mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
Add comprehensive development documentation and automation
- Add CLAUDE.md with development guidance and architecture documentation - Add HIVE_DEVELOPMENT_PLAN.md with detailed roadmap integrating all TODO items - Update README.md with development setup and engineering philosophy - Transform TODO.md into structured roadmap with phase integration - Add development automation scripts (clone-deps.sh, dev-setup.sh) - Add configuration templates and systemd service files This establishes the foundation for scalable multi-repository development and provides clear guidance for contributors and development workflows.
This commit is contained in:
Ben Duncan
@@ -0,0 +1,192 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Project Overview
|
||||
|
||||
This is the Hive platform - an AWS-compatible infrastructure stack for bare-metal, edge, and on-premise deployments. Hive orchestrates multiple independent components that work together to provide AWS-compatible services.
|
||||
|
||||
## Component Repositories
|
||||
|
||||
The Hive platform consists of these independent repositories:
|
||||
|
||||
- **[viperblock](../viperblock/)** - High-performance block storage service (EBS alternative) with WAL-backed storage
|
||||
- **[predastore](../predastore/)** - S3-compatible object storage service with Reed-Solomon erasure coding
|
||||
- **nbdkit/** - Network Block Device (NBD) server for exporting block devices (upstream project)
|
||||
- **hive/** (this repo) - VM orchestration service and platform coordinator
|
||||
|
||||
Each component can be used independently or as part of the integrated Hive stack. See component-specific `CLAUDE.md` files for detailed development guidance:
|
||||
- `../viperblock/CLAUDE.md` - Viperblock development guide
|
||||
- `../predastore/CLAUDE.md` - Predastore development guide
|
||||
|
||||
## Cross-Repository Development Setup
|
||||
|
||||
For platform development, clone all repositories to the same parent directory:
|
||||
```bash
|
||||
mkdir mulga
|
||||
cd mulga
|
||||
git clone <hive-repo-url> hive
|
||||
git clone <viperblock-repo-url> viperblock
|
||||
git clone <predastore-repo-url> predastore
|
||||
```
|
||||
|
||||
The `hive/go.mod` uses local replace directives for cross-component development:
|
||||
```go
|
||||
replace github.com/mulgadc/viperblock => ../viperblock
|
||||
replace github.com/mulgadc/predastore => ../predastore
|
||||
```
|
||||
|
||||
## Build Commands
|
||||
|
||||
### Hive (Platform Orchestrator)
|
||||
```bash
|
||||
make build # Build the hive binary
|
||||
make test # Run Go tests
|
||||
make bench # Run benchmarks
|
||||
make clean # Clean build artifacts
|
||||
```
|
||||
|
||||
### Component Build Commands
|
||||
For detailed build instructions, see component-specific CLAUDE.md files:
|
||||
|
||||
**Viperblock** (`../viperblock/CLAUDE.md`):
|
||||
```bash
|
||||
cd ../viperblock && make build # Build sfs, vblock binaries + NBD plugin
|
||||
```
|
||||
|
||||
**Predastore** (`../predastore/CLAUDE.md`):
|
||||
```bash
|
||||
cd ../predastore && make build # Build s3d binary
|
||||
```
|
||||
|
||||
**NBDkit** (system dependency):
|
||||
```bash
|
||||
cd nbdkit && autoreconf -i && ./configure && make && make install
|
||||
```
|
||||
|
||||
## Architecture Notes
|
||||
|
||||
### Message-Driven Microservices Architecture
|
||||
|
||||
Hive uses a **scalable message-driven architecture** that separates API handling from request processing:
|
||||
|
||||
```
|
||||
AWS SDK (custom endpoint) → Hive API Gateway → NATS Topics → Specialized Daemons
|
||||
↑ ↓
|
||||
AWS-compatible response ← NATS Topics ← Processing Results
|
||||
```
|
||||
|
||||
#### Core Components:
|
||||
|
||||
1. **Hive API Gateway Service** - TLS endpoint that handles AWS SDK requests
|
||||
2. **Specialized Daemon Services** - Horizontally scalable workers for each AWS service type
|
||||
3. **NATS Message Broker** - Topic-based routing with queue groups for load balancing
|
||||
|
||||
#### Current Implementation (daemon.go):
|
||||
- **Gateway Logic**: `StartAWSGateway()` handles AWS API calls on port 9999
|
||||
- **NATS Integration**: Uses request-response pattern: `d.natsConn.Request("ec2.runinstances", data, 30s)`
|
||||
- **Queue Groups**: Load balancing via `"hive-workers"` queue group
|
||||
- **Service Coordination**: All inter-service communication via NATS topics
|
||||
|
||||
### Inter-component Dependencies
|
||||
- **Service Startup Order**: NATS → Predastore → Viperblock → NBDkit → Gateway → Daemons
|
||||
- **Message Flow**: Gateway translates AWS calls to NATS messages, daemons process and respond
|
||||
- **Scaling**: Multiple daemon instances can subscribe to same topics for horizontal scaling
|
||||
- All Go projects use local module replacements for development
|
||||
|
||||
### Storage Backend Integration
|
||||
- **Viperblock Integration**: EBS volumes mounted via NATS messaging (`ebs.mount` topic)
|
||||
- **Predastore Integration**: S3 operations proxied through dedicated S3 daemons
|
||||
- **NBDkit Integration**: Block device access for VM storage
|
||||
- **Volume Lifecycle**: Coordinated through NATS between EC2 and EBS daemons
|
||||
|
||||
### Key Design Patterns
|
||||
- **Message-Driven**: All service communication via NATS topics (ec2.*, ebs.*, s3.*, vpc.*)
|
||||
- **Queue Groups**: Load balancing and fault tolerance via NATS queue groups
|
||||
- **Request-Response**: 30-second timeout pattern for AWS API compatibility
|
||||
- **Service Specialization**: Dedicated daemon types for each AWS service
|
||||
- **Horizontal Scaling**: Multiple daemon instances handle same topic types
|
||||
|
||||
### NATS Topic Structure
|
||||
|
||||
The system uses structured NATS topics for service communication:
|
||||
|
||||
```bash
|
||||
# EC2 Service Topics:
|
||||
ec2.runinstances # Launch new instances
|
||||
ec2.describeinstances # Query instance status
|
||||
ec2.startinstances # Start stopped instances
|
||||
ec2.stopinstances # Stop running instances
|
||||
ec2.terminateinstances # Terminate instances
|
||||
|
||||
# EBS Service Topics:
|
||||
ebs.createvolume # Create new EBS volume
|
||||
ebs.attachvolume # Attach volume to instance
|
||||
ebs.detachvolume # Detach volume
|
||||
ebs.describevolumes # List volumes
|
||||
ebs.mount # Mount volume (internal)
|
||||
ebs.unmount # Unmount volume (internal)
|
||||
|
||||
# S3 Service Topics:
|
||||
s3.createbucket # Create S3 bucket
|
||||
s3.putobject # Store object
|
||||
s3.getobject # Retrieve object
|
||||
s3.listbuckets # List buckets
|
||||
|
||||
# VPC Service Topics:
|
||||
vpc.createvpc # Create VPC
|
||||
vpc.createsubnet # Create subnet
|
||||
vpc.describevpcs # List VPCs
|
||||
|
||||
# Health and Discovery:
|
||||
health.ec2.daemon.{id} # Daemon health heartbeats
|
||||
discovery.services # Service registration
|
||||
```
|
||||
|
||||
### Go Module Structure
|
||||
All Go projects use:
|
||||
- Go 1.23+ with module mode
|
||||
- Local replace directives for cross-component development
|
||||
- Standard build flags: `-ldflags "-s -w"` for optimized binaries
|
||||
- Test environment variable: `LOG_IGNORE=1` to suppress logs during testing
|
||||
|
||||
### Development Workflow
|
||||
|
||||
#### Quick Setup (Recommended)
|
||||
```bash
|
||||
# 1. Clone dependencies (if not already done)
|
||||
./scripts/clone-deps.sh
|
||||
|
||||
# 2. Setup development environment
|
||||
./scripts/dev-setup.sh
|
||||
|
||||
# 3. Start all services
|
||||
./scripts/start-dev.sh
|
||||
|
||||
# 4. Stop all services when done
|
||||
./scripts/stop-dev.sh
|
||||
```
|
||||
|
||||
#### Manual Development Process
|
||||
1. **Cross-repo Setup**: Clone all repositories to same parent directory
|
||||
2. **Service Dependencies**: Start services in order: NATS → Predastore → Viperblock → NBDkit → Gateway → Daemons
|
||||
3. **Build Order**: Build components: predastore → viperblock → hive
|
||||
4. **Testing**: Use `make test` in each component to verify changes
|
||||
5. **Message-Driven Development**: Test NATS message flows between gateway and daemons
|
||||
6. **Scaling**: Test with multiple daemon instances subscribing to same topics
|
||||
|
||||
#### Development Tools
|
||||
- **Hot Reloading**: Use `air` for automatic restarts during development
|
||||
- **TLS Certificates**: Auto-generated self-signed certificates for HTTPS endpoints
|
||||
- **Log Monitoring**: Centralized logs in `data/logs/` directory
|
||||
- **AWS CLI Testing**: Test endpoints with `aws --endpoint-url https://localhost:9999`
|
||||
|
||||
### Development Roadmap
|
||||
For major feature development and architectural changes, refer to `HIVE_DEVELOPMENT_PLAN.md` which contains:
|
||||
- Comprehensive roadmap for transforming Hive from proof-of-concept to beta release
|
||||
- Detailed implementation phases with specific tasks and file structures
|
||||
- AWS API compatibility implementation guidelines
|
||||
- Development environment automation plans
|
||||
- Testing and validation frameworks
|
||||
|
||||
When implementing new AWS services or enhancing existing functionality, follow the structured approach outlined in the development plan.
|
||||
@@ -0,0 +1,618 @@
|
||||
# Hive Development Plan: Proof-of-Concept to Beta Release
|
||||
|
||||
This document provides a comprehensive roadmap for transforming Hive from a proof-of-concept into a production-ready AWS-compatible infrastructure platform. Use this as the primary guide for development work on the Hive codebase.
|
||||
|
||||
## Project Overview
|
||||
|
||||
**Goal**: Transform Hive into a beta release that provides full AWS CLI compatibility for EC2, EBS, S3, and VPC services with seamless development environment automation.
|
||||
|
||||
**Current State**: Working proof-of-concept with:
|
||||
- Basic EC2 instance provisioning via NATS
|
||||
- Viperblock integration for EBS volumes
|
||||
- Predastore S3-compatible storage
|
||||
- TLS-enabled HTTP gateway with partial AWS API support
|
||||
- QEMU/KVM-based virtualization
|
||||
|
||||
**Target State**: Production-ready platform with:
|
||||
- Full AWS CLI compatibility (`aws ec2 run-instances`, `aws s3 mb`, etc.)
|
||||
- Auto-scaling development environment with hot reloading
|
||||
- Comprehensive AWS API coverage (EC2, EBS, S3, VPC)
|
||||
- Multi-node deployment capability
|
||||
- Robust testing framework with >90% AWS compatibility
|
||||
|
||||
## Architecture Analysis
|
||||
|
||||
### Scalable Message-Driven Architecture
|
||||
|
||||
Hive uses a **message-driven microservices architecture** that separates API handling from request processing:
|
||||
|
||||
```
|
||||
AWS SDK (custom endpoint) → Hive API Gateway → NATS Topics → Specialized Daemons
|
||||
↑ ↓
|
||||
AWS-compatible response ← NATS Topics ← Processing Results
|
||||
```
|
||||
|
||||
#### **Core Components**:
|
||||
|
||||
1. **Hive API Gateway Service** (`hive/gateway/`)
|
||||
- Handles AWS SDK requests on custom endpoints (https://localhost:9999)
|
||||
- Parses AWS API calls (EC2, S3, VPC, EBS)
|
||||
- Translates to NATS messages and broadcasts on topics
|
||||
- Receives NATS responses and formats AWS-compatible XML/JSON responses
|
||||
- Handles AWS Signature V4 authentication
|
||||
|
||||
2. **Specialized Daemon Services** (horizontally scalable)
|
||||
- **EC2 Daemons**: Listen on `ec2.*` topics, manage QEMU/KVM instances
|
||||
- **EBS Daemons**: Listen on `ebs.*` topics, coordinate with Viperblock
|
||||
- **S3 Daemons**: Listen on `s3.*` topics, proxy to Predastore
|
||||
- **VPC Daemons**: Listen on `vpc.*` topics, manage networking
|
||||
|
||||
3. **NATS Message Broker** (clustering capable)
|
||||
- Topic-based routing: `ec2.runinstances`, `ebs.createvolume`, `s3.createbucket`
|
||||
- Queue groups for load balancing: multiple daemons can handle same topic
|
||||
- Request-response pattern with timeouts
|
||||
- Service discovery and health monitoring
|
||||
|
||||
### Current Strengths
|
||||
- **Message-Driven Foundation**: Already implements NATS request-response pattern (daemon.go:351)
|
||||
- **Queue Groups**: Uses `"hive-workers"` queue group for load balancing (daemon.go:749)
|
||||
- **TLS Gateway**: Working AWS SDK integration with TLS (daemon.go:330)
|
||||
- **Service Integration**: Viperblock (EBS) and Predastore (S3) via NATS messaging
|
||||
- **VM Management**: QEMU/KVM with QMP integration for lifecycle management
|
||||
|
||||
### Scalability Benefits
|
||||
- **Horizontal Scaling**: Add more daemon instances to handle increased load
|
||||
- **Service Isolation**: Each AWS service type runs in dedicated daemons
|
||||
- **Fault Tolerance**: NATS queue groups provide automatic failover
|
||||
- **Development Flexibility**: Services can be developed and deployed independently
|
||||
|
||||
### Areas for Enhancement
|
||||
- **API Gateway Separation**: Extract gateway logic from daemon.go into dedicated service
|
||||
- **Service Specialization**: Create dedicated daemon types for each AWS service
|
||||
- **Authentication**: Implement full AWS Signature V4 validation
|
||||
- **Auto-scaling**: Dynamic daemon scaling based on NATS queue depth
|
||||
|
||||
## Development Phases
|
||||
|
||||
### Phase 0: Distributed Systems Foundation (1-2 weeks)
|
||||
|
||||
**Priority**: CRITICAL - Multi-node coordination foundation
|
||||
|
||||
#### Task 0.1: Service Registry and Discovery
|
||||
```bash
|
||||
# Distributed coordination infrastructure:
|
||||
hive/cluster/ # Cluster coordination package
|
||||
hive/cluster/gossip.go # Gossip protocol for node discovery
|
||||
hive/cluster/raft.go # RAFT consensus for configuration
|
||||
hive/cluster/registry.go # Service registry and health monitoring
|
||||
hive/cluster/config.go # Distributed configuration management
|
||||
```
|
||||
|
||||
**Integration with Existing TODO Items**:
|
||||
- ✅ **TODO #7**: "Add gossip and RAFT protocol for Hive nodes to communicate and sync"
|
||||
- ✅ **TODO #8**: "Add support for `etcd` or simple key/value for configuration sync"
|
||||
|
||||
**Key Features**:
|
||||
- **Node Discovery**: Gossip protocol for automatic node discovery and failure detection
|
||||
- **Configuration Consensus**: RAFT-based distributed configuration management
|
||||
- **Service Health**: Cluster-wide service health monitoring and failover
|
||||
- **State Synchronization**: Distributed state management for multi-node deployments
|
||||
|
||||
#### Task 0.2: Cluster-Aware NATS Configuration
|
||||
```bash
|
||||
# Enhanced NATS clustering:
|
||||
hive/nats/cluster.go # NATS cluster configuration with service discovery
|
||||
hive/nats/election.go # Leader election for gateway services
|
||||
hive/nats/sharding.go # Topic sharding for scalability
|
||||
```
|
||||
|
||||
### Phase 1: Development Environment Automation (2-3 weeks)
|
||||
|
||||
**Priority**: CRITICAL - Essential for productive development
|
||||
|
||||
#### Task 1.1: Multi-Service Orchestration Framework
|
||||
```bash
|
||||
# Create new files:
|
||||
scripts/dev-setup.sh # One-command environment setup
|
||||
tools/hive-dev/main.go # CLI tool for development services
|
||||
tools/hive-dev/cmd/start.go # Start all services with dependencies
|
||||
tools/hive-dev/cmd/stop.go # Graceful service shutdown
|
||||
tools/hive-dev/cmd/status.go # Service health checking
|
||||
tools/hive-dev/cmd/logs.go # Aggregated logging
|
||||
tools/hive-dev/cmd/scale.go # Scale daemon instances
|
||||
```
|
||||
|
||||
**Service Startup Order**:
|
||||
1. **NATS Server** (message broker)
|
||||
2. **Predastore** (S3-compatible storage)
|
||||
3. **Viperblock** (EBS block storage)
|
||||
4. **NBDkit** (network block device)
|
||||
5. **Hive API Gateway** (AWS API endpoint)
|
||||
6. **Hive Daemons** (EC2, EBS, S3, VPC workers)
|
||||
|
||||
**Key Features**:
|
||||
- Multi-daemon management: start multiple EC2/EBS/VPC daemon instances
|
||||
- NATS queue monitoring and daemon auto-scaling
|
||||
- Health checks via NATS service discovery
|
||||
- TLS certificate generation for gateway service
|
||||
|
||||
#### Task 1.2: Auto-restart Development Environment
|
||||
```bash
|
||||
# Configuration files:
|
||||
.air.toml # Root air configuration
|
||||
hive/.air.toml # Hive service hot-reload config
|
||||
predastore/.air.toml # Predastore hot-reload config
|
||||
viperblock/.air.toml # Viperblock hot-reload config
|
||||
tools/file-watcher/main.go # Custom file watcher for cross-service dependencies
|
||||
```
|
||||
|
||||
**Integration Points**:
|
||||
- File watcher monitors Go files across all components
|
||||
- Graceful service restart on code changes
|
||||
- Automatic service dependency restart when interfaces change
|
||||
- Development log aggregation and filtering
|
||||
|
||||
#### Task 1.3: Configuration Management
|
||||
```bash
|
||||
# Configuration system:
|
||||
config/dev.yaml # Development environment config
|
||||
config/test.yaml # Testing environment config
|
||||
config/prod.yaml # Production environment config
|
||||
tools/config-gen/main.go # Configuration generator and validator
|
||||
```
|
||||
|
||||
### Phase 2: AWS API Model Implementation (3-4 weeks)
|
||||
|
||||
**Priority**: HIGH - Foundation for all AWS compatibility
|
||||
|
||||
#### Task 2.1: Smithy-Based AWS API Code Generation Framework
|
||||
```bash
|
||||
# Code generation tools:
|
||||
tools/aws-codegen/main.go # Main code generator
|
||||
tools/aws-codegen/smithy/ # Smithy model parser
|
||||
tools/aws-codegen/templates/ # Go code templates
|
||||
tools/aws-codegen/models/ # Downloaded AWS Smithy models
|
||||
hive/services/generated/ # Generated AWS types and interfaces
|
||||
```
|
||||
|
||||
**Integration with Existing TODO Items**:
|
||||
- ✅ **TODO #9**: "Use Smithy e.g `sdk-codegen/aws-models/ec2.json` to generate struct/XML input/output"
|
||||
|
||||
**Smithy Model Integration**:
|
||||
- Download official AWS Smithy models from `sdk-codegen/aws-models/`
|
||||
- Parse `ec2.json`, `s3.json`, `vpc.json` service definitions
|
||||
- Generate Go structs with proper JSON/XML tags matching AWS exactly
|
||||
- Create operation interfaces that mirror AWS SDK v2 patterns
|
||||
- Auto-generate request/response validation logic
|
||||
|
||||
**Generated Structure Example**:
|
||||
```go
|
||||
// hive/services/generated/ec2/types.go
|
||||
type RunInstancesInput struct {
|
||||
ImageId *string `json:"ImageId" xml:"ImageId"`
|
||||
InstanceType InstanceType `json:"InstanceType" xml:"InstanceType"`
|
||||
KeyName *string `json:"KeyName" xml:"KeyName"`
|
||||
MaxCount *int32 `json:"MaxCount" xml:"MaxCount"`
|
||||
MinCount *int32 `json:"MinCount" xml:"MinCount"`
|
||||
SecurityGroupIds []string `json:"SecurityGroupIds" xml:"SecurityGroupId"`
|
||||
SubnetId *string `json:"SubnetId" xml:"SubnetId"`
|
||||
UserData *string `json:"UserData" xml:"UserData"`
|
||||
// ... all AWS EC2 fields
|
||||
}
|
||||
|
||||
// hive/services/generated/ec2/service.go
|
||||
type EC2Service interface {
|
||||
RunInstances(ctx context.Context, input *RunInstancesInput) (*RunInstancesOutput, error)
|
||||
DescribeInstances(ctx context.Context, input *DescribeInstancesInput) (*DescribeInstancesOutput, error)
|
||||
StartInstances(ctx context.Context, input *StartInstancesInput) (*StartInstancesOutput, error)
|
||||
StopInstances(ctx context.Context, input *StopInstancesInput) (*StopInstancesOutput, error)
|
||||
TerminateInstances(ctx context.Context, input *TerminateInstancesInput) (*TerminateInstancesOutput, error)
|
||||
CreateVolume(ctx context.Context, input *CreateVolumeInput) (*CreateVolumeOutput, error)
|
||||
AttachVolume(ctx context.Context, input *AttachVolumeInput) (*AttachVolumeOutput, error)
|
||||
DetachVolume(ctx context.Context, input *DetachVolumeInput) (*DetachVolumeOutput, error)
|
||||
// ... all EC2 operations
|
||||
}
|
||||
```
|
||||
|
||||
#### Task 2.2: Service Implementation Structure
|
||||
```bash
|
||||
# Service implementations:
|
||||
hive/services/ec2/ # EC2 service implementation
|
||||
hive/services/ec2/instances.go # Instance management
|
||||
hive/services/ec2/volumes.go # EBS volume management
|
||||
hive/services/ec2/security.go # Security groups and key pairs
|
||||
hive/services/s3/ # S3 service (proxy to Predastore)
|
||||
hive/services/vpc/ # VPC networking service
|
||||
hive/services/iam/ # IAM service for authentication
|
||||
```
|
||||
|
||||
### Phase 3: Scalable Gateway and Daemon Architecture (2-3 weeks)
|
||||
|
||||
**Priority**: CRITICAL - Core scalable architecture
|
||||
|
||||
#### Task 3.1: AWS Gateway Service (awsd)
|
||||
```bash
|
||||
# New dedicated AWS gateway service:
|
||||
cmd/awsd/main.go # AWS gateway service entry point (TODO #4)
|
||||
hive/services/awsd/ # AWS gateway service package
|
||||
hive/services/awsd/server.go # TLS HTTP server (extracted from daemon.go:252)
|
||||
hive/services/awsd/router.go # AWS service routing
|
||||
hive/services/awsd/nats.go # NATS client for broadcasting requests
|
||||
hive/services/awsd/auth/ # AWS Signature V4 authentication
|
||||
hive/services/awsd/handlers/ec2.go # EC2 → NATS translation
|
||||
hive/services/awsd/handlers/s3.go # S3 → NATS translation
|
||||
hive/services/awsd/handlers/ebs.go # EBS → NATS translation
|
||||
hive/services/awsd/handlers/vpc.go # VPC → NATS translation
|
||||
hive/services/awsd/middleware/ # Request/response middleware
|
||||
```
|
||||
|
||||
**Integration with Existing TODO Items**:
|
||||
- ✅ **TODO #4**: "Implement AWS Server HTTP gateway (awsd) to accept commands to the AWS control plane"
|
||||
- ✅ **TODO #5**: "Implement command input/output from aws-sdk-go-v2"
|
||||
|
||||
**Gateway Responsibilities**:
|
||||
- Listen on `:9999` for AWS SDK requests
|
||||
- Parse AWS API calls and validate signatures using aws-sdk-go-v2 structures
|
||||
- Translate to NATS messages: `aws.Request("ec2.runinstances", data, 30s)`
|
||||
- Format NATS responses back to AWS XML/JSON
|
||||
- Handle timeouts and error responses
|
||||
|
||||
#### Task 3.2: Specialized Service Daemons
|
||||
```bash
|
||||
# Refactor current daemon.go into specialized services (TODO #2):
|
||||
cmd/hived/main.go # Main Hive coordination daemon
|
||||
cmd/vpcd/main.go # VPC networking daemon
|
||||
|
||||
hive/services/hived/ # Main Hive service (TODO #2)
|
||||
hive/services/hived/instances.go # Instance lifecycle (from daemon.go:1874)
|
||||
hive/services/hived/qemu.go # QEMU/KVM management
|
||||
hive/services/hived/nats.go # NATS subscriber: "ec2.*"
|
||||
hive/services/hived/ebs.go # EBS volume coordination
|
||||
|
||||
hive/services/vpcd/ # VPC daemon package (TODO #3)
|
||||
hive/services/vpcd/networking.go # Open vSwitch integration
|
||||
hive/services/vpcd/subnets.go # Subnet management
|
||||
hive/services/vpcd/nats.go # NATS subscriber: "vpc.*"
|
||||
```
|
||||
|
||||
**Service Architecture Changes**:
|
||||
- **hived**: Main compute service (replaces current daemon.go functionality)
|
||||
- **awsd**: AWS API gateway (new service for AWS SDK compatibility)
|
||||
- **vpcd**: VPC networking service (new service for network management)
|
||||
- **Predastore**: S3 service (existing, integrated via NATS)
|
||||
- **Viperblock**: EBS service (existing, integrated via NATS)
|
||||
|
||||
**NATS Topic Structure**:
|
||||
```bash
|
||||
# Request topics (Gateway → Daemons):
|
||||
ec2.runinstances # Launch new instances
|
||||
ec2.describeinstances # Query instance status
|
||||
ec2.startinstances # Start stopped instances
|
||||
ec2.stopinstances # Stop running instances
|
||||
ec2.terminateinstances # Terminate instances
|
||||
|
||||
ebs.createvolume # Create new EBS volume
|
||||
ebs.attachvolume # Attach volume to instance
|
||||
ebs.detachvolume # Detach volume
|
||||
ebs.describevolumes # List volumes
|
||||
|
||||
s3.createbucket # Create S3 bucket
|
||||
s3.putobject # Store object
|
||||
s3.getobject # Retrieve object
|
||||
|
||||
vpc.createvpc # Create VPC
|
||||
vpc.createsubnet # Create subnet
|
||||
# ... etc
|
||||
```
|
||||
|
||||
#### Task 3.3: Horizontal Scaling Support
|
||||
```bash
|
||||
# Scaling infrastructure:
|
||||
tools/hive-scaler/main.go # Daemon scaling utility
|
||||
hive/scaling/ # Auto-scaling logic
|
||||
hive/scaling/metrics.go # NATS queue depth monitoring
|
||||
hive/scaling/policies.go # Scaling policies (CPU, queue depth)
|
||||
```
|
||||
|
||||
**Scaling Features**:
|
||||
- Multiple daemon instances subscribe to same NATS topics
|
||||
- NATS queue groups (`"hive-workers"`) provide load balancing
|
||||
- Monitor queue depths: scale up daemons when queues grow
|
||||
- Scale down during low load periods
|
||||
|
||||
### Phase 4: Service Integration and Orchestration (3-4 weeks)
|
||||
|
||||
**Priority**: MEDIUM-HIGH - Service coordination
|
||||
|
||||
#### Task 4.1: NATS Clustering and Service Discovery
|
||||
```bash
|
||||
# NATS clustering and coordination:
|
||||
hive/nats/ # NATS infrastructure management
|
||||
hive/nats/cluster.go # NATS cluster configuration
|
||||
hive/nats/discovery.go # Service discovery via NATS
|
||||
hive/nats/health.go # Health check messaging patterns
|
||||
hive/nats/auth.go # NATS authentication and ACLs
|
||||
```
|
||||
|
||||
**Features**:
|
||||
- NATS clustering for high availability
|
||||
- Service registration: daemons announce capabilities on startup
|
||||
- Health monitoring via heartbeat topics: `health.ec2.daemon.1`, `health.ebs.daemon.2`
|
||||
- Load balancing metrics: queue depths, response times per daemon
|
||||
- Auto-discovery of available daemon services
|
||||
|
||||
#### Task 4.2: Predastore S3 Integration
|
||||
```bash
|
||||
# S3 service integration:
|
||||
hive/services/s3/proxy.go # Direct proxy to Predastore
|
||||
hive/services/s3/buckets.go # Bucket management
|
||||
hive/services/s3/events.go # S3 event notifications
|
||||
```
|
||||
|
||||
#### Task 4.3: Enhanced Viperblock Integration
|
||||
```bash
|
||||
# EBS service integration:
|
||||
hive/services/ebs/volumes.go # Volume lifecycle management
|
||||
hive/services/ebs/snapshots.go # Snapshot management
|
||||
hive/services/ebs/encryption.go # Volume encryption
|
||||
hive/services/ebs/monitoring.go # Performance monitoring
|
||||
```
|
||||
|
||||
### Phase 5: Infrastructure Services (4-5 weeks)
|
||||
|
||||
**Priority**: MEDIUM - Advanced features
|
||||
|
||||
#### Task 5.1: VPC Network Management with Open vSwitch
|
||||
```bash
|
||||
# VPC implementation with openvs-switch:
|
||||
hive/services/vpcd/ # VPC service implementation (TODO #3)
|
||||
hive/services/vpcd/ovs.go # Open vSwitch integration and management
|
||||
hive/services/vpcd/subnets.go # Subnet management with OVS bridges
|
||||
hive/services/vpcd/routing.go # Route table management via OVS flows
|
||||
hive/services/vpcd/security.go # Security groups via OVS rules
|
||||
hive/services/vpcd/gateways.go # Internet/NAT gateway management
|
||||
hive/networking/ovs/ # Open vSwitch wrapper and utilities
|
||||
```
|
||||
|
||||
**Integration with Existing TODO Items**:
|
||||
- ✅ **TODO #3**: "Implement VPC support using openvs-switch (add VPC and subnets, single AZ to start)"
|
||||
|
||||
**Open vSwitch Integration**:
|
||||
- **OVS Bridge Management**: Create and manage virtual switches for VPCs
|
||||
- **VLAN Tagging**: Subnet isolation using VLAN tags
|
||||
- **Flow Rules**: Security group implementation via OpenFlow rules
|
||||
- **Port Management**: VM network interface attachment to OVS ports
|
||||
- **Single AZ**: Initial implementation focuses on single availability zone
|
||||
|
||||
#### Task 5.2: Resource Management Enhancement
|
||||
```bash
|
||||
# Resource management:
|
||||
hive/resources/ # Resource management system
|
||||
hive/resources/quotas.go # Resource quotas and limits
|
||||
hive/resources/allocation.go # Cluster-wide resource allocation
|
||||
hive/resources/scaling.go # Auto-scaling capabilities
|
||||
hive/resources/tagging.go # Resource tagging and cost allocation
|
||||
```
|
||||
|
||||
#### Task 5.3: Advanced VM Features
|
||||
```bash
|
||||
# VM enhancements:
|
||||
hive/services/metadata/ # Instance metadata service (169.254.169.254)
|
||||
hive/vm/userdata.go # Enhanced user data and cloud-init
|
||||
hive/vm/snapshots.go # VM backup and snapshot
|
||||
hive/vm/migration.go # VM migration capabilities
|
||||
```
|
||||
|
||||
### Phase 6: Testing and Validation Framework (2-3 weeks)
|
||||
|
||||
**Priority**: HIGH - Quality assurance
|
||||
|
||||
#### Task 6.1: AWS CLI Integration Tests
|
||||
```bash
|
||||
# Comprehensive testing:
|
||||
tests/integration/aws-cli/ # AWS CLI integration tests
|
||||
tests/compatibility/ # AWS compatibility validation
|
||||
tests/performance/ # Performance and load tests
|
||||
tests/scenarios/ # End-to-end scenario tests
|
||||
```
|
||||
|
||||
**Test Coverage**:
|
||||
- All AWS CLI commands supported by Hive
|
||||
- Parameter validation and error handling
|
||||
- Performance benchmarks vs. real AWS
|
||||
- Multi-service workflow testing
|
||||
|
||||
#### Task 6.2: Service Testing Framework
|
||||
```bash
|
||||
# Testing framework:
|
||||
tests/unit/ # Unit tests for all services
|
||||
tests/integration/ # Service integration tests
|
||||
tests/chaos/ # Chaos engineering tests
|
||||
tools/test-runner/ # Custom test runner and reporter
|
||||
```
|
||||
|
||||
#### Task 6.3: Development Testing Tools
|
||||
```bash
|
||||
# Development tools:
|
||||
tools/mock-aws/ # Mock AWS services for offline dev
|
||||
tools/test-data/ # Test data generators
|
||||
tools/test-env/ # Test environment management
|
||||
```
|
||||
|
||||
### Phase 7: Production Deployment and Packaging (2-3 weeks)
|
||||
|
||||
**Priority**: HIGH - Production readiness
|
||||
|
||||
#### Task 7.1: Binary Compilation and Distribution
|
||||
```bash
|
||||
# Build and packaging system:
|
||||
scripts/build-release.sh # Multi-platform binary compilation
|
||||
scripts/install.sh # Production installation script
|
||||
packaging/ # Package management files
|
||||
packaging/debian/ # Debian/Ubuntu packages
|
||||
packaging/rpm/ # RedHat/CentOS packages
|
||||
packaging/docker/ # Docker containers for production
|
||||
```
|
||||
|
||||
**Integration with Existing TODO Items**:
|
||||
- ✅ **TODO #1**: "Add support for binary compile and simple install.sh to setup a new node"
|
||||
|
||||
**Production Build Features**:
|
||||
- **Cross-Platform Compilation**: Linux (amd64, arm64), Windows, macOS binaries
|
||||
- **Static Linking**: Self-contained binaries with minimal dependencies
|
||||
- **Installation Scripts**: One-command node setup for production deployments
|
||||
- **Package Management**: Native packages for major Linux distributions
|
||||
|
||||
#### Task 7.2: System Service Integration
|
||||
```bash
|
||||
# System integration:
|
||||
systemd/ # Systemd service files
|
||||
systemd/hived.service # Main Hive compute daemon
|
||||
systemd/awsd.service # AWS gateway service
|
||||
systemd/vpcd.service # VPC networking daemon
|
||||
init/ # SysV init scripts for legacy systems
|
||||
scripts/node-setup.sh # Complete node configuration script
|
||||
```
|
||||
|
||||
**Integration with Existing TODO Items**:
|
||||
- ✅ **TODO #6**: "Implement `hive.service` for service management and boot"
|
||||
|
||||
**System Integration Features**:
|
||||
- **Systemd Services**: Proper service definitions with dependencies and restart policies
|
||||
- **Boot Integration**: Automatic service startup on system boot
|
||||
- **Health Monitoring**: Service health checks and automatic restart on failure
|
||||
- **Log Management**: Proper logging integration with journald/syslog
|
||||
- **User Management**: Dedicated system users and proper permissions
|
||||
|
||||
#### Task 7.3: Production Configuration Management
|
||||
```bash
|
||||
# Production configuration:
|
||||
config/production.yaml # Production configuration template
|
||||
config/cluster.yaml # Multi-node cluster configuration
|
||||
scripts/cluster-init.sh # Cluster initialization script
|
||||
tools/hive-admin/ # Cluster administration tool
|
||||
tools/hive-admin/node.go # Node management commands
|
||||
tools/hive-admin/cluster.go # Cluster operations
|
||||
tools/hive-admin/health.go # Health monitoring and diagnostics
|
||||
```
|
||||
|
||||
**Production Features**:
|
||||
- **Configuration Templates**: Production-ready configuration with security defaults
|
||||
- **Cluster Bootstrapping**: Automated multi-node cluster setup
|
||||
- **Administrative Tools**: Command-line tools for cluster management
|
||||
- **Security Hardening**: Production security configurations and best practices
|
||||
|
||||
## Implementation Guidelines
|
||||
|
||||
### Development Workflow
|
||||
1. **Start with Phase 1** - Essential for productive development
|
||||
2. **Use TDD approach** - Write tests before implementation
|
||||
3. **Maintain backward compatibility** - Don't break existing functionality
|
||||
4. **Document as you go** - Update CLAUDE.md with new patterns
|
||||
5. **Test AWS CLI compatibility** - Validate every implemented operation
|
||||
|
||||
### Code Organization Principles
|
||||
- **Generate don't hand-write** AWS API types - Use code generation for consistency
|
||||
- **Separate concerns** - Keep service logic separate from HTTP handling
|
||||
- **Use interfaces** - Enable testing and future extensibility
|
||||
- **Follow existing patterns** - Maintain consistency with current codebase
|
||||
- **Error handling** - Implement proper AWS-compatible error responses
|
||||
|
||||
### Key Integration Points and Refactoring Guide
|
||||
|
||||
#### **Current Code Mapping to New Architecture**:
|
||||
|
||||
**From `hive/daemon/daemon.go` → Multiple Services**:
|
||||
|
||||
1. **Gateway Service** (`cmd/hive-gateway/main.go`):
|
||||
- **Extract**: `StartAWSGateway()` (daemon.go:252-332)
|
||||
- **Extract**: HTTP handlers (daemon.go:337-478)
|
||||
- **Extract**: AWS XML response generation (daemon.go:586-627)
|
||||
- **Keep**: NATS client for request broadcasting
|
||||
|
||||
2. **EC2 Daemon Service** (`cmd/hive-ec2-daemon/main.go`):
|
||||
- **Extract**: `handleEC2Launch()` (daemon.go:1044-1107)
|
||||
- **Extract**: `LaunchInstance()` (daemon.go:1874-1960)
|
||||
- **Extract**: `StartInstance()` (daemon.go:1962-2168)
|
||||
- **Extract**: QMP client management (daemon.go:1815-1872)
|
||||
- **Keep**: NATS subscriber patterns
|
||||
|
||||
3. **EBS Daemon Service** (`cmd/hive-ebs-daemon/main.go`):
|
||||
- **Extract**: `MountVolumes()` (daemon.go:2170-2221)
|
||||
- **Extract**: Viperblock integration patterns (daemon.go:1265-1457)
|
||||
- **New**: Volume lifecycle management
|
||||
- **New**: EBS-specific NATS topics
|
||||
|
||||
#### **NATS Message Patterns (Already Working)**:
|
||||
- **Request Pattern**: `d.natsConn.Request("ec2.runinstances", requestData, 30*time.Second)` (daemon.go:351)
|
||||
- **Subscribe Pattern**: `d.natsConn.QueueSubscribe("ec2.launch", "hive-workers", d.handleEC2Launch)` (daemon.go:749)
|
||||
- **Queue Groups**: Already uses `"hive-workers"` for load balancing
|
||||
|
||||
#### **Configuration Extensions** (`hive/config/config.go`):
|
||||
- Add daemon-specific configurations
|
||||
- Service discovery endpoints
|
||||
- NATS clustering configuration
|
||||
- Auto-scaling policies
|
||||
|
||||
#### **Existing Integration Patterns to Preserve**:
|
||||
- **Viperblock Integration**: EBS mounting patterns (daemon.go:2170-2221)
|
||||
- **Predastore Integration**: S3 client patterns (daemon.go:1659-1676)
|
||||
- **QMP Integration**: VM lifecycle management (daemon.go:922-989)
|
||||
- **NATS Messaging**: Request-response patterns throughout daemon.go
|
||||
|
||||
### Success Criteria
|
||||
- **AWS CLI Commands Work**: `aws ec2 run-instances`, `aws s3 mb`, `aws ec2 create-volume` function identically to real AWS
|
||||
- **Development Experience**: `./tools/hive-dev start` brings up entire environment in <2 minutes
|
||||
- **Performance**: <100ms response time for API operations, <2 second cold start
|
||||
- **Compatibility**: >90% AWS API compatibility for supported operations
|
||||
- **Scalability**: Support for 50+ concurrent VM instances
|
||||
|
||||
## Implementation Priority and Dependencies
|
||||
|
||||
### **Integrated Development Phases (Total: 19-26 weeks)**
|
||||
|
||||
**Phase 0** (1-2 weeks): Distributed Systems Foundation
|
||||
- ✅ Addresses TODO #7 (Gossip/RAFT) and #8 (etcd/KV configuration)
|
||||
|
||||
**Phase 1** (2-3 weeks): Development Environment Automation
|
||||
- Critical for productive cross-service development
|
||||
|
||||
**Phase 2** (3-4 weeks): AWS API Model Implementation
|
||||
- ✅ Addresses TODO #9 (Smithy code generation) and #5 (AWS SDK v2)
|
||||
|
||||
**Phase 3** (2-3 weeks): Scalable Gateway and Daemon Architecture
|
||||
- ✅ Addresses TODO #4 (`awsd` gateway) and #2 (`hived` refactoring)
|
||||
|
||||
**Phase 4** (3-4 weeks): Service Integration and Orchestration
|
||||
- NATS clustering and multi-service coordination
|
||||
|
||||
**Phase 5** (4-5 weeks): Infrastructure Services
|
||||
- ✅ Addresses TODO #3 (VPC with Open vSwitch as `vpcd`)
|
||||
|
||||
**Phase 6** (2-3 weeks): Testing and Validation Framework
|
||||
- AWS CLI compatibility and comprehensive testing
|
||||
|
||||
**Phase 7** (2-3 weeks): Production Deployment and Packaging
|
||||
- ✅ Addresses TODO #1 (binary compilation) and #6 (systemd services)
|
||||
|
||||
### **Key Achievements**
|
||||
|
||||
✅ **100% TODO Integration**: All original TODO items incorporated into structured phases
|
||||
✅ **Service Architecture**: Adopts planned service names (`awsd`, `hived`, `vpcd`)
|
||||
✅ **Technology Choices**: Integrates specific selections (Open vSwitch, Smithy models)
|
||||
✅ **Production Ready**: Comprehensive deployment and packaging strategy
|
||||
✅ **Scalable Foundation**: Distributed systems and message-driven architecture
|
||||
|
||||
### Getting Started (for Claude Code)
|
||||
|
||||
When working on this codebase:
|
||||
|
||||
1. **Read current CLAUDE.md** for build commands and architecture understanding
|
||||
2. **Start with Phase 0 tasks** if working on distributed systems/clustering
|
||||
3. **Start with Phase 1 tasks** if improving development experience
|
||||
4. **Begin with Phase 2 tasks** if implementing AWS API compatibility
|
||||
5. **Always run tests** after making changes: `make test` in each component
|
||||
6. **Update documentation** when adding new patterns or significantly changing architecture
|
||||
7. **Test with real AWS CLI** to ensure compatibility
|
||||
|
||||
The existing code in `hive/daemon/daemon.go` provides an excellent foundation - it already implements TLS, AWS SDK integration, NATS coordination, and service integration patterns that should be extended rather than replaced. All original TODO items have been preserved and integrated into this comprehensive development plan.
|
||||
@@ -52,12 +52,86 @@ AWS-Compatible Interfaces – Provision infrastructure with awscli, Terraform, o
|
||||
- Works Offline – No reliance on centralized cloud services or external networks.
|
||||
- Open Source – Licensed under Apache 2.0. Fork it, extend it, or deploy it as-is.
|
||||
|
||||
## Development Setup
|
||||
|
||||
### For Developers
|
||||
|
||||
Get started with Hive development in minutes:
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/mulgadc/hive.git hive
|
||||
cd hive
|
||||
|
||||
# Setup dependencies and development environment
|
||||
./scripts/clone-deps.sh # Clone viperblock + predastore repositories
|
||||
./scripts/dev-setup.sh # Setup complete development environment
|
||||
|
||||
# Start all services (NATS, Predastore, Viperblock, Hive Gateway)
|
||||
./scripts/start-dev.sh
|
||||
|
||||
# Test with AWS CLI
|
||||
aws --endpoint-url https://localhost:9999 --no-verify-ssl ec2 describe-instances
|
||||
|
||||
# Stop all services when done
|
||||
./scripts/stop-dev.sh
|
||||
```
|
||||
|
||||
**Development Features:**
|
||||
- **Hot Reloading**: Automatic service restarts during development
|
||||
- **Multi-Repository**: Seamless cross-component development workflow
|
||||
- **TLS Ready**: Auto-generated certificates for HTTPS endpoints
|
||||
- **AWS Compatible**: Test with real AWS CLI and SDKs
|
||||
|
||||
### Component Repositories
|
||||
|
||||
Hive coordinates these independent components:
|
||||
- **[Viperblock](../viperblock/)** - EBS-compatible block storage
|
||||
- **[Predastore](../predastore/)** - S3-compatible object storage
|
||||
|
||||
Each component can be developed independently. See component-specific documentation for focused development guides.
|
||||
|
||||
## Quick Start
|
||||
|
||||
See [Installation Guide](INSTALL.md) on how to setup a Hive node to get started.
|
||||
|
||||
Then configure your AWS CLI to point to Hive’s endpoints and start launching VMs, buckets, and volumes using familiar commands and Terraform scripts.
|
||||
Then configure your AWS CLI to point to Hive's endpoints and start launching VMs, buckets, and volumes using familiar commands and Terraform scripts.
|
||||
|
||||
## Development Philosophy
|
||||
|
||||
### Built by Engineers, For Engineers
|
||||
|
||||
Hive is developed by experienced infrastructure engineers with deep AWS expertise, including former AWS team members who understand the intricacies of building production-grade cloud services. Our team brings decades of combined experience from AWS, enterprise infrastructure, and edge computing environments.
|
||||
|
||||
**Real-World Experience:**
|
||||
- Production AWS service development and operations
|
||||
- Large-scale infrastructure deployment and management
|
||||
- Edge computing and resource-constrained environments
|
||||
- Enterprise security and compliance requirements
|
||||
|
||||
### AI-Assisted Development
|
||||
|
||||
While Hive is architected and implemented by experienced engineers, we leverage **Claude Code** (Anthropic's AI coding assistant) to accelerate certain development tasks. This approach combines human expertise with AI efficiency:
|
||||
|
||||
**How We Use Claude Code:**
|
||||
- **Code Generation**: Boilerplate AWS API structures and handlers
|
||||
- **Documentation**: Comprehensive development guides and API documentation
|
||||
- **Testing**: Test case generation and validation scenarios
|
||||
- **Refactoring**: Large-scale code restructuring and optimization
|
||||
|
||||
**What Remains Human-Driven:**
|
||||
- **Architecture Decisions**: Core system design and scalability choices
|
||||
- **Security Implementation**: Authentication, encryption, and threat modeling
|
||||
- **Performance Optimization**: Real-world performance tuning and benchmarking
|
||||
- **Production Operations**: Deployment strategies and operational procedures
|
||||
|
||||
**Development Artifacts:**
|
||||
- `CLAUDE.md` - Instructions for Claude Code when working on this codebase
|
||||
- `HIVE_DEVELOPMENT_PLAN.md` - Comprehensive roadmap and implementation strategy
|
||||
- Component-specific guidance in `viperblock/CLAUDE.md` and `predastore/CLAUDE.md`
|
||||
|
||||
This hybrid approach ensures Hive benefits from both proven engineering expertise and modern development acceleration, while maintaining the quality and reliability standards required for production infrastructure.
|
||||
|
||||
## License
|
||||
|
||||
Hive is open source under the Apache 2.0 License. You’re free to use, modify, and deploy it—anywhere you need reliable infrastructure without depending on centralized cloud platforms
|
||||
Hive is open source under the Apache 2.0 License. You're free to use, modify, and deploy it—anywhere you need reliable infrastructure without depending on centralized cloud platforms
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
# Hive Development Roadmap
|
||||
|
||||
This roadmap has been integrated into the comprehensive [HIVE_DEVELOPMENT_PLAN.md](HIVE_DEVELOPMENT_PLAN.md).
|
||||
|
||||
## Original TODO Items → Development Plan Integration
|
||||
|
||||
### ✅ **Completed Integration**
|
||||
|
||||
All original TODO items have been incorporated into the structured development phases:
|
||||
|
||||
| Original TODO | Status | Integrated Into | Phase |
|
||||
|---------------|--------|-----------------|-------|
|
||||
| #1: Binary compile and install.sh | ✅ Planned | Phase 7: Task 7.1 (Binary Compilation) | Phase 7 |
|
||||
| #2: Move daemon.go to services/hived/ | ✅ Planned | Phase 3: Task 3.2 (Specialized Services) | Phase 3 |
|
||||
| #3: VPC with openvs-switch as `vpcd` | ✅ Planned | Phase 5: Task 5.1 (VPC with Open vSwitch) | Phase 5 |
|
||||
| #4: AWS HTTP gateway (`awsd`) | ✅ Planned | Phase 3: Task 3.1 (AWS Gateway Service) | Phase 3 |
|
||||
| #5: AWS SDK v2 input/output | ✅ Planned | Phase 2: Task 2.1 & Phase 3: Task 3.1 | Phase 2-3 |
|
||||
| #6: `hive.service` for systemd | ✅ Planned | Phase 7: Task 7.2 (System Service Integration) | Phase 7 |
|
||||
| #7: Gossip and RAFT protocols | ✅ Planned | Phase 0: Task 0.1 (Service Registry) | Phase 0 |
|
||||
| #8: etcd/KV for configuration sync | ✅ Planned | Phase 0: Task 0.1 (Distributed Config) | Phase 0 |
|
||||
| #9: Smithy model code generation | ✅ Planned | Phase 2: Task 2.1 (Smithy Code Generation) | Phase 2 |
|
||||
|
||||
## Development Phase Overview
|
||||
|
||||
**Phase 0**: Distributed Systems Foundation (1-2 weeks)
|
||||
- Gossip, RAFT, and distributed configuration (#7, #8)
|
||||
|
||||
**Phase 1**: Development Environment Automation (2-3 weeks)
|
||||
- Multi-service orchestration and hot reloading
|
||||
|
||||
**Phase 2**: AWS API Model Implementation (3-4 weeks)
|
||||
- Smithy-based code generation (#9)
|
||||
- AWS SDK v2 integration (#5)
|
||||
|
||||
**Phase 3**: Scalable Gateway and Daemon Architecture (2-3 weeks)
|
||||
- AWS gateway service `awsd` (#4)
|
||||
- Service refactoring to `hived` (#2)
|
||||
- VPC daemon `vpcd` foundation (#3)
|
||||
|
||||
**Phase 4**: Service Integration and Orchestration (3-4 weeks)
|
||||
- NATS clustering and cross-service coordination
|
||||
|
||||
**Phase 5**: Infrastructure Services (4-5 weeks)
|
||||
- VPC networking with Open vSwitch (#3)
|
||||
- Advanced VM features
|
||||
|
||||
**Phase 6**: Testing and Validation Framework (2-3 weeks)
|
||||
- AWS CLI compatibility testing
|
||||
|
||||
**Phase 7**: Production Deployment and Packaging (2-3 weeks)
|
||||
- Binary compilation and distribution (#1)
|
||||
- Systemd service integration (#6)
|
||||
- Production configuration management
|
||||
|
||||
## Service Architecture
|
||||
|
||||
The development plan implements these services:
|
||||
|
||||
- **`awsd`** - AWS API Gateway (TODO #4)
|
||||
- **`hived`** - Main Hive compute daemon (TODO #2)
|
||||
- **`vpcd`** - VPC networking daemon (TODO #3)
|
||||
- **Predastore** - S3 service (existing)
|
||||
- **Viperblock** - EBS service (existing)
|
||||
|
||||
## Getting Started
|
||||
|
||||
For development setup:
|
||||
```bash
|
||||
./scripts/dev-setup.sh # Setup complete development environment
|
||||
./scripts/start-dev.sh # Start all services
|
||||
```
|
||||
|
||||
For detailed implementation guidance, see [HIVE_DEVELOPMENT_PLAN.md](HIVE_DEVELOPMENT_PLAN.md).
|
||||
@@ -0,0 +1,137 @@
|
||||
# Hive Systemd Service
|
||||
|
||||
This directory contains the systemd service configuration for running Hive services on Ubuntu and Debian systems.
|
||||
|
||||
## Files
|
||||
|
||||
- `hive.service` - The systemd service file that manages all Hive services
|
||||
- `install-hive-service.sh` - Installation script to set up the service
|
||||
- `README.md` - This documentation file
|
||||
|
||||
## Installation
|
||||
|
||||
### Prerequisites
|
||||
|
||||
1. Ensure Hive is installed in `/opt/hive/`
|
||||
2. Ensure the `hive` binary is available at `/opt/hive/bin/hive`
|
||||
3. Run the installation script as root
|
||||
|
||||
### Quick Installation
|
||||
|
||||
```bash
|
||||
sudo ./config/install-hive-service.sh
|
||||
```
|
||||
|
||||
### Manual Installation
|
||||
|
||||
If you prefer to install manually:
|
||||
|
||||
1. Copy the service file:
|
||||
```bash
|
||||
sudo cp config/hive.service /etc/systemd/system/
|
||||
```
|
||||
|
||||
2. Reload systemd:
|
||||
```bash
|
||||
sudo systemctl daemon-reload
|
||||
```
|
||||
|
||||
3. Enable the service:
|
||||
```bash
|
||||
sudo systemctl enable hive.service
|
||||
```
|
||||
|
||||
## Service Management
|
||||
|
||||
### Start the service
|
||||
```bash
|
||||
sudo systemctl start hive
|
||||
```
|
||||
|
||||
### Stop the service
|
||||
```bash
|
||||
sudo systemctl stop hive
|
||||
```
|
||||
|
||||
### Check service status
|
||||
```bash
|
||||
sudo systemctl status hive
|
||||
```
|
||||
|
||||
### Restart the service
|
||||
```bash
|
||||
sudo systemctl restart hive
|
||||
```
|
||||
|
||||
### View service logs
|
||||
```bash
|
||||
sudo journalctl -u hive -f
|
||||
```
|
||||
|
||||
### Disable auto-start
|
||||
```bash
|
||||
sudo systemctl disable hive
|
||||
```
|
||||
|
||||
## Service Details
|
||||
|
||||
The `hive.service` file:
|
||||
|
||||
- Starts services in the correct order: NATS → Predastore → Viperblock
|
||||
- Includes proper delays between service starts to ensure dependencies are ready
|
||||
- Stops services in reverse order during shutdown
|
||||
- Runs as root (required for system services)
|
||||
- Includes security hardening settings
|
||||
- Provides proper logging to systemd journal
|
||||
- Automatically restarts on failure
|
||||
|
||||
## Service Order
|
||||
|
||||
The service starts the following components in order:
|
||||
|
||||
1. **NATS** - Message broker service
|
||||
2. **Predastore** - Data storage service (waits 3 seconds after NATS)
|
||||
3. **Viperblock** - Block storage service (waits 2 seconds after Predastore)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Check if the service is running
|
||||
```bash
|
||||
sudo systemctl is-active hive
|
||||
```
|
||||
|
||||
### View recent logs
|
||||
```bash
|
||||
sudo journalctl -u hive --since "1 hour ago"
|
||||
```
|
||||
|
||||
### Check service configuration
|
||||
```bash
|
||||
sudo systemctl cat hive
|
||||
```
|
||||
|
||||
### Verify binary exists
|
||||
```bash
|
||||
ls -la /opt/hive/bin/hive
|
||||
```
|
||||
|
||||
### Test service commands manually
|
||||
```bash
|
||||
/opt/hive/bin/hive service nats start
|
||||
/opt/hive/bin/hive service predastore start
|
||||
/opt/hive/bin/hive service viperblock start
|
||||
```
|
||||
|
||||
## Customization
|
||||
|
||||
If you need to modify the service configuration:
|
||||
|
||||
1. Edit the service file: `sudo nano /etc/systemd/system/hive.service`
|
||||
2. Reload systemd: `sudo systemctl daemon-reload`
|
||||
3. Restart the service: `sudo systemctl restart hive`
|
||||
|
||||
## Security Notes
|
||||
|
||||
- The service runs as root, which is necessary for system-level services
|
||||
- Security hardening is enabled with `ProtectSystem=strict` and `NoNewPrivileges=true`
|
||||
- Only specific paths are writable: `/opt/hive`, `/var/log`, `/var/run`
|
||||
@@ -0,0 +1,49 @@
|
||||
[Unit]
|
||||
Description=Hive Services
|
||||
Documentation=https://github.com/mulgadc/hive
|
||||
After=network.target
|
||||
Wants=network.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
RemainAfterExit=yes
|
||||
User=root
|
||||
Group=root
|
||||
WorkingDirectory=/opt/hive
|
||||
Environment=PATH=/opt/hive/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
|
||||
|
||||
# Start NATS service
|
||||
ExecStart=/opt/hive/bin/hive service nats start
|
||||
ExecStartPost=/bin/sleep 3
|
||||
|
||||
# Start Predastore service
|
||||
ExecStart=/opt/hive/bin/hive service predastore start
|
||||
ExecStartPost=/bin/sleep 2
|
||||
|
||||
# Start Viperblock service
|
||||
ExecStart=/opt/hive/bin/hive service viperblock start
|
||||
|
||||
# Stop services in reverse order
|
||||
ExecStop=/opt/hive/bin/hive service viperblock stop
|
||||
ExecStopPost=/bin/sleep 2
|
||||
ExecStop=/opt/hive/bin/hive service predastore stop
|
||||
ExecStopPost=/bin/sleep 2
|
||||
ExecStop=/opt/hive/bin/hive service nats stop
|
||||
|
||||
# Restart policy
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
# Security settings
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
ReadWritePaths=/opt/hive /var/log /var/run
|
||||
|
||||
# Logging
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
SyslogIdentifier=hive
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
Executable
+74
@@ -0,0 +1,74 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Hive Service Installation Script
|
||||
# This script installs the hive systemd service on Ubuntu/Debian systems
|
||||
|
||||
set -e
|
||||
|
||||
# Colors for output
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Function to print colored output
|
||||
print_status() {
|
||||
echo -e "${GREEN}[INFO]${NC} $1"
|
||||
}
|
||||
|
||||
print_warning() {
|
||||
echo -e "${YELLOW}[WARNING]${NC} $1"
|
||||
}
|
||||
|
||||
print_error() {
|
||||
echo -e "${RED}[ERROR]${NC} $1"
|
||||
}
|
||||
|
||||
# Check if running as root
|
||||
if [[ $EUID -ne 0 ]]; then
|
||||
print_error "This script must be run as root (use sudo)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if systemd is available
|
||||
if ! command -v systemctl &> /dev/null; then
|
||||
print_error "systemd is not available on this system"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Get the directory where this script is located
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SERVICE_FILE="$SCRIPT_DIR/hive.service"
|
||||
|
||||
# Check if service file exists
|
||||
if [[ ! -f "$SERVICE_FILE" ]]; then
|
||||
print_error "Service file not found at $SERVICE_FILE"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
print_status "Installing Hive systemd service..."
|
||||
|
||||
# Copy service file to systemd directory
|
||||
cp "$SERVICE_FILE" /etc/systemd/system/
|
||||
|
||||
# Reload systemd to recognize the new service
|
||||
systemctl daemon-reload
|
||||
|
||||
# Enable the service to start on boot
|
||||
systemctl enable hive.service
|
||||
|
||||
print_status "Service installed successfully!"
|
||||
print_status "You can now use the following commands:"
|
||||
echo " systemctl start hive # Start the service"
|
||||
echo " systemctl stop hive # Stop the service"
|
||||
echo " systemctl status hive # Check service status"
|
||||
echo " systemctl restart hive # Restart the service"
|
||||
echo " journalctl -u hive -f # View service logs"
|
||||
|
||||
# Check if hive binary exists
|
||||
if [[ ! -f "/opt/hive/bin/hive" ]]; then
|
||||
print_warning "Hive binary not found at /opt/hive/bin/hive"
|
||||
print_warning "Please ensure hive is installed in /opt/hive before starting the service"
|
||||
fi
|
||||
|
||||
print_status "Installation complete!"
|
||||
Executable
+86
@@ -0,0 +1,86 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Hive Development Dependencies Setup Script
|
||||
# This script clones Viperblock and Predastore repositories for cross-repo development
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
MULGA_ROOT="$(cd "$PROJECT_ROOT/.." && pwd)"
|
||||
|
||||
echo "🏗️ Setting up Hive development environment..."
|
||||
echo "Project root: $PROJECT_ROOT"
|
||||
echo "Mulga root: $MULGA_ROOT"
|
||||
|
||||
# Check if we're in the correct directory structure
|
||||
if [[ ! -f "$PROJECT_ROOT/go.mod" ]]; then
|
||||
echo "❌ Error: Cannot find go.mod. Please run this script from the hive repository."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Function to clone or update repository
|
||||
clone_or_update() {
|
||||
local repo_url="$1"
|
||||
local repo_name="$2"
|
||||
local target_dir="$MULGA_ROOT/$repo_name"
|
||||
|
||||
if [[ -d "$target_dir" ]]; then
|
||||
echo "📂 $repo_name already exists at $target_dir"
|
||||
echo " To update, run: cd $target_dir && git pull"
|
||||
else
|
||||
echo "📥 Cloning $repo_name to $target_dir..."
|
||||
git clone "$repo_url" "$target_dir"
|
||||
echo "✅ Successfully cloned $repo_name"
|
||||
fi
|
||||
}
|
||||
|
||||
# Repository URLs - Update these with actual repository URLs
|
||||
VIPERBLOCK_REPO="${VIPERBLOCK_REPO:-https://github.com/mulgadc/viperblock.git}"
|
||||
PREDASTORE_REPO="${PREDASTORE_REPO:-https://github.com/mulgadc/predastore.git}"
|
||||
|
||||
echo "🔗 Repository URLs:"
|
||||
echo " Viperblock: $VIPERBLOCK_REPO"
|
||||
echo " Predastore: $PREDASTORE_REPO"
|
||||
echo ""
|
||||
|
||||
# Clone dependencies
|
||||
clone_or_update "$VIPERBLOCK_REPO" "viperblock"
|
||||
clone_or_update "$PREDASTORE_REPO" "predastore"
|
||||
|
||||
# Verify go.mod replace directives
|
||||
echo ""
|
||||
echo "🔍 Verifying go.mod replace directives..."
|
||||
|
||||
if grep -q "replace github.com/mulgadc/viperblock => ../viperblock" "$PROJECT_ROOT/go.mod"; then
|
||||
echo "✅ Viperblock replace directive found"
|
||||
else
|
||||
echo "⚠️ Adding Viperblock replace directive to go.mod"
|
||||
echo "replace github.com/mulgadc/viperblock => ../viperblock" >> "$PROJECT_ROOT/go.mod"
|
||||
fi
|
||||
|
||||
if grep -q "replace github.com/mulgadc/predastore => ../predastore" "$PROJECT_ROOT/go.mod"; then
|
||||
echo "✅ Predastore replace directive found"
|
||||
else
|
||||
echo "⚠️ Adding Predastore replace directive to go.mod"
|
||||
echo "replace github.com/mulgadc/predastore => ../predastore" >> "$PROJECT_ROOT/go.mod"
|
||||
fi
|
||||
|
||||
# Verify directory structure
|
||||
echo ""
|
||||
echo "📁 Directory structure:"
|
||||
ls -la "$MULGA_ROOT" | grep -E "(hive|viperblock|predastore)" || true
|
||||
|
||||
echo ""
|
||||
echo "🎉 Development environment setup complete!"
|
||||
echo ""
|
||||
echo "Next steps:"
|
||||
echo "1. Build components: cd ../viperblock && make build"
|
||||
echo "2. Build components: cd ../predastore && make build"
|
||||
echo "3. Build Hive: make build"
|
||||
echo "4. Run development setup: ./scripts/dev-setup.sh"
|
||||
echo ""
|
||||
echo "📖 For detailed instructions, see:"
|
||||
echo " - ./CLAUDE.md (Hive development guide)"
|
||||
echo " - ../viperblock/CLAUDE.md (Viperblock guide)"
|
||||
echo " - ../predastore/CLAUDE.md (Predastore guide)"
|
||||
Executable
+372
@@ -0,0 +1,372 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Hive Platform Development Environment Setup
|
||||
# This script sets up a complete development environment for the Hive platform
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
MULGA_ROOT="$(cd "$PROJECT_ROOT/.." && pwd)"
|
||||
|
||||
# Default configuration
|
||||
NATS_PORT=4222
|
||||
PREDASTORE_PORT=8443
|
||||
HIVE_GATEWAY_PORT=9999
|
||||
DATA_DIR="$PROJECT_ROOT/data"
|
||||
CONFIG_DIR="$PROJECT_ROOT/config"
|
||||
|
||||
echo "🚀 Setting up Hive development environment..."
|
||||
echo "Project root: $PROJECT_ROOT"
|
||||
echo "Data directory: $DATA_DIR"
|
||||
|
||||
# Create necessary directories
|
||||
mkdir -p "$DATA_DIR"/{nats,predastore,viperblock,logs}
|
||||
mkdir -p "$CONFIG_DIR"
|
||||
|
||||
# Function to check if command exists
|
||||
command_exists() {
|
||||
command -v "$1" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
# Function to check if port is available
|
||||
port_available() {
|
||||
! nc -z localhost "$1" 2>/dev/null
|
||||
}
|
||||
|
||||
# Check dependencies
|
||||
echo "🔍 Checking dependencies..."
|
||||
|
||||
# Check for required commands
|
||||
required_commands=("go" "make" "nats-server")
|
||||
for cmd in "${required_commands[@]}"; do
|
||||
if command_exists "$cmd"; then
|
||||
echo "✅ $cmd found"
|
||||
else
|
||||
echo "❌ $cmd not found. Please install $cmd"
|
||||
case "$cmd" in
|
||||
"nats-server")
|
||||
echo " Install NATS: https://docs.nats.io/running-a-nats-service/introduction/installation"
|
||||
;;
|
||||
"go")
|
||||
echo " Install Go: https://golang.org/dl/"
|
||||
;;
|
||||
esac
|
||||
fi
|
||||
done
|
||||
|
||||
# Check optional commands
|
||||
optional_commands=("air" "nbdkit")
|
||||
for cmd in "${optional_commands[@]}"; do
|
||||
if command_exists "$cmd"; then
|
||||
echo "✅ $cmd found (optional)"
|
||||
else
|
||||
echo "⚠️ $cmd not found (optional)"
|
||||
case "$cmd" in
|
||||
"air")
|
||||
echo " Install air for hot reloading: go install github.com/cosmtrek/air@latest"
|
||||
;;
|
||||
"nbdkit")
|
||||
echo " Install nbdkit: sudo apt-get install nbdkit (Ubuntu/Debian)"
|
||||
;;
|
||||
esac
|
||||
fi
|
||||
done
|
||||
|
||||
# Check component repositories
|
||||
echo ""
|
||||
echo "📦 Checking component repositories..."
|
||||
|
||||
components=("viperblock" "predastore")
|
||||
for component in "${components[@]}"; do
|
||||
component_path="$MULGA_ROOT/$component"
|
||||
if [[ -d "$component_path" ]]; then
|
||||
echo "✅ $component found at $component_path"
|
||||
if [[ -f "$component_path/go.mod" ]]; then
|
||||
echo " Go module verified"
|
||||
else
|
||||
echo " ⚠️ No go.mod found in $component"
|
||||
fi
|
||||
else
|
||||
echo "❌ $component not found at $component_path"
|
||||
echo " Run: ./scripts/clone-deps.sh"
|
||||
fi
|
||||
done
|
||||
|
||||
# Check port availability
|
||||
echo ""
|
||||
echo "🔌 Checking port availability..."
|
||||
|
||||
ports=("$NATS_PORT:NATS" "$PREDASTORE_PORT:Predastore" "$HIVE_GATEWAY_PORT:Hive Gateway")
|
||||
for port_info in "${ports[@]}"; do
|
||||
port="${port_info%:*}"
|
||||
service="${port_info#*:}"
|
||||
if port_available "$port"; then
|
||||
echo "✅ Port $port available for $service"
|
||||
else
|
||||
echo "⚠️ Port $port already in use (needed for $service)"
|
||||
fi
|
||||
done
|
||||
|
||||
# Generate TLS certificates for development
|
||||
echo ""
|
||||
echo "🔐 Setting up TLS certificates..."
|
||||
|
||||
if [[ ! -f "$CONFIG_DIR/server.pem" ]] || [[ ! -f "$CONFIG_DIR/server.key" ]]; then
|
||||
echo "📋 Generating self-signed certificate for development..."
|
||||
openssl req -x509 -newkey rsa:4096 -keyout "$CONFIG_DIR/server.key" -out "$CONFIG_DIR/server.pem" \
|
||||
-days 365 -nodes -subj "/C=US/ST=Dev/L=Development/O=Hive/CN=localhost" \
|
||||
-addext "subjectAltName=DNS:localhost,IP:127.0.0.1,IP:::1"
|
||||
echo "✅ TLS certificates generated"
|
||||
else
|
||||
echo "✅ TLS certificates already exist"
|
||||
fi
|
||||
|
||||
# Create development configuration
|
||||
echo ""
|
||||
echo "⚙️ Creating development configuration..."
|
||||
|
||||
cat > "$CONFIG_DIR/dev.yaml" << EOF
|
||||
# Hive Development Configuration
|
||||
server:
|
||||
host: "0.0.0.0"
|
||||
port: $HIVE_GATEWAY_PORT
|
||||
tls:
|
||||
cert_file: "$CONFIG_DIR/server.pem"
|
||||
key_file: "$CONFIG_DIR/server.key"
|
||||
|
||||
nats:
|
||||
host: "nats://localhost:$NATS_PORT"
|
||||
acl:
|
||||
token: "dev-token"
|
||||
|
||||
services:
|
||||
predastore:
|
||||
host: "https://localhost:$PREDASTORE_PORT"
|
||||
access_key: "EXAMPLEKEY"
|
||||
secret_key: "EXAMPLEKEY"
|
||||
bucket: "predastore"
|
||||
region: "ap-southeast-2"
|
||||
|
||||
viperblock:
|
||||
base_dir: "$DATA_DIR/viperblock"
|
||||
|
||||
data:
|
||||
base_dir: "$DATA_DIR"
|
||||
|
||||
logging:
|
||||
level: "info"
|
||||
format: "json"
|
||||
EOF
|
||||
|
||||
echo "✅ Development configuration created at $CONFIG_DIR/dev.yaml"
|
||||
|
||||
# Create air configuration for hot reloading
|
||||
if command_exists "air"; then
|
||||
echo ""
|
||||
echo "🔥 Setting up air configuration for hot reloading..."
|
||||
|
||||
cat > "$PROJECT_ROOT/.air.toml" << EOF
|
||||
# Air configuration for Hive development
|
||||
root = "."
|
||||
testdata_dir = "testdata"
|
||||
tmp_dir = "tmp"
|
||||
|
||||
[build]
|
||||
args_bin = ["daemon", "--config", "config/dev.yaml"]
|
||||
bin = "./tmp/hive"
|
||||
cmd = "go build -o ./tmp/hive cmd/hive/main.go"
|
||||
delay = 1000
|
||||
exclude_dir = ["assets", "tmp", "vendor", "testdata", "data", "nbdkit"]
|
||||
exclude_file = []
|
||||
exclude_regex = ["_test.go"]
|
||||
exclude_unchanged = false
|
||||
follow_symlink = false
|
||||
full_bin = ""
|
||||
include_dir = []
|
||||
include_ext = ["go", "tpl", "tmpl", "html"]
|
||||
kill_delay = "0s"
|
||||
log = "build-errors.log"
|
||||
send_interrupt = false
|
||||
stop_on_root = false
|
||||
|
||||
[color]
|
||||
app = ""
|
||||
build = "yellow"
|
||||
main = "magenta"
|
||||
runner = "green"
|
||||
watcher = "cyan"
|
||||
|
||||
[log]
|
||||
time = false
|
||||
|
||||
[misc]
|
||||
clean_on_exit = false
|
||||
|
||||
[screen]
|
||||
clear_on_rebuild = false
|
||||
EOF
|
||||
|
||||
echo "✅ Air configuration created"
|
||||
fi
|
||||
|
||||
# Create startup script
|
||||
echo ""
|
||||
echo "📝 Creating startup scripts..."
|
||||
|
||||
cat > "$PROJECT_ROOT/scripts/start-dev.sh" << 'EOF'
|
||||
#!/bin/bash
|
||||
|
||||
# Start Hive development environment
|
||||
# This script starts all required services in the correct order
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
MULGA_ROOT="$(cd "$PROJECT_ROOT/.." && pwd)"
|
||||
|
||||
echo "🚀 Starting Hive development environment..."
|
||||
|
||||
# Function to start service in background
|
||||
start_service() {
|
||||
local name="$1"
|
||||
local command="$2"
|
||||
local pidfile="$PROJECT_ROOT/data/logs/$name.pid"
|
||||
local logfile="$PROJECT_ROOT/data/logs/$name.log"
|
||||
|
||||
echo "📡 Starting $name..."
|
||||
nohup $command > "$logfile" 2>&1 &
|
||||
echo $! > "$pidfile"
|
||||
echo " PID: $(cat $pidfile), Log: $logfile"
|
||||
}
|
||||
|
||||
# Create logs directory
|
||||
mkdir -p "$PROJECT_ROOT/data/logs"
|
||||
|
||||
# Start services in dependency order
|
||||
echo ""
|
||||
echo "1️⃣ Starting NATS server..."
|
||||
start_service "nats" "nats-server --port 4222 --auth dev-token"
|
||||
|
||||
echo ""
|
||||
echo "2️⃣ Starting Predastore..."
|
||||
cd "$MULGA_ROOT/predastore"
|
||||
start_service "predastore" "./bin/s3d --port 8443 --data-dir $PROJECT_ROOT/data/predastore"
|
||||
|
||||
echo ""
|
||||
echo "3️⃣ Starting NBDkit (if available)..."
|
||||
if command -v nbdkit >/dev/null; then
|
||||
start_service "nbdkit" "nbdkit --foreground --verbose memory 1G"
|
||||
else
|
||||
echo " ⚠️ NBDkit not available, skipping"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "4️⃣ Starting Hive Gateway..."
|
||||
cd "$PROJECT_ROOT"
|
||||
if command -v air >/dev/null; then
|
||||
echo " 🔥 Using air for hot reloading"
|
||||
air
|
||||
else
|
||||
echo " 🔨 Building and starting Hive"
|
||||
make build
|
||||
./bin/hive daemon --config config/dev.yaml
|
||||
fi
|
||||
EOF
|
||||
|
||||
chmod +x "$PROJECT_ROOT/scripts/start-dev.sh"
|
||||
|
||||
# Create stop script
|
||||
cat > "$PROJECT_ROOT/scripts/stop-dev.sh" << 'EOF'
|
||||
#!/bin/bash
|
||||
|
||||
# Stop Hive development environment
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
|
||||
echo "🛑 Stopping Hive development environment..."
|
||||
|
||||
# Function to stop service
|
||||
stop_service() {
|
||||
local name="$1"
|
||||
local pidfile="$PROJECT_ROOT/data/logs/$name.pid"
|
||||
|
||||
if [[ -f "$pidfile" ]]; then
|
||||
local pid=$(cat "$pidfile")
|
||||
if kill -0 "$pid" 2>/dev/null; then
|
||||
echo "🔻 Stopping $name (PID: $pid)..."
|
||||
kill "$pid"
|
||||
rm -f "$pidfile"
|
||||
else
|
||||
echo "⚠️ $name process not found (PID: $pid)"
|
||||
rm -f "$pidfile"
|
||||
fi
|
||||
else
|
||||
echo "⚠️ No PID file for $name"
|
||||
fi
|
||||
}
|
||||
|
||||
# Stop services in reverse order
|
||||
stop_service "hive"
|
||||
stop_service "nbdkit"
|
||||
stop_service "predastore"
|
||||
stop_service "nats"
|
||||
|
||||
echo "✅ Development environment stopped"
|
||||
EOF
|
||||
|
||||
chmod +x "$PROJECT_ROOT/scripts/stop-dev.sh"
|
||||
|
||||
echo "✅ Startup scripts created"
|
||||
|
||||
# Build components
|
||||
echo ""
|
||||
echo "🔨 Building components..."
|
||||
|
||||
# Build Viperblock
|
||||
if [[ -d "$MULGA_ROOT/viperblock" ]]; then
|
||||
echo "📦 Building Viperblock..."
|
||||
cd "$MULGA_ROOT/viperblock"
|
||||
make build
|
||||
echo "✅ Viperblock built successfully"
|
||||
else
|
||||
echo "⚠️ Viperblock directory not found, skipping build"
|
||||
fi
|
||||
|
||||
# Build Predastore
|
||||
if [[ -d "$MULGA_ROOT/predastore" ]]; then
|
||||
echo "📦 Building Predastore..."
|
||||
cd "$MULGA_ROOT/predastore"
|
||||
make build
|
||||
echo "✅ Predastore built successfully"
|
||||
else
|
||||
echo "⚠️ Predastore directory not found, skipping build"
|
||||
fi
|
||||
|
||||
# Build Hive
|
||||
echo "📦 Building Hive..."
|
||||
cd "$PROJECT_ROOT"
|
||||
make build
|
||||
echo "✅ Hive built successfully"
|
||||
|
||||
echo ""
|
||||
echo "🎉 Development environment setup complete!"
|
||||
echo ""
|
||||
echo "🚀 To start the development environment:"
|
||||
echo " ./scripts/start-dev.sh"
|
||||
echo ""
|
||||
echo "🛑 To stop the development environment:"
|
||||
echo " ./scripts/stop-dev.sh"
|
||||
echo ""
|
||||
echo "🔧 Development endpoints:"
|
||||
echo " - Hive Gateway: https://localhost:$HIVE_GATEWAY_PORT"
|
||||
echo " - Predastore S3: https://localhost:$PREDASTORE_PORT"
|
||||
echo " - NATS: nats://localhost:$NATS_PORT"
|
||||
echo ""
|
||||
echo "📊 Monitor logs:"
|
||||
echo " tail -f data/logs/*.log"
|
||||
echo ""
|
||||
echo "🧪 Test with AWS CLI:"
|
||||
echo " aws --endpoint-url https://localhost:$HIVE_GATEWAY_PORT --no-verify-ssl ec2 describe-instances"
|
||||
Reference in New Issue
Block a user