spinifex/hive/daemon/TESTING.md

Daemon Testing Guide

Overview

The daemon_test.go file provides comprehensive unit tests for the daemon's EC2 launch functionality, focusing on message handling, validation, and resource management.

Test Structure

Core Test Components

1. NATS Server Setup (startTestNATSServer)

   • Creates an embedded NATS server for testing
   • Uses port -1 for automatic port allocation
   • Returns server instance and connection URL
   • Automatically starts and waits for readiness

2. Daemon Initialization (createTestDaemon)

   • Creates a minimal daemon configuration for testing
   • Sets up temporary directories for test data
   • Establishes NATS connection
   • Includes automatic cleanup via t.Cleanup()

3. Test Fixtures (createValidRunInstancesInput)

   • Provides valid ec2.RunInstancesInput structures
   • Pre-configured with sensible defaults
   • Easy to modify for specific test scenarios

Test Suites

1. TestHandleEC2Launch_MessageParsing

Tests the daemon's ability to parse and validate incoming NATS messages:

• ✅ Valid RunInstancesInput processing
• ✅ Invalid instance type handling
• ✅ Missing required fields (ImageId)
• ✅ Invalid parameter values (MinCount = 0)
• ✅ Malformed JSON handling

Expected Behaviors:

• Valid inputs should return a Reservation with pending instance
• Invalid inputs should return error payloads (not transport errors)
• All responses should be properly formatted JSON

2. TestHandleEC2Launch_ResourceManagement

Tests resource allocation and instance type validation:

• ✅ Valid instance types (t3.micro, t3.nano)
• ✅ Invalid instance types (t99.invalid)
• ✅ Resource availability checking
• ✅ Allocation/deallocation cycles

3. TestDaemon_Initialization

Tests daemon initialization logic:

• ✅ Configuration loading
• ✅ Resource manager creation
• ✅ Instance storage initialization

4. TestResourceManager

Tests the resource management subsystem:

• ✅ System resource detection
• ✅ Instance type allocation
• ✅ Resource deallocation
• ✅ Capacity validation

Running Tests

Basic Test Execution

# Run all daemon tests
go test -v ./hive/daemon/

# Run specific test
go test -v ./hive/daemon/ -run TestHandleEC2Launch_MessageParsing

# Run with race detection
go test -race -v ./hive/daemon/

# Run with coverage
go test -cover -v ./hive/daemon/

Skipping Integration Tests

The tests are designed to skip integration components when running on macOS or when dependencies are unavailable:

# Skip integration tests that require viperblock/nbdkit
SKIP_INTEGRATION=1 go test -v ./hive/daemon/

macOS Testing Limitations

Current Limitations

The full integration test suite has limitations on macOS due to missing dependencies:

1. Viperblock - Not yet ported to macOS

   • NBD (Network Block Device) support not available
   • Volume mounting operations will fail

2. NBDkit - Linux-specific tool

   • Requires kernel NBD support
   • Not available natively on macOS

3. QEMU/KVM - Limited on macOS

   • KVM acceleration not available (Linux-only)
   • Can use QEMU with TCG (slow emulation)

What Works on macOS

✅ Message-driven architecture testing:

• NATS message parsing and routing
• Request/response validation
• Error handling and payload generation
• Resource manager logic

✅ Unit tests without infrastructure:

• Input validation (ValidateRunInstancesInput)
• JSON marshaling/unmarshaling
• Error code generation
• Instance ID generation

What Requires Linux

❌ Full integration testing:

• Volume creation and mounting (requires viperblock)
• NBD device attachment (requires nbdkit)
• VM launch with KVM acceleration
• EBS volume operations
• Cloud-init ISO creation and mounting

Recommendations for macOS Development

Option 1: Mock Integration Points

Create mock implementations for macOS testing:

// daemon_test_helpers.go
// +build darwin

// Mock viperblock operations for macOS
type MockViperblock struct {
    // Mock implementation
}

func (m *MockViperblock) Mount(volumeId string) error {
    // Return mock NBD URI
    return nil
}

Option 2: Use Docker Linux Container

Run tests in a Linux container with full dependencies:

# Build Linux test container
docker build -t hive-test -f Dockerfile.test .

# Run tests in container
docker run --rm \
  --privileged \
  -v $(pwd):/workspace \
  hive-test \
  go test -v ./hive/daemon/

Dockerfile.test example:

FROM ubuntu:22.04

RUN apt-get update && apt-get install -y \
    golang-1.21 \
    qemu-kvm \
    nbdkit \
    build-essential

WORKDIR /workspace

Option 3: GitHub Actions / CI Pipeline

Use CI for full integration tests:

# .github/workflows/test.yml
name: Integration Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-go@v4
      - name: Install dependencies
        run: |
          sudo apt-get update
          sudo apt-get install -y nbdkit qemu-kvm
      - name: Run tests
        run: make test

Option 4: Virtual Linux VM on macOS

Use a Linux VM for development testing:

# Using UTM or Multipass
multipass launch --name hive-dev --cpus 4 --memory 8G --disk 20G
multipass shell hive-dev

# Inside VM
cd /path/to/hive
make test

Hybrid Testing Strategy

On macOS (Unit Tests):

# Test message handling, validation, resource management
SKIP_INTEGRATION=1 go test -v ./hive/daemon/

On Linux (Full Integration):

# Test complete EC2 launch flow with viperblock/nbdkit
go test -v ./hive/daemon/

Test Coverage Goals

Current Coverage

• ✅ Message parsing and validation
• ✅ Resource allocation logic
• ✅ Error handling and responses
• ✅ NATS integration (pub/sub)

Additional Tests Needed (Linux)

Volume Management Tests

func TestHandleEC2Launch_VolumeCreation(t *testing.T) {
    // Test AMI volume cloning
    // Test EFI partition creation
    // Test cloud-init ISO generation
}

NBD Mount Tests

func TestMountVolumes(t *testing.T) {
    // Test NBD URI generation
    // Test mount request/response
    // Test mount failures
}

VM Lifecycle Tests

func TestLaunchInstance(t *testing.T) {
    // Test QEMU process startup
    // Test QMP socket connection
    // Test instance state transitions
}

End-to-End Tests

func TestEC2Launch_EndToEnd(t *testing.T) {
    // Full flow: message -> volumes -> VM -> running
    // Test with multiple instance types
    // Test concurrent launches
}

Mocking Strategy for macOS

Mock EBS Mount Service

// Create a mock EBS mount responder
func startMockEBSService(t *testing.T, nc *nats.Conn) {
    nc.QueueSubscribe("ebs.mount", "mock-ebs", func(msg *nats.Msg) {
        var req config.EBSRequest
        json.Unmarshal(msg.Data, &req)

        resp := config.EBSMountResponse{
            URI:     fmt.Sprintf("nbd://localhost:10809/%s", req.Name),
            Mounted: true,
            Error:   "",
        }

        data, _ := json.Marshal(resp)
        msg.Respond(data)
    })
}

Mock S3 Client

// Mock S3 client for SSH key retrieval
type MockS3Client struct {
    keys map[string]string
}

func (m *MockS3Client) Read(path string) ([]byte, error) {
    if key, ok := m.keys[path]; ok {
        return []byte(key), nil
    }
    return nil, fmt.Errorf("key not found")
}

Example Mock Test

func TestHandleEC2Launch_WithMocks(t *testing.T) {
    if runtime.GOOS == "darwin" {
        // Run with mocks on macOS
        ns, natsURL := startTestNATSServer(t)
        defer ns.Shutdown()

        nc, _ := nats.Connect(natsURL)
        defer nc.Close()

        // Start mock services
        startMockEBSService(t, nc)
        startMockS3Service(t, nc)

        // Run test...
    } else {
        // Run with real services on Linux
        // ...
    }
}

Debugging Tests

Enable Verbose Logging

# Run tests with verbose output
go test -v ./hive/daemon/ 2>&1 | tee test.log

Debug NATS Messages

// Add debug subscriber to see all messages
nc.Subscribe(">", func(msg *nats.Msg) {
    t.Logf("NATS [%s]: %s", msg.Subject, string(msg.Data))
})

Common Issues

Issue: "NATS server failed to start"

• Port already in use
• Use -1 for auto port allocation
• Check firewall settings

Issue: "Request timeout"

• Increase timeout duration
• Check NATS connectivity
• Verify handler subscription

Issue: "Test fails on Linux but works on macOS"

• Missing dependencies (nbdkit, QEMU)
• Permission issues (KVM access)
• File system differences

Next Steps

1. ✅ Create basic message handling tests (DONE)
2. ⏳ Add mock services for macOS testing
3. ⏳ Create Linux CI pipeline for full integration tests
4. ⏳ Add Docker-based test environment
5. ⏳ Implement volume lifecycle tests (Linux only)
6. ⏳ Add concurrent launch stress tests
7. ⏳ Create performance benchmarks

Contributing

When adding new tests:

1. Check platform compatibility:

   if runtime.GOOS == "darwin" && needsLinux {
       t.Skip("Test requires Linux")
   }
   

2. Use table-driven tests:

   tests := []struct {
       name string
       input interface{}
       want interface{}
   }{ /* ... */ }
   

3. Clean up resources:

   t.Cleanup(func() {
       // Cleanup code
   })
   

4. Document dependencies:

   • Add comments explaining Linux-only requirements
   • Use build tags where appropriate
   • Provide alternative test paths for macOS