# RunInstances capacity routing via per-instance-type NATS subscriptions

**Status:** Implemented (local tests pass, pending remote verification)

## Problem

When `RunInstances` is called, the gateway publishes to NATS topic `ec2.RunInstances` with queue group `hive-workers`. NATS round-robins the request to one daemon. If that daemon has insufficient capacity, it returns `InsufficientInstanceCapacity` and the gateway passes the error to the client. No other node is tried.

This means a cluster with available capacity can reject instance launches if the randomly-selected node happens to be full.

### Observed behavior (3-node cluster)

Node 3 already runs a `t3.medium` (2 vCPUs, 4 GB) and has no remaining capacity:

```
# First attempt — NATS routes to node3 (full) → error
$ aws ec2 run-instances --image-id $HIVE_AMI --instance-type t3.medium ...
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation: None

# Second attempt — NATS routes to node1 or node2 (free) → success
$ aws ec2 run-instances --image-id $HIVE_AMI --instance-type t3.medium ...
{ "Instances": [{ "InstanceId": "i-6c97c523f1ead0eef", "State": { "Name": "pending" } }] }
```

The user must retry manually until NATS happens to pick a node with capacity.

### Why this happens

```
AWS SDK → Gateway → NATS "ec2.RunInstances" [queue: hive-workers] → ONE daemon (random)
                                                                        ↓
                                                          canAllocate() fails
                                                                        ↓
                                                      InsufficientInstanceCapacity
                                                                        ↓
                                                    Gateway returns error to client
```

There is no capacity-aware routing. NATS blindly round-robins to any daemon, regardless of whether it can serve the requested instance type.

## Solution: per-instance-type NATS topics with dynamic subscription

Leverage NATS subscription semantics directly: each daemon subscribes to `ec2.RunInstances.{instanceType}` only for types it has capacity to serve. When a node fills up for a given type, it unsubscribes. When capacity is freed, it re-subscribes.

This means NATS itself becomes the capacity-aware router — requests are only delivered to nodes that can serve them. No retry logic, no discovery, no hacks.

### Fixed NATS topic flow

```
Gateway → "ec2.RunInstances.t3.medium" [queue: hive-workers] → only daemons with capacity
  → Daemon receives (guaranteed to have capacity) → allocates → launches VM
  → Daemon recalculates: t3.medium no longer fits → unsubscribes from ec2.RunInstances.t3.medium
  → Other daemons still subscribed → next request routed to them

All nodes full for t3.medium:
Gateway → "ec2.RunInstances.t3.medium" → no subscribers → NATS returns ErrNoResponders
  → Gateway maps ErrNoResponders to InsufficientInstanceCapacity → returns to client
```

### How dynamic subscription works

Each daemon maintains a set of per-instance-type NATS subscriptions. After every resource change (allocate or deallocate), it recalculates which types can fit at least 1 instance:

```
Node starts (8 vCPUs, 32 GB free):
  → subscribes to: ec2.RunInstances.t3.nano, .micro, .small, .medium, .large, .xlarge, .2xlarge
                    ec2.RunInstances.m8a.nano, .micro, .small, .medium, .large, .xlarge, .2xlarge

After launching t3.2xlarge (8 vCPU, 32 GB → 0 free):
  → unsubscribes from ALL ec2.RunInstances.* types (nothing fits)

After terminating t3.2xlarge (resources freed → 8 vCPU, 32 GB free again):
  → re-subscribes to all 14 instance types
```

### Race condition handling

There's a small window between a subscription existing and resources being allocated where two requests for the last available slot could arrive at the same node. This is handled naturally:

1. Request A arrives → daemon allocates → success → recalculates subscriptions
2. Request B arrives before unsubscribe completes → daemon tries to allocate → `canAllocate()` returns 0
3. Daemon responds with `InsufficientInstanceCapacity` (same as today)

This is an edge case that only occurs when a node is at exact capacity boundary with concurrent requests. The existing `canAllocate()` check remains as a safety net. This is acceptable — the alternative (distributed locking) adds complexity far exceeding the rare failure.

## Implementation

### Step 1: Gateway publishes to instance-type-specific topic

**File:** `hive/handlers/ec2/instance/service_nats.go`

Change:
```go
func (s *NATSInstanceService) RunInstances(input *ec2.RunInstancesInput) (*ec2.Reservation, error) {
    topic := fmt.Sprintf("ec2.RunInstances.%s", aws.StringValue(input.InstanceType))
    return utils.NATSRequest[ec2.Reservation](s.natsConn, topic, input, 5*time.Minute)
}
```

### Step 2: Gateway maps ErrNoResponders to InsufficientInstanceCapacity

**File:** `hive/utils/nats.go`

Update `NATSRequest` to detect `nats.ErrNoResponders` and return a specific error the gateway can map to `InsufficientInstanceCapacity`:

```go
msg, err := conn.Request(subject, jsonData, timeout)
if err != nil {
    if errors.Is(err, nats.ErrNoResponders) {
        return nil, fmt.Errorf("no responders for subject %s: %w", subject, nats.ErrNoResponders)
    }
    return nil, fmt.Errorf("NATS request failed: %w", err)
}
```

**File:** `hive/gateway/ec2/instance/RunInstances.go`

Map the no-responders error to the AWS-compatible error:
```go
reservationPtr, err := service.RunInstances(input)
if err != nil {
    if errors.Is(err, nats.ErrNoResponders) {
        return reservation, errors.New(awserrors.ErrorInsufficientInstanceCapacity)
    }
    return reservation, err
}
```

### Step 3: Add subscription management to ResourceManager

**File:** `hive/daemon/daemon.go`

Add to `ResourceManager`:
```go
type ResourceManager struct {
    // ... existing fields ...
    natsConn    *nats.Conn
    node        string
    instanceSubs map[string]*nats.Subscription  // topic → subscription
    handler      nats.MsgHandler                // RunInstances handler
}
```

Add methods:
```go
// updateInstanceSubscriptions recalculates which instance types can fit
// and subscribes/unsubscribes accordingly. Called after every allocate/deallocate.
func (rm *ResourceManager) updateInstanceSubscriptions() {
    rm.mu.RLock()
    defer rm.mu.RUnlock()

    for typeName, typeInfo := range rm.instanceTypes {
        topic := fmt.Sprintf("ec2.RunInstances.%s", typeName)
        canFit := rm.canAllocateUnlocked(typeInfo, 1) >= 1

        _, subscribed := rm.instanceSubs[topic]
        if canFit && !subscribed {
            sub, err := rm.natsConn.QueueSubscribe(topic, "hive-workers", rm.handler)
            if err != nil {
                slog.Error("Failed to subscribe to instance type topic", "topic", topic, "err", err)
                continue
            }
            rm.instanceSubs[topic] = sub
            slog.Info("Subscribed to instance type", "topic", topic)
        } else if !canFit && subscribed {
            rm.instanceSubs[topic].Unsubscribe()
            delete(rm.instanceSubs, topic)
            slog.Info("Unsubscribed from instance type (capacity full)", "topic", topic)
        }
    }
}
```

### Step 4: Call updateInstanceSubscriptions after allocate/deallocate

**File:** `hive/daemon/daemon.go`

Update `allocate()` to recalculate subscriptions after reserving resources:
```go
func (rm *ResourceManager) allocate(instanceType *ec2.InstanceTypeInfo) error {
    // ... existing allocation logic ...
    rm.updateInstanceSubscriptions()
    return nil
}
```

Update `deallocate()` to recalculate subscriptions after freeing resources:
```go
func (rm *ResourceManager) deallocate(instanceType *ec2.InstanceTypeInfo) {
    // ... existing deallocation logic ...
    rm.updateInstanceSubscriptions()
}
```

### Step 5: Remove static ec2.RunInstances subscription from subscribeAll()

**File:** `hive/daemon/daemon.go`

Remove `{"ec2.RunInstances", d.handleEC2RunInstances, "hive-workers"}` from the `subscribeAll()` table. The initial subscriptions are created by `updateInstanceSubscriptions()` called during daemon startup (after `NewResourceManager()` sets up the handler).

### Step 6: Initialize subscription management on daemon start

**File:** `hive/daemon/daemon.go`

After `subscribeAll()` in `Start()`, initialize the instance type subscriptions:
```go
d.resourceMgr.handler = d.handleEC2RunInstances
d.resourceMgr.natsConn = d.natsConn
d.resourceMgr.node = d.node
d.resourceMgr.instanceSubs = make(map[string]*nats.Subscription)
d.resourceMgr.updateInstanceSubscriptions()
```

### Step 7: Update tests

**File:** `hive/daemon/daemon_test.go`

- Update tests that publish to `ec2.RunInstances` to publish to `ec2.RunInstances.{instanceType}` instead
- Add test: verify that after filling a node, it unsubscribes from the filled type
- Add test: verify that after deallocating, it re-subscribes
- Add test: verify `ErrNoResponders` when no node has capacity

**File:** `hive/handlers/ec2/instance/service_nats_test.go` (if exists)

- Update topic assertions

## Files summary

| File | Change |
|------|--------|
| `hive/handlers/ec2/instance/service_nats.go` | Publish to `ec2.RunInstances.{instanceType}` |
| `hive/utils/nats.go` | Preserve `ErrNoResponders` in error chain |
| `hive/gateway/ec2/instance/RunInstances.go` | Map `ErrNoResponders` → `InsufficientInstanceCapacity` |
| `hive/daemon/daemon.go` | Add subscription management to ResourceManager, remove static subscription |
| `hive/daemon/daemon_test.go` | Update topics, add subscribe/unsubscribe tests |

## Existing behavior preserved

- `canAllocate()` still runs as a safety net (handles race conditions)
- Instance type validation (`instanceType, exists := d.resourceMgr.instanceTypes[...]`) still runs
- All other NATS subscriptions unchanged
- DescribeInstanceTypes still uses fan-out (no queue group) — unchanged

## Verification

1. `make test` — all tests pass
2. `make build` — compiles
3. 3-node cluster test:
   - Start cluster, all nodes subscribe to all 14 instance types
   - Fill node1 with t3.2xlarge — node1 unsubscribes from all types
   - `RunInstances t3.medium` → NATS routes to node2 or node3 (never node1)
   - Fill all nodes → `RunInstances` returns `InsufficientInstanceCapacity` (NATS ErrNoResponders)
   - Terminate instance on node1 → node1 re-subscribes → next RunInstances succeeds