mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
Julian Sommer
142 lines
9.1 KiB
Markdown
142 lines
9.1 KiB
Markdown
# Distributed StartInstances
|
|
|
|
Decouple stopped EC2 instances from their original daemon node, allowing any available daemon to handle `start-instances` requests.
|
|
|
|
## Problem
|
|
|
|
Previously, stopped instances remained bound to the daemon that ran them. The daemon kept an active `ec2.cmd.{instanceID}` NATS subscription after stop, so `start-instances` always routed back to the same node. In multi-node deployments, this meant a stopped instance could not be started if its original daemon was down or at capacity.
|
|
|
|
## Request Flow
|
|
|
|
```
|
|
AWS SDK (start-instances)
|
|
-> Gateway (port 9999): parse AWS query params
|
|
-> NATS ec2.start (queue group: hive-workers): any daemon can pick up
|
|
-> Daemon handleEC2StartStoppedInstance:
|
|
1. Load instance from shared NATS KV (key: instance.{instanceID})
|
|
2. Validate state is "stopped"
|
|
3. Re-initialize non-serialized fields (QMPClient, EBSRequests.Mu)
|
|
4. Allocate resources (CPU, memory)
|
|
5. LaunchInstance (mount volumes, start QEMU, subscribe to ec2.cmd.{instanceID})
|
|
6. Delete from shared KV
|
|
<- NATS response: {"status":"running","instanceId":"..."}
|
|
<- Gateway: build InstanceStateChange XML
|
|
<- AWS SDK: StartInstancesOutput
|
|
```
|
|
|
|
## How Ownership Release Works
|
|
|
|
When an instance is stopped (not terminated), the daemon releases ownership in this order:
|
|
|
|
1. Set `inst.LastNode = d.node` (record which daemon last ran it)
|
|
2. Write instance to shared KV at key `instance.{instanceID}`
|
|
3. Unsubscribe from `ec2.cmd.{instanceID}` NATS topic
|
|
4. Remove instance from local `d.Instances.VMS` map
|
|
5. Persist local state via `d.WriteState()` (now without the stopped instance)
|
|
|
|
The order is deliberate: shared KV is written **first**, then local state is cleaned up. If the daemon crashes between steps 2 and 5, `restoreInstances` on restart detects the overlap and completes the migration.
|
|
|
|
## Shared KV Storage
|
|
|
|
Stopped instances are stored in the existing `hive-instance-state` NATS JetStream KV bucket using the key prefix `instance.` (distinct from the per-node prefix `node.`).
|
|
|
|
Four methods on `JetStreamManager`:
|
|
- `WriteStoppedInstance(instanceID, *vm.VM)` — serialize and write to `instance.{instanceID}`
|
|
- `LoadStoppedInstance(instanceID)` — returns `nil, nil` if not found (not an error)
|
|
- `DeleteStoppedInstance(instanceID)` — idempotent (ignores `ErrKeyNotFound`)
|
|
- `ListStoppedInstances()` — iterates all `instance.*` keys, returns `[]*vm.VM`
|
|
|
|
## DescribeInstances Integration
|
|
|
|
`DescribeInstances` now includes stopped instances from shared KV. After the existing fan-out to all daemons (which returns running/pending instances from each node's local map), the gateway sends an additional NATS request to `ec2.DescribeStoppedInstances` (queue group, 3s timeout). The response contains reservations for all stopped instances in shared KV. These are merged into the final output.
|
|
|
|
If the stopped-instance query fails (timeout, no responders), it is logged and skipped — running instances are still returned. This matches the existing pattern where individual daemon fan-out failures are non-fatal.
|
|
|
|
## Migration on Restart
|
|
|
|
`restoreInstances` migrates pre-existing stopped instances from per-node state to shared KV:
|
|
|
|
- Instances in `StateStopped`: set `LastNode`, write to shared KV, remove from local map
|
|
- Instances in `StateStopping`: transition to `StateStopped`, then migrate
|
|
- `WriteState()` is called at the end to persist the removals
|
|
|
|
If migration fails for an instance (KV write error), the instance stays in local state as a fallback and will be retried on next restart.
|
|
|
|
## Files Changed
|
|
|
|
### VM struct: `hive/vm/vm.go`
|
|
|
|
Added `LastNode string` field (json: `last_node,omitempty`). Records which daemon node last ran the instance. Set during ownership release on stop. Used for diagnostics and future WAL recovery.
|
|
|
|
### KV methods: `hive/daemon/jetstream.go`
|
|
|
|
Added `StoppedInstancePrefix = "instance."` constant and four CRUD methods for stopped instance state. These reuse the `hive-instance-state` bucket — no new bucket required.
|
|
|
|
### Stop flow: `hive/daemon/daemon_handlers.go`
|
|
|
|
Extended the background goroutine in `handleStopOrTerminateInstance`. After `TransitionState(inst, StateStopped)`, the stop-only path (not terminate) releases ownership to shared KV. The terminate path is unchanged.
|
|
|
|
### New handlers: `hive/daemon/daemon_handlers.go`
|
|
|
|
`handleEC2StartStoppedInstance` — loads from shared KV, validates stopped state, re-initializes non-serialized fields, allocates resources, launches instance, removes from shared KV. On failure, rolls back resource allocation and local map insertion.
|
|
|
|
`handleEC2DescribeStoppedInstances` — lists all stopped instances from shared KV, applies instance ID filters from request payload, groups by reservation ID, returns `DescribeInstancesOutput`.
|
|
|
|
### Subscriptions: `hive/daemon/daemon.go`
|
|
|
|
Added to `subscribeAll()`:
|
|
- `ec2.start` with queue group `hive-workers` — load-balanced start requests
|
|
- `ec2.DescribeStoppedInstances` with queue group `hive-workers` — only one daemon needs to read shared KV
|
|
|
|
### Gateway: `hive/gateway/ec2/instance/StartInstances.go`
|
|
|
|
Rewritten to send `{"instance_id":"..."}` to `ec2.start` topic instead of constructing QMP commands and routing to `ec2.cmd.{instanceID}`. Removed `qmp` import.
|
|
|
|
### Gateway: `hive/gateway/ec2/instance/DescribeInstances.go`
|
|
|
|
Added a NATS request to `ec2.DescribeStoppedInstances` after the fan-out loop. Merges stopped instance reservations into the response.
|
|
|
|
### Restore: `hive/daemon/daemon.go`
|
|
|
|
`restoreInstances` now migrates stopped instances from per-node state to shared KV instead of re-subscribing to per-instance NATS topics.
|
|
|
|
## Design Decisions
|
|
|
|
**Why queue group for `ec2.start`?** NATS queue groups deliver each message to exactly one subscriber. This prevents two daemons from simultaneously trying to start the same instance, without needing distributed locks.
|
|
|
|
**Why shared KV instead of a database?** The `hive-instance-state` KV bucket already exists for per-node state. Reusing it avoids new infrastructure. NATS JetStream KV provides consistency guarantees sufficient for this use case (single writer per key during normal operation).
|
|
|
|
**Why write shared KV before removing local state?** Crash safety. If the daemon crashes after the KV write but before local cleanup, the instance exists in both locations. `restoreInstances` detects this overlap and completes the migration. The reverse order (remove local first, then write KV) would risk losing the instance entirely on crash.
|
|
|
|
**Why `LastNode` instead of re-routing to the original daemon?** The original daemon may be down, at capacity, or decommissioned. Recording which node last ran an instance is useful for debugging and future WAL recovery (Phase 2), but the start request goes to whichever daemon is available.
|
|
|
|
**Why not a dedicated KV bucket for stopped instances?** The `instance.` prefix provides sufficient namespace separation from `node.` prefix entries. A separate bucket would require additional JetStream configuration for no practical benefit.
|
|
|
|
## Error Codes
|
|
|
|
| Condition | AWS Error Code |
|
|
|-----------|---------------|
|
|
| No instance IDs provided | `InvalidParameterValue` (gateway) |
|
|
| Instance not found in shared KV | `InvalidInstanceID.NotFound` |
|
|
| Instance not in stopped state | `IncorrectInstanceState` |
|
|
| Resource allocation failure | `InsufficientInstanceCapacity` |
|
|
| LaunchInstance failure | `InternalError` |
|
|
| NATS timeout (no daemons available) | State reported as still stopped |
|
|
|
|
## Testing
|
|
|
|
**JetStream KV methods** (`jetstream_test.go`): 6 tests covering write/load round-trip, missing key returns nil, delete (including non-existent), list multiple instances, no interference with per-node state, nil KV bucket handling.
|
|
|
|
**Handler error paths** (`daemon_handlers_test.go`): 5 tests covering missing instance ID, instance not found in KV, non-stopped state, describe returns stopped instances, describe with instance ID filter.
|
|
|
|
**Gateway** (`StartInstances_test.go`): 6 tests covering single instance success, multiple instances, empty input, nil IDs skipped, NATS failure (reports stopped state), mixed success and failure.
|
|
|
|
**State restore** (`state_test.go`): 5 tests updated to verify stopped instances are migrated to shared KV during `restoreInstances` instead of kept in local state.
|
|
|
|
## Known Limitations
|
|
|
|
- **All daemons must be updated together.** Old daemons don't subscribe to `ec2.start` and won't release stopped instances to shared KV.
|
|
- **No WAL flush before ownership release.** Phase 1 relies on `ebs.unmount` during stop to flush volume data. Phase 2 will add explicit WAL flush to Predastore before releasing ownership.
|
|
- **No capacity-aware scheduling.** `ec2.start` uses simple NATS queue group round-robin. A daemon might accept a start request even if it lacks resources, failing at allocation time. The gateway reports this as "still stopped" to the caller.
|
|
- **TOCTOU on shared KV.** Between loading an instance from KV and deleting it after successful start, another daemon could theoretically load the same key. In practice this doesn't happen because NATS queue group delivers each `ec2.start` message to exactly one subscriber, and the instance ID in the message maps to a single KV key.
|