mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 09:01:31 +00:00
Move internal dev docs to private repo, trim CLAUDE.md for public use
Internal development plans, roadmap, and detailed dev process docs moved to private mulga repo. CLAUDE.md trimmed to remove SSH credentials, node IPs, and internal deployment workflows. Gosec fixes for file permissions and unhandled error.
This commit is contained in:
Ben Duncan
+3
-99
@@ -59,80 +59,6 @@ scripts/stop-dev.sh # Stop all services
|
||||
|
||||
Data directory: `~/hive/` (logs in `~/hive/logs/`, config in `~/hive/config/`)
|
||||
|
||||
## Development Process
|
||||
|
||||
All non-trivial work (features, bug fixes, improvements) follows a plan-first, document-as-you-go process. Plans are committed to git so they are visible to the whole team, reviewable in PRs, and serve as permanent documentation.
|
||||
|
||||
### When to write a plan
|
||||
|
||||
- **Required**: New features, multi-file changes, architectural changes, performance work, non-obvious bug fixes
|
||||
- **Not required**: Typo fixes, single-line bug fixes, config tweaks, small refactors contained to one function
|
||||
|
||||
### Plan location
|
||||
|
||||
Plans live in `docs/development/` under the appropriate category:
|
||||
|
||||
```
|
||||
docs/development/
|
||||
├── bugs/ # Bug fixes
|
||||
├── feature/ # New features
|
||||
└── improvements/ # Performance, refactoring, tech debt
|
||||
```
|
||||
|
||||
File naming: lowercase kebab-case describing the work, e.g. `nbd-performance.md`, `attach-volume.md`, `instance-state.md`.
|
||||
|
||||
### Plan lifecycle
|
||||
|
||||
#### 1. Planning phase
|
||||
|
||||
When entering plan mode for non-trivial work, **create the plan document first** and write it to the appropriate `docs/development/{category}/` path. The plan must include:
|
||||
|
||||
- **Summary** — What and why, in 1-3 sentences
|
||||
- **Context / Problem Statement** — Current behavior, what's wrong or missing
|
||||
- **Proposed Changes** — What will change, which files, key design decisions
|
||||
- **Files to Modify** — List of files with brief description of changes
|
||||
- **Testing** — How the changes will be verified
|
||||
|
||||
The plan document is committed to git before implementation begins. This makes the plan reviewable and visible to the team.
|
||||
|
||||
#### 2. Implementation phase
|
||||
|
||||
During implementation, **update the plan document as you go**:
|
||||
|
||||
- Mark completed steps (use `[x]` checkboxes or note "Done" inline)
|
||||
- Record findings, corrections, or deviations from the original plan
|
||||
- Add code references (file paths, line numbers, struct/function names) for what was actually implemented
|
||||
- Note any follow-up work discovered during implementation
|
||||
|
||||
#### 3. Completion
|
||||
|
||||
When development is complete, update the document to serve as **permanent reference documentation**:
|
||||
|
||||
- Add a status line near the top: `**Status: Complete**` (or `**Status: In Progress**`, `**Status: Planned**`)
|
||||
- Replace future-tense plan language with past-tense description of what was done
|
||||
- Add a **"Files Modified"** summary table
|
||||
- Add a **"Future Work"** section if there are known follow-ups
|
||||
- Keep the original context (benchmarks, research, root cause analysis) — this is valuable reference material
|
||||
- The document should read as a complete technical reference, not just a plan
|
||||
|
||||
#### Example status markers
|
||||
|
||||
```markdown
|
||||
**Status: Planned** — Awaiting review/approval before implementation
|
||||
**Status: In Progress** — Implementation underway
|
||||
**Status: Complete** — All changes implemented and tests passing
|
||||
```
|
||||
|
||||
### Git history as context
|
||||
|
||||
When starting work on a feature or investigating an issue, review the last 10-20 git commits (`git log --oneline -20`) to understand the current development context. Recent commits contain valuable context about in-progress features, architectural decisions, and related changes.
|
||||
|
||||
### Existing examples
|
||||
|
||||
- `docs/development/improvements/nbd-performance.md` — Complete improvement with benchmarks, root cause analysis, implementation details, and future work
|
||||
- `docs/development/feature/attach-volume.md` — Complete feature with request flow, files changed, and design decisions
|
||||
- `docs/development/bugs/instance-state.md` — Bug fix with problem statement, proposed architecture, and state transitions
|
||||
|
||||
## Project Standards
|
||||
|
||||
- Use `log/slog` instead of `log`. Use appropriate log level, eg `slog.Debug`
|
||||
@@ -140,6 +66,7 @@ When starting work on a feature or investigating an issue, review the last 10-20
|
||||
- For returning AWS errors, use `awserrors` package instead of manually typed strings
|
||||
|
||||
**Preflight Policy:**
|
||||
- **Claude MUST run `make preflight` and `make test` after any Go code changes** — these catch security issues (gosec), lint errors, and test failures before they reach CI. Do not skip this step.
|
||||
- Run `make preflight` before pushing any major changes — it runs the same checks as GitHub Actions (gofmt, go vet, gosec, staticcheck, govulncheck, and all unit tests)
|
||||
- All developers must install the pre-push hook to enforce this automatically:
|
||||
|
||||
@@ -188,29 +115,6 @@ mulga/
|
||||
AWS_PROFILE=hive aws ec2 describe-instances
|
||||
```
|
||||
|
||||
### Remote Test Cluster
|
||||
### IaC / Test Cluster
|
||||
|
||||
A multi-node Proxmox test cluster is available for validating changes on real hardware. Node IPs, SSH credentials, and connection details are in `docs/DEVNOTES.md` (gitignored, private to the repo).
|
||||
|
||||
**Lifecycle scripts** (`scripts/iac/`):
|
||||
|
||||
```bash
|
||||
# Source environment first
|
||||
source scripts/iac/proxmox/.env && source ~/prox
|
||||
|
||||
# Provision, configure, test, destroy
|
||||
./scripts/iac/hive-test.sh up <cluster-name> # Create VMs
|
||||
./scripts/iac/hive-test.sh configure <cluster-name> # Clone, build, form cluster, start services
|
||||
./scripts/iac/hive-test.sh test <cluster-name> # Smoke tests (describe-regions, instances, etc.)
|
||||
./scripts/iac/hive-test.sh status <cluster-name> # Health check all nodes
|
||||
./scripts/iac/hive-test.sh ssh <cluster-name> <N> # SSH to node N
|
||||
./scripts/iac/hive-test.sh down <cluster-name> # Destroy VMs
|
||||
```
|
||||
|
||||
**Deploying local changes to a running cluster:**
|
||||
|
||||
1. Read `docs/DEVNOTES.md` for the current cluster node IPs and SSH key path
|
||||
2. SSH key: `~/.ssh/us-west-1-tf-cloudinit`, user: `tf-user`
|
||||
3. Use `git diff` patches or `scp` to push changes, then rebuild on each node
|
||||
4. Daemon health endpoint is HTTP (not HTTPS): `curl -s http://<ip>:4432/health`
|
||||
5. Always `export PATH=/usr/local/go/bin:$PATH` on remote nodes before builds
|
||||
Infrastructure-as-code scripts for provisioning multi-node test clusters are in `scripts/iac/`. See `scripts/iac/proxmox/` for Proxmox-based provisioning with OpenTofu.
|
||||
|
||||
+6
-1
@@ -30,7 +30,12 @@ Thumbs.db
|
||||
# Test reports
|
||||
*-report.txt
|
||||
|
||||
# Agent direction (internal use only)
|
||||
# Internal docs (moved to private mulga repo)
|
||||
docs/INTERNAL.md
|
||||
docs/PRIVATE.md
|
||||
docs/DEVNOTES.md
|
||||
docs/HIVE_DEVELOPMENT_PLAN.md
|
||||
docs/PLAN.md
|
||||
docs/TODO.md
|
||||
docs/development/
|
||||
docs/internal/
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
-145
@@ -1,145 +0,0 @@
|
||||
# Plan: Implement CreateImage API
|
||||
|
||||
## Context
|
||||
|
||||
Users need the ability to create a new AMI from a running (or stopped) instance so they can launch new instances from snapshots of existing ones. This is the standard AWS `CreateImage` flow: snapshot the instance's root volume, then register a new AMI pointing at that snapshot. All the building blocks exist (CreateSnapshot, AMIMetadata, OpenFromSnapshot, cloneAMIToVolume) — they just need to be wired together.
|
||||
|
||||
## Flow
|
||||
|
||||
```
|
||||
AWS CLI: create-image --instance-id i-xxx --name "my-image"
|
||||
→ Gateway (ec2.go)
|
||||
→ NATS "ec2.CreateImage"
|
||||
→ Daemon handler
|
||||
→ Finds instance → gets root volume ID
|
||||
→ Calls snapshotService.CreateSnapshot(volumeID)
|
||||
→ Creates new AMI viperblock config with AMIMetadata.SnapshotID
|
||||
→ Stores as ami-xxx/config.json in S3
|
||||
→ Returns ami-xxx
|
||||
```
|
||||
|
||||
## Changes
|
||||
|
||||
### 1. Gateway: Add CreateImage action entry
|
||||
**File**: `hive/gateway/ec2.go`
|
||||
- Add `"CreateImage"` to the `ec2Actions` map, following the DescribeImages pattern
|
||||
|
||||
### 2. Gateway: CreateImage handler
|
||||
**File**: `hive/gateway/ec2/image/CreateImage.go` (new)
|
||||
- `ValidateCreateImageInput()` — validate InstanceId is present and starts with `i-`
|
||||
- `CreateImage()` — create NATSImageService, call `svc.CreateImage(input)`
|
||||
- Follow exact pattern of `gateway/ec2/snapshot/CreateSnapshot.go`
|
||||
|
||||
### 3. NATS service: Wire CreateImage to NATS
|
||||
**File**: `hive/handlers/ec2/image/service_nats.go`
|
||||
- Replace the `CreateImage` stub with a real NATS request to `"ec2.CreateImage"` topic
|
||||
- Use `utils.NATSRequest[ec2.CreateImageOutput]` with 120s timeout (snapshot can be slow)
|
||||
|
||||
### 4. Daemon: Add NATS subscription and handler
|
||||
**File**: `hive/daemon/daemon.go`
|
||||
- Add `{"ec2.CreateImage", d.handleEC2CreateImage, "hive-workers"}` to subscriptions
|
||||
- Update `NewImageServiceImpl(d.config)` → `NewImageServiceImpl(d.config, d.natsConn)` (~line 515)
|
||||
|
||||
**File**: `hive/daemon/daemon_handlers.go`
|
||||
- Add `handleEC2CreateImage` — this is a **stateful handler** (NOT handleNATSRequest) because it needs instance state to find the root volume
|
||||
- Steps:
|
||||
1. Unmarshal `ec2.CreateImageInput`
|
||||
2. Lock `d.Instances.Mu`, find instance in `d.Instances.VMS[*input.InstanceId]`, unlock
|
||||
3. Validate instance exists and is in running or stopped state
|
||||
4. Find root volume ID from `instance.Instance.BlockDeviceMappings` (first entry with `Ebs.VolumeId`)
|
||||
5. Get instance status (running vs stopped) — pass to service so it knows which snapshot path to use
|
||||
6. Get source AMI ID from `instance.Instance.ImageId` (for architecture, platform, etc.)
|
||||
7. Call `d.imageService.CreateImageFromInstance(params)` passing the extracted data
|
||||
8. Respond with JSON
|
||||
|
||||
### 5. Image service: Update interface and add CreateImageParams
|
||||
**File**: `hive/handlers/ec2/image/service.go`
|
||||
- Keep `CreateImage(input *ec2.CreateImageInput)` signature unchanged on the interface (gateway NATS side doesn't use extra params)
|
||||
|
||||
**File**: `hive/handlers/ec2/image/service_impl.go`
|
||||
- Add a `CreateImageParams` struct for the daemon-side call:
|
||||
```go
|
||||
type CreateImageParams struct {
|
||||
Input *ec2.CreateImageInput
|
||||
RootVolumeID string
|
||||
SourceImageID string
|
||||
IsRunning bool // true = use ebs.snapshot NATS, false = snapshot from S3 state
|
||||
}
|
||||
```
|
||||
- The daemon handler calls `d.imageService.CreateImageFromInstance(params)` directly (not through the generic interface)
|
||||
|
||||
### 6. Image service: Implement CreateImageFromInstance
|
||||
**File**: `hive/handlers/ec2/image/service_impl.go`
|
||||
|
||||
Add `natsConn *nats.Conn` to `ImageServiceImpl` (for running-instance NATS snapshot calls and offline viperblock instantiation for stopped instances). Update constructor.
|
||||
|
||||
Implement `CreateImageFromInstance(params CreateImageParams)`:
|
||||
1. Generate AMI ID: `utils.GenerateResourceID("ami")`
|
||||
2. Generate snapshot ID: `utils.GenerateResourceID("snap")`
|
||||
3. **Snapshot the root volume** — two paths:
|
||||
- **Running instance**: NATS `ebs.snapshot.{volumeID}` — reuse the exact pattern from `snapshot/service_impl.go:191-218`:
|
||||
- Marshal `config.EBSSnapshotRequest{Volume: rootVolumeID, SnapshotID: snapshotID}`
|
||||
- Send NATS request to `ebs.snapshot.{rootVolumeID}` with 30s timeout
|
||||
- Validate response
|
||||
- **Stopped instance**: Volume is not mounted in viperblockd. Data is fully persisted in S3. Create snapshot offline:
|
||||
- Instantiate viperblock via `newViperblock(rootVolumeID, ...)` (reuse pattern from `instance/service_impl.go:238-258`)
|
||||
- `backend.Init()` + `LoadStateRequest("")` to load config + block map from S3
|
||||
- Call `vb.CreateSnapshot(snapshotID)` directly — the flush/WAL steps are no-ops on a freshly loaded instance, it just serializes the block map into the snapshot checkpoint
|
||||
- `vb.RemoveLocalFiles()` to clean up temp files
|
||||
4. **Store snapshot metadata** in `{snapshotID}/metadata.json` — same pattern as `snapshot/service_impl.go:220-252`
|
||||
5. **Read source AMI config** from S3 `{sourceImageID}/config.json` to get architecture, platform, etc.
|
||||
6. **Read root volume config** from S3 `{rootVolumeID}/config.json` to get volume size
|
||||
7. **Build AMI VolumeConfig** with populated `AMIMetadata`:
|
||||
- `ImageID`: new ami-xxx
|
||||
- `Name`: from input
|
||||
- `Description`: from input
|
||||
- `SnapshotID`: the snap-xxx just created
|
||||
- `Architecture`, `PlatformDetails`, `Virtualization`, etc.: from source AMI (or defaults)
|
||||
- `VolumeSizeGiB`: from root volume config
|
||||
- `CreationDate`: now
|
||||
- `RootDeviceType`: "ebs"
|
||||
- `ImageOwnerAlias`: "self"
|
||||
8. **Store AMI config** to S3 as `{amiID}/config.json` — create a `VBState` with the VolumeConfig. This is what `DescribeImages` reads.
|
||||
9. Return `&ec2.CreateImageOutput{ImageId: &amiID}`
|
||||
|
||||
### 7. Tests
|
||||
- Add unit test for `CreateImage` in `hive/handlers/ec2/image/` with mock object store
|
||||
- Add gateway validation test in `hive/gateway/ec2/image/`
|
||||
|
||||
## Key Files
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `hive/gateway/ec2.go` | Add CreateImage action |
|
||||
| `hive/gateway/ec2/image/CreateImage.go` | New gateway handler |
|
||||
| `hive/handlers/ec2/image/service.go` | Update interface |
|
||||
| `hive/handlers/ec2/image/service_nats.go` | Wire NATS request |
|
||||
| `hive/handlers/ec2/image/service_impl.go` | Core implementation |
|
||||
| `hive/daemon/daemon.go` | Add subscription, update constructor |
|
||||
| `hive/daemon/daemon_handlers.go` | Add stateful handler |
|
||||
|
||||
## Existing Code to Reuse
|
||||
|
||||
- `snapshot/service_impl.go:191-218` — EBS snapshot NATS pattern (`ebs.snapshot.{volumeID}`)
|
||||
- `snapshot/service_impl.go:107-123` — `putSnapshotConfig` S3 storage pattern
|
||||
- `image/service_impl.go:96-126` — Reading viperblock VBState from S3 (`config.json`)
|
||||
- `instance/service_impl.go:238-258` — `newViperblock()` for offline viperblock instantiation
|
||||
- `utils.GenerateResourceID()` — ID generation
|
||||
- `utils.NATSRequest[T]()` — Generic NATS request helper
|
||||
- `viperblock.VBState`, `viperblock.VolumeConfig`, `viperblock.AMIMetadata` — Existing types
|
||||
|
||||
## Verification
|
||||
|
||||
1. `make build` — confirm compilation
|
||||
2. `make test` — all existing tests pass
|
||||
3. Manual test:
|
||||
```bash
|
||||
# Launch instance from existing AMI
|
||||
aws ec2 run-instances --image-id ami-xxx --instance-type t2.micro --key-name mykey
|
||||
# Create image from running instance
|
||||
aws ec2 create-image --instance-id i-xxx --name "test-snapshot-image"
|
||||
# Verify it appears
|
||||
aws ec2 describe-images --owners self
|
||||
# Launch new instance from the snapshot-based AMI
|
||||
aws ec2 run-instances --image-id ami-yyy --instance-type t2.micro --key-name mykey
|
||||
```
|
||||
-240
@@ -1,240 +0,0 @@
|
||||
# Hive Development Roadmap
|
||||
|
||||
## Update Jan 2026
|
||||
|
||||
### Big rocks:
|
||||
|
||||
- [DONE] Add multi-node support for predastore (S3) using reed solomon encoding
|
||||
- [DONE] Implement KV store for object lookup, WAL files, offset. Use hash-ring to determine which nodes an object belongs to.
|
||||
- Implement VPC support using Open vSwitch across multiple nodes, core VPC functionality included
|
||||
- Add support for NVIDIA Bluefield DPU with Open vSwitch
|
||||
- Implement basic IAM using NATS Jetstream as KV store, vs IAM/access-keys in local config/TOML files for beta.
|
||||
- [DONE] Move `daemon.go` instances.json state to Jetstream KV
|
||||
- Add support using the `hive` CLI tool to provision a new user with AWS access-keys/IAM.
|
||||
- Support multi-tenant operations and isolation
|
||||
- Add support to include capabilities when adding a new hardware node to MulgaOS (e.g EC2 target, S3, EBS, NATs, etc) - Features can be turned on/off depending on hardware scope.
|
||||
- [DONE] Add simple Web UI console, using the AWS JS SDK, communicating to local AWS gateway.
|
||||
- [DONE] Implement ShadCNblocks for UI framework
|
||||
- [DONE] Simple Go webserver, static files, easy build process.
|
||||
|
||||
### Implementation gaps:
|
||||
|
||||
- EC2 - Support extended features for `run-instance`
|
||||
- [DONE] Volume resize of AMI. Note `nbd` does not support live resize events, instance needs to be stopped, resized, and started.
|
||||
- Confirm cloud-init will resize volume on boot and configured correctly.
|
||||
- [DONE] Attach additional volumes
|
||||
- Attach to VPC / Security group (required Open vSwitch implementation)
|
||||
- only allow hive to use x% of cpu and memory. dont allow allocating 100% of resources.
|
||||
- placement group instances, allow spreading instance group between nodes.
|
||||
- capabilities flag
|
||||
- more instance types - per cpu gen
|
||||
- [DONE] multi part uploads on frontend
|
||||
- [DONE] delete volumes on termination
|
||||
- fix describe instances
|
||||
- [DONE] change all log into slog
|
||||
- `nbd` does not support resizing disks live. Requires instance to be stopped, boot/root volume, or attached volume will need to be resized, and instance started again. Limitation for Hive v1 using NBD, aiming to resolve with `vhost-user-blk` to create a `virtio-blk` device for extended functionality and performance.
|
||||
- Migrate to net/http + chi from fiber
|
||||
- remove insecure skip verify - need to add predastore multi node ips to certs
|
||||
- migrate to list objects v2
|
||||
- split up utils.go
|
||||
- make sure nats is running before we start daemon? eigw and account service will fallback to memory versions but that breaks all functionality
|
||||
- the BlockDeviceMappings > DeviceName (/dev/xvda) is not the true volume, i think it's /dev/vda on mine. I think qmp has a query device stats which will provide block devices so we can match these up. not a blocker, but needs to be fixed eventually
|
||||
- aws ec2 modify-instance-attribute
|
||||
|
||||
### Issues to investigate
|
||||
|
||||
- When the host disk volume is full and a new instance is launched, disk corruption will occur within the instance since the host volume is out of space. Consider a pre-flight check, check available storage on the S3 server, local viperblock for WAL cache availability and provide improved guard rails to prevent a new instance running, if available disk space is nearing to be exceeded.
|
||||
|
||||
Host:
|
||||
|
||||
```json
|
||||
{
|
||||
"time": "2026-01-27T21:30:29.368164556+10:00",
|
||||
"level": "ERROR",
|
||||
"msg": "WAL sync failed",
|
||||
"error": "write /home/hive/hive/predastore/distributed/nodes/node-2/0000000000000737.wal: no space left on device"
|
||||
}
|
||||
```
|
||||
|
||||
Instance `dmesg`:
|
||||
|
||||
```
|
||||
[ 1445.573495] EXT4-fs error (device vda1): ext4_journal_check_start:83: comm systemd-journal: Detected aborted journal
|
||||
[ 1445.583565] EXT4-fs (vda1): Remounting filesystem read-only
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
|
||||
|
||||
## Update Nov 2025
|
||||
|
||||
[PARTIAL] _ Implement multi-tenant support
|
||||
[PARTIAL] _ Move config settings from ~/hive/\*.toml, to using Nats Jetstream for core config which can be synced between nodes.
|
||||
|
||||
- Implement a lightweight IAM using Nats Jetstream, vs current config files for auth settings.
|
||||
[ONGOING] _ Implement Reed Solomon Encoding for Predastore (S3) objects and a WAL implementation, for storing multiple objects in a single shard (e.g 4MB), with the WAL referencing the location of each object (e.g 4kb min)
|
||||
_ Implement basic KY lookup, object key sha512 (bucket/key), location to shard on S3 (obj.0000124.bin), read WAL (e.g first 4096 bytes) to determine location of the object. e.g key-1234 => obj.0000124.bin, wal header (4096 bytes) > key-1234 == offset location (seek) 16384, len 32768.
|
||||
_ Read multiple nodes (e.g 5 predastore instances, k = 3 (data shards), n = 5 (total shards), n - k = 2 (parity shards) )
|
||||
[ONGOING] _ Complete core scaffolding AWS SDK/API requirements (ec2 describe-instances, run-instances, etc)
|
||||
- Implement UEFI support for image downloads and `qemu` exec in `vm.go`
|
||||
- Confirm Alpine Linux, fails import image AMI > (run-instance) ""Failed to read block from AMI source" err="request out of range" - Block size correct?
|
||||
- Improve shutdown gracefully, `./scripts/stop-dev.sh` waits 60 seconds, while qemu/nbd could still be shutting down.
|
||||
- [DONE] Add delete-volume support via EBS (s3 vol-\*) for terminated instance (DeleteOnTermination flag)
|
||||
- Add default LRU cache support for viperblock, depending on the instance type / volume size and system memory available.
|
||||
|
||||
## Multi-node setup
|
||||
|
||||
[DONE] - Development release - v0.1.0 (single node)
|
||||
[PARTIAL] - Production release - v0.2.0 (multi node)
|
||||
|
||||
Design layout for multi-node configuration.
|
||||
|
||||
```
|
||||
# node1
|
||||
hive admin init --region ap-southeast-2 --az ap-southeast-2a --node node1 --nodes 3 --hive-dir ~/hive/
|
||||
|
||||
# node2
|
||||
hive admin join --region ap-southeast-2 --az ap-southeast-2a --node node2 --host node1.local:8443 --data-dir ~/hive/
|
||||
|
||||
# node3, optionally toggle EBS/EC2/NATs support only
|
||||
hive admin join --region ap-southeast-2 --az ap-southeast-2a --node node3 --host node1.local:8443 --data-dir ~/hive/ --cap ebs,ec2,nats
|
||||
```
|
||||
|
||||
If --host is missing, the join command tries multicast broadcast to find parent (leader) node.
|
||||
|
||||
Set region with `--region ap-southeast-2` which will create a new Hive cluster for the specified region.
|
||||
|
||||
For production, it is recommended to run Hive on at least three physical nodes. In this minimum setup, Hive S3 and EBS volumes use 2 data shards and 1 parity shard RS(2,1) to distribute each chunk across the cluster. This configuration tolerates a full node failure while adding only 1.5x storage overhead.
|
||||
|
||||
### Overview:
|
||||
|
||||
- init creates a cluster-id, node-id, and a short join token, starts a tiny control server on :8443, and writes DNS hint files if available.
|
||||
|
||||
- init sets target size to 3 and waits until all 3 nodes join and ack.
|
||||
|
||||
- join nodes contact node1, present the token or cluster-id, and advertise capabilities.
|
||||
|
||||
- node1 appends them to the member set and immediately pushes the current settings bundle to them.
|
||||
|
||||
- When member count reaches 3, node1 broadcasts the final cluster settings to all nodes.
|
||||
|
||||
- Every node writes the same cluster.json and starts the services that match its capabilities.
|
||||
|
||||
Node exchange payload
|
||||
|
||||
```json
|
||||
{
|
||||
"cluster_id": "c-82d5",
|
||||
"node_id": "n-rpi2",
|
||||
"addr": "rpi2.local:8443",
|
||||
"version": "0.1.0",
|
||||
"caps": ["ec2", "s3"],
|
||||
"ts": 1731388800
|
||||
}
|
||||
```
|
||||
|
||||
Settings bundle (identical on all 3 once committed)
|
||||
|
||||
```json
|
||||
{
|
||||
"cluster_id": "c-82d5",
|
||||
"target_size": 3,
|
||||
"members": [
|
||||
{
|
||||
"node_id": "n-rpi1",
|
||||
"addr": "rpi1.local:8443",
|
||||
"caps": ["ec2", "s3", "nats", "ebs"]
|
||||
},
|
||||
{ "node_id": "n-rpi2", "addr": "rpi2.local:8443", "caps": ["ec2", "s3"] },
|
||||
{ "node_id": "n-rpi3", "addr": "rpi3.local:8443", "caps": ["nats", "ebs"] }
|
||||
],
|
||||
"services": {
|
||||
"ec2": { "api_bind": ":9001" },
|
||||
"s3": { "api_bind": ":9002", "replicas": 2 },
|
||||
"nats": { "cluster": "c-82d5", "quorum": 2 },
|
||||
"ebs": { "replicas": 3, "block_size": 4096 }
|
||||
},
|
||||
"epoch": 1,
|
||||
"sig": "ed25519:..." // signed by rpi1 during init
|
||||
}
|
||||
```
|
||||
|
||||
## Original TODO Items → Development Plan Integration
|
||||
|
||||
### ✅ **Completed Integration**
|
||||
|
||||
All original TODO items have been incorporated into the structured development phases:
|
||||
|
||||
| Original TODO | Status | Integrated Into | Phase |
|
||||
| ------------------------------------- | ---------- | ---------------------------------------------- | --------- |
|
||||
| #1: Binary compile and install.sh | ✅ Planned | Phase 7: Task 7.1 (Binary Compilation) | Phase 7 |
|
||||
| #2: Move daemon.go to services/hived/ | ✅ Planned | Phase 3: Task 3.2 (Specialized Services) | Phase 3 |
|
||||
| #3: VPC with openvs-switch as `vpcd` | ✅ Planned | Phase 5: Task 5.1 (VPC with Open vSwitch) | Phase 5 |
|
||||
| #4: AWS HTTP gateway (`awsd`) | ✅ Planned | Phase 3: Task 3.1 (AWS Gateway Service) | Phase 3 |
|
||||
| #5: AWS SDK v2 input/output | ✅ Planned | Phase 2: Task 2.1 & Phase 3: Task 3.1 | Phase 2-3 |
|
||||
| #6: `hive.service` for systemd | ✅ Planned | Phase 7: Task 7.2 (System Service Integration) | Phase 7 |
|
||||
| #7: Gossip and RAFT protocols | ✅ Planned | Phase 0: Task 0.1 (Service Registry) | Phase 0 |
|
||||
| #8: etcd/KV for configuration sync | ✅ Planned | Phase 0: Task 0.1 (Distributed Config) | Phase 0 |
|
||||
| #9: Smithy model code generation | ✅ Planned | Phase 2: Task 2.1 (Smithy Code Generation) | Phase 2 |
|
||||
|
||||
## Development Phase Overview
|
||||
|
||||
**Phase 0**: Distributed Systems Foundation (1-2 weeks)
|
||||
|
||||
- Gossip, RAFT, and distributed configuration (#7, #8)
|
||||
|
||||
**Phase 1**: Development Environment Automation (2-3 weeks)
|
||||
|
||||
- Multi-service orchestration and hot reloading
|
||||
|
||||
**Phase 2**: AWS API Model Implementation (3-4 weeks)
|
||||
|
||||
- Smithy-based code generation (#9)
|
||||
- AWS SDK v2 integration (#5)
|
||||
|
||||
**Phase 3**: Scalable Gateway and Daemon Architecture (2-3 weeks)
|
||||
|
||||
- AWS gateway service `awsd` (#4)
|
||||
- Service refactoring to `hived` (#2)
|
||||
- VPC daemon `vpcd` foundation (#3)
|
||||
|
||||
**Phase 4**: Service Integration and Orchestration (3-4 weeks)
|
||||
|
||||
- NATS clustering and cross-service coordination
|
||||
|
||||
**Phase 5**: Infrastructure Services (4-5 weeks)
|
||||
|
||||
- VPC networking with Open vSwitch (#3)
|
||||
- Advanced VM features
|
||||
|
||||
**Phase 6**: Testing and Validation Framework (2-3 weeks)
|
||||
|
||||
- AWS CLI compatibility testing
|
||||
|
||||
**Phase 7**: Production Deployment and Packaging (2-3 weeks)
|
||||
|
||||
- Binary compilation and distribution (#1)
|
||||
- Systemd service integration (#6)
|
||||
- Production configuration management
|
||||
|
||||
## Service Architecture
|
||||
|
||||
The development plan implements these services:
|
||||
|
||||
- **`awsd`** - AWS API Gateway (TODO #4)
|
||||
- **`hived`** - Main Hive compute daemon (TODO #2)
|
||||
- **`vpcd`** - VPC networking daemon (TODO #3)
|
||||
- **Predastore** - S3 service (existing)
|
||||
- **Viperblock** - EBS service (existing)
|
||||
|
||||
## Getting Started
|
||||
|
||||
For development setup:
|
||||
|
||||
```bash
|
||||
./scripts/dev-setup.sh # Setup complete development environment
|
||||
./scripts/start-dev.sh # Start all services
|
||||
```
|
||||
|
||||
For detailed implementation guidance, see [HIVE_DEVELOPMENT_PLAN.md](HIVE_DEVELOPMENT_PLAN.md).
|
||||
@@ -1,253 +0,0 @@
|
||||
# RunInstances capacity routing via per-instance-type NATS subscriptions
|
||||
|
||||
**Status:** Implemented (local tests pass, pending remote verification)
|
||||
|
||||
## Problem
|
||||
|
||||
When `RunInstances` is called, the gateway publishes to NATS topic `ec2.RunInstances` with queue group `hive-workers`. NATS round-robins the request to one daemon. If that daemon has insufficient capacity, it returns `InsufficientInstanceCapacity` and the gateway passes the error to the client. No other node is tried.
|
||||
|
||||
This means a cluster with available capacity can reject instance launches if the randomly-selected node happens to be full.
|
||||
|
||||
### Observed behavior (3-node cluster)
|
||||
|
||||
Node 3 already runs a `t3.medium` (2 vCPUs, 4 GB) and has no remaining capacity:
|
||||
|
||||
```
|
||||
# First attempt — NATS routes to node3 (full) → error
|
||||
$ aws ec2 run-instances --image-id $HIVE_AMI --instance-type t3.medium ...
|
||||
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation: None
|
||||
|
||||
# Second attempt — NATS routes to node1 or node2 (free) → success
|
||||
$ aws ec2 run-instances --image-id $HIVE_AMI --instance-type t3.medium ...
|
||||
{ "Instances": [{ "InstanceId": "i-6c97c523f1ead0eef", "State": { "Name": "pending" } }] }
|
||||
```
|
||||
|
||||
The user must retry manually until NATS happens to pick a node with capacity.
|
||||
|
||||
### Why this happens
|
||||
|
||||
```
|
||||
AWS SDK → Gateway → NATS "ec2.RunInstances" [queue: hive-workers] → ONE daemon (random)
|
||||
↓
|
||||
canAllocate() fails
|
||||
↓
|
||||
InsufficientInstanceCapacity
|
||||
↓
|
||||
Gateway returns error to client
|
||||
```
|
||||
|
||||
There is no capacity-aware routing. NATS blindly round-robins to any daemon, regardless of whether it can serve the requested instance type.
|
||||
|
||||
## Solution: per-instance-type NATS topics with dynamic subscription
|
||||
|
||||
Leverage NATS subscription semantics directly: each daemon subscribes to `ec2.RunInstances.{instanceType}` only for types it has capacity to serve. When a node fills up for a given type, it unsubscribes. When capacity is freed, it re-subscribes.
|
||||
|
||||
This means NATS itself becomes the capacity-aware router — requests are only delivered to nodes that can serve them. No retry logic, no discovery, no hacks.
|
||||
|
||||
### Fixed NATS topic flow
|
||||
|
||||
```
|
||||
Gateway → "ec2.RunInstances.t3.medium" [queue: hive-workers] → only daemons with capacity
|
||||
→ Daemon receives (guaranteed to have capacity) → allocates → launches VM
|
||||
→ Daemon recalculates: t3.medium no longer fits → unsubscribes from ec2.RunInstances.t3.medium
|
||||
→ Other daemons still subscribed → next request routed to them
|
||||
|
||||
All nodes full for t3.medium:
|
||||
Gateway → "ec2.RunInstances.t3.medium" → no subscribers → NATS returns ErrNoResponders
|
||||
→ Gateway maps ErrNoResponders to InsufficientInstanceCapacity → returns to client
|
||||
```
|
||||
|
||||
### How dynamic subscription works
|
||||
|
||||
Each daemon maintains a set of per-instance-type NATS subscriptions. After every resource change (allocate or deallocate), it recalculates which types can fit at least 1 instance:
|
||||
|
||||
```
|
||||
Node starts (8 vCPUs, 32 GB free):
|
||||
→ subscribes to: ec2.RunInstances.t3.nano, .micro, .small, .medium, .large, .xlarge, .2xlarge
|
||||
ec2.RunInstances.m8a.nano, .micro, .small, .medium, .large, .xlarge, .2xlarge
|
||||
|
||||
After launching t3.2xlarge (8 vCPU, 32 GB → 0 free):
|
||||
→ unsubscribes from ALL ec2.RunInstances.* types (nothing fits)
|
||||
|
||||
After terminating t3.2xlarge (resources freed → 8 vCPU, 32 GB free again):
|
||||
→ re-subscribes to all 14 instance types
|
||||
```
|
||||
|
||||
### Race condition handling
|
||||
|
||||
There's a small window between a subscription existing and resources being allocated where two requests for the last available slot could arrive at the same node. This is handled naturally:
|
||||
|
||||
1. Request A arrives → daemon allocates → success → recalculates subscriptions
|
||||
2. Request B arrives before unsubscribe completes → daemon tries to allocate → `canAllocate()` returns 0
|
||||
3. Daemon responds with `InsufficientInstanceCapacity` (same as today)
|
||||
|
||||
This is an edge case that only occurs when a node is at exact capacity boundary with concurrent requests. The existing `canAllocate()` check remains as a safety net. This is acceptable — the alternative (distributed locking) adds complexity far exceeding the rare failure.
|
||||
|
||||
## Implementation
|
||||
|
||||
### Step 1: Gateway publishes to instance-type-specific topic
|
||||
|
||||
**File:** `hive/handlers/ec2/instance/service_nats.go`
|
||||
|
||||
Change:
|
||||
```go
|
||||
func (s *NATSInstanceService) RunInstances(input *ec2.RunInstancesInput) (*ec2.Reservation, error) {
|
||||
topic := fmt.Sprintf("ec2.RunInstances.%s", aws.StringValue(input.InstanceType))
|
||||
return utils.NATSRequest[ec2.Reservation](s.natsConn, topic, input, 5*time.Minute)
|
||||
}
|
||||
```
|
||||
|
||||
### Step 2: Gateway maps ErrNoResponders to InsufficientInstanceCapacity
|
||||
|
||||
**File:** `hive/utils/nats.go`
|
||||
|
||||
Update `NATSRequest` to detect `nats.ErrNoResponders` and return a specific error the gateway can map to `InsufficientInstanceCapacity`:
|
||||
|
||||
```go
|
||||
msg, err := conn.Request(subject, jsonData, timeout)
|
||||
if err != nil {
|
||||
if errors.Is(err, nats.ErrNoResponders) {
|
||||
return nil, fmt.Errorf("no responders for subject %s: %w", subject, nats.ErrNoResponders)
|
||||
}
|
||||
return nil, fmt.Errorf("NATS request failed: %w", err)
|
||||
}
|
||||
```
|
||||
|
||||
**File:** `hive/gateway/ec2/instance/RunInstances.go`
|
||||
|
||||
Map the no-responders error to the AWS-compatible error:
|
||||
```go
|
||||
reservationPtr, err := service.RunInstances(input)
|
||||
if err != nil {
|
||||
if errors.Is(err, nats.ErrNoResponders) {
|
||||
return reservation, errors.New(awserrors.ErrorInsufficientInstanceCapacity)
|
||||
}
|
||||
return reservation, err
|
||||
}
|
||||
```
|
||||
|
||||
### Step 3: Add subscription management to ResourceManager
|
||||
|
||||
**File:** `hive/daemon/daemon.go`
|
||||
|
||||
Add to `ResourceManager`:
|
||||
```go
|
||||
type ResourceManager struct {
|
||||
// ... existing fields ...
|
||||
natsConn *nats.Conn
|
||||
node string
|
||||
instanceSubs map[string]*nats.Subscription // topic → subscription
|
||||
handler nats.MsgHandler // RunInstances handler
|
||||
}
|
||||
```
|
||||
|
||||
Add methods:
|
||||
```go
|
||||
// updateInstanceSubscriptions recalculates which instance types can fit
|
||||
// and subscribes/unsubscribes accordingly. Called after every allocate/deallocate.
|
||||
func (rm *ResourceManager) updateInstanceSubscriptions() {
|
||||
rm.mu.RLock()
|
||||
defer rm.mu.RUnlock()
|
||||
|
||||
for typeName, typeInfo := range rm.instanceTypes {
|
||||
topic := fmt.Sprintf("ec2.RunInstances.%s", typeName)
|
||||
canFit := rm.canAllocateUnlocked(typeInfo, 1) >= 1
|
||||
|
||||
_, subscribed := rm.instanceSubs[topic]
|
||||
if canFit && !subscribed {
|
||||
sub, err := rm.natsConn.QueueSubscribe(topic, "hive-workers", rm.handler)
|
||||
if err != nil {
|
||||
slog.Error("Failed to subscribe to instance type topic", "topic", topic, "err", err)
|
||||
continue
|
||||
}
|
||||
rm.instanceSubs[topic] = sub
|
||||
slog.Info("Subscribed to instance type", "topic", topic)
|
||||
} else if !canFit && subscribed {
|
||||
rm.instanceSubs[topic].Unsubscribe()
|
||||
delete(rm.instanceSubs, topic)
|
||||
slog.Info("Unsubscribed from instance type (capacity full)", "topic", topic)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Step 4: Call updateInstanceSubscriptions after allocate/deallocate
|
||||
|
||||
**File:** `hive/daemon/daemon.go`
|
||||
|
||||
Update `allocate()` to recalculate subscriptions after reserving resources:
|
||||
```go
|
||||
func (rm *ResourceManager) allocate(instanceType *ec2.InstanceTypeInfo) error {
|
||||
// ... existing allocation logic ...
|
||||
rm.updateInstanceSubscriptions()
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
Update `deallocate()` to recalculate subscriptions after freeing resources:
|
||||
```go
|
||||
func (rm *ResourceManager) deallocate(instanceType *ec2.InstanceTypeInfo) {
|
||||
// ... existing deallocation logic ...
|
||||
rm.updateInstanceSubscriptions()
|
||||
}
|
||||
```
|
||||
|
||||
### Step 5: Remove static ec2.RunInstances subscription from subscribeAll()
|
||||
|
||||
**File:** `hive/daemon/daemon.go`
|
||||
|
||||
Remove `{"ec2.RunInstances", d.handleEC2RunInstances, "hive-workers"}` from the `subscribeAll()` table. The initial subscriptions are created by `updateInstanceSubscriptions()` called during daemon startup (after `NewResourceManager()` sets up the handler).
|
||||
|
||||
### Step 6: Initialize subscription management on daemon start
|
||||
|
||||
**File:** `hive/daemon/daemon.go`
|
||||
|
||||
After `subscribeAll()` in `Start()`, initialize the instance type subscriptions:
|
||||
```go
|
||||
d.resourceMgr.handler = d.handleEC2RunInstances
|
||||
d.resourceMgr.natsConn = d.natsConn
|
||||
d.resourceMgr.node = d.node
|
||||
d.resourceMgr.instanceSubs = make(map[string]*nats.Subscription)
|
||||
d.resourceMgr.updateInstanceSubscriptions()
|
||||
```
|
||||
|
||||
### Step 7: Update tests
|
||||
|
||||
**File:** `hive/daemon/daemon_test.go`
|
||||
|
||||
- Update tests that publish to `ec2.RunInstances` to publish to `ec2.RunInstances.{instanceType}` instead
|
||||
- Add test: verify that after filling a node, it unsubscribes from the filled type
|
||||
- Add test: verify that after deallocating, it re-subscribes
|
||||
- Add test: verify `ErrNoResponders` when no node has capacity
|
||||
|
||||
**File:** `hive/handlers/ec2/instance/service_nats_test.go` (if exists)
|
||||
|
||||
- Update topic assertions
|
||||
|
||||
## Files summary
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `hive/handlers/ec2/instance/service_nats.go` | Publish to `ec2.RunInstances.{instanceType}` |
|
||||
| `hive/utils/nats.go` | Preserve `ErrNoResponders` in error chain |
|
||||
| `hive/gateway/ec2/instance/RunInstances.go` | Map `ErrNoResponders` → `InsufficientInstanceCapacity` |
|
||||
| `hive/daemon/daemon.go` | Add subscription management to ResourceManager, remove static subscription |
|
||||
| `hive/daemon/daemon_test.go` | Update topics, add subscribe/unsubscribe tests |
|
||||
|
||||
## Existing behavior preserved
|
||||
|
||||
- `canAllocate()` still runs as a safety net (handles race conditions)
|
||||
- Instance type validation (`instanceType, exists := d.resourceMgr.instanceTypes[...]`) still runs
|
||||
- All other NATS subscriptions unchanged
|
||||
- DescribeInstanceTypes still uses fan-out (no queue group) — unchanged
|
||||
|
||||
## Verification
|
||||
|
||||
1. `make test` — all tests pass
|
||||
2. `make build` — compiles
|
||||
3. 3-node cluster test:
|
||||
- Start cluster, all nodes subscribe to all 14 instance types
|
||||
- Fill node1 with t3.2xlarge — node1 unsubscribes from all types
|
||||
- `RunInstances t3.medium` → NATS routes to node2 or node3 (never node1)
|
||||
- Fill all nodes → `RunInstances` returns `InsufficientInstanceCapacity` (NATS ErrNoResponders)
|
||||
- Terminate instance on node1 → node1 re-subscribes → next RunInstances succeeds
|
||||
@@ -1,127 +0,0 @@
|
||||
# Instance State Management
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The current implementation tightly couples EC2 instances to specific daemon nodes throughout their entire lifecycle. When an instance is stopped, the daemon retains ownership and NATS subscriptions, preventing instances from being started on different nodes. This creates issues in multi-node deployments and prevents proper recovery from ungraceful shutdowns.
|
||||
|
||||
## Goals
|
||||
|
||||
1. Enable stopped instances to be started on any available node
|
||||
2. Handle ungraceful shutdown scenarios (QEMU killed, viperblock/nbdkit crashes)
|
||||
3. Ensure data consistency when instances migrate between nodes
|
||||
4. Improve test coverage for edge cases and failure scenarios
|
||||
|
||||
## Current Behavior
|
||||
|
||||
- Daemon owns an instance throughout its entire lifecycle (running and stopped)
|
||||
- `ec2.cmd.$instance` NATS subscription remains on the original node even when stopped
|
||||
- StartInstance commands route directly to the original daemon
|
||||
- This works correctly for single-node deployments but breaks multi-node scenarios
|
||||
|
||||
## Proposed Architecture
|
||||
|
||||
### Instance Ownership Model
|
||||
|
||||
**Running Instance:**
|
||||
- Daemon owns the instance and subscribes to `ec2.cmd.$instance`
|
||||
- All commands (stop, reboot, terminate, etc.) route directly to the owning daemon
|
||||
|
||||
**Stopped Instance:**
|
||||
- Daemon releases the `ec2.cmd.$instance` subscription
|
||||
- Instance state persists in NATS KV (no daemon ownership)
|
||||
- Gateway handles StartInstance, routing to any available daemon
|
||||
|
||||
### State Transitions
|
||||
|
||||
```
|
||||
[Stopped] --StartInstance--> Gateway routes to available daemon --> [Running]
|
||||
[Running] --StopInstance--> Daemon releases subscription --> [Stopped]
|
||||
[Running] --Crash/Kill--> Daemon detects, handles cleanup --> [Requires Recovery]
|
||||
```
|
||||
|
||||
### Viperblock WAL Synchronization
|
||||
|
||||
Viperblock maintains a local WAL on NVMe for performance. On ungraceful shutdown, the WAL may contain blocks not yet pushed to Predastore (S3).
|
||||
|
||||
**Graceful Shutdown:**
|
||||
1. Stop QEMU
|
||||
2. Flush viperblock WAL to Predastore
|
||||
3. Release `ec2.cmd.$instance` subscription
|
||||
4. Instance can start on any node
|
||||
|
||||
**Ungraceful Shutdown (crash, kill, hardware failure):**
|
||||
1. Instance marked as `requires-recovery` in NATS KV
|
||||
2. WAL data remains on original node's NVMe
|
||||
3. Instance MUST restart on same node OR original node must sync WAL first
|
||||
|
||||
**Recovery Sequence:**
|
||||
1. StartInstance received for instance in `requires-recovery` state
|
||||
2. Gateway queries original node: "Can you sync WAL for instance X?"
|
||||
3. Original node syncs WAL to Predastore
|
||||
4. Instance state updated to `stopped`
|
||||
5. Any available node can now start the instance
|
||||
|
||||
### Edge Case: Original Node at Capacity
|
||||
|
||||
If the original node has no resources available and holds unsync'd WAL data:
|
||||
1. Original node still syncs WAL to Predastore (sync doesn't require running the instance)
|
||||
2. After sync completes, instance can start on any other available node
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Refactor Instance Ownership
|
||||
|
||||
1. **Modify StopInstance handler** - Release `ec2.cmd.$instance` subscription after graceful stop
|
||||
2. **Add Gateway StartInstance handler** - Route to available daemon via queue group
|
||||
3. **Update daemon availability tracking** - Only listen for new instances when resources available
|
||||
4. **Update NATS KV schema** - Track instance state and last-known node
|
||||
|
||||
### Phase 2: Graceful Shutdown Flow
|
||||
|
||||
1. **Implement WAL sync on stop** - Ensure viperblock flushes to Predastore before releasing ownership
|
||||
2. **Add sync confirmation** - Daemon confirms WAL sync complete before releasing subscription
|
||||
3. **Update instance metadata** - Store `wal_synced: true` in NATS KV
|
||||
|
||||
### Phase 3: Crash Detection and Recovery
|
||||
|
||||
1. **Detect ungraceful shutdown** - Monitor QEMU, nbdkit, viperblock processes
|
||||
2. **Mark instances for recovery** - Set `requires_recovery: true` and `last_node: $node_id`
|
||||
3. **Implement recovery endpoint** - `POST /recover` triggers WAL sync from original node
|
||||
4. **Add recovery check to StartInstance** - Block start until WAL synced
|
||||
|
||||
### Phase 4: Testing
|
||||
|
||||
**Unit Tests:**
|
||||
- Instance state transitions (running → stopped → running)
|
||||
- Ownership release on stop
|
||||
- Ownership acquisition on start
|
||||
- WAL sync completion tracking
|
||||
|
||||
**Integration Tests:**
|
||||
- Multi-node instance migration
|
||||
- Graceful shutdown and restart on different node
|
||||
- Concurrent StartInstance requests
|
||||
|
||||
**Edge Case Tests:**
|
||||
- Kill QEMU process mid-operation
|
||||
- Kill viperblock/nbdkit processes
|
||||
- Corrupt NATS KV state (instance shows wrong state)
|
||||
- Node failure during WAL sync
|
||||
- Start instance when original node unavailable
|
||||
- Start instance when original node at capacity
|
||||
|
||||
**E2E Tests:**
|
||||
- Full instance lifecycle across multiple nodes
|
||||
- Recovery from simulated hardware failure
|
||||
|
||||
## Technical Notes
|
||||
|
||||
### Predastore Replication
|
||||
|
||||
All instance data is stored in Predastore, which uses Reed-Solomon erasure coding (e.g., RS(3,2) = 3 data shards, 2 parity shards). Objects are distributed across nodes via consistent hashing. If a data node is unavailable, parity shards can reconstruct the data. See [predastore/DESIGN.md](https://github.com/mulgadc/predastore/blob/main/DESIGN.md).
|
||||
|
||||
### NATS Topics
|
||||
|
||||
- `ec2.cmd.$instance` - Commands for running instances (daemon subscribes)
|
||||
- `ec2.start` - StartInstance requests (queue group, any available daemon)
|
||||
- `ec2.recovery.$node` - WAL sync requests for specific node
|
||||
@@ -1,125 +0,0 @@
|
||||
# EC2 AttachVolume
|
||||
|
||||
Hot-plug an EBS volume to a running instance via QMP, implementing the AWS `ec2 attach-volume` API.
|
||||
|
||||
## Request Flow
|
||||
|
||||
```
|
||||
AWS SDK (attach-volume)
|
||||
-> Gateway (port 9999): parse AWS query params, validate input
|
||||
-> NATS ec2.cmd.{instanceId}: per-instance topic routing
|
||||
-> Daemon handleEC2Events: owns the instance's QEMU process
|
||||
1. Validate volume is "available" (Predastore/S3 lookup)
|
||||
2. ebs.mount via NATS -> viperblock starts NBD server
|
||||
3. QMP blockdev-add -> register NBD block device in QEMU
|
||||
4. QMP device_add -> attach as virtio-blk-pci to guest
|
||||
5. Update instance metadata + volume metadata
|
||||
<- NATS response: ec2.VolumeAttachment JSON
|
||||
<- Gateway: wrap in XML
|
||||
<- AWS SDK: VolumeAttachment response
|
||||
```
|
||||
|
||||
## Why Per-Instance NATS Routing
|
||||
|
||||
AttachVolume uses `ec2.cmd.{instanceId}` instead of a generic `ec2.AttachVolume` topic. This is because the command must reach the specific daemon process that owns the QEMU instance. Each daemon subscribes to `ec2.cmd.{instanceId}` for every instance it manages. The same pattern is used by Start, Stop, and Terminate. If no daemon owns the instance, NATS returns `ErrNoResponders`, which the gateway maps to `InvalidInstanceID.NotFound`.
|
||||
|
||||
Other volume operations (CreateVolume, DeleteVolume, DescribeVolumes) use generic topics like `ec2.CreateVolume` because they don't require access to a specific QEMU process.
|
||||
|
||||
## Files Changed
|
||||
|
||||
### Wire Format: `hive/qmp/qmp.go`
|
||||
|
||||
Added `AttachVolume bool` to `Attributes` and a new `AttachVolumeData` struct carried as an optional pointer on `Command`. The pointer-with-omitempty pattern means the field is absent from JSON for non-attach commands, keeping the wire format clean. The `Attributes` struct uses boolean flags for dispatch (matching the existing Start/Stop/Terminate pattern).
|
||||
|
||||
### Gateway Handler: `hive/gateway/ec2/volume/AttachVolume.go`
|
||||
|
||||
Validates `VolumeId` and `InstanceId` are present, constructs a `qmp.Command` with `AttachVolume: true` and `AttachVolumeData`, sends it to `ec2.cmd.{instanceId}` via NATS request-response (30s timeout). The `Device` field is optional per the AWS API spec.
|
||||
|
||||
NATS errors are differentiated: `ErrNoResponders` returns `InvalidInstanceID.NotFound` (no daemon owns this instance), while timeouts and connection errors return `InternalError` to avoid misleading users about non-existent instances when the real problem is infrastructure.
|
||||
|
||||
### Gateway Switch: `hive/gateway/ec2.go`
|
||||
|
||||
Added `"AttachVolume"` case to the EC2 action switch statement.
|
||||
|
||||
### Daemon Handler: `hive/daemon/daemon_handlers.go`
|
||||
|
||||
The core logic, placed as the first branch in `handleEC2Events` (before Start/Stop/Terminate). Implements a three-phase commit with reverse-order rollback:
|
||||
|
||||
**Phase 1 - EBS Mount:** Sends `ebs.mount` via NATS to viperblock, which starts an NBD server and returns a URI (unix socket or TCP). If this fails, no cleanup is needed since nothing was mounted.
|
||||
|
||||
**Phase 2 - QMP blockdev-add:** Registers the NBD block device in QEMU. Uses `nbd-{volumeId}` as the node-name for predictable identification during future detach. The `export` field is set to `""` per viperblock convention (default export). If this fails, Phase 1 is rolled back via `ebs.unmount`.
|
||||
|
||||
**Phase 3 - QMP device_add:** Attaches the block device to the guest as `virtio-blk-pci` with id `vdisk-{volumeId}`. If this fails, Phase 2 is rolled back via `blockdev-del`, then Phase 1 via `ebs.unmount`. The `blockdev-del` rollback error is checked: if it fails, the EBS unmount is skipped to avoid tearing down storage that QEMU still references (which could crash the VM).
|
||||
|
||||
After all three phases succeed, the handler updates in-memory state (`EBSRequests`, `BlockDeviceMappings`), persists to Predastore (volume metadata) and JetStream (instance state), then responds with `ec2.VolumeAttachment` JSON.
|
||||
|
||||
### Volume Service: `hive/handlers/ec2/volume/service_impl.go`
|
||||
|
||||
`getVolumeConfig` was renamed to `GetVolumeConfig` (exported directly instead of using a wrapper) so the daemon can validate that a volume exists and check its state before attempting attachment.
|
||||
|
||||
`UpdateVolumeState` was extended with a `deviceName` parameter rather than creating a separate `UpdateVolumeAttachment` method. The existing callers pass `""` for deviceName since boot volumes don't track device names this way. This avoids near-duplicate methods.
|
||||
|
||||
### Volume Service Interface: `hive/handlers/ec2/volume/service.go`
|
||||
|
||||
The `VolumeService` interface is kept to AWS-facing operations only (CreateVolume, DescribeVolumes, ModifyVolume, DeleteVolume). `GetVolumeConfig` and `UpdateVolumeState` are methods on the concrete `VolumeServiceImpl` type, not the interface, because they are internal operations used by the daemon rather than AWS API endpoints. The daemon holds a `*VolumeServiceImpl` directly so it can call these methods without polluting the interface that the gateway-side `NATSVolumeService` also implements.
|
||||
|
||||
### Device Name Assignment: `hive/daemon/daemon.go`
|
||||
|
||||
`nextAvailableDevice` scans both `EBSRequests` and `BlockDeviceMappings` to find the next available `/dev/sd[f-p]` slot. Both collections are checked because there's a window during attachment where a volume appears in `EBSRequests` but not yet in `BlockDeviceMappings`. Returns `""` when all 11 slots are exhausted, which the handler maps to `AttachmentLimitExceeded`.
|
||||
|
||||
The `/dev/sd[f-p]` range follows the AWS convention. The guest OS sees these as `/dev/vd*` when using virtio drivers, but the metadata device name is the `/dev/sd*` form per AWS API compatibility.
|
||||
|
||||
### Rollback Helper: `hive/daemon/daemon.go`
|
||||
|
||||
`rollbackEBSMount` sends `ebs.unmount` via NATS and verifies the response: checks both the `Error` field and the `Mounted` boolean. Rollback failures are logged at Error level but not propagated to callers since rollback is best-effort cleanup during an already-failed operation. This is a deliberate tradeoff: propagating rollback errors would complicate the handler without changing the user-visible outcome (the attach already failed).
|
||||
|
||||
### Config: `hive/config/config.go`
|
||||
|
||||
Added `DeviceName` field to `EBSRequest`. This tracks which `/dev/sd*` device name is assigned to a hot-plugged volume. Boot-time volumes don't use this field since their device assignment is handled by QEMU command-line args.
|
||||
|
||||
### NBD URI Parsing: `hive/utils/utils.go`
|
||||
|
||||
`ParseNBDURI` is the inverse of the existing `FormatNBDSocketURI`/`FormatNBDTCPURI` functions. It extracts the components needed for QMP's `blockdev-add` server argument:
|
||||
- `nbd:unix:/path/to/socket.sock` -> `type=unix, path=/path/to/socket.sock`
|
||||
- `nbd://host:port` -> `type=inet, host=host, port=port`
|
||||
|
||||
## Design Decisions
|
||||
|
||||
**Why QMP hot-plug instead of restarting the VM?** Hot-plug via `blockdev-add` + `device_add` attaches the volume without downtime. This matches AWS behavior where `attach-volume` works on running instances immediately.
|
||||
|
||||
**Why `nbd-{volumeId}` and `vdisk-{volumeId}` naming?** These are predictable identifiers derived from the volume ID. This makes future `detach-volume` straightforward: the daemon can construct the node-name and device-id from the volume ID without maintaining a separate mapping.
|
||||
|
||||
**Why `export: ""`?** Viperblock's NBD server uses a blank export name as the default. This is a viperblock convention, not a universal NBD convention.
|
||||
|
||||
**Why rollback skips EBS unmount when blockdev-del fails?** If `blockdev-del` fails, QEMU still has a reference to the NBD backend. Unmounting the EBS volume would tear down the NBD server while QEMU still tries to do I/O through it, potentially crashing or hanging the VM. Leaving the mount in place is the safer failure mode: the volume is leaked but the VM stays healthy.
|
||||
|
||||
**Why `DeleteOnTermination` defaults to false?** This matches AWS behavior: hot-attached volumes default to `DeleteOnTermination: false`. Boot volumes set this to true. Users can change it via `modify-instance-attribute`.
|
||||
|
||||
## Error Codes
|
||||
|
||||
| Condition | AWS Error Code |
|
||||
|-----------|---------------|
|
||||
| Missing VolumeId/InstanceId | `InvalidParameterValue` |
|
||||
| Volume not found in Predastore | `InvalidVolume.NotFound` |
|
||||
| Volume already in-use | `VolumeInUse` |
|
||||
| Instance not found (no NATS responder) | `InvalidInstanceID.NotFound` |
|
||||
| Instance not running | `IncorrectInstanceState` |
|
||||
| All device slots full | `AttachmentLimitExceeded` |
|
||||
| Internal failures (mount, QMP, etc.) | `InternalError` |
|
||||
|
||||
## Testing
|
||||
|
||||
**Gateway validation** (`AttachVolume_test.go`): Table-driven tests for nil input, empty input, missing fields, empty strings, valid inputs with/without device. Follows the pattern from `CreateVolume_test.go`.
|
||||
|
||||
**Device name assignment** (`daemon_test.go`): 6 scenarios covering empty instance, existing block device mappings, existing EBS requests, all devices used, nil instance, mixed sources.
|
||||
|
||||
**Handler error paths** (`daemon_test.go`): Tests for missing attach volume data, instance not running, and volume not found. Uses a real embedded NATS server (not mocks). The happy path is not currently testable because `volumeService` is a concrete type — testing the full flow would require introducing interface-based dependency injection for the volume service and QMP client.
|
||||
|
||||
**NBD URI parsing** (`utils_test.go`): 8 cases covering both formats, empty paths, missing ports, invalid ports, unsupported formats.
|
||||
|
||||
## Known Limitations
|
||||
|
||||
- No validation of user-supplied device names against the `/dev/sd[f-p]` range or conflict checking.
|
||||
- TOCTOU gap between checking instance status/volume state and performing the actual QMP operations. A concurrent stop could race with an attach.
|
||||
- The `Attributes` struct uses boolean flags for mutually exclusive operations. A string-based action field would be structurally safer but is a breaking wire-format change.
|
||||
- Volume metadata and JetStream state persistence failures after a successful attach are logged but do not cause the response to indicate failure, since the volume is already physically attached to the VM.
|
||||
@@ -1,114 +0,0 @@
|
||||
# EC2 DescribeVolumeStatus
|
||||
|
||||
Report the health and status of EBS volumes, implementing the AWS `ec2 describe-volume-status` API. Returns IO-enabled status and volume health for all or specific volumes.
|
||||
|
||||
## Request Flow
|
||||
|
||||
```
|
||||
AWS SDK (describe-volume-status)
|
||||
-> Gateway (port 9999): parse AWS query params, validate vol- prefix
|
||||
-> NATS ec2.DescribeVolumeStatus: generic topic (no instance affinity)
|
||||
-> Daemon DescribeVolumeStatus handler
|
||||
Fast path (VolumeIds provided):
|
||||
1. Parallel GetVolumeConfig for each volume ID
|
||||
2. Build VolumeStatusItem per volume
|
||||
Slow path (no VolumeIds):
|
||||
1. S3 ListObjects to find vol-* prefixes
|
||||
2. Skip internal sub-volumes (-efi, -cloudinit)
|
||||
3. GetVolumeConfig + build VolumeStatusItem per volume
|
||||
<- NATS response: DescribeVolumeStatusOutput JSON
|
||||
<- Gateway: wrap in XML as DescribeVolumeStatusResponse
|
||||
<- AWS SDK: DescribeVolumeStatusOutput
|
||||
```
|
||||
|
||||
## Files Created
|
||||
|
||||
### Gateway Handler: `hive/gateway/ec2/volume/DescribeVolumeStatus.go`
|
||||
|
||||
`ValidateDescribeVolumeStatusInput` validates the vol- prefix on all VolumeIds. Follows the same pattern as `ValidateDescribeVolumesInput` — nil input is valid (means "all volumes"), nil entries in the VolumeIds slice are skipped, any non-`vol-` prefixed ID returns `InvalidVolume.Malformed`.
|
||||
|
||||
`DescribeVolumeStatus` validates input, creates a `NATSVolumeService`, and delegates to `volumeService.DescribeVolumeStatus(input)`.
|
||||
|
||||
### Gateway Tests: `hive/gateway/ec2/volume/DescribeVolumeStatus_test.go`
|
||||
|
||||
9 table-driven test cases mirroring `DescribeVolumes_test.go`: nil input, empty input, valid single ID, valid multiple IDs, no prefix, wrong prefix, empty string, mixed valid/invalid, nil in list.
|
||||
|
||||
## Files Modified
|
||||
|
||||
### Gateway Switch: `hive/gateway/ec2.go`
|
||||
|
||||
Added `"DescribeVolumeStatus"` case to the EC2 action switch. Parses `ec2.DescribeVolumeStatusInput`, calls the gateway handler, wraps result as `"DescribeVolumeStatusResponse"` XML.
|
||||
|
||||
### Volume Service Interface: `hive/handlers/ec2/volume/service.go`
|
||||
|
||||
Added `DescribeVolumeStatus(input *ec2.DescribeVolumeStatusInput) (*ec2.DescribeVolumeStatusOutput, error)` to the `VolumeService` interface. This is an AWS-facing operation so it belongs on the interface (unlike internal helpers like `GetVolumeConfig`).
|
||||
|
||||
### NATS Client: `hive/handlers/ec2/volume/service_nats.go`
|
||||
|
||||
Added `DescribeVolumeStatus` method following the same marshal-request-validate-unmarshal pattern as `DescribeVolumes`. Sends to `ec2.DescribeVolumeStatus` topic with 30s timeout.
|
||||
|
||||
### Service Implementation: `hive/handlers/ec2/volume/service_impl.go`
|
||||
|
||||
Three new methods:
|
||||
|
||||
**`DescribeVolumeStatus`** — Entry point. Defaults nil input to empty. Takes the fast path (parallel by-ID lookup) when VolumeIds are provided, or the slow path (S3 ListObjects with vol-* filtering) when no IDs are specified. Same dual-path pattern as `DescribeVolumes`.
|
||||
|
||||
**`fetchVolumeStatusByIDs`** — Parallel goroutine fan-out with mutex-protected slice append, matching the `fetchVolumesByIDs` pattern. Volumes that fail to load are logged and skipped.
|
||||
|
||||
**`getVolumeStatusByID`** — Reads volume config via the existing `GetVolumeConfig` and builds an `ec2.VolumeStatusItem` with:
|
||||
- `VolumeId` and `AvailabilityZone` from volume metadata
|
||||
- `VolumeStatus`: status `"ok"` with two details entries
|
||||
- `Actions`: empty slice
|
||||
- `Events`: empty slice
|
||||
|
||||
### Service Implementation Tests: `hive/handlers/ec2/volume/service_impl_test.go`
|
||||
|
||||
Added `TestDescribeVolumeStatus_NilInputDefaults` — verifies that nil input is correctly defaulted to an empty input (the nil guard runs) before reaching the S3 layer.
|
||||
|
||||
## Volume Status Details
|
||||
|
||||
Every volume currently returns the same static status:
|
||||
|
||||
| Detail | Value | Reason |
|
||||
|--------|-------|--------|
|
||||
| `io-enabled` | `passed` | Viperblock doesn't have degraded-IO states |
|
||||
| `io-performance` | `not-applicable` | Viperblock doesn't track this metric |
|
||||
| Status | `ok` | No health degradation model yet |
|
||||
| Actions | empty | No pending actions |
|
||||
| Events | empty | No scheduled events |
|
||||
|
||||
This matches AWS behavior for healthy volumes. When viperblock adds health monitoring, these values can be derived from actual backend state.
|
||||
|
||||
## Error Codes
|
||||
|
||||
| Condition | AWS Error Code |
|
||||
|-----------|---------------|
|
||||
| Invalid volume ID format (no vol- prefix) | `InvalidVolume.Malformed` |
|
||||
| S3 listing failure | `InternalError` |
|
||||
| Volume not found | Silently skipped (matching DescribeVolumes behavior) |
|
||||
|
||||
## Design Decisions
|
||||
|
||||
**Why reuse the DescribeVolumes patterns?** DescribeVolumeStatus has the same data access pattern: optionally filter by volume IDs, otherwise list all. Reusing `GetVolumeConfig`, the parallel fan-out helper, and the S3 listing/filtering logic keeps the implementation consistent and avoids duplication.
|
||||
|
||||
**Why static status values?** Viperblock currently has no concept of degraded IO or volume health checks. Returning `"ok"` / `"passed"` is accurate — if a volume's config is readable from S3, it is healthy. When viperblock adds health monitoring, `getVolumeStatusByID` is the single point to update.
|
||||
|
||||
**Why empty Actions and Events?** AWS uses these for scheduled maintenance events (e.g., `enable-volume-io`, `retire-volume`). Hive doesn't have scheduled maintenance, so these are always empty.
|
||||
|
||||
**Why no Filters/MaxResults/NextToken?** Consistent with the existing DescribeVolumes implementation. These can be added in a future iteration across all Describe* endpoints.
|
||||
|
||||
## Testing
|
||||
|
||||
```sh
|
||||
# All volumes
|
||||
aws --endpoint-url https://localhost:9999 ec2 describe-volume-status
|
||||
|
||||
# Specific volumes
|
||||
aws --endpoint-url https://localhost:9999 ec2 describe-volume-status --volume-ids vol-abc123
|
||||
```
|
||||
|
||||
## Known Limitations
|
||||
|
||||
- Filters, MaxResults, and NextToken are accepted but ignored.
|
||||
- Volume status is always `"ok"` regardless of actual backend health.
|
||||
- The slow path (list all volumes) is sequential per-volume after the S3 list call, unlike the fast path which is parallel.
|
||||
@@ -1,99 +0,0 @@
|
||||
# EC2 DetachVolume
|
||||
|
||||
Hot-unplug an EBS volume from a running instance via QMP, implementing the AWS `ec2 detach-volume` API. This is the inverse of `attach-volume`.
|
||||
|
||||
## Request Flow
|
||||
|
||||
```
|
||||
AWS SDK (detach-volume)
|
||||
-> Gateway (port 9999): parse AWS query params, validate input
|
||||
-> NATS ec2.cmd.{instanceId}: per-instance topic routing
|
||||
-> Daemon handleEC2Events: owns the instance's QEMU process
|
||||
1. Validate volume is attached to instance
|
||||
2. Reject boot/EFI/CloudInit volumes
|
||||
3. QMP device_del -> remove virtio-blk-pci from guest
|
||||
4. QMP blockdev-del -> remove NBD block device from QEMU
|
||||
5. ebs.unmount via NATS -> viperblock stops NBD server
|
||||
6. Update instance metadata + volume metadata
|
||||
<- NATS response: ec2.VolumeAttachment JSON
|
||||
<- Gateway: wrap in XML
|
||||
<- AWS SDK: VolumeAttachment response
|
||||
```
|
||||
|
||||
## Instance ID Resolution
|
||||
|
||||
Unlike AttachVolume, DetachVolume does not require an InstanceId parameter (per the AWS API spec). If InstanceId is omitted, the gateway resolves it by calling DescribeVolumes via NATS to look up the volume's current attachment. If the volume has no attachments, it returns `IncorrectState`.
|
||||
|
||||
## Files Changed
|
||||
|
||||
### Wire Format: `hive/qmp/qmp.go`
|
||||
|
||||
Added `DetachVolume bool` to `Attributes` and a new `DetachVolumeData` struct with `VolumeID`, `Device` (optional), and `Force` (optional). Carried as a pointer with omitempty on `Command`, matching the AttachVolumeData pattern.
|
||||
|
||||
### Gateway Handler: `hive/gateway/ec2/volume/DetachVolume.go`
|
||||
|
||||
Validates `VolumeId` is present (InstanceId, Device, Force are optional). If InstanceId is not provided, resolves it via `NewNATSVolumeService.DescribeVolumes()`. Constructs `qmp.Command` with `DetachVolume: true` and sends to `ec2.cmd.{instanceId}`.
|
||||
|
||||
### Gateway Switch: `hive/gateway/ec2.go`
|
||||
|
||||
Added `"DetachVolume"` case to the EC2 action switch statement.
|
||||
|
||||
### Daemon Handler: `hive/daemon/daemon_handlers.go`
|
||||
|
||||
The core logic, placed after AttachVolume in `handleEC2Events`. Implements a three-phase hot-unplug (reverse order of attach):
|
||||
|
||||
**Validation:**
|
||||
- Volume must be in instance's `EBSRequests`
|
||||
- Boot, EFI, and CloudInit volumes cannot be detached
|
||||
- Optional device name cross-check against stored device name
|
||||
|
||||
**Phase 1 - QMP device_del:** Removes the virtio-blk-pci device from the guest. Uses `vdisk-{volumeId}` as the device ID (set during attach). If this fails and Force is false, the operation aborts. With Force=true, a warning is logged and execution continues.
|
||||
|
||||
**Phase 2 - QMP blockdev-del:** Removes the NBD block device node from QEMU. Uses `nbd-{volumeId}` as the node-name. If this fails, Phase 3 is skipped because the block node is still referenced by QEMU — unmounting would crash the VM.
|
||||
|
||||
**Phase 3 - ebs.unmount:** Sends `ebs.unmount` via NATS to viperblock to stop the NBD server. Only executed if Phase 2 succeeded.
|
||||
|
||||
**State cleanup (always runs after QMP phases):**
|
||||
- Remove volume from `instance.EBSRequests.Requests`
|
||||
- Remove matching entry from `instance.Instance.BlockDeviceMappings`
|
||||
- `UpdateVolumeState(volumeID, "available", "", "")` — clears attachment in S3
|
||||
- `WriteState()` — persist instance to JetStream
|
||||
|
||||
**Response:** `ec2.VolumeAttachment` with `State: "detaching"`
|
||||
|
||||
## Error Codes
|
||||
|
||||
| Condition | AWS Error Code |
|
||||
|-----------|---------------|
|
||||
| Missing VolumeId | `InvalidParameterValue` |
|
||||
| Volume not found (gateway lookup) | `InvalidVolume.NotFound` |
|
||||
| Volume not attached | `IncorrectState` |
|
||||
| Instance not found (no NATS responder) | `InvalidInstanceID.NotFound` |
|
||||
| Instance not running | `IncorrectInstanceState` |
|
||||
| Boot/EFI/CloudInit volume | `OperationNotPermitted` |
|
||||
| Device mismatch | `InvalidParameterValue` |
|
||||
| Internal failures | `ServerInternal` |
|
||||
|
||||
## Design Decisions
|
||||
|
||||
**Why three phases in reverse order?** The attach order is mount -> blockdev-add -> device_add. Detach reverses this: device_del -> blockdev-del -> unmount. The guest device must be removed before the block node, and the block node before the NBD server, to avoid I/O errors or VM crashes.
|
||||
|
||||
**Why skip unmount when blockdev-del fails?** If QEMU still references the block node, tearing down the NBD server would cause I/O errors in the VM. The volume is leaked but the VM stays healthy. This matches the same safety rationale used in attach rollback.
|
||||
|
||||
**Why the 1-second sleep between device_del and blockdev-del?** The guest OS needs time to acknowledge the PCI device removal. Without this pause, blockdev-del may fail because the device still has a reference to the block node.
|
||||
|
||||
**Why allow Force=true to continue past device_del failure?** This matches AWS `--force` behavior for stuck volumes. The device may already be partially removed or the guest may be unresponsive.
|
||||
|
||||
**Why state cleanup runs even when QMP phases partially fail?** The metadata should reflect the detach intent. If the QMP commands partially succeeded, the volume is in an inconsistent state from QEMU's perspective but the metadata should be cleaned up so the volume can be re-attached or managed.
|
||||
|
||||
## Testing
|
||||
|
||||
**Gateway validation** (`DetachVolume_test.go`): Table-driven tests for nil input, empty input, missing VolumeId, empty VolumeId, valid inputs with various optional field combinations.
|
||||
|
||||
**Daemon handler tests** (`daemon_test.go`): Tests for missing DetachVolumeData, instance not running, volume not attached, and boot volume protection.
|
||||
|
||||
## Known Limitations
|
||||
|
||||
- TOCTOU gap between finding the volume in EBSRequests and performing QMP operations.
|
||||
- No timeout on the 1-second sleep between device_del and blockdev-del.
|
||||
- Force mode continues past device_del failure but does not retry.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,194 +0,0 @@
|
||||
# Egress-Only Internet Gateway (EIGW)
|
||||
|
||||
## Overview
|
||||
|
||||
Egress-Only Internet Gateways provide outbound-only IPv6 connectivity for instances in a VPC. In Hive, EIGWs are lightweight metadata objects persisted in NATS JetStream KV. The implementation covers the three standard EC2 API operations: Create, Delete, and Describe.
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
### 1. NATS JetStream KV for persistence
|
||||
|
||||
EIGW records are stored in a dedicated KV bucket (`hive-eigw`) with 10-revision history. This avoids introducing a database dependency -- the same NATS infrastructure used for messaging also handles persistence.
|
||||
|
||||
The `getOrCreateKVBucket` helper tries `CreateKeyValue` first, then falls back to `KeyValue` (open existing). This makes the service idempotent across restarts.
|
||||
|
||||
**Trade-off**: KV is not suited for complex queries (e.g. filtering by tag values). Describe currently iterates all keys and filters in-memory, which is fine for typical EIGW counts but wouldn't scale to thousands of records.
|
||||
|
||||
### 2. Constructor fails loudly, daemon falls back gracefully
|
||||
|
||||
`NewEgressOnlyIGWServiceImplWithNATS` returns an error if the KV bucket can't be created or opened. This ensures the service never operates with a nil KV handle, eliminating an entire class of nil-pointer bugs.
|
||||
|
||||
The daemon handles this with a graceful fallback:
|
||||
|
||||
```go
|
||||
if eigwSvc, err := handlers_ec2_eigw.NewEgressOnlyIGWServiceImplWithNATS(d.config, d.natsConn); err != nil {
|
||||
slog.Warn("Failed to initialize EIGW service, falling back to in-memory", "error", err)
|
||||
d.eigwService = handlers_ec2_eigw.NewEgressOnlyIGWServiceImpl(d.config)
|
||||
} else {
|
||||
d.eigwService = eigwSvc
|
||||
}
|
||||
```
|
||||
|
||||
**Trade-off**: The in-memory fallback (`NewEgressOnlyIGWServiceImpl`) has no KV backing, so EIGW state won't survive daemon restarts. This is acceptable for degraded mode -- the alternative is refusing to start the daemon entirely.
|
||||
|
||||
### 3. crypto/rand for ID generation
|
||||
|
||||
EIGW IDs (`eigw-` + 16 hex chars) are generated using `crypto/rand` + `encoding/hex`. This matches the pattern used by instance and volume ID generation elsewhere in the codebase.
|
||||
|
||||
```go
|
||||
func generateEgressOnlyIGWID() string {
|
||||
b := make([]byte, 8)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
panic("crypto/rand failed: " + err.Error())
|
||||
}
|
||||
return "eigw-" + hex.EncodeToString(b)
|
||||
}
|
||||
```
|
||||
|
||||
The panic on `crypto/rand` failure is intentional -- if the OS entropy source is broken, the system has larger problems.
|
||||
|
||||
### 4. Error strings are awserrors constants
|
||||
|
||||
All errors returned from `service_impl.go` use `errors.New(awserrors.ErrorXxx)` with exact constant values. This is critical because the error string travels through a round-trip:
|
||||
|
||||
```
|
||||
service_impl: errors.New(awserrors.ErrorMissingParameter)
|
||||
-> daemon handleNATSRequest: utils.GenerateErrorPayload(err.Error())
|
||||
-> JSON over NATS: {"Code": "MissingParameter"}
|
||||
-> gateway NATSRequest[Out]: ValidateErrorPayload extracts Code
|
||||
-> gateway ErrorHandler: awserrors.ErrorLookup[err.Error()] -> HTTP status + XML
|
||||
```
|
||||
|
||||
If the error string doesn't match a key in `ErrorLookup`, the gateway returns a generic 500. Using `fmt.Errorf` with extra context would break this chain.
|
||||
|
||||
### 5. Centralized nil NATS check in the gateway
|
||||
|
||||
Rather than checking `natsConn == nil` in each gateway function, a single check in `EC2_Request` covers all NATS-dependent actions. An allowlist (`ec2LocalActions`) exempts actions that work without NATS:
|
||||
|
||||
```go
|
||||
var ec2LocalActions = map[string]bool{
|
||||
"DescribeRegions": true,
|
||||
"DescribeAvailabilityZones": true,
|
||||
"DescribeAccountAttributes": true,
|
||||
}
|
||||
```
|
||||
|
||||
If `gw.NATSConn` is nil and the action isn't in the allowlist, the gateway returns `ErrorServerInternal` immediately.
|
||||
|
||||
### 6. Tag filtering by resource type
|
||||
|
||||
Create accepts `TagSpecifications` but only applies tags where `ResourceType` equals `"egress-only-internet-gateway"`. Tags with other resource types (e.g. `"instance"`) are silently ignored, matching AWS behavior.
|
||||
|
||||
## Request Flow
|
||||
|
||||
```
|
||||
AWS SDK (create-egress-only-internet-gateway --vpc-id vpc-xxx)
|
||||
-> Gateway (port 9999): ec2Handler parses AWS query params into SDK struct
|
||||
-> Validate input (nil check, required fields)
|
||||
-> NATS ec2.CreateEgressOnlyInternetGateway: routed to daemon via "hive-workers" queue group
|
||||
-> Daemon handleEC2CreateEgressOnlyInternetGateway
|
||||
-> handleNATSRequest: unmarshal -> EgressOnlyIGWServiceImpl.Create -> marshal -> respond
|
||||
1. Validate VpcId is present
|
||||
2. Generate eigw-{16hex} ID via crypto/rand
|
||||
3. Build EgressOnlyIGWRecord with state "attached"
|
||||
4. Process matching tag specifications
|
||||
5. JSON marshal and store in NATS KV (hive-eigw bucket)
|
||||
<- NATS response: ec2.CreateEgressOnlyInternetGatewayOutput JSON
|
||||
<- Gateway: wrap in XML (CreateEgressOnlyInternetGatewayResponse)
|
||||
<- AWS SDK: response
|
||||
```
|
||||
|
||||
## NATS Topics
|
||||
|
||||
| Topic | Direction | Purpose |
|
||||
|-------|-----------|---------|
|
||||
| `ec2.CreateEgressOnlyInternetGateway` | Gateway -> Daemon | Create new EIGW attached to a VPC |
|
||||
| `ec2.DeleteEgressOnlyInternetGateway` | Gateway -> Daemon | Delete EIGW by ID |
|
||||
| `ec2.DescribeEgressOnlyInternetGateways` | Gateway -> Daemon | List/filter EIGWs |
|
||||
|
||||
All topics use the `hive-workers` queue group for load balancing across daemon instances.
|
||||
|
||||
## Storage Layout
|
||||
|
||||
```
|
||||
NATS JetStream KV bucket: hive-eigw (history: 10)
|
||||
|
||||
Key: eigw-a1b2c3d4e5f6g7h8
|
||||
Value: {
|
||||
"egress_only_internet_gateway_id": "eigw-a1b2c3d4e5f6g7h8",
|
||||
"vpc_id": "vpc-abc123",
|
||||
"state": "attached",
|
||||
"tags": {"Name": "my-eigw", "Env": "prod"},
|
||||
"created_at": "2026-02-10T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
## Files Changed
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `hive/handlers/ec2/eigw/service.go` | `EgressOnlyIGWService` interface (3 methods) |
|
||||
| `hive/handlers/ec2/eigw/service_impl.go` | Direct implementation with NATS JetStream KV persistence |
|
||||
| `hive/handlers/ec2/eigw/service_impl_test.go` | 12 test cases against embedded NATS server |
|
||||
| `hive/handlers/ec2/eigw/service_nats.go` | NATS proxy (gateway -> daemon via `utils.NATSRequest`) |
|
||||
| `hive/gateway/ec2/eigw/CreateEgressOnlyInternetGateway.go` | Gateway validation + handler for Create |
|
||||
| `hive/gateway/ec2/eigw/DeleteEgressOnlyInternetGateway.go` | Gateway validation + handler for Delete |
|
||||
| `hive/gateway/ec2/eigw/DescribeEgressOnlyInternetGateways.go` | Gateway validation + handler for Describe |
|
||||
| `hive/gateway/ec2/eigw/eigw_test.go` | Gateway validation tests (9 cases) |
|
||||
| `hive/gateway/ec2.go` | 3 entries in `ec2Actions` map; centralized nil NATS check |
|
||||
| `hive/daemon/daemon.go` | `eigwService` field, 3 NATS subscriptions, service initialization with fallback |
|
||||
| `hive/daemon/daemon_handlers.go` | 3 handler functions delegating to `handleNATSRequest` |
|
||||
| `hive/awserrors/awserrors.go` | `ErrorInvalidEgressOnlyInternetGatewayId.Malformed` and `.NotFound` constants + lookup entries |
|
||||
|
||||
## Test Coverage
|
||||
|
||||
### Service Implementation Tests (`service_impl_test.go`)
|
||||
|
||||
| Test | What it validates |
|
||||
|------|-------------------|
|
||||
| `TestCreateEgressOnlyInternetGateway` | Basic create: ID format, VPC attachment, state |
|
||||
| `TestCreateEgressOnlyInternetGateway_MissingVpcId` | Rejects nil VpcId with MissingParameter |
|
||||
| `TestCreateEgressOnlyInternetGateway_EmptyVpcId` | Rejects empty string VpcId |
|
||||
| `TestCreateEgressOnlyInternetGateway_WithTags` | Tags applied and persisted through describe round-trip |
|
||||
| `TestCreateEgressOnlyInternetGateway_TagsWrongResourceType` | Tags with non-matching resource type are ignored |
|
||||
| `TestDeleteEgressOnlyInternetGateway` | Delete succeeds, describe confirms removal |
|
||||
| `TestDeleteEgressOnlyInternetGateway_NotFound` | Returns NotFound for nonexistent ID |
|
||||
| `TestDeleteEgressOnlyInternetGateway_MissingID` | Rejects nil ID with MissingParameter |
|
||||
| `TestDeleteEgressOnlyInternetGateway_EmptyID` | Rejects empty string ID |
|
||||
| `TestDescribeEgressOnlyInternetGateways_All` | Lists all EIGWs when no filter specified |
|
||||
| `TestDescribeEgressOnlyInternetGateways_ByID` | Filters by specific EIGW ID |
|
||||
| `TestDescribeEgressOnlyInternetGateways_Empty` | Returns empty list for empty bucket |
|
||||
|
||||
All tests run against an embedded NATS server with JetStream enabled.
|
||||
|
||||
### Gateway Validation Tests (`eigw_test.go`)
|
||||
|
||||
| Test | What it validates |
|
||||
|------|-------------------|
|
||||
| `ValidateCreateEgressOnlyInternetGatewayInput` | nil input, missing VpcId, empty VpcId, valid input |
|
||||
| `ValidateDeleteEgressOnlyInternetGatewayInput` | nil input, missing ID, empty ID, valid input |
|
||||
| `ValidateDescribeEgressOnlyInternetGatewaysInput` | nil input, empty input, with filter IDs |
|
||||
|
||||
## AWS CLI Usage
|
||||
|
||||
```bash
|
||||
# Create
|
||||
aws ec2 create-egress-only-internet-gateway \
|
||||
--vpc-id vpc-abc123 \
|
||||
--tag-specifications 'ResourceType=egress-only-internet-gateway,Tags=[{Key=Name,Value=my-eigw}]' \
|
||||
--endpoint-url https://localhost:9999
|
||||
|
||||
# Describe all
|
||||
aws ec2 describe-egress-only-internet-gateways \
|
||||
--endpoint-url https://localhost:9999
|
||||
|
||||
# Describe by ID
|
||||
aws ec2 describe-egress-only-internet-gateways \
|
||||
--egress-only-internet-gateway-ids eigw-a1b2c3d4e5f6g7h8 \
|
||||
--endpoint-url https://localhost:9999
|
||||
|
||||
# Delete
|
||||
aws ec2 delete-egress-only-internet-gateway \
|
||||
--egress-only-internet-gateway-id eigw-a1b2c3d4e5f6g7h8 \
|
||||
--endpoint-url https://localhost:9999
|
||||
```
|
||||
@@ -1,613 +0,0 @@
|
||||
# Cluster Lifecycle: Heartbeats, Shutdown, Upgrades, Resilience, and VM Migration
|
||||
|
||||
## Phase 2 Roadmap
|
||||
|
||||
| Phase | Feature | Status | Description |
|
||||
|-------|---------|--------|-------------|
|
||||
| 2.1 | Heartbeats | **Complete** | Daemon heartbeats publish node status to KV every 10s |
|
||||
| 2.4 | Coordinated Shutdown | **Complete** | Phased cluster-wide shutdown via NATS |
|
||||
| 2.2 | Node Modes & Health Monitoring | **Planned** | Drain/maintenance/offline modes, stale heartbeat detection |
|
||||
| 2.5 | Coordinated Rolling Upgrade | **Planned** | Cluster-wide binary upgrade without full downtime |
|
||||
| 2.6 | Degraded Storage Writes | **Planned** | Predastore shard redirection when nodes are unavailable |
|
||||
| 2.7 | VM Migration | **Planned** | Relocate running VMs to other nodes during drain/maintenance |
|
||||
| 2.8 | [EC2 Health & Auto-Restart](ec2-health-restart.md) | **Planned** | QEMU crash detection, auto-restart, OOM protection, `DescribeInstanceStatus` API |
|
||||
|
||||
---
|
||||
|
||||
## Phase 2.1 — Heartbeats (Complete)
|
||||
|
||||
### Summary
|
||||
|
||||
Each daemon publishes its status to JetStream KV every 10 seconds — node name, epoch, running services, VM count, and resource allocation.
|
||||
|
||||
### Implementation
|
||||
|
||||
- `Heartbeat` struct in `jetstream.go` with node, epoch, services, VM count, resource allocation
|
||||
- `WriteHeartbeat`/`ReadHeartbeat` KV methods on `JetStreamManager`
|
||||
- `startHeartbeat()` goroutine in `heartbeat.go` — fires immediately, then every 10s
|
||||
- `buildHeartbeat()` reads from `ResourceManager` and `Instances`
|
||||
|
||||
### Files Modified
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `hive/daemon/jetstream.go` | Edit | `Heartbeat` struct + KV methods |
|
||||
| `hive/daemon/heartbeat.go` | New | `startHeartbeat()`, `buildHeartbeat()` goroutine |
|
||||
| `hive/daemon/daemon.go` | Edit | Call `startHeartbeat()` in `Start()` |
|
||||
| `hive/daemon/heartbeat_test.go` | New | Heartbeat tests (build, allocation, KV round-trip) |
|
||||
|
||||
---
|
||||
|
||||
## Phase 2.4 — Coordinated Cluster Shutdown (Complete)
|
||||
|
||||
### Summary
|
||||
|
||||
`hive admin cluster shutdown` drives a phased shutdown across all nodes via NATS, replacing the error-prone per-node `stop-dev.sh` approach for multi-node clusters.
|
||||
|
||||
### Design Decisions
|
||||
|
||||
- **Option B chosen**: Daemons handle stopping local services internally via `utils.StopProcess()` when they receive NATS shutdown phase messages. The CLI coordinator just sends messages and collects ACKs.
|
||||
- **Shutdown phases**: GATE → DRAIN → STORAGE → PERSIST → INFRA — ordered to prevent data loss
|
||||
- **INFRA phase is fire-and-forget**: NATS is going down, no ACK possible
|
||||
- **Scripts**: `stop-dev.sh` auto-detects multi-node and delegates to `hive admin cluster shutdown`; `HIVE_FORCE_LOCAL_STOP=1` overrides for per-service stop
|
||||
|
||||
### Implementation
|
||||
|
||||
- Five shutdown phase handlers in `shutdown.go`: GATE, DRAIN, STORAGE, PERSIST, INFRA
|
||||
- `ShutdownRequest`, `ShutdownACK`, `ShutdownProgress` message types
|
||||
- `shuttingDown atomic.Bool` on Daemon — set during GATE, checked by `setupShutdown()` to skip redundant VM stops
|
||||
- NATS subscriptions for `hive.cluster.shutdown.{phase}` (fan-out, no queue group)
|
||||
- Progress updates published during DRAIN phase
|
||||
- CLI coordinator in `cmd/hive/cmd/cluster.go` with `--force`, `--timeout`, `--dry-run` flags
|
||||
- `stop-dev.sh`: `is_multinode()` detection, delegates to coordinated shutdown (skipped when data-dir arg passed, e.g. E2E cleanup)
|
||||
- `start-dev.sh`: `is_multinode()` detection, cluster health poll at end
|
||||
|
||||
### Files Modified
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `hive/daemon/jetstream.go` | Edit | `ClusterShutdownState` struct + KV methods |
|
||||
| `hive/daemon/shutdown.go` | New | Shutdown types + 5 phase handlers |
|
||||
| `hive/daemon/daemon.go` | Edit | `shuttingDown atomic.Bool`, shutdown subscriptions, updated `setupShutdown()` |
|
||||
| `cmd/hive/cmd/admin.go` | Edit | `clusterCmd` + `clusterShutdownCmd` registration |
|
||||
| `cmd/hive/cmd/cluster.go` | New | `runClusterShutdown()` coordinator |
|
||||
| `scripts/stop-dev.sh` | Edit | Multi-node detection + coordinated shutdown delegation |
|
||||
| `scripts/start-dev.sh` | Edit | Multi-node detection + cluster health poll |
|
||||
| `hive/daemon/shutdown_test.go` | New | Shutdown tests (marshal, KV round-trip, flag behavior) |
|
||||
|
||||
### Testing
|
||||
|
||||
- `make preflight` — all checks pass
|
||||
- 9 new unit tests
|
||||
- `make test-docker-multi` — full multi-node E2E passes (2 consecutive runs confirmed)
|
||||
- `hive admin cluster shutdown --dry-run` prints phase plan
|
||||
- `stop-dev.sh` on multi-node delegates to coordinated shutdown
|
||||
- `HIVE_FORCE_LOCAL_STOP=1 stop-dev.sh` on multi-node uses per-service stop
|
||||
- `stop-dev.sh` on single-node uses per-service stop (unchanged)
|
||||
|
||||
---
|
||||
|
||||
## Phase 2.2 — Node Modes & Health Monitoring (Planned)
|
||||
|
||||
### Problem Statement
|
||||
|
||||
Currently, nodes are either "running" or "not responding." There is no concept of a node being in a transitional state — draining VMs, undergoing maintenance, or recovering from a crash. The health endpoint returns hardcoded `"running"`, and `hive get nodes` shows `"Ready"` or `"NotReady"` with no in-between.
|
||||
|
||||
This means:
|
||||
- No way to gracefully take a node offline for maintenance without a full cluster shutdown
|
||||
- No distinction between a planned offline and a crash — rejoining is the same either way
|
||||
- No stale heartbeat detection — if a node stops publishing heartbeats, nobody notices
|
||||
- The scheduler keeps routing work to draining nodes until they're fully dead
|
||||
|
||||
### Design: Node Modes
|
||||
|
||||
```
|
||||
┌─────────┐
|
||||
│ NORMAL │ ← default, accepts all work
|
||||
└────┬────┘
|
||||
drain cmd │ ▲ drain-cancel / rejoin
|
||||
▼ │
|
||||
┌─────────┐
|
||||
│DRAINING │ ← rejects new VMs, existing VMs run until migrated/stopped
|
||||
└────┬────┘
|
||||
all VMs │
|
||||
stopped │
|
||||
▼
|
||||
┌─────────┐
|
||||
│CORDONED │ ← no VMs, services still running, ready for maintenance
|
||||
└────┬────┘
|
||||
maintenance │ ▲ uncordon
|
||||
cmd ▼ │
|
||||
┌─────────┐
|
||||
│MAINTENANCE│ ← services may be stopped, node offline
|
||||
└─────────┘
|
||||
```
|
||||
|
||||
Add a `NodeMode` type to the Daemon struct (atomic for lock-free reads). Store in JetStream KV (`node.<name>.mode`) so it survives daemon restarts and is visible cluster-wide.
|
||||
|
||||
#### Mode Behaviors
|
||||
|
||||
| Mode | Accepts new VMs | Existing VMs | Services | Heartbeats |
|
||||
|------|----------------|--------------|----------|------------|
|
||||
| `normal` | Yes | Run | All running | Published |
|
||||
| `draining` | No | Run (being migrated/stopped) | All running | Published with mode |
|
||||
| `cordoned` | No | None | All running | Published with mode |
|
||||
| `maintenance` | No | None | May be stopped | Published until NATS stops |
|
||||
|
||||
#### CLI Commands
|
||||
|
||||
```bash
|
||||
hive node drain <node-name> # NORMAL → DRAINING (triggers VM migration/stop)
|
||||
hive node drain-cancel <node-name> # DRAINING → NORMAL (abort drain)
|
||||
hive node cordon <node-name> # NORMAL/CORDONED (reject new VMs, don't touch existing)
|
||||
hive node uncordon <node-name> # CORDONED/DRAINING → NORMAL
|
||||
hive node maintenance <node-name> # CORDONED → MAINTENANCE
|
||||
```
|
||||
|
||||
#### Implementation Details
|
||||
|
||||
**Heartbeat extension** — add `Mode` field to `Heartbeat` struct:
|
||||
```go
|
||||
type Heartbeat struct {
|
||||
// ... existing fields ...
|
||||
Mode string `json:"mode"` // "normal", "draining", "cordoned", "maintenance"
|
||||
}
|
||||
```
|
||||
|
||||
**Resource manager integration** — when mode != `normal`, unsubscribe from all `ec2.RunInstances.*` topics. The existing `updateInstanceSubscriptions()` already unsubscribes when capacity is 0; extend it to check mode.
|
||||
|
||||
**Stale heartbeat detection** — new goroutine `monitorPeerHeartbeats()`:
|
||||
- Every 30s, read all `heartbeat.*` keys from KV
|
||||
- If any peer's timestamp is >60s old (6 missed heartbeats), mark as `"suspect"`
|
||||
- If >120s old, log warning and publish alert to `hive.cluster.alert`
|
||||
- Display in `hive get nodes` as `"Unreachable"` status
|
||||
|
||||
**Graceful rejoin** — when a node starts and finds an existing `node.<name>.mode` key:
|
||||
- If `maintenance`: transition to `normal` after services are healthy
|
||||
- If `draining`: resume drain (check for remaining VMs)
|
||||
- Publish heartbeat immediately on startup to signal liveness
|
||||
|
||||
### Proposed Changes
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `hive/config/config.go` | Edit | Add `NodeMode` type and constants |
|
||||
| `hive/daemon/daemon.go` | Edit | Add `nodeMode atomic.Value`, check mode in request handlers |
|
||||
| `hive/daemon/heartbeat.go` | Edit | Add `Mode` to heartbeat, add `monitorPeerHeartbeats()` |
|
||||
| `hive/daemon/jetstream.go` | Edit | Add `WriteNodeMode`/`ReadNodeMode` KV methods |
|
||||
| `hive/daemon/daemon_handlers.go` | Edit | Check mode before accepting RunInstances |
|
||||
| `cmd/hive/cmd/node.go` | New | `hive node drain/cordon/uncordon/maintenance` commands |
|
||||
| `cmd/hive/cmd/get.go` | Edit | Show mode in `hive get nodes` output |
|
||||
|
||||
### Testing
|
||||
|
||||
- Unit tests for mode transitions, heartbeat staleness detection
|
||||
- `make test-docker-multi`: drain a node, verify new VMs route elsewhere
|
||||
- Remote cluster: `hive node drain node3`, verify VMs stop, verify `hive get nodes` shows "Draining"
|
||||
|
||||
---
|
||||
|
||||
## Phase 2.5 — Coordinated Rolling Upgrade (Planned)
|
||||
|
||||
### Problem Statement
|
||||
|
||||
Upgrading the Hive binary across a cluster currently requires either:
|
||||
1. Full cluster shutdown → upgrade all nodes → restart (downtime)
|
||||
2. Manual per-node SSH, stop, upgrade, restart (error-prone, no ordering guarantees)
|
||||
|
||||
Neither approach is acceptable for production clusters. We need a coordinated upgrade that takes nodes offline one at a time, upgrades them, and brings them back — while the cluster continues serving requests on the remaining nodes.
|
||||
|
||||
### Design: Rolling Upgrade via NATS Coordination
|
||||
|
||||
The cluster already has the building blocks: coordinated shutdown phases, heartbeats, and (with Phase 2.2) node modes. A rolling upgrade orchestrates these per-node:
|
||||
|
||||
```
|
||||
For each node (one at a time):
|
||||
1. DRAIN node (Phase 2.2) — migrate/stop VMs
|
||||
2. CORDON node — no new work
|
||||
3. Stop services on node (reuse shutdown phases, targeted to single node)
|
||||
4. Upgrade binary
|
||||
5. Start services
|
||||
6. UNCORDON — resume normal operation
|
||||
7. Wait for health check before proceeding to next node
|
||||
```
|
||||
|
||||
#### Upgrade Delivery Mechanism
|
||||
|
||||
**Option A: Binary push via Predastore (recommended)**
|
||||
|
||||
The upgrade coordinator (any node or external CLI) uploads the new `hive` binary to Predastore (S3-compatible), then each node downloads and replaces its own binary. No SSH needed, no inter-node trust beyond what NATS already provides.
|
||||
|
||||
```bash
|
||||
# Operator uploads new binary
|
||||
hive admin cluster upgrade --binary ./bin/hive
|
||||
|
||||
# Internally:
|
||||
# 1. Upload binary to predastore: s3://hive-system/upgrades/hive-<version>
|
||||
# 2. For each node:
|
||||
# a. Send NATS "hive.node.upgrade" to target node
|
||||
# b. Node downloads binary from predastore
|
||||
# c. Node replaces /usr/local/bin/hive (or wherever binary lives)
|
||||
# d. Node restarts services
|
||||
# e. Wait for heartbeat to confirm healthy
|
||||
```
|
||||
|
||||
**Option B: Manual per-node (simpler, good enough for now)**
|
||||
|
||||
The operator runs `hive node drain <name>`, SSHes in, replaces the binary, and runs `hive node uncordon <name>`. The CLI provides the orchestration framework but the binary swap is manual.
|
||||
|
||||
**Option C: SSH-based push (rejected)**
|
||||
|
||||
Having Hive nodes SSH into each other introduces key management complexity, a lateral movement attack surface, and fragile dependency on SSH config. Not recommended.
|
||||
|
||||
#### Decision: Option A for automation, Option B as fallback
|
||||
|
||||
Option A (binary via Predastore) is the right long-term approach — the storage layer already exists, and nodes can self-serve the upgrade. Option B is the immediate path since it requires no new infrastructure.
|
||||
|
||||
### Implementation Plan
|
||||
|
||||
**Phase 2.5a: Manual rolling upgrade (Option B)**
|
||||
|
||||
CLI commands that coordinate the drain/upgrade/uncordon flow but leave the binary swap to the operator:
|
||||
|
||||
```bash
|
||||
hive admin cluster upgrade --manual
|
||||
# Prints step-by-step instructions per node:
|
||||
# 1. Draining node1... done (0 VMs migrated)
|
||||
# 2. SSH to node1 and run: sudo cp /path/to/new/hive /usr/local/bin/hive
|
||||
# 3. Press Enter when done...
|
||||
# 4. Restarting node1 services...
|
||||
# 5. Waiting for node1 health... healthy
|
||||
# 6. Proceeding to node2...
|
||||
```
|
||||
|
||||
**Phase 2.5b: Automated rolling upgrade (Option A)**
|
||||
|
||||
Add binary upload to Predastore and NATS-triggered self-upgrade:
|
||||
|
||||
```bash
|
||||
hive admin cluster upgrade --binary ./bin/hive [--version v1.2.3]
|
||||
```
|
||||
|
||||
- New NATS handler `hive.node.upgrade` on each daemon
|
||||
- Handler downloads binary from predastore, validates checksum, replaces in-place
|
||||
- Daemon exec's itself to restart with new binary (or uses systemd restart)
|
||||
|
||||
### Proposed Changes
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `cmd/hive/cmd/cluster.go` | Edit | Add `clusterUpgradeCmd` with `--manual`/`--binary` flags |
|
||||
| `hive/daemon/daemon_handlers.go` | Edit | Add `hive.node.upgrade` handler (Phase 2.5b) |
|
||||
| `hive/daemon/upgrade.go` | New | Binary download, validation, self-replace logic |
|
||||
|
||||
### Testing
|
||||
|
||||
- `make test-docker-multi`: run upgrade flow (swap binary, verify services restart)
|
||||
- Remote cluster: full rolling upgrade with `hive admin cluster upgrade --manual`
|
||||
|
||||
---
|
||||
|
||||
## Phase 2.6 — Degraded Storage Writes (Planned)
|
||||
|
||||
### Problem Statement
|
||||
|
||||
**Current behavior**: Predastore uses Reed-Solomon RS(K, M) erasure coding (default RS(3,2) for 5 nodes, RS(2,1) for 3 nodes). When writing an object, it splits into K data shards + M parity shards and sends each to a deterministic node via consistent hashing. **If any shard destination is unreachable, the entire write fails immediately.** There is no retry, no alternate node selection, no degraded write mode.
|
||||
|
||||
This means a single node being drained or offline makes **all writes fail** — even though RS coding is designed to tolerate M node failures for reads.
|
||||
|
||||
**Desired behavior**: When a node is unavailable (draining, maintenance, or crashed), writes should succeed by redirecting shards to an available alternate node. The metadata should track where shards actually landed, and a background repair process should move shards back to their correct home when the node returns.
|
||||
|
||||
### Current Architecture
|
||||
|
||||
```
|
||||
PUT /bucket/key
|
||||
→ Hash object key → consistent hash ring → [node0, node1, node2] for RS(2,1)
|
||||
→ Split into 2 data shards + 1 parity shard
|
||||
→ QUIC Put to node0 (data shard 0) ← fails if node0 is down
|
||||
→ QUIC Put to node1 (data shard 1)
|
||||
→ QUIC Put to node2 (parity shard 0)
|
||||
→ Store ObjectToShardNodes metadata in s3db (Raft)
|
||||
```
|
||||
|
||||
Key code: `putObjectViaQUIC()` in `predastore/backend/distributed/distributed.go:506`
|
||||
|
||||
### Design: Shard Redirection with Repair Queue
|
||||
|
||||
#### Write Path Changes
|
||||
|
||||
When a shard destination is unreachable:
|
||||
1. Catch the QUIC dial/write error for the failing shard
|
||||
2. Query the hash ring for the next available node (skip nodes marked as draining/maintenance/offline)
|
||||
3. Write the shard to the alternate node
|
||||
4. Update `ObjectToShardNodes` metadata to reflect actual placement
|
||||
5. Enqueue a repair job: "shard X of object Y should be on node Z, currently on node W"
|
||||
|
||||
```
|
||||
PUT with node0 down:
|
||||
→ QUIC Put to node0 → FAIL
|
||||
→ Redirect: QUIC Put to node3 (next on ring)
|
||||
→ QUIC Put to node1 (data shard 1) → OK
|
||||
→ QUIC Put to node2 (parity shard 0) → OK
|
||||
→ Store metadata: DataShardNodes=[3, 1], ParityShardNodes=[2]
|
||||
→ Enqueue repair: {object: key, shard: 0, expected: node0, actual: node3}
|
||||
```
|
||||
|
||||
#### Node Health Awareness in Predastore
|
||||
|
||||
Predastore needs to know which nodes are available. Options:
|
||||
|
||||
**Option A: Heartbeat-based (NATS KV)** — Predastore reads hive daemon heartbeats from the `hive-cluster-state` KV bucket. Requires Predastore to connect to NATS (it currently doesn't).
|
||||
|
||||
**Option B: Predastore-internal health checks** — Each Predastore node pings peers via QUIC periodically. If a peer fails to respond after N attempts, mark it as unavailable locally. Simpler, self-contained.
|
||||
|
||||
**Option C: Configuration-driven** — The `hive node drain` CLI command updates predastore config (via NATS → daemon → predastore config reload). Predastore skips nodes marked as unavailable. Relies on the Hive daemon to propagate state.
|
||||
|
||||
**Recommendation**: Option B for crash detection (self-contained, no new dependencies), Option C for planned drain/maintenance (explicit, reliable). Both can coexist.
|
||||
|
||||
#### Repair Queue
|
||||
|
||||
When the unavailable node returns:
|
||||
1. The returning node announces itself (heartbeat resumes or health check passes)
|
||||
2. A background repair goroutine scans the repair queue
|
||||
3. For each misplaced shard:
|
||||
- Read shard from alternate node
|
||||
- Write shard to correct node
|
||||
- Update `ObjectToShardNodes` metadata
|
||||
- Remove repair queue entry
|
||||
4. Optionally delete the shard from the alternate node (after confirming the correct node has it)
|
||||
|
||||
Repair queue storage: JetStream stream (`HIVE-REPAIR`) or a dedicated KV bucket. Durable, replicated, survives restarts.
|
||||
|
||||
#### Write Consistency Guarantees
|
||||
|
||||
- **Minimum write quorum**: At least K data shards + M parity shards must be written for the PUT to succeed. Redirected shards count toward the quorum.
|
||||
- **If fewer than K+M nodes are available**: Write fails (cannot satisfy RS requirements). For RS(2,1) with 3 nodes, at most 1 node can be down.
|
||||
- **Metadata is always authoritative**: `ObjectToShardNodes` records where shards actually are, not where the hash ring says they should be. Reads use metadata, not the ring.
|
||||
|
||||
### Proposed Changes
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `predastore/backend/distributed/distributed.go` | Edit | Shard redirection in `putObjectViaQUIC()`, alternate node selection |
|
||||
| `predastore/backend/distributed/repair.go` | New | Repair queue, background repair goroutine |
|
||||
| `predastore/backend/distributed/health.go` | New | Peer health checks, node availability tracking |
|
||||
| `predastore/backend/distributed/distributed.go` | Edit | `ObjectToShardNodes` — ensure metadata reflects actual placement |
|
||||
|
||||
### Testing
|
||||
|
||||
- Unit tests: write with simulated node failure, verify redirect and metadata
|
||||
- `make test-docker-multi`: stop one Predastore node, verify S3 PUTs still succeed, restart node, verify repair
|
||||
- Remote cluster: drain a node, upload files via AWS CLI, verify objects readable
|
||||
|
||||
---
|
||||
|
||||
## Phase 2.7 — VM Migration (Planned)
|
||||
|
||||
### Problem Statement
|
||||
|
||||
When a node is drained for maintenance (Phase 2.2), its running VMs are currently stopped and terminated. The user's workload goes offline until they manually launch new instances on other nodes. For a platform aiming at AWS compatibility, this is unacceptable — VMs should be transparently relocated to other nodes with minimal or zero downtime.
|
||||
|
||||
### Background: Why Hive Is Well-Suited for Migration
|
||||
|
||||
Hive's architecture makes VM migration simpler than typical hypervisors:
|
||||
|
||||
1. **Storage is already shared** — all VM disks are backed by Viperblock/Predastore over the network (NBD). Both source and destination nodes can access the same volumes. No disk data transfer needed during migration.
|
||||
2. **QMP is already implemented** — `hive/qmp/qmp.go` has a working QMP client over UNIX sockets. Migration commands are just more QMP messages.
|
||||
3. **VM state is in JetStream KV** — instance metadata (`hive-instance-state` bucket) is already shared across the cluster. Cross-node handoff just updates the KV key.
|
||||
4. **`ResetNodeLocalState()`** — `hive/vm/vm.go` already has a method that zeros out PID, PTS, QMPClient for cross-node handoff.
|
||||
|
||||
### Design: Two-Phase Migration Approach
|
||||
|
||||
#### Phase 2.7a: Cold Migration (Stop → Save → Restore)
|
||||
|
||||
Simplest path — stop the VM, save its memory state to a file, transfer the file, restore on destination. Total downtime: seconds to minutes depending on RAM size.
|
||||
|
||||
**When to use**: Planned maintenance, low-priority VMs, initial implementation.
|
||||
|
||||
```
|
||||
Source Node Destination Node
|
||||
──────────── ────────────────
|
||||
1. QMP: stop (pause vCPUs)
|
||||
2. QMP: migrate "exec:cat > /tmp/state"
|
||||
3. Wait for migration complete
|
||||
4. Transfer state file ──────────────→ 5. Receive state file
|
||||
6. Start nbdkit for same volumes
|
||||
7. Launch QEMU with -incoming "exec:cat /tmp/state"
|
||||
8. VM resumes execution
|
||||
9. Cleanup: stop nbdkit, remove state
|
||||
10. Update KV: instance now on dest
|
||||
```
|
||||
|
||||
**State file transfer options**:
|
||||
- **Predastore (S3)**: Upload state to `s3://hive-system/migrations/<instance-id>.state`, destination downloads it. Reuses existing infrastructure, works across any network topology.
|
||||
- **Direct TCP**: Pipe state directly via QEMU's `tcp:` transport. Faster (no intermediate storage), but requires direct connectivity between nodes.
|
||||
- **NATS object store**: For smaller VMs (<1GB RAM), NATS JetStream object store could work. Built-in replication.
|
||||
|
||||
**Recommendation**: Start with Predastore for simplicity (upload/download pattern), add direct TCP later for performance.
|
||||
|
||||
#### Phase 2.7b: Live Migration (Pre-Copy)
|
||||
|
||||
Minimal downtime — VM runs on source while memory is copied to destination. Final switchover pauses VM for <500ms.
|
||||
|
||||
**When to use**: Production workloads, load balancing, zero-downtime maintenance.
|
||||
|
||||
```
|
||||
Source Node Destination Node
|
||||
──────────── ────────────────
|
||||
1. Start nbdkit for same volumes
|
||||
2. Launch QEMU with -incoming defer
|
||||
3. QMP: migrate-incoming "tcp:0:PORT"
|
||||
4. QMP: migrate-set-capabilities
|
||||
[auto-converge, multifd]
|
||||
5. QMP: migrate "tcp:dest:PORT"
|
||||
6. ── pre-copy: bulk RAM transfer ──→
|
||||
7. ── iterative: dirty pages ──────→
|
||||
8. ── converged: final switchover ──→ 9. VM resumes on destination
|
||||
10. Cleanup source
|
||||
11. Update KV
|
||||
```
|
||||
|
||||
**QMP commands** (existing `SendQMPCommand` infrastructure):
|
||||
|
||||
```go
|
||||
// On source:
|
||||
d.SendQMPCommand(qmpClient, QMPCommand{
|
||||
Execute: "migrate-set-capabilities",
|
||||
Arguments: map[string]any{"capabilities": []map[string]any{
|
||||
{"capability": "auto-converge", "state": true},
|
||||
}},
|
||||
}, instanceID)
|
||||
|
||||
d.SendQMPCommand(qmpClient, QMPCommand{
|
||||
Execute: "migrate",
|
||||
Arguments: map[string]any{"uri": fmt.Sprintf("tcp:%s:%d", destHost, destPort)},
|
||||
}, instanceID)
|
||||
|
||||
// Monitor progress:
|
||||
resp := d.SendQMPCommand(qmpClient, QMPCommand{Execute: "query-migrate"}, instanceID)
|
||||
// resp.Return.status: "active", "completed", "failed"
|
||||
```
|
||||
|
||||
**Convergence**: For typical VMs (512MB-8GB RAM) on 10GbE, pre-copy converges in 5-30 seconds with <300ms switchover. `auto-converge` throttles guest vCPUs if dirty page rate exceeds transfer rate.
|
||||
|
||||
**TLS for migration streams**: Use the cluster's existing CA (generated by `dev-setup.sh` / `hive admin init`). Each node gets a dual-role TLS cert for QEMU migration. QEMU's `-object tls-creds-x509` handles the TLS handshake natively.
|
||||
|
||||
### Migration Coordination Flow (NATS)
|
||||
|
||||
```
|
||||
CLI: hive node drain node3
|
||||
→ NATS "hive.node.setmode" {node: "node3", mode: "draining"}
|
||||
|
||||
Daemon (node3): mode → draining
|
||||
→ Unsubscribe from ec2.RunInstances.* topics
|
||||
→ For each running VM:
|
||||
1. Select destination: pick node with most available resources + mode=normal
|
||||
2. NATS Request "hive.node.prepare-migration" → destination daemon
|
||||
Payload: {instanceID, volumes, vmConfig, migrationType: "live"|"cold"}
|
||||
3. Destination prepares: start nbdkit, launch QEMU with -incoming
|
||||
4. Destination responds: {ready: true, migrationEndpoint: "tcp:10.1.3.170:49152"}
|
||||
5. Source initiates QEMU migration
|
||||
6. Monitor progress, publish to "hive.cluster.migration.progress"
|
||||
7. On completion:
|
||||
- Source: cleanup local QEMU, nbdkit, update KV
|
||||
- Destination: update KV with new ownership
|
||||
8. Move to next VM
|
||||
|
||||
When all VMs migrated:
|
||||
→ mode → cordoned (ready for maintenance)
|
||||
```
|
||||
|
||||
### Destination VM Preparation
|
||||
|
||||
The destination daemon must reconstruct an identical QEMU environment:
|
||||
|
||||
1. **Mount volumes**: Send `ebs.mount` NATS requests for each of the VM's volumes. Viperblock starts nbdkit processes pointing at the same underlying data.
|
||||
2. **Build QEMU command**: Same CPU, memory, machine type, devices, network config as source. Add `-incoming defer` flag.
|
||||
3. **Network setup**: Same MAC address (preserved in migration stream), same port forwarding rules.
|
||||
4. **Start QEMU**: Process starts but vCPUs are paused, waiting for migration data.
|
||||
|
||||
Key reuse point: the VM struct's `Config.Execute()` method in `hive/vm/vm.go` already builds the QEMU command. A variant with `-incoming` is straightforward.
|
||||
|
||||
### Post-Migration Cleanup
|
||||
|
||||
On the source node after successful migration:
|
||||
1. Kill the source QEMU process (should already be exited after migration completes)
|
||||
2. Unmount volumes (send `ebs.unmount` for each volume)
|
||||
3. Remove local state (PID file, QMP socket, serial PTS)
|
||||
4. Update JetStream KV: move instance from `node.<source>.instances` to `node.<dest>.instances`
|
||||
|
||||
The existing `ResetNodeLocalState()` method zeros out PID, PTS, QMPClient — exactly what's needed for the handoff.
|
||||
|
||||
### Failure Handling
|
||||
|
||||
| Failure | During Cold Migration | During Live Migration |
|
||||
|---------|----------------------|----------------------|
|
||||
| Source dies mid-migration | State file may be incomplete; VM is lost. Restart from stopped state on any node. | Migration aborts. If source QEMU still running, VM continues on source. If not, VM lost. |
|
||||
| Destination dies mid-migration | Source VM is paused; resume on source. Retry with different destination. | Migration aborts. VM continues running on source (pre-switchover). |
|
||||
| Network partition | State file transfer fails; resume on source. | Pre-copy: migration stalls, eventually times out, VM stays on source. Post-switchover: VM on destination, source can't clean up (handled on reconnect). |
|
||||
| Storage failure | Both sides fail equally (shared storage). | Same — storage is shared, not part of migration stream. |
|
||||
|
||||
### Proposed Changes
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `hive/daemon/migration.go` | New | Migration coordinator: prepare, execute (cold/live), cleanup |
|
||||
| `hive/daemon/daemon_handlers.go` | Edit | Add `hive.node.prepare-migration` NATS handler |
|
||||
| `hive/qmp/qmp.go` | Edit | Add migration QMP command types and helpers |
|
||||
| `hive/vm/vm.go` | Edit | Add `IncomingConfig()` variant for destination QEMU launch |
|
||||
| `cmd/hive/cmd/node.go` | Edit | `hive node drain` triggers migration flow |
|
||||
|
||||
### Testing
|
||||
|
||||
- Unit tests: migration state machine, QMP command construction
|
||||
- `make test-docker-multi`: cold migration between nodes (Docker E2E doesn't have KVM so live migration won't work there — use remote cluster)
|
||||
- Remote cluster (`scripts/iac/hive-test.sh`): full live migration test
|
||||
1. Launch VM on node3
|
||||
2. SSH into VM, start a long-running process
|
||||
3. `hive node drain node3`
|
||||
4. Verify VM appears on another node
|
||||
5. SSH into VM again, verify process still running (live) or VM rebooted cleanly (cold)
|
||||
|
||||
---
|
||||
|
||||
## Implementation Priority & Dependencies
|
||||
|
||||
```
|
||||
Phase 2.8 A-B-D1 (Crash Detection, Auto-Restart, OOM Scores)
|
||||
│ ↑ no external dependencies, can start immediately
|
||||
│
|
||||
Phase 2.2 (Node Modes) ◄── Phase 2.8 C+E (QMP Health, DescribeInstanceStatus API)
|
||||
│ ↑ shares health state infrastructure with 2.2
|
||||
│
|
||||
├──→ Phase 2.7a (Cold Migration) — requires drain mode + crash detection (2.8 A)
|
||||
│ │
|
||||
│ └──→ Phase 2.7b (Live Migration) — builds on cold migration
|
||||
│
|
||||
├──→ Phase 2.5 (Rolling Upgrade) — requires drain/uncordon
|
||||
│
|
||||
├──→ Phase 2.6 (Degraded Writes) — requires node health awareness
|
||||
│ ↑ benefits from memory pressure signals (2.8 D2-D3)
|
||||
│
|
||||
└──→ Phase 2.8 D2-D3-D1b (Resource Reservation, Pressure Monitoring, cgroups)
|
||||
```
|
||||
|
||||
**Recommended order**: 2.8(A,B,D1) → 2.2 + 2.8(C,E) → 2.7a → 2.5 → 2.6 → 2.7b → 2.8(D2,D3,D1b)
|
||||
|
||||
Rationale:
|
||||
- **2.8 A,B,D1 first**: Zero external dependencies. Fixes the silent QEMU crash problem immediately. OOM scores are 2 lines of code for massive protection. See [ec2-health-restart.md](ec2-health-restart.md) for details.
|
||||
- **2.2 + 2.8 C,E together**: Node modes and health monitoring share the same state infrastructure. DescribeInstanceStatus API consumes the health data that 2.2's `monitorPeerHeartbeats()` and 2.8 C's QMP health check produce.
|
||||
- **2.7a third**: Cold migration provides immediate value — VMs survive maintenance windows. Uses crash detection (2.8 A) for failure cases.
|
||||
- **2.5 fourth**: Rolling upgrades become practical once drain + cold migration work.
|
||||
- **2.6 fifth**: Degraded writes are important but require changes to Predastore (separate repo). Can be done in parallel with 2.7a.
|
||||
- **2.7b sixth**: Live migration is the polish — sub-second downtime instead of seconds. Only worth the complexity after cold migration is proven.
|
||||
- **2.8 D2,D3,D1b last**: Operational polish. Resource reservation and cgroups prevent OOM at the source but aren't blocking for other features.
|
||||
|
||||
---
|
||||
|
||||
## Key Existing Code to Reuse
|
||||
|
||||
| Function / Type | File | Relevance |
|
||||
|----------------|------|-----------|
|
||||
| `SendQMPCommand()` | `hive/daemon/daemon.go` | QMP dispatch — add migration commands |
|
||||
| `Config.Execute()` | `hive/vm/vm.go` | QEMU command builder — add `-incoming` variant |
|
||||
| `ResetNodeLocalState()` | `hive/vm/vm.go` | Zeros PID/PTS/QMP for cross-node handoff |
|
||||
| `MountVolumes()` | `hive/daemon/daemon.go` | NBD volume setup — reuse on destination |
|
||||
| `stopInstance()` | `hive/daemon/daemon.go` | QMP shutdown + cleanup — reuse for source cleanup |
|
||||
| `collectResponses()` | `cmd/hive/cmd/get.go` | NATS fan-out pattern — reuse for node queries |
|
||||
| `loadConfigAndConnect()` | `cmd/hive/cmd/get.go` | CLI NATS connection — reuse for new CLI commands |
|
||||
| `utils.StopProcess()` | `hive/utils/utils.go` | PID-based service stop — reuse in drain/upgrade |
|
||||
| `putObjectViaQUIC()` | `predastore/backend/distributed/distributed.go` | Write path — modify for shard redirection |
|
||||
| `ObjectToShardNodes` | `predastore/backend/distributed/distributed.go` | Shard metadata — extend for actual placement |
|
||||
| `hashRing.GetClosestN()` | `predastore/backend/distributed/distributed.go` | Node selection — extend to skip unavailable nodes |
|
||||
|
||||
---
|
||||
|
||||
## Verification Strategy
|
||||
|
||||
All phases will be verified with:
|
||||
1. `make preflight` — format, vet, security, all unit tests
|
||||
2. `make test-docker-multi` — multi-node E2E in Docker
|
||||
3. Remote Proxmox cluster — real hardware validation via `scripts/iac/hive-test.sh`
|
||||
|
||||
The remote test cluster is critical for Phase 2.7b (live migration) since Docker E2E doesn't support nested KVM for QEMU-to-QEMU migration.
|
||||
@@ -1,154 +0,0 @@
|
||||
# Multi-Node Predastore Configuration
|
||||
|
||||
## Overview
|
||||
|
||||
Predastore uses Reed-Solomon erasure coding RS(2,1) -- 2 data shards and 1 parity shard -- requiring a minimum of 3 distinct nodes. Each shard is placed on a separate node via a consistent hash ring (`GetClosestN(hash, 3)`), which means every node must know the full cluster topology at startup to make identical placement decisions.
|
||||
|
||||
This is fundamentally different from NATS, which uses gossip-based discovery and can add members dynamically. Predastore's hash ring is static -- if nodes disagree about membership, they'll compute different shard placements and data becomes unreachable.
|
||||
|
||||
## How It Works
|
||||
|
||||
### Init (leader node)
|
||||
|
||||
```
|
||||
hive admin init --predastore-nodes "10.11.12.1,10.11.12.2,10.11.12.3" --bind 10.11.12.1 ...
|
||||
```
|
||||
|
||||
1. The comma-separated IPs are parsed and validated. Each gets a sequential node ID (1, 2, 3).
|
||||
2. The leader's `--bind` IP is matched against the list to determine its own node ID. If the bind IP isn't in the list, init fails.
|
||||
3. `GenerateMultiNodePredastoreConfig` renders a single `predastore.toml` containing the full cluster topology -- all DB entries (port 6660), all shard entries (port 9991), RS config, auth, and buckets. Node ID 1 is marked as the Raft bootstrap leader.
|
||||
4. The predastore.toml is written to `config/predastore/predastore.toml`.
|
||||
5. The leader's node ID is written into `hive.toml` as `node_id = 1` under the `[predastore]` section.
|
||||
|
||||
### Join (follower nodes)
|
||||
|
||||
```
|
||||
hive admin join --leader https://10.11.12.1:7777 --bind 10.11.12.2 ...
|
||||
```
|
||||
|
||||
1. The follower contacts the leader's `/join` HTTP endpoint.
|
||||
2. The leader reads its `predastore.toml` from disk and includes the full content in the join response (`PredastoreConfig` field).
|
||||
3. The follower writes this verbatim to its own `config/predastore/predastore.toml`. Now all nodes have an identical copy.
|
||||
4. `ParsePredastoreNodeIDFromConfig` unmarshals the TOML, extracts the `[[db]]` array, and matches the follower's `--bind` IP to find its node ID. If the IP isn't found, join fails with an error.
|
||||
5. The detected node ID is written into the follower's `hive.toml` (e.g., `node_id = 2`).
|
||||
|
||||
### Startup
|
||||
|
||||
When `start-dev.sh` runs, it reads `node_id` from `hive.toml` using awk and exports it as `HIVE_PREDASTORE_NODE_ID`. Predastore reads this environment variable to know which DB and shard entries in the shared config belong to it.
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
### Shared config, per-node identity
|
||||
|
||||
Every node gets the **same** `predastore.toml` (the full cluster topology). The only difference per node is `node_id` in `hive.toml`. This separation exists because:
|
||||
|
||||
- Predastore needs all members known at startup for consistent hash ring agreement.
|
||||
- The `predastore.toml` describes the cluster. The `node_id` in `hive.toml` says "I am this member."
|
||||
- This makes config distribution simple: the leader just sends one file, and the follower figures out its own identity by matching its bind IP.
|
||||
|
||||
### Why IPs must be specified at init time
|
||||
|
||||
Unlike NATS (gossip discovery), Predastore requires static membership. The hash ring maps `GetClosestN(hash, 3)` to determine which 3 nodes store each object's shards. If a node joins without being in the ring, existing nodes won't route data to it. If a node is in the ring but missing, data placed on it becomes unavailable.
|
||||
|
||||
This means all Predastore node IPs must be known when the leader is initialized. The `--predastore-nodes` flag captures this.
|
||||
|
||||
### Node ID 1 is the Raft bootstrap leader
|
||||
|
||||
The generated config marks `leader = true` on the `[[db]]` entry with `id = 1`. This is the node that bootstraps the Raft cluster. It's always the first IP in the `--predastore-nodes` list, which is also the `--bind` IP of the init command (the leader node).
|
||||
|
||||
### Config distribution via join protocol
|
||||
|
||||
Rather than requiring operators to manually copy `predastore.toml` to each node, the existing HTTP join protocol distributes it. The leader's daemon reads its predastore.toml and includes it in the join response. This is the same pattern used for CA certificates.
|
||||
|
||||
The leader currently sends its predastore.toml unconditionally (even in single-node setups). The follower treats any non-empty `PredastoreConfig` in the response as "leader provided a multi-node config" and skips generating its own template-based config.
|
||||
|
||||
### Node ID detection uses proper TOML parsing
|
||||
|
||||
The follower determines its node ID by parsing the received predastore.toml with `go-toml/v2` and calling `FindNodeIDByIP` against the `[[db]]` entries. If parsing fails or the bind IP isn't found, join exits with an error. This mirrors the init path, which also exits on failure.
|
||||
|
||||
## Backward Compatibility
|
||||
|
||||
When `--predastore-nodes` is omitted from `hive admin init`:
|
||||
|
||||
- No multi-node predastore.toml is generated.
|
||||
- `PredastoreNodeID` stays at 0, so `node_id` is not emitted in hive.toml.
|
||||
- The default single-node predastore.toml template is used instead.
|
||||
- `start-dev.sh` finds no `node_id` in hive.toml and leaves `HIVE_PREDASTORE_NODE_ID` empty.
|
||||
- Predastore runs in its default single-node mode.
|
||||
|
||||
Existing single-node deployments are unaffected.
|
||||
|
||||
## Files Involved
|
||||
|
||||
| File | Role |
|
||||
|------|------|
|
||||
| `cmd/hive/cmd/admin.go` | CLI flag parsing, init/join orchestration |
|
||||
| `hive/admin/admin.go` | `GenerateMultiNodePredastoreConfig`, `FindNodeIDByIP`, `ParsePredastoreNodeIDFromConfig`, `BaseConfigFiles` |
|
||||
| `hive/config/config.go` | `PredastoreConfig.NodeID` field, `NodeJoinResponse.PredastoreConfig` field |
|
||||
| `hive/daemon/daemon.go` | `/join` endpoint reads and sends predastore.toml |
|
||||
| `cmd/hive/cmd/templates/hive.toml` | Conditional `node_id` rendering |
|
||||
| `scripts/start-dev.sh` | Auto-detects `NODE_ID` from hive.toml at startup |
|
||||
|
||||
## Generated Config Example
|
||||
|
||||
For `--predastore-nodes "10.11.12.1,10.11.12.2,10.11.12.3"`:
|
||||
|
||||
```toml
|
||||
version = "1.0"
|
||||
region = "ap-southeast-2"
|
||||
|
||||
host = "0.0.0.0"
|
||||
port = 8443
|
||||
|
||||
[rs]
|
||||
data = 2
|
||||
parity = 1
|
||||
|
||||
[[db]]
|
||||
id = 1
|
||||
host = "10.11.12.1"
|
||||
port = 6660
|
||||
path = "distributed/db/node-1/"
|
||||
leader = true
|
||||
|
||||
[[db]]
|
||||
id = 2
|
||||
host = "10.11.12.2"
|
||||
port = 6660
|
||||
path = "distributed/db/node-2/"
|
||||
|
||||
[[db]]
|
||||
id = 3
|
||||
host = "10.11.12.3"
|
||||
port = 6660
|
||||
path = "distributed/db/node-3/"
|
||||
|
||||
[[nodes]]
|
||||
id = 1
|
||||
host = "10.11.12.1"
|
||||
port = 9991
|
||||
path = "distributed/nodes/node-1/"
|
||||
|
||||
[[nodes]]
|
||||
id = 2
|
||||
host = "10.11.12.2"
|
||||
port = 9991
|
||||
path = "distributed/nodes/node-2/"
|
||||
|
||||
[[nodes]]
|
||||
id = 3
|
||||
host = "10.11.12.3"
|
||||
port = 9991
|
||||
path = "distributed/nodes/node-3/"
|
||||
|
||||
[[buckets]]
|
||||
name = "predastore"
|
||||
region = "ap-southeast-2"
|
||||
type = "distributed"
|
||||
|
||||
[[auth]]
|
||||
access_key_id = "..."
|
||||
secret_access_key = "..."
|
||||
```
|
||||
|
||||
Each node receives this identical file. Node 1's `hive.toml` has `node_id = 1`, node 2 has `node_id = 2`, etc.
|
||||
@@ -1,164 +0,0 @@
|
||||
# Viperblock Snapshot Feature
|
||||
|
||||
## Overview
|
||||
|
||||
Viperblock snapshots enable instant, zero-copy point-in-time captures of block storage volumes. A snapshot freezes the block-to-object mapping at a moment in time without copying any data. Both the snapshot and the live volume share the same immutable chunk files on S3. Volumes created from snapshots use a copy-on-write (COW) layered read path, reading unmodified blocks from the source volume's chunks and writing new data to their own chunks.
|
||||
|
||||
This is the foundation for AMI-backed instance launches -- `aws ec2 run-instances --image-id ami-ubuntu` will boot a VM that reads Ubuntu blocks on demand from the AMI's frozen snapshot, with zero image copying.
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
### 1. Snapshots are frozen block maps, not data copies
|
||||
|
||||
Viperblock's chunk files are immutable -- once `chunk.00000001.bin` is written, it never changes. New writes always produce new chunks with higher ObjectIDs. This means a snapshot only needs to capture the current `BlocksToObject` mapping (which block lives at which offset in which chunk file). The actual block data stays in place.
|
||||
|
||||
**Trade-off**: This makes snapshot creation O(mapping size) instead of O(data size), but it means the source volume's chunk files cannot be garbage collected while any snapshot references them.
|
||||
|
||||
### 2. Copy-on-write via layered block maps
|
||||
|
||||
A clone volume created from a snapshot has two block maps:
|
||||
|
||||
- **Own map** (`BlocksToObject`) -- blocks written since clone creation
|
||||
- **Base map** (`BaseBlockMap`) -- frozen read-only map from the parent snapshot
|
||||
|
||||
The read path checks own map first, then falls back to the base map, then returns zeros:
|
||||
|
||||
```
|
||||
Read block N:
|
||||
1. Own BlocksToObject / BlockStore -> found? return data from own chunks
|
||||
2. BaseBlockMap -> found? return data from source volume's chunks via ReadFrom()
|
||||
3. Neither -> zero block (never written)
|
||||
```
|
||||
|
||||
Writes always go to the clone's own chunks. Once block N is written in the clone, it shadows the snapshot's version permanently.
|
||||
|
||||
**Trade-off**: Reads of unmodified blocks require cross-volume I/O (reading from the source volume's S3 prefix). This is the same latency as a normal backend read but targets a different path. The BlockStore caches fetched base blocks so repeated reads are fast.
|
||||
|
||||
### 3. Backend interface extended with ReadFrom/WriteTo
|
||||
|
||||
The existing `Backend.Read()` and `Backend.Write()` methods construct S3 keys using the backend's configured volume name. Snapshots need to read/write across volume boundaries:
|
||||
|
||||
- `CreateSnapshot` writes checkpoint and config to `{snapshotID}/` prefix
|
||||
- Clone volumes read chunk data from `{sourceVolumeName}/` prefix
|
||||
|
||||
Rather than temporarily swapping the backend's volume name (error-prone, not thread-safe), we added two new methods to the `Backend` interface:
|
||||
|
||||
```go
|
||||
ReadFrom(volumeName string, fileType FileType, objectId uint64, offset uint32, length uint32) ([]byte, error)
|
||||
WriteTo(volumeName string, fileType FileType, objectId uint64, headers *[]byte, data *[]byte) error
|
||||
```
|
||||
|
||||
These are identical to `Read`/`Write` but use the provided volume name for path construction instead of the backend's configured name.
|
||||
|
||||
**Trade-off**: This adds two methods to an interface that all backends must implement. The alternative was adding a `volumeName` parameter to the existing methods (breaks all callers) or using a separate backend instance per snapshot operation (heavier). The `ReadFrom`/`WriteTo` approach is minimal and backward-compatible.
|
||||
|
||||
### 4. Snapshot state persisted in VBState
|
||||
|
||||
Two fields were added to `VBState` (the serialized volume config):
|
||||
|
||||
```go
|
||||
SnapshotID string // ID of parent snapshot (empty for non-clones)
|
||||
SourceVolumeName string // Volume whose chunks the base map references
|
||||
```
|
||||
|
||||
On `LoadState()`, if `SnapshotID` is set, `OpenFromSnapshot()` is called automatically to reload the base block map. This means a clone volume survives restarts -- the snapshot relationship is restored from the persisted config.
|
||||
|
||||
### 5. Both read paths modified (BlockStore and legacy)
|
||||
|
||||
Viperblock has two read paths: the optimized `readBlockStore()` (O(1) sharded index) and the legacy `read()` (map rebuilding). Both were modified identically:
|
||||
|
||||
- When a block is not found in the volume's own data structures, check `BaseBlockMap`
|
||||
- Blocks found in the base map are collected into a separate `baseConsecutiveBlocks` list
|
||||
- These are fetched via `fetchBaseBlocksFromBackend()` which uses `ReadFrom(sourceVolumeName, ...)`
|
||||
- Fetched base blocks are cached in the BlockStore/LRU for subsequent reads
|
||||
|
||||
The two-list approach (own blocks vs base blocks) avoids mixing `Read()` and `ReadFrom()` calls in the same fetch loop.
|
||||
|
||||
### 6. Snapshot checkpoint reuses existing serialization
|
||||
|
||||
The snapshot's frozen block map is serialized in the same binary format as `SaveBlockState()` -- a header followed by 26-byte `BlockLookup` entries with CRC32 checksums. `LoadSnapshotBlockMap()` deserializes using the same `readBlockWalChunk()` function.
|
||||
|
||||
This means snapshot checkpoints are validated with the same integrity checks as regular checkpoints, and no new serialization code was needed.
|
||||
|
||||
## S3 Layout
|
||||
|
||||
```
|
||||
vol-abc123/ # Source volume
|
||||
config.json
|
||||
chunks/chunk.00000001.bin # Shared -- snapshot references these
|
||||
chunks/chunk.00000002.bin
|
||||
checkpoints/blocks.00000001.bin
|
||||
|
||||
snap-xyz789/ # Snapshot
|
||||
config.json # SnapshotState metadata
|
||||
checkpoints/blocks.00000000.bin # Frozen block map at snapshot time
|
||||
|
||||
vol-newclone/ # Clone volume
|
||||
config.json # VBState with SnapshotID set
|
||||
chunks/chunk.00000001.bin # Only new writes after clone creation
|
||||
checkpoints/blocks.00000001.bin # Only clone's own block mappings
|
||||
```
|
||||
|
||||
## Files Changed (viperblock repo)
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `types/types.go` | Added `ReadFrom`, `WriteTo` to `Backend` interface |
|
||||
| `viperblock/viperblock.go` | Added `BaseBlockMap`, `SourceVolumeName`, `SnapshotID` to VB struct. Added snapshot fields to `VBState`. Modified `readBlockStore()` and legacy `read()` for base map fallback. Added `fetchBaseBlocksFromBackend()`. Updated `SaveState`/`LoadState`. |
|
||||
| `viperblock/snapshot.go` | New file: `SnapshotState` type, `CreateSnapshot()`, `LoadSnapshotBlockMap()`, `OpenFromSnapshot()`, `LookupBaseBlockToObject()` |
|
||||
| `viperblock/backends/file/file.go` | Implemented `ReadFrom()`, `WriteTo()` |
|
||||
| `viperblock/backends/s3/s3.go` | Implemented `ReadFrom()`, `WriteTo()` |
|
||||
| `viperblock/backends/memory/memory.go` | Updated stub to satisfy new interface |
|
||||
| `viperblock/snapshot_test.go` | 6 test cases across file and S3 backends |
|
||||
|
||||
## Test Coverage
|
||||
|
||||
| Test | What it validates |
|
||||
|------|-------------------|
|
||||
| `TestCreateSnapshot` | Flush + checkpoint + config written to backend; round-trip via LoadSnapshotBlockMap |
|
||||
| `TestSnapshotReadFallback` | Clone reads unmodified blocks from source volume's chunks |
|
||||
| `TestSnapshotCopyOnWrite` | Overwritten blocks return new data; unmodified blocks return snapshot data |
|
||||
| `TestSnapshotZeroBlocks` | Blocks never written in snapshot or clone return zeros |
|
||||
| `TestSnapshotMultiChunk` | Snapshot spanning multiple 4MB chunk files |
|
||||
| `TestLoadFromSnapshot` | Clone survives save/reset/reload cycle with base map restored |
|
||||
|
||||
All tests run against 4 backend configurations: file, file_nocache, s3, s3_nocache.
|
||||
|
||||
## Hive API Integration
|
||||
|
||||
### Request Flow
|
||||
|
||||
```
|
||||
AWS SDK (create-snapshot --volume-id vol-xxx)
|
||||
-> Gateway (port 9999): parse AWS query params, validate VolumeId
|
||||
-> NATS ec2.CreateSnapshot: routed to daemon via queue group
|
||||
-> Daemon handleEC2CreateSnapshot -> SnapshotServiceImpl.CreateSnapshot()
|
||||
1. Fetch volume config from Predastore (validate volume exists)
|
||||
2. NATS ebs.snapshot -> viperblockd: flush + freeze block map
|
||||
3. Write snapshot metadata to Predastore ({snapId}/metadata.json)
|
||||
<- NATS response: ec2.Snapshot JSON
|
||||
<- Gateway: wrap in XML
|
||||
<- AWS SDK: Snapshot response
|
||||
```
|
||||
|
||||
### NATS Topics
|
||||
|
||||
| Topic | Direction | Purpose |
|
||||
|-------|-----------|---------|
|
||||
| `ec2.CreateSnapshot` | Gateway -> Daemon | AWS API entry point |
|
||||
| `ebs.snapshot` | Daemon -> Viperblockd | Flush live VB instance and create frozen block map checkpoint |
|
||||
| `ec2.DescribeSnapshots` | Gateway -> Daemon | List/filter snapshots |
|
||||
| `ec2.DeleteSnapshot` | Gateway -> Daemon | Delete snapshot metadata and checkpoint |
|
||||
| `ec2.CopySnapshot` | Gateway -> Daemon | Copy snapshot metadata (metadata-only) |
|
||||
|
||||
### Files Changed (hive repo)
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `hive/handlers/ec2/snapshot/service.go` | `SnapshotService` interface |
|
||||
| `hive/handlers/ec2/snapshot/service_impl.go` | Direct implementation with S3 storage + NATS `ebs.snapshot` call |
|
||||
| `hive/handlers/ec2/snapshot/service_nats.go` | NATS proxy (gateway -> daemon) |
|
||||
| `hive/gateway/ec2/snapshot/CreateSnapshot.go` | Gateway validation + NATS dispatch |
|
||||
| `hive/daemon/daemon_handlers.go` | `handleEC2CreateSnapshot` wiring |
|
||||
| `hive/services/viperblockd/viperblockd.go` | `ebs.snapshot` subscription handler |
|
||||
| `hive/config/config.go` | `EBSSnapshotRequest` / `EBSSnapshotResponse` types |
|
||||
@@ -1,369 +0,0 @@
|
||||
# Distributed StartInstances - Manual Testing Guide
|
||||
|
||||
Manual test plan for verifying stop/start instance lifecycle across daemon nodes, including daemon restarts. Requires a multi-node cluster (see INSTALL.md "Multi-node Configuration").
|
||||
|
||||
## Prerequisites
|
||||
|
||||
```bash
|
||||
# Build
|
||||
make build
|
||||
|
||||
# Set up 3-node cluster (if not already running)
|
||||
./scripts/create-multi-node.sh
|
||||
./scripts/start-dev.sh ~/node1/
|
||||
./scripts/start-dev.sh ~/node2/
|
||||
./scripts/start-dev.sh ~/node3/
|
||||
|
||||
# AWS CLI setup
|
||||
export AWS_PROFILE=hive
|
||||
```
|
||||
|
||||
Pick a gateway endpoint for all commands (any node works):
|
||||
|
||||
```bash
|
||||
export EP="--endpoint-url https://10.11.12.1:9999"
|
||||
```
|
||||
|
||||
Import an SSH key and AMI if not already done (see INSTALL.md).
|
||||
|
||||
---
|
||||
|
||||
## Test 1: Basic Stop and Start (Same Cluster)
|
||||
|
||||
Verify an instance can be stopped and restarted by any daemon.
|
||||
|
||||
```bash
|
||||
# 1. Launch an instance
|
||||
aws $EP ec2 run-instances \
|
||||
--image-id $HIVE_AMI \
|
||||
--instance-type $HIVE_INSTANCE \
|
||||
--key-name hive-key \
|
||||
--count 1
|
||||
|
||||
export INSTANCE_ID="i-XXX" # from output
|
||||
```
|
||||
|
||||
```bash
|
||||
# 2. Confirm running
|
||||
aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
|
||||
# Expect: State.Name = "running", State.Code = 16
|
||||
```
|
||||
|
||||
```bash
|
||||
# 3. Stop the instance
|
||||
aws $EP ec2 stop-instances --instance-ids $INSTANCE_ID
|
||||
# Expect: PreviousState.Name = "running", CurrentState.Name = "stopping"
|
||||
```
|
||||
|
||||
```bash
|
||||
# 4. Wait for stop to complete, then verify stopped
|
||||
aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
|
||||
# Expect: State.Name = "stopped", State.Code = 80
|
||||
```
|
||||
|
||||
```bash
|
||||
# 5. Check daemon logs on the original node — should show KV release
|
||||
grep -i "Released stopped instance ownership" ~/node*/logs/hive.log
|
||||
# Expect: "Released stopped instance ownership to shared KV" with instanceId and lastNode
|
||||
```
|
||||
|
||||
```bash
|
||||
# 6. Start the instance
|
||||
aws $EP ec2 start-instances --instance-ids $INSTANCE_ID
|
||||
# Expect: PreviousState.Name = "stopped", CurrentState.Name = "pending"
|
||||
```
|
||||
|
||||
```bash
|
||||
# 7. Confirm running again
|
||||
aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
|
||||
# Expect: State.Name = "running", State.Code = 16
|
||||
```
|
||||
|
||||
```bash
|
||||
# 8. Check which node picked up the start
|
||||
grep -i "Started stopped instance from shared KV" ~/node*/logs/hive.log
|
||||
# Note the node — it may differ from the original
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test 2: Stop, Restart Daemon, Then Start
|
||||
|
||||
Verify a stopped instance survives a daemon restart and can be started after.
|
||||
|
||||
```bash
|
||||
# 1. Launch and stop an instance (repeat Test 1 steps 1-5)
|
||||
export INSTANCE_ID="i-XXX"
|
||||
```
|
||||
|
||||
```bash
|
||||
# 2. Identify which node originally ran the instance
|
||||
grep "Released stopped instance" ~/node*/logs/hive.log | grep $INSTANCE_ID
|
||||
# Example: ~/node1/logs/hive.log — node1 was the original
|
||||
```
|
||||
|
||||
```bash
|
||||
# 3. Restart that daemon
|
||||
./scripts/stop-dev.sh ~/node1/
|
||||
./scripts/start-dev.sh ~/node1/
|
||||
```
|
||||
|
||||
```bash
|
||||
# 4. Verify the instance is still visible as stopped
|
||||
aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
|
||||
# Expect: State.Name = "stopped"
|
||||
# The instance is in shared KV, so any running daemon can report it
|
||||
```
|
||||
|
||||
```bash
|
||||
# 5. Start the instance
|
||||
aws $EP ec2 start-instances --instance-ids $INSTANCE_ID
|
||||
# Expect: PreviousState.Name = "stopped", CurrentState.Name = "pending"
|
||||
```
|
||||
|
||||
```bash
|
||||
# 6. Confirm running and check which node picked it up
|
||||
aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
|
||||
grep "Started stopped instance from shared KV" ~/node*/logs/hive.log | grep $INSTANCE_ID
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test 3: Stop on Node A, Kill Node A, Start on Node B
|
||||
|
||||
Verify a stopped instance can be started when its original daemon is down.
|
||||
|
||||
```bash
|
||||
# 1. Launch instance (it lands on some node)
|
||||
aws $EP ec2 run-instances \
|
||||
--image-id $HIVE_AMI \
|
||||
--instance-type $HIVE_INSTANCE \
|
||||
--key-name hive-key \
|
||||
--count 1
|
||||
|
||||
export INSTANCE_ID="i-XXX"
|
||||
```
|
||||
|
||||
```bash
|
||||
# 2. Stop the instance
|
||||
aws $EP ec2 stop-instances --instance-ids $INSTANCE_ID
|
||||
```
|
||||
|
||||
```bash
|
||||
# 3. Wait for stopped state
|
||||
aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
|
||||
# Expect: State.Name = "stopped"
|
||||
```
|
||||
|
||||
```bash
|
||||
# 4. Identify the original node and shut it down
|
||||
grep "Released stopped instance" ~/node*/logs/hive.log | grep $INSTANCE_ID
|
||||
# If node1 — stop node1
|
||||
./scripts/stop-dev.sh ~/node1/
|
||||
```
|
||||
|
||||
```bash
|
||||
# 5. Point CLI to a surviving node and start the instance
|
||||
export EP="--endpoint-url https://10.11.12.2:9999"
|
||||
aws $EP ec2 start-instances --instance-ids $INSTANCE_ID
|
||||
# Expect: CurrentState.Name = "pending" — a different daemon picked it up
|
||||
```
|
||||
|
||||
```bash
|
||||
# 6. Confirm running on the new node
|
||||
aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
|
||||
# Expect: State.Name = "running"
|
||||
grep "Started stopped instance from shared KV" ~/node2/logs/hive.log ~/node3/logs/hive.log
|
||||
```
|
||||
|
||||
```bash
|
||||
# 7. Bring node1 back up (cleanup)
|
||||
./scripts/start-dev.sh ~/node1/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test 4: Repeated Stop/Start Cycles
|
||||
|
||||
Verify the instance can be stopped and started multiple times without state corruption.
|
||||
|
||||
```bash
|
||||
# 1. Launch an instance
|
||||
aws $EP ec2 run-instances \
|
||||
--image-id $HIVE_AMI \
|
||||
--instance-type $HIVE_INSTANCE \
|
||||
--key-name hive-key \
|
||||
--count 1
|
||||
|
||||
export INSTANCE_ID="i-XXX"
|
||||
```
|
||||
|
||||
```bash
|
||||
# 2. Cycle 3 times
|
||||
for i in 1 2 3; do
|
||||
echo "=== Cycle $i ==="
|
||||
|
||||
# Wait for running
|
||||
sleep 5
|
||||
aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID \
|
||||
--query 'Reservations[0].Instances[0].State.Name' --output text
|
||||
# Expect: running
|
||||
|
||||
# Stop
|
||||
aws $EP ec2 stop-instances --instance-ids $INSTANCE_ID
|
||||
sleep 10
|
||||
|
||||
aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID \
|
||||
--query 'Reservations[0].Instances[0].State.Name' --output text
|
||||
# Expect: stopped
|
||||
|
||||
# Start
|
||||
aws $EP ec2 start-instances --instance-ids $INSTANCE_ID
|
||||
sleep 10
|
||||
done
|
||||
```
|
||||
|
||||
```bash
|
||||
# 3. Final state check
|
||||
aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
|
||||
# Expect: State.Name = "running"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test 5: DescribeInstances Includes Stopped Instances
|
||||
|
||||
Verify stopped instances appear alongside running ones in describe output.
|
||||
|
||||
```bash
|
||||
# 1. Launch two instances
|
||||
aws $EP ec2 run-instances \
|
||||
--image-id $HIVE_AMI \
|
||||
--instance-type $HIVE_INSTANCE \
|
||||
--key-name hive-key \
|
||||
--count 1
|
||||
export RUNNING_ID="i-XXX"
|
||||
|
||||
aws $EP ec2 run-instances \
|
||||
--image-id $HIVE_AMI \
|
||||
--instance-type $HIVE_INSTANCE \
|
||||
--key-name hive-key \
|
||||
--count 1
|
||||
export STOPPED_ID="i-YYY"
|
||||
```
|
||||
|
||||
```bash
|
||||
# 2. Stop one, keep the other running
|
||||
aws $EP ec2 stop-instances --instance-ids $STOPPED_ID
|
||||
sleep 10
|
||||
```
|
||||
|
||||
```bash
|
||||
# 3. Describe all instances — both should appear
|
||||
aws $EP ec2 describe-instances
|
||||
# Expect: $RUNNING_ID with State.Name = "running"
|
||||
# Expect: $STOPPED_ID with State.Name = "stopped"
|
||||
```
|
||||
|
||||
```bash
|
||||
# 4. Describe with instance ID filter
|
||||
aws $EP ec2 describe-instances --instance-ids $STOPPED_ID
|
||||
# Expect: only the stopped instance
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test 6: Start a Non-Existent Instance
|
||||
|
||||
Verify proper error handling.
|
||||
|
||||
```bash
|
||||
aws $EP ec2 start-instances --instance-ids i-doesnotexist
|
||||
# Expect: instance remains reported as stopped (state unchanged)
|
||||
# Check daemon logs for "instance not found in shared KV"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test 7: Start an Already Running Instance
|
||||
|
||||
Verify that starting a running instance does not cause issues.
|
||||
|
||||
```bash
|
||||
# 1. Launch and confirm running
|
||||
aws $EP ec2 run-instances \
|
||||
--image-id $HIVE_AMI \
|
||||
--instance-type $HIVE_INSTANCE \
|
||||
--key-name hive-key \
|
||||
--count 1
|
||||
export INSTANCE_ID="i-XXX"
|
||||
sleep 5
|
||||
```
|
||||
|
||||
```bash
|
||||
# 2. Try to start it again
|
||||
aws $EP ec2 start-instances --instance-ids $INSTANCE_ID
|
||||
# Expect: instance not found in shared KV (running instances are not in shared KV)
|
||||
# The response should report the instance as still "stopped" (no-op from gateway perspective)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test 8: Daemon Crash During Stop (Migration Recovery)
|
||||
|
||||
Verify that a daemon crash between KV write and local cleanup is handled on restart.
|
||||
|
||||
```bash
|
||||
# 1. Launch and stop an instance
|
||||
aws $EP ec2 run-instances \
|
||||
--image-id $HIVE_AMI \
|
||||
--instance-type $HIVE_INSTANCE \
|
||||
--key-name hive-key \
|
||||
--count 1
|
||||
export INSTANCE_ID="i-XXX"
|
||||
|
||||
aws $EP ec2 stop-instances --instance-ids $INSTANCE_ID
|
||||
sleep 10
|
||||
```
|
||||
|
||||
```bash
|
||||
# 2. Force kill the daemon process (simulate crash — skips cleanup)
|
||||
# Identify the daemon PID on the node that ran the instance
|
||||
grep "Released stopped instance" ~/node*/logs/hive.log | grep $INSTANCE_ID
|
||||
# If node1:
|
||||
kill -9 $(pgrep -f "hive.*node1")
|
||||
```
|
||||
|
||||
```bash
|
||||
# 3. Restart the daemon
|
||||
./scripts/start-dev.sh ~/node1/
|
||||
```
|
||||
|
||||
```bash
|
||||
# 4. Check logs for migration handling
|
||||
grep -i "Migrated stopped instance" ~/node1/logs/hive.log
|
||||
# If the instance was still in per-node state, restoreInstances migrates it to shared KV
|
||||
```
|
||||
|
||||
```bash
|
||||
# 5. Verify instance is still startable
|
||||
aws $EP ec2 start-instances --instance-ids $INSTANCE_ID
|
||||
aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
|
||||
# Expect: State.Name = "running"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## What to Look For in Logs
|
||||
|
||||
Key log messages and where to find them (`~/node*/logs/hive.log`):
|
||||
|
||||
| Log Message | When |
|
||||
|-------------|------|
|
||||
| `Released stopped instance ownership to shared KV` | After successful stop + KV write |
|
||||
| `Started stopped instance from shared KV` | After a daemon picks up and launches a stopped instance |
|
||||
| `Migrated stopped instance to shared KV` | During `restoreInstances` on daemon startup |
|
||||
| `Failed to write stopped instance to shared KV, keeping local ownership` | KV write failed, instance stays local |
|
||||
| `instance not found in shared KV` | Start request for non-existent instance |
|
||||
| `instance not in stopped state` | Start request for instance not in stopped state |
|
||||
| `DescribeInstances: Collected stopped instance reservations` | Gateway merged stopped instances into describe output |
|
||||
@@ -1,141 +0,0 @@
|
||||
# Distributed StartInstances
|
||||
|
||||
Decouple stopped EC2 instances from their original daemon node, allowing any available daemon to handle `start-instances` requests.
|
||||
|
||||
## Problem
|
||||
|
||||
Previously, stopped instances remained bound to the daemon that ran them. The daemon kept an active `ec2.cmd.{instanceID}` NATS subscription after stop, so `start-instances` always routed back to the same node. In multi-node deployments, this meant a stopped instance could not be started if its original daemon was down or at capacity.
|
||||
|
||||
## Request Flow
|
||||
|
||||
```
|
||||
AWS SDK (start-instances)
|
||||
-> Gateway (port 9999): parse AWS query params
|
||||
-> NATS ec2.start (queue group: hive-workers): any daemon can pick up
|
||||
-> Daemon handleEC2StartStoppedInstance:
|
||||
1. Load instance from shared NATS KV (key: instance.{instanceID})
|
||||
2. Validate state is "stopped"
|
||||
3. Re-initialize non-serialized fields (QMPClient, EBSRequests.Mu)
|
||||
4. Allocate resources (CPU, memory)
|
||||
5. LaunchInstance (mount volumes, start QEMU, subscribe to ec2.cmd.{instanceID})
|
||||
6. Delete from shared KV
|
||||
<- NATS response: {"status":"running","instanceId":"..."}
|
||||
<- Gateway: build InstanceStateChange XML
|
||||
<- AWS SDK: StartInstancesOutput
|
||||
```
|
||||
|
||||
## How Ownership Release Works
|
||||
|
||||
When an instance is stopped (not terminated), the daemon releases ownership in this order:
|
||||
|
||||
1. Set `inst.LastNode = d.node` (record which daemon last ran it)
|
||||
2. Write instance to shared KV at key `instance.{instanceID}`
|
||||
3. Unsubscribe from `ec2.cmd.{instanceID}` NATS topic
|
||||
4. Remove instance from local `d.Instances.VMS` map
|
||||
5. Persist local state via `d.WriteState()` (now without the stopped instance)
|
||||
|
||||
The order is deliberate: shared KV is written **first**, then local state is cleaned up. If the daemon crashes between steps 2 and 5, `restoreInstances` on restart detects the overlap and completes the migration.
|
||||
|
||||
## Shared KV Storage
|
||||
|
||||
Stopped instances are stored in the existing `hive-instance-state` NATS JetStream KV bucket using the key prefix `instance.` (distinct from the per-node prefix `node.`).
|
||||
|
||||
Four methods on `JetStreamManager`:
|
||||
- `WriteStoppedInstance(instanceID, *vm.VM)` — serialize and write to `instance.{instanceID}`
|
||||
- `LoadStoppedInstance(instanceID)` — returns `nil, nil` if not found (not an error)
|
||||
- `DeleteStoppedInstance(instanceID)` — idempotent (ignores `ErrKeyNotFound`)
|
||||
- `ListStoppedInstances()` — iterates all `instance.*` keys, returns `[]*vm.VM`
|
||||
|
||||
## DescribeInstances Integration
|
||||
|
||||
`DescribeInstances` now includes stopped instances from shared KV. After the existing fan-out to all daemons (which returns running/pending instances from each node's local map), the gateway sends an additional NATS request to `ec2.DescribeStoppedInstances` (queue group, 3s timeout). The response contains reservations for all stopped instances in shared KV. These are merged into the final output.
|
||||
|
||||
If the stopped-instance query fails (timeout, no responders), it is logged and skipped — running instances are still returned. This matches the existing pattern where individual daemon fan-out failures are non-fatal.
|
||||
|
||||
## Migration on Restart
|
||||
|
||||
`restoreInstances` migrates pre-existing stopped instances from per-node state to shared KV:
|
||||
|
||||
- Instances in `StateStopped`: set `LastNode`, write to shared KV, remove from local map
|
||||
- Instances in `StateStopping`: transition to `StateStopped`, then migrate
|
||||
- `WriteState()` is called at the end to persist the removals
|
||||
|
||||
If migration fails for an instance (KV write error), the instance stays in local state as a fallback and will be retried on next restart.
|
||||
|
||||
## Files Changed
|
||||
|
||||
### VM struct: `hive/vm/vm.go`
|
||||
|
||||
Added `LastNode string` field (json: `last_node,omitempty`). Records which daemon node last ran the instance. Set during ownership release on stop. Used for diagnostics and future WAL recovery.
|
||||
|
||||
### KV methods: `hive/daemon/jetstream.go`
|
||||
|
||||
Added `StoppedInstancePrefix = "instance."` constant and four CRUD methods for stopped instance state. These reuse the `hive-instance-state` bucket — no new bucket required.
|
||||
|
||||
### Stop flow: `hive/daemon/daemon_handlers.go`
|
||||
|
||||
Extended the background goroutine in `handleStopOrTerminateInstance`. After `TransitionState(inst, StateStopped)`, the stop-only path (not terminate) releases ownership to shared KV. The terminate path is unchanged.
|
||||
|
||||
### New handlers: `hive/daemon/daemon_handlers.go`
|
||||
|
||||
`handleEC2StartStoppedInstance` — loads from shared KV, validates stopped state, re-initializes non-serialized fields, allocates resources, launches instance, removes from shared KV. On failure, rolls back resource allocation and local map insertion.
|
||||
|
||||
`handleEC2DescribeStoppedInstances` — lists all stopped instances from shared KV, applies instance ID filters from request payload, groups by reservation ID, returns `DescribeInstancesOutput`.
|
||||
|
||||
### Subscriptions: `hive/daemon/daemon.go`
|
||||
|
||||
Added to `subscribeAll()`:
|
||||
- `ec2.start` with queue group `hive-workers` — load-balanced start requests
|
||||
- `ec2.DescribeStoppedInstances` with queue group `hive-workers` — only one daemon needs to read shared KV
|
||||
|
||||
### Gateway: `hive/gateway/ec2/instance/StartInstances.go`
|
||||
|
||||
Rewritten to send `{"instance_id":"..."}` to `ec2.start` topic instead of constructing QMP commands and routing to `ec2.cmd.{instanceID}`. Removed `qmp` import.
|
||||
|
||||
### Gateway: `hive/gateway/ec2/instance/DescribeInstances.go`
|
||||
|
||||
Added a NATS request to `ec2.DescribeStoppedInstances` after the fan-out loop. Merges stopped instance reservations into the response.
|
||||
|
||||
### Restore: `hive/daemon/daemon.go`
|
||||
|
||||
`restoreInstances` now migrates stopped instances from per-node state to shared KV instead of re-subscribing to per-instance NATS topics.
|
||||
|
||||
## Design Decisions
|
||||
|
||||
**Why queue group for `ec2.start`?** NATS queue groups deliver each message to exactly one subscriber. This prevents two daemons from simultaneously trying to start the same instance, without needing distributed locks.
|
||||
|
||||
**Why shared KV instead of a database?** The `hive-instance-state` KV bucket already exists for per-node state. Reusing it avoids new infrastructure. NATS JetStream KV provides consistency guarantees sufficient for this use case (single writer per key during normal operation).
|
||||
|
||||
**Why write shared KV before removing local state?** Crash safety. If the daemon crashes after the KV write but before local cleanup, the instance exists in both locations. `restoreInstances` detects this overlap and completes the migration. The reverse order (remove local first, then write KV) would risk losing the instance entirely on crash.
|
||||
|
||||
**Why `LastNode` instead of re-routing to the original daemon?** The original daemon may be down, at capacity, or decommissioned. Recording which node last ran an instance is useful for debugging and future WAL recovery (Phase 2), but the start request goes to whichever daemon is available.
|
||||
|
||||
**Why not a dedicated KV bucket for stopped instances?** The `instance.` prefix provides sufficient namespace separation from `node.` prefix entries. A separate bucket would require additional JetStream configuration for no practical benefit.
|
||||
|
||||
## Error Codes
|
||||
|
||||
| Condition | AWS Error Code |
|
||||
|-----------|---------------|
|
||||
| No instance IDs provided | `InvalidParameterValue` (gateway) |
|
||||
| Instance not found in shared KV | `InvalidInstanceID.NotFound` |
|
||||
| Instance not in stopped state | `IncorrectInstanceState` |
|
||||
| Resource allocation failure | `InsufficientInstanceCapacity` |
|
||||
| LaunchInstance failure | `InternalError` |
|
||||
| NATS timeout (no daemons available) | State reported as still stopped |
|
||||
|
||||
## Testing
|
||||
|
||||
**JetStream KV methods** (`jetstream_test.go`): 6 tests covering write/load round-trip, missing key returns nil, delete (including non-existent), list multiple instances, no interference with per-node state, nil KV bucket handling.
|
||||
|
||||
**Handler error paths** (`daemon_handlers_test.go`): 5 tests covering missing instance ID, instance not found in KV, non-stopped state, describe returns stopped instances, describe with instance ID filter.
|
||||
|
||||
**Gateway** (`StartInstances_test.go`): 6 tests covering single instance success, multiple instances, empty input, nil IDs skipped, NATS failure (reports stopped state), mixed success and failure.
|
||||
|
||||
**State restore** (`state_test.go`): 5 tests updated to verify stopped instances are migrated to shared KV during `restoreInstances` instead of kept in local state.
|
||||
|
||||
## Known Limitations
|
||||
|
||||
- **All daemons must be updated together.** Old daemons don't subscribe to `ec2.start` and won't release stopped instances to shared KV.
|
||||
- **No WAL flush before ownership release.** Phase 1 relies on `ebs.unmount` during stop to flush volume data. Phase 2 will add explicit WAL flush to Predastore before releasing ownership.
|
||||
- **No capacity-aware scheduling.** `ec2.start` uses simple NATS queue group round-robin. A daemon might accept a start request even if it lacks resources, failing at allocation time. The gateway reports this as "still stopped" to the caller.
|
||||
- **TOCTOU on shared KV.** Between loading an instance from KV and deleting it after successful start, another daemon could theoretically load the same key. In practice this doesn't happen because NATS queue group delivers each `ec2.start` message to exactly one subscriber, and the instance ID in the message maps to a single KV key.
|
||||
@@ -1,107 +0,0 @@
|
||||
# DeleteOnTermination for EBS Volumes
|
||||
|
||||
## Overview
|
||||
|
||||
When an EC2 instance is terminated, attached EBS volumes with `DeleteOnTermination=true` are automatically deleted. Volumes with `DeleteOnTermination=false` (the default) remain in "available" state. Internal volumes (EFI, cloud-init) are always cleaned up on termination regardless of the flag.
|
||||
|
||||
This matches the AWS EC2 behavior where the root volume's lifecycle can be tied to its instance.
|
||||
|
||||
## How It Works
|
||||
|
||||
### Volume Types During Termination
|
||||
|
||||
Each instance has three volume categories in `EBSRequests.Requests`:
|
||||
|
||||
| Volume Type | Identified By | Behavior on Terminate |
|
||||
|---|---|---|
|
||||
| Root (user-visible) | `Boot=true` | Respects `DeleteOnTermination` flag |
|
||||
| EFI partition | `EFI=true` | Always cleaned up via `ebs.delete` NATS |
|
||||
| Cloud-init ISO | `CloudInit=true` | Always cleaned up via `ebs.delete` NATS |
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
RunInstances request
|
||||
→ BlockDeviceMappings[0].Ebs.DeleteOnTermination parsed in GenerateVolumes()
|
||||
→ Stored on config.EBSRequest.DeleteOnTermination (per-volume)
|
||||
→ Persisted in viperblock VolumeMetadata.DeleteOnTermination (S3 config.json)
|
||||
→ Persisted in JetStream VM state (EBSRequests serialized with the instance)
|
||||
|
||||
TerminateInstances request
|
||||
→ handleEC2Events sets deleteVolume=true
|
||||
→ stopInstance() iterates instance.EBSRequests.Requests:
|
||||
1. All volumes: unmounted via ebs.unmount NATS
|
||||
2. EFI/cloud-init: ebs.delete NATS sent to stop viperblockd processes
|
||||
3. Root with DeleteOnTermination=true: volumeService.DeleteVolume() called
|
||||
→ Sends ebs.delete NATS to viperblockd
|
||||
→ Deletes S3 prefixes: vol-xxx/, vol-xxx-efi/, vol-xxx-cloudinit/
|
||||
4. Root with DeleteOnTermination=false: skipped, remains "available"
|
||||
```
|
||||
|
||||
### Stop vs Terminate
|
||||
|
||||
- **Stop** (`deleteVolume=false`): Volumes are unmounted only. No deletion occurs regardless of the `DeleteOnTermination` flag. The volume state is set to "available".
|
||||
- **Terminate** (`deleteVolume=true`): The full deletion logic above runs.
|
||||
|
||||
## Design Decisions
|
||||
|
||||
### Why store the flag on EBSRequest rather than reading it from VolumeMetadata at termination time?
|
||||
|
||||
`EBSRequest` is the struct available on the instance at termination time (`instance.EBSRequests.Requests`). Reading from S3 (`VolumeMetadata`) would add a network round-trip during termination and could fail if predastore is degraded. The flag is set once during `GenerateVolumes()` and carried with the instance state in JetStream.
|
||||
|
||||
### Why do internal volumes use ebs.delete directly instead of DeleteVolume?
|
||||
|
||||
EFI and cloud-init volumes are sub-prefixes of the root volume in S3 (`vol-xxx-efi/`, `vol-xxx-cloudinit/`). When `DeleteVolume` is called on the root volume, it already deletes these sub-prefixes. The direct `ebs.delete` NATS message to viperblockd is only needed to stop the running nbdkit/WAL syncer processes for those volumes. This avoids double-deletion of S3 data.
|
||||
|
||||
### Why do errors not block termination?
|
||||
|
||||
Following AWS behavior, termination is a best-effort cleanup. If a volume deletion fails (e.g., predastore temporarily unavailable), the instance still transitions to terminated state. The volume may remain as orphaned S3 data that can be cleaned up later. This prevents a broken storage backend from making instances un-terminable.
|
||||
|
||||
### Why is DeleteOnTermination=false the default?
|
||||
|
||||
This matches AWS EC2 defaults. When no `BlockDeviceMappings` are specified in `RunInstances`, volumes are preserved after termination. This is the safer default since accidental data loss is worse than orphaned volumes.
|
||||
|
||||
## Files Changed
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `hive/config/config.go` | Added `DeleteOnTermination bool` to `EBSRequest` |
|
||||
| `hive/handlers/ec2/instance/service_impl.go` | Wired flag through `prepareRootVolume`, set on `EBSRequest` and `VolumeMetadata` |
|
||||
| `hive/daemon/daemon.go` | Implemented deletion logic in `stopInstance()` |
|
||||
| `hive/handlers/ec2/volume/service_impl.go` | `DescribeVolumes` reads `DeleteOnTermination` from `VolumeMetadata` instead of hardcoded `false` |
|
||||
| `viperblock/viperblock/viperblock.go` | Added `DeleteOnTermination bool` to `VolumeMetadata` |
|
||||
| `hive/daemon/daemon_test.go` | Tests for flag propagation, termination deletion, stop-not-delete, and JSON round-trip |
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit Tests
|
||||
|
||||
Run the DeleteOnTermination-specific tests:
|
||||
|
||||
```bash
|
||||
go test -v -run "TestEBSRequest|TestGenerateVolumes_DeleteOnTermination|TestStopInstance_DeleteOnTermination|TestStopInstance_NoDelete|TestEBSRequest_JSON" ./hive/daemon/
|
||||
```
|
||||
|
||||
### Manual Testing
|
||||
|
||||
```bash
|
||||
# Launch with DeleteOnTermination=true
|
||||
aws --endpoint-url https://localhost:9999 ec2 run-instances \
|
||||
--image-id ami-<id> --instance-type t3.micro --key-name <key> \
|
||||
--block-device-mappings '[{"DeviceName":"/dev/xvda","Ebs":{"DeleteOnTermination":true,"VolumeSize":4}}]' \
|
||||
--no-verify-ssl
|
||||
|
||||
# Terminate and verify volume is gone
|
||||
aws --endpoint-url https://localhost:9999 ec2 terminate-instances --instance-ids <id> --no-verify-ssl
|
||||
aws --endpoint-url https://localhost:9999 ec2 describe-volumes --volume-ids <vol-id> --no-verify-ssl
|
||||
|
||||
# Launch with default (DeleteOnTermination=false)
|
||||
# Terminate and verify volume remains in "available" state
|
||||
```
|
||||
|
||||
### Log Messages to Watch
|
||||
|
||||
- `"Volume has DeleteOnTermination=false, skipping deletion"` - volume preserved
|
||||
- `"Deleting volume with DeleteOnTermination=true"` - deletion triggered
|
||||
- `"Sent ebs.delete for internal volume"` - EFI/cloud-init process cleanup
|
||||
- `"Failed to delete volume on termination"` - deletion failed (non-blocking)
|
||||
@@ -1,82 +0,0 @@
|
||||
# Fix Docker E2E Tests and Enable Local Reproduction
|
||||
|
||||
**Status: Complete**
|
||||
|
||||
## Summary
|
||||
|
||||
Both CI E2E tests (single-node and multi-node) are failing after recent multi-server changes. This improvement fixes the root causes and adds a local runner script so developers can reproduce E2E tests without pushing to CI.
|
||||
|
||||
## Context / Problem Statement
|
||||
|
||||
### Single-Node E2E Failure
|
||||
|
||||
**Symptom:** `DescribeInstanceTypes` returns empty — logs show "nats: no responders available for request".
|
||||
|
||||
**Root cause:** Race condition. The E2E script waits for the gateway HTTP endpoint (`curl -s https://localhost:9999`), but this only confirms the gateway process is listening — not that the daemon has connected to NATS and subscribed to topics.
|
||||
|
||||
Daemon startup path: `connectNATS()` -> `ClusterManager()` -> `initJetStream()` -> create services -> `restoreInstances()` -> `subscribeAll()`. If the gateway accepts requests before the daemon finishes `subscribeAll()`, NATS publishes find no subscribers.
|
||||
|
||||
Note: `DescribeRegions` and `DescribeAvailabilityZones` succeed because they're handled locally in the gateway (`gateway/ec2.go`) without NATS. `DescribeInstanceTypes` is the first action that requires daemon NATS subscriptions.
|
||||
|
||||
### Multi-Node E2E Failure
|
||||
|
||||
**Symptom:** Formation server waits 10 minutes for nodes to join, then times out (1/3 nodes joined).
|
||||
|
||||
**Root cause:** `init_leader_node()` calls `hive admin init` without explicit `--nodes` flag. The default is `--nodes 3` (`admin.go:156`). Combined with `--bind 10.11.12.1` (not `0.0.0.0`), this triggers the multi-node formation path:
|
||||
|
||||
```go
|
||||
isMultiNode := nodes >= 2 && bindIP != "0.0.0.0"
|
||||
```
|
||||
|
||||
The formation server starts and blocks at `WaitForCompletion(formationTimeout)` waiting for 2 more nodes. But the E2E script is sequential — `join_follower_node 2` and `3` run AFTER `init_leader_node` returns, creating a deadlock.
|
||||
|
||||
Additionally, `hive admin join` also blocks — it polls `/formation/status` until `complete: true` (all nodes joined). So even if init were backgrounded, running join_2 then join_3 sequentially would deadlock because join_2 blocks waiting for node_3.
|
||||
|
||||
### Timing of CA cert and config files
|
||||
|
||||
The `runAdminInit` flow (`admin.go`):
|
||||
1. **Cert generation** (`admin.GenerateCertificatesIfNeeded`) — happens BEFORE multi-node check
|
||||
2. **Multi-node path** starts formation server, blocks at `WaitForCompletion`
|
||||
3. **Config generation** happens AFTER all nodes join
|
||||
|
||||
This means:
|
||||
- `~/node1/config/ca.pem` — available as soon as formation server starts (generated beforehand)
|
||||
- `~/node1/config/hive.toml` and other configs — only available after ALL nodes join
|
||||
|
||||
## Proposed Changes
|
||||
|
||||
### 1. Single-node fix: daemon readiness check
|
||||
|
||||
Added `wait_for_daemon_ready()` helper in `multinode-helpers.sh` that polls `describe-instance-types` until it returns a non-empty result. Called after gateway health check in both E2E scripts.
|
||||
|
||||
### 2. Multi-node fix: concurrent formation orchestration
|
||||
|
||||
Restructured `init_leader_node()` to background the init process and poll for formation server readiness. Restructured Phase 2 of `run-multinode-e2e.sh` to:
|
||||
1. Background init (starts formation server, generates certs first)
|
||||
2. Trust CA cert (exists before formation completes)
|
||||
3. Background BOTH joins concurrently (they poll `/formation/status`)
|
||||
4. Wait for all three processes to finish
|
||||
5. Start services after formation completes (configs now exist)
|
||||
|
||||
This mirrors production where init and joins run on separate machines simultaneously.
|
||||
|
||||
### 3. Local E2E runner script
|
||||
|
||||
Created `scripts/run-e2e-local.sh` that mirrors the CI workflow (`e2e.yml`) for local Docker E2E testing. Supports running single-node, multi-node, or both suites.
|
||||
|
||||
## Files to Modify
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `tests/e2e/lib/multinode-helpers.sh` | Added `wait_for_daemon_ready()`, restructured `init_leader_node()` to background init and poll formation health |
|
||||
| `tests/e2e/run-e2e.sh` | Added `wait_for_daemon_ready` call after gateway wait |
|
||||
| `tests/e2e/run-multinode-e2e.sh` | Restructured Phase 2 for concurrent formation, added `wait_for_daemon_ready` |
|
||||
| `scripts/run-e2e-local.sh` | New local Docker E2E runner script |
|
||||
| `.claude/CLAUDE.md` | Added git history guidance to Development Process section |
|
||||
|
||||
## Testing
|
||||
|
||||
1. **Multi-node fix:** Run `./scripts/run-e2e-local.sh multi` — formation server starts, all 3 joins complete concurrently, configs generated, services start, tests pass
|
||||
2. **Single-node fix:** Run `./scripts/run-e2e-local.sh single` — daemon readiness check passes, `describe-instance-types` returns instance types
|
||||
3. **Full local E2E:** Run `./scripts/run-e2e-local.sh` — both suites pass
|
||||
4. **CI validation:** Push and verify GitHub Actions E2E workflow passes
|
||||
@@ -1,129 +0,0 @@
|
||||
# Hive Cluster Stats CLI
|
||||
|
||||
**Status: Complete**
|
||||
|
||||
## Summary
|
||||
|
||||
Added `hive get {nodes,vms}` and `hive top nodes` commands to provide kubectl-style cluster visibility. Operators can now see node health, running VMs, and resource utilization across the cluster from a single command.
|
||||
|
||||
This also satisfies the `hive admin status` command proposed in `multi-server-restart.md` (Section 3.3) — `hive get nodes` and `hive top nodes` together provide the same information with better kubectl-style UX.
|
||||
|
||||
## Commands
|
||||
|
||||
### `hive get nodes`
|
||||
|
||||
Like `kubectl get nodes -o wide`. Shows all physical nodes in the cluster.
|
||||
|
||||
```
|
||||
$ hive get nodes
|
||||
NAME STATUS IP REGION AZ UPTIME VMs SERVICES
|
||||
node1 Ready 10.11.12.1 ap-southeast-2 ap-southeast-2a 3d4h 2 nats,predastore,viperblock,daemon,awsgw,ui
|
||||
node2 Ready 10.11.12.2 ap-southeast-2 ap-southeast-2a 3d4h 1 nats,predastore,viperblock,daemon
|
||||
node3 NotReady 10.11.12.3 ap-southeast-2 ap-southeast-2b - - nats,predastore,viperblock,daemon
|
||||
```
|
||||
|
||||
- **Status values**: `Ready` (daemon responded), `NotReady` (no response within timeout)
|
||||
- Nodes known from config but not responding are shown as `NotReady`
|
||||
- `--timeout` flag controls response collection window (default 3s)
|
||||
|
||||
### `hive get vms` (alias: `hive get instances`)
|
||||
|
||||
Like `kubectl get pods -A -o wide`. Shows all VMs across the cluster.
|
||||
|
||||
```
|
||||
$ hive get vms
|
||||
INSTANCE STATUS TYPE VCPU MEM NODE IP AGE
|
||||
i-896fdcfd299a39193 running t3.nano 2 512Mi node1 10.11.12.1 0m
|
||||
i-2b61dfa9fdb8c2b39 running t3.nano 2 512Mi node2 10.11.12.2 0m
|
||||
i-b097988f78db6553d running t3.nano 2 512Mi node2 10.11.12.2 0m
|
||||
```
|
||||
|
||||
### `hive top nodes`
|
||||
|
||||
Like `kubectl top nodes`. Shows resource usage per node plus aggregated available instance types.
|
||||
|
||||
```
|
||||
$ hive top nodes
|
||||
NAME CPU (used/total) MEM (used/total) VMs
|
||||
node1 4/16 1.0Gi/30.6Gi 2
|
||||
node2 2/16 512Mi/30.6Gi 1
|
||||
node3 - - -
|
||||
|
||||
INSTANCE TYPE AVAILABLE VCPU MEMORY
|
||||
t3.nano 22 2 512Mi
|
||||
t3.micro 22 2 1.0Gi
|
||||
t3.small 16 2 2.0Gi
|
||||
t3.medium 11 2 4.0Gi
|
||||
t3.large 5 2 8.0Gi
|
||||
t3.xlarge 2 4 16.0Gi
|
||||
```
|
||||
|
||||
Bottom table aggregates free capacity across all Ready nodes.
|
||||
|
||||
## Design
|
||||
|
||||
### Config as Source of Truth
|
||||
|
||||
During multi-node formation, each node's `hive.toml` includes ALL cluster nodes — the local node with full config (daemon, nats, predastore subsections) and remote nodes with basic info (host, region, AZ, services). This makes config the authoritative list of expected cluster members.
|
||||
|
||||
The CLI builds its node list from the union of config nodes + NATS responders:
|
||||
- **Config nodes that respond** → shown as `Ready` with live data from NATS
|
||||
- **Config nodes that don't respond** → shown as `NotReady` with host/region/AZ from config
|
||||
- **NATS responders not in config** → shown as `Ready` (safety net for dynamic additions)
|
||||
|
||||
### NATS Topics
|
||||
|
||||
Two fan-out topics (no queue group — all daemons respond):
|
||||
|
||||
| Topic | Purpose | Response Type |
|
||||
|-------|---------|---------------|
|
||||
| `hive.node.status` | Node stats + resource usage | `NodeStatusResponse` |
|
||||
| `hive.node.vms` | Running VMs on this node | `NodeVMsResponse` |
|
||||
|
||||
### NATS Fan-Out Collection Pattern
|
||||
|
||||
CLI connects to NATS using the config's token, publishes to a fan-out topic with an inbox reply address, and collects all responses within the timeout window:
|
||||
|
||||
```go
|
||||
inbox := nats.NewInbox()
|
||||
sub, _ := nc.SubscribeSync(inbox)
|
||||
nc.PublishRequest("hive.node.status", inbox, nil)
|
||||
for {
|
||||
msg, err := sub.NextMsg(remaining)
|
||||
if err != nil { break }
|
||||
// unmarshal and append
|
||||
}
|
||||
```
|
||||
|
||||
### ResourceManager.GetResourceStats()
|
||||
|
||||
New method that returns total/allocated vCPU and memory plus per-instance-type available capacity in a single lock acquisition. All data was already tracked — no new system calls needed.
|
||||
|
||||
## Files Modified
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `hive/config/config.go` | Added `NodeStatusResponse`, `InstanceTypeCap`, `VMInfo`, `NodeVMsResponse` types |
|
||||
| `hive/admin/admin.go` | Added `RemoteNode` type and `RemoteNodes` field to `ConfigSettings` |
|
||||
| `hive/daemon/daemon_handlers.go` | Added `daemonIP()`, `handleNodeStatus()`, `handleNodeVMs()` handlers |
|
||||
| `hive/daemon/daemon.go` | Added `hive.node.status` and `hive.node.vms` subscriptions in `subscribeAll()`, added `GetResourceStats()` method to `ResourceManager` |
|
||||
| `cmd/hive/cmd/admin.go` | Added `buildRemoteNodes()` helper; populate `RemoteNodes` in init and join flows |
|
||||
| `cmd/hive/cmd/templates/hive.toml` | Added `host` to local node, template loop for remote nodes |
|
||||
| `cmd/hive/cmd/get.go` | **New file** — `get nodes`, `get vms` commands with pterm table output |
|
||||
| `cmd/hive/cmd/top.go` | **New file** — `top nodes` command with node resource + instance type capacity tables |
|
||||
| `tests/e2e/run-e2e.sh` | Added Phase 1b (get nodes, top nodes, get vms empty) and Phase 5a-pre (get vms with running VM) |
|
||||
| `tests/e2e/run-multinode-e2e.sh` | Added Phase 3b (multi-node get/top) and post-launch get vms verification |
|
||||
|
||||
## Testing
|
||||
|
||||
- `make preflight` — all checks pass (gofmt, vet, gosec, staticcheck, govulncheck, unit tests)
|
||||
- `make test-docker-single` — single-node E2E passes with new CLI test phases
|
||||
- `make test-docker-multi` — 3-node E2E passes, `get vms` correctly shows instances across nodes with correct IPs
|
||||
|
||||
### E2E test coverage added
|
||||
|
||||
- `hive get nodes` shows Ready nodes with correct IP, region, AZ, services
|
||||
- `hive top nodes` shows CPU/MEM stats and instance type capacity table
|
||||
- `hive get vms` returns "No VMs found" when empty
|
||||
- `hive get vms` shows running instances with correct node placement after launch
|
||||
- Multi-node: VMs distributed across nodes are all visible in `get vms` output
|
||||
@@ -1,414 +0,0 @@
|
||||
# IaC Compatibility: Terraform/OpenTofu with AWS Provider
|
||||
|
||||
**Status: Planned**
|
||||
|
||||
## Summary
|
||||
|
||||
Enable infrastructure-as-code workflows using the standard Terraform/OpenTofu AWS provider (`hashicorp/aws`) pointed at MulgaOS/Hive endpoints. The goal is to run `tofu apply` against a Hive cluster and have it create real infrastructure — EC2 instances, EBS volumes, key pairs — using the same `.tf` files that work against AWS.
|
||||
|
||||
## Context / Problem Statement
|
||||
|
||||
Hive already implements 37 EC2 API actions (instances, volumes, snapshots, images, key pairs, tags, etc.), but the Terraform AWS provider requires several additional APIs during `plan` and `apply` — particularly VPC networking resources that Terraform treats as mandatory dependencies for EC2 instances.
|
||||
|
||||
The existing `scripts/iac/aws/main.tf` defines a full VPC + subnet + security group + instance stack, but `tofu apply` fails because the VPC/networking APIs aren't implemented. We need to either:
|
||||
|
||||
1. **Stub** the networking APIs (accept input, return valid-looking output, store state) so Terraform can proceed, or
|
||||
2. **Implement** them with real backing logic
|
||||
|
||||
For the initial milestone, stubbing is acceptable — Hive VMs already get networking through QEMU user-mode or bridge networking at the daemon level, so the VPC/subnet/SG resources are metadata-only for now. The critical requirement is that Terraform can complete a full `plan` → `apply` → `destroy` cycle.
|
||||
|
||||
## Current API Coverage
|
||||
|
||||
### Fully Implemented (works with Terraform)
|
||||
|
||||
| Resource | Terraform Type | Hive API Actions |
|
||||
|---|---|---|
|
||||
| EC2 Instances | `aws_instance` | `RunInstances`, `DescribeInstances`, `StartInstances`, `StopInstances`, `TerminateInstances`, `DescribeInstanceTypes` |
|
||||
| EBS Volumes | `aws_ebs_volume` | `CreateVolume`, `DescribeVolumes`, `DescribeVolumeStatus`, `ModifyVolume`, `DeleteVolume` |
|
||||
| Volume Attachment | `aws_volume_attachment` | `AttachVolume`, `DetachVolume` |
|
||||
| Key Pairs | `aws_key_pair` | `CreateKeyPair`, `ImportKeyPair`, `DescribeKeyPairs`, `DeleteKeyPair` |
|
||||
| AMI/Images | `aws_ami`, data source | `DescribeImages`, `CreateImage` |
|
||||
| Snapshots | `aws_ebs_snapshot` | `CreateSnapshot`, `DescribeSnapshots`, `DeleteSnapshot`, `CopySnapshot` |
|
||||
| Tags | (all resources) | `CreateTags`, `DeleteTags`, `DescribeTags` |
|
||||
| Regions/AZs | data sources | `DescribeRegions`, `DescribeAvailabilityZones` |
|
||||
| Account | data source | `DescribeAccountAttributes` |
|
||||
| Egress IGW | `aws_egress_only_internet_gateway` | `Create/Describe/DeleteEgressOnlyInternetGateway` |
|
||||
|
||||
### Missing — Required by Terraform AWS Provider
|
||||
|
||||
These APIs are called by Terraform during `plan`/`apply` for common infrastructure patterns. Each needs at minimum a stub that accepts input, generates an ID, stores state in NATS KV, and returns a valid response.
|
||||
|
||||
#### Priority 1: VPC Networking (blocks `tofu apply` on `main.tf`)
|
||||
|
||||
| Terraform Resource | Required API Actions | Complexity |
|
||||
|---|---|---|
|
||||
| `aws_vpc` | `CreateVpc`, `DescribeVpcs`, `DeleteVpc`, `ModifyVpcAttribute` | Medium — needs ID generation, CIDR storage, state tracking |
|
||||
| `aws_subnet` | `CreateSubnet`, `DescribeSubnets`, `DeleteSubnet` | Medium — references VPC, needs AZ assignment |
|
||||
| `aws_security_group` | `CreateSecurityGroup`, `DescribeSecurityGroups`, `DeleteSecurityGroup`, `AuthorizeSecurityGroupIngress`, `AuthorizeSecurityGroupEgress`, `RevokeSecurityGroupIngress`, `RevokeSecurityGroupEgress` | High — complex rule model, Terraform creates a default SG per VPC |
|
||||
| `aws_internet_gateway` | `CreateInternetGateway`, `DescribeInternetGateways`, `DeleteInternetGateway`, `AttachInternetGateway`, `DetachInternetGateway` | Medium — references VPC |
|
||||
| `aws_route_table` | `CreateRouteTable`, `DescribeRouteTables`, `DeleteRouteTable`, `CreateRoute`, `DeleteRoute` | Medium — references VPC, subnet associations |
|
||||
| `aws_route_table_association` | `AssociateRouteTable`, `DisassociateRouteTable` | Low — links subnet to route table |
|
||||
|
||||
#### Priority 2: Provider Initialization (called during `tofu init`/`plan`)
|
||||
|
||||
| Action | Why Terraform Calls It | Stub Needed |
|
||||
|---|---|---|
|
||||
| `GetCallerIdentity` (STS) | Provider startup — validates credentials | Yes — return static account ID |
|
||||
| `DescribeVpcAttribute` | After VPC creation — queries DNS settings | Yes — return defaults |
|
||||
| `DescribeNetworkInterfaces` | Instance creation — checks ENI state | Yes — return empty |
|
||||
| `DescribeInstanceAttribute` | After instance creation — checks attributes | Yes — return defaults |
|
||||
|
||||
#### Priority 3: Nice-to-Have
|
||||
|
||||
| Terraform Resource | Required API Actions | Notes |
|
||||
|---|---|---|
|
||||
| `aws_placement_group` | `CreatePlacementGroup`, `DescribePlacementGroups`, `DeletePlacementGroup` | Used in `main.tf` for spread strategy |
|
||||
| `aws_eip` | `AllocateAddress`, `DescribeAddresses`, `ReleaseAddress`, `AssociateAddress` | Static IP assignment |
|
||||
| `aws_network_interface` | `CreateNetworkInterface`, `DescribeNetworkInterfaces`, `DeleteNetworkInterface` | Advanced networking |
|
||||
|
||||
## Proposed Changes
|
||||
|
||||
### Phase 1: VPC Networking Stubs
|
||||
|
||||
Implement stub handlers for all Priority 1 + Priority 2 APIs. Each stub:
|
||||
|
||||
- Accepts the AWS SDK input struct
|
||||
- Generates a resource ID (e.g., `vpc-<random>`, `subnet-<random>`)
|
||||
- Stores the resource in NATS KV (bucket per resource type)
|
||||
- Returns a valid AWS-compatible response
|
||||
- Supports `Describe*` to list stored resources
|
||||
- Supports `Delete*` to remove from KV
|
||||
|
||||
Architecture:
|
||||
```
|
||||
hive/gateway/ec2/vpc/ # CreateVpc, DescribeVpcs, DeleteVpc
|
||||
hive/gateway/ec2/subnet/ # CreateSubnet, DescribeSubnets, DeleteSubnet
|
||||
hive/gateway/ec2/sg/ # Security group CRUD + rule management
|
||||
hive/gateway/ec2/igw/ # Internet gateway CRUD + attach/detach
|
||||
hive/gateway/ec2/rtb/ # Route table CRUD + associations
|
||||
hive/gateway/sts/ # GetCallerIdentity stub
|
||||
```
|
||||
|
||||
Storage: NATS KV buckets — `vpc`, `subnet`, `security-group`, `internet-gateway`, `route-table`. Each entry keyed by resource ID, value is JSON-serialized resource state.
|
||||
|
||||
### Phase 2: Terraform Integration Testing
|
||||
|
||||
Update `scripts/iac/aws/main.tf` and add automated validation:
|
||||
|
||||
1. **Simplify `main.tf`** — start with a minimal config that exercises implemented features:
|
||||
- Key pair (works today)
|
||||
- VPC + subnet + security group (after Phase 1 stubs)
|
||||
- EC2 instance with EBS volume
|
||||
- Tags on all resources
|
||||
|
||||
2. **Add `scripts/iac/aws/test.sh`** — automated test script:
|
||||
```bash
|
||||
tofu init
|
||||
tofu plan -out=plan.tfplan
|
||||
tofu apply plan.tfplan
|
||||
# Verify resources exist via AWS CLI
|
||||
tofu destroy -auto-approve
|
||||
```
|
||||
|
||||
3. **Add to CI** — optional E2E stage that runs the Terraform test against a running Hive instance
|
||||
|
||||
### Phase 3: Real VPC Networking (Future)
|
||||
|
||||
Replace stubs with real implementations as Hive networking matures:
|
||||
- Map VPCs to Linux bridges or network namespaces
|
||||
- Map subnets to DHCP ranges on bridges
|
||||
- Map security groups to nftables/iptables rules
|
||||
- Map internet gateways to NAT/routing rules
|
||||
|
||||
## Files to Modify
|
||||
|
||||
### Phase 1: New Gateway Handlers
|
||||
|
||||
| File | Description |
|
||||
|---|---|
|
||||
| `hive/gateway/ec2/vpc/CreateVpc.go` | Create VPC stub — generate ID, store in NATS KV |
|
||||
| `hive/gateway/ec2/vpc/DescribeVpcs.go` | List VPCs from NATS KV |
|
||||
| `hive/gateway/ec2/vpc/DeleteVpc.go` | Remove VPC from NATS KV |
|
||||
| `hive/gateway/ec2/subnet/CreateSubnet.go` | Create subnet stub |
|
||||
| `hive/gateway/ec2/subnet/DescribeSubnets.go` | List subnets |
|
||||
| `hive/gateway/ec2/subnet/DeleteSubnet.go` | Delete subnet |
|
||||
| `hive/gateway/ec2/sg/CreateSecurityGroup.go` | Create security group |
|
||||
| `hive/gateway/ec2/sg/DescribeSecurityGroups.go` | List security groups |
|
||||
| `hive/gateway/ec2/sg/DeleteSecurityGroup.go` | Delete security group |
|
||||
| `hive/gateway/ec2/sg/AuthorizeIngress.go` | Add ingress rule |
|
||||
| `hive/gateway/ec2/sg/AuthorizeEgress.go` | Add egress rule |
|
||||
| `hive/gateway/ec2/sg/RevokeIngress.go` | Remove ingress rule |
|
||||
| `hive/gateway/ec2/sg/RevokeEgress.go` | Remove egress rule |
|
||||
| `hive/gateway/ec2/igw/CreateInternetGateway.go` | Create IGW stub |
|
||||
| `hive/gateway/ec2/igw/DescribeInternetGateways.go` | List IGWs |
|
||||
| `hive/gateway/ec2/igw/DeleteInternetGateway.go` | Delete IGW |
|
||||
| `hive/gateway/ec2/igw/AttachInternetGateway.go` | Attach IGW to VPC |
|
||||
| `hive/gateway/ec2/igw/DetachInternetGateway.go` | Detach IGW from VPC |
|
||||
| `hive/gateway/ec2/rtb/CreateRouteTable.go` | Create route table |
|
||||
| `hive/gateway/ec2/rtb/DescribeRouteTables.go` | List route tables |
|
||||
| `hive/gateway/ec2/rtb/DeleteRouteTable.go` | Delete route table |
|
||||
| `hive/gateway/ec2/rtb/CreateRoute.go` | Add route to table |
|
||||
| `hive/gateway/ec2/rtb/DeleteRoute.go` | Remove route from table |
|
||||
| `hive/gateway/ec2/rtb/AssociateRouteTable.go` | Associate subnet with route table |
|
||||
| `hive/gateway/ec2/rtb/DisassociateRouteTable.go` | Disassociate route table |
|
||||
| `hive/gateway/sts/GetCallerIdentity.go` | STS stub — return static account |
|
||||
|
||||
### Existing Files to Modify
|
||||
|
||||
| File | Description |
|
||||
|---|---|
|
||||
| `hive/gateway/ec2.go` | Register new VPC/subnet/SG/IGW/RTB actions in `ec2Actions` map |
|
||||
| `hive/gateway/gateway.go` | Add STS service routing if needed |
|
||||
| `scripts/iac/aws/main.tf` | Update endpoint config, adjust resources for current capabilities |
|
||||
| `scripts/iac/aws/README.md` | Update with current API support status |
|
||||
|
||||
## Terraform AWS Provider Behavior Notes
|
||||
|
||||
Key things to know about how the Terraform AWS provider interacts with APIs:
|
||||
|
||||
1. **Provider init** calls `sts:GetCallerIdentity` unless `skip_requesting_account_id = true` — we set this in the provider config
|
||||
2. **VPC creation** triggers `CreateVpc` then immediately `DescribeVpcs` to wait for `available` state, then `DescribeVpcAttribute` for DNS settings
|
||||
3. **Security group** — Terraform creates a default SG when creating a VPC, and `CreateSecurityGroup` must return a `groupId`; rules are added via separate `AuthorizeSecurityGroup{Ingress,Egress}` calls
|
||||
4. **Instance creation** — Terraform passes `SubnetId` and `SecurityGroupId` in `RunInstances`; we need to accept these without erroring even if networking is stubbed
|
||||
5. **Destroy order** — Terraform destroys in reverse dependency order (instances first, then SGs, subnets, IGW, VPC). Each `Delete*` must succeed or Terraform retries.
|
||||
6. **State refresh** — On every `plan`/`apply`, Terraform calls `Describe*` for every resource in state to detect drift. All `Describe*` endpoints must return consistent data.
|
||||
|
||||
## Testing
|
||||
|
||||
### Manual Validation
|
||||
|
||||
```bash
|
||||
cd scripts/iac/aws
|
||||
export TF_VAR_mulgaos_endpoint="https://localhost:9999"
|
||||
export AWS_ACCESS_KEY_ID=test
|
||||
export AWS_SECRET_ACCESS_KEY=test
|
||||
|
||||
tofu init
|
||||
tofu plan # Should complete without errors
|
||||
tofu apply # Should create all resources
|
||||
tofu destroy # Should clean up all resources
|
||||
```
|
||||
|
||||
### Automated (Future)
|
||||
|
||||
Add a Terraform E2E stage to CI that runs the full `init → plan → apply → verify → destroy` cycle against a Hive instance in Docker.
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. VPC + DescribeVpcs + DeleteVpc (unblocks subnet, SG, IGW)
|
||||
2. Subnet + DescribeSubnets + DeleteSubnet
|
||||
3. Security Group (full CRUD + rules)
|
||||
4. Internet Gateway (CRUD + attach/detach)
|
||||
5. Route Table (CRUD + routes + associations)
|
||||
6. Wire all into `ec2.go` action map
|
||||
7. Test with `tofu plan` (catches missing APIs)
|
||||
8. Test with `tofu apply` (catches response format issues)
|
||||
9. Update `scripts/iac/aws/main.tf` and README
|
||||
|
||||
## Future Work
|
||||
|
||||
- Replace VPC stubs with real Linux bridge/namespace networking
|
||||
- Map security groups to nftables rules on hypervisor
|
||||
- Implement Elastic IPs for static addressing
|
||||
- Add S3 endpoint support (Predastore already S3-compatible)
|
||||
- Implement IAM basics for multi-tenant access control
|
||||
- Add CloudWatch-compatible metrics endpoint
|
||||
|
||||
---
|
||||
|
||||
## TODO: Pre-Baked Proxmox VM Template for Test Clusters
|
||||
|
||||
### Problem
|
||||
|
||||
Spinning up a new Hive test cluster on Proxmox currently takes a long time because every new VM starts from a base Debian 12 cloud image and must:
|
||||
|
||||
1. `apt-get update && apt-get install` ~20 packages (QEMU, nbdkit, libvirt, gcc, jq, etc.)
|
||||
2. Download and install Go 1.26.0 (~150MB)
|
||||
3. Download and install AWS CLI v2 (~100MB)
|
||||
4. `git clone` hive, viperblock, and predastore repos
|
||||
5. `go mod download` for all three repos (~hundreds of MB of Go modules)
|
||||
6. `make build` / `dev-setup.sh` to compile everything
|
||||
|
||||
This takes 10-15+ minutes per node, multiplied by 3 nodes. A pre-baked template with all dependencies pre-installed would reduce cluster spin-up to: boot VM, `git pull`, `make build` (~1-2 minutes).
|
||||
|
||||
### Current State
|
||||
|
||||
The existing Proxmox IaC (`scripts/iac/proxmox/`) uses:
|
||||
- Base image: `debian-12-genericcloud-amd64-20240211-1654.img`
|
||||
- Cloud-init: installs only `qemu-guest-agent`, `net-tools`, `git`, `make`
|
||||
- Everything else is done manually via SSH after boot
|
||||
|
||||
The E2E Docker image (`tests/e2e/Dockerfile.e2e`) already has the canonical install sequence:
|
||||
```
|
||||
make install-system # QEMU, nbdkit, libvirt, gcc, jq, curl, etc.
|
||||
make install-go # Go 1.26.0
|
||||
make install-aws # AWS CLI v2
|
||||
```
|
||||
|
||||
### Approaches Considered
|
||||
|
||||
#### Option A: Packer (Recommended)
|
||||
|
||||
[Packer](https://www.packer.io/) is the industry standard for building VM images. It has a native [Proxmox builder](https://developer.hashicorp.com/packer/integrations/hashicorp/proxmox) (`proxmox-iso` / `proxmox-clone`).
|
||||
|
||||
**How it works:**
|
||||
1. Packer boots a VM from the base Debian cloud image on Proxmox
|
||||
2. Runs provisioners (shell scripts) to install dependencies
|
||||
3. Converts the VM to a Proxmox template
|
||||
4. Future VMs clone from this template
|
||||
|
||||
**Pros:**
|
||||
- Industry standard — used by Kubernetes (image-builder), HashiCorp (Nomad/Consul AMIs), and most cloud-native projects
|
||||
- Reproducible and version-controlled (`.pkr.hcl` files in git)
|
||||
- Can reuse existing Makefile targets (`make install-system install-go install-aws`)
|
||||
- Supports multi-platform (same Packer template could build Proxmox templates AND local QEMU images)
|
||||
- Integrates with existing Proxmox IaC — just change `var.os_image` to point to the new template
|
||||
- Template can be rebuilt automatically when dependencies change (CI/scheduled)
|
||||
|
||||
**Cons:**
|
||||
- Adds Packer as a dependency (though it's a single binary, same as Terraform/OpenTofu)
|
||||
- Template build takes 10-15 minutes (but only done once, not per-cluster)
|
||||
|
||||
**Proposed file structure:**
|
||||
```
|
||||
scripts/iac/packer/
|
||||
├── hive-node.pkr.hcl # Main Packer template
|
||||
├── variables.pkr.hcl # Proxmox connection vars, image settings
|
||||
├── scripts/
|
||||
│ ├── base-setup.sh # System deps (calls make install-system)
|
||||
│ ├── dev-tools.sh # Go + AWS CLI (calls make install-go install-aws)
|
||||
│ ├── clone-repos.sh # git clone hive, viperblock, predastore
|
||||
│ └── cleanup.sh # Clear apt cache, truncate logs, zero free space
|
||||
└── README.md
|
||||
```
|
||||
|
||||
**Layered template strategy:**
|
||||
- **Layer 1 — `hive-base`**: System deps + Go + AWS CLI + cloud image cached. Rebuilt monthly or when deps change. This is the slow layer (~10 min) but changes rarely.
|
||||
- **Layer 2 — `hive-dev`** (optional): Extends `hive-base` with git clone + `go mod download` + pre-built binaries. Rebuilt more frequently. Cluster spin-up becomes just `git pull && make build`.
|
||||
|
||||
Alternatively, a single template with everything is simpler and sufficient for our scale.
|
||||
|
||||
**Example Packer template sketch:**
|
||||
```hcl
|
||||
source "proxmox-iso" "hive-node" {
|
||||
proxmox_url = var.proxmox_endpoint
|
||||
node = var.proxmox_node
|
||||
iso_file = var.base_image
|
||||
os = "l26"
|
||||
cores = 4
|
||||
memory = 8192
|
||||
disk_size = "32G"
|
||||
ssh_username = "root"
|
||||
template_name = "hive-node-template"
|
||||
template_description = "Hive test node with all dependencies pre-installed"
|
||||
}
|
||||
|
||||
build {
|
||||
sources = ["source.proxmox-iso.hive-node"]
|
||||
|
||||
# Copy Makefile for install targets
|
||||
provisioner "file" {
|
||||
source = "../../../Makefile"
|
||||
destination = "/tmp/Makefile"
|
||||
}
|
||||
|
||||
provisioner "shell" {
|
||||
inline = [
|
||||
"cd /tmp && make -f Makefile install-system",
|
||||
"cd /tmp && make -f Makefile install-go",
|
||||
"cd /tmp && make -f Makefile install-aws",
|
||||
]
|
||||
}
|
||||
|
||||
# Clone repos
|
||||
provisioner "shell" {
|
||||
script = "scripts/clone-repos.sh"
|
||||
}
|
||||
|
||||
# Cleanup for smaller template
|
||||
provisioner "shell" {
|
||||
script = "scripts/cleanup.sh"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Option B: Enhanced Cloud-Init
|
||||
|
||||
Expand the existing cloud-init in `scripts/iac/proxmox/cloud-config.tf` to install everything on first boot.
|
||||
|
||||
**Pros:**
|
||||
- No extra tools — works with current Terraform setup
|
||||
- Single source of truth (cloud-config.yaml)
|
||||
|
||||
**Cons:**
|
||||
- Runs on **every** new VM, not once-and-reuse (10-15 min per node, every time)
|
||||
- Cloud-init has timeouts for long-running operations (Go download, module cache)
|
||||
- Not a template — doesn't solve the speed problem, just automates the manual steps
|
||||
- Harder to debug when things fail (cloud-init logs are buried)
|
||||
|
||||
**Verdict:** Good for basic setup (current use), but doesn't solve the speed problem.
|
||||
|
||||
#### Option C: Manual Template + Script
|
||||
|
||||
SSH into a running VM, install everything, then convert to a Proxmox template via API:
|
||||
```bash
|
||||
# On the Proxmox host:
|
||||
qm shutdown <vmid>
|
||||
qm template <vmid>
|
||||
```
|
||||
|
||||
**Pros:**
|
||||
- Simplest approach, works right now
|
||||
- No extra tools
|
||||
|
||||
**Cons:**
|
||||
- Not reproducible — "it works on my cluster" problem
|
||||
- Not version-controlled
|
||||
- Easy to forget what was installed or configured
|
||||
- Stale templates are hard to update (must redo from scratch or maintain a VM)
|
||||
|
||||
**Verdict:** Acceptable for prototyping, but should be replaced with Packer.
|
||||
|
||||
#### Option D: qemu-img + chroot (Build Locally)
|
||||
|
||||
Mount a cloud image locally with `qemu-nbd`, chroot into it, install dependencies, then upload to Proxmox.
|
||||
|
||||
**Pros:**
|
||||
- No need to boot a VM on Proxmox
|
||||
- Fast iteration locally
|
||||
|
||||
**Cons:**
|
||||
- Needs root access and `qemu-nbd` on the build machine
|
||||
- Architecture-dependent (must match target arch or use binfmt_misc)
|
||||
- Fragile — chroot environments often have networking/DNS issues
|
||||
- Not standard practice
|
||||
|
||||
**Verdict:** Interesting for CI-built images, but too complex for our needs.
|
||||
|
||||
### What Other Projects Do
|
||||
|
||||
| Project | Approach | Notes |
|
||||
|---|---|---|
|
||||
| **Kubernetes image-builder** | Packer + Ansible | Builds node images for every cloud + bare-metal. Packer templates with Ansible provisioners. |
|
||||
| **HashiCorp reference architectures** | Packer | Pre-baked AMIs/templates for Nomad, Consul, Vault with all binaries installed |
|
||||
| **Talos Linux** | Custom OS | Entire immutable OS image built from scratch — overkill for us |
|
||||
| **Flatcar / CoreOS** | Immutable OS + Ignition | Pre-built OS with container runtime, config applied at boot via Ignition |
|
||||
| **Proxmox community** | Packer (`proxmox-iso` builder) | Dominant approach in the Proxmox ecosystem |
|
||||
| **GitLab CI runners** | Packer | Pre-baked runner images with Docker, tools pre-installed |
|
||||
|
||||
### Recommendation
|
||||
|
||||
**Start with Option A (Packer)** for the reusable template, with **Option C as an interim** step if we want something working today.
|
||||
|
||||
Concrete next steps:
|
||||
1. Manually create a template once (Option C) to unblock testing immediately
|
||||
2. Codify the process as a Packer template (Option A) for reproducibility
|
||||
3. Integrate template rebuilds into CI (rebuild weekly or on Makefile dependency changes)
|
||||
|
||||
### Open Questions
|
||||
|
||||
- **Single template vs layered?** One template with everything is simpler. Two layers (base + dev) would allow faster rebuilds when only code changes but deps stay the same.
|
||||
- **Include pre-built binaries?** If the template includes compiled Hive/Viperblock/Predastore, spin-up is nearly instant but the template goes stale faster. If it only includes Go modules, `make build` takes ~30s.
|
||||
- **Go module cache strategy?** Pre-populate `~/go/pkg/mod/` in the template? Or just `go mod download` on first boot?
|
||||
- **Cloud image caching?** Pre-download the Ubuntu/Debian cloud image used for nested VMs (like the Docker E2E image does with `/root/images/ubuntu-24.04.img`)?
|
||||
- **Template naming/versioning?** `hive-node-v1`, `hive-node-YYYYMMDD`, or `hive-node-<git-short-hash>`?
|
||||
@@ -1,341 +0,0 @@
|
||||
# Reusable IaC Test Environments for Hive
|
||||
|
||||
## Summary
|
||||
|
||||
Make the Proxmox IaC in `scripts/iac/proxmox/` reusable and parameterized so that multiple isolated Hive clusters can be provisioned, configured, tested, and destroyed independently. Introduce a four-phase lifecycle (Provision, Configure, Test, Destroy) with a single wrapper script, inspired by Kubernetes kubetest2, QEMU avocado, and SUSE Rancher patterns.
|
||||
|
||||
**Status: Planned**
|
||||
|
||||
## Context / Problem Statement
|
||||
|
||||
The current Proxmox IaC works well for its original purpose — provisioning the 3 static dev servers (10.1.3.170-172). But it has several limitations that prevent reuse:
|
||||
|
||||
### What exists today
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `scripts/iac/proxmox/main.tf` | Provider config (`bpg/proxmox` v0.52.0), locals for node map |
|
||||
| `scripts/iac/proxmox/vms.tf` | VM resources, hardcoded 4 vCPU / 8GB RAM / 32GB disk, output `iac_mulgaos_ips` |
|
||||
| `scripts/iac/proxmox/variables.tf` | Input vars: `proxmox_endpoint`, `ssh_*_key_path`, `nodes` (exactly 3) |
|
||||
| `scripts/iac/proxmox/cloud-config.tf` | Cloud-init: installs packages, clones hive, runs `quickinstall` + `dev-setup.sh` |
|
||||
| `scripts/iac/proxmox/README.md` | Manual cluster setup instructions (init node 1, join nodes 2-3) |
|
||||
|
||||
### Current gaps
|
||||
|
||||
1. **Hardcoded to 3 nodes** — `variables.tf` validates `length(var.nodes) == 3`, can't run a 1-node smoke test or a 5-node stress test
|
||||
2. **Single state file** — `terraform.tfstate` lives in the working directory, no way to have parallel environments without manually copying the directory (which is what `proxmox.orig/` is doing today)
|
||||
3. **Hardcoded VM names** — `iac-dev-mulgaos-${index}` means two environments would collide on VM names
|
||||
4. **Hardcoded specs** — 4 vCPU, 8GB RAM, 32GB disk baked into `vms.tf`, no way to test with different sizes
|
||||
5. **Cloud-init does too much** — clones hive from GitHub, runs `quickinstall`, runs `dev-setup.sh`. This couples provisioning to a specific hive version and makes iteration slow (full cloud-init reruns on every `tofu apply`)
|
||||
6. **No automated post-provisioning** — after `tofu apply`, the README has manual SSH commands for `hive admin init` on node 1, `hive admin join` on nodes 2-3, then manual `hive start` on each
|
||||
7. **No test orchestration** — no way to automatically run E2E tests against a provisioned cluster
|
||||
|
||||
## Design Principles
|
||||
|
||||
Drawing from how other infrastructure projects handle automated test environments:
|
||||
|
||||
### Kubernetes kubetest2
|
||||
|
||||
Pluggable deployer interface with lifecycle phases: `up -> test -> down`. Deployers are swappable (kind, GKE, EKS) but the lifecycle contract is the same. The test harness is decoupled from infrastructure creation — tests don't know or care whether they're running against kind or GKE.
|
||||
|
||||
**Takeaway**: Separate the lifecycle wrapper from the infrastructure backend. Today we have Proxmox; tomorrow we might add AWS or libvirt. The `up/test/down` contract stays the same.
|
||||
|
||||
### QEMU avocado
|
||||
|
||||
Parameterized test matrices — vary CPU count, memory, disk type, architecture across test runs. Test cases are decoupled from environment parameters. A single test suite runs against many different configurations.
|
||||
|
||||
**Takeaway**: Make node count, CPU, memory, and disk size variables, not constants. Enable test matrix runs like "3-node/4-core vs 5-node/8-core".
|
||||
|
||||
### SUSE Rancher
|
||||
|
||||
Two-layer IaC — Layer 1 provisions VMs (bare OS), Layer 2 deploys software (Kubernetes/Rancher). Clean separation means you can re-deploy software without re-provisioning VMs, and vice versa.
|
||||
|
||||
**Takeaway**: Cloud-init should only set up the OS. Hive installation and cluster formation belong in a separate configure phase that can be re-run independently.
|
||||
|
||||
### Terraform best practices
|
||||
|
||||
Separate state per environment (workspaces are for identical configs in different accounts, not divergent environments). Variables for everything that changes. Outputs from one phase become inputs to the next.
|
||||
|
||||
**Takeaway**: Use `-state=clusters/{name}/terraform.tfstate` for state isolation. Write structured outputs (node IPs, SSH config) to `clusters/{name}/inventory.json` for the configure phase to consume.
|
||||
|
||||
## Proposed Changes
|
||||
|
||||
### Four-phase lifecycle
|
||||
|
||||
```
|
||||
Provision (tofu) -> Configure (shell) -> Test (shell) -> Destroy (tofu)
|
||||
| | | |
|
||||
v v v v
|
||||
VMs exist Hive running Tests pass VMs gone
|
||||
IPs known Cluster formed Results saved State cleaned
|
||||
```
|
||||
|
||||
### Phase 1: Provision
|
||||
|
||||
Terraform/OpenTofu creates VMs with a minimal OS. No hive-specific setup.
|
||||
|
||||
**Changes to `scripts/iac/proxmox/variables.tf`:**
|
||||
|
||||
```hcl
|
||||
variable "cluster_name" {
|
||||
type = string
|
||||
description = "Unique name for this cluster (used in VM names, state path, tags)"
|
||||
|
||||
validation {
|
||||
condition = can(regex("^[a-z0-9-]+$", var.cluster_name))
|
||||
error_message = "cluster_name must contain only lowercase letters, numbers, and hyphens."
|
||||
}
|
||||
}
|
||||
|
||||
variable "node_count" {
|
||||
type = number
|
||||
description = "Number of nodes in the cluster (1-10)"
|
||||
default = 3
|
||||
|
||||
validation {
|
||||
condition = var.node_count >= 1 && var.node_count <= 10
|
||||
error_message = "node_count must be between 1 and 10."
|
||||
}
|
||||
}
|
||||
|
||||
variable "cpu_cores" {
|
||||
type = number
|
||||
description = "vCPU cores per VM"
|
||||
default = 4
|
||||
}
|
||||
|
||||
variable "memory_mb" {
|
||||
type = number
|
||||
description = "Memory in MB per VM"
|
||||
default = 8192
|
||||
}
|
||||
|
||||
variable "disk_size_gb" {
|
||||
type = number
|
||||
description = "Root disk size in GB per VM"
|
||||
default = 32
|
||||
}
|
||||
|
||||
variable "os_image" {
|
||||
type = string
|
||||
description = "Cloud image file ID on Proxmox (local:iso/...)"
|
||||
default = "local:iso/debian-12-genericcloud-amd64-20240211-1654.img"
|
||||
}
|
||||
```
|
||||
|
||||
The existing `nodes` variable (list of Proxmox host objects) stays — it defines *where* VMs land. The new `node_count` controls *how many* VMs are created. VMs are distributed across Proxmox hosts round-robin. The `length(var.nodes) == 3` validation is removed; any number of Proxmox hosts is valid.
|
||||
|
||||
**Changes to `scripts/iac/proxmox/vms.tf`:**
|
||||
|
||||
```hcl
|
||||
resource "proxmox_virtual_environment_vm" "hive_node" {
|
||||
count = var.node_count
|
||||
|
||||
name = "${var.cluster_name}-${count.index + 1}"
|
||||
description = "Managed by Terraform - cluster: ${var.cluster_name}"
|
||||
tags = ["terraform", "hive", var.cluster_name]
|
||||
node_name = var.nodes[count.index % length(var.nodes)].name
|
||||
|
||||
cpu {
|
||||
cores = var.cpu_cores
|
||||
type = "host"
|
||||
}
|
||||
|
||||
memory {
|
||||
dedicated = var.memory_mb
|
||||
}
|
||||
|
||||
disk {
|
||||
file_id = var.os_image
|
||||
datastore_id = var.nodes[count.index % length(var.nodes)].datastore_id
|
||||
interface = "virtio0"
|
||||
iothread = true
|
||||
discard = "on"
|
||||
size = var.disk_size_gb
|
||||
}
|
||||
|
||||
# ... initialization, network_device, agent blocks unchanged
|
||||
}
|
||||
```
|
||||
|
||||
**Changes to `scripts/iac/proxmox/cloud-config.tf`:**
|
||||
|
||||
Slim down to OS essentials only. Remove hive clone, `quickinstall`, `clone-deps.sh`, `dev-setup.sh`:
|
||||
|
||||
```yaml
|
||||
#cloud-config
|
||||
package_update: true
|
||||
package_upgrade: true
|
||||
packages:
|
||||
- qemu-guest-agent
|
||||
- net-tools
|
||||
- git
|
||||
- make
|
||||
runcmd:
|
||||
- timedatectl set-timezone America/Los_Angeles
|
||||
- systemctl enable qemu-guest-agent
|
||||
- systemctl start qemu-guest-agent
|
||||
- ufw disable
|
||||
- sysctl -w net.core.rmem_max=4194304
|
||||
- sysctl -w net.core.wmem_max=4194304
|
||||
- usermod -aG kvm tf-user
|
||||
- echo "done" > /tmp/vendor-cloud-init-done
|
||||
```
|
||||
|
||||
Hive installation moves to Phase 2 (Configure), where it can be iterated without re-provisioning VMs.
|
||||
|
||||
**New `scripts/iac/proxmox/outputs.tf`:**
|
||||
|
||||
Structured inventory output for the configure phase:
|
||||
|
||||
```hcl
|
||||
output "inventory" {
|
||||
description = "Cluster inventory for configure phase"
|
||||
value = {
|
||||
cluster_name = var.cluster_name
|
||||
nodes = [
|
||||
for i, vm in proxmox_virtual_environment_vm.hive_node : {
|
||||
name = "${var.cluster_name}-${i + 1}"
|
||||
index = i + 1
|
||||
management = vm.ipv4_addresses[1][0]
|
||||
data = vm.ipv4_addresses[2][0]
|
||||
}
|
||||
]
|
||||
ssh_user = "tf-user"
|
||||
ssh_key_path = var.ssh_private_key_path
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**State isolation:**
|
||||
|
||||
Each cluster gets its own state file via `-state` flag:
|
||||
|
||||
```bash
|
||||
tofu apply -state="clusters/${CLUSTER_NAME}/terraform.tfstate"
|
||||
```
|
||||
|
||||
The `clusters/` directory is gitignored. Each cluster's state, inventory, and logs live in `clusters/{name}/`.
|
||||
|
||||
### Phase 2: Configure
|
||||
|
||||
A shell script that reads the Terraform outputs and sets up Hive on the provisioned VMs.
|
||||
|
||||
**`scripts/iac/configure-cluster.sh <cluster_name>`:**
|
||||
|
||||
1. Read inventory from `tofu output -json -state=clusters/{name}/terraform.tfstate`
|
||||
2. Wait for cloud-init to complete on all nodes (poll `/tmp/vendor-cloud-init-done`)
|
||||
3. On each node (parallel): clone/pull hive repo, run `make quickinstall`, `clone-deps.sh`, `dev-setup.sh`, `make build`
|
||||
4. On node 1: `hive admin init` with management IPs from inventory
|
||||
5. On remaining nodes (parallel): `hive admin join` pointing at node 1
|
||||
6. On all nodes: `hive start`
|
||||
7. Wait for health checks: poll `http://<ip>:4432/health` on all nodes
|
||||
|
||||
This replaces the manual SSH commands currently in `scripts/iac/proxmox/README.md`. The `hive admin init` and `hive admin join` commands are generated from inventory data rather than hardcoded IPs.
|
||||
|
||||
### Phase 3: Test
|
||||
|
||||
**`scripts/iac/test-cluster.sh <cluster_name>`:**
|
||||
|
||||
1. Read inventory to get node IPs
|
||||
2. Set `HIVE_ENDPOINT=https://<node1-management-ip>:9999`
|
||||
3. Run E2E tests from `tests/e2e/` against the real cluster
|
||||
4. Save results to `clusters/{name}/test-results/`
|
||||
|
||||
This reuses the existing E2E test logic, currently Docker-only (`tests/e2e/run-e2e.sh`, `tests/e2e/run-multinode-e2e.sh`), pointed at real cluster IPs instead of localhost containers.
|
||||
|
||||
### Phase 4: Destroy
|
||||
|
||||
```bash
|
||||
tofu destroy -state="clusters/${CLUSTER_NAME}/terraform.tfstate"
|
||||
rm -rf "clusters/${CLUSTER_NAME}/"
|
||||
```
|
||||
|
||||
### Lifecycle wrapper
|
||||
|
||||
**`scripts/iac/hive-test.sh`** — single entry point:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Usage: hive-test.sh <command> <cluster_name> [options]
|
||||
#
|
||||
# Commands:
|
||||
# up Provision VMs (Phase 1)
|
||||
# configure Install and start Hive (Phase 2)
|
||||
# test Run E2E tests (Phase 3)
|
||||
# down Destroy VMs and clean up (Phase 4)
|
||||
# full Run all phases: up -> configure -> test -> down
|
||||
# status Show cluster status (node IPs, health)
|
||||
#
|
||||
# Options (passed through to tofu apply):
|
||||
# --node-count=N Number of nodes (default: 3)
|
||||
# --cpu-cores=N vCPU per node (default: 4)
|
||||
# --memory-mb=N Memory per node in MB (default: 8192)
|
||||
# --disk-size-gb=N Disk per node in GB (default: 32)
|
||||
|
||||
# Examples:
|
||||
# hive-test.sh full my-test-cluster
|
||||
# hive-test.sh up stress-test --node-count=5 --cpu-cores=8
|
||||
# hive-test.sh down stress-test
|
||||
```
|
||||
|
||||
The `full` command mirrors kubetest2's `--up --test --down` pattern: provision, configure, test, then tear down regardless of test outcome (with exit code preserved).
|
||||
|
||||
## Files to Modify
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `scripts/iac/proxmox/variables.tf` | Remove `length == 3` validation, add `cluster_name`, `node_count`, `cpu_cores`, `memory_mb`, `disk_size_gb`, `os_image` |
|
||||
| `scripts/iac/proxmox/vms.tf` | Use `count` instead of `for_each` on `local.nodes`, parameterize VM name/specs, round-robin placement |
|
||||
| `scripts/iac/proxmox/cloud-config.tf` | Remove hive clone/quickinstall/dev-setup, keep OS essentials only |
|
||||
| `scripts/iac/proxmox/main.tf` | No changes (provider config stays the same) |
|
||||
| `scripts/iac/proxmox/outputs.tf` | **New** — structured inventory output (IPs, SSH config, cluster name) |
|
||||
| `scripts/iac/configure-cluster.sh` | **New** — Phase 2: install hive, form cluster, start services |
|
||||
| `scripts/iac/test-cluster.sh` | **New** — Phase 3: run E2E tests against real cluster |
|
||||
| `scripts/iac/hive-test.sh` | **New** — lifecycle wrapper (up/configure/test/down/full) |
|
||||
| `scripts/iac/proxmox/.gitignore` | Add `clusters/` directory |
|
||||
|
||||
## Multi-Environment Usage Examples
|
||||
|
||||
```bash
|
||||
# Source Proxmox credentials (unchanged from today)
|
||||
source scripts/iac/proxmox/.env
|
||||
|
||||
# Environment 1: standard 3-node dev cluster
|
||||
./scripts/iac/hive-test.sh up hive-dev1
|
||||
|
||||
# Environment 2: 5-node stress test with larger VMs
|
||||
./scripts/iac/hive-test.sh up hive-stress --node-count=5 --cpu-cores=8 --memory-mb=16384
|
||||
|
||||
# Configure and test environment 1
|
||||
./scripts/iac/hive-test.sh configure hive-dev1
|
||||
./scripts/iac/hive-test.sh test hive-dev1
|
||||
|
||||
# Full lifecycle (CI use case): provision, configure, test, destroy
|
||||
./scripts/iac/hive-test.sh full hive-ci-$(date +%s)
|
||||
|
||||
# Check status of any cluster
|
||||
./scripts/iac/hive-test.sh status hive-dev1
|
||||
|
||||
# Tear down when done
|
||||
./scripts/iac/hive-test.sh down hive-stress
|
||||
./scripts/iac/hive-test.sh down hive-dev1
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
- Provision a 1-node cluster, verify VM comes up with correct name and specs
|
||||
- Provision a 3-node cluster (matches current setup), verify parity with existing behavior
|
||||
- Run configure phase, verify all nodes have hive running and cluster formed
|
||||
- Run test phase, verify E2E tests execute against real cluster
|
||||
- Run destroy phase, verify VMs are removed and state is cleaned
|
||||
- Provision two clusters simultaneously, verify no naming or state conflicts
|
||||
- Test the `full` lifecycle end-to-end
|
||||
|
||||
## Future Work
|
||||
|
||||
- **CI integration**: GitHub Actions workflow that provisions real Proxmox VMs for E2E tests on PR, with automatic teardown
|
||||
- **AWS provider parity**: Same lifecycle wrapper for a future `scripts/iac/aws/` backend, keeping the `up/configure/test/down` contract
|
||||
- **Test matrices**: Parameterized runs varying node count and instance sizes across a matrix (like avocado), e.g. `{1,3,5} nodes x {4,8} cores`
|
||||
- **Ansible migration**: Replace `configure-cluster.sh` with Ansible playbooks when shell scripting complexity grows (inventory.json maps cleanly to Ansible inventory)
|
||||
- **Re-configure without re-provision**: Since cloud-init no longer installs hive, `configure` can be re-run to deploy a new hive build without destroying/recreating VMs
|
||||
@@ -1,681 +0,0 @@
|
||||
# Multi-Server Cluster Restart
|
||||
|
||||
**Status: In Progress** — Phase 0 and Phase 1 complete. Phase 2 and 3 planned.
|
||||
|
||||
## Summary
|
||||
|
||||
When a multi-node Hive cluster restarts (stop-dev.sh then start-dev.sh on each node), every daemon independently loads the full instance state from NATS JetStream KV and attempts to relaunch all VMs simultaneously. This creates race conditions, duplicate launch attempts, and resource exhaustion. We need a coordinated recovery mechanism inspired by Ceph, Kubernetes, and etcd patterns.
|
||||
|
||||
Additionally, the cluster assumes every node runs all services (NATS, Predastore, Viperblock, Daemon, AWSGW, UI). In practice, nodes should be able to run a subset — e.g., a compute-only node running just Daemon + Viperblock that connects to remote NATS and Predastore. The config, startup scripts, health checks, and recovery logic all need to be capability-aware.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
### Current behavior
|
||||
|
||||
Each daemon's `restoreInstances()` (`daemon.go:634-719`) runs independently on startup:
|
||||
|
||||
1. Loads `node.{nodeID}` from JetStream KV — gets its own previous instances
|
||||
2. For each instance that was `StateRunning` but whose QEMU exited, resets to `StatePending` and calls `LaunchInstance()`
|
||||
3. All instances launch simultaneously with no throttling
|
||||
4. No coordination with other daemons — no awareness of cluster readiness
|
||||
|
||||
### Race conditions and failure modes
|
||||
|
||||
1. **Thundering herd on storage**: All nodes launch all their VMs at once. Each VM requires Viperblock volume mounts, NBD connections, and AMI downloads from Predastore. This saturates storage I/O before services are fully ready.
|
||||
|
||||
2. **JetStream quorum unavailable**: In a 3-node cluster with 3 replicas, JetStream needs 2 NATS servers for Raft quorum. If node1 starts restoring instances before node2's NATS is up, KV writes may fail or block. The current 10-attempt retry in `initJetStream()` (`daemon.go:574-615`) helps but doesn't guarantee other services (Predastore, Viperblock) are ready.
|
||||
|
||||
3. **No cluster-wide readiness gate**: Daemons start restoring VMs as soon as their own JetStream init succeeds, regardless of whether Predastore, Viperblock, or peer daemons are healthy. Volume mounts fail because Viperblock isn't ready; AMI fetches fail because Predastore hasn't elected a leader.
|
||||
|
||||
4. **No recovery throttling**: A node with 10 previously-running VMs will attempt to launch all 10 simultaneously — flooding QEMU starts, volume mounts, and metadata server allocations.
|
||||
|
||||
5. **Stopped instance coordination**: Stopped instances live in shared KV (`instance.{id}`) and can be started by any node via `ec2.start` with the `hive-workers` queue group. On restart, the `handleEC2StartStoppedInstance` handler (`daemon_handlers.go:1363-1463`) uses `Put` (not CAS) to claim instances, so two nodes could theoretically race on the same start request.
|
||||
|
||||
6. **No clean vs crash shutdown distinction**: The daemon writes state to KV during graceful shutdown (`setupShutdown()` at `daemon.go:1224-1228`), but there's no marker distinguishing a clean shutdown from a crash. After a crash, the KV state may be stale (last write before crash), and QEMU processes may be orphaned.
|
||||
|
||||
### Heterogeneous node problems
|
||||
|
||||
7. **All-or-nothing service startup**: `start-dev.sh` unconditionally starts all 6 services (NATS, Predastore, Viperblock, Daemon, AWSGW, UI) on every node. A compute-only node that connects to remote NATS/Predastore will fail because it tries to start local instances of services it doesn't need.
|
||||
|
||||
8. **No capability model in config**: `Config` struct (`config/config.go:22-41`) has no field indicating which services a node runs. The `hive.toml` template always generates all service sections. There's no way to express "this node runs daemon + viperblock, but uses remote NATS and Predastore."
|
||||
|
||||
9. **Health checks assume local services**: The daemon's `waitForClusterReady()` (proposed in Phase 1) must distinguish between "wait for local Viperblock to start" vs "wait for remote Predastore at 10.1.3.170:8443 to be reachable" — currently there's no way to know which is which.
|
||||
|
||||
10. **Formation and heartbeats lack capability info**: `formation.NodeInfo` and `NodeHealthResponse` have no capability/role fields. The recovery coordinator can't make capacity-aware decisions without knowing which nodes can run VMs vs which are storage-only.
|
||||
|
||||
## Research: Distributed Systems Patterns
|
||||
|
||||
### Ceph OSD restart patterns
|
||||
|
||||
- **Backfill reservation**: Limits concurrent recovery operations per OSD via `osd_max_backfills` (default: 1). Each OSD has local and remote reservers — a semaphore-based system that prevents thundering herd. Priority ordering ensures degraded/critical recovery happens first.
|
||||
- **Recovery delay**: `osd_recovery_delay_start` introduces a deliberate delay after OSD boot before recovery begins, allowing the cluster to stabilize (peering, Raft leader election).
|
||||
- **`noout` flag**: Distinguishes planned restart (don't rebalance) from permanent failure (rebalance after grace period). Set before maintenance, unset after.
|
||||
- **Peering and epochs**: Before serving I/O, an OSD must verify its cluster map epoch is current via `up_thru`. Prevents stale decisions based on outdated topology.
|
||||
|
||||
### Kubernetes node restart patterns
|
||||
|
||||
- **Node lease heartbeats**: Kubelets maintain a `Lease` object with `renewTime`. If not renewed within `leaseDurationSeconds` (40s), the node controller marks the node `NotReady`.
|
||||
- **Taint-based eviction timeline**: T+0 miss heartbeat → T+40s mark NotReady → T+340s (5min grace) evict pods. Deliberately slow to tolerate transient failures. When a node recovers, taints auto-clear but evicted pods stay on their new nodes.
|
||||
- **Pod Disruption Budgets**: Limit concurrent disruptions cluster-wide, ensuring minimum availability during rolling restarts.
|
||||
- **Learner pattern** (from etcd): New members join as non-voting learners, catch up on state, then get promoted to voters. Prevents a partially-synced member from affecting quorum.
|
||||
- **Node labels and taints**: Kubernetes uses labels (not hard-coded types) to describe node capabilities. Scheduling matches workload requirements to node labels via affinity rules. A node can have multiple roles (control-plane + worker) or be specialized.
|
||||
|
||||
### etcd cluster recovery
|
||||
|
||||
- **Quorum requirement**: Needs `(N/2)+1` members for reads/writes. In a 3-node cluster, losing 2 nodes makes the cluster read-only or unavailable.
|
||||
- **Raft log replay**: On restart, members replay their WAL before accepting requests. No client traffic until replay complete.
|
||||
- **Learner promotion**: Members start as learners, sync data, then promote to full members. Safe and reversible.
|
||||
|
||||
### Consul/Nomad server vs client pattern
|
||||
|
||||
- **Two-tier architecture**: Server nodes participate in Raft consensus; client nodes run workloads and forward RPCs to servers. A client is configured with `retry_join` — a list of server addresses.
|
||||
- **Role advertising via gossip tags**: Serf gossip carries per-node metadata (role, datacenter, port). Any node can inspect peers' capabilities without a central registry.
|
||||
- **Service registration**: Each node's local agent registers only the services it runs. Health checks are per-service, not per-node.
|
||||
|
||||
### Key takeaways for Hive
|
||||
|
||||
| Pattern | Source | Application to Hive |
|
||||
|---------|--------|---------------------|
|
||||
| Recovery delay | Ceph | Wait for cluster readiness before restoring VMs |
|
||||
| Concurrency limit | Ceph | Max N concurrent VM launches per node during recovery |
|
||||
| Heartbeat leases | K8s | Daemons publish heartbeats to JetStream KV for peer monitoring |
|
||||
| Grace period | K8s | Don't reschedule VMs from a down node for 5 minutes |
|
||||
| Quorum gate | etcd | Don't begin recovery until JetStream has quorum |
|
||||
| Fencing tokens | Kleppmann | Use KV revision-based CAS to prevent duplicate VM claims |
|
||||
| Clean shutdown marker | General | Distinguish crash from graceful stop for recovery decisions |
|
||||
| Node labels | K8s | Capabilities as config-driven labels, not hard-coded types |
|
||||
| Static bootstrap + dynamic registration | Consul/Ceph | Config lists remote service addresses; node advertises its own capabilities at runtime |
|
||||
| Per-service health checks | Consul | Only health-check what the node actually runs |
|
||||
|
||||
## Proposed Changes
|
||||
|
||||
### Phase 0: Node capabilities and service topology — Complete
|
||||
|
||||
Foundation for everything else — the cluster must know what each node runs.
|
||||
|
||||
#### 0.1 Capabilities in config — Done
|
||||
|
||||
Added `Services []string` field to the `Config` struct (`config/config.go:29`) and two helper methods:
|
||||
- `HasService(name string) bool` — checks if a service is in the list (or returns true for all if list is empty, for backward compat)
|
||||
- `GetServices() []string` — returns the services list, defaulting to `AllServices` when empty
|
||||
|
||||
The `AllServices` package variable defines the canonical list: `["nats", "predastore", "viperblock", "daemon", "awsgw", "ui"]`.
|
||||
|
||||
The originally-planned role helpers (`IsComputeNode`, `IsStorageNode`, `IsInfraNode`) were deferred — `HasService()` is sufficient and more flexible.
|
||||
|
||||
Updated `NodeHealthResponse` to include `Services []string` and `ServiceHealth map[string]string`.
|
||||
|
||||
Unit tests added in `hive/config/config_test.go`: `TestHasService_ExplicitList`, `TestHasService_EmptyListDefaultsAll`, `TestHasService_UnknownService`, `TestGetServices_DefaultsToAll`, `TestGetServices_ExplicitList`.
|
||||
|
||||
**Files modified:**
|
||||
- `hive/config/config.go` — `Services` field, `AllServices`, `HasService()`, `GetServices()`, updated `NodeHealthResponse`
|
||||
- `hive/config/config_test.go` — 5 new tests
|
||||
|
||||
#### 0.2 Formation carries capabilities — Done
|
||||
|
||||
Added `Services []string` to `NodeInfo` in `hive/formation/formation.go`. Since `JoinRequest` embeds `NodeInfo`, it automatically gets the field.
|
||||
|
||||
Updated `BuildClusterRoutes()` to filter nodes by `"nats"` service — only nodes running NATS are included in cluster routes. Updated `BuildPredastoreNodes()` to filter by `"predastore"` service. Added `hasService()` helper with backward-compat behavior (empty services list = all services).
|
||||
|
||||
Unit tests added in `hive/formation/helpers_test.go`: `TestHasService_EmptyListMeansAll`, `TestHasService_ExplicitList`, `TestBuildClusterRoutes_FiltersNonNATSNodes`, `TestBuildClusterRoutes_EmptyServicesIncludesAll`, `TestBuildPredastoreNodes_FiltersByPredastore`, `TestBuildPredastoreNodes_EmptyServicesIncludesAll`, `TestBuildClusterRoutes_MixedServicesAndEmpty`.
|
||||
|
||||
**Files modified:**
|
||||
- `hive/formation/formation.go` — `Services` field on `NodeInfo`
|
||||
- `hive/formation/helpers.go` — `hasService()` helper, filtered `BuildClusterRoutes()` and `BuildPredastoreNodes()`
|
||||
- `hive/formation/helpers_test.go` — 7 new tests
|
||||
|
||||
#### 0.3 Config template and CLI flags — Done
|
||||
|
||||
Added `Services []string` to `ConfigSettings` in `hive/admin/admin.go`.
|
||||
|
||||
Added `services` template rendering in `cmd/hive/cmd/templates/hive.toml` (after the `az` line).
|
||||
|
||||
Added `--services` StringSlice flag to both `adminInitCmd` and `adminJoinCmd` in `cmd/hive/cmd/admin.go`. The flag is read in `runAdminInit`, `runAdminInitMultiNode`, and `runAdminJoin`, and passed through to `ConfigSettings`.
|
||||
|
||||
**Files modified:**
|
||||
- `hive/admin/admin.go` — `Services` field on `ConfigSettings`
|
||||
- `cmd/hive/cmd/templates/hive.toml` — `services` template line
|
||||
- `cmd/hive/cmd/admin.go` — `--services` flag on init/join, wiring to `ConfigSettings`
|
||||
|
||||
#### 0.4 Capability-aware start/stop scripts — Done
|
||||
|
||||
Added `parse_services()` and `has_service()` bash functions to both `scripts/start-dev.sh` and `scripts/stop-dev.sh`.
|
||||
|
||||
`parse_services()` reads the `services` line from `hive.toml` and defaults to all services if not set. All 6 service start/stop blocks are wrapped in `has_service` conditionals with skip messages for non-local services.
|
||||
|
||||
The stop script maps between process names and config service names (e.g., `hive` → `daemon`, `hive-ui` → `ui`).
|
||||
|
||||
**Files modified:**
|
||||
- `scripts/start-dev.sh` — `parse_services()`, `has_service()`, conditional service startup
|
||||
- `scripts/stop-dev.sh` — `parse_services()`, `has_service()` (with name mapping), conditional service stop
|
||||
|
||||
#### 0.5 Health endpoint and cluster state KV — Done
|
||||
|
||||
**Health endpoint**: Updated the `/health` HTTP handler in `daemon.go` to include `Services` (from config) and `ServiceHealth` (per-service status map). For local services, status is `"ok"`. For remote NATS dependency (when nats is not a local service), checks `natsConn.IsConnected()` and reports `"remote_ok"` or `"remote_unreachable"`.
|
||||
|
||||
**Cluster state KV bucket**: Added `hive-cluster-state` bucket alongside the existing `hive-instance-state` bucket. New `clusterKV` field on `JetStreamManager`. `InitClusterStateBucket()` creates the bucket with 1-hour TTL and history of 1. Called from `initJetStream()` after `InitKVBucket()`.
|
||||
|
||||
**Service manifest**: `WriteServiceManifest()` writes `node.{nodeID}.services` to the cluster state KV on daemon startup, containing the node's services, NATS host, predastore host, and timestamp.
|
||||
|
||||
`UpdateReplicas()` updated to also update the cluster-state bucket replicas alongside the instance-state bucket.
|
||||
|
||||
**Files modified:**
|
||||
- `hive/daemon/daemon.go` — health endpoint with services/serviceHealth, `InitClusterStateBucket()` call in `initJetStream()`, `WriteServiceManifest()` call in `Start()`
|
||||
- `hive/daemon/jetstream.go` — `ClusterStateBucket` const, `clusterKV` field, `InitClusterStateBucket()`, `WriteServiceManifest()`, updated `UpdateReplicas()`
|
||||
|
||||
### Phase 1: Self-recovery improvements (single-node correctness) — Complete
|
||||
|
||||
These changes fix the immediate restart bugs without requiring cross-node coordination.
|
||||
|
||||
#### 1.1 Cluster readiness gate — Done
|
||||
|
||||
Added `waitForClusterReady()` method to `Daemon`, called in `Start()` just before `restoreInstances()`. Uses a 2-minute max wait with 2-second polling interval.
|
||||
|
||||
Two checks run in sequence:
|
||||
1. **Viperblock** — `checkViperblockReady()` verifies NATS connection is live. Initially implemented as a NATS request to `ebs.{node}.health`, but viperblock has no health NATS topic — this caused a 2-minute hang during E2E tests. Fixed to simply check `d.natsConn.IsConnected()` since viperblock runs alongside the daemon and subscribes to NATS.
|
||||
2. **Predastore** — `checkPredastoreReady()` does a TCP dial to the predastore host:port from config with a 3-second timeout. Skipped if no predastore host is configured.
|
||||
|
||||
If all checks pass, recovery proceeds immediately. If the 2-minute timeout expires, recovery proceeds anyway with a warning log.
|
||||
|
||||
**Finding**: Viperblock does not expose a health endpoint via NATS. Any future viperblock health check should use TCP connectivity or an HTTP endpoint, not NATS request-reply.
|
||||
|
||||
**Files modified:**
|
||||
- `hive/daemon/daemon.go` — `waitForClusterReady()`, `checkViperblockReady()`, `checkPredastoreReady()`, added `"net"` import
|
||||
|
||||
#### 1.2 Recovery throttling with semaphore — Done
|
||||
|
||||
Refactored `restoreInstances()` into two phases:
|
||||
|
||||
**Phase 1 (sequential)**: Iterates all instances from KV state. For each instance:
|
||||
- Reconnects to still-running QEMU processes (checks PID file, instant, no I/O)
|
||||
- Finalizes transitional states (stopping → stopped, shutting-down → terminated)
|
||||
- Collects instances that need relaunching into a `toLaunch` list
|
||||
|
||||
**Phase 2 (throttled)**: Launches crashed VMs using a semaphore channel with `maxConcurrentRecovery = 2`. Each launch runs in a goroutine that acquires the semaphore before proceeding and releases on completion. A `sync.WaitGroup` ensures all launches complete before `restoreInstances()` returns.
|
||||
|
||||
This prevents thundering herd on storage during recovery while keeping the lightweight reconnect/finalize steps sequential.
|
||||
|
||||
**Files modified:**
|
||||
- `hive/daemon/daemon.go` — refactored `restoreInstances()` with two-phase approach and semaphore
|
||||
|
||||
#### 1.3 Clean shutdown marker — Done
|
||||
|
||||
Three new methods on `JetStreamManager`:
|
||||
- `WriteShutdownMarker(nodeID)` — writes `shutdown.{nodeID}` to `hive-cluster-state` KV with node name and timestamp
|
||||
- `ReadShutdownMarker(nodeID)` — checks if a shutdown marker exists for the node
|
||||
- `DeleteShutdownMarker(nodeID)` — deletes the marker (tolerates not-found)
|
||||
|
||||
**Write**: In `setupShutdown()`, the shutdown marker is written to cluster state KV before `WriteState()` persists instance state. This ensures the marker is present if the state write also succeeds.
|
||||
|
||||
**Read**: At the start of `restoreInstances()`, the marker is checked. If present: log "Clean shutdown marker found, trusting KV state", delete the marker, proceed normally. If absent: log a warning about possible crash recovery, sleep 3 seconds to let orphaned QEMU processes finish exiting, then proceed with careful PID validation.
|
||||
|
||||
**Files modified:**
|
||||
- `hive/daemon/jetstream.go` — `WriteShutdownMarker()`, `ReadShutdownMarker()`, `DeleteShutdownMarker()`
|
||||
- `hive/daemon/daemon.go` — marker write in `setupShutdown()`, marker check/delete in `restoreInstances()`
|
||||
|
||||
#### 1.4 Service manifest on startup — Done
|
||||
|
||||
Added `WriteServiceManifest()` to `JetStreamManager`, which writes `node.{nodeID}.services` to the cluster state KV containing the node's service list, NATS host, predastore host, and timestamp.
|
||||
|
||||
Called in `Start()` after `initJetStream()` succeeds. Failure is non-fatal (logged as warning).
|
||||
|
||||
**Files modified:**
|
||||
- `hive/daemon/jetstream.go` — `WriteServiceManifest()`
|
||||
- `hive/daemon/daemon.go` — call in `Start()`
|
||||
|
||||
### Phase 2: Cross-node coordination
|
||||
|
||||
These changes enable daemons to coordinate lifecycle events — both failure recovery and graceful cluster-wide shutdown.
|
||||
|
||||
#### 2.1 Heartbeat publishing
|
||||
|
||||
Each daemon publishes a heartbeat to `hive-cluster-state` KV every 10 seconds:
|
||||
|
||||
```
|
||||
Key: heartbeat.{nodeID}
|
||||
Value: {"node": "node1", "epoch": 1, "timestamp": "...", "services": ["daemon", "viperblock", ...], "vm_count": 4, "vcpu_avail": 8, "mem_avail_gb": 16.0}
|
||||
```
|
||||
|
||||
Use `kv.Put()` — no need for CAS since only the owning daemon writes its own heartbeat. Other daemons use `kv.Watch("heartbeat.*")` for push-based notification of changes — no polling. The 1-hour TTL on the bucket auto-cleans dead node heartbeats.
|
||||
|
||||
Heartbeat publishing starts after self-recovery completes (after `restoreInstances()` returns), so the heartbeat reflects accurate resource state.
|
||||
|
||||
**Files:**
|
||||
- `hive/daemon/jetstream.go` — new `WriteHeartbeat()` / `WatchHeartbeats()`
|
||||
- `hive/daemon/daemon.go` — start heartbeat goroutine after `restoreInstances()`
|
||||
|
||||
#### 2.2 Peer failure detection
|
||||
|
||||
After self-recovery and heartbeat publishing begin, each daemon monitors peer heartbeats:
|
||||
|
||||
- **Stale threshold**: 60 seconds (6 missed heartbeats). Mark peer as `suspect`.
|
||||
- **Failed threshold**: 300 seconds (5 minutes). Mark peer as `failed` — eligible for VM recovery.
|
||||
|
||||
These thresholds are deliberately conservative (matching Kubernetes' approach) to avoid false positives during rolling restarts where nodes come back quickly.
|
||||
|
||||
**Files:**
|
||||
- `hive/daemon/daemon.go` — new `monitorPeerHealth()` goroutine, started alongside heartbeat publisher
|
||||
|
||||
#### 2.3 Leader-elected recovery coordinator (capability-aware)
|
||||
|
||||
When a peer is detected as `failed`, surviving daemons elect a recovery leader using JetStream KV CAS:
|
||||
|
||||
```
|
||||
Key: leader.recovery
|
||||
Value: {"node": "node2", "epoch": 42, "acquired": "..."}
|
||||
```
|
||||
|
||||
Acquisition uses `kv.Create()` (create-if-not-exists) for atomic leader election. The leader holds the lease for 60 seconds and must renew it. If the leader fails, another daemon acquires it after the lease expires.
|
||||
|
||||
The recovery leader:
|
||||
1. Reads the failed node's state from `node.{failedNode}` in `hive-instance-state` KV
|
||||
2. Filters to VMs that were `StateRunning` or `StatePending`
|
||||
3. Assigns VMs to surviving **compute-capable** nodes (those with `"daemon"` in their services, from heartbeat data) based on resource availability
|
||||
4. Writes recovery assignments to KV: `recovery.{instanceID} = {"target_node": "node2", "epoch": 42}`
|
||||
5. Each target daemon watches for its assignments and launches them
|
||||
|
||||
**Capability-aware placement**: The leader only assigns VMs to nodes whose heartbeats include `"daemon"` in their `services` list and have sufficient `vcpu_avail` and `mem_avail_gb`. Storage-only nodes are never assigned VMs.
|
||||
|
||||
**Fencing**: Each assignment includes the recovery epoch. Before launching, the target daemon verifies the epoch is still current via CAS. If another leader took over with a higher epoch, the stale assignment is ignored.
|
||||
|
||||
**Files:**
|
||||
- `hive/daemon/jetstream.go` — new `AcquireRecoveryLease()` / `RenewRecoveryLease()` / `WriteRecoveryAssignment()` / `WatchRecoveryAssignments()`
|
||||
- `hive/daemon/recovery.go` — new file, recovery coordinator logic
|
||||
- `hive/daemon/daemon.go` — integrate recovery coordinator startup
|
||||
|
||||
#### 2.4 Coordinated cluster shutdown
|
||||
|
||||
##### Problem
|
||||
|
||||
The current `stop-dev.sh` operates per-node: each node independently shuts down its own services in reverse startup order. In a multi-node cluster, this creates ordering hazards:
|
||||
|
||||
1. **Storage pulled from under running VMs**: If node1 stops Predastore before node2's VMs have finished flushing writes through Viperblock, data can be lost or corrupted.
|
||||
2. **Cascade failures during rolling stop**: Stopping NATS on one node can break JetStream quorum before other nodes have persisted their state.
|
||||
3. **Manual coordination required**: An operator must SSH into each node and run `stop-dev.sh` in the right order — infrastructure nodes last, compute nodes first. This is error-prone and doesn't scale.
|
||||
4. **No cluster-wide drain**: There's no way to stop accepting new API requests across all gateways simultaneously. A request arriving at node2's AWSGW during node1's shutdown could try to launch a VM on a node that's mid-teardown.
|
||||
5. **Orphaned nbdkit processes**: If a daemon exits before cleanly unmounting all volumes, nbdkit processes may be left running with no parent to clean them up. The current `stop-dev.sh` handles this via `pgrep`/`pkill` but only for the local node.
|
||||
|
||||
##### Design inspiration
|
||||
|
||||
The coordinated shutdown draws from several distributed systems patterns:
|
||||
|
||||
| Source | Pattern | Application to Hive |
|
||||
|--------|---------|---------------------|
|
||||
| Ceph | Pre-shutdown flags (`noout`, `norecover`, `nobackfill`) set before maintenance; monitors shut down last | Write cluster shutdown marker before any service stops; NATS shuts down last |
|
||||
| Ceph | Ordered daemon shutdown: RGW → MDS → OSD → MON | AWSGW → Daemon/VMs → Viperblock → Predastore → NATS |
|
||||
| Kubernetes | Graceful node shutdown (KEP-2000): kubelet uses systemd inhibitor locks, pods sorted by priority class, `terminationGracePeriodSeconds` per pod | Per-VM QMP `system_powerdown` with configurable timeout, force kill after grace period |
|
||||
| Kubernetes | Pod termination lifecycle: preStop hook → SIGTERM → grace period → SIGKILL | Phase-based shutdown with ACK between phases; no phase proceeds until prior phase completes |
|
||||
| Kubernetes | `kubectl drain` cordons a node (marks unschedulable) before evicting pods | GATE phase stops AWSGW to prevent new work before draining VMs |
|
||||
| Rancher/Harvester | Ordered HCI shutdown: cordon node → migrate VMs → drain workloads → power off; fleet-managed lifecycle for multi-cluster | Coordinator-driven phased shutdown across all nodes; progress tracking via NATS |
|
||||
| RKE2 | `rke2-killall.sh` handles both graceful and forceful modes with explicit cleanup of iptables, mounts, and network namespaces | `--force` flag for immediate shutdown; explicit nbdkit/QEMU cleanup in DRAIN phase |
|
||||
| etcd | Learner removal before shutdown; leader transfer to minimize disruption | Coordinator is last to stop NATS; non-coordinator nodes stop NATS first |
|
||||
|
||||
##### Command
|
||||
|
||||
```bash
|
||||
# Graceful shutdown — waits for VMs to power down cleanly
|
||||
./bin/hive admin cluster shutdown
|
||||
|
||||
# Force shutdown — SIGKILL VMs immediately, skip grace periods
|
||||
./bin/hive admin cluster shutdown --force
|
||||
|
||||
# Custom timeout for VM drain phase (default: 120s)
|
||||
./bin/hive admin cluster shutdown --timeout 300s
|
||||
|
||||
# Dry run — show what would happen without executing
|
||||
./bin/hive admin cluster shutdown --dry-run
|
||||
```
|
||||
|
||||
The command can be run from **any node** in the cluster. The issuing node becomes the shutdown coordinator. If the coordinator crashes mid-shutdown, any surviving node can detect the `cluster.shutdown` KV marker and resume coordination (similar to the recovery leader election pattern from 2.3).
|
||||
|
||||
##### Shutdown protocol
|
||||
|
||||
The protocol uses five phases, each gated by acknowledgements from all participating nodes. The coordinator drives transitions and aggregates progress. NATS request-reply ensures reliable delivery with timeouts per phase.
|
||||
|
||||
```
|
||||
Phase 1: GATE ──────► Phase 2: DRAIN ──────► Phase 3: STORAGE ──────► Phase 4: PERSIST ──────► Phase 5: INFRA
|
||||
(stop new work) (stop VMs) (stop block I/O) (stop object storage) (stop NATS)
|
||||
```
|
||||
|
||||
**Phase 1: GATE** — stop accepting new work
|
||||
|
||||
Purpose: Ensure no new API requests enter the system while shutdown proceeds. Analogous to Ceph's `noout`/`norecover` flags and Kubernetes' node cordon.
|
||||
|
||||
1. Coordinator writes `cluster.shutdown` marker to `hive-cluster-state` KV:
|
||||
```
|
||||
Key: cluster.shutdown
|
||||
Value: {"initiator": "node1", "phase": "gate", "started": "2024-...", "timeout": "120s", "force": false}
|
||||
```
|
||||
2. Coordinator publishes `hive.cluster.shutdown.gate` (NATS request, expecting N replies)
|
||||
3. Each node on receiving the gate request:
|
||||
- Stops AWSGW service (no new API requests accepted)
|
||||
- Stops hive-ui service
|
||||
- Sets internal `shuttingDown` flag to reject any in-flight NATS work requests
|
||||
- ACKs to coordinator: `{"node": "node2", "phase": "gate", "stopped": ["awsgw", "ui"]}`
|
||||
4. Coordinator waits for all ACKs (timeout: 30s per node)
|
||||
5. On timeout: log which nodes didn't respond, proceed anyway (they may be already down)
|
||||
|
||||
**Phase 2: DRAIN** — gracefully terminate all VMs
|
||||
|
||||
Purpose: Ensure all VMs are cleanly shut down and their volumes unmounted before storage services stop. This is the critical data-safety phase. Mirrors Kubernetes' pod termination lifecycle: preStop → SIGTERM → grace period → SIGKILL.
|
||||
|
||||
1. Coordinator publishes `hive.cluster.shutdown.drain`
|
||||
2. Each daemon node:
|
||||
- Sends QMP `system_powerdown` to each running QEMU instance (same as `setupShutdown()` today)
|
||||
- Waits for QEMU processes to exit, polling PID files
|
||||
- Reports progress via `hive.cluster.shutdown.progress`:
|
||||
```json
|
||||
{"node": "node2", "phase": "drain", "total": 4, "remaining": 2, "vms": ["i-abc123", "i-def456"]}
|
||||
```
|
||||
- After per-VM grace period (default 60s), sends SIGTERM then SIGKILL to stubborn QEMU processes
|
||||
- Unmounts all EBS volumes via `ebs.unmount` NATS requests (viperblock still running)
|
||||
- Cleans up any orphaned nbdkit processes attached to terminated VMs
|
||||
- Writes instance state to JetStream KV (same as `WriteState()` in `setupShutdown()`)
|
||||
- Writes per-node shutdown marker (reuses Phase 1's `WriteShutdownMarker()`)
|
||||
- ACKs when all VMs have exited and volumes are unmounted
|
||||
3. Coordinator aggregates progress, prints live status
|
||||
4. `--force` mode: skip QMP `system_powerdown`, immediately SIGKILL all QEMU processes
|
||||
5. On timeout: coordinator logs remaining VMs, force-kills them, proceeds
|
||||
|
||||
**Phase 3: STORAGE** — stop block storage
|
||||
|
||||
Purpose: Shut down viperblock and clean up all nbdkit processes. Safe to do now because all VMs are terminated and volumes unmounted.
|
||||
|
||||
1. Coordinator publishes `hive.cluster.shutdown.storage`
|
||||
2. Each node with viperblock service:
|
||||
- Stops viperblock service
|
||||
- Scans for and kills any orphaned nbdkit processes (`pkill -f nbdkit`)
|
||||
- Verifies no NBD block devices remain active
|
||||
- ACKs when complete
|
||||
3. Nodes without viperblock: ACK immediately
|
||||
4. Coordinator waits for all ACKs (timeout: 30s)
|
||||
|
||||
**Phase 4: PERSIST** — stop object storage
|
||||
|
||||
Purpose: Shut down Predastore cleanly. Safe because no viperblock or AWSGW requests remain. Predastore may still have internal replication traffic between nodes — the coordinator ensures all nodes stop together.
|
||||
|
||||
1. Coordinator publishes `hive.cluster.shutdown.persist`
|
||||
2. Each node with predastore service:
|
||||
- Stops predastore service
|
||||
- ACKs when complete
|
||||
3. Coordinator waits for all ACKs (timeout: 30s)
|
||||
|
||||
**Phase 5: INFRA** — stop NATS (coordinator last)
|
||||
|
||||
Purpose: Shut down the coordination layer itself. This is fire-and-forget because the communication channel is being torn down.
|
||||
|
||||
1. Coordinator publishes `hive.cluster.shutdown.infra` to all **non-coordinator** nodes
|
||||
2. Non-coordinator nodes: stop local NATS and exit the shutdown handler
|
||||
3. Coordinator waits 5s for NATS cluster to stabilize with fewer members
|
||||
4. Coordinator stops its own NATS
|
||||
5. Coordinator logs "Cluster shutdown complete" and exits
|
||||
|
||||
##### Coordinator election and failure handling
|
||||
|
||||
The coordinator is simply the node that runs the `hive admin cluster shutdown` command. If the coordinator crashes mid-shutdown:
|
||||
|
||||
1. The `cluster.shutdown` KV marker persists (1-hour TTL, long enough for any shutdown)
|
||||
2. The marker records the current phase and which nodes have ACKed
|
||||
3. Any surviving daemon detects the marker via `kv.Watch("cluster.shutdown")`
|
||||
4. After a coordinator timeout (30s without progress updates), a surviving daemon takes over using the same CAS pattern as recovery leader election (2.3)
|
||||
5. The new coordinator resumes from the last completed phase
|
||||
|
||||
If a node is unreachable during any phase:
|
||||
- Coordinator logs a warning and continues after the per-phase timeout
|
||||
- Unreachable nodes may have already shut down (e.g., power loss)
|
||||
- If they come back later, they'll see the `cluster.shutdown` marker and refuse to start services until the marker is cleared
|
||||
|
||||
##### Status reporting
|
||||
|
||||
The coordinator prints live progress to stdout:
|
||||
|
||||
```
|
||||
Cluster shutdown initiated from node1 (3 nodes)
|
||||
|
||||
Phase 1/5: GATE — Stopping API gateways and UI...
|
||||
node1: awsgw stopped, ui stopped [ok]
|
||||
node2: awsgw stopped, ui stopped [ok]
|
||||
node3: awsgw stopped, ui stopped [ok]
|
||||
|
||||
Phase 2/5: DRAIN — Stopping virtual machines...
|
||||
node1: 4/4 VMs stopped [ok] 12.3s
|
||||
node2: 2/2 VMs stopped [ok] 8.7s
|
||||
node3: 1/1 VMs stopped [ok] 6.1s
|
||||
|
||||
Phase 3/5: STORAGE — Stopping block storage...
|
||||
node1: viperblock stopped, 0 nbdkit cleaned [ok]
|
||||
node2: viperblock stopped, 0 nbdkit cleaned [ok]
|
||||
node3: viperblock stopped, 0 nbdkit cleaned [ok]
|
||||
|
||||
Phase 4/5: PERSIST — Stopping object storage...
|
||||
node1: predastore stopped [ok]
|
||||
node2: predastore stopped [ok]
|
||||
node3: predastore stopped [ok]
|
||||
|
||||
Phase 5/5: INFRA — Stopping NATS...
|
||||
node2: nats stopped
|
||||
node3: nats stopped
|
||||
node1: nats stopped (coordinator, last)
|
||||
|
||||
Cluster shutdown complete (47.3s)
|
||||
```
|
||||
|
||||
##### NATS topics
|
||||
|
||||
| Topic | Pattern | Purpose |
|
||||
|-------|---------|---------|
|
||||
| `hive.cluster.shutdown.gate` | Request-Reply | Phase 1: stop AWSGW + UI, await ACK |
|
||||
| `hive.cluster.shutdown.drain` | Request-Reply | Phase 2: stop VMs, await ACK |
|
||||
| `hive.cluster.shutdown.progress` | Publish (broadcast) | Progress updates during drain (VM counts) |
|
||||
| `hive.cluster.shutdown.storage` | Request-Reply | Phase 3: stop viperblock + nbdkit, await ACK |
|
||||
| `hive.cluster.shutdown.persist` | Request-Reply | Phase 4: stop predastore, await ACK |
|
||||
| `hive.cluster.shutdown.infra` | Publish (fire-and-forget) | Phase 5: stop NATS on non-coordinator nodes |
|
||||
|
||||
##### Cluster shutdown KV state
|
||||
|
||||
```
|
||||
Bucket: hive-cluster-state
|
||||
Key: cluster.shutdown
|
||||
Value: {
|
||||
"initiator": "node1",
|
||||
"phase": "drain",
|
||||
"started": "2024-01-15T10:30:00Z",
|
||||
"timeout": "120s",
|
||||
"force": false,
|
||||
"nodes_total": 3,
|
||||
"nodes_acked": {
|
||||
"node1": {"phase": "gate", "at": "..."},
|
||||
"node2": {"phase": "gate", "at": "..."},
|
||||
"node3": {"phase": "gate", "at": "..."}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
##### Integration with existing code
|
||||
|
||||
- **Reuses `stopInstance()`**: The daemon's existing `stopInstance()` function handles QMP shutdown, PID polling, volume unmount, and force kill. The DRAIN phase calls this same code path rather than reimplementing it.
|
||||
- **Reuses `WriteShutdownMarker()`**: Each node writes its per-node shutdown marker during DRAIN, so on next startup, `restoreInstances()` sees a clean shutdown.
|
||||
- **Reuses `./bin/hive service <name> stop`**: For stopping AWSGW, UI, viperblock, predastore, and NATS, the coordinator tells each daemon to invoke the same service stop mechanism used by `stop-dev.sh`.
|
||||
- **`setupShutdown()` becomes cluster-aware**: When a daemon receives a cluster shutdown request, it skips independent shutdown and instead participates in the coordinated protocol. If a SIGTERM arrives while a coordinated shutdown is in progress, the daemon ACKs the current phase and exits.
|
||||
- **`stop-dev.sh` remains for single-node dev**: The script continues to work for local development. For multi-node clusters, `hive admin cluster shutdown` is the preferred approach.
|
||||
|
||||
##### Comparison with per-node `stop-dev.sh`
|
||||
|
||||
| Aspect | `stop-dev.sh` (current) | `hive admin cluster shutdown` (proposed) |
|
||||
|--------|------------------------|------------------------------------------|
|
||||
| Scope | Single node | Entire cluster from any node |
|
||||
| Ordering | Per-node reverse order | Cluster-wide phased ordering |
|
||||
| VM safety | Daemon exits, then waits for QEMU via `pgrep` | Daemon actively monitors each VM's QMP shutdown |
|
||||
| Storage safety | Hopes VMs exited before predastore stop | Guarantees all VMs stopped before storage phases |
|
||||
| nbdkit cleanup | `pgrep`/`pkill` after the fact | Explicit unmount + cleanup during DRAIN |
|
||||
| Progress | Per-service echo messages | Cluster-wide live progress with VM counts |
|
||||
| Failure handling | None — operator must check each node | Automatic coordinator failover, timeout handling |
|
||||
| NATS safety | Stops NATS whenever it gets there | NATS is always last, coordinator node last of all |
|
||||
|
||||
**Files:**
|
||||
- `hive/daemon/daemon.go` — shutdown handler that participates in coordinated protocol, `shuttingDown` flag to reject new work
|
||||
- `hive/daemon/shutdown.go` — **new file**, coordinator logic: phase management, ACK aggregation, progress reporting, coordinator failover
|
||||
- `hive/daemon/jetstream.go` — `WriteClusterShutdown()` / `ReadClusterShutdown()` / `DeleteClusterShutdown()` / `WatchClusterShutdown()`
|
||||
- `cmd/hive/cmd/admin.go` — new `cluster shutdown` subcommand with `--force`, `--timeout`, `--dry-run` flags
|
||||
- `hive/daemon/daemon_handlers.go` — NATS handlers for `hive.cluster.shutdown.*` topics
|
||||
|
||||
### Phase 3: Operational tooling
|
||||
|
||||
#### 3.1 Maintenance mode flag
|
||||
|
||||
Add a `maintenance.{nodeID}` KV key that daemons can set before planned restarts:
|
||||
|
||||
```bash
|
||||
./bin/hive admin maintenance enable --node node1
|
||||
```
|
||||
|
||||
When maintenance mode is set for a node:
|
||||
- Peer daemons skip the 5-minute failure detection grace period for that node (they know it's planned)
|
||||
- The node itself skips VM recovery on restart if the maintenance flag is still set (operator controls when VMs come back)
|
||||
- Analogous to Ceph's `noout` flag
|
||||
- Works with coordinated shutdown: the coordinator sets maintenance mode on all nodes before Phase 1, clears it after Phase 5
|
||||
|
||||
**Files:**
|
||||
- `hive/daemon/jetstream.go` — new `SetMaintenanceMode()` / `ClearMaintenanceMode()` / `IsMaintenanceMode()`
|
||||
- `cmd/hive/cmd/admin.go` — new `maintenance` subcommand
|
||||
|
||||
#### 3.2 Recovery status visibility
|
||||
|
||||
Add admin endpoints and NATS subjects for recovery observability:
|
||||
|
||||
- `GET /admin/recovery/status` — shows recovery state, assignments, peer health
|
||||
- `GET /admin/cluster/topology` — shows all nodes, their capabilities, and service health
|
||||
- `hive.admin.recovery.status` NATS subject — cluster-wide recovery state query
|
||||
|
||||
**Files:**
|
||||
- `hive/daemon/daemon.go` — new cluster manager routes
|
||||
|
||||
#### 3.3 Admin cluster status command
|
||||
|
||||
Add a CLI command for operators to inspect the cluster topology:
|
||||
|
||||
```bash
|
||||
$ ./bin/hive admin status
|
||||
Cluster: hive (epoch 3)
|
||||
|
||||
NODE SERVICES STATUS VMS vCPU(free) MEM(free)
|
||||
node1 nats,predastore,viperblock,daemon,awsgw,ui healthy 4 8/16 12.0/32.0
|
||||
node2 nats,predastore,viperblock,daemon,awsgw healthy 3 10/16 16.0/32.0
|
||||
node3 nats,predastore,viperblock,daemon healthy 2 12/16 20.0/32.0
|
||||
node4 viperblock,daemon healthy 6 2/8 4.0/16.0
|
||||
```
|
||||
|
||||
This reads from the `hive-cluster-state` KV bucket (heartbeats + service manifests) and fans out health checks via NATS.
|
||||
|
||||
**Files:**
|
||||
- `cmd/hive/cmd/admin.go` — new `status` subcommand
|
||||
|
||||
## Files Modified (Phase 0 + 1)
|
||||
|
||||
| File | Changes | Phase |
|
||||
|------|---------|-------|
|
||||
| `hive/config/config.go` | Added `Services` field, `AllServices`, `HasService()`, `GetServices()`, updated `NodeHealthResponse` with `Services` and `ServiceHealth` | 0.1, 0.5 |
|
||||
| `hive/config/config_test.go` | 5 new tests for `HasService` and `GetServices` | 0.1 |
|
||||
| `hive/admin/admin.go` | Added `Services []string` to `ConfigSettings` | 0.3 |
|
||||
| `cmd/hive/cmd/templates/hive.toml` | Added `services` template rendering | 0.3 |
|
||||
| `cmd/hive/cmd/admin.go` | Added `--services` flag to init/join, wired to `ConfigSettings` | 0.3 |
|
||||
| `hive/formation/formation.go` | Added `Services` to `NodeInfo` | 0.2 |
|
||||
| `hive/formation/helpers.go` | Added `hasService()`, filtered `BuildClusterRoutes()` by nats, `BuildPredastoreNodes()` by predastore | 0.2 |
|
||||
| `hive/formation/helpers_test.go` | 7 new tests for capability-filtered routing and predastore node building | 0.2 |
|
||||
| `scripts/start-dev.sh` | Added `parse_services()`, `has_service()`, conditional service startup | 0.4 |
|
||||
| `scripts/stop-dev.sh` | Added `parse_services()`, `has_service()` (with name mapping), conditional stop | 0.4 |
|
||||
| `hive/daemon/daemon.go` | Updated health endpoint, `waitForClusterReady()`, `checkViperblockReady()`, `checkPredastoreReady()`, refactored `restoreInstances()` with semaphore, shutdown marker in `setupShutdown()`, service manifest write in `Start()` | 0.5, 1.1–1.4 |
|
||||
| `hive/daemon/jetstream.go` | `ClusterStateBucket`, `clusterKV` field, `InitClusterStateBucket()`, `WriteShutdownMarker()`, `ReadShutdownMarker()`, `DeleteShutdownMarker()`, `WriteServiceManifest()`, updated `UpdateReplicas()` | 0.5, 1.3, 1.4 |
|
||||
|
||||
## Files to Modify (Phase 2 + 3)
|
||||
|
||||
| File | Changes | Phase |
|
||||
|------|---------|-------|
|
||||
| `hive/daemon/jetstream.go` | Heartbeat, recovery lease, cluster shutdown KV, maintenance mode | 2.1–2.4, 3.1 |
|
||||
| `hive/daemon/recovery.go` | **New file** — recovery coordinator: leader election, capability-aware VM assignment, peer monitoring | 2.3 |
|
||||
| `hive/daemon/shutdown.go` | **New file** — coordinated shutdown coordinator: phase management, ACK aggregation, progress reporting, failover | 2.4 |
|
||||
| `hive/daemon/daemon.go` | Heartbeat/recovery integration, cluster shutdown handler, `shuttingDown` flag | 2.1–2.4 |
|
||||
| `hive/daemon/daemon_handlers.go` | NATS handlers for `hive.cluster.shutdown.*` topics | 2.4 |
|
||||
| `cmd/hive/cmd/admin.go` | `cluster shutdown`, `maintenance`, `status` subcommands | 2.4, 3.1, 3.3 |
|
||||
|
||||
## Implementation Order
|
||||
|
||||
Phase 0 is the prerequisite — all subsequent phases depend on nodes knowing their capabilities. Phase 1 fixes immediate restart bugs. Phases 2-3 add cross-node resilience and tooling.
|
||||
|
||||
**Phase 0** — node capabilities (foundation) — **Complete**:
|
||||
1. [x] `Services` field in Config + helpers
|
||||
2. [x] Formation carries capabilities
|
||||
3. [x] Config template and CLI flags
|
||||
4. [x] Capability-aware start/stop scripts
|
||||
5. [x] Health endpoint includes capabilities + cluster state KV bucket + service manifest
|
||||
|
||||
**Phase 1** — self-recovery (fixes immediate restart bugs) — **Complete**:
|
||||
6. [x] Cluster readiness gate
|
||||
7. [x] Recovery throttling (semaphore in `restoreInstances`)
|
||||
8. [x] Clean shutdown marker
|
||||
9. [x] Service manifest on startup
|
||||
|
||||
**Phase 2** — cross-node coordination — **Planned**:
|
||||
10. Heartbeat publishing (with capabilities)
|
||||
11. Peer failure detection
|
||||
12. Leader-elected recovery coordinator (capability-aware placement)
|
||||
13. Coordinated cluster shutdown (`hive admin cluster shutdown`)
|
||||
|
||||
**Phase 3** — operational tooling — **Planned**:
|
||||
14. Maintenance mode flag
|
||||
15. Recovery status endpoints
|
||||
16. Admin cluster status command
|
||||
|
||||
## Testing
|
||||
|
||||
### Phase 0 + 1 verification — Passed
|
||||
|
||||
| Check | Result |
|
||||
|-------|--------|
|
||||
| `make test` (all unit tests) | Passed |
|
||||
| `make preflight` (format + vet + security + tests) | Passed |
|
||||
| `make test-docker-single` (single-node E2E) | Passed — all 9 phases |
|
||||
| `make test-docker-multi` (multi-node E2E, 3 nodes) | Passed — all phases including cluster health, cross-node operations |
|
||||
| Backward compatibility (no `services` field in config) | Verified — defaults to all services |
|
||||
|
||||
**New unit tests added:**
|
||||
- `hive/config/config_test.go` — `HasService` (explicit, empty/default, unknown), `GetServices` (default, explicit)
|
||||
- `hive/formation/helpers_test.go` — `hasService` (empty=all, explicit), `BuildClusterRoutes` (filters by nats, empty=all, mixed), `BuildPredastoreNodes` (filters by predastore, empty=all)
|
||||
|
||||
**Bug found during E2E**: `checkViperblockReady()` initially sent a NATS request to `ebs.{node}.health`, but viperblock does not subscribe to any health NATS topic. This caused the daemon to block in `waitForClusterReady()` for the full 2-minute timeout on every startup. Fixed by checking `natsConn.IsConnected()` instead.
|
||||
|
||||
### Phase 2 + 3 tests (planned)
|
||||
|
||||
#### Unit tests
|
||||
- Recovery coordinator: test leader election, capability-filtered VM assignment, epoch-based fencing
|
||||
- Heartbeat: test staleness detection thresholds
|
||||
- Coordinated shutdown: test phase transitions, ACK aggregation, timeout handling, coordinator failover
|
||||
- Shutdown protocol: test `cluster.shutdown` KV marker lifecycle (write, read, resume, clear)
|
||||
|
||||
#### Integration tests (multi-node Docker)
|
||||
- Start 3-node cluster, launch 4 VMs, stop all, restart — verify all 4 VMs recover exactly once
|
||||
- Start 3+1 cluster (3 full + 1 compute-only), verify compute-only connects to remote NATS/Predastore
|
||||
- Kill one node (SIGKILL), verify surviving nodes recover its VMs after grace period
|
||||
- Rolling restart: stop/start nodes one at a time, verify zero duplicate launches
|
||||
- Coordinated shutdown: launch VMs across 3 nodes, run `hive admin cluster shutdown` from node2, verify phase ordering (AWSGW stops before VMs, VMs stop before viperblock, viperblock before predastore, NATS last)
|
||||
- Coordinated shutdown with `--force`: verify VMs are killed immediately, no grace period
|
||||
- Shutdown coordinator failure: start cluster shutdown from node1, kill node1 mid-drain, verify another node resumes coordination
|
||||
- Post-shutdown restart: after coordinated shutdown, restart all nodes, verify clean shutdown markers present and VMs restore correctly
|
||||
- No orphaned processes: after coordinated shutdown, verify zero remaining QEMU/nbdkit processes on all nodes
|
||||
|
||||
#### Manual verification
|
||||
- Deploy to 3-node test cluster (10.1.3.170-172), run stop/start cycle, check logs for race conditions
|
||||
- Add a 4th compute-only node, verify it joins and accepts workloads
|
||||
- Run `hive admin cluster shutdown` from each of the 3 nodes in turn, verify it works from any node
|
||||
- Run `hive admin cluster shutdown --dry-run` and verify output matches expected phase plan
|
||||
|
||||
## Future Work
|
||||
|
||||
- **Coordinated cluster startup** (`hive admin cluster start`): Complement to `cluster shutdown`. Coordinator starts services across all nodes in dependency order (NATS → Predastore → Viperblock → Daemon → AWSGW → UI), with health checks between phases. Would replace per-node `start-dev.sh` for multi-node clusters.
|
||||
- **Persistent VM pinning**: Optional affinity rules to prefer restarting VMs on their original node
|
||||
- **Live migration**: Move running VMs between nodes without stopping them (requires QEMU live migration support). Would enable `cluster shutdown` to migrate VMs to surviving nodes instead of stopping them.
|
||||
- **Automatic scaling**: Detect sustained capacity pressure and suggest adding nodes
|
||||
- **Recovery metrics**: Prometheus-compatible metrics for recovery duration, failures, leader elections, shutdown phase timings
|
||||
- **Dynamic service reconfiguration**: Add/remove services from a running node without full restart
|
||||
- **Storage-only node health**: Extend recovery to detect failed storage nodes and reroute Viperblock/Predastore traffic
|
||||
- **Rolling cluster upgrade**: Coordinated restart that drains nodes one at a time, upgrades binaries, and restarts — maintaining cluster availability throughout
|
||||
@@ -1,675 +0,0 @@
|
||||
# Multi-Server Cluster Formation
|
||||
|
||||
## Problem
|
||||
|
||||
The current multi-server configuration has a fundamental design flaw: cluster formation and cluster operation are entangled. The `/join` endpoint lives inside the Hive daemon, which requires NATS to be running. This creates a chicken-and-egg problem — nodes cannot join until services are running, but services cannot be correctly configured until all nodes have joined.
|
||||
|
||||
### Current bugs
|
||||
|
||||
1. **NATS config missing all cluster routes** — each node's `nats.conf` only contains the leader's route, not routes to all other nodes. NATS cannot form a full mesh cluster.
|
||||
2. **NATS cluster name hardcoded to `C1`** — should be configurable per-cluster.
|
||||
3. **Predastore "Distributed mode requires -node flag"** — NODE_ID detection fails in multi-node setups.
|
||||
4. **Predastore "no known peers" on joining nodes** — Raft starts with `servers=[]`, cannot form consensus.
|
||||
5. **Sequence dependency** — node 1 must run `./scripts/start-dev.sh` before node 2 can `hive admin join`, because the `/join` endpoint is inside the daemon.
|
||||
6. **Certs not installed to system CA store** — joining nodes receive the CA cert but don't install it, causing TLS failures.
|
||||
|
||||
### Root cause
|
||||
|
||||
All bugs trace back to one issue: configs are generated per-node during init/join with incomplete information (only the leader IP is known). The correct approach is to generate configs AFTER all nodes have registered, using complete cluster information.
|
||||
|
||||
## Design
|
||||
|
||||
Separate cluster **formation** (lightweight HTTP) from cluster **operation** (NATS, Predastore, etc).
|
||||
|
||||
### Formation flow
|
||||
|
||||
```
|
||||
┌─────────────────────────────────┐
|
||||
│ hive admin init --nodes 3 │
|
||||
│ Starts formation HTTP server │
|
||||
│ Registers self as node 1 │
|
||||
│ Waits for 2 more nodes... │
|
||||
└──────────┬──────────────────────┘
|
||||
│ :4432
|
||||
┌────────────────┼────────────────┐
|
||||
│ │ │
|
||||
┌─────────▼──────┐ ┌─────▼────────┐ ┌───▼──────────┐
|
||||
│ hive admin join │ │ (waiting) │ │ (waiting) │
|
||||
│ POST /join │ │ │ │ │
|
||||
│ polls /status │ │ │ │ │
|
||||
└────────────────┘ └──────────────┘ └──────────────┘
|
||||
│ │ │
|
||||
└────────────────┼────────────────┘
|
||||
│ All 3 nodes joined
|
||||
┌──────────▼──────────────────────┐
|
||||
│ Formation complete │
|
||||
│ Generate configs with ALL IPs │
|
||||
│ Distribute CA, credentials │
|
||||
│ All processes exit │
|
||||
└──────────┬──────────────────────┘
|
||||
│
|
||||
┌────────────────┼────────────────┐
|
||||
│ │ │
|
||||
┌─────────▼──────┐ ┌─────▼────────┐ ┌───▼──────────┐
|
||||
│ start-dev.sh │ │ start-dev.sh │ │ start-dev.sh │
|
||||
│ Node 1 │ │ Node 2 │ │ Node 3 │
|
||||
└────────────────┘ └──────────────┘ └──────────────┘
|
||||
```
|
||||
|
||||
### Phase 1: Formation (init + join)
|
||||
|
||||
**Node 1** starts a lightweight HTTP formation server (no NATS dependency):
|
||||
|
||||
```sh
|
||||
./bin/hive admin init \
|
||||
--node node1 \
|
||||
--nodes 3 \
|
||||
--bind 10.11.12.1 \
|
||||
--cluster-bind 10.11.12.1 \
|
||||
--port 4432 \
|
||||
--hive-dir ~/hive/ \
|
||||
--config-dir ~/hive/config/ \
|
||||
--region ap-southeast-2 \
|
||||
--az ap-southeast-2a
|
||||
```
|
||||
|
||||
This generates credentials (AWS keys, NATS token, CA certs) and starts the formation HTTP server on `10.11.12.1:4432`. It registers itself as the first node and waits for 2 more.
|
||||
|
||||
**Node 2 and Node 3** join while init is running:
|
||||
|
||||
```sh
|
||||
./bin/hive admin join \
|
||||
--node node2 \
|
||||
--bind 10.11.12.2 \
|
||||
--cluster-bind 10.11.12.2 \
|
||||
--host 10.11.12.1:4432 \
|
||||
--data-dir ~/hive/ \
|
||||
--config-dir ~/hive/config/ \
|
||||
--region ap-southeast-2 \
|
||||
--az ap-southeast-2a
|
||||
```
|
||||
|
||||
Each joining node POSTs to the formation server, then polls for completion.
|
||||
|
||||
Once all 3 nodes register, the formation server:
|
||||
1. Computes NATS cluster routes from ALL node IPs (full mesh)
|
||||
2. Builds predastore node configs from ALL node IPs (with correct node IDs)
|
||||
3. Returns complete config package (credentials, CA, node list) to each polling node
|
||||
4. Each node generates its own configs locally with complete cluster data
|
||||
5. CA cert is installed to system trust store on each node
|
||||
6. All processes (init + joins) display cluster summary and exit
|
||||
|
||||
**Key improvements over current design:**
|
||||
- `--predastore-nodes` is now **optional** — derived automatically from joined nodes
|
||||
- `--cluster-routes` on join is now **optional** — derived from formation data
|
||||
- No sequence dependency — init starts the formation server directly (no NATS needed)
|
||||
- All configs have complete cluster data (all IPs, all routes)
|
||||
|
||||
### Phase 2: Service startup
|
||||
|
||||
After formation completes, run on each node:
|
||||
|
||||
```sh
|
||||
./scripts/start-dev.sh
|
||||
```
|
||||
|
||||
Services start in order: NATS → Predastore → Viperblock → Daemon → AWS Gateway → UI.
|
||||
|
||||
Since all configs already have the full cluster topology, NATS forms a mesh immediately and Predastore Raft finds all peers.
|
||||
|
||||
## Network recommendations
|
||||
|
||||
It is recommended that at least two network links are used by Hive:
|
||||
|
||||
1. **Management link** — cluster sync, NATS, Predastore Raft, and daemon communication. This requires low latency (< 5ms) and dedicated bandwidth.
|
||||
2. **Data/VM link** — EC2 instances, VPC traffic (Geneve/Open vSwitch), and user-facing services. Isolated from management to prevent contention.
|
||||
|
||||
This follows the same model as Proxmox VE, which recommends a physically separated NIC for cluster (corosync) traffic: "The network should not be used heavily by other members, as while corosync does not use much bandwidth it is sensitive to latency jitters."
|
||||
|
||||
## Proxmox Cluster Manager comparison
|
||||
|
||||
Reviewed [Proxmox Cluster Manager](https://pve.proxmox.com/wiki/Cluster_Manager) for best practices.
|
||||
|
||||
**What we adopt from Proxmox:**
|
||||
- Formation-first workflow (`pvecm create` / `pvecm add` completes before services use the cluster)
|
||||
- Shared CA model (joining node's cert is replaced with one signed by cluster CA)
|
||||
- Dedicated cluster network recommendation (separate NIC for management traffic)
|
||||
- Hostname/IP finalized before formation (cannot change post-creation)
|
||||
|
||||
**Why native Go over Corosync:**
|
||||
|
||||
Corosync is a C group communication library providing reliable messaging, quorum, and membership. Hive already has equivalent capabilities:
|
||||
|
||||
| Capability | Corosync | Hive |
|
||||
|---|---|---|
|
||||
| Message bus | Totem protocol | NATS |
|
||||
| State replication | pmxcfs (replicated filesystem) | NATS JetStream KV |
|
||||
| Consensus/quorum | Corosync votequorum | Predastore Raft |
|
||||
| Config distribution | pmxcfs | Formation server + hive.toml |
|
||||
|
||||
Adding Corosync would be a redundant external C dependency providing overlapping functionality. The native Go implementation gives us:
|
||||
- Single binary deployment (no `corosync` daemon to install/manage)
|
||||
- Direct integration with NATS and Predastore
|
||||
- Full control over the join protocol
|
||||
- No CGo or external library dependency
|
||||
|
||||
## Implementation
|
||||
|
||||
### New package: `hive/formation/`
|
||||
|
||||
Lightweight HTTP formation server using `net/http`. Zero dependency on NATS or any operational service.
|
||||
|
||||
**`formation.go`** — core server:
|
||||
- `FormationServer` struct: tracks expected nodes, joined nodes, shared credentials, CA cert/key
|
||||
- `POST /formation/join` — register a node, return progress
|
||||
- `GET /formation/status` — poll completion; returns full config data when all nodes joined
|
||||
- `GET /formation/health` — liveness check
|
||||
- `WaitForCompletion()` — blocks until all nodes join or timeout (default 10 minutes)
|
||||
|
||||
**`helpers.go`** — config builders:
|
||||
- `BuildClusterRoutes(nodes)` — returns all node cluster-bind IPs as `IP:4248` routes
|
||||
- `BuildPredastoreNodes(nodes)` — returns `[]PredastoreNodeConfig` with 1-based IDs
|
||||
|
||||
### Changes to `cmd/hive/cmd/admin.go`
|
||||
|
||||
**`runAdminInit`:**
|
||||
- `--nodes 1` (or no cluster flags): existing single-node behaviour, unchanged
|
||||
- `--nodes >= 2`: start formation server, register self, wait for joins, generate configs with complete data
|
||||
- `--predastore-nodes` becomes optional (derived from formation when not specified)
|
||||
- New flag: `--formation-timeout` (default "10m")
|
||||
|
||||
**`runAdminJoin`:**
|
||||
- POST to `http://leader:port/formation/join` (not the daemon `/join`)
|
||||
- Poll `GET /formation/status` every 2s until complete
|
||||
- Extract full node list, CA, credentials from response
|
||||
- Generate all configs locally with complete cluster data
|
||||
- Install CA to system trust store
|
||||
|
||||
### Changes to templates
|
||||
|
||||
**`templates/nats.conf`:** Change `name: C1` to `name: {{ .ClusterName }}`
|
||||
|
||||
**`hive/admin/admin.go`:** Add `ClusterName string` to `ConfigSettings`
|
||||
|
||||
### Changes to `scripts/start-dev.sh`
|
||||
|
||||
- Remove awk-based NODE_ID auto-detection (formation now writes correct node_id into hive.toml)
|
||||
- Keep `HIVE_PREDASTORE_NODE_ID` env var override for manual use
|
||||
|
||||
### Daemon `/join` endpoint (no changes)
|
||||
|
||||
The existing `/join` endpoint in `hive/daemon/daemon.go` stays as-is for future "hot-add node to running cluster" scenarios. Formation handles initial bootstrap; daemon handles runtime expansion.
|
||||
|
||||
### Graceful cluster shutdown
|
||||
|
||||
The current `stop-dev.sh` sends SIGTERM to each service sequentially with no cluster coordination. In a multi-node cluster this causes cascading failures — if the Raft leader is stopped first, remaining predastore nodes enter an election storm trying to contact dead peers, goroutines block, and the cluster hangs instead of shutting down cleanly.
|
||||
|
||||
#### Two shutdown modes
|
||||
|
||||
**1. Cluster-wide shutdown** (`hive admin shutdown`):
|
||||
|
||||
Coordinated shutdown of the entire cluster. The initiating node broadcasts a shutdown intent, all nodes agree, and services stop in a safe order across the cluster.
|
||||
|
||||
```sh
|
||||
# On any node — broadcasts to entire cluster
|
||||
./bin/hive admin shutdown
|
||||
```
|
||||
|
||||
**2. Single-node shutdown** (`stop-dev.sh` or `hive admin stop`):
|
||||
|
||||
Stops services on one node only. The node announces its departure so peers can adjust (e.g., Raft leader transfer) rather than treating it as a crash.
|
||||
|
||||
```sh
|
||||
# On the node being stopped
|
||||
./scripts/stop-dev.sh
|
||||
```
|
||||
|
||||
#### Cluster-wide shutdown flow
|
||||
|
||||
```
|
||||
Operator runs: hive admin shutdown (on any node)
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────┐
|
||||
│ 1. Connect to local daemon │
|
||||
│ 2. POST /cluster/shutdown │
|
||||
│ 3. Daemon broadcasts │
|
||||
│ hive.cluster.shutdown via NATS │
|
||||
└──────────────┬───────────────────┘
|
||||
│
|
||||
┌──────────┼──────────┐
|
||||
▼ ▼ ▼
|
||||
Node 1 Node 2 Node 3
|
||||
│ │ │
|
||||
│ All nodes enter "shutting_down" state
|
||||
│ Stop accepting new work (API returns 503)
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
Phase 1: Stop VMs (system_powerdown, wait, write state to JetStream)
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
Phase 2: Stop daemon + gateway (unsubscribe NATS, close connections)
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
Phase 3: Stop storage (predastore Raft shutdown, viperblock)
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
Phase 4: Stop NATS (last — needed for coordination until this point)
|
||||
```
|
||||
|
||||
#### Shutdown protocol
|
||||
|
||||
1. **Initiation** — any node's daemon receives `POST /cluster/shutdown` (from `hive admin shutdown` CLI)
|
||||
2. **Broadcast** — daemon publishes `hive.cluster.shutdown` on NATS with initiator node ID and epoch
|
||||
3. **Acknowledgement** — each daemon responds on `hive.cluster.shutdown.ack` with its node ID. Initiator waits up to 10s for all known nodes to ACK (proceeds anyway if some nodes are unreachable)
|
||||
4. **Phase 1: VM drain** — all daemons stop accepting new `ec2.RunInstances` requests, send `system_powerdown` to running VMs, write instance state to JetStream, wait up to 60s for VMs to power off
|
||||
5. **Phase 2: Daemon stop** — daemons unsubscribe from all NATS topics, shut down the cluster HTTP server and AWS Gateway
|
||||
6. **Phase 3: Storage stop** — predastore Raft leader (if local) calls `raft.Shutdown()` first, then followers shut down Raft; viperblock stops. Because all nodes are shutting down simultaneously, there's no election storm — every node's Raft transport is closed before heartbeat timeouts fire
|
||||
7. **Phase 4: NATS stop** — NATS server is the last to stop (it was needed for the shutdown broadcast and ACKs)
|
||||
8. **Cleanup** — each node's `stop-dev.sh` waits for all service PIDs to exit, force-kills any stragglers after 120s
|
||||
|
||||
#### Predastore Raft shutdown handling
|
||||
|
||||
The core problem: when a Raft leader disappears without warning, followers detect missing heartbeats and start elections. With 2-of-3 nodes gone, elections can never succeed and goroutines spin indefinitely.
|
||||
|
||||
**Solution for cluster-wide shutdown:**
|
||||
- All predastore nodes receive the shutdown broadcast via NATS (Phase 3)
|
||||
- Each node closes the Raft transport first (`transport.Close()`), which prevents any new RPCs
|
||||
- Then calls `raft.Shutdown()` which drains in-flight operations
|
||||
- Because all nodes close transport near-simultaneously, no node has time to start an election
|
||||
- The 5-second Raft shutdown timeout in predastore (`s3db/raft.go`) is sufficient since no elections are triggered
|
||||
|
||||
**Solution for single-node shutdown:**
|
||||
- The departing node publishes `hive.cluster.node-leaving` with its node ID
|
||||
- If the departing node is the Raft leader, it calls `raft.LeadershipTransfer()` before shutting down — this hands leadership to a follower cleanly
|
||||
- If it's a follower, it calls `raft.Shutdown()` directly — the leader and remaining followers continue normally
|
||||
- Remaining nodes log the departure and continue operating (Raft maintains quorum with N-1 nodes if majority survives)
|
||||
|
||||
#### Single-node stop flow
|
||||
|
||||
When `stop-dev.sh` runs on one node in a multi-node cluster:
|
||||
|
||||
1. Daemon receives SIGTERM
|
||||
2. Daemon publishes `hive.cluster.node-leaving` with its node ID on NATS
|
||||
3. If this node is the predastore Raft leader: call `raft.LeadershipTransfer()`, wait up to 5s
|
||||
4. Stop VMs: `system_powerdown` → wait → write state to JetStream
|
||||
5. Unsubscribe from NATS topics, shut down cluster HTTP server
|
||||
6. Stop predastore (Raft transport close, then `raft.Shutdown()`)
|
||||
7. Stop viperblock
|
||||
8. Close NATS connection
|
||||
9. Stop NATS server
|
||||
|
||||
The remaining cluster nodes see the `node-leaving` message and:
|
||||
- Remove the node from active heartbeat tracking (no false alarms)
|
||||
- Predastore Raft has already transferred leadership if needed
|
||||
- AWS Gateway's next `hive.nodes.discover` broadcast gets fewer responses (expected)
|
||||
- No config changes — the node is still in `hive.toml` but simply offline
|
||||
|
||||
#### Changes to `scripts/stop-dev.sh`
|
||||
|
||||
- Before stopping services, check if this is a multi-node cluster (read `hive.toml` node count)
|
||||
- If multi-node: send `hive.cluster.node-leaving` via the daemon before stopping it
|
||||
- If daemon is unreachable (already crashed): proceed with direct SIGTERM as today (best effort)
|
||||
- Keep the existing service stop order but add a brief pause after daemon stop to allow the `node-leaving` broadcast to propagate
|
||||
|
||||
#### New CLI command: `hive admin shutdown`
|
||||
|
||||
```sh
|
||||
./bin/hive admin shutdown [--force] [--timeout 120s]
|
||||
```
|
||||
|
||||
- Connects to local daemon's cluster HTTP endpoint
|
||||
- `POST /cluster/shutdown` triggers the coordinated shutdown flow
|
||||
- `--force` skips VM drain and ACK wait (immediate SIGTERM to all services)
|
||||
- `--timeout` controls max wait for VM powerdown (default 120s)
|
||||
- Prints progress: "Waiting for ACKs... 3/3", "Draining VMs... 2 remaining", "Stopping storage...", "Shutdown complete"
|
||||
|
||||
## Files
|
||||
|
||||
| File | Action |
|
||||
|---|---|
|
||||
| `hive/formation/formation.go` | New — formation HTTP server |
|
||||
| `hive/formation/helpers.go` | New — route and node config builders |
|
||||
| `hive/formation/formation_test.go` | New — tests |
|
||||
| `hive/admin/admin.go` | Modify — add ClusterName to ConfigSettings |
|
||||
| `cmd/hive/cmd/admin.go` | Modify — rewrite init multi-node path + join, add `shutdown` subcommand |
|
||||
| `cmd/hive/cmd/templates/nats.conf` | Modify — C1 → {{ .ClusterName }} |
|
||||
| `scripts/start-dev.sh` | Modify — simplify NODE_ID detection |
|
||||
| `scripts/stop-dev.sh` | Modify — check cluster mode, send `node-leaving` before stopping services |
|
||||
| `hive/daemon/daemon.go` | Modify — add shutdown broadcast handler, `node-leaving` publish on SIGTERM, Raft leader transfer coordination |
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit tests
|
||||
|
||||
- Formation server: all nodes join, duplicate rejection, timeout, status polling
|
||||
- Helper functions: route building, predastore node building
|
||||
- NATS template: renders cluster name and all routes correctly
|
||||
- Shutdown broadcast: all nodes ACK, timeout when node unreachable, phase ordering
|
||||
- Node-leaving: Raft leader transfer triggered when departing node is leader
|
||||
|
||||
### Integration test
|
||||
|
||||
Note: `--config-dir ~/$NODE/config/` needs to be specified for each node, so not to conflict when testing locally to simulate 3 nodes.
|
||||
|
||||
1. `hive admin init --nodes 3 --bind 127.0.0.1 --node node1` in terminal 1
|
||||
2. `hive admin join --node node2 --host 127.0.0.1:4432 --bind 127.0.0.2` in terminal 2
|
||||
3. `hive admin join --node node3 --host 127.0.0.1:4432 --bind 127.0.0.3` in terminal 3
|
||||
4. Verify: all 3 exit with cluster summary
|
||||
5. Verify: each `nats.conf` has routes to ALL other nodes
|
||||
6. Verify: each `predastore.toml` has all 3 nodes
|
||||
7. Verify: each `hive.toml` has all 3 nodes in `[nodes.*]`
|
||||
8. Run `./scripts/start-dev.sh` on all 3, verify NATS mesh and Predastore Raft form correctly
|
||||
|
||||
### Shutdown integration test
|
||||
|
||||
1. Start 3-node cluster (from formation integration test above)
|
||||
2. Run `hive admin shutdown` on node1
|
||||
3. Verify: all 3 nodes receive shutdown broadcast (check logs for ACK)
|
||||
4. Verify: all predastore Raft nodes shut down cleanly — no election storms in logs
|
||||
5. Verify: all NATS servers stop last
|
||||
6. Verify: all service PIDs are gone on all 3 nodes
|
||||
|
||||
### Single-node stop integration test
|
||||
|
||||
1. Start 3-node cluster
|
||||
2. Run `./scripts/stop-dev.sh` on node3 (a Raft follower)
|
||||
3. Verify: node3 publishes `node-leaving`, services stop cleanly
|
||||
4. Verify: nodes 1 and 2 continue operating, Predastore Raft maintains quorum (2/3)
|
||||
5. Run `./scripts/stop-dev.sh` on node1 (if node1 is Raft leader)
|
||||
6. Verify: Raft leadership transfers to node2 before node1's predastore stops
|
||||
7. Verify: node2 continues operating as single-node (degraded but functional)
|
||||
|
||||
---
|
||||
|
||||
## Stage 2: Runtime Node Expansion
|
||||
|
||||
### Overview
|
||||
|
||||
After the initial cluster is formed (Stage 1) and services are running, operators need the ability to add new nodes without stopping the cluster. This covers "day 2" operations — e.g., a 3-node cluster formed on Monday, with a 4th node added on Tuesday.
|
||||
|
||||
The daemon takes over formation duties once the cluster is running. The existing `/join` endpoint in `hive/daemon/daemon.go` is replaced by the formation server operating in "Stage 2 mode" — same formation package, different context.
|
||||
|
||||
### Key difference from Stage 1
|
||||
|
||||
Stage 1 formation happens BEFORE any services start — it's a clean-room config generation step where all processes exit after writing configs. Stage 2 happens WHILE services are running, so each service must be notified of topology changes and reconfigure itself live.
|
||||
|
||||
### Expansion flow
|
||||
|
||||
```
|
||||
Existing running cluster (3 nodes) New node
|
||||
┌──────────┐ ┌──────────┐ ┌──────────┐
|
||||
│ Node 1 │ │ Node 2 │ │ Node 3 │
|
||||
│ daemon │ │ daemon │ │ daemon │
|
||||
│ :4432 │ │ :4432 │ │ :4432 │
|
||||
└─────┬─────┘ └─────┬─────┘ └─────┬─────┘
|
||||
│ NATS mesh │ │
|
||||
└───────┬───────┘ │
|
||||
│ │ ┌─────────────────┐
|
||||
│ │ │ hive admin join │
|
||||
│ │ │ --host node1 │
|
||||
│ │ │ --node node4 │
|
||||
│ │ │ --capability │
|
||||
│ │ │ nats,viperblock│
|
||||
│ │ │ ,hive,awsgw │
|
||||
│ │ └────────┬────────┘
|
||||
│ │ │
|
||||
│ │ POST /formation/join
|
||||
│ │ │
|
||||
┌─────▼──────┐ │ │
|
||||
│ Node 1 │◄─────────────┼──────────────────┘
|
||||
│ validates │ │
|
||||
│ updates toml │ │
|
||||
│ returns creds│─────────────┘
|
||||
│ broadcasts │──► hive.cluster.config-update (NATS)
|
||||
└─────┬───────┘
|
||||
│
|
||||
▼
|
||||
All existing nodes receive broadcast:
|
||||
- Regenerate nats.conf → SIGHUP nats-server
|
||||
- Update predastore.toml if needed → restart predastore
|
||||
- Update hive.toml with new node entry
|
||||
│
|
||||
▼
|
||||
New node generates configs, starts services:
|
||||
./scripts/start-dev.sh (on node4 only)
|
||||
```
|
||||
|
||||
### Node capabilities
|
||||
|
||||
Not all nodes need to run all services. The `--capability` flag controls which services a node provides:
|
||||
|
||||
```sh
|
||||
./bin/hive admin join \
|
||||
--node node4 \
|
||||
--bind 10.11.12.4 \
|
||||
--cluster-bind 10.11.12.4 \
|
||||
--host 10.11.12.1:4432 \
|
||||
--data-dir ~/hive/ \
|
||||
--config-dir ~/hive/config/ \
|
||||
--region ap-southeast-2 \
|
||||
--az ap-southeast-2a \
|
||||
--capability nats,viperblock,hive,awsgw
|
||||
```
|
||||
|
||||
Available capabilities:
|
||||
|
||||
| Capability | Service | Notes |
|
||||
|---|---|---|
|
||||
| `nats` | NATS server | Cluster messaging — required for all nodes |
|
||||
| `predastore` | Predastore storage node | Object shard storage via QUIC |
|
||||
| `predastoredb` | Predastore DB node | Raft-replicated metadata database |
|
||||
| `viperblock` | Viperblock | Local NVMe WAL for block storage — required if running VMs |
|
||||
| `hive` | Hive daemon | VM orchestration and scheduling |
|
||||
| `awsgw` | AWS Gateway | TLS API endpoint for AWS SDK compatibility |
|
||||
| `hive-ui` | Hive UI | Web management interface |
|
||||
|
||||
Default: all capabilities enabled.
|
||||
|
||||
**Example: compute-only node** (runs VMs, no object storage):
|
||||
```sh
|
||||
--capability nats,viperblock,hive,awsgw
|
||||
```
|
||||
|
||||
**Example: storage-only node** (object storage, no VMs):
|
||||
```sh
|
||||
--capability nats,predastore
|
||||
```
|
||||
|
||||
**Predastore DB scaling note:** The `predastoredb` capability controls whether a node participates in the Predastore Raft metadata cluster. For small clusters (3-5 nodes), every predastore node should also run predastoredb. For larger clusters (10+ nodes), Raft only needs 3-5 voters for consensus — additional predastore nodes store object shards via the hash ring but don't replicate the metadata database. This avoids replicating the KV store 25 times in a 25-node cluster.
|
||||
|
||||
The `--capability` flag also applies to Stage 1 `hive admin join` for initial formation. Stage 1 `hive admin init` always has all capabilities (it's the bootstrap node).
|
||||
|
||||
### Join protocol (Stage 2)
|
||||
|
||||
The `hive admin join` command detects whether it's joining a formation (Stage 1) or a running cluster (Stage 2) based on the response from the leader.
|
||||
|
||||
1. New node runs `hive admin join --host leader:4432 --node node4 --capability ...`
|
||||
2. New node POSTs to `http://leader:4432/formation/join` with node info + capabilities
|
||||
3. Leader daemon validates:
|
||||
- Node name not already in cluster (409 Conflict if duplicate)
|
||||
- Bind IP not already in use by another node
|
||||
- Capabilities are valid strings
|
||||
4. Leader updates local `hive.toml` with new node entry, increments epoch
|
||||
5. Leader returns: CA cert/key, credentials (AWS access/secret key, NATS token), full node list including the new node
|
||||
6. New node writes CA cert/key, generates server cert signed by cluster CA
|
||||
7. New node generates local configs (nats.conf, hive.toml, predastore.toml, etc.) using complete cluster data, filtered by its capabilities
|
||||
8. New node installs CA cert to system trust store
|
||||
9. Leader broadcasts config update to all existing nodes via NATS (`hive.cluster.config-update`)
|
||||
10. Existing nodes receive update and reconfigure affected services (see below)
|
||||
11. `hive admin join` exits with cluster summary — operator then runs `start-dev.sh` on the new node
|
||||
|
||||
### Service reconfiguration on existing nodes
|
||||
|
||||
When a new node joins a running cluster, the leader broadcasts a `hive.cluster.config-update` message via NATS containing the updated node list, capabilities, and epoch. Each existing daemon handles the update:
|
||||
|
||||
**NATS:**
|
||||
1. Regenerate `nats.conf` with new route added (only if new node has `nats` capability)
|
||||
2. Send `SIGHUP` to the `nats-server` process — NATS supports config reload via signal, including adding new cluster routes
|
||||
3. NATS discovers and connects to the new node once it starts
|
||||
4. If SIGHUP reload fails, log error and flag for manual restart
|
||||
|
||||
**Predastore storage (`predastore` capability):**
|
||||
- If the new node has `predastore` capability: regenerate `predastore.toml` with new `[[nodes]]` entry and restart the predastore process (predastore does not support hot config reload)
|
||||
- The consistent hash ring updates on restart — new PUTs distribute across all nodes including the new one
|
||||
- Existing objects are unaffected (see "Hash ring rebalancing" below)
|
||||
|
||||
**Predastore DB (`predastoredb` capability):**
|
||||
- If the new node has `predastoredb` capability: the leader calls `raft.AddVoter()` to add the new node to the Raft cluster
|
||||
- No restart needed on existing DB nodes — Raft handles membership changes natively via joint consensus
|
||||
- If the cluster already has sufficient DB voters (e.g., 5), the new node is not added as a voter even if it has the `predastoredb` capability — log a warning
|
||||
|
||||
**Viperblock:**
|
||||
- No reconfiguration needed on existing nodes — viperblock is local to each node
|
||||
- New node starts its own viperblock instance independently
|
||||
|
||||
**JetStream:**
|
||||
- Call `UpdateReplicas()` to match new cluster size (already implemented in `jetstream.go`)
|
||||
- Capped at the NATS cluster size (JetStream replicas cannot exceed the number of NATS nodes)
|
||||
|
||||
**AWS Gateway:**
|
||||
- No reconfiguration needed — uses dynamic NATS node discovery (`hive.nodes.discover` fan-out with 500ms timeout)
|
||||
- New node's daemon responds to discovery automatically once it's connected to NATS
|
||||
|
||||
**Hive UI:**
|
||||
- No reconfiguration needed — UI queries the AWS Gateway, which discovers nodes dynamically
|
||||
|
||||
### Predastore hash ring rebalancing
|
||||
|
||||
When a new predastore storage node joins, the consistent hash ring (xxhash + `buraksezer/consistent`) changes. Shard placement for new writes shifts, but existing data is safe.
|
||||
|
||||
**Immediate behaviour (no rebalancing — Stage 2 scope):**
|
||||
- Existing objects stay on their original nodes. Shard locations are recorded per-object in `ObjectToShardNodes` in the Raft-replicated s3db — GETs look up this metadata, not the ring
|
||||
- New objects (PUTs) use the updated ring, distributing across all nodes including the new one
|
||||
- DELETEs use stored metadata to find and remove shards from the correct nodes
|
||||
- The cluster is fully functional without rebalancing — the only downside is uneven storage distribution until a rebalance runs
|
||||
|
||||
**Why this is safe:** The hash ring is only used at write time to decide where to place shards. Once placed, the mapping `object_hash → [node_ids]` is persisted in s3db. All subsequent reads and deletes use this stored mapping, not the ring. Adding a node to the ring does not invalidate any existing mappings.
|
||||
|
||||
**Background rebalancing (future — not in Stage 2 scope):**
|
||||
A future `hive admin rebalance` command would:
|
||||
1. Scan all objects in s3db
|
||||
2. For each object, compute new ring placement with current topology
|
||||
3. If placement changed, migrate shards from old nodes to new node via QUIC
|
||||
4. Update s3db metadata with new shard locations (atomic Raft write)
|
||||
5. Delete old shard copies after successful migration
|
||||
|
||||
This is a significant feature (data migration, progress tracking, failure recovery, bandwidth throttling) and is deferred to a dedicated implementation.
|
||||
|
||||
### Config sync protocol
|
||||
|
||||
All nodes must converge on the same cluster state. The protocol uses epoch-based versioning:
|
||||
|
||||
1. **Epoch counter** — every config change (node join, capability change) increments the epoch on the leader
|
||||
2. **Broadcast on change** — leader publishes `hive.cluster.config-update` via NATS with full node list + epoch
|
||||
3. **Periodic heartbeat** — each daemon periodically publishes its epoch on `hive.cluster.heartbeat`; if a node detects it has a stale epoch (e.g., it was temporarily disconnected when the broadcast happened), it requests the latest config from the leader via `hive.cluster.config-request`
|
||||
4. **Config hash** — SHA256 of shared cluster data (already implemented via `computeConfigHash()`) included in `/health` endpoint response; monitoring can detect inconsistencies
|
||||
5. **Idempotent updates** — if a node receives a config update with the same or older epoch, it ignores it
|
||||
|
||||
### New node startup sequence
|
||||
|
||||
After `hive admin join` completes on the new node:
|
||||
|
||||
```sh
|
||||
./scripts/start-dev.sh # On the new node only
|
||||
```
|
||||
|
||||
`start-dev.sh` reads the node's capabilities from `hive.toml` and starts only matching services:
|
||||
|
||||
1. **NATS** (if `nats` capability) — joins existing mesh via routes in nats.conf
|
||||
2. **Predastore** (if `predastore` capability) — starts storage node; existing nodes have already been restarted with updated config
|
||||
3. **Predastore DB** (if `predastoredb` capability) — starts DB node; leader has already called `AddVoter()` so Raft accepts it
|
||||
4. **Viperblock** (if `viperblock` capability) — starts local instance
|
||||
5. **Daemon** (if `hive` capability) — connects to NATS, subscribes to topics, announces via heartbeat
|
||||
6. **AWS Gateway** (if `awsgw` capability) — starts and discovers nodes via NATS fan-out
|
||||
7. **UI** (if `hive-ui` capability) — starts web interface
|
||||
|
||||
### Stage 2 implementation
|
||||
|
||||
**Changes to `hive/formation/formation.go`:**
|
||||
- Add `Stage2Join()` handler — validates node, updates config, returns credentials + full cluster data
|
||||
- Add `BroadcastConfigUpdate(natsConn, clusterConfig)` — publishes update to `hive.cluster.config-update` NATS topic
|
||||
- `FormationServer` gains `mode` field: `ModeFormation` (Stage 1) vs `ModeRunning` (Stage 2)
|
||||
- In Stage 2 mode, the server has access to the NATS connection for broadcasting
|
||||
|
||||
**Changes to `hive/daemon/daemon.go`:**
|
||||
- On startup, start formation server in Stage 2 mode (replaces the existing `/join` handler)
|
||||
- Remove old `/join` endpoint handler (lines 814-931) — replaced by formation server
|
||||
- Add `handleConfigUpdate()` — NATS subscription on `hive.cluster.config-update`, applies received config
|
||||
- Add `reloadNATS()` — regenerates nats.conf from current cluster config, sends SIGHUP to nats-server PID
|
||||
- Add `reloadPredastore()` — regenerates predastore.toml, restarts predastore process
|
||||
- Add `handleClusterHeartbeat()` — periodic epoch broadcast + stale detection
|
||||
|
||||
**Changes to `hive/config/config.go`:**
|
||||
- Add `Capabilities []string` to per-node `Config` struct
|
||||
- Add `Capabilities []string` to `NodeJoinRequest`
|
||||
- Add `HasCapability(cap string) bool` method on `Config`
|
||||
|
||||
**Changes to `cmd/hive/cmd/admin.go`:**
|
||||
- `runAdminJoin`: new `--capability` flag (comma-separated string, default all capabilities)
|
||||
- Join detects Stage 1 vs Stage 2: if formation server responds with `mode: "formation"`, poll for completion (Stage 1); if `mode: "running"`, configs are returned immediately (Stage 2)
|
||||
|
||||
**Changes to `hive/formation/helpers.go`:**
|
||||
- `BuildClusterRoutes(nodes)` → `BuildClusterRoutes(nodes, capabilityFilter)` — only include nodes with `nats` capability
|
||||
- `BuildPredastoreNodes(nodes)` → filter by `predastore` capability
|
||||
- New: `BuildPredastoreDBNodes(nodes)` — filter by `predastoredb` capability
|
||||
- New: `FilterNodesByCapability(nodes map[string]Config, cap string) []Config`
|
||||
|
||||
**Changes to config templates:**
|
||||
- `nats.conf` — routes only include nodes with `nats` capability
|
||||
- `predastore-multinode.toml` — `[[nodes]]` only include `predastore` capable nodes; `[[db]]` only include `predastoredb` capable nodes
|
||||
|
||||
**Changes to `scripts/start-dev.sh`:**
|
||||
- Read capabilities from `hive.toml` for the local node
|
||||
- Only start services matching the node's capabilities
|
||||
- Skip predastore/predastoredb/viperblock/awsgw/hive-ui if not in capability list
|
||||
|
||||
### Stage 2 files
|
||||
|
||||
| File | Action |
|
||||
|---|---|
|
||||
| `hive/formation/formation.go` | Modify — add Stage 2 mode, Stage2Join handler, broadcast |
|
||||
| `hive/formation/helpers.go` | Modify — capability-aware filtering for all config builders |
|
||||
| `hive/formation/formation_test.go` | Modify — add Stage 2 tests |
|
||||
| `hive/daemon/daemon.go` | Modify — replace `/join` with formation Stage 2, add config update + service reload handlers |
|
||||
| `hive/config/config.go` | Modify — add Capabilities field to Config and NodeJoinRequest |
|
||||
| `cmd/hive/cmd/admin.go` | Modify — add `--capability` flag, Stage 1 vs Stage 2 detection logic |
|
||||
| `cmd/hive/cmd/templates/nats.conf` | Modify — capability-filtered routes |
|
||||
| `cmd/hive/cmd/templates/predastore-multinode.toml` | Modify — capability-filtered `[[nodes]]` and `[[db]]` sections |
|
||||
| `scripts/start-dev.sh` | Modify — capability-aware service startup |
|
||||
| `hive/daemon/jetstream.go` | No changes (UpdateReplicas already handles dynamic scaling) |
|
||||
|
||||
### Stage 2 testing
|
||||
|
||||
**Unit tests:**
|
||||
- Stage 2 join: validates capabilities, rejects duplicate node names and IPs, increments epoch
|
||||
- Capability filtering: correct nodes appear in NATS routes and predastore config
|
||||
- Config broadcast: serialization/deserialization of `hive.cluster.config-update` messages
|
||||
- Service reload: NATS config regeneration produces valid config, predastore config includes correct node subsets
|
||||
- Epoch-based sync: stale epoch detection triggers config request
|
||||
|
||||
**Integration test:**
|
||||
1. Form 3-node cluster via Stage 1 (all capabilities)
|
||||
2. Start all 3 nodes, verify NATS mesh + Predastore Raft healthy
|
||||
3. Run `hive admin join --node node4 --host 127.0.0.1:4432 --bind 127.0.0.4 --capability nats,viperblock,hive,awsgw`
|
||||
4. Verify: node4 added to all nodes' `hive.toml` with correct capabilities
|
||||
5. Verify: all `nats.conf` files updated with node4 route
|
||||
6. Verify: `predastore.toml` NOT updated (node4 has no `predastore` capability)
|
||||
7. Start node4 with `start-dev.sh`, verify it joins NATS mesh
|
||||
8. Verify: AWS Gateway discovers node4 via `hive.nodes.discover`
|
||||
9. Verify: predastore cluster unchanged (still 3 nodes)
|
||||
10. Run `hive admin join --node node5 --host 127.0.0.1:4432 --bind 127.0.0.5 --capability nats,predastore`
|
||||
11. Verify: `predastore.toml` updated on all nodes with node5 as storage node
|
||||
12. Verify: predastore `[[db]]` section unchanged (node5 has no `predastoredb` capability)
|
||||
13. Start node5, write new object via S3 API, verify shards distributed across 4 predastore nodes
|
||||
|
||||
## Remote testing
|
||||
|
||||
Integration tests above use localhost IPs for local simulation. For real multi-server testing, a 3-node cluster environment is available — see `docs/INTERNAL.md` for SSH access, deployment workflow (git diff → scp → git apply → make build), and cluster init/join commands with real IPs.
|
||||
|
||||
The remote test flow:
|
||||
1. Deploy code patches to all 3 nodes (INTERNAL.md: "Deploying code changes")
|
||||
2. Fresh install or patch-only depending on scope (INTERNAL.md: "Testing workflows")
|
||||
3. Run `hive admin init` on node 1, `hive admin join` on nodes 2 and 3 (INTERNAL.md: "Cluster init and join")
|
||||
4. Start services on all nodes, verify NATS mesh, Predastore Raft, and daemon health
|
||||
5. Run AWS CLI smoke test against the cluster endpoint
|
||||
@@ -1,295 +0,0 @@
|
||||
# NBD / QEMU Performance
|
||||
|
||||
## Summary
|
||||
|
||||
Hive VMs using NBD-backed boot volumes showed ~38x lower IOPS and ~35-40x higher latency compared to a Proxmox VM on the same host. The root cause was not the NBD transport itself, but missing QEMU tuning flags: no iothread, no cache policy, no multiqueue. Proxmox applies all of these by default; Hive applied none.
|
||||
|
||||
**Status: Implemented.** All boot volumes and hot-attached data volumes now get dedicated iothreads, `cache=none`, and multiqueue. Auxiliary volumes (cloud-init, EFI) are excluded — they are small, rarely read, and don't benefit from these flags. The viperblock LRU cache was also doubled from 64MB to 128MB for main volumes.
|
||||
|
||||
## What Changed
|
||||
|
||||
### QEMU flags (boot volumes)
|
||||
|
||||
`hive/daemon/daemon.go` — `StartInstance()` boot drive setup:
|
||||
|
||||
- Boot volume drive gets `cache=none`
|
||||
- A dedicated iothread (`ioth-os`) is created
|
||||
- virtio-blk-pci device gets `iothread=ioth-os,num-queues=<vCPU count>`
|
||||
|
||||
Before:
|
||||
```
|
||||
-drive file=nbd:unix:/path.sock,format=raw,if=none,media=disk,id=os
|
||||
-device virtio-blk-pci,drive=os,bootindex=1
|
||||
```
|
||||
|
||||
After:
|
||||
```
|
||||
-object iothread,id=ioth-os
|
||||
-drive file=nbd:unix:/path.sock,format=raw,if=none,media=disk,id=os,cache=none
|
||||
-device virtio-blk-pci,drive=os,iothread=ioth-os,num-queues=2,bootindex=1
|
||||
```
|
||||
|
||||
### QEMU flags (hot-attached data volumes)
|
||||
|
||||
`hive/daemon/daemon_handlers.go` — `handleAttachVolume()`:
|
||||
|
||||
Hot-attached volumes (e.g. a 1TB data drive attached to a running instance) also get a per-volume iothread via QMP:
|
||||
|
||||
1. `object-add` creates `ioth-<volumeID>`
|
||||
2. `blockdev-add` connects the NBD backend
|
||||
3. `device_add` creates `virtio-blk-pci` with `iothread=ioth-<volumeID>`
|
||||
|
||||
On detach (`handleDetachVolume()`), cleanup is:
|
||||
|
||||
1. `device_del` removes the guest device
|
||||
2. `blockdev-del` removes the block node
|
||||
3. `object-del` removes the iothread (best-effort, non-fatal if it fails)
|
||||
|
||||
### Excluded: cloud-init and EFI volumes
|
||||
|
||||
These auxiliary volumes are intentionally excluded from performance tuning:
|
||||
|
||||
- **Cloud-init** (`v.CloudInit`): Mounted as `if=virtio,media=cdrom`. Read once at boot, tiny payload. No iothread, no cache tuning.
|
||||
- **EFI** (`v.EFI`): Currently skipped entirely (`continue` in the volume loop). No QEMU drive emitted.
|
||||
- **Viperblock cache**: Volumes with names ending in `-cloudinit` or `-efi` get `cache_size=0` in both the viperblock in-process LRU and the nbdkit plugin. Main volumes get 128MB (32768 blocks at 4096B block size).
|
||||
|
||||
### Viperblock LRU cache doubled
|
||||
|
||||
`hive/services/viperblockd/viperblockd.go`:
|
||||
|
||||
- `defaultCache` changed from `(64 * 1024 * 1024) / DefaultBlockSize` → `(128 * 1024 * 1024) / DefaultBlockSize`
|
||||
- This is 16384 → 32768 blocks (at `DefaultBlockSize = 4096`)
|
||||
- Applies to both the in-process viperblock golang-lru cache and the `cache_size` parameter passed to the nbdkit plugin
|
||||
- Cloud-init/EFI volumes still get `cache_size=0`
|
||||
|
||||
### Struct changes
|
||||
|
||||
`hive/vm/vm.go`:
|
||||
|
||||
```go
|
||||
type Drive struct {
|
||||
File string `json:"file"`
|
||||
Format string `json:"format"`
|
||||
If string `json:"if"`
|
||||
Media string `json:"media"`
|
||||
ID string `json:"id"`
|
||||
Cache string `json:"cache,omitempty"` // NEW — e.g. "none"
|
||||
}
|
||||
|
||||
type IOThread struct { // NEW
|
||||
ID string `json:"id"`
|
||||
}
|
||||
|
||||
type Config struct {
|
||||
// ... existing fields ...
|
||||
Drives []Drive `json:"drives"`
|
||||
IOThreads []IOThread `json:"io_threads,omitempty"` // NEW
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
`Config.Execute()` emits `-object iothread,id=...` before `-drive` args, and appends `cache=<value>` to drive option strings when set.
|
||||
|
||||
## Benchmark Comparison (pre-fix)
|
||||
|
||||
Test: `fio --rw=randrw --rwmixread=70 --bs=4k --numjobs=4 --iodepth=32 --ioengine=libaio --direct=1`
|
||||
|
||||
| Metric | Proxmox VM (direct) | Hive VM (nested, NBD) | Delta |
|
||||
|---|---|---|---|
|
||||
| Read IOPS | 85,600 | 2,243 | 38x lower |
|
||||
| Write IOPS | 36,800 | 970 | 38x lower |
|
||||
| Read BW | 334 MiB/s | 8.8 MiB/s | 38x lower |
|
||||
| Write BW | 144 MiB/s | 3.8 MiB/s | 38x lower |
|
||||
| Read avg latency | 1.0 ms | 41.6 ms | 40x higher |
|
||||
| Write avg latency | 1.1 ms | 35.5 ms | 32x higher |
|
||||
| Read p99 latency | ~2.9 ms | ~75 ms | 26x higher |
|
||||
| Write p99 latency | ~3.0 ms | ~68 ms | 23x higher |
|
||||
| Disk util | 97% | 79% | Guest not saturating device |
|
||||
|
||||
Note: Proxmox test used 512 MiB/job (2 GiB total), Hive test used 128 MiB/job (512 MiB total). This affects cache behavior but does not explain a 38x gap.
|
||||
|
||||
The 79% utilization with 35-41 ms average latency in the Hive VM indicates requests were queueing behind a serialization point, not that the physical disk was saturated.
|
||||
|
||||
## Root Cause Analysis
|
||||
|
||||
**I/O path comparison:**
|
||||
|
||||
Proxmox VM:
|
||||
```
|
||||
fio → virtio-blk (iothread, multiqueue) → QEMU block layer (cache=none, aio=io_uring) → qcow2 file → host FS → physical disk
|
||||
```
|
||||
|
||||
Hive VM (nested inside Proxmox, before fix):
|
||||
```
|
||||
fio → virtio-blk (no iothread, single queue) → QEMU block layer (defaults) → NBD client → Unix socket → nbdkit → viperblock → [outer VM disk path]
|
||||
```
|
||||
|
||||
**What Proxmox set that Hive did not (now fixed):**
|
||||
|
||||
| Flag | Proxmox | Hive (before) | Hive (after) |
|
||||
|---|---|---|---|
|
||||
| `iothread` | Yes (`iothread-virtio0`) | No | Yes (`ioth-os`, `ioth-<volID>`) |
|
||||
| `cache` | `none` | Default (writeback) | `none` |
|
||||
| `num-queues` | Implicit (1, but with iothread) | 1, no iothread | vCPU count (boot), not set (hot-attach) |
|
||||
| `aio` | `io_uring` | Default (threads) | Default (threads) — correct for NBD |
|
||||
|
||||
The iothread was the highest-impact missing flag. Without it, all block I/O processing ran in the main QEMU event loop alongside vCPU emulation.
|
||||
|
||||
**Remaining factors (not addressed by QEMU flags):**
|
||||
|
||||
1. **QEMU NBD client has a 16 in-flight request limit** — The NBD protocol allows negotiating queue depth, but QEMU's built-in NBD client caps concurrent requests at 16. With `numjobs=4 --iodepth=32` (128 potential in-flight), requests queue inside QEMU before they even reach the socket. This is a hard protocol-level limit that cannot be changed via flags.
|
||||
|
||||
2. **Unix socket buffer size** — The default socket buffer (`net.core.wmem_default`, typically 212992 bytes) can bottleneck when many small 4K NBD request/response frames are in flight. Tuning `SO_SNDBUF`/`SO_RCVBUF` on the NBD socket or increasing system defaults may help throughput.
|
||||
|
||||
3. **Double virtualization overhead** — The benchmark Hive VM ran inside a Proxmox VM. Every I/O traverses two virtualization boundaries. This is a test environment artifact and won't apply on bare-metal deployments.
|
||||
|
||||
## QEMU Best Practices for NBD Backends
|
||||
|
||||
### iothread
|
||||
|
||||
An iothread is a dedicated QEMU thread for processing virtio-blk requests. Without one, I/O completion runs in the main loop and competes with vCPU scheduling. For any workload beyond trivial single-threaded sequential I/O, an iothread is required.
|
||||
|
||||
Each volume should have its own iothread. Boot volumes get `ioth-os`, hot-attached volumes get `ioth-<volumeID>`.
|
||||
|
||||
### cache=none
|
||||
|
||||
Sets QEMU's block layer to skip host page caching. For file backends, this means `O_DIRECT`. For NBD socket backends, there is no file to `O_DIRECT` on — but the flag still controls QEMU's internal write-back cache and flush forwarding behavior. `cache=none` ensures `FLUSH` commands from the guest are forwarded through NBD to the server, which is correct for data integrity.
|
||||
|
||||
### aio= (not meaningful for NBD)
|
||||
|
||||
The `aio=` option selects the host async I/O engine for file operations:
|
||||
- `aio=threads` — POSIX AIO via thread pool (default)
|
||||
- `aio=io_uring` — Linux io_uring submission queue
|
||||
- `aio=native` — Linux native AIO (`io_submit`)
|
||||
|
||||
These engines optimize `preadv`/`pwritev` syscalls on file descriptors. QEMU's NBD client does not use these syscalls — it uses coroutine-based socket I/O through `QIOChannel`. Setting `aio=io_uring` on an NBD drive changes nothing in the I/O path. The default (`aio=threads`) is correct and we intentionally omit this flag.
|
||||
|
||||
### virtio-blk multiqueue (num-queues)
|
||||
|
||||
By default, virtio-blk exposes a single virtqueue. With multiple vCPUs, all cores contend on that single queue's notification and completion path. Setting `num-queues=N` (where N = vCPU count) allows each vCPU to submit I/O to its own queue, eliminating cross-CPU contention.
|
||||
|
||||
Requires guest kernel support (Linux 3.13+, always available on modern distros).
|
||||
|
||||
Currently applied to boot volumes only (where the vCPU count is known from `Config.CPUCount`). Hot-attached volumes do not set `num-queues` because the device_add QMP path doesn't easily support it — the iothread alone provides the primary benefit.
|
||||
|
||||
### NBD protocol limits
|
||||
|
||||
QEMU's NBD client implementation enforces a maximum of 16 concurrent in-flight requests per connection. This is defined in the QEMU source and cannot be changed via command-line flags. At high queue depths, this becomes a bottleneck regardless of other tuning.
|
||||
|
||||
Potential future mitigation: multiple NBD connections to the same export (requires changes to both QEMU invocation and nbdkit/viperblock).
|
||||
|
||||
## Viperblock Cache Architecture
|
||||
|
||||
The viperblock cache operates at two layers:
|
||||
|
||||
1. **In-process golang-lru** — Viperblock's Go process maintains a hashicorp/golang-lru cache of recently accessed blocks. This avoids round-trips to the S3-compatible backend (predastore) for hot blocks.
|
||||
|
||||
2. **nbdkit plugin cache** — The nbdkit plugin (which runs as a separate process) has its own `cache_size` parameter that controls a similar LRU cache within the C plugin.
|
||||
|
||||
Both caches are sized identically via `defaultCache`:
|
||||
|
||||
| Volume type | Cache size | Blocks (4KB each) |
|
||||
|---|---|---|
|
||||
| Main (boot, data) | 128 MB | 32,768 |
|
||||
| Cloud-init (`-cloudinit` suffix) | 0 (disabled) | 0 |
|
||||
| EFI (`-efi` suffix) | 0 (disabled) | 0 |
|
||||
|
||||
Cloud-init and EFI volumes are small (typically < 1MB), read once at boot, and would waste memory with a cache enabled.
|
||||
|
||||
## Reference: QEMU Commands
|
||||
|
||||
### Proxmox outer VM (relevant flags only)
|
||||
|
||||
```sh
|
||||
/usr/bin/kvm \
|
||||
-object iothread,id=iothread-virtio0 \
|
||||
-drive file=/var/lib/vz/images/146/vm-146-disk-0.qcow2,if=none,id=drive-virtio0,cache=none,aio=io_uring,discard=on,format=qcow2,detect-zeroes=unmap \
|
||||
-device virtio-blk-pci,drive=drive-virtio0,id=virtio0,iothread=iothread-virtio0,bootindex=100 \
|
||||
-smp 4,sockets=1,cores=4,maxcpus=4 \
|
||||
-m 8192 \
|
||||
-cpu host
|
||||
```
|
||||
|
||||
### Hive inner VM (before fix)
|
||||
|
||||
```sh
|
||||
qemu-system-x86_64 \
|
||||
-enable-kvm -cpu host -smp 2 -m 4096 \
|
||||
-drive file=nbd:unix:/run/user/1000/nbd-vol-xxx.sock,format=raw,if=none,media=disk,id=os \
|
||||
-device virtio-blk-pci,drive=os,bootindex=1 \
|
||||
-M q35
|
||||
```
|
||||
|
||||
### Hive inner VM (after fix)
|
||||
|
||||
```sh
|
||||
qemu-system-x86_64 \
|
||||
-enable-kvm -cpu host -smp 2 -m 4096 \
|
||||
-object iothread,id=ioth-os \
|
||||
-drive file=nbd:unix:/run/user/1000/nbd-vol-xxx.sock,format=raw,if=none,media=disk,id=os,cache=none \
|
||||
-device virtio-blk-pci,drive=os,iothread=ioth-os,num-queues=2,bootindex=1 \
|
||||
-M q35
|
||||
```
|
||||
|
||||
### Hot-attach QMP sequence (after fix)
|
||||
|
||||
```json
|
||||
{"execute": "object-add", "arguments": {"qom-type": "iothread", "id": "ioth-vol-abc123"}}
|
||||
{"execute": "blockdev-add", "arguments": {"node-name": "nbd-vol-abc123", "driver": "nbd", "server": {"type": "unix", "path": "/run/user/1000/nbd-vol-abc123.sock"}, "export": "", "read-only": false}}
|
||||
{"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "vdisk-vol-abc123", "drive": "nbd-vol-abc123", "iothread": "ioth-vol-abc123", "bus": "hotplug1"}}
|
||||
```
|
||||
|
||||
### Hot-detach QMP sequence (after fix)
|
||||
|
||||
```json
|
||||
{"execute": "device_del", "arguments": {"id": "vdisk-vol-abc123"}}
|
||||
{"execute": "blockdev-del", "arguments": {"node-name": "nbd-vol-abc123"}}
|
||||
{"execute": "object-del", "arguments": {"id": "ioth-vol-abc123"}}
|
||||
```
|
||||
|
||||
## Files Modified
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `hive/vm/vm.go` | Added `Cache` field to `Drive`, added `IOThread` struct and `IOThreads` to `Config`, updated `Execute()` to emit `-object iothread` and `cache=` |
|
||||
| `hive/daemon/daemon.go` | Boot drive setup: `cache=none`, iothread, `num-queues=<vCPU count>` |
|
||||
| `hive/daemon/daemon_handlers.go` | Hot-attach: QMP `object-add` iothread + `iothread` in `device_add`. Hot-detach: QMP `object-del` cleanup |
|
||||
| `hive/daemon/daemon_test.go` | Updated detach test assertions to expect `object-del` in QMP command sequence |
|
||||
| `hive/vm/vm_test.go` | Added `TestExecute_IOThreadAndCache`, `TestExecute_NoCacheWhenEmpty`, `TestExecute_MultipleIOThreads` |
|
||||
| `hive/services/viperblockd/viperblockd.go` | Doubled `defaultCache` from 64MB (16384 blocks) to 128MB (32768 blocks) |
|
||||
|
||||
## Diagnostic Tests
|
||||
|
||||
Run inside the Hive guest VM to measure the impact of changes.
|
||||
|
||||
**Baseline per-I/O overhead** (isolates single-request latency):
|
||||
```sh
|
||||
fio --name=lat1 --directory=$HOME/bench --rw=randrw --rwmixread=70 \
|
||||
--bs=4k --size=512M --numjobs=1 --iodepth=1 \
|
||||
--ioengine=libaio --direct=1 --group_reporting
|
||||
```
|
||||
|
||||
**Queue depth scaling** (detects serialization):
|
||||
```sh
|
||||
fio --name=qdtest --directory=$HOME/bench --rw=randrw --rwmixread=70 \
|
||||
--bs=4k --size=512M --numjobs=1 --iodepth=32 \
|
||||
--ioengine=libaio --direct=1 --group_reporting
|
||||
```
|
||||
|
||||
If `iodepth=1` latency is already in the tens of ms, the NBD/viperblock backend is fundamentally slow per-I/O. If `iodepth=1` is reasonable but `iodepth=32` explodes, there is a serialization or flush amplification problem in the stack.
|
||||
|
||||
**Before/after comparison** (run with target QEMU flags applied):
|
||||
```sh
|
||||
fio --name=randrw_70_30 --directory=$HOME/bench --rw=randrw --rwmixread=70 \
|
||||
--bs=4k --size=512M --numjobs=4 --iodepth=32 \
|
||||
--ioengine=libaio --direct=1 --group_reporting
|
||||
```
|
||||
|
||||
## Future Work
|
||||
|
||||
- **Post-fix benchmarks**: Re-run fio with the new QEMU flags and 128MB cache to quantify improvement
|
||||
- **Bare-metal benchmarks**: The pre-fix numbers were from a nested Proxmox environment (double virtualization). Bare-metal numbers will show the true NBD overhead
|
||||
- **NBD 16-request limit**: Investigate multi-connection NBD or QEMU patches to raise the in-flight cap
|
||||
- **Socket buffer tuning**: Test `SO_SNDBUF`/`SO_RCVBUF` sizing on NBD Unix sockets
|
||||
- **nbdkit threading**: nbdkit `--threads` parameter (defaults to 16 for parallel plugins) — verify this matches or exceeds the NBD in-flight limit
|
||||
- **Dynamic cache sizing**: Scale viperblock cache based on available system memory rather than a fixed 128MB
|
||||
@@ -114,7 +114,7 @@ func (d *Daemon) handleInstanceCrash(instance *vm.VM, waitErr error) {
|
||||
|
||||
// Clean up stale QMP socket so QEMU can rebind on restart
|
||||
if instance.Config.QMPSocket != "" {
|
||||
os.Remove(instance.Config.QMPSocket)
|
||||
_ = os.Remove(instance.Config.QMPSocket)
|
||||
}
|
||||
|
||||
// Unmount EBS volumes (same pattern as stopInstance)
|
||||
|
||||
+1
-1
@@ -177,7 +177,7 @@ func SetOOMScore(pid int, score int) error {
|
||||
return fmt.Errorf("OOM score adjustment is only supported on Linux")
|
||||
}
|
||||
path := fmt.Sprintf("/proc/%d/oom_score_adj", pid)
|
||||
return os.WriteFile(path, []byte(strconv.Itoa(score)), 0644)
|
||||
return os.WriteFile(path, []byte(strconv.Itoa(score)), 0600)
|
||||
}
|
||||
|
||||
func ReadPidFile(name string) (int, error) {
|
||||
|
||||
Reference in New Issue
Block a user