mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 09:01:31 +00:00
multi-server tweaks, ec2 health, IAC templates, misc cleanup
This commit is contained in:
Ben Duncan
@@ -187,3 +187,30 @@ mulga/
|
||||
```bash
|
||||
AWS_PROFILE=hive aws ec2 describe-instances
|
||||
```
|
||||
|
||||
### Remote Test Cluster
|
||||
|
||||
A multi-node Proxmox test cluster is available for validating changes on real hardware. Node IPs, SSH credentials, and connection details are in `docs/DEVNOTES.md` (gitignored, private to the repo).
|
||||
|
||||
**Lifecycle scripts** (`scripts/iac/`):
|
||||
|
||||
```bash
|
||||
# Source environment first
|
||||
source scripts/iac/proxmox/.env && source ~/prox
|
||||
|
||||
# Provision, configure, test, destroy
|
||||
./scripts/iac/hive-test.sh up <cluster-name> # Create VMs
|
||||
./scripts/iac/hive-test.sh configure <cluster-name> # Clone, build, form cluster, start services
|
||||
./scripts/iac/hive-test.sh test <cluster-name> # Smoke tests (describe-regions, instances, etc.)
|
||||
./scripts/iac/hive-test.sh status <cluster-name> # Health check all nodes
|
||||
./scripts/iac/hive-test.sh ssh <cluster-name> <N> # SSH to node N
|
||||
./scripts/iac/hive-test.sh down <cluster-name> # Destroy VMs
|
||||
```
|
||||
|
||||
**Deploying local changes to a running cluster:**
|
||||
|
||||
1. Read `docs/DEVNOTES.md` for the current cluster node IPs and SSH key path
|
||||
2. SSH key: `~/.ssh/us-west-1-tf-cloudinit`, user: `tf-user`
|
||||
3. Use `git diff` patches or `scp` to push changes, then rebuild on each node
|
||||
4. Daemon health endpoint is HTTP (not HTTPS): `curl -s http://<ip>:4432/health`
|
||||
5. Always `export PATH=/usr/local/go/bin:$PATH` on remote nodes before builds
|
||||
|
||||
+46
-8
@@ -674,7 +674,7 @@ ip addr show | grep "inet "
|
||||
|
||||
#### 1. Set Variables
|
||||
|
||||
Export the IPs for your three servers (replace with your actual IPs):
|
||||
Export the IPs for your three servers (replace with your actual IPs) on each server:
|
||||
|
||||
```bash
|
||||
export HIVE_NODE1=192.168.1.10
|
||||
@@ -868,16 +868,54 @@ sudo ip addr del 10.11.12.3/24 dev eth0
|
||||
|
||||
## Shutdown
|
||||
|
||||
Stop services on each node:
|
||||
When running in the multi-server cluster mode a graceful shutdown is required to synchronize and coordinate nodes.
|
||||
|
||||
On any node, issue the following command to shutdown nodes cluster wide:
|
||||
|
||||
```sh
|
||||
./bin/hive admin cluster shutdown
|
||||
```
|
||||
|
||||
Example output:
|
||||
```bash
|
||||
# Simulated — stop in reverse order:
|
||||
./scripts/stop-dev.sh ~/node3/
|
||||
./scripts/stop-dev.sh ~/node2/
|
||||
./scripts/stop-dev.sh ~/node1/
|
||||
Starting coordinated cluster shutdown (3 nodes)
|
||||
Phases: gate -> drain -> storage -> persist -> infra
|
||||
Timeout per phase: 2m0s
|
||||
|
||||
[GATE] Sending to 3 node(s)...
|
||||
[GATE] node1: stopped awsgw, hive-ui
|
||||
[GATE] node3: stopped awsgw, hive-ui
|
||||
[GATE] node2: stopped awsgw, hive-ui
|
||||
[GATE] Complete (3/3 nodes, 2.005s)
|
||||
|
||||
[DRAIN] Sending to 3 node(s)...
|
||||
[DRAIN] node1: 0/0 VMs remaining
|
||||
[DRAIN] node1: 0/0 VMs remaining
|
||||
[DRAIN] node2: 0/0 VMs remaining
|
||||
[DRAIN] node2: 0/0 VMs remaining
|
||||
[DRAIN] node3: 0/0 VMs remaining
|
||||
[DRAIN] node3: 0/0 VMs remaining
|
||||
[DRAIN] node1: OK
|
||||
[DRAIN] node3: OK
|
||||
[DRAIN] node2: OK
|
||||
[DRAIN] Complete (3/3 nodes, 5ms)
|
||||
|
||||
[STORAGE] Sending to 3 node(s)...
|
||||
[STORAGE] node1: stopped viperblock
|
||||
[STORAGE] node3: stopped viperblock
|
||||
[STORAGE] node2: stopped viperblock
|
||||
[STORAGE] Complete (3/3 nodes, 1.09s)
|
||||
|
||||
[PERSIST] Sending to 3 node(s)...
|
||||
[PERSIST] node1: stopped predastore
|
||||
[PERSIST] node3: stopped predastore
|
||||
[PERSIST] node2: stopped predastore
|
||||
[PERSIST] Complete (3/3 nodes, 1.003s)
|
||||
|
||||
[INFRA] Sending final shutdown to all nodes...
|
||||
[INFRA] Complete
|
||||
Cluster shutdown complete (6.104s)
|
||||
|
||||
# Real multi-server — run on each server:
|
||||
./scripts/stop-dev.sh
|
||||
```
|
||||
|
||||
If EC2 instances are running, the stop process will gracefully terminate them, unmount attached EBS volumes (via NBD), and flush the write-ahead-log (WAL) to the S3 server (Predastore). This may take a few minutes depending on instance volume sizes.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,78 +1,613 @@
|
||||
# Phase 2.1 Heartbeats + Phase 2.4 Coordinated Cluster Shutdown
|
||||
# Cluster Lifecycle: Heartbeats, Shutdown, Upgrades, Resilience, and VM Migration
|
||||
|
||||
**Status: Complete**
|
||||
## Phase 2 Roadmap
|
||||
|
||||
## Summary
|
||||
| Phase | Feature | Status | Description |
|
||||
|-------|---------|--------|-------------|
|
||||
| 2.1 | Heartbeats | **Complete** | Daemon heartbeats publish node status to KV every 10s |
|
||||
| 2.4 | Coordinated Shutdown | **Complete** | Phased cluster-wide shutdown via NATS |
|
||||
| 2.2 | Node Modes & Health Monitoring | **Planned** | Drain/maintenance/offline modes, stale heartbeat detection |
|
||||
| 2.5 | Coordinated Rolling Upgrade | **Planned** | Cluster-wide binary upgrade without full downtime |
|
||||
| 2.6 | Degraded Storage Writes | **Planned** | Predastore shard redirection when nodes are unavailable |
|
||||
| 2.7 | VM Migration | **Planned** | Relocate running VMs to other nodes during drain/maintenance |
|
||||
| 2.8 | [EC2 Health & Auto-Restart](ec2-health-restart.md) | **Planned** | QEMU crash detection, auto-restart, OOM protection, `DescribeInstanceStatus` API |
|
||||
|
||||
Implemented daemon heartbeats (Phase 2.1) and coordinated cluster shutdown (Phase 2.4). Heartbeats publish node status to KV every 10s. Coordinated shutdown drives a phased shutdown across all nodes via NATS, replacing the error-prone per-node approach for multi-node clusters.
|
||||
---
|
||||
|
||||
## Context / Problem Statement
|
||||
## Phase 2.1 — Heartbeats (Complete)
|
||||
|
||||
- No cluster-wide health awareness — nodes don't know peer status
|
||||
- Multi-node shutdown required manual per-node `stop-dev.sh`, which is error-prone and doesn't guarantee correct phase ordering (VMs must stop before storage, storage before NATS)
|
||||
- Phase 0+1 (capabilities, recovery throttling, shutdown markers) were already complete
|
||||
### Summary
|
||||
|
||||
## Design Decisions
|
||||
Each daemon publishes its status to JetStream KV every 10 seconds — node name, epoch, running services, VM count, and resource allocation.
|
||||
|
||||
- **Option B chosen**: Daemons handle stopping local services internally via `utils.StopProcess()` when they receive NATS shutdown phase messages. The CLI coordinator just sends messages and collects ACKs.
|
||||
- **Heartbeat interval**: 10 seconds — fast enough for health monitoring, low overhead
|
||||
- **Shutdown phases**: GATE → DRAIN → STORAGE → PERSIST → INFRA — ordered to prevent data loss
|
||||
- **INFRA phase is fire-and-forget**: NATS is going down, no ACK possible
|
||||
- **Scripts**: `stop-dev.sh` auto-detects multi-node and delegates to `hive admin cluster shutdown`; `HIVE_FORCE_LOCAL_STOP=1` overrides for per-service stop
|
||||
### Implementation
|
||||
|
||||
## Proposed Changes (Implemented)
|
||||
|
||||
### Heartbeat System
|
||||
- `Heartbeat` struct in `jetstream.go` with node, epoch, services, VM count, resource allocation
|
||||
- `WriteHeartbeat`/`ReadHeartbeat` KV methods on `JetStreamManager`
|
||||
- `startHeartbeat()` goroutine in `heartbeat.go` — fires immediately, then every 10s
|
||||
- `buildHeartbeat()` reads from `ResourceManager` and `Instances`
|
||||
|
||||
### Coordinated Shutdown
|
||||
### Files Modified
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `hive/daemon/jetstream.go` | Edit | `Heartbeat` struct + KV methods |
|
||||
| `hive/daemon/heartbeat.go` | New | `startHeartbeat()`, `buildHeartbeat()` goroutine |
|
||||
| `hive/daemon/daemon.go` | Edit | Call `startHeartbeat()` in `Start()` |
|
||||
| `hive/daemon/heartbeat_test.go` | New | Heartbeat tests (build, allocation, KV round-trip) |
|
||||
|
||||
---
|
||||
|
||||
## Phase 2.4 — Coordinated Cluster Shutdown (Complete)
|
||||
|
||||
### Summary
|
||||
|
||||
`hive admin cluster shutdown` drives a phased shutdown across all nodes via NATS, replacing the error-prone per-node `stop-dev.sh` approach for multi-node clusters.
|
||||
|
||||
### Design Decisions
|
||||
|
||||
- **Option B chosen**: Daemons handle stopping local services internally via `utils.StopProcess()` when they receive NATS shutdown phase messages. The CLI coordinator just sends messages and collects ACKs.
|
||||
- **Shutdown phases**: GATE → DRAIN → STORAGE → PERSIST → INFRA — ordered to prevent data loss
|
||||
- **INFRA phase is fire-and-forget**: NATS is going down, no ACK possible
|
||||
- **Scripts**: `stop-dev.sh` auto-detects multi-node and delegates to `hive admin cluster shutdown`; `HIVE_FORCE_LOCAL_STOP=1` overrides for per-service stop
|
||||
|
||||
### Implementation
|
||||
|
||||
- Five shutdown phase handlers in `shutdown.go`: GATE, DRAIN, STORAGE, PERSIST, INFRA
|
||||
- `ShutdownRequest`, `ShutdownACK`, `ShutdownProgress` message types
|
||||
- `shuttingDown atomic.Bool` on Daemon — set during GATE, checked by `setupShutdown()` to skip redundant VM stops
|
||||
- NATS subscriptions for `hive.cluster.shutdown.{phase}` (fan-out, no queue group)
|
||||
- Progress updates published during DRAIN phase
|
||||
|
||||
### CLI Coordinator
|
||||
- `hive admin cluster shutdown` command in `cluster.go`
|
||||
- Flags: `--force`, `--timeout`, `--dry-run`
|
||||
- Executes phases sequentially, collecting ACKs with timeout
|
||||
- Writes `ClusterShutdownState` to KV for observability
|
||||
- Live progress display during DRAIN phase
|
||||
|
||||
### Script Updates
|
||||
- `stop-dev.sh`: `is_multinode()` detection, delegates to coordinated shutdown
|
||||
- CLI coordinator in `cmd/hive/cmd/cluster.go` with `--force`, `--timeout`, `--dry-run` flags
|
||||
- `stop-dev.sh`: `is_multinode()` detection, delegates to coordinated shutdown (skipped when data-dir arg passed, e.g. E2E cleanup)
|
||||
- `start-dev.sh`: `is_multinode()` detection, cluster health poll at end
|
||||
|
||||
## Files Modified
|
||||
### Files Modified
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `hive/daemon/jetstream.go` | Edit | `Heartbeat`, `ClusterShutdownState` structs + KV methods |
|
||||
| `hive/daemon/heartbeat.go` | New | `startHeartbeat()`, `buildHeartbeat()` goroutine |
|
||||
| `hive/daemon/jetstream.go` | Edit | `ClusterShutdownState` struct + KV methods |
|
||||
| `hive/daemon/shutdown.go` | New | Shutdown types + 5 phase handlers |
|
||||
| `hive/daemon/daemon.go` | Edit | `shuttingDown atomic.Bool`, shutdown subscriptions, `startHeartbeat()`, updated `setupShutdown()` |
|
||||
| `hive/daemon/daemon.go` | Edit | `shuttingDown atomic.Bool`, shutdown subscriptions, updated `setupShutdown()` |
|
||||
| `cmd/hive/cmd/admin.go` | Edit | `clusterCmd` + `clusterShutdownCmd` registration |
|
||||
| `cmd/hive/cmd/cluster.go` | New | `runClusterShutdown()` coordinator |
|
||||
| `scripts/stop-dev.sh` | Edit | Multi-node detection + coordinated shutdown delegation |
|
||||
| `scripts/start-dev.sh` | Edit | Multi-node detection + cluster health poll |
|
||||
| `hive/daemon/heartbeat_test.go` | New | Heartbeat tests (build, allocation, KV round-trip) |
|
||||
| `hive/daemon/shutdown_test.go` | New | Shutdown tests (marshal, KV round-trip, flag behavior) |
|
||||
|
||||
## Testing
|
||||
### Testing
|
||||
|
||||
- `make preflight` — all checks pass (gofmt, go vet, gosec, staticcheck, govulncheck, all tests)
|
||||
- 9 new unit tests covering heartbeat building, resource reflection, KV round-trips, shutdown message marshaling, and shutdown state management
|
||||
- `make preflight` — all checks pass
|
||||
- 9 new unit tests
|
||||
- `make test-docker-multi` — full multi-node E2E passes (2 consecutive runs confirmed)
|
||||
- `hive admin cluster shutdown --dry-run` prints phase plan
|
||||
- `stop-dev.sh` on multi-node delegates to coordinated shutdown
|
||||
- `HIVE_FORCE_LOCAL_STOP=1 stop-dev.sh` on multi-node uses per-service stop
|
||||
- `stop-dev.sh` on single-node uses per-service stop (unchanged behavior)
|
||||
- `stop-dev.sh` on single-node uses per-service stop (unchanged)
|
||||
|
||||
## Future Work
|
||||
---
|
||||
|
||||
- Phase 2.2: Heartbeat-driven health monitoring (stale heartbeat detection, node failure alerts)
|
||||
- Phase 2.3: Automatic VM migration on node failure
|
||||
- Integration with `hive top` to display heartbeat data
|
||||
- Cluster shutdown progress bar in UI
|
||||
## Phase 2.2 — Node Modes & Health Monitoring (Planned)
|
||||
|
||||
### Problem Statement
|
||||
|
||||
Currently, nodes are either "running" or "not responding." There is no concept of a node being in a transitional state — draining VMs, undergoing maintenance, or recovering from a crash. The health endpoint returns hardcoded `"running"`, and `hive get nodes` shows `"Ready"` or `"NotReady"` with no in-between.
|
||||
|
||||
This means:
|
||||
- No way to gracefully take a node offline for maintenance without a full cluster shutdown
|
||||
- No distinction between a planned offline and a crash — rejoining is the same either way
|
||||
- No stale heartbeat detection — if a node stops publishing heartbeats, nobody notices
|
||||
- The scheduler keeps routing work to draining nodes until they're fully dead
|
||||
|
||||
### Design: Node Modes
|
||||
|
||||
```
|
||||
┌─────────┐
|
||||
│ NORMAL │ ← default, accepts all work
|
||||
└────┬────┘
|
||||
drain cmd │ ▲ drain-cancel / rejoin
|
||||
▼ │
|
||||
┌─────────┐
|
||||
│DRAINING │ ← rejects new VMs, existing VMs run until migrated/stopped
|
||||
└────┬────┘
|
||||
all VMs │
|
||||
stopped │
|
||||
▼
|
||||
┌─────────┐
|
||||
│CORDONED │ ← no VMs, services still running, ready for maintenance
|
||||
└────┬────┘
|
||||
maintenance │ ▲ uncordon
|
||||
cmd ▼ │
|
||||
┌─────────┐
|
||||
│MAINTENANCE│ ← services may be stopped, node offline
|
||||
└─────────┘
|
||||
```
|
||||
|
||||
Add a `NodeMode` type to the Daemon struct (atomic for lock-free reads). Store in JetStream KV (`node.<name>.mode`) so it survives daemon restarts and is visible cluster-wide.
|
||||
|
||||
#### Mode Behaviors
|
||||
|
||||
| Mode | Accepts new VMs | Existing VMs | Services | Heartbeats |
|
||||
|------|----------------|--------------|----------|------------|
|
||||
| `normal` | Yes | Run | All running | Published |
|
||||
| `draining` | No | Run (being migrated/stopped) | All running | Published with mode |
|
||||
| `cordoned` | No | None | All running | Published with mode |
|
||||
| `maintenance` | No | None | May be stopped | Published until NATS stops |
|
||||
|
||||
#### CLI Commands
|
||||
|
||||
```bash
|
||||
hive node drain <node-name> # NORMAL → DRAINING (triggers VM migration/stop)
|
||||
hive node drain-cancel <node-name> # DRAINING → NORMAL (abort drain)
|
||||
hive node cordon <node-name> # NORMAL/CORDONED (reject new VMs, don't touch existing)
|
||||
hive node uncordon <node-name> # CORDONED/DRAINING → NORMAL
|
||||
hive node maintenance <node-name> # CORDONED → MAINTENANCE
|
||||
```
|
||||
|
||||
#### Implementation Details
|
||||
|
||||
**Heartbeat extension** — add `Mode` field to `Heartbeat` struct:
|
||||
```go
|
||||
type Heartbeat struct {
|
||||
// ... existing fields ...
|
||||
Mode string `json:"mode"` // "normal", "draining", "cordoned", "maintenance"
|
||||
}
|
||||
```
|
||||
|
||||
**Resource manager integration** — when mode != `normal`, unsubscribe from all `ec2.RunInstances.*` topics. The existing `updateInstanceSubscriptions()` already unsubscribes when capacity is 0; extend it to check mode.
|
||||
|
||||
**Stale heartbeat detection** — new goroutine `monitorPeerHeartbeats()`:
|
||||
- Every 30s, read all `heartbeat.*` keys from KV
|
||||
- If any peer's timestamp is >60s old (6 missed heartbeats), mark as `"suspect"`
|
||||
- If >120s old, log warning and publish alert to `hive.cluster.alert`
|
||||
- Display in `hive get nodes` as `"Unreachable"` status
|
||||
|
||||
**Graceful rejoin** — when a node starts and finds an existing `node.<name>.mode` key:
|
||||
- If `maintenance`: transition to `normal` after services are healthy
|
||||
- If `draining`: resume drain (check for remaining VMs)
|
||||
- Publish heartbeat immediately on startup to signal liveness
|
||||
|
||||
### Proposed Changes
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `hive/config/config.go` | Edit | Add `NodeMode` type and constants |
|
||||
| `hive/daemon/daemon.go` | Edit | Add `nodeMode atomic.Value`, check mode in request handlers |
|
||||
| `hive/daemon/heartbeat.go` | Edit | Add `Mode` to heartbeat, add `monitorPeerHeartbeats()` |
|
||||
| `hive/daemon/jetstream.go` | Edit | Add `WriteNodeMode`/`ReadNodeMode` KV methods |
|
||||
| `hive/daemon/daemon_handlers.go` | Edit | Check mode before accepting RunInstances |
|
||||
| `cmd/hive/cmd/node.go` | New | `hive node drain/cordon/uncordon/maintenance` commands |
|
||||
| `cmd/hive/cmd/get.go` | Edit | Show mode in `hive get nodes` output |
|
||||
|
||||
### Testing
|
||||
|
||||
- Unit tests for mode transitions, heartbeat staleness detection
|
||||
- `make test-docker-multi`: drain a node, verify new VMs route elsewhere
|
||||
- Remote cluster: `hive node drain node3`, verify VMs stop, verify `hive get nodes` shows "Draining"
|
||||
|
||||
---
|
||||
|
||||
## Phase 2.5 — Coordinated Rolling Upgrade (Planned)
|
||||
|
||||
### Problem Statement
|
||||
|
||||
Upgrading the Hive binary across a cluster currently requires either:
|
||||
1. Full cluster shutdown → upgrade all nodes → restart (downtime)
|
||||
2. Manual per-node SSH, stop, upgrade, restart (error-prone, no ordering guarantees)
|
||||
|
||||
Neither approach is acceptable for production clusters. We need a coordinated upgrade that takes nodes offline one at a time, upgrades them, and brings them back — while the cluster continues serving requests on the remaining nodes.
|
||||
|
||||
### Design: Rolling Upgrade via NATS Coordination
|
||||
|
||||
The cluster already has the building blocks: coordinated shutdown phases, heartbeats, and (with Phase 2.2) node modes. A rolling upgrade orchestrates these per-node:
|
||||
|
||||
```
|
||||
For each node (one at a time):
|
||||
1. DRAIN node (Phase 2.2) — migrate/stop VMs
|
||||
2. CORDON node — no new work
|
||||
3. Stop services on node (reuse shutdown phases, targeted to single node)
|
||||
4. Upgrade binary
|
||||
5. Start services
|
||||
6. UNCORDON — resume normal operation
|
||||
7. Wait for health check before proceeding to next node
|
||||
```
|
||||
|
||||
#### Upgrade Delivery Mechanism
|
||||
|
||||
**Option A: Binary push via Predastore (recommended)**
|
||||
|
||||
The upgrade coordinator (any node or external CLI) uploads the new `hive` binary to Predastore (S3-compatible), then each node downloads and replaces its own binary. No SSH needed, no inter-node trust beyond what NATS already provides.
|
||||
|
||||
```bash
|
||||
# Operator uploads new binary
|
||||
hive admin cluster upgrade --binary ./bin/hive
|
||||
|
||||
# Internally:
|
||||
# 1. Upload binary to predastore: s3://hive-system/upgrades/hive-<version>
|
||||
# 2. For each node:
|
||||
# a. Send NATS "hive.node.upgrade" to target node
|
||||
# b. Node downloads binary from predastore
|
||||
# c. Node replaces /usr/local/bin/hive (or wherever binary lives)
|
||||
# d. Node restarts services
|
||||
# e. Wait for heartbeat to confirm healthy
|
||||
```
|
||||
|
||||
**Option B: Manual per-node (simpler, good enough for now)**
|
||||
|
||||
The operator runs `hive node drain <name>`, SSHes in, replaces the binary, and runs `hive node uncordon <name>`. The CLI provides the orchestration framework but the binary swap is manual.
|
||||
|
||||
**Option C: SSH-based push (rejected)**
|
||||
|
||||
Having Hive nodes SSH into each other introduces key management complexity, a lateral movement attack surface, and fragile dependency on SSH config. Not recommended.
|
||||
|
||||
#### Decision: Option A for automation, Option B as fallback
|
||||
|
||||
Option A (binary via Predastore) is the right long-term approach — the storage layer already exists, and nodes can self-serve the upgrade. Option B is the immediate path since it requires no new infrastructure.
|
||||
|
||||
### Implementation Plan
|
||||
|
||||
**Phase 2.5a: Manual rolling upgrade (Option B)**
|
||||
|
||||
CLI commands that coordinate the drain/upgrade/uncordon flow but leave the binary swap to the operator:
|
||||
|
||||
```bash
|
||||
hive admin cluster upgrade --manual
|
||||
# Prints step-by-step instructions per node:
|
||||
# 1. Draining node1... done (0 VMs migrated)
|
||||
# 2. SSH to node1 and run: sudo cp /path/to/new/hive /usr/local/bin/hive
|
||||
# 3. Press Enter when done...
|
||||
# 4. Restarting node1 services...
|
||||
# 5. Waiting for node1 health... healthy
|
||||
# 6. Proceeding to node2...
|
||||
```
|
||||
|
||||
**Phase 2.5b: Automated rolling upgrade (Option A)**
|
||||
|
||||
Add binary upload to Predastore and NATS-triggered self-upgrade:
|
||||
|
||||
```bash
|
||||
hive admin cluster upgrade --binary ./bin/hive [--version v1.2.3]
|
||||
```
|
||||
|
||||
- New NATS handler `hive.node.upgrade` on each daemon
|
||||
- Handler downloads binary from predastore, validates checksum, replaces in-place
|
||||
- Daemon exec's itself to restart with new binary (or uses systemd restart)
|
||||
|
||||
### Proposed Changes
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `cmd/hive/cmd/cluster.go` | Edit | Add `clusterUpgradeCmd` with `--manual`/`--binary` flags |
|
||||
| `hive/daemon/daemon_handlers.go` | Edit | Add `hive.node.upgrade` handler (Phase 2.5b) |
|
||||
| `hive/daemon/upgrade.go` | New | Binary download, validation, self-replace logic |
|
||||
|
||||
### Testing
|
||||
|
||||
- `make test-docker-multi`: run upgrade flow (swap binary, verify services restart)
|
||||
- Remote cluster: full rolling upgrade with `hive admin cluster upgrade --manual`
|
||||
|
||||
---
|
||||
|
||||
## Phase 2.6 — Degraded Storage Writes (Planned)
|
||||
|
||||
### Problem Statement
|
||||
|
||||
**Current behavior**: Predastore uses Reed-Solomon RS(K, M) erasure coding (default RS(3,2) for 5 nodes, RS(2,1) for 3 nodes). When writing an object, it splits into K data shards + M parity shards and sends each to a deterministic node via consistent hashing. **If any shard destination is unreachable, the entire write fails immediately.** There is no retry, no alternate node selection, no degraded write mode.
|
||||
|
||||
This means a single node being drained or offline makes **all writes fail** — even though RS coding is designed to tolerate M node failures for reads.
|
||||
|
||||
**Desired behavior**: When a node is unavailable (draining, maintenance, or crashed), writes should succeed by redirecting shards to an available alternate node. The metadata should track where shards actually landed, and a background repair process should move shards back to their correct home when the node returns.
|
||||
|
||||
### Current Architecture
|
||||
|
||||
```
|
||||
PUT /bucket/key
|
||||
→ Hash object key → consistent hash ring → [node0, node1, node2] for RS(2,1)
|
||||
→ Split into 2 data shards + 1 parity shard
|
||||
→ QUIC Put to node0 (data shard 0) ← fails if node0 is down
|
||||
→ QUIC Put to node1 (data shard 1)
|
||||
→ QUIC Put to node2 (parity shard 0)
|
||||
→ Store ObjectToShardNodes metadata in s3db (Raft)
|
||||
```
|
||||
|
||||
Key code: `putObjectViaQUIC()` in `predastore/backend/distributed/distributed.go:506`
|
||||
|
||||
### Design: Shard Redirection with Repair Queue
|
||||
|
||||
#### Write Path Changes
|
||||
|
||||
When a shard destination is unreachable:
|
||||
1. Catch the QUIC dial/write error for the failing shard
|
||||
2. Query the hash ring for the next available node (skip nodes marked as draining/maintenance/offline)
|
||||
3. Write the shard to the alternate node
|
||||
4. Update `ObjectToShardNodes` metadata to reflect actual placement
|
||||
5. Enqueue a repair job: "shard X of object Y should be on node Z, currently on node W"
|
||||
|
||||
```
|
||||
PUT with node0 down:
|
||||
→ QUIC Put to node0 → FAIL
|
||||
→ Redirect: QUIC Put to node3 (next on ring)
|
||||
→ QUIC Put to node1 (data shard 1) → OK
|
||||
→ QUIC Put to node2 (parity shard 0) → OK
|
||||
→ Store metadata: DataShardNodes=[3, 1], ParityShardNodes=[2]
|
||||
→ Enqueue repair: {object: key, shard: 0, expected: node0, actual: node3}
|
||||
```
|
||||
|
||||
#### Node Health Awareness in Predastore
|
||||
|
||||
Predastore needs to know which nodes are available. Options:
|
||||
|
||||
**Option A: Heartbeat-based (NATS KV)** — Predastore reads hive daemon heartbeats from the `hive-cluster-state` KV bucket. Requires Predastore to connect to NATS (it currently doesn't).
|
||||
|
||||
**Option B: Predastore-internal health checks** — Each Predastore node pings peers via QUIC periodically. If a peer fails to respond after N attempts, mark it as unavailable locally. Simpler, self-contained.
|
||||
|
||||
**Option C: Configuration-driven** — The `hive node drain` CLI command updates predastore config (via NATS → daemon → predastore config reload). Predastore skips nodes marked as unavailable. Relies on the Hive daemon to propagate state.
|
||||
|
||||
**Recommendation**: Option B for crash detection (self-contained, no new dependencies), Option C for planned drain/maintenance (explicit, reliable). Both can coexist.
|
||||
|
||||
#### Repair Queue
|
||||
|
||||
When the unavailable node returns:
|
||||
1. The returning node announces itself (heartbeat resumes or health check passes)
|
||||
2. A background repair goroutine scans the repair queue
|
||||
3. For each misplaced shard:
|
||||
- Read shard from alternate node
|
||||
- Write shard to correct node
|
||||
- Update `ObjectToShardNodes` metadata
|
||||
- Remove repair queue entry
|
||||
4. Optionally delete the shard from the alternate node (after confirming the correct node has it)
|
||||
|
||||
Repair queue storage: JetStream stream (`HIVE-REPAIR`) or a dedicated KV bucket. Durable, replicated, survives restarts.
|
||||
|
||||
#### Write Consistency Guarantees
|
||||
|
||||
- **Minimum write quorum**: At least K data shards + M parity shards must be written for the PUT to succeed. Redirected shards count toward the quorum.
|
||||
- **If fewer than K+M nodes are available**: Write fails (cannot satisfy RS requirements). For RS(2,1) with 3 nodes, at most 1 node can be down.
|
||||
- **Metadata is always authoritative**: `ObjectToShardNodes` records where shards actually are, not where the hash ring says they should be. Reads use metadata, not the ring.
|
||||
|
||||
### Proposed Changes
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `predastore/backend/distributed/distributed.go` | Edit | Shard redirection in `putObjectViaQUIC()`, alternate node selection |
|
||||
| `predastore/backend/distributed/repair.go` | New | Repair queue, background repair goroutine |
|
||||
| `predastore/backend/distributed/health.go` | New | Peer health checks, node availability tracking |
|
||||
| `predastore/backend/distributed/distributed.go` | Edit | `ObjectToShardNodes` — ensure metadata reflects actual placement |
|
||||
|
||||
### Testing
|
||||
|
||||
- Unit tests: write with simulated node failure, verify redirect and metadata
|
||||
- `make test-docker-multi`: stop one Predastore node, verify S3 PUTs still succeed, restart node, verify repair
|
||||
- Remote cluster: drain a node, upload files via AWS CLI, verify objects readable
|
||||
|
||||
---
|
||||
|
||||
## Phase 2.7 — VM Migration (Planned)
|
||||
|
||||
### Problem Statement
|
||||
|
||||
When a node is drained for maintenance (Phase 2.2), its running VMs are currently stopped and terminated. The user's workload goes offline until they manually launch new instances on other nodes. For a platform aiming at AWS compatibility, this is unacceptable — VMs should be transparently relocated to other nodes with minimal or zero downtime.
|
||||
|
||||
### Background: Why Hive Is Well-Suited for Migration
|
||||
|
||||
Hive's architecture makes VM migration simpler than typical hypervisors:
|
||||
|
||||
1. **Storage is already shared** — all VM disks are backed by Viperblock/Predastore over the network (NBD). Both source and destination nodes can access the same volumes. No disk data transfer needed during migration.
|
||||
2. **QMP is already implemented** — `hive/qmp/qmp.go` has a working QMP client over UNIX sockets. Migration commands are just more QMP messages.
|
||||
3. **VM state is in JetStream KV** — instance metadata (`hive-instance-state` bucket) is already shared across the cluster. Cross-node handoff just updates the KV key.
|
||||
4. **`ResetNodeLocalState()`** — `hive/vm/vm.go` already has a method that zeros out PID, PTS, QMPClient for cross-node handoff.
|
||||
|
||||
### Design: Two-Phase Migration Approach
|
||||
|
||||
#### Phase 2.7a: Cold Migration (Stop → Save → Restore)
|
||||
|
||||
Simplest path — stop the VM, save its memory state to a file, transfer the file, restore on destination. Total downtime: seconds to minutes depending on RAM size.
|
||||
|
||||
**When to use**: Planned maintenance, low-priority VMs, initial implementation.
|
||||
|
||||
```
|
||||
Source Node Destination Node
|
||||
──────────── ────────────────
|
||||
1. QMP: stop (pause vCPUs)
|
||||
2. QMP: migrate "exec:cat > /tmp/state"
|
||||
3. Wait for migration complete
|
||||
4. Transfer state file ──────────────→ 5. Receive state file
|
||||
6. Start nbdkit for same volumes
|
||||
7. Launch QEMU with -incoming "exec:cat /tmp/state"
|
||||
8. VM resumes execution
|
||||
9. Cleanup: stop nbdkit, remove state
|
||||
10. Update KV: instance now on dest
|
||||
```
|
||||
|
||||
**State file transfer options**:
|
||||
- **Predastore (S3)**: Upload state to `s3://hive-system/migrations/<instance-id>.state`, destination downloads it. Reuses existing infrastructure, works across any network topology.
|
||||
- **Direct TCP**: Pipe state directly via QEMU's `tcp:` transport. Faster (no intermediate storage), but requires direct connectivity between nodes.
|
||||
- **NATS object store**: For smaller VMs (<1GB RAM), NATS JetStream object store could work. Built-in replication.
|
||||
|
||||
**Recommendation**: Start with Predastore for simplicity (upload/download pattern), add direct TCP later for performance.
|
||||
|
||||
#### Phase 2.7b: Live Migration (Pre-Copy)
|
||||
|
||||
Minimal downtime — VM runs on source while memory is copied to destination. Final switchover pauses VM for <500ms.
|
||||
|
||||
**When to use**: Production workloads, load balancing, zero-downtime maintenance.
|
||||
|
||||
```
|
||||
Source Node Destination Node
|
||||
──────────── ────────────────
|
||||
1. Start nbdkit for same volumes
|
||||
2. Launch QEMU with -incoming defer
|
||||
3. QMP: migrate-incoming "tcp:0:PORT"
|
||||
4. QMP: migrate-set-capabilities
|
||||
[auto-converge, multifd]
|
||||
5. QMP: migrate "tcp:dest:PORT"
|
||||
6. ── pre-copy: bulk RAM transfer ──→
|
||||
7. ── iterative: dirty pages ──────→
|
||||
8. ── converged: final switchover ──→ 9. VM resumes on destination
|
||||
10. Cleanup source
|
||||
11. Update KV
|
||||
```
|
||||
|
||||
**QMP commands** (existing `SendQMPCommand` infrastructure):
|
||||
|
||||
```go
|
||||
// On source:
|
||||
d.SendQMPCommand(qmpClient, QMPCommand{
|
||||
Execute: "migrate-set-capabilities",
|
||||
Arguments: map[string]any{"capabilities": []map[string]any{
|
||||
{"capability": "auto-converge", "state": true},
|
||||
}},
|
||||
}, instanceID)
|
||||
|
||||
d.SendQMPCommand(qmpClient, QMPCommand{
|
||||
Execute: "migrate",
|
||||
Arguments: map[string]any{"uri": fmt.Sprintf("tcp:%s:%d", destHost, destPort)},
|
||||
}, instanceID)
|
||||
|
||||
// Monitor progress:
|
||||
resp := d.SendQMPCommand(qmpClient, QMPCommand{Execute: "query-migrate"}, instanceID)
|
||||
// resp.Return.status: "active", "completed", "failed"
|
||||
```
|
||||
|
||||
**Convergence**: For typical VMs (512MB-8GB RAM) on 10GbE, pre-copy converges in 5-30 seconds with <300ms switchover. `auto-converge` throttles guest vCPUs if dirty page rate exceeds transfer rate.
|
||||
|
||||
**TLS for migration streams**: Use the cluster's existing CA (generated by `dev-setup.sh` / `hive admin init`). Each node gets a dual-role TLS cert for QEMU migration. QEMU's `-object tls-creds-x509` handles the TLS handshake natively.
|
||||
|
||||
### Migration Coordination Flow (NATS)
|
||||
|
||||
```
|
||||
CLI: hive node drain node3
|
||||
→ NATS "hive.node.setmode" {node: "node3", mode: "draining"}
|
||||
|
||||
Daemon (node3): mode → draining
|
||||
→ Unsubscribe from ec2.RunInstances.* topics
|
||||
→ For each running VM:
|
||||
1. Select destination: pick node with most available resources + mode=normal
|
||||
2. NATS Request "hive.node.prepare-migration" → destination daemon
|
||||
Payload: {instanceID, volumes, vmConfig, migrationType: "live"|"cold"}
|
||||
3. Destination prepares: start nbdkit, launch QEMU with -incoming
|
||||
4. Destination responds: {ready: true, migrationEndpoint: "tcp:10.1.3.170:49152"}
|
||||
5. Source initiates QEMU migration
|
||||
6. Monitor progress, publish to "hive.cluster.migration.progress"
|
||||
7. On completion:
|
||||
- Source: cleanup local QEMU, nbdkit, update KV
|
||||
- Destination: update KV with new ownership
|
||||
8. Move to next VM
|
||||
|
||||
When all VMs migrated:
|
||||
→ mode → cordoned (ready for maintenance)
|
||||
```
|
||||
|
||||
### Destination VM Preparation
|
||||
|
||||
The destination daemon must reconstruct an identical QEMU environment:
|
||||
|
||||
1. **Mount volumes**: Send `ebs.mount` NATS requests for each of the VM's volumes. Viperblock starts nbdkit processes pointing at the same underlying data.
|
||||
2. **Build QEMU command**: Same CPU, memory, machine type, devices, network config as source. Add `-incoming defer` flag.
|
||||
3. **Network setup**: Same MAC address (preserved in migration stream), same port forwarding rules.
|
||||
4. **Start QEMU**: Process starts but vCPUs are paused, waiting for migration data.
|
||||
|
||||
Key reuse point: the VM struct's `Config.Execute()` method in `hive/vm/vm.go` already builds the QEMU command. A variant with `-incoming` is straightforward.
|
||||
|
||||
### Post-Migration Cleanup
|
||||
|
||||
On the source node after successful migration:
|
||||
1. Kill the source QEMU process (should already be exited after migration completes)
|
||||
2. Unmount volumes (send `ebs.unmount` for each volume)
|
||||
3. Remove local state (PID file, QMP socket, serial PTS)
|
||||
4. Update JetStream KV: move instance from `node.<source>.instances` to `node.<dest>.instances`
|
||||
|
||||
The existing `ResetNodeLocalState()` method zeros out PID, PTS, QMPClient — exactly what's needed for the handoff.
|
||||
|
||||
### Failure Handling
|
||||
|
||||
| Failure | During Cold Migration | During Live Migration |
|
||||
|---------|----------------------|----------------------|
|
||||
| Source dies mid-migration | State file may be incomplete; VM is lost. Restart from stopped state on any node. | Migration aborts. If source QEMU still running, VM continues on source. If not, VM lost. |
|
||||
| Destination dies mid-migration | Source VM is paused; resume on source. Retry with different destination. | Migration aborts. VM continues running on source (pre-switchover). |
|
||||
| Network partition | State file transfer fails; resume on source. | Pre-copy: migration stalls, eventually times out, VM stays on source. Post-switchover: VM on destination, source can't clean up (handled on reconnect). |
|
||||
| Storage failure | Both sides fail equally (shared storage). | Same — storage is shared, not part of migration stream. |
|
||||
|
||||
### Proposed Changes
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `hive/daemon/migration.go` | New | Migration coordinator: prepare, execute (cold/live), cleanup |
|
||||
| `hive/daemon/daemon_handlers.go` | Edit | Add `hive.node.prepare-migration` NATS handler |
|
||||
| `hive/qmp/qmp.go` | Edit | Add migration QMP command types and helpers |
|
||||
| `hive/vm/vm.go` | Edit | Add `IncomingConfig()` variant for destination QEMU launch |
|
||||
| `cmd/hive/cmd/node.go` | Edit | `hive node drain` triggers migration flow |
|
||||
|
||||
### Testing
|
||||
|
||||
- Unit tests: migration state machine, QMP command construction
|
||||
- `make test-docker-multi`: cold migration between nodes (Docker E2E doesn't have KVM so live migration won't work there — use remote cluster)
|
||||
- Remote cluster (`scripts/iac/hive-test.sh`): full live migration test
|
||||
1. Launch VM on node3
|
||||
2. SSH into VM, start a long-running process
|
||||
3. `hive node drain node3`
|
||||
4. Verify VM appears on another node
|
||||
5. SSH into VM again, verify process still running (live) or VM rebooted cleanly (cold)
|
||||
|
||||
---
|
||||
|
||||
## Implementation Priority & Dependencies
|
||||
|
||||
```
|
||||
Phase 2.8 A-B-D1 (Crash Detection, Auto-Restart, OOM Scores)
|
||||
│ ↑ no external dependencies, can start immediately
|
||||
│
|
||||
Phase 2.2 (Node Modes) ◄── Phase 2.8 C+E (QMP Health, DescribeInstanceStatus API)
|
||||
│ ↑ shares health state infrastructure with 2.2
|
||||
│
|
||||
├──→ Phase 2.7a (Cold Migration) — requires drain mode + crash detection (2.8 A)
|
||||
│ │
|
||||
│ └──→ Phase 2.7b (Live Migration) — builds on cold migration
|
||||
│
|
||||
├──→ Phase 2.5 (Rolling Upgrade) — requires drain/uncordon
|
||||
│
|
||||
├──→ Phase 2.6 (Degraded Writes) — requires node health awareness
|
||||
│ ↑ benefits from memory pressure signals (2.8 D2-D3)
|
||||
│
|
||||
└──→ Phase 2.8 D2-D3-D1b (Resource Reservation, Pressure Monitoring, cgroups)
|
||||
```
|
||||
|
||||
**Recommended order**: 2.8(A,B,D1) → 2.2 + 2.8(C,E) → 2.7a → 2.5 → 2.6 → 2.7b → 2.8(D2,D3,D1b)
|
||||
|
||||
Rationale:
|
||||
- **2.8 A,B,D1 first**: Zero external dependencies. Fixes the silent QEMU crash problem immediately. OOM scores are 2 lines of code for massive protection. See [ec2-health-restart.md](ec2-health-restart.md) for details.
|
||||
- **2.2 + 2.8 C,E together**: Node modes and health monitoring share the same state infrastructure. DescribeInstanceStatus API consumes the health data that 2.2's `monitorPeerHeartbeats()` and 2.8 C's QMP health check produce.
|
||||
- **2.7a third**: Cold migration provides immediate value — VMs survive maintenance windows. Uses crash detection (2.8 A) for failure cases.
|
||||
- **2.5 fourth**: Rolling upgrades become practical once drain + cold migration work.
|
||||
- **2.6 fifth**: Degraded writes are important but require changes to Predastore (separate repo). Can be done in parallel with 2.7a.
|
||||
- **2.7b sixth**: Live migration is the polish — sub-second downtime instead of seconds. Only worth the complexity after cold migration is proven.
|
||||
- **2.8 D2,D3,D1b last**: Operational polish. Resource reservation and cgroups prevent OOM at the source but aren't blocking for other features.
|
||||
|
||||
---
|
||||
|
||||
## Key Existing Code to Reuse
|
||||
|
||||
| Function / Type | File | Relevance |
|
||||
|----------------|------|-----------|
|
||||
| `SendQMPCommand()` | `hive/daemon/daemon.go` | QMP dispatch — add migration commands |
|
||||
| `Config.Execute()` | `hive/vm/vm.go` | QEMU command builder — add `-incoming` variant |
|
||||
| `ResetNodeLocalState()` | `hive/vm/vm.go` | Zeros PID/PTS/QMP for cross-node handoff |
|
||||
| `MountVolumes()` | `hive/daemon/daemon.go` | NBD volume setup — reuse on destination |
|
||||
| `stopInstance()` | `hive/daemon/daemon.go` | QMP shutdown + cleanup — reuse for source cleanup |
|
||||
| `collectResponses()` | `cmd/hive/cmd/get.go` | NATS fan-out pattern — reuse for node queries |
|
||||
| `loadConfigAndConnect()` | `cmd/hive/cmd/get.go` | CLI NATS connection — reuse for new CLI commands |
|
||||
| `utils.StopProcess()` | `hive/utils/utils.go` | PID-based service stop — reuse in drain/upgrade |
|
||||
| `putObjectViaQUIC()` | `predastore/backend/distributed/distributed.go` | Write path — modify for shard redirection |
|
||||
| `ObjectToShardNodes` | `predastore/backend/distributed/distributed.go` | Shard metadata — extend for actual placement |
|
||||
| `hashRing.GetClosestN()` | `predastore/backend/distributed/distributed.go` | Node selection — extend to skip unavailable nodes |
|
||||
|
||||
---
|
||||
|
||||
## Verification Strategy
|
||||
|
||||
All phases will be verified with:
|
||||
1. `make preflight` — format, vet, security, all unit tests
|
||||
2. `make test-docker-multi` — multi-node E2E in Docker
|
||||
3. Remote Proxmox cluster — real hardware validation via `scripts/iac/hive-test.sh`
|
||||
|
||||
The remote test cluster is critical for Phase 2.7b (live migration) since Docker E2E doesn't support nested KVM for QEMU-to-QEMU migration.
|
||||
|
||||
@@ -213,3 +213,202 @@ Add a Terraform E2E stage to CI that runs the full `init → plan → apply →
|
||||
- Add S3 endpoint support (Predastore already S3-compatible)
|
||||
- Implement IAM basics for multi-tenant access control
|
||||
- Add CloudWatch-compatible metrics endpoint
|
||||
|
||||
---
|
||||
|
||||
## TODO: Pre-Baked Proxmox VM Template for Test Clusters
|
||||
|
||||
### Problem
|
||||
|
||||
Spinning up a new Hive test cluster on Proxmox currently takes a long time because every new VM starts from a base Debian 12 cloud image and must:
|
||||
|
||||
1. `apt-get update && apt-get install` ~20 packages (QEMU, nbdkit, libvirt, gcc, jq, etc.)
|
||||
2. Download and install Go 1.26.0 (~150MB)
|
||||
3. Download and install AWS CLI v2 (~100MB)
|
||||
4. `git clone` hive, viperblock, and predastore repos
|
||||
5. `go mod download` for all three repos (~hundreds of MB of Go modules)
|
||||
6. `make build` / `dev-setup.sh` to compile everything
|
||||
|
||||
This takes 10-15+ minutes per node, multiplied by 3 nodes. A pre-baked template with all dependencies pre-installed would reduce cluster spin-up to: boot VM, `git pull`, `make build` (~1-2 minutes).
|
||||
|
||||
### Current State
|
||||
|
||||
The existing Proxmox IaC (`scripts/iac/proxmox/`) uses:
|
||||
- Base image: `debian-12-genericcloud-amd64-20240211-1654.img`
|
||||
- Cloud-init: installs only `qemu-guest-agent`, `net-tools`, `git`, `make`
|
||||
- Everything else is done manually via SSH after boot
|
||||
|
||||
The E2E Docker image (`tests/e2e/Dockerfile.e2e`) already has the canonical install sequence:
|
||||
```
|
||||
make install-system # QEMU, nbdkit, libvirt, gcc, jq, curl, etc.
|
||||
make install-go # Go 1.26.0
|
||||
make install-aws # AWS CLI v2
|
||||
```
|
||||
|
||||
### Approaches Considered
|
||||
|
||||
#### Option A: Packer (Recommended)
|
||||
|
||||
[Packer](https://www.packer.io/) is the industry standard for building VM images. It has a native [Proxmox builder](https://developer.hashicorp.com/packer/integrations/hashicorp/proxmox) (`proxmox-iso` / `proxmox-clone`).
|
||||
|
||||
**How it works:**
|
||||
1. Packer boots a VM from the base Debian cloud image on Proxmox
|
||||
2. Runs provisioners (shell scripts) to install dependencies
|
||||
3. Converts the VM to a Proxmox template
|
||||
4. Future VMs clone from this template
|
||||
|
||||
**Pros:**
|
||||
- Industry standard — used by Kubernetes (image-builder), HashiCorp (Nomad/Consul AMIs), and most cloud-native projects
|
||||
- Reproducible and version-controlled (`.pkr.hcl` files in git)
|
||||
- Can reuse existing Makefile targets (`make install-system install-go install-aws`)
|
||||
- Supports multi-platform (same Packer template could build Proxmox templates AND local QEMU images)
|
||||
- Integrates with existing Proxmox IaC — just change `var.os_image` to point to the new template
|
||||
- Template can be rebuilt automatically when dependencies change (CI/scheduled)
|
||||
|
||||
**Cons:**
|
||||
- Adds Packer as a dependency (though it's a single binary, same as Terraform/OpenTofu)
|
||||
- Template build takes 10-15 minutes (but only done once, not per-cluster)
|
||||
|
||||
**Proposed file structure:**
|
||||
```
|
||||
scripts/iac/packer/
|
||||
├── hive-node.pkr.hcl # Main Packer template
|
||||
├── variables.pkr.hcl # Proxmox connection vars, image settings
|
||||
├── scripts/
|
||||
│ ├── base-setup.sh # System deps (calls make install-system)
|
||||
│ ├── dev-tools.sh # Go + AWS CLI (calls make install-go install-aws)
|
||||
│ ├── clone-repos.sh # git clone hive, viperblock, predastore
|
||||
│ └── cleanup.sh # Clear apt cache, truncate logs, zero free space
|
||||
└── README.md
|
||||
```
|
||||
|
||||
**Layered template strategy:**
|
||||
- **Layer 1 — `hive-base`**: System deps + Go + AWS CLI + cloud image cached. Rebuilt monthly or when deps change. This is the slow layer (~10 min) but changes rarely.
|
||||
- **Layer 2 — `hive-dev`** (optional): Extends `hive-base` with git clone + `go mod download` + pre-built binaries. Rebuilt more frequently. Cluster spin-up becomes just `git pull && make build`.
|
||||
|
||||
Alternatively, a single template with everything is simpler and sufficient for our scale.
|
||||
|
||||
**Example Packer template sketch:**
|
||||
```hcl
|
||||
source "proxmox-iso" "hive-node" {
|
||||
proxmox_url = var.proxmox_endpoint
|
||||
node = var.proxmox_node
|
||||
iso_file = var.base_image
|
||||
os = "l26"
|
||||
cores = 4
|
||||
memory = 8192
|
||||
disk_size = "32G"
|
||||
ssh_username = "root"
|
||||
template_name = "hive-node-template"
|
||||
template_description = "Hive test node with all dependencies pre-installed"
|
||||
}
|
||||
|
||||
build {
|
||||
sources = ["source.proxmox-iso.hive-node"]
|
||||
|
||||
# Copy Makefile for install targets
|
||||
provisioner "file" {
|
||||
source = "../../../Makefile"
|
||||
destination = "/tmp/Makefile"
|
||||
}
|
||||
|
||||
provisioner "shell" {
|
||||
inline = [
|
||||
"cd /tmp && make -f Makefile install-system",
|
||||
"cd /tmp && make -f Makefile install-go",
|
||||
"cd /tmp && make -f Makefile install-aws",
|
||||
]
|
||||
}
|
||||
|
||||
# Clone repos
|
||||
provisioner "shell" {
|
||||
script = "scripts/clone-repos.sh"
|
||||
}
|
||||
|
||||
# Cleanup for smaller template
|
||||
provisioner "shell" {
|
||||
script = "scripts/cleanup.sh"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Option B: Enhanced Cloud-Init
|
||||
|
||||
Expand the existing cloud-init in `scripts/iac/proxmox/cloud-config.tf` to install everything on first boot.
|
||||
|
||||
**Pros:**
|
||||
- No extra tools — works with current Terraform setup
|
||||
- Single source of truth (cloud-config.yaml)
|
||||
|
||||
**Cons:**
|
||||
- Runs on **every** new VM, not once-and-reuse (10-15 min per node, every time)
|
||||
- Cloud-init has timeouts for long-running operations (Go download, module cache)
|
||||
- Not a template — doesn't solve the speed problem, just automates the manual steps
|
||||
- Harder to debug when things fail (cloud-init logs are buried)
|
||||
|
||||
**Verdict:** Good for basic setup (current use), but doesn't solve the speed problem.
|
||||
|
||||
#### Option C: Manual Template + Script
|
||||
|
||||
SSH into a running VM, install everything, then convert to a Proxmox template via API:
|
||||
```bash
|
||||
# On the Proxmox host:
|
||||
qm shutdown <vmid>
|
||||
qm template <vmid>
|
||||
```
|
||||
|
||||
**Pros:**
|
||||
- Simplest approach, works right now
|
||||
- No extra tools
|
||||
|
||||
**Cons:**
|
||||
- Not reproducible — "it works on my cluster" problem
|
||||
- Not version-controlled
|
||||
- Easy to forget what was installed or configured
|
||||
- Stale templates are hard to update (must redo from scratch or maintain a VM)
|
||||
|
||||
**Verdict:** Acceptable for prototyping, but should be replaced with Packer.
|
||||
|
||||
#### Option D: qemu-img + chroot (Build Locally)
|
||||
|
||||
Mount a cloud image locally with `qemu-nbd`, chroot into it, install dependencies, then upload to Proxmox.
|
||||
|
||||
**Pros:**
|
||||
- No need to boot a VM on Proxmox
|
||||
- Fast iteration locally
|
||||
|
||||
**Cons:**
|
||||
- Needs root access and `qemu-nbd` on the build machine
|
||||
- Architecture-dependent (must match target arch or use binfmt_misc)
|
||||
- Fragile — chroot environments often have networking/DNS issues
|
||||
- Not standard practice
|
||||
|
||||
**Verdict:** Interesting for CI-built images, but too complex for our needs.
|
||||
|
||||
### What Other Projects Do
|
||||
|
||||
| Project | Approach | Notes |
|
||||
|---|---|---|
|
||||
| **Kubernetes image-builder** | Packer + Ansible | Builds node images for every cloud + bare-metal. Packer templates with Ansible provisioners. |
|
||||
| **HashiCorp reference architectures** | Packer | Pre-baked AMIs/templates for Nomad, Consul, Vault with all binaries installed |
|
||||
| **Talos Linux** | Custom OS | Entire immutable OS image built from scratch — overkill for us |
|
||||
| **Flatcar / CoreOS** | Immutable OS + Ignition | Pre-built OS with container runtime, config applied at boot via Ignition |
|
||||
| **Proxmox community** | Packer (`proxmox-iso` builder) | Dominant approach in the Proxmox ecosystem |
|
||||
| **GitLab CI runners** | Packer | Pre-baked runner images with Docker, tools pre-installed |
|
||||
|
||||
### Recommendation
|
||||
|
||||
**Start with Option A (Packer)** for the reusable template, with **Option C as an interim** step if we want something working today.
|
||||
|
||||
Concrete next steps:
|
||||
1. Manually create a template once (Option C) to unblock testing immediately
|
||||
2. Codify the process as a Packer template (Option A) for reproducibility
|
||||
3. Integrate template rebuilds into CI (rebuild weekly or on Makefile dependency changes)
|
||||
|
||||
### Open Questions
|
||||
|
||||
- **Single template vs layered?** One template with everything is simpler. Two layers (base + dev) would allow faster rebuilds when only code changes but deps stay the same.
|
||||
- **Include pre-built binaries?** If the template includes compiled Hive/Viperblock/Predastore, spin-up is nearly instant but the template goes stale faster. If it only includes Go modules, `make build` takes ~30s.
|
||||
- **Go module cache strategy?** Pre-populate `~/go/pkg/mod/` in the template? Or just `go mod download` on first boot?
|
||||
- **Cloud image caching?** Pre-download the Ubuntu/Debian cloud image used for nested VMs (like the Docker E2E image does with `/root/images/ubuntu-24.04.img`)?
|
||||
- **Template naming/versioning?** `hive-node-v1`, `hive-node-YYYYMMDD`, or `hive-node-<git-short-hash>`?
|
||||
|
||||
@@ -303,7 +303,8 @@ func LoadConfig(configPath string) (*ClusterConfig, error) {
|
||||
if err := viper.ReadInConfig(); err != nil {
|
||||
return nil, fmt.Errorf("error reading config file: %w", err)
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "Using config file: %s\n", viper.ConfigFileUsed())
|
||||
//fmt.Fprintf(os.Stderr, "Using config file: %s\n", viper.ConfigFileUsed())
|
||||
|
||||
} else {
|
||||
fmt.Fprintf(os.Stderr, "Config file not found: %s, using environment variables and defaults\n", configPath)
|
||||
}
|
||||
|
||||
+32
-8
@@ -609,6 +609,11 @@ func (d *Daemon) Start() error {
|
||||
}
|
||||
d.accountService = accountSvc
|
||||
|
||||
// Protect daemon from OOM killer (prefer killing QEMU VMs instead)
|
||||
if err := utils.SetOOMScore(os.Getpid(), -500); err != nil {
|
||||
slog.Warn("Failed to set daemon OOM score", "err", err)
|
||||
}
|
||||
|
||||
d.waitForClusterReady()
|
||||
d.restoreInstances()
|
||||
|
||||
@@ -1732,6 +1737,7 @@ func (d *Daemon) StartInstance(instance *vm.VM) error {
|
||||
processChan := make(chan int, 1)
|
||||
exitChan := make(chan int, 1)
|
||||
ptsChan := make(chan int, 1)
|
||||
startupConfirmed := make(chan bool, 1)
|
||||
|
||||
go func() {
|
||||
cmd, err := instance.Config.Execute()
|
||||
@@ -1766,6 +1772,11 @@ func (d *Daemon) StartInstance(instance *vm.VM) error {
|
||||
|
||||
slog.Info("VM started successfully", "pid", cmd.Process.Pid)
|
||||
|
||||
// Set OOM score for QEMU process (prefer killing VMs over system services)
|
||||
if err := utils.SetOOMScore(cmd.Process.Pid, 500); err != nil {
|
||||
slog.Warn("Failed to set QEMU OOM score", "pid", cmd.Process.Pid, "err", err)
|
||||
}
|
||||
|
||||
// TODO: Consider workaround using QMP
|
||||
// (QEMU) query-chardev
|
||||
// {"return": [{"frontend-open": true, "filename": "vc", "label": "parallel0"}, {"frontend-open": true, "filename": "unix:/run/user/1000/qmp-i-150340b52b20c0b43.sock,server=on", "label": "compat_monitor0"}, {"frontend-open": true, "filename": "pty:/dev/pts/9", "label": "serial0"}]}
|
||||
@@ -1813,15 +1824,27 @@ func (d *Daemon) StartInstance(instance *vm.VM) error {
|
||||
|
||||
processChan <- cmd.Process.Pid
|
||||
|
||||
// Read the pts from launch
|
||||
err = cmd.Wait()
|
||||
// Block until QEMU exits
|
||||
waitErr := cmd.Wait()
|
||||
|
||||
if err != nil {
|
||||
slog.Error("Failed to wait for VM:", "err", err)
|
||||
exitChan <- 1
|
||||
return
|
||||
if waitErr != nil {
|
||||
slog.Error("VM process exited", "instance", instance.ID, "err", waitErr)
|
||||
}
|
||||
|
||||
// Signal startup check (non-blocking)
|
||||
select {
|
||||
case exitChan <- 1:
|
||||
default:
|
||||
}
|
||||
|
||||
// Wait for startup phase to complete before deciding on crash handling
|
||||
confirmed := <-startupConfirmed
|
||||
if !confirmed {
|
||||
return // Startup failed, LaunchInstance handles the error
|
||||
}
|
||||
|
||||
// Runtime crash — handle it
|
||||
d.handleInstanceCrash(instance, waitErr)
|
||||
}()
|
||||
|
||||
// Wait for startup result
|
||||
@@ -1843,16 +1866,17 @@ func (d *Daemon) StartInstance(instance *vm.VM) error {
|
||||
//return fmt.Errorf("failed to get pts")
|
||||
}
|
||||
|
||||
// Check if nbdkit exited immediately with an error
|
||||
// Check if QEMU exited immediately with an error
|
||||
select {
|
||||
case exitErr := <-exitChan:
|
||||
startupConfirmed <- false // tell goroutine not to handle crash
|
||||
if exitErr != 0 {
|
||||
errorMsg := fmt.Errorf("failed: %v", exitErr)
|
||||
slog.Error("Failed to launch qemu", "err", errorMsg)
|
||||
return errorMsg
|
||||
}
|
||||
default:
|
||||
// nbdkit is still running after 1 second, which means it started successfully
|
||||
startupConfirmed <- true // goroutine will handle future crashes
|
||||
slog.Info("QEMU started successfully and is running", "pts", pts)
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,274 @@
|
||||
package daemon
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"os/exec"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/hive/hive/vm"
|
||||
)
|
||||
|
||||
const (
|
||||
maxRestartsInWindow = 3
|
||||
restartWindow = 10 * time.Minute
|
||||
restartBackoffBase = 5 * time.Second
|
||||
restartBackoffMax = 2 * time.Minute
|
||||
)
|
||||
|
||||
// classifyCrashReason extracts a human-readable crash reason from the error
|
||||
// returned by cmd.Wait(). Uses exec.ExitError + syscall.WaitStatus to
|
||||
// distinguish OOM kills (SIGKILL), segfaults (SIGSEGV), etc.
|
||||
func classifyCrashReason(waitErr error) string {
|
||||
if waitErr == nil {
|
||||
return "clean-exit"
|
||||
}
|
||||
|
||||
var exitErr *exec.ExitError
|
||||
if !errors.As(waitErr, &exitErr) {
|
||||
return "unknown"
|
||||
}
|
||||
|
||||
status, ok := exitErr.Sys().(syscall.WaitStatus)
|
||||
if !ok {
|
||||
return "unknown"
|
||||
}
|
||||
|
||||
if status.Signaled() {
|
||||
switch status.Signal() {
|
||||
case syscall.SIGKILL:
|
||||
return "oom-killed"
|
||||
case syscall.SIGSEGV:
|
||||
return "segfault"
|
||||
case syscall.SIGABRT:
|
||||
return "abort"
|
||||
default:
|
||||
return fmt.Sprintf("signal-%d", status.Signal())
|
||||
}
|
||||
}
|
||||
|
||||
if status.Exited() {
|
||||
return fmt.Sprintf("exit-%d", status.ExitStatus())
|
||||
}
|
||||
|
||||
return "unknown"
|
||||
}
|
||||
|
||||
// handleInstanceCrash is called by the QEMU launch goroutine after cmd.Wait()
|
||||
// returns during runtime (after startup was confirmed successful). It detects
|
||||
// the crash reason, transitions the instance to error state, cleans up resources,
|
||||
// and schedules a restart if policy allows.
|
||||
func (d *Daemon) handleInstanceCrash(instance *vm.VM, waitErr error) {
|
||||
// Guard: if instance is not running, this was an expected exit
|
||||
// (stopInstance/terminateInstance set status before QEMU exits)
|
||||
d.Instances.Mu.Lock()
|
||||
status := instance.Status
|
||||
d.Instances.Mu.Unlock()
|
||||
|
||||
if status != vm.StateRunning {
|
||||
slog.Debug("QEMU exited but instance not in running state, skipping crash handler",
|
||||
"instance", instance.ID, "status", status)
|
||||
return
|
||||
}
|
||||
|
||||
// Guard: coordinated shutdown in progress
|
||||
if d.shuttingDown.Load() {
|
||||
slog.Debug("QEMU exited during coordinated shutdown, skipping crash handler",
|
||||
"instance", instance.ID)
|
||||
return
|
||||
}
|
||||
|
||||
reason := classifyCrashReason(waitErr)
|
||||
slog.Error("VM process crashed", "instance", instance.ID, "reason", reason, "err", waitErr)
|
||||
|
||||
// Transition to error state
|
||||
if err := d.TransitionState(instance, vm.StateError); err != nil {
|
||||
slog.Error("Failed to transition crashed instance to error state",
|
||||
"instance", instance.ID, "err", err)
|
||||
}
|
||||
|
||||
// Update health tracking
|
||||
now := time.Now()
|
||||
d.Instances.Mu.Lock()
|
||||
instance.Health.CrashCount++
|
||||
instance.Health.LastCrashTime = now
|
||||
instance.Health.LastCrashReason = reason
|
||||
if instance.Health.FirstCrashTime.IsZero() {
|
||||
instance.Health.FirstCrashTime = now
|
||||
}
|
||||
instance.Running = false
|
||||
instance.PID = 0
|
||||
d.Instances.Mu.Unlock()
|
||||
|
||||
// Deallocate resources to fix phantom reservation
|
||||
instanceType, ok := d.resourceMgr.instanceTypes[instance.InstanceType]
|
||||
if ok && instanceType != nil {
|
||||
slog.Info("Deallocating resources for crashed instance",
|
||||
"instance", instance.ID, "type", instance.InstanceType)
|
||||
d.resourceMgr.deallocate(instanceType)
|
||||
}
|
||||
|
||||
// Clean up stale QMP socket so QEMU can rebind on restart
|
||||
if instance.Config.QMPSocket != "" {
|
||||
os.Remove(instance.Config.QMPSocket)
|
||||
}
|
||||
|
||||
// Unmount EBS volumes (same pattern as stopInstance)
|
||||
d.unmountInstanceVolumes(instance)
|
||||
|
||||
// Persist state
|
||||
if err := d.WriteState(); err != nil {
|
||||
slog.Error("Failed to persist state after crash handling",
|
||||
"instance", instance.ID, "err", err)
|
||||
}
|
||||
|
||||
d.maybeRestartInstance(instance)
|
||||
}
|
||||
|
||||
// unmountInstanceVolumes sends NATS unmount requests for all volumes attached
|
||||
// to the instance and updates their state to "available".
|
||||
func (d *Daemon) unmountInstanceVolumes(instance *vm.VM) {
|
||||
instance.EBSRequests.Mu.Lock()
|
||||
defer instance.EBSRequests.Mu.Unlock()
|
||||
|
||||
for _, ebsRequest := range instance.EBSRequests.Requests {
|
||||
ebsUnMountRequest, err := json.Marshal(ebsRequest)
|
||||
if err != nil {
|
||||
slog.Error("Failed to marshal volume payload for crash cleanup",
|
||||
"err", err)
|
||||
continue
|
||||
}
|
||||
|
||||
msg, err := d.natsConn.Request(d.ebsTopic("unmount"), ebsUnMountRequest, 30*time.Second)
|
||||
if err != nil {
|
||||
slog.Error("Failed to unmount volume after crash",
|
||||
"name", ebsRequest.Name, "instance", instance.ID, "err", err)
|
||||
} else {
|
||||
slog.Info("Unmounted volume after crash",
|
||||
"instance", instance.ID, "volume", ebsRequest.Name, "data", string(msg.Data))
|
||||
}
|
||||
|
||||
// Update user-visible volume state to "available"
|
||||
if !ebsRequest.EFI && !ebsRequest.CloudInit {
|
||||
if err := d.volumeService.UpdateVolumeState(ebsRequest.Name, "available", "", ""); err != nil {
|
||||
slog.Error("Failed to update volume state to available after crash",
|
||||
"volumeId", ebsRequest.Name, "err", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// maybeRestartInstance checks restart policy and schedules a restart if allowed.
|
||||
func (d *Daemon) maybeRestartInstance(instance *vm.VM) {
|
||||
if d.shuttingDown.Load() {
|
||||
slog.Info("Skipping restart during shutdown", "instance", instance.ID)
|
||||
return
|
||||
}
|
||||
|
||||
now := time.Now()
|
||||
|
||||
d.Instances.Mu.Lock()
|
||||
health := &instance.Health
|
||||
|
||||
// If crashes are outside the restart window, reset counters
|
||||
if !health.FirstCrashTime.IsZero() && now.Sub(health.FirstCrashTime) > restartWindow {
|
||||
slog.Info("Crash window expired, resetting counters", "instance", instance.ID)
|
||||
health.CrashCount = 1
|
||||
health.FirstCrashTime = now
|
||||
health.RestartCount = 0
|
||||
}
|
||||
|
||||
// Check if we've exceeded the max restarts in the window
|
||||
if health.CrashCount > maxRestartsInWindow {
|
||||
d.Instances.Mu.Unlock()
|
||||
slog.Error("Instance exceeded max restarts in window, leaving in error state",
|
||||
"instance", instance.ID,
|
||||
"crashes", health.CrashCount,
|
||||
"window", restartWindow,
|
||||
"max", maxRestartsInWindow)
|
||||
return
|
||||
}
|
||||
|
||||
restartCount := health.RestartCount
|
||||
d.Instances.Mu.Unlock()
|
||||
|
||||
// Check resource availability
|
||||
instanceType, ok := d.resourceMgr.instanceTypes[instance.InstanceType]
|
||||
if !ok || instanceType == nil {
|
||||
slog.Error("Unknown instance type, cannot restart",
|
||||
"instance", instance.ID, "type", instance.InstanceType)
|
||||
return
|
||||
}
|
||||
|
||||
if d.resourceMgr.canAllocate(instanceType, 1) < 1 {
|
||||
slog.Error("Insufficient resources to restart instance",
|
||||
"instance", instance.ID, "type", instance.InstanceType)
|
||||
return
|
||||
}
|
||||
|
||||
// Calculate exponential backoff
|
||||
delay := restartBackoffBase
|
||||
for i := 0; i < restartCount; i++ {
|
||||
delay *= 2
|
||||
if delay > restartBackoffMax {
|
||||
delay = restartBackoffMax
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
slog.Info("Scheduling instance restart",
|
||||
"instance", instance.ID,
|
||||
"delay", delay,
|
||||
"restartCount", restartCount+1)
|
||||
|
||||
time.AfterFunc(delay, func() {
|
||||
d.restartCrashedInstance(instance)
|
||||
})
|
||||
}
|
||||
|
||||
// restartCrashedInstance re-verifies the instance is still in error state
|
||||
// and relaunches it via LaunchInstance.
|
||||
func (d *Daemon) restartCrashedInstance(instance *vm.VM) {
|
||||
d.Instances.Mu.Lock()
|
||||
if instance.Status != vm.StateError {
|
||||
d.Instances.Mu.Unlock()
|
||||
slog.Info("Instance no longer in error state, skipping restart",
|
||||
"instance", instance.ID, "status", instance.Status)
|
||||
return
|
||||
}
|
||||
|
||||
if d.shuttingDown.Load() {
|
||||
d.Instances.Mu.Unlock()
|
||||
slog.Info("Daemon shutting down, skipping restart", "instance", instance.ID)
|
||||
return
|
||||
}
|
||||
|
||||
instance.Health.RestartCount++
|
||||
d.Instances.Mu.Unlock()
|
||||
|
||||
slog.Info("Restarting crashed instance",
|
||||
"instance", instance.ID,
|
||||
"restartCount", instance.Health.RestartCount)
|
||||
|
||||
// Transition Error → Pending (valid now that we added it to the transitions map)
|
||||
if err := d.TransitionState(instance, vm.StatePending); err != nil {
|
||||
slog.Error("Failed to transition instance to pending for restart",
|
||||
"instance", instance.ID, "err", err)
|
||||
return
|
||||
}
|
||||
|
||||
// LaunchInstance handles the full relaunch flow:
|
||||
// check PID → MountVolumes → StartInstance → CreateQMPClient → NATS subscribe → Running
|
||||
if err := d.LaunchInstance(instance); err != nil {
|
||||
slog.Error("Failed to restart crashed instance",
|
||||
"instance", instance.ID, "err", err)
|
||||
if err := d.TransitionState(instance, vm.StateError); err != nil {
|
||||
slog.Error("Failed to transition instance back to error after restart failure",
|
||||
"instance", instance.ID, "err", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,202 @@
|
||||
package daemon
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/hive/hive/config"
|
||||
"github.com/mulgadc/hive/hive/vm"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestClassifyCrashReason_CleanExit(t *testing.T) {
|
||||
assert.Equal(t, "clean-exit", classifyCrashReason(nil))
|
||||
}
|
||||
|
||||
func TestClassifyCrashReason_OOMKill(t *testing.T) {
|
||||
// Start a process and kill it with SIGKILL to get a real ExitError
|
||||
cmd := exec.Command("sleep", "60")
|
||||
require.NoError(t, cmd.Start())
|
||||
|
||||
require.NoError(t, cmd.Process.Kill()) // sends SIGKILL
|
||||
waitErr := cmd.Wait()
|
||||
require.Error(t, waitErr)
|
||||
|
||||
assert.Equal(t, "oom-killed", classifyCrashReason(waitErr))
|
||||
}
|
||||
|
||||
func TestClassifyCrashReason_ExitCode(t *testing.T) {
|
||||
// Run a command that exits with a non-zero code
|
||||
cmd := exec.Command("sh", "-c", "exit 42")
|
||||
waitErr := cmd.Run()
|
||||
require.Error(t, waitErr)
|
||||
|
||||
assert.Equal(t, "exit-42", classifyCrashReason(waitErr))
|
||||
}
|
||||
|
||||
func TestClassifyCrashReason_Unknown(t *testing.T) {
|
||||
// A non-ExitError error
|
||||
err := fmt.Errorf("some random error")
|
||||
assert.Equal(t, "unknown", classifyCrashReason(err))
|
||||
}
|
||||
|
||||
func TestHandleInstanceCrash_SkipsNonRunning(t *testing.T) {
|
||||
nc, err := nats.Connect(sharedNATSURL)
|
||||
require.NoError(t, err)
|
||||
defer nc.Close()
|
||||
|
||||
d := &Daemon{
|
||||
natsConn: nc,
|
||||
resourceMgr: NewResourceManager(),
|
||||
Instances: vm.Instances{VMS: make(map[string]*vm.VM)},
|
||||
}
|
||||
|
||||
instance := &vm.VM{
|
||||
ID: "i-test-stopped",
|
||||
Status: vm.StateStopped,
|
||||
}
|
||||
d.Instances.VMS[instance.ID] = instance
|
||||
|
||||
// Should return without action (no panic, no state change)
|
||||
d.handleInstanceCrash(instance, fmt.Errorf("test error"))
|
||||
assert.Equal(t, vm.StateStopped, instance.Status)
|
||||
}
|
||||
|
||||
func TestHandleInstanceCrash_SkipsShuttingDown(t *testing.T) {
|
||||
nc, err := nats.Connect(sharedNATSURL)
|
||||
require.NoError(t, err)
|
||||
defer nc.Close()
|
||||
|
||||
d := &Daemon{
|
||||
natsConn: nc,
|
||||
resourceMgr: NewResourceManager(),
|
||||
Instances: vm.Instances{VMS: make(map[string]*vm.VM)},
|
||||
}
|
||||
d.shuttingDown.Store(true)
|
||||
|
||||
instance := &vm.VM{
|
||||
ID: "i-test-shutdown",
|
||||
Status: vm.StateRunning,
|
||||
}
|
||||
d.Instances.VMS[instance.ID] = instance
|
||||
|
||||
d.handleInstanceCrash(instance, fmt.Errorf("test error"))
|
||||
|
||||
// Status unchanged because shuttingDown is true
|
||||
assert.Equal(t, vm.StateRunning, instance.Status)
|
||||
}
|
||||
|
||||
func TestMaybeRestart_ExceedsMaxInWindow(t *testing.T) {
|
||||
nc, err := nats.Connect(sharedNATSURL)
|
||||
require.NoError(t, err)
|
||||
defer nc.Close()
|
||||
|
||||
d := &Daemon{
|
||||
node: "test-node",
|
||||
natsConn: nc,
|
||||
config: &config.Config{
|
||||
Services: []string{"daemon"},
|
||||
},
|
||||
resourceMgr: NewResourceManager(),
|
||||
Instances: vm.Instances{VMS: make(map[string]*vm.VM)},
|
||||
}
|
||||
|
||||
// Find an allocatable instance type
|
||||
var allocType string
|
||||
for typeName := range d.resourceMgr.instanceTypes {
|
||||
allocType = typeName
|
||||
break
|
||||
}
|
||||
require.NotEmpty(t, allocType)
|
||||
|
||||
instance := &vm.VM{
|
||||
ID: "i-test-maxcrash",
|
||||
Status: vm.StateError,
|
||||
InstanceType: allocType,
|
||||
Health: vm.InstanceHealthState{
|
||||
CrashCount: maxRestartsInWindow + 1, // exceeds limit
|
||||
FirstCrashTime: time.Now(),
|
||||
RestartCount: maxRestartsInWindow,
|
||||
},
|
||||
}
|
||||
d.Instances.VMS[instance.ID] = instance
|
||||
|
||||
// Should not schedule a restart (no panic, instance stays in error)
|
||||
d.maybeRestartInstance(instance)
|
||||
|
||||
// Instance should remain in error state
|
||||
assert.Equal(t, vm.StateError, instance.Status)
|
||||
}
|
||||
|
||||
func TestMaybeRestart_ResetsAfterWindow(t *testing.T) {
|
||||
nc, err := nats.Connect(sharedNATSURL)
|
||||
require.NoError(t, err)
|
||||
defer nc.Close()
|
||||
|
||||
d := &Daemon{
|
||||
node: "test-node",
|
||||
natsConn: nc,
|
||||
config: &config.Config{
|
||||
Services: []string{"daemon"},
|
||||
},
|
||||
resourceMgr: NewResourceManager(),
|
||||
Instances: vm.Instances{VMS: make(map[string]*vm.VM)},
|
||||
}
|
||||
|
||||
// Find an allocatable instance type
|
||||
var allocType string
|
||||
for typeName := range d.resourceMgr.instanceTypes {
|
||||
allocType = typeName
|
||||
break
|
||||
}
|
||||
require.NotEmpty(t, allocType)
|
||||
|
||||
instance := &vm.VM{
|
||||
ID: "i-test-windowreset",
|
||||
Status: vm.StateError,
|
||||
InstanceType: allocType,
|
||||
Health: vm.InstanceHealthState{
|
||||
CrashCount: 5,
|
||||
FirstCrashTime: time.Now().Add(-restartWindow - time.Minute), // outside window
|
||||
RestartCount: 5,
|
||||
},
|
||||
}
|
||||
d.Instances.VMS[instance.ID] = instance
|
||||
|
||||
// maybeRestartInstance should reset counters and schedule a restart
|
||||
// (restart will fail because there's no real QEMU, but counters should reset)
|
||||
d.maybeRestartInstance(instance)
|
||||
|
||||
// Counters should be reset
|
||||
assert.Equal(t, 1, instance.Health.CrashCount)
|
||||
assert.Equal(t, 0, instance.Health.RestartCount)
|
||||
}
|
||||
|
||||
func TestRestartBackoff_Exponential(t *testing.T) {
|
||||
// Verify the backoff calculation logic
|
||||
expected := []time.Duration{
|
||||
5 * time.Second, // restart 0
|
||||
10 * time.Second, // restart 1
|
||||
20 * time.Second, // restart 2
|
||||
40 * time.Second, // restart 3
|
||||
80 * time.Second, // restart 4
|
||||
120 * time.Second, // restart 5 (capped at 2min)
|
||||
120 * time.Second, // restart 6 (capped at 2min)
|
||||
}
|
||||
|
||||
for i, want := range expected {
|
||||
delay := restartBackoffBase
|
||||
for j := 0; j < i; j++ {
|
||||
delay *= 2
|
||||
if delay > restartBackoffMax {
|
||||
delay = restartBackoffMax
|
||||
break
|
||||
}
|
||||
}
|
||||
assert.Equal(t, want, delay, "Backoff mismatch at restart count %d", i)
|
||||
}
|
||||
}
|
||||
@@ -51,7 +51,7 @@ func (m *JetStreamManager) InitKVBucket() error {
|
||||
if err != nil {
|
||||
if errors.Is(err, nats.ErrBucketNotFound) {
|
||||
// Bucket doesn't exist, create it
|
||||
slog.Info("Creating JetStream KV bucket", "bucket", InstanceStateBucket, "replicas", m.replicas)
|
||||
slog.Debug("Creating JetStream KV bucket", "bucket", InstanceStateBucket, "replicas", m.replicas)
|
||||
kv, err = m.js.CreateKeyValue(&nats.KeyValueConfig{
|
||||
Bucket: InstanceStateBucket,
|
||||
Description: "Hive instance state storage",
|
||||
@@ -65,7 +65,7 @@ func (m *JetStreamManager) InitKVBucket() error {
|
||||
return err
|
||||
}
|
||||
} else {
|
||||
slog.Info("Connected to existing JetStream KV bucket", "bucket", InstanceStateBucket)
|
||||
slog.Debug("Connected to existing JetStream KV bucket", "bucket", InstanceStateBucket)
|
||||
}
|
||||
|
||||
m.kv = kv
|
||||
@@ -77,7 +77,7 @@ func (m *JetStreamManager) InitClusterStateBucket() error {
|
||||
kv, err := m.js.KeyValue(ClusterStateBucket)
|
||||
if err != nil {
|
||||
if errors.Is(err, nats.ErrBucketNotFound) {
|
||||
slog.Info("Creating JetStream KV bucket", "bucket", ClusterStateBucket, "replicas", m.replicas)
|
||||
slog.Debug("Creating JetStream KV bucket", "bucket", ClusterStateBucket, "replicas", m.replicas)
|
||||
kv, err = m.js.CreateKeyValue(&nats.KeyValueConfig{
|
||||
Bucket: ClusterStateBucket,
|
||||
Description: "Hive cluster state (heartbeats, shutdown markers, service maps)",
|
||||
@@ -92,7 +92,7 @@ func (m *JetStreamManager) InitClusterStateBucket() error {
|
||||
return err
|
||||
}
|
||||
} else {
|
||||
slog.Info("Connected to existing JetStream KV bucket", "bucket", ClusterStateBucket)
|
||||
slog.Debug("Connected to existing JetStream KV bucket", "bucket", ClusterStateBucket)
|
||||
}
|
||||
|
||||
m.clusterKV = kv
|
||||
|
||||
+3
-3
@@ -17,10 +17,10 @@ func ConnectNATS(host, token string) (*nats.Conn, error) {
|
||||
nats.ReconnectWait(time.Second),
|
||||
nats.MaxReconnects(-1),
|
||||
nats.DisconnectErrHandler(func(nc *nats.Conn, err error) {
|
||||
slog.Warn("NATS disconnected", "err", err)
|
||||
slog.Debug("NATS disconnected", "err", err)
|
||||
}),
|
||||
nats.ReconnectHandler(func(nc *nats.Conn) {
|
||||
slog.Info("NATS reconnected", "url", nc.ConnectedUrl())
|
||||
slog.Debug("NATS reconnected", "url", nc.ConnectedUrl())
|
||||
}),
|
||||
}
|
||||
|
||||
@@ -33,7 +33,7 @@ func ConnectNATS(host, token string) (*nats.Conn, error) {
|
||||
return nil, fmt.Errorf("NATS connect failed: %w", err)
|
||||
}
|
||||
|
||||
slog.Info("Connected to NATS server", "host", host)
|
||||
slog.Debug("Connected to NATS server", "host", host)
|
||||
return nc, nil
|
||||
}
|
||||
|
||||
|
||||
@@ -169,6 +169,17 @@ var AvailableImages = map[string]Images{
|
||||
|
||||
}
|
||||
|
||||
// SetOOMScore sets the OOM score adjustment for a process.
|
||||
// Score range: -1000 (never kill) to 1000 (always kill first).
|
||||
// Linux-only; returns an error on non-Linux systems.
|
||||
func SetOOMScore(pid int, score int) error {
|
||||
if runtime.GOOS != "linux" {
|
||||
return fmt.Errorf("OOM score adjustment is only supported on Linux")
|
||||
}
|
||||
path := fmt.Sprintf("/proc/%d/oom_score_adj", pid)
|
||||
return os.WriteFile(path, []byte(strconv.Itoa(score)), 0644)
|
||||
}
|
||||
|
||||
func ReadPidFile(name string) (int, error) {
|
||||
|
||||
pidPath := pidPath()
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"runtime"
|
||||
"strings"
|
||||
"syscall"
|
||||
"testing"
|
||||
@@ -826,3 +827,40 @@ func TestGenerateSocketFile_EmptyName(t *testing.T) {
|
||||
_, err := GenerateSocketFile("")
|
||||
assert.Error(t, err)
|
||||
}
|
||||
|
||||
func TestSetOOMScore(t *testing.T) {
|
||||
if runtime.GOOS != "linux" {
|
||||
t.Skip("OOM score adjustment only supported on Linux")
|
||||
}
|
||||
|
||||
// Read current value so we can set something higher (unprivileged processes
|
||||
// can only increase their OOM score, not decrease it)
|
||||
pid := os.Getpid()
|
||||
current, err := os.ReadFile(fmt.Sprintf("/proc/%d/oom_score_adj", pid))
|
||||
if err != nil {
|
||||
t.Skipf("Cannot read OOM score: %v", err)
|
||||
}
|
||||
|
||||
// Set a positive score (always allowed for unprivileged processes)
|
||||
err = SetOOMScore(pid, 100)
|
||||
if err != nil {
|
||||
t.Skipf("Insufficient permissions to set OOM score: %v", err)
|
||||
}
|
||||
|
||||
// Read back and verify
|
||||
data, err := os.ReadFile(fmt.Sprintf("/proc/%d/oom_score_adj", pid))
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, "100", strings.TrimSpace(string(data)))
|
||||
|
||||
// Best-effort restore (may fail without privileges if original was lower)
|
||||
_ = os.WriteFile(fmt.Sprintf("/proc/%d/oom_score_adj", pid), current, 0644)
|
||||
}
|
||||
|
||||
func TestSetOOMScore_InvalidPID(t *testing.T) {
|
||||
if runtime.GOOS != "linux" {
|
||||
t.Skip("OOM score adjustment only supported on Linux")
|
||||
}
|
||||
|
||||
err := SetOOMScore(999999999, 100)
|
||||
assert.Error(t, err)
|
||||
}
|
||||
|
||||
+1
-1
@@ -45,7 +45,7 @@ var ValidTransitions = map[InstanceState][]InstanceState{
|
||||
StateStopping: {StateStopped, StateShuttingDown, StateError},
|
||||
StateStopped: {StateRunning, StateShuttingDown, StateError},
|
||||
StateShuttingDown: {StateTerminated, StateError},
|
||||
StateError: {StateRunning, StateShuttingDown},
|
||||
StateError: {StatePending, StateRunning, StateShuttingDown},
|
||||
}
|
||||
|
||||
// IsValidTransition checks whether moving from current to target is allowed.
|
||||
|
||||
@@ -8,12 +8,22 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/hive/hive/config"
|
||||
"github.com/mulgadc/hive/hive/qmp"
|
||||
)
|
||||
|
||||
// InstanceHealthState tracks crash detection and auto-restart metadata for a VM.
|
||||
type InstanceHealthState struct {
|
||||
CrashCount int `json:"crash_count"`
|
||||
LastCrashTime time.Time `json:"last_crash_time,omitempty"`
|
||||
LastCrashReason string `json:"last_crash_reason,omitempty"`
|
||||
RestartCount int `json:"restart_count"`
|
||||
FirstCrashTime time.Time `json:"first_crash_time,omitempty"`
|
||||
}
|
||||
|
||||
type VM struct {
|
||||
ID string `json:"id"`
|
||||
PID int `json:"pid"`
|
||||
@@ -47,6 +57,9 @@ type VM struct {
|
||||
|
||||
// Metadata server address (e.g., "127.0.0.1:12345") for EC2 metadata service
|
||||
MetadataServerAddress string `json:"metadata_server_address,omitempty"`
|
||||
|
||||
// Health tracks crash detection and auto-restart state
|
||||
Health InstanceHealthState `json:"health"`
|
||||
}
|
||||
|
||||
// ResetNodeLocalState zeroes out fields that are specific to the daemon node
|
||||
|
||||
@@ -6,3 +6,4 @@ Collection of OpenTofu/Terraform templates for creating Hive instances.
|
||||
|---|---|
|
||||
| [proxmox/](proxmox/) | Provisions a 3-node Hive cluster on Proxmox VE bare-metal hypervisors |
|
||||
| [aws/](aws/) | Provisions infrastructure using the Terraform AWS provider, with MulgaOS AWS-compatible API as the local endpoint |
|
||||
| [proxmox-template/](proxmox-template/) | Builds a Proxmox VM template with all Hive dependencies pre-installed (QEMU, Go, repos, modules) |
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
# Proxmox Template Builder — Environment Configuration
|
||||
#
|
||||
# Copy to .env and fill in your values:
|
||||
# cp .env.example .env
|
||||
#
|
||||
# Then source it before running tofu:
|
||||
# source .env && tofu plan
|
||||
|
||||
# --- Proxmox API ---
|
||||
|
||||
# API token for Proxmox VE (read by the provider directly)
|
||||
export PROXMOX_VE_API_TOKEN="terraform@pve!provider=YOUR_TOKEN_HERE"
|
||||
|
||||
# Proxmox VE API endpoint URL
|
||||
export TF_VAR_proxmox_endpoint="https://pve-host.example.com:8006/"
|
||||
|
||||
# --- Proxmox Node ---
|
||||
# Build the template on ONE node. After creation, use vzdump to copy to others.
|
||||
#
|
||||
# name - Proxmox node name (as shown in the Proxmox UI)
|
||||
# address - SSH-reachable hostname or IP of the Proxmox host
|
||||
|
||||
export TF_VAR_proxmox_node_name="node1"
|
||||
export TF_VAR_proxmox_node_address="node1.example.com"
|
||||
|
||||
# --- SSH Access ---
|
||||
|
||||
# SSH username on Proxmox hosts (used by provider for file uploads + qm commands)
|
||||
export TF_VAR_proxmox_ssh_username="terraform"
|
||||
|
||||
# Path to SSH private key for Proxmox host access
|
||||
export TF_VAR_ssh_private_key_path="~/.ssh/proxmox-host-tf"
|
||||
|
||||
# Path to SSH private key for VM access (must be the pair of the public key below)
|
||||
export TF_VAR_vm_ssh_private_key_path="~/.ssh/cloud-init"
|
||||
|
||||
# Path to SSH public key injected into VMs via cloud-init (for tf-user)
|
||||
export TF_VAR_ssh_public_key_path="~/.ssh/cloud-init.pub"
|
||||
|
||||
# --- Template ---
|
||||
|
||||
# VM ID for the template (pick a high number to avoid conflicts)
|
||||
export TF_VAR_template_vmid=9000
|
||||
|
||||
# Short tag for the base image (used in Proxmox template tags)
|
||||
export TF_VAR_base_image_tag="debian-12"
|
||||
|
||||
# Storage and network for the build node
|
||||
export TF_VAR_datastore_id="local-lvm"
|
||||
export TF_VAR_bridge="vmbr0"
|
||||
|
||||
# --- Optional ---
|
||||
|
||||
# Base image — change this when testing a newer Debian/Ubuntu version:
|
||||
# export TF_VAR_base_image="local:iso/debian-13-genericcloud-amd64.img"
|
||||
# export TF_VAR_base_image_tag="debian-13"
|
||||
# export TF_VAR_template_vmid=9010
|
||||
|
||||
# Clone a specific branch instead of main:
|
||||
# export TF_VAR_git_branch="feature/my-branch"
|
||||
@@ -0,0 +1,6 @@
|
||||
.env
|
||||
.terraform/
|
||||
.terraform.lock.hcl
|
||||
*.tfstate
|
||||
*.tfstate.backup
|
||||
*.tfplan
|
||||
@@ -0,0 +1,124 @@
|
||||
# Hive Node Template Builder
|
||||
|
||||
Builds a Proxmox VM template with all Hive dependencies pre-installed. Cloning from this template reduces cluster spin-up from ~15 minutes to ~1-2 minutes (`git pull` + `make build`).
|
||||
|
||||
## What's in the template
|
||||
|
||||
| Component | Details |
|
||||
|---|---|
|
||||
| System packages | QEMU, nbdkit, libvirt, gcc, jq, curl, etc. (`make install-system`) |
|
||||
| Go | 1.26.0 (`make install-go`) |
|
||||
| AWS CLI | v2 (`make install-aws`) |
|
||||
| Repositories | hive, viperblock, predastore (with `go.work` configured) |
|
||||
| Module cache | `go mod download` completed for all repos |
|
||||
| Cloud images | Ubuntu 24.04 for nested VMs (`~/images/ubuntu-24.04.img`) |
|
||||
| Tuning | sysctl (rmem_max/wmem_max), kvm group, ufw disabled |
|
||||
| Generalized | Clean machine-id, SSH host keys, cloud-init — each clone boots fresh |
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
cd scripts/iac/proxmox-template
|
||||
|
||||
# 1. Configure
|
||||
cp .env.example .env
|
||||
# Edit .env — set your Proxmox endpoint, API token, node, and SSH key paths.
|
||||
|
||||
# 2. Set API token (not stored in .env for security)
|
||||
export PROXMOX_VE_API_TOKEN="terraform@pve!provider=YOUR_TOKEN"
|
||||
|
||||
# 3. Build
|
||||
source .env
|
||||
tofu init
|
||||
tofu plan
|
||||
tofu apply
|
||||
```
|
||||
|
||||
Build takes ~10-15 minutes (apt packages, Go, cloud image download). When done, you'll have a template on the build node at the configured VMID (default 9000).
|
||||
|
||||
## Distribute to Other Nodes
|
||||
|
||||
After `tofu apply`, the output prints ready-to-run commands. Example for a 3-node cluster:
|
||||
|
||||
```bash
|
||||
# Export from build node
|
||||
ssh -i ~/.ssh/proxmox-host-tf terraform@<BUILD_NODE> \
|
||||
'vzdump 9000 --dumpdir /tmp --mode stop --compress zstd'
|
||||
|
||||
# Copy to other nodes
|
||||
scp -3 -i ~/.ssh/proxmox-host-tf \
|
||||
terraform@<BUILD_NODE>:/tmp/vzdump-qemu-9000-*.vma.zst \
|
||||
terraform@<TARGET_NODE>:/tmp/
|
||||
|
||||
# Restore on each target node (use a unique VMID and the node's datastore)
|
||||
ssh -i ~/.ssh/proxmox-host-tf terraform@<TARGET_NODE> \
|
||||
'qmrestore /tmp/vzdump-qemu-9000-*.vma.zst <NEW_VMID> --storage <DATASTORE> && qm template <NEW_VMID>'
|
||||
```
|
||||
|
||||
After distributing, each node has its own copy of the template with a unique VMID.
|
||||
|
||||
## Using the Template
|
||||
|
||||
Update `scripts/iac/proxmox/vms.tf` to clone from the template instead of booting from a raw cloud image. Or clone manually from the Proxmox UI.
|
||||
|
||||
On each cloned VM after boot:
|
||||
|
||||
```bash
|
||||
# Pull latest code
|
||||
cd ~/Development/mulga/hive && git pull
|
||||
cd ~/Development/mulga/viperblock && git pull
|
||||
cd ~/Development/mulga/predastore && git pull
|
||||
|
||||
# Build (~30s with cached modules)
|
||||
cd ~/Development/mulga/hive && export PATH=/usr/local/go/bin:$PATH && make build
|
||||
```
|
||||
|
||||
## Testing a Different Base Image
|
||||
|
||||
Upload the new image to Proxmox, then build a separate template:
|
||||
|
||||
```bash
|
||||
# Debian 13 example
|
||||
export TF_VAR_base_image="local:iso/debian-13-genericcloud-amd64.img"
|
||||
export TF_VAR_base_image_tag="debian-13"
|
||||
export TF_VAR_template_vmid=9010 # Keep the debian-12 template at 9000
|
||||
|
||||
source .env
|
||||
tofu apply
|
||||
```
|
||||
|
||||
Templates are tagged in the Proxmox UI with the base image name (`debian-12`, `debian-13`, etc.) so you can tell them apart.
|
||||
|
||||
## Rebuilding
|
||||
|
||||
When dependencies change (new Go version, new system packages):
|
||||
|
||||
```bash
|
||||
source .env
|
||||
tofu destroy # Remove the old template
|
||||
tofu apply # Build fresh
|
||||
# Then re-distribute with vzdump
|
||||
```
|
||||
|
||||
## SSH Key Setup
|
||||
|
||||
This template uses **two separate key pairs** (matching the `scripts/iac/proxmox/` setup):
|
||||
|
||||
| Key | Used for | Variable |
|
||||
|---|---|---|
|
||||
| Proxmox host key | SSH to Proxmox hosts (provider, `qm template` command) | `ssh_private_key_path` |
|
||||
| VM / cloud-init key | SSH to VMs (cloud-init injection, provisioning) | `vm_ssh_private_key_path` + `ssh_public_key_path` |
|
||||
|
||||
If you use the same key for both, set all three variables to the same key pair.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `main.tf` | Proxmox provider configuration |
|
||||
| `variables.tf` | All configurable variables with defaults |
|
||||
| `template-builder.tf` | VM creation, provisioning, template conversion |
|
||||
| `outputs.tf` | Template info + vzdump/restore commands |
|
||||
| `scripts/provision.sh` | Install deps, clone repos, cache modules, generalize |
|
||||
| `.env.example` | Environment variable template |
|
||||
| `.gitignore` | Ignores .env, state files, provider plugins |
|
||||
@@ -0,0 +1,27 @@
|
||||
terraform {
|
||||
required_version = ">= 1.6.0"
|
||||
|
||||
required_providers {
|
||||
proxmox = {
|
||||
source = "bpg/proxmox"
|
||||
version = "0.52.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# API token is read from PROXMOX_VE_API_TOKEN env var automatically
|
||||
provider "proxmox" {
|
||||
endpoint = var.proxmox_endpoint
|
||||
insecure = true
|
||||
|
||||
ssh {
|
||||
agent = false
|
||||
username = var.proxmox_ssh_username
|
||||
private_key = file(pathexpand(var.ssh_private_key_path))
|
||||
|
||||
node {
|
||||
name = var.proxmox_node_name
|
||||
address = var.proxmox_node_address
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
output "template_vmid" {
|
||||
description = "VM ID of the created template"
|
||||
value = proxmox_virtual_environment_vm.builder.vm_id
|
||||
}
|
||||
|
||||
output "template_node" {
|
||||
description = "Proxmox node where the template lives"
|
||||
value = var.proxmox_node_name
|
||||
}
|
||||
|
||||
output "base_image_tag" {
|
||||
description = "Base image tag for this template"
|
||||
value = var.base_image_tag
|
||||
}
|
||||
|
||||
# Ready-to-run command: export the template for distribution to other Proxmox nodes
|
||||
output "vzdump_export_command" {
|
||||
description = "Run this to export the template (vzdump) for copying to other nodes"
|
||||
value = <<-EOT
|
||||
ssh -i ${var.ssh_private_key_path} ${var.proxmox_ssh_username}@${var.proxmox_node_address} \
|
||||
'vzdump ${var.template_vmid} --dumpdir /tmp --mode stop --compress zstd'
|
||||
EOT
|
||||
}
|
||||
|
||||
# After exporting, copy to each target node and restore
|
||||
output "distribute_commands" {
|
||||
description = "Copy and restore the template on other Proxmox nodes"
|
||||
value = <<-EOT
|
||||
# 1. Export from build node (${var.proxmox_node_name}):
|
||||
ssh -i ${var.ssh_private_key_path} ${var.proxmox_ssh_username}@${var.proxmox_node_address} \
|
||||
'vzdump ${var.template_vmid} --dumpdir /tmp --mode stop --compress zstd'
|
||||
|
||||
# 2. Copy to each target node:
|
||||
scp -3 \
|
||||
-i ${var.ssh_private_key_path} \
|
||||
${var.proxmox_ssh_username}@${var.proxmox_node_address}:/tmp/vzdump-qemu-${var.template_vmid}-*.vma.zst \
|
||||
${var.proxmox_ssh_username}@<TARGET_NODE_ADDRESS>:/tmp/
|
||||
|
||||
# 3. On each target node — restore and convert to template:
|
||||
ssh -i ${var.ssh_private_key_path} ${var.proxmox_ssh_username}@<TARGET_NODE_ADDRESS> \
|
||||
'qmrestore /tmp/vzdump-qemu-${var.template_vmid}-*.vma.zst <NEW_VMID> --storage <DATASTORE> && qm template <NEW_VMID>'
|
||||
EOT
|
||||
}
|
||||
+176
@@ -0,0 +1,176 @@
|
||||
#!/bin/bash
|
||||
# provision.sh — Installs all Hive dependencies, clones repos, caches Go modules,
|
||||
# and generalizes the VM for use as a Proxmox template.
|
||||
#
|
||||
# Runs as tf-user via Terraform remote-exec. Uses sudo for system operations.
|
||||
# Environment: GIT_BRANCH (default: main)
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
GIT_BRANCH="${GIT_BRANCH:-main}"
|
||||
WORK_DIR="$HOME/Development/mulga"
|
||||
|
||||
echo "========================================"
|
||||
echo " Hive Template Provisioning"
|
||||
echo " Branch: ${GIT_BRANCH}"
|
||||
echo " Arch: $(uname -m)"
|
||||
echo "========================================"
|
||||
|
||||
# ── 1. Wait for cloud-init ──────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "=== [1/8] Waiting for cloud-init to complete ==="
|
||||
cloud-init status --wait
|
||||
|
||||
# ── 2. Clone repositories ───────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "=== [2/8] Cloning repositories ==="
|
||||
mkdir -p "$WORK_DIR"
|
||||
cd "$WORK_DIR"
|
||||
|
||||
for repo in hive viperblock predastore; do
|
||||
if [ ! -d "$repo" ]; then
|
||||
echo " Cloning $repo (branch: $GIT_BRANCH)..."
|
||||
git clone -b "$GIT_BRANCH" "https://github.com/mulgadc/${repo}.git"
|
||||
else
|
||||
echo " $repo already exists, skipping"
|
||||
fi
|
||||
done
|
||||
|
||||
# ── 3. Install system dependencies via Makefile ─────────────────────────────
|
||||
echo ""
|
||||
echo "=== [3/8] Installing system dependencies ==="
|
||||
cd "$WORK_DIR/hive"
|
||||
sudo make quickinstall
|
||||
|
||||
# Ensure Go is in PATH for the rest of the script
|
||||
export PATH="/usr/local/go/bin:$HOME/go/bin:$PATH"
|
||||
echo " Go version: $(go version)"
|
||||
echo " AWS CLI: $(aws --version)"
|
||||
|
||||
# ── 4. Set up go.work ───────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "=== [4/8] Creating go.work ==="
|
||||
|
||||
# Detect Go version from go.mod
|
||||
GO_VERSION=$(grep '^go ' "$WORK_DIR/hive/go.mod" | awk '{print $2}')
|
||||
echo " Using Go version from go.mod: ${GO_VERSION}"
|
||||
|
||||
cat > "$WORK_DIR/hive/go.work" << GOWORK
|
||||
go ${GO_VERSION}
|
||||
|
||||
use (
|
||||
.
|
||||
../viperblock
|
||||
../viperblock/nbd
|
||||
../viperblock/nbd/libguestfs.org/nbdkit
|
||||
../predastore
|
||||
)
|
||||
GOWORK
|
||||
|
||||
echo " Created $WORK_DIR/hive/go.work"
|
||||
|
||||
# ── 5. Download Go modules ──────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "=== [5/8] Downloading Go modules ==="
|
||||
cd "$WORK_DIR/hive" && go mod download && echo " hive: done"
|
||||
cd "$WORK_DIR/viperblock" && go mod download && echo " viperblock: done"
|
||||
cd "$WORK_DIR/predastore" && go mod download && echo " predastore: done"
|
||||
|
||||
# Sub-modules
|
||||
if [ -f "$WORK_DIR/viperblock/nbd/go.mod" ]; then
|
||||
cd "$WORK_DIR/viperblock/nbd" && go mod download && echo " viperblock/nbd: done"
|
||||
fi
|
||||
if [ -f "$WORK_DIR/viperblock/nbd/libguestfs.org/nbdkit/go.mod" ]; then
|
||||
cd "$WORK_DIR/viperblock/nbd/libguestfs.org/nbdkit" && go mod download && echo " viperblock/nbd/nbdkit: done"
|
||||
fi
|
||||
|
||||
# ── 6. Pre-download cloud images for nested VMs ─────────────────────────────
|
||||
echo ""
|
||||
echo "=== [6/8] Pre-downloading cloud images ==="
|
||||
mkdir -p "$HOME/images"
|
||||
|
||||
ARCH=$(dpkg --print-architecture)
|
||||
IMAGE_URL=""
|
||||
if [ "$ARCH" = "amd64" ]; then
|
||||
IMAGE_URL="https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img"
|
||||
elif [ "$ARCH" = "arm64" ]; then
|
||||
IMAGE_URL="https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-arm64.img"
|
||||
fi
|
||||
|
||||
if [ -n "$IMAGE_URL" ]; then
|
||||
echo " Downloading Ubuntu 24.04 cloud image ($ARCH)..."
|
||||
curl -L -o "$HOME/images/ubuntu-24.04.img" "$IMAGE_URL"
|
||||
echo " Saved to ~/images/ubuntu-24.04.img"
|
||||
else
|
||||
echo " Unknown arch ($ARCH), skipping cloud image download"
|
||||
fi
|
||||
|
||||
# ── 7. System tuning ────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "=== [7/8] Applying system tuning ==="
|
||||
|
||||
# Persistent sysctl for NATS / high-throughput networking
|
||||
sudo tee /etc/sysctl.d/99-hive.conf > /dev/null << 'SYSCTL'
|
||||
net.core.rmem_max=4194304
|
||||
net.core.wmem_max=4194304
|
||||
SYSCTL
|
||||
echo " sysctl: net.core.rmem_max/wmem_max = 4MB"
|
||||
|
||||
# Add user to kvm group for QEMU access
|
||||
sudo usermod -aG kvm "$(whoami)"
|
||||
echo " Added $(whoami) to kvm group"
|
||||
|
||||
# Disable ufw if present
|
||||
if command -v ufw >/dev/null 2>&1; then
|
||||
sudo ufw disable 2>/dev/null || true
|
||||
echo " ufw disabled"
|
||||
fi
|
||||
|
||||
# ── 8. Generalize for templating ────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "=== [8/8] Cleaning up for template ==="
|
||||
|
||||
# Apt caches
|
||||
sudo apt-get clean
|
||||
sudo rm -rf /var/lib/apt/lists/*
|
||||
echo " Cleared apt cache"
|
||||
|
||||
# Logs
|
||||
sudo find /var/log -type f -name "*.log" -exec truncate -s 0 {} \; 2>/dev/null || true
|
||||
sudo rm -rf /var/log/journal/*
|
||||
echo " Truncated logs"
|
||||
|
||||
# Temp files
|
||||
sudo rm -rf /tmp/* /var/tmp/*
|
||||
|
||||
# Machine ID — systemd regenerates on next boot.
|
||||
# This is critical: without this, all clones get the same DHCP client ID → IP conflicts.
|
||||
sudo truncate -s 0 /etc/machine-id
|
||||
sudo rm -f /var/lib/dbus/machine-id
|
||||
echo " Cleared machine-id (will regenerate on boot)"
|
||||
|
||||
# SSH host keys — regenerated on boot by cloud-init or systemd
|
||||
sudo rm -f /etc/ssh/ssh_host_*
|
||||
echo " Removed SSH host keys (will regenerate on boot)"
|
||||
|
||||
# Cloud-init state — forces cloud-init to run again on next boot.
|
||||
# This ensures each clone gets its own hostname, SSH keys, user config, etc.
|
||||
sudo cloud-init clean
|
||||
echo " Cleaned cloud-init state (will re-run on boot)"
|
||||
|
||||
# Shell history
|
||||
history -c 2>/dev/null || true
|
||||
cat /dev/null > "$HOME/.bash_history" 2>/dev/null || true
|
||||
|
||||
# Provisioning script itself
|
||||
rm -f /tmp/provision.sh
|
||||
|
||||
# Sync filesystem
|
||||
sync
|
||||
|
||||
echo ""
|
||||
echo "========================================"
|
||||
echo " Template provisioning complete!"
|
||||
echo " VM is ready to be shut down and"
|
||||
echo " converted to a Proxmox template."
|
||||
echo "========================================"
|
||||
@@ -0,0 +1,130 @@
|
||||
data "local_file" "ssh_public_key" {
|
||||
filename = pathexpand(var.ssh_public_key_path)
|
||||
}
|
||||
|
||||
# Minimal cloud-init: just enough to bootstrap SSH + guest agent.
|
||||
# Heavy provisioning is done via remote-exec after boot.
|
||||
resource "proxmox_virtual_environment_file" "cloud_config" {
|
||||
content_type = "snippets"
|
||||
datastore_id = "local"
|
||||
node_name = var.proxmox_node_name
|
||||
|
||||
source_raw {
|
||||
# NOTE: #cloud-config MUST start at column 0 — cloud-init uses it as a magic header.
|
||||
data = <<EOF
|
||||
#cloud-config
|
||||
package_update: true
|
||||
packages:
|
||||
- qemu-guest-agent
|
||||
- git
|
||||
- make
|
||||
- curl
|
||||
- sudo
|
||||
- ca-certificates
|
||||
runcmd:
|
||||
- systemctl enable qemu-guest-agent
|
||||
- systemctl start qemu-guest-agent
|
||||
EOF
|
||||
|
||||
file_name = "hive-template-cloud-config.yaml"
|
||||
}
|
||||
}
|
||||
|
||||
resource "proxmox_virtual_environment_vm" "builder" {
|
||||
name = "${var.template_name}-builder"
|
||||
node_name = var.proxmox_node_name
|
||||
vm_id = var.template_vmid
|
||||
|
||||
tags = sort(["terraform", "hive-template", var.base_image_tag])
|
||||
|
||||
initialization {
|
||||
datastore_id = "local"
|
||||
vendor_data_file_id = proxmox_virtual_environment_file.cloud_config.id
|
||||
|
||||
user_account {
|
||||
username = var.vm_ssh_user
|
||||
keys = [trimspace(data.local_file.ssh_public_key.content)]
|
||||
}
|
||||
|
||||
ip_config {
|
||||
ipv4 {
|
||||
address = "dhcp"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
cpu {
|
||||
cores = var.cpu_cores
|
||||
type = "host"
|
||||
}
|
||||
|
||||
memory {
|
||||
dedicated = var.memory_mb
|
||||
}
|
||||
|
||||
agent {
|
||||
enabled = true
|
||||
}
|
||||
|
||||
network_device {
|
||||
bridge = var.bridge
|
||||
}
|
||||
|
||||
disk {
|
||||
file_id = var.base_image
|
||||
datastore_id = var.datastore_id
|
||||
interface = "virtio0"
|
||||
iothread = true
|
||||
discard = "on"
|
||||
size = var.disk_size_gb
|
||||
}
|
||||
|
||||
# SSH connection to the VM for provisioning.
|
||||
# Uses the VM key (vm_ssh_private_key_path), NOT the Proxmox host key.
|
||||
# Waits for guest agent to report an IP, then connects.
|
||||
connection {
|
||||
type = "ssh"
|
||||
user = var.vm_ssh_user
|
||||
private_key = file(pathexpand(var.vm_ssh_private_key_path))
|
||||
host = element([for addr in flatten(self.ipv4_addresses) : addr if !startswith(addr, "127.")], 0)
|
||||
timeout = "10m"
|
||||
}
|
||||
|
||||
# Copy provisioning script to VM
|
||||
provisioner "file" {
|
||||
source = "${path.module}/scripts/provision.sh"
|
||||
destination = "/tmp/provision.sh"
|
||||
}
|
||||
|
||||
# Run provisioning (installs deps, clones repos, caches modules, cleans up)
|
||||
provisioner "remote-exec" {
|
||||
inline = [
|
||||
"chmod +x /tmp/provision.sh",
|
||||
"GIT_BRANCH='${var.git_branch}' /tmp/provision.sh",
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
# After provisioning completes, shut down the VM and convert to a Proxmox template.
|
||||
# This runs on the Proxmox host via SSH (not on the VM — it's being shut down).
|
||||
resource "null_resource" "convert_template" {
|
||||
depends_on = [proxmox_virtual_environment_vm.builder]
|
||||
|
||||
triggers = {
|
||||
builder_id = proxmox_virtual_environment_vm.builder.vm_id
|
||||
}
|
||||
|
||||
provisioner "local-exec" {
|
||||
command = <<-EOT
|
||||
ssh -i '${pathexpand(var.ssh_private_key_path)}' \
|
||||
-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
|
||||
'${var.proxmox_ssh_username}@${var.proxmox_node_address}' \
|
||||
"sudo qm shutdown ${proxmox_virtual_environment_vm.builder.vm_id} --timeout 60 && \
|
||||
echo 'Waiting for VM to stop...' && \
|
||||
while sudo qm status ${proxmox_virtual_environment_vm.builder.vm_id} | grep -q running; do sleep 2; done && \
|
||||
echo 'Converting to template...' && \
|
||||
sudo qm template ${proxmox_virtual_environment_vm.builder.vm_id} && \
|
||||
echo 'Template ${var.template_vmid} created successfully'"
|
||||
EOT
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,117 @@
|
||||
# --- Proxmox Connection ---
|
||||
|
||||
variable "proxmox_endpoint" {
|
||||
type = string
|
||||
description = "Proxmox VE API endpoint URL (e.g. https://pve.example.com:8006/)"
|
||||
}
|
||||
|
||||
variable "proxmox_node_name" {
|
||||
type = string
|
||||
description = "Proxmox node name to build the template on (as shown in the Proxmox UI)"
|
||||
}
|
||||
|
||||
variable "proxmox_node_address" {
|
||||
type = string
|
||||
description = "SSH-reachable hostname or IP of the Proxmox node"
|
||||
}
|
||||
|
||||
variable "proxmox_ssh_username" {
|
||||
type = string
|
||||
description = "SSH username for Proxmox host access"
|
||||
default = "terraform"
|
||||
}
|
||||
|
||||
# --- SSH Keys ---
|
||||
# Proxmox host and VM typically use different key pairs.
|
||||
# - ssh_private_key_path → authenticates to the Proxmox host (provider + qm commands)
|
||||
# - vm_ssh_private_key_path → authenticates to the VM for provisioning (remote-exec)
|
||||
# - ssh_public_key_path → injected into the VM via cloud-init
|
||||
|
||||
variable "ssh_private_key_path" {
|
||||
type = string
|
||||
description = "Path to SSH private key for Proxmox host access"
|
||||
}
|
||||
|
||||
variable "vm_ssh_private_key_path" {
|
||||
type = string
|
||||
description = "Path to SSH private key for VM access (must match ssh_public_key_path)"
|
||||
}
|
||||
|
||||
variable "ssh_public_key_path" {
|
||||
type = string
|
||||
description = "Path to SSH public key injected into the VM via cloud-init"
|
||||
}
|
||||
|
||||
# --- Base Image ---
|
||||
|
||||
variable "base_image" {
|
||||
type = string
|
||||
description = "Proxmox image file ID for the base OS (e.g. local:iso/debian-12-genericcloud-amd64-20240211-1654.img)"
|
||||
default = "local:iso/debian-12-genericcloud-amd64-20240211-1654.img"
|
||||
}
|
||||
|
||||
variable "base_image_tag" {
|
||||
type = string
|
||||
description = "Short tag identifying the base image, used in template tags for tracking"
|
||||
default = "debian-12"
|
||||
}
|
||||
|
||||
# --- Template Settings ---
|
||||
|
||||
variable "template_name" {
|
||||
type = string
|
||||
description = "Name for the resulting Proxmox template"
|
||||
default = "hive-node-template"
|
||||
}
|
||||
|
||||
variable "template_vmid" {
|
||||
type = number
|
||||
description = "VM ID for the template (pick a high number to avoid conflicts, e.g. 9000)"
|
||||
default = 9000
|
||||
}
|
||||
|
||||
variable "vm_ssh_user" {
|
||||
type = string
|
||||
description = "Username created in the VM via cloud-init"
|
||||
default = "tf-user"
|
||||
}
|
||||
|
||||
# --- VM Resources ---
|
||||
|
||||
variable "cpu_cores" {
|
||||
type = number
|
||||
description = "CPU cores for the builder VM (more = faster builds)"
|
||||
default = 4
|
||||
}
|
||||
|
||||
variable "memory_mb" {
|
||||
type = number
|
||||
description = "Memory in MB for the builder VM"
|
||||
default = 8192
|
||||
}
|
||||
|
||||
variable "disk_size_gb" {
|
||||
type = number
|
||||
description = "Disk size in GB (needs room for OS + Go + modules + cloud images)"
|
||||
default = 32
|
||||
}
|
||||
|
||||
variable "datastore_id" {
|
||||
type = string
|
||||
description = "Proxmox storage for the VM disk"
|
||||
default = "local-lvm"
|
||||
}
|
||||
|
||||
variable "bridge" {
|
||||
type = string
|
||||
description = "Network bridge for the VM"
|
||||
default = "vmbr0"
|
||||
}
|
||||
|
||||
# --- Git ---
|
||||
|
||||
variable "git_branch" {
|
||||
type = string
|
||||
description = "Git branch to clone for hive, viperblock, and predastore"
|
||||
default = "main"
|
||||
}
|
||||
@@ -29,10 +29,14 @@ export TF_VAR_ssh_public_key_path="~/.ssh/cloud-init.pub"
|
||||
# You can define 1 or more nodes — VMs are distributed across them.
|
||||
#
|
||||
# Each node object:
|
||||
# name - Proxmox node name (as shown in the Proxmox UI)
|
||||
# address - SSH-reachable hostname or IP of the Proxmox host
|
||||
# bridge - Network bridge for VM interfaces (e.g. vmbr0, vmbr3)
|
||||
# datastore_id - Proxmox storage ID for VM disks (e.g. local, local-lvm)
|
||||
# name - Proxmox node name (as shown in the Proxmox UI)
|
||||
# address - SSH-reachable hostname or IP of the Proxmox host
|
||||
# bridge - Network bridge for VM interfaces (e.g. vmbr0, vmbr3)
|
||||
# datastore_id - Proxmox storage ID for VM disks (e.g. local, local-lvm)
|
||||
# template_vmid - (optional) VMID of a pre-built template on this node.
|
||||
# Build with scripts/iac/proxmox-template/, then distribute
|
||||
# with vzdump. When set, VMs clone from the template (fast).
|
||||
# When omitted, VMs boot from os_image (slower).
|
||||
|
||||
export TF_VAR_nodes='[
|
||||
{
|
||||
@@ -54,3 +58,11 @@ export TF_VAR_nodes='[
|
||||
"datastore_id": "local"
|
||||
}
|
||||
]'
|
||||
|
||||
# Example with templates (VMIDs may differ per node after vzdump restore):
|
||||
#
|
||||
# export TF_VAR_nodes='[
|
||||
# { "name": "node1", "address": "node1.example.com", "bridge": "vmbr0", "datastore_id": "local", "template_vmid": 9999 },
|
||||
# { "name": "node2", "address": "node2.example.com", "bridge": "vmbr0", "datastore_id": "local-lvm", "template_vmid": 9997 },
|
||||
# { "name": "node3", "address": "node3.example.com", "bridge": "vmbr0", "datastore_id": "ssd", "template_vmid": 9998 }
|
||||
# ]'
|
||||
|
||||
@@ -2,6 +2,38 @@ data "local_file" "ssh_public_key" {
|
||||
filename = pathexpand(var.ssh_public_key_path)
|
||||
}
|
||||
|
||||
# When cloning from a template, packages are pre-installed — skip apt update/upgrade.
|
||||
locals {
|
||||
cloud_config_template = <<-EOF
|
||||
#cloud-config
|
||||
runcmd:
|
||||
- timedatectl set-timezone America/Los_Angeles
|
||||
- systemctl enable qemu-guest-agent
|
||||
- systemctl start qemu-guest-agent
|
||||
- echo "done" > /tmp/vendor-cloud-init-done
|
||||
EOF
|
||||
|
||||
cloud_config_image = <<-EOF
|
||||
#cloud-config
|
||||
package_update: true
|
||||
package_upgrade: true
|
||||
packages:
|
||||
- qemu-guest-agent
|
||||
- net-tools
|
||||
- git
|
||||
- make
|
||||
runcmd:
|
||||
- timedatectl set-timezone America/Los_Angeles
|
||||
- systemctl enable qemu-guest-agent
|
||||
- systemctl start qemu-guest-agent
|
||||
- ufw disable
|
||||
- sysctl -w net.core.rmem_max=4194304
|
||||
- sysctl -w net.core.wmem_max=4194304
|
||||
- usermod -aG kvm tf-user
|
||||
- echo "done" > /tmp/vendor-cloud-init-done
|
||||
EOF
|
||||
}
|
||||
|
||||
resource "proxmox_virtual_environment_file" "cloud_config" {
|
||||
count = var.node_count
|
||||
|
||||
@@ -10,26 +42,7 @@ resource "proxmox_virtual_environment_file" "cloud_config" {
|
||||
node_name = var.nodes[count.index % length(var.nodes)].name
|
||||
|
||||
source_raw {
|
||||
data = <<EOF
|
||||
#cloud-config
|
||||
package_update: true
|
||||
package_upgrade: true
|
||||
packages:
|
||||
- qemu-guest-agent
|
||||
- net-tools
|
||||
- git
|
||||
- make
|
||||
runcmd:
|
||||
- timedatectl set-timezone America/Los_Angeles
|
||||
- systemctl enable qemu-guest-agent
|
||||
- systemctl start qemu-guest-agent
|
||||
- ufw disable
|
||||
- sysctl -w net.core.rmem_max=4194304
|
||||
- sysctl -w net.core.wmem_max=4194304
|
||||
- usermod -aG kvm tf-user
|
||||
- echo "done" > /tmp/vendor-cloud-init-done
|
||||
EOF
|
||||
|
||||
data = var.nodes[count.index % length(var.nodes)].template_vmid != null ? local.cloud_config_template : local.cloud_config_image
|
||||
file_name = "${var.cluster_name}-${count.index + 1}-cloud-config.yaml"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -36,12 +36,13 @@ variable "ssh_public_key_path" {
|
||||
|
||||
variable "nodes" {
|
||||
type = list(object({
|
||||
name = string
|
||||
address = string
|
||||
bridge = string
|
||||
datastore_id = string
|
||||
name = string
|
||||
address = string
|
||||
bridge = string
|
||||
datastore_id = string
|
||||
template_vmid = optional(number)
|
||||
}))
|
||||
description = "Proxmox hosts for VM placement (VMs are distributed round-robin)"
|
||||
description = "Proxmox hosts for VM placement (VMs are distributed round-robin). Set template_vmid to clone from a pre-built template instead of booting from os_image."
|
||||
|
||||
validation {
|
||||
condition = length(var.nodes) > 0
|
||||
@@ -100,6 +101,6 @@ variable "disk_size_gb" {
|
||||
|
||||
variable "os_image" {
|
||||
type = string
|
||||
description = "Proxmox image file ID for VM boot disk"
|
||||
description = "Proxmox image file ID for VM boot disk (used when node has no template_vmid)"
|
||||
default = "local:iso/debian-12-genericcloud-amd64-20240211-1654.img"
|
||||
}
|
||||
|
||||
@@ -51,12 +51,24 @@ resource "proxmox_virtual_environment_vm" "hive_node" {
|
||||
bridge = var.nodes[count.index % length(var.nodes)].bridge
|
||||
}
|
||||
|
||||
disk {
|
||||
file_id = var.os_image
|
||||
datastore_id = var.nodes[count.index % length(var.nodes)].datastore_id
|
||||
interface = "virtio0"
|
||||
iothread = true
|
||||
discard = "on"
|
||||
size = var.disk_size_gb
|
||||
# Clone from template if the target node has template_vmid set, otherwise boot from cloud image.
|
||||
dynamic "clone" {
|
||||
for_each = var.nodes[count.index % length(var.nodes)].template_vmid != null ? [1] : []
|
||||
content {
|
||||
vm_id = var.nodes[count.index % length(var.nodes)].template_vmid
|
||||
full = true
|
||||
}
|
||||
}
|
||||
|
||||
dynamic "disk" {
|
||||
for_each = var.nodes[count.index % length(var.nodes)].template_vmid == null ? [1] : []
|
||||
content {
|
||||
file_id = var.os_image
|
||||
datastore_id = var.nodes[count.index % length(var.nodes)].datastore_id
|
||||
interface = "virtio0"
|
||||
iothread = true
|
||||
discard = "on"
|
||||
size = var.disk_size_gb
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -126,6 +126,22 @@ start_service_foreground() {
|
||||
$command
|
||||
}
|
||||
|
||||
# Function to set OOM score for a service (Linux only)
|
||||
# Protects infrastructure services from OOM killer (-500 = less likely to be killed)
|
||||
set_oom_score() {
|
||||
local name="$1"
|
||||
local score="$2"
|
||||
local pid_file="$LOGS_DIR/${name}.pid"
|
||||
if [ -f "$pid_file" ]; then
|
||||
local pid=$(cat "$pid_file")
|
||||
if [ -d "/proc/$pid" ]; then
|
||||
echo "$score" > "/proc/$pid/oom_score_adj" 2>/dev/null && \
|
||||
echo " OOM score for $name (PID $pid): $score" || \
|
||||
echo " Warning: Could not set OOM score for $name"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to check if service is responsive
|
||||
check_service() {
|
||||
local name="$1"
|
||||
@@ -188,6 +204,7 @@ if has_service "nats"; then
|
||||
|
||||
NATS_CMD="./bin/hive service nats start"
|
||||
start_service "nats" "$NATS_CMD"
|
||||
set_oom_score "nats" "-500"
|
||||
else
|
||||
echo "1️⃣ Skipping NATS (not a local service)"
|
||||
fi
|
||||
@@ -231,6 +248,7 @@ if has_service "predastore"; then
|
||||
|
||||
PREDASTORE_CMD="./bin/hive service predastore start"
|
||||
start_service "predastore" "$PREDASTORE_CMD"
|
||||
set_oom_score "predastore" "-500"
|
||||
check_service "Predastore" "$HIVE_PREDASTORE_HOST" "$HIVE_PREDASTORE_PORT"
|
||||
else
|
||||
echo "2️⃣ Skipping Predastore (not a local service)"
|
||||
@@ -266,6 +284,7 @@ if has_service "viperblock"; then
|
||||
|
||||
VIPERBLOCK_CMD="./bin/hive service viperblock start"
|
||||
start_service "viperblock" "$VIPERBLOCK_CMD"
|
||||
set_oom_score "viperblock" "-500"
|
||||
else
|
||||
echo "3️⃣ Skipping Viperblock (not a local service)"
|
||||
fi
|
||||
@@ -281,6 +300,7 @@ if has_service "daemon"; then
|
||||
|
||||
HIVE_CMD="./bin/hive service hive start"
|
||||
start_service "hive" "$HIVE_CMD"
|
||||
set_oom_score "hive" "-500"
|
||||
else
|
||||
echo "4️⃣. Skipping Hive Gateway (not a local service)"
|
||||
fi
|
||||
@@ -300,6 +320,7 @@ if has_service "awsgw"; then
|
||||
|
||||
AWSGW_CMD="./bin/hive service awsgw start"
|
||||
start_service "awsgw" "$AWSGW_CMD"
|
||||
set_oom_score "awsgw" "-500"
|
||||
else
|
||||
echo "5️⃣. Skipping AWS Gateway (not a local service)"
|
||||
fi
|
||||
@@ -312,6 +333,7 @@ if [ "${UI}" != "false" ] && has_service "ui"; then
|
||||
|
||||
HIVEUI_CMD="./bin/hive service hive-ui start"
|
||||
start_service "hive-ui" "$HIVEUI_CMD"
|
||||
set_oom_score "hive-ui" "-500"
|
||||
else
|
||||
echo ""
|
||||
echo "6️⃣. Skipping Hive UI"
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
#!/bin/sh
|
||||
|
||||
# Workload performance testing (memory, disk, cpu, network)
|
||||
|
||||
sudo apt update
|
||||
sudo apt install -y make gcc git
|
||||
|
||||
wget https://go.dev/dl/go1.26.0.linux-amd64.tar.gz
|
||||
sudo tar -C /usr/local -xzf go1.26.0.linux-amd64.tar.gz
|
||||
|
||||
echo "export PATH=$PATH:/usr/local/go/bin" >> $HOME/.bashrc
|
||||
source $HOME/.bashrc
|
||||
|
||||
git clone https://github.com/dgraph-io/badger.git
|
||||
cd badger
|
||||
|
||||
# disable jemalloc
|
||||
sed -i 's/-tags=jemalloc/-tags=nojemalloc/g' test.sh
|
||||
|
||||
# run 4 tests, otherwise OOM on <8GB RAM
|
||||
sed -i 's/-parallel 16/-parallel 4/g; s/-parallel=16/-parallel=4/g' test.sh
|
||||
|
||||
# run tests
|
||||
./test.sh
|
||||
@@ -495,6 +495,68 @@ dump_all_node_logs() {
|
||||
echo "=========================================="
|
||||
}
|
||||
|
||||
# Get the QEMU process PID for an instance
|
||||
# Usage: get_qemu_pid <instance_id>
|
||||
# Returns: the PID, or empty string if not found
|
||||
get_qemu_pid() {
|
||||
local instance_id="$1"
|
||||
|
||||
local pid
|
||||
pid=$(ps auxw | grep "$instance_id" | grep qemu-system | grep -v grep | awk '{print $2}' | head -1)
|
||||
|
||||
if [ -n "$pid" ]; then
|
||||
echo "$pid"
|
||||
return 0
|
||||
fi
|
||||
|
||||
return 1
|
||||
}
|
||||
|
||||
# Wait for an instance to recover from a crash (error → running)
|
||||
# Usage: wait_for_instance_recovery <instance_id> [max_attempts]
|
||||
# Expects state transition: error → pending → running
|
||||
wait_for_instance_recovery() {
|
||||
local instance_id="$1"
|
||||
local max_attempts="${2:-60}"
|
||||
local attempt=0
|
||||
local saw_error=false
|
||||
|
||||
echo " Waiting for instance $instance_id to recover..."
|
||||
|
||||
while [ $attempt -lt $max_attempts ]; do
|
||||
local state
|
||||
state=$(aws --endpoint-url https://${NODE1_IP}:${AWSGW_PORT} ec2 describe-instances \
|
||||
--instance-ids "$instance_id" \
|
||||
--query 'Reservations[0].Instances[0].State.Name' \
|
||||
--output text 2>/dev/null) || {
|
||||
sleep 2
|
||||
attempt=$((attempt + 1))
|
||||
continue
|
||||
}
|
||||
|
||||
if [ "$state" == "error" ] || [ "$state" == "pending" ]; then
|
||||
saw_error=true
|
||||
echo " State: $state (attempt $((attempt + 1))/$max_attempts)"
|
||||
fi
|
||||
|
||||
if [ "$state" == "running" ] && [ "$saw_error" = true ]; then
|
||||
echo " Instance recovered to running state"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [ "$state" == "running" ] && [ "$saw_error" = false ]; then
|
||||
# Still in original running state, crash not detected yet
|
||||
:
|
||||
fi
|
||||
|
||||
sleep 2
|
||||
attempt=$((attempt + 1))
|
||||
done
|
||||
|
||||
echo " ERROR: Instance did not recover within $max_attempts attempts"
|
||||
return 1
|
||||
}
|
||||
|
||||
# Global variable for init PID tracking (used by multi-node formation)
|
||||
LEADER_INIT_PID=""
|
||||
|
||||
|
||||
@@ -958,6 +958,176 @@ verify_nats_cluster 3 || {
|
||||
echo "WARNING: NATS cluster verification failed after operations"
|
||||
}
|
||||
|
||||
# Test 5: VM Crash Recovery (kill -9 → detect → auto-restart)
|
||||
echo ""
|
||||
echo "Test 5: VM Crash Recovery"
|
||||
echo "----------------------------------------"
|
||||
echo "Testing QEMU crash detection and auto-restart..."
|
||||
|
||||
# Use the second instance for crash testing (first was used for stop/start in Test 3)
|
||||
CRASH_INSTANCE="${INSTANCE_IDS[1]}"
|
||||
echo " Crash test instance: $CRASH_INSTANCE"
|
||||
|
||||
# Verify instance is running before crash
|
||||
CRASH_STATE=$($AWS_EC2 describe-instances --instance-ids "$CRASH_INSTANCE" \
|
||||
--query 'Reservations[0].Instances[0].State.Name' --output text)
|
||||
if [ "$CRASH_STATE" != "running" ]; then
|
||||
echo " ERROR: Instance not in running state before crash test (state: $CRASH_STATE)"
|
||||
exit 1
|
||||
fi
|
||||
echo " Instance is running"
|
||||
|
||||
# Get the QEMU PID
|
||||
QEMU_PID=$(get_qemu_pid "$CRASH_INSTANCE")
|
||||
if [ -z "$QEMU_PID" ]; then
|
||||
echo " ERROR: Could not find QEMU PID for $CRASH_INSTANCE"
|
||||
exit 1
|
||||
fi
|
||||
echo " QEMU PID: $QEMU_PID"
|
||||
|
||||
# Kill QEMU with SIGKILL (simulates OOM kill)
|
||||
echo " Killing QEMU process with SIGKILL (simulating OOM kill)..."
|
||||
kill -9 "$QEMU_PID"
|
||||
|
||||
# Brief pause for the daemon to detect the crash
|
||||
sleep 3
|
||||
|
||||
# Verify the daemon detected the crash (state should be error or already recovering)
|
||||
echo " Checking post-crash state..."
|
||||
POST_CRASH_STATE=$($AWS_EC2 describe-instances --instance-ids "$CRASH_INSTANCE" \
|
||||
--query 'Reservations[0].Instances[0].State.Name' --output text)
|
||||
echo " Post-crash state: $POST_CRASH_STATE"
|
||||
|
||||
if [ "$POST_CRASH_STATE" == "running" ]; then
|
||||
# Might have recovered very quickly, check if it's a new PID
|
||||
NEW_PID=$(get_qemu_pid "$CRASH_INSTANCE" || echo "")
|
||||
if [ "$NEW_PID" == "$QEMU_PID" ]; then
|
||||
echo " ERROR: Instance still has same PID after kill -9, crash not detected"
|
||||
exit 1
|
||||
fi
|
||||
echo " Instance already recovered with new PID: $NEW_PID (was: $QEMU_PID)"
|
||||
else
|
||||
# Wait for auto-restart (backoff starts at 5s)
|
||||
echo " Instance in $POST_CRASH_STATE state, waiting for auto-restart..."
|
||||
wait_for_instance_recovery "$CRASH_INSTANCE" 60 || {
|
||||
echo " ERROR: Instance failed to recover from crash"
|
||||
# Dump daemon logs for debugging
|
||||
for i in 1 2 3; do
|
||||
if [ -f "$HOME/node$i/logs/hive.log" ]; then
|
||||
echo ""
|
||||
echo " --- node$i daemon log (last 30 lines) ---"
|
||||
tail -30 "$HOME/node$i/logs/hive.log"
|
||||
fi
|
||||
done
|
||||
exit 1
|
||||
}
|
||||
fi
|
||||
|
||||
# Verify the instance is running with a new QEMU process
|
||||
RECOVERED_PID=$(get_qemu_pid "$CRASH_INSTANCE")
|
||||
if [ -z "$RECOVERED_PID" ]; then
|
||||
echo " ERROR: No QEMU process found after recovery"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "$RECOVERED_PID" == "$QEMU_PID" ]; then
|
||||
echo " ERROR: QEMU PID unchanged after crash recovery (expected new process)"
|
||||
exit 1
|
||||
fi
|
||||
echo " New QEMU PID: $RECOVERED_PID (was: $QEMU_PID)"
|
||||
|
||||
# Verify describe-instances shows running state
|
||||
FINAL_STATE=$($AWS_EC2 describe-instances --instance-ids "$CRASH_INSTANCE" \
|
||||
--query 'Reservations[0].Instances[0].State.Name' --output text)
|
||||
if [ "$FINAL_STATE" != "running" ]; then
|
||||
echo " ERROR: Instance not running after recovery (state: $FINAL_STATE)"
|
||||
exit 1
|
||||
fi
|
||||
echo " Instance is running after crash recovery"
|
||||
|
||||
# Verify SSH works after recovery (VM rebooted with fresh OS from root volume)
|
||||
echo " Verifying SSH after crash recovery..."
|
||||
CRASH_SSH_PORT=$(get_ssh_port "$CRASH_INSTANCE" 30)
|
||||
CRASH_SSH_HOST=$(get_ssh_host "$CRASH_INSTANCE")
|
||||
if [ -n "$CRASH_SSH_PORT" ]; then
|
||||
echo " SSH endpoint after recovery: $CRASH_SSH_HOST:$CRASH_SSH_PORT"
|
||||
# Update SSH details for later termination verification
|
||||
SSH_PORTS[1]="$CRASH_SSH_PORT"
|
||||
SSH_HOSTS[1]="$CRASH_SSH_HOST"
|
||||
wait_for_ssh "$CRASH_SSH_HOST" "$CRASH_SSH_PORT" "multinode-test-key.pem" || {
|
||||
echo " WARNING: SSH not ready after crash recovery (non-fatal, VM may still be booting)"
|
||||
}
|
||||
else
|
||||
echo " WARNING: Could not determine SSH port after recovery (non-fatal)"
|
||||
fi
|
||||
|
||||
# Test 5b: Crash Loop Prevention (kill 4 times rapidly to exceed max restarts)
|
||||
echo ""
|
||||
echo "Test 5b: Crash Loop Prevention"
|
||||
echo "----------------------------------------"
|
||||
echo "Testing that crash loop is detected and restarts stop after max attempts..."
|
||||
|
||||
# Use the third instance for crash loop testing
|
||||
LOOP_INSTANCE="${INSTANCE_IDS[2]}"
|
||||
echo " Crash loop test instance: $LOOP_INSTANCE"
|
||||
|
||||
# Verify instance is running
|
||||
LOOP_STATE=$($AWS_EC2 describe-instances --instance-ids "$LOOP_INSTANCE" \
|
||||
--query 'Reservations[0].Instances[0].State.Name' --output text)
|
||||
if [ "$LOOP_STATE" != "running" ]; then
|
||||
echo " ERROR: Instance not running before crash loop test (state: $LOOP_STATE)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Kill QEMU repeatedly to exhaust restart attempts (max 3 in 10 min window)
|
||||
for crash_num in 1 2 3 4; do
|
||||
echo " Crash $crash_num/4: killing QEMU..."
|
||||
LOOP_PID=$(get_qemu_pid "$LOOP_INSTANCE" || echo "")
|
||||
if [ -z "$LOOP_PID" ]; then
|
||||
echo " No QEMU process found (instance may be in error state)"
|
||||
break
|
||||
fi
|
||||
|
||||
kill -9 "$LOOP_PID"
|
||||
|
||||
if [ $crash_num -lt 4 ]; then
|
||||
# Wait for restart (backoff increases: 5s, 10s, 20s)
|
||||
# Give generous time for each restart cycle
|
||||
local_max=$((15 + crash_num * 10))
|
||||
echo " Waiting up to ${local_max}s for restart or error state..."
|
||||
attempt=0
|
||||
while [ $attempt -lt $local_max ]; do
|
||||
state=$($AWS_EC2 describe-instances --instance-ids "$LOOP_INSTANCE" \
|
||||
--query 'Reservations[0].Instances[0].State.Name' --output text 2>/dev/null || echo "unknown")
|
||||
if [ "$state" == "running" ]; then
|
||||
echo " Instance restarted (crash $crash_num)"
|
||||
break
|
||||
fi
|
||||
if [ "$state" == "error" ] && [ $crash_num -ge 3 ]; then
|
||||
echo " Instance in error state after crash $crash_num (restart limit may be reached)"
|
||||
break
|
||||
fi
|
||||
sleep 2
|
||||
attempt=$((attempt + 2))
|
||||
done
|
||||
fi
|
||||
done
|
||||
|
||||
# After 4 rapid crashes, the instance should be in error state (exceeded max 3 restarts)
|
||||
sleep 5
|
||||
LOOP_FINAL_STATE=$($AWS_EC2 describe-instances --instance-ids "$LOOP_INSTANCE" \
|
||||
--query 'Reservations[0].Instances[0].State.Name' --output text)
|
||||
echo " Final state after crash loop: $LOOP_FINAL_STATE"
|
||||
|
||||
if [ "$LOOP_FINAL_STATE" == "error" ]; then
|
||||
echo " Crash loop prevention working: instance stayed in error state"
|
||||
else
|
||||
echo " WARNING: Instance in state '$LOOP_FINAL_STATE' (expected 'error' after exceeding max restarts)"
|
||||
echo " This may be expected if timing allowed restarts to spread across the window"
|
||||
fi
|
||||
|
||||
echo " Crash recovery tests passed"
|
||||
|
||||
# Cleanup: Terminate all test instances
|
||||
echo ""
|
||||
echo "Cleanup: Deleting test resources"
|
||||
|
||||
Reference in New Issue
Block a user