diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 3bc8ceb92..4cbe05d1b 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -187,3 +187,30 @@ mulga/ ```bash AWS_PROFILE=hive aws ec2 describe-instances ``` + +### Remote Test Cluster + +A multi-node Proxmox test cluster is available for validating changes on real hardware. Node IPs, SSH credentials, and connection details are in `docs/DEVNOTES.md` (gitignored, private to the repo). + +**Lifecycle scripts** (`scripts/iac/`): + +```bash +# Source environment first +source scripts/iac/proxmox/.env && source ~/prox + +# Provision, configure, test, destroy +./scripts/iac/hive-test.sh up # Create VMs +./scripts/iac/hive-test.sh configure # Clone, build, form cluster, start services +./scripts/iac/hive-test.sh test # Smoke tests (describe-regions, instances, etc.) +./scripts/iac/hive-test.sh status # Health check all nodes +./scripts/iac/hive-test.sh ssh # SSH to node N +./scripts/iac/hive-test.sh down # Destroy VMs +``` + +**Deploying local changes to a running cluster:** + +1. Read `docs/DEVNOTES.md` for the current cluster node IPs and SSH key path +2. SSH key: `~/.ssh/us-west-1-tf-cloudinit`, user: `tf-user` +3. Use `git diff` patches or `scp` to push changes, then rebuild on each node +4. Daemon health endpoint is HTTP (not HTTPS): `curl -s http://:4432/health` +5. Always `export PATH=/usr/local/go/bin:$PATH` on remote nodes before builds diff --git a/INSTALL.md b/INSTALL.md index 31325dc55..fec4ae6b5 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -674,7 +674,7 @@ ip addr show | grep "inet " #### 1. Set Variables -Export the IPs for your three servers (replace with your actual IPs): +Export the IPs for your three servers (replace with your actual IPs) on each server: ```bash export HIVE_NODE1=192.168.1.10 @@ -868,16 +868,54 @@ sudo ip addr del 10.11.12.3/24 dev eth0 ## Shutdown -Stop services on each node: +When running in the multi-server cluster mode a graceful shutdown is required to synchronize and coordinate nodes. +On any node, issue the following command to shutdown nodes cluster wide: + +```sh +./bin/hive admin cluster shutdown +``` + +Example output: ```bash -# Simulated — stop in reverse order: -./scripts/stop-dev.sh ~/node3/ -./scripts/stop-dev.sh ~/node2/ -./scripts/stop-dev.sh ~/node1/ +Starting coordinated cluster shutdown (3 nodes) +Phases: gate -> drain -> storage -> persist -> infra +Timeout per phase: 2m0s + +[GATE] Sending to 3 node(s)... + [GATE] node1: stopped awsgw, hive-ui + [GATE] node3: stopped awsgw, hive-ui + [GATE] node2: stopped awsgw, hive-ui +[GATE] Complete (3/3 nodes, 2.005s) + +[DRAIN] Sending to 3 node(s)... + [DRAIN] node1: 0/0 VMs remaining + [DRAIN] node1: 0/0 VMs remaining + [DRAIN] node2: 0/0 VMs remaining + [DRAIN] node2: 0/0 VMs remaining + [DRAIN] node3: 0/0 VMs remaining + [DRAIN] node3: 0/0 VMs remaining + [DRAIN] node1: OK + [DRAIN] node3: OK + [DRAIN] node2: OK +[DRAIN] Complete (3/3 nodes, 5ms) + +[STORAGE] Sending to 3 node(s)... + [STORAGE] node1: stopped viperblock + [STORAGE] node3: stopped viperblock + [STORAGE] node2: stopped viperblock +[STORAGE] Complete (3/3 nodes, 1.09s) + +[PERSIST] Sending to 3 node(s)... + [PERSIST] node1: stopped predastore + [PERSIST] node3: stopped predastore + [PERSIST] node2: stopped predastore +[PERSIST] Complete (3/3 nodes, 1.003s) + +[INFRA] Sending final shutdown to all nodes... +[INFRA] Complete +Cluster shutdown complete (6.104s) -# Real multi-server — run on each server: -./scripts/stop-dev.sh ``` If EC2 instances are running, the stop process will gracefully terminate them, unmount attached EBS volumes (via NBD), and flush the write-ahead-log (WAL) to the S3 server (Predastore). This may take a few minutes depending on instance volume sizes. diff --git a/docs/development/feature/ec2-health-restart.md b/docs/development/feature/ec2-health-restart.md new file mode 100644 index 000000000..ac3beb97e --- /dev/null +++ b/docs/development/feature/ec2-health-restart.md @@ -0,0 +1,1031 @@ +# EC2 Instance Health Monitoring & Auto-Restart + +**Status: In Progress** — Phases A (Crash Detection), B (Auto-Restart), D1 (OOM Scores) implemented + +Related: [Cluster Lifecycle Plan](heartbeat-cluster-shutdown.md) (Phases 2.1-2.7) + +## Summary + +Detect QEMU process crashes at runtime (OOM kills, segfaults, unexpected exits) and automatically restart affected VMs. Protect the host from resource exhaustion by enforcing memory/CPU limits on both system services and VM processes. Expose health state through the AWS-compatible `DescribeInstanceStatus` API so clients get standard SystemStatus/InstanceStatus responses matching AWS behavior. This is a prerequisite for production-grade reliability — without it, a single OOM event can silently kill VMs with no recovery. + +## Context / Problem Statement + +### The OOM Kill Scenario + +Real-world example from development: +``` +[ 6269.170115] oom-kill:constraint=CONSTRAINT_NONE,...task=qemu-system-x86,pid=37970,uid=1000 +[ 6269.171405] Out of memory: Killed process 37970 (qemu-system-x86) + total-vm:9125308kB, anon-rss:8408868kB +``` + +The Linux OOM killer silently terminated QEMU. The daemon had no idea — the VM appeared "running" in state until the next manual check. Resources stayed allocated, the instance was unreachable, and no recovery happened. + +### Current Gaps + +1. **No runtime crash detection**: `cmd.Wait()` completes when QEMU exits (daemon.go:1817), but the exit result channel is only read during the startup handshake (waiting for PID). After that, nobody listens. A crashed QEMU goes unnoticed. + +2. **QMP heartbeat doesn't act on failures**: The 30-second QMP heartbeat (daemon.go:1458-1490) sends `query-status` and logs the result. On error, it `continue`s — no state transition, no alert, no restart attempt. + +3. **No resource limits on any process**: QEMU, predastore, viperblock, nats — all run as bare processes with no cgroup limits, no OOM score adjustments, no memory caps. A single runaway VM or service can exhaust the host. + +4. **Resource accounting leak**: If QEMU crashes without going through `stopInstance()`, the ResourceManager never calls `deallocate()`. The vCPU/memory stay "allocated" — phantom reservations that block new instances from launching. + +5. **Daemon restart recovery exists but runtime recovery doesn't**: `restoreInstances()` (daemon.go:770-891) handles the case where the daemon itself restarted and QEMU may have survived or died. But during normal operation, there's no equivalent — QEMU can crash and sit in a broken state indefinitely. + +### What Already Works + +| Component | File | Status | +|-----------|------|--------| +| QMP heartbeat goroutine (30s `query-status`) | `daemon.go:1458-1490` | Logs only, no action on failure | +| `cmd.Wait()` goroutine per QEMU process | `daemon.go:1817` | Fires on exit, result channel unread after startup | +| `isInstanceProcessRunning()` (signal 0 check) | `daemon.go:893-904` | Used only during recovery, not runtime | +| `restoreInstances()` (daemon restart recovery) | `daemon.go:770-891` | Relaunches crashed VMs after daemon restart | +| `deallocate()` on ResourceManager | `daemon.go:2077-2087` | Called by `stopInstance()`, not by crash handler | +| VM state machine with `StateError` transition | `vm/state.go:41-49` | Exists but rarely used | + +--- + +## Proposed Changes + +### Phase A: Runtime Crash Detection + +The infrastructure for detection already exists — it just needs to be connected. + +#### A1: QEMU Exit Watcher Goroutine + +The `cmd.Wait()` in `StartInstance()` (daemon.go:1817) already blocks until QEMU exits. Currently it sends the exit code to a channel that nobody reads after startup. Fix: launch a dedicated watcher goroutine that outlives the startup phase. + +```go +// In StartInstance(), after successful QEMU launch: +go d.watchInstanceProcess(instance, cmd, exitCh) +``` + +```go +func (d *Daemon) watchInstanceProcess(instance *vm.VM, cmd *exec.Cmd, exitCh <-chan error) { + exitErr := <-exitCh // blocks until QEMU process exits + + // Check if this was an expected exit (stopInstance/terminate already handled it) + if d.shuttingDown.Load() { + return + } + instance.Mu.Lock() + status := instance.Status + instance.Mu.Unlock() + if status == vm.StateStopping || status == vm.StateStopped || + status == vm.StateShuttingDown || status == vm.StateTerminated { + return // expected exit, stopInstance() is handling cleanup + } + + // Unexpected crash + slog.Error("QEMU process exited unexpectedly", + "instance", instance.ID, + "status", status, + "exitErr", exitErr, + "pid", cmd.Process.Pid) + + d.handleInstanceCrash(instance, exitErr) +} +``` + +#### A2: Crash Handler + +```go +func (d *Daemon) handleInstanceCrash(instance *vm.VM, exitErr error) { + // 1. Transition to error state + d.TransitionState(instance, vm.StateError) + + // 2. Deallocate resources (fix phantom reservation) + d.resourceMgr.deallocate(instance) + + // 3. Unmount volumes (cleanup nbdkit) + d.unmountVolumes(instance) + + // 4. Record crash metadata for observability + instance.Mu.Lock() + instance.CrashCount++ + instance.LastCrashTime = time.Now() + instance.LastCrashReason = classifyCrashReason(exitErr) + instance.Mu.Unlock() + + // 5. Persist state + d.WriteState() + + // 6. Attempt auto-restart (if policy allows) + d.maybeRestartInstance(instance) +} +``` + +#### A3: Crash Reason Classification + +Determine why QEMU died — this informs whether restart is safe: + +```go +func classifyCrashReason(exitErr error) string { + if exitErr == nil { + return "clean-exit" // QEMU exited 0 unexpectedly (shouldn't happen) + } + var exitError *exec.ExitError + if errors.As(exitErr, &exitError) { + status := exitError.Sys().(syscall.WaitStatus) + if status.Signaled() { + switch status.Signal() { + case syscall.SIGKILL: + return "oom-killed" // signal 9 = almost certainly OOM + case syscall.SIGSEGV: + return "segfault" + case syscall.SIGABRT: + return "abort" + default: + return fmt.Sprintf("signal-%d", status.Signal()) + } + } + return fmt.Sprintf("exit-%d", status.ExitStatus()) + } + return "unknown" +} +``` + +Cross-reference with dmesg for OOM confirmation: +```go +// After detecting SIGKILL, check dmesg for OOM kill of this PID +func checkDmesgOOM(pid int) bool { + out, err := exec.Command("dmesg", "--time-format", "iso", "-T").Output() + // Search for "oom-kill" + PID in last 60 seconds of dmesg + // Return true if found +} +``` + +### Phase B: Auto-Restart Policy + +Not all crashes should trigger restart. The policy should prevent restart loops and respect resource constraints. + +#### B1: Restart Policy + +```go +type RestartPolicy struct { + MaxRestarts int // max restarts before giving up (default: 3) + RestartWindow time.Duration // window for counting restarts (default: 10 minutes) + BackoffBase time.Duration // initial backoff delay (default: 5 seconds) + BackoffMax time.Duration // maximum backoff delay (default: 2 minutes) + RestartOnOOM bool // restart after OOM kill (default: true) + RestartOnCrash bool // restart after other crashes (default: true) +} +``` + +#### B2: Restart Decision + +```go +func (d *Daemon) maybeRestartInstance(instance *vm.VM) { + policy := d.getRestartPolicy() + + // Check restart count within window + if instance.CrashCount > policy.MaxRestarts { + recentCrashes := countCrashesSince(instance, time.Now().Add(-policy.RestartWindow)) + if recentCrashes >= policy.MaxRestarts { + slog.Error("Instance exceeded max restart attempts, leaving in error state", + "instance", instance.ID, + "crashes", recentCrashes, + "window", policy.RestartWindow) + return + } + } + + // Check crash type + reason := instance.LastCrashReason + if reason == "oom-killed" && !policy.RestartOnOOM { + slog.Warn("OOM restart disabled by policy", "instance", instance.ID) + return + } + + // Check node mode (don't restart on draining node) + if d.nodeMode.Load().(string) != "normal" { + slog.Info("Skipping restart, node not in normal mode", "instance", instance.ID) + return + } + + // Check resource availability + if !d.resourceMgr.canAllocate(instance.InstanceType) { + slog.Warn("Insufficient resources for restart", "instance", instance.ID) + return + } + + // Exponential backoff + delay := policy.BackoffBase * time.Duration(1< policy.BackoffMax { + delay = policy.BackoffMax + } + slog.Info("Scheduling instance restart", + "instance", instance.ID, + "reason", reason, + "delay", delay, + "attempt", instance.CrashCount) + + time.AfterFunc(delay, func() { + d.restartCrashedInstance(instance) + }) +} +``` + +#### B3: Restart Execution + +```go +func (d *Daemon) restartCrashedInstance(instance *vm.VM) { + // Re-verify instance should still restart (state may have changed during backoff) + instance.Mu.Lock() + if instance.Status != vm.StateError { + instance.Mu.Unlock() + return + } + instance.Mu.Unlock() + + slog.Info("Restarting crashed instance", "instance", instance.ID, "attempt", instance.CrashCount) + + // Transition: error → pending + d.TransitionState(instance, vm.StatePending) + + // Relaunch (reuses existing LaunchInstance flow: mount volumes → start QEMU) + err := d.LaunchInstance(instance) + if err != nil { + slog.Error("Failed to restart instance", "instance", instance.ID, "error", err) + d.TransitionState(instance, vm.StateError) + return + } + + slog.Info("Instance restarted successfully", "instance", instance.ID) +} +``` + +### Phase C: Improved QMP Health Monitoring + +The existing 30-second QMP heartbeat should detect more failure modes and trigger recovery. + +#### C1: Enhanced QMP Heartbeat + +Replace the current log-only heartbeat (daemon.go:1458-1490) with an actionable one: + +```go +func (d *Daemon) qmpHealthCheck(instance *vm.VM) { + ticker := time.NewTicker(30 * time.Second) + defer ticker.Stop() + + consecutiveFailures := 0 + const maxFailures = 3 // 90 seconds of QMP unresponsiveness + + for { + select { + case <-d.ctx.Done(): + return + case <-ticker.C: + // Skip terminal states + instance.Mu.Lock() + status := instance.Status + instance.Mu.Unlock() + if status == vm.StateStopping || status == vm.StateStopped || + status == vm.StateShuttingDown || status == vm.StateTerminated { + return + } + + resp, err := d.SendQMPCommand(instance.QMPClient, + qmp.QMPCommand{Execute: "query-status"}, instance.ID) + if err != nil { + consecutiveFailures++ + slog.Warn("QMP health check failed", + "instance", instance.ID, + "consecutive_failures", consecutiveFailures, + "error", err) + + if consecutiveFailures >= maxFailures { + slog.Error("QMP unresponsive, checking process liveness", + "instance", instance.ID) + + if !d.isInstanceProcessRunning(instance) { + // Process is dead — the exit watcher should handle this, + // but if it missed it (race), trigger crash handler + slog.Error("QEMU process dead, triggering crash recovery", + "instance", instance.ID) + d.handleInstanceCrash(instance, fmt.Errorf("qmp unresponsive, process dead")) + return + } + // Process alive but QMP hung — QEMU may be stuck + // Log and continue monitoring; operator can intervene + slog.Error("QEMU process alive but QMP unresponsive", + "instance", instance.ID) + } + continue + } + + // Reset failure counter on success + consecutiveFailures = 0 + } + } +} +``` + +#### C2: Instance Health Status + +Add health fields to the VM struct for observability: + +```go +// In vm/vm.go, add to VM struct: +CrashCount int `json:"crash_count"` +LastCrashTime time.Time `json:"last_crash_time,omitempty"` +LastCrashReason string `json:"last_crash_reason,omitempty"` +HealthStatus string `json:"health_status"` // "healthy", "unhealthy", "recovering" +``` + +Expose via `hive get vms` and the EC2 DescribeInstances response: +``` +INSTANCE | STATUS | HEALTH | TYPE | NODE | CRASHES | LAST CRASH +i-abc123 | running | healthy | t3.nano | node1 | 0 | +i-def456 | running | recovering | t3.nano | node2 | 2 | OOM 3m ago +i-ghi789 | error | unhealthy | t3.nano | node1 | 4 | exceeded max restarts +``` + +### Phase D: Host Resource Protection + +Prevent OOM kills by enforcing resource limits at the OS level. + +> **Dependency**: Phase D3 (memory pressure monitoring) feeds into Phase E's SystemStatus. When host memory pressure is high, the node's SystemStatus transitions to `impaired`. + +#### D1: QEMU Process Memory Limits + +When launching QEMU, set the process memory limit to slightly above the allocated VM memory. This ensures QEMU can't grow beyond its allocation and trigger a host-wide OOM. + +```go +// In vm.go Config.Execute(), add to cmd.SysProcAttr: +func (c *Config) Execute() *exec.Cmd { + cmd := exec.Command(c.QEMUPath, c.Args()...) + + // Set memory limit: VM memory + 256MB overhead (QEMU internal use) + memLimitBytes := (c.MemoryMB + 256) * 1024 * 1024 + cmd.SysProcAttr = &syscall.SysProcAttr{ + // On Linux, use RLIMIT_AS or cgroups + } + + return cmd +} +``` + +**Option A: cgroups v2 (recommended for production)** + +Create a cgroup per QEMU process with a hard memory limit: + +```go +func setCgroupMemoryLimit(pid int, limitBytes int64, instanceID string) error { + cgroupPath := fmt.Sprintf("/sys/fs/cgroup/hive-vms/%s", instanceID) + os.MkdirAll(cgroupPath, 0755) + + // Set memory limit + os.WriteFile(filepath.Join(cgroupPath, "memory.max"), []byte(strconv.FormatInt(limitBytes, 10)), 0644) + + // Move process into cgroup + os.WriteFile(filepath.Join(cgroupPath, "cgroup.procs"), []byte(strconv.Itoa(pid)), 0644) + + return nil +} +``` + +With cgroups, the OOM killer targets only the QEMU process in the cgroup — not random host processes. The `memory.max` is a hard limit; QEMU is killed cleanly if it exceeds it. + +**Option B: OOM score adjustment (simpler, defense-in-depth)** + +Protect system services by lowering their OOM score, and raise QEMU's score so VMs are killed before infrastructure: + +```go +// Protect system services (predastore, viperblock, nats) +func protectSystemProcess(pid int) { + os.WriteFile(fmt.Sprintf("/proc/%d/oom_score_adj", pid), []byte("-500"), 0644) +} + +// Make QEMU processes preferred OOM targets over system services +func setQEMUOOMScore(pid int) { + os.WriteFile(fmt.Sprintf("/proc/%d/oom_score_adj", pid), []byte("500"), 0644) +} +``` + +OOM score range: -1000 (never kill) to 1000 (kill first). Setting system services to -500 and QEMU to +500 ensures the OOM killer sacrifices VMs before infrastructure. + +**Recommendation**: Implement Option B immediately (trivial, no dependencies). Add Option A (cgroups) as a follow-up for hard isolation. + +#### D2: Service Resource Reservations + +Reserve a portion of host resources for system services, reducing the amount available for VMs: + +```go +// In NewResourceManager(), reserve resources for system services +const ( + systemReservedVCPU = 2 // 2 cores for nats/predastore/viperblock/daemon + systemReservedMemGB = 2.0 // 2 GB for system services +) + +func NewResourceManager(...) *ResourceManager { + totalCPU := runtime.NumCPU() + totalMem := getSystemMemoryGB() + + rm := &ResourceManager{ + availableVCPU: totalCPU - systemReservedVCPU, + availableMem: totalMem - systemReservedMemGB, + // ... + } +} +``` + +This is a soft limit (accounting only) but prevents the scheduler from over-committing the host. Hard enforcement comes from cgroups (D1). + +#### D3: Memory Pressure Monitoring + +Monitor host memory pressure and proactively stop accepting new VMs before OOM occurs: + +```go +func (d *Daemon) monitorMemoryPressure() { + ticker := time.NewTicker(10 * time.Second) + defer ticker.Stop() + + for { + select { + case <-d.ctx.Done(): + return + case <-ticker.C: + available := getAvailableMemoryGB() // from /proc/meminfo MemAvailable + total := getTotalMemoryGB() + usedPercent := (1.0 - available/total) * 100 + + if usedPercent > 90 { + slog.Warn("Host memory pressure HIGH", + "used_percent", usedPercent, + "available_gb", available) + // Unsubscribe from RunInstances topics to stop accepting new VMs + // (re-subscribe when pressure drops below 80%) + } + } + } +} +``` + +### Phase E: AWS-Compatible DescribeInstanceStatus API + +Expose VM and host health through the standard AWS `DescribeInstanceStatus` API so clients (AWS CLI, SDKs, Terraform) get the same response format they expect from EC2. + +> **Dependencies**: +> - Phases A-C provide the data (crash state, QMP health, process liveness) +> - Phase D provides host-level health (memory pressure, resource exhaustion) +> - [Phase 2.2 Node Modes](heartbeat-cluster-shutdown.md#phase-22--node-modes--health-monitoring-planned) provides `draining`/`maintenance` states that map to scheduled events + +#### E1: AWS Response Format + +The API returns two independent health dimensions per instance: + +```json +{ + "InstanceStatuses": [ + { + "InstanceId": "i-abc123", + "AvailabilityZone": "ap-southeast-2a", + "InstanceState": { + "Code": 16, + "Name": "running" + }, + "SystemStatus": { + "Status": "ok", + "Details": [ + { + "Name": "reachability", + "Status": "passed" + } + ] + }, + "InstanceStatus": { + "Status": "ok", + "Details": [ + { + "Name": "reachability", + "Status": "passed" + } + ] + }, + "Events": [] + } + ] +} +``` + +**SystemStatus** = host/infrastructure health (is the underlying node healthy?): + +| Condition | SystemStatus | Detail Status | Source | +|-----------|-------------|---------------|--------| +| Node healthy, heartbeat current | `ok` | `passed` | Phase 2.1 heartbeat | +| Node heartbeat stale (>60s) | `impaired` | `failed` | Phase 2.2 `monitorPeerHeartbeats()` | +| Node in `draining` mode | `ok` (still operational) | `passed` | Phase 2.2 node modes | +| Node in `maintenance` mode | `impaired` | `failed` | Phase 2.2 node modes | +| Host memory pressure >90% | `impaired` | `failed` | Phase D3 memory pressure | +| Node unreachable (>120s no heartbeat) | `impaired` | `failed` | Phase 2.2 | + +**InstanceStatus** = VM-level health (is this specific QEMU process healthy?): + +| Condition | InstanceStatus | Detail Status | Source | +|-----------|---------------|---------------|--------| +| QEMU running, QMP responsive | `ok` | `passed` | Phase C QMP health | +| QMP unresponsive (1-2 failures) | `ok` | `passed` | Transient, not yet impaired | +| QMP unresponsive (3+ failures) | `impaired` | `failed` | Phase C consecutive failures | +| QEMU crashed, pending restart | `impaired` | `failed` | Phase A crash detection | +| QEMU crashed, restart failed | `impaired` | `failed` | Phase B restart exceeded | +| Instance stopped/terminated | `not-applicable` | `not-applicable` | Standard EC2 behavior | +| Instance just launched (<2 min) | `initializing` | `initializing` | Grace period | + +**Events** — scheduled events map from [Phase 2.2 node modes](heartbeat-cluster-shutdown.md#phase-22--node-modes--health-monitoring-planned): + +| Node Mode | Event Code | Description | +|-----------|------------|-------------| +| `draining` | `instance-retirement` | "The instance is scheduled for retirement" | +| `maintenance` | `system-maintenance` | "System maintenance is scheduled" | +| Instance recovering from crash | `instance-reboot` | "The instance is being restarted" | + +```json +{ + "Events": [ + { + "InstanceEventId": "evt-drain-node3-1707868800", + "Code": "instance-retirement", + "Description": "Node node3 is being drained for maintenance", + "NotBefore": "2026-02-14T06:00:00+00:00" + } + ] +} +``` + +#### E2: Health State Tracking on VM Struct + +Extend the VM struct to track both dimensions independently: + +```go +// In vm/vm.go, add to VM struct: +type InstanceHealthState struct { + // Instance-level (QEMU process health) + InstanceStatus string `json:"instance_status"` // "ok", "impaired", "initializing", "not-applicable" + InstanceReachability string `json:"instance_reachability"` // "passed", "failed", "initializing" + ImpairedSince time.Time `json:"impaired_since,omitempty"` + + // Crash tracking (feeds into InstanceStatus) + CrashCount int `json:"crash_count"` + LastCrashTime time.Time `json:"last_crash_time,omitempty"` + LastCrashReason string `json:"last_crash_reason,omitempty"` + + // QMP health (feeds into InstanceStatus) + QMPConsecutiveFailures int `json:"qmp_consecutive_failures"` + LastQMPSuccess time.Time `json:"last_qmp_success,omitempty"` +} +``` + +SystemStatus is not per-VM — it's derived from the node's state at response time: + +```go +func (d *Daemon) buildSystemStatus() *ec2.InstanceStatusSummary { + status := "ok" + detailStatus := "passed" + + // Check node mode + mode := d.nodeMode.Load().(string) + if mode == "maintenance" { + status = "impaired" + detailStatus = "failed" + } + + // Check memory pressure + if d.memoryPressureHigh.Load() { + status = "impaired" + detailStatus = "failed" + } + + return &ec2.InstanceStatusSummary{ + Status: aws.String(status), + Details: []*ec2.InstanceStatusDetails{{ + Name: aws.String("reachability"), + Status: aws.String(detailStatus), + }}, + } +} +``` + +InstanceStatus is per-VM, derived from the VM's health state: + +```go +func (d *Daemon) buildInstanceStatus(instance *vm.VM) *ec2.InstanceStatusSummary { + instance.Mu.Lock() + health := instance.Health + status := instance.Status + launchTime := instance.Instance.LaunchTime + instance.Mu.Unlock() + + // Not running → not-applicable + if status != vm.StateRunning && status != vm.StateError { + return &ec2.InstanceStatusSummary{ + Status: aws.String("not-applicable"), + Details: []*ec2.InstanceStatusDetails{{ + Name: aws.String("reachability"), + Status: aws.String("not-applicable"), + }}, + } + } + + // Grace period: <2 minutes since launch → initializing + if launchTime != nil && time.Since(*launchTime) < 2*time.Minute { + return &ec2.InstanceStatusSummary{ + Status: aws.String("initializing"), + Details: []*ec2.InstanceStatusDetails{{ + Name: aws.String("reachability"), + Status: aws.String("initializing"), + }}, + } + } + + // Crashed or QMP unresponsive → impaired + if status == vm.StateError || health.QMPConsecutiveFailures >= 3 { + return &ec2.InstanceStatusSummary{ + Status: aws.String("impaired"), + Details: []*ec2.InstanceStatusDetails{{ + Name: aws.String("reachability"), + Status: aws.String("failed"), + ImpairedSince: &health.ImpairedSince, + }}, + } + } + + return &ec2.InstanceStatusSummary{ + Status: aws.String("ok"), + Details: []*ec2.InstanceStatusDetails{{ + Name: aws.String("reachability"), + Status: aws.String("passed"), + }}, + } +} +``` + +#### E3: Gateway + NATS Implementation + +Follow the exact pattern from `DescribeInstances`: + +**Gateway handler** (`hive/gateway/ec2/instance/DescribeInstanceStatus.go`): +```go +func DescribeInstanceStatus(input *ec2.DescribeInstanceStatusInput, natsConn *nats.Conn, expectedNodes int) (*ec2.DescribeInstanceStatusOutput, error) { + // 1. Marshal input to JSON + // 2. Create inbox, subscribe + // 3. PublishRequest to "ec2.DescribeInstanceStatus" (fan-out) + // 4. Collect responses with 3s timeout + // 5. Aggregate InstanceStatuses from all nodes + // 6. Handle IncludeAllInstances flag (include non-running instances) + // 7. Also query stopped instances via "ec2.DescribeStoppedInstanceStatus" (queue group) + // 8. Return aggregated output +} +``` + +**Daemon handler** (`hive/daemon/daemon_handlers.go`): +```go +func (d *Daemon) handleEC2DescribeInstanceStatus(msg *nats.Msg) { + var input ec2.DescribeInstanceStatusInput + utils.UnmarshalJsonPayload(&input, msg.Data) + + systemStatus := d.buildSystemStatus() // same for all instances on this node + var statuses []*ec2.InstanceStatus + + d.Instances.Mu.Lock() + for _, instance := range d.Instances.VMS { + // Apply instance ID filter + // Apply IncludeAllInstances (default: only running instances) + + instanceStatus := d.buildInstanceStatus(instance) + events := d.buildInstanceEvents(instance) + + statuses = append(statuses, &ec2.InstanceStatus{ + InstanceId: aws.String(instance.ID), + AvailabilityZone: aws.String(d.config.AZ), + InstanceState: &ec2.InstanceState{ + Code: aws.Int64(vm.EC2StateCodes[instance.Status].Code), + Name: aws.String(vm.EC2StateCodes[instance.Status].Name), + }, + SystemStatus: systemStatus, + InstanceStatus: instanceStatus, + Events: events, + }) + } + d.Instances.Mu.Unlock() + + output := &ec2.DescribeInstanceStatusOutput{InstanceStatuses: statuses} + jsonResponse, _ := json.Marshal(output) + msg.Respond(jsonResponse) +} +``` + +**NATS subscription** (`hive/daemon/daemon.go` in `subscribeAll()`): +```go +{"ec2.DescribeInstanceStatus", d.handleEC2DescribeInstanceStatus, ""}, +``` + +**Gateway route** (`hive/gateway/ec2.go` in `ec2Actions`): +```go +"DescribeInstanceStatus": ec2Handler(func(input *ec2.DescribeInstanceStatusInput, gw *GatewayConfig) (any, error) { + return gateway_ec2_instance.DescribeInstanceStatus(input, gw.NATSConn, gw.DiscoverActiveNodes()) +}), +``` + +#### E4: IncludeAllInstances Flag + +Per AWS behavior: +- **Default** (`IncludeAllInstances=false`): Only return instances in `running` state +- **`IncludeAllInstances=true`**: Return all instances including stopped/terminated + +For non-running instances, both SystemStatus and InstanceStatus are `not-applicable`. + +#### E5: Cross-Node SystemStatus for Unreachable Nodes + +When a node is unreachable, the gateway can still report status for instances that were last known on that node. The gateway queries stopped instance state from JetStream KV and builds a response with `SystemStatus: impaired`: + +```go +// In the gateway, after fan-out collection: +// Check for nodes that didn't respond within timeout +// For instances on non-responding nodes (from KV state): +// SystemStatus = "impaired" (host unreachable) +// InstanceStatus = "impaired" (can't verify) +``` + +This requires the gateway to know which nodes own which instances — the JetStream KV `node.` keys already track this. + +--- + +## VM Struct Extensions + +```go +// New fields on vm.VM +type VM struct { + // ... existing fields ... + + // Health monitoring (Phase A-C) + Health InstanceHealthState `json:"health"` +} + +type InstanceHealthState struct { + // Instance-level (QEMU process health) + InstanceStatus string `json:"instance_status"` // "ok", "impaired", "initializing", "not-applicable" + InstanceReachability string `json:"instance_reachability"` // "passed", "failed", "initializing" + ImpairedSince time.Time `json:"impaired_since,omitempty"` + + // Crash tracking + CrashCount int `json:"crash_count"` + LastCrashTime time.Time `json:"last_crash_time,omitempty"` + LastCrashReason string `json:"last_crash_reason,omitempty"` + + // QMP health + QMPConsecutiveFailures int `json:"qmp_consecutive_failures"` + LastQMPSuccess time.Time `json:"last_qmp_success,omitempty"` +} +``` + +--- + +## Files to Modify + +| File | Phase | Action | Description | +|------|-------|--------|-------------| +| `hive/daemon/daemon.go` | A | Edit | Add `watchInstanceProcess()`, `handleInstanceCrash()`, call from `StartInstance()` and `reconnectInstance()` | +| `hive/daemon/daemon.go` | B | Edit | Add `maybeRestartInstance()`, `restartCrashedInstance()` | +| `hive/daemon/daemon.go` | C | Edit | Replace QMP heartbeat goroutine with `qmpHealthCheck()` | +| `hive/daemon/daemon.go` | D | Edit | Add system resource reservation to `NewResourceManager()` | +| `hive/daemon/daemon.go` | E | Edit | Add `ec2.DescribeInstanceStatus` to `subscribeAll()` | +| `hive/daemon/health.go` | A-D | New | `classifyCrashReason()`, `checkDmesgOOM()`, `RestartPolicy`, memory pressure monitor, `buildSystemStatus()`, `buildInstanceStatus()` | +| `hive/daemon/daemon_handlers.go` | E | Edit | Add `handleEC2DescribeInstanceStatus()`, include health in DescribeInstances | +| `hive/vm/vm.go` | A | Edit | Add `InstanceHealthState` struct and `Health` field to VM | +| `hive/vm/vm.go` | D | Edit | OOM score adjustment in `Config.Execute()` | +| `hive/utils/utils.go` | D | Edit | Add `protectSystemProcess()` for OOM score on services | +| `hive/gateway/ec2/instance/DescribeInstanceStatus.go` | E | New | Gateway handler: fan-out, aggregate, XML response | +| `hive/gateway/ec2.go` | E | Edit | Add `DescribeInstanceStatus` to `ec2Actions` map | +| `cmd/hive/cmd/get.go` | C | Edit | Show health status in `hive get vms` output | +| `hive/daemon/health_test.go` | A-C | New | Tests for crash classification, restart policy, backoff | +| `hive/gateway/ec2/instance/DescribeInstanceStatus_test.go` | E | New | Gateway handler tests, multi-node aggregation | + +--- + +## Testing Strategy + +### Unit Tests + +- `TestCrashReasonClassification` — SIGKILL → "oom-killed", SIGSEGV → "segfault", exit code 1 → "exit-1" +- `TestRestartPolicyMaxRestarts` — verify restart is denied after max restarts in window +- `TestRestartPolicyBackoff` — verify exponential backoff: 5s, 10s, 20s, 40s, 80s, 120s (capped) +- `TestRestartPolicyDrainingNode` — no restart when node mode != normal +- `TestResourceDeallocationOnCrash` — verify phantom reservations are freed +- `TestQMPHealthCheckConsecutiveFailures` — 3 failures triggers liveness check +- `TestBuildSystemStatus` — node mode → SystemStatus mapping +- `TestBuildInstanceStatus` — QMP failures/crash state → InstanceStatus mapping +- `TestDescribeInstanceStatusHandler` — NATS handler returns correct format +- `TestDescribeInstanceStatusGateway` — fan-out aggregation, XML response + +### Integration Tests (Docker E2E) + +**Test: QEMU crash detection and restart** + +```bash +# 1. Launch a VM +aws ec2 run-instances --instance-type t3.nano ... + +# 2. Verify running and healthy via describe-instance-status +aws ec2 describe-instance-status --instance-ids $INSTANCE_ID +# SystemStatus: ok, InstanceStatus: ok (or initializing if <2min) + +# 3. Kill QEMU process (simulates crash) +# Inside Docker: kill -9 $(cat ~/node1/run/i-.pid) + +# 4. Wait 5-10 seconds + +# 5. Verify describe-instance-status shows impaired then recovering +aws ec2 describe-instance-status --instance-ids $INSTANCE_ID --include-all-instances +# InstanceStatus: impaired (during crash), then ok (after restart) + +# 6. Verify auto-restart — instance is running again +aws ec2 describe-instances # state: running + +# 7. Verify SSH still works (new QEMU process, same volumes) +ssh -p ec2-user@127.0.0.1 +``` + +**Test: DescribeInstanceStatus AWS CLI compatibility** + +```bash +# 1. Launch 3 VMs across nodes +# 2. describe-instance-status with no args (running only) +aws ec2 describe-instance-status +# Returns 3 instances, all SystemStatus: ok, InstanceStatus: ok + +# 3. Stop one instance +aws ec2 stop-instances --instance-ids $INSTANCE_ID + +# 4. describe-instance-status (default: running only) +aws ec2 describe-instance-status +# Returns 2 instances (stopped one excluded) + +# 5. describe-instance-status --include-all-instances +aws ec2 describe-instance-status --include-all-instances +# Returns 3 instances, stopped one has status: not-applicable +``` + +**Test: OOM score protection** + +```bash +# 1. Check OOM scores +cat /proc/$(cat ~/node1/run/nats.pid)/oom_score_adj # should be -500 +cat /proc/$(cat ~/node1/run/predastore.pid)/oom_score_adj # should be -500 +cat /proc//oom_score_adj # should be +500 +``` + +**Test: Restart loop prevention** + +```bash +# 1. Launch VM +# 2. Kill QEMU 4 times rapidly (exceeds max restarts of 3 in 10 min) +# 3. Verify instance stays in "error" state after 3rd crash +# 4. describe-instance-status shows InstanceStatus: impaired persistently +# 5. Verify hive get vms shows "unhealthy" with crash count +``` + +### Remote Cluster Tests + +**Test: Real OOM kill scenario** + +```bash +# 1. Launch a VM with most of the host memory +# 2. SSH into VM, run: stress-ng --vm 1 --vm-bytes 95% --timeout 120s +# 3. Alternatively, from host: run a memory-hungry process to trigger OOM +# 4. Verify QEMU gets OOM-killed (check dmesg) +# 5. Verify daemon detects crash, logs reason as "oom-killed" +# 6. describe-instance-status shows InstanceStatus: impaired, SystemStatus: impaired (memory pressure) +# 7. Verify auto-restart with backoff +# 8. Verify system services (predastore, viperblock) survived (protected by OOM score) +``` + +**Test: Resource pressure response** + +```bash +# 1. Launch VMs until host is at 85% memory +# 2. Verify hive top shows accurate allocation +# 3. Try to launch another VM — should succeed if capacity allows +# 4. Push to 90% — verify scheduler stops accepting new VMs +# 5. describe-instance-status shows SystemStatus: impaired for all instances on that node +# 6. Terminate a VM — verify scheduler resumes accepting, SystemStatus recovers to ok +``` + +--- + +## Implementation Priority + +``` +Phase A (Crash Detection) ✅ DONE — detects QEMU crashes at runtime + │ + ├── A1: Exit watcher goroutine (startupConfirmed channel in StartInstance) + ├── A2: Crash handler + resource deallocation (handleInstanceCrash) + └── A3: Crash reason classification (classifyCrashReason) + │ +Phase B (Auto-Restart) ✅ DONE — exponential backoff, restart limits + │ + ├── B1: Restart policy (maxRestartsInWindow, restartWindow constants) + ├── B2: Restart decision with backoff (maybeRestartInstance) + └── B3: Restart execution (restartCrashedInstance via LaunchInstance) + │ +Phase C (QMP Improvements) ← defense-in-depth + │ + ├── C1: Enhanced QMP heartbeat + └── C2: Health status fields on VM struct + │ +Phase D (Resource Protection) ← prevents the problem at the source + │ + ├── D1: OOM score adjustment ✅ DONE (QEMU +500, daemon -500, dev services -500) + ├── D1b: cgroups per QEMU (follow-up) + ├── D2: System resource reservation + └── D3: Memory pressure monitoring + │ +Phase E (AWS API) ← exposes all the above via standard EC2 API + │ + ├── E1: DescribeInstanceStatus gateway handler + ├── E2: SystemStatus + InstanceStatus builders in daemon + ├── E3: NATS handler + subscription + ├── E4: IncludeAllInstances flag + └── E5: Cross-node status for unreachable nodes +``` + +**Recommended order**: A → B → D1 → C → E → D2 → D3 → D1b + +Phases A and B are the core — detect crashes and restart VMs. D1 (OOM scores) is trivial and prevents the worst-case scenario. C adds the health state fields that Phase E consumes. E (DescribeInstanceStatus API) is the externally visible surface — once A-C provide the data, E is straightforward plumbing (follows the exact DescribeInstances pattern). D2-D3 are operational polish. D1b (cgroups) is the hardest and least urgent since OOM scores provide 80% of the protection. + +--- + +## Key Design Decisions + +1. **Exit watcher vs polling**: Use `cmd.Wait()` (already exists) rather than periodic PID polling. It's event-driven, zero overhead, and catches the exact moment of exit. The QMP heartbeat is a secondary check for edge cases where the exit channel is missed. + +2. **Restart on same node**: Crashed VMs restart on the same node by default. Cross-node migration is Phase 2.7's concern. If the crash was OOM and resources are tight, the restart policy can optionally defer to the scheduler (future work). + +3. **State file not needed for crash restart**: Unlike cold migration (Phase 2.7a), a crash restart doesn't preserve VM memory — the VM boots fresh from its root volume. This is equivalent to a hard power cycle, which is the correct behavior for an unexpected crash. + +4. **OOM scores over cgroups**: OOM score adjustment is 2 lines of code and works everywhere. Cgroups require root, cgroup v2 availability, and careful setup. Start with OOM scores. + +5. **No restart for terminated instances**: If the user explicitly terminated the VM, don't restart it. The exit watcher checks the VM state before triggering recovery. + +--- + +## Cross-References & Dependencies + +This plan and [heartbeat-cluster-shutdown.md](heartbeat-cluster-shutdown.md) are tightly coupled. Here's the dependency map: + +### What this plan depends on from heartbeat-cluster-shutdown.md + +| Dependency | Phase | Why | +|------------|-------|-----| +| Heartbeats (Phase 2.1) | **Complete** | SystemStatus uses heartbeat staleness to detect unreachable nodes | +| Node Modes (Phase 2.2) | **Planned** | `draining`/`maintenance` modes map to SystemStatus `impaired` and scheduled Events | +| Node Modes (Phase 2.2) | **Planned** | `nodeMode` check in restart policy — don't restart VMs on draining nodes | +| VM Migration (Phase 2.7) | **Planned** | When auto-restart fails (OOM, no resources), cross-node migration is the fallback | + +### What heartbeat-cluster-shutdown.md depends on from this plan + +| Dependency | Phase | Why | +|------------|-------|-----| +| Crash Detection (Phase A) | Core | Phase 2.7 (VM Migration) needs to detect when a VM dies on a failing node to trigger migration | +| Health State (Phase C2) | Core | Phase 2.2 `monitorPeerHeartbeats()` can use QMP health data to decide if a node is "suspect" | +| DescribeInstanceStatus (Phase E) | API | Phase 2.2 node drain should update SystemStatus for all VMs on the draining node | +| Resource Protection (Phase D) | Operational | Phase 2.6 (Degraded Writes) benefits from memory pressure signals to throttle writes | + +### Implementation Order (Combined) + +``` +heartbeat-cluster-shutdown.md ec2-health-restart.md +───────────────────────────── ───────────────────── +Phase 2.1 Heartbeats ✅ +Phase 2.4 Shutdown ✅ + Phase A: Crash Detection ✅ + Phase B: Auto-Restart ✅ + Phase D1: OOM Scores ✅ +Phase 2.2 Node Modes ◄────────────────Phase C: QMP Health + Status Fields + Phase E: DescribeInstanceStatus API +Phase 2.7a Cold Migration ◄───────────(uses crash detection) +Phase 2.5 Rolling Upgrade +Phase 2.6 Degraded Writes ◄───────────(uses memory pressure signals) + Phase D2-D3: Resource Reservation +Phase 2.7b Live Migration + Phase D1b: cgroups +``` + +Phases A, B, D1 from this plan should be implemented **before** Phase 2.2 from heartbeat-cluster-shutdown.md. Phase C and E should be implemented **alongside** Phase 2.2 since they share the health state infrastructure. + +--- + +## Files Modified (Phases A, B, D1) + +| File | Action | Description | +|------|--------|-------------| +| `hive/vm/vm.go` | Edit | Added `InstanceHealthState` struct and `Health` field to VM | +| `hive/vm/state.go` | Edit | Added `StatePending` to `StateError` valid transitions (Error→Pending for restart) | +| `hive/utils/utils.go` | Edit | Added `SetOOMScore()` helper (Linux-only, writes to `/proc//oom_score_adj`) | +| `hive/daemon/health.go` | New | `classifyCrashReason()`, `handleInstanceCrash()`, `unmountInstanceVolumes()`, `maybeRestartInstance()`, `restartCrashedInstance()` | +| `hive/daemon/daemon.go` | Edit | Added `startupConfirmed` channel to QEMU launch goroutine for crash watcher, QEMU OOM score (+500) after `cmd.Start()`, daemon self-protection OOM score (-500) in `Start()` | +| `scripts/start-dev.sh` | Edit | Added `set_oom_score()` helper, set -500 for all infrastructure services | +| `hive/daemon/health_test.go` | New | Tests: crash classification (OOM, exit code, clean, unknown), skip guards, restart limits, window reset, exponential backoff | +| `hive/utils/utils_test.go` | Edit | Added `TestSetOOMScore` and `TestSetOOMScore_InvalidPID` | +| `docs/development/feature/ec2-health-restart.md` | Edit | Updated status, marked A/B/D1 complete | diff --git a/docs/development/feature/heartbeat-cluster-shutdown.md b/docs/development/feature/heartbeat-cluster-shutdown.md index 46b89585f..c604c4252 100644 --- a/docs/development/feature/heartbeat-cluster-shutdown.md +++ b/docs/development/feature/heartbeat-cluster-shutdown.md @@ -1,78 +1,613 @@ -# Phase 2.1 Heartbeats + Phase 2.4 Coordinated Cluster Shutdown +# Cluster Lifecycle: Heartbeats, Shutdown, Upgrades, Resilience, and VM Migration -**Status: Complete** +## Phase 2 Roadmap -## Summary +| Phase | Feature | Status | Description | +|-------|---------|--------|-------------| +| 2.1 | Heartbeats | **Complete** | Daemon heartbeats publish node status to KV every 10s | +| 2.4 | Coordinated Shutdown | **Complete** | Phased cluster-wide shutdown via NATS | +| 2.2 | Node Modes & Health Monitoring | **Planned** | Drain/maintenance/offline modes, stale heartbeat detection | +| 2.5 | Coordinated Rolling Upgrade | **Planned** | Cluster-wide binary upgrade without full downtime | +| 2.6 | Degraded Storage Writes | **Planned** | Predastore shard redirection when nodes are unavailable | +| 2.7 | VM Migration | **Planned** | Relocate running VMs to other nodes during drain/maintenance | +| 2.8 | [EC2 Health & Auto-Restart](ec2-health-restart.md) | **Planned** | QEMU crash detection, auto-restart, OOM protection, `DescribeInstanceStatus` API | -Implemented daemon heartbeats (Phase 2.1) and coordinated cluster shutdown (Phase 2.4). Heartbeats publish node status to KV every 10s. Coordinated shutdown drives a phased shutdown across all nodes via NATS, replacing the error-prone per-node approach for multi-node clusters. +--- -## Context / Problem Statement +## Phase 2.1 — Heartbeats (Complete) -- No cluster-wide health awareness — nodes don't know peer status -- Multi-node shutdown required manual per-node `stop-dev.sh`, which is error-prone and doesn't guarantee correct phase ordering (VMs must stop before storage, storage before NATS) -- Phase 0+1 (capabilities, recovery throttling, shutdown markers) were already complete +### Summary -## Design Decisions +Each daemon publishes its status to JetStream KV every 10 seconds — node name, epoch, running services, VM count, and resource allocation. -- **Option B chosen**: Daemons handle stopping local services internally via `utils.StopProcess()` when they receive NATS shutdown phase messages. The CLI coordinator just sends messages and collects ACKs. -- **Heartbeat interval**: 10 seconds — fast enough for health monitoring, low overhead -- **Shutdown phases**: GATE → DRAIN → STORAGE → PERSIST → INFRA — ordered to prevent data loss -- **INFRA phase is fire-and-forget**: NATS is going down, no ACK possible -- **Scripts**: `stop-dev.sh` auto-detects multi-node and delegates to `hive admin cluster shutdown`; `HIVE_FORCE_LOCAL_STOP=1` overrides for per-service stop +### Implementation -## Proposed Changes (Implemented) - -### Heartbeat System - `Heartbeat` struct in `jetstream.go` with node, epoch, services, VM count, resource allocation - `WriteHeartbeat`/`ReadHeartbeat` KV methods on `JetStreamManager` - `startHeartbeat()` goroutine in `heartbeat.go` — fires immediately, then every 10s - `buildHeartbeat()` reads from `ResourceManager` and `Instances` -### Coordinated Shutdown +### Files Modified + +| File | Action | Description | +|------|--------|-------------| +| `hive/daemon/jetstream.go` | Edit | `Heartbeat` struct + KV methods | +| `hive/daemon/heartbeat.go` | New | `startHeartbeat()`, `buildHeartbeat()` goroutine | +| `hive/daemon/daemon.go` | Edit | Call `startHeartbeat()` in `Start()` | +| `hive/daemon/heartbeat_test.go` | New | Heartbeat tests (build, allocation, KV round-trip) | + +--- + +## Phase 2.4 — Coordinated Cluster Shutdown (Complete) + +### Summary + +`hive admin cluster shutdown` drives a phased shutdown across all nodes via NATS, replacing the error-prone per-node `stop-dev.sh` approach for multi-node clusters. + +### Design Decisions + +- **Option B chosen**: Daemons handle stopping local services internally via `utils.StopProcess()` when they receive NATS shutdown phase messages. The CLI coordinator just sends messages and collects ACKs. +- **Shutdown phases**: GATE → DRAIN → STORAGE → PERSIST → INFRA — ordered to prevent data loss +- **INFRA phase is fire-and-forget**: NATS is going down, no ACK possible +- **Scripts**: `stop-dev.sh` auto-detects multi-node and delegates to `hive admin cluster shutdown`; `HIVE_FORCE_LOCAL_STOP=1` overrides for per-service stop + +### Implementation + - Five shutdown phase handlers in `shutdown.go`: GATE, DRAIN, STORAGE, PERSIST, INFRA - `ShutdownRequest`, `ShutdownACK`, `ShutdownProgress` message types - `shuttingDown atomic.Bool` on Daemon — set during GATE, checked by `setupShutdown()` to skip redundant VM stops - NATS subscriptions for `hive.cluster.shutdown.{phase}` (fan-out, no queue group) - Progress updates published during DRAIN phase - -### CLI Coordinator -- `hive admin cluster shutdown` command in `cluster.go` -- Flags: `--force`, `--timeout`, `--dry-run` -- Executes phases sequentially, collecting ACKs with timeout -- Writes `ClusterShutdownState` to KV for observability -- Live progress display during DRAIN phase - -### Script Updates -- `stop-dev.sh`: `is_multinode()` detection, delegates to coordinated shutdown +- CLI coordinator in `cmd/hive/cmd/cluster.go` with `--force`, `--timeout`, `--dry-run` flags +- `stop-dev.sh`: `is_multinode()` detection, delegates to coordinated shutdown (skipped when data-dir arg passed, e.g. E2E cleanup) - `start-dev.sh`: `is_multinode()` detection, cluster health poll at end -## Files Modified +### Files Modified | File | Action | Description | |------|--------|-------------| -| `hive/daemon/jetstream.go` | Edit | `Heartbeat`, `ClusterShutdownState` structs + KV methods | -| `hive/daemon/heartbeat.go` | New | `startHeartbeat()`, `buildHeartbeat()` goroutine | +| `hive/daemon/jetstream.go` | Edit | `ClusterShutdownState` struct + KV methods | | `hive/daemon/shutdown.go` | New | Shutdown types + 5 phase handlers | -| `hive/daemon/daemon.go` | Edit | `shuttingDown atomic.Bool`, shutdown subscriptions, `startHeartbeat()`, updated `setupShutdown()` | +| `hive/daemon/daemon.go` | Edit | `shuttingDown atomic.Bool`, shutdown subscriptions, updated `setupShutdown()` | | `cmd/hive/cmd/admin.go` | Edit | `clusterCmd` + `clusterShutdownCmd` registration | | `cmd/hive/cmd/cluster.go` | New | `runClusterShutdown()` coordinator | | `scripts/stop-dev.sh` | Edit | Multi-node detection + coordinated shutdown delegation | | `scripts/start-dev.sh` | Edit | Multi-node detection + cluster health poll | -| `hive/daemon/heartbeat_test.go` | New | Heartbeat tests (build, allocation, KV round-trip) | | `hive/daemon/shutdown_test.go` | New | Shutdown tests (marshal, KV round-trip, flag behavior) | -## Testing +### Testing -- `make preflight` — all checks pass (gofmt, go vet, gosec, staticcheck, govulncheck, all tests) -- 9 new unit tests covering heartbeat building, resource reflection, KV round-trips, shutdown message marshaling, and shutdown state management +- `make preflight` — all checks pass +- 9 new unit tests +- `make test-docker-multi` — full multi-node E2E passes (2 consecutive runs confirmed) - `hive admin cluster shutdown --dry-run` prints phase plan - `stop-dev.sh` on multi-node delegates to coordinated shutdown - `HIVE_FORCE_LOCAL_STOP=1 stop-dev.sh` on multi-node uses per-service stop -- `stop-dev.sh` on single-node uses per-service stop (unchanged behavior) +- `stop-dev.sh` on single-node uses per-service stop (unchanged) -## Future Work +--- -- Phase 2.2: Heartbeat-driven health monitoring (stale heartbeat detection, node failure alerts) -- Phase 2.3: Automatic VM migration on node failure -- Integration with `hive top` to display heartbeat data -- Cluster shutdown progress bar in UI +## Phase 2.2 — Node Modes & Health Monitoring (Planned) + +### Problem Statement + +Currently, nodes are either "running" or "not responding." There is no concept of a node being in a transitional state — draining VMs, undergoing maintenance, or recovering from a crash. The health endpoint returns hardcoded `"running"`, and `hive get nodes` shows `"Ready"` or `"NotReady"` with no in-between. + +This means: +- No way to gracefully take a node offline for maintenance without a full cluster shutdown +- No distinction between a planned offline and a crash — rejoining is the same either way +- No stale heartbeat detection — if a node stops publishing heartbeats, nobody notices +- The scheduler keeps routing work to draining nodes until they're fully dead + +### Design: Node Modes + +``` + ┌─────────┐ + │ NORMAL │ ← default, accepts all work + └────┬────┘ + drain cmd │ ▲ drain-cancel / rejoin + ▼ │ + ┌─────────┐ + │DRAINING │ ← rejects new VMs, existing VMs run until migrated/stopped + └────┬────┘ + all VMs │ + stopped │ + ▼ + ┌─────────┐ + │CORDONED │ ← no VMs, services still running, ready for maintenance + └────┬────┘ + maintenance │ ▲ uncordon + cmd ▼ │ + ┌─────────┐ + │MAINTENANCE│ ← services may be stopped, node offline + └─────────┘ +``` + +Add a `NodeMode` type to the Daemon struct (atomic for lock-free reads). Store in JetStream KV (`node..mode`) so it survives daemon restarts and is visible cluster-wide. + +#### Mode Behaviors + +| Mode | Accepts new VMs | Existing VMs | Services | Heartbeats | +|------|----------------|--------------|----------|------------| +| `normal` | Yes | Run | All running | Published | +| `draining` | No | Run (being migrated/stopped) | All running | Published with mode | +| `cordoned` | No | None | All running | Published with mode | +| `maintenance` | No | None | May be stopped | Published until NATS stops | + +#### CLI Commands + +```bash +hive node drain # NORMAL → DRAINING (triggers VM migration/stop) +hive node drain-cancel # DRAINING → NORMAL (abort drain) +hive node cordon # NORMAL/CORDONED (reject new VMs, don't touch existing) +hive node uncordon # CORDONED/DRAINING → NORMAL +hive node maintenance # CORDONED → MAINTENANCE +``` + +#### Implementation Details + +**Heartbeat extension** — add `Mode` field to `Heartbeat` struct: +```go +type Heartbeat struct { + // ... existing fields ... + Mode string `json:"mode"` // "normal", "draining", "cordoned", "maintenance" +} +``` + +**Resource manager integration** — when mode != `normal`, unsubscribe from all `ec2.RunInstances.*` topics. The existing `updateInstanceSubscriptions()` already unsubscribes when capacity is 0; extend it to check mode. + +**Stale heartbeat detection** — new goroutine `monitorPeerHeartbeats()`: +- Every 30s, read all `heartbeat.*` keys from KV +- If any peer's timestamp is >60s old (6 missed heartbeats), mark as `"suspect"` +- If >120s old, log warning and publish alert to `hive.cluster.alert` +- Display in `hive get nodes` as `"Unreachable"` status + +**Graceful rejoin** — when a node starts and finds an existing `node..mode` key: +- If `maintenance`: transition to `normal` after services are healthy +- If `draining`: resume drain (check for remaining VMs) +- Publish heartbeat immediately on startup to signal liveness + +### Proposed Changes + +| File | Action | Description | +|------|--------|-------------| +| `hive/config/config.go` | Edit | Add `NodeMode` type and constants | +| `hive/daemon/daemon.go` | Edit | Add `nodeMode atomic.Value`, check mode in request handlers | +| `hive/daemon/heartbeat.go` | Edit | Add `Mode` to heartbeat, add `monitorPeerHeartbeats()` | +| `hive/daemon/jetstream.go` | Edit | Add `WriteNodeMode`/`ReadNodeMode` KV methods | +| `hive/daemon/daemon_handlers.go` | Edit | Check mode before accepting RunInstances | +| `cmd/hive/cmd/node.go` | New | `hive node drain/cordon/uncordon/maintenance` commands | +| `cmd/hive/cmd/get.go` | Edit | Show mode in `hive get nodes` output | + +### Testing + +- Unit tests for mode transitions, heartbeat staleness detection +- `make test-docker-multi`: drain a node, verify new VMs route elsewhere +- Remote cluster: `hive node drain node3`, verify VMs stop, verify `hive get nodes` shows "Draining" + +--- + +## Phase 2.5 — Coordinated Rolling Upgrade (Planned) + +### Problem Statement + +Upgrading the Hive binary across a cluster currently requires either: +1. Full cluster shutdown → upgrade all nodes → restart (downtime) +2. Manual per-node SSH, stop, upgrade, restart (error-prone, no ordering guarantees) + +Neither approach is acceptable for production clusters. We need a coordinated upgrade that takes nodes offline one at a time, upgrades them, and brings them back — while the cluster continues serving requests on the remaining nodes. + +### Design: Rolling Upgrade via NATS Coordination + +The cluster already has the building blocks: coordinated shutdown phases, heartbeats, and (with Phase 2.2) node modes. A rolling upgrade orchestrates these per-node: + +``` +For each node (one at a time): + 1. DRAIN node (Phase 2.2) — migrate/stop VMs + 2. CORDON node — no new work + 3. Stop services on node (reuse shutdown phases, targeted to single node) + 4. Upgrade binary + 5. Start services + 6. UNCORDON — resume normal operation + 7. Wait for health check before proceeding to next node +``` + +#### Upgrade Delivery Mechanism + +**Option A: Binary push via Predastore (recommended)** + +The upgrade coordinator (any node or external CLI) uploads the new `hive` binary to Predastore (S3-compatible), then each node downloads and replaces its own binary. No SSH needed, no inter-node trust beyond what NATS already provides. + +```bash +# Operator uploads new binary +hive admin cluster upgrade --binary ./bin/hive + +# Internally: +# 1. Upload binary to predastore: s3://hive-system/upgrades/hive- +# 2. For each node: +# a. Send NATS "hive.node.upgrade" to target node +# b. Node downloads binary from predastore +# c. Node replaces /usr/local/bin/hive (or wherever binary lives) +# d. Node restarts services +# e. Wait for heartbeat to confirm healthy +``` + +**Option B: Manual per-node (simpler, good enough for now)** + +The operator runs `hive node drain `, SSHes in, replaces the binary, and runs `hive node uncordon `. The CLI provides the orchestration framework but the binary swap is manual. + +**Option C: SSH-based push (rejected)** + +Having Hive nodes SSH into each other introduces key management complexity, a lateral movement attack surface, and fragile dependency on SSH config. Not recommended. + +#### Decision: Option A for automation, Option B as fallback + +Option A (binary via Predastore) is the right long-term approach — the storage layer already exists, and nodes can self-serve the upgrade. Option B is the immediate path since it requires no new infrastructure. + +### Implementation Plan + +**Phase 2.5a: Manual rolling upgrade (Option B)** + +CLI commands that coordinate the drain/upgrade/uncordon flow but leave the binary swap to the operator: + +```bash +hive admin cluster upgrade --manual +# Prints step-by-step instructions per node: +# 1. Draining node1... done (0 VMs migrated) +# 2. SSH to node1 and run: sudo cp /path/to/new/hive /usr/local/bin/hive +# 3. Press Enter when done... +# 4. Restarting node1 services... +# 5. Waiting for node1 health... healthy +# 6. Proceeding to node2... +``` + +**Phase 2.5b: Automated rolling upgrade (Option A)** + +Add binary upload to Predastore and NATS-triggered self-upgrade: + +```bash +hive admin cluster upgrade --binary ./bin/hive [--version v1.2.3] +``` + +- New NATS handler `hive.node.upgrade` on each daemon +- Handler downloads binary from predastore, validates checksum, replaces in-place +- Daemon exec's itself to restart with new binary (or uses systemd restart) + +### Proposed Changes + +| File | Action | Description | +|------|--------|-------------| +| `cmd/hive/cmd/cluster.go` | Edit | Add `clusterUpgradeCmd` with `--manual`/`--binary` flags | +| `hive/daemon/daemon_handlers.go` | Edit | Add `hive.node.upgrade` handler (Phase 2.5b) | +| `hive/daemon/upgrade.go` | New | Binary download, validation, self-replace logic | + +### Testing + +- `make test-docker-multi`: run upgrade flow (swap binary, verify services restart) +- Remote cluster: full rolling upgrade with `hive admin cluster upgrade --manual` + +--- + +## Phase 2.6 — Degraded Storage Writes (Planned) + +### Problem Statement + +**Current behavior**: Predastore uses Reed-Solomon RS(K, M) erasure coding (default RS(3,2) for 5 nodes, RS(2,1) for 3 nodes). When writing an object, it splits into K data shards + M parity shards and sends each to a deterministic node via consistent hashing. **If any shard destination is unreachable, the entire write fails immediately.** There is no retry, no alternate node selection, no degraded write mode. + +This means a single node being drained or offline makes **all writes fail** — even though RS coding is designed to tolerate M node failures for reads. + +**Desired behavior**: When a node is unavailable (draining, maintenance, or crashed), writes should succeed by redirecting shards to an available alternate node. The metadata should track where shards actually landed, and a background repair process should move shards back to their correct home when the node returns. + +### Current Architecture + +``` +PUT /bucket/key + → Hash object key → consistent hash ring → [node0, node1, node2] for RS(2,1) + → Split into 2 data shards + 1 parity shard + → QUIC Put to node0 (data shard 0) ← fails if node0 is down + → QUIC Put to node1 (data shard 1) + → QUIC Put to node2 (parity shard 0) + → Store ObjectToShardNodes metadata in s3db (Raft) +``` + +Key code: `putObjectViaQUIC()` in `predastore/backend/distributed/distributed.go:506` + +### Design: Shard Redirection with Repair Queue + +#### Write Path Changes + +When a shard destination is unreachable: +1. Catch the QUIC dial/write error for the failing shard +2. Query the hash ring for the next available node (skip nodes marked as draining/maintenance/offline) +3. Write the shard to the alternate node +4. Update `ObjectToShardNodes` metadata to reflect actual placement +5. Enqueue a repair job: "shard X of object Y should be on node Z, currently on node W" + +``` +PUT with node0 down: + → QUIC Put to node0 → FAIL + → Redirect: QUIC Put to node3 (next on ring) + → QUIC Put to node1 (data shard 1) → OK + → QUIC Put to node2 (parity shard 0) → OK + → Store metadata: DataShardNodes=[3, 1], ParityShardNodes=[2] + → Enqueue repair: {object: key, shard: 0, expected: node0, actual: node3} +``` + +#### Node Health Awareness in Predastore + +Predastore needs to know which nodes are available. Options: + +**Option A: Heartbeat-based (NATS KV)** — Predastore reads hive daemon heartbeats from the `hive-cluster-state` KV bucket. Requires Predastore to connect to NATS (it currently doesn't). + +**Option B: Predastore-internal health checks** — Each Predastore node pings peers via QUIC periodically. If a peer fails to respond after N attempts, mark it as unavailable locally. Simpler, self-contained. + +**Option C: Configuration-driven** — The `hive node drain` CLI command updates predastore config (via NATS → daemon → predastore config reload). Predastore skips nodes marked as unavailable. Relies on the Hive daemon to propagate state. + +**Recommendation**: Option B for crash detection (self-contained, no new dependencies), Option C for planned drain/maintenance (explicit, reliable). Both can coexist. + +#### Repair Queue + +When the unavailable node returns: +1. The returning node announces itself (heartbeat resumes or health check passes) +2. A background repair goroutine scans the repair queue +3. For each misplaced shard: + - Read shard from alternate node + - Write shard to correct node + - Update `ObjectToShardNodes` metadata + - Remove repair queue entry +4. Optionally delete the shard from the alternate node (after confirming the correct node has it) + +Repair queue storage: JetStream stream (`HIVE-REPAIR`) or a dedicated KV bucket. Durable, replicated, survives restarts. + +#### Write Consistency Guarantees + +- **Minimum write quorum**: At least K data shards + M parity shards must be written for the PUT to succeed. Redirected shards count toward the quorum. +- **If fewer than K+M nodes are available**: Write fails (cannot satisfy RS requirements). For RS(2,1) with 3 nodes, at most 1 node can be down. +- **Metadata is always authoritative**: `ObjectToShardNodes` records where shards actually are, not where the hash ring says they should be. Reads use metadata, not the ring. + +### Proposed Changes + +| File | Action | Description | +|------|--------|-------------| +| `predastore/backend/distributed/distributed.go` | Edit | Shard redirection in `putObjectViaQUIC()`, alternate node selection | +| `predastore/backend/distributed/repair.go` | New | Repair queue, background repair goroutine | +| `predastore/backend/distributed/health.go` | New | Peer health checks, node availability tracking | +| `predastore/backend/distributed/distributed.go` | Edit | `ObjectToShardNodes` — ensure metadata reflects actual placement | + +### Testing + +- Unit tests: write with simulated node failure, verify redirect and metadata +- `make test-docker-multi`: stop one Predastore node, verify S3 PUTs still succeed, restart node, verify repair +- Remote cluster: drain a node, upload files via AWS CLI, verify objects readable + +--- + +## Phase 2.7 — VM Migration (Planned) + +### Problem Statement + +When a node is drained for maintenance (Phase 2.2), its running VMs are currently stopped and terminated. The user's workload goes offline until they manually launch new instances on other nodes. For a platform aiming at AWS compatibility, this is unacceptable — VMs should be transparently relocated to other nodes with minimal or zero downtime. + +### Background: Why Hive Is Well-Suited for Migration + +Hive's architecture makes VM migration simpler than typical hypervisors: + +1. **Storage is already shared** — all VM disks are backed by Viperblock/Predastore over the network (NBD). Both source and destination nodes can access the same volumes. No disk data transfer needed during migration. +2. **QMP is already implemented** — `hive/qmp/qmp.go` has a working QMP client over UNIX sockets. Migration commands are just more QMP messages. +3. **VM state is in JetStream KV** — instance metadata (`hive-instance-state` bucket) is already shared across the cluster. Cross-node handoff just updates the KV key. +4. **`ResetNodeLocalState()`** — `hive/vm/vm.go` already has a method that zeros out PID, PTS, QMPClient for cross-node handoff. + +### Design: Two-Phase Migration Approach + +#### Phase 2.7a: Cold Migration (Stop → Save → Restore) + +Simplest path — stop the VM, save its memory state to a file, transfer the file, restore on destination. Total downtime: seconds to minutes depending on RAM size. + +**When to use**: Planned maintenance, low-priority VMs, initial implementation. + +``` +Source Node Destination Node +──────────── ──────────────── +1. QMP: stop (pause vCPUs) +2. QMP: migrate "exec:cat > /tmp/state" +3. Wait for migration complete +4. Transfer state file ──────────────→ 5. Receive state file + 6. Start nbdkit for same volumes + 7. Launch QEMU with -incoming "exec:cat /tmp/state" + 8. VM resumes execution +9. Cleanup: stop nbdkit, remove state +10. Update KV: instance now on dest +``` + +**State file transfer options**: +- **Predastore (S3)**: Upload state to `s3://hive-system/migrations/.state`, destination downloads it. Reuses existing infrastructure, works across any network topology. +- **Direct TCP**: Pipe state directly via QEMU's `tcp:` transport. Faster (no intermediate storage), but requires direct connectivity between nodes. +- **NATS object store**: For smaller VMs (<1GB RAM), NATS JetStream object store could work. Built-in replication. + +**Recommendation**: Start with Predastore for simplicity (upload/download pattern), add direct TCP later for performance. + +#### Phase 2.7b: Live Migration (Pre-Copy) + +Minimal downtime — VM runs on source while memory is copied to destination. Final switchover pauses VM for <500ms. + +**When to use**: Production workloads, load balancing, zero-downtime maintenance. + +``` +Source Node Destination Node +──────────── ──────────────── + 1. Start nbdkit for same volumes + 2. Launch QEMU with -incoming defer + 3. QMP: migrate-incoming "tcp:0:PORT" +4. QMP: migrate-set-capabilities + [auto-converge, multifd] +5. QMP: migrate "tcp:dest:PORT" +6. ── pre-copy: bulk RAM transfer ──→ +7. ── iterative: dirty pages ──────→ +8. ── converged: final switchover ──→ 9. VM resumes on destination +10. Cleanup source +11. Update KV +``` + +**QMP commands** (existing `SendQMPCommand` infrastructure): + +```go +// On source: +d.SendQMPCommand(qmpClient, QMPCommand{ + Execute: "migrate-set-capabilities", + Arguments: map[string]any{"capabilities": []map[string]any{ + {"capability": "auto-converge", "state": true}, + }}, +}, instanceID) + +d.SendQMPCommand(qmpClient, QMPCommand{ + Execute: "migrate", + Arguments: map[string]any{"uri": fmt.Sprintf("tcp:%s:%d", destHost, destPort)}, +}, instanceID) + +// Monitor progress: +resp := d.SendQMPCommand(qmpClient, QMPCommand{Execute: "query-migrate"}, instanceID) +// resp.Return.status: "active", "completed", "failed" +``` + +**Convergence**: For typical VMs (512MB-8GB RAM) on 10GbE, pre-copy converges in 5-30 seconds with <300ms switchover. `auto-converge` throttles guest vCPUs if dirty page rate exceeds transfer rate. + +**TLS for migration streams**: Use the cluster's existing CA (generated by `dev-setup.sh` / `hive admin init`). Each node gets a dual-role TLS cert for QEMU migration. QEMU's `-object tls-creds-x509` handles the TLS handshake natively. + +### Migration Coordination Flow (NATS) + +``` +CLI: hive node drain node3 + → NATS "hive.node.setmode" {node: "node3", mode: "draining"} + +Daemon (node3): mode → draining + → Unsubscribe from ec2.RunInstances.* topics + → For each running VM: + 1. Select destination: pick node with most available resources + mode=normal + 2. NATS Request "hive.node.prepare-migration" → destination daemon + Payload: {instanceID, volumes, vmConfig, migrationType: "live"|"cold"} + 3. Destination prepares: start nbdkit, launch QEMU with -incoming + 4. Destination responds: {ready: true, migrationEndpoint: "tcp:10.1.3.170:49152"} + 5. Source initiates QEMU migration + 6. Monitor progress, publish to "hive.cluster.migration.progress" + 7. On completion: + - Source: cleanup local QEMU, nbdkit, update KV + - Destination: update KV with new ownership + 8. Move to next VM + +When all VMs migrated: + → mode → cordoned (ready for maintenance) +``` + +### Destination VM Preparation + +The destination daemon must reconstruct an identical QEMU environment: + +1. **Mount volumes**: Send `ebs.mount` NATS requests for each of the VM's volumes. Viperblock starts nbdkit processes pointing at the same underlying data. +2. **Build QEMU command**: Same CPU, memory, machine type, devices, network config as source. Add `-incoming defer` flag. +3. **Network setup**: Same MAC address (preserved in migration stream), same port forwarding rules. +4. **Start QEMU**: Process starts but vCPUs are paused, waiting for migration data. + +Key reuse point: the VM struct's `Config.Execute()` method in `hive/vm/vm.go` already builds the QEMU command. A variant with `-incoming` is straightforward. + +### Post-Migration Cleanup + +On the source node after successful migration: +1. Kill the source QEMU process (should already be exited after migration completes) +2. Unmount volumes (send `ebs.unmount` for each volume) +3. Remove local state (PID file, QMP socket, serial PTS) +4. Update JetStream KV: move instance from `node..instances` to `node..instances` + +The existing `ResetNodeLocalState()` method zeros out PID, PTS, QMPClient — exactly what's needed for the handoff. + +### Failure Handling + +| Failure | During Cold Migration | During Live Migration | +|---------|----------------------|----------------------| +| Source dies mid-migration | State file may be incomplete; VM is lost. Restart from stopped state on any node. | Migration aborts. If source QEMU still running, VM continues on source. If not, VM lost. | +| Destination dies mid-migration | Source VM is paused; resume on source. Retry with different destination. | Migration aborts. VM continues running on source (pre-switchover). | +| Network partition | State file transfer fails; resume on source. | Pre-copy: migration stalls, eventually times out, VM stays on source. Post-switchover: VM on destination, source can't clean up (handled on reconnect). | +| Storage failure | Both sides fail equally (shared storage). | Same — storage is shared, not part of migration stream. | + +### Proposed Changes + +| File | Action | Description | +|------|--------|-------------| +| `hive/daemon/migration.go` | New | Migration coordinator: prepare, execute (cold/live), cleanup | +| `hive/daemon/daemon_handlers.go` | Edit | Add `hive.node.prepare-migration` NATS handler | +| `hive/qmp/qmp.go` | Edit | Add migration QMP command types and helpers | +| `hive/vm/vm.go` | Edit | Add `IncomingConfig()` variant for destination QEMU launch | +| `cmd/hive/cmd/node.go` | Edit | `hive node drain` triggers migration flow | + +### Testing + +- Unit tests: migration state machine, QMP command construction +- `make test-docker-multi`: cold migration between nodes (Docker E2E doesn't have KVM so live migration won't work there — use remote cluster) +- Remote cluster (`scripts/iac/hive-test.sh`): full live migration test + 1. Launch VM on node3 + 2. SSH into VM, start a long-running process + 3. `hive node drain node3` + 4. Verify VM appears on another node + 5. SSH into VM again, verify process still running (live) or VM rebooted cleanly (cold) + +--- + +## Implementation Priority & Dependencies + +``` +Phase 2.8 A-B-D1 (Crash Detection, Auto-Restart, OOM Scores) + │ ↑ no external dependencies, can start immediately + │ +Phase 2.2 (Node Modes) ◄── Phase 2.8 C+E (QMP Health, DescribeInstanceStatus API) + │ ↑ shares health state infrastructure with 2.2 + │ + ├──→ Phase 2.7a (Cold Migration) — requires drain mode + crash detection (2.8 A) + │ │ + │ └──→ Phase 2.7b (Live Migration) — builds on cold migration + │ + ├──→ Phase 2.5 (Rolling Upgrade) — requires drain/uncordon + │ + ├──→ Phase 2.6 (Degraded Writes) — requires node health awareness + │ ↑ benefits from memory pressure signals (2.8 D2-D3) + │ + └──→ Phase 2.8 D2-D3-D1b (Resource Reservation, Pressure Monitoring, cgroups) +``` + +**Recommended order**: 2.8(A,B,D1) → 2.2 + 2.8(C,E) → 2.7a → 2.5 → 2.6 → 2.7b → 2.8(D2,D3,D1b) + +Rationale: +- **2.8 A,B,D1 first**: Zero external dependencies. Fixes the silent QEMU crash problem immediately. OOM scores are 2 lines of code for massive protection. See [ec2-health-restart.md](ec2-health-restart.md) for details. +- **2.2 + 2.8 C,E together**: Node modes and health monitoring share the same state infrastructure. DescribeInstanceStatus API consumes the health data that 2.2's `monitorPeerHeartbeats()` and 2.8 C's QMP health check produce. +- **2.7a third**: Cold migration provides immediate value — VMs survive maintenance windows. Uses crash detection (2.8 A) for failure cases. +- **2.5 fourth**: Rolling upgrades become practical once drain + cold migration work. +- **2.6 fifth**: Degraded writes are important but require changes to Predastore (separate repo). Can be done in parallel with 2.7a. +- **2.7b sixth**: Live migration is the polish — sub-second downtime instead of seconds. Only worth the complexity after cold migration is proven. +- **2.8 D2,D3,D1b last**: Operational polish. Resource reservation and cgroups prevent OOM at the source but aren't blocking for other features. + +--- + +## Key Existing Code to Reuse + +| Function / Type | File | Relevance | +|----------------|------|-----------| +| `SendQMPCommand()` | `hive/daemon/daemon.go` | QMP dispatch — add migration commands | +| `Config.Execute()` | `hive/vm/vm.go` | QEMU command builder — add `-incoming` variant | +| `ResetNodeLocalState()` | `hive/vm/vm.go` | Zeros PID/PTS/QMP for cross-node handoff | +| `MountVolumes()` | `hive/daemon/daemon.go` | NBD volume setup — reuse on destination | +| `stopInstance()` | `hive/daemon/daemon.go` | QMP shutdown + cleanup — reuse for source cleanup | +| `collectResponses()` | `cmd/hive/cmd/get.go` | NATS fan-out pattern — reuse for node queries | +| `loadConfigAndConnect()` | `cmd/hive/cmd/get.go` | CLI NATS connection — reuse for new CLI commands | +| `utils.StopProcess()` | `hive/utils/utils.go` | PID-based service stop — reuse in drain/upgrade | +| `putObjectViaQUIC()` | `predastore/backend/distributed/distributed.go` | Write path — modify for shard redirection | +| `ObjectToShardNodes` | `predastore/backend/distributed/distributed.go` | Shard metadata — extend for actual placement | +| `hashRing.GetClosestN()` | `predastore/backend/distributed/distributed.go` | Node selection — extend to skip unavailable nodes | + +--- + +## Verification Strategy + +All phases will be verified with: +1. `make preflight` — format, vet, security, all unit tests +2. `make test-docker-multi` — multi-node E2E in Docker +3. Remote Proxmox cluster — real hardware validation via `scripts/iac/hive-test.sh` + +The remote test cluster is critical for Phase 2.7b (live migration) since Docker E2E doesn't support nested KVM for QEMU-to-QEMU migration. diff --git a/docs/development/improvements/iac-testing-aws-terraform.md b/docs/development/improvements/iac-testing-aws-terraform.md index 53cbf2fdb..83ef65ea8 100644 --- a/docs/development/improvements/iac-testing-aws-terraform.md +++ b/docs/development/improvements/iac-testing-aws-terraform.md @@ -213,3 +213,202 @@ Add a Terraform E2E stage to CI that runs the full `init → plan → apply → - Add S3 endpoint support (Predastore already S3-compatible) - Implement IAM basics for multi-tenant access control - Add CloudWatch-compatible metrics endpoint + +--- + +## TODO: Pre-Baked Proxmox VM Template for Test Clusters + +### Problem + +Spinning up a new Hive test cluster on Proxmox currently takes a long time because every new VM starts from a base Debian 12 cloud image and must: + +1. `apt-get update && apt-get install` ~20 packages (QEMU, nbdkit, libvirt, gcc, jq, etc.) +2. Download and install Go 1.26.0 (~150MB) +3. Download and install AWS CLI v2 (~100MB) +4. `git clone` hive, viperblock, and predastore repos +5. `go mod download` for all three repos (~hundreds of MB of Go modules) +6. `make build` / `dev-setup.sh` to compile everything + +This takes 10-15+ minutes per node, multiplied by 3 nodes. A pre-baked template with all dependencies pre-installed would reduce cluster spin-up to: boot VM, `git pull`, `make build` (~1-2 minutes). + +### Current State + +The existing Proxmox IaC (`scripts/iac/proxmox/`) uses: +- Base image: `debian-12-genericcloud-amd64-20240211-1654.img` +- Cloud-init: installs only `qemu-guest-agent`, `net-tools`, `git`, `make` +- Everything else is done manually via SSH after boot + +The E2E Docker image (`tests/e2e/Dockerfile.e2e`) already has the canonical install sequence: +``` +make install-system # QEMU, nbdkit, libvirt, gcc, jq, curl, etc. +make install-go # Go 1.26.0 +make install-aws # AWS CLI v2 +``` + +### Approaches Considered + +#### Option A: Packer (Recommended) + +[Packer](https://www.packer.io/) is the industry standard for building VM images. It has a native [Proxmox builder](https://developer.hashicorp.com/packer/integrations/hashicorp/proxmox) (`proxmox-iso` / `proxmox-clone`). + +**How it works:** +1. Packer boots a VM from the base Debian cloud image on Proxmox +2. Runs provisioners (shell scripts) to install dependencies +3. Converts the VM to a Proxmox template +4. Future VMs clone from this template + +**Pros:** +- Industry standard — used by Kubernetes (image-builder), HashiCorp (Nomad/Consul AMIs), and most cloud-native projects +- Reproducible and version-controlled (`.pkr.hcl` files in git) +- Can reuse existing Makefile targets (`make install-system install-go install-aws`) +- Supports multi-platform (same Packer template could build Proxmox templates AND local QEMU images) +- Integrates with existing Proxmox IaC — just change `var.os_image` to point to the new template +- Template can be rebuilt automatically when dependencies change (CI/scheduled) + +**Cons:** +- Adds Packer as a dependency (though it's a single binary, same as Terraform/OpenTofu) +- Template build takes 10-15 minutes (but only done once, not per-cluster) + +**Proposed file structure:** +``` +scripts/iac/packer/ +├── hive-node.pkr.hcl # Main Packer template +├── variables.pkr.hcl # Proxmox connection vars, image settings +├── scripts/ +│ ├── base-setup.sh # System deps (calls make install-system) +│ ├── dev-tools.sh # Go + AWS CLI (calls make install-go install-aws) +│ ├── clone-repos.sh # git clone hive, viperblock, predastore +│ └── cleanup.sh # Clear apt cache, truncate logs, zero free space +└── README.md +``` + +**Layered template strategy:** +- **Layer 1 — `hive-base`**: System deps + Go + AWS CLI + cloud image cached. Rebuilt monthly or when deps change. This is the slow layer (~10 min) but changes rarely. +- **Layer 2 — `hive-dev`** (optional): Extends `hive-base` with git clone + `go mod download` + pre-built binaries. Rebuilt more frequently. Cluster spin-up becomes just `git pull && make build`. + +Alternatively, a single template with everything is simpler and sufficient for our scale. + +**Example Packer template sketch:** +```hcl +source "proxmox-iso" "hive-node" { + proxmox_url = var.proxmox_endpoint + node = var.proxmox_node + iso_file = var.base_image + os = "l26" + cores = 4 + memory = 8192 + disk_size = "32G" + ssh_username = "root" + template_name = "hive-node-template" + template_description = "Hive test node with all dependencies pre-installed" +} + +build { + sources = ["source.proxmox-iso.hive-node"] + + # Copy Makefile for install targets + provisioner "file" { + source = "../../../Makefile" + destination = "/tmp/Makefile" + } + + provisioner "shell" { + inline = [ + "cd /tmp && make -f Makefile install-system", + "cd /tmp && make -f Makefile install-go", + "cd /tmp && make -f Makefile install-aws", + ] + } + + # Clone repos + provisioner "shell" { + script = "scripts/clone-repos.sh" + } + + # Cleanup for smaller template + provisioner "shell" { + script = "scripts/cleanup.sh" + } +} +``` + +#### Option B: Enhanced Cloud-Init + +Expand the existing cloud-init in `scripts/iac/proxmox/cloud-config.tf` to install everything on first boot. + +**Pros:** +- No extra tools — works with current Terraform setup +- Single source of truth (cloud-config.yaml) + +**Cons:** +- Runs on **every** new VM, not once-and-reuse (10-15 min per node, every time) +- Cloud-init has timeouts for long-running operations (Go download, module cache) +- Not a template — doesn't solve the speed problem, just automates the manual steps +- Harder to debug when things fail (cloud-init logs are buried) + +**Verdict:** Good for basic setup (current use), but doesn't solve the speed problem. + +#### Option C: Manual Template + Script + +SSH into a running VM, install everything, then convert to a Proxmox template via API: +```bash +# On the Proxmox host: +qm shutdown +qm template +``` + +**Pros:** +- Simplest approach, works right now +- No extra tools + +**Cons:** +- Not reproducible — "it works on my cluster" problem +- Not version-controlled +- Easy to forget what was installed or configured +- Stale templates are hard to update (must redo from scratch or maintain a VM) + +**Verdict:** Acceptable for prototyping, but should be replaced with Packer. + +#### Option D: qemu-img + chroot (Build Locally) + +Mount a cloud image locally with `qemu-nbd`, chroot into it, install dependencies, then upload to Proxmox. + +**Pros:** +- No need to boot a VM on Proxmox +- Fast iteration locally + +**Cons:** +- Needs root access and `qemu-nbd` on the build machine +- Architecture-dependent (must match target arch or use binfmt_misc) +- Fragile — chroot environments often have networking/DNS issues +- Not standard practice + +**Verdict:** Interesting for CI-built images, but too complex for our needs. + +### What Other Projects Do + +| Project | Approach | Notes | +|---|---|---| +| **Kubernetes image-builder** | Packer + Ansible | Builds node images for every cloud + bare-metal. Packer templates with Ansible provisioners. | +| **HashiCorp reference architectures** | Packer | Pre-baked AMIs/templates for Nomad, Consul, Vault with all binaries installed | +| **Talos Linux** | Custom OS | Entire immutable OS image built from scratch — overkill for us | +| **Flatcar / CoreOS** | Immutable OS + Ignition | Pre-built OS with container runtime, config applied at boot via Ignition | +| **Proxmox community** | Packer (`proxmox-iso` builder) | Dominant approach in the Proxmox ecosystem | +| **GitLab CI runners** | Packer | Pre-baked runner images with Docker, tools pre-installed | + +### Recommendation + +**Start with Option A (Packer)** for the reusable template, with **Option C as an interim** step if we want something working today. + +Concrete next steps: +1. Manually create a template once (Option C) to unblock testing immediately +2. Codify the process as a Packer template (Option A) for reproducibility +3. Integrate template rebuilds into CI (rebuild weekly or on Makefile dependency changes) + +### Open Questions + +- **Single template vs layered?** One template with everything is simpler. Two layers (base + dev) would allow faster rebuilds when only code changes but deps stay the same. +- **Include pre-built binaries?** If the template includes compiled Hive/Viperblock/Predastore, spin-up is nearly instant but the template goes stale faster. If it only includes Go modules, `make build` takes ~30s. +- **Go module cache strategy?** Pre-populate `~/go/pkg/mod/` in the template? Or just `go mod download` on first boot? +- **Cloud image caching?** Pre-download the Ubuntu/Debian cloud image used for nested VMs (like the Docker E2E image does with `/root/images/ubuntu-24.04.img`)? +- **Template naming/versioning?** `hive-node-v1`, `hive-node-YYYYMMDD`, or `hive-node-`? diff --git a/hive/config/config.go b/hive/config/config.go index 4530b3f41..f79354058 100644 --- a/hive/config/config.go +++ b/hive/config/config.go @@ -303,7 +303,8 @@ func LoadConfig(configPath string) (*ClusterConfig, error) { if err := viper.ReadInConfig(); err != nil { return nil, fmt.Errorf("error reading config file: %w", err) } - fmt.Fprintf(os.Stderr, "Using config file: %s\n", viper.ConfigFileUsed()) + //fmt.Fprintf(os.Stderr, "Using config file: %s\n", viper.ConfigFileUsed()) + } else { fmt.Fprintf(os.Stderr, "Config file not found: %s, using environment variables and defaults\n", configPath) } diff --git a/hive/daemon/daemon.go b/hive/daemon/daemon.go index 441ac7aee..6ca22ccd3 100644 --- a/hive/daemon/daemon.go +++ b/hive/daemon/daemon.go @@ -609,6 +609,11 @@ func (d *Daemon) Start() error { } d.accountService = accountSvc + // Protect daemon from OOM killer (prefer killing QEMU VMs instead) + if err := utils.SetOOMScore(os.Getpid(), -500); err != nil { + slog.Warn("Failed to set daemon OOM score", "err", err) + } + d.waitForClusterReady() d.restoreInstances() @@ -1732,6 +1737,7 @@ func (d *Daemon) StartInstance(instance *vm.VM) error { processChan := make(chan int, 1) exitChan := make(chan int, 1) ptsChan := make(chan int, 1) + startupConfirmed := make(chan bool, 1) go func() { cmd, err := instance.Config.Execute() @@ -1766,6 +1772,11 @@ func (d *Daemon) StartInstance(instance *vm.VM) error { slog.Info("VM started successfully", "pid", cmd.Process.Pid) + // Set OOM score for QEMU process (prefer killing VMs over system services) + if err := utils.SetOOMScore(cmd.Process.Pid, 500); err != nil { + slog.Warn("Failed to set QEMU OOM score", "pid", cmd.Process.Pid, "err", err) + } + // TODO: Consider workaround using QMP // (QEMU) query-chardev // {"return": [{"frontend-open": true, "filename": "vc", "label": "parallel0"}, {"frontend-open": true, "filename": "unix:/run/user/1000/qmp-i-150340b52b20c0b43.sock,server=on", "label": "compat_monitor0"}, {"frontend-open": true, "filename": "pty:/dev/pts/9", "label": "serial0"}]} @@ -1813,15 +1824,27 @@ func (d *Daemon) StartInstance(instance *vm.VM) error { processChan <- cmd.Process.Pid - // Read the pts from launch - err = cmd.Wait() + // Block until QEMU exits + waitErr := cmd.Wait() - if err != nil { - slog.Error("Failed to wait for VM:", "err", err) - exitChan <- 1 - return + if waitErr != nil { + slog.Error("VM process exited", "instance", instance.ID, "err", waitErr) } + // Signal startup check (non-blocking) + select { + case exitChan <- 1: + default: + } + + // Wait for startup phase to complete before deciding on crash handling + confirmed := <-startupConfirmed + if !confirmed { + return // Startup failed, LaunchInstance handles the error + } + + // Runtime crash — handle it + d.handleInstanceCrash(instance, waitErr) }() // Wait for startup result @@ -1843,16 +1866,17 @@ func (d *Daemon) StartInstance(instance *vm.VM) error { //return fmt.Errorf("failed to get pts") } - // Check if nbdkit exited immediately with an error + // Check if QEMU exited immediately with an error select { case exitErr := <-exitChan: + startupConfirmed <- false // tell goroutine not to handle crash if exitErr != 0 { errorMsg := fmt.Errorf("failed: %v", exitErr) slog.Error("Failed to launch qemu", "err", errorMsg) return errorMsg } default: - // nbdkit is still running after 1 second, which means it started successfully + startupConfirmed <- true // goroutine will handle future crashes slog.Info("QEMU started successfully and is running", "pts", pts) } diff --git a/hive/daemon/health.go b/hive/daemon/health.go new file mode 100644 index 000000000..9ca559679 --- /dev/null +++ b/hive/daemon/health.go @@ -0,0 +1,274 @@ +package daemon + +import ( + "encoding/json" + "errors" + "fmt" + "log/slog" + "os" + "os/exec" + "syscall" + "time" + + "github.com/mulgadc/hive/hive/vm" +) + +const ( + maxRestartsInWindow = 3 + restartWindow = 10 * time.Minute + restartBackoffBase = 5 * time.Second + restartBackoffMax = 2 * time.Minute +) + +// classifyCrashReason extracts a human-readable crash reason from the error +// returned by cmd.Wait(). Uses exec.ExitError + syscall.WaitStatus to +// distinguish OOM kills (SIGKILL), segfaults (SIGSEGV), etc. +func classifyCrashReason(waitErr error) string { + if waitErr == nil { + return "clean-exit" + } + + var exitErr *exec.ExitError + if !errors.As(waitErr, &exitErr) { + return "unknown" + } + + status, ok := exitErr.Sys().(syscall.WaitStatus) + if !ok { + return "unknown" + } + + if status.Signaled() { + switch status.Signal() { + case syscall.SIGKILL: + return "oom-killed" + case syscall.SIGSEGV: + return "segfault" + case syscall.SIGABRT: + return "abort" + default: + return fmt.Sprintf("signal-%d", status.Signal()) + } + } + + if status.Exited() { + return fmt.Sprintf("exit-%d", status.ExitStatus()) + } + + return "unknown" +} + +// handleInstanceCrash is called by the QEMU launch goroutine after cmd.Wait() +// returns during runtime (after startup was confirmed successful). It detects +// the crash reason, transitions the instance to error state, cleans up resources, +// and schedules a restart if policy allows. +func (d *Daemon) handleInstanceCrash(instance *vm.VM, waitErr error) { + // Guard: if instance is not running, this was an expected exit + // (stopInstance/terminateInstance set status before QEMU exits) + d.Instances.Mu.Lock() + status := instance.Status + d.Instances.Mu.Unlock() + + if status != vm.StateRunning { + slog.Debug("QEMU exited but instance not in running state, skipping crash handler", + "instance", instance.ID, "status", status) + return + } + + // Guard: coordinated shutdown in progress + if d.shuttingDown.Load() { + slog.Debug("QEMU exited during coordinated shutdown, skipping crash handler", + "instance", instance.ID) + return + } + + reason := classifyCrashReason(waitErr) + slog.Error("VM process crashed", "instance", instance.ID, "reason", reason, "err", waitErr) + + // Transition to error state + if err := d.TransitionState(instance, vm.StateError); err != nil { + slog.Error("Failed to transition crashed instance to error state", + "instance", instance.ID, "err", err) + } + + // Update health tracking + now := time.Now() + d.Instances.Mu.Lock() + instance.Health.CrashCount++ + instance.Health.LastCrashTime = now + instance.Health.LastCrashReason = reason + if instance.Health.FirstCrashTime.IsZero() { + instance.Health.FirstCrashTime = now + } + instance.Running = false + instance.PID = 0 + d.Instances.Mu.Unlock() + + // Deallocate resources to fix phantom reservation + instanceType, ok := d.resourceMgr.instanceTypes[instance.InstanceType] + if ok && instanceType != nil { + slog.Info("Deallocating resources for crashed instance", + "instance", instance.ID, "type", instance.InstanceType) + d.resourceMgr.deallocate(instanceType) + } + + // Clean up stale QMP socket so QEMU can rebind on restart + if instance.Config.QMPSocket != "" { + os.Remove(instance.Config.QMPSocket) + } + + // Unmount EBS volumes (same pattern as stopInstance) + d.unmountInstanceVolumes(instance) + + // Persist state + if err := d.WriteState(); err != nil { + slog.Error("Failed to persist state after crash handling", + "instance", instance.ID, "err", err) + } + + d.maybeRestartInstance(instance) +} + +// unmountInstanceVolumes sends NATS unmount requests for all volumes attached +// to the instance and updates their state to "available". +func (d *Daemon) unmountInstanceVolumes(instance *vm.VM) { + instance.EBSRequests.Mu.Lock() + defer instance.EBSRequests.Mu.Unlock() + + for _, ebsRequest := range instance.EBSRequests.Requests { + ebsUnMountRequest, err := json.Marshal(ebsRequest) + if err != nil { + slog.Error("Failed to marshal volume payload for crash cleanup", + "err", err) + continue + } + + msg, err := d.natsConn.Request(d.ebsTopic("unmount"), ebsUnMountRequest, 30*time.Second) + if err != nil { + slog.Error("Failed to unmount volume after crash", + "name", ebsRequest.Name, "instance", instance.ID, "err", err) + } else { + slog.Info("Unmounted volume after crash", + "instance", instance.ID, "volume", ebsRequest.Name, "data", string(msg.Data)) + } + + // Update user-visible volume state to "available" + if !ebsRequest.EFI && !ebsRequest.CloudInit { + if err := d.volumeService.UpdateVolumeState(ebsRequest.Name, "available", "", ""); err != nil { + slog.Error("Failed to update volume state to available after crash", + "volumeId", ebsRequest.Name, "err", err) + } + } + } +} + +// maybeRestartInstance checks restart policy and schedules a restart if allowed. +func (d *Daemon) maybeRestartInstance(instance *vm.VM) { + if d.shuttingDown.Load() { + slog.Info("Skipping restart during shutdown", "instance", instance.ID) + return + } + + now := time.Now() + + d.Instances.Mu.Lock() + health := &instance.Health + + // If crashes are outside the restart window, reset counters + if !health.FirstCrashTime.IsZero() && now.Sub(health.FirstCrashTime) > restartWindow { + slog.Info("Crash window expired, resetting counters", "instance", instance.ID) + health.CrashCount = 1 + health.FirstCrashTime = now + health.RestartCount = 0 + } + + // Check if we've exceeded the max restarts in the window + if health.CrashCount > maxRestartsInWindow { + d.Instances.Mu.Unlock() + slog.Error("Instance exceeded max restarts in window, leaving in error state", + "instance", instance.ID, + "crashes", health.CrashCount, + "window", restartWindow, + "max", maxRestartsInWindow) + return + } + + restartCount := health.RestartCount + d.Instances.Mu.Unlock() + + // Check resource availability + instanceType, ok := d.resourceMgr.instanceTypes[instance.InstanceType] + if !ok || instanceType == nil { + slog.Error("Unknown instance type, cannot restart", + "instance", instance.ID, "type", instance.InstanceType) + return + } + + if d.resourceMgr.canAllocate(instanceType, 1) < 1 { + slog.Error("Insufficient resources to restart instance", + "instance", instance.ID, "type", instance.InstanceType) + return + } + + // Calculate exponential backoff + delay := restartBackoffBase + for i := 0; i < restartCount; i++ { + delay *= 2 + if delay > restartBackoffMax { + delay = restartBackoffMax + break + } + } + + slog.Info("Scheduling instance restart", + "instance", instance.ID, + "delay", delay, + "restartCount", restartCount+1) + + time.AfterFunc(delay, func() { + d.restartCrashedInstance(instance) + }) +} + +// restartCrashedInstance re-verifies the instance is still in error state +// and relaunches it via LaunchInstance. +func (d *Daemon) restartCrashedInstance(instance *vm.VM) { + d.Instances.Mu.Lock() + if instance.Status != vm.StateError { + d.Instances.Mu.Unlock() + slog.Info("Instance no longer in error state, skipping restart", + "instance", instance.ID, "status", instance.Status) + return + } + + if d.shuttingDown.Load() { + d.Instances.Mu.Unlock() + slog.Info("Daemon shutting down, skipping restart", "instance", instance.ID) + return + } + + instance.Health.RestartCount++ + d.Instances.Mu.Unlock() + + slog.Info("Restarting crashed instance", + "instance", instance.ID, + "restartCount", instance.Health.RestartCount) + + // Transition Error → Pending (valid now that we added it to the transitions map) + if err := d.TransitionState(instance, vm.StatePending); err != nil { + slog.Error("Failed to transition instance to pending for restart", + "instance", instance.ID, "err", err) + return + } + + // LaunchInstance handles the full relaunch flow: + // check PID → MountVolumes → StartInstance → CreateQMPClient → NATS subscribe → Running + if err := d.LaunchInstance(instance); err != nil { + slog.Error("Failed to restart crashed instance", + "instance", instance.ID, "err", err) + if err := d.TransitionState(instance, vm.StateError); err != nil { + slog.Error("Failed to transition instance back to error after restart failure", + "instance", instance.ID, "err", err) + } + } +} diff --git a/hive/daemon/health_test.go b/hive/daemon/health_test.go new file mode 100644 index 000000000..8b57c2e02 --- /dev/null +++ b/hive/daemon/health_test.go @@ -0,0 +1,202 @@ +package daemon + +import ( + "fmt" + "os/exec" + "testing" + "time" + + "github.com/mulgadc/hive/hive/config" + "github.com/mulgadc/hive/hive/vm" + "github.com/nats-io/nats.go" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestClassifyCrashReason_CleanExit(t *testing.T) { + assert.Equal(t, "clean-exit", classifyCrashReason(nil)) +} + +func TestClassifyCrashReason_OOMKill(t *testing.T) { + // Start a process and kill it with SIGKILL to get a real ExitError + cmd := exec.Command("sleep", "60") + require.NoError(t, cmd.Start()) + + require.NoError(t, cmd.Process.Kill()) // sends SIGKILL + waitErr := cmd.Wait() + require.Error(t, waitErr) + + assert.Equal(t, "oom-killed", classifyCrashReason(waitErr)) +} + +func TestClassifyCrashReason_ExitCode(t *testing.T) { + // Run a command that exits with a non-zero code + cmd := exec.Command("sh", "-c", "exit 42") + waitErr := cmd.Run() + require.Error(t, waitErr) + + assert.Equal(t, "exit-42", classifyCrashReason(waitErr)) +} + +func TestClassifyCrashReason_Unknown(t *testing.T) { + // A non-ExitError error + err := fmt.Errorf("some random error") + assert.Equal(t, "unknown", classifyCrashReason(err)) +} + +func TestHandleInstanceCrash_SkipsNonRunning(t *testing.T) { + nc, err := nats.Connect(sharedNATSURL) + require.NoError(t, err) + defer nc.Close() + + d := &Daemon{ + natsConn: nc, + resourceMgr: NewResourceManager(), + Instances: vm.Instances{VMS: make(map[string]*vm.VM)}, + } + + instance := &vm.VM{ + ID: "i-test-stopped", + Status: vm.StateStopped, + } + d.Instances.VMS[instance.ID] = instance + + // Should return without action (no panic, no state change) + d.handleInstanceCrash(instance, fmt.Errorf("test error")) + assert.Equal(t, vm.StateStopped, instance.Status) +} + +func TestHandleInstanceCrash_SkipsShuttingDown(t *testing.T) { + nc, err := nats.Connect(sharedNATSURL) + require.NoError(t, err) + defer nc.Close() + + d := &Daemon{ + natsConn: nc, + resourceMgr: NewResourceManager(), + Instances: vm.Instances{VMS: make(map[string]*vm.VM)}, + } + d.shuttingDown.Store(true) + + instance := &vm.VM{ + ID: "i-test-shutdown", + Status: vm.StateRunning, + } + d.Instances.VMS[instance.ID] = instance + + d.handleInstanceCrash(instance, fmt.Errorf("test error")) + + // Status unchanged because shuttingDown is true + assert.Equal(t, vm.StateRunning, instance.Status) +} + +func TestMaybeRestart_ExceedsMaxInWindow(t *testing.T) { + nc, err := nats.Connect(sharedNATSURL) + require.NoError(t, err) + defer nc.Close() + + d := &Daemon{ + node: "test-node", + natsConn: nc, + config: &config.Config{ + Services: []string{"daemon"}, + }, + resourceMgr: NewResourceManager(), + Instances: vm.Instances{VMS: make(map[string]*vm.VM)}, + } + + // Find an allocatable instance type + var allocType string + for typeName := range d.resourceMgr.instanceTypes { + allocType = typeName + break + } + require.NotEmpty(t, allocType) + + instance := &vm.VM{ + ID: "i-test-maxcrash", + Status: vm.StateError, + InstanceType: allocType, + Health: vm.InstanceHealthState{ + CrashCount: maxRestartsInWindow + 1, // exceeds limit + FirstCrashTime: time.Now(), + RestartCount: maxRestartsInWindow, + }, + } + d.Instances.VMS[instance.ID] = instance + + // Should not schedule a restart (no panic, instance stays in error) + d.maybeRestartInstance(instance) + + // Instance should remain in error state + assert.Equal(t, vm.StateError, instance.Status) +} + +func TestMaybeRestart_ResetsAfterWindow(t *testing.T) { + nc, err := nats.Connect(sharedNATSURL) + require.NoError(t, err) + defer nc.Close() + + d := &Daemon{ + node: "test-node", + natsConn: nc, + config: &config.Config{ + Services: []string{"daemon"}, + }, + resourceMgr: NewResourceManager(), + Instances: vm.Instances{VMS: make(map[string]*vm.VM)}, + } + + // Find an allocatable instance type + var allocType string + for typeName := range d.resourceMgr.instanceTypes { + allocType = typeName + break + } + require.NotEmpty(t, allocType) + + instance := &vm.VM{ + ID: "i-test-windowreset", + Status: vm.StateError, + InstanceType: allocType, + Health: vm.InstanceHealthState{ + CrashCount: 5, + FirstCrashTime: time.Now().Add(-restartWindow - time.Minute), // outside window + RestartCount: 5, + }, + } + d.Instances.VMS[instance.ID] = instance + + // maybeRestartInstance should reset counters and schedule a restart + // (restart will fail because there's no real QEMU, but counters should reset) + d.maybeRestartInstance(instance) + + // Counters should be reset + assert.Equal(t, 1, instance.Health.CrashCount) + assert.Equal(t, 0, instance.Health.RestartCount) +} + +func TestRestartBackoff_Exponential(t *testing.T) { + // Verify the backoff calculation logic + expected := []time.Duration{ + 5 * time.Second, // restart 0 + 10 * time.Second, // restart 1 + 20 * time.Second, // restart 2 + 40 * time.Second, // restart 3 + 80 * time.Second, // restart 4 + 120 * time.Second, // restart 5 (capped at 2min) + 120 * time.Second, // restart 6 (capped at 2min) + } + + for i, want := range expected { + delay := restartBackoffBase + for j := 0; j < i; j++ { + delay *= 2 + if delay > restartBackoffMax { + delay = restartBackoffMax + break + } + } + assert.Equal(t, want, delay, "Backoff mismatch at restart count %d", i) + } +} diff --git a/hive/daemon/jetstream.go b/hive/daemon/jetstream.go index 658ab2baa..1915dc5d9 100644 --- a/hive/daemon/jetstream.go +++ b/hive/daemon/jetstream.go @@ -51,7 +51,7 @@ func (m *JetStreamManager) InitKVBucket() error { if err != nil { if errors.Is(err, nats.ErrBucketNotFound) { // Bucket doesn't exist, create it - slog.Info("Creating JetStream KV bucket", "bucket", InstanceStateBucket, "replicas", m.replicas) + slog.Debug("Creating JetStream KV bucket", "bucket", InstanceStateBucket, "replicas", m.replicas) kv, err = m.js.CreateKeyValue(&nats.KeyValueConfig{ Bucket: InstanceStateBucket, Description: "Hive instance state storage", @@ -65,7 +65,7 @@ func (m *JetStreamManager) InitKVBucket() error { return err } } else { - slog.Info("Connected to existing JetStream KV bucket", "bucket", InstanceStateBucket) + slog.Debug("Connected to existing JetStream KV bucket", "bucket", InstanceStateBucket) } m.kv = kv @@ -77,7 +77,7 @@ func (m *JetStreamManager) InitClusterStateBucket() error { kv, err := m.js.KeyValue(ClusterStateBucket) if err != nil { if errors.Is(err, nats.ErrBucketNotFound) { - slog.Info("Creating JetStream KV bucket", "bucket", ClusterStateBucket, "replicas", m.replicas) + slog.Debug("Creating JetStream KV bucket", "bucket", ClusterStateBucket, "replicas", m.replicas) kv, err = m.js.CreateKeyValue(&nats.KeyValueConfig{ Bucket: ClusterStateBucket, Description: "Hive cluster state (heartbeats, shutdown markers, service maps)", @@ -92,7 +92,7 @@ func (m *JetStreamManager) InitClusterStateBucket() error { return err } } else { - slog.Info("Connected to existing JetStream KV bucket", "bucket", ClusterStateBucket) + slog.Debug("Connected to existing JetStream KV bucket", "bucket", ClusterStateBucket) } m.clusterKV = kv diff --git a/hive/utils/nats.go b/hive/utils/nats.go index 8ed4ddb67..b098832cf 100644 --- a/hive/utils/nats.go +++ b/hive/utils/nats.go @@ -17,10 +17,10 @@ func ConnectNATS(host, token string) (*nats.Conn, error) { nats.ReconnectWait(time.Second), nats.MaxReconnects(-1), nats.DisconnectErrHandler(func(nc *nats.Conn, err error) { - slog.Warn("NATS disconnected", "err", err) + slog.Debug("NATS disconnected", "err", err) }), nats.ReconnectHandler(func(nc *nats.Conn) { - slog.Info("NATS reconnected", "url", nc.ConnectedUrl()) + slog.Debug("NATS reconnected", "url", nc.ConnectedUrl()) }), } @@ -33,7 +33,7 @@ func ConnectNATS(host, token string) (*nats.Conn, error) { return nil, fmt.Errorf("NATS connect failed: %w", err) } - slog.Info("Connected to NATS server", "host", host) + slog.Debug("Connected to NATS server", "host", host) return nc, nil } diff --git a/hive/utils/utils.go b/hive/utils/utils.go index 22b3e86e0..6fb82ea60 100644 --- a/hive/utils/utils.go +++ b/hive/utils/utils.go @@ -169,6 +169,17 @@ var AvailableImages = map[string]Images{ } +// SetOOMScore sets the OOM score adjustment for a process. +// Score range: -1000 (never kill) to 1000 (always kill first). +// Linux-only; returns an error on non-Linux systems. +func SetOOMScore(pid int, score int) error { + if runtime.GOOS != "linux" { + return fmt.Errorf("OOM score adjustment is only supported on Linux") + } + path := fmt.Sprintf("/proc/%d/oom_score_adj", pid) + return os.WriteFile(path, []byte(strconv.Itoa(score)), 0644) +} + func ReadPidFile(name string) (int, error) { pidPath := pidPath() diff --git a/hive/utils/utils_test.go b/hive/utils/utils_test.go index 4a0a5408b..a0e735ab8 100644 --- a/hive/utils/utils_test.go +++ b/hive/utils/utils_test.go @@ -4,6 +4,7 @@ import ( "fmt" "os" "os/exec" + "runtime" "strings" "syscall" "testing" @@ -826,3 +827,40 @@ func TestGenerateSocketFile_EmptyName(t *testing.T) { _, err := GenerateSocketFile("") assert.Error(t, err) } + +func TestSetOOMScore(t *testing.T) { + if runtime.GOOS != "linux" { + t.Skip("OOM score adjustment only supported on Linux") + } + + // Read current value so we can set something higher (unprivileged processes + // can only increase their OOM score, not decrease it) + pid := os.Getpid() + current, err := os.ReadFile(fmt.Sprintf("/proc/%d/oom_score_adj", pid)) + if err != nil { + t.Skipf("Cannot read OOM score: %v", err) + } + + // Set a positive score (always allowed for unprivileged processes) + err = SetOOMScore(pid, 100) + if err != nil { + t.Skipf("Insufficient permissions to set OOM score: %v", err) + } + + // Read back and verify + data, err := os.ReadFile(fmt.Sprintf("/proc/%d/oom_score_adj", pid)) + assert.NoError(t, err) + assert.Equal(t, "100", strings.TrimSpace(string(data))) + + // Best-effort restore (may fail without privileges if original was lower) + _ = os.WriteFile(fmt.Sprintf("/proc/%d/oom_score_adj", pid), current, 0644) +} + +func TestSetOOMScore_InvalidPID(t *testing.T) { + if runtime.GOOS != "linux" { + t.Skip("OOM score adjustment only supported on Linux") + } + + err := SetOOMScore(999999999, 100) + assert.Error(t, err) +} diff --git a/hive/vm/state.go b/hive/vm/state.go index 4de290db9..d3a18b6e7 100644 --- a/hive/vm/state.go +++ b/hive/vm/state.go @@ -45,7 +45,7 @@ var ValidTransitions = map[InstanceState][]InstanceState{ StateStopping: {StateStopped, StateShuttingDown, StateError}, StateStopped: {StateRunning, StateShuttingDown, StateError}, StateShuttingDown: {StateTerminated, StateError}, - StateError: {StateRunning, StateShuttingDown}, + StateError: {StatePending, StateRunning, StateShuttingDown}, } // IsValidTransition checks whether moving from current to target is allowed. diff --git a/hive/vm/vm.go b/hive/vm/vm.go index 82d6b270d..622713e24 100644 --- a/hive/vm/vm.go +++ b/hive/vm/vm.go @@ -8,12 +8,22 @@ import ( "strconv" "strings" "sync" + "time" "github.com/aws/aws-sdk-go/service/ec2" "github.com/mulgadc/hive/hive/config" "github.com/mulgadc/hive/hive/qmp" ) +// InstanceHealthState tracks crash detection and auto-restart metadata for a VM. +type InstanceHealthState struct { + CrashCount int `json:"crash_count"` + LastCrashTime time.Time `json:"last_crash_time,omitempty"` + LastCrashReason string `json:"last_crash_reason,omitempty"` + RestartCount int `json:"restart_count"` + FirstCrashTime time.Time `json:"first_crash_time,omitempty"` +} + type VM struct { ID string `json:"id"` PID int `json:"pid"` @@ -47,6 +57,9 @@ type VM struct { // Metadata server address (e.g., "127.0.0.1:12345") for EC2 metadata service MetadataServerAddress string `json:"metadata_server_address,omitempty"` + + // Health tracks crash detection and auto-restart state + Health InstanceHealthState `json:"health"` } // ResetNodeLocalState zeroes out fields that are specific to the daemon node diff --git a/scripts/iac/README.md b/scripts/iac/README.md index 277aeefd2..cfd1d25de 100644 --- a/scripts/iac/README.md +++ b/scripts/iac/README.md @@ -6,3 +6,4 @@ Collection of OpenTofu/Terraform templates for creating Hive instances. |---|---| | [proxmox/](proxmox/) | Provisions a 3-node Hive cluster on Proxmox VE bare-metal hypervisors | | [aws/](aws/) | Provisions infrastructure using the Terraform AWS provider, with MulgaOS AWS-compatible API as the local endpoint | +| [proxmox-template/](proxmox-template/) | Builds a Proxmox VM template with all Hive dependencies pre-installed (QEMU, Go, repos, modules) | diff --git a/scripts/iac/proxmox-template/.env.example b/scripts/iac/proxmox-template/.env.example new file mode 100644 index 000000000..0f542cc31 --- /dev/null +++ b/scripts/iac/proxmox-template/.env.example @@ -0,0 +1,60 @@ +# Proxmox Template Builder — Environment Configuration +# +# Copy to .env and fill in your values: +# cp .env.example .env +# +# Then source it before running tofu: +# source .env && tofu plan + +# --- Proxmox API --- + +# API token for Proxmox VE (read by the provider directly) +export PROXMOX_VE_API_TOKEN="terraform@pve!provider=YOUR_TOKEN_HERE" + +# Proxmox VE API endpoint URL +export TF_VAR_proxmox_endpoint="https://pve-host.example.com:8006/" + +# --- Proxmox Node --- +# Build the template on ONE node. After creation, use vzdump to copy to others. +# +# name - Proxmox node name (as shown in the Proxmox UI) +# address - SSH-reachable hostname or IP of the Proxmox host + +export TF_VAR_proxmox_node_name="node1" +export TF_VAR_proxmox_node_address="node1.example.com" + +# --- SSH Access --- + +# SSH username on Proxmox hosts (used by provider for file uploads + qm commands) +export TF_VAR_proxmox_ssh_username="terraform" + +# Path to SSH private key for Proxmox host access +export TF_VAR_ssh_private_key_path="~/.ssh/proxmox-host-tf" + +# Path to SSH private key for VM access (must be the pair of the public key below) +export TF_VAR_vm_ssh_private_key_path="~/.ssh/cloud-init" + +# Path to SSH public key injected into VMs via cloud-init (for tf-user) +export TF_VAR_ssh_public_key_path="~/.ssh/cloud-init.pub" + +# --- Template --- + +# VM ID for the template (pick a high number to avoid conflicts) +export TF_VAR_template_vmid=9000 + +# Short tag for the base image (used in Proxmox template tags) +export TF_VAR_base_image_tag="debian-12" + +# Storage and network for the build node +export TF_VAR_datastore_id="local-lvm" +export TF_VAR_bridge="vmbr0" + +# --- Optional --- + +# Base image — change this when testing a newer Debian/Ubuntu version: +# export TF_VAR_base_image="local:iso/debian-13-genericcloud-amd64.img" +# export TF_VAR_base_image_tag="debian-13" +# export TF_VAR_template_vmid=9010 + +# Clone a specific branch instead of main: +# export TF_VAR_git_branch="feature/my-branch" diff --git a/scripts/iac/proxmox-template/.gitignore b/scripts/iac/proxmox-template/.gitignore new file mode 100644 index 000000000..537e90ed3 --- /dev/null +++ b/scripts/iac/proxmox-template/.gitignore @@ -0,0 +1,6 @@ +.env +.terraform/ +.terraform.lock.hcl +*.tfstate +*.tfstate.backup +*.tfplan diff --git a/scripts/iac/proxmox-template/README.md b/scripts/iac/proxmox-template/README.md new file mode 100644 index 000000000..09647b216 --- /dev/null +++ b/scripts/iac/proxmox-template/README.md @@ -0,0 +1,124 @@ +# Hive Node Template Builder + +Builds a Proxmox VM template with all Hive dependencies pre-installed. Cloning from this template reduces cluster spin-up from ~15 minutes to ~1-2 minutes (`git pull` + `make build`). + +## What's in the template + +| Component | Details | +|---|---| +| System packages | QEMU, nbdkit, libvirt, gcc, jq, curl, etc. (`make install-system`) | +| Go | 1.26.0 (`make install-go`) | +| AWS CLI | v2 (`make install-aws`) | +| Repositories | hive, viperblock, predastore (with `go.work` configured) | +| Module cache | `go mod download` completed for all repos | +| Cloud images | Ubuntu 24.04 for nested VMs (`~/images/ubuntu-24.04.img`) | +| Tuning | sysctl (rmem_max/wmem_max), kvm group, ufw disabled | +| Generalized | Clean machine-id, SSH host keys, cloud-init — each clone boots fresh | + +## Quick Start + +```bash +cd scripts/iac/proxmox-template + +# 1. Configure +cp .env.example .env +# Edit .env — set your Proxmox endpoint, API token, node, and SSH key paths. + +# 2. Set API token (not stored in .env for security) +export PROXMOX_VE_API_TOKEN="terraform@pve!provider=YOUR_TOKEN" + +# 3. Build +source .env +tofu init +tofu plan +tofu apply +``` + +Build takes ~10-15 minutes (apt packages, Go, cloud image download). When done, you'll have a template on the build node at the configured VMID (default 9000). + +## Distribute to Other Nodes + +After `tofu apply`, the output prints ready-to-run commands. Example for a 3-node cluster: + +```bash +# Export from build node +ssh -i ~/.ssh/proxmox-host-tf terraform@ \ + 'vzdump 9000 --dumpdir /tmp --mode stop --compress zstd' + +# Copy to other nodes +scp -3 -i ~/.ssh/proxmox-host-tf \ + terraform@:/tmp/vzdump-qemu-9000-*.vma.zst \ + terraform@:/tmp/ + +# Restore on each target node (use a unique VMID and the node's datastore) +ssh -i ~/.ssh/proxmox-host-tf terraform@ \ + 'qmrestore /tmp/vzdump-qemu-9000-*.vma.zst --storage && qm template ' +``` + +After distributing, each node has its own copy of the template with a unique VMID. + +## Using the Template + +Update `scripts/iac/proxmox/vms.tf` to clone from the template instead of booting from a raw cloud image. Or clone manually from the Proxmox UI. + +On each cloned VM after boot: + +```bash +# Pull latest code +cd ~/Development/mulga/hive && git pull +cd ~/Development/mulga/viperblock && git pull +cd ~/Development/mulga/predastore && git pull + +# Build (~30s with cached modules) +cd ~/Development/mulga/hive && export PATH=/usr/local/go/bin:$PATH && make build +``` + +## Testing a Different Base Image + +Upload the new image to Proxmox, then build a separate template: + +```bash +# Debian 13 example +export TF_VAR_base_image="local:iso/debian-13-genericcloud-amd64.img" +export TF_VAR_base_image_tag="debian-13" +export TF_VAR_template_vmid=9010 # Keep the debian-12 template at 9000 + +source .env +tofu apply +``` + +Templates are tagged in the Proxmox UI with the base image name (`debian-12`, `debian-13`, etc.) so you can tell them apart. + +## Rebuilding + +When dependencies change (new Go version, new system packages): + +```bash +source .env +tofu destroy # Remove the old template +tofu apply # Build fresh +# Then re-distribute with vzdump +``` + +## SSH Key Setup + +This template uses **two separate key pairs** (matching the `scripts/iac/proxmox/` setup): + +| Key | Used for | Variable | +|---|---|---| +| Proxmox host key | SSH to Proxmox hosts (provider, `qm template` command) | `ssh_private_key_path` | +| VM / cloud-init key | SSH to VMs (cloud-init injection, provisioning) | `vm_ssh_private_key_path` + `ssh_public_key_path` | + +If you use the same key for both, set all three variables to the same key pair. + +## Files + +| File | Description | +|------|-------------| +| `main.tf` | Proxmox provider configuration | +| `variables.tf` | All configurable variables with defaults | +| `template-builder.tf` | VM creation, provisioning, template conversion | +| `outputs.tf` | Template info + vzdump/restore commands | +| `scripts/provision.sh` | Install deps, clone repos, cache modules, generalize | +| `.env.example` | Environment variable template | +| `.gitignore` | Ignores .env, state files, provider plugins | diff --git a/scripts/iac/proxmox-template/main.tf b/scripts/iac/proxmox-template/main.tf new file mode 100644 index 000000000..461399faa --- /dev/null +++ b/scripts/iac/proxmox-template/main.tf @@ -0,0 +1,27 @@ +terraform { + required_version = ">= 1.6.0" + + required_providers { + proxmox = { + source = "bpg/proxmox" + version = "0.52.0" + } + } +} + +# API token is read from PROXMOX_VE_API_TOKEN env var automatically +provider "proxmox" { + endpoint = var.proxmox_endpoint + insecure = true + + ssh { + agent = false + username = var.proxmox_ssh_username + private_key = file(pathexpand(var.ssh_private_key_path)) + + node { + name = var.proxmox_node_name + address = var.proxmox_node_address + } + } +} diff --git a/scripts/iac/proxmox-template/outputs.tf b/scripts/iac/proxmox-template/outputs.tf new file mode 100644 index 000000000..25d9fc1a5 --- /dev/null +++ b/scripts/iac/proxmox-template/outputs.tf @@ -0,0 +1,43 @@ +output "template_vmid" { + description = "VM ID of the created template" + value = proxmox_virtual_environment_vm.builder.vm_id +} + +output "template_node" { + description = "Proxmox node where the template lives" + value = var.proxmox_node_name +} + +output "base_image_tag" { + description = "Base image tag for this template" + value = var.base_image_tag +} + +# Ready-to-run command: export the template for distribution to other Proxmox nodes +output "vzdump_export_command" { + description = "Run this to export the template (vzdump) for copying to other nodes" + value = <<-EOT + ssh -i ${var.ssh_private_key_path} ${var.proxmox_ssh_username}@${var.proxmox_node_address} \ + 'vzdump ${var.template_vmid} --dumpdir /tmp --mode stop --compress zstd' + EOT +} + +# After exporting, copy to each target node and restore +output "distribute_commands" { + description = "Copy and restore the template on other Proxmox nodes" + value = <<-EOT + # 1. Export from build node (${var.proxmox_node_name}): + ssh -i ${var.ssh_private_key_path} ${var.proxmox_ssh_username}@${var.proxmox_node_address} \ + 'vzdump ${var.template_vmid} --dumpdir /tmp --mode stop --compress zstd' + + # 2. Copy to each target node: + scp -3 \ + -i ${var.ssh_private_key_path} \ + ${var.proxmox_ssh_username}@${var.proxmox_node_address}:/tmp/vzdump-qemu-${var.template_vmid}-*.vma.zst \ + ${var.proxmox_ssh_username}@:/tmp/ + + # 3. On each target node — restore and convert to template: + ssh -i ${var.ssh_private_key_path} ${var.proxmox_ssh_username}@ \ + 'qmrestore /tmp/vzdump-qemu-${var.template_vmid}-*.vma.zst --storage && qm template ' + EOT +} diff --git a/scripts/iac/proxmox-template/scripts/provision.sh b/scripts/iac/proxmox-template/scripts/provision.sh new file mode 100755 index 000000000..52a967e2d --- /dev/null +++ b/scripts/iac/proxmox-template/scripts/provision.sh @@ -0,0 +1,176 @@ +#!/bin/bash +# provision.sh — Installs all Hive dependencies, clones repos, caches Go modules, +# and generalizes the VM for use as a Proxmox template. +# +# Runs as tf-user via Terraform remote-exec. Uses sudo for system operations. +# Environment: GIT_BRANCH (default: main) + +set -euo pipefail + +GIT_BRANCH="${GIT_BRANCH:-main}" +WORK_DIR="$HOME/Development/mulga" + +echo "========================================" +echo " Hive Template Provisioning" +echo " Branch: ${GIT_BRANCH}" +echo " Arch: $(uname -m)" +echo "========================================" + +# ── 1. Wait for cloud-init ────────────────────────────────────────────────── +echo "" +echo "=== [1/8] Waiting for cloud-init to complete ===" +cloud-init status --wait + +# ── 2. Clone repositories ─────────────────────────────────────────────────── +echo "" +echo "=== [2/8] Cloning repositories ===" +mkdir -p "$WORK_DIR" +cd "$WORK_DIR" + +for repo in hive viperblock predastore; do + if [ ! -d "$repo" ]; then + echo " Cloning $repo (branch: $GIT_BRANCH)..." + git clone -b "$GIT_BRANCH" "https://github.com/mulgadc/${repo}.git" + else + echo " $repo already exists, skipping" + fi +done + +# ── 3. Install system dependencies via Makefile ───────────────────────────── +echo "" +echo "=== [3/8] Installing system dependencies ===" +cd "$WORK_DIR/hive" +sudo make quickinstall + +# Ensure Go is in PATH for the rest of the script +export PATH="/usr/local/go/bin:$HOME/go/bin:$PATH" +echo " Go version: $(go version)" +echo " AWS CLI: $(aws --version)" + +# ── 4. Set up go.work ─────────────────────────────────────────────────────── +echo "" +echo "=== [4/8] Creating go.work ===" + +# Detect Go version from go.mod +GO_VERSION=$(grep '^go ' "$WORK_DIR/hive/go.mod" | awk '{print $2}') +echo " Using Go version from go.mod: ${GO_VERSION}" + +cat > "$WORK_DIR/hive/go.work" << GOWORK +go ${GO_VERSION} + +use ( + . + ../viperblock + ../viperblock/nbd + ../viperblock/nbd/libguestfs.org/nbdkit + ../predastore +) +GOWORK + +echo " Created $WORK_DIR/hive/go.work" + +# ── 5. Download Go modules ────────────────────────────────────────────────── +echo "" +echo "=== [5/8] Downloading Go modules ===" +cd "$WORK_DIR/hive" && go mod download && echo " hive: done" +cd "$WORK_DIR/viperblock" && go mod download && echo " viperblock: done" +cd "$WORK_DIR/predastore" && go mod download && echo " predastore: done" + +# Sub-modules +if [ -f "$WORK_DIR/viperblock/nbd/go.mod" ]; then + cd "$WORK_DIR/viperblock/nbd" && go mod download && echo " viperblock/nbd: done" +fi +if [ -f "$WORK_DIR/viperblock/nbd/libguestfs.org/nbdkit/go.mod" ]; then + cd "$WORK_DIR/viperblock/nbd/libguestfs.org/nbdkit" && go mod download && echo " viperblock/nbd/nbdkit: done" +fi + +# ── 6. Pre-download cloud images for nested VMs ───────────────────────────── +echo "" +echo "=== [6/8] Pre-downloading cloud images ===" +mkdir -p "$HOME/images" + +ARCH=$(dpkg --print-architecture) +IMAGE_URL="" +if [ "$ARCH" = "amd64" ]; then + IMAGE_URL="https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img" +elif [ "$ARCH" = "arm64" ]; then + IMAGE_URL="https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-arm64.img" +fi + +if [ -n "$IMAGE_URL" ]; then + echo " Downloading Ubuntu 24.04 cloud image ($ARCH)..." + curl -L -o "$HOME/images/ubuntu-24.04.img" "$IMAGE_URL" + echo " Saved to ~/images/ubuntu-24.04.img" +else + echo " Unknown arch ($ARCH), skipping cloud image download" +fi + +# ── 7. System tuning ──────────────────────────────────────────────────────── +echo "" +echo "=== [7/8] Applying system tuning ===" + +# Persistent sysctl for NATS / high-throughput networking +sudo tee /etc/sysctl.d/99-hive.conf > /dev/null << 'SYSCTL' +net.core.rmem_max=4194304 +net.core.wmem_max=4194304 +SYSCTL +echo " sysctl: net.core.rmem_max/wmem_max = 4MB" + +# Add user to kvm group for QEMU access +sudo usermod -aG kvm "$(whoami)" +echo " Added $(whoami) to kvm group" + +# Disable ufw if present +if command -v ufw >/dev/null 2>&1; then + sudo ufw disable 2>/dev/null || true + echo " ufw disabled" +fi + +# ── 8. Generalize for templating ──────────────────────────────────────────── +echo "" +echo "=== [8/8] Cleaning up for template ===" + +# Apt caches +sudo apt-get clean +sudo rm -rf /var/lib/apt/lists/* +echo " Cleared apt cache" + +# Logs +sudo find /var/log -type f -name "*.log" -exec truncate -s 0 {} \; 2>/dev/null || true +sudo rm -rf /var/log/journal/* +echo " Truncated logs" + +# Temp files +sudo rm -rf /tmp/* /var/tmp/* + +# Machine ID — systemd regenerates on next boot. +# This is critical: without this, all clones get the same DHCP client ID → IP conflicts. +sudo truncate -s 0 /etc/machine-id +sudo rm -f /var/lib/dbus/machine-id +echo " Cleared machine-id (will regenerate on boot)" + +# SSH host keys — regenerated on boot by cloud-init or systemd +sudo rm -f /etc/ssh/ssh_host_* +echo " Removed SSH host keys (will regenerate on boot)" + +# Cloud-init state — forces cloud-init to run again on next boot. +# This ensures each clone gets its own hostname, SSH keys, user config, etc. +sudo cloud-init clean +echo " Cleaned cloud-init state (will re-run on boot)" + +# Shell history +history -c 2>/dev/null || true +cat /dev/null > "$HOME/.bash_history" 2>/dev/null || true + +# Provisioning script itself +rm -f /tmp/provision.sh + +# Sync filesystem +sync + +echo "" +echo "========================================" +echo " Template provisioning complete!" +echo " VM is ready to be shut down and" +echo " converted to a Proxmox template." +echo "========================================" diff --git a/scripts/iac/proxmox-template/template-builder.tf b/scripts/iac/proxmox-template/template-builder.tf new file mode 100644 index 000000000..294ff124b --- /dev/null +++ b/scripts/iac/proxmox-template/template-builder.tf @@ -0,0 +1,130 @@ +data "local_file" "ssh_public_key" { + filename = pathexpand(var.ssh_public_key_path) +} + +# Minimal cloud-init: just enough to bootstrap SSH + guest agent. +# Heavy provisioning is done via remote-exec after boot. +resource "proxmox_virtual_environment_file" "cloud_config" { + content_type = "snippets" + datastore_id = "local" + node_name = var.proxmox_node_name + + source_raw { + # NOTE: #cloud-config MUST start at column 0 — cloud-init uses it as a magic header. + data = < /tmp/vendor-cloud-init-done + EOF + + cloud_config_image = <<-EOF + #cloud-config + package_update: true + package_upgrade: true + packages: + - qemu-guest-agent + - net-tools + - git + - make + runcmd: + - timedatectl set-timezone America/Los_Angeles + - systemctl enable qemu-guest-agent + - systemctl start qemu-guest-agent + - ufw disable + - sysctl -w net.core.rmem_max=4194304 + - sysctl -w net.core.wmem_max=4194304 + - usermod -aG kvm tf-user + - echo "done" > /tmp/vendor-cloud-init-done + EOF +} + resource "proxmox_virtual_environment_file" "cloud_config" { count = var.node_count @@ -10,26 +42,7 @@ resource "proxmox_virtual_environment_file" "cloud_config" { node_name = var.nodes[count.index % length(var.nodes)].name source_raw { - data = < /tmp/vendor-cloud-init-done -EOF - + data = var.nodes[count.index % length(var.nodes)].template_vmid != null ? local.cloud_config_template : local.cloud_config_image file_name = "${var.cluster_name}-${count.index + 1}-cloud-config.yaml" } } diff --git a/scripts/iac/proxmox/variables.tf b/scripts/iac/proxmox/variables.tf index a1d0027ee..cae8bdfe2 100644 --- a/scripts/iac/proxmox/variables.tf +++ b/scripts/iac/proxmox/variables.tf @@ -36,12 +36,13 @@ variable "ssh_public_key_path" { variable "nodes" { type = list(object({ - name = string - address = string - bridge = string - datastore_id = string + name = string + address = string + bridge = string + datastore_id = string + template_vmid = optional(number) })) - description = "Proxmox hosts for VM placement (VMs are distributed round-robin)" + description = "Proxmox hosts for VM placement (VMs are distributed round-robin). Set template_vmid to clone from a pre-built template instead of booting from os_image." validation { condition = length(var.nodes) > 0 @@ -100,6 +101,6 @@ variable "disk_size_gb" { variable "os_image" { type = string - description = "Proxmox image file ID for VM boot disk" + description = "Proxmox image file ID for VM boot disk (used when node has no template_vmid)" default = "local:iso/debian-12-genericcloud-amd64-20240211-1654.img" } diff --git a/scripts/iac/proxmox/vms.tf b/scripts/iac/proxmox/vms.tf index 708be2d7d..dc830c959 100644 --- a/scripts/iac/proxmox/vms.tf +++ b/scripts/iac/proxmox/vms.tf @@ -51,12 +51,24 @@ resource "proxmox_virtual_environment_vm" "hive_node" { bridge = var.nodes[count.index % length(var.nodes)].bridge } - disk { - file_id = var.os_image - datastore_id = var.nodes[count.index % length(var.nodes)].datastore_id - interface = "virtio0" - iothread = true - discard = "on" - size = var.disk_size_gb + # Clone from template if the target node has template_vmid set, otherwise boot from cloud image. + dynamic "clone" { + for_each = var.nodes[count.index % length(var.nodes)].template_vmid != null ? [1] : [] + content { + vm_id = var.nodes[count.index % length(var.nodes)].template_vmid + full = true + } + } + + dynamic "disk" { + for_each = var.nodes[count.index % length(var.nodes)].template_vmid == null ? [1] : [] + content { + file_id = var.os_image + datastore_id = var.nodes[count.index % length(var.nodes)].datastore_id + interface = "virtio0" + iothread = true + discard = "on" + size = var.disk_size_gb + } } } diff --git a/scripts/start-dev.sh b/scripts/start-dev.sh index 6bfbd902f..72f15db63 100755 --- a/scripts/start-dev.sh +++ b/scripts/start-dev.sh @@ -126,6 +126,22 @@ start_service_foreground() { $command } +# Function to set OOM score for a service (Linux only) +# Protects infrastructure services from OOM killer (-500 = less likely to be killed) +set_oom_score() { + local name="$1" + local score="$2" + local pid_file="$LOGS_DIR/${name}.pid" + if [ -f "$pid_file" ]; then + local pid=$(cat "$pid_file") + if [ -d "/proc/$pid" ]; then + echo "$score" > "/proc/$pid/oom_score_adj" 2>/dev/null && \ + echo " OOM score for $name (PID $pid): $score" || \ + echo " Warning: Could not set OOM score for $name" + fi + fi +} + # Function to check if service is responsive check_service() { local name="$1" @@ -188,6 +204,7 @@ if has_service "nats"; then NATS_CMD="./bin/hive service nats start" start_service "nats" "$NATS_CMD" + set_oom_score "nats" "-500" else echo "1️⃣ Skipping NATS (not a local service)" fi @@ -231,6 +248,7 @@ if has_service "predastore"; then PREDASTORE_CMD="./bin/hive service predastore start" start_service "predastore" "$PREDASTORE_CMD" + set_oom_score "predastore" "-500" check_service "Predastore" "$HIVE_PREDASTORE_HOST" "$HIVE_PREDASTORE_PORT" else echo "2️⃣ Skipping Predastore (not a local service)" @@ -266,6 +284,7 @@ if has_service "viperblock"; then VIPERBLOCK_CMD="./bin/hive service viperblock start" start_service "viperblock" "$VIPERBLOCK_CMD" + set_oom_score "viperblock" "-500" else echo "3️⃣ Skipping Viperblock (not a local service)" fi @@ -281,6 +300,7 @@ if has_service "daemon"; then HIVE_CMD="./bin/hive service hive start" start_service "hive" "$HIVE_CMD" + set_oom_score "hive" "-500" else echo "4️⃣. Skipping Hive Gateway (not a local service)" fi @@ -300,6 +320,7 @@ if has_service "awsgw"; then AWSGW_CMD="./bin/hive service awsgw start" start_service "awsgw" "$AWSGW_CMD" + set_oom_score "awsgw" "-500" else echo "5️⃣. Skipping AWS Gateway (not a local service)" fi @@ -312,6 +333,7 @@ if [ "${UI}" != "false" ] && has_service "ui"; then HIVEUI_CMD="./bin/hive service hive-ui start" start_service "hive-ui" "$HIVEUI_CMD" + set_oom_score "hive-ui" "-500" else echo "" echo "6️⃣. Skipping Hive UI" diff --git a/scripts/workload-performance.sh b/scripts/workload-performance.sh new file mode 100644 index 000000000..2ca77bbd7 --- /dev/null +++ b/scripts/workload-performance.sh @@ -0,0 +1,24 @@ +#!/bin/sh + +# Workload performance testing (memory, disk, cpu, network) + +sudo apt update +sudo apt install -y make gcc git + +wget https://go.dev/dl/go1.26.0.linux-amd64.tar.gz +sudo tar -C /usr/local -xzf go1.26.0.linux-amd64.tar.gz + +echo "export PATH=$PATH:/usr/local/go/bin" >> $HOME/.bashrc +source $HOME/.bashrc + +git clone https://github.com/dgraph-io/badger.git +cd badger + +# disable jemalloc +sed -i 's/-tags=jemalloc/-tags=nojemalloc/g' test.sh + +# run 4 tests, otherwise OOM on <8GB RAM +sed -i 's/-parallel 16/-parallel 4/g; s/-parallel=16/-parallel=4/g' test.sh + +# run tests +./test.sh diff --git a/tests/e2e/lib/multinode-helpers.sh b/tests/e2e/lib/multinode-helpers.sh index a5e8f0d68..97d59e313 100755 --- a/tests/e2e/lib/multinode-helpers.sh +++ b/tests/e2e/lib/multinode-helpers.sh @@ -495,6 +495,68 @@ dump_all_node_logs() { echo "==========================================" } +# Get the QEMU process PID for an instance +# Usage: get_qemu_pid +# Returns: the PID, or empty string if not found +get_qemu_pid() { + local instance_id="$1" + + local pid + pid=$(ps auxw | grep "$instance_id" | grep qemu-system | grep -v grep | awk '{print $2}' | head -1) + + if [ -n "$pid" ]; then + echo "$pid" + return 0 + fi + + return 1 +} + +# Wait for an instance to recover from a crash (error → running) +# Usage: wait_for_instance_recovery [max_attempts] +# Expects state transition: error → pending → running +wait_for_instance_recovery() { + local instance_id="$1" + local max_attempts="${2:-60}" + local attempt=0 + local saw_error=false + + echo " Waiting for instance $instance_id to recover..." + + while [ $attempt -lt $max_attempts ]; do + local state + state=$(aws --endpoint-url https://${NODE1_IP}:${AWSGW_PORT} ec2 describe-instances \ + --instance-ids "$instance_id" \ + --query 'Reservations[0].Instances[0].State.Name' \ + --output text 2>/dev/null) || { + sleep 2 + attempt=$((attempt + 1)) + continue + } + + if [ "$state" == "error" ] || [ "$state" == "pending" ]; then + saw_error=true + echo " State: $state (attempt $((attempt + 1))/$max_attempts)" + fi + + if [ "$state" == "running" ] && [ "$saw_error" = true ]; then + echo " Instance recovered to running state" + return 0 + fi + + if [ "$state" == "running" ] && [ "$saw_error" = false ]; then + # Still in original running state, crash not detected yet + : + fi + + sleep 2 + attempt=$((attempt + 1)) + done + + echo " ERROR: Instance did not recover within $max_attempts attempts" + return 1 +} + # Global variable for init PID tracking (used by multi-node formation) LEADER_INIT_PID="" diff --git a/tests/e2e/run-multinode-e2e.sh b/tests/e2e/run-multinode-e2e.sh index 8cc32f632..a7037a3dd 100755 --- a/tests/e2e/run-multinode-e2e.sh +++ b/tests/e2e/run-multinode-e2e.sh @@ -958,6 +958,176 @@ verify_nats_cluster 3 || { echo "WARNING: NATS cluster verification failed after operations" } +# Test 5: VM Crash Recovery (kill -9 → detect → auto-restart) +echo "" +echo "Test 5: VM Crash Recovery" +echo "----------------------------------------" +echo "Testing QEMU crash detection and auto-restart..." + +# Use the second instance for crash testing (first was used for stop/start in Test 3) +CRASH_INSTANCE="${INSTANCE_IDS[1]}" +echo " Crash test instance: $CRASH_INSTANCE" + +# Verify instance is running before crash +CRASH_STATE=$($AWS_EC2 describe-instances --instance-ids "$CRASH_INSTANCE" \ + --query 'Reservations[0].Instances[0].State.Name' --output text) +if [ "$CRASH_STATE" != "running" ]; then + echo " ERROR: Instance not in running state before crash test (state: $CRASH_STATE)" + exit 1 +fi +echo " Instance is running" + +# Get the QEMU PID +QEMU_PID=$(get_qemu_pid "$CRASH_INSTANCE") +if [ -z "$QEMU_PID" ]; then + echo " ERROR: Could not find QEMU PID for $CRASH_INSTANCE" + exit 1 +fi +echo " QEMU PID: $QEMU_PID" + +# Kill QEMU with SIGKILL (simulates OOM kill) +echo " Killing QEMU process with SIGKILL (simulating OOM kill)..." +kill -9 "$QEMU_PID" + +# Brief pause for the daemon to detect the crash +sleep 3 + +# Verify the daemon detected the crash (state should be error or already recovering) +echo " Checking post-crash state..." +POST_CRASH_STATE=$($AWS_EC2 describe-instances --instance-ids "$CRASH_INSTANCE" \ + --query 'Reservations[0].Instances[0].State.Name' --output text) +echo " Post-crash state: $POST_CRASH_STATE" + +if [ "$POST_CRASH_STATE" == "running" ]; then + # Might have recovered very quickly, check if it's a new PID + NEW_PID=$(get_qemu_pid "$CRASH_INSTANCE" || echo "") + if [ "$NEW_PID" == "$QEMU_PID" ]; then + echo " ERROR: Instance still has same PID after kill -9, crash not detected" + exit 1 + fi + echo " Instance already recovered with new PID: $NEW_PID (was: $QEMU_PID)" +else + # Wait for auto-restart (backoff starts at 5s) + echo " Instance in $POST_CRASH_STATE state, waiting for auto-restart..." + wait_for_instance_recovery "$CRASH_INSTANCE" 60 || { + echo " ERROR: Instance failed to recover from crash" + # Dump daemon logs for debugging + for i in 1 2 3; do + if [ -f "$HOME/node$i/logs/hive.log" ]; then + echo "" + echo " --- node$i daemon log (last 30 lines) ---" + tail -30 "$HOME/node$i/logs/hive.log" + fi + done + exit 1 + } +fi + +# Verify the instance is running with a new QEMU process +RECOVERED_PID=$(get_qemu_pid "$CRASH_INSTANCE") +if [ -z "$RECOVERED_PID" ]; then + echo " ERROR: No QEMU process found after recovery" + exit 1 +fi + +if [ "$RECOVERED_PID" == "$QEMU_PID" ]; then + echo " ERROR: QEMU PID unchanged after crash recovery (expected new process)" + exit 1 +fi +echo " New QEMU PID: $RECOVERED_PID (was: $QEMU_PID)" + +# Verify describe-instances shows running state +FINAL_STATE=$($AWS_EC2 describe-instances --instance-ids "$CRASH_INSTANCE" \ + --query 'Reservations[0].Instances[0].State.Name' --output text) +if [ "$FINAL_STATE" != "running" ]; then + echo " ERROR: Instance not running after recovery (state: $FINAL_STATE)" + exit 1 +fi +echo " Instance is running after crash recovery" + +# Verify SSH works after recovery (VM rebooted with fresh OS from root volume) +echo " Verifying SSH after crash recovery..." +CRASH_SSH_PORT=$(get_ssh_port "$CRASH_INSTANCE" 30) +CRASH_SSH_HOST=$(get_ssh_host "$CRASH_INSTANCE") +if [ -n "$CRASH_SSH_PORT" ]; then + echo " SSH endpoint after recovery: $CRASH_SSH_HOST:$CRASH_SSH_PORT" + # Update SSH details for later termination verification + SSH_PORTS[1]="$CRASH_SSH_PORT" + SSH_HOSTS[1]="$CRASH_SSH_HOST" + wait_for_ssh "$CRASH_SSH_HOST" "$CRASH_SSH_PORT" "multinode-test-key.pem" || { + echo " WARNING: SSH not ready after crash recovery (non-fatal, VM may still be booting)" + } +else + echo " WARNING: Could not determine SSH port after recovery (non-fatal)" +fi + +# Test 5b: Crash Loop Prevention (kill 4 times rapidly to exceed max restarts) +echo "" +echo "Test 5b: Crash Loop Prevention" +echo "----------------------------------------" +echo "Testing that crash loop is detected and restarts stop after max attempts..." + +# Use the third instance for crash loop testing +LOOP_INSTANCE="${INSTANCE_IDS[2]}" +echo " Crash loop test instance: $LOOP_INSTANCE" + +# Verify instance is running +LOOP_STATE=$($AWS_EC2 describe-instances --instance-ids "$LOOP_INSTANCE" \ + --query 'Reservations[0].Instances[0].State.Name' --output text) +if [ "$LOOP_STATE" != "running" ]; then + echo " ERROR: Instance not running before crash loop test (state: $LOOP_STATE)" + exit 1 +fi + +# Kill QEMU repeatedly to exhaust restart attempts (max 3 in 10 min window) +for crash_num in 1 2 3 4; do + echo " Crash $crash_num/4: killing QEMU..." + LOOP_PID=$(get_qemu_pid "$LOOP_INSTANCE" || echo "") + if [ -z "$LOOP_PID" ]; then + echo " No QEMU process found (instance may be in error state)" + break + fi + + kill -9 "$LOOP_PID" + + if [ $crash_num -lt 4 ]; then + # Wait for restart (backoff increases: 5s, 10s, 20s) + # Give generous time for each restart cycle + local_max=$((15 + crash_num * 10)) + echo " Waiting up to ${local_max}s for restart or error state..." + attempt=0 + while [ $attempt -lt $local_max ]; do + state=$($AWS_EC2 describe-instances --instance-ids "$LOOP_INSTANCE" \ + --query 'Reservations[0].Instances[0].State.Name' --output text 2>/dev/null || echo "unknown") + if [ "$state" == "running" ]; then + echo " Instance restarted (crash $crash_num)" + break + fi + if [ "$state" == "error" ] && [ $crash_num -ge 3 ]; then + echo " Instance in error state after crash $crash_num (restart limit may be reached)" + break + fi + sleep 2 + attempt=$((attempt + 2)) + done + fi +done + +# After 4 rapid crashes, the instance should be in error state (exceeded max 3 restarts) +sleep 5 +LOOP_FINAL_STATE=$($AWS_EC2 describe-instances --instance-ids "$LOOP_INSTANCE" \ + --query 'Reservations[0].Instances[0].State.Name' --output text) +echo " Final state after crash loop: $LOOP_FINAL_STATE" + +if [ "$LOOP_FINAL_STATE" == "error" ]; then + echo " Crash loop prevention working: instance stayed in error state" +else + echo " WARNING: Instance in state '$LOOP_FINAL_STATE' (expected 'error' after exceeding max restarts)" + echo " This may be expected if timing allowed restarts to spread across the window" +fi + +echo " Crash recovery tests passed" + # Cleanup: Terminate all test instances echo "" echo "Cleanup: Deleting test resources"