fix: merge

.claude/CLAUDE.md

@@ -1,189 +0,0 @@
-# CLAUDE.md
-
-This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
-
-## Project Overview
-
-This is the Hive platform - an AWS-compatible infrastructure stack for bare-metal, edge, and on-premise deployments. Hive orchestrates multiple independent components that work together to provide AWS-compatible services.
-
-## Component Repositories
-
-The Hive platform consists of these independent repositories:
-
-- **[viperblock](../viperblock/)** - High-performance block storage service (EBS alternative) with WAL-backed storage
-- **[predastore](../predastore/)** - S3-compatible object storage service with Reed-Solomon erasure coding
-- **nbdkit/** - Network Block Device (NBD) server for exporting block devices (upstream project)
-- **hive/** (this repo) - VM orchestration service and platform coordinator
-
-Each component can be used independently or as part of the integrated Hive stack. See component-specific `CLAUDE.md` files for detailed development guidance:
-- `../viperblock/CLAUDE.md`
-- `../predastore/CLAUDE.md`
-
-## Commands
-
-### Go (backend)
-
-```bash
-make preflight            # Run ALL CI checks locally (format + vet + security + tests)
-make build                # Build hive binary to ./bin/hive
-make test                 # Run all unit tests (sets LOG_IGNORE=1)
-make check-format         # Check gofmt (fails on diff, same as CI)
-make format               # Fix gofmt in place
-make vet                  # Run go vet (fails on issues)
-make security             # Run go vet + govulncheck, gosec, staticcheck (fails on issues)
-make bench                # Run benchmarks
-make clean                # Remove build artifacts
-make test-docker          # Run single + multi-node E2E in Docker (requires /dev/kvm)
-make test-docker-single   # Single-node E2E only
-make test-docker-multi    # Multi-node E2E only
-```
-
-### Frontend (hive-ui)
-
-```bash
-make build-ui                                    # Production build
-cd hive/services/hiveui/frontend && pnpm dev     # Dev server on port 3000
-cd hive/services/hiveui/frontend && pnpm fix     # Biome lint/format fix
-cd hive/services/hiveui/frontend && pnpm build   # Build + typecheck
-```
-
-Frontend stack: React 19, TanStack Router/Query, Tailwind CSS v4, Biome (lint/format), Vite, pnpm. See `.claude/rules/frontend-standards.md` for coding standards.
-
-### Dev Environment
-
-```bash
-scripts/dev-setup.sh    # One-time setup: creates directories, generates config/certs
-scripts/start-dev.sh    # Start all services (NATS → Predastore → Viperblock → NBDkit → Gateway → Daemon → UI)
-scripts/stop-dev.sh     # Stop all services
-```
-
-Data directory: `~/hive/` (logs in `~/hive/logs/`, config in `~/hive/config/`)
-
-## Development Process
-
-All non-trivial work (features, bug fixes, improvements) follows a plan-first, document-as-you-go process. Plans are committed to git so they are visible to the whole team, reviewable in PRs, and serve as permanent documentation.
-
-### When to write a plan
-
-- **Required**: New features, multi-file changes, architectural changes, performance work, non-obvious bug fixes
-- **Not required**: Typo fixes, single-line bug fixes, config tweaks, small refactors contained to one function
-
-### Plan location
-
-Plans live in `docs/development/` under the appropriate category:
-
-```
-docs/development/
-├── bugs/              # Bug fixes
-├── feature/           # New features
-└── improvements/      # Performance, refactoring, tech debt
-```
-
-File naming: lowercase kebab-case describing the work, e.g. `nbd-performance.md`, `attach-volume.md`, `instance-state.md`.
-
-### Plan lifecycle
-
-#### 1. Planning phase
-
-When entering plan mode for non-trivial work, **create the plan document first** and write it to the appropriate `docs/development/{category}/` path. The plan must include:
-
-- **Summary** — What and why, in 1-3 sentences
-- **Context / Problem Statement** — Current behavior, what's wrong or missing
-- **Proposed Changes** — What will change, which files, key design decisions
-- **Files to Modify** — List of files with brief description of changes
-- **Testing** — How the changes will be verified
-
-The plan document is committed to git before implementation begins. This makes the plan reviewable and visible to the team.
-
-#### 2. Implementation phase
-
-During implementation, **update the plan document as you go**:
-
-- Mark completed steps (use `[x]` checkboxes or note "Done" inline)
-- Record findings, corrections, or deviations from the original plan
-- Add code references (file paths, line numbers, struct/function names) for what was actually implemented
-- Note any follow-up work discovered during implementation
-
-#### 3. Completion
-
-When development is complete, update the document to serve as **permanent reference documentation**:
-
-- Add a status line near the top: `**Status: Complete**` (or `**Status: In Progress**`, `**Status: Planned**`)
-- Replace future-tense plan language with past-tense description of what was done
-- Add a **"Files Modified"** summary table
-- Add a **"Future Work"** section if there are known follow-ups
-- Keep the original context (benchmarks, research, root cause analysis) — this is valuable reference material
-- The document should read as a complete technical reference, not just a plan
-
-#### Example status markers
-
-```markdown
-**Status: Planned** — Awaiting review/approval before implementation
-**Status: In Progress** — Implementation underway
-**Status: Complete** — All changes implemented and tests passing
-```
-
-### Git history as context
-
-When starting work on a feature or investigating an issue, review the last 10-20 git commits (`git log --oneline -20`) to understand the current development context. Recent commits contain valuable context about in-progress features, architectural decisions, and related changes.
-
-### Existing examples
-
-- `docs/development/improvements/nbd-performance.md` — Complete improvement with benchmarks, root cause analysis, implementation details, and future work
-- `docs/development/feature/attach-volume.md` — Complete feature with request flow, files changed, and design decisions
-- `docs/development/bugs/instance-state.md` — Bug fix with problem statement, proposed architecture, and state transitions
-
-## Project Standards
-
-- Use `log/slog` instead of `log`. Use appropriate log level, eg `slog.Debug`
-- All new features must have comprehensive unit tests
-- For returning AWS errors, use `awserrors` package instead of manually typed strings
-
-**Preflight Policy:**
-- Run `make preflight` before pushing any major changes — it runs the same checks as GitHub Actions (gofmt, go vet, gosec, staticcheck, govulncheck, and all unit tests)
-- All developers must install the pre-push hook to enforce this automatically:
-
-```bash
-cat > .git/hooks/pre-push << 'EOF'
-#!/bin/sh
-make preflight
-make test
-EOF
-chmod +x .git/hooks/pre-push
-```
-
-**Testing Policy:**
-- **All unit tests MUST pass** before committing changes
-- **No exceptions** - failing tests block commits
-- Tests must complete without errors or panics
-
-## Architecture
-
-Read `DESIGN.md` to understand the full architecture.
-
-### Message-Driven Architecture
-
-```
-AWS SDK (custom endpoint) → AWS Gateway (port 9999) → NATS → Specialized Daemons → Response
-```
-
-- **AWS Gateway** (`hive/services/awsgw/`): TLS endpoint, SigV4 auth, routes to service handlers via NATS
-- **Daemons** (`hive/daemon/`): Subscribe to NATS topics, process requests, manage QEMU/KVM VMs
-- **NATS Topics**: `ec2.*`, `ebs.*`, `s3.*`, `vpc.*` — queue groups (`hive-workers`) for load balancing, no queue group for fan-out (e.g., DescribeInstances)
-- **Storage**: Viperblock for EBS volumes (`ebs.mount` topic), Predastore for S3/AMI/key storage
-
-### Go Workspace
-
-Uses `go.work` with local module replacements — all repos must be cloned to the same parent directory:
-```
-mulga/
-├── hive/         # This repo
-├── predastore/   # S3-compatible storage
-└── viperblock/   # Block storage
-```
-
-### Testing with AWS CLI
-
-```bash
-AWS_PROFILE=hive aws ec2 describe-instances
-```

.claude/rules/frontend-standards.md

@@ -1,90 +0,0 @@
-# Quick Reference
-
-Most formatting and common issues are automatically fixed by Biome. Run `pnpm fix` before committing to ensure compliance and run `pnpm build` for typechecking and route tree generation.
-
-## Core Principles
-
-Write code that is **accessible, performant, type-safe, and maintainable**. Focus on clarity and explicit intent over brevity.
-
-### Type Safety & Explicitness
-
-- Use explicit types for function parameters and return values when they enhance clarity
-- Prefer `unknown` over `any` when the type is genuinely unknown
-- Use const assertions (`as const`) for immutable values and literal types
-- Leverage TypeScript's type narrowing instead of type assertions
-- Use meaningful variable names instead of magic numbers - extract constants with descriptive names
-
-### Modern JavaScript/TypeScript
-
-- Use arrow functions for callbacks and short functions
-- Prefer `for...of` loops over `.forEach()` and indexed `for` loops
-- Use optional chaining (`?.`) and nullish coalescing (`??`) for safer property access
-- Prefer template literals over string concatenation
-- Use destructuring for object and array assignments
-- Use `const` by default, `let` only when reassignment is needed, never `var`
-- Use interfaces for objects, otherwise use type aliases
-- Prefer using function declarations over arrow functions for named functions
-
-### Async & Promises
-
-- Always `await` promises in async functions - don't forget to use the return value
-- Use `async/await` syntax instead of promise chains for better readability
-- Handle errors appropriately in async code with try-catch blocks
-- Don't use async functions as Promise executors
-
-### React & JSX
-
-- Use function components over class components
-- Call hooks at the top level only, never conditionally
-- Specify all dependencies in hook dependency arrays correctly
-- Use the `key` prop for elements in iterables (prefer unique IDs over array indices)
-- Nest children between opening and closing tags instead of passing as props
-- Don't define components inside other components
-- Use semantic HTML and ARIA attributes for accessibility:
-  - Provide meaningful alt text for images
-  - Use proper heading hierarchy
-  - Add labels for form inputs
-  - Include keyboard event handlers alongside mouse events
-  - Use semantic elements (`<button>`, `<nav>`, etc.) instead of divs with roles
-- Don't use `useMemo` or `useCallback`
-- Use ref as a prop instead of `React.forwardRef`
-
-### Error Handling & Debugging
-
-- Remove `console.log`, `debugger`, and `alert` statements from production code
-- Throw `Error` objects with descriptive messages, not strings or other values
-- Use `try-catch` blocks meaningfully - don't catch errors just to rethrow them
-- Prefer early returns over nested conditionals for error cases
-
-### Code Organization
-
-- Keep functions focused and under reasonable cognitive complexity limits
-- Extract complex conditions into well-named boolean variables
-- Use early returns to reduce nesting
-- Prefer simple conditionals over nested ternary operators
-- Group related code together and separate concerns
-
-### Security
-
-- Add `rel="noopener"` when using `target="_blank"` on links
-- Avoid `dangerouslySetInnerHTML` unless absolutely necessary
-- Don't use `eval()` or assign directly to `document.cookie`
-- Validate and sanitize user input
-
-### Performance
-
-- Avoid spread syntax in accumulators within loops
-- Use top-level regex literals instead of creating them in loops
-- Prefer specific imports over namespace imports
-- Avoid barrel files (index files that re-export everything)
-
-## When Biome Can't Help
-
-Biome's linter will catch most issues automatically. Focus your attention on:
-
-1. **Business logic correctness** - Biome can't validate your algorithms
-2. **Meaningful naming** - Use descriptive names for functions, variables, and types
-3. **Architecture decisions** - Component structure, data flow, and API design
-4. **Edge cases** - Handle boundary conditions and error states
-5. **User experience** - Accessibility, performance, and usability considerations
-6. **Documentation** - Add comments for complex logic, but prefer self-documenting code

.claude/settings.json

@@ -1,80 +0,0 @@
-{
-  "permissions": {
-    "allow": [
-      "Skill(feature-dev:feature-dev)",
-      "Skill(pr-review-toolkit:review-pr)",
-
-      "Skill(frontend-design:frontend-design)",
-      "Skill(react-best-practices)",
-      "Skill(react-composition-patterns)",
-      "Skill(web-design-guidelines)",
-
-      "Bash(make build)",
-      "Bash(make build-ui)",
-      "Bash(LOG_IGNORE=1 make test:*)",
-      "Bash(make test)",
-      "Bash(make bench)",
-      "Bash(make clean)",
-      "Bash(make security)",
-
-      "Bash(go build*)",
-      "Bash(go doc:*)",
-      "Bash(LOG_IGNORE=1 go test:*)",
-      "Bash(go test:*)",
-
-      "Bash(ls*)",
-      "Bash(ls -la*)",
-      "Bash(ls -l*)",
-      "Bash(ll*)",
-      "Bash(ls -a*)",
-      "Bash(ls -lh*)",
-      "Bash(ls -la */)",
-      "Bash(cat *)",
-      "Bash(cat -n *)",
-      "Bash(less *)",
-      "Bash(more *)",
-      "Bash(head *)",
-      "Bash(head -n *)",
-      "Bash(head -*)",
-      "Bash(tail *)",
-      "Bash(tail -n *)",
-      "Bash(tail -*)",
-      "Bash(tail -f *)",
-      "Bash(grep *)",
-      "Bash(grep -r *)",
-      "Bash(grep -i *)",
-      "Bash(grep -n *)",
-      "Bash(grep -E *)",
-      "Bash(grep -v *)",
-      "Bash(grep -A* -B* *)",
-      "Bash(find *)",
-      "Bash(find . -name *)",
-      "Bash(find . -type *)",
-      "Bash(find . -path *)",
-      "Bash(diff *)",
-      "Bash(tree *)",
-      "Bash(pwd*)",
-      "Bash(cd *)",
-      "Bash(cd * && *)",
-      "Bash(cd ../viperblock && *)",
-      "Bash(cd ../predastore && *)",
-      "Bash(wc:*)",
-
-      "Bash(pnpm fix:*)",
-      "Bash(pnpm check:*)",
-      "Bash(pnpm build:*)",
-      "Bash(pnpm tsc *)",
-
-      "Bash(git diff*)",
-      "Bash(git diff --staged*)",
-      "Bash(git diff HEAD*)",
-      "Bash(git log*)",
-      "Bash(git log --oneline*)",
-      "Bash(git log --graph*)",
-      "Bash(git status*)",
-      "Bash(git stash*)",
-      "Bash(git stash pop*)",
-      "Bash(git stash list*)"
-    ]
-  }
-}

.claude/skills/go-best-practices/SKILL.md

@@ -1,75 +0,0 @@
----
-name: go-best-practices
-description: Go best practices and security guidelines. This skill should be used when writing, reviewing, or refactoring Go code to ensure correct error handling, secure HTTP servers, proper concurrency patterns, and idiomatic Go. Triggers on tasks involving Go backend code, HTTP handlers, goroutines, or system-level programming.
----
-
-# Go Best Practices
-
-Comprehensive best practices for writing secure, correct, and idiomatic Go code. Contains rules prioritized by impact, covering security (gosec compliance), error handling, HTTP servers, concurrency, and common pitfalls. All rules are enforced by `make security` (go vet, gosec, govulncheck, staticcheck).
-
-## When to Apply
-
-Reference these guidelines when:
-- Writing new Go code (handlers, services, packages)
-- Creating or modifying HTTP servers
-- Working with goroutines, channels, or mutexes
-- Writing functions that return errors
-- Reviewing Go code for correctness and security
-- Implementing AWS-compatible service handlers
-
-## Rule Categories by Priority
-
-| Priority | Category | Impact | Prefix |
-|----------|----------|--------|--------|
-| 1 | HTTP Server Security | CRITICAL | `http-` |
-| 2 | Error Handling | HIGH | `errors-` |
-| 3 | Concurrency | HIGH | `concurrency-` |
-| 4 | Resource Management | MEDIUM | `resources-` |
-| 5 | Code Style & Idioms | LOW-MEDIUM | `style-` |
-
-## Quick Reference
-
-### 1. HTTP Server Security (CRITICAL)
-
-- `http-server-timeouts` - Always configure all timeouts on http.Server
-- `http-handle-write-errors` - Always handle errors from ResponseWriter.Write and JSON encoding
-- `http-input-validation` - Validate and limit request bodies, never trust client input
-
-### 2. Error Handling (HIGH)
-
-- `errors-always-check` - Never ignore returned errors (gosec G104)
-- `errors-wrap-context` - Wrap errors with fmt.Errorf and %w for context
-- `errors-sentinel` - Use sentinel errors and errors.Is/errors.As for comparison
-
-### 3. Concurrency (HIGH)
-
-- `concurrency-goroutine-leaks` - Ensure goroutines can always exit
-- `concurrency-mutex-patterns` - Correct mutex usage and defer unlock
-- `concurrency-channel-ownership` - Clear channel ownership and closing conventions
-
-### 4. Resource Management (MEDIUM)
-
-- `resources-close-bodies` - Always close http response bodies and file handles
-- `resources-context-propagation` - Pass and respect context.Context for cancellation
-- `resources-defer-cleanup` - Use defer for cleanup to ensure execution on all paths
-
-### 5. Code Style & Idioms (LOW-MEDIUM)
-
-- `style-slog-logging` - Use log/slog instead of log, with appropriate levels
-- `style-error-types` - Use awserrors package for AWS error responses
-- `style-table-driven-tests` - Use table-driven tests with subtests
-
-## How to Use
-
-Read individual rule files for detailed explanations and code examples:
-
-```
-rules/http-server-timeouts.md
-rules/errors-always-check.md
-```
-
-Each rule file contains:
-- Brief explanation of why it matters
-- Incorrect code example with explanation
-- Correct code example with explanation
-- gosec/vet rule ID where applicable

.claude/skills/go-best-practices/rules/concurrency-channel-ownership.md

@@ -1,54 +0,0 @@
----
-title: Clear Channel Ownership and Closing
-impact: HIGH
-impactDescription: prevents panics from double-close and sends on closed channels
-tags: go, concurrency, channels, ownership
----
-
-## Clear Channel Ownership and Closing
-
-Follow clear ownership rules for channels: only the sender (producer) should close a channel, never the receiver. Closing a channel from the wrong goroutine causes panics.
-
-**Incorrect (receiver closes, or double close):**
-
-```go
-func producer(ch chan<- int) {
-    for i := 0; i < 10; i++ {
-        ch <- i
-    }
-    // forgot to close — receiver blocks forever
-}
-
-func consumer(ch <-chan int, done chan struct{}) {
-    for val := range ch {
-        process(val)
-    }
-    close(done)
-    close(done)  // panic: close of closed channel
-}
-```
-
-**Correct (sender closes, clear ownership):**
-
-```go
-func producer(ch chan<- int) {
-    defer close(ch)  // sender closes when done
-    for i := 0; i < 10; i++ {
-        ch <- i
-    }
-}
-
-func consumer(ch <-chan int) {
-    for val := range ch {
-        process(val)
-    }
-    // channel was closed by producer, range exits naturally
-}
-```
-
-**Key points:**
-- The goroutine that creates/sends on a channel should close it
-- Use `defer close(ch)` at the top of the sending function
-- Use directional channel types (`chan<-`, `<-chan`) in function signatures to enforce ownership
-- For signaling (done channels), use `chan struct{}` and close it once — don't send values
-- Use `sync.Once` if multiple goroutines might try to close the same channel

.claude/skills/go-best-practices/rules/concurrency-goroutine-leaks.md

@@ -1,77 +0,0 @@
----
-title: Prevent Goroutine Leaks
-impact: HIGH
-impactDescription: prevents memory leaks and resource exhaustion from orphaned goroutines
-tags: go, concurrency, goroutines, context, leaks
----
-
-## Prevent Goroutine Leaks
-
-Every goroutine must have a clear exit path. Goroutines that block forever on channels or I/O without a cancellation mechanism leak memory and resources.
-
-**Incorrect (goroutine can never exit):**
-
-```go
-func startWorker(jobs <-chan Job) {
-    go func() {
-        for job := range jobs {
-            process(job)
-        }
-    }()
-    // If no one closes jobs channel and no context, this goroutine lives forever
-}
-
-func pollStatus(url string, interval time.Duration) {
-    go func() {
-        for {
-            resp, _ := http.Get(url)
-            // runs forever, no way to stop
-            time.Sleep(interval)
-        }
-    }()
-}
-```
-
-**Correct (goroutine exits via context or done channel):**
-
-```go
-func startWorker(ctx context.Context, jobs <-chan Job) {
-    go func() {
-        for {
-            select {
-            case <-ctx.Done():
-                return
-            case job, ok := <-jobs:
-                if !ok {
-                    return
-                }
-                process(job)
-            }
-        }
-    }()
-}
-
-func pollStatus(ctx context.Context, url string, interval time.Duration) {
-    go func() {
-        ticker := time.NewTicker(interval)
-        defer ticker.Stop()
-        for {
-            select {
-            case <-ctx.Done():
-                return
-            case <-ticker.C:
-                // use context-aware HTTP request
-                req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
-                http.DefaultClient.Do(req)
-            }
-        }
-    }()
-}
-```
-
-**Key points:**
-- Always pass `context.Context` to functions that spawn goroutines
-- Use `select` with `ctx.Done()` to enable cancellation
-- Use `time.NewTicker` (and defer `Stop()`) instead of `time.Sleep` in loops
-- Close channels from the sender side to signal completion
-- In server shutdown, use `server.Shutdown(ctx)` to drain connections gracefully

.claude/skills/go-best-practices/rules/concurrency-mutex-patterns.md

@@ -1,82 +0,0 @@
----
-title: Correct Mutex Usage Patterns
-impact: HIGH
-impactDescription: prevents data races, deadlocks, and race condition bugs
-tags: go, concurrency, mutex, sync, data-race
----
-
-## Correct Mutex Usage Patterns
-
-Use mutexes correctly to protect shared state. Common mistakes include forgetting to unlock, holding locks across blocking operations, and using the wrong lock type.
-
-**Incorrect (common mutex mistakes):**
-
-```go
-// Forgetting to unlock on error paths
-func (s *Store) Get(key string) (string, error) {
-    s.mu.Lock()
-    val, ok := s.data[key]
-    if !ok {
-        return "", ErrNotFound  // mutex never unlocked!
-    }
-    s.mu.Unlock()
-    return val, nil
-}
-
-// Holding lock across blocking operation
-func (s *Store) Refresh() {
-    s.mu.Lock()
-    defer s.mu.Unlock()
-    data, _ := http.Get(s.url)  // blocks while holding lock!
-    s.data = parse(data)
-}
-
-// Using RWMutex wrong — write lock where read lock suffices
-func (s *Store) Count() int {
-    s.mu.Lock()  // should be RLock for read-only access
-    defer s.mu.Unlock()
-    return len(s.data)
-}
-```
-
-**Correct:**
-
-```go
-// Always use defer Unlock
-func (s *Store) Get(key string) (string, error) {
-    s.mu.RLock()
-    defer s.mu.RUnlock()
-    val, ok := s.data[key]
-    if !ok {
-        return "", ErrNotFound
-    }
-    return val, nil
-}
-
-// Don't hold lock across blocking ops
-func (s *Store) Refresh() {
-    data, err := http.Get(s.url)  // fetch without lock
-    if err != nil {
-        return
-    }
-    parsed := parse(data)
-
-    s.mu.Lock()
-    s.data = parsed  // only hold lock for assignment
-    s.mu.Unlock()
-}
-
-// Use RLock for read-only access
-func (s *Store) Count() int {
-    s.mu.RLock()
-    defer s.mu.RUnlock()
-    return len(s.data)
-}
-```
-
-**Key points:**
-- Always use `defer mu.Unlock()` immediately after `mu.Lock()` to prevent missed unlocks
-- Use `sync.RWMutex` when reads vastly outnumber writes — `RLock()` for reads, `Lock()` for writes
-- Never hold a lock while doing I/O, network calls, or channel operations
-- Keep critical sections as small as possible
-- Document which fields a mutex protects with a comment above the mutex field

.claude/skills/go-best-practices/rules/errors-always-check.md

@@ -1,61 +0,0 @@
----
-title: Always Check Returned Errors
-impact: HIGH
-impactDescription: prevents silent failures and data corruption (gosec G104)
-tags: go, errors, gosec, reliability
----
-
-## Always Check Returned Errors
-
-Never ignore error return values. Every function that returns an error must have that error checked. This is the single most common source of bugs in Go code.
-
-**Incorrect (errors discarded):**
-
-```go
-file, _ := os.Open(path)           // crash if file doesn't exist
-json.Unmarshal(data, &config)       // silent corruption if data is invalid
-db.Exec("DELETE FROM users WHERE id = ?", id)  // silent failure
-conn.Close()                        // resource leak on error
-```
-
-**Correct (errors handled):**
-
-```go
-file, err := os.Open(path)
-if err != nil {
-    return fmt.Errorf("open config: %w", err)
-}
-
-if err := json.Unmarshal(data, &config); err != nil {
-    return fmt.Errorf("parse config: %w", err)
-}
-
-if _, err := db.Exec("DELETE FROM users WHERE id = ?", id); err != nil {
-    return fmt.Errorf("delete user %s: %w", id, err)
-}
-
-if err := conn.Close(); err != nil {
-    slog.Error("Failed to close connection", "error", err)
-}
-```
-
-**Acceptable exceptions:**
-
-```go
-// In deferred cleanup where you can't return the error
-defer func() {
-    if err := resp.Body.Close(); err != nil {
-        slog.Debug("Failed to close response body", "error", err)
-    }
-}()
-
-// fmt.Fprintf to a bytes.Buffer never fails — but document why
-_, _ = fmt.Fprintf(&buf, "value: %d", n)  // bytes.Buffer.Write never returns error
-```
-
-**Key points:**
-- Use `_ =` explicitly when you intentionally discard an error, with a comment explaining why
-- Log errors in cleanup paths (defer) where you can't propagate them
-- Wrap errors with `%w` to preserve the error chain for callers
-
-**gosec rule:** G104 (CWE-703) — Errors unhandled

.claude/skills/go-best-practices/rules/errors-sentinel.md

@@ -1,57 +0,0 @@
----
-title: Use Sentinel Errors and errors.Is/As
-impact: HIGH
-impactDescription: enables correct error matching across wrapped error chains
-tags: go, errors, sentinel, comparison
----
-
-## Use Sentinel Errors and errors.Is/As
-
-Never compare errors with `==`. Use `errors.Is()` for sentinel error comparison and `errors.As()` for type assertion. Direct comparison breaks when errors are wrapped.
-
-**Incorrect (direct comparison):**
-
-```go
-if err == sql.ErrNoRows {
-    return nil, nil  // breaks if err is wrapped
-}
-
-if err.Error() == "not found" {
-    // fragile string comparison
-}
-
-switch err.(type) {
-case *os.PathError:
-    // breaks with wrapped errors
-}
-```
-
-**Correct (errors.Is and errors.As):**
-
-```go
-if errors.Is(err, sql.ErrNoRows) {
-    return nil, nil  // works even if wrapped
-}
-
-var pathErr *os.PathError
-if errors.As(err, &pathErr) {
-    slog.Error("Path error", "path", pathErr.Path, "op", pathErr.Op)
-}
-```
-
-**Defining sentinel errors:**
-
-```go
-var (
-    ErrNotFound     = errors.New("not found")
-    ErrUnauthorized = errors.New("unauthorized")
-)
-
-// Usage
-return fmt.Errorf("get volume %s: %w", id, ErrNotFound)
-
-// Checking
-if errors.Is(err, ErrNotFound) {
-    w.WriteHeader(http.StatusNotFound)
-}
-```

.claude/skills/go-best-practices/rules/errors-wrap-context.md

@@ -1,52 +0,0 @@
----
-title: Wrap Errors with Context
-impact: HIGH
-impactDescription: enables debugging by preserving error chains and adding call context
-tags: go, errors, debugging, fmt
----
-
-## Wrap Errors with Context
-
-When returning an error from a called function, wrap it with `fmt.Errorf` and `%w` to add context about what operation failed. Bare error returns make debugging difficult because you lose the call chain.
-
-**Incorrect (bare error returns):**
-
-```go
-func LoadConfig(path string) (*Config, error) {
-    data, err := os.ReadFile(path)
-    if err != nil {
-        return nil, err  // caller gets "open /etc/config.yaml: no such file or directory"
-    }
-
-    var cfg Config
-    if err := json.Unmarshal(data, &cfg); err != nil {
-        return nil, err  // caller gets "invalid character '}'" — no idea what file
-    }
-
-    return &cfg, nil
-}
-```
-
-**Correct (wrapped with context):**
-
-```go
-func LoadConfig(path string) (*Config, error) {
-    data, err := os.ReadFile(path)
-    if err != nil {
-        return nil, fmt.Errorf("read config %s: %w", path, err)
-    }
-
-    var cfg Config
-    if err := json.Unmarshal(data, &cfg); err != nil {
-        return nil, fmt.Errorf("parse config %s: %w", path, err)
-    }
-
-    return &cfg, nil
-}
-```
-
-**Key points:**
-- Use `%w` (not `%v`) to wrap errors — this preserves `errors.Is()` and `errors.As()` checks
-- Add the operation name and key parameters (file path, ID, etc.)
-- Keep wrap messages lowercase, no trailing punctuation (Go convention)
-- Don't prefix with "failed to" or "error" — the error itself communicates failure

.claude/skills/go-best-practices/rules/http-handle-write-errors.md

@@ -1,52 +0,0 @@
----
-title: Handle HTTP Response Write Errors
-impact: CRITICAL
-impactDescription: prevents silent failures and satisfies gosec G104
-tags: go, http, error-handling, gosec, handlers
----
-
-## Handle HTTP Response Write Errors
-
-Always check errors returned by `http.ResponseWriter.Write()`, `io.Writer.Write()`, `json.Encoder.Encode()`, and `fmt.Fprintf(w, ...)` in HTTP handlers. Ignoring these errors means you won't know when clients disconnect or responses fail to send.
-
-**Incorrect (errors ignored — gosec G104):**
-
-```go
-func handleHealth(w http.ResponseWriter, r *http.Request) {
-    w.WriteHeader(http.StatusOK)
-    w.Write([]byte("ok"))  // error ignored
-}
-
-func writeJSON(w http.ResponseWriter, status int, v any) {
-    w.Header().Set("Content-Type", "application/json")
-    w.WriteHeader(status)
-    json.NewEncoder(w).Encode(v)  // error ignored
-}
-```
-
-**Correct (errors handled):**
-
-```go
-func handleHealth(w http.ResponseWriter, r *http.Request) {
-    w.WriteHeader(http.StatusOK)
-    if _, err := w.Write([]byte("ok")); err != nil {
-        slog.Error("Failed to write health response", "error", err)
-    }
-}
-
-func writeJSON(w http.ResponseWriter, status int, v any) {
-    w.Header().Set("Content-Type", "application/json")
-    w.WriteHeader(status)
-    if err := json.NewEncoder(w).Encode(v); err != nil {
-        slog.Error("Failed to encode JSON response", "error", err)
-    }
-}
-```
-
-**Key points:**
-- Log write errors at error level — they indicate client disconnects or network issues
-- Don't try to write an HTTP error response after a write failure (the connection is likely broken)
-- This applies to all `io.Writer` methods, not just `http.ResponseWriter`
-- `fmt.Fprintf(w, ...)` also returns an error that must be checked
-
-**gosec rule:** G104 (CWE-703) — Errors unhandled

.claude/skills/go-best-practices/rules/http-input-validation.md

@@ -1,52 +0,0 @@
----
-title: Validate and Limit HTTP Request Input
-impact: CRITICAL
-impactDescription: prevents denial of service via oversized payloads and injection attacks
-tags: go, http, security, validation, input
----
-
-## Validate and Limit HTTP Request Input
-
-Always limit request body size and validate input in HTTP handlers. An unbounded `json.NewDecoder(r.Body).Decode()` allows attackers to send arbitrarily large payloads, exhausting server memory.
-
-**Incorrect (unbounded request body):**
-
-```go
-func handleCreate(w http.ResponseWriter, r *http.Request) {
-    var req CreateRequest
-    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
-        http.Error(w, "bad request", http.StatusBadRequest)
-        return
-    }
-    // process req...
-}
-```
-
-**Correct (limited and validated):**
-
-```go
-const maxRequestBodySize = 1 << 20 // 1 MB
-
-func handleCreate(w http.ResponseWriter, r *http.Request) {
-    r.Body = http.MaxBytesReader(w, r.Body, maxRequestBodySize)
-
-    var req CreateRequest
-    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
-        http.Error(w, "bad request", http.StatusBadRequest)
-        return
-    }
-
-    // Validate required fields
-    if req.Name == "" {
-        http.Error(w, "name is required", http.StatusBadRequest)
-        return
-    }
-    // process req...
-}
-```
-
-**Key points:**
-- Use `http.MaxBytesReader` to limit body size — it returns a 413 error if exceeded
-- Validate all required fields after decoding
-- Sanitize string inputs if they'll be used in commands, queries, or file paths
-- Use early returns for validation failures

.claude/skills/go-best-practices/rules/http-server-timeouts.md

@@ -1,50 +0,0 @@
----
-title: Configure HTTP Server Timeouts
-impact: CRITICAL
-impactDescription: prevents Slowloris attacks and resource exhaustion (gosec G112)
-tags: go, http, security, gosec, server, timeouts
----
-
-## Configure HTTP Server Timeouts
-
-Every `http.Server` must have `ReadHeaderTimeout` configured at minimum. Omitting timeouts allows Slowloris-style attacks where an attacker opens connections and sends headers slowly, exhausting server resources.
-
-**Incorrect (no timeouts — gosec G112):**
-
-```go
-server := &http.Server{
-    Addr:    ":8080",
-    Handler: mux,
-}
-```
-
-This server has no timeout protection. A malicious client can hold connections open indefinitely.
-
-**Correct (all timeouts configured):**
-
-```go
-server := &http.Server{
-    Addr:              ":8080",
-    Handler:           mux,
-    ReadTimeout:       10 * time.Second,
-    WriteTimeout:      10 * time.Second,
-    IdleTimeout:       30 * time.Second,
-    ReadHeaderTimeout: 5 * time.Second,
-}
-```
-
-**Timeout guidelines:**
-
-| Timeout | Purpose | Typical Value |
-|---------|---------|---------------|
-| `ReadHeaderTimeout` | Time to read request headers (Slowloris protection) | 5s |
-| `ReadTimeout` | Time to read entire request including body | 10-30s |
-| `WriteTimeout` | Time to write the response | 10-30s |
-| `IdleTimeout` | Time to keep idle keep-alive connections open | 30-120s |
-
-**Adjust for your use case:**
-- Short-lived internal APIs (formation, health): 10s read/write, 30s idle
-- User-facing APIs with uploads: 30s+ read, 15s write
-- Long-polling endpoints: increase WriteTimeout accordingly
-
-**gosec rule:** G112 (CWE-400) — Potential Slowloris Attack

.claude/skills/go-best-practices/rules/resources-close-bodies.md

@@ -1,47 +0,0 @@
----
-title: Close Response Bodies and File Handles
-impact: MEDIUM
-impactDescription: prevents resource leaks that exhaust file descriptors and connections
-tags: go, resources, http, files, defer, leak
----
-
-## Close Response Bodies and File Handles
-
-Always close `http.Response.Body`, files, and other `io.Closer` resources. Failing to close HTTP response bodies leaks TCP connections; failing to close files leaks file descriptors.
-
-**Incorrect (response body not closed):**
-
-```go
-func fetchData(url string) ([]byte, error) {
-    resp, err := http.Get(url)
-    if err != nil {
-        return nil, err
-    }
-    // resp.Body is never closed — leaks TCP connection
-    return io.ReadAll(resp.Body)
-}
-```
-
-**Correct (defer close immediately after error check):**
-
-```go
-func fetchData(url string) ([]byte, error) {
-    resp, err := http.Get(url)
-    if err != nil {
-        return nil, fmt.Errorf("fetch %s: %w", url, err)
-    }
-    defer resp.Body.Close()
-
-    if resp.StatusCode != http.StatusOK {
-        return nil, fmt.Errorf("fetch %s: status %d", url, resp.StatusCode)
-    }
-
-    return io.ReadAll(resp.Body)
-}
-```
-
-**Key points:**
-- Place `defer Close()` immediately after the nil-error check, before any other logic
-- For files: `defer f.Close()` right after `os.Open`/`os.Create`
-- The body must be fully read and closed even for non-2xx responses — otherwise the connection can't be reused
-- If you need the close error (e.g., for writes): use a named return and check in defer

.claude/skills/go-best-practices/rules/resources-context-propagation.md

@@ -1,49 +0,0 @@
----
-title: Propagate and Respect context.Context
-impact: MEDIUM
-impactDescription: enables cancellation, timeouts, and graceful shutdown across the call chain
-tags: go, context, cancellation, timeout, graceful-shutdown
----
-
-## Propagate and Respect context.Context
-
-Pass `context.Context` as the first parameter to functions that do I/O, run goroutines, or may need cancellation. Never store contexts in structs — they represent a single request lifecycle.
-
-**Incorrect (context ignored or stored):**
-
-```go
-type Service struct {
-    ctx context.Context  // don't store context in structs
-}
-
-func (s *Service) FetchData() ([]byte, error) {
-    resp, err := http.Get(s.url)  // no context — can't cancel
-    return io.ReadAll(resp.Body)
-}
-```
-
-**Correct (context flows through the call chain):**
-
-```go
-func (s *Service) FetchData(ctx context.Context) ([]byte, error) {
-    req, err := http.NewRequestWithContext(ctx, "GET", s.url, nil)
-    if err != nil {
-        return nil, fmt.Errorf("create request: %w", err)
-    }
-
-    resp, err := s.client.Do(req)
-    if err != nil {
-        return nil, fmt.Errorf("fetch data: %w", err)
-    }
-    defer resp.Body.Close()
-
-    return io.ReadAll(resp.Body)
-}
-```
-
-**Key points:**
-- `context.Context` is always the first parameter, named `ctx`
-- Use `context.WithTimeout` or `context.WithCancel` to set deadlines
-- Check `ctx.Err()` or `select` on `ctx.Done()` in long-running loops
-- Use `http.NewRequestWithContext` instead of `http.Get`/`http.Post`
-- In tests, use `context.Background()` or `t.Context()` (Go 1.24+)

.claude/skills/go-best-practices/rules/resources-defer-cleanup.md

@@ -1,54 +0,0 @@
----
-title: Use defer for Cleanup
-impact: MEDIUM
-impactDescription: ensures resources are released on all exit paths including panics
-tags: go, defer, cleanup, resources
----
-
-## Use defer for Cleanup
-
-Use `defer` to ensure cleanup code runs on all function exit paths, including early returns and panics. Place the defer immediately after acquiring the resource.
-
-**Incorrect (cleanup missed on error paths):**
-
-```go
-func processFile(path string) error {
-    f, err := os.Open(path)
-    if err != nil {
-        return err
-    }
-
-    data, err := io.ReadAll(f)
-    if err != nil {
-        return err  // f is never closed!
-    }
-
-    f.Close()
-    return process(data)
-}
-```
-
-**Correct (defer immediately after acquire):**
-
-```go
-func processFile(path string) error {
-    f, err := os.Open(path)
-    if err != nil {
-        return fmt.Errorf("open %s: %w", path, err)
-    }
-    defer f.Close()
-
-    data, err := io.ReadAll(f)
-    if err != nil {
-        return fmt.Errorf("read %s: %w", path, err)
-    }
-
-    return process(data)
-}
-```
-
-**Key points:**
-- Place `defer` on the line immediately after the error check for the acquire operation
-- Defers execute in LIFO order — last deferred runs first
-- For mutexes: `mu.Lock()` then `defer mu.Unlock()` on the next line
-- For write operations where you need the close error, use a named return

.claude/skills/go-best-practices/rules/style-error-types.md

@@ -1,40 +0,0 @@
----
-title: Use awserrors Package for AWS Error Responses
-impact: LOW-MEDIUM
-impactDescription: ensures consistent AWS-compatible error responses across all handlers
-tags: go, aws, errors, handlers, compatibility
----
-
-## Use awserrors Package for AWS Error Responses
-
-When returning AWS-compatible error responses, use the `awserrors` package instead of manually constructing error strings or XML. This ensures consistent error formatting across all service handlers.
-
-**Incorrect (manual error strings):**
-
-```go
-func handleDescribeInstances(w http.ResponseWriter, r *http.Request) {
-    // ...
-    if err != nil {
-        w.WriteHeader(http.StatusInternalServerError)
-        fmt.Fprintf(w, `<Error><Code>InternalError</Code><Message>%s</Message></Error>`, err)
-        return
-    }
-}
-```
-
-**Correct (awserrors package):**
-
-```go
-func handleDescribeInstances(w http.ResponseWriter, r *http.Request) {
-    // ...
-    if err != nil {
-        awserrors.WriteError(w, awserrors.InternalError(err.Error()))
-        return
-    }
-}
-```
-
-**Key points:**
-- The `awserrors` package handles XML formatting, status codes, and AWS error code conventions
-- Use predefined error constructors where available
-- This ensures AWS SDK clients can parse error responses correctly

.claude/skills/go-best-practices/rules/style-slog-logging.md

@@ -1,42 +0,0 @@
----
-title: Use log/slog for Structured Logging
-impact: LOW-MEDIUM
-impactDescription: enables consistent, queryable structured logs across the codebase
-tags: go, logging, slog, structured
----
-
-## Use log/slog for Structured Logging
-
-Use `log/slog` instead of `log` or `fmt.Println` for all logging. Use appropriate log levels and structured key-value pairs.
-
-**Incorrect:**
-
-```go
-import "log"
-
-log.Printf("failed to connect to %s: %v", addr, err)
-fmt.Println("starting server on", addr)
-log.Fatal("database connection failed")
-```
-
-**Correct:**
-
-```go
-import "log/slog"
-
-slog.Error("Failed to connect", "addr", addr, "error", err)
-slog.Info("Server starting", "addr", addr)
-slog.Debug("Cache hit", "key", key, "ttl", ttl)
-```
-
-**Log levels:**
-- `slog.Debug` — verbose diagnostic info, disabled in production
-- `slog.Info` — normal operational events (startup, shutdown, config loaded)
-- `slog.Warn` — unexpected but recoverable situations
-- `slog.Error` — failures that need attention (lost connections, failed operations)
-
-**Key points:**
-- Key-value pairs must alternate: key (string), value (any)
-- Keep keys lowercase, snake_case: `"node_id"`, `"bind_ip"`, `"error"`
-- Always include `"error", err` when logging an error
-- Don't log and return the same error — pick one (usually return)

.claude/skills/go-best-practices/rules/style-table-driven-tests.md

@@ -1,67 +0,0 @@
----
-title: Use Table-Driven Tests
-impact: LOW-MEDIUM
-impactDescription: improves test coverage, readability, and makes it easy to add new cases
-tags: go, testing, table-driven, subtests
----
-
-## Use Table-Driven Tests
-
-Use table-driven tests with `t.Run` subtests for functions with multiple input/output cases. This makes tests easy to extend and provides clear failure messages.
-
-**Incorrect (repetitive test functions):**
-
-```go
-func TestValidatePort_Valid(t *testing.T) {
-    if err := ValidatePort(8080); err != nil {
-        t.Fatalf("unexpected error: %v", err)
-    }
-}
-
-func TestValidatePort_Zero(t *testing.T) {
-    if err := ValidatePort(0); err == nil {
-        t.Fatal("expected error for port 0")
-    }
-}
-
-func TestValidatePort_Negative(t *testing.T) {
-    if err := ValidatePort(-1); err == nil {
-        t.Fatal("expected error for port -1")
-    }
-}
-```
-
-**Correct (table-driven):**
-
-```go
-func TestValidatePort(t *testing.T) {
-    tests := []struct {
-        name    string
-        port    int
-        wantErr bool
-    }{
-        {"valid port", 8080, false},
-        {"min valid", 1, false},
-        {"max valid", 65535, false},
-        {"zero", 0, true},
-        {"negative", -1, true},
-        {"too high", 65536, true},
-    }
-
-    for _, tt := range tests {
-        t.Run(tt.name, func(t *testing.T) {
-            err := ValidatePort(tt.port)
-            if (err != nil) != tt.wantErr {
-                t.Errorf("ValidatePort(%d) error = %v, wantErr %v", tt.port, err, tt.wantErr)
-            }
-        })
-    }
-}
-```
-
-**Key points:**
-- Name each test case descriptively — it appears in test output on failure
-- Use `t.Run` for subtests so individual cases can be run with `-run`
-- Keep test struct fields minimal — only what varies between cases
-- For complex expected outputs, add a `want` field with the expected value
-- Use `t.Parallel()` in subtests when cases are independent

.claude/skills/react-best-practices/SKILL.md

@@ -1,99 +0,0 @@
----
-name: react-best-practices
-description: React performance optimization guidelines. This skill should be used when writing, reviewing, or refactoring React code to ensure optimal performance patterns. Triggers on tasks involving React components, data fetching, bundle optimization, or performance improvements.
----
-
-# React Best Practices
-
-Comprehensive performance optimization guide for React applications. Contains 36 rules across 6 categories, prioritized by impact to guide automated refactoring and code generation.
-
-## When to Apply
-
-Reference these guidelines when:
-- Writing new React components
-- Implementing data fetching
-- Reviewing code for performance issues
-- Refactoring existing React code
-- Optimizing bundle size or load times
-
-## Rule Categories by Priority
-
-| Priority | Category | Impact | Prefix |
-|----------|----------|--------|--------|
-| 1 | Eliminating Waterfalls | CRITICAL | `async-` |
-| 2 | Client-Side Data Fetching | MEDIUM-HIGH | `client-` |
-| 3 | Re-render Optimization | MEDIUM | `rerender-` |
-| 4 | Rendering Performance | MEDIUM | `rendering-` |
-| 5 | JavaScript Performance | LOW-MEDIUM | `js-` |
-| 6 | Advanced Patterns | LOW | `advanced-` |
-
-## Quick Reference
-
-### 1. Eliminating Waterfalls (CRITICAL)
-
-- `async-defer-await` - Move await into branches where actually used
-- `async-parallel` - Use Promise.all() for independent operations
-- `async-dependencies` - Use better-all for partial dependencies
-
-### 2. Client-Side Data Fetching (MEDIUM-HIGH)
-
-- `client-passive-event-listeners` - Use passive listeners for scroll
-- `client-localstorage-schema` - Version and minimize localStorage data
-
-### 3. Re-render Optimization (MEDIUM)
-
-- `rerender-defer-reads` - Don't subscribe to state only used in callbacks
-- `rerender-dependencies` - Use primitive dependencies in effects
-- `rerender-derived-state` - Subscribe to derived booleans, not raw values
-- `rerender-derived-state-no-effect` - Derive state during render, not effects
-- `rerender-functional-setstate` - Use functional setState for stable callbacks
-- `rerender-lazy-state-init` - Pass function to useState for expensive values
-- `rerender-move-effect-to-event` - Put interaction logic in event handlers
-- `rerender-transitions` - Use startTransition for non-urgent updates
-- `rerender-use-ref-transient-values` - Use refs for transient frequent values
-
-### 4. Rendering Performance (MEDIUM)
-
-- `rendering-animate-svg-wrapper` - Animate div wrapper, not SVG element
-- `rendering-content-visibility` - Use content-visibility for long lists
-- `rendering-hoist-jsx` - Extract static JSX outside components
-- `rendering-svg-precision` - Reduce SVG coordinate precision
-- `rendering-activity` - Use Activity component for show/hide
-- `rendering-conditional-render` - Use ternary, not && for conditionals
-- `rendering-usetransition-loading` - Prefer useTransition for loading state
-
-### 5. JavaScript Performance (LOW-MEDIUM)
-
-- `js-batch-dom-css` - Group CSS changes via classes or cssText
-- `js-index-maps` - Build Map for repeated lookups
-- `js-cache-property-access` - Cache object properties in loops
-- `js-cache-function-results` - Cache function results in module-level Map
-- `js-cache-storage` - Cache localStorage/sessionStorage reads
-- `js-combine-iterations` - Combine multiple filter/map into one loop
-- `js-length-check-first` - Check array length before expensive comparison
-- `js-early-exit` - Return early from functions
-- `js-hoist-regexp` - Hoist RegExp creation outside loops
-- `js-min-max-loop` - Use loop for min/max instead of sort
-- `js-set-map-lookups` - Use Set/Map for O(1) lookups
-- `js-tosorted-immutable` - Use toSorted() for immutability
-
-### 6. Advanced Patterns (LOW)
-
-- `advanced-event-handler-refs` - Store event handlers in refs
-- `advanced-init-once` - Initialize app once per app load
-- `advanced-use-latest` - useLatest for stable callback refs
-
-## How to Use
-
-Read individual rule files for detailed explanations and code examples:
-
-```
-rules/async-parallel.md
-rules/bundle-barrel-imports.md
-```
-
-Each rule file contains:
-- Brief explanation of why it matters
-- Incorrect code example with explanation
-- Correct code example with explanation
-- Additional context and references

.claude/skills/react-best-practices/rules/advanced-event-handler-refs.md

@@ -1,55 +0,0 @@
----
-title: Store Event Handlers in Refs
-impact: LOW
-impactDescription: stable subscriptions
-tags: advanced, hooks, refs, event-handlers, optimization
----
-
-## Store Event Handlers in Refs
-
-Store callbacks in refs when used in effects that shouldn't re-subscribe on callback changes.
-
-**Incorrect (re-subscribes on every render):**
-
-```tsx
-function useWindowEvent(event: string, handler: (e) => void) {
-  useEffect(() => {
-    window.addEventListener(event, handler)
-    return () => window.removeEventListener(event, handler)
-  }, [event, handler])
-}
-```
-
-**Correct (stable subscription):**
-
-```tsx
-function useWindowEvent(event: string, handler: (e) => void) {
-  const handlerRef = useRef(handler)
-  useEffect(() => {
-    handlerRef.current = handler
-  }, [handler])
-
-  useEffect(() => {
-    const listener = (e) => handlerRef.current(e)
-    window.addEventListener(event, listener)
-    return () => window.removeEventListener(event, listener)
-  }, [event])
-}
-```
-
-**Alternative: use `useEffectEvent` if you're on latest React:**
-
-```tsx
-import { useEffectEvent } from 'react'
-
-function useWindowEvent(event: string, handler: (e) => void) {
-  const onEvent = useEffectEvent(handler)
-
-  useEffect(() => {
-    window.addEventListener(event, onEvent)
-    return () => window.removeEventListener(event, onEvent)
-  }, [event])
-}
-```
-
-`useEffectEvent` provides a cleaner API for the same pattern: it creates a stable function reference that always calls the latest version of the handler.

.claude/skills/react-best-practices/rules/advanced-init-once.md

@@ -1,42 +0,0 @@
----
-title: Initialize App Once, Not Per Mount
-impact: LOW-MEDIUM
-impactDescription: avoids duplicate init in development
-tags: initialization, useEffect, app-startup, side-effects
----
-
-## Initialize App Once, Not Per Mount
-
-Do not put app-wide initialization that must run once per app load inside `useEffect([])` of a component. Components can remount and effects will re-run. Use a module-level guard or top-level init in the entry module instead.
-
-**Incorrect (runs twice in dev, re-runs on remount):**
-
-```tsx
-function Comp() {
-  useEffect(() => {
-    loadFromStorage()
-    checkAuthToken()
-  }, [])
-
-  // ...
-}
-```
-
-**Correct (once per app load):**
-
-```tsx
-let didInit = false
-
-function Comp() {
-  useEffect(() => {
-    if (didInit) return
-    didInit = true
-    loadFromStorage()
-    checkAuthToken()
-  }, [])
-
-  // ...
-}
-```
-
-Reference: [Initializing the application](https://react.dev/learn/you-might-not-need-an-effect#initializing-the-application)

.claude/skills/react-best-practices/rules/advanced-use-latest.md

@@ -1,39 +0,0 @@
----
-title: useEffectEvent for Stable Callback Refs
-impact: LOW
-impactDescription: prevents effect re-runs
-tags: advanced, hooks, useEffectEvent, refs, optimization
----
-
-## useEffectEvent for Stable Callback Refs
-
-Access latest values in callbacks without adding them to dependency arrays. Prevents effect re-runs while avoiding stale closures.
-
-**Incorrect (effect re-runs on every callback change):**
-
-```tsx
-function SearchInput({ onSearch }: { onSearch: (q: string) => void }) {
-  const [query, setQuery] = useState('')
-
-  useEffect(() => {
-    const timeout = setTimeout(() => onSearch(query), 300)
-    return () => clearTimeout(timeout)
-  }, [query, onSearch])
-}
-```
-
-**Correct (using React's useEffectEvent):**
-
-```tsx
-import { useEffectEvent } from 'react';
-
-function SearchInput({ onSearch }: { onSearch: (q: string) => void }) {
-  const [query, setQuery] = useState('')
-  const onSearchEvent = useEffectEvent(onSearch)
-
-  useEffect(() => {
-    const timeout = setTimeout(() => onSearchEvent(query), 300)
-    return () => clearTimeout(timeout)
-  }, [query])
-}
-```

.claude/skills/react-best-practices/rules/async-defer-await.md

@@ -1,80 +0,0 @@
----
-title: Defer Await Until Needed
-impact: HIGH
-impactDescription: avoids blocking unused code paths
-tags: async, await, conditional, optimization
----
-
-## Defer Await Until Needed
-
-Move `await` operations into the branches where they're actually used to avoid blocking code paths that don't need them.
-
-**Incorrect (blocks both branches):**
-
-```typescript
-async function handleRequest(userId: string, skipProcessing: boolean) {
-  const userData = await fetchUserData(userId)
-  
-  if (skipProcessing) {
-    // Returns immediately but still waited for userData
-    return { skipped: true }
-  }
-  
-  // Only this branch uses userData
-  return processUserData(userData)
-}
-```
-
-**Correct (only blocks when needed):**
-
-```typescript
-async function handleRequest(userId: string, skipProcessing: boolean) {
-  if (skipProcessing) {
-    // Returns immediately without waiting
-    return { skipped: true }
-  }
-  
-  // Fetch only when needed
-  const userData = await fetchUserData(userId)
-  return processUserData(userData)
-}
-```
-
-**Another example (early return optimization):**
-
-```typescript
-// Incorrect: always fetches permissions
-async function updateResource(resourceId: string, userId: string) {
-  const permissions = await fetchPermissions(userId)
-  const resource = await getResource(resourceId)
-  
-  if (!resource) {
-    return { error: 'Not found' }
-  }
-  
-  if (!permissions.canEdit) {
-    return { error: 'Forbidden' }
-  }
-  
-  return await updateResourceData(resource, permissions)
-}
-
-// Correct: fetches only when needed
-async function updateResource(resourceId: string, userId: string) {
-  const resource = await getResource(resourceId)
-  
-  if (!resource) {
-    return { error: 'Not found' }
-  }
-  
-  const permissions = await fetchPermissions(userId)
-  
-  if (!permissions.canEdit) {
-    return { error: 'Forbidden' }
-  }
-  
-  return await updateResourceData(resource, permissions)
-}
-```
-
-This optimization is especially valuable when the skipped branch is frequently taken, or when the deferred operation is expensive.

.claude/skills/react-best-practices/rules/async-dependencies.md

@@ -1,51 +0,0 @@
----
-title: Dependency-Based Parallelization
-impact: CRITICAL
-impactDescription: 2-10× improvement
-tags: async, parallelization, dependencies, better-all
----
-
-## Dependency-Based Parallelization
-
-For operations with partial dependencies, use `better-all` to maximize parallelism. It automatically starts each task at the earliest possible moment.
-
-**Incorrect (profile waits for config unnecessarily):**
-
-```typescript
-const [user, config] = await Promise.all([
-  fetchUser(),
-  fetchConfig()
-])
-const profile = await fetchProfile(user.id)
-```
-
-**Correct (config and profile run in parallel):**
-
-```typescript
-import { all } from 'better-all'
-
-const { user, config, profile } = await all({
-  async user() { return fetchUser() },
-  async config() { return fetchConfig() },
-  async profile() {
-    return fetchProfile((await this.$.user).id)
-  }
-})
-```
-
-**Alternative without extra dependencies:**
-
-We can also create all the promises first, and do `Promise.all()` at the end.
-
-```typescript
-const userPromise = fetchUser()
-const profilePromise = userPromise.then(user => fetchProfile(user.id))
-
-const [user, config, profile] = await Promise.all([
-  userPromise,
-  fetchConfig(),
-  profilePromise
-])
-```
-
-Reference: [https://github.com/shuding/better-all](https://github.com/shuding/better-all)

.claude/skills/react-best-practices/rules/async-parallel.md

@@ -1,28 +0,0 @@
----
-title: Promise.all() for Independent Operations
-impact: CRITICAL
-impactDescription: 2-10× improvement
-tags: async, parallelization, promises, waterfalls
----
-
-## Promise.all() for Independent Operations
-
-When async operations have no interdependencies, execute them concurrently using `Promise.all()`.
-
-**Incorrect (sequential execution, 3 round trips):**
-
-```typescript
-const user = await fetchUser()
-const posts = await fetchPosts()
-const comments = await fetchComments()
-```
-
-**Correct (parallel execution, 1 round trip):**
-
-```typescript
-const [user, posts, comments] = await Promise.all([
-  fetchUser(),
-  fetchPosts(),
-  fetchComments()
-])
-```

.claude/skills/react-best-practices/rules/client-localstorage-schema.md

@@ -1,71 +0,0 @@
----
-title: Version and Minimize localStorage Data
-impact: MEDIUM
-impactDescription: prevents schema conflicts, reduces storage size
-tags: client, localStorage, storage, versioning, data-minimization
----
-
-## Version and Minimize localStorage Data
-
-Add version prefix to keys and store only needed fields. Prevents schema conflicts and accidental storage of sensitive data.
-
-**Incorrect:**
-
-```typescript
-// No version, stores everything, no error handling
-localStorage.setItem('userConfig', JSON.stringify(fullUserObject))
-const data = localStorage.getItem('userConfig')
-```
-
-**Correct:**
-
-```typescript
-const VERSION = 'v2'
-
-function saveConfig(config: { theme: string; language: string }) {
-  try {
-    localStorage.setItem(`userConfig:${VERSION}`, JSON.stringify(config))
-  } catch {
-    // Throws in incognito/private browsing, quota exceeded, or disabled
-  }
-}
-
-function loadConfig() {
-  try {
-    const data = localStorage.getItem(`userConfig:${VERSION}`)
-    return data ? JSON.parse(data) : null
-  } catch {
-    return null
-  }
-}
-
-// Migration from v1 to v2
-function migrate() {
-  try {
-    const v1 = localStorage.getItem('userConfig:v1')
-    if (v1) {
-      const old = JSON.parse(v1)
-      saveConfig({ theme: old.darkMode ? 'dark' : 'light', language: old.lang })
-      localStorage.removeItem('userConfig:v1')
-    }
-  } catch {}
-}
-```
-
-**Store minimal fields from server responses:**
-
-```typescript
-// User object has 20+ fields, only store what UI needs
-function cachePrefs(user: FullUser) {
-  try {
-    localStorage.setItem('prefs:v1', JSON.stringify({
-      theme: user.preferences.theme,
-      notifications: user.preferences.notifications
-    }))
-  } catch {}
-}
-```
-
-**Always wrap in try-catch:** `getItem()` and `setItem()` throw in incognito/private browsing (Safari, Firefox), when quota exceeded, or when disabled.
-
-**Benefits:** Schema evolution via versioning, reduced storage size, prevents storing tokens/PII/internal flags.

.claude/skills/react-best-practices/rules/client-passive-event-listeners.md

@@ -1,48 +0,0 @@
----
-title: Use Passive Event Listeners for Scrolling Performance
-impact: MEDIUM
-impactDescription: eliminates scroll delay caused by event listeners
-tags: client, event-listeners, scrolling, performance, touch, wheel
----
-
-## Use Passive Event Listeners for Scrolling Performance
-
-Add `{ passive: true }` to touch and wheel event listeners to enable immediate scrolling. Browsers normally wait for listeners to finish to check if `preventDefault()` is called, causing scroll delay.
-
-**Incorrect:**
-
-```typescript
-useEffect(() => {
-  const handleTouch = (e: TouchEvent) => console.log(e.touches[0].clientX)
-  const handleWheel = (e: WheelEvent) => console.log(e.deltaY)
-  
-  document.addEventListener('touchstart', handleTouch)
-  document.addEventListener('wheel', handleWheel)
-  
-  return () => {
-    document.removeEventListener('touchstart', handleTouch)
-    document.removeEventListener('wheel', handleWheel)
-  }
-}, [])
-```
-
-**Correct:**
-
-```typescript
-useEffect(() => {
-  const handleTouch = (e: TouchEvent) => console.log(e.touches[0].clientX)
-  const handleWheel = (e: WheelEvent) => console.log(e.deltaY)
-  
-  document.addEventListener('touchstart', handleTouch, { passive: true })
-  document.addEventListener('wheel', handleWheel, { passive: true })
-  
-  return () => {
-    document.removeEventListener('touchstart', handleTouch)
-    document.removeEventListener('wheel', handleWheel)
-  }
-}, [])
-```
-
-**Use passive when:** tracking/analytics, logging, any listener that doesn't call `preventDefault()`.
-
-**Don't use passive when:** implementing custom swipe gestures, custom zoom controls, or any listener that needs `preventDefault()`.

.claude/skills/react-best-practices/rules/js-batch-dom-css.md

@@ -1,107 +0,0 @@
----
-title: Avoid Layout Thrashing
-impact: MEDIUM
-impactDescription: prevents forced synchronous layouts and reduces performance bottlenecks
-tags: javascript, dom, css, performance, reflow, layout-thrashing
----
-
-## Avoid Layout Thrashing
-
-Avoid interleaving style writes with layout reads. When you read a layout property (like `offsetWidth`, `getBoundingClientRect()`, or `getComputedStyle()`) between style changes, the browser is forced to trigger a synchronous reflow.
-
-**This is OK (browser batches style changes):**
-```typescript
-function updateElementStyles(element: HTMLElement) {
-  // Each line invalidates style, but browser batches the recalculation
-  element.style.width = '100px'
-  element.style.height = '200px'
-  element.style.backgroundColor = 'blue'
-  element.style.border = '1px solid black'
-}
-```
-
-**Incorrect (interleaved reads and writes force reflows):**
-```typescript
-function layoutThrashing(element: HTMLElement) {
-  element.style.width = '100px'
-  const width = element.offsetWidth  // Forces reflow
-  element.style.height = '200px'
-  const height = element.offsetHeight  // Forces another reflow
-}
-```
-
-**Correct (batch writes, then read once):**
-```typescript
-function updateElementStyles(element: HTMLElement) {
-  // Batch all writes together
-  element.style.width = '100px'
-  element.style.height = '200px'
-  element.style.backgroundColor = 'blue'
-  element.style.border = '1px solid black'
-  
-  // Read after all writes are done (single reflow)
-  const { width, height } = element.getBoundingClientRect()
-}
-```
-
-**Correct (batch reads, then writes):**
-```typescript
-function avoidThrashing(element: HTMLElement) {
-  // Read phase - all layout queries first
-  const rect1 = element.getBoundingClientRect()
-  const offsetWidth = element.offsetWidth
-  const offsetHeight = element.offsetHeight
-  
-  // Write phase - all style changes after
-  element.style.width = '100px'
-  element.style.height = '200px'
-}
-```
-
-**Better: use CSS classes**
-```css
-.highlighted-box {
-  width: 100px;
-  height: 200px;
-  background-color: blue;
-  border: 1px solid black;
-}
-```
-```typescript
-function updateElementStyles(element: HTMLElement) {
-  element.classList.add('highlighted-box')
-  
-  const { width, height } = element.getBoundingClientRect()
-}
-```
-
-**React example:**
-```tsx
-// Incorrect: interleaving style changes with layout queries
-function Box({ isHighlighted }: { isHighlighted: boolean }) {
-  const ref = useRef<HTMLDivElement>(null)
-  
-  useEffect(() => {
-    if (ref.current && isHighlighted) {
-      ref.current.style.width = '100px'
-      const width = ref.current.offsetWidth // Forces layout
-      ref.current.style.height = '200px'
-    }
-  }, [isHighlighted])
-  
-  return <div ref={ref}>Content</div>
-}
-
-// Correct: toggle class
-function Box({ isHighlighted }: { isHighlighted: boolean }) {
-  return (
-    <div className={isHighlighted ? 'highlighted-box' : ''}>
-      Content
-    </div>
-  )
-}
-```
-
-Prefer CSS classes over inline styles when possible. CSS files are cached by the browser, and classes provide better separation of concerns and are easier to maintain.
-
-See [this gist](https://gist.github.com/paulirish/5d52fb081b3570c81e3a) and [CSS Triggers](https://csstriggers.com/) for more information on layout-forcing operations.

.claude/skills/react-best-practices/rules/js-cache-function-results.md

@@ -1,80 +0,0 @@
----
-title: Cache Repeated Function Calls
-impact: MEDIUM
-impactDescription: avoid redundant computation
-tags: javascript, cache, memoization, performance
----
-
-## Cache Repeated Function Calls
-
-Use a module-level Map to cache function results when the same function is called repeatedly with the same inputs during render.
-
-**Incorrect (redundant computation):**
-
-```typescript
-function ProjectList({ projects }: { projects: Project[] }) {
-  return (
-    <div>
-      {projects.map(project => {
-        // slugify() called 100+ times for same project names
-        const slug = slugify(project.name)
-        
-        return <ProjectCard key={project.id} slug={slug} />
-      })}
-    </div>
-  )
-}
-```
-
-**Correct (cached results):**
-
-```typescript
-// Module-level cache
-const slugifyCache = new Map<string, string>()
-
-function cachedSlugify(text: string): string {
-  if (slugifyCache.has(text)) {
-    return slugifyCache.get(text)!
-  }
-  const result = slugify(text)
-  slugifyCache.set(text, result)
-  return result
-}
-
-function ProjectList({ projects }: { projects: Project[] }) {
-  return (
-    <div>
-      {projects.map(project => {
-        // Computed only once per unique project name
-        const slug = cachedSlugify(project.name)
-        
-        return <ProjectCard key={project.id} slug={slug} />
-      })}
-    </div>
-  )
-}
-```
-
-**Simpler pattern for single-value functions:**
-
-```typescript
-let isLoggedInCache: boolean | null = null
-
-function isLoggedIn(): boolean {
-  if (isLoggedInCache !== null) {
-    return isLoggedInCache
-  }
-  
-  isLoggedInCache = document.cookie.includes('auth=')
-  return isLoggedInCache
-}
-
-// Clear cache when auth changes
-function onAuthChange() {
-  isLoggedInCache = null
-}
-```
-
-Use a Map (not a hook) so it works everywhere: utilities, event handlers, not just React components.
-
-Reference: [How we made the Vercel Dashboard twice as fast](https://vercel.com/blog/how-we-made-the-vercel-dashboard-twice-as-fast)

.claude/skills/react-best-practices/rules/js-cache-property-access.md

@@ -1,28 +0,0 @@
----
-title: Cache Property Access in Loops
-impact: LOW-MEDIUM
-impactDescription: reduces lookups
-tags: javascript, loops, optimization, caching
----
-
-## Cache Property Access in Loops
-
-Cache object property lookups in hot paths.
-
-**Incorrect (3 lookups × N iterations):**
-
-```typescript
-for (let i = 0; i < arr.length; i++) {
-  process(obj.config.settings.value)
-}
-```
-
-**Correct (1 lookup total):**
-
-```typescript
-const value = obj.config.settings.value
-const len = arr.length
-for (let i = 0; i < len; i++) {
-  process(value)
-}
-```

.claude/skills/react-best-practices/rules/js-cache-storage.md

@@ -1,70 +0,0 @@
----
-title: Cache Storage API Calls
-impact: LOW-MEDIUM
-impactDescription: reduces expensive I/O
-tags: javascript, localStorage, storage, caching, performance
----
-
-## Cache Storage API Calls
-
-`localStorage`, `sessionStorage`, and `document.cookie` are synchronous and expensive. Cache reads in memory.
-
-**Incorrect (reads storage on every call):**
-
-```typescript
-function getTheme() {
-  return localStorage.getItem('theme') ?? 'light'
-}
-// Called 10 times = 10 storage reads
-```
-
-**Correct (Map cache):**
-
-```typescript
-const storageCache = new Map<string, string | null>()
-
-function getLocalStorage(key: string) {
-  if (!storageCache.has(key)) {
-    storageCache.set(key, localStorage.getItem(key))
-  }
-  return storageCache.get(key)
-}
-
-function setLocalStorage(key: string, value: string) {
-  localStorage.setItem(key, value)
-  storageCache.set(key, value)  // keep cache in sync
-}
-```
-
-Use a Map (not a hook) so it works everywhere: utilities, event handlers, not just React components.
-
-**Cookie caching:**
-
-```typescript
-let cookieCache: Record<string, string> | null = null
-
-function getCookie(name: string) {
-  if (!cookieCache) {
-    cookieCache = Object.fromEntries(
-      document.cookie.split('; ').map(c => c.split('='))
-    )
-  }
-  return cookieCache[name]
-}
-```
-
-**Important (invalidate on external changes):**
-
-If storage can change externally (another tab, server-set cookies), invalidate cache:
-
-```typescript
-window.addEventListener('storage', (e) => {
-  if (e.key) storageCache.delete(e.key)
-})
-
-document.addEventListener('visibilitychange', () => {
-  if (document.visibilityState === 'visible') {
-    storageCache.clear()
-  }
-})
-```

.claude/skills/react-best-practices/rules/js-combine-iterations.md

@@ -1,32 +0,0 @@
----
-title: Combine Multiple Array Iterations
-impact: LOW-MEDIUM
-impactDescription: reduces iterations
-tags: javascript, arrays, loops, performance
----
-
-## Combine Multiple Array Iterations
-
-Multiple `.filter()` or `.map()` calls iterate the array multiple times. Combine into one loop.
-
-**Incorrect (3 iterations):**
-
-```typescript
-const admins = users.filter(u => u.isAdmin)
-const testers = users.filter(u => u.isTester)
-const inactive = users.filter(u => !u.isActive)
-```
-
-**Correct (1 iteration):**
-
-```typescript
-const admins: User[] = []
-const testers: User[] = []
-const inactive: User[] = []
-
-for (const user of users) {
-  if (user.isAdmin) admins.push(user)
-  if (user.isTester) testers.push(user)
-  if (!user.isActive) inactive.push(user)
-}
-```

.claude/skills/react-best-practices/rules/js-early-exit.md

@@ -1,50 +0,0 @@
----
-title: Early Return from Functions
-impact: LOW-MEDIUM
-impactDescription: avoids unnecessary computation
-tags: javascript, functions, optimization, early-return
----
-
-## Early Return from Functions
-
-Return early when result is determined to skip unnecessary processing.
-
-**Incorrect (processes all items even after finding answer):**
-
-```typescript
-function validateUsers(users: User[]) {
-  let hasError = false
-  let errorMessage = ''
-  
-  for (const user of users) {
-    if (!user.email) {
-      hasError = true
-      errorMessage = 'Email required'
-    }
-    if (!user.name) {
-      hasError = true
-      errorMessage = 'Name required'
-    }
-    // Continues checking all users even after error found
-  }
-  
-  return hasError ? { valid: false, error: errorMessage } : { valid: true }
-}
-```
-
-**Correct (returns immediately on first error):**
-
-```typescript
-function validateUsers(users: User[]) {
-  for (const user of users) {
-    if (!user.email) {
-      return { valid: false, error: 'Email required' }
-    }
-    if (!user.name) {
-      return { valid: false, error: 'Name required' }
-    }
-  }
-
-  return { valid: true }
-}
-```

.claude/skills/react-best-practices/rules/js-hoist-regexp.md

@@ -1,45 +0,0 @@
----
-title: Hoist RegExp Creation
-impact: LOW-MEDIUM
-impactDescription: avoids recreation
-tags: javascript, regexp, optimization, memoization
----
-
-## Hoist RegExp Creation
-
-Don't create RegExp inside render. Hoist to module scope or memoize with `useMemo()`.
-
-**Incorrect (new RegExp every render):**
-
-```tsx
-function Highlighter({ text, query }: Props) {
-  const regex = new RegExp(`(${query})`, 'gi')
-  const parts = text.split(regex)
-  return <>{parts.map((part, i) => ...)}</>
-}
-```
-
-**Correct (memoize or hoist):**
-
-```tsx
-const EMAIL_REGEX = /^[^\s@]+@[^\s@]+\.[^\s@]+$/
-
-function Highlighter({ text, query }: Props) {
-  const regex = useMemo(
-    () => new RegExp(`(${escapeRegex(query)})`, 'gi'),
-    [query]
-  )
-  const parts = text.split(regex)
-  return <>{parts.map((part, i) => ...)}</>
-}
-```
-
-**Warning (global regex has mutable state):**
-
-Global regex (`/g`) has mutable `lastIndex` state:
-
-```typescript
-const regex = /foo/g
-regex.test('foo')  // true, lastIndex = 3
-regex.test('foo')  // false, lastIndex = 0
-```

.claude/skills/react-best-practices/rules/js-index-maps.md

@@ -1,37 +0,0 @@
----
-title: Build Index Maps for Repeated Lookups
-impact: LOW-MEDIUM
-impactDescription: 1M ops to 2K ops
-tags: javascript, map, indexing, optimization, performance
----
-
-## Build Index Maps for Repeated Lookups
-
-Multiple `.find()` calls by the same key should use a Map.
-
-**Incorrect (O(n) per lookup):**
-
-```typescript
-function processOrders(orders: Order[], users: User[]) {
-  return orders.map(order => ({
-    ...order,
-    user: users.find(u => u.id === order.userId)
-  }))
-}
-```
-
-**Correct (O(1) per lookup):**
-
-```typescript
-function processOrders(orders: Order[], users: User[]) {
-  const userById = new Map(users.map(u => [u.id, u]))
-
-  return orders.map(order => ({
-    ...order,
-    user: userById.get(order.userId)
-  }))
-}
-```
-
-Build map once (O(n)), then all lookups are O(1).
-For 1000 orders × 1000 users: 1M ops → 2K ops.

.claude/skills/react-best-practices/rules/js-length-check-first.md

@@ -1,49 +0,0 @@
----
-title: Early Length Check for Array Comparisons
-impact: MEDIUM-HIGH
-impactDescription: avoids expensive operations when lengths differ
-tags: javascript, arrays, performance, optimization, comparison
----
-
-## Early Length Check for Array Comparisons
-
-When comparing arrays with expensive operations (sorting, deep equality, serialization), check lengths first. If lengths differ, the arrays cannot be equal.
-
-In real-world applications, this optimization is especially valuable when the comparison runs in hot paths (event handlers, render loops).
-
-**Incorrect (always runs expensive comparison):**
-
-```typescript
-function hasChanges(current: string[], original: string[]) {
-  // Always sorts and joins, even when lengths differ
-  return current.sort().join() !== original.sort().join()
-}
-```
-
-Two O(n log n) sorts run even when `current.length` is 5 and `original.length` is 100. There is also overhead of joining the arrays and comparing the strings.
-
-**Correct (O(1) length check first):**
-
-```typescript
-function hasChanges(current: string[], original: string[]) {
-  // Early return if lengths differ
-  if (current.length !== original.length) {
-    return true
-  }
-  // Only sort when lengths match
-  const currentSorted = current.toSorted()
-  const originalSorted = original.toSorted()
-  for (let i = 0; i < currentSorted.length; i++) {
-    if (currentSorted[i] !== originalSorted[i]) {
-      return true
-    }
-  }
-  return false
-}
-```
-
-This new approach is more efficient because:
-- It avoids the overhead of sorting and joining the arrays when lengths differ
-- It avoids consuming memory for the joined strings (especially important for large arrays)
-- It avoids mutating the original arrays
-- It returns early when a difference is found

.claude/skills/react-best-practices/rules/js-min-max-loop.md

@@ -1,82 +0,0 @@
----
-title: Use Loop for Min/Max Instead of Sort
-impact: LOW
-impactDescription: O(n) instead of O(n log n)
-tags: javascript, arrays, performance, sorting, algorithms
----
-
-## Use Loop for Min/Max Instead of Sort
-
-Finding the smallest or largest element only requires a single pass through the array. Sorting is wasteful and slower.
-
-**Incorrect (O(n log n) - sort to find latest):**
-
-```typescript
-interface Project {
-  id: string
-  name: string
-  updatedAt: number
-}
-
-function getLatestProject(projects: Project[]) {
-  const sorted = [...projects].sort((a, b) => b.updatedAt - a.updatedAt)
-  return sorted[0]
-}
-```
-
-Sorts the entire array just to find the maximum value.
-
-**Incorrect (O(n log n) - sort for oldest and newest):**
-
-```typescript
-function getOldestAndNewest(projects: Project[]) {
-  const sorted = [...projects].sort((a, b) => a.updatedAt - b.updatedAt)
-  return { oldest: sorted[0], newest: sorted[sorted.length - 1] }
-}
-```
-
-Still sorts unnecessarily when only min/max are needed.
-
-**Correct (O(n) - single loop):**
-
-```typescript
-function getLatestProject(projects: Project[]) {
-  if (projects.length === 0) return null
-  
-  let latest = projects[0]
-  
-  for (let i = 1; i < projects.length; i++) {
-    if (projects[i].updatedAt > latest.updatedAt) {
-      latest = projects[i]
-    }
-  }
-  
-  return latest
-}
-
-function getOldestAndNewest(projects: Project[]) {
-  if (projects.length === 0) return { oldest: null, newest: null }
-  
-  let oldest = projects[0]
-  let newest = projects[0]
-  
-  for (let i = 1; i < projects.length; i++) {
-    if (projects[i].updatedAt < oldest.updatedAt) oldest = projects[i]
-    if (projects[i].updatedAt > newest.updatedAt) newest = projects[i]
-  }
-  
-  return { oldest, newest }
-}
-```
-
-Single pass through the array, no copying, no sorting.
-
-**Alternative (Math.min/Math.max for small arrays):**
-
-```typescript
-const numbers = [5, 2, 8, 1, 9]
-const min = Math.min(...numbers)
-const max = Math.max(...numbers)
-```
-
-This works for small arrays, but can be slower or just throw an error for very large arrays due to spread operator limitations. Maximal array length is approximately 124000 in Chrome 143 and 638000 in Safari 18; exact numbers may vary - see [the fiddle](https://jsfiddle.net/qw1jabsx/4/). Use the loop approach for reliability.

.claude/skills/react-best-practices/rules/js-set-map-lookups.md

@@ -1,24 +0,0 @@
----
-title: Use Set/Map for O(1) Lookups
-impact: LOW-MEDIUM
-impactDescription: O(n) to O(1)
-tags: javascript, set, map, data-structures, performance
----
-
-## Use Set/Map for O(1) Lookups
-
-Convert arrays to Set/Map for repeated membership checks.
-
-**Incorrect (O(n) per check):**
-
-```typescript
-const allowedIds = ['a', 'b', 'c', ...]
-items.filter(item => allowedIds.includes(item.id))
-```
-
-**Correct (O(1) per check):**
-
-```typescript
-const allowedIds = new Set(['a', 'b', 'c', ...])
-items.filter(item => allowedIds.has(item.id))
-```

.claude/skills/react-best-practices/rules/js-tosorted-immutable.md

@@ -1,57 +0,0 @@
----
-title: Use toSorted() Instead of sort() for Immutability
-impact: MEDIUM-HIGH
-impactDescription: prevents mutation bugs in React state
-tags: javascript, arrays, immutability, react, state, mutation
----
-
-## Use toSorted() Instead of sort() for Immutability
-
-`.sort()` mutates the array in place, which can cause bugs with React state and props. Use `.toSorted()` to create a new sorted array without mutation.
-
-**Incorrect (mutates original array):**
-
-```typescript
-function UserList({ users }: { users: User[] }) {
-  // Mutates the users prop array!
-  const sorted = useMemo(
-    () => users.sort((a, b) => a.name.localeCompare(b.name)),
-    [users]
-  )
-  return <div>{sorted.map(renderUser)}</div>
-}
-```
-
-**Correct (creates new array):**
-
-```typescript
-function UserList({ users }: { users: User[] }) {
-  // Creates new sorted array, original unchanged
-  const sorted = useMemo(
-    () => users.toSorted((a, b) => a.name.localeCompare(b.name)),
-    [users]
-  )
-  return <div>{sorted.map(renderUser)}</div>
-}
-```
-
-**Why this matters in React:**
-
-1. Props/state mutations break React's immutability model - React expects props and state to be treated as read-only
-2. Causes stale closure bugs - Mutating arrays inside closures (callbacks, effects) can lead to unexpected behavior
-
-**Browser support (fallback for older browsers):**
-
-`.toSorted()` is available in all modern browsers (Chrome 110+, Safari 16+, Firefox 115+, Node.js 20+). For older environments, use spread operator:
-
-```typescript
-// Fallback for older browsers
-const sorted = [...items].sort((a, b) => a.value - b.value)
-```
-
-**Other immutable array methods:**
-
-- `.toSorted()` - immutable sort
-- `.toReversed()` - immutable reverse
-- `.toSpliced()` - immutable splice
-- `.with()` - immutable element replacement

.claude/skills/react-best-practices/rules/rendering-activity.md

@@ -1,26 +0,0 @@
----
-title: Use Activity Component for Show/Hide
-impact: MEDIUM
-impactDescription: preserves state/DOM
-tags: rendering, activity, visibility, state-preservation
----
-
-## Use Activity Component for Show/Hide
-
-Use React's `<Activity>` to preserve state/DOM for expensive components that frequently toggle visibility.
-
-**Usage:**
-
-```tsx
-import { Activity } from 'react'
-
-function Dropdown({ isOpen }: Props) {
-  return (
-    <Activity mode={isOpen ? 'visible' : 'hidden'}>
-      <ExpensiveMenu />
-    </Activity>
-  )
-}
-```
-
-Avoids expensive re-renders and state loss.

.claude/skills/react-best-practices/rules/rendering-animate-svg-wrapper.md

@@ -1,47 +0,0 @@
----
-title: Animate SVG Wrapper Instead of SVG Element
-impact: LOW
-impactDescription: enables hardware acceleration
-tags: rendering, svg, css, animation, performance
----
-
-## Animate SVG Wrapper Instead of SVG Element
-
-Many browsers don't have hardware acceleration for CSS3 animations on SVG elements. Wrap SVG in a `<div>` and animate the wrapper instead.
-
-**Incorrect (animating SVG directly - no hardware acceleration):**
-
-```tsx
-function LoadingSpinner() {
-  return (
-    <svg 
-      className="animate-spin"
-      width="24" 
-      height="24" 
-      viewBox="0 0 24 24"
-    >
-      <circle cx="12" cy="12" r="10" stroke="currentColor" />
-    </svg>
-  )
-}
-```
-
-**Correct (animating wrapper div - hardware accelerated):**
-
-```tsx
-function LoadingSpinner() {
-  return (
-    <div className="animate-spin">
-      <svg 
-        width="24" 
-        height="24" 
-        viewBox="0 0 24 24"
-      >
-        <circle cx="12" cy="12" r="10" stroke="currentColor" />
-      </svg>
-    </div>
-  )
-}
-```
-
-This applies to all CSS transforms and transitions (`transform`, `opacity`, `translate`, `scale`, `rotate`). The wrapper div allows browsers to use GPU acceleration for smoother animations.

.claude/skills/react-best-practices/rules/rendering-conditional-render.md

@@ -1,40 +0,0 @@
----
-title: Use Explicit Conditional Rendering
-impact: LOW
-impactDescription: prevents rendering 0 or NaN
-tags: rendering, conditional, jsx, falsy-values
----
-
-## Use Explicit Conditional Rendering
-
-Use explicit ternary operators (`? :`) instead of `&&` for conditional rendering when the condition can be `0`, `NaN`, or other falsy values that render.
-
-**Incorrect (renders "0" when count is 0):**
-
-```tsx
-function Badge({ count }: { count: number }) {
-  return (
-    <div>
-      {count && <span className="badge">{count}</span>}
-    </div>
-  )
-}
-
-// When count = 0, renders: <div>0</div>
-// When count = 5, renders: <div><span class="badge">5</span></div>
-```
-
-**Correct (renders nothing when count is 0):**
-
-```tsx
-function Badge({ count }: { count: number }) {
-  return (
-    <div>
-      {count > 0 ? <span className="badge">{count}</span> : null}
-    </div>
-  )
-}
-
-// When count = 0, renders: <div></div>
-// When count = 5, renders: <div><span class="badge">5</span></div>
-```

.claude/skills/react-best-practices/rules/rendering-content-visibility.md

@@ -1,38 +0,0 @@
----
-title: CSS content-visibility for Long Lists
-impact: HIGH
-impactDescription: faster initial render
-tags: rendering, css, content-visibility, long-lists
----
-
-## CSS content-visibility for Long Lists
-
-Apply `content-visibility: auto` to defer off-screen rendering.
-
-**CSS:**
-
-```css
-.message-item {
-  content-visibility: auto;
-  contain-intrinsic-size: 0 80px;
-}
-```
-
-**Example:**
-
-```tsx
-function MessageList({ messages }: { messages: Message[] }) {
-  return (
-    <div className="overflow-y-auto h-screen">
-      {messages.map(msg => (
-        <div key={msg.id} className="message-item">
-          <Avatar user={msg.author} />
-          <div>{msg.content}</div>
-        </div>
-      ))}
-    </div>
-  )
-}
-```
-
-For 1000 messages, browser skips layout/paint for ~990 off-screen items (10× faster initial render).

.claude/skills/react-best-practices/rules/rendering-hoist-jsx.md

@@ -1,46 +0,0 @@
----
-title: Hoist Static JSX Elements
-impact: LOW
-impactDescription: avoids re-creation
-tags: rendering, jsx, static, optimization
----
-
-## Hoist Static JSX Elements
-
-Extract static JSX outside components to avoid re-creation.
-
-**Incorrect (recreates element every render):**
-
-```tsx
-function LoadingSkeleton() {
-  return <div className="animate-pulse h-20 bg-gray-200" />
-}
-
-function Container() {
-  return (
-    <div>
-      {loading && <LoadingSkeleton />}
-    </div>
-  )
-}
-```
-
-**Correct (reuses same element):**
-
-```tsx
-const loadingSkeleton = (
-  <div className="animate-pulse h-20 bg-gray-200" />
-)
-
-function Container() {
-  return (
-    <div>
-      {loading && loadingSkeleton}
-    </div>
-  )
-}
-```
-
-This is especially helpful for large and static SVG nodes, which can be expensive to recreate on every render.
-
-**Note:** If your project has [React Compiler](https://react.dev/learn/react-compiler) enabled, the compiler automatically hoists static JSX elements and optimizes component re-renders, making manual hoisting unnecessary.

.claude/skills/react-best-practices/rules/rendering-svg-precision.md

@@ -1,28 +0,0 @@
----
-title: Optimize SVG Precision
-impact: LOW
-impactDescription: reduces file size
-tags: rendering, svg, optimization, svgo
----
-
-## Optimize SVG Precision
-
-Reduce SVG coordinate precision to decrease file size. The optimal precision depends on the viewBox size, but in general reducing precision should be considered.
-
-**Incorrect (excessive precision):**
-
-```svg
-<path d="M 10.293847 20.847362 L 30.938472 40.192837" />
-```
-
-**Correct (1 decimal place):**
-
-```svg
-<path d="M 10.3 20.8 L 30.9 40.2" />
-```
-
-**Automate with SVGO:**
-
-```bash
-npx svgo --precision=1 --multipass icon.svg
-```

.claude/skills/react-best-practices/rules/rendering-usetransition-loading.md

@@ -1,75 +0,0 @@
----
-title: Use useTransition Over Manual Loading States
-impact: LOW
-impactDescription: reduces re-renders and improves code clarity
-tags: rendering, transitions, useTransition, loading, state
----
-
-## Use useTransition Over Manual Loading States
-
-Use `useTransition` instead of manual `useState` for loading states. This provides built-in `isPending` state and automatically manages transitions.
-
-**Incorrect (manual loading state):**
-
-```tsx
-function SearchResults() {
-  const [query, setQuery] = useState('')
-  const [results, setResults] = useState([])
-  const [isLoading, setIsLoading] = useState(false)
-
-  const handleSearch = async (value: string) => {
-    setIsLoading(true)
-    setQuery(value)
-    const data = await fetchResults(value)
-    setResults(data)
-    setIsLoading(false)
-  }
-
-  return (
-    <>
-      <input onChange={(e) => handleSearch(e.target.value)} />
-      {isLoading && <Spinner />}
-      <ResultsList results={results} />
-    </>
-  )
-}
-```
-
-**Correct (useTransition with built-in pending state):**
-
-```tsx
-import { useTransition, useState } from 'react'
-
-function SearchResults() {
-  const [query, setQuery] = useState('')
-  const [results, setResults] = useState([])
-  const [isPending, startTransition] = useTransition()
-
-  const handleSearch = (value: string) => {
-    setQuery(value) // Update input immediately
-    
-    startTransition(async () => {
-      // Fetch and update results
-      const data = await fetchResults(value)
-      setResults(data)
-    })
-  }
-
-  return (
-    <>
-      <input onChange={(e) => handleSearch(e.target.value)} />
-      {isPending && <Spinner />}
-      <ResultsList results={results} />
-    </>
-  )
-}
-```
-
-**Benefits:**
-
-- **Automatic pending state**: No need to manually manage `setIsLoading(true/false)`
-- **Error resilience**: Pending state correctly resets even if the transition throws
-- **Better responsiveness**: Keeps the UI responsive during updates
-- **Interrupt handling**: New transitions automatically cancel pending ones
-
-Reference: [useTransition](https://react.dev/reference/react/useTransition)

.claude/skills/react-best-practices/rules/rerender-defer-reads.md

@@ -1,39 +0,0 @@
----
-title: Defer State Reads to Usage Point
-impact: MEDIUM
-impactDescription: avoids unnecessary subscriptions
-tags: rerender, searchParams, localStorage, optimization
----
-
-## Defer State Reads to Usage Point
-
-Don't subscribe to dynamic state (searchParams, localStorage) if you only read it inside callbacks.
-
-**Incorrect (subscribes to all searchParams changes):**
-
-```tsx
-function ShareButton({ chatId }: { chatId: string }) {
-  const searchParams = useSearchParams()
-
-  const handleShare = () => {
-    const ref = searchParams.get('ref')
-    shareChat(chatId, { ref })
-  }
-
-  return <button onClick={handleShare}>Share</button>
-}
-```
-
-**Correct (reads on demand, no subscription):**
-
-```tsx
-function ShareButton({ chatId }: { chatId: string }) {
-  const handleShare = () => {
-    const params = new URLSearchParams(window.location.search)
-    const ref = params.get('ref')
-    shareChat(chatId, { ref })
-  }
-
-  return <button onClick={handleShare}>Share</button>
-}
-```

.claude/skills/react-best-practices/rules/rerender-dependencies.md

@@ -1,45 +0,0 @@
----
-title: Narrow Effect Dependencies
-impact: LOW
-impactDescription: minimizes effect re-runs
-tags: rerender, useEffect, dependencies, optimization
----
-
-## Narrow Effect Dependencies
-
-Specify primitive dependencies instead of objects to minimize effect re-runs.
-
-**Incorrect (re-runs on any user field change):**
-
-```tsx
-useEffect(() => {
-  console.log(user.id)
-}, [user])
-```
-
-**Correct (re-runs only when id changes):**
-
-```tsx
-useEffect(() => {
-  console.log(user.id)
-}, [user.id])
-```
-
-**For derived state, compute outside effect:**
-
-```tsx
-// Incorrect: runs on width=767, 766, 765...
-useEffect(() => {
-  if (width < 768) {
-    enableMobileMode()
-  }
-}, [width])
-
-// Correct: runs only on boolean transition
-const isMobile = width < 768
-useEffect(() => {
-  if (isMobile) {
-    enableMobileMode()
-  }
-}, [isMobile])
-```

.claude/skills/react-best-practices/rules/rerender-derived-state-no-effect.md

@@ -1,40 +0,0 @@
----
-title: Calculate Derived State During Rendering
-impact: MEDIUM
-impactDescription: avoids redundant renders and state drift
-tags: rerender, derived-state, useEffect, state
----
-
-## Calculate Derived State During Rendering
-
-If a value can be computed from current props/state, do not store it in state or update it in an effect. Derive it during render to avoid extra renders and state drift. Do not set state in effects solely in response to prop changes; prefer derived values or keyed resets instead.
-
-**Incorrect (redundant state and effect):**
-
-```tsx
-function Form() {
-  const [firstName, setFirstName] = useState('First')
-  const [lastName, setLastName] = useState('Last')
-  const [fullName, setFullName] = useState('')
-
-  useEffect(() => {
-    setFullName(firstName + ' ' + lastName)
-  }, [firstName, lastName])
-
-  return <p>{fullName}</p>
-}
-```
-
-**Correct (derive during render):**
-
-```tsx
-function Form() {
-  const [firstName, setFirstName] = useState('First')
-  const [lastName, setLastName] = useState('Last')
-  const fullName = firstName + ' ' + lastName
-
-  return <p>{fullName}</p>
-}
-```
-
-References: [You Might Not Need an Effect](https://react.dev/learn/you-might-not-need-an-effect)

.claude/skills/react-best-practices/rules/rerender-derived-state.md

@@ -1,29 +0,0 @@
----
-title: Subscribe to Derived State
-impact: MEDIUM
-impactDescription: reduces re-render frequency
-tags: rerender, derived-state, media-query, optimization
----
-
-## Subscribe to Derived State
-
-Subscribe to derived boolean state instead of continuous values to reduce re-render frequency.
-
-**Incorrect (re-renders on every pixel change):**
-
-```tsx
-function Sidebar() {
-  const width = useWindowWidth()  // updates continuously
-  const isMobile = width < 768
-  return <nav className={isMobile ? 'mobile' : 'desktop'} />
-}
-```
-
-**Correct (re-renders only when boolean changes):**
-
-```tsx
-function Sidebar() {
-  const isMobile = useMediaQuery('(max-width: 767px)')
-  return <nav className={isMobile ? 'mobile' : 'desktop'} />
-}
-```

.claude/skills/react-best-practices/rules/rerender-functional-setstate.md

@@ -1,74 +0,0 @@
----
-title: Use Functional setState Updates
-impact: MEDIUM
-impactDescription: prevents stale closures and unnecessary callback recreations
-tags: react, hooks, useState, useCallback, callbacks, closures
----
-
-## Use Functional setState Updates
-
-When updating state based on the current state value, use the functional update form of setState instead of directly referencing the state variable. This prevents stale closures, eliminates unnecessary dependencies, and creates stable callback references.
-
-**Incorrect (requires state as dependency):**
-
-```tsx
-function TodoList() {
-  const [items, setItems] = useState(initialItems)
-  
-  // Callback must depend on items, recreated on every items change
-  const addItems = useCallback((newItems: Item[]) => {
-    setItems([...items, ...newItems])
-  }, [items])  // ❌ items dependency causes recreations
-  
-  // Risk of stale closure if dependency is forgotten
-  const removeItem = useCallback((id: string) => {
-    setItems(items.filter(item => item.id !== id))
-  }, [])  // ❌ Missing items dependency - will use stale items!
-  
-  return <ItemsEditor items={items} onAdd={addItems} onRemove={removeItem} />
-}
-```
-
-The first callback is recreated every time `items` changes, which can cause child components to re-render unnecessarily. The second callback has a stale closure bug—it will always reference the initial `items` value.
-
-**Correct (stable callbacks, no stale closures):**
-
-```tsx
-function TodoList() {
-  const [items, setItems] = useState(initialItems)
-  
-  // Stable callback, never recreated
-  const addItems = useCallback((newItems: Item[]) => {
-    setItems(curr => [...curr, ...newItems])
-  }, [])  // ✅ No dependencies needed
-  
-  // Always uses latest state, no stale closure risk
-  const removeItem = useCallback((id: string) => {
-    setItems(curr => curr.filter(item => item.id !== id))
-  }, [])  // ✅ Safe and stable
-  
-  return <ItemsEditor items={items} onAdd={addItems} onRemove={removeItem} />
-}
-```
-
-**Benefits:**
-
-1. **Stable callback references** - Callbacks don't need to be recreated when state changes
-2. **No stale closures** - Always operates on the latest state value
-3. **Fewer dependencies** - Simplifies dependency arrays and reduces memory leaks
-4. **Prevents bugs** - Eliminates the most common source of React closure bugs
-
-**When to use functional updates:**
-
-- Any setState that depends on the current state value
-- Inside useCallback/useMemo when state is needed
-- Event handlers that reference state
-- Async operations that update state
-
-**When direct updates are fine:**
-
-- Setting state to a static value: `setCount(0)`
-- Setting state from props/arguments only: `setName(newName)`
-- State doesn't depend on previous value
-
-**Note:** If your project has [React Compiler](https://react.dev/learn/react-compiler) enabled, the compiler can automatically optimize some cases, but functional updates are still recommended for correctness and to prevent stale closure bugs.

.claude/skills/react-best-practices/rules/rerender-lazy-state-init.md

@@ -1,58 +0,0 @@
----
-title: Use Lazy State Initialization
-impact: MEDIUM
-impactDescription: wasted computation on every render
-tags: react, hooks, useState, performance, initialization
----
-
-## Use Lazy State Initialization
-
-Pass a function to `useState` for expensive initial values. Without the function form, the initializer runs on every render even though the value is only used once.
-
-**Incorrect (runs on every render):**
-
-```tsx
-function FilteredList({ items }: { items: Item[] }) {
-  // buildSearchIndex() runs on EVERY render, even after initialization
-  const [searchIndex, setSearchIndex] = useState(buildSearchIndex(items))
-  const [query, setQuery] = useState('')
-  
-  // When query changes, buildSearchIndex runs again unnecessarily
-  return <SearchResults index={searchIndex} query={query} />
-}
-
-function UserProfile() {
-  // JSON.parse runs on every render
-  const [settings, setSettings] = useState(
-    JSON.parse(localStorage.getItem('settings') || '{}')
-  )
-  
-  return <SettingsForm settings={settings} onChange={setSettings} />
-}
-```
-
-**Correct (runs only once):**
-
-```tsx
-function FilteredList({ items }: { items: Item[] }) {
-  // buildSearchIndex() runs ONLY on initial render
-  const [searchIndex, setSearchIndex] = useState(() => buildSearchIndex(items))
-  const [query, setQuery] = useState('')
-  
-  return <SearchResults index={searchIndex} query={query} />
-}
-
-function UserProfile() {
-  // JSON.parse runs only on initial render
-  const [settings, setSettings] = useState(() => {
-    const stored = localStorage.getItem('settings')
-    return stored ? JSON.parse(stored) : {}
-  })
-  
-  return <SettingsForm settings={settings} onChange={setSettings} />
-}
-```
-
-Use lazy initialization when computing initial values from localStorage/sessionStorage, building data structures (indexes, maps), reading from the DOM, or performing heavy transformations.
-
-For simple primitives (`useState(0)`), direct references (`useState(props.value)`), or cheap literals (`useState({})`), the function form is unnecessary.

.claude/skills/react-best-practices/rules/rerender-move-effect-to-event.md

@@ -1,45 +0,0 @@
----
-title: Put Interaction Logic in Event Handlers
-impact: MEDIUM
-impactDescription: avoids effect re-runs and duplicate side effects
-tags: rerender, useEffect, events, side-effects, dependencies
----
-
-## Put Interaction Logic in Event Handlers
-
-If a side effect is triggered by a specific user action (submit, click, drag), run it in that event handler. Do not model the action as state + effect; it makes effects re-run on unrelated changes and can duplicate the action.
-
-**Incorrect (event modeled as state + effect):**
-
-```tsx
-function Form() {
-  const [submitted, setSubmitted] = useState(false)
-  const theme = useContext(ThemeContext)
-
-  useEffect(() => {
-    if (submitted) {
-      post('/api/register')
-      showToast('Registered', theme)
-    }
-  }, [submitted, theme])
-
-  return <button onClick={() => setSubmitted(true)}>Submit</button>
-}
-```
-
-**Correct (do it in the handler):**
-
-```tsx
-function Form() {
-  const theme = useContext(ThemeContext)
-
-  function handleSubmit() {
-    post('/api/register')
-    showToast('Registered', theme)
-  }
-
-  return <button onClick={handleSubmit}>Submit</button>
-}
-```
-
-Reference: [Should this code move to an event handler?](https://react.dev/learn/removing-effect-dependencies#should-this-code-move-to-an-event-handler)

.claude/skills/react-best-practices/rules/rerender-transitions.md

@@ -1,40 +0,0 @@
----
-title: Use Transitions for Non-Urgent Updates
-impact: MEDIUM
-impactDescription: maintains UI responsiveness
-tags: rerender, transitions, startTransition, performance
----
-
-## Use Transitions for Non-Urgent Updates
-
-Mark frequent, non-urgent state updates as transitions to maintain UI responsiveness.
-
-**Incorrect (blocks UI on every scroll):**
-
-```tsx
-function ScrollTracker() {
-  const [scrollY, setScrollY] = useState(0)
-  useEffect(() => {
-    const handler = () => setScrollY(window.scrollY)
-    window.addEventListener('scroll', handler, { passive: true })
-    return () => window.removeEventListener('scroll', handler)
-  }, [])
-}
-```
-
-**Correct (non-blocking updates):**
-
-```tsx
-import { startTransition } from 'react'
-
-function ScrollTracker() {
-  const [scrollY, setScrollY] = useState(0)
-  useEffect(() => {
-    const handler = () => {
-      startTransition(() => setScrollY(window.scrollY))
-    }
-    window.addEventListener('scroll', handler, { passive: true })
-    return () => window.removeEventListener('scroll', handler)
-  }, [])
-}
-```

.claude/skills/react-best-practices/rules/rerender-use-ref-transient-values.md

@@ -1,73 +0,0 @@
----
-title: Use useRef for Transient Values
-impact: MEDIUM
-impactDescription: avoids unnecessary re-renders on frequent updates
-tags: rerender, useref, state, performance
----
-
-## Use useRef for Transient Values
-
-When a value changes frequently and you don't want a re-render on every update (e.g., mouse trackers, intervals, transient flags), store it in `useRef` instead of `useState`. Keep component state for UI; use refs for temporary DOM-adjacent values. Updating a ref does not trigger a re-render.
-
-**Incorrect (renders every update):**
-
-```tsx
-function Tracker() {
-  const [lastX, setLastX] = useState(0)
-
-  useEffect(() => {
-    const onMove = (e: MouseEvent) => setLastX(e.clientX)
-    window.addEventListener('mousemove', onMove)
-    return () => window.removeEventListener('mousemove', onMove)
-  }, [])
-
-  return (
-    <div
-      style={{
-        position: 'fixed',
-        top: 0,
-        left: lastX,
-        width: 8,
-        height: 8,
-        background: 'black',
-      }}
-    />
-  )
-}
-```
-
-**Correct (no re-render for tracking):**
-
-```tsx
-function Tracker() {
-  const lastXRef = useRef(0)
-  const dotRef = useRef<HTMLDivElement>(null)
-
-  useEffect(() => {
-    const onMove = (e: MouseEvent) => {
-      lastXRef.current = e.clientX
-      const node = dotRef.current
-      if (node) {
-        node.style.transform = `translateX(${e.clientX}px)`
-      }
-    }
-    window.addEventListener('mousemove', onMove)
-    return () => window.removeEventListener('mousemove', onMove)
-  }, [])
-
-  return (
-    <div
-      ref={dotRef}
-      style={{
-        position: 'fixed',
-        top: 0,
-        left: 0,
-        width: 8,
-        height: 8,
-        background: 'black',
-        transform: 'translateX(0px)',
-      }}
-    />
-  )
-}
-```

.claude/skills/react-composition-patterns/SKILL.md

@@ -1,73 +0,0 @@
----
-name: react-composition-patterns
-description:
-  React composition patterns that scale. Use when refactoring components with
-  boolean prop proliferation, building flexible component libraries, or
-  designing reusable APIs. Triggers on tasks involving compound components,
-  render props, context providers, or component architecture.
----
-
-# React Composition Patterns
-
-Composition patterns for building flexible, maintainable React components. Avoid
-boolean prop proliferation by using compound components, lifting state, and
-composing internals. These patterns make codebases easier for both humans and AI
-agents to work with as they scale.
-
-## When to Apply
-
-Reference these guidelines when:
-
-- Refactoring components with many boolean props
-- Building reusable component libraries
-- Designing flexible component APIs
-- Reviewing component architecture
-- Working with compound components or context providers
-
-## Rule Categories by Priority
-
-| Priority | Category                | Impact | Prefix          |
-| -------- | ----------------------- | ------ | --------------- |
-| 1        | Component Architecture  | HIGH   | `architecture-` |
-| 2        | State Management        | MEDIUM | `state-`        |
-| 3        | Implementation Patterns | MEDIUM | `patterns-`     |
-
-## Quick Reference
-
-### 1. Component Architecture (HIGH)
-
-- `architecture-avoid-boolean-props` - Don't add boolean props to customize
-  behavior; use composition
-- `architecture-compound-components` - Structure complex components with shared
-  context
-
-### 2. State Management (MEDIUM)
-
-- `state-decouple-implementation` - Provider is the only place that knows how
-  state is managed
-- `state-context-interface` - Define generic interface with state, actions, meta
-  for dependency injection
-- `state-lift-state` - Move state into provider components for sibling access
-
-### 3. Implementation Patterns (MEDIUM)
-
-- `patterns-explicit-variants` - Create explicit variant components instead of
-  boolean modes
-- `patterns-children-over-render-props` - Use children for composition instead
-  of renderX props
-
-## How to Use
-
-Read individual rule files for detailed explanations and code examples:
-
-```
-rules/architecture-avoid-boolean-props.md
-rules/state-context-interface.md
-```
-
-Each rule file contains:
-
-- Brief explanation of why it matters
-- Incorrect code example with explanation
-- Correct code example with explanation
-- Additional context and references

.claude/skills/react-composition-patterns/rules/architecture-avoid-boolean-props.md

@@ -1,100 +0,0 @@
----
-title: Avoid Boolean Prop Proliferation
-impact: CRITICAL
-impactDescription: prevents unmaintainable component variants
-tags: composition, props, architecture
----
-
-## Avoid Boolean Prop Proliferation
-
-Don't add boolean props like `isThread`, `isEditing`, `isDMThread` to customize
-component behavior. Each boolean doubles possible states and creates
-unmaintainable conditional logic. Use composition instead.
-
-**Incorrect (boolean props create exponential complexity):**
-
-```tsx
-function Composer({
-  onSubmit,
-  isThread,
-  channelId,
-  isDMThread,
-  dmId,
-  isEditing,
-  isForwarding,
-}: Props) {
-  return (
-    <form>
-      <Header />
-      <Input />
-      {isDMThread ? (
-        <AlsoSendToDMField id={dmId} />
-      ) : isThread ? (
-        <AlsoSendToChannelField id={channelId} />
-      ) : null}
-      {isEditing ? (
-        <EditActions />
-      ) : isForwarding ? (
-        <ForwardActions />
-      ) : (
-        <DefaultActions />
-      )}
-      <Footer onSubmit={onSubmit} />
-    </form>
-  )
-}
-```
-
-**Correct (composition eliminates conditionals):**
-
-```tsx
-// Channel composer
-function ChannelComposer() {
-  return (
-    <Composer.Frame>
-      <Composer.Header />
-      <Composer.Input />
-      <Composer.Footer>
-        <Composer.Attachments />
-        <Composer.Formatting />
-        <Composer.Emojis />
-        <Composer.Submit />
-      </Composer.Footer>
-    </Composer.Frame>
-  )
-}
-
-// Thread composer - adds "also send to channel" field
-function ThreadComposer({ channelId }: { channelId: string }) {
-  return (
-    <Composer.Frame>
-      <Composer.Header />
-      <Composer.Input />
-      <AlsoSendToChannelField id={channelId} />
-      <Composer.Footer>
-        <Composer.Formatting />
-        <Composer.Emojis />
-        <Composer.Submit />
-      </Composer.Footer>
-    </Composer.Frame>
-  )
-}
-
-// Edit composer - different footer actions
-function EditComposer() {
-  return (
-    <Composer.Frame>
-      <Composer.Input />
-      <Composer.Footer>
-        <Composer.Formatting />
-        <Composer.Emojis />
-        <Composer.CancelEdit />
-        <Composer.SaveEdit />
-      </Composer.Footer>
-    </Composer.Frame>
-  )
-}
-```
-
-Each variant is explicit about what it renders. We can share internals without
-sharing a single monolithic parent.

.claude/skills/react-composition-patterns/rules/architecture-compound-components.md

@@ -1,112 +0,0 @@
----
-title: Use Compound Components
-impact: HIGH
-impactDescription: enables flexible composition without prop drilling
-tags: composition, compound-components, architecture
----
-
-## Use Compound Components
-
-Structure complex components as compound components with a shared context. Each
-subcomponent accesses shared state via context, not props. Consumers compose the
-pieces they need.
-
-**Incorrect (monolithic component with render props):**
-
-```tsx
-function Composer({
-  renderHeader,
-  renderFooter,
-  renderActions,
-  showAttachments,
-  showFormatting,
-  showEmojis,
-}: Props) {
-  return (
-    <form>
-      {renderHeader?.()}
-      <Input />
-      {showAttachments && <Attachments />}
-      {renderFooter ? (
-        renderFooter()
-      ) : (
-        <Footer>
-          {showFormatting && <Formatting />}
-          {showEmojis && <Emojis />}
-          {renderActions?.()}
-        </Footer>
-      )}
-    </form>
-  )
-}
-```
-
-**Correct (compound components with shared context):**
-
-```tsx
-const ComposerContext = createContext<ComposerContextValue | null>(null)
-
-function ComposerProvider({ children, state, actions, meta }: ProviderProps) {
-  return (
-    <ComposerContext value={{ state, actions, meta }}>
-      {children}
-    </ComposerContext>
-  )
-}
-
-function ComposerFrame({ children }: { children: React.ReactNode }) {
-  return <form>{children}</form>
-}
-
-function ComposerInput() {
-  const {
-    state,
-    actions: { update },
-    meta: { inputRef },
-  } = use(ComposerContext)
-  return (
-    <TextInput
-      ref={inputRef}
-      value={state.input}
-      onChangeText={(text) => update((s) => ({ ...s, input: text }))}
-    />
-  )
-}
-
-function ComposerSubmit() {
-  const {
-    actions: { submit },
-  } = use(ComposerContext)
-  return <Button onPress={submit}>Send</Button>
-}
-
-// Export as compound component
-const Composer = {
-  Provider: ComposerProvider,
-  Frame: ComposerFrame,
-  Input: ComposerInput,
-  Submit: ComposerSubmit,
-  Header: ComposerHeader,
-  Footer: ComposerFooter,
-  Attachments: ComposerAttachments,
-  Formatting: ComposerFormatting,
-  Emojis: ComposerEmojis,
-}
-```
-
-**Usage:**
-
-```tsx
-<Composer.Provider state={state} actions={actions} meta={meta}>
-  <Composer.Frame>
-    <Composer.Header />
-    <Composer.Input />
-    <Composer.Footer>
-      <Composer.Formatting />
-      <Composer.Submit />
-    </Composer.Footer>
-  </Composer.Frame>
-</Composer.Provider>
-```
-
-Consumers explicitly compose exactly what they need. No hidden conditionals. And the state, actions and meta are dependency-injected by a parent provider, allowing multiple usages of the same component structure.

.claude/skills/react-composition-patterns/rules/patterns-children-over-render-props.md

@@ -1,87 +0,0 @@
----
-title: Prefer Composing Children Over Render Props
-impact: MEDIUM
-impactDescription: cleaner composition, better readability
-tags: composition, children, render-props
----
-
-## Prefer Children Over Render Props
-
-Use `children` for composition instead of `renderX` props. Children are more
-readable, compose naturally, and don't require understanding callback
-signatures.
-
-**Incorrect (render props):**
-
-```tsx
-function Composer({
-  renderHeader,
-  renderFooter,
-  renderActions,
-}: {
-  renderHeader?: () => React.ReactNode
-  renderFooter?: () => React.ReactNode
-  renderActions?: () => React.ReactNode
-}) {
-  return (
-    <form>
-      {renderHeader?.()}
-      <Input />
-      {renderFooter ? renderFooter() : <DefaultFooter />}
-      {renderActions?.()}
-    </form>
-  )
-}
-
-// Usage is awkward and inflexible
-return (
-  <Composer
-    renderHeader={() => <CustomHeader />}
-    renderFooter={() => (
-      <>
-        <Formatting />
-        <Emojis />
-      </>
-    )}
-    renderActions={() => <SubmitButton />}
-  />
-)
-```
-
-**Correct (compound components with children):**
-
-```tsx
-function ComposerFrame({ children }: { children: React.ReactNode }) {
-  return <form>{children}</form>
-}
-
-function ComposerFooter({ children }: { children: React.ReactNode }) {
-  return <footer className='flex'>{children}</footer>
-}
-
-// Usage is flexible
-return (
-  <Composer.Frame>
-    <CustomHeader />
-    <Composer.Input />
-    <Composer.Footer>
-      <Composer.Formatting />
-      <Composer.Emojis />
-      <SubmitButton />
-    </Composer.Footer>
-  </Composer.Frame>
-)
-```
-
-**When render props are appropriate:**
-
-```tsx
-// Render props work well when you need to pass data back
-<List
-  data={items}
-  renderItem={({ item, index }) => <Item item={item} index={index} />}
-/>
-```
-
-Use render props when the parent needs to provide data or state to the child.
-Use children when composing static structure.

.claude/skills/react-composition-patterns/rules/patterns-explicit-variants.md

@@ -1,100 +0,0 @@
----
-title: Create Explicit Component Variants
-impact: MEDIUM
-impactDescription: self-documenting code, no hidden conditionals
-tags: composition, variants, architecture
----
-
-## Create Explicit Component Variants
-
-Instead of one component with many boolean props, create explicit variant
-components. Each variant composes the pieces it needs. The code documents
-itself.
-
-**Incorrect (one component, many modes):**
-
-```tsx
-// What does this component actually render?
-<Composer
-  isThread
-  isEditing={false}
-  channelId='abc'
-  showAttachments
-  showFormatting={false}
-/>
-```
-
-**Correct (explicit variants):**
-
-```tsx
-// Immediately clear what this renders
-<ThreadComposer channelId="abc" />
-
-// Or
-<EditMessageComposer messageId="xyz" />
-
-// Or
-<ForwardMessageComposer messageId="123" />
-```
-
-Each implementation is unique, explicit and self-contained. Yet they can each
-use shared parts.
-
-**Implementation:**
-
-```tsx
-function ThreadComposer({ channelId }: { channelId: string }) {
-  return (
-    <ThreadProvider channelId={channelId}>
-      <Composer.Frame>
-        <Composer.Input />
-        <AlsoSendToChannelField channelId={channelId} />
-        <Composer.Footer>
-          <Composer.Formatting />
-          <Composer.Emojis />
-          <Composer.Submit />
-        </Composer.Footer>
-      </Composer.Frame>
-    </ThreadProvider>
-  )
-}
-
-function EditMessageComposer({ messageId }: { messageId: string }) {
-  return (
-    <EditMessageProvider messageId={messageId}>
-      <Composer.Frame>
-        <Composer.Input />
-        <Composer.Footer>
-          <Composer.Formatting />
-          <Composer.Emojis />
-          <Composer.CancelEdit />
-          <Composer.SaveEdit />
-        </Composer.Footer>
-      </Composer.Frame>
-    </EditMessageProvider>
-  )
-}
-
-function ForwardMessageComposer({ messageId }: { messageId: string }) {
-  return (
-    <ForwardMessageProvider messageId={messageId}>
-      <Composer.Frame>
-        <Composer.Input placeholder="Add a message, if you'd like." />
-        <Composer.Footer>
-          <Composer.Formatting />
-          <Composer.Emojis />
-          <Composer.Mentions />
-        </Composer.Footer>
-      </Composer.Frame>
-    </ForwardMessageProvider>
-  )
-}
-```
-
-Each variant is explicit about:
-
-- What provider/state it uses
-- What UI elements it includes
-- What actions are available
-
-No boolean prop combinations to reason about. No impossible states.

.claude/skills/react-composition-patterns/rules/state-context-interface.md

@@ -1,191 +0,0 @@
----
-title: Define Generic Context Interfaces for Dependency Injection
-impact: HIGH
-impactDescription: enables dependency-injectable state across use-cases
-tags: composition, context, state, typescript, dependency-injection
----
-
-## Define Generic Context Interfaces for Dependency Injection
-
-Define a **generic interface** for your component context with three parts:
-`state`, `actions`, and `meta`. This interface is a contract that any provider
-can implement—enabling the same UI components to work with completely different
-state implementations.
-
-**Core principle:** Lift state, compose internals, make state
-dependency-injectable.
-
-**Incorrect (UI coupled to specific state implementation):**
-
-```tsx
-function ComposerInput() {
-  // Tightly coupled to a specific hook
-  const { input, setInput } = useChannelComposerState()
-  return <TextInput value={input} onChangeText={setInput} />
-}
-```
-
-**Correct (generic interface enables dependency injection):**
-
-```tsx
-// Define a GENERIC interface that any provider can implement
-interface ComposerState {
-  input: string
-  attachments: Attachment[]
-  isSubmitting: boolean
-}
-
-interface ComposerActions {
-  update: (updater: (state: ComposerState) => ComposerState) => void
-  submit: () => void
-}
-
-interface ComposerMeta {
-  inputRef: React.RefObject<TextInput>
-}
-
-interface ComposerContextValue {
-  state: ComposerState
-  actions: ComposerActions
-  meta: ComposerMeta
-}
-
-const ComposerContext = createContext<ComposerContextValue | null>(null)
-```
-
-**UI components consume the interface, not the implementation:**
-
-```tsx
-function ComposerInput() {
-  const {
-    state,
-    actions: { update },
-    meta,
-  } = use(ComposerContext)
-
-  // This component works with ANY provider that implements the interface
-  return (
-    <TextInput
-      ref={meta.inputRef}
-      value={state.input}
-      onChangeText={(text) => update((s) => ({ ...s, input: text }))}
-    />
-  )
-}
-```
-
-**Different providers implement the same interface:**
-
-```tsx
-// Provider A: Local state for ephemeral forms
-function ForwardMessageProvider({ children }: { children: React.ReactNode }) {
-  const [state, setState] = useState(initialState)
-  const inputRef = useRef(null)
-  const submit = useForwardMessage()
-
-  return (
-    <ComposerContext
-      value={{
-        state,
-        actions: { update: setState, submit },
-        meta: { inputRef },
-      }}
-    >
-      {children}
-    </ComposerContext>
-  )
-}
-
-// Provider B: Global synced state for channels
-function ChannelProvider({ channelId, children }: Props) {
-  const { state, update, submit } = useGlobalChannel(channelId)
-  const inputRef = useRef(null)
-
-  return (
-    <ComposerContext
-      value={{
-        state,
-        actions: { update, submit },
-        meta: { inputRef },
-      }}
-    >
-      {children}
-    </ComposerContext>
-  )
-}
-```
-
-**The same composed UI works with both:**
-
-```tsx
-// Works with ForwardMessageProvider (local state)
-<ForwardMessageProvider>
-  <Composer.Frame>
-    <Composer.Input />
-    <Composer.Submit />
-  </Composer.Frame>
-</ForwardMessageProvider>
-
-// Works with ChannelProvider (global synced state)
-<ChannelProvider channelId="abc">
-  <Composer.Frame>
-    <Composer.Input />
-    <Composer.Submit />
-  </Composer.Frame>
-</ChannelProvider>
-```
-
-**Custom UI outside the component can access state and actions:**
-
-The provider boundary is what matters—not the visual nesting. Components that
-need shared state don't have to be inside the `Composer.Frame`. They just need
-to be within the provider.
-
-```tsx
-function ForwardMessageDialog() {
-  return (
-    <ForwardMessageProvider>
-      <Dialog>
-        {/* The composer UI */}
-        <Composer.Frame>
-          <Composer.Input placeholder="Add a message, if you'd like." />
-          <Composer.Footer>
-            <Composer.Formatting />
-            <Composer.Emojis />
-          </Composer.Footer>
-        </Composer.Frame>
-
-        {/* Custom UI OUTSIDE the composer, but INSIDE the provider */}
-        <MessagePreview />
-
-        {/* Actions at the bottom of the dialog */}
-        <DialogActions>
-          <CancelButton />
-          <ForwardButton />
-        </DialogActions>
-      </Dialog>
-    </ForwardMessageProvider>
-  )
-}
-
-// This button lives OUTSIDE Composer.Frame but can still submit based on its context!
-function ForwardButton() {
-  const {
-    actions: { submit },
-  } = use(ComposerContext)
-  return <Button onPress={submit}>Forward</Button>
-}
-
-// This preview lives OUTSIDE Composer.Frame but can read composer's state!
-function MessagePreview() {
-  const { state } = use(ComposerContext)
-  return <Preview message={state.input} attachments={state.attachments} />
-}
-```
-
-The `ForwardButton` and `MessagePreview` are not visually inside the composer
-box, but they can still access its state and actions. This is the power of
-lifting state into providers.
-
-The UI is reusable bits you compose together. The state is dependency-injected
-by the provider. Swap the provider, keep the UI.

.claude/skills/react-composition-patterns/rules/state-decouple-implementation.md

@@ -1,113 +0,0 @@
----
-title: Decouple State Management from UI
-impact: MEDIUM
-impactDescription: enables swapping state implementations without changing UI
-tags: composition, state, architecture
----
-
-## Decouple State Management from UI
-
-The provider component should be the only place that knows how state is managed.
-UI components consume the context interface—they don't know if state comes from
-useState, Zustand, or a server sync.
-
-**Incorrect (UI coupled to state implementation):**
-
-```tsx
-function ChannelComposer({ channelId }: { channelId: string }) {
-  // UI component knows about global state implementation
-  const state = useGlobalChannelState(channelId)
-  const { submit, updateInput } = useChannelSync(channelId)
-
-  return (
-    <Composer.Frame>
-      <Composer.Input
-        value={state.input}
-        onChange={(text) => sync.updateInput(text)}
-      />
-      <Composer.Submit onPress={() => sync.submit()} />
-    </Composer.Frame>
-  )
-}
-```
-
-**Correct (state management isolated in provider):**
-
-```tsx
-// Provider handles all state management details
-function ChannelProvider({
-  channelId,
-  children,
-}: {
-  channelId: string
-  children: React.ReactNode
-}) {
-  const { state, update, submit } = useGlobalChannel(channelId)
-  const inputRef = useRef(null)
-
-  return (
-    <Composer.Provider
-      state={state}
-      actions={{ update, submit }}
-      meta={{ inputRef }}
-    >
-      {children}
-    </Composer.Provider>
-  )
-}
-
-// UI component only knows about the context interface
-function ChannelComposer() {
-  return (
-    <Composer.Frame>
-      <Composer.Header />
-      <Composer.Input />
-      <Composer.Footer>
-        <Composer.Submit />
-      </Composer.Footer>
-    </Composer.Frame>
-  )
-}
-
-// Usage
-function Channel({ channelId }: { channelId: string }) {
-  return (
-    <ChannelProvider channelId={channelId}>
-      <ChannelComposer />
-    </ChannelProvider>
-  )
-}
-```
-
-**Different providers, same UI:**
-
-```tsx
-// Local state for ephemeral forms
-function ForwardMessageProvider({ children }) {
-  const [state, setState] = useState(initialState)
-  const forwardMessage = useForwardMessage()
-
-  return (
-    <Composer.Provider
-      state={state}
-      actions={{ update: setState, submit: forwardMessage }}
-    >
-      {children}
-    </Composer.Provider>
-  )
-}
-
-// Global synced state for channels
-function ChannelProvider({ channelId, children }) {
-  const { state, update, submit } = useGlobalChannel(channelId)
-
-  return (
-    <Composer.Provider state={state} actions={{ update, submit }}>
-      {children}
-    </Composer.Provider>
-  )
-}
-```
-
-The same `Composer.Input` component works with both providers because it only
-depends on the context interface, not the implementation.

.claude/skills/react-composition-patterns/rules/state-lift-state.md

@@ -1,125 +0,0 @@
----
-title: Lift State into Provider Components
-impact: HIGH
-impactDescription: enables state sharing outside component boundaries
-tags: composition, state, context, providers
----
-
-## Lift State into Provider Components
-
-Move state management into dedicated provider components. This allows sibling
-components outside the main UI to access and modify state without prop drilling
-or awkward refs.
-
-**Incorrect (state trapped inside component):**
-
-```tsx
-function ForwardMessageComposer() {
-  const [state, setState] = useState(initialState)
-  const forwardMessage = useForwardMessage()
-
-  return (
-    <Composer.Frame>
-      <Composer.Input />
-      <Composer.Footer />
-    </Composer.Frame>
-  )
-}
-
-// Problem: How does this button access composer state?
-function ForwardMessageDialog() {
-  return (
-    <Dialog>
-      <ForwardMessageComposer />
-      <MessagePreview /> {/* Needs composer state */}
-      <DialogActions>
-        <CancelButton />
-        <ForwardButton /> {/* Needs to call submit */}
-      </DialogActions>
-    </Dialog>
-  )
-}
-```
-
-**Incorrect (useEffect to sync state up):**
-
-```tsx
-function ForwardMessageDialog() {
-  const [input, setInput] = useState('')
-  return (
-    <Dialog>
-      <ForwardMessageComposer onInputChange={setInput} />
-      <MessagePreview input={input} />
-    </Dialog>
-  )
-}
-
-function ForwardMessageComposer({ onInputChange }) {
-  const [state, setState] = useState(initialState)
-  useEffect(() => {
-    onInputChange(state.input) // Sync on every change 😬
-  }, [state.input])
-}
-```
-
-**Incorrect (reading state from ref on submit):**
-
-```tsx
-function ForwardMessageDialog() {
-  const stateRef = useRef(null)
-  return (
-    <Dialog>
-      <ForwardMessageComposer stateRef={stateRef} />
-      <ForwardButton onPress={() => submit(stateRef.current)} />
-    </Dialog>
-  )
-}
-```
-
-**Correct (state lifted to provider):**
-
-```tsx
-function ForwardMessageProvider({ children }: { children: React.ReactNode }) {
-  const [state, setState] = useState(initialState)
-  const forwardMessage = useForwardMessage()
-  const inputRef = useRef(null)
-
-  return (
-    <Composer.Provider
-      state={state}
-      actions={{ update: setState, submit: forwardMessage }}
-      meta={{ inputRef }}
-    >
-      {children}
-    </Composer.Provider>
-  )
-}
-
-function ForwardMessageDialog() {
-  return (
-    <ForwardMessageProvider>
-      <Dialog>
-        <ForwardMessageComposer />
-        <MessagePreview /> {/* Custom components can access state and actions */}
-        <DialogActions>
-          <CancelButton />
-          <ForwardButton /> {/* Custom components can access state and actions */}
-        </DialogActions>
-      </Dialog>
-    </ForwardMessageProvider>
-  )
-}
-
-function ForwardButton() {
-  const { actions } = use(Composer.Context)
-  return <Button onPress={actions.submit}>Forward</Button>
-}
-```
-
-The ForwardButton lives outside the Composer.Frame but still has access to the
-submit action because it's within the provider. Even though it's a one-off
-component, it can still access the composer's state and actions from outside the
-UI itself.
-
-**Key insight:** Components that need shared state don't have to be visually
-nested inside each other—they just need to be within the same provider.

.claude/skills/web-design-guidelines/SKILL.md

@@ -1,19 +0,0 @@
----
-name: web-design-guidelines
-description: Review UI code for Web Interface Guidelines compliance. Use when asked to "review my UI", "check accessibility", "audit design", "review UX", or "check my site against best practices".
-argument-hint: <file-or-pattern>
----
-
-# Web Interface Guidelines
-
-Review files for compliance with Web Interface Guidelines.
-
-## Usage
-
-When a user provides a file or pattern argument:
-1. Read guidelines from rules/accessibility.md
-2. Read the specified files
-3. Apply all rules from the fetched guidelines
-4. Output findings using the format specified in the guidelines
-
-If no files specified, ask the user which files to review.

.claude/skills/web-design-guidelines/rules/accessibility.md

@@ -1,179 +0,0 @@
----
-description: Review UI code for Web Interface Guidelines compliance
-argument-hint: <file-or-pattern>
----
-
-# Web Interface Guidelines
-
-Review these files for compliance: $ARGUMENTS
-
-Read files, check against rules below. Output concise but comprehensive—sacrifice grammar for brevity. High signal-to-noise.
-
-## Rules
-
-### Accessibility
-
-- Icon-only buttons need `aria-label`
-- Form controls need `<label>` or `aria-label`
-- Interactive elements need keyboard handlers (`onKeyDown`/`onKeyUp`)
-- `<button>` for actions, `<a>`/`<Link>` for navigation (not `<div onClick>`)
-- Images need `alt` (or `alt=""` if decorative)
-- Decorative icons need `aria-hidden="true"`
-- Async updates (toasts, validation) need `aria-live="polite"`
-- Use semantic HTML (`<button>`, `<a>`, `<label>`, `<table>`) before ARIA
-- Headings hierarchical `<h1>`–`<h6>`; include skip link for main content
-- `scroll-margin-top` on heading anchors
-
-### Focus States
-
-- Interactive elements need visible focus: `focus-visible:ring-*` or equivalent
-- Never `outline-none` / `outline: none` without focus replacement
-- Use `:focus-visible` over `:focus` (avoid focus ring on click)
-- Group focus with `:focus-within` for compound controls
-
-### Forms
-
-- Inputs need `autocomplete` and meaningful `name`
-- Use correct `type` (`email`, `tel`, `url`, `number`) and `inputmode`
-- Never block paste (`onPaste` + `preventDefault`)
-- Labels clickable (`htmlFor` or wrapping control)
-- Disable spellcheck on emails, codes, usernames (`spellCheck={false}`)
-- Checkboxes/radios: label + control share single hit target (no dead zones)
-- Submit button stays enabled until request starts; spinner during request
-- Errors inline next to fields; focus first error on submit
-- Placeholders end with `…` and show example pattern
-- `autocomplete="off"` on non-auth fields to avoid password manager triggers
-- Warn before navigation with unsaved changes (`beforeunload` or router guard)
-
-### Animation
-
-- Honor `prefers-reduced-motion` (provide reduced variant or disable)
-- Animate `transform`/`opacity` only (compositor-friendly)
-- Never `transition: all`—list properties explicitly
-- Set correct `transform-origin`
-- SVG: transforms on `<g>` wrapper with `transform-box: fill-box; transform-origin: center`
-- Animations interruptible—respond to user input mid-animation
-
-### Typography
-
-- `…` not `...`
-- Curly quotes `"` `"` not straight `"`
-- Non-breaking spaces: `10&nbsp;MB`, `⌘&nbsp;K`, brand names
-- Loading states end with `…`: `"Loading…"`, `"Saving…"`
-- `font-variant-numeric: tabular-nums` for number columns/comparisons
-- Use `text-wrap: balance` or `text-pretty` on headings (prevents widows)
-
-### Content Handling
-
-- Text containers handle long content: `truncate`, `line-clamp-*`, or `break-words`
-- Flex children need `min-w-0` to allow text truncation
-- Handle empty states—don't render broken UI for empty strings/arrays
-- User-generated content: anticipate short, average, and very long inputs
-
-### Images
-
-- `<img>` needs explicit `width` and `height` (prevents CLS)
-- Below-fold images: `loading="lazy"`
-- Above-fold critical images: `priority` or `fetchpriority="high"`
-
-### Performance
-
-- Large lists (>50 items): virtualize (`virtua`, `content-visibility: auto`)
-- No layout reads in render (`getBoundingClientRect`, `offsetHeight`, `offsetWidth`, `scrollTop`)
-- Batch DOM reads/writes; avoid interleaving
-- Prefer uncontrolled inputs; controlled inputs must be cheap per keystroke
-- Add `<link rel="preconnect">` for CDN/asset domains
-- Critical fonts: `<link rel="preload" as="font">` with `font-display: swap`
-
-### Navigation & State
-
-- URL reflects state—filters, tabs, pagination, expanded panels in query params
-- Links use `<a>`/`<Link>` (Cmd/Ctrl+click, middle-click support)
-- Deep-link all stateful UI (if uses `useState`, consider URL sync via nuqs or similar)
-- Destructive actions need confirmation modal or undo window—never immediate
-
-### Touch & Interaction
-
-- `touch-action: manipulation` (prevents double-tap zoom delay)
-- `-webkit-tap-highlight-color` set intentionally
-- `overscroll-behavior: contain` in modals/drawers/sheets
-- During drag: disable text selection, `inert` on dragged elements
-- `autoFocus` sparingly—desktop only, single primary input; avoid on mobile
-
-### Safe Areas & Layout
-
-- Full-bleed layouts need `env(safe-area-inset-*)` for notches
-- Avoid unwanted scrollbars: `overflow-x-hidden` on containers, fix content overflow
-- Flex/grid over JS measurement for layout
-
-### Dark Mode & Theming
-
-- `color-scheme: dark` on `<html>` for dark themes (fixes scrollbar, inputs)
-- `<meta name="theme-color">` matches page background
-- Native `<select>`: explicit `background-color` and `color` (Windows dark mode)
-
-### Locale & i18n
-
-- Dates/times: use `Intl.DateTimeFormat` not hardcoded formats
-- Numbers/currency: use `Intl.NumberFormat` not hardcoded formats
-- Detect language via `Accept-Language` / `navigator.languages`, not IP
-
-### Hydration Safety
-
-- Inputs with `value` need `onChange` (or use `defaultValue` for uncontrolled)
-- Date/time rendering: guard against hydration mismatch (server vs client)
-- `suppressHydrationWarning` only where truly needed
-
-### Hover & Interactive States
-
-- Buttons/links need `hover:` state (visual feedback)
-- Interactive states increase contrast: hover/active/focus more prominent than rest
-
-### Content & Copy
-
-- Active voice: "Install the CLI" not "The CLI will be installed"
-- Title Case for headings/buttons (Chicago style)
-- Numerals for counts: "8 deployments" not "eight"
-- Specific button labels: "Save API Key" not "Continue"
-- Error messages include fix/next step, not just problem
-- Second person; avoid first person
-- `&` over "and" where space-constrained
-
-### Anti-patterns (flag these)
-
-- `user-scalable=no` or `maximum-scale=1` disabling zoom
-- `onPaste` with `preventDefault`
-- `transition: all`
-- `outline-none` without focus-visible replacement
-- Inline `onClick` navigation without `<a>`
-- `<div>` or `<span>` with click handlers (should be `<button>`)
-- Images without dimensions
-- Large arrays `.map()` without virtualization
-- Form inputs without labels
-- Icon buttons without `aria-label`
-- Hardcoded date/number formats (use `Intl.*`)
-- `autoFocus` without clear justification
-
-## Output Format
-
-Group by file. Use `file:line` format (VS Code clickable). Terse findings.
-
-```text
-## src/Button.tsx
-
-src/Button.tsx:42 - icon button missing aria-label
-src/Button.tsx:18 - input lacks label
-src/Button.tsx:55 - animation missing prefers-reduced-motion
-src/Button.tsx:67 - transition: all → list properties
-
-## src/Modal.tsx
-
-src/Modal.tsx:12 - missing overscroll-behavior: contain
-src/Modal.tsx:34 - "..." → "…"
-
-## src/Card.tsx
-
-✓ pass
-```
-
-State issue + location. Skip explanation unless fix non-obvious. No preamble.

.gitignore

@@ -30,7 +30,12 @@ Thumbs.db
 # Test reports
 *-report.txt
 
-# Agent direction (internal use only)
+# Internal docs (moved to private mulga repo)
 docs/INTERNAL.md
 docs/PRIVATE.md
 docs/DEVNOTES.md
+docs/HIVE_DEVELOPMENT_PLAN.md
+docs/PLAN.md
+docs/TODO.md
+docs/development/
+docs/internal/

INSTALL.md

@@ -674,7 +674,7 @@ ip addr show | grep "inet "
 
 #### 1. Set Variables
 
-Export the IPs for your three servers (replace with your actual IPs):
+Export the IPs for your three servers (replace with your actual IPs) on each server:
 
 ```bash
 export HIVE_NODE1=192.168.1.10
@@ -868,16 +868,54 @@ sudo ip addr del 10.11.12.3/24 dev eth0
 
 ## Shutdown
 
-Stop services on each node:
+When running in the multi-server cluster mode a graceful shutdown is required to synchronize and coordinate nodes.
 
+On any node, issue the following command to shutdown nodes cluster wide:
+
+```sh
+./bin/hive admin cluster shutdown
+```
+
+Example output:
 ```bash
-# Simulated — stop in reverse order:
-./scripts/stop-dev.sh ~/node3/
-./scripts/stop-dev.sh ~/node2/
-./scripts/stop-dev.sh ~/node1/
+Starting coordinated cluster shutdown (3 nodes)
+Phases: gate -> drain -> storage -> persist -> infra
+Timeout per phase: 2m0s
+
+[GATE] Sending to 3 node(s)...
+  [GATE] node1: stopped awsgw, hive-ui
+  [GATE] node3: stopped awsgw, hive-ui
+  [GATE] node2: stopped awsgw, hive-ui
+[GATE] Complete (3/3 nodes, 2.005s)
+
+[DRAIN] Sending to 3 node(s)...
+  [DRAIN] node1: 0/0 VMs remaining
+  [DRAIN] node1: 0/0 VMs remaining
+  [DRAIN] node2: 0/0 VMs remaining
+  [DRAIN] node2: 0/0 VMs remaining
+  [DRAIN] node3: 0/0 VMs remaining
+  [DRAIN] node3: 0/0 VMs remaining
+  [DRAIN] node1: OK
+  [DRAIN] node3: OK
+  [DRAIN] node2: OK
+[DRAIN] Complete (3/3 nodes, 5ms)
+
+[STORAGE] Sending to 3 node(s)...
+  [STORAGE] node1: stopped viperblock
+  [STORAGE] node3: stopped viperblock
+  [STORAGE] node2: stopped viperblock
+[STORAGE] Complete (3/3 nodes, 1.09s)
+
+[PERSIST] Sending to 3 node(s)...
+  [PERSIST] node1: stopped predastore
+  [PERSIST] node3: stopped predastore
+  [PERSIST] node2: stopped predastore
+[PERSIST] Complete (3/3 nodes, 1.003s)
+
+[INFRA] Sending final shutdown to all nodes...
+[INFRA] Complete
+Cluster shutdown complete (6.104s)
 
-# Real multi-server — run on each server:
-./scripts/stop-dev.sh
 ```
 
 If EC2 instances are running, the stop process will gracefully terminate them, unmount attached EBS volumes (via NBD), and flush the write-ahead-log (WAL) to the S3 server (Predastore). This may take a few minutes depending on instance volume sizes.

Makefile

@@ -1,4 +1,5 @@
 GO_PROJECT_NAME := hive
+SHELL := /bin/bash
 
 # Detect architecture for cross-platform support
 ARCH := $(shell uname -m)
@@ -37,30 +38,30 @@ build:
 
 # Build hive-ui frontend (requires pnpm)
 build-ui:
-	@echo "\n....Building hive-ui frontend...."
+	@echo -e "\n....Building hive-ui frontend...."
 	cd hive/services/hiveui/frontend && pnpm build
 
 # GO commands
 go_build:
-	@echo "\n....Building $(GO_PROJECT_NAME)"
+	@echo -e "\n....Building $(GO_PROJECT_NAME)"
 	go build $(GO_BUILD_MOD) -ldflags "-s -w" -o ./bin/$(GO_PROJECT_NAME) cmd/hive/main.go
 
 go_run:
-	@echo "\n....Running $(GO_PROJECT_NAME)...."
+	@echo -e "\n....Running $(GO_PROJECT_NAME)...."
 	$(GOPATH)/bin/$(GO_PROJECT_NAME)
 
 # Preflight — runs the same checks as GitHub Actions (format + lint + security + tests).
 # Use this before committing to catch CI failures locally.
 preflight: check-format vet security-check test
-	@echo "\n✅ Preflight passed — safe to commit."
+	@echo -e "\n ✅ Preflight passed — safe to commit."
 
 # Run unit tests
 test:
-	@echo "\n....Running tests for $(GO_PROJECT_NAME)...."
+	@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
 	LOG_IGNORE=1 go test -v -timeout 120s ./...
 
 bench:
-	@echo "\n....Running benchmarks for $(GO_PROJECT_NAME)...."
+	@echo -e "\n....Running benchmarks for $(GO_PROJECT_NAME)...."
 	$(MAKE) easyjson
 	LOG_IGNORE=1 go test -benchmem -run=. -bench=. ./...
 
@@ -73,7 +74,7 @@ clean:
 	rm -rf hive/services/hiveui/frontend/dist
 
 install-system:
-	@echo "\n....Installing system dependencies for $(ARCH)...."
+	@echo -e "\n....Installing system dependencies for $(ARCH)...."
 	@echo "QEMU packages: $(QEMU_PACKAGES)"
 	apt-get update && sudo apt-get install -y \
 		nbdkit nbdkit-plugin-dev pkg-config $(QEMU_PACKAGES) qemu-utils qemu-kvm \
@@ -81,7 +82,7 @@ install-system:
 		iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file
 
 install-go:
-	@echo "\n....Installing Go 1.26.0 for $(ARCH) ($(GO_ARCH))...."
+	@echo -e "\n....Installing Go 1.26.0 for $(ARCH) ($(GO_ARCH))...."
 	@if [ ! -d "/usr/local/go" ]; then \
 		curl -L https://go.dev/dl/go1.26.0.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
 	else \
@@ -90,7 +91,7 @@ install-go:
 	@echo "Go version: $$(go version)"
 
 install-aws:
-	@echo "\n....Installing AWS CLI v2 for $(ARCH) ($(AWS_ARCH))...."
+	@echo -e "\n....Installing AWS CLI v2 for $(ARCH) ($(AWS_ARCH))...."
 	@if ! command -v aws >/dev/null 2>&1; then \
 		curl "https://awscli.amazonaws.com/awscli-exe-linux-$(AWS_ARCH).zip" -o "awscliv2.zip"; \
 		unzip -o awscliv2.zip; \
@@ -101,7 +102,7 @@ install-aws:
 	fi
 
 quickinstall: install-system install-go install-aws
-	@echo "\n✅ Quickinstall complete for $(ARCH)."
+	@echo -e "\n✅ Quickinstall complete for $(ARCH)."
 	@echo "   Please ensure /usr/local/go/bin is in your PATH."
 	@echo "   Installed: Go ($(GO_ARCH)), AWS CLI ($(AWS_ARCH)), QEMU ($(QEMU_PACKAGES))"
 
@@ -130,12 +131,12 @@ vet:
 # Security checks — each tool fails the build on findings (matches CI).
 # Reports are also saved to tests/ for review.
 security-check:
-	@echo "\n....Running security checks for $(GO_PROJECT_NAME)...."
-	go tool govulncheck ./... 2>&1 | tee tests/govulncheck-report.txt
+	@echo -e "\n....Running security checks for $(GO_PROJECT_NAME)...."
+	set -o pipefail && go tool govulncheck ./... 2>&1 | tee tests/govulncheck-report.txt
 	@echo "  govulncheck ok"
-	go tool gosec -exclude=G204,G304,G402 -exclude-generated -exclude-dir=cmd ./... 2>&1 | tee tests/gosec-report.txt
+	set -o pipefail && go tool gosec -exclude=G204,G304,G402 -exclude-generated -exclude-dir=cmd ./... 2>&1 | tee tests/gosec-report.txt
 	@echo "  gosec ok"
-	go tool staticcheck -checks="all,-ST1000,-ST1003,-ST1016,-ST1020,-ST1021,-ST1022,-SA1019,-SA9005" ./... 2>&1 | tee tests/staticcheck-report.txt
+	set -o pipefail && go tool staticcheck -checks="all,-ST1000,-ST1003,-ST1016,-ST1020,-ST1021,-ST1022,-SA1019,-SA9005" ./... 2>&1 | tee tests/staticcheck-report.txt
 	@echo "  staticcheck ok"
 
 # Legacy alias — now runs the strict checks (same as security-check)
@@ -149,7 +150,7 @@ PARENT_DIR := $(shell cd .. && pwd)
 E2E_IMAGE := hive-e2e:latest
 
 test-docker-build:
-	@echo "\n....Building E2E Docker image...."
+	@echo -e "\n....Building E2E Docker image...."
 	@for dep in viperblock predastore; do \
 		if [ ! -d "$(PARENT_DIR)/$$dep" ]; then \
 			echo "Missing sibling repo $$dep — running clone-deps.sh"; \
@@ -160,17 +161,17 @@ test-docker-build:
 	docker build -t $(E2E_IMAGE) -f tests/e2e/Dockerfile.e2e $(PARENT_DIR)
 
 test-docker-single: test-docker-build
-	@echo "\n....Running Single-Node E2E Tests...."
+	@echo -e "\n....Running Single-Node E2E Tests...."
 	docker run --privileged --rm -v /dev/kvm:/dev/kvm $(E2E_IMAGE)
 
 test-docker-multi: test-docker-build
-	@echo "\n....Running Multi-Node E2E Tests...."
+	@echo -e "\n....Running Multi-Node E2E Tests...."
 	docker run --privileged --rm -v /dev/kvm:/dev/kvm --cap-add=NET_ADMIN $(E2E_IMAGE) ./tests/e2e/run-multinode-e2e.sh
 
 test-docker: test-docker-build
-	@echo "\n....Running Single-Node E2E Tests...."
+	@echo -e "\n....Running Single-Node E2E Tests...."
 	docker run --privileged --rm -v /dev/kvm:/dev/kvm $(E2E_IMAGE)
-	@echo "\n....Running Multi-Node E2E Tests...."
+	@echo -e "\n....Running Multi-Node E2E Tests...."
 	docker run --privileged --rm -v /dev/kvm:/dev/kvm --cap-add=NET_ADMIN $(E2E_IMAGE) ./tests/e2e/run-multinode-e2e.sh
 
 .PHONY: build build-ui go_build go_run preflight test bench run clean \

docs/PLAN.md

@@ -1,145 +0,0 @@
-# Plan: Implement CreateImage API
-
-## Context
-
-Users need the ability to create a new AMI from a running (or stopped) instance so they can launch new instances from snapshots of existing ones. This is the standard AWS `CreateImage` flow: snapshot the instance's root volume, then register a new AMI pointing at that snapshot. All the building blocks exist (CreateSnapshot, AMIMetadata, OpenFromSnapshot, cloneAMIToVolume) — they just need to be wired together.
-
-## Flow
-
-```
-AWS CLI: create-image --instance-id i-xxx --name "my-image"
-  → Gateway (ec2.go)
-    → NATS "ec2.CreateImage"
-      → Daemon handler
-        → Finds instance → gets root volume ID
-        → Calls snapshotService.CreateSnapshot(volumeID)
-        → Creates new AMI viperblock config with AMIMetadata.SnapshotID
-        → Stores as ami-xxx/config.json in S3
-        → Returns ami-xxx
-```
-
-## Changes
-
-### 1. Gateway: Add CreateImage action entry
-**File**: `hive/gateway/ec2.go`
-- Add `"CreateImage"` to the `ec2Actions` map, following the DescribeImages pattern
-
-### 2. Gateway: CreateImage handler
-**File**: `hive/gateway/ec2/image/CreateImage.go` (new)
-- `ValidateCreateImageInput()` — validate InstanceId is present and starts with `i-`
-- `CreateImage()` — create NATSImageService, call `svc.CreateImage(input)`
-- Follow exact pattern of `gateway/ec2/snapshot/CreateSnapshot.go`
-
-### 3. NATS service: Wire CreateImage to NATS
-**File**: `hive/handlers/ec2/image/service_nats.go`
-- Replace the `CreateImage` stub with a real NATS request to `"ec2.CreateImage"` topic
-- Use `utils.NATSRequest[ec2.CreateImageOutput]` with 120s timeout (snapshot can be slow)
-
-### 4. Daemon: Add NATS subscription and handler
-**File**: `hive/daemon/daemon.go`
-- Add `{"ec2.CreateImage", d.handleEC2CreateImage, "hive-workers"}` to subscriptions
-- Update `NewImageServiceImpl(d.config)` → `NewImageServiceImpl(d.config, d.natsConn)` (~line 515)
-
-**File**: `hive/daemon/daemon_handlers.go`
-- Add `handleEC2CreateImage` — this is a **stateful handler** (NOT handleNATSRequest) because it needs instance state to find the root volume
-- Steps:
-  1. Unmarshal `ec2.CreateImageInput`
-  2. Lock `d.Instances.Mu`, find instance in `d.Instances.VMS[*input.InstanceId]`, unlock
-  3. Validate instance exists and is in running or stopped state
-  4. Find root volume ID from `instance.Instance.BlockDeviceMappings` (first entry with `Ebs.VolumeId`)
-  5. Get instance status (running vs stopped) — pass to service so it knows which snapshot path to use
-  6. Get source AMI ID from `instance.Instance.ImageId` (for architecture, platform, etc.)
-  7. Call `d.imageService.CreateImageFromInstance(params)` passing the extracted data
-  8. Respond with JSON
-
-### 5. Image service: Update interface and add CreateImageParams
-**File**: `hive/handlers/ec2/image/service.go`
-- Keep `CreateImage(input *ec2.CreateImageInput)` signature unchanged on the interface (gateway NATS side doesn't use extra params)
-
-**File**: `hive/handlers/ec2/image/service_impl.go`
-- Add a `CreateImageParams` struct for the daemon-side call:
-  ```go
-  type CreateImageParams struct {
-      Input         *ec2.CreateImageInput
-      RootVolumeID  string
-      SourceImageID string
-      IsRunning     bool // true = use ebs.snapshot NATS, false = snapshot from S3 state
-  }
-  ```
-- The daemon handler calls `d.imageService.CreateImageFromInstance(params)` directly (not through the generic interface)
-
-### 6. Image service: Implement CreateImageFromInstance
-**File**: `hive/handlers/ec2/image/service_impl.go`
-
-Add `natsConn *nats.Conn` to `ImageServiceImpl` (for running-instance NATS snapshot calls and offline viperblock instantiation for stopped instances). Update constructor.
-
-Implement `CreateImageFromInstance(params CreateImageParams)`:
-1. Generate AMI ID: `utils.GenerateResourceID("ami")`
-2. Generate snapshot ID: `utils.GenerateResourceID("snap")`
-3. **Snapshot the root volume** — two paths:
-   - **Running instance**: NATS `ebs.snapshot.{volumeID}` — reuse the exact pattern from `snapshot/service_impl.go:191-218`:
-     - Marshal `config.EBSSnapshotRequest{Volume: rootVolumeID, SnapshotID: snapshotID}`
-     - Send NATS request to `ebs.snapshot.{rootVolumeID}` with 30s timeout
-     - Validate response
-   - **Stopped instance**: Volume is not mounted in viperblockd. Data is fully persisted in S3. Create snapshot offline:
-     - Instantiate viperblock via `newViperblock(rootVolumeID, ...)` (reuse pattern from `instance/service_impl.go:238-258`)
-     - `backend.Init()` + `LoadStateRequest("")` to load config + block map from S3
-     - Call `vb.CreateSnapshot(snapshotID)` directly — the flush/WAL steps are no-ops on a freshly loaded instance, it just serializes the block map into the snapshot checkpoint
-     - `vb.RemoveLocalFiles()` to clean up temp files
-4. **Store snapshot metadata** in `{snapshotID}/metadata.json` — same pattern as `snapshot/service_impl.go:220-252`
-5. **Read source AMI config** from S3 `{sourceImageID}/config.json` to get architecture, platform, etc.
-6. **Read root volume config** from S3 `{rootVolumeID}/config.json` to get volume size
-7. **Build AMI VolumeConfig** with populated `AMIMetadata`:
-   - `ImageID`: new ami-xxx
-   - `Name`: from input
-   - `Description`: from input
-   - `SnapshotID`: the snap-xxx just created
-   - `Architecture`, `PlatformDetails`, `Virtualization`, etc.: from source AMI (or defaults)
-   - `VolumeSizeGiB`: from root volume config
-   - `CreationDate`: now
-   - `RootDeviceType`: "ebs"
-   - `ImageOwnerAlias`: "self"
-8. **Store AMI config** to S3 as `{amiID}/config.json` — create a `VBState` with the VolumeConfig. This is what `DescribeImages` reads.
-9. Return `&ec2.CreateImageOutput{ImageId: &amiID}`
-
-### 7. Tests
-- Add unit test for `CreateImage` in `hive/handlers/ec2/image/` with mock object store
-- Add gateway validation test in `hive/gateway/ec2/image/`
-
-## Key Files
-
-| File | Change |
-|------|--------|
-| `hive/gateway/ec2.go` | Add CreateImage action |
-| `hive/gateway/ec2/image/CreateImage.go` | New gateway handler |
-| `hive/handlers/ec2/image/service.go` | Update interface |
-| `hive/handlers/ec2/image/service_nats.go` | Wire NATS request |
-| `hive/handlers/ec2/image/service_impl.go` | Core implementation |
-| `hive/daemon/daemon.go` | Add subscription, update constructor |
-| `hive/daemon/daemon_handlers.go` | Add stateful handler |
-
-## Existing Code to Reuse
-
-- `snapshot/service_impl.go:191-218` — EBS snapshot NATS pattern (`ebs.snapshot.{volumeID}`)
-- `snapshot/service_impl.go:107-123` — `putSnapshotConfig` S3 storage pattern
-- `image/service_impl.go:96-126` — Reading viperblock VBState from S3 (`config.json`)
-- `instance/service_impl.go:238-258` — `newViperblock()` for offline viperblock instantiation
-- `utils.GenerateResourceID()` — ID generation
-- `utils.NATSRequest[T]()` — Generic NATS request helper
-- `viperblock.VBState`, `viperblock.VolumeConfig`, `viperblock.AMIMetadata` — Existing types
-
-## Verification
-
-1. `make build` — confirm compilation
-2. `make test` — all existing tests pass
-3. Manual test:
-   ```bash
-   # Launch instance from existing AMI
-   aws ec2 run-instances --image-id ami-xxx --instance-type t2.micro --key-name mykey
-   # Create image from running instance
-   aws ec2 create-image --instance-id i-xxx --name "test-snapshot-image"
-   # Verify it appears
-   aws ec2 describe-images --owners self
-   # Launch new instance from the snapshot-based AMI
-   aws ec2 run-instances --image-id ami-yyy --instance-type t2.micro --key-name mykey
-   ```

docs/TODO.md

@@ -1,240 +0,0 @@
-# Hive Development Roadmap
-
-## Update Jan 2026
-
-### Big rocks:
-
-- [DONE] Add multi-node support for predastore (S3) using reed solomon encoding
-  - [DONE] Implement KV store for object lookup, WAL files, offset. Use hash-ring to determine which nodes an object belongs to.
-- Implement VPC support using Open vSwitch across multiple nodes, core VPC functionality included
-  - Add support for NVIDIA Bluefield DPU with Open vSwitch
-- Implement basic IAM using NATS Jetstream as KV store, vs IAM/access-keys in local config/TOML files for beta.
-  - [DONE] Move `daemon.go` instances.json state to Jetstream KV
-- Add support using the `hive` CLI tool to provision a new user with AWS access-keys/IAM.
-  - Support multi-tenant operations and isolation
-- Add support to include capabilities when adding a new hardware node to MulgaOS (e.g EC2 target, S3, EBS, NATs, etc) - Features can be turned on/off depending on hardware scope.
-- [DONE] Add simple Web UI console, using the AWS JS SDK, communicating to local AWS gateway.
-  - [DONE] Implement ShadCNblocks for UI framework
-  - [DONE] Simple Go webserver, static files, easy build process.
-
-### Implementation gaps:
-
-- EC2 - Support extended features for `run-instance`
-  - [DONE] Volume resize of AMI. Note `nbd` does not support live resize events, instance needs to be stopped, resized, and started.
-  - Confirm cloud-init will resize volume on boot and configured correctly.
-  - [DONE] Attach additional volumes
-  - Attach to VPC / Security group (required Open vSwitch implementation)
-- only allow hive to use x% of cpu and memory. dont allow allocating 100% of resources.
-- placement group instances, allow spreading instance group between nodes.
-- capabilities flag
-- more instance types - per cpu gen
-- [DONE] multi part uploads on frontend
-- [DONE] delete volumes on termination
-- fix describe instances
-- [DONE] change all log into slog
-- `nbd` does not support resizing disks live. Requires instance to be stopped, boot/root volume, or attached volume will need to be resized, and instance started again. Limitation for Hive v1 using NBD, aiming to resolve with `vhost-user-blk` to create a `virtio-blk` device for extended functionality and performance.
-- Migrate to net/http + chi from fiber
-- remove insecure skip verify - need to add predastore multi node ips to certs
-- migrate to list objects v2
-- split up utils.go
-- make sure nats is running before we start daemon? eigw and account service will fallback to memory versions but that breaks all functionality
-- the BlockDeviceMappings > DeviceName (/dev/xvda) is not the true volume, i think it's /dev/vda on mine. I think qmp has a query device stats which will provide block devices so we can match these up. not a blocker, but needs to be fixed eventually
-- aws ec2 modify-instance-attribute
-
-### Issues to investigate
-
-- When the host disk volume is full and a new instance is launched, disk corruption will occur within the instance since the host volume is out of space. Consider a pre-flight check, check available storage on the S3 server, local viperblock for WAL cache availability and provide improved guard rails to prevent a new instance running, if available disk space is nearing to be exceeded.
-
-Host:
-
-```json
-{
-  "time": "2026-01-27T21:30:29.368164556+10:00",
-  "level": "ERROR",
-  "msg": "WAL sync failed",
-  "error": "write /home/hive/hive/predastore/distributed/nodes/node-2/0000000000000737.wal: no space left on device"
-}
-```
-
-Instance `dmesg`:
-
-```
-[ 1445.573495] EXT4-fs error (device vda1): ext4_journal_check_start:83: comm systemd-journal: Detected aborted journal
-[ 1445.583565] EXT4-fs (vda1): Remounting filesystem read-only
-```
-
----
-
-
-
-## Update Nov 2025
-
-[PARTIAL] _ Implement multi-tenant support
-[PARTIAL] _ Move config settings from ~/hive/\*.toml, to using Nats Jetstream for core config which can be synced between nodes.
-
-- Implement a lightweight IAM using Nats Jetstream, vs current config files for auth settings.
-  [ONGOING] _ Implement Reed Solomon Encoding for Predastore (S3) objects and a WAL implementation, for storing multiple objects in a single shard (e.g 4MB), with the WAL referencing the location of each object (e.g 4kb min)
-  _ Implement basic KY lookup, object key sha512 (bucket/key), location to shard on S3 (obj.0000124.bin), read WAL (e.g first 4096 bytes) to determine location of the object. e.g key-1234 => obj.0000124.bin, wal header (4096 bytes) > key-1234 == offset location (seek) 16384, len 32768.
-  _ Read multiple nodes (e.g 5 predastore instances, k = 3 (data shards), n = 5 (total shards), n - k = 2 (parity shards) )
-  [ONGOING] _ Complete core scaffolding AWS SDK/API requirements (ec2 describe-instances, run-instances, etc)
-- Implement UEFI support for image downloads and `qemu` exec in `vm.go`
-- Confirm Alpine Linux, fails import image AMI > (run-instance) ""Failed to read block from AMI source" err="request out of range" - Block size correct?
-- Improve shutdown gracefully, `./scripts/stop-dev.sh` waits 60 seconds, while qemu/nbd could still be shutting down.
-- [DONE] Add delete-volume support via EBS (s3 vol-\*) for terminated instance (DeleteOnTermination flag)
-- Add default LRU cache support for viperblock, depending on the instance type / volume size and system memory available.
-
-## Multi-node setup
-
-[DONE] - Development release - v0.1.0 (single node)
-[PARTIAL] - Production release - v0.2.0 (multi node)
-
-Design layout for multi-node configuration.
-
-```
-# node1
-hive admin init --region ap-southeast-2 --az ap-southeast-2a --node node1 --nodes 3 --hive-dir ~/hive/
-
-# node2
-hive admin join --region ap-southeast-2 --az ap-southeast-2a --node node2 --host node1.local:8443 --data-dir ~/hive/
-
-# node3, optionally toggle EBS/EC2/NATs support only
-hive admin join --region ap-southeast-2 --az ap-southeast-2a --node node3 --host node1.local:8443 --data-dir ~/hive/ --cap ebs,ec2,nats
-```
-
-If --host is missing, the join command tries multicast broadcast to find parent (leader) node.
-
-Set region with `--region ap-southeast-2` which will create a new Hive cluster for the specified region.
-
-For production, it is recommended to run Hive on at least three physical nodes. In this minimum setup, Hive S3 and EBS volumes use 2 data shards and 1 parity shard RS(2,1) to distribute each chunk across the cluster. This configuration tolerates a full node failure while adding only 1.5x storage overhead.
-
-### Overview:
-
-- init creates a cluster-id, node-id, and a short join token, starts a tiny control server on :8443, and writes DNS hint files if available.
-
-- init sets target size to 3 and waits until all 3 nodes join and ack.
-
-- join nodes contact node1, present the token or cluster-id, and advertise capabilities.
-
-- node1 appends them to the member set and immediately pushes the current settings bundle to them.
-
-- When member count reaches 3, node1 broadcasts the final cluster settings to all nodes.
-
-- Every node writes the same cluster.json and starts the services that match its capabilities.
-
-Node exchange payload
-
-```json
-{
-  "cluster_id": "c-82d5",
-  "node_id": "n-rpi2",
-  "addr": "rpi2.local:8443",
-  "version": "0.1.0",
-  "caps": ["ec2", "s3"],
-  "ts": 1731388800
-}
-```
-
-Settings bundle (identical on all 3 once committed)
-
-```json
-{
-  "cluster_id": "c-82d5",
-  "target_size": 3,
-  "members": [
-    {
-      "node_id": "n-rpi1",
-      "addr": "rpi1.local:8443",
-      "caps": ["ec2", "s3", "nats", "ebs"]
-    },
-    { "node_id": "n-rpi2", "addr": "rpi2.local:8443", "caps": ["ec2", "s3"] },
-    { "node_id": "n-rpi3", "addr": "rpi3.local:8443", "caps": ["nats", "ebs"] }
-  ],
-  "services": {
-    "ec2": { "api_bind": ":9001" },
-    "s3": { "api_bind": ":9002", "replicas": 2 },
-    "nats": { "cluster": "c-82d5", "quorum": 2 },
-    "ebs": { "replicas": 3, "block_size": 4096 }
-  },
-  "epoch": 1,
-  "sig": "ed25519:..." // signed by rpi1 during init
-}
-```
-
-## Original TODO Items → Development Plan Integration
-
-### ✅ **Completed Integration**
-
-All original TODO items have been incorporated into the structured development phases:
-
-| Original TODO                         | Status     | Integrated Into                                | Phase     |
-| ------------------------------------- | ---------- | ---------------------------------------------- | --------- |
-| #1: Binary compile and install.sh     | ✅ Planned | Phase 7: Task 7.1 (Binary Compilation)         | Phase 7   |
-| #2: Move daemon.go to services/hived/ | ✅ Planned | Phase 3: Task 3.2 (Specialized Services)       | Phase 3   |
-| #3: VPC with openvs-switch as `vpcd`  | ✅ Planned | Phase 5: Task 5.1 (VPC with Open vSwitch)      | Phase 5   |
-| #4: AWS HTTP gateway (`awsd`)         | ✅ Planned | Phase 3: Task 3.1 (AWS Gateway Service)        | Phase 3   |
-| #5: AWS SDK v2 input/output           | ✅ Planned | Phase 2: Task 2.1 & Phase 3: Task 3.1          | Phase 2-3 |
-| #6: `hive.service` for systemd        | ✅ Planned | Phase 7: Task 7.2 (System Service Integration) | Phase 7   |
-| #7: Gossip and RAFT protocols         | ✅ Planned | Phase 0: Task 0.1 (Service Registry)           | Phase 0   |
-| #8: etcd/KV for configuration sync    | ✅ Planned | Phase 0: Task 0.1 (Distributed Config)         | Phase 0   |
-| #9: Smithy model code generation      | ✅ Planned | Phase 2: Task 2.1 (Smithy Code Generation)     | Phase 2   |
-
-## Development Phase Overview
-
-**Phase 0**: Distributed Systems Foundation (1-2 weeks)
-
-- Gossip, RAFT, and distributed configuration (#7, #8)
-
-**Phase 1**: Development Environment Automation (2-3 weeks)
-
-- Multi-service orchestration and hot reloading
-
-**Phase 2**: AWS API Model Implementation (3-4 weeks)
-
-- Smithy-based code generation (#9)
-- AWS SDK v2 integration (#5)
-
-**Phase 3**: Scalable Gateway and Daemon Architecture (2-3 weeks)
-
-- AWS gateway service `awsd` (#4)
-- Service refactoring to `hived` (#2)
-- VPC daemon `vpcd` foundation (#3)
-
-**Phase 4**: Service Integration and Orchestration (3-4 weeks)
-
-- NATS clustering and cross-service coordination
-
-**Phase 5**: Infrastructure Services (4-5 weeks)
-
-- VPC networking with Open vSwitch (#3)
-- Advanced VM features
-
-**Phase 6**: Testing and Validation Framework (2-3 weeks)
-
-- AWS CLI compatibility testing
-
-**Phase 7**: Production Deployment and Packaging (2-3 weeks)
-
-- Binary compilation and distribution (#1)
-- Systemd service integration (#6)
-- Production configuration management
-
-## Service Architecture
-
-The development plan implements these services:
-
-- **`awsd`** - AWS API Gateway (TODO #4)
-- **`hived`** - Main Hive compute daemon (TODO #2)
-- **`vpcd`** - VPC networking daemon (TODO #3)
-- **Predastore** - S3 service (existing)
-- **Viperblock** - EBS service (existing)
-
-## Getting Started
-
-For development setup:
-
-```bash
-./scripts/dev-setup.sh     # Setup complete development environment
-./scripts/start-dev.sh     # Start all services
-```
-
-For detailed implementation guidance, see [HIVE_DEVELOPMENT_PLAN.md](HIVE_DEVELOPMENT_PLAN.md).

docs/development/bugs/capacity.md

@@ -1,253 +0,0 @@
-# RunInstances capacity routing via per-instance-type NATS subscriptions
-
-**Status:** Implemented (local tests pass, pending remote verification)
-
-## Problem
-
-When `RunInstances` is called, the gateway publishes to NATS topic `ec2.RunInstances` with queue group `hive-workers`. NATS round-robins the request to one daemon. If that daemon has insufficient capacity, it returns `InsufficientInstanceCapacity` and the gateway passes the error to the client. No other node is tried.
-
-This means a cluster with available capacity can reject instance launches if the randomly-selected node happens to be full.
-
-### Observed behavior (3-node cluster)
-
-Node 3 already runs a `t3.medium` (2 vCPUs, 4 GB) and has no remaining capacity:
-
-```
-# First attempt — NATS routes to node3 (full) → error
-$ aws ec2 run-instances --image-id $HIVE_AMI --instance-type t3.medium ...
-An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation: None
-
-# Second attempt — NATS routes to node1 or node2 (free) → success
-$ aws ec2 run-instances --image-id $HIVE_AMI --instance-type t3.medium ...
-{ "Instances": [{ "InstanceId": "i-6c97c523f1ead0eef", "State": { "Name": "pending" } }] }
-```
-
-The user must retry manually until NATS happens to pick a node with capacity.
-
-### Why this happens
-
-```
-AWS SDK → Gateway → NATS "ec2.RunInstances" [queue: hive-workers] → ONE daemon (random)
-                                                                        ↓
-                                                          canAllocate() fails
-                                                                        ↓
-                                                      InsufficientInstanceCapacity
-                                                                        ↓
-                                                    Gateway returns error to client
-```
-
-There is no capacity-aware routing. NATS blindly round-robins to any daemon, regardless of whether it can serve the requested instance type.
-
-## Solution: per-instance-type NATS topics with dynamic subscription
-
-Leverage NATS subscription semantics directly: each daemon subscribes to `ec2.RunInstances.{instanceType}` only for types it has capacity to serve. When a node fills up for a given type, it unsubscribes. When capacity is freed, it re-subscribes.
-
-This means NATS itself becomes the capacity-aware router — requests are only delivered to nodes that can serve them. No retry logic, no discovery, no hacks.
-
-### Fixed NATS topic flow
-
-```
-Gateway → "ec2.RunInstances.t3.medium" [queue: hive-workers] → only daemons with capacity
-  → Daemon receives (guaranteed to have capacity) → allocates → launches VM
-  → Daemon recalculates: t3.medium no longer fits → unsubscribes from ec2.RunInstances.t3.medium
-  → Other daemons still subscribed → next request routed to them
-
-All nodes full for t3.medium:
-Gateway → "ec2.RunInstances.t3.medium" → no subscribers → NATS returns ErrNoResponders
-  → Gateway maps ErrNoResponders to InsufficientInstanceCapacity → returns to client
-```
-
-### How dynamic subscription works
-
-Each daemon maintains a set of per-instance-type NATS subscriptions. After every resource change (allocate or deallocate), it recalculates which types can fit at least 1 instance:
-
-```
-Node starts (8 vCPUs, 32 GB free):
-  → subscribes to: ec2.RunInstances.t3.nano, .micro, .small, .medium, .large, .xlarge, .2xlarge
-                    ec2.RunInstances.m8a.nano, .micro, .small, .medium, .large, .xlarge, .2xlarge
-
-After launching t3.2xlarge (8 vCPU, 32 GB → 0 free):
-  → unsubscribes from ALL ec2.RunInstances.* types (nothing fits)
-
-After terminating t3.2xlarge (resources freed → 8 vCPU, 32 GB free again):
-  → re-subscribes to all 14 instance types
-```
-
-### Race condition handling
-
-There's a small window between a subscription existing and resources being allocated where two requests for the last available slot could arrive at the same node. This is handled naturally:
-
-1. Request A arrives → daemon allocates → success → recalculates subscriptions
-2. Request B arrives before unsubscribe completes → daemon tries to allocate → `canAllocate()` returns 0
-3. Daemon responds with `InsufficientInstanceCapacity` (same as today)
-
-This is an edge case that only occurs when a node is at exact capacity boundary with concurrent requests. The existing `canAllocate()` check remains as a safety net. This is acceptable — the alternative (distributed locking) adds complexity far exceeding the rare failure.
-
-## Implementation
-
-### Step 1: Gateway publishes to instance-type-specific topic
-
-**File:** `hive/handlers/ec2/instance/service_nats.go`
-
-Change:
-```go
-func (s *NATSInstanceService) RunInstances(input *ec2.RunInstancesInput) (*ec2.Reservation, error) {
-    topic := fmt.Sprintf("ec2.RunInstances.%s", aws.StringValue(input.InstanceType))
-    return utils.NATSRequest[ec2.Reservation](s.natsConn, topic, input, 5*time.Minute)
-}
-```
-
-### Step 2: Gateway maps ErrNoResponders to InsufficientInstanceCapacity
-
-**File:** `hive/utils/nats.go`
-
-Update `NATSRequest` to detect `nats.ErrNoResponders` and return a specific error the gateway can map to `InsufficientInstanceCapacity`:
-
-```go
-msg, err := conn.Request(subject, jsonData, timeout)
-if err != nil {
-    if errors.Is(err, nats.ErrNoResponders) {
-        return nil, fmt.Errorf("no responders for subject %s: %w", subject, nats.ErrNoResponders)
-    }
-    return nil, fmt.Errorf("NATS request failed: %w", err)
-}
-```
-
-**File:** `hive/gateway/ec2/instance/RunInstances.go`
-
-Map the no-responders error to the AWS-compatible error:
-```go
-reservationPtr, err := service.RunInstances(input)
-if err != nil {
-    if errors.Is(err, nats.ErrNoResponders) {
-        return reservation, errors.New(awserrors.ErrorInsufficientInstanceCapacity)
-    }
-    return reservation, err
-}
-```
-
-### Step 3: Add subscription management to ResourceManager
-
-**File:** `hive/daemon/daemon.go`
-
-Add to `ResourceManager`:
-```go
-type ResourceManager struct {
-    // ... existing fields ...
-    natsConn    *nats.Conn
-    node        string
-    instanceSubs map[string]*nats.Subscription  // topic → subscription
-    handler      nats.MsgHandler                // RunInstances handler
-}
-```
-
-Add methods:
-```go
-// updateInstanceSubscriptions recalculates which instance types can fit
-// and subscribes/unsubscribes accordingly. Called after every allocate/deallocate.
-func (rm *ResourceManager) updateInstanceSubscriptions() {
-    rm.mu.RLock()
-    defer rm.mu.RUnlock()
-
-    for typeName, typeInfo := range rm.instanceTypes {
-        topic := fmt.Sprintf("ec2.RunInstances.%s", typeName)
-        canFit := rm.canAllocateUnlocked(typeInfo, 1) >= 1
-
-        _, subscribed := rm.instanceSubs[topic]
-        if canFit && !subscribed {
-            sub, err := rm.natsConn.QueueSubscribe(topic, "hive-workers", rm.handler)
-            if err != nil {
-                slog.Error("Failed to subscribe to instance type topic", "topic", topic, "err", err)
-                continue
-            }
-            rm.instanceSubs[topic] = sub
-            slog.Info("Subscribed to instance type", "topic", topic)
-        } else if !canFit && subscribed {
-            rm.instanceSubs[topic].Unsubscribe()
-            delete(rm.instanceSubs, topic)
-            slog.Info("Unsubscribed from instance type (capacity full)", "topic", topic)
-        }
-    }
-}
-```
-
-### Step 4: Call updateInstanceSubscriptions after allocate/deallocate
-
-**File:** `hive/daemon/daemon.go`
-
-Update `allocate()` to recalculate subscriptions after reserving resources:
-```go
-func (rm *ResourceManager) allocate(instanceType *ec2.InstanceTypeInfo) error {
-    // ... existing allocation logic ...
-    rm.updateInstanceSubscriptions()
-    return nil
-}
-```
-
-Update `deallocate()` to recalculate subscriptions after freeing resources:
-```go
-func (rm *ResourceManager) deallocate(instanceType *ec2.InstanceTypeInfo) {
-    // ... existing deallocation logic ...
-    rm.updateInstanceSubscriptions()
-}
-```
-
-### Step 5: Remove static ec2.RunInstances subscription from subscribeAll()
-
-**File:** `hive/daemon/daemon.go`
-
-Remove `{"ec2.RunInstances", d.handleEC2RunInstances, "hive-workers"}` from the `subscribeAll()` table. The initial subscriptions are created by `updateInstanceSubscriptions()` called during daemon startup (after `NewResourceManager()` sets up the handler).
-
-### Step 6: Initialize subscription management on daemon start
-
-**File:** `hive/daemon/daemon.go`
-
-After `subscribeAll()` in `Start()`, initialize the instance type subscriptions:
-```go
-d.resourceMgr.handler = d.handleEC2RunInstances
-d.resourceMgr.natsConn = d.natsConn
-d.resourceMgr.node = d.node
-d.resourceMgr.instanceSubs = make(map[string]*nats.Subscription)
-d.resourceMgr.updateInstanceSubscriptions()
-```
-
-### Step 7: Update tests
-
-**File:** `hive/daemon/daemon_test.go`
-
-- Update tests that publish to `ec2.RunInstances` to publish to `ec2.RunInstances.{instanceType}` instead
-- Add test: verify that after filling a node, it unsubscribes from the filled type
-- Add test: verify that after deallocating, it re-subscribes
-- Add test: verify `ErrNoResponders` when no node has capacity
-
-**File:** `hive/handlers/ec2/instance/service_nats_test.go` (if exists)
-
-- Update topic assertions
-
-## Files summary
-
-| File | Change |
-|------|--------|
-| `hive/handlers/ec2/instance/service_nats.go` | Publish to `ec2.RunInstances.{instanceType}` |
-| `hive/utils/nats.go` | Preserve `ErrNoResponders` in error chain |
-| `hive/gateway/ec2/instance/RunInstances.go` | Map `ErrNoResponders` → `InsufficientInstanceCapacity` |
-| `hive/daemon/daemon.go` | Add subscription management to ResourceManager, remove static subscription |
-| `hive/daemon/daemon_test.go` | Update topics, add subscribe/unsubscribe tests |
-
-## Existing behavior preserved
-
-- `canAllocate()` still runs as a safety net (handles race conditions)
-- Instance type validation (`instanceType, exists := d.resourceMgr.instanceTypes[...]`) still runs
-- All other NATS subscriptions unchanged
-- DescribeInstanceTypes still uses fan-out (no queue group) — unchanged
-
-## Verification
-
-1. `make test` — all tests pass
-2. `make build` — compiles
-3. 3-node cluster test:
-   - Start cluster, all nodes subscribe to all 14 instance types
-   - Fill node1 with t3.2xlarge — node1 unsubscribes from all types
-   - `RunInstances t3.medium` → NATS routes to node2 or node3 (never node1)
-   - Fill all nodes → `RunInstances` returns `InsufficientInstanceCapacity` (NATS ErrNoResponders)
-   - Terminate instance on node1 → node1 re-subscribes → next RunInstances succeeds

docs/development/bugs/instance-state.md

@@ -1,127 +0,0 @@
-# Instance State Management
-
-## Problem Statement
-
-The current implementation tightly couples EC2 instances to specific daemon nodes throughout their entire lifecycle. When an instance is stopped, the daemon retains ownership and NATS subscriptions, preventing instances from being started on different nodes. This creates issues in multi-node deployments and prevents proper recovery from ungraceful shutdowns.
-
-## Goals
-
-1. Enable stopped instances to be started on any available node
-2. Handle ungraceful shutdown scenarios (QEMU killed, viperblock/nbdkit crashes)
-3. Ensure data consistency when instances migrate between nodes
-4. Improve test coverage for edge cases and failure scenarios
-
-## Current Behavior
-
-- Daemon owns an instance throughout its entire lifecycle (running and stopped)
-- `ec2.cmd.$instance` NATS subscription remains on the original node even when stopped
-- StartInstance commands route directly to the original daemon
-- This works correctly for single-node deployments but breaks multi-node scenarios
-
-## Proposed Architecture
-
-### Instance Ownership Model
-
-**Running Instance:**
-- Daemon owns the instance and subscribes to `ec2.cmd.$instance`
-- All commands (stop, reboot, terminate, etc.) route directly to the owning daemon
-
-**Stopped Instance:**
-- Daemon releases the `ec2.cmd.$instance` subscription
-- Instance state persists in NATS KV (no daemon ownership)
-- Gateway handles StartInstance, routing to any available daemon
-
-### State Transitions
-
-```
-[Stopped] --StartInstance--> Gateway routes to available daemon --> [Running]
-[Running] --StopInstance-->  Daemon releases subscription       --> [Stopped]
-[Running] --Crash/Kill-->    Daemon detects, handles cleanup    --> [Requires Recovery]
-```
-
-### Viperblock WAL Synchronization
-
-Viperblock maintains a local WAL on NVMe for performance. On ungraceful shutdown, the WAL may contain blocks not yet pushed to Predastore (S3).
-
-**Graceful Shutdown:**
-1. Stop QEMU
-2. Flush viperblock WAL to Predastore
-3. Release `ec2.cmd.$instance` subscription
-4. Instance can start on any node
-
-**Ungraceful Shutdown (crash, kill, hardware failure):**
-1. Instance marked as `requires-recovery` in NATS KV
-2. WAL data remains on original node's NVMe
-3. Instance MUST restart on same node OR original node must sync WAL first
-
-**Recovery Sequence:**
-1. StartInstance received for instance in `requires-recovery` state
-2. Gateway queries original node: "Can you sync WAL for instance X?"
-3. Original node syncs WAL to Predastore
-4. Instance state updated to `stopped`
-5. Any available node can now start the instance
-
-### Edge Case: Original Node at Capacity
-
-If the original node has no resources available and holds unsync'd WAL data:
-1. Original node still syncs WAL to Predastore (sync doesn't require running the instance)
-2. After sync completes, instance can start on any other available node
-
-## Implementation Plan
-
-### Phase 1: Refactor Instance Ownership
-
-1. **Modify StopInstance handler** - Release `ec2.cmd.$instance` subscription after graceful stop
-2. **Add Gateway StartInstance handler** - Route to available daemon via queue group
-3. **Update daemon availability tracking** - Only listen for new instances when resources available
-4. **Update NATS KV schema** - Track instance state and last-known node
-
-### Phase 2: Graceful Shutdown Flow
-
-1. **Implement WAL sync on stop** - Ensure viperblock flushes to Predastore before releasing ownership
-2. **Add sync confirmation** - Daemon confirms WAL sync complete before releasing subscription
-3. **Update instance metadata** - Store `wal_synced: true` in NATS KV
-
-### Phase 3: Crash Detection and Recovery
-
-1. **Detect ungraceful shutdown** - Monitor QEMU, nbdkit, viperblock processes
-2. **Mark instances for recovery** - Set `requires_recovery: true` and `last_node: $node_id`
-3. **Implement recovery endpoint** - `POST /recover` triggers WAL sync from original node
-4. **Add recovery check to StartInstance** - Block start until WAL synced
-
-### Phase 4: Testing
-
-**Unit Tests:**
-- Instance state transitions (running → stopped → running)
-- Ownership release on stop
-- Ownership acquisition on start
-- WAL sync completion tracking
-
-**Integration Tests:**
-- Multi-node instance migration
-- Graceful shutdown and restart on different node
-- Concurrent StartInstance requests
-
-**Edge Case Tests:**
-- Kill QEMU process mid-operation
-- Kill viperblock/nbdkit processes
-- Corrupt NATS KV state (instance shows wrong state)
-- Node failure during WAL sync
-- Start instance when original node unavailable
-- Start instance when original node at capacity
-
-**E2E Tests:**
-- Full instance lifecycle across multiple nodes
-- Recovery from simulated hardware failure
-
-## Technical Notes
-
-### Predastore Replication
-
-All instance data is stored in Predastore, which uses Reed-Solomon erasure coding (e.g., RS(3,2) = 3 data shards, 2 parity shards). Objects are distributed across nodes via consistent hashing. If a data node is unavailable, parity shards can reconstruct the data. See [predastore/DESIGN.md](https://github.com/mulgadc/predastore/blob/main/DESIGN.md).
-
-### NATS Topics
-
-- `ec2.cmd.$instance` - Commands for running instances (daemon subscribes)
-- `ec2.start` - StartInstance requests (queue group, any available daemon)
-- `ec2.recovery.$node` - WAL sync requests for specific node

docs/development/feature/attach-volume.md

@@ -1,125 +0,0 @@
-# EC2 AttachVolume
-
-Hot-plug an EBS volume to a running instance via QMP, implementing the AWS `ec2 attach-volume` API.
-
-## Request Flow
-
-```
-AWS SDK (attach-volume)
-  -> Gateway (port 9999): parse AWS query params, validate input
-    -> NATS ec2.cmd.{instanceId}: per-instance topic routing
-      -> Daemon handleEC2Events: owns the instance's QEMU process
-        1. Validate volume is "available" (Predastore/S3 lookup)
-        2. ebs.mount via NATS -> viperblock starts NBD server
-        3. QMP blockdev-add -> register NBD block device in QEMU
-        4. QMP device_add -> attach as virtio-blk-pci to guest
-        5. Update instance metadata + volume metadata
-      <- NATS response: ec2.VolumeAttachment JSON
-    <- Gateway: wrap in XML
-  <- AWS SDK: VolumeAttachment response
-```
-
-## Why Per-Instance NATS Routing
-
-AttachVolume uses `ec2.cmd.{instanceId}` instead of a generic `ec2.AttachVolume` topic. This is because the command must reach the specific daemon process that owns the QEMU instance. Each daemon subscribes to `ec2.cmd.{instanceId}` for every instance it manages. The same pattern is used by Start, Stop, and Terminate. If no daemon owns the instance, NATS returns `ErrNoResponders`, which the gateway maps to `InvalidInstanceID.NotFound`.
-
-Other volume operations (CreateVolume, DeleteVolume, DescribeVolumes) use generic topics like `ec2.CreateVolume` because they don't require access to a specific QEMU process.
-
-## Files Changed
-
-### Wire Format: `hive/qmp/qmp.go`
-
-Added `AttachVolume bool` to `Attributes` and a new `AttachVolumeData` struct carried as an optional pointer on `Command`. The pointer-with-omitempty pattern means the field is absent from JSON for non-attach commands, keeping the wire format clean. The `Attributes` struct uses boolean flags for dispatch (matching the existing Start/Stop/Terminate pattern).
-
-### Gateway Handler: `hive/gateway/ec2/volume/AttachVolume.go`
-
-Validates `VolumeId` and `InstanceId` are present, constructs a `qmp.Command` with `AttachVolume: true` and `AttachVolumeData`, sends it to `ec2.cmd.{instanceId}` via NATS request-response (30s timeout). The `Device` field is optional per the AWS API spec.
-
-NATS errors are differentiated: `ErrNoResponders` returns `InvalidInstanceID.NotFound` (no daemon owns this instance), while timeouts and connection errors return `InternalError` to avoid misleading users about non-existent instances when the real problem is infrastructure.
-
-### Gateway Switch: `hive/gateway/ec2.go`
-
-Added `"AttachVolume"` case to the EC2 action switch statement.
-
-### Daemon Handler: `hive/daemon/daemon_handlers.go`
-
-The core logic, placed as the first branch in `handleEC2Events` (before Start/Stop/Terminate). Implements a three-phase commit with reverse-order rollback:
-
-**Phase 1 - EBS Mount:** Sends `ebs.mount` via NATS to viperblock, which starts an NBD server and returns a URI (unix socket or TCP). If this fails, no cleanup is needed since nothing was mounted.
-
-**Phase 2 - QMP blockdev-add:** Registers the NBD block device in QEMU. Uses `nbd-{volumeId}` as the node-name for predictable identification during future detach. The `export` field is set to `""` per viperblock convention (default export). If this fails, Phase 1 is rolled back via `ebs.unmount`.
-
-**Phase 3 - QMP device_add:** Attaches the block device to the guest as `virtio-blk-pci` with id `vdisk-{volumeId}`. If this fails, Phase 2 is rolled back via `blockdev-del`, then Phase 1 via `ebs.unmount`. The `blockdev-del` rollback error is checked: if it fails, the EBS unmount is skipped to avoid tearing down storage that QEMU still references (which could crash the VM).
-
-After all three phases succeed, the handler updates in-memory state (`EBSRequests`, `BlockDeviceMappings`), persists to Predastore (volume metadata) and JetStream (instance state), then responds with `ec2.VolumeAttachment` JSON.
-
-### Volume Service: `hive/handlers/ec2/volume/service_impl.go`
-
-`getVolumeConfig` was renamed to `GetVolumeConfig` (exported directly instead of using a wrapper) so the daemon can validate that a volume exists and check its state before attempting attachment.
-
-`UpdateVolumeState` was extended with a `deviceName` parameter rather than creating a separate `UpdateVolumeAttachment` method. The existing callers pass `""` for deviceName since boot volumes don't track device names this way. This avoids near-duplicate methods.
-
-### Volume Service Interface: `hive/handlers/ec2/volume/service.go`
-
-The `VolumeService` interface is kept to AWS-facing operations only (CreateVolume, DescribeVolumes, ModifyVolume, DeleteVolume). `GetVolumeConfig` and `UpdateVolumeState` are methods on the concrete `VolumeServiceImpl` type, not the interface, because they are internal operations used by the daemon rather than AWS API endpoints. The daemon holds a `*VolumeServiceImpl` directly so it can call these methods without polluting the interface that the gateway-side `NATSVolumeService` also implements.
-
-### Device Name Assignment: `hive/daemon/daemon.go`
-
-`nextAvailableDevice` scans both `EBSRequests` and `BlockDeviceMappings` to find the next available `/dev/sd[f-p]` slot. Both collections are checked because there's a window during attachment where a volume appears in `EBSRequests` but not yet in `BlockDeviceMappings`. Returns `""` when all 11 slots are exhausted, which the handler maps to `AttachmentLimitExceeded`.
-
-The `/dev/sd[f-p]` range follows the AWS convention. The guest OS sees these as `/dev/vd*` when using virtio drivers, but the metadata device name is the `/dev/sd*` form per AWS API compatibility.
-
-### Rollback Helper: `hive/daemon/daemon.go`
-
-`rollbackEBSMount` sends `ebs.unmount` via NATS and verifies the response: checks both the `Error` field and the `Mounted` boolean. Rollback failures are logged at Error level but not propagated to callers since rollback is best-effort cleanup during an already-failed operation. This is a deliberate tradeoff: propagating rollback errors would complicate the handler without changing the user-visible outcome (the attach already failed).
-
-### Config: `hive/config/config.go`
-
-Added `DeviceName` field to `EBSRequest`. This tracks which `/dev/sd*` device name is assigned to a hot-plugged volume. Boot-time volumes don't use this field since their device assignment is handled by QEMU command-line args.
-
-### NBD URI Parsing: `hive/utils/utils.go`
-
-`ParseNBDURI` is the inverse of the existing `FormatNBDSocketURI`/`FormatNBDTCPURI` functions. It extracts the components needed for QMP's `blockdev-add` server argument:
-- `nbd:unix:/path/to/socket.sock` -> `type=unix, path=/path/to/socket.sock`
-- `nbd://host:port` -> `type=inet, host=host, port=port`
-
-## Design Decisions
-
-**Why QMP hot-plug instead of restarting the VM?** Hot-plug via `blockdev-add` + `device_add` attaches the volume without downtime. This matches AWS behavior where `attach-volume` works on running instances immediately.
-
-**Why `nbd-{volumeId}` and `vdisk-{volumeId}` naming?** These are predictable identifiers derived from the volume ID. This makes future `detach-volume` straightforward: the daemon can construct the node-name and device-id from the volume ID without maintaining a separate mapping.
-
-**Why `export: ""`?** Viperblock's NBD server uses a blank export name as the default. This is a viperblock convention, not a universal NBD convention.
-
-**Why rollback skips EBS unmount when blockdev-del fails?** If `blockdev-del` fails, QEMU still has a reference to the NBD backend. Unmounting the EBS volume would tear down the NBD server while QEMU still tries to do I/O through it, potentially crashing or hanging the VM. Leaving the mount in place is the safer failure mode: the volume is leaked but the VM stays healthy.
-
-**Why `DeleteOnTermination` defaults to false?** This matches AWS behavior: hot-attached volumes default to `DeleteOnTermination: false`. Boot volumes set this to true. Users can change it via `modify-instance-attribute`.
-
-## Error Codes
-
-| Condition | AWS Error Code |
-|-----------|---------------|
-| Missing VolumeId/InstanceId | `InvalidParameterValue` |
-| Volume not found in Predastore | `InvalidVolume.NotFound` |
-| Volume already in-use | `VolumeInUse` |
-| Instance not found (no NATS responder) | `InvalidInstanceID.NotFound` |
-| Instance not running | `IncorrectInstanceState` |
-| All device slots full | `AttachmentLimitExceeded` |
-| Internal failures (mount, QMP, etc.) | `InternalError` |
-
-## Testing
-
-**Gateway validation** (`AttachVolume_test.go`): Table-driven tests for nil input, empty input, missing fields, empty strings, valid inputs with/without device. Follows the pattern from `CreateVolume_test.go`.
-
-**Device name assignment** (`daemon_test.go`): 6 scenarios covering empty instance, existing block device mappings, existing EBS requests, all devices used, nil instance, mixed sources.
-
-**Handler error paths** (`daemon_test.go`): Tests for missing attach volume data, instance not running, and volume not found. Uses a real embedded NATS server (not mocks). The happy path is not currently testable because `volumeService` is a concrete type — testing the full flow would require introducing interface-based dependency injection for the volume service and QMP client.
-
-**NBD URI parsing** (`utils_test.go`): 8 cases covering both formats, empty paths, missing ports, invalid ports, unsupported formats.
-
-## Known Limitations
-
-- No validation of user-supplied device names against the `/dev/sd[f-p]` range or conflict checking.
-- TOCTOU gap between checking instance status/volume state and performing the actual QMP operations. A concurrent stop could race with an attach.
-- The `Attributes` struct uses boolean flags for mutually exclusive operations. A string-based action field would be structurally safer but is a breaking wire-format change.
-- Volume metadata and JetStream state persistence failures after a successful attach are logged but do not cause the response to indicate failure, since the volume is already physically attached to the VM.

docs/development/feature/describe-volume-status.md

@@ -1,114 +0,0 @@
-# EC2 DescribeVolumeStatus
-
-Report the health and status of EBS volumes, implementing the AWS `ec2 describe-volume-status` API. Returns IO-enabled status and volume health for all or specific volumes.
-
-## Request Flow
-
-```
-AWS SDK (describe-volume-status)
-  -> Gateway (port 9999): parse AWS query params, validate vol- prefix
-    -> NATS ec2.DescribeVolumeStatus: generic topic (no instance affinity)
-      -> Daemon DescribeVolumeStatus handler
-        Fast path (VolumeIds provided):
-          1. Parallel GetVolumeConfig for each volume ID
-          2. Build VolumeStatusItem per volume
-        Slow path (no VolumeIds):
-          1. S3 ListObjects to find vol-* prefixes
-          2. Skip internal sub-volumes (-efi, -cloudinit)
-          3. GetVolumeConfig + build VolumeStatusItem per volume
-      <- NATS response: DescribeVolumeStatusOutput JSON
-    <- Gateway: wrap in XML as DescribeVolumeStatusResponse
-  <- AWS SDK: DescribeVolumeStatusOutput
-```
-
-## Files Created
-
-### Gateway Handler: `hive/gateway/ec2/volume/DescribeVolumeStatus.go`
-
-`ValidateDescribeVolumeStatusInput` validates the vol- prefix on all VolumeIds. Follows the same pattern as `ValidateDescribeVolumesInput` — nil input is valid (means "all volumes"), nil entries in the VolumeIds slice are skipped, any non-`vol-` prefixed ID returns `InvalidVolume.Malformed`.
-
-`DescribeVolumeStatus` validates input, creates a `NATSVolumeService`, and delegates to `volumeService.DescribeVolumeStatus(input)`.
-
-### Gateway Tests: `hive/gateway/ec2/volume/DescribeVolumeStatus_test.go`
-
-9 table-driven test cases mirroring `DescribeVolumes_test.go`: nil input, empty input, valid single ID, valid multiple IDs, no prefix, wrong prefix, empty string, mixed valid/invalid, nil in list.
-
-## Files Modified
-
-### Gateway Switch: `hive/gateway/ec2.go`
-
-Added `"DescribeVolumeStatus"` case to the EC2 action switch. Parses `ec2.DescribeVolumeStatusInput`, calls the gateway handler, wraps result as `"DescribeVolumeStatusResponse"` XML.
-
-### Volume Service Interface: `hive/handlers/ec2/volume/service.go`
-
-Added `DescribeVolumeStatus(input *ec2.DescribeVolumeStatusInput) (*ec2.DescribeVolumeStatusOutput, error)` to the `VolumeService` interface. This is an AWS-facing operation so it belongs on the interface (unlike internal helpers like `GetVolumeConfig`).
-
-### NATS Client: `hive/handlers/ec2/volume/service_nats.go`
-
-Added `DescribeVolumeStatus` method following the same marshal-request-validate-unmarshal pattern as `DescribeVolumes`. Sends to `ec2.DescribeVolumeStatus` topic with 30s timeout.
-
-### Service Implementation: `hive/handlers/ec2/volume/service_impl.go`
-
-Three new methods:
-
-**`DescribeVolumeStatus`** — Entry point. Defaults nil input to empty. Takes the fast path (parallel by-ID lookup) when VolumeIds are provided, or the slow path (S3 ListObjects with vol-* filtering) when no IDs are specified. Same dual-path pattern as `DescribeVolumes`.
-
-**`fetchVolumeStatusByIDs`** — Parallel goroutine fan-out with mutex-protected slice append, matching the `fetchVolumesByIDs` pattern. Volumes that fail to load are logged and skipped.
-
-**`getVolumeStatusByID`** — Reads volume config via the existing `GetVolumeConfig` and builds an `ec2.VolumeStatusItem` with:
-- `VolumeId` and `AvailabilityZone` from volume metadata
-- `VolumeStatus`: status `"ok"` with two details entries
-- `Actions`: empty slice
-- `Events`: empty slice
-
-### Service Implementation Tests: `hive/handlers/ec2/volume/service_impl_test.go`
-
-Added `TestDescribeVolumeStatus_NilInputDefaults` — verifies that nil input is correctly defaulted to an empty input (the nil guard runs) before reaching the S3 layer.
-
-## Volume Status Details
-
-Every volume currently returns the same static status:
-
-| Detail | Value | Reason |
-|--------|-------|--------|
-| `io-enabled` | `passed` | Viperblock doesn't have degraded-IO states |
-| `io-performance` | `not-applicable` | Viperblock doesn't track this metric |
-| Status | `ok` | No health degradation model yet |
-| Actions | empty | No pending actions |
-| Events | empty | No scheduled events |
-
-This matches AWS behavior for healthy volumes. When viperblock adds health monitoring, these values can be derived from actual backend state.
-
-## Error Codes
-
-| Condition | AWS Error Code |
-|-----------|---------------|
-| Invalid volume ID format (no vol- prefix) | `InvalidVolume.Malformed` |
-| S3 listing failure | `InternalError` |
-| Volume not found | Silently skipped (matching DescribeVolumes behavior) |
-
-## Design Decisions
-
-**Why reuse the DescribeVolumes patterns?** DescribeVolumeStatus has the same data access pattern: optionally filter by volume IDs, otherwise list all. Reusing `GetVolumeConfig`, the parallel fan-out helper, and the S3 listing/filtering logic keeps the implementation consistent and avoids duplication.
-
-**Why static status values?** Viperblock currently has no concept of degraded IO or volume health checks. Returning `"ok"` / `"passed"` is accurate — if a volume's config is readable from S3, it is healthy. When viperblock adds health monitoring, `getVolumeStatusByID` is the single point to update.
-
-**Why empty Actions and Events?** AWS uses these for scheduled maintenance events (e.g., `enable-volume-io`, `retire-volume`). Hive doesn't have scheduled maintenance, so these are always empty.
-
-**Why no Filters/MaxResults/NextToken?** Consistent with the existing DescribeVolumes implementation. These can be added in a future iteration across all Describe* endpoints.
-
-## Testing
-
-```sh
-# All volumes
-aws --endpoint-url https://localhost:9999 ec2 describe-volume-status
-
-# Specific volumes
-aws --endpoint-url https://localhost:9999 ec2 describe-volume-status --volume-ids vol-abc123
-```
-
-## Known Limitations
-
-- Filters, MaxResults, and NextToken are accepted but ignored.
-- Volume status is always `"ok"` regardless of actual backend health.
-- The slow path (list all volumes) is sequential per-volume after the S3 list call, unlike the fast path which is parallel.

docs/development/feature/detach-volume.md

@@ -1,99 +0,0 @@
-# EC2 DetachVolume
-
-Hot-unplug an EBS volume from a running instance via QMP, implementing the AWS `ec2 detach-volume` API. This is the inverse of `attach-volume`.
-
-## Request Flow
-
-```
-AWS SDK (detach-volume)
-  -> Gateway (port 9999): parse AWS query params, validate input
-    -> NATS ec2.cmd.{instanceId}: per-instance topic routing
-      -> Daemon handleEC2Events: owns the instance's QEMU process
-        1. Validate volume is attached to instance
-        2. Reject boot/EFI/CloudInit volumes
-        3. QMP device_del -> remove virtio-blk-pci from guest
-        4. QMP blockdev-del -> remove NBD block device from QEMU
-        5. ebs.unmount via NATS -> viperblock stops NBD server
-        6. Update instance metadata + volume metadata
-      <- NATS response: ec2.VolumeAttachment JSON
-    <- Gateway: wrap in XML
-  <- AWS SDK: VolumeAttachment response
-```
-
-## Instance ID Resolution
-
-Unlike AttachVolume, DetachVolume does not require an InstanceId parameter (per the AWS API spec). If InstanceId is omitted, the gateway resolves it by calling DescribeVolumes via NATS to look up the volume's current attachment. If the volume has no attachments, it returns `IncorrectState`.
-
-## Files Changed
-
-### Wire Format: `hive/qmp/qmp.go`
-
-Added `DetachVolume bool` to `Attributes` and a new `DetachVolumeData` struct with `VolumeID`, `Device` (optional), and `Force` (optional). Carried as a pointer with omitempty on `Command`, matching the AttachVolumeData pattern.
-
-### Gateway Handler: `hive/gateway/ec2/volume/DetachVolume.go`
-
-Validates `VolumeId` is present (InstanceId, Device, Force are optional). If InstanceId is not provided, resolves it via `NewNATSVolumeService.DescribeVolumes()`. Constructs `qmp.Command` with `DetachVolume: true` and sends to `ec2.cmd.{instanceId}`.
-
-### Gateway Switch: `hive/gateway/ec2.go`
-
-Added `"DetachVolume"` case to the EC2 action switch statement.
-
-### Daemon Handler: `hive/daemon/daemon_handlers.go`
-
-The core logic, placed after AttachVolume in `handleEC2Events`. Implements a three-phase hot-unplug (reverse order of attach):
-
-**Validation:**
-- Volume must be in instance's `EBSRequests`
-- Boot, EFI, and CloudInit volumes cannot be detached
-- Optional device name cross-check against stored device name
-
-**Phase 1 - QMP device_del:** Removes the virtio-blk-pci device from the guest. Uses `vdisk-{volumeId}` as the device ID (set during attach). If this fails and Force is false, the operation aborts. With Force=true, a warning is logged and execution continues.
-
-**Phase 2 - QMP blockdev-del:** Removes the NBD block device node from QEMU. Uses `nbd-{volumeId}` as the node-name. If this fails, Phase 3 is skipped because the block node is still referenced by QEMU — unmounting would crash the VM.
-
-**Phase 3 - ebs.unmount:** Sends `ebs.unmount` via NATS to viperblock to stop the NBD server. Only executed if Phase 2 succeeded.
-
-**State cleanup (always runs after QMP phases):**
-- Remove volume from `instance.EBSRequests.Requests`
-- Remove matching entry from `instance.Instance.BlockDeviceMappings`
-- `UpdateVolumeState(volumeID, "available", "", "")` — clears attachment in S3
-- `WriteState()` — persist instance to JetStream
-
-**Response:** `ec2.VolumeAttachment` with `State: "detaching"`
-
-## Error Codes
-
-| Condition | AWS Error Code |
-|-----------|---------------|
-| Missing VolumeId | `InvalidParameterValue` |
-| Volume not found (gateway lookup) | `InvalidVolume.NotFound` |
-| Volume not attached | `IncorrectState` |
-| Instance not found (no NATS responder) | `InvalidInstanceID.NotFound` |
-| Instance not running | `IncorrectInstanceState` |
-| Boot/EFI/CloudInit volume | `OperationNotPermitted` |
-| Device mismatch | `InvalidParameterValue` |
-| Internal failures | `ServerInternal` |
-
-## Design Decisions
-
-**Why three phases in reverse order?** The attach order is mount -> blockdev-add -> device_add. Detach reverses this: device_del -> blockdev-del -> unmount. The guest device must be removed before the block node, and the block node before the NBD server, to avoid I/O errors or VM crashes.
-
-**Why skip unmount when blockdev-del fails?** If QEMU still references the block node, tearing down the NBD server would cause I/O errors in the VM. The volume is leaked but the VM stays healthy. This matches the same safety rationale used in attach rollback.
-
-**Why the 1-second sleep between device_del and blockdev-del?** The guest OS needs time to acknowledge the PCI device removal. Without this pause, blockdev-del may fail because the device still has a reference to the block node.
-
-**Why allow Force=true to continue past device_del failure?** This matches AWS `--force` behavior for stuck volumes. The device may already be partially removed or the guest may be unresponsive.
-
-**Why state cleanup runs even when QMP phases partially fail?** The metadata should reflect the detach intent. If the QMP commands partially succeeded, the volume is in an inconsistent state from QEMU's perspective but the metadata should be cleaned up so the volume can be re-attached or managed.
-
-## Testing
-
-**Gateway validation** (`DetachVolume_test.go`): Table-driven tests for nil input, empty input, missing VolumeId, empty VolumeId, valid inputs with various optional field combinations.
-
-**Daemon handler tests** (`daemon_test.go`): Tests for missing DetachVolumeData, instance not running, volume not attached, and boot volume protection.
-
-## Known Limitations
-
-- TOCTOU gap between finding the volume in EBSRequests and performing QMP operations.
-- No timeout on the 1-second sleep between device_del and blockdev-del.
-- Force mode continues past device_del failure but does not retry.

docs/development/feature/eigw.md

@@ -1,194 +0,0 @@
-# Egress-Only Internet Gateway (EIGW)
-
-## Overview
-
-Egress-Only Internet Gateways provide outbound-only IPv6 connectivity for instances in a VPC. In Hive, EIGWs are lightweight metadata objects persisted in NATS JetStream KV. The implementation covers the three standard EC2 API operations: Create, Delete, and Describe.
-
-## Key Design Decisions
-
-### 1. NATS JetStream KV for persistence
-
-EIGW records are stored in a dedicated KV bucket (`hive-eigw`) with 10-revision history. This avoids introducing a database dependency -- the same NATS infrastructure used for messaging also handles persistence.
-
-The `getOrCreateKVBucket` helper tries `CreateKeyValue` first, then falls back to `KeyValue` (open existing). This makes the service idempotent across restarts.
-
-**Trade-off**: KV is not suited for complex queries (e.g. filtering by tag values). Describe currently iterates all keys and filters in-memory, which is fine for typical EIGW counts but wouldn't scale to thousands of records.
-
-### 2. Constructor fails loudly, daemon falls back gracefully
-
-`NewEgressOnlyIGWServiceImplWithNATS` returns an error if the KV bucket can't be created or opened. This ensures the service never operates with a nil KV handle, eliminating an entire class of nil-pointer bugs.
-
-The daemon handles this with a graceful fallback:
-
-```go
-if eigwSvc, err := handlers_ec2_eigw.NewEgressOnlyIGWServiceImplWithNATS(d.config, d.natsConn); err != nil {
-    slog.Warn("Failed to initialize EIGW service, falling back to in-memory", "error", err)
-    d.eigwService = handlers_ec2_eigw.NewEgressOnlyIGWServiceImpl(d.config)
-} else {
-    d.eigwService = eigwSvc
-}
-```
-
-**Trade-off**: The in-memory fallback (`NewEgressOnlyIGWServiceImpl`) has no KV backing, so EIGW state won't survive daemon restarts. This is acceptable for degraded mode -- the alternative is refusing to start the daemon entirely.
-
-### 3. crypto/rand for ID generation
-
-EIGW IDs (`eigw-` + 16 hex chars) are generated using `crypto/rand` + `encoding/hex`. This matches the pattern used by instance and volume ID generation elsewhere in the codebase.
-
-```go
-func generateEgressOnlyIGWID() string {
-    b := make([]byte, 8)
-    if _, err := rand.Read(b); err != nil {
-        panic("crypto/rand failed: " + err.Error())
-    }
-    return "eigw-" + hex.EncodeToString(b)
-}
-```
-
-The panic on `crypto/rand` failure is intentional -- if the OS entropy source is broken, the system has larger problems.
-
-### 4. Error strings are awserrors constants
-
-All errors returned from `service_impl.go` use `errors.New(awserrors.ErrorXxx)` with exact constant values. This is critical because the error string travels through a round-trip:
-
-```
-service_impl: errors.New(awserrors.ErrorMissingParameter)
-  -> daemon handleNATSRequest: utils.GenerateErrorPayload(err.Error())
-    -> JSON over NATS: {"Code": "MissingParameter"}
-      -> gateway NATSRequest[Out]: ValidateErrorPayload extracts Code
-        -> gateway ErrorHandler: awserrors.ErrorLookup[err.Error()] -> HTTP status + XML
-```
-
-If the error string doesn't match a key in `ErrorLookup`, the gateway returns a generic 500. Using `fmt.Errorf` with extra context would break this chain.
-
-### 5. Centralized nil NATS check in the gateway
-
-Rather than checking `natsConn == nil` in each gateway function, a single check in `EC2_Request` covers all NATS-dependent actions. An allowlist (`ec2LocalActions`) exempts actions that work without NATS:
-
-```go
-var ec2LocalActions = map[string]bool{
-    "DescribeRegions":           true,
-    "DescribeAvailabilityZones": true,
-    "DescribeAccountAttributes": true,
-}
-```
-
-If `gw.NATSConn` is nil and the action isn't in the allowlist, the gateway returns `ErrorServerInternal` immediately.
-
-### 6. Tag filtering by resource type
-
-Create accepts `TagSpecifications` but only applies tags where `ResourceType` equals `"egress-only-internet-gateway"`. Tags with other resource types (e.g. `"instance"`) are silently ignored, matching AWS behavior.
-
-## Request Flow
-
-```
-AWS SDK (create-egress-only-internet-gateway --vpc-id vpc-xxx)
-  -> Gateway (port 9999): ec2Handler parses AWS query params into SDK struct
-    -> Validate input (nil check, required fields)
-    -> NATS ec2.CreateEgressOnlyInternetGateway: routed to daemon via "hive-workers" queue group
-      -> Daemon handleEC2CreateEgressOnlyInternetGateway
-        -> handleNATSRequest: unmarshal -> EgressOnlyIGWServiceImpl.Create -> marshal -> respond
-          1. Validate VpcId is present
-          2. Generate eigw-{16hex} ID via crypto/rand
-          3. Build EgressOnlyIGWRecord with state "attached"
-          4. Process matching tag specifications
-          5. JSON marshal and store in NATS KV (hive-eigw bucket)
-        <- NATS response: ec2.CreateEgressOnlyInternetGatewayOutput JSON
-    <- Gateway: wrap in XML (CreateEgressOnlyInternetGatewayResponse)
-  <- AWS SDK: response
-```
-
-## NATS Topics
-
-| Topic | Direction | Purpose |
-|-------|-----------|---------|
-| `ec2.CreateEgressOnlyInternetGateway` | Gateway -> Daemon | Create new EIGW attached to a VPC |
-| `ec2.DeleteEgressOnlyInternetGateway` | Gateway -> Daemon | Delete EIGW by ID |
-| `ec2.DescribeEgressOnlyInternetGateways` | Gateway -> Daemon | List/filter EIGWs |
-
-All topics use the `hive-workers` queue group for load balancing across daemon instances.
-
-## Storage Layout
-
-```
-NATS JetStream KV bucket: hive-eigw (history: 10)
-
-Key: eigw-a1b2c3d4e5f6g7h8
-Value: {
-  "egress_only_internet_gateway_id": "eigw-a1b2c3d4e5f6g7h8",
-  "vpc_id": "vpc-abc123",
-  "state": "attached",
-  "tags": {"Name": "my-eigw", "Env": "prod"},
-  "created_at": "2026-02-10T12:00:00Z"
-}
-```
-
-## Files Changed
-
-| File | Change |
-|------|--------|
-| `hive/handlers/ec2/eigw/service.go` | `EgressOnlyIGWService` interface (3 methods) |
-| `hive/handlers/ec2/eigw/service_impl.go` | Direct implementation with NATS JetStream KV persistence |
-| `hive/handlers/ec2/eigw/service_impl_test.go` | 12 test cases against embedded NATS server |
-| `hive/handlers/ec2/eigw/service_nats.go` | NATS proxy (gateway -> daemon via `utils.NATSRequest`) |
-| `hive/gateway/ec2/eigw/CreateEgressOnlyInternetGateway.go` | Gateway validation + handler for Create |
-| `hive/gateway/ec2/eigw/DeleteEgressOnlyInternetGateway.go` | Gateway validation + handler for Delete |
-| `hive/gateway/ec2/eigw/DescribeEgressOnlyInternetGateways.go` | Gateway validation + handler for Describe |
-| `hive/gateway/ec2/eigw/eigw_test.go` | Gateway validation tests (9 cases) |
-| `hive/gateway/ec2.go` | 3 entries in `ec2Actions` map; centralized nil NATS check |
-| `hive/daemon/daemon.go` | `eigwService` field, 3 NATS subscriptions, service initialization with fallback |
-| `hive/daemon/daemon_handlers.go` | 3 handler functions delegating to `handleNATSRequest` |
-| `hive/awserrors/awserrors.go` | `ErrorInvalidEgressOnlyInternetGatewayId.Malformed` and `.NotFound` constants + lookup entries |
-
-## Test Coverage
-
-### Service Implementation Tests (`service_impl_test.go`)
-
-| Test | What it validates |
-|------|-------------------|
-| `TestCreateEgressOnlyInternetGateway` | Basic create: ID format, VPC attachment, state |
-| `TestCreateEgressOnlyInternetGateway_MissingVpcId` | Rejects nil VpcId with MissingParameter |
-| `TestCreateEgressOnlyInternetGateway_EmptyVpcId` | Rejects empty string VpcId |
-| `TestCreateEgressOnlyInternetGateway_WithTags` | Tags applied and persisted through describe round-trip |
-| `TestCreateEgressOnlyInternetGateway_TagsWrongResourceType` | Tags with non-matching resource type are ignored |
-| `TestDeleteEgressOnlyInternetGateway` | Delete succeeds, describe confirms removal |
-| `TestDeleteEgressOnlyInternetGateway_NotFound` | Returns NotFound for nonexistent ID |
-| `TestDeleteEgressOnlyInternetGateway_MissingID` | Rejects nil ID with MissingParameter |
-| `TestDeleteEgressOnlyInternetGateway_EmptyID` | Rejects empty string ID |
-| `TestDescribeEgressOnlyInternetGateways_All` | Lists all EIGWs when no filter specified |
-| `TestDescribeEgressOnlyInternetGateways_ByID` | Filters by specific EIGW ID |
-| `TestDescribeEgressOnlyInternetGateways_Empty` | Returns empty list for empty bucket |
-
-All tests run against an embedded NATS server with JetStream enabled.
-
-### Gateway Validation Tests (`eigw_test.go`)
-
-| Test | What it validates |
-|------|-------------------|
-| `ValidateCreateEgressOnlyInternetGatewayInput` | nil input, missing VpcId, empty VpcId, valid input |
-| `ValidateDeleteEgressOnlyInternetGatewayInput` | nil input, missing ID, empty ID, valid input |
-| `ValidateDescribeEgressOnlyInternetGatewaysInput` | nil input, empty input, with filter IDs |
-
-## AWS CLI Usage
-
-```bash
-# Create
-aws ec2 create-egress-only-internet-gateway \
-  --vpc-id vpc-abc123 \
-  --tag-specifications 'ResourceType=egress-only-internet-gateway,Tags=[{Key=Name,Value=my-eigw}]' \
-  --endpoint-url https://localhost:9999
-
-# Describe all
-aws ec2 describe-egress-only-internet-gateways \
-  --endpoint-url https://localhost:9999
-
-# Describe by ID
-aws ec2 describe-egress-only-internet-gateways \
-  --egress-only-internet-gateway-ids eigw-a1b2c3d4e5f6g7h8 \
-  --endpoint-url https://localhost:9999
-
-# Delete
-aws ec2 delete-egress-only-internet-gateway \
-  --egress-only-internet-gateway-id eigw-a1b2c3d4e5f6g7h8 \
-  --endpoint-url https://localhost:9999
-```

docs/development/feature/heartbeat-cluster-shutdown.md

@@ -1,78 +0,0 @@
-# Phase 2.1 Heartbeats + Phase 2.4 Coordinated Cluster Shutdown
-
-**Status: Complete**
-
-## Summary
-
-Implemented daemon heartbeats (Phase 2.1) and coordinated cluster shutdown (Phase 2.4). Heartbeats publish node status to KV every 10s. Coordinated shutdown drives a phased shutdown across all nodes via NATS, replacing the error-prone per-node approach for multi-node clusters.
-
-## Context / Problem Statement
-
-- No cluster-wide health awareness — nodes don't know peer status
-- Multi-node shutdown required manual per-node `stop-dev.sh`, which is error-prone and doesn't guarantee correct phase ordering (VMs must stop before storage, storage before NATS)
-- Phase 0+1 (capabilities, recovery throttling, shutdown markers) were already complete
-
-## Design Decisions
-
-- **Option B chosen**: Daemons handle stopping local services internally via `utils.StopProcess()` when they receive NATS shutdown phase messages. The CLI coordinator just sends messages and collects ACKs.
-- **Heartbeat interval**: 10 seconds — fast enough for health monitoring, low overhead
-- **Shutdown phases**: GATE → DRAIN → STORAGE → PERSIST → INFRA — ordered to prevent data loss
-- **INFRA phase is fire-and-forget**: NATS is going down, no ACK possible
-- **Scripts**: `stop-dev.sh` auto-detects multi-node and delegates to `hive admin cluster shutdown`; `HIVE_FORCE_LOCAL_STOP=1` overrides for per-service stop
-
-## Proposed Changes (Implemented)
-
-### Heartbeat System
-- `Heartbeat` struct in `jetstream.go` with node, epoch, services, VM count, resource allocation
-- `WriteHeartbeat`/`ReadHeartbeat` KV methods on `JetStreamManager`
-- `startHeartbeat()` goroutine in `heartbeat.go` — fires immediately, then every 10s
-- `buildHeartbeat()` reads from `ResourceManager` and `Instances`
-
-### Coordinated Shutdown
-- Five shutdown phase handlers in `shutdown.go`: GATE, DRAIN, STORAGE, PERSIST, INFRA
-- `ShutdownRequest`, `ShutdownACK`, `ShutdownProgress` message types
-- `shuttingDown atomic.Bool` on Daemon — set during GATE, checked by `setupShutdown()` to skip redundant VM stops
-- NATS subscriptions for `hive.cluster.shutdown.{phase}` (fan-out, no queue group)
-- Progress updates published during DRAIN phase
-
-### CLI Coordinator
-- `hive admin cluster shutdown` command in `cluster.go`
-- Flags: `--force`, `--timeout`, `--dry-run`
-- Executes phases sequentially, collecting ACKs with timeout
-- Writes `ClusterShutdownState` to KV for observability
-- Live progress display during DRAIN phase
-
-### Script Updates
-- `stop-dev.sh`: `is_multinode()` detection, delegates to coordinated shutdown
-- `start-dev.sh`: `is_multinode()` detection, cluster health poll at end
-
-## Files Modified
-
-| File | Action | Description |
-|------|--------|-------------|
-| `hive/daemon/jetstream.go` | Edit | `Heartbeat`, `ClusterShutdownState` structs + KV methods |
-| `hive/daemon/heartbeat.go` | New | `startHeartbeat()`, `buildHeartbeat()` goroutine |
-| `hive/daemon/shutdown.go` | New | Shutdown types + 5 phase handlers |
-| `hive/daemon/daemon.go` | Edit | `shuttingDown atomic.Bool`, shutdown subscriptions, `startHeartbeat()`, updated `setupShutdown()` |
-| `cmd/hive/cmd/admin.go` | Edit | `clusterCmd` + `clusterShutdownCmd` registration |
-| `cmd/hive/cmd/cluster.go` | New | `runClusterShutdown()` coordinator |
-| `scripts/stop-dev.sh` | Edit | Multi-node detection + coordinated shutdown delegation |
-| `scripts/start-dev.sh` | Edit | Multi-node detection + cluster health poll |
-| `hive/daemon/heartbeat_test.go` | New | Heartbeat tests (build, allocation, KV round-trip) |
-| `hive/daemon/shutdown_test.go` | New | Shutdown tests (marshal, KV round-trip, flag behavior) |
-
-## Testing
-
-- `make preflight` — all checks pass (gofmt, go vet, gosec, staticcheck, govulncheck, all tests)
-- 9 new unit tests covering heartbeat building, resource reflection, KV round-trips, shutdown message marshaling, and shutdown state management
-- `hive admin cluster shutdown --dry-run` prints phase plan
-- `stop-dev.sh` on multi-node delegates to coordinated shutdown
-- `HIVE_FORCE_LOCAL_STOP=1 stop-dev.sh` on multi-node uses per-service stop
-- `stop-dev.sh` on single-node uses per-service stop (unchanged behavior)
-
-## Future Work
-
-- Phase 2.2: Heartbeat-driven health monitoring (stale heartbeat detection, node failure alerts)
-- Phase 2.3: Automatic VM migration on node failure
-- Integration with `hive top` to display heartbeat data
-- Cluster shutdown progress bar in UI

docs/development/feature/multi-node-predastore-config.md

@@ -1,154 +0,0 @@
-# Multi-Node Predastore Configuration
-
-## Overview
-
-Predastore uses Reed-Solomon erasure coding RS(2,1) -- 2 data shards and 1 parity shard -- requiring a minimum of 3 distinct nodes. Each shard is placed on a separate node via a consistent hash ring (`GetClosestN(hash, 3)`), which means every node must know the full cluster topology at startup to make identical placement decisions.
-
-This is fundamentally different from NATS, which uses gossip-based discovery and can add members dynamically. Predastore's hash ring is static -- if nodes disagree about membership, they'll compute different shard placements and data becomes unreachable.
-
-## How It Works
-
-### Init (leader node)
-
-```
-hive admin init --predastore-nodes "10.11.12.1,10.11.12.2,10.11.12.3" --bind 10.11.12.1 ...
-```
-
-1. The comma-separated IPs are parsed and validated. Each gets a sequential node ID (1, 2, 3).
-2. The leader's `--bind` IP is matched against the list to determine its own node ID. If the bind IP isn't in the list, init fails.
-3. `GenerateMultiNodePredastoreConfig` renders a single `predastore.toml` containing the full cluster topology -- all DB entries (port 6660), all shard entries (port 9991), RS config, auth, and buckets. Node ID 1 is marked as the Raft bootstrap leader.
-4. The predastore.toml is written to `config/predastore/predastore.toml`.
-5. The leader's node ID is written into `hive.toml` as `node_id = 1` under the `[predastore]` section.
-
-### Join (follower nodes)
-
-```
-hive admin join --leader https://10.11.12.1:7777 --bind 10.11.12.2 ...
-```
-
-1. The follower contacts the leader's `/join` HTTP endpoint.
-2. The leader reads its `predastore.toml` from disk and includes the full content in the join response (`PredastoreConfig` field).
-3. The follower writes this verbatim to its own `config/predastore/predastore.toml`. Now all nodes have an identical copy.
-4. `ParsePredastoreNodeIDFromConfig` unmarshals the TOML, extracts the `[[db]]` array, and matches the follower's `--bind` IP to find its node ID. If the IP isn't found, join fails with an error.
-5. The detected node ID is written into the follower's `hive.toml` (e.g., `node_id = 2`).
-
-### Startup
-
-When `start-dev.sh` runs, it reads `node_id` from `hive.toml` using awk and exports it as `HIVE_PREDASTORE_NODE_ID`. Predastore reads this environment variable to know which DB and shard entries in the shared config belong to it.
-
-## Key Design Decisions
-
-### Shared config, per-node identity
-
-Every node gets the **same** `predastore.toml` (the full cluster topology). The only difference per node is `node_id` in `hive.toml`. This separation exists because:
-
-- Predastore needs all members known at startup for consistent hash ring agreement.
-- The `predastore.toml` describes the cluster. The `node_id` in `hive.toml` says "I am this member."
-- This makes config distribution simple: the leader just sends one file, and the follower figures out its own identity by matching its bind IP.
-
-### Why IPs must be specified at init time
-
-Unlike NATS (gossip discovery), Predastore requires static membership. The hash ring maps `GetClosestN(hash, 3)` to determine which 3 nodes store each object's shards. If a node joins without being in the ring, existing nodes won't route data to it. If a node is in the ring but missing, data placed on it becomes unavailable.
-
-This means all Predastore node IPs must be known when the leader is initialized. The `--predastore-nodes` flag captures this.
-
-### Node ID 1 is the Raft bootstrap leader
-
-The generated config marks `leader = true` on the `[[db]]` entry with `id = 1`. This is the node that bootstraps the Raft cluster. It's always the first IP in the `--predastore-nodes` list, which is also the `--bind` IP of the init command (the leader node).
-
-### Config distribution via join protocol
-
-Rather than requiring operators to manually copy `predastore.toml` to each node, the existing HTTP join protocol distributes it. The leader's daemon reads its predastore.toml and includes it in the join response. This is the same pattern used for CA certificates.
-
-The leader currently sends its predastore.toml unconditionally (even in single-node setups). The follower treats any non-empty `PredastoreConfig` in the response as "leader provided a multi-node config" and skips generating its own template-based config.
-
-### Node ID detection uses proper TOML parsing
-
-The follower determines its node ID by parsing the received predastore.toml with `go-toml/v2` and calling `FindNodeIDByIP` against the `[[db]]` entries. If parsing fails or the bind IP isn't found, join exits with an error. This mirrors the init path, which also exits on failure.
-
-## Backward Compatibility
-
-When `--predastore-nodes` is omitted from `hive admin init`:
-
-- No multi-node predastore.toml is generated.
-- `PredastoreNodeID` stays at 0, so `node_id` is not emitted in hive.toml.
-- The default single-node predastore.toml template is used instead.
-- `start-dev.sh` finds no `node_id` in hive.toml and leaves `HIVE_PREDASTORE_NODE_ID` empty.
-- Predastore runs in its default single-node mode.
-
-Existing single-node deployments are unaffected.
-
-## Files Involved
-
-| File | Role |
-|------|------|
-| `cmd/hive/cmd/admin.go` | CLI flag parsing, init/join orchestration |
-| `hive/admin/admin.go` | `GenerateMultiNodePredastoreConfig`, `FindNodeIDByIP`, `ParsePredastoreNodeIDFromConfig`, `BaseConfigFiles` |
-| `hive/config/config.go` | `PredastoreConfig.NodeID` field, `NodeJoinResponse.PredastoreConfig` field |
-| `hive/daemon/daemon.go` | `/join` endpoint reads and sends predastore.toml |
-| `cmd/hive/cmd/templates/hive.toml` | Conditional `node_id` rendering |
-| `scripts/start-dev.sh` | Auto-detects `NODE_ID` from hive.toml at startup |
-
-## Generated Config Example
-
-For `--predastore-nodes "10.11.12.1,10.11.12.2,10.11.12.3"`:
-
-```toml
-version = "1.0"
-region = "ap-southeast-2"
-
-host = "0.0.0.0"
-port = 8443
-
-[rs]
-data = 2
-parity = 1
-
-[[db]]
-id = 1
-host = "10.11.12.1"
-port = 6660
-path = "distributed/db/node-1/"
-leader = true
-
-[[db]]
-id = 2
-host = "10.11.12.2"
-port = 6660
-path = "distributed/db/node-2/"
-
-[[db]]
-id = 3
-host = "10.11.12.3"
-port = 6660
-path = "distributed/db/node-3/"
-
-[[nodes]]
-id = 1
-host = "10.11.12.1"
-port = 9991
-path = "distributed/nodes/node-1/"
-
-[[nodes]]
-id = 2
-host = "10.11.12.2"
-port = 9991
-path = "distributed/nodes/node-2/"
-
-[[nodes]]
-id = 3
-host = "10.11.12.3"
-port = 9991
-path = "distributed/nodes/node-3/"
-
-[[buckets]]
-name = "predastore"
-region = "ap-southeast-2"
-type = "distributed"
-
-[[auth]]
-access_key_id = "..."
-secret_access_key = "..."
-```
-
-Each node receives this identical file. Node 1's `hive.toml` has `node_id = 1`, node 2 has `node_id = 2`, etc.

docs/development/feature/snapshot.md

@@ -1,164 +0,0 @@
-# Viperblock Snapshot Feature
-
-## Overview
-
-Viperblock snapshots enable instant, zero-copy point-in-time captures of block storage volumes. A snapshot freezes the block-to-object mapping at a moment in time without copying any data. Both the snapshot and the live volume share the same immutable chunk files on S3. Volumes created from snapshots use a copy-on-write (COW) layered read path, reading unmodified blocks from the source volume's chunks and writing new data to their own chunks.
-
-This is the foundation for AMI-backed instance launches -- `aws ec2 run-instances --image-id ami-ubuntu` will boot a VM that reads Ubuntu blocks on demand from the AMI's frozen snapshot, with zero image copying.
-
-## Key Design Decisions
-
-### 1. Snapshots are frozen block maps, not data copies
-
-Viperblock's chunk files are immutable -- once `chunk.00000001.bin` is written, it never changes. New writes always produce new chunks with higher ObjectIDs. This means a snapshot only needs to capture the current `BlocksToObject` mapping (which block lives at which offset in which chunk file). The actual block data stays in place.
-
-**Trade-off**: This makes snapshot creation O(mapping size) instead of O(data size), but it means the source volume's chunk files cannot be garbage collected while any snapshot references them.
-
-### 2. Copy-on-write via layered block maps
-
-A clone volume created from a snapshot has two block maps:
-
-- **Own map** (`BlocksToObject`) -- blocks written since clone creation
-- **Base map** (`BaseBlockMap`) -- frozen read-only map from the parent snapshot
-
-The read path checks own map first, then falls back to the base map, then returns zeros:
-
-```
-Read block N:
-  1. Own BlocksToObject / BlockStore  -> found? return data from own chunks
-  2. BaseBlockMap                     -> found? return data from source volume's chunks via ReadFrom()
-  3. Neither                          -> zero block (never written)
-```
-
-Writes always go to the clone's own chunks. Once block N is written in the clone, it shadows the snapshot's version permanently.
-
-**Trade-off**: Reads of unmodified blocks require cross-volume I/O (reading from the source volume's S3 prefix). This is the same latency as a normal backend read but targets a different path. The BlockStore caches fetched base blocks so repeated reads are fast.
-
-### 3. Backend interface extended with ReadFrom/WriteTo
-
-The existing `Backend.Read()` and `Backend.Write()` methods construct S3 keys using the backend's configured volume name. Snapshots need to read/write across volume boundaries:
-
-- `CreateSnapshot` writes checkpoint and config to `{snapshotID}/` prefix
-- Clone volumes read chunk data from `{sourceVolumeName}/` prefix
-
-Rather than temporarily swapping the backend's volume name (error-prone, not thread-safe), we added two new methods to the `Backend` interface:
-
-```go
-ReadFrom(volumeName string, fileType FileType, objectId uint64, offset uint32, length uint32) ([]byte, error)
-WriteTo(volumeName string, fileType FileType, objectId uint64, headers *[]byte, data *[]byte) error
-```
-
-These are identical to `Read`/`Write` but use the provided volume name for path construction instead of the backend's configured name.
-
-**Trade-off**: This adds two methods to an interface that all backends must implement. The alternative was adding a `volumeName` parameter to the existing methods (breaks all callers) or using a separate backend instance per snapshot operation (heavier). The `ReadFrom`/`WriteTo` approach is minimal and backward-compatible.
-
-### 4. Snapshot state persisted in VBState
-
-Two fields were added to `VBState` (the serialized volume config):
-
-```go
-SnapshotID       string  // ID of parent snapshot (empty for non-clones)
-SourceVolumeName string  // Volume whose chunks the base map references
-```
-
-On `LoadState()`, if `SnapshotID` is set, `OpenFromSnapshot()` is called automatically to reload the base block map. This means a clone volume survives restarts -- the snapshot relationship is restored from the persisted config.
-
-### 5. Both read paths modified (BlockStore and legacy)
-
-Viperblock has two read paths: the optimized `readBlockStore()` (O(1) sharded index) and the legacy `read()` (map rebuilding). Both were modified identically:
-
-- When a block is not found in the volume's own data structures, check `BaseBlockMap`
-- Blocks found in the base map are collected into a separate `baseConsecutiveBlocks` list
-- These are fetched via `fetchBaseBlocksFromBackend()` which uses `ReadFrom(sourceVolumeName, ...)`
-- Fetched base blocks are cached in the BlockStore/LRU for subsequent reads
-
-The two-list approach (own blocks vs base blocks) avoids mixing `Read()` and `ReadFrom()` calls in the same fetch loop.
-
-### 6. Snapshot checkpoint reuses existing serialization
-
-The snapshot's frozen block map is serialized in the same binary format as `SaveBlockState()` -- a header followed by 26-byte `BlockLookup` entries with CRC32 checksums. `LoadSnapshotBlockMap()` deserializes using the same `readBlockWalChunk()` function.
-
-This means snapshot checkpoints are validated with the same integrity checks as regular checkpoints, and no new serialization code was needed.
-
-## S3 Layout
-
-```
-vol-abc123/                              # Source volume
-  config.json
-  chunks/chunk.00000001.bin              # Shared -- snapshot references these
-  chunks/chunk.00000002.bin
-  checkpoints/blocks.00000001.bin
-
-snap-xyz789/                             # Snapshot
-  config.json                            # SnapshotState metadata
-  checkpoints/blocks.00000000.bin        # Frozen block map at snapshot time
-
-vol-newclone/                            # Clone volume
-  config.json                            # VBState with SnapshotID set
-  chunks/chunk.00000001.bin              # Only new writes after clone creation
-  checkpoints/blocks.00000001.bin        # Only clone's own block mappings
-```
-
-## Files Changed (viperblock repo)
-
-| File | Change |
-|------|--------|
-| `types/types.go` | Added `ReadFrom`, `WriteTo` to `Backend` interface |
-| `viperblock/viperblock.go` | Added `BaseBlockMap`, `SourceVolumeName`, `SnapshotID` to VB struct. Added snapshot fields to `VBState`. Modified `readBlockStore()` and legacy `read()` for base map fallback. Added `fetchBaseBlocksFromBackend()`. Updated `SaveState`/`LoadState`. |
-| `viperblock/snapshot.go` | New file: `SnapshotState` type, `CreateSnapshot()`, `LoadSnapshotBlockMap()`, `OpenFromSnapshot()`, `LookupBaseBlockToObject()` |
-| `viperblock/backends/file/file.go` | Implemented `ReadFrom()`, `WriteTo()` |
-| `viperblock/backends/s3/s3.go` | Implemented `ReadFrom()`, `WriteTo()` |
-| `viperblock/backends/memory/memory.go` | Updated stub to satisfy new interface |
-| `viperblock/snapshot_test.go` | 6 test cases across file and S3 backends |
-
-## Test Coverage
-
-| Test | What it validates |
-|------|-------------------|
-| `TestCreateSnapshot` | Flush + checkpoint + config written to backend; round-trip via LoadSnapshotBlockMap |
-| `TestSnapshotReadFallback` | Clone reads unmodified blocks from source volume's chunks |
-| `TestSnapshotCopyOnWrite` | Overwritten blocks return new data; unmodified blocks return snapshot data |
-| `TestSnapshotZeroBlocks` | Blocks never written in snapshot or clone return zeros |
-| `TestSnapshotMultiChunk` | Snapshot spanning multiple 4MB chunk files |
-| `TestLoadFromSnapshot` | Clone survives save/reset/reload cycle with base map restored |
-
-All tests run against 4 backend configurations: file, file_nocache, s3, s3_nocache.
-
-## Hive API Integration
-
-### Request Flow
-
-```
-AWS SDK (create-snapshot --volume-id vol-xxx)
-  -> Gateway (port 9999): parse AWS query params, validate VolumeId
-    -> NATS ec2.CreateSnapshot: routed to daemon via queue group
-      -> Daemon handleEC2CreateSnapshot -> SnapshotServiceImpl.CreateSnapshot()
-        1. Fetch volume config from Predastore (validate volume exists)
-        2. NATS ebs.snapshot -> viperblockd: flush + freeze block map
-        3. Write snapshot metadata to Predastore ({snapId}/metadata.json)
-      <- NATS response: ec2.Snapshot JSON
-    <- Gateway: wrap in XML
-  <- AWS SDK: Snapshot response
-```
-
-### NATS Topics
-
-| Topic | Direction | Purpose |
-|-------|-----------|---------|
-| `ec2.CreateSnapshot` | Gateway -> Daemon | AWS API entry point |
-| `ebs.snapshot` | Daemon -> Viperblockd | Flush live VB instance and create frozen block map checkpoint |
-| `ec2.DescribeSnapshots` | Gateway -> Daemon | List/filter snapshots |
-| `ec2.DeleteSnapshot` | Gateway -> Daemon | Delete snapshot metadata and checkpoint |
-| `ec2.CopySnapshot` | Gateway -> Daemon | Copy snapshot metadata (metadata-only) |
-
-### Files Changed (hive repo)
-
-| File | Change |
-|------|--------|
-| `hive/handlers/ec2/snapshot/service.go` | `SnapshotService` interface |
-| `hive/handlers/ec2/snapshot/service_impl.go` | Direct implementation with S3 storage + NATS `ebs.snapshot` call |
-| `hive/handlers/ec2/snapshot/service_nats.go` | NATS proxy (gateway -> daemon) |
-| `hive/gateway/ec2/snapshot/CreateSnapshot.go` | Gateway validation + NATS dispatch |
-| `hive/daemon/daemon_handlers.go` | `handleEC2CreateSnapshot` wiring |
-| `hive/services/viperblockd/viperblockd.go` | `ebs.snapshot` subscription handler |
-| `hive/config/config.go` | `EBSSnapshotRequest` / `EBSSnapshotResponse` types |

docs/development/feature/start-instance-distributed-testing.md

@@ -1,369 +0,0 @@
-# Distributed StartInstances - Manual Testing Guide
-
-Manual test plan for verifying stop/start instance lifecycle across daemon nodes, including daemon restarts. Requires a multi-node cluster (see INSTALL.md "Multi-node Configuration").
-
-## Prerequisites
-
-```bash
-# Build
-make build
-
-# Set up 3-node cluster (if not already running)
-./scripts/create-multi-node.sh
-./scripts/start-dev.sh ~/node1/
-./scripts/start-dev.sh ~/node2/
-./scripts/start-dev.sh ~/node3/
-
-# AWS CLI setup
-export AWS_PROFILE=hive
-```
-
-Pick a gateway endpoint for all commands (any node works):
-
-```bash
-export EP="--endpoint-url https://10.11.12.1:9999"
-```
-
-Import an SSH key and AMI if not already done (see INSTALL.md).
-
----
-
-## Test 1: Basic Stop and Start (Same Cluster)
-
-Verify an instance can be stopped and restarted by any daemon.
-
-```bash
-# 1. Launch an instance
-aws $EP ec2 run-instances \
-  --image-id $HIVE_AMI \
-  --instance-type $HIVE_INSTANCE \
-  --key-name hive-key \
-  --count 1
-
-export INSTANCE_ID="i-XXX"  # from output
-```
-
-```bash
-# 2. Confirm running
-aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
-# Expect: State.Name = "running", State.Code = 16
-```
-
-```bash
-# 3. Stop the instance
-aws $EP ec2 stop-instances --instance-ids $INSTANCE_ID
-# Expect: PreviousState.Name = "running", CurrentState.Name = "stopping"
-```
-
-```bash
-# 4. Wait for stop to complete, then verify stopped
-aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
-# Expect: State.Name = "stopped", State.Code = 80
-```
-
-```bash
-# 5. Check daemon logs on the original node — should show KV release
-grep -i "Released stopped instance ownership" ~/node*/logs/hive.log
-# Expect: "Released stopped instance ownership to shared KV" with instanceId and lastNode
-```
-
-```bash
-# 6. Start the instance
-aws $EP ec2 start-instances --instance-ids $INSTANCE_ID
-# Expect: PreviousState.Name = "stopped", CurrentState.Name = "pending"
-```
-
-```bash
-# 7. Confirm running again
-aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
-# Expect: State.Name = "running", State.Code = 16
-```
-
-```bash
-# 8. Check which node picked up the start
-grep -i "Started stopped instance from shared KV" ~/node*/logs/hive.log
-# Note the node — it may differ from the original
-```
-
----
-
-## Test 2: Stop, Restart Daemon, Then Start
-
-Verify a stopped instance survives a daemon restart and can be started after.
-
-```bash
-# 1. Launch and stop an instance (repeat Test 1 steps 1-5)
-export INSTANCE_ID="i-XXX"
-```
-
-```bash
-# 2. Identify which node originally ran the instance
-grep "Released stopped instance" ~/node*/logs/hive.log | grep $INSTANCE_ID
-# Example: ~/node1/logs/hive.log — node1 was the original
-```
-
-```bash
-# 3. Restart that daemon
-./scripts/stop-dev.sh ~/node1/
-./scripts/start-dev.sh ~/node1/
-```
-
-```bash
-# 4. Verify the instance is still visible as stopped
-aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
-# Expect: State.Name = "stopped"
-# The instance is in shared KV, so any running daemon can report it
-```
-
-```bash
-# 5. Start the instance
-aws $EP ec2 start-instances --instance-ids $INSTANCE_ID
-# Expect: PreviousState.Name = "stopped", CurrentState.Name = "pending"
-```
-
-```bash
-# 6. Confirm running and check which node picked it up
-aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
-grep "Started stopped instance from shared KV" ~/node*/logs/hive.log | grep $INSTANCE_ID
-```
-
----
-
-## Test 3: Stop on Node A, Kill Node A, Start on Node B
-
-Verify a stopped instance can be started when its original daemon is down.
-
-```bash
-# 1. Launch instance (it lands on some node)
-aws $EP ec2 run-instances \
-  --image-id $HIVE_AMI \
-  --instance-type $HIVE_INSTANCE \
-  --key-name hive-key \
-  --count 1
-
-export INSTANCE_ID="i-XXX"
-```
-
-```bash
-# 2. Stop the instance
-aws $EP ec2 stop-instances --instance-ids $INSTANCE_ID
-```
-
-```bash
-# 3. Wait for stopped state
-aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
-# Expect: State.Name = "stopped"
-```
-
-```bash
-# 4. Identify the original node and shut it down
-grep "Released stopped instance" ~/node*/logs/hive.log | grep $INSTANCE_ID
-# If node1 — stop node1
-./scripts/stop-dev.sh ~/node1/
-```
-
-```bash
-# 5. Point CLI to a surviving node and start the instance
-export EP="--endpoint-url https://10.11.12.2:9999"
-aws $EP ec2 start-instances --instance-ids $INSTANCE_ID
-# Expect: CurrentState.Name = "pending" — a different daemon picked it up
-```
-
-```bash
-# 6. Confirm running on the new node
-aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
-# Expect: State.Name = "running"
-grep "Started stopped instance from shared KV" ~/node2/logs/hive.log ~/node3/logs/hive.log
-```
-
-```bash
-# 7. Bring node1 back up (cleanup)
-./scripts/start-dev.sh ~/node1/
-```
-
----
-
-## Test 4: Repeated Stop/Start Cycles
-
-Verify the instance can be stopped and started multiple times without state corruption.
-
-```bash
-# 1. Launch an instance
-aws $EP ec2 run-instances \
-  --image-id $HIVE_AMI \
-  --instance-type $HIVE_INSTANCE \
-  --key-name hive-key \
-  --count 1
-
-export INSTANCE_ID="i-XXX"
-```
-
-```bash
-# 2. Cycle 3 times
-for i in 1 2 3; do
-  echo "=== Cycle $i ==="
-
-  # Wait for running
-  sleep 5
-  aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID \
-    --query 'Reservations[0].Instances[0].State.Name' --output text
-  # Expect: running
-
-  # Stop
-  aws $EP ec2 stop-instances --instance-ids $INSTANCE_ID
-  sleep 10
-
-  aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID \
-    --query 'Reservations[0].Instances[0].State.Name' --output text
-  # Expect: stopped
-
-  # Start
-  aws $EP ec2 start-instances --instance-ids $INSTANCE_ID
-  sleep 10
-done
-```
-
-```bash
-# 3. Final state check
-aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
-# Expect: State.Name = "running"
-```
-
----
-
-## Test 5: DescribeInstances Includes Stopped Instances
-
-Verify stopped instances appear alongside running ones in describe output.
-
-```bash
-# 1. Launch two instances
-aws $EP ec2 run-instances \
-  --image-id $HIVE_AMI \
-  --instance-type $HIVE_INSTANCE \
-  --key-name hive-key \
-  --count 1
-export RUNNING_ID="i-XXX"
-
-aws $EP ec2 run-instances \
-  --image-id $HIVE_AMI \
-  --instance-type $HIVE_INSTANCE \
-  --key-name hive-key \
-  --count 1
-export STOPPED_ID="i-YYY"
-```
-
-```bash
-# 2. Stop one, keep the other running
-aws $EP ec2 stop-instances --instance-ids $STOPPED_ID
-sleep 10
-```
-
-```bash
-# 3. Describe all instances — both should appear
-aws $EP ec2 describe-instances
-# Expect: $RUNNING_ID with State.Name = "running"
-# Expect: $STOPPED_ID with State.Name = "stopped"
-```
-
-```bash
-# 4. Describe with instance ID filter
-aws $EP ec2 describe-instances --instance-ids $STOPPED_ID
-# Expect: only the stopped instance
-```
-
----
-
-## Test 6: Start a Non-Existent Instance
-
-Verify proper error handling.
-
-```bash
-aws $EP ec2 start-instances --instance-ids i-doesnotexist
-# Expect: instance remains reported as stopped (state unchanged)
-# Check daemon logs for "instance not found in shared KV"
-```
-
----
-
-## Test 7: Start an Already Running Instance
-
-Verify that starting a running instance does not cause issues.
-
-```bash
-# 1. Launch and confirm running
-aws $EP ec2 run-instances \
-  --image-id $HIVE_AMI \
-  --instance-type $HIVE_INSTANCE \
-  --key-name hive-key \
-  --count 1
-export INSTANCE_ID="i-XXX"
-sleep 5
-```
-
-```bash
-# 2. Try to start it again
-aws $EP ec2 start-instances --instance-ids $INSTANCE_ID
-# Expect: instance not found in shared KV (running instances are not in shared KV)
-# The response should report the instance as still "stopped" (no-op from gateway perspective)
-```
-
----
-
-## Test 8: Daemon Crash During Stop (Migration Recovery)
-
-Verify that a daemon crash between KV write and local cleanup is handled on restart.
-
-```bash
-# 1. Launch and stop an instance
-aws $EP ec2 run-instances \
-  --image-id $HIVE_AMI \
-  --instance-type $HIVE_INSTANCE \
-  --key-name hive-key \
-  --count 1
-export INSTANCE_ID="i-XXX"
-
-aws $EP ec2 stop-instances --instance-ids $INSTANCE_ID
-sleep 10
-```
-
-```bash
-# 2. Force kill the daemon process (simulate crash — skips cleanup)
-# Identify the daemon PID on the node that ran the instance
-grep "Released stopped instance" ~/node*/logs/hive.log | grep $INSTANCE_ID
-# If node1:
-kill -9 $(pgrep -f "hive.*node1")
-```
-
-```bash
-# 3. Restart the daemon
-./scripts/start-dev.sh ~/node1/
-```
-
-```bash
-# 4. Check logs for migration handling
-grep -i "Migrated stopped instance" ~/node1/logs/hive.log
-# If the instance was still in per-node state, restoreInstances migrates it to shared KV
-```
-
-```bash
-# 5. Verify instance is still startable
-aws $EP ec2 start-instances --instance-ids $INSTANCE_ID
-aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
-# Expect: State.Name = "running"
-```
-
----
-
-## What to Look For in Logs
-
-Key log messages and where to find them (`~/node*/logs/hive.log`):
-
-| Log Message | When |
-|-------------|------|
-| `Released stopped instance ownership to shared KV` | After successful stop + KV write |
-| `Started stopped instance from shared KV` | After a daemon picks up and launches a stopped instance |
-| `Migrated stopped instance to shared KV` | During `restoreInstances` on daemon startup |
-| `Failed to write stopped instance to shared KV, keeping local ownership` | KV write failed, instance stays local |
-| `instance not found in shared KV` | Start request for non-existent instance |
-| `instance not in stopped state` | Start request for instance not in stopped state |
-| `DescribeInstances: Collected stopped instance reservations` | Gateway merged stopped instances into describe output |

docs/development/feature/start-instance-distributed.md

@@ -1,141 +0,0 @@
-# Distributed StartInstances
-
-Decouple stopped EC2 instances from their original daemon node, allowing any available daemon to handle `start-instances` requests.
-
-## Problem
-
-Previously, stopped instances remained bound to the daemon that ran them. The daemon kept an active `ec2.cmd.{instanceID}` NATS subscription after stop, so `start-instances` always routed back to the same node. In multi-node deployments, this meant a stopped instance could not be started if its original daemon was down or at capacity.
-
-## Request Flow
-
-```
-AWS SDK (start-instances)
-  -> Gateway (port 9999): parse AWS query params
-    -> NATS ec2.start (queue group: hive-workers): any daemon can pick up
-      -> Daemon handleEC2StartStoppedInstance:
-        1. Load instance from shared NATS KV (key: instance.{instanceID})
-        2. Validate state is "stopped"
-        3. Re-initialize non-serialized fields (QMPClient, EBSRequests.Mu)
-        4. Allocate resources (CPU, memory)
-        5. LaunchInstance (mount volumes, start QEMU, subscribe to ec2.cmd.{instanceID})
-        6. Delete from shared KV
-      <- NATS response: {"status":"running","instanceId":"..."}
-    <- Gateway: build InstanceStateChange XML
-  <- AWS SDK: StartInstancesOutput
-```
-
-## How Ownership Release Works
-
-When an instance is stopped (not terminated), the daemon releases ownership in this order:
-
-1. Set `inst.LastNode = d.node` (record which daemon last ran it)
-2. Write instance to shared KV at key `instance.{instanceID}`
-3. Unsubscribe from `ec2.cmd.{instanceID}` NATS topic
-4. Remove instance from local `d.Instances.VMS` map
-5. Persist local state via `d.WriteState()` (now without the stopped instance)
-
-The order is deliberate: shared KV is written **first**, then local state is cleaned up. If the daemon crashes between steps 2 and 5, `restoreInstances` on restart detects the overlap and completes the migration.
-
-## Shared KV Storage
-
-Stopped instances are stored in the existing `hive-instance-state` NATS JetStream KV bucket using the key prefix `instance.` (distinct from the per-node prefix `node.`).
-
-Four methods on `JetStreamManager`:
-- `WriteStoppedInstance(instanceID, *vm.VM)` — serialize and write to `instance.{instanceID}`
-- `LoadStoppedInstance(instanceID)` — returns `nil, nil` if not found (not an error)
-- `DeleteStoppedInstance(instanceID)` — idempotent (ignores `ErrKeyNotFound`)
-- `ListStoppedInstances()` — iterates all `instance.*` keys, returns `[]*vm.VM`
-
-## DescribeInstances Integration
-
-`DescribeInstances` now includes stopped instances from shared KV. After the existing fan-out to all daemons (which returns running/pending instances from each node's local map), the gateway sends an additional NATS request to `ec2.DescribeStoppedInstances` (queue group, 3s timeout). The response contains reservations for all stopped instances in shared KV. These are merged into the final output.
-
-If the stopped-instance query fails (timeout, no responders), it is logged and skipped — running instances are still returned. This matches the existing pattern where individual daemon fan-out failures are non-fatal.
-
-## Migration on Restart
-
-`restoreInstances` migrates pre-existing stopped instances from per-node state to shared KV:
-
-- Instances in `StateStopped`: set `LastNode`, write to shared KV, remove from local map
-- Instances in `StateStopping`: transition to `StateStopped`, then migrate
-- `WriteState()` is called at the end to persist the removals
-
-If migration fails for an instance (KV write error), the instance stays in local state as a fallback and will be retried on next restart.
-
-## Files Changed
-
-### VM struct: `hive/vm/vm.go`
-
-Added `LastNode string` field (json: `last_node,omitempty`). Records which daemon node last ran the instance. Set during ownership release on stop. Used for diagnostics and future WAL recovery.
-
-### KV methods: `hive/daemon/jetstream.go`
-
-Added `StoppedInstancePrefix = "instance."` constant and four CRUD methods for stopped instance state. These reuse the `hive-instance-state` bucket — no new bucket required.
-
-### Stop flow: `hive/daemon/daemon_handlers.go`
-
-Extended the background goroutine in `handleStopOrTerminateInstance`. After `TransitionState(inst, StateStopped)`, the stop-only path (not terminate) releases ownership to shared KV. The terminate path is unchanged.
-
-### New handlers: `hive/daemon/daemon_handlers.go`
-
-`handleEC2StartStoppedInstance` — loads from shared KV, validates stopped state, re-initializes non-serialized fields, allocates resources, launches instance, removes from shared KV. On failure, rolls back resource allocation and local map insertion.
-
-`handleEC2DescribeStoppedInstances` — lists all stopped instances from shared KV, applies instance ID filters from request payload, groups by reservation ID, returns `DescribeInstancesOutput`.
-
-### Subscriptions: `hive/daemon/daemon.go`
-
-Added to `subscribeAll()`:
-- `ec2.start` with queue group `hive-workers` — load-balanced start requests
-- `ec2.DescribeStoppedInstances` with queue group `hive-workers` — only one daemon needs to read shared KV
-
-### Gateway: `hive/gateway/ec2/instance/StartInstances.go`
-
-Rewritten to send `{"instance_id":"..."}` to `ec2.start` topic instead of constructing QMP commands and routing to `ec2.cmd.{instanceID}`. Removed `qmp` import.
-
-### Gateway: `hive/gateway/ec2/instance/DescribeInstances.go`
-
-Added a NATS request to `ec2.DescribeStoppedInstances` after the fan-out loop. Merges stopped instance reservations into the response.
-
-### Restore: `hive/daemon/daemon.go`
-
-`restoreInstances` now migrates stopped instances from per-node state to shared KV instead of re-subscribing to per-instance NATS topics.
-
-## Design Decisions
-
-**Why queue group for `ec2.start`?** NATS queue groups deliver each message to exactly one subscriber. This prevents two daemons from simultaneously trying to start the same instance, without needing distributed locks.
-
-**Why shared KV instead of a database?** The `hive-instance-state` KV bucket already exists for per-node state. Reusing it avoids new infrastructure. NATS JetStream KV provides consistency guarantees sufficient for this use case (single writer per key during normal operation).
-
-**Why write shared KV before removing local state?** Crash safety. If the daemon crashes after the KV write but before local cleanup, the instance exists in both locations. `restoreInstances` detects this overlap and completes the migration. The reverse order (remove local first, then write KV) would risk losing the instance entirely on crash.
-
-**Why `LastNode` instead of re-routing to the original daemon?** The original daemon may be down, at capacity, or decommissioned. Recording which node last ran an instance is useful for debugging and future WAL recovery (Phase 2), but the start request goes to whichever daemon is available.
-
-**Why not a dedicated KV bucket for stopped instances?** The `instance.` prefix provides sufficient namespace separation from `node.` prefix entries. A separate bucket would require additional JetStream configuration for no practical benefit.
-
-## Error Codes
-
-| Condition | AWS Error Code |
-|-----------|---------------|
-| No instance IDs provided | `InvalidParameterValue` (gateway) |
-| Instance not found in shared KV | `InvalidInstanceID.NotFound` |
-| Instance not in stopped state | `IncorrectInstanceState` |
-| Resource allocation failure | `InsufficientInstanceCapacity` |
-| LaunchInstance failure | `InternalError` |
-| NATS timeout (no daemons available) | State reported as still stopped |
-
-## Testing
-
-**JetStream KV methods** (`jetstream_test.go`): 6 tests covering write/load round-trip, missing key returns nil, delete (including non-existent), list multiple instances, no interference with per-node state, nil KV bucket handling.
-
-**Handler error paths** (`daemon_handlers_test.go`): 5 tests covering missing instance ID, instance not found in KV, non-stopped state, describe returns stopped instances, describe with instance ID filter.
-
-**Gateway** (`StartInstances_test.go`): 6 tests covering single instance success, multiple instances, empty input, nil IDs skipped, NATS failure (reports stopped state), mixed success and failure.
-
-**State restore** (`state_test.go`): 5 tests updated to verify stopped instances are migrated to shared KV during `restoreInstances` instead of kept in local state.
-
-## Known Limitations
-
-- **All daemons must be updated together.** Old daemons don't subscribe to `ec2.start` and won't release stopped instances to shared KV.
-- **No WAL flush before ownership release.** Phase 1 relies on `ebs.unmount` during stop to flush volume data. Phase 2 will add explicit WAL flush to Predastore before releasing ownership.
-- **No capacity-aware scheduling.** `ec2.start` uses simple NATS queue group round-robin. A daemon might accept a start request even if it lacks resources, failing at allocation time. The gateway reports this as "still stopped" to the caller.
-- **TOCTOU on shared KV.** Between loading an instance from KV and deleting it after successful start, another daemon could theoretically load the same key. In practice this doesn't happen because NATS queue group delivers each `ec2.start` message to exactly one subscriber, and the instance ID in the message maps to a single KV key.

docs/development/feature/terminate-on-delete.md

@@ -1,107 +0,0 @@
-# DeleteOnTermination for EBS Volumes
-
-## Overview
-
-When an EC2 instance is terminated, attached EBS volumes with `DeleteOnTermination=true` are automatically deleted. Volumes with `DeleteOnTermination=false` (the default) remain in "available" state. Internal volumes (EFI, cloud-init) are always cleaned up on termination regardless of the flag.
-
-This matches the AWS EC2 behavior where the root volume's lifecycle can be tied to its instance.
-
-## How It Works
-
-### Volume Types During Termination
-
-Each instance has three volume categories in `EBSRequests.Requests`:
-
-| Volume Type | Identified By | Behavior on Terminate |
-|---|---|---|
-| Root (user-visible) | `Boot=true` | Respects `DeleteOnTermination` flag |
-| EFI partition | `EFI=true` | Always cleaned up via `ebs.delete` NATS |
-| Cloud-init ISO | `CloudInit=true` | Always cleaned up via `ebs.delete` NATS |
-
-### Data Flow
-
-```
-RunInstances request
-  → BlockDeviceMappings[0].Ebs.DeleteOnTermination parsed in GenerateVolumes()
-  → Stored on config.EBSRequest.DeleteOnTermination (per-volume)
-  → Persisted in viperblock VolumeMetadata.DeleteOnTermination (S3 config.json)
-  → Persisted in JetStream VM state (EBSRequests serialized with the instance)
-
-TerminateInstances request
-  → handleEC2Events sets deleteVolume=true
-  → stopInstance() iterates instance.EBSRequests.Requests:
-      1. All volumes: unmounted via ebs.unmount NATS
-      2. EFI/cloud-init: ebs.delete NATS sent to stop viperblockd processes
-      3. Root with DeleteOnTermination=true: volumeService.DeleteVolume() called
-         → Sends ebs.delete NATS to viperblockd
-         → Deletes S3 prefixes: vol-xxx/, vol-xxx-efi/, vol-xxx-cloudinit/
-      4. Root with DeleteOnTermination=false: skipped, remains "available"
-```
-
-### Stop vs Terminate
-
-- **Stop** (`deleteVolume=false`): Volumes are unmounted only. No deletion occurs regardless of the `DeleteOnTermination` flag. The volume state is set to "available".
-- **Terminate** (`deleteVolume=true`): The full deletion logic above runs.
-
-## Design Decisions
-
-### Why store the flag on EBSRequest rather than reading it from VolumeMetadata at termination time?
-
-`EBSRequest` is the struct available on the instance at termination time (`instance.EBSRequests.Requests`). Reading from S3 (`VolumeMetadata`) would add a network round-trip during termination and could fail if predastore is degraded. The flag is set once during `GenerateVolumes()` and carried with the instance state in JetStream.
-
-### Why do internal volumes use ebs.delete directly instead of DeleteVolume?
-
-EFI and cloud-init volumes are sub-prefixes of the root volume in S3 (`vol-xxx-efi/`, `vol-xxx-cloudinit/`). When `DeleteVolume` is called on the root volume, it already deletes these sub-prefixes. The direct `ebs.delete` NATS message to viperblockd is only needed to stop the running nbdkit/WAL syncer processes for those volumes. This avoids double-deletion of S3 data.
-
-### Why do errors not block termination?
-
-Following AWS behavior, termination is a best-effort cleanup. If a volume deletion fails (e.g., predastore temporarily unavailable), the instance still transitions to terminated state. The volume may remain as orphaned S3 data that can be cleaned up later. This prevents a broken storage backend from making instances un-terminable.
-
-### Why is DeleteOnTermination=false the default?
-
-This matches AWS EC2 defaults. When no `BlockDeviceMappings` are specified in `RunInstances`, volumes are preserved after termination. This is the safer default since accidental data loss is worse than orphaned volumes.
-
-## Files Changed
-
-| File | Change |
-|---|---|
-| `hive/config/config.go` | Added `DeleteOnTermination bool` to `EBSRequest` |
-| `hive/handlers/ec2/instance/service_impl.go` | Wired flag through `prepareRootVolume`, set on `EBSRequest` and `VolumeMetadata` |
-| `hive/daemon/daemon.go` | Implemented deletion logic in `stopInstance()` |
-| `hive/handlers/ec2/volume/service_impl.go` | `DescribeVolumes` reads `DeleteOnTermination` from `VolumeMetadata` instead of hardcoded `false` |
-| `viperblock/viperblock/viperblock.go` | Added `DeleteOnTermination bool` to `VolumeMetadata` |
-| `hive/daemon/daemon_test.go` | Tests for flag propagation, termination deletion, stop-not-delete, and JSON round-trip |
-
-## Testing
-
-### Unit Tests
-
-Run the DeleteOnTermination-specific tests:
-
-```bash
-go test -v -run "TestEBSRequest|TestGenerateVolumes_DeleteOnTermination|TestStopInstance_DeleteOnTermination|TestStopInstance_NoDelete|TestEBSRequest_JSON" ./hive/daemon/
-```
-
-### Manual Testing
-
-```bash
-# Launch with DeleteOnTermination=true
-aws --endpoint-url https://localhost:9999 ec2 run-instances \
-  --image-id ami-<id> --instance-type t3.micro --key-name <key> \
-  --block-device-mappings '[{"DeviceName":"/dev/xvda","Ebs":{"DeleteOnTermination":true,"VolumeSize":4}}]' \
-  --no-verify-ssl
-
-# Terminate and verify volume is gone
-aws --endpoint-url https://localhost:9999 ec2 terminate-instances --instance-ids <id> --no-verify-ssl
-aws --endpoint-url https://localhost:9999 ec2 describe-volumes --volume-ids <vol-id> --no-verify-ssl
-
-# Launch with default (DeleteOnTermination=false)
-# Terminate and verify volume remains in "available" state
-```
-
-### Log Messages to Watch
-
-- `"Volume has DeleteOnTermination=false, skipping deletion"` - volume preserved
-- `"Deleting volume with DeleteOnTermination=true"` - deletion triggered
-- `"Sent ebs.delete for internal volume"` - EFI/cloud-init process cleanup
-- `"Failed to delete volume on termination"` - deletion failed (non-blocking)

docs/development/improvements/docker-e2e-local.md

@@ -1,82 +0,0 @@
-# Fix Docker E2E Tests and Enable Local Reproduction
-
-**Status: Complete**
-
-## Summary
-
-Both CI E2E tests (single-node and multi-node) are failing after recent multi-server changes. This improvement fixes the root causes and adds a local runner script so developers can reproduce E2E tests without pushing to CI.
-
-## Context / Problem Statement
-
-### Single-Node E2E Failure
-
-**Symptom:** `DescribeInstanceTypes` returns empty — logs show "nats: no responders available for request".
-
-**Root cause:** Race condition. The E2E script waits for the gateway HTTP endpoint (`curl -s https://localhost:9999`), but this only confirms the gateway process is listening — not that the daemon has connected to NATS and subscribed to topics.
-
-Daemon startup path: `connectNATS()` -> `ClusterManager()` -> `initJetStream()` -> create services -> `restoreInstances()` -> `subscribeAll()`. If the gateway accepts requests before the daemon finishes `subscribeAll()`, NATS publishes find no subscribers.
-
-Note: `DescribeRegions` and `DescribeAvailabilityZones` succeed because they're handled locally in the gateway (`gateway/ec2.go`) without NATS. `DescribeInstanceTypes` is the first action that requires daemon NATS subscriptions.
-
-### Multi-Node E2E Failure
-
-**Symptom:** Formation server waits 10 minutes for nodes to join, then times out (1/3 nodes joined).
-
-**Root cause:** `init_leader_node()` calls `hive admin init` without explicit `--nodes` flag. The default is `--nodes 3` (`admin.go:156`). Combined with `--bind 10.11.12.1` (not `0.0.0.0`), this triggers the multi-node formation path:
-
-```go
-isMultiNode := nodes >= 2 && bindIP != "0.0.0.0"
-```
-
-The formation server starts and blocks at `WaitForCompletion(formationTimeout)` waiting for 2 more nodes. But the E2E script is sequential — `join_follower_node 2` and `3` run AFTER `init_leader_node` returns, creating a deadlock.
-
-Additionally, `hive admin join` also blocks — it polls `/formation/status` until `complete: true` (all nodes joined). So even if init were backgrounded, running join_2 then join_3 sequentially would deadlock because join_2 blocks waiting for node_3.
-
-### Timing of CA cert and config files
-
-The `runAdminInit` flow (`admin.go`):
-1. **Cert generation** (`admin.GenerateCertificatesIfNeeded`) — happens BEFORE multi-node check
-2. **Multi-node path** starts formation server, blocks at `WaitForCompletion`
-3. **Config generation** happens AFTER all nodes join
-
-This means:
-- `~/node1/config/ca.pem` — available as soon as formation server starts (generated beforehand)
-- `~/node1/config/hive.toml` and other configs — only available after ALL nodes join
-
-## Proposed Changes
-
-### 1. Single-node fix: daemon readiness check
-
-Added `wait_for_daemon_ready()` helper in `multinode-helpers.sh` that polls `describe-instance-types` until it returns a non-empty result. Called after gateway health check in both E2E scripts.
-
-### 2. Multi-node fix: concurrent formation orchestration
-
-Restructured `init_leader_node()` to background the init process and poll for formation server readiness. Restructured Phase 2 of `run-multinode-e2e.sh` to:
-1. Background init (starts formation server, generates certs first)
-2. Trust CA cert (exists before formation completes)
-3. Background BOTH joins concurrently (they poll `/formation/status`)
-4. Wait for all three processes to finish
-5. Start services after formation completes (configs now exist)
-
-This mirrors production where init and joins run on separate machines simultaneously.
-
-### 3. Local E2E runner script
-
-Created `scripts/run-e2e-local.sh` that mirrors the CI workflow (`e2e.yml`) for local Docker E2E testing. Supports running single-node, multi-node, or both suites.
-
-## Files to Modify
-
-| File | Change |
-|---|---|
-| `tests/e2e/lib/multinode-helpers.sh` | Added `wait_for_daemon_ready()`, restructured `init_leader_node()` to background init and poll formation health |
-| `tests/e2e/run-e2e.sh` | Added `wait_for_daemon_ready` call after gateway wait |
-| `tests/e2e/run-multinode-e2e.sh` | Restructured Phase 2 for concurrent formation, added `wait_for_daemon_ready` |
-| `scripts/run-e2e-local.sh` | New local Docker E2E runner script |
-| `.claude/CLAUDE.md` | Added git history guidance to Development Process section |
-
-## Testing
-
-1. **Multi-node fix:** Run `./scripts/run-e2e-local.sh multi` — formation server starts, all 3 joins complete concurrently, configs generated, services start, tests pass
-2. **Single-node fix:** Run `./scripts/run-e2e-local.sh single` — daemon readiness check passes, `describe-instance-types` returns instance types
-3. **Full local E2E:** Run `./scripts/run-e2e-local.sh` — both suites pass
-4. **CI validation:** Push and verify GitHub Actions E2E workflow passes

docs/development/improvements/hive-cluster-stats.md

@@ -1,129 +0,0 @@
-# Hive Cluster Stats CLI
-
-**Status: Complete**
-
-## Summary
-
-Added `hive get {nodes,vms}` and `hive top nodes` commands to provide kubectl-style cluster visibility. Operators can now see node health, running VMs, and resource utilization across the cluster from a single command.
-
-This also satisfies the `hive admin status` command proposed in `multi-server-restart.md` (Section 3.3) — `hive get nodes` and `hive top nodes` together provide the same information with better kubectl-style UX.
-
-## Commands
-
-### `hive get nodes`
-
-Like `kubectl get nodes -o wide`. Shows all physical nodes in the cluster.
-
-```
-$ hive get nodes
-NAME    STATUS    IP            REGION             AZ                  UPTIME   VMs   SERVICES
-node1   Ready     10.11.12.1    ap-southeast-2     ap-southeast-2a     3d4h     2     nats,predastore,viperblock,daemon,awsgw,ui
-node2   Ready     10.11.12.2    ap-southeast-2     ap-southeast-2a     3d4h     1     nats,predastore,viperblock,daemon
-node3   NotReady  10.11.12.3    ap-southeast-2     ap-southeast-2b     -        -     nats,predastore,viperblock,daemon
-```
-
-- **Status values**: `Ready` (daemon responded), `NotReady` (no response within timeout)
-- Nodes known from config but not responding are shown as `NotReady`
-- `--timeout` flag controls response collection window (default 3s)
-
-### `hive get vms` (alias: `hive get instances`)
-
-Like `kubectl get pods -A -o wide`. Shows all VMs across the cluster.
-
-```
-$ hive get vms
-INSTANCE             STATUS    TYPE       VCPU   MEM     NODE    IP            AGE
-i-896fdcfd299a39193  running   t3.nano    2      512Mi   node1   10.11.12.1    0m
-i-2b61dfa9fdb8c2b39  running   t3.nano    2      512Mi   node2   10.11.12.2    0m
-i-b097988f78db6553d  running   t3.nano    2      512Mi   node2   10.11.12.2    0m
-```
-
-### `hive top nodes`
-
-Like `kubectl top nodes`. Shows resource usage per node plus aggregated available instance types.
-
-```
-$ hive top nodes
-NAME    CPU (used/total)   MEM (used/total)    VMs
-node1   4/16               1.0Gi/30.6Gi        2
-node2   2/16               512Mi/30.6Gi        1
-node3   -                  -                   -
-
-INSTANCE TYPE   AVAILABLE   VCPU   MEMORY
-t3.nano         22          2      512Mi
-t3.micro        22          2      1.0Gi
-t3.small        16          2      2.0Gi
-t3.medium       11          2      4.0Gi
-t3.large        5           2      8.0Gi
-t3.xlarge       2           4      16.0Gi
-```
-
-Bottom table aggregates free capacity across all Ready nodes.
-
-## Design
-
-### Config as Source of Truth
-
-During multi-node formation, each node's `hive.toml` includes ALL cluster nodes — the local node with full config (daemon, nats, predastore subsections) and remote nodes with basic info (host, region, AZ, services). This makes config the authoritative list of expected cluster members.
-
-The CLI builds its node list from the union of config nodes + NATS responders:
-- **Config nodes that respond** → shown as `Ready` with live data from NATS
-- **Config nodes that don't respond** → shown as `NotReady` with host/region/AZ from config
-- **NATS responders not in config** → shown as `Ready` (safety net for dynamic additions)
-
-### NATS Topics
-
-Two fan-out topics (no queue group — all daemons respond):
-
-| Topic | Purpose | Response Type |
-|-------|---------|---------------|
-| `hive.node.status` | Node stats + resource usage | `NodeStatusResponse` |
-| `hive.node.vms` | Running VMs on this node | `NodeVMsResponse` |
-
-### NATS Fan-Out Collection Pattern
-
-CLI connects to NATS using the config's token, publishes to a fan-out topic with an inbox reply address, and collects all responses within the timeout window:
-
-```go
-inbox := nats.NewInbox()
-sub, _ := nc.SubscribeSync(inbox)
-nc.PublishRequest("hive.node.status", inbox, nil)
-for {
-    msg, err := sub.NextMsg(remaining)
-    if err != nil { break }
-    // unmarshal and append
-}
-```
-
-### ResourceManager.GetResourceStats()
-
-New method that returns total/allocated vCPU and memory plus per-instance-type available capacity in a single lock acquisition. All data was already tracked — no new system calls needed.
-
-## Files Modified
-
-| File | Change |
-|------|--------|
-| `hive/config/config.go` | Added `NodeStatusResponse`, `InstanceTypeCap`, `VMInfo`, `NodeVMsResponse` types |
-| `hive/admin/admin.go` | Added `RemoteNode` type and `RemoteNodes` field to `ConfigSettings` |
-| `hive/daemon/daemon_handlers.go` | Added `daemonIP()`, `handleNodeStatus()`, `handleNodeVMs()` handlers |
-| `hive/daemon/daemon.go` | Added `hive.node.status` and `hive.node.vms` subscriptions in `subscribeAll()`, added `GetResourceStats()` method to `ResourceManager` |
-| `cmd/hive/cmd/admin.go` | Added `buildRemoteNodes()` helper; populate `RemoteNodes` in init and join flows |
-| `cmd/hive/cmd/templates/hive.toml` | Added `host` to local node, template loop for remote nodes |
-| `cmd/hive/cmd/get.go` | **New file** — `get nodes`, `get vms` commands with pterm table output |
-| `cmd/hive/cmd/top.go` | **New file** — `top nodes` command with node resource + instance type capacity tables |
-| `tests/e2e/run-e2e.sh` | Added Phase 1b (get nodes, top nodes, get vms empty) and Phase 5a-pre (get vms with running VM) |
-| `tests/e2e/run-multinode-e2e.sh` | Added Phase 3b (multi-node get/top) and post-launch get vms verification |
-
-## Testing
-
-- `make preflight` — all checks pass (gofmt, vet, gosec, staticcheck, govulncheck, unit tests)
-- `make test-docker-single` — single-node E2E passes with new CLI test phases
-- `make test-docker-multi` — 3-node E2E passes, `get vms` correctly shows instances across nodes with correct IPs
-
-### E2E test coverage added
-
-- `hive get nodes` shows Ready nodes with correct IP, region, AZ, services
-- `hive top nodes` shows CPU/MEM stats and instance type capacity table
-- `hive get vms` returns "No VMs found" when empty
-- `hive get vms` shows running instances with correct node placement after launch
-- Multi-node: VMs distributed across nodes are all visible in `get vms` output

docs/development/improvements/iac-testing-aws-terraform.md

@@ -1,215 +0,0 @@
-# IaC Compatibility: Terraform/OpenTofu with AWS Provider
-
-**Status: Planned**
-
-## Summary
-
-Enable infrastructure-as-code workflows using the standard Terraform/OpenTofu AWS provider (`hashicorp/aws`) pointed at MulgaOS/Hive endpoints. The goal is to run `tofu apply` against a Hive cluster and have it create real infrastructure — EC2 instances, EBS volumes, key pairs — using the same `.tf` files that work against AWS.
-
-## Context / Problem Statement
-
-Hive already implements 37 EC2 API actions (instances, volumes, snapshots, images, key pairs, tags, etc.), but the Terraform AWS provider requires several additional APIs during `plan` and `apply` — particularly VPC networking resources that Terraform treats as mandatory dependencies for EC2 instances.
-
-The existing `scripts/iac/aws/main.tf` defines a full VPC + subnet + security group + instance stack, but `tofu apply` fails because the VPC/networking APIs aren't implemented. We need to either:
-
-1. **Stub** the networking APIs (accept input, return valid-looking output, store state) so Terraform can proceed, or
-2. **Implement** them with real backing logic
-
-For the initial milestone, stubbing is acceptable — Hive VMs already get networking through QEMU user-mode or bridge networking at the daemon level, so the VPC/subnet/SG resources are metadata-only for now. The critical requirement is that Terraform can complete a full `plan` → `apply` → `destroy` cycle.
-
-## Current API Coverage
-
-### Fully Implemented (works with Terraform)
-
-| Resource | Terraform Type | Hive API Actions |
-|---|---|---|
-| EC2 Instances | `aws_instance` | `RunInstances`, `DescribeInstances`, `StartInstances`, `StopInstances`, `TerminateInstances`, `DescribeInstanceTypes` |
-| EBS Volumes | `aws_ebs_volume` | `CreateVolume`, `DescribeVolumes`, `DescribeVolumeStatus`, `ModifyVolume`, `DeleteVolume` |
-| Volume Attachment | `aws_volume_attachment` | `AttachVolume`, `DetachVolume` |
-| Key Pairs | `aws_key_pair` | `CreateKeyPair`, `ImportKeyPair`, `DescribeKeyPairs`, `DeleteKeyPair` |
-| AMI/Images | `aws_ami`, data source | `DescribeImages`, `CreateImage` |
-| Snapshots | `aws_ebs_snapshot` | `CreateSnapshot`, `DescribeSnapshots`, `DeleteSnapshot`, `CopySnapshot` |
-| Tags | (all resources) | `CreateTags`, `DeleteTags`, `DescribeTags` |
-| Regions/AZs | data sources | `DescribeRegions`, `DescribeAvailabilityZones` |
-| Account | data source | `DescribeAccountAttributes` |
-| Egress IGW | `aws_egress_only_internet_gateway` | `Create/Describe/DeleteEgressOnlyInternetGateway` |
-
-### Missing — Required by Terraform AWS Provider
-
-These APIs are called by Terraform during `plan`/`apply` for common infrastructure patterns. Each needs at minimum a stub that accepts input, generates an ID, stores state in NATS KV, and returns a valid response.
-
-#### Priority 1: VPC Networking (blocks `tofu apply` on `main.tf`)
-
-| Terraform Resource | Required API Actions | Complexity |
-|---|---|---|
-| `aws_vpc` | `CreateVpc`, `DescribeVpcs`, `DeleteVpc`, `ModifyVpcAttribute` | Medium — needs ID generation, CIDR storage, state tracking |
-| `aws_subnet` | `CreateSubnet`, `DescribeSubnets`, `DeleteSubnet` | Medium — references VPC, needs AZ assignment |
-| `aws_security_group` | `CreateSecurityGroup`, `DescribeSecurityGroups`, `DeleteSecurityGroup`, `AuthorizeSecurityGroupIngress`, `AuthorizeSecurityGroupEgress`, `RevokeSecurityGroupIngress`, `RevokeSecurityGroupEgress` | High — complex rule model, Terraform creates a default SG per VPC |
-| `aws_internet_gateway` | `CreateInternetGateway`, `DescribeInternetGateways`, `DeleteInternetGateway`, `AttachInternetGateway`, `DetachInternetGateway` | Medium — references VPC |
-| `aws_route_table` | `CreateRouteTable`, `DescribeRouteTables`, `DeleteRouteTable`, `CreateRoute`, `DeleteRoute` | Medium — references VPC, subnet associations |
-| `aws_route_table_association` | `AssociateRouteTable`, `DisassociateRouteTable` | Low — links subnet to route table |
-
-#### Priority 2: Provider Initialization (called during `tofu init`/`plan`)
-
-| Action | Why Terraform Calls It | Stub Needed |
-|---|---|---|
-| `GetCallerIdentity` (STS) | Provider startup — validates credentials | Yes — return static account ID |
-| `DescribeVpcAttribute` | After VPC creation — queries DNS settings | Yes — return defaults |
-| `DescribeNetworkInterfaces` | Instance creation — checks ENI state | Yes — return empty |
-| `DescribeInstanceAttribute` | After instance creation — checks attributes | Yes — return defaults |
-
-#### Priority 3: Nice-to-Have
-
-| Terraform Resource | Required API Actions | Notes |
-|---|---|---|
-| `aws_placement_group` | `CreatePlacementGroup`, `DescribePlacementGroups`, `DeletePlacementGroup` | Used in `main.tf` for spread strategy |
-| `aws_eip` | `AllocateAddress`, `DescribeAddresses`, `ReleaseAddress`, `AssociateAddress` | Static IP assignment |
-| `aws_network_interface` | `CreateNetworkInterface`, `DescribeNetworkInterfaces`, `DeleteNetworkInterface` | Advanced networking |
-
-## Proposed Changes
-
-### Phase 1: VPC Networking Stubs
-
-Implement stub handlers for all Priority 1 + Priority 2 APIs. Each stub:
-
-- Accepts the AWS SDK input struct
-- Generates a resource ID (e.g., `vpc-<random>`, `subnet-<random>`)
-- Stores the resource in NATS KV (bucket per resource type)
-- Returns a valid AWS-compatible response
-- Supports `Describe*` to list stored resources
-- Supports `Delete*` to remove from KV
-
-Architecture:
-```
-hive/gateway/ec2/vpc/          # CreateVpc, DescribeVpcs, DeleteVpc
-hive/gateway/ec2/subnet/       # CreateSubnet, DescribeSubnets, DeleteSubnet
-hive/gateway/ec2/sg/           # Security group CRUD + rule management
-hive/gateway/ec2/igw/          # Internet gateway CRUD + attach/detach
-hive/gateway/ec2/rtb/          # Route table CRUD + associations
-hive/gateway/sts/              # GetCallerIdentity stub
-```
-
-Storage: NATS KV buckets — `vpc`, `subnet`, `security-group`, `internet-gateway`, `route-table`. Each entry keyed by resource ID, value is JSON-serialized resource state.
-
-### Phase 2: Terraform Integration Testing
-
-Update `scripts/iac/aws/main.tf` and add automated validation:
-
-1. **Simplify `main.tf`** — start with a minimal config that exercises implemented features:
-   - Key pair (works today)
-   - VPC + subnet + security group (after Phase 1 stubs)
-   - EC2 instance with EBS volume
-   - Tags on all resources
-
-2. **Add `scripts/iac/aws/test.sh`** — automated test script:
-   ```bash
-   tofu init
-   tofu plan -out=plan.tfplan
-   tofu apply plan.tfplan
-   # Verify resources exist via AWS CLI
-   tofu destroy -auto-approve
-   ```
-
-3. **Add to CI** — optional E2E stage that runs the Terraform test against a running Hive instance
-
-### Phase 3: Real VPC Networking (Future)
-
-Replace stubs with real implementations as Hive networking matures:
-- Map VPCs to Linux bridges or network namespaces
-- Map subnets to DHCP ranges on bridges
-- Map security groups to nftables/iptables rules
-- Map internet gateways to NAT/routing rules
-
-## Files to Modify
-
-### Phase 1: New Gateway Handlers
-
-| File | Description |
-|---|---|
-| `hive/gateway/ec2/vpc/CreateVpc.go` | Create VPC stub — generate ID, store in NATS KV |
-| `hive/gateway/ec2/vpc/DescribeVpcs.go` | List VPCs from NATS KV |
-| `hive/gateway/ec2/vpc/DeleteVpc.go` | Remove VPC from NATS KV |
-| `hive/gateway/ec2/subnet/CreateSubnet.go` | Create subnet stub |
-| `hive/gateway/ec2/subnet/DescribeSubnets.go` | List subnets |
-| `hive/gateway/ec2/subnet/DeleteSubnet.go` | Delete subnet |
-| `hive/gateway/ec2/sg/CreateSecurityGroup.go` | Create security group |
-| `hive/gateway/ec2/sg/DescribeSecurityGroups.go` | List security groups |
-| `hive/gateway/ec2/sg/DeleteSecurityGroup.go` | Delete security group |
-| `hive/gateway/ec2/sg/AuthorizeIngress.go` | Add ingress rule |
-| `hive/gateway/ec2/sg/AuthorizeEgress.go` | Add egress rule |
-| `hive/gateway/ec2/sg/RevokeIngress.go` | Remove ingress rule |
-| `hive/gateway/ec2/sg/RevokeEgress.go` | Remove egress rule |
-| `hive/gateway/ec2/igw/CreateInternetGateway.go` | Create IGW stub |
-| `hive/gateway/ec2/igw/DescribeInternetGateways.go` | List IGWs |
-| `hive/gateway/ec2/igw/DeleteInternetGateway.go` | Delete IGW |
-| `hive/gateway/ec2/igw/AttachInternetGateway.go` | Attach IGW to VPC |
-| `hive/gateway/ec2/igw/DetachInternetGateway.go` | Detach IGW from VPC |
-| `hive/gateway/ec2/rtb/CreateRouteTable.go` | Create route table |
-| `hive/gateway/ec2/rtb/DescribeRouteTables.go` | List route tables |
-| `hive/gateway/ec2/rtb/DeleteRouteTable.go` | Delete route table |
-| `hive/gateway/ec2/rtb/CreateRoute.go` | Add route to table |
-| `hive/gateway/ec2/rtb/DeleteRoute.go` | Remove route from table |
-| `hive/gateway/ec2/rtb/AssociateRouteTable.go` | Associate subnet with route table |
-| `hive/gateway/ec2/rtb/DisassociateRouteTable.go` | Disassociate route table |
-| `hive/gateway/sts/GetCallerIdentity.go` | STS stub — return static account |
-
-### Existing Files to Modify
-
-| File | Description |
-|---|---|
-| `hive/gateway/ec2.go` | Register new VPC/subnet/SG/IGW/RTB actions in `ec2Actions` map |
-| `hive/gateway/gateway.go` | Add STS service routing if needed |
-| `scripts/iac/aws/main.tf` | Update endpoint config, adjust resources for current capabilities |
-| `scripts/iac/aws/README.md` | Update with current API support status |
-
-## Terraform AWS Provider Behavior Notes
-
-Key things to know about how the Terraform AWS provider interacts with APIs:
-
-1. **Provider init** calls `sts:GetCallerIdentity` unless `skip_requesting_account_id = true` — we set this in the provider config
-2. **VPC creation** triggers `CreateVpc` then immediately `DescribeVpcs` to wait for `available` state, then `DescribeVpcAttribute` for DNS settings
-3. **Security group** — Terraform creates a default SG when creating a VPC, and `CreateSecurityGroup` must return a `groupId`; rules are added via separate `AuthorizeSecurityGroup{Ingress,Egress}` calls
-4. **Instance creation** — Terraform passes `SubnetId` and `SecurityGroupId` in `RunInstances`; we need to accept these without erroring even if networking is stubbed
-5. **Destroy order** — Terraform destroys in reverse dependency order (instances first, then SGs, subnets, IGW, VPC). Each `Delete*` must succeed or Terraform retries.
-6. **State refresh** — On every `plan`/`apply`, Terraform calls `Describe*` for every resource in state to detect drift. All `Describe*` endpoints must return consistent data.
-
-## Testing
-
-### Manual Validation
-
-```bash
-cd scripts/iac/aws
-export TF_VAR_mulgaos_endpoint="https://localhost:9999"
-export AWS_ACCESS_KEY_ID=test
-export AWS_SECRET_ACCESS_KEY=test
-
-tofu init
-tofu plan        # Should complete without errors
-tofu apply       # Should create all resources
-tofu destroy     # Should clean up all resources
-```
-
-### Automated (Future)
-
-Add a Terraform E2E stage to CI that runs the full `init → plan → apply → verify → destroy` cycle against a Hive instance in Docker.
-
-## Implementation Order
-
-1. VPC + DescribeVpcs + DeleteVpc (unblocks subnet, SG, IGW)
-2. Subnet + DescribeSubnets + DeleteSubnet
-3. Security Group (full CRUD + rules)
-4. Internet Gateway (CRUD + attach/detach)
-5. Route Table (CRUD + routes + associations)
-6. Wire all into `ec2.go` action map
-7. Test with `tofu plan` (catches missing APIs)
-8. Test with `tofu apply` (catches response format issues)
-9. Update `scripts/iac/aws/main.tf` and README
-
-## Future Work
-
-- Replace VPC stubs with real Linux bridge/namespace networking
-- Map security groups to nftables rules on hypervisor
-- Implement Elastic IPs for static addressing
-- Add S3 endpoint support (Predastore already S3-compatible)
-- Implement IAM basics for multi-tenant access control
-- Add CloudWatch-compatible metrics endpoint

docs/development/improvements/iac-testing-hive.md

@@ -1,341 +0,0 @@
-# Reusable IaC Test Environments for Hive
-
-## Summary
-
-Make the Proxmox IaC in `scripts/iac/proxmox/` reusable and parameterized so that multiple isolated Hive clusters can be provisioned, configured, tested, and destroyed independently. Introduce a four-phase lifecycle (Provision, Configure, Test, Destroy) with a single wrapper script, inspired by Kubernetes kubetest2, QEMU avocado, and SUSE Rancher patterns.
-
-**Status: Planned**
-
-## Context / Problem Statement
-
-The current Proxmox IaC works well for its original purpose — provisioning the 3 static dev servers (10.1.3.170-172). But it has several limitations that prevent reuse:
-
-### What exists today
-
-| File | Purpose |
-|---|---|
-| `scripts/iac/proxmox/main.tf` | Provider config (`bpg/proxmox` v0.52.0), locals for node map |
-| `scripts/iac/proxmox/vms.tf` | VM resources, hardcoded 4 vCPU / 8GB RAM / 32GB disk, output `iac_mulgaos_ips` |
-| `scripts/iac/proxmox/variables.tf` | Input vars: `proxmox_endpoint`, `ssh_*_key_path`, `nodes` (exactly 3) |
-| `scripts/iac/proxmox/cloud-config.tf` | Cloud-init: installs packages, clones hive, runs `quickinstall` + `dev-setup.sh` |
-| `scripts/iac/proxmox/README.md` | Manual cluster setup instructions (init node 1, join nodes 2-3) |
-
-### Current gaps
-
-1. **Hardcoded to 3 nodes** — `variables.tf` validates `length(var.nodes) == 3`, can't run a 1-node smoke test or a 5-node stress test
-2. **Single state file** — `terraform.tfstate` lives in the working directory, no way to have parallel environments without manually copying the directory (which is what `proxmox.orig/` is doing today)
-3. **Hardcoded VM names** — `iac-dev-mulgaos-${index}` means two environments would collide on VM names
-4. **Hardcoded specs** — 4 vCPU, 8GB RAM, 32GB disk baked into `vms.tf`, no way to test with different sizes
-5. **Cloud-init does too much** — clones hive from GitHub, runs `quickinstall`, runs `dev-setup.sh`. This couples provisioning to a specific hive version and makes iteration slow (full cloud-init reruns on every `tofu apply`)
-6. **No automated post-provisioning** — after `tofu apply`, the README has manual SSH commands for `hive admin init` on node 1, `hive admin join` on nodes 2-3, then manual `hive start` on each
-7. **No test orchestration** — no way to automatically run E2E tests against a provisioned cluster
-
-## Design Principles
-
-Drawing from how other infrastructure projects handle automated test environments:
-
-### Kubernetes kubetest2
-
-Pluggable deployer interface with lifecycle phases: `up -> test -> down`. Deployers are swappable (kind, GKE, EKS) but the lifecycle contract is the same. The test harness is decoupled from infrastructure creation — tests don't know or care whether they're running against kind or GKE.
-
-**Takeaway**: Separate the lifecycle wrapper from the infrastructure backend. Today we have Proxmox; tomorrow we might add AWS or libvirt. The `up/test/down` contract stays the same.
-
-### QEMU avocado
-
-Parameterized test matrices — vary CPU count, memory, disk type, architecture across test runs. Test cases are decoupled from environment parameters. A single test suite runs against many different configurations.
-
-**Takeaway**: Make node count, CPU, memory, and disk size variables, not constants. Enable test matrix runs like "3-node/4-core vs 5-node/8-core".
-
-### SUSE Rancher
-
-Two-layer IaC — Layer 1 provisions VMs (bare OS), Layer 2 deploys software (Kubernetes/Rancher). Clean separation means you can re-deploy software without re-provisioning VMs, and vice versa.
-
-**Takeaway**: Cloud-init should only set up the OS. Hive installation and cluster formation belong in a separate configure phase that can be re-run independently.
-
-### Terraform best practices
-
-Separate state per environment (workspaces are for identical configs in different accounts, not divergent environments). Variables for everything that changes. Outputs from one phase become inputs to the next.
-
-**Takeaway**: Use `-state=clusters/{name}/terraform.tfstate` for state isolation. Write structured outputs (node IPs, SSH config) to `clusters/{name}/inventory.json` for the configure phase to consume.
-
-## Proposed Changes
-
-### Four-phase lifecycle
-
-```
-Provision (tofu)  ->  Configure (shell)  ->  Test (shell)  ->  Destroy (tofu)
-     |                      |                     |                   |
-     v                      v                     v                   v
-  VMs exist            Hive running          Tests pass          VMs gone
-  IPs known            Cluster formed        Results saved       State cleaned
-```
-
-### Phase 1: Provision
-
-Terraform/OpenTofu creates VMs with a minimal OS. No hive-specific setup.
-
-**Changes to `scripts/iac/proxmox/variables.tf`:**
-
-```hcl
-variable "cluster_name" {
-  type        = string
-  description = "Unique name for this cluster (used in VM names, state path, tags)"
-
-  validation {
-    condition     = can(regex("^[a-z0-9-]+$", var.cluster_name))
-    error_message = "cluster_name must contain only lowercase letters, numbers, and hyphens."
-  }
-}
-
-variable "node_count" {
-  type        = number
-  description = "Number of nodes in the cluster (1-10)"
-  default     = 3
-
-  validation {
-    condition     = var.node_count >= 1 && var.node_count <= 10
-    error_message = "node_count must be between 1 and 10."
-  }
-}
-
-variable "cpu_cores" {
-  type        = number
-  description = "vCPU cores per VM"
-  default     = 4
-}
-
-variable "memory_mb" {
-  type        = number
-  description = "Memory in MB per VM"
-  default     = 8192
-}
-
-variable "disk_size_gb" {
-  type        = number
-  description = "Root disk size in GB per VM"
-  default     = 32
-}
-
-variable "os_image" {
-  type        = string
-  description = "Cloud image file ID on Proxmox (local:iso/...)"
-  default     = "local:iso/debian-12-genericcloud-amd64-20240211-1654.img"
-}
-```
-
-The existing `nodes` variable (list of Proxmox host objects) stays — it defines *where* VMs land. The new `node_count` controls *how many* VMs are created. VMs are distributed across Proxmox hosts round-robin. The `length(var.nodes) == 3` validation is removed; any number of Proxmox hosts is valid.
-
-**Changes to `scripts/iac/proxmox/vms.tf`:**
-
-```hcl
-resource "proxmox_virtual_environment_vm" "hive_node" {
-  count = var.node_count
-
-  name        = "${var.cluster_name}-${count.index + 1}"
-  description = "Managed by Terraform - cluster: ${var.cluster_name}"
-  tags        = ["terraform", "hive", var.cluster_name]
-  node_name   = var.nodes[count.index % length(var.nodes)].name
-
-  cpu {
-    cores = var.cpu_cores
-    type  = "host"
-  }
-
-  memory {
-    dedicated = var.memory_mb
-  }
-
-  disk {
-    file_id      = var.os_image
-    datastore_id = var.nodes[count.index % length(var.nodes)].datastore_id
-    interface    = "virtio0"
-    iothread     = true
-    discard      = "on"
-    size         = var.disk_size_gb
-  }
-
-  # ... initialization, network_device, agent blocks unchanged
-}
-```
-
-**Changes to `scripts/iac/proxmox/cloud-config.tf`:**
-
-Slim down to OS essentials only. Remove hive clone, `quickinstall`, `clone-deps.sh`, `dev-setup.sh`:
-
-```yaml
-#cloud-config
-package_update: true
-package_upgrade: true
-packages:
-  - qemu-guest-agent
-  - net-tools
-  - git
-  - make
-runcmd:
-  - timedatectl set-timezone America/Los_Angeles
-  - systemctl enable qemu-guest-agent
-  - systemctl start qemu-guest-agent
-  - ufw disable
-  - sysctl -w net.core.rmem_max=4194304
-  - sysctl -w net.core.wmem_max=4194304
-  - usermod -aG kvm tf-user
-  - echo "done" > /tmp/vendor-cloud-init-done
-```
-
-Hive installation moves to Phase 2 (Configure), where it can be iterated without re-provisioning VMs.
-
-**New `scripts/iac/proxmox/outputs.tf`:**
-
-Structured inventory output for the configure phase:
-
-```hcl
-output "inventory" {
-  description = "Cluster inventory for configure phase"
-  value = {
-    cluster_name = var.cluster_name
-    nodes = [
-      for i, vm in proxmox_virtual_environment_vm.hive_node : {
-        name       = "${var.cluster_name}-${i + 1}"
-        index      = i + 1
-        management = vm.ipv4_addresses[1][0]
-        data       = vm.ipv4_addresses[2][0]
-      }
-    ]
-    ssh_user         = "tf-user"
-    ssh_key_path     = var.ssh_private_key_path
-  }
-}
-```
-
-**State isolation:**
-
-Each cluster gets its own state file via `-state` flag:
-
-```bash
-tofu apply -state="clusters/${CLUSTER_NAME}/terraform.tfstate"
-```
-
-The `clusters/` directory is gitignored. Each cluster's state, inventory, and logs live in `clusters/{name}/`.
-
-### Phase 2: Configure
-
-A shell script that reads the Terraform outputs and sets up Hive on the provisioned VMs.
-
-**`scripts/iac/configure-cluster.sh <cluster_name>`:**
-
-1. Read inventory from `tofu output -json -state=clusters/{name}/terraform.tfstate`
-2. Wait for cloud-init to complete on all nodes (poll `/tmp/vendor-cloud-init-done`)
-3. On each node (parallel): clone/pull hive repo, run `make quickinstall`, `clone-deps.sh`, `dev-setup.sh`, `make build`
-4. On node 1: `hive admin init` with management IPs from inventory
-5. On remaining nodes (parallel): `hive admin join` pointing at node 1
-6. On all nodes: `hive start`
-7. Wait for health checks: poll `http://<ip>:4432/health` on all nodes
-
-This replaces the manual SSH commands currently in `scripts/iac/proxmox/README.md`. The `hive admin init` and `hive admin join` commands are generated from inventory data rather than hardcoded IPs.
-
-### Phase 3: Test
-
-**`scripts/iac/test-cluster.sh <cluster_name>`:**
-
-1. Read inventory to get node IPs
-2. Set `HIVE_ENDPOINT=https://<node1-management-ip>:9999`
-3. Run E2E tests from `tests/e2e/` against the real cluster
-4. Save results to `clusters/{name}/test-results/`
-
-This reuses the existing E2E test logic, currently Docker-only (`tests/e2e/run-e2e.sh`, `tests/e2e/run-multinode-e2e.sh`), pointed at real cluster IPs instead of localhost containers.
-
-### Phase 4: Destroy
-
-```bash
-tofu destroy -state="clusters/${CLUSTER_NAME}/terraform.tfstate"
-rm -rf "clusters/${CLUSTER_NAME}/"
-```
-
-### Lifecycle wrapper
-
-**`scripts/iac/hive-test.sh`** — single entry point:
-
-```bash
-#!/bin/bash
-# Usage: hive-test.sh <command> <cluster_name> [options]
-#
-# Commands:
-#   up          Provision VMs (Phase 1)
-#   configure   Install and start Hive (Phase 2)
-#   test        Run E2E tests (Phase 3)
-#   down        Destroy VMs and clean up (Phase 4)
-#   full        Run all phases: up -> configure -> test -> down
-#   status      Show cluster status (node IPs, health)
-#
-# Options (passed through to tofu apply):
-#   --node-count=N      Number of nodes (default: 3)
-#   --cpu-cores=N       vCPU per node (default: 4)
-#   --memory-mb=N       Memory per node in MB (default: 8192)
-#   --disk-size-gb=N    Disk per node in GB (default: 32)
-
-# Examples:
-#   hive-test.sh full my-test-cluster
-#   hive-test.sh up stress-test --node-count=5 --cpu-cores=8
-#   hive-test.sh down stress-test
-```
-
-The `full` command mirrors kubetest2's `--up --test --down` pattern: provision, configure, test, then tear down regardless of test outcome (with exit code preserved).
-
-## Files to Modify
-
-| File | Change |
-|---|---|
-| `scripts/iac/proxmox/variables.tf` | Remove `length == 3` validation, add `cluster_name`, `node_count`, `cpu_cores`, `memory_mb`, `disk_size_gb`, `os_image` |
-| `scripts/iac/proxmox/vms.tf` | Use `count` instead of `for_each` on `local.nodes`, parameterize VM name/specs, round-robin placement |
-| `scripts/iac/proxmox/cloud-config.tf` | Remove hive clone/quickinstall/dev-setup, keep OS essentials only |
-| `scripts/iac/proxmox/main.tf` | No changes (provider config stays the same) |
-| `scripts/iac/proxmox/outputs.tf` | **New** — structured inventory output (IPs, SSH config, cluster name) |
-| `scripts/iac/configure-cluster.sh` | **New** — Phase 2: install hive, form cluster, start services |
-| `scripts/iac/test-cluster.sh` | **New** — Phase 3: run E2E tests against real cluster |
-| `scripts/iac/hive-test.sh` | **New** — lifecycle wrapper (up/configure/test/down/full) |
-| `scripts/iac/proxmox/.gitignore` | Add `clusters/` directory |
-
-## Multi-Environment Usage Examples
-
-```bash
-# Source Proxmox credentials (unchanged from today)
-source scripts/iac/proxmox/.env
-
-# Environment 1: standard 3-node dev cluster
-./scripts/iac/hive-test.sh up hive-dev1
-
-# Environment 2: 5-node stress test with larger VMs
-./scripts/iac/hive-test.sh up hive-stress --node-count=5 --cpu-cores=8 --memory-mb=16384
-
-# Configure and test environment 1
-./scripts/iac/hive-test.sh configure hive-dev1
-./scripts/iac/hive-test.sh test hive-dev1
-
-# Full lifecycle (CI use case): provision, configure, test, destroy
-./scripts/iac/hive-test.sh full hive-ci-$(date +%s)
-
-# Check status of any cluster
-./scripts/iac/hive-test.sh status hive-dev1
-
-# Tear down when done
-./scripts/iac/hive-test.sh down hive-stress
-./scripts/iac/hive-test.sh down hive-dev1
-```
-
-## Testing
-
-- Provision a 1-node cluster, verify VM comes up with correct name and specs
-- Provision a 3-node cluster (matches current setup), verify parity with existing behavior
-- Run configure phase, verify all nodes have hive running and cluster formed
-- Run test phase, verify E2E tests execute against real cluster
-- Run destroy phase, verify VMs are removed and state is cleaned
-- Provision two clusters simultaneously, verify no naming or state conflicts
-- Test the `full` lifecycle end-to-end
-
-## Future Work
-
-- **CI integration**: GitHub Actions workflow that provisions real Proxmox VMs for E2E tests on PR, with automatic teardown
-- **AWS provider parity**: Same lifecycle wrapper for a future `scripts/iac/aws/` backend, keeping the `up/configure/test/down` contract
-- **Test matrices**: Parameterized runs varying node count and instance sizes across a matrix (like avocado), e.g. `{1,3,5} nodes x {4,8} cores`
-- **Ansible migration**: Replace `configure-cluster.sh` with Ansible playbooks when shell scripting complexity grows (inventory.json maps cleanly to Ansible inventory)
-- **Re-configure without re-provision**: Since cloud-init no longer installs hive, `configure` can be re-run to deploy a new hive build without destroying/recreating VMs

docs/development/improvements/multi-server-restart.md

@@ -1,681 +0,0 @@
-# Multi-Server Cluster Restart
-
-**Status: In Progress** — Phase 0 and Phase 1 complete. Phase 2 and 3 planned.
-
-## Summary
-
-When a multi-node Hive cluster restarts (stop-dev.sh then start-dev.sh on each node), every daemon independently loads the full instance state from NATS JetStream KV and attempts to relaunch all VMs simultaneously. This creates race conditions, duplicate launch attempts, and resource exhaustion. We need a coordinated recovery mechanism inspired by Ceph, Kubernetes, and etcd patterns.
-
-Additionally, the cluster assumes every node runs all services (NATS, Predastore, Viperblock, Daemon, AWSGW, UI). In practice, nodes should be able to run a subset — e.g., a compute-only node running just Daemon + Viperblock that connects to remote NATS and Predastore. The config, startup scripts, health checks, and recovery logic all need to be capability-aware.
-
-## Problem Statement
-
-### Current behavior
-
-Each daemon's `restoreInstances()` (`daemon.go:634-719`) runs independently on startup:
-
-1. Loads `node.{nodeID}` from JetStream KV — gets its own previous instances
-2. For each instance that was `StateRunning` but whose QEMU exited, resets to `StatePending` and calls `LaunchInstance()`
-3. All instances launch simultaneously with no throttling
-4. No coordination with other daemons — no awareness of cluster readiness
-
-### Race conditions and failure modes
-
-1. **Thundering herd on storage**: All nodes launch all their VMs at once. Each VM requires Viperblock volume mounts, NBD connections, and AMI downloads from Predastore. This saturates storage I/O before services are fully ready.
-
-2. **JetStream quorum unavailable**: In a 3-node cluster with 3 replicas, JetStream needs 2 NATS servers for Raft quorum. If node1 starts restoring instances before node2's NATS is up, KV writes may fail or block. The current 10-attempt retry in `initJetStream()` (`daemon.go:574-615`) helps but doesn't guarantee other services (Predastore, Viperblock) are ready.
-
-3. **No cluster-wide readiness gate**: Daemons start restoring VMs as soon as their own JetStream init succeeds, regardless of whether Predastore, Viperblock, or peer daemons are healthy. Volume mounts fail because Viperblock isn't ready; AMI fetches fail because Predastore hasn't elected a leader.
-
-4. **No recovery throttling**: A node with 10 previously-running VMs will attempt to launch all 10 simultaneously — flooding QEMU starts, volume mounts, and metadata server allocations.
-
-5. **Stopped instance coordination**: Stopped instances live in shared KV (`instance.{id}`) and can be started by any node via `ec2.start` with the `hive-workers` queue group. On restart, the `handleEC2StartStoppedInstance` handler (`daemon_handlers.go:1363-1463`) uses `Put` (not CAS) to claim instances, so two nodes could theoretically race on the same start request.
-
-6. **No clean vs crash shutdown distinction**: The daemon writes state to KV during graceful shutdown (`setupShutdown()` at `daemon.go:1224-1228`), but there's no marker distinguishing a clean shutdown from a crash. After a crash, the KV state may be stale (last write before crash), and QEMU processes may be orphaned.
-
-### Heterogeneous node problems
-
-7. **All-or-nothing service startup**: `start-dev.sh` unconditionally starts all 6 services (NATS, Predastore, Viperblock, Daemon, AWSGW, UI) on every node. A compute-only node that connects to remote NATS/Predastore will fail because it tries to start local instances of services it doesn't need.
-
-8. **No capability model in config**: `Config` struct (`config/config.go:22-41`) has no field indicating which services a node runs. The `hive.toml` template always generates all service sections. There's no way to express "this node runs daemon + viperblock, but uses remote NATS and Predastore."
-
-9. **Health checks assume local services**: The daemon's `waitForClusterReady()` (proposed in Phase 1) must distinguish between "wait for local Viperblock to start" vs "wait for remote Predastore at 10.1.3.170:8443 to be reachable" — currently there's no way to know which is which.
-
-10. **Formation and heartbeats lack capability info**: `formation.NodeInfo` and `NodeHealthResponse` have no capability/role fields. The recovery coordinator can't make capacity-aware decisions without knowing which nodes can run VMs vs which are storage-only.
-
-## Research: Distributed Systems Patterns
-
-### Ceph OSD restart patterns
-
-- **Backfill reservation**: Limits concurrent recovery operations per OSD via `osd_max_backfills` (default: 1). Each OSD has local and remote reservers — a semaphore-based system that prevents thundering herd. Priority ordering ensures degraded/critical recovery happens first.
-- **Recovery delay**: `osd_recovery_delay_start` introduces a deliberate delay after OSD boot before recovery begins, allowing the cluster to stabilize (peering, Raft leader election).
-- **`noout` flag**: Distinguishes planned restart (don't rebalance) from permanent failure (rebalance after grace period). Set before maintenance, unset after.
-- **Peering and epochs**: Before serving I/O, an OSD must verify its cluster map epoch is current via `up_thru`. Prevents stale decisions based on outdated topology.
-
-### Kubernetes node restart patterns
-
-- **Node lease heartbeats**: Kubelets maintain a `Lease` object with `renewTime`. If not renewed within `leaseDurationSeconds` (40s), the node controller marks the node `NotReady`.
-- **Taint-based eviction timeline**: T+0 miss heartbeat → T+40s mark NotReady → T+340s (5min grace) evict pods. Deliberately slow to tolerate transient failures. When a node recovers, taints auto-clear but evicted pods stay on their new nodes.
-- **Pod Disruption Budgets**: Limit concurrent disruptions cluster-wide, ensuring minimum availability during rolling restarts.
-- **Learner pattern** (from etcd): New members join as non-voting learners, catch up on state, then get promoted to voters. Prevents a partially-synced member from affecting quorum.
-- **Node labels and taints**: Kubernetes uses labels (not hard-coded types) to describe node capabilities. Scheduling matches workload requirements to node labels via affinity rules. A node can have multiple roles (control-plane + worker) or be specialized.
-
-### etcd cluster recovery
-
-- **Quorum requirement**: Needs `(N/2)+1` members for reads/writes. In a 3-node cluster, losing 2 nodes makes the cluster read-only or unavailable.
-- **Raft log replay**: On restart, members replay their WAL before accepting requests. No client traffic until replay complete.
-- **Learner promotion**: Members start as learners, sync data, then promote to full members. Safe and reversible.
-
-### Consul/Nomad server vs client pattern
-
-- **Two-tier architecture**: Server nodes participate in Raft consensus; client nodes run workloads and forward RPCs to servers. A client is configured with `retry_join` — a list of server addresses.
-- **Role advertising via gossip tags**: Serf gossip carries per-node metadata (role, datacenter, port). Any node can inspect peers' capabilities without a central registry.
-- **Service registration**: Each node's local agent registers only the services it runs. Health checks are per-service, not per-node.
-
-### Key takeaways for Hive
-
-| Pattern | Source | Application to Hive |
-|---------|--------|---------------------|
-| Recovery delay | Ceph | Wait for cluster readiness before restoring VMs |
-| Concurrency limit | Ceph | Max N concurrent VM launches per node during recovery |
-| Heartbeat leases | K8s | Daemons publish heartbeats to JetStream KV for peer monitoring |
-| Grace period | K8s | Don't reschedule VMs from a down node for 5 minutes |
-| Quorum gate | etcd | Don't begin recovery until JetStream has quorum |
-| Fencing tokens | Kleppmann | Use KV revision-based CAS to prevent duplicate VM claims |
-| Clean shutdown marker | General | Distinguish crash from graceful stop for recovery decisions |
-| Node labels | K8s | Capabilities as config-driven labels, not hard-coded types |
-| Static bootstrap + dynamic registration | Consul/Ceph | Config lists remote service addresses; node advertises its own capabilities at runtime |
-| Per-service health checks | Consul | Only health-check what the node actually runs |
-
-## Proposed Changes
-
-### Phase 0: Node capabilities and service topology — Complete
-
-Foundation for everything else — the cluster must know what each node runs.
-
-#### 0.1 Capabilities in config — Done
-
-Added `Services []string` field to the `Config` struct (`config/config.go:29`) and two helper methods:
-- `HasService(name string) bool` — checks if a service is in the list (or returns true for all if list is empty, for backward compat)
-- `GetServices() []string` — returns the services list, defaulting to `AllServices` when empty
-
-The `AllServices` package variable defines the canonical list: `["nats", "predastore", "viperblock", "daemon", "awsgw", "ui"]`.
-
-The originally-planned role helpers (`IsComputeNode`, `IsStorageNode`, `IsInfraNode`) were deferred — `HasService()` is sufficient and more flexible.
-
-Updated `NodeHealthResponse` to include `Services []string` and `ServiceHealth map[string]string`.
-
-Unit tests added in `hive/config/config_test.go`: `TestHasService_ExplicitList`, `TestHasService_EmptyListDefaultsAll`, `TestHasService_UnknownService`, `TestGetServices_DefaultsToAll`, `TestGetServices_ExplicitList`.
-
-**Files modified:**
-- `hive/config/config.go` — `Services` field, `AllServices`, `HasService()`, `GetServices()`, updated `NodeHealthResponse`
-- `hive/config/config_test.go` — 5 new tests
-
-#### 0.2 Formation carries capabilities — Done
-
-Added `Services []string` to `NodeInfo` in `hive/formation/formation.go`. Since `JoinRequest` embeds `NodeInfo`, it automatically gets the field.
-
-Updated `BuildClusterRoutes()` to filter nodes by `"nats"` service — only nodes running NATS are included in cluster routes. Updated `BuildPredastoreNodes()` to filter by `"predastore"` service. Added `hasService()` helper with backward-compat behavior (empty services list = all services).
-
-Unit tests added in `hive/formation/helpers_test.go`: `TestHasService_EmptyListMeansAll`, `TestHasService_ExplicitList`, `TestBuildClusterRoutes_FiltersNonNATSNodes`, `TestBuildClusterRoutes_EmptyServicesIncludesAll`, `TestBuildPredastoreNodes_FiltersByPredastore`, `TestBuildPredastoreNodes_EmptyServicesIncludesAll`, `TestBuildClusterRoutes_MixedServicesAndEmpty`.
-
-**Files modified:**
-- `hive/formation/formation.go` — `Services` field on `NodeInfo`
-- `hive/formation/helpers.go` — `hasService()` helper, filtered `BuildClusterRoutes()` and `BuildPredastoreNodes()`
-- `hive/formation/helpers_test.go` — 7 new tests
-
-#### 0.3 Config template and CLI flags — Done
-
-Added `Services []string` to `ConfigSettings` in `hive/admin/admin.go`.
-
-Added `services` template rendering in `cmd/hive/cmd/templates/hive.toml` (after the `az` line).
-
-Added `--services` StringSlice flag to both `adminInitCmd` and `adminJoinCmd` in `cmd/hive/cmd/admin.go`. The flag is read in `runAdminInit`, `runAdminInitMultiNode`, and `runAdminJoin`, and passed through to `ConfigSettings`.
-
-**Files modified:**
-- `hive/admin/admin.go` — `Services` field on `ConfigSettings`
-- `cmd/hive/cmd/templates/hive.toml` — `services` template line
-- `cmd/hive/cmd/admin.go` — `--services` flag on init/join, wiring to `ConfigSettings`
-
-#### 0.4 Capability-aware start/stop scripts — Done
-
-Added `parse_services()` and `has_service()` bash functions to both `scripts/start-dev.sh` and `scripts/stop-dev.sh`.
-
-`parse_services()` reads the `services` line from `hive.toml` and defaults to all services if not set. All 6 service start/stop blocks are wrapped in `has_service` conditionals with skip messages for non-local services.
-
-The stop script maps between process names and config service names (e.g., `hive` → `daemon`, `hive-ui` → `ui`).
-
-**Files modified:**
-- `scripts/start-dev.sh` — `parse_services()`, `has_service()`, conditional service startup
-- `scripts/stop-dev.sh` — `parse_services()`, `has_service()` (with name mapping), conditional service stop
-
-#### 0.5 Health endpoint and cluster state KV — Done
-
-**Health endpoint**: Updated the `/health` HTTP handler in `daemon.go` to include `Services` (from config) and `ServiceHealth` (per-service status map). For local services, status is `"ok"`. For remote NATS dependency (when nats is not a local service), checks `natsConn.IsConnected()` and reports `"remote_ok"` or `"remote_unreachable"`.
-
-**Cluster state KV bucket**: Added `hive-cluster-state` bucket alongside the existing `hive-instance-state` bucket. New `clusterKV` field on `JetStreamManager`. `InitClusterStateBucket()` creates the bucket with 1-hour TTL and history of 1. Called from `initJetStream()` after `InitKVBucket()`.
-
-**Service manifest**: `WriteServiceManifest()` writes `node.{nodeID}.services` to the cluster state KV on daemon startup, containing the node's services, NATS host, predastore host, and timestamp.
-
-`UpdateReplicas()` updated to also update the cluster-state bucket replicas alongside the instance-state bucket.
-
-**Files modified:**
-- `hive/daemon/daemon.go` — health endpoint with services/serviceHealth, `InitClusterStateBucket()` call in `initJetStream()`, `WriteServiceManifest()` call in `Start()`
-- `hive/daemon/jetstream.go` — `ClusterStateBucket` const, `clusterKV` field, `InitClusterStateBucket()`, `WriteServiceManifest()`, updated `UpdateReplicas()`
-
-### Phase 1: Self-recovery improvements (single-node correctness) — Complete
-
-These changes fix the immediate restart bugs without requiring cross-node coordination.
-
-#### 1.1 Cluster readiness gate — Done
-
-Added `waitForClusterReady()` method to `Daemon`, called in `Start()` just before `restoreInstances()`. Uses a 2-minute max wait with 2-second polling interval.
-
-Two checks run in sequence:
-1. **Viperblock** — `checkViperblockReady()` verifies NATS connection is live. Initially implemented as a NATS request to `ebs.{node}.health`, but viperblock has no health NATS topic — this caused a 2-minute hang during E2E tests. Fixed to simply check `d.natsConn.IsConnected()` since viperblock runs alongside the daemon and subscribes to NATS.
-2. **Predastore** — `checkPredastoreReady()` does a TCP dial to the predastore host:port from config with a 3-second timeout. Skipped if no predastore host is configured.
-
-If all checks pass, recovery proceeds immediately. If the 2-minute timeout expires, recovery proceeds anyway with a warning log.
-
-**Finding**: Viperblock does not expose a health endpoint via NATS. Any future viperblock health check should use TCP connectivity or an HTTP endpoint, not NATS request-reply.
-
-**Files modified:**
-- `hive/daemon/daemon.go` — `waitForClusterReady()`, `checkViperblockReady()`, `checkPredastoreReady()`, added `"net"` import
-
-#### 1.2 Recovery throttling with semaphore — Done
-
-Refactored `restoreInstances()` into two phases:
-
-**Phase 1 (sequential)**: Iterates all instances from KV state. For each instance:
-- Reconnects to still-running QEMU processes (checks PID file, instant, no I/O)
-- Finalizes transitional states (stopping → stopped, shutting-down → terminated)
-- Collects instances that need relaunching into a `toLaunch` list
-
-**Phase 2 (throttled)**: Launches crashed VMs using a semaphore channel with `maxConcurrentRecovery = 2`. Each launch runs in a goroutine that acquires the semaphore before proceeding and releases on completion. A `sync.WaitGroup` ensures all launches complete before `restoreInstances()` returns.
-
-This prevents thundering herd on storage during recovery while keeping the lightweight reconnect/finalize steps sequential.
-
-**Files modified:**
-- `hive/daemon/daemon.go` — refactored `restoreInstances()` with two-phase approach and semaphore
-
-#### 1.3 Clean shutdown marker — Done
-
-Three new methods on `JetStreamManager`:
-- `WriteShutdownMarker(nodeID)` — writes `shutdown.{nodeID}` to `hive-cluster-state` KV with node name and timestamp
-- `ReadShutdownMarker(nodeID)` — checks if a shutdown marker exists for the node
-- `DeleteShutdownMarker(nodeID)` — deletes the marker (tolerates not-found)
-
-**Write**: In `setupShutdown()`, the shutdown marker is written to cluster state KV before `WriteState()` persists instance state. This ensures the marker is present if the state write also succeeds.
-
-**Read**: At the start of `restoreInstances()`, the marker is checked. If present: log "Clean shutdown marker found, trusting KV state", delete the marker, proceed normally. If absent: log a warning about possible crash recovery, sleep 3 seconds to let orphaned QEMU processes finish exiting, then proceed with careful PID validation.
-
-**Files modified:**
-- `hive/daemon/jetstream.go` — `WriteShutdownMarker()`, `ReadShutdownMarker()`, `DeleteShutdownMarker()`
-- `hive/daemon/daemon.go` — marker write in `setupShutdown()`, marker check/delete in `restoreInstances()`
-
-#### 1.4 Service manifest on startup — Done
-
-Added `WriteServiceManifest()` to `JetStreamManager`, which writes `node.{nodeID}.services` to the cluster state KV containing the node's service list, NATS host, predastore host, and timestamp.
-
-Called in `Start()` after `initJetStream()` succeeds. Failure is non-fatal (logged as warning).
-
-**Files modified:**
-- `hive/daemon/jetstream.go` — `WriteServiceManifest()`
-- `hive/daemon/daemon.go` — call in `Start()`
-
-### Phase 2: Cross-node coordination
-
-These changes enable daemons to coordinate lifecycle events — both failure recovery and graceful cluster-wide shutdown.
-
-#### 2.1 Heartbeat publishing
-
-Each daemon publishes a heartbeat to `hive-cluster-state` KV every 10 seconds:
-
-```
-Key:   heartbeat.{nodeID}
-Value: {"node": "node1", "epoch": 1, "timestamp": "...", "services": ["daemon", "viperblock", ...], "vm_count": 4, "vcpu_avail": 8, "mem_avail_gb": 16.0}
-```
-
-Use `kv.Put()` — no need for CAS since only the owning daemon writes its own heartbeat. Other daemons use `kv.Watch("heartbeat.*")` for push-based notification of changes — no polling. The 1-hour TTL on the bucket auto-cleans dead node heartbeats.
-
-Heartbeat publishing starts after self-recovery completes (after `restoreInstances()` returns), so the heartbeat reflects accurate resource state.
-
-**Files:**
-- `hive/daemon/jetstream.go` — new `WriteHeartbeat()` / `WatchHeartbeats()`
-- `hive/daemon/daemon.go` — start heartbeat goroutine after `restoreInstances()`
-
-#### 2.2 Peer failure detection
-
-After self-recovery and heartbeat publishing begin, each daemon monitors peer heartbeats:
-
-- **Stale threshold**: 60 seconds (6 missed heartbeats). Mark peer as `suspect`.
-- **Failed threshold**: 300 seconds (5 minutes). Mark peer as `failed` — eligible for VM recovery.
-
-These thresholds are deliberately conservative (matching Kubernetes' approach) to avoid false positives during rolling restarts where nodes come back quickly.
-
-**Files:**
-- `hive/daemon/daemon.go` — new `monitorPeerHealth()` goroutine, started alongside heartbeat publisher
-
-#### 2.3 Leader-elected recovery coordinator (capability-aware)
-
-When a peer is detected as `failed`, surviving daemons elect a recovery leader using JetStream KV CAS:
-
-```
-Key:   leader.recovery
-Value: {"node": "node2", "epoch": 42, "acquired": "..."}
-```
-
-Acquisition uses `kv.Create()` (create-if-not-exists) for atomic leader election. The leader holds the lease for 60 seconds and must renew it. If the leader fails, another daemon acquires it after the lease expires.
-
-The recovery leader:
-1. Reads the failed node's state from `node.{failedNode}` in `hive-instance-state` KV
-2. Filters to VMs that were `StateRunning` or `StatePending`
-3. Assigns VMs to surviving **compute-capable** nodes (those with `"daemon"` in their services, from heartbeat data) based on resource availability
-4. Writes recovery assignments to KV: `recovery.{instanceID} = {"target_node": "node2", "epoch": 42}`
-5. Each target daemon watches for its assignments and launches them
-
-**Capability-aware placement**: The leader only assigns VMs to nodes whose heartbeats include `"daemon"` in their `services` list and have sufficient `vcpu_avail` and `mem_avail_gb`. Storage-only nodes are never assigned VMs.
-
-**Fencing**: Each assignment includes the recovery epoch. Before launching, the target daemon verifies the epoch is still current via CAS. If another leader took over with a higher epoch, the stale assignment is ignored.
-
-**Files:**
-- `hive/daemon/jetstream.go` — new `AcquireRecoveryLease()` / `RenewRecoveryLease()` / `WriteRecoveryAssignment()` / `WatchRecoveryAssignments()`
-- `hive/daemon/recovery.go` — new file, recovery coordinator logic
-- `hive/daemon/daemon.go` — integrate recovery coordinator startup
-
-#### 2.4 Coordinated cluster shutdown
-
-##### Problem
-
-The current `stop-dev.sh` operates per-node: each node independently shuts down its own services in reverse startup order. In a multi-node cluster, this creates ordering hazards:
-
-1. **Storage pulled from under running VMs**: If node1 stops Predastore before node2's VMs have finished flushing writes through Viperblock, data can be lost or corrupted.
-2. **Cascade failures during rolling stop**: Stopping NATS on one node can break JetStream quorum before other nodes have persisted their state.
-3. **Manual coordination required**: An operator must SSH into each node and run `stop-dev.sh` in the right order — infrastructure nodes last, compute nodes first. This is error-prone and doesn't scale.
-4. **No cluster-wide drain**: There's no way to stop accepting new API requests across all gateways simultaneously. A request arriving at node2's AWSGW during node1's shutdown could try to launch a VM on a node that's mid-teardown.
-5. **Orphaned nbdkit processes**: If a daemon exits before cleanly unmounting all volumes, nbdkit processes may be left running with no parent to clean them up. The current `stop-dev.sh` handles this via `pgrep`/`pkill` but only for the local node.
-
-##### Design inspiration
-
-The coordinated shutdown draws from several distributed systems patterns:
-
-| Source | Pattern | Application to Hive |
-|--------|---------|---------------------|
-| Ceph | Pre-shutdown flags (`noout`, `norecover`, `nobackfill`) set before maintenance; monitors shut down last | Write cluster shutdown marker before any service stops; NATS shuts down last |
-| Ceph | Ordered daemon shutdown: RGW → MDS → OSD → MON | AWSGW → Daemon/VMs → Viperblock → Predastore → NATS |
-| Kubernetes | Graceful node shutdown (KEP-2000): kubelet uses systemd inhibitor locks, pods sorted by priority class, `terminationGracePeriodSeconds` per pod | Per-VM QMP `system_powerdown` with configurable timeout, force kill after grace period |
-| Kubernetes | Pod termination lifecycle: preStop hook → SIGTERM → grace period → SIGKILL | Phase-based shutdown with ACK between phases; no phase proceeds until prior phase completes |
-| Kubernetes | `kubectl drain` cordons a node (marks unschedulable) before evicting pods | GATE phase stops AWSGW to prevent new work before draining VMs |
-| Rancher/Harvester | Ordered HCI shutdown: cordon node → migrate VMs → drain workloads → power off; fleet-managed lifecycle for multi-cluster | Coordinator-driven phased shutdown across all nodes; progress tracking via NATS |
-| RKE2 | `rke2-killall.sh` handles both graceful and forceful modes with explicit cleanup of iptables, mounts, and network namespaces | `--force` flag for immediate shutdown; explicit nbdkit/QEMU cleanup in DRAIN phase |
-| etcd | Learner removal before shutdown; leader transfer to minimize disruption | Coordinator is last to stop NATS; non-coordinator nodes stop NATS first |
-
-##### Command
-
-```bash
-# Graceful shutdown — waits for VMs to power down cleanly
-./bin/hive admin cluster shutdown
-
-# Force shutdown — SIGKILL VMs immediately, skip grace periods
-./bin/hive admin cluster shutdown --force
-
-# Custom timeout for VM drain phase (default: 120s)
-./bin/hive admin cluster shutdown --timeout 300s
-
-# Dry run — show what would happen without executing
-./bin/hive admin cluster shutdown --dry-run
-```
-
-The command can be run from **any node** in the cluster. The issuing node becomes the shutdown coordinator. If the coordinator crashes mid-shutdown, any surviving node can detect the `cluster.shutdown` KV marker and resume coordination (similar to the recovery leader election pattern from 2.3).
-
-##### Shutdown protocol
-
-The protocol uses five phases, each gated by acknowledgements from all participating nodes. The coordinator drives transitions and aggregates progress. NATS request-reply ensures reliable delivery with timeouts per phase.
-
-```
-Phase 1: GATE ──────► Phase 2: DRAIN ──────► Phase 3: STORAGE ──────► Phase 4: PERSIST ──────► Phase 5: INFRA
-(stop new work)        (stop VMs)             (stop block I/O)         (stop object storage)    (stop NATS)
-```
-
-**Phase 1: GATE** — stop accepting new work
-
-Purpose: Ensure no new API requests enter the system while shutdown proceeds. Analogous to Ceph's `noout`/`norecover` flags and Kubernetes' node cordon.
-
-1. Coordinator writes `cluster.shutdown` marker to `hive-cluster-state` KV:
-   ```
-   Key:   cluster.shutdown
-   Value: {"initiator": "node1", "phase": "gate", "started": "2024-...", "timeout": "120s", "force": false}
-   ```
-2. Coordinator publishes `hive.cluster.shutdown.gate` (NATS request, expecting N replies)
-3. Each node on receiving the gate request:
-   - Stops AWSGW service (no new API requests accepted)
-   - Stops hive-ui service
-   - Sets internal `shuttingDown` flag to reject any in-flight NATS work requests
-   - ACKs to coordinator: `{"node": "node2", "phase": "gate", "stopped": ["awsgw", "ui"]}`
-4. Coordinator waits for all ACKs (timeout: 30s per node)
-5. On timeout: log which nodes didn't respond, proceed anyway (they may be already down)
-
-**Phase 2: DRAIN** — gracefully terminate all VMs
-
-Purpose: Ensure all VMs are cleanly shut down and their volumes unmounted before storage services stop. This is the critical data-safety phase. Mirrors Kubernetes' pod termination lifecycle: preStop → SIGTERM → grace period → SIGKILL.
-
-1. Coordinator publishes `hive.cluster.shutdown.drain`
-2. Each daemon node:
-   - Sends QMP `system_powerdown` to each running QEMU instance (same as `setupShutdown()` today)
-   - Waits for QEMU processes to exit, polling PID files
-   - Reports progress via `hive.cluster.shutdown.progress`:
-     ```json
-     {"node": "node2", "phase": "drain", "total": 4, "remaining": 2, "vms": ["i-abc123", "i-def456"]}
-     ```
-   - After per-VM grace period (default 60s), sends SIGTERM then SIGKILL to stubborn QEMU processes
-   - Unmounts all EBS volumes via `ebs.unmount` NATS requests (viperblock still running)
-   - Cleans up any orphaned nbdkit processes attached to terminated VMs
-   - Writes instance state to JetStream KV (same as `WriteState()` in `setupShutdown()`)
-   - Writes per-node shutdown marker (reuses Phase 1's `WriteShutdownMarker()`)
-   - ACKs when all VMs have exited and volumes are unmounted
-3. Coordinator aggregates progress, prints live status
-4. `--force` mode: skip QMP `system_powerdown`, immediately SIGKILL all QEMU processes
-5. On timeout: coordinator logs remaining VMs, force-kills them, proceeds
-
-**Phase 3: STORAGE** — stop block storage
-
-Purpose: Shut down viperblock and clean up all nbdkit processes. Safe to do now because all VMs are terminated and volumes unmounted.
-
-1. Coordinator publishes `hive.cluster.shutdown.storage`
-2. Each node with viperblock service:
-   - Stops viperblock service
-   - Scans for and kills any orphaned nbdkit processes (`pkill -f nbdkit`)
-   - Verifies no NBD block devices remain active
-   - ACKs when complete
-3. Nodes without viperblock: ACK immediately
-4. Coordinator waits for all ACKs (timeout: 30s)
-
-**Phase 4: PERSIST** — stop object storage
-
-Purpose: Shut down Predastore cleanly. Safe because no viperblock or AWSGW requests remain. Predastore may still have internal replication traffic between nodes — the coordinator ensures all nodes stop together.
-
-1. Coordinator publishes `hive.cluster.shutdown.persist`
-2. Each node with predastore service:
-   - Stops predastore service
-   - ACKs when complete
-3. Coordinator waits for all ACKs (timeout: 30s)
-
-**Phase 5: INFRA** — stop NATS (coordinator last)
-
-Purpose: Shut down the coordination layer itself. This is fire-and-forget because the communication channel is being torn down.
-
-1. Coordinator publishes `hive.cluster.shutdown.infra` to all **non-coordinator** nodes
-2. Non-coordinator nodes: stop local NATS and exit the shutdown handler
-3. Coordinator waits 5s for NATS cluster to stabilize with fewer members
-4. Coordinator stops its own NATS
-5. Coordinator logs "Cluster shutdown complete" and exits
-
-##### Coordinator election and failure handling
-
-The coordinator is simply the node that runs the `hive admin cluster shutdown` command. If the coordinator crashes mid-shutdown:
-
-1. The `cluster.shutdown` KV marker persists (1-hour TTL, long enough for any shutdown)
-2. The marker records the current phase and which nodes have ACKed
-3. Any surviving daemon detects the marker via `kv.Watch("cluster.shutdown")`
-4. After a coordinator timeout (30s without progress updates), a surviving daemon takes over using the same CAS pattern as recovery leader election (2.3)
-5. The new coordinator resumes from the last completed phase
-
-If a node is unreachable during any phase:
-- Coordinator logs a warning and continues after the per-phase timeout
-- Unreachable nodes may have already shut down (e.g., power loss)
-- If they come back later, they'll see the `cluster.shutdown` marker and refuse to start services until the marker is cleared
-
-##### Status reporting
-
-The coordinator prints live progress to stdout:
-
-```
-Cluster shutdown initiated from node1 (3 nodes)
-
-Phase 1/5: GATE — Stopping API gateways and UI...
-  node1: awsgw stopped, ui stopped               [ok]
-  node2: awsgw stopped, ui stopped               [ok]
-  node3: awsgw stopped, ui stopped               [ok]
-
-Phase 2/5: DRAIN — Stopping virtual machines...
-  node1: 4/4 VMs stopped                         [ok]     12.3s
-  node2: 2/2 VMs stopped                         [ok]      8.7s
-  node3: 1/1 VMs stopped                         [ok]      6.1s
-
-Phase 3/5: STORAGE — Stopping block storage...
-  node1: viperblock stopped, 0 nbdkit cleaned    [ok]
-  node2: viperblock stopped, 0 nbdkit cleaned    [ok]
-  node3: viperblock stopped, 0 nbdkit cleaned    [ok]
-
-Phase 4/5: PERSIST — Stopping object storage...
-  node1: predastore stopped                      [ok]
-  node2: predastore stopped                      [ok]
-  node3: predastore stopped                      [ok]
-
-Phase 5/5: INFRA — Stopping NATS...
-  node2: nats stopped
-  node3: nats stopped
-  node1: nats stopped (coordinator, last)
-
-Cluster shutdown complete (47.3s)
-```
-
-##### NATS topics
-
-| Topic | Pattern | Purpose |
-|-------|---------|---------|
-| `hive.cluster.shutdown.gate` | Request-Reply | Phase 1: stop AWSGW + UI, await ACK |
-| `hive.cluster.shutdown.drain` | Request-Reply | Phase 2: stop VMs, await ACK |
-| `hive.cluster.shutdown.progress` | Publish (broadcast) | Progress updates during drain (VM counts) |
-| `hive.cluster.shutdown.storage` | Request-Reply | Phase 3: stop viperblock + nbdkit, await ACK |
-| `hive.cluster.shutdown.persist` | Request-Reply | Phase 4: stop predastore, await ACK |
-| `hive.cluster.shutdown.infra` | Publish (fire-and-forget) | Phase 5: stop NATS on non-coordinator nodes |
-
-##### Cluster shutdown KV state
-
-```
-Bucket:  hive-cluster-state
-Key:     cluster.shutdown
-Value:   {
-    "initiator": "node1",
-    "phase": "drain",
-    "started": "2024-01-15T10:30:00Z",
-    "timeout": "120s",
-    "force": false,
-    "nodes_total": 3,
-    "nodes_acked": {
-        "node1": {"phase": "gate", "at": "..."},
-        "node2": {"phase": "gate", "at": "..."},
-        "node3": {"phase": "gate", "at": "..."}
-    }
-}
-```
-
-##### Integration with existing code
-
-- **Reuses `stopInstance()`**: The daemon's existing `stopInstance()` function handles QMP shutdown, PID polling, volume unmount, and force kill. The DRAIN phase calls this same code path rather than reimplementing it.
-- **Reuses `WriteShutdownMarker()`**: Each node writes its per-node shutdown marker during DRAIN, so on next startup, `restoreInstances()` sees a clean shutdown.
-- **Reuses `./bin/hive service <name> stop`**: For stopping AWSGW, UI, viperblock, predastore, and NATS, the coordinator tells each daemon to invoke the same service stop mechanism used by `stop-dev.sh`.
-- **`setupShutdown()` becomes cluster-aware**: When a daemon receives a cluster shutdown request, it skips independent shutdown and instead participates in the coordinated protocol. If a SIGTERM arrives while a coordinated shutdown is in progress, the daemon ACKs the current phase and exits.
-- **`stop-dev.sh` remains for single-node dev**: The script continues to work for local development. For multi-node clusters, `hive admin cluster shutdown` is the preferred approach.
-
-##### Comparison with per-node `stop-dev.sh`
-
-| Aspect | `stop-dev.sh` (current) | `hive admin cluster shutdown` (proposed) |
-|--------|------------------------|------------------------------------------|
-| Scope | Single node | Entire cluster from any node |
-| Ordering | Per-node reverse order | Cluster-wide phased ordering |
-| VM safety | Daemon exits, then waits for QEMU via `pgrep` | Daemon actively monitors each VM's QMP shutdown |
-| Storage safety | Hopes VMs exited before predastore stop | Guarantees all VMs stopped before storage phases |
-| nbdkit cleanup | `pgrep`/`pkill` after the fact | Explicit unmount + cleanup during DRAIN |
-| Progress | Per-service echo messages | Cluster-wide live progress with VM counts |
-| Failure handling | None — operator must check each node | Automatic coordinator failover, timeout handling |
-| NATS safety | Stops NATS whenever it gets there | NATS is always last, coordinator node last of all |
-
-**Files:**
-- `hive/daemon/daemon.go` — shutdown handler that participates in coordinated protocol, `shuttingDown` flag to reject new work
-- `hive/daemon/shutdown.go` — **new file**, coordinator logic: phase management, ACK aggregation, progress reporting, coordinator failover
-- `hive/daemon/jetstream.go` — `WriteClusterShutdown()` / `ReadClusterShutdown()` / `DeleteClusterShutdown()` / `WatchClusterShutdown()`
-- `cmd/hive/cmd/admin.go` — new `cluster shutdown` subcommand with `--force`, `--timeout`, `--dry-run` flags
-- `hive/daemon/daemon_handlers.go` — NATS handlers for `hive.cluster.shutdown.*` topics
-
-### Phase 3: Operational tooling
-
-#### 3.1 Maintenance mode flag
-
-Add a `maintenance.{nodeID}` KV key that daemons can set before planned restarts:
-
-```bash
-./bin/hive admin maintenance enable --node node1
-```
-
-When maintenance mode is set for a node:
-- Peer daemons skip the 5-minute failure detection grace period for that node (they know it's planned)
-- The node itself skips VM recovery on restart if the maintenance flag is still set (operator controls when VMs come back)
-- Analogous to Ceph's `noout` flag
-- Works with coordinated shutdown: the coordinator sets maintenance mode on all nodes before Phase 1, clears it after Phase 5
-
-**Files:**
-- `hive/daemon/jetstream.go` — new `SetMaintenanceMode()` / `ClearMaintenanceMode()` / `IsMaintenanceMode()`
-- `cmd/hive/cmd/admin.go` — new `maintenance` subcommand
-
-#### 3.2 Recovery status visibility
-
-Add admin endpoints and NATS subjects for recovery observability:
-
-- `GET /admin/recovery/status` — shows recovery state, assignments, peer health
-- `GET /admin/cluster/topology` — shows all nodes, their capabilities, and service health
-- `hive.admin.recovery.status` NATS subject — cluster-wide recovery state query
-
-**Files:**
-- `hive/daemon/daemon.go` — new cluster manager routes
-
-#### 3.3 Admin cluster status command
-
-Add a CLI command for operators to inspect the cluster topology:
-
-```bash
-$ ./bin/hive admin status
-Cluster: hive (epoch 3)
-
-NODE     SERVICES                                    STATUS   VMS  vCPU(free)  MEM(free)
-node1    nats,predastore,viperblock,daemon,awsgw,ui  healthy  4    8/16        12.0/32.0
-node2    nats,predastore,viperblock,daemon,awsgw     healthy  3    10/16       16.0/32.0
-node3    nats,predastore,viperblock,daemon            healthy  2    12/16       20.0/32.0
-node4    viperblock,daemon                            healthy  6    2/8         4.0/16.0
-```
-
-This reads from the `hive-cluster-state` KV bucket (heartbeats + service manifests) and fans out health checks via NATS.
-
-**Files:**
-- `cmd/hive/cmd/admin.go` — new `status` subcommand
-
-## Files Modified (Phase 0 + 1)
-
-| File | Changes | Phase |
-|------|---------|-------|
-| `hive/config/config.go` | Added `Services` field, `AllServices`, `HasService()`, `GetServices()`, updated `NodeHealthResponse` with `Services` and `ServiceHealth` | 0.1, 0.5 |
-| `hive/config/config_test.go` | 5 new tests for `HasService` and `GetServices` | 0.1 |
-| `hive/admin/admin.go` | Added `Services []string` to `ConfigSettings` | 0.3 |
-| `cmd/hive/cmd/templates/hive.toml` | Added `services` template rendering | 0.3 |
-| `cmd/hive/cmd/admin.go` | Added `--services` flag to init/join, wired to `ConfigSettings` | 0.3 |
-| `hive/formation/formation.go` | Added `Services` to `NodeInfo` | 0.2 |
-| `hive/formation/helpers.go` | Added `hasService()`, filtered `BuildClusterRoutes()` by nats, `BuildPredastoreNodes()` by predastore | 0.2 |
-| `hive/formation/helpers_test.go` | 7 new tests for capability-filtered routing and predastore node building | 0.2 |
-| `scripts/start-dev.sh` | Added `parse_services()`, `has_service()`, conditional service startup | 0.4 |
-| `scripts/stop-dev.sh` | Added `parse_services()`, `has_service()` (with name mapping), conditional stop | 0.4 |
-| `hive/daemon/daemon.go` | Updated health endpoint, `waitForClusterReady()`, `checkViperblockReady()`, `checkPredastoreReady()`, refactored `restoreInstances()` with semaphore, shutdown marker in `setupShutdown()`, service manifest write in `Start()` | 0.5, 1.1–1.4 |
-| `hive/daemon/jetstream.go` | `ClusterStateBucket`, `clusterKV` field, `InitClusterStateBucket()`, `WriteShutdownMarker()`, `ReadShutdownMarker()`, `DeleteShutdownMarker()`, `WriteServiceManifest()`, updated `UpdateReplicas()` | 0.5, 1.3, 1.4 |
-
-## Files to Modify (Phase 2 + 3)
-
-| File | Changes | Phase |
-|------|---------|-------|
-| `hive/daemon/jetstream.go` | Heartbeat, recovery lease, cluster shutdown KV, maintenance mode | 2.1–2.4, 3.1 |
-| `hive/daemon/recovery.go` | **New file** — recovery coordinator: leader election, capability-aware VM assignment, peer monitoring | 2.3 |
-| `hive/daemon/shutdown.go` | **New file** — coordinated shutdown coordinator: phase management, ACK aggregation, progress reporting, failover | 2.4 |
-| `hive/daemon/daemon.go` | Heartbeat/recovery integration, cluster shutdown handler, `shuttingDown` flag | 2.1–2.4 |
-| `hive/daemon/daemon_handlers.go` | NATS handlers for `hive.cluster.shutdown.*` topics | 2.4 |
-| `cmd/hive/cmd/admin.go` | `cluster shutdown`, `maintenance`, `status` subcommands | 2.4, 3.1, 3.3 |
-
-## Implementation Order
-
-Phase 0 is the prerequisite — all subsequent phases depend on nodes knowing their capabilities. Phase 1 fixes immediate restart bugs. Phases 2-3 add cross-node resilience and tooling.
-
-**Phase 0** — node capabilities (foundation) — **Complete**:
-1. [x] `Services` field in Config + helpers
-2. [x] Formation carries capabilities
-3. [x] Config template and CLI flags
-4. [x] Capability-aware start/stop scripts
-5. [x] Health endpoint includes capabilities + cluster state KV bucket + service manifest
-
-**Phase 1** — self-recovery (fixes immediate restart bugs) — **Complete**:
-6. [x] Cluster readiness gate
-7. [x] Recovery throttling (semaphore in `restoreInstances`)
-8. [x] Clean shutdown marker
-9. [x] Service manifest on startup
-
-**Phase 2** — cross-node coordination — **Planned**:
-10. Heartbeat publishing (with capabilities)
-11. Peer failure detection
-12. Leader-elected recovery coordinator (capability-aware placement)
-13. Coordinated cluster shutdown (`hive admin cluster shutdown`)
-
-**Phase 3** — operational tooling — **Planned**:
-14. Maintenance mode flag
-15. Recovery status endpoints
-16. Admin cluster status command
-
-## Testing
-
-### Phase 0 + 1 verification — Passed
-
-| Check | Result |
-|-------|--------|
-| `make test` (all unit tests) | Passed |
-| `make preflight` (format + vet + security + tests) | Passed |
-| `make test-docker-single` (single-node E2E) | Passed — all 9 phases |
-| `make test-docker-multi` (multi-node E2E, 3 nodes) | Passed — all phases including cluster health, cross-node operations |
-| Backward compatibility (no `services` field in config) | Verified — defaults to all services |
-
-**New unit tests added:**
-- `hive/config/config_test.go` — `HasService` (explicit, empty/default, unknown), `GetServices` (default, explicit)
-- `hive/formation/helpers_test.go` — `hasService` (empty=all, explicit), `BuildClusterRoutes` (filters by nats, empty=all, mixed), `BuildPredastoreNodes` (filters by predastore, empty=all)
-
-**Bug found during E2E**: `checkViperblockReady()` initially sent a NATS request to `ebs.{node}.health`, but viperblock does not subscribe to any health NATS topic. This caused the daemon to block in `waitForClusterReady()` for the full 2-minute timeout on every startup. Fixed by checking `natsConn.IsConnected()` instead.
-
-### Phase 2 + 3 tests (planned)
-
-#### Unit tests
-- Recovery coordinator: test leader election, capability-filtered VM assignment, epoch-based fencing
-- Heartbeat: test staleness detection thresholds
-- Coordinated shutdown: test phase transitions, ACK aggregation, timeout handling, coordinator failover
-- Shutdown protocol: test `cluster.shutdown` KV marker lifecycle (write, read, resume, clear)
-
-#### Integration tests (multi-node Docker)
-- Start 3-node cluster, launch 4 VMs, stop all, restart — verify all 4 VMs recover exactly once
-- Start 3+1 cluster (3 full + 1 compute-only), verify compute-only connects to remote NATS/Predastore
-- Kill one node (SIGKILL), verify surviving nodes recover its VMs after grace period
-- Rolling restart: stop/start nodes one at a time, verify zero duplicate launches
-- Coordinated shutdown: launch VMs across 3 nodes, run `hive admin cluster shutdown` from node2, verify phase ordering (AWSGW stops before VMs, VMs stop before viperblock, viperblock before predastore, NATS last)
-- Coordinated shutdown with `--force`: verify VMs are killed immediately, no grace period
-- Shutdown coordinator failure: start cluster shutdown from node1, kill node1 mid-drain, verify another node resumes coordination
-- Post-shutdown restart: after coordinated shutdown, restart all nodes, verify clean shutdown markers present and VMs restore correctly
-- No orphaned processes: after coordinated shutdown, verify zero remaining QEMU/nbdkit processes on all nodes
-
-#### Manual verification
-- Deploy to 3-node test cluster (10.1.3.170-172), run stop/start cycle, check logs for race conditions
-- Add a 4th compute-only node, verify it joins and accepts workloads
-- Run `hive admin cluster shutdown` from each of the 3 nodes in turn, verify it works from any node
-- Run `hive admin cluster shutdown --dry-run` and verify output matches expected phase plan
-
-## Future Work
-
-- **Coordinated cluster startup** (`hive admin cluster start`): Complement to `cluster shutdown`. Coordinator starts services across all nodes in dependency order (NATS → Predastore → Viperblock → Daemon → AWSGW → UI), with health checks between phases. Would replace per-node `start-dev.sh` for multi-node clusters.
-- **Persistent VM pinning**: Optional affinity rules to prefer restarting VMs on their original node
-- **Live migration**: Move running VMs between nodes without stopping them (requires QEMU live migration support). Would enable `cluster shutdown` to migrate VMs to surviving nodes instead of stopping them.
-- **Automatic scaling**: Detect sustained capacity pressure and suggest adding nodes
-- **Recovery metrics**: Prometheus-compatible metrics for recovery duration, failures, leader elections, shutdown phase timings
-- **Dynamic service reconfiguration**: Add/remove services from a running node without full restart
-- **Storage-only node health**: Extend recovery to detect failed storage nodes and reroute Viperblock/Predastore traffic
-- **Rolling cluster upgrade**: Coordinated restart that drains nodes one at a time, upgrades binaries, and restarts — maintaining cluster availability throughout

docs/development/improvements/multi-server.md

@@ -1,675 +0,0 @@
-# Multi-Server Cluster Formation
-
-## Problem
-
-The current multi-server configuration has a fundamental design flaw: cluster formation and cluster operation are entangled. The `/join` endpoint lives inside the Hive daemon, which requires NATS to be running. This creates a chicken-and-egg problem — nodes cannot join until services are running, but services cannot be correctly configured until all nodes have joined.
-
-### Current bugs
-
-1. **NATS config missing all cluster routes** — each node's `nats.conf` only contains the leader's route, not routes to all other nodes. NATS cannot form a full mesh cluster.
-2. **NATS cluster name hardcoded to `C1`** — should be configurable per-cluster.
-3. **Predastore "Distributed mode requires -node flag"** — NODE_ID detection fails in multi-node setups.
-4. **Predastore "no known peers" on joining nodes** — Raft starts with `servers=[]`, cannot form consensus.
-5. **Sequence dependency** — node 1 must run `./scripts/start-dev.sh` before node 2 can `hive admin join`, because the `/join` endpoint is inside the daemon.
-6. **Certs not installed to system CA store** — joining nodes receive the CA cert but don't install it, causing TLS failures.
-
-### Root cause
-
-All bugs trace back to one issue: configs are generated per-node during init/join with incomplete information (only the leader IP is known). The correct approach is to generate configs AFTER all nodes have registered, using complete cluster information.
-
-## Design
-
-Separate cluster **formation** (lightweight HTTP) from cluster **operation** (NATS, Predastore, etc).
-
-### Formation flow
-
-```
-                    ┌─────────────────────────────────┐
-                    │   hive admin init --nodes 3      │
-                    │   Starts formation HTTP server    │
-                    │   Registers self as node 1        │
-                    │   Waits for 2 more nodes...       │
-                    └──────────┬──────────────────────┘
-                               │ :4432
-              ┌────────────────┼────────────────┐
-              │                │                │
-    ┌─────────▼──────┐  ┌─────▼────────┐  ┌───▼──────────┐
-    │ hive admin join │  │  (waiting)   │  │  (waiting)   │
-    │ POST /join      │  │              │  │              │
-    │ polls /status   │  │              │  │              │
-    └────────────────┘  └──────────────┘  └──────────────┘
-              │                │                │
-              └────────────────┼────────────────┘
-                               │ All 3 nodes joined
-                    ┌──────────▼──────────────────────┐
-                    │   Formation complete              │
-                    │   Generate configs with ALL IPs   │
-                    │   Distribute CA, credentials      │
-                    │   All processes exit               │
-                    └──────────┬──────────────────────┘
-                               │
-              ┌────────────────┼────────────────┐
-              │                │                │
-    ┌─────────▼──────┐  ┌─────▼────────┐  ┌───▼──────────┐
-    │ start-dev.sh   │  │ start-dev.sh │  │ start-dev.sh │
-    │ Node 1         │  │ Node 2       │  │ Node 3       │
-    └────────────────┘  └──────────────┘  └──────────────┘
-```
-
-### Phase 1: Formation (init + join)
-
-**Node 1** starts a lightweight HTTP formation server (no NATS dependency):
-
-```sh
-./bin/hive admin init \
-  --node node1 \
-  --nodes 3 \
-  --bind 10.11.12.1 \
-  --cluster-bind 10.11.12.1 \
-  --port 4432 \
-  --hive-dir ~/hive/ \
-  --config-dir ~/hive/config/ \
-  --region ap-southeast-2 \
-  --az ap-southeast-2a
-```
-
-This generates credentials (AWS keys, NATS token, CA certs) and starts the formation HTTP server on `10.11.12.1:4432`. It registers itself as the first node and waits for 2 more.
-
-**Node 2 and Node 3** join while init is running:
-
-```sh
-./bin/hive admin join \
-  --node node2 \
-  --bind 10.11.12.2 \
-  --cluster-bind 10.11.12.2 \
-  --host 10.11.12.1:4432 \
-  --data-dir ~/hive/ \
-  --config-dir ~/hive/config/ \
-  --region ap-southeast-2 \
-  --az ap-southeast-2a
-```
-
-Each joining node POSTs to the formation server, then polls for completion.
-
-Once all 3 nodes register, the formation server:
-1. Computes NATS cluster routes from ALL node IPs (full mesh)
-2. Builds predastore node configs from ALL node IPs (with correct node IDs)
-3. Returns complete config package (credentials, CA, node list) to each polling node
-4. Each node generates its own configs locally with complete cluster data
-5. CA cert is installed to system trust store on each node
-6. All processes (init + joins) display cluster summary and exit
-
-**Key improvements over current design:**
-- `--predastore-nodes` is now **optional** — derived automatically from joined nodes
-- `--cluster-routes` on join is now **optional** — derived from formation data
-- No sequence dependency — init starts the formation server directly (no NATS needed)
-- All configs have complete cluster data (all IPs, all routes)
-
-### Phase 2: Service startup
-
-After formation completes, run on each node:
-
-```sh
-./scripts/start-dev.sh
-```
-
-Services start in order: NATS → Predastore → Viperblock → Daemon → AWS Gateway → UI.
-
-Since all configs already have the full cluster topology, NATS forms a mesh immediately and Predastore Raft finds all peers.
-
-## Network recommendations
-
-It is recommended that at least two network links are used by Hive:
-
-1. **Management link** — cluster sync, NATS, Predastore Raft, and daemon communication. This requires low latency (< 5ms) and dedicated bandwidth.
-2. **Data/VM link** — EC2 instances, VPC traffic (Geneve/Open vSwitch), and user-facing services. Isolated from management to prevent contention.
-
-This follows the same model as Proxmox VE, which recommends a physically separated NIC for cluster (corosync) traffic: "The network should not be used heavily by other members, as while corosync does not use much bandwidth it is sensitive to latency jitters."
-
-## Proxmox Cluster Manager comparison
-
-Reviewed [Proxmox Cluster Manager](https://pve.proxmox.com/wiki/Cluster_Manager) for best practices.
-
-**What we adopt from Proxmox:**
-- Formation-first workflow (`pvecm create` / `pvecm add` completes before services use the cluster)
-- Shared CA model (joining node's cert is replaced with one signed by cluster CA)
-- Dedicated cluster network recommendation (separate NIC for management traffic)
-- Hostname/IP finalized before formation (cannot change post-creation)
-
-**Why native Go over Corosync:**
-
-Corosync is a C group communication library providing reliable messaging, quorum, and membership. Hive already has equivalent capabilities:
-
-| Capability | Corosync | Hive |
-|---|---|---|
-| Message bus | Totem protocol | NATS |
-| State replication | pmxcfs (replicated filesystem) | NATS JetStream KV |
-| Consensus/quorum | Corosync votequorum | Predastore Raft |
-| Config distribution | pmxcfs | Formation server + hive.toml |
-
-Adding Corosync would be a redundant external C dependency providing overlapping functionality. The native Go implementation gives us:
-- Single binary deployment (no `corosync` daemon to install/manage)
-- Direct integration with NATS and Predastore
-- Full control over the join protocol
-- No CGo or external library dependency
-
-## Implementation
-
-### New package: `hive/formation/`
-
-Lightweight HTTP formation server using `net/http`. Zero dependency on NATS or any operational service.
-
-**`formation.go`** — core server:
-- `FormationServer` struct: tracks expected nodes, joined nodes, shared credentials, CA cert/key
-- `POST /formation/join` — register a node, return progress
-- `GET /formation/status` — poll completion; returns full config data when all nodes joined
-- `GET /formation/health` — liveness check
-- `WaitForCompletion()` — blocks until all nodes join or timeout (default 10 minutes)
-
-**`helpers.go`** — config builders:
-- `BuildClusterRoutes(nodes)` — returns all node cluster-bind IPs as `IP:4248` routes
-- `BuildPredastoreNodes(nodes)` — returns `[]PredastoreNodeConfig` with 1-based IDs
-
-### Changes to `cmd/hive/cmd/admin.go`
-
-**`runAdminInit`:**
-- `--nodes 1` (or no cluster flags): existing single-node behaviour, unchanged
-- `--nodes >= 2`: start formation server, register self, wait for joins, generate configs with complete data
-- `--predastore-nodes` becomes optional (derived from formation when not specified)
-- New flag: `--formation-timeout` (default "10m")
-
-**`runAdminJoin`:**
-- POST to `http://leader:port/formation/join` (not the daemon `/join`)
-- Poll `GET /formation/status` every 2s until complete
-- Extract full node list, CA, credentials from response
-- Generate all configs locally with complete cluster data
-- Install CA to system trust store
-
-### Changes to templates
-
-**`templates/nats.conf`:** Change `name: C1` to `name: {{ .ClusterName }}`
-
-**`hive/admin/admin.go`:** Add `ClusterName string` to `ConfigSettings`
-
-### Changes to `scripts/start-dev.sh`
-
-- Remove awk-based NODE_ID auto-detection (formation now writes correct node_id into hive.toml)
-- Keep `HIVE_PREDASTORE_NODE_ID` env var override for manual use
-
-### Daemon `/join` endpoint (no changes)
-
-The existing `/join` endpoint in `hive/daemon/daemon.go` stays as-is for future "hot-add node to running cluster" scenarios. Formation handles initial bootstrap; daemon handles runtime expansion.
-
-### Graceful cluster shutdown
-
-The current `stop-dev.sh` sends SIGTERM to each service sequentially with no cluster coordination. In a multi-node cluster this causes cascading failures — if the Raft leader is stopped first, remaining predastore nodes enter an election storm trying to contact dead peers, goroutines block, and the cluster hangs instead of shutting down cleanly.
-
-#### Two shutdown modes
-
-**1. Cluster-wide shutdown** (`hive admin shutdown`):
-
-Coordinated shutdown of the entire cluster. The initiating node broadcasts a shutdown intent, all nodes agree, and services stop in a safe order across the cluster.
-
-```sh
-# On any node — broadcasts to entire cluster
-./bin/hive admin shutdown
-```
-
-**2. Single-node shutdown** (`stop-dev.sh` or `hive admin stop`):
-
-Stops services on one node only. The node announces its departure so peers can adjust (e.g., Raft leader transfer) rather than treating it as a crash.
-
-```sh
-# On the node being stopped
-./scripts/stop-dev.sh
-```
-
-#### Cluster-wide shutdown flow
-
-```
-Operator runs: hive admin shutdown (on any node)
-              │
-              ▼
-┌──────────────────────────────────┐
-│ 1. Connect to local daemon       │
-│ 2. POST /cluster/shutdown        │
-│ 3. Daemon broadcasts             │
-│    hive.cluster.shutdown via NATS │
-└──────────────┬───────────────────┘
-               │
-    ┌──────────┼──────────┐
-    ▼          ▼          ▼
-  Node 1     Node 2     Node 3
-    │          │          │
-    │  All nodes enter "shutting_down" state
-    │  Stop accepting new work (API returns 503)
-    │          │          │
-    ▼          ▼          ▼
-  Phase 1: Stop VMs (system_powerdown, wait, write state to JetStream)
-    │          │          │
-    ▼          ▼          ▼
-  Phase 2: Stop daemon + gateway (unsubscribe NATS, close connections)
-    │          │          │
-    ▼          ▼          ▼
-  Phase 3: Stop storage (predastore Raft shutdown, viperblock)
-    │          │          │
-    ▼          ▼          ▼
-  Phase 4: Stop NATS (last — needed for coordination until this point)
-```
-
-#### Shutdown protocol
-
-1. **Initiation** — any node's daemon receives `POST /cluster/shutdown` (from `hive admin shutdown` CLI)
-2. **Broadcast** — daemon publishes `hive.cluster.shutdown` on NATS with initiator node ID and epoch
-3. **Acknowledgement** — each daemon responds on `hive.cluster.shutdown.ack` with its node ID. Initiator waits up to 10s for all known nodes to ACK (proceeds anyway if some nodes are unreachable)
-4. **Phase 1: VM drain** — all daemons stop accepting new `ec2.RunInstances` requests, send `system_powerdown` to running VMs, write instance state to JetStream, wait up to 60s for VMs to power off
-5. **Phase 2: Daemon stop** — daemons unsubscribe from all NATS topics, shut down the cluster HTTP server and AWS Gateway
-6. **Phase 3: Storage stop** — predastore Raft leader (if local) calls `raft.Shutdown()` first, then followers shut down Raft; viperblock stops. Because all nodes are shutting down simultaneously, there's no election storm — every node's Raft transport is closed before heartbeat timeouts fire
-7. **Phase 4: NATS stop** — NATS server is the last to stop (it was needed for the shutdown broadcast and ACKs)
-8. **Cleanup** — each node's `stop-dev.sh` waits for all service PIDs to exit, force-kills any stragglers after 120s
-
-#### Predastore Raft shutdown handling
-
-The core problem: when a Raft leader disappears without warning, followers detect missing heartbeats and start elections. With 2-of-3 nodes gone, elections can never succeed and goroutines spin indefinitely.
-
-**Solution for cluster-wide shutdown:**
-- All predastore nodes receive the shutdown broadcast via NATS (Phase 3)
-- Each node closes the Raft transport first (`transport.Close()`), which prevents any new RPCs
-- Then calls `raft.Shutdown()` which drains in-flight operations
-- Because all nodes close transport near-simultaneously, no node has time to start an election
-- The 5-second Raft shutdown timeout in predastore (`s3db/raft.go`) is sufficient since no elections are triggered
-
-**Solution for single-node shutdown:**
-- The departing node publishes `hive.cluster.node-leaving` with its node ID
-- If the departing node is the Raft leader, it calls `raft.LeadershipTransfer()` before shutting down — this hands leadership to a follower cleanly
-- If it's a follower, it calls `raft.Shutdown()` directly — the leader and remaining followers continue normally
-- Remaining nodes log the departure and continue operating (Raft maintains quorum with N-1 nodes if majority survives)
-
-#### Single-node stop flow
-
-When `stop-dev.sh` runs on one node in a multi-node cluster:
-
-1. Daemon receives SIGTERM
-2. Daemon publishes `hive.cluster.node-leaving` with its node ID on NATS
-3. If this node is the predastore Raft leader: call `raft.LeadershipTransfer()`, wait up to 5s
-4. Stop VMs: `system_powerdown` → wait → write state to JetStream
-5. Unsubscribe from NATS topics, shut down cluster HTTP server
-6. Stop predastore (Raft transport close, then `raft.Shutdown()`)
-7. Stop viperblock
-8. Close NATS connection
-9. Stop NATS server
-
-The remaining cluster nodes see the `node-leaving` message and:
-- Remove the node from active heartbeat tracking (no false alarms)
-- Predastore Raft has already transferred leadership if needed
-- AWS Gateway's next `hive.nodes.discover` broadcast gets fewer responses (expected)
-- No config changes — the node is still in `hive.toml` but simply offline
-
-#### Changes to `scripts/stop-dev.sh`
-
-- Before stopping services, check if this is a multi-node cluster (read `hive.toml` node count)
-- If multi-node: send `hive.cluster.node-leaving` via the daemon before stopping it
-- If daemon is unreachable (already crashed): proceed with direct SIGTERM as today (best effort)
-- Keep the existing service stop order but add a brief pause after daemon stop to allow the `node-leaving` broadcast to propagate
-
-#### New CLI command: `hive admin shutdown`
-
-```sh
-./bin/hive admin shutdown [--force] [--timeout 120s]
-```
-
-- Connects to local daemon's cluster HTTP endpoint
-- `POST /cluster/shutdown` triggers the coordinated shutdown flow
-- `--force` skips VM drain and ACK wait (immediate SIGTERM to all services)
-- `--timeout` controls max wait for VM powerdown (default 120s)
-- Prints progress: "Waiting for ACKs... 3/3", "Draining VMs... 2 remaining", "Stopping storage...", "Shutdown complete"
-
-## Files
-
-| File | Action |
-|---|---|
-| `hive/formation/formation.go` | New — formation HTTP server |
-| `hive/formation/helpers.go` | New — route and node config builders |
-| `hive/formation/formation_test.go` | New — tests |
-| `hive/admin/admin.go` | Modify — add ClusterName to ConfigSettings |
-| `cmd/hive/cmd/admin.go` | Modify — rewrite init multi-node path + join, add `shutdown` subcommand |
-| `cmd/hive/cmd/templates/nats.conf` | Modify — C1 → {{ .ClusterName }} |
-| `scripts/start-dev.sh` | Modify — simplify NODE_ID detection |
-| `scripts/stop-dev.sh` | Modify — check cluster mode, send `node-leaving` before stopping services |
-| `hive/daemon/daemon.go` | Modify — add shutdown broadcast handler, `node-leaving` publish on SIGTERM, Raft leader transfer coordination |
-
-## Testing
-
-### Unit tests
-
-- Formation server: all nodes join, duplicate rejection, timeout, status polling
-- Helper functions: route building, predastore node building
-- NATS template: renders cluster name and all routes correctly
-- Shutdown broadcast: all nodes ACK, timeout when node unreachable, phase ordering
-- Node-leaving: Raft leader transfer triggered when departing node is leader
-
-### Integration test
-
-Note: `--config-dir ~/$NODE/config/` needs to be specified for each node, so not to conflict when testing locally to simulate 3 nodes.
-
-1. `hive admin init --nodes 3 --bind 127.0.0.1 --node node1` in terminal 1
-2. `hive admin join --node node2 --host 127.0.0.1:4432 --bind 127.0.0.2` in terminal 2
-3. `hive admin join --node node3 --host 127.0.0.1:4432 --bind 127.0.0.3` in terminal 3
-4. Verify: all 3 exit with cluster summary
-5. Verify: each `nats.conf` has routes to ALL other nodes
-6. Verify: each `predastore.toml` has all 3 nodes
-7. Verify: each `hive.toml` has all 3 nodes in `[nodes.*]`
-8. Run `./scripts/start-dev.sh` on all 3, verify NATS mesh and Predastore Raft form correctly
-
-### Shutdown integration test
-
-1. Start 3-node cluster (from formation integration test above)
-2. Run `hive admin shutdown` on node1
-3. Verify: all 3 nodes receive shutdown broadcast (check logs for ACK)
-4. Verify: all predastore Raft nodes shut down cleanly — no election storms in logs
-5. Verify: all NATS servers stop last
-6. Verify: all service PIDs are gone on all 3 nodes
-
-### Single-node stop integration test
-
-1. Start 3-node cluster
-2. Run `./scripts/stop-dev.sh` on node3 (a Raft follower)
-3. Verify: node3 publishes `node-leaving`, services stop cleanly
-4. Verify: nodes 1 and 2 continue operating, Predastore Raft maintains quorum (2/3)
-5. Run `./scripts/stop-dev.sh` on node1 (if node1 is Raft leader)
-6. Verify: Raft leadership transfers to node2 before node1's predastore stops
-7. Verify: node2 continues operating as single-node (degraded but functional)
-
----
-
-## Stage 2: Runtime Node Expansion
-
-### Overview
-
-After the initial cluster is formed (Stage 1) and services are running, operators need the ability to add new nodes without stopping the cluster. This covers "day 2" operations — e.g., a 3-node cluster formed on Monday, with a 4th node added on Tuesday.
-
-The daemon takes over formation duties once the cluster is running. The existing `/join` endpoint in `hive/daemon/daemon.go` is replaced by the formation server operating in "Stage 2 mode" — same formation package, different context.
-
-### Key difference from Stage 1
-
-Stage 1 formation happens BEFORE any services start — it's a clean-room config generation step where all processes exit after writing configs. Stage 2 happens WHILE services are running, so each service must be notified of topology changes and reconfigure itself live.
-
-### Expansion flow
-
-```
-Existing running cluster (3 nodes)                  New node
-┌──────────┐  ┌──────────┐  ┌──────────┐
-│  Node 1   │  │  Node 2   │  │  Node 3   │
-│  daemon   │  │  daemon   │  │  daemon   │
-│  :4432    │  │  :4432    │  │  :4432    │
-└─────┬─────┘  └─────┬─────┘  └─────┬─────┘
-      │  NATS mesh    │              │
-      └───────┬───────┘              │
-              │                      │         ┌─────────────────┐
-              │                      │         │ hive admin join  │
-              │                      │         │   --host node1   │
-              │                      │         │   --node node4   │
-              │                      │         │   --capability   │
-              │                      │         │   nats,viperblock│
-              │                      │         │   ,hive,awsgw    │
-              │                      │         └────────┬────────┘
-              │                      │                  │
-              │                      │       POST /formation/join
-              │                      │                  │
-        ┌─────▼──────┐              │                  │
-        │ Node 1      │◄─────────────┼──────────────────┘
-        │ validates    │             │
-        │ updates toml │             │
-        │ returns creds│─────────────┘
-        │ broadcasts   │──► hive.cluster.config-update (NATS)
-        └─────┬───────┘
-              │
-              ▼
-   All existing nodes receive broadcast:
-   - Regenerate nats.conf → SIGHUP nats-server
-   - Update predastore.toml if needed → restart predastore
-   - Update hive.toml with new node entry
-              │
-              ▼
-   New node generates configs, starts services:
-   ./scripts/start-dev.sh  (on node4 only)
-```
-
-### Node capabilities
-
-Not all nodes need to run all services. The `--capability` flag controls which services a node provides:
-
-```sh
-./bin/hive admin join \
-  --node node4 \
-  --bind 10.11.12.4 \
-  --cluster-bind 10.11.12.4 \
-  --host 10.11.12.1:4432 \
-  --data-dir ~/hive/ \
-  --config-dir ~/hive/config/ \
-  --region ap-southeast-2 \
-  --az ap-southeast-2a \
-  --capability nats,viperblock,hive,awsgw
-```
-
-Available capabilities:
-
-| Capability | Service | Notes |
-|---|---|---|
-| `nats` | NATS server | Cluster messaging — required for all nodes |
-| `predastore` | Predastore storage node | Object shard storage via QUIC |
-| `predastoredb` | Predastore DB node | Raft-replicated metadata database |
-| `viperblock` | Viperblock | Local NVMe WAL for block storage — required if running VMs |
-| `hive` | Hive daemon | VM orchestration and scheduling |
-| `awsgw` | AWS Gateway | TLS API endpoint for AWS SDK compatibility |
-| `hive-ui` | Hive UI | Web management interface |
-
-Default: all capabilities enabled.
-
-**Example: compute-only node** (runs VMs, no object storage):
-```sh
---capability nats,viperblock,hive,awsgw
-```
-
-**Example: storage-only node** (object storage, no VMs):
-```sh
---capability nats,predastore
-```
-
-**Predastore DB scaling note:** The `predastoredb` capability controls whether a node participates in the Predastore Raft metadata cluster. For small clusters (3-5 nodes), every predastore node should also run predastoredb. For larger clusters (10+ nodes), Raft only needs 3-5 voters for consensus — additional predastore nodes store object shards via the hash ring but don't replicate the metadata database. This avoids replicating the KV store 25 times in a 25-node cluster.
-
-The `--capability` flag also applies to Stage 1 `hive admin join` for initial formation. Stage 1 `hive admin init` always has all capabilities (it's the bootstrap node).
-
-### Join protocol (Stage 2)
-
-The `hive admin join` command detects whether it's joining a formation (Stage 1) or a running cluster (Stage 2) based on the response from the leader.
-
-1. New node runs `hive admin join --host leader:4432 --node node4 --capability ...`
-2. New node POSTs to `http://leader:4432/formation/join` with node info + capabilities
-3. Leader daemon validates:
-   - Node name not already in cluster (409 Conflict if duplicate)
-   - Bind IP not already in use by another node
-   - Capabilities are valid strings
-4. Leader updates local `hive.toml` with new node entry, increments epoch
-5. Leader returns: CA cert/key, credentials (AWS access/secret key, NATS token), full node list including the new node
-6. New node writes CA cert/key, generates server cert signed by cluster CA
-7. New node generates local configs (nats.conf, hive.toml, predastore.toml, etc.) using complete cluster data, filtered by its capabilities
-8. New node installs CA cert to system trust store
-9. Leader broadcasts config update to all existing nodes via NATS (`hive.cluster.config-update`)
-10. Existing nodes receive update and reconfigure affected services (see below)
-11. `hive admin join` exits with cluster summary — operator then runs `start-dev.sh` on the new node
-
-### Service reconfiguration on existing nodes
-
-When a new node joins a running cluster, the leader broadcasts a `hive.cluster.config-update` message via NATS containing the updated node list, capabilities, and epoch. Each existing daemon handles the update:
-
-**NATS:**
-1. Regenerate `nats.conf` with new route added (only if new node has `nats` capability)
-2. Send `SIGHUP` to the `nats-server` process — NATS supports config reload via signal, including adding new cluster routes
-3. NATS discovers and connects to the new node once it starts
-4. If SIGHUP reload fails, log error and flag for manual restart
-
-**Predastore storage (`predastore` capability):**
-- If the new node has `predastore` capability: regenerate `predastore.toml` with new `[[nodes]]` entry and restart the predastore process (predastore does not support hot config reload)
-- The consistent hash ring updates on restart — new PUTs distribute across all nodes including the new one
-- Existing objects are unaffected (see "Hash ring rebalancing" below)
-
-**Predastore DB (`predastoredb` capability):**
-- If the new node has `predastoredb` capability: the leader calls `raft.AddVoter()` to add the new node to the Raft cluster
-- No restart needed on existing DB nodes — Raft handles membership changes natively via joint consensus
-- If the cluster already has sufficient DB voters (e.g., 5), the new node is not added as a voter even if it has the `predastoredb` capability — log a warning
-
-**Viperblock:**
-- No reconfiguration needed on existing nodes — viperblock is local to each node
-- New node starts its own viperblock instance independently
-
-**JetStream:**
-- Call `UpdateReplicas()` to match new cluster size (already implemented in `jetstream.go`)
-- Capped at the NATS cluster size (JetStream replicas cannot exceed the number of NATS nodes)
-
-**AWS Gateway:**
-- No reconfiguration needed — uses dynamic NATS node discovery (`hive.nodes.discover` fan-out with 500ms timeout)
-- New node's daemon responds to discovery automatically once it's connected to NATS
-
-**Hive UI:**
-- No reconfiguration needed — UI queries the AWS Gateway, which discovers nodes dynamically
-
-### Predastore hash ring rebalancing
-
-When a new predastore storage node joins, the consistent hash ring (xxhash + `buraksezer/consistent`) changes. Shard placement for new writes shifts, but existing data is safe.
-
-**Immediate behaviour (no rebalancing — Stage 2 scope):**
-- Existing objects stay on their original nodes. Shard locations are recorded per-object in `ObjectToShardNodes` in the Raft-replicated s3db — GETs look up this metadata, not the ring
-- New objects (PUTs) use the updated ring, distributing across all nodes including the new one
-- DELETEs use stored metadata to find and remove shards from the correct nodes
-- The cluster is fully functional without rebalancing — the only downside is uneven storage distribution until a rebalance runs
-
-**Why this is safe:** The hash ring is only used at write time to decide where to place shards. Once placed, the mapping `object_hash → [node_ids]` is persisted in s3db. All subsequent reads and deletes use this stored mapping, not the ring. Adding a node to the ring does not invalidate any existing mappings.
-
-**Background rebalancing (future — not in Stage 2 scope):**
-A future `hive admin rebalance` command would:
-1. Scan all objects in s3db
-2. For each object, compute new ring placement with current topology
-3. If placement changed, migrate shards from old nodes to new node via QUIC
-4. Update s3db metadata with new shard locations (atomic Raft write)
-5. Delete old shard copies after successful migration
-
-This is a significant feature (data migration, progress tracking, failure recovery, bandwidth throttling) and is deferred to a dedicated implementation.
-
-### Config sync protocol
-
-All nodes must converge on the same cluster state. The protocol uses epoch-based versioning:
-
-1. **Epoch counter** — every config change (node join, capability change) increments the epoch on the leader
-2. **Broadcast on change** — leader publishes `hive.cluster.config-update` via NATS with full node list + epoch
-3. **Periodic heartbeat** — each daemon periodically publishes its epoch on `hive.cluster.heartbeat`; if a node detects it has a stale epoch (e.g., it was temporarily disconnected when the broadcast happened), it requests the latest config from the leader via `hive.cluster.config-request`
-4. **Config hash** — SHA256 of shared cluster data (already implemented via `computeConfigHash()`) included in `/health` endpoint response; monitoring can detect inconsistencies
-5. **Idempotent updates** — if a node receives a config update with the same or older epoch, it ignores it
-
-### New node startup sequence
-
-After `hive admin join` completes on the new node:
-
-```sh
-./scripts/start-dev.sh  # On the new node only
-```
-
-`start-dev.sh` reads the node's capabilities from `hive.toml` and starts only matching services:
-
-1. **NATS** (if `nats` capability) — joins existing mesh via routes in nats.conf
-2. **Predastore** (if `predastore` capability) — starts storage node; existing nodes have already been restarted with updated config
-3. **Predastore DB** (if `predastoredb` capability) — starts DB node; leader has already called `AddVoter()` so Raft accepts it
-4. **Viperblock** (if `viperblock` capability) — starts local instance
-5. **Daemon** (if `hive` capability) — connects to NATS, subscribes to topics, announces via heartbeat
-6. **AWS Gateway** (if `awsgw` capability) — starts and discovers nodes via NATS fan-out
-7. **UI** (if `hive-ui` capability) — starts web interface
-
-### Stage 2 implementation
-
-**Changes to `hive/formation/formation.go`:**
-- Add `Stage2Join()` handler — validates node, updates config, returns credentials + full cluster data
-- Add `BroadcastConfigUpdate(natsConn, clusterConfig)` — publishes update to `hive.cluster.config-update` NATS topic
-- `FormationServer` gains `mode` field: `ModeFormation` (Stage 1) vs `ModeRunning` (Stage 2)
-- In Stage 2 mode, the server has access to the NATS connection for broadcasting
-
-**Changes to `hive/daemon/daemon.go`:**
-- On startup, start formation server in Stage 2 mode (replaces the existing `/join` handler)
-- Remove old `/join` endpoint handler (lines 814-931) — replaced by formation server
-- Add `handleConfigUpdate()` — NATS subscription on `hive.cluster.config-update`, applies received config
-- Add `reloadNATS()` — regenerates nats.conf from current cluster config, sends SIGHUP to nats-server PID
-- Add `reloadPredastore()` — regenerates predastore.toml, restarts predastore process
-- Add `handleClusterHeartbeat()` — periodic epoch broadcast + stale detection
-
-**Changes to `hive/config/config.go`:**
-- Add `Capabilities []string` to per-node `Config` struct
-- Add `Capabilities []string` to `NodeJoinRequest`
-- Add `HasCapability(cap string) bool` method on `Config`
-
-**Changes to `cmd/hive/cmd/admin.go`:**
-- `runAdminJoin`: new `--capability` flag (comma-separated string, default all capabilities)
-- Join detects Stage 1 vs Stage 2: if formation server responds with `mode: "formation"`, poll for completion (Stage 1); if `mode: "running"`, configs are returned immediately (Stage 2)
-
-**Changes to `hive/formation/helpers.go`:**
-- `BuildClusterRoutes(nodes)` → `BuildClusterRoutes(nodes, capabilityFilter)` — only include nodes with `nats` capability
-- `BuildPredastoreNodes(nodes)` → filter by `predastore` capability
-- New: `BuildPredastoreDBNodes(nodes)` — filter by `predastoredb` capability
-- New: `FilterNodesByCapability(nodes map[string]Config, cap string) []Config`
-
-**Changes to config templates:**
-- `nats.conf` — routes only include nodes with `nats` capability
-- `predastore-multinode.toml` — `[[nodes]]` only include `predastore` capable nodes; `[[db]]` only include `predastoredb` capable nodes
-
-**Changes to `scripts/start-dev.sh`:**
-- Read capabilities from `hive.toml` for the local node
-- Only start services matching the node's capabilities
-- Skip predastore/predastoredb/viperblock/awsgw/hive-ui if not in capability list
-
-### Stage 2 files
-
-| File | Action |
-|---|---|
-| `hive/formation/formation.go` | Modify — add Stage 2 mode, Stage2Join handler, broadcast |
-| `hive/formation/helpers.go` | Modify — capability-aware filtering for all config builders |
-| `hive/formation/formation_test.go` | Modify — add Stage 2 tests |
-| `hive/daemon/daemon.go` | Modify — replace `/join` with formation Stage 2, add config update + service reload handlers |
-| `hive/config/config.go` | Modify — add Capabilities field to Config and NodeJoinRequest |
-| `cmd/hive/cmd/admin.go` | Modify — add `--capability` flag, Stage 1 vs Stage 2 detection logic |
-| `cmd/hive/cmd/templates/nats.conf` | Modify — capability-filtered routes |
-| `cmd/hive/cmd/templates/predastore-multinode.toml` | Modify — capability-filtered `[[nodes]]` and `[[db]]` sections |
-| `scripts/start-dev.sh` | Modify — capability-aware service startup |
-| `hive/daemon/jetstream.go` | No changes (UpdateReplicas already handles dynamic scaling) |
-
-### Stage 2 testing
-
-**Unit tests:**
-- Stage 2 join: validates capabilities, rejects duplicate node names and IPs, increments epoch
-- Capability filtering: correct nodes appear in NATS routes and predastore config
-- Config broadcast: serialization/deserialization of `hive.cluster.config-update` messages
-- Service reload: NATS config regeneration produces valid config, predastore config includes correct node subsets
-- Epoch-based sync: stale epoch detection triggers config request
-
-**Integration test:**
-1. Form 3-node cluster via Stage 1 (all capabilities)
-2. Start all 3 nodes, verify NATS mesh + Predastore Raft healthy
-3. Run `hive admin join --node node4 --host 127.0.0.1:4432 --bind 127.0.0.4 --capability nats,viperblock,hive,awsgw`
-4. Verify: node4 added to all nodes' `hive.toml` with correct capabilities
-5. Verify: all `nats.conf` files updated with node4 route
-6. Verify: `predastore.toml` NOT updated (node4 has no `predastore` capability)
-7. Start node4 with `start-dev.sh`, verify it joins NATS mesh
-8. Verify: AWS Gateway discovers node4 via `hive.nodes.discover`
-9. Verify: predastore cluster unchanged (still 3 nodes)
-10. Run `hive admin join --node node5 --host 127.0.0.1:4432 --bind 127.0.0.5 --capability nats,predastore`
-11. Verify: `predastore.toml` updated on all nodes with node5 as storage node
-12. Verify: predastore `[[db]]` section unchanged (node5 has no `predastoredb` capability)
-13. Start node5, write new object via S3 API, verify shards distributed across 4 predastore nodes
-
-## Remote testing
-
-Integration tests above use localhost IPs for local simulation. For real multi-server testing, a 3-node cluster environment is available — see `docs/INTERNAL.md` for SSH access, deployment workflow (git diff → scp → git apply → make build), and cluster init/join commands with real IPs.
-
-The remote test flow:
-1. Deploy code patches to all 3 nodes (INTERNAL.md: "Deploying code changes")
-2. Fresh install or patch-only depending on scope (INTERNAL.md: "Testing workflows")
-3. Run `hive admin init` on node 1, `hive admin join` on nodes 2 and 3 (INTERNAL.md: "Cluster init and join")
-4. Start services on all nodes, verify NATS mesh, Predastore Raft, and daemon health
-5. Run AWS CLI smoke test against the cluster endpoint

docs/development/improvements/nbd-performance.md

@@ -1,295 +0,0 @@
-# NBD / QEMU Performance
-
-## Summary
-
-Hive VMs using NBD-backed boot volumes showed ~38x lower IOPS and ~35-40x higher latency compared to a Proxmox VM on the same host. The root cause was not the NBD transport itself, but missing QEMU tuning flags: no iothread, no cache policy, no multiqueue. Proxmox applies all of these by default; Hive applied none.
-
-**Status: Implemented.** All boot volumes and hot-attached data volumes now get dedicated iothreads, `cache=none`, and multiqueue. Auxiliary volumes (cloud-init, EFI) are excluded — they are small, rarely read, and don't benefit from these flags. The viperblock LRU cache was also doubled from 64MB to 128MB for main volumes.
-
-## What Changed
-
-### QEMU flags (boot volumes)
-
-`hive/daemon/daemon.go` — `StartInstance()` boot drive setup:
-
-- Boot volume drive gets `cache=none`
-- A dedicated iothread (`ioth-os`) is created
-- virtio-blk-pci device gets `iothread=ioth-os,num-queues=<vCPU count>`
-
-Before:
-```
--drive file=nbd:unix:/path.sock,format=raw,if=none,media=disk,id=os
--device virtio-blk-pci,drive=os,bootindex=1
-```
-
-After:
-```
--object iothread,id=ioth-os
--drive file=nbd:unix:/path.sock,format=raw,if=none,media=disk,id=os,cache=none
--device virtio-blk-pci,drive=os,iothread=ioth-os,num-queues=2,bootindex=1
-```
-
-### QEMU flags (hot-attached data volumes)
-
-`hive/daemon/daemon_handlers.go` — `handleAttachVolume()`:
-
-Hot-attached volumes (e.g. a 1TB data drive attached to a running instance) also get a per-volume iothread via QMP:
-
-1. `object-add` creates `ioth-<volumeID>`
-2. `blockdev-add` connects the NBD backend
-3. `device_add` creates `virtio-blk-pci` with `iothread=ioth-<volumeID>`
-
-On detach (`handleDetachVolume()`), cleanup is:
-
-1. `device_del` removes the guest device
-2. `blockdev-del` removes the block node
-3. `object-del` removes the iothread (best-effort, non-fatal if it fails)
-
-### Excluded: cloud-init and EFI volumes
-
-These auxiliary volumes are intentionally excluded from performance tuning:
-
-- **Cloud-init** (`v.CloudInit`): Mounted as `if=virtio,media=cdrom`. Read once at boot, tiny payload. No iothread, no cache tuning.
-- **EFI** (`v.EFI`): Currently skipped entirely (`continue` in the volume loop). No QEMU drive emitted.
-- **Viperblock cache**: Volumes with names ending in `-cloudinit` or `-efi` get `cache_size=0` in both the viperblock in-process LRU and the nbdkit plugin. Main volumes get 128MB (32768 blocks at 4096B block size).
-
-### Viperblock LRU cache doubled
-
-`hive/services/viperblockd/viperblockd.go`:
-
-- `defaultCache` changed from `(64 * 1024 * 1024) / DefaultBlockSize` → `(128 * 1024 * 1024) / DefaultBlockSize`
-- This is 16384 → 32768 blocks (at `DefaultBlockSize = 4096`)
-- Applies to both the in-process viperblock golang-lru cache and the `cache_size` parameter passed to the nbdkit plugin
-- Cloud-init/EFI volumes still get `cache_size=0`
-
-### Struct changes
-
-`hive/vm/vm.go`:
-
-```go
-type Drive struct {
-    File   string `json:"file"`
-    Format string `json:"format"`
-    If     string `json:"if"`
-    Media  string `json:"media"`
-    ID     string `json:"id"`
-    Cache  string `json:"cache,omitempty"`  // NEW — e.g. "none"
-}
-
-type IOThread struct {  // NEW
-    ID string `json:"id"`
-}
-
-type Config struct {
-    // ... existing fields ...
-    Drives    []Drive    `json:"drives"`
-    IOThreads []IOThread `json:"io_threads,omitempty"`  // NEW
-    // ...
-}
-```
-
-`Config.Execute()` emits `-object iothread,id=...` before `-drive` args, and appends `cache=<value>` to drive option strings when set.
-
-## Benchmark Comparison (pre-fix)
-
-Test: `fio --rw=randrw --rwmixread=70 --bs=4k --numjobs=4 --iodepth=32 --ioengine=libaio --direct=1`
-
-| Metric | Proxmox VM (direct) | Hive VM (nested, NBD) | Delta |
-|---|---|---|---|
-| Read IOPS | 85,600 | 2,243 | 38x lower |
-| Write IOPS | 36,800 | 970 | 38x lower |
-| Read BW | 334 MiB/s | 8.8 MiB/s | 38x lower |
-| Write BW | 144 MiB/s | 3.8 MiB/s | 38x lower |
-| Read avg latency | 1.0 ms | 41.6 ms | 40x higher |
-| Write avg latency | 1.1 ms | 35.5 ms | 32x higher |
-| Read p99 latency | ~2.9 ms | ~75 ms | 26x higher |
-| Write p99 latency | ~3.0 ms | ~68 ms | 23x higher |
-| Disk util | 97% | 79% | Guest not saturating device |
-
-Note: Proxmox test used 512 MiB/job (2 GiB total), Hive test used 128 MiB/job (512 MiB total). This affects cache behavior but does not explain a 38x gap.
-
-The 79% utilization with 35-41 ms average latency in the Hive VM indicates requests were queueing behind a serialization point, not that the physical disk was saturated.
-
-## Root Cause Analysis
-
-**I/O path comparison:**
-
-Proxmox VM:
-```
-fio → virtio-blk (iothread, multiqueue) → QEMU block layer (cache=none, aio=io_uring) → qcow2 file → host FS → physical disk
-```
-
-Hive VM (nested inside Proxmox, before fix):
-```
-fio → virtio-blk (no iothread, single queue) → QEMU block layer (defaults) → NBD client → Unix socket → nbdkit → viperblock → [outer VM disk path]
-```
-
-**What Proxmox set that Hive did not (now fixed):**
-
-| Flag | Proxmox | Hive (before) | Hive (after) |
-|---|---|---|---|
-| `iothread` | Yes (`iothread-virtio0`) | No | Yes (`ioth-os`, `ioth-<volID>`) |
-| `cache` | `none` | Default (writeback) | `none` |
-| `num-queues` | Implicit (1, but with iothread) | 1, no iothread | vCPU count (boot), not set (hot-attach) |
-| `aio` | `io_uring` | Default (threads) | Default (threads) — correct for NBD |
-
-The iothread was the highest-impact missing flag. Without it, all block I/O processing ran in the main QEMU event loop alongside vCPU emulation.
-
-**Remaining factors (not addressed by QEMU flags):**
-
-1. **QEMU NBD client has a 16 in-flight request limit** — The NBD protocol allows negotiating queue depth, but QEMU's built-in NBD client caps concurrent requests at 16. With `numjobs=4 --iodepth=32` (128 potential in-flight), requests queue inside QEMU before they even reach the socket. This is a hard protocol-level limit that cannot be changed via flags.
-
-2. **Unix socket buffer size** — The default socket buffer (`net.core.wmem_default`, typically 212992 bytes) can bottleneck when many small 4K NBD request/response frames are in flight. Tuning `SO_SNDBUF`/`SO_RCVBUF` on the NBD socket or increasing system defaults may help throughput.
-
-3. **Double virtualization overhead** — The benchmark Hive VM ran inside a Proxmox VM. Every I/O traverses two virtualization boundaries. This is a test environment artifact and won't apply on bare-metal deployments.
-
-## QEMU Best Practices for NBD Backends
-
-### iothread
-
-An iothread is a dedicated QEMU thread for processing virtio-blk requests. Without one, I/O completion runs in the main loop and competes with vCPU scheduling. For any workload beyond trivial single-threaded sequential I/O, an iothread is required.
-
-Each volume should have its own iothread. Boot volumes get `ioth-os`, hot-attached volumes get `ioth-<volumeID>`.
-
-### cache=none
-
-Sets QEMU's block layer to skip host page caching. For file backends, this means `O_DIRECT`. For NBD socket backends, there is no file to `O_DIRECT` on — but the flag still controls QEMU's internal write-back cache and flush forwarding behavior. `cache=none` ensures `FLUSH` commands from the guest are forwarded through NBD to the server, which is correct for data integrity.
-
-### aio= (not meaningful for NBD)
-
-The `aio=` option selects the host async I/O engine for file operations:
-- `aio=threads` — POSIX AIO via thread pool (default)
-- `aio=io_uring` — Linux io_uring submission queue
-- `aio=native` — Linux native AIO (`io_submit`)
-
-These engines optimize `preadv`/`pwritev` syscalls on file descriptors. QEMU's NBD client does not use these syscalls — it uses coroutine-based socket I/O through `QIOChannel`. Setting `aio=io_uring` on an NBD drive changes nothing in the I/O path. The default (`aio=threads`) is correct and we intentionally omit this flag.
-
-### virtio-blk multiqueue (num-queues)
-
-By default, virtio-blk exposes a single virtqueue. With multiple vCPUs, all cores contend on that single queue's notification and completion path. Setting `num-queues=N` (where N = vCPU count) allows each vCPU to submit I/O to its own queue, eliminating cross-CPU contention.
-
-Requires guest kernel support (Linux 3.13+, always available on modern distros).
-
-Currently applied to boot volumes only (where the vCPU count is known from `Config.CPUCount`). Hot-attached volumes do not set `num-queues` because the device_add QMP path doesn't easily support it — the iothread alone provides the primary benefit.
-
-### NBD protocol limits
-
-QEMU's NBD client implementation enforces a maximum of 16 concurrent in-flight requests per connection. This is defined in the QEMU source and cannot be changed via command-line flags. At high queue depths, this becomes a bottleneck regardless of other tuning.
-
-Potential future mitigation: multiple NBD connections to the same export (requires changes to both QEMU invocation and nbdkit/viperblock).
-
-## Viperblock Cache Architecture
-
-The viperblock cache operates at two layers:
-
-1. **In-process golang-lru** — Viperblock's Go process maintains a hashicorp/golang-lru cache of recently accessed blocks. This avoids round-trips to the S3-compatible backend (predastore) for hot blocks.
-
-2. **nbdkit plugin cache** — The nbdkit plugin (which runs as a separate process) has its own `cache_size` parameter that controls a similar LRU cache within the C plugin.
-
-Both caches are sized identically via `defaultCache`:
-
-| Volume type | Cache size | Blocks (4KB each) |
-|---|---|---|
-| Main (boot, data) | 128 MB | 32,768 |
-| Cloud-init (`-cloudinit` suffix) | 0 (disabled) | 0 |
-| EFI (`-efi` suffix) | 0 (disabled) | 0 |
-
-Cloud-init and EFI volumes are small (typically < 1MB), read once at boot, and would waste memory with a cache enabled.
-
-## Reference: QEMU Commands
-
-### Proxmox outer VM (relevant flags only)
-
-```sh
-/usr/bin/kvm \
-  -object iothread,id=iothread-virtio0 \
-  -drive file=/var/lib/vz/images/146/vm-146-disk-0.qcow2,if=none,id=drive-virtio0,cache=none,aio=io_uring,discard=on,format=qcow2,detect-zeroes=unmap \
-  -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,iothread=iothread-virtio0,bootindex=100 \
-  -smp 4,sockets=1,cores=4,maxcpus=4 \
-  -m 8192 \
-  -cpu host
-```
-
-### Hive inner VM (before fix)
-
-```sh
-qemu-system-x86_64 \
-  -enable-kvm -cpu host -smp 2 -m 4096 \
-  -drive file=nbd:unix:/run/user/1000/nbd-vol-xxx.sock,format=raw,if=none,media=disk,id=os \
-  -device virtio-blk-pci,drive=os,bootindex=1 \
-  -M q35
-```
-
-### Hive inner VM (after fix)
-
-```sh
-qemu-system-x86_64 \
-  -enable-kvm -cpu host -smp 2 -m 4096 \
-  -object iothread,id=ioth-os \
-  -drive file=nbd:unix:/run/user/1000/nbd-vol-xxx.sock,format=raw,if=none,media=disk,id=os,cache=none \
-  -device virtio-blk-pci,drive=os,iothread=ioth-os,num-queues=2,bootindex=1 \
-  -M q35
-```
-
-### Hot-attach QMP sequence (after fix)
-
-```json
-{"execute": "object-add", "arguments": {"qom-type": "iothread", "id": "ioth-vol-abc123"}}
-{"execute": "blockdev-add", "arguments": {"node-name": "nbd-vol-abc123", "driver": "nbd", "server": {"type": "unix", "path": "/run/user/1000/nbd-vol-abc123.sock"}, "export": "", "read-only": false}}
-{"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "vdisk-vol-abc123", "drive": "nbd-vol-abc123", "iothread": "ioth-vol-abc123", "bus": "hotplug1"}}
-```
-
-### Hot-detach QMP sequence (after fix)
-
-```json
-{"execute": "device_del", "arguments": {"id": "vdisk-vol-abc123"}}
-{"execute": "blockdev-del", "arguments": {"node-name": "nbd-vol-abc123"}}
-{"execute": "object-del", "arguments": {"id": "ioth-vol-abc123"}}
-```
-
-## Files Modified
-
-| File | Change |
-|---|---|
-| `hive/vm/vm.go` | Added `Cache` field to `Drive`, added `IOThread` struct and `IOThreads` to `Config`, updated `Execute()` to emit `-object iothread` and `cache=` |
-| `hive/daemon/daemon.go` | Boot drive setup: `cache=none`, iothread, `num-queues=<vCPU count>` |
-| `hive/daemon/daemon_handlers.go` | Hot-attach: QMP `object-add` iothread + `iothread` in `device_add`. Hot-detach: QMP `object-del` cleanup |
-| `hive/daemon/daemon_test.go` | Updated detach test assertions to expect `object-del` in QMP command sequence |
-| `hive/vm/vm_test.go` | Added `TestExecute_IOThreadAndCache`, `TestExecute_NoCacheWhenEmpty`, `TestExecute_MultipleIOThreads` |
-| `hive/services/viperblockd/viperblockd.go` | Doubled `defaultCache` from 64MB (16384 blocks) to 128MB (32768 blocks) |
-
-## Diagnostic Tests
-
-Run inside the Hive guest VM to measure the impact of changes.
-
-**Baseline per-I/O overhead** (isolates single-request latency):
-```sh
-fio --name=lat1 --directory=$HOME/bench --rw=randrw --rwmixread=70 \
-    --bs=4k --size=512M --numjobs=1 --iodepth=1 \
-    --ioengine=libaio --direct=1 --group_reporting
-```
-
-**Queue depth scaling** (detects serialization):
-```sh
-fio --name=qdtest --directory=$HOME/bench --rw=randrw --rwmixread=70 \
-    --bs=4k --size=512M --numjobs=1 --iodepth=32 \
-    --ioengine=libaio --direct=1 --group_reporting
-```
-
-If `iodepth=1` latency is already in the tens of ms, the NBD/viperblock backend is fundamentally slow per-I/O. If `iodepth=1` is reasonable but `iodepth=32` explodes, there is a serialization or flush amplification problem in the stack.
-
-**Before/after comparison** (run with target QEMU flags applied):
-```sh
-fio --name=randrw_70_30 --directory=$HOME/bench --rw=randrw --rwmixread=70 \
-    --bs=4k --size=512M --numjobs=4 --iodepth=32 \
-    --ioengine=libaio --direct=1 --group_reporting
-```
-
-## Future Work
-
-- **Post-fix benchmarks**: Re-run fio with the new QEMU flags and 128MB cache to quantify improvement
-- **Bare-metal benchmarks**: The pre-fix numbers were from a nested Proxmox environment (double virtualization). Bare-metal numbers will show the true NBD overhead
-- **NBD 16-request limit**: Investigate multi-connection NBD or QEMU patches to raise the in-flight cap
-- **Socket buffer tuning**: Test `SO_SNDBUF`/`SO_RCVBUF` sizing on NBD Unix sockets
-- **nbdkit threading**: nbdkit `--threads` parameter (defaults to 16 for parallel plugins) — verify this matches or exceeds the NBD in-flight limit
-- **Dynamic cache sizing**: Scale viperblock cache based on available system memory rather than a fixed 128MB

hive/config/config.go

@@ -303,7 +303,8 @@ func LoadConfig(configPath string) (*ClusterConfig, error) {
 			if err := viper.ReadInConfig(); err != nil {
 				return nil, fmt.Errorf("error reading config file: %w", err)
 			}
-			fmt.Fprintf(os.Stderr, "Using config file: %s\n", viper.ConfigFileUsed())
+			//fmt.Fprintf(os.Stderr, "Using config file: %s\n", viper.ConfigFileUsed())
+
 		} else {
 			fmt.Fprintf(os.Stderr, "Config file not found: %s, using environment variables and defaults\n", configPath)
 		}

hive/daemon/daemon.go

@@ -246,10 +246,12 @@ func generateInstanceTypes(family, arch string) map[string]*ec2.InstanceTypeInfo
 		{"2xlarge", 8, 32.0},
 	}
 
-	// Determine the burstable family based on architecture
+	// Determine the burstable family based on architecture and CPU vendor
 	burstableFamily := "t3"
 	if arch == "arm64" {
 		burstableFamily = "t4g"
+	} else if strings.HasSuffix(family, "a") {
+		burstableFamily = "t3a"
 	}
 
 	// Build list of families to generate: CPU-specific + burstable (if different)
@@ -328,6 +330,8 @@ func NewResourceManager() *ResourceManager {
 	burstableFamily := "t3"
 	if runtime.GOARCH == "arm64" {
 		burstableFamily = "t4g"
+	} else if strings.HasSuffix(instanceFamily, "a") {
+		burstableFamily = "t3a"
 	}
 	slog.Info("System resources detected",
 		"vCPUs", numCPU, "memGB", totalMemGB, "cpu", cpuModel,
@@ -592,8 +596,12 @@ func (d *Daemon) Start() error {
 	d.instanceService = handlers_ec2_instance.NewInstanceServiceImpl(d.config, d.resourceMgr.instanceTypes, d.natsConn, &d.Instances, store)
 	d.keyService = handlers_ec2_key.NewKeyServiceImpl(d.config)
 	d.imageService = handlers_ec2_image.NewImageServiceImpl(d.config, d.natsConn)
-	d.volumeService = handlers_ec2_volume.NewVolumeServiceImpl(d.config, d.natsConn)
-	d.snapshotService = handlers_ec2_snapshot.NewSnapshotServiceImpl(d.config, d.natsConn)
+	snapSvc, snapshotKV, err := handlers_ec2_snapshot.NewSnapshotServiceImplWithNATS(d.config, d.natsConn)
+	if err != nil {
+		return fmt.Errorf("failed to initialize snapshot service with NATS KV: %w", err)
+	}
+	d.snapshotService = snapSvc
+	d.volumeService = handlers_ec2_volume.NewVolumeServiceImpl(d.config, d.natsConn, snapshotKV)
 	d.tagsService = handlers_ec2_tags.NewTagsServiceImpl(d.config)
 
 	if eigwSvc, err := handlers_ec2_eigw.NewEgressOnlyIGWServiceImplWithNATS(d.config, d.natsConn); err != nil {
@@ -609,6 +617,11 @@ func (d *Daemon) Start() error {
 	}
 	d.accountService = accountSvc
 
+	// Protect daemon from OOM killer (prefer killing QEMU VMs instead)
+	if err := utils.SetOOMScore(os.Getpid(), -500); err != nil {
+		slog.Warn("Failed to set daemon OOM score", "err", err)
+	}
+
 	d.waitForClusterReady()
 	d.restoreInstances()
 
@@ -741,7 +754,7 @@ func (d *Daemon) checkPredastoreReady() bool {
 	if err != nil {
 		return false
 	}
-	conn.Close()
+	_ = conn.Close()
 	return true
 }
 
@@ -1732,6 +1745,7 @@ func (d *Daemon) StartInstance(instance *vm.VM) error {
 	processChan := make(chan int, 1)
 	exitChan := make(chan int, 1)
 	ptsChan := make(chan int, 1)
+	startupConfirmed := make(chan bool, 1)
 
 	go func() {
 		cmd, err := instance.Config.Execute()
@@ -1766,6 +1780,11 @@ func (d *Daemon) StartInstance(instance *vm.VM) error {
 
 		slog.Info("VM started successfully", "pid", cmd.Process.Pid)
 
+		// Set OOM score for QEMU process (prefer killing VMs over system services)
+		if err := utils.SetOOMScore(cmd.Process.Pid, 500); err != nil {
+			slog.Warn("Failed to set QEMU OOM score", "pid", cmd.Process.Pid, "err", err)
+		}
+
 		// TODO: Consider workaround using QMP
 		//  (QEMU) query-chardev
 		// {"return": [{"frontend-open": true, "filename": "vc", "label": "parallel0"}, {"frontend-open": true, "filename": "unix:/run/user/1000/qmp-i-150340b52b20c0b43.sock,server=on", "label": "compat_monitor0"}, {"frontend-open": true, "filename": "pty:/dev/pts/9", "label": "serial0"}]}
@@ -1813,15 +1832,27 @@ func (d *Daemon) StartInstance(instance *vm.VM) error {
 
 		processChan <- cmd.Process.Pid
 
-		// Read the pts from launch
-		err = cmd.Wait()
+		// Block until QEMU exits
+		waitErr := cmd.Wait()
 
-		if err != nil {
-			slog.Error("Failed to wait for VM:", "err", err)
-			exitChan <- 1
-			return
+		if waitErr != nil {
+			slog.Error("VM process exited", "instance", instance.ID, "err", waitErr)
 		}
 
+		// Signal startup check (non-blocking)
+		select {
+		case exitChan <- 1:
+		default:
+		}
+
+		// Wait for startup phase to complete before deciding on crash handling
+		confirmed := <-startupConfirmed
+		if !confirmed {
+			return // Startup failed, LaunchInstance handles the error
+		}
+
+		// Runtime crash — handle it
+		d.handleInstanceCrash(instance, waitErr)
 	}()
 
 	// Wait for startup result
@@ -1843,16 +1874,17 @@ func (d *Daemon) StartInstance(instance *vm.VM) error {
 		//return fmt.Errorf("failed to get pts")
 	}
 
-	// Check if nbdkit exited immediately with an error
+	// Check if QEMU exited immediately with an error
 	select {
 	case exitErr := <-exitChan:
+		startupConfirmed <- false // tell goroutine not to handle crash
 		if exitErr != 0 {
 			errorMsg := fmt.Errorf("failed: %v", exitErr)
 			slog.Error("Failed to launch qemu", "err", errorMsg)
 			return errorMsg
 		}
 	default:
-		// nbdkit is still running after 1 second, which means it started successfully
+		startupConfirmed <- true // goroutine will handle future crashes
 		slog.Info("QEMU started successfully and is running", "pts", pts)
 	}
 

hive/daemon/daemon_test.go

@@ -76,7 +76,7 @@ func createTestDaemon(t *testing.T, natsURL string) *Daemon {
 
 	// Initialize services (needed for handler tests)
 	daemon.instanceService = handlers_ec2_instance.NewInstanceServiceImpl(cfg, daemon.resourceMgr.instanceTypes, nc, &daemon.Instances, objectstore.NewMemoryObjectStore())
-	daemon.volumeService = handlers_ec2_volume.NewVolumeServiceImpl(cfg, nc)
+	daemon.volumeService = handlers_ec2_volume.NewVolumeServiceImpl(cfg, nc, nil)
 
 	t.Cleanup(func() {
 		if daemon.natsConn != nil {
@@ -3256,6 +3256,28 @@ func TestGenerateInstanceTypes_SingleFamily(t *testing.T) {
 	}
 }
 
+func TestGenerateInstanceTypes_AMDBurstable(t *testing.T) {
+	types := generateInstanceTypes("m8a", "x86_64")
+	assert.Len(t, types, 14) // 7 m8a + 7 t3a
+
+	m8aCount, t3aCount := 0, 0
+	for name := range types {
+		if strings.HasPrefix(name, "m8a.") {
+			m8aCount++
+		}
+		if strings.HasPrefix(name, "t3a.") {
+			t3aCount++
+		}
+	}
+	assert.Equal(t, 7, m8aCount)
+	assert.Equal(t, 7, t3aCount)
+
+	// Verify t3 (Intel burstable) is NOT present
+	for name := range types {
+		assert.False(t, strings.HasPrefix(name, "t3."), "AMD should get t3a, not t3: %s", name)
+	}
+}
+
 func TestGenerateInstanceTypes_ARM64(t *testing.T) {
 	types := generateInstanceTypes("m8g", "arm64")
 	// m8g is not t4g, so we should get both families

hive/daemon/health.go

@@ -0,0 +1,274 @@
+package daemon
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"log/slog"
+	"os"
+	"os/exec"
+	"syscall"
+	"time"
+
+	"github.com/mulgadc/hive/hive/vm"
+)
+
+const (
+	maxRestartsInWindow = 3
+	restartWindow       = 10 * time.Minute
+	restartBackoffBase  = 5 * time.Second
+	restartBackoffMax   = 2 * time.Minute
+)
+
+// classifyCrashReason extracts a human-readable crash reason from the error
+// returned by cmd.Wait(). Uses exec.ExitError + syscall.WaitStatus to
+// distinguish OOM kills (SIGKILL), segfaults (SIGSEGV), etc.
+func classifyCrashReason(waitErr error) string {
+	if waitErr == nil {
+		return "clean-exit"
+	}
+
+	var exitErr *exec.ExitError
+	if !errors.As(waitErr, &exitErr) {
+		return "unknown"
+	}
+
+	status, ok := exitErr.Sys().(syscall.WaitStatus)
+	if !ok {
+		return "unknown"
+	}
+
+	if status.Signaled() {
+		switch status.Signal() {
+		case syscall.SIGKILL:
+			return "oom-killed"
+		case syscall.SIGSEGV:
+			return "segfault"
+		case syscall.SIGABRT:
+			return "abort"
+		default:
+			return fmt.Sprintf("signal-%d", status.Signal())
+		}
+	}
+
+	if status.Exited() {
+		return fmt.Sprintf("exit-%d", status.ExitStatus())
+	}
+
+	return "unknown"
+}
+
+// handleInstanceCrash is called by the QEMU launch goroutine after cmd.Wait()
+// returns during runtime (after startup was confirmed successful). It detects
+// the crash reason, transitions the instance to error state, cleans up resources,
+// and schedules a restart if policy allows.
+func (d *Daemon) handleInstanceCrash(instance *vm.VM, waitErr error) {
+	// Guard: if instance is not running, this was an expected exit
+	// (stopInstance/terminateInstance set status before QEMU exits)
+	d.Instances.Mu.Lock()
+	status := instance.Status
+	d.Instances.Mu.Unlock()
+
+	if status != vm.StateRunning {
+		slog.Debug("QEMU exited but instance not in running state, skipping crash handler",
+			"instance", instance.ID, "status", status)
+		return
+	}
+
+	// Guard: coordinated shutdown in progress
+	if d.shuttingDown.Load() {
+		slog.Debug("QEMU exited during coordinated shutdown, skipping crash handler",
+			"instance", instance.ID)
+		return
+	}
+
+	reason := classifyCrashReason(waitErr)
+	slog.Error("VM process crashed", "instance", instance.ID, "reason", reason, "err", waitErr)
+
+	// Transition to error state
+	if err := d.TransitionState(instance, vm.StateError); err != nil {
+		slog.Error("Failed to transition crashed instance to error state",
+			"instance", instance.ID, "err", err)
+	}
+
+	// Update health tracking
+	now := time.Now()
+	d.Instances.Mu.Lock()
+	instance.Health.CrashCount++
+	instance.Health.LastCrashTime = now
+	instance.Health.LastCrashReason = reason
+	if instance.Health.FirstCrashTime.IsZero() {
+		instance.Health.FirstCrashTime = now
+	}
+	instance.Running = false
+	instance.PID = 0
+	d.Instances.Mu.Unlock()
+
+	// Deallocate resources to fix phantom reservation
+	instanceType, ok := d.resourceMgr.instanceTypes[instance.InstanceType]
+	if ok && instanceType != nil {
+		slog.Info("Deallocating resources for crashed instance",
+			"instance", instance.ID, "type", instance.InstanceType)
+		d.resourceMgr.deallocate(instanceType)
+	}
+
+	// Clean up stale QMP socket so QEMU can rebind on restart
+	if instance.Config.QMPSocket != "" {
+		_ = os.Remove(instance.Config.QMPSocket)
+	}
+
+	// Unmount EBS volumes (same pattern as stopInstance)
+	d.unmountInstanceVolumes(instance)
+
+	// Persist state
+	if err := d.WriteState(); err != nil {
+		slog.Error("Failed to persist state after crash handling",
+			"instance", instance.ID, "err", err)
+	}
+
+	d.maybeRestartInstance(instance)
+}
+
+// unmountInstanceVolumes sends NATS unmount requests for all volumes attached
+// to the instance and updates their state to "available".
+func (d *Daemon) unmountInstanceVolumes(instance *vm.VM) {
+	instance.EBSRequests.Mu.Lock()
+	defer instance.EBSRequests.Mu.Unlock()
+
+	for _, ebsRequest := range instance.EBSRequests.Requests {
+		ebsUnMountRequest, err := json.Marshal(ebsRequest)
+		if err != nil {
+			slog.Error("Failed to marshal volume payload for crash cleanup",
+				"err", err)
+			continue
+		}
+
+		msg, err := d.natsConn.Request(d.ebsTopic("unmount"), ebsUnMountRequest, 30*time.Second)
+		if err != nil {
+			slog.Error("Failed to unmount volume after crash",
+				"name", ebsRequest.Name, "instance", instance.ID, "err", err)
+		} else {
+			slog.Info("Unmounted volume after crash",
+				"instance", instance.ID, "volume", ebsRequest.Name, "data", string(msg.Data))
+		}
+
+		// Update user-visible volume state to "available"
+		if !ebsRequest.EFI && !ebsRequest.CloudInit {
+			if err := d.volumeService.UpdateVolumeState(ebsRequest.Name, "available", "", ""); err != nil {
+				slog.Error("Failed to update volume state to available after crash",
+					"volumeId", ebsRequest.Name, "err", err)
+			}
+		}
+	}
+}
+
+// maybeRestartInstance checks restart policy and schedules a restart if allowed.
+func (d *Daemon) maybeRestartInstance(instance *vm.VM) {
+	if d.shuttingDown.Load() {
+		slog.Info("Skipping restart during shutdown", "instance", instance.ID)
+		return
+	}
+
+	now := time.Now()
+
+	d.Instances.Mu.Lock()
+	health := &instance.Health
+
+	// If crashes are outside the restart window, reset counters
+	if !health.FirstCrashTime.IsZero() && now.Sub(health.FirstCrashTime) > restartWindow {
+		slog.Info("Crash window expired, resetting counters", "instance", instance.ID)
+		health.CrashCount = 1
+		health.FirstCrashTime = now
+		health.RestartCount = 0
+	}
+
+	// Check if we've exceeded the max restarts in the window
+	if health.CrashCount > maxRestartsInWindow {
+		d.Instances.Mu.Unlock()
+		slog.Error("Instance exceeded max restarts in window, leaving in error state",
+			"instance", instance.ID,
+			"crashes", health.CrashCount,
+			"window", restartWindow,
+			"max", maxRestartsInWindow)
+		return
+	}
+
+	restartCount := health.RestartCount
+	d.Instances.Mu.Unlock()
+
+	// Check resource availability
+	instanceType, ok := d.resourceMgr.instanceTypes[instance.InstanceType]
+	if !ok || instanceType == nil {
+		slog.Error("Unknown instance type, cannot restart",
+			"instance", instance.ID, "type", instance.InstanceType)
+		return
+	}
+
+	if d.resourceMgr.canAllocate(instanceType, 1) < 1 {
+		slog.Error("Insufficient resources to restart instance",
+			"instance", instance.ID, "type", instance.InstanceType)
+		return
+	}
+
+	// Calculate exponential backoff
+	delay := restartBackoffBase
+	for i := 0; i < restartCount; i++ {
+		delay *= 2
+		if delay > restartBackoffMax {
+			delay = restartBackoffMax
+			break
+		}
+	}
+
+	slog.Info("Scheduling instance restart",
+		"instance", instance.ID,
+		"delay", delay,
+		"restartCount", restartCount+1)
+
+	time.AfterFunc(delay, func() {
+		d.restartCrashedInstance(instance)
+	})
+}
+
+// restartCrashedInstance re-verifies the instance is still in error state
+// and relaunches it via LaunchInstance.
+func (d *Daemon) restartCrashedInstance(instance *vm.VM) {
+	d.Instances.Mu.Lock()
+	if instance.Status != vm.StateError {
+		d.Instances.Mu.Unlock()
+		slog.Info("Instance no longer in error state, skipping restart",
+			"instance", instance.ID, "status", instance.Status)
+		return
+	}
+
+	if d.shuttingDown.Load() {
+		d.Instances.Mu.Unlock()
+		slog.Info("Daemon shutting down, skipping restart", "instance", instance.ID)
+		return
+	}
+
+	instance.Health.RestartCount++
+	d.Instances.Mu.Unlock()
+
+	slog.Info("Restarting crashed instance",
+		"instance", instance.ID,
+		"restartCount", instance.Health.RestartCount)
+
+	// Transition Error → Pending (valid now that we added it to the transitions map)
+	if err := d.TransitionState(instance, vm.StatePending); err != nil {
+		slog.Error("Failed to transition instance to pending for restart",
+			"instance", instance.ID, "err", err)
+		return
+	}
+
+	// LaunchInstance handles the full relaunch flow:
+	// check PID → MountVolumes → StartInstance → CreateQMPClient → NATS subscribe → Running
+	if err := d.LaunchInstance(instance); err != nil {
+		slog.Error("Failed to restart crashed instance",
+			"instance", instance.ID, "err", err)
+		if err := d.TransitionState(instance, vm.StateError); err != nil {
+			slog.Error("Failed to transition instance back to error after restart failure",
+				"instance", instance.ID, "err", err)
+		}
+	}
+}

hive/daemon/health_test.go

@@ -0,0 +1,202 @@
+package daemon
+
+import (
+	"fmt"
+	"os/exec"
+	"testing"
+	"time"
+
+	"github.com/mulgadc/hive/hive/config"
+	"github.com/mulgadc/hive/hive/vm"
+	"github.com/nats-io/nats.go"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestClassifyCrashReason_CleanExit(t *testing.T) {
+	assert.Equal(t, "clean-exit", classifyCrashReason(nil))
+}
+
+func TestClassifyCrashReason_OOMKill(t *testing.T) {
+	// Start a process and kill it with SIGKILL to get a real ExitError
+	cmd := exec.Command("sleep", "60")
+	require.NoError(t, cmd.Start())
+
+	require.NoError(t, cmd.Process.Kill()) // sends SIGKILL
+	waitErr := cmd.Wait()
+	require.Error(t, waitErr)
+
+	assert.Equal(t, "oom-killed", classifyCrashReason(waitErr))
+}
+
+func TestClassifyCrashReason_ExitCode(t *testing.T) {
+	// Run a command that exits with a non-zero code
+	cmd := exec.Command("sh", "-c", "exit 42")
+	waitErr := cmd.Run()
+	require.Error(t, waitErr)
+
+	assert.Equal(t, "exit-42", classifyCrashReason(waitErr))
+}
+
+func TestClassifyCrashReason_Unknown(t *testing.T) {
+	// A non-ExitError error
+	err := fmt.Errorf("some random error")
+	assert.Equal(t, "unknown", classifyCrashReason(err))
+}
+
+func TestHandleInstanceCrash_SkipsNonRunning(t *testing.T) {
+	nc, err := nats.Connect(sharedNATSURL)
+	require.NoError(t, err)
+	defer nc.Close()
+
+	d := &Daemon{
+		natsConn:    nc,
+		resourceMgr: NewResourceManager(),
+		Instances:   vm.Instances{VMS: make(map[string]*vm.VM)},
+	}
+
+	instance := &vm.VM{
+		ID:     "i-test-stopped",
+		Status: vm.StateStopped,
+	}
+	d.Instances.VMS[instance.ID] = instance
+
+	// Should return without action (no panic, no state change)
+	d.handleInstanceCrash(instance, fmt.Errorf("test error"))
+	assert.Equal(t, vm.StateStopped, instance.Status)
+}
+
+func TestHandleInstanceCrash_SkipsShuttingDown(t *testing.T) {
+	nc, err := nats.Connect(sharedNATSURL)
+	require.NoError(t, err)
+	defer nc.Close()
+
+	d := &Daemon{
+		natsConn:    nc,
+		resourceMgr: NewResourceManager(),
+		Instances:   vm.Instances{VMS: make(map[string]*vm.VM)},
+	}
+	d.shuttingDown.Store(true)
+
+	instance := &vm.VM{
+		ID:     "i-test-shutdown",
+		Status: vm.StateRunning,
+	}
+	d.Instances.VMS[instance.ID] = instance
+
+	d.handleInstanceCrash(instance, fmt.Errorf("test error"))
+
+	// Status unchanged because shuttingDown is true
+	assert.Equal(t, vm.StateRunning, instance.Status)
+}
+
+func TestMaybeRestart_ExceedsMaxInWindow(t *testing.T) {
+	nc, err := nats.Connect(sharedNATSURL)
+	require.NoError(t, err)
+	defer nc.Close()
+
+	d := &Daemon{
+		node:     "test-node",
+		natsConn: nc,
+		config: &config.Config{
+			Services: []string{"daemon"},
+		},
+		resourceMgr: NewResourceManager(),
+		Instances:   vm.Instances{VMS: make(map[string]*vm.VM)},
+	}
+
+	// Find an allocatable instance type
+	var allocType string
+	for typeName := range d.resourceMgr.instanceTypes {
+		allocType = typeName
+		break
+	}
+	require.NotEmpty(t, allocType)
+
+	instance := &vm.VM{
+		ID:           "i-test-maxcrash",
+		Status:       vm.StateError,
+		InstanceType: allocType,
+		Health: vm.InstanceHealthState{
+			CrashCount:     maxRestartsInWindow + 1, // exceeds limit
+			FirstCrashTime: time.Now(),
+			RestartCount:   maxRestartsInWindow,
+		},
+	}
+	d.Instances.VMS[instance.ID] = instance
+
+	// Should not schedule a restart (no panic, instance stays in error)
+	d.maybeRestartInstance(instance)
+
+	// Instance should remain in error state
+	assert.Equal(t, vm.StateError, instance.Status)
+}
+
+func TestMaybeRestart_ResetsAfterWindow(t *testing.T) {
+	nc, err := nats.Connect(sharedNATSURL)
+	require.NoError(t, err)
+	defer nc.Close()
+
+	d := &Daemon{
+		node:     "test-node",
+		natsConn: nc,
+		config: &config.Config{
+			Services: []string{"daemon"},
+		},
+		resourceMgr: NewResourceManager(),
+		Instances:   vm.Instances{VMS: make(map[string]*vm.VM)},
+	}
+
+	// Find an allocatable instance type
+	var allocType string
+	for typeName := range d.resourceMgr.instanceTypes {
+		allocType = typeName
+		break
+	}
+	require.NotEmpty(t, allocType)
+
+	instance := &vm.VM{
+		ID:           "i-test-windowreset",
+		Status:       vm.StateError,
+		InstanceType: allocType,
+		Health: vm.InstanceHealthState{
+			CrashCount:     5,
+			FirstCrashTime: time.Now().Add(-restartWindow - time.Minute), // outside window
+			RestartCount:   5,
+		},
+	}
+	d.Instances.VMS[instance.ID] = instance
+
+	// maybeRestartInstance should reset counters and schedule a restart
+	// (restart will fail because there's no real QEMU, but counters should reset)
+	d.maybeRestartInstance(instance)
+
+	// Counters should be reset
+	assert.Equal(t, 1, instance.Health.CrashCount)
+	assert.Equal(t, 0, instance.Health.RestartCount)
+}
+
+func TestRestartBackoff_Exponential(t *testing.T) {
+	// Verify the backoff calculation logic
+	expected := []time.Duration{
+		5 * time.Second,   // restart 0
+		10 * time.Second,  // restart 1
+		20 * time.Second,  // restart 2
+		40 * time.Second,  // restart 3
+		80 * time.Second,  // restart 4
+		120 * time.Second, // restart 5 (capped at 2min)
+		120 * time.Second, // restart 6 (capped at 2min)
+	}
+
+	for i, want := range expected {
+		delay := restartBackoffBase
+		for j := 0; j < i; j++ {
+			delay *= 2
+			if delay > restartBackoffMax {
+				delay = restartBackoffMax
+				break
+			}
+		}
+		assert.Equal(t, want, delay, "Backoff mismatch at restart count %d", i)
+	}
+}

hive/daemon/jetstream.go

@@ -51,7 +51,7 @@ func (m *JetStreamManager) InitKVBucket() error {
 	if err != nil {
 		if errors.Is(err, nats.ErrBucketNotFound) {
 			// Bucket doesn't exist, create it
-			slog.Info("Creating JetStream KV bucket", "bucket", InstanceStateBucket, "replicas", m.replicas)
+			slog.Debug("Creating JetStream KV bucket", "bucket", InstanceStateBucket, "replicas", m.replicas)
 			kv, err = m.js.CreateKeyValue(&nats.KeyValueConfig{
 				Bucket:      InstanceStateBucket,
 				Description: "Hive instance state storage",
@@ -65,7 +65,7 @@ func (m *JetStreamManager) InitKVBucket() error {
 			return err
 		}
 	} else {
-		slog.Info("Connected to existing JetStream KV bucket", "bucket", InstanceStateBucket)
+		slog.Debug("Connected to existing JetStream KV bucket", "bucket", InstanceStateBucket)
 	}
 
 	m.kv = kv
@@ -77,7 +77,7 @@ func (m *JetStreamManager) InitClusterStateBucket() error {
 	kv, err := m.js.KeyValue(ClusterStateBucket)
 	if err != nil {
 		if errors.Is(err, nats.ErrBucketNotFound) {
-			slog.Info("Creating JetStream KV bucket", "bucket", ClusterStateBucket, "replicas", m.replicas)
+			slog.Debug("Creating JetStream KV bucket", "bucket", ClusterStateBucket, "replicas", m.replicas)
 			kv, err = m.js.CreateKeyValue(&nats.KeyValueConfig{
 				Bucket:      ClusterStateBucket,
 				Description: "Hive cluster state (heartbeats, shutdown markers, service maps)",
@@ -92,7 +92,7 @@ func (m *JetStreamManager) InitClusterStateBucket() error {
 			return err
 		}
 	} else {
-		slog.Info("Connected to existing JetStream KV bucket", "bucket", ClusterStateBucket)
+		slog.Debug("Connected to existing JetStream KV bucket", "bucket", ClusterStateBucket)
 	}
 
 	m.clusterKV = kv

hive/handlers/ec2/snapshot/service_impl.go

@@ -24,11 +24,14 @@ import (
 // Ensure SnapshotServiceImpl implements SnapshotService
 var _ SnapshotService = (*SnapshotServiceImpl)(nil)
 
+const KVBucketVolumeSnapshots = "hive-volume-snapshots"
+
 // SnapshotServiceImpl implements SnapshotService with S3-backed storage
 type SnapshotServiceImpl struct {
 	config   *config.Config
 	store    objectstore.ObjectStore
 	natsConn *nats.Conn
+	snapKV   nats.KeyValue
 	mutex    sync.RWMutex
 }
 
@@ -47,8 +50,8 @@ type SnapshotConfig struct {
 	Tags             map[string]string `json:"tags"`
 }
 
-// NewSnapshotServiceImpl creates a new snapshot service implementation
-func NewSnapshotServiceImpl(cfg *config.Config, natsConn *nats.Conn) *SnapshotServiceImpl {
+// NewSnapshotServiceImplWithNATS creates a snapshot service with JetStream KV for volume-snapshot tracking
+func NewSnapshotServiceImplWithNATS(cfg *config.Config, natsConn *nats.Conn) (*SnapshotServiceImpl, nats.KeyValue, error) {
 	store := objectstore.NewS3ObjectStoreFromConfig(
 		cfg.Predastore.Host,
 		cfg.Predastore.Region,
@@ -56,20 +59,52 @@ func NewSnapshotServiceImpl(cfg *config.Config, natsConn *nats.Conn) *SnapshotSe
 		cfg.Predastore.SecretKey,
 	)
 
+	js, err := natsConn.JetStream()
+	if err != nil {
+		return nil, nil, fmt.Errorf("failed to get JetStream context: %w", err)
+	}
+
+	kv, err := getOrCreateKVBucket(js, KVBucketVolumeSnapshots, 10)
+	if err != nil {
+		return nil, nil, fmt.Errorf("failed to create KV bucket %s: %w", KVBucketVolumeSnapshots, err)
+	}
+
+	slog.Info("Snapshot service initialized with JetStream KV", "bucket", KVBucketVolumeSnapshots)
+
 	return &SnapshotServiceImpl{
 		config:   cfg,
 		store:    store,
 		natsConn: natsConn,
-	}
+		snapKV:   kv,
+	}, kv, nil
 }
 
-// NewSnapshotServiceImplWithStore creates a snapshot service with a custom ObjectStore (for testing)
-func NewSnapshotServiceImplWithStore(cfg *config.Config, store objectstore.ObjectStore, natsConn *nats.Conn) *SnapshotServiceImpl {
-	return &SnapshotServiceImpl{
+// NewSnapshotServiceImplWithStore creates a snapshot service with a custom ObjectStore (for testing).
+// An optional snapshotKV can be provided for KV-backed volume-snapshot tracking.
+func NewSnapshotServiceImplWithStore(cfg *config.Config, store objectstore.ObjectStore, natsConn *nats.Conn, snapshotKV ...nats.KeyValue) *SnapshotServiceImpl {
+	svc := &SnapshotServiceImpl{
 		config:   cfg,
 		store:    store,
 		natsConn: natsConn,
 	}
+	if len(snapshotKV) > 0 {
+		svc.snapKV = snapshotKV[0]
+	}
+	return svc
+}
+
+func getOrCreateKVBucket(js nats.JetStreamContext, bucketName string, history int) (nats.KeyValue, error) {
+	kv, err := js.CreateKeyValue(&nats.KeyValueConfig{
+		Bucket:  bucketName,
+		History: utils.SafeIntToUint8(history),
+	})
+	if err != nil {
+		kv, err = js.KeyValue(bucketName)
+		if err != nil {
+			return nil, err
+		}
+	}
+	return kv, nil
 }
 
 // getSnapshotKey returns the S3 key for storing snapshot metadata.
@@ -197,22 +232,25 @@ func (s *SnapshotServiceImpl) CreateSnapshot(input *ec2.CreateSnapshotInput) (*e
 		}
 
 		msg, err := s.natsConn.Request(fmt.Sprintf("ebs.snapshot.%s", volumeID), snapData, 30*time.Second)
-		if err != nil {
+		if errors.Is(err, nats.ErrNoResponders) {
+			// Volume is not mounted — data is already persisted to S3, proceed with metadata-only snapshot.
+			slog.Info("CreateSnapshot: volume not mounted, creating metadata-only snapshot", "volumeId", volumeID, "snapshotId", snapshotID)
+		} else if err != nil {
 			slog.Error("CreateSnapshot: ebs.snapshot NATS request failed", "volumeId", volumeID, "snapshotId", snapshotID, "err", err)
 			return nil, errors.New(awserrors.ErrorServerInternal)
-		}
+		} else {
+			var snapResp config.EBSSnapshotResponse
+			if err := json.Unmarshal(msg.Data, &snapResp); err != nil {
+				slog.Error("CreateSnapshot: failed to unmarshal ebs.snapshot response", "volumeId", volumeID, "snapshotId", snapshotID, "err", err)
+				return nil, errors.New(awserrors.ErrorServerInternal)
+			}
+			if !snapResp.Success || snapResp.Error != "" {
+				slog.Error("CreateSnapshot: viperblock snapshot failed", "volumeId", volumeID, "snapshotId", snapshotID, "err", snapResp.Error)
+				return nil, errors.New(awserrors.ErrorServerInternal)
+			}
 
-		var snapResp config.EBSSnapshotResponse
-		if err := json.Unmarshal(msg.Data, &snapResp); err != nil {
-			slog.Error("CreateSnapshot: failed to unmarshal ebs.snapshot response", "volumeId", volumeID, "snapshotId", snapshotID, "err", err)
-			return nil, errors.New(awserrors.ErrorServerInternal)
+			slog.Info("CreateSnapshot: viperblock snapshot created", "volumeId", volumeID, "snapshotId", snapshotID)
 		}
-		if !snapResp.Success || snapResp.Error != "" {
-			slog.Error("CreateSnapshot: viperblock snapshot failed", "volumeId", volumeID, "snapshotId", snapshotID, "err", snapResp.Error)
-			return nil, errors.New(awserrors.ErrorServerInternal)
-		}
-
-		slog.Info("CreateSnapshot: viperblock snapshot created", "volumeId", volumeID, "snapshotId", snapshotID)
 	} else {
 		slog.Warn("CreateSnapshot: natsConn is nil, skipping viperblock snapshot (metadata-only)", "volumeId", volumeID)
 	}
@@ -246,8 +284,16 @@ func (s *SnapshotServiceImpl) CreateSnapshot(input *ec2.CreateSnapshotInput) (*e
 		}
 	}
 
+	// Track the volume→snapshot dependency in KV before persisting to S3.
+	// This ensures we never have an untracked snapshot in S3.
+	if err := s.addSnapshotRef(volumeID, snapshotID); err != nil {
+		slog.Error("CreateSnapshot failed to add snapshot ref to KV", "snapshotId", snapshotID, "volumeId", volumeID, "err", err)
+		return nil, errors.New(awserrors.ErrorServerInternal)
+	}
+
 	if err := s.putSnapshotConfig(snapshotID, snapshotCfg); err != nil {
 		slog.Error("CreateSnapshot failed to write config", "snapshotId", snapshotID, "err", err)
+		_ = s.removeSnapshotRef(volumeID, snapshotID) // best-effort cleanup
 		return nil, errors.New(awserrors.ErrorServerInternal)
 	}
 
@@ -318,7 +364,7 @@ func (s *SnapshotServiceImpl) DeleteSnapshot(input *ec2.DeleteSnapshotInput) (*e
 
 	slog.Info("DeleteSnapshot request", "snapshotId", snapshotID)
 
-	_, err := s.getSnapshotConfig(snapshotID)
+	cfg, err := s.getSnapshotConfig(snapshotID)
 	if err != nil {
 		slog.Error("DeleteSnapshot snapshot not found", "snapshotId", snapshotID, "err", err)
 		return nil, err
@@ -346,6 +392,12 @@ func (s *SnapshotServiceImpl) DeleteSnapshot(input *ec2.DeleteSnapshotInput) (*e
 		}
 	}
 
+	// Remove from KV after S3 cleanup. Failure is logged but not fatal —
+	// a phantom entry safely blocks volume deletion rather than allowing it.
+	if err := s.removeSnapshotRef(cfg.VolumeID, snapshotID); err != nil {
+		slog.Warn("DeleteSnapshot failed to remove snapshot ref from KV", "snapshotId", snapshotID, "volumeId", cfg.VolumeID, "err", err)
+	}
+
 	slog.Info("DeleteSnapshot completed", "snapshotId", snapshotID)
 
 	return &ec2.DeleteSnapshotOutput{}, nil
@@ -391,8 +443,15 @@ func (s *SnapshotServiceImpl) CopySnapshot(input *ec2.CopySnapshotInput) (*ec2.C
 		newCfg.Tags[k] = v
 	}
 
+	// Track the volume→snapshot dependency in KV before persisting to S3.
+	if err := s.addSnapshotRef(sourceCfg.VolumeID, newSnapshotID); err != nil {
+		slog.Error("CopySnapshot failed to add snapshot ref to KV", "snapshotId", newSnapshotID, "volumeId", sourceCfg.VolumeID, "err", err)
+		return nil, errors.New(awserrors.ErrorServerInternal)
+	}
+
 	if err := s.putSnapshotConfig(newSnapshotID, newCfg); err != nil {
 		slog.Error("CopySnapshot failed to write config", "snapshotId", newSnapshotID, "err", err)
+		_ = s.removeSnapshotRef(sourceCfg.VolumeID, newSnapshotID) // best-effort cleanup
 		return nil, errors.New(awserrors.ErrorServerInternal)
 	}
 
@@ -402,3 +461,137 @@ func (s *SnapshotServiceImpl) CopySnapshot(input *ec2.CopySnapshotInput) (*ec2.C
 		SnapshotId: aws.String(newSnapshotID),
 	}, nil
 }
+
+// addSnapshotRef adds snapshotID to the volume's snapshot list in KV.
+// Uses CAS (Create/Update with revision) to prevent lost updates under concurrency.
+func (s *SnapshotServiceImpl) addSnapshotRef(volumeID, snapshotID string) error {
+	if s.snapKV == nil {
+		slog.Debug("addSnapshotRef: snapshotKV is nil, skipping", "volumeId", volumeID, "snapshotId", snapshotID)
+		return nil
+	}
+
+	const maxRetries = 5
+	for attempt := range maxRetries {
+		entry, err := s.snapKV.Get(volumeID)
+		var snapshots []string
+
+		if err != nil {
+			if !errors.Is(err, nats.ErrKeyNotFound) {
+				return fmt.Errorf("addSnapshotRef: failed to get KV key %s: %w", volumeID, err)
+			}
+			// Key doesn't exist yet — create with just this snapshot
+			data, err := json.Marshal([]string{snapshotID})
+			if err != nil {
+				return fmt.Errorf("addSnapshotRef: failed to marshal snapshot list: %w", err)
+			}
+			if _, err := s.snapKV.Create(volumeID, data); err != nil {
+				if attempt < maxRetries-1 {
+					continue // concurrent Create/Update — retry
+				}
+				return fmt.Errorf("addSnapshotRef: failed to create KV key %s: %w", volumeID, err)
+			}
+			slog.Info("addSnapshotRef: added snapshot ref", "volumeId", volumeID, "snapshotId", snapshotID)
+			return nil
+		}
+
+		if err := json.Unmarshal(entry.Value(), &snapshots); err != nil {
+			return fmt.Errorf("addSnapshotRef: failed to unmarshal KV value for %s: %w", volumeID, err)
+		}
+
+		snapshots = append(snapshots, snapshotID)
+
+		data, err := json.Marshal(snapshots)
+		if err != nil {
+			return fmt.Errorf("addSnapshotRef: failed to marshal snapshot list: %w", err)
+		}
+
+		if _, err := s.snapKV.Update(volumeID, data, entry.Revision()); err != nil {
+			if attempt < maxRetries-1 {
+				continue // concurrent update — retry
+			}
+			return fmt.Errorf("addSnapshotRef: failed to update KV key %s: %w", volumeID, err)
+		}
+
+		slog.Info("addSnapshotRef: added snapshot ref", "volumeId", volumeID, "snapshotId", snapshotID)
+		return nil
+	}
+
+	return fmt.Errorf("addSnapshotRef: exhausted retries for KV key %s", volumeID)
+}
+
+// removeSnapshotRef removes snapshotID from the volume's snapshot list in KV.
+// Deletes the key if the list becomes empty.
+// Uses CAS (Update with revision) to prevent lost updates under concurrency.
+func (s *SnapshotServiceImpl) removeSnapshotRef(volumeID, snapshotID string) error {
+	if s.snapKV == nil {
+		slog.Debug("removeSnapshotRef: snapshotKV is nil, skipping", "volumeId", volumeID, "snapshotId", snapshotID)
+		return nil
+	}
+
+	const maxRetries = 5
+	for attempt := range maxRetries {
+		entry, err := s.snapKV.Get(volumeID)
+		if err != nil {
+			if errors.Is(err, nats.ErrKeyNotFound) {
+				return nil
+			}
+			return fmt.Errorf("removeSnapshotRef: failed to get KV key %s: %w", volumeID, err)
+		}
+
+		var snapshots []string
+		if err := json.Unmarshal(entry.Value(), &snapshots); err != nil {
+			return fmt.Errorf("removeSnapshotRef: failed to unmarshal KV value for %s: %w", volumeID, err)
+		}
+
+		filtered := snapshots[:0]
+		for _, snap := range snapshots {
+			if snap != snapshotID {
+				filtered = append(filtered, snap)
+			}
+		}
+
+		if len(filtered) == 0 {
+			if err := s.snapKV.Delete(volumeID); err != nil && !errors.Is(err, nats.ErrKeyNotFound) {
+				return fmt.Errorf("removeSnapshotRef: failed to delete KV key %s: %w", volumeID, err)
+			}
+		} else {
+			data, err := json.Marshal(filtered)
+			if err != nil {
+				return fmt.Errorf("removeSnapshotRef: failed to marshal snapshot list: %w", err)
+			}
+			if _, err := s.snapKV.Update(volumeID, data, entry.Revision()); err != nil {
+				if attempt < maxRetries-1 {
+					continue // concurrent update — retry
+				}
+				return fmt.Errorf("removeSnapshotRef: failed to update KV key %s: %w", volumeID, err)
+			}
+		}
+
+		slog.Info("removeSnapshotRef: removed snapshot ref", "volumeId", volumeID, "snapshotId", snapshotID)
+		return nil
+	}
+
+	return fmt.Errorf("removeSnapshotRef: exhausted retries for KV key %s", volumeID)
+}
+
+// volumeHasSnapshots returns true if the volume has any snapshots in KV.
+func (s *SnapshotServiceImpl) volumeHasSnapshots(volumeID string) (bool, error) {
+	if s.snapKV == nil {
+		return false, nil
+	}
+
+	entry, err := s.snapKV.Get(volumeID)
+	if err != nil {
+		if errors.Is(err, nats.ErrKeyNotFound) {
+			return false, nil
+		}
+		return false, fmt.Errorf("volumeHasSnapshots: failed to get KV key %s: %w", volumeID, err)
+	}
+
+	var snapshots []string
+	if err := json.Unmarshal(entry.Value(), &snapshots); err != nil {
+		return false, fmt.Errorf("volumeHasSnapshots: failed to unmarshal KV value for %s: %w", volumeID, err)
+	}
+
+	return len(snapshots) > 0, nil
+}

hive/handlers/ec2/snapshot/service_impl_test.go

@@ -4,6 +4,7 @@ import (
 	"encoding/json"
 	"strings"
 	"testing"
+	"time"
 
 	"github.com/aws/aws-sdk-go/aws"
 	"github.com/aws/aws-sdk-go/service/ec2"
@@ -12,6 +13,8 @@ import (
 	"github.com/mulgadc/hive/hive/config"
 	"github.com/mulgadc/hive/hive/objectstore"
 	"github.com/mulgadc/viperblock/viperblock"
+	"github.com/nats-io/nats-server/v2/server"
+	"github.com/nats-io/nats.go"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 )
@@ -349,3 +352,209 @@ func TestCopySnapshot_PreservesTags(t *testing.T) {
 	assert.Equal(t, "Environment", *result.Snapshots[0].Tags[0].Key)
 	assert.Equal(t, "test", *result.Snapshots[0].Tags[0].Value)
 }
+
+// setupTestNATSKV creates a NATS JetStream test server and returns a KV bucket for testing.
+func setupTestNATSKV(t *testing.T) nats.KeyValue {
+	t.Helper()
+	opts := &server.Options{
+		Host:      "127.0.0.1",
+		Port:      -1,
+		JetStream: true,
+		StoreDir:  t.TempDir(),
+		NoLog:     true,
+		NoSigs:    true,
+	}
+	ns, err := server.NewServer(opts)
+	require.NoError(t, err)
+	go ns.Start()
+	require.True(t, ns.ReadyForConnections(5*time.Second))
+	t.Cleanup(func() { ns.Shutdown() })
+
+	nc, err := nats.Connect(ns.ClientURL())
+	require.NoError(t, err)
+	t.Cleanup(func() { nc.Close() })
+
+	js, err := nc.JetStream()
+	require.NoError(t, err)
+
+	kv, err := js.CreateKeyValue(&nats.KeyValueConfig{
+		Bucket: KVBucketVolumeSnapshots,
+	})
+	require.NoError(t, err)
+	return kv
+}
+
+func TestAddSnapshotRef(t *testing.T) {
+	kv := setupTestNATSKV(t)
+	svc := &SnapshotServiceImpl{snapKV: kv}
+
+	require.NoError(t, svc.addSnapshotRef("vol-1", "snap-a"))
+	require.NoError(t, svc.addSnapshotRef("vol-1", "snap-b"))
+
+	entry, err := kv.Get("vol-1")
+	require.NoError(t, err)
+	var snapshots []string
+	require.NoError(t, json.Unmarshal(entry.Value(), &snapshots))
+	assert.Equal(t, []string{"snap-a", "snap-b"}, snapshots)
+}
+
+func TestRemoveSnapshotRef(t *testing.T) {
+	kv := setupTestNATSKV(t)
+	svc := &SnapshotServiceImpl{snapKV: kv}
+
+	require.NoError(t, svc.addSnapshotRef("vol-1", "snap-a"))
+	require.NoError(t, svc.addSnapshotRef("vol-1", "snap-b"))
+
+	// Remove one
+	require.NoError(t, svc.removeSnapshotRef("vol-1", "snap-a"))
+
+	entry, err := kv.Get("vol-1")
+	require.NoError(t, err)
+	var snapshots []string
+	require.NoError(t, json.Unmarshal(entry.Value(), &snapshots))
+	assert.Equal(t, []string{"snap-b"}, snapshots)
+
+	// Remove last — key should be deleted
+	require.NoError(t, svc.removeSnapshotRef("vol-1", "snap-b"))
+
+	_, err = kv.Get("vol-1")
+	assert.ErrorIs(t, err, nats.ErrKeyNotFound)
+}
+
+func TestRemoveSnapshotRef_NonExistentKey(t *testing.T) {
+	kv := setupTestNATSKV(t)
+	svc := &SnapshotServiceImpl{snapKV: kv}
+
+	// Should not error on non-existent key
+	require.NoError(t, svc.removeSnapshotRef("vol-nonexistent", "snap-x"))
+}
+
+func TestVolumeHasSnapshots(t *testing.T) {
+	kv := setupTestNATSKV(t)
+	svc := &SnapshotServiceImpl{snapKV: kv}
+
+	// No entry → false
+	has, err := svc.volumeHasSnapshots("vol-1")
+	require.NoError(t, err)
+	assert.False(t, has)
+
+	// Add one → true
+	require.NoError(t, svc.addSnapshotRef("vol-1", "snap-a"))
+	has, err = svc.volumeHasSnapshots("vol-1")
+	require.NoError(t, err)
+	assert.True(t, has)
+
+	// Remove it → false
+	require.NoError(t, svc.removeSnapshotRef("vol-1", "snap-a"))
+	has, err = svc.volumeHasSnapshots("vol-1")
+	require.NoError(t, err)
+	assert.False(t, has)
+}
+
+func TestKVNilFallback(t *testing.T) {
+	svc := &SnapshotServiceImpl{snapKV: nil}
+
+	// All methods should be no-ops when KV is nil
+	require.NoError(t, svc.addSnapshotRef("vol-1", "snap-a"))
+	require.NoError(t, svc.removeSnapshotRef("vol-1", "snap-a"))
+	has, err := svc.volumeHasSnapshots("vol-1")
+	require.NoError(t, err)
+	assert.False(t, has)
+}
+
+func TestCreateSnapshot_WritesKVEntry(t *testing.T) {
+	kv := setupTestNATSKV(t)
+	store := objectstore.NewMemoryObjectStore()
+	cfg := &config.Config{
+		Predastore: config.PredastoreConfig{
+			Bucket:    "test-bucket",
+			AccessKey: "test-owner-123",
+		},
+	}
+	svc := NewSnapshotServiceImplWithStore(cfg, store, nil, kv)
+
+	volumeID := "vol-kvtest"
+	createTestVolume(t, store, volumeID, 10)
+
+	snap, err := svc.CreateSnapshot(&ec2.CreateSnapshotInput{
+		VolumeId: aws.String(volumeID),
+	})
+	require.NoError(t, err)
+
+	// Verify KV entry exists
+	has, err := svc.volumeHasSnapshots(volumeID)
+	require.NoError(t, err)
+	assert.True(t, has)
+
+	// Verify snapshot ID is in the list
+	entry, err := kv.Get(volumeID)
+	require.NoError(t, err)
+	var snapshots []string
+	require.NoError(t, json.Unmarshal(entry.Value(), &snapshots))
+	assert.Contains(t, snapshots, *snap.SnapshotId)
+}
+
+func TestDeleteSnapshot_RemovesKVEntry(t *testing.T) {
+	kv := setupTestNATSKV(t)
+	store := objectstore.NewMemoryObjectStore()
+	cfg := &config.Config{
+		Predastore: config.PredastoreConfig{
+			Bucket:    "test-bucket",
+			AccessKey: "test-owner-123",
+		},
+	}
+	svc := NewSnapshotServiceImplWithStore(cfg, store, nil, kv)
+
+	volumeID := "vol-kvdelete"
+	createTestVolume(t, store, volumeID, 10)
+
+	snap, err := svc.CreateSnapshot(&ec2.CreateSnapshotInput{
+		VolumeId: aws.String(volumeID),
+	})
+	require.NoError(t, err)
+
+	// Delete the snapshot
+	_, err = svc.DeleteSnapshot(&ec2.DeleteSnapshotInput{
+		SnapshotId: snap.SnapshotId,
+	})
+	require.NoError(t, err)
+
+	// KV should now be empty for this volume
+	has, err := svc.volumeHasSnapshots(volumeID)
+	require.NoError(t, err)
+	assert.False(t, has)
+}
+
+func TestCopySnapshot_AddsKVEntry(t *testing.T) {
+	kv := setupTestNATSKV(t)
+	store := objectstore.NewMemoryObjectStore()
+	cfg := &config.Config{
+		Predastore: config.PredastoreConfig{
+			Bucket:    "test-bucket",
+			AccessKey: "test-owner-123",
+		},
+	}
+	svc := NewSnapshotServiceImplWithStore(cfg, store, nil, kv)
+
+	volumeID := "vol-kvcopy"
+	createTestVolume(t, store, volumeID, 10)
+
+	snap, err := svc.CreateSnapshot(&ec2.CreateSnapshotInput{
+		VolumeId: aws.String(volumeID),
+	})
+	require.NoError(t, err)
+
+	copyResult, err := svc.CopySnapshot(&ec2.CopySnapshotInput{
+		SourceSnapshotId: snap.SnapshotId,
+	})
+	require.NoError(t, err)
+
+	// Both snapshot IDs should be in KV
+	entry, err := kv.Get(volumeID)
+	require.NoError(t, err)
+	var snapshots []string
+	require.NoError(t, json.Unmarshal(entry.Value(), &snapshots))
+	assert.Contains(t, snapshots, *snap.SnapshotId)
+	assert.Contains(t, snapshots, *copyResult.SnapshotId)
+	assert.Len(t, snapshots, 2)
+}

hive/handlers/ec2/volume/service_impl.go

@@ -34,10 +34,13 @@ type VolumeServiceImpl struct {
 	store      objectstore.ObjectStore
 	bucketName string
 	natsConn   *nats.Conn
+	snapshotKV nats.KeyValue
 }
 
-// NewVolumeServiceImpl creates a new daemon-side volume service
-func NewVolumeServiceImpl(cfg *config.Config, natsConn *nats.Conn) *VolumeServiceImpl {
+// NewVolumeServiceImpl creates a new daemon-side volume service.
+// snapshotKV is optional — when non-nil, DeleteVolume uses O(1) KV lookup
+// instead of scanning all snapshots in S3.
+func NewVolumeServiceImpl(cfg *config.Config, natsConn *nats.Conn, snapshotKV nats.KeyValue) *VolumeServiceImpl {
 	store := objectstore.NewS3ObjectStoreFromConfig(
 		cfg.Predastore.Host,
 		cfg.Predastore.Region,
@@ -50,21 +53,26 @@ func NewVolumeServiceImpl(cfg *config.Config, natsConn *nats.Conn) *VolumeServic
 		store:      store,
 		bucketName: cfg.Predastore.Bucket,
 		natsConn:   natsConn,
+		snapshotKV: snapshotKV,
 	}
 }
 
 // NewVolumeServiceImplWithStore creates a volume service with a custom ObjectStore (for testing)
-func NewVolumeServiceImplWithStore(cfg *config.Config, store objectstore.ObjectStore, natsConn *nats.Conn) *VolumeServiceImpl {
+func NewVolumeServiceImplWithStore(cfg *config.Config, store objectstore.ObjectStore, natsConn *nats.Conn, snapshotKV ...nats.KeyValue) *VolumeServiceImpl {
 	bucketName := ""
 	if cfg != nil {
 		bucketName = cfg.Predastore.Bucket
 	}
-	return &VolumeServiceImpl{
+	svc := &VolumeServiceImpl{
 		config:     cfg,
 		store:      store,
 		bucketName: bucketName,
 		natsConn:   natsConn,
 	}
+	if len(snapshotKV) > 0 {
+		svc.snapshotKV = snapshotKV[0]
+	}
+	return svc
 }
 
 // CreateVolume creates a new EBS volume via viperblock and persists its config to S3
@@ -705,9 +713,9 @@ func (s *VolumeServiceImpl) DeleteVolume(input *ec2.DeleteVolumeInput) (*ec2.Del
 		return nil, errors.New(awserrors.ErrorVolumeInUse)
 	}
 
-	// Check if any snapshots reference this volume. Snapshot-backed clones
-	// read chunk files from the source volume's S3 prefix via ReadFrom().
-	// Deleting the source volume would silently break all clones.
+	// Check if any snapshots reference this volume via JetStream KV.
+	// Snapshot-backed clones read chunk files from the source volume's
+	// S3 prefix via ReadFrom(). Deleting the source would break all clones.
 	if err := s.checkVolumeHasNoSnapshots(volumeID); err != nil {
 		return nil, err
 	}
@@ -831,90 +839,40 @@ func (s *VolumeServiceImpl) getSnapshotMetadata(snapshotID string) (*snapshotMet
 	return &meta, nil
 }
 
-// snapshotVolumeRef reads VolumeID from hive metadata.json (volume_id)
-// or viperblock config.json (SourceVolumeName).
-type snapshotVolumeRef struct {
-	VolumeID         string `json:"volume_id"`
-	SourceVolumeName string `json:"SourceVolumeName"`
-}
-
-func (r snapshotVolumeRef) referencesVolume(volumeID string) bool {
-	return r.VolumeID == volumeID || r.SourceVolumeName == volumeID
-}
-
-// checkVolumeHasNoSnapshots scans all snap-* prefixes in S3 and returns an
-// error if any snapshot references the given volume. Checks both hive's
-// metadata.json and viperblock's config.json to handle either format.
+// checkVolumeHasNoSnapshots checks if a volume has dependent snapshots
+// using the JetStream KV index.
 func (s *VolumeServiceImpl) checkVolumeHasNoSnapshots(volumeID string) error {
-	listResult, err := s.store.ListObjects(&s3.ListObjectsInput{
-		Bucket:    aws.String(s.bucketName),
-		Prefix:    aws.String("snap-"),
-		Delimiter: aws.String("/"),
-	})
-	if err != nil {
-		slog.Error("checkVolumeHasNoSnapshots: failed to list snapshots", "volumeId", volumeID, "err", err)
+	if s.snapshotKV == nil {
+		slog.Error("checkVolumeHasNoSnapshots: snapshotKV is nil", "volumeId", volumeID)
 		return errors.New(awserrors.ErrorServerInternal)
 	}
 
-	for _, prefix := range listResult.CommonPrefixes {
-		if prefix.Prefix == nil {
-			continue
-		}
-
-		snapshotID := strings.TrimSuffix(*prefix.Prefix, "/")
-
-		if err := s.snapshotReferencesVolume(snapshotID, volumeID); err != nil {
-			return err
-		}
+	has, err := s.volumeHasSnapshotsKV(volumeID)
+	if err != nil {
+		slog.Error("checkVolumeHasNoSnapshots: KV lookup failed", "volumeId", volumeID, "err", err)
+		return errors.New(awserrors.ErrorServerInternal)
+	}
+	if has {
+		slog.Error("DeleteVolume blocked: volume has snapshots", "volumeId", volumeID)
+		return errors.New(awserrors.ErrorVolumeInUse)
 	}
-
 	return nil
 }
 
-// snapshotReferencesVolume checks if a snapshot references the given volume.
-// Returns an error if the snapshot references the volume or if a non-recoverable
-// S3/IO error occurs. Returns nil if the snapshot does not reference the volume.
-func (s *VolumeServiceImpl) snapshotReferencesVolume(snapshotID, volumeID string) error {
-	// Try metadata.json first (hive format), then config.json (viperblock format)
-	for _, filename := range []string{"metadata.json", "config.json"} {
-		key := snapshotID + "/" + filename
-
-		getResult, err := s.store.GetObject(&s3.GetObjectInput{
-			Bucket: aws.String(s.bucketName),
-			Key:    aws.String(key),
-		})
-		if err != nil {
-			if objectstore.IsNoSuchKeyError(err) {
-				continue // Expected: this format file doesn't exist for this snapshot
-			}
-			slog.Error("checkVolumeHasNoSnapshots: failed to read snapshot metadata",
-				"volumeId", volumeID, "snapshotId", snapshotID, "key", key, "err", err)
-			return errors.New(awserrors.ErrorServerInternal)
-		}
-
-		body, err := io.ReadAll(getResult.Body)
-		if closeErr := getResult.Body.Close(); closeErr != nil {
-			slog.Error("checkVolumeHasNoSnapshots: failed to close response body",
-				"volumeId", volumeID, "snapshotId", snapshotID, "key", key, "err", closeErr)
-		}
-		if err != nil {
-			slog.Error("checkVolumeHasNoSnapshots: failed to read snapshot body",
-				"volumeId", volumeID, "snapshotId", snapshotID, "key", key, "err", err)
-			return errors.New(awserrors.ErrorServerInternal)
-		}
-
-		var ref snapshotVolumeRef
-		if err := json.Unmarshal(body, &ref); err != nil {
-			slog.Warn("checkVolumeHasNoSnapshots: failed to parse snapshot metadata, skipping file",
-				"volumeId", volumeID, "snapshotId", snapshotID, "key", key, "err", err)
-			continue // JSON parse errors for individual files are not fatal -- try next file
-		}
-
-		if ref.referencesVolume(volumeID) {
-			slog.Error("DeleteVolume blocked: volume has snapshots", "volumeId", volumeID, "snapshotId", snapshotID)
-			return fmt.Errorf("%s: volume %s has existing snapshot %s. Delete snapshots first", awserrors.ErrorVolumeInUse, volumeID, snapshotID)
+// volumeHasSnapshotsKV checks the JetStream KV index for snapshot references.
+func (s *VolumeServiceImpl) volumeHasSnapshotsKV(volumeID string) (bool, error) {
+	entry, err := s.snapshotKV.Get(volumeID)
+	if err != nil {
+		if errors.Is(err, nats.ErrKeyNotFound) {
+			return false, nil
 		}
+		return false, err
 	}
 
-	return nil
+	var snapshots []string
+	if err := json.Unmarshal(entry.Value(), &snapshots); err != nil {
+		return false, err
+	}
+
+	return len(snapshots) > 0, nil
 }