Compare commits

..

328 Commits

Author SHA1 Message Date
Julian Sommer 404b3d6246 release: update go.mod for v0.2.3 2026-03-10 09:36:58 +10:00
Julian Sommer 021b0982f4 fix: bump go 1.26.1 references and fix gosec G104 in hiveui 2026-03-10 08:33:04 +10:00
Julian Sommer 7b590957e1 fix: bump go 1.26.1 2026-03-10 08:19:38 +10:00
Julian Sommer c9c3cda651 fix: redirect HTTP to HTTPS on same port for hive-ui
Uses a TLS-detecting listener that peeks the first byte of each
connection. TLS handshakes (0x16) proceed normally; plain HTTP
requests get a 301 redirect to the HTTPS URL. This handles
http://localhost:3000 gracefully instead of showing a cryptic error.
2026-03-10 08:14:19 +10:00
Julian Sommer 1efb4c4a3b fix: config 2026-03-06 14:51:21 +10:00
Julian Sommer 05cb4ac108 feat: frontend refactors (#66) 2026-03-06 15:42:01 +11:00
Julian Sommer 6eac4d6598 feat: IAM Frontend (#65) 2026-03-06 15:14:50 +11:00
Julian Sommer f03a2314c6 feat: terminated instances expiry (#63) 2026-03-06 12:51:27 +11:00
Julian Sommer 2c444b1eba chore: install tsgo 2026-03-06 11:46:21 +10:00
Julian Sommer aea2e2c05a ci: setup go work after 2026-03-06 10:41:45 +10:00
Julian Sommer d90bf53eaf fix: log rotate ovn 2026-03-06 10:10:34 +10:00
Julian Sommer 14c52366da fix: decouple gateway from qmp 2026-03-06 10:06:59 +10:00
Julian Sommer 6737c42601 refactor: consolidate NATS/API types into types/ package (phase 1)
Move EBS, cluster, and node message types from config/ to types/.
Remove duplicate NodeDiscoverResponse from gateway and daemon.
Delete dead EC2StartInstances types and tests from config.
Update all imports across source and test files.
2026-03-06 09:56:04 +10:00
Julian Sommer 19988af768 fix: gofmt formatting 2026-03-06 09:48:16 +10:00
Julian Sommer d20fcd2c38 fix merge 2026-03-06 09:45:34 +10:00
Julian Sommer 1c6950b89e release: update go.mod for v0.2.2 2026-03-06 09:45:34 +10:00
Julian Sommer 94ec56bb59 fix: extract shared type 2026-03-06 09:45:34 +10:00
Julian Sommer 0b2e613c6c fix: diff-coverage.sh 2026-03-06 08:22:30 +10:00
Julian Sommer 2d304528b8 chore: bump deps 2026-03-06 08:15:48 +10:00
Julian Sommer 35f2104325 feat: more tests 2026-03-05 15:09:24 +10:00
Julian Sommer c4027a0380 test: add gateway core routing, IAM auth, and handler test coverage
Squash of dev commits: gateway core routing/policy/middleware tests,
IAM auth and handler tests, awserrors tests, respondWithJSON helper,
utils refactor into focused files, and daemon test speedups.
2026-03-05 14:20:00 +10:00
Julian Sommer 77c6d05042 refactor: replace NYTimes/gziphandler with stdlib compress/gzip
Drop the external gziphandler dependency in hiveui. The replacement
uses a simple http.ResponseWriter wrapper that checks Content-Type
against an allowlist before enabling gzip compression. Same behavior,
zero external deps.
2026-03-05 12:53:46 +10:00
Julian Sommer 8af3d2e5fa feat: migrate off fiber (#62) 2026-03-05 12:53:46 +10:00
Julian Sommer 49c5807ef1 refactor: accounts 2026-03-05 11:41:51 +10:00
Julian Sommer 5fad038b87 feat: IAM account scoping (#61) 2026-03-05 09:06:35 +10:00
Julian Sommer 48c0565d47 fix: ovn shut down on stop-dev.sh 2026-03-03 15:12:44 +10:00
Julian Sommer 371193a832 feat: IAM v1 (#59) 2026-03-03 15:23:19 +11:00
Julian Sommer 9d66ef8233 fix: sanitize error codes to prevent leaking internal errors to clients
Add awserrors.ValidErrorCode() to validate error strings against
ErrorLookup before sending to clients. Replace hardcoded
"ValidationError" strings in utils.go with awserrors constants.
2026-02-27 08:49:18 +10:00
Julian Sommer 992111e809 fix: race condition + frontend refetching 2026-02-27 08:14:50 +10:00
Julian Sommer 12c2e2789e fix: skip diff coverage when instrumentable lines below minimum
The MIN_LOC=100 threshold checked total diff lines, not instrumentable
lines. A commit with many non-instrumentable lines (structs, comments)
but only 1-2 instrumentable lines would unfairly fail at 0% coverage.
Add a minimum instrumentable lines threshold (5) so the check skips
when the sample size is too small to be meaningful.
2026-02-26 11:35:22 +10:00
Julian Sommer 3720a5ba90 feat: blockdevicemappings fix (#55) 2026-02-26 11:47:37 +11:00
Julian Sommer de0e049b16 fix: e2e console output 2026-02-26 08:26:34 +10:00
Julian Sommer 6e3fe148af feat: frontend features 2026-02-26 08:22:55 +10:00
Julian Sommer 81b990a238 fix: e2e verify errored instance didnt restart on launch 2026-02-25 11:33:44 +10:00
Julian Sommer 52f2498699 fix: proper recovery checks e2e 2026-02-25 11:16:07 +10:00
Julian Sommer 61e8eb9d91 fix: reset LaunchTime on instance relaunch to prevent watchdog termination
During cluster restart, restoreInstances resets running instances (with
dead QEMU) to pending for relaunch, but didn't reset LaunchTime. The
pending watchdog (5min timeout) saw the stale LaunchTime from the
original launch and immediately marked instances as failed/terminated,
preventing any relaunch from succeeding.
2026-02-25 10:48:16 +10:00
Julian Sommer 1453beca6b Merge remote-tracking branch 'origin/main' into dev 2026-02-25 10:45:45 +10:00
Julian Sommer d0056ee925 fix: daemon-level retry with backoff for NATS/JetStream startup (#52) 2026-02-25 11:43:10 +11:00
Julian Sommer 55486490aa fix: kv stream recovery (#54) 2026-02-25 11:41:28 +11:00
Julian Sommer b370f6336e docs 2026-02-25 10:01:12 +10:00
Julian Sommer 1910347cce fix: make building on start-dev.sh opt in 2026-02-25 09:56:13 +10:00
dependabot[bot] e08b8140a6 chore: bump github.com/gofiber/fiber/v2 from 2.52.11 to 2.52.12 (#51) 2026-02-25 10:36:09 +11:00
dependabot[bot] a7e0cb6972 chore: bump github.com/nats-io/nats-server/v2 (#50) 2026-02-25 10:35:49 +11:00
Julian Sommer 6b9fc7b600 refactor: daemon 2026-02-25 08:22:46 +10:00
Julian Sommer c2e66e1ab4 refactor: extract respondWithError; fix nil panic in RunInstance
Extract duplicated respondWithError closures into a single package-level
function. Guard against nil ImageId in handleEC2RunInstances to prevent
panic under race conditions. Fix diff-coverage base ref for main branch.
2026-02-24 15:02:57 +10:00
Julian Sommer deae613a75 refactor: split daemon_handlers.go into per-domain files
Split the monolithic daemon_handlers.go into focused files:
  - daemon_handlers_account.go
  - daemon_handlers_console.go
  - daemon_handlers_igw.go
  - daemon_handlers_image.go
  - daemon_handlers_instance.go
  - daemon_handlers_key.go
  - daemon_handlers_snapshot.go
  - daemon_handlers_tag.go
  - daemon_handlers_volume.go
  - daemon_handlers_vpc.go

Add tests for ModifyVolume success, TerminateStoppedInstance with volumes,
DescribeInstanceTypes capacity filter, StartStoppedInstance type-not-available,
AttachVolume/DetachVolume validation paths, RunInstances error paths, and
DescribeInstances malformed ID.

Fix diff-coverage.sh to use git rename/copy detection (-M -C) so file
splits are not incorrectly flagged as all-new code.
2026-02-24 14:28:24 +10:00
Julian Sommer 8188707c91 feat: VPC networking v1 — OVN-based virtual networking for EC2 instances 2026-02-24 14:13:57 +10:00
Julian Sommer b9c528ce4c fix frontend vuln 2026-02-24 11:56:17 +10:00
Julian Sommer 5992d3f056 test: general test coverage improvements across hive packages
Coverage improvements:
- awsec2query: parser byte slice, invalid input, locationName fallback (73% → 83%)
- config: EC2StartInstancesResponse.Respond NATS round-trip, NodeBaseDir, ShardWAL
- daemon: 14 NATS delegate handlers, device allocation, canAllocate (count=0 boundary),
  resource management, shutdown ACK/progress, JetStream WriteServiceManifest,
  fix smallestAllocType to score by both vCPU and memory
- gateway: GenerateEC2ErrorResponse XML validation (fixed no-op assertion),
  DiscoverActiveNodes (5 scenarios), ParseAWSQueryArgs, ParseArgsToStruct (41.6% → 52.1%)
- utils: ConnectNATS (success, token, wrong token, bad address), NATSRequest
- vm: ResetNodeLocalState, PIDFile/QMP, NoGraphic, Serial+Console partial configs,
  NetDevs, ARM64, KVM, full config (68.1% → 91.7%)
2026-02-20 15:09:33 +10:00
Julian Sommer 6bc36455fc test coverage improvements: instance, hiveui, health, utils
Total hive coverage: 54.3% → 55.8% (+1.5%)

- instance/service_impl.go: new test file (generateHostname, RunInstance,
  cloud-init template, constructor)
- hiveui.go: new test file (security headers, gzip middleware, New(),
  Status(), Reload())
- daemon/health.go: extend existing tests (handleInstanceCrash core flow,
  first crash time, unknown instance type, maybeRestart scheduling)
- utils/utils.go: extend existing tests (pidPath all branches, RuntimeDir,
  extractDiskImagePath, WritePidFile error path)
- fix: use smallestAllocType in health tests to avoid CI resource failures
2026-02-20 13:46:31 +10:00
Julian Sommer db45c028be test: e2e + unit coverage for ModifyInstanceAttribute and key service
- Add Phase 7b to single-node E2E: stop instance, modify type (nano →
  xlarge), start, SSH in, verify vCPU count (nproc) and memory
  (/proc/meminfo) match the new instance type
- Add Phase 8q negative test: modify-instance-attribute on running
  instance expects InvalidInstanceID.NotFound
- Improve key service integration test coverage (mulga-580)
- Pin viperblock/predastore CI checkout to main (fixes feature branch CI)
- Update TEST_COVERAGE.md
2026-02-20 12:01:03 +10:00
Julian Sommer ff6ca8a057 feat: modify-instance-attribute (#48) 2026-02-20 12:19:20 +11:00
Julian Sommer df0e8a6459 ci: add race detection, fix test race conditions 2026-02-20 09:40:02 +10:00
Julian Sommer 3dc3d41d79 ci: enforce code coverage 2026-02-20 08:58:44 +10:00
Julian Sommer eba702b1f2 test: add 10 viperblockd coverage tests, fix daemon test sleep
viperblockd: add snapshot handler, sync/unmount/delete with VB instance,
invalid JSON, dual-publish, generic topic routing, and auxiliary volume
suffix tests. Coverage 43.1% → 53.6%.

daemon: write clean shutdown marker in restore tests to skip the 3s
crash-recovery sleep (no QEMU runs in tests). Suite 36s → 6s.
2026-02-20 08:42:30 +10:00
Julian Sommer e6ffecafca gofix: exclude stringsbuilder analyzer from modernize checks 2026-02-19 15:15:24 +10:00
Julian Sommer 27a27f660a feat: add go fix check in ci 2026-02-19 14:43:59 +10:00
Julian Sommer b4dcd3ead9 gofix: modernize codebase with go fix analyzers
Apply safe modernizations across the codebase:
- minmax: replace if/else clamping with min/max builtins
- fmtappendf: replace []byte(fmt.Sprintf) with fmt.Appendf
- forvar: remove redundant loop variable re-declarations
- inline: rename reflect.Ptr to reflect.Pointer
- mapsloop: replace explicit map loops with maps package
- omitzero: remove ineffective omitempty on time.Time fields
- rangeint: replace 3-clause for loops with range-over-int
- slicescontains: replace manual loops with slices.Contains
2026-02-19 14:40:16 +10:00
Julian Sommer 43acb57b9f feat: add e2e tests for serial console access and error compliance 2026-02-19 14:23:55 +10:00
Julian Sommer 17c011e9db fix 2026-02-19 13:17:20 +10:00
Julian Sommer a330e35d5f feat: account settings console access (#47) 2026-02-19 13:05:11 +10:00
Julian Sommer a5ff1ca5a7 feat: more awserrors (#46) 2026-02-19 12:15:45 +11:00
Julian Sommer 8573800914 fix: move instances with unavailable type to stopped on recovery (#45) 2026-02-19 10:31:13 +11:00
Julian Sommer cefbcea485 fix: README.md 2026-02-19 08:30:25 +10:00
Julian Sommer 1bf4a9cfab fix: ci branches 2026-02-18 15:05:28 +10:00
Julian Sommer 09a1e142eb release: update go.mod for v0.2.1 2026-02-18 15:01:36 +10:00
Julian Sommer ca14ecf2bc fix: port splitting (#44) 2026-02-18 15:20:40 +11:00
Julian Sommer 0c65d75371 feat: add coordinated cluster shutdown to multi-node E2E tests (#42) 2026-02-18 13:22:12 +11:00
Julian Sommer b5d2c22e68 chore: remove legacy security alias from Makefile (#43) 2026-02-18 12:23:22 +11:00
Julian Sommer 135f02c386 feat: make hive test use dev versions 2026-02-18 09:36:25 +10:00
Ben Duncan 4986e3fb87 fix: disable sharded WAL by default, bump viperblock v1.0.14
Sharded WAL causes incomplete write errors in E2E tests. Disable by
default in hive.toml template, service.go fallback, and viperblock
constructor. Can be re-enabled via hive.toml shardwal=true.
2026-02-18 07:08:11 +10:00
Ben Duncan 20e67051b0 feat: EC2 GetConsoleOutput via QEMU socket chardev + logfile capture
Replace -serial pty with socket chardev + logfile for serial output:
- Console log written to {RuntimeDir}/console-{instanceID}.log
- Serial socket at {RuntimeDir}/serial-{instanceID}.sock for admin access
- Eliminates PTS extraction deadlock (mulga-lfo) — no more ptsChan
- GetConsoleOutput API handler reads log, returns base64-encoded output
- Per-instance NATS topic ec2.{instanceID}.GetConsoleOutput (no fan-out)
- Admin debugging via: socat -,raw,echo=0 UNIX-CONNECT:serial-*.sock
2026-02-18 06:25:43 +10:00
Ben Duncan 5f370ab75e fix: bump viperblock v1.0.13, add local Docker E2E test docs
Update viperblock dependency to v1.0.13 which fixes CreateSnapshot
skipping WAL flush/consolidate when the VB instance doesn't own the
WAL files (viperblockd snapshot scenario).

Add local Docker E2E test instructions to TESTING.md.
2026-02-18 06:25:20 +10:00
Ben Duncan a1a801b26a fix: bump viperblock v1.0.12 — shard WAL snapshot corruption fix
Viperblock v1.0.12 fixes WriteShardedWALToChunk corrupting NBD plugin's
active WAL files when called from a VB instance that doesn't own the WAL
(viperblockd snapshot path). This caused CreateImage to fail with
"checksum mismatch in shard 0" on the second snapshot.
2026-02-17 22:57:36 +10:00
Ben Duncan c2265920eb fix: stop-dev.sh incorrectly detects single-node as multi-node in Docker E2E
is_multinode() counted all [nodes.*] TOML subsections (daemon, nats, awsgw,
etc.) instead of just top-level node entries. A single-node config has ~8
such sections, causing the single-node Docker E2E to use coordinated cluster
shutdown and exit with code 254.

Fix grep to match only [nodes.X] (not [nodes.X.Y]) and preserve the original
test exit code in the cleanup trap.
2026-02-17 22:15:49 +10:00
Ben Duncan bb1fe5e485 feat: add configurable shard WAL toggle, bump viperblock v1.0.11
Add [viperblock] section to hive.toml with shardwal config (default
true). The setting flows from config through viperblockd to nbdkit
plugin args, allowing operators to disable sharded WAL if needed.

Bumps viperblock dependency from v1.0.10 to v1.0.11 which includes
the sharded WAL implementation (UseShardedWAL/OpenShardedWAL), fixing
CI build failures.
2026-02-17 21:48:44 +10:00
Ben Duncan 1b431b29f7 Merge branch 'main' of github.com:mulgadc/hive 2026-02-17 16:14:49 +10:00
Ben Duncan c645fc46b1 fix: dispatch chunk WAL opening for EFI and cloud-init volumes
With UseShardedWAL now defaulting to true, the EFI and cloud-init
volume preparation must open sharded WAL files instead of always
opening the legacy WAL. Without this, Flush() dispatches to the
sharded path but shard DB files are nil, causing zero-byte writes.
2026-02-17 16:14:00 +10:00
Julian Sommer 83b1435c99 fix: refactor instance type detection (#41) 2026-02-17 15:54:45 +11:00
Julian Sommer 7128a22fdb feat: detect CPU generation instead of vendor for instance types (#40) 2026-02-17 14:59:29 +11:00
Julian Sommer b7396b97c0 fix: use available instance types in allocation test to prevent flaky CI
The FilterUnavailableTypesAfterAllocation test picked a random 2-vCPU
type from the raw instance type map, which could include memory-hungry
types (e.g. r5a.large at 16 GB) that exceed host RAM on CI runners
(~15.6 GB). Now picks from the already-filtered available types list.
2026-02-17 13:30:44 +10:00
Julian Sommer 1735a46f11 fix: e2e 2026-02-17 12:32:40 +10:00
Julian Sommer 8d9da69abb fix: e2e 2026-02-17 11:52:31 +10:00
Julian Sommer c6b89e873a feat: more testing 2026-02-17 11:47:56 +10:00
Ben Duncan 88ee6e3171 merge: resolve conflicts, accept remote changes 2026-02-17 11:43:38 +10:00
Julian Sommer dad0df0e33 fix: remove .vscode 2026-02-17 10:50:24 +10:00
Julian Sommer aeb802448b feat: more instance types 2026-02-17 10:39:59 +10:00
Julian Sommer b648fe7390 fix: merge 2026-02-17 10:32:35 +10:00
Ben Duncan 0e47a376c7 merge 2026-02-17 10:29:29 +10:00
Ben Duncan 81bf36bfc1 Moved .claude to internal repo for development 2026-02-17 10:28:21 +10:00
Ben Duncan 93c5b09ac5 Merge branch 'main' of github.com:mulgadc/hive 2026-02-17 09:42:48 +10:00
Ben Duncan dd7bce20e6 Move internal dev docs to private repo, trim CLAUDE.md for public use
Internal development plans, roadmap, and detailed dev process docs moved
to private mulga repo. CLAUDE.md trimmed to remove SSH credentials, node
IPs, and internal deployment workflows. Gosec fixes for file permissions
and unhandled error.
2026-02-17 09:38:28 +10:00
Julian Sommer 2e9ede516a fix: lint errors 2026-02-17 09:37:09 +10:00
Ben Duncan 244cb69715 Merge branch 'main' of github.com:mulgadc/hive 2026-02-17 09:11:37 +10:00
Ben Duncan bb533dfbb6 multi-server tweaks, ec2 health, IAC templates, misc cleanup 2026-02-17 09:10:13 +10:00
Julian Sommer dae9d1bb42 fix: add t3a family + fix ci 2026-02-17 08:56:06 +10:00
Julian Sommer e4db948910 feat: O(1) snapshot volume checks (#38) 2026-02-17 09:23:02 +11:00
Ben Duncan 962d85edf2 Improved IAC automation testing 2026-02-14 19:27:13 +10:00
Ben Duncan 24924be941 Cluster heartbeat support, improved cluster start/stop, coordinated shutdown in multi-server mode 2026-02-14 16:59:32 +10:00
Ben Duncan c5e561abbf Improved cluster stats and INSTALL.md doc 2026-02-14 16:19:25 +10:00
Ben Duncan 774655eff0 hive top, hive get cmds for cluster health and stats. improved docs for iac, cluster, multi-server restart 2026-02-14 15:38:03 +10:00
Ben Duncan 5863aeed13 Phase 0/1 multi-server-restart enhancements 2026-02-14 13:23:24 +10:00
Ben Duncan febeae8c47 preflight fixes for gh errors 2026-02-14 09:23:42 +10:00
Ben Duncan 3095a90090 e2e tests, multi server improvements, doc updates for workflow 2026-02-14 08:50:24 +10:00
Ben Duncan d80dd0996a Fixed gosec and added go skills 2026-02-14 06:54:00 +10:00
Ben Duncan 25d98dda7e Consolidating documentation 2026-02-13 21:00:22 +10:00
Ben Duncan 6df221309d Merge branch 'main' of github.com:mulgadc/hive 2026-02-13 20:32:08 +10:00
Ben Duncan db0d46ca5b Improved multi-server node testing and tweaks 2026-02-13 20:31:45 +10:00
Ben Duncan cf12a3272e IAC for Proxmox / AWS terraform examples 2026-02-13 11:42:43 +10:00
Julian Sommer 8774fb1e95 feat: distributed start instances (#37) 2026-02-12 15:15:05 +11:00
Julian Sommer fb361dfe49 fix: update instance status BEFORE calling it 2026-02-12 11:11:59 +10:00
Julian Sommer 9d3747bf26 feat: ssh tests in e2e (#36) 2026-02-12 11:57:31 +11:00
Ben Duncan 5b4bad3cb3 Merge branch 'main' of github.com:mulgadc/hive 2026-02-12 10:50:35 +10:00
Ben Duncan 6c4caec652 instance state 2026-02-12 10:37:45 +10:00
Julian Sommer 22af6c4a46 feat: more e2e (#35) 2026-02-12 11:00:00 +11:00
Julian Sommer 815e9a8c36 fix: duplicate subscription 2026-02-12 09:03:24 +10:00
Julian Sommer c7d832a823 fix: nats topic 2026-02-12 08:30:23 +10:00
Julian Sommer 32e2f7bf78 fix: daemon reboot 2026-02-11 16:05:21 +10:00
Julian Sommer 202fffc8db chore: utils tests 2026-02-11 15:21:29 +10:00
Julian Sommer 72709dda12 linter fix 2026-02-11 15:15:30 +10:00
Julian Sommer 0ff03be0dd fix 2026-02-11 15:10:55 +10:00
Julian Sommer 998649cbab feat: createImage (#34) 2026-02-11 16:07:19 +11:00
Julian Sommer efe07a9ed4 chore: bump go version 2026-02-11 12:59:32 +10:00
Julian Sommer 27530ba270 chore: daemon tests 2026-02-11 12:48:21 +10:00
Julian Sommer 467eb0252d chore: more gateway tests 2026-02-11 12:24:00 +10:00
Julian Sommer 2d30743ba5 chore: gateway tests 2026-02-11 12:12:40 +10:00
Julian Sommer 10e65ab66a fix: update ci triggers 2026-02-11 11:44:36 +10:00
Julian Sommer c514d6b937 format 2026-02-11 11:37:52 +10:00
Julian Sommer df70750f09 chore: more tests 2026-02-11 11:37:17 +10:00
Julian Sommer 210ca92b2c fix: dead code 2026-02-11 11:09:00 +10:00
Julian Sommer 5ce3773c42 fix: cryptographically secure ids 2026-02-11 10:51:41 +10:00
Julian Sommer 39b9e0dcd1 fix: split daemon ec2 paths 2026-02-11 10:32:32 +10:00
Julian Sommer b1762f445c fix: centralise nats connection logic 2026-02-11 10:18:30 +10:00
Julian Sommer 221a1ef258 fix: use typed aws errors 2026-02-11 10:11:37 +10:00
Julian Sommer e95c2536da feat: EIGW (#33) 2026-02-11 11:00:53 +11:00
Julian Sommer 2937c8f425 feat: account functionality (#27) 2026-02-11 10:58:52 +11:00
Julian Sommer 1b424928f4 fix: move matrix out of TODO.md 2026-02-11 09:49:22 +10:00
Julian Sommer 7a0d148e71 fix: update Makefile 2026-02-11 09:44:36 +10:00
Julian Sommer 6b4728e702 fix: update CLAUDE.md 2026-02-11 09:43:55 +10:00
Julian Sommer 63e476323d fix: url decode keys 2026-02-11 08:27:02 +10:00
Julian Sommer 6abae11122 feat: real snapshots (#32) 2026-02-10 16:57:50 +11:00
Julian Sommer 8d3808c364 fix: parallelize ci 2026-02-10 10:47:07 +10:00
Julian Sommer 51ffc4c3ab fix: add missing error handling 2026-02-10 10:38:36 +10:00
Julian Sommer b443bbd3e2 fix: permissions 2026-02-10 10:10:39 +10:00
Julian Sommer e545e81e59 fix: run modernize again 2026-02-10 10:02:06 +10:00
Julian Sommer 2491c6f2c2 feat: snapshot e2e (#31) 2026-02-10 10:54:48 +11:00
Julian Sommer 59c57cc074 chore: run modernize 2026-02-10 09:23:25 +10:00
Julian Sommer c5bfedcd98 chore: fmt 2026-02-10 09:10:42 +10:00
Julian Sommer 05a608855c fix: add compile time interface checks 2026-02-10 08:55:23 +10:00
Julian Sommer 513fd07ced feat: snapshots (#30) 2026-02-10 09:45:53 +11:00
Julian Sommer 2977b03cef feat: tags e2e (#29) 2026-02-09 16:28:58 +11:00
Julian Sommer 560e5cb58c feat: tags (#28) 2026-02-09 15:49:59 +11:00
Julian Sommer 09f78fe29b update TODO.md 2026-02-09 11:53:51 +10:00
Julian Sommer f0323bc081 fix: delete dead code (unused handlers) 2026-02-09 11:52:12 +10:00
Julian Sommer dcb08c0bd1 fix: e2e predastore multi node 2026-02-09 11:17:00 +10:00
Julian Sommer 9ba912b745 fix: duplicated unmarshal 2026-02-09 11:14:38 +10:00
Julian Sommer a554b9b884 fix: state change 2026-02-09 10:51:21 +10:00
Julian Sommer e15c077d3a fix: simplify nats handlers 2026-02-09 10:42:47 +10:00
Julian Sommer 88f07a9f3a simplify gateway more 2026-02-09 10:34:12 +10:00
Julian Sommer 45a466f71d fix: viperblock init 2026-02-09 10:03:06 +10:00
Julian Sommer e5038d4a35 fix: deduplicate nilchecks 2026-02-09 09:56:11 +10:00
Julian Sommer 25dd435bc6 fix: use object store instead of s3 client 2026-02-09 09:52:12 +10:00
Julian Sommer daf5f0c780 fix: refactor simple daemon handlers 2026-02-09 09:43:57 +10:00
Julian Sommer e0fea35b19 fix: refactor gateway 2026-02-09 09:35:07 +10:00
Julian Sommer e31e523380 feat: describe-account-attributes (#26) 2026-02-09 10:19:33 +11:00
Julian Sommer 0efa570cc5 fix: update to go 1.25.7 2026-02-09 08:52:32 +10:00
Julian Sommer ef87b41858 feat: describe-availbility-zones (#25) 2026-02-09 09:47:20 +11:00
Julian Sommer 1c2174a10f fix: refactor handlers to use ObjectStore interface (#24) 2026-02-04 16:32:11 +11:00
Julian Sommer 801f0f8135 cleanup dead code 2026-02-04 14:02:48 +10:00
Julian Sommer d0dfea56e8 cleanup comments 2026-02-04 13:54:43 +10:00
Julian Sommer 2566d9d528 make gateway consistent 2026-02-04 13:35:03 +10:00
Julian Sommer a3d7e20e31 feat: describe-volume-status (#23) 2026-02-04 12:46:06 +11:00
Julian Sommer ba990ffd97 feat: attach and detach volume e2e (#22) 2026-02-04 11:39:24 +11:00
Julian Sommer d81e417410 feat: detach-volumes (#21) 2026-02-04 10:54:02 +11:00
Julian Sommer 45ac7c4357 update docs 2026-02-04 08:48:10 +10:00
Julian Sommer fca82939f5 feat: attach-volume (#20) 2026-02-04 09:44:54 +11:00
Julian Sommer 65ad19e604 bump viperblock 2026-02-03 14:00:10 +10:00
Julian Sommer d14158a619 feat: delete on termination 2026-02-03 13:55:19 +10:00
Julian Sommer 3ab1a839db fix: fallback 2026-02-03 12:22:38 +10:00
Julian Sommer 10ded4608b fix: frontend error handling 2026-02-03 12:19:39 +10:00
Julian Sommer 2de2899724 fix: accessibility 2026-02-03 12:05:23 +10:00
Julian Sommer 130e4b9662 feat: create-bucket frontend 2026-02-03 11:54:14 +10:00
Julian Sommer 941aaa6504 move claude rules and skills to root 2026-02-03 11:31:11 +10:00
Julian Sommer 917b66beb9 feat: predastore multinode (#19) 2026-02-03 12:07:06 +11:00
Julian Sommer fdb5cca77e feat: delete-volumes (#18) 2026-02-03 10:47:00 +11:00
Julian Sommer b772b2e0bf feat: use state machine for daemon (#16) 2026-02-02 16:14:36 +11:00
Julian Sommer 875328cfef feat: create-volume (#17) 2026-02-02 15:47:46 +11:00
Julian Sommer 6baf49885b e2e: add modify-volumes (#15) 2026-02-02 14:22:55 +11:00
Julian Sommer 6067f0ccd4 tweak 2026-02-02 11:30:05 +10:00
Julian Sommer e4a2e2972a fix: split daemon handlers 2026-02-02 11:14:35 +10:00
Julian Sommer 731ba4fbe8 feat: refactor Start() daemon 2026-02-02 10:51:16 +10:00
Julian Sommer 7e7d358135 update viperblock version 2026-02-02 10:39:24 +10:00
Julian Sommer ad763eb41d feat: add attach time to volume 2026-02-02 10:34:10 +10:00
Julian Sommer 1a8e273ac6 fix: use slog instead of log 2026-02-02 10:18:04 +10:00
Julian Sommer 91384856ec fix: delete on termination false 2026-02-02 10:12:09 +10:00
Julian Sommer 0e2b715d60 fix: form error handling 2026-02-02 10:08:55 +10:00
Julian Sommer f45a106dda feat: resize volume frontend 2026-02-02 09:36:49 +10:00
Julian Sommer 7aa688e564 fix: table based NATS subscriptions 2026-02-02 08:59:30 +10:00
Julian Sommer 50a0a2e1f1 fix: dead code 2026-02-02 08:35:19 +10:00
Julian Sommer 7f39da8718 fix: state on launch 2026-02-02 08:14:40 +10:00
Julian Sommer d1d6e6d004 todo.md 2026-01-30 15:07:53 +10:00
Julian Sommer 6d24ec2111 feat: modify-volume (resize) (#14) 2026-01-30 16:06:45 +11:00
Julian Sommer 230335adca add rules for claude 2026-01-30 10:59:07 +10:00
Julian Sommer 0b261743d2 fix: use aws-sdk for uploads 2026-01-30 09:38:11 +10:00
Julian Sommer f04d553a02 fix: multipart upload frontend 2026-01-30 08:59:15 +10:00
Julian Sommer dfd36370b0 tweak 2026-01-30 08:18:57 +10:00
benduncan 2fae29d486 Go version bump 2026-01-29 20:23:58 +10:00
benduncan 98701e5208 Install.md tweaks 2026-01-29 20:18:33 +10:00
benduncan 3232d45ff8 Merge branch 'main' of github.com:mulgadc/hive 2026-01-29 20:13:00 +10:00
benduncan d6bfef02ac Latest todo 2026-01-29 20:12:55 +10:00
Julian Sommer 5b55e1d1e0 chore: cut CLAUDE.md down more 2026-01-29 15:55:25 +10:00
Julian Sommer 2aaed28404 chore: trim CLAUDE.md 2026-01-29 15:52:15 +10:00
Julian Sommer e92d67caf0 fix: add default values to run-instances 2026-01-29 15:18:06 +10:00
Julian Sommer 0b1c44d908 fix: volume naming 2026-01-29 15:03:43 +10:00
Julian Sommer 5e53ab1f7e fix: lint issue 2026-01-29 14:14:20 +10:00
Julian Sommer 3bca0e69df feat: block device mappings 2026-01-29 14:10:04 +10:00
Julian Sommer 20de0a7989 feat: count support for run-instances 2026-01-29 12:19:18 +10:00
Julian Sommer 118fb35a1e add hive-ui to start-dev 2026-01-29 11:00:08 +10:00
Julian Sommer dc77e2547b fix: INSTALL.md 2026-01-29 10:46:55 +10:00
Julian Sommer 936828ed53 fix: upgrade to go 1.25.6 to fix vuln 2026-01-29 09:26:27 +10:00
Julian Sommer 454e8dcace revert daemon.go (#13) 2026-01-29 10:22:56 +11:00
Julian Sommer 785d2d6c7b feat: commit frontend build so node isnt a dep for building hive 2026-01-29 09:15:09 +10:00
Julian 5cd5189b56 chore: add logs 2026-01-28 15:49:50 +10:00
Julian a4e7b71d6c fix: e2e verify ssl 2026-01-28 15:36:11 +10:00
Julian adfc54a7f0 fix: check nats sub 2026-01-28 15:26:11 +10:00
Julian 82fa85f992 fix: build first 2026-01-28 15:07:56 +10:00
Julian Sommer de0e9a915e fix: backoffs for jetstream (#12) 2026-01-28 16:00:54 +11:00
Julian 1d816721e8 feat: move hive-ui into main hive binary 2026-01-28 15:00:30 +10:00
Julian 84c0b3e715 fix: remove ssh tests 2026-01-28 14:36:10 +10:00
Julian a423de3380 fix: actual fix 2026-01-28 13:45:13 +10:00
Julian 21a8e2b004 fix: jetstream 2026-01-28 13:36:47 +10:00
Julian 04197d1adc fix: instance-state 2026-01-28 13:19:47 +10:00
Julian bd0c12d45e fix: add t3 and t4g back 2026-01-28 13:15:59 +10:00
Julian a3a32d37c3 fix: SigV4AuthMiddleware 2026-01-28 13:00:02 +10:00
Julian 07056f42a9 fix: pathnames in hive-ui 2026-01-28 11:06:08 +10:00
Julian 528cbc0acd fix: buckets on hive-ui 2026-01-28 10:36:54 +10:00
Julian Sommer 8f0e8d7ead feat: multi-node certs (#11) 2026-01-28 11:36:09 +11:00
Ben Duncan 772da5a181 Updated todo for volumes 2026-01-28 10:26:34 +10:00
Ben Duncan 3912a2a84d Merge branch 'main' of github.com:mulgadc/hive 2026-01-28 10:20:20 +10:00
Ben Duncan bf99270095 EC2 volume todo 2026-01-28 10:20:02 +10:00
Julian 11c1d74a79 embed frontend into binary 2026-01-28 10:15:34 +10:00
Julian 7dca292e67 fix: web-ui to use hive certs 2026-01-28 09:55:01 +10:00
Ben Duncan 1bfb07c643 Fresh install tweaks 2026-01-27 21:54:10 +10:00
Ben Duncan f57bc9f2e6 Added cert authority feature for init 2026-01-27 17:08:46 +10:00
Ben Duncan 2252764da7 Latest changes for fresh install 2026-01-27 16:16:03 +10:00
Ben Duncan 36847bbf54 Merge branch 'main' of github.com:mulgadc/hive 2026-01-27 16:11:02 +10:00
Ben Duncan 7f8bbf9b31 build tweak for quickinstall 2026-01-27 16:10:52 +10:00
Julian Sommer 52b8550788 feat: hive-ui (#10) 2026-01-27 17:05:08 +11:00
Ben Duncan fc52284ec5 Installer and Makefile tweaks for fresh install 2026-01-27 16:04:53 +10:00
Ben Duncan b7bcdcaf1c Bumped viperblock version 2026-01-27 15:29:48 +10:00
Ben Duncan bd73a602af Multi-node predastore default config 2026-01-27 15:06:30 +10:00
Ben Duncan 2156e5c591 Merging latest 2026-01-27 15:04:58 +10:00
Ben Duncan ed75a63e15 Viperblock update 2026-01-27 14:29:12 +10:00
Julian 6067450684 go mod tidy 2026-01-27 14:18:37 +10:00
Julian 67b9ffa423 predastore 1.0 2026-01-27 14:13:51 +10:00
Julian Sommer 5878669eb0 fix for predastore multi node (#9) 2026-01-27 13:43:01 +11:00
Julian Sommer 7b7be2c26b feat: multi-node e2e test v2 (#8) 2026-01-27 10:59:58 +11:00
Ben Duncan 6c6909ad1c multi-node predastore config 2026-01-27 08:19:29 +10:00
Ben Duncan 83fd425a60 Latest changes 2026-01-27 08:17:06 +10:00
Ben Duncan 8280cb7102 merge latest 2026-01-25 07:08:17 +10:00
Ben Duncan 7f5b53c5df v1 support for distributed backend 2026-01-25 07:06:53 +10:00
Julian 0138b7c970 move image download to before deps layer 2026-01-23 11:26:33 +10:00
Julian 6a87b62e06 feat: cache Ubuntu image in e2e Docker build 2026-01-23 11:23:07 +10:00
Julian b1ec9b2122 upgrade viperblock 2026-01-23 09:22:12 +10:00
Julian 0a4e0858f1 remove replace directive 2026-01-23 09:04:48 +10:00
Julian Sommer 31158932b8 fix: run-instances in e2e 2026-01-23 09:49:28 +11:00
Julian 31f180c144 fix: stop-instances edge case 2026-01-23 08:23:01 +10:00
Ben Duncan d64e03c2f9 Merge branch 'main' of github.com:mulgadc/hive 2026-01-22 19:53:00 +10:00
Julian Sommer 6677b741e1 fix: stop-instances 2026-01-22 04:46:25 +00:00
Julian Sommer bfb909b365 feat: jetstream kv for instance state 2026-01-22 04:08:46 +00:00
Julian Sommer ca3427d64b feat: E2E tests in GitHub Actions 2026-01-22 12:37:45 +11:00
Julian Sommer d29069db83 fix: match new viperblock err type 2026-01-21 04:02:20 +00:00
Julian Sommer 72b706a00d remove paths in action 2026-01-21 01:48:31 +00:00
Julian Sommer 217fcf221a chore: use go tool for linters 2026-01-21 01:34:04 +00:00
Julian Sommer e8e9b72cd3 feat: e2e test 2026-01-20 04:40:38 +00:00
Julian Sommer 896b70e508 feat: describe-volumes 2026-01-20 13:07:16 +11:00
Ben Duncan 302c893340 Merge branch 'main' of github.com:mulgadc/hive 2026-01-19 14:09:32 +10:00
Ben Duncan e3fae3d96f Latest predastore changes for multi-node 2026-01-19 14:08:18 +10:00
Julian Sommer 43fdb3f373 pin viperblock to v0.1.3 2026-01-16 05:44:06 +00:00
Julian Sommer 0c00a6ce99 add go.mod and go.sum to action 2026-01-16 05:34:51 +00:00
Julian Sommer 049ae6f66b pin viperblock to specific commit until we release 2026-01-16 05:32:34 +00:00
Julian Sommer f056dff083 fix 2026-01-16 05:25:04 +00:00
Julian Sommer 3bc8ee7538 fix type for new viperblock 2026-01-16 05:19:50 +00:00
Julian Sommer 824faa6202 safe int conversions 2026-01-16 04:00:06 +00:00
Julian Sommer 4a3f158977 run linters in ci 2026-01-16 03:45:28 +00:00
Julian Sommer a074cafc51 (feat) add expectedNodes to avoid nats timeout 2026-01-14 05:18:08 +00:00
Julian Sommer aeaaba2bca enable greentea gc 2026-01-14 04:33:47 +00:00
Julian Sommer 35db549d7e (feat) describe-regions 2026-01-14 03:48:43 +00:00
Julian Sommer 0c164a4807 (fix) Daemon resource allocation 2026-01-14 13:06:09 +11:00
Julian Sommer 4ae360acf0 Feat/describe instance types 2026-01-13 15:18:31 +11:00
Julian Sommer c37f7ea279 add cors 2026-01-12 02:05:30 +00:00
benduncan 92fd276d34 TODO updated for next ec2 implementation 2026-01-09 15:02:21 +10:00
Julian Sommer d5aa6908a7 unique key id 2026-01-09 02:53:20 +00:00
Julian Sommer e6476ba3b0 minify xml 2026-01-08 03:32:36 +00:00
Julian Sommer 907db178de fix typo 2026-01-08 09:58:29 +11:00
Julian Sommer b9b8c65e79 (fix) Linter fixes
* Fix go vet mutex copy issues and remove report files from git

* remove redundant nil checks (len of nil slice is 0)

* handle error

* add missing error handling

* run modernize to change interface{} to any

* run other modernise commands

* use crypto instead of rand.Intn
2026-01-06 14:20:56 +11:00
Ben Duncan 66ad247c34 Improved formatting 2026-01-06 12:07:07 +10:00
Ben Duncan c42a9d8321 Merge branch 'main' of github.com:mulgadc/hive 2026-01-06 11:57:54 +10:00
Ben Duncan 1c88eaf253 Updated latest todo 2026-01-06 11:57:05 +10:00
Julian Sommer 11983b8095 hide debug log 2026-01-06 00:47:36 +00:00
Julian Sommer 4b593029aa fix ubuntu version 2026-01-06 11:22:56 +11:00
Ben Duncan f5d954d86b Updated design doc 2026-01-06 10:00:07 +10:00
benduncan 52d17d61b7 go build tweaks 2025-12-15 16:08:58 +10:00
benduncan a5934ae2c6 go build tweaks 2025-12-15 15:54:08 +10:00
benduncan a23677604a go build tweaks 2025-12-15 15:49:07 +10:00
benduncan 3645523e59 Merge branch 'main' of github.com:mulgadc/hive 2025-12-15 15:40:56 +10:00
benduncan 36cc7b2515 Installer tweaks 2025-12-15 15:39:25 +10:00
Julian Sommer 49351d2fc2 upgrade go version to fix cert vuln 2025-12-09 11:27:20 +11:00
Julian Sommer 0b90abda64 add repo checkout false to govulncheck 2025-12-09 11:22:41 +11:00
benduncan 9f20664ece Makefile tweaks for security tests 2025-11-27 19:59:15 +11:00
benduncan 35769524e3 [SEC] Updated vul import libs. Adding automation for security checks on commit 2025-11-27 17:09:10 +11:00
benduncan 58d75102ca Added EFI for ARM, RunInstances tweak for provisining disks after instance response 2025-11-25 19:49:22 +11:00
benduncan 243463be2b Set correct machine type for ARM 2025-11-25 16:50:27 +11:00
benduncan 803a3d3b35 Fixed pts 0 issue, tweaked install docs, and unit tests on MacOS 2025-11-25 14:49:52 +11:00
Ben Duncan a2552c57f3 Merge pull request #1 from mulgadc/install-tweaks
docs: hive install tweaks
2025-11-25 13:36:41 +11:00
Julian Sommer 36e467fb8d feat: add workflow_dispatch to action 2025-11-25 11:21:38 +11:00
Julian Sommer d2f72a7a1d docs: hive install tweaks 2025-11-25 11:12:59 +11:00
benduncan fb4da120ea Installer tweaks 2025-11-25 09:10:52 +11:00
Ben Duncan f2a04dd4c2 Added multi-node support and updated INSTALLER. Fix vm.go bug on qemu exec with args error 2025-11-21 23:06:14 +11:00
Ben Duncan a64874e477 Tweaked install instructions 2025-11-18 11:28:04 +08:00
Ben Duncan 9fec764945 Trigger github action for fresh build 2025-11-18 10:12:08 +08:00
Ben Duncan b6faa2c8bf Multi-node details 2025-11-18 10:10:09 +08:00
Ben Duncan b79eaeeed2 Move go.mod replace to go.work 2025-11-18 10:06:54 +08:00
Ben Duncan 6c4953c72e Fixed UT mistakes 2025-11-18 10:02:09 +08:00
Ben Duncan ff22a08948 Added support for multi-node config settings 2025-11-16 21:20:08 +08:00
Ben Duncan bacd1f3ba2 Tweaked qemu launch for non KVM hosts for development 2025-11-15 11:15:03 +00:00
Ben Duncan 42b3ba7fa1 ImportKeyPair bug fix on URL encode base64 encoding 2025-11-15 17:58:27 +08:00
Ben Duncan f8d469220d Fresh install tweaks 2025-11-14 21:42:09 +08:00
557 changed files with 93717 additions and 10943 deletions
+80
View File
@@ -0,0 +1,80 @@
name: Hive E2E Test
on:
push:
branches:
- main
- dev
paths:
- "**/*.go"
- go.mod
- go.sum
- Makefile
- tests/e2e/**
- scripts/**
- .github/workflows/e2e.yml
workflow_dispatch:
jobs:
build_e2e_image:
name: Build E2E Image
runs-on: self-hosted
steps:
- name: Checkout Code
uses: actions/checkout@v6
with:
path: hive
- name: Clone Dependencies
run: ./hive/scripts/clone-deps.sh
- name: Build E2E Image
run: |
docker build \
-t hive-e2e:latest \
-f hive/tests/e2e/Dockerfile.e2e \
.
hive_e2e_test:
name: Hive Single-Node E2E Test
runs-on: self-hosted
needs: build_e2e_image
steps:
- name: Run E2E Tests
run: |
docker run --privileged --rm \
-v /dev/kvm:/dev/kvm \
--name hive-e2e-test \
hive-e2e:latest
hive_multinode_e2e_test:
name: Hive Multi-Node E2E Test
runs-on: self-hosted
needs: build_e2e_image
steps:
- name: Run Multi-Node E2E Tests
run: |
docker run --privileged --rm \
-v /dev/kvm:/dev/kvm \
--cap-add=NET_ADMIN \
--name hive-multinode-e2e-test \
hive-e2e:latest \
./tests/e2e/run-multinode-e2e.sh
cleanup:
name: Cleanup
runs-on: self-hosted
needs: [hive_e2e_test, hive_multinode_e2e_test]
if: always()
steps:
- name: Remove Stale Containers
run: |
docker rm -f hive-e2e-test 2>/dev/null || true
docker rm -f hive-multinode-e2e-test 2>/dev/null || true
- name: Prune Docker Images
run: docker image prune -f
+122 -12
View File
@@ -1,23 +1,133 @@
name: Build and Test Hive
name: Hive Build and Test
on:
push:
branches: ["main"]
paths: ["hive/**", ".github/workflows/**"]
branches:
- main
- dev
paths:
- "**/*.go"
- go.mod
- go.sum
- Makefile
- .github/workflows/hive.yml
workflow_dispatch:
jobs:
hive_test_and_push:
runs-on: ubuntu-latest
build_and_test:
name: Hive Build and Test
runs-on: ubuntu-24.04
steps:
- name: Checkout Code
uses: actions/checkout@v4
- name: Checkout Hive
uses: actions/checkout@v6
with:
path: hive
fetch-depth: 2
- name: Checkout Viperblock
uses: actions/checkout@v6
with:
repository: mulgadc/viperblock
path: viperblock
- name: Checkout Predastore
uses: actions/checkout@v6
with:
repository: mulgadc/predastore
path: predastore
- name: Setup Go
uses: actions/setup-go@v5
uses: actions/setup-go@v6
with:
go-version-file: "go.mod"
cache-dependency-path: "go.sum"
cache-dependency-path: "hive/go.sum"
go-version-file: "hive/go.mod"
- name: Run Tests
run: make test
- name: Setup Go Workspace
run: go work init ./hive ./viperblock ./predastore
- name: Build Hive
run: make -C hive build
- name: Run Tests with Coverage
run: make -C hive test-cover
- name: Check Diff Coverage
run: cd hive && scripts/diff-coverage.sh coverage.out --base HEAD~1
race_detection:
name: Hive Race Detection
runs-on: ubuntu-24.04
steps:
- name: Checkout Hive
uses: actions/checkout@v6
with:
path: hive
- name: Checkout Viperblock
uses: actions/checkout@v6
with:
repository: mulgadc/viperblock
path: viperblock
- name: Checkout Predastore
uses: actions/checkout@v6
with:
repository: mulgadc/predastore
path: predastore
- name: Setup Go
uses: actions/setup-go@v6
with:
cache-dependency-path: "hive/go.sum"
go-version-file: "hive/go.mod"
- name: Setup Go Workspace
run: go work init ./hive ./viperblock ./predastore
- name: Run Tests with Race Detector
run: make -C hive test-race
lint_and_security:
name: Hive Lint and Security
runs-on: ubuntu-24.04
steps:
- name: Checkout Hive
uses: actions/checkout@v6
with:
path: hive
- name: Checkout Viperblock
uses: actions/checkout@v6
with:
repository: mulgadc/viperblock
path: viperblock
- name: Checkout Predastore
uses: actions/checkout@v6
with:
repository: mulgadc/predastore
path: predastore
- name: Setup Go
uses: actions/setup-go@v6
with:
cache-dependency-path: "hive/go.sum"
go-version-file: "hive/go.mod"
- name: Setup Go Workspace
run: go work init ./hive ./viperblock ./predastore
- name: Check Format
run: make -C hive check-format
- name: Check Modernize
run: make -C hive check-modernize
- name: Go Vet
run: make -C hive vet
- name: Security Checks
run: make -C hive security-check
+46
View File
@@ -0,0 +1,46 @@
name: UI Build and Test
on:
push:
branches:
- main
- dev
paths:
- hive/services/hiveui/frontend/**
- .github/workflows/ui.yml
workflow_dispatch:
jobs:
ui_build_and_test:
name: UI Build and Test
runs-on: ubuntu-24.04
defaults:
run:
working-directory: hive/services/hiveui/frontend
steps:
- name: Checkout Code
uses: actions/checkout@v6
with:
sparse-checkout: hive/services/hiveui/frontend
- name: Setup pnpm
uses: pnpm/action-setup@v4
with:
package_json_file: hive/services/hiveui/frontend/package.json
- name: Setup Node.js
uses: actions/setup-node@v6
with:
cache: "pnpm"
cache-dependency-path: hive/services/hiveui/frontend/pnpm-lock.yaml
node-version-file: hive/services/hiveui/frontend/.nvmrc
- name: Install Dependencies
run: pnpm install --frozen-lockfile
- name: Run Biome Check
run: pnpm check
- name: Build Frontend
run: pnpm build
+36 -2
View File
@@ -1,7 +1,41 @@
.vscode
# Terraform/OpenTofu
.terraform/
.terraform.lock.hcl
*.tfstate
*.tfstate.*
*.tfvars
!example.tfvars
scripts/iac/proxmox/clusters/
# Go workspace
go.work
go.work.sum
bin/hive
# Build outputs
bin/
tmp/
*.test
*.out
# Config
config/dev.yaml
# OS files
.DS_Store
Thumbs.db
# Environment
.env
# Test reports
*-report.txt
# Internal docs (moved to private mulga repo)
docs/INTERNAL.md
docs/PRIVATE.md
docs/DEVNOTES.md
docs/HIVE_DEVELOPMENT_PLAN.md
docs/PLAN.md
docs/TODO.md
docs/development/
docs/internal/
-214
View File
@@ -1,214 +0,0 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
This is the Hive platform - an AWS-compatible infrastructure stack for bare-metal, edge, and on-premise deployments. Hive orchestrates multiple independent components that work together to provide AWS-compatible services.
## Component Repositories
The Hive platform consists of these independent repositories:
- **[viperblock](../viperblock/)** - High-performance block storage service (EBS alternative) with WAL-backed storage
- **[predastore](../predastore/)** - S3-compatible object storage service with Reed-Solomon erasure coding
- **nbdkit/** - Network Block Device (NBD) server for exporting block devices (upstream project)
- **hive/** (this repo) - VM orchestration service and platform coordinator
Each component can be used independently or as part of the integrated Hive stack. See component-specific `CLAUDE.md` files for detailed development guidance:
- `../viperblock/CLAUDE.md` - Viperblock development guide
- `../predastore/CLAUDE.md` - Predastore development guide
## Cross-Repository Development Setup
For platform development, clone all repositories to the same parent directory:
```bash
mkdir mulga
cd mulga
git clone <hive-repo-url> hive
git clone <viperblock-repo-url> viperblock
git clone <predastore-repo-url> predastore
```
The `hive/go.mod` uses local replace directives for cross-component development:
```go
replace github.com/mulgadc/viperblock => ../viperblock
replace github.com/mulgadc/predastore => ../predastore
```
## Build Commands
### Hive (Platform Orchestrator)
```bash
make build # Build the hive binary
make test # Run Go tests
make bench # Run benchmarks
make clean # Clean build artifacts
```
### **MANDATORY: Unit Testing Requirements**
**⚠️ CRITICAL: Always run unit tests before any commit or push operation.**
```bash
# REQUIRED before any git commit or push
make test # Must pass with zero failures
# Example workflow:
make test # Verify all tests pass
git add . # Stage changes
git commit -m "..." # Only after tests pass
git push origin main # Only after tests pass
```
**Testing Policy:**
- **All unit tests MUST pass** before committing changes
- **No exceptions** - failing tests block commits
- Tests must complete without errors or panics
- Use `make test` command to run the full test suite
- If tests fail, fix issues before proceeding with git operations
### Component Build Commands
For detailed build instructions, see component-specific CLAUDE.md files:
**Viperblock** (`../viperblock/CLAUDE.md`):
```bash
cd ../viperblock && make build # Build sfs, vblock binaries + NBD plugin
```
**Predastore** (`../predastore/CLAUDE.md`):
```bash
cd ../predastore && make build # Build s3d binary
```
**NBDkit** (system dependency):
```bash
cd nbdkit && autoreconf -i && ./configure && make && make install
```
## Architecture Notes
### Message-Driven Microservices Architecture
Hive uses a **scalable message-driven architecture** that separates API handling from request processing:
```
AWS SDK (custom endpoint) → Hive API Gateway → NATS Topics → Specialized Daemons
↑ ↓
AWS-compatible response ← NATS Topics ← Processing Results
```
#### Core Components:
1. **Hive API Gateway Service** - TLS endpoint that handles AWS SDK requests
2. **Specialized Daemon Services** - Horizontally scalable workers for each AWS service type
3. **NATS Message Broker** - Topic-based routing with queue groups for load balancing
#### Current Implementation (daemon.go):
- **Gateway Logic**: `StartAWSGateway()` handles AWS API calls on port 9999
- **NATS Integration**: Uses request-response pattern: `d.natsConn.Request("ec2.runinstances", data, 30s)`
- **Queue Groups**: Load balancing via `"hive-workers"` queue group
- **Service Coordination**: All inter-service communication via NATS topics
### Inter-component Dependencies
- **Service Startup Order**: NATS → Predastore → Viperblock → NBDkit → Gateway → Daemons
- **Message Flow**: Gateway translates AWS calls to NATS messages, daemons process and respond
- **Scaling**: Multiple daemon instances can subscribe to same topics for horizontal scaling
- All Go projects use local module replacements for development
### Storage Backend Integration
- **Viperblock Integration**: EBS volumes mounted via NATS messaging (`ebs.mount` topic)
- **Predastore Integration**: S3 operations proxied through dedicated S3 daemons
- **NBDkit Integration**: Block device access for VM storage
- **Volume Lifecycle**: Coordinated through NATS between EC2 and EBS daemons
### Key Design Patterns
- **Message-Driven**: All service communication via NATS topics (ec2.*, ebs.*, s3.*, vpc.*)
- **Queue Groups**: Load balancing and fault tolerance via NATS queue groups
- **Request-Response**: 30-second timeout pattern for AWS API compatibility
- **Service Specialization**: Dedicated daemon types for each AWS service
- **Horizontal Scaling**: Multiple daemon instances handle same topic types
### NATS Topic Structure
The system uses structured NATS topics for service communication:
```bash
# EC2 Service Topics:
ec2.runinstances # Launch new instances
ec2.describeinstances # Query instance status
ec2.startinstances # Start stopped instances
ec2.stopinstances # Stop running instances
ec2.terminateinstances # Terminate instances
# EBS Service Topics:
ebs.createvolume # Create new EBS volume
ebs.attachvolume # Attach volume to instance
ebs.detachvolume # Detach volume
ebs.describevolumes # List volumes
ebs.mount # Mount volume (internal)
ebs.unmount # Unmount volume (internal)
# S3 Service Topics:
s3.createbucket # Create S3 bucket
s3.putobject # Store object
s3.getobject # Retrieve object
s3.listbuckets # List buckets
# VPC Service Topics:
vpc.createvpc # Create VPC
vpc.createsubnet # Create subnet
vpc.describevpcs # List VPCs
# Health and Discovery:
health.ec2.daemon.{id} # Daemon health heartbeats
discovery.services # Service registration
```
### Go Module Structure
All Go projects use:
- Go 1.23+ with module mode
- Local replace directives for cross-component development
- Standard build flags: `-ldflags "-s -w"` for optimized binaries
- Test environment variable: `LOG_IGNORE=1` to suppress logs during testing
### Development Workflow
#### Quick Setup (Recommended)
```bash
# 1. Clone dependencies (if not already done)
./scripts/clone-deps.sh
# 2. Setup development environment
./scripts/dev-setup.sh
# 3. Start all services
./scripts/start-dev.sh
# 4. Stop all services when done
./scripts/stop-dev.sh
```
#### Manual Development Process
1. **Cross-repo Setup**: Clone all repositories to same parent directory
2. **Service Dependencies**: Start services in order: NATS → Predastore → Viperblock → NBDkit → Gateway → Daemons
3. **Build Order**: Build components: predastore → viperblock → hive
4. **Testing**: Use `make test` in each component to verify changes
5. **Message-Driven Development**: Test NATS message flows between gateway and daemons
6. **Scaling**: Test with multiple daemon instances subscribing to same topics
#### Development Tools
- **Hot Reloading**: Use `air` for automatic restarts during development
- **TLS Certificates**: Auto-generated self-signed certificates for HTTPS endpoints
- **Log Monitoring**: Centralized logs in `data/logs/` directory
- **AWS CLI Testing**: Test endpoints with `aws --endpoint-url https://localhost:9999`
### Development Roadmap
For major feature development and architectural changes, refer to `HIVE_DEVELOPMENT_PLAN.md` which contains:
- Comprehensive roadmap for transforming Hive from proof-of-concept to beta release
- Detailed implementation phases with specific tasks and file structures
- AWS API compatibility implementation guidelines
- Development environment automation plans
- Testing and validation frameworks
When implementing new AWS services or enhancing existing functionality, follow the structured approach outlined in the development plan.
-74
View File
@@ -1,74 +0,0 @@
# Design ideas
### hivectl
Basic commands to setup a compute instance, block storage (viperblock), object storage (predastore)
`hivectl configure --endpoint-url https://localhost:8443/`
`hivectl auth --access_key X --secret_key Y --profile hivelocal --endpoint-url https://localhost:8443/`
Writes to ~/.aws/credentials
`hivectl vol add --size 128M --no-verify-ssl --endpoint-url https://localhost:8443/ --`
Authenticates using ~/.aws/credentials - Backend creates a unique volume id
Connects to S3, creates the /vol-uniquestr/, uploads `config.json`
```
{
"VolumeName": "vol-uniquestr",
"VolumeSize": 134217728,
"BlockSize": 4096,
"ObjBlockSize": 4194304,
"SeqNum": 0,
"ObjectNum": 0,
"WALNum": 0,
"BlockToObjectWALNum": 0,
"Version": 1
}
```
`hivectl ec2 import-key-pair --key-name my-key-pair --public-key-material fileb://~/.ssh/id_rsa.pub`
This will upload the users SSH public key to their S3 bucket, /bucket/ec2/my-key-pair.pub
`hivectl ec2 run-instances --image-id ami-X --instance-type t3.micro --key-name my-key-pair`
This will clone an existing image-id (vol-ID, e.g debian snapshot), copy to the users S3 bucket, and start the new volume from the fresh copy.
Note when starting an image additional volumes are required.
```
:~/qemu-host$ truncate -s 64m varstore.img
:~/qemu-host$ truncate -s 64m efi.img
:~/qemu-host$ dd if=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd of=efi.img conv=notrunc
```
Sample launch
Require to start NBD (EFI)
./server/nbdkit -p 10810 --pidfile /tmp/vb-vol-1-efi.pid ./plugins/golang/examples/ramdisk/nbdkit-goramdisk-plugin.so -v -f size=67108864 volume=vol-1-efi bucket=predastore region=ap-southeast-2 access_key="AKIAIOSFODNN7EXAMPLE" secret_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" base_dir="/tmp/vb" host="https://127.0.0.1:8443"
Require to start Debian volume
./server/nbdkit -p 10809 --pidfile /tmp/vb-vol-1-root.pid ./plugins/golang/examples/ramdisk/nbdkit-goramdisk-plugin.so -v -f size=4294967296 volume=vol-1-root bucket=predastore region=ap-southeast-2 access_key="AKIAIOSFODNN7EXAMPLE" secret_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" base_dir="/tmp/vb" host="https://127.0.0.1:8443"
```
:~/qemu-host$ ~/qemu/build/aarch64-softmmu/qemu-system-aarch64 -M virt \
-machine virtualization=true -machine virt,gic-version=3 \
-cpu max,pauth-impdef=on -smp 2 -m 4096 \
-drive if=pflash,format=raw,file=efi.img,readonly=on \
-drive if=pflash,format=raw,file=varstore.img \
-drive if=virtio,format=qcow2,file=disk.img \
-device virtio-scsi-pci,id=scsi0 \
-object rng-random,filename=/dev/urandom,id=rng0 \
-device virtio-rng-pci,rng=rng0 \
-device virtio-net-pci,netdev=net0 \
-netdev user,id=net0,hostfwd=tcp::8022-:22 \
-nographic \
-drive if=none,id=cd,file=debian-12.2.0-arm64-netinst.iso \
-device scsi-cd,drive=cd
```
File diff suppressed because it is too large Load Diff
+1013 -189
View File
File diff suppressed because it is too large Load Diff
+208 -8
View File
@@ -1,23 +1,99 @@
GO_PROJECT_NAME := hive
SHELL := /bin/bash
# Detect architecture for cross-platform support
ARCH := $(shell uname -m)
ifeq ($(ARCH),x86_64)
GO_ARCH := amd64
AWS_ARCH := x86_64
# On x86, we can run ARM VMs via emulation
QEMU_PACKAGES := qemu-system-x86 qemu-system-arm
else ifeq ($(ARCH),aarch64)
GO_ARCH := arm64
AWS_ARCH := aarch64
# On ARM, we can run x86 VMs via emulation
QEMU_PACKAGES := qemu-system-aarch64 qemu-system-x86
else ifeq ($(ARCH),arm64)
GO_ARCH := arm64
AWS_ARCH := aarch64
QEMU_PACKAGES := qemu-system-aarch64 qemu-system-x86
else
$(error Unsupported architecture: $(ARCH). Only x86_64 and aarch64/arm64 are supported.)
endif
# Ask Go whether workspace mode is active
IN_WORKSPACE := $(shell go env GOWORK)
# Use -mod=mod unless Go reports an active workspace path
ifeq ($(IN_WORKSPACE),)
GO_BUILD_MOD := -mod=mod
else ifeq ($(IN_WORKSPACE),off)
GO_BUILD_MOD := -mod=mod
else
GO_BUILD_MOD :=
endif
# Quiet-mode filters (active when QUIET=1, set by preflight via recursive make)
ifdef QUIET
_Q = @
_COVQ = 2>&1 | grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' | grep -v 'coverage: 0\.0%' || true
_RACEQ = 2>&1 | { grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' || true; }; exit $${PIPESTATUS[0]}
_SECQ = >
else
_Q =
_COVQ = || true
_RACEQ =
_SECQ = 2>&1 | tee
endif
build:
$(MAKE) go_build
# Build hive-ui frontend (requires pnpm)
build-ui:
@echo -e "\n....Building hive-ui frontend...."
cd hive/services/hiveui/frontend && pnpm build
# GO commands
go_build:
@echo "\n....Building $(GO_PROJECT_NAME)"
go build -ldflags "-s -w" -o ./bin/$(GO_PROJECT_NAME) cmd/hive/main.go
@echo -e "\n....Building $(GO_PROJECT_NAME)"
go build $(GO_BUILD_MOD) -ldflags "-s -w" -o ./bin/$(GO_PROJECT_NAME) cmd/hive/main.go
go_run:
@echo "\n....Running $(GO_PROJECT_NAME)...."
@echo -e "\n....Running $(GO_PROJECT_NAME)...."
$(GOPATH)/bin/$(GO_PROJECT_NAME)
# Preflight — runs the same checks as GitHub Actions (format + lint + security + tests).
# Use this before committing to catch CI failures locally.
preflight:
@$(MAKE) --no-print-directory QUIET=1 check-format check-modernize vet security-check test-cover diff-coverage test-race
@echo -e "\n ✅ Preflight passed — safe to commit."
# Run unit tests
test:
@echo "\n....Running tests for $(GO_PROJECT_NAME)...."
LOG_IGNORE=1 go test -v -timeout 300s ./...
@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
LOG_IGNORE=1 go test -timeout 120s ./hive/...
# Run unit tests with coverage profile
# Note: go test may exit non-zero due to Go version mismatch in coverage instrumentation
# for packages without test files. We check actual test results + coverage threshold instead.
COVERPROFILE ?= coverage.out
test-cover:
@echo -e "\n....Running tests with coverage for $(GO_PROJECT_NAME)...."
$(_Q)LOG_IGNORE=1 go test -timeout 120s -coverprofile=$(COVERPROFILE) -covermode=atomic ./hive/... $(_COVQ)
@scripts/check-coverage.sh $(COVERPROFILE) $(QUIET)
# Run unit tests with race detector
test-race:
@echo -e "\n....Running tests with race detector for $(GO_PROJECT_NAME)...."
$(_Q)LOG_IGNORE=1 go test -race -timeout 300s ./hive/... $(_RACEQ)
# Check that new/changed code meets coverage threshold (runs tests first)
diff-coverage: test-cover
@QUIET=$(QUIET) scripts/diff-coverage.sh $(COVERPROFILE)
bench:
@echo "\n....Running benchmarks for $(GO_PROJECT_NAME)...."
@echo -e "\n....Running benchmarks for $(GO_PROJECT_NAME)...."
$(MAKE) easyjson
LOG_IGNORE=1 go test -benchmem -run=. -bench=. ./...
@@ -26,6 +102,130 @@ run:
$(MAKE) go_run
clean:
rm ./bin/$(GO_PROJECT_NAME)
rm -f ./bin/$(GO_PROJECT_NAME)
rm -rf hive/services/hiveui/frontend/dist
.PHONY: go_build go_run build run test clean
install-system:
@echo -e "\n....Installing system dependencies for $(ARCH)...."
@echo "QEMU packages: $(QEMU_PACKAGES)"
apt-get update && sudo apt-get install -y \
nbdkit nbdkit-plugin-dev pkg-config $(QEMU_PACKAGES) qemu-utils qemu-kvm \
libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
ovn-central ovn-host openvswitch-switch
install-go:
@echo -e "\n....Installing Go 1.26.1 for $(ARCH) ($(GO_ARCH))...."
@if [ ! -d "/usr/local/go" ]; then \
curl -L https://go.dev/dl/go1.26.1.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
else \
echo "Go already installed in /usr/local/go"; \
fi
@echo "Go version: $$(go version)"
install-aws:
@echo -e "\n....Installing AWS CLI v2 for $(ARCH) ($(AWS_ARCH))...."
@if ! command -v aws >/dev/null 2>&1; then \
curl "https://awscli.amazonaws.com/awscli-exe-linux-$(AWS_ARCH).zip" -o "awscliv2.zip"; \
unzip -o awscliv2.zip; \
./aws/install; \
rm -rf awscliv2.zip aws/; \
else \
echo "AWS CLI already installed"; \
fi
quickinstall: install-system install-go install-aws
@echo -e "\n✅ Quickinstall complete for $(ARCH)."
@echo " Please ensure /usr/local/go/bin is in your PATH."
@echo " Installed: Go ($(GO_ARCH)), AWS CLI ($(AWS_ARCH)), QEMU ($(QEMU_PACKAGES))"
# Format all Go files in place
format:
gofmt -w .
# Check that all Go files are formatted (CI-compatible, fails on diff)
check-format:
@echo "Checking gofmt..."
@UNFORMATTED=$$(gofmt -l .); \
if [ -n "$$UNFORMATTED" ]; then \
echo "Files not formatted:"; \
echo "$$UNFORMATTED"; \
echo "Run 'make format' to fix."; \
exit 1; \
fi
@echo " gofmt ok"
# Go vet (fails on issues, matches CI)
vet:
@echo "Running go vet..."
$(_Q)go vet ./...
@echo " go vet ok"
# Excluded: newexpr (replaces aws.String with new, not idiomatic for AWS SDK)
# Excluded: stringsbuilder (replaces string += in loops with strings.Builder, not worth the complexity for small loops)
GOFIX_EXCLUDE := -newexpr=false -stringsbuilder=false
# Apply go fix modernizations
modernize:
@echo "Applying go fix modernizations..."
go fix $(GOFIX_EXCLUDE) ./...
@echo " go fix applied"
# Check that code is modernized (CI-compatible, fails on diff)
check-modernize:
@echo "Checking go fix modernizations..."
@DIFF=$$(go fix $(GOFIX_EXCLUDE) -diff ./...); \
if [ -n "$$DIFF" ]; then \
echo "$$DIFF"; \
echo "Run 'make modernize' to fix."; \
exit 1; \
fi
@echo " go fix ok"
# Security checks — each tool fails the build on findings (matches CI).
# Reports are also saved to tests/ for review.
security-check:
@echo -e "\n....Running security checks for $(GO_PROJECT_NAME)...."
$(_Q)set -o pipefail && go tool govulncheck ./... $(_SECQ) tests/govulncheck-report.txt $(if $(QUIET),|| { cat tests/govulncheck-report.txt; exit 1; })
@echo " govulncheck ok"
$(_Q)set -o pipefail && go tool gosec -quiet -exclude=G204,G304,G402,G117,G703,G705,G706 -exclude-generated -exclude-dir=cmd ./... $(_SECQ) tests/gosec-report.txt $(if $(QUIET),|| { cat tests/gosec-report.txt; exit 1; })
@echo " gosec ok"
$(_Q)set -o pipefail && go tool staticcheck -checks="all,-ST1000,-ST1003,-ST1016,-ST1020,-ST1021,-ST1022,-SA1019,-SA9005" ./... $(_SECQ) tests/staticcheck-report.txt $(if $(QUIET),|| { cat tests/staticcheck-report.txt; exit 1; })
@echo " staticcheck ok"
# Docker E2E tests (mirrors GitHub Actions e2e.yml)
# Usage: make test-docker # both suites
# make test-docker-single # single-node only
# make test-docker-multi # multi-node only
PARENT_DIR := $(shell cd .. && pwd)
E2E_IMAGE := hive-e2e:latest
test-docker-build:
@echo -e "\n....Building E2E Docker image...."
@for dep in viperblock predastore; do \
if [ ! -d "$(PARENT_DIR)/$$dep" ]; then \
echo "Missing sibling repo $$dep — running clone-deps.sh"; \
./scripts/clone-deps.sh; \
break; \
fi; \
done
docker build -t $(E2E_IMAGE) -f tests/e2e/Dockerfile.e2e $(PARENT_DIR)
test-docker-single: test-docker-build
@echo -e "\n....Running Single-Node E2E Tests...."
docker run --privileged --rm -v /dev/kvm:/dev/kvm $(E2E_IMAGE)
test-docker-multi: test-docker-build
@echo -e "\n....Running Multi-Node E2E Tests...."
docker run --privileged --rm -v /dev/kvm:/dev/kvm --cap-add=NET_ADMIN $(E2E_IMAGE) ./tests/e2e/run-multinode-e2e.sh
test-docker: test-docker-build
@echo -e "\n....Running Single-Node E2E Tests...."
docker run --privileged --rm -v /dev/kvm:/dev/kvm $(E2E_IMAGE)
@echo -e "\n....Running Multi-Node E2E Tests...."
docker run --privileged --rm -v /dev/kvm:/dev/kvm --cap-add=NET_ADMIN $(E2E_IMAGE) ./tests/e2e/run-multinode-e2e.sh
.PHONY: build build-ui go_build go_run preflight test test-cover test-race diff-coverage bench run clean \
install-system install-go install-aws quickinstall \
format check-format modernize check-modernize vet security-check \
test-docker-build test-docker-single test-docker-multi test-docker
+3 -9
View File
@@ -60,7 +60,7 @@ Get started with Hive development in minutes:
```bash
# Clone the repository
git clone https://github.com/mulgadc/hive.git hive
git clone https://github.com/mulgadc/hive.git
cd hive
# Setup dependencies and development environment
@@ -79,7 +79,7 @@ Hive coordinates these independent components:
Each component can be developed independently. See component-specific documentation for focused development guides.
## Detailed Installation method
## Detailed Installation Method
See [Installation Guide](INSTALL.md) on how to setup a Hive node to get started.
@@ -116,14 +116,8 @@ While Hive is architected and implemented by experienced engineers, we leverage
- **Performance Optimization**: Real-world performance tuning and benchmarking
- **Production Operations**: Deployment strategies and operational procedures
**Development Artifacts:**
- `CLAUDE.md` - Instructions for Claude Code when working on this codebase
- `HIVE_DEVELOPMENT_PLAN.md` - Comprehensive roadmap and implementation strategy
- Component-specific guidance in `viperblock/CLAUDE.md` and `predastore/CLAUDE.md`
This hybrid approach ensures Hive benefits from both proven engineering expertise and modern development acceleration, while maintaining the quality and reliability standards required for production infrastructure.
## License
Hive is open source under the Apache 2.0 License. You're free to use, modify, and deploy it—anywhere you need reliable infrastructure without depending on centralized cloud platforms
Hive is open source under the Apache 2.0 License. You're free to use, modify, and deploy it—anywhere you need reliable infrastructure without depending on centralized cloud platforms.
-509
View File
@@ -1,509 +0,0 @@
# Testing Implementation Summary
## ✅ Completed Work
### 1. Daemon Test Suite (`hive/daemon/daemon_test.go`)
Created comprehensive unit tests for the daemon's EC2 launch functionality:
**Test Components:**
- `startTestNATSServer()` - Embedded NATS server with auto port allocation
- `createTestDaemon()` - Test daemon factory with automatic cleanup
- `createValidRunInstancesInput()` - Test fixture for valid EC2 requests
**Test Suites:**
- ✅ `TestHandleEC2Launch_MessageParsing` - NATS message handling (5 test cases)
- ⏸️ `TestHandleEC2Launch_ResourceManagement` - Skipped (requires full infrastructure)
- ✅ `TestDaemon_Initialization` - Daemon creation and configuration
- ✅ `TestResourceManager` - Resource allocation logic
**Test Results:**
```bash
$ go test -v ./hive/daemon/
PASS: TestHandleEC2Launch_MessageParsing (0.17s)
✓ Valid_RunInstancesInput
✓ Invalid_Instance_Type
✓ Missing_Required_ImageId
✓ Invalid_MinCount_(zero)
✓ Malformed_JSON
SKIP: TestHandleEC2Launch_ResourceManagement
PASS: TestDaemon_Initialization (0.00s)
PASS: TestResourceManager (0.00s)
ok github.com/mulgadc/hive/hive/daemon 0.592s
```
### 2. Utils Test Suite (`hive/utils/utils_test.go`)
Added comprehensive tests for previously untested utility functions:
**New Tests:**
- ✅ `TestUnmarshalJsonPayload` - JSON unmarshaling (4 test cases)
- ✅ `TestMarshalJsonPayload` - JSON marshaling (2 test cases)
- ✅ `TestGenerateErrorPayload` - Error generation (3 test cases)
- ✅ `TestValidateErrorPayload` - Error validation (4 test cases)
- ✅ `TestMarshalToXML` - XML marshaling (3 test cases)
- ✅ `TestKillProcess` - Process termination
- ✅ `TestStopProcess` - Service stop with PID cleanup
**Test Results:**
```bash
$ go test -v ./hive/utils/
PASS: TestGeneratePidFile (0.00s)
PASS: TestGenerateSocketFile (0.00s)
PASS: TestExecProcessAndKill (1.71s)
PASS: TestUnmarshalJsonPayload (0.00s)
PASS: TestMarshalJsonPayload (0.00s)
PASS: TestGenerateErrorPayload (0.00s)
PASS: TestValidateErrorPayload (0.00s)
PASS: TestMarshalToXML (0.00s)
PASS: TestKillProcess (4.51s)
PASS: TestStopProcess (4.01s)
ok github.com/mulgadc/hive/hive/utils 10.844s
```
### 3. Bug Fixes
Fixed critical bugs that prevented tests from passing:
#### Bug 1: Pointer-to-Pointer in UnmarshalJsonPayload
**Location:** `hive/utils/utils.go:283`
**Before:**
```go
err := decoder.Decode(&input) // input already a pointer, creates **Type
```
**After:**
```go
err := decoder.Decode(input) // Correct: input is already a pointer
```
#### Bug 2: Nil Pointer in handleEC2Launch
**Location:** `hive/daemon/daemon.go:638-645`
**Before:**
```go
var runInstancesInput *ec2.RunInstancesInput // nil pointer
errResp = utils.UnmarshalJsonPayload(runInstancesInput, msg.Data) // crashes
```
**After:**
```go
runInstancesInput := &ec2.RunInstancesInput{} // Initialize before use
errResp = utils.UnmarshalJsonPayload(runInstancesInput, msg.Data)
```
#### Bug 3: Uninitialized Slice Elements in launchEC2Instance
**Location:** `hive/daemon/daemon.go:852-856`
**Before:**
```go
reservation.Instances = make([]*ec2.Instance, 1)
reservation.Instances[0].SetInstanceId(instance.ID) // nil pointer dereference
```
**After:**
```go
reservation.Instances = make([]*ec2.Instance, 1)
reservation.Instances[0] = &ec2.Instance{
State: &ec2.InstanceState{},
}
reservation.Instances[0].SetInstanceId(instance.ID)
```
#### Bug 4: Same Issue in MarshalJsonPayload
**Location:** `hive/utils/utils.go:299`
**Before:**
```go
err := decoder.Decode(&input)
```
**After:**
```go
err := decoder.Decode(input)
```
### 4. Documentation
Created comprehensive testing documentation:
**Files Created:**
1. `hive/daemon/TESTING.md` (423 lines)
- Test structure and components
- macOS limitations and workarounds
- Test execution instructions
- Mock service strategies
- Debugging guide
2. `hive/daemon/TEST_SUMMARY.md` (467 lines)
- Test architecture highlights
- Known issues and limitations
- Recommendations for different platforms
- Next steps and roadmap
- Current test results
3. `TESTING_COMPLETE.md` (this file)
- Summary of all completed work
- Bug fixes
- Test results
- Future enhancements
## 📊 Test Coverage
### Overall Test Results
```bash
$ make test
✓ hive/awsec2query - All tests passing
✓ hive/awserrors - All tests passing
✓ hive/daemon - 3 passing, 1 skipped (infrastructure required)
✓ hive/gateway/ec2/instance - All tests passing
✓ hive/service - All tests passing
✓ hive/services/predastore - All tests passing (integration skipped)
✓ hive/services/viperblockd - All tests passing
✓ hive/utils - All tests passing (10 tests, 26 cases)
✓ hive/vm - All tests passing
OVERALL: PASS
```
### Code Coverage by Package
| Package | Tests | Coverage | Notes |
|---------|-------|----------|-------|
| daemon | 4 | ~60% | Message handling, resource mgmt, initialization |
| utils | 10 | ~85% | All major functions tested |
| gateway/ec2/instance | 2 | ~75% | RunInstances validation |
| vm | 2 | ~40% | Config generation, ID creation |
| services/* | 8 | ~70% | Service lifecycle, NATS integration |
## 🎯 Test Framework Features
### Daemon Tests
**Strengths:**
- ✅ Isolated NATS server per test
- ✅ Automatic resource cleanup
- ✅ Table-driven test design
- ✅ Comprehensive error scenarios
- ✅ AWS SDK integration testing
**Limitations:**
- ⚠️ Requires mocks for viperblock/nbdkit on macOS
- ⚠️ Full VM launch needs Linux + infrastructure
- ⚠️ Cloud-init volume creation not tested
### Utils Tests
**Strengths:**
- ✅ Comprehensive coverage of JSON/XML marshaling
- ✅ Process management testing
- ✅ Error handling validation
- ✅ Platform-aware (macOS vs Linux)
**Coverage:**
- ✅ PID file operations
- ✅ Socket file generation
- ✅ Process lifecycle (start/stop/kill)
- ✅ JSON payload marshaling/unmarshaling
- ✅ XML marshaling
- ✅ Error payload generation/validation
## 🐛 Bugs Fixed
### Critical (Blocking Tests)
1. **Pointer-to-pointer bugs** (2 instances)
- Impact: Caused all JSON unmarshaling to fail
- Fix: Remove unnecessary `&` operator
- Files: `utils.go:283, 299`
2. **Nil pointer dereference in handleEC2Launch**
- Impact: All EC2 launch requests crashed
- Fix: Initialize struct before passing to unmarshal
- File: `daemon.go:638`
3. **Uninitialized slice elements**
- Impact: Crash when setting instance fields
- Fix: Initialize slice elements before use
- File: `daemon.go:852-856`
### Total Lines Changed
- **Modified:** ~15 lines across 3 files
- **Added:** ~550 lines of tests
- **Documentation:** ~1,200 lines
## 📝 Test Scenarios Covered
### Daemon Tests
**Message Parsing:**
- ✅ Valid RunInstancesInput with all fields
- ✅ Invalid instance type rejection
- ✅ Missing required ImageId
- ✅ Invalid MinCount (zero)
- ✅ Malformed JSON handling
**Resource Management:**
- ✅ Instance type lookup
- ✅ Resource allocation/deallocation
- ✅ Capacity checking
- ✅ System resource detection
### Utils Tests
**JSON Operations:**
- ✅ Valid JSON unmarshaling
- ✅ Malformed JSON error handling
- ✅ Unknown field rejection (DisallowUnknownFields)
- ✅ Empty JSON handling
**Error Handling:**
- ✅ Error payload generation
- ✅ Error payload validation
- ✅ Success vs error discrimination
**XML Operations:**
- ✅ Struct marshaling
- ✅ Pointer marshaling
- ✅ Invalid type error handling
**Process Management:**
- ✅ Process creation and termination
- ✅ PID file read/write/remove
- ✅ Signal handling
- ✅ Non-existent process error handling
## 🚀 Running Tests
### Quick Test
```bash
# Run all tests
make test
# Run specific package
go test -v ./hive/daemon/
go test -v ./hive/utils/
# Run with race detection
go test -race ./...
# Run with coverage
go test -cover ./...
```
### Platform-Specific
**macOS (Unit Tests Only):**
```bash
# All passing unit tests
go test -v ./hive/daemon/
go test -v ./hive/utils/
```
**Linux (Full Integration):**
```bash
# Requires: predastore, viperblock, nbdkit, QEMU
# Remove SKIP_INTEGRATION flag
unset SKIP_INTEGRATION
go test -v ./hive/daemon/
```
**Docker (Recommended for Full Tests):**
```bash
docker-compose up test-env
docker exec -it hive-test go test -v ./...
```
## 🔍 What's Tested vs Not Tested
### ✅ Fully Tested
- NATS message parsing and routing
- JSON marshaling/unmarshaling
- Error generation and validation
- Resource manager logic
- PID file operations
- Process lifecycle management
- XML marshaling
- Input validation
- Instance type lookup
### ⏸️ Partially Tested (Mocks Needed for macOS)
- EC2 launch flow (stops at viperblock creation)
- Volume mounting (requires nbdkit)
- Cloud-init generation (requires S3 backend)
### ❌ Not Yet Tested (Requires Full Infrastructure)
- Complete VM launch
- QEMU process management
- QMP socket communication
- NBD volume attachment
- EBS volume lifecycle
- VM state transitions
- Concurrent instance launches
- Resource exhaustion scenarios
## 🎯 Future Enhancements
### Short Term (Next Sprint)
1. **Mock Services**
```go
// Mock EBS mount service for macOS
func startMockEBSService(t *testing.T, nc *nats.Conn) {
nc.Subscribe("ebs.mount", func(msg *nats.Msg) {
// Return mock NBD URI
})
}
```
2. **Integration Tests**
- Add `//go:build linux` tags
- Full VM launch tests
- Volume lifecycle tests
3. **Benchmark Tests**
```go
func BenchmarkEC2Launch(b *testing.B) {
// Measure launch performance
}
```
### Medium Term
1. **CI/CD Integration**
- GitHub Actions workflow
- Docker-based test environment
- Coverage reporting (codecov.io)
2. **Load Testing**
- Concurrent launch stress tests
- Resource exhaustion tests
- NATS throughput tests
3. **Mock Framework**
- Reusable mock services
- Test helpers package
- Platform-aware test skipping
### Long Term
1. **E2E Test Suite**
- Complete AWS SDK compatibility tests
- Multi-instance orchestration
- Failure recovery scenarios
2. **Performance Suite**
- Latency measurements
- Throughput benchmarks
- Resource utilization profiling
3. **Chaos Engineering**
- Network partition tests
- Service failure simulation
- Recovery time validation
## 📚 Documentation Structure
```
hive/
├── hive/daemon/
│ ├── daemon.go [FIXED: 3 bugs]
│ ├── daemon_test.go [NEW: 4 test suites]
│ ├── TESTING.md [NEW: 423 lines]
│ └── TEST_SUMMARY.md [NEW: 467 lines]
├── hive/utils/
│ ├── utils.go [FIXED: 2 bugs]
│ └── utils_test.go [ENHANCED: +10 tests]
└── TESTING_COMPLETE.md [NEW: This file]
```
## 🎉 Success Metrics
### Before
- ❌ daemon tests: 0 tests, crashes on startup
- ❌ utils tests: 3 tests, missing coverage for 13+ functions
- ❌ Critical bugs: 4 blocking issues
- ❌ Documentation: None
### After
- ✅ daemon tests: 4 test suites, 5 test cases passing
- ✅ utils tests: 10 test suites, 26 test cases passing
- ✅ Critical bugs: All fixed
- ✅ Documentation: 1,200+ lines
- ✅ Total test cases: 50+ across all packages
- ✅ Overall: `make test` passes 100%
## 💡 Key Learnings
### Test Design
1. **Embedded NATS server is ideal for testing**
- No external dependencies
- Port auto-allocation prevents conflicts
- Fast startup/teardown
2. **Table-driven tests scale well**
- Easy to add new cases
- Clear test structure
- Minimal code duplication
3. **Platform differences matter**
- macOS vs Linux signal handling
- Process cleanup timing
- File system behavior
### Bug Patterns
1. **Pointer confusion is common**
- Double-check `&` usage
- Verify struct initialization
- Test nil pointer paths
2. **AWS SDK requires explicit initialization**
- Can't use SetX() on nil pointers
- Must initialize nested structs
- Slice make() != slice element initialization
3. **JSON unmarshaling needs initialized target**
- Can't unmarshal into nil pointer
- DisallowUnknownFields is strict
- Empty {} is valid but may be treated as error
## 🔗 Related Files
- `CLAUDE.md` - Project overview and development guide
- `HIVE_DEVELOPMENT_PLAN.md` - Roadmap and feature planning
- `hive/daemon/TESTING.md` - Detailed testing guide
- `hive/daemon/TEST_SUMMARY.md` - Test architecture and status
- `hive/utils/utils_test.go` - Utils test implementation
- `hive/daemon/daemon_test.go` - Daemon test implementation
## ✅ Acceptance Criteria Met
- [x] Created `daemon_test.go` with NATS server setup
- [x] Implemented `handleEC2Launch` message tests
- [x] Created sample RunInstancesInput test cases
- [x] Validated expected arguments and error handling
- [x] Documented macOS limitations (viperblock, nbdkit)
- [x] Provided suggestions for scaffolding
- [x] Fixed existing bugs (pointer-to-pointer issues)
- [x] Created comprehensive utils tests
- [x] All tests passing with `make test`
- [x] Detailed documentation created
## 🏁 Conclusion
The test framework is now production-ready for unit testing on both macOS and Linux. The daemon EC2 launch functionality has comprehensive message handling tests, and the utils package is thoroughly tested. All critical bugs have been fixed, and the codebase is ready for further development.
**Next Steps:**
1. Implement mock services for full macOS testing
2. Add Linux-specific integration tests
3. Set up CI/CD pipeline with Docker
4. Expand coverage to volume lifecycle operations
-88
View File
@@ -1,88 +0,0 @@
# Hive Development Roadmap
This roadmap has been integrated into the comprehensive [HIVE_DEVELOPMENT_PLAN.md](HIVE_DEVELOPMENT_PLAN.md).
## Update Nov 2025
* Implement multi-tenant support
* Move config settings from ~/hive/*.toml, to using Nats Jetstream for core config which can be synced between nodes.
* Implement a lightweight IAM using Nats Jetstream, vs current config files for auth settings.
* Implement Reed Solomon Encoding for Predastore (S3) objects and a WAL implementation, for storing multiple objects in a single shard (e.g 4MB), with the WAL referencing the location of each object (e.g 4kb min)
* Implement basic KY lookup, object key sha512 (bucket/key), location to shard on S3 (obj.0000124.bin), read WAL (e.g first 4096 bytes) to determine location of the object. e.g key-1234 => obj.0000124.bin, wal header (4096 bytes) > key-1234 == offset location (seek) 16384, len 32768.
* Read multiple nodes (e.g 5 predastore instances, k = 3 (data shards), n = 5 (total shards), n - k = 2 (parity shards) )
* Complete core scaffolding AWS SDK/API requirements (ec2 describe-instances, run-instances, etc)
* Implement UEFI support for image downloads and `qemu` exec in `vm.go`
* Confirm Alpine Linux, fails import image AMI > (run-instance) ""Failed to read block from AMI source" err="request out of range" - Block size correct?
* Improve shutdown gracefully, `./scripts/stop-dev.sh` waits 60 seconds, while qemu/nbd could still be shutting down.
* Add delete-volume support via EBS (s3 vol-*) for terminated instance
* Add default LRU cache support for viperblock, depending on the instance type / volume size and system memory available.
## Original TODO Items → Development Plan Integration
### ✅ **Completed Integration**
All original TODO items have been incorporated into the structured development phases:
| Original TODO | Status | Integrated Into | Phase |
|---------------|--------|-----------------|-------|
| #1: Binary compile and install.sh | ✅ Planned | Phase 7: Task 7.1 (Binary Compilation) | Phase 7 |
| #2: Move daemon.go to services/hived/ | ✅ Planned | Phase 3: Task 3.2 (Specialized Services) | Phase 3 |
| #3: VPC with openvs-switch as `vpcd` | ✅ Planned | Phase 5: Task 5.1 (VPC with Open vSwitch) | Phase 5 |
| #4: AWS HTTP gateway (`awsd`) | ✅ Planned | Phase 3: Task 3.1 (AWS Gateway Service) | Phase 3 |
| #5: AWS SDK v2 input/output | ✅ Planned | Phase 2: Task 2.1 & Phase 3: Task 3.1 | Phase 2-3 |
| #6: `hive.service` for systemd | ✅ Planned | Phase 7: Task 7.2 (System Service Integration) | Phase 7 |
| #7: Gossip and RAFT protocols | ✅ Planned | Phase 0: Task 0.1 (Service Registry) | Phase 0 |
| #8: etcd/KV for configuration sync | ✅ Planned | Phase 0: Task 0.1 (Distributed Config) | Phase 0 |
| #9: Smithy model code generation | ✅ Planned | Phase 2: Task 2.1 (Smithy Code Generation) | Phase 2 |
## Development Phase Overview
**Phase 0**: Distributed Systems Foundation (1-2 weeks)
- Gossip, RAFT, and distributed configuration (#7, #8)
**Phase 1**: Development Environment Automation (2-3 weeks)
- Multi-service orchestration and hot reloading
**Phase 2**: AWS API Model Implementation (3-4 weeks)
- Smithy-based code generation (#9)
- AWS SDK v2 integration (#5)
**Phase 3**: Scalable Gateway and Daemon Architecture (2-3 weeks)
- AWS gateway service `awsd` (#4)
- Service refactoring to `hived` (#2)
- VPC daemon `vpcd` foundation (#3)
**Phase 4**: Service Integration and Orchestration (3-4 weeks)
- NATS clustering and cross-service coordination
**Phase 5**: Infrastructure Services (4-5 weeks)
- VPC networking with Open vSwitch (#3)
- Advanced VM features
**Phase 6**: Testing and Validation Framework (2-3 weeks)
- AWS CLI compatibility testing
**Phase 7**: Production Deployment and Packaging (2-3 weeks)
- Binary compilation and distribution (#1)
- Systemd service integration (#6)
- Production configuration management
## Service Architecture
The development plan implements these services:
- **`awsd`** - AWS API Gateway (TODO #4)
- **`hived`** - Main Hive compute daemon (TODO #2)
- **`vpcd`** - VPC networking daemon (TODO #3)
- **Predastore** - S3 service (existing)
- **Viperblock** - EBS service (existing)
## Getting Started
For development setup:
```bash
./scripts/dev-setup.sh # Setup complete development environment
./scripts/start-dev.sh # Start all services
```
For detailed implementation guidance, see [HIVE_DEVELOPMENT_PLAN.md](HIVE_DEVELOPMENT_PLAN.md).
+926 -277
View File
File diff suppressed because it is too large Load Diff
+195
View File
@@ -0,0 +1,195 @@
package cmd
import (
"encoding/json"
"fmt"
"os"
"strings"
"time"
"github.com/mulgadc/hive/hive/daemon"
"github.com/nats-io/nats.go"
"github.com/spf13/cobra"
)
// runClusterShutdown orchestrates a phased, coordinated shutdown of the cluster.
func runClusterShutdown(cmd *cobra.Command, args []string) {
force, _ := cmd.Flags().GetBool("force")
timeout, _ := cmd.Flags().GetDuration("timeout")
dryRun, _ := cmd.Flags().GetBool("dry-run")
cfg, nc, err := loadConfigAndConnect()
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
os.Exit(1)
}
defer nc.Close()
nodeCount := len(cfg.Nodes)
phases := []string{"gate", "drain", "storage", "persist", "infra"}
if dryRun {
fmt.Println("Cluster Shutdown Plan (dry-run)")
fmt.Printf(" Nodes: %d\n", nodeCount)
for name, nodeCfg := range cfg.Nodes {
fmt.Printf(" - %s (services: %s)\n", name, strings.Join(nodeCfg.GetServices(), ", "))
}
fmt.Printf(" Phases: %s\n", strings.Join(phases, " -> "))
fmt.Printf(" Timeout per phase: %s\n", timeout)
fmt.Printf(" Force: %v\n", force)
fmt.Println("\nPhase details:")
fmt.Println(" 1. GATE - Stop API gateway and UI, reject new work")
fmt.Println(" 2. DRAIN - Gracefully stop all VMs, persist state")
fmt.Println(" 3. STORAGE - Stop viperblock, cleanup nbdkit")
fmt.Println(" 4. PERSIST - Stop predastore")
fmt.Println(" 5. INFRA - Stop NATS, exit daemons")
return
}
fmt.Printf("Starting coordinated cluster shutdown (%d nodes)\n", nodeCount)
fmt.Printf("Phases: %s\n", strings.Join(phases, " -> "))
fmt.Printf("Timeout per phase: %s\n\n", timeout)
// Write cluster shutdown marker to KV
jsm, err := daemon.NewJetStreamManager(nc, 1)
if err != nil {
fmt.Fprintf(os.Stderr, "Warning: failed to create JetStream manager: %v\n", err)
} else {
if err := jsm.InitClusterStateBucket(); err != nil {
fmt.Fprintf(os.Stderr, "Warning: failed to init cluster state bucket: %v\n", err)
} else {
state := &daemon.ClusterShutdownState{
Initiator: cfg.Node,
Phase: "starting",
Started: time.Now().UTC().Format(time.RFC3339),
Timeout: timeout.String(),
Force: force,
NodesTotal: nodeCount,
NodesAcked: make(map[string]string),
}
if err := jsm.WriteClusterShutdown(state); err != nil {
fmt.Fprintf(os.Stderr, "Warning: failed to write shutdown state: %v\n", err)
}
}
}
start := time.Now()
// Execute phases sequentially (except INFRA which is fire-and-forget)
for _, phase := range phases {
topic := "hive.cluster.shutdown." + phase
req := daemon.ShutdownRequest{
Phase: phase,
Force: force,
Timeout: int(timeout.Seconds()),
}
reqData, err := json.Marshal(req)
if err != nil {
fmt.Fprintf(os.Stderr, "Error marshaling request: %v\n", err)
os.Exit(1)
}
if phase == "infra" {
// INFRA is fire-and-forget — NATS is going down, no ACKs possible
fmt.Printf("[INFRA] Sending final shutdown to all nodes...\n")
if err := nc.Publish(topic, reqData); err != nil {
fmt.Fprintf(os.Stderr, "Error publishing infra shutdown: %v\n", err)
}
nc.Flush()
// Wait briefly for messages to propagate
time.Sleep(2 * time.Second)
fmt.Printf("[INFRA] Complete\n")
break
}
// For DRAIN phase, subscribe to progress updates
var progressSub *nats.Subscription
if phase == "drain" {
progressSub, err = nc.Subscribe("hive.cluster.shutdown.progress", func(msg *nats.Msg) {
var progress daemon.ShutdownProgress
if err := json.Unmarshal(msg.Data, &progress); err != nil {
return
}
fmt.Printf(" [%s] %s: %d/%d VMs remaining\n", strings.ToUpper(phase), progress.Node, progress.Remaining, progress.Total)
})
if err != nil {
fmt.Fprintf(os.Stderr, "Warning: failed to subscribe to progress: %v\n", err)
}
}
// Collect ACKs from all nodes
phaseStart := time.Now()
fmt.Printf("[%s] Sending to %d node(s)...\n", strings.ToUpper(phase), nodeCount)
acks, err := collectShutdownACKs(nc, topic, reqData, nodeCount, timeout)
if err != nil {
fmt.Fprintf(os.Stderr, "[%s] Error: %v\n", strings.ToUpper(phase), err)
if !force {
fmt.Fprintf(os.Stderr, "Aborting shutdown. Use --force to continue despite errors.\n")
os.Exit(1)
}
}
// Unsubscribe from progress
if progressSub != nil {
progressSub.Unsubscribe()
}
// Print results
for _, ack := range acks {
if ack.Error != "" {
fmt.Printf(" [%s] %s: ERROR - %s\n", strings.ToUpper(phase), ack.Node, ack.Error)
} else if len(ack.Stopped) > 0 {
fmt.Printf(" [%s] %s: stopped %s\n", strings.ToUpper(phase), ack.Node, strings.Join(ack.Stopped, ", "))
} else {
fmt.Printf(" [%s] %s: OK\n", strings.ToUpper(phase), ack.Node)
}
}
ackedCount := len(acks)
if ackedCount < nodeCount && !force {
fmt.Fprintf(os.Stderr, "[%s] Only %d/%d nodes responded. Use --force to continue.\n",
strings.ToUpper(phase), ackedCount, nodeCount)
os.Exit(1)
}
fmt.Printf("[%s] Complete (%d/%d nodes, %s)\n\n", strings.ToUpper(phase), ackedCount, nodeCount, time.Since(phaseStart).Round(time.Millisecond))
}
fmt.Printf("Cluster shutdown complete (%s)\n", time.Since(start).Round(time.Millisecond))
}
// collectShutdownACKs publishes a shutdown request and collects ACKs from nodes.
func collectShutdownACKs(nc *nats.Conn, topic string, reqData []byte, nodeCount int, timeout time.Duration) ([]daemon.ShutdownACK, error) {
inbox := nats.NewInbox()
sub, err := nc.SubscribeSync(inbox)
if err != nil {
return nil, fmt.Errorf("failed to subscribe to inbox: %w", err)
}
defer sub.Unsubscribe()
if err := nc.PublishRequest(topic, inbox, reqData); err != nil {
return nil, fmt.Errorf("failed to publish request: %w", err)
}
nc.Flush()
var acks []daemon.ShutdownACK
deadline := time.Now().Add(timeout)
for len(acks) < nodeCount {
remaining := time.Until(deadline)
if remaining <= 0 {
break
}
msg, err := sub.NextMsg(remaining)
if err != nil {
break
}
var ack daemon.ShutdownACK
if err := json.Unmarshal(msg.Data, &ack); err != nil {
continue
}
acks = append(acks, ack)
}
return acks, nil
}
+290
View File
@@ -0,0 +1,290 @@
/*
Copyright © 2025 Mulga Defense Corporation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package cmd
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"time"
"github.com/mulgadc/hive/hive/config"
"github.com/mulgadc/hive/hive/types"
"github.com/mulgadc/hive/hive/utils"
"github.com/nats-io/nats.go"
"github.com/pterm/pterm"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
var getCmd = &cobra.Command{
Use: "get",
Short: "Display cluster resources",
Long: `Display cluster resources such as nodes and VMs.`,
}
var getNodesCmd = &cobra.Command{
Use: "nodes",
Short: "Display cluster nodes",
Long: `Display all physical nodes in the cluster with status, IP, region, and services.`,
Run: runGetNodes,
}
var getVMsCmd = &cobra.Command{
Use: "vms",
Aliases: []string{"instances"},
Short: "Display VMs across the cluster",
Long: `Display all VMs running across the cluster with instance type, resources, and placement.`,
Run: runGetVMs,
}
func init() {
rootCmd.AddCommand(getCmd)
getCmd.AddCommand(getNodesCmd)
getCmd.AddCommand(getVMsCmd)
getCmd.PersistentFlags().Duration("timeout", 3*time.Second, "Timeout for collecting responses from nodes")
}
// loadConfigAndConnect loads the cluster config and connects to NATS.
func loadConfigAndConnect() (*config.ClusterConfig, *nats.Conn, error) {
cfgPath := viper.GetString("config")
if cfgPath == "" {
homeDir, _ := os.UserHomeDir()
cfgPath = fmt.Sprintf("%s/hive/config/hive.toml", homeDir)
}
cfg, err := config.LoadConfig(cfgPath)
if err != nil {
return nil, nil, fmt.Errorf("failed to load config: %w", err)
}
// Backfill BaseDir from config file path when not set in the config.
// Config lives at <baseDir>/config/hive.toml, so baseDir is two levels up.
if nodeConfig, ok := cfg.Nodes[cfg.Node]; ok && nodeConfig.BaseDir == "" {
nodeConfig.BaseDir = filepath.Dir(filepath.Dir(cfgPath))
cfg.Nodes[cfg.Node] = nodeConfig
}
nodeConfig := cfg.Nodes[cfg.Node]
nc, err := utils.ConnectNATS(nodeConfig.NATS.Host, nodeConfig.NATS.ACL.Token)
if err != nil {
return nil, nil, fmt.Errorf("failed to connect to NATS: %w", err)
}
return cfg, nc, nil
}
// collectResponses publishes to a fan-out topic and collects all responses within the timeout.
func collectResponses(nc *nats.Conn, topic string, timeout time.Duration) ([][]byte, error) {
inbox := nats.NewInbox()
sub, err := nc.SubscribeSync(inbox)
if err != nil {
return nil, fmt.Errorf("failed to subscribe to inbox: %w", err)
}
defer sub.Unsubscribe()
if err := nc.PublishRequest(topic, inbox, nil); err != nil {
return nil, fmt.Errorf("failed to publish request: %w", err)
}
nc.Flush()
var responses [][]byte
deadline := time.Now().Add(timeout)
for {
remaining := time.Until(deadline)
if remaining <= 0 {
break
}
msg, err := sub.NextMsg(remaining)
if err != nil {
break // timeout or other error — done collecting
}
responses = append(responses, msg.Data)
}
return responses, nil
}
func formatUptime(seconds int64) string {
if seconds <= 0 {
return "-"
}
d := time.Duration(seconds) * time.Second
days := int(d.Hours()) / 24
hours := int(d.Hours()) % 24
mins := int(d.Minutes()) % 60
if days > 0 {
return fmt.Sprintf("%dd%dh", days, hours)
}
if hours > 0 {
return fmt.Sprintf("%dh%dm", hours, mins)
}
return fmt.Sprintf("%dm", mins)
}
func formatMemGB(gb float64) string {
if gb >= 1.0 {
return fmt.Sprintf("%.1fGi", gb)
}
return fmt.Sprintf("%dMi", int(gb*1024))
}
func runGetNodes(cmd *cobra.Command, args []string) {
cfg, nc, err := loadConfigAndConnect()
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
os.Exit(1)
}
defer nc.Close()
timeout, _ := cmd.Flags().GetDuration("timeout")
responses, err := collectResponses(nc, "hive.node.status", timeout)
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
os.Exit(1)
}
// Parse responses into a map by node name
respondedNodes := make(map[string]types.NodeStatusResponse)
for _, data := range responses {
var resp types.NodeStatusResponse
if err := json.Unmarshal(data, &resp); err != nil {
continue
}
respondedNodes[resp.Node] = resp
}
// Build table: union of config-known nodes + NATS responders
tableData := pterm.TableData{
{"NAME", "STATUS", "IP", "REGION", "AZ", "UPTIME", "VMs", "SERVICES"},
}
// Collect all node names: config + responded (union)
nodeSet := make(map[string]struct{})
for name := range cfg.Nodes {
nodeSet[name] = struct{}{}
}
for name := range respondedNodes {
nodeSet[name] = struct{}{}
}
nodeNames := make([]string, 0, len(nodeSet))
for name := range nodeSet {
nodeNames = append(nodeNames, name)
}
sort.Strings(nodeNames)
for _, name := range nodeNames {
nodeCfg := cfg.Nodes[name]
if resp, ok := respondedNodes[name]; ok {
tableData = append(tableData, []string{
resp.Node,
resp.Status,
resp.Host,
resp.Region,
resp.AZ,
formatUptime(resp.Uptime),
fmt.Sprintf("%d", resp.VMCount),
strings.Join(resp.Services, ","),
})
} else {
tableData = append(tableData, []string{
name,
"NotReady",
nodeCfg.Host,
nodeCfg.Region,
nodeCfg.AZ,
"-",
"-",
strings.Join(nodeCfg.GetServices(), ","),
})
}
}
pterm.DefaultTable.WithHasHeader().WithLeftAlignment().WithData(tableData).Render()
}
func runGetVMs(cmd *cobra.Command, args []string) {
_, nc, err := loadConfigAndConnect()
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
os.Exit(1)
}
defer nc.Close()
timeout, _ := cmd.Flags().GetDuration("timeout")
responses, err := collectResponses(nc, "hive.node.vms", timeout)
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
os.Exit(1)
}
type vmRow struct {
types.VMInfo
Node string
Host string
}
var allVMs []vmRow
for _, data := range responses {
var resp types.NodeVMsResponse
if err := json.Unmarshal(data, &resp); err != nil {
continue
}
for _, v := range resp.VMs {
allVMs = append(allVMs, vmRow{VMInfo: v, Node: resp.Node, Host: resp.Host})
}
}
if len(allVMs) == 0 {
fmt.Println("No VMs found.")
return
}
// Sort by node then instance ID
sort.Slice(allVMs, func(i, j int) bool {
if allVMs[i].Node != allVMs[j].Node {
return allVMs[i].Node < allVMs[j].Node
}
return allVMs[i].InstanceID < allVMs[j].InstanceID
})
tableData := pterm.TableData{
{"INSTANCE", "STATUS", "TYPE", "VCPU", "MEM", "NODE", "IP", "AGE"},
}
for _, v := range allVMs {
age := "-"
if v.LaunchTime > 0 {
age = formatUptime(time.Now().Unix() - v.LaunchTime)
}
tableData = append(tableData, []string{
v.InstanceID,
v.Status,
v.InstanceType,
fmt.Sprintf("%d", v.VCPU),
formatMemGB(v.MemoryGB),
v.Node,
v.Host,
age,
})
}
pterm.DefaultTable.WithHasHeader().WithLeftAlignment().WithData(tableData).Render()
}
+12 -11
View File
@@ -25,8 +25,9 @@ import (
)
var (
cfgFile string
appConfig *config.Config
cfgFile string
//lint:ignore U1000 fixing config loading later
appConfig *config.ClusterConfig
)
// rootCmd represents the base command when called without any subcommands
@@ -34,16 +35,16 @@ var rootCmd = &cobra.Command{
Use: "hive",
Short: "Hive - Open source AWS-compatible platform for secure edge deployments",
Long: `
__/\\\________/\\\__/\\\\\\\\\\\__/\\\________/\\\__/\\\\\\\\\\\\\\\_
_\/\\\_______\/\\\_\/////\\\///__\/\\\_______\/\\\_\/\\\///////////__
_\/\\\_______\/\\\_____\/\\\_____\//\\\______/\\\__\/\\\_____________
_\/\\\\\\\\\\\\\\\_____\/\\\______\//\\\____/\\\___\/\\\\\\\\\\\_____
_\/\\\/////////\\\_____\/\\\_______\//\\\__/\\\____\/\\\///////______
_\/\\\_______\/\\\_____\/\\\________\//\\\/\\\_____\/\\\_____________
_\/\\\_______\/\\\_____\/\\\_________\//\\\\\______\/\\\_____________
_\/\\\_______\/\\\__/\\\\\\\\\\\______\//\\\_______\/\\\\\\\\\\\\\\\_
__/\\\________/\\\__/\\\\\\\\\\\__/\\\________/\\\__/\\\\\\\\\\\\\\\_
_\/\\\_______\/\\\_\/////\\\///__\/\\\_______\/\\\_\/\\\///////////__
_\/\\\_______\/\\\_____\/\\\_____\//\\\______/\\\__\/\\\_____________
_\/\\\\\\\\\\\\\\\_____\/\\\______\//\\\____/\\\___\/\\\\\\\\\\\_____
_\/\\\/////////\\\_____\/\\\_______\//\\\__/\\\____\/\\\///////______
_\/\\\_______\/\\\_____\/\\\________\//\\\/\\\_____\/\\\_____________
_\/\\\_______\/\\\_____\/\\\_________\//\\\\\______\/\\\_____________
_\/\\\_______\/\\\__/\\\\\\\\\\\______\//\\\_______\/\\\\\\\\\\\\\\\_
_\///________\///__\///////////________\///________\///////////////__
Hive – Open source AWS-compatible platform for secure edge deployments.
Run EC2, VPC, S3, and EBS-like services on bare metal with full control.
Built for environments where running in the cloud isn’t an option.
+275 -34
View File
@@ -23,9 +23,12 @@ import (
"github.com/mulgadc/hive/hive/config"
"github.com/mulgadc/hive/hive/service"
"github.com/mulgadc/hive/hive/services/hiveui"
"github.com/mulgadc/hive/hive/services/nats"
"github.com/mulgadc/hive/hive/services/predastore"
"github.com/mulgadc/hive/hive/services/viperblockd"
"github.com/mulgadc/hive/hive/services/vpcd"
"github.com/mulgadc/predastore/s3"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
@@ -60,6 +63,17 @@ var awsgwCmd = &cobra.Command{
Short: "Manage the awsgw (AWS gateway) service",
}
var vpcdCmd = &cobra.Command{
Use: "vpcd",
Short: "Manage the vpcd (VPC daemon) service",
}
var hiveUICmd = &cobra.Command{
Use: "hive-ui",
Aliases: []string{"ui", "hiveui"},
Short: "Manage the hive-ui service",
}
var predastoreStartCmd = &cobra.Command{
Use: "start",
Short: "Start the predastore service",
@@ -100,6 +114,19 @@ var predastoreStartCmd = &cobra.Command{
return
}
backendType := viper.GetString("backend")
var backend s3.BackendType
if backendType == "distributed" {
backend = s3.BackendDistributed
} else {
backend = s3.BackendFilesystem
}
nodeID := viper.GetInt("node-id")
pprofEnabled := viper.GetBool("pprof")
pprofOutput := viper.GetString("pprof-output")
service, err := service.New("predastore", &predastore.Config{
Port: port,
Host: host,
@@ -108,6 +135,12 @@ var predastoreStartCmd = &cobra.Command{
Debug: debug,
TlsCert: tlsCert,
TlsKey: tlsKey,
Backend: backend,
NodeID: nodeID,
PprofEnabled: pprofEnabled,
PprofOutputPath: pprofOutput,
})
if err != nil {
@@ -167,7 +200,8 @@ var viperblockStartCmd = &cobra.Command{
fmt.Println("Loading config from:", cfgFile)
// TODO: Support ENV vars, CLI, otherwise revert to config.LoadConfig()
appConfig, err := config.LoadConfig(cfgFile)
clusterConfig, err := config.LoadConfig(cfgFile)
nodeConfig := clusterConfig.Nodes[clusterConfig.Node]
if err != nil {
fmt.Println("Error loading config file:", err)
@@ -178,48 +212,51 @@ var viperblockStartCmd = &cobra.Command{
if natsHost != "" {
fmt.Println("Overwriting natsHost:", natsHost)
appConfig.NATS.Host = natsHost
nodeConfig.NATS.Host = natsHost
}
s3Host := viper.GetString("s3-host")
if s3Host != "" {
fmt.Println("Overwriting s3host:", s3Host)
appConfig.Predastore.Host = s3Host
nodeConfig.Predastore.Host = s3Host
}
s3Bucket := viper.GetString("s3-bucket")
if s3Bucket != "" {
fmt.Println("Overwriting s3bucket:", s3Bucket)
appConfig.Predastore.Bucket = s3Bucket
nodeConfig.Predastore.Bucket = s3Bucket
}
s3Region := viper.GetString("s3-region")
if s3Region != "" {
fmt.Println("Overwriting s3Region:", s3Region)
appConfig.Predastore.Region = s3Region
nodeConfig.Predastore.Region = s3Region
}
accessKey := viper.GetString("access-key")
if accessKey != "" {
fmt.Println("Overwriting access-key:", accessKey)
appConfig.Predastore.AccessKey = accessKey
nodeConfig.Predastore.AccessKey = accessKey
}
secretKey := viper.GetString("secret-key")
if secretKey != "" {
fmt.Println("Overwriting secret-key:", secretKey)
appConfig.Predastore.SecretKey = secretKey
nodeConfig.Predastore.SecretKey = secretKey
}
baseDir := viper.GetString("base-dir")
if baseDir != "" {
fmt.Println("Overwriting base-dir:", baseDir)
appConfig.Predastore.BaseDir = baseDir
nodeConfig.Predastore.BaseDir = baseDir
}
// Apply changes back to cluster config
clusterConfig.Nodes[clusterConfig.Node] = nodeConfig
pluginPath := viper.GetString("plugin-path")
if pluginPath == "" {
@@ -235,15 +272,23 @@ var viperblockStartCmd = &cobra.Command{
os.Exit(1)
}
// Resolve sharded WAL setting: default false unless explicitly set to true
shardWAL := false
if nodeConfig.Viperblock.ShardWAL != nil {
shardWAL = *nodeConfig.Viperblock.ShardWAL
}
service, err := service.New("viperblock", &viperblockd.Config{
NatsHost: appConfig.NATS.Host,
NatsHost: nodeConfig.NATS.Host,
PluginPath: pluginPath,
S3Host: appConfig.Predastore.Host,
Bucket: appConfig.Predastore.Bucket,
Region: appConfig.Predastore.Region,
AccessKey: appConfig.Predastore.AccessKey,
SecretKey: appConfig.Predastore.SecretKey,
BaseDir: appConfig.Predastore.BaseDir,
S3Host: nodeConfig.Predastore.Host,
Bucket: nodeConfig.Predastore.Bucket,
Region: nodeConfig.Predastore.Region,
AccessKey: nodeConfig.Predastore.AccessKey,
SecretKey: nodeConfig.Predastore.SecretKey,
BaseDir: nodeConfig.Predastore.BaseDir,
NodeName: clusterConfig.Node,
ShardWAL: shardWAL,
})
if err != nil {
@@ -303,12 +348,15 @@ var natsStartCmd = &cobra.Command{
dataDir := viper.GetString("data-dir")
jetStream := viper.GetBool("jetstream")
cfgFile := viper.GetString("config")
service, err := service.New("nats", &nats.Config{
Port: port,
Host: host,
Debug: debug,
DataDir: dataDir,
JetStream: jetStream,
ConfigFile: cfgFile,
Port: port,
Host: host,
Debug: debug,
DataDir: dataDir,
JetStream: jetStream,
})
if err != nil {
@@ -363,7 +411,8 @@ var hiveStartCmd = &cobra.Command{
}
// TODO: Support ENV vars, CLI, otherwise revert to config.LoadConfig()
appConfig, err := config.LoadConfig(cfgFile)
clusterConfig, err := config.LoadConfig(cfgFile)
nodeConfig := clusterConfig.Nodes[clusterConfig.Node]
if err != nil {
fmt.Println("Error loading config file:", err)
@@ -375,7 +424,7 @@ var hiveStartCmd = &cobra.Command{
if baseDir != "" {
fmt.Println("Overwriting base-dir to:", baseDir)
appConfig.BaseDir = baseDir
nodeConfig.BaseDir = baseDir
}
// Overwrite defaults (CLI first, config second, env third)
@@ -383,17 +432,25 @@ var hiveStartCmd = &cobra.Command{
if walDir != "" {
fmt.Println("Overwriting wal-dir to:", walDir)
appConfig.WalDir = walDir
nodeConfig.WalDir = walDir
}
service, err := service.New("hive", appConfig)
// Apply changes back to cluster config
clusterConfig.Nodes[clusterConfig.Node] = nodeConfig
svc, err := service.New("hive", clusterConfig)
if err != nil {
fmt.Println("Error starting hive service:", err)
return
}
service.Start()
// Set config path for cluster manager
if hiveSvc, ok := svc.(interface{ SetConfigPath(string) }); ok {
hiveSvc.SetConfigPath(cfgFile)
}
svc.Start()
fmt.Println("HIVE service started")
},
}
@@ -404,7 +461,7 @@ var hiveStopCmd = &cobra.Command{
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("Stopping hive service...")
service, err := service.New("hive", &config.Config{})
service, err := service.New("hive", &config.ClusterConfig{})
if err != nil {
fmt.Println("Error stopping hive service:", err)
@@ -443,7 +500,8 @@ var awsgwStartCmd = &cobra.Command{
fmt.Println("Loading config from:", cfgFile)
// TODO: Support ENV vars, CLI, otherwise revert to config.LoadConfig()
appConfig, err := config.LoadConfig(cfgFile)
clusterConfig, err := config.LoadConfig(cfgFile)
nodeConfig := clusterConfig.Nodes[clusterConfig.Node]
if err != nil {
fmt.Println("Error loading config file:", err)
@@ -454,30 +512,33 @@ var awsgwStartCmd = &cobra.Command{
awsgwHost := viper.GetString("host")
if awsgwHost != "" {
fmt.Println("Overwriting awsgw host to:", awsgwHost)
appConfig.AWSGW.Host = awsgwHost
//nodeConfig.AWSGW.Host = awsgwHost
}
awsgwTlsCert := viper.GetString("tls-cert")
if awsgwTlsCert != "" {
fmt.Println("Overwriting awsgw tls-cert to:", awsgwTlsCert)
appConfig.AWSGW.TLSCert = awsgwTlsCert
nodeConfig.AWSGW.TLSCert = awsgwTlsCert
}
awsgwTlsKey := viper.GetString("tls-key")
if awsgwTlsKey != "" {
fmt.Println("Overwriting awsgw tls-key to:", awsgwTlsKey)
appConfig.AWSGW.TLSKey = awsgwTlsKey
nodeConfig.AWSGW.TLSKey = awsgwTlsKey
}
baseDir := viper.GetString("base-dir")
if awsgwTlsKey != "" {
if baseDir != "" {
fmt.Println("Overwriting awsgw base-dir to:", baseDir)
appConfig.BaseDir = baseDir
nodeConfig.BaseDir = baseDir
}
service, err := service.New("awsgw", appConfig)
// Apply changes back to cluster config
clusterConfig.Nodes[clusterConfig.Node] = nodeConfig
service, err := service.New("awsgw", clusterConfig)
if err != nil {
fmt.Println("Error starting awsgw service:", err)
@@ -495,7 +556,7 @@ var awsgwStopCmd = &cobra.Command{
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("Stopping awsgw service...")
service, err := service.New("awsgw", &config.Config{})
service, err := service.New("awsgw", &config.ClusterConfig{})
if err != nil {
fmt.Println("Error stopping awsgw service:", err)
@@ -516,6 +577,136 @@ var awsgwStatusCmd = &cobra.Command{
},
}
var hiveUIStartCmd = &cobra.Command{
Use: "start",
Short: "Start the hive-ui service",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("Starting hive-ui service...")
port := viper.GetInt("hive-ui-port")
host := viper.GetString("hive-ui-host")
tlsCert := viper.GetString("hive-ui-tls-cert")
tlsKey := viper.GetString("hive-ui-tls-key")
svc, err := service.New("hive-ui", &hiveui.Config{
Port: port,
Host: host,
TLSCert: tlsCert,
TLSKey: tlsKey,
})
if err != nil {
fmt.Println("Error starting hive-ui service:", err)
return
}
svc.Start()
fmt.Println("hive-ui service started")
},
}
var hiveUIStopCmd = &cobra.Command{
Use: "stop",
Short: "Stop the hive-ui service",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("Stopping hive-ui service...")
svc, err := service.New("hive-ui", &hiveui.Config{})
if err != nil {
fmt.Println("Error stopping hive-ui service:", err)
return
}
svc.Stop()
fmt.Println("hive-ui service stopped")
},
}
var hiveUIStatusCmd = &cobra.Command{
Use: "status",
Short: "Get status of the hive-ui service",
Run: func(cmd *cobra.Command, args []string) {
svc, err := service.New("hive-ui", &hiveui.Config{})
if err != nil {
fmt.Println("Error getting hive-ui service status:", err)
return
}
status, err := svc.Status()
if err != nil {
fmt.Println("Error getting hive-ui service status:", err)
return
}
fmt.Println("hive-ui service status:", status)
},
}
var vpcdStartCmd = &cobra.Command{
Use: "start",
Short: "Start the vpcd service",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("Starting vpcd service...")
cfgFile := viper.GetString("config")
if cfgFile == "" {
fmt.Println("Config file is not set")
return
}
clusterConfig, err := config.LoadConfig(cfgFile)
if err != nil {
fmt.Println("Error loading config file:", err)
return
}
nodeConfig := clusterConfig.Nodes[clusterConfig.Node]
svc, err := service.New("vpcd", &vpcd.Config{
NatsHost: nodeConfig.NATS.Host,
NatsToken: nodeConfig.NATS.ACL.Token,
OVNNBAddr: nodeConfig.VPCD.OVNNBAddr,
OVNSBAddr: nodeConfig.VPCD.OVNSBAddr,
BaseDir: nodeConfig.BaseDir,
Debug: false,
})
if err != nil {
fmt.Println("Error starting vpcd service:", err)
return
}
svc.Start()
fmt.Println("vpcd service started")
},
}
var vpcdStopCmd = &cobra.Command{
Use: "stop",
Short: "Stop the vpcd service",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("Stopping vpcd service...")
svc, err := service.New("vpcd", &vpcd.Config{})
if err != nil {
fmt.Println("Error stopping vpcd service:", err)
return
}
svc.Stop()
fmt.Println("vpcd service stopped")
},
}
var vpcdStatusCmd = &cobra.Command{
Use: "status",
Short: "Get status of the vpcd service",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("vpcd service status: ...")
},
}
func init() {
viper.SetEnvPrefix("HIVE") // Prefix for environment variables
@@ -562,6 +753,26 @@ func init() {
viper.BindEnv("tls-key", "HIVE_PREDASTORE_TLS_KEY")
viper.BindPFlag("tls-key", predastoreCmd.PersistentFlags().Lookup("tls-key"))
// Predastore Backend
predastoreCmd.PersistentFlags().String("backend", "distributed", "Predastore (S3) backend")
viper.BindEnv("backend", "HIVE_PREDASTORE_BACKEND")
viper.BindPFlag("backend", predastoreCmd.PersistentFlags().Lookup("backend"))
// Predastore Node ID
predastoreCmd.PersistentFlags().Int("node-id", 0, "Predastore (S3) node ID")
viper.BindEnv("node-id", "HIVE_PREDASTORE_NODE_ID")
viper.BindPFlag("node-id", predastoreCmd.PersistentFlags().Lookup("node-id"))
// Predastore CPU Profiling
predastoreCmd.PersistentFlags().Bool("pprof", false, "Enable CPU profiling (also via PPROF_ENABLED=1)")
viper.BindEnv("pprof", "PPROF_ENABLED")
viper.BindPFlag("pprof", predastoreCmd.PersistentFlags().Lookup("pprof"))
// Predastore CPU Profile Output Path
predastoreCmd.PersistentFlags().String("pprof-output", "/tmp/predastore-cpu.prof", "CPU profile output path")
viper.BindEnv("pprof-output", "PPROF_OUTPUT")
viper.BindPFlag("pprof-output", predastoreCmd.PersistentFlags().Lookup("pprof-output"))
predastoreCmd.AddCommand(predastoreStartCmd)
predastoreCmd.AddCommand(predastoreStopCmd)
predastoreCmd.AddCommand(predastoreStatusCmd)
@@ -652,4 +863,34 @@ func init() {
awsgwCmd.AddCommand(awsgwStopCmd)
awsgwCmd.AddCommand(awsgwStatusCmd)
// hive-ui
serviceCmd.AddCommand(hiveUICmd)
hiveUICmd.PersistentFlags().Int("port", 3000, "hive-ui server port")
viper.BindEnv("hive-ui-port", "HIVE_UI_PORT")
viper.BindPFlag("hive-ui-port", hiveUICmd.PersistentFlags().Lookup("port"))
hiveUICmd.PersistentFlags().String("host", "0.0.0.0", "hive-ui server host")
viper.BindEnv("hive-ui-host", "HIVE_UI_HOST")
viper.BindPFlag("hive-ui-host", hiveUICmd.PersistentFlags().Lookup("host"))
hiveUICmd.PersistentFlags().String("tls-cert", "", "TLS certificate path")
viper.BindEnv("hive-ui-tls-cert", "HIVE_UI_TLS_CERT")
viper.BindPFlag("hive-ui-tls-cert", hiveUICmd.PersistentFlags().Lookup("tls-cert"))
hiveUICmd.PersistentFlags().String("tls-key", "", "TLS key path")
viper.BindEnv("hive-ui-tls-key", "HIVE_UI_TLS_KEY")
viper.BindPFlag("hive-ui-tls-key", hiveUICmd.PersistentFlags().Lookup("tls-key"))
hiveUICmd.AddCommand(hiveUIStartCmd)
hiveUICmd.AddCommand(hiveUIStopCmd)
hiveUICmd.AddCommand(hiveUIStatusCmd)
// vpcd
serviceCmd.AddCommand(vpcdCmd)
vpcdCmd.AddCommand(vpcdStartCmd)
vpcdCmd.AddCommand(vpcdStopCmd)
vpcdCmd.AddCommand(vpcdStatusCmd)
}
-7
View File
@@ -4,10 +4,3 @@ region = "{{.Region}}"
# Enable verbose debugging
debug = false
[[auth]]
access_key_id = "{{.AccessKey}}"
secret_access_key = "{{.SecretKey}}"
policy = [
{ bucket = "predastore", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
]
+45 -15
View File
@@ -1,34 +1,64 @@
# Domain configuration file in TOML format.
version = "1.0"
epoch = 1
node = "{{.Node}}"
[nodes.{{.Node}}]
node = "{{.Node}}"
host = "{{.BindIP}}"
region = "{{.Region}}"
az = "{{.Az}}"
{{- if .Services}}
services = [{{range $i, $s := .Services}}{{if $i}}, {{end}}"{{$s}}"{{end}}]
{{- end}}
account_id = "{{.AccountID}}"
access_key = "{{.AccessKey}}"
secret_key = "{{.SecretKey}}"
[daemon]
host = "0.0.0.0:4432"
[nodes.{{.Node}}.daemon]
host = "{{.BindIP}}:{{.Port}}"
tlskey = "config/server.key"
tlscert = "config/server.pem"
dev_networking = true
[awsgw]
host = "0.0.0.0:9999"
[nodes.{{.Node}}.awsgw]
host = "{{.BindIP}}:9999"
tlskey = "config/server.key"
tlscert = "config/server.pem"
debug = true
config = "awsgw/awsgw.toml"
[nats]
host = "0.0.0.0:4222"
[nodes.{{.Node}}.nats]
host = "{{.BindIP}}:4222"
[nats.acl]
[nodes.{{.Node}}.nats.acl]
token = "{{.NatsToken}}"
[nats.sub]
[nodes.{{.Node}}.nats.sub]
[predastore]
host = "0.0.0.0:8443"
[nodes.{{.Node}}.viperblock]
shardwal = false
[nodes.{{.Node}}.vpcd]
ovn_nb_addr = "tcp:127.0.0.1:6641"
ovn_sb_addr = "tcp:127.0.0.1:6642"
[nodes.{{.Node}}.predastore]
host = "{{.BindIP}}:8443"
region = "{{.Region}}"
bucket = "predastore"
access_key = "{{.AccessKey}}"
secret_key = "{{.SecretKey}}"
accesskey = "{{.AccessKey}}"
secretkey = "{{.SecretKey}}"
{{- if gt .PredastoreNodeID 0}}
node_id = {{.PredastoreNodeID}}
{{- end}}
{{- range .RemoteNodes}}
[nodes.{{.Name}}]
node = "{{.Name}}"
host = "{{.Host}}"
region = "{{.Region}}"
az = "{{.AZ}}"
{{- if .Services}}
services = [{{range $i, $s := .Services}}{{if $i}}, {{end}}"{{$s}}"{{end}}]
{{- end}}
{{- end}}
+29 -5
View File
@@ -2,17 +2,36 @@
# Generated by Hive admin init
# Server configuration
port: 4222
server_name: hive-nats
{{- if .BindIP }}
listen: {{.BindIP}}:4222
{{- else }}
listen: 0.0.0.0:4222
{{- end }}
{{- if and .ClusterBindIP .ClusterRoutes }}
cluster {
name: {{ .ClusterName }}
listen: {{.ClusterBindIP}}:4248
routes = [
{{- range .ClusterRoutes }}
"nats://{{ . }}",
{{- end }}
]
}
{{- end }}
server_name: hive-nats-{{.Node}}
# Authorization
authorization {
token: "{{.NatsToken}}"
# token: "{{.NatsToken}}"
}
# JetStream for persistent messaging
# JetStream for persistent messaging and KV store
jetstream {
store_dir: "{{.DataDir}}/nats/storage"
store_dir: "{{.DataDir}}/nats/"
max_memory_store: 1GB
max_file_store: 10GB
}
@@ -29,4 +48,9 @@ max_payload: 1MB
max_pending: 64MB
# Monitoring
{{- if .BindIP }}
http: {{.BindIP}}:8222
{{- else }}
http_port: 8222
{{- end }}
@@ -0,0 +1,48 @@
# Predastore S3-compatible storage configuration file in TOML format.
# Generated for multi-node cluster
version = "1.0"
region = "{{.Region}}"
host = "0.0.0.0"
port = 8443
debug = false
[rs]
data = 2
parity = 1
{{- range .Nodes}}
[[db]]
id = {{.ID}}
host = "{{.Host}}"
port = 6660
path = "distributed/db/node-{{.ID}}/"
access_key_id = "{{$.AccessKey}}"
secret_access_key = "{{$.SecretKey}}"
{{- if eq .ID 1}}
leader = true
{{- end}}
{{- end}}
{{- range .Nodes}}
[[nodes]]
id = {{.ID}}
host = "{{.Host}}"
port = 9991
path = "distributed/nodes/node-{{.ID}}/"
{{- end}}
[[buckets]]
name = "predastore"
region = "{{.Region}}"
type = "distributed"
public = false
encryption = ""
[[auth]]
access_key_id = "{{.AccessKey}}"
secret_access_key = "{{.SecretKey}}"
policy = [
{ bucket = "predastore", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
]
+58 -4
View File
@@ -1,4 +1,5 @@
# Predastore S3-compatible storage configuration file in TOML format.
# MULTI NODE used for testing
version = "1.0"
region = "{{.Region}}"
@@ -12,11 +13,64 @@ debug = false
# Optionally configure base-path if full directory path not defined for "fs" buckets
# base_path = "/home/predastore/data/"
# Specify reed solomon encoding -
# RS(2,1), 2 data, 1 parity. Minimum
# RS(3,2), 3 data, 2 parity. Recommended for production
[rs]
data = 2
parity = 1
[[db]]
id = 1
host = "127.0.0.1"
port = 6660
#raft_port = 7660 # Optional: Raft consensus port (default: port + 1000)
#advertise_host = "" # Optional: Address to advertise to other Raft nodes
path = "distributed/db/node-1/"
access_key_id = "{{.AccessKey}}"
secret_access_key = "{{.SecretKey}}"
leader = true # Bootstrap leader hint
[[db]]
id = 2
host = "127.0.0.1"
port = 6661
path = "distributed/db/node-2/"
access_key_id = "{{.AccessKey}}"
secret_access_key = "{{.SecretKey}}"
[[db]]
id = 3
host = "127.0.0.1"
port = 6662
path = "distributed/db/node-3/"
access_key_id = "{{.AccessKey}}"
secret_access_key = "{{.SecretKey}}"
# Shard nodes
# Using distributed hash ring for object > shard placement.
[[nodes]]
id = 1
host = "0.0.0.0"
port = 9991
path = "distributed/nodes/node-1/"
[[nodes]]
id = 2
host = "0.0.0.0"
port = 9992
path = "distributed/nodes/node-2/"
[[nodes]]
id = 3
host = "0.0.0.0"
port = 9993
path = "distributed/nodes/node-3/"
[[buckets]]
name = "predastore"
region = "{{.Region}}"
type = "fs"
pathname = "data/predastore"
region = "ap-southeast-2"
type = "distributed"
public = false
encryption = ""
@@ -25,4 +79,4 @@ access_key_id = "{{.AccessKey}}"
secret_access_key = "{{.SecretKey}}"
policy = [
{ bucket = "predastore", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
]
]
+161
View File
@@ -0,0 +1,161 @@
/*
Copyright © 2025 Mulga Defense Corporation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package cmd
import (
"encoding/json"
"fmt"
"os"
"sort"
"time"
"github.com/mulgadc/hive/hive/types"
"github.com/pterm/pterm"
"github.com/spf13/cobra"
)
var topCmd = &cobra.Command{
Use: "top",
Short: "Display cluster resource usage",
Long: `Display resource usage (CPU, memory) for cluster nodes.`,
}
var topNodesCmd = &cobra.Command{
Use: "nodes",
Short: "Display resource usage per node",
Long: `Display CPU and memory usage per node, plus a summary of available instance types across the cluster.`,
Run: runTopNodes,
}
func init() {
rootCmd.AddCommand(topCmd)
topCmd.AddCommand(topNodesCmd)
topCmd.PersistentFlags().Duration("timeout", 3*time.Second, "Timeout for collecting responses from nodes")
}
func runTopNodes(cmd *cobra.Command, args []string) {
cfg, nc, err := loadConfigAndConnect()
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
os.Exit(1)
}
defer nc.Close()
timeout, _ := cmd.Flags().GetDuration("timeout")
responses, err := collectResponses(nc, "hive.node.status", timeout)
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
os.Exit(1)
}
respondedNodes := make(map[string]types.NodeStatusResponse)
for _, data := range responses {
var resp types.NodeStatusResponse
if err := json.Unmarshal(data, &resp); err != nil {
continue
}
respondedNodes[resp.Node] = resp
}
// Node resource table: union of config-known nodes + NATS responders
nodeSet := make(map[string]struct{})
for name := range cfg.Nodes {
nodeSet[name] = struct{}{}
}
for name := range respondedNodes {
nodeSet[name] = struct{}{}
}
nodeNames := make([]string, 0, len(nodeSet))
for name := range nodeSet {
nodeNames = append(nodeNames, name)
}
sort.Strings(nodeNames)
nodeTable := pterm.TableData{
{"NAME", "CPU (used/total)", "MEM (used/total)", "VMs"},
}
// Aggregate instance type capacity across all nodes
capacityMap := make(map[string]*aggregatedCap)
for _, name := range nodeNames {
if resp, ok := respondedNodes[name]; ok {
nodeTable = append(nodeTable, []string{
resp.Node,
fmt.Sprintf("%d/%d", resp.AllocVCPU, resp.TotalVCPU),
fmt.Sprintf("%s/%s", formatMemGB(resp.AllocMemGB), formatMemGB(resp.TotalMemGB)),
fmt.Sprintf("%d", resp.VMCount),
})
for _, cap := range resp.InstanceTypes {
if agg, ok := capacityMap[cap.Name]; ok {
agg.Available += cap.Available
} else {
capacityMap[cap.Name] = &aggregatedCap{
VCPU: cap.VCPU,
MemoryGB: cap.MemoryGB,
Available: cap.Available,
}
}
}
} else {
nodeTable = append(nodeTable, []string{
name,
"-",
"-",
"-",
})
}
}
pterm.DefaultTable.WithHasHeader().WithLeftAlignment().WithData(nodeTable).Render()
// Instance type capacity summary
if len(capacityMap) == 0 {
return
}
fmt.Println()
capNames := make([]string, 0, len(capacityMap))
for name := range capacityMap {
capNames = append(capNames, name)
}
sort.Strings(capNames)
capTable := pterm.TableData{
{"INSTANCE TYPE", "AVAILABLE", "VCPU", "MEMORY"},
}
for _, name := range capNames {
agg := capacityMap[name]
capTable = append(capTable, []string{
name,
fmt.Sprintf("%d", agg.Available),
fmt.Sprintf("%d", agg.VCPU),
formatMemGB(agg.MemoryGB),
})
}
pterm.DefaultTable.WithHasHeader().WithLeftAlignment().WithData(capTable).Render()
}
type aggregatedCap struct {
VCPU int
MemoryGB float64
Available int
}
+1 -1
View File
@@ -5,7 +5,7 @@ Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
+80
View File
@@ -0,0 +1,80 @@
# Predastore S3-compatible storage configuration file in TOML format.
# MULTI NODE used for testing
version = "1.0"
region = "ap-southeast-2"
# Optionally configure port and host, or specify via daemon CLI options
host = "0.0.0.0"
port = 8443
# Enable verbose debugging
debug = false
# Optionally configure base-path if full directory path not defined for "fs" buckets
# base_path = "/home/predastore/data/"
# Specify reed solomon encoding
[rs]
data = 2
parity = 1
[[db]]
id = 1
host = "127.0.0.1"
port = 6660
#raft_port = 7660 # Optional: Raft consensus port (default: port + 1000)
#advertise_host = "" # Optional: Address to advertise to other Raft nodes
path = "distributed/db/node-1/"
access_key_id = "X"
secret_access_key = "X"
leader = true # Bootstrap leader hint
[[db]]
id = 2
host = "127.0.0.1"
port = 6661
path = "distributed/db/node-2/"
access_key_id = "X"
secret_access_key = "X"
[[db]]
id = 3
host = "127.0.0.1"
port = 6662
path = "distributed/db/node-3/"
access_key_id = "X"
secret_access_key = "X"
[[nodes]]
id = 1
host = "0.0.0.0"
port = 9991
path = "distributed/nodes/node-1/"
[[nodes]]
id = 2
host = "0.0.0.0"
port = 9992
path = "distributed/nodes/node-2/"
[[nodes]]
id = 3
host = "0.0.0.0"
port = 9993
path = "distributed/nodes/node-3/"
[[buckets]]
name = "predastore"
region = "ap-southeast-2"
type = "distributed"
public = false
encryption = ""
[[auth]]
access_key_id = "X"
secret_access_key = "X"
policy = [
{ bucket = "predastore", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
]
+479
View File
@@ -0,0 +1,479 @@
# Command Implementation Matrix
## Hive Admin CLI
Platform management commands not exposed via the AWS gateway API. These are CLI-only commands.
### Service Management
Service lifecycle commands for starting, stopping, and checking status of all Hive cluster services. Each service subcommand supports `start`, `stop`, and `status` operations.
| Command | Flags | Prerequisites | Basic Logic | Status |
|---------|-------|---------------|-------------|--------|
| `hive service predastore start` | `--port` (default: 8443), `--host` (default: 0.0.0.0), `--base-path`, `--config-path`, `--debug`, `--tls-cert`, `--tls-key`, `--backend` (distributed/filesystem, default: distributed), `--node-id` (default: 0), `--pprof`, `--pprof-output` | TLS cert/key, base-path, config-path required | Creates predastore service instance with S3-compatible storage backend → starts service | **DONE** |
| `hive service predastore stop` | — | Predastore must be running | Stops the predastore service | **DONE** |
| `hive service predastore status` | — | None | Reports predastore service status | **DONE** |
| `hive service viperblock start` | `--s3-host` (default: 0.0.0.0:8443), `--s3-bucket` (default: predastore), `--s3-region` (default: ap-southeast-2), `--plugin-path` (default: /opt/hive/lib/nbdkit-viperblock-plugin.so) | `--config` required, plugin-path must exist on disk | Loads cluster config → connects to NATS and Predastore → starts viperblock block storage service with NBD plugin | **DONE** |
| `hive service viperblock stop` | — | Viperblock must be running | Stops the viperblock service | **DONE** |
| `hive service viperblock status` | — | None | Reports viperblock service status | **DONE** |
| `hive service nats start` | `--port` (default: 4222), `--host` (default: 0.0.0.0), `--debug`, `--data-dir`, `--jetstream` | None | Starts embedded NATS server with optional JetStream | **DONE** |
| `hive service nats stop` | — | NATS must be running | Stops the NATS service | **DONE** |
| `hive service nats status` | — | None | Reports NATS service status | **DONE** |
| `hive service hive start` | `--wal-dir` | `--config` required | Loads cluster config → starts hive daemon (VM orchestration, NATS subscriptions, health endpoint) | **DONE** |
| `hive service hive stop` | — | Hive daemon must be running | Stops the hive daemon service | **DONE** |
| `hive service hive status` | — | None | Reports hive daemon service status | **DONE** |
| `hive service awsgw start` | `--host` (default: 0.0.0.0:9999), `--tls-cert`, `--tls-key`, `--debug` | `--config` required | Loads cluster config → starts AWS-compatible gateway with SigV4 auth, IAM policy enforcement, TLS | **DONE** |
| `hive service awsgw stop` | — | AWS gateway must be running | Stops the AWS gateway service | **DONE** |
| `hive service awsgw status` | — | None | Reports AWS gateway service status | **DONE** |
| `hive service hive-ui start` | `--port` (default: 3000), `--host` (default: 0.0.0.0), `--tls-cert`, `--tls-key` | None | Starts embedded web UI server serving the React frontend. Aliases: `ui`, `hiveui` | **DONE** |
| `hive service hive-ui stop` | — | hive-ui must be running | Stops the hive-ui service | **DONE** |
| `hive service hive-ui status` | — | None | Reports hive-ui service status | **DONE** |
| `hive service vpcd start` | — | `--config` required, OVN/OVS installed | Loads cluster config → starts VPC daemon (subscribes to `vpc.*` NATS events, translates to OVN logical switches/ports/routers) | **DONE** |
| `hive service vpcd stop` | — | vpcd must be running | Stops the vpcd service | **DONE** |
| `hive service vpcd status` | — | None | Reports vpcd service status | **DONE** |
### Cluster Inspection
Operational commands for inspecting cluster state. These fan out NATS requests to all nodes and aggregate responses.
| Command | Flags | Prerequisites | Basic Logic | Output Columns | Status |
|---------|-------|---------------|-------------|----------------|--------|
| `hive get nodes` | `--timeout` (default: 3s) | Cluster must be running (NATS) | Loads config → publishes to `hive.node.status` fan-out topic → collects responses within timeout → merges config-known nodes with NATS responders → nodes that don't respond shown as `NotReady` | NAME, STATUS, IP, REGION, AZ, UPTIME, VMs, SERVICES | **DONE** |
| `hive get vms` | `--timeout` (default: 3s) | Cluster must be running (NATS) | Publishes to `hive.node.vms` fan-out topic → collects VM info from all nodes → sorts by node then instance ID → prints table. Alias: `hive get instances` | INSTANCE, STATUS, TYPE, VCPU, MEM, NODE, IP, AGE | **DONE** |
### Resource Monitoring
| Command | Flags | Prerequisites | Basic Logic | Output | Status |
|---------|-------|---------------|-------------|--------|--------|
| `hive top nodes` | `--timeout` (default: 3s) | Cluster must be running (NATS) | Publishes to `hive.node.status` fan-out topic → collects CPU/memory usage per node → aggregates instance type capacity across all nodes → prints two tables: per-node resource usage and cluster-wide instance type availability | Table 1: NAME, CPU (used/total), MEM (used/total), VMs. Table 2: INSTANCE TYPE, AVAILABLE, VCPU, MEMORY | **DONE** |
### Cluster Initialization
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------|---------------|-------------|------------|--------|
| `hive admin init` | `--nodes`, `--node`, `--bind`, `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--services`, `--formation-timeout`, `--force` | None (first-time setup) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → creates NATS config with auth token → writes hive.toml, awsgw.toml, predastore.toml → configures AWS CLI `hive` profile → creates directory structure under `~/hive/` | 1. Init creates all config files<br>2. Root credentials printed once<br>3. master.key is 32 bytes, mode 0600<br>4. bootstrap.json consumed on first start<br>5. `--force` re-initializes existing config<br>6. AWS CLI profile `hive` auto-configured | **DONE** |
| `hive admin join` | `--host` (required), `--node` (required), `--bind`, `--port`, `--region`, `--az`, `--cluster-bind`, `--cluster-routes`, `--data-dir`, `--services` | Leader node must be running | Connects to leader node → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations | 1. Join existing cluster<br>2. Missing host (error)<br>3. Missing node name (error) | **DONE** |
### Cluster Operations
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------|---------------|-------------|------------|--------|
| `hive admin cluster shutdown` | — | Cluster must be running | Gracefully shuts down the entire cluster | 1. Shutdown running cluster<br>2. All nodes stop cleanly | **DONE** |
### Account Management
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------|---------------|-------------|------------|--------|
| `hive admin account create` | `--name` | Cluster must be running (NATS), master key must exist | Connects to NATS → CAS loop on `hive-account-counter:next_id` for sequential 12-digit ID → creates Account record in `hive-accounts` KV → creates `admin` user in new account → creates access key for admin → creates AdministratorAccess policy (Action:*, Resource:*) → attaches policy → prints credentials | 1. Create first account (ID 000000000001)<br>2. Sequential IDs (000000000002, etc.)<br>3. Admin user has full access<br>4. Credentials printed once | **DONE** |
| `hive admin account list` | — | Cluster must be running (NATS), master key must exist | Connects to NATS → IAMService.ListAccounts() → prints table with Account ID, Name, Status, Created | 1. List all accounts<br>2. Empty list (only global account) | **DONE** |
### Image Management
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------|---------------|-------------|------------|--------|
| `hive admin images import` | `--name` | Cluster must be running, image file must exist locally | Imports a local disk image (QCOW2) as an AMI → uploads to Predastore S3 → registers as available AMI | 1. Import valid QCOW2 image<br>2. Image appears in `describe-images` | **DONE** |
| `hive admin images list` | — | None | Lists available OS images that can be imported or downloaded | 1. List available images | **DONE** |
## AWS Commands
### EC2 - Instance Management
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `run-instances` | `--image-id`, `--instance-type`, `--count` (Min/MaxCount), `--key-name`, `--user-data`, `--subnet-id` (auto-creates ENI, assigns private IP), `--block-device-mappings` (DeviceName, VolumeSize, VolumeType, Iops, DeleteOnTermination) | `--security-group-ids`, `--placement`, `--tag-specifications`, `--dry-run`, `--client-token`, `--disable-api-termination`, `--ebs-optimized`, `--iam-instance-profile`, `--network-interfaces`, `--private-ip-address`, `--monitoring`, `--credit-specification`, `--cpu-options`, `--metadata-options`, `--launch-template`, `--hibernate-options` | `describe-images` (AMI must exist), `create-key-pair` (optional), VPC/SG (optional) | Gateway parses AWS query → NATS `ec2.runinstances` → daemon creates QEMU/KVM VM with viperblock-backed root volume via NBD → if SubnetId provided, auto-creates ENI with private IP → cloud-init injects user-data/keys → returns reservation with instance ID | 1. Launch with valid AMI and key pair<br>2. Launch with invalid AMI ID (error)<br>3. Launch with block device mappings (custom volume size)<br>4. Launch multiple instances (MinCount/MaxCount)<br>5. Launch with subnet-id (auto-creates ENI)<br>6. Invalid instance type returns error | **DONE** |
| `describe-instances` | `--instance-ids` | `--filters`, `--max-results`, `--next-token`, `--dry-run` | None | Gateway fans out NATS `ec2.DescribeInstances` to all nodes (no queue group) → each daemon returns local instances → gateway aggregates and returns combined list | 1. Describe all instances (no filter)<br>2. Describe by instance ID<br>3. Describe with filters (e.g. instance-state-name)<br>4. Instance not found returns empty set<br>5. Multi-node aggregation returns instances from all nodes | **DONE** |
| `start-instances` | `--instance-ids` | `--dry-run`, `--force` | `run-instances` (instance must exist in stopped state) | Gateway sends NATS `ec2.cmd.{instance-id}` → daemon restarts stopped QEMU process with same config → state transitions stopped→pending→running | 1. Start a stopped instance<br>2. Start already-running instance (error: IncorrectInstanceState)<br>3. Start with invalid instance ID<br>4. Verify volumes re-mount on start | **DONE** |
| `stop-instances` | `--instance-ids` | `--force`, `--hibernate`, `--dry-run` | `run-instances` (instance must be running) | Gateway sends NATS to target node → daemon issues QMP `system_powerdown` for graceful shutdown → monitors heartbeat until QEMU exits → state transitions running→stopping→stopped | 1. Graceful stop of running instance<br>2. Force stop (kills QEMU process)<br>3. Stop already-stopped instance (error)<br>4. Verify ~30s heartbeat detection | **DONE** |
| `terminate-instances` | `--instance-ids`, `DeleteOnTermination` (per-volume flag, default true) | `--dry-run` | `run-instances` (instance must exist) | Gateway sends NATS to target node → daemon kills QEMU process → cleans up NBD mounts → deletes volumes with `DeleteOnTermination=true` via `volumeService.DeleteVolume()` (S3 cleanup of vol/, vol-efi/, vol-cloudinit/) → internal volumes (EFI, cloud-init) always cleaned up via `ebs.delete` NATS → volumes with `DeleteOnTermination=false` left in available state → state→terminated | 1. Terminate running instance<br>2. Terminate stopped instance<br>3. Terminate with DeleteOnTermination=true deletes volumes<br>4. Terminate with DeleteOnTermination=false preserves volumes<br>5. Terminate already-terminated (idempotent)<br>6. Internal volumes (EFI, cloud-init) always cleaned up<br>7. Invalid instance ID | **DONE** |
| `reboot-instances` | — | `--instance-ids`, `--dry-run` | `run-instances` (instance must be running) | Gateway sends NATS to target node → daemon issues QMP `system_reset` → instance reboots without stopping | 1. Reboot running instance<br>2. Reboot stopped instance (error)<br>3. Verify instance stays in running state after reboot | **NOT STARTED** (parser test exists) |
| `describe-instance-types` | `--filters` (capacity filter only) | `--instance-types`, `--max-results`, `--next-token`, `--dry-run`, all other filters | None | Gateway fans out NATS `ec2.DescribeInstanceTypes` to all nodes → each daemon reports supported types (t3.micro/small/medium/large) with vCPU/memory specs → gateway deduplicates and returns | 1. List all instance types<br>2. Filter by specific type<br>3. Filter with `capacity=true` shows available slots<br>4. Verify vCPU/memory specs match hardware | **DONE** |
| `modify-instance-attribute` | `--instance-id`, `--instance-type`, `--user-data` | `--disable-api-termination`, `--ebs-optimized`, `--source-dest-check`, `--instance-initiated-shutdown-behavior`, `--block-device-mappings`, `--groups`, `--ena-support`, `--sriov-net-support` | Instance must be stopped (in NATS KV) | Gateway validates input (exactly one attribute per call, instance ID format) → NATS `ec2.ModifyInstanceAttribute` with `hive-workers` queue group → daemon loads stopped instance from JetStream KV → applies attribute change → writes back to KV → returns `{}` on success. **InstanceType**: updates vm.InstanceType, Config, and Instance fields; clears StateReason (enables recovery from instance-type-missing bug). **UserData**: stores decoded content in vm.UserData and re-encodes to base64 for RunInstancesInput (cloud-init on next start). No instance type pre-validation (matches AWS — invalid types accepted, fail at StartInstances time). | 1. Change instance type while stopped<br>2. Change user data while stopped<br>3. Modify running instance (error: NotFound — running instances not in KV)<br>4. Instance not found (error: InvalidInstanceID.NotFound)<br>5. Instance not stopped (error: IncorrectInstanceState)<br>6. Invalid instance type accepted (fails on start with InsufficientInstanceCapacity)<br>7. StateReason cleared on type change (recovery from capacity-unavailable)<br>8. Missing/malformed instance ID (error: InvalidInstanceID.Malformed)<br>9. No attribute set (error: InvalidParameterValue)<br>10. Multiple attributes in one call (error: InvalidParameterValue) | **DONE** |
| `get-console-output` | `--instance-id` | `--latest` (always returns latest), `--dry-run` | Instance must be running on a node | Gateway sends NATS `ec2.{instanceId}.GetConsoleOutput` (per-instance topic, routed to owning node) → daemon reads console log file from disk → returns last 64KB base64-encoded with timestamp. Always available regardless of serial console access setting (matches AWS behavior). | 1. Get output from running instance<br>2. Empty log file returns empty output<br>3. Instance not found (error: InvalidInstanceID.NotFound) | **DONE** |
| `monitor-instances` | — | `--instance-ids` | Instance must exist | Enable basic monitoring (CPU, disk, network) → store metrics in NATS KV → return monitoring state | 1. Enable monitoring<br>2. Verify monitoring state in describe-instances | **NOT STARTED** |
| `unmonitor-instances` | — | `--instance-ids` | Instance must exist | Disable detailed monitoring → revert to basic monitoring → return monitoring state | 1. Disable monitoring<br>2. Verify monitoring state reverts in describe-instances | **NOT STARTED** |
### EC2 - Key Pair Management
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-key-pair` | `--key-name`, `--key-type` (rsa/ed25519) | `--key-format` (pem/ppk), `--tag-specifications`, `--dry-run` | None | NATS `ec2.CreateKeyPair` → daemon generates SSH keypair → stores public key in Predastore S3 (`/bucket/ec2/{name}.pub`) → returns private key material (one-time) | 1. Create RSA key pair<br>2. Create ED25519 key pair<br>3. Duplicate key name (error: InvalidKeyPair.Duplicate)<br>4. Verify key material returned only on creation | **DONE** |
| `describe-key-pairs` | `--key-names`, `--key-pair-ids` | `--filters`, `--max-results`, `--dry-run` | None | NATS `ec2.DescribeKeyPairs` → daemon lists public keys from Predastore S3 → returns key names and fingerprints | 1. List all key pairs<br>2. Filter by key name<br>3. Filter by key pair ID<br>4. Non-existent key returns empty | **DONE** |
| `delete-key-pair` | `--key-name`, `--key-pair-id` | `--dry-run` | Key must exist | NATS `ec2.DeleteKeyPair` → daemon deletes public key from Predastore S3 → returns success | 1. Delete existing key pair<br>2. Delete non-existent key (idempotent, no error)<br>3. Verify key no longer in describe-key-pairs | **DONE** |
| `import-key-pair` | `--key-name`, `--public-key-material` | `--tag-specifications`, `--dry-run` | None | NATS `ec2.ImportKeyPair` → daemon stores provided public key in Predastore S3 → returns key name and fingerprint | 1. Import valid SSH public key<br>2. Import invalid key material (error)<br>3. Import duplicate name (error)<br>4. Verify imported key usable with run-instances | **DONE** |
### EC2 - AMI Image Management
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `describe-images` | `--image-ids` (format validation only), `--owners` (self, account ID, alias) | `--executable-users`, `--filters`, `--include-deprecated`, `--include-disabled`, `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeImages` → daemon reads AMI metadata from Predastore S3 buckets (ami-*) → filters by ImageIds and Owners → returns image list with state, architecture, block device mappings | 1. List all images<br>2. Filter by image ID<br>3. Filter by owner (self/amazon)<br>4. Non-existent AMI returns empty<br>5. Verify metadata fields (architecture, state, rootDeviceName) | **DONE** |
| `create-image` | `--instance-id`, `--name`, `--description`, `--tag-specifications` | `--no-reboot`, `--block-device-mappings`, `--dry-run` | Instance must exist (running or stopped) | Gateway validates input → NATS `ec2.{instanceId}.CreateImage` (per-instance topic) → daemon extracts root volume → snapshots via `ebs.snapshot` NATS (running) or offline S3 copy (stopped) → creates AMI metadata in S3 (`{amiId}/config.json`) → stores tags → returns ami-ID. Duplicate AMI name validation enforced. | 1. Create image from running instance<br>2. Create image from stopped instance<br>3. Invalid instance ID (error)<br>4. Duplicate AMI name (error)<br>5. Verify new AMI appears in describe-images<br>6. Launch new instance from created AMI | **DONE** |
| `register-image` | — | `--name`, `--description`, `--architecture`, `--root-device-name`, `--virtualization-type`, `--block-device-mappings`, `--boot-mode`, `--ena-support`, `--sriov-net-support`, `--tpm-support`, `--imds-support` | Image data must exist in S3 | Create AMI metadata entry in Predastore → assign ami-ID → set state to available → return ami-ID | 1. Register with valid metadata<br>2. Missing required name (error)<br>3. Verify registered image in describe-images | **NOT STARTED** |
| `deregister-image` | — | `--image-id`, `--dry-run` | AMI must exist, no instances using it | Mark AMI metadata as deregistered in S3 → AMI no longer available for new launches → existing instances unaffected | 1. Deregister existing AMI<br>2. Deregister non-existent AMI (error)<br>3. Verify deregistered AMI not in describe-images<br>4. Existing instances from AMI still run | **NOT STARTED** |
| `copy-image` | — | `--source-image-id`, `--source-region`, `--name`, `--description`, `--encrypted`, `--kms-key-id`, `--client-token`, `--copy-image-tags`, `--tag-specifications` | Source AMI must exist | Copy AMI data between regions/nodes in Predastore → create new AMI metadata with new ami-ID → return new ami-ID | 1. Copy image within same region<br>2. Copy non-existent image (error)<br>3. Verify copied image independent of source | **NOT STARTED** |
| `import-image` | — | `--disk-containers` (Format, Url/S3Bucket+S3Key), `--description`, `--architecture`, `--platform` | S3 bucket with disk image | Download disk image from S3 → convert format (VMDK/VHD/RAW→QCOW2) → create viperblock volume → register as AMI | 1. Import QCOW2 image<br>2. Import RAW image<br>3. Invalid format (error)<br>4. Verify imported image launchable | **NOT STARTED** |
| `describe-image-attribute` | — | `--image-id`, `--attribute`, `--dry-run` | AMI must exist | Read specific attribute from AMI metadata in S3 → return attribute value | 1. Get description attribute<br>2. Get launch permission<br>3. Invalid attribute name (error) | **NOT STARTED** |
| `modify-image-attribute` | — | `--image-id`, `--launch-permission`, `--description`, `--operation-type`, `--user-ids`, `--user-groups`, `--organization-arns`, `--dry-run` | AMI must exist | Update specific attribute in AMI metadata → persist to S3 | 1. Add launch permission<br>2. Modify description<br>3. Invalid AMI (error) | **NOT STARTED** |
| `reset-image-attribute` | — | `--image-id`, `--attribute`, `--dry-run` | AMI must exist | Reset attribute to default value in AMI metadata | 1. Reset launch permission to owner-only<br>2. Invalid AMI (error) | **NOT STARTED** |
### EC2 - Volume (EBS) Management
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `describe-volumes` | `--volume-ids` (fast-path lookup), `DeleteOnTermination` (from persisted VolumeMetadata) | `--filters`, `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeVolumes` → daemon queries viperblock for volume metadata → returns volume list with state, size, attachments, type, DeleteOnTermination flag | 1. List all volumes<br>2. Filter by volume ID<br>3. Filter by attachment state<br>4. Non-existent volume returns empty<br>5. DeleteOnTermination reflects persisted value | **DONE** |
| `modify-volume` | `--volume-id`, `--size`, `--volume-type`, `--iops` | `--throughput`, `--dry-run`, `--multi-attach-enabled` | Volume must exist | NATS `ec2.ModifyVolume` → daemon sends resize request to viperblock → NBD does not support live resize, instance must be stopped → returns modification state | 1. Increase volume size<br>2. Modify volume type<br>3. Decrease size (error - not supported)<br>4. Modify attached volume (requires stop/start) | **DONE** |
| `create-volume` | `--size`, `--availability-zone`, `--volume-type` (gp3 only), `--snapshot-id` (creates volume from snapshot) | `--iops` (hardcoded 3000), `--encrypted` (hardcoded false), `--throughput`, `--tag-specifications` | Valid AZ configured via `hive init` | Gateway validates input → NATS `ec2.CreateVolume` → daemon generates vol-ID via viperblock → creates volume (empty or from snapshot) of specified size → persists config.json to Predastore S3 → returns vol-ID with state=available | 1. Create 80GB gp3 volume<br>2. Boundary sizes (1 GiB min, 16384 GiB max)<br>3. Invalid AZ (error)<br>4. Verify volume in describe-volumes<br>5. Unsupported volume type (error - only gp3)<br>6. Size out of range (error)<br>7. Create from snapshot | **DONE** |
| `delete-volume` | `--volume-id` | `--dry-run` | Volume must exist and be detached (state=available) | Gateway validates vol- prefix → NATS `ec2.DeleteVolume` → daemon confirms state=available and no AttachedInstance → NATS `ebs.delete` to viperblockd (stops nbdkit/WAL) → deletes S3 objects under vol-id/, vol-id-efi/, vol-id-cloudinit/ → returns success | 1. Delete detached volume<br>2. Delete attached volume (error: VolumeInUse)<br>3. Delete non-existent volume (error: InvalidVolume.NotFound)<br>4. Verify volume gone from describe-volumes<br>5. Malformed volume ID (error: InvalidVolumeID.Malformed)<br>6. Double delete (idempotent NotFound) | **DONE** |
| `attach-volume` | `--volume-id`, `--instance-id`, `--device` (optional, auto-assigns `/dev/sd[f-p]`) | `--dry-run` | Volume must exist (available), instance must exist (running) | Gateway sends to `ec2.cmd.{instanceId}` → daemon validates volume (Predastore) → `ebs.mount` via NATS (viperblock starts NBD server) → QMP `blockdev-add` (nbd-{volId}) → QMP `device_add` (virtio-blk-pci, vdisk-{volId}) → three-phase rollback on failure → update EBSRequests + BlockDeviceMappings → persist to JetStream + Predastore → respond with VolumeAttachment | 1. Attach volume to running instance<br>2. Auto-assign device name<br>3. Attach already-attached volume (VolumeInUse)<br>4. Attach to non-existent instance (InvalidInstanceID.NotFound)<br>5. Attach to stopped instance (IncorrectInstanceState)<br>6. Volume not found (InvalidVolume.NotFound)<br>7. All device slots full (AttachmentLimitExceeded)<br>8. Volume persists across stop/start | **DONE** |
| `detach-volume` | `--volume-id`, `--instance-id` (optional, resolved via DescribeVolumes), `--device` (optional cross-check), `--force` | `--dry-run` | Volume must be attached, instance must be running | Gateway resolves InstanceId if omitted (via DescribeVolumes) → sends to `ec2.cmd.{instanceId}` → daemon validates (running, attached, not boot/EFI/CloudInit, device match) → three-phase hot-unplug: QMP `device_del` (force continues on failure) → QMP `blockdev-del` (abort if fails, preserves state to prevent double-attach) → `ebs.unmount` via NATS (best-effort) → remove from EBSRequests + BlockDeviceMappings → update volume metadata to available → persist state → respond with VolumeAttachment (state=detaching) | 1. Detach with explicit InstanceId<br>2. Detach without InstanceId (gateway resolution)<br>3. Detach with correct --device cross-check<br>4. Missing VolumeId (InvalidParameterValue)<br>5. Volume not attached (IncorrectState)<br>6. Nonexistent volume (InvalidVolume.NotFound)<br>7. Nonexistent instance (InvalidInstanceID.NotFound)<br>8. Instance not running (IncorrectInstanceState)<br>9. Device mismatch (InvalidParameterValue)<br>10. Boot volume protection (OperationNotPermitted)<br>11. Force flag (continues past device_del failure)<br>12. Volume reusability (re-attach after detach) | **DONE** |
| `describe-volume-status` | `--volume-ids` | `--filters`, `--max-results`, `--next-token`, `--dry-run` | None | Gateway validates vol- prefix → NATS `ec2.DescribeVolumeStatus` → daemon fetches VolumeConfig from Predastore S3 (parallel for specific IDs, sequential list-all for no IDs) → builds VolumeStatusItem per volume (status=ok, io-enabled=passed, io-performance=not-applicable) → returns InvalidVolume.NotFound for missing explicit IDs → skips internal sub-volumes (-efi, -cloudinit) | 1. List all volume statuses<br>2. Filter by specific volume IDs (fast path)<br>3. Non-existent volume ID returns InvalidVolume.NotFound<br>4. Invalid volume ID format (InvalidVolume.Malformed)<br>5. Internal sub-volumes excluded from listing<br>6. Nil/empty input defaults to all volumes | **DONE** |
| `describe-volumes-modifications` | — | `--volume-ids`, `--filters`, `--max-results` | None | Query pending/completed volume modifications → return modification state, progress, original/target size | 1. Check in-progress modification<br>2. Check completed modification<br>3. No modifications returns empty | **NOT STARTED** |
### EC2 - Snapshot Management
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-snapshot` | `--volume-id`, `--description`, `--tag-specifications` | `--dry-run` | Volume must exist in Predastore | Gateway validates vol- prefix → NATS `ec2.CreateSnapshot` → daemon reads VolumeConfig from Predastore → generates snap-ID via viperblock → stores SnapshotConfig (metadata-only, points to source volume) as completed → returns Snapshot with state=completed, progress=100% | 1. Snapshot from valid volume<br>2. Missing volume ID (InvalidParameterValue)<br>3. Volume not found (InvalidVolume.NotFound)<br>4. Invalid volume ID format (InvalidVolume.Malformed)<br>5. Snapshot with description<br>6. Snapshot with tag specifications | **DONE** |
| `create-snapshots` | — | `--instance-specification`, `--description`, `--tag-specifications` | Instance must exist, instance-volume attachment tracking | Create snapshots of all volumes attached to instance → return list of snapshot IDs. Blocked: requires instance-volume attachment tracking to discover which volumes to snapshot. | 1. Snapshot all volumes on instance<br>2. Instance with no volumes | **NOT STARTED** |
| `delete-snapshot` | `--snapshot-id` | `--dry-run` | Snapshot must exist | Gateway validates snap- prefix → NATS `ec2.DeleteSnapshot` → daemon verifies snapshot exists in Predastore → lists and deletes all objects under snapshot prefix → returns success | 1. Delete existing snapshot<br>2. Delete non-existent snapshot (InvalidSnapshot.NotFound)<br>3. Missing snapshot ID (InvalidParameterValue)<br>4. Invalid snapshot ID format (InvalidSnapshot.Malformed) | **DONE** |
| `describe-snapshots` | `--snapshot-ids` | `--owner-ids`, `--filters`, `--max-results`, `--dry-run` | None | Gateway validates snap- prefix on IDs → NATS `ec2.DescribeSnapshots` → daemon lists snap- prefixed objects in Predastore → reads SnapshotConfig for each → filters by snapshot ID if specified → returns snapshot list | 1. List all snapshots<br>2. Filter by snapshot ID<br>3. Empty snapshot list<br>4. Invalid snapshot ID format (InvalidSnapshot.Malformed) | **DONE** |
| `copy-snapshot` | `--source-snapshot-id`, `--source-region`, `--description` | `--encrypted`, `--dry-run` | Source snapshot must exist | Gateway validates snap- prefix + source region → NATS `ec2.CopySnapshot` → daemon reads source SnapshotConfig → generates new snap-ID → copies metadata (preserves tags, description override) → stores as completed → returns new snapshot ID | 1. Copy within same region<br>2. Copy non-existent snapshot (InvalidSnapshot.NotFound)<br>3. Missing source ID (InvalidParameterValue)<br>4. Missing source region (MissingParameter)<br>5. Copy preserves tags<br>6. Copy with description override | **DONE** |
### EC2 - Tags
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-tags` | `--resources`, `--tags` (Key=,Value=) | `--dry-run` | None | Gateway validates input (resources non-empty, tags non-empty, keys non-empty) → NATS `ec2.CreateTags` → daemon merges tags into per-resource JSON in Predastore S3 (`tags/{resourceId}.json`) → return success | 1. Tag an instance<br>2. Tag multiple resources at once<br>3. Overwrite existing tag value<br>4. Empty resources list (error: MissingParameter)<br>5. Empty tags list (error: MissingParameter)<br>6. Empty tag key (error: InvalidParameterValue) | **DONE** |
| `delete-tags` | `--resources`, `--tags` | `--dry-run` | None | Gateway validates input (resources non-empty) → NATS `ec2.DeleteTags` → daemon removes specified tag keys from resource JSON, or all tags if no keys specified → return success | 1. Delete specific tag by key<br>2. Delete all tags (no tags specified)<br>3. Empty resources list (error: MissingParameter) | **DONE** |
| `describe-tags` | `--filters` (resource-id, resource-type, key, value) | `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeTags` → daemon lists all `tags/*.json` from Predastore S3 → applies filters (resource-id, resource-type, key, value) → returns tag list with resource type auto-detected from ID prefix | 1. List all tags<br>2. Filter by resource-id<br>3. Filter by resource-type (instance, volume, image, snapshot, vpc, subnet, security-group, route-table, internet-gateway, egress-only-internet-gateway)<br>4. Filter by key<br>5. Filter by value<br>6. Empty result when no tags exist | **DONE** |
### EC2 - Regions & Availability Zones
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `describe-regions` | *(returns configured region, endpoint, opt-in status from config — input params ignored)* | `--region-names`, `--filters`, `--all-regions`, `--dry-run` | None | Return configured region from hive init config with Endpoint and OptInStatus → local-only response, no NATS | 1. List all regions<br>2. Filter by region name<br>3. Verify endpoint URL returned<br>4. Verify OptInStatus returned | **DONE** |
| `describe-availability-zones` | *(returns configured AZ, region, zone ID, state, opt-in status from config — input params ignored)* | `--zone-names`, `--filters`, `--all-availability-zones` | None | Return configured AZ from hive init config with zone ID, state, group name, network border group → local-only response, no NATS | 1. List all AZs<br>2. Filter by zone name<br>3. Verify zone state is available<br>4. Verify region name matches config | **DONE** |
### EC2 - Account Attributes
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `describe-account-attributes` | `--attribute-names` | `--dry-run` | None | Gateway parses input → returns static account attributes: supported-platforms=VPC, default-vpc=none, max-instances=100, vpc-max-security-groups-per-interface=5, max-elastic-ips=5, vpc-max-elastic-ips=20 → local-only response, no NATS | 1. List all account attributes<br>2. Filter by attribute name<br>3. Verify all 6 attributes returned with correct values | **DONE** |
### EC2 - Account Settings (Deferred)
Persistence layer works (NATS JetStream KV), but stored values are not yet enforced by downstream services. Will be completed post-merge.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `enable-ebs-encryption-by-default` | *(no flags needed)* | `--dry-run` | None | Gateway validates input → NATS `ec2.EnableEbsEncryptionByDefault` with `hive-workers` queue group → daemon stores `EbsEncryptionByDefault=true` in JetStream KV (`hive-ec2-account-settings` bucket) → return `EbsEncryptionByDefault=true`. Enforcement deferred: stored value not yet checked by CreateVolume/CreateSnapshot. | 1. Enable encryption<br>2. Verify via get | **STARTED** (enforcement pending) |
| `disable-ebs-encryption-by-default` | *(no flags needed)* | `--dry-run` | None | Gateway validates input → NATS `ec2.DisableEbsEncryptionByDefault` with `hive-workers` queue group → daemon stores `EbsEncryptionByDefault=false` in JetStream KV → return `EbsEncryptionByDefault=false`. | 1. Disable encryption<br>2. Verify via get | **STARTED** (enforcement pending) |
| `get-ebs-encryption-by-default` | *(no flags needed)* | `--dry-run` | None | Gateway validates input → NATS `ec2.GetEbsEncryptionByDefault` with `hive-workers` queue group → daemon reads `EbsEncryptionByDefault` from JetStream KV → return current state. | 1. Get default state<br>2. Verify matches last enable/disable | **STARTED** (enforcement pending) |
| `enable-serial-console-access` | *(no flags needed)* | `--dry-run` | None | Gateway validates input → NATS `ec2.EnableSerialConsoleAccess` with `hive-workers` queue group → daemon stores `SerialConsoleAccessEnabled=true` in JetStream KV (`hive-ec2-account-settings` bucket) → return enabled=true. Stored for future interactive serial console support (not currently enforced by any operation). | 1. Enable access<br>2. Verify via get | **STARTED** (enforcement pending) |
| `disable-serial-console-access` | *(no flags needed)* | `--dry-run` | None | Gateway validates input → NATS `ec2.DisableSerialConsoleAccess` with `hive-workers` queue group → daemon stores `SerialConsoleAccessEnabled=false` in JetStream KV → return enabled=false. Stored for future interactive serial console support (not currently enforced by any operation). | 1. Disable access<br>2. Verify via get | **STARTED** (enforcement pending) |
| `get-serial-console-access-status` | *(no flags needed)* | `--dry-run` | None | Gateway validates input → NATS `ec2.GetSerialConsoleAccessStatus` with `hive-workers` queue group → daemon reads `SerialConsoleAccessEnabled` from JetStream KV → return current state. | 1. Get current status<br>2. Verify matches last enable/disable | **DONE** |
| `enable-snapshot-block-public-access` | — | `--state` | None | Store snapshot block public access state in JetStream KV | 1. Enable with block-all-sharing<br>2. Enable with block-new-sharing | **NOT STARTED** (enforcement pending) |
| `disable-snapshot-block-public-access` | — | `--dry-run` | None | Clear snapshot block public access state in JetStream KV | 1. Disable access | **NOT STARTED** (enforcement pending) |
| `get-snapshot-block-public-access-state` | — | `--dry-run` | None | Read snapshot block public access state from JetStream KV | 1. Get current state | **NOT STARTED** (enforcement pending) |
| `enable-image-block-public-access` | — | `--image-block-public-access-state` | None | Store image block public access state in JetStream KV | 1. Enable with block-new-sharing | **NOT STARTED** (enforcement pending) |
| `disable-image-block-public-access` | — | `--dry-run` | None | Clear image block public access state in JetStream KV | 1. Disable access | **NOT STARTED** (enforcement pending) |
| `get-image-block-public-access-state` | — | `--dry-run` | None | Read image block public access state from JetStream KV | 1. Get current state | **NOT STARTED** (enforcement pending) |
| `modify-instance-metadata-defaults` | — | `--http-tokens`, `--http-put-response-hop-limit`, `--http-endpoint`, `--instance-metadata-tags` | None | Store instance metadata defaults in JetStream KV | 1. Set httpTokens=required<br>2. Set hop limit | **NOT STARTED** (enforcement pending) |
| `get-instance-metadata-defaults` | — | `--dry-run` | None | Read instance metadata defaults from JetStream KV | 1. Get current defaults | **NOT STARTED** (enforcement pending) |
### EC2 - Egress-Only Internet Gateway
Gateway routing and KV CRUD are wired up but no OVN/OVS integration exists. EIGWs are stored in `hive-eigw` KV bucket but have no effect on network topology (no vpcd events published, unlike IGW which publishes `vpc.igw-attach`/`vpc.igw-detach`).
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-egress-only-internet-gateway` | `--vpc-id`, `--tag-specifications` | `--client-token`, `--dry-run` | VPC must exist | Gateway validates VpcId → NATS `ec2.CreateEgressOnlyInternetGateway` → daemon generates eigw-ID → creates EgressOnlyIGWRecord in `hive-eigw` KV bucket with state=attached (always attached at creation, unlike IGW) → returns EgressOnlyInternetGateway with attachment | 1. Create with valid VpcId<br>2. Missing VpcId (error: MissingParameter)<br>3. Create with inline tags<br>4. Verify ID format eigw-{16hex} | **STARTED** (KV only, no OVN) |
| `delete-egress-only-internet-gateway` | `--egress-only-internet-gateway-id` | `--dry-run` | EIGW must exist | NATS `ec2.DeleteEgressOnlyInternetGateway` → daemon verifies EIGW exists in KV → deletes from `hive-eigw` KV → returns ReturnCode=true. No detach required (unlike IGW). | 1. Delete existing EIGW<br>2. Delete non-existent EIGW (error)<br>3. Missing ID (error: MissingParameter) | **STARTED** (KV only, no OVN) |
| `describe-egress-only-internet-gateways` | `--egress-only-internet-gateway-ids` | `--filters`, `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeEgressOnlyInternetGateways` → daemon lists all keys from `hive-eigw` KV bucket → filters by ID if specified → always includes Attachments array → returns list | 1. Describe all EIGWs<br>2. Filter by specific ID<br>3. Non-existent ID returns empty list | **STARTED** (KV only, no OVN) |
### EC2 - Placement Groups
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-placement-group` | — | `--group-name`, `--strategy` (cluster/spread/partition), `--partition-count`, `--spread-level`, `--tag-specifications` | None | NATS `ec2.CreatePlacementGroup` → daemon stores placement group in JetStream KV with strategy, state=available → return group metadata | 1. Create with cluster strategy<br>2. Create with spread strategy<br>3. Create with partition strategy + partition count<br>4. Duplicate name (error)<br>5. Invalid strategy (error) | **NOT STARTED** |
| `delete-placement-group` | — | `--group-name`, `--dry-run` | Group must exist, no instances in group | NATS `ec2.DeletePlacementGroup` → daemon verifies group exists and is empty → delete from KV → return success | 1. Delete empty group<br>2. Delete group with instances (error)<br>3. Delete non-existent group (error) | **NOT STARTED** |
| `describe-placement-groups` | — | `--group-names`, `--group-ids`, `--filters` | None | NATS `ec2.DescribePlacementGroups` → daemon lists groups from KV → return list with strategy and state | 1. List all groups<br>2. Filter by name<br>3. Filter by strategy | **NOT STARTED** |
### EC2 - Dedicated Hosts
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `allocate-hosts` | — | `--availability-zone`, `--instance-type`, `--quantity`, `--auto-placement`, `--tag-specifications` | None | NATS `ec2.AllocateHosts` → daemon allocates host resources → stores in JetStream KV with h-ID → return host ID list | 1. Allocate single host<br>2. Allocate multiple hosts<br>3. Missing AZ or instance type (error) | **NOT STARTED** |
| `describe-hosts` | — | `--host-ids`, `--filters`, `--max-results` | None | NATS `ec2.DescribeHosts` → daemon lists hosts from KV → return host list with capacity and instance info | 1. List all hosts<br>2. Filter by host ID<br>3. Filter by instance type | **NOT STARTED** |
| `release-hosts` | — | `--host-ids` | Host must exist, no running instances | NATS `ec2.ReleaseHosts` → daemon verifies no instances on host → delete from KV → return success/failure per host | 1. Release empty host<br>2. Release host with instances (error)<br>3. Release non-existent host (error) | **NOT STARTED** |
### EC2 - IPv4 Pools
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-public-ipv4-pool` | — | `--tag-specifications`, `--dry-run` | None | NATS `ec2.CreatePublicIpv4Pool` → daemon creates pool in JetStream KV with ipv4pool-ID → return pool ID | 1. Create pool<br>2. Verify in describe | **NOT STARTED** |
| `delete-public-ipv4-pool` | — | `--pool-id`, `--dry-run` | Pool must exist and be empty | NATS `ec2.DeletePublicIpv4Pool` → daemon verifies pool empty → delete from KV → return success | 1. Delete empty pool<br>2. Delete pool with addresses (error) | **NOT STARTED** |
| `describe-public-ipv4-pools` | — | `--pool-ids`, `--filters`, `--max-results` | None | NATS `ec2.DescribePublicIpv4Pools` → daemon lists pools from KV → return pool list with address counts | 1. List all pools<br>2. Filter by pool ID | **NOT STARTED** |
### EC2 - DHCP Options
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-dhcp-options` | — | `--dhcp-configurations` (domain-name, domain-name-servers, ntp-servers, netbios-name-servers, netbios-node-type), `--tag-specifications` | None | NATS `ec2.CreateDhcpOptions` → daemon stores DHCP option set in JetStream KV with dopt-ID → return DHCP options with configurations | 1. Create with domain-name + DNS servers<br>2. Create with NTP servers<br>3. Verify in describe | **NOT STARTED** |
| `delete-dhcp-options` | — | `--dhcp-options-id`, `--dry-run` | DHCP options must exist and not be associated with a VPC | NATS `ec2.DeleteDhcpOptions` → daemon verifies not associated → delete from KV → return success | 1. Delete unassociated options<br>2. Delete associated options (error)<br>3. Delete non-existent (error) | **NOT STARTED** |
| `describe-dhcp-options` | — | `--dhcp-options-ids`, `--filters`, `--max-results` | None | NATS `ec2.DescribeDhcpOptions` → daemon lists DHCP options from KV → return list with configurations | 1. List all DHCP options<br>2. Filter by ID | **NOT STARTED** |
| `associate-dhcp-options` | — | `--dhcp-options-id`, `--vpc-id`, `--dry-run` | DHCP options and VPC must exist | NATS `ec2.AssociateDhcpOptions` → daemon links DHCP options to VPC → update VPC metadata → return success | 1. Associate with VPC<br>2. Missing DHCP options ID (error)<br>3. Missing VPC ID (error) | **NOT STARTED** |
### EC2 - Capacity Reservations
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-capacity-reservation` | — | `--instance-type`, `--instance-platform`, `--availability-zone`, `--instance-count`, `--end-date`, `--end-date-type`, `--instance-match-criteria`, `--tag-specifications` | None | NATS `ec2.CreateCapacityReservation` → daemon stores reservation in JetStream KV with cr-ID → return reservation with state=active | 1. Create with required fields<br>2. Missing instance type (error)<br>3. Verify in describe | **NOT STARTED** |
| `cancel-capacity-reservation` | — | `--capacity-reservation-id`, `--dry-run` | Reservation must exist | NATS `ec2.CancelCapacityReservation` → daemon marks reservation as cancelled in KV → return success | 1. Cancel active reservation<br>2. Cancel non-existent (error) | **NOT STARTED** |
| `describe-capacity-reservations` | — | `--capacity-reservation-ids`, `--filters`, `--max-results` | None | NATS `ec2.DescribeCapacityReservations` → daemon lists reservations from KV → return list with state and counts | 1. List all reservations<br>2. Filter by ID<br>3. Filter by instance type | **NOT STARTED** |
| `modify-capacity-reservation` | — | `--capacity-reservation-id`, `--instance-count`, `--end-date`, `--end-date-type` | Reservation must exist | NATS `ec2.ModifyCapacityReservation` → daemon updates reservation in KV → return success | 1. Modify instance count<br>2. Modify end date<br>3. Missing reservation ID (error) | **NOT STARTED** |
### EC2 - Elastic IP
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `allocate-address` | — | `--domain` (vpc), `--tag-specifications`, `--dry-run` | None | NATS `ec2.AllocateAddress` → daemon allocates IP from pool → stores in JetStream KV with eipalloc-ID → return AllocationId, PublicIp, Domain | 1. Allocate EIP<br>2. Verify in describe-addresses<br>3. Allocate without --domain uses default | **NOT STARTED** |
| `release-address` | — | `--allocation-id`, `--dry-run` | EIP must exist and be disassociated | NATS `ec2.ReleaseAddress` → daemon verifies not associated → release IP back to pool → delete from KV → return success | 1. Release disassociated EIP<br>2. Release associated EIP (error)<br>3. Release non-existent (error) | **NOT STARTED** |
| `associate-address` | — | `--allocation-id`, `--instance-id` or `--network-interface-id`, `--dry-run` | EIP and target must exist | NATS `ec2.AssociateAddress` → daemon links EIP to instance/ENI → configure OVS DNAT/SNAT rules → return eipassoc-ID | 1. Associate with instance<br>2. Re-associate (moves EIP)<br>3. Associate non-existent EIP (error) | **NOT STARTED** |
| `disassociate-address` | — | `--association-id`, `--dry-run` | Association must exist | NATS `ec2.DisassociateAddress` → daemon unlinks EIP from instance/ENI → remove OVS DNAT/SNAT rules → return success | 1. Disassociate existing<br>2. Disassociate non-existent (error) | **NOT STARTED** |
| `describe-addresses` | — | `--allocation-ids`, `--filters`, `--public-ips` | None | NATS `ec2.DescribeAddresses` → daemon lists EIPs from KV → return list with association info | 1. List all EIPs<br>2. Filter by allocation ID<br>3. Filter by public IP | **NOT STARTED** |
### EC2 - NAT Gateway
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-nat-gateway` | — | `--subnet-id`, `--allocation-id`, `--connectivity-type`, `--tag-specifications` | Subnet must exist, EIP must be allocated (for public type) | NATS `ec2.CreateNatGateway` → daemon creates NAT gateway in JetStream KV with nat-ID → configure OVS SNAT rules → state=pending→available → return NatGateway | 1. Create public NAT gateway<br>2. Create private NAT gateway (no allocation-id)<br>3. Invalid subnet (error) | **NOT STARTED** |
| `delete-nat-gateway` | — | `--nat-gateway-id`, `--dry-run` | NAT gateway must exist | NATS `ec2.DeleteNatGateway` → daemon marks as deleting → remove OVS SNAT rules → delete from KV → return NatGatewayId | 1. Delete existing NAT gateway<br>2. Delete non-existent (error) | **NOT STARTED** |
| `describe-nat-gateways` | — | `--nat-gateway-ids`, `--filters`, `--max-results` | None | NATS `ec2.DescribeNatGateways` → daemon lists NAT gateways from KV → return list with state, subnet, addresses | 1. List all NAT gateways<br>2. Filter by ID<br>3. Filter by VPC ID | **NOT STARTED** |
| `assign-private-nat-gateway-address` | — | `--nat-gateway-id`, `--private-ip-addresses` | NAT gateway must exist (private type) | NATS `ec2.AssignPrivateNatGatewayAddress` → daemon assigns private IPs to NAT gateway → return assigned addresses | 1. Assign address<br>2. Missing NAT gateway ID (error) | **NOT STARTED** |
| `associate-nat-gateway-address` | — | `--nat-gateway-id`, `--allocation-ids` | NAT gateway and EIPs must exist | NATS `ec2.AssociateNatGatewayAddress` → daemon associates EIPs with NAT gateway → return association info | 1. Associate EIP<br>2. Missing NAT gateway ID (error) | **NOT STARTED** |
### EC2 - VPC Networking (Core)
Requires OVN/OVS (`apt install openvswitch-switch ovn-host`). VPC/Subnet/ENI CRUD stores metadata in NATS KV and publishes events to vpcd (the network topology daemon) which translates to OVN logical switches, ports, and routers. Single AZ for Hive v1. IPAM uses CAS-based allocation from NATS KV (`hive-vpc-ipam` bucket).
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-vpc` | `--cidr-block`, `--tag-specifications` | `--instance-tenancy`, `--dry-run` | None | Gateway validates CidrBlock (required, valid CIDR, /16–/28 prefix) → NATS `ec2.CreateVpc` → daemon allocates unique VNI from `hive-vpc-vni-counter` KV → generates vpc-ID → stores VPCRecord in `hive-vpc-vpcs` KV → publishes `vpc.create` event to vpcd (creates OVN logical switch) → returns VPC with state=available | 1. Create VPC with /16 CIDR<br>2. Invalid CIDR format (error: InvalidVpcRange)<br>3. CIDR prefix outside /16–/28 (error)<br>4. Verify in describe-vpcs<br>5. Tags applied at creation | **DONE** |
| `delete-vpc` | `--vpc-id` | `--dry-run` | VPC must be empty (no subnets) | Gateway validates VpcId → NATS `ec2.DeleteVpc` → daemon checks for dependent subnets in `hive-vpc-subnets` KV → returns DependencyViolation if subnets exist → deletes from `hive-vpc-vpcs` KV → publishes `vpc.delete` event to vpcd (removes OVN resources) | 1. Delete empty VPC<br>2. Delete VPC with subnets (DependencyViolation)<br>3. Delete non-existent VPC (error) | **DONE** |
| `describe-vpcs` | `--vpc-ids` | `--filters`, `--max-results`, `--dry-run` | None | NATS `ec2.DescribeVpcs` → daemon lists all keys from `hive-vpc-vpcs` KV → filters by VPC IDs if specified → returns error for non-existent requested IDs → returns VPC list with CIDR, state, CIDR associations | 1. List all VPCs<br>2. Filter by VPC ID<br>3. Non-existent VPC ID returns error | **DONE** |
| `modify-vpc-attribute` | — | `--vpc-id`, `--enable-dns-support`, `--enable-dns-hostnames` | VPC must exist | NATS `ec2.ModifyVpcAttribute` → daemon updates VPC attribute in KV → return success | 1. Enable DNS support<br>2. Enable DNS hostnames<br>3. Missing VPC ID (error) | **NOT STARTED** |
| `associate-vpc-cidr-block` | — | `--vpc-id`, `--cidr-block` | VPC must exist | NATS `ec2.AssociateVpcCidrBlock` → daemon adds secondary CIDR to VPC metadata → return association | 1. Add secondary CIDR<br>2. Overlapping CIDR (error)<br>3. Max CIDR blocks exceeded (error) | **NOT STARTED** |
| `disassociate-vpc-cidr-block` | — | `--association-id` | Association must exist | NATS `ec2.DisassociateVpcCidrBlock` → daemon removes secondary CIDR from VPC → return success | 1. Remove secondary CIDR<br>2. Remove primary CIDR (error) | **NOT STARTED** |
| `create-subnet` | `--vpc-id`, `--cidr-block`, `--availability-zone`, `--tag-specifications` | `--dry-run` | VPC must exist | Gateway validates VpcId + CidrBlock (required, valid CIDR, /16–/28 prefix) → NATS `ec2.CreateSubnet` → daemon verifies parent VPC exists → validates subnet CIDR is within VPC CIDR range → checks for overlapping subnets in same VPC → calculates available IPs (total hosts - 5 AWS-reserved: .0/.1/.2/.3/.255) → generates subnet-ID → stores SubnetRecord in `hive-vpc-subnets` KV → publishes `vpc.create-subnet` event to vpcd → defaults AZ from config if omitted | 1. Create subnet within VPC CIDR<br>2. Subnet CIDR outside VPC range (error: InvalidSubnetRange)<br>3. Overlapping subnet CIDRs (error: InvalidSubnetConflict)<br>4. CIDR prefix outside /16–/28 (error)<br>5. Missing VPC or CIDR (error: MissingParameter) | **DONE** |
| `delete-subnet` | `--subnet-id` | `--dry-run` | Subnet must exist | NATS `ec2.DeleteSubnet` → daemon verifies subnet exists → deletes from `hive-vpc-subnets` KV → publishes `vpc.delete-subnet` event to vpcd | 1. Delete existing subnet<br>2. Delete non-existent subnet (error) | **DONE** |
| `describe-subnets` | `--subnet-ids`, `--filters` (vpc-id) | `--max-results`, `--dry-run` | None | NATS `ec2.DescribeSubnets` → daemon lists all keys from `hive-vpc-subnets` KV → applies vpc-id filter if present → filters by subnet IDs if specified → recalculates available IPs from CIDR → returns error for non-existent requested IDs | 1. List all subnets<br>2. Filter by VPC ID<br>3. Filter by subnet ID<br>4. Non-existent subnet returns error | **DONE** |
| `modify-subnet-attribute` | — | `--subnet-id`, `--map-public-ip-on-launch`, `--assign-ipv6-address-on-creation` | Subnet must exist | NATS `ec2.ModifySubnetAttribute` → daemon updates subnet attribute in KV → return success | 1. Enable auto-assign public IP<br>2. Missing subnet ID (error) | **NOT STARTED** |
| `associate-subnet-cidr-block` | — | `--subnet-id`, `--ipv6-cidr-block` | Subnet must exist | NATS `ec2.AssociateSubnetCidrBlock` → daemon adds IPv6 CIDR to subnet → return association | 1. Add IPv6 CIDR<br>2. Missing subnet ID (error) | **NOT STARTED** |
| `disassociate-subnet-cidr-block` | — | `--association-id` | Association must exist | NATS `ec2.DisassociateSubnetCidrBlock` → daemon removes IPv6 CIDR from subnet → return success | 1. Remove IPv6 CIDR<br>2. Invalid association (error) | **NOT STARTED** |
| `create-security-group` | — | `--group-name`, `--description`, `--vpc-id`, `--tag-specifications` | VPC must exist | Create security group metadata → create default OVS flow rules (deny all inbound, allow all outbound) → assign sg-ID | 1. Create SG in VPC<br>2. Duplicate name in same VPC (error)<br>3. Verify default rules | **NOT STARTED** |
| `delete-security-group` | — | `--group-id` | SG must not be in use by instances | Verify no instances reference SG → remove OVS flow rules → delete metadata | 1. Delete unused SG<br>2. Delete SG in use (error) | **NOT STARTED** |
| `describe-security-groups` | — | `--group-ids`, `--group-names`, `--filters`, `--max-results` | None | Query NATS KV for SG metadata → return SG list with rules | 1. List all SGs<br>2. Filter by VPC ID<br>3. Filter by group name | **NOT STARTED** |
| `authorize-security-group-ingress` | — | `--group-id`, `--protocol`, `--port`, `--cidr`, `--source-group` | SG must exist | Add inbound rule → create OVS OpenFlow rule → persist to metadata | 1. Allow SSH (port 22) from 0.0.0.0/0<br>2. Allow from specific CIDR<br>3. Duplicate rule (idempotent) | **NOT STARTED** |
| `authorize-security-group-egress` | — | `--group-id`, `--protocol`, `--port`, `--cidr` | SG must exist | Add outbound rule → create OVS OpenFlow rule → persist to metadata | 1. Allow HTTPS outbound<br>2. Restrict to specific CIDR | **NOT STARTED** |
| `revoke-security-group-ingress` | — | `--group-id`, `--protocol`, `--port`, `--cidr` | SG must exist, rule must exist | Remove inbound rule → delete OVS OpenFlow rule → update metadata | 1. Revoke existing rule<br>2. Revoke non-existent rule (error) | **NOT STARTED** |
| `revoke-security-group-egress` | — | `--group-id`, `--protocol`, `--port`, `--cidr` | SG must exist, rule must exist | Remove outbound rule → delete OVS OpenFlow rule → update metadata | 1. Revoke existing rule<br>2. Revoke non-existent rule (error) | **NOT STARTED** |
| `create-internet-gateway` | `--tag-specifications` | `--dry-run` | None | NATS `ec2.CreateInternetGateway` → daemon generates igw-ID → creates IGWRecord in `hive-igw` KV bucket with state=available (not attached) → returns InternetGateway with empty Attachments | 1. Create IGW<br>2. Verify in describe-internet-gateways<br>3. Tags applied at creation | **DONE** |
| `attach-internet-gateway` | `--internet-gateway-id`, `--vpc-id` | `--dry-run` | IGW and VPC must exist | NATS `ec2.AttachInternetGateway` → daemon validates IGW exists and is not already attached (ResourceAlreadyAssociated) → sets VpcId on record → state=attached → publishes `vpc.igw-attach` event to vpcd (creates OVN external switch, gateway port, SNAT rules) | 1. Attach IGW to VPC<br>2. Attach already-attached IGW (ResourceAlreadyAssociated)<br>3. Non-existent IGW (error)<br>4. Missing params (MissingParameter) | **DONE** |
| `detach-internet-gateway` | `--internet-gateway-id`, `--vpc-id` | `--dry-run` | IGW must be attached to specified VPC | NATS `ec2.DetachInternetGateway` → daemon validates IGW exists and is attached to specified VPC (GatewayNotAttached if wrong VPC) → clears VpcId → state=available → publishes `vpc.igw-detach` event to vpcd (removes OVN resources) | 1. Detach attached IGW<br>2. Detach from wrong VPC (GatewayNotAttached)<br>3. Detach unattached IGW (error) | **DONE** |
| `delete-internet-gateway` | `--internet-gateway-id` | `--dry-run` | IGW must be detached | NATS `ec2.DeleteInternetGateway` → daemon validates IGW exists and VpcId is empty (DependencyViolation if attached) → deletes from `hive-igw` KV | 1. Delete detached IGW<br>2. Delete attached IGW (DependencyViolation)<br>3. Delete non-existent (error) | **DONE** |
| `describe-internet-gateways` | `--internet-gateway-ids` | `--filters`, `--max-results`, `--dry-run` | None | NATS `ec2.DescribeInternetGateways` → daemon lists all keys from `hive-igw` KV → filters by IGW IDs if specified → returns list with Attachments array (if attached) | 1. List all IGWs<br>2. Filter by IGW ID<br>3. Verify attachment info | **DONE** |
### EC2 - Route Tables
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-route-table` | — | `--vpc-id`, `--tag-specifications` | VPC must exist | NATS `ec2.CreateRouteTable` → daemon creates route table in KV with rtb-ID → add default local route for VPC CIDR → return route table | 1. Create route table<br>2. Verify default local route<br>3. Missing VPC ID (error) | **NOT STARTED** |
| `delete-route-table` | — | `--route-table-id`, `--dry-run` | Route table must exist and not be main route table | NATS `ec2.DeleteRouteTable` → daemon verifies not main table → delete from KV → return success | 1. Delete non-main route table<br>2. Delete main route table (error)<br>3. Delete non-existent (error) | **NOT STARTED** |
| `describe-route-tables` | — | `--route-table-ids`, `--filters`, `--max-results` | None | NATS `ec2.DescribeRouteTables` → daemon lists route tables from KV → return tables with routes | 1. List all route tables<br>2. Filter by VPC | **NOT STARTED** |
| `create-route` | — | `--route-table-id`, `--destination-cidr-block`, `--gateway-id` or `--nat-gateway-id` or `--egress-only-internet-gateway-id` | Route table and target must exist | NATS `ec2.CreateRoute` → daemon adds route entry → configure OVS flow for destination CIDR → update metadata | 1. Add internet route via IGW<br>2. Add route to NAT gateway<br>3. Add route to EIGW<br>4. Conflicting route (error) | **NOT STARTED** |
| `delete-route` | — | `--route-table-id`, `--destination-cidr-block` | Route must exist | NATS `ec2.DeleteRoute` → daemon removes route entry → remove OVS flow → update metadata | 1. Delete existing route<br>2. Delete non-existent route (error)<br>3. Delete local route (error) | **NOT STARTED** |
| `replace-route` | — | `--route-table-id`, `--destination-cidr-block`, `--gateway-id` or `--nat-gateway-id` | Route must exist | NATS `ec2.ReplaceRoute` → daemon replaces route target → update OVS flow → update metadata | 1. Replace IGW route with NAT<br>2. Replace non-existent route (error) | **NOT STARTED** |
| `associate-route-table` | — | `--route-table-id`, `--subnet-id` or `--gateway-id` | Route table and target must exist | NATS `ec2.AssociateRouteTable` → daemon links route table to subnet/gateway → return rtbassoc-ID | 1. Associate with subnet<br>2. Missing route table ID (error) | **NOT STARTED** |
| `disassociate-route-table` | — | `--association-id` | Association must exist, not main table association | NATS `ec2.DisassociateRouteTable` → daemon unlinks route table → return success | 1. Disassociate existing<br>2. Disassociate main table (error) | **NOT STARTED** |
### EC2 - VPC Peering
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-vpc-peering-connection` | — | `--vpc-id`, `--peer-vpc-id`, `--peer-owner-id`, `--peer-region`, `--tag-specifications` | Both VPCs must exist | NATS `ec2.CreateVpcPeeringConnection` → daemon creates peering in KV with pcx-ID → state=pending-acceptance → return peering connection | 1. Create peering between two VPCs<br>2. Missing VPC ID (error)<br>3. Overlapping CIDRs (error) | **NOT STARTED** |
| `accept-vpc-peering-connection` | — | `--vpc-peering-connection-id`, `--dry-run` | Peering must exist in pending-acceptance state | NATS `ec2.AcceptVpcPeeringConnection` → daemon updates state to active → configure OVS cross-VPC flows → return peering | 1. Accept pending peering<br>2. Accept already active (error) | **NOT STARTED** |
| `reject-vpc-peering-connection` | — | `--vpc-peering-connection-id`, `--dry-run` | Peering must exist in pending-acceptance state | NATS `ec2.RejectVpcPeeringConnection` → daemon updates state to rejected → return peering | 1. Reject pending peering<br>2. Reject already active (error) | **NOT STARTED** |
| `delete-vpc-peering-connection` | — | `--vpc-peering-connection-id`, `--dry-run` | Peering must exist | NATS `ec2.DeleteVpcPeeringConnection` → daemon removes OVS cross-VPC flows → delete from KV → return success | 1. Delete active peering<br>2. Delete non-existent (error) | **NOT STARTED** |
| `describe-vpc-peering-connections` | — | `--vpc-peering-connection-ids`, `--filters`, `--max-results` | None | NATS `ec2.DescribeVpcPeeringConnections` → daemon lists peerings from KV → return list with state | 1. List all peerings<br>2. Filter by ID<br>3. Filter by VPC | **NOT STARTED** |
| `modify-vpc-peering-connection-options` | — | `--vpc-peering-connection-id`, `--requester-peering-connection-options`, `--accepter-peering-connection-options` | Peering must be active | NATS `ec2.ModifyVpcPeeringConnectionOptions` → daemon updates peering options in KV → return modified options | 1. Enable DNS resolution<br>2. Missing peering ID (error) | **NOT STARTED** |
### EC2 - VPC Endpoints
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-vpc-endpoint` | — | `--vpc-id`, `--service-name`, `--vpc-endpoint-type` (Gateway/Interface), `--route-table-ids`, `--subnet-ids`, `--tag-specifications` | VPC must exist | NATS `ec2.CreateVpcEndpoint` → daemon creates endpoint in KV with vpce-ID → return endpoint | 1. Create Gateway endpoint for S3<br>2. Create Interface endpoint<br>3. Missing VPC ID (error) | **NOT STARTED** |
| `delete-vpc-endpoints` | — | `--vpc-endpoint-ids`, `--dry-run` | Endpoints must exist | NATS `ec2.DeleteVpcEndpoints` → daemon deletes endpoints from KV → return success/failure per endpoint | 1. Delete existing endpoint<br>2. Delete non-existent (error) | **NOT STARTED** |
| `describe-vpc-endpoints` | — | `--vpc-endpoint-ids`, `--filters`, `--max-results` | None | NATS `ec2.DescribeVpcEndpoints` → daemon lists endpoints from KV → return list with state and type | 1. List all endpoints<br>2. Filter by VPC<br>3. Filter by service name | **NOT STARTED** |
| `describe-vpc-endpoint-services` | — | `--service-names`, `--filters`, `--max-results` | None | NATS `ec2.DescribeVpcEndpointServices` → daemon returns available services (S3, DynamoDB equivalents) → return service list | 1. List available services<br>2. Filter by service name | **NOT STARTED** |
| `modify-vpc-endpoint` | — | `--vpc-endpoint-id`, `--add-route-table-ids`, `--remove-route-table-ids`, `--add-subnet-ids`, `--remove-subnet-ids` | Endpoint must exist | NATS `ec2.ModifyVpcEndpoint` → daemon updates endpoint route tables/subnets in KV → return success | 1. Add route table<br>2. Remove subnet<br>3. Missing endpoint ID (error) | **NOT STARTED** |
### EC2 - VPN Gateway & Customer Gateway
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-customer-gateway` | — | `--type` (ipsec.1), `--bgp-asn`, `--ip-address`, `--tag-specifications` | None | NATS `ec2.CreateCustomerGateway` → daemon stores in JetStream KV with cgw-ID → return customer gateway | 1. Create with BGP ASN and IP<br>2. Missing required fields (error) | **NOT STARTED** |
| `delete-customer-gateway` | — | `--customer-gateway-id`, `--dry-run` | Must exist, no VPN connections | NATS `ec2.DeleteCustomerGateway` → daemon verifies no active connections → delete from KV → return success | 1. Delete existing<br>2. Delete with active VPN (error) | **NOT STARTED** |
| `describe-customer-gateways` | — | `--customer-gateway-ids`, `--filters` | None | NATS `ec2.DescribeCustomerGateways` → daemon lists from KV → return list with BGP/IP info | 1. List all<br>2. Filter by ID | **NOT STARTED** |
| `create-vpn-gateway` | — | `--type` (ipsec.1), `--amazon-side-asn`, `--tag-specifications` | None | NATS `ec2.CreateVpnGateway` → daemon stores in JetStream KV with vgw-ID → return VPN gateway | 1. Create VPN gateway<br>2. Verify in describe | **NOT STARTED** |
| `delete-vpn-gateway` | — | `--vpn-gateway-id`, `--dry-run` | Must be detached | NATS `ec2.DeleteVpnGateway` → daemon verifies detached → delete from KV → return success | 1. Delete detached gateway<br>2. Delete attached (error) | **NOT STARTED** |
| `attach-vpn-gateway` | — | `--vpn-gateway-id`, `--vpc-id` | Both must exist | NATS `ec2.AttachVpnGateway` → daemon links VPN gateway to VPC → return attachment | 1. Attach to VPC<br>2. Already attached (error) | **NOT STARTED** |
| `detach-vpn-gateway` | — | `--vpn-gateway-id`, `--vpc-id` | Must be attached | NATS `ec2.DetachVpnGateway` → daemon unlinks VPN gateway from VPC → return success | 1. Detach from VPC<br>2. Not attached (error) | **NOT STARTED** |
| `describe-vpn-gateways` | — | `--vpn-gateway-ids`, `--filters` | None | NATS `ec2.DescribeVpnGateways` → daemon lists from KV → return list with attachment info | 1. List all<br>2. Filter by VPC attachment | **NOT STARTED** |
### EC2 - VPN Connections
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-vpn-connection` | — | `--type` (ipsec.1), `--customer-gateway-id`, `--vpn-gateway-id`, `--options`, `--tag-specifications` | Customer gateway and VPN gateway must exist | NATS `ec2.CreateVpnConnection` → daemon stores connection in JetStream KV with vpn-ID → return connection with tunnel info | 1. Create IPsec connection<br>2. Missing customer gateway (error)<br>3. Missing VPN gateway (error) | **NOT STARTED** |
| `delete-vpn-connection` | — | `--vpn-connection-id`, `--dry-run` | Connection must exist | NATS `ec2.DeleteVpnConnection` → daemon deletes from KV → return success | 1. Delete existing connection<br>2. Delete non-existent (error) | **NOT STARTED** |
| `describe-vpn-connections` | — | `--vpn-connection-ids`, `--filters` | None | NATS `ec2.DescribeVpnConnections` → daemon lists from KV → return list with state and tunnel info | 1. List all connections<br>2. Filter by ID<br>3. Filter by VPN gateway | **NOT STARTED** |
| `modify-vpn-connection` | — | `--vpn-connection-id`, `--vpn-gateway-id`, `--customer-gateway-id` | Connection must exist | NATS `ec2.ModifyVpnConnection` → daemon updates connection target in KV → return modified connection | 1. Change VPN gateway<br>2. Missing connection ID (error) | **NOT STARTED** |
| `modify-vpn-connection-options` | — | `--vpn-connection-id`, `--local-ipv4-network-cidr`, `--remote-ipv4-network-cidr` | Connection must exist | NATS `ec2.ModifyVpnConnectionOptions` → daemon updates tunnel options in KV → return modified connection | 1. Modify CIDR ranges<br>2. Missing connection ID (error) | **NOT STARTED** |
### EC2 - Network ACLs
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-network-acl` | — | `--vpc-id`, `--tag-specifications` | VPC must exist | NATS `ec2.CreateNetworkAcl` → daemon creates NACL in KV with acl-ID → add default deny-all rules → return NACL | 1. Create NACL in VPC<br>2. Missing VPC ID (error)<br>3. Verify default deny rules | **NOT STARTED** |
| `delete-network-acl` | — | `--network-acl-id`, `--dry-run` | Must not be default NACL | NATS `ec2.DeleteNetworkAcl` → daemon verifies not default → delete from KV → return success | 1. Delete non-default NACL<br>2. Delete default NACL (error) | **NOT STARTED** |
| `describe-network-acls` | — | `--network-acl-ids`, `--filters`, `--max-results` | None | NATS `ec2.DescribeNetworkAcls` → daemon lists NACLs from KV → return list with entries | 1. List all NACLs<br>2. Filter by VPC | **NOT STARTED** |
| `create-network-acl-entry` | — | `--network-acl-id`, `--rule-number`, `--protocol`, `--rule-action` (allow/deny), `--cidr-block`, `--ingress`/`--egress`, `--port-range` | NACL must exist | NATS `ec2.CreateNetworkAclEntry` → daemon adds rule entry to NACL in KV → configure OVS flow → return success | 1. Add ingress allow rule<br>2. Add egress deny rule<br>3. ICMP type/code parsing<br>4. Missing NACL ID (error) | **NOT STARTED** |
| `delete-network-acl-entry` | — | `--network-acl-id`, `--rule-number`, `--ingress`/`--egress` | Entry must exist | NATS `ec2.DeleteNetworkAclEntry` → daemon removes rule from NACL → remove OVS flow → return success | 1. Delete existing entry<br>2. Delete non-existent entry (error) | **NOT STARTED** |
| `replace-network-acl-association` | — | `--association-id`, `--network-acl-id` | Both must exist | NATS `ec2.ReplaceNetworkAclAssociation` → daemon swaps NACL for subnet → return new association ID | 1. Replace association<br>2. Missing association ID (error) | **NOT STARTED** |
| `replace-network-acl-entry` | — | `--network-acl-id`, `--rule-number`, `--protocol`, `--rule-action`, `--cidr-block`, `--ingress`/`--egress`, `--port-range` | Entry must exist | NATS `ec2.ReplaceNetworkAclEntry` → daemon replaces rule in KV → update OVS flow → return success | 1. Replace existing entry<br>2. Replace non-existent entry (error) | **NOT STARTED** |
### EC2 - Prefix Lists
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-managed-prefix-list` | — | `--prefix-list-name`, `--address-family` (IPv4/IPv6), `--max-entries`, `--entries`, `--tag-specifications` | None | NATS `ec2.CreateManagedPrefixList` → daemon stores prefix list in JetStream KV with pl-ID → return prefix list with version | 1. Create with IPv4 entries<br>2. Name, address family, max entries required<br>3. Verify in describe | **NOT STARTED** |
| `delete-managed-prefix-list` | — | `--prefix-list-id`, `--dry-run` | Must exist, not referenced by security groups/route tables | NATS `ec2.DeleteManagedPrefixList` → daemon verifies not referenced → delete from KV → return success | 1. Delete unreferenced list<br>2. Delete referenced list (error) | **NOT STARTED** |
| `describe-managed-prefix-lists` | — | `--prefix-list-ids`, `--filters`, `--max-results` | None | NATS `ec2.DescribeManagedPrefixLists` → daemon lists from KV → return list with state and version | 1. List all prefix lists<br>2. Filter by ID | **NOT STARTED** |
| `modify-managed-prefix-list` | — | `--prefix-list-id`, `--current-version`, `--add-entries`, `--remove-entries`, `--prefix-list-name` | Must exist, current version must match | NATS `ec2.ModifyManagedPrefixList` → daemon validates version → apply changes → increment version → return updated list | 1. Add entries<br>2. Remove entries<br>3. Version mismatch (error) | **NOT STARTED** |
| `get-managed-prefix-list-entries` | — | `--prefix-list-id`, `--target-version`, `--max-results` | Prefix list must exist | NATS `ec2.GetManagedPrefixListEntries` → daemon reads entries from KV → return entry list | 1. List entries<br>2. List entries at specific version | **NOT STARTED** |
| `get-managed-prefix-list-associations` | — | `--prefix-list-id`, `--max-results` | Prefix list must exist | NATS `ec2.GetManagedPrefixListAssociations` → daemon finds references in SGs/route tables → return association list | 1. List associations<br>2. No associations returns empty | **NOT STARTED** |
### EC2 - Network Interfaces
ENIs are managed via `hive-vpc-enis` KV bucket with CAS-based IPAM allocation from `hive-vpc-ipam`. ENIs are auto-created by `run-instances --subnet-id` and auto-deleted on termination. MAC addresses are deterministically generated from the ENI ID. Attach/detach are internal operations used by instance lifecycle — no standalone attach/detach API yet.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-network-interface` | `--subnet-id`, `--private-ip-address`, `--description`, `--tag-specifications` | `--groups`, `--dry-run` | Subnet must exist | Gateway validates SubnetId (required) → NATS `ec2.CreateNetworkInterface` → daemon verifies subnet exists in `hive-vpc-subnets` KV → if no PrivateIpAddress, allocates from IPAM pool (CAS-based, up to 5 retries) → generates eni-ID → generates deterministic MAC from ENI ID → stores ENIRecord in `hive-vpc-enis` KV → publishes `vpc.create-port` event to vpcd (creates OVN logical port) → returns NetworkInterface with status=available | 1. Create in subnet (auto-allocate IP)<br>2. Create with specific private IP<br>3. Missing subnet ID (MissingParameter)<br>4. Non-existent subnet (error)<br>5. Tags applied at creation | **DONE** |
| `delete-network-interface` | `--network-interface-id` | `--dry-run` | Must not be attached (status != in-use) | NATS `ec2.DeleteNetworkInterface` → daemon verifies ENI exists and status is not "in-use" (InvalidNetworkInterfaceInUse) → releases IP back to IPAM pool → deletes from `hive-vpc-enis` KV → publishes `vpc.delete-port` event to vpcd | 1. Delete detached ENI<br>2. Delete attached ENI (InvalidNetworkInterfaceInUse)<br>3. Delete non-existent (error) | **DONE** |
| `describe-network-interfaces` | `--network-interface-ids`, `--filters` (subnet-id, vpc-id, attachment.instance-id) | `--max-results`, `--dry-run` | None | NATS `ec2.DescribeNetworkInterfaces` → daemon lists all keys from `hive-vpc-enis` KV → applies filters (subnet-id, vpc-id, attachment.instance-id) → filters by ENI IDs if specified → returns error for non-existent requested IDs | 1. List all ENIs<br>2. Filter by subnet-id<br>3. Filter by vpc-id<br>4. Filter by attachment.instance-id<br>5. Non-existent ENI returns error | **DONE** |
| `attach-network-interface` | — | `--network-interface-id`, `--instance-id`, `--device-index` | ENI and instance must exist | Internal operation used by `run-instances --subnet-id`. Marks ENI as "in-use", sets instance attachment, generates attachment ID. No standalone API endpoint yet. | 1. Attach ENI to instance<br>2. Already attached (error)<br>3. Missing device index (error) | **NOT STARTED** (internal only) |
| `detach-network-interface` | — | `--attachment-id`, `--force` | ENI must be attached | Internal operation used by terminate-instances. Marks ENI as "available", clears attachment info. No standalone API endpoint yet. | 1. Detach ENI<br>2. Force detach<br>3. Not attached (error) | **NOT STARTED** (internal only) |
| `modify-network-interface-attribute` | — | `--network-interface-id`, `--description`, `--source-dest-check`, `--groups` | ENI must exist | NATS `ec2.ModifyNetworkInterfaceAttribute` → daemon updates ENI attributes in KV → return success | 1. Modify description<br>2. Toggle source/dest check<br>3. Update security groups | **NOT STARTED** |
| `assign-private-ip-addresses` | — | `--network-interface-id`, `--private-ip-addresses` or `--secondary-private-ip-address-count` | ENI must exist | NATS `ec2.AssignPrivateIpAddresses` → daemon allocates IPs from subnet pool → assign to ENI → return assigned IPs | 1. Assign specific IPs<br>2. Assign by count<br>3. Cannot specify both (error) | **NOT STARTED** |
| `unassign-private-ip-addresses` | — | `--network-interface-id`, `--private-ip-addresses` | IPs must be assigned to ENI | NATS `ec2.UnassignPrivateIpAddresses` → daemon releases IPs → return success | 1. Unassign secondary IP<br>2. Unassign primary IP (error) | **NOT STARTED** |
### EC2 - Launch Templates
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-launch-template` | — | `--launch-template-name`, `--launch-template-data` (ImageId, InstanceType, etc.), `--tag-specifications` | None | NATS `ec2.CreateLaunchTemplate` → daemon stores template in JetStream KV with lt-ID → version 1 → return template metadata | 1. Create template with full config<br>2. Duplicate name (error)<br>3. Verify in describe output | **NOT STARTED** |
| `create-launch-template-version` | — | `--launch-template-id` or `--launch-template-name`, `--launch-template-data`, `--source-version` | Template must exist | NATS `ec2.CreateLaunchTemplateVersion` → daemon creates new version in KV → return version details | 1. Create version from scratch<br>2. Create version from source version<br>3. Template not found (error) | **NOT STARTED** |
| `delete-launch-template` | — | `--launch-template-id` or `--launch-template-name`, `--dry-run` | Template must exist | NATS `ec2.DeleteLaunchTemplate` → daemon deletes template and all versions from KV → return deleted template info | 1. Delete by ID<br>2. Delete by name<br>3. Non-existent template (error) | **NOT STARTED** |
| `describe-launch-templates` | — | `--launch-template-ids`, `--launch-template-names`, `--filters` | None | NATS `ec2.DescribeLaunchTemplates` → daemon lists templates from KV → return list | 1. List all templates<br>2. Filter by name<br>3. Filter by ID | **NOT STARTED** |
| `describe-launch-template-versions` | — | `--launch-template-id` or `--launch-template-name`, `--versions`, `--min-version`, `--max-version` | Template must exist | NATS `ec2.DescribeLaunchTemplateVersions` → daemon lists versions from KV → return version list with data | 1. List all versions<br>2. List specific version<br>3. Template not found (error) | **NOT STARTED** |
### EC2 - Misc Operations
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `delete-network-interface-permission` | — | `--network-interface-permission-id`, `--force` | Permission must exist | NATS `ec2.DeleteNetworkInterfacePermission` → daemon deletes permission from KV → return success | 1. Delete existing permission<br>2. Force delete | **NOT STARTED** |
| `enable-address-transfer` | — | `--allocation-id`, `--transfer-account-id` | EIP must exist | NATS `ec2.EnableAddressTransfer` → daemon stores transfer in JetStream KV with 7-day expiration → return AllocationId and TransferAccountId | 1. Enable transfer<br>2. Verify transfer stored with expiration | **NOT STARTED** |
| `disable-address-transfer` | — | `--allocation-id` | Transfer must exist | NATS `ec2.DisableAddressTransfer` → daemon updates transfer status to disabled in KV → return status | 1. Disable active transfer<br>2. Disable non-existent transfer (error) | **NOT STARTED** |
### EBS Direct API
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `start-snapshot` | — | `--volume-size`, `--parent-snapshot-id`, `--description`, `--encrypted` | None | Initialize snapshot in viperblock → return snap-ID for block-level writes | 1. Start new snapshot<br>2. Start incremental from parent | **NOT STARTED** |
| `put-snapshot-block` | — | `--snapshot-id`, `--block-index`, `--block-data`, `--checksum` | start-snapshot | Write block data to snapshot at specified index in viperblock → verify checksum | 1. Write single block<br>2. Write with bad checksum (error) | **NOT STARTED** |
| `get-snapshot-block` | — | `--snapshot-id`, `--block-index` | Snapshot must exist | Read block data from snapshot at specified index → return data with checksum | 1. Read existing block<br>2. Read sparse block (zeros) | **NOT STARTED** |
| `complete-snapshot` | — | `--snapshot-id`, `--changed-blocks-count` | start-snapshot, put-snapshot-block | Finalize snapshot → mark as completed → sync to Predastore | 1. Complete valid snapshot<br>2. Complete with wrong block count (error) | **NOT STARTED** |
| `list-snapshot-blocks` | — | `--snapshot-id`, `--max-results`, `--next-token` | Snapshot must exist | List all non-empty blocks in snapshot with tokens and sizes | 1. List blocks of populated snapshot<br>2. List blocks of empty snapshot | **NOT STARTED** |
| `list-changed-blocks` | — | `--second-snapshot-id`, `--first-snapshot-id`, `--max-results` | Snapshots must exist | Compare two snapshots → return list of differing block indexes | 1. Compare parent and child snapshots<br>2. Compare unrelated snapshots | **NOT STARTED** |
### S3 (via Predastore)
Currently S3 operations are handled directly by Predastore. Consider moving control/data plane to Hive format for AWS gateway integration.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `s3 mb` (CreateBucket) | — | `s3://bucket-name` | None | NATS `s3.createbucket` → Predastore creates bucket → return success | 1. Create valid bucket<br>2. Duplicate bucket name (error)<br>3. Invalid bucket name (error) | **NOT STARTED** (Predastore handles directly) |
| `s3 rb` (DeleteBucket) | — | `s3://bucket-name`, `--force` | Bucket must exist | NATS `s3.deletebucket` → verify bucket empty (or force) → Predastore deletes bucket | 1. Delete empty bucket<br>2. Delete non-empty bucket (error)<br>3. Force delete non-empty bucket | **NOT STARTED** |
| `s3 ls` (ListBuckets/Objects) | — | `s3://bucket/prefix`, `--recursive` | None | NATS `s3.listobjects` → Predastore lists objects with prefix → return list | 1. List all buckets<br>2. List objects in bucket<br>3. List with prefix filter | **NOT STARTED** |
| `s3 cp` (PutObject/GetObject) | — | `source dest`, `--recursive`, `--acl` | Bucket must exist | NATS `s3.putobject`/`s3.getobject` → Predastore stores/retrieves with Reed-Solomon encoding | 1. Upload single file<br>2. Download single file<br>3. Recursive upload/download<br>4. Large file (multipart) | **NOT STARTED** |
| `s3 rm` (DeleteObject) | — | `s3://bucket/key`, `--recursive` | Object must exist | NATS `s3.deleteobject` → Predastore deletes object shards | 1. Delete single object<br>2. Recursive delete<br>3. Delete non-existent (idempotent) | **NOT STARTED** |
| `s3 sync` | — | `source dest`, `--delete`, `--exclude`, `--include` | Bucket must exist | Compare source and dest → upload changed/new files → optionally delete removed files | 1. Sync local dir to S3<br>2. Sync with --delete<br>3. Sync with exclude filter | **NOT STARTED** |
### IAM - User Management
All IAM operations are account-scoped. Users, policies, and access keys are namespaced per account. The gateway extracts the account ID from the authenticated access key via `c.Locals("sigv4.accountId")` and passes it to all IAM service methods. Root user (account `000000000000`) bypasses policy evaluation entirely.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-user` | `--user-name`, `--path` | `--tags`, `--permissions-boundary` | None | Gateway validates UserName non-empty → IAMService.CreateUser(accountID, input) → generates AIDA-prefixed UserId → stores in KV at key `{accountID}.{userName}` → ARN includes real account ID → conditional create for race safety | 1. Create user<br>2. Duplicate name in same account (EntityAlreadyExists)<br>3. Same name in different account (succeeds)<br>4. Missing username (InvalidInput)<br>5. Default path is `/` | **DONE** |
| `get-user` | `--user-name` | — | User must exist in caller's account | Gateway validates UserName → IAMService.GetUser(accountID, input) → reads from KV at `{accountID}.{userName}` → returns User with ARN, Path, CreateDate | 1. Get existing user<br>2. Get non-existent user (NoSuchEntity)<br>3. Cannot see user in different account (NoSuchEntity) | **DONE** |
| `list-users` | `--path-prefix` | `--max-items`, `--marker` | None | IAMService.ListUsers(accountID, input) → prefix-scans KV for `{accountID}.*` → filters by path prefix if specified → returns user list scoped to caller's account | 1. List all users in account<br>2. List with path prefix filter<br>3. Empty list<br>4. Does not show other accounts' users | **DONE** |
| `delete-user` | `--user-name` | — | User must exist, no access keys attached | Gateway validates UserName → IAMService.DeleteUser(accountID, input) → checks user has no access keys → deletes from KV | 1. Delete user with no keys<br>2. Delete user with keys attached (DeleteConflict)<br>3. Delete non-existent user (NoSuchEntity) | **DONE** |
### IAM - Access Key Management
Access keys are globally unique (keyed by AccessKeyID) but associated with an account. Max 2 keys per user. Secrets encrypted with AES-256-GCM using master key.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-access-key` | `--user-name` | — | User must exist | IAMService.CreateAccessKey(accountID, input) → generates AKIA-prefixed key ID + random secret → encrypts secret with master key → stores in KV (globally unique key ID) → updates user's AccessKeys list → returns plaintext secret (one-time only) | 1. Create key for user (AKIA prefix)<br>2. Max 2 keys per user (LimitExceeded on 3rd)<br>3. Non-existent user (NoSuchEntity)<br>4. Secret only returned on creation | **DONE** |
| `list-access-keys` | `--user-name` | `--max-items`, `--marker` | None | IAMService.ListAccessKeys(accountID, input) → reads user's AccessKeys list → fetches each key's metadata → returns key IDs and status (no secrets) | 1. List keys for user<br>2. No keys returns empty list<br>3. Shows Status (Active/Inactive) but not secrets | **DONE** |
| `delete-access-key` | `--access-key-id`, `--user-name` | — | Key must exist | IAMService.DeleteAccessKey(accountID, input) → removes key from global KV → removes key ID from user's AccessKeys list | 1. Delete existing key<br>2. Delete non-existent key (NoSuchEntity) | **DONE** |
| `update-access-key` | `--access-key-id`, `--user-name`, `--status` (Active/Inactive) | — | Key must exist | IAMService.UpdateAccessKey(input) → reads key from global KV (key ID is globally unique, no accountID needed) → updates Status field → inactive keys rejected at SigV4 auth | 1. Deactivate key (Inactive)<br>2. Reactivate key (Active)<br>3. Non-existent key (NoSuchEntity)<br>4. Inactive key → auth returns InvalidClientTokenId | **DONE** |
### IAM - Policy Management
Policies are account-scoped. KV key is `{accountID}.{policyName}`. Policy documents require `Version: "2012-10-17"` and valid Statement array. Supports wildcard action matching (`ec2:*`, `ec2:Describe*`, `*`). Evaluation order: explicit Deny > explicit Allow > implicit Deny.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `create-policy` | `--policy-name`, `--policy-document`, `--path`, `--description` | `--tags` | None | Gateway validates PolicyName and PolicyDocument non-empty → IAMService.CreatePolicy(accountID, input) → validates JSON (Version, Statement with Effect/Action/Resource) → generates ANPA-prefixed PolicyId → stores at `{accountID}.{policyName}` → ARN includes real account ID | 1. Create with valid document<br>2. Malformed JSON (MalformedPolicyDocument)<br>3. Missing Effect/Action/Resource (MalformedPolicyDocument)<br>4. Duplicate name in same account (EntityAlreadyExists)<br>5. Same name in different account (succeeds)<br>6. With path and description | **DONE** |
| `get-policy` | `--policy-arn` | — | Policy must exist | Gateway validates PolicyArn → extracts account ID and policy name from ARN → IAMService.GetPolicy(accountID, input) → returns policy metadata (no document) with AttachmentCount | 1. Get existing policy<br>2. Non-existent policy (NoSuchEntity)<br>3. Returns metadata only, not document | **DONE** |
| `get-policy-version` | `--policy-arn`, `--version-id` | — | Policy must exist | Gateway validates PolicyArn and VersionId → IAMService.GetPolicyVersion(accountID, input) → returns PolicyVersion with URL-encoded Document, VersionId, IsDefaultVersion | 1. Get version v1 (returns document)<br>2. Non-existent version (NoSuchEntity)<br>3. Non-existent policy (NoSuchEntity) | **DONE** |
| `list-policies` | — | `--scope`, `--only-attached`, `--path-prefix`, `--max-items`, `--marker` | None | IAMService.ListPolicies(accountID, input) → prefix-scans KV for `{accountID}.*` → returns all policies in caller's account | 1. List all policies in account<br>2. Empty list<br>3. Does not show other accounts' policies | **DONE** |
| `delete-policy` | `--policy-arn` | — | Policy must exist, not attached to any user | Gateway validates PolicyArn → IAMService.DeletePolicy(accountID, input) → checks no users have this policy attached (scans account's users) → deletes from KV | 1. Delete unattached policy<br>2. Delete attached policy (DeleteConflict)<br>3. Delete non-existent (NoSuchEntity) | **DONE** |
### IAM - Policy Attachment
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---------|-------------------|---------------|---------------|-------------|------------|--------|
| `attach-user-policy` | `--user-name`, `--policy-arn` | — | User and policy must exist in caller's account | Gateway validates both params → IAMService.AttachUserPolicy(accountID, input) → verifies policy exists → adds policy ARN to user's AttachedPolicies list → idempotent (re-attach succeeds without duplicate) | 1. Attach policy to user<br>2. Already attached (idempotent, no error)<br>3. Non-existent user (NoSuchEntity)<br>4. Non-existent policy (NoSuchEntity) | **DONE** |
| `detach-user-policy` | `--user-name`, `--policy-arn` | — | Policy must be attached to user | Gateway validates both params → IAMService.DetachUserPolicy(accountID, input) → removes policy ARN from user's AttachedPolicies list | 1. Detach attached policy<br>2. Policy not attached (NoSuchEntity)<br>3. Non-existent user (NoSuchEntity) | **DONE** |
| `list-attached-user-policies` | `--user-name` | `--path-prefix`, `--max-items`, `--marker` | User must exist | IAMService.ListAttachedUserPolicies(accountID, input) → reads user's AttachedPolicies list → returns policy ARNs and names | 1. List attached policies<br>2. No policies returns empty list | **DONE** |
### IAM - Policy Evaluation
The gateway evaluates IAM policies on every request (except root user bypass). Policy evaluation is internal — not an AWS API.
| Feature | Description | Status |
|---------|-------------|--------|
| Root user bypass | Root user (`000000000000:root`) skips all policy checks | **DONE** |
| Default deny | Users with no attached policies are denied all actions | **DONE** |
| Explicit allow | Matching Allow statement grants access | **DONE** |
| Explicit deny | Deny statements override Allow (deny wins) | **DONE** |
| Wildcard matching | `*` matches all, `ec2:*` matches all EC2, `ec2:Describe*` matches prefix | **DONE** |
| Account-scoped evaluation | Policies resolved from caller's account context | **DONE** |
| EC2 action mapping | 54 EC2 actions mapped to IAM policy strings | **DONE** |
| IAM action mapping | 16 IAM actions mapped to IAM policy strings | **DONE** |
+373
View File
@@ -0,0 +1,373 @@
# Hive Architecture
Hive is an AWS-compatible infrastructure platform for bare-metal, edge, and on-premise deployments. It provides EC2-compatible VM orchestration backed by QEMU/KVM, with storage provided by companion projects Viperblock (EBS) and Predastore (S3).
## High-Level Architecture
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ AWS SDK / CLI │
│ (aws ec2 run-instances --endpoint-url ...) │
└─────────────────────────────────────────────────────────────────────────────┘
│
│ HTTPS (SigV4 Auth)
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ AWS Gateway (Port 9999) │
│ │
│ • TLS termination • AWS SigV4 authentication │
│ • EC2 Query Protocol parsing • Routes to service handlers │
│ • XML response generation • NATS client connection │
│ │
│ File: hive/services/awsgw/awsgw.go │
└─────────────────────────────────────────────────────────────────────────────┘
│
│ JSON over NATS
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ NATS Message Broker │
│ │
│ Topics: Queue Groups: │
│ • ec2.RunInstances • hive-workers (load balanced) │
│ • ec2.DescribeInstances • No queue (fan-out to all nodes) │
│ • ec2.cmd.{instanceID} (per-instance start/stop/terminate) │
│ • ebs.mount / ebs.unmount │
└─────────────────────────────────────────────────────────────────────────────┘
│
┌─────────────────┼─────────────────┐
▼ ▼ ▼
┌──────────────────────┐ ┌──────────────────────┐ ┌──────────────────────┐
│ Hive Daemon │ │ Hive Daemon │ │ Hive Daemon │
│ (Node 1) │ │ (Node 2) │ │ (Node N) │
│ │ │ │ │ │
│ • NATS subscriber │ │ • NATS subscriber │ │ • NATS subscriber │
│ • Resource manager │ │ • Resource manager │ │ • Resource manager │
│ • QEMU/KVM launcher │ │ • QEMU/KVM launcher │ │ • QEMU/KVM launcher │
│ • QMP monitoring │ │ • QMP monitoring │ │ • QMP monitoring │
│ │ │ │ │ │
│ File: daemon.go │ │ File: daemon.go │ │ File: daemon.go │
└──────────────────────┘ └──────────────────────┘ └──────────────────────┘
│ │ │
▼ ▼ ▼
┌──────────────────────┐ ┌──────────────────────┐ ┌──────────────────────┐
│ QEMU/KVM VMs │ │ QEMU/KVM VMs │ │ QEMU/KVM VMs │
└──────────────────────┘ └──────────────────────┘ └──────────────────────┘
```
## Request Flow
### 1. AWS SDK Request
Users interact with Hive using standard AWS SDKs or the AWS CLI by pointing to a custom endpoint:
```bash
aws --endpoint-url https://localhost:9999 --no-verify-ssl ec2 run-instances \
--image-id ami-debian12 \
--instance-type t3.micro \
--key-name my-keypair
```
The AWS SDK formats this as an HTTP POST with:
- AWS SigV4 authentication headers
- EC2 Query Protocol body (Action=RunInstances&ImageId=ami-debian12...)
### 2. AWS Gateway
The gateway (`hive/services/awsgw/awsgw.go:57-107`) is the entry point:
```go
// awsgw.go:74 - Connect to NATS
natsConn, err := nats.Connect(nodeConfig.NATS.Host, opts...)
// awsgw.go:89-94 - Create gateway with NATS connection
gw := gateway.GatewayConfig{
NATSConn: natsConn,
Config: nodeConfig.AWSGW.Config,
}
// awsgw.go:103 - Start TLS listener
log.Fatal(app.ListenTLS(nodeConfig.AWSGW.Host, nodeConfig.AWSGW.TLSCert, nodeConfig.AWSGW.TLSKey))
```
Request routing (`hive/gateway/gateway.go:125-156`):
1. **Authentication**: SigV4 middleware validates AWS credentials (`gateway.go:104`)
2. **Service Detection**: Extracts service name (ec2, iam, account) from Authorization header (`gateway.go:158-186`)
3. **Action Dispatch**: Routes to service-specific handler (`gateway.go:136-145`)
```go
// gateway.go:136-145
switch svc {
case "ec2":
err = gw.EC2_Request(ctx)
case "account":
err = gw.Account_Request(ctx)
case "iam":
err = gw.IAM_Request(ctx)
}
```
### 3. EC2 Handler
The EC2 handler (`hive/gateway/ec2.go`) parses the Action parameter and delegates to specific handlers:
```go
// ec2.go - Action routing
switch action {
case "RunInstances":
output, err := gateway_ec2_instance.RunInstances(input, gw.NATSConn)
case "DescribeInstances":
output, err := gateway_ec2_instance.DescribeInstances(input, gw.NATSConn, expectedNodes)
// ... more actions
}
```
### 4. NATS Messaging
The gateway communicates with daemons via NATS request/response (`hive/handlers/ec2/instance/service_nats.go:24-57`):
```go
// service_nats.go:24-57
func (s *NATSInstanceService) RunInstances(input *ec2.RunInstancesInput) (*ec2.Reservation, error) {
// Marshal input to JSON
jsonData, err := json.Marshal(input)
// Send request to daemon via NATS with 5 minute timeout
msg, err := s.natsConn.Request("ec2.RunInstances", jsonData, 5*time.Minute)
// Unmarshal response
var reservation ec2.Reservation
err = json.Unmarshal(msg.Data, &reservation)
return &reservation, nil
}
```
### 5. Daemon Processing
Daemons (`hive/daemon/daemon.go`) subscribe to NATS topics and handle requests:
```go
// daemon.go:319 - Subscribe with queue group for load balancing
d.natsSubscriptions["ec2.RunInstances"], err = d.natsConn.QueueSubscribe(
"ec2.RunInstances",
"hive-workers", // Queue group - only one daemon handles each request
d.handleEC2RunInstances,
)
// daemon.go:373 - Subscribe without queue group for fan-out
d.natsSubscriptions["ec2.DescribeInstances"], err = d.natsConn.Subscribe(
"ec2.DescribeInstances",
d.handleEC2DescribeInstances, // All daemons respond
)
```
**Queue Groups**: Topics subscribed with a queue group (`hive-workers`) are load-balanced - only one daemon handles each request. Topics without a queue group fan out to all daemons.
### 6. VM Launch
When a daemon handles `RunInstances`:
1. **Resource Check**: Validates CPU/memory availability (`daemon.go:66-74`)
2. **Volume Generation**: Creates boot, cloud-init, and EFI volumes via Viperblock
3. **Volume Mount**: Sends `ebs.mount` request to Viperblock, receives NBD URI
4. **QEMU Launch**: Builds and executes QEMU command with NBD-backed disks
5. **QMP Monitoring**: Establishes QEMU Machine Protocol connection for VM management
```go
// daemon.go - Volume mounting via NATS
msg, err := d.natsConn.Request("ebs.mount", ebsMountRequest, 10*time.Second)
// daemon.go - QEMU launch with NBD storage
cmd := exec.Command("qemu-system-x86_64",
"-drive", fmt.Sprintf("file=nbd:%s,format=raw", nbdURI),
// ... additional QEMU args
)
```
## Key Components
### Daemon Structure
```go
// daemon.go:76-102
type Daemon struct {
node string // Node identifier
clusterConfig *config.ClusterConfig // Cluster-wide configuration
config *config.Config // Node-specific configuration
natsConn *nats.Conn // NATS connection
resourceMgr *ResourceManager // CPU/Memory tracking
instanceService *InstanceServiceImpl // EC2 instance operations
keyService *KeyServiceImpl // SSH key operations
imageService *ImageServiceImpl // AMI operations
Instances vm.Instances // Local VMs
clusterApp *fiber.App // HTTP cluster manager
}
```
### Resource Manager
Tracks available and allocated CPU/memory to prevent overcommit:
```go
// daemon.go:66-74
type ResourceManager struct {
mu sync.RWMutex
availableVCPU int
availableMem float64
allocatedVCPU int
allocatedMem float64
instanceTypes map[string]InstanceType // t3.micro, t3.small, etc.
}
```
### NATS Topics
| Topic | Queue Group | Purpose |
|-------|-------------|---------|
| `ec2.RunInstances` | `hive-workers` | Launch new instances |
| `ec2.DescribeInstances` | None (fan-out) | Query all nodes for instances |
| `ec2.cmd.{instanceID}` | None (owner only) | Per-instance commands (start/stop/terminate, attach/detach volume) |
| `ec2.CreateKeyPair` | `hive-workers` | Generate SSH keypair |
| `ec2.DescribeKeyPairs` | `hive-workers` | List SSH keypairs |
| `ec2.DescribeImages` | `hive-workers` | List AMIs |
| `ebs.mount` | - | Mount volume via Viperblock |
| `ebs.unmount` | - | Unmount volume |
| `hive.admin.{node}.health` | None | Node health checks |
### Multi-Node Aggregation
For operations that need data from all nodes (like `DescribeInstances`), the gateway uses inbox-based fan-out (`hive/gateway/ec2/instance/DescribeInstances.go`):
```go
// DescribeInstances.go:16-111
func DescribeInstances(...) {
// Create unique inbox for collecting responses
inbox := nats.NewInbox()
sub, err := natsConn.SubscribeSync(inbox)
// Publish to all nodes (no queue group = all daemons respond)
err = natsConn.PublishRequest("ec2.DescribeInstances", inbox, jsonData)
// Collect responses from all nodes with 3-second timeout
for {
msg, err := sub.NextMsg(3 * time.Second)
allReservations = append(allReservations, nodeOutput.Reservations...)
}
return allReservations
}
```
## Storage Integration
### Viperblock (EBS)
Block storage volumes are managed via NATS messages to Viperblock:
```go
// Volume mount request
type EBSRequest struct {
Name string // Volume name (vol-xxxxx)
VolType string // gp2, io1, etc.
Boot bool // Boot volume flag
EFI bool // EFI boot volume
CloudInit bool // Cloud-init volume
NBDURI string // NBD URI returned from mount
}
```
Viperblock responds with an NBD URI that QEMU uses to access the block device.
### Predastore (S3)
Object storage used for:
- AMI image storage
- SSH public key storage (`/bucket/ec2/{key-name}.pub`)
- Volume metadata and configuration
## Configuration
Cluster configuration (`hive/config/config.go`):
```go
type ClusterConfig struct {
Epoch uint64 // Incremented on config changes
Node string // This node's identifier
Version string // Hive version
Nodes map[string]Config // Configuration for all nodes
}
type Config struct {
Node, Host, Region, AZ string
DataDir string
Daemon DaemonConfig // Cluster manager port
NATS NATSConfig // NATS broker address
Predastore PredastoreConfig // S3 endpoint
AWSGW AWSGWConfig // Gateway port, TLS certs
AccessKey, SecretKey string
}
```
## File Structure
```
hive/
├── hive/
│ ├── services/
│ │ └── awsgw/
│ │ └── awsgw.go # AWS Gateway service entry point
│ ├── gateway/
│ │ ├── gateway.go # Request routing, auth middleware
│ │ ├── ec2.go # EC2 action dispatcher
│ │ └── ec2/
│ │ └── instance/
│ │ ├── RunInstances.go
│ │ └── DescribeInstances.go
│ ├── handlers/
│ │ └── ec2/
│ │ ├── instance/
│ │ │ ├── service_nats.go # NATS request/response
│ │ │ └── service_impl.go # Business logic
│ │ ├── key/
│ │ └── image/
│ ├── daemon/
│ │ └── daemon.go # Daemon entry point, NATS subscriptions
│ ├── config/
│ │ └── config.go # Configuration structures
│ ├── vm/
│ │ └── instance.go # VM instance representation
│ └── qmp/
│ └── qmp.go # QEMU Machine Protocol client
└── cmd/
└── hive/
└── main.go # CLI entry point
```
## Development
### Running the Stack
1. Start NATS broker
2. Start Predastore (S3)
3. Start Viperblock (EBS)
4. Start Hive Gateway (`hive awsgw`)
5. Start Hive Daemon(s) (`hive daemon`)
### Testing with AWS CLI
```bash
# Set endpoint
export AWS_ENDPOINT_URL=https://localhost:9999
# Run instance
aws --no-verify-ssl ec2 run-instances \
--image-id ami-debian12 \
--instance-type t3.micro
# List instances (queries all nodes)
aws --no-verify-ssl ec2 describe-instances
# Terminate instance
aws --no-verify-ssl ec2 terminate-instances --instance-ids i-xxxxx
```
+116 -40
View File
@@ -1,66 +1,142 @@
module github.com/mulgadc/hive
go 1.25
go 1.26.1
require (
github.com/aws/aws-sdk-go v1.55.8
github.com/davecgh/go-spew v1.1.1
github.com/gofiber/fiber/v2 v2.52.9
github.com/go-chi/chi/v5 v5.2.5
github.com/google/uuid v1.6.0
github.com/kdomanski/iso9660 v0.4.0
github.com/mulgadc/predastore v0.1.5
github.com/mulgadc/viperblock v0.1.1
github.com/nats-io/nats-server/v2 v2.11.6
github.com/nats-io/nats.go v1.43.0
github.com/pterm/pterm v0.12.82
github.com/spf13/cobra v1.9.1
github.com/spf13/viper v1.20.1
github.com/stretchr/testify v1.10.0
github.com/klauspost/cpuid/v2 v2.3.0
github.com/mulgadc/predastore v1.0.3
github.com/mulgadc/viperblock v1.0.17
github.com/nats-io/nats-server/v2 v2.12.4
github.com/nats-io/nats.go v1.49.0
github.com/ovn-kubernetes/libovsdb v0.8.1
github.com/pelletier/go-toml/v2 v2.2.4
github.com/pterm/pterm v0.12.83
github.com/spf13/cobra v1.10.2
github.com/spf13/viper v1.21.0
github.com/stretchr/testify v1.11.1
go.uber.org/automaxprocs v1.6.0
gopkg.in/ini.v1 v1.67.0
golang.org/x/net v0.51.0
gopkg.in/ini.v1 v1.67.1
)
require (
atomicgo.dev/cursor v0.2.0 // indirect
atomicgo.dev/keyboard v0.2.9 // indirect
atomicgo.dev/schedule v0.1.0 // indirect
github.com/andybalholm/brotli v1.1.1 // indirect
cloud.google.com/go v0.123.0 // indirect
cloud.google.com/go/auth v0.18.2 // indirect
cloud.google.com/go/compute/metadata v0.9.0 // indirect
github.com/BurntSushi/toml v1.6.0 // indirect
github.com/anthropics/anthropic-sdk-go v1.26.0 // indirect
github.com/antithesishq/antithesis-sdk-go v0.6.0 // indirect
github.com/armon/go-metrics v0.4.1 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/boltdb/bolt v1.3.1 // indirect
github.com/buraksezer/consistent v0.10.0 // indirect
github.com/ccojocar/zxcvbn-go v1.0.4 // indirect
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
github.com/cenkalti/hub v1.0.2 // indirect
github.com/cenkalti/rpc2 v1.0.5 // indirect
github.com/cespare/xxhash v1.1.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/clipperhouse/uax29/v2 v2.7.0 // indirect
github.com/containerd/console v1.0.5 // indirect
github.com/fsnotify/fsnotify v1.8.0 // indirect
github.com/go-viper/mapstructure/v2 v2.2.1 // indirect
github.com/google/go-tpm v0.9.5 // indirect
github.com/gookit/color v1.5.4 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dgraph-io/badger/v4 v4.9.1 // indirect
github.com/dgraph-io/ristretto/v2 v2.4.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/fatih/color v1.18.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/fsnotify/fsnotify v1.9.0 // indirect
github.com/gabriel-vasile/mimetype v1.4.13 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-playground/validator/v10 v10.30.1 // indirect
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
github.com/google/flatbuffers v25.12.19+incompatible // indirect
github.com/google/go-cmp v0.7.0 // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/google/s2a-go v0.1.9 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.13 // indirect
github.com/googleapis/gax-go/v2 v2.17.0 // indirect
github.com/gookit/color v1.6.0 // indirect
github.com/gorilla/websocket v1.5.3 // indirect
github.com/hashicorp/go-hclog v1.6.3 // indirect
github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
github.com/hashicorp/go-metrics v0.5.4 // indirect
github.com/hashicorp/go-msgpack/v2 v2.1.5 // indirect
github.com/hashicorp/golang-lru v1.0.2 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/hashicorp/raft v1.7.3 // indirect
github.com/hashicorp/raft-boltdb/v2 v2.3.1 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/klauspost/compress v1.18.0 // indirect
github.com/klauspost/compress v1.18.4 // indirect
github.com/klauspost/reedsolomon v1.13.2 // indirect
github.com/leodido/go-urn v1.4.0 // indirect
github.com/lithammer/fuzzysearch v1.1.8 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.16 // indirect
github.com/minio/highwayhash v1.0.3 // indirect
github.com/nats-io/jwt/v2 v2.7.4 // indirect
github.com/nats-io/nkeys v0.4.11 // indirect
github.com/mattn/go-runewidth v0.0.20 // indirect
github.com/minio/crc64nvme v1.1.1 // indirect
github.com/minio/highwayhash v1.0.4-0.20251030100505-070ab1a87a76 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/nats-io/jwt/v2 v2.8.0 // indirect
github.com/nats-io/nkeys v0.4.15 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
github.com/openai/openai-go/v3 v3.24.0 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/sagikazarmark/locafero v0.7.0 // indirect
github.com/sourcegraph/conc v0.3.0 // indirect
github.com/spf13/afero v1.12.0 // indirect
github.com/spf13/cast v1.7.1 // indirect
github.com/spf13/pflag v1.0.6 // indirect
github.com/prometheus/client_golang v1.23.2 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.67.5 // indirect
github.com/prometheus/procfs v0.20.1 // indirect
github.com/quic-go/quic-go v0.59.0 // indirect
github.com/sagikazarmark/locafero v0.12.0 // indirect
github.com/securego/gosec/v2 v2.24.7 // indirect
github.com/spf13/afero v1.15.0 // indirect
github.com/spf13/cast v1.10.0 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
github.com/valyala/bytebufferpool v1.0.0 // indirect
github.com/valyala/fasthttp v1.57.0 // indirect
github.com/valyala/tcplisten v1.0.0 // indirect
github.com/tidwall/gjson v1.18.0 // indirect
github.com/tidwall/match v1.2.0 // indirect
github.com/tidwall/pretty v1.2.1 // indirect
github.com/tidwall/sjson v1.2.5 // indirect
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
go.uber.org/atomic v1.9.0 // indirect
go.uber.org/multierr v1.9.0 // indirect
golang.org/x/crypto v0.39.0 // indirect
golang.org/x/sys v0.33.0 // indirect
golang.org/x/term v0.32.0 // indirect
golang.org/x/text v0.26.0 // indirect
golang.org/x/time v0.12.0 // indirect
go.etcd.io/bbolt v1.4.3 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.66.0 // indirect
go.opentelemetry.io/otel v1.41.0 // indirect
go.opentelemetry.io/otel/metric v1.41.0 // indirect
go.opentelemetry.io/otel/trace v1.41.0 // indirect
go.yaml.in/yaml/v2 v2.4.3 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/crypto v0.48.0 // indirect
golang.org/x/exp/typeparams v0.0.0-20260218203240-3dfff04db8fa // indirect
golang.org/x/mod v0.33.0 // indirect
golang.org/x/sync v0.19.0 // indirect
golang.org/x/sys v0.41.0 // indirect
golang.org/x/telemetry v0.0.0-20260304144227-18da59047661 // indirect
golang.org/x/term v0.40.0 // indirect
golang.org/x/text v0.34.0 // indirect
golang.org/x/time v0.14.0 // indirect
golang.org/x/tools v0.42.0 // indirect
golang.org/x/vuln v1.1.4 // indirect
google.golang.org/genai v1.49.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260226221140-a57be14db171 // indirect
google.golang.org/grpc v1.79.1 // indirect
google.golang.org/protobuf v1.36.11 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
honnef.co/go/tools v0.7.0 // indirect
)
tool (
github.com/securego/gosec/v2/cmd/gosec
golang.org/x/vuln/cmd/govulncheck
honnef.co/go/tools/cmd/staticcheck
)
+455 -90
View File
@@ -6,6 +6,16 @@ atomicgo.dev/keyboard v0.2.9 h1:tOsIid3nlPLZ3lwgG8KZMp/SFmr7P0ssEN5JUsm78K8=
atomicgo.dev/keyboard v0.2.9/go.mod h1:BC4w9g00XkxH/f1HXhW2sXmJFOCWbKn9xrOunSFtExQ=
atomicgo.dev/schedule v0.1.0 h1:nTthAbhZS5YZmgYbb2+DH8uQIZcTlIrd4eYr3UQxEjs=
atomicgo.dev/schedule v0.1.0/go.mod h1:xeUa3oAkiuHYh8bKiQBRojqAMq3PXXbJujjb0hw8pEU=
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.123.0 h1:2NAUJwPR47q+E35uaJeYoNhuNEM9kM8SjgRgdeOJUSE=
cloud.google.com/go v0.123.0/go.mod h1:xBoMV08QcqUGuPW65Qfm1o9Y4zKZBpGS+7bImXLTAZU=
cloud.google.com/go/auth v0.18.2 h1:+Nbt5Ev0xEqxlNjd6c+yYUeosQ5TtEUaNcN/3FozlaM=
cloud.google.com/go/auth v0.18.2/go.mod h1:xD+oY7gcahcu7G2SG2DsBerfFxgPAJz17zz2joOFF3M=
cloud.google.com/go/compute/metadata v0.9.0 h1:pDUj4QMoPejqq20dK0Pg2N4yG9zIkYGdBtwLoEkH9Zs=
cloud.google.com/go/compute/metadata v0.9.0/go.mod h1:E0bWwX5wTnLPedCKqk3pJmVgCBSM6qQI1yTBdEb3C10=
github.com/BurntSushi/toml v1.6.0 h1:dRaEfpa2VI55EwlIW72hMRHdWouJeRF7TPYhI+AUQjk=
github.com/BurntSushi/toml v1.6.0/go.mod h1:ukJfTF/6rtPPRCnwkur4qwRxa8vTRFBF0uk2lLoLwho=
github.com/DataDog/datadog-go v3.2.0+incompatible/go.mod h1:LButxg5PwREeZtORoXG3tL4fMGNddJ+vMq1mwgfaqoQ=
github.com/MarvinJWendt/testza v0.1.0/go.mod h1:7AxNvlfeHP7Z/hDQ5JtE3OKYT3XFUeLCDE2DQninSqs=
github.com/MarvinJWendt/testza v0.2.1/go.mod h1:God7bhG8n6uQxwdScay+gjm9/LnO4D3kkcZX4hv9Rp8=
github.com/MarvinJWendt/testza v0.2.8/go.mod h1:nwIcjmr0Zz+Rcwfh3/4UhBp7ePKVhuBExvZqnKYWlII=
@@ -15,13 +25,87 @@ github.com/MarvinJWendt/testza v0.3.0/go.mod h1:eFcL4I0idjtIx8P9C6KkAuLgATNKpX4/
github.com/MarvinJWendt/testza v0.4.2/go.mod h1:mSdhXiKH8sg/gQehJ63bINcCKp7RtYewEjXsvsVUPbE=
github.com/MarvinJWendt/testza v0.5.2 h1:53KDo64C1z/h/d/stCYCPY69bt/OSwjq5KpFNwi+zB4=
github.com/MarvinJWendt/testza v0.5.2/go.mod h1:xu53QFE5sCdjtMCKk8YMQ2MnymimEctc4n3EjyIYvEY=
github.com/andybalholm/brotli v1.1.1 h1:PR2pgnyFznKEugtsUo0xLdDop5SKXd5Qf5ysW+7XdTA=
github.com/andybalholm/brotli v1.1.1/go.mod h1:05ib4cKhjx3OQYUY22hTVd34Bc8upXjOLL2rKwwZBoA=
github.com/antithesishq/antithesis-sdk-go v0.4.3-default-no-op h1:+OSa/t11TFhqfrX0EOSqQBDJ0YlpmK0rDSiB19dg9M0=
github.com/antithesishq/antithesis-sdk-go v0.4.3-default-no-op/go.mod h1:IUpT2DPAKh6i/YhSbt6Gl3v2yvUZjmKncl7U91fup7E=
github.com/Masterminds/semver/v3 v3.4.0 h1:Zog+i5UMtVoCU8oKka5P7i9q9HgrJeGzI9SA1Xbatp0=
github.com/Masterminds/semver/v3 v3.4.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM=
github.com/OneOfOne/xxhash v1.2.2 h1:KMrpdQIwFcEqXDklaen+P1axHaj9BSKzvpUUfnHldSE=
github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d/go.mod h1:rBZYJk541a8SKzHPHnH3zbiI+7dagKZ0cgpgrD7Fyho=
github.com/anthropics/anthropic-sdk-go v1.26.0 h1:oUTzFaUpAevfuELAP1sjL6CQJ9HHAfT7CoSYSac11PY=
github.com/anthropics/anthropic-sdk-go v1.26.0/go.mod h1:qUKmaW+uuPB64iy1l+4kOSvaLqPXnHTTBKH6RVZ7q5Q=
github.com/antithesishq/antithesis-sdk-go v0.6.0 h1:v/YViLhFYkZOEEof4AXjD5AgGnGM84YHF4RqEwp6I2g=
github.com/antithesishq/antithesis-sdk-go v0.6.0/go.mod h1:IUpT2DPAKh6i/YhSbt6Gl3v2yvUZjmKncl7U91fup7E=
github.com/armon/go-metrics v0.4.1 h1:hR91U9KYmb6bLBYLQjyM+3j+rcd/UhE+G78SFnF8gJA=
github.com/armon/go-metrics v0.4.1/go.mod h1:E6amYzXo6aW1tqzoZGT755KkbgrJsSdpwZ+3JqfkOG4=
github.com/atomicgo/cursor v0.0.1/go.mod h1:cBON2QmmrysudxNBFthvMtN32r3jxVRIvzkUiF/RuIk=
github.com/aws/aws-sdk-go v1.55.8 h1:JRmEUbU52aJQZ2AjX4q4Wu7t4uZjOu71uyNmaWlUkJQ=
github.com/aws/aws-sdk-go v1.55.8/go.mod h1:ZkViS9AqA6otK+JBBNH2++sx1sgxrPKcSzPPvQkUtXk=
github.com/aws/aws-sdk-go-v2 v1.40.0 h1:/WMUA0kjhZExjOQN2z3oLALDREea1A7TobfuiBrKlwc=
github.com/aws/aws-sdk-go-v2 v1.40.0/go.mod h1:c9pm7VwuW0UPxAEYGyTmyurVcNrbF6Rt/wixFqDhcjE=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.3 h1:DHctwEM8P8iTXFxC/QK0MRjwEpWQeM9yzidCRjldUz0=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.3/go.mod h1:xdCzcZEtnSTKVDOmUZs4l/j3pSV6rpo1WXl5ugNsL8Y=
github.com/aws/aws-sdk-go-v2/config v1.32.2 h1:4liUsdEpUUPZs5WVapsJLx5NPmQhQdez7nYFcovrytk=
github.com/aws/aws-sdk-go-v2/config v1.32.2/go.mod h1:l0hs06IFz1eCT+jTacU/qZtC33nvcnLADAPL/XyrkZI=
github.com/aws/aws-sdk-go-v2/credentials v1.19.2 h1:qZry8VUyTK4VIo5aEdUcBjPZHL2v4FyQ3QEOaWcFLu4=
github.com/aws/aws-sdk-go-v2/credentials v1.19.2/go.mod h1:YUqm5a1/kBnoK+/NY5WEiMocZihKSo15/tJdmdXnM5g=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.14 h1:WZVR5DbDgxzA0BJeudId89Kmgy6DIU4ORpxwsVHz0qA=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.14/go.mod h1:Dadl9QO0kHgbrH1GRqGiZdYtW5w+IXXaBNCHTIaheM4=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.14 h1:PZHqQACxYb8mYgms4RZbhZG0a7dPW06xOjmaH0EJC/I=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.14/go.mod h1:VymhrMJUWs69D8u0/lZ7jSB6WgaG/NqHi3gX0aYf6U0=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.14 h1:bOS19y6zlJwagBfHxs0ESzr1XCOU2KXJCWcq3E2vfjY=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.14/go.mod h1:1ipeGBMAxZ0xcTm6y6paC2C/J6f6OO7LBODV9afuAyM=
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 h1:WKuaxf++XKWlHWu9ECbMlha8WOEGm0OUEZqm4K/Gcfk=
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4/go.mod h1:ZWy7j6v1vWGmPReu0iSGvRiise4YI5SkR3OHKTZ6Wuc=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.14 h1:ITi7qiDSv/mSGDSWNpZ4k4Ve0DQR6Ug2SJQ8zEHoDXg=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.14/go.mod h1:k1xtME53H1b6YpZt74YmwlONMWf4ecM+lut1WQLAF/U=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.3 h1:x2Ibm/Af8Fi+BH+Hsn9TXGdT+hKbDd5XOTZxTMxDk7o=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.3/go.mod h1:IW1jwyrQgMdhisceG8fQLmQIydcT/jWY21rFhzgaKwo=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.5 h1:Hjkh7kE6D81PgrHlE/m9gx+4TyyeLHuY8xJs7yXN5C4=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.5/go.mod h1:nPRXgyCfAurhyaTMoBMwRBYBhaHI4lNPAnJmjM0Tslc=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.14 h1:FIouAnCE46kyYqyhs0XEBDFFSREtdnr8HQuLPQPLCrY=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.14/go.mod h1:UTwDc5COa5+guonQU8qBikJo1ZJ4ln2r1MkF7Dqag1E=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.14 h1:FzQE21lNtUor0Fb7QNgnEyiRCBlolLTX/Z1j65S7teM=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.14/go.mod h1:s1ydyWG9pm3ZwmmYN21HKyG9WzAZhYVW85wMHs5FV6w=
github.com/aws/aws-sdk-go-v2/service/s3 v1.92.1 h1:OgQy/+0+Kc3khtqiEOk23xQAglXi3Tj0y5doOxbi5tg=
github.com/aws/aws-sdk-go-v2/service/s3 v1.92.1/go.mod h1:wYNqY3L02Z3IgRYxOBPH9I1zD9Cjh9hI5QOy/eOjQvw=
github.com/aws/aws-sdk-go-v2/service/signin v1.0.2 h1:MxMBdKTYBjPQChlJhi4qlEueqB1p1KcbTEa7tD5aqPs=
github.com/aws/aws-sdk-go-v2/service/signin v1.0.2/go.mod h1:iS6EPmNeqCsGo+xQmXv0jIMjyYtQfnwg36zl2FwEouk=
github.com/aws/aws-sdk-go-v2/service/sso v1.30.5 h1:ksUT5KtgpZd3SAiFJNJ0AFEJVva3gjBmN7eXUZjzUwQ=
github.com/aws/aws-sdk-go-v2/service/sso v1.30.5/go.mod h1:av+ArJpoYf3pgyrj6tcehSFW+y9/QvAY8kMooR9bZCw=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.10 h1:GtsxyiF3Nd3JahRBJbxLCCdYW9ltGQYrFWg8XdkGDd8=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.10/go.mod h1:/j67Z5XBVDx8nZVp9EuFM9/BS5dvBznbqILGuu73hug=
github.com/aws/aws-sdk-go-v2/service/sts v1.41.2 h1:a5UTtD4mHBU3t0o6aHQZFJTNKVfxFWfPX7J0Lr7G+uY=
github.com/aws/aws-sdk-go-v2/service/sts v1.41.2/go.mod h1:6TxbXoDSgBQ225Qd8Q+MbxUxUh6TtNKwbRt/EPS9xso=
github.com/aws/smithy-go v1.23.2 h1:Crv0eatJUQhaManss33hS5r40CG3ZFH+21XSkqMrIUM=
github.com/aws/smithy-go v1.23.2/go.mod h1:LEj2LM3rBRQJxPZTB4KuzZkaZYnZPnvgIhb4pu07mx0=
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/boltdb/bolt v1.3.1 h1:JQmyP4ZBrce+ZQu0dY660FMfatumYDLun9hBCUVIkF4=
github.com/boltdb/bolt v1.3.1/go.mod h1:clJnj/oiGkjum5o1McbSZDSLxVThjynRyGBgiAx27Ps=
github.com/buraksezer/consistent v0.10.0 h1:hqBgz1PvNLC5rkWcEBVAL9dFMBWz6I0VgUCW25rrZlU=
github.com/buraksezer/consistent v0.10.0/go.mod h1:6BrVajWq7wbKZlTOUPs/XVfR8c0maujuPowduSpZqmw=
github.com/ccojocar/zxcvbn-go v1.0.4 h1:FWnCIRMXPj43ukfX000kvBZvV6raSxakYr1nzyNrUcc=
github.com/ccojocar/zxcvbn-go v1.0.4/go.mod h1:3GxGX+rHmueTUMvm5ium7irpyjmm7ikxYFOSJB21Das=
github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8=
github.com/cenkalti/backoff/v4 v4.3.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE=
github.com/cenkalti/hub v1.0.2 h1:Nqv9TNaA9boeO2wQFW8o87BY3zKthtnzXmWGmJqhAV8=
github.com/cenkalti/hub v1.0.2/go.mod h1:8LAFAZcCasb83vfxatMUnZHRoQcffho2ELpHb+kaTJU=
github.com/cenkalti/rpc2 v1.0.5 h1:T6l4SS3ja3eaJfRyZrn7Oco/PSx/pr3YK5cjCgLVLTk=
github.com/cenkalti/rpc2 v1.0.5/go.mod h1:2yfU5b86vOr16+iY1jN3MvT6Kxc9Nf8j5iZWwUf7iaw=
github.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/circonus-labs/circonus-gometrics v2.3.1+incompatible/go.mod h1:nmEj6Dob7S7YxXgwXpfOuvO54S+tGdZdw9fuRZt25Ag=
github.com/circonus-labs/circonusllhist v0.1.3/go.mod h1:kMXHVDlOchFAehlya5ePtbp5jckzBHf4XRpQvBOLI+I=
github.com/clipperhouse/uax29/v2 v2.7.0 h1:+gs4oBZ2gPfVrKPthwbMzWZDaAFPGYK72F0NJv2v7Vk=
github.com/clipperhouse/uax29/v2 v2.7.0/go.mod h1:EFJ2TJMRUaplDxHKj1qAEhCtQPW2tJSwu5BF98AuoVM=
github.com/containerd/console v1.0.3/go.mod h1:7LqA/THxQ86k76b8c/EMSiaJ3h1eZkMkXar0TQ1gf3U=
github.com/containerd/console v1.0.5 h1:R0ymNeydRqH2DmakFNdmjR2k0t7UPuiOV/N/27/qqsc=
github.com/containerd/console v1.0.5/go.mod h1:YynlIjWYF8myEu6sdkwKIvGQq+cOckRm6So2avqoYAk=
@@ -29,41 +113,153 @@ github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6N
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dgraph-io/badger/v4 v4.9.1 h1:DocZXZkg5JJHJPtUErA0ibyHxOVUDVoXLSCV6t8NC8w=
github.com/dgraph-io/badger/v4 v4.9.1/go.mod h1:5/MEx97uzdPUHR4KtkNt8asfI2T4JiEiQlV7kWUo8c0=
github.com/dgraph-io/ristretto/v2 v2.4.0 h1:I/w09yLjhdcVD2QV192UJcq8dPBaAJb9pOuMyNy0XlU=
github.com/dgraph-io/ristretto/v2 v2.4.0/go.mod h1:0KsrXtXvnv0EqnzyowllbVJB8yBonswa2lTCK2gGo9E=
github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da h1:aIftn67I1fkbMa512G+w+Pxci9hJPB8oMnkcP3iZF38=
github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw=
github.com/dnaeon/go-vcr v1.2.0 h1:zHCHvJYTMh1N7xnV7zf1m1GPBF9Ad0Jk/whtQ1663qI=
github.com/dnaeon/go-vcr v1.2.0/go.mod h1:R4UdLID7HZT3taECzJs4YgbbH6PIGXB6W/sc5OLb6RQ=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/fatih/color v1.13.0/go.mod h1:kLAiJbzzSOZDVNGyDpeOxJ47H46qBXwg5ILebYFFOfk=
github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM=
github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU=
github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/fsnotify/fsnotify v1.8.0 h1:dAwr6QBTBZIkG8roQaJjGof0pp0EeF+tNV7YBP3F/8M=
github.com/fsnotify/fsnotify v1.8.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
github.com/go-viper/mapstructure/v2 v2.2.1 h1:ZAaOCxANMuZx5RCeg0mBdEZk7DZasvvZIxtHqx8aGss=
github.com/go-viper/mapstructure/v2 v2.2.1/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/gofiber/fiber/v2 v2.52.9 h1:YjKl5DOiyP3j0mO61u3NTmK7or8GzzWzCFzkboyP5cw=
github.com/gofiber/fiber/v2 v2.52.9/go.mod h1:YEcBbO/FB+5M1IZNBP9FO3J9281zgPAreiI1oqg8nDw=
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/go-tpm v0.9.5 h1:ocUmnDebX54dnW+MQWGQRbdaAcJELsa6PqZhJ48KwVU=
github.com/google/go-tpm v0.9.5/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
github.com/gabriel-vasile/mimetype v1.4.13 h1:46nXokslUBsAJE/wMsp5gtO500a4F3Nkz9Ufpk2AcUM=
github.com/gabriel-vasile/mimetype v1.4.13/go.mod h1:d+9Oxyo1wTzWdyVUPMmXFvp4F9tea18J8ufA774AB3s=
github.com/go-chi/chi/v5 v5.2.5 h1:Eg4myHZBjyvJmAFjFvWgrqDTXFyOzjj7YIm3L3mu6Ug=
github.com/go-chi/chi/v5 v5.2.5/go.mod h1:X7Gx4mteadT3eDOMTsXzmI4/rwUpOwBHLpAfupzFJP0=
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-kit/log v0.1.0/go.mod h1:zbhenjAZHb184qTLMA9ZjW7ThYL0H2mk7Q6pNt4vbaY=
github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/go-playground/assert/v2 v2.2.0 h1:JvknZsQTYeFEAhQwI4qEt9cyV5ONwRHC+lYKSsYSR8s=
github.com/go-playground/assert/v2 v2.2.0/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
github.com/go-playground/locales v0.14.1 h1:EWaQ/wswjilfKLTECiXz7Rh+3BjFhfDFKv/oXslEjJA=
github.com/go-playground/locales v0.14.1/go.mod h1:hxrqLVvrK65+Rwrd5Fc6F2O76J/NuW9t0sjnWqG1slY=
github.com/go-playground/universal-translator v0.18.1 h1:Bcnm0ZwsGyWbCzImXv+pAJnYK9S473LQFuzCbDbfSFY=
github.com/go-playground/universal-translator v0.18.1/go.mod h1:xekY+UJKNuX9WP91TpwSH2VMlDf28Uj24BCp08ZFTUY=
github.com/go-playground/validator/v10 v10.30.1 h1:f3zDSN/zOma+w6+1Wswgd9fLkdwy06ntQJp0BBvFG0w=
github.com/go-playground/validator/v10 v10.30.1/go.mod h1:oSuBIQzuJxL//3MelwSLD5hc2Tu889bF0Idm9Dg26cM=
github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI=
github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8=
github.com/go-viper/mapstructure/v2 v2.5.0 h1:vM5IJoUAy3d7zRSVtIwQgBj7BiWtMPfmPEgAXnvj1Ro=
github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs=
github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
github.com/google/flatbuffers v25.12.19+incompatible h1:haMV2JRRJCe1998HeW/p0X9UaMTK6SDo0ffLn2+DbLs=
github.com/google/flatbuffers v25.12.19+incompatible/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
github.com/google/go-cmdtest v0.4.1-0.20220921163831-55ab3332a786 h1:rcv+Ippz6RAtvaGgKxc+8FQIpxHgsF+HBzPyYL2cyVU=
github.com/google/go-cmdtest v0.4.1-0.20220921163831-55ab3332a786/go.mod h1:apVn/GCasLZUVpAJ6oWAuyP7Ne7CEsQbTnc0plM3m+o=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/pprof v0.0.0-20260115054156-294ebfa9ad83 h1:z2ogiKUYzX5Is6zr/vP9vJGqPwcdqsWjOt+V8J7+bTc=
github.com/google/pprof v0.0.0-20260115054156-294ebfa9ad83/go.mod h1:MxpfABSjhmINe3F1It9d+8exIHFvUqtLIRCdOGNXqiI=
github.com/google/renameio v0.1.0 h1:GOZbcHa3HfsPKPlmyPyN2KEohoMXOhdMbHrvbpl2QaA=
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
github.com/google/s2a-go v0.1.9 h1:LGD7gtMgezd8a/Xak7mEWL0PjoTQFvpRudN895yqKW0=
github.com/google/s2a-go v0.1.9/go.mod h1:YA0Ei2ZQL3acow2O62kdp9UlnvMmU7kA6Eutn0dXayM=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/googleapis/enterprise-certificate-proxy v0.3.13 h1:hSPAhW3NX+7HNlTsmrvU0jL75cIzxFktheceg95Nq14=
github.com/googleapis/enterprise-certificate-proxy v0.3.13/go.mod h1:vqVt9yG9480NtzREnTlmGSBmFrA+bzb0yl0TxoBQXOg=
github.com/googleapis/gax-go/v2 v2.17.0 h1:RksgfBpxqff0EZkDWYuz9q/uWsTVz+kf43LsZ1J6SMc=
github.com/googleapis/gax-go/v2 v2.17.0/go.mod h1:mzaqghpQp4JDh3HvADwrat+6M3MOIDp5YKHhb9PAgDY=
github.com/gookit/assert v0.1.1 h1:lh3GcawXe/p+cU7ESTZ5Ui3Sm/x8JWpIis4/1aF0mY0=
github.com/gookit/assert v0.1.1/go.mod h1:jS5bmIVQZTIwk42uXl4lyj4iaaxx32tqH16CFj0VX2E=
github.com/gookit/color v1.4.2/go.mod h1:fqRyamkC1W8uxl+lxCQxOT09l/vYfZ+QeiX3rKQHCoQ=
github.com/gookit/color v1.5.0/go.mod h1:43aQb+Zerm/BWh2GnrgOQm7ffz7tvQXEKV6BFMl7wAo=
github.com/gookit/color v1.5.4 h1:FZmqs7XOyGgCAxmWyPslpiok1k05wmY3SJTytgvYFs0=
github.com/gookit/color v1.5.4/go.mod h1:pZJOeOS8DM43rXbp4AZo1n9zCU2qjpcRko0b6/QJi9w=
github.com/gookit/color v1.6.0 h1:JjJXBTk1ETNyqyilJhkTXJYYigHG24TM9Xa2M1xAhRA=
github.com/gookit/color v1.6.0/go.mod h1:9ACFc7/1IpHGBW8RwuDm/0YEnhg3dwwXpoMsmtyHfjs=
github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/hashicorp/go-cleanhttp v0.5.0/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80=
github.com/hashicorp/go-hclog v1.6.3 h1:Qr2kF+eVWjTiYmU7Y31tYlP1h0q/X3Nl3tPGdaB11/k=
github.com/hashicorp/go-hclog v1.6.3/go.mod h1:W4Qnvbt70Wk/zYJryRzDRU/4r0kIg0PVHBcfoyhpF5M=
github.com/hashicorp/go-immutable-radix v1.0.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60=
github.com/hashicorp/go-immutable-radix v1.3.1 h1:DKHmCUm2hRBK510BaiZlwvpD40f8bJFeZnpfm2KLowc=
github.com/hashicorp/go-immutable-radix v1.3.1/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60=
github.com/hashicorp/go-metrics v0.5.4 h1:8mmPiIJkTPPEbAiV97IxdAGNdRdaWwVap1BU6elejKY=
github.com/hashicorp/go-metrics v0.5.4/go.mod h1:CG5yz4NZ/AI/aQt9Ucm/vdBnbh7fvmv4lxZ350i+QQI=
github.com/hashicorp/go-msgpack v0.5.5 h1:i9R9JSrqIz0QVLz3sz+i3YJdT7TTSLcfLLzJi9aZTuI=
github.com/hashicorp/go-msgpack v0.5.5/go.mod h1:ahLV/dePpqEmjfWmKiqvPkv/twdG7iPBM1vqhUKIvfM=
github.com/hashicorp/go-msgpack/v2 v2.1.5 h1:Ue879bPnutj/hXfmUk6s/jtIK90XxgiUIcXRl656T44=
github.com/hashicorp/go-msgpack/v2 v2.1.5/go.mod h1:bjCsRXpZ7NsJdk45PoCQnzRGDaK8TKm5ZnDI/9y3J4M=
github.com/hashicorp/go-retryablehttp v0.5.3/go.mod h1:9B5zBasrRhHXnJnui7y6sL7es7NDiJgTc6Er0maI1Xs=
github.com/hashicorp/go-uuid v1.0.0 h1:RS8zrF7PhGwyNPOtxSClXXj9HA8feRnJzgnI1RJCSnM=
github.com/hashicorp/go-uuid v1.0.0/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v1.0.2 h1:dV3g9Z/unq5DpblPpw+Oqcv4dU/1omnb4Ok8iPY6p1c=
github.com/hashicorp/golang-lru v1.0.2/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/hashicorp/raft v1.7.3 h1:DxpEqZJysHN0wK+fviai5mFcSYsCkNpFUl1xpAW8Rbo=
github.com/hashicorp/raft v1.7.3/go.mod h1:DfvCGFxpAUPE0L4Uc8JLlTPtc3GzSbdH0MTJCLgnmJQ=
github.com/hashicorp/raft-boltdb v0.0.0-20230125174641-2a8082862702 h1:RLKEcCuKcZ+qp2VlaaZsYZfLOmIiuJNpEi48Rl8u9cQ=
github.com/hashicorp/raft-boltdb v0.0.0-20230125174641-2a8082862702/go.mod h1:nTakvJ4XYq45UXtn0DbwR4aU9ZdjlnIenpbs6Cd+FM0=
github.com/hashicorp/raft-boltdb/v2 v2.3.1 h1:ackhdCNPKblmOhjEU9+4lHSJYFkJd6Jqyvj6eW9pwkc=
github.com/hashicorp/raft-boltdb/v2 v2.3.1/go.mod h1:n4S+g43dXF1tqDT+yzcXHhXM6y7MrlUd3TTwGRcUvQE=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo=
github.com/jmespath/go-jmespath/internal/testify v1.5.1 h1:shLQSRRSCCPj3f2gpwzGwWFoC7ycTf1rcQZHOlsJ6N8=
github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U=
github.com/jpillora/backoff v1.0.0/go.mod h1:J/6gKK9jxlEcS3zixgDgUAsiuZ7yrSoa/FX5e0EB2j4=
github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
github.com/json-iterator/go v1.1.9/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/json-iterator/go v1.1.11/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
github.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM=
github.com/kdomanski/iso9660 v0.4.0 h1:BPKKdcINz3m0MdjIMwS0wx1nofsOjxOq8TOr45WGHFg=
github.com/kdomanski/iso9660 v0.4.0/go.mod h1:OxUSupHsO9ceI8lBLPJKWBTphLemjrCQY8LPXM7qSzU=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/klauspost/compress v1.18.4 h1:RPhnKRAQ4Fh8zU2FY/6ZFDwTVTxgJ/EMydqSTzE9a2c=
github.com/klauspost/compress v1.18.4/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/klauspost/cpuid/v2 v2.0.10/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuObKfj5c0PQa7c=
github.com/klauspost/cpuid/v2 v2.0.12/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuObKfj5c0PQa7c=
github.com/klauspost/cpuid/v2 v2.2.3 h1:sxCkb+qR91z4vsqw4vGGZlDgPz3G7gjaLyK3V8y70BU=
github.com/klauspost/cpuid/v2 v2.2.3/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY=
github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y=
github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
github.com/klauspost/reedsolomon v1.13.2 h1:9qtQy2tKEVpVB8Pfq87ZljHZb60/LbeTQ1OxV8EGzdE=
github.com/klauspost/reedsolomon v1.13.2/go.mod h1:ggJT9lc71Vu+cSOPBlxGvBN6TfAS77qB4fp8vJ05NSA=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
@@ -71,38 +267,92 @@ github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/leodido/go-urn v1.4.0 h1:WT9HwE9SGECu3lg4d/dIA+jxlljEa1/ffXKmRjqdmIQ=
github.com/leodido/go-urn v1.4.0/go.mod h1:bvxc+MVxLKB4z00jd1z+Dvzr47oO32F/QSNjSBOlFxI=
github.com/lithammer/fuzzysearch v1.1.8 h1:/HIuJnjHuXS8bKaiTMeeDlW2/AyIWk2brx1V8LFgLN4=
github.com/lithammer/fuzzysearch v1.1.8/go.mod h1:IdqeyBClc3FFqSzYq/MXESsS4S0FsZ5ajtkr5xPLts4=
github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
github.com/mattn/go-colorable v0.1.9/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
github.com/mattn/go-colorable v0.1.12/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4=
github.com/mattn/go-colorable v0.1.14 h1:9A9LHSqF/7dyVVX6g0U9cwm9pG3kP9gSzcuIPHPsaIE=
github.com/mattn/go-colorable v0.1.14/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8=
github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU=
github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/mattn/go-runewidth v0.0.13/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
github.com/mattn/go-runewidth v0.0.16 h1:E5ScNMtiwvlvB5paMFdw9p4kSQzbXFikJ5SQO6TULQc=
github.com/mattn/go-runewidth v0.0.16/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
github.com/minio/highwayhash v1.0.3 h1:kbnuUMoHYyVl7szWjSxJnxw11k2U709jqFPPmIUyD6Q=
github.com/minio/highwayhash v1.0.3/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/mulgadc/predastore v0.1.5 h1:7SySkQQ5MQ1NTcKcB14pCUIJipq0YVgCJZeKM6ytrLM=
github.com/mulgadc/predastore v0.1.5/go.mod h1:BhMymRV3zMTu2S+5pJnNDDm5+AGakjhSc/PKX0q8Xfg=
github.com/mulgadc/viperblock v0.1.1 h1:R1DVX1S+/xPPMQ4LIFkwIX7Es+RS3Wgdd8bsWLTEvgY=
github.com/mulgadc/viperblock v0.1.1/go.mod h1:zCjdQ76fi2Qs2a5ign9A/Uqgbz77xts5WV0OR1be48I=
github.com/nats-io/jwt/v2 v2.7.4 h1:jXFuDDxs/GQjGDZGhNgH4tXzSUK6WQi2rsj4xmsNOtI=
github.com/nats-io/jwt/v2 v2.7.4/go.mod h1:me11pOkwObtcBNR8AiMrUbtVOUGkqYjMQZ6jnSdVUIA=
github.com/nats-io/nats-server/v2 v2.11.6 h1:4VXRjbTUFKEB+7UoaKL3F5Y83xC7MxPoIONOnGgpkHw=
github.com/nats-io/nats-server/v2 v2.11.6/go.mod h1:2xoztlcb4lDL5Blh1/BiukkKELXvKQ5Vy29FPVRBUYs=
github.com/nats-io/nats.go v1.43.0 h1:uRFZ2FEoRvP64+UUhaTokyS18XBCR/xM2vQZKO4i8ug=
github.com/nats-io/nats.go v1.43.0/go.mod h1:iRWIPokVIFbVijxuMQq4y9ttaBTMe0SFdlZfMDd+33g=
github.com/nats-io/nkeys v0.4.11 h1:q44qGV008kYd9W1b1nEBkNzvnWxtRSQ7A8BoqRrcfa0=
github.com/nats-io/nkeys v0.4.11/go.mod h1:szDimtgmfOi9n25JpfIdGw12tZFYXqhGxjhVxsatHVE=
github.com/mattn/go-runewidth v0.0.20 h1:WcT52H91ZUAwy8+HUkdM3THM6gXqXuLJi9O3rjcQQaQ=
github.com/mattn/go-runewidth v0.0.20/go.mod h1:XBkDxAl56ILZc9knddidhrOlY5R/pDhgLpndooCuJAs=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
github.com/minio/crc64nvme v1.1.1 h1:8dwx/Pz49suywbO+auHCBpCtlW1OfpcLN7wYgVR6wAI=
github.com/minio/crc64nvme v1.1.1/go.mod h1:eVfm2fAzLlxMdUGc0EEBGSMmPwmXD5XiNRpnu9J3bvg=
github.com/minio/highwayhash v1.0.4-0.20251030100505-070ab1a87a76 h1:KGuD/pM2JpL9FAYvBrnBBeENKZNh6eNtjqytV6TYjnk=
github.com/minio/highwayhash v1.0.4-0.20251030100505-070ab1a87a76/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/mulgadc/predastore v1.0.3 h1:qkuOEUH9B98QzVByn5osKeXm7SSPgk+ardQ3MibPSok=
github.com/mulgadc/predastore v1.0.3/go.mod h1:x3Z6Ts/PONpmVx/R6t/hkHDJ3+0xRieYS3BuuVaNgyA=
github.com/mulgadc/viperblock v1.0.17 h1:bGRVlXgcqnmH+Xxon03W/ZsbXqu/EE5+/VG3J4E3EBU=
github.com/mulgadc/viperblock v1.0.17/go.mod h1:w9V0DjUH5hMXbXsiLbDr9lpqzXBGhEiYB1BeV6TYBkw=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
github.com/nats-io/jwt/v2 v2.8.0 h1:K7uzyz50+yGZDO5o772eRE7atlcSEENpL7P+b74JV1g=
github.com/nats-io/jwt/v2 v2.8.0/go.mod h1:me11pOkwObtcBNR8AiMrUbtVOUGkqYjMQZ6jnSdVUIA=
github.com/nats-io/nats-server/v2 v2.12.4 h1:ZnT10v2LU2Xcoiy8ek9X6Se4YG8EuMfIfvAEuFVx1Ts=
github.com/nats-io/nats-server/v2 v2.12.4/go.mod h1:5MCp/pqm5SEfsvVZ31ll1088ZTwEUdvRX1Hmh/mTTDg=
github.com/nats-io/nats.go v1.49.0 h1:yh/WvY59gXqYpgl33ZI+XoVPKyut/IcEaqtsiuTJpoE=
github.com/nats-io/nats.go v1.49.0/go.mod h1:fDCn3mN5cY8HooHwE2ukiLb4p4G4ImmzvXyJt+tGwdw=
github.com/nats-io/nkeys v0.4.15 h1:JACV5jRVO9V856KOapQ7x+EY8Jo3qw1vJt/9Jpwzkk4=
github.com/nats-io/nkeys v0.4.15/go.mod h1:CpMchTXC9fxA5zrMo4KpySxNjiDVvr8ANOSZdiNfUrs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
github.com/onsi/ginkgo/v2 v2.28.1 h1:S4hj+HbZp40fNKuLUQOYLDgZLwNUVn19N3Atb98NCyI=
github.com/onsi/ginkgo/v2 v2.28.1/go.mod h1:CLtbVInNckU3/+gC8LzkGUb9oF+e8W8TdUsxPwvdOgE=
github.com/onsi/gomega v1.39.1 h1:1IJLAad4zjPn2PsnhH70V4DKRFlrCzGBNrNaru+Vf28=
github.com/onsi/gomega v1.39.1/go.mod h1:hL6yVALoTOxeWudERyfppUcZXjMwIMLnuSfruD2lcfg=
github.com/openai/openai-go/v3 v3.24.0 h1:08x6GnYiB+AAejTo6yzPY8RkZMJQ8NpreiOyM5QfyYU=
github.com/openai/openai-go/v3 v3.24.0/go.mod h1:cdufnVK14cWcT9qA1rRtrXx4FTRsgbDPW7Ia7SS5cZo=
github.com/ovn-kubernetes/libovsdb v0.8.1 h1:M2J8bcJt5mXCom0HqzfEtuHkT80CTSQRcYG7acT8gf4=
github.com/ovn-kubernetes/libovsdb v0.8.1/go.mod h1:ZlnHLzagmLOSvyd9qfxBIZp6wOSOw0IsRsc+6lNUGbU=
github.com/pascaldekloe/goe v0.1.0 h1:cBOtyMzM9HTpWjXfbbunk26uA6nG3a8n06Wieeh0MwY=
github.com/pascaldekloe/goe v0.1.0/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0t5Ec4=
github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prashantv/gostub v1.1.0 h1:BTyx3RfQjRHnUWaGF9oQos79AlQ5k8WNktv7VGvVH4g=
github.com/prashantv/gostub v1.1.0/go.mod h1:A5zLQHz7ieHGG7is6LLXLz7I8+3LZzsrV0P1IAHhP5U=
github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo=
github.com/prometheus/client_golang v1.4.0/go.mod h1:e9GMxYsXl05ICDXkRhurwBS4Q3OK1iX/F2sw+iXX5zU=
github.com/prometheus/client_golang v1.7.1/go.mod h1:PY5Wy2awLA44sXw4AOSfFBetzPP4j5+D6mVACh+pe2M=
github.com/prometheus/client_golang v1.11.1/go.mod h1:Z6t4BnS23TR94PD6BsDNk8yVqroYurpAkEiz0P2BEV0=
github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o=
github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg=
github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/client_model v0.2.0/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
github.com/prometheus/common v0.4.1/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
github.com/prometheus/common v0.9.1/go.mod h1:yhUN8i9wzaXS3w1O07YhxHEBxD+W35wd8bs7vj7HSQ4=
github.com/prometheus/common v0.10.0/go.mod h1:Tlit/dnDKsSWFlCLTWaA1cyBgKHSMdTB80sz/V91rCo=
github.com/prometheus/common v0.26.0/go.mod h1:M7rCNAaPfAosfx8veZJCuw84e35h3Cfd9VFqTh1DIvc=
github.com/prometheus/common v0.67.5 h1:pIgK94WWlQt1WLwAC5j2ynLaBRDiinoAb86HZHTUGI4=
github.com/prometheus/common v0.67.5/go.mod h1:SjE/0MzDEEAyrdr5Gqc6G+sXI67maCxzaT3A2+HqjUw=
github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
github.com/prometheus/procfs v0.0.8/go.mod h1:7Qr8sr6344vo1JqZ6HhLceV9o3AJ1Ff+GxbHq6oeK9A=
github.com/prometheus/procfs v0.1.3/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU=
github.com/prometheus/procfs v0.6.0/go.mod h1:cz+aTbrPOrUb4q7XlbU9ygM+/jj0fzG6c1xBZuNvfVA=
github.com/prometheus/procfs v0.20.1 h1:XwbrGOIplXW/AU3YhIhLODXMJYyC1isLFfYCsTEycfc=
github.com/prometheus/procfs v0.20.1/go.mod h1:o9EMBZGRyvDrSPH1RqdxhojkuXstoe4UlK79eF5TGGo=
github.com/pterm/pterm v0.12.27/go.mod h1:PhQ89w4i95rhgE+xedAoqous6K9X+r6aSOI2eFF7DZI=
github.com/pterm/pterm v0.12.29/go.mod h1:WI3qxgvoQFFGKGjGnJR849gU0TsEOvKn5Q8LlY1U7lg=
github.com/pterm/pterm v0.12.30/go.mod h1:MOqLIyMOgmTDz9yorcYbcw+HsgoZo3BQfg2wtl3HEFE=
@@ -110,119 +360,234 @@ github.com/pterm/pterm v0.12.31/go.mod h1:32ZAWZVXD7ZfG0s8qqHXePte42kdz8ECtRyEej
github.com/pterm/pterm v0.12.33/go.mod h1:x+h2uL+n7CP/rel9+bImHD5lF3nM9vJj80k9ybiiTTE=
github.com/pterm/pterm v0.12.36/go.mod h1:NjiL09hFhT/vWjQHSj1athJpx6H8cjpHXNAK5bUw8T8=
github.com/pterm/pterm v0.12.40/go.mod h1:ffwPLwlbXxP+rxT0GsgDTzS3y3rmpAO1NMjUkGTYf8s=
github.com/pterm/pterm v0.12.82 h1:+D9wYhCaeaK0FIQoZtqbNQuNpe2lB2tajKKsTd5paVQ=
github.com/pterm/pterm v0.12.82/go.mod h1:TyuyrPjnxfwP+ccJdBTeWHtd/e0ybQHkOS/TakajZCw=
github.com/pterm/pterm v0.12.83 h1:ie+YmGmA727VuhxBlyGr74Ks+7McV6kT99IB8EU80aA=
github.com/pterm/pterm v0.12.83/go.mod h1:xlgc6bFWyJIMtmLJvGim+L7jhSReilOlOnodeIYe4Tk=
github.com/quic-go/quic-go v0.59.0 h1:OLJkp1Mlm/aS7dpKgTc6cnpynnD2Xg7C1pwL6vy/SAw=
github.com/quic-go/quic-go v0.59.0/go.mod h1:upnsH4Ju1YkqpLXC305eW3yDZ4NfnNbmQRCMWS58IKU=
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/sagikazarmark/locafero v0.7.0 h1:5MqpDsTGNDhY8sGp0Aowyf0qKsPrhewaLSsFaodPcyo=
github.com/sagikazarmark/locafero v0.7.0/go.mod h1:2za3Cg5rMaTMoG/2Ulr9AwtFaIppKXTRYnozin4aB5k=
github.com/sagikazarmark/locafero v0.12.0 h1:/NQhBAkUb4+fH1jivKHWusDYFjMOOKU88eegjfxfHb4=
github.com/sagikazarmark/locafero v0.12.0/go.mod h1:sZh36u/YSZ918v0Io+U9ogLYQJ9tLLBmM4eneO6WwsI=
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU=
github.com/securego/gosec/v2 v2.24.7 h1:3k5yJnrhT1TTdsG0ZsnenlfCcT+7Y/+zeCPHbL7QAn8=
github.com/securego/gosec/v2 v2.24.7/go.mod h1:AdDJbjcG/XxFgVv7pW19vMNYlFM6+Q6Qy3t6lWAUcEY=
github.com/sergi/go-diff v1.2.0 h1:XU+rvMAioB0UC3q1MFrIQy4Vo5/4VsRDQQXHsEya6xQ=
github.com/sergi/go-diff v1.2.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM=
github.com/sourcegraph/conc v0.3.0 h1:OQTbbt6P72L20UqAkXXuLOj79LfEanQ+YQFNpLA9ySo=
github.com/sourcegraph/conc v0.3.0/go.mod h1:Sdozi7LEKbFPqYX2/J+iBAM6HpqSLTASQIKqDmF7Mt0=
github.com/spf13/afero v1.12.0 h1:UcOPyRBYczmFn6yvphxkn9ZEOY65cpwGKb5mL36mrqs=
github.com/spf13/afero v1.12.0/go.mod h1:ZTlWwG4/ahT8W7T0WQ5uYmjI9duaLQGy3Q2OAl4sk/4=
github.com/spf13/cast v1.7.1 h1:cuNEagBQEHWN1FnbGEjCXL2szYEXqfJPbP2HNUaca9Y=
github.com/spf13/cast v1.7.1/go.mod h1:ancEpBxwJDODSW/UG4rDrAqiKolqNNh2DX3mk86cAdo=
github.com/spf13/cobra v1.9.1 h1:CXSaggrXdbHK9CF+8ywj8Amf7PBRmPCOJugH954Nnlo=
github.com/spf13/cobra v1.9.1/go.mod h1:nDyEzZ8ogv936Cinf6g1RU9MRY64Ir93oCnqb9wxYW0=
github.com/spf13/pflag v1.0.6 h1:jFzHGLGAlb3ruxLB8MhbI6A8+AQX/2eW4qeyNZXNp2o=
github.com/spf13/pflag v1.0.6/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.20.1 h1:ZMi+z/lvLyPSCoNtFCpqjy0S4kPbirhpTMwl8BkW9X4=
github.com/spf13/viper v1.20.1/go.mod h1:P9Mdzt1zoHIG8m2eZQinpiBjo6kCmZSKBClNNqjJvu4=
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88=
github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72 h1:qLC7fQah7D6K1B0ujays3HV9gkFtllcxhzImRR7ArPQ=
github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
github.com/spf13/afero v1.15.0 h1:b/YBCLWAJdFWJTN9cLhiXXcD7mzKn9Dm86dNnfyQw1I=
github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg=
github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY=
github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo=
github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU=
github.com/spf13/viper v1.21.0/go.mod h1:P0lhsswPGWD/1lZJ9ny3fYnVqxiegrlNrEmgLjbTCAY=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY=
github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.2/go.mod h1:R6va5+xMeoiuVRoj+gSkQ7d3FALtqAAGI1FQKckRals=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8=
github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU=
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
github.com/valyala/fasthttp v1.57.0 h1:Xw8SjWGEP/+wAAgyy5XTvgrWlOD1+TxbbvNADYCm1Tg=
github.com/valyala/fasthttp v1.57.0/go.mod h1:h6ZBaPRlzpZ6O3H5t2gEk1Qi33+TmLvfwgLLp0t9CpE=
github.com/valyala/tcplisten v1.0.0 h1:rBHj/Xf+E1tRGZyWIWwJDiRY0zc1Js+CV5DqwacVSA8=
github.com/valyala/tcplisten v1.0.0/go.mod h1:T0xQ8SeCZGxckz9qRXTfG43PvQ/mcWh7FwZEA7Ioqkc=
github.com/tidwall/gjson v1.14.2/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/gjson v1.18.0 h1:FIDeeyB800efLX89e5a8Y0BNH+LOngJyGrIWxG2FKQY=
github.com/tidwall/gjson v1.18.0/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/match v1.1.1/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM=
github.com/tidwall/match v1.2.0 h1:0pt8FlkOwjN2fPt4bIl4BoNxb98gGHN2ObFEDkrfZnM=
github.com/tidwall/match v1.2.0/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM=
github.com/tidwall/pretty v1.2.0/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
github.com/tidwall/pretty v1.2.1 h1:qjsOFOWWQl+N3RsoF5/ssm1pHmJJwhjlSbZ51I6wMl4=
github.com/tidwall/pretty v1.2.1/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
github.com/tidwall/sjson v1.2.5 h1:kLy8mja+1c9jlljvWTlSazM7cKDRfJuR/bOJhcY5NcY=
github.com/tidwall/sjson v1.2.5/go.mod h1:Fvgq9kS/6ociJEDnK0Fk1cpYF4FIW6ZF7LAe+6jwd28=
github.com/tv42/httpunix v0.0.0-20150427012821-b75d8614f926/go.mod h1:9ESjWnEqriFuLhtthL60Sar/7RFoluCcXsuvEwTV5KM=
github.com/xo/terminfo v0.0.0-20210125001918-ca9a967f8778/go.mod h1:2MuV+tbUrU1zIOPMxZ5EncGwgmMJsa+9ucAQZXxsObs=
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e h1:JVG44RsyaB9T2KIHavMF/ppJZNG9ZpyihvCd0w101no=
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM=
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.uber.org/atomic v1.9.0 h1:ECmE8Bn/WFTYwEW/bpKD3M8VtR/zQVbavAoalC1PYyE=
go.uber.org/atomic v1.9.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
go.etcd.io/bbolt v1.4.3 h1:dEadXpI6G79deX5prL3QRNP6JB8UxVkqo4UPnHaNXJo=
go.etcd.io/bbolt v1.4.3/go.mod h1:tKQlpPaYCVFctUIgFKFnAlvbmB3tpy1vkTnDWohtc0E=
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.66.0 h1:PnV4kVnw0zOmwwFkAzCN5O07fw1YOIQor120zrh0AVo=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.66.0/go.mod h1:ofAwF4uinaf8SXdVzzbL4OsxJ3VfeEg3f/F6CeF49/Y=
go.opentelemetry.io/otel v1.41.0 h1:YlEwVsGAlCvczDILpUXpIpPSL/VPugt7zHThEMLce1c=
go.opentelemetry.io/otel v1.41.0/go.mod h1:Yt4UwgEKeT05QbLwbyHXEwhnjxNO6D8L5PQP51/46dE=
go.opentelemetry.io/otel/metric v1.41.0 h1:rFnDcs4gRzBcsO9tS8LCpgR0dxg4aaxWlJxCno7JlTQ=
go.opentelemetry.io/otel/metric v1.41.0/go.mod h1:xPvCwd9pU0VN8tPZYzDZV/BMj9CM9vs00GuBjeKhJps=
go.opentelemetry.io/otel/sdk v1.41.0 h1:YPIEXKmiAwkGl3Gu1huk1aYWwtpRLeskpV+wPisxBp8=
go.opentelemetry.io/otel/sdk v1.41.0/go.mod h1:ahFdU0G5y8IxglBf0QBJXgSe7agzjE4GiTJ6HT9ud90=
go.opentelemetry.io/otel/sdk/metric v1.41.0 h1:siZQIYBAUd1rlIWQT2uCxWJxcCO7q3TriaMlf08rXw8=
go.opentelemetry.io/otel/sdk/metric v1.41.0/go.mod h1:HNBuSvT7ROaGtGI50ArdRLUnvRTRGniSUZbxiWxSO8Y=
go.opentelemetry.io/otel/trace v1.41.0 h1:Vbk2co6bhj8L59ZJ6/xFTskY+tGAbOnCtQGVVa9TIN0=
go.opentelemetry.io/otel/trace v1.41.0/go.mod h1:U1NU4ULCoxeDKc09yCWdWe+3QoyweJcISEVa1RBzOis=
go.uber.org/automaxprocs v1.6.0 h1:O3y2/QNTOdbF+e/dpXNNW7Rx2hZ4sTIPyybbxyNqTUs=
go.uber.org/automaxprocs v1.6.0/go.mod h1:ifeIMSnPZuznNm6jmdzmU3/bfk01Fe2fotchwEFJ8r8=
go.uber.org/multierr v1.9.0 h1:7fIwc/ZtS0q++VgcfqFDxSBZVv/Xo49/SYnDFupUwlI=
go.uber.org/multierr v1.9.0/go.mod h1:X2jQV1h+kxSjClGpnseKVIxpmcjrj7MNnI0bnlfKTVQ=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
go.uber.org/mock v0.5.2 h1:LbtPTcP8A5k9WPXj54PPPbjcI4Y6lhyOZXn+VS7wNko=
go.uber.org/mock v0.5.2/go.mod h1:wLlUxC2vVTPTaE3UD51E0BGOAElKrILxhVSDYQLld5o=
go.yaml.in/yaml/v2 v2.4.3 h1:6gvOSjQoTB3vt1l+CU+tSyi/HOjfOjRLJ4YwYZGwRO0=
go.yaml.in/yaml/v2 v2.4.3/go.mod h1:zSxWcmIDjOzPXpjlTTbAsKokqkDNAVtZO0WOMiT90s8=
go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.39.0 h1:SHs+kF4LP+f+p14esP5jAoDpHU8Gu/v9lFRK6IT5imM=
golang.org/x/crypto v0.39.0/go.mod h1:L+Xg3Wf6HoL4Bn4238Z6ft6KfEpN0tJGo53AAPC632U=
golang.org/x/exp v0.0.0-20220909182711-5c715a9e8561 h1:MDc5xs78ZrZr3HMQugiXOAkSZtfTpbJLDr/lwfgO53E=
golang.org/x/exp v0.0.0-20220909182711-5c715a9e8561/go.mod h1:cyybsKvd6eL0RnXn6p/Grxp8F5bW7iYuBgsNCOHpMYE=
golang.org/x/crypto v0.48.0 h1:/VRzVqiRSggnhY7gNRxPauEQ5Drw9haKdM0jqfcCFts=
golang.org/x/crypto v0.48.0/go.mod h1:r0kV5h3qnFPlQnBSrULhlsRfryS2pmewsg+XfMgkVos=
golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa h1:FRnLl4eNAQl8hwxVVC17teOw8kdjVDVAiFMtgUdTSRQ=
golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa/go.mod h1:zk2irFbV9DP96SEBUUAy67IdHUaZuSnrz1n472HUCLE=
golang.org/x/exp/typeparams v0.0.0-20260218203240-3dfff04db8fa h1:6Wi43P0isP8Nl8D4qJmA3VC4FswVH0RJkr5cauo67SQ=
golang.org/x/exp/typeparams v0.0.0-20260218203240-3dfff04db8fa/go.mod h1:PqrXSW65cXDZH0k4IeUbhmg/bcAZDbzNz3byBpKCsXo=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.33.0 h1:tHFzIWbBifEmbwtGz65eaWyGiGZatSrT9prnU8DbVL8=
golang.org/x/mod v0.33.0/go.mod h1:swjeQEj+6r7fODbD2cqrnje9PnziFuw4bmLbBZFrQ5w=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.51.0 h1:94R/GTO7mt3/4wIKpcR5gkGmRLOuE/2hNGeWq/GBIFo=
golang.org/x/net v0.51.0/go.mod h1:aamm+2QF5ogm02fjy5Bb7CQ0WMt1/WVM7FtyaTLlA9Y=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200106162015-b016eb3dc98e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200122134326-e047566fdf82/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200615200032-f1bc736245b1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211013075003-97ac67df715c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220319134239-a9b59b0215f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220503163025-988cb79eb6c6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.33.0 h1:q3i8TbbEz+JRD9ywIRlyRAQbM0qF7hu24q3teo2hbuw=
golang.org/x/sys v0.33.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/telemetry v0.0.0-20260304144227-18da59047661 h1:AjMsnhVxCTd9HssPcdXPhNTQ0YazDt5+52EqmcWQ0PA=
golang.org/x/telemetry v0.0.0-20260304144227-18da59047661/go.mod h1:NuITXsA9cTiqnXtVk+/wrBT2Ja4X5hsfGOYRJ6kgYjs=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/term v0.32.0 h1:DR4lr0TjUs3epypdhTOkMmuF5CDFJ/8pOnbzMZPQ7bg=
golang.org/x/term v0.32.0/go.mod h1:uZG1FhGx848Sqfsq4/DlJr3xGGsYMu/L5GW4abiaEPQ=
golang.org/x/term v0.40.0 h1:36e4zGLqU4yhjlmxEaagx2KuYbJq3EwY8K943ZsHcvg=
golang.org/x/term v0.40.0/go.mod h1:w2P8uVp06p2iyKKuvXIm7N/y0UCRt3UfJTfZ7oOpglM=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/text v0.26.0 h1:P42AVeLghgTYr4+xUnTRKDMqpar+PtX7KWuNQL21L8M=
golang.org/x/text v0.26.0/go.mod h1:QK15LZJUUQVJxhz7wXgxSy/CJaTFjd0G+YLonydOVQA=
golang.org/x/time v0.12.0 h1:ScB/8o8olJvc+CQPWrK3fPZNfh7qgwCrY0zJmoEQLSE=
golang.org/x/time v0.12.0/go.mod h1:CDIdPxbZBQxdj6cxyCIdrNogrJKMJ7pr37NYpMcMDSg=
golang.org/x/text v0.34.0 h1:oL/Qq0Kdaqxa1KbNeMKwQq0reLCCaFtqu2eNuSeNHbk=
golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA=
golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI=
golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.42.0 h1:uNgphsn75Tdz5Ji2q36v/nsFSfR/9BRFvqhGBaJGd5k=
golang.org/x/tools v0.42.0/go.mod h1:Ma6lCIwGZvHK6XtgbswSoWroEkhugApmsXyrUmBhfr0=
golang.org/x/tools/go/expect v0.1.1-deprecated h1:jpBZDwmgPhXsKZC6WhL20P4b/wmnpsEAGHaNy0n/rJM=
golang.org/x/tools/go/expect v0.1.1-deprecated/go.mod h1:eihoPOH+FgIqa3FpoTwguz/bVUSGBlGQU67vpBeOrBY=
golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated h1:1h2MnaIAIXISqTFKdENegdpAgUXz6NrPEsbIeWaBRvM=
golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated/go.mod h1:RVAQXBGNv1ib0J382/DPCRS/BPnsGebyM1Gj5VSDpG8=
golang.org/x/vuln v1.1.4 h1:Ju8QsuyhX3Hk8ma3CesTbO8vfJD9EvUBgHvkxHBzj0I=
golang.org/x/vuln v1.1.4/go.mod h1:F+45wmU18ym/ca5PLTPLsSzr2KppzswxPP603ldA67s=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gonum.org/v1/gonum v0.16.0 h1:5+ul4Swaf3ESvrOnidPp4GZbzf0mxVQpDCYUQE7OJfk=
gonum.org/v1/gonum v0.16.0/go.mod h1:fef3am4MQ93R2HHpKnLk4/Tbh/s0+wqD5nfa6Pnwy4E=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/genai v1.49.0 h1:Se+QJaH2GYK1aaR1o5S38mlU2GD5FnVvP76nfkV7LH0=
google.golang.org/genai v1.49.0/go.mod h1:A3kkl0nyBjyFlNjgxIwKq70julKbIxpSxqKO5gw/gmk=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260226221140-a57be14db171 h1:ggcbiqK8WWh6l1dnltU4BgWGIGo+EVYxCaAPih/zQXQ=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260226221140-a57be14db171/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
google.golang.org/grpc v1.79.1 h1:zGhSi45ODB9/p3VAawt9a+O/MULLl9dpizzNNpq7flY=
google.golang.org/grpc v1.79.1/go.mod h1:KmT0Kjez+0dde/v2j9vzwoAScgEPx/Bw1CYChhHLrHQ=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA=
gopkg.in/ini.v1 v1.67.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/ini.v1 v1.67.1 h1:tVBILHy0R6e4wkYOn3XmiITt/hEVH4TFMYvAX2Ytz6k=
gopkg.in/ini.v1 v1.67.1/go.mod h1:x/cyOwCgZqOkJoDIJ3c1KNHMo10+nLGAhh+kn3Zizss=
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10=
gopkg.in/yaml.v2 v2.2.5/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
honnef.co/go/tools v0.7.0 h1:w6WUp1VbkqPEgLz4rkBzH/CSU6HkoqNLp6GstyTx3lU=
honnef.co/go/tools v0.7.0/go.mod h1:pm29oPxeP3P82ISxZDgIYeOaf9ta6Pi0EWvCFoLG2vc=
+650
View File
@@ -0,0 +1,650 @@
package admin
import (
"crypto/rand"
"crypto/rsa"
"crypto/x509"
"crypto/x509/pkix"
"encoding/base64"
"encoding/pem"
"fmt"
"log/slog"
"math/big"
"net"
"os"
"path/filepath"
"strings"
"text/template"
"time"
toml "github.com/pelletier/go-toml/v2"
"gopkg.in/ini.v1"
)
// RemoteNode holds basic info about a remote cluster node for config generation.
type RemoteNode struct {
Name string
Host string
Region string
AZ string
Services []string
}
type ConfigSettings struct {
AccessKey string
SecretKey string
AccountID string
Region string
NatsToken string
DataDir string
// Add more fields as needed
Node string
Az string
Port string
BindIP string
// Cluster settings
ClusterBindIP string
ClusterRoutes []string
ClusterName string
// Predastore multi-node
PredastoreNodeID int
// Node capabilities
Services []string
// Other nodes in the cluster (for config source of truth)
RemoteNodes []RemoteNode
}
// PredastoreNodeConfig describes a single Predastore node for multi-node config generation.
type PredastoreNodeConfig struct {
ID int
Host string
}
type ConfigFile struct {
Name string
Path string
Template string
}
func GenerateConfigFiles(configs []ConfigFile, configSettings ConfigSettings) error {
for _, cfg := range configs {
if err := GenerateConfigFile(cfg.Path, cfg.Template, configSettings); err != nil {
return fmt.Errorf("error creating %s: %v", cfg.Name, err)
}
fmt.Printf("✅ Created: %s\n", cfg.Name)
}
return nil
}
// generateConfigFile creates a configuration file from a template
func GenerateConfigFile(configPath string, configTemplate string, configSettings ConfigSettings) error {
// Parse the embedded template
tmpl, err := template.New("config").Parse(configTemplate)
if err != nil {
return fmt.Errorf("failed to parse template: %w", err)
}
// Create file with secure permissions
f, err := os.OpenFile(configPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600)
if err != nil {
return fmt.Errorf("failed to create config file: %w", err)
}
defer f.Close()
// Execute template
if err := tmpl.Execute(f, configSettings); err != nil {
return fmt.Errorf("failed to execute template: %w", err)
}
return nil
}
func GenerateCertificatesIfNeeded(configDir string, force bool, bindIP string) (caCertPath string) {
// Certificate paths
caCertPath = filepath.Join(configDir, "ca.pem")
caKeyPath := filepath.Join(configDir, "ca.key")
serverCertPath := filepath.Join(configDir, "server.pem")
serverKeyPath := filepath.Join(configDir, "server.key")
// Check if we need to generate certificates
needsGeneration := force ||
!FileExists(caCertPath) || !FileExists(caKeyPath) ||
!FileExists(serverCertPath) || !FileExists(serverKeyPath)
if needsGeneration {
fmt.Println("\n🔐 Generating Certificate Authority and SSL certificates...")
// Step 1: Generate CA certificate
if err := GenerateCACert(caCertPath, caKeyPath); err != nil {
fmt.Fprintf(os.Stderr, "Error generating CA certificate: %v\n", err)
os.Exit(1)
}
fmt.Printf("✅ CA certificate generated:\n")
fmt.Printf(" CA Certificate: %s\n", caCertPath)
fmt.Printf(" CA Key: %s\n", caKeyPath)
// Step 2: Generate server certificate signed by CA (with bind IP in SANs)
if err := GenerateSignedCert(serverCertPath, serverKeyPath, caCertPath, caKeyPath, bindIP); err != nil {
fmt.Fprintf(os.Stderr, "Error generating server certificate: %v\n", err)
os.Exit(1)
}
fmt.Printf("✅ Server certificate generated (signed by CA):\n")
fmt.Printf(" Certificate: %s\n", serverCertPath)
fmt.Printf(" Key: %s\n", serverKeyPath)
// Print instructions for adding CA to system trust store
fmt.Println("\n📋 To trust the Hive CA system-wide (recommended):")
fmt.Printf(" sudo cp %s /usr/local/share/ca-certificates/hive-ca.crt\n", caCertPath)
fmt.Println(" sudo update-ca-certificates")
fmt.Println("\n This allows AWS CLI and other tools to trust Hive services automatically.")
} else {
fmt.Println("\n✅ CA and SSL certificates already exist")
}
return caCertPath
}
// GenerateServerCertOnly generates a server certificate signed by an existing CA.
// Used by joining nodes that receive the CA from the leader.
func GenerateServerCertOnly(configDir string, bindIP string) error {
caCertPath := filepath.Join(configDir, "ca.pem")
caKeyPath := filepath.Join(configDir, "ca.key")
serverCertPath := filepath.Join(configDir, "server.pem")
serverKeyPath := filepath.Join(configDir, "server.key")
// Verify CA files exist
if !FileExists(caCertPath) || !FileExists(caKeyPath) {
return fmt.Errorf("CA files not found in %s", configDir)
}
// Generate server cert signed by CA with this node's bind IP
return GenerateSignedCert(serverCertPath, serverKeyPath, caCertPath, caKeyPath, bindIP)
}
func CreateServiceDirectories(hiveRoot string) {
// Create additional directories
dirs := []string{
filepath.Join(hiveRoot, "images"),
filepath.Join(hiveRoot, "amis"),
filepath.Join(hiveRoot, "volumes"),
filepath.Join(hiveRoot, "state"),
filepath.Join(hiveRoot, "logs"),
filepath.Join(hiveRoot, "nats"),
filepath.Join(hiveRoot, "predastore"),
filepath.Join(hiveRoot, "viperblock"),
filepath.Join(hiveRoot, "hive"),
}
fmt.Println("\n📁 Creating directory structure...")
for _, dir := range dirs {
// Check if directory exists
if _, err := os.Stat(dir); os.IsNotExist(err) {
if err := os.MkdirAll(dir, 0750); err != nil {
fmt.Fprintf(os.Stderr, "Warning: Could not create %s: %v\n", dir, err)
}
}
}
fmt.Printf("✅ Directory structure created in %s\n", hiveRoot)
}
// Helper functions
func FileExists(path string) bool {
_, err := os.Stat(path)
return err == nil
}
// updateAWSINIFile updates or creates an AWS INI file section with given key-value pairs
func UpdateAWSINIFile(path, section string, values map[string]string) error {
var cfg *ini.File
var err error
// Load existing file or create new one
if FileExists(path) {
cfg, err = ini.Load(path)
if err != nil {
return fmt.Errorf("failed to load INI file: %w", err)
}
} else {
cfg = ini.Empty()
}
// Get or create section
sec, err := cfg.NewSection(section)
if err != nil {
// Section already exists, get it
sec, err = cfg.GetSection(section)
if err != nil {
return fmt.Errorf("failed to get section: %w", err)
}
}
// Set key-value pairs
for key, value := range values {
sec.Key(key).SetValue(value)
}
// Save with proper permissions
return cfg.SaveTo(path)
}
// generateAWSAccessKey generates an AWS-style access key
// Format: AKIA + 16 random uppercase alphanumeric characters
func GenerateAWSAccessKey() string {
const prefix = "AKIA"
const charset = "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"
const length = 16
result := make([]byte, length)
for i := range result {
num, _ := rand.Int(rand.Reader, big.NewInt(int64(len(charset))))
result[i] = charset[num.Int64()]
}
return prefix + string(result)
}
// generateAWSSecretKey generates an AWS-style secret key
// 40 character base64-encoded string
func GenerateAWSSecretKey() string {
bytes := make([]byte, 30) // 30 bytes = 40 chars in base64
if _, err := rand.Read(bytes); err != nil {
panic(err)
}
return base64.StdEncoding.EncodeToString(bytes)
}
// GenerateAccountID returns the global platform account ID (000000000000).
// This is used during bootstrap (hive admin init) for the root/platform account.
// Customer accounts are created via IAMService.CreateAccount() with sequential IDs.
func GenerateAccountID() string {
return "000000000000"
}
// generateNATSToken generates a secure random token for NATS
func GenerateNATSToken() string {
bytes := make([]byte, 32)
if _, err := rand.Read(bytes); err != nil {
panic(err)
}
return "nats_" + base64.URLEncoding.EncodeToString(bytes)[:32]
}
// GenerateCACert generates a Certificate Authority certificate and key
func GenerateCACert(caCertPath, caKeyPath string) error {
// Generate CA private key
caPrivateKey, err := rsa.GenerateKey(rand.Reader, 4096)
if err != nil {
return fmt.Errorf("failed to generate CA private key: %w", err)
}
// Create CA certificate template
notBefore := time.Now()
notAfter := notBefore.Add(3650 * 24 * time.Hour) // 10 years
serialNumber, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128))
if err != nil {
return fmt.Errorf("failed to generate serial number: %w", err)
}
caTemplate := x509.Certificate{
SerialNumber: serialNumber,
Subject: pkix.Name{
CommonName: "Hive Local CA",
Organization: []string{"Hive Platform"},
},
NotBefore: notBefore,
NotAfter: notAfter,
KeyUsage: x509.KeyUsageCertSign | x509.KeyUsageCRLSign | x509.KeyUsageDigitalSignature,
BasicConstraintsValid: true,
IsCA: true,
MaxPathLen: 1,
}
// Self-sign the CA certificate
caDerBytes, err := x509.CreateCertificate(rand.Reader, &caTemplate, &caTemplate, &caPrivateKey.PublicKey, caPrivateKey)
if err != nil {
return fmt.Errorf("failed to create CA certificate: %w", err)
}
// Write CA certificate to file
caCertOut, err := os.Create(caCertPath)
if err != nil {
return fmt.Errorf("failed to create CA cert file: %w", err)
}
defer caCertOut.Close()
if err := pem.Encode(caCertOut, &pem.Block{Type: "CERTIFICATE", Bytes: caDerBytes}); err != nil {
return fmt.Errorf("failed to write CA cert: %w", err)
}
// Write CA private key to file
caKeyOut, err := os.OpenFile(caKeyPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600)
if err != nil {
return fmt.Errorf("failed to create CA key file: %w", err)
}
defer caKeyOut.Close()
caPrivBytes, err := x509.MarshalPKCS8PrivateKey(caPrivateKey)
if err != nil {
return fmt.Errorf("failed to marshal CA private key: %w", err)
}
if err := pem.Encode(caKeyOut, &pem.Block{Type: "PRIVATE KEY", Bytes: caPrivBytes}); err != nil {
return fmt.Errorf("failed to write CA key: %w", err)
}
return nil
}
// GenerateSignedCert generates a server certificate signed by the CA.
// extraIPs are additional IP addresses to include in the certificate's SANs.
func GenerateSignedCert(certPath, keyPath, caCertPath, caKeyPath string, extraIPs ...string) error {
// Load CA certificate
caCertPEM, err := os.ReadFile(caCertPath)
if err != nil {
return fmt.Errorf("failed to read CA cert: %w", err)
}
caCertBlock, _ := pem.Decode(caCertPEM)
if caCertBlock == nil {
return fmt.Errorf("failed to decode CA cert PEM")
}
caCert, err := x509.ParseCertificate(caCertBlock.Bytes)
if err != nil {
return fmt.Errorf("failed to parse CA cert: %w", err)
}
// Load CA private key
caKeyPEM, err := os.ReadFile(caKeyPath)
if err != nil {
return fmt.Errorf("failed to read CA key: %w", err)
}
caKeyBlock, _ := pem.Decode(caKeyPEM)
if caKeyBlock == nil {
return fmt.Errorf("failed to decode CA key PEM")
}
caPrivateKey, err := x509.ParsePKCS8PrivateKey(caKeyBlock.Bytes)
if err != nil {
return fmt.Errorf("failed to parse CA private key: %w", err)
}
caRSAKey, ok := caPrivateKey.(*rsa.PrivateKey)
if !ok {
return fmt.Errorf("CA key is not RSA")
}
// Generate server private key
serverPrivateKey, err := rsa.GenerateKey(rand.Reader, 4096)
if err != nil {
return fmt.Errorf("failed to generate server private key: %w", err)
}
// Create server certificate template
notBefore := time.Now()
notAfter := notBefore.Add(365 * 24 * time.Hour) // 1 year for server certs
serialNumber, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128))
if err != nil {
return fmt.Errorf("failed to generate serial number: %w", err)
}
// Build IP list: localhost IPs + any extra IPs (e.g., bind IP for multi-node)
ipAddresses := []net.IP{net.ParseIP("127.0.0.1"), net.ParseIP("::1")}
for _, ip := range extraIPs {
if ip != "" && ip != "127.0.0.1" && ip != "::1" && ip != "0.0.0.0" {
if parsed := net.ParseIP(ip); parsed != nil {
ipAddresses = append(ipAddresses, parsed)
}
}
}
serverTemplate := x509.Certificate{
SerialNumber: serialNumber,
Subject: pkix.Name{
CommonName: "localhost",
Organization: []string{"Hive Platform"},
},
NotBefore: notBefore,
NotAfter: notAfter,
KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
BasicConstraintsValid: true,
DNSNames: []string{"localhost"},
IPAddresses: ipAddresses,
}
// Sign the server certificate with the CA
serverDerBytes, err := x509.CreateCertificate(rand.Reader, &serverTemplate, caCert, &serverPrivateKey.PublicKey, caRSAKey)
if err != nil {
return fmt.Errorf("failed to create server certificate: %w", err)
}
// Write server certificate to file
certOut, err := os.Create(certPath)
if err != nil {
return fmt.Errorf("failed to create cert file: %w", err)
}
defer certOut.Close()
if err := pem.Encode(certOut, &pem.Block{Type: "CERTIFICATE", Bytes: serverDerBytes}); err != nil {
return fmt.Errorf("failed to write cert: %w", err)
}
// Write server private key to file
keyOut, err := os.OpenFile(keyPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600)
if err != nil {
return fmt.Errorf("failed to create key file: %w", err)
}
defer keyOut.Close()
privBytes, err := x509.MarshalPKCS8PrivateKey(serverPrivateKey)
if err != nil {
return fmt.Errorf("failed to marshal private key: %w", err)
}
if err := pem.Encode(keyOut, &pem.Block{Type: "PRIVATE KEY", Bytes: privBytes}); err != nil {
return fmt.Errorf("failed to write key: %w", err)
}
return nil
}
// generateSelfSignedCert generates a self-signed SSL certificate (legacy, kept for compatibility)
func GenerateSelfSignedCert(certPath, keyPath string) error {
// Generate private key
privateKey, err := rsa.GenerateKey(rand.Reader, 4096)
if err != nil {
return fmt.Errorf("failed to generate private key: %w", err)
}
// Create certificate template
notBefore := time.Now()
notAfter := notBefore.Add(3650 * 24 * time.Hour) // 10 years
serialNumber, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128))
if err != nil {
return fmt.Errorf("failed to generate serial number: %w", err)
}
template := x509.Certificate{
SerialNumber: serialNumber,
Subject: pkix.Name{
CommonName: "localhost",
Organization: []string{"Hive Platform"},
},
NotBefore: notBefore,
NotAfter: notAfter,
KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
BasicConstraintsValid: true,
DNSNames: []string{"localhost"},
IPAddresses: []net.IP{net.ParseIP("127.0.0.1"), net.ParseIP("::1")},
}
// Create certificate
derBytes, err := x509.CreateCertificate(rand.Reader, &template, &template, &privateKey.PublicKey, privateKey)
if err != nil {
return fmt.Errorf("failed to create certificate: %w", err)
}
// Write certificate to file
certOut, err := os.Create(certPath)
if err != nil {
return fmt.Errorf("failed to create cert file: %w", err)
}
defer certOut.Close()
if err := pem.Encode(certOut, &pem.Block{Type: "CERTIFICATE", Bytes: derBytes}); err != nil {
return fmt.Errorf("failed to write cert: %w", err)
}
// Write private key to file
keyOut, err := os.OpenFile(keyPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600)
if err != nil {
return fmt.Errorf("failed to create key file: %w", err)
}
defer keyOut.Close()
privBytes, err := x509.MarshalPKCS8PrivateKey(privateKey)
if err != nil {
return fmt.Errorf("failed to marshal private key: %w", err)
}
if err := pem.Encode(keyOut, &pem.Block{Type: "PRIVATE KEY", Bytes: privBytes}); err != nil {
return fmt.Errorf("failed to write key: %w", err)
}
return nil
}
// SetupAWSCredentials updates ~/.aws/credentials and ~/.aws/config.
// bindIP is the IP the AWS gateway listens on. If empty or "0.0.0.0", defaults to "localhost".
func SetupAWSCredentials(accessKey, secretKey, region, certPath, bindIP string) error {
homeDir, err := os.UserHomeDir()
if err != nil {
return err
}
awsDir := filepath.Join(homeDir, ".aws")
if err := os.MkdirAll(awsDir, 0700); err != nil {
return err
}
credPath := filepath.Join(awsDir, "credentials")
configPath := filepath.Join(awsDir, "config")
// Determine profile name
//profileName := "default"
// Use hive as the default profile
profileName := "hive"
if FileExists(credPath) {
// Check if default profile already exists
cfg, err := ini.Load(credPath)
if err == nil && cfg.HasSection("default") {
profileName = "hive"
}
}
// Update credentials file
if err := UpdateAWSINIFile(credPath, profileName, map[string]string{
"aws_access_key_id": accessKey,
"aws_secret_access_key": secretKey,
}); err != nil {
return err
}
// Update config file
configSection := profileName
if profileName != "default" {
configSection = "profile " + profileName
}
endpointHost := bindIP
if endpointHost == "" || endpointHost == "0.0.0.0" {
endpointHost = "localhost"
}
if err := UpdateAWSINIFile(configPath, configSection, map[string]string{
"region": region,
"endpoint_url": fmt.Sprintf("https://%s:9999", endpointHost),
"ca_bundle": certPath,
"output": "json",
}); err != nil {
return err
}
fmt.Printf(" Profile: %s\n", profileName)
if profileName != "default" {
fmt.Printf(" Use: export AWS_PROFILE=%s\n", profileName)
}
return nil
}
// GenerateMultiNodePredastoreConfig produces a complete predastore.toml for a
// multi-node Predastore cluster. Each node gets its own DB entry (port 6660)
// and shard entry (port 9991) on a distinct IP. Node ID 1 is the bootstrap leader.
func GenerateMultiNodePredastoreConfig(templateStr string, nodes []PredastoreNodeConfig, accessKey, secretKey, region string) (string, error) {
if len(nodes) < 3 {
return "", fmt.Errorf("multi-node predastore requires at least 3 nodes, got %d", len(nodes))
}
data := struct {
Nodes []PredastoreNodeConfig
AccessKey string
SecretKey string
Region string
}{nodes, accessKey, secretKey, region}
tmpl, err := template.New("predastore-multinode").Parse(templateStr)
if err != nil {
return "", fmt.Errorf("failed to parse predastore template: %w", err)
}
var b strings.Builder
if err := tmpl.Execute(&b, data); err != nil {
return "", fmt.Errorf("failed to execute predastore template: %w", err)
}
return b.String(), nil
}
// FindNodeIDByIP returns the node ID for the given IP in the node list,
// or 0 if the IP is not found.
func FindNodeIDByIP(nodes []PredastoreNodeConfig, ip string) int {
for _, n := range nodes {
if n.Host == ip {
return n.ID
}
}
return 0
}
// ParsePredastoreNodeIDFromConfig parses a predastore.toml string and returns
// the node ID whose host matches the given IP, or 0 if not found.
func ParsePredastoreNodeIDFromConfig(tomlContent string, ip string) int {
var cfg struct {
DB []PredastoreNodeConfig `toml:"db"`
}
if err := toml.Unmarshal([]byte(tomlContent), &cfg); err != nil {
slog.Warn("Failed to parse predastore.toml content", "error", err)
return 0
}
return FindNodeIDByIP(cfg.DB, ip)
}
+617
View File
@@ -0,0 +1,617 @@
package admin
import (
"crypto/x509"
"encoding/base64"
"encoding/pem"
"fmt"
"net"
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// --- Key / Token generation ---
func TestGenerateAWSAccessKey_Format(t *testing.T) {
key := GenerateAWSAccessKey()
assert.Len(t, key, 20)
assert.True(t, strings.HasPrefix(key, "AKIA"))
for _, c := range key[4:] {
assert.True(t, (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9'),
"unexpected character %c in access key suffix", c)
}
}
func TestGenerateAWSAccessKey_Uniqueness(t *testing.T) {
assert.NotEqual(t, GenerateAWSAccessKey(), GenerateAWSAccessKey())
}
func TestGenerateAWSSecretKey_Format(t *testing.T) {
key := GenerateAWSSecretKey()
assert.Len(t, key, 40)
_, err := base64.StdEncoding.DecodeString(key)
assert.NoError(t, err, "secret key should be valid base64")
}
func TestGenerateAWSSecretKey_Uniqueness(t *testing.T) {
assert.NotEqual(t, GenerateAWSSecretKey(), GenerateAWSSecretKey())
}
func TestGenerateAccountID_ReturnsGlobalID(t *testing.T) {
id := GenerateAccountID()
assert.Equal(t, "000000000000", id)
assert.Len(t, id, 12)
}
func TestGenerateAccountID_Deterministic(t *testing.T) {
assert.Equal(t, GenerateAccountID(), GenerateAccountID())
}
func TestGenerateNATSToken_Format(t *testing.T) {
token := GenerateNATSToken()
assert.True(t, strings.HasPrefix(token, "nats_"))
assert.Len(t, token, 37) // 5 prefix + 32 random
// URL-safe base64: no '+' or '/'
assert.NotContains(t, token, "+")
assert.NotContains(t, token, "/")
}
func TestGenerateNATSToken_Uniqueness(t *testing.T) {
assert.NotEqual(t, GenerateNATSToken(), GenerateNATSToken())
}
// --- Config file generation ---
func TestGenerateConfigFile_Success(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "test.conf")
tmpl := "region = {{.Region}}\nnode = {{.Node}}"
settings := ConfigSettings{Region: "us-east-1", Node: "node1"}
err := GenerateConfigFile(path, tmpl, settings)
require.NoError(t, err)
data, err := os.ReadFile(path)
require.NoError(t, err)
assert.Contains(t, string(data), "region = us-east-1")
assert.Contains(t, string(data), "node = node1")
info, _ := os.Stat(path)
assert.Equal(t, os.FileMode(0600), info.Mode().Perm())
}
func TestGenerateConfigFile_InvalidTemplate(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "bad.conf")
err := GenerateConfigFile(path, "{{.Unclosed", ConfigSettings{})
assert.Error(t, err)
assert.Contains(t, err.Error(), "failed to parse template")
}
func TestGenerateConfigFile_InvalidPath(t *testing.T) {
err := GenerateConfigFile("/nonexistent/dir/file.conf", "ok", ConfigSettings{})
assert.Error(t, err)
assert.Contains(t, err.Error(), "failed to create config file")
}
func TestGenerateConfigFile_OverwritesExisting(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "overwrite.conf")
require.NoError(t, GenerateConfigFile(path, "old={{.Region}}", ConfigSettings{Region: "old"}))
require.NoError(t, GenerateConfigFile(path, "new={{.Region}}", ConfigSettings{Region: "new"}))
data, _ := os.ReadFile(path)
assert.Contains(t, string(data), "new=new")
assert.NotContains(t, string(data), "old")
}
func TestGenerateConfigFiles_AllSucceed(t *testing.T) {
dir := t.TempDir()
configs := []ConfigFile{
{Name: "a", Path: filepath.Join(dir, "a.conf"), Template: "a={{.Region}}"},
{Name: "b", Path: filepath.Join(dir, "b.conf"), Template: "b={{.Node}}"},
}
err := GenerateConfigFiles(configs, ConfigSettings{Region: "us-west-2", Node: "n1"})
require.NoError(t, err)
for _, cfg := range configs {
assert.True(t, FileExists(cfg.Path))
}
}
func TestGenerateConfigFiles_StopsOnFirstError(t *testing.T) {
dir := t.TempDir()
configs := []ConfigFile{
{Name: "ok", Path: filepath.Join(dir, "ok.conf"), Template: "ok"},
{Name: "bad", Path: "/nonexistent/dir/bad.conf", Template: "bad"},
{Name: "never", Path: filepath.Join(dir, "never.conf"), Template: "never"},
}
err := GenerateConfigFiles(configs, ConfigSettings{})
assert.Error(t, err)
assert.Contains(t, err.Error(), "bad")
assert.False(t, FileExists(filepath.Join(dir, "never.conf")))
}
// --- FileExists ---
func TestFileExists(t *testing.T) {
dir := t.TempDir()
existing := filepath.Join(dir, "exists.txt")
require.NoError(t, os.WriteFile(existing, []byte("hi"), 0644))
assert.True(t, FileExists(existing))
assert.False(t, FileExists(filepath.Join(dir, "nope.txt")))
assert.True(t, FileExists(dir)) // directory also returns true
}
// --- UpdateAWSINIFile ---
func TestUpdateAWSINIFile_CreateNew(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "credentials")
err := UpdateAWSINIFile(path, "hive", map[string]string{
"aws_access_key_id": "AKIATEST",
"aws_secret_access_key": "secrettest",
})
require.NoError(t, err)
data, _ := os.ReadFile(path)
content := string(data)
assert.Contains(t, content, "[hive]")
assert.Contains(t, content, "AKIATEST")
assert.Contains(t, content, "secrettest")
}
func TestUpdateAWSINIFile_UpdateExisting(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "credentials")
// Create with initial value
require.NoError(t, UpdateAWSINIFile(path, "hive", map[string]string{"key": "old"}))
// Update
require.NoError(t, UpdateAWSINIFile(path, "hive", map[string]string{"key": "new"}))
data, _ := os.ReadFile(path)
content := string(data)
assert.Contains(t, content, "new")
}
func TestUpdateAWSINIFile_AddNewSection(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "credentials")
require.NoError(t, UpdateAWSINIFile(path, "default", map[string]string{"key": "default-val"}))
require.NoError(t, UpdateAWSINIFile(path, "hive", map[string]string{"key": "hive-val"}))
data, _ := os.ReadFile(path)
content := string(data)
assert.Contains(t, content, "[default]")
assert.Contains(t, content, "[hive]")
assert.Contains(t, content, "default-val")
assert.Contains(t, content, "hive-val")
}
// --- SetupAWSCredentials ---
func TestSetupAWSCredentials_CreatesFiles(t *testing.T) {
dir := t.TempDir()
t.Setenv("HOME", dir)
err := SetupAWSCredentials("AKIATEST123", "secret123", "us-east-1", "/path/to/ca.pem", "")
require.NoError(t, err)
credData, _ := os.ReadFile(filepath.Join(dir, ".aws", "credentials"))
configData, _ := os.ReadFile(filepath.Join(dir, ".aws", "config"))
assert.Contains(t, string(credData), "AKIATEST123")
assert.Contains(t, string(credData), "secret123")
assert.Contains(t, string(configData), "us-east-1")
assert.Contains(t, string(configData), "https://localhost:9999")
assert.Contains(t, string(configData), "/path/to/ca.pem")
}
func TestSetupAWSCredentials_PreservesExistingProfiles(t *testing.T) {
dir := t.TempDir()
t.Setenv("HOME", dir)
awsDir := filepath.Join(dir, ".aws")
require.NoError(t, os.MkdirAll(awsDir, 0700))
require.NoError(t, UpdateAWSINIFile(filepath.Join(awsDir, "credentials"), "default", map[string]string{
"aws_access_key_id": "EXISTING_KEY",
}))
err := SetupAWSCredentials("NEWAKEY", "NEWSECRET", "us-west-2", "/ca.pem", "")
require.NoError(t, err)
data, _ := os.ReadFile(filepath.Join(awsDir, "credentials"))
content := string(data)
assert.Contains(t, content, "EXISTING_KEY")
assert.Contains(t, content, "NEWAKEY")
}
func TestSetupAWSCredentials_UsesBindIP(t *testing.T) {
dir := t.TempDir()
t.Setenv("HOME", dir)
err := SetupAWSCredentials("AKIATEST123", "secret123", "us-east-1", "/ca.pem", "10.11.12.1")
require.NoError(t, err)
configData, _ := os.ReadFile(filepath.Join(dir, ".aws", "config"))
assert.Contains(t, string(configData), "https://10.11.12.1:9999")
assert.NotContains(t, string(configData), "localhost")
}
func TestSetupAWSCredentials_FallsBackToLocalhostForWildcard(t *testing.T) {
dir := t.TempDir()
t.Setenv("HOME", dir)
err := SetupAWSCredentials("AKIATEST123", "secret123", "us-east-1", "/ca.pem", "0.0.0.0")
require.NoError(t, err)
configData, _ := os.ReadFile(filepath.Join(dir, ".aws", "config"))
assert.Contains(t, string(configData), "https://localhost:9999")
}
// --- Certificate generation ---
func TestGenerateCACert_CreatesValidCA(t *testing.T) {
t.Parallel()
dir := t.TempDir()
certPath := filepath.Join(dir, "ca.pem")
keyPath := filepath.Join(dir, "ca.key")
err := GenerateCACert(certPath, keyPath)
require.NoError(t, err)
// Parse certificate
certPEM, _ := os.ReadFile(certPath)
block, _ := pem.Decode(certPEM)
require.NotNil(t, block)
assert.Equal(t, "CERTIFICATE", block.Type)
cert, err := x509.ParseCertificate(block.Bytes)
require.NoError(t, err)
assert.True(t, cert.IsCA)
assert.Equal(t, "Hive Local CA", cert.Subject.CommonName)
assert.NotZero(t, cert.KeyUsage&x509.KeyUsageCertSign)
// Parse key
keyPEM, _ := os.ReadFile(keyPath)
keyBlock, _ := pem.Decode(keyPEM)
require.NotNil(t, keyBlock)
assert.Equal(t, "PRIVATE KEY", keyBlock.Type)
// Verify key file permissions
info, _ := os.Stat(keyPath)
assert.Equal(t, os.FileMode(0600), info.Mode().Perm())
}
// TestGenerateSignedCert uses subtests sharing a single CA to avoid repeated 4096-bit key generation.
func TestGenerateSignedCert(t *testing.T) {
t.Parallel()
// Generate CA once for all subtests (~0.7s instead of ~0.7s x 3)
caDir := t.TempDir()
caCertPath := filepath.Join(caDir, "ca.pem")
caKeyPath := filepath.Join(caDir, "ca.key")
require.NoError(t, GenerateCACert(caCertPath, caKeyPath))
t.Run("CreatesValidCert", func(t *testing.T) {
t.Parallel()
dir := t.TempDir()
certPath := filepath.Join(dir, "server.pem")
keyPath := filepath.Join(dir, "server.key")
require.NoError(t, GenerateSignedCert(certPath, keyPath, caCertPath, caKeyPath))
certPEM, _ := os.ReadFile(certPath)
block, _ := pem.Decode(certPEM)
require.NotNil(t, block)
cert, err := x509.ParseCertificate(block.Bytes)
require.NoError(t, err)
assert.False(t, cert.IsCA)
assert.Equal(t, "localhost", cert.Subject.CommonName)
assert.Contains(t, cert.DNSNames, "localhost")
hasLoopback := false
for _, ip := range cert.IPAddresses {
if ip.Equal(net.ParseIP("127.0.0.1")) {
hasLoopback = true
}
}
assert.True(t, hasLoopback)
// Verify against CA
caCertPEM, _ := os.ReadFile(caCertPath)
caBlock, _ := pem.Decode(caCertPEM)
caCert, _ := x509.ParseCertificate(caBlock.Bytes)
pool := x509.NewCertPool()
pool.AddCert(caCert)
_, err = cert.Verify(x509.VerifyOptions{Roots: pool})
assert.NoError(t, err)
info, _ := os.Stat(keyPath)
assert.Equal(t, os.FileMode(0600), info.Mode().Perm())
})
t.Run("ExtraIPs", func(t *testing.T) {
t.Parallel()
dir := t.TempDir()
certPath := filepath.Join(dir, "server.pem")
keyPath := filepath.Join(dir, "server.key")
require.NoError(t, GenerateSignedCert(certPath, keyPath, caCertPath, caKeyPath, "192.168.1.100"))
certPEM, _ := os.ReadFile(certPath)
block, _ := pem.Decode(certPEM)
cert, _ := x509.ParseCertificate(block.Bytes)
hasExtraIP := false
for _, ip := range cert.IPAddresses {
if ip.Equal(net.ParseIP("192.168.1.100")) {
hasExtraIP = true
}
}
assert.True(t, hasExtraIP, "cert should contain extra IP 192.168.1.100")
})
t.Run("SkipsDuplicateAndSpecialIPs", func(t *testing.T) {
t.Parallel()
dir := t.TempDir()
certPath := filepath.Join(dir, "server.pem")
keyPath := filepath.Join(dir, "server.key")
require.NoError(t, GenerateSignedCert(certPath, keyPath, caCertPath, caKeyPath, "127.0.0.1", "::1", "0.0.0.0", ""))
certPEM, _ := os.ReadFile(certPath)
block, _ := pem.Decode(certPEM)
cert, _ := x509.ParseCertificate(block.Bytes)
assert.Len(t, cert.IPAddresses, 2)
})
t.Run("InvalidCACert", func(t *testing.T) {
t.Parallel()
dir := t.TempDir()
badCACert := filepath.Join(dir, "bad-ca.pem")
require.NoError(t, os.WriteFile(badCACert, []byte("not-a-cert"), 0600))
err := GenerateSignedCert(filepath.Join(dir, "s.pem"), filepath.Join(dir, "s.key"), badCACert, caKeyPath)
assert.Error(t, err)
})
}
func TestGenerateSelfSignedCert_CreatesValidCert(t *testing.T) {
t.Parallel()
dir := t.TempDir()
certPath := filepath.Join(dir, "self.pem")
keyPath := filepath.Join(dir, "self.key")
require.NoError(t, GenerateSelfSignedCert(certPath, keyPath))
certPEM, _ := os.ReadFile(certPath)
block, _ := pem.Decode(certPEM)
require.NotNil(t, block)
cert, err := x509.ParseCertificate(block.Bytes)
require.NoError(t, err)
assert.Contains(t, cert.DNSNames, "localhost")
info, _ := os.Stat(keyPath)
assert.Equal(t, os.FileMode(0600), info.Mode().Perm())
}
// --- Certificate orchestrator ---
// TestGenerateCertificatesIfNeeded uses subtests to share the initial generation.
func TestGenerateCertificatesIfNeeded(t *testing.T) {
t.Parallel()
dir := t.TempDir()
// First call generates all certs
caCertPath := GenerateCertificatesIfNeeded(dir, false, "")
assert.Equal(t, filepath.Join(dir, "ca.pem"), caCertPath)
assert.True(t, FileExists(filepath.Join(dir, "ca.pem")))
assert.True(t, FileExists(filepath.Join(dir, "ca.key")))
assert.True(t, FileExists(filepath.Join(dir, "server.pem")))
assert.True(t, FileExists(filepath.Join(dir, "server.key")))
t.Run("SkipsWhenAllExist", func(t *testing.T) {
caInfo, _ := os.Stat(filepath.Join(dir, "ca.pem"))
origModTime := caInfo.ModTime()
GenerateCertificatesIfNeeded(dir, false, "")
caInfo2, _ := os.Stat(filepath.Join(dir, "ca.pem"))
assert.Equal(t, origModTime, caInfo2.ModTime())
})
t.Run("ForceRegenerates", func(t *testing.T) {
origCA, _ := os.ReadFile(filepath.Join(dir, "ca.pem"))
GenerateCertificatesIfNeeded(dir, true, "")
newCA, _ := os.ReadFile(filepath.Join(dir, "ca.pem"))
assert.NotEqual(t, origCA, newCA)
})
}
func TestGenerateServerCertOnly(t *testing.T) {
t.Parallel()
// Generate CA once, reuse for subtests
caDir := t.TempDir()
require.NoError(t, GenerateCACert(filepath.Join(caDir, "ca.pem"), filepath.Join(caDir, "ca.key")))
t.Run("Success", func(t *testing.T) {
dir := t.TempDir()
// Copy CA files into test dir
caCert, _ := os.ReadFile(filepath.Join(caDir, "ca.pem"))
caKey, _ := os.ReadFile(filepath.Join(caDir, "ca.key"))
require.NoError(t, os.WriteFile(filepath.Join(dir, "ca.pem"), caCert, 0600))
require.NoError(t, os.WriteFile(filepath.Join(dir, "ca.key"), caKey, 0600))
err := GenerateServerCertOnly(dir, "10.0.0.5")
require.NoError(t, err)
assert.True(t, FileExists(filepath.Join(dir, "server.pem")))
assert.True(t, FileExists(filepath.Join(dir, "server.key")))
certPEM, _ := os.ReadFile(filepath.Join(dir, "server.pem"))
block, _ := pem.Decode(certPEM)
cert, _ := x509.ParseCertificate(block.Bytes)
hasBindIP := false
for _, ip := range cert.IPAddresses {
if ip.Equal(net.ParseIP("10.0.0.5")) {
hasBindIP = true
}
}
assert.True(t, hasBindIP)
})
t.Run("MissingCA", func(t *testing.T) {
dir := t.TempDir()
err := GenerateServerCertOnly(dir, "10.0.0.5")
assert.Error(t, err)
assert.Contains(t, err.Error(), "CA files not found")
})
}
// --- Directory creation ---
func TestCreateServiceDirectories_CreatesAll(t *testing.T) {
dir := t.TempDir()
CreateServiceDirectories(dir)
expected := []string{"images", "amis", "volumes", "state", "logs", "nats", "predastore", "viperblock", "hive"}
for _, name := range expected {
path := filepath.Join(dir, name)
info, err := os.Stat(path)
assert.NoError(t, err, "directory %s should exist", name)
if err == nil {
assert.True(t, info.IsDir())
}
}
}
func TestCreateServiceDirectories_Idempotent(t *testing.T) {
dir := t.TempDir()
CreateServiceDirectories(dir)
// Should not error on second call
CreateServiceDirectories(dir)
info, err := os.Stat(filepath.Join(dir, "images"))
assert.NoError(t, err)
assert.True(t, info.IsDir())
}
// --- Predastore multi-node config ---
func TestGenerateMultiNodePredastoreConfig_Success(t *testing.T) {
tmpl := `{{range .Nodes}}[[db]]
id = {{.ID}}
host = "{{.Host}}"
{{end}}`
nodes := []PredastoreNodeConfig{
{ID: 1, Host: "10.0.0.1"},
{ID: 2, Host: "10.0.0.2"},
{ID: 3, Host: "10.0.0.3"},
}
result, err := GenerateMultiNodePredastoreConfig(tmpl, nodes, "AK", "SK", "us-east-1")
require.NoError(t, err)
assert.Contains(t, result, `host = "10.0.0.1"`)
assert.Contains(t, result, `host = "10.0.0.3"`)
}
func TestGenerateMultiNodePredastoreConfig_MinimumNodes(t *testing.T) {
tmpl := "{{range .Nodes}}{{.ID}}{{end}}"
_, err := GenerateMultiNodePredastoreConfig(tmpl, []PredastoreNodeConfig{
{ID: 1, Host: "10.0.0.1"},
{ID: 2, Host: "10.0.0.2"},
}, "AK", "SK", "us-east-1")
assert.Error(t, err)
assert.Contains(t, err.Error(), "at least 3 nodes")
}
func TestGenerateMultiNodePredastoreConfig_InvalidTemplate(t *testing.T) {
_, err := GenerateMultiNodePredastoreConfig("{{.Unclosed", []PredastoreNodeConfig{
{ID: 1, Host: "a"}, {ID: 2, Host: "b"}, {ID: 3, Host: "c"},
}, "AK", "SK", "us-east-1")
assert.Error(t, err)
assert.Contains(t, err.Error(), "failed to parse")
}
// --- FindNodeIDByIP ---
func TestFindNodeIDByIP(t *testing.T) {
nodes := []PredastoreNodeConfig{
{ID: 1, Host: "10.0.0.1"},
{ID: 2, Host: "10.0.0.2"},
{ID: 3, Host: "10.0.0.3"},
}
assert.Equal(t, 2, FindNodeIDByIP(nodes, "10.0.0.2"))
assert.Equal(t, 0, FindNodeIDByIP(nodes, "10.0.0.99"))
assert.Equal(t, 0, FindNodeIDByIP(nil, "10.0.0.1"))
}
// --- ParsePredastoreNodeIDFromConfig ---
func TestParsePredastoreNodeIDFromConfig(t *testing.T) {
tomlContent := `
[[db]]
id = 1
host = "10.0.0.1"
[[db]]
id = 2
host = "10.0.0.2"
[[db]]
id = 3
host = "10.0.0.3"
`
assert.Equal(t, 2, ParsePredastoreNodeIDFromConfig(tomlContent, "10.0.0.2"))
assert.Equal(t, 0, ParsePredastoreNodeIDFromConfig(tomlContent, "10.0.0.99"))
assert.Equal(t, 0, ParsePredastoreNodeIDFromConfig("invalid toml {{{", "10.0.0.1"))
assert.Equal(t, 0, ParsePredastoreNodeIDFromConfig("", "10.0.0.1"))
}
// --- Integration: Full config generation flow ---
func TestGenerateConfigFile_HiveTomlTemplate(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "hive.toml")
tmpl := `version = "1.0"
epoch = 1
node = "{{.Node}}"
[nodes.{{.Node}}]
region = "{{.Region}}"
accesskey = "{{.AccessKey}}"
secretkey = "{{.SecretKey}}"
`
settings := ConfigSettings{
Node: "node1",
Region: "us-east-1",
AccessKey: "AKIATEST",
SecretKey: "SECRET",
}
require.NoError(t, GenerateConfigFile(path, tmpl, settings))
data, _ := os.ReadFile(path)
content := string(data)
assert.Contains(t, content, `node = "node1"`)
assert.Contains(t, content, `region = "us-east-1"`)
assert.Contains(t, content, fmt.Sprintf(`accesskey = "%s"`, settings.AccessKey))
}
+14 -7
View File
@@ -29,11 +29,11 @@ args["TagSpecification.2.Tag.1.Value"] = "Production"
*/
func QueryParamsToStruct(params map[string]string, out interface{}) error {
func QueryParamsToStruct(params map[string]string, out any) error {
v := reflect.ValueOf(out)
// Must be a pointer to a struct
if v.Kind() != reflect.Ptr {
if v.Kind() != reflect.Pointer {
return fmt.Errorf("out must be a pointer to a struct")
}
@@ -62,10 +62,17 @@ func setStructFields(v reflect.Value, params map[string]string, prefix string) e
// Check for locationName tag (AWS SDK uses this for query parameter names)
locationName := fieldType.Tag.Get("locationName")
// Try both the field name and locationName (if different)
// Try the field name, locationName, and title-cased locationName.
// AWS query params use title case (e.g. "ResourceId") but some SDK
// structs use camelCase locationName (e.g. "resourceId" in DeleteTagsInput
// vs "ResourceId" in CreateTagsInput).
queryKeys := []string{prefix + fieldName}
if locationName != "" && locationName != fieldName {
queryKeys = append(queryKeys, prefix+locationName)
titled := strings.ToUpper(locationName[:1]) + locationName[1:]
if titled != fieldName && titled != locationName {
queryKeys = append(queryKeys, prefix+titled)
}
}
// Check if this is a simple field (string, int, bool, etc.)
@@ -79,7 +86,7 @@ func setStructFields(v reflect.Value, params map[string]string, prefix string) e
}
// Handle pointer to struct (nested structs)
if field.Kind() == reflect.Ptr && field.Type().Elem().Kind() == reflect.Struct {
if field.Kind() == reflect.Pointer && field.Type().Elem().Kind() == reflect.Struct {
for _, queryKey := range queryKeys {
// Check if there are any params with this prefix
hasParams := false
@@ -115,7 +122,7 @@ func setStructFields(v reflect.Value, params map[string]string, prefix string) e
}
// Handle pointer to slice
if field.Kind() == reflect.Ptr && field.Type().Elem().Kind() == reflect.Slice {
if field.Kind() == reflect.Pointer && field.Type().Elem().Kind() == reflect.Slice {
for _, queryKey := range queryKeys {
sliceField := reflect.New(field.Type().Elem()).Elem()
if err := setSliceField(sliceField, params, queryKey); err != nil {
@@ -154,7 +161,7 @@ func setFieldValue(field reflect.Value, value string) error {
case reflect.String:
field.SetString(value)
case reflect.Ptr:
case reflect.Pointer:
// Handle pointer types
if field.IsNil() {
field.Set(reflect.New(field.Type().Elem()))
@@ -222,7 +229,7 @@ func setSliceField(field reflect.Value, params map[string]string, prefix string)
if val, ok := params[indexPrefix]; ok {
elem.SetString(val)
}
case reflect.Ptr:
case reflect.Pointer:
// Handle pointer to string
if elemType.Elem().Kind() == reflect.String {
if val, ok := params[indexPrefix]; ok {
+100
View File
@@ -379,6 +379,106 @@ func TestQueryParamsToStruct_CompleteRunInstancesExample(t *testing.T) {
assert.Equal(t, "Production", aws.StringValue(tagSpec.Tags[1].Value))
}
// --- Gap-filling tests for setFieldValue and setStructFields uncovered branches ---
func TestQueryParamsToStruct_ByteSliceBase64(t *testing.T) {
// ImportKeyPairInput.PublicKeyMaterial is []byte — tests base64 decoding path
args := map[string]string{
"KeyName": "my-imported-key",
"PublicKeyMaterial": "c3NoLXJzYSB0ZXN0LWtleS1kYXRh", // base64 of "ssh-rsa test-key-data"
}
input := &ec2.ImportKeyPairInput{}
err := QueryParamsToStruct(args, input)
assert.NoError(t, err)
assert.Equal(t, "my-imported-key", aws.StringValue(input.KeyName))
assert.Equal(t, "ssh-rsa test-key-data", string(input.PublicKeyMaterial))
}
func TestQueryParamsToStruct_ByteSliceRawText(t *testing.T) {
// Non-base64 text falls back to raw bytes
args := map[string]string{
"KeyName": "raw-key",
"PublicKeyMaterial": "not-valid-base64!!!",
}
input := &ec2.ImportKeyPairInput{}
err := QueryParamsToStruct(args, input)
assert.NoError(t, err)
assert.Equal(t, []byte("not-valid-base64!!!"), input.PublicKeyMaterial)
}
func TestQueryParamsToStruct_InvalidIntValue(t *testing.T) {
args := map[string]string{
"MinCount": "not-a-number",
}
input := &ec2.RunInstancesInput{}
err := QueryParamsToStruct(args, input)
assert.Error(t, err)
assert.Contains(t, err.Error(), "error setting field MinCount")
}
func TestQueryParamsToStruct_InvalidBoolValue(t *testing.T) {
args := map[string]string{
"InstanceId.1": "i-1234567890abcdef0",
"Force": "not-a-bool",
}
input := &ec2.StopInstancesInput{}
err := QueryParamsToStruct(args, input)
assert.Error(t, err)
assert.Contains(t, err.Error(), "error setting field Force")
}
func TestQueryParamsToStruct_LocationNameCamelCaseFallback(t *testing.T) {
// DeleteTagsInput.Resources has locationName:"resourceId"
// AWS query params use "ResourceId" (titled), testing the camelCase → titleCase fallback
args := map[string]string{
"ResourceId.1": "i-1234567890abcdef0",
"ResourceId.2": "i-0987654321fedcba0",
"Tag.1.Key": "Environment",
"Tag.1.Value": "Production",
}
input := &ec2.DeleteTagsInput{}
err := QueryParamsToStruct(args, input)
assert.NoError(t, err)
assert.Len(t, input.Resources, 2)
assert.Equal(t, "i-1234567890abcdef0", aws.StringValue(input.Resources[0]))
assert.Equal(t, "i-0987654321fedcba0", aws.StringValue(input.Resources[1]))
assert.Len(t, input.Tags, 1)
assert.Equal(t, "Environment", aws.StringValue(input.Tags[0].Key))
assert.Equal(t, "Production", aws.StringValue(input.Tags[0].Value))
}
func TestQueryParamsToStruct_LocationNameDryRun(t *testing.T) {
// DryRun has locationName:"dryRun" — tests that titled locationName matches "DryRun" query param
args := map[string]string{
"InstanceId.1": "i-1234567890abcdef0",
"DryRun": "true",
}
input := &ec2.TerminateInstancesInput{}
err := QueryParamsToStruct(args, input)
assert.NoError(t, err)
assert.Equal(t, true, aws.BoolValue(input.DryRun))
}
func TestQueryParamsToStruct_NonStructPointer(t *testing.T) {
args := map[string]string{"Key": "value"}
str := "hello"
err := QueryParamsToStruct(args, &str)
assert.Error(t, err)
assert.Contains(t, err.Error(), "must be a pointer to a struct")
}
func TestQueryParamsToStruct_BlockDeviceMappingsWithIOPS(t *testing.T) {
args := map[string]string{
"Action": "RunInstances",
+29
View File
@@ -367,6 +367,7 @@ var (
ErrorSecurityGroupLimitExceeded = "SecurityGroupLimitExceeded"
ErrorSecurityGroupsPerInstanceLimitExceeded = "SecurityGroupsPerInstanceLimitExceeded"
ErrorSecurityGroupsPerInterfaceLimitExceeded = "SecurityGroupsPerInterfaceLimitExceeded"
ErrorSerialConsoleSessionUnavailable = "SerialConsoleSessionUnavailable"
ErrorServerInternal = "ServerInternal"
ErrorServiceUnavailable = "ServiceUnavailable"
ErrorSignatureDoesNotMatch = "SignatureDoesNotMatch"
@@ -422,8 +423,27 @@ var (
ErrorVpnGatewayAttachmentLimitExceeded = "VpnGatewayAttachmentLimitExceeded"
ErrorVpnGatewayLimitExceeded = "VpnGatewayLimitExceeded"
ErrorZonesMismatched = "ZonesMismatched"
// IAM-specific error codes
ErrorIAMNoSuchEntity = "NoSuchEntity"
ErrorIAMEntityAlreadyExists = "EntityAlreadyExists"
ErrorIAMDeleteConflict = "DeleteConflict"
ErrorIAMLimitExceeded = "LimitExceeded"
ErrorIAMInvalidInput = ErrorInvalidInput // alias for IAM usage
ErrorIAMMalformedPolicyDocument = "MalformedPolicyDocument"
ErrorAccessDenied = "AccessDenied"
)
// ValidErrorCode returns the error code if it exists in ErrorLookup,
// otherwise returns ErrorServerInternal. Use this to sanitize error strings
// before sending them to clients.
func ValidErrorCode(code string) string {
if _, ok := ErrorLookup[code]; ok {
return code
}
return ErrorServerInternal
}
var ErrorLookup = map[string]ErrorMessage{
ErrorAccountDisabled: {HTTPCode: 400, Message: "The functionality you have requested has been administratively disabled for this account."},
ErrorActiveVpcPeeringConnectionPerVpcLimitExceeded: {HTTPCode: 400, Message: "You've reached the limit on the number of active VPC peering connections you can have for the specified VPC."},
@@ -786,6 +806,7 @@ var ErrorLookup = map[string]ErrorMessage{
ErrorSecurityGroupLimitExceeded: {HTTPCode: 400, Message: "You've reached the limit on the number of security groups that you can create, or that you can assign to an instance."},
ErrorSecurityGroupsPerInstanceLimitExceeded: {HTTPCode: 400, Message: "You've reached the limit on the number of security groups that you can assign to an instance. For more information, see Amazon EC2 security groups."},
ErrorSecurityGroupsPerInterfaceLimitExceeded: {HTTPCode: 400, Message: "You've reached the limit on the number of security groups you can associate with the specified network interface. For more information, see Amazon VPC quotas."},
ErrorSerialConsoleSessionUnavailable: {HTTPCode: 403, Message: "The serial console access is not enabled for this account. Use EnableSerialConsoleAccess to enable access."},
ErrorServerInternal: {HTTPCode: 500, Message: "An internal error has occurred. Retry your request, but if the problem persists, contact us with details by posting a message on AWS re:Post."},
ErrorServiceUnavailable: {HTTPCode: 503, Message: "The request has failed due to a temporary failure of the server."},
ErrorSignatureDoesNotMatch: {HTTPCode: 403, Message: "The request signature that Amazon has does not match the signature that you provided. Check your AWS credentials and signing method."},
@@ -841,4 +862,12 @@ var ErrorLookup = map[string]ErrorMessage{
ErrorVpnGatewayAttachmentLimitExceeded: {HTTPCode: 400, Message: "You've reached the limit on the number of VPCs that can be attached to the specified virtual private gateway."},
ErrorVpnGatewayLimitExceeded: {HTTPCode: 400, Message: "You've reached the limit on the number of virtual private gateways that you can create. For more information about limits, see Amazon VPC quotas."},
ErrorZonesMismatched: {HTTPCode: 400, Message: "The Availability Zone for the instance does not match that of the Dedicated Host"},
// IAM error codes
ErrorIAMNoSuchEntity: {HTTPCode: 404, Message: "The request was rejected because it referenced a resource entity that does not exist."},
ErrorIAMEntityAlreadyExists: {HTTPCode: 409, Message: "The request was rejected because it attempted to create a resource that already exists."},
ErrorIAMDeleteConflict: {HTTPCode: 409, Message: "The request was rejected because it attempted to delete a resource that has attached subordinate entities."},
ErrorIAMLimitExceeded: {HTTPCode: 409, Message: "The request was rejected because it attempted to create resources beyond the current AWS account limits."},
ErrorIAMMalformedPolicyDocument: {HTTPCode: 400, Message: "The policy document is malformed."},
ErrorAccessDenied: {HTTPCode: 403, Message: "User is not authorized to perform this action."},
}
+31
View File
@@ -369,6 +369,7 @@ func TestErrorLookup(t *testing.T) {
{code: "SecurityGroupLimitExceeded", http: 400, message: "You've reached the limit on the number of security groups that you can create, or that you can assign to an instance."},
{code: "SecurityGroupsPerInstanceLimitExceeded", http: 400, message: "You've reached the limit on the number of security groups that you can assign to an instance. For more information, see Amazon EC2 security groups."},
{code: "SecurityGroupsPerInterfaceLimitExceeded", http: 400, message: "You've reached the limit on the number of security groups you can associate with the specified network interface. For more information, see Amazon VPC quotas."},
{code: "SerialConsoleSessionUnavailable", http: 403, message: "The serial console access is not enabled for this account. Use EnableSerialConsoleAccess to enable access."},
{code: "ServerInternal", http: 500, message: "An internal error has occurred. Retry your request, but if the problem persists, contact us with details by posting a message on AWS re:Post."},
{code: "ServiceUnavailable", http: 503, message: "The request has failed due to a temporary failure of the server."},
{code: "SignatureDoesNotMatch", http: 403, message: "The request signature that Amazon has does not match the signature that you provided. Check your AWS credentials and signing method."},
@@ -424,6 +425,14 @@ func TestErrorLookup(t *testing.T) {
{code: "VpnGatewayAttachmentLimitExceeded", http: 400, message: "You've reached the limit on the number of VPCs that can be attached to the specified virtual private gateway."},
{code: "VpnGatewayLimitExceeded", http: 400, message: "You've reached the limit on the number of virtual private gateways that you can create. For more information about limits, see Amazon VPC quotas."},
{code: "ZonesMismatched", http: 400, message: "The Availability Zone for the instance does not match that of the Dedicated Host"},
// IAM error codes
{code: "NoSuchEntity", http: 404, message: "The request was rejected because it referenced a resource entity that does not exist."},
{code: "EntityAlreadyExists", http: 409, message: "The request was rejected because it attempted to create a resource that already exists."},
{code: "DeleteConflict", http: 409, message: "The request was rejected because it attempted to delete a resource that has attached subordinate entities."},
{code: "LimitExceeded", http: 409, message: "The request was rejected because it attempted to create resources beyond the current AWS account limits."},
{code: "MalformedPolicyDocument", http: 400, message: "The policy document is malformed."},
{code: "AccessDenied", http: 403, message: "User is not authorized to perform this action."},
}
if len(ErrorLookup) != len(expected) {
@@ -443,3 +452,25 @@ func TestErrorLookup(t *testing.T) {
}
}
}
func TestValidErrorCode(t *testing.T) {
tests := []struct {
name string
code string
want string
}{
{name: "known error code", code: ErrorAuthFailure, want: ErrorAuthFailure},
{name: "another known code", code: ErrorInvalidParameterValue, want: ErrorInvalidParameterValue},
{name: "unknown code returns ServerInternal", code: "CompletelyMadeUp", want: ErrorServerInternal},
{name: "empty string returns ServerInternal", code: "", want: ErrorServerInternal},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := ValidErrorCode(tt.code)
if got != tt.want {
t.Errorf("ValidErrorCode(%q) = %q, want %q", tt.code, got, tt.want)
}
})
}
}
-16
View File
@@ -1,16 +0,0 @@
package awssmithy
type FilterList struct {
Name string
Values []string
}
type DescribeInstancesRequest struct {
InstanceIds []string `xml:"InstanceId"`
DryRun bool `xml:"DryRun"`
Filters []FilterList `xml:"Filter"`
NextToken string `xml:"nextToken"`
MaxResults int `xml:"maxResults"`
}
+72 -135
View File
@@ -1,29 +1,40 @@
package config
import (
"encoding/json"
"fmt"
"log/slog"
"os"
"sync"
"slices"
"github.com/nats-io/nats.go"
"github.com/spf13/viper"
)
type ClusterConfig struct {
Epoch uint64 `mapstructure:"epoch"` // bump when leader commits changes
Node string `mapstructure:"node"` // my node name
Version string `mapstructure:"version"` // hive version
Nodes map[string]Config `mapstructure:"nodes"` // full config for every node
}
// Config holds all configuration for the application
type Config struct {
// Node config
Node string `mapstructure:"node"`
Host string `mapstructure:"host"` // Unique hostname or IP of this node
Region string `mapstructure:"region"`
AZ string `mapstructure:"az"`
DataDir string `mapstructure:"data_dir"`
Services []string `mapstructure:"services"` // Which services this node runs locally
Daemon DaemonConfig `mapstructure:"daemon"`
NATS NATSConfig `mapstructure:"nats"`
Predastore PredastoreConfig `mapstructure:"predastore"`
Viperblock ViperblockConfig `mapstructure:"viperblock"`
AWSGW AWSGWConfig `mapstructure:"awsgw"`
VPCD VPCDConfig `mapstructure:"vpcd"`
// Authentication
AccessKey string `mapstructure:"access_key"`
SecretKey string `mapstructure:"secret_key"`
Host string `mapstructure:"host"`
BaseDir string `mapstructure:"base_dir"`
WalDir string `mapstructure:"wal_dir"`
BaseDir string `mapstructure:"base_dir"`
WalDir string `mapstructure:"wal_dir"`
}
type AWSGWConfig struct {
@@ -33,23 +44,35 @@ type AWSGWConfig struct {
Config string `mapstructure:"config"`
Debug bool `mapstructure:"debug"`
ExpectedNodes int `mapstructure:"expected_nodes"`
ExpectedNodes int `mapstructure:"expected_nodes"` // TODO: Replace with root cluster config
}
type ViperblockConfig struct {
ShardWAL *bool `mapstructure:"shardwal"` // Enable sharded WAL (default false when nil)
}
// VPCDConfig holds the VPC daemon (vpcd) configuration.
type VPCDConfig struct {
OVNNBAddr string `mapstructure:"ovn_nb_addr"` // OVN Northbound DB address (e.g., "tcp:127.0.0.1:6641")
OVNSBAddr string `mapstructure:"ovn_sb_addr"` // OVN Southbound DB address (e.g., "tcp:127.0.0.1:6642")
}
type PredastoreConfig struct {
Host string `mapstructure:"host"`
Bucket string `mapstructure:"bucket"`
Region string `mapstructure:"region"`
AccessKey string `mapstructure:"access_key"`
SecretKey string `mapstructure:"secret_key"`
AccessKey string `mapstructure:"accesskey"`
SecretKey string `mapstructure:"secretkey"`
BaseDir string `mapstructure:"base_dir"`
NodeID int `mapstructure:"node_id"`
}
// DaemonConfig holds the daemon configuration
type DaemonConfig struct {
Host string `mapstructure:"host"`
TLSKey string `mapstructure:"tlskey"`
TLSCert string `mapstructure:"tlscert"`
Host string `mapstructure:"host"`
TLSKey string `mapstructure:"tlskey"`
TLSCert string `mapstructure:"tlscert"`
DevNetworking bool `mapstructure:"dev_networking"` // VPC instances get both TAP + hostfwd for SSH dev access
}
// NATSConfig holds the NATS configuration
@@ -69,122 +92,51 @@ type NATSSub struct {
Subject string `mapstructure:"subject"`
}
type EBSRequests struct {
Requests []EBSRequest `mapstructure:"ebs_requests"`
Mu sync.Mutex `json:"-"`
}
type EBSRequest struct {
Name string
VolType string
Boot bool
EFI bool
CloudInit bool
NBDURI string
}
type EBSMountResponse struct {
URI string
Mounted bool
Error string
}
type EBSUnMountResponse struct {
Volume string
Mounted bool
Error string
}
type EBSDeleteRequest struct {
Volume string
}
type EBSDeleteResponse struct {
Volume string
Success bool
Error string
}
// EC2, TODO: Move to vm.go or more applicable place
type EC2Response struct {
InstanceID string
Hostname string
Success bool
Status string
Error string
}
type EC2DescribeRequest struct {
InstanceID string
}
type EC2DescribeResponse struct {
InstanceID string
Hostname string
Status string
Error string
}
type EC2StartInstancesRequest struct {
InstanceID string
}
type EC2StartInstancesResponse struct {
InstanceID string
Status string
Error string
}
// TODO: Make a generic function for the response
func (ec2StartInstanceResponse EC2StartInstancesResponse) Respond(msg *nats.Msg) {
response, err := json.Marshal(ec2StartInstanceResponse)
if err != nil {
slog.Error("Failed to marshal response", "err", err)
return
// NodeBaseDir returns the BaseDir for the current node, or "" if the config
// is nil, the node name is unset, or the node is not found in the Nodes map.
func (cc *ClusterConfig) NodeBaseDir() string {
if cc == nil || cc.Node == "" {
slog.Warn("NodeBaseDir: no config or node name set, using global PID path")
return ""
}
msg.Respond(response)
}
func (ec2DescribeResponse EC2DescribeResponse) Respond(msg *nats.Msg) {
response, err := json.Marshal(ec2DescribeResponse)
if err != nil {
slog.Error("Failed to marshal response", "err", err)
return
node, ok := cc.Nodes[cc.Node]
if !ok {
slog.Error("NodeBaseDir: node not found in config", "node", cc.Node)
return ""
}
msg.Respond(response)
if node.BaseDir == "" {
slog.Warn("NodeBaseDir: BaseDir is empty for node, using global PID path", "node", cc.Node)
}
return node.BaseDir
}
func (ec2Response EC2Response) Respond(msg *nats.Msg) {
// AllServices is the default service list when Services is empty (backward compat).
var AllServices = []string{"nats", "predastore", "viperblock", "daemon", "awsgw", "vpcd", "ui"}
response, err := json.Marshal(ec2Response)
if err != nil {
slog.Error("Failed to marshal response", "err", err)
return
// HasService reports whether the node runs the named service.
// An empty Services list means all services (backward compat).
func (c Config) HasService(name string) bool {
services := c.Services
if len(services) == 0 {
services = AllServices
}
return slices.Contains(services, name)
}
msg.Respond(response)
// GetServices returns the configured service list, defaulting to AllServices.
func (c Config) GetServices() []string {
if len(c.Services) == 0 {
return AllServices
}
return c.Services
}
// LoadConfig loads the configuration from file and environment variables
func LoadConfig(configPath string) (*Config, error) {
func LoadConfig(configPath string) (*ClusterConfig, error) {
// Set environment variable prefix
viper.SetEnvPrefix("HIVE")
viper.AutomaticEnv()
// Set default values
//viper.SetDefault("host", "https://localhost:8443/")
//viper.SetDefault("base_dir", "/tmp/vb/")
//viper.SetDefault("daemon.host", "0.0.0.0:4432")
//viper.SetDefault("nats.host", "0.0.0.0:4222")
//viper.SetDefault("nats.sub.subject", "ec2.>")
// Try to load config file if it exists
if configPath != "" {
// Check if file exists
@@ -195,33 +147,18 @@ func LoadConfig(configPath string) (*Config, error) {
if err := viper.ReadInConfig(); err != nil {
return nil, fmt.Errorf("error reading config file: %w", err)
}
fmt.Fprintf(os.Stderr, "Using config file: %s\n", viper.ConfigFileUsed())
//fmt.Fprintf(os.Stderr, "Using config file: %s\n", viper.ConfigFileUsed())
} else {
fmt.Fprintf(os.Stderr, "Config file not found: %s, using environment variables and defaults\n", configPath)
}
}
// Create config struct
var config Config
var config ClusterConfig
if err := viper.Unmarshal(&config); err != nil {
return nil, fmt.Errorf("error unmarshaling config: %w", err)
}
// Validate required fields (can be overwritten by env vars)
/*
if config.NATS.Host == "" {
return nil, fmt.Errorf("NATS host is required")
}
if config.AccessKey == "" {
return nil, fmt.Errorf("access key is required")
}
if config.SecretKey == "" {
return nil, fmt.Errorf("secret key is required")
}
*/
// fmt.Println("Config: ", config)
return &config, nil
}
+371
View File
@@ -0,0 +1,371 @@
package config
import (
"os"
"path/filepath"
"testing"
"github.com/spf13/viper"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func resetViper(t *testing.T) {
t.Cleanup(func() { viper.Reset() })
}
func TestLoadConfig_ValidTOMLFile(t *testing.T) {
resetViper(t)
dir := t.TempDir()
path := filepath.Join(dir, "hive.toml")
toml := `
version = "1.0"
epoch = 1
node = "node1"
[nodes.node1]
node = "node1"
region = "us-east-1"
az = "us-east-1a"
[nodes.node1.daemon]
host = "127.0.0.1:8080"
[nodes.node1.nats]
host = "127.0.0.1:4222"
[nodes.node1.nats.acl]
token = "nats_testtoken"
[nodes.node1.predastore]
host = "127.0.0.1:8443"
bucket = "predastore"
region = "us-east-1"
`
require.NoError(t, os.WriteFile(path, []byte(toml), 0600))
cfg, err := LoadConfig(path)
require.NoError(t, err)
require.NotNil(t, cfg)
assert.Equal(t, uint64(1), cfg.Epoch)
assert.Equal(t, "node1", cfg.Node)
assert.Equal(t, "1.0", cfg.Version)
node, ok := cfg.Nodes["node1"]
require.True(t, ok, "node1 should exist in Nodes map")
assert.Equal(t, "us-east-1", node.Region)
assert.Equal(t, "us-east-1a", node.AZ)
assert.Equal(t, "127.0.0.1:8080", node.Daemon.Host)
assert.Equal(t, "127.0.0.1:4222", node.NATS.Host)
assert.Equal(t, "nats_testtoken", node.NATS.ACL.Token)
assert.Equal(t, "127.0.0.1:8443", node.Predastore.Host)
assert.Equal(t, "predastore", node.Predastore.Bucket)
}
func TestLoadConfig_MultipleNodes(t *testing.T) {
resetViper(t)
dir := t.TempDir()
path := filepath.Join(dir, "hive.toml")
toml := `
epoch = 2
node = "leader"
version = "2.0"
[nodes.leader]
region = "us-east-1"
[nodes.follower]
region = "us-west-2"
`
require.NoError(t, os.WriteFile(path, []byte(toml), 0600))
cfg, err := LoadConfig(path)
require.NoError(t, err)
assert.Len(t, cfg.Nodes, 2)
assert.Equal(t, "us-east-1", cfg.Nodes["leader"].Region)
assert.Equal(t, "us-west-2", cfg.Nodes["follower"].Region)
}
func TestLoadConfig_EmptyConfigPath(t *testing.T) {
resetViper(t)
cfg, err := LoadConfig("")
require.NoError(t, err)
require.NotNil(t, cfg)
// All zero values
assert.Equal(t, uint64(0), cfg.Epoch)
assert.Empty(t, cfg.Node)
}
func TestLoadConfig_NonexistentFile(t *testing.T) {
resetViper(t)
cfg, err := LoadConfig("/tmp/nonexistent-hive-config-test-12345.toml")
// Not an error - falls through to defaults
require.NoError(t, err)
require.NotNil(t, cfg)
}
func TestLoadConfig_MalformedTOML(t *testing.T) {
resetViper(t)
dir := t.TempDir()
path := filepath.Join(dir, "bad.toml")
require.NoError(t, os.WriteFile(path, []byte("this is not valid toml {{{"), 0600))
cfg, err := LoadConfig(path)
assert.Error(t, err)
assert.Nil(t, cfg)
assert.Contains(t, err.Error(), "error reading config file")
}
func TestLoadConfig_PartialConfig(t *testing.T) {
resetViper(t)
dir := t.TempDir()
path := filepath.Join(dir, "partial.toml")
require.NoError(t, os.WriteFile(path, []byte(`node = "partial-node"
epoch = 5
`), 0600))
cfg, err := LoadConfig(path)
require.NoError(t, err)
assert.Equal(t, "partial-node", cfg.Node)
assert.Equal(t, uint64(5), cfg.Epoch)
assert.Empty(t, cfg.Version)
assert.Nil(t, cfg.Nodes)
}
func TestLoadConfig_EnvVarOverrideWithFile(t *testing.T) {
resetViper(t)
dir := t.TempDir()
path := filepath.Join(dir, "hive.toml")
// Viper's AutomaticEnv only works for keys Viper already knows about
// (from a config file or explicit BindEnv). Provide a minimal config
// so Viper registers the "epoch" key, then override via env.
require.NoError(t, os.WriteFile(path, []byte(`epoch = 1
node = "file-node"
`), 0600))
t.Setenv("HIVE_NODE", "env-node")
cfg, err := LoadConfig(path)
require.NoError(t, err)
// Env vars override file values for keys Viper knows about
assert.Equal(t, "env-node", cfg.Node)
}
func TestLoadConfig_NestedStructParsing(t *testing.T) {
resetViper(t)
dir := t.TempDir()
path := filepath.Join(dir, "nested.toml")
toml := `
node = "n1"
[nodes.n1]
region = "ap-southeast-2"
[nodes.n1.daemon]
host = "0.0.0.0:8080"
tlskey = "server.key"
tlscert = "server.pem"
[nodes.n1.nats]
host = "0.0.0.0:4222"
[nodes.n1.nats.acl]
token = "secret-token"
[nodes.n1.nats.sub]
subject = "test-subject"
[nodes.n1.predastore]
host = "0.0.0.0:8443"
bucket = "mybucket"
region = "ap-southeast-2"
accesskey = "AK"
secretkey = "SK"
base_dir = "/data"
node_id = 1
[nodes.n1.awsgw]
host = "0.0.0.0:9999"
debug = true
expected_nodes = 3
`
require.NoError(t, os.WriteFile(path, []byte(toml), 0600))
cfg, err := LoadConfig(path)
require.NoError(t, err)
n := cfg.Nodes["n1"]
assert.Equal(t, "0.0.0.0:8080", n.Daemon.Host)
assert.Equal(t, "server.key", n.Daemon.TLSKey)
assert.Equal(t, "server.pem", n.Daemon.TLSCert)
assert.Equal(t, "0.0.0.0:4222", n.NATS.Host)
assert.Equal(t, "secret-token", n.NATS.ACL.Token)
assert.Equal(t, "test-subject", n.NATS.Sub.Subject)
assert.Equal(t, "0.0.0.0:8443", n.Predastore.Host)
assert.Equal(t, "mybucket", n.Predastore.Bucket)
assert.Equal(t, "AK", n.Predastore.AccessKey)
assert.Equal(t, "/data", n.Predastore.BaseDir)
assert.Equal(t, 1, n.Predastore.NodeID)
assert.Equal(t, "0.0.0.0:9999", n.AWSGW.Host)
assert.True(t, n.AWSGW.Debug)
assert.Equal(t, 3, n.AWSGW.ExpectedNodes)
}
// Tests for HasService / GetServices
func TestHasService_ExplicitList(t *testing.T) {
c := Config{Services: []string{"nats", "daemon"}}
assert.True(t, c.HasService("nats"))
assert.True(t, c.HasService("daemon"))
assert.False(t, c.HasService("predastore"))
assert.False(t, c.HasService("viperblock"))
assert.False(t, c.HasService("ui"))
}
func TestHasService_EmptyListBackwardCompat(t *testing.T) {
c := Config{} // no Services set
// Empty list means all services
for _, svc := range AllServices {
assert.True(t, c.HasService(svc), "expected %s to be available with empty list", svc)
}
}
func TestHasService_UnknownService(t *testing.T) {
c := Config{Services: []string{"nats"}}
assert.False(t, c.HasService("unknown"))
}
func TestGetServices_DefaultsToAll(t *testing.T) {
c := Config{}
services := c.GetServices()
assert.Equal(t, AllServices, services)
}
func TestGetServices_ExplicitList(t *testing.T) {
c := Config{Services: []string{"nats", "predastore"}}
services := c.GetServices()
assert.Equal(t, []string{"nats", "predastore"}, services)
}
// Tests for NodeBaseDir
func TestNodeBaseDir_HappyPath(t *testing.T) {
cc := &ClusterConfig{
Node: "node1",
Nodes: map[string]Config{
"node1": {BaseDir: "/data/node1"},
},
}
assert.Equal(t, "/data/node1", cc.NodeBaseDir())
}
func TestNodeBaseDir_NilConfig(t *testing.T) {
var cc *ClusterConfig
assert.Equal(t, "", cc.NodeBaseDir())
}
func TestNodeBaseDir_EmptyNode(t *testing.T) {
cc := &ClusterConfig{
Node: "",
Nodes: map[string]Config{
"node1": {BaseDir: "/data/node1"},
},
}
assert.Equal(t, "", cc.NodeBaseDir())
}
func TestNodeBaseDir_NodeNotInMap(t *testing.T) {
cc := &ClusterConfig{
Node: "missing",
Nodes: map[string]Config{
"node1": {BaseDir: "/data/node1"},
},
}
assert.Equal(t, "", cc.NodeBaseDir())
}
func TestNodeBaseDir_EmptyBaseDir(t *testing.T) {
cc := &ClusterConfig{
Node: "node1",
Nodes: map[string]Config{
"node1": {BaseDir: ""},
},
}
assert.Equal(t, "", cc.NodeBaseDir())
}
// Tests for ViperblockConfig
func TestLoadConfig_ViperblockShardWAL_Explicit(t *testing.T) {
resetViper(t)
dir := t.TempDir()
path := filepath.Join(dir, "hive.toml")
toml := `
node = "n1"
[nodes.n1]
region = "us-east-1"
[nodes.n1.viperblock]
shardwal = false
`
require.NoError(t, os.WriteFile(path, []byte(toml), 0600))
cfg, err := LoadConfig(path)
require.NoError(t, err)
n := cfg.Nodes["n1"]
require.NotNil(t, n.Viperblock.ShardWAL, "ShardWAL should be set when explicitly configured")
assert.False(t, *n.Viperblock.ShardWAL)
}
func TestLoadConfig_ViperblockShardWAL_DefaultNil(t *testing.T) {
resetViper(t)
dir := t.TempDir()
path := filepath.Join(dir, "hive.toml")
toml := `
node = "n1"
[nodes.n1]
region = "us-east-1"
`
require.NoError(t, os.WriteFile(path, []byte(toml), 0600))
cfg, err := LoadConfig(path)
require.NoError(t, err)
n := cfg.Nodes["n1"]
assert.Nil(t, n.Viperblock.ShardWAL, "ShardWAL should be nil when not configured (defaults to false in service)")
}
func TestLoadConfig_ViperblockShardWAL_True(t *testing.T) {
resetViper(t)
dir := t.TempDir()
path := filepath.Join(dir, "hive.toml")
toml := `
node = "n1"
[nodes.n1]
region = "us-east-1"
[nodes.n1.viperblock]
shardwal = true
`
require.NoError(t, os.WriteFile(path, []byte(toml), 0600))
cfg, err := LoadConfig(path)
require.NoError(t, err)
n := cfg.Nodes["n1"]
require.NotNil(t, n.Viperblock.ShardWAL)
assert.True(t, *n.Viperblock.ShardWAL)
}
-420
View File
@@ -1,420 +0,0 @@
# Daemon Testing Guide
## Overview
The `daemon_test.go` file provides comprehensive unit tests for the daemon's EC2 launch functionality, focusing on message handling, validation, and resource management.
## Test Structure
### Core Test Components
1. **NATS Server Setup** (`startTestNATSServer`)
- Creates an embedded NATS server for testing
- Uses port `-1` for automatic port allocation
- Returns server instance and connection URL
- Automatically starts and waits for readiness
2. **Daemon Initialization** (`createTestDaemon`)
- Creates a minimal daemon configuration for testing
- Sets up temporary directories for test data
- Establishes NATS connection
- Includes automatic cleanup via `t.Cleanup()`
3. **Test Fixtures** (`createValidRunInstancesInput`)
- Provides valid `ec2.RunInstancesInput` structures
- Pre-configured with sensible defaults
- Easy to modify for specific test scenarios
### Test Suites
#### 1. TestHandleEC2Launch_MessageParsing
Tests the daemon's ability to parse and validate incoming NATS messages:
- ✅ Valid RunInstancesInput processing
- ✅ Invalid instance type handling
- ✅ Missing required fields (ImageId)
- ✅ Invalid parameter values (MinCount = 0)
- ✅ Malformed JSON handling
**Expected Behaviors:**
- Valid inputs should return a Reservation with pending instance
- Invalid inputs should return error payloads (not transport errors)
- All responses should be properly formatted JSON
#### 2. TestHandleEC2Launch_ResourceManagement
Tests resource allocation and instance type validation:
- ✅ Valid instance types (t3.micro, t3.nano)
- ✅ Invalid instance types (t99.invalid)
- ✅ Resource availability checking
- ✅ Allocation/deallocation cycles
#### 3. TestDaemon_Initialization
Tests daemon initialization logic:
- ✅ Configuration loading
- ✅ Resource manager creation
- ✅ Instance storage initialization
#### 4. TestResourceManager
Tests the resource management subsystem:
- ✅ System resource detection
- ✅ Instance type allocation
- ✅ Resource deallocation
- ✅ Capacity validation
## Running Tests
### Basic Test Execution
```bash
# Run all daemon tests
go test -v ./hive/daemon/
# Run specific test
go test -v ./hive/daemon/ -run TestHandleEC2Launch_MessageParsing
# Run with race detection
go test -race -v ./hive/daemon/
# Run with coverage
go test -cover -v ./hive/daemon/
```
### Skipping Integration Tests
The tests are designed to skip integration components when running on macOS or when dependencies are unavailable:
```bash
# Skip integration tests that require viperblock/nbdkit
SKIP_INTEGRATION=1 go test -v ./hive/daemon/
```
## macOS Testing Limitations
### Current Limitations
The full integration test suite has limitations on macOS due to missing dependencies:
1. **Viperblock** - Not yet ported to macOS
- NBD (Network Block Device) support not available
- Volume mounting operations will fail
2. **NBDkit** - Linux-specific tool
- Requires kernel NBD support
- Not available natively on macOS
3. **QEMU/KVM** - Limited on macOS
- KVM acceleration not available (Linux-only)
- Can use QEMU with TCG (slow emulation)
### What Works on macOS
✅ **Message-driven architecture testing:**
- NATS message parsing and routing
- Request/response validation
- Error handling and payload generation
- Resource manager logic
✅ **Unit tests without infrastructure:**
- Input validation (`ValidateRunInstancesInput`)
- JSON marshaling/unmarshaling
- Error code generation
- Instance ID generation
### What Requires Linux
❌ **Full integration testing:**
- Volume creation and mounting (requires viperblock)
- NBD device attachment (requires nbdkit)
- VM launch with KVM acceleration
- EBS volume operations
- Cloud-init ISO creation and mounting
### Recommendations for macOS Development
#### Option 1: Mock Integration Points
Create mock implementations for macOS testing:
```go
// daemon_test_helpers.go
// +build darwin
// Mock viperblock operations for macOS
type MockViperblock struct {
// Mock implementation
}
func (m *MockViperblock) Mount(volumeId string) error {
// Return mock NBD URI
return nil
}
```
#### Option 2: Use Docker Linux Container
Run tests in a Linux container with full dependencies:
```bash
# Build Linux test container
docker build -t hive-test -f Dockerfile.test .
# Run tests in container
docker run --rm \
--privileged \
-v $(pwd):/workspace \
hive-test \
go test -v ./hive/daemon/
```
**Dockerfile.test example:**
```dockerfile
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y \
golang-1.21 \
qemu-kvm \
nbdkit \
build-essential
WORKDIR /workspace
```
#### Option 3: GitHub Actions / CI Pipeline
Use CI for full integration tests:
```yaml
# .github/workflows/test.yml
name: Integration Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-go@v4
- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get install -y nbdkit qemu-kvm
- name: Run tests
run: make test
```
#### Option 4: Virtual Linux VM on macOS
Use a Linux VM for development testing:
```bash
# Using UTM or Multipass
multipass launch --name hive-dev --cpus 4 --memory 8G --disk 20G
multipass shell hive-dev
# Inside VM
cd /path/to/hive
make test
```
### Hybrid Testing Strategy
**On macOS (Unit Tests):**
```bash
# Test message handling, validation, resource management
SKIP_INTEGRATION=1 go test -v ./hive/daemon/
```
**On Linux (Full Integration):**
```bash
# Test complete EC2 launch flow with viperblock/nbdkit
go test -v ./hive/daemon/
```
## Test Coverage Goals
### Current Coverage
- ✅ Message parsing and validation
- ✅ Resource allocation logic
- ✅ Error handling and responses
- ✅ NATS integration (pub/sub)
### Additional Tests Needed (Linux)
#### Volume Management Tests
```go
func TestHandleEC2Launch_VolumeCreation(t *testing.T) {
// Test AMI volume cloning
// Test EFI partition creation
// Test cloud-init ISO generation
}
```
#### NBD Mount Tests
```go
func TestMountVolumes(t *testing.T) {
// Test NBD URI generation
// Test mount request/response
// Test mount failures
}
```
#### VM Lifecycle Tests
```go
func TestLaunchInstance(t *testing.T) {
// Test QEMU process startup
// Test QMP socket connection
// Test instance state transitions
}
```
#### End-to-End Tests
```go
func TestEC2Launch_EndToEnd(t *testing.T) {
// Full flow: message -> volumes -> VM -> running
// Test with multiple instance types
// Test concurrent launches
}
```
## Mocking Strategy for macOS
### Mock EBS Mount Service
```go
// Create a mock EBS mount responder
func startMockEBSService(t *testing.T, nc *nats.Conn) {
nc.QueueSubscribe("ebs.mount", "mock-ebs", func(msg *nats.Msg) {
var req config.EBSRequest
json.Unmarshal(msg.Data, &req)
resp := config.EBSMountResponse{
URI: fmt.Sprintf("nbd://localhost:10809/%s", req.Name),
Mounted: true,
Error: "",
}
data, _ := json.Marshal(resp)
msg.Respond(data)
})
}
```
### Mock S3 Client
```go
// Mock S3 client for SSH key retrieval
type MockS3Client struct {
keys map[string]string
}
func (m *MockS3Client) Read(path string) ([]byte, error) {
if key, ok := m.keys[path]; ok {
return []byte(key), nil
}
return nil, fmt.Errorf("key not found")
}
```
### Example Mock Test
```go
func TestHandleEC2Launch_WithMocks(t *testing.T) {
if runtime.GOOS == "darwin" {
// Run with mocks on macOS
ns, natsURL := startTestNATSServer(t)
defer ns.Shutdown()
nc, _ := nats.Connect(natsURL)
defer nc.Close()
// Start mock services
startMockEBSService(t, nc)
startMockS3Service(t, nc)
// Run test...
} else {
// Run with real services on Linux
// ...
}
}
```
## Debugging Tests
### Enable Verbose Logging
```bash
# Run tests with verbose output
go test -v ./hive/daemon/ 2>&1 | tee test.log
```
### Debug NATS Messages
```go
// Add debug subscriber to see all messages
nc.Subscribe(">", func(msg *nats.Msg) {
t.Logf("NATS [%s]: %s", msg.Subject, string(msg.Data))
})
```
### Common Issues
**Issue: "NATS server failed to start"**
- Port already in use
- Use `-1` for auto port allocation
- Check firewall settings
**Issue: "Request timeout"**
- Increase timeout duration
- Check NATS connectivity
- Verify handler subscription
**Issue: "Test fails on Linux but works on macOS"**
- Missing dependencies (nbdkit, QEMU)
- Permission issues (KVM access)
- File system differences
## Next Steps
1. ✅ Create basic message handling tests (DONE)
2. ⏳ Add mock services for macOS testing
3. ⏳ Create Linux CI pipeline for full integration tests
4. ⏳ Add Docker-based test environment
5. ⏳ Implement volume lifecycle tests (Linux only)
6. ⏳ Add concurrent launch stress tests
7. ⏳ Create performance benchmarks
## Contributing
When adding new tests:
1. **Check platform compatibility:**
```go
if runtime.GOOS == "darwin" && needsLinux {
t.Skip("Test requires Linux")
}
```
2. **Use table-driven tests:**
```go
tests := []struct {
name string
input interface{}
want interface{}
}{ /* ... */ }
```
3. **Clean up resources:**
```go
t.Cleanup(func() {
// Cleanup code
})
```
4. **Document dependencies:**
- Add comments explaining Linux-only requirements
- Use build tags where appropriate
- Provide alternative test paths for macOS
-411
View File
@@ -1,411 +0,0 @@
# Daemon Unit Test Summary
## Created Files
1. **`daemon_test.go`** - Comprehensive unit test suite for daemon EC2 launch functionality
2. **`TESTING.md`** - Detailed testing guide with macOS limitations and strategies
3. **`TEST_SUMMARY.md`** (this file) - Summary of test implementation
## Test Framework Components
### 1. NATS Test Server (`startTestNATSServer`)
```go
func startTestNATSServer(t *testing.T) (*server.Server, string)
```
- Creates embedded NATS server for isolated testing
- Uses port `-1` for automatic port allocation (avoids conflicts)
- Returns server instance and connection URL
- Automatically waits for server readiness with 5-second timeout
**Key Features:**
- No external NATS dependency
- Clean isolation between tests
- Automatic cleanup via defer
- Debug logging of assigned port
### 2. Test Daemon Factory (`createTestDaemon`)
```go
func createTestDaemon(t *testing.T, natsURL string) *Daemon
```
- Creates minimal daemon configuration for testing
- Sets up temporary directories with automatic cleanup
- Configures mock Predastore and S3 settings
- Establishes NATS connection
**Cleanup Strategy:**
- Uses `t.Cleanup()` for automatic resource cleanup
- Removes temporary directories
- Closes NATS connections
- No manual cleanup required
### 3. Test Fixtures (`createValidRunInstancesInput`)
```go
func createValidRunInstancesInput() *ec2.RunInstancesInput
```
- Provides fully-formed valid `RunInstancesInput`
- Pre-configured with AWS-compatible values
- Easy to clone and modify for specific tests
- Includes all required fields
## Test Suites
### TestHandleEC2Launch_MessageParsing ✅
**Purpose:** Validates NATS message parsing and error handling
**Test Cases:**
1. Valid RunInstancesInput - expects Reservation with pending instance
2. Invalid instance type - expects error response
3. Missing required ImageId - expects MissingParameter error
4. Invalid MinCount (zero) - expects InvalidParameterValue error
5. Malformed JSON - expects ValidationError
**Status:** ⚠️ Skipped (requires SKIP_INTEGRATION not set)
**Current Issues:**
- Line 638 in daemon.go: `var runInstancesInput *ec2.RunInstancesInput` is nil
- Line 645: `utils.UnmarshalJsonPayload(runInstancesInput, msg.Data)` fails with nil pointer
- The UnmarshalJsonPayload function in utils.go:283 does `decoder.Decode(&input)` which creates a pointer-to-pointer issue
- Even valid inputs are rejected with ValidationError
### TestHandleEC2Launch_ResourceManagement ⏸️
**Purpose:** Tests resource allocation and instance type validation
**Status:** Skipped (requires full infrastructure)
**Reasoning:**
- handleEC2Launch attempts to create viperblock volumes
- Requires S3 backend (predastore) running
- Requires nbdkit for NBD mounting
- Requires QEMU for VM launch
- None of these are available on macOS
**When to Enable:**
- On Linux with full Hive stack running
- In CI/CD pipeline with all dependencies
- With mock services (future enhancement)
### TestDaemon_Initialization ✅
**Purpose:** Tests daemon creation and configuration
**Test Coverage:**
- Daemon struct initialization
- Resource manager creation
- Instance storage initialization
- Configuration loading
**Status:** ✅ PASSING
### TestResourceManager ✅
**Purpose:** Tests resource allocation logic
**Test Coverage:**
- System resource detection (CPU, Memory)
- Instance type definitions
- Resource allocation/deallocation
- Capacity checking
**Status:** ✅ PASSING
## Current Test Results
```bash
$ go test -v ./hive/daemon/
=== RUN TestHandleEC2Launch_MessageParsing
daemon_test.go:115: Skipping integration test - SKIP_INTEGRATION is set
--- SKIP: TestHandleEC2Launch_MessageParsing (0.00s)
=== RUN TestHandleEC2Launch_ResourceManagement
daemon_test.go:261: Skipping resource management test - requires full hive infrastructure
--- SKIP: TestHandleEC2Launch_ResourceManagement (0.00s)
=== RUN TestDaemon_Initialization
--- PASS: TestDaemon_Initialization (0.01s)
=== RUN TestResourceManager
--- PASS: TestResourceManager (0.01s)
PASS
ok github.com/mulgadc/hive/hive/daemon 0.786s
```
## Known Issues & Limitations
### 1. Nil Pointer Bug in handleEC2Launch (daemon.go:638-645)
**Issue:**
```go
var runInstancesInput *ec2.RunInstancesInput // nil pointer
// ...
errResp = utils.UnmarshalJsonPayload(runInstancesInput, msg.Data) // fails
```
**Root Cause:**
- `runInstancesInput` declared as pointer but never initialized
- `UnmarshalJsonPayload` takes `&input` creating pointer-to-pointer
- Results in ValidationError even for valid inputs
**Suggested Fix (not applied per instructions):**
```go
runInstancesInput := &ec2.RunInstancesInput{} // Initialize before use
errResp = utils.UnmarshalJsonPayload(runInstancesInput, msg.Data)
```
Or fix the utils function:
```go
func UnmarshalJsonPayload(input interface{}, jsonData []byte) []byte {
decoder := json.NewDecoder(bytes.NewReader(jsonData))
decoder.DisallowUnknownFields()
err := decoder.Decode(input) // Remove & since input is already a pointer
// ...
}
```
### 2. macOS Limitations
**Missing Dependencies:**
- ❌ viperblock (not ported to macOS)
- ❌ nbdkit (Linux-only, requires kernel NBD)
- ❌ KVM acceleration (Linux-only)
- ❌ Predastore running instance
**Impact:**
- Cannot test volume creation/mounting
- Cannot test VM launch
- Cannot test full EC2 launch flow
**Workarounds:**
- Use Docker Linux container for integration tests
- Create mock services for macOS
- Run full tests in CI/CD on Linux
- Use VM (UTM, Multipass) for local Linux testing
### 3. Test Coverage Gaps
**Not Tested:**
- Volume lifecycle (create, mount, unmount)
- NBD URI generation and validation
- Cloud-init ISO creation
- EFI partition creation
- QMP socket communication
- VM state transitions
- Concurrent launch requests
- Resource exhaustion scenarios
**Future Enhancements:**
- Add mock EBS mount service
- Add mock S3 client
- Add mock viperblock backend
- Test error paths more thoroughly
- Add benchmark tests
- Add stress tests
## Running Tests
### On macOS (Unit Tests Only)
```bash
# Run all passing tests
go test -v ./hive/daemon/
# Run with race detection
go test -race -v ./hive/daemon/
# Run with coverage
go test -cover -v ./hive/daemon/
# Run specific test
go test -v ./hive/daemon/ -run TestResourceManager
```
### On Linux (Full Integration)
```bash
# Requires: predastore, viperblock, nbdkit, qemu running
# Run all tests including integration
SKIP_INTEGRATION="" go test -v ./hive/daemon/
# Or simply
go test -v ./hive/daemon/
```
### In Docker (Recommended)
```bash
# Build test container
docker build -t hive-daemon-test -f Dockerfile.test .
# Run tests
docker run --rm --privileged \
-v $(pwd):/workspace \
hive-daemon-test \
go test -v ./hive/daemon/
```
## Test Architecture Highlights
### ✅ Strengths
1. **Isolated NATS Server**
- Each test gets fresh NATS instance
- No interference between tests
- Automatic port allocation prevents conflicts
2. **Clean Resource Management**
- Automatic cleanup via `t.Cleanup()`
- Temporary directories auto-removed
- No test artifacts left behind
3. **Table-Driven Tests**
- Easy to add new test cases
- Clear test structure
- Minimal code duplication
4. **Clear Documentation**
- Comments explain what each test validates
- Reasoning for skipped tests documented
- macOS limitations clearly stated
5. **Framework Foundation**
- Easy to extend with new tests
- Mock services can be added later
- Ready for CI/CD integration
### ⚠️ Weaknesses
1. **Limited Coverage**
- Only tests message parsing and resource logic
- Cannot test actual VM launch on macOS
- Missing integration with viperblock/nbdkit
2. **Existing Code Issues**
- Nil pointer bug blocks message parsing tests
- utils.UnmarshalJsonPayload has pointer-to-pointer issue
- These bugs need fixing before tests can pass
3. **No Mocking Yet**
- Tests require real infrastructure for full validation
- No mock EBS service
- No mock S3 client
## Next Steps
### Immediate (Required for Tests to Pass)
1. **Fix nil pointer in handleEC2Launch**
```go
// daemon.go:638
runInstancesInput := &ec2.RunInstancesInput{} // Initialize
```
2. **Fix UnmarshalJsonPayload**
```go
// utils.go:283
err := decoder.Decode(input) // Remove &
```
### Short Term (Enhance Test Coverage)
1. **Add Mock Services**
- Mock EBS mount service (NATS responder)
- Mock S3 client for SSH keys
- Mock viperblock for volume operations
2. **Add Platform-Specific Tests**
```go
// +build linux
func TestHandleEC2Launch_FullFlow(t *testing.T) { ... }
```
3. **Add Helper Functions**
- `createMockEBSService(t, nc)`
- `createMockS3Client(t, keys map[string]string)`
- `waitForInstanceState(t, daemon, instanceId, expectedState)`
### Long Term (Production Ready)
1. **CI/CD Integration**
- GitHub Actions workflow for Linux tests
- Docker-based test environment
- Coverage reporting
2. **Comprehensive Test Suite**
- End-to-end VM launch tests
- Concurrent launch stress tests
- Failure recovery tests
- Performance benchmarks
3. **Test Documentation**
- Video walkthrough of test setup
- Troubleshooting guide
- Contributing guidelines
## Recommendations
### For macOS Development
**Option 1: Mock Services** (Fastest)
```go
func TestHandleEC2Launch_WithMocks(t *testing.T) {
ns, natsURL := startTestNATSServer(t)
startMockEBSService(t, nc)
startMockS3Service(t, nc)
// Test with mocked dependencies
}
```
**Option 2: Docker Container** (Most Realistic)
```bash
docker-compose up test-env
go test -v ./hive/daemon/
```
**Option 3: Linux VM** (Balanced)
```bash
multipass launch --name hive-dev
multipass shell hive-dev
# Run tests in VM
```
### For Linux CI/CD
```yaml
# .github/workflows/daemon-tests.yml
name: Daemon Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Install dependencies
run: sudo apt-get install -y nbdkit qemu-kvm
- name: Start services
run: ./scripts/start-test-env.sh
- name: Run tests
run: go test -v ./hive/daemon/
```
## Conclusion
This test framework provides:
✅ **Solid Foundation**
- NATS message handling tests
- Resource manager validation
- Clean test isolation
- Easy to extend
⚠️ **Current Limitations**
- Requires bug fixes in daemon.go and utils.go
- Limited to unit tests on macOS
- Needs mock services for full coverage
🎯 **Future Potential**
- Can support full integration tests on Linux
- Mock services enable macOS testing
- Ready for CI/CD integration
- Foundation for comprehensive test suite
The test framework is **production-ready for unit testing** but requires the identified bug fixes and either mock services (macOS) or full infrastructure (Linux) for integration testing.
+1637 -1074
View File
File diff suppressed because it is too large Load Diff
+209
View File
@@ -0,0 +1,209 @@
package daemon
import (
"encoding/json"
"fmt"
"log/slog"
"net"
"time"
"github.com/mulgadc/hive/hive/awserrors"
"github.com/mulgadc/hive/hive/types"
"github.com/mulgadc/hive/hive/utils"
"github.com/mulgadc/hive/hive/vm"
"github.com/nats-io/nats.go"
)
// respondWithError sends an error payload for the given error code on the NATS message.
func respondWithError(msg *nats.Msg, errCode string) {
if err := msg.Respond(utils.GenerateErrorPayload(errCode)); err != nil {
slog.Error("Failed to respond to NATS request", "err", err)
}
}
// respondWithJSON marshals data to JSON and sends it as a NATS response.
// On marshal failure it responds with an internal server error.
func respondWithJSON(msg *nats.Msg, data any) {
jsonResponse, err := json.Marshal(data)
if err != nil {
slog.Error("Failed to marshal response", "type", fmt.Sprintf("%T", data), "err", err)
respondWithError(msg, awserrors.ErrorServerInternal)
return
}
if err := msg.Respond(jsonResponse); err != nil {
slog.Error("Failed to respond to NATS request", "err", err)
}
}
// handleNATSRequest is a generic helper for the common unmarshal → service → marshal → respond pattern.
// It extracts the account ID from the NATS message header and passes it to the service function.
func handleNATSRequest[I any, O any](msg *nats.Msg, serviceFn func(*I, string) (*O, error)) {
accountID := utils.AccountIDFromMsg(msg)
input := new(I)
if errResp := utils.UnmarshalJsonPayload(input, msg.Data); errResp != nil {
if err := msg.Respond(errResp); err != nil {
slog.Error("Failed to respond to NATS request", "err", err)
}
return
}
output, err := serviceFn(input, accountID)
if err != nil {
respondWithError(msg, awserrors.ValidErrorCode(err.Error()))
return
}
respondWithJSON(msg, output)
}
// handleEC2Events processes incoming EC2 instance events (start, stop, terminate, attach-volume)
func (d *Daemon) handleEC2Events(msg *nats.Msg) {
var command types.EC2InstanceCommand
if err := json.Unmarshal(msg.Data, &command); err != nil {
slog.Error("Error unmarshaling EC2 instance command", "err", err)
respondWithError(msg, awserrors.ErrorServerInternal)
return
}
slog.Debug("Received message", "subject", msg.Subject, "data", string(msg.Data))
d.Instances.Mu.Lock()
instance, ok := d.Instances.VMS[command.ID]
d.Instances.Mu.Unlock()
if !ok {
slog.Warn("Instance is not running on this node", "id", command.ID)
respondWithError(msg, awserrors.ErrorInvalidInstanceIDNotFound)
return
}
// Verify the caller owns this instance
if !checkInstanceOwnership(msg, command.ID, instance.AccountID) {
return
}
switch {
case command.Attributes.AttachVolume:
d.handleAttachVolume(msg, command, instance)
case command.Attributes.DetachVolume:
d.handleDetachVolume(msg, command, instance)
case command.Attributes.StartInstance:
d.handleStartInstance(msg, command, instance)
case command.Attributes.StopInstance, command.Attributes.TerminateInstance:
d.handleStopOrTerminateInstance(msg, command, instance)
default:
slog.Warn("Unhandled EC2 instance command", "id", command.ID, "attributes", command.Attributes)
respondWithError(msg, awserrors.ErrorServerInternal)
}
}
// --- Admin / node management handlers ---
// handleHealthCheck processes NATS health check requests
func (d *Daemon) handleHealthCheck(msg *nats.Msg) {
configHash, err := d.computeConfigHash()
if err != nil {
slog.Error("Failed to compute config hash for health check", "error", err)
configHash = "error"
}
status := "running"
if !d.ready.Load() {
status = "starting"
}
response := types.NodeHealthResponse{
Node: d.node,
Status: status,
ConfigHash: configHash,
Epoch: d.clusterConfig.Epoch,
Uptime: int64(time.Since(d.startTime).Seconds()),
}
respondWithJSON(msg, response)
slog.Debug("Health check responded", "node", d.node, "epoch", d.clusterConfig.Epoch)
}
// handleNodeDiscover responds to node discovery requests with this node's ID
// Used by the gateway to dynamically discover active hive nodes in the cluster
func (d *Daemon) handleNodeDiscover(msg *nats.Msg) {
response := types.NodeDiscoverResponse{
Node: d.node,
}
respondWithJSON(msg, response)
slog.Debug("Node discovery responded", "node", d.node)
}
// daemonIP extracts the IP portion from the daemon host (host:port format).
func (d *Daemon) daemonIP() string {
host := d.config.Daemon.Host
if h, _, err := net.SplitHostPort(host); err == nil {
return h
}
return host
}
// handleNodeStatus responds with this node's status and resource stats.
// Used by the CLI: hive get nodes, hive top nodes.
func (d *Daemon) handleNodeStatus(msg *nats.Msg) {
totalVCPU, totalMemGB, allocVCPU, allocMemGB, caps := d.resourceMgr.GetResourceStats()
d.Instances.Mu.Lock()
vmCount := 0
for _, v := range d.Instances.VMS {
if v.Status == vm.StateRunning {
vmCount++
}
}
d.Instances.Mu.Unlock()
resp := types.NodeStatusResponse{
Node: d.node,
Status: "Ready",
Host: d.daemonIP(),
Region: d.config.Region,
AZ: d.config.AZ,
Uptime: int64(time.Since(d.startTime).Seconds()),
Services: d.config.GetServices(),
TotalVCPU: totalVCPU,
TotalMemGB: totalMemGB,
AllocVCPU: allocVCPU,
AllocMemGB: allocMemGB,
VMCount: vmCount,
InstanceTypes: caps,
}
respondWithJSON(msg, resp)
}
// handleNodeVMs responds with the list of VMs running on this node.
// Used by the CLI: hive get vms.
func (d *Daemon) handleNodeVMs(msg *nats.Msg) {
d.Instances.Mu.Lock()
vms := make([]types.VMInfo, 0, len(d.Instances.VMS))
for _, v := range d.Instances.VMS {
info := types.VMInfo{
InstanceID: v.ID,
Status: string(v.Status),
InstanceType: v.InstanceType,
}
// Get vCPU/memory from the resource manager's instance type info
if it, ok := d.resourceMgr.instanceTypes[v.InstanceType]; ok {
info.VCPU = int(instanceTypeVCPUs(it))
info.MemoryGB = float64(instanceTypeMemoryMiB(it)) / 1024.0
}
if v.Instance != nil && v.Instance.LaunchTime != nil {
info.LaunchTime = v.Instance.LaunchTime.Unix()
}
vms = append(vms, info)
}
d.Instances.Mu.Unlock()
resp := types.NodeVMsResponse{
Node: d.node,
Host: d.daemonIP(),
VMs: vms,
}
respondWithJSON(msg, resp)
}
+29
View File
@@ -0,0 +1,29 @@
package daemon
import (
"github.com/nats-io/nats.go"
)
func (d *Daemon) handleEC2EnableEbsEncryptionByDefault(msg *nats.Msg) {
handleNATSRequest(msg, d.accountService.EnableEbsEncryptionByDefault)
}
func (d *Daemon) handleEC2DisableEbsEncryptionByDefault(msg *nats.Msg) {
handleNATSRequest(msg, d.accountService.DisableEbsEncryptionByDefault)
}
func (d *Daemon) handleEC2GetEbsEncryptionByDefault(msg *nats.Msg) {
handleNATSRequest(msg, d.accountService.GetEbsEncryptionByDefault)
}
func (d *Daemon) handleEC2GetSerialConsoleAccessStatus(msg *nats.Msg) {
handleNATSRequest(msg, d.accountService.GetSerialConsoleAccessStatus)
}
func (d *Daemon) handleEC2EnableSerialConsoleAccess(msg *nats.Msg) {
handleNATSRequest(msg, d.accountService.EnableSerialConsoleAccess)
}
func (d *Daemon) handleEC2DisableSerialConsoleAccess(msg *nats.Msg) {
handleNATSRequest(msg, d.accountService.DisableSerialConsoleAccess)
}
+92
View File
@@ -0,0 +1,92 @@
package daemon
import (
"encoding/base64"
"log/slog"
"os"
"time"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/mulgadc/hive/hive/awserrors"
"github.com/mulgadc/hive/hive/utils"
"github.com/nats-io/nats.go"
)
// handleEC2GetConsoleOutput reads the console log file for an instance and returns
// base64-encoded output matching the AWS GetConsoleOutput API response format.
func (d *Daemon) handleEC2GetConsoleOutput(msg *nats.Msg) {
slog.Debug("Received GetConsoleOutput request", "subject", msg.Subject, "data", string(msg.Data))
var input ec2.GetConsoleOutputInput
if errResp := utils.UnmarshalJsonPayload(&input, msg.Data); errResp != nil {
if err := msg.Respond(errResp); err != nil {
slog.Error("Failed to respond to NATS request", "err", err)
}
return
}
if input.InstanceId == nil {
respondWithError(msg, awserrors.ErrorMissingParameter)
return
}
instanceID := *input.InstanceId
// Find the instance on this node
d.Instances.Mu.Lock()
instance, exists := d.Instances.VMS[instanceID]
d.Instances.Mu.Unlock()
if !exists {
respondWithError(msg, awserrors.ErrorInvalidInstanceIDNotFound)
return
}
// Verify the caller owns this instance
if !checkInstanceOwnership(msg, instanceID, instance.AccountID) {
return
}
logPath := instance.Config.ConsoleLogPath
var outputData []byte
var modTime time.Time
if logPath != "" {
info, err := os.Stat(logPath)
if err == nil {
modTime = info.ModTime()
data, err := os.ReadFile(logPath)
if err != nil {
slog.Error("Failed to read console log", "path", logPath, "err", err)
} else {
// Return last 64KB (AWS limit)
const maxConsoleOutput = 64 * 1024
if len(data) > maxConsoleOutput {
data = data[len(data)-maxConsoleOutput:]
}
outputData = data
}
}
}
// Base64-encode the output (AWS returns base64)
var encodedOutput string
if len(outputData) > 0 {
encodedOutput = base64.StdEncoding.EncodeToString(outputData)
}
now := time.Now()
if modTime.IsZero() {
modTime = now
}
output := &ec2.GetConsoleOutputOutput{
InstanceId: &instanceID,
Output: &encodedOutput,
Timestamp: &modTime,
}
respondWithJSON(msg, output)
slog.Info("handleEC2GetConsoleOutput completed", "instance_id", instanceID, "output_bytes", len(outputData))
}
+37
View File
@@ -0,0 +1,37 @@
package daemon
import (
"github.com/nats-io/nats.go"
)
func (d *Daemon) handleEC2CreateInternetGateway(msg *nats.Msg) {
handleNATSRequest(msg, d.igwService.CreateInternetGateway)
}
func (d *Daemon) handleEC2DeleteInternetGateway(msg *nats.Msg) {
handleNATSRequest(msg, d.igwService.DeleteInternetGateway)
}
func (d *Daemon) handleEC2DescribeInternetGateways(msg *nats.Msg) {
handleNATSRequest(msg, d.igwService.DescribeInternetGateways)
}
func (d *Daemon) handleEC2AttachInternetGateway(msg *nats.Msg) {
handleNATSRequest(msg, d.igwService.AttachInternetGateway)
}
func (d *Daemon) handleEC2DetachInternetGateway(msg *nats.Msg) {
handleNATSRequest(msg, d.igwService.DetachInternetGateway)
}
func (d *Daemon) handleEC2CreateEgressOnlyInternetGateway(msg *nats.Msg) {
handleNATSRequest(msg, d.eigwService.CreateEgressOnlyInternetGateway)
}
func (d *Daemon) handleEC2DeleteEgressOnlyInternetGateway(msg *nats.Msg) {
handleNATSRequest(msg, d.eigwService.DeleteEgressOnlyInternetGateway)
}
func (d *Daemon) handleEC2DescribeEgressOnlyInternetGateways(msg *nats.Msg) {
handleNATSRequest(msg, d.eigwService.DescribeEgressOnlyInternetGateways)
}
+100
View File
@@ -0,0 +1,100 @@
package daemon
import (
"log/slog"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/mulgadc/hive/hive/awserrors"
handlers_ec2_image "github.com/mulgadc/hive/hive/handlers/ec2/image"
"github.com/mulgadc/hive/hive/utils"
"github.com/mulgadc/hive/hive/vm"
"github.com/nats-io/nats.go"
)
func (d *Daemon) handleEC2DescribeImages(msg *nats.Msg) {
handleNATSRequest(msg, d.imageService.DescribeImages)
}
// handleEC2CreateImage is a stateful handler that extracts instance context
// (root volume ID, source AMI, running state) before delegating to the image service.
func (d *Daemon) handleEC2CreateImage(msg *nats.Msg) {
slog.Debug("Received message", "subject", msg.Subject)
input := &ec2.CreateImageInput{}
if errResp := utils.UnmarshalJsonPayload(input, msg.Data); errResp != nil {
if err := msg.Respond(errResp); err != nil {
slog.Error("Failed to respond to NATS request", "err", err)
}
return
}
accountID := utils.AccountIDFromMsg(msg)
if input.InstanceId == nil || *input.InstanceId == "" {
respondWithError(msg, awserrors.ErrorMissingParameter)
return
}
instanceID := *input.InstanceId
// Extract all instance context in a single critical section
d.Instances.Mu.Lock()
instance, ok := d.Instances.VMS[instanceID]
var status vm.InstanceState
var rootVolumeID, sourceImageID string
if ok {
status = instance.Status
if instance.Instance != nil {
for _, bdm := range instance.Instance.BlockDeviceMappings {
if bdm.Ebs != nil && bdm.Ebs.VolumeId != nil {
rootVolumeID = *bdm.Ebs.VolumeId
break
}
}
if instance.Instance.ImageId != nil {
sourceImageID = *instance.Instance.ImageId
}
}
}
d.Instances.Mu.Unlock()
if !ok {
slog.Warn("CreateImage: instance not found", "instanceId", instanceID)
respondWithError(msg, awserrors.ErrorInvalidInstanceIDNotFound)
return
}
// Verify the caller owns this instance
if !checkInstanceOwnership(msg, instanceID, instance.AccountID) {
return
}
if status != vm.StateRunning && status != vm.StateStopped {
slog.Warn("CreateImage: instance not in valid state", "instanceId", instanceID, "status", status)
respondWithError(msg, awserrors.ErrorIncorrectInstanceState)
return
}
if rootVolumeID == "" {
slog.Error("CreateImage: no root volume found", "instanceId", instanceID)
respondWithError(msg, awserrors.ErrorServerInternal)
return
}
params := handlers_ec2_image.CreateImageParams{
Input: input,
RootVolumeID: rootVolumeID,
SourceImageID: sourceImageID,
IsRunning: status == vm.StateRunning,
}
output, err := d.imageService.CreateImageFromInstance(params, accountID)
if err != nil {
slog.Error("CreateImage: service failed", "instanceId", instanceID, "err", err)
respondWithError(msg, awserrors.ValidErrorCode(err.Error()))
return
}
respondWithJSON(msg, output)
slog.Info("CreateImage completed", "instanceId", instanceID, "imageId", *output.ImageId)
}
File diff suppressed because it is too large Load Diff
+21
View File
@@ -0,0 +1,21 @@
package daemon
import (
"github.com/nats-io/nats.go"
)
func (d *Daemon) handleEC2CreateKeyPair(msg *nats.Msg) {
handleNATSRequest(msg, d.keyService.CreateKeyPair)
}
func (d *Daemon) handleEC2DeleteKeyPair(msg *nats.Msg) {
handleNATSRequest(msg, d.keyService.DeleteKeyPair)
}
func (d *Daemon) handleEC2DescribeKeyPairs(msg *nats.Msg) {
handleNATSRequest(msg, d.keyService.DescribeKeyPairs)
}
func (d *Daemon) handleEC2ImportKeyPair(msg *nats.Msg) {
handleNATSRequest(msg, d.keyService.ImportKeyPair)
}
+21
View File
@@ -0,0 +1,21 @@
package daemon
import (
"github.com/nats-io/nats.go"
)
func (d *Daemon) handleEC2CreateSnapshot(msg *nats.Msg) {
handleNATSRequest(msg, d.snapshotService.CreateSnapshot)
}
func (d *Daemon) handleEC2DescribeSnapshots(msg *nats.Msg) {
handleNATSRequest(msg, d.snapshotService.DescribeSnapshots)
}
func (d *Daemon) handleEC2DeleteSnapshot(msg *nats.Msg) {
handleNATSRequest(msg, d.snapshotService.DeleteSnapshot)
}
func (d *Daemon) handleEC2CopySnapshot(msg *nats.Msg) {
handleNATSRequest(msg, d.snapshotService.CopySnapshot)
}
+15
View File
@@ -0,0 +1,15 @@
package daemon
import "github.com/nats-io/nats.go"
func (d *Daemon) handleEC2CreateTags(msg *nats.Msg) {
handleNATSRequest(msg, d.tagsService.CreateTags)
}
func (d *Daemon) handleEC2DeleteTags(msg *nats.Msg) {
handleNATSRequest(msg, d.tagsService.DeleteTags)
}
func (d *Daemon) handleEC2DescribeTags(msg *nats.Msg) {
handleNATSRequest(msg, d.tagsService.DescribeTags)
}
File diff suppressed because it is too large Load Diff
+501
View File
@@ -0,0 +1,501 @@
package daemon
import (
"encoding/json"
"fmt"
"log/slog"
"time"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/mulgadc/hive/hive/awserrors"
"github.com/mulgadc/hive/hive/qmp"
"github.com/mulgadc/hive/hive/types"
"github.com/mulgadc/hive/hive/utils"
"github.com/mulgadc/hive/hive/vm"
"github.com/nats-io/nats.go"
)
// handleAttachVolume performs a three-phase hot-plug:
//
// Phase 1: ebs.mount via NATS (rolls back with ebs.unmount)
// Phase 2: QMP blockdev-add (rolls back Phase 1)
// Phase 3: QMP device_add (rolls back Phase 2 + Phase 1)
func (d *Daemon) handleAttachVolume(msg *nats.Msg, command types.EC2InstanceCommand, instance *vm.VM) {
slog.Info("Attaching volume to instance", "instanceId", command.ID)
// Validate AttachVolumeData
if command.AttachVolumeData == nil || command.AttachVolumeData.VolumeID == "" {
slog.Error("AttachVolume: missing attach volume data")
respondWithError(msg, awserrors.ErrorInvalidParameterValue)
return
}
volumeID := command.AttachVolumeData.VolumeID
device := command.AttachVolumeData.Device
// Validate instance is running
d.Instances.Mu.Lock()
status := instance.Status
d.Instances.Mu.Unlock()
if status != vm.StateRunning {
slog.Error("AttachVolume: instance not running", "instanceId", command.ID, "status", status)
respondWithError(msg, awserrors.ErrorIncorrectInstanceState)
return
}
// Validate volume exists and is available
volCfg, err := d.volumeService.GetVolumeConfig(volumeID)
if err != nil {
slog.Error("AttachVolume: failed to get volume config", "volumeId", volumeID, "err", err)
respondWithError(msg, awserrors.ErrorInvalidVolumeNotFound)
return
}
// Account scoping: verify the caller owns this volume
callerAccountID := utils.AccountIDFromMsg(msg)
if callerAccountID != "" && volCfg.VolumeMetadata.TenantID != "" && volCfg.VolumeMetadata.TenantID != callerAccountID {
slog.Warn("AttachVolume: account does not own volume", "volumeId", volumeID, "callerAccount", callerAccountID, "ownerAccount", volCfg.VolumeMetadata.TenantID)
respondWithError(msg, awserrors.ErrorInvalidVolumeNotFound)
return
}
if volCfg.VolumeMetadata.State != "available" {
slog.Error("AttachVolume: volume not available", "volumeId", volumeID, "state", volCfg.VolumeMetadata.State)
respondWithError(msg, awserrors.ErrorVolumeInUse)
return
}
// Check AZ compatibility — volume and instance must be in the same AZ
if volCfg.VolumeMetadata.AvailabilityZone != "" && d.config.AZ != "" &&
volCfg.VolumeMetadata.AvailabilityZone != d.config.AZ {
slog.Error("AttachVolume: volume and instance are in different AZs",
"volumeId", volumeID, "volumeAZ", volCfg.VolumeMetadata.AvailabilityZone,
"instanceAZ", d.config.AZ)
respondWithError(msg, awserrors.ErrorInvalidVolumeZoneMismatch)
return
}
// Determine device name
if device == "" {
d.Instances.Mu.Lock()
device = nextAvailableDevice(instance)
d.Instances.Mu.Unlock()
if device == "" {
slog.Error("AttachVolume: no available device names")
respondWithError(msg, awserrors.ErrorAttachmentLimitExceeded)
return
}
}
// Create EBS mount request
ebsRequest := types.EBSRequest{
Name: volumeID,
DeviceName: device,
}
ebsMountData, err := json.Marshal(ebsRequest)
if err != nil {
slog.Error("AttachVolume: failed to marshal ebs.mount request", "err", err)
respondWithError(msg, awserrors.ErrorServerInternal)
return
}
mountReply, err := d.natsConn.Request(d.ebsTopic("mount"), ebsMountData, 30*time.Second)
if err != nil {
slog.Error("AttachVolume: ebs.mount failed", "volumeId", volumeID, "err", err)
respondWithError(msg, awserrors.ErrorServerInternal)
return
}
var mountResp types.EBSMountResponse
if err := json.Unmarshal(mountReply.Data, &mountResp); err != nil {
slog.Error("AttachVolume: failed to unmarshal mount response", "err", err)
respondWithError(msg, awserrors.ErrorServerInternal)
return
}
if mountResp.Error != "" {
slog.Error("AttachVolume: mount returned error", "volumeId", volumeID, "err", mountResp.Error)
respondWithError(msg, awserrors.ErrorServerInternal)
return
}
nbdURI := mountResp.URI
if nbdURI == "" {
slog.Error("AttachVolume: mount response has empty URI", "volumeId", volumeID)
d.rollbackEBSMount(ebsRequest)
respondWithError(msg, awserrors.ErrorServerInternal)
return
}
ebsRequest.NBDURI = nbdURI
// Parse NBDURI for QMP blockdev-add
serverType, socketPath, nbdHost, nbdPort, err := utils.ParseNBDURI(nbdURI)
if err != nil {
slog.Error("AttachVolume: failed to parse NBDURI", "uri", nbdURI, "err", err)
// Rollback: unmount
d.rollbackEBSMount(ebsRequest)
respondWithError(msg, awserrors.ErrorServerInternal)
return
}
// Build QMP server argument
var serverArg map[string]any
if serverType == "unix" {
serverArg = map[string]any{"type": "unix", "path": socketPath}
} else {
serverArg = map[string]any{"type": "inet", "host": nbdHost, "port": fmt.Sprintf("%d", nbdPort)}
}
nodeName := fmt.Sprintf("nbd-%s", volumeID)
deviceID := fmt.Sprintf("vdisk-%s", volumeID)
iothreadID := fmt.Sprintf("ioth-%s", volumeID)
// QMP object-add: create iothread for this volume
iothreadCmd := qmp.QMPCommand{
Execute: "object-add",
Arguments: map[string]any{
"qom-type": "iothread",
"id": iothreadID,
},
}
_, err = d.SendQMPCommand(instance.QMPClient, iothreadCmd, instance.ID)
if err != nil {
slog.Error("AttachVolume: QMP object-add iothread failed", "volumeId", volumeID, "err", err)
d.rollbackEBSMount(ebsRequest)
respondWithError(msg, awserrors.ErrorServerInternal)
return
}
// QMP blockdev-add
blockdevCmd := qmp.QMPCommand{
Execute: "blockdev-add",
Arguments: map[string]any{
"node-name": nodeName,
"driver": "nbd",
"server": serverArg,
"export": "",
"read-only": false,
},
}
_, err = d.SendQMPCommand(instance.QMPClient, blockdevCmd, instance.ID)
if err != nil {
slog.Error("AttachVolume: QMP blockdev-add failed", "volumeId", volumeID, "err", err)
d.rollbackEBSMount(ebsRequest)
respondWithError(msg, awserrors.ErrorServerInternal)
return
}
// Determine which hotplug root port to use based on device letter.
// /dev/sdf -> hotplug1, /dev/sdg -> hotplug2, etc.
hotplugBus := ""
if len(device) > 0 {
letter := device[len(device)-1]
if letter >= 'f' && letter <= 'p' {
hotplugBus = fmt.Sprintf("hotplug%d", letter-'f'+1)
}
}
// QMP device_add
deviceAddArgs := map[string]any{
"driver": "virtio-blk-pci",
"id": deviceID,
"drive": nodeName,
"iothread": iothreadID,
}
if hotplugBus != "" {
deviceAddArgs["bus"] = hotplugBus
}
deviceAddCmd := qmp.QMPCommand{
Execute: "device_add",
Arguments: deviceAddArgs,
}
_, err = d.SendQMPCommand(instance.QMPClient, deviceAddCmd, instance.ID)
if err != nil {
slog.Error("AttachVolume: QMP device_add failed, rolling back blockdev", "volumeId", volumeID, "err", err)
// Rollback: blockdev-del then unmount
if _, delErr := d.SendQMPCommand(instance.QMPClient, qmp.QMPCommand{
Execute: "blockdev-del",
Arguments: map[string]any{"node-name": nodeName},
}, instance.ID); delErr != nil {
slog.Error("AttachVolume: rollback blockdev-del failed, skipping EBS unmount", "volumeId", volumeID, "err", delErr)
} else {
d.rollbackEBSMount(ebsRequest)
}
respondWithError(msg, awserrors.ErrorServerInternal)
return
}
// Discover actual guest device name via QMP query-block.
// Use the retry variant because query-block may not include the device
// immediately after device_add.
guestDevice := device // fallback to AWS API name
deviceMap, qmpErr := queryGuestDeviceMapWait(d, instance.QMPClient, instance.ID, deviceID)
if qmpErr != nil {
slog.Warn("AttachVolume: failed to query guest device map, using API device name", "volumeId", volumeID, "err", qmpErr)
} else if gd, ok := deviceMap[deviceID]; ok {
guestDevice = gd
slog.Info("AttachVolume: discovered guest device", "volumeId", volumeID, "qemuDevice", deviceID, "guestDevice", guestDevice)
} else {
slog.Error("AttachVolume: device not found in QMP device map after retries, using API device name",
"volumeId", volumeID, "qemuDevice", deviceID, "deviceMap", deviceMap)
}
// Update instance state: replace existing entry for this volume (handles
// stop/start cycles that keep stale entries) or append a new one.
instance.EBSRequests.Mu.Lock()
replaced := false
for i, req := range instance.EBSRequests.Requests {
if req.Name == volumeID {
instance.EBSRequests.Requests[i] = ebsRequest
replaced = true
break
}
}
if !replaced {
instance.EBSRequests.Requests = append(instance.EBSRequests.Requests, ebsRequest)
}
instance.EBSRequests.Mu.Unlock()
// Update BlockDeviceMappings on the ec2.Instance using actual guest device name
d.Instances.Mu.Lock()
if instance.Instance != nil {
now := time.Now()
mapping := &ec2.InstanceBlockDeviceMapping{}
mapping.SetDeviceName(guestDevice)
mapping.Ebs = &ec2.EbsInstanceBlockDevice{}
mapping.Ebs.SetVolumeId(volumeID)
mapping.Ebs.SetAttachTime(now)
mapping.Ebs.SetDeleteOnTermination(false)
mapping.Ebs.SetStatus("attached")
instance.Instance.BlockDeviceMappings = append(instance.Instance.BlockDeviceMappings, mapping)
}
d.Instances.Mu.Unlock()
// Update volume metadata in S3
if err := d.volumeService.UpdateVolumeState(volumeID, "in-use", command.ID, guestDevice); err != nil {
slog.Error("AttachVolume: failed to update volume metadata", "volumeId", volumeID, "err", err)
}
// Persist state
if err := d.WriteState(); err != nil {
slog.Error("AttachVolume: failed to write state", "err", err)
}
d.respondWithVolumeAttachment(msg, volumeID, command.ID, guestDevice, "attached")
slog.Info("Volume attached successfully", "volumeId", volumeID, "instanceId", command.ID, "apiDevice", device, "guestDevice", guestDevice)
}
// handleDetachVolume performs a three-phase hot-unplug (reverse of attach):
//
// Phase 1: QMP device_del (remove guest device)
// Phase 2: QMP blockdev-del (remove block node)
// Phase 3: ebs.unmount NATS (stop NBD server)
func (d *Daemon) handleDetachVolume(msg *nats.Msg, command types.EC2InstanceCommand, instance *vm.VM) {
slog.Info("Detaching volume from instance", "instanceId", command.ID)
// Validate DetachVolumeData
if command.DetachVolumeData == nil || command.DetachVolumeData.VolumeID == "" {
slog.Error("DetachVolume: missing detach volume data")
respondWithError(msg, awserrors.ErrorInvalidParameterValue)
return
}
volumeID := command.DetachVolumeData.VolumeID
device := command.DetachVolumeData.Device
force := command.DetachVolumeData.Force
// Validate instance is running
d.Instances.Mu.Lock()
status := instance.Status
d.Instances.Mu.Unlock()
if status != vm.StateRunning {
slog.Error("DetachVolume: instance not running", "instanceId", command.ID, "status", status)
respondWithError(msg, awserrors.ErrorIncorrectInstanceState)
return
}
// Find the volume in EBSRequests
instance.EBSRequests.Mu.Lock()
var ebsReq types.EBSRequest
found := false
for _, req := range instance.EBSRequests.Requests {
if req.Name == volumeID {
ebsReq = req
found = true
break
}
}
instance.EBSRequests.Mu.Unlock()
if !found {
slog.Error("DetachVolume: volume not attached to instance", "volumeId", volumeID, "instanceId", command.ID)
respondWithError(msg, awserrors.ErrorIncorrectState)
return
}
// Reject detaching boot/EFI/CloudInit volumes
if ebsReq.Boot || ebsReq.EFI || ebsReq.CloudInit {
slog.Error("DetachVolume: cannot detach boot/EFI/CloudInit volume", "volumeId", volumeID)
respondWithError(msg, awserrors.ErrorOperationNotPermitted)
return
}
// Optional device cross-check
if device != "" && ebsReq.DeviceName != "" && device != ebsReq.DeviceName {
slog.Error("DetachVolume: device mismatch", "requested", device, "actual", ebsReq.DeviceName)
respondWithError(msg, awserrors.ErrorInvalidParameterValue)
return
}
deviceID := fmt.Sprintf("vdisk-%s", volumeID)
nodeName := fmt.Sprintf("nbd-%s", volumeID)
iothreadID := fmt.Sprintf("ioth-%s", volumeID)
// Phase 1: QMP device_del (remove guest device)
_, err := d.SendQMPCommand(instance.QMPClient, qmp.QMPCommand{
Execute: "device_del",
Arguments: map[string]any{"id": deviceID},
}, instance.ID)
if err != nil {
if !force {
slog.Error("DetachVolume: QMP device_del failed", "volumeId", volumeID, "err", err)
respondWithError(msg, awserrors.ErrorServerInternal)
return
}
slog.Warn("DetachVolume: QMP device_del failed (force=true, continuing)", "volumeId", volumeID, "err", err)
}
// Brief pause for guest to acknowledge PCI removal
time.Sleep(d.detachDelay)
// Phase 2: QMP blockdev-del (remove block node)
_, blockdevErr := d.SendQMPCommand(instance.QMPClient, qmp.QMPCommand{
Execute: "blockdev-del",
Arguments: map[string]any{"node-name": nodeName},
}, instance.ID)
if blockdevErr != nil {
// Block node still referenced by QEMU; do not clean up state or unmount —
// tearing down the NBD server would crash the VM, and removing metadata
// would allow the volume to be double-attached.
slog.Error("DetachVolume: QMP blockdev-del failed, leaving volume state intact", "volumeId", volumeID, "err", blockdevErr)
respondWithError(msg, awserrors.ErrorServerInternal)
return
}
// Phase 2b: QMP object-del (remove iothread, best-effort)
_, iothreadErr := d.SendQMPCommand(instance.QMPClient, qmp.QMPCommand{
Execute: "object-del",
Arguments: map[string]any{"id": iothreadID},
}, instance.ID)
if iothreadErr != nil {
slog.Warn("DetachVolume: QMP object-del iothread failed (non-fatal)", "volumeId", volumeID, "err", iothreadErr)
}
// Phase 3: ebs.unmount via NATS (best-effort)
d.rollbackEBSMount(ebsReq)
// State cleanup: remove volume from EBSRequests (search by name to avoid stale index)
instance.EBSRequests.Mu.Lock()
for i, req := range instance.EBSRequests.Requests {
if req.Name == volumeID {
instance.EBSRequests.Requests = append(instance.EBSRequests.Requests[:i], instance.EBSRequests.Requests[i+1:]...)
break
}
}
instance.EBSRequests.Mu.Unlock()
// Remove from BlockDeviceMappings
d.Instances.Mu.Lock()
if instance.Instance != nil {
filtered := make([]*ec2.InstanceBlockDeviceMapping, 0, len(instance.Instance.BlockDeviceMappings))
for _, bdm := range instance.Instance.BlockDeviceMappings {
if bdm.Ebs != nil && bdm.Ebs.VolumeId != nil && *bdm.Ebs.VolumeId == volumeID {
continue
}
filtered = append(filtered, bdm)
}
instance.Instance.BlockDeviceMappings = filtered
}
d.Instances.Mu.Unlock()
// Update volume metadata to "available"
if err := d.volumeService.UpdateVolumeState(volumeID, "available", "", ""); err != nil {
slog.Error("DetachVolume: failed to update volume metadata", "volumeId", volumeID, "err", err)
}
// Persist state
if err := d.WriteState(); err != nil {
slog.Error("DetachVolume: failed to write state", "err", err)
}
d.respondWithVolumeAttachment(msg, volumeID, command.ID, ebsReq.DeviceName, "detaching")
slog.Info("Volume detached successfully", "volumeId", volumeID, "instanceId", command.ID)
}
func (d *Daemon) handleEC2CreateVolume(msg *nats.Msg) {
handleNATSRequest(msg, d.volumeService.CreateVolume)
}
func (d *Daemon) handleEC2DescribeVolumes(msg *nats.Msg) {
handleNATSRequest(msg, d.volumeService.DescribeVolumes)
}
func (d *Daemon) handleEC2DescribeVolumeStatus(msg *nats.Msg) {
handleNATSRequest(msg, d.volumeService.DescribeVolumeStatus)
}
// handleEC2ModifyVolume processes incoming EC2 ModifyVolume requests
func (d *Daemon) handleEC2ModifyVolume(msg *nats.Msg) {
slog.Debug("Received message", "subject", msg.Subject)
slog.Debug("Message data", "data", string(msg.Data))
accountID := utils.AccountIDFromMsg(msg)
modifyVolumeInput := &ec2.ModifyVolumeInput{}
errResp := utils.UnmarshalJsonPayload(modifyVolumeInput, msg.Data)
if errResp != nil {
if err := msg.Respond(errResp); err != nil {
slog.Error("Failed to respond to NATS request", "err", err)
}
slog.Error("Request does not match ModifyVolumeInput")
return
}
slog.Info("Processing ModifyVolume request", "volumeId", modifyVolumeInput.VolumeId, "accountID", accountID)
output, err := d.volumeService.ModifyVolume(modifyVolumeInput, accountID)
if err != nil {
slog.Error("handleEC2ModifyVolume service.ModifyVolume failed", "err", err)
respondWithError(msg, awserrors.ValidErrorCode(err.Error()))
return
}
respondWithJSON(msg, output)
// Notify viperblockd to reload state after volume modification (e.g. resize)
if modifyVolumeInput.VolumeId != nil {
syncData, err := json.Marshal(types.EBSSyncRequest{Volume: *modifyVolumeInput.VolumeId})
if err != nil {
slog.Error("failed to marshal ebs.sync request", "volumeId", *modifyVolumeInput.VolumeId, "err", err)
} else {
_, syncErr := d.natsConn.Request("ebs.sync", syncData, 5*time.Second)
if syncErr != nil {
slog.Warn("ebs.sync notification failed (volume may not be mounted)",
"volumeId", *modifyVolumeInput.VolumeId, "err", syncErr)
}
}
}
slog.Info("handleEC2ModifyVolume completed", "volumeId", modifyVolumeInput.VolumeId)
}
func (d *Daemon) handleEC2DeleteVolume(msg *nats.Msg) {
handleNATSRequest(msg, d.volumeService.DeleteVolume)
}
+41
View File
@@ -0,0 +1,41 @@
package daemon
import (
"github.com/nats-io/nats.go"
)
func (d *Daemon) handleEC2CreateVpc(msg *nats.Msg) {
handleNATSRequest(msg, d.vpcService.CreateVpc)
}
func (d *Daemon) handleEC2DeleteVpc(msg *nats.Msg) {
handleNATSRequest(msg, d.vpcService.DeleteVpc)
}
func (d *Daemon) handleEC2DescribeVpcs(msg *nats.Msg) {
handleNATSRequest(msg, d.vpcService.DescribeVpcs)
}
func (d *Daemon) handleEC2CreateSubnet(msg *nats.Msg) {
handleNATSRequest(msg, d.vpcService.CreateSubnet)
}
func (d *Daemon) handleEC2DeleteSubnet(msg *nats.Msg) {
handleNATSRequest(msg, d.vpcService.DeleteSubnet)
}
func (d *Daemon) handleEC2DescribeSubnets(msg *nats.Msg) {
handleNATSRequest(msg, d.vpcService.DescribeSubnets)
}
func (d *Daemon) handleEC2CreateNetworkInterface(msg *nats.Msg) {
handleNATSRequest(msg, d.vpcService.CreateNetworkInterface)
}
func (d *Daemon) handleEC2DeleteNetworkInterface(msg *nats.Msg) {
handleNATSRequest(msg, d.vpcService.DeleteNetworkInterface)
}
func (d *Daemon) handleEC2DescribeNetworkInterfaces(msg *nats.Msg) {
handleNATSRequest(msg, d.vpcService.DescribeNetworkInterfaces)
}
+3493 -75
View File
File diff suppressed because it is too large Load Diff
+242
View File
@@ -0,0 +1,242 @@
package daemon
import (
"encoding/json"
"fmt"
"log/slog"
"regexp"
"sort"
"strconv"
"strings"
"time"
"github.com/mulgadc/hive/hive/qmp"
"github.com/mulgadc/hive/hive/vm"
)
// pciAddrRegexp extracts the device index from a boot-time QDev path.
// Example: "/machine/peripheral-anon/device[0]/virtio-backend" → 0
var pciAddrRegexp = regexp.MustCompile(`device\[(\d+)\]`)
// hotplugPortRegexp extracts the hotplug port number from a hot-plugged QDev path.
// Example: "/machine/peripheral/vdisk-vol-xxx/hotplug3/virtio-backend" or
//
// "/machine/peripheral/vdisk-vol-xxx/virtio-backend" (bus assigned by QEMU)
//
// The chassis number from the QEMU command line determines PCI slot order.
var hotplugPortRegexp = regexp.MustCompile(`hotplug(\d+)`)
// queryGuestDeviceMap uses QMP query-block to build a map from QEMU device ID
// (e.g. "os", "cloudinit", "vdisk-vol-xxx") to the guest device path
// (e.g. "/dev/vda", "/dev/vdb", "/dev/vdc").
//
// The mapping is derived from PCI address order: virtio-blk-pci devices are
// enumerated by the guest kernel in PCI bus order, which corresponds to the
// device index in the QDev path.
func queryGuestDeviceMap(d *Daemon, qmpClient *qmp.QMPClient, instanceID string) (map[string]string, error) {
resp, err := d.SendQMPCommand(qmpClient, qmp.QMPCommand{Execute: "query-block"}, instanceID)
if err != nil {
return nil, fmt.Errorf("query-block failed: %w", err)
}
var devices []qmp.BlockDevice
if err := json.Unmarshal(resp.Return, &devices); err != nil {
return nil, fmt.Errorf("failed to parse query-block response: %w", err)
}
return buildDeviceMap(devices), nil
}
// queryGuestDeviceMapWait retries queryGuestDeviceMap until expectedDevice
// appears in the result. This handles the race where query-block is called
// immediately after device_add, before QEMU has registered the new device.
func queryGuestDeviceMapWait(d *Daemon, qmpClient *qmp.QMPClient, instanceID, expectedDevice string) (map[string]string, error) {
const maxAttempts = 5
const retryDelay = 200 * time.Millisecond
for attempt := range maxAttempts {
deviceMap, err := queryGuestDeviceMap(d, qmpClient, instanceID)
if err != nil {
return nil, err
}
if _, ok := deviceMap[expectedDevice]; ok {
return deviceMap, nil
}
if attempt < maxAttempts-1 {
slog.Debug("queryGuestDeviceMapWait: device not yet visible, retrying",
"expectedDevice", expectedDevice, "attempt", attempt+1, "maxAttempts", maxAttempts)
time.Sleep(retryDelay)
}
}
// Return the last result even though the device wasn't found — caller decides fallback.
return queryGuestDeviceMap(d, qmpClient, instanceID)
}
// buildDeviceMap takes a list of BlockDevices from QMP and returns a map
// from QEMU device ID to guest /dev/vdX path, sorted by PCI address.
//
// Two QDev path formats exist:
// - Boot-time: /machine/peripheral-anon/device[N]/virtio-backend
// device field contains the ID (e.g. "os", "cloudinit")
// - Hot-plugged: /machine/peripheral/<device-id>/virtio-backend
// device field is empty; ID is extracted from the QDev path
func buildDeviceMap(devices []qmp.BlockDevice) map[string]string {
type deviceEntry struct {
name string
pciIndex int
}
var virtioDevices []deviceEntry
for _, dev := range devices {
if dev.QDev == "" || !strings.Contains(dev.QDev, "virtio-backend") {
continue
}
name := dev.Device
pciIndex := extractPCIIndex(dev.QDev)
if pciIndex < 0 {
// Hot-plugged device: extract name and port from QDev path.
// QDev looks like "/machine/peripheral/<name>/virtio-backend"
if name == "" {
name = extractPeripheralName(dev.QDev)
}
if name == "" {
slog.Warn("Could not extract device name from QDev path", "qdev", dev.QDev)
continue
}
// Use hotplug port number for ordering; fall back to a high
// index so hot-plugged devices sort after boot devices.
pciIndex = extractHotplugPort(dev.QDev)
if pciIndex < 0 {
pciIndex = 1000 // arbitrary high value
}
}
virtioDevices = append(virtioDevices, deviceEntry{
name: name,
pciIndex: pciIndex,
})
}
// Sort by PCI index — this determines /dev/vd* letter assignment
sort.Slice(virtioDevices, func(i, j int) bool {
return virtioDevices[i].pciIndex < virtioDevices[j].pciIndex
})
result := make(map[string]string, len(virtioDevices))
for i, entry := range virtioDevices {
if i >= 26 {
slog.Warn("buildDeviceMap: more than 26 virtio devices, cannot map remaining",
"totalDevices", len(virtioDevices))
break
}
guestDev := fmt.Sprintf("/dev/vd%c", 'a'+i)
result[entry.name] = guestDev
}
return result
}
// updateGuestDeviceNames queries the running VM's QMP to discover actual guest device
// paths and updates the instance's BlockDeviceMappings accordingly.
func (d *Daemon) updateGuestDeviceNames(instance *vm.VM) {
if instance.QMPClient == nil {
slog.Warn("updateGuestDeviceNames: QMPClient is nil, cannot discover guest device names",
"instanceId", instance.ID)
return
}
deviceMap, err := queryGuestDeviceMap(d, instance.QMPClient, instance.ID)
if err != nil {
slog.Warn("Failed to query guest device map, BlockDeviceMappings will use API names",
"instanceId", instance.ID, "err", err)
return
}
// Build volume ID → guest device path mapping from EBSRequests.
// Collect under EBSRequests lock, then release before acquiring Instances lock
// to maintain consistent lock ordering.
instance.EBSRequests.Mu.Lock()
volToGuest := make(map[string]string, len(instance.EBSRequests.Requests))
for _, req := range instance.EBSRequests.Requests {
var qemuID string
if req.Boot {
qemuID = "os"
} else if req.CloudInit {
qemuID = "cloudinit"
} else {
qemuID = fmt.Sprintf("vdisk-%s", req.Name)
}
if gd, ok := deviceMap[qemuID]; ok {
volToGuest[req.Name] = gd
}
}
instance.EBSRequests.Mu.Unlock()
// Update BlockDeviceMappings using the mapping
d.Instances.Mu.Lock()
if instance.Instance != nil {
for _, bdm := range instance.Instance.BlockDeviceMappings {
if bdm.Ebs == nil || bdm.Ebs.VolumeId == nil || bdm.DeviceName == nil {
continue
}
if gd, ok := volToGuest[*bdm.Ebs.VolumeId]; ok {
bdm.DeviceName = &gd
}
}
}
d.Instances.Mu.Unlock()
if err := d.WriteState(); err != nil {
slog.Error("Failed to persist state after guest device name update", "instanceId", instance.ID, "err", err)
}
slog.Info("Updated guest device names", "instanceId", instance.ID, "deviceMap", deviceMap)
}
// extractPCIIndex parses the device index from a QDev path.
// For example: "/machine/peripheral-anon/device[3]/virtio-backend" → 3
func extractPCIIndex(qdev string) int {
matches := pciAddrRegexp.FindStringSubmatch(qdev)
if len(matches) < 2 {
return -1
}
idx, err := strconv.Atoi(matches[1])
if err != nil {
return -1
}
return idx
}
// extractPeripheralName extracts the device ID from a hot-plugged QDev path.
// For example: "/machine/peripheral/vdisk-vol-abc123/virtio-backend" → "vdisk-vol-abc123"
func extractPeripheralName(qdev string) string {
// Path format: /machine/peripheral/<name>/virtio-backend
const prefix = "/machine/peripheral/"
if !strings.HasPrefix(qdev, prefix) {
return ""
}
rest := qdev[len(prefix):]
if idx := strings.Index(rest, "/"); idx > 0 {
return rest[:idx]
}
return ""
}
// extractHotplugPort extracts the hotplug port number from a QDev path.
// For example: "/machine/peripheral/vdisk-vol-xxx/hotplug3/virtio-backend" → 3
// Returns -1 if no hotplug port is found.
func extractHotplugPort(qdev string) int {
matches := hotplugPortRegexp.FindStringSubmatch(qdev)
if len(matches) < 2 {
return -1
}
idx, err := strconv.Atoi(matches[1])
if err != nil {
return -1
}
return idx
}
+257
View File
@@ -0,0 +1,257 @@
package daemon
import (
"testing"
"github.com/mulgadc/hive/hive/qmp"
"github.com/stretchr/testify/assert"
)
func TestExtractPCIIndex(t *testing.T) {
tests := []struct {
name string
qdev string
want int
}{
{
name: "standard peripheral-anon device",
qdev: "/machine/peripheral-anon/device[0]/virtio-backend",
want: 0,
},
{
name: "device index 3",
qdev: "/machine/peripheral-anon/device[3]/virtio-backend",
want: 3,
},
{
name: "hotplug device with higher index",
qdev: "/machine/peripheral/hotplug1/device[12]/virtio-backend",
want: 12,
},
{
name: "unattached device path",
qdev: "/machine/unattached/device[24]",
want: 24,
},
{
name: "empty string",
qdev: "",
want: -1,
},
{
name: "no device brackets",
qdev: "/machine/peripheral/something",
want: -1,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := extractPCIIndex(tt.qdev)
assert.Equal(t, tt.want, got)
})
}
}
// TestBuildDeviceMap tests the core device mapping logic using the example
// QMP query-block response from qmp.go:72
func TestBuildDeviceMap(t *testing.T) {
// Simulate the exact QMP response from the example in qmp.go
devices := []qmp.BlockDevice{
{
IOStatus: "ok",
Device: "os",
Locked: false,
Inserted: &qmp.BlockInserted{
Image: qmp.BlockImage{
VirtualSize: 4294967296,
Filename: "nbd://127.0.0.1:44801",
Format: "raw",
},
},
QDev: "/machine/peripheral-anon/device[0]/virtio-backend",
Type: "unknown",
},
{
IOStatus: "ok",
Device: "cloudinit",
Locked: false,
Inserted: &qmp.BlockInserted{
Image: qmp.BlockImage{
VirtualSize: 1048576,
Filename: "nbd://127.0.0.1:42911",
Format: "raw",
},
RO: true,
},
QDev: "/machine/peripheral-anon/device[3]/virtio-backend",
Type: "unknown",
},
{
IOStatus: "ok",
Device: "ide1-cd0",
Locked: false,
Removable: true,
QDev: "/machine/unattached/device[24]",
Type: "unknown",
},
{
Device: "floppy0",
Locked: false,
Removable: true,
QDev: "/machine/unattached/device[18]",
Type: "unknown",
},
{
Device: "sd0",
Locked: false,
Removable: true,
Type: "unknown",
},
}
result := buildDeviceMap(devices)
assert.Equal(t, "/dev/vda", result["os"], "root disk should be /dev/vda")
assert.Equal(t, "/dev/vdb", result["cloudinit"], "cloudinit should be /dev/vdb")
assert.NotContains(t, result, "ide1-cd0", "ide device should be excluded")
assert.NotContains(t, result, "floppy0", "floppy should be excluded")
assert.NotContains(t, result, "sd0", "sd device should be excluded")
assert.Len(t, result, 2, "should only have 2 virtio devices")
}
func TestBuildDeviceMapWithHotplug(t *testing.T) {
// Hot-plugged devices have empty Device field and use
// /machine/peripheral/<id>/virtio-backend QDev path format
devices := []qmp.BlockDevice{
{
Device: "os",
Inserted: &qmp.BlockInserted{},
QDev: "/machine/peripheral-anon/device[0]/virtio-backend",
},
{
Device: "cloudinit",
Inserted: &qmp.BlockInserted{},
QDev: "/machine/peripheral-anon/device[3]/virtio-backend",
},
{
Device: "",
Inserted: &qmp.BlockInserted{},
QDev: "/machine/peripheral/vdisk-vol-abc123/virtio-backend",
},
{
Device: "",
Inserted: &qmp.BlockInserted{},
QDev: "/machine/peripheral/vdisk-vol-def456/virtio-backend",
},
// Legacy devices to filter out
{Device: "floppy0", QDev: "/machine/unattached/device[18]"},
{Device: "sd0"},
{Device: "ide1-cd0", QDev: "/machine/unattached/device[24]"},
}
result := buildDeviceMap(devices)
assert.Equal(t, "/dev/vda", result["os"])
assert.Equal(t, "/dev/vdb", result["cloudinit"])
assert.Equal(t, "/dev/vdc", result["vdisk-vol-abc123"])
assert.Equal(t, "/dev/vdd", result["vdisk-vol-def456"])
assert.Len(t, result, 4)
}
func TestBuildDeviceMapEmpty(t *testing.T) {
result := buildDeviceMap(nil)
assert.Empty(t, result)
}
func TestBuildDeviceMapPCIOrdering(t *testing.T) {
// Boot devices returned out of order + hot-plugged device
devices := []qmp.BlockDevice{
{
Device: "cloudinit",
Inserted: &qmp.BlockInserted{},
QDev: "/machine/peripheral-anon/device[5]/virtio-backend",
},
{
Device: "os",
Inserted: &qmp.BlockInserted{},
QDev: "/machine/peripheral-anon/device[1]/virtio-backend",
},
{
Device: "",
Inserted: &qmp.BlockInserted{},
QDev: "/machine/peripheral/vdisk-vol-123/virtio-backend",
},
}
result := buildDeviceMap(devices)
assert.Equal(t, "/dev/vda", result["os"], "lowest PCI index gets /dev/vda")
assert.Equal(t, "/dev/vdb", result["cloudinit"], "second boot device")
assert.Equal(t, "/dev/vdc", result["vdisk-vol-123"], "hot-plugged device sorts after boot devices")
}
func TestExtractPeripheralName(t *testing.T) {
tests := []struct {
name string
qdev string
want string
}{
{
name: "hot-plugged volume",
qdev: "/machine/peripheral/vdisk-vol-abc123/virtio-backend",
want: "vdisk-vol-abc123",
},
{
name: "boot device path",
qdev: "/machine/peripheral-anon/device[0]/virtio-backend",
want: "",
},
{
name: "empty",
qdev: "",
want: "",
},
{
name: "no trailing slash",
qdev: "/machine/peripheral/",
want: "",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
assert.Equal(t, tt.want, extractPeripheralName(tt.qdev))
})
}
}
func TestExtractHotplugPort(t *testing.T) {
tests := []struct {
name string
qdev string
want int
}{
{
name: "hotplug port 3",
qdev: "/machine/peripheral/vdisk-vol-xxx/hotplug3/virtio-backend",
want: 3,
},
{
name: "no hotplug in path",
qdev: "/machine/peripheral/vdisk-vol-xxx/virtio-backend",
want: -1,
},
{
name: "boot device",
qdev: "/machine/peripheral-anon/device[0]/virtio-backend",
want: -1,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
assert.Equal(t, tt.want, extractHotplugPort(tt.qdev))
})
}
}
+279
View File
@@ -0,0 +1,279 @@
package daemon
import (
"encoding/json"
"errors"
"fmt"
"log/slog"
"os"
"os/exec"
"syscall"
"time"
"github.com/mulgadc/hive/hive/vm"
)
const (
maxRestartsInWindow = 3
restartWindow = 10 * time.Minute
restartBackoffBase = 5 * time.Second
restartBackoffMax = 2 * time.Minute
)
// restartBackoff computes the exponential backoff delay for the given
// restart count. Pure function — no side effects.
func restartBackoff(restartCount int) time.Duration {
delay := restartBackoffBase
for range restartCount {
delay *= 2
if delay > restartBackoffMax {
return restartBackoffMax
}
}
return delay
}
// classifyCrashReason extracts a human-readable crash reason from the error
// returned by cmd.Wait(). Uses exec.ExitError + syscall.WaitStatus to
// distinguish OOM kills (SIGKILL), segfaults (SIGSEGV), etc.
func classifyCrashReason(waitErr error) string {
if waitErr == nil {
return "clean-exit"
}
var exitErr *exec.ExitError
if !errors.As(waitErr, &exitErr) {
return "unknown"
}
status, ok := exitErr.Sys().(syscall.WaitStatus)
if !ok {
return "unknown"
}
if status.Signaled() {
switch status.Signal() {
case syscall.SIGKILL:
return "oom-killed"
case syscall.SIGSEGV:
return "segfault"
case syscall.SIGABRT:
return "abort"
default:
return fmt.Sprintf("signal-%d", status.Signal())
}
}
if status.Exited() {
return fmt.Sprintf("exit-%d", status.ExitStatus())
}
return "unknown"
}
// handleInstanceCrash is called by the QEMU launch goroutine after cmd.Wait()
// returns during runtime (after startup was confirmed successful). It detects
// the crash reason, transitions the instance to error state, cleans up resources,
// and schedules a restart if policy allows.
func (d *Daemon) handleInstanceCrash(instance *vm.VM, waitErr error) {
// Guard: if instance is not running, this was an expected exit
// (stopInstance/terminateInstance set status before QEMU exits)
d.Instances.Mu.Lock()
status := instance.Status
d.Instances.Mu.Unlock()
if status != vm.StateRunning {
slog.Debug("QEMU exited but instance not in running state, skipping crash handler",
"instance", instance.ID, "status", status)
return
}
// Guard: coordinated shutdown in progress
if d.shuttingDown.Load() {
slog.Debug("QEMU exited during coordinated shutdown, skipping crash handler",
"instance", instance.ID)
return
}
reason := classifyCrashReason(waitErr)
slog.Error("VM process crashed", "instance", instance.ID, "reason", reason, "err", waitErr)
// Transition to error state
if err := d.TransitionState(instance, vm.StateError); err != nil {
slog.Error("Failed to transition crashed instance to error state",
"instance", instance.ID, "err", err)
}
// Update health tracking
now := time.Now()
d.Instances.Mu.Lock()
instance.Health.CrashCount++
instance.Health.LastCrashTime = now
instance.Health.LastCrashReason = reason
if instance.Health.FirstCrashTime.IsZero() {
instance.Health.FirstCrashTime = now
}
instance.Running = false
instance.PID = 0
d.Instances.Mu.Unlock()
// Deallocate resources to fix phantom reservation
instanceType, ok := d.resourceMgr.instanceTypes[instance.InstanceType]
if ok && instanceType != nil {
slog.Info("Deallocating resources for crashed instance",
"instance", instance.ID, "type", instance.InstanceType)
d.resourceMgr.deallocate(instanceType)
}
// Clean up stale QMP socket so QEMU can rebind on restart
if instance.Config.QMPSocket != "" {
_ = os.Remove(instance.Config.QMPSocket)
}
// Unmount EBS volumes (same pattern as stopInstance)
d.unmountInstanceVolumes(instance)
// Persist state
if err := d.WriteState(); err != nil {
slog.Error("Failed to persist state after crash handling",
"instance", instance.ID, "err", err)
}
d.maybeRestartInstance(instance)
}
// unmountInstanceVolumes sends NATS unmount requests for all volumes attached
// to the instance and updates their state to "available".
func (d *Daemon) unmountInstanceVolumes(instance *vm.VM) {
instance.EBSRequests.Mu.Lock()
defer instance.EBSRequests.Mu.Unlock()
for _, ebsRequest := range instance.EBSRequests.Requests {
ebsUnMountRequest, err := json.Marshal(ebsRequest)
if err != nil {
slog.Error("Failed to marshal volume payload for crash cleanup",
"err", err)
continue
}
msg, err := d.natsConn.Request(d.ebsTopic("unmount"), ebsUnMountRequest, 30*time.Second)
if err != nil {
slog.Error("Failed to unmount volume after crash",
"name", ebsRequest.Name, "instance", instance.ID, "err", err)
} else {
slog.Info("Unmounted volume after crash",
"instance", instance.ID, "volume", ebsRequest.Name, "data", string(msg.Data))
}
// Update user-visible volume state to "available"
if !ebsRequest.EFI && !ebsRequest.CloudInit {
if err := d.volumeService.UpdateVolumeState(ebsRequest.Name, "available", "", ""); err != nil {
slog.Error("Failed to update volume state to available after crash",
"volumeId", ebsRequest.Name, "err", err)
}
}
}
}
// maybeRestartInstance checks restart policy and schedules a restart if allowed.
func (d *Daemon) maybeRestartInstance(instance *vm.VM) {
if d.shuttingDown.Load() {
slog.Info("Skipping restart during shutdown", "instance", instance.ID)
return
}
now := time.Now()
d.Instances.Mu.Lock()
health := &instance.Health
// If crashes are outside the restart window, reset counters
if !health.FirstCrashTime.IsZero() && now.Sub(health.FirstCrashTime) > restartWindow {
slog.Info("Crash window expired, resetting counters", "instance", instance.ID)
health.CrashCount = 1
health.FirstCrashTime = now
health.RestartCount = 0
}
// Check if we've exceeded the max restarts in the window
if health.CrashCount > maxRestartsInWindow {
d.Instances.Mu.Unlock()
slog.Error("Instance exceeded max restarts in window, leaving in error state",
"instance", instance.ID,
"crashes", health.CrashCount,
"window", restartWindow,
"max", maxRestartsInWindow)
return
}
restartCount := health.RestartCount
d.Instances.Mu.Unlock()
// Check resource availability
instanceType, ok := d.resourceMgr.instanceTypes[instance.InstanceType]
if !ok || instanceType == nil {
slog.Error("Unknown instance type, cannot restart",
"instance", instance.ID, "type", instance.InstanceType)
return
}
if d.resourceMgr.canAllocate(instanceType, 1) < 1 {
slog.Error("Insufficient resources to restart instance",
"instance", instance.ID, "type", instance.InstanceType)
return
}
delay := restartBackoff(restartCount)
slog.Info("Scheduling instance restart",
"instance", instance.ID,
"delay", delay,
"restartCount", restartCount+1)
time.AfterFunc(delay, func() {
d.restartCrashedInstance(instance)
})
}
// restartCrashedInstance re-verifies the instance is still in error state
// and relaunches it via LaunchInstance.
func (d *Daemon) restartCrashedInstance(instance *vm.VM) {
d.Instances.Mu.Lock()
if instance.Status != vm.StateError {
d.Instances.Mu.Unlock()
slog.Info("Instance no longer in error state, skipping restart",
"instance", instance.ID, "status", instance.Status)
return
}
if d.shuttingDown.Load() {
d.Instances.Mu.Unlock()
slog.Info("Daemon shutting down, skipping restart", "instance", instance.ID)
return
}
instance.Health.RestartCount++
d.Instances.Mu.Unlock()
slog.Info("Restarting crashed instance",
"instance", instance.ID,
"restartCount", instance.Health.RestartCount)
// Transition Error → Pending (valid now that we added it to the transitions map)
if err := d.TransitionState(instance, vm.StatePending); err != nil {
slog.Error("Failed to transition instance to pending for restart",
"instance", instance.ID, "err", err)
return
}
// LaunchInstance handles the full relaunch flow:
// check PID → MountVolumes → StartInstance → CreateQMPClient → NATS subscribe → Running
if err := d.LaunchInstance(instance); err != nil {
slog.Error("Failed to restart crashed instance",
"instance", instance.ID, "err", err)
if err := d.TransitionState(instance, vm.StateError); err != nil {
slog.Error("Failed to transition instance back to error after restart failure",
"instance", instance.ID, "err", err)
}
}
}
+465
View File
@@ -0,0 +1,465 @@
package daemon
import (
"fmt"
"os"
"os/exec"
"path/filepath"
"testing"
"time"
"github.com/mulgadc/hive/hive/config"
"github.com/mulgadc/hive/hive/vm"
"github.com/nats-io/nats.go"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestClassifyCrashReason_CleanExit(t *testing.T) {
assert.Equal(t, "clean-exit", classifyCrashReason(nil))
}
func TestClassifyCrashReason_OOMKill(t *testing.T) {
// Start a process and kill it with SIGKILL to get a real ExitError
cmd := exec.Command("sleep", "60")
require.NoError(t, cmd.Start())
require.NoError(t, cmd.Process.Kill()) // sends SIGKILL
waitErr := cmd.Wait()
require.Error(t, waitErr)
assert.Equal(t, "oom-killed", classifyCrashReason(waitErr))
}
func TestClassifyCrashReason_ExitCode(t *testing.T) {
// Run a command that exits with a non-zero code
cmd := exec.Command("sh", "-c", "exit 42")
waitErr := cmd.Run()
require.Error(t, waitErr)
assert.Equal(t, "exit-42", classifyCrashReason(waitErr))
}
func TestClassifyCrashReason_Unknown(t *testing.T) {
// A non-ExitError error
err := fmt.Errorf("some random error")
assert.Equal(t, "unknown", classifyCrashReason(err))
}
func TestHandleInstanceCrash_SkipsNonRunning(t *testing.T) {
nc, err := nats.Connect(sharedNATSURL)
require.NoError(t, err)
defer nc.Close()
d := &Daemon{
natsConn: nc,
resourceMgr: NewResourceManager(),
Instances: vm.Instances{VMS: make(map[string]*vm.VM)},
}
instance := &vm.VM{
ID: "i-test-stopped",
Status: vm.StateStopped,
}
d.Instances.VMS[instance.ID] = instance
// Should return without action (no panic, no state change)
d.handleInstanceCrash(instance, fmt.Errorf("test error"))
assert.Equal(t, vm.StateStopped, instance.Status)
}
func TestHandleInstanceCrash_SkipsShuttingDown(t *testing.T) {
nc, err := nats.Connect(sharedNATSURL)
require.NoError(t, err)
defer nc.Close()
d := &Daemon{
natsConn: nc,
resourceMgr: NewResourceManager(),
Instances: vm.Instances{VMS: make(map[string]*vm.VM)},
}
d.shuttingDown.Store(true)
instance := &vm.VM{
ID: "i-test-shutdown",
Status: vm.StateRunning,
}
d.Instances.VMS[instance.ID] = instance
d.handleInstanceCrash(instance, fmt.Errorf("test error"))
// Status unchanged because shuttingDown is true
assert.Equal(t, vm.StateRunning, instance.Status)
}
func TestMaybeRestart_ExceedsMaxInWindow(t *testing.T) {
nc, err := nats.Connect(sharedNATSURL)
require.NoError(t, err)
defer nc.Close()
d := &Daemon{
node: "test-node",
natsConn: nc,
config: &config.Config{
Services: []string{"daemon"},
},
resourceMgr: NewResourceManager(),
Instances: vm.Instances{VMS: make(map[string]*vm.VM)},
}
// Find an allocatable instance type
var allocType string
for typeName := range d.resourceMgr.instanceTypes {
allocType = typeName
break
}
require.NotEmpty(t, allocType)
instance := &vm.VM{
ID: "i-test-maxcrash",
Status: vm.StateError,
InstanceType: allocType,
Health: vm.InstanceHealthState{
CrashCount: maxRestartsInWindow + 1, // exceeds limit
FirstCrashTime: time.Now(),
RestartCount: maxRestartsInWindow,
},
}
d.Instances.VMS[instance.ID] = instance
// Should not schedule a restart (no panic, instance stays in error)
d.maybeRestartInstance(instance)
// Instance should remain in error state
assert.Equal(t, vm.StateError, instance.Status)
}
func TestMaybeRestart_ResetsAfterWindow(t *testing.T) {
nc, err := nats.Connect(sharedNATSURL)
require.NoError(t, err)
defer nc.Close()
d := &Daemon{
node: "test-node",
natsConn: nc,
config: &config.Config{
Services: []string{"daemon"},
},
resourceMgr: NewResourceManager(),
Instances: vm.Instances{VMS: make(map[string]*vm.VM)},
}
// Find an allocatable instance type
var allocType string
for typeName := range d.resourceMgr.instanceTypes {
allocType = typeName
break
}
require.NotEmpty(t, allocType)
instance := &vm.VM{
ID: "i-test-windowreset",
Status: vm.StateError,
InstanceType: allocType,
Health: vm.InstanceHealthState{
CrashCount: 5,
FirstCrashTime: time.Now().Add(-restartWindow - time.Minute), // outside window
RestartCount: 5,
},
}
d.Instances.VMS[instance.ID] = instance
// maybeRestartInstance should reset counters and schedule a restart
// (restart will fail because there's no real QEMU, but counters should reset)
d.maybeRestartInstance(instance)
// Counters should be reset
assert.Equal(t, 1, instance.Health.CrashCount)
assert.Equal(t, 0, instance.Health.RestartCount)
}
func TestRestartBackoff_Exponential(t *testing.T) {
tests := []struct {
restartCount int
want time.Duration
}{
{0, 5 * time.Second},
{1, 10 * time.Second},
{2, 20 * time.Second},
{3, 40 * time.Second},
{4, 80 * time.Second},
{5, 120 * time.Second}, // capped at 2min
{6, 120 * time.Second}, // stays capped
{100, 120 * time.Second}, // large count stays capped
}
for _, tc := range tests {
got := restartBackoff(tc.restartCount)
assert.Equal(t, tc.want, got, "Backoff mismatch at restart count %d", tc.restartCount)
}
}
// newTestDaemon creates a Daemon with shared NATS and a ResourceManager for health tests.
// jsManager is nil — WriteState will error (logged, not fatal).
func newTestDaemon(t *testing.T) (*Daemon, func()) {
t.Helper()
nc, err := nats.Connect(sharedNATSURL)
require.NoError(t, err)
d := &Daemon{
node: "test-node",
natsConn: nc,
config: &config.Config{
Services: []string{"daemon"},
},
resourceMgr: NewResourceManager(),
Instances: vm.Instances{VMS: make(map[string]*vm.VM)},
}
return d, func() { nc.Close() }
}
// smallestAllocType returns the name of the smallest allocatable instance type
// in the resource manager, considering both vCPU and memory to ensure it fits
// on the current host (important for CI runners with limited resources).
func smallestAllocType(t *testing.T, rm *ResourceManager) string {
t.Helper()
var smallest string
var smallestScore int64 = 1<<63 - 1
for name, it := range rm.instanceTypes {
vcpu := instanceTypeVCPUs(it)
memMiB := instanceTypeMemoryMiB(it)
// Score by total resource footprint so we pick the type smallest in both dimensions
score := vcpu*1024 + memMiB
if score < smallestScore {
smallestScore = score
smallest = name
}
}
if smallest == "" {
t.Fatal("resource manager has no instance types")
}
return smallest
}
func TestHandleInstanceCrash_CoreFlow(t *testing.T) {
d, cleanup := newTestDaemon(t)
defer cleanup()
allocType := smallestAllocType(t, d.resourceMgr)
instanceType := d.resourceMgr.instanceTypes[allocType]
require.NoError(t, d.resourceMgr.allocate(instanceType))
// Record resource levels before crash
d.resourceMgr.mu.RLock()
allocVCPUBefore := d.resourceMgr.allocatedVCPU
allocMemBefore := d.resourceMgr.allocatedMem
d.resourceMgr.mu.RUnlock()
require.Greater(t, allocVCPUBefore, 0)
// Create a temp QMP socket file to verify removal
tmpDir := t.TempDir()
qmpPath := filepath.Join(tmpDir, "qmp.sock")
require.NoError(t, os.WriteFile(qmpPath, []byte("dummy"), 0o600))
instance := &vm.VM{
ID: "i-test-core",
Status: vm.StateRunning,
Running: true,
PID: 12345,
InstanceType: allocType,
Config: vm.Config{QMPSocket: qmpPath},
}
d.Instances.VMS[instance.ID] = instance
d.handleInstanceCrash(instance, fmt.Errorf("test crash"))
// Status transitioned to error
assert.Equal(t, vm.StateError, instance.Status)
// Health tracking updated
assert.Equal(t, 1, instance.Health.CrashCount)
assert.False(t, instance.Health.LastCrashTime.IsZero())
assert.Equal(t, "unknown", instance.Health.LastCrashReason) // fmt.Errorf is not *exec.ExitError
assert.False(t, instance.Health.FirstCrashTime.IsZero())
// Running and PID cleared
assert.False(t, instance.Running)
assert.Equal(t, 0, instance.PID)
// Resources deallocated
d.resourceMgr.mu.RLock()
assert.Less(t, d.resourceMgr.allocatedVCPU, allocVCPUBefore)
assert.Less(t, d.resourceMgr.allocatedMem, allocMemBefore)
d.resourceMgr.mu.RUnlock()
// QMP socket file removed
_, err := os.Stat(qmpPath)
assert.True(t, os.IsNotExist(err), "QMP socket should be removed")
}
func TestHandleInstanceCrash_FirstCrashSetsTime(t *testing.T) {
d, cleanup := newTestDaemon(t)
defer cleanup()
allocType := smallestAllocType(t, d.resourceMgr)
instance := &vm.VM{
ID: "i-test-firstcrash",
Status: vm.StateRunning,
Running: true,
PID: 111,
InstanceType: allocType,
}
d.Instances.VMS[instance.ID] = instance
// First crash
d.handleInstanceCrash(instance, fmt.Errorf("crash 1"))
firstTime := instance.Health.FirstCrashTime
assert.False(t, firstTime.IsZero())
assert.Equal(t, 1, instance.Health.CrashCount)
// Reset to running for a second crash
d.Instances.Mu.Lock()
instance.Status = vm.StateRunning
instance.Running = true
instance.PID = 222
d.Instances.Mu.Unlock()
time.Sleep(time.Millisecond) // ensure time advances
d.handleInstanceCrash(instance, fmt.Errorf("crash 2"))
// FirstCrashTime should be unchanged from the first crash
assert.Equal(t, firstTime, instance.Health.FirstCrashTime)
assert.Equal(t, 2, instance.Health.CrashCount)
}
func TestHandleInstanceCrash_UnknownInstanceType(t *testing.T) {
d, cleanup := newTestDaemon(t)
defer cleanup()
instance := &vm.VM{
ID: "i-test-unknown-type",
Status: vm.StateRunning,
Running: true,
PID: 333,
InstanceType: "z99.nonexistent",
}
d.Instances.VMS[instance.ID] = instance
// Should not panic even though instance type is not in resourceMgr
d.handleInstanceCrash(instance, fmt.Errorf("crash"))
assert.Equal(t, vm.StateError, instance.Status)
assert.Equal(t, 1, instance.Health.CrashCount)
assert.False(t, instance.Running)
assert.Equal(t, 0, instance.PID)
}
func TestMaybeRestart_SkipsShuttingDown(t *testing.T) {
d, cleanup := newTestDaemon(t)
defer cleanup()
d.shuttingDown.Store(true)
allocType := smallestAllocType(t, d.resourceMgr)
instance := &vm.VM{
ID: "i-test-restart-shutdown",
Status: vm.StateError,
InstanceType: allocType,
Health: vm.InstanceHealthState{
CrashCount: 1,
FirstCrashTime: time.Now(),
},
}
d.Instances.VMS[instance.ID] = instance
d.maybeRestartInstance(instance)
// RestartCount should not have changed — restart was skipped
assert.Equal(t, 0, instance.Health.RestartCount)
}
func TestMaybeRestart_UnknownInstanceType(t *testing.T) {
d, cleanup := newTestDaemon(t)
defer cleanup()
instance := &vm.VM{
ID: "i-test-restart-unknown",
Status: vm.StateError,
InstanceType: "z99.nonexistent",
Health: vm.InstanceHealthState{
CrashCount: 1,
FirstCrashTime: time.Now(),
},
}
d.Instances.VMS[instance.ID] = instance
// Should not panic — just log and return
d.maybeRestartInstance(instance)
// No restart scheduled
assert.Equal(t, 0, instance.Health.RestartCount)
}
func TestMaybeRestart_InsufficientResources(t *testing.T) {
d, cleanup := newTestDaemon(t)
defer cleanup()
allocType := smallestAllocType(t, d.resourceMgr)
// Exhaust all resources by setting allocated = available
d.resourceMgr.mu.Lock()
d.resourceMgr.allocatedVCPU = d.resourceMgr.availableVCPU
d.resourceMgr.allocatedMem = d.resourceMgr.availableMem
d.resourceMgr.mu.Unlock()
instance := &vm.VM{
ID: "i-test-restart-nores",
Status: vm.StateError,
InstanceType: allocType,
Health: vm.InstanceHealthState{
CrashCount: 1,
FirstCrashTime: time.Now(),
},
}
d.Instances.VMS[instance.ID] = instance
d.maybeRestartInstance(instance)
// No restart scheduled
assert.Equal(t, 0, instance.Health.RestartCount)
}
func TestMaybeRestart_SchedulesRestart(t *testing.T) {
d, cleanup := newTestDaemon(t)
defer cleanup()
allocType := smallestAllocType(t, d.resourceMgr)
instance := &vm.VM{
ID: "i-test-restart-schedule",
Status: vm.StateError,
InstanceType: allocType,
Health: vm.InstanceHealthState{
CrashCount: 1,
FirstCrashTime: time.Now(),
RestartCount: 0,
},
}
d.Instances.VMS[instance.ID] = instance
// maybeRestartInstance should pass all guards and call time.AfterFunc.
// The scheduled restartCrashedInstance will increment RestartCount and
// attempt LaunchInstance (which will fail without real infra — that's fine).
d.maybeRestartInstance(instance)
// The restart is scheduled via time.AfterFunc with 5s delay.
// We verify that the scheduling code path was reached by checking that
// the function didn't bail out early (all early returns leave RestartCount at 0
// and don't call time.AfterFunc). The counter reset branch does not apply since
// FirstCrashTime is recent. If we got here without panic, the scheduling path ran.
// We can't easily assert the timer fired without waiting 5s, but we can
// verify the counters were not reset (proving we didn't take the window-expired path).
assert.Equal(t, 1, instance.Health.CrashCount)
assert.Equal(t, 0, instance.Health.RestartCount) // incremented by restartCrashedInstance, not maybeRestart
}
+69
View File
@@ -0,0 +1,69 @@
package daemon
import (
"log/slog"
"time"
)
const heartbeatInterval = 10 * time.Second
// startHeartbeat launches a goroutine that publishes this daemon's heartbeat
// to the cluster-state KV store every heartbeatInterval. It fires immediately
// on start, then repeats on a ticker. The goroutine exits when d.ctx is cancelled.
func (d *Daemon) startHeartbeat() {
if d.jsManager == nil {
slog.Warn("JetStream not initialized, skipping heartbeat")
return
}
go func() {
// Fire immediately on startup
d.publishHeartbeat()
ticker := time.NewTicker(heartbeatInterval)
defer ticker.Stop()
for {
select {
case <-d.ctx.Done():
slog.Info("Heartbeat goroutine stopping")
return
case <-ticker.C:
d.publishHeartbeat()
}
}
}()
slog.Info("Heartbeat started", "interval", heartbeatInterval)
}
// publishHeartbeat builds and writes a heartbeat entry to KV.
func (d *Daemon) publishHeartbeat() {
h := d.buildHeartbeat()
if err := d.jsManager.WriteHeartbeat(h); err != nil {
slog.Warn("Failed to publish heartbeat", "error", err)
} else {
slog.Debug("Heartbeat published", "node", h.Node, "vms", h.VMCount)
}
}
// buildHeartbeat constructs a Heartbeat from the daemon's current state.
func (d *Daemon) buildHeartbeat() *Heartbeat {
totalVCPU, totalMem, allocVCPU, allocMem, _ := d.resourceMgr.GetResourceStats()
d.Instances.Mu.Lock()
vmCount := len(d.Instances.VMS)
d.Instances.Mu.Unlock()
return &Heartbeat{
Node: d.node,
Epoch: d.clusterConfig.Epoch,
Timestamp: time.Now().UTC().Format(time.RFC3339),
Services: d.config.GetServices(),
VMCount: vmCount,
AllocatedVCPU: allocVCPU,
AvailableVCPU: totalVCPU - allocVCPU,
AllocatedMem: allocMem,
AvailableMem: totalMem - allocMem,
}
}
+126
View File
@@ -0,0 +1,126 @@
package daemon
import (
"testing"
"github.com/mulgadc/hive/hive/config"
"github.com/mulgadc/hive/hive/vm"
"github.com/nats-io/nats.go"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// TestBuildHeartbeat verifies that buildHeartbeat populates the struct from daemon state.
func TestBuildHeartbeat(t *testing.T) {
d := &Daemon{
node: "test-node",
clusterConfig: &config.ClusterConfig{
Epoch: 5,
},
config: &config.Config{
Services: []string{"daemon", "nats", "viperblock"},
},
resourceMgr: NewResourceManager(),
Instances: vm.Instances{VMS: make(map[string]*vm.VM)},
}
h := d.buildHeartbeat()
assert.Equal(t, "test-node", h.Node)
assert.Equal(t, uint64(5), h.Epoch)
assert.NotEmpty(t, h.Timestamp)
assert.Equal(t, []string{"daemon", "nats", "viperblock"}, h.Services)
assert.Equal(t, 0, h.VMCount)
assert.Equal(t, 0, h.AllocatedVCPU)
assert.Greater(t, h.AvailableVCPU, 0)
assert.Greater(t, h.AvailableMem, 0.0)
}
// TestHeartbeatReflectsAllocation verifies that allocating resources changes the heartbeat values.
func TestHeartbeatReflectsAllocation(t *testing.T) {
d := &Daemon{
node: "alloc-node",
clusterConfig: &config.ClusterConfig{
Epoch: 1,
},
config: &config.Config{
Services: []string{"daemon"},
},
resourceMgr: NewResourceManager(),
Instances: vm.Instances{VMS: make(map[string]*vm.VM)},
}
// Take a heartbeat before allocation
before := d.buildHeartbeat()
// Find an instance type we can allocate
var allocType string
for typeName := range d.resourceMgr.instanceTypes {
if d.resourceMgr.canAllocate(d.resourceMgr.instanceTypes[typeName], 1) >= 1 {
allocType = typeName
break
}
}
require.NotEmpty(t, allocType, "Should have at least one allocatable instance type")
// Allocate resources
err := d.resourceMgr.allocate(d.resourceMgr.instanceTypes[allocType])
require.NoError(t, err)
// Add a VM to the instance map
d.Instances.Mu.Lock()
d.Instances.VMS["i-test-001"] = &vm.VM{
ID: "i-test-001",
Status: vm.StateRunning,
InstanceType: allocType,
}
d.Instances.Mu.Unlock()
// Take a heartbeat after allocation
after := d.buildHeartbeat()
assert.Equal(t, 1, after.VMCount, "Should reflect 1 VM")
assert.Greater(t, after.AllocatedVCPU, before.AllocatedVCPU, "AllocatedVCPU should increase")
assert.Less(t, after.AvailableVCPU, before.AvailableVCPU, "AvailableVCPU should decrease")
}
// TestHeartbeatKVRoundTrip verifies that heartbeat can be written to and read from KV.
func TestHeartbeatKVRoundTrip(t *testing.T) {
nc, err := nats.Connect(sharedJSNATSURL)
require.NoError(t, err)
defer nc.Close()
jsm, err := NewJetStreamManager(nc, 1)
require.NoError(t, err)
err = jsm.InitClusterStateBucket()
require.NoError(t, err)
h := &Heartbeat{
Node: "kv-test-node",
Epoch: 3,
Timestamp: "2025-01-01T00:00:00Z",
Services: []string{"daemon", "nats"},
VMCount: 2,
AllocatedVCPU: 4,
AvailableVCPU: 12,
AllocatedMem: 8.0,
AvailableMem: 24.0,
}
err = jsm.WriteHeartbeat(h)
require.NoError(t, err)
loaded, err := jsm.ReadHeartbeat("kv-test-node")
require.NoError(t, err)
require.NotNil(t, loaded)
assert.Equal(t, h.Node, loaded.Node)
assert.Equal(t, h.Epoch, loaded.Epoch)
assert.Equal(t, h.Timestamp, loaded.Timestamp)
assert.Equal(t, h.Services, loaded.Services)
assert.Equal(t, h.VMCount, loaded.VMCount)
assert.Equal(t, h.AllocatedVCPU, loaded.AllocatedVCPU)
assert.Equal(t, h.AvailableVCPU, loaded.AvailableVCPU)
assert.Equal(t, h.AllocatedMem, loaded.AllocatedMem)
assert.Equal(t, h.AvailableMem, loaded.AvailableMem)
}
+853
View File
@@ -0,0 +1,853 @@
package daemon
import (
"encoding/json"
"errors"
"log/slog"
"strings"
"sync"
"time"
"github.com/mulgadc/hive/hive/vm"
"github.com/nats-io/nats.go"
)
const (
// InstanceStateBucket is the name of the KV bucket for storing instance state
InstanceStateBucket = "hive-instance-state"
// ClusterStateBucket is the name of the KV bucket for cluster state (heartbeats, shutdown markers, service maps)
ClusterStateBucket = "hive-cluster-state"
// InstanceStatePrefix is the key prefix for per-node instance state entries
InstanceStatePrefix = "node."
// StoppedInstancePrefix is the key prefix for stopped instances in shared KV
StoppedInstancePrefix = "instance."
// TerminatedInstanceBucket is the name of the KV bucket for terminated instances (auto-expiry via TTL)
TerminatedInstanceBucket = "hive-terminated-instances"
// TerminatedInstancePrefix is the key prefix for terminated instances
TerminatedInstancePrefix = "terminated."
)
// JetStreamManager manages JetStream KV store operations for instance state
type JetStreamManager struct {
js nats.JetStreamContext
kv nats.KeyValue // hive-instance-state
clusterKV nats.KeyValue // hive-cluster-state
terminatedKV nats.KeyValue // hive-terminated-instances
replicas int
kvMu sync.Mutex // protects kv during recovery
}
// NewJetStreamManager creates a new JetStreamManager from a NATS connection.
// replicas specifies the number of replicas for the KV bucket (typically matches cluster node count).
func NewJetStreamManager(nc *nats.Conn, replicas int) (*JetStreamManager, error) {
js, err := nc.JetStream()
if err != nil {
return nil, err
}
return &JetStreamManager{
js: js,
replicas: replicas,
}, nil
}
// InitKVBucket initializes the KV bucket, creating it if it doesn't exist
func (m *JetStreamManager) InitKVBucket() error {
// Try to get the existing bucket first
kv, err := m.js.KeyValue(InstanceStateBucket)
if err != nil {
if errors.Is(err, nats.ErrBucketNotFound) {
// Bucket doesn't exist, create it
slog.Debug("Creating JetStream KV bucket", "bucket", InstanceStateBucket, "replicas", m.replicas)
kv, err = m.js.CreateKeyValue(&nats.KeyValueConfig{
Bucket: InstanceStateBucket,
Description: "Hive instance state storage",
History: 1, // Only keep latest value
Replicas: m.replicas, // Replication across cluster nodes
})
if err != nil {
return err
}
} else {
return err
}
} else {
slog.Debug("Connected to existing JetStream KV bucket", "bucket", InstanceStateBucket)
}
m.kv = kv
return nil
}
// InitClusterStateBucket initializes the cluster-state KV bucket, creating it if it doesn't exist.
func (m *JetStreamManager) InitClusterStateBucket() error {
kv, err := m.js.KeyValue(ClusterStateBucket)
if err != nil {
if errors.Is(err, nats.ErrBucketNotFound) {
slog.Debug("Creating JetStream KV bucket", "bucket", ClusterStateBucket, "replicas", m.replicas)
kv, err = m.js.CreateKeyValue(&nats.KeyValueConfig{
Bucket: ClusterStateBucket,
Description: "Hive cluster state (heartbeats, shutdown markers, service maps)",
History: 1,
Replicas: m.replicas,
TTL: 1 * time.Hour,
})
if err != nil {
return err
}
} else {
return err
}
} else {
slog.Debug("Connected to existing JetStream KV bucket", "bucket", ClusterStateBucket)
}
m.clusterKV = kv
return nil
}
// InitTerminatedInstanceBucket initializes the terminated-instances KV bucket with a 1-hour TTL.
// JetStream automatically purges keys after 1 hour, matching AWS behavior for terminated instances.
func (m *JetStreamManager) InitTerminatedInstanceBucket() error {
kv, err := m.js.KeyValue(TerminatedInstanceBucket)
if err != nil {
if errors.Is(err, nats.ErrBucketNotFound) {
slog.Debug("Creating JetStream KV bucket", "bucket", TerminatedInstanceBucket, "replicas", m.replicas)
kv, err = m.js.CreateKeyValue(&nats.KeyValueConfig{
Bucket: TerminatedInstanceBucket,
Description: "Terminated instances (auto-expire after 1 hour)",
History: 1,
Replicas: m.replicas,
TTL: 1 * time.Hour,
})
if err != nil {
return err
}
} else {
return err
}
} else {
slog.Debug("Connected to existing JetStream KV bucket", "bucket", TerminatedInstanceBucket)
}
m.terminatedKV = kv
return nil
}
// isStreamUnavailable checks if an error indicates the underlying JetStream stream
// was lost or is unreachable. This can happen during NATS cluster formation when
// streams created with low replication are disrupted by node join/catchup operations.
// Different KV operations surface different errors when the stream is gone:
// - Get/Keys → ErrNoResponders ("no responders available for request")
// - Put/Delete → ErrNoStreamResponse ("no response from stream")
// - Direct stream queries → ErrStreamNotFound ("stream not found")
func isStreamUnavailable(err error) bool {
if err == nil {
return false
}
if errors.Is(err, nats.ErrStreamNotFound) ||
errors.Is(err, nats.ErrNoStreamResponse) ||
errors.Is(err, nats.ErrNoResponders) {
return true
}
return strings.Contains(err.Error(), "stream not found")
}
// recoverBucket attempts to reconnect to or re-create a KV bucket after the
// underlying JetStream stream was lost during cluster formation.
// Returns the recovered KV handle directly so callers avoid a racy re-read.
func (m *JetStreamManager) recoverBucket(cfg *nats.KeyValueConfig, field *nats.KeyValue) (nats.KeyValue, error) {
m.kvMu.Lock()
defer m.kvMu.Unlock()
// Try to reconnect to existing bucket first (another goroutine may have recovered it)
kv, err := m.js.KeyValue(cfg.Bucket)
if err == nil {
*field = kv
slog.Info("Reconnected to KV bucket", "bucket", cfg.Bucket)
return kv, nil
}
if !errors.Is(err, nats.ErrBucketNotFound) && !isStreamUnavailable(err) {
return nil, err
}
// Bucket truly doesn't exist — recreate it
slog.Warn("KV bucket stream lost, recreating", "bucket", cfg.Bucket, "replicas", m.replicas)
cfg.History = 1
cfg.Replicas = m.replicas
kv, err = m.js.CreateKeyValue(cfg)
if err != nil {
slog.Error("Failed to recreate KV bucket", "bucket", cfg.Bucket, "err", err)
return nil, err
}
*field = kv
slog.Info("KV bucket recreated successfully", "bucket", cfg.Bucket)
return kv, nil
}
func (m *JetStreamManager) recoverKVBucket() (nats.KeyValue, error) {
return m.recoverBucket(&nats.KeyValueConfig{
Bucket: InstanceStateBucket,
Description: "Hive instance state storage",
}, &m.kv)
}
func (m *JetStreamManager) recoverTerminatedKVBucket() (nats.KeyValue, error) {
return m.recoverBucket(&nats.KeyValueConfig{
Bucket: TerminatedInstanceBucket,
Description: "Terminated instances (auto-expire after 1 hour)",
TTL: 1 * time.Hour,
}, &m.terminatedKV)
}
// Heartbeat represents a daemon's periodic health status published to cluster KV.
type Heartbeat struct {
Node string `json:"node"`
Epoch uint64 `json:"epoch"`
Timestamp string `json:"timestamp"`
Services []string `json:"services"`
VMCount int `json:"vm_count"`
AllocatedVCPU int `json:"allocated_vcpu"`
AvailableVCPU int `json:"available_vcpu"`
AllocatedMem float64 `json:"allocated_mem_gb"`
AvailableMem float64 `json:"available_mem_gb"`
}
// WriteHeartbeat writes a heartbeat entry for the given node to the cluster-state KV.
func (m *JetStreamManager) WriteHeartbeat(h *Heartbeat) error {
if m.clusterKV == nil {
return errors.New("cluster state KV not initialized")
}
data, err := json.Marshal(h)
if err != nil {
return err
}
_, err = m.clusterKV.Put("heartbeat."+h.Node, data)
return err
}
// ReadHeartbeat reads the heartbeat entry for the given node from the cluster-state KV.
func (m *JetStreamManager) ReadHeartbeat(nodeID string) (*Heartbeat, error) {
if m.clusterKV == nil {
return nil, errors.New("cluster state KV not initialized")
}
entry, err := m.clusterKV.Get("heartbeat." + nodeID)
if err != nil {
return nil, err
}
var h Heartbeat
if err := json.Unmarshal(entry.Value(), &h); err != nil {
return nil, err
}
return &h, nil
}
// ClusterShutdownState tracks the coordinated cluster shutdown progress in KV.
type ClusterShutdownState struct {
Initiator string `json:"initiator"`
Phase string `json:"phase"`
Started string `json:"started"`
Timeout string `json:"timeout"`
Force bool `json:"force"`
NodesTotal int `json:"nodes_total"`
NodesAcked map[string]string `json:"nodes_acked"`
}
// WriteClusterShutdown writes the cluster shutdown state to KV.
func (m *JetStreamManager) WriteClusterShutdown(state *ClusterShutdownState) error {
if m.clusterKV == nil {
return errors.New("cluster state KV not initialized")
}
data, err := json.Marshal(state)
if err != nil {
return err
}
_, err = m.clusterKV.Put("cluster.shutdown", data)
return err
}
// ReadClusterShutdown reads the cluster shutdown state from KV.
func (m *JetStreamManager) ReadClusterShutdown() (*ClusterShutdownState, error) {
if m.clusterKV == nil {
return nil, errors.New("cluster state KV not initialized")
}
entry, err := m.clusterKV.Get("cluster.shutdown")
if err != nil {
return nil, err
}
var state ClusterShutdownState
if err := json.Unmarshal(entry.Value(), &state); err != nil {
return nil, err
}
return &state, nil
}
// DeleteClusterShutdown removes the cluster shutdown state from KV.
func (m *JetStreamManager) DeleteClusterShutdown() error {
if m.clusterKV == nil {
return errors.New("cluster state KV not initialized")
}
err := m.clusterKV.Delete("cluster.shutdown")
if err != nil && !errors.Is(err, nats.ErrKeyNotFound) {
return err
}
return nil
}
// WriteShutdownMarker writes a shutdown marker for the given node to the cluster-state KV.
func (m *JetStreamManager) WriteShutdownMarker(nodeID string) error {
if m.clusterKV == nil {
return errors.New("cluster state KV not initialized")
}
data, _ := json.Marshal(map[string]any{
"node": nodeID,
"timestamp": time.Now().UTC().Format(time.RFC3339),
})
_, err := m.clusterKV.Put("shutdown."+nodeID, data)
return err
}
// ReadShutdownMarker checks if a clean shutdown marker exists for the given node.
func (m *JetStreamManager) ReadShutdownMarker(nodeID string) (bool, error) {
if m.clusterKV == nil {
return false, errors.New("cluster state KV not initialized")
}
_, err := m.clusterKV.Get("shutdown." + nodeID)
if errors.Is(err, nats.ErrKeyNotFound) {
return false, nil
}
if err != nil {
return false, err
}
return true, nil
}
// DeleteShutdownMarker removes the shutdown marker for the given node.
func (m *JetStreamManager) DeleteShutdownMarker(nodeID string) error {
if m.clusterKV == nil {
return errors.New("cluster state KV not initialized")
}
err := m.clusterKV.Delete("shutdown." + nodeID)
if err != nil && !errors.Is(err, nats.ErrKeyNotFound) {
return err
}
return nil
}
// WriteServiceManifest writes the service manifest for the given node to the cluster-state KV.
func (m *JetStreamManager) WriteServiceManifest(nodeID string, services []string, natsHost, predastoreHost string) error {
if m.clusterKV == nil {
return errors.New("cluster state KV not initialized")
}
data, _ := json.Marshal(map[string]any{
"node": nodeID,
"services": services,
"nats_host": natsHost,
"predastore_host": predastoreHost,
"timestamp": time.Now().UTC().Format(time.RFC3339),
})
_, err := m.clusterKV.Put("node."+nodeID+".services", data)
return err
}
// WriteState writes the instance state to the KV store for the given node.
// It acquires instances.Mu internally.
func (m *JetStreamManager) WriteState(nodeID string, instances *vm.Instances) error {
instances.Mu.Lock()
defer instances.Mu.Unlock()
if m.kv == nil {
return errors.New("KV bucket not initialized")
}
// Create a struct without the mutex to avoid copying the lock
state := struct {
VMS map[string]*vm.VM `json:"vms"`
}{
VMS: instances.VMS,
}
jsonData, err := json.Marshal(state)
if err != nil {
return err
}
key := InstanceStatePrefix + nodeID
_, err = m.kv.Put(key, jsonData)
if err != nil {
if isStreamUnavailable(err) {
slog.Warn("KV stream unavailable, attempting recovery", "operation", "WriteState", "key", key, "err", err)
kv, recoverErr := m.recoverKVBucket()
if recoverErr != nil {
return err
}
if _, retryErr := kv.Put(key, jsonData); retryErr != nil {
return retryErr
}
slog.Debug("Wrote state to JetStream KV (after recovery)", "key", key, "instances", len(instances.VMS))
return nil
}
return err
}
slog.Debug("Wrote state to JetStream KV", "key", key, "instances", len(instances.VMS))
return nil
}
// LoadState loads the instance state from the KV store for the given node
func (m *JetStreamManager) LoadState(nodeID string) (*vm.Instances, error) {
if m.kv == nil {
return nil, errors.New("KV bucket not initialized")
}
key := InstanceStatePrefix + nodeID
entry, err := m.kv.Get(key)
if err != nil {
if errors.Is(err, nats.ErrKeyNotFound) {
slog.Debug("No existing state in JetStream KV, returning empty state", "key", key)
return &vm.Instances{VMS: make(map[string]*vm.VM)}, nil
}
if isStreamUnavailable(err) {
slog.Warn("KV stream unavailable, attempting recovery", "operation", "LoadState", "key", key, "err", err)
kv, recoverErr := m.recoverKVBucket()
if recoverErr != nil {
return nil, err
}
// Retry the read — if we reconnected, data may still exist
entry, err = kv.Get(key)
if err != nil {
if errors.Is(err, nats.ErrKeyNotFound) {
slog.Warn("No state found after KV recovery", "key", key)
return &vm.Instances{VMS: make(map[string]*vm.VM)}, nil
}
return nil, err
}
// Fall through to unmarshal below
} else {
return nil, err
}
}
var instances vm.Instances
if err := json.Unmarshal(entry.Value(), &instances); err != nil {
return nil, err
}
// Ensure the VMS map is initialized
if instances.VMS == nil {
instances.VMS = make(map[string]*vm.VM)
}
slog.Debug("Loaded state from JetStream KV", "key", key, "instances", len(instances.VMS))
return &instances, nil
}
// DeleteState removes the instance state from the KV store for the given node
func (m *JetStreamManager) DeleteState(nodeID string) error {
if m.kv == nil {
return errors.New("KV bucket not initialized")
}
key := InstanceStatePrefix + nodeID
err := m.kv.Delete(key)
if err != nil && !errors.Is(err, nats.ErrKeyNotFound) {
if isStreamUnavailable(err) {
slog.Warn("KV stream unavailable, attempting recovery", "operation", "DeleteState", "key", key, "err", err)
kv, recoverErr := m.recoverKVBucket()
if recoverErr != nil {
return err
}
// Retry — if we reconnected the key may still exist
if retryErr := kv.Delete(key); retryErr != nil && !errors.Is(retryErr, nats.ErrKeyNotFound) {
return retryErr
}
return nil
}
return err
}
slog.Debug("Deleted state from JetStream KV", "key", key)
return nil
}
// UpdateReplicas updates the replica count for the KV bucket's underlying stream.
// This should be called when new nodes join the cluster.
// Note: Increasing replicas requires the new replica count of NATS servers to be available.
func (m *JetStreamManager) UpdateReplicas(newReplicas int) error {
if m.js == nil {
return errors.New("JetStream context not initialized")
}
// KV buckets are backed by streams with name "KV_<bucket>"
streamName := "KV_" + InstanceStateBucket
// Get current stream info
streamInfo, err := m.js.StreamInfo(streamName)
if err != nil {
return err
}
currentReplicas := streamInfo.Config.Replicas
if currentReplicas >= newReplicas {
slog.Debug("Stream already has sufficient replicas", "current", currentReplicas, "requested", newReplicas)
return nil
}
// Update the stream config with new replica count
streamInfo.Config.Replicas = newReplicas
_, err = m.js.UpdateStream(&streamInfo.Config)
if err != nil {
return err
}
m.replicas = newReplicas
slog.Info("Updated JetStream KV bucket replicas", "bucket", InstanceStateBucket, "oldReplicas", currentReplicas, "newReplicas", newReplicas)
// Also update the cluster-state bucket if it exists
if m.clusterKV != nil {
clusterStreamName := "KV_" + ClusterStateBucket
clusterInfo, err := m.js.StreamInfo(clusterStreamName)
if err == nil && clusterInfo.Config.Replicas < newReplicas {
clusterInfo.Config.Replicas = newReplicas
if _, err := m.js.UpdateStream(&clusterInfo.Config); err != nil {
slog.Warn("Failed to update cluster-state bucket replicas", "error", err)
} else {
slog.Info("Updated cluster-state bucket replicas", "bucket", ClusterStateBucket, "newReplicas", newReplicas)
}
}
}
// Also update the terminated-instances bucket if it exists
if m.terminatedKV != nil {
terminatedStreamName := "KV_" + TerminatedInstanceBucket
terminatedInfo, err := m.js.StreamInfo(terminatedStreamName)
if err == nil && terminatedInfo.Config.Replicas < newReplicas {
terminatedInfo.Config.Replicas = newReplicas
if _, err := m.js.UpdateStream(&terminatedInfo.Config); err != nil {
slog.Warn("Failed to update terminated-instances bucket replicas", "error", err)
} else {
slog.Info("Updated terminated-instances bucket replicas", "bucket", TerminatedInstanceBucket, "newReplicas", newReplicas)
}
}
}
return nil
}
// WriteStoppedInstance writes a stopped instance to the shared KV store.
func (m *JetStreamManager) WriteStoppedInstance(instanceID string, instance *vm.VM) error {
if m.kv == nil {
return errors.New("KV bucket not initialized")
}
jsonData, err := json.Marshal(instance)
if err != nil {
return err
}
key := StoppedInstancePrefix + instanceID
_, err = m.kv.Put(key, jsonData)
if err != nil {
if isStreamUnavailable(err) {
slog.Warn("KV stream unavailable, attempting recovery", "operation", "WriteStoppedInstance", "key", key, "err", err)
kv, recoverErr := m.recoverKVBucket()
if recoverErr != nil {
return err
}
if _, retryErr := kv.Put(key, jsonData); retryErr != nil {
return retryErr
}
slog.Debug("Wrote stopped instance to JetStream KV (after recovery)", "key", key, "instanceId", instanceID)
return nil
}
return err
}
slog.Debug("Wrote stopped instance to JetStream KV", "key", key, "instanceId", instanceID)
return nil
}
// LoadStoppedInstance loads a stopped instance from the shared KV store.
// Returns nil, nil if the key does not exist.
func (m *JetStreamManager) LoadStoppedInstance(instanceID string) (*vm.VM, error) {
if m.kv == nil {
return nil, errors.New("KV bucket not initialized")
}
key := StoppedInstancePrefix + instanceID
entry, err := m.kv.Get(key)
if err != nil {
if errors.Is(err, nats.ErrKeyNotFound) {
return nil, nil
}
if isStreamUnavailable(err) {
slog.Warn("KV stream unavailable, attempting recovery", "operation", "LoadStoppedInstance", "key", key, "err", err)
kv, recoverErr := m.recoverKVBucket()
if recoverErr != nil {
return nil, err
}
// Retry — if we reconnected, data may still exist
entry, err = kv.Get(key)
if err != nil {
if errors.Is(err, nats.ErrKeyNotFound) {
return nil, nil
}
return nil, err
}
// Fall through to unmarshal below
} else {
return nil, err
}
}
var instance vm.VM
if err := json.Unmarshal(entry.Value(), &instance); err != nil {
return nil, err
}
return &instance, nil
}
// DeleteStoppedInstance removes a stopped instance from the shared KV store.
// It is idempotent — deleting a non-existent key is not an error.
func (m *JetStreamManager) DeleteStoppedInstance(instanceID string) error {
if m.kv == nil {
return errors.New("KV bucket not initialized")
}
key := StoppedInstancePrefix + instanceID
err := m.kv.Delete(key)
if err != nil && !errors.Is(err, nats.ErrKeyNotFound) {
if isStreamUnavailable(err) {
slog.Warn("KV stream unavailable, attempting recovery", "operation", "DeleteStoppedInstance", "key", key, "err", err)
kv, recoverErr := m.recoverKVBucket()
if recoverErr != nil {
return err
}
// Retry — if we reconnected the key may still exist
if retryErr := kv.Delete(key); retryErr != nil && !errors.Is(retryErr, nats.ErrKeyNotFound) {
return retryErr
}
return nil
}
return err
}
slog.Debug("Deleted stopped instance from JetStream KV", "key", key)
return nil
}
// ListStoppedInstances returns all stopped instances from the shared KV store.
func (m *JetStreamManager) ListStoppedInstances() ([]*vm.VM, error) {
if m.kv == nil {
return nil, errors.New("KV bucket not initialized")
}
keys, err := m.kv.Keys()
if err != nil {
if errors.Is(err, nats.ErrNoKeysFound) {
return nil, nil
}
if isStreamUnavailable(err) {
slog.Warn("KV stream unavailable, attempting recovery", "operation", "ListStoppedInstances", "err", err)
kv, recoverErr := m.recoverKVBucket()
if recoverErr != nil {
return nil, err
}
// Retry — if we reconnected, data may still exist
keys, err = kv.Keys()
if err != nil {
if errors.Is(err, nats.ErrNoKeysFound) {
return nil, nil
}
return nil, err
}
// Fall through to iterate keys below
} else {
return nil, err
}
}
var instances []*vm.VM
for _, key := range keys {
if !strings.HasPrefix(key, StoppedInstancePrefix) {
continue
}
entry, err := m.kv.Get(key)
if err != nil {
if errors.Is(err, nats.ErrKeyNotFound) {
continue
}
return nil, err
}
var instance vm.VM
if err := json.Unmarshal(entry.Value(), &instance); err != nil {
slog.Error("Failed to unmarshal stopped instance", "key", key, "err", err)
continue
}
instances = append(instances, &instance)
}
return instances, nil
}
// WriteTerminatedInstance writes a terminated instance to the terminated KV bucket.
// The entry will auto-expire after the bucket's TTL (1 hour).
func (m *JetStreamManager) WriteTerminatedInstance(instanceID string, instance *vm.VM) error {
if m.terminatedKV == nil {
return errors.New("terminated instance KV bucket not initialized")
}
jsonData, err := json.Marshal(instance)
if err != nil {
return err
}
key := TerminatedInstancePrefix + instanceID
_, err = m.terminatedKV.Put(key, jsonData)
if err != nil {
if isStreamUnavailable(err) {
slog.Warn("KV stream unavailable, attempting recovery", "operation", "WriteTerminatedInstance", "key", key, "err", err)
kv, recoverErr := m.recoverTerminatedKVBucket()
if recoverErr != nil {
return err
}
if _, retryErr := kv.Put(key, jsonData); retryErr != nil {
return retryErr
}
slog.Debug("Wrote terminated instance to JetStream KV (after recovery)", "key", key, "instanceId", instanceID)
return nil
}
return err
}
slog.Debug("Wrote terminated instance to JetStream KV", "key", key, "instanceId", instanceID)
return nil
}
// ListTerminatedInstances returns all terminated instances from the terminated KV bucket.
func (m *JetStreamManager) ListTerminatedInstances() ([]*vm.VM, error) {
if m.terminatedKV == nil {
return nil, errors.New("terminated instance KV bucket not initialized")
}
keys, err := m.terminatedKV.Keys()
if err != nil {
if errors.Is(err, nats.ErrNoKeysFound) {
return nil, nil
}
if isStreamUnavailable(err) {
slog.Warn("KV stream unavailable, attempting recovery", "operation", "ListTerminatedInstances", "err", err)
kv, recoverErr := m.recoverTerminatedKVBucket()
if recoverErr != nil {
return nil, err
}
keys, err = kv.Keys()
if err != nil {
if errors.Is(err, nats.ErrNoKeysFound) {
return nil, nil
}
return nil, err
}
} else {
return nil, err
}
}
var instances []*vm.VM
for _, key := range keys {
if !strings.HasPrefix(key, TerminatedInstancePrefix) {
continue
}
entry, err := m.terminatedKV.Get(key)
if err != nil {
if errors.Is(err, nats.ErrKeyNotFound) {
continue
}
return nil, err
}
var instance vm.VM
if err := json.Unmarshal(entry.Value(), &instance); err != nil {
slog.Error("Failed to unmarshal terminated instance", "key", key, "err", err)
continue
}
instances = append(instances, &instance)
}
return instances, nil
}
// DeleteTerminatedInstance removes a terminated instance from the terminated KV bucket.
func (m *JetStreamManager) DeleteTerminatedInstance(instanceID string) error {
if m.terminatedKV == nil {
return errors.New("terminated instance KV bucket not initialized")
}
key := TerminatedInstancePrefix + instanceID
err := m.terminatedKV.Delete(key)
if err != nil && !errors.Is(err, nats.ErrKeyNotFound) {
if isStreamUnavailable(err) {
slog.Warn("KV stream unavailable, attempting recovery", "operation", "DeleteTerminatedInstance", "key", key, "err", err)
kv, recoverErr := m.recoverTerminatedKVBucket()
if recoverErr != nil {
return err
}
if retryErr := kv.Delete(key); retryErr != nil && !errors.Is(retryErr, nats.ErrKeyNotFound) {
return retryErr
}
return nil
}
return err
}
slog.Debug("Deleted terminated instance from JetStream KV", "key", key)
return nil
}
// LoadTerminatedInstance loads a single terminated instance from the terminated KV bucket.
// Returns nil, nil if the key does not exist.
func (m *JetStreamManager) LoadTerminatedInstance(instanceID string) (*vm.VM, error) {
if m.terminatedKV == nil {
return nil, errors.New("terminated instance KV bucket not initialized")
}
key := TerminatedInstancePrefix + instanceID
entry, err := m.terminatedKV.Get(key)
if err != nil {
if errors.Is(err, nats.ErrKeyNotFound) {
return nil, nil
}
if isStreamUnavailable(err) {
slog.Warn("KV stream unavailable, attempting recovery", "operation", "LoadTerminatedInstance", "key", key, "err", err)
kv, recoverErr := m.recoverTerminatedKVBucket()
if recoverErr != nil {
return nil, err
}
entry, err = kv.Get(key)
if err != nil {
if errors.Is(err, nats.ErrKeyNotFound) {
return nil, nil
}
return nil, err
}
} else {
return nil, err
}
}
var instance vm.VM
if err := json.Unmarshal(entry.Value(), &instance); err != nil {
return nil, err
}
return &instance, nil
}
File diff suppressed because it is too large Load Diff
+73
View File
@@ -0,0 +1,73 @@
package daemon
import (
"fmt"
"os"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
)
// Shared NATS servers for all daemon tests — started once in TestMain.
var (
sharedNATSServer *server.Server
sharedNATSURL string
sharedJSServer *server.Server
sharedJSNATSURL string
)
func TestMain(m *testing.M) {
// Start a plain NATS server (no JetStream)
ns, err := server.NewServer(&server.Options{
Host: "127.0.0.1",
Port: -1,
NoLog: true,
NoSigs: true,
})
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to create shared NATS server: %v\n", err)
os.Exit(1)
}
go ns.Start()
if !ns.ReadyForConnections(5 * time.Second) {
fmt.Fprintln(os.Stderr, "Shared NATS server failed to start")
os.Exit(1)
}
sharedNATSServer = ns
sharedNATSURL = ns.ClientURL()
// Start a JetStream-enabled NATS server
jsTmpDir, err := os.MkdirTemp("", "nats-js-shared-*")
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to create JetStream temp dir: %v\n", err)
os.Exit(1)
}
jsNS, err := server.NewServer(&server.Options{
Host: "127.0.0.1",
Port: -1,
JetStream: true,
StoreDir: jsTmpDir,
NoLog: true,
NoSigs: true,
})
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to create shared JetStream NATS server: %v\n", err)
os.Exit(1)
}
go jsNS.Start()
if !jsNS.ReadyForConnections(5 * time.Second) {
fmt.Fprintln(os.Stderr, "Shared JetStream NATS server failed to start")
os.Exit(1)
}
sharedJSServer = jsNS
sharedJSNATSURL = jsNS.ClientURL()
code := m.Run()
ns.Shutdown()
jsNS.Shutdown()
os.RemoveAll(jsTmpDir)
os.Exit(code)
}
+275
View File
@@ -0,0 +1,275 @@
package daemon
import (
"fmt"
"log/slog"
"net"
"os"
"os/exec"
"strings"
)
// sudoCommand wraps exec.Command with sudo when running as non-root.
// OVS/OVN and ip commands require elevated privileges; in Docker and
// production the daemon runs as root, but in dev environments it may not.
func sudoCommand(name string, args ...string) *exec.Cmd {
if os.Getuid() == 0 {
return exec.Command(name, args...)
}
return exec.Command("sudo", append([]string{name}, args...)...)
}
// NetworkPlumber handles tap device and OVS bridge operations for VM networking.
// The live implementation runs system commands (ip, ovs-vsctl); tests use a mock.
type NetworkPlumber interface {
// SetupTapDevice creates a tap device and adds it to the OVS br-int bridge
// with the correct iface-id for OVN port binding.
SetupTapDevice(eniId, mac string) error
// CleanupTapDevice removes the tap device from br-int and deletes it.
CleanupTapDevice(eniId string) error
}
// OVSNetworkPlumber implements NetworkPlumber using system commands.
type OVSNetworkPlumber struct{}
func (p *OVSNetworkPlumber) SetupTapDevice(eniId, mac string) error {
tapName := TapDeviceName(eniId)
ifaceID := OVSIfaceID(eniId)
// 0. If tap already exists (e.g. unclean shutdown), clean it up first
if _, err := os.Stat("/sys/class/net/" + tapName); err == nil {
slog.Warn("Stale tap device found, cleaning up before recreate", "tap", tapName)
_ = sudoCommand("ovs-vsctl", "--if-exists", "del-port", "br-int", tapName).Run()
_ = sudoCommand("ip", "tuntap", "del", "dev", tapName, "mode", "tap").Run()
}
// 1. Create tap device
if out, err := sudoCommand("ip", "tuntap", "add", "dev", tapName, "mode", "tap").CombinedOutput(); err != nil {
return fmt.Errorf("create tap %s: %s: %w", tapName, strings.TrimSpace(string(out)), err)
}
// 2. Bring tap up
if out, err := sudoCommand("ip", "link", "set", tapName, "up").CombinedOutput(); err != nil {
_ = sudoCommand("ip", "tuntap", "del", "dev", tapName, "mode", "tap").Run()
return fmt.Errorf("bring up tap %s: %s: %w", tapName, strings.TrimSpace(string(out)), err)
}
// 3. Add to br-int with iface-id for OVN port binding
if out, err := sudoCommand("ovs-vsctl",
"add-port", "br-int", tapName,
"--", "set", "Interface", tapName,
fmt.Sprintf("external_ids:iface-id=%s", ifaceID),
fmt.Sprintf("external_ids:attached-mac=%s", mac),
).CombinedOutput(); err != nil {
_ = sudoCommand("ip", "tuntap", "del", "dev", tapName, "mode", "tap").Run()
return fmt.Errorf("add tap to br-int: %s: %w", strings.TrimSpace(string(out)), err)
}
slog.Info("Network plumbing complete", "tap", tapName, "iface-id", ifaceID, "mac", mac)
return nil
}
func (p *OVSNetworkPlumber) CleanupTapDevice(eniId string) error {
tapName := TapDeviceName(eniId)
// 1. Remove from br-int (--if-exists avoids error if already gone)
if out, err := sudoCommand("ovs-vsctl", "--if-exists", "del-port", "br-int", tapName).CombinedOutput(); err != nil {
slog.Warn("Failed to remove tap from br-int", "tap", tapName, "err", err, "out", strings.TrimSpace(string(out)))
}
// 2. Delete tap device
if out, err := sudoCommand("ip", "tuntap", "del", "dev", tapName, "mode", "tap").CombinedOutput(); err != nil {
return fmt.Errorf("delete tap %s: %s: %w", tapName, strings.TrimSpace(string(out)), err)
}
slog.Info("Network cleanup complete", "tap", tapName)
return nil
}
// TapDeviceName returns the Linux tap device name for an ENI.
// Linux IFNAMSIZ limits interface names to 15 characters.
// ENI IDs like "eni-abc123def456789" are too long, so we truncate.
func TapDeviceName(eniId string) string {
id := strings.TrimPrefix(eniId, "eni-")
name := "tap" + id
if len(name) > 15 {
name = name[:15]
}
return name
}
// OVSIfaceID returns the OVS external_ids:iface-id value for an ENI.
// This must match the OVN LogicalSwitchPort name for ovn-controller binding.
func OVSIfaceID(eniId string) string {
return "port-" + eniId
}
// generateDevMAC creates a locally-administered unicast MAC for the dev/hostfwd NIC.
// Uses prefix 02:de:00 to distinguish from ENI MACs (02:00:00).
// All octets must be valid hex for QEMU's virtio-net-pci mac property.
func generateDevMAC(instanceId string) string {
h := uint32(0)
for _, c := range instanceId {
h = h*31 + uint32(c) // #nosec G115 -- intentional overflow for hashing
}
return fmt.Sprintf("02:de:00:%02x:%02x:%02x", (h>>16)&0xff, (h>>8)&0xff, h&0xff)
}
// OVNHealthStatus reports the readiness of OVN networking on this compute node.
type OVNHealthStatus struct {
BrIntExists bool `json:"br_int_exists"`
OVNControllerUp bool `json:"ovn_controller_up"`
ChassisID string `json:"chassis_id,omitempty"`
EncapIP string `json:"encap_ip,omitempty"`
OVNRemote string `json:"ovn_remote,omitempty"`
}
// CheckOVNHealth probes local OVS/OVN state to determine network readiness.
func CheckOVNHealth() OVNHealthStatus {
status := OVNHealthStatus{}
// Check br-int exists
if err := sudoCommand("ovs-vsctl", "br-exists", "br-int").Run(); err == nil {
status.BrIntExists = true
}
// Check ovn-controller is running via ovs-appctl (more reliable than pgrep)
if out, err := sudoCommand("ovs-appctl", "-t", "ovn-controller", "version").CombinedOutput(); err == nil && len(out) > 0 {
status.OVNControllerUp = true
}
// Read chassis identity from OVS external_ids
if out, err := sudoCommand("ovs-vsctl", "get", "Open_vSwitch", ".", "external_ids:system-id").CombinedOutput(); err == nil {
status.ChassisID = strings.Trim(strings.TrimSpace(string(out)), "\"")
}
if out, err := sudoCommand("ovs-vsctl", "get", "Open_vSwitch", ".", "external_ids:ovn-encap-ip").CombinedOutput(); err == nil {
status.EncapIP = strings.Trim(strings.TrimSpace(string(out)), "\"")
}
if out, err := sudoCommand("ovs-vsctl", "get", "Open_vSwitch", ".", "external_ids:ovn-remote").CombinedOutput(); err == nil {
status.OVNRemote = strings.Trim(strings.TrimSpace(string(out)), "\"")
}
return status
}
// SetupComputeNode configures OVS for OVN on this compute node.
// It creates br-int with secure fail-mode and sets the OVN external_ids.
func SetupComputeNode(chassisID, ovnRemote, encapIP string) error {
// Create br-int if it doesn't exist
if out, err := sudoCommand("ovs-vsctl", "--may-exist", "add-br", "br-int").CombinedOutput(); err != nil {
return fmt.Errorf("create br-int: %s: %w", strings.TrimSpace(string(out)), err)
}
// Set fail-mode=secure (preserves flows during ovn-controller restart)
if out, err := sudoCommand("ovs-vsctl", "set", "Bridge", "br-int", "fail-mode=secure").CombinedOutput(); err != nil {
return fmt.Errorf("set br-int fail-mode: %s: %w", strings.TrimSpace(string(out)), err)
}
// Disable in-band management (prevents OVS from adding its own flows)
if out, err := sudoCommand("ovs-vsctl", "set", "Bridge", "br-int", "other-config:disable-in-band=true").CombinedOutput(); err != nil {
return fmt.Errorf("set br-int disable-in-band: %s: %w", strings.TrimSpace(string(out)), err)
}
// Bring br-int up
if out, err := sudoCommand("ip", "link", "set", "br-int", "up").CombinedOutput(); err != nil {
return fmt.Errorf("bring up br-int: %s: %w", strings.TrimSpace(string(out)), err)
}
// Set OVN external_ids on the Open_vSwitch table
if out, err := sudoCommand("ovs-vsctl", "set", "Open_vSwitch", ".",
fmt.Sprintf("external_ids:system-id=%s", chassisID),
fmt.Sprintf("external_ids:ovn-remote=%s", ovnRemote),
fmt.Sprintf("external_ids:ovn-encap-ip=%s", encapIP),
"external_ids:ovn-encap-type=geneve",
).CombinedOutput(); err != nil {
return fmt.Errorf("set OVN external_ids: %s: %w", strings.TrimSpace(string(out)), err)
}
slog.Info("OVN compute node configured",
"chassis_id", chassisID,
"ovn_remote", ovnRemote,
"encap_ip", encapIP,
)
// Ensure the data NIC is preferred for Geneve tunnel routing.
if err := EnsureDataRoute(encapIP); err != nil {
slog.Warn("Failed to configure data NIC routing (Geneve tunnels may use wrong source IP)", "err", err)
}
return nil
}
// EnsureDataRoute ensures the kernel routes Geneve tunnel traffic through the
// data NIC (the interface holding the encap IP). When management and data NICs
// share the same subnet with equal route metrics, the kernel may pick the
// management NIC, causing Geneve packets to have the wrong source IP. Remote
// OVS nodes then drop these packets because the source doesn't match the
// expected tunnel remote_ip.
//
// Fix: find the data NIC's subnet route and replace it with a lower metric (50)
// so it's preferred over the management NIC's route (typically metric 100+).
func EnsureDataRoute(encapIP string) error {
dataIface, err := findInterfaceByIP(encapIP)
if err != nil {
return fmt.Errorf("find data interface for %s: %w", encapIP, err)
}
// Read the existing subnet route for the data interface
out, err := sudoCommand("ip", "-o", "-4", "route", "show", "dev", dataIface, "proto", "kernel", "scope", "link").CombinedOutput()
if err != nil {
return fmt.Errorf("read routes for %s: %s: %w", dataIface, strings.TrimSpace(string(out)), err)
}
// Parse the subnet CIDR from the route output (e.g. "10.1.0.0/16 dev eth1 ...")
fields := strings.Fields(strings.TrimSpace(string(out)))
if len(fields) == 0 {
return fmt.Errorf("no kernel route found for %s", dataIface)
}
subnet := fields[0]
// Replace the route with a lower metric so the data NIC is preferred
if out, err := sudoCommand("ip", "route", "replace", subnet,
"dev", dataIface, "src", encapIP, "metric", "50",
).CombinedOutput(); err != nil {
return fmt.Errorf("replace route for %s: %s: %w", subnet, strings.TrimSpace(string(out)), err)
}
slog.Info("Data NIC route configured for Geneve tunnels",
"interface", dataIface,
"subnet", subnet,
"encap_ip", encapIP,
"metric", 50,
)
return nil
}
// findInterfaceByIP returns the network interface name that holds the given IP address.
func findInterfaceByIP(ipAddr string) (string, error) {
targetIP := net.ParseIP(ipAddr)
if targetIP == nil {
return "", fmt.Errorf("invalid IP address: %s", ipAddr)
}
ifaces, err := net.Interfaces()
if err != nil {
return "", fmt.Errorf("list interfaces: %w", err)
}
for _, iface := range ifaces {
addrs, err := iface.Addrs()
if err != nil {
continue
}
for _, addr := range addrs {
ipNet, ok := addr.(*net.IPNet)
if !ok {
continue
}
if ipNet.IP.Equal(targetIP) {
return iface.Name, nil
}
}
}
return "", fmt.Errorf("no interface found with IP %s", ipAddr)
}
+437
View File
@@ -0,0 +1,437 @@
package daemon
import (
"fmt"
"os"
"strings"
"testing"
"github.com/mulgadc/hive/hive/vm"
)
func TestTapDeviceName(t *testing.T) {
tests := []struct {
eniId string
expected string
}{
{"eni-abc123", "tapabc123"}, // Short ID
{"eni-abc123def456789", "tapabc123def456"}, // Truncated to 15 chars
{"eni-a", "tapa"}, // Minimal
{"eni-123456789abcdef", "tap123456789abc"}, // Exactly 15 chars
{"eni-123456789abcdefg", "tap123456789abc"}, // Truncated at 15
}
for _, tt := range tests {
t.Run(tt.eniId, func(t *testing.T) {
got := TapDeviceName(tt.eniId)
if got != tt.expected {
t.Errorf("TapDeviceName(%q) = %q, want %q", tt.eniId, got, tt.expected)
}
if len(got) > 15 {
t.Errorf("TapDeviceName(%q) = %q (len %d), exceeds IFNAMSIZ limit of 15", tt.eniId, got, len(got))
}
})
}
}
func TestOVSIfaceID(t *testing.T) {
tests := []struct {
eniId string
expected string
}{
{"eni-abc123", "port-eni-abc123"},
{"eni-abc123def456789", "port-eni-abc123def456789"},
}
for _, tt := range tests {
t.Run(tt.eniId, func(t *testing.T) {
got := OVSIfaceID(tt.eniId)
if got != tt.expected {
t.Errorf("OVSIfaceID(%q) = %q, want %q", tt.eniId, got, tt.expected)
}
})
}
}
// MockNetworkPlumber records calls for testing.
type MockNetworkPlumber struct {
SetupCalls []mockSetupCall
CleanupCalls []string
SetupErr error
CleanupErr error
}
type mockSetupCall struct {
ENIId string
MAC string
}
func (m *MockNetworkPlumber) SetupTapDevice(eniId, mac string) error {
m.SetupCalls = append(m.SetupCalls, mockSetupCall{ENIId: eniId, MAC: mac})
return m.SetupErr
}
func (m *MockNetworkPlumber) CleanupTapDevice(eniId string) error {
m.CleanupCalls = append(m.CleanupCalls, eniId)
return m.CleanupErr
}
func TestStartInstance_VPCNetworking(t *testing.T) {
instance := &vm.VM{
ID: "i-test123",
InstanceType: "t3.micro",
ENIId: "eni-abc123def456789",
ENIMac: "02:00:00:11:22:33",
}
// When ENI is set, config should use tap networking with MAC
instance.Config = vm.Config{
CPUCount: 1,
Memory: 512,
Architecture: "x86_64",
}
// Simulate what StartInstance does for VPC mode
tapName := TapDeviceName(instance.ENIId)
instance.Config.NetDevs = append(instance.Config.NetDevs, vm.NetDev{
Value: "tap,id=net0,ifname=" + tapName + ",script=no,downscript=no",
})
instance.Config.Devices = append(instance.Config.Devices, vm.Device{
Value: "virtio-net-pci,netdev=net0,mac=" + instance.ENIMac,
})
// Verify QEMU args
if len(instance.Config.NetDevs) != 1 {
t.Fatalf("expected 1 netdev, got %d", len(instance.Config.NetDevs))
}
expected := "tap,id=net0,ifname=tapabc123def456,script=no,downscript=no"
if instance.Config.NetDevs[0].Value != expected {
t.Errorf("netdev = %q, want %q", instance.Config.NetDevs[0].Value, expected)
}
expectedDev := "virtio-net-pci,netdev=net0,mac=02:00:00:11:22:33"
if instance.Config.Devices[0].Value != expectedDev {
t.Errorf("device = %q, want %q", instance.Config.Devices[0].Value, expectedDev)
}
}
func TestStartInstance_FallbackNetworking(t *testing.T) {
instance := &vm.VM{
ID: "i-test456",
InstanceType: "t3.micro",
// No ENI — should use user-mode networking
}
instance.Config = vm.Config{
CPUCount: 1,
Memory: 512,
Architecture: "x86_64",
}
// Simulate what StartInstance does for non-VPC mode
instance.Config.NetDevs = append(instance.Config.NetDevs, vm.NetDev{
Value: "user,id=net0,hostfwd=tcp:127.0.0.1:22222-:22",
})
instance.Config.Devices = append(instance.Config.Devices, vm.Device{
Value: "virtio-net-pci,netdev=net0",
})
// Verify no MAC is specified (QEMU auto-assigns)
if len(instance.Config.Devices) != 1 {
t.Fatalf("expected 1 device, got %d", len(instance.Config.Devices))
}
// User-mode networking should not include MAC
dev := instance.Config.Devices[0].Value
if dev != "virtio-net-pci,netdev=net0" {
t.Errorf("device = %q, want 'virtio-net-pci,netdev=net0'", dev)
}
}
func TestMockNetworkPlumber_SetupAndCleanup(t *testing.T) {
mock := &MockNetworkPlumber{}
err := mock.SetupTapDevice("eni-abc123", "02:00:00:aa:bb:cc")
if err != nil {
t.Fatalf("SetupTapDevice: %v", err)
}
if len(mock.SetupCalls) != 1 {
t.Fatalf("expected 1 setup call, got %d", len(mock.SetupCalls))
}
if mock.SetupCalls[0].ENIId != "eni-abc123" {
t.Errorf("setup eniId = %q, want 'eni-abc123'", mock.SetupCalls[0].ENIId)
}
if mock.SetupCalls[0].MAC != "02:00:00:aa:bb:cc" {
t.Errorf("setup mac = %q, want '02:00:00:aa:bb:cc'", mock.SetupCalls[0].MAC)
}
err = mock.CleanupTapDevice("eni-abc123")
if err != nil {
t.Fatalf("CleanupTapDevice: %v", err)
}
if len(mock.CleanupCalls) != 1 {
t.Fatalf("expected 1 cleanup call, got %d", len(mock.CleanupCalls))
}
if mock.CleanupCalls[0] != "eni-abc123" {
t.Errorf("cleanup eniId = %q, want 'eni-abc123'", mock.CleanupCalls[0])
}
}
func TestSudoCommand_NonRoot(t *testing.T) {
// When not root, sudoCommand should prefix with sudo
cmd := sudoCommand("ovs-vsctl", "br-exists", "br-int")
args := cmd.Args
if os.Getuid() == 0 {
// Running as root: should NOT use sudo
if args[0] != "ovs-vsctl" {
t.Errorf("as root, expected args[0]='ovs-vsctl', got %q", args[0])
}
} else {
// Running as non-root: should use sudo
if args[0] != "sudo" {
t.Errorf("as non-root, expected args[0]='sudo', got %q", args[0])
}
if args[1] != "ovs-vsctl" {
t.Errorf("as non-root, expected args[1]='ovs-vsctl', got %q", args[1])
}
if len(args) != 4 {
t.Errorf("expected 4 args [sudo ovs-vsctl br-exists br-int], got %d: %v", len(args), args)
}
}
}
func TestGenerateDevMAC(t *testing.T) {
tests := []struct {
instanceId string
}{
{"i-abc123"},
{"i-def456"},
{"i-ghi789"},
}
// All MACs should be unique and have the 02:de:v0 prefix
seen := make(map[string]bool)
for _, tt := range tests {
mac := generateDevMAC(tt.instanceId)
if !strings.HasPrefix(mac, "02:de:00:") {
t.Errorf("generateDevMAC(%q) = %q, want prefix '02:de:00:'", tt.instanceId, mac)
}
if seen[mac] {
t.Errorf("generateDevMAC(%q) = %q, duplicate MAC", tt.instanceId, mac)
}
seen[mac] = true
}
// Same input should produce same output (deterministic)
mac1 := generateDevMAC("i-test123")
mac2 := generateDevMAC("i-test123")
if mac1 != mac2 {
t.Errorf("generateDevMAC not deterministic: %q != %q", mac1, mac2)
}
}
func TestNetworkPlumber_InterfaceCompliance(t *testing.T) {
// Verify both types satisfy the interface
var _ NetworkPlumber = &OVSNetworkPlumber{}
var _ NetworkPlumber = &MockNetworkPlumber{}
}
func TestOVNHealthStatus_Fields(t *testing.T) {
// Verify OVNHealthStatus struct can be used for health reporting
status := OVNHealthStatus{
BrIntExists: true,
OVNControllerUp: true,
ChassisID: "chassis-node1",
EncapIP: "10.0.0.1",
OVNRemote: "tcp:10.0.0.1:6642",
}
if !status.BrIntExists {
t.Error("expected BrIntExists to be true")
}
if !status.OVNControllerUp {
t.Error("expected OVNControllerUp to be true")
}
if status.ChassisID != "chassis-node1" {
t.Errorf("ChassisID = %q, want 'chassis-node1'", status.ChassisID)
}
if status.EncapIP != "10.0.0.1" {
t.Errorf("EncapIP = %q, want '10.0.0.1'", status.EncapIP)
}
if status.OVNRemote != "tcp:10.0.0.1:6642" {
t.Errorf("OVNRemote = %q, want 'tcp:10.0.0.1:6642'", status.OVNRemote)
}
}
func TestOVNHealthStatus_Defaults(t *testing.T) {
// Zero-value OVNHealthStatus should indicate nothing is ready
var status OVNHealthStatus
if status.BrIntExists {
t.Error("zero-value BrIntExists should be false")
}
if status.OVNControllerUp {
t.Error("zero-value OVNControllerUp should be false")
}
if status.ChassisID != "" {
t.Errorf("zero-value ChassisID should be empty, got %q", status.ChassisID)
}
}
func TestCheckOVNHealth_ReturnsStatus(t *testing.T) {
// CheckOVNHealth should return a status struct without panicking,
// even when OVS/OVN tools are not installed (CI environment).
// On a dev machine without OVN, all fields will be zero values.
status := CheckOVNHealth()
// On CI without OVS, both should be false — just verify no panic
_ = status.BrIntExists
_ = status.OVNControllerUp
_ = status.ChassisID
_ = status.EncapIP
_ = status.OVNRemote
}
func TestFindInterfaceByIP_InvalidIP(t *testing.T) {
_, err := findInterfaceByIP("not-an-ip")
if err == nil {
t.Fatal("expected error for invalid IP")
}
if !strings.Contains(err.Error(), "invalid IP address") {
t.Errorf("expected 'invalid IP address' error, got: %v", err)
}
}
func TestFindInterfaceByIP_Loopback(t *testing.T) {
// 127.0.0.1 should always be on the loopback interface
iface, err := findInterfaceByIP("127.0.0.1")
if err != nil {
t.Fatalf("findInterfaceByIP(127.0.0.1): %v", err)
}
if iface != "lo" {
t.Errorf("findInterfaceByIP(127.0.0.1) = %q, want 'lo'", iface)
}
}
func TestFindInterfaceByIP_NotFound(t *testing.T) {
_, err := findInterfaceByIP("192.0.2.1")
if err == nil {
t.Fatal("expected error for non-existent IP")
}
if !strings.Contains(err.Error(), "no interface found") {
t.Errorf("expected 'no interface found' error, got: %v", err)
}
}
func TestEnsureDataRoute_NoOVS(t *testing.T) {
// EnsureDataRoute requires ip commands which may not work in CI.
// On loopback, there's no kernel subnet route, so it should return an error.
err := EnsureDataRoute("127.0.0.1")
// We expect an error (no kernel route for lo), but no panic.
_ = err
}
func TestSetupComputeNode_ValidatesArgs(t *testing.T) {
// SetupComputeNode requires ovs-vsctl which may not be available in CI.
// This test verifies the function signature and that it returns an error
// when OVS is not installed (expected on CI).
err := SetupComputeNode("chassis-test", "tcp:127.0.0.1:6642", "10.0.0.1")
// We expect an error in CI (no OVS), but the function should not panic.
// On a dev machine with OVS, it would succeed. Either result is acceptable.
_ = err
}
func TestMockNetworkPlumber_SetupError(t *testing.T) {
mock := &MockNetworkPlumber{
SetupErr: fmt.Errorf("simulated setup failure"),
}
err := mock.SetupTapDevice("eni-abc123", "02:00:00:aa:bb:cc")
if err == nil {
t.Fatal("expected error from SetupTapDevice")
}
if err.Error() != "simulated setup failure" {
t.Errorf("unexpected error: %v", err)
}
// Call should still be recorded
if len(mock.SetupCalls) != 1 {
t.Fatalf("expected 1 setup call, got %d", len(mock.SetupCalls))
}
}
func TestMockNetworkPlumber_CleanupError(t *testing.T) {
mock := &MockNetworkPlumber{
CleanupErr: fmt.Errorf("simulated cleanup failure"),
}
err := mock.CleanupTapDevice("eni-abc123")
if err == nil {
t.Fatal("expected error from CleanupTapDevice")
}
if err.Error() != "simulated cleanup failure" {
t.Errorf("unexpected error: %v", err)
}
if len(mock.CleanupCalls) != 1 {
t.Fatalf("expected 1 cleanup call, got %d", len(mock.CleanupCalls))
}
}
func TestGenerateDevMAC_Format(t *testing.T) {
tests := []string{
"i-abc123",
"i-def456",
"i-00000000",
"i-ffffffff",
"i-a",
"i-very-long-instance-id-with-many-characters",
}
for _, id := range tests {
mac := generateDevMAC(id)
// Must be 17 chars: xx:xx:xx:xx:xx:xx
if len(mac) != 17 {
t.Errorf("generateDevMAC(%q) = %q, expected 17 chars", id, mac)
}
// Must start with locally-administered unicast prefix 02:de:00
if !strings.HasPrefix(mac, "02:de:00:") {
t.Errorf("generateDevMAC(%q) = %q, expected prefix 02:de:00:", id, mac)
}
// All chars must be valid hex or colons
for i, c := range mac {
if i%3 == 2 {
if c != ':' {
t.Errorf("generateDevMAC(%q) = %q, expected ':' at pos %d", id, mac, i)
}
} else {
if !((c >= '0' && c <= '9') || (c >= 'a' && c <= 'f')) {
t.Errorf("generateDevMAC(%q) = %q, invalid hex char '%c' at pos %d", id, mac, c, i)
}
}
}
}
}
func TestTapDeviceName_EmptyInput(t *testing.T) {
// Even with empty string (no eni- prefix), should not panic
name := TapDeviceName("")
if name != "tap" {
t.Errorf("TapDeviceName('') = %q, want 'tap'", name)
}
}
func TestOVSIfaceID_Format(t *testing.T) {
tests := []struct {
eniId string
expected string
}{
{"eni-short", "port-eni-short"},
{"eni-", "port-eni-"},
{"", "port-"},
}
for _, tt := range tests {
got := OVSIfaceID(tt.eniId)
if got != tt.expected {
t.Errorf("OVSIfaceID(%q) = %q, want %q", tt.eniId, got, tt.expected)
}
}
}
+45
View File
@@ -0,0 +1,45 @@
package daemon
import (
"log/slog"
"github.com/mulgadc/hive/hive/awserrors"
"github.com/mulgadc/hive/hive/utils"
"github.com/nats-io/nats.go"
)
// checkInstanceOwnership verifies the caller owns the instance.
// Returns true if access is allowed, false if denied (error already sent).
// Pre-Phase4 instances (empty AccountID) are only visible to root (GlobalAccountID).
func checkInstanceOwnership(msg *nats.Msg, instanceID, ownerAccountID string) bool {
callerAccountID := utils.AccountIDFromMsg(msg)
// Pre-Phase4 instance: only root can access
if ownerAccountID == "" {
if callerAccountID != utils.GlobalAccountID {
slog.Warn("Pre-Phase4 instance access denied (not root)",
"instanceId", instanceID, "callerAccount", callerAccountID)
respondWithError(msg, awserrors.ErrorInvalidInstanceIDNotFound)
return false
}
return true
}
// Normal ownership check
if callerAccountID != ownerAccountID {
slog.Warn("Account does not own instance",
"instanceId", instanceID, "callerAccount", callerAccountID, "ownerAccount", ownerAccountID)
respondWithError(msg, awserrors.ErrorInvalidInstanceIDNotFound)
return false
}
return true
}
// isInstanceVisible checks if the caller can see this instance (for Describe operations).
// Pre-Phase4 instances (empty AccountID) are only visible to root (GlobalAccountID).
func isInstanceVisible(callerAccountID, ownerAccountID string) bool {
if ownerAccountID == "" {
return callerAccountID == utils.GlobalAccountID
}
return callerAccountID == ownerAccountID
}
+74
View File
@@ -0,0 +1,74 @@
package daemon
import (
"errors"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/mulgadc/hive/hive/types"
)
// errInsufficientCapacity is returned by allocateForLaunch when MinCount
// cannot be satisfied.
var errInsufficientCapacity = errors.New("insufficient capacity to satisfy MinCount")
// canAllocateCount returns how many instances of the given type can fit
// in the remaining capacity, capped at maxCount. Pure function — no locks
// or side effects.
func canAllocateCount(availVCPU, allocVCPU int, availMem, allocMem float64,
vCPUs int64, memMiB int64, maxCount int) int {
remainingVCPU := availVCPU - allocVCPU
remainingMem := availMem - allocMem
memoryGB := float64(memMiB) / 1024.0
countByCPU := maxCount
if vCPUs > 0 {
countByCPU = remainingVCPU / int(vCPUs)
}
countByMem := maxCount
if memoryGB > 0 {
countByMem = int(remainingMem / memoryGB)
}
result := min(countByMem, countByCPU)
result = min(result, maxCount)
return max(result, 0)
}
// resourceStatsForType computes the InstanceTypeCap for a single instance type
// given the remaining host resources. Pure function — no locks or side effects.
func resourceStatsForType(remainVCPU int, remainMem float64, it *ec2.InstanceTypeInfo) types.InstanceTypeCap {
vCPUs := instanceTypeVCPUs(it)
memGB := float64(instanceTypeMemoryMiB(it)) / 1024.0
count := 0
if vCPUs > 0 && memGB > 0 {
countVCPU := remainVCPU / int(vCPUs)
countMem := int(remainMem / memGB)
count = max(min(countMem, countVCPU), 0)
}
name := ""
if it.InstanceType != nil {
name = *it.InstanceType
}
return types.InstanceTypeCap{
Name: name,
VCPU: int(vCPUs),
MemoryGB: memGB,
Available: count,
}
}
// allocateForLaunch determines the number of instances to launch given
// available capacity and the MinCount/MaxCount constraints from a
// RunInstances request. Returns the launch count or an error if the
// minimum cannot be satisfied.
func allocateForLaunch(canAlloc, minCount, maxCount int) (int, error) {
if canAlloc < minCount {
return 0, errInsufficientCapacity
}
return min(canAlloc, maxCount), nil
}
+265
View File
@@ -0,0 +1,265 @@
package daemon
import (
"testing"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/stretchr/testify/assert"
)
func TestCanAllocateCount(t *testing.T) {
tests := []struct {
name string
availVCPU int
allocVCPU int
availMem float64
allocMem float64
vCPUs int64
memMiB int64
maxCount int
want int
}{
{
name: "exact fit single instance",
availVCPU: 4, allocVCPU: 2,
availMem: 8.0, allocMem: 4.0,
vCPUs: 2, memMiB: 4096,
maxCount: 10,
want: 1,
},
{
name: "multiple instances fit",
availVCPU: 16, allocVCPU: 0,
availMem: 32.0, allocMem: 0.0,
vCPUs: 2, memMiB: 4096,
maxCount: 10,
want: 8, // limited by CPU: 16/2 = 8, mem: 32/4 = 8
},
{
name: "CPU limited",
availVCPU: 4, allocVCPU: 0,
availMem: 64.0, allocMem: 0.0,
vCPUs: 2, memMiB: 4096,
maxCount: 10,
want: 2, // CPU: 4/2=2, mem: 64/4=16 → min=2
},
{
name: "memory limited",
availVCPU: 64, allocVCPU: 0,
availMem: 8.0, allocMem: 0.0,
vCPUs: 2, memMiB: 4096,
maxCount: 10,
want: 2, // CPU: 64/2=32, mem: 8/4=2 → min=2
},
{
name: "capped by maxCount",
availVCPU: 64, allocVCPU: 0,
availMem: 128.0, allocMem: 0.0,
vCPUs: 2, memMiB: 4096,
maxCount: 3,
want: 3,
},
{
name: "zero remaining resources",
availVCPU: 4, allocVCPU: 4,
availMem: 8.0, allocMem: 8.0,
vCPUs: 2, memMiB: 4096,
maxCount: 5,
want: 0,
},
{
name: "negative remaining (overallocated)",
availVCPU: 4, allocVCPU: 6,
availMem: 8.0, allocMem: 10.0,
vCPUs: 2, memMiB: 4096,
maxCount: 5,
want: 0,
},
{
name: "zero vCPUs bypasses CPU check",
availVCPU: 4, allocVCPU: 0,
availMem: 16.0, allocMem: 0.0,
vCPUs: 0, memMiB: 4096,
maxCount: 5,
want: 4, // CPU check skipped (maxCount=5), mem: 16/4=4 → min=4
},
{
name: "zero memory bypasses mem check",
availVCPU: 8, allocVCPU: 0,
availMem: 16.0, allocMem: 0.0,
vCPUs: 2, memMiB: 0,
maxCount: 5,
want: 4, // CPU: 8/2=4, mem check skipped (maxCount=5) → min=4
},
{
name: "maxCount zero",
availVCPU: 16, allocVCPU: 0,
availMem: 32.0, allocMem: 0.0,
vCPUs: 2, memMiB: 4096,
maxCount: 0,
want: 0,
},
{
name: "off by one CPU",
availVCPU: 5, allocVCPU: 0,
availMem: 64.0, allocMem: 0.0,
vCPUs: 2, memMiB: 4096,
maxCount: 10,
want: 2, // 5/2 = 2 (integer division)
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
got := canAllocateCount(tc.availVCPU, tc.allocVCPU, tc.availMem, tc.allocMem, tc.vCPUs, tc.memMiB, tc.maxCount)
assert.Equal(t, tc.want, got)
})
}
}
func TestResourceStatsForType(t *testing.T) {
tests := []struct {
name string
remainVCPU int
remainMem float64
it *ec2.InstanceTypeInfo
wantName string
wantVCPU int
wantMemGB float64
wantAvail int
}{
{
name: "standard instance type",
remainVCPU: 8,
remainMem: 16.0,
it: &ec2.InstanceTypeInfo{
InstanceType: aws.String("t3.medium"),
VCpuInfo: &ec2.VCpuInfo{DefaultVCpus: aws.Int64(2)},
MemoryInfo: &ec2.MemoryInfo{SizeInMiB: aws.Int64(4096)},
},
wantName: "t3.medium",
wantVCPU: 2,
wantMemGB: 4.0,
wantAvail: 4, // min(8/2=4, 16/4=4)
},
{
name: "CPU limited",
remainVCPU: 2,
remainMem: 64.0,
it: &ec2.InstanceTypeInfo{
InstanceType: aws.String("c5.xlarge"),
VCpuInfo: &ec2.VCpuInfo{DefaultVCpus: aws.Int64(4)},
MemoryInfo: &ec2.MemoryInfo{SizeInMiB: aws.Int64(8192)},
},
wantName: "c5.xlarge",
wantVCPU: 4,
wantMemGB: 8.0,
wantAvail: 0, // CPU: 2/4=0
},
{
name: "nil instance type name",
remainVCPU: 16,
remainMem: 32.0,
it: &ec2.InstanceTypeInfo{
VCpuInfo: &ec2.VCpuInfo{DefaultVCpus: aws.Int64(2)},
MemoryInfo: &ec2.MemoryInfo{SizeInMiB: aws.Int64(2048)},
},
wantName: "",
wantVCPU: 2,
wantMemGB: 2.0,
wantAvail: 8, // min(16/2=8, 32/2=16)
},
{
name: "zero vCPU gives zero available",
remainVCPU: 16,
remainMem: 32.0,
it: &ec2.InstanceTypeInfo{
InstanceType: aws.String("broken"),
VCpuInfo: &ec2.VCpuInfo{DefaultVCpus: aws.Int64(0)},
MemoryInfo: &ec2.MemoryInfo{SizeInMiB: aws.Int64(4096)},
},
wantName: "broken",
wantVCPU: 0,
wantMemGB: 4.0,
wantAvail: 0,
},
{
name: "nil VCpuInfo",
remainVCPU: 16,
remainMem: 32.0,
it: &ec2.InstanceTypeInfo{
InstanceType: aws.String("broken2"),
MemoryInfo: &ec2.MemoryInfo{SizeInMiB: aws.Int64(4096)},
},
wantName: "broken2",
wantVCPU: 0,
wantMemGB: 4.0,
wantAvail: 0,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
cap := resourceStatsForType(tc.remainVCPU, tc.remainMem, tc.it)
assert.Equal(t, tc.wantName, cap.Name)
assert.Equal(t, tc.wantVCPU, cap.VCPU)
assert.InDelta(t, tc.wantMemGB, cap.MemoryGB, 0.001)
assert.Equal(t, tc.wantAvail, cap.Available)
})
}
}
func TestAllocateForLaunch(t *testing.T) {
tests := []struct {
name string
canAlloc int
minCount int
maxCount int
want int
wantErr bool
}{
{
name: "exact min equals max",
canAlloc: 5, minCount: 5, maxCount: 5,
want: 5,
},
{
name: "capacity exceeds max",
canAlloc: 10, minCount: 1, maxCount: 5,
want: 5,
},
{
name: "capacity between min and max",
canAlloc: 3, minCount: 1, maxCount: 5,
want: 3,
},
{
name: "capacity below min",
canAlloc: 2, minCount: 3, maxCount: 5,
wantErr: true,
},
{
name: "zero capacity",
canAlloc: 0, minCount: 1, maxCount: 5,
wantErr: true,
},
{
name: "zero min count always succeeds",
canAlloc: 0, minCount: 0, maxCount: 5,
want: 0,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
got, err := allocateForLaunch(tc.canAlloc, tc.minCount, tc.maxCount)
if tc.wantErr {
assert.ErrorIs(t, err, errInsufficientCapacity)
} else {
assert.NoError(t, err)
assert.Equal(t, tc.want, got)
}
})
}
}
+298
View File
@@ -0,0 +1,298 @@
package daemon
import (
"context"
"encoding/json"
"log/slog"
"os"
"os/exec"
"path/filepath"
"time"
"github.com/mulgadc/hive/hive/utils"
"github.com/nats-io/nats.go"
)
// pidDir returns the directory containing shell-level PID files for this node.
// start-dev.sh writes PID files to $DATA_DIR/logs/ and the daemon's BaseDir is
// always $DATA_DIR/hive/, so we derive the logs directory from that. Each node
// has its own DATA_DIR, so this is safe for simulated multi-node on a single host.
func (d *Daemon) pidDir() string {
if d.config.BaseDir != "" {
return filepath.Join(filepath.Dir(filepath.Clean(d.config.BaseDir)), "logs")
}
return ""
}
// ShutdownRequest is sent by the coordinator to each daemon per phase.
type ShutdownRequest struct {
Phase string `json:"phase"`
Force bool `json:"force"`
Timeout int `json:"timeout_seconds"`
}
// ShutdownACK is the response from a daemon after completing a shutdown phase.
type ShutdownACK struct {
Node string `json:"node"`
Phase string `json:"phase"`
Stopped []string `json:"stopped,omitempty"`
Error string `json:"error,omitempty"`
}
// ShutdownProgress is published by daemons during the DRAIN phase to report VM shutdown progress.
type ShutdownProgress struct {
Node string `json:"node"`
Phase string `json:"phase"`
Total int `json:"total"`
Remaining int `json:"remaining"`
}
// handleShutdownGate stops the API gateway and UI, then sets shuttingDown flag
// so the daemon rejects new work. Phase: GATE.
func (d *Daemon) handleShutdownGate(msg *nats.Msg) {
var req ShutdownRequest
if err := json.Unmarshal(msg.Data, &req); err != nil {
slog.Error("handleShutdownGate: failed to unmarshal request", "error", err)
return
}
slog.Info("Shutdown GATE phase starting", "node", d.node, "force", req.Force)
var stopped []string
// Stop AWSGW
if d.config.HasService("awsgw") {
if err := utils.StopProcessAt(d.pidDir(), "awsgw"); err != nil {
slog.Warn("Failed to stop awsgw", "error", err)
} else {
stopped = append(stopped, "awsgw")
}
}
// Stop UI
if d.config.HasService("ui") {
if err := utils.StopProcessAt(d.pidDir(), "hive-ui"); err != nil {
slog.Warn("Failed to stop hive-ui", "error", err)
} else {
stopped = append(stopped, "hive-ui")
}
}
// Stop vpcd (VPC daemon)
if d.config.HasService("vpcd") {
if err := utils.StopProcessAt(d.pidDir(), "vpcd"); err != nil {
slog.Warn("Failed to stop vpcd", "error", err)
} else {
stopped = append(stopped, "vpcd")
}
}
// Set shuttingDown flag so daemon rejects new work
d.shuttingDown.Store(true)
ack := ShutdownACK{
Node: d.node,
Phase: "gate",
Stopped: stopped,
}
d.respondShutdownACK(msg, ack)
slog.Info("Shutdown GATE phase complete", "node", d.node, "stopped", stopped)
}
// handleShutdownDrain gracefully stops all running VMs, writes shutdown marker
// and persists state. Phase: DRAIN.
func (d *Daemon) handleShutdownDrain(msg *nats.Msg) {
var req ShutdownRequest
if err := json.Unmarshal(msg.Data, &req); err != nil {
slog.Error("handleShutdownDrain: failed to unmarshal request", "error", err)
return
}
slog.Info("Shutdown DRAIN phase starting", "node", d.node)
// Count total VMs
d.Instances.Mu.Lock()
total := len(d.Instances.VMS)
d.Instances.Mu.Unlock()
// Publish initial progress
d.publishShutdownProgress("drain", total, total)
// Stop all instances (graceful shutdown, no volume deletion)
if total > 0 {
d.Instances.Mu.Lock()
vms := d.Instances.VMS
d.Instances.Mu.Unlock()
if err := d.stopInstance(vms, false); err != nil {
slog.Error("Failed to stop instances during DRAIN", "error", err)
ack := ShutdownACK{
Node: d.node,
Phase: "drain",
Error: err.Error(),
}
d.respondShutdownACK(msg, ack)
return
}
}
// Publish final progress
d.publishShutdownProgress("drain", total, 0)
// Write shutdown marker and persist state
if d.jsManager != nil {
if err := d.jsManager.WriteShutdownMarker(d.node); err != nil {
slog.Error("Failed to write shutdown marker during DRAIN", "error", err)
}
}
if err := d.WriteState(); err != nil {
slog.Error("Failed to write state during DRAIN", "error", err)
}
ack := ShutdownACK{
Node: d.node,
Phase: "drain",
}
d.respondShutdownACK(msg, ack)
slog.Info("Shutdown DRAIN phase complete", "node", d.node, "vms_stopped", total)
}
// handleShutdownStorage stops viperblock and cleans up orphan nbdkit processes. Phase: STORAGE.
func (d *Daemon) handleShutdownStorage(msg *nats.Msg) {
var req ShutdownRequest
if err := json.Unmarshal(msg.Data, &req); err != nil {
slog.Error("handleShutdownStorage: failed to unmarshal request", "error", err)
return
}
slog.Info("Shutdown STORAGE phase starting", "node", d.node)
var stopped []string
if d.config.HasService("viperblock") {
if err := utils.StopProcessAt(d.pidDir(), "viperblock"); err != nil {
slog.Warn("Failed to stop viperblock", "error", err)
} else {
stopped = append(stopped, "viperblock")
}
}
// Best-effort cleanup of orphaned nbdkit processes
if err := exec.Command("pkill", "-f", "nbdkit").Run(); err != nil {
slog.Debug("pkill nbdkit (best-effort)", "result", err)
}
ack := ShutdownACK{
Node: d.node,
Phase: "storage",
Stopped: stopped,
}
d.respondShutdownACK(msg, ack)
slog.Info("Shutdown STORAGE phase complete", "node", d.node, "stopped", stopped)
}
// handleShutdownPersist stops predastore. Phase: PERSIST.
func (d *Daemon) handleShutdownPersist(msg *nats.Msg) {
var req ShutdownRequest
if err := json.Unmarshal(msg.Data, &req); err != nil {
slog.Error("handleShutdownPersist: failed to unmarshal request", "error", err)
return
}
slog.Info("Shutdown PERSIST phase starting", "node", d.node)
var stopped []string
if d.config.HasService("predastore") {
if err := utils.StopProcessAt(d.pidDir(), "predastore"); err != nil {
slog.Warn("Failed to stop predastore", "error", err)
} else {
stopped = append(stopped, "predastore")
}
}
ack := ShutdownACK{
Node: d.node,
Phase: "persist",
Stopped: stopped,
}
d.respondShutdownACK(msg, ack)
slog.Info("Shutdown PERSIST phase complete", "node", d.node, "stopped", stopped)
}
// handleShutdownInfra stops NATS and exits the daemon. Phase: INFRA.
// No ACK is sent because NATS is going down — this is fire-and-forget from the coordinator.
func (d *Daemon) handleShutdownInfra(msg *nats.Msg) {
var req ShutdownRequest
if err := json.Unmarshal(msg.Data, &req); err != nil {
slog.Error("handleShutdownInfra: failed to unmarshal request", "error", err)
return
}
slog.Info("Shutdown INFRA phase starting", "node", d.node)
// Cancel context to stop heartbeat and other goroutines
d.cancel()
// Unsubscribe from all NATS topics
for topic, sub := range d.natsSubscriptions {
slog.Debug("Unsubscribing from NATS", "topic", topic)
if err := sub.Unsubscribe(); err != nil {
slog.Warn("Failed to unsubscribe", "topic", topic, "error", err)
}
}
// Shutdown cluster manager
if d.clusterServer != nil {
if err := d.clusterServer.Shutdown(context.Background()); err != nil {
slog.Warn("Failed to shutdown cluster manager", "error", err)
}
}
// Close NATS connection
d.natsConn.Close()
// Stop NATS server if this node runs it
if d.config.HasService("nats") {
if err := utils.StopProcessAt(d.pidDir(), "nats"); err != nil {
slog.Warn("Failed to stop nats", "error", err)
} else {
slog.Info("NATS server stopped", "node", d.node)
}
}
slog.Info("Shutdown INFRA phase complete, exiting", "node", d.node)
// Give a moment for log flush
time.Sleep(500 * time.Millisecond)
os.Exit(0)
}
// respondShutdownACK marshals and sends a ShutdownACK response.
func (d *Daemon) respondShutdownACK(msg *nats.Msg, ack ShutdownACK) {
data, err := json.Marshal(ack)
if err != nil {
slog.Error("Failed to marshal shutdown ACK", "error", err)
return
}
if err := msg.Respond(data); err != nil {
slog.Error("Failed to respond with shutdown ACK", "error", err)
}
}
// publishShutdownProgress publishes a progress update during the DRAIN phase.
func (d *Daemon) publishShutdownProgress(phase string, total, remaining int) {
progress := ShutdownProgress{
Node: d.node,
Phase: phase,
Total: total,
Remaining: remaining,
}
data, err := json.Marshal(progress)
if err != nil {
return
}
if err := d.natsConn.Publish("hive.cluster.shutdown.progress", data); err != nil {
slog.Debug("Failed to publish shutdown progress", "error", err)
}
}
+273
View File
@@ -0,0 +1,273 @@
package daemon
import (
"encoding/json"
"fmt"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// TestShutdownACKMarshal verifies JSON round-trip for ShutdownACK.
func TestShutdownACKMarshal(t *testing.T) {
ack := ShutdownACK{
Node: "node1",
Phase: "gate",
Stopped: []string{"awsgw", "hive-ui"},
}
data, err := json.Marshal(ack)
require.NoError(t, err)
var decoded ShutdownACK
err = json.Unmarshal(data, &decoded)
require.NoError(t, err)
assert.Equal(t, ack.Node, decoded.Node)
assert.Equal(t, ack.Phase, decoded.Phase)
assert.Equal(t, ack.Stopped, decoded.Stopped)
assert.Empty(t, decoded.Error)
}
// TestShutdownACKWithError verifies JSON round-trip for ShutdownACK with an error.
func TestShutdownACKWithError(t *testing.T) {
ack := ShutdownACK{
Node: "node2",
Phase: "drain",
Error: "failed to stop VMs",
}
data, err := json.Marshal(ack)
require.NoError(t, err)
var decoded ShutdownACK
err = json.Unmarshal(data, &decoded)
require.NoError(t, err)
assert.Equal(t, "failed to stop VMs", decoded.Error)
assert.Nil(t, decoded.Stopped)
}
// TestShutdownRequestMarshal verifies JSON round-trip for ShutdownRequest.
func TestShutdownRequestMarshal(t *testing.T) {
req := ShutdownRequest{
Phase: "drain",
Force: true,
Timeout: 120,
}
data, err := json.Marshal(req)
require.NoError(t, err)
var decoded ShutdownRequest
err = json.Unmarshal(data, &decoded)
require.NoError(t, err)
assert.Equal(t, "drain", decoded.Phase)
assert.True(t, decoded.Force)
assert.Equal(t, 120, decoded.Timeout)
}
// TestShutdownProgressMarshal verifies JSON round-trip for ShutdownProgress.
func TestShutdownProgressMarshal(t *testing.T) {
progress := ShutdownProgress{
Node: "node1",
Phase: "drain",
Total: 5,
Remaining: 3,
}
data, err := json.Marshal(progress)
require.NoError(t, err)
var decoded ShutdownProgress
err = json.Unmarshal(data, &decoded)
require.NoError(t, err)
assert.Equal(t, "node1", decoded.Node)
assert.Equal(t, "drain", decoded.Phase)
assert.Equal(t, 5, decoded.Total)
assert.Equal(t, 3, decoded.Remaining)
}
// TestClusterShutdownStateKVRoundTrip verifies cluster shutdown state can be stored and retrieved from KV.
func TestClusterShutdownStateKVRoundTrip(t *testing.T) {
nc, err := nats.Connect(sharedJSNATSURL)
require.NoError(t, err)
defer nc.Close()
jsm, err := NewJetStreamManager(nc, 1)
require.NoError(t, err)
err = jsm.InitClusterStateBucket()
require.NoError(t, err)
state := &ClusterShutdownState{
Initiator: "node1",
Phase: "drain",
Started: "2025-01-01T00:00:00Z",
Timeout: "2m0s",
Force: false,
NodesTotal: 3,
NodesAcked: map[string]string{
"node1": "gate",
"node2": "gate",
},
}
err = jsm.WriteClusterShutdown(state)
require.NoError(t, err)
loaded, err := jsm.ReadClusterShutdown()
require.NoError(t, err)
require.NotNil(t, loaded)
assert.Equal(t, state.Initiator, loaded.Initiator)
assert.Equal(t, state.Phase, loaded.Phase)
assert.Equal(t, state.Started, loaded.Started)
assert.Equal(t, state.NodesTotal, loaded.NodesTotal)
assert.Equal(t, state.Force, loaded.Force)
assert.Len(t, loaded.NodesAcked, 2)
// Cleanup
err = jsm.DeleteClusterShutdown()
require.NoError(t, err)
}
// TestShuttingDownFlagSkipsVMStop verifies that the shuttingDown flag is respected.
func TestShuttingDownFlagSkipsVMStop(t *testing.T) {
d := &Daemon{}
// Default should be false
assert.False(t, d.shuttingDown.Load())
// Set to true (as GATE phase does)
d.shuttingDown.Store(true)
assert.True(t, d.shuttingDown.Load())
}
// TestRespondShutdownACK verifies respondShutdownACK marshals and sends the ACK via NATS request/reply.
func TestRespondShutdownACK(t *testing.T) {
nc, err := nats.Connect(sharedNATSURL)
require.NoError(t, err)
defer nc.Close()
d := &Daemon{node: "test-node", natsConn: nc}
tests := []struct {
name string
ack ShutdownACK
}{
{
name: "gate phase with stopped services",
ack: ShutdownACK{
Node: "test-node",
Phase: "gate",
Stopped: []string{"awsgw", "hive-ui"},
},
},
{
name: "drain phase with error",
ack: ShutdownACK{
Node: "test-node",
Phase: "drain",
Error: "failed to stop VMs",
},
},
{
name: "storage phase empty stopped list",
ack: ShutdownACK{
Node: "test-node",
Phase: "storage",
},
},
}
for i, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
subject := fmt.Sprintf("test.shutdown.ack.%d", i)
// Subscribe and set up a handler that receives requests
sub, err := nc.SubscribeSync(subject)
require.NoError(t, err)
defer sub.Unsubscribe()
require.NoError(t, nc.Flush())
// Send a NATS request — the handler will call msg.Respond()
inbox := nc.NewRespInbox()
replySub, err := nc.SubscribeSync(inbox)
require.NoError(t, err)
defer replySub.Unsubscribe()
require.NoError(t, nc.Flush())
err = nc.PublishRequest(subject, inbox, []byte("{}"))
require.NoError(t, err)
// Receive the request message and pass it to respondShutdownACK
msg, err := sub.NextMsg(2 * time.Second)
require.NoError(t, err)
d.respondShutdownACK(msg, tt.ack)
// Read the reply
reply, err := replySub.NextMsg(2 * time.Second)
require.NoError(t, err)
var decoded ShutdownACK
err = json.Unmarshal(reply.Data, &decoded)
require.NoError(t, err)
assert.Equal(t, tt.ack.Node, decoded.Node)
assert.Equal(t, tt.ack.Phase, decoded.Phase)
assert.Equal(t, tt.ack.Stopped, decoded.Stopped)
assert.Equal(t, tt.ack.Error, decoded.Error)
})
}
}
// TestPublishShutdownProgress verifies publishShutdownProgress publishes correct progress to the NATS topic.
func TestPublishShutdownProgress(t *testing.T) {
nc, err := nats.Connect(sharedNATSURL)
require.NoError(t, err)
defer nc.Close()
d := &Daemon{node: "progress-node", natsConn: nc}
tests := []struct {
name string
phase string
total int
remaining int
}{
{"initial drain progress", "drain", 5, 5},
{"partial drain progress", "drain", 5, 2},
{"final drain progress", "drain", 5, 0},
{"zero VMs", "drain", 0, 0},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
sub, err := nc.SubscribeSync("hive.cluster.shutdown.progress")
require.NoError(t, err)
defer sub.Unsubscribe()
require.NoError(t, nc.Flush())
d.publishShutdownProgress(tt.phase, tt.total, tt.remaining)
require.NoError(t, nc.Flush())
msg, err := sub.NextMsg(2 * time.Second)
require.NoError(t, err)
var progress ShutdownProgress
err = json.Unmarshal(msg.Data, &progress)
require.NoError(t, err)
assert.Equal(t, "progress-node", progress.Node)
assert.Equal(t, tt.phase, progress.Phase)
assert.Equal(t, tt.total, progress.Total)
assert.Equal(t, tt.remaining, progress.Remaining)
})
}
}
+34
View File
@@ -0,0 +1,34 @@
package daemon
import (
"fmt"
"log/slog"
"github.com/mulgadc/hive/hive/vm"
)
// TransitionState validates and applies a state transition on the given instance.
// It sets instance.Status, persists via WriteState, and logs.
// On validation failure, the instance status is unchanged and an error is returned.
// If WriteState fails, the in-memory status retains the new value (the VM has
// physically changed state regardless) and an error is returned.
// The caller must NOT hold d.Instances.Mu; this method acquires it internally.
func (d *Daemon) TransitionState(instance *vm.VM, target vm.InstanceState) error {
d.Instances.Mu.Lock()
current := instance.Status
if !vm.IsValidTransition(current, target) {
d.Instances.Mu.Unlock()
return fmt.Errorf("invalid state transition: %s -> %s for instance %s", current, target, instance.ID)
}
instance.Status = target
d.Instances.Mu.Unlock()
slog.Info("Instance state transition", "instanceId", instance.ID, "from", string(current), "to", string(target))
if err := d.WriteState(); err != nil {
slog.Error("Failed to persist state after transition", "instanceId", instance.ID,
"from", string(current), "to", string(target), "err", err)
return fmt.Errorf("state transition applied but write failed: %w", err)
}
return nil
}
File diff suppressed because it is too large Load Diff
+269
View File
@@ -0,0 +1,269 @@
package formation
import (
"context"
"encoding/json"
"fmt"
"log/slog"
"maps"
"net"
"net/http"
"sync"
"time"
)
// NodeInfo describes a node participating in cluster formation.
type NodeInfo struct {
Name string `json:"name"`
BindIP string `json:"bind_ip"`
ClusterIP string `json:"cluster_ip"`
Region string `json:"region"`
AZ string `json:"az"`
Port int `json:"port"`
Services []string `json:"services,omitempty"`
}
// JoinRequest is the payload POSTed by joining nodes.
type JoinRequest struct {
NodeInfo
}
// JoinResponse is returned by the formation server on join.
type JoinResponse struct {
Success bool `json:"success"`
Message string `json:"message"`
Joined int `json:"joined"`
Expected int `json:"expected"`
}
// StatusResponse is returned by the formation server status endpoint.
type StatusResponse struct {
Complete bool `json:"complete"`
Joined int `json:"joined"`
Expected int `json:"expected"`
Nodes map[string]NodeInfo `json:"nodes,omitempty"`
Credentials *SharedCredentials `json:"credentials,omitempty"`
CACert string `json:"ca_cert,omitempty"`
CAKey string `json:"ca_key,omitempty"`
MasterKey string `json:"master_key,omitempty"`
}
// SharedCredentials contains the cluster-wide credentials distributed during formation.
type SharedCredentials struct {
AccessKey string `json:"access_key"`
SecretKey string `json:"secret_key"`
AccountID string `json:"account_id"`
NatsToken string `json:"nats_token"`
ClusterName string `json:"cluster_name"`
Region string `json:"region"`
}
// FormationServer is a lightweight HTTP server that coordinates cluster formation.
// Nodes register themselves via POST /formation/join. Once the expected number of
// nodes have joined, the done channel is closed and full cluster data (credentials,
// CA, node list) becomes available via GET /formation/status.
type FormationServer struct {
mu sync.RWMutex
expected int
nodes map[string]NodeInfo
credentials *SharedCredentials
caCert string
caKey string
masterKey string
done chan struct{}
server *http.Server
}
// NewFormationServer creates a new formation server expecting the given number of nodes.
func NewFormationServer(expected int, creds *SharedCredentials, caCert, caKey string) *FormationServer {
return &FormationServer{
expected: expected,
nodes: make(map[string]NodeInfo),
credentials: creds,
caCert: caCert,
caKey: caKey,
done: make(chan struct{}),
}
}
// RegisterNode validates and adds a node. Returns an error for duplicates.
func (fs *FormationServer) RegisterNode(info NodeInfo) error {
fs.mu.Lock()
defer fs.mu.Unlock()
if info.Name == "" {
return fmt.Errorf("node name is required")
}
if info.BindIP == "" {
return fmt.Errorf("bind_ip is required")
}
// Check for duplicate name
if _, exists := fs.nodes[info.Name]; exists {
return fmt.Errorf("node %q already registered", info.Name)
}
// Check for duplicate bind IP
for _, n := range fs.nodes {
if n.BindIP == info.BindIP {
return fmt.Errorf("bind IP %s already registered by node %q", info.BindIP, n.Name)
}
}
fs.nodes[info.Name] = info
slog.Info("Node registered", "name", info.Name, "bind_ip", info.BindIP, "joined", len(fs.nodes), "expected", fs.expected)
if fs.isComplete() {
close(fs.done)
}
return nil
}
// isComplete returns true when we have enough nodes. Must be called with lock held.
func (fs *FormationServer) isComplete() bool {
return len(fs.nodes) >= fs.expected
}
// IsComplete returns true when the expected number of nodes have registered.
func (fs *FormationServer) IsComplete() bool {
fs.mu.RLock()
defer fs.mu.RUnlock()
return fs.isComplete()
}
// WaitForCompletion blocks until all nodes have joined or the timeout fires.
func (fs *FormationServer) WaitForCompletion(timeout time.Duration) error {
select {
case <-fs.done:
return nil
case <-time.After(timeout):
fs.mu.RLock()
joined := len(fs.nodes)
fs.mu.RUnlock()
return fmt.Errorf("formation timed out after %s: %d/%d nodes joined", timeout, joined, fs.expected)
}
}
// Nodes returns a copy of the registered nodes.
func (fs *FormationServer) Nodes() map[string]NodeInfo {
fs.mu.RLock()
defer fs.mu.RUnlock()
out := make(map[string]NodeInfo, len(fs.nodes))
maps.Copy(out, fs.nodes)
return out
}
// SetMasterKey sets the base64-encoded IAM master key for distribution to joining nodes.
func (fs *FormationServer) SetMasterKey(key string) {
fs.mu.Lock()
defer fs.mu.Unlock()
fs.masterKey = key
}
// Start launches the HTTP server on the given address (e.g. "10.0.0.1:4432").
func (fs *FormationServer) Start(bindAddr string) error {
mux := http.NewServeMux()
mux.HandleFunc("POST /formation/join", fs.handleJoin)
mux.HandleFunc("GET /formation/status", fs.handleStatus)
mux.HandleFunc("GET /formation/health", fs.handleHealth)
fs.server = &http.Server{
Addr: bindAddr,
Handler: mux,
ReadTimeout: 10 * time.Second,
WriteTimeout: 10 * time.Second,
IdleTimeout: 30 * time.Second,
ReadHeaderTimeout: 5 * time.Second,
}
ln, err := net.Listen("tcp", bindAddr)
if err != nil {
return fmt.Errorf("formation server listen: %w", err)
}
go func() {
if err := fs.server.Serve(ln); err != nil && err != http.ErrServerClosed {
slog.Error("Formation server error", "error", err)
}
}()
slog.Info("Formation server started", "addr", bindAddr)
return nil
}
// Shutdown gracefully stops the formation server.
func (fs *FormationServer) Shutdown(ctx context.Context) error {
if fs.server == nil {
return nil
}
return fs.server.Shutdown(ctx)
}
func (fs *FormationServer) handleJoin(w http.ResponseWriter, r *http.Request) {
var req JoinRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeJSON(w, http.StatusBadRequest, JoinResponse{
Success: false,
Message: fmt.Sprintf("invalid request body: %v", err),
})
return
}
if err := fs.RegisterNode(req.NodeInfo); err != nil {
writeJSON(w, http.StatusConflict, JoinResponse{
Success: false,
Message: err.Error(),
})
return
}
fs.mu.RLock()
joined := len(fs.nodes)
fs.mu.RUnlock()
writeJSON(w, http.StatusOK, JoinResponse{
Success: true,
Message: fmt.Sprintf("node %q registered", req.Name),
Joined: joined,
Expected: fs.expected,
})
}
func (fs *FormationServer) handleStatus(w http.ResponseWriter, r *http.Request) {
fs.mu.RLock()
defer fs.mu.RUnlock()
resp := StatusResponse{
Complete: fs.isComplete(),
Joined: len(fs.nodes),
Expected: fs.expected,
}
// Only expose full data when formation is complete
if fs.isComplete() {
resp.Nodes = make(map[string]NodeInfo, len(fs.nodes))
maps.Copy(resp.Nodes, fs.nodes)
resp.Credentials = fs.credentials
resp.CACert = fs.caCert
resp.CAKey = fs.caKey
resp.MasterKey = fs.masterKey
}
writeJSON(w, http.StatusOK, resp)
}
func (fs *FormationServer) handleHealth(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
if _, err := w.Write([]byte("ok")); err != nil {
slog.Error("Failed to write health response", "error", err)
}
}
func writeJSON(w http.ResponseWriter, status int, v any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
if err := json.NewEncoder(w).Encode(v); err != nil {
slog.Error("Failed to encode JSON response", "error", err)
}
}
+341
View File
@@ -0,0 +1,341 @@
package formation
import (
"bytes"
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func testCreds() *SharedCredentials {
return &SharedCredentials{
AccessKey: "AKIATEST1234567890AB",
SecretKey: "testSecretKey1234567890",
AccountID: "123456789012",
NatsToken: "nats_testtoken123456",
ClusterName: "hive",
Region: "us-west-1",
}
}
func testNode(name, ip string) NodeInfo {
return NodeInfo{
Name: name,
BindIP: ip,
Region: "us-west-1",
AZ: "us-west-1a",
Port: 4432,
}
}
func TestNewFormationServer(t *testing.T) {
t.Parallel()
fs := NewFormationServer(3, testCreds(), "ca-cert-pem", "ca-key-pem")
assert.Equal(t, 3, fs.expected)
assert.Empty(t, fs.nodes)
assert.False(t, fs.IsComplete())
}
func TestRegisterNode(t *testing.T) {
t.Parallel()
fs := NewFormationServer(3, testCreds(), "", "")
err := fs.RegisterNode(testNode("node1", "10.0.0.1"))
require.NoError(t, err)
nodes := fs.Nodes()
assert.Len(t, nodes, 1)
assert.Equal(t, "10.0.0.1", nodes["node1"].BindIP)
}
func TestRegisterDuplicateName(t *testing.T) {
t.Parallel()
fs := NewFormationServer(3, testCreds(), "", "")
require.NoError(t, fs.RegisterNode(testNode("node1", "10.0.0.1")))
err := fs.RegisterNode(testNode("node1", "10.0.0.2"))
assert.Error(t, err)
assert.Contains(t, err.Error(), "already registered")
}
func TestRegisterDuplicateIP(t *testing.T) {
t.Parallel()
fs := NewFormationServer(3, testCreds(), "", "")
require.NoError(t, fs.RegisterNode(testNode("node1", "10.0.0.1")))
err := fs.RegisterNode(testNode("node2", "10.0.0.1"))
assert.Error(t, err)
assert.Contains(t, err.Error(), "bind IP")
}
func TestIsComplete(t *testing.T) {
t.Parallel()
fs := NewFormationServer(2, testCreds(), "", "")
assert.False(t, fs.IsComplete())
require.NoError(t, fs.RegisterNode(testNode("node1", "10.0.0.1")))
assert.False(t, fs.IsComplete())
require.NoError(t, fs.RegisterNode(testNode("node2", "10.0.0.2")))
assert.True(t, fs.IsComplete())
}
func TestWaitForCompletion(t *testing.T) {
t.Parallel()
fs := NewFormationServer(2, testCreds(), "", "")
go func() {
time.Sleep(50 * time.Millisecond)
fs.RegisterNode(testNode("node1", "10.0.0.1"))
fs.RegisterNode(testNode("node2", "10.0.0.2"))
}()
err := fs.WaitForCompletion(5 * time.Second)
require.NoError(t, err)
assert.True(t, fs.IsComplete())
}
func TestWaitForCompletionTimeout(t *testing.T) {
t.Parallel()
fs := NewFormationServer(3, testCreds(), "", "")
require.NoError(t, fs.RegisterNode(testNode("node1", "10.0.0.1")))
err := fs.WaitForCompletion(100 * time.Millisecond)
assert.Error(t, err)
assert.Contains(t, err.Error(), "timed out")
}
func testServer(t *testing.T, fs *FormationServer) *httptest.Server {
t.Helper()
mux := http.NewServeMux()
mux.HandleFunc("POST /formation/join", fs.handleJoin)
mux.HandleFunc("GET /formation/status", fs.handleStatus)
mux.HandleFunc("GET /formation/health", fs.handleHealth)
return httptest.NewServer(mux)
}
func TestJoinEndpoint(t *testing.T) {
t.Parallel()
fs := NewFormationServer(3, testCreds(), "", "")
ts := testServer(t, fs)
defer ts.Close()
body, _ := json.Marshal(JoinRequest{NodeInfo: testNode("node1", "10.0.0.1")})
resp, err := http.Post(ts.URL+"/formation/join", "application/json", bytes.NewReader(body))
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode)
var jr JoinResponse
require.NoError(t, json.NewDecoder(resp.Body).Decode(&jr))
assert.True(t, jr.Success)
assert.Equal(t, 1, jr.Joined)
assert.Equal(t, 3, jr.Expected)
}
func TestJoinEndpointDuplicate(t *testing.T) {
t.Parallel()
fs := NewFormationServer(3, testCreds(), "", "")
ts := testServer(t, fs)
defer ts.Close()
body, _ := json.Marshal(JoinRequest{NodeInfo: testNode("node1", "10.0.0.1")})
resp1, err := http.Post(ts.URL+"/formation/join", "application/json", bytes.NewReader(body))
require.NoError(t, err)
resp1.Body.Close()
assert.Equal(t, http.StatusOK, resp1.StatusCode)
resp2, err := http.Post(ts.URL+"/formation/join", "application/json", bytes.NewReader(body))
require.NoError(t, err)
defer resp2.Body.Close()
assert.Equal(t, http.StatusConflict, resp2.StatusCode)
}
func TestStatusEndpointIncomplete(t *testing.T) {
t.Parallel()
fs := NewFormationServer(3, testCreds(), "ca-cert", "ca-key")
ts := testServer(t, fs)
defer ts.Close()
require.NoError(t, fs.RegisterNode(testNode("node1", "10.0.0.1")))
resp, err := http.Get(ts.URL + "/formation/status")
require.NoError(t, err)
defer resp.Body.Close()
var sr StatusResponse
require.NoError(t, json.NewDecoder(resp.Body).Decode(&sr))
assert.False(t, sr.Complete)
assert.Equal(t, 1, sr.Joined)
assert.Equal(t, 3, sr.Expected)
assert.Nil(t, sr.Credentials)
assert.Empty(t, sr.CACert)
assert.Nil(t, sr.Nodes)
}
func TestStatusEndpointComplete(t *testing.T) {
t.Parallel()
creds := testCreds()
fs := NewFormationServer(2, creds, "ca-cert-data", "ca-key-data")
ts := testServer(t, fs)
defer ts.Close()
require.NoError(t, fs.RegisterNode(testNode("node1", "10.0.0.1")))
require.NoError(t, fs.RegisterNode(testNode("node2", "10.0.0.2")))
resp, err := http.Get(ts.URL + "/formation/status")
require.NoError(t, err)
defer resp.Body.Close()
var sr StatusResponse
require.NoError(t, json.NewDecoder(resp.Body).Decode(&sr))
assert.True(t, sr.Complete)
assert.Equal(t, 2, sr.Joined)
assert.Equal(t, 2, sr.Expected)
assert.Len(t, sr.Nodes, 2)
require.NotNil(t, sr.Credentials)
assert.Equal(t, creds.AccessKey, sr.Credentials.AccessKey)
assert.Equal(t, "ca-cert-data", sr.CACert)
assert.Equal(t, "ca-key-data", sr.CAKey)
}
func TestStatusEndpointMasterKey(t *testing.T) {
t.Parallel()
fs := NewFormationServer(2, testCreds(), "ca-cert", "ca-key")
fs.SetMasterKey("dGVzdC1tYXN0ZXIta2V5LWJhc2U2NC1lbmNvZGVk")
ts := testServer(t, fs)
defer ts.Close()
// Before completion: master key should not be exposed
resp, err := http.Get(ts.URL + "/formation/status")
require.NoError(t, err)
var sr StatusResponse
require.NoError(t, json.NewDecoder(resp.Body).Decode(&sr))
resp.Body.Close()
assert.Empty(t, sr.MasterKey)
// Complete formation
require.NoError(t, fs.RegisterNode(testNode("node1", "10.0.0.1")))
require.NoError(t, fs.RegisterNode(testNode("node2", "10.0.0.2")))
// After completion: master key should be present
resp, err = http.Get(ts.URL + "/formation/status")
require.NoError(t, err)
defer resp.Body.Close()
require.NoError(t, json.NewDecoder(resp.Body).Decode(&sr))
assert.True(t, sr.Complete)
assert.Equal(t, "dGVzdC1tYXN0ZXIta2V5LWJhc2U2NC1lbmNvZGVk", sr.MasterKey)
}
func TestHealthEndpoint(t *testing.T) {
t.Parallel()
fs := NewFormationServer(1, testCreds(), "", "")
ts := testServer(t, fs)
defer ts.Close()
resp, err := http.Get(ts.URL + "/formation/health")
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode)
}
func TestBuildClusterRoutes(t *testing.T) {
t.Parallel()
nodes := map[string]NodeInfo{
"node3": {Name: "node3", BindIP: "10.0.0.3"},
"node1": {Name: "node1", BindIP: "10.0.0.1"},
"node2": {Name: "node2", BindIP: "10.0.0.2"},
}
routes := BuildClusterRoutes(nodes)
assert.Equal(t, []string{"10.0.0.1:4248", "10.0.0.2:4248", "10.0.0.3:4248"}, routes)
}
func TestBuildClusterRoutesUsesClusterIP(t *testing.T) {
t.Parallel()
nodes := map[string]NodeInfo{
"node1": {Name: "node1", BindIP: "10.0.0.1", ClusterIP: "192.168.1.1"},
"node2": {Name: "node2", BindIP: "10.0.0.2"}, // no cluster IP, fallback to bind
}
routes := BuildClusterRoutes(nodes)
assert.Equal(t, []string{"192.168.1.1:4248", "10.0.0.2:4248"}, routes)
}
func TestBuildPredastoreNodes(t *testing.T) {
t.Parallel()
nodes := map[string]NodeInfo{
"node3": {Name: "node3", BindIP: "10.0.0.3"},
"node1": {Name: "node1", BindIP: "10.0.0.1"},
"node2": {Name: "node2", BindIP: "10.0.0.2"},
}
pnodes := BuildPredastoreNodes(nodes)
require.Len(t, pnodes, 3)
assert.Equal(t, 1, pnodes[0].ID)
assert.Equal(t, "10.0.0.1", pnodes[0].Host)
assert.Equal(t, 2, pnodes[1].ID)
assert.Equal(t, "10.0.0.2", pnodes[1].Host)
assert.Equal(t, 3, pnodes[2].ID)
assert.Equal(t, "10.0.0.3", pnodes[2].Host)
}
func TestFullFormationFlow(t *testing.T) {
t.Parallel()
creds := testCreds()
fs := NewFormationServer(3, creds, "-----BEGIN CERTIFICATE-----\ntest\n-----END CERTIFICATE-----", "-----BEGIN PRIVATE KEY-----\ntest\n-----END PRIVATE KEY-----")
ts := testServer(t, fs)
defer ts.Close()
// 3 nodes join sequentially
for i, n := range []struct{ name, ip string }{
{"node1", "10.0.0.1"},
{"node2", "10.0.0.2"},
{"node3", "10.0.0.3"},
} {
body, _ := json.Marshal(JoinRequest{NodeInfo: testNode(n.name, n.ip)})
resp, err := http.Post(ts.URL+"/formation/join", "application/json", bytes.NewReader(body))
require.NoError(t, err)
var jr JoinResponse
require.NoError(t, json.NewDecoder(resp.Body).Decode(&jr))
resp.Body.Close()
assert.True(t, jr.Success)
assert.Equal(t, i+1, jr.Joined)
}
// All nodes poll status — should be complete
resp, err := http.Get(ts.URL + "/formation/status")
require.NoError(t, err)
defer resp.Body.Close()
var sr StatusResponse
require.NoError(t, json.NewDecoder(resp.Body).Decode(&sr))
assert.True(t, sr.Complete)
assert.Equal(t, 3, sr.Joined)
assert.Len(t, sr.Nodes, 3)
require.NotNil(t, sr.Credentials)
assert.Equal(t, creds.AccessKey, sr.Credentials.AccessKey)
assert.Equal(t, creds.SecretKey, sr.Credentials.SecretKey)
assert.Equal(t, creds.NatsToken, sr.Credentials.NatsToken)
assert.NotEmpty(t, sr.CACert)
assert.NotEmpty(t, sr.CAKey)
// Verify helper outputs
routes := BuildClusterRoutes(sr.Nodes)
assert.Len(t, routes, 3)
pnodes := BuildPredastoreNodes(sr.Nodes)
assert.Len(t, pnodes, 3)
assert.Equal(t, 1, pnodes[0].ID)
}
+77
View File
@@ -0,0 +1,77 @@
package formation
import (
"slices"
"sort"
"github.com/mulgadc/hive/hive/admin"
)
// hasService reports whether the node's service list includes name.
// An empty list means all services (backward compat).
func hasService(services []string, name string) bool {
if len(services) == 0 {
return true // backward compat: empty = all services
}
return slices.Contains(services, name)
}
// BuildClusterRoutes returns a sorted list of "IP:4248" NATS cluster routes
// for nodes that run the "nats" service. If ClusterIP is set it is used,
// otherwise BindIP. Nodes are sorted by name for deterministic ordering.
func BuildClusterRoutes(nodes map[string]NodeInfo) []string {
// Filter to nodes that run NATS
natsNodes := make(map[string]NodeInfo)
for k, n := range nodes {
if hasService(n.Services, "nats") {
natsNodes[k] = n
}
}
sorted := sortedNodes(natsNodes)
routes := make([]string, len(sorted))
for i, n := range sorted {
ip := n.ClusterIP
if ip == "" {
ip = n.BindIP
}
routes[i] = ip + ":4248"
}
return routes
}
// BuildPredastoreNodes returns a sorted list of PredastoreNodeConfig with
// 1-based IDs for nodes that run the "predastore" service.
// Nodes are sorted by name for deterministic ordering.
func BuildPredastoreNodes(nodes map[string]NodeInfo) []admin.PredastoreNodeConfig {
// Filter to nodes that run Predastore
predaNodes := make(map[string]NodeInfo)
for k, n := range nodes {
if hasService(n.Services, "predastore") {
predaNodes[k] = n
}
}
sorted := sortedNodes(predaNodes)
out := make([]admin.PredastoreNodeConfig, len(sorted))
for i, n := range sorted {
out[i] = admin.PredastoreNodeConfig{
ID: i + 1,
Host: n.BindIP,
}
}
return out
}
// sortedNodes returns nodes sorted by name.
func sortedNodes(nodes map[string]NodeInfo) []NodeInfo {
names := make([]string, 0, len(nodes))
for name := range nodes {
names = append(names, name)
}
sort.Strings(names)
sorted := make([]NodeInfo, len(names))
for i, name := range names {
sorted[i] = nodes[name]
}
return sorted
}
+87
View File
@@ -0,0 +1,87 @@
package formation
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestHasService_EmptyListMeansAll(t *testing.T) {
t.Parallel()
assert.True(t, hasService(nil, "nats"))
assert.True(t, hasService([]string{}, "predastore"))
}
func TestHasService_ExplicitList(t *testing.T) {
t.Parallel()
services := []string{"nats", "daemon"}
assert.True(t, hasService(services, "nats"))
assert.True(t, hasService(services, "daemon"))
assert.False(t, hasService(services, "predastore"))
}
func TestBuildClusterRoutes_FiltersNonNATSNodes(t *testing.T) {
t.Parallel()
nodes := map[string]NodeInfo{
"node1": {Name: "node1", BindIP: "10.0.0.1", Services: []string{"nats", "daemon"}},
"node2": {Name: "node2", BindIP: "10.0.0.2", Services: []string{"daemon"}}, // no NATS
"node3": {Name: "node3", BindIP: "10.0.0.3", Services: []string{"nats", "predastore"}},
}
routes := BuildClusterRoutes(nodes)
assert.Equal(t, []string{"10.0.0.1:4248", "10.0.0.3:4248"}, routes)
}
func TestBuildClusterRoutes_EmptyServicesIncludesAll(t *testing.T) {
t.Parallel()
// Nodes with empty services (backward compat) should all be included
nodes := map[string]NodeInfo{
"node1": {Name: "node1", BindIP: "10.0.0.1"},
"node2": {Name: "node2", BindIP: "10.0.0.2"},
}
routes := BuildClusterRoutes(nodes)
assert.Equal(t, []string{"10.0.0.1:4248", "10.0.0.2:4248"}, routes)
}
func TestBuildPredastoreNodes_FiltersByPredastore(t *testing.T) {
t.Parallel()
nodes := map[string]NodeInfo{
"node1": {Name: "node1", BindIP: "10.0.0.1", Services: []string{"nats", "predastore"}},
"node2": {Name: "node2", BindIP: "10.0.0.2", Services: []string{"daemon"}}, // no predastore
"node3": {Name: "node3", BindIP: "10.0.0.3", Services: []string{"predastore"}},
}
pnodes := BuildPredastoreNodes(nodes)
require.Len(t, pnodes, 2)
assert.Equal(t, 1, pnodes[0].ID)
assert.Equal(t, "10.0.0.1", pnodes[0].Host)
assert.Equal(t, 2, pnodes[1].ID)
assert.Equal(t, "10.0.0.3", pnodes[1].Host)
}
func TestBuildPredastoreNodes_EmptyServicesIncludesAll(t *testing.T) {
t.Parallel()
nodes := map[string]NodeInfo{
"node1": {Name: "node1", BindIP: "10.0.0.1"},
"node2": {Name: "node2", BindIP: "10.0.0.2"},
"node3": {Name: "node3", BindIP: "10.0.0.3"},
}
pnodes := BuildPredastoreNodes(nodes)
require.Len(t, pnodes, 3)
}
func TestBuildClusterRoutes_MixedServicesAndEmpty(t *testing.T) {
t.Parallel()
nodes := map[string]NodeInfo{
"node1": {Name: "node1", BindIP: "10.0.0.1"}, // empty = all
"node2": {Name: "node2", BindIP: "10.0.0.2", Services: []string{"daemon"}}, // no nats
"node3": {Name: "node3", BindIP: "10.0.0.3", Services: []string{"nats"}}, // has nats
}
routes := BuildClusterRoutes(nodes)
// node1 (empty=all, included), node2 (no nats, excluded), node3 (has nats, included)
assert.Equal(t, []string{"10.0.0.1:4248", "10.0.0.3:4248"}, routes)
}
+2 -2
View File
@@ -1,8 +1,8 @@
package gateway
import "github.com/gofiber/fiber/v2"
import "net/http"
func (gw *GatewayConfig) Account_Request(ctx *fiber.Ctx) error {
func (gw *GatewayConfig) Account_Request(w http.ResponseWriter, r *http.Request) error {
return nil
}
+320
View File
@@ -0,0 +1,320 @@
package gateway
import (
"bytes"
"context"
"crypto/subtle"
"fmt"
"io"
"log/slog"
"net/http"
"sort"
"strings"
"time"
"github.com/google/uuid"
"github.com/mulgadc/hive/hive/awserrors"
handlers_iam "github.com/mulgadc/hive/hive/handlers/iam"
"github.com/mulgadc/predastore/auth"
)
const (
// Maximum allowed clock skew for signature validation (5 minutes)
maxClockSkew = 5 * time.Minute
)
// SigV4AuthMiddleware returns stdlib middleware that validates AWS Signature V4 authentication.
func (gw *GatewayConfig) SigV4AuthMiddleware() func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Skip OPTIONS requests (CORS preflight)
if r.Method == "OPTIONS" {
next.ServeHTTP(w, r)
return
}
authHeader := r.Header.Get("Authorization")
if authHeader == "" {
gw.writeSigV4Error(w, r, awserrors.ErrorMissingAuthenticationToken)
return
}
// Parse the Authorization header
// Format: AWS4-HMAC-SHA256 Credential=ACCESS_KEY/DATE/REGION/SERVICE/aws4_request, SignedHeaders=..., Signature=...
if !strings.HasPrefix(authHeader, "AWS4-HMAC-SHA256 ") {
gw.writeSigV4Error(w, r, awserrors.ErrorIncompleteSignature)
return
}
// Extract components from the Authorization header
parts := strings.Split(authHeader[len("AWS4-HMAC-SHA256 "):], ", ")
if len(parts) != 3 {
gw.writeSigV4Error(w, r, awserrors.ErrorIncompleteSignature)
return
}
// Parse Credential
var accessKey, date, region, service string
if after, ok := strings.CutPrefix(parts[0], "Credential="); ok {
credParts := strings.Split(after, "/")
if len(credParts) != 5 || credParts[4] != "aws4_request" {
gw.writeSigV4Error(w, r, awserrors.ErrorIncompleteSignature)
return
}
accessKey = credParts[0]
date = credParts[1]
region = credParts[2]
service = credParts[3]
} else {
gw.writeSigV4Error(w, r, awserrors.ErrorIncompleteSignature)
return
}
// Parse SignedHeaders
var signedHeaders string
if after, ok := strings.CutPrefix(parts[1], "SignedHeaders="); ok {
signedHeaders = after
} else {
gw.writeSigV4Error(w, r, awserrors.ErrorIncompleteSignature)
return
}
// Parse Signature
var providedSignature string
if after, ok := strings.CutPrefix(parts[2], "Signature="); ok {
providedSignature = after
} else {
gw.writeSigV4Error(w, r, awserrors.ErrorIncompleteSignature)
return
}
// Lookup access key in IAM KV store
if gw.IAMService == nil {
slog.Error("SigV4 auth: IAM service not initialized")
gw.writeSigV4Error(w, r, awserrors.ErrorInternalError)
return
}
ak, err := gw.IAMService.LookupAccessKey(accessKey)
if err != nil {
if strings.Contains(err.Error(), awserrors.ErrorIAMNoSuchEntity) {
slog.Debug("Access key not found", "accessKeyID", accessKey)
gw.writeSigV4Error(w, r, awserrors.ErrorInvalidClientTokenId)
return
}
slog.Error("IAM lookup failed", "accessKeyID", accessKey, "err", err)
gw.writeSigV4Error(w, r, awserrors.ErrorInternalError)
return
}
if ak.Status != handlers_iam.AccessKeyStatusActive {
slog.Debug("Access key inactive", "accessKeyID", accessKey)
gw.writeSigV4Error(w, r, awserrors.ErrorInvalidClientTokenId)
return
}
secret, err := gw.IAMService.DecryptSecret(ak.SecretAccessKey)
if err != nil {
slog.Error("Failed to decrypt IAM secret", "accessKeyID", accessKey, "err", err)
gw.writeSigV4Error(w, r, awserrors.ErrorInternalError)
return
}
// Get timestamp from X-Amz-Date header
timestamp := r.Header.Get("X-Amz-Date")
if timestamp == "" {
gw.writeSigV4Error(w, r, awserrors.ErrorIncompleteSignature)
return
}
// Validate timestamp is within acceptable bounds to prevent replay attacks
parsedTime, err := time.Parse("20060102T150405Z", timestamp)
if err != nil {
slog.Debug("Invalid timestamp format", "timestamp", timestamp)
gw.writeSigV4Error(w, r, awserrors.ErrorIncompleteSignature)
return
}
if time.Since(parsedTime).Abs() > maxClockSkew {
slog.Debug("Signature expired", "timestamp", timestamp, "skew", time.Since(parsedTime))
gw.writeSigV4Error(w, r, awserrors.ErrorSignatureDoesNotMatch)
return
}
// Read body once and re-buffer for downstream handlers
body, err := io.ReadAll(r.Body)
if err != nil {
slog.Error("Failed to read request body", "err", err)
gw.writeSigV4Error(w, r, awserrors.ErrorInternalError)
return
}
r.Body = io.NopCloser(bytes.NewReader(body))
// Compute expected signature using decrypted secret
expectedSignature := computeSignatureWithSecret(r, body, secret, date, timestamp, region, service, signedHeaders)
// Compare signatures using constant-time comparison to prevent timing attacks
if subtle.ConstantTimeCompare([]byte(expectedSignature), []byte(providedSignature)) != 1 {
slog.Debug("Signature mismatch",
"expected", expectedSignature,
"provided", providedSignature,
)
gw.writeSigV4Error(w, r, awserrors.ErrorSignatureDoesNotMatch)
return
}
// Store parsed auth data in context for downstream handlers
ctx := r.Context()
ctx = context.WithValue(ctx, ctxIdentity, ak.UserName)
ctx = context.WithValue(ctx, ctxAccountID, ak.AccountID)
ctx = context.WithValue(ctx, ctxService, service)
ctx = context.WithValue(ctx, ctxRegion, region)
ctx = context.WithValue(ctx, ctxAccessKey, accessKey)
slog.Debug("SigV4 authentication successful", "accessKey", accessKey, "identity", ak.UserName)
next.ServeHTTP(w, r.WithContext(ctx))
})
}
}
// computeSignatureWithSecret builds the canonical request and computes the AWS Signature V4 signature
// using the provided secret key. The body is passed explicitly since it has already been read from r.Body.
func computeSignatureWithSecret(r *http.Request, body []byte, secretKey, date, timestamp, region, service, signedHeaders string) string {
// Build canonical URI (URI-encoded path)
path := r.URL.Path
if path == "" {
path = "/"
}
canonicalURI := auth.UriEncode(path, false)
if canonicalURI == "" {
canonicalURI = "/"
}
// Build canonical query string (sorted, encoded)
canonicalQueryString := buildCanonicalQueryString(r.URL.RawQuery)
// Build canonical headers from SignedHeaders list
headersList := strings.Split(signedHeaders, ";")
sort.Strings(headersList)
var canonicalHeaders strings.Builder
for _, header := range headersList {
header = strings.ToLower(strings.TrimSpace(header))
var value string
if header == "host" {
value = r.Host
} else {
value = r.Header.Get(canonicalHeaderName(header))
}
// Trim leading/trailing whitespace and collapse multiple spaces
value = strings.TrimSpace(value)
canonicalHeaders.WriteString(header)
canonicalHeaders.WriteString(":")
canonicalHeaders.WriteString(value)
canonicalHeaders.WriteString("\n")
}
// Hash payload body with SHA256
payloadHash := auth.HashSHA256(string(body))
// Build canonical request
canonicalRequest := fmt.Sprintf(
"%s\n%s\n%s\n%s\n%s\n%s",
r.Method,
canonicalURI,
canonicalQueryString,
canonicalHeaders.String(),
signedHeaders,
payloadHash,
)
hashedCanonicalRequest := auth.HashSHA256(canonicalRequest)
// Build string-to-sign
scope := fmt.Sprintf("%s/%s/%s/aws4_request", date, region, service)
stringToSign := fmt.Sprintf(
"AWS4-HMAC-SHA256\n%s\n%s\n%s",
timestamp,
scope,
hashedCanonicalRequest,
)
// Derive signing key and compute signature
signingKey := auth.GetSigningKey(secretKey, date, region, service)
signature := auth.HmacSHA256Hex(signingKey, stringToSign)
return signature
}
// buildCanonicalQueryString creates the canonical query string according to AWS specs.
func buildCanonicalQueryString(queryString string) string {
if queryString == "" {
return ""
}
// Parse query parameters
params := make(map[string][]string)
pairs := strings.SplitSeq(queryString, "&")
for pair := range pairs {
if pair == "" {
continue
}
kv := strings.SplitN(pair, "=", 2)
key := kv[0]
value := ""
if len(kv) == 2 {
value = kv[1]
}
params[key] = append(params[key], value)
}
// Sort keys
keys := make([]string, 0, len(params))
for k := range params {
keys = append(keys, k)
}
sort.Strings(keys)
// Build canonical query string
var result []string
for _, key := range keys {
values := params[key]
sort.Strings(values)
encodedKey := auth.UriEncode(key, true)
for _, v := range values {
encodedValue := auth.UriEncode(v, true)
result = append(result, fmt.Sprintf("%s=%s", encodedKey, encodedValue))
}
}
return strings.Join(result, "&")
}
// canonicalHeaderName converts a lowercase header name to the canonical form for lookup.
func canonicalHeaderName(header string) string {
// Convert header names like "x-amz-date" to "X-Amz-Date" for http.Header.Get
parts := strings.Split(header, "-")
for i, part := range parts {
if len(part) > 0 {
parts[i] = strings.ToUpper(part[:1]) + part[1:]
}
}
return strings.Join(parts, "-")
}
// writeSigV4Error writes an EC2-compatible XML error response for authentication failures.
func (gw *GatewayConfig) writeSigV4Error(w http.ResponseWriter, r *http.Request, errorCode string) {
requestID := r.Header.Get("x-amz-request-id")
if requestID == "" {
requestID = uuid.NewString()
}
errorMsg, exists := awserrors.ErrorLookup[errorCode]
if !exists {
errorMsg = awserrors.ErrorMessage{HTTPCode: 500, Message: "Internal error"}
}
xmlError := GenerateEC2ErrorResponse(errorCode, errorMsg.Message, requestID)
w.Header().Set("Content-Type", "application/xml")
w.WriteHeader(errorMsg.HTTPCode)
_, _ = w.Write(xmlError)
}
File diff suppressed because it is too large Load Diff
+235 -353
View File
@@ -1,385 +1,267 @@
// Package gateway provides the AWS Gateway for the Hive platform.
// It handles the incoming requests from the AWS SDK and delegates to the appropriate gateway functions (which calls the NATS handlers).
package gateway
import (
"errors"
"io"
"log/slog"
"net/http"
"strings"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/gofiber/fiber/v2"
"github.com/mulgadc/hive/hive/awsec2query"
"github.com/mulgadc/hive/hive/awserrors"
gateway_ec2_account "github.com/mulgadc/hive/hive/gateway/ec2/account"
gateway_ec2_eigw "github.com/mulgadc/hive/hive/gateway/ec2/eigw"
gateway_ec2_igw "github.com/mulgadc/hive/hive/gateway/ec2/igw"
gateway_ec2_image "github.com/mulgadc/hive/hive/gateway/ec2/image"
gateway_ec2_instance "github.com/mulgadc/hive/hive/gateway/ec2/instance"
gateway_ec2_key "github.com/mulgadc/hive/hive/gateway/ec2/key"
gateway_ec2_snapshot "github.com/mulgadc/hive/hive/gateway/ec2/snapshot"
gateway_ec2_tags "github.com/mulgadc/hive/hive/gateway/ec2/tags"
gateway_ec2_volume "github.com/mulgadc/hive/hive/gateway/ec2/volume"
gateway_ec2_vpc "github.com/mulgadc/hive/hive/gateway/ec2/vpc"
gateway_ec2_zone "github.com/mulgadc/hive/hive/gateway/ec2/zone"
"github.com/mulgadc/hive/hive/utils"
)
func (gw *GatewayConfig) EC2_Request(ctx *fiber.Ctx) error {
// EC2Handler processes parsed query args and returns XML response bytes.
// The action parameter is the EC2 API action name, passed from the map key.
// accountID is the caller's AWS account ID extracted from SigV4 auth context.
type EC2Handler func(action string, q map[string]string, gw *GatewayConfig, accountID string) ([]byte, error)
queryArgs := ParseAWSQueryArgs(string(ctx.Body()))
var xmlOutput []byte
var err error
// Run the action
// TODO: Generate for each action, unit test each, and invalid action
switch queryArgs["Action"] {
case "DescribeInstances":
var input = &ec2.DescribeInstancesInput{}
err = awsec2query.QueryParamsToStruct(queryArgs, input)
if err != nil {
return err
// ec2Handler creates a type-safe EC2Handler that allocates the typed input struct,
// parses query params into it, calls the handler, and marshals the output to XML.
func ec2Handler[In any](handler func(*In, *GatewayConfig, string) (any, error)) EC2Handler {
return func(action string, q map[string]string, gw *GatewayConfig, accountID string) ([]byte, error) {
input := new(In)
if err := awsec2query.QueryParamsToStruct(q, input); err != nil {
return nil, err
}
output, err := gateway_ec2_instance.DescribeInstances(input, gw.NATSConn, gw.ExpectedNodes)
output, err := handler(input, gw, accountID)
if err != nil {
return err
return nil, err
}
// Convert to XML
payload := utils.GenerateXMLPayload("DescribeInstancesResponse", output)
xmlOutput, err = utils.MarshalToXML(payload)
payload := utils.GenerateXMLPayload(action+"Response", output)
xmlOutput, err := utils.MarshalToXML(payload)
if err != nil {
return errors.New("failed to marshal response to XML")
return nil, errors.New("failed to marshal response to XML")
}
case "RunInstances":
return xmlOutput, nil
}
}
var input = &ec2.RunInstancesInput{}
err = awsec2query.QueryParamsToStruct(queryArgs, input)
if err != nil {
return err
var ec2Actions = map[string]EC2Handler{
"DescribeInstances": ec2Handler(func(input *ec2.DescribeInstancesInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_instance.DescribeInstances(input, gw.NATSConn, gw.DiscoverActiveNodes(), accountID)
}),
"RunInstances": ec2Handler(func(input *ec2.RunInstancesInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_instance.RunInstances(input, gw.NATSConn, accountID)
}),
"StartInstances": ec2Handler(func(input *ec2.StartInstancesInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_instance.StartInstances(input, gw.NATSConn, accountID)
}),
"StopInstances": ec2Handler(func(input *ec2.StopInstancesInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_instance.StopInstances(input, gw.NATSConn, accountID)
}),
"TerminateInstances": ec2Handler(func(input *ec2.TerminateInstancesInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_instance.TerminateInstances(input, gw.NATSConn, accountID)
}),
"DescribeInstanceTypes": ec2Handler(func(input *ec2.DescribeInstanceTypesInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_instance.DescribeInstanceTypes(input, gw.NATSConn, gw.ExpectedNodes)
}),
"GetConsoleOutput": ec2Handler(func(input *ec2.GetConsoleOutputInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_instance.GetConsoleOutput(input, gw.NATSConn, accountID)
}),
"ModifyInstanceAttribute": ec2Handler(func(input *ec2.ModifyInstanceAttributeInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_instance.ModifyInstanceAttribute(input, gw.NATSConn, accountID)
}),
"CreateKeyPair": ec2Handler(func(input *ec2.CreateKeyPairInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_key.CreateKeyPair(input, gw.NATSConn, accountID)
}),
"DeleteKeyPair": ec2Handler(func(input *ec2.DeleteKeyPairInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_key.DeleteKeyPair(input, gw.NATSConn, accountID)
}),
"DescribeKeyPairs": ec2Handler(func(input *ec2.DescribeKeyPairsInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_key.DescribeKeyPairs(input, gw.NATSConn, accountID)
}),
"ImportKeyPair": func(action string, q map[string]string, gw *GatewayConfig, accountID string) ([]byte, error) {
// Workaround: parser leaves Base64 padding URL-encoded
if strings.HasSuffix(q["PublicKeyMaterial"], "%3D%3D") {
q["PublicKeyMaterial"] = strings.Replace(q["PublicKeyMaterial"], "%3D%3D", "==", 1)
}
return ec2Handler(func(input *ec2.ImportKeyPairInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_key.ImportKeyPair(input, gw.NATSConn, accountID)
})(action, q, gw, accountID)
},
"DescribeImages": ec2Handler(func(input *ec2.DescribeImagesInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_image.DescribeImages(input, gw.NATSConn, accountID)
}),
"CreateImage": ec2Handler(func(input *ec2.CreateImageInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_image.CreateImage(input, gw.NATSConn, gw.DiscoverActiveNodes(), accountID)
}),
"DescribeRegions": ec2Handler(func(input *ec2.DescribeRegionsInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_zone.DescribeRegions(input, gw.Region)
}),
"DescribeAvailabilityZones": ec2Handler(func(input *ec2.DescribeAvailabilityZonesInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_zone.DescribeAvailabilityZones(input, gw.Region, gw.AZ)
}),
"DescribeVolumes": ec2Handler(func(input *ec2.DescribeVolumesInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_volume.DescribeVolumes(input, gw.NATSConn, accountID)
}),
"ModifyVolume": ec2Handler(func(input *ec2.ModifyVolumeInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_volume.ModifyVolume(input, gw.NATSConn, accountID)
}),
"CreateVolume": ec2Handler(func(input *ec2.CreateVolumeInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_volume.CreateVolume(input, gw.NATSConn, accountID)
}),
"DeleteVolume": ec2Handler(func(input *ec2.DeleteVolumeInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_volume.DeleteVolume(input, gw.NATSConn, accountID)
}),
"AttachVolume": ec2Handler(func(input *ec2.AttachVolumeInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_volume.AttachVolume(input, gw.NATSConn, accountID)
}),
"DescribeVolumeStatus": ec2Handler(func(input *ec2.DescribeVolumeStatusInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_volume.DescribeVolumeStatus(input, gw.NATSConn, accountID)
}),
"DetachVolume": ec2Handler(func(input *ec2.DetachVolumeInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_volume.DetachVolume(input, gw.NATSConn, accountID)
}),
"DescribeAccountAttributes": ec2Handler(func(input *ec2.DescribeAccountAttributesInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_account.DescribeAccountAttributes(input)
}),
"EnableEbsEncryptionByDefault": ec2Handler(func(input *ec2.EnableEbsEncryptionByDefaultInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_account.EnableEbsEncryptionByDefault(input, gw.NATSConn, accountID)
}),
"DisableEbsEncryptionByDefault": ec2Handler(func(input *ec2.DisableEbsEncryptionByDefaultInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_account.DisableEbsEncryptionByDefault(input, gw.NATSConn, accountID)
}),
"GetEbsEncryptionByDefault": ec2Handler(func(input *ec2.GetEbsEncryptionByDefaultInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_account.GetEbsEncryptionByDefault(input, gw.NATSConn, accountID)
}),
"GetSerialConsoleAccessStatus": ec2Handler(func(input *ec2.GetSerialConsoleAccessStatusInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_account.GetSerialConsoleAccessStatus(input, gw.NATSConn, accountID)
}),
"EnableSerialConsoleAccess": ec2Handler(func(input *ec2.EnableSerialConsoleAccessInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_account.EnableSerialConsoleAccess(input, gw.NATSConn, accountID)
}),
"DisableSerialConsoleAccess": ec2Handler(func(input *ec2.DisableSerialConsoleAccessInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_account.DisableSerialConsoleAccess(input, gw.NATSConn, accountID)
}),
"CreateTags": ec2Handler(func(input *ec2.CreateTagsInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_tags.CreateTags(input, gw.NATSConn, accountID)
}),
"DeleteTags": ec2Handler(func(input *ec2.DeleteTagsInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_tags.DeleteTags(input, gw.NATSConn, accountID)
}),
"DescribeTags": ec2Handler(func(input *ec2.DescribeTagsInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_tags.DescribeTags(input, gw.NATSConn, accountID)
}),
"CreateSnapshot": ec2Handler(func(input *ec2.CreateSnapshotInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_snapshot.CreateSnapshot(input, gw.NATSConn, accountID)
}),
"DeleteSnapshot": ec2Handler(func(input *ec2.DeleteSnapshotInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_snapshot.DeleteSnapshot(input, gw.NATSConn, accountID)
}),
"DescribeSnapshots": ec2Handler(func(input *ec2.DescribeSnapshotsInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_snapshot.DescribeSnapshots(input, gw.NATSConn, accountID)
}),
"CopySnapshot": ec2Handler(func(input *ec2.CopySnapshotInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_snapshot.CopySnapshot(input, gw.NATSConn, accountID)
}),
"CreateInternetGateway": ec2Handler(func(input *ec2.CreateInternetGatewayInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_igw.CreateInternetGateway(input, gw.NATSConn, accountID)
}),
"DeleteInternetGateway": ec2Handler(func(input *ec2.DeleteInternetGatewayInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_igw.DeleteInternetGateway(input, gw.NATSConn, accountID)
}),
"DescribeInternetGateways": ec2Handler(func(input *ec2.DescribeInternetGatewaysInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_igw.DescribeInternetGateways(input, gw.NATSConn, accountID)
}),
"AttachInternetGateway": ec2Handler(func(input *ec2.AttachInternetGatewayInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_igw.AttachInternetGateway(input, gw.NATSConn, accountID)
}),
"DetachInternetGateway": ec2Handler(func(input *ec2.DetachInternetGatewayInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_igw.DetachInternetGateway(input, gw.NATSConn, accountID)
}),
"CreateEgressOnlyInternetGateway": ec2Handler(func(input *ec2.CreateEgressOnlyInternetGatewayInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_eigw.CreateEgressOnlyInternetGateway(input, gw.NATSConn, accountID)
}),
"DeleteEgressOnlyInternetGateway": ec2Handler(func(input *ec2.DeleteEgressOnlyInternetGatewayInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_eigw.DeleteEgressOnlyInternetGateway(input, gw.NATSConn, accountID)
}),
"DescribeEgressOnlyInternetGateways": ec2Handler(func(input *ec2.DescribeEgressOnlyInternetGatewaysInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_eigw.DescribeEgressOnlyInternetGateways(input, gw.NATSConn, accountID)
}),
"CreateVpc": ec2Handler(func(input *ec2.CreateVpcInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_vpc.CreateVpc(input, gw.NATSConn, accountID)
}),
"DeleteVpc": ec2Handler(func(input *ec2.DeleteVpcInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_vpc.DeleteVpc(input, gw.NATSConn, accountID)
}),
"DescribeVpcs": ec2Handler(func(input *ec2.DescribeVpcsInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_vpc.DescribeVpcs(input, gw.NATSConn, accountID)
}),
"CreateSubnet": ec2Handler(func(input *ec2.CreateSubnetInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_vpc.CreateSubnet(input, gw.NATSConn, accountID)
}),
"DeleteSubnet": ec2Handler(func(input *ec2.DeleteSubnetInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_vpc.DeleteSubnet(input, gw.NATSConn, accountID)
}),
"DescribeSubnets": ec2Handler(func(input *ec2.DescribeSubnetsInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_vpc.DescribeSubnets(input, gw.NATSConn, accountID)
}),
"CreateNetworkInterface": ec2Handler(func(input *ec2.CreateNetworkInterfaceInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_vpc.CreateNetworkInterface(input, gw.NATSConn, accountID)
}),
"DeleteNetworkInterface": ec2Handler(func(input *ec2.DeleteNetworkInterfaceInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_vpc.DeleteNetworkInterface(input, gw.NATSConn, accountID)
}),
"DescribeNetworkInterfaces": ec2Handler(func(input *ec2.DescribeNetworkInterfacesInput, gw *GatewayConfig, accountID string) (any, error) {
return gateway_ec2_vpc.DescribeNetworkInterfaces(input, gw.NATSConn, accountID)
}),
}
output, err := gateway_ec2_instance.RunInstances(input, gw.NATSConn)
// ec2LocalActions are actions that don't require a NATS connection.
var ec2LocalActions = map[string]bool{
"DescribeRegions": true,
"DescribeAvailabilityZones": true,
"DescribeAccountAttributes": true,
}
if err != nil {
return err
}
func (gw *GatewayConfig) EC2_Request(w http.ResponseWriter, r *http.Request) error {
body, _ := io.ReadAll(r.Body)
queryArgs := ParseAWSQueryArgs(string(body))
// Convert to XML
payload := utils.GenerateXMLPayload("RunInstanceResponse", output)
xmlOutput, err = utils.MarshalToXML(payload)
if err != nil {
return errors.New("failed to marshal response to XML")
}
case "StartInstances":
var input = &ec2.StartInstancesInput{}
err = awsec2query.QueryParamsToStruct(queryArgs, input)
if err != nil {
return err
}
output, err := gateway_ec2_instance.StartInstances(input, gw.NATSConn)
if err != nil {
return err
}
// Convert to XML
payload := utils.GenerateXMLPayload("StartInstancesResponse", output)
xmlOutput, err = utils.MarshalToXML(payload)
if err != nil {
return errors.New("failed to marshal response to XML")
}
case "StopInstances":
var input = &ec2.StopInstancesInput{}
err = awsec2query.QueryParamsToStruct(queryArgs, input)
if err != nil {
return err
}
output, err := gateway_ec2_instance.StopInstances(input, gw.NATSConn)
if err != nil {
return err
}
// Convert to XML
payload := utils.GenerateXMLPayload("StopInstancesResponse", output)
xmlOutput, err = utils.MarshalToXML(payload)
if err != nil {
return errors.New("failed to marshal response to XML")
}
case "TerminateInstances":
var input = &ec2.TerminateInstancesInput{}
err = awsec2query.QueryParamsToStruct(queryArgs, input)
if err != nil {
return err
}
output, err := gateway_ec2_instance.TerminateInstances(input, gw.NATSConn)
if err != nil {
return err
}
// Convert to XML
payload := utils.GenerateXMLPayload("TerminateInstancesResponse", output)
xmlOutput, err = utils.MarshalToXML(payload)
if err != nil {
return errors.New("failed to marshal response to XML")
}
case "CreateKeyPair":
var input = &ec2.CreateKeyPairInput{}
err = awsec2query.QueryParamsToStruct(queryArgs, input)
if err != nil {
return err
}
output, err := gateway_ec2_key.CreateKeyPair(input, gw.NATSConn)
if err != nil {
return err
}
// Convert to XML
payload := utils.GenerateXMLPayload("CreateKeyPairResponse", output)
xmlOutput, err = utils.MarshalToXML(payload)
if err != nil {
return errors.New("failed to marshal response to XML")
}
case "DeleteKeyPair":
var input = &ec2.DeleteKeyPairInput{}
err = awsec2query.QueryParamsToStruct(queryArgs, input)
if err != nil {
return err
}
output, err := gateway_ec2_key.DeleteKeyPair(input, gw.NATSConn)
if err != nil {
return err
}
// Convert to XML
payload := utils.GenerateXMLPayload("DeleteKeyPairResponse", output)
xmlOutput, err = utils.MarshalToXML(payload)
if err != nil {
return errors.New("failed to marshal response to XML")
}
case "DescribeKeyPairs":
var input = &ec2.DescribeKeyPairsInput{}
err = awsec2query.QueryParamsToStruct(queryArgs, input)
if err != nil {
return err
}
output, err := gateway_ec2_key.DescribeKeyPairs(input, gw.NATSConn)
if err != nil {
return err
}
// Convert to XML
payload := utils.GenerateXMLPayload("DescribeKeyPairsResponse", output)
xmlOutput, err = utils.MarshalToXML(payload)
if err != nil {
return errors.New("failed to marshal response to XML")
}
case "ImportKeyPair":
var input = &ec2.ImportKeyPairInput{}
err = awsec2query.QueryParamsToStruct(queryArgs, input)
if err != nil {
return err
}
output, err := gateway_ec2_key.ImportKeyPair(input, gw.NATSConn)
if err != nil {
return err
}
// Convert to XML
payload := utils.GenerateXMLPayload("ImportKeyPairResponse", output)
xmlOutput, err = utils.MarshalToXML(payload)
if err != nil {
return errors.New("failed to marshal response to XML")
}
case "DescribeImages":
var input = &ec2.DescribeImagesInput{}
err = awsec2query.QueryParamsToStruct(queryArgs, input)
if err != nil {
return err
}
output, err := gateway_ec2_image.DescribeImages(input, gw.NATSConn)
if err != nil {
return err
}
// Convert to XML
payload := utils.GenerateXMLPayload("DescribeImagesResponse", output)
xmlOutput, err = utils.MarshalToXML(payload)
if err != nil {
return errors.New("failed to marshal response to XML")
}
default:
err = errors.New("InvalidAction")
action := queryArgs["Action"]
handler, ok := ec2Actions[action]
if !ok {
return errors.New(awserrors.ErrorInvalidAction)
}
// Return an error XML
if err := gw.checkPolicy(r, "ec2", action); err != nil {
return err
}
if gw.NATSConn == nil && !ec2LocalActions[action] {
return errors.New(awserrors.ErrorServerInternal)
}
// Extract account ID from auth context
accountID, _ := r.Context().Value(ctxAccountID).(string)
if accountID == "" {
slog.Error("EC2_Request: no account ID in auth context")
return errors.New(awserrors.ErrorServerInternal)
}
xmlOutput, err := handler(action, queryArgs, gw, accountID)
if err != nil {
return err
}
ctx.Status(fiber.StatusOK).Type("text/xml").Send(xmlOutput)
w.Header().Set("Content-Type", "text/xml")
w.WriteHeader(http.StatusOK)
_, _ = w.Write(xmlOutput)
return nil
}
/*
func (gw *GatewayConfig) EC2_DescribeInstances(ctx *fiber.Ctx, args map[string]string) error {
slog.Info("EC2 DescribeInstances called")
// Return a dummy response
response := `<?xml version="1.0" encoding="UTF-8"?>
<DescribeInstancesResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
<requestId>2b7ac2f1-9acd-4d73-b0a1-1b2e5607f9ab</requestId>
<reservationSet>
<item>
<reservationId>r-0f6b3c4d5e6789abc</reservationId>
<ownerId>123456789012</ownerId>
<groupSet/>
<instancesSet>
<item>
<instanceId>i-0123456789abcdef0</instanceId>
<imageId>ami-08d4ac5b634553e16</imageId>
<instanceState>
<code>16</code>
<name>running</name>
</instanceState>
<privateDnsName>ip-10-0-1-25.ap-southeast-2.compute.internal</privateDnsName>
<dnsName>ec2-3-26-14-112.ap-southeast-2.compute.amazonaws.com</dnsName>
<reason/>
<keyName>my-keypair</keyName>
<amiLaunchIndex>0</amiLaunchIndex>
<productCodes/>
<instanceType>t3.medium</instanceType>
<launchTime>2025-02-18T04:12:07.000Z</launchTime>
<placement>
<availabilityZone>ap-southeast-2a</availabilityZone>
<groupName/>
<tenancy>default</tenancy>
</placement>
<monitoring>
<state>disabled</state>
</monitoring>
<privateIpAddress>10.0.1.25</privateIpAddress>
<ipAddress>3.26.14.112</ipAddress>
<subnetId>subnet-0ab1c2d3e4f567890</subnetId>
<vpcId>vpc-0123abcd4567efgh8</vpcId>
<sourceDestCheck>true</sourceDestCheck>
<groupSet>
<item>
<groupId>sg-0abc1234def567890</groupId>
<groupName>default</groupName>
</item>
</groupSet>
<stateReason/>
<architecture>x86_64</architecture>
<rootDeviceType>ebs</rootDeviceType>
<rootDeviceName>/dev/xvda</rootDeviceName>
<blockDeviceMapping>
<item>
<deviceName>/dev/xvda</deviceName>
<ebs>
<volumeId>vol-06f5e4d3c2b1a0f98</volumeId>
<status>attached</status>
<attachTime>2025-02-18T04:12:09.000Z</attachTime>
<deleteOnTermination>true</deleteOnTermination>
</ebs>
</item>
</blockDeviceMapping>
<virtualizationType>hvm</virtualizationType>
<tagSet>
<item>
<key>Name</key>
<value>web-01</value>
</item>
</tagSet>
<hypervisor>xen</hypervisor>
<networkInterfaceSet>
<item>
<networkInterfaceId>eni-0a12b34c56d78ef90</networkInterfaceId>
<subnetId>subnet-0ab1c2d3e4f567890</subnetId>
<vpcId>vpc-0123abcd4567efgh8</vpcId>
<description/>
<ownerId>123456789012</ownerId>
<status>in-use</status>
<macAddress>0a:1b:2c:3d:4e:5f</macAddress>
<privateIpAddress>10.0.1.25</privateIpAddress>
<privateDnsName>ip-10-0-1-25.ap-southeast-2.compute.internal</privateDnsName>
<sourceDestCheck>true</sourceDestCheck>
<groupSet>
<item>
<groupId>sg-0abc1234def567890</groupId>
<groupName>default</groupName>
</item>
</groupSet>
<attachment>
<attachmentId>eni-attach-0abc123def4567890</attachmentId>
<deviceIndex>0</deviceIndex>
<status>attached</status>
<attachTime>2025-02-18T04:12:07.000Z</attachTime>
<deleteOnTermination>true</deleteOnTermination>
</attachment>
<privateIpAddressesSet>
<item>
<privateIpAddress>10.0.1.25</privateIpAddress>
<privateDnsName>ip-10-0-1-25.ap-southeast-2.compute.internal</privateDnsName>
<primary>true</primary>
</item>
</privateIpAddressesSet>
</item>
</networkInterfaceSet>
</item>
</instancesSet>
</item>
</reservationSet>
</DescribeInstancesResponse>
`
return ctx.SendString(response)
}
*/
@@ -0,0 +1,93 @@
package gateway_ec2_account
import (
"log/slog"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/ec2"
)
// DescribeAccountAttributes returns static account attributes for the Hive platform
func DescribeAccountAttributes(input *ec2.DescribeAccountAttributesInput) (*ec2.DescribeAccountAttributesOutput, error) {
slog.Debug("DescribeAccountAttributes called")
// Build set of requested attribute names for filtering
requestedAttrs := make(map[string]bool)
for _, name := range input.AttributeNames {
if name != nil {
requestedAttrs[*name] = true
}
}
// If no specific attributes requested, return all
returnAll := len(requestedAttrs) == 0
var accountAttributes []*ec2.AccountAttribute
// supported-platforms - VPC only (no EC2-Classic)
if returnAll || requestedAttrs["supported-platforms"] {
accountAttributes = append(accountAttributes, &ec2.AccountAttribute{
AttributeName: aws.String("supported-platforms"),
AttributeValues: []*ec2.AccountAttributeValue{
{AttributeValue: aws.String("VPC")},
},
})
}
// default-vpc
if returnAll || requestedAttrs["default-vpc"] {
accountAttributes = append(accountAttributes, &ec2.AccountAttribute{
AttributeName: aws.String("default-vpc"),
AttributeValues: []*ec2.AccountAttributeValue{
{AttributeValue: aws.String("none")},
},
})
}
// max-instances
if returnAll || requestedAttrs["max-instances"] {
accountAttributes = append(accountAttributes, &ec2.AccountAttribute{
AttributeName: aws.String("max-instances"),
AttributeValues: []*ec2.AccountAttributeValue{
{AttributeValue: aws.String("100")},
},
})
}
// vpc-max-security-groups-per-interface
if returnAll || requestedAttrs["vpc-max-security-groups-per-interface"] {
accountAttributes = append(accountAttributes, &ec2.AccountAttribute{
AttributeName: aws.String("vpc-max-security-groups-per-interface"),
AttributeValues: []*ec2.AccountAttributeValue{
{AttributeValue: aws.String("5")},
},
})
}
// max-elastic-ips
if returnAll || requestedAttrs["max-elastic-ips"] {
accountAttributes = append(accountAttributes, &ec2.AccountAttribute{
AttributeName: aws.String("max-elastic-ips"),
AttributeValues: []*ec2.AccountAttributeValue{
{AttributeValue: aws.String("5")},
},
})
}
// vpc-max-elastic-ips
if returnAll || requestedAttrs["vpc-max-elastic-ips"] {
accountAttributes = append(accountAttributes, &ec2.AccountAttribute{
AttributeName: aws.String("vpc-max-elastic-ips"),
AttributeValues: []*ec2.AccountAttributeValue{
{AttributeValue: aws.String("20")},
},
})
}
output := &ec2.DescribeAccountAttributesOutput{
AccountAttributes: accountAttributes,
}
slog.Info("DescribeAccountAttributes completed", "attributeCount", len(accountAttributes))
return output, nil
}
@@ -0,0 +1,33 @@
package gateway_ec2_account
import (
"errors"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/mulgadc/hive/hive/awserrors"
handlers_ec2_account "github.com/mulgadc/hive/hive/handlers/ec2/account"
"github.com/nats-io/nats.go"
)
func ValidateDisableEbsEncryptionByDefaultInput(input *ec2.DisableEbsEncryptionByDefaultInput) error {
if input == nil {
return errors.New(awserrors.ErrorInvalidParameterValue)
}
return nil
}
func DisableEbsEncryptionByDefault(input *ec2.DisableEbsEncryptionByDefaultInput, natsConn *nats.Conn, accountID string) (ec2.DisableEbsEncryptionByDefaultOutput, error) {
var output ec2.DisableEbsEncryptionByDefaultOutput
if err := ValidateDisableEbsEncryptionByDefaultInput(input); err != nil {
return output, err
}
svc := handlers_ec2_account.NewNATSAccountSettingsService(natsConn)
result, err := svc.DisableEbsEncryptionByDefault(input, accountID)
if err != nil {
return output, err
}
return *result, nil
}
@@ -0,0 +1,33 @@
package gateway_ec2_account
import (
"errors"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/mulgadc/hive/hive/awserrors"
handlers_ec2_account "github.com/mulgadc/hive/hive/handlers/ec2/account"
"github.com/nats-io/nats.go"
)
func ValidateDisableSerialConsoleAccessInput(input *ec2.DisableSerialConsoleAccessInput) error {
if input == nil {
return errors.New(awserrors.ErrorInvalidParameterValue)
}
return nil
}
func DisableSerialConsoleAccess(input *ec2.DisableSerialConsoleAccessInput, natsConn *nats.Conn, accountID string) (ec2.DisableSerialConsoleAccessOutput, error) {
var output ec2.DisableSerialConsoleAccessOutput
if err := ValidateDisableSerialConsoleAccessInput(input); err != nil {
return output, err
}
svc := handlers_ec2_account.NewNATSAccountSettingsService(natsConn)
result, err := svc.DisableSerialConsoleAccess(input, accountID)
if err != nil {
return output, err
}
return *result, nil
}
@@ -0,0 +1,33 @@
package gateway_ec2_account
import (
"errors"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/mulgadc/hive/hive/awserrors"
handlers_ec2_account "github.com/mulgadc/hive/hive/handlers/ec2/account"
"github.com/nats-io/nats.go"
)
func ValidateEnableEbsEncryptionByDefaultInput(input *ec2.EnableEbsEncryptionByDefaultInput) error {
if input == nil {
return errors.New(awserrors.ErrorInvalidParameterValue)
}
return nil
}
func EnableEbsEncryptionByDefault(input *ec2.EnableEbsEncryptionByDefaultInput, natsConn *nats.Conn, accountID string) (ec2.EnableEbsEncryptionByDefaultOutput, error) {
var output ec2.EnableEbsEncryptionByDefaultOutput
if err := ValidateEnableEbsEncryptionByDefaultInput(input); err != nil {
return output, err
}
svc := handlers_ec2_account.NewNATSAccountSettingsService(natsConn)
result, err := svc.EnableEbsEncryptionByDefault(input, accountID)
if err != nil {
return output, err
}
return *result, nil
}
@@ -0,0 +1,33 @@
package gateway_ec2_account
import (
"errors"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/mulgadc/hive/hive/awserrors"
handlers_ec2_account "github.com/mulgadc/hive/hive/handlers/ec2/account"
"github.com/nats-io/nats.go"
)
func ValidateEnableSerialConsoleAccessInput(input *ec2.EnableSerialConsoleAccessInput) error {
if input == nil {
return errors.New(awserrors.ErrorInvalidParameterValue)
}
return nil
}
func EnableSerialConsoleAccess(input *ec2.EnableSerialConsoleAccessInput, natsConn *nats.Conn, accountID string) (ec2.EnableSerialConsoleAccessOutput, error) {
var output ec2.EnableSerialConsoleAccessOutput
if err := ValidateEnableSerialConsoleAccessInput(input); err != nil {
return output, err
}
svc := handlers_ec2_account.NewNATSAccountSettingsService(natsConn)
result, err := svc.EnableSerialConsoleAccess(input, accountID)
if err != nil {
return output, err
}
return *result, nil
}
@@ -0,0 +1,33 @@
package gateway_ec2_account
import (
"errors"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/mulgadc/hive/hive/awserrors"
handlers_ec2_account "github.com/mulgadc/hive/hive/handlers/ec2/account"
"github.com/nats-io/nats.go"
)
func ValidateGetEbsEncryptionByDefaultInput(input *ec2.GetEbsEncryptionByDefaultInput) error {
if input == nil {
return errors.New(awserrors.ErrorInvalidParameterValue)
}
return nil
}
func GetEbsEncryptionByDefault(input *ec2.GetEbsEncryptionByDefaultInput, natsConn *nats.Conn, accountID string) (ec2.GetEbsEncryptionByDefaultOutput, error) {
var output ec2.GetEbsEncryptionByDefaultOutput
if err := ValidateGetEbsEncryptionByDefaultInput(input); err != nil {
return output, err
}
svc := handlers_ec2_account.NewNATSAccountSettingsService(natsConn)
result, err := svc.GetEbsEncryptionByDefault(input, accountID)
if err != nil {
return output, err
}
return *result, nil
}
@@ -0,0 +1,33 @@
package gateway_ec2_account
import (
"errors"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/mulgadc/hive/hive/awserrors"
handlers_ec2_account "github.com/mulgadc/hive/hive/handlers/ec2/account"
"github.com/nats-io/nats.go"
)
func ValidateGetSerialConsoleAccessStatusInput(input *ec2.GetSerialConsoleAccessStatusInput) error {
if input == nil {
return errors.New(awserrors.ErrorInvalidParameterValue)
}
return nil
}
func GetSerialConsoleAccessStatus(input *ec2.GetSerialConsoleAccessStatusInput, natsConn *nats.Conn, accountID string) (ec2.GetSerialConsoleAccessStatusOutput, error) {
var output ec2.GetSerialConsoleAccessStatusOutput
if err := ValidateGetSerialConsoleAccessStatusInput(input); err != nil {
return output, err
}
svc := handlers_ec2_account.NewNATSAccountSettingsService(natsConn)
result, err := svc.GetSerialConsoleAccessStatus(input, accountID)
if err != nil {
return output, err
}
return *result, nil
}
+342
View File
@@ -0,0 +1,342 @@
package gateway_ec2_account
import (
"testing"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/mulgadc/hive/hive/awserrors"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestDescribeAccountAttributes_AllAttributes(t *testing.T) {
input := &ec2.DescribeAccountAttributesInput{}
output, err := DescribeAccountAttributes(input)
require.NoError(t, err)
require.NotNil(t, output)
assert.Len(t, output.AccountAttributes, 6)
attrMap := make(map[string]string)
for _, attr := range output.AccountAttributes {
attrMap[*attr.AttributeName] = *attr.AttributeValues[0].AttributeValue
}
assert.Equal(t, "VPC", attrMap["supported-platforms"])
assert.Equal(t, "none", attrMap["default-vpc"])
assert.Equal(t, "100", attrMap["max-instances"])
assert.Equal(t, "5", attrMap["vpc-max-security-groups-per-interface"])
assert.Equal(t, "5", attrMap["max-elastic-ips"])
assert.Equal(t, "20", attrMap["vpc-max-elastic-ips"])
}
func TestDescribeAccountAttributes_FilterSingle(t *testing.T) {
input := &ec2.DescribeAccountAttributesInput{
AttributeNames: []*string{aws.String("max-instances")},
}
output, err := DescribeAccountAttributes(input)
require.NoError(t, err)
require.NotNil(t, output)
assert.Len(t, output.AccountAttributes, 1)
assert.Equal(t, "max-instances", *output.AccountAttributes[0].AttributeName)
assert.Equal(t, "100", *output.AccountAttributes[0].AttributeValues[0].AttributeValue)
}
func TestDescribeAccountAttributes_FilterMultiple(t *testing.T) {
input := &ec2.DescribeAccountAttributesInput{
AttributeNames: []*string{
aws.String("supported-platforms"),
aws.String("default-vpc"),
},
}
output, err := DescribeAccountAttributes(input)
require.NoError(t, err)
require.NotNil(t, output)
assert.Len(t, output.AccountAttributes, 2)
names := make(map[string]bool)
for _, attr := range output.AccountAttributes {
names[*attr.AttributeName] = true
}
assert.True(t, names["supported-platforms"])
assert.True(t, names["default-vpc"])
}
func TestDescribeAccountAttributes_FilterNonExistent(t *testing.T) {
input := &ec2.DescribeAccountAttributesInput{
AttributeNames: []*string{aws.String("nonexistent-attribute")},
}
output, err := DescribeAccountAttributes(input)
require.NoError(t, err)
require.NotNil(t, output)
assert.Len(t, output.AccountAttributes, 0)
}
func TestDescribeAccountAttributes_EmptyAttributeNames(t *testing.T) {
input := &ec2.DescribeAccountAttributesInput{
AttributeNames: []*string{},
}
output, err := DescribeAccountAttributes(input)
require.NoError(t, err)
require.NotNil(t, output)
// Empty slice means return all
assert.Len(t, output.AccountAttributes, 6)
}
func TestValidateEnableEbsEncryptionByDefaultInput(t *testing.T) {
tests := []struct {
name string
input *ec2.EnableEbsEncryptionByDefaultInput
wantErr bool
}{
{
name: "NilInput",
input: nil,
wantErr: true,
},
{
name: "ValidInput",
input: &ec2.EnableEbsEncryptionByDefaultInput{},
wantErr: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateEnableEbsEncryptionByDefaultInput(tt.input)
if tt.wantErr {
assert.Error(t, err)
assert.Equal(t, awserrors.ErrorInvalidParameterValue, err.Error())
} else {
assert.NoError(t, err)
}
})
}
}
func TestValidateDisableEbsEncryptionByDefaultInput(t *testing.T) {
tests := []struct {
name string
input *ec2.DisableEbsEncryptionByDefaultInput
wantErr bool
}{
{
name: "NilInput",
input: nil,
wantErr: true,
},
{
name: "ValidInput",
input: &ec2.DisableEbsEncryptionByDefaultInput{},
wantErr: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateDisableEbsEncryptionByDefaultInput(tt.input)
if tt.wantErr {
assert.Error(t, err)
assert.Equal(t, awserrors.ErrorInvalidParameterValue, err.Error())
} else {
assert.NoError(t, err)
}
})
}
}
func TestValidateGetEbsEncryptionByDefaultInput(t *testing.T) {
tests := []struct {
name string
input *ec2.GetEbsEncryptionByDefaultInput
wantErr bool
}{
{
name: "NilInput",
input: nil,
wantErr: true,
},
{
name: "ValidInput",
input: &ec2.GetEbsEncryptionByDefaultInput{},
wantErr: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateGetEbsEncryptionByDefaultInput(tt.input)
if tt.wantErr {
assert.Error(t, err)
assert.Equal(t, awserrors.ErrorInvalidParameterValue, err.Error())
} else {
assert.NoError(t, err)
}
})
}
}
func TestValidateEnableSerialConsoleAccessInput(t *testing.T) {
tests := []struct {
name string
input *ec2.EnableSerialConsoleAccessInput
wantErr bool
}{
{
name: "NilInput",
input: nil,
wantErr: true,
},
{
name: "ValidInput",
input: &ec2.EnableSerialConsoleAccessInput{},
wantErr: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateEnableSerialConsoleAccessInput(tt.input)
if tt.wantErr {
assert.Error(t, err)
assert.Equal(t, awserrors.ErrorInvalidParameterValue, err.Error())
} else {
assert.NoError(t, err)
}
})
}
}
func TestValidateDisableSerialConsoleAccessInput(t *testing.T) {
tests := []struct {
name string
input *ec2.DisableSerialConsoleAccessInput
wantErr bool
}{
{
name: "NilInput",
input: nil,
wantErr: true,
},
{
name: "ValidInput",
input: &ec2.DisableSerialConsoleAccessInput{},
wantErr: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateDisableSerialConsoleAccessInput(tt.input)
if tt.wantErr {
assert.Error(t, err)
assert.Equal(t, awserrors.ErrorInvalidParameterValue, err.Error())
} else {
assert.NoError(t, err)
}
})
}
}
func TestValidateGetSerialConsoleAccessStatusInput(t *testing.T) {
tests := []struct {
name string
input *ec2.GetSerialConsoleAccessStatusInput
wantErr bool
}{
{
name: "NilInput",
input: nil,
wantErr: true,
},
{
name: "ValidInput",
input: &ec2.GetSerialConsoleAccessStatusInput{},
wantErr: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateGetSerialConsoleAccessStatusInput(tt.input)
if tt.wantErr {
assert.Error(t, err)
assert.Equal(t, awserrors.ErrorInvalidParameterValue, err.Error())
} else {
assert.NoError(t, err)
}
})
}
}
// Handler tests — call handlers with valid input + nil NATS to cover error paths
func TestEnableEbsEncryptionByDefault_ValidationErrors(t *testing.T) {
_, err := EnableEbsEncryptionByDefault(nil, nil, "")
assert.EqualError(t, err, awserrors.ErrorInvalidParameterValue)
}
func TestEnableEbsEncryptionByDefault_NilNATS(t *testing.T) {
_, err := EnableEbsEncryptionByDefault(&ec2.EnableEbsEncryptionByDefaultInput{}, nil, "acct-123")
assert.Error(t, err)
}
func TestDisableEbsEncryptionByDefault_ValidationErrors(t *testing.T) {
_, err := DisableEbsEncryptionByDefault(nil, nil, "")
assert.EqualError(t, err, awserrors.ErrorInvalidParameterValue)
}
func TestDisableEbsEncryptionByDefault_NilNATS(t *testing.T) {
_, err := DisableEbsEncryptionByDefault(&ec2.DisableEbsEncryptionByDefaultInput{}, nil, "acct-123")
assert.Error(t, err)
}
func TestGetEbsEncryptionByDefault_ValidationErrors(t *testing.T) {
_, err := GetEbsEncryptionByDefault(nil, nil, "")
assert.EqualError(t, err, awserrors.ErrorInvalidParameterValue)
}
func TestGetEbsEncryptionByDefault_NilNATS(t *testing.T) {
_, err := GetEbsEncryptionByDefault(&ec2.GetEbsEncryptionByDefaultInput{}, nil, "acct-123")
assert.Error(t, err)
}
func TestEnableSerialConsoleAccess_ValidationErrors(t *testing.T) {
_, err := EnableSerialConsoleAccess(nil, nil, "")
assert.EqualError(t, err, awserrors.ErrorInvalidParameterValue)
}
func TestEnableSerialConsoleAccess_NilNATS(t *testing.T) {
_, err := EnableSerialConsoleAccess(&ec2.EnableSerialConsoleAccessInput{}, nil, "acct-123")
assert.Error(t, err)
}
func TestDisableSerialConsoleAccess_ValidationErrors(t *testing.T) {
_, err := DisableSerialConsoleAccess(nil, nil, "")
assert.EqualError(t, err, awserrors.ErrorInvalidParameterValue)
}
func TestDisableSerialConsoleAccess_NilNATS(t *testing.T) {
_, err := DisableSerialConsoleAccess(&ec2.DisableSerialConsoleAccessInput{}, nil, "acct-123")
assert.Error(t, err)
}
func TestGetSerialConsoleAccessStatus_ValidationErrors(t *testing.T) {
_, err := GetSerialConsoleAccessStatus(nil, nil, "")
assert.EqualError(t, err, awserrors.ErrorInvalidParameterValue)
}
func TestGetSerialConsoleAccessStatus_NilNATS(t *testing.T) {
_, err := GetSerialConsoleAccessStatus(&ec2.GetSerialConsoleAccessStatusInput{}, nil, "acct-123")
assert.Error(t, err)
}
@@ -0,0 +1,40 @@
package gateway_ec2_eigw
import (
"errors"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/mulgadc/hive/hive/awserrors"
handlers_ec2_eigw "github.com/mulgadc/hive/hive/handlers/ec2/eigw"
"github.com/nats-io/nats.go"
)
// ValidateCreateEgressOnlyInternetGatewayInput validates the input parameters
func ValidateCreateEgressOnlyInternetGatewayInput(input *ec2.CreateEgressOnlyInternetGatewayInput) error {
if input == nil {
return errors.New(awserrors.ErrorInvalidParameterValue)
}
if input.VpcId == nil || *input.VpcId == "" {
return errors.New(awserrors.ErrorMissingParameter)
}
return nil
}
// CreateEgressOnlyInternetGateway handles the EC2 CreateEgressOnlyInternetGateway API call
func CreateEgressOnlyInternetGateway(input *ec2.CreateEgressOnlyInternetGatewayInput, natsConn *nats.Conn, accountID string) (ec2.CreateEgressOnlyInternetGatewayOutput, error) {
var output ec2.CreateEgressOnlyInternetGatewayOutput
if err := ValidateCreateEgressOnlyInternetGatewayInput(input); err != nil {
return output, err
}
svc := handlers_ec2_eigw.NewNATSEgressOnlyIGWService(natsConn)
result, err := svc.CreateEgressOnlyInternetGateway(input, accountID)
if err != nil {
return output, err
}
return *result, nil
}
@@ -0,0 +1,40 @@
package gateway_ec2_eigw
import (
"errors"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/mulgadc/hive/hive/awserrors"
handlers_ec2_eigw "github.com/mulgadc/hive/hive/handlers/ec2/eigw"
"github.com/nats-io/nats.go"
)
// ValidateDeleteEgressOnlyInternetGatewayInput validates the input parameters
func ValidateDeleteEgressOnlyInternetGatewayInput(input *ec2.DeleteEgressOnlyInternetGatewayInput) error {
if input == nil {
return errors.New(awserrors.ErrorInvalidParameterValue)
}
if input.EgressOnlyInternetGatewayId == nil || *input.EgressOnlyInternetGatewayId == "" {
return errors.New(awserrors.ErrorMissingParameter)
}
return nil
}
// DeleteEgressOnlyInternetGateway handles the EC2 DeleteEgressOnlyInternetGateway API call
func DeleteEgressOnlyInternetGateway(input *ec2.DeleteEgressOnlyInternetGatewayInput, natsConn *nats.Conn, accountID string) (ec2.DeleteEgressOnlyInternetGatewayOutput, error) {
var output ec2.DeleteEgressOnlyInternetGatewayOutput
if err := ValidateDeleteEgressOnlyInternetGatewayInput(input); err != nil {
return output, err
}
svc := handlers_ec2_eigw.NewNATSEgressOnlyIGWService(natsConn)
result, err := svc.DeleteEgressOnlyInternetGateway(input, accountID)
if err != nil {
return output, err
}
return *result, nil
}
@@ -0,0 +1,36 @@
package gateway_ec2_eigw
import (
"errors"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/mulgadc/hive/hive/awserrors"
handlers_ec2_eigw "github.com/mulgadc/hive/hive/handlers/ec2/eigw"
"github.com/nats-io/nats.go"
)
// ValidateDescribeEgressOnlyInternetGatewaysInput validates the input parameters
func ValidateDescribeEgressOnlyInternetGatewaysInput(input *ec2.DescribeEgressOnlyInternetGatewaysInput) error {
if input == nil {
return errors.New(awserrors.ErrorInvalidParameterValue)
}
return nil
}
// DescribeEgressOnlyInternetGateways handles the EC2 DescribeEgressOnlyInternetGateways API call
func DescribeEgressOnlyInternetGateways(input *ec2.DescribeEgressOnlyInternetGatewaysInput, natsConn *nats.Conn, accountID string) (ec2.DescribeEgressOnlyInternetGatewaysOutput, error) {
var output ec2.DescribeEgressOnlyInternetGatewaysOutput
if err := ValidateDescribeEgressOnlyInternetGatewaysInput(input); err != nil {
return output, err
}
svc := handlers_ec2_eigw.NewNATSEgressOnlyIGWService(natsConn)
result, err := svc.DescribeEgressOnlyInternetGateways(input, accountID)
if err != nil {
return output, err
}
return *result, nil
}
+169
View File
@@ -0,0 +1,169 @@
package gateway_ec2_eigw
import (
"testing"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/mulgadc/hive/hive/awserrors"
"github.com/stretchr/testify/assert"
)
func TestValidateCreateEgressOnlyInternetGatewayInput(t *testing.T) {
tests := []struct {
name string
input *ec2.CreateEgressOnlyInternetGatewayInput
wantErr string
}{
{
name: "nil input",
input: nil,
wantErr: awserrors.ErrorInvalidParameterValue,
},
{
name: "missing VpcId",
input: &ec2.CreateEgressOnlyInternetGatewayInput{},
wantErr: awserrors.ErrorMissingParameter,
},
{
name: "empty VpcId",
input: &ec2.CreateEgressOnlyInternetGatewayInput{VpcId: aws.String("")},
wantErr: awserrors.ErrorMissingParameter,
},
{
name: "valid input",
input: &ec2.CreateEgressOnlyInternetGatewayInput{VpcId: aws.String("vpc-123")},
wantErr: "",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateCreateEgressOnlyInternetGatewayInput(tt.input)
if tt.wantErr == "" {
assert.NoError(t, err)
} else {
assert.EqualError(t, err, tt.wantErr)
}
})
}
}
func TestValidateDeleteEgressOnlyInternetGatewayInput(t *testing.T) {
tests := []struct {
name string
input *ec2.DeleteEgressOnlyInternetGatewayInput
wantErr string
}{
{
name: "nil input",
input: nil,
wantErr: awserrors.ErrorInvalidParameterValue,
},
{
name: "missing ID",
input: &ec2.DeleteEgressOnlyInternetGatewayInput{},
wantErr: awserrors.ErrorMissingParameter,
},
{
name: "empty ID",
input: &ec2.DeleteEgressOnlyInternetGatewayInput{EgressOnlyInternetGatewayId: aws.String("")},
wantErr: awserrors.ErrorMissingParameter,
},
{
name: "valid input",
input: &ec2.DeleteEgressOnlyInternetGatewayInput{EgressOnlyInternetGatewayId: aws.String("eigw-abc123")},
wantErr: "",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateDeleteEgressOnlyInternetGatewayInput(tt.input)
if tt.wantErr == "" {
assert.NoError(t, err)
} else {
assert.EqualError(t, err, tt.wantErr)
}
})
}
}
func TestValidateDescribeEgressOnlyInternetGatewaysInput(t *testing.T) {
tests := []struct {
name string
input *ec2.DescribeEgressOnlyInternetGatewaysInput
wantErr string
}{
{
name: "nil input",
input: nil,
wantErr: awserrors.ErrorInvalidParameterValue,
},
{
name: "empty input",
input: &ec2.DescribeEgressOnlyInternetGatewaysInput{},
wantErr: "",
},
{
name: "with filter IDs",
input: &ec2.DescribeEgressOnlyInternetGatewaysInput{
EgressOnlyInternetGatewayIds: []*string{aws.String("eigw-abc")},
},
wantErr: "",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateDescribeEgressOnlyInternetGatewaysInput(tt.input)
if tt.wantErr == "" {
assert.NoError(t, err)
} else {
assert.EqualError(t, err, tt.wantErr)
}
})
}
}
// Handler tests — call handlers directly to cover validation + NATS error paths
func TestCreateEgressOnlyInternetGateway_ValidationErrors(t *testing.T) {
_, err := CreateEgressOnlyInternetGateway(nil, nil, "")
assert.EqualError(t, err, awserrors.ErrorInvalidParameterValue)
_, err = CreateEgressOnlyInternetGateway(&ec2.CreateEgressOnlyInternetGatewayInput{}, nil, "")
assert.EqualError(t, err, awserrors.ErrorMissingParameter)
}
func TestCreateEgressOnlyInternetGateway_NilNATS(t *testing.T) {
_, err := CreateEgressOnlyInternetGateway(&ec2.CreateEgressOnlyInternetGatewayInput{
VpcId: aws.String("vpc-123"),
}, nil, "acct-123")
assert.Error(t, err)
}
func TestDeleteEgressOnlyInternetGateway_ValidationErrors(t *testing.T) {
_, err := DeleteEgressOnlyInternetGateway(nil, nil, "")
assert.EqualError(t, err, awserrors.ErrorInvalidParameterValue)
_, err = DeleteEgressOnlyInternetGateway(&ec2.DeleteEgressOnlyInternetGatewayInput{}, nil, "")
assert.EqualError(t, err, awserrors.ErrorMissingParameter)
}
func TestDeleteEgressOnlyInternetGateway_NilNATS(t *testing.T) {
_, err := DeleteEgressOnlyInternetGateway(&ec2.DeleteEgressOnlyInternetGatewayInput{
EgressOnlyInternetGatewayId: aws.String("eigw-abc123"),
}, nil, "acct-123")
assert.Error(t, err)
}
func TestDescribeEgressOnlyInternetGateways_ValidationErrors(t *testing.T) {
_, err := DescribeEgressOnlyInternetGateways(nil, nil, "")
assert.EqualError(t, err, awserrors.ErrorInvalidParameterValue)
}
func TestDescribeEgressOnlyInternetGateways_NilNATS(t *testing.T) {
_, err := DescribeEgressOnlyInternetGateways(&ec2.DescribeEgressOnlyInternetGatewaysInput{}, nil, "acct-123")
assert.Error(t, err)
}
@@ -0,0 +1,33 @@
package gateway_ec2_igw
import (
"errors"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/mulgadc/hive/hive/awserrors"
handlers_ec2_igw "github.com/mulgadc/hive/hive/handlers/ec2/igw"
"github.com/nats-io/nats.go"
)
// AttachInternetGateway handles the EC2 AttachInternetGateway API call
func AttachInternetGateway(input *ec2.AttachInternetGatewayInput, natsConn *nats.Conn, accountID string) (ec2.AttachInternetGatewayOutput, error) {
var output ec2.AttachInternetGatewayOutput
if input == nil {
return output, errors.New(awserrors.ErrorInvalidParameterValue)
}
if input.InternetGatewayId == nil || *input.InternetGatewayId == "" {
return output, errors.New(awserrors.ErrorMissingParameter)
}
if input.VpcId == nil || *input.VpcId == "" {
return output, errors.New(awserrors.ErrorMissingParameter)
}
svc := handlers_ec2_igw.NewNATSIGWService(natsConn)
result, err := svc.AttachInternetGateway(input, accountID)
if err != nil {
return output, err
}
return *result, nil
}
@@ -0,0 +1,27 @@
package gateway_ec2_igw
import (
"errors"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/mulgadc/hive/hive/awserrors"
handlers_ec2_igw "github.com/mulgadc/hive/hive/handlers/ec2/igw"
"github.com/nats-io/nats.go"
)
// CreateInternetGateway handles the EC2 CreateInternetGateway API call
func CreateInternetGateway(input *ec2.CreateInternetGatewayInput, natsConn *nats.Conn, accountID string) (ec2.CreateInternetGatewayOutput, error) {
var output ec2.CreateInternetGatewayOutput
if input == nil {
return output, errors.New(awserrors.ErrorInvalidParameterValue)
}
svc := handlers_ec2_igw.NewNATSIGWService(natsConn)
result, err := svc.CreateInternetGateway(input, accountID)
if err != nil {
return output, err
}
return *result, nil
}
@@ -0,0 +1,30 @@
package gateway_ec2_igw
import (
"errors"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/mulgadc/hive/hive/awserrors"
handlers_ec2_igw "github.com/mulgadc/hive/hive/handlers/ec2/igw"
"github.com/nats-io/nats.go"
)
// DeleteInternetGateway handles the EC2 DeleteInternetGateway API call
func DeleteInternetGateway(input *ec2.DeleteInternetGatewayInput, natsConn *nats.Conn, accountID string) (ec2.DeleteInternetGatewayOutput, error) {
var output ec2.DeleteInternetGatewayOutput
if input == nil {
return output, errors.New(awserrors.ErrorInvalidParameterValue)
}
if input.InternetGatewayId == nil || *input.InternetGatewayId == "" {
return output, errors.New(awserrors.ErrorMissingParameter)
}
svc := handlers_ec2_igw.NewNATSIGWService(natsConn)
result, err := svc.DeleteInternetGateway(input, accountID)
if err != nil {
return output, err
}
return *result, nil
}
@@ -0,0 +1,27 @@
package gateway_ec2_igw
import (
"errors"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/mulgadc/hive/hive/awserrors"
handlers_ec2_igw "github.com/mulgadc/hive/hive/handlers/ec2/igw"
"github.com/nats-io/nats.go"
)
// DescribeInternetGateways handles the EC2 DescribeInternetGateways API call
func DescribeInternetGateways(input *ec2.DescribeInternetGatewaysInput, natsConn *nats.Conn, accountID string) (ec2.DescribeInternetGatewaysOutput, error) {
var output ec2.DescribeInternetGatewaysOutput
if input == nil {
return output, errors.New(awserrors.ErrorInvalidParameterValue)
}
svc := handlers_ec2_igw.NewNATSIGWService(natsConn)
result, err := svc.DescribeInternetGateways(input, accountID)
if err != nil {
return output, err
}
return *result, nil
}
@@ -0,0 +1,33 @@
package gateway_ec2_igw
import (
"errors"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/mulgadc/hive/hive/awserrors"
handlers_ec2_igw "github.com/mulgadc/hive/hive/handlers/ec2/igw"
"github.com/nats-io/nats.go"
)
// DetachInternetGateway handles the EC2 DetachInternetGateway API call
func DetachInternetGateway(input *ec2.DetachInternetGatewayInput, natsConn *nats.Conn, accountID string) (ec2.DetachInternetGatewayOutput, error) {
var output ec2.DetachInternetGatewayOutput
if input == nil {
return output, errors.New(awserrors.ErrorInvalidParameterValue)
}
if input.InternetGatewayId == nil || *input.InternetGatewayId == "" {
return output, errors.New(awserrors.ErrorMissingParameter)
}
if input.VpcId == nil || *input.VpcId == "" {
return output, errors.New(awserrors.ErrorMissingParameter)
}
svc := handlers_ec2_igw.NewNATSIGWService(natsConn)
result, err := svc.DetachInternetGateway(input, accountID)
if err != nil {
return output, err
}
return *result, nil
}

Some files were not shown because too many files have changed in this diff Show More