* feat: add ALB target health reporting via HAProxy stats Agent-side: periodically query HAProxy stats socket and publish target health status to NATS (elbv2.alb.<id>.health topic). Control-plane: healthChecker subscribes to health reports from all ALB agents and updates target health state in the KV store. Wire into ELBv2ServiceImpl lifecycle and DeregisterTargets cleanup. E2E: remove health check skip — targets must now become healthy. * feat: allocate public IP for internet-facing ALB VMs in daemon Add Scheme field to SystemInstanceInput so the daemon knows whether an ALB is internet-facing or internal. For internet-facing, allocate a public IP via externalIPAM and publish vpc.add-nat for the OVN dnat_and_snat rule. Internal ALBs skip public IP assignment entirely. * fix: release auto-assigned public IPs on DeleteNetworkInterface DeleteNetworkInterface previously leaked public IPs and NAT rules. Now checks if the ENI has a public IP, verifies it's not EIP-owned, and releases via externalIPAM + publishes vpc.delete-nat. Daemon injects externalIPAM and EIP KV into VPC service at init time. Tests cover: auto-assigned IP release with NAT event, EIP-owned IP skipped, and no-op when externalIPAM is not configured. * feat: enforce multi-AZ for internet-facing ALBs in pool mode (mulga-770) In CreateLoadBalancer, when DevNetworking is false (pool mode), require 2+ subnets in different AZs for internet-facing scheme. Dev mode and internal scheme skip the validation. * feat: ALB scheme unit tests + DNS internal prefix + public IP in AZ response (mulga-771) Add 9 unit tests covering scheme-dependent ALB behavior: - DNS name uses internal- prefix for internal scheme (AWS-compatible) - Internet-facing ALBs include public IP in AvailabilityZones response - Internal ALBs have no public IP in AZ response - Launcher receives correct scheme, delete cleans up VM + ENIs Also implements the code changes these tests exercise: - AvailZoneInfo.PublicIP field populated from launcher output - lbRecordToSDK populates LoadBalancerAddresses for internet-facing * feat: add custom AMI for ALB to images list * feat: add internet-facing ALB data plane E2E test (mulga-772) New E2E script that validates internet-facing ALBs get public IPs and are reachable from a peer cluster node (external validation). Skips in dev mode. * fix: set ENDPOINT to runner node IP in ALB E2E workflow * fix: correct paths filter to reference alb-e2e.yml * fix: alb ami name and workflow step to import the image * fix: spx path * fix: remove ALB AMI import from workflow and added to bootstrap.sh * fix: use two subnets in different AZs for internet-facing ALB E2E Multi-AZ enforcement (mulga-770) requires internet-facing ALBs to have subnets in 2+ AZs in pool mode. Discover AZs and create a subnet in each. * refactor: remove multi-AZ enforcement for internet-facing ALBs Team decision: ALBs must work with single-AZ and single-node clusters. Removes the multi-AZ validation, pool mode helper, associated tests, and error constant. E2E script reverted to single subnet. * fix: use correct account ID for ALB ENI public IP assignment The ENI is created under the caller's account but LaunchSystemInstance was looking it up with GlobalAccountID (system account), causing the UpdateENIPublicIP call to fail with "ENI not found". Pass the caller's account ID via SystemInstanceInput.AccountID. * fix: add debug logging for ALB public IP assignment failure * fix: expand debug logging to include peer node and more keywords * fix: lazy system AMI discovery for ALB VMs The daemon was caching the system AMI at startup, but the ALB image is imported after services start (bootstrap step 9). Replace SetSystemAMI with SetSystemAMIFunc so the AMI is resolved at request time when CreateLoadBalancer actually needs it. * fix: resolve ALB VM instance type dynamically from resource manager The instance type was hardcoded to t3.nano which may not exist on all clusters. Now the daemon provides a resolver that picks the smallest available instance type from the resource manager. * push interim changes * refactor: rewrite ALB agent from NATS client to HTTP server Replace NATS subscriptions with an HTTP server on :8405 exposing POST /config, GET /ping, and GET /health. The daemon will push config and poll health over HTTP through the VPC network, removing the agent's direct NATS dependency. - agent.go: net/http server replaces nats.Conn/Subscriptions - health.go: remove push-based reportHealth; /health handler calls queryHAProxyStats synchronously on each request - cmd/alb-agent: --nats flag replaced with --listen - tests: rewritten with httptest instead of NATS test server - health_checker_test: inline topic strings (HealthTopic removed) * refactor: replace NATS with HTTP client for daemon-to-ALB agent comms Daemon now pushes config via POST and polls ping via GET to the ALB agent's HTTP server on the VPC IP, eliminating the agent's NATS dependency from the daemon side. - service_impl: pushConfig uses http.Post, waitForAgentReady uses http.Get, both target http://<vpcIP>:8405 - albVMUserData: drop natsURL param (agent no longer needs NATS) - types: add VPCIP field to LoadBalancerRecord - daemon: remove SetNATSURL call and 10.0.2.2 translation block - arn_test: update user data tests for new signature * refactor: rewrite ALB health checker from NATS subscription to HTTP polling Health checker now polls each active ALB's GET /health endpoint on a 10s ticker instead of subscribing to a NATS wildcard topic. Removes the NATS connection dependency from the health checker. - health_checker: replace nc/sub with httpClient + stopCh polling loop - newHealthChecker: drop nc param, add agentURLFn for testability - service_impl: update newHealthChecker call site - tests: rewrite with httptest, test pollAll/pollOne paths * refactor: remove NATS wiring from ALB image and make hostfwd dev-only - alb-setup.sh: remove --nats flag, use --listen=:8405; remove placeholder HAProxy frontend on 8405 (agent owns the port now); update description and comments - service_impl: HostfwdPorts (80/443) now only set when DevNetworking is enabled — production ALBs are reached via VPC IP directly * feat: system credentials file + daemon wiring for ALB agent SigV4 auth Write config/system-credentials.json during admin init/join with the system access key. Daemon reads it at startup, resolves the gateway URL from external IPAM (or 10.0.2.2 in dev mode), and wires both into the ELBv2 service. albVMUserData now injects ALB_GATEWAY_URL, ALB_ACCESS_KEY, and ALB_SECRET_KEY into cloud-init for ALB VMs. * feat: ALB agent gateway actions + daemon handlers (heartbeat, config) Add ALBAgentHeartbeat and GetALBConfig actions to the gateway so the ALB agent can communicate via the AWS gateway instead of direct HTTP. - Add agent_types.go with input/output types for both actions - Add ALBAgentHeartbeat: processes heartbeat, transitions provisioning→active on first contact, updates LastHeartbeat, processes health report - Add GetALBConfig: returns pre-computed HAProxy config text and hash - Replace pushConfig (HTTP POST to agent) with updateStoredConfig (stores config + SHA256 hash on LB record for agent pull) - Remove waitForAgentReady polling goroutine (heartbeat handles transition) - Remove HTTP client from ELBv2ServiceImpl (no longer needed) - Strip health checker HTTP polling (pollLoop/pollAll/pollOne) — health reports now arrive directly via ALBAgentHeartbeat - Add ConfigText, ConfigHash, LastHeartbeat fields to LoadBalancerRecord - Wire NATS subscriptions + daemon handlers for both new actions - Add gateway handler files and elbv2Actions map entries - Comprehensive tests for heartbeat, config, and updateStoredConfig * feat: rewrite ALB agent as AWS SDK HTTP client (Bead 3r) Replace the HTTP server agent with a poll-based client that communicates with the gateway using SigV4-signed requests. The agent now sends ALBAgentHeartbeat every 5s with health reports and fetches config via GetALBConfig when the config hash changes. * feat: add VPC service route for ALB agent gateway access ALB VMs need to reach the gateway for heartbeat/config polling, but VPCs without an IGW have no external routing. Instead of requiring an IGW, the daemon now publishes a vpc.ensure-service-route event before launching ALB VMs. vpcd handles this by creating minimal external connectivity (external switch, localnet port, gateway router port, chassis bindings) if not already present, then adds a /32 static route for the gateway IP. Refactors handleIGWAttach to share the external connectivity setup via ensureExternalConnectivity helper. * Revert "feat: add VPC service route for ALB agent gateway access" This reverts commitd4220d8806. * feat: add management NIC infrastructure for system instance control plane Add config, IP allocator, TAP helpers, and VM struct fields for the management NIC that gives system instances (ALBs etc.) L3 access to the host control plane via br-mgmt, independent of VPC routing. - DaemonConfig.MgmtBridge (default "br-mgmt") - MgmtIPAllocator: thread-safe static IP allocation (.10-.249), rebuild from VMs on restart - generateMgmtMAC (02:a0:00 prefix), MgmtTapName, GetBridgeIPv4 - SetupMgmtTapDevice / CleanupMgmtTapDevice (plain Linux bridge TAP) - VM struct: MgmtMAC, MgmtIP, MgmtTap fields * feat: wire management NIC into QEMU launch and cloud-init config Add QEMU NIC injection for mgmt TAP (virtio-net-pci on br-mgmt), extend generateNetworkConfig to emit mgmt0 stanza with static IP, and wire mgmt MAC/IP allocation into LaunchSystemInstance. Includes mgmt TAP cleanup on instance stop and failed launch cleanup. * test: add coverage for ELBv2 service lifecycle and setter methods * fix: move br-mgmt detection before wireALBAgentConfig so gateway URL uses management bridge IP * fix: generate per-interface cloud-init config when eniMAC present without devMAC The guard in generateNetworkConfig required both eniMAC and devMAC to produce per-interface config, falling back to wildcard DHCP otherwise. System instances (ALBs) have eniMAC + mgmtMAC but no devMAC (DevNetworking is independent), so the mgmt NIC stanza was never emitted. The wildcard config does DHCP on all NICs, but br-mgmt has no DHCP server, leaving the management NIC unconfigured and the agent unable to reach the gateway. * fix: add wildcard support to ENI description filter + add ca-certificates to ALB image - ENI DescribeNetworkInterfaces description filter did exact string match, but AWS supports * wildcards. E2E test uses "ELB *" which didn't match "ELB app/my-alb/lb-123". Added matchFilter helper for wildcard matching. - ALB image was missing ca-certificates package, so cloud-init ca_certs module couldn't add the Spinifex CA to the system trust store. Agent TLS handshakes failed with "certificate signed by unknown authority". - Fixed data race in TestDeleteLoadBalancer_TerminatesVM_WithPublicIP where async TerminateSystemInstance goroutine wrote to mock fields while the test read them. * fix: detach ENI before delete on termination + defer ENI cleanup to stopInstance DeleteLoadBalancer was trying to delete ENIs synchronously, but they were still attached (in-use) to the running VM. The delete was rejected with InvalidNetworkInterface.InUse. Fix: DeleteLoadBalancer only deletes ENIs when no VM was launched (InstanceID empty). When a VM exists, ENI cleanup is deferred to stopInstance which runs after QEMU exits. stopInstance now calls DetachENI before DeleteNetworkInterface to clear the in-use status. E2E test now polls for ENI cleanup (up to 30s) since VM termination is async. * fix: detach then delete ENIs in DeleteLoadBalancer ENIs are attached (in-use) to the ALB VM, so DeleteNetworkInterface rejects them. Fix: call DetachENI first to clear in-use status, then delete. stopInstance tolerates already-deleted ENIs. * test: add internal ALB dataplane E2E test Add run-elbv2-dataplane-internal-e2e.sh replacing the generic dataplane test. Internet-facing coverage is already provided by run-elbv2-dataplane-internet-facing-e2e.sh. The internal test validates: - ALB reaches active via mgmt NIC heartbeat (no IGW dependency) - Scheme=internal, no public IP, internal- DNS prefix - Health checks pass via VPC L2 (OVN logical switch) - Round-robin traffic from a VPC client to the ALB's private IP - ENI cleanup on ALB deletion Uses a client VM inside the VPC that curls the ALB private IP and serves results via HTTP on its public IP for host-side verification. * feat: add NLB constants and data model changes (mulga-838) * feat: type-aware validation in CreateLoadBalancer (mulga-839) Accept Type=network for NLB, reject SecurityGroups for NLB, set CrossZoneEnabled defaults (false for NLB, true for ALB), and use /net/ ARN path segment for network load balancers. * feat: type-aware validation in CreateTargetGroup (mulga-840) Accept TCP/UDP/TLS/TCP_UDP protocols and apply NLB health check defaults (TCP protocol, no path/matcher, 10s timeout, 3/3 thresholds). Reject invalid protocols with InvalidParameterValue. * feat: type-aware validation in CreateListener (mulga-841) * feat: add NLB listener and protocol compatibility unit tests Cover the remaining Phase 1 test gaps: NLB CreateListener with all L4 protocols, protocol rejection for wrong LB type, listener-to-target-group protocol compatibility validation, and default protocol rejection on NLB. * feat: add TCP HAProxy config generation for NLB Add TCP mode template and generateNLBConfig function for Network Load Balancer support. GenerateHAProxyConfig now branches on LB type, producing mode-tcp configs for NLBs with tcp-check or httpchk health checks depending on the target group's health check protocol. * refactor: rename albagent to lbagent for NLB generalization Rename the ALB-specific agent package and binary to generic LB names in preparation for NLB support. Both ALB and NLB VMs use the same agent — it just applies whatever HAProxy config the control plane generates. - Rename albagent/ → lbagent/, cmd/alb-agent/ → cmd/lb-agent/ - Rename types: ALBAgentHeartbeat* → LBAgentHeartbeat*, GetALBConfig* → GetLBConfig* - Rename NATS topics: elbv2.ALBAgentHeartbeat → elbv2.LBAgentHeartbeat - Rename cloud-init user data: albVMUserData → lbVMUserData with LB_* env vars - Rename Makefile targets: build-alb-agent → build-lb-agent - Update gateway action map, daemon subscriptions, service interface - Fix pre-existing unused matchFilter function lint error - Fix awserrors test count for InvalidConfigurationRequest * feat: add sys.micro system instance type for LB VMs (mulga-845) Add sys.micro (1 vCPU, 128MB) as an internal-only instance type for system VMs (load balancers, NAT gateways). Registered in the type map for allocation but excluded from DescribeInstanceTypes, NATS subscriptions, and resource stats. Daemon SetSystemInstanceTypeFunc now returns sys.micro instead of scanning for smallest guest type. * feat: LB image shrink and rename (mulga-846) Rename alb.conf → lb.conf, alb-setup.sh → lb-setup.sh with IMAGE_SIZE reduced from 512M to 192M. Update all references to use lb-agent binary and LB_* env vars. Update COMMANDS.md for NLB support. * chore: rename file * feat: NLB E2E test and bug fixes (mulga-847) Add full NLB data plane E2E test (33 assertions, 9 phases): VPC/subnet setup, TCP target group with health checks, NLB creation (type=network), TCP listener, agent heartbeat, target health verification, TCP traffic round-robin, target deregistration, and full cleanup verification. Bug fixes found during E2E testing: - Fix HAProxy stats socket path mismatch: config used alb-/nlb- prefix but agent expects lb- prefix, preventing health reports - Fix system AMI lookup to match new -lb- image name, falling back to legacy -alb- name - Bump LB image size from 192M to 256M (Alpine base is 200M) * test: validate NLB internal scheme (mulga-857) NLB with scheme=internal works out of the box — shared code paths with ALB handle it correctly. Add unit test and E2E coverage: - Unit test: NLB internal creates with no public IP, internal- DNS prefix - E2E phases 10-14: internal NLB lifecycle with VPC client TCP traffic - E2E passed 56/56 with perfect round-robin (20/20 requests) * feat: consolidate LB e2e tests into run-lb-e2e.sh (mulga-854) Replace 3 separate LB dataplane scripts with a single consolidated test that covers all 4 variants (ALB/NLB × internet-facing/internal). - Share one VPC, subnet, and 2 dual-purpose app instances across all suites - ALB + NLB provision in parallel within each phase (internet-facing / internal) - One client VM tests both internal LBs simultaneously - Replace sleep 100 with LB health check gates - Delete standalone alb-e2e.yml, integrate into e2e.yml for both single and multi-node * chore: disable LB E2E tests in CI until dev env issue resolved * fix: read LB agent credentials from spinifex.toml instead of system-credentials.json The daemon couldn't read system-credentials.json due to systemd hardening (file owned by root:root 0600, not in ReadOnlyPaths). Use the same system credentials already loaded from spinifex.toml via the predastore config section — these are identical keys written by admin init. * feat: automate LB image build and import (mulga-861) - dev-install.sh: build lb-agent and run build-system-image.sh --import after services start so the LB image is available out of the box - build-system-image.sh: add --quiet flag to suppress build output, remove hardcoded --config path (uses auto-detected default) - images.go: rename alb-alpine → lb-alpine to match current naming * update go version * add LB e2e tests * fix: clean LB image build dir and wait for nbd device readiness Previous CI runs can leave stale state in /tmp/lb-image-build/ causing e2fsck failures. Clean the build dir before each CI build. Also add a readiness check after qemu-nbd connect to avoid race conditions. * fix: serialize nbd access with flock for concurrent CI builds Single-node and multi-node CI jobs share the same host and both build the LB system image using /dev/nbd0. Use flock to serialize the nbd connect/mount/customize/disconnect section so concurrent builds wait rather than collide. * fix: move flock to cover entire image build, not just nbd section Concurrent CI jobs share BUILD_DIR and /dev/nbd0. The lock must be acquired before downloading/copying the image, not just before nbd connect, otherwise the second job can clobber the first job's files. * fix: skip LB image build if output is fresh from concurrent CI job When concurrent CI jobs build the same image, the second job now reuses the first's output if it's less than 10 minutes old, avoiding a duplicate download + build cycle. * fix: filter AMI discovery by name in E2E tests With the LB Alpine image now imported alongside Ubuntu, the unfiltered describe-images call picked up the wrong AMI. Filter by the expected image name so tests use the Ubuntu image for instance lifecycle. * fix: filter AMI discovery by name in multi-node E2E tests Same fix as run-e2e.sh — with the LB Alpine image now imported, the unfiltered describe-images picks the wrong AMI. * fix: use awsgw bind IP as LB agent gateway URL fallback When br-mgmt is not configured and dev_networking is off (production with external pool), the gateway URL was falling through to the external IPAM pool's GatewayIP — the WAN router, not the Spinifex node. LB VMs couldn't reach the awsgw, so heartbeats never arrived and ALBs stayed stuck in provisioning. Use the awsgw bind IP instead — LB VMs on the OVN overlay can reach the host node's WAN IP where the gateway listens. * fix: capture StateReason and daemon logs when LB app instances fail Instances with user-data terminate immediately in CI. Add describe-instances StateReason query and daemon log dump to diagnose whether it's volume_preparation_failed or launch_failed. * fix: prevent orphaned AMI from breaking LB E2E and avoid stuck provisioning Three fixes for CI E2E failures: 1. run-e2e.sh: Remove custom AMI from predastore after deleting its backing snapshot. Previously the AMI remained discoverable but unusable, causing run-lb-e2e.sh to pick it and fail with NoSuchKey on the deleted snapshot. 2. run-lb-e2e.sh: Prefer ubuntu AMI by name for app instances instead of picking any non-alpine image. Also improve debug log dump to use journalctl (binary-install CI uses systemd, not log files). 3. CreateLoadBalancer: Only set provisioning state when a system VM was actually launched. Previously a failed launch still set provisioning, leaving the LB stuck forever with no agent to send heartbeats. * fix: use AWSGW bind IP as LB agent gateway URL in multi-node In multi-node deployments, the AWSGW binds to the WAN IP (e.g. 192.168.1.26:9999), but wireLBAgentConfig was using the br-mgmt IP as the gateway URL. Since the AWSGW doesn't listen on br-mgmt, the LB agent's heartbeat was refused, leaving internet-facing ALBs stuck in provisioning. Internet-facing LBs now use the AWSGW bind IP directly — the VM reaches it via VPC/external networking. Internal LBs get a host route on the mgmt NIC so they can reach the AWSGW bind IP through br-mgmt. * fix: add mgmt route via bootcmd for multi-node internal LBs Alpine cloud-init does not support v2 routes under ethernets — adding a routes: block to the network-config breaks all networking in the VM. Instead, add the host route via a bootcmd in the LB user data. This runs early in cloud-init and adds `ip route add {awsgw}/32 via {br-mgmt}` so internal LB agents can reach the AWSGW through the management NIC when the gateway binds to a specific WAN IP. Also moves the networking service from boot to default runlevel in the LB Alpine image to prevent a race with cloud-init. Verified locally: internal ALB transitions to active with AWSGW bound to WAN IP (simulating multi-node). * add LB enable flag to e2e * test: add elbv2 tests to meet diff coverage threshold Cover health checker edge cases (invalid JSON, empty servers, zero thresholds, port-0 fallback) and service_impl paths (mgmt route cloud-init, protocol validation, missing target group). * fix: swallowed health report error + needless marshal round-trip in LBAgentHeartbeat Add handleHealthReportDirect(report) to healthChecker so LBAgentHeartbeat can pass the struct directly instead of marshaling to JSON and immediately unmarshaling. Promote handleHealthReport unmarshal error from Debug to Warn so failures are visible in logs. * fix: validate credentials before launching LB VM + set failed state on launch error (mulga-867) Guard lbVMUserData with empty-value check for gatewayURL, systemAccessKey, and systemSecretKey — blank env vars previously caused agent auth-loop with no daemon-side log. Set LB state to failed (not active) when VM launch fails so the API reflects the broken data-plane instead of black-holing traffic. * fix: data race in getSystemAMI/getSystemInstanceType lazy caching Use sync.Once to guard lazy resolution of systemAMI and systemInstanceType fields. Concurrent CreateLoadBalancer calls previously raced on unsynchronized read+write of these cached values. The setter methods reset the Once so re-resolution works when the func is set after initial construction. Also fix pre-existing nosprintfhostport lint in daemon.go. * fix: use utils.PublishEvent for NAT events + return errors from isEIPOwned publishNATEvent in eni.go and daemon was copy-pasted boilerplate — now delegates to utils.PublishEvent (matching the EIP service pattern). isEIPOwned previously swallowed all KV errors and returned false, which could release an EIP-owned public IP if the store was temporarily unavailable. Now returns (bool, error) so the caller skips release on KV failure instead of assuming no EIP ownership. * fix: narrow health checker to only scan target groups for reporting LB handleHealthReportDirect was calling ListTargetGroups() on every heartbeat (every 5s per LB agent), scanning all TGs. Now uses report.LBID to follow LB → listeners → TG ARNs and fetch only the attached target groups via direct key lookups. Falls back to full scan if LBID is missing. * fix: pass region through to LB agent SigV4 signing (mulga-871) LB agent hardcoded us-east-1 as the SigV4 signing region. Thread region from ELBv2 service config → cloud-init LB_REGION env var → lb-agent --region flag → lbagent.New() → signer.Sign() call. * fix: add account-level authorization to LBAgentHeartbeat and GetLBConfig Both endpoints accepted any accountID without verifying the LB belongs to that account. Now verify accountID matches the LB's AccountID, with an exception for the system account (000000000000) which the ALB agent uses for heartbeats and config fetches. * fix: upgrade silent failure paths from Debug to Warn/Error - healthChecker target group list failure: Debug → Warn - HAProxy stats socket error in agent tick(): Debug → Warn - ENI delete EIP-owned skip: Debug → Info (operational visibility) - wireLBAgentConfig: warn when no gateway host is resolved - detectMgmtIP: warn when bridge exists but IP is empty - stale mgmt TAP cleanup: log errors instead of discarding * refactor: deduplicate MAC hash function into utils.HashMAC Three identical hash-to-MAC implementations (generateDevMAC, generateMgmtMAC, generateENIMac) now delegate to a shared utils.HashMAC(prefix, id) helper. * fix: guard LBAgentHeartbeat KV writes to state transitions or stale timestamps Previously the full LoadBalancerRecord (including ConfigText) was written to JetStream KV on every heartbeat (every 5s). Now only persists on state transitions (provisioning→active) or when LastHeartbeat is older than 60s. * fix: fail ALB launch when public IP allocation fails for internet-facing scheme Previously, IPAM allocation failure was logged as a warning and the ALB VM continued launching without a public IP. The ALB reached active state but was unreachable from the internet. Now the launch returns an error and cleans up the partially-created instance, causing the ALB to enter the failed state. * fix: retry system AMI resolution when image not yet imported sync.Once cached empty results permanently — if the daemon started before the system image was imported, all CreateLoadBalancer calls silently skipped VM creation. Replace sync.Once with a mutex that only caches non-empty results, allowing retry on subsequent calls. Same fix applied to getSystemInstanceType for consistency. * fix: propagate updateStoredConfig errors to API callers (mulga-884) updateStoredConfig and updateStoredConfigForTargetGroup now return errors instead of silently swallowing them. Callers (CreateListener, DeleteListener, RegisterTargets, DeregisterTargets) return ErrorServerInternal when config persistence fails, preventing the API from reporting success while HAProxy serves stale configuration. * test: add unit tests for wireLBAgentConfig, isEIPOwned, toHealthReport (mulga-885) * fix: parse HAProxy templates once at package level (mulga-875) generateALBConfig and generateNLBConfig were calling template.New().Parse() on every invocation. Move to package-level template.Must for one-time parsing. * fix: fail launch for internal-scheme ALBs when mgmt NIC setup fails (mulga-886) When management TAP or IP allocation fails, internal-scheme ALBs now return an error instead of silently continuing without a management NIC. Without the mgmt NIC, the LB agent cannot reach the gateway and the ALB gets stuck in provisioning forever. Internet-facing ALBs retain the existing degrade-and-continue behavior. * refactor: extract lbArnPathSegment helper for ALB/NLB ARN construction (mulga-877) The app/net path segment derivation was duplicated in buildLBArn, buildListenerArn, and inline in CreateLoadBalancer. Extracted into a single lbArnPathSegment helper. * refactor: use aws.StringValue in toHealthReport (mulga-878) Replace manual nil-check-then-deref with aws.StringValue for cleaner pointer-to-value conversion in the heartbeat health report builder. * refactor: use testutil.StartTestJetStream in health_checker_test.go (mulga-876) Replace hand-rolled setupTestNATS (~20 lines) with the shared testutil.StartTestJetStream helper. Simplify return to just *Store since no caller needed the *nats.Conn. * fix: address PR #122 review comments (mulga-887) - Extract NATEvent named type from anonymous struct in publishNATEvent - Remove unused build-lb-image.sh thin wrapper - Remove SPX_ENABLE_LB env var from e2e workflow (unused) - Remove no-op healthChecker start/stop methods - Export simple ELBv2 fields, remove trivial setter methods - Delete old e2e scripts replaced by consolidated run-lb-e2e.sh * fix: check specific error type for ENI deletion on termination (mulga-888) ENI deletion errors were all silently downgraded to Debug level, masking genuine failures (KV issues, permissions, ENI stuck in-use) that cause orphaned ENIs and leaked IPAM addresses. Now only NotFound is logged at Debug; all other errors log at Error. * fix: propagate non-ENODEV errors from GetBridgeIPv4 (mulga-890) GetBridgeIPv4 returned ("", nil) for ALL InterfaceByName failures, not just missing devices. Permission errors on /sys/class/net/ were silently treated as "bridge absent". Now only "no such network interface" returns nil; other errors propagate to the caller. * fix: remove backend address from proxy error response (mulga-892) The reverse proxy error handler included the internal backend host:port in the XML error body sent to clients. The address is already logged server-side; the client now sees a generic message. * fix: release mutex before KV store I/O in health checker (mulga-893) handleHealthReportDirect held hc.mu across PutTargetGroup calls (JetStream KV Put — network I/O), serializing concurrent heartbeats from independent LBs. Now collects changed TGs under the lock and persists them after releasing it. * fix: warn when system credentials missing in wireLBAgentConfig (mulga-894) fix: log DetachENI errors instead of discarding with blank identifier (mulga-895) * fix: detect veth bridge mode and use centralized NAT for it The veth-pair bridge topology (Linux bridge ↔ veth ↔ OVS bridge) was misclassified as direct bridge mode, causing OVN to use distributed NAT. Distributed NAT hairpin routing fails through the Linux bridge intermediary, breaking host-to-local-LB connectivity. Add BridgeModeVeth constant, detect veth-wan-ovs interface in detectBridgeMode(), and introduce useCentralizedNAT() to gate NAT mode and nat-addresses=router for both macvlan and veth modes. isMacvlanMode() is preserved for macvlan-specific behavior (MAC alignment). * fix: add retry loop for peer NLB reachability test With centralized NAT, ARP resolution for NAT external IPs relies on gratuitous ARPs from the gateway chassis. These take time to propagate to peer nodes. The peer→NLB test had a single 3s attempt while all other connectivity tests use retry loops. * fix: use request-reply for vpc.add-nat to prevent ARP propagation races publishNATEvent was fire-and-forget for vpc.add-nat, meaning the daemon continued before vpcd committed the OVN NAT rule. This caused intermittent NLB connectivity failures: the LB was reported active before OVN had the DNAT rule and sent GARPs for the public IP. Switch vpc.add-nat to request-reply (RequestEvent) so the daemon waits for OVN confirmation. Also increase nc timeout from 2-3s to 5s in E2E tests for more reliable TCP probing during ARP resolution. * fix: clean up stale NAT rules before adding new ones for reused IPs When a public IP is returned to the external pool and quickly reallocated (e.g. instance terminated, then LB created), the fire-and-forget vpc.delete-nat may not have been processed yet. handleAddNAT now removes any existing dnat_and_snat rule for the same external IP before creating the new one, preventing duplicate/conflicting OVN NAT rules. This was causing intermittent NLB connectivity failures in multi-node E2E where NLB public IPs collided with recently-freed instance IPs. * fix: clean up stale NAT rules across all VPC routers on IP reuse The previous fix (0a93081) only removed stale dnat_and_snat rules from the target VPC's router. When a public IP is freed from one VPC and reallocated to a different VPC before the fire-and-forget vpc.delete-nat is processed, the stale rule on the old VPC's router caused duplicate OVN ARP responses — traffic was black-holed when the host resolved to the wrong router's MAC. DeleteAllNATsByExternalIP now searches every logical router for stale NAT rules matching the external IP and removes them in one transaction. Also adds host connectivity diagnostics (ARP, routing, OVN NAT state) to run-lb-e2e.sh so the next CI failure surfaces the root cause of the persistent NLB-from-host issue. * fix: run LB inet tests from actual hosting node, not hardcoded node1 The host/peer connectivity tests were hardcoded: "host" = node1 (where the test script runs), "peer" = node2. But LB VMs land on whichever node picks up the NATS queue message, so the tests were randomly testing local vs remote connectivity depending on placement luck. Now the test: 1. Accepts --nodes <ip1> <ip2> ... (all cluster node IPs) 2. After LB creation, discovers which node hosts each LB VM by checking for the ENI's tap device 3. Runs "local" tests from the hosting node and "remote" tests from a different node Also updates e2e.yml to pass --nodes instead of --peer.
141 KiB
Command Implementation Matrix
Spinifex Admin CLI
Platform management commands not exposed via the AWS gateway API. These are CLI-only commands.
Service Management
Service lifecycle commands for starting, stopping, and checking status of all Spinifex cluster services. Each service subcommand supports start, stop, and status operations.
| Command | Flags | Prerequisites | Basic Logic | Status |
|---|---|---|---|---|
spx service predastore start |
--port (default: 8443), --host (default: 0.0.0.0), --base-path, --config-path, --debug, --tls-cert, --tls-key, --backend (distributed/filesystem, default: distributed), --node-id (default: 0), --pprof, --pprof-output |
TLS cert/key, base-path, config-path required | Creates predastore service instance with S3-compatible storage backend → starts service | DONE |
spx service predastore stop |
— | Predastore must be running | Stops the predastore service | DONE |
spx service predastore status |
— | None | Reports predastore service status | DONE |
spx service viperblock start |
--s3-host (default: 0.0.0.0:8443), --s3-bucket (default: predastore), --s3-region (default: ap-southeast-2), --plugin-path (default: /opt/spinifex/lib/nbdkit-viperblock-plugin.so) |
--config required, plugin-path must exist on disk |
Loads cluster config → connects to NATS and Predastore → starts viperblock block storage service with NBD plugin | DONE |
spx service viperblock stop |
— | Viperblock must be running | Stops the viperblock service | DONE |
spx service viperblock status |
— | None | Reports viperblock service status | DONE |
spx service nats start |
--port (default: 4222), --host (default: 0.0.0.0), --debug, --data-dir, --jetstream |
None | Starts embedded NATS server with optional JetStream | DONE |
spx service nats stop |
— | NATS must be running | Stops the NATS service | DONE |
spx service nats status |
— | None | Reports NATS service status | DONE |
spx service spinifex start |
--wal-dir |
--config required |
Loads cluster config → starts spinifex daemon (VM orchestration, NATS subscriptions, health endpoint) | DONE |
spx service spinifex stop |
— | Spinifex daemon must be running | Stops the spinifex daemon service | DONE |
spx service spinifex status |
— | None | Reports spinifex daemon service status | DONE |
spx service awsgw start |
--host (default: 0.0.0.0:9999), --tls-cert, --tls-key, --debug |
--config required |
Loads cluster config → starts AWS-compatible gateway with SigV4 auth, IAM policy enforcement, TLS | DONE |
spx service awsgw stop |
— | AWS gateway must be running | Stops the AWS gateway service | DONE |
spx service awsgw status |
— | None | Reports AWS gateway service status | DONE |
spx service spinifex-ui start |
--port (default: 3000), --host (default: 0.0.0.0), --tls-cert, --tls-key |
None | Starts embedded web UI server serving the React frontend. Aliases: ui, spinifexui |
DONE |
spx service spinifex-ui stop |
— | spinifex-ui must be running | Stops the spinifex-ui service | DONE |
spx service spinifex-ui status |
— | None | Reports spinifex-ui service status | DONE |
spx service vpcd start |
— | --config required, OVN/OVS installed |
Loads cluster config → starts VPC daemon (subscribes to vpc.* NATS events, translates to OVN logical switches/ports/routers) |
DONE |
spx service vpcd stop |
— | vpcd must be running | Stops the vpcd service | DONE |
spx service vpcd status |
— | None | Reports vpcd service status | DONE |
Cluster Inspection
Operational commands for inspecting cluster state. These fan out NATS requests to all nodes and aggregate responses.
| Command | Flags | Prerequisites | Basic Logic | Output Columns | Status |
|---|---|---|---|---|---|
spx get nodes |
--timeout (default: 3s) |
Cluster must be running (NATS) | Loads config → publishes to spinifex.node.status fan-out topic → collects responses within timeout → merges config-known nodes with NATS responders → nodes that don't respond shown as NotReady |
NAME, STATUS, IP, REGION, AZ, UPTIME, VMs, SERVICES | DONE |
spx get vms |
--timeout (default: 3s) |
Cluster must be running (NATS) | Publishes to spinifex.node.vms fan-out topic → collects VM info from all nodes → sorts by node then instance ID → prints table. Alias: spx get instances |
INSTANCE, STATUS, TYPE, VCPU, MEM, NODE, IP, AGE | DONE |
Resource Monitoring
| Command | Flags | Prerequisites | Basic Logic | Output | Status |
|---|---|---|---|---|---|
spx top nodes |
--timeout (default: 3s) |
Cluster must be running (NATS) | Publishes to spinifex.node.status fan-out topic → collects CPU/memory usage per node → aggregates instance type capacity across all nodes → prints two tables: per-node resource usage and cluster-wide instance type availability |
Table 1: NAME, CPU (used/total), MEM (used/total), VMs. Table 2: INSTANCE TYPE, AVAILABLE, VCPU, MEMORY | DONE |
Cluster Initialization
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|
spx admin init |
--nodes, --node, --bind, --port, --region, --az, --cluster-name, --cluster-bind, --cluster-routes, --predastore-nodes, --services, --formation-timeout, --force |
None (first-time setup) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI spx profile → creates directory structure under ~/spinifex/ |
1. Init creates all config files 2. Root credentials printed once 3. master.key is 32 bytes, mode 0600 4. bootstrap.json consumed on first start 5. --force re-initializes existing config6. AWS CLI profile spx auto-configured |
DONE |
spx admin join |
--host (required), --node (required), --bind, --port, --region, --az, --cluster-bind, --cluster-routes, --data-dir, --services |
Leader node must be running | Connects to leader node → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations | 1. Join existing cluster 2. Missing host (error) 3. Missing node name (error) |
DONE |
Version
| Command | Flags | Prerequisites | Basic Logic | Status |
|---|---|---|---|---|
spx version |
— | None | Prints Spinifex version, commit hash, OS, and architecture (populated via build-time ldflags) | DONE |
Cluster Operations
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|
spx admin cluster shutdown |
--force (shutdown even if nodes don't respond), --timeout (max wait per phase, default 120s), --dry-run (print phase plan without executing) |
Cluster must be running | Performs coordinated, phased shutdown of entire cluster. Phases execute in order: GATE (stop API/UI) → DRAIN (stop VMs) → STORAGE (stop viperblock) → PERSIST (stop predastore) → INFRA (stop NATS/daemon). Each phase waits for all nodes to ACK before proceeding. Uses JetStream state tracking. | 1. Shutdown running cluster 2. All nodes stop cleanly 3. Force shutdown with unresponsive nodes 4. Dry-run prints plan |
DONE |
Certificate Management
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|
spx admin cert renew |
--extra-ip (additional IPs for SANs), --extra-dns (additional DNS names for SANs) |
Existing CA from spx admin init |
Reads existing CA → regenerates server certificate with all current network interface IPs and machine hostname in SANs → writes new cert. Use after adding a new network interface or changing IP addresses. | 1. Renew with auto-detected IPs 2. Renew with extra IPs 3. Renew with extra DNS names |
DONE |
Account Management
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|
spx admin account create |
--name |
Cluster must be running (NATS), master key must exist | Connects to NATS → CAS loop on spinifex-account-counter:next_id for sequential 12-digit ID → creates Account record in spinifex-accounts KV → creates admin user in new account → creates access key for admin → creates AdministratorAccess policy (Action:, Resource:) → attaches policy → prints credentials |
1. Create first account (ID 000000000001) 2. Sequential IDs (000000000002, etc.) 3. Admin user has full access 4. Credentials printed once |
DONE |
spx admin account list |
— | Cluster must be running (NATS), master key must exist | Connects to NATS → IAMService.ListAccounts() → prints table with Account ID, Name, Status, Created | 1. List all accounts 2. Empty list (only global account) |
DONE |
Image Management
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|
spx admin images import |
--name |
Cluster must be running, image file must exist locally | Imports a local disk image (QCOW2) as an AMI → uploads to Predastore S3 → registers as available AMI | 1. Import valid QCOW2 image 2. Image appears in describe-images |
DONE |
spx admin images list |
— | None | Lists available OS images that can be imported or downloaded | 1. List available images | DONE |
AWS Commands
EC2 - Instance Management
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
run-instances |
--image-id, --instance-type, --count (Min/MaxCount), --key-name, --user-data, --subnet-id (auto-creates ENI, assigns private IP), --block-device-mappings (DeviceName, VolumeSize, VolumeType, Iops, DeleteOnTermination), --placement (GroupName only — routes via spread or cluster strategy) |
--security-group-ids, --tag-specifications, --dry-run, --client-token, --disable-api-termination, --ebs-optimized, --iam-instance-profile, --network-interfaces, --private-ip-address, --monitoring, --credit-specification, --cpu-options, --metadata-options, --launch-template, --hibernate-options |
describe-images (AMI must exist), create-key-pair (optional), VPC/SG (optional) |
Gateway parses AWS query → if Placement.GroupName set, looks up strategy: spread → distributeInstancesSpread() (1 instance per node, atomic CAS reservation), cluster → distributeInstancesCluster() (pin all to single node); otherwise NATS ec2.runinstances → daemon creates QEMU/KVM VM with viperblock-backed root volume via NBD → if SubnetId provided, auto-creates ENI with private IP → cloud-init injects user-data/keys → on termination, removes instance from placement group → returns reservation with instance ID |
1. Launch with valid AMI and key pair 2. Launch with invalid AMI ID (error) 3. Launch with block device mappings (custom volume size) 4. Launch multiple instances (MinCount/MaxCount) 5. Launch with subnet-id (auto-creates ENI) 6. Invalid instance type returns error 7. Launch with spread placement group (1 per node) 8. Launch with cluster placement group (all on one node) 9. Insufficient capacity for placement group (error) |
DONE |
describe-instances |
--instance-ids, --filters (instance-state-name, instance-id, instance-type, vpc-id, subnet-id, tag:*, tag-key, tag-value) |
--max-results, --next-token, --dry-run |
None | Gateway fans out NATS ec2.DescribeInstances to all nodes (no queue group) → each daemon returns local instances → gateway aggregates and returns combined list. Filters applied per-node before aggregation (reduces payload). Also applies to stopped/terminated instances via describeInstancesFromKV(). |
1. Describe all instances (no filter) 2. Describe by instance ID 3. Describe with filters (e.g. instance-state-name) 4. Instance not found returns empty set 5. Multi-node aggregation returns instances from all nodes 6. Filter by tag 7. Unknown filter returns InvalidParameterValue |
DONE |
start-instances |
--instance-ids |
--dry-run, --force |
run-instances (instance must exist in stopped state) |
Gateway sends NATS ec2.cmd.{instance-id} → daemon restarts stopped QEMU process with same config → state transitions stopped→pending→running |
1. Start a stopped instance 2. Start already-running instance (error: IncorrectInstanceState) 3. Start with invalid instance ID 4. Verify volumes re-mount on start |
DONE |
stop-instances |
--instance-ids |
--force, --hibernate, --dry-run |
run-instances (instance must be running) |
Gateway sends NATS to target node → daemon issues QMP system_powerdown for graceful shutdown → monitors heartbeat until QEMU exits → state transitions running→stopping→stopped |
1. Graceful stop of running instance 2. Force stop (kills QEMU process) 3. Stop already-stopped instance (error) 4. Verify ~30s heartbeat detection |
DONE |
terminate-instances |
--instance-ids, DeleteOnTermination (per-volume flag, default true) |
--dry-run |
run-instances (instance must exist) |
Gateway sends NATS to target node → daemon kills QEMU process → cleans up NBD mounts → deletes volumes with DeleteOnTermination=true via volumeService.DeleteVolume() (S3 cleanup of vol/, vol-efi/, vol-cloudinit/) → internal volumes (EFI, cloud-init) always cleaned up via ebs.delete NATS → volumes with DeleteOnTermination=false left in available state → state→terminated |
1. Terminate running instance 2. Terminate stopped instance 3. Terminate with DeleteOnTermination=true deletes volumes 4. Terminate with DeleteOnTermination=false preserves volumes 5. Terminate already-terminated (idempotent) 6. Internal volumes (EFI, cloud-init) always cleaned up 7. Invalid instance ID |
DONE |
reboot-instances |
--instance-ids |
--dry-run |
run-instances (instance must be running) |
Gateway validates instance IDs → sends EC2InstanceCommand with RebootInstance=true via NATS ec2.cmd.{instanceId} → daemon validates instance is in StateRunning (returns IncorrectInstanceState if stopped) → issues QMP system_reset → instance reboots without stopping, stays in running state |
1. Reboot running instance 2. Reboot multiple instances 3. Reboot stopped instance (error: IncorrectInstanceState) 4. Instance not found (error: InvalidInstanceID.NotFound) 5. Verify instance stays in running state after reboot |
DONE |
describe-instance-types |
--filters (capacity filter only) |
--instance-types, --max-results, --next-token, --dry-run, all other filters |
None | Gateway fans out NATS ec2.DescribeInstanceTypes to all nodes → each daemon reports supported types (t3.micro/small/medium/large) with vCPU/memory specs → gateway deduplicates and returns |
1. List all instance types 2. Filter by specific type 3. Filter with capacity=true shows available slots4. Verify vCPU/memory specs match hardware |
DONE |
modify-instance-attribute |
--instance-id, --instance-type, --user-data |
--disable-api-termination, --ebs-optimized, --source-dest-check, --instance-initiated-shutdown-behavior, --block-device-mappings, --groups, --ena-support, --sriov-net-support |
Instance must be stopped (in NATS KV) | Gateway validates input (exactly one attribute per call, instance ID format) → NATS ec2.ModifyInstanceAttribute with spinifex-workers queue group → daemon loads stopped instance from JetStream KV → applies attribute change → writes back to KV → returns {} on success. InstanceType: updates vm.InstanceType, Config, and Instance fields; clears StateReason (enables recovery from instance-type-missing bug). UserData: stores decoded content in vm.UserData and re-encodes to base64 for RunInstancesInput (cloud-init on next start). No instance type pre-validation (matches AWS — invalid types accepted, fail at StartInstances time). |
1. Change instance type while stopped 2. Change user data while stopped 3. Modify running instance (error: NotFound — running instances not in KV) 4. Instance not found (error: InvalidInstanceID.NotFound) 5. Instance not stopped (error: IncorrectInstanceState) 6. Invalid instance type accepted (fails on start with InsufficientInstanceCapacity) 7. StateReason cleared on type change (recovery from capacity-unavailable) 8. Missing/malformed instance ID (error: InvalidInstanceID.Malformed) 9. No attribute set (error: InvalidParameterValue) 10. Multiple attributes in one call (error: InvalidParameterValue) |
DONE |
get-console-output |
--instance-id |
--latest (always returns latest), --dry-run |
Instance must be running on a node | Gateway sends NATS ec2.{instanceId}.GetConsoleOutput (per-instance topic, routed to owning node) → daemon reads console log file from disk → returns last 64KB base64-encoded with timestamp. Always available regardless of serial console access setting (matches AWS behavior). |
1. Get output from running instance 2. Empty log file returns empty output 3. Instance not found (error: InvalidInstanceID.NotFound) |
DONE |
describe-instance-attribute |
--instance-id, --attribute (instanceType, userData, instanceInitiatedShutdownBehavior, disableApiTermination, disableApiStop, ebsOptimized, enaSupport, sourceDestCheck, rootDeviceName, kernel, ramdisk) |
--dry-run |
Instance must exist (running or stopped) | Gateway validates input → NATS ec2.DescribeInstanceAttribute with spinifex-workers queue group → daemon checks running instances first (d.Instances.VMS), then stopped instances in JetStream KV → returns single attribute per call (matches AWS behavior). Stored attributes (instanceType, userData) return real values; unstored attributes return AWS defaults (instanceInitiatedShutdownBehavior=stop, disableApiTermination=false, etc.) |
1. Get instanceType from running instance 2. Get userData from stopped instance 3. Get default disableApiTermination 4. Invalid attribute name (error) 5. Instance not found (error: InvalidInstanceID.NotFound) |
DONE |
describe-instance-credit-specifications |
--instance-ids |
--filters, --max-results, --dry-run |
None | Gateway-only stub — returns CpuCredits: "standard" for each requested instance ID. No daemon round-trip. T-series credit mode is not persisted. |
1. Get credit spec for T-series instance 2. Multiple instance IDs |
DONE |
monitor-instances |
— | --instance-ids |
Instance must exist | Enable basic monitoring (CPU, disk, network) → store metrics in NATS KV → return monitoring state | 1. Enable monitoring 2. Verify monitoring state in describe-instances |
NOT STARTED |
unmonitor-instances |
— | --instance-ids |
Instance must exist | Disable detailed monitoring → revert to basic monitoring → return monitoring state | 1. Disable monitoring 2. Verify monitoring state reverts in describe-instances |
NOT STARTED |
EC2 - Key Pair Management
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-key-pair |
--key-name, --key-type (rsa/ed25519) |
--key-format (pem/ppk), --tag-specifications, --dry-run |
None | NATS ec2.CreateKeyPair → daemon generates SSH keypair → stores public key in Predastore S3 (/bucket/ec2/{name}.pub) → returns private key material (one-time) |
1. Create RSA key pair 2. Create ED25519 key pair 3. Duplicate key name (error: InvalidKeyPair.Duplicate) 4. Verify key material returned only on creation |
DONE |
describe-key-pairs |
--key-names, --key-pair-ids, --filters (key-pair-id, key-name, fingerprint, tag:*) |
--max-results, --dry-run |
None | NATS ec2.DescribeKeyPairs → daemon lists public keys from Predastore S3 → applies filters → returns key names and fingerprints |
1. List all key pairs 2. Filter by key name 3. Filter by key pair ID 4. Non-existent key returns empty 5. Unknown filter returns InvalidParameterValue |
DONE |
delete-key-pair |
--key-name, --key-pair-id |
--dry-run |
Key must exist | NATS ec2.DeleteKeyPair → daemon deletes public key from Predastore S3 → returns success |
1. Delete existing key pair 2. Delete non-existent key (idempotent, no error) 3. Verify key no longer in describe-key-pairs |
DONE |
import-key-pair |
--key-name, --public-key-material |
--tag-specifications, --dry-run |
None | NATS ec2.ImportKeyPair → daemon stores provided public key in Predastore S3 → returns key name and fingerprint |
1. Import valid SSH public key 2. Import invalid key material (error) 3. Import duplicate name (error) 4. Verify imported key usable with run-instances |
DONE |
EC2 - AMI Image Management
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
describe-images |
--image-ids (format validation only), --owners (self, account ID, alias), --filters (name, state, architecture, image-id, is-public, owner-id, description, image-type, tag:*) |
--executable-users, --include-deprecated, --include-disabled, --max-results, --next-token, --dry-run |
None | NATS ec2.DescribeImages → daemon reads AMI metadata from Predastore S3 buckets (ami-*) → filters by ImageIds, Owners, and Filters → returns image list with state, architecture, block device mappings |
1. List all images 2. Filter by image ID 3. Filter by owner (self/amazon) 4. Non-existent AMI returns empty 5. Verify metadata fields (architecture, state, rootDeviceName) 6. Filter by name wildcard 7. Unknown filter returns InvalidParameterValue |
DONE |
create-image |
--instance-id, --name, --description, --tag-specifications |
--no-reboot, --block-device-mappings, --dry-run |
Instance must exist (running or stopped) | Gateway validates input → NATS ec2.{instanceId}.CreateImage (per-instance topic) → daemon extracts root volume → snapshots via ebs.snapshot NATS (running) or offline S3 copy (stopped) → creates AMI metadata in S3 ({amiId}/config.json) → stores tags → returns ami-ID. Duplicate AMI name validation enforced. |
1. Create image from running instance 2. Create image from stopped instance 3. Invalid instance ID (error) 4. Duplicate AMI name (error) 5. Verify new AMI appears in describe-images 6. Launch new instance from created AMI |
DONE |
register-image |
— | --name, --description, --architecture, --root-device-name, --virtualization-type, --block-device-mappings, --boot-mode, --ena-support, --sriov-net-support, --tpm-support, --imds-support |
Image data must exist in S3 | Create AMI metadata entry in Predastore → assign ami-ID → set state to available → return ami-ID | 1. Register with valid metadata 2. Missing required name (error) 3. Verify registered image in describe-images |
NOT STARTED |
deregister-image |
— | --image-id, --dry-run |
AMI must exist, no instances using it | Mark AMI metadata as deregistered in S3 → AMI no longer available for new launches → existing instances unaffected | 1. Deregister existing AMI 2. Deregister non-existent AMI (error) 3. Verify deregistered AMI not in describe-images 4. Existing instances from AMI still run |
NOT STARTED |
copy-image |
— | --source-image-id, --source-region, --name, --description, --encrypted, --kms-key-id, --client-token, --copy-image-tags, --tag-specifications |
Source AMI must exist | Copy AMI data between regions/nodes in Predastore → create new AMI metadata with new ami-ID → return new ami-ID | 1. Copy image within same region 2. Copy non-existent image (error) 3. Verify copied image independent of source |
NOT STARTED |
import-image |
— | --disk-containers (Format, Url/S3Bucket+S3Key), --description, --architecture, --platform |
S3 bucket with disk image | Download disk image from S3 → convert format (VMDK/VHD/RAW→QCOW2) → create viperblock volume → register as AMI | 1. Import QCOW2 image 2. Import RAW image 3. Invalid format (error) 4. Verify imported image launchable |
NOT STARTED |
describe-image-attribute |
— | --image-id, --attribute, --dry-run |
AMI must exist | Read specific attribute from AMI metadata in S3 → return attribute value | 1. Get description attribute 2. Get launch permission 3. Invalid attribute name (error) |
NOT STARTED |
modify-image-attribute |
— | --image-id, --launch-permission, --description, --operation-type, --user-ids, --user-groups, --organization-arns, --dry-run |
AMI must exist | Update specific attribute in AMI metadata → persist to S3 | 1. Add launch permission 2. Modify description 3. Invalid AMI (error) |
NOT STARTED |
reset-image-attribute |
— | --image-id, --attribute, --dry-run |
AMI must exist | Reset attribute to default value in AMI metadata | 1. Reset launch permission to owner-only 2. Invalid AMI (error) |
NOT STARTED |
EC2 - Volume (EBS) Management
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
describe-volumes |
--volume-ids (fast-path lookup), DeleteOnTermination (from persisted VolumeMetadata), --filters (volume-id, status, size, volume-type, attachment.instance-id, attachment.status, attachment.device, availability-zone, tag:*) |
--max-results, --next-token, --dry-run |
None | NATS ec2.DescribeVolumes → daemon queries viperblock for volume metadata → applies filters → returns volume list with state, size, attachments, type, DeleteOnTermination flag |
1. List all volumes 2. Filter by volume ID 3. Filter by attachment state 4. Non-existent volume returns empty 5. DeleteOnTermination reflects persisted value 6. Filter by status, size, volume-type 7. Unknown filter returns InvalidParameterValue |
DONE |
modify-volume |
--volume-id, --size, --volume-type, --iops |
--throughput, --dry-run, --multi-attach-enabled |
Volume must exist | NATS ec2.ModifyVolume → daemon sends resize request to viperblock → NBD does not support live resize, instance must be stopped → returns modification state |
1. Increase volume size 2. Modify volume type 3. Decrease size (error - not supported) 4. Modify attached volume (requires stop/start) |
DONE |
create-volume |
--size, --availability-zone, --volume-type (gp3 only), --snapshot-id (creates volume from snapshot) |
--iops (hardcoded 3000), --encrypted (hardcoded false), --throughput, --tag-specifications |
Valid AZ configured via spinifex init |
Gateway validates input → NATS ec2.CreateVolume → daemon generates vol-ID via viperblock → creates volume (empty or from snapshot) of specified size → persists config.json to Predastore S3 → returns vol-ID with state=available |
1. Create 80GB gp3 volume 2. Boundary sizes (1 GiB min, 16384 GiB max) 3. Invalid AZ (error) 4. Verify volume in describe-volumes 5. Unsupported volume type (error - only gp3) 6. Size out of range (error) 7. Create from snapshot |
DONE |
delete-volume |
--volume-id |
--dry-run |
Volume must exist and be detached (state=available) | Gateway validates vol- prefix → NATS ec2.DeleteVolume → daemon confirms state=available and no AttachedInstance → NATS ebs.delete to viperblockd (stops nbdkit/WAL) → deletes S3 objects under vol-id/, vol-id-efi/, vol-id-cloudinit/ → returns success |
1. Delete detached volume 2. Delete attached volume (error: VolumeInUse) 3. Delete non-existent volume (error: InvalidVolume.NotFound) 4. Verify volume gone from describe-volumes 5. Malformed volume ID (error: InvalidVolumeID.Malformed) 6. Double delete (idempotent NotFound) |
DONE |
attach-volume |
--volume-id, --instance-id, --device (optional, auto-assigns /dev/sd[f-p]) |
--dry-run |
Volume must exist (available), instance must exist (running) | Gateway sends to ec2.cmd.{instanceId} → daemon validates volume (Predastore) → ebs.mount via NATS (viperblock starts NBD server) → QMP blockdev-add (nbd-{volId}) → QMP device_add (virtio-blk-pci, vdisk-{volId}) → three-phase rollback on failure → update EBSRequests + BlockDeviceMappings → persist to JetStream + Predastore → respond with VolumeAttachment |
1. Attach volume to running instance 2. Auto-assign device name 3. Attach already-attached volume (VolumeInUse) 4. Attach to non-existent instance (InvalidInstanceID.NotFound) 5. Attach to stopped instance (IncorrectInstanceState) 6. Volume not found (InvalidVolume.NotFound) 7. All device slots full (AttachmentLimitExceeded) 8. Volume persists across stop/start |
DONE |
detach-volume |
--volume-id, --instance-id (optional, resolved via DescribeVolumes), --device (optional cross-check), --force |
--dry-run |
Volume must be attached, instance must be running | Gateway resolves InstanceId if omitted (via DescribeVolumes) → sends to ec2.cmd.{instanceId} → daemon validates (running, attached, not boot/EFI/CloudInit, device match) → three-phase hot-unplug: QMP device_del (force continues on failure) → QMP blockdev-del (abort if fails, preserves state to prevent double-attach) → ebs.unmount via NATS (best-effort) → remove from EBSRequests + BlockDeviceMappings → update volume metadata to available → persist state → respond with VolumeAttachment (state=detaching) |
1. Detach with explicit InstanceId 2. Detach without InstanceId (gateway resolution) 3. Detach with correct --device cross-check 4. Missing VolumeId (InvalidParameterValue) 5. Volume not attached (IncorrectState) 6. Nonexistent volume (InvalidVolume.NotFound) 7. Nonexistent instance (InvalidInstanceID.NotFound) 8. Instance not running (IncorrectInstanceState) 9. Device mismatch (InvalidParameterValue) 10. Boot volume protection (OperationNotPermitted) 11. Force flag (continues past device_del failure) 12. Volume reusability (re-attach after detach) |
DONE |
describe-volume-status |
--volume-ids, --filters (volume-id, volume-status.status, availability-zone) |
--max-results, --next-token, --dry-run |
None | Gateway validates vol- prefix → NATS ec2.DescribeVolumeStatus → daemon fetches VolumeConfig from Predastore S3 (parallel for specific IDs, sequential list-all for no IDs) → applies filters → builds VolumeStatusItem per volume (status=ok, io-enabled=passed, io-performance=not-applicable) → returns InvalidVolume.NotFound for missing explicit IDs → skips internal sub-volumes (-efi, -cloudinit) |
1. List all volume statuses 2. Filter by specific volume IDs (fast path) 3. Non-existent volume ID returns InvalidVolume.NotFound 4. Invalid volume ID format (InvalidVolume.Malformed) 5. Internal sub-volumes excluded from listing 6. Nil/empty input defaults to all volumes 7. Unknown filter returns InvalidParameterValue |
DONE |
describe-volumes-modifications |
— | --volume-ids, --filters, --max-results |
None | Query pending/completed volume modifications → return modification state, progress, original/target size | 1. Check in-progress modification 2. Check completed modification 3. No modifications returns empty |
NOT STARTED |
EC2 - Snapshot Management
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-snapshot |
--volume-id, --description, --tag-specifications |
--dry-run |
Volume must exist in Predastore | Gateway validates vol- prefix → NATS ec2.CreateSnapshot → daemon reads VolumeConfig from Predastore → generates snap-ID via viperblock → stores SnapshotConfig (metadata-only, points to source volume) as completed → returns Snapshot with state=completed, progress=100% |
1. Snapshot from valid volume 2. Missing volume ID (InvalidParameterValue) 3. Volume not found (InvalidVolume.NotFound) 4. Invalid volume ID format (InvalidVolume.Malformed) 5. Snapshot with description 6. Snapshot with tag specifications |
DONE |
create-snapshots |
— | --instance-specification, --description, --tag-specifications |
Instance must exist, instance-volume attachment tracking | Create snapshots of all volumes attached to instance → return list of snapshot IDs. Blocked: requires instance-volume attachment tracking to discover which volumes to snapshot. | 1. Snapshot all volumes on instance 2. Instance with no volumes |
NOT STARTED |
delete-snapshot |
--snapshot-id |
--dry-run |
Snapshot must exist | Gateway validates snap- prefix → NATS ec2.DeleteSnapshot → daemon verifies snapshot exists in Predastore → lists and deletes all objects under snapshot prefix → returns success |
1. Delete existing snapshot 2. Delete non-existent snapshot (InvalidSnapshot.NotFound) 3. Missing snapshot ID (InvalidParameterValue) 4. Invalid snapshot ID format (InvalidSnapshot.Malformed) |
DONE |
describe-snapshots |
--snapshot-ids, --filters (snapshot-id, status, volume-id, volume-size, owner-id, tag:*) |
--owner-ids, --max-results, --dry-run |
None | Gateway validates snap- prefix on IDs → NATS ec2.DescribeSnapshots → daemon lists snap- prefixed objects in Predastore → reads SnapshotConfig for each → applies filters → returns snapshot list |
1. List all snapshots 2. Filter by snapshot ID 3. Empty snapshot list 4. Invalid snapshot ID format (InvalidSnapshot.Malformed) 5. Filter by status, volume-id, volume-size 6. Unknown filter returns InvalidParameterValue |
DONE |
copy-snapshot |
--source-snapshot-id, --source-region, --description |
--encrypted, --dry-run |
Source snapshot must exist | Gateway validates snap- prefix + source region → NATS ec2.CopySnapshot → daemon reads source SnapshotConfig → generates new snap-ID → copies metadata (preserves tags, description override) → stores as completed → returns new snapshot ID |
1. Copy within same region 2. Copy non-existent snapshot (InvalidSnapshot.NotFound) 3. Missing source ID (InvalidParameterValue) 4. Missing source region (MissingParameter) 5. Copy preserves tags 6. Copy with description override |
DONE |
EC2 - Tags
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-tags |
--resources, --tags (Key=,Value=) |
--dry-run |
None | Gateway validates input (resources non-empty, tags non-empty, keys non-empty) → NATS ec2.CreateTags → daemon merges tags into per-resource JSON in Predastore S3 (tags/{resourceId}.json) → return success |
1. Tag an instance 2. Tag multiple resources at once 3. Overwrite existing tag value 4. Empty resources list (error: MissingParameter) 5. Empty tags list (error: MissingParameter) 6. Empty tag key (error: InvalidParameterValue) |
DONE |
delete-tags |
--resources, --tags |
--dry-run |
None | Gateway validates input (resources non-empty) → NATS ec2.DeleteTags → daemon removes specified tag keys from resource JSON, or all tags if no keys specified → return success |
1. Delete specific tag by key 2. Delete all tags (no tags specified) 3. Empty resources list (error: MissingParameter) |
DONE |
describe-tags |
--filters (resource-id, resource-type, key, value) |
--max-results, --next-token, --dry-run |
None | NATS ec2.DescribeTags → daemon lists all tags/*.json from Predastore S3 → applies filters (resource-id, resource-type, key, value) → returns tag list with resource type auto-detected from ID prefix |
1. List all tags 2. Filter by resource-id 3. Filter by resource-type (instance, volume, image, snapshot, vpc, subnet, security-group, route-table, internet-gateway, egress-only-internet-gateway) 4. Filter by key 5. Filter by value 6. Empty result when no tags exist |
DONE |
EC2 - Regions & Availability Zones
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
describe-regions |
(returns configured region, endpoint, opt-in status from config — input params ignored) | --region-names, --filters, --all-regions, --dry-run |
None | Return configured region from spinifex init config with Endpoint and OptInStatus → local-only response, no NATS | 1. List all regions 2. Filter by region name 3. Verify endpoint URL returned 4. Verify OptInStatus returned |
DONE |
describe-availability-zones |
(returns configured AZ, region, zone ID, state, opt-in status from config — input params ignored) | --zone-names, --filters, --all-availability-zones |
None | Return configured AZ from spinifex init config with zone ID, state, group name, network border group → local-only response, no NATS | 1. List all AZs 2. Filter by zone name 3. Verify zone state is available 4. Verify region name matches config |
DONE |
EC2 - Account Attributes
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
describe-account-attributes |
--attribute-names |
--dry-run |
None | Gateway parses input → returns static account attributes: supported-platforms=VPC, default-vpc=none, max-instances=100, vpc-max-security-groups-per-interface=5, max-elastic-ips=5, vpc-max-elastic-ips=20 → local-only response, no NATS | 1. List all account attributes 2. Filter by attribute name 3. Verify all 6 attributes returned with correct values |
DONE |
EC2 - Account Settings (Deferred)
Persistence layer works (NATS JetStream KV), but stored values are not yet enforced by downstream services. Will be completed post-merge.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
enable-ebs-encryption-by-default |
(no flags needed) | --dry-run |
None | Gateway validates input → NATS ec2.EnableEbsEncryptionByDefault with spinifex-workers queue group → daemon stores EbsEncryptionByDefault=true in JetStream KV (spinifex-ec2-account-settings bucket) → return EbsEncryptionByDefault=true. Enforcement deferred: stored value not yet checked by CreateVolume/CreateSnapshot. |
1. Enable encryption 2. Verify via get |
STARTED (enforcement pending) |
disable-ebs-encryption-by-default |
(no flags needed) | --dry-run |
None | Gateway validates input → NATS ec2.DisableEbsEncryptionByDefault with spinifex-workers queue group → daemon stores EbsEncryptionByDefault=false in JetStream KV → return EbsEncryptionByDefault=false. |
1. Disable encryption 2. Verify via get |
STARTED (enforcement pending) |
get-ebs-encryption-by-default |
(no flags needed) | --dry-run |
None | Gateway validates input → NATS ec2.GetEbsEncryptionByDefault with spinifex-workers queue group → daemon reads EbsEncryptionByDefault from JetStream KV → return current state. |
1. Get default state 2. Verify matches last enable/disable |
STARTED (enforcement pending) |
enable-serial-console-access |
(no flags needed) | --dry-run |
None | Gateway validates input → NATS ec2.EnableSerialConsoleAccess with spinifex-workers queue group → daemon stores SerialConsoleAccessEnabled=true in JetStream KV (spinifex-ec2-account-settings bucket) → return enabled=true. Stored for future interactive serial console support (not currently enforced by any operation). |
1. Enable access 2. Verify via get |
STARTED (enforcement pending) |
disable-serial-console-access |
(no flags needed) | --dry-run |
None | Gateway validates input → NATS ec2.DisableSerialConsoleAccess with spinifex-workers queue group → daemon stores SerialConsoleAccessEnabled=false in JetStream KV → return enabled=false. Stored for future interactive serial console support (not currently enforced by any operation). |
1. Disable access 2. Verify via get |
STARTED (enforcement pending) |
get-serial-console-access-status |
(no flags needed) | --dry-run |
None | Gateway validates input → NATS ec2.GetSerialConsoleAccessStatus with spinifex-workers queue group → daemon reads SerialConsoleAccessEnabled from JetStream KV → return current state. |
1. Get current status 2. Verify matches last enable/disable |
DONE |
enable-snapshot-block-public-access |
— | --state |
None | Store snapshot block public access state in JetStream KV | 1. Enable with block-all-sharing 2. Enable with block-new-sharing |
NOT STARTED (enforcement pending) |
disable-snapshot-block-public-access |
— | --dry-run |
None | Clear snapshot block public access state in JetStream KV | 1. Disable access | NOT STARTED (enforcement pending) |
get-snapshot-block-public-access-state |
— | --dry-run |
None | Read snapshot block public access state from JetStream KV | 1. Get current state | NOT STARTED (enforcement pending) |
enable-image-block-public-access |
— | --image-block-public-access-state |
None | Store image block public access state in JetStream KV | 1. Enable with block-new-sharing | NOT STARTED (enforcement pending) |
disable-image-block-public-access |
— | --dry-run |
None | Clear image block public access state in JetStream KV | 1. Disable access | NOT STARTED (enforcement pending) |
get-image-block-public-access-state |
— | --dry-run |
None | Read image block public access state from JetStream KV | 1. Get current state | NOT STARTED (enforcement pending) |
modify-instance-metadata-defaults |
— | --http-tokens, --http-put-response-hop-limit, --http-endpoint, --instance-metadata-tags |
None | Store instance metadata defaults in JetStream KV | 1. Set httpTokens=required 2. Set hop limit |
NOT STARTED (enforcement pending) |
get-instance-metadata-defaults |
— | --dry-run |
None | Read instance metadata defaults from JetStream KV | 1. Get current defaults | NOT STARTED (enforcement pending) |
EC2 - Egress-Only Internet Gateway
Gateway routing and KV CRUD are wired up but no OVN/OVS integration exists. EIGWs are stored in spinifex-eigw KV bucket but have no effect on network topology (no vpcd events published, unlike IGW which publishes vpc.igw-attach/vpc.igw-detach).
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-egress-only-internet-gateway |
--vpc-id, --tag-specifications |
--client-token, --dry-run |
VPC must exist | Gateway validates VpcId → NATS ec2.CreateEgressOnlyInternetGateway → daemon generates eigw-ID → creates EgressOnlyIGWRecord in spinifex-eigw KV bucket with state=attached (always attached at creation, unlike IGW) → returns EgressOnlyInternetGateway with attachment |
1. Create with valid VpcId 2. Missing VpcId (error: MissingParameter) 3. Create with inline tags 4. Verify ID format eigw-{16hex} |
STARTED (KV only, no OVN) |
delete-egress-only-internet-gateway |
--egress-only-internet-gateway-id |
--dry-run |
EIGW must exist | NATS ec2.DeleteEgressOnlyInternetGateway → daemon verifies EIGW exists in KV → deletes from spinifex-eigw KV → returns ReturnCode=true. No detach required (unlike IGW). |
1. Delete existing EIGW 2. Delete non-existent EIGW (error) 3. Missing ID (error: MissingParameter) |
STARTED (KV only, no OVN) |
describe-egress-only-internet-gateways |
--egress-only-internet-gateway-ids, --filters (egress-only-internet-gateway-id, tag:*) |
--max-results, --next-token, --dry-run |
None | NATS ec2.DescribeEgressOnlyInternetGateways → daemon lists all keys from spinifex-eigw KV bucket → filters by ID if specified → applies filters → always includes Attachments array → returns list |
1. Describe all EIGWs 2. Filter by specific ID 3. Non-existent ID returns empty list 4. Unknown filter returns InvalidParameterValue |
STARTED (KV only, no OVN) |
EC2 - Placement Groups
Placement groups are stored in spinifex-placement-groups JetStream KV bucket, keyed by {accountID}.{groupName}. All mutations use CAS (compare-and-swap) with 5-retry limit for optimistic concurrency. Supported strategies: spread (strict 1 instance per node) and cluster (all instances pinned to single node). Partition strategy is rejected. Internal NATS operations (ReserveSpreadNodes, FinalizeSpreadInstances, ReleaseSpreadNodes, RemoveInstanceFromPlacementGroup, ReserveClusterNode, FinalizeClusterInstances) support atomic multi-node coordination for RunInstances and TerminateInstances.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-placement-group |
--group-name, --strategy (spread/cluster) |
--partition-count, --spread-level (hardcoded to "host"), --tag-specifications, --dry-run |
None | Gateway validates GroupName + Strategy → NATS ec2.CreatePlacementGroup with spinifex-workers queue group → daemon generates pg-ID (pg-{16hex}) → stores PlacementGroupRecord in spinifex-placement-groups KV with state=available, SpreadLevel="host" for spread → rejects partition strategy (InvalidParameterValue) → rejects duplicate names (InvalidPlacementGroup.Duplicate) → returns PlacementGroup |
1. Create with spread strategy 2. Create with cluster strategy 3. Create with partition strategy (error: InvalidParameterValue) 4. Duplicate name (error: InvalidPlacementGroup.Duplicate) 5. Missing GroupName (error) 6. Missing Strategy (error) |
DONE |
delete-placement-group |
--group-name |
--dry-run |
Group must exist, no instances in group | NATS ec2.DeletePlacementGroup → daemon verifies group exists (InvalidPlacementGroup.Unknown) → counts instances in NodeInstances map → rejects if non-empty (InvalidPlacementGroup.InUse) → deletes from KV → returns success |
1. Delete empty group 2. Delete group with instances (error: InvalidPlacementGroup.InUse) 3. Delete non-existent group (error: InvalidPlacementGroup.Unknown) |
DONE |
describe-placement-groups |
--group-names, --group-ids, --filters (strategy, state, spread-level, group-name) |
--dry-run |
None | NATS ec2.DescribePlacementGroups → daemon lists all keys from spinifex-placement-groups KV → filters by GroupNames, GroupIds, or Filters → returns list with Strategy, State, SpreadLevel, GroupId |
1. List all groups 2. Filter by name 3. Filter by group ID 4. Filter by strategy 5. Filter by state 6. Empty result when no groups exist |
DONE |
EC2 - Dedicated Hosts
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
allocate-hosts |
— | --availability-zone, --instance-type, --quantity, --auto-placement, --tag-specifications |
None | NATS ec2.AllocateHosts → daemon allocates host resources → stores in JetStream KV with h-ID → return host ID list |
1. Allocate single host 2. Allocate multiple hosts 3. Missing AZ or instance type (error) |
NOT STARTED |
describe-hosts |
— | --host-ids, --filters, --max-results |
None | NATS ec2.DescribeHosts → daemon lists hosts from KV → return host list with capacity and instance info |
1. List all hosts 2. Filter by host ID 3. Filter by instance type |
NOT STARTED |
release-hosts |
— | --host-ids |
Host must exist, no running instances | NATS ec2.ReleaseHosts → daemon verifies no instances on host → delete from KV → return success/failure per host |
1. Release empty host 2. Release host with instances (error) 3. Release non-existent host (error) |
NOT STARTED |
EC2 - IPv4 Pools
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-public-ipv4-pool |
— | --tag-specifications, --dry-run |
None | NATS ec2.CreatePublicIpv4Pool → daemon creates pool in JetStream KV with ipv4pool-ID → return pool ID |
1. Create pool 2. Verify in describe |
NOT STARTED |
delete-public-ipv4-pool |
— | --pool-id, --dry-run |
Pool must exist and be empty | NATS ec2.DeletePublicIpv4Pool → daemon verifies pool empty → delete from KV → return success |
1. Delete empty pool 2. Delete pool with addresses (error) |
NOT STARTED |
describe-public-ipv4-pools |
— | --pool-ids, --filters, --max-results |
None | NATS ec2.DescribePublicIpv4Pools → daemon lists pools from KV → return pool list with address counts |
1. List all pools 2. Filter by pool ID |
NOT STARTED |
EC2 - DHCP Options
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-dhcp-options |
— | --dhcp-configurations (domain-name, domain-name-servers, ntp-servers, netbios-name-servers, netbios-node-type), --tag-specifications |
None | NATS ec2.CreateDhcpOptions → daemon stores DHCP option set in JetStream KV with dopt-ID → return DHCP options with configurations |
1. Create with domain-name + DNS servers 2. Create with NTP servers 3. Verify in describe |
NOT STARTED |
delete-dhcp-options |
— | --dhcp-options-id, --dry-run |
DHCP options must exist and not be associated with a VPC | NATS ec2.DeleteDhcpOptions → daemon verifies not associated → delete from KV → return success |
1. Delete unassociated options 2. Delete associated options (error) 3. Delete non-existent (error) |
NOT STARTED |
describe-dhcp-options |
— | --dhcp-options-ids, --filters, --max-results |
None | NATS ec2.DescribeDhcpOptions → daemon lists DHCP options from KV → return list with configurations |
1. List all DHCP options 2. Filter by ID |
NOT STARTED |
associate-dhcp-options |
— | --dhcp-options-id, --vpc-id, --dry-run |
DHCP options and VPC must exist | NATS ec2.AssociateDhcpOptions → daemon links DHCP options to VPC → update VPC metadata → return success |
1. Associate with VPC 2. Missing DHCP options ID (error) 3. Missing VPC ID (error) |
NOT STARTED |
EC2 - Capacity Reservations
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-capacity-reservation |
— | --instance-type, --instance-platform, --availability-zone, --instance-count, --end-date, --end-date-type, --instance-match-criteria, --tag-specifications |
None | NATS ec2.CreateCapacityReservation → daemon stores reservation in JetStream KV with cr-ID → return reservation with state=active |
1. Create with required fields 2. Missing instance type (error) 3. Verify in describe |
NOT STARTED |
cancel-capacity-reservation |
— | --capacity-reservation-id, --dry-run |
Reservation must exist | NATS ec2.CancelCapacityReservation → daemon marks reservation as cancelled in KV → return success |
1. Cancel active reservation 2. Cancel non-existent (error) |
NOT STARTED |
describe-capacity-reservations |
— | --capacity-reservation-ids, --filters, --max-results |
None | NATS ec2.DescribeCapacityReservations → daemon lists reservations from KV → return list with state and counts |
1. List all reservations 2. Filter by ID 3. Filter by instance type |
NOT STARTED |
modify-capacity-reservation |
— | --capacity-reservation-id, --instance-count, --end-date, --end-date-type |
Reservation must exist | NATS ec2.ModifyCapacityReservation → daemon updates reservation in KV → return success |
1. Modify instance count 2. Modify end date 3. Missing reservation ID (error) |
NOT STARTED |
EC2 - Elastic IP
EIP operations are conditionally available — only registered when external IPAM is configured (pool mode). EIPs are stored in spinifex-eip-allocations KV bucket. Association publishes vpc.add-nat/vpc.delete-nat events to vpcd for OVN NAT configuration.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
allocate-address |
--public-ipv4-pool (optional, specific IPAM pool), --tag-specifications |
--domain (hardcoded vpc), --dry-run |
EIP service must be initialized (external IPAM configured) | Gateway validates input → NATS ec2.AllocateAddress with spinifex-workers queue group → daemon allocates IP from external IPAM pool (named or best-matching) → creates EIPRecord with state=allocated in spinifex-eip-allocations KV → returns AllocationId, PublicIp, Domain=vpc, PublicIpv4Pool |
1. Allocate EIP from default pool 2. Allocate from specific pool 3. Pool exhausted (InsufficientAddressCapacity) 4. With tag specifications 5. Verify in describe-addresses |
DONE |
release-address |
--allocation-id |
--dry-run |
EIP must exist and be disassociated | Gateway validates AllocationId non-empty → NATS ec2.ReleaseAddress → daemon retrieves EIP record from KV → checks state != associated (InvalidAddressLocked if still associated) → releases IP back to IPAM pool → deletes KV record → returns success |
1. Release disassociated EIP 2. Release associated EIP (error: InvalidAddressLocked) 3. Release non-existent (error: InvalidAllocationID.NotFound) 4. Missing AllocationId (error: MissingParameter) |
DONE |
associate-address |
--allocation-id, --network-interface-id OR --instance-id, --private-ip-address (optional override) |
--dry-run, --allow-reassociation |
EIP and target ENI/instance must exist | Gateway validates AllocationId + one of NetworkInterfaceId/InstanceId required → NATS ec2.AssociateAddress → daemon retrieves EIP → resolves target ENI (by ID or instance lookup) → generates AssociationId → updates record state=associated with ENI/instance/VPC/private IP → publishes vpc.add-nat event to vpcd (OVN NAT configuration) → returns AssociationId |
1. Associate with instance 2. Associate with specific ENI 3. Associate with private IP override 4. Missing AllocationId (MissingParameter) 5. Missing target (MissingParameter) 6. Non-existent EIP (InvalidAllocationID.NotFound) 7. Non-existent ENI (InvalidNetworkInterfaceID.NotFound) |
DONE |
disassociate-address |
--association-id |
--dry-run |
Association must exist | Gateway validates AssociationId non-empty → NATS ec2.DisassociateAddress → daemon scans KV to find EIP by AssociationId → publishes vpc.delete-nat event to vpcd → clears association fields → reverts state to allocated → updates KV → returns success |
1. Disassociate existing 2. Non-existent association (InvalidAssociationID.NotFound) 3. Missing AssociationId (MissingParameter) |
DONE |
describe-addresses |
--allocation-ids, --public-ips, --filters (allocation-id, public-ip, instance-id, association-id, domain, tag:*) |
--dry-run |
None | NATS ec2.DescribeAddresses → daemon scans all EIP records in account → filters by AllocationIds and/or PublicIps if specified → applies filters → returns Address list with allocation, association, PublicIp, PrivateIpAddress, NetworkInterfaceId, InstanceId, Domain |
1. List all EIPs 2. Filter by allocation ID 3. Filter by public IP 4. Non-existent AllocationId returns error 5. Empty result when no EIPs 6. Filter by instance-id, domain 7. Unknown filter returns InvalidParameterValue |
DONE |
describe-addresses-attribute |
--allocation-ids (optional filter) |
--attribute, --dry-run, --max-results, --next-token |
None | Gateway validates input → NATS ec2.DescribeAddressesAttribute with spinifex-workers queue group → daemon looks up EIP records by AllocationIds (or scans all account EIPs if none specified) → returns AddressAttribute list with AllocationId and PublicIp per entry. Unlike DescribeAddresses, returns empty list (not error) for non-existent AllocationIds. PtrRecord always nil (reverse-DNS not supported). |
1. List all EIP attributes 2. Filter by allocation IDs 3. Non-existent AllocationId returns empty (not error) 4. Nil input returns error |
DONE |
EC2 - NAT Gateway
NAT gateways are stored in spinifex-vpc-nat-gateways KV bucket. Deleted gateways are moved to spinifex-vpc-deleted-nat-gateways with a 1-hour TTL for Terraform polling. SNAT rule coordination uses vpc.add-nat-gateway and vpc.delete-nat-gateway NATS events (published via route table service on CreateRoute and during DeleteNatGateway respectively).
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-nat-gateway |
--subnet-id, --allocation-id |
--connectivity-type (public only), --tag-specifications, --dry-run |
Subnet must exist, EIP must be allocated and unassociated | Gateway validates SubnetId + AllocationId (required) → NATS ec2.CreateNatGateway with spinifex-workers queue group → daemon verifies subnet exists in spinifex-vpc-subnets KV → verifies EIP exists in spinifex-vpc-eips KV and has no AssociationId (Resource.AlreadyAssociated if in use) → generates nat-ID → stores NatGatewayRecord in spinifex-vpc-nat-gateways KV with state=available → returns NatGateway with addresses |
1. Create NAT gateway (happy path) 2. Subnet not found (InvalidSubnetID.NotFound) 3. EIP not found (InvalidAllocationID.NotFound) 4. EIP already associated (Resource.AlreadyAssociated) 5. Missing SubnetId or AllocationId (MissingParameter) |
DONE |
delete-nat-gateway |
--nat-gateway-id |
--dry-run |
NAT gateway must exist | Gateway validates NatGatewayId (required) → NATS ec2.DeleteNatGateway → daemon retrieves record from KV → scans all route tables in VPC for routes targeting this NAT gateway → publishes vpc.delete-nat-gateway event per affected subnet (SNAT rule cleanup) → sets state=deleted → stores in spinifex-vpc-deleted-nat-gateways with 1-hour TTL → deletes from active KV bucket |
1. Delete existing NAT gateway 2. Delete non-existent (InvalidNatGatewayID.NotFound) 3. Deleted gateway visible when queried by ID (state=deleted) 4. Deleted gateway excluded from general list queries (TTL auto-expire) |
DONE |
describe-nat-gateways |
--nat-gateway-ids, --filters (vpc-id, state) |
--max-results, --next-token, --dry-run |
None | NATS ec2.DescribeNatGateways → daemon scans spinifex-vpc-nat-gateways KV → applies NatGatewayIds filter and/or vpc-id/state filters → for specific IDs not found in active bucket, checks deleted bucket → returns InvalidNatGatewayID.NotFound if ID never existed → returns list with state, addresses, subnet, VPC |
1. List all NAT gateways 2. Filter by specific IDs 3. Filter by state 4. Filter by vpc-id 5. Non-existent ID (InvalidNatGatewayID.NotFound) 6. Deleted gateway visible by ID query |
DONE |
assign-private-nat-gateway-address |
— | --nat-gateway-id, --private-ip-addresses |
NAT gateway must exist (private type) | NATS ec2.AssignPrivateNatGatewayAddress → daemon assigns private IPs to NAT gateway → return assigned addresses |
1. Assign address 2. Missing NAT gateway ID (error) |
NOT STARTED |
associate-nat-gateway-address |
— | --nat-gateway-id, --allocation-ids |
NAT gateway and EIPs must exist | NATS ec2.AssociateNatGatewayAddress → daemon associates EIPs with NAT gateway → return association info |
1. Associate EIP 2. Missing NAT gateway ID (error) |
NOT STARTED |
EC2 - VPC Networking (Core)
Requires OVN/OVS (apt install openvswitch-switch ovn-host). VPC/Subnet/ENI/SG CRUD stores metadata in NATS KV and publishes events to vpcd (the network topology daemon) which translates to OVN logical switches, ports, routers, and ACLs. Single AZ for Spinifex v1. IPAM uses CAS-based allocation from NATS KV (spinifex-vpc-ipam bucket). Security groups stored in spinifex-vpc-security-groups KV bucket with optimistic locking on rule updates.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-vpc |
--cidr-block, --tag-specifications |
--instance-tenancy, --dry-run |
None | Gateway validates CidrBlock (required, valid CIDR, /16–/28 prefix) → NATS ec2.CreateVpc → daemon allocates unique VNI from spinifex-vpc-vni-counter KV → generates vpc-ID → stores VPCRecord in spinifex-vpc-vpcs KV → publishes vpc.create event to vpcd (creates OVN logical switch) → returns VPC with state=available |
1. Create VPC with /16 CIDR 2. Invalid CIDR format (error: InvalidVpcRange) 3. CIDR prefix outside /16–/28 (error) 4. Verify in describe-vpcs 5. Tags applied at creation |
DONE |
delete-vpc |
--vpc-id |
--dry-run |
VPC must be empty (no subnets) | Gateway validates VpcId → NATS ec2.DeleteVpc → daemon checks for dependent subnets in spinifex-vpc-subnets KV → returns DependencyViolation if subnets exist → deletes from spinifex-vpc-vpcs KV → publishes vpc.delete event to vpcd (removes OVN resources) |
1. Delete empty VPC 2. Delete VPC with subnets (DependencyViolation) 3. Delete non-existent VPC (error) |
DONE |
describe-vpcs |
--vpc-ids, --filters (vpc-id, state, cidr-block, is-default, owner-id, tag:*) |
--max-results, --dry-run |
None | NATS ec2.DescribeVpcs → daemon lists all keys from spinifex-vpc-vpcs KV → filters by VPC IDs if specified → applies filters → returns error for non-existent requested IDs → returns VPC list with CIDR, state, CIDR associations |
1. List all VPCs 2. Filter by VPC ID 3. Non-existent VPC ID returns error 4. Filter by state, cidr-block, is-default 5. Unknown filter returns InvalidParameterValue |
DONE |
modify-vpc-attribute |
--vpc-id, --enable-dns-hostnames, --enable-dns-support, --enable-network-address-usage-metrics |
--dry-run |
VPC must exist | Gateway validates VpcId non-empty → NATS ec2.ModifyVpcAttribute → daemon retrieves VPC record from spinifex-vpc-vpcs KV → updates boolean flags if provided → updates KV with optimistic locking → returns empty output |
1. Enable DNS hostnames 2. Enable DNS support 3. Enable network address usage metrics 4. Missing VPC ID (MissingParameter) 5. Non-existent VPC (InvalidVpcID.NotFound) |
DONE |
describe-vpc-attribute |
--vpc-id, --attribute (enableDnsHostnames, enableDnsSupport, enableNetworkAddressUsageMetrics) |
--dry-run |
VPC must exist | Gateway validates VpcId + Attribute non-empty → NATS ec2.DescribeVpcAttribute → daemon retrieves VPC record → returns single attribute as AttributeBooleanValue (one attribute per call, matches AWS behavior) → validates attribute name |
1. Get enableDnsHostnames 2. Get enableDnsSupport 3. Get enableNetworkAddressUsageMetrics 4. Invalid attribute name (InvalidParameterValue) 5. Missing VPC ID (MissingParameter) 6. Non-existent VPC (InvalidVpcID.NotFound) |
DONE |
associate-vpc-cidr-block |
— | --vpc-id, --cidr-block |
VPC must exist | NATS ec2.AssociateVpcCidrBlock → daemon adds secondary CIDR to VPC metadata → return association |
1. Add secondary CIDR 2. Overlapping CIDR (error) 3. Max CIDR blocks exceeded (error) |
NOT STARTED |
disassociate-vpc-cidr-block |
— | --association-id |
Association must exist | NATS ec2.DisassociateVpcCidrBlock → daemon removes secondary CIDR from VPC → return success |
1. Remove secondary CIDR 2. Remove primary CIDR (error) |
NOT STARTED |
create-subnet |
--vpc-id, --cidr-block, --availability-zone, --tag-specifications |
--dry-run |
VPC must exist | Gateway validates VpcId + CidrBlock (required, valid CIDR, /16–/28 prefix) → NATS ec2.CreateSubnet → daemon verifies parent VPC exists → validates subnet CIDR is within VPC CIDR range → checks for overlapping subnets in same VPC → calculates available IPs (total hosts - 5 AWS-reserved: .0/.1/.2/.3/.255) → generates subnet-ID → stores SubnetRecord in spinifex-vpc-subnets KV → publishes vpc.create-subnet event to vpcd → defaults AZ from config if omitted |
1. Create subnet within VPC CIDR 2. Subnet CIDR outside VPC range (error: InvalidSubnetRange) 3. Overlapping subnet CIDRs (error: InvalidSubnetConflict) 4. CIDR prefix outside /16–/28 (error) 5. Missing VPC or CIDR (error: MissingParameter) |
DONE |
delete-subnet |
--subnet-id |
--dry-run |
Subnet must exist | NATS ec2.DeleteSubnet → daemon verifies subnet exists → deletes from spinifex-vpc-subnets KV → publishes vpc.delete-subnet event to vpcd |
1. Delete existing subnet 2. Delete non-existent subnet (error) |
DONE |
describe-subnets |
--subnet-ids, --filters (vpc-id, subnet-id, availability-zone, cidr-block, state, default-for-az, tag:*) |
--max-results, --dry-run |
None | NATS ec2.DescribeSubnets → daemon lists all keys from spinifex-vpc-subnets KV → applies filters → filters by subnet IDs if specified → recalculates available IPs from CIDR → returns error for non-existent requested IDs |
1. List all subnets 2. Filter by VPC ID 3. Filter by subnet ID 4. Non-existent subnet returns error 5. Filter by availability-zone, cidr-block, state 6. Unknown filter returns InvalidParameterValue |
DONE |
modify-subnet-attribute |
--subnet-id, --map-public-ip-on-launch |
--assign-ipv6-address-on-creation, --dry-run |
Subnet must exist | Gateway validates SubnetId non-empty → NATS ec2.ModifySubnetAttribute → daemon retrieves Subnet record from spinifex-vpc-subnets KV → updates MapPublicIpOnLaunch flag if provided → updates KV with optimistic locking → returns empty output |
1. Enable auto-assign public IP 2. Disable auto-assign public IP 3. Missing SubnetId (MissingParameter) 4. Non-existent subnet (InvalidSubnetID.NotFound) |
DONE |
associate-subnet-cidr-block |
— | --subnet-id, --ipv6-cidr-block |
Subnet must exist | NATS ec2.AssociateSubnetCidrBlock → daemon adds IPv6 CIDR to subnet → return association |
1. Add IPv6 CIDR 2. Missing subnet ID (error) |
NOT STARTED |
disassociate-subnet-cidr-block |
— | --association-id |
Association must exist | NATS ec2.DisassociateSubnetCidrBlock → daemon removes IPv6 CIDR from subnet → return success |
1. Remove IPv6 CIDR 2. Invalid association (error) |
NOT STARTED |
create-security-group |
--group-name, --description, --vpc-id, --tag-specifications |
--dry-run |
VPC must exist | Gateway validates GroupName + VpcId required → NATS ec2.CreateSecurityGroup → daemon verifies VPC exists → checks duplicate name in same VPC → generates sg-ID → creates SecurityGroupRecord in spinifex-vpc-security-groups KV with default egress rule (allow all outbound 0.0.0.0/0), no ingress rules → publishes vpc.create-sg event to vpcd (OVN ACL configuration) → returns GroupId |
1. Create SG in VPC 2. Duplicate name in same VPC (error: InvalidGroup.Duplicate) 3. Verify default egress rule (allow all) 4. Verify no default ingress rules 5. Missing GroupName (MissingParameter) 6. Non-existent VPC (InvalidVpcID.NotFound) 7. With tag specifications |
DONE |
delete-security-group |
--group-id |
--dry-run |
SG must exist | Gateway validates GroupId non-empty → NATS ec2.DeleteSecurityGroup → daemon retrieves SG from KV → deletes from spinifex-vpc-security-groups KV → publishes vpc.delete-sg event to vpcd → returns success |
1. Delete existing SG 2. Delete non-existent SG (error: InvalidGroup.NotFound) 3. Missing GroupId (MissingParameter) |
DONE |
describe-security-groups |
--group-ids, --filters (vpc-id, group-name, group-id, description, ip-permission.cidr, tag:*) |
--group-names, --max-results, --dry-run |
None | NATS ec2.DescribeSecurityGroups → daemon scans all SG records in account → applies filters (AND logic across filter names, OR across values) → returns SecurityGroup list with ingress/egress rules → returns error for non-existent requested GroupIds |
1. List all SGs 2. Filter by VPC ID 3. Filter by group name 4. Filter by GroupId 5. Non-existent GroupId (InvalidGroup.NotFound) 6. Empty result when no SGs 7. Filter by description, ip-permission.cidr 8. Unknown filter returns InvalidParameterValue |
DONE |
authorize-security-group-ingress |
--group-id, --ip-permissions (IpProtocol, FromPort, ToPort, IpRanges/CidrIp, UserIdGroupPairs/GroupId) |
--dry-run |
SG must exist | Gateway validates GroupId non-empty → NATS ec2.AuthorizeSecurityGroupIngress → daemon retrieves SG → converts IpPermissions to SGRule objects → appends to IngressRules → updates KV with optimistic locking (revision check) → publishes vpc.update-sg event to vpcd → returns true |
1. Allow SSH (port 22) from 0.0.0.0/0 2. Allow from specific CIDR 3. Allow from source security group 4. Missing GroupId (MissingParameter) 5. Non-existent SG (InvalidGroup.NotFound) |
DONE |
authorize-security-group-egress |
--group-id, --ip-permissions |
--dry-run |
SG must exist | Gateway validates GroupId → NATS ec2.AuthorizeSecurityGroupEgress → daemon retrieves SG → appends rules to EgressRules → updates KV with optimistic locking → publishes vpc.update-sg event → returns true |
1. Allow HTTPS outbound 2. Restrict to specific CIDR 3. Missing GroupId (MissingParameter) 4. Non-existent SG (InvalidGroup.NotFound) |
DONE |
revoke-security-group-ingress |
--group-id, --ip-permissions (matching rules to remove) |
--dry-run |
SG must exist | Gateway validates GroupId → NATS ec2.RevokeSecurityGroupIngress → daemon retrieves SG → removes matching rules from IngressRules (set difference) → updates KV with optimistic locking → publishes vpc.update-sg event → returns true |
1. Revoke existing rule 2. Missing GroupId (MissingParameter) 3. Non-existent SG (InvalidGroup.NotFound) |
DONE |
revoke-security-group-egress |
--group-id, --ip-permissions (matching rules to remove) |
--dry-run |
SG must exist | Gateway validates GroupId → NATS ec2.RevokeSecurityGroupEgress → daemon retrieves SG → removes matching rules from EgressRules (set difference) → updates KV with optimistic locking → publishes vpc.update-sg event → returns true |
1. Revoke existing rule 2. Missing GroupId (MissingParameter) 3. Non-existent SG (InvalidGroup.NotFound) |
DONE |
create-internet-gateway |
--tag-specifications |
--dry-run |
None | NATS ec2.CreateInternetGateway → daemon generates igw-ID → creates IGWRecord in spinifex-igw KV bucket with state=available (not attached) → returns InternetGateway with empty Attachments |
1. Create IGW 2. Verify in describe-internet-gateways 3. Tags applied at creation |
DONE |
attach-internet-gateway |
--internet-gateway-id, --vpc-id |
--dry-run |
IGW and VPC must exist | NATS ec2.AttachInternetGateway → daemon validates IGW exists and is not already attached (ResourceAlreadyAssociated) → sets VpcId on record → state=attached → publishes vpc.igw-attach event to vpcd (creates OVN external switch, gateway port, SNAT rules) |
1. Attach IGW to VPC 2. Attach already-attached IGW (ResourceAlreadyAssociated) 3. Non-existent IGW (error) 4. Missing params (MissingParameter) |
DONE |
detach-internet-gateway |
--internet-gateway-id, --vpc-id |
--dry-run |
IGW must be attached to specified VPC | NATS ec2.DetachInternetGateway → daemon validates IGW exists and is attached to specified VPC (GatewayNotAttached if wrong VPC) → clears VpcId → state=available → publishes vpc.igw-detach event to vpcd (removes OVN resources) |
1. Detach attached IGW 2. Detach from wrong VPC (GatewayNotAttached) 3. Detach unattached IGW (error) |
DONE |
delete-internet-gateway |
--internet-gateway-id |
--dry-run |
IGW must be detached | NATS ec2.DeleteInternetGateway → daemon validates IGW exists and VpcId is empty (DependencyViolation if attached) → deletes from spinifex-igw KV |
1. Delete detached IGW 2. Delete attached IGW (DependencyViolation) 3. Delete non-existent (error) |
DONE |
describe-internet-gateways |
--internet-gateway-ids, --filters (internet-gateway-id, attachment.vpc-id, attachment.state, tag:*) |
--max-results, --dry-run |
None | NATS ec2.DescribeInternetGateways → daemon lists all keys from spinifex-igw KV → filters by IGW IDs if specified → applies filters → returns list with Attachments array (if attached) |
1. List all IGWs 2. Filter by IGW ID 3. Verify attachment info 4. Filter by attachment.vpc-id, attachment.state 5. Unknown filter returns InvalidParameterValue |
DONE |
EC2 - Route Tables
Route tables are stored in spinifex-vpc-route-tables JetStream KV bucket, keyed by {accountID}.{routeTableId}. Each route table has a Routes array (with local route pre-populated) and an Associations array. All operations validate VPC consistency. When a route targeting a NAT gateway is created, vpc.add-nat-gateway events are published per associated subnet for vpcd SNAT rule creation.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-route-table |
--vpc-id |
--tag-specifications, --dry-run |
VPC must exist | Gateway validates VpcId (required) → NATS ec2.CreateRouteTable with spinifex-workers queue group → daemon verifies VPC exists in spinifex-vpc-vpcs KV → retrieves VPC CIDR → generates rtb-ID → creates RouteTableRecord with IsMain=false, local route pre-populated (destination=VPC CIDR, gateway="local"), empty associations → stores in spinifex-vpc-route-tables KV → returns RouteTable |
1. Create route table with local route (happy path) 2. VPC not found (InvalidVpcID.NotFound) 3. Missing VpcId (MissingParameter) |
DONE |
delete-route-table |
--route-table-id |
--dry-run |
Route table must exist, not main, no subnet associations | Gateway validates RouteTableId (required) → NATS ec2.DeleteRouteTable → daemon retrieves record → rejects if IsMain=true (DependencyViolation) → rejects if has non-main subnet associations (DependencyViolation) → deletes from KV → returns success |
1. Delete custom route table 2. Delete main route table (DependencyViolation) 3. Delete table with subnet associations (DependencyViolation) 4. Delete non-existent (InvalidRouteTableID.NotFound) |
DONE |
describe-route-tables |
--route-table-ids, --filters (vpc-id, route-table-id, association.main, association.route-table-association-id, association.subnet-id, route.destination-cidr-block, route.gateway-id) |
--max-results, --next-token, --dry-run |
None | NATS ec2.DescribeRouteTables → daemon scans all keys in spinifex-vpc-route-tables KV → filters by RouteTableIds if specified → applies all provided filters (AND logic) → returns error for non-existent requested IDs → converts records to EC2 RouteTable objects |
1. List all route tables 2. Filter by vpc-id 3. Filter by association.main (true/false) 4. Non-existent ID (InvalidRouteTableID.NotFound) 5. Empty result when no route tables |
DONE |
create-route |
--route-table-id, --destination-cidr-block, --gateway-id, --nat-gateway-id |
--egress-only-internet-gateway-id, --vpc-peering-connection-id, --dry-run |
Route table and target must exist in same VPC | Gateway validates RouteTableId + DestinationCidrBlock (required) → NATS ec2.CreateRoute → daemon retrieves route table → checks for duplicate destination (RouteAlreadyExists) → validates exactly one of GatewayId/NatGatewayId provided → if GatewayId: verifies IGW exists in spinifex-igw KV and is attached to same VPC → if NatGatewayId: verifies NAT GW exists in spinifex-vpc-nat-gateways KV and is in same VPC → for NAT GW routes, publishes vpc.add-nat-gateway event per associated subnet → appends RouteRecord (state=active, origin=CreateRoute) → updates KV → returns Return=true |
1. Add IGW route (happy path) 2. Duplicate destination (RouteAlreadyExists) 3. IGW not found (InvalidInternetGatewayID.NotFound) 4. NAT GW not found (InvalidNatGatewayID.NotFound) 5. Gateway not in same VPC (InvalidParameterValue) 6. Missing RouteTableId or DestinationCidrBlock (MissingParameter) |
DONE |
delete-route |
--route-table-id, --destination-cidr-block |
--dry-run |
Route must exist | Gateway validates RouteTableId + DestinationCidrBlock (required) → NATS ec2.DeleteRoute → daemon retrieves route table → finds matching route → rejects if gateway is "local" (InvalidParameterValue — local routes cannot be deleted) → removes route from Routes array → updates KV → returns success |
1. Delete IGW route (happy path, local route remains) 2. Delete local route (InvalidParameterValue) 3. Route not found (InvalidRoute.NotFound) 4. Missing parameters (MissingParameter) |
DONE |
replace-route |
--route-table-id, --destination-cidr-block, --gateway-id |
--nat-gateway-id, --dry-run |
Route must exist, new target must exist in same VPC | Gateway validates RouteTableId + DestinationCidrBlock + GatewayId (required) → NATS ec2.ReplaceRoute → daemon retrieves route table → finds matching route → rejects if local route → verifies IGW exists and is attached to same VPC → updates route GatewayId, clears NatGatewayId, sets state=active → updates KV → returns success |
1. Replace route target (IGW swap) 2. Replace local route (InvalidParameterValue) 3. Route not found (InvalidRoute.NotFound) 4. IGW not found (InvalidInternetGatewayID.NotFound) |
DONE |
associate-route-table |
--route-table-id, --subnet-id |
--gateway-id, --dry-run |
Route table and subnet must exist in same VPC | Gateway validates RouteTableId + SubnetId (required) → NATS ec2.AssociateRouteTable → daemon retrieves route table → verifies subnet exists and is in same VPC → scans ALL route tables in VPC to check subnet isn't already explicitly associated (Resource.AlreadyAssociated) → generates rtbassoc-ID → appends AssociationRecord (Main=false) → updates KV → returns AssociationId with state=associated |
1. Associate subnet (happy path) 2. Duplicate subnet association (Resource.AlreadyAssociated) 3. Subnet not found (InvalidSubnetID.NotFound) 4. Subnet in different VPC (InvalidParameterValue) 5. Missing parameters (MissingParameter) |
DONE |
disassociate-route-table |
--association-id |
--dry-run |
Association must exist, not main table association | Gateway validates AssociationId (required) → NATS ec2.DisassociateRouteTable → daemon scans ALL route tables in account to find matching AssociationId → rejects if Main=true (InvalidParameterValue — main associations are immutable) → removes association from Associations array → updates KV → returns success |
1. Disassociate subnet (happy path) 2. Disassociate main table (InvalidParameterValue) 3. Association not found (InvalidAssociationID.NotFound) |
DONE |
replace-route-table-association |
--association-id, --route-table-id |
--dry-run |
Association and new route table must exist | Gateway validates AssociationId + RouteTableId (required) → NATS ec2.ReplaceRouteTableAssociation → daemon verifies new route table exists → scans all route tables to find matching AssociationId → if Main=true: atomically swaps IsMain flag between old and new table → two-phase commit: removes association from old table, creates new AssociationId on new table → compensating write on failure (logs CRITICAL if compensation fails) → returns new AssociationId with state=associated |
1. Move subnet between tables (happy path) 2. Replace main table association (swaps IsMain flags) 3. Association not found (InvalidAssociationID.NotFound) 4. Route table not found (InvalidRouteTableID.NotFound) |
DONE |
EC2 - VPC Peering
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-vpc-peering-connection |
— | --vpc-id, --peer-vpc-id, --peer-owner-id, --peer-region, --tag-specifications |
Both VPCs must exist | NATS ec2.CreateVpcPeeringConnection → daemon creates peering in KV with pcx-ID → state=pending-acceptance → return peering connection |
1. Create peering between two VPCs 2. Missing VPC ID (error) 3. Overlapping CIDRs (error) |
NOT STARTED |
accept-vpc-peering-connection |
— | --vpc-peering-connection-id, --dry-run |
Peering must exist in pending-acceptance state | NATS ec2.AcceptVpcPeeringConnection → daemon updates state to active → configure OVS cross-VPC flows → return peering |
1. Accept pending peering 2. Accept already active (error) |
NOT STARTED |
reject-vpc-peering-connection |
— | --vpc-peering-connection-id, --dry-run |
Peering must exist in pending-acceptance state | NATS ec2.RejectVpcPeeringConnection → daemon updates state to rejected → return peering |
1. Reject pending peering 2. Reject already active (error) |
NOT STARTED |
delete-vpc-peering-connection |
— | --vpc-peering-connection-id, --dry-run |
Peering must exist | NATS ec2.DeleteVpcPeeringConnection → daemon removes OVS cross-VPC flows → delete from KV → return success |
1. Delete active peering 2. Delete non-existent (error) |
NOT STARTED |
describe-vpc-peering-connections |
— | --vpc-peering-connection-ids, --filters, --max-results |
None | NATS ec2.DescribeVpcPeeringConnections → daemon lists peerings from KV → return list with state |
1. List all peerings 2. Filter by ID 3. Filter by VPC |
NOT STARTED |
modify-vpc-peering-connection-options |
— | --vpc-peering-connection-id, --requester-peering-connection-options, --accepter-peering-connection-options |
Peering must be active | NATS ec2.ModifyVpcPeeringConnectionOptions → daemon updates peering options in KV → return modified options |
1. Enable DNS resolution 2. Missing peering ID (error) |
NOT STARTED |
EC2 - VPC Endpoints
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-vpc-endpoint |
— | --vpc-id, --service-name, --vpc-endpoint-type (Gateway/Interface), --route-table-ids, --subnet-ids, --tag-specifications |
VPC must exist | NATS ec2.CreateVpcEndpoint → daemon creates endpoint in KV with vpce-ID → return endpoint |
1. Create Gateway endpoint for S3 2. Create Interface endpoint 3. Missing VPC ID (error) |
NOT STARTED |
delete-vpc-endpoints |
— | --vpc-endpoint-ids, --dry-run |
Endpoints must exist | NATS ec2.DeleteVpcEndpoints → daemon deletes endpoints from KV → return success/failure per endpoint |
1. Delete existing endpoint 2. Delete non-existent (error) |
NOT STARTED |
describe-vpc-endpoints |
— | --vpc-endpoint-ids, --filters, --max-results |
None | NATS ec2.DescribeVpcEndpoints → daemon lists endpoints from KV → return list with state and type |
1. List all endpoints 2. Filter by VPC 3. Filter by service name |
NOT STARTED |
describe-vpc-endpoint-services |
— | --service-names, --filters, --max-results |
None | NATS ec2.DescribeVpcEndpointServices → daemon returns available services (S3, DynamoDB equivalents) → return service list |
1. List available services 2. Filter by service name |
NOT STARTED |
modify-vpc-endpoint |
— | --vpc-endpoint-id, --add-route-table-ids, --remove-route-table-ids, --add-subnet-ids, --remove-subnet-ids |
Endpoint must exist | NATS ec2.ModifyVpcEndpoint → daemon updates endpoint route tables/subnets in KV → return success |
1. Add route table 2. Remove subnet 3. Missing endpoint ID (error) |
NOT STARTED |
EC2 - VPN Gateway & Customer Gateway
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-customer-gateway |
— | --type (ipsec.1), --bgp-asn, --ip-address, --tag-specifications |
None | NATS ec2.CreateCustomerGateway → daemon stores in JetStream KV with cgw-ID → return customer gateway |
1. Create with BGP ASN and IP 2. Missing required fields (error) |
NOT STARTED |
delete-customer-gateway |
— | --customer-gateway-id, --dry-run |
Must exist, no VPN connections | NATS ec2.DeleteCustomerGateway → daemon verifies no active connections → delete from KV → return success |
1. Delete existing 2. Delete with active VPN (error) |
NOT STARTED |
describe-customer-gateways |
— | --customer-gateway-ids, --filters |
None | NATS ec2.DescribeCustomerGateways → daemon lists from KV → return list with BGP/IP info |
1. List all 2. Filter by ID |
NOT STARTED |
create-vpn-gateway |
— | --type (ipsec.1), --amazon-side-asn, --tag-specifications |
None | NATS ec2.CreateVpnGateway → daemon stores in JetStream KV with vgw-ID → return VPN gateway |
1. Create VPN gateway 2. Verify in describe |
NOT STARTED |
delete-vpn-gateway |
— | --vpn-gateway-id, --dry-run |
Must be detached | NATS ec2.DeleteVpnGateway → daemon verifies detached → delete from KV → return success |
1. Delete detached gateway 2. Delete attached (error) |
NOT STARTED |
attach-vpn-gateway |
— | --vpn-gateway-id, --vpc-id |
Both must exist | NATS ec2.AttachVpnGateway → daemon links VPN gateway to VPC → return attachment |
1. Attach to VPC 2. Already attached (error) |
NOT STARTED |
detach-vpn-gateway |
— | --vpn-gateway-id, --vpc-id |
Must be attached | NATS ec2.DetachVpnGateway → daemon unlinks VPN gateway from VPC → return success |
1. Detach from VPC 2. Not attached (error) |
NOT STARTED |
describe-vpn-gateways |
— | --vpn-gateway-ids, --filters |
None | NATS ec2.DescribeVpnGateways → daemon lists from KV → return list with attachment info |
1. List all 2. Filter by VPC attachment |
NOT STARTED |
EC2 - VPN Connections
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-vpn-connection |
— | --type (ipsec.1), --customer-gateway-id, --vpn-gateway-id, --options, --tag-specifications |
Customer gateway and VPN gateway must exist | NATS ec2.CreateVpnConnection → daemon stores connection in JetStream KV with vpn-ID → return connection with tunnel info |
1. Create IPsec connection 2. Missing customer gateway (error) 3. Missing VPN gateway (error) |
NOT STARTED |
delete-vpn-connection |
— | --vpn-connection-id, --dry-run |
Connection must exist | NATS ec2.DeleteVpnConnection → daemon deletes from KV → return success |
1. Delete existing connection 2. Delete non-existent (error) |
NOT STARTED |
describe-vpn-connections |
— | --vpn-connection-ids, --filters |
None | NATS ec2.DescribeVpnConnections → daemon lists from KV → return list with state and tunnel info |
1. List all connections 2. Filter by ID 3. Filter by VPN gateway |
NOT STARTED |
modify-vpn-connection |
— | --vpn-connection-id, --vpn-gateway-id, --customer-gateway-id |
Connection must exist | NATS ec2.ModifyVpnConnection → daemon updates connection target in KV → return modified connection |
1. Change VPN gateway 2. Missing connection ID (error) |
NOT STARTED |
modify-vpn-connection-options |
— | --vpn-connection-id, --local-ipv4-network-cidr, --remote-ipv4-network-cidr |
Connection must exist | NATS ec2.ModifyVpnConnectionOptions → daemon updates tunnel options in KV → return modified connection |
1. Modify CIDR ranges 2. Missing connection ID (error) |
NOT STARTED |
EC2 - Network ACLs
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-network-acl |
— | --vpc-id, --tag-specifications |
VPC must exist | NATS ec2.CreateNetworkAcl → daemon creates NACL in KV with acl-ID → add default deny-all rules → return NACL |
1. Create NACL in VPC 2. Missing VPC ID (error) 3. Verify default deny rules |
NOT STARTED |
delete-network-acl |
— | --network-acl-id, --dry-run |
Must not be default NACL | NATS ec2.DeleteNetworkAcl → daemon verifies not default → delete from KV → return success |
1. Delete non-default NACL 2. Delete default NACL (error) |
NOT STARTED |
describe-network-acls |
— | --network-acl-ids, --filters, --max-results |
None | NATS ec2.DescribeNetworkAcls → daemon lists NACLs from KV → return list with entries |
1. List all NACLs 2. Filter by VPC |
NOT STARTED |
create-network-acl-entry |
— | --network-acl-id, --rule-number, --protocol, --rule-action (allow/deny), --cidr-block, --ingress/--egress, --port-range |
NACL must exist | NATS ec2.CreateNetworkAclEntry → daemon adds rule entry to NACL in KV → configure OVS flow → return success |
1. Add ingress allow rule 2. Add egress deny rule 3. ICMP type/code parsing 4. Missing NACL ID (error) |
NOT STARTED |
delete-network-acl-entry |
— | --network-acl-id, --rule-number, --ingress/--egress |
Entry must exist | NATS ec2.DeleteNetworkAclEntry → daemon removes rule from NACL → remove OVS flow → return success |
1. Delete existing entry 2. Delete non-existent entry (error) |
NOT STARTED |
replace-network-acl-association |
— | --association-id, --network-acl-id |
Both must exist | NATS ec2.ReplaceNetworkAclAssociation → daemon swaps NACL for subnet → return new association ID |
1. Replace association 2. Missing association ID (error) |
NOT STARTED |
replace-network-acl-entry |
— | --network-acl-id, --rule-number, --protocol, --rule-action, --cidr-block, --ingress/--egress, --port-range |
Entry must exist | NATS ec2.ReplaceNetworkAclEntry → daemon replaces rule in KV → update OVS flow → return success |
1. Replace existing entry 2. Replace non-existent entry (error) |
NOT STARTED |
EC2 - Prefix Lists
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-managed-prefix-list |
— | --prefix-list-name, --address-family (IPv4/IPv6), --max-entries, --entries, --tag-specifications |
None | NATS ec2.CreateManagedPrefixList → daemon stores prefix list in JetStream KV with pl-ID → return prefix list with version |
1. Create with IPv4 entries 2. Name, address family, max entries required 3. Verify in describe |
NOT STARTED |
delete-managed-prefix-list |
— | --prefix-list-id, --dry-run |
Must exist, not referenced by security groups/route tables | NATS ec2.DeleteManagedPrefixList → daemon verifies not referenced → delete from KV → return success |
1. Delete unreferenced list 2. Delete referenced list (error) |
NOT STARTED |
describe-managed-prefix-lists |
— | --prefix-list-ids, --filters, --max-results |
None | NATS ec2.DescribeManagedPrefixLists → daemon lists from KV → return list with state and version |
1. List all prefix lists 2. Filter by ID |
NOT STARTED |
modify-managed-prefix-list |
— | --prefix-list-id, --current-version, --add-entries, --remove-entries, --prefix-list-name |
Must exist, current version must match | NATS ec2.ModifyManagedPrefixList → daemon validates version → apply changes → increment version → return updated list |
1. Add entries 2. Remove entries 3. Version mismatch (error) |
NOT STARTED |
get-managed-prefix-list-entries |
— | --prefix-list-id, --target-version, --max-results |
Prefix list must exist | NATS ec2.GetManagedPrefixListEntries → daemon reads entries from KV → return entry list |
1. List entries 2. List entries at specific version |
NOT STARTED |
get-managed-prefix-list-associations |
— | --prefix-list-id, --max-results |
Prefix list must exist | NATS ec2.GetManagedPrefixListAssociations → daemon finds references in SGs/route tables → return association list |
1. List associations 2. No associations returns empty |
NOT STARTED |
EC2 - Network Interfaces
ENIs are managed via spinifex-vpc-enis KV bucket with CAS-based IPAM allocation from spinifex-vpc-ipam. ENIs are auto-created by run-instances --subnet-id and auto-deleted on termination. MAC addresses are deterministically generated from the ENI ID. Attach/detach are internal operations used by instance lifecycle — no standalone attach/detach API yet.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-network-interface |
--subnet-id, --private-ip-address, --description, --tag-specifications |
--groups, --dry-run |
Subnet must exist | Gateway validates SubnetId (required) → NATS ec2.CreateNetworkInterface → daemon verifies subnet exists in spinifex-vpc-subnets KV → if no PrivateIpAddress, allocates from IPAM pool (CAS-based, up to 5 retries) → generates eni-ID → generates deterministic MAC from ENI ID → stores ENIRecord in spinifex-vpc-enis KV → publishes vpc.create-port event to vpcd (creates OVN logical port) → returns NetworkInterface with status=available |
1. Create in subnet (auto-allocate IP) 2. Create with specific private IP 3. Missing subnet ID (MissingParameter) 4. Non-existent subnet (error) 5. Tags applied at creation |
DONE |
delete-network-interface |
--network-interface-id |
--dry-run |
Must not be attached (status != in-use) | NATS ec2.DeleteNetworkInterface → daemon verifies ENI exists and status is not "in-use" (InvalidNetworkInterfaceInUse) → releases IP back to IPAM pool → deletes from spinifex-vpc-enis KV → publishes vpc.delete-port event to vpcd |
1. Delete detached ENI 2. Delete attached ENI (InvalidNetworkInterfaceInUse) 3. Delete non-existent (error) |
DONE |
describe-network-interfaces |
--network-interface-ids, --filters (subnet-id, vpc-id, attachment.instance-id) |
--max-results, --dry-run |
None | NATS ec2.DescribeNetworkInterfaces → daemon lists all keys from spinifex-vpc-enis KV → applies filters (subnet-id, vpc-id, attachment.instance-id) → filters by ENI IDs if specified → returns error for non-existent requested IDs |
1. List all ENIs 2. Filter by subnet-id 3. Filter by vpc-id 4. Filter by attachment.instance-id 5. Non-existent ENI returns error |
DONE |
attach-network-interface |
— | --network-interface-id, --instance-id, --device-index |
ENI and instance must exist | Internal operation used by run-instances --subnet-id. Marks ENI as "in-use", sets instance attachment, generates attachment ID. No standalone API endpoint yet. |
1. Attach ENI to instance 2. Already attached (error) 3. Missing device index (error) |
NOT STARTED (internal only) |
detach-network-interface |
— | --attachment-id, --force |
ENI must be attached | Internal operation used by terminate-instances. Marks ENI as "available", clears attachment info. No standalone API endpoint yet. | 1. Detach ENI 2. Force detach 3. Not attached (error) |
NOT STARTED (internal only) |
modify-network-interface-attribute |
— | --network-interface-id, --description, --groups |
ENI must exist; at least one attribute required | NATS ec2.ModifyNetworkInterfaceAttribute → daemon validates ENI exists → updates description and/or security groups in spinifex-vpc-enis KV (optimistic locking) → return success |
1. Modify description 2. Update security groups 3. No attributes (InvalidParameterValue) 4. Non-existent ENI (error) |
DONE |
assign-private-ip-addresses |
— | --network-interface-id, --private-ip-addresses or --secondary-private-ip-address-count |
ENI must exist | NATS ec2.AssignPrivateIpAddresses → daemon allocates IPs from subnet pool → assign to ENI → return assigned IPs |
1. Assign specific IPs 2. Assign by count 3. Cannot specify both (error) |
NOT STARTED |
unassign-private-ip-addresses |
— | --network-interface-id, --private-ip-addresses |
IPs must be assigned to ENI | NATS ec2.UnassignPrivateIpAddresses → daemon releases IPs → return success |
1. Unassign secondary IP 2. Unassign primary IP (error) |
NOT STARTED |
EC2 - Launch Templates
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-launch-template |
— | --launch-template-name, --launch-template-data (ImageId, InstanceType, etc.), --tag-specifications |
None | NATS ec2.CreateLaunchTemplate → daemon stores template in JetStream KV with lt-ID → version 1 → return template metadata |
1. Create template with full config 2. Duplicate name (error) 3. Verify in describe output |
NOT STARTED |
create-launch-template-version |
— | --launch-template-id or --launch-template-name, --launch-template-data, --source-version |
Template must exist | NATS ec2.CreateLaunchTemplateVersion → daemon creates new version in KV → return version details |
1. Create version from scratch 2. Create version from source version 3. Template not found (error) |
NOT STARTED |
delete-launch-template |
— | --launch-template-id or --launch-template-name, --dry-run |
Template must exist | NATS ec2.DeleteLaunchTemplate → daemon deletes template and all versions from KV → return deleted template info |
1. Delete by ID 2. Delete by name 3. Non-existent template (error) |
NOT STARTED |
describe-launch-templates |
— | --launch-template-ids, --launch-template-names, --filters |
None | NATS ec2.DescribeLaunchTemplates → daemon lists templates from KV → return list |
1. List all templates 2. Filter by name 3. Filter by ID |
NOT STARTED |
describe-launch-template-versions |
— | --launch-template-id or --launch-template-name, --versions, --min-version, --max-version |
Template must exist | NATS ec2.DescribeLaunchTemplateVersions → daemon lists versions from KV → return version list with data |
1. List all versions 2. List specific version 3. Template not found (error) |
NOT STARTED |
EC2 - Misc Operations
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
delete-network-interface-permission |
— | --network-interface-permission-id, --force |
Permission must exist | NATS ec2.DeleteNetworkInterfacePermission → daemon deletes permission from KV → return success |
1. Delete existing permission 2. Force delete |
NOT STARTED |
enable-address-transfer |
— | --allocation-id, --transfer-account-id |
EIP must exist | NATS ec2.EnableAddressTransfer → daemon stores transfer in JetStream KV with 7-day expiration → return AllocationId and TransferAccountId |
1. Enable transfer 2. Verify transfer stored with expiration |
NOT STARTED |
disable-address-transfer |
— | --allocation-id |
Transfer must exist | NATS ec2.DisableAddressTransfer → daemon updates transfer status to disabled in KV → return status |
1. Disable active transfer 2. Disable non-existent transfer (error) |
NOT STARTED |
EBS Direct API
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
start-snapshot |
— | --volume-size, --parent-snapshot-id, --description, --encrypted |
None | Initialize snapshot in viperblock → return snap-ID for block-level writes | 1. Start new snapshot 2. Start incremental from parent |
NOT STARTED |
put-snapshot-block |
— | --snapshot-id, --block-index, --block-data, --checksum |
start-snapshot | Write block data to snapshot at specified index in viperblock → verify checksum | 1. Write single block 2. Write with bad checksum (error) |
NOT STARTED |
get-snapshot-block |
— | --snapshot-id, --block-index |
Snapshot must exist | Read block data from snapshot at specified index → return data with checksum | 1. Read existing block 2. Read sparse block (zeros) |
NOT STARTED |
complete-snapshot |
— | --snapshot-id, --changed-blocks-count |
start-snapshot, put-snapshot-block | Finalize snapshot → mark as completed → sync to Predastore | 1. Complete valid snapshot 2. Complete with wrong block count (error) |
NOT STARTED |
list-snapshot-blocks |
— | --snapshot-id, --max-results, --next-token |
Snapshot must exist | List all non-empty blocks in snapshot with tokens and sizes | 1. List blocks of populated snapshot 2. List blocks of empty snapshot |
NOT STARTED |
list-changed-blocks |
— | --second-snapshot-id, --first-snapshot-id, --max-results |
Snapshots must exist | Compare two snapshots → return list of differing block indexes | 1. Compare parent and child snapshots 2. Compare unrelated snapshots |
NOT STARTED |
IAM - User Management
All IAM operations are account-scoped. Users, policies, and access keys are namespaced per account. The gateway extracts the account ID from the authenticated access key via c.Locals("sigv4.accountId") and passes it to all IAM service methods. Root user (account 000000000000) bypasses policy evaluation entirely.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-user |
--user-name, --path |
--tags, --permissions-boundary |
None | Gateway validates UserName non-empty → IAMService.CreateUser(accountID, input) → generates AIDA-prefixed UserId → stores in KV at key {accountID}.{userName} → ARN includes real account ID → conditional create for race safety |
1. Create user 2. Duplicate name in same account (EntityAlreadyExists) 3. Same name in different account (succeeds) 4. Missing username (InvalidInput) 5. Default path is / |
DONE |
get-user |
--user-name |
— | User must exist in caller's account | Gateway validates UserName → IAMService.GetUser(accountID, input) → reads from KV at {accountID}.{userName} → returns User with ARN, Path, CreateDate |
1. Get existing user 2. Get non-existent user (NoSuchEntity) 3. Cannot see user in different account (NoSuchEntity) |
DONE |
list-users |
--path-prefix |
--max-items, --marker |
None | IAMService.ListUsers(accountID, input) → prefix-scans KV for {accountID}.* → filters by path prefix if specified → returns user list scoped to caller's account |
1. List all users in account 2. List with path prefix filter 3. Empty list 4. Does not show other accounts' users |
DONE |
delete-user |
--user-name |
— | User must exist, no access keys attached | Gateway validates UserName → IAMService.DeleteUser(accountID, input) → checks user has no access keys → deletes from KV | 1. Delete user with no keys 2. Delete user with keys attached (DeleteConflict) 3. Delete non-existent user (NoSuchEntity) |
DONE |
IAM - Access Key Management
Access keys are globally unique (keyed by AccessKeyID) but associated with an account. Max 2 keys per user. Secrets encrypted with AES-256-GCM using master key.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-access-key |
--user-name |
— | User must exist | IAMService.CreateAccessKey(accountID, input) → generates AKIA-prefixed key ID + random secret → encrypts secret with master key → stores in KV (globally unique key ID) → updates user's AccessKeys list → returns plaintext secret (one-time only) | 1. Create key for user (AKIA prefix) 2. Max 2 keys per user (LimitExceeded on 3rd) 3. Non-existent user (NoSuchEntity) 4. Secret only returned on creation |
DONE |
list-access-keys |
--user-name |
--max-items, --marker |
None | IAMService.ListAccessKeys(accountID, input) → reads user's AccessKeys list → fetches each key's metadata → returns key IDs and status (no secrets) | 1. List keys for user 2. No keys returns empty list 3. Shows Status (Active/Inactive) but not secrets |
DONE |
delete-access-key |
--access-key-id, --user-name |
— | Key must exist | IAMService.DeleteAccessKey(accountID, input) → removes key from global KV → removes key ID from user's AccessKeys list | 1. Delete existing key 2. Delete non-existent key (NoSuchEntity) |
DONE |
update-access-key |
--access-key-id, --user-name, --status (Active/Inactive) |
— | Key must exist | IAMService.UpdateAccessKey(input) → reads key from global KV (key ID is globally unique, no accountID needed) → updates Status field → inactive keys rejected at SigV4 auth | 1. Deactivate key (Inactive) 2. Reactivate key (Active) 3. Non-existent key (NoSuchEntity) 4. Inactive key → auth returns InvalidClientTokenId |
DONE |
IAM - Policy Management
Policies are account-scoped. KV key is {accountID}.{policyName}. Policy documents require Version: "2012-10-17" and valid Statement array. Supports wildcard action matching (ec2:*, ec2:Describe*, *). Evaluation order: explicit Deny > explicit Allow > implicit Deny.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-policy |
--policy-name, --policy-document, --path, --description |
--tags |
None | Gateway validates PolicyName and PolicyDocument non-empty → IAMService.CreatePolicy(accountID, input) → validates JSON (Version, Statement with Effect/Action/Resource) → generates ANPA-prefixed PolicyId → stores at {accountID}.{policyName} → ARN includes real account ID |
1. Create with valid document 2. Malformed JSON (MalformedPolicyDocument) 3. Missing Effect/Action/Resource (MalformedPolicyDocument) 4. Duplicate name in same account (EntityAlreadyExists) 5. Same name in different account (succeeds) 6. With path and description |
DONE |
get-policy |
--policy-arn |
— | Policy must exist | Gateway validates PolicyArn → extracts account ID and policy name from ARN → IAMService.GetPolicy(accountID, input) → returns policy metadata (no document) with AttachmentCount | 1. Get existing policy 2. Non-existent policy (NoSuchEntity) 3. Returns metadata only, not document |
DONE |
get-policy-version |
--policy-arn, --version-id |
— | Policy must exist | Gateway validates PolicyArn and VersionId → IAMService.GetPolicyVersion(accountID, input) → returns PolicyVersion with URL-encoded Document, VersionId, IsDefaultVersion | 1. Get version v1 (returns document) 2. Non-existent version (NoSuchEntity) 3. Non-existent policy (NoSuchEntity) |
DONE |
list-policies |
— | --scope, --only-attached, --path-prefix, --max-items, --marker |
None | IAMService.ListPolicies(accountID, input) → prefix-scans KV for {accountID}.* → returns all policies in caller's account |
1. List all policies in account 2. Empty list 3. Does not show other accounts' policies |
DONE |
delete-policy |
--policy-arn |
— | Policy must exist, not attached to any user | Gateway validates PolicyArn → IAMService.DeletePolicy(accountID, input) → checks no users have this policy attached (scans account's users) → deletes from KV | 1. Delete unattached policy 2. Delete attached policy (DeleteConflict) 3. Delete non-existent (NoSuchEntity) |
DONE |
IAM - Policy Attachment
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
attach-user-policy |
--user-name, --policy-arn |
— | User and policy must exist in caller's account | Gateway validates both params → IAMService.AttachUserPolicy(accountID, input) → verifies policy exists → adds policy ARN to user's AttachedPolicies list → idempotent (re-attach succeeds without duplicate) | 1. Attach policy to user 2. Already attached (idempotent, no error) 3. Non-existent user (NoSuchEntity) 4. Non-existent policy (NoSuchEntity) |
DONE |
detach-user-policy |
--user-name, --policy-arn |
— | Policy must be attached to user | Gateway validates both params → IAMService.DetachUserPolicy(accountID, input) → removes policy ARN from user's AttachedPolicies list | 1. Detach attached policy 2. Policy not attached (NoSuchEntity) 3. Non-existent user (NoSuchEntity) |
DONE |
list-attached-user-policies |
--user-name |
--path-prefix, --max-items, --marker |
User must exist | IAMService.ListAttachedUserPolicies(accountID, input) → reads user's AttachedPolicies list → returns policy ARNs and names | 1. List attached policies 2. No policies returns empty list |
DONE |
IAM - Policy Evaluation
The gateway evaluates IAM policies on every request (except root user bypass). Policy evaluation is internal — not an AWS API.
| Feature | Description | Status |
|---|---|---|
| Root user bypass | Root user (000000000000:root) skips all policy checks |
DONE |
| Default deny | Users with no attached policies are denied all actions | DONE |
| Explicit allow | Matching Allow statement grants access | DONE |
| Explicit deny | Deny statements override Allow (deny wins) | DONE |
| Wildcard matching | * matches all, ec2:* matches all EC2, ec2:Describe* matches prefix |
DONE |
| Account-scoped evaluation | Policies resolved from caller's account context | DONE |
| EC2 action mapping | All registered EC2 actions mapped to IAM policy strings | DONE |
| IAM action mapping | 16 IAM actions mapped to IAM policy strings | DONE |
IAM - Roles
IAM roles are required for instance profiles, cross-account access, and service-linked permissions. Without roles, workloads must use static access keys. Roles should be stored in a spinifex-iam-roles KV bucket keyed by {accountID}.{roleName}. Trust policies (AssumeRolePolicyDocument) define who can assume the role.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-role |
— | --role-name, --assume-role-policy-document, --path, --description, --max-session-duration, --permissions-boundary, --tags |
None | IAMService.CreateRole(accountID, input) → validate RoleName + AssumeRolePolicyDocument (required) → validate trust policy JSON → generate AROA-prefixed RoleId → store RoleRecord in spinifex-iam-roles KV at {accountID}.{roleName} → ARN: arn:aws:iam::{accountID}:role/{path}{roleName} → return Role |
1. Create with valid trust policy 2. Malformed trust policy (MalformedPolicyDocument) 3. Duplicate name (EntityAlreadyExists) 4. Missing RoleName (InvalidInput) 5. With path and description |
NOT STARTED |
get-role |
— | --role-name |
Role must exist | IAMService.GetRole(accountID, input) → reads from KV at {accountID}.{roleName} → returns Role with ARN, Path, AssumeRolePolicyDocument, CreateDate |
1. Get existing role 2. Non-existent role (NoSuchEntity) |
NOT STARTED |
list-roles |
— | --path-prefix, --max-items, --marker |
None | IAMService.ListRoles(accountID, input) → prefix-scans KV for {accountID}.* → filters by path prefix → returns role list scoped to caller's account |
1. List all roles 2. List with path prefix 3. Empty list 4. Account isolation |
NOT STARTED |
delete-role |
— | --role-name |
Role must exist, no attached policies, no instance profiles | IAMService.DeleteRole(accountID, input) → verify no attached policies (DeleteConflict) → verify not in any instance profile (DeleteConflict) → delete from KV | 1. Delete role with no attachments 2. Delete role with policies (DeleteConflict) 3. Delete role in instance profile (DeleteConflict) 4. Non-existent role (NoSuchEntity) |
NOT STARTED |
update-role |
— | --role-name, --description, --max-session-duration |
Role must exist | IAMService.UpdateRole(accountID, input) → update description and/or session duration in KV → return success | 1. Update description 2. Update max session duration 3. Non-existent role (NoSuchEntity) |
NOT STARTED |
update-assume-role-policy |
— | --role-name, --policy-document |
Role must exist | IAMService.UpdateAssumeRolePolicy(accountID, input) → validate trust policy JSON → update AssumeRolePolicyDocument in KV → return success | 1. Update trust policy 2. Malformed policy (MalformedPolicyDocument) 3. Non-existent role (NoSuchEntity) |
NOT STARTED |
attach-role-policy |
— | --role-name, --policy-arn |
Role and policy must exist | IAMService.AttachRolePolicy(accountID, input) → verify policy exists → add ARN to role's AttachedPolicies → idempotent | 1. Attach policy 2. Already attached (idempotent) 3. Non-existent role (NoSuchEntity) 4. Non-existent policy (NoSuchEntity) |
NOT STARTED |
detach-role-policy |
— | --role-name, --policy-arn |
Policy must be attached | IAMService.DetachRolePolicy(accountID, input) → remove ARN from role's AttachedPolicies | 1. Detach attached policy 2. Not attached (NoSuchEntity) 3. Non-existent role (NoSuchEntity) |
NOT STARTED |
list-attached-role-policies |
— | --role-name, --path-prefix, --max-items, --marker |
Role must exist | IAMService.ListAttachedRolePolicies(accountID, input) → reads role's AttachedPolicies list → returns ARNs and names | 1. List attached policies 2. No policies (empty list) 3. Non-existent role (NoSuchEntity) |
NOT STARTED |
IAM - Instance Profiles
Instance profiles are containers for IAM roles that allow EC2 instances to assume a role and obtain temporary credentials. Required for run-instances --iam-instance-profile. Stored in spinifex-iam-instance-profiles KV bucket keyed by {accountID}.{profileName}.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-instance-profile |
— | --instance-profile-name, --path, --tags |
None | IAMService.CreateInstanceProfile(accountID, input) → validate name (required) → generate AIPA-prefixed InstanceProfileId → store in spinifex-iam-instance-profiles KV with empty Roles array → ARN: arn:aws:iam::{accountID}:instance-profile/{path}{name} → return InstanceProfile |
1. Create profile 2. Duplicate name (EntityAlreadyExists) 3. Missing name (InvalidInput) |
NOT STARTED |
get-instance-profile |
— | --instance-profile-name |
Profile must exist | IAMService.GetInstanceProfile(accountID, input) → reads from KV → returns InstanceProfile with Roles array | 1. Get existing profile 2. Non-existent (NoSuchEntity) |
NOT STARTED |
list-instance-profiles |
— | --path-prefix, --max-items, --marker |
None | IAMService.ListInstanceProfiles(accountID, input) → prefix-scans KV → returns list | 1. List all profiles 2. Empty list |
NOT STARTED |
list-instance-profiles-for-role |
— | --role-name, --max-items, --marker |
Role must exist | IAMService.ListInstanceProfilesForRole(accountID, input) → scan all profiles → filter where Roles contains role name → return list | 1. List profiles for role 2. Role not in any profile (empty) |
NOT STARTED |
delete-instance-profile |
— | --instance-profile-name |
Profile must exist, Roles must be empty | IAMService.DeleteInstanceProfile(accountID, input) → verify Roles empty (DeleteConflict if not) → delete from KV | 1. Delete empty profile 2. Delete profile with role (DeleteConflict) 3. Non-existent (NoSuchEntity) |
NOT STARTED |
add-role-to-instance-profile |
— | --instance-profile-name, --role-name |
Both must exist, max 1 role per profile | IAMService.AddRoleToInstanceProfile(accountID, input) → verify profile exists → verify role exists → check Roles array is empty (LimitExceeded if role already present) → append role → update KV | 1. Add role to empty profile 2. Profile already has role (LimitExceeded) 3. Non-existent profile (NoSuchEntity) 4. Non-existent role (NoSuchEntity) |
NOT STARTED |
remove-role-from-instance-profile |
— | --instance-profile-name, --role-name |
Role must be in profile | IAMService.RemoveRoleFromInstanceProfile(accountID, input) → remove role from Roles array → update KV | 1. Remove role 2. Role not in profile (NoSuchEntity) |
NOT STARTED |
IAM - Groups
Groups provide a way to attach policies to multiple users. Stored in spinifex-iam-groups KV bucket keyed by {accountID}.{groupName}.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-group |
— | --group-name, --path |
None | IAMService.CreateGroup(accountID, input) → validate GroupName (required) → generate AGPA-prefixed GroupId → store in KV → return Group | 1. Create group 2. Duplicate name (EntityAlreadyExists) 3. Missing name (InvalidInput) |
NOT STARTED |
get-group |
— | --group-name |
Group must exist | IAMService.GetGroup(accountID, input) → reads group from KV → returns Group with member Users list | 1. Get existing group 2. Non-existent (NoSuchEntity) |
NOT STARTED |
list-groups |
— | --path-prefix, --max-items, --marker |
None | IAMService.ListGroups(accountID, input) → prefix-scans KV → returns list | 1. List all groups 2. Empty list |
NOT STARTED |
delete-group |
— | --group-name |
Group must exist, no members, no attached policies | IAMService.DeleteGroup(accountID, input) → verify no members → verify no attached policies → delete from KV | 1. Delete empty group 2. Delete group with members (DeleteConflict) 3. Delete group with policies (DeleteConflict) 4. Non-existent (NoSuchEntity) |
NOT STARTED |
add-user-to-group |
— | --group-name, --user-name |
Both must exist | IAMService.AddUserToGroup(accountID, input) → verify group + user exist → add user to group's Members list → idempotent | 1. Add user 2. Already a member (idempotent) 3. Non-existent group (NoSuchEntity) 4. Non-existent user (NoSuchEntity) |
NOT STARTED |
remove-user-from-group |
— | --group-name, --user-name |
User must be in group | IAMService.RemoveUserFromGroup(accountID, input) → remove user from Members list → update KV | 1. Remove member 2. Not a member (NoSuchEntity) |
NOT STARTED |
list-groups-for-user |
— | --user-name, --max-items, --marker |
User must exist | IAMService.ListGroupsForUser(accountID, input) → scan all groups → filter where Members contains user → return list | 1. List groups for user 2. User in no groups (empty) |
NOT STARTED |
attach-group-policy |
— | --group-name, --policy-arn |
Group and policy must exist | IAMService.AttachGroupPolicy(accountID, input) → verify policy exists → add ARN to group's AttachedPolicies → idempotent | 1. Attach policy 2. Already attached (idempotent) 3. Non-existent group (NoSuchEntity) |
NOT STARTED |
detach-group-policy |
— | --group-name, --policy-arn |
Policy must be attached | IAMService.DetachGroupPolicy(accountID, input) → remove ARN from AttachedPolicies | 1. Detach policy 2. Not attached (NoSuchEntity) |
NOT STARTED |
list-attached-group-policies |
— | --group-name, --path-prefix, --max-items, --marker |
Group must exist | IAMService.ListAttachedGroupPolicies(accountID, input) → reads group's AttachedPolicies → returns ARNs and names | 1. List attached policies 2. No policies (empty) |
NOT STARTED |
STS (Security Token Service)
STS provides temporary credentials and identity verification. GetCallerIdentity is called by every AWS SDK/CLI invocation. AssumeRole is required once IAM roles are implemented. STS needs its own gateway service endpoint (sts service in the supportedServices map) with a dedicated action map.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
get-caller-identity |
— | (no flags) | Valid SigV4 credentials | Gateway extracts accountID and username from authenticated request → returns Account, Arn (arn:aws:iam::{accountID}:user/{userName} or arn:aws:sts::{accountID}:assumed-role/{roleName}/{sessionName}), UserId. Note: a Spinifex-custom version exists at spinifex.GetCallerIdentity but needs a proper STS endpoint. |
1. Get identity for IAM user 2. Get identity for assumed role 3. Get identity for root user 4. Invalid credentials (AuthFailure) |
NOT STARTED |
assume-role |
— | --role-arn, --role-session-name, --duration-seconds, --policy, --external-id, --serial-number, --token-code, --tags, --transitive-tag-keys |
Role must exist, caller must be allowed by trust policy | Gateway validates RoleArn + RoleSessionName (required) → looks up role → evaluates trust policy (AssumeRolePolicyDocument) against caller → generates temporary credentials (AccessKeyId with ASIA prefix, SecretAccessKey, SessionToken) → credentials expire after DurationSeconds (default 3600, max 43200) → return Credentials + AssumedRoleUser (ARN, AssumedRoleId) | 1. Assume role with valid trust policy 2. Trust policy denies caller (AccessDenied) 3. Non-existent role (NoSuchEntity) 4. Expired session token rejected at SigV4 5. Custom duration 6. Session tags |
NOT STARTED |
get-session-token |
— | --duration-seconds, --serial-number, --token-code |
Valid IAM user credentials (not role) | Gateway validates caller is IAM user (not assumed role) → generates temporary credentials (ASIA prefix) → session inherits user's permissions → return Credentials | 1. Get token for IAM user 2. Duration within range 3. Reject if caller is assumed role |
NOT STARTED |
ELBv2 (Elastic Load Balancing v2 — Application & Network Load Balancer)
All ELBv2 resources are stored in the spinifex-elbv2 JetStream KV bucket. Key format: lb.{lbId}, tg.{tgId}, listener.{listenerId}. The data plane uses a system-managed LB VM running HAProxy, launched automatically during CreateLoadBalancer. HAProxy configuration is pushed via NATS (elbv2.lb.{lbId}.config topic) whenever listeners or targets change. Agent readiness is polled via elbv2.lb.{lbId}.ping.
ELBv2 - Load Balancers
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-load-balancer |
--name, --subnets, --security-groups, --scheme (internet-facing/internal), --tags, --ip-address-type (ipv4 only) |
--type (hardcoded application), --customer-owned-ipv4-pool, --dry-run |
None (subnets optional) | Gateway validates Name (required) → NATS elbv2.CreateLoadBalancer with spinifex-workers queue group → daemon validates scheme (internet-facing or internal, InvalidScheme if other) → checks duplicate name in account (DuplicateLoadBalancerName) → generates LB ID → builds DNS name {name}-{lbId}.{region}.elb.spinifex.local → if subnets provided, creates one ENI per subnet with spinifex:managed-by=elbv2 tag (rolls back on failure) → if system AMI available, launches sys.micro LB VM with cloud-init running /usr/local/bin/lb-agent → state=provisioning → background goroutine polls agent ping for ~5min → on agent ready: state=active, on timeout: state=failed → if no VM launcher: state=active immediately → persists to KV → returns LoadBalancer with ARN, DNSName, State |
1. Create with name (happy path) 2. Internal scheme 3. Invalid scheme (InvalidScheme) 4. Duplicate name (DuplicateLoadBalancerName) 5. Missing name (MissingParameter) 6. With tags 7. With subnets (ENI creation) |
DONE |
delete-load-balancer |
--load-balancer-arn |
--dry-run |
LB must exist | Gateway validates LoadBalancerArn (required) → NATS elbv2.DeleteLoadBalancer → daemon verifies LB exists and belongs to account → terminates LB VM (async background) → deletes all listeners for this LB → deletes system-managed ENIs → deletes LB record from KV → returns success |
1. Delete existing LB 2. Not found (LoadBalancerNotFound) 3. Cascades listener cleanup 4. ENI cleanup |
DONE |
describe-load-balancers |
--load-balancer-arns, --names |
--page-size, --marker, --dry-run |
None | NATS elbv2.DescribeLoadBalancers → daemon lists all LB records from KV → filters by account ID → applies ARN filter and/or Name filter → returns LoadBalancer list with State, DNSName, Scheme, VpcId, AvailabilityZones |
1. List all LBs 2. Filter by name 3. Account isolation |
DONE |
ELBv2 - Target Groups
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-target-group |
--name, --protocol (HTTP), --port (default 80), --vpc-id, --target-type (instance only), --health-check-protocol, --health-check-port, --health-check-path (default /), --health-check-interval-seconds (default 30), --health-check-timeout-seconds (default 5), --healthy-threshold-count (default 5), --unhealthy-threshold-count (default 2), --matcher (HttpCode, default 200), --tags |
--health-check-enabled, --protocol-version, --ip-address-type, --dry-run |
None | Gateway validates Name (required) → NATS elbv2.CreateTargetGroup → daemon checks duplicate name in same VPC (DuplicateTargetGroupName) → generates TG ID → creates HealthCheckConfig with provided values or defaults → stores record with empty Targets array → returns TargetGroup with ARN |
1. Create with defaults 2. Custom health check 3. Duplicate name (DuplicateTargetGroupName) 4. Missing name (MissingParameter) |
DONE |
delete-target-group |
--target-group-arn |
--dry-run |
TG must exist, not referenced by listener | Gateway validates TargetGroupArn (required) → NATS elbv2.DeleteTargetGroup → daemon verifies TG exists → scans all listeners to check if any DefaultActions reference this TG (TargetGroupInUse if referenced) → deletes from KV |
1. Delete TG 2. TG in use by listener (TargetGroupInUse) 3. Not found (TargetGroupNotFound) |
DONE |
describe-target-groups |
--target-group-arns, --names, --load-balancer-arn |
--page-size, --marker, --dry-run |
None | NATS elbv2.DescribeTargetGroups → daemon lists all TG records → filters by account → applies ARN/Name filters → if LoadBalancerArn specified, lists listeners for that LB and includes only TGs referenced by those listeners → returns TargetGroup list |
1. List all TGs 2. Filter by LB ARN (via listener lookup) 3. Account isolation |
DONE |
ELBv2 - Targets
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
register-targets |
--target-group-arn, --targets (Id=instance-id, Port=optional override) |
--dry-run |
TG must exist, instances should exist | Gateway validates TargetGroupArn + Targets (required) → NATS elbv2.RegisterTargets → daemon fetches TG → deduplicates against existing targets (key: {id}:{port}, idempotent) → resolves instance ID → private IP via VPC ENI lookup → adds target with state=initial → persists to KV → calls pushConfigForTargetGroup() to regenerate HAProxy config on all LBs using this TG → returns success |
1. Register targets 2. Idempotent re-registration 3. TG not found (TargetGroupNotFound) 4. Missing ARN or targets (MissingParameter) |
DONE |
deregister-targets |
--target-group-arn, --targets (Id, Port) |
--dry-run |
TG must exist | Gateway validates TargetGroupArn + Targets (required) → NATS elbv2.DeregisterTargets → daemon fetches TG → builds removal set (key: {id}:{port}) → filters Targets array to keep only non-matching → persists to KV → calls pushConfigForTargetGroup() to update HAProxy config → returns success |
1. Deregister targets 2. TG not found (TargetGroupNotFound) 3. Missing parameters (MissingParameter) |
DONE |
describe-target-health |
--target-group-arn, --targets (optional filter to specific targets) |
--include |
TG must exist | Gateway validates TargetGroupArn (required) → NATS elbv2.DescribeTargetHealth → daemon fetches TG → builds filter set from Targets param (if specified) → for each target, returns TargetHealthDescription with Target (Id, Port) and TargetHealth (State, Description). Note: health state is stored in records, not actively polled — HAProxy runs its own independent health checks. |
1. Describe all targets 2. Filter to specific targets 3. TG not found (TargetGroupNotFound) |
DONE |
ELBv2 - Listeners
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-listener |
--load-balancer-arn, --default-actions (Type=forward, TargetGroupArn), --protocol (HTTP, default), --port (default 80) |
--ssl-policy, --certificates, --alpn-policy, --mutual-authentication, --dry-run |
LB must exist | Gateway validates LoadBalancerArn + DefaultActions (required) → NATS elbv2.CreateListener → daemon verifies LB exists → checks for duplicate port on same LB (DuplicateListener) → generates listener ID → builds ARN → stores ListenerRecord in KV → calls pushConfig(lb) to regenerate HAProxy config (creates frontend for listener port, backend for target group with health checks) → returns Listener with ARN, Protocol, Port, DefaultActions |
1. Create listener 2. Duplicate port (DuplicateListener) 3. LB not found (LoadBalancerNotFound) 4. Graceful handling when NATS unavailable |
DONE |
delete-listener |
--listener-arn |
--dry-run |
Listener must exist | Gateway validates ListenerArn (required) → NATS elbv2.DeleteListener → daemon verifies listener exists and belongs to account → deletes from KV → fetches parent LB → calls pushConfig(lb) to regenerate HAProxy config without this listener → returns success |
1. Delete listener 2. Not found (ListenerNotFound) |
DONE |
describe-listeners |
--load-balancer-arn, --listener-arns |
--page-size, --marker, --dry-run |
None | NATS elbv2.DescribeListeners → daemon lists listeners → if LoadBalancerArn specified, filters to that LB → applies ARN filter → account isolation → returns Listener list |
1. Filter by LB ARN 2. Account isolation |
DONE |
ELBv2 - Not Implemented
| Feature | Description | Priority | Status |
|---|---|---|---|
| Listener rules | Path-based and hostname-based routing (CreateRule, DeleteRule, DescribeRules, ModifyRule) | High | NOT STARTED |
| HTTPS/TLS termination | SSL certificates, SSL policies, ALPN | High | NOT STARTED |
| Modify operations | ModifyLoadBalancerAttributes, ModifyTargetGroup, ModifyTargetGroupAttributes, ModifyListener | Medium | NOT STARTED |
| Connection draining | Deregistration delay for graceful target removal | Medium | NOT STARTED |
| Stickiness | Session affinity / sticky sessions on target groups | Medium | NOT STARTED |
| Active health checking | API-driven health state updates (currently HAProxy-only) | Medium | NOT STARTED |
| Target types | IP and Lambda target types (only instance supported) | Low | NOT STARTED |
| Access logs | S3 access log delivery | Low | NOT STARTED |
| WAF integration | AWS WAF association for web application firewall | Low | NOT STARTED |
CloudWatch (Basic Monitoring)
Basic CloudWatch metrics support is required for monitor-instances/unmonitor-instances to be meaningful. Daemon nodes should publish CPU, disk, and network metrics for running instances. Metrics stored in a time-series-friendly structure in JetStream KV or dedicated NATS subjects.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
put-metric-data |
— | --namespace, --metric-data (MetricName, Value, Unit, Timestamp, Dimensions) |
None | Gateway validates Namespace + MetricData (required) → NATS cloudwatch.PutMetricData → daemon stores metric data points in time-series KV (key: {namespace}.{metricName}.{dimensionHash}) → return success |
1. Put single metric 2. Put with dimensions (InstanceId) 3. Missing namespace (MissingParameter) |
NOT STARTED |
get-metric-statistics |
— | --namespace, --metric-name, --start-time, --end-time, --period, --statistics (Average, Sum, Min, Max, SampleCount), --dimensions |
None | Gateway validates required fields → NATS cloudwatch.GetMetricStatistics → daemon queries time-series data → aggregates by period → returns Datapoints with requested statistics |
1. Get CPU utilization over 1 hour 2. Get with dimensions (InstanceId) 3. No data returns empty |
NOT STARTED |
list-metrics |
— | --namespace, --metric-name, --dimensions, --recently-active |
None | NATS cloudwatch.ListMetrics → daemon scans metric namespaces → returns Metrics list with Namespace, MetricName, Dimensions |
1. List all metrics 2. Filter by namespace 3. Filter by metric name |
NOT STARTED |
describe-alarms |
— | --alarm-names, --alarm-name-prefix, --state-value, --action-prefix |
None | NATS cloudwatch.DescribeAlarms → daemon lists alarms from KV → return MetricAlarms with State, Threshold, ComparisonOperator |
1. List all alarms 2. Filter by state (OK, ALARM, INSUFFICIENT_DATA) |
NOT STARTED |
put-metric-alarm |
— | --alarm-name, --namespace, --metric-name, --statistic, --period, --evaluation-periods, --threshold, --comparison-operator, --alarm-actions, --dimensions |
None | NATS cloudwatch.PutMetricAlarm → daemon stores alarm config in KV → alarm evaluation runs on metric publish → state transitions trigger alarm actions |
1. Create alarm 2. Update existing alarm 3. Missing required fields |
NOT STARTED |
delete-alarms |
— | --alarm-names |
Alarms must exist | NATS cloudwatch.DeleteAlarms → daemon deletes alarm records from KV → return success |
1. Delete existing alarm 2. Non-existent alarm (ResourceNotFound) |
NOT STARTED |
Auto Scaling
Auto Scaling manages instance fleets based on scaling policies. Depends on Launch Templates (EC2) and optionally ELBv2 target groups for health-based scaling. Stored in spinifex-autoscaling KV bucket.
| Command | Implemented Flags | Missing Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|---|
create-auto-scaling-group |
— | --auto-scaling-group-name, --launch-template (Id/Name, Version), --min-size, --max-size, --desired-capacity, --vpc-zone-identifier (subnet IDs), --target-group-arns, --health-check-type (EC2/ELB), --health-check-grace-period, --tags |
Launch template must exist | NATS autoscaling.CreateAutoScalingGroup → daemon validates launch template → stores ASG record in KV → launches DesiredCapacity instances using template → registers with target groups if specified → starts health check loop → return success |
1. Create ASG with launch template 2. Scale to desired capacity 3. Missing launch template (ValidationError) 4. Min > Max (ValidationError) |
NOT STARTED |
update-auto-scaling-group |
— | --auto-scaling-group-name, --min-size, --max-size, --desired-capacity, --launch-template, --health-check-type, --health-check-grace-period |
ASG must exist | NATS autoscaling.UpdateAutoScalingGroup → daemon updates ASG config in KV → if DesiredCapacity changed, scale up/down accordingly → return success |
1. Update desired capacity 2. Update launch template 3. Non-existent ASG (ValidationError) |
NOT STARTED |
delete-auto-scaling-group |
— | --auto-scaling-group-name, --force-delete |
ASG must exist | NATS autoscaling.DeleteAutoScalingGroup → if force: terminate all instances → deregister from target groups → delete from KV → return success. Without force: reject if instances > 0 |
1. Delete empty ASG 2. Force delete with instances 3. Delete with instances (ResourceInUse) |
NOT STARTED |
describe-auto-scaling-groups |
— | --auto-scaling-group-names, --filters, --max-records |
None | NATS autoscaling.DescribeAutoScalingGroups → daemon lists ASG records → return list with Instances, DesiredCapacity, MinSize, MaxSize |
1. List all ASGs 2. Filter by name |
NOT STARTED |
set-desired-capacity |
— | --auto-scaling-group-name, --desired-capacity, --honor-cooldown |
ASG must exist | NATS autoscaling.SetDesiredCapacity → daemon updates desired count → scale up (launch) or scale down (terminate oldest) → return success |
1. Scale up 2. Scale down 3. Exceeds max (ValidationError) |
NOT STARTED |
describe-auto-scaling-instances |
— | --instance-ids, --max-records |
None | NATS autoscaling.DescribeAutoScalingInstances → daemon lists all ASG-managed instances → return list with ASG name, lifecycle state, health status |
1. List all ASG instances 2. Filter by instance ID |
NOT STARTED |
put-scaling-policy |
— | --auto-scaling-group-name, --policy-name, --policy-type (TargetTrackingScaling, StepScaling, SimpleScaling), --target-tracking-configuration, --scaling-adjustment, --cooldown |
ASG must exist | NATS autoscaling.PutScalingPolicy → daemon stores policy in KV → links to CloudWatch alarms for metric-based triggers → return PolicyARN |
1. Create target tracking policy 2. Create step scaling policy 3. Non-existent ASG (ValidationError) |
NOT STARTED |
delete-scaling-policy |
— | --auto-scaling-group-name, --policy-name |
Policy must exist | NATS autoscaling.DeleteScalingPolicy → daemon deletes policy → remove CloudWatch alarm links → return success |
1. Delete existing policy 2. Non-existent policy |
NOT STARTED |