Files
spinifex/docs/COMMANDS.md
T
Josh Siviour ab95566795 feat: initial ELBv2 implementation (#122)
* feat: add ALB target health reporting via HAProxy stats

Agent-side: periodically query HAProxy stats socket and publish
target health status to NATS (elbv2.alb.<id>.health topic).

Control-plane: healthChecker subscribes to health reports from all
ALB agents and updates target health state in the KV store. Wire
into ELBv2ServiceImpl lifecycle and DeregisterTargets cleanup.

E2E: remove health check skip — targets must now become healthy.

* feat: allocate public IP for internet-facing ALB VMs in daemon

Add Scheme field to SystemInstanceInput so the daemon knows whether
an ALB is internet-facing or internal. For internet-facing, allocate
a public IP via externalIPAM and publish vpc.add-nat for the OVN
dnat_and_snat rule. Internal ALBs skip public IP assignment entirely.

* fix: release auto-assigned public IPs on DeleteNetworkInterface

DeleteNetworkInterface previously leaked public IPs and NAT rules.
Now checks if the ENI has a public IP, verifies it's not EIP-owned,
and releases via externalIPAM + publishes vpc.delete-nat. Daemon
injects externalIPAM and EIP KV into VPC service at init time.

Tests cover: auto-assigned IP release with NAT event, EIP-owned IP
skipped, and no-op when externalIPAM is not configured.

* feat: enforce multi-AZ for internet-facing ALBs in pool mode (mulga-770)

In CreateLoadBalancer, when DevNetworking is false (pool mode), require
2+ subnets in different AZs for internet-facing scheme. Dev mode and
internal scheme skip the validation.

* feat: ALB scheme unit tests + DNS internal prefix + public IP in AZ response (mulga-771)

Add 9 unit tests covering scheme-dependent ALB behavior:
- DNS name uses internal- prefix for internal scheme (AWS-compatible)
- Internet-facing ALBs include public IP in AvailabilityZones response
- Internal ALBs have no public IP in AZ response
- Launcher receives correct scheme, delete cleans up VM + ENIs

Also implements the code changes these tests exercise:
- AvailZoneInfo.PublicIP field populated from launcher output
- lbRecordToSDK populates LoadBalancerAddresses for internet-facing

* feat: add custom AMI for ALB to images list

* feat: add internet-facing ALB data plane E2E test (mulga-772)

New E2E script that validates internet-facing ALBs get public IPs and are
reachable from a peer cluster node (external validation). Skips in dev mode.

* fix: set ENDPOINT to runner node IP in ALB E2E workflow

* fix: correct paths filter to reference alb-e2e.yml

* fix: alb ami name and workflow step to import the image

* fix: spx path

* fix: remove ALB AMI import from workflow and added to bootstrap.sh

* fix: use two subnets in different AZs for internet-facing ALB E2E

Multi-AZ enforcement (mulga-770) requires internet-facing ALBs to have
subnets in 2+ AZs in pool mode. Discover AZs and create a subnet in each.

* refactor: remove multi-AZ enforcement for internet-facing ALBs

Team decision: ALBs must work with single-AZ and single-node clusters.
Removes the multi-AZ validation, pool mode helper, associated tests,
and error constant. E2E script reverted to single subnet.

* fix: use correct account ID for ALB ENI public IP assignment

The ENI is created under the caller's account but LaunchSystemInstance
was looking it up with GlobalAccountID (system account), causing the
UpdateENIPublicIP call to fail with "ENI not found". Pass the caller's
account ID via SystemInstanceInput.AccountID.

* fix: add debug logging for ALB public IP assignment failure

* fix: expand debug logging to include peer node and more keywords

* fix: lazy system AMI discovery for ALB VMs

The daemon was caching the system AMI at startup, but the ALB image is
imported after services start (bootstrap step 9). Replace SetSystemAMI
with SetSystemAMIFunc so the AMI is resolved at request time when
CreateLoadBalancer actually needs it.

* fix: resolve ALB VM instance type dynamically from resource manager

The instance type was hardcoded to t3.nano which may not exist on all
clusters. Now the daemon provides a resolver that picks the smallest
available instance type from the resource manager.

* push interim changes

* refactor: rewrite ALB agent from NATS client to HTTP server

Replace NATS subscriptions with an HTTP server on :8405 exposing
POST /config, GET /ping, and GET /health. The daemon will push
config and poll health over HTTP through the VPC network, removing
the agent's direct NATS dependency.

- agent.go: net/http server replaces nats.Conn/Subscriptions
- health.go: remove push-based reportHealth; /health handler
  calls queryHAProxyStats synchronously on each request
- cmd/alb-agent: --nats flag replaced with --listen
- tests: rewritten with httptest instead of NATS test server
- health_checker_test: inline topic strings (HealthTopic removed)

* refactor: replace NATS with HTTP client for daemon-to-ALB agent comms

Daemon now pushes config via POST and polls ping via GET to the ALB
agent's HTTP server on the VPC IP, eliminating the agent's NATS
dependency from the daemon side.

- service_impl: pushConfig uses http.Post, waitForAgentReady uses
  http.Get, both target http://<vpcIP>:8405
- albVMUserData: drop natsURL param (agent no longer needs NATS)
- types: add VPCIP field to LoadBalancerRecord
- daemon: remove SetNATSURL call and 10.0.2.2 translation block
- arn_test: update user data tests for new signature

* refactor: rewrite ALB health checker from NATS subscription to HTTP polling

Health checker now polls each active ALB's GET /health endpoint on a
10s ticker instead of subscribing to a NATS wildcard topic. Removes
the NATS connection dependency from the health checker.

- health_checker: replace nc/sub with httpClient + stopCh polling loop
- newHealthChecker: drop nc param, add agentURLFn for testability
- service_impl: update newHealthChecker call site
- tests: rewrite with httptest, test pollAll/pollOne paths

* refactor: remove NATS wiring from ALB image and make hostfwd dev-only

- alb-setup.sh: remove --nats flag, use --listen=:8405; remove
  placeholder HAProxy frontend on 8405 (agent owns the port now);
  update description and comments
- service_impl: HostfwdPorts (80/443) now only set when DevNetworking
  is enabled — production ALBs are reached via VPC IP directly

* feat: system credentials file + daemon wiring for ALB agent SigV4 auth

Write config/system-credentials.json during admin init/join with the
system access key. Daemon reads it at startup, resolves the gateway URL
from external IPAM (or 10.0.2.2 in dev mode), and wires both into the
ELBv2 service. albVMUserData now injects ALB_GATEWAY_URL, ALB_ACCESS_KEY,
and ALB_SECRET_KEY into cloud-init for ALB VMs.

* feat: ALB agent gateway actions + daemon handlers (heartbeat, config)

Add ALBAgentHeartbeat and GetALBConfig actions to the gateway so the ALB
agent can communicate via the AWS gateway instead of direct HTTP.

- Add agent_types.go with input/output types for both actions
- Add ALBAgentHeartbeat: processes heartbeat, transitions provisioning→active
  on first contact, updates LastHeartbeat, processes health report
- Add GetALBConfig: returns pre-computed HAProxy config text and hash
- Replace pushConfig (HTTP POST to agent) with updateStoredConfig (stores
  config + SHA256 hash on LB record for agent pull)
- Remove waitForAgentReady polling goroutine (heartbeat handles transition)
- Remove HTTP client from ELBv2ServiceImpl (no longer needed)
- Strip health checker HTTP polling (pollLoop/pollAll/pollOne) — health
  reports now arrive directly via ALBAgentHeartbeat
- Add ConfigText, ConfigHash, LastHeartbeat fields to LoadBalancerRecord
- Wire NATS subscriptions + daemon handlers for both new actions
- Add gateway handler files and elbv2Actions map entries
- Comprehensive tests for heartbeat, config, and updateStoredConfig

* feat: rewrite ALB agent as AWS SDK HTTP client (Bead 3r)

Replace the HTTP server agent with a poll-based client that communicates
with the gateway using SigV4-signed requests. The agent now sends
ALBAgentHeartbeat every 5s with health reports and fetches config via
GetALBConfig when the config hash changes.

* feat: add VPC service route for ALB agent gateway access

ALB VMs need to reach the gateway for heartbeat/config polling, but
VPCs without an IGW have no external routing. Instead of requiring
an IGW, the daemon now publishes a vpc.ensure-service-route event
before launching ALB VMs. vpcd handles this by creating minimal
external connectivity (external switch, localnet port, gateway router
port, chassis bindings) if not already present, then adds a /32 static
route for the gateway IP.

Refactors handleIGWAttach to share the external connectivity setup via
ensureExternalConnectivity helper.

* Revert "feat: add VPC service route for ALB agent gateway access"

This reverts commit d4220d8806.

* feat: add management NIC infrastructure for system instance control plane

Add config, IP allocator, TAP helpers, and VM struct fields for the
management NIC that gives system instances (ALBs etc.) L3 access to the
host control plane via br-mgmt, independent of VPC routing.

- DaemonConfig.MgmtBridge (default "br-mgmt")
- MgmtIPAllocator: thread-safe static IP allocation (.10-.249), rebuild from VMs on restart
- generateMgmtMAC (02:a0:00 prefix), MgmtTapName, GetBridgeIPv4
- SetupMgmtTapDevice / CleanupMgmtTapDevice (plain Linux bridge TAP)
- VM struct: MgmtMAC, MgmtIP, MgmtTap fields

* feat: wire management NIC into QEMU launch and cloud-init config

Add QEMU NIC injection for mgmt TAP (virtio-net-pci on br-mgmt),
extend generateNetworkConfig to emit mgmt0 stanza with static IP,
and wire mgmt MAC/IP allocation into LaunchSystemInstance. Includes
mgmt TAP cleanup on instance stop and failed launch cleanup.

* test: add coverage for ELBv2 service lifecycle and setter methods

* fix: move br-mgmt detection before wireALBAgentConfig so gateway URL uses management bridge IP

* fix: generate per-interface cloud-init config when eniMAC present without devMAC

The guard in generateNetworkConfig required both eniMAC and devMAC to
produce per-interface config, falling back to wildcard DHCP otherwise.
System instances (ALBs) have eniMAC + mgmtMAC but no devMAC (DevNetworking
is independent), so the mgmt NIC stanza was never emitted. The wildcard
config does DHCP on all NICs, but br-mgmt has no DHCP server, leaving the
management NIC unconfigured and the agent unable to reach the gateway.

* fix: add wildcard support to ENI description filter + add ca-certificates to ALB image

- ENI DescribeNetworkInterfaces description filter did exact string match,
  but AWS supports * wildcards. E2E test uses "ELB *" which didn't match
  "ELB app/my-alb/lb-123". Added matchFilter helper for wildcard matching.
- ALB image was missing ca-certificates package, so cloud-init ca_certs
  module couldn't add the Spinifex CA to the system trust store. Agent
  TLS handshakes failed with "certificate signed by unknown authority".
- Fixed data race in TestDeleteLoadBalancer_TerminatesVM_WithPublicIP
  where async TerminateSystemInstance goroutine wrote to mock fields
  while the test read them.

* fix: detach ENI before delete on termination + defer ENI cleanup to stopInstance

DeleteLoadBalancer was trying to delete ENIs synchronously, but they were
still attached (in-use) to the running VM. The delete was rejected with
InvalidNetworkInterface.InUse.

Fix: DeleteLoadBalancer only deletes ENIs when no VM was launched (InstanceID
empty). When a VM exists, ENI cleanup is deferred to stopInstance which runs
after QEMU exits. stopInstance now calls DetachENI before DeleteNetworkInterface
to clear the in-use status.

E2E test now polls for ENI cleanup (up to 30s) since VM termination is async.

* fix: detach then delete ENIs in DeleteLoadBalancer

ENIs are attached (in-use) to the ALB VM, so DeleteNetworkInterface
rejects them. Fix: call DetachENI first to clear in-use status, then
delete. stopInstance tolerates already-deleted ENIs.

* test: add internal ALB dataplane E2E test

Add run-elbv2-dataplane-internal-e2e.sh replacing the generic dataplane
test. Internet-facing coverage is already provided by
run-elbv2-dataplane-internet-facing-e2e.sh.

The internal test validates:
- ALB reaches active via mgmt NIC heartbeat (no IGW dependency)
- Scheme=internal, no public IP, internal- DNS prefix
- Health checks pass via VPC L2 (OVN logical switch)
- Round-robin traffic from a VPC client to the ALB's private IP
- ENI cleanup on ALB deletion

Uses a client VM inside the VPC that curls the ALB private IP and
serves results via HTTP on its public IP for host-side verification.

* feat: add NLB constants and data model changes (mulga-838)

* feat: type-aware validation in CreateLoadBalancer (mulga-839)

Accept Type=network for NLB, reject SecurityGroups for NLB, set
CrossZoneEnabled defaults (false for NLB, true for ALB), and use
/net/ ARN path segment for network load balancers.

* feat: type-aware validation in CreateTargetGroup (mulga-840)

Accept TCP/UDP/TLS/TCP_UDP protocols and apply NLB health check
defaults (TCP protocol, no path/matcher, 10s timeout, 3/3 thresholds).
Reject invalid protocols with InvalidParameterValue.

* feat: type-aware validation in CreateListener (mulga-841)

* feat: add NLB listener and protocol compatibility unit tests

Cover the remaining Phase 1 test gaps: NLB CreateListener with all L4
protocols, protocol rejection for wrong LB type, listener-to-target-group
protocol compatibility validation, and default protocol rejection on NLB.

* feat: add TCP HAProxy config generation for NLB

Add TCP mode template and generateNLBConfig function for Network Load
Balancer support. GenerateHAProxyConfig now branches on LB type,
producing mode-tcp configs for NLBs with tcp-check or httpchk health
checks depending on the target group's health check protocol.

* refactor: rename albagent to lbagent for NLB generalization

Rename the ALB-specific agent package and binary to generic LB names
in preparation for NLB support. Both ALB and NLB VMs use the same
agent — it just applies whatever HAProxy config the control plane
generates.

- Rename albagent/ → lbagent/, cmd/alb-agent/ → cmd/lb-agent/
- Rename types: ALBAgentHeartbeat* → LBAgentHeartbeat*, GetALBConfig* → GetLBConfig*
- Rename NATS topics: elbv2.ALBAgentHeartbeat → elbv2.LBAgentHeartbeat
- Rename cloud-init user data: albVMUserData → lbVMUserData with LB_* env vars
- Rename Makefile targets: build-alb-agent → build-lb-agent
- Update gateway action map, daemon subscriptions, service interface
- Fix pre-existing unused matchFilter function lint error
- Fix awserrors test count for InvalidConfigurationRequest

* feat: add sys.micro system instance type for LB VMs (mulga-845)

Add sys.micro (1 vCPU, 128MB) as an internal-only instance type for
system VMs (load balancers, NAT gateways). Registered in the type map
for allocation but excluded from DescribeInstanceTypes, NATS
subscriptions, and resource stats. Daemon SetSystemInstanceTypeFunc
now returns sys.micro instead of scanning for smallest guest type.

* feat: LB image shrink and rename (mulga-846)

Rename alb.conf → lb.conf, alb-setup.sh → lb-setup.sh with IMAGE_SIZE
reduced from 512M to 192M. Update all references to use lb-agent binary
and LB_* env vars. Update COMMANDS.md for NLB support.

* chore: rename file

* feat: NLB E2E test and bug fixes (mulga-847)

Add full NLB data plane E2E test (33 assertions, 9 phases):
VPC/subnet setup, TCP target group with health checks, NLB
creation (type=network), TCP listener, agent heartbeat,
target health verification, TCP traffic round-robin, target
deregistration, and full cleanup verification.

Bug fixes found during E2E testing:
- Fix HAProxy stats socket path mismatch: config used alb-/nlb-
  prefix but agent expects lb- prefix, preventing health reports
- Fix system AMI lookup to match new -lb- image name, falling
  back to legacy -alb- name
- Bump LB image size from 192M to 256M (Alpine base is 200M)

* test: validate NLB internal scheme (mulga-857)

NLB with scheme=internal works out of the box — shared code paths
with ALB handle it correctly. Add unit test and E2E coverage:

- Unit test: NLB internal creates with no public IP, internal- DNS prefix
- E2E phases 10-14: internal NLB lifecycle with VPC client TCP traffic
- E2E passed 56/56 with perfect round-robin (20/20 requests)

* feat: consolidate LB e2e tests into run-lb-e2e.sh (mulga-854)

Replace 3 separate LB dataplane scripts with a single consolidated test
that covers all 4 variants (ALB/NLB × internet-facing/internal).

- Share one VPC, subnet, and 2 dual-purpose app instances across all suites
- ALB + NLB provision in parallel within each phase (internet-facing / internal)
- One client VM tests both internal LBs simultaneously
- Replace sleep 100 with LB health check gates
- Delete standalone alb-e2e.yml, integrate into e2e.yml for both single and multi-node

* chore: disable LB E2E tests in CI until dev env issue resolved

* fix: read LB agent credentials from spinifex.toml instead of system-credentials.json

The daemon couldn't read system-credentials.json due to systemd
hardening (file owned by root:root 0600, not in ReadOnlyPaths).
Use the same system credentials already loaded from spinifex.toml
via the predastore config section — these are identical keys written
by admin init.

* feat: automate LB image build and import (mulga-861)

- dev-install.sh: build lb-agent and run build-system-image.sh --import
  after services start so the LB image is available out of the box
- build-system-image.sh: add --quiet flag to suppress build output,
  remove hardcoded --config path (uses auto-detected default)
- images.go: rename alb-alpine → lb-alpine to match current naming

* update go version

* add LB e2e tests

* fix: clean LB image build dir and wait for nbd device readiness

Previous CI runs can leave stale state in /tmp/lb-image-build/ causing
e2fsck failures. Clean the build dir before each CI build. Also add a
readiness check after qemu-nbd connect to avoid race conditions.

* fix: serialize nbd access with flock for concurrent CI builds

Single-node and multi-node CI jobs share the same host and both build
the LB system image using /dev/nbd0. Use flock to serialize the nbd
connect/mount/customize/disconnect section so concurrent builds wait
rather than collide.

* fix: move flock to cover entire image build, not just nbd section

Concurrent CI jobs share BUILD_DIR and /dev/nbd0. The lock must be
acquired before downloading/copying the image, not just before nbd
connect, otherwise the second job can clobber the first job's files.

* fix: skip LB image build if output is fresh from concurrent CI job

When concurrent CI jobs build the same image, the second job now reuses
the first's output if it's less than 10 minutes old, avoiding a
duplicate download + build cycle.

* fix: filter AMI discovery by name in E2E tests

With the LB Alpine image now imported alongside Ubuntu, the unfiltered
describe-images call picked up the wrong AMI. Filter by the expected
image name so tests use the Ubuntu image for instance lifecycle.

* fix: filter AMI discovery by name in multi-node E2E tests

Same fix as run-e2e.sh — with the LB Alpine image now imported,
the unfiltered describe-images picks the wrong AMI.

* fix: use awsgw bind IP as LB agent gateway URL fallback

When br-mgmt is not configured and dev_networking is off (production
with external pool), the gateway URL was falling through to the
external IPAM pool's GatewayIP — the WAN router, not the Spinifex
node. LB VMs couldn't reach the awsgw, so heartbeats never arrived
and ALBs stayed stuck in provisioning.

Use the awsgw bind IP instead — LB VMs on the OVN overlay can reach
the host node's WAN IP where the gateway listens.

* fix: capture StateReason and daemon logs when LB app instances fail

Instances with user-data terminate immediately in CI. Add
describe-instances StateReason query and daemon log dump to diagnose
whether it's volume_preparation_failed or launch_failed.

* fix: prevent orphaned AMI from breaking LB E2E and avoid stuck provisioning

Three fixes for CI E2E failures:

1. run-e2e.sh: Remove custom AMI from predastore after deleting its
   backing snapshot. Previously the AMI remained discoverable but
   unusable, causing run-lb-e2e.sh to pick it and fail with NoSuchKey
   on the deleted snapshot.

2. run-lb-e2e.sh: Prefer ubuntu AMI by name for app instances instead
   of picking any non-alpine image. Also improve debug log dump to use
   journalctl (binary-install CI uses systemd, not log files).

3. CreateLoadBalancer: Only set provisioning state when a system VM
   was actually launched. Previously a failed launch still set
   provisioning, leaving the LB stuck forever with no agent to
   send heartbeats.

* fix: use AWSGW bind IP as LB agent gateway URL in multi-node

In multi-node deployments, the AWSGW binds to the WAN IP (e.g.
192.168.1.26:9999), but wireLBAgentConfig was using the br-mgmt IP as
the gateway URL. Since the AWSGW doesn't listen on br-mgmt, the LB
agent's heartbeat was refused, leaving internet-facing ALBs stuck in
provisioning.

Internet-facing LBs now use the AWSGW bind IP directly — the VM reaches
it via VPC/external networking. Internal LBs get a host route on the
mgmt NIC so they can reach the AWSGW bind IP through br-mgmt.

* fix: add mgmt route via bootcmd for multi-node internal LBs

Alpine cloud-init does not support v2 routes under ethernets — adding
a routes: block to the network-config breaks all networking in the VM.

Instead, add the host route via a bootcmd in the LB user data. This
runs early in cloud-init and adds `ip route add {awsgw}/32 via
{br-mgmt}` so internal LB agents can reach the AWSGW through the
management NIC when the gateway binds to a specific WAN IP.

Also moves the networking service from boot to default runlevel in the
LB Alpine image to prevent a race with cloud-init.

Verified locally: internal ALB transitions to active with AWSGW bound
to WAN IP (simulating multi-node).

* add LB enable flag to e2e

* test: add elbv2 tests to meet diff coverage threshold

Cover health checker edge cases (invalid JSON, empty servers, zero
thresholds, port-0 fallback) and service_impl paths (mgmt route
cloud-init, protocol validation, missing target group).

* fix: swallowed health report error + needless marshal round-trip in LBAgentHeartbeat

Add handleHealthReportDirect(report) to healthChecker so LBAgentHeartbeat
can pass the struct directly instead of marshaling to JSON and immediately
unmarshaling. Promote handleHealthReport unmarshal error from Debug to Warn
so failures are visible in logs.

* fix: validate credentials before launching LB VM + set failed state on launch error (mulga-867)

Guard lbVMUserData with empty-value check for gatewayURL, systemAccessKey,
and systemSecretKey — blank env vars previously caused agent auth-loop with
no daemon-side log. Set LB state to failed (not active) when VM launch fails
so the API reflects the broken data-plane instead of black-holing traffic.

* fix: data race in getSystemAMI/getSystemInstanceType lazy caching

Use sync.Once to guard lazy resolution of systemAMI and
systemInstanceType fields. Concurrent CreateLoadBalancer calls
previously raced on unsynchronized read+write of these cached values.
The setter methods reset the Once so re-resolution works when the
func is set after initial construction.

Also fix pre-existing nosprintfhostport lint in daemon.go.

* fix: use utils.PublishEvent for NAT events + return errors from isEIPOwned

publishNATEvent in eni.go and daemon was copy-pasted boilerplate — now
delegates to utils.PublishEvent (matching the EIP service pattern).

isEIPOwned previously swallowed all KV errors and returned false, which
could release an EIP-owned public IP if the store was temporarily
unavailable. Now returns (bool, error) so the caller skips release on
KV failure instead of assuming no EIP ownership.

* fix: narrow health checker to only scan target groups for reporting LB

handleHealthReportDirect was calling ListTargetGroups() on every heartbeat
(every 5s per LB agent), scanning all TGs. Now uses report.LBID to follow
LB → listeners → TG ARNs and fetch only the attached target groups via
direct key lookups. Falls back to full scan if LBID is missing.

* fix: pass region through to LB agent SigV4 signing (mulga-871)

LB agent hardcoded us-east-1 as the SigV4 signing region. Thread region
from ELBv2 service config → cloud-init LB_REGION env var → lb-agent
--region flag → lbagent.New() → signer.Sign() call.

* fix: add account-level authorization to LBAgentHeartbeat and GetLBConfig

Both endpoints accepted any accountID without verifying the LB belongs
to that account. Now verify accountID matches the LB's AccountID,
with an exception for the system account (000000000000) which the
ALB agent uses for heartbeats and config fetches.

* fix: upgrade silent failure paths from Debug to Warn/Error

- healthChecker target group list failure: Debug → Warn
- HAProxy stats socket error in agent tick(): Debug → Warn
- ENI delete EIP-owned skip: Debug → Info (operational visibility)
- wireLBAgentConfig: warn when no gateway host is resolved
- detectMgmtIP: warn when bridge exists but IP is empty
- stale mgmt TAP cleanup: log errors instead of discarding

* refactor: deduplicate MAC hash function into utils.HashMAC

Three identical hash-to-MAC implementations (generateDevMAC, generateMgmtMAC,
generateENIMac) now delegate to a shared utils.HashMAC(prefix, id) helper.

* fix: guard LBAgentHeartbeat KV writes to state transitions or stale timestamps

Previously the full LoadBalancerRecord (including ConfigText) was written
to JetStream KV on every heartbeat (every 5s). Now only persists on state
transitions (provisioning→active) or when LastHeartbeat is older than 60s.

* fix: fail ALB launch when public IP allocation fails for internet-facing scheme

Previously, IPAM allocation failure was logged as a warning and the ALB
VM continued launching without a public IP. The ALB reached active state
but was unreachable from the internet. Now the launch returns an error
and cleans up the partially-created instance, causing the ALB to enter
the failed state.

* fix: retry system AMI resolution when image not yet imported

sync.Once cached empty results permanently — if the daemon started
before the system image was imported, all CreateLoadBalancer calls
silently skipped VM creation. Replace sync.Once with a mutex that
only caches non-empty results, allowing retry on subsequent calls.

Same fix applied to getSystemInstanceType for consistency.

* fix: propagate updateStoredConfig errors to API callers (mulga-884)

updateStoredConfig and updateStoredConfigForTargetGroup now return
errors instead of silently swallowing them. Callers (CreateListener,
DeleteListener, RegisterTargets, DeregisterTargets) return
ErrorServerInternal when config persistence fails, preventing the API
from reporting success while HAProxy serves stale configuration.

* test: add unit tests for wireLBAgentConfig, isEIPOwned, toHealthReport (mulga-885)

* fix: parse HAProxy templates once at package level (mulga-875)

generateALBConfig and generateNLBConfig were calling template.New().Parse()
on every invocation. Move to package-level template.Must for one-time parsing.

* fix: fail launch for internal-scheme ALBs when mgmt NIC setup fails (mulga-886)

When management TAP or IP allocation fails, internal-scheme ALBs now
return an error instead of silently continuing without a management NIC.
Without the mgmt NIC, the LB agent cannot reach the gateway and the ALB
gets stuck in provisioning forever. Internet-facing ALBs retain the
existing degrade-and-continue behavior.

* refactor: extract lbArnPathSegment helper for ALB/NLB ARN construction (mulga-877)

The app/net path segment derivation was duplicated in buildLBArn,
buildListenerArn, and inline in CreateLoadBalancer. Extracted into a
single lbArnPathSegment helper.

* refactor: use aws.StringValue in toHealthReport (mulga-878)

Replace manual nil-check-then-deref with aws.StringValue for cleaner
pointer-to-value conversion in the heartbeat health report builder.

* refactor: use testutil.StartTestJetStream in health_checker_test.go (mulga-876)

Replace hand-rolled setupTestNATS (~20 lines) with the shared
testutil.StartTestJetStream helper. Simplify return to just *Store
since no caller needed the *nats.Conn.

* fix: address PR #122 review comments (mulga-887)

- Extract NATEvent named type from anonymous struct in publishNATEvent
- Remove unused build-lb-image.sh thin wrapper
- Remove SPX_ENABLE_LB env var from e2e workflow (unused)
- Remove no-op healthChecker start/stop methods
- Export simple ELBv2 fields, remove trivial setter methods
- Delete old e2e scripts replaced by consolidated run-lb-e2e.sh

* fix: check specific error type for ENI deletion on termination (mulga-888)

ENI deletion errors were all silently downgraded to Debug level,
masking genuine failures (KV issues, permissions, ENI stuck in-use)
that cause orphaned ENIs and leaked IPAM addresses. Now only
NotFound is logged at Debug; all other errors log at Error.

* fix: propagate non-ENODEV errors from GetBridgeIPv4 (mulga-890)

GetBridgeIPv4 returned ("", nil) for ALL InterfaceByName failures,
not just missing devices. Permission errors on /sys/class/net/ were
silently treated as "bridge absent". Now only "no such network
interface" returns nil; other errors propagate to the caller.

* fix: remove backend address from proxy error response (mulga-892)

The reverse proxy error handler included the internal backend
host:port in the XML error body sent to clients. The address is
already logged server-side; the client now sees a generic message.

* fix: release mutex before KV store I/O in health checker (mulga-893)

handleHealthReportDirect held hc.mu across PutTargetGroup calls
(JetStream KV Put — network I/O), serializing concurrent heartbeats
from independent LBs. Now collects changed TGs under the lock and
persists them after releasing it.

* fix: warn when system credentials missing in wireLBAgentConfig (mulga-894)

fix: log DetachENI errors instead of discarding with blank identifier (mulga-895)

* fix: detect veth bridge mode and use centralized NAT for it

The veth-pair bridge topology (Linux bridge ↔ veth ↔ OVS bridge) was
misclassified as direct bridge mode, causing OVN to use distributed NAT.
Distributed NAT hairpin routing fails through the Linux bridge intermediary,
breaking host-to-local-LB connectivity.

Add BridgeModeVeth constant, detect veth-wan-ovs interface in
detectBridgeMode(), and introduce useCentralizedNAT() to gate NAT mode
and nat-addresses=router for both macvlan and veth modes. isMacvlanMode()
is preserved for macvlan-specific behavior (MAC alignment).

* fix: add retry loop for peer NLB reachability test

With centralized NAT, ARP resolution for NAT external IPs relies on
gratuitous ARPs from the gateway chassis. These take time to propagate
to peer nodes. The peer→NLB test had a single 3s attempt while all
other connectivity tests use retry loops.

* fix: use request-reply for vpc.add-nat to prevent ARP propagation races

publishNATEvent was fire-and-forget for vpc.add-nat, meaning the daemon
continued before vpcd committed the OVN NAT rule. This caused intermittent
NLB connectivity failures: the LB was reported active before OVN had
the DNAT rule and sent GARPs for the public IP.

Switch vpc.add-nat to request-reply (RequestEvent) so the daemon waits
for OVN confirmation. Also increase nc timeout from 2-3s to 5s in E2E
tests for more reliable TCP probing during ARP resolution.

* fix: clean up stale NAT rules before adding new ones for reused IPs

When a public IP is returned to the external pool and quickly reallocated
(e.g. instance terminated, then LB created), the fire-and-forget
vpc.delete-nat may not have been processed yet. handleAddNAT now removes
any existing dnat_and_snat rule for the same external IP before creating
the new one, preventing duplicate/conflicting OVN NAT rules.

This was causing intermittent NLB connectivity failures in multi-node E2E
where NLB public IPs collided with recently-freed instance IPs.

* fix: clean up stale NAT rules across all VPC routers on IP reuse

The previous fix (0a93081) only removed stale dnat_and_snat rules from
the target VPC's router. When a public IP is freed from one VPC and
reallocated to a different VPC before the fire-and-forget vpc.delete-nat
is processed, the stale rule on the old VPC's router caused duplicate
OVN ARP responses — traffic was black-holed when the host resolved to
the wrong router's MAC.

DeleteAllNATsByExternalIP now searches every logical router for stale
NAT rules matching the external IP and removes them in one transaction.

Also adds host connectivity diagnostics (ARP, routing, OVN NAT state)
to run-lb-e2e.sh so the next CI failure surfaces the root cause of the
persistent NLB-from-host issue.

* fix: run LB inet tests from actual hosting node, not hardcoded node1

The host/peer connectivity tests were hardcoded: "host" = node1 (where
the test script runs), "peer" = node2. But LB VMs land on whichever
node picks up the NATS queue message, so the tests were randomly testing
local vs remote connectivity depending on placement luck.

Now the test:
1. Accepts --nodes <ip1> <ip2> ... (all cluster node IPs)
2. After LB creation, discovers which node hosts each LB VM by checking
   for the ENI's tap device
3. Runs "local" tests from the hosting node and "remote" tests from a
   different node

Also updates e2e.yml to pass --nodes instead of --peer.
2026-04-10 10:27:20 +10:00

141 KiB
Raw Blame History

Command Implementation Matrix

Spinifex Admin CLI

Platform management commands not exposed via the AWS gateway API. These are CLI-only commands.

Service Management

Service lifecycle commands for starting, stopping, and checking status of all Spinifex cluster services. Each service subcommand supports start, stop, and status operations.

Command Flags Prerequisites Basic Logic Status
spx service predastore start --port (default: 8443), --host (default: 0.0.0.0), --base-path, --config-path, --debug, --tls-cert, --tls-key, --backend (distributed/filesystem, default: distributed), --node-id (default: 0), --pprof, --pprof-output TLS cert/key, base-path, config-path required Creates predastore service instance with S3-compatible storage backend → starts service DONE
spx service predastore stop — Predastore must be running Stops the predastore service DONE
spx service predastore status — None Reports predastore service status DONE
spx service viperblock start --s3-host (default: 0.0.0.0:8443), --s3-bucket (default: predastore), --s3-region (default: ap-southeast-2), --plugin-path (default: /opt/spinifex/lib/nbdkit-viperblock-plugin.so) --config required, plugin-path must exist on disk Loads cluster config → connects to NATS and Predastore → starts viperblock block storage service with NBD plugin DONE
spx service viperblock stop — Viperblock must be running Stops the viperblock service DONE
spx service viperblock status — None Reports viperblock service status DONE
spx service nats start --port (default: 4222), --host (default: 0.0.0.0), --debug, --data-dir, --jetstream None Starts embedded NATS server with optional JetStream DONE
spx service nats stop — NATS must be running Stops the NATS service DONE
spx service nats status — None Reports NATS service status DONE
spx service spinifex start --wal-dir --config required Loads cluster config → starts spinifex daemon (VM orchestration, NATS subscriptions, health endpoint) DONE
spx service spinifex stop — Spinifex daemon must be running Stops the spinifex daemon service DONE
spx service spinifex status — None Reports spinifex daemon service status DONE
spx service awsgw start --host (default: 0.0.0.0:9999), --tls-cert, --tls-key, --debug --config required Loads cluster config → starts AWS-compatible gateway with SigV4 auth, IAM policy enforcement, TLS DONE
spx service awsgw stop — AWS gateway must be running Stops the AWS gateway service DONE
spx service awsgw status — None Reports AWS gateway service status DONE
spx service spinifex-ui start --port (default: 3000), --host (default: 0.0.0.0), --tls-cert, --tls-key None Starts embedded web UI server serving the React frontend. Aliases: ui, spinifexui DONE
spx service spinifex-ui stop — spinifex-ui must be running Stops the spinifex-ui service DONE
spx service spinifex-ui status — None Reports spinifex-ui service status DONE
spx service vpcd start — --config required, OVN/OVS installed Loads cluster config → starts VPC daemon (subscribes to vpc.* NATS events, translates to OVN logical switches/ports/routers) DONE
spx service vpcd stop — vpcd must be running Stops the vpcd service DONE
spx service vpcd status — None Reports vpcd service status DONE

Cluster Inspection

Operational commands for inspecting cluster state. These fan out NATS requests to all nodes and aggregate responses.

Command Flags Prerequisites Basic Logic Output Columns Status
spx get nodes --timeout (default: 3s) Cluster must be running (NATS) Loads config → publishes to spinifex.node.status fan-out topic → collects responses within timeout → merges config-known nodes with NATS responders → nodes that don't respond shown as NotReady NAME, STATUS, IP, REGION, AZ, UPTIME, VMs, SERVICES DONE
spx get vms --timeout (default: 3s) Cluster must be running (NATS) Publishes to spinifex.node.vms fan-out topic → collects VM info from all nodes → sorts by node then instance ID → prints table. Alias: spx get instances INSTANCE, STATUS, TYPE, VCPU, MEM, NODE, IP, AGE DONE

Resource Monitoring

Command Flags Prerequisites Basic Logic Output Status
spx top nodes --timeout (default: 3s) Cluster must be running (NATS) Publishes to spinifex.node.status fan-out topic → collects CPU/memory usage per node → aggregates instance type capacity across all nodes → prints two tables: per-node resource usage and cluster-wide instance type availability Table 1: NAME, CPU (used/total), MEM (used/total), VMs. Table 2: INSTANCE TYPE, AVAILABLE, VCPU, MEMORY DONE

Cluster Initialization

Command Flags Prerequisites Basic Logic Test Cases Status
spx admin init --nodes, --node, --bind, --port, --region, --az, --cluster-name, --cluster-bind, --cluster-routes, --predastore-nodes, --services, --formation-timeout, --force None (first-time setup) Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI spx profile → creates directory structure under ~/spinifex/ 1. Init creates all config files
2. Root credentials printed once
3. master.key is 32 bytes, mode 0600
4. bootstrap.json consumed on first start
5. --force re-initializes existing config
6. AWS CLI profile spx auto-configured
DONE
spx admin join --host (required), --node (required), --bind, --port, --region, --az, --cluster-bind, --cluster-routes, --data-dir, --services Leader node must be running Connects to leader node → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations 1. Join existing cluster
2. Missing host (error)
3. Missing node name (error)
DONE

Version

Command Flags Prerequisites Basic Logic Status
spx version — None Prints Spinifex version, commit hash, OS, and architecture (populated via build-time ldflags) DONE

Cluster Operations

Command Flags Prerequisites Basic Logic Test Cases Status
spx admin cluster shutdown --force (shutdown even if nodes don't respond), --timeout (max wait per phase, default 120s), --dry-run (print phase plan without executing) Cluster must be running Performs coordinated, phased shutdown of entire cluster. Phases execute in order: GATE (stop API/UI) → DRAIN (stop VMs) → STORAGE (stop viperblock) → PERSIST (stop predastore) → INFRA (stop NATS/daemon). Each phase waits for all nodes to ACK before proceeding. Uses JetStream state tracking. 1. Shutdown running cluster
2. All nodes stop cleanly
3. Force shutdown with unresponsive nodes
4. Dry-run prints plan
DONE

Certificate Management

Command Flags Prerequisites Basic Logic Test Cases Status
spx admin cert renew --extra-ip (additional IPs for SANs), --extra-dns (additional DNS names for SANs) Existing CA from spx admin init Reads existing CA → regenerates server certificate with all current network interface IPs and machine hostname in SANs → writes new cert. Use after adding a new network interface or changing IP addresses. 1. Renew with auto-detected IPs
2. Renew with extra IPs
3. Renew with extra DNS names
DONE

Account Management

Command Flags Prerequisites Basic Logic Test Cases Status
spx admin account create --name Cluster must be running (NATS), master key must exist Connects to NATS → CAS loop on spinifex-account-counter:next_id for sequential 12-digit ID → creates Account record in spinifex-accounts KV → creates admin user in new account → creates access key for admin → creates AdministratorAccess policy (Action:, Resource:) → attaches policy → prints credentials 1. Create first account (ID 000000000001)
2. Sequential IDs (000000000002, etc.)
3. Admin user has full access
4. Credentials printed once
DONE
spx admin account list — Cluster must be running (NATS), master key must exist Connects to NATS → IAMService.ListAccounts() → prints table with Account ID, Name, Status, Created 1. List all accounts
2. Empty list (only global account)
DONE

Image Management

Command Flags Prerequisites Basic Logic Test Cases Status
spx admin images import --name Cluster must be running, image file must exist locally Imports a local disk image (QCOW2) as an AMI → uploads to Predastore S3 → registers as available AMI 1. Import valid QCOW2 image
2. Image appears in describe-images
DONE
spx admin images list — None Lists available OS images that can be imported or downloaded 1. List available images DONE

AWS Commands

EC2 - Instance Management

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
run-instances --image-id, --instance-type, --count (Min/MaxCount), --key-name, --user-data, --subnet-id (auto-creates ENI, assigns private IP), --block-device-mappings (DeviceName, VolumeSize, VolumeType, Iops, DeleteOnTermination), --placement (GroupName only — routes via spread or cluster strategy) --security-group-ids, --tag-specifications, --dry-run, --client-token, --disable-api-termination, --ebs-optimized, --iam-instance-profile, --network-interfaces, --private-ip-address, --monitoring, --credit-specification, --cpu-options, --metadata-options, --launch-template, --hibernate-options describe-images (AMI must exist), create-key-pair (optional), VPC/SG (optional) Gateway parses AWS query → if Placement.GroupName set, looks up strategy: spread → distributeInstancesSpread() (1 instance per node, atomic CAS reservation), cluster → distributeInstancesCluster() (pin all to single node); otherwise NATS ec2.runinstances → daemon creates QEMU/KVM VM with viperblock-backed root volume via NBD → if SubnetId provided, auto-creates ENI with private IP → cloud-init injects user-data/keys → on termination, removes instance from placement group → returns reservation with instance ID 1. Launch with valid AMI and key pair
2. Launch with invalid AMI ID (error)
3. Launch with block device mappings (custom volume size)
4. Launch multiple instances (MinCount/MaxCount)
5. Launch with subnet-id (auto-creates ENI)
6. Invalid instance type returns error
7. Launch with spread placement group (1 per node)
8. Launch with cluster placement group (all on one node)
9. Insufficient capacity for placement group (error)
DONE
describe-instances --instance-ids, --filters (instance-state-name, instance-id, instance-type, vpc-id, subnet-id, tag:*, tag-key, tag-value) --max-results, --next-token, --dry-run None Gateway fans out NATS ec2.DescribeInstances to all nodes (no queue group) → each daemon returns local instances → gateway aggregates and returns combined list. Filters applied per-node before aggregation (reduces payload). Also applies to stopped/terminated instances via describeInstancesFromKV(). 1. Describe all instances (no filter)
2. Describe by instance ID
3. Describe with filters (e.g. instance-state-name)
4. Instance not found returns empty set
5. Multi-node aggregation returns instances from all nodes
6. Filter by tag
7. Unknown filter returns InvalidParameterValue
DONE
start-instances --instance-ids --dry-run, --force run-instances (instance must exist in stopped state) Gateway sends NATS ec2.cmd.{instance-id} → daemon restarts stopped QEMU process with same config → state transitions stopped→pending→running 1. Start a stopped instance
2. Start already-running instance (error: IncorrectInstanceState)
3. Start with invalid instance ID
4. Verify volumes re-mount on start
DONE
stop-instances --instance-ids --force, --hibernate, --dry-run run-instances (instance must be running) Gateway sends NATS to target node → daemon issues QMP system_powerdown for graceful shutdown → monitors heartbeat until QEMU exits → state transitions running→stopping→stopped 1. Graceful stop of running instance
2. Force stop (kills QEMU process)
3. Stop already-stopped instance (error)
4. Verify ~30s heartbeat detection
DONE
terminate-instances --instance-ids, DeleteOnTermination (per-volume flag, default true) --dry-run run-instances (instance must exist) Gateway sends NATS to target node → daemon kills QEMU process → cleans up NBD mounts → deletes volumes with DeleteOnTermination=true via volumeService.DeleteVolume() (S3 cleanup of vol/, vol-efi/, vol-cloudinit/) → internal volumes (EFI, cloud-init) always cleaned up via ebs.delete NATS → volumes with DeleteOnTermination=false left in available state → state→terminated 1. Terminate running instance
2. Terminate stopped instance
3. Terminate with DeleteOnTermination=true deletes volumes
4. Terminate with DeleteOnTermination=false preserves volumes
5. Terminate already-terminated (idempotent)
6. Internal volumes (EFI, cloud-init) always cleaned up
7. Invalid instance ID
DONE
reboot-instances --instance-ids --dry-run run-instances (instance must be running) Gateway validates instance IDs → sends EC2InstanceCommand with RebootInstance=true via NATS ec2.cmd.{instanceId} → daemon validates instance is in StateRunning (returns IncorrectInstanceState if stopped) → issues QMP system_reset → instance reboots without stopping, stays in running state 1. Reboot running instance
2. Reboot multiple instances
3. Reboot stopped instance (error: IncorrectInstanceState)
4. Instance not found (error: InvalidInstanceID.NotFound)
5. Verify instance stays in running state after reboot
DONE
describe-instance-types --filters (capacity filter only) --instance-types, --max-results, --next-token, --dry-run, all other filters None Gateway fans out NATS ec2.DescribeInstanceTypes to all nodes → each daemon reports supported types (t3.micro/small/medium/large) with vCPU/memory specs → gateway deduplicates and returns 1. List all instance types
2. Filter by specific type
3. Filter with capacity=true shows available slots
4. Verify vCPU/memory specs match hardware
DONE
modify-instance-attribute --instance-id, --instance-type, --user-data --disable-api-termination, --ebs-optimized, --source-dest-check, --instance-initiated-shutdown-behavior, --block-device-mappings, --groups, --ena-support, --sriov-net-support Instance must be stopped (in NATS KV) Gateway validates input (exactly one attribute per call, instance ID format) → NATS ec2.ModifyInstanceAttribute with spinifex-workers queue group → daemon loads stopped instance from JetStream KV → applies attribute change → writes back to KV → returns {} on success. InstanceType: updates vm.InstanceType, Config, and Instance fields; clears StateReason (enables recovery from instance-type-missing bug). UserData: stores decoded content in vm.UserData and re-encodes to base64 for RunInstancesInput (cloud-init on next start). No instance type pre-validation (matches AWS — invalid types accepted, fail at StartInstances time). 1. Change instance type while stopped
2. Change user data while stopped
3. Modify running instance (error: NotFound — running instances not in KV)
4. Instance not found (error: InvalidInstanceID.NotFound)
5. Instance not stopped (error: IncorrectInstanceState)
6. Invalid instance type accepted (fails on start with InsufficientInstanceCapacity)
7. StateReason cleared on type change (recovery from capacity-unavailable)
8. Missing/malformed instance ID (error: InvalidInstanceID.Malformed)
9. No attribute set (error: InvalidParameterValue)
10. Multiple attributes in one call (error: InvalidParameterValue)
DONE
get-console-output --instance-id --latest (always returns latest), --dry-run Instance must be running on a node Gateway sends NATS ec2.{instanceId}.GetConsoleOutput (per-instance topic, routed to owning node) → daemon reads console log file from disk → returns last 64KB base64-encoded with timestamp. Always available regardless of serial console access setting (matches AWS behavior). 1. Get output from running instance
2. Empty log file returns empty output
3. Instance not found (error: InvalidInstanceID.NotFound)
DONE
describe-instance-attribute --instance-id, --attribute (instanceType, userData, instanceInitiatedShutdownBehavior, disableApiTermination, disableApiStop, ebsOptimized, enaSupport, sourceDestCheck, rootDeviceName, kernel, ramdisk) --dry-run Instance must exist (running or stopped) Gateway validates input → NATS ec2.DescribeInstanceAttribute with spinifex-workers queue group → daemon checks running instances first (d.Instances.VMS), then stopped instances in JetStream KV → returns single attribute per call (matches AWS behavior). Stored attributes (instanceType, userData) return real values; unstored attributes return AWS defaults (instanceInitiatedShutdownBehavior=stop, disableApiTermination=false, etc.) 1. Get instanceType from running instance
2. Get userData from stopped instance
3. Get default disableApiTermination
4. Invalid attribute name (error)
5. Instance not found (error: InvalidInstanceID.NotFound)
DONE
describe-instance-credit-specifications --instance-ids --filters, --max-results, --dry-run None Gateway-only stub — returns CpuCredits: "standard" for each requested instance ID. No daemon round-trip. T-series credit mode is not persisted. 1. Get credit spec for T-series instance
2. Multiple instance IDs
DONE
monitor-instances — --instance-ids Instance must exist Enable basic monitoring (CPU, disk, network) → store metrics in NATS KV → return monitoring state 1. Enable monitoring
2. Verify monitoring state in describe-instances
NOT STARTED
unmonitor-instances — --instance-ids Instance must exist Disable detailed monitoring → revert to basic monitoring → return monitoring state 1. Disable monitoring
2. Verify monitoring state reverts in describe-instances
NOT STARTED

EC2 - Key Pair Management

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-key-pair --key-name, --key-type (rsa/ed25519) --key-format (pem/ppk), --tag-specifications, --dry-run None NATS ec2.CreateKeyPair → daemon generates SSH keypair → stores public key in Predastore S3 (/bucket/ec2/{name}.pub) → returns private key material (one-time) 1. Create RSA key pair
2. Create ED25519 key pair
3. Duplicate key name (error: InvalidKeyPair.Duplicate)
4. Verify key material returned only on creation
DONE
describe-key-pairs --key-names, --key-pair-ids, --filters (key-pair-id, key-name, fingerprint, tag:*) --max-results, --dry-run None NATS ec2.DescribeKeyPairs → daemon lists public keys from Predastore S3 → applies filters → returns key names and fingerprints 1. List all key pairs
2. Filter by key name
3. Filter by key pair ID
4. Non-existent key returns empty
5. Unknown filter returns InvalidParameterValue
DONE
delete-key-pair --key-name, --key-pair-id --dry-run Key must exist NATS ec2.DeleteKeyPair → daemon deletes public key from Predastore S3 → returns success 1. Delete existing key pair
2. Delete non-existent key (idempotent, no error)
3. Verify key no longer in describe-key-pairs
DONE
import-key-pair --key-name, --public-key-material --tag-specifications, --dry-run None NATS ec2.ImportKeyPair → daemon stores provided public key in Predastore S3 → returns key name and fingerprint 1. Import valid SSH public key
2. Import invalid key material (error)
3. Import duplicate name (error)
4. Verify imported key usable with run-instances
DONE

EC2 - AMI Image Management

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
describe-images --image-ids (format validation only), --owners (self, account ID, alias), --filters (name, state, architecture, image-id, is-public, owner-id, description, image-type, tag:*) --executable-users, --include-deprecated, --include-disabled, --max-results, --next-token, --dry-run None NATS ec2.DescribeImages → daemon reads AMI metadata from Predastore S3 buckets (ami-*) → filters by ImageIds, Owners, and Filters → returns image list with state, architecture, block device mappings 1. List all images
2. Filter by image ID
3. Filter by owner (self/amazon)
4. Non-existent AMI returns empty
5. Verify metadata fields (architecture, state, rootDeviceName)
6. Filter by name wildcard
7. Unknown filter returns InvalidParameterValue
DONE
create-image --instance-id, --name, --description, --tag-specifications --no-reboot, --block-device-mappings, --dry-run Instance must exist (running or stopped) Gateway validates input → NATS ec2.{instanceId}.CreateImage (per-instance topic) → daemon extracts root volume → snapshots via ebs.snapshot NATS (running) or offline S3 copy (stopped) → creates AMI metadata in S3 ({amiId}/config.json) → stores tags → returns ami-ID. Duplicate AMI name validation enforced. 1. Create image from running instance
2. Create image from stopped instance
3. Invalid instance ID (error)
4. Duplicate AMI name (error)
5. Verify new AMI appears in describe-images
6. Launch new instance from created AMI
DONE
register-image — --name, --description, --architecture, --root-device-name, --virtualization-type, --block-device-mappings, --boot-mode, --ena-support, --sriov-net-support, --tpm-support, --imds-support Image data must exist in S3 Create AMI metadata entry in Predastore → assign ami-ID → set state to available → return ami-ID 1. Register with valid metadata
2. Missing required name (error)
3. Verify registered image in describe-images
NOT STARTED
deregister-image — --image-id, --dry-run AMI must exist, no instances using it Mark AMI metadata as deregistered in S3 → AMI no longer available for new launches → existing instances unaffected 1. Deregister existing AMI
2. Deregister non-existent AMI (error)
3. Verify deregistered AMI not in describe-images
4. Existing instances from AMI still run
NOT STARTED
copy-image — --source-image-id, --source-region, --name, --description, --encrypted, --kms-key-id, --client-token, --copy-image-tags, --tag-specifications Source AMI must exist Copy AMI data between regions/nodes in Predastore → create new AMI metadata with new ami-ID → return new ami-ID 1. Copy image within same region
2. Copy non-existent image (error)
3. Verify copied image independent of source
NOT STARTED
import-image — --disk-containers (Format, Url/S3Bucket+S3Key), --description, --architecture, --platform S3 bucket with disk image Download disk image from S3 → convert format (VMDK/VHD/RAW→QCOW2) → create viperblock volume → register as AMI 1. Import QCOW2 image
2. Import RAW image
3. Invalid format (error)
4. Verify imported image launchable
NOT STARTED
describe-image-attribute — --image-id, --attribute, --dry-run AMI must exist Read specific attribute from AMI metadata in S3 → return attribute value 1. Get description attribute
2. Get launch permission
3. Invalid attribute name (error)
NOT STARTED
modify-image-attribute — --image-id, --launch-permission, --description, --operation-type, --user-ids, --user-groups, --organization-arns, --dry-run AMI must exist Update specific attribute in AMI metadata → persist to S3 1. Add launch permission
2. Modify description
3. Invalid AMI (error)
NOT STARTED
reset-image-attribute — --image-id, --attribute, --dry-run AMI must exist Reset attribute to default value in AMI metadata 1. Reset launch permission to owner-only
2. Invalid AMI (error)
NOT STARTED

EC2 - Volume (EBS) Management

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
describe-volumes --volume-ids (fast-path lookup), DeleteOnTermination (from persisted VolumeMetadata), --filters (volume-id, status, size, volume-type, attachment.instance-id, attachment.status, attachment.device, availability-zone, tag:*) --max-results, --next-token, --dry-run None NATS ec2.DescribeVolumes → daemon queries viperblock for volume metadata → applies filters → returns volume list with state, size, attachments, type, DeleteOnTermination flag 1. List all volumes
2. Filter by volume ID
3. Filter by attachment state
4. Non-existent volume returns empty
5. DeleteOnTermination reflects persisted value
6. Filter by status, size, volume-type
7. Unknown filter returns InvalidParameterValue
DONE
modify-volume --volume-id, --size, --volume-type, --iops --throughput, --dry-run, --multi-attach-enabled Volume must exist NATS ec2.ModifyVolume → daemon sends resize request to viperblock → NBD does not support live resize, instance must be stopped → returns modification state 1. Increase volume size
2. Modify volume type
3. Decrease size (error - not supported)
4. Modify attached volume (requires stop/start)
DONE
create-volume --size, --availability-zone, --volume-type (gp3 only), --snapshot-id (creates volume from snapshot) --iops (hardcoded 3000), --encrypted (hardcoded false), --throughput, --tag-specifications Valid AZ configured via spinifex init Gateway validates input → NATS ec2.CreateVolume → daemon generates vol-ID via viperblock → creates volume (empty or from snapshot) of specified size → persists config.json to Predastore S3 → returns vol-ID with state=available 1. Create 80GB gp3 volume
2. Boundary sizes (1 GiB min, 16384 GiB max)
3. Invalid AZ (error)
4. Verify volume in describe-volumes
5. Unsupported volume type (error - only gp3)
6. Size out of range (error)
7. Create from snapshot
DONE
delete-volume --volume-id --dry-run Volume must exist and be detached (state=available) Gateway validates vol- prefix → NATS ec2.DeleteVolume → daemon confirms state=available and no AttachedInstance → NATS ebs.delete to viperblockd (stops nbdkit/WAL) → deletes S3 objects under vol-id/, vol-id-efi/, vol-id-cloudinit/ → returns success 1. Delete detached volume
2. Delete attached volume (error: VolumeInUse)
3. Delete non-existent volume (error: InvalidVolume.NotFound)
4. Verify volume gone from describe-volumes
5. Malformed volume ID (error: InvalidVolumeID.Malformed)
6. Double delete (idempotent NotFound)
DONE
attach-volume --volume-id, --instance-id, --device (optional, auto-assigns /dev/sd[f-p]) --dry-run Volume must exist (available), instance must exist (running) Gateway sends to ec2.cmd.{instanceId} → daemon validates volume (Predastore) → ebs.mount via NATS (viperblock starts NBD server) → QMP blockdev-add (nbd-{volId}) → QMP device_add (virtio-blk-pci, vdisk-{volId}) → three-phase rollback on failure → update EBSRequests + BlockDeviceMappings → persist to JetStream + Predastore → respond with VolumeAttachment 1. Attach volume to running instance
2. Auto-assign device name
3. Attach already-attached volume (VolumeInUse)
4. Attach to non-existent instance (InvalidInstanceID.NotFound)
5. Attach to stopped instance (IncorrectInstanceState)
6. Volume not found (InvalidVolume.NotFound)
7. All device slots full (AttachmentLimitExceeded)
8. Volume persists across stop/start
DONE
detach-volume --volume-id, --instance-id (optional, resolved via DescribeVolumes), --device (optional cross-check), --force --dry-run Volume must be attached, instance must be running Gateway resolves InstanceId if omitted (via DescribeVolumes) → sends to ec2.cmd.{instanceId} → daemon validates (running, attached, not boot/EFI/CloudInit, device match) → three-phase hot-unplug: QMP device_del (force continues on failure) → QMP blockdev-del (abort if fails, preserves state to prevent double-attach) → ebs.unmount via NATS (best-effort) → remove from EBSRequests + BlockDeviceMappings → update volume metadata to available → persist state → respond with VolumeAttachment (state=detaching) 1. Detach with explicit InstanceId
2. Detach without InstanceId (gateway resolution)
3. Detach with correct --device cross-check
4. Missing VolumeId (InvalidParameterValue)
5. Volume not attached (IncorrectState)
6. Nonexistent volume (InvalidVolume.NotFound)
7. Nonexistent instance (InvalidInstanceID.NotFound)
8. Instance not running (IncorrectInstanceState)
9. Device mismatch (InvalidParameterValue)
10. Boot volume protection (OperationNotPermitted)
11. Force flag (continues past device_del failure)
12. Volume reusability (re-attach after detach)
DONE
describe-volume-status --volume-ids, --filters (volume-id, volume-status.status, availability-zone) --max-results, --next-token, --dry-run None Gateway validates vol- prefix → NATS ec2.DescribeVolumeStatus → daemon fetches VolumeConfig from Predastore S3 (parallel for specific IDs, sequential list-all for no IDs) → applies filters → builds VolumeStatusItem per volume (status=ok, io-enabled=passed, io-performance=not-applicable) → returns InvalidVolume.NotFound for missing explicit IDs → skips internal sub-volumes (-efi, -cloudinit) 1. List all volume statuses
2. Filter by specific volume IDs (fast path)
3. Non-existent volume ID returns InvalidVolume.NotFound
4. Invalid volume ID format (InvalidVolume.Malformed)
5. Internal sub-volumes excluded from listing
6. Nil/empty input defaults to all volumes
7. Unknown filter returns InvalidParameterValue
DONE
describe-volumes-modifications — --volume-ids, --filters, --max-results None Query pending/completed volume modifications → return modification state, progress, original/target size 1. Check in-progress modification
2. Check completed modification
3. No modifications returns empty
NOT STARTED

EC2 - Snapshot Management

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-snapshot --volume-id, --description, --tag-specifications --dry-run Volume must exist in Predastore Gateway validates vol- prefix → NATS ec2.CreateSnapshot → daemon reads VolumeConfig from Predastore → generates snap-ID via viperblock → stores SnapshotConfig (metadata-only, points to source volume) as completed → returns Snapshot with state=completed, progress=100% 1. Snapshot from valid volume
2. Missing volume ID (InvalidParameterValue)
3. Volume not found (InvalidVolume.NotFound)
4. Invalid volume ID format (InvalidVolume.Malformed)
5. Snapshot with description
6. Snapshot with tag specifications
DONE
create-snapshots — --instance-specification, --description, --tag-specifications Instance must exist, instance-volume attachment tracking Create snapshots of all volumes attached to instance → return list of snapshot IDs. Blocked: requires instance-volume attachment tracking to discover which volumes to snapshot. 1. Snapshot all volumes on instance
2. Instance with no volumes
NOT STARTED
delete-snapshot --snapshot-id --dry-run Snapshot must exist Gateway validates snap- prefix → NATS ec2.DeleteSnapshot → daemon verifies snapshot exists in Predastore → lists and deletes all objects under snapshot prefix → returns success 1. Delete existing snapshot
2. Delete non-existent snapshot (InvalidSnapshot.NotFound)
3. Missing snapshot ID (InvalidParameterValue)
4. Invalid snapshot ID format (InvalidSnapshot.Malformed)
DONE
describe-snapshots --snapshot-ids, --filters (snapshot-id, status, volume-id, volume-size, owner-id, tag:*) --owner-ids, --max-results, --dry-run None Gateway validates snap- prefix on IDs → NATS ec2.DescribeSnapshots → daemon lists snap- prefixed objects in Predastore → reads SnapshotConfig for each → applies filters → returns snapshot list 1. List all snapshots
2. Filter by snapshot ID
3. Empty snapshot list
4. Invalid snapshot ID format (InvalidSnapshot.Malformed)
5. Filter by status, volume-id, volume-size
6. Unknown filter returns InvalidParameterValue
DONE
copy-snapshot --source-snapshot-id, --source-region, --description --encrypted, --dry-run Source snapshot must exist Gateway validates snap- prefix + source region → NATS ec2.CopySnapshot → daemon reads source SnapshotConfig → generates new snap-ID → copies metadata (preserves tags, description override) → stores as completed → returns new snapshot ID 1. Copy within same region
2. Copy non-existent snapshot (InvalidSnapshot.NotFound)
3. Missing source ID (InvalidParameterValue)
4. Missing source region (MissingParameter)
5. Copy preserves tags
6. Copy with description override
DONE

EC2 - Tags

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-tags --resources, --tags (Key=,Value=) --dry-run None Gateway validates input (resources non-empty, tags non-empty, keys non-empty) → NATS ec2.CreateTags → daemon merges tags into per-resource JSON in Predastore S3 (tags/{resourceId}.json) → return success 1. Tag an instance
2. Tag multiple resources at once
3. Overwrite existing tag value
4. Empty resources list (error: MissingParameter)
5. Empty tags list (error: MissingParameter)
6. Empty tag key (error: InvalidParameterValue)
DONE
delete-tags --resources, --tags --dry-run None Gateway validates input (resources non-empty) → NATS ec2.DeleteTags → daemon removes specified tag keys from resource JSON, or all tags if no keys specified → return success 1. Delete specific tag by key
2. Delete all tags (no tags specified)
3. Empty resources list (error: MissingParameter)
DONE
describe-tags --filters (resource-id, resource-type, key, value) --max-results, --next-token, --dry-run None NATS ec2.DescribeTags → daemon lists all tags/*.json from Predastore S3 → applies filters (resource-id, resource-type, key, value) → returns tag list with resource type auto-detected from ID prefix 1. List all tags
2. Filter by resource-id
3. Filter by resource-type (instance, volume, image, snapshot, vpc, subnet, security-group, route-table, internet-gateway, egress-only-internet-gateway)
4. Filter by key
5. Filter by value
6. Empty result when no tags exist
DONE

EC2 - Regions & Availability Zones

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
describe-regions (returns configured region, endpoint, opt-in status from config — input params ignored) --region-names, --filters, --all-regions, --dry-run None Return configured region from spinifex init config with Endpoint and OptInStatus → local-only response, no NATS 1. List all regions
2. Filter by region name
3. Verify endpoint URL returned
4. Verify OptInStatus returned
DONE
describe-availability-zones (returns configured AZ, region, zone ID, state, opt-in status from config — input params ignored) --zone-names, --filters, --all-availability-zones None Return configured AZ from spinifex init config with zone ID, state, group name, network border group → local-only response, no NATS 1. List all AZs
2. Filter by zone name
3. Verify zone state is available
4. Verify region name matches config
DONE

EC2 - Account Attributes

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
describe-account-attributes --attribute-names --dry-run None Gateway parses input → returns static account attributes: supported-platforms=VPC, default-vpc=none, max-instances=100, vpc-max-security-groups-per-interface=5, max-elastic-ips=5, vpc-max-elastic-ips=20 → local-only response, no NATS 1. List all account attributes
2. Filter by attribute name
3. Verify all 6 attributes returned with correct values
DONE

EC2 - Account Settings (Deferred)

Persistence layer works (NATS JetStream KV), but stored values are not yet enforced by downstream services. Will be completed post-merge.

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
enable-ebs-encryption-by-default (no flags needed) --dry-run None Gateway validates input → NATS ec2.EnableEbsEncryptionByDefault with spinifex-workers queue group → daemon stores EbsEncryptionByDefault=true in JetStream KV (spinifex-ec2-account-settings bucket) → return EbsEncryptionByDefault=true. Enforcement deferred: stored value not yet checked by CreateVolume/CreateSnapshot. 1. Enable encryption
2. Verify via get
STARTED (enforcement pending)
disable-ebs-encryption-by-default (no flags needed) --dry-run None Gateway validates input → NATS ec2.DisableEbsEncryptionByDefault with spinifex-workers queue group → daemon stores EbsEncryptionByDefault=false in JetStream KV → return EbsEncryptionByDefault=false. 1. Disable encryption
2. Verify via get
STARTED (enforcement pending)
get-ebs-encryption-by-default (no flags needed) --dry-run None Gateway validates input → NATS ec2.GetEbsEncryptionByDefault with spinifex-workers queue group → daemon reads EbsEncryptionByDefault from JetStream KV → return current state. 1. Get default state
2. Verify matches last enable/disable
STARTED (enforcement pending)
enable-serial-console-access (no flags needed) --dry-run None Gateway validates input → NATS ec2.EnableSerialConsoleAccess with spinifex-workers queue group → daemon stores SerialConsoleAccessEnabled=true in JetStream KV (spinifex-ec2-account-settings bucket) → return enabled=true. Stored for future interactive serial console support (not currently enforced by any operation). 1. Enable access
2. Verify via get
STARTED (enforcement pending)
disable-serial-console-access (no flags needed) --dry-run None Gateway validates input → NATS ec2.DisableSerialConsoleAccess with spinifex-workers queue group → daemon stores SerialConsoleAccessEnabled=false in JetStream KV → return enabled=false. Stored for future interactive serial console support (not currently enforced by any operation). 1. Disable access
2. Verify via get
STARTED (enforcement pending)
get-serial-console-access-status (no flags needed) --dry-run None Gateway validates input → NATS ec2.GetSerialConsoleAccessStatus with spinifex-workers queue group → daemon reads SerialConsoleAccessEnabled from JetStream KV → return current state. 1. Get current status
2. Verify matches last enable/disable
DONE
enable-snapshot-block-public-access — --state None Store snapshot block public access state in JetStream KV 1. Enable with block-all-sharing
2. Enable with block-new-sharing
NOT STARTED (enforcement pending)
disable-snapshot-block-public-access — --dry-run None Clear snapshot block public access state in JetStream KV 1. Disable access NOT STARTED (enforcement pending)
get-snapshot-block-public-access-state — --dry-run None Read snapshot block public access state from JetStream KV 1. Get current state NOT STARTED (enforcement pending)
enable-image-block-public-access — --image-block-public-access-state None Store image block public access state in JetStream KV 1. Enable with block-new-sharing NOT STARTED (enforcement pending)
disable-image-block-public-access — --dry-run None Clear image block public access state in JetStream KV 1. Disable access NOT STARTED (enforcement pending)
get-image-block-public-access-state — --dry-run None Read image block public access state from JetStream KV 1. Get current state NOT STARTED (enforcement pending)
modify-instance-metadata-defaults — --http-tokens, --http-put-response-hop-limit, --http-endpoint, --instance-metadata-tags None Store instance metadata defaults in JetStream KV 1. Set httpTokens=required
2. Set hop limit
NOT STARTED (enforcement pending)
get-instance-metadata-defaults — --dry-run None Read instance metadata defaults from JetStream KV 1. Get current defaults NOT STARTED (enforcement pending)

EC2 - Egress-Only Internet Gateway

Gateway routing and KV CRUD are wired up but no OVN/OVS integration exists. EIGWs are stored in spinifex-eigw KV bucket but have no effect on network topology (no vpcd events published, unlike IGW which publishes vpc.igw-attach/vpc.igw-detach).

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-egress-only-internet-gateway --vpc-id, --tag-specifications --client-token, --dry-run VPC must exist Gateway validates VpcId → NATS ec2.CreateEgressOnlyInternetGateway → daemon generates eigw-ID → creates EgressOnlyIGWRecord in spinifex-eigw KV bucket with state=attached (always attached at creation, unlike IGW) → returns EgressOnlyInternetGateway with attachment 1. Create with valid VpcId
2. Missing VpcId (error: MissingParameter)
3. Create with inline tags
4. Verify ID format eigw-{16hex}
STARTED (KV only, no OVN)
delete-egress-only-internet-gateway --egress-only-internet-gateway-id --dry-run EIGW must exist NATS ec2.DeleteEgressOnlyInternetGateway → daemon verifies EIGW exists in KV → deletes from spinifex-eigw KV → returns ReturnCode=true. No detach required (unlike IGW). 1. Delete existing EIGW
2. Delete non-existent EIGW (error)
3. Missing ID (error: MissingParameter)
STARTED (KV only, no OVN)
describe-egress-only-internet-gateways --egress-only-internet-gateway-ids, --filters (egress-only-internet-gateway-id, tag:*) --max-results, --next-token, --dry-run None NATS ec2.DescribeEgressOnlyInternetGateways → daemon lists all keys from spinifex-eigw KV bucket → filters by ID if specified → applies filters → always includes Attachments array → returns list 1. Describe all EIGWs
2. Filter by specific ID
3. Non-existent ID returns empty list
4. Unknown filter returns InvalidParameterValue
STARTED (KV only, no OVN)

EC2 - Placement Groups

Placement groups are stored in spinifex-placement-groups JetStream KV bucket, keyed by {accountID}.{groupName}. All mutations use CAS (compare-and-swap) with 5-retry limit for optimistic concurrency. Supported strategies: spread (strict 1 instance per node) and cluster (all instances pinned to single node). Partition strategy is rejected. Internal NATS operations (ReserveSpreadNodes, FinalizeSpreadInstances, ReleaseSpreadNodes, RemoveInstanceFromPlacementGroup, ReserveClusterNode, FinalizeClusterInstances) support atomic multi-node coordination for RunInstances and TerminateInstances.

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-placement-group --group-name, --strategy (spread/cluster) --partition-count, --spread-level (hardcoded to "host"), --tag-specifications, --dry-run None Gateway validates GroupName + Strategy → NATS ec2.CreatePlacementGroup with spinifex-workers queue group → daemon generates pg-ID (pg-{16hex}) → stores PlacementGroupRecord in spinifex-placement-groups KV with state=available, SpreadLevel="host" for spread → rejects partition strategy (InvalidParameterValue) → rejects duplicate names (InvalidPlacementGroup.Duplicate) → returns PlacementGroup 1. Create with spread strategy
2. Create with cluster strategy
3. Create with partition strategy (error: InvalidParameterValue)
4. Duplicate name (error: InvalidPlacementGroup.Duplicate)
5. Missing GroupName (error)
6. Missing Strategy (error)
DONE
delete-placement-group --group-name --dry-run Group must exist, no instances in group NATS ec2.DeletePlacementGroup → daemon verifies group exists (InvalidPlacementGroup.Unknown) → counts instances in NodeInstances map → rejects if non-empty (InvalidPlacementGroup.InUse) → deletes from KV → returns success 1. Delete empty group
2. Delete group with instances (error: InvalidPlacementGroup.InUse)
3. Delete non-existent group (error: InvalidPlacementGroup.Unknown)
DONE
describe-placement-groups --group-names, --group-ids, --filters (strategy, state, spread-level, group-name) --dry-run None NATS ec2.DescribePlacementGroups → daemon lists all keys from spinifex-placement-groups KV → filters by GroupNames, GroupIds, or Filters → returns list with Strategy, State, SpreadLevel, GroupId 1. List all groups
2. Filter by name
3. Filter by group ID
4. Filter by strategy
5. Filter by state
6. Empty result when no groups exist
DONE

EC2 - Dedicated Hosts

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
allocate-hosts — --availability-zone, --instance-type, --quantity, --auto-placement, --tag-specifications None NATS ec2.AllocateHosts → daemon allocates host resources → stores in JetStream KV with h-ID → return host ID list 1. Allocate single host
2. Allocate multiple hosts
3. Missing AZ or instance type (error)
NOT STARTED
describe-hosts — --host-ids, --filters, --max-results None NATS ec2.DescribeHosts → daemon lists hosts from KV → return host list with capacity and instance info 1. List all hosts
2. Filter by host ID
3. Filter by instance type
NOT STARTED
release-hosts — --host-ids Host must exist, no running instances NATS ec2.ReleaseHosts → daemon verifies no instances on host → delete from KV → return success/failure per host 1. Release empty host
2. Release host with instances (error)
3. Release non-existent host (error)
NOT STARTED

EC2 - IPv4 Pools

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-public-ipv4-pool — --tag-specifications, --dry-run None NATS ec2.CreatePublicIpv4Pool → daemon creates pool in JetStream KV with ipv4pool-ID → return pool ID 1. Create pool
2. Verify in describe
NOT STARTED
delete-public-ipv4-pool — --pool-id, --dry-run Pool must exist and be empty NATS ec2.DeletePublicIpv4Pool → daemon verifies pool empty → delete from KV → return success 1. Delete empty pool
2. Delete pool with addresses (error)
NOT STARTED
describe-public-ipv4-pools — --pool-ids, --filters, --max-results None NATS ec2.DescribePublicIpv4Pools → daemon lists pools from KV → return pool list with address counts 1. List all pools
2. Filter by pool ID
NOT STARTED

EC2 - DHCP Options

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-dhcp-options — --dhcp-configurations (domain-name, domain-name-servers, ntp-servers, netbios-name-servers, netbios-node-type), --tag-specifications None NATS ec2.CreateDhcpOptions → daemon stores DHCP option set in JetStream KV with dopt-ID → return DHCP options with configurations 1. Create with domain-name + DNS servers
2. Create with NTP servers
3. Verify in describe
NOT STARTED
delete-dhcp-options — --dhcp-options-id, --dry-run DHCP options must exist and not be associated with a VPC NATS ec2.DeleteDhcpOptions → daemon verifies not associated → delete from KV → return success 1. Delete unassociated options
2. Delete associated options (error)
3. Delete non-existent (error)
NOT STARTED
describe-dhcp-options — --dhcp-options-ids, --filters, --max-results None NATS ec2.DescribeDhcpOptions → daemon lists DHCP options from KV → return list with configurations 1. List all DHCP options
2. Filter by ID
NOT STARTED
associate-dhcp-options — --dhcp-options-id, --vpc-id, --dry-run DHCP options and VPC must exist NATS ec2.AssociateDhcpOptions → daemon links DHCP options to VPC → update VPC metadata → return success 1. Associate with VPC
2. Missing DHCP options ID (error)
3. Missing VPC ID (error)
NOT STARTED

EC2 - Capacity Reservations

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-capacity-reservation — --instance-type, --instance-platform, --availability-zone, --instance-count, --end-date, --end-date-type, --instance-match-criteria, --tag-specifications None NATS ec2.CreateCapacityReservation → daemon stores reservation in JetStream KV with cr-ID → return reservation with state=active 1. Create with required fields
2. Missing instance type (error)
3. Verify in describe
NOT STARTED
cancel-capacity-reservation — --capacity-reservation-id, --dry-run Reservation must exist NATS ec2.CancelCapacityReservation → daemon marks reservation as cancelled in KV → return success 1. Cancel active reservation
2. Cancel non-existent (error)
NOT STARTED
describe-capacity-reservations — --capacity-reservation-ids, --filters, --max-results None NATS ec2.DescribeCapacityReservations → daemon lists reservations from KV → return list with state and counts 1. List all reservations
2. Filter by ID
3. Filter by instance type
NOT STARTED
modify-capacity-reservation — --capacity-reservation-id, --instance-count, --end-date, --end-date-type Reservation must exist NATS ec2.ModifyCapacityReservation → daemon updates reservation in KV → return success 1. Modify instance count
2. Modify end date
3. Missing reservation ID (error)
NOT STARTED

EC2 - Elastic IP

EIP operations are conditionally available — only registered when external IPAM is configured (pool mode). EIPs are stored in spinifex-eip-allocations KV bucket. Association publishes vpc.add-nat/vpc.delete-nat events to vpcd for OVN NAT configuration.

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
allocate-address --public-ipv4-pool (optional, specific IPAM pool), --tag-specifications --domain (hardcoded vpc), --dry-run EIP service must be initialized (external IPAM configured) Gateway validates input → NATS ec2.AllocateAddress with spinifex-workers queue group → daemon allocates IP from external IPAM pool (named or best-matching) → creates EIPRecord with state=allocated in spinifex-eip-allocations KV → returns AllocationId, PublicIp, Domain=vpc, PublicIpv4Pool 1. Allocate EIP from default pool
2. Allocate from specific pool
3. Pool exhausted (InsufficientAddressCapacity)
4. With tag specifications
5. Verify in describe-addresses
DONE
release-address --allocation-id --dry-run EIP must exist and be disassociated Gateway validates AllocationId non-empty → NATS ec2.ReleaseAddress → daemon retrieves EIP record from KV → checks state != associated (InvalidAddressLocked if still associated) → releases IP back to IPAM pool → deletes KV record → returns success 1. Release disassociated EIP
2. Release associated EIP (error: InvalidAddressLocked)
3. Release non-existent (error: InvalidAllocationID.NotFound)
4. Missing AllocationId (error: MissingParameter)
DONE
associate-address --allocation-id, --network-interface-id OR --instance-id, --private-ip-address (optional override) --dry-run, --allow-reassociation EIP and target ENI/instance must exist Gateway validates AllocationId + one of NetworkInterfaceId/InstanceId required → NATS ec2.AssociateAddress → daemon retrieves EIP → resolves target ENI (by ID or instance lookup) → generates AssociationId → updates record state=associated with ENI/instance/VPC/private IP → publishes vpc.add-nat event to vpcd (OVN NAT configuration) → returns AssociationId 1. Associate with instance
2. Associate with specific ENI
3. Associate with private IP override
4. Missing AllocationId (MissingParameter)
5. Missing target (MissingParameter)
6. Non-existent EIP (InvalidAllocationID.NotFound)
7. Non-existent ENI (InvalidNetworkInterfaceID.NotFound)
DONE
disassociate-address --association-id --dry-run Association must exist Gateway validates AssociationId non-empty → NATS ec2.DisassociateAddress → daemon scans KV to find EIP by AssociationId → publishes vpc.delete-nat event to vpcd → clears association fields → reverts state to allocated → updates KV → returns success 1. Disassociate existing
2. Non-existent association (InvalidAssociationID.NotFound)
3. Missing AssociationId (MissingParameter)
DONE
describe-addresses --allocation-ids, --public-ips, --filters (allocation-id, public-ip, instance-id, association-id, domain, tag:*) --dry-run None NATS ec2.DescribeAddresses → daemon scans all EIP records in account → filters by AllocationIds and/or PublicIps if specified → applies filters → returns Address list with allocation, association, PublicIp, PrivateIpAddress, NetworkInterfaceId, InstanceId, Domain 1. List all EIPs
2. Filter by allocation ID
3. Filter by public IP
4. Non-existent AllocationId returns error
5. Empty result when no EIPs
6. Filter by instance-id, domain
7. Unknown filter returns InvalidParameterValue
DONE
describe-addresses-attribute --allocation-ids (optional filter) --attribute, --dry-run, --max-results, --next-token None Gateway validates input → NATS ec2.DescribeAddressesAttribute with spinifex-workers queue group → daemon looks up EIP records by AllocationIds (or scans all account EIPs if none specified) → returns AddressAttribute list with AllocationId and PublicIp per entry. Unlike DescribeAddresses, returns empty list (not error) for non-existent AllocationIds. PtrRecord always nil (reverse-DNS not supported). 1. List all EIP attributes
2. Filter by allocation IDs
3. Non-existent AllocationId returns empty (not error)
4. Nil input returns error
DONE

EC2 - NAT Gateway

NAT gateways are stored in spinifex-vpc-nat-gateways KV bucket. Deleted gateways are moved to spinifex-vpc-deleted-nat-gateways with a 1-hour TTL for Terraform polling. SNAT rule coordination uses vpc.add-nat-gateway and vpc.delete-nat-gateway NATS events (published via route table service on CreateRoute and during DeleteNatGateway respectively).

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-nat-gateway --subnet-id, --allocation-id --connectivity-type (public only), --tag-specifications, --dry-run Subnet must exist, EIP must be allocated and unassociated Gateway validates SubnetId + AllocationId (required) → NATS ec2.CreateNatGateway with spinifex-workers queue group → daemon verifies subnet exists in spinifex-vpc-subnets KV → verifies EIP exists in spinifex-vpc-eips KV and has no AssociationId (Resource.AlreadyAssociated if in use) → generates nat-ID → stores NatGatewayRecord in spinifex-vpc-nat-gateways KV with state=available → returns NatGateway with addresses 1. Create NAT gateway (happy path)
2. Subnet not found (InvalidSubnetID.NotFound)
3. EIP not found (InvalidAllocationID.NotFound)
4. EIP already associated (Resource.AlreadyAssociated)
5. Missing SubnetId or AllocationId (MissingParameter)
DONE
delete-nat-gateway --nat-gateway-id --dry-run NAT gateway must exist Gateway validates NatGatewayId (required) → NATS ec2.DeleteNatGateway → daemon retrieves record from KV → scans all route tables in VPC for routes targeting this NAT gateway → publishes vpc.delete-nat-gateway event per affected subnet (SNAT rule cleanup) → sets state=deleted → stores in spinifex-vpc-deleted-nat-gateways with 1-hour TTL → deletes from active KV bucket 1. Delete existing NAT gateway
2. Delete non-existent (InvalidNatGatewayID.NotFound)
3. Deleted gateway visible when queried by ID (state=deleted)
4. Deleted gateway excluded from general list queries (TTL auto-expire)
DONE
describe-nat-gateways --nat-gateway-ids, --filters (vpc-id, state) --max-results, --next-token, --dry-run None NATS ec2.DescribeNatGateways → daemon scans spinifex-vpc-nat-gateways KV → applies NatGatewayIds filter and/or vpc-id/state filters → for specific IDs not found in active bucket, checks deleted bucket → returns InvalidNatGatewayID.NotFound if ID never existed → returns list with state, addresses, subnet, VPC 1. List all NAT gateways
2. Filter by specific IDs
3. Filter by state
4. Filter by vpc-id
5. Non-existent ID (InvalidNatGatewayID.NotFound)
6. Deleted gateway visible by ID query
DONE
assign-private-nat-gateway-address — --nat-gateway-id, --private-ip-addresses NAT gateway must exist (private type) NATS ec2.AssignPrivateNatGatewayAddress → daemon assigns private IPs to NAT gateway → return assigned addresses 1. Assign address
2. Missing NAT gateway ID (error)
NOT STARTED
associate-nat-gateway-address — --nat-gateway-id, --allocation-ids NAT gateway and EIPs must exist NATS ec2.AssociateNatGatewayAddress → daemon associates EIPs with NAT gateway → return association info 1. Associate EIP
2. Missing NAT gateway ID (error)
NOT STARTED

EC2 - VPC Networking (Core)

Requires OVN/OVS (apt install openvswitch-switch ovn-host). VPC/Subnet/ENI/SG CRUD stores metadata in NATS KV and publishes events to vpcd (the network topology daemon) which translates to OVN logical switches, ports, routers, and ACLs. Single AZ for Spinifex v1. IPAM uses CAS-based allocation from NATS KV (spinifex-vpc-ipam bucket). Security groups stored in spinifex-vpc-security-groups KV bucket with optimistic locking on rule updates.

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-vpc --cidr-block, --tag-specifications --instance-tenancy, --dry-run None Gateway validates CidrBlock (required, valid CIDR, /16–/28 prefix) → NATS ec2.CreateVpc → daemon allocates unique VNI from spinifex-vpc-vni-counter KV → generates vpc-ID → stores VPCRecord in spinifex-vpc-vpcs KV → publishes vpc.create event to vpcd (creates OVN logical switch) → returns VPC with state=available 1. Create VPC with /16 CIDR
2. Invalid CIDR format (error: InvalidVpcRange)
3. CIDR prefix outside /16–/28 (error)
4. Verify in describe-vpcs
5. Tags applied at creation
DONE
delete-vpc --vpc-id --dry-run VPC must be empty (no subnets) Gateway validates VpcId → NATS ec2.DeleteVpc → daemon checks for dependent subnets in spinifex-vpc-subnets KV → returns DependencyViolation if subnets exist → deletes from spinifex-vpc-vpcs KV → publishes vpc.delete event to vpcd (removes OVN resources) 1. Delete empty VPC
2. Delete VPC with subnets (DependencyViolation)
3. Delete non-existent VPC (error)
DONE
describe-vpcs --vpc-ids, --filters (vpc-id, state, cidr-block, is-default, owner-id, tag:*) --max-results, --dry-run None NATS ec2.DescribeVpcs → daemon lists all keys from spinifex-vpc-vpcs KV → filters by VPC IDs if specified → applies filters → returns error for non-existent requested IDs → returns VPC list with CIDR, state, CIDR associations 1. List all VPCs
2. Filter by VPC ID
3. Non-existent VPC ID returns error
4. Filter by state, cidr-block, is-default
5. Unknown filter returns InvalidParameterValue
DONE
modify-vpc-attribute --vpc-id, --enable-dns-hostnames, --enable-dns-support, --enable-network-address-usage-metrics --dry-run VPC must exist Gateway validates VpcId non-empty → NATS ec2.ModifyVpcAttribute → daemon retrieves VPC record from spinifex-vpc-vpcs KV → updates boolean flags if provided → updates KV with optimistic locking → returns empty output 1. Enable DNS hostnames
2. Enable DNS support
3. Enable network address usage metrics
4. Missing VPC ID (MissingParameter)
5. Non-existent VPC (InvalidVpcID.NotFound)
DONE
describe-vpc-attribute --vpc-id, --attribute (enableDnsHostnames, enableDnsSupport, enableNetworkAddressUsageMetrics) --dry-run VPC must exist Gateway validates VpcId + Attribute non-empty → NATS ec2.DescribeVpcAttribute → daemon retrieves VPC record → returns single attribute as AttributeBooleanValue (one attribute per call, matches AWS behavior) → validates attribute name 1. Get enableDnsHostnames
2. Get enableDnsSupport
3. Get enableNetworkAddressUsageMetrics
4. Invalid attribute name (InvalidParameterValue)
5. Missing VPC ID (MissingParameter)
6. Non-existent VPC (InvalidVpcID.NotFound)
DONE
associate-vpc-cidr-block — --vpc-id, --cidr-block VPC must exist NATS ec2.AssociateVpcCidrBlock → daemon adds secondary CIDR to VPC metadata → return association 1. Add secondary CIDR
2. Overlapping CIDR (error)
3. Max CIDR blocks exceeded (error)
NOT STARTED
disassociate-vpc-cidr-block — --association-id Association must exist NATS ec2.DisassociateVpcCidrBlock → daemon removes secondary CIDR from VPC → return success 1. Remove secondary CIDR
2. Remove primary CIDR (error)
NOT STARTED
create-subnet --vpc-id, --cidr-block, --availability-zone, --tag-specifications --dry-run VPC must exist Gateway validates VpcId + CidrBlock (required, valid CIDR, /16–/28 prefix) → NATS ec2.CreateSubnet → daemon verifies parent VPC exists → validates subnet CIDR is within VPC CIDR range → checks for overlapping subnets in same VPC → calculates available IPs (total hosts - 5 AWS-reserved: .0/.1/.2/.3/.255) → generates subnet-ID → stores SubnetRecord in spinifex-vpc-subnets KV → publishes vpc.create-subnet event to vpcd → defaults AZ from config if omitted 1. Create subnet within VPC CIDR
2. Subnet CIDR outside VPC range (error: InvalidSubnetRange)
3. Overlapping subnet CIDRs (error: InvalidSubnetConflict)
4. CIDR prefix outside /16–/28 (error)
5. Missing VPC or CIDR (error: MissingParameter)
DONE
delete-subnet --subnet-id --dry-run Subnet must exist NATS ec2.DeleteSubnet → daemon verifies subnet exists → deletes from spinifex-vpc-subnets KV → publishes vpc.delete-subnet event to vpcd 1. Delete existing subnet
2. Delete non-existent subnet (error)
DONE
describe-subnets --subnet-ids, --filters (vpc-id, subnet-id, availability-zone, cidr-block, state, default-for-az, tag:*) --max-results, --dry-run None NATS ec2.DescribeSubnets → daemon lists all keys from spinifex-vpc-subnets KV → applies filters → filters by subnet IDs if specified → recalculates available IPs from CIDR → returns error for non-existent requested IDs 1. List all subnets
2. Filter by VPC ID
3. Filter by subnet ID
4. Non-existent subnet returns error
5. Filter by availability-zone, cidr-block, state
6. Unknown filter returns InvalidParameterValue
DONE
modify-subnet-attribute --subnet-id, --map-public-ip-on-launch --assign-ipv6-address-on-creation, --dry-run Subnet must exist Gateway validates SubnetId non-empty → NATS ec2.ModifySubnetAttribute → daemon retrieves Subnet record from spinifex-vpc-subnets KV → updates MapPublicIpOnLaunch flag if provided → updates KV with optimistic locking → returns empty output 1. Enable auto-assign public IP
2. Disable auto-assign public IP
3. Missing SubnetId (MissingParameter)
4. Non-existent subnet (InvalidSubnetID.NotFound)
DONE
associate-subnet-cidr-block — --subnet-id, --ipv6-cidr-block Subnet must exist NATS ec2.AssociateSubnetCidrBlock → daemon adds IPv6 CIDR to subnet → return association 1. Add IPv6 CIDR
2. Missing subnet ID (error)
NOT STARTED
disassociate-subnet-cidr-block — --association-id Association must exist NATS ec2.DisassociateSubnetCidrBlock → daemon removes IPv6 CIDR from subnet → return success 1. Remove IPv6 CIDR
2. Invalid association (error)
NOT STARTED
create-security-group --group-name, --description, --vpc-id, --tag-specifications --dry-run VPC must exist Gateway validates GroupName + VpcId required → NATS ec2.CreateSecurityGroup → daemon verifies VPC exists → checks duplicate name in same VPC → generates sg-ID → creates SecurityGroupRecord in spinifex-vpc-security-groups KV with default egress rule (allow all outbound 0.0.0.0/0), no ingress rules → publishes vpc.create-sg event to vpcd (OVN ACL configuration) → returns GroupId 1. Create SG in VPC
2. Duplicate name in same VPC (error: InvalidGroup.Duplicate)
3. Verify default egress rule (allow all)
4. Verify no default ingress rules
5. Missing GroupName (MissingParameter)
6. Non-existent VPC (InvalidVpcID.NotFound)
7. With tag specifications
DONE
delete-security-group --group-id --dry-run SG must exist Gateway validates GroupId non-empty → NATS ec2.DeleteSecurityGroup → daemon retrieves SG from KV → deletes from spinifex-vpc-security-groups KV → publishes vpc.delete-sg event to vpcd → returns success 1. Delete existing SG
2. Delete non-existent SG (error: InvalidGroup.NotFound)
3. Missing GroupId (MissingParameter)
DONE
describe-security-groups --group-ids, --filters (vpc-id, group-name, group-id, description, ip-permission.cidr, tag:*) --group-names, --max-results, --dry-run None NATS ec2.DescribeSecurityGroups → daemon scans all SG records in account → applies filters (AND logic across filter names, OR across values) → returns SecurityGroup list with ingress/egress rules → returns error for non-existent requested GroupIds 1. List all SGs
2. Filter by VPC ID
3. Filter by group name
4. Filter by GroupId
5. Non-existent GroupId (InvalidGroup.NotFound)
6. Empty result when no SGs
7. Filter by description, ip-permission.cidr
8. Unknown filter returns InvalidParameterValue
DONE
authorize-security-group-ingress --group-id, --ip-permissions (IpProtocol, FromPort, ToPort, IpRanges/CidrIp, UserIdGroupPairs/GroupId) --dry-run SG must exist Gateway validates GroupId non-empty → NATS ec2.AuthorizeSecurityGroupIngress → daemon retrieves SG → converts IpPermissions to SGRule objects → appends to IngressRules → updates KV with optimistic locking (revision check) → publishes vpc.update-sg event to vpcd → returns true 1. Allow SSH (port 22) from 0.0.0.0/0
2. Allow from specific CIDR
3. Allow from source security group
4. Missing GroupId (MissingParameter)
5. Non-existent SG (InvalidGroup.NotFound)
DONE
authorize-security-group-egress --group-id, --ip-permissions --dry-run SG must exist Gateway validates GroupId → NATS ec2.AuthorizeSecurityGroupEgress → daemon retrieves SG → appends rules to EgressRules → updates KV with optimistic locking → publishes vpc.update-sg event → returns true 1. Allow HTTPS outbound
2. Restrict to specific CIDR
3. Missing GroupId (MissingParameter)
4. Non-existent SG (InvalidGroup.NotFound)
DONE
revoke-security-group-ingress --group-id, --ip-permissions (matching rules to remove) --dry-run SG must exist Gateway validates GroupId → NATS ec2.RevokeSecurityGroupIngress → daemon retrieves SG → removes matching rules from IngressRules (set difference) → updates KV with optimistic locking → publishes vpc.update-sg event → returns true 1. Revoke existing rule
2. Missing GroupId (MissingParameter)
3. Non-existent SG (InvalidGroup.NotFound)
DONE
revoke-security-group-egress --group-id, --ip-permissions (matching rules to remove) --dry-run SG must exist Gateway validates GroupId → NATS ec2.RevokeSecurityGroupEgress → daemon retrieves SG → removes matching rules from EgressRules (set difference) → updates KV with optimistic locking → publishes vpc.update-sg event → returns true 1. Revoke existing rule
2. Missing GroupId (MissingParameter)
3. Non-existent SG (InvalidGroup.NotFound)
DONE
create-internet-gateway --tag-specifications --dry-run None NATS ec2.CreateInternetGateway → daemon generates igw-ID → creates IGWRecord in spinifex-igw KV bucket with state=available (not attached) → returns InternetGateway with empty Attachments 1. Create IGW
2. Verify in describe-internet-gateways
3. Tags applied at creation
DONE
attach-internet-gateway --internet-gateway-id, --vpc-id --dry-run IGW and VPC must exist NATS ec2.AttachInternetGateway → daemon validates IGW exists and is not already attached (ResourceAlreadyAssociated) → sets VpcId on record → state=attached → publishes vpc.igw-attach event to vpcd (creates OVN external switch, gateway port, SNAT rules) 1. Attach IGW to VPC
2. Attach already-attached IGW (ResourceAlreadyAssociated)
3. Non-existent IGW (error)
4. Missing params (MissingParameter)
DONE
detach-internet-gateway --internet-gateway-id, --vpc-id --dry-run IGW must be attached to specified VPC NATS ec2.DetachInternetGateway → daemon validates IGW exists and is attached to specified VPC (GatewayNotAttached if wrong VPC) → clears VpcId → state=available → publishes vpc.igw-detach event to vpcd (removes OVN resources) 1. Detach attached IGW
2. Detach from wrong VPC (GatewayNotAttached)
3. Detach unattached IGW (error)
DONE
delete-internet-gateway --internet-gateway-id --dry-run IGW must be detached NATS ec2.DeleteInternetGateway → daemon validates IGW exists and VpcId is empty (DependencyViolation if attached) → deletes from spinifex-igw KV 1. Delete detached IGW
2. Delete attached IGW (DependencyViolation)
3. Delete non-existent (error)
DONE
describe-internet-gateways --internet-gateway-ids, --filters (internet-gateway-id, attachment.vpc-id, attachment.state, tag:*) --max-results, --dry-run None NATS ec2.DescribeInternetGateways → daemon lists all keys from spinifex-igw KV → filters by IGW IDs if specified → applies filters → returns list with Attachments array (if attached) 1. List all IGWs
2. Filter by IGW ID
3. Verify attachment info
4. Filter by attachment.vpc-id, attachment.state
5. Unknown filter returns InvalidParameterValue
DONE

EC2 - Route Tables

Route tables are stored in spinifex-vpc-route-tables JetStream KV bucket, keyed by {accountID}.{routeTableId}. Each route table has a Routes array (with local route pre-populated) and an Associations array. All operations validate VPC consistency. When a route targeting a NAT gateway is created, vpc.add-nat-gateway events are published per associated subnet for vpcd SNAT rule creation.

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-route-table --vpc-id --tag-specifications, --dry-run VPC must exist Gateway validates VpcId (required) → NATS ec2.CreateRouteTable with spinifex-workers queue group → daemon verifies VPC exists in spinifex-vpc-vpcs KV → retrieves VPC CIDR → generates rtb-ID → creates RouteTableRecord with IsMain=false, local route pre-populated (destination=VPC CIDR, gateway="local"), empty associations → stores in spinifex-vpc-route-tables KV → returns RouteTable 1. Create route table with local route (happy path)
2. VPC not found (InvalidVpcID.NotFound)
3. Missing VpcId (MissingParameter)
DONE
delete-route-table --route-table-id --dry-run Route table must exist, not main, no subnet associations Gateway validates RouteTableId (required) → NATS ec2.DeleteRouteTable → daemon retrieves record → rejects if IsMain=true (DependencyViolation) → rejects if has non-main subnet associations (DependencyViolation) → deletes from KV → returns success 1. Delete custom route table
2. Delete main route table (DependencyViolation)
3. Delete table with subnet associations (DependencyViolation)
4. Delete non-existent (InvalidRouteTableID.NotFound)
DONE
describe-route-tables --route-table-ids, --filters (vpc-id, route-table-id, association.main, association.route-table-association-id, association.subnet-id, route.destination-cidr-block, route.gateway-id) --max-results, --next-token, --dry-run None NATS ec2.DescribeRouteTables → daemon scans all keys in spinifex-vpc-route-tables KV → filters by RouteTableIds if specified → applies all provided filters (AND logic) → returns error for non-existent requested IDs → converts records to EC2 RouteTable objects 1. List all route tables
2. Filter by vpc-id
3. Filter by association.main (true/false)
4. Non-existent ID (InvalidRouteTableID.NotFound)
5. Empty result when no route tables
DONE
create-route --route-table-id, --destination-cidr-block, --gateway-id, --nat-gateway-id --egress-only-internet-gateway-id, --vpc-peering-connection-id, --dry-run Route table and target must exist in same VPC Gateway validates RouteTableId + DestinationCidrBlock (required) → NATS ec2.CreateRoute → daemon retrieves route table → checks for duplicate destination (RouteAlreadyExists) → validates exactly one of GatewayId/NatGatewayId provided → if GatewayId: verifies IGW exists in spinifex-igw KV and is attached to same VPC → if NatGatewayId: verifies NAT GW exists in spinifex-vpc-nat-gateways KV and is in same VPC → for NAT GW routes, publishes vpc.add-nat-gateway event per associated subnet → appends RouteRecord (state=active, origin=CreateRoute) → updates KV → returns Return=true 1. Add IGW route (happy path)
2. Duplicate destination (RouteAlreadyExists)
3. IGW not found (InvalidInternetGatewayID.NotFound)
4. NAT GW not found (InvalidNatGatewayID.NotFound)
5. Gateway not in same VPC (InvalidParameterValue)
6. Missing RouteTableId or DestinationCidrBlock (MissingParameter)
DONE
delete-route --route-table-id, --destination-cidr-block --dry-run Route must exist Gateway validates RouteTableId + DestinationCidrBlock (required) → NATS ec2.DeleteRoute → daemon retrieves route table → finds matching route → rejects if gateway is "local" (InvalidParameterValue — local routes cannot be deleted) → removes route from Routes array → updates KV → returns success 1. Delete IGW route (happy path, local route remains)
2. Delete local route (InvalidParameterValue)
3. Route not found (InvalidRoute.NotFound)
4. Missing parameters (MissingParameter)
DONE
replace-route --route-table-id, --destination-cidr-block, --gateway-id --nat-gateway-id, --dry-run Route must exist, new target must exist in same VPC Gateway validates RouteTableId + DestinationCidrBlock + GatewayId (required) → NATS ec2.ReplaceRoute → daemon retrieves route table → finds matching route → rejects if local route → verifies IGW exists and is attached to same VPC → updates route GatewayId, clears NatGatewayId, sets state=active → updates KV → returns success 1. Replace route target (IGW swap)
2. Replace local route (InvalidParameterValue)
3. Route not found (InvalidRoute.NotFound)
4. IGW not found (InvalidInternetGatewayID.NotFound)
DONE
associate-route-table --route-table-id, --subnet-id --gateway-id, --dry-run Route table and subnet must exist in same VPC Gateway validates RouteTableId + SubnetId (required) → NATS ec2.AssociateRouteTable → daemon retrieves route table → verifies subnet exists and is in same VPC → scans ALL route tables in VPC to check subnet isn't already explicitly associated (Resource.AlreadyAssociated) → generates rtbassoc-ID → appends AssociationRecord (Main=false) → updates KV → returns AssociationId with state=associated 1. Associate subnet (happy path)
2. Duplicate subnet association (Resource.AlreadyAssociated)
3. Subnet not found (InvalidSubnetID.NotFound)
4. Subnet in different VPC (InvalidParameterValue)
5. Missing parameters (MissingParameter)
DONE
disassociate-route-table --association-id --dry-run Association must exist, not main table association Gateway validates AssociationId (required) → NATS ec2.DisassociateRouteTable → daemon scans ALL route tables in account to find matching AssociationId → rejects if Main=true (InvalidParameterValue — main associations are immutable) → removes association from Associations array → updates KV → returns success 1. Disassociate subnet (happy path)
2. Disassociate main table (InvalidParameterValue)
3. Association not found (InvalidAssociationID.NotFound)
DONE
replace-route-table-association --association-id, --route-table-id --dry-run Association and new route table must exist Gateway validates AssociationId + RouteTableId (required) → NATS ec2.ReplaceRouteTableAssociation → daemon verifies new route table exists → scans all route tables to find matching AssociationId → if Main=true: atomically swaps IsMain flag between old and new table → two-phase commit: removes association from old table, creates new AssociationId on new table → compensating write on failure (logs CRITICAL if compensation fails) → returns new AssociationId with state=associated 1. Move subnet between tables (happy path)
2. Replace main table association (swaps IsMain flags)
3. Association not found (InvalidAssociationID.NotFound)
4. Route table not found (InvalidRouteTableID.NotFound)
DONE

EC2 - VPC Peering

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-vpc-peering-connection — --vpc-id, --peer-vpc-id, --peer-owner-id, --peer-region, --tag-specifications Both VPCs must exist NATS ec2.CreateVpcPeeringConnection → daemon creates peering in KV with pcx-ID → state=pending-acceptance → return peering connection 1. Create peering between two VPCs
2. Missing VPC ID (error)
3. Overlapping CIDRs (error)
NOT STARTED
accept-vpc-peering-connection — --vpc-peering-connection-id, --dry-run Peering must exist in pending-acceptance state NATS ec2.AcceptVpcPeeringConnection → daemon updates state to active → configure OVS cross-VPC flows → return peering 1. Accept pending peering
2. Accept already active (error)
NOT STARTED
reject-vpc-peering-connection — --vpc-peering-connection-id, --dry-run Peering must exist in pending-acceptance state NATS ec2.RejectVpcPeeringConnection → daemon updates state to rejected → return peering 1. Reject pending peering
2. Reject already active (error)
NOT STARTED
delete-vpc-peering-connection — --vpc-peering-connection-id, --dry-run Peering must exist NATS ec2.DeleteVpcPeeringConnection → daemon removes OVS cross-VPC flows → delete from KV → return success 1. Delete active peering
2. Delete non-existent (error)
NOT STARTED
describe-vpc-peering-connections — --vpc-peering-connection-ids, --filters, --max-results None NATS ec2.DescribeVpcPeeringConnections → daemon lists peerings from KV → return list with state 1. List all peerings
2. Filter by ID
3. Filter by VPC
NOT STARTED
modify-vpc-peering-connection-options — --vpc-peering-connection-id, --requester-peering-connection-options, --accepter-peering-connection-options Peering must be active NATS ec2.ModifyVpcPeeringConnectionOptions → daemon updates peering options in KV → return modified options 1. Enable DNS resolution
2. Missing peering ID (error)
NOT STARTED

EC2 - VPC Endpoints

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-vpc-endpoint — --vpc-id, --service-name, --vpc-endpoint-type (Gateway/Interface), --route-table-ids, --subnet-ids, --tag-specifications VPC must exist NATS ec2.CreateVpcEndpoint → daemon creates endpoint in KV with vpce-ID → return endpoint 1. Create Gateway endpoint for S3
2. Create Interface endpoint
3. Missing VPC ID (error)
NOT STARTED
delete-vpc-endpoints — --vpc-endpoint-ids, --dry-run Endpoints must exist NATS ec2.DeleteVpcEndpoints → daemon deletes endpoints from KV → return success/failure per endpoint 1. Delete existing endpoint
2. Delete non-existent (error)
NOT STARTED
describe-vpc-endpoints — --vpc-endpoint-ids, --filters, --max-results None NATS ec2.DescribeVpcEndpoints → daemon lists endpoints from KV → return list with state and type 1. List all endpoints
2. Filter by VPC
3. Filter by service name
NOT STARTED
describe-vpc-endpoint-services — --service-names, --filters, --max-results None NATS ec2.DescribeVpcEndpointServices → daemon returns available services (S3, DynamoDB equivalents) → return service list 1. List available services
2. Filter by service name
NOT STARTED
modify-vpc-endpoint — --vpc-endpoint-id, --add-route-table-ids, --remove-route-table-ids, --add-subnet-ids, --remove-subnet-ids Endpoint must exist NATS ec2.ModifyVpcEndpoint → daemon updates endpoint route tables/subnets in KV → return success 1. Add route table
2. Remove subnet
3. Missing endpoint ID (error)
NOT STARTED

EC2 - VPN Gateway & Customer Gateway

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-customer-gateway — --type (ipsec.1), --bgp-asn, --ip-address, --tag-specifications None NATS ec2.CreateCustomerGateway → daemon stores in JetStream KV with cgw-ID → return customer gateway 1. Create with BGP ASN and IP
2. Missing required fields (error)
NOT STARTED
delete-customer-gateway — --customer-gateway-id, --dry-run Must exist, no VPN connections NATS ec2.DeleteCustomerGateway → daemon verifies no active connections → delete from KV → return success 1. Delete existing
2. Delete with active VPN (error)
NOT STARTED
describe-customer-gateways — --customer-gateway-ids, --filters None NATS ec2.DescribeCustomerGateways → daemon lists from KV → return list with BGP/IP info 1. List all
2. Filter by ID
NOT STARTED
create-vpn-gateway — --type (ipsec.1), --amazon-side-asn, --tag-specifications None NATS ec2.CreateVpnGateway → daemon stores in JetStream KV with vgw-ID → return VPN gateway 1. Create VPN gateway
2. Verify in describe
NOT STARTED
delete-vpn-gateway — --vpn-gateway-id, --dry-run Must be detached NATS ec2.DeleteVpnGateway → daemon verifies detached → delete from KV → return success 1. Delete detached gateway
2. Delete attached (error)
NOT STARTED
attach-vpn-gateway — --vpn-gateway-id, --vpc-id Both must exist NATS ec2.AttachVpnGateway → daemon links VPN gateway to VPC → return attachment 1. Attach to VPC
2. Already attached (error)
NOT STARTED
detach-vpn-gateway — --vpn-gateway-id, --vpc-id Must be attached NATS ec2.DetachVpnGateway → daemon unlinks VPN gateway from VPC → return success 1. Detach from VPC
2. Not attached (error)
NOT STARTED
describe-vpn-gateways — --vpn-gateway-ids, --filters None NATS ec2.DescribeVpnGateways → daemon lists from KV → return list with attachment info 1. List all
2. Filter by VPC attachment
NOT STARTED

EC2 - VPN Connections

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-vpn-connection — --type (ipsec.1), --customer-gateway-id, --vpn-gateway-id, --options, --tag-specifications Customer gateway and VPN gateway must exist NATS ec2.CreateVpnConnection → daemon stores connection in JetStream KV with vpn-ID → return connection with tunnel info 1. Create IPsec connection
2. Missing customer gateway (error)
3. Missing VPN gateway (error)
NOT STARTED
delete-vpn-connection — --vpn-connection-id, --dry-run Connection must exist NATS ec2.DeleteVpnConnection → daemon deletes from KV → return success 1. Delete existing connection
2. Delete non-existent (error)
NOT STARTED
describe-vpn-connections — --vpn-connection-ids, --filters None NATS ec2.DescribeVpnConnections → daemon lists from KV → return list with state and tunnel info 1. List all connections
2. Filter by ID
3. Filter by VPN gateway
NOT STARTED
modify-vpn-connection — --vpn-connection-id, --vpn-gateway-id, --customer-gateway-id Connection must exist NATS ec2.ModifyVpnConnection → daemon updates connection target in KV → return modified connection 1. Change VPN gateway
2. Missing connection ID (error)
NOT STARTED
modify-vpn-connection-options — --vpn-connection-id, --local-ipv4-network-cidr, --remote-ipv4-network-cidr Connection must exist NATS ec2.ModifyVpnConnectionOptions → daemon updates tunnel options in KV → return modified connection 1. Modify CIDR ranges
2. Missing connection ID (error)
NOT STARTED

EC2 - Network ACLs

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-network-acl — --vpc-id, --tag-specifications VPC must exist NATS ec2.CreateNetworkAcl → daemon creates NACL in KV with acl-ID → add default deny-all rules → return NACL 1. Create NACL in VPC
2. Missing VPC ID (error)
3. Verify default deny rules
NOT STARTED
delete-network-acl — --network-acl-id, --dry-run Must not be default NACL NATS ec2.DeleteNetworkAcl → daemon verifies not default → delete from KV → return success 1. Delete non-default NACL
2. Delete default NACL (error)
NOT STARTED
describe-network-acls — --network-acl-ids, --filters, --max-results None NATS ec2.DescribeNetworkAcls → daemon lists NACLs from KV → return list with entries 1. List all NACLs
2. Filter by VPC
NOT STARTED
create-network-acl-entry — --network-acl-id, --rule-number, --protocol, --rule-action (allow/deny), --cidr-block, --ingress/--egress, --port-range NACL must exist NATS ec2.CreateNetworkAclEntry → daemon adds rule entry to NACL in KV → configure OVS flow → return success 1. Add ingress allow rule
2. Add egress deny rule
3. ICMP type/code parsing
4. Missing NACL ID (error)
NOT STARTED
delete-network-acl-entry — --network-acl-id, --rule-number, --ingress/--egress Entry must exist NATS ec2.DeleteNetworkAclEntry → daemon removes rule from NACL → remove OVS flow → return success 1. Delete existing entry
2. Delete non-existent entry (error)
NOT STARTED
replace-network-acl-association — --association-id, --network-acl-id Both must exist NATS ec2.ReplaceNetworkAclAssociation → daemon swaps NACL for subnet → return new association ID 1. Replace association
2. Missing association ID (error)
NOT STARTED
replace-network-acl-entry — --network-acl-id, --rule-number, --protocol, --rule-action, --cidr-block, --ingress/--egress, --port-range Entry must exist NATS ec2.ReplaceNetworkAclEntry → daemon replaces rule in KV → update OVS flow → return success 1. Replace existing entry
2. Replace non-existent entry (error)
NOT STARTED

EC2 - Prefix Lists

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-managed-prefix-list — --prefix-list-name, --address-family (IPv4/IPv6), --max-entries, --entries, --tag-specifications None NATS ec2.CreateManagedPrefixList → daemon stores prefix list in JetStream KV with pl-ID → return prefix list with version 1. Create with IPv4 entries
2. Name, address family, max entries required
3. Verify in describe
NOT STARTED
delete-managed-prefix-list — --prefix-list-id, --dry-run Must exist, not referenced by security groups/route tables NATS ec2.DeleteManagedPrefixList → daemon verifies not referenced → delete from KV → return success 1. Delete unreferenced list
2. Delete referenced list (error)
NOT STARTED
describe-managed-prefix-lists — --prefix-list-ids, --filters, --max-results None NATS ec2.DescribeManagedPrefixLists → daemon lists from KV → return list with state and version 1. List all prefix lists
2. Filter by ID
NOT STARTED
modify-managed-prefix-list — --prefix-list-id, --current-version, --add-entries, --remove-entries, --prefix-list-name Must exist, current version must match NATS ec2.ModifyManagedPrefixList → daemon validates version → apply changes → increment version → return updated list 1. Add entries
2. Remove entries
3. Version mismatch (error)
NOT STARTED
get-managed-prefix-list-entries — --prefix-list-id, --target-version, --max-results Prefix list must exist NATS ec2.GetManagedPrefixListEntries → daemon reads entries from KV → return entry list 1. List entries
2. List entries at specific version
NOT STARTED
get-managed-prefix-list-associations — --prefix-list-id, --max-results Prefix list must exist NATS ec2.GetManagedPrefixListAssociations → daemon finds references in SGs/route tables → return association list 1. List associations
2. No associations returns empty
NOT STARTED

EC2 - Network Interfaces

ENIs are managed via spinifex-vpc-enis KV bucket with CAS-based IPAM allocation from spinifex-vpc-ipam. ENIs are auto-created by run-instances --subnet-id and auto-deleted on termination. MAC addresses are deterministically generated from the ENI ID. Attach/detach are internal operations used by instance lifecycle — no standalone attach/detach API yet.

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-network-interface --subnet-id, --private-ip-address, --description, --tag-specifications --groups, --dry-run Subnet must exist Gateway validates SubnetId (required) → NATS ec2.CreateNetworkInterface → daemon verifies subnet exists in spinifex-vpc-subnets KV → if no PrivateIpAddress, allocates from IPAM pool (CAS-based, up to 5 retries) → generates eni-ID → generates deterministic MAC from ENI ID → stores ENIRecord in spinifex-vpc-enis KV → publishes vpc.create-port event to vpcd (creates OVN logical port) → returns NetworkInterface with status=available 1. Create in subnet (auto-allocate IP)
2. Create with specific private IP
3. Missing subnet ID (MissingParameter)
4. Non-existent subnet (error)
5. Tags applied at creation
DONE
delete-network-interface --network-interface-id --dry-run Must not be attached (status != in-use) NATS ec2.DeleteNetworkInterface → daemon verifies ENI exists and status is not "in-use" (InvalidNetworkInterfaceInUse) → releases IP back to IPAM pool → deletes from spinifex-vpc-enis KV → publishes vpc.delete-port event to vpcd 1. Delete detached ENI
2. Delete attached ENI (InvalidNetworkInterfaceInUse)
3. Delete non-existent (error)
DONE
describe-network-interfaces --network-interface-ids, --filters (subnet-id, vpc-id, attachment.instance-id) --max-results, --dry-run None NATS ec2.DescribeNetworkInterfaces → daemon lists all keys from spinifex-vpc-enis KV → applies filters (subnet-id, vpc-id, attachment.instance-id) → filters by ENI IDs if specified → returns error for non-existent requested IDs 1. List all ENIs
2. Filter by subnet-id
3. Filter by vpc-id
4. Filter by attachment.instance-id
5. Non-existent ENI returns error
DONE
attach-network-interface — --network-interface-id, --instance-id, --device-index ENI and instance must exist Internal operation used by run-instances --subnet-id. Marks ENI as "in-use", sets instance attachment, generates attachment ID. No standalone API endpoint yet. 1. Attach ENI to instance
2. Already attached (error)
3. Missing device index (error)
NOT STARTED (internal only)
detach-network-interface — --attachment-id, --force ENI must be attached Internal operation used by terminate-instances. Marks ENI as "available", clears attachment info. No standalone API endpoint yet. 1. Detach ENI
2. Force detach
3. Not attached (error)
NOT STARTED (internal only)
modify-network-interface-attribute — --network-interface-id, --description, --groups ENI must exist; at least one attribute required NATS ec2.ModifyNetworkInterfaceAttribute → daemon validates ENI exists → updates description and/or security groups in spinifex-vpc-enis KV (optimistic locking) → return success 1. Modify description
2. Update security groups
3. No attributes (InvalidParameterValue)
4. Non-existent ENI (error)
DONE
assign-private-ip-addresses — --network-interface-id, --private-ip-addresses or --secondary-private-ip-address-count ENI must exist NATS ec2.AssignPrivateIpAddresses → daemon allocates IPs from subnet pool → assign to ENI → return assigned IPs 1. Assign specific IPs
2. Assign by count
3. Cannot specify both (error)
NOT STARTED
unassign-private-ip-addresses — --network-interface-id, --private-ip-addresses IPs must be assigned to ENI NATS ec2.UnassignPrivateIpAddresses → daemon releases IPs → return success 1. Unassign secondary IP
2. Unassign primary IP (error)
NOT STARTED

EC2 - Launch Templates

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-launch-template — --launch-template-name, --launch-template-data (ImageId, InstanceType, etc.), --tag-specifications None NATS ec2.CreateLaunchTemplate → daemon stores template in JetStream KV with lt-ID → version 1 → return template metadata 1. Create template with full config
2. Duplicate name (error)
3. Verify in describe output
NOT STARTED
create-launch-template-version — --launch-template-id or --launch-template-name, --launch-template-data, --source-version Template must exist NATS ec2.CreateLaunchTemplateVersion → daemon creates new version in KV → return version details 1. Create version from scratch
2. Create version from source version
3. Template not found (error)
NOT STARTED
delete-launch-template — --launch-template-id or --launch-template-name, --dry-run Template must exist NATS ec2.DeleteLaunchTemplate → daemon deletes template and all versions from KV → return deleted template info 1. Delete by ID
2. Delete by name
3. Non-existent template (error)
NOT STARTED
describe-launch-templates — --launch-template-ids, --launch-template-names, --filters None NATS ec2.DescribeLaunchTemplates → daemon lists templates from KV → return list 1. List all templates
2. Filter by name
3. Filter by ID
NOT STARTED
describe-launch-template-versions — --launch-template-id or --launch-template-name, --versions, --min-version, --max-version Template must exist NATS ec2.DescribeLaunchTemplateVersions → daemon lists versions from KV → return version list with data 1. List all versions
2. List specific version
3. Template not found (error)
NOT STARTED

EC2 - Misc Operations

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
delete-network-interface-permission — --network-interface-permission-id, --force Permission must exist NATS ec2.DeleteNetworkInterfacePermission → daemon deletes permission from KV → return success 1. Delete existing permission
2. Force delete
NOT STARTED
enable-address-transfer — --allocation-id, --transfer-account-id EIP must exist NATS ec2.EnableAddressTransfer → daemon stores transfer in JetStream KV with 7-day expiration → return AllocationId and TransferAccountId 1. Enable transfer
2. Verify transfer stored with expiration
NOT STARTED
disable-address-transfer — --allocation-id Transfer must exist NATS ec2.DisableAddressTransfer → daemon updates transfer status to disabled in KV → return status 1. Disable active transfer
2. Disable non-existent transfer (error)
NOT STARTED

EBS Direct API

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
start-snapshot — --volume-size, --parent-snapshot-id, --description, --encrypted None Initialize snapshot in viperblock → return snap-ID for block-level writes 1. Start new snapshot
2. Start incremental from parent
NOT STARTED
put-snapshot-block — --snapshot-id, --block-index, --block-data, --checksum start-snapshot Write block data to snapshot at specified index in viperblock → verify checksum 1. Write single block
2. Write with bad checksum (error)
NOT STARTED
get-snapshot-block — --snapshot-id, --block-index Snapshot must exist Read block data from snapshot at specified index → return data with checksum 1. Read existing block
2. Read sparse block (zeros)
NOT STARTED
complete-snapshot — --snapshot-id, --changed-blocks-count start-snapshot, put-snapshot-block Finalize snapshot → mark as completed → sync to Predastore 1. Complete valid snapshot
2. Complete with wrong block count (error)
NOT STARTED
list-snapshot-blocks — --snapshot-id, --max-results, --next-token Snapshot must exist List all non-empty blocks in snapshot with tokens and sizes 1. List blocks of populated snapshot
2. List blocks of empty snapshot
NOT STARTED
list-changed-blocks — --second-snapshot-id, --first-snapshot-id, --max-results Snapshots must exist Compare two snapshots → return list of differing block indexes 1. Compare parent and child snapshots
2. Compare unrelated snapshots
NOT STARTED

IAM - User Management

All IAM operations are account-scoped. Users, policies, and access keys are namespaced per account. The gateway extracts the account ID from the authenticated access key via c.Locals("sigv4.accountId") and passes it to all IAM service methods. Root user (account 000000000000) bypasses policy evaluation entirely.

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-user --user-name, --path --tags, --permissions-boundary None Gateway validates UserName non-empty → IAMService.CreateUser(accountID, input) → generates AIDA-prefixed UserId → stores in KV at key {accountID}.{userName} → ARN includes real account ID → conditional create for race safety 1. Create user
2. Duplicate name in same account (EntityAlreadyExists)
3. Same name in different account (succeeds)
4. Missing username (InvalidInput)
5. Default path is /
DONE
get-user --user-name — User must exist in caller's account Gateway validates UserName → IAMService.GetUser(accountID, input) → reads from KV at {accountID}.{userName} → returns User with ARN, Path, CreateDate 1. Get existing user
2. Get non-existent user (NoSuchEntity)
3. Cannot see user in different account (NoSuchEntity)
DONE
list-users --path-prefix --max-items, --marker None IAMService.ListUsers(accountID, input) → prefix-scans KV for {accountID}.* → filters by path prefix if specified → returns user list scoped to caller's account 1. List all users in account
2. List with path prefix filter
3. Empty list
4. Does not show other accounts' users
DONE
delete-user --user-name — User must exist, no access keys attached Gateway validates UserName → IAMService.DeleteUser(accountID, input) → checks user has no access keys → deletes from KV 1. Delete user with no keys
2. Delete user with keys attached (DeleteConflict)
3. Delete non-existent user (NoSuchEntity)
DONE

IAM - Access Key Management

Access keys are globally unique (keyed by AccessKeyID) but associated with an account. Max 2 keys per user. Secrets encrypted with AES-256-GCM using master key.

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-access-key --user-name — User must exist IAMService.CreateAccessKey(accountID, input) → generates AKIA-prefixed key ID + random secret → encrypts secret with master key → stores in KV (globally unique key ID) → updates user's AccessKeys list → returns plaintext secret (one-time only) 1. Create key for user (AKIA prefix)
2. Max 2 keys per user (LimitExceeded on 3rd)
3. Non-existent user (NoSuchEntity)
4. Secret only returned on creation
DONE
list-access-keys --user-name --max-items, --marker None IAMService.ListAccessKeys(accountID, input) → reads user's AccessKeys list → fetches each key's metadata → returns key IDs and status (no secrets) 1. List keys for user
2. No keys returns empty list
3. Shows Status (Active/Inactive) but not secrets
DONE
delete-access-key --access-key-id, --user-name — Key must exist IAMService.DeleteAccessKey(accountID, input) → removes key from global KV → removes key ID from user's AccessKeys list 1. Delete existing key
2. Delete non-existent key (NoSuchEntity)
DONE
update-access-key --access-key-id, --user-name, --status (Active/Inactive) — Key must exist IAMService.UpdateAccessKey(input) → reads key from global KV (key ID is globally unique, no accountID needed) → updates Status field → inactive keys rejected at SigV4 auth 1. Deactivate key (Inactive)
2. Reactivate key (Active)
3. Non-existent key (NoSuchEntity)
4. Inactive key → auth returns InvalidClientTokenId
DONE

IAM - Policy Management

Policies are account-scoped. KV key is {accountID}.{policyName}. Policy documents require Version: "2012-10-17" and valid Statement array. Supports wildcard action matching (ec2:*, ec2:Describe*, *). Evaluation order: explicit Deny > explicit Allow > implicit Deny.

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-policy --policy-name, --policy-document, --path, --description --tags None Gateway validates PolicyName and PolicyDocument non-empty → IAMService.CreatePolicy(accountID, input) → validates JSON (Version, Statement with Effect/Action/Resource) → generates ANPA-prefixed PolicyId → stores at {accountID}.{policyName} → ARN includes real account ID 1. Create with valid document
2. Malformed JSON (MalformedPolicyDocument)
3. Missing Effect/Action/Resource (MalformedPolicyDocument)
4. Duplicate name in same account (EntityAlreadyExists)
5. Same name in different account (succeeds)
6. With path and description
DONE
get-policy --policy-arn — Policy must exist Gateway validates PolicyArn → extracts account ID and policy name from ARN → IAMService.GetPolicy(accountID, input) → returns policy metadata (no document) with AttachmentCount 1. Get existing policy
2. Non-existent policy (NoSuchEntity)
3. Returns metadata only, not document
DONE
get-policy-version --policy-arn, --version-id — Policy must exist Gateway validates PolicyArn and VersionId → IAMService.GetPolicyVersion(accountID, input) → returns PolicyVersion with URL-encoded Document, VersionId, IsDefaultVersion 1. Get version v1 (returns document)
2. Non-existent version (NoSuchEntity)
3. Non-existent policy (NoSuchEntity)
DONE
list-policies — --scope, --only-attached, --path-prefix, --max-items, --marker None IAMService.ListPolicies(accountID, input) → prefix-scans KV for {accountID}.* → returns all policies in caller's account 1. List all policies in account
2. Empty list
3. Does not show other accounts' policies
DONE
delete-policy --policy-arn — Policy must exist, not attached to any user Gateway validates PolicyArn → IAMService.DeletePolicy(accountID, input) → checks no users have this policy attached (scans account's users) → deletes from KV 1. Delete unattached policy
2. Delete attached policy (DeleteConflict)
3. Delete non-existent (NoSuchEntity)
DONE

IAM - Policy Attachment

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
attach-user-policy --user-name, --policy-arn — User and policy must exist in caller's account Gateway validates both params → IAMService.AttachUserPolicy(accountID, input) → verifies policy exists → adds policy ARN to user's AttachedPolicies list → idempotent (re-attach succeeds without duplicate) 1. Attach policy to user
2. Already attached (idempotent, no error)
3. Non-existent user (NoSuchEntity)
4. Non-existent policy (NoSuchEntity)
DONE
detach-user-policy --user-name, --policy-arn — Policy must be attached to user Gateway validates both params → IAMService.DetachUserPolicy(accountID, input) → removes policy ARN from user's AttachedPolicies list 1. Detach attached policy
2. Policy not attached (NoSuchEntity)
3. Non-existent user (NoSuchEntity)
DONE
list-attached-user-policies --user-name --path-prefix, --max-items, --marker User must exist IAMService.ListAttachedUserPolicies(accountID, input) → reads user's AttachedPolicies list → returns policy ARNs and names 1. List attached policies
2. No policies returns empty list
DONE

IAM - Policy Evaluation

The gateway evaluates IAM policies on every request (except root user bypass). Policy evaluation is internal — not an AWS API.

Feature Description Status
Root user bypass Root user (000000000000:root) skips all policy checks DONE
Default deny Users with no attached policies are denied all actions DONE
Explicit allow Matching Allow statement grants access DONE
Explicit deny Deny statements override Allow (deny wins) DONE
Wildcard matching * matches all, ec2:* matches all EC2, ec2:Describe* matches prefix DONE
Account-scoped evaluation Policies resolved from caller's account context DONE
EC2 action mapping All registered EC2 actions mapped to IAM policy strings DONE
IAM action mapping 16 IAM actions mapped to IAM policy strings DONE

IAM - Roles

IAM roles are required for instance profiles, cross-account access, and service-linked permissions. Without roles, workloads must use static access keys. Roles should be stored in a spinifex-iam-roles KV bucket keyed by {accountID}.{roleName}. Trust policies (AssumeRolePolicyDocument) define who can assume the role.

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-role — --role-name, --assume-role-policy-document, --path, --description, --max-session-duration, --permissions-boundary, --tags None IAMService.CreateRole(accountID, input) → validate RoleName + AssumeRolePolicyDocument (required) → validate trust policy JSON → generate AROA-prefixed RoleId → store RoleRecord in spinifex-iam-roles KV at {accountID}.{roleName} → ARN: arn:aws:iam::{accountID}:role/{path}{roleName} → return Role 1. Create with valid trust policy
2. Malformed trust policy (MalformedPolicyDocument)
3. Duplicate name (EntityAlreadyExists)
4. Missing RoleName (InvalidInput)
5. With path and description
NOT STARTED
get-role — --role-name Role must exist IAMService.GetRole(accountID, input) → reads from KV at {accountID}.{roleName} → returns Role with ARN, Path, AssumeRolePolicyDocument, CreateDate 1. Get existing role
2. Non-existent role (NoSuchEntity)
NOT STARTED
list-roles — --path-prefix, --max-items, --marker None IAMService.ListRoles(accountID, input) → prefix-scans KV for {accountID}.* → filters by path prefix → returns role list scoped to caller's account 1. List all roles
2. List with path prefix
3. Empty list
4. Account isolation
NOT STARTED
delete-role — --role-name Role must exist, no attached policies, no instance profiles IAMService.DeleteRole(accountID, input) → verify no attached policies (DeleteConflict) → verify not in any instance profile (DeleteConflict) → delete from KV 1. Delete role with no attachments
2. Delete role with policies (DeleteConflict)
3. Delete role in instance profile (DeleteConflict)
4. Non-existent role (NoSuchEntity)
NOT STARTED
update-role — --role-name, --description, --max-session-duration Role must exist IAMService.UpdateRole(accountID, input) → update description and/or session duration in KV → return success 1. Update description
2. Update max session duration
3. Non-existent role (NoSuchEntity)
NOT STARTED
update-assume-role-policy — --role-name, --policy-document Role must exist IAMService.UpdateAssumeRolePolicy(accountID, input) → validate trust policy JSON → update AssumeRolePolicyDocument in KV → return success 1. Update trust policy
2. Malformed policy (MalformedPolicyDocument)
3. Non-existent role (NoSuchEntity)
NOT STARTED
attach-role-policy — --role-name, --policy-arn Role and policy must exist IAMService.AttachRolePolicy(accountID, input) → verify policy exists → add ARN to role's AttachedPolicies → idempotent 1. Attach policy
2. Already attached (idempotent)
3. Non-existent role (NoSuchEntity)
4. Non-existent policy (NoSuchEntity)
NOT STARTED
detach-role-policy — --role-name, --policy-arn Policy must be attached IAMService.DetachRolePolicy(accountID, input) → remove ARN from role's AttachedPolicies 1. Detach attached policy
2. Not attached (NoSuchEntity)
3. Non-existent role (NoSuchEntity)
NOT STARTED
list-attached-role-policies — --role-name, --path-prefix, --max-items, --marker Role must exist IAMService.ListAttachedRolePolicies(accountID, input) → reads role's AttachedPolicies list → returns ARNs and names 1. List attached policies
2. No policies (empty list)
3. Non-existent role (NoSuchEntity)
NOT STARTED

IAM - Instance Profiles

Instance profiles are containers for IAM roles that allow EC2 instances to assume a role and obtain temporary credentials. Required for run-instances --iam-instance-profile. Stored in spinifex-iam-instance-profiles KV bucket keyed by {accountID}.{profileName}.

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-instance-profile — --instance-profile-name, --path, --tags None IAMService.CreateInstanceProfile(accountID, input) → validate name (required) → generate AIPA-prefixed InstanceProfileId → store in spinifex-iam-instance-profiles KV with empty Roles array → ARN: arn:aws:iam::{accountID}:instance-profile/{path}{name} → return InstanceProfile 1. Create profile
2. Duplicate name (EntityAlreadyExists)
3. Missing name (InvalidInput)
NOT STARTED
get-instance-profile — --instance-profile-name Profile must exist IAMService.GetInstanceProfile(accountID, input) → reads from KV → returns InstanceProfile with Roles array 1. Get existing profile
2. Non-existent (NoSuchEntity)
NOT STARTED
list-instance-profiles — --path-prefix, --max-items, --marker None IAMService.ListInstanceProfiles(accountID, input) → prefix-scans KV → returns list 1. List all profiles
2. Empty list
NOT STARTED
list-instance-profiles-for-role — --role-name, --max-items, --marker Role must exist IAMService.ListInstanceProfilesForRole(accountID, input) → scan all profiles → filter where Roles contains role name → return list 1. List profiles for role
2. Role not in any profile (empty)
NOT STARTED
delete-instance-profile — --instance-profile-name Profile must exist, Roles must be empty IAMService.DeleteInstanceProfile(accountID, input) → verify Roles empty (DeleteConflict if not) → delete from KV 1. Delete empty profile
2. Delete profile with role (DeleteConflict)
3. Non-existent (NoSuchEntity)
NOT STARTED
add-role-to-instance-profile — --instance-profile-name, --role-name Both must exist, max 1 role per profile IAMService.AddRoleToInstanceProfile(accountID, input) → verify profile exists → verify role exists → check Roles array is empty (LimitExceeded if role already present) → append role → update KV 1. Add role to empty profile
2. Profile already has role (LimitExceeded)
3. Non-existent profile (NoSuchEntity)
4. Non-existent role (NoSuchEntity)
NOT STARTED
remove-role-from-instance-profile — --instance-profile-name, --role-name Role must be in profile IAMService.RemoveRoleFromInstanceProfile(accountID, input) → remove role from Roles array → update KV 1. Remove role
2. Role not in profile (NoSuchEntity)
NOT STARTED

IAM - Groups

Groups provide a way to attach policies to multiple users. Stored in spinifex-iam-groups KV bucket keyed by {accountID}.{groupName}.

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-group — --group-name, --path None IAMService.CreateGroup(accountID, input) → validate GroupName (required) → generate AGPA-prefixed GroupId → store in KV → return Group 1. Create group
2. Duplicate name (EntityAlreadyExists)
3. Missing name (InvalidInput)
NOT STARTED
get-group — --group-name Group must exist IAMService.GetGroup(accountID, input) → reads group from KV → returns Group with member Users list 1. Get existing group
2. Non-existent (NoSuchEntity)
NOT STARTED
list-groups — --path-prefix, --max-items, --marker None IAMService.ListGroups(accountID, input) → prefix-scans KV → returns list 1. List all groups
2. Empty list
NOT STARTED
delete-group — --group-name Group must exist, no members, no attached policies IAMService.DeleteGroup(accountID, input) → verify no members → verify no attached policies → delete from KV 1. Delete empty group
2. Delete group with members (DeleteConflict)
3. Delete group with policies (DeleteConflict)
4. Non-existent (NoSuchEntity)
NOT STARTED
add-user-to-group — --group-name, --user-name Both must exist IAMService.AddUserToGroup(accountID, input) → verify group + user exist → add user to group's Members list → idempotent 1. Add user
2. Already a member (idempotent)
3. Non-existent group (NoSuchEntity)
4. Non-existent user (NoSuchEntity)
NOT STARTED
remove-user-from-group — --group-name, --user-name User must be in group IAMService.RemoveUserFromGroup(accountID, input) → remove user from Members list → update KV 1. Remove member
2. Not a member (NoSuchEntity)
NOT STARTED
list-groups-for-user — --user-name, --max-items, --marker User must exist IAMService.ListGroupsForUser(accountID, input) → scan all groups → filter where Members contains user → return list 1. List groups for user
2. User in no groups (empty)
NOT STARTED
attach-group-policy — --group-name, --policy-arn Group and policy must exist IAMService.AttachGroupPolicy(accountID, input) → verify policy exists → add ARN to group's AttachedPolicies → idempotent 1. Attach policy
2. Already attached (idempotent)
3. Non-existent group (NoSuchEntity)
NOT STARTED
detach-group-policy — --group-name, --policy-arn Policy must be attached IAMService.DetachGroupPolicy(accountID, input) → remove ARN from AttachedPolicies 1. Detach policy
2. Not attached (NoSuchEntity)
NOT STARTED
list-attached-group-policies — --group-name, --path-prefix, --max-items, --marker Group must exist IAMService.ListAttachedGroupPolicies(accountID, input) → reads group's AttachedPolicies → returns ARNs and names 1. List attached policies
2. No policies (empty)
NOT STARTED

STS (Security Token Service)

STS provides temporary credentials and identity verification. GetCallerIdentity is called by every AWS SDK/CLI invocation. AssumeRole is required once IAM roles are implemented. STS needs its own gateway service endpoint (sts service in the supportedServices map) with a dedicated action map.

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
get-caller-identity — (no flags) Valid SigV4 credentials Gateway extracts accountID and username from authenticated request → returns Account, Arn (arn:aws:iam::{accountID}:user/{userName} or arn:aws:sts::{accountID}:assumed-role/{roleName}/{sessionName}), UserId. Note: a Spinifex-custom version exists at spinifex.GetCallerIdentity but needs a proper STS endpoint. 1. Get identity for IAM user
2. Get identity for assumed role
3. Get identity for root user
4. Invalid credentials (AuthFailure)
NOT STARTED
assume-role — --role-arn, --role-session-name, --duration-seconds, --policy, --external-id, --serial-number, --token-code, --tags, --transitive-tag-keys Role must exist, caller must be allowed by trust policy Gateway validates RoleArn + RoleSessionName (required) → looks up role → evaluates trust policy (AssumeRolePolicyDocument) against caller → generates temporary credentials (AccessKeyId with ASIA prefix, SecretAccessKey, SessionToken) → credentials expire after DurationSeconds (default 3600, max 43200) → return Credentials + AssumedRoleUser (ARN, AssumedRoleId) 1. Assume role with valid trust policy
2. Trust policy denies caller (AccessDenied)
3. Non-existent role (NoSuchEntity)
4. Expired session token rejected at SigV4
5. Custom duration
6. Session tags
NOT STARTED
get-session-token — --duration-seconds, --serial-number, --token-code Valid IAM user credentials (not role) Gateway validates caller is IAM user (not assumed role) → generates temporary credentials (ASIA prefix) → session inherits user's permissions → return Credentials 1. Get token for IAM user
2. Duration within range
3. Reject if caller is assumed role
NOT STARTED

ELBv2 (Elastic Load Balancing v2 — Application & Network Load Balancer)

All ELBv2 resources are stored in the spinifex-elbv2 JetStream KV bucket. Key format: lb.{lbId}, tg.{tgId}, listener.{listenerId}. The data plane uses a system-managed LB VM running HAProxy, launched automatically during CreateLoadBalancer. HAProxy configuration is pushed via NATS (elbv2.lb.{lbId}.config topic) whenever listeners or targets change. Agent readiness is polled via elbv2.lb.{lbId}.ping.

ELBv2 - Load Balancers

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-load-balancer --name, --subnets, --security-groups, --scheme (internet-facing/internal), --tags, --ip-address-type (ipv4 only) --type (hardcoded application), --customer-owned-ipv4-pool, --dry-run None (subnets optional) Gateway validates Name (required) → NATS elbv2.CreateLoadBalancer with spinifex-workers queue group → daemon validates scheme (internet-facing or internal, InvalidScheme if other) → checks duplicate name in account (DuplicateLoadBalancerName) → generates LB ID → builds DNS name {name}-{lbId}.{region}.elb.spinifex.local → if subnets provided, creates one ENI per subnet with spinifex:managed-by=elbv2 tag (rolls back on failure) → if system AMI available, launches sys.micro LB VM with cloud-init running /usr/local/bin/lb-agent → state=provisioning → background goroutine polls agent ping for ~5min → on agent ready: state=active, on timeout: state=failed → if no VM launcher: state=active immediately → persists to KV → returns LoadBalancer with ARN, DNSName, State 1. Create with name (happy path)
2. Internal scheme
3. Invalid scheme (InvalidScheme)
4. Duplicate name (DuplicateLoadBalancerName)
5. Missing name (MissingParameter)
6. With tags
7. With subnets (ENI creation)
DONE
delete-load-balancer --load-balancer-arn --dry-run LB must exist Gateway validates LoadBalancerArn (required) → NATS elbv2.DeleteLoadBalancer → daemon verifies LB exists and belongs to account → terminates LB VM (async background) → deletes all listeners for this LB → deletes system-managed ENIs → deletes LB record from KV → returns success 1. Delete existing LB
2. Not found (LoadBalancerNotFound)
3. Cascades listener cleanup
4. ENI cleanup
DONE
describe-load-balancers --load-balancer-arns, --names --page-size, --marker, --dry-run None NATS elbv2.DescribeLoadBalancers → daemon lists all LB records from KV → filters by account ID → applies ARN filter and/or Name filter → returns LoadBalancer list with State, DNSName, Scheme, VpcId, AvailabilityZones 1. List all LBs
2. Filter by name
3. Account isolation
DONE

ELBv2 - Target Groups

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-target-group --name, --protocol (HTTP), --port (default 80), --vpc-id, --target-type (instance only), --health-check-protocol, --health-check-port, --health-check-path (default /), --health-check-interval-seconds (default 30), --health-check-timeout-seconds (default 5), --healthy-threshold-count (default 5), --unhealthy-threshold-count (default 2), --matcher (HttpCode, default 200), --tags --health-check-enabled, --protocol-version, --ip-address-type, --dry-run None Gateway validates Name (required) → NATS elbv2.CreateTargetGroup → daemon checks duplicate name in same VPC (DuplicateTargetGroupName) → generates TG ID → creates HealthCheckConfig with provided values or defaults → stores record with empty Targets array → returns TargetGroup with ARN 1. Create with defaults
2. Custom health check
3. Duplicate name (DuplicateTargetGroupName)
4. Missing name (MissingParameter)
DONE
delete-target-group --target-group-arn --dry-run TG must exist, not referenced by listener Gateway validates TargetGroupArn (required) → NATS elbv2.DeleteTargetGroup → daemon verifies TG exists → scans all listeners to check if any DefaultActions reference this TG (TargetGroupInUse if referenced) → deletes from KV 1. Delete TG
2. TG in use by listener (TargetGroupInUse)
3. Not found (TargetGroupNotFound)
DONE
describe-target-groups --target-group-arns, --names, --load-balancer-arn --page-size, --marker, --dry-run None NATS elbv2.DescribeTargetGroups → daemon lists all TG records → filters by account → applies ARN/Name filters → if LoadBalancerArn specified, lists listeners for that LB and includes only TGs referenced by those listeners → returns TargetGroup list 1. List all TGs
2. Filter by LB ARN (via listener lookup)
3. Account isolation
DONE

ELBv2 - Targets

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
register-targets --target-group-arn, --targets (Id=instance-id, Port=optional override) --dry-run TG must exist, instances should exist Gateway validates TargetGroupArn + Targets (required) → NATS elbv2.RegisterTargets → daemon fetches TG → deduplicates against existing targets (key: {id}:{port}, idempotent) → resolves instance ID → private IP via VPC ENI lookup → adds target with state=initial → persists to KV → calls pushConfigForTargetGroup() to regenerate HAProxy config on all LBs using this TG → returns success 1. Register targets
2. Idempotent re-registration
3. TG not found (TargetGroupNotFound)
4. Missing ARN or targets (MissingParameter)
DONE
deregister-targets --target-group-arn, --targets (Id, Port) --dry-run TG must exist Gateway validates TargetGroupArn + Targets (required) → NATS elbv2.DeregisterTargets → daemon fetches TG → builds removal set (key: {id}:{port}) → filters Targets array to keep only non-matching → persists to KV → calls pushConfigForTargetGroup() to update HAProxy config → returns success 1. Deregister targets
2. TG not found (TargetGroupNotFound)
3. Missing parameters (MissingParameter)
DONE
describe-target-health --target-group-arn, --targets (optional filter to specific targets) --include TG must exist Gateway validates TargetGroupArn (required) → NATS elbv2.DescribeTargetHealth → daemon fetches TG → builds filter set from Targets param (if specified) → for each target, returns TargetHealthDescription with Target (Id, Port) and TargetHealth (State, Description). Note: health state is stored in records, not actively polled — HAProxy runs its own independent health checks. 1. Describe all targets
2. Filter to specific targets
3. TG not found (TargetGroupNotFound)
DONE

ELBv2 - Listeners

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-listener --load-balancer-arn, --default-actions (Type=forward, TargetGroupArn), --protocol (HTTP, default), --port (default 80) --ssl-policy, --certificates, --alpn-policy, --mutual-authentication, --dry-run LB must exist Gateway validates LoadBalancerArn + DefaultActions (required) → NATS elbv2.CreateListener → daemon verifies LB exists → checks for duplicate port on same LB (DuplicateListener) → generates listener ID → builds ARN → stores ListenerRecord in KV → calls pushConfig(lb) to regenerate HAProxy config (creates frontend for listener port, backend for target group with health checks) → returns Listener with ARN, Protocol, Port, DefaultActions 1. Create listener
2. Duplicate port (DuplicateListener)
3. LB not found (LoadBalancerNotFound)
4. Graceful handling when NATS unavailable
DONE
delete-listener --listener-arn --dry-run Listener must exist Gateway validates ListenerArn (required) → NATS elbv2.DeleteListener → daemon verifies listener exists and belongs to account → deletes from KV → fetches parent LB → calls pushConfig(lb) to regenerate HAProxy config without this listener → returns success 1. Delete listener
2. Not found (ListenerNotFound)
DONE
describe-listeners --load-balancer-arn, --listener-arns --page-size, --marker, --dry-run None NATS elbv2.DescribeListeners → daemon lists listeners → if LoadBalancerArn specified, filters to that LB → applies ARN filter → account isolation → returns Listener list 1. Filter by LB ARN
2. Account isolation
DONE

ELBv2 - Not Implemented

Feature Description Priority Status
Listener rules Path-based and hostname-based routing (CreateRule, DeleteRule, DescribeRules, ModifyRule) High NOT STARTED
HTTPS/TLS termination SSL certificates, SSL policies, ALPN High NOT STARTED
Modify operations ModifyLoadBalancerAttributes, ModifyTargetGroup, ModifyTargetGroupAttributes, ModifyListener Medium NOT STARTED
Connection draining Deregistration delay for graceful target removal Medium NOT STARTED
Stickiness Session affinity / sticky sessions on target groups Medium NOT STARTED
Active health checking API-driven health state updates (currently HAProxy-only) Medium NOT STARTED
Target types IP and Lambda target types (only instance supported) Low NOT STARTED
Access logs S3 access log delivery Low NOT STARTED
WAF integration AWS WAF association for web application firewall Low NOT STARTED

CloudWatch (Basic Monitoring)

Basic CloudWatch metrics support is required for monitor-instances/unmonitor-instances to be meaningful. Daemon nodes should publish CPU, disk, and network metrics for running instances. Metrics stored in a time-series-friendly structure in JetStream KV or dedicated NATS subjects.

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
put-metric-data — --namespace, --metric-data (MetricName, Value, Unit, Timestamp, Dimensions) None Gateway validates Namespace + MetricData (required) → NATS cloudwatch.PutMetricData → daemon stores metric data points in time-series KV (key: {namespace}.{metricName}.{dimensionHash}) → return success 1. Put single metric
2. Put with dimensions (InstanceId)
3. Missing namespace (MissingParameter)
NOT STARTED
get-metric-statistics — --namespace, --metric-name, --start-time, --end-time, --period, --statistics (Average, Sum, Min, Max, SampleCount), --dimensions None Gateway validates required fields → NATS cloudwatch.GetMetricStatistics → daemon queries time-series data → aggregates by period → returns Datapoints with requested statistics 1. Get CPU utilization over 1 hour
2. Get with dimensions (InstanceId)
3. No data returns empty
NOT STARTED
list-metrics — --namespace, --metric-name, --dimensions, --recently-active None NATS cloudwatch.ListMetrics → daemon scans metric namespaces → returns Metrics list with Namespace, MetricName, Dimensions 1. List all metrics
2. Filter by namespace
3. Filter by metric name
NOT STARTED
describe-alarms — --alarm-names, --alarm-name-prefix, --state-value, --action-prefix None NATS cloudwatch.DescribeAlarms → daemon lists alarms from KV → return MetricAlarms with State, Threshold, ComparisonOperator 1. List all alarms
2. Filter by state (OK, ALARM, INSUFFICIENT_DATA)
NOT STARTED
put-metric-alarm — --alarm-name, --namespace, --metric-name, --statistic, --period, --evaluation-periods, --threshold, --comparison-operator, --alarm-actions, --dimensions None NATS cloudwatch.PutMetricAlarm → daemon stores alarm config in KV → alarm evaluation runs on metric publish → state transitions trigger alarm actions 1. Create alarm
2. Update existing alarm
3. Missing required fields
NOT STARTED
delete-alarms — --alarm-names Alarms must exist NATS cloudwatch.DeleteAlarms → daemon deletes alarm records from KV → return success 1. Delete existing alarm
2. Non-existent alarm (ResourceNotFound)
NOT STARTED

Auto Scaling

Auto Scaling manages instance fleets based on scaling policies. Depends on Launch Templates (EC2) and optionally ELBv2 target groups for health-based scaling. Stored in spinifex-autoscaling KV bucket.

Command Implemented Flags Missing Flags Prerequisites Basic Logic Test Cases Status
create-auto-scaling-group — --auto-scaling-group-name, --launch-template (Id/Name, Version), --min-size, --max-size, --desired-capacity, --vpc-zone-identifier (subnet IDs), --target-group-arns, --health-check-type (EC2/ELB), --health-check-grace-period, --tags Launch template must exist NATS autoscaling.CreateAutoScalingGroup → daemon validates launch template → stores ASG record in KV → launches DesiredCapacity instances using template → registers with target groups if specified → starts health check loop → return success 1. Create ASG with launch template
2. Scale to desired capacity
3. Missing launch template (ValidationError)
4. Min > Max (ValidationError)
NOT STARTED
update-auto-scaling-group — --auto-scaling-group-name, --min-size, --max-size, --desired-capacity, --launch-template, --health-check-type, --health-check-grace-period ASG must exist NATS autoscaling.UpdateAutoScalingGroup → daemon updates ASG config in KV → if DesiredCapacity changed, scale up/down accordingly → return success 1. Update desired capacity
2. Update launch template
3. Non-existent ASG (ValidationError)
NOT STARTED
delete-auto-scaling-group — --auto-scaling-group-name, --force-delete ASG must exist NATS autoscaling.DeleteAutoScalingGroup → if force: terminate all instances → deregister from target groups → delete from KV → return success. Without force: reject if instances > 0 1. Delete empty ASG
2. Force delete with instances
3. Delete with instances (ResourceInUse)
NOT STARTED
describe-auto-scaling-groups — --auto-scaling-group-names, --filters, --max-records None NATS autoscaling.DescribeAutoScalingGroups → daemon lists ASG records → return list with Instances, DesiredCapacity, MinSize, MaxSize 1. List all ASGs
2. Filter by name
NOT STARTED
set-desired-capacity — --auto-scaling-group-name, --desired-capacity, --honor-cooldown ASG must exist NATS autoscaling.SetDesiredCapacity → daemon updates desired count → scale up (launch) or scale down (terminate oldest) → return success 1. Scale up
2. Scale down
3. Exceeds max (ValidationError)
NOT STARTED
describe-auto-scaling-instances — --instance-ids, --max-records None NATS autoscaling.DescribeAutoScalingInstances → daemon lists all ASG-managed instances → return list with ASG name, lifecycle state, health status 1. List all ASG instances
2. Filter by instance ID
NOT STARTED
put-scaling-policy — --auto-scaling-group-name, --policy-name, --policy-type (TargetTrackingScaling, StepScaling, SimpleScaling), --target-tracking-configuration, --scaling-adjustment, --cooldown ASG must exist NATS autoscaling.PutScalingPolicy → daemon stores policy in KV → links to CloudWatch alarms for metric-based triggers → return PolicyARN 1. Create target tracking policy
2. Create step scaling policy
3. Non-existent ASG (ValidationError)
NOT STARTED
delete-scaling-policy — --auto-scaling-group-name, --policy-name Policy must exist NATS autoscaling.DeleteScalingPolicy → daemon deletes policy → remove CloudWatch alarm links → return success 1. Delete existing policy
2. Non-existent policy
NOT STARTED