spinifex/docs/development/feature/eigw.md

Egress-Only Internet Gateway (EIGW)

Overview

Egress-Only Internet Gateways provide outbound-only IPv6 connectivity for instances in a VPC. In Hive, EIGWs are lightweight metadata objects persisted in NATS JetStream KV. The implementation covers the three standard EC2 API operations: Create, Delete, and Describe.

Key Design Decisions

1. NATS JetStream KV for persistence

EIGW records are stored in a dedicated KV bucket (hive-eigw) with 10-revision history. This avoids introducing a database dependency -- the same NATS infrastructure used for messaging also handles persistence.

The getOrCreateKVBucket helper tries CreateKeyValue first, then falls back to KeyValue (open existing). This makes the service idempotent across restarts.

Trade-off: KV is not suited for complex queries (e.g. filtering by tag values). Describe currently iterates all keys and filters in-memory, which is fine for typical EIGW counts but wouldn't scale to thousands of records.

2. Constructor fails loudly, daemon falls back gracefully

NewEgressOnlyIGWServiceImplWithNATS returns an error if the KV bucket can't be created or opened. This ensures the service never operates with a nil KV handle, eliminating an entire class of nil-pointer bugs.

The daemon handles this with a graceful fallback:

if eigwSvc, err := handlers_ec2_eigw.NewEgressOnlyIGWServiceImplWithNATS(d.config, d.natsConn); err != nil {
    slog.Warn("Failed to initialize EIGW service, falling back to in-memory", "error", err)
    d.eigwService = handlers_ec2_eigw.NewEgressOnlyIGWServiceImpl(d.config)
} else {
    d.eigwService = eigwSvc
}

Trade-off: The in-memory fallback (NewEgressOnlyIGWServiceImpl) has no KV backing, so EIGW state won't survive daemon restarts. This is acceptable for degraded mode -- the alternative is refusing to start the daemon entirely.

3. crypto/rand for ID generation

EIGW IDs (eigw- + 16 hex chars) are generated using crypto/rand + encoding/hex. This matches the pattern used by instance and volume ID generation elsewhere in the codebase.

func generateEgressOnlyIGWID() string {
    b := make([]byte, 8)
    if _, err := rand.Read(b); err != nil {
        panic("crypto/rand failed: " + err.Error())
    }
    return "eigw-" + hex.EncodeToString(b)
}

The panic on crypto/rand failure is intentional -- if the OS entropy source is broken, the system has larger problems.

4. Error strings are awserrors constants

All errors returned from service_impl.go use errors.New(awserrors.ErrorXxx) with exact constant values. This is critical because the error string travels through a round-trip:

service_impl: errors.New(awserrors.ErrorMissingParameter)
  -> daemon handleNATSRequest: utils.GenerateErrorPayload(err.Error())
    -> JSON over NATS: {"Code": "MissingParameter"}
      -> gateway NATSRequest[Out]: ValidateErrorPayload extracts Code
        -> gateway ErrorHandler: awserrors.ErrorLookup[err.Error()] -> HTTP status + XML

If the error string doesn't match a key in ErrorLookup, the gateway returns a generic 500. Using fmt.Errorf with extra context would break this chain.

5. Centralized nil NATS check in the gateway

Rather than checking natsConn == nil in each gateway function, a single check in EC2_Request covers all NATS-dependent actions. An allowlist (ec2LocalActions) exempts actions that work without NATS:

var ec2LocalActions = map[string]bool{
    "DescribeRegions":           true,
    "DescribeAvailabilityZones": true,
    "DescribeAccountAttributes": true,
}

If gw.NATSConn is nil and the action isn't in the allowlist, the gateway returns ErrorServerInternal immediately.

6. Tag filtering by resource type

Create accepts TagSpecifications but only applies tags where ResourceType equals "egress-only-internet-gateway". Tags with other resource types (e.g. "instance") are silently ignored, matching AWS behavior.

Request Flow

AWS SDK (create-egress-only-internet-gateway --vpc-id vpc-xxx)
  -> Gateway (port 9999): ec2Handler parses AWS query params into SDK struct
    -> Validate input (nil check, required fields)
    -> NATS ec2.CreateEgressOnlyInternetGateway: routed to daemon via "hive-workers" queue group
      -> Daemon handleEC2CreateEgressOnlyInternetGateway
        -> handleNATSRequest: unmarshal -> EgressOnlyIGWServiceImpl.Create -> marshal -> respond
          1. Validate VpcId is present
          2. Generate eigw-{16hex} ID via crypto/rand
          3. Build EgressOnlyIGWRecord with state "attached"
          4. Process matching tag specifications
          5. JSON marshal and store in NATS KV (hive-eigw bucket)
        <- NATS response: ec2.CreateEgressOnlyInternetGatewayOutput JSON
    <- Gateway: wrap in XML (CreateEgressOnlyInternetGatewayResponse)
  <- AWS SDK: response

NATS Topics

Topic	Direction	Purpose
ec2.CreateEgressOnlyInternetGateway	Gateway -> Daemon	Create new EIGW attached to a VPC
ec2.DeleteEgressOnlyInternetGateway	Gateway -> Daemon	Delete EIGW by ID
ec2.DescribeEgressOnlyInternetGateways	Gateway -> Daemon	List/filter EIGWs

All topics use the hive-workers queue group for load balancing across daemon instances.

Storage Layout

NATS JetStream KV bucket: hive-eigw (history: 10)

Key: eigw-a1b2c3d4e5f6g7h8
Value: {
  "egress_only_internet_gateway_id": "eigw-a1b2c3d4e5f6g7h8",
  "vpc_id": "vpc-abc123",
  "state": "attached",
  "tags": {"Name": "my-eigw", "Env": "prod"},
  "created_at": "2026-02-10T12:00:00Z"
}

Files Changed

File	Change
hive/handlers/ec2/eigw/service.go	EgressOnlyIGWService interface (3 methods)
hive/handlers/ec2/eigw/service_impl.go	Direct implementation with NATS JetStream KV persistence
hive/handlers/ec2/eigw/service_impl_test.go	12 test cases against embedded NATS server
hive/handlers/ec2/eigw/service_nats.go	NATS proxy (gateway -> daemon via utils.NATSRequest)
hive/gateway/ec2/eigw/CreateEgressOnlyInternetGateway.go	Gateway validation + handler for Create
hive/gateway/ec2/eigw/DeleteEgressOnlyInternetGateway.go	Gateway validation + handler for Delete
hive/gateway/ec2/eigw/DescribeEgressOnlyInternetGateways.go	Gateway validation + handler for Describe
hive/gateway/ec2/eigw/eigw_test.go	Gateway validation tests (9 cases)
hive/gateway/ec2.go	3 entries in ec2Actions map; centralized nil NATS check
hive/daemon/daemon.go	eigwService field, 3 NATS subscriptions, service initialization with fallback
hive/daemon/daemon_handlers.go	3 handler functions delegating to handleNATSRequest
hive/awserrors/awserrors.go	ErrorInvalidEgressOnlyInternetGatewayId.Malformed and .NotFound constants + lookup entries

Test Coverage

Service Implementation Tests (service_impl_test.go)

Test	What it validates
TestCreateEgressOnlyInternetGateway	Basic create: ID format, VPC attachment, state
TestCreateEgressOnlyInternetGateway_MissingVpcId	Rejects nil VpcId with MissingParameter
TestCreateEgressOnlyInternetGateway_EmptyVpcId	Rejects empty string VpcId
TestCreateEgressOnlyInternetGateway_WithTags	Tags applied and persisted through describe round-trip
TestCreateEgressOnlyInternetGateway_TagsWrongResourceType	Tags with non-matching resource type are ignored
TestDeleteEgressOnlyInternetGateway	Delete succeeds, describe confirms removal
TestDeleteEgressOnlyInternetGateway_NotFound	Returns NotFound for nonexistent ID
TestDeleteEgressOnlyInternetGateway_MissingID	Rejects nil ID with MissingParameter
TestDeleteEgressOnlyInternetGateway_EmptyID	Rejects empty string ID
TestDescribeEgressOnlyInternetGateways_All	Lists all EIGWs when no filter specified
TestDescribeEgressOnlyInternetGateways_ByID	Filters by specific EIGW ID
TestDescribeEgressOnlyInternetGateways_Empty	Returns empty list for empty bucket

All tests run against an embedded NATS server with JetStream enabled.

Gateway Validation Tests (eigw_test.go)

Test	What it validates
ValidateCreateEgressOnlyInternetGatewayInput	nil input, missing VpcId, empty VpcId, valid input
ValidateDeleteEgressOnlyInternetGatewayInput	nil input, missing ID, empty ID, valid input
ValidateDescribeEgressOnlyInternetGatewaysInput	nil input, empty input, with filter IDs

AWS CLI Usage

# Create
aws ec2 create-egress-only-internet-gateway \
  --vpc-id vpc-abc123 \
  --tag-specifications 'ResourceType=egress-only-internet-gateway,Tags=[{Key=Name,Value=my-eigw}]' \
  --endpoint-url https://localhost:9999

# Describe all
aws ec2 describe-egress-only-internet-gateways \
  --endpoint-url https://localhost:9999

# Describe by ID
aws ec2 describe-egress-only-internet-gateways \
  --egress-only-internet-gateway-ids eigw-a1b2c3d4e5f6g7h8 \
  --endpoint-url https://localhost:9999

# Delete
aws ec2 delete-egress-only-internet-gateway \
  --egress-only-internet-gateway-id eigw-a1b2c3d4e5f6g7h8 \
  --endpoint-url https://localhost:9999