spinifex/tests/e2e/TEST_COVERAGE.md

# E2E Test Coverage

## Single-Node (`run-e2e.sh`)

### Phase 1: Environment Setup
- KVM support check
- `spx admin init` (region/az/node config)
- CA certificate trust
- Start all services (`start-dev.sh`)
- Wait for AWS gateway on `localhost:9999`

### Phase 1b: Cluster Stats CLI
- `spx get nodes` — verify node shows Ready
- `spx top nodes` — verify CPU/MEM resource stats
- `spx get vms` — verify "No VMs found" before any launches

### Phase 2: Discovery & Metadata
- `describe-regions`
- `describe-availability-zones` (verify zone name and state)
- `describe-instance-types` (discover available types)
- Select nano instance type and detect architecture

### Phase 2b: Serial Console Access Settings
- `get-serial-console-access-status` (verify default disabled)
- `enable-serial-console-access` (verify returns true, get confirms true)
- `disable-serial-console-access` (verify returns false, get confirms false)

### Phase 3: SSH Key Management
- `create-key-pair` (test-key-1, verify private key material)
- `import-key-pair` (test-key-2, from local RSA key)
- `describe-key-pairs` (verify both keys exist)
- `delete-key-pair` (test-key-2, verify only test-key-1 remains)

### Phase 4: Image Management
- `spx admin images import` (file-based, architecture-aware)
- `describe-images` (verify AMI by ID)

### Phase 5: Instance Lifecycle
- `run-instances` (launch VM with key pair)
- `describe-instances` (poll pending -> running)

### Phase 5a-pre: Cluster Stats CLI (with running VM)
- `spx get vms` — verify running instance appears in output

### Phase 5a: Instance Metadata Validation
- `describe-instances` — verify InstanceType matches requested type
- Verify KeyName matches requested key
- Verify ImageId matches requested AMI
- Verify at least 1 BlockDeviceMapping present

### Phase 5a-ii: SSH Connectivity & Volume Verification
- SSH into instance via QEMU hostfwd port
- Verify SSH connectivity (`id` command returns ec2-user)
- Verify root volume size from inside VM matches API-reported size (`lsblk` vs `describe-volumes`)
- Verify VM hostname

### Phase 5a-iii: Console Output
- `get-console-output` succeeds (verify InstanceId in response)

### Phase 5 (cont): Root Volume
- `describe-volumes` (verify root volume attached)

### Phase 5b: Volume Lifecycle
- `create-volume` (10GB, ap-southeast-2a)
- `modify-volume` (resize to 20GB, poll to verify)
- `attach-volume` (to running instance, /dev/sdf)
- `describe-volumes` (verify in-use + attached state)
- `detach-volume` (verify available state)
- `delete-volume` (verify gone)

### Phase 5b-ii: DescribeVolumeStatus
- `describe-volume-status` (on root volume, verify VolumeId in response)

### Phase 5c: Snapshot Lifecycle
- Uses root volume already attached to running instance (snapshots require a mounted VB instance)
- `create-snapshot` (from attached root volume, with description)
- Verify create response fields (VolumeId, VolumeSize, State, Progress)
- `describe-snapshots` (by ID, verify VolumeId/Size/Description)
- `copy-snapshot` (with new description, verify distinct ID)
- `describe-snapshots` (verify both original + copy visible)
- `delete-snapshot` (original, verify gone while copy survives)
- `delete-snapshot` (copy, cleanup)

### Phase 5d: Verify Snapshot-Backed Instance Launch
- All `run-instances` calls use the snapshot path (`cloneAMIToVolume` -> `OpenFromSnapshot`), so the Phase 5 instance is already snapshot-backed
- Verify AMI snapshot exists in Predastore (`snap-{amiId}/config.json`)
- Read Phase 5 root volume's `config.json` from Predastore
- Verify `SnapshotID` and `SourceVolumeName` are set (proves zero-copy clone)

### Phase 5e: CreateImage Lifecycle
- `create-image` (from running instance, with name and description)
- Verify returned ImageId is non-empty
- `describe-images` (verify custom AMI name and state)
- Extract backing snapshot ID from Predastore config (for cleanup before termination)

### Phase 6: Tag Management
- `create-tags` (3 tags on instance)
- `describe-tags` (filter by resource-id)
- `create-tags` (2 tags on volume)
- `describe-tags` (filter by key)
- `describe-tags` (filter by resource-type)
- `create-tags` (overwrite existing tag value)
- `delete-tags` (unconditional by key)
- `delete-tags` (with wrong value — should be no-op)
- `delete-tags` (with correct value)
- Verify final tag count

### Phase 7: Instance State Transitions
- `stop-instances` (poll -> stopped)

### Phase 7a: Attach Volume to Stopped Instance (Error Path)
- `create-volume` (for attach test)
- `attach-volume` to stopped instance (expect `IncorrectInstanceState` error)
- `delete-volume` (cleanup)

### Phase 7b: ModifyInstanceAttribute
- `modify-instance-attribute` (change instance type from nano → xlarge while stopped)
- `describe-instances` (verify type updated in KV)
- `describe-instance-types` (get expected vCPU count and memory for new type)
- `start-instances` (poll -> running with new type)
- SSH: `nproc` — verify vCPU count matches xlarge (4 vCPUs)
- SSH: `/proc/meminfo` MemTotal — verify memory matches xlarge (within 85% of expected)

### Phase 7c: RunInstances with count > 1
- `run-instances --count 2` (launch 2 instances in a single call)
- Verify 2 instances returned in response
- Poll both to running state
- `terminate-instances` (both, poll -> terminated)

### Phase 8: Negative / Error Path Tests
- `run-instances` with malformed AMI ID (expect `InvalidAMIID.Malformed`)
- `run-instances` with invalid instance type (expect `InvalidInstanceType`)
- `attach-volume` on in-use volume (expect `VolumeInUse`)
- `detach-volume` on boot volume (expect `OperationNotPermitted`)
- `delete-snapshot` on non-existent snapshot (expect `InvalidSnapshot.NotFound`)
- Unsupported Action via raw HTTP (expect `InvalidAction` or error response)
- `run-instances` with non-existent AMI ID (expect `InvalidAMIID.NotFound`)
- `run-instances` with non-existent key pair (expect `InvalidKeyPair.NotFound`)
- `delete-volume` on non-existent volume (expect `InvalidVolume.NotFound`)
- `create-key-pair` with duplicate name (expect `InvalidKeyPair.Duplicate`)
- `import-key-pair` with duplicate name (expect `InvalidKeyPair.Duplicate`)
- `import-key-pair` with invalid key format (expect `InvalidKey.Format`)
- `describe-volumes` with non-existent volume ID (expect `InvalidVolume.NotFound`)
- `describe-images` with non-existent AMI ID (expect `InvalidAMIID.NotFound`)
- `create-image` with duplicate name (expect `InvalidAMIName.Duplicate`)
- `delete-key-pair` on non-existent key (expect success — idempotent, matches AWS)
- `modify-instance-attribute` on running instance (expect `InvalidInstanceID.NotFound` — running instances not in stopped KV)

### IAM Phase 1: User CRUD
- Root auth via `iam list-users` (root user exists)
- `create-user` (alice, bob with path)
- Duplicate user (expect `EntityAlreadyExists`)
- `get-user` (alice)
- `get-user` non-existent (expect `NoSuchEntity`)
- `list-users` (verify count)
- `list-users` with `--path-prefix` filter

### IAM Phase 2: Access Key Lifecycle
- `create-access-key` (alice key 1, key 2)
- Third key exceeds limit (expect `LimitExceeded`)
- `create-access-key` for non-existent user (expect `NoSuchEntity`)
- `list-access-keys` (alice: 2 keys, bob: 0 keys)
- `update-access-key` (Inactive, verify status, reactivate)
- `delete-access-key` (verify count decremented)

### IAM Phase 3: User Authentication
- Configure IAM user profile with access key
- Deactivate key → auth rejected (`InvalidClientTokenId`)
- Reactivate key
- Wrong secret → `SignatureDoesNotMatch`
- Non-existent key ID → `InvalidClientTokenId`
- Multi-user simultaneous auth (root + bob)

### IAM Phase 4: Policy CRUD
- `create-policy` (EC2ReadOnly, FullAdmin, DenyTerminate, IAMReadOnly, EC2DescribeAll)
- Duplicate policy (expect `EntityAlreadyExists`)
- Malformed JSON (expect `MalformedPolicyDocument`)
- `get-policy` (by ARN)
- `get-policy` non-existent (expect `NoSuchEntity`)
- `get-policy-version` (v1)
- `list-policies` (verify count)

### IAM Phase 5: Policy Attachment & Enforcement
- Create charlie with access key
- `attach-user-policy` (alice: EC2ReadOnly + IAMReadOnly, bob: DenyTerminate)
- `list-attached-user-policies` (verify count)
- Idempotent attach (no duplicate)
- Attach non-existent policy (expect `NoSuchEntity`)
- Attach to non-existent user (expect `NoSuchEntity`)
- **Default Deny**: charlie (no policies) → `AccessDenied` on ec2 + iam
- **Explicit Allow**: alice → ec2:Describe{Instances,Vpcs} allowed, ec2:DescribeKeyPairs denied, iam:ListUsers allowed, iam:CreateUser denied
- **Deny Override**: bob → ec2:Describe allowed, ec2:TerminateInstances denied (explicit Deny), iam denied
- **Root Bypass**: root user → all actions succeed
- **Prefix Wildcard**: swap alice to EC2DescribeAll (ec2:Describe*) → Describe* allowed, non-Describe denied
- **FullAdmin**: charlie with FullAdmin → all actions allowed

### IAM Phase 6: Policy Lifecycle — Detach & Delete
- `detach-user-policy` → user loses access
- `delete-policy` while attached (expect `DeleteConflict`)
- Detach then delete → `get-policy` returns `NoSuchEntity`

### IAM Phase 7: IAM Cleanup
- Delete all test users (remove keys + policies first)
- Verify root-only remains (`list-users` count = 1)
- Delete all test policies (`list-policies` count = 0)
- Clean up AWS CLI profiles

### Phase 8: EC2 Account Scoping

Tests that EC2 resources are properly isolated between tenant accounts (Alpha, Beta). Based on `docs/development/feature/iam-phase4-e2e-test-guide.md`. Skips CreateImage (mulga-612) and instance tags (mulga-613).

#### Step 1: Account Setup
- `spx admin account create` (Team Alpha, Team Beta)
- Configure AWS CLI profiles
- Verify auth for both accounts

#### Step 2: Instance Scoping
- Alpha + Beta each launch 1 instance
- `describe-instances` isolation (each sees only own)
- OwnerId field matches account ID
- Cross-account `stop-instances` → `InvalidInstanceID.NotFound`
- Cross-account `terminate-instances` → `InvalidInstanceID.NotFound`
- Cross-account `start-instances` → `InvalidInstanceID.NotFound`
- Cross-account `modify-instance-attribute` → `InvalidInstanceID.NotFound`
- Cross-account `get-console-output` → `InvalidInstanceID.NotFound`

#### Step 3: Volume Scoping
- Alpha + Beta each create 1 volume
- `describe-volumes` isolation
- Cross-account `describe-volumes` by ID → `InvalidVolume.NotFound`
- Cross-account `delete-volume` → `InvalidVolume.NotFound`
- Cross-account `attach-volume` → `InvalidVolume.NotFound`
- Cross-account `detach-volume` → `InvalidVolume.NotFound`
- Cross-account `modify-volume` → `InvalidVolume.NotFound`

#### Step 4: Key Pair Scoping
- Alpha + Beta each create key pair
- `describe-key-pairs` isolation
- Same key name in both accounts (namespace isolation — different KeyPairIds)
- Cross-account `delete-key-pair` → no effect on other account's key
- `import-key-pair` → invisible to other account

#### Step 5: Snapshot Scoping
- Alpha + Beta each create snapshot from own volume
- `describe-snapshots` isolation + OwnerId verification
- Cross-account `delete-snapshot` → `UnauthorizedOperation`
- Cross-account `create-snapshot` from other's volume → `InvalidVolume.NotFound`

#### Step 6: VPC/Subnet Scoping
- Alpha + Beta each create VPC (same CIDR — no conflict)
- `describe-vpcs` isolation
- Cross-account `describe-vpcs` by ID → `InvalidVpcID.NotFound`
- Cross-account `delete-vpc` → `InvalidVpcID.NotFound`
- Alpha + Beta each create subnet
- `describe-subnets` isolation
- Cross-account `create-subnet` in other's VPC → `InvalidVpcID.NotFound`
- Cross-account `delete-subnet` → `InvalidSubnetID.NotFound`

#### Step 7: IGW + EIGW Scoping
- Alpha + Beta each create IGW
- `describe-internet-gateways` isolation
- Cross-account `describe-internet-gateways` by ID → `InvalidInternetGatewayID.NotFound`
- Cross-account `delete-internet-gateway` → `InvalidInternetGatewayID.NotFound`
- Cross-account `attach-internet-gateway` → `InvalidInternetGatewayID.NotFound`
- Cross-account `detach-internet-gateway` → `InvalidInternetGatewayID.NotFound`
- Alpha + Beta each create EIGW
- `describe-egress-only-internet-gateways` isolation
- Cross-account EIGW delete → no effect

#### Step 8: Account Settings
- Alpha `enable-ebs-encryption-by-default` → Beta unaffected
- Independent toggle verification

#### Step 9: Global Resources
- `describe-regions` identical for both accounts
- `describe-availability-zones` identical
- `describe-instance-types` identical

#### Step 10: Edge Cases
- Empty account (Gamma) — no resources visible
- Root isolation from tenants (root cannot see tenant instances)
- Non-existent resource IDs → same error as cross-account (no info leakage)

#### Step 11: Cleanup
- Terminate instances, delete volumes/snapshots/keys/VPCs/IGWs/EIGWs/subnets
- Clean up AWS CLI profiles

### Phase 9: Terminate and Verify Cleanup
- `delete-snapshot` (CreateImage backing snapshot, so DeleteOnTermination is not blocked)
- `terminate-instances` (poll -> terminated)

### Phase 9a: SSH Unreachable Verification
- Verify SSH connection is refused/unreachable after termination

### Phase 9b: Volume Cleanup Verification
- `describe-volumes` on root volume after termination
- Verify root volume is deleted (DeleteOnTermination)

---

## Pseudo Multi-Node (`run-pseudo-multinode-e2e.sh`)

### Phase 1: Environment Setup
- KVM support check
- Simulated network IPs (10.11.12.{1,2,3} on loopback)
- Simulated network IPs (no ramdisk — start-dev.sh uses disk-backed WAL/VB in CI)

### Phase 2: Cluster Initialization
- `spx admin init` (leader node1)
- CA certificate trust
- Start node1 services
- `spx admin join` (node2, node3)
- Start node2 + node3 services

### Phase 3: Cluster Health Verification
- Verify NATS cluster (3 nodes)
- Verify Predastore cluster (3 nodes)
- Wait for gateway on node1
- Wait for daemon NATS readiness
- `describe-regions` (gateway connectivity check)

### Phase 3b: Cluster Stats CLI
- `spx get nodes` — verify all 3 nodes show Ready
- `spx top nodes` — verify instance type capacity table
- `spx get vms` — verify empty (no instances yet)

### Phase 4: Image and Key Setup
- `describe-instance-types` (discover + select nano)
- `create-key-pair`
- `spx admin images import` (with node1 config paths)
- `describe-images` (verify AMI)

### Phase 4b: Multi-Node Key Pair Operations
- `import-key-pair` (multinode-test-key-2, from local RSA key)
- `describe-key-pairs` (verify both keys visible across cluster)
- `delete-key-pair` (multinode-test-key-2, verify deletion)

### Phase 5: Multi-Node Instance Tests

#### Test 1: Instance Distribution
- `run-instances` x3 (distribute across nodes)
- Poll all instances to running state
- Check instance distribution across nodes
- `spx get vms` — verify all instances visible

#### Test 1a-ii: SSH Connectivity & Volume Verification
- SSH into all 3 instances via QEMU hostfwd port
- Verify SSH connectivity (`id` command returns ec2-user)
- Verify root volume size from inside VM matches API-reported size (`lsblk` vs `describe-volumes`)
- Verify VM hostname

#### Test 1b: Volume Lifecycle
- `create-volume` (10GB)
- `modify-volume` (resize to 20GB)
- `attach-volume` (to first instance)
- `detach-volume`
- `delete-volume`

#### Test 1c: Snapshot Lifecycle
- Uses root volume of first instance (snapshots require a mounted VB instance)
- `create-snapshot` (from attached root volume, with description)
- Verify create response fields (VolumeId, VolumeSize, State)
- `describe-snapshots` (by ID, verify fields)
- `copy-snapshot` (with new description)
- `describe-snapshots` (verify both exist)
- `delete-snapshot` (original, verify copy survives)
- `delete-snapshot` (copy, cleanup)

#### Test 1c-ii: Verify Snapshot-Backed Instance Launch
- All `run-instances` calls use the snapshot path (`cloneAMIToVolume` -> `OpenFromSnapshot`), so the Test 1 instances are already snapshot-backed
- Verify AMI snapshot exists in Predastore (`snap-{amiId}/config.json`)
- Read first instance's root volume `config.json` from Predastore
- Verify `SnapshotID` and `SourceVolumeName` are set (proves zero-copy clone)

#### Test 1d: Tag Management (Instances)
- `create-tags` (3 tags on instance)
- `describe-tags` (filter by resource-id)
- `describe-tags` (filter by key)
- `describe-tags` (filter by resource-type)
- `create-tags` (overwrite tag value)
- `delete-tags` (unconditional by key)
- `delete-tags` (wrong value — no-op)
- `delete-tags` (correct value)
- Verify final tag count

#### Test 1d-ii: Tag Management (Volumes)
- `create-tags` (2 tags on root volume)
- `describe-tags` (filter by resource-id, verify count)
- `describe-tags` (filter by resource-type=volume)
- `delete-tags` (both tags, verify cleanup)

#### Test 2: DescribeInstances Aggregation
- `describe-instances` (fan-out across all nodes, verify count)

#### Test 3: Cross-Node Operations
- `stop-instances` (poll -> stopped)
- `start-instances` (poll -> running)

#### Test 4: NATS Cluster Health (Post-Operations)
- Verify NATS cluster still healthy after all operations

#### Test 5: VM Crash Recovery
- Kill QEMU process with SIGKILL (simulate OOM kill)
- Verify daemon detects crash (instance transitions to error/pending)
- Wait for auto-restart (backoff starts at 5s)
- Verify new QEMU PID differs from original
- Verify instance reaches running state
- Verify SSH connectivity after recovery

#### Test 5b: Crash Loop Prevention
- Kill QEMU 4 times rapidly on a third instance
- Verify crash loop is detected and restarts stop after max attempts (3 in 10-min window)
- Verify instance reaches error state (won't restart further)

### Phase 5c: IAM Accounts & Cross-Account Isolation

#### Step 1: Create Accounts
- `spx admin account create` (Team Alpha, Team Beta)
- Verify sequential 12-digit account IDs
- `spx admin account list` (verify both accounts)

#### Step 2: Account Admin Auth
- Alpha admin authenticates via ec2 + iam
- Beta admin authenticates via ec2 + iam

#### Step 3: Account-Scoped Users
- `create-user` alice in Alpha + Beta (same name, different accounts)
- `create-user` team-member (Alpha), dev-user (Beta)
- `list-users` scoped per account (verify different user lists)
- Cross-account isolation: Alpha cannot see Beta's users and vice versa (`NoSuchEntity`)

#### Step 4: Account-Scoped Access Keys
- `create-access-key` for alice in Alpha + Beta
- Configure separate AWS CLI profiles per account user

#### Step 5: Account-Scoped Policies & Enforcement
- `create-policy` EC2ReadOnly in Alpha (narrow: DescribeInstances + DescribeVpcs)
- `create-policy` EC2ReadOnly in Beta (broad: ec2:Describe*)
- `attach-user-policy` to alice in both accounts
- Alpha alice: DescribeInstances allowed, DescribeKeyPairs denied (narrow)
- Beta alice: DescribeInstances allowed, DescribeKeyPairs allowed (broad Describe*)
- Both denied: CreateKeyPair

#### Step 6: Cross-Account Delete Isolation
- Delete Alpha's alice (detach policy, delete key, delete user)
- Verify Alpha alice gone (`NoSuchEntity`)
- Verify Beta alice unaffected (still exists, auth still works)

#### Step 7: EC2 Resource Scoping

Tests that EC2 resources are properly isolated between the Alpha/Beta accounts. Skips CreateImage (mulga-612) and instance tags (mulga-613).

**7a: Instance Scoping**
- Alpha + Beta each launch 1 instance
- `describe-instances` isolation (each sees only own)
- OwnerId field matches account ID
- Cross-account `stop-instances` → `InvalidInstanceID.NotFound`
- Cross-account `terminate-instances` → `InvalidInstanceID.NotFound`
- Cross-account `get-console-output` → `InvalidInstanceID.NotFound`

**7b: Volume Scoping**
- Alpha + Beta each create 1 volume
- `describe-volumes` isolation
- Cross-account `describe-volumes` by ID → `InvalidVolume.NotFound`
- Cross-account `delete-volume` → `InvalidVolume.NotFound`
- Cross-account `attach-volume` → `InvalidVolume.NotFound`
- Cross-account `detach-volume` → `InvalidVolume.NotFound`
- Cross-account `modify-volume` → `InvalidVolume.NotFound`

**7c: Key Pair Scoping**
- Alpha + Beta each create key pair
- `describe-key-pairs` isolation
- Same key name in both accounts (namespace isolation — different KeyPairIds)
- Cross-account `delete-key-pair` → no effect on other account's key
- `import-key-pair` → invisible to other account

**7d: Snapshot Scoping**
- Alpha + Beta each create snapshot from own volume
- `describe-snapshots` isolation + OwnerId verification
- Cross-account `delete-snapshot` → `UnauthorizedOperation`
- Cross-account `create-snapshot` from other's volume → `InvalidVolume.NotFound`

**7e: VPC/Subnet Scoping**
- Alpha + Beta each create VPC (same CIDR — no conflict)
- `describe-vpcs` / `describe-subnets` isolation
- Cross-account VPC/subnet operations → `NotFound`

**7f: IGW + EIGW Scoping**
- Alpha + Beta each create IGW + EIGW
- Describe isolation for both
- Cross-account attach/detach/delete → `NotFound`

**7g: Account Settings**
- `enable-ebs-encryption-by-default` per account — independent toggle

**7h: Global Resources**
- `describe-regions`, `describe-availability-zones`, `describe-instance-types` identical for both accounts

#### Step 8: Edge Cases
- Empty account (Gamma) — no resources visible
- Root isolation from tenants
- Non-existent resource IDs → same error as cross-account
- Parallel key creation (race condition) — no cross-contamination

#### Step 9: EC2 + IAM Cleanup
- Terminate instances, delete volumes/snapshots/keys/VPCs/IGWs/EIGWs/subnets
- Delete all IAM users, policies, access keys
- Clean up AWS CLI profiles

### Phase 5d: VPC Networking
- Step 1: `create-vpc` (10.100.0.0/16) + `create-subnet` (10.100.1.0/24)
- Step 2: `run-instances` x3 with `--subnet-id` (launch into VPC subnet)
- Poll all VPC instances to running state
- Step 3: Verify `PrivateIpAddress` in `describe-instances` for each instance
- Verify `SubnetId` and `VpcId` match requested values
- Verify at least 1 `NetworkInterface` per instance
- Verify all IPs are unique and in subnet range (10.100.1.x)
- Step 4: SSH + ping connectivity (skipped in CI — OVN DHCP wait too slow)
- Step 5: Stop/start IP persistence
  - `stop-instances` (all VPC instances, poll -> stopped)
  - Verify `PrivateIpAddress` persists in stopped state
  - `start-instances` (all VPC instances, poll -> running)
  - Verify `PrivateIpAddress` identical after restart
- Step 6: Cleanup — terminate VPC instances, delete subnet, delete VPC

### Phase 6: Cluster Shutdown + Restart

#### Test 6a: Dry-Run Shutdown
- `spx admin cluster shutdown --dry-run`
- Validate output contains all 5 phases (GATE, DRAIN, STORAGE, PERSIST, INFRA)

#### Test 6b: Coordinated Cluster Shutdown
- `spx admin cluster shutdown --force --timeout 60s`
- Verify all services down on all nodes (gateway, NATS, QEMU)

#### Test 6c: Cluster Restart + Recovery
- Restart all 3 node services
- Verify NATS cluster reforms (3 members)
- Wait for gateway and daemon readiness
- Smoke test: `describe-instance-types` returns valid results

#### Test 6d: Instance Relaunch + Terminate
- Wait for instances to finish relaunching after restart (pending → running/error)
- `terminate-instances` (all 3 instances)
- Poll all to terminated state

### Cleanup
- Coordinated cluster shutdown (with fallback to per-node PID stops)
- Remove simulated IPs

---

## Real Multi-Node (`run-multinode-e2e.sh`)

Runs on a real 3-node libvirt cluster provisioned by OpenTofu (`scripts/tofu-cluster/`). Each node is a separate VM with its own OVN, NATS, Predastore, and Spinifex daemon. Bootstrap (`bootstrap.sh`) handles all provisioning before the test script runs.

### Bootstrap (pre-test)

1. OpenTofu provisions 3 libvirt VMs (bottlebrush, ironbark, casuarina) with cloud-init
2. Wait for all nodes to be SSH-reachable
3. Copy SSH key to node1 for peer access
4. `git fetch` + checkout test branch on all nodes (all 3 repos)
5. `make build` on all nodes (parallel)
6. `setup-ovn.sh` — primary gets `--management`, secondaries connect to primary's OVN central
7. `spx admin init` on primary, `spx admin join` on secondaries (formation server handshake)
8. `start-dev.sh` on all nodes
9. Wait for `/health` (awsgw=ok) on all nodes
10. Install Spinifex CA certificate on all nodes
11. `import-key-pair` (spinifex-key) + `spx admin images import` (Ubuntu 24.04)
12. `create-vpc` (10.200.0.0/16) + `create-subnet` (10.200.1.0/24)

### Phase 1: Pre-flight Validation
- KVM support check (`/dev/kvm` writable)
- SSH connectivity to all peer nodes

### Phase 2: Cluster Health
- NATS cluster: verify 2 unique route peers (3-node quorum)
- Predastore reachable on all 3 nodes
- AWS gateway reachable on all 3 nodes
- Daemon readiness (`describe-instance-types` returns results)
- `spx get nodes` — verify all 3 nodes show Ready
- `spx get vms` — verify empty (no instances yet)

### Phase 3: Instance Lifecycle + Distribution
- Discover nano instance type and AMI (from bootstrap import)
- `run-instances` x3 with staggered launches
- Poll all instances to running state
- Check instance distribution across physical nodes (QEMU process check)
- Verify at least 2 different hosting nodes (non-deterministic, non-fatal if all on 1)
- `spx get vms` — verify all 3 instances visible

### Phase 4: SSH into Guest VMs
- For each instance across all nodes:
  - Find hosting node via QEMU process scan
  - Extract SSH hostfwd port from QEMU command line on remote node
  - Wait for SSH readiness (up to 60s)
  - Verify SSH connectivity (`id` returns ec2-user)
  - Verify block device (`lsblk`)

### Phase 5: Volume Lifecycle
- `create-volume` (10GB)
- `attach-volume` (to first instance, /dev/sdf)
- Poll for in-use + attached state
- `detach-volume` (poll for available)
- `delete-volume` (verify gone)

### Phase 6: Cross-Node Gateway Access
- `describe-instances` via node1 gateway (baseline count)
- `describe-instances` via node2 and node3 gateways
- Verify all gateways return the same instance count (fan-out aggregation)

### Phase 7: Cross-Node Operations
- Find which node hosts instance 0
- Pick a different node's gateway for stop
- `stop-instances` via remote gateway (poll -> stopped)
- Pick a third node's gateway for start
- `start-instances` via another remote gateway (poll -> running)

### Phase 8: Node Failure
- `stop-dev.sh` on node2 (simulate node failure)
- Verify node1 still serves requests (`describe-instance-types`)
- Verify node3 still serves requests
- Check NATS degraded state (1 peer instead of 2)
- `describe-instances` still works from surviving nodes

### Phase 9: Node Recovery
- `start-dev.sh` on node2 (restart failed node)
- Wait for NATS cluster to reform (2 peers again)
- Verify node2 gateway is back
- `spx get nodes` — verify all 3 nodes Ready again
- Verify node2 serves requests after recovery

### Phase 10: Cleanup
- `terminate-instances` (all 3 instances, poll -> terminated)

---

## VPC (`run-vpc-e2e.sh`)

Standalone VPC networking test suite. Runs against a running Spinifex endpoint (configurable via `ENDPOINT` env var). OVN topology tests are skipped when OVN is unavailable.

### Phase 1: VPC CRUD
- `create-vpc` (10.99.0.0/16, verify VpcId)
- `describe-vpcs` (by ID, verify exactly 1 returned)

### Phase 2: Subnet CRUD
- `create-subnet` (10.99.1.0/24 in VPC, verify SubnetId)
- `describe-subnets` (by ID, verify exactly 1 returned)

### Phase 3: Internet Gateway CRUD
- `create-internet-gateway` (verify InternetGatewayId)
- `describe-internet-gateways` (by ID, verify exactly 1 returned)

### Phase 4: Internet Gateway Attach / Detach
- `attach-internet-gateway` (IGW to VPC)
- `describe-internet-gateways` (verify attachment VpcId)
- `delete-internet-gateway` on attached IGW (expect `DependencyViolation` or rejection)
- `detach-internet-gateway` (IGW from VPC)
- `describe-internet-gateways` (verify no attachments)

### Phase 5: OVN Topology Verification (requires OVN)
- Re-attach IGW for topology inspection
- Verify OVN logical switch exists for subnet
- Verify OVN logical router exists for VPC
- Verify OVN external switch exists (IGW attached)
- Verify SNAT rule on VPC router
- Verify default route (0.0.0.0/0) on VPC router
- Dump full OVN NB DB topology for debugging
- Detach IGW
- Skipped when `ovn-nbctl` is not available

### Phase 6: Cleanup
- `delete-internet-gateway`
- `delete-subnet`
- `delete-vpc`
- Verify OVN cleanup (no VPC routers remaining, when OVN available)