spinifex/docs/development/feature/start-instance-distributed-testing.md

Distributed StartInstances - Manual Testing Guide

Manual test plan for verifying stop/start instance lifecycle across daemon nodes, including daemon restarts. Requires a multi-node cluster (see INSTALL.md "Multi-node Configuration").

Prerequisites

# Build
make build

# Set up 3-node cluster (if not already running)
./scripts/create-multi-node.sh
./scripts/start-dev.sh ~/node1/
./scripts/start-dev.sh ~/node2/
./scripts/start-dev.sh ~/node3/

# AWS CLI setup
export AWS_PROFILE=hive

Pick a gateway endpoint for all commands (any node works):

export EP="--endpoint-url https://10.11.12.1:9999"

Import an SSH key and AMI if not already done (see INSTALL.md).

---

Test 1: Basic Stop and Start (Same Cluster)

Verify an instance can be stopped and restarted by any daemon.

# 1. Launch an instance
aws $EP ec2 run-instances \
  --image-id $HIVE_AMI \
  --instance-type $HIVE_INSTANCE \
  --key-name hive-key \
  --count 1

export INSTANCE_ID="i-XXX"  # from output

# 2. Confirm running
aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
# Expect: State.Name = "running", State.Code = 16

# 3. Stop the instance
aws $EP ec2 stop-instances --instance-ids $INSTANCE_ID
# Expect: PreviousState.Name = "running", CurrentState.Name = "stopping"

# 4. Wait for stop to complete, then verify stopped
aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
# Expect: State.Name = "stopped", State.Code = 80

# 5. Check daemon logs on the original node — should show KV release
grep -i "Released stopped instance ownership" ~/node*/logs/hive.log
# Expect: "Released stopped instance ownership to shared KV" with instanceId and lastNode

# 6. Start the instance
aws $EP ec2 start-instances --instance-ids $INSTANCE_ID
# Expect: PreviousState.Name = "stopped", CurrentState.Name = "pending"

# 7. Confirm running again
aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
# Expect: State.Name = "running", State.Code = 16

# 8. Check which node picked up the start
grep -i "Started stopped instance from shared KV" ~/node*/logs/hive.log
# Note the node — it may differ from the original

---

Test 2: Stop, Restart Daemon, Then Start

Verify a stopped instance survives a daemon restart and can be started after.

# 1. Launch and stop an instance (repeat Test 1 steps 1-5)
export INSTANCE_ID="i-XXX"

# 2. Identify which node originally ran the instance
grep "Released stopped instance" ~/node*/logs/hive.log | grep $INSTANCE_ID
# Example: ~/node1/logs/hive.log — node1 was the original

# 3. Restart that daemon
./scripts/stop-dev.sh ~/node1/
./scripts/start-dev.sh ~/node1/

# 4. Verify the instance is still visible as stopped
aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
# Expect: State.Name = "stopped"
# The instance is in shared KV, so any running daemon can report it

# 5. Start the instance
aws $EP ec2 start-instances --instance-ids $INSTANCE_ID
# Expect: PreviousState.Name = "stopped", CurrentState.Name = "pending"

# 6. Confirm running and check which node picked it up
aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
grep "Started stopped instance from shared KV" ~/node*/logs/hive.log | grep $INSTANCE_ID

---

Test 3: Stop on Node A, Kill Node A, Start on Node B

Verify a stopped instance can be started when its original daemon is down.

# 1. Launch instance (it lands on some node)
aws $EP ec2 run-instances \
  --image-id $HIVE_AMI \
  --instance-type $HIVE_INSTANCE \
  --key-name hive-key \
  --count 1

export INSTANCE_ID="i-XXX"

# 2. Stop the instance
aws $EP ec2 stop-instances --instance-ids $INSTANCE_ID

# 3. Wait for stopped state
aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
# Expect: State.Name = "stopped"

# 4. Identify the original node and shut it down
grep "Released stopped instance" ~/node*/logs/hive.log | grep $INSTANCE_ID
# If node1 — stop node1
./scripts/stop-dev.sh ~/node1/

# 5. Point CLI to a surviving node and start the instance
export EP="--endpoint-url https://10.11.12.2:9999"
aws $EP ec2 start-instances --instance-ids $INSTANCE_ID
# Expect: CurrentState.Name = "pending" — a different daemon picked it up

# 6. Confirm running on the new node
aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
# Expect: State.Name = "running"
grep "Started stopped instance from shared KV" ~/node2/logs/hive.log ~/node3/logs/hive.log

# 7. Bring node1 back up (cleanup)
./scripts/start-dev.sh ~/node1/

---

Test 4: Repeated Stop/Start Cycles

Verify the instance can be stopped and started multiple times without state corruption.

# 1. Launch an instance
aws $EP ec2 run-instances \
  --image-id $HIVE_AMI \
  --instance-type $HIVE_INSTANCE \
  --key-name hive-key \
  --count 1

export INSTANCE_ID="i-XXX"

# 2. Cycle 3 times
for i in 1 2 3; do
  echo "=== Cycle $i ==="

  # Wait for running
  sleep 5
  aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID \
    --query 'Reservations[0].Instances[0].State.Name' --output text
  # Expect: running

  # Stop
  aws $EP ec2 stop-instances --instance-ids $INSTANCE_ID
  sleep 10

  aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID \
    --query 'Reservations[0].Instances[0].State.Name' --output text
  # Expect: stopped

  # Start
  aws $EP ec2 start-instances --instance-ids $INSTANCE_ID
  sleep 10
done

# 3. Final state check
aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
# Expect: State.Name = "running"

---

Test 5: DescribeInstances Includes Stopped Instances

Verify stopped instances appear alongside running ones in describe output.

# 1. Launch two instances
aws $EP ec2 run-instances \
  --image-id $HIVE_AMI \
  --instance-type $HIVE_INSTANCE \
  --key-name hive-key \
  --count 1
export RUNNING_ID="i-XXX"

aws $EP ec2 run-instances \
  --image-id $HIVE_AMI \
  --instance-type $HIVE_INSTANCE \
  --key-name hive-key \
  --count 1
export STOPPED_ID="i-YYY"

# 2. Stop one, keep the other running
aws $EP ec2 stop-instances --instance-ids $STOPPED_ID
sleep 10

# 3. Describe all instances — both should appear
aws $EP ec2 describe-instances
# Expect: $RUNNING_ID with State.Name = "running"
# Expect: $STOPPED_ID with State.Name = "stopped"

# 4. Describe with instance ID filter
aws $EP ec2 describe-instances --instance-ids $STOPPED_ID
# Expect: only the stopped instance

---

Test 6: Start a Non-Existent Instance

Verify proper error handling.

aws $EP ec2 start-instances --instance-ids i-doesnotexist
# Expect: instance remains reported as stopped (state unchanged)
# Check daemon logs for "instance not found in shared KV"

---

Test 7: Start an Already Running Instance

Verify that starting a running instance does not cause issues.

# 1. Launch and confirm running
aws $EP ec2 run-instances \
  --image-id $HIVE_AMI \
  --instance-type $HIVE_INSTANCE \
  --key-name hive-key \
  --count 1
export INSTANCE_ID="i-XXX"
sleep 5

# 2. Try to start it again
aws $EP ec2 start-instances --instance-ids $INSTANCE_ID
# Expect: instance not found in shared KV (running instances are not in shared KV)
# The response should report the instance as still "stopped" (no-op from gateway perspective)

---

Test 8: Daemon Crash During Stop (Migration Recovery)

Verify that a daemon crash between KV write and local cleanup is handled on restart.

# 1. Launch and stop an instance
aws $EP ec2 run-instances \
  --image-id $HIVE_AMI \
  --instance-type $HIVE_INSTANCE \
  --key-name hive-key \
  --count 1
export INSTANCE_ID="i-XXX"

aws $EP ec2 stop-instances --instance-ids $INSTANCE_ID
sleep 10

# 2. Force kill the daemon process (simulate crash — skips cleanup)
# Identify the daemon PID on the node that ran the instance
grep "Released stopped instance" ~/node*/logs/hive.log | grep $INSTANCE_ID
# If node1:
kill -9 $(pgrep -f "hive.*node1")

# 3. Restart the daemon
./scripts/start-dev.sh ~/node1/

# 4. Check logs for migration handling
grep -i "Migrated stopped instance" ~/node1/logs/hive.log
# If the instance was still in per-node state, restoreInstances migrates it to shared KV

# 5. Verify instance is still startable
aws $EP ec2 start-instances --instance-ids $INSTANCE_ID
aws $EP ec2 describe-instances --instance-ids $INSTANCE_ID
# Expect: State.Name = "running"

---

What to Look For in Logs

Key log messages and where to find them (~/node*/logs/hive.log):

Log Message	When
Released stopped instance ownership to shared KV	After successful stop + KV write
Started stopped instance from shared KV	After a daemon picks up and launches a stopped instance
Migrated stopped instance to shared KV	During restoreInstances on daemon startup
Failed to write stopped instance to shared KV, keeping local ownership	KV write failed, instance stays local
instance not found in shared KV	Start request for non-existent instance
instance not in stopped state	Start request for instance not in stopped state
DescribeInstances: Collected stopped instance reservations	Gateway merged stopped instances into describe output