Files
spinifex/docs/getting-started/setting-up-your-cluster
Josh Siviour 4ae2a4ac53 feat(elbv2): QEMU microvm direct-boot for LB system instances (#228)
* feat(vm): direct-boot foundations + config/input extensions + image build pipeline

vm package (mulga-siv-62):
- Add KernelImage, Initrd, KernelCmdline, FwCfg/FwCfgEntry to vm.Config
- Emit -kernel/-initrd/-append/-fw_cfg args when fields set
- Relax drive invariant: accept kernel image in place of drives
- Add devices.go: IsMMIO predicate, NetDevice/BlkDevice/RngDevice helpers
- Refactor lifecycle.go + network_helpers.go call sites to use helpers
- shutdown.go: os.Remove fw_cfg tmpfiles after QEMU exits
- Add devices_test.go, microvm_test.go; extend vm_test.go

config/input extensions (mulga-siv-63):
- Add MicrovmConfig{ELBv2Enabled, ImagePath} to DaemonConfig
- ImagePath falls back to /usr/share/spinifex/microvm when empty
- Add commented [microvm] section to spinifex.toml template
- Add DirectBoot bool + NICs []NICConfig to SystemInstanceInput

image build pipeline (mulga-siv-64):
- scripts/build-microvm-image.sh: Alpine chroot build producing vmlinuz + initramfs.cpio.gz
- build/microvm/init.sh: fw_cfg-driven PID 1 init (mounts, modprobe, NIC config, lb-agent-env, ca-cert)
- build/microvm/inittab: BusyBox init with lb-agent respawn
- make build-microvm-image target
- setup.sh: install microvm/* to /usr/share/spinifex/microvm/
- .gitignore: exclude built vmlinuz + initramfs.cpio.gz artifacts

* feat(daemon): direct-boot launch path for microvm system instances

When SystemInstanceInput.DirectBoot=true:
- Skip GetAMIConfig and GenerateVolumes
- Build vm.Config directly (no buildBaseVMConfig, no PCIe root ports)
- MachineType=microvm,x-option-roms=off,pic=off,pit=off,rtc=on,acpi=on,isa-serial=on
- Resolve kernel/initrd from Microvm.ImagePath config field
- Serialise NICs to three fw_cfg tmpfiles (netcfg, lb-agent-env, ca-cert)
- Validate exactly one NICConfig.IsDefault=true before writing netcfg blob
- Shutdown path cleans up tmpfiles via existing Config.FwCfg iteration

PC-machine path (DirectBoot=false) unchanged.

Closes mulga-siv-65

* feat(elbv2): wire microvm.elbv2_enabled flag into ELBv2 service

When ELBv2Enabled=true:
- Drop lbVMUserData cloud-config; emit KEY=value LBAgentEnv blob instead
- Set SystemInstanceInput.DirectBoot=true; omit ImageID
- Populate NICs []NICConfig (index 0=primary ENI IsDefault, 1=mgmt, 2+=extra ENIs)
- Validate exactly one IsDefault NIC at service layer

When ELBv2Enabled=false: existing PC-machine path unchanged.

Adds buildLBAgentEnv, buildMicrovmNICs, subnetCIDRForIP, subnetGatewayIP helpers.
Tests in arn_test.go cover both flag-on and flag-off paths.

Closes mulga-siv-66

* test(daemon,vm): add unit tests for direct-boot helpers to meet diff coverage threshold

* feat(daemon): Phase 1 perf instrumentation + e2e runner microvm flag

- Log t1→t2 boot timing via slog on the direct-boot path
- measureQEMURSSMiB helper reads /proc/<pid>/status (tested)
- Gate constants: phase1BootGateMs=500, phase1RSSGateMiB=25
- run-lb-e2e.sh: --microvm flag, dual-path CI annotation, density test note

Closes mulga-siv-67

* fix(build): remove busybox-initscripts — not in Alpine 3.20, not needed

* fix(microvm): end-to-end fixes for direct-boot LB path

Serial output was silent (x-option-roms=off suppresses BIOS serial init);
fixed by removing it from the machine string and using -serial chardev:console0
to attach to the machine-created ISA serial device instead of adding a
duplicate isa-serial device (which would create ttyS1 while kernel uses ttyS0).

virtio-net interfaces were invisible to the kernel because QEMU's
auto-kernel-cmdline does not append virtio_mmio.device= params when -append
is used. Fixed by generating explicit virtio_mmio.device=0x200@0xfebNNNNN:N
params for each NIC in KernelCmdline, and modprobing virtio_mmio + virtio_net
in init.sh.

sys.micro bumped to 512 MiB — 128 MiB caused kernel OOM during initramfs
decompression (~68 MiB uncompressed).

BusyBox init ::respawn: entries do not source or export env files; fixed
inittab to use set -a; . /etc/conf.d/lb-agent; set +a before exec.

wireLBAgentConfig was documented as reading the CA cert but never set
elbv2Service.CACert; lb-agent TLS handshakes to AWSGW failed with
x509: certificate signed by unknown authority. Fixed by reading
d.config.NATS.CACert (same local CA that signs the AWSGW server cert)
and assigning to elbv2Service.CACert so writeFwCfgBlobs writes a
non-empty cacert blob into fw_cfg.

Adds vm_test TestExecute_MicrovmFileChardev and updates lifecycle_test
to assert SerialSocket is intentionally empty on direct-boot VMs.

* fix(build): auto-detect docker/podman when apk unavailable for microvm image build

* fix(build): create /dev/console and /dev/null as proper device nodes in initramfs

tar extraction without root silently demotes device files to 0-byte regular
files (mknod requires CAP_MKNOD). The kernel opens /dev/console when wiring
init's stdio to the serial console; a regular file there causes all init output
to be silently discarded to RAM — the VM appears stuck at 'Run /init as init
process' with no subsequent console output and lb-agent never heartbeats.

Fix: after rootfs population, wipe /dev and recreate console (c 5 1) and
null (c 1 3) via sudo mknod. devtmpfs provides all other devices at runtime
once init.sh mounts it.

* fix(microvm): fix mgmt NIC routing for all LB schemes

Two bugs prevented lb-agent from reaching the AWSGW via the mgmt interface:

1. RouteDst/RouteVia swapped — resolveMgmtRoute returns (gateway, target) but
   buildMicrovmNICs assigned them as (Dst=gateway, Via=target), producing
   'ip route add 10.15.8.1 via 192.168.1.33' instead of the correct
   'ip route add 192.168.1.33 via 10.15.8.1'.

2. NIC[1] MAC and CIDR never filled — the daemon allocates mgmt IP/MAC
   after buildMicrovmNICs returns, but didn't inject them into input.NICs[1]
   before calling writeFwCfgBlobs, leaving NIC1_MAC= empty in the netcfg
   blob and skipping interface setup in init.sh entirely.

Also extends mgmt routing to internet-facing scheme in the microvm path:
VPC SNAT is not reliable across all VPC topologies for control-plane
heartbeats; mgmt is always a direct L2 path to the host.

* feat(microvm): Phase 2 kernel stripping + boot optimisation

- cpu: switch to host,-pmu (disable PMU — not needed in microvm guest)
- machine type: make isa-serial configurable via microvm.isa_serial config
  field; default true (on) preserves console log path
- build: strip unused kernel modules (USB, IDE, ATA, GPU, media, input,
  HID, bluetooth, staging, sound, wireless) from initramfs before cpio
- build: add Phase 2 artifact size gate — fail if vmlinuz + initramfs.cpio.gz
  exceeds MICROVM_ARTIFACT_MAX_MiB (default 50 MiB); override via env var
  during development
- perf: add phase2BootGateMs=300 and phase2ArtifactGateMiB=50 constants
  with canary tests to catch unintentional gate changes
- tests: cover buildMicrovmNICs nil-VPC and extra-ENI paths
- tests: cover microvmMachineType nil/true/false branches

* fix(build): use find-based virtio+fw_cfg module keep-list in initramfs strip

The directory-only keep-list (drivers/virtio, drivers/firmware) missed
virtio_net.ko which lives in drivers/net/, virtio_blk in drivers/block/,
and virtio-rng in drivers/char/hw_random/. Switch to find-based approach:
collect all virtio*.ko* and *fw_cfg*.ko* regardless of subdirectory, then
restore them into their original relative paths under kernel/.

Artifacts now: 42 MiB (vmlinuz 11 MiB + initramfs 31 MiB), gate passes.

* fix(build): keep failover modules in initramfs strip

virtio_net depends on net_failover/failover for live-migration support.
Stripping them left virtio_net with unresolved symbols
(net_failover_create/destroy), so the module failed to load and the
guest came up with only lo. init.sh then skipped every NIC ("no
interface found for MAC ..."), leaving lb-agent with no network path
to AWSGW and the LB stuck in provisioning.

Add *failover*.ko* to the keep-list. Image size unchanged at 42 MiB
(under the 50 MiB Phase 2 gate). ELBv2 e2e now reaches active and
2/2 targets healthy for both ALB and NLB internal LBs.

* fix(daemon): restore slog default after viperblock.New mutates it

viperblock.New unconditionally calls vb.SetDebug, which calls
slog.SetDefault with a TextHandler at LevelError — globally silencing
every slog.Info and slog.Warn in the caller process for the rest of
its life. Discovered while validating microvm Phase 2 (mulga-siv-68):
the daemon stopped emitting Info logs the moment the first customer
VM with volumes launched, hiding direct-boot timing,
LaunchSystemInstance completed, and CreateLoadBalancer completed
lines that we need for boot-time tuning.

Workaround in newViperblock: re-install an Info-level TextHandler on
os.Stderr after each viperblock.New call. Also drop the three explicit
vb.SetDebug(false) calls in prepareRootVolume / prepareEFIVolume /
prepareCloudInitVolume — they were no-ops (viperblock.New already
runs SetDebug internally) and each one re-triggered the global
mutation we are trying to undo.

Proper fix is in viperblock itself — tracked as mulga-siv-70: SetDebug
should not call slog.SetDefault. Until then, this restore call is the
shortest path to getting the daemon's observability back so we can
measure t1_to_t2_ms and tune Phase 2.

Preflight note: golangci-lint not available on this dev host, so the
project preflight target cannot run end-to-end. Verified with
go vet ./spinifex/handlers/ec2/instance/... (clean) and
go test ./spinifex/handlers/ec2/instance/... (pass).

* perf(microvm): hit Phase 2 boot/memory gates

t1→t2 hypervisor: 3224 ms → 264 ms (12×). 128 MiB guest now boots
cleanly without OOM, reverting the temporary 512 MiB bump.

Root causes:

1. build-microvm-image.sh staged the entire chroot into the cpio
   without dropping /boot, embedding a duplicate vmlinuz, Alpine's
   stock initramfs-virt, System.map, and a "boot → ." symlink loop
   (~25 MiB). Decompression peak exceeded 128 MiB. Strip /boot after
   the kernel is copied out, before the cpio. Initramfs drops from
   31 MiB → 11 MiB (53 MiB → 28 MiB uncompressed); total artifact
   42 MiB → 22 MiB.

2. vm/lifecycle.go burned 3 s of fixed sleep on every direct-boot
   launch: 2 s nbdkit pre-exec wait (no nbdkit on direct-boot) and
   1 s post-fork settle (QEMU binds QMP within ~10 ms when there is
   no firmware, ROM, or block layer). Both extracted to
   directBoot-aware helpers — nbdkitPreExecWait collapses to 0, and
   qemuStartSettleWait to 50 ms.

3. KernelCmdline only carried `console=ttyS0 virtio_mmio.device=…`.
   Added boot-time perf flags: quiet loglevel=3 (silence printk over
   115200-baud isa-serial), mitigations=off (KVM guest on trusted
   host, single-tenant LB workload), tsc=reliable, no_timer_check,
   reboot=t, i8042.{nopnp,noaux,nokbd} (microvm has no PS/2).

4. init.sh dropped modprobe virtio_mmio (built-in:
   CONFIG_VIRTIO_MMIO=y) and tightened polling: fw_cfg/iface waits go
   from 10 × 200 ms (up to 2 s) to 5 × 20 ms (up to 100 ms).

Phase 2 gates:
  - t1→t2 ≤ 300 ms       ✅ 264 ms / 281 ms
  - artifact ≤ 50 MiB    ✅ 22 MiB (gate already in build script)
  - ELBv2 e2e green      ✅ 32/33 (pre-existing client-VM failure,
                                   unrelated to microvm path)

* build(microvm): wire build into deploy + distro + release pipeline

Adds install-microvm Makefile target (with grouped-target staleness
detection on vmlinuz/initramfs.cpio.gz), wires it into dev-install.sh
and reset-dev-env.sh, makes 'make deploy' reinstall artifacts when
present, and makes 'make distro-amd64' stage build/microvm/* into the
release tarball staging dir. arm64 distro stays unchanged (no aarch64
microvm artifact yet — Phase 4).

Release workflow gains a setup-go step and a 'make build-microvm-image'
step before 'make distro-amd64' on the amd64 runner so the resulting
tarball includes the kernel + initramfs without a manual copy.

Precursor for the Phase 3 PC-machine LB removal (mulga-siv-69).
Closes mulga-siv-71.

* build(ansible): include microvm artifacts in build role tarball

Adds a build-microvm-image task to the ansible build role and stages
build/microvm/* into the role's release tarball so the install role's
setup.sh run picks them up via INSTALL_SPINIFEX_TARBALL. Skipped
gracefully when the build script is absent (older branches without
direct-boot LB wiring).

* feat(config): enable microvm direct-boot LB path by default

Uncomments the [nodes.<node>.daemon.microvm] section in the generated
spinifex.toml template and sets elbv2_enabled = true so all new
clusters bring up ELBv2 LBs via the direct-boot microvm path. Operators
can opt back to the legacy PC-machine path by flipping the flag to
false in the rendered config.

Required for end-to-end CI on this branch to exercise the direct-boot
path. Existing clusters' configs are unchanged (the template only
affects fresh 'spx admin init' renders).

* fix(microvm): bootstrap apk.static when no host rootfs tool exists

Self-hosted CI runners and the GitHub-hosted ubuntu-24.04 image often
have none of apk, docker, or podman installed. apk-tools is not in the
Debian 13 (trixie) repositories. Without one of these the rootfs build
fails up front with 'no rootfs build tool available'.

When all three are absent, fetch apk-tools-static from the configured
Alpine mirror, extract it to a tempdir, and prepend it to PATH so the
existing native-apk strategy below picks it up transparently. Requires
only curl + tar, both of which are present on every CI runner that can
already check out source.

No-op when apk, docker, or podman is already present.

Unblocks https://github.com/mulgadc/spinifex/actions/runs/25988830244

* revert: drop apk.static bootstrap in build-microvm-image

Reverts 0bd76b7. Self-hosted runners will get docker.io installed
out of band; existing Strategy 2 (docker run alpine:3.20) handles
the rootfs build without host sudo or unsigned package install.

* docs(workbook): drop lb-alpine import for microvm direct-boot LB

LBs now launch via direct-boot QEMU microvm using kernel + initramfs
bundled at /usr/share/spinifex/microvm/, so the legacy lb-alpine system
AMI import is no longer required.

* fix(daemon): own tap devices by daemon user so QEMU can attach without CAP_NET_ADMIN

When the daemon runs non-root (production: User=spinifex-daemon), child
QEMU processes inherit that uid but lack CAP_NET_ADMIN. The kernel's tun
attach check then requires the tap owner uid to match the caller's euid.
Without the `user` flag, `ip tuntap add` defaults to root:root and qemu's
TUNSETIFF fails with EPERM ("could not configure /dev/net/tun: Operation
not permitted").

Surfaced by the e2e-nightly cell-17 microvm LB launch on the ci-pseudo
runner where the kernel enforces strictly; legacy EC2 launches on
ci-single runners happened to attach successfully despite the same code
path. Adding `user $current group $gid` to `ip tuntap add` ensures the
tap is owned by whatever uid the daemon (and its qemu children) run as.
Root daemons skip the flag — they have CAP_NET_ADMIN ambient.

* fix(e2e): redirect lb probe chatter to stderr so stalls are visible

test_http_from / test_tcp_from echoed progress to stdout, which the
callers captured into *_OK vars via $(...). Result: 6+ minute retry
loops ran silently in CI, and the "$ok" comparison never matched
"true" because the captured value contained the full multi-line
chatter. Reachability pass/fail counters were also lost in the
subshell.

Redirect helper output to stderr, leave only the true/false flag on
stdout, and move pass/fail to the caller so they actually count.

* fix(microvm): don't /32 AdvertiseIP via mgmt for internet-facing single-node

buildMicrovmNICs overrode resolveMgmtRoute's deliberate empty return
for the single-node internet-facing case by falling back to
(MgmtBridgeIP, AdvertiseIP). That installed a /32 route to AdvertiseIP
via the mgmt NIC inside the LB microvm. On single-node where the
AWSGW host IP equals the WAN client IP, the /32 also catches reply
traffic for inbound HTTP/TCP to the LB's floating IP — replies leave
via the mgmt NIC with src=data-NIC IP, the host kernel drops the
asymmetric flow, and same-chassis ingress hangs.

resolveMgmtRoute already encodes the correct policy:
  - multi-node any scheme        → explicit MgmtRoute{Gateway,Target}
  - single-node internal         → fallback to (MgmtBridgeIP, AdvertiseIP)
  - single-node internet-facing  → empty (reach AWSGW via VPC SNAT)

Honor it. Matches the legacy lbVMUserData path, whose own comment
flagged the stolen-WAN-return-path symptom this introduced.

Test covers all three branches.

* feat(ansible): dev-deploy playbook for in-place artifact swap

Fills the gap between ansible-dev-install (skips when already installed)
and ansible-dev-reset (full teardown + reinstall + smoketest). Mirrors
Makefile `deploy` target: rebuild + replace binaries and microvm
artifacts on a running node, restart spinifex.target, wait for awsgw.

Skips setup.sh (no user/dir/systemd-unit changes) and smoketest so
iteration is fast. Use ansible-dev-reset when changing systemd units,
helper scripts, logrotate, or setup.sh itself.

Adds:
- roles/deploy/tasks/main.yml — install spx + nbdkit plugin + microvm
  artifacts, with plugindir auto-detection via `nbdkit --dump-config`.
- playbooks/dev-deploy.yml — chains build + deploy roles.
- Makefile ansible-dev-deploy target.
- README playbook table + "when to use which" guidance.

* chore(microvm): remove legacy LB Alpine image build pipeline

Phase 3 microvm removal — strip the PC-machine LB image build
artifacts that direct-boot no longer needs. The microvm initramfs
already carries lb-agent + HAProxy via build-microvm-image, so the
lb.conf manifest, lb-setup.sh chroot script, build-lb-image target,
dev-install / reset-dev-env import steps, and ansible LB-image tasks
all become dead.

build-system-image stays — other manifests (e.g. GPU images) still
use it.

* refactor(microvm): remove legacy PC-machine LB boot path

Phase 3 — direct-boot becomes the only ELBv2 LB launch path.

- Drop SystemInstanceInput.DirectBoot / ImageID / UserData (every LB
  caller set DirectBoot=true; legacy cloud-init userdata is gone).
- Drop ELBv2ServiceImpl.lbVMUserData, systemAMI* fields, SetSystemAMIFunc,
  getSystemAMI, and the lazy-AMI wiring in daemon.startCluster.
- Collapse CreateLoadBalancer to a single direct-boot launch input.
- Collapse daemon.LaunchSystemInstance: drop AMI validation, ImageID
  on RunInstancesInput, UserData base64 encode, GenerateVolumes branch.
  buildDirectBootConfig hardcodes /usr/share/spinifex/microvm and
  isa-serial=on (microvmMachineType no longer takes a flag).
- Drop MicrovmConfig from config.DaemonConfig and the
  [daemon.microvm] section from the spinifex.toml template.
- Prune dead tests (lbVMUserData_*, PCMachine_Unchanged, AMI resolver
  paths) and refresh stale comments that still mentioned lbVMUserData.

ARM hosts have no LB launch path until Phase 4 adds direct-boot on
virt — CreateLoadBalancer on aarch64 fails naturally at QEMU exec.

* docs: drop lb-alpine catalog entry from cluster setup guide

LB system instances no longer use an AMI — direct-boot microvm
artifacts ship in the spinifex package.

* feat(ui): remove LB image import banner + LB-AMI gating

Direct-boot microvm artifacts ship in the spinifex package, so the
LB system AMI no longer exists — the "Load balancer image not imported"
banner and the Create-button disable on the LB list / wizard pages
can never fire. Strip:

- lb-image-missing-banner.tsx component
- hasLbImage / isLbImage / LB_IMAGE_NAME / LB_MANAGED_BY_VALUE from
  system-managed.ts (isSystemManagedImage stays — still used by the
  EC2 Images pages to hide system AMIs)
- Banner gating + ec2ImagesQueryOptions imports from the LB list and
  wizard pages
- The corresponding tests that asserted the banner appears

* revert: restore lb.conf + lb-setup.sh for mulga install pipeline

Phase 3 microvm removal (commit 6313d74) deleted these too early — the
mulga install/iso artifact builders (scripts/tofu-cluster/build-install
-artifacts.sh, scripts/tofu-iso/build-iso-artifacts.sh) still invoke
scripts/build-system-image.sh scripts/images/lb.conf, breaking e2e with
"ERROR: Manifest not found: scripts/images/lb.conf".

Restore both files verbatim so the legacy pipeline keeps building until
the mulga-side migration to microvm-only artifacts lands post-merge.
2026-05-18 15:48:04 +10:00
..

title, description, category, tags, resources
title description category tags resources
Setting Up Your Cluster Import an AMI, create a VPC with a public subnet, and launch your first EC2 instance on Spinifex. Getting Started
setup
vpc
ec2
quickstart
title url
Spinifex Repository https://github.com/mulgadc/spinifex
title url
AWS EC2 CLI Reference https://docs.aws.amazon.com/cli/latest/reference/ec2/

Setting Up Your Cluster

Import an AMI, create a VPC with a public subnet, and launch your first instance.

Table of Contents


Overview

This guide assumes Spinifex is already installed and running. If not, follow one of the installation guides first:

Instructions

Prerequisites

Ensure the AWS profile is set:

export AWS_PROFILE=spinifex

1. Import an AMI

List available images and import one matching your architecture:

spx admin images list
NAME                    | DISTRO | VERSION | ARCH   | BOOT
debian-13-arm64         | debian | 13      | arm64  | bios
debian-13-x86_64        | debian | 13      | x86_64 | bios
ubuntu-26.04-arm64      | ubuntu | 26.04   | arm64  | bios
ubuntu-26.04-x86_64     | ubuntu | 26.04   | x86_64 | bios

Import an image:

spx admin images import --name ubuntu-26.04-x86_64

Or import a local image file:

spx admin images import --file ~/images/ubuntu-26.04.img --distro ubuntu --version 26.04 --arch x86_64

Verify the import and note the AMI ID:

aws ec2 describe-images

2. Create an SSH Key

Option A: Import an existing key

aws ec2 import-key-pair \
  --key-name "spinifex-key" \
  --public-key-material fileb://~/.ssh/id_rsa.pub

Option B: Create a new key pair

aws ec2 create-key-pair --key-name spinifex-key \
  | jq -r '.KeyMaterial | rtrimstr("\n")' > ~/.ssh/spinifex-key

chmod 600 ~/.ssh/spinifex-key
ssh-keygen -y -f ~/.ssh/spinifex-key > ~/.ssh/spinifex-key.pub

Verify:

aws ec2 describe-key-pairs

3. Create a VPC and Public Subnet

Create a VPC

VPC_ID=$(aws ec2 create-vpc --cidr-block 10.200.0.0/16 \
  --query 'Vpc.VpcId' --output text)

echo "VPC: $VPC_ID"

Create an Internet Gateway

An Internet Gateway enables instances in public subnets to reach the internet and be reachable from the LAN/WAN.

IGW_ID=$(aws ec2 create-internet-gateway \
  --query 'InternetGateway.InternetGatewayId' --output text)

aws ec2 attach-internet-gateway \
  --internet-gateway-id $IGW_ID \
  --vpc-id $VPC_ID

echo "IGW: $IGW_ID (attached to $VPC_ID)"

Create a Public Subnet

A public subnet auto-assigns a routable IP to each instance, making it directly reachable from your network.

SUBNET_ID=$(aws ec2 create-subnet \
  --vpc-id $VPC_ID \
  --cidr-block 10.200.1.0/24 \
  --query 'Subnet.SubnetId' --output text)

# Enable auto-assign public IP
aws ec2 modify-subnet-attribute \
  --subnet-id $SUBNET_ID \
  --map-public-ip-on-launch

echo "Subnet: $SUBNET_ID (public)"

Verify

aws ec2 describe-vpcs --vpc-ids $VPC_ID
aws ec2 describe-subnets --subnet-ids $SUBNET_ID

4. Launch an Instance

Query available instance types for your hardware:

aws ec2 describe-instance-types

Launch an instance in the public subnet, note this will select an instance type with 2 VCPU and 1024MB of memory.

Note: Replace AMI_NAME with the previously imported image above with the ami- prefix.

AMI_NAME="ami-ubuntu-26.04-x86_64"

AMI_ID=$(aws ec2 describe-images \
  --filters "Name=name,Values=$AMI_NAME" \
  --query 'Images | sort_by(@, &CreationDate) | [-1].ImageId' \
  --output text)

INSTANCE_TYPE=$(aws ec2 describe-instance-types \
  --query "sort_by(InstanceTypes[?VCpuInfo.DefaultVCpus==\`2\` && MemoryInfo.SizeInMiB>=\`1024\`], &MemoryInfo.SizeInMiB)[0].InstanceType" \
  --output text)

INSTANCE_ID=$(aws ec2 run-instances \
  --image-id $AMI_ID \
  --instance-type $INSTANCE_TYPE \
  --key-name spinifex-key \
  --subnet-id $SUBNET_ID \
  --count 1 \
  --query 'Instances[0].InstanceId' --output text)

echo "Instance: $INSTANCE_ID"

Wait for the instance to reach running state:

aws ec2 describe-instances --instance-ids $INSTANCE_ID \
  --query 'Reservations[0].Instances[0].[State.Name, PrivateIpAddress, PublicIpAddress]' \
  --output text

Expected output:

running    10.200.1.4    192.168.1.155

The instance has both a private IP (VPC overlay) and a public IP (from your external pool, routable on your network).

5. Connect via SSH

SSH directly to the instance's public IP:

PUBLIC_IP=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID \
  --query 'Reservations[0].Instances[0].PublicIpAddress' --output text)

ssh -i ~/.ssh/spinifex-key ec2-user@$PUBLIC_IP

Note: cloud-init takes 30-60 seconds to configure the instance. If SSH is refused, wait and retry.

Once connected, verify the instance has internet access:

curl -s http://ifconfig.me

This should return the instance's public IP or the gateway's SNAT address.

6. Managing Instances

Stop

aws ec2 stop-instances --instance-ids $INSTANCE_ID

Start

aws ec2 start-instances --instance-ids $INSTANCE_ID

Terminate

aws ec2 terminate-instances --instance-ids $INSTANCE_ID

Console Output

View the instance's serial console log (useful for debugging boot issues):

aws ec2 get-console-output --instance-id $INSTANCE_ID \
  --query 'Output' --output text

Multi-Node: Check Instance Placement

On a multi-node cluster, instances are distributed across nodes:

spx get vms

7. Launching the Web UI

Spinifex ships with a built-in web console — an alternative to the AWS CLI, SDKs, and Terraform, analogous to the AWS Management Console. Every action in the UI is a standard AWS SigV4 API call, so the same IAM policies and audit behaviour apply.

Spinifex Web UI

Open the Console

The UI is served by each node on port 3000 over TLS. Replace YOUR_NODE_IP with the address of the node you installed Spinifex on (or localhost if you're on the node itself):

https://YOUR_NODE_IP:3000

Trust the Self-Signed Certificate (required)

On first load, your browser will show a TLS warning — Spinifex generates a self-signed certificate at install time. This is expected.

  1. Accept the warning to reach the login page (exact wording varies by browser — e.g. Chrome: Advanced → Proceed to ..., Firefox: Advanced → Accept the Risk and Continue).

  2. On the login page, click Download Certificate and save spinifex-ca.pem to your machine.

  3. Install the certificate as a trusted root on your workstation:

    • macOS: open spinifex-ca.pem in Keychain Access → System keychain → set Trust to Always Trust.
    • Linux: sudo cp spinifex-ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt && sudo update-ca-certificates
    • Windows: double-click the file → Install Certificate → Local Machine → Trusted Root Certification Authorities.
    • Browser-only (Firefox): Settings → Privacy & Security → Certificates → View Certificates → Authorities → Import and tick Trust this CA to identify websites.
  4. Restart your browser and reload https://YOUR_NODE_IP:3000. The padlock should now show a valid certificate.

Why this is required: the UI logs in by reading your AWS credentials through a trusted TLS channel. Browsers refuse to send credentials over an untrusted connection, so the certificate must be installed as trusted — temporary "Proceed anyway" exceptions won't work for login.

Log In with AWS Credentials

The console authenticates against the AWS credentials in ~/.aws/credentials on the node where Spinifex is installed:

[spinifex]
aws_access_key_id     = AKIA...
aws_secret_access_key = ...

At the Spinifex login screen, paste the Access Key ID and Secret Access Key from the [spinifex] profile (or whichever profile maps to the IAM user or role you want to use). Additional users and policies can be managed through the UI or via aws iam commands — see IAM Users and Policies.

Once logged in, you have browser-based access to every Spinifex feature: launch and manage instances, attach EBS volumes, browse S3 buckets, configure VPCs and security groups, and manage IAM users and keys — all backed by the same AWS-compatible control plane the CLI uses.

Additional Options

Private Subnets (No Public IP)

Create a subnet without --map-public-ip-on-launch:

PRIVATE_SUBNET=$(aws ec2 create-subnet \
  --vpc-id $VPC_ID \
  --cidr-block 10.200.2.0/24 \
  --query 'Subnet.SubnetId' --output text)

Instances in private subnets get a private IP only. They can still reach the internet via SNAT through the VPC router, but are not directly reachable from your network.

Multiple Accounts

Create isolated accounts with their own resources:

spx admin account create --name myteam
export AWS_PROFILE=spinifex-myteam

Troubleshooting

Instance Stuck in Pending

journalctl -u spinifex-daemon -f
aws ec2 describe-images

SSH Connection Refused

cloud-init needs 30-60 seconds after boot. Check instance state:

aws ec2 describe-instances --instance-ids $INSTANCE_ID

No Public IP Assigned

Verify the subnet has MapPublicIpOnLaunch enabled:

aws ec2 describe-subnets --subnet-ids $SUBNET_ID

If MapPublicIpOnLaunch is false:

aws ec2 modify-subnet-attribute --subnet-id $SUBNET_ID --map-public-ip-on-launch

Also verify an Internet Gateway is attached to the VPC:

aws ec2 describe-internet-gateways

Instance Has No Internet Access

Check the VPC router's NAT rules (from the host):

sudo ovn-nbctl lr-nat-list $(sudo ovn-nbctl lr-list | awk '{print $2}' | head -1)

Verify the default route exists:

sudo ovn-nbctl lr-route-list $(sudo ovn-nbctl lr-list | awk '{print $2}' | head -1)