Two bugs prevented lb-agent from reaching the AWSGW via the mgmt interface:
1. RouteDst/RouteVia swapped — resolveMgmtRoute returns (gateway, target) but
buildMicrovmNICs assigned them as (Dst=gateway, Via=target), producing
'ip route add 10.15.8.1 via 192.168.1.33' instead of the correct
'ip route add 192.168.1.33 via 10.15.8.1'.
2. NIC[1] MAC and CIDR never filled — the daemon allocates mgmt IP/MAC
after buildMicrovmNICs returns, but didn't inject them into input.NICs[1]
before calling writeFwCfgBlobs, leaving NIC1_MAC= empty in the netcfg
blob and skipping interface setup in init.sh entirely.
Also extends mgmt routing to internet-facing scheme in the microvm path:
VPC SNAT is not reliable across all VPC topologies for control-plane
heartbeats; mgmt is always a direct L2 path to the host.
tar extraction without root silently demotes device files to 0-byte regular
files (mknod requires CAP_MKNOD). The kernel opens /dev/console when wiring
init's stdio to the serial console; a regular file there causes all init output
to be silently discarded to RAM — the VM appears stuck at 'Run /init as init
process' with no subsequent console output and lb-agent never heartbeats.
Fix: after rootfs population, wipe /dev and recreate console (c 5 1) and
null (c 1 3) via sudo mknod. devtmpfs provides all other devices at runtime
once init.sh mounts it.
Serial output was silent (x-option-roms=off suppresses BIOS serial init);
fixed by removing it from the machine string and using -serial chardev:console0
to attach to the machine-created ISA serial device instead of adding a
duplicate isa-serial device (which would create ttyS1 while kernel uses ttyS0).
virtio-net interfaces were invisible to the kernel because QEMU's
auto-kernel-cmdline does not append virtio_mmio.device= params when -append
is used. Fixed by generating explicit virtio_mmio.device=0x200@0xfebNNNNN:N
params for each NIC in KernelCmdline, and modprobing virtio_mmio + virtio_net
in init.sh.
sys.micro bumped to 512 MiB — 128 MiB caused kernel OOM during initramfs
decompression (~68 MiB uncompressed).
BusyBox init ::respawn: entries do not source or export env files; fixed
inittab to use set -a; . /etc/conf.d/lb-agent; set +a before exec.
wireLBAgentConfig was documented as reading the CA cert but never set
elbv2Service.CACert; lb-agent TLS handshakes to AWSGW failed with
x509: certificate signed by unknown authority. Fixed by reading
d.config.NATS.CACert (same local CA that signs the AWSGW server cert)
and assigning to elbv2Service.CACert so writeFwCfgBlobs writes a
non-empty cacert blob into fw_cfg.
Adds vm_test TestExecute_MicrovmFileChardev and updates lifecycle_test
to assert SerialSocket is intentionally empty on direct-boot VMs.
- Log t1→t2 boot timing via slog on the direct-boot path
- measureQEMURSSMiB helper reads /proc/<pid>/status (tested)
- Gate constants: phase1BootGateMs=500, phase1RSSGateMiB=25
- run-lb-e2e.sh: --microvm flag, dual-path CI annotation, density test note
Closes mulga-siv-67
When ELBv2Enabled=true:
- Drop lbVMUserData cloud-config; emit KEY=value LBAgentEnv blob instead
- Set SystemInstanceInput.DirectBoot=true; omit ImageID
- Populate NICs []NICConfig (index 0=primary ENI IsDefault, 1=mgmt, 2+=extra ENIs)
- Validate exactly one IsDefault NIC at service layer
When ELBv2Enabled=false: existing PC-machine path unchanged.
Adds buildLBAgentEnv, buildMicrovmNICs, subnetCIDRForIP, subnetGatewayIP helpers.
Tests in arn_test.go cover both flag-on and flag-off paths.
Closes mulga-siv-66
The 5s ec2.start.{lastNode} forward timeout was firing before the target
node finished StartStoppedInstance (cold start = volume rehydrate + QEMU
launch + QMP handshake, ~10-20s on a t3a.nano). The forwarding daemon
then fell back to starting the same instance locally, racing the
still-running launch on the original node. Both attempts derive the same
deterministic tap name from the ENI, so the second one fails with
TUNSETIFF: Device or resource busy.
Surfaced on the pseudo-multinode E2E (run 26006933491) where all three
daemons share one kernel and the collision is unavoidable; real
multi-node hides it because separate kernels can each hold the same tap
name without conflict (but the duplicate Run is still semantically
wrong).
Changes:
* Bump startStoppedForwardTimeout from 5s to 30s so it matches the
awsgw upstream NATS request budget. A slow-but-alive target node
finishes well within this window.
* Distinguish nats.ErrNoResponders (no active subscriber on
ec2.start.{lastNode} — node genuinely down) from a timeout or other
transport error (subscriber existed but didn't respond). Only the
no-responders case justifies a local-fallback Run; everything else
surfaces ServerInternal so the caller retries instead of double-
launching.
* Add an info log on every forward attempt for easier diagnosis.
* feat(predastore): support 2-node distributed mode with RS(1,1)
Lower minimum node threshold from 3 to 2. Emit data=1,parity=1 when
exactly 2 nodes configured — effectively mirroring. 3+ nodes continue
to use RS(2,1).
* fix(ec2): route StartInstances back to original node with node-down fallback
ec2.start used a plain queue group, so NATS could route a restart request to
any daemon — including a capacity-constrained NUC that can't fit a second
instance. This caused InsufficientInstanceCapacity in the Phase 10 stop/start
test on 2-node bare-metal clusters where one node has significantly fewer
resources.
Each daemon now also subscribes to ec2.start.{nodeId} (no queue group). The
queue-group handler peeks at the instance ID, looks up LastNode from the
stopped KV, and forwards to ec2.start.{lastNode}. If the original node does
not respond within 5 seconds (down or partitioned), the handler falls back to
starting locally on whichever node picked up the request — preserving the
existing behaviour for node-failure scenarios.
The direct handler (ec2.start.{nodeId}) always starts locally and never
forwards, preventing routing loops.
* fix(ec2): fall back to local start when original node returns InsufficientInstanceCapacity
The forwarding handler was relaying all responses from the targeted node
unconditionally, including InsufficientInstanceCapacity. This meant that if
the NUC (node2) was the LastNode but had no remaining capacity, the error
propagated to the caller rather than falling back to the local node which may
have room.
Now only success and non-capacity errors are relayed; an InsufficientInstance-
Capacity response from the targeted node triggers a local start attempt, giving
any other node in the cluster a chance to absorb the instance.
* test(e2e): make VPC stop/start restart non-fatal on InsufficientInstanceCapacity
Asymmetric hardware (NUC node2) may not have capacity to restart all
VPC instances. IP-persists-while-stopped is still verified and fails
hard; restart verification is skipped with a warning on capacity errors.
* test(e2e): density-aware capacity guard across bare-metal multi-node suite
Sub-3-node bare-metal clusters with asymmetric/undersized nodes (e.g. a
4-vCPU NUC alongside a 16-vCPU box) can fit only one t3.nano on the
smallest node after the daemon's host reserve. Phases that direct or
indirectly land >1 instance on that node hit InsufficientInstanceCapacity
and previously hard-failed the entire suite.
Add is_density_skip helper and skip_test counter, and wrap every
density-sensitive run-instances / start-instances / wait-for-running call
site (phases 3, 5, 7, 10, 11) so capacity exhaustion on sub-3-node
clusters records a SKIP rather than a FAIL. On 3+ node clusters the
guard short-circuits to false, preserving existing failure semantics.
Summary block now prints Skipped count + names so degraded coverage is
visible in CI logs.
Refs: mulga-siv-72
* test(e2e): parameterise lb SSH key + user via PEER_SSH_KEY / PEER_SSH_USER
Allow run-lb-e2e.sh to run against clusters that don't use tf-user with the
tofu-cluster SSH key (e.g. bare-metal nodes which use spinifex@ + bm-ci key).
Defaults preserve existing behaviour on the tofu-cluster CI path.
Refs: mulga-siv-73
- SSH wait: 60 → 120 attempts; NBD-backed VMs on bare metal take
longer to boot than tofu-cluster VMs with local storage
- Diagnostic console log: add /run/spinifex/console-<id>.log which
is where Spinifex QEMU actually writes serial output
/routez reports outbound routes; with N nodes each node has N-1 routes.
A 2-node cluster has 1 route, not 2. Test was written for 3-node tofu
cluster and failed on 2-node baremetal.
Hardcoded 'for i in 1 2' loops assumed 3+ nodes, accessing NODE_IPS[2]
which doesn't exist on a 2-node cluster. With set -u this is a fatal
unbound variable error. Use seq 1 $((NODE_COUNT-1)) instead.
Phase 8.3 polled VPC instance IPs via two-hop SSH. In single-node macvlan
mode the host has no route to VPC subnets, so SSH always timed out (10 min
of dead retries) even when the VMs were healthy (ALB targets passing health
checks).
Replace with a LaunchTime comparison: the daemon resets LaunchTime to
time.Now() when it relaunches QEMU on recovery, so LaunchTime >= REBOOT_START
is a fast, network-topology-independent reboot confirmation.
GIT_BRANCH was hardcoded to dev, so workflow_dispatch runs ignored the
ref chosen in the UI. Use github.ref_name for dispatch (and any non-cron
trigger), keep dev as the explicit target for scheduled cron runs.
Trixie OVN moved the pid/socket to /var/run/ovn/; older installs
use /var/run/openvswitch/. Probe which exists so the same binary
works on both layouts.
Add SkipFormation bool to firstboot.Config. When set, the cluster
formation block (spx admin init/join) is replaced with a no-op echo,
leaving formation to the provisioning controller.
Wired through: autoinstall reads SPINIFEX_SKIP_FORMATION=1 from env,
install.go passes it to firstboot.Config. spinifex-init.sh parses the
kernel cmdline param and exports it. boot-handler.go hardcodes
SPINIFEX_SKIP_FORMATION=1 so all bare-metal CI nodes skip formation —
bm-bootstrap.sh always runs spx admin init/join with the correct
--nodes N from nodes.json, enabling proper multi-node formation.
Add a step before the E2E run that ensures the Ubuntu 24.04 noble cloud
image is cached at BM_ISO_CACHE. Downloads from ubuntu.com if missing,
prints cached size if present. Passes path via BM_UBUNTU_IMAGE to
bm-bootstrap.sh which SCPs it to the node for --file import.
The --no-nvram flag removed the UEFI NVRAM boot entry, causing firmware to
fall through to PXE on every reboot. Restore original --recheck behaviour
which was confirmed working in manual testing.
grub-probe scans all block devices even without --recheck. On N100 mini-PCs
with eMMC + NVMe, scanning the eMMC causes grub-install to hang. Also skip
NVRAM writes (--no-nvram) since --removable installs to the default EFI path
without needing to register a boot entry in UEFI firmware variables.
--recheck triggers grub to rescan all block devices. On N100 mini-PCs with
eMMC + NVMe, scanning the eMMC causes grub-install to hang indefinitely.
Device maps are generated fresh on each install so --recheck is redundant.
On N100 mini-PCs with eMMC + NVMe, update-grub's os-prober probes all
block devices via the bind-mounted /dev, hanging indefinitely on the eMMC.
GRUB_DISABLE_OS_PROBER=true skips os-prober in the chroot.
Also apply modernize lint fixes to vm test files (atomic.Int32).
joshsiv-mulga