* feat: add GPU discovery package (mulga-992)
Adds spinifex/gpu package with sysfs-based GPU enumeration, IOMMU group
member lookup, and a PCI device ID model table. Vendor-specific enrichment
via nvidia-smi is attempted opportunistically and falls back to the built-in
table when unavailable. All discovery and IOMMU logic is covered by unit tests
using mock sysfs trees.
* feat: add VFIO bind/unbind and GPU pool manager (mulga-993)
Adds gpu.BindVFIO/UnbindVFIO using sysfs driver_override for per-device
driver switching, and gpu.Manager which owns the claim/release lifecycle
for the GPU pool including full IOMMU group binding, rollback on partial
failure, and marking entries unavailable when rebind fails after VM
termination.
* feat: add G5 instance type generator with GPU model support (mulga-994)
- Add GPUModel type, NVIDIAa10g constant, and GPUModelForVendorDevice lookup
- Add generateGPUTypes to emit g5.* types with GpuInfo populated
- Add IsGPUType helper for consumers filtering GPU-capable types
- Update DetectAndGenerate to accept gpuModels []GPUModel parameter
* feat: integrate GPU passthrough into daemon lifecycle (mulga-995)
- Add gpu_passthrough config flag; discover GPUs and build manager at startup
- Gate g5 instance types on GPU pool availability via canAllocateCount
- Claim GPU (VFIO bind) in LaunchInstance; inject vfio-pci QEMU device
- Release GPU (VFIO unbind, rebind to host driver) in stopInstance
- Re-claim GPU on daemon restart when QEMU process is still alive
- Add UseUEFI / OVMF support for x86_64 VMs (optional, falls back to SeaBIOS)
- Fix Claim to use stored OriginalDriver when device is already vfio-pci bound
* feat: add GPU fields to NodeStatus and spx top nodes (mulga-996)
- Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes)
- Populate from gpuManager in handleNodeStatus
- Show GPU (used/total) column in spx top nodes; "-" when no GPUs present
* feat: Ubuntu 24.04 GPU AMI build pipeline (mulga-997)
Extends build-system-image.sh to support Ubuntu alongside Alpine.
Key differences handled: partition table (parted resizepart p1),
bind mounts for DKMS (/proc /sys /dev), apt vs apk, systemctl vs
rc-update. DISTRO defaults to "alpine" for backward compatibility.
Adds ubuntu-gpu.conf manifest and ubuntu-gpu-setup.sh chroot script
that installs nvidia-headless-550-server with DKMS pre-built against
a pinned linux-image-generic kernel, blacklists nouveau, and prevents
unattended-upgrades from invalidating the pre-built module.
* Fixed GPU passthrough and added test script
* feat: always-probe GPU at startup, surface capability in NodeStatusResponse
- Add gpuProbeResult struct and probeGPU() which runs at daemon startup
regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU
groups, and vfio_pci module presence.
- Store gpuProbe on Daemon struct; split NewDaemon into probe (always)
and activate (opt-in) phases.
- Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse
so operators can see hardware state before enabling passthrough.
- Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but
passthrough not active, 'used/total' = passthrough active.
* feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001)
- Export GenerateGPUTypes for use by the reload path
- Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place
(preserves references held by instanceService) and updates NATS
subscriptions without full daemon restart
- Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP
in a context-aware loop; setupShutdown no longer intercepts SIGHUP
- Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal,
handles false→true (re-probe + activate) and true→false (refuse if
instances running, otherwise deactivate) transitions
* feat: spx admin gpu status/enable/disable commands (mulga-1000)
- admin.SetGPUPassthrough: Go implementation of the idempotent TOML write
(replaces the inline Python heredoc in spx-test-gpu.sh)
- spx admin gpu status: shows GPU hardware, IOMMU/vfio state, passthrough
activation, and advertised g5 instance types via NATS NodeStatusResponse
- spx admin gpu enable: checks prerequisites, writes TOML, sends SIGHUP,
polls until daemon confirms GPUPassthrough=true
- spx admin gpu disable: refuses while GPU instances are running, otherwise
writes TOML, sends SIGHUP, polls until confirmed
- spx-test-gpu.sh: replace Python TOML heredoc with spx admin gpu enable
* fixed admin
* fix: filter Intel iGPU from passthrough pool, use lspci for consumer GPU names
Discover() now skips non-NVIDIA/AMD vendors so Intel iGPUs never enter
the passthrough pool. enrichFromLspci() uses the system PCI ID database
via lspci -vmm to populate the model name for any card not in the
hardcoded table, without needing nvidia-smi or driver tools.
* fix: grant CAP_SYS_ADMIN and vfio device access to spinifex-daemon
GPU passthrough requires writing to /sys/bus/pci/drivers/vfio-pci/unbind
on instance stop. The kernel gates this on CAP_SYS_ADMIN regardless of
file permissions. QEMU subprocesses also need char-vfio rw access to
open /dev/vfio/N for the passed-through device.
* fixing gpu bining permissions
* working state of gpu passthrough on phyiscal box
* installer grub image fixes
* made gpu-passthrough more user facing
* fixed banner
* fixed banner
* banner fix
* gpu passthrough docs
* preflight tweak
* vpcd fixes to set br-wan to promisc for public IP DHCP processes
* fixed vpcd br-wan promisc and veth pair
* ssh keygen fix
* fixes after running /review-branch
* feat: re-wire GPU claim and device injection after vm.Manager refactor
GPU passthrough claim (VFIO bind) and QEMU device injection were lost
when dev's vm.Manager refactor (#185/#186) moved StartInstance/
CreateQMPClient into vm/lifecycle.go. Re-add both in their correct
new homes:
- daemon_handlers_instance.go: claim GPU before vmMgr.Run() in both
RunInstances and StartStoppedInstance handlers; release on failure
- vm/lifecycle.go startQEMU(): inject vfio-pci device with x-vga flag
derived from instance.GPUXVGAEnabled when GPUPCIAddress is set
* fix: Create spinifex user ~/.ssh and keys in firstboot
---------
Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
Replace fragile image-name substring match (-lb-/-alb-) with a
DescribeImages tag filter on spinifex:managed-by=elbv2. Remove the
silent fallback that picks the first image in the store when no LB
image exists — CreateLoadBalancer now returns ServerInternal with a
remediation hint instead of persisting a record in StateFailed.
- daemon: tag filter + error return from the resolver closure
- service_impl: resolver signature is func() (string, error); caches
only successful results so the next call retries after an error
- CreateLoadBalancer gate now requires launcher + networking; AMI is
resolved inside the gate and failures short-circuit before persist
- admin images import: add --name+--file combo (catalog metadata with
a local file source) and a --tag key=value flag for explicit tagging
of file-only imports
- build-system-image.sh + lb.conf: propagate SYSTEM_TAG via --tag so
locally built system AMIs get the right tag even when the catalog
is bypassed
Closes mulga-969
* feat: add ALB target health reporting via HAProxy stats
Agent-side: periodically query HAProxy stats socket and publish
target health status to NATS (elbv2.alb.<id>.health topic).
Control-plane: healthChecker subscribes to health reports from all
ALB agents and updates target health state in the KV store. Wire
into ELBv2ServiceImpl lifecycle and DeregisterTargets cleanup.
E2E: remove health check skip — targets must now become healthy.
* feat: allocate public IP for internet-facing ALB VMs in daemon
Add Scheme field to SystemInstanceInput so the daemon knows whether
an ALB is internet-facing or internal. For internet-facing, allocate
a public IP via externalIPAM and publish vpc.add-nat for the OVN
dnat_and_snat rule. Internal ALBs skip public IP assignment entirely.
* fix: release auto-assigned public IPs on DeleteNetworkInterface
DeleteNetworkInterface previously leaked public IPs and NAT rules.
Now checks if the ENI has a public IP, verifies it's not EIP-owned,
and releases via externalIPAM + publishes vpc.delete-nat. Daemon
injects externalIPAM and EIP KV into VPC service at init time.
Tests cover: auto-assigned IP release with NAT event, EIP-owned IP
skipped, and no-op when externalIPAM is not configured.
* feat: enforce multi-AZ for internet-facing ALBs in pool mode (mulga-770)
In CreateLoadBalancer, when DevNetworking is false (pool mode), require
2+ subnets in different AZs for internet-facing scheme. Dev mode and
internal scheme skip the validation.
* feat: ALB scheme unit tests + DNS internal prefix + public IP in AZ response (mulga-771)
Add 9 unit tests covering scheme-dependent ALB behavior:
- DNS name uses internal- prefix for internal scheme (AWS-compatible)
- Internet-facing ALBs include public IP in AvailabilityZones response
- Internal ALBs have no public IP in AZ response
- Launcher receives correct scheme, delete cleans up VM + ENIs
Also implements the code changes these tests exercise:
- AvailZoneInfo.PublicIP field populated from launcher output
- lbRecordToSDK populates LoadBalancerAddresses for internet-facing
* feat: add custom AMI for ALB to images list
* feat: add internet-facing ALB data plane E2E test (mulga-772)
New E2E script that validates internet-facing ALBs get public IPs and are
reachable from a peer cluster node (external validation). Skips in dev mode.
* fix: set ENDPOINT to runner node IP in ALB E2E workflow
* fix: correct paths filter to reference alb-e2e.yml
* fix: alb ami name and workflow step to import the image
* fix: spx path
* fix: remove ALB AMI import from workflow and added to bootstrap.sh
* fix: use two subnets in different AZs for internet-facing ALB E2E
Multi-AZ enforcement (mulga-770) requires internet-facing ALBs to have
subnets in 2+ AZs in pool mode. Discover AZs and create a subnet in each.
* refactor: remove multi-AZ enforcement for internet-facing ALBs
Team decision: ALBs must work with single-AZ and single-node clusters.
Removes the multi-AZ validation, pool mode helper, associated tests,
and error constant. E2E script reverted to single subnet.
* fix: use correct account ID for ALB ENI public IP assignment
The ENI is created under the caller's account but LaunchSystemInstance
was looking it up with GlobalAccountID (system account), causing the
UpdateENIPublicIP call to fail with "ENI not found". Pass the caller's
account ID via SystemInstanceInput.AccountID.
* fix: add debug logging for ALB public IP assignment failure
* fix: expand debug logging to include peer node and more keywords
* fix: lazy system AMI discovery for ALB VMs
The daemon was caching the system AMI at startup, but the ALB image is
imported after services start (bootstrap step 9). Replace SetSystemAMI
with SetSystemAMIFunc so the AMI is resolved at request time when
CreateLoadBalancer actually needs it.
* fix: resolve ALB VM instance type dynamically from resource manager
The instance type was hardcoded to t3.nano which may not exist on all
clusters. Now the daemon provides a resolver that picks the smallest
available instance type from the resource manager.
* push interim changes
* refactor: rewrite ALB agent from NATS client to HTTP server
Replace NATS subscriptions with an HTTP server on :8405 exposing
POST /config, GET /ping, and GET /health. The daemon will push
config and poll health over HTTP through the VPC network, removing
the agent's direct NATS dependency.
- agent.go: net/http server replaces nats.Conn/Subscriptions
- health.go: remove push-based reportHealth; /health handler
calls queryHAProxyStats synchronously on each request
- cmd/alb-agent: --nats flag replaced with --listen
- tests: rewritten with httptest instead of NATS test server
- health_checker_test: inline topic strings (HealthTopic removed)
* refactor: replace NATS with HTTP client for daemon-to-ALB agent comms
Daemon now pushes config via POST and polls ping via GET to the ALB
agent's HTTP server on the VPC IP, eliminating the agent's NATS
dependency from the daemon side.
- service_impl: pushConfig uses http.Post, waitForAgentReady uses
http.Get, both target http://<vpcIP>:8405
- albVMUserData: drop natsURL param (agent no longer needs NATS)
- types: add VPCIP field to LoadBalancerRecord
- daemon: remove SetNATSURL call and 10.0.2.2 translation block
- arn_test: update user data tests for new signature
* refactor: rewrite ALB health checker from NATS subscription to HTTP polling
Health checker now polls each active ALB's GET /health endpoint on a
10s ticker instead of subscribing to a NATS wildcard topic. Removes
the NATS connection dependency from the health checker.
- health_checker: replace nc/sub with httpClient + stopCh polling loop
- newHealthChecker: drop nc param, add agentURLFn for testability
- service_impl: update newHealthChecker call site
- tests: rewrite with httptest, test pollAll/pollOne paths
* refactor: remove NATS wiring from ALB image and make hostfwd dev-only
- alb-setup.sh: remove --nats flag, use --listen=:8405; remove
placeholder HAProxy frontend on 8405 (agent owns the port now);
update description and comments
- service_impl: HostfwdPorts (80/443) now only set when DevNetworking
is enabled — production ALBs are reached via VPC IP directly
* feat: system credentials file + daemon wiring for ALB agent SigV4 auth
Write config/system-credentials.json during admin init/join with the
system access key. Daemon reads it at startup, resolves the gateway URL
from external IPAM (or 10.0.2.2 in dev mode), and wires both into the
ELBv2 service. albVMUserData now injects ALB_GATEWAY_URL, ALB_ACCESS_KEY,
and ALB_SECRET_KEY into cloud-init for ALB VMs.
* feat: ALB agent gateway actions + daemon handlers (heartbeat, config)
Add ALBAgentHeartbeat and GetALBConfig actions to the gateway so the ALB
agent can communicate via the AWS gateway instead of direct HTTP.
- Add agent_types.go with input/output types for both actions
- Add ALBAgentHeartbeat: processes heartbeat, transitions provisioning→active
on first contact, updates LastHeartbeat, processes health report
- Add GetALBConfig: returns pre-computed HAProxy config text and hash
- Replace pushConfig (HTTP POST to agent) with updateStoredConfig (stores
config + SHA256 hash on LB record for agent pull)
- Remove waitForAgentReady polling goroutine (heartbeat handles transition)
- Remove HTTP client from ELBv2ServiceImpl (no longer needed)
- Strip health checker HTTP polling (pollLoop/pollAll/pollOne) — health
reports now arrive directly via ALBAgentHeartbeat
- Add ConfigText, ConfigHash, LastHeartbeat fields to LoadBalancerRecord
- Wire NATS subscriptions + daemon handlers for both new actions
- Add gateway handler files and elbv2Actions map entries
- Comprehensive tests for heartbeat, config, and updateStoredConfig
* feat: rewrite ALB agent as AWS SDK HTTP client (Bead 3r)
Replace the HTTP server agent with a poll-based client that communicates
with the gateway using SigV4-signed requests. The agent now sends
ALBAgentHeartbeat every 5s with health reports and fetches config via
GetALBConfig when the config hash changes.
* feat: add VPC service route for ALB agent gateway access
ALB VMs need to reach the gateway for heartbeat/config polling, but
VPCs without an IGW have no external routing. Instead of requiring
an IGW, the daemon now publishes a vpc.ensure-service-route event
before launching ALB VMs. vpcd handles this by creating minimal
external connectivity (external switch, localnet port, gateway router
port, chassis bindings) if not already present, then adds a /32 static
route for the gateway IP.
Refactors handleIGWAttach to share the external connectivity setup via
ensureExternalConnectivity helper.
* Revert "feat: add VPC service route for ALB agent gateway access"
This reverts commit d4220d8806.
* feat: add management NIC infrastructure for system instance control plane
Add config, IP allocator, TAP helpers, and VM struct fields for the
management NIC that gives system instances (ALBs etc.) L3 access to the
host control plane via br-mgmt, independent of VPC routing.
- DaemonConfig.MgmtBridge (default "br-mgmt")
- MgmtIPAllocator: thread-safe static IP allocation (.10-.249), rebuild from VMs on restart
- generateMgmtMAC (02:a0:00 prefix), MgmtTapName, GetBridgeIPv4
- SetupMgmtTapDevice / CleanupMgmtTapDevice (plain Linux bridge TAP)
- VM struct: MgmtMAC, MgmtIP, MgmtTap fields
* feat: wire management NIC into QEMU launch and cloud-init config
Add QEMU NIC injection for mgmt TAP (virtio-net-pci on br-mgmt),
extend generateNetworkConfig to emit mgmt0 stanza with static IP,
and wire mgmt MAC/IP allocation into LaunchSystemInstance. Includes
mgmt TAP cleanup on instance stop and failed launch cleanup.
* test: add coverage for ELBv2 service lifecycle and setter methods
* fix: move br-mgmt detection before wireALBAgentConfig so gateway URL uses management bridge IP
* fix: generate per-interface cloud-init config when eniMAC present without devMAC
The guard in generateNetworkConfig required both eniMAC and devMAC to
produce per-interface config, falling back to wildcard DHCP otherwise.
System instances (ALBs) have eniMAC + mgmtMAC but no devMAC (DevNetworking
is independent), so the mgmt NIC stanza was never emitted. The wildcard
config does DHCP on all NICs, but br-mgmt has no DHCP server, leaving the
management NIC unconfigured and the agent unable to reach the gateway.
* fix: add wildcard support to ENI description filter + add ca-certificates to ALB image
- ENI DescribeNetworkInterfaces description filter did exact string match,
but AWS supports * wildcards. E2E test uses "ELB *" which didn't match
"ELB app/my-alb/lb-123". Added matchFilter helper for wildcard matching.
- ALB image was missing ca-certificates package, so cloud-init ca_certs
module couldn't add the Spinifex CA to the system trust store. Agent
TLS handshakes failed with "certificate signed by unknown authority".
- Fixed data race in TestDeleteLoadBalancer_TerminatesVM_WithPublicIP
where async TerminateSystemInstance goroutine wrote to mock fields
while the test read them.
* fix: detach ENI before delete on termination + defer ENI cleanup to stopInstance
DeleteLoadBalancer was trying to delete ENIs synchronously, but they were
still attached (in-use) to the running VM. The delete was rejected with
InvalidNetworkInterface.InUse.
Fix: DeleteLoadBalancer only deletes ENIs when no VM was launched (InstanceID
empty). When a VM exists, ENI cleanup is deferred to stopInstance which runs
after QEMU exits. stopInstance now calls DetachENI before DeleteNetworkInterface
to clear the in-use status.
E2E test now polls for ENI cleanup (up to 30s) since VM termination is async.
* fix: detach then delete ENIs in DeleteLoadBalancer
ENIs are attached (in-use) to the ALB VM, so DeleteNetworkInterface
rejects them. Fix: call DetachENI first to clear in-use status, then
delete. stopInstance tolerates already-deleted ENIs.
* test: add internal ALB dataplane E2E test
Add run-elbv2-dataplane-internal-e2e.sh replacing the generic dataplane
test. Internet-facing coverage is already provided by
run-elbv2-dataplane-internet-facing-e2e.sh.
The internal test validates:
- ALB reaches active via mgmt NIC heartbeat (no IGW dependency)
- Scheme=internal, no public IP, internal- DNS prefix
- Health checks pass via VPC L2 (OVN logical switch)
- Round-robin traffic from a VPC client to the ALB's private IP
- ENI cleanup on ALB deletion
Uses a client VM inside the VPC that curls the ALB private IP and
serves results via HTTP on its public IP for host-side verification.
* feat: add NLB constants and data model changes (mulga-838)
* feat: type-aware validation in CreateLoadBalancer (mulga-839)
Accept Type=network for NLB, reject SecurityGroups for NLB, set
CrossZoneEnabled defaults (false for NLB, true for ALB), and use
/net/ ARN path segment for network load balancers.
* feat: type-aware validation in CreateTargetGroup (mulga-840)
Accept TCP/UDP/TLS/TCP_UDP protocols and apply NLB health check
defaults (TCP protocol, no path/matcher, 10s timeout, 3/3 thresholds).
Reject invalid protocols with InvalidParameterValue.
* feat: type-aware validation in CreateListener (mulga-841)
* feat: add NLB listener and protocol compatibility unit tests
Cover the remaining Phase 1 test gaps: NLB CreateListener with all L4
protocols, protocol rejection for wrong LB type, listener-to-target-group
protocol compatibility validation, and default protocol rejection on NLB.
* feat: add TCP HAProxy config generation for NLB
Add TCP mode template and generateNLBConfig function for Network Load
Balancer support. GenerateHAProxyConfig now branches on LB type,
producing mode-tcp configs for NLBs with tcp-check or httpchk health
checks depending on the target group's health check protocol.
* refactor: rename albagent to lbagent for NLB generalization
Rename the ALB-specific agent package and binary to generic LB names
in preparation for NLB support. Both ALB and NLB VMs use the same
agent — it just applies whatever HAProxy config the control plane
generates.
- Rename albagent/ → lbagent/, cmd/alb-agent/ → cmd/lb-agent/
- Rename types: ALBAgentHeartbeat* → LBAgentHeartbeat*, GetALBConfig* → GetLBConfig*
- Rename NATS topics: elbv2.ALBAgentHeartbeat → elbv2.LBAgentHeartbeat
- Rename cloud-init user data: albVMUserData → lbVMUserData with LB_* env vars
- Rename Makefile targets: build-alb-agent → build-lb-agent
- Update gateway action map, daemon subscriptions, service interface
- Fix pre-existing unused matchFilter function lint error
- Fix awserrors test count for InvalidConfigurationRequest
* feat: add sys.micro system instance type for LB VMs (mulga-845)
Add sys.micro (1 vCPU, 128MB) as an internal-only instance type for
system VMs (load balancers, NAT gateways). Registered in the type map
for allocation but excluded from DescribeInstanceTypes, NATS
subscriptions, and resource stats. Daemon SetSystemInstanceTypeFunc
now returns sys.micro instead of scanning for smallest guest type.
* feat: LB image shrink and rename (mulga-846)
Rename alb.conf → lb.conf, alb-setup.sh → lb-setup.sh with IMAGE_SIZE
reduced from 512M to 192M. Update all references to use lb-agent binary
and LB_* env vars. Update COMMANDS.md for NLB support.
* chore: rename file
* feat: NLB E2E test and bug fixes (mulga-847)
Add full NLB data plane E2E test (33 assertions, 9 phases):
VPC/subnet setup, TCP target group with health checks, NLB
creation (type=network), TCP listener, agent heartbeat,
target health verification, TCP traffic round-robin, target
deregistration, and full cleanup verification.
Bug fixes found during E2E testing:
- Fix HAProxy stats socket path mismatch: config used alb-/nlb-
prefix but agent expects lb- prefix, preventing health reports
- Fix system AMI lookup to match new -lb- image name, falling
back to legacy -alb- name
- Bump LB image size from 192M to 256M (Alpine base is 200M)
* test: validate NLB internal scheme (mulga-857)
NLB with scheme=internal works out of the box — shared code paths
with ALB handle it correctly. Add unit test and E2E coverage:
- Unit test: NLB internal creates with no public IP, internal- DNS prefix
- E2E phases 10-14: internal NLB lifecycle with VPC client TCP traffic
- E2E passed 56/56 with perfect round-robin (20/20 requests)
* feat: consolidate LB e2e tests into run-lb-e2e.sh (mulga-854)
Replace 3 separate LB dataplane scripts with a single consolidated test
that covers all 4 variants (ALB/NLB × internet-facing/internal).
- Share one VPC, subnet, and 2 dual-purpose app instances across all suites
- ALB + NLB provision in parallel within each phase (internet-facing / internal)
- One client VM tests both internal LBs simultaneously
- Replace sleep 100 with LB health check gates
- Delete standalone alb-e2e.yml, integrate into e2e.yml for both single and multi-node
* chore: disable LB E2E tests in CI until dev env issue resolved
* fix: read LB agent credentials from spinifex.toml instead of system-credentials.json
The daemon couldn't read system-credentials.json due to systemd
hardening (file owned by root:root 0600, not in ReadOnlyPaths).
Use the same system credentials already loaded from spinifex.toml
via the predastore config section — these are identical keys written
by admin init.
* feat: automate LB image build and import (mulga-861)
- dev-install.sh: build lb-agent and run build-system-image.sh --import
after services start so the LB image is available out of the box
- build-system-image.sh: add --quiet flag to suppress build output,
remove hardcoded --config path (uses auto-detected default)
- images.go: rename alb-alpine → lb-alpine to match current naming
* update go version
* add LB e2e tests
* fix: clean LB image build dir and wait for nbd device readiness
Previous CI runs can leave stale state in /tmp/lb-image-build/ causing
e2fsck failures. Clean the build dir before each CI build. Also add a
readiness check after qemu-nbd connect to avoid race conditions.
* fix: serialize nbd access with flock for concurrent CI builds
Single-node and multi-node CI jobs share the same host and both build
the LB system image using /dev/nbd0. Use flock to serialize the nbd
connect/mount/customize/disconnect section so concurrent builds wait
rather than collide.
* fix: move flock to cover entire image build, not just nbd section
Concurrent CI jobs share BUILD_DIR and /dev/nbd0. The lock must be
acquired before downloading/copying the image, not just before nbd
connect, otherwise the second job can clobber the first job's files.
* fix: skip LB image build if output is fresh from concurrent CI job
When concurrent CI jobs build the same image, the second job now reuses
the first's output if it's less than 10 minutes old, avoiding a
duplicate download + build cycle.
* fix: filter AMI discovery by name in E2E tests
With the LB Alpine image now imported alongside Ubuntu, the unfiltered
describe-images call picked up the wrong AMI. Filter by the expected
image name so tests use the Ubuntu image for instance lifecycle.
* fix: filter AMI discovery by name in multi-node E2E tests
Same fix as run-e2e.sh — with the LB Alpine image now imported,
the unfiltered describe-images picks the wrong AMI.
* fix: use awsgw bind IP as LB agent gateway URL fallback
When br-mgmt is not configured and dev_networking is off (production
with external pool), the gateway URL was falling through to the
external IPAM pool's GatewayIP — the WAN router, not the Spinifex
node. LB VMs couldn't reach the awsgw, so heartbeats never arrived
and ALBs stayed stuck in provisioning.
Use the awsgw bind IP instead — LB VMs on the OVN overlay can reach
the host node's WAN IP where the gateway listens.
* fix: capture StateReason and daemon logs when LB app instances fail
Instances with user-data terminate immediately in CI. Add
describe-instances StateReason query and daemon log dump to diagnose
whether it's volume_preparation_failed or launch_failed.
* fix: prevent orphaned AMI from breaking LB E2E and avoid stuck provisioning
Three fixes for CI E2E failures:
1. run-e2e.sh: Remove custom AMI from predastore after deleting its
backing snapshot. Previously the AMI remained discoverable but
unusable, causing run-lb-e2e.sh to pick it and fail with NoSuchKey
on the deleted snapshot.
2. run-lb-e2e.sh: Prefer ubuntu AMI by name for app instances instead
of picking any non-alpine image. Also improve debug log dump to use
journalctl (binary-install CI uses systemd, not log files).
3. CreateLoadBalancer: Only set provisioning state when a system VM
was actually launched. Previously a failed launch still set
provisioning, leaving the LB stuck forever with no agent to
send heartbeats.
* fix: use AWSGW bind IP as LB agent gateway URL in multi-node
In multi-node deployments, the AWSGW binds to the WAN IP (e.g.
192.168.1.26:9999), but wireLBAgentConfig was using the br-mgmt IP as
the gateway URL. Since the AWSGW doesn't listen on br-mgmt, the LB
agent's heartbeat was refused, leaving internet-facing ALBs stuck in
provisioning.
Internet-facing LBs now use the AWSGW bind IP directly — the VM reaches
it via VPC/external networking. Internal LBs get a host route on the
mgmt NIC so they can reach the AWSGW bind IP through br-mgmt.
* fix: add mgmt route via bootcmd for multi-node internal LBs
Alpine cloud-init does not support v2 routes under ethernets — adding
a routes: block to the network-config breaks all networking in the VM.
Instead, add the host route via a bootcmd in the LB user data. This
runs early in cloud-init and adds `ip route add {awsgw}/32 via
{br-mgmt}` so internal LB agents can reach the AWSGW through the
management NIC when the gateway binds to a specific WAN IP.
Also moves the networking service from boot to default runlevel in the
LB Alpine image to prevent a race with cloud-init.
Verified locally: internal ALB transitions to active with AWSGW bound
to WAN IP (simulating multi-node).
* add LB enable flag to e2e
* test: add elbv2 tests to meet diff coverage threshold
Cover health checker edge cases (invalid JSON, empty servers, zero
thresholds, port-0 fallback) and service_impl paths (mgmt route
cloud-init, protocol validation, missing target group).
* fix: swallowed health report error + needless marshal round-trip in LBAgentHeartbeat
Add handleHealthReportDirect(report) to healthChecker so LBAgentHeartbeat
can pass the struct directly instead of marshaling to JSON and immediately
unmarshaling. Promote handleHealthReport unmarshal error from Debug to Warn
so failures are visible in logs.
* fix: validate credentials before launching LB VM + set failed state on launch error (mulga-867)
Guard lbVMUserData with empty-value check for gatewayURL, systemAccessKey,
and systemSecretKey — blank env vars previously caused agent auth-loop with
no daemon-side log. Set LB state to failed (not active) when VM launch fails
so the API reflects the broken data-plane instead of black-holing traffic.
* fix: data race in getSystemAMI/getSystemInstanceType lazy caching
Use sync.Once to guard lazy resolution of systemAMI and
systemInstanceType fields. Concurrent CreateLoadBalancer calls
previously raced on unsynchronized read+write of these cached values.
The setter methods reset the Once so re-resolution works when the
func is set after initial construction.
Also fix pre-existing nosprintfhostport lint in daemon.go.
* fix: use utils.PublishEvent for NAT events + return errors from isEIPOwned
publishNATEvent in eni.go and daemon was copy-pasted boilerplate — now
delegates to utils.PublishEvent (matching the EIP service pattern).
isEIPOwned previously swallowed all KV errors and returned false, which
could release an EIP-owned public IP if the store was temporarily
unavailable. Now returns (bool, error) so the caller skips release on
KV failure instead of assuming no EIP ownership.
* fix: narrow health checker to only scan target groups for reporting LB
handleHealthReportDirect was calling ListTargetGroups() on every heartbeat
(every 5s per LB agent), scanning all TGs. Now uses report.LBID to follow
LB → listeners → TG ARNs and fetch only the attached target groups via
direct key lookups. Falls back to full scan if LBID is missing.
* fix: pass region through to LB agent SigV4 signing (mulga-871)
LB agent hardcoded us-east-1 as the SigV4 signing region. Thread region
from ELBv2 service config → cloud-init LB_REGION env var → lb-agent
--region flag → lbagent.New() → signer.Sign() call.
* fix: add account-level authorization to LBAgentHeartbeat and GetLBConfig
Both endpoints accepted any accountID without verifying the LB belongs
to that account. Now verify accountID matches the LB's AccountID,
with an exception for the system account (000000000000) which the
ALB agent uses for heartbeats and config fetches.
* fix: upgrade silent failure paths from Debug to Warn/Error
- healthChecker target group list failure: Debug → Warn
- HAProxy stats socket error in agent tick(): Debug → Warn
- ENI delete EIP-owned skip: Debug → Info (operational visibility)
- wireLBAgentConfig: warn when no gateway host is resolved
- detectMgmtIP: warn when bridge exists but IP is empty
- stale mgmt TAP cleanup: log errors instead of discarding
* refactor: deduplicate MAC hash function into utils.HashMAC
Three identical hash-to-MAC implementations (generateDevMAC, generateMgmtMAC,
generateENIMac) now delegate to a shared utils.HashMAC(prefix, id) helper.
* fix: guard LBAgentHeartbeat KV writes to state transitions or stale timestamps
Previously the full LoadBalancerRecord (including ConfigText) was written
to JetStream KV on every heartbeat (every 5s). Now only persists on state
transitions (provisioning→active) or when LastHeartbeat is older than 60s.
* fix: fail ALB launch when public IP allocation fails for internet-facing scheme
Previously, IPAM allocation failure was logged as a warning and the ALB
VM continued launching without a public IP. The ALB reached active state
but was unreachable from the internet. Now the launch returns an error
and cleans up the partially-created instance, causing the ALB to enter
the failed state.
* fix: retry system AMI resolution when image not yet imported
sync.Once cached empty results permanently — if the daemon started
before the system image was imported, all CreateLoadBalancer calls
silently skipped VM creation. Replace sync.Once with a mutex that
only caches non-empty results, allowing retry on subsequent calls.
Same fix applied to getSystemInstanceType for consistency.
* fix: propagate updateStoredConfig errors to API callers (mulga-884)
updateStoredConfig and updateStoredConfigForTargetGroup now return
errors instead of silently swallowing them. Callers (CreateListener,
DeleteListener, RegisterTargets, DeregisterTargets) return
ErrorServerInternal when config persistence fails, preventing the API
from reporting success while HAProxy serves stale configuration.
* test: add unit tests for wireLBAgentConfig, isEIPOwned, toHealthReport (mulga-885)
* fix: parse HAProxy templates once at package level (mulga-875)
generateALBConfig and generateNLBConfig were calling template.New().Parse()
on every invocation. Move to package-level template.Must for one-time parsing.
* fix: fail launch for internal-scheme ALBs when mgmt NIC setup fails (mulga-886)
When management TAP or IP allocation fails, internal-scheme ALBs now
return an error instead of silently continuing without a management NIC.
Without the mgmt NIC, the LB agent cannot reach the gateway and the ALB
gets stuck in provisioning forever. Internet-facing ALBs retain the
existing degrade-and-continue behavior.
* refactor: extract lbArnPathSegment helper for ALB/NLB ARN construction (mulga-877)
The app/net path segment derivation was duplicated in buildLBArn,
buildListenerArn, and inline in CreateLoadBalancer. Extracted into a
single lbArnPathSegment helper.
* refactor: use aws.StringValue in toHealthReport (mulga-878)
Replace manual nil-check-then-deref with aws.StringValue for cleaner
pointer-to-value conversion in the heartbeat health report builder.
* refactor: use testutil.StartTestJetStream in health_checker_test.go (mulga-876)
Replace hand-rolled setupTestNATS (~20 lines) with the shared
testutil.StartTestJetStream helper. Simplify return to just *Store
since no caller needed the *nats.Conn.
* fix: address PR #122 review comments (mulga-887)
- Extract NATEvent named type from anonymous struct in publishNATEvent
- Remove unused build-lb-image.sh thin wrapper
- Remove SPX_ENABLE_LB env var from e2e workflow (unused)
- Remove no-op healthChecker start/stop methods
- Export simple ELBv2 fields, remove trivial setter methods
- Delete old e2e scripts replaced by consolidated run-lb-e2e.sh
* fix: check specific error type for ENI deletion on termination (mulga-888)
ENI deletion errors were all silently downgraded to Debug level,
masking genuine failures (KV issues, permissions, ENI stuck in-use)
that cause orphaned ENIs and leaked IPAM addresses. Now only
NotFound is logged at Debug; all other errors log at Error.
* fix: propagate non-ENODEV errors from GetBridgeIPv4 (mulga-890)
GetBridgeIPv4 returned ("", nil) for ALL InterfaceByName failures,
not just missing devices. Permission errors on /sys/class/net/ were
silently treated as "bridge absent". Now only "no such network
interface" returns nil; other errors propagate to the caller.
* fix: remove backend address from proxy error response (mulga-892)
The reverse proxy error handler included the internal backend
host:port in the XML error body sent to clients. The address is
already logged server-side; the client now sees a generic message.
* fix: release mutex before KV store I/O in health checker (mulga-893)
handleHealthReportDirect held hc.mu across PutTargetGroup calls
(JetStream KV Put — network I/O), serializing concurrent heartbeats
from independent LBs. Now collects changed TGs under the lock and
persists them after releasing it.
* fix: warn when system credentials missing in wireLBAgentConfig (mulga-894)
fix: log DetachENI errors instead of discarding with blank identifier (mulga-895)
* fix: detect veth bridge mode and use centralized NAT for it
The veth-pair bridge topology (Linux bridge ↔ veth ↔ OVS bridge) was
misclassified as direct bridge mode, causing OVN to use distributed NAT.
Distributed NAT hairpin routing fails through the Linux bridge intermediary,
breaking host-to-local-LB connectivity.
Add BridgeModeVeth constant, detect veth-wan-ovs interface in
detectBridgeMode(), and introduce useCentralizedNAT() to gate NAT mode
and nat-addresses=router for both macvlan and veth modes. isMacvlanMode()
is preserved for macvlan-specific behavior (MAC alignment).
* fix: add retry loop for peer NLB reachability test
With centralized NAT, ARP resolution for NAT external IPs relies on
gratuitous ARPs from the gateway chassis. These take time to propagate
to peer nodes. The peer→NLB test had a single 3s attempt while all
other connectivity tests use retry loops.
* fix: use request-reply for vpc.add-nat to prevent ARP propagation races
publishNATEvent was fire-and-forget for vpc.add-nat, meaning the daemon
continued before vpcd committed the OVN NAT rule. This caused intermittent
NLB connectivity failures: the LB was reported active before OVN had
the DNAT rule and sent GARPs for the public IP.
Switch vpc.add-nat to request-reply (RequestEvent) so the daemon waits
for OVN confirmation. Also increase nc timeout from 2-3s to 5s in E2E
tests for more reliable TCP probing during ARP resolution.
* fix: clean up stale NAT rules before adding new ones for reused IPs
When a public IP is returned to the external pool and quickly reallocated
(e.g. instance terminated, then LB created), the fire-and-forget
vpc.delete-nat may not have been processed yet. handleAddNAT now removes
any existing dnat_and_snat rule for the same external IP before creating
the new one, preventing duplicate/conflicting OVN NAT rules.
This was causing intermittent NLB connectivity failures in multi-node E2E
where NLB public IPs collided with recently-freed instance IPs.
* fix: clean up stale NAT rules across all VPC routers on IP reuse
The previous fix (0a93081) only removed stale dnat_and_snat rules from
the target VPC's router. When a public IP is freed from one VPC and
reallocated to a different VPC before the fire-and-forget vpc.delete-nat
is processed, the stale rule on the old VPC's router caused duplicate
OVN ARP responses — traffic was black-holed when the host resolved to
the wrong router's MAC.
DeleteAllNATsByExternalIP now searches every logical router for stale
NAT rules matching the external IP and removes them in one transaction.
Also adds host connectivity diagnostics (ARP, routing, OVN NAT state)
to run-lb-e2e.sh so the next CI failure surfaces the root cause of the
persistent NLB-from-host issue.
* fix: run LB inet tests from actual hosting node, not hardcoded node1
The host/peer connectivity tests were hardcoded: "host" = node1 (where
the test script runs), "peer" = node2. But LB VMs land on whichever
node picks up the NATS queue message, so the tests were randomly testing
local vs remote connectivity depending on placement luck.
Now the test:
1. Accepts --nodes <ip1> <ip2> ... (all cluster node IPs)
2. After LB creation, discovers which node hosts each LB VM by checking
for the ENI's tap device
3. Runs "local" tests from the hosting node and "remote" tests from a
different node
Also updates e2e.yml to pass --nodes instead of --peer.
tomnewton-mulga