mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-22 12:16:52 +00:00

Files

T

Josh Siviour ab95566795 feat: initial ELBv2 implementation (#122 )

* feat: add ALB target health reporting via HAProxy stats

Agent-side: periodically query HAProxy stats socket and publish
target health status to NATS (elbv2.alb.<id>.health topic).

Control-plane: healthChecker subscribes to health reports from all
ALB agents and updates target health state in the KV store. Wire
into ELBv2ServiceImpl lifecycle and DeregisterTargets cleanup.

E2E: remove health check skip — targets must now become healthy.

* feat: allocate public IP for internet-facing ALB VMs in daemon

Add Scheme field to SystemInstanceInput so the daemon knows whether
an ALB is internet-facing or internal. For internet-facing, allocate
a public IP via externalIPAM and publish vpc.add-nat for the OVN
dnat_and_snat rule. Internal ALBs skip public IP assignment entirely.

* fix: release auto-assigned public IPs on DeleteNetworkInterface

DeleteNetworkInterface previously leaked public IPs and NAT rules.
Now checks if the ENI has a public IP, verifies it's not EIP-owned,
and releases via externalIPAM + publishes vpc.delete-nat. Daemon
injects externalIPAM and EIP KV into VPC service at init time.

Tests cover: auto-assigned IP release with NAT event, EIP-owned IP
skipped, and no-op when externalIPAM is not configured.

* feat: enforce multi-AZ for internet-facing ALBs in pool mode (mulga-770)

In CreateLoadBalancer, when DevNetworking is false (pool mode), require
2+ subnets in different AZs for internet-facing scheme. Dev mode and
internal scheme skip the validation.

* feat: ALB scheme unit tests + DNS internal prefix + public IP in AZ response (mulga-771)

Add 9 unit tests covering scheme-dependent ALB behavior:
- DNS name uses internal- prefix for internal scheme (AWS-compatible)
- Internet-facing ALBs include public IP in AvailabilityZones response
- Internal ALBs have no public IP in AZ response
- Launcher receives correct scheme, delete cleans up VM + ENIs

Also implements the code changes these tests exercise:
- AvailZoneInfo.PublicIP field populated from launcher output
- lbRecordToSDK populates LoadBalancerAddresses for internet-facing

* feat: add custom AMI for ALB to images list

* feat: add internet-facing ALB data plane E2E test (mulga-772)

New E2E script that validates internet-facing ALBs get public IPs and are
reachable from a peer cluster node (external validation). Skips in dev mode.

* fix: set ENDPOINT to runner node IP in ALB E2E workflow

* fix: correct paths filter to reference alb-e2e.yml

* fix: alb ami name and workflow step to import the image

* fix: spx path

* fix: remove ALB AMI import from workflow and added to bootstrap.sh

* fix: use two subnets in different AZs for internet-facing ALB E2E

Multi-AZ enforcement (mulga-770) requires internet-facing ALBs to have
subnets in 2+ AZs in pool mode. Discover AZs and create a subnet in each.

* refactor: remove multi-AZ enforcement for internet-facing ALBs

Team decision: ALBs must work with single-AZ and single-node clusters.
Removes the multi-AZ validation, pool mode helper, associated tests,
and error constant. E2E script reverted to single subnet.

* fix: use correct account ID for ALB ENI public IP assignment

The ENI is created under the caller's account but LaunchSystemInstance
was looking it up with GlobalAccountID (system account), causing the
UpdateENIPublicIP call to fail with "ENI not found". Pass the caller's
account ID via SystemInstanceInput.AccountID.

* fix: add debug logging for ALB public IP assignment failure

* fix: expand debug logging to include peer node and more keywords

* fix: lazy system AMI discovery for ALB VMs

The daemon was caching the system AMI at startup, but the ALB image is
imported after services start (bootstrap step 9). Replace SetSystemAMI
with SetSystemAMIFunc so the AMI is resolved at request time when
CreateLoadBalancer actually needs it.

* fix: resolve ALB VM instance type dynamically from resource manager

The instance type was hardcoded to t3.nano which may not exist on all
clusters. Now the daemon provides a resolver that picks the smallest
available instance type from the resource manager.

* push interim changes

* refactor: rewrite ALB agent from NATS client to HTTP server

Replace NATS subscriptions with an HTTP server on :8405 exposing
POST /config, GET /ping, and GET /health. The daemon will push
config and poll health over HTTP through the VPC network, removing
the agent's direct NATS dependency.

- agent.go: net/http server replaces nats.Conn/Subscriptions
- health.go: remove push-based reportHealth; /health handler
  calls queryHAProxyStats synchronously on each request
- cmd/alb-agent: --nats flag replaced with --listen
- tests: rewritten with httptest instead of NATS test server
- health_checker_test: inline topic strings (HealthTopic removed)

* refactor: replace NATS with HTTP client for daemon-to-ALB agent comms

Daemon now pushes config via POST and polls ping via GET to the ALB
agent's HTTP server on the VPC IP, eliminating the agent's NATS
dependency from the daemon side.

- service_impl: pushConfig uses http.Post, waitForAgentReady uses
  http.Get, both target http://<vpcIP>:8405
- albVMUserData: drop natsURL param (agent no longer needs NATS)
- types: add VPCIP field to LoadBalancerRecord
- daemon: remove SetNATSURL call and 10.0.2.2 translation block
- arn_test: update user data tests for new signature

* refactor: rewrite ALB health checker from NATS subscription to HTTP polling

Health checker now polls each active ALB's GET /health endpoint on a
10s ticker instead of subscribing to a NATS wildcard topic. Removes
the NATS connection dependency from the health checker.

- health_checker: replace nc/sub with httpClient + stopCh polling loop
- newHealthChecker: drop nc param, add agentURLFn for testability
- service_impl: update newHealthChecker call site
- tests: rewrite with httptest, test pollAll/pollOne paths

* refactor: remove NATS wiring from ALB image and make hostfwd dev-only

- alb-setup.sh: remove --nats flag, use --listen=:8405; remove
  placeholder HAProxy frontend on 8405 (agent owns the port now);
  update description and comments
- service_impl: HostfwdPorts (80/443) now only set when DevNetworking
  is enabled — production ALBs are reached via VPC IP directly

* feat: system credentials file + daemon wiring for ALB agent SigV4 auth

Write config/system-credentials.json during admin init/join with the
system access key. Daemon reads it at startup, resolves the gateway URL
from external IPAM (or 10.0.2.2 in dev mode), and wires both into the
ELBv2 service. albVMUserData now injects ALB_GATEWAY_URL, ALB_ACCESS_KEY,
and ALB_SECRET_KEY into cloud-init for ALB VMs.

* feat: ALB agent gateway actions + daemon handlers (heartbeat, config)

Add ALBAgentHeartbeat and GetALBConfig actions to the gateway so the ALB
agent can communicate via the AWS gateway instead of direct HTTP.

- Add agent_types.go with input/output types for both actions
- Add ALBAgentHeartbeat: processes heartbeat, transitions provisioning→active
  on first contact, updates LastHeartbeat, processes health report
- Add GetALBConfig: returns pre-computed HAProxy config text and hash
- Replace pushConfig (HTTP POST to agent) with updateStoredConfig (stores
  config + SHA256 hash on LB record for agent pull)
- Remove waitForAgentReady polling goroutine (heartbeat handles transition)
- Remove HTTP client from ELBv2ServiceImpl (no longer needed)
- Strip health checker HTTP polling (pollLoop/pollAll/pollOne) — health
  reports now arrive directly via ALBAgentHeartbeat
- Add ConfigText, ConfigHash, LastHeartbeat fields to LoadBalancerRecord
- Wire NATS subscriptions + daemon handlers for both new actions
- Add gateway handler files and elbv2Actions map entries
- Comprehensive tests for heartbeat, config, and updateStoredConfig

* feat: rewrite ALB agent as AWS SDK HTTP client (Bead 3r)

Replace the HTTP server agent with a poll-based client that communicates
with the gateway using SigV4-signed requests. The agent now sends
ALBAgentHeartbeat every 5s with health reports and fetches config via
GetALBConfig when the config hash changes.

* feat: add VPC service route for ALB agent gateway access

ALB VMs need to reach the gateway for heartbeat/config polling, but
VPCs without an IGW have no external routing. Instead of requiring
an IGW, the daemon now publishes a vpc.ensure-service-route event
before launching ALB VMs. vpcd handles this by creating minimal
external connectivity (external switch, localnet port, gateway router
port, chassis bindings) if not already present, then adds a /32 static
route for the gateway IP.

Refactors handleIGWAttach to share the external connectivity setup via
ensureExternalConnectivity helper.

* Revert "feat: add VPC service route for ALB agent gateway access"

This reverts commit d4220d8806.

* feat: add management NIC infrastructure for system instance control plane

Add config, IP allocator, TAP helpers, and VM struct fields for the
management NIC that gives system instances (ALBs etc.) L3 access to the
host control plane via br-mgmt, independent of VPC routing.

- DaemonConfig.MgmtBridge (default "br-mgmt")
- MgmtIPAllocator: thread-safe static IP allocation (.10-.249), rebuild from VMs on restart
- generateMgmtMAC (02:a0:00 prefix), MgmtTapName, GetBridgeIPv4
- SetupMgmtTapDevice / CleanupMgmtTapDevice (plain Linux bridge TAP)
- VM struct: MgmtMAC, MgmtIP, MgmtTap fields

* feat: wire management NIC into QEMU launch and cloud-init config

Add QEMU NIC injection for mgmt TAP (virtio-net-pci on br-mgmt),
extend generateNetworkConfig to emit mgmt0 stanza with static IP,
and wire mgmt MAC/IP allocation into LaunchSystemInstance. Includes
mgmt TAP cleanup on instance stop and failed launch cleanup.

* test: add coverage for ELBv2 service lifecycle and setter methods

* fix: move br-mgmt detection before wireALBAgentConfig so gateway URL uses management bridge IP

* fix: generate per-interface cloud-init config when eniMAC present without devMAC

The guard in generateNetworkConfig required both eniMAC and devMAC to
produce per-interface config, falling back to wildcard DHCP otherwise.
System instances (ALBs) have eniMAC + mgmtMAC but no devMAC (DevNetworking
is independent), so the mgmt NIC stanza was never emitted. The wildcard
config does DHCP on all NICs, but br-mgmt has no DHCP server, leaving the
management NIC unconfigured and the agent unable to reach the gateway.

* fix: add wildcard support to ENI description filter + add ca-certificates to ALB image

- ENI DescribeNetworkInterfaces description filter did exact string match,
  but AWS supports * wildcards. E2E test uses "ELB *" which didn't match
  "ELB app/my-alb/lb-123". Added matchFilter helper for wildcard matching.
- ALB image was missing ca-certificates package, so cloud-init ca_certs
  module couldn't add the Spinifex CA to the system trust store. Agent
  TLS handshakes failed with "certificate signed by unknown authority".
- Fixed data race in TestDeleteLoadBalancer_TerminatesVM_WithPublicIP
  where async TerminateSystemInstance goroutine wrote to mock fields
  while the test read them.

* fix: detach ENI before delete on termination + defer ENI cleanup to stopInstance

DeleteLoadBalancer was trying to delete ENIs synchronously, but they were
still attached (in-use) to the running VM. The delete was rejected with
InvalidNetworkInterface.InUse.

Fix: DeleteLoadBalancer only deletes ENIs when no VM was launched (InstanceID
empty). When a VM exists, ENI cleanup is deferred to stopInstance which runs
after QEMU exits. stopInstance now calls DetachENI before DeleteNetworkInterface
to clear the in-use status.

E2E test now polls for ENI cleanup (up to 30s) since VM termination is async.

* fix: detach then delete ENIs in DeleteLoadBalancer

ENIs are attached (in-use) to the ALB VM, so DeleteNetworkInterface
rejects them. Fix: call DetachENI first to clear in-use status, then
delete. stopInstance tolerates already-deleted ENIs.

* test: add internal ALB dataplane E2E test

Add run-elbv2-dataplane-internal-e2e.sh replacing the generic dataplane
test. Internet-facing coverage is already provided by
run-elbv2-dataplane-internet-facing-e2e.sh.

The internal test validates:
- ALB reaches active via mgmt NIC heartbeat (no IGW dependency)
- Scheme=internal, no public IP, internal- DNS prefix
- Health checks pass via VPC L2 (OVN logical switch)
- Round-robin traffic from a VPC client to the ALB's private IP
- ENI cleanup on ALB deletion

Uses a client VM inside the VPC that curls the ALB private IP and
serves results via HTTP on its public IP for host-side verification.

* feat: add NLB constants and data model changes (mulga-838)

* feat: type-aware validation in CreateLoadBalancer (mulga-839)

Accept Type=network for NLB, reject SecurityGroups for NLB, set
CrossZoneEnabled defaults (false for NLB, true for ALB), and use
/net/ ARN path segment for network load balancers.

* feat: type-aware validation in CreateTargetGroup (mulga-840)

Accept TCP/UDP/TLS/TCP_UDP protocols and apply NLB health check
defaults (TCP protocol, no path/matcher, 10s timeout, 3/3 thresholds).
Reject invalid protocols with InvalidParameterValue.

* feat: type-aware validation in CreateListener (mulga-841)

* feat: add NLB listener and protocol compatibility unit tests

Cover the remaining Phase 1 test gaps: NLB CreateListener with all L4
protocols, protocol rejection for wrong LB type, listener-to-target-group
protocol compatibility validation, and default protocol rejection on NLB.

* feat: add TCP HAProxy config generation for NLB

Add TCP mode template and generateNLBConfig function for Network Load
Balancer support. GenerateHAProxyConfig now branches on LB type,
producing mode-tcp configs for NLBs with tcp-check or httpchk health
checks depending on the target group's health check protocol.

* refactor: rename albagent to lbagent for NLB generalization

Rename the ALB-specific agent package and binary to generic LB names
in preparation for NLB support. Both ALB and NLB VMs use the same
agent — it just applies whatever HAProxy config the control plane
generates.

- Rename albagent/ → lbagent/, cmd/alb-agent/ → cmd/lb-agent/
- Rename types: ALBAgentHeartbeat* → LBAgentHeartbeat*, GetALBConfig* → GetLBConfig*
- Rename NATS topics: elbv2.ALBAgentHeartbeat → elbv2.LBAgentHeartbeat
- Rename cloud-init user data: albVMUserData → lbVMUserData with LB_* env vars
- Rename Makefile targets: build-alb-agent → build-lb-agent
- Update gateway action map, daemon subscriptions, service interface
- Fix pre-existing unused matchFilter function lint error
- Fix awserrors test count for InvalidConfigurationRequest

* feat: add sys.micro system instance type for LB VMs (mulga-845)

Add sys.micro (1 vCPU, 128MB) as an internal-only instance type for
system VMs (load balancers, NAT gateways). Registered in the type map
for allocation but excluded from DescribeInstanceTypes, NATS
subscriptions, and resource stats. Daemon SetSystemInstanceTypeFunc
now returns sys.micro instead of scanning for smallest guest type.

* feat: LB image shrink and rename (mulga-846)

Rename alb.conf → lb.conf, alb-setup.sh → lb-setup.sh with IMAGE_SIZE
reduced from 512M to 192M. Update all references to use lb-agent binary
and LB_* env vars. Update COMMANDS.md for NLB support.

* chore: rename file

* feat: NLB E2E test and bug fixes (mulga-847)

Add full NLB data plane E2E test (33 assertions, 9 phases):
VPC/subnet setup, TCP target group with health checks, NLB
creation (type=network), TCP listener, agent heartbeat,
target health verification, TCP traffic round-robin, target
deregistration, and full cleanup verification.

Bug fixes found during E2E testing:
- Fix HAProxy stats socket path mismatch: config used alb-/nlb-
  prefix but agent expects lb- prefix, preventing health reports
- Fix system AMI lookup to match new -lb- image name, falling
  back to legacy -alb- name
- Bump LB image size from 192M to 256M (Alpine base is 200M)

* test: validate NLB internal scheme (mulga-857)

NLB with scheme=internal works out of the box — shared code paths
with ALB handle it correctly. Add unit test and E2E coverage:

- Unit test: NLB internal creates with no public IP, internal- DNS prefix
- E2E phases 10-14: internal NLB lifecycle with VPC client TCP traffic
- E2E passed 56/56 with perfect round-robin (20/20 requests)

* feat: consolidate LB e2e tests into run-lb-e2e.sh (mulga-854)

Replace 3 separate LB dataplane scripts with a single consolidated test
that covers all 4 variants (ALB/NLB × internet-facing/internal).

- Share one VPC, subnet, and 2 dual-purpose app instances across all suites
- ALB + NLB provision in parallel within each phase (internet-facing / internal)
- One client VM tests both internal LBs simultaneously
- Replace sleep 100 with LB health check gates
- Delete standalone alb-e2e.yml, integrate into e2e.yml for both single and multi-node

* chore: disable LB E2E tests in CI until dev env issue resolved

* fix: read LB agent credentials from spinifex.toml instead of system-credentials.json

The daemon couldn't read system-credentials.json due to systemd
hardening (file owned by root:root 0600, not in ReadOnlyPaths).
Use the same system credentials already loaded from spinifex.toml
via the predastore config section — these are identical keys written
by admin init.

* feat: automate LB image build and import (mulga-861)

- dev-install.sh: build lb-agent and run build-system-image.sh --import
  after services start so the LB image is available out of the box
- build-system-image.sh: add --quiet flag to suppress build output,
  remove hardcoded --config path (uses auto-detected default)
- images.go: rename alb-alpine → lb-alpine to match current naming

* update go version

* add LB e2e tests

* fix: clean LB image build dir and wait for nbd device readiness

Previous CI runs can leave stale state in /tmp/lb-image-build/ causing
e2fsck failures. Clean the build dir before each CI build. Also add a
readiness check after qemu-nbd connect to avoid race conditions.

* fix: serialize nbd access with flock for concurrent CI builds

Single-node and multi-node CI jobs share the same host and both build
the LB system image using /dev/nbd0. Use flock to serialize the nbd
connect/mount/customize/disconnect section so concurrent builds wait
rather than collide.

* fix: move flock to cover entire image build, not just nbd section

Concurrent CI jobs share BUILD_DIR and /dev/nbd0. The lock must be
acquired before downloading/copying the image, not just before nbd
connect, otherwise the second job can clobber the first job's files.

* fix: skip LB image build if output is fresh from concurrent CI job

When concurrent CI jobs build the same image, the second job now reuses
the first's output if it's less than 10 minutes old, avoiding a
duplicate download + build cycle.

* fix: filter AMI discovery by name in E2E tests

With the LB Alpine image now imported alongside Ubuntu, the unfiltered
describe-images call picked up the wrong AMI. Filter by the expected
image name so tests use the Ubuntu image for instance lifecycle.

* fix: filter AMI discovery by name in multi-node E2E tests

Same fix as run-e2e.sh — with the LB Alpine image now imported,
the unfiltered describe-images picks the wrong AMI.

* fix: use awsgw bind IP as LB agent gateway URL fallback

When br-mgmt is not configured and dev_networking is off (production
with external pool), the gateway URL was falling through to the
external IPAM pool's GatewayIP — the WAN router, not the Spinifex
node. LB VMs couldn't reach the awsgw, so heartbeats never arrived
and ALBs stayed stuck in provisioning.

Use the awsgw bind IP instead — LB VMs on the OVN overlay can reach
the host node's WAN IP where the gateway listens.

* fix: capture StateReason and daemon logs when LB app instances fail

Instances with user-data terminate immediately in CI. Add
describe-instances StateReason query and daemon log dump to diagnose
whether it's volume_preparation_failed or launch_failed.

* fix: prevent orphaned AMI from breaking LB E2E and avoid stuck provisioning

Three fixes for CI E2E failures:

1. run-e2e.sh: Remove custom AMI from predastore after deleting its
   backing snapshot. Previously the AMI remained discoverable but
   unusable, causing run-lb-e2e.sh to pick it and fail with NoSuchKey
   on the deleted snapshot.

2. run-lb-e2e.sh: Prefer ubuntu AMI by name for app instances instead
   of picking any non-alpine image. Also improve debug log dump to use
   journalctl (binary-install CI uses systemd, not log files).

3. CreateLoadBalancer: Only set provisioning state when a system VM
   was actually launched. Previously a failed launch still set
   provisioning, leaving the LB stuck forever with no agent to
   send heartbeats.

* fix: use AWSGW bind IP as LB agent gateway URL in multi-node

In multi-node deployments, the AWSGW binds to the WAN IP (e.g.
192.168.1.26:9999), but wireLBAgentConfig was using the br-mgmt IP as
the gateway URL. Since the AWSGW doesn't listen on br-mgmt, the LB
agent's heartbeat was refused, leaving internet-facing ALBs stuck in
provisioning.

Internet-facing LBs now use the AWSGW bind IP directly — the VM reaches
it via VPC/external networking. Internal LBs get a host route on the
mgmt NIC so they can reach the AWSGW bind IP through br-mgmt.

* fix: add mgmt route via bootcmd for multi-node internal LBs

Alpine cloud-init does not support v2 routes under ethernets — adding
a routes: block to the network-config breaks all networking in the VM.

Instead, add the host route via a bootcmd in the LB user data. This
runs early in cloud-init and adds `ip route add {awsgw}/32 via
{br-mgmt}` so internal LB agents can reach the AWSGW through the
management NIC when the gateway binds to a specific WAN IP.

Also moves the networking service from boot to default runlevel in the
LB Alpine image to prevent a race with cloud-init.

Verified locally: internal ALB transitions to active with AWSGW bound
to WAN IP (simulating multi-node).

* add LB enable flag to e2e

* test: add elbv2 tests to meet diff coverage threshold

Cover health checker edge cases (invalid JSON, empty servers, zero
thresholds, port-0 fallback) and service_impl paths (mgmt route
cloud-init, protocol validation, missing target group).

* fix: swallowed health report error + needless marshal round-trip in LBAgentHeartbeat

Add handleHealthReportDirect(report) to healthChecker so LBAgentHeartbeat
can pass the struct directly instead of marshaling to JSON and immediately
unmarshaling. Promote handleHealthReport unmarshal error from Debug to Warn
so failures are visible in logs.

* fix: validate credentials before launching LB VM + set failed state on launch error (mulga-867)

Guard lbVMUserData with empty-value check for gatewayURL, systemAccessKey,
and systemSecretKey — blank env vars previously caused agent auth-loop with
no daemon-side log. Set LB state to failed (not active) when VM launch fails
so the API reflects the broken data-plane instead of black-holing traffic.

* fix: data race in getSystemAMI/getSystemInstanceType lazy caching

Use sync.Once to guard lazy resolution of systemAMI and
systemInstanceType fields. Concurrent CreateLoadBalancer calls
previously raced on unsynchronized read+write of these cached values.
The setter methods reset the Once so re-resolution works when the
func is set after initial construction.

Also fix pre-existing nosprintfhostport lint in daemon.go.

* fix: use utils.PublishEvent for NAT events + return errors from isEIPOwned

publishNATEvent in eni.go and daemon was copy-pasted boilerplate — now
delegates to utils.PublishEvent (matching the EIP service pattern).

isEIPOwned previously swallowed all KV errors and returned false, which
could release an EIP-owned public IP if the store was temporarily
unavailable. Now returns (bool, error) so the caller skips release on
KV failure instead of assuming no EIP ownership.

* fix: narrow health checker to only scan target groups for reporting LB

handleHealthReportDirect was calling ListTargetGroups() on every heartbeat
(every 5s per LB agent), scanning all TGs. Now uses report.LBID to follow
LB → listeners → TG ARNs and fetch only the attached target groups via
direct key lookups. Falls back to full scan if LBID is missing.

* fix: pass region through to LB agent SigV4 signing (mulga-871)

LB agent hardcoded us-east-1 as the SigV4 signing region. Thread region
from ELBv2 service config → cloud-init LB_REGION env var → lb-agent
--region flag → lbagent.New() → signer.Sign() call.

* fix: add account-level authorization to LBAgentHeartbeat and GetLBConfig

Both endpoints accepted any accountID without verifying the LB belongs
to that account. Now verify accountID matches the LB's AccountID,
with an exception for the system account (000000000000) which the
ALB agent uses for heartbeats and config fetches.

* fix: upgrade silent failure paths from Debug to Warn/Error

- healthChecker target group list failure: Debug → Warn
- HAProxy stats socket error in agent tick(): Debug → Warn
- ENI delete EIP-owned skip: Debug → Info (operational visibility)
- wireLBAgentConfig: warn when no gateway host is resolved
- detectMgmtIP: warn when bridge exists but IP is empty
- stale mgmt TAP cleanup: log errors instead of discarding

* refactor: deduplicate MAC hash function into utils.HashMAC

Three identical hash-to-MAC implementations (generateDevMAC, generateMgmtMAC,
generateENIMac) now delegate to a shared utils.HashMAC(prefix, id) helper.

* fix: guard LBAgentHeartbeat KV writes to state transitions or stale timestamps

Previously the full LoadBalancerRecord (including ConfigText) was written
to JetStream KV on every heartbeat (every 5s). Now only persists on state
transitions (provisioning→active) or when LastHeartbeat is older than 60s.

* fix: fail ALB launch when public IP allocation fails for internet-facing scheme

Previously, IPAM allocation failure was logged as a warning and the ALB
VM continued launching without a public IP. The ALB reached active state
but was unreachable from the internet. Now the launch returns an error
and cleans up the partially-created instance, causing the ALB to enter
the failed state.

* fix: retry system AMI resolution when image not yet imported

sync.Once cached empty results permanently — if the daemon started
before the system image was imported, all CreateLoadBalancer calls
silently skipped VM creation. Replace sync.Once with a mutex that
only caches non-empty results, allowing retry on subsequent calls.

Same fix applied to getSystemInstanceType for consistency.

* fix: propagate updateStoredConfig errors to API callers (mulga-884)

updateStoredConfig and updateStoredConfigForTargetGroup now return
errors instead of silently swallowing them. Callers (CreateListener,
DeleteListener, RegisterTargets, DeregisterTargets) return
ErrorServerInternal when config persistence fails, preventing the API
from reporting success while HAProxy serves stale configuration.

* test: add unit tests for wireLBAgentConfig, isEIPOwned, toHealthReport (mulga-885)

* fix: parse HAProxy templates once at package level (mulga-875)

generateALBConfig and generateNLBConfig were calling template.New().Parse()
on every invocation. Move to package-level template.Must for one-time parsing.

* fix: fail launch for internal-scheme ALBs when mgmt NIC setup fails (mulga-886)

When management TAP or IP allocation fails, internal-scheme ALBs now
return an error instead of silently continuing without a management NIC.
Without the mgmt NIC, the LB agent cannot reach the gateway and the ALB
gets stuck in provisioning forever. Internet-facing ALBs retain the
existing degrade-and-continue behavior.

* refactor: extract lbArnPathSegment helper for ALB/NLB ARN construction (mulga-877)

The app/net path segment derivation was duplicated in buildLBArn,
buildListenerArn, and inline in CreateLoadBalancer. Extracted into a
single lbArnPathSegment helper.

* refactor: use aws.StringValue in toHealthReport (mulga-878)

Replace manual nil-check-then-deref with aws.StringValue for cleaner
pointer-to-value conversion in the heartbeat health report builder.

* refactor: use testutil.StartTestJetStream in health_checker_test.go (mulga-876)

Replace hand-rolled setupTestNATS (~20 lines) with the shared
testutil.StartTestJetStream helper. Simplify return to just *Store
since no caller needed the *nats.Conn.

* fix: address PR #122 review comments (mulga-887)

- Extract NATEvent named type from anonymous struct in publishNATEvent
- Remove unused build-lb-image.sh thin wrapper
- Remove SPX_ENABLE_LB env var from e2e workflow (unused)
- Remove no-op healthChecker start/stop methods
- Export simple ELBv2 fields, remove trivial setter methods
- Delete old e2e scripts replaced by consolidated run-lb-e2e.sh

* fix: check specific error type for ENI deletion on termination (mulga-888)

ENI deletion errors were all silently downgraded to Debug level,
masking genuine failures (KV issues, permissions, ENI stuck in-use)
that cause orphaned ENIs and leaked IPAM addresses. Now only
NotFound is logged at Debug; all other errors log at Error.

* fix: propagate non-ENODEV errors from GetBridgeIPv4 (mulga-890)

GetBridgeIPv4 returned ("", nil) for ALL InterfaceByName failures,
not just missing devices. Permission errors on /sys/class/net/ were
silently treated as "bridge absent". Now only "no such network
interface" returns nil; other errors propagate to the caller.

* fix: remove backend address from proxy error response (mulga-892)

The reverse proxy error handler included the internal backend
host:port in the XML error body sent to clients. The address is
already logged server-side; the client now sees a generic message.

* fix: release mutex before KV store I/O in health checker (mulga-893)

handleHealthReportDirect held hc.mu across PutTargetGroup calls
(JetStream KV Put — network I/O), serializing concurrent heartbeats
from independent LBs. Now collects changed TGs under the lock and
persists them after releasing it.

* fix: warn when system credentials missing in wireLBAgentConfig (mulga-894)

fix: log DetachENI errors instead of discarding with blank identifier (mulga-895)

* fix: detect veth bridge mode and use centralized NAT for it

The veth-pair bridge topology (Linux bridge ↔ veth ↔ OVS bridge) was
misclassified as direct bridge mode, causing OVN to use distributed NAT.
Distributed NAT hairpin routing fails through the Linux bridge intermediary,
breaking host-to-local-LB connectivity.

Add BridgeModeVeth constant, detect veth-wan-ovs interface in
detectBridgeMode(), and introduce useCentralizedNAT() to gate NAT mode
and nat-addresses=router for both macvlan and veth modes. isMacvlanMode()
is preserved for macvlan-specific behavior (MAC alignment).

* fix: add retry loop for peer NLB reachability test

With centralized NAT, ARP resolution for NAT external IPs relies on
gratuitous ARPs from the gateway chassis. These take time to propagate
to peer nodes. The peer→NLB test had a single 3s attempt while all
other connectivity tests use retry loops.

* fix: use request-reply for vpc.add-nat to prevent ARP propagation races

publishNATEvent was fire-and-forget for vpc.add-nat, meaning the daemon
continued before vpcd committed the OVN NAT rule. This caused intermittent
NLB connectivity failures: the LB was reported active before OVN had
the DNAT rule and sent GARPs for the public IP.

Switch vpc.add-nat to request-reply (RequestEvent) so the daemon waits
for OVN confirmation. Also increase nc timeout from 2-3s to 5s in E2E
tests for more reliable TCP probing during ARP resolution.

* fix: clean up stale NAT rules before adding new ones for reused IPs

When a public IP is returned to the external pool and quickly reallocated
(e.g. instance terminated, then LB created), the fire-and-forget
vpc.delete-nat may not have been processed yet. handleAddNAT now removes
any existing dnat_and_snat rule for the same external IP before creating
the new one, preventing duplicate/conflicting OVN NAT rules.

This was causing intermittent NLB connectivity failures in multi-node E2E
where NLB public IPs collided with recently-freed instance IPs.

* fix: clean up stale NAT rules across all VPC routers on IP reuse

The previous fix (0a93081) only removed stale dnat_and_snat rules from
the target VPC's router. When a public IP is freed from one VPC and
reallocated to a different VPC before the fire-and-forget vpc.delete-nat
is processed, the stale rule on the old VPC's router caused duplicate
OVN ARP responses — traffic was black-holed when the host resolved to
the wrong router's MAC.

DeleteAllNATsByExternalIP now searches every logical router for stale
NAT rules matching the external IP and removes them in one transaction.

Also adds host connectivity diagnostics (ARP, routing, OVN NAT state)
to run-lb-e2e.sh so the next CI failure surfaces the root cause of the
persistent NLB-from-host issue.

* fix: run LB inet tests from actual hosting node, not hardcoded node1

The host/peer connectivity tests were hardcoded: "host" = node1 (where
the test script runs), "peer" = node2. But LB VMs land on whichever
node picks up the NATS queue message, so the tests were randomly testing
local vs remote connectivity depending on placement luck.

Now the test:
1. Accepts --nodes <ip1> <ip2> ... (all cluster node IPs)
2. After LB creation, discovers which node hosts each LB VM by checking
   for the ENI's tap device
3. Runs "local" tests from the hosting node and "remote" tests from a
   different node

Also updates e2e.yml to pass --nodes instead of --peer.

2026-04-10 10:27:20 +10:00

33 KiB

Raw Blame History

E2E Test Coverage

Single-Node (`run-e2e.sh`)

Phase 1: Environment Setup

KVM support check
spx admin init (region/az/node config)
CA certificate trust
Start all services (start-dev.sh)
Wait for AWS gateway on localhost:9999

Phase 1b: Cluster Stats CLI

spx get nodes — verify node shows Ready
spx top nodes — verify CPU/MEM resource stats
spx get vms — verify "No VMs found" before any launches

Phase 2: Discovery & Metadata

describe-regions
describe-availability-zones (verify zone name and state)
describe-instance-types (discover available types)
Select nano instance type and detect architecture

Phase 2b: Serial Console Access Settings

get-serial-console-access-status (verify default disabled)
enable-serial-console-access (verify returns true, get confirms true)
disable-serial-console-access (verify returns false, get confirms false)

Phase 3: SSH Key Management

create-key-pair (test-key-1, verify private key material)
import-key-pair (test-key-2, from local RSA key)
describe-key-pairs (verify both keys exist)
delete-key-pair (test-key-2, verify only test-key-1 remains)

Phase 4: Image Management

spx admin images import (file-based, architecture-aware)
describe-images (verify AMI by ID)

Phase 5: Instance Lifecycle

run-instances (launch VM with key pair)
describe-instances (poll pending -> running)

Phase 5a-pre: Cluster Stats CLI (with running VM)

spx get vms — verify running instance appears in output

Phase 5a: Instance Metadata Validation

describe-instances — verify InstanceType matches requested type
Verify KeyName matches requested key
Verify ImageId matches requested AMI
Verify at least 1 BlockDeviceMapping present

Phase 5a-ii: SSH Connectivity & Volume Verification

SSH into instance via QEMU hostfwd port
Verify SSH connectivity (id command returns ec2-user)
Verify root volume size from inside VM matches API-reported size (lsblk vs describe-volumes)
Verify VM hostname

Phase 5a-iii: Console Output

get-console-output succeeds (verify InstanceId in response)

Phase 5 (cont): Root Volume

describe-volumes (verify root volume attached)

Phase 5b: Volume Lifecycle

create-volume (10GB, ap-southeast-2a)
modify-volume (resize to 20GB, poll to verify)
attach-volume (to running instance, /dev/sdf)
describe-volumes (verify in-use + attached state)
detach-volume (verify available state)
delete-volume (verify gone)

Phase 5b-ii: DescribeVolumeStatus

describe-volume-status (on root volume, verify VolumeId in response)

Phase 5c: Snapshot Lifecycle

Uses root volume already attached to running instance (snapshots require a mounted VB instance)
create-snapshot (from attached root volume, with description)
Verify create response fields (VolumeId, VolumeSize, State, Progress)
describe-snapshots (by ID, verify VolumeId/Size/Description)
copy-snapshot (with new description, verify distinct ID)
describe-snapshots (verify both original + copy visible)
delete-snapshot (original, verify gone while copy survives)
delete-snapshot (copy, cleanup)

Phase 5d: Verify Snapshot-Backed Instance Launch

All run-instances calls use the snapshot path (cloneAMIToVolume -> OpenFromSnapshot), so the Phase 5 instance is already snapshot-backed
Verify AMI snapshot exists in Predastore (snap-{amiId}/config.json)
Read Phase 5 root volume's config.json from Predastore
Verify SnapshotID and SourceVolumeName are set (proves zero-copy clone)

Phase 5e: CreateImage Lifecycle

create-image (from running instance, with name and description)
Verify returned ImageId is non-empty
describe-images (verify custom AMI name and state)
Extract backing snapshot ID from Predastore config (for cleanup before termination)

Phase 7: Instance State Transitions

stop-instances (poll -> stopped)

Phase 7a: Attach Volume to Stopped Instance (Error Path)

create-volume (for attach test)
attach-volume to stopped instance (expect IncorrectInstanceState error)
delete-volume (cleanup)

Phase 7b: ModifyInstanceAttribute

modify-instance-attribute (change instance type from nano → xlarge while stopped)
describe-instances (verify type updated in KV)
describe-instance-types (get expected vCPU count and memory for new type)
start-instances (poll -> running with new type)
SSH: nproc — verify vCPU count matches xlarge (4 vCPUs)
SSH: /proc/meminfo MemTotal — verify memory matches xlarge (within 85% of expected)

Phase 7c: RunInstances with count > 1

run-instances --count 2 (launch 2 instances in a single call)
Verify 2 instances returned in response
Poll both to running state
terminate-instances (both, poll -> terminated)

Phase 8: Negative / Error Path Tests

run-instances with malformed AMI ID (expect InvalidAMIID.Malformed)
run-instances with invalid instance type (expect InvalidInstanceType)
attach-volume on in-use volume (expect VolumeInUse)
detach-volume on boot volume (expect OperationNotPermitted)
delete-snapshot on non-existent snapshot (expect InvalidSnapshot.NotFound)
Unsupported Action via raw HTTP (expect InvalidAction or error response)
run-instances with non-existent AMI ID (expect InvalidAMIID.NotFound)
run-instances with non-existent key pair (expect InvalidKeyPair.NotFound)
delete-volume on non-existent volume (expect InvalidVolume.NotFound)
create-key-pair with duplicate name (expect InvalidKeyPair.Duplicate)
import-key-pair with duplicate name (expect InvalidKeyPair.Duplicate)
import-key-pair with invalid key format (expect InvalidKey.Format)
describe-volumes with non-existent volume ID (expect InvalidVolume.NotFound)
describe-images with non-existent AMI ID (expect InvalidAMIID.NotFound)
create-image with duplicate name (expect InvalidAMIName.Duplicate)
delete-key-pair on non-existent key (expect success — idempotent, matches AWS)
modify-instance-attribute on running instance (expect InvalidInstanceID.NotFound — running instances not in stopped KV)

IAM Phase 1: User CRUD

Root auth via iam list-users (root user exists)
create-user (alice, bob with path)
Duplicate user (expect EntityAlreadyExists)
get-user (alice)
get-user non-existent (expect NoSuchEntity)
list-users (verify count)
list-users with --path-prefix filter

IAM Phase 2: Access Key Lifecycle

create-access-key (alice key 1, key 2)
Third key exceeds limit (expect LimitExceeded)
create-access-key for non-existent user (expect NoSuchEntity)
list-access-keys (alice: 2 keys, bob: 0 keys)
update-access-key (Inactive, verify status, reactivate)
delete-access-key (verify count decremented)

IAM Phase 3: User Authentication

Configure IAM user profile with access key
Deactivate key → auth rejected (InvalidClientTokenId)
Reactivate key
Wrong secret → SignatureDoesNotMatch
Non-existent key ID → InvalidClientTokenId
Multi-user simultaneous auth (root + bob)

IAM Phase 4: Policy CRUD

create-policy (EC2ReadOnly, FullAdmin, DenyTerminate, IAMReadOnly, EC2DescribeAll)
Duplicate policy (expect EntityAlreadyExists)
Malformed JSON (expect MalformedPolicyDocument)
get-policy (by ARN)
get-policy non-existent (expect NoSuchEntity)
get-policy-version (v1)
list-policies (verify count)

IAM Phase 5: Policy Attachment & Enforcement

Create charlie with access key
attach-user-policy (alice: EC2ReadOnly + IAMReadOnly, bob: DenyTerminate)
list-attached-user-policies (verify count)
Idempotent attach (no duplicate)
Attach non-existent policy (expect NoSuchEntity)
Attach to non-existent user (expect NoSuchEntity)
Default Deny: charlie (no policies) → AccessDenied on ec2 + iam
Explicit Allow: alice → ec2:Describe{Instances,Vpcs} allowed, ec2:DescribeKeyPairs denied, iam:ListUsers allowed, iam:CreateUser denied
Deny Override: bob → ec2:Describe allowed, ec2:TerminateInstances denied (explicit Deny), iam denied
Root Bypass: root user → all actions succeed
Prefix Wildcard: swap alice to EC2DescribeAll (ec2:Describe*) → Describe* allowed, non-Describe denied
FullAdmin: charlie with FullAdmin → all actions allowed

IAM Phase 6: Policy Lifecycle — Detach & Delete

detach-user-policy → user loses access
delete-policy while attached (expect DeleteConflict)
Detach then delete → get-policy returns NoSuchEntity

IAM Phase 7: IAM Cleanup

Delete all test users (remove keys + policies first)
Verify root-only remains (list-users count = 1)
Delete all test policies (list-policies count = 0)
Clean up AWS CLI profiles

Phase 8: EC2 Account Scoping

Tests that EC2 resources are properly isolated between tenant accounts (Alpha, Beta). Based on docs/development/feature/iam-phase4-e2e-test-guide.md. Skips CreateImage (mulga-612) and instance tags (mulga-613).

Step 1: Account Setup

spx admin account create (Team Alpha, Team Beta)
Configure AWS CLI profiles
Verify auth for both accounts

Step 2: Instance Scoping

Alpha + Beta each launch 1 instance
describe-instances isolation (each sees only own)
OwnerId field matches account ID
Cross-account stop-instances → InvalidInstanceID.NotFound
Cross-account terminate-instances → InvalidInstanceID.NotFound
Cross-account start-instances → InvalidInstanceID.NotFound
Cross-account modify-instance-attribute → InvalidInstanceID.NotFound
Cross-account get-console-output → InvalidInstanceID.NotFound

Step 3: Volume Scoping

Alpha + Beta each create 1 volume
describe-volumes isolation
Cross-account describe-volumes by ID → InvalidVolume.NotFound
Cross-account delete-volume → InvalidVolume.NotFound
Cross-account attach-volume → InvalidVolume.NotFound
Cross-account detach-volume → InvalidVolume.NotFound
Cross-account modify-volume → InvalidVolume.NotFound

Step 4: Key Pair Scoping

Alpha + Beta each create key pair
describe-key-pairs isolation
Same key name in both accounts (namespace isolation — different KeyPairIds)
Cross-account delete-key-pair → no effect on other account's key
import-key-pair → invisible to other account

Step 5: Snapshot Scoping

Alpha + Beta each create snapshot from own volume
describe-snapshots isolation + OwnerId verification
Cross-account delete-snapshot → UnauthorizedOperation
Cross-account create-snapshot from other's volume → InvalidVolume.NotFound

Step 6: VPC/Subnet Scoping

Alpha + Beta each create VPC (same CIDR — no conflict)
describe-vpcs isolation
Cross-account describe-vpcs by ID → InvalidVpcID.NotFound
Cross-account delete-vpc → InvalidVpcID.NotFound
Alpha + Beta each create subnet
describe-subnets isolation
Cross-account create-subnet in other's VPC → InvalidVpcID.NotFound
Cross-account delete-subnet → InvalidSubnetID.NotFound

Step 7: IGW + EIGW Scoping

Alpha + Beta each create IGW
describe-internet-gateways isolation
Cross-account describe-internet-gateways by ID → InvalidInternetGatewayID.NotFound
Cross-account delete-internet-gateway → InvalidInternetGatewayID.NotFound
Cross-account attach-internet-gateway → InvalidInternetGatewayID.NotFound
Cross-account detach-internet-gateway → InvalidInternetGatewayID.NotFound
Alpha + Beta each create EIGW
describe-egress-only-internet-gateways isolation
Cross-account EIGW delete → no effect

Step 8: Account Settings

Alpha enable-ebs-encryption-by-default → Beta unaffected
Independent toggle verification

Step 9: Global Resources

describe-regions identical for both accounts
describe-availability-zones identical
describe-instance-types identical

Step 10: Edge Cases

Empty account (Gamma) — no resources visible
Root isolation from tenants (root cannot see tenant instances)
Non-existent resource IDs → same error as cross-account (no info leakage)

Step 11: Cleanup

Terminate instances, delete volumes/snapshots/keys/VPCs/IGWs/EIGWs/subnets
Clean up AWS CLI profiles

Phase 9: Terminate and Verify Cleanup

delete-snapshot (CreateImage backing snapshot, so DeleteOnTermination is not blocked)
terminate-instances (poll -> terminated)

Phase 9a: SSH Unreachable Verification

Verify SSH connection is refused/unreachable after termination

Phase 9b: Volume Cleanup Verification

describe-volumes on root volume after termination
Verify root volume is deleted (DeleteOnTermination)

Pseudo Multi-Node (`run-pseudo-multinode-e2e.sh`)

Phase 1: Environment Setup

KVM support check
Simulated network IPs (10.11.12.{1,2,3} on loopback)
Simulated network IPs (no ramdisk — start-dev.sh uses disk-backed WAL/VB in CI)

Phase 2: Cluster Initialization

spx admin init (leader node1)
CA certificate trust
Start node1 services
spx admin join (node2, node3)
Start node2 + node3 services

Phase 3: Cluster Health Verification

Verify NATS cluster (3 nodes)
Verify Predastore cluster (3 nodes)
Wait for gateway on node1
Wait for daemon NATS readiness
describe-regions (gateway connectivity check)

Phase 3b: Cluster Stats CLI

spx get nodes — verify all 3 nodes show Ready
spx top nodes — verify instance type capacity table
spx get vms — verify empty (no instances yet)

Phase 4: Image and Key Setup

describe-instance-types (discover + select nano)
create-key-pair
spx admin images import (with node1 config paths)
describe-images (verify AMI)

Phase 4b: Multi-Node Key Pair Operations

import-key-pair (multinode-test-key-2, from local RSA key)
describe-key-pairs (verify both keys visible across cluster)
delete-key-pair (multinode-test-key-2, verify deletion)

Phase 5: Multi-Node Instance Tests

Test 1: Instance Distribution

run-instances x3 (distribute across nodes)
Poll all instances to running state
Check instance distribution across nodes
spx get vms — verify all instances visible

Test 1a-ii: SSH Connectivity & Volume Verification

SSH into all 3 instances via QEMU hostfwd port
Verify SSH connectivity (id command returns ec2-user)
Verify root volume size from inside VM matches API-reported size (lsblk vs describe-volumes)
Verify VM hostname

Test 1b: Volume Lifecycle

create-volume (10GB)
modify-volume (resize to 20GB)
attach-volume (to first instance)
detach-volume
delete-volume

Test 1c: Snapshot Lifecycle

Uses root volume of first instance (snapshots require a mounted VB instance)
create-snapshot (from attached root volume, with description)
Verify create response fields (VolumeId, VolumeSize, State)
describe-snapshots (by ID, verify fields)
copy-snapshot (with new description)
describe-snapshots (verify both exist)
delete-snapshot (original, verify copy survives)
delete-snapshot (copy, cleanup)

Test 1c-ii: Verify Snapshot-Backed Instance Launch

All run-instances calls use the snapshot path (cloneAMIToVolume -> OpenFromSnapshot), so the Test 1 instances are already snapshot-backed
Verify AMI snapshot exists in Predastore (snap-{amiId}/config.json)
Read first instance's root volume config.json from Predastore
Verify SnapshotID and SourceVolumeName are set (proves zero-copy clone)

Test 1d-ii: Tag Management (Volumes)

create-tags (2 tags on root volume)
describe-tags (filter by resource-id, verify count)
describe-tags (filter by resource-type=volume)
delete-tags (both tags, verify cleanup)

Test 2: DescribeInstances Aggregation

describe-instances (fan-out across all nodes, verify count)

Test 3: Cross-Node Operations

stop-instances (poll -> stopped)
start-instances (poll -> running)

Test 4: NATS Cluster Health (Post-Operations)

Verify NATS cluster still healthy after all operations

Test 5: VM Crash Recovery

Kill QEMU process with SIGKILL (simulate OOM kill)
Verify daemon detects crash (instance transitions to error/pending)
Wait for auto-restart (backoff starts at 5s)
Verify new QEMU PID differs from original
Verify instance reaches running state
Verify SSH connectivity after recovery

Test 5b: Crash Loop Prevention

Kill QEMU 4 times rapidly on a third instance
Verify crash loop is detected and restarts stop after max attempts (3 in 10-min window)
Verify instance reaches error state (won't restart further)

Phase 5c: IAM Accounts & Cross-Account Isolation

Step 1: Create Accounts

spx admin account create (Team Alpha, Team Beta)
Verify sequential 12-digit account IDs
spx admin account list (verify both accounts)

Step 2: Account Admin Auth

Alpha admin authenticates via ec2 + iam
Beta admin authenticates via ec2 + iam

Step 3: Account-Scoped Users

create-user alice in Alpha + Beta (same name, different accounts)
create-user team-member (Alpha), dev-user (Beta)
list-users scoped per account (verify different user lists)
Cross-account isolation: Alpha cannot see Beta's users and vice versa (NoSuchEntity)

Step 4: Account-Scoped Access Keys

create-access-key for alice in Alpha + Beta
Configure separate AWS CLI profiles per account user

Step 5: Account-Scoped Policies & Enforcement

create-policy EC2ReadOnly in Alpha (narrow: DescribeInstances + DescribeVpcs)
create-policy EC2ReadOnly in Beta (broad: ec2:Describe*)
attach-user-policy to alice in both accounts
Alpha alice: DescribeInstances allowed, DescribeKeyPairs denied (narrow)
Beta alice: DescribeInstances allowed, DescribeKeyPairs allowed (broad Describe*)
Both denied: CreateKeyPair

Step 6: Cross-Account Delete Isolation

Delete Alpha's alice (detach policy, delete key, delete user)
Verify Alpha alice gone (NoSuchEntity)
Verify Beta alice unaffected (still exists, auth still works)

Step 7: EC2 Resource Scoping

Tests that EC2 resources are properly isolated between the Alpha/Beta accounts. Skips CreateImage (mulga-612) and instance tags (mulga-613).

7a: Instance Scoping

Alpha + Beta each launch 1 instance
describe-instances isolation (each sees only own)
OwnerId field matches account ID
Cross-account stop-instances → InvalidInstanceID.NotFound
Cross-account terminate-instances → InvalidInstanceID.NotFound
Cross-account get-console-output → InvalidInstanceID.NotFound

7b: Volume Scoping

Alpha + Beta each create 1 volume
describe-volumes isolation
Cross-account describe-volumes by ID → InvalidVolume.NotFound
Cross-account delete-volume → InvalidVolume.NotFound
Cross-account attach-volume → InvalidVolume.NotFound
Cross-account detach-volume → InvalidVolume.NotFound
Cross-account modify-volume → InvalidVolume.NotFound

7c: Key Pair Scoping

Alpha + Beta each create key pair
describe-key-pairs isolation
Same key name in both accounts (namespace isolation — different KeyPairIds)
Cross-account delete-key-pair → no effect on other account's key
import-key-pair → invisible to other account

7d: Snapshot Scoping

Alpha + Beta each create snapshot from own volume
describe-snapshots isolation + OwnerId verification
Cross-account delete-snapshot → UnauthorizedOperation
Cross-account create-snapshot from other's volume → InvalidVolume.NotFound

7e: VPC/Subnet Scoping

Alpha + Beta each create VPC (same CIDR — no conflict)
describe-vpcs / describe-subnets isolation
Cross-account VPC/subnet operations → NotFound

7f: IGW + EIGW Scoping

Alpha + Beta each create IGW + EIGW
Describe isolation for both
Cross-account attach/detach/delete → NotFound

7g: Account Settings

enable-ebs-encryption-by-default per account — independent toggle

7h: Global Resources

describe-regions, describe-availability-zones, describe-instance-types identical for both accounts

Step 8: Edge Cases

Empty account (Gamma) — no resources visible
Root isolation from tenants
Non-existent resource IDs → same error as cross-account
Parallel key creation (race condition) — no cross-contamination

Step 9: EC2 + IAM Cleanup

Terminate instances, delete volumes/snapshots/keys/VPCs/IGWs/EIGWs/subnets
Delete all IAM users, policies, access keys
Clean up AWS CLI profiles

Phase 5d: VPC Networking

Step 1: create-vpc (10.100.0.0/16) + create-subnet (10.100.1.0/24)
Step 2: run-instances x3 with --subnet-id (launch into VPC subnet)
Poll all VPC instances to running state
Step 3: Verify PrivateIpAddress in describe-instances for each instance
Verify SubnetId and VpcId match requested values
Verify at least 1 NetworkInterface per instance
Verify all IPs are unique and in subnet range (10.100.1.x)
Step 4: SSH + ping connectivity (skipped in CI — OVN DHCP wait too slow)
Step 5: Stop/start IP persistence
- stop-instances (all VPC instances, poll -> stopped)
- Verify PrivateIpAddress persists in stopped state
- start-instances (all VPC instances, poll -> running)
- Verify PrivateIpAddress identical after restart
Step 6: Cleanup — terminate VPC instances, delete subnet, delete VPC

Phase 6: Cluster Shutdown + Restart

Test 6a: Dry-Run Shutdown

spx admin cluster shutdown --dry-run
Validate output contains all 5 phases (GATE, DRAIN, STORAGE, PERSIST, INFRA)

Test 6b: Coordinated Cluster Shutdown

spx admin cluster shutdown --force --timeout 60s
Verify all services down on all nodes (gateway, NATS, QEMU)

Test 6c: Cluster Restart + Recovery

Restart all 3 node services
Verify NATS cluster reforms (3 members)
Wait for gateway and daemon readiness
Smoke test: describe-instance-types returns valid results

Test 6d: Instance Relaunch + Terminate

Wait for instances to finish relaunching after restart (pending → running/error)
terminate-instances (all 3 instances)
Poll all to terminated state

Cleanup

Coordinated cluster shutdown (with fallback to per-node PID stops)
Remove simulated IPs

Real Multi-Node (`run-multinode-e2e.sh`)

Runs on a real 3-node libvirt cluster provisioned by OpenTofu (scripts/tofu-cluster/). Each node is a separate VM with its own OVN, NATS, Predastore, and Spinifex daemon. Bootstrap (bootstrap.sh) handles all provisioning before the test script runs.

Bootstrap (pre-test)

OpenTofu provisions 3 libvirt VMs (bottlebrush, ironbark, casuarina) with cloud-init
Wait for all nodes to be SSH-reachable
Copy SSH key to node1 for peer access
git fetch + checkout test branch on all nodes (all 3 repos)
make build on all nodes (parallel)
setup-ovn.sh — primary gets --management, secondaries connect to primary's OVN central
spx admin init on primary, spx admin join on secondaries (formation server handshake)
start-dev.sh on all nodes
Wait for /health (awsgw=ok) on all nodes
Install Spinifex CA certificate on all nodes
import-key-pair (spinifex-key) + spx admin images import (Ubuntu 24.04)
create-vpc (10.200.0.0/16) + create-subnet (10.200.1.0/24)

Phase 1: Pre-flight Validation

KVM support check (/dev/kvm writable)
SSH connectivity to all peer nodes

Phase 2: Cluster Health

NATS cluster: verify 2 unique route peers (3-node quorum)
Predastore reachable on all 3 nodes
AWS gateway reachable on all 3 nodes
Daemon readiness (describe-instance-types returns results)
spx get nodes — verify all 3 nodes show Ready
spx get vms — verify empty (no instances yet)

Phase 3: Instance Lifecycle + Distribution

Discover nano instance type and AMI (from bootstrap import)
run-instances x3 with staggered launches
Poll all instances to running state
Check instance distribution across physical nodes (QEMU process check)
Verify at least 2 different hosting nodes (non-deterministic, non-fatal if all on 1)
spx get vms — verify all 3 instances visible

Phase 4: SSH into Guest VMs

For each instance across all nodes:
- Find hosting node via QEMU process scan
- Extract SSH hostfwd port from QEMU command line on remote node
- Wait for SSH readiness (up to 60s)
- Verify SSH connectivity (id returns ec2-user)
- Verify block device (lsblk)

Phase 5: Volume Lifecycle

create-volume (10GB)
attach-volume (to first instance, /dev/sdf)
Poll for in-use + attached state
detach-volume (poll for available)
delete-volume (verify gone)

Phase 6: Cross-Node Gateway Access

describe-instances via node1 gateway (baseline count)
describe-instances via node2 and node3 gateways
Verify all gateways return the same instance count (fan-out aggregation)

Phase 7: Cross-Node Operations

Find which node hosts instance 0
Pick a different node's gateway for stop
stop-instances via remote gateway (poll -> stopped)
Pick a third node's gateway for start
start-instances via another remote gateway (poll -> running)

Phase 8: Node Failure

stop-dev.sh on node2 (simulate node failure)
Verify node1 still serves requests (describe-instance-types)
Verify node3 still serves requests
Check NATS degraded state (1 peer instead of 2)
describe-instances still works from surviving nodes

Phase 9: Node Recovery

start-dev.sh on node2 (restart failed node)
Wait for NATS cluster to reform (2 peers again)
Verify node2 gateway is back
spx get nodes — verify all 3 nodes Ready again
Verify node2 serves requests after recovery

Phase 10: Cleanup

terminate-instances (all 3 instances, poll -> terminated)

VPC (`run-vpc-e2e.sh`)

Standalone VPC networking test suite. Runs against a running Spinifex endpoint (configurable via ENDPOINT env var). OVN topology tests are skipped when OVN is unavailable.

Phase 1: VPC CRUD

create-vpc (10.99.0.0/16, verify VpcId)
describe-vpcs (by ID, verify exactly 1 returned)

Phase 2: Subnet CRUD

create-subnet (10.99.1.0/24 in VPC, verify SubnetId)
describe-subnets (by ID, verify exactly 1 returned)

Phase 3: Internet Gateway CRUD

create-internet-gateway (verify InternetGatewayId)
describe-internet-gateways (by ID, verify exactly 1 returned)

Phase 4: Internet Gateway Attach / Detach

attach-internet-gateway (IGW to VPC)
describe-internet-gateways (verify attachment VpcId)
delete-internet-gateway on attached IGW (expect DependencyViolation or rejection)
detach-internet-gateway (IGW from VPC)
describe-internet-gateways (verify no attachments)

Phase 5: OVN Topology Verification (requires OVN)

Re-attach IGW for topology inspection
Verify OVN logical switch exists for subnet
Verify OVN logical router exists for VPC
Verify OVN external switch exists (IGW attached)
Verify SNAT rule on VPC router
Verify default route (0.0.0.0/0) on VPC router
Dump full OVN NB DB topology for debugging
Detach IGW
Skipped when ovn-nbctl is not available

Phase 6: Cleanup

delete-internet-gateway
delete-subnet
delete-vpc
Verify OVN cleanup (no VPC routers remaining, when OVN available)

ELBv2 / ALB (`run-elbv2-e2e.sh`)

Standalone ELBv2 (Application Load Balancer) E2E test suite. Requires VPC networking for ALB ENI creation. Configurable endpoint via ENDPOINT env var.

Phase 0: VPC + Subnet Setup

create-vpc (10.200.0.0/16, prerequisite for ALB)
create-subnet (10.200.1.0/24)

Phase 1: Target Group CRUD

create-target-group (HTTP, port 80, verify health check defaults)
describe-target-groups (by ARN, verify single result)
create-target-group (second TG, port 8080)
describe-target-groups (all, verify >= 2)
Duplicate name detection (same name → error)

Phase 2: Target Registration

register-targets (2 fake instances)
describe-target-health (verify 2 targets, initial state)
deregister-targets (remove 1 target)
describe-target-health (verify 1 target remains)

Phase 3: Load Balancer CRUD

create-load-balancer (ALB with subnet)
Verify fields: type=application, state=active, DNS name, scheme=internet-facing
Verify ENIs created (describe-network-interfaces with ELB filter)
describe-load-balancers (by ARN, verify single result)
Duplicate name detection (same name → error)

Phase 4: Listener CRUD

create-listener (port 80, forward to TG)
Verify fields: port=80, protocol=HTTP
describe-listeners (by LB ARN, verify single result)
Duplicate port detection (same port → error)

Phase 5: In-Use Protection

delete-target-group while referenced by listener → error (ResourceInUse)

Phase 6: Listener Deletion

delete-listener
Verify 0 listeners remain
delete-target-group (now succeeds after listener removed)

Phase 7: Load Balancer Deletion

delete-load-balancer
Verify ALB gone from describe
Verify ENIs cleaned up
delete-target-group (second TG)

Phase 8: Error Path Tests

Describe non-existent ALB → empty result
Delete non-existent ALB → error
Delete non-existent TG → error
Create TG without name → error
Create listener on non-existent ALB → error

Consolidated LB Data Plane (`run-lb-e2e.sh`)

Consolidated LB data plane test covering all 4 variants: ALB internet-facing, ALB internal, NLB internet-facing, NLB internal. Shares a single VPC, subnet, and set of dual-purpose app instances (HTTP:80 + TCP:9000) across all suites. Requires pool mode with external IPAM (NOT dev_networking).

Phase 0: Prerequisites

Dev-mode gate (skip if dev_networking enabled)
SSH to peer node (if --peer provided)
Discover instance types (nano), AMI (non-Alpine)
Create key pair (lb-e2e-key)

Phase 1: Shared VPC + Subnet

create-vpc (10.200.0.0/16)
create-internet-gateway + attach
create-subnet (10.200.1.0/24) + MapPublicIpOnLaunch

Phase 2: Launch App Instances

run-instances x2 with dual-purpose cloud-init (HTTP on 80, TCP echo on 9000)
Poll instances to running state, collect private IPs
Poll app ports for readiness (replaces sleep 100)

Phase 3a: ALB Internet-Facing [skip if no --peer]

Create HTTP target group, register 2 targets
create-load-balancer (scheme=internet-facing), verify scheme
Verify ENI has public IP, DNS has no internal- prefix
create-listener (HTTP:80)
Wait for ALB active (agent heartbeat)
Host HTTP traffic test (curl 20x, verify round-robin)
Peer validation (curl from --peer node, 20x)
Cleanup: delete listener, LB, verify ENI removal, delete TG

Phase 3b: ALB Internal

Create HTTP target group, register 2 targets
create-load-balancer (scheme=internal), verify scheme
Verify ENI has NO public IP, DNS has internal- prefix
create-listener (HTTP:80)
Wait for ALB active (mgmt NIC heartbeat)
Wait for targets healthy
Launch client VM (cloud-init curls ALB private IP, serves results)
Poll client status.txt, fetch results, verify round-robin
Terminate client VM
Cleanup: delete LB, verify ENI removal, delete TG

Phase 3c: NLB Internet-Facing [skip if no --peer]

Create TCP target group (port 9000), verify protocol=TCP
Register 2 targets
create-load-balancer (type=network), verify type, ARN /net/
create-listener (TCP:9000)
Wait for NLB active (agent heartbeat)
Verify ENI has public IP
Wait for targets healthy (TCP health check)
Host TCP traffic test (nc 20x, verify round-robin)
Peer validation (nc from --peer node)
Deregister target, verify draining state
Cleanup: delete listener, LB, verify ENI removal, delete TG

Phase 3d: NLB Internal

Create TCP target group (port 9000), register 2 targets
create-load-balancer (type=network, scheme=internal)
Verify type=network, scheme=internal, ARN /net/
Verify ENI has NO public IP, DNS has internal- prefix
create-listener (TCP:9000)
Wait for NLB active (mgmt NIC heartbeat)
Wait for targets healthy (TCP health check)
Launch client VM (cloud-init nc's NLB private IP, serves results)
Poll client status.txt, fetch results, verify TCP responses
Terminate client VM
Cleanup: delete LB, verify ENI removal, delete TG

Phase 4: Cleanup (trap)

Terminate app instances, wait for terminated
Delete key pair, subnet, IGW, VPC

33 KiB Raw Blame History

E2E Test Coverage

Single-Node (run-e2e.sh)

Phase 1: Environment Setup

Phase 1b: Cluster Stats CLI

Phase 2: Discovery & Metadata

Phase 2b: Serial Console Access Settings

Phase 3: SSH Key Management

Phase 4: Image Management

Phase 5: Instance Lifecycle

Phase 5a-pre: Cluster Stats CLI (with running VM)

Phase 5a: Instance Metadata Validation

Phase 5a-ii: SSH Connectivity & Volume Verification

Phase 5a-iii: Console Output

Phase 5 (cont): Root Volume

Phase 5b: Volume Lifecycle

Phase 5b-ii: DescribeVolumeStatus

Phase 5c: Snapshot Lifecycle

Phase 5d: Verify Snapshot-Backed Instance Launch

Phase 5e: CreateImage Lifecycle

Phase 6: Tag Management

Phase 7: Instance State Transitions

Phase 7a: Attach Volume to Stopped Instance (Error Path)

Phase 7b: ModifyInstanceAttribute

Phase 7c: RunInstances with count > 1

Phase 8: Negative / Error Path Tests

IAM Phase 1: User CRUD

IAM Phase 2: Access Key Lifecycle

IAM Phase 3: User Authentication

IAM Phase 4: Policy CRUD

IAM Phase 5: Policy Attachment & Enforcement

IAM Phase 6: Policy Lifecycle — Detach & Delete

IAM Phase 7: IAM Cleanup

Phase 8: EC2 Account Scoping

Step 1: Account Setup

Step 2: Instance Scoping

Step 3: Volume Scoping

Step 4: Key Pair Scoping

Step 5: Snapshot Scoping

Step 6: VPC/Subnet Scoping

Step 7: IGW + EIGW Scoping

Step 8: Account Settings

Step 9: Global Resources

Step 10: Edge Cases

Step 11: Cleanup

Phase 9: Terminate and Verify Cleanup

Phase 9a: SSH Unreachable Verification

Phase 9b: Volume Cleanup Verification

Pseudo Multi-Node (run-pseudo-multinode-e2e.sh)

Phase 1: Environment Setup

Phase 2: Cluster Initialization

Phase 3: Cluster Health Verification

Phase 3b: Cluster Stats CLI

Phase 4: Image and Key Setup

Phase 4b: Multi-Node Key Pair Operations

Phase 5: Multi-Node Instance Tests

Test 1: Instance Distribution

Test 1a-ii: SSH Connectivity & Volume Verification

Test 1b: Volume Lifecycle

Test 1c: Snapshot Lifecycle

Test 1c-ii: Verify Snapshot-Backed Instance Launch

Test 1d: Tag Management (Instances)

Test 1d-ii: Tag Management (Volumes)

Test 2: DescribeInstances Aggregation

Test 3: Cross-Node Operations

Test 4: NATS Cluster Health (Post-Operations)

Test 5: VM Crash Recovery

Test 5b: Crash Loop Prevention

Phase 5c: IAM Accounts & Cross-Account Isolation

Step 1: Create Accounts

Step 2: Account Admin Auth

Step 3: Account-Scoped Users

Step 4: Account-Scoped Access Keys

Step 5: Account-Scoped Policies & Enforcement

Step 6: Cross-Account Delete Isolation

Step 7: EC2 Resource Scoping

Step 8: Edge Cases

Step 9: EC2 + IAM Cleanup

Phase 5d: VPC Networking

Phase 6: Cluster Shutdown + Restart

33 KiB

Raw Blame History

Single-Node (`run-e2e.sh`)

Pseudo Multi-Node (`run-pseudo-multinode-e2e.sh`)

Real Multi-Node (`run-multinode-e2e.sh`)

VPC (`run-vpc-e2e.sh`)

ELBv2 / ALB (`run-elbv2-e2e.sh`)

Consolidated LB Data Plane (`run-lb-e2e.sh`)