docker export of a container leaks its overlay mount on Docker's
overlay2 driver — the mount survives docker rm, so the layer dirs are
never reaped except by a daemon restart. The lb-agent microVM build ran
docker export on every e2e job across ~9 runners, accumulating ~69G of
orphaned overlay2 layers on the CI host (disk hit 94%).
Build the rootfs with `docker buildx --output type=local` from a small
generated Dockerfile instead, which writes the image filesystem to a
directory and releases all mounts — nothing leaks. Keep the podman path
(its store is unaffected). Tighten the e2e cleanup image prune to
dangling-only so the alpine base stays cached instead of re-pulling.
* test(e2e): add targeted suite selector + shadow-mode CI step
select-suites maps a diff to the e2e suites it must run: changed files →
services (manifest path/additional_paths) → consumer closure over
depends_on vs publishes/subscribes → suites whose covers intersect.
Special cases force-all (infra/harness/workflow, ≥3 services, submodule
bumps) or a lone suite dir; empty diffs fall back to default suites. An
AST scan of each suite's harness.Ensure* call graph derives per-test
service coverage and emits an optional -test.run subset (RUN_PATTERN).
Wired into e2e.yml as a non-gating 'Select Suites (shadow)' job that
logs the computed selection to the run summary so predicted vs actual
can be compared before the selector drives SUITES.
* ci(e2e): add dispatch-only per-suite job blocks grouped by env
Restructure the looped e2e flow into env-grouped provision -> matrix
suites -> teardown blocks so each suite surfaces as its own node in the
Actions graph instead of a single green tick:
single_provision -> single_suite [single,cert,lb,eks] -> single_teardown
multi_provision -> multi_suite [multinode] -> multi_teardown
Provision resolves the VM/cluster and ships test binaries, emitting WAN
IP + ssh key path as job outputs; each matrix leg runs one suite over
SSH and uploads its own junit/analysis artifacts; teardown (if:always)
reuses the persisted tofu workspace on the same runner to destroy.
workflow_dispatch only -- runs alongside the gating e2e.yml for
validation and posts no commit status.
* test(e2e): add manifest drift lint (fixture-create + NATS subject)
Bead 5 of the targeted-suite-selection plan. Two ratcheted drift guards in
tests/e2e/manifest-lint, wired into make preflight and the spinifex.yml PR job:
- Fixture lint: AST-flags direct resource-create AWS calls
(c.EC2.RunInstances, c.ELBv2.CreateLoadBalancer, ...) in suite test files,
steering them through harness.Ensure* fixtures. ~104 pre-existing call sites
are captured in a checked-in baseline; the lint fails only on NEW creates.
Intentional create-path tests opt out with an inline // e2e:allow-create.
- Subject lint: extracts NATS subject literals from the natsSub{} registry and
literal Subscribe/Publish/Request args, cross-referenced wildcard-aware
against service-interfaces.yaml. Code subjects missing from the manifest fail
(ratcheted); manifest subjects with no static reference warn only (many are
produced dynamically). Fixed real drift surfaced on first run: declared the
five spinifex.cluster.shutdown.* phase subjects under spinifex_daemon.
manifest-check is now CI-enforced alongside manifest-lint (previously
preflight/local-only).
* feat: run-scoped e2e teardown sweep
Tag every Ensure* fixture resource with e2e:run=<run-id> when
SPINIFEX_E2E_RUN_ID is set, and add a teardown.test suite that sweeps
everything carrying that tag at end of run — reclaiming resources leaked
by a crashed suite, an interrupted run, or a persistent cluster.
- harness/sweep.go: tagRunResources (EC2 CreateTags) + tagRunELB (ELBv2
AddTags) + SweepRunResources in dependency order (LB, instances, NAT,
snapshots, volumes, images, SGs, subnets, key pairs). Blank run id
no-ops both tagging and sweeping, so the sweep can never touch an
untagged or foreign resource.
- harness/fixtures.go: stamp the run tag on each created resource.
- tests/e2e/teardown: thin suite that runs the sweep; skips without a
cluster or run id, logs the report, never fails the run.
- e2e-suites.yml: build + ship teardown.test, pass the run id into suite
legs, sweep on the VM in both teardown jobs before tofu destroy.
* Revert "fix(network): prime host neigh with gateway port MAC on centralised EIP attach"
This reverts commit f7446c69c6.
* fix(network): round-robin external pool cursor so a released IP is not reused instantly
The static external pool handed out the lowest free address and freed IPs
immediately on release, so a just-released EIP was reassigned within ~0.6s. A
recycled EIP front-end is unreachable from the same host: the kernel still
holds the prior owner's ARP binding (or a negative cache after the neigh
flush) and OVN does not answer the re-ARP in time, so the neighbour entry
settles to FAILED (EHOSTUNREACH). This surfaced as EKS TestEKS/KubectlGetNodes
failing in single-node CI once #315 routed kubectl through the cluster NLB's
external EIP. Dev never hit it — it uses DHCP-random addresses.
Add a persisted per-pool cursor: allocation resumes one past the last address
handed out and wraps at RangeEnd, so a freed IP is not reused until the cursor
cycles the whole range. Empty cursor (fresh pool / older KV record) starts at
RangeStart, preserving lowest-free order for the first cycle — backward
compatible, no migration.
* ci: drop e2e-eks from default e2e suites
EKS data-plane subtest (TestEKS/KubectlGetNodes) is a persistent flake on
single-node CI — a freshly-attached centralised EIP is intermittently
unreachable from the host (no route to host to the cluster endpoint), which
fails the whole E2E run and blocks unrelated suites. Remove e2e-eks from the
default SUITES so PR/push runs no longer gate on it. The e2e-eks target and
the workflow_dispatch suites input are unchanged, so EKS stays runnable on
demand and via make e2e-all.
EKS create-cluster fails deterministically on the multinode job with
ServiceUnavailableException: the EKS KV is a clustered (Raft) JetStream
stream and create-cluster needs an elected leader, but multinode
JetStream clustering isn't reliably forming yet (see mulga-omq/z2z/5yg).
EKS is validated single-node only at this stage.
Drop eks from the multinode build+run suite filters (alongside the
existing reboot/single exclusions) so the multinode job reflects what's
actually supported. Single-node continues to run the full eks suite.
build-microvm-image required depmod (kmod) as a mandatory tool to
generate modules.dep. Self-hosted CI runners don't ship kmod and can't
reliably apt-install it, so the inline 'apt-get install kmod' steps added
to e2e/e2e-nightly/e2e-ddil were unreliable.
modules.dep is an optimisation, not a boot requirement: the guest init's
load_mod() (build/microvm/init.sh) falls back to insmod-by-path when
modprobe can't resolve a module. So:
- Drop depmod from the mandatory-tool preflight.
- Run depmod only when present; still assert a non-empty modules.dep in
that case to keep catching the trixie-kmod empty-dep bug.
- Warn and skip when depmod is absent (guest uses insmod fallback).
- Remove the apt-get kmod install steps from all three workflows.
Gold-image bake still installs kmod (provision.sh), so production images
get a proper modules.dep; only kmod-less CI runners take the fallback.
* ci(e2e): install kmod (depmod) before microVM image build
#305 made depmod a hard requirement in build-microvm-image.sh, but the
self-hosted ci-single/ci-multi runners don't ship the kmod package, so the
Build Install Artifacts step aborts with 'required tool not found: depmod'
before any E2E suite runs. Add an idempotent kmod install ahead of each
Build Install Artifacts step (single/multi/eks jobs); the gold-image builder
bakes kmod in too, so this no-ops once runners pick up the new image.
* ci: install kmod (depmod) in nightly + ddil too
e2e-nightly and e2e-ddil call the same build-install-artifacts.sh (-> make
build-microvm-image) on the same self-hosted ci-single/ci-multi runners as
e2e.yml, so they hit the identical 'depmod not found' abort. Add the same
idempotent kmod guard before Build Install Artifacts in both.
release distro builds run on GitHub-hosted ubuntu (kmod present); e2e-baremetal
runs on ci-baremetal which already has it (green today) — neither needs the
guard.
* test(eks): EKS control-plane e2e suite (Phase 1, API-level)
Add tests/e2e/eks/eks_test.go driving the EKS lifecycle against awsgw:
CreateCluster -> ACTIVE (endpoint + CA asserted), kubeconfig artifact
assembly, AccessEntry CRUD, get-token presign decode, DeleteCluster -> gone.
Harness: EKS SDK client on AWSClient + WaitForEKSCluster{Active,Deleted}
pollers. Wire e2e-eks into the Makefile and the single-node CI suite list.
* test(eks): Phase 2 kubectl get-nodes data-plane gate
Add the reachability shim (harness/eks_mgmt.go) that reads the control-plane
VM's br-mgmt IP from the cluster KV meta over NATS, plus a kubectl wrapper
(harness/kubectl.go) that skips when kubectl is absent. KubectlGetNodes points
kubectl at https://<mgmtIP>:6443 (insecure-skip-tls-verify; cert SANs the NLB
DNS, not the mgmt IP) with an aws eks get-token exec block and polls until the
control-plane node is Ready — the get-token webhook auth regression gate.
IRSA pod + the real NLB/Route53 passthrough are deferred to a follow-up.
* test(eks): tolerate doubled NotFound code + dial loopback NATS
DeleteCluster detection matched the SDK constant ResourceNotFoundException,
but awsgw emits ResourceNotFoundExceptionException on the wire — match the
NotFound code by substring so deletion is detected immediately instead of
timing out. The mgmt-IP shim dialed NATS.Host verbatim (0.0.0.0:4222), which
the serving cert does not SAN; rewrite the wildcard bind to loopback.
Validated live: all 6 subtests green, kubectl get nodes shows the
control-plane node Ready via the get-token webhook auth path.
* feat: build system images with libguestfs (no host block devices)
Replace the qemu-nbd + sudo mount + chroot + /dev-bind flow in
build-system-image.sh with libguestfs (virt-customize / virt-resize /
guestfish). Every operation runs in an isolated userspace appliance VM with
its own kernel, so the build touches no host block device, no host mount, and
needs no sudo — removing the failure class that wiped the host's /dev on shared
CI runners (a leaked /dev bind traversed by rm -rf).
Applies to all distros: alpine resizes via guestfish e2fsck+resize2fs, ubuntu
via virt-resize; customization (packages, binaries, files, setup script,
service enable, cache cleanup) is one ordered virt-customize invocation.
LIBGUESTFS_BACKEND=direct (no libvirt on runners).
Wire the eks-node AMI into the single-node e2e job: the runner builds the
device-free raw, ships it to the VM, and the VM imports it (predastore/NATS are
VM-only) so the eks suite's CreateCluster has an AMI to boot.
* feat(eks): bootstrap + state-report via AWS gateway broker
Converge the EKS control-plane → host channel onto the ELBv2 lb-agent model:
the K3s server VM publishes its bootstrap envelopes and periodic state reports
via SigV4-signed HTTPS POST to the AWS gateway, instead of dialing core NATS
directly. A new gateway action (PublishInternal, POST
/clusters/{name}/internal-publish) relays the payload verbatim onto the
existing eks.bus.* / eks.state.* subjects, so the daemon-side subscriber, KV
persistence/encryption, and reconciler are unchanged.
This removes the VM's need to reach NATS at all, so NATS can stay
cluster-internal behind its bind/firewall/mTLS controls — the prior direct
path broke when NATS was not VM-reachable (e2e CreateCluster timeout) and was
DDIL-fragile (fire-and-forget nats pub, no ack/retry).
- gateway/eks: PublishInternal broker + route; closed-set channel/kind validation
- cmd/eks-gateway-publish: one-shot SigV4 publisher (retry-until-ack) baked into
the eks-node AMI; replaces the nats-cli publisher in k3s-first-boot.sh and
mulga-eks-state-report.sh
- handlers/eks + daemon: hand the K3s server VM the mgmt-reachable gateway URL +
system (Predastore) SigV4 creds + gateway CA instead of NATS URL/token/CA;
new resolveSystemGatewayBaseURL (mgmt-bridge-first)
- eks-node image: build/install eks-gateway-publish, drop nats-cli download
* feat: migrate eks-token-webhook to the AWS gateway broker
The TokenReview webhook dialed core NATS directly (STS verify + AccessEntry
KV read). On a mgmt-only control-plane VM with NATS bound off-subnet that
fails, so the webhook exited at startup, the apiserver token-webhook
kubeconfig was never written, and the apiserver crash-looped — the cluster
never reached ACTIVE.
Converge it onto the same gateway-broker model as bootstrap/state-report and
ELBv2's lb-agent: the webhook SigV4-signs an HTTPS POST {accountId,token} to
/clusters/{name}/token-review, and the gateway runs the verify + lookup
host-side and returns the resolved identity. The VM now speaks no core NATS.
- internal/eksgw: shared on-VM SigV4 client (also adopted by
eks-gateway-publish).
- handlers/eks/token_review.go: Authenticate decision core (moved from the
webhook binary) + ResolveTokenReview host-side wiring.
- gateway/eks token-review route + WebhookTokenReview handler.
- webhook: drop NATS, relay via eksgw.Client; broker fault -> 503 (retryable)
not a silent deny.
- initd: export EKS_GATEWAY_* / EKS_ACCESS_* creds, not SPINIFEX_NATS_*.
* fix: build eks-node helpers with GOFIPS140; cover token-review broker
eks-gateway-publish blank-imports internal/fipsboot, whose init() panics
unless the binary is built with GOFIPS140=v1.0.0. The eks-node AMI build
omitted the flag, so the binary paniced at first boot right after k3s came
up — k3s-first-boot (set -eu) died before publishing the bootstrap token and
the cluster never reached ACTIVE. Build both eks-node helpers with
GOFIPS140=v1.0.0, matching cmd/lb-agent.
Add NATS-backed unit tests for ResolveTokenReview and the WebhookTokenReview
gateway handler (verify responder + account KV) to cover the new broker path.
* fix: export first-boot env so eks-gateway-publish child sees it
k3s-first-boot.sh and mulga-eks-state-report.sh sourced /etc/spinifex-eks/
first-boot.env but never exported the vars, so the eks-gateway-publish child
process inherited an empty environment and exited '--account-id is required'.
The control-plane VM came up healthy but never published its bootstrap
envelopes, so the cluster never reached ACTIVE. Wrap the source in set -a /
set +a so the KEY=value lines export to child processes.
* fix: make awsgw start --host override actually bind
The 'awsgw start --host' override printed 'Overwriting awsgw host' but the
assignment was commented out, so AWSGW always bound the spinifex.toml host
(a specific WAN IP, e.g. 192.168.0.10). A system instance (EKS K3s control
plane, internal LB agent) has a mgmt-only NIC on-link to the br-mgmt IP with
no route to the WAN IP, so its SigV4 POSTs to the gateway were refused and the
cluster never bootstrapped. Honour --host (e2e passes 0.0.0.0:9999) so AWSGW
listens on all interfaces incl. br-mgmt; the serving cert already SANs every
local interface IP, and WAN clients still reach it.
* test(eks): install kubectl on e2e VM so KubectlGetNodes runs
The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full
aws eks get-token -> kubectl -> apiserver -> token-webhook -> gateway broker
-> STS verify path — the only end-to-end exercise of the token-review broker.
The e2e binaries run on the VM, where kubectl was absent, so
harness.NewKubectl t.Skip'd it and the path went unvalidated. Install a
checksum-verified kubectl (pinned to the k3s server version) on the VM,
gated on e2e-eks.
A killed ubuntu system-image build can leave `mount --bind /dev` under
its build dir. The e2e Clean Workspace step then ran
`sudo rm -rf "$GITHUB_WORKSPACE"/*` as root, traversed the leaked bind,
and deleted the host's /dev (incl. /dev/null), wedging the runner.
- e2e.yml: umount anything under the workspace (deepest-first) before
deleting, and pass `rm --one-file-system` so rm can never cross a mount
boundary into host /dev. Hard backstop, applied to all 3 Clean Workspace
steps.
- build-system-image.sh: recursive lazy unmount in cleanup() detaches the
chroot binds even when busy, and mark the build subtree rprivate so the
/dev bind can't propagate into the host namespace.
NLB target health stalls at 0/N in CI while the data plane works locally.
The lb-agent's nginx engine activation, any nginx -t/reload error, and its
active-probe results are logged to the LB microVM serial console — which the
host journal never captures and the GetConsoleOutput API path reports as
NotFound, so the failure is undiagnosable from CI artifacts.
- e2e.yml: pull /run/spinifex/console-*.log off each node in the single- and
multi-node collect steps (files persist past VM teardown) into lb-console/.
- lbagent healthcheck: log probe failures at Warn (was Debug) so unreachable
targets surface at the agent's default level on the captured console.
Two bugs in afd63ed9:
1. `sudo mkdir` makes /tmp/spinifex-diag root-owned 0755. The
subsequent `> /tmp/spinifex-diag/tcpdump.log` is performed by
the calling tf-user shell BEFORE sudo gets the command, failing
with Permission denied. action shell runs with `set -e` so the
step aborts.
2. `sudo nohup ... &; echo $!` captures the wrong PID — `$!` is the
backgrounded sudo, which may exit when its target dies, leaving
stale tcpdump.pid.
Fixes:
- chmod 0777 on /tmp/spinifex-diag after creating it (next steps'
redirects from tf-user shell now work).
- Wrap the tcpdump launch in `sudo bash -c '...'` so backgrounding
and `$!` happen inside the sudo'd shell — `$!` is the actual
tcpdump PID (nohup execs, sudo bash holds it). Captures stdin
to /dev/null, stdout/stderr to /tmp/spinifex-diag/tcpdump.log.
- sleep 2 (was 1) — gives tcpdump time to fully init pcap header.
- Hard fail (exit 1) if PID is missing or process dead, so the
step's failure is visible instead of a silent green.
bash -n verified.
Mulga-siv-167 Phase 1. Adds evidence-capture for the LB
single-host return-path drop class:
e2e_single only:
- New step 'Start Packet Capture' between Bootstrap and Run E2E
Tests. SSHes to test VM and starts background tcpdump on br-wan
capturing TCP/22/80/443/9999 + ARP into /tmp/spinifex-diag/br-wan.pcap.
PID stored at /tmp/spinifex-diag/tcpdump.pid.
All three jobs (single, pseudo-multi, real-multi):
- Collect Diagnostics: kill tcpdump (single only — PID file
guarded) -INT for graceful flush, falls through to -KILL after 5s.
- Adds journal-spinifex-full.log (no -n truncation) alongside
the existing -n 10000 capped one.
- Adds bridge fdb show, bridge link, and hairpin_mode/learning/flood
for every br-wan brif — needed to confirm/disconfirm the
hairpin-loss hypothesis in the plan doc.
- Adds 'chmod -R a+r /tmp/spinifex-diag' before tar to ensure pcap
(created by sudo tcpdump as root) is readable by tf-user.
bash -n verified on all four rendered scripts. YAML parses.
Zero new gold-image deps (tcpdump, bridge present in base image).
Adds a Collect Diagnostics + Upload Diagnostics step pair to all three
E2E jobs (single, pseudo-multi, real-multi). Runs before Teardown,
if: always() so green runs produce baselines too. SSHes to test
VM(s), dumps:
- journalctl for spinifex-*, ovn-*, openvswitch-*
- ovs-vsctl/ofctl/appctl: bridge state, br-int + br-ex flows,
dpctl flow + conntrack tables
- ovn-nbctl/sbctl: show, Port_Binding, Chassis, NAT, Logical_Router
- ovn-appctl ct-zone-list (LR -> ct_zone mapping)
- ip addr/route/neigh, iptables-save, nft list, dmesg, virsh list
Tarred, scp'd back, uploaded as e2e-diag-{single,pseudo,multi}-<run>
artifact (14-day retention).
OVS-native ovs-appctl dpctl/dump-conntrack — no new gold-image deps.
Multi-node job writes script to RUNNER_TEMP then ssh-stdin's it
per-node (avoids YAML/heredoc indent traps inside for-loop). bash -n
verified on all three rendered scripts.
Unblocks siv-111 debug loop: bash LB driver does not emit an
analysis bundle on failure, so post-W2 recurrences have been blind.
Workflow-level capture is hypothesis-free.
- workflows: add 'dev' to push/PR triggers (spinifex.yml, ui.yml, e2e.yml)
- workflows: drop "Resolve Mulga Branch" fallback step; mulga is single-branch
so checkout@v6 uses the default branch implicitly
- dependabot: route gomod/github-actions/npm PRs to dev
- diff-coverage.sh: pick origin/dev as base for feat branches, origin/main for dev
- iac/proxmox-template: default git_branch to dev for cluster builds
Refs docs/development/improvements/reintroduce-dev-branch.md
* feat: add ALB target health reporting via HAProxy stats
Agent-side: periodically query HAProxy stats socket and publish
target health status to NATS (elbv2.alb.<id>.health topic).
Control-plane: healthChecker subscribes to health reports from all
ALB agents and updates target health state in the KV store. Wire
into ELBv2ServiceImpl lifecycle and DeregisterTargets cleanup.
E2E: remove health check skip — targets must now become healthy.
* feat: allocate public IP for internet-facing ALB VMs in daemon
Add Scheme field to SystemInstanceInput so the daemon knows whether
an ALB is internet-facing or internal. For internet-facing, allocate
a public IP via externalIPAM and publish vpc.add-nat for the OVN
dnat_and_snat rule. Internal ALBs skip public IP assignment entirely.
* fix: release auto-assigned public IPs on DeleteNetworkInterface
DeleteNetworkInterface previously leaked public IPs and NAT rules.
Now checks if the ENI has a public IP, verifies it's not EIP-owned,
and releases via externalIPAM + publishes vpc.delete-nat. Daemon
injects externalIPAM and EIP KV into VPC service at init time.
Tests cover: auto-assigned IP release with NAT event, EIP-owned IP
skipped, and no-op when externalIPAM is not configured.
* feat: enforce multi-AZ for internet-facing ALBs in pool mode (mulga-770)
In CreateLoadBalancer, when DevNetworking is false (pool mode), require
2+ subnets in different AZs for internet-facing scheme. Dev mode and
internal scheme skip the validation.
* feat: ALB scheme unit tests + DNS internal prefix + public IP in AZ response (mulga-771)
Add 9 unit tests covering scheme-dependent ALB behavior:
- DNS name uses internal- prefix for internal scheme (AWS-compatible)
- Internet-facing ALBs include public IP in AvailabilityZones response
- Internal ALBs have no public IP in AZ response
- Launcher receives correct scheme, delete cleans up VM + ENIs
Also implements the code changes these tests exercise:
- AvailZoneInfo.PublicIP field populated from launcher output
- lbRecordToSDK populates LoadBalancerAddresses for internet-facing
* feat: add custom AMI for ALB to images list
* feat: add internet-facing ALB data plane E2E test (mulga-772)
New E2E script that validates internet-facing ALBs get public IPs and are
reachable from a peer cluster node (external validation). Skips in dev mode.
* fix: set ENDPOINT to runner node IP in ALB E2E workflow
* fix: correct paths filter to reference alb-e2e.yml
* fix: alb ami name and workflow step to import the image
* fix: spx path
* fix: remove ALB AMI import from workflow and added to bootstrap.sh
* fix: use two subnets in different AZs for internet-facing ALB E2E
Multi-AZ enforcement (mulga-770) requires internet-facing ALBs to have
subnets in 2+ AZs in pool mode. Discover AZs and create a subnet in each.
* refactor: remove multi-AZ enforcement for internet-facing ALBs
Team decision: ALBs must work with single-AZ and single-node clusters.
Removes the multi-AZ validation, pool mode helper, associated tests,
and error constant. E2E script reverted to single subnet.
* fix: use correct account ID for ALB ENI public IP assignment
The ENI is created under the caller's account but LaunchSystemInstance
was looking it up with GlobalAccountID (system account), causing the
UpdateENIPublicIP call to fail with "ENI not found". Pass the caller's
account ID via SystemInstanceInput.AccountID.
* fix: add debug logging for ALB public IP assignment failure
* fix: expand debug logging to include peer node and more keywords
* fix: lazy system AMI discovery for ALB VMs
The daemon was caching the system AMI at startup, but the ALB image is
imported after services start (bootstrap step 9). Replace SetSystemAMI
with SetSystemAMIFunc so the AMI is resolved at request time when
CreateLoadBalancer actually needs it.
* fix: resolve ALB VM instance type dynamically from resource manager
The instance type was hardcoded to t3.nano which may not exist on all
clusters. Now the daemon provides a resolver that picks the smallest
available instance type from the resource manager.
* push interim changes
* refactor: rewrite ALB agent from NATS client to HTTP server
Replace NATS subscriptions with an HTTP server on :8405 exposing
POST /config, GET /ping, and GET /health. The daemon will push
config and poll health over HTTP through the VPC network, removing
the agent's direct NATS dependency.
- agent.go: net/http server replaces nats.Conn/Subscriptions
- health.go: remove push-based reportHealth; /health handler
calls queryHAProxyStats synchronously on each request
- cmd/alb-agent: --nats flag replaced with --listen
- tests: rewritten with httptest instead of NATS test server
- health_checker_test: inline topic strings (HealthTopic removed)
* refactor: replace NATS with HTTP client for daemon-to-ALB agent comms
Daemon now pushes config via POST and polls ping via GET to the ALB
agent's HTTP server on the VPC IP, eliminating the agent's NATS
dependency from the daemon side.
- service_impl: pushConfig uses http.Post, waitForAgentReady uses
http.Get, both target http://<vpcIP>:8405
- albVMUserData: drop natsURL param (agent no longer needs NATS)
- types: add VPCIP field to LoadBalancerRecord
- daemon: remove SetNATSURL call and 10.0.2.2 translation block
- arn_test: update user data tests for new signature
* refactor: rewrite ALB health checker from NATS subscription to HTTP polling
Health checker now polls each active ALB's GET /health endpoint on a
10s ticker instead of subscribing to a NATS wildcard topic. Removes
the NATS connection dependency from the health checker.
- health_checker: replace nc/sub with httpClient + stopCh polling loop
- newHealthChecker: drop nc param, add agentURLFn for testability
- service_impl: update newHealthChecker call site
- tests: rewrite with httptest, test pollAll/pollOne paths
* refactor: remove NATS wiring from ALB image and make hostfwd dev-only
- alb-setup.sh: remove --nats flag, use --listen=:8405; remove
placeholder HAProxy frontend on 8405 (agent owns the port now);
update description and comments
- service_impl: HostfwdPorts (80/443) now only set when DevNetworking
is enabled — production ALBs are reached via VPC IP directly
* feat: system credentials file + daemon wiring for ALB agent SigV4 auth
Write config/system-credentials.json during admin init/join with the
system access key. Daemon reads it at startup, resolves the gateway URL
from external IPAM (or 10.0.2.2 in dev mode), and wires both into the
ELBv2 service. albVMUserData now injects ALB_GATEWAY_URL, ALB_ACCESS_KEY,
and ALB_SECRET_KEY into cloud-init for ALB VMs.
* feat: ALB agent gateway actions + daemon handlers (heartbeat, config)
Add ALBAgentHeartbeat and GetALBConfig actions to the gateway so the ALB
agent can communicate via the AWS gateway instead of direct HTTP.
- Add agent_types.go with input/output types for both actions
- Add ALBAgentHeartbeat: processes heartbeat, transitions provisioning→active
on first contact, updates LastHeartbeat, processes health report
- Add GetALBConfig: returns pre-computed HAProxy config text and hash
- Replace pushConfig (HTTP POST to agent) with updateStoredConfig (stores
config + SHA256 hash on LB record for agent pull)
- Remove waitForAgentReady polling goroutine (heartbeat handles transition)
- Remove HTTP client from ELBv2ServiceImpl (no longer needed)
- Strip health checker HTTP polling (pollLoop/pollAll/pollOne) — health
reports now arrive directly via ALBAgentHeartbeat
- Add ConfigText, ConfigHash, LastHeartbeat fields to LoadBalancerRecord
- Wire NATS subscriptions + daemon handlers for both new actions
- Add gateway handler files and elbv2Actions map entries
- Comprehensive tests for heartbeat, config, and updateStoredConfig
* feat: rewrite ALB agent as AWS SDK HTTP client (Bead 3r)
Replace the HTTP server agent with a poll-based client that communicates
with the gateway using SigV4-signed requests. The agent now sends
ALBAgentHeartbeat every 5s with health reports and fetches config via
GetALBConfig when the config hash changes.
* feat: add VPC service route for ALB agent gateway access
ALB VMs need to reach the gateway for heartbeat/config polling, but
VPCs without an IGW have no external routing. Instead of requiring
an IGW, the daemon now publishes a vpc.ensure-service-route event
before launching ALB VMs. vpcd handles this by creating minimal
external connectivity (external switch, localnet port, gateway router
port, chassis bindings) if not already present, then adds a /32 static
route for the gateway IP.
Refactors handleIGWAttach to share the external connectivity setup via
ensureExternalConnectivity helper.
* Revert "feat: add VPC service route for ALB agent gateway access"
This reverts commit d4220d8806.
* feat: add management NIC infrastructure for system instance control plane
Add config, IP allocator, TAP helpers, and VM struct fields for the
management NIC that gives system instances (ALBs etc.) L3 access to the
host control plane via br-mgmt, independent of VPC routing.
- DaemonConfig.MgmtBridge (default "br-mgmt")
- MgmtIPAllocator: thread-safe static IP allocation (.10-.249), rebuild from VMs on restart
- generateMgmtMAC (02:a0:00 prefix), MgmtTapName, GetBridgeIPv4
- SetupMgmtTapDevice / CleanupMgmtTapDevice (plain Linux bridge TAP)
- VM struct: MgmtMAC, MgmtIP, MgmtTap fields
* feat: wire management NIC into QEMU launch and cloud-init config
Add QEMU NIC injection for mgmt TAP (virtio-net-pci on br-mgmt),
extend generateNetworkConfig to emit mgmt0 stanza with static IP,
and wire mgmt MAC/IP allocation into LaunchSystemInstance. Includes
mgmt TAP cleanup on instance stop and failed launch cleanup.
* test: add coverage for ELBv2 service lifecycle and setter methods
* fix: move br-mgmt detection before wireALBAgentConfig so gateway URL uses management bridge IP
* fix: generate per-interface cloud-init config when eniMAC present without devMAC
The guard in generateNetworkConfig required both eniMAC and devMAC to
produce per-interface config, falling back to wildcard DHCP otherwise.
System instances (ALBs) have eniMAC + mgmtMAC but no devMAC (DevNetworking
is independent), so the mgmt NIC stanza was never emitted. The wildcard
config does DHCP on all NICs, but br-mgmt has no DHCP server, leaving the
management NIC unconfigured and the agent unable to reach the gateway.
* fix: add wildcard support to ENI description filter + add ca-certificates to ALB image
- ENI DescribeNetworkInterfaces description filter did exact string match,
but AWS supports * wildcards. E2E test uses "ELB *" which didn't match
"ELB app/my-alb/lb-123". Added matchFilter helper for wildcard matching.
- ALB image was missing ca-certificates package, so cloud-init ca_certs
module couldn't add the Spinifex CA to the system trust store. Agent
TLS handshakes failed with "certificate signed by unknown authority".
- Fixed data race in TestDeleteLoadBalancer_TerminatesVM_WithPublicIP
where async TerminateSystemInstance goroutine wrote to mock fields
while the test read them.
* fix: detach ENI before delete on termination + defer ENI cleanup to stopInstance
DeleteLoadBalancer was trying to delete ENIs synchronously, but they were
still attached (in-use) to the running VM. The delete was rejected with
InvalidNetworkInterface.InUse.
Fix: DeleteLoadBalancer only deletes ENIs when no VM was launched (InstanceID
empty). When a VM exists, ENI cleanup is deferred to stopInstance which runs
after QEMU exits. stopInstance now calls DetachENI before DeleteNetworkInterface
to clear the in-use status.
E2E test now polls for ENI cleanup (up to 30s) since VM termination is async.
* fix: detach then delete ENIs in DeleteLoadBalancer
ENIs are attached (in-use) to the ALB VM, so DeleteNetworkInterface
rejects them. Fix: call DetachENI first to clear in-use status, then
delete. stopInstance tolerates already-deleted ENIs.
* test: add internal ALB dataplane E2E test
Add run-elbv2-dataplane-internal-e2e.sh replacing the generic dataplane
test. Internet-facing coverage is already provided by
run-elbv2-dataplane-internet-facing-e2e.sh.
The internal test validates:
- ALB reaches active via mgmt NIC heartbeat (no IGW dependency)
- Scheme=internal, no public IP, internal- DNS prefix
- Health checks pass via VPC L2 (OVN logical switch)
- Round-robin traffic from a VPC client to the ALB's private IP
- ENI cleanup on ALB deletion
Uses a client VM inside the VPC that curls the ALB private IP and
serves results via HTTP on its public IP for host-side verification.
* feat: add NLB constants and data model changes (mulga-838)
* feat: type-aware validation in CreateLoadBalancer (mulga-839)
Accept Type=network for NLB, reject SecurityGroups for NLB, set
CrossZoneEnabled defaults (false for NLB, true for ALB), and use
/net/ ARN path segment for network load balancers.
* feat: type-aware validation in CreateTargetGroup (mulga-840)
Accept TCP/UDP/TLS/TCP_UDP protocols and apply NLB health check
defaults (TCP protocol, no path/matcher, 10s timeout, 3/3 thresholds).
Reject invalid protocols with InvalidParameterValue.
* feat: type-aware validation in CreateListener (mulga-841)
* feat: add NLB listener and protocol compatibility unit tests
Cover the remaining Phase 1 test gaps: NLB CreateListener with all L4
protocols, protocol rejection for wrong LB type, listener-to-target-group
protocol compatibility validation, and default protocol rejection on NLB.
* feat: add TCP HAProxy config generation for NLB
Add TCP mode template and generateNLBConfig function for Network Load
Balancer support. GenerateHAProxyConfig now branches on LB type,
producing mode-tcp configs for NLBs with tcp-check or httpchk health
checks depending on the target group's health check protocol.
* refactor: rename albagent to lbagent for NLB generalization
Rename the ALB-specific agent package and binary to generic LB names
in preparation for NLB support. Both ALB and NLB VMs use the same
agent — it just applies whatever HAProxy config the control plane
generates.
- Rename albagent/ → lbagent/, cmd/alb-agent/ → cmd/lb-agent/
- Rename types: ALBAgentHeartbeat* → LBAgentHeartbeat*, GetALBConfig* → GetLBConfig*
- Rename NATS topics: elbv2.ALBAgentHeartbeat → elbv2.LBAgentHeartbeat
- Rename cloud-init user data: albVMUserData → lbVMUserData with LB_* env vars
- Rename Makefile targets: build-alb-agent → build-lb-agent
- Update gateway action map, daemon subscriptions, service interface
- Fix pre-existing unused matchFilter function lint error
- Fix awserrors test count for InvalidConfigurationRequest
* feat: add sys.micro system instance type for LB VMs (mulga-845)
Add sys.micro (1 vCPU, 128MB) as an internal-only instance type for
system VMs (load balancers, NAT gateways). Registered in the type map
for allocation but excluded from DescribeInstanceTypes, NATS
subscriptions, and resource stats. Daemon SetSystemInstanceTypeFunc
now returns sys.micro instead of scanning for smallest guest type.
* feat: LB image shrink and rename (mulga-846)
Rename alb.conf → lb.conf, alb-setup.sh → lb-setup.sh with IMAGE_SIZE
reduced from 512M to 192M. Update all references to use lb-agent binary
and LB_* env vars. Update COMMANDS.md for NLB support.
* chore: rename file
* feat: NLB E2E test and bug fixes (mulga-847)
Add full NLB data plane E2E test (33 assertions, 9 phases):
VPC/subnet setup, TCP target group with health checks, NLB
creation (type=network), TCP listener, agent heartbeat,
target health verification, TCP traffic round-robin, target
deregistration, and full cleanup verification.
Bug fixes found during E2E testing:
- Fix HAProxy stats socket path mismatch: config used alb-/nlb-
prefix but agent expects lb- prefix, preventing health reports
- Fix system AMI lookup to match new -lb- image name, falling
back to legacy -alb- name
- Bump LB image size from 192M to 256M (Alpine base is 200M)
* test: validate NLB internal scheme (mulga-857)
NLB with scheme=internal works out of the box — shared code paths
with ALB handle it correctly. Add unit test and E2E coverage:
- Unit test: NLB internal creates with no public IP, internal- DNS prefix
- E2E phases 10-14: internal NLB lifecycle with VPC client TCP traffic
- E2E passed 56/56 with perfect round-robin (20/20 requests)
* feat: consolidate LB e2e tests into run-lb-e2e.sh (mulga-854)
Replace 3 separate LB dataplane scripts with a single consolidated test
that covers all 4 variants (ALB/NLB × internet-facing/internal).
- Share one VPC, subnet, and 2 dual-purpose app instances across all suites
- ALB + NLB provision in parallel within each phase (internet-facing / internal)
- One client VM tests both internal LBs simultaneously
- Replace sleep 100 with LB health check gates
- Delete standalone alb-e2e.yml, integrate into e2e.yml for both single and multi-node
* chore: disable LB E2E tests in CI until dev env issue resolved
* fix: read LB agent credentials from spinifex.toml instead of system-credentials.json
The daemon couldn't read system-credentials.json due to systemd
hardening (file owned by root:root 0600, not in ReadOnlyPaths).
Use the same system credentials already loaded from spinifex.toml
via the predastore config section — these are identical keys written
by admin init.
* feat: automate LB image build and import (mulga-861)
- dev-install.sh: build lb-agent and run build-system-image.sh --import
after services start so the LB image is available out of the box
- build-system-image.sh: add --quiet flag to suppress build output,
remove hardcoded --config path (uses auto-detected default)
- images.go: rename alb-alpine → lb-alpine to match current naming
* update go version
* add LB e2e tests
* fix: clean LB image build dir and wait for nbd device readiness
Previous CI runs can leave stale state in /tmp/lb-image-build/ causing
e2fsck failures. Clean the build dir before each CI build. Also add a
readiness check after qemu-nbd connect to avoid race conditions.
* fix: serialize nbd access with flock for concurrent CI builds
Single-node and multi-node CI jobs share the same host and both build
the LB system image using /dev/nbd0. Use flock to serialize the nbd
connect/mount/customize/disconnect section so concurrent builds wait
rather than collide.
* fix: move flock to cover entire image build, not just nbd section
Concurrent CI jobs share BUILD_DIR and /dev/nbd0. The lock must be
acquired before downloading/copying the image, not just before nbd
connect, otherwise the second job can clobber the first job's files.
* fix: skip LB image build if output is fresh from concurrent CI job
When concurrent CI jobs build the same image, the second job now reuses
the first's output if it's less than 10 minutes old, avoiding a
duplicate download + build cycle.
* fix: filter AMI discovery by name in E2E tests
With the LB Alpine image now imported alongside Ubuntu, the unfiltered
describe-images call picked up the wrong AMI. Filter by the expected
image name so tests use the Ubuntu image for instance lifecycle.
* fix: filter AMI discovery by name in multi-node E2E tests
Same fix as run-e2e.sh — with the LB Alpine image now imported,
the unfiltered describe-images picks the wrong AMI.
* fix: use awsgw bind IP as LB agent gateway URL fallback
When br-mgmt is not configured and dev_networking is off (production
with external pool), the gateway URL was falling through to the
external IPAM pool's GatewayIP — the WAN router, not the Spinifex
node. LB VMs couldn't reach the awsgw, so heartbeats never arrived
and ALBs stayed stuck in provisioning.
Use the awsgw bind IP instead — LB VMs on the OVN overlay can reach
the host node's WAN IP where the gateway listens.
* fix: capture StateReason and daemon logs when LB app instances fail
Instances with user-data terminate immediately in CI. Add
describe-instances StateReason query and daemon log dump to diagnose
whether it's volume_preparation_failed or launch_failed.
* fix: prevent orphaned AMI from breaking LB E2E and avoid stuck provisioning
Three fixes for CI E2E failures:
1. run-e2e.sh: Remove custom AMI from predastore after deleting its
backing snapshot. Previously the AMI remained discoverable but
unusable, causing run-lb-e2e.sh to pick it and fail with NoSuchKey
on the deleted snapshot.
2. run-lb-e2e.sh: Prefer ubuntu AMI by name for app instances instead
of picking any non-alpine image. Also improve debug log dump to use
journalctl (binary-install CI uses systemd, not log files).
3. CreateLoadBalancer: Only set provisioning state when a system VM
was actually launched. Previously a failed launch still set
provisioning, leaving the LB stuck forever with no agent to
send heartbeats.
* fix: use AWSGW bind IP as LB agent gateway URL in multi-node
In multi-node deployments, the AWSGW binds to the WAN IP (e.g.
192.168.1.26:9999), but wireLBAgentConfig was using the br-mgmt IP as
the gateway URL. Since the AWSGW doesn't listen on br-mgmt, the LB
agent's heartbeat was refused, leaving internet-facing ALBs stuck in
provisioning.
Internet-facing LBs now use the AWSGW bind IP directly — the VM reaches
it via VPC/external networking. Internal LBs get a host route on the
mgmt NIC so they can reach the AWSGW bind IP through br-mgmt.
* fix: add mgmt route via bootcmd for multi-node internal LBs
Alpine cloud-init does not support v2 routes under ethernets — adding
a routes: block to the network-config breaks all networking in the VM.
Instead, add the host route via a bootcmd in the LB user data. This
runs early in cloud-init and adds `ip route add {awsgw}/32 via
{br-mgmt}` so internal LB agents can reach the AWSGW through the
management NIC when the gateway binds to a specific WAN IP.
Also moves the networking service from boot to default runlevel in the
LB Alpine image to prevent a race with cloud-init.
Verified locally: internal ALB transitions to active with AWSGW bound
to WAN IP (simulating multi-node).
* add LB enable flag to e2e
* test: add elbv2 tests to meet diff coverage threshold
Cover health checker edge cases (invalid JSON, empty servers, zero
thresholds, port-0 fallback) and service_impl paths (mgmt route
cloud-init, protocol validation, missing target group).
* fix: swallowed health report error + needless marshal round-trip in LBAgentHeartbeat
Add handleHealthReportDirect(report) to healthChecker so LBAgentHeartbeat
can pass the struct directly instead of marshaling to JSON and immediately
unmarshaling. Promote handleHealthReport unmarshal error from Debug to Warn
so failures are visible in logs.
* fix: validate credentials before launching LB VM + set failed state on launch error (mulga-867)
Guard lbVMUserData with empty-value check for gatewayURL, systemAccessKey,
and systemSecretKey — blank env vars previously caused agent auth-loop with
no daemon-side log. Set LB state to failed (not active) when VM launch fails
so the API reflects the broken data-plane instead of black-holing traffic.
* fix: data race in getSystemAMI/getSystemInstanceType lazy caching
Use sync.Once to guard lazy resolution of systemAMI and
systemInstanceType fields. Concurrent CreateLoadBalancer calls
previously raced on unsynchronized read+write of these cached values.
The setter methods reset the Once so re-resolution works when the
func is set after initial construction.
Also fix pre-existing nosprintfhostport lint in daemon.go.
* fix: use utils.PublishEvent for NAT events + return errors from isEIPOwned
publishNATEvent in eni.go and daemon was copy-pasted boilerplate — now
delegates to utils.PublishEvent (matching the EIP service pattern).
isEIPOwned previously swallowed all KV errors and returned false, which
could release an EIP-owned public IP if the store was temporarily
unavailable. Now returns (bool, error) so the caller skips release on
KV failure instead of assuming no EIP ownership.
* fix: narrow health checker to only scan target groups for reporting LB
handleHealthReportDirect was calling ListTargetGroups() on every heartbeat
(every 5s per LB agent), scanning all TGs. Now uses report.LBID to follow
LB → listeners → TG ARNs and fetch only the attached target groups via
direct key lookups. Falls back to full scan if LBID is missing.
* fix: pass region through to LB agent SigV4 signing (mulga-871)
LB agent hardcoded us-east-1 as the SigV4 signing region. Thread region
from ELBv2 service config → cloud-init LB_REGION env var → lb-agent
--region flag → lbagent.New() → signer.Sign() call.
* fix: add account-level authorization to LBAgentHeartbeat and GetLBConfig
Both endpoints accepted any accountID without verifying the LB belongs
to that account. Now verify accountID matches the LB's AccountID,
with an exception for the system account (000000000000) which the
ALB agent uses for heartbeats and config fetches.
* fix: upgrade silent failure paths from Debug to Warn/Error
- healthChecker target group list failure: Debug → Warn
- HAProxy stats socket error in agent tick(): Debug → Warn
- ENI delete EIP-owned skip: Debug → Info (operational visibility)
- wireLBAgentConfig: warn when no gateway host is resolved
- detectMgmtIP: warn when bridge exists but IP is empty
- stale mgmt TAP cleanup: log errors instead of discarding
* refactor: deduplicate MAC hash function into utils.HashMAC
Three identical hash-to-MAC implementations (generateDevMAC, generateMgmtMAC,
generateENIMac) now delegate to a shared utils.HashMAC(prefix, id) helper.
* fix: guard LBAgentHeartbeat KV writes to state transitions or stale timestamps
Previously the full LoadBalancerRecord (including ConfigText) was written
to JetStream KV on every heartbeat (every 5s). Now only persists on state
transitions (provisioning→active) or when LastHeartbeat is older than 60s.
* fix: fail ALB launch when public IP allocation fails for internet-facing scheme
Previously, IPAM allocation failure was logged as a warning and the ALB
VM continued launching without a public IP. The ALB reached active state
but was unreachable from the internet. Now the launch returns an error
and cleans up the partially-created instance, causing the ALB to enter
the failed state.
* fix: retry system AMI resolution when image not yet imported
sync.Once cached empty results permanently — if the daemon started
before the system image was imported, all CreateLoadBalancer calls
silently skipped VM creation. Replace sync.Once with a mutex that
only caches non-empty results, allowing retry on subsequent calls.
Same fix applied to getSystemInstanceType for consistency.
* fix: propagate updateStoredConfig errors to API callers (mulga-884)
updateStoredConfig and updateStoredConfigForTargetGroup now return
errors instead of silently swallowing them. Callers (CreateListener,
DeleteListener, RegisterTargets, DeregisterTargets) return
ErrorServerInternal when config persistence fails, preventing the API
from reporting success while HAProxy serves stale configuration.
* test: add unit tests for wireLBAgentConfig, isEIPOwned, toHealthReport (mulga-885)
* fix: parse HAProxy templates once at package level (mulga-875)
generateALBConfig and generateNLBConfig were calling template.New().Parse()
on every invocation. Move to package-level template.Must for one-time parsing.
* fix: fail launch for internal-scheme ALBs when mgmt NIC setup fails (mulga-886)
When management TAP or IP allocation fails, internal-scheme ALBs now
return an error instead of silently continuing without a management NIC.
Without the mgmt NIC, the LB agent cannot reach the gateway and the ALB
gets stuck in provisioning forever. Internet-facing ALBs retain the
existing degrade-and-continue behavior.
* refactor: extract lbArnPathSegment helper for ALB/NLB ARN construction (mulga-877)
The app/net path segment derivation was duplicated in buildLBArn,
buildListenerArn, and inline in CreateLoadBalancer. Extracted into a
single lbArnPathSegment helper.
* refactor: use aws.StringValue in toHealthReport (mulga-878)
Replace manual nil-check-then-deref with aws.StringValue for cleaner
pointer-to-value conversion in the heartbeat health report builder.
* refactor: use testutil.StartTestJetStream in health_checker_test.go (mulga-876)
Replace hand-rolled setupTestNATS (~20 lines) with the shared
testutil.StartTestJetStream helper. Simplify return to just *Store
since no caller needed the *nats.Conn.
* fix: address PR #122 review comments (mulga-887)
- Extract NATEvent named type from anonymous struct in publishNATEvent
- Remove unused build-lb-image.sh thin wrapper
- Remove SPX_ENABLE_LB env var from e2e workflow (unused)
- Remove no-op healthChecker start/stop methods
- Export simple ELBv2 fields, remove trivial setter methods
- Delete old e2e scripts replaced by consolidated run-lb-e2e.sh
* fix: check specific error type for ENI deletion on termination (mulga-888)
ENI deletion errors were all silently downgraded to Debug level,
masking genuine failures (KV issues, permissions, ENI stuck in-use)
that cause orphaned ENIs and leaked IPAM addresses. Now only
NotFound is logged at Debug; all other errors log at Error.
* fix: propagate non-ENODEV errors from GetBridgeIPv4 (mulga-890)
GetBridgeIPv4 returned ("", nil) for ALL InterfaceByName failures,
not just missing devices. Permission errors on /sys/class/net/ were
silently treated as "bridge absent". Now only "no such network
interface" returns nil; other errors propagate to the caller.
* fix: remove backend address from proxy error response (mulga-892)
The reverse proxy error handler included the internal backend
host:port in the XML error body sent to clients. The address is
already logged server-side; the client now sees a generic message.
* fix: release mutex before KV store I/O in health checker (mulga-893)
handleHealthReportDirect held hc.mu across PutTargetGroup calls
(JetStream KV Put — network I/O), serializing concurrent heartbeats
from independent LBs. Now collects changed TGs under the lock and
persists them after releasing it.
* fix: warn when system credentials missing in wireLBAgentConfig (mulga-894)
fix: log DetachENI errors instead of discarding with blank identifier (mulga-895)
* fix: detect veth bridge mode and use centralized NAT for it
The veth-pair bridge topology (Linux bridge ↔ veth ↔ OVS bridge) was
misclassified as direct bridge mode, causing OVN to use distributed NAT.
Distributed NAT hairpin routing fails through the Linux bridge intermediary,
breaking host-to-local-LB connectivity.
Add BridgeModeVeth constant, detect veth-wan-ovs interface in
detectBridgeMode(), and introduce useCentralizedNAT() to gate NAT mode
and nat-addresses=router for both macvlan and veth modes. isMacvlanMode()
is preserved for macvlan-specific behavior (MAC alignment).
* fix: add retry loop for peer NLB reachability test
With centralized NAT, ARP resolution for NAT external IPs relies on
gratuitous ARPs from the gateway chassis. These take time to propagate
to peer nodes. The peer→NLB test had a single 3s attempt while all
other connectivity tests use retry loops.
* fix: use request-reply for vpc.add-nat to prevent ARP propagation races
publishNATEvent was fire-and-forget for vpc.add-nat, meaning the daemon
continued before vpcd committed the OVN NAT rule. This caused intermittent
NLB connectivity failures: the LB was reported active before OVN had
the DNAT rule and sent GARPs for the public IP.
Switch vpc.add-nat to request-reply (RequestEvent) so the daemon waits
for OVN confirmation. Also increase nc timeout from 2-3s to 5s in E2E
tests for more reliable TCP probing during ARP resolution.
* fix: clean up stale NAT rules before adding new ones for reused IPs
When a public IP is returned to the external pool and quickly reallocated
(e.g. instance terminated, then LB created), the fire-and-forget
vpc.delete-nat may not have been processed yet. handleAddNAT now removes
any existing dnat_and_snat rule for the same external IP before creating
the new one, preventing duplicate/conflicting OVN NAT rules.
This was causing intermittent NLB connectivity failures in multi-node E2E
where NLB public IPs collided with recently-freed instance IPs.
* fix: clean up stale NAT rules across all VPC routers on IP reuse
The previous fix (0a93081) only removed stale dnat_and_snat rules from
the target VPC's router. When a public IP is freed from one VPC and
reallocated to a different VPC before the fire-and-forget vpc.delete-nat
is processed, the stale rule on the old VPC's router caused duplicate
OVN ARP responses — traffic was black-holed when the host resolved to
the wrong router's MAC.
DeleteAllNATsByExternalIP now searches every logical router for stale
NAT rules matching the external IP and removes them in one transaction.
Also adds host connectivity diagnostics (ARP, routing, OVN NAT state)
to run-lb-e2e.sh so the next CI failure surfaces the root cause of the
persistent NLB-from-host issue.
* fix: run LB inet tests from actual hosting node, not hardcoded node1
The host/peer connectivity tests were hardcoded: "host" = node1 (where
the test script runs), "peer" = node2. But LB VMs land on whichever
node picks up the NATS queue message, so the tests were randomly testing
local vs remote connectivity depending on placement luck.
Now the test:
1. Accepts --nodes <ip1> <ip2> ... (all cluster node IPs)
2. After LB creation, discovers which node hosts each LB VM by checking
for the ENI's tap device
3. Runs "local" tests from the hosting node and "remote" tests from a
different node
Also updates e2e.yml to pass --nodes instead of --peer.
Auto-discover all node IPs (WAN, mgmt, VPC) and hostname for server
cert SANs — fixes TLS failures when accessing Spinifex via non-localhost
IPs. Adds CA download endpoint (/api/ca.pem) with install instructions
on the login page. Injects CA cert into guest VM cloud-init so EC2
instances trust Spinifex services without --insecure. Adds 'spx admin
cert renew' command for cert regeneration after IP changes.
Includes certificate validation E2E test suite (run-cert-e2e.sh) with
system trust store, SAN, and in-guest TLS verification across single,
pseudo multi-node, and real multi-node modes.
Closes: mulga-777, mulga-778, mulga-779, mulga-780, mulga-781
Plan: docs/development/feature/cert-manager.md
The pseudo multi-node test was not passing --external-mode/--external-pool
to spx admin init, so instances never got public IPs. SSH then failed
trying to use hostfwd which doesn't exist with tap networking.
- init_leader_node: pass SPX_EXTERNAL_POOL as --external-mode=pool flags
- e2e.yml: set SPX_EXTERNAL_POOL=192.168.1.220-229 for pseudo test run
- Add Phase 8b to run-e2e.sh: public subnet instance gets PublicIpAddress
and dnat_and_snat rule; private subnet instance gets neither; no blanket
VPC CIDR SNAT (validates mulga-754 fix)
- Remove legacy external_mode="nat" — only "pool" with source=static or
source=dhcp. Admin init defaults to pool/dhcp when WAN detected.
- Enable dev_networking always (was disabled with external mode, breaking
E2E SSH tests)
- Update bootstrap.sh + e2e.yml: SPX_EXTERNAL_POOL env var configures
static IP pool for CI (230-240 single-node, 240-250 multi-node)
- Fix run-vpc-e2e.sh SNAT assertion for mulga-754 (no blanket SNAT)
- Add unit tests for parseDHCPCDLeasedIP, ReconcileFromKV edge cases,
DHCP pool config validation to meet 60% diff coverage threshold
Julian Sommer