CreateLoadBalancer is async: InstanceID is only set when the LB VM's QEMU
process starts, ~600ms after the create burst. The reactive
updateStoredConfig calls from CreateListener/RegisterTargets all no-op
during that window (InstanceID empty), and nothing rebuilds config once
the VM launches. The lb-agent then heartbeats, gets an empty ConfigHash,
never sees it change, and probes zero backends -> e2e 0/2 healthy.
Build the data-plane config on the provisioning->active heartbeat
transition instead: that is the one point where InstanceID is guaranteed
set and the agent is alive to consume config. The InstanceID guard in
updateStoredConfig stays as a cheap no-data-plane skip, no longer the
primary gate.
DHCP-source external-pool leases are DHCPRELEASEd only on explicit
instance/EIP terminate. Manager.Stop() preserves leases for
adopt-on-reboot, so env teardown/reset kills vpcd and destroys the
per-AZ lease KV without releasing — stranding every live lease
(instance ENIs, EIP allocations, per-VPC LRP gateways) on the upstream
DHCP server until TTL. Across CI churn with fresh client-ids the
upstream scope fills and new launches get no public IP.
Add Manager.DrainAll + a vpc.dhcp.drain RPC and a standalone
'spx admin cluster drain-dhcp' command that releases every held lease.
Wired into both teardown paths (single-node teardown role and
multi-node cluster-teardown.yml) before services stop. Stop() is
unchanged, so normal restarts still adopt leases on reboot.
* fix(ec2): fail RunInstances when public-IP auto-assign fails
PrepareRunInstances swallowed an AllocateIP failure for a
MapPublicIpOnLaunch subnet (logged WARN, fell through), so the instance
booted running with no public IP. With no qemu-hostfwd fallback such an
instance is unreachable. Mirror the adjacent AddNAT-failure path: detach
and delete the auto-created ENI, deallocate capacity, drop the instance,
and surface InsufficientAddressCapacity.
* test(daemon): deterministically pick a fitting 2-vCPU type in capacity filter test
CapacityFilterRespectsAllocation selected the first 2-vCPU type by Go
map-iteration order, intermittently picking a type near the
schedulable-memory boundary the allocator then rejected (~3/5 failures in
isolation on small hosts). Select the smallest-memory 2-vCPU type from the
capacity-aware set so it always fits.
CreateLoadBalancer's PutLoadBalancer-failure path only released the name
claim, orphaning the ENIs and managed front-end SG already created. Extract
the ENI+SG+claim teardown into rollbackLBInfra and call it from both the
ENI-create-failure and PutLoadBalancer-failure paths.
CreateListener persisted the listener record before authorizing the NLB
managed-SG port and reloading config; a failure in either left an orphan
listener record (and, after a config failure, a dangling SG authorization).
Add rollbackListener to revoke the port (when already opened) and
cascade-delete the listener on every post-persist failure.
Also wait for the async worker launch in the two CreateNodegroup disk-size
tests so they observe the launch under the record-before-launch contract.
* fix: make platform create handlers async to stop responder starvation
EKS CreateCluster/CreateNodegroup and ELBv2 CreateLoadBalancer ran their
full VM/control-plane launch synchronously inside the NATS subscription
callback on the shared spinifex-workers queue. A multi-minute create
head-of-line blocked the worker, so system.LaunchInstance/TerminateInstance
found no responder within the gateway timeout and failed with
'nats: no responders available' (mulga-siv-267.4).
Each create now runs validation + the atomic name claim synchronously,
snapshots the CREATING/provisioning output, then spawns the slow launch on
a per-impl launchWG and returns immediately. Launch failures mark the
record FAILED/CREATE_FAILED so 267.3 reclaim makes a retry idempotent. The
service interfaces are shared by the gateway NATS clients and the daemon
impls, so the async split lives inside the impl method and leaves the
interface, handleNATSRequest, and all daemon handlers unchanged.
The host-pinned system.LaunchInstance/TerminateInstance responders now
dispatch each request on its own goroutine (systemDispatchWg) so a slow
boot, which must respond after it completes, no longer blocks the subject.
ELBv2 keeps ENI creation synchronous (cheap VPC calls that yield the
AZ/VpcId the create response needs); only the LB-VM boot goes async,
folding InstanceID/IPs back into the record for a later Describe.
* fix: keep node-targeted launch subscription live under capacity pressure
The node-targeted (unicast) system.LaunchInstance.{type}.{node} subject was
torn down whenever the node filled, same as the queue (anycast) subject. For
the queue subject that is correct — NATS reroutes the launch to a node with
room. For the node-targeted subject there is no other node: a committed spread
reservation (EKS HA control-plane leg) that targets the now-unsubscribed node
gets 'nats: no responders available' instead of a capacity decision.
Keep the node-targeted subscription live for any type the node can host;
enforce capacity at launch time via ResourceManager.allocate, which already
rejects an over-capacity launch atomically. A reserved launch always reaches a
responder and, when the slot is still good, succeeds.
launchWorkers built RunInstancesInput with no BlockDeviceMappings, so
parseVolumeParams applied the 4GB root-volume default and the nodegroup
DiskSize was silently ignored. Wire rec.DiskSize into a BlockDeviceMapping
when set.
Run 27254808154 showed TestRebootResilience fail with ovn-controller on
the live chassis claiming NO local Port_Bindings post-reboot: ENIs stuck
on a dead chassis UUID, cr-gw chassis empty, NAT external_mac empty —
while NB intent was correct and taps carried the right iface-id. The SB
Chassis _uuid churned across the reboot (old UUID gone, new one created)
while the SB DB persisted, orphaning every claim.
The current artifacts cannot pin WHY the new chassis won't reclaim: the
ovn-controller vlog lives under /var/log/ovn, not journald, so it is
absent from the systemd-journal capture.
Add:
- phase5Snapshot: capture chassis identity PRE-reboot (system-id, SB
Chassis/Chassis_Private, get-connection) to ovn-pre-chassis.txt as the
churn baseline.
- dumpOVNClaimState: at post-reboot failure, capture the same chassis
state POST-reboot plus ovn-controller connection-status/debug, the
gw/eni Port_Binding claim detail, ovn-central-vs-ovn-controller boot
ordering, and tails of ovn-controller.log / ovn-northd.log →
diag-ovn-claim.txt.
Disambiguates chassis-identity churn from an SB/NB startup-ordering race
on the next reboot-suite run. Diagnostic only; no behavior change.
* feat: atomic create idempotency across EKS, ELBv2 and EC2 ClientToken
Every create handler took a non-atomic Get-check then a blind Put, so a
gateway/NATS retry that spawned a second concurrent handler could pass the
check in both and double-launch + leak VMs (267 failure-mode C).
Make each create claim its identity key atomically before any launching work:
- EKS CreateCluster: claim the cluster meta via kv.Create; a prior FAILED
attempt is reclaimed via a CREATING CAS so exactly one concurrent retry wins.
purgeClusterInfra gains a deleteMeta flag so reclaim purges infra while
keeping its claim key.
- EKS CreateNodegroup: claim the nodegroup record before SG provisioning or
worker launch; duplicates return ResourceInUse without provisioning.
- ELBv2 CreateLoadBalancer: account-scoped name claim (lbname.<acct>.<name>)
taken before ENI/VM work, released on every failure path and in
DeleteLoadBalancer; a name claim whose owner lbID has no live record is a
crashed-create orphan and is reclaimed.
- EC2 RunInstances: ClientToken dedup over a dedicated TTL bucket with a
two-phase claim (in-flight -> done) and reservation replay; a token reused
with different params returns IdempotentParameterMismatch.
Race tests under -race assert exactly one owner and a single launch for
concurrent same-name creates on all paths.
* test: cover ClientToken idempotency; extract orchestration for testability
Extract the claim->replay->launch->finalize/abort flow out of RunInstances
into runInstancesWithClientToken(store, ..., launch func) so the idempotency
path is unit-testable without a live gateway/NATS. RunInstances now just
resolves the store, hashes params, and passes a launch closure.
Make the in-flight poll timeout/step package vars so a duplicate-caller
wait-timeout is deterministically testable.
Adds tests: replay-skips-launch, owner-launches-once-then-replay,
launch-failure-aborts, param-mismatch maps to AWS error, getClientTokenStore
binds once, and in-flight wait-timeout.
After a host reboot the Northbound gateway_chassis intent recovers correctly
(SetGatewayChassis re-asserts it idempotently), but ovn-controller can leave the
Southbound Port_Binding for the gateway router port unclaimed (chassis == []).
With no claimed binding it installs no logical->physical flows for the centralised
gateway redirect, so every floating IP on the VPC is dark: ofproto/trace drops the
FIP ingress at br-int table 0, host->FIP is 100% packet loss, and the ALB lb-agent
heartbeat times out until the 5-min health window expires (TestRebootResilience).
rebindGatewayChassis only asserted NB intent. Now, after SetGatewayChassis, the
reconciler verifies the SB Port_Binding has a non-empty chassis and, if not, nudges
the local ovn-controller to recompute (inc-engine/recompute), polling until the
claim converges or a bounded deadline elapses — logging an error rather than
blocking reconcile. SB read + recompute are exec-only (ovn-sbctl/ovn-appctl) and
live in network/host behind a reconcile.GatewayClaimVerifier interface, wired in
vpcd; reconcile stays NB-pure and mockable. No NB object create/delete (ADR-0006 S5).
Verified by run 27251769281's diag-ovn-flows.txt (SB chassis empty + table-0 drop).
Validation completes on the next reboot-suite nightly (single-node self-heals too
fast to reproduce locally).
* test(reboot): capture heartbeat-path diagnostics on post-reboot health timeout
When post-reboot ALB targets fail to go healthy, the lb-agent's heartbeat to the
gateway times out with 'context deadline exceeded while awaiting headers' — a
symptom that alone cannot distinguish a network-level drop from an app/gateway
stall. Replace the fatal WaitForTargetsHealthy with a non-fatal poll that, on
timeout, tcpdumps br-wan for the heartbeat 5-tuple + ARP for the ALB floating IP
and pulls the lb-agent console + awsgw journal, then fails. Disambiguates the
layer before any fix (mulga-siv-265).
* test(reboot): re-aim heartbeat capture + add OVN dataplane flow-dump
Run 27250238163 proved the post-reboot failure is network-level (all 3 floating
IPs 100% packet-loss from host) but NOT neigh/NAT priming — neigh is primed and
NAT rows are present, only br-int forwarding is dead. The prior capture watched
br-wan for the ALB floating IP, while the real lb-agent heartbeat posts to the
gateway port on the host's advertise interface, so the L4 classifier could never
see the heartbeat and defaulted regardless.
- heartbeat tcpdump now watches the gateway port on `any` (real path) plus a
secondary br-wan floating-IP capture for north-south; classifier keys off
gateway-port L4 presence.
- new dumpOVNDataplane captures ovs-vsctl show, ovs-ofctl dump-flows br-int,
ovn-sbctl chassis + Port_Binding, dnat_and_snat NAT rows, host neigh, and a
host->floating-IP ping + ofproto/trace, to pin the flow-restore step.
- failure path dumps diag-ovn-flows.txt alongside diag-heartbeat.txt.
Test-only, build-tagged e2e; no production code.
CreateCluster set the control-plane VM/ENI/spread-group refs into the cluster
meta but first persisted them to KV only after allocateClusterEgressIP. A
failure between placement and that persist left the live sys.medium CP VMs
unrecorded, so neither DeleteCluster nor the FAILED-cluster reclaim could reach
them and they leaked, reserving host capacity.
Persist the meta immediately after placeControlPlane returns. The reclaim path
keys off the persisted ControlPlaneNodes + ControlPlaneSpreadGroup, so this also
frees reserved spread capacity on teardown.
A retried DeleteCluster (or the egress-delete cascade) can release the
hidden-pool allocation before the release step runs, so ReleaseAddress
returns InvalidAllocationID.NotFound. That was treated as a fatal teardown
error, so the cluster wedged in DELETING forever and its KV meta plus
downstream resources leaked. Swallow the NotFound variants as idempotent
success — mirroring the 165.17 ENI-teardown fix — so the SG + KV sweep
proceeds. A genuine release failure still wedges DELETING for retry.
* feat(eks): node-targetable control-plane VM (HA spread transport prereq)
Groundwork for HA control-plane placement (mulga-siv-231.7.1): make the k3s
control-plane VM placeable on a specific Spinifex host.
- Add a sys.medium instance type (2 vCPU / 4 GB) and boot the control-plane VM
as it instead of t3.medium. A sys.* type keeps the CP VM out of customer
DescribeInstanceTypes and makes the daemon register the node-targeted
system.LaunchInstance.{type}.{nodeID} subject (only sys.* types get it).
- Add K3sServerInput.TargetNodeID and a LaunchSystemInstanceOnNode method on
the launcher: empty/local node launches in-process (unchanged single-CP
path); any other node dispatches over the node-targeted subject, where the
receiving daemon runs the launch and owns the VM.
The spread orchestrator (reserve N hosts, launch per host, verify, rollback,
<3-host fallback) lands separately; CreateCluster still launches locally
(TargetNodeID empty), so behaviour is unchanged.
* feat(eks): HA control-plane spread orchestrator
Place N=3 k3s server VMs across distinct Spinifex hosts via a per-cluster
spread placement group, with all-or-nothing rollback and single-CP fallback.
- placeControlPlane: capacity fan-out (HostScheduler) -> >=3-host gate ->
ensure spread group (system account) -> ReserveSpreadNodes(Min=Max=3) ->
parallel per-node LaunchK3sServerVM(TargetNodeID) -> node-vms placement
verify -> FinalizeSpreadInstances. Any launch/verify/finalize failure
terminates launched VMs, releases the reservation, drops the group.
- <3 schedulable hosts or a lost reservation race falls back to a single
local control-plane VM (prior behaviour).
- ClusterMeta gains ControlPlaneNodes[] + ControlPlaneSpreadGroup; [0] is
mirrored into the scalar ControlPlane* fields for back-compat. DeleteCluster
tears down every CP VM and the spread group.
- Wire PlacementGroup + Scheduler into EKSServiceDeps + orchestration-deps gate.
* fix(eks): size HA control-plane spread on raw host headroom
SchedulableHosts matched the CP type by name in node.status's per-type
capacity list, but that list omits sys.* types (system types are hidden
from customers). The CP VM is sys.medium, so the match never hit and
SchedulableHosts returned 0 on every node — placeControlPlane always fell
back to a single CP, making the >=3-host spread unreachable in production.
System VMs consume host vCPU/memory like any guest, so size the fit against
each node's raw schedulable headroom (Total - Reserved - Alloc) and the
type's footprint instead. Adds instancetypes.SpecForSystemType for the
sys.* vCPU/memory spec and a pure nodeFitsSystemInstance helper.
* feat(eks): k3s server-join role for HA control-plane quorum
Servers 2..N now boot in join mode: server: https://<first-server>:6443 +
a shared cluster token, without cluster-init, so they join the first server's
embedded-etcd quorum instead of each cluster-initing an independent datastore.
- K3sServerInput gains JoinToken (shared cluster secret, set on every server)
and ServerURL (set => join server, no cluster-init, server-join role).
- buildK3sUserData branches cluster-init vs server:, emits token:, and selects
SPINIFEX_K3S_ROLE server vs server-join.
- launchControlPlaneSpread launches nodes[0] first (cluster-init), derives the
join URL from its ENI IP, then launches the rest in parallel as join servers;
a first-server failure skips the joins and rolls back.
- CreateCluster mints a per-cluster crypto-random token (GenerateK3sClusterToken)
threaded into the placement template.
- eks-node-role.sh adds a server-join case: token-webhook + k3s + state-report,
but not k3s-first-boot (the first server already publishes the bootstrap
artifacts; a join re-publish only races the bootstrap bus).
* fix: open etcd-peer/kubelet ingress on EKS control-plane SG for HA join
Join servers (231.7.2) register over the supervisor port 6443 but their
embedded etcd could not peer with the quorum: the control-plane SG only
admitted 6443 from the VPC CIDR, omitting etcd client+peer 2379-2380 and
kubelet 10250 between control-plane nodes. A single cluster-init server never
exercises those ports, so the gap stayed hidden until multi-server HA — the
join registered but its etcd never replicated the raft log, leaving the node
perpetually NotReady.
Add EnsureControlPlaneHAIngress with self-referencing (cpSG->cpSG) rules for
etcd 2379-2380 and kubelet 10250, called from CreateCluster after
EnsureControlPlaneIngress. Verified live on a 3-node cluster: join servers
reach Ready and form a 3-member etcd quorum that survives loss of the original
first server.
* feat: register all control-plane ENI IPs as EKS cluster NLB targets
Register every HA control-plane server ENI IP in the cluster NLB target
group, not just the primary. The TG already health-checks :6443, so a
dead member drains and kubectl/apiserver traffic fails over to surviving
etcd-quorum members. Add RegisterClusterTargets batch helper; the single
RegisterClusterTarget now delegates to it.
Associated Elastic IPs lost external connectivity after a host reboot:
vpcd's reconcile re-applied the dnat_and_snat rule without the ENI MAC,
collapsing it to the centralised shape (external_mac/logical_port unset),
and the daemon's relaunch republish only fired for auto-assigned public
IPs, never for store-backed EIPs.
- EIPRecord gains MacAddress, captured at AssociateAddress and cleared at
DisassociateAddress, so reconcile can re-apply the distributed shape
without re-querying the ENI.
- loadEIPs feeds rec.MacAddress into the reconcile spec; policy.AddEIP
self-heals a stale centralised row when given a MAC.
- AssociatedPublicIPForInstance lets the relaunch hook resolve an
associated EIP from the store and republish vpc.add-nat with the MAC.
* feat(eks): node-targetable control-plane VM (HA spread transport prereq)
Groundwork for HA control-plane placement (mulga-siv-231.7.1): make the k3s
control-plane VM placeable on a specific Spinifex host.
- Add a sys.medium instance type (2 vCPU / 4 GB) and boot the control-plane VM
as it instead of t3.medium. A sys.* type keeps the CP VM out of customer
DescribeInstanceTypes and makes the daemon register the node-targeted
system.LaunchInstance.{type}.{nodeID} subject (only sys.* types get it).
- Add K3sServerInput.TargetNodeID and a LaunchSystemInstanceOnNode method on
the launcher: empty/local node launches in-process (unchanged single-CP
path); any other node dispatches over the node-targeted subject, where the
receiving daemon runs the launch and owns the VM.
The spread orchestrator (reserve N hosts, launch per host, verify, rollback,
<3-host fallback) lands separately; CreateCluster still launches locally
(TargetNodeID empty), so behaviour is unchanged.
* feat(eks): HA control-plane spread orchestrator
Place N=3 k3s server VMs across distinct Spinifex hosts via a per-cluster
spread placement group, with all-or-nothing rollback and single-CP fallback.
- placeControlPlane: capacity fan-out (HostScheduler) -> >=3-host gate ->
ensure spread group (system account) -> ReserveSpreadNodes(Min=Max=3) ->
parallel per-node LaunchK3sServerVM(TargetNodeID) -> node-vms placement
verify -> FinalizeSpreadInstances. Any launch/verify/finalize failure
terminates launched VMs, releases the reservation, drops the group.
- <3 schedulable hosts or a lost reservation race falls back to a single
local control-plane VM (prior behaviour).
- ClusterMeta gains ControlPlaneNodes[] + ControlPlaneSpreadGroup; [0] is
mirrored into the scalar ControlPlane* fields for back-compat. DeleteCluster
tears down every CP VM and the spread group.
- Wire PlacementGroup + Scheduler into EKSServiceDeps + orchestration-deps gate.
* fix(eks): size HA control-plane spread on raw host headroom
SchedulableHosts matched the CP type by name in node.status's per-type
capacity list, but that list omits sys.* types (system types are hidden
from customers). The CP VM is sys.medium, so the match never hit and
SchedulableHosts returned 0 on every node — placeControlPlane always fell
back to a single CP, making the >=3-host spread unreachable in production.
System VMs consume host vCPU/memory like any guest, so size the fit against
each node's raw schedulable headroom (Total - Reserved - Alloc) and the
type's footprint instead. Adds instancetypes.SpecForSystemType for the
sys.* vCPU/memory spec and a pure nodeFitsSystemInstance helper.
* fix: strip duplicate lb- prefix from haproxy socket and cert paths
LoadBalancerID, ListenerID and friends already carry their type prefix
(lb-/lst-), so the stats-socket, frontend cert, and HAProxyManager
config/pid filename templates double-prefixed: lb-lb-<id>.sock,
lb-lb-<id>-<lst>.pem, alb-lb-<id>.cfg.
The socket path is computed independently on both sides (gateway render
in buildHAProxyConfig, agent poll in lbagent.New); any divergence in the
lb- prefix convention between the two leaves the agent polling a socket
HAProxy never bound, spamming 'HAProxy stats unavailable: no such file'
while the data plane serves fine. Drop the redundant literal prefix so
both sides resolve to <id>.sock and the alb-/nginx cert paths match the
delivered config.
* refactor: remove unused HAProxyManager
HAProxyManager was never constructed (NewHAProxyManager had no callers).
The live ALB data plane runs in the lb-agent via reloadHAProxy with the
config generated daemon-side and shipped; this per-daemon process manager
was dead code. Remove the type, its methods, tests, and now-unused
imports (log/slog, os/exec, sync, syscall).
* ci(e2e-nightly): run Go harness suites instead of bash run-*.sh
Swap nightly cells 1-3 (single), 8-10 (real-multi) and 18 (reboot) to the
compiled <suite>.test binaries, mirroring e2e.yml: SPINIFEX_E2E=1 with
SPINIFEX_MODE single|multinode, per-suite logs to ARTIFACT_DIR, then
go-junit-report + e2e-analyze + upload-artifact. Adds a spinifex checkout for
the composite actions and pulls /run/spinifex/console-*.log so LB microVM
(lb-agent) state is captured on failure. Cells 17 (tofu) and 20 (baremetal)
stay on bash — no Go suite exists.
* ci(e2e-nightly): cell-17 RCA analytics + cell-20 cosmetic alignment
run-tofu-examples-e2e.sh now emits go test -v markers per workbook so the
cell-17 step's tee'd test-tofu.log flows through the same go-junit-report ->
e2e-analyze -> upload-artifact chain as the Go suites, producing junit-tofu.xml
and an analysis bundle for RCA. Align e2e-baremetal.yml (cell 20) ARTIFACT_DIR,
analyze title, and artifact name with the matrix cells so CI-RCA discovers its
bundle uniformly.
* ci(e2e-nightly): wire host journal + VM consoles into RCA bundles
Collect now emits the canonical spinifex-journal.log (concat of all nodes)
that the e2e-analyze bundler slices into each failure's journal.log, so
elbv2-service/vpcd/daemon lines in the failure window travel with the bundle
(previously the journal uploaded as spinifex-journal-node*.log, a name the
bundler never reads). Rename the console capture to vm-console-node* and note
it also carries guest cloud-init/sshd consoles, covering the cell-3 internal-LB,
cell-18 reboot-datapath, and cell-9/10 guest-ssh blind spots.
* ci(e2e): self-heal runner /dev + --one-file-system on workspace clean
Clean Workspace now detaches any leaked iso-builder rootfs mount, recreates
/dev/null if a prior build clobbered it, and uses rm --one-file-system so the
clean can't descend into a still-mounted devtmpfs and delete host device nodes.
Guards both the matrix cells and the bare-metal job against the host-wide
/dev/null wipe that fails go build with 'open /dev/null: no such file or
directory'.
The LB VM's external-pool floating-IP NAT (dnat_and_snat external_ip -> LB
VPC IP) was reaped only conditionally on teardown: DeleteNetworkInterface
skips EIP-owned addresses (defers to the EIP lifecycle), and that lifecycle's
reap runs in the async TerminateSystemInstance goroutine gated on the EIP
association. So repeated create/delete cycles accumulated stale rules that
collapsed onto the reused LB VIP, making OVN SNAT/un-DNAT ambiguous and
dropping the lb-agent heartbeat return path.
DeleteLoadBalancer now publishes vpc.delete-nat deterministically from the LB
record (external=AvailZones public IP, logical=VPCIP). DeleteEIP matches on
external+logical and is idempotent, so redundant reaps from other paths are
harmless. Internal LBs (no floating IP) are a no-op.
* fix: re-prime EIP reachability on AddEIP idempotent-skip path
AddEIP early-returned on the idempotent-skip path (OVN DNAT row already
matches) before emitting GARP and priming the host neighbour. Stop->start
and daemon reboot-recovery re-attach the same EIP with an unchanged row, so
the host neigh entry / OVS datapath never got re-primed and the EIP/ALB-VIP
stayed dark until the kernel ARP entry timed out (60-300s). inject-garp is
unsupported on some chassis, making neighPrime the only working remediation
— exactly the step the skip bypassed.
Extract garp + neighPrime/flush into primeReachability and run it on the
skip path as well as after a fresh AddNAT. The FlowsBarrier still fires only
on row churn. Resolves cell-9/10 TestMultinodeGuestSSH and cell-18
TestRebootResilience nightly failures.
* refactor: remove dead inject-garp GARP emitter from EIP path
ovn-appctl inject-garp is not a valid command on the OVN build Spinifex
deploys and tests against — every AddEIP GARP attempt failed with
'inject-garp is not a valid command' and fell through to the host neigh
prime/flush, which is the only remediation that works. The GARP hook was a
guaranteed-failing shell-out plus a misleading WARN on every EIP attach.
Delete host.InjectGARP, the GARPEmitter type, WithGARPEmitter option, the
garp field + its m.garp() call in primeReachability, and the vpcd
injectGARPEmitter wiring. host.ReplaceNeigh (distributed) and host.FlushNeigh
(centralised) already cover both NAT shapes and become the sole EIP
reachability path. Reword the neigh hook docs that framed themselves as an
inject-garp-independent fallback. Prune GARP-only tests; move the shared
recordingRunner helper into neigh_test.go.
* feat(daemon): live MemAvailable admission gate + fixed-services reserve scope
Admission previously charged only guest -m against a flat reserve, blind to the
co-located storage VMs, nbdkit, and daemon stack — so it greenlit launches that
overcommit real RAM and OOM-kill guests on memory-tight hosts.
Add a second admission gate that clamps the accounting-derived launch count by
live /proc/meminfo MemAvailable (which already nets reclaimable cache and sees
every untracked consumer): admitted = min(accounting, (MemAvailable-reserve)/
guestMem). Reads live per admission, fails open on a /proc read error so a
transient fault can't wedge launches, and is disabled by SPINIFEX_ADMISSION_LIVE_MEM=0.
Re-scope the host reserve to the fixed infrastructure tier only; per-instance
nbdkit is no longer expected to fit inside it (explicit per-instance charge is a
follow-up; the live gate covers it implicitly in the meantime).
Pure helpers (liveMemCount, parseMemAvailableKB, liveMemAdmissionEnabled) are
unit-tested; liveMemGate covered via an injected reader.
* feat(daemon): charge nbdkit per-instance in admission accounting
An instance's admission memory charge is now its guest -m plus the nbdkit
processes backing its volumes (one per attached volume), not guest -m alone.
The static reserve no longer has to absorb per-instance nbdkit, which was
unsizable and the root of the host-memory-overcommit OOM.
Per-volume charge is measured per class (main 768 MiB: 128 cache + plugin
working set + in-flight I/O; aux 96 MiB: process floor, 0 cache), env-tunable
via SPINIFEX_NBDKIT_MAIN_MIB / SPINIFEX_NBDKIT_AUX_MIB. Default layout is one
main + two aux volumes (root + cloudinit + efi) ~= 0.94 GB/instance. Applied at
every accounting site through instanceMemChargeMiB so charge and release never
drift; the live MemAvailable gate backstops block-device-mapping volumes the
static count misses.
* feat(vm): three-tier guest OOM ladder + spinifex-guests.slice
Replace the flat oom_score_adj=500 on every guest QEMU with a ManagedBy-derived
ladder: user guests (ManagedBy=='') stay +500 and are reaped first under host
memory pressure; system instances (ELBv2 LB-VMs, EKS control-plane nodes,
ManagedBy!='') score 0, ranking above user guests but below the infra tier
(-500). Killing a system instance degrades the data path or cluster control
plane for many user VMs, so it must outlive them.
Add spinifex-guests.slice as a sibling of spinifex-system under spinifex.slice —
the parent tier for the per-instance leaves that land with RG-7. Guest QEMU
placement into the slice is deferred to the per-instance work.
* refactor(systemd): lean unit comments to RG-n tags + RG-9/RG-11 invariants
Strip the multi-paragraph rationale novels from spinifex-daemon.service (48->7
comment lines), spinifex-vpcd.service (29->8) and spinifex.slice (15->5),
replacing them with settings + terse # RG-n clause tags. Remove plan-doc, bead
and CI-run references from comments (project policy). Directives are byte-for-byte
unchanged — comment-only.
Add RG-9/RG-11 invariant tests asserting per-tier confinement (locked-down
storage/control tier drops all caps + NoNewPrivileges=yes; daemon keeps exactly
CAP_SYS_ADMIN+CAP_DAC_OVERRIDE and the four-entry DeviceAllow; vpcd is the
documented network-tier exception) and the lean-comment budget + no-plan-ref rule.
Project internal StateError onto AWS stopped (code 80) in DescribeInstances
so SDK/UI clients no longer break on the non-AWS 'error' state label.
Route start-instances through the per-instance ec2.cmd topic first so a
crash-pinned StateError instance is restarted by its live owner node, whose
StartInstance handler now accepts StateError (Manager.Start transitions
error->pending before relaunch). Fall back to the ec2.start queue-group path
on ErrNoResponders for genuinely stopped instances in the shared KV.
* ci(eks-ami): build on self-hosted ci-single-1 runner
GitHub-hosted ubuntu-24.04 cannot run the libguestfs appliance network:
passt fails (PID file denied under its AppArmor confine / privilege drop)
and the SLIRP fallback has no outbound, so virt-customize --network dies.
The self-hosted ci-single-1 runner has /dev/kvm and real networking, so
the existing libguestfs build works unmodified.
* fix(build-system-image): add --no-cache, force fresh build on publish
The <10min cached-raw skip avoids duplicate work for concurrent e2e
jobs, but on the persistent publish runner it could republish a stale
image from a prior or concurrent run. Add --no-cache to bypass the skip
and have publish-system-image.sh pass it on every build.
* ci(eks-ami): target ci-single label (no ci-single-1 label exists)
The self-hosted runners are named ci-single/-2/-3 but all share the
single label 'ci-single'; runs-on matches labels, not names, so
ci-single-1 never schedules.
* ci(eks-ami): disable Go cache on self-hosted runner
setup-go's cache-save (Post Setup Go) re-uploaded the Go cache to the
Actions backend on every run, taking ~2.5min vs the 36s build. The
self-hosted runner already keeps the cache on local disk, so cache:false.
* feat(systemd): bound spinifex to a resource-limited slice
Add a spinifex.slice parent ceiling (MemoryHigh=80%, MemoryMax=90%,
MemorySwapMax=0) with a spinifex-system.slice child holding the seven
control/infra services, so a runaway hits the slice instead of the host.
A system.slice drop-in (MemoryMin=1G, CPUWeight=300) reserves RAM and CPU
for sshd/journald/the operator, keeping the box reachable when spinifex is
maxed.
The drop-in is generated in setup.sh install_systemd() rather than shipped
as a staged file, since the packaging globs flatten the systemd/ dir; this
mirrors the existing sshd-keygen drop-in pattern. The two .slice files flow
through every staging path unchanged.
Within the single effective cgroup, a slice MemoryMax OOM picks the highest
oom_score victim — guest QEMU (+500) before any service (-500) — so a slice
OOM degrades by killing one guest while the daemon, storage tier, NATS, and
the host all survive.
* feat(setup): provision 8G host swapfile + vm.swappiness=10
Add a setup_swap stage that creates an 8G /swapfile, enables it, persists it
to fstab, and writes vm.swappiness=10 so the kernel only swaps under real
pressure. Backs spinifex.slice MemorySwapMax so the slice can spill to swap
for graceful degradation instead of OOMing a guest. Idempotent; skipped under
ISO_BUILD (firstboot provisions swap on ISO hosts). Reverses the prior swap=0
assumption.
* Revert "fix(network): prime host neigh with gateway port MAC on centralised EIP attach"
This reverts commit f7446c69c6.
* fix(network): round-robin external pool cursor so a released IP is not reused instantly
The static external pool handed out the lowest free address and freed IPs
immediately on release, so a just-released EIP was reassigned within ~0.6s. A
recycled EIP front-end is unreachable from the same host: the kernel still
holds the prior owner's ARP binding (or a negative cache after the neigh
flush) and OVN does not answer the re-ARP in time, so the neighbour entry
settles to FAILED (EHOSTUNREACH). This surfaced as EKS TestEKS/KubectlGetNodes
failing in single-node CI once #315 routed kubectl through the cluster NLB's
external EIP. Dev never hit it — it uses DHCP-random addresses.
Add a persisted per-pool cursor: allocation resumes one past the last address
handed out and wraps at RangeEnd, so a freed IP is not reused until the cursor
cycles the whole range. Empty cursor (fresh pool / older KV record) starts at
RangeStart, preserving lowest-free order for the first cycle — backward
compatible, no migration.
* ci: drop e2e-eks from default e2e suites
EKS data-plane subtest (TestEKS/KubectlGetNodes) is a persistent flake on
single-node CI — a freshly-attached centralised EIP is intermittently
unreachable from the host (no route to host to the cluster endpoint), which
fails the whole E2E run and blocks unrelated suites. Remove e2e-eks from the
default SUITES so PR/push runs no longer gate on it. The e2e-eks target and
the workflow_dispatch suites input are unchanged, so EKS stays runnable on
demand and via make e2e-all.
Centralised-mode EIP attach only flushed the host neighbour entry and relied
on OVN re-answering the host's ARP. On OVN builds whose ovn-controller lacks
inject-garp, no node answers a same-chassis recycled IP's re-ARP, so the entry
decays to FAILED and the external IP is unreachable ("no route to host"). This
sank the EKS e2e suite: the cluster NLB front-end EIP never resolved on br-wan,
so kubectl could not reach the published apiserver endpoint.
Mirror the distributed-mode prime: resolve the VPC gateway router port MAC (the
L2 owner of a centralised EIP on the WAN segment) and program the host
neighbour entry directly, falling back to a flush only when no MAC resolves.
* feat: un-stub EKS Addon API with managed-addon state machine
Wire the 6 EKS Addon ops (List/DescribeVersions/Create/Describe/Update/
Delete) to a per-cluster managed-addon state machine in the per-account KV.
- addon_catalog.go: static air-gapped catalog (aws-load-balancer-controller,
coredns) with version validation.
- addon_state.go: AddonRecord + AddonStatus, KV CRUD + CAS helpers; list skips
staged-manifest sub-keys; delete sweeps record + manifest.
- addon_installer.go: transport-agnostic AddonInstaller; stagingInstaller
stages a manifest descriptor in KV for the (future) VM-side delivery slice.
- addons.go: op impls + AWS-shape converters; installer failure flips
CREATE_FAILED. Status stays CREATING until delivery confirms ACTIVE (private
apiserver is host-unreachable, so delivery must be VM-side).
- service_impl.go: drop notImpl stubs; add AddonInstaller dep (nil-defaults to
staging installer).
DeleteClusterPrefix already sweeps clusters/{name}/ so addon keys clean up on
cluster delete.
* fix: route ListAddons at GET /clusters/{name}/addons
ListAddons was wired at GET /addons with no cluster param, so the AWS CLI's
GET /clusters/{name}/addons hit no route (InvalidActionException) and the
wrapper couldn't pass a cluster to the service. Match the real EKS path and
thread the cluster name through.
Recycled WAN EIPs stayed unreachable from the host (kubectl to a cluster
endpoint: "no route to host", host neigh FAILED) on OVN builds whose
ovn-controller lacks the inject-garp appctl. OVN advertises a recycled
external IP's new MAC only via the GARP it emits on LSP binding-chassis
migration; a same-chassis rebind emits none, and the neigh flush added for
recycle reachability then waits on an ARP reply no node sends.
In distributed mode the external_mac is known at attach time, so program the
host neighbour entry directly (ip neigh replace <ip> lladdr <mac> nud
reachable) instead of flushing. Inbound traffic refreshes the entry from
there. Centralized attach (no external_mac) and all detaches keep the flush.
Adds host.ReplaceNeigh + policy.NeighPrimer hook wired in vpcd alongside the
existing flusher.
Workloads need a working ingress/servicelb path before the
aws-load-balancer-controller add-on lands. Flip builtinIngressEnabled to
default ON; opt out per-cluster for AWS parity with the CreateCluster tag
spinifex.io/managed-ingress=false. Interim divergence — revert to
default-off when the controller add-on ships.
Recycled WAN external IPs stayed unreachable for the kernel ARP timeout
(60-300s) when ovn-controller lacks the inject-garp appctl: the GARP refresh
after AddEIP is then a no-op, leaving the host neighbour cache pointed at the
prior owner's MAC.
Add host.FlushNeigh (ip neigh flush) and a policy.NeighFlusher hook wired
into AddEIP and DeleteEIP, so a recycled IP re-resolves L2 against the current
owner independent of inject-garp. DeleteEIP now takes the external IP (the
vpc.delete-nat event already carries it) to flush on release.
* feat(eks): public/private endpoint access drives NLB scheme + IP:port reachability
Parse resourcesVpcConfig.endpointPublicAccess/endpointPrivateAccess/
publicAccessCidrs (defaults public=true, private=false, cidrs=0.0.0.0/0) and
reject the both-disabled combination. Public access provisions an
internet-facing NLB and publishes its external frontend IP as the cluster
endpoint; private access keeps the NLB internal and uses the VPC frontend IP.
Inject the resolved endpoint IP into the k3s apiserver tls-san so the
published IP:443 endpoint serves a cert valid for verified TLS, and surface
the access fields in DescribeCluster. The e2e suite now connects via the
verified endpoint with TLS verification on, dropping the insecure mgmt-IP
shim.
* fix(eks): authorize NLB->apiserver ingress on control-plane SG
The cluster control-plane security group was created with empty ingress, so
the NLB's backing LB VM could not reach the apiserver on 6443 from inside the
VPC. The target group stayed unhealthy and the published endpoint never
served. Authorize tcp/6443 on the control-plane SG from the VPC CIDR after the
SGs are created; public exposure remains gated at the NLB front-end by
publicAccessCidrs, so this only opens the in-VPC LB->apiserver hop.
Extend SubnetVPCResolver with GetVPCCIDR so the service can resolve the VPC
CIDR for the rule.
* feat(eks): thread publicAccessCidrs to NLB SetLoadBalancerIngressCIDRs
Wire the cluster's publicAccessCidrs (parsed from CreateCluster
ResourcesVpcConfig) through EnsureClusterNLB to the ELBv2
SetLoadBalancerIngressCIDRs override so a public cluster endpoint can be
narrowed below the wide-open scheme default. The override only fires for
an internet-facing NLB whose CIDRs restrict below 0.0.0.0/0; internal
NLBs and the default keep the LB's out-of-the-box ingress.
NLB ENIs fell back to the VPC default security group, whose intra-SG-only
ingress made the OVN ACL drop inbound listener traffic at the front-end VIP
even with a healthy target group. NLBs reject customer SGs, so mint an internal
managed SG per network LB instead: attach it to every LB ENI at create, open
each listener port on it from the resolved client CIDRs at CreateListener,
close ports on DeleteListener, and delete the SG on DeleteLoadBalancer.
Default ingress CIDRs are scheme-based (internet-facing: 0.0.0.0/0; internal:
the VPC CIDR). SetLoadBalancerIngressCIDRs overrides them and re-authorizes
existing listeners, providing the hook for EKS publicAccessCidrs. ALBs are
unchanged.
* feat(eks): publish eks-node AMI to R2 + spx catalogue entry
Add a spinifex-eks-node catalogue entry so operators install the EKS
node image with a single command:
spx admin images import --name spinifex-eks-node
The entry pulls the Alpine+K3s image from iso.mulgadc.com, verifies its
sha256, and stamps spinifex:managed-by=eks — the tag the control-plane
and node-group launchers resolve the boot AMI by, so no extra config is
needed.
Add scripts/publish-system-image.sh (+ make publish-eks-node-image) to
convert the built raw to a compressed qcow2, emit a checksum sidecar,
and upload both to the R2 bucket backing iso.mulgadc.com/system-ami via
the S3 API.
* ci(eks): add manual workflow to build + publish eks-node AMI to R2
Build the eks-node AMI on the runner (libguestfs) and publish via
publish-system-image.sh using the existing CF_R2_* secrets. Manual
workflow_dispatch with a dry-run input. Fix the publish script's default
bucket to spinifex-iso (the bucket iso.mulgadc.com is served from).
The LB system microVM stripped its module tree to virtio + fw_cfg +
failover, but init.sh only loaded qemu_fw_cfg and virtio_net. virtio_net
depends on net_failover -> failover; modprobe resolves that chain via
modules.dep, and load_mod()'s insmod-by-path fallback loads a single .ko
with no dependency resolution.
When the image is built on a kmod-less host (depmod skipped, no
modules.dep) the chain can't be resolved either way, so virtio_net fails
to load, no NIC appears ('no interface found for MAC'), the lb-agent has
no network, heartbeats never reach AWSGW and every LB hangs in
provisioning until the 5m e2e timeout. Baremetal runners ship kmod so
modules.dep is generated and the chain resolves — which is why the LB
suite passed there but failed on normal CI.
Load failover -> net_failover -> virtio_net explicitly bottom-up so the
NIC comes up regardless of whether modules.dep exists.
EKS create-cluster fails deterministically on the multinode job with
ServiceUnavailableException: the EKS KV is a clustered (Raft) JetStream
stream and create-cluster needs an elected leader, but multinode
JetStream clustering isn't reliably forming yet (see mulga-omq/z2z/5yg).
EKS is validated single-node only at this stage.
Drop eks from the multinode build+run suite filters (alongside the
existing reboot/single exclusions) so the multinode job reflects what's
actually supported. Single-node continues to run the full eks suite.
* feat(elbv2): SetSubnets — full add+remove LB subnet re-homing
Add the ELBv2 SetSubnets operation across the interface, NATS subject,
daemon handler/subscription, and gateway route/forwarder.
Re-homing is a full add+remove: subnets new to the LB get a fresh managed
ENI, dropped subnets have their ENI deleted, then the backing system VM is
relaunched on the new ENI set. The system-instance launcher exposes only
Launch/Terminate (no live ENI hotplug for system VMs), so applying new
subnets relaunches the LB VM — a brief data-plane interruption. The record
is mutated only after the new ENIs exist and the VM is relaunched.
Extract a shared launchLBVM helper from CreateLoadBalancer so create and
SetSubnets boot the LB VM identically. Idempotent on no-op; record-only
fallback when no VPC service is wired.
Completes the elbv2 mutation-API surface (SetSubnets was the last gap).
* feat(tagging): resourcegroupstaggingapi GetResources for controller discovery
Add the Resource Groups Tagging API as a new gateway service (SigV4 scope
\"tagging\", AWS JSON 1.1 / X-Amz-Target, mirroring ACM). GetResources is a
gateway-side aggregator over the existing elbv2 and ec2 tag stores — no new
NATS subject or daemon handler.
It lists the account's load balancers and target groups (attaching tags via
batched DescribeTags) plus EC2 resources from the tag store, then applies the
request's ResourceTypeFilters (service-wide or fully-qualified) and TagFilters
(AND across filters, OR within values). Results are ARN-sorted with opaque
offset pagination.
This is how aws-load-balancer-controller v2 discovers and adopts the elbv2
resources it manages (filtering on its cluster/stack tags); DescribeLoadBalancers
has no tag-filter field, so RGT GetResources is the required path.
* fix(eks): emit canonical exception code, no doubled suffix
GenerateEKSErrorResponse appended \"Exception\" unconditionally, so codes that
already carry it (e.g. awserrors.ErrorEKSResourceNotFound =
\"ResourceNotFoundException\") serialized as ResourceNotFoundExceptionException.
SDK clients comparing eks.ErrCodeResourceNotFoundException then fail to match.
Append the suffix idempotently. Also covers the many other awserrors codes that
already end in \"Exception\" and flow through the shared JSON error envelope
(EKS, ACM, tagging).
* refactor(tagging): split GetResources fetch from orchestration for testing
Introduce a resourceLister seam: GetResources wires a NATS-backed lister, while
getResources (parse + tag-filter + sort + paginate) and buildEC2Mappings (tag
grouping + ARN synthesis) are now pure and unit-tested without a live backend.
No behaviour change.
* feat(eks): per-cluster opt-in K3s built-in ingress + enforce parity default
The runtime-rendered /etc/rancher/k3s/config.yaml emitted no disable: block, so
traefik + servicelb actually ran on every cluster — the AMI config.yaml.skel
that disables them is a never-applied boot fallback (cloud-init overrides it).
Render the disable block by default so traefik + servicelb are off for AWS
parity (Service type=LoadBalancer / Ingress are the AWS Load Balancer
Controller's job). A cluster created with tag spinifex.io/managed-ingress=true
keeps the K3s built-ins for interim in-VPC app exposure before the controller
add-on lands. The flag is derived into ClusterMeta.BuiltinIngress at
CreateCluster and threaded onto K3sServerInput.
* feat(ui): subnet editor on load balancer detail page (SetSubnets)
Add a Subnets tab to the LB detail page backed by the elbv2 SetSubnets
API. The editor mirrors the security-groups editor (checkbox list of the
VPC's subnets, dirty-tracking, Reset/Save) and warns that applying a
change re-homes the load balancer by relaunching its system VM, causing
a brief data-plane interruption.
* fix(elbv2): SetSubnets must detach ENIs before delete/relaunch + roll back on terminate failure
TerminateSystemInstance tears down the LB VM but leaves its ENIs marked
in-use, so the relaunch path could neither delete removed-subnet ENIs
(InvalidNetworkInterface.InUse leak) nor re-attach retained ENIs to the
relaunched VM (attach failed InUse, dropping the retained subnet's data
plane). Detach the existing ENIs explicitly after terminate — mirroring
DeleteLoadBalancer — before deleting removed ones and relaunching.
Also roll back the freshly-created added-subnet ENIs when the terminate
step itself fails, so a failed relaunch no longer leaks ENIs (amplified
by SDK retries).
* fix(daemon): re-arm system-instance terminate subscription on recovery
system.TerminateInstance.{id} was bound only by the system.LaunchInstance
handler at first launch, so it was lost across a daemon restart: recovery
reconnects to the surviving QEMU (or relaunches) but never re-bound the
subject, leaving system.TerminateInstance.{id} with no responder. Any LB
that predated the restart could then never be torn down — SetSubnets
relaunch and DeleteLoadBalancer both failed with 'no responders'.
Re-arm it in onInstanceUpHook, which fires on both the relaunch and the
reconnect-to-surviving-QEMU recovery paths, for ManagedByELBv2 VMs; drop
it in onInstanceDownHook. Split subscribeSystemTerminate into a locked
helper so the hook (which already holds d.mu) can reuse it.
Make e2e-baremetal.yml a reusable workflow (workflow_call + a ref input)
so the nightly and release pipelines can invoke the real PXE-reimaged
hardware suite, which previously only ran on manual dispatch.
Nightly: add a baremetal job (conceptual cell 20) that runs every
weeknight on the existing cron, or on dispatch when 20 is selected. It
tests dev on schedule, matching GIT_BRANCH. Sits outside the tofu matrix
(different runner, provisioning, 120m timeout) and shares the
e2e-baremetal-cluster concurrency group, so all baremetal runs serialise
on the single physical node.
Release: a release builds and uploads an ISO to R2 but never booted that
ISO/squashfs path on hardware. Add a baremetal job (needs: release) and
hard-gate on it: promotion moves to a dedicated promote job
needs:[verify, baremetal] (with a matching mark-failed job on failure),
and the iso job's upload gate now waits for both verify and baremetal to
conclude success before pushing to R2. Bump the iso job timeout to 180m
since it now waits on the parallel ~2h baremetal run.
Julian Sommer