* Revert "fix(network): prime host neigh with gateway port MAC on centralised EIP attach"
This reverts commit f7446c69c6.
* fix(network): round-robin external pool cursor so a released IP is not reused instantly
The static external pool handed out the lowest free address and freed IPs
immediately on release, so a just-released EIP was reassigned within ~0.6s. A
recycled EIP front-end is unreachable from the same host: the kernel still
holds the prior owner's ARP binding (or a negative cache after the neigh
flush) and OVN does not answer the re-ARP in time, so the neighbour entry
settles to FAILED (EHOSTUNREACH). This surfaced as EKS TestEKS/KubectlGetNodes
failing in single-node CI once #315 routed kubectl through the cluster NLB's
external EIP. Dev never hit it — it uses DHCP-random addresses.
Add a persisted per-pool cursor: allocation resumes one past the last address
handed out and wraps at RangeEnd, so a freed IP is not reused until the cursor
cycles the whole range. Empty cursor (fresh pool / older KV record) starts at
RangeStart, preserving lowest-free order for the first cycle — backward
compatible, no migration.
* ci: drop e2e-eks from default e2e suites
EKS data-plane subtest (TestEKS/KubectlGetNodes) is a persistent flake on
single-node CI — a freshly-attached centralised EIP is intermittently
unreachable from the host (no route to host to the cluster endpoint), which
fails the whole E2E run and blocks unrelated suites. Remove e2e-eks from the
default SUITES so PR/push runs no longer gate on it. The e2e-eks target and
the workflow_dispatch suites input are unchanged, so EKS stays runnable on
demand and via make e2e-all.
Centralised-mode EIP attach only flushed the host neighbour entry and relied
on OVN re-answering the host's ARP. On OVN builds whose ovn-controller lacks
inject-garp, no node answers a same-chassis recycled IP's re-ARP, so the entry
decays to FAILED and the external IP is unreachable ("no route to host"). This
sank the EKS e2e suite: the cluster NLB front-end EIP never resolved on br-wan,
so kubectl could not reach the published apiserver endpoint.
Mirror the distributed-mode prime: resolve the VPC gateway router port MAC (the
L2 owner of a centralised EIP on the WAN segment) and program the host
neighbour entry directly, falling back to a flush only when no MAC resolves.
* feat: un-stub EKS Addon API with managed-addon state machine
Wire the 6 EKS Addon ops (List/DescribeVersions/Create/Describe/Update/
Delete) to a per-cluster managed-addon state machine in the per-account KV.
- addon_catalog.go: static air-gapped catalog (aws-load-balancer-controller,
coredns) with version validation.
- addon_state.go: AddonRecord + AddonStatus, KV CRUD + CAS helpers; list skips
staged-manifest sub-keys; delete sweeps record + manifest.
- addon_installer.go: transport-agnostic AddonInstaller; stagingInstaller
stages a manifest descriptor in KV for the (future) VM-side delivery slice.
- addons.go: op impls + AWS-shape converters; installer failure flips
CREATE_FAILED. Status stays CREATING until delivery confirms ACTIVE (private
apiserver is host-unreachable, so delivery must be VM-side).
- service_impl.go: drop notImpl stubs; add AddonInstaller dep (nil-defaults to
staging installer).
DeleteClusterPrefix already sweeps clusters/{name}/ so addon keys clean up on
cluster delete.
* fix: route ListAddons at GET /clusters/{name}/addons
ListAddons was wired at GET /addons with no cluster param, so the AWS CLI's
GET /clusters/{name}/addons hit no route (InvalidActionException) and the
wrapper couldn't pass a cluster to the service. Match the real EKS path and
thread the cluster name through.
Recycled WAN EIPs stayed unreachable from the host (kubectl to a cluster
endpoint: "no route to host", host neigh FAILED) on OVN builds whose
ovn-controller lacks the inject-garp appctl. OVN advertises a recycled
external IP's new MAC only via the GARP it emits on LSP binding-chassis
migration; a same-chassis rebind emits none, and the neigh flush added for
recycle reachability then waits on an ARP reply no node sends.
In distributed mode the external_mac is known at attach time, so program the
host neighbour entry directly (ip neigh replace <ip> lladdr <mac> nud
reachable) instead of flushing. Inbound traffic refreshes the entry from
there. Centralized attach (no external_mac) and all detaches keep the flush.
Adds host.ReplaceNeigh + policy.NeighPrimer hook wired in vpcd alongside the
existing flusher.
Workloads need a working ingress/servicelb path before the
aws-load-balancer-controller add-on lands. Flip builtinIngressEnabled to
default ON; opt out per-cluster for AWS parity with the CreateCluster tag
spinifex.io/managed-ingress=false. Interim divergence — revert to
default-off when the controller add-on ships.
Recycled WAN external IPs stayed unreachable for the kernel ARP timeout
(60-300s) when ovn-controller lacks the inject-garp appctl: the GARP refresh
after AddEIP is then a no-op, leaving the host neighbour cache pointed at the
prior owner's MAC.
Add host.FlushNeigh (ip neigh flush) and a policy.NeighFlusher hook wired
into AddEIP and DeleteEIP, so a recycled IP re-resolves L2 against the current
owner independent of inject-garp. DeleteEIP now takes the external IP (the
vpc.delete-nat event already carries it) to flush on release.
* feat(eks): public/private endpoint access drives NLB scheme + IP:port reachability
Parse resourcesVpcConfig.endpointPublicAccess/endpointPrivateAccess/
publicAccessCidrs (defaults public=true, private=false, cidrs=0.0.0.0/0) and
reject the both-disabled combination. Public access provisions an
internet-facing NLB and publishes its external frontend IP as the cluster
endpoint; private access keeps the NLB internal and uses the VPC frontend IP.
Inject the resolved endpoint IP into the k3s apiserver tls-san so the
published IP:443 endpoint serves a cert valid for verified TLS, and surface
the access fields in DescribeCluster. The e2e suite now connects via the
verified endpoint with TLS verification on, dropping the insecure mgmt-IP
shim.
* fix(eks): authorize NLB->apiserver ingress on control-plane SG
The cluster control-plane security group was created with empty ingress, so
the NLB's backing LB VM could not reach the apiserver on 6443 from inside the
VPC. The target group stayed unhealthy and the published endpoint never
served. Authorize tcp/6443 on the control-plane SG from the VPC CIDR after the
SGs are created; public exposure remains gated at the NLB front-end by
publicAccessCidrs, so this only opens the in-VPC LB->apiserver hop.
Extend SubnetVPCResolver with GetVPCCIDR so the service can resolve the VPC
CIDR for the rule.
* feat(eks): thread publicAccessCidrs to NLB SetLoadBalancerIngressCIDRs
Wire the cluster's publicAccessCidrs (parsed from CreateCluster
ResourcesVpcConfig) through EnsureClusterNLB to the ELBv2
SetLoadBalancerIngressCIDRs override so a public cluster endpoint can be
narrowed below the wide-open scheme default. The override only fires for
an internet-facing NLB whose CIDRs restrict below 0.0.0.0/0; internal
NLBs and the default keep the LB's out-of-the-box ingress.
NLB ENIs fell back to the VPC default security group, whose intra-SG-only
ingress made the OVN ACL drop inbound listener traffic at the front-end VIP
even with a healthy target group. NLBs reject customer SGs, so mint an internal
managed SG per network LB instead: attach it to every LB ENI at create, open
each listener port on it from the resolved client CIDRs at CreateListener,
close ports on DeleteListener, and delete the SG on DeleteLoadBalancer.
Default ingress CIDRs are scheme-based (internet-facing: 0.0.0.0/0; internal:
the VPC CIDR). SetLoadBalancerIngressCIDRs overrides them and re-authorizes
existing listeners, providing the hook for EKS publicAccessCidrs. ALBs are
unchanged.
* feat(eks): publish eks-node AMI to R2 + spx catalogue entry
Add a spinifex-eks-node catalogue entry so operators install the EKS
node image with a single command:
spx admin images import --name spinifex-eks-node
The entry pulls the Alpine+K3s image from iso.mulgadc.com, verifies its
sha256, and stamps spinifex:managed-by=eks — the tag the control-plane
and node-group launchers resolve the boot AMI by, so no extra config is
needed.
Add scripts/publish-system-image.sh (+ make publish-eks-node-image) to
convert the built raw to a compressed qcow2, emit a checksum sidecar,
and upload both to the R2 bucket backing iso.mulgadc.com/system-ami via
the S3 API.
* ci(eks): add manual workflow to build + publish eks-node AMI to R2
Build the eks-node AMI on the runner (libguestfs) and publish via
publish-system-image.sh using the existing CF_R2_* secrets. Manual
workflow_dispatch with a dry-run input. Fix the publish script's default
bucket to spinifex-iso (the bucket iso.mulgadc.com is served from).
The LB system microVM stripped its module tree to virtio + fw_cfg +
failover, but init.sh only loaded qemu_fw_cfg and virtio_net. virtio_net
depends on net_failover -> failover; modprobe resolves that chain via
modules.dep, and load_mod()'s insmod-by-path fallback loads a single .ko
with no dependency resolution.
When the image is built on a kmod-less host (depmod skipped, no
modules.dep) the chain can't be resolved either way, so virtio_net fails
to load, no NIC appears ('no interface found for MAC'), the lb-agent has
no network, heartbeats never reach AWSGW and every LB hangs in
provisioning until the 5m e2e timeout. Baremetal runners ship kmod so
modules.dep is generated and the chain resolves — which is why the LB
suite passed there but failed on normal CI.
Load failover -> net_failover -> virtio_net explicitly bottom-up so the
NIC comes up regardless of whether modules.dep exists.
EKS create-cluster fails deterministically on the multinode job with
ServiceUnavailableException: the EKS KV is a clustered (Raft) JetStream
stream and create-cluster needs an elected leader, but multinode
JetStream clustering isn't reliably forming yet (see mulga-omq/z2z/5yg).
EKS is validated single-node only at this stage.
Drop eks from the multinode build+run suite filters (alongside the
existing reboot/single exclusions) so the multinode job reflects what's
actually supported. Single-node continues to run the full eks suite.
* feat(elbv2): SetSubnets — full add+remove LB subnet re-homing
Add the ELBv2 SetSubnets operation across the interface, NATS subject,
daemon handler/subscription, and gateway route/forwarder.
Re-homing is a full add+remove: subnets new to the LB get a fresh managed
ENI, dropped subnets have their ENI deleted, then the backing system VM is
relaunched on the new ENI set. The system-instance launcher exposes only
Launch/Terminate (no live ENI hotplug for system VMs), so applying new
subnets relaunches the LB VM — a brief data-plane interruption. The record
is mutated only after the new ENIs exist and the VM is relaunched.
Extract a shared launchLBVM helper from CreateLoadBalancer so create and
SetSubnets boot the LB VM identically. Idempotent on no-op; record-only
fallback when no VPC service is wired.
Completes the elbv2 mutation-API surface (SetSubnets was the last gap).
* feat(tagging): resourcegroupstaggingapi GetResources for controller discovery
Add the Resource Groups Tagging API as a new gateway service (SigV4 scope
\"tagging\", AWS JSON 1.1 / X-Amz-Target, mirroring ACM). GetResources is a
gateway-side aggregator over the existing elbv2 and ec2 tag stores — no new
NATS subject or daemon handler.
It lists the account's load balancers and target groups (attaching tags via
batched DescribeTags) plus EC2 resources from the tag store, then applies the
request's ResourceTypeFilters (service-wide or fully-qualified) and TagFilters
(AND across filters, OR within values). Results are ARN-sorted with opaque
offset pagination.
This is how aws-load-balancer-controller v2 discovers and adopts the elbv2
resources it manages (filtering on its cluster/stack tags); DescribeLoadBalancers
has no tag-filter field, so RGT GetResources is the required path.
* fix(eks): emit canonical exception code, no doubled suffix
GenerateEKSErrorResponse appended \"Exception\" unconditionally, so codes that
already carry it (e.g. awserrors.ErrorEKSResourceNotFound =
\"ResourceNotFoundException\") serialized as ResourceNotFoundExceptionException.
SDK clients comparing eks.ErrCodeResourceNotFoundException then fail to match.
Append the suffix idempotently. Also covers the many other awserrors codes that
already end in \"Exception\" and flow through the shared JSON error envelope
(EKS, ACM, tagging).
* refactor(tagging): split GetResources fetch from orchestration for testing
Introduce a resourceLister seam: GetResources wires a NATS-backed lister, while
getResources (parse + tag-filter + sort + paginate) and buildEC2Mappings (tag
grouping + ARN synthesis) are now pure and unit-tested without a live backend.
No behaviour change.
* feat(eks): per-cluster opt-in K3s built-in ingress + enforce parity default
The runtime-rendered /etc/rancher/k3s/config.yaml emitted no disable: block, so
traefik + servicelb actually ran on every cluster — the AMI config.yaml.skel
that disables them is a never-applied boot fallback (cloud-init overrides it).
Render the disable block by default so traefik + servicelb are off for AWS
parity (Service type=LoadBalancer / Ingress are the AWS Load Balancer
Controller's job). A cluster created with tag spinifex.io/managed-ingress=true
keeps the K3s built-ins for interim in-VPC app exposure before the controller
add-on lands. The flag is derived into ClusterMeta.BuiltinIngress at
CreateCluster and threaded onto K3sServerInput.
* feat(ui): subnet editor on load balancer detail page (SetSubnets)
Add a Subnets tab to the LB detail page backed by the elbv2 SetSubnets
API. The editor mirrors the security-groups editor (checkbox list of the
VPC's subnets, dirty-tracking, Reset/Save) and warns that applying a
change re-homes the load balancer by relaunching its system VM, causing
a brief data-plane interruption.
* fix(elbv2): SetSubnets must detach ENIs before delete/relaunch + roll back on terminate failure
TerminateSystemInstance tears down the LB VM but leaves its ENIs marked
in-use, so the relaunch path could neither delete removed-subnet ENIs
(InvalidNetworkInterface.InUse leak) nor re-attach retained ENIs to the
relaunched VM (attach failed InUse, dropping the retained subnet's data
plane). Detach the existing ENIs explicitly after terminate — mirroring
DeleteLoadBalancer — before deleting removed ones and relaunching.
Also roll back the freshly-created added-subnet ENIs when the terminate
step itself fails, so a failed relaunch no longer leaks ENIs (amplified
by SDK retries).
* fix(daemon): re-arm system-instance terminate subscription on recovery
system.TerminateInstance.{id} was bound only by the system.LaunchInstance
handler at first launch, so it was lost across a daemon restart: recovery
reconnects to the surviving QEMU (or relaunches) but never re-bound the
subject, leaving system.TerminateInstance.{id} with no responder. Any LB
that predated the restart could then never be torn down — SetSubnets
relaunch and DeleteLoadBalancer both failed with 'no responders'.
Re-arm it in onInstanceUpHook, which fires on both the relaunch and the
reconnect-to-surviving-QEMU recovery paths, for ManagedByELBv2 VMs; drop
it in onInstanceDownHook. Split subscribeSystemTerminate into a locked
helper so the hook (which already holds d.mu) can reuse it.
Make e2e-baremetal.yml a reusable workflow (workflow_call + a ref input)
so the nightly and release pipelines can invoke the real PXE-reimaged
hardware suite, which previously only ran on manual dispatch.
Nightly: add a baremetal job (conceptual cell 20) that runs every
weeknight on the existing cron, or on dispatch when 20 is selected. It
tests dev on schedule, matching GIT_BRANCH. Sits outside the tofu matrix
(different runner, provisioning, 120m timeout) and shares the
e2e-baremetal-cluster concurrency group, so all baremetal runs serialise
on the single physical node.
Release: a release builds and uploads an ISO to R2 but never booted that
ISO/squashfs path on hardware. Add a baremetal job (needs: release) and
hard-gate on it: promotion moves to a dedicated promote job
needs:[verify, baremetal] (with a matching mark-failed job on failure),
and the iso job's upload gate now waits for both verify and baremetal to
conclude success before pushing to R2. Bump the iso job timeout to 180m
since it now waits on the parallel ~2h baremetal run.
build-microvm-image required depmod (kmod) as a mandatory tool to
generate modules.dep. Self-hosted CI runners don't ship kmod and can't
reliably apt-install it, so the inline 'apt-get install kmod' steps added
to e2e/e2e-nightly/e2e-ddil were unreliable.
modules.dep is an optimisation, not a boot requirement: the guest init's
load_mod() (build/microvm/init.sh) falls back to insmod-by-path when
modprobe can't resolve a module. So:
- Drop depmod from the mandatory-tool preflight.
- Run depmod only when present; still assert a non-empty modules.dep in
that case to keep catching the trixie-kmod empty-dep bug.
- Warn and skip when depmod is absent (guest uses insmod fallback).
- Remove the apt-get kmod install steps from all three workflows.
Gold-image bake still installs kmod (provision.sh), so production images
get a proper modules.dep; only kmod-less CI runners take the fallback.
* ci(e2e): install kmod (depmod) before microVM image build
#305 made depmod a hard requirement in build-microvm-image.sh, but the
self-hosted ci-single/ci-multi runners don't ship the kmod package, so the
Build Install Artifacts step aborts with 'required tool not found: depmod'
before any E2E suite runs. Add an idempotent kmod install ahead of each
Build Install Artifacts step (single/multi/eks jobs); the gold-image builder
bakes kmod in too, so this no-ops once runners pick up the new image.
* ci: install kmod (depmod) in nightly + ddil too
e2e-nightly and e2e-ddil call the same build-install-artifacts.sh (-> make
build-microvm-image) on the same self-hosted ci-single/ci-multi runners as
e2e.yml, so they hit the identical 'depmod not found' abort. Add the same
idempotent kmod guard before Build Install Artifacts in both.
release distro builds run on GitHub-hosted ubuntu (kmod present); e2e-baremetal
runs on ci-baremetal which already has it (green today) — neither needs the
guard.
* feat(elbv2): localize internal-LB heartbeat failures for baremetal debug
Add diagnostics so a stuck-in-provisioning internal LB can be pinned to
one of three points on the heartbeat path:
- lb-agent failure log now carries the gateway target + region, so the
microVM serial console shows whether the heartbeat hit a transport
error (never reached AWSGW) or an HTTP-level rejection.
- AWSGW LBAgentHeartbeat logs receipt (debug) and warns on the
previously-silent not-found/account-mismatch reject path, so the
journal distinguishes 'heartbeat arrived but cannot transition' from
'heartbeat never arrived'.
* test(e2e/lb): fail fast on stuck internal LB + probe fw_cfg on microvm boot
- lb suite: gate the remaining internal subtests on Internal_ALB. The
internal LBs share one mgmt-return datapath and fixture, so if the
first never reaches active the rest time out identically — skip them
instead of burning ~5m per subtest (~20m saved on a known-bad run).
- microvm init.sh: widen the fw_cfg netcfg poll window and, on the
not-available path, dump DMI / loaded modules / qemu_fw_cfg by_name
listing / dmesg fw_cfg to the serial console before init exits, so a
boot panic is diagnosable from the captured console log.
* fix(microvm): insmod fallback for fw_cfg + fail build on bad modules.dep
Baremetal-built microVM images shipped an unusable modules.dep (the build
host's depmod errors were swallowed), so busybox modprobe in the guest init
silently no-oped, qemu_fw_cfg never loaded, /sys/firmware/qemu_fw_cfg was
absent, init could not read opt/spinifex/netcfg, and the LB microVM panicked
before lb-agent ran — leaving internal LBs stuck in provisioning.
init.sh: load_mod() tries modprobe, then falls back to insmod by explicit
path (.ko/.ko.gz) which needs no modules.dep, so the guest boots regardless
of which host built the image.
build-microvm-image.sh: require depmod, drop the error swallow, and assert
modules.dep is non-empty after depmod so a boot-broken image fails the build
loudly instead of silently.
* fix(microvm): decompress kernel modules before depmod
Debian trixie's kmod 34 depmod silently emits an empty modules.dep when fed
Alpine's gzip-compressed (.ko.gz) modules — reproduced: 928 modules in, 0-byte
modules.dep out. The guest's busybox modprobe then no-ops, qemu_fw_cfg never
loads, and internal LB microVMs panic before lb-agent runs (stuck provisioning
on bare metal; GitHub-hosted runners ship a kmod that reads .gz, so normal CI
was unaffected).
Decompress kept modules to plain .ko after the keep-list strip, before depmod.
Host depmod then produces a usable map regardless of its gzip support, and the
guest loads modules without a decompressor.
* feat(e2e): capacity-aware scheduling for dense single-node runs
Denominate daemon schedulable capacity in physical cores instead of
logical threads, and auto-serialise e2e launches when the cluster is at
capacity, so a single hyperthreaded node stops over-packing guests.
Daemon: NewResourceManager sizes hostVCPU from physicalCoreCount()
(distinct physical-id:core-id pairs in /proc/cpuinfo) rather than
runtime.NumCPU(). Falls back to logical CPUs when topology is absent
(non-Linux, container, ARM cpuinfo without core id). On a hyperthreaded
box this halves the over-pack: with the unchanged 2 vCPU reserve the
i5-12600H schedules 10 vCPU (12 cores - 2) instead of 14, keeping the
boot-spike CPU within real cores.
Harness: capacityRetryEC2 decorates the EC2 client and, on
InsufficientInstanceCapacity, waits and retries RunInstances instead of
failing — turning the daemon admission gate into back-pressure so
concurrent subtests queue. AWSClient.EC2 is now the wrapped
ec2iface.EC2API; EC2Conf keeps the raw *ec2.EC2 for .Config access.
Adds unit tests for parsePhysicalCores (hyperthreaded/hybrid/dual-socket/
fallback) and the retry decorator (retry-then-succeed, non-capacity
passthrough, deadline).
* fix(daemon): SPINIFEX_HOST_VCPU override so small CI hosts can start
Denominating capacity by physical cores starves tiny hosts: a 2-physical-
core CI runner leaves 0 schedulable after the default 2-vCPU reserve, so
NewResourceManager refuses to start and every daemon-constructing unit test
fails. The previous runtime.NumCPU() count (4 logical on the same runner)
incidentally cleared the reserve floor.
Add a SPINIFEX_HOST_VCPU escape hatch (symmetric with the existing
SPINIFEX_RESERVED_* overrides) that pins the detected core count for hosts
whose /proc/cpuinfo topology misreports, and pin it to 4 in the daemon
TestMain so unit tests are deterministic regardless of runner topology.
Reserve stays at the default 2.
* test(eks): EKS control-plane e2e suite (Phase 1, API-level)
Add tests/e2e/eks/eks_test.go driving the EKS lifecycle against awsgw:
CreateCluster -> ACTIVE (endpoint + CA asserted), kubeconfig artifact
assembly, AccessEntry CRUD, get-token presign decode, DeleteCluster -> gone.
Harness: EKS SDK client on AWSClient + WaitForEKSCluster{Active,Deleted}
pollers. Wire e2e-eks into the Makefile and the single-node CI suite list.
* test(eks): Phase 2 kubectl get-nodes data-plane gate
Add the reachability shim (harness/eks_mgmt.go) that reads the control-plane
VM's br-mgmt IP from the cluster KV meta over NATS, plus a kubectl wrapper
(harness/kubectl.go) that skips when kubectl is absent. KubectlGetNodes points
kubectl at https://<mgmtIP>:6443 (insecure-skip-tls-verify; cert SANs the NLB
DNS, not the mgmt IP) with an aws eks get-token exec block and polls until the
control-plane node is Ready — the get-token webhook auth regression gate.
IRSA pod + the real NLB/Route53 passthrough are deferred to a follow-up.
* test(eks): tolerate doubled NotFound code + dial loopback NATS
DeleteCluster detection matched the SDK constant ResourceNotFoundException,
but awsgw emits ResourceNotFoundExceptionException on the wire — match the
NotFound code by substring so deletion is detected immediately instead of
timing out. The mgmt-IP shim dialed NATS.Host verbatim (0.0.0.0:4222), which
the serving cert does not SAN; rewrite the wildcard bind to loopback.
Validated live: all 6 subtests green, kubectl get nodes shows the
control-plane node Ready via the get-token webhook auth path.
* feat: build system images with libguestfs (no host block devices)
Replace the qemu-nbd + sudo mount + chroot + /dev-bind flow in
build-system-image.sh with libguestfs (virt-customize / virt-resize /
guestfish). Every operation runs in an isolated userspace appliance VM with
its own kernel, so the build touches no host block device, no host mount, and
needs no sudo — removing the failure class that wiped the host's /dev on shared
CI runners (a leaked /dev bind traversed by rm -rf).
Applies to all distros: alpine resizes via guestfish e2fsck+resize2fs, ubuntu
via virt-resize; customization (packages, binaries, files, setup script,
service enable, cache cleanup) is one ordered virt-customize invocation.
LIBGUESTFS_BACKEND=direct (no libvirt on runners).
Wire the eks-node AMI into the single-node e2e job: the runner builds the
device-free raw, ships it to the VM, and the VM imports it (predastore/NATS are
VM-only) so the eks suite's CreateCluster has an AMI to boot.
* feat(eks): bootstrap + state-report via AWS gateway broker
Converge the EKS control-plane → host channel onto the ELBv2 lb-agent model:
the K3s server VM publishes its bootstrap envelopes and periodic state reports
via SigV4-signed HTTPS POST to the AWS gateway, instead of dialing core NATS
directly. A new gateway action (PublishInternal, POST
/clusters/{name}/internal-publish) relays the payload verbatim onto the
existing eks.bus.* / eks.state.* subjects, so the daemon-side subscriber, KV
persistence/encryption, and reconciler are unchanged.
This removes the VM's need to reach NATS at all, so NATS can stay
cluster-internal behind its bind/firewall/mTLS controls — the prior direct
path broke when NATS was not VM-reachable (e2e CreateCluster timeout) and was
DDIL-fragile (fire-and-forget nats pub, no ack/retry).
- gateway/eks: PublishInternal broker + route; closed-set channel/kind validation
- cmd/eks-gateway-publish: one-shot SigV4 publisher (retry-until-ack) baked into
the eks-node AMI; replaces the nats-cli publisher in k3s-first-boot.sh and
mulga-eks-state-report.sh
- handlers/eks + daemon: hand the K3s server VM the mgmt-reachable gateway URL +
system (Predastore) SigV4 creds + gateway CA instead of NATS URL/token/CA;
new resolveSystemGatewayBaseURL (mgmt-bridge-first)
- eks-node image: build/install eks-gateway-publish, drop nats-cli download
* feat: migrate eks-token-webhook to the AWS gateway broker
The TokenReview webhook dialed core NATS directly (STS verify + AccessEntry
KV read). On a mgmt-only control-plane VM with NATS bound off-subnet that
fails, so the webhook exited at startup, the apiserver token-webhook
kubeconfig was never written, and the apiserver crash-looped — the cluster
never reached ACTIVE.
Converge it onto the same gateway-broker model as bootstrap/state-report and
ELBv2's lb-agent: the webhook SigV4-signs an HTTPS POST {accountId,token} to
/clusters/{name}/token-review, and the gateway runs the verify + lookup
host-side and returns the resolved identity. The VM now speaks no core NATS.
- internal/eksgw: shared on-VM SigV4 client (also adopted by
eks-gateway-publish).
- handlers/eks/token_review.go: Authenticate decision core (moved from the
webhook binary) + ResolveTokenReview host-side wiring.
- gateway/eks token-review route + WebhookTokenReview handler.
- webhook: drop NATS, relay via eksgw.Client; broker fault -> 503 (retryable)
not a silent deny.
- initd: export EKS_GATEWAY_* / EKS_ACCESS_* creds, not SPINIFEX_NATS_*.
* fix: build eks-node helpers with GOFIPS140; cover token-review broker
eks-gateway-publish blank-imports internal/fipsboot, whose init() panics
unless the binary is built with GOFIPS140=v1.0.0. The eks-node AMI build
omitted the flag, so the binary paniced at first boot right after k3s came
up — k3s-first-boot (set -eu) died before publishing the bootstrap token and
the cluster never reached ACTIVE. Build both eks-node helpers with
GOFIPS140=v1.0.0, matching cmd/lb-agent.
Add NATS-backed unit tests for ResolveTokenReview and the WebhookTokenReview
gateway handler (verify responder + account KV) to cover the new broker path.
* fix: export first-boot env so eks-gateway-publish child sees it
k3s-first-boot.sh and mulga-eks-state-report.sh sourced /etc/spinifex-eks/
first-boot.env but never exported the vars, so the eks-gateway-publish child
process inherited an empty environment and exited '--account-id is required'.
The control-plane VM came up healthy but never published its bootstrap
envelopes, so the cluster never reached ACTIVE. Wrap the source in set -a /
set +a so the KEY=value lines export to child processes.
* fix: make awsgw start --host override actually bind
The 'awsgw start --host' override printed 'Overwriting awsgw host' but the
assignment was commented out, so AWSGW always bound the spinifex.toml host
(a specific WAN IP, e.g. 192.168.0.10). A system instance (EKS K3s control
plane, internal LB agent) has a mgmt-only NIC on-link to the br-mgmt IP with
no route to the WAN IP, so its SigV4 POSTs to the gateway were refused and the
cluster never bootstrapped. Honour --host (e2e passes 0.0.0.0:9999) so AWSGW
listens on all interfaces incl. br-mgmt; the serving cert already SANs every
local interface IP, and WAN clients still reach it.
* test(eks): install kubectl on e2e VM so KubectlGetNodes runs
The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full
aws eks get-token -> kubectl -> apiserver -> token-webhook -> gateway broker
-> STS verify path — the only end-to-end exercise of the token-review broker.
The e2e binaries run on the VM, where kubectl was absent, so
harness.NewKubectl t.Skip'd it and the path went unvalidated. Install a
checksum-verified kubectl (pinned to the k3s server version) on the VM,
gated on e2e-eks.
Internal-scheme LBs on single-node have no dedicated mgmt-bind IP on
br-mgmt, so resolveMgmtRoute falls back to routing the agent to the
advertised WAN IP via the mgmt NIC. That advertised IP lives on a
different bridge than the agent's mgmt NIC, and the cross-bridge return
path never delivers — the agent never heartbeats and the LB hangs in
provisioning (normal single-node dodges this entirely via dev_networking
slirp at 10.0.2.2; bare-metal external-mode is the first real exercise of
this path).
Point the agent at the br-mgmt host IP, which shares its mgmt L2 — no
cross-bridge hop — mirroring the multi-node datapath. Requires AWSGW to
listen on br-mgmt (bind 0.0.0.0); the server cert already SANs that IP.
Internal-scheme LBs only leave provisioning once the lb-agent heartbeats
AWSGW, which requires the mgmt return route resolved by resolveMgmtRoute
(MgmtRoute{Gateway,Target}, or MgmtBridgeIP+AdvertiseIP on single-node).
When none resolve, the route was silently skipped and the LB hung in
provisioning until the client timed out, with no signal as to why.
Log which precondition is empty on the internal skip path, and mark the
LB failed instead of provisioning when no return route exists, so the
broken data-plane surfaces immediately.
InitBuckets was passed the OVN chassis count as the replica factor. When the
chassis count exceeds the NATS node count (single-node bare-metal: 1 node but
2 SBDB chassis), CreateKeyValue requested more replicas than JetStream could
satisfy, the create-or-open fallback returned "bucket not found", and vpcd
crash-looped — cascading to RunInstances ServerInternal. Create at 1 replica
like STS/IAM; the daemon's deferred upgradeJetStreamReplicas raises all KV_*
streams to cluster size when the cluster grows.
* feat(eks): webhook observability — console log sink + per-request TokenReview decision
* fix(eks): wire token-webhook authenticator into generated k3s apiserver config
The generated /etc/rancher/k3s/config.yaml overrides the AMI-baked
config.yaml.skel, but its kube-apiserver-arg list omitted the
authentication-token-webhook-config-file arg. The running apiserver
therefore never loaded the token webhook, so bearer tokens minted by
'aws eks get-token' 401'd without the webhook ever being invoked.
* fix(eks): echo TokenReview audiences so api-audiences apiserver accepts the principal
With --api-audiences set, kube-apiserver sends spec.audiences in the
TokenReview and discards an authenticated=true response that does not
confirm a matching audience. The webhook returned an empty
status.audiences, so a valid IAM principal surfaced as a 401. Echo the
request's audiences on success.
* fix(eks): pin token-webhook version v1 so apiserver accepts the response
kube-apiserver --authentication-token-webhook-version defaults to
v1beta1, but the eks-token-webhook responds with an
authentication.k8s.io/v1 TokenReview. The apiserver decodes the response
against the expected GVK and rejects the mismatch, surfacing a valid
principal as a 401. Pin v1 to match the webhook.
Bare-metal e2e produced no artifact, so RCA fell back to truncated
log-failed output. Tee bootstrap + e2e output to test-baremetal.log,
generate junit-baremetal.xml, run e2e-analyze to emit analysis.md, and
upload the bundle as e2e-baremetal-artifacts-<run_id>.
aws eks list-clusters returned InternalErrorException for a few seconds after a
daemon restart, while describe-cluster (by name) succeeded. During the
post-restart window the per-account KV stream's Raft group has no leader yet, so
kv.Keys() (an ordered consumer over the warming stream) fails with a transient
NATS error. ListClusters wrapped it as a fmt.Errorf string the daemon's
ValidErrorCode cannot map, so it sanitized to ServerInternal — a misleading 500
that tells AWS clients the request is non-retryable.
Classify transient NATS conditions (no-responders, timeout, no-stream-response,
connection-closed) and return a retryable ServiceUnavailable (503) instead;
genuine faults still pass through wrapped. describe-cluster survived only
because kv.Get on the same bucket is cheaper than the consumer kv.Keys spins up.
* feat(iam,awsgw): OIDC provider CRUD + IRSA discovery endpoints (6e)
Complete the AWS-SDK-facing half of IRSA:
- IAM Create/Get/List/DeleteOpenIDConnectProvider, backed by the per-account
oidc-providers registry that STS AssumeRoleWithWebIdentity already reads to
authorise an issuer. Create writes the key STS checks; the provider ARN is
the scheme-stripped issuer per AWS convention.
- awsgw serves the per-cluster OIDC discovery document
(/.well-known/openid-configuration) and JWKS (/keys) as unauthenticated
public endpoints, mounted ahead of the SigV4 surface in their own route
group. The discovery issuer is the cluster's stored OIDCIssuer, byte-exact
with the apiserver iss claim; JWKS is served from the EKS account bucket.
Unit + route tests cover CRUD happy/idempotent/not-found/invalid-url, the
ARN<->issuer round-trip, discovery issuer/jwks_uri equality, JWKS bytes, 404s,
and the no-auth bypass. make preflight green.
* fix(awsgw): route anonymous AssumeRoleWithWebIdentity past SigV4 (6e)
AssumeRoleWithWebIdentity is the IRSA credential-bootstrap call: the AWS SDK
sends it unsigned because the caller holds no AWS credentials, only a projected
ServiceAccount JWT in the body. The gateway ran the SigV4 middleware first and
rejected it with MissingAuthenticationToken, so the true anonymous path never
reached the handler.
Add anonymousSTSInterceptor, a pre-auth middleware mounted ahead of the SigV4
surface. For an unsigned POST it body-peeks the Action and, for the
anonymousSTSActions set, dispatches straight to STS_Request; STS_Request now
skips SigV4-caller resolution for those actions (the handler authenticates the
JWT and gates on the role's web-identity trust policy). Signed requests carry an
Authorization header and pass through untouched, off the body-buffering path.
Surfaced by the 6e live exit gate: a real in-pod assume-role-with-web-identity
would otherwise have been rejected before reaching token validation.
* feat(eks): nodegroup provisioning + K3s agent join (6d)
CreateNodegroup launches N customer-owned worker EC2 from the unified
eks-node AMI in agent role; each joins the cluster via K3S_URL + K3S_TOKEN
baked into cloud-init (first-boot.env SPINIFEX_K3S_ROLE=agent + agent.env).
Workers are customer-owned and DescribeInstances-visible (no ManagedBy,
no mgmt NIC) via the normal RunInstances path, matching real EKS.
K3S_URL targets the control-plane ENI's in-VPC private IP on :6443 — the
cluster NLB DNS endpoint is not resolvable in-VPC yet; switch to the NLB
endpoint on :443 once authoritative in-VPC DNS lands. K3S_TOKEN is the
decrypted cluster node-token persisted at server bootstrap.
Describe/List/Update(scaling)/Delete round out CRUD. Cluster<->nodegroup
SG ingress rules (apiserver 6443, flannel VXLAN 8472/udp, kubelet 10250,
intra-nodegroup) authorized idempotently. UpdateNodegroupVersion deferred.
* feat: gate EKS cluster health via NATS state reports
The K3s apiserver binds the VPC node-ip, unreachable from the host-side
spinifex daemon, so the reconciler cannot probe /healthz directly. Add a
control-plane state reporter that publishes {healthz, node_count} over the
management NATS bus the daemon already shares.
- eks-node server role runs mulga-eks-state-report as an OpenRC service,
publishing eks.state.{account}.{cluster}.server every 30s.
- ClusterReconciler subscribes to that subject, gates CREATING -> ACTIVE on
a healthy report, and records Ready node_count into ClusterMeta.
- node_count counts Ready nodes only; k3s retains terminated workers as
NotReady until pruned, so a raw count never drops on scale-down.
The iso job mounts a kernel-global devtmpfs under the workspace
(.cache/rootfs/dev). A killed job skips its EXIT-trap unmount and leaks
the bind; the verify job (same ci-single runner) then ran a bare
'rm -rf $WORKSPACE/*' that could traverse it and wipe the host /dev.
Detach any mount under the workspace before deleting and add
--one-file-system so rm never crosses a mount boundary. Same backstop
added to the iso job's own cleanup.
A killed ubuntu system-image build can leave `mount --bind /dev` under
its build dir. The e2e Clean Workspace step then ran
`sudo rm -rf "$GITHUB_WORKSPACE"/*` as root, traversed the leaked bind,
and deleted the host's /dev (incl. /dev/null), wedging the runner.
- e2e.yml: umount anything under the workspace (deepest-first) before
deleting, and pass `rm --one-file-system` so rm can never cross a mount
boundary into host /dev. Hard backstop, applied to all 3 Clean Workspace
steps.
- build-system-image.sh: recursive lazy unmount in cleanup() detaches the
chroot binds even when busy, and mark the build subtree rprivate so the
/dev bind can't propagate into the host namespace.
* fix: switch EKS datastore to embedded etcd + fix Alpine cloud-init IMDS route
Flip the K3s control-plane datastore from SQLite/kine back to embedded etcd
(cluster-init: true, etcd-expose-metrics: true). The earlier abandonment
premise — that etcd's per-commit fsync stalls past the apiserver's 5s deadline
on the viperblock-backed root volume — was wrong: etcd inits fine on an
unchanged viperblock (apiserver ready 25-30s, clusters reach ACTIVE). The real
blocker was a network-config regression, fixed here.
Alpine's cloud-init eni renderer crashes (KeyError: 'gateway') on the
gateway-less IMDS on-link route, which failed the whole network-config render
so the static mgmt NIC never came up (no NATS bootstrap, no /healthz). Fixes:
- generateNetworkConfig: add emitIMDSRoute bool; selectNetworkConfigForFamily
passes false for Alpine so the eni renderer never sees the route.
- Deliver the IMDS route out-of-band via a persistent /etc/local.d script +
OpenRC local service. Resolve the VPC NIC via its default route at runtime
(ip route show default), not a hardcoded name: the persistent-net rename to
vpc0 doesn't apply on the Alpine AMI, so the NIC is eth0 at first boot.
- EKS path bakes the same route into buildK3sUserData's own write_files/runcmd;
the generic template Alpine block fires only when no #cloud-config user-data,
avoiding a duplicate write_files key (yaml.safe_load keeps last, silently
dropping a block).
- eks-node image: config.yaml.skel adds cluster-init; comments updated.
* fix: surface CreateCluster step failures at ERROR + map subnet faults; add auth-mode/delete test cases
CreateCluster wrapped its step failures in bare fmt.Errorf that the request
surface collapses to an opaque ServerInternal ("contact AWS re:Post"), so the
real cause never reached the operator. Add logCreateErr to log each step's
precise cause at ERROR before returning, and map a subnet-resolve failure
(caller-supplied parameter) to InvalidParameterValue instead of a retryable
ServerInternal the SDK pointlessly retries. (mulga-siv-165.4)
Tests (mulga-siv-165.15): reject authenticationMode=API_AND_CONFIG_MAP (only
the CONFIG_MAP value was covered); DeleteCluster on a nonexistent cluster via
the full-deps path returns ResourceNotFound (the shim path short-circuits to
ServiceUnavailable before the meta lookup).
* fix: hide EKS control-plane VMs from customer DescribeInstances
EKS K3s control-plane VMs are owned by the customer account (their ENI
lives in the customer VPC for the NLB target), so unlike LB microVMs
(owned by the system account) they leaked into customer
ec2:DescribeInstances. The spinifex:managed-by tag only filtered the UI
Nodes page, not the EC2 API.
Add a caller-aware exclusion: skip any ManagedBy-tagged VM unless the
caller is root. Preserves operator visibility, matches LB behaviour, and
leaves future customer-owned worker nodes (no ManagedBy) visible.
* docs: correct stale SQLite datastore comments in eks-node image (now embedded etcd)
Julian Sommer