EKS create-cluster fails deterministically on the multinode job with
ServiceUnavailableException: the EKS KV is a clustered (Raft) JetStream
stream and create-cluster needs an elected leader, but multinode
JetStream clustering isn't reliably forming yet (see mulga-omq/z2z/5yg).
EKS is validated single-node only at this stage.
Drop eks from the multinode build+run suite filters (alongside the
existing reboot/single exclusions) so the multinode job reflects what's
actually supported. Single-node continues to run the full eks suite.
* feat(elbv2): SetSubnets — full add+remove LB subnet re-homing
Add the ELBv2 SetSubnets operation across the interface, NATS subject,
daemon handler/subscription, and gateway route/forwarder.
Re-homing is a full add+remove: subnets new to the LB get a fresh managed
ENI, dropped subnets have their ENI deleted, then the backing system VM is
relaunched on the new ENI set. The system-instance launcher exposes only
Launch/Terminate (no live ENI hotplug for system VMs), so applying new
subnets relaunches the LB VM — a brief data-plane interruption. The record
is mutated only after the new ENIs exist and the VM is relaunched.
Extract a shared launchLBVM helper from CreateLoadBalancer so create and
SetSubnets boot the LB VM identically. Idempotent on no-op; record-only
fallback when no VPC service is wired.
Completes the elbv2 mutation-API surface (SetSubnets was the last gap).
* feat(tagging): resourcegroupstaggingapi GetResources for controller discovery
Add the Resource Groups Tagging API as a new gateway service (SigV4 scope
\"tagging\", AWS JSON 1.1 / X-Amz-Target, mirroring ACM). GetResources is a
gateway-side aggregator over the existing elbv2 and ec2 tag stores — no new
NATS subject or daemon handler.
It lists the account's load balancers and target groups (attaching tags via
batched DescribeTags) plus EC2 resources from the tag store, then applies the
request's ResourceTypeFilters (service-wide or fully-qualified) and TagFilters
(AND across filters, OR within values). Results are ARN-sorted with opaque
offset pagination.
This is how aws-load-balancer-controller v2 discovers and adopts the elbv2
resources it manages (filtering on its cluster/stack tags); DescribeLoadBalancers
has no tag-filter field, so RGT GetResources is the required path.
* fix(eks): emit canonical exception code, no doubled suffix
GenerateEKSErrorResponse appended \"Exception\" unconditionally, so codes that
already carry it (e.g. awserrors.ErrorEKSResourceNotFound =
\"ResourceNotFoundException\") serialized as ResourceNotFoundExceptionException.
SDK clients comparing eks.ErrCodeResourceNotFoundException then fail to match.
Append the suffix idempotently. Also covers the many other awserrors codes that
already end in \"Exception\" and flow through the shared JSON error envelope
(EKS, ACM, tagging).
* refactor(tagging): split GetResources fetch from orchestration for testing
Introduce a resourceLister seam: GetResources wires a NATS-backed lister, while
getResources (parse + tag-filter + sort + paginate) and buildEC2Mappings (tag
grouping + ARN synthesis) are now pure and unit-tested without a live backend.
No behaviour change.
* feat(eks): per-cluster opt-in K3s built-in ingress + enforce parity default
The runtime-rendered /etc/rancher/k3s/config.yaml emitted no disable: block, so
traefik + servicelb actually ran on every cluster — the AMI config.yaml.skel
that disables them is a never-applied boot fallback (cloud-init overrides it).
Render the disable block by default so traefik + servicelb are off for AWS
parity (Service type=LoadBalancer / Ingress are the AWS Load Balancer
Controller's job). A cluster created with tag spinifex.io/managed-ingress=true
keeps the K3s built-ins for interim in-VPC app exposure before the controller
add-on lands. The flag is derived into ClusterMeta.BuiltinIngress at
CreateCluster and threaded onto K3sServerInput.
* feat(ui): subnet editor on load balancer detail page (SetSubnets)
Add a Subnets tab to the LB detail page backed by the elbv2 SetSubnets
API. The editor mirrors the security-groups editor (checkbox list of the
VPC's subnets, dirty-tracking, Reset/Save) and warns that applying a
change re-homes the load balancer by relaunching its system VM, causing
a brief data-plane interruption.
* fix(elbv2): SetSubnets must detach ENIs before delete/relaunch + roll back on terminate failure
TerminateSystemInstance tears down the LB VM but leaves its ENIs marked
in-use, so the relaunch path could neither delete removed-subnet ENIs
(InvalidNetworkInterface.InUse leak) nor re-attach retained ENIs to the
relaunched VM (attach failed InUse, dropping the retained subnet's data
plane). Detach the existing ENIs explicitly after terminate — mirroring
DeleteLoadBalancer — before deleting removed ones and relaunching.
Also roll back the freshly-created added-subnet ENIs when the terminate
step itself fails, so a failed relaunch no longer leaks ENIs (amplified
by SDK retries).
* fix(daemon): re-arm system-instance terminate subscription on recovery
system.TerminateInstance.{id} was bound only by the system.LaunchInstance
handler at first launch, so it was lost across a daemon restart: recovery
reconnects to the surviving QEMU (or relaunches) but never re-bound the
subject, leaving system.TerminateInstance.{id} with no responder. Any LB
that predated the restart could then never be torn down — SetSubnets
relaunch and DeleteLoadBalancer both failed with 'no responders'.
Re-arm it in onInstanceUpHook, which fires on both the relaunch and the
reconnect-to-surviving-QEMU recovery paths, for ManagedByELBv2 VMs; drop
it in onInstanceDownHook. Split subscribeSystemTerminate into a locked
helper so the hook (which already holds d.mu) can reuse it.
Make e2e-baremetal.yml a reusable workflow (workflow_call + a ref input)
so the nightly and release pipelines can invoke the real PXE-reimaged
hardware suite, which previously only ran on manual dispatch.
Nightly: add a baremetal job (conceptual cell 20) that runs every
weeknight on the existing cron, or on dispatch when 20 is selected. It
tests dev on schedule, matching GIT_BRANCH. Sits outside the tofu matrix
(different runner, provisioning, 120m timeout) and shares the
e2e-baremetal-cluster concurrency group, so all baremetal runs serialise
on the single physical node.
Release: a release builds and uploads an ISO to R2 but never booted that
ISO/squashfs path on hardware. Add a baremetal job (needs: release) and
hard-gate on it: promotion moves to a dedicated promote job
needs:[verify, baremetal] (with a matching mark-failed job on failure),
and the iso job's upload gate now waits for both verify and baremetal to
conclude success before pushing to R2. Bump the iso job timeout to 180m
since it now waits on the parallel ~2h baremetal run.
build-microvm-image required depmod (kmod) as a mandatory tool to
generate modules.dep. Self-hosted CI runners don't ship kmod and can't
reliably apt-install it, so the inline 'apt-get install kmod' steps added
to e2e/e2e-nightly/e2e-ddil were unreliable.
modules.dep is an optimisation, not a boot requirement: the guest init's
load_mod() (build/microvm/init.sh) falls back to insmod-by-path when
modprobe can't resolve a module. So:
- Drop depmod from the mandatory-tool preflight.
- Run depmod only when present; still assert a non-empty modules.dep in
that case to keep catching the trixie-kmod empty-dep bug.
- Warn and skip when depmod is absent (guest uses insmod fallback).
- Remove the apt-get kmod install steps from all three workflows.
Gold-image bake still installs kmod (provision.sh), so production images
get a proper modules.dep; only kmod-less CI runners take the fallback.
* ci(e2e): install kmod (depmod) before microVM image build
#305 made depmod a hard requirement in build-microvm-image.sh, but the
self-hosted ci-single/ci-multi runners don't ship the kmod package, so the
Build Install Artifacts step aborts with 'required tool not found: depmod'
before any E2E suite runs. Add an idempotent kmod install ahead of each
Build Install Artifacts step (single/multi/eks jobs); the gold-image builder
bakes kmod in too, so this no-ops once runners pick up the new image.
* ci: install kmod (depmod) in nightly + ddil too
e2e-nightly and e2e-ddil call the same build-install-artifacts.sh (-> make
build-microvm-image) on the same self-hosted ci-single/ci-multi runners as
e2e.yml, so they hit the identical 'depmod not found' abort. Add the same
idempotent kmod guard before Build Install Artifacts in both.
release distro builds run on GitHub-hosted ubuntu (kmod present); e2e-baremetal
runs on ci-baremetal which already has it (green today) — neither needs the
guard.
* feat(elbv2): localize internal-LB heartbeat failures for baremetal debug
Add diagnostics so a stuck-in-provisioning internal LB can be pinned to
one of three points on the heartbeat path:
- lb-agent failure log now carries the gateway target + region, so the
microVM serial console shows whether the heartbeat hit a transport
error (never reached AWSGW) or an HTTP-level rejection.
- AWSGW LBAgentHeartbeat logs receipt (debug) and warns on the
previously-silent not-found/account-mismatch reject path, so the
journal distinguishes 'heartbeat arrived but cannot transition' from
'heartbeat never arrived'.
* test(e2e/lb): fail fast on stuck internal LB + probe fw_cfg on microvm boot
- lb suite: gate the remaining internal subtests on Internal_ALB. The
internal LBs share one mgmt-return datapath and fixture, so if the
first never reaches active the rest time out identically — skip them
instead of burning ~5m per subtest (~20m saved on a known-bad run).
- microvm init.sh: widen the fw_cfg netcfg poll window and, on the
not-available path, dump DMI / loaded modules / qemu_fw_cfg by_name
listing / dmesg fw_cfg to the serial console before init exits, so a
boot panic is diagnosable from the captured console log.
* fix(microvm): insmod fallback for fw_cfg + fail build on bad modules.dep
Baremetal-built microVM images shipped an unusable modules.dep (the build
host's depmod errors were swallowed), so busybox modprobe in the guest init
silently no-oped, qemu_fw_cfg never loaded, /sys/firmware/qemu_fw_cfg was
absent, init could not read opt/spinifex/netcfg, and the LB microVM panicked
before lb-agent ran — leaving internal LBs stuck in provisioning.
init.sh: load_mod() tries modprobe, then falls back to insmod by explicit
path (.ko/.ko.gz) which needs no modules.dep, so the guest boots regardless
of which host built the image.
build-microvm-image.sh: require depmod, drop the error swallow, and assert
modules.dep is non-empty after depmod so a boot-broken image fails the build
loudly instead of silently.
* fix(microvm): decompress kernel modules before depmod
Debian trixie's kmod 34 depmod silently emits an empty modules.dep when fed
Alpine's gzip-compressed (.ko.gz) modules — reproduced: 928 modules in, 0-byte
modules.dep out. The guest's busybox modprobe then no-ops, qemu_fw_cfg never
loads, and internal LB microVMs panic before lb-agent runs (stuck provisioning
on bare metal; GitHub-hosted runners ship a kmod that reads .gz, so normal CI
was unaffected).
Decompress kept modules to plain .ko after the keep-list strip, before depmod.
Host depmod then produces a usable map regardless of its gzip support, and the
guest loads modules without a decompressor.
* feat(e2e): capacity-aware scheduling for dense single-node runs
Denominate daemon schedulable capacity in physical cores instead of
logical threads, and auto-serialise e2e launches when the cluster is at
capacity, so a single hyperthreaded node stops over-packing guests.
Daemon: NewResourceManager sizes hostVCPU from physicalCoreCount()
(distinct physical-id:core-id pairs in /proc/cpuinfo) rather than
runtime.NumCPU(). Falls back to logical CPUs when topology is absent
(non-Linux, container, ARM cpuinfo without core id). On a hyperthreaded
box this halves the over-pack: with the unchanged 2 vCPU reserve the
i5-12600H schedules 10 vCPU (12 cores - 2) instead of 14, keeping the
boot-spike CPU within real cores.
Harness: capacityRetryEC2 decorates the EC2 client and, on
InsufficientInstanceCapacity, waits and retries RunInstances instead of
failing — turning the daemon admission gate into back-pressure so
concurrent subtests queue. AWSClient.EC2 is now the wrapped
ec2iface.EC2API; EC2Conf keeps the raw *ec2.EC2 for .Config access.
Adds unit tests for parsePhysicalCores (hyperthreaded/hybrid/dual-socket/
fallback) and the retry decorator (retry-then-succeed, non-capacity
passthrough, deadline).
* fix(daemon): SPINIFEX_HOST_VCPU override so small CI hosts can start
Denominating capacity by physical cores starves tiny hosts: a 2-physical-
core CI runner leaves 0 schedulable after the default 2-vCPU reserve, so
NewResourceManager refuses to start and every daemon-constructing unit test
fails. The previous runtime.NumCPU() count (4 logical on the same runner)
incidentally cleared the reserve floor.
Add a SPINIFEX_HOST_VCPU escape hatch (symmetric with the existing
SPINIFEX_RESERVED_* overrides) that pins the detected core count for hosts
whose /proc/cpuinfo topology misreports, and pin it to 4 in the daemon
TestMain so unit tests are deterministic regardless of runner topology.
Reserve stays at the default 2.
* test(eks): EKS control-plane e2e suite (Phase 1, API-level)
Add tests/e2e/eks/eks_test.go driving the EKS lifecycle against awsgw:
CreateCluster -> ACTIVE (endpoint + CA asserted), kubeconfig artifact
assembly, AccessEntry CRUD, get-token presign decode, DeleteCluster -> gone.
Harness: EKS SDK client on AWSClient + WaitForEKSCluster{Active,Deleted}
pollers. Wire e2e-eks into the Makefile and the single-node CI suite list.
* test(eks): Phase 2 kubectl get-nodes data-plane gate
Add the reachability shim (harness/eks_mgmt.go) that reads the control-plane
VM's br-mgmt IP from the cluster KV meta over NATS, plus a kubectl wrapper
(harness/kubectl.go) that skips when kubectl is absent. KubectlGetNodes points
kubectl at https://<mgmtIP>:6443 (insecure-skip-tls-verify; cert SANs the NLB
DNS, not the mgmt IP) with an aws eks get-token exec block and polls until the
control-plane node is Ready — the get-token webhook auth regression gate.
IRSA pod + the real NLB/Route53 passthrough are deferred to a follow-up.
* test(eks): tolerate doubled NotFound code + dial loopback NATS
DeleteCluster detection matched the SDK constant ResourceNotFoundException,
but awsgw emits ResourceNotFoundExceptionException on the wire — match the
NotFound code by substring so deletion is detected immediately instead of
timing out. The mgmt-IP shim dialed NATS.Host verbatim (0.0.0.0:4222), which
the serving cert does not SAN; rewrite the wildcard bind to loopback.
Validated live: all 6 subtests green, kubectl get nodes shows the
control-plane node Ready via the get-token webhook auth path.
* feat: build system images with libguestfs (no host block devices)
Replace the qemu-nbd + sudo mount + chroot + /dev-bind flow in
build-system-image.sh with libguestfs (virt-customize / virt-resize /
guestfish). Every operation runs in an isolated userspace appliance VM with
its own kernel, so the build touches no host block device, no host mount, and
needs no sudo — removing the failure class that wiped the host's /dev on shared
CI runners (a leaked /dev bind traversed by rm -rf).
Applies to all distros: alpine resizes via guestfish e2fsck+resize2fs, ubuntu
via virt-resize; customization (packages, binaries, files, setup script,
service enable, cache cleanup) is one ordered virt-customize invocation.
LIBGUESTFS_BACKEND=direct (no libvirt on runners).
Wire the eks-node AMI into the single-node e2e job: the runner builds the
device-free raw, ships it to the VM, and the VM imports it (predastore/NATS are
VM-only) so the eks suite's CreateCluster has an AMI to boot.
* feat(eks): bootstrap + state-report via AWS gateway broker
Converge the EKS control-plane → host channel onto the ELBv2 lb-agent model:
the K3s server VM publishes its bootstrap envelopes and periodic state reports
via SigV4-signed HTTPS POST to the AWS gateway, instead of dialing core NATS
directly. A new gateway action (PublishInternal, POST
/clusters/{name}/internal-publish) relays the payload verbatim onto the
existing eks.bus.* / eks.state.* subjects, so the daemon-side subscriber, KV
persistence/encryption, and reconciler are unchanged.
This removes the VM's need to reach NATS at all, so NATS can stay
cluster-internal behind its bind/firewall/mTLS controls — the prior direct
path broke when NATS was not VM-reachable (e2e CreateCluster timeout) and was
DDIL-fragile (fire-and-forget nats pub, no ack/retry).
- gateway/eks: PublishInternal broker + route; closed-set channel/kind validation
- cmd/eks-gateway-publish: one-shot SigV4 publisher (retry-until-ack) baked into
the eks-node AMI; replaces the nats-cli publisher in k3s-first-boot.sh and
mulga-eks-state-report.sh
- handlers/eks + daemon: hand the K3s server VM the mgmt-reachable gateway URL +
system (Predastore) SigV4 creds + gateway CA instead of NATS URL/token/CA;
new resolveSystemGatewayBaseURL (mgmt-bridge-first)
- eks-node image: build/install eks-gateway-publish, drop nats-cli download
* feat: migrate eks-token-webhook to the AWS gateway broker
The TokenReview webhook dialed core NATS directly (STS verify + AccessEntry
KV read). On a mgmt-only control-plane VM with NATS bound off-subnet that
fails, so the webhook exited at startup, the apiserver token-webhook
kubeconfig was never written, and the apiserver crash-looped — the cluster
never reached ACTIVE.
Converge it onto the same gateway-broker model as bootstrap/state-report and
ELBv2's lb-agent: the webhook SigV4-signs an HTTPS POST {accountId,token} to
/clusters/{name}/token-review, and the gateway runs the verify + lookup
host-side and returns the resolved identity. The VM now speaks no core NATS.
- internal/eksgw: shared on-VM SigV4 client (also adopted by
eks-gateway-publish).
- handlers/eks/token_review.go: Authenticate decision core (moved from the
webhook binary) + ResolveTokenReview host-side wiring.
- gateway/eks token-review route + WebhookTokenReview handler.
- webhook: drop NATS, relay via eksgw.Client; broker fault -> 503 (retryable)
not a silent deny.
- initd: export EKS_GATEWAY_* / EKS_ACCESS_* creds, not SPINIFEX_NATS_*.
* fix: build eks-node helpers with GOFIPS140; cover token-review broker
eks-gateway-publish blank-imports internal/fipsboot, whose init() panics
unless the binary is built with GOFIPS140=v1.0.0. The eks-node AMI build
omitted the flag, so the binary paniced at first boot right after k3s came
up — k3s-first-boot (set -eu) died before publishing the bootstrap token and
the cluster never reached ACTIVE. Build both eks-node helpers with
GOFIPS140=v1.0.0, matching cmd/lb-agent.
Add NATS-backed unit tests for ResolveTokenReview and the WebhookTokenReview
gateway handler (verify responder + account KV) to cover the new broker path.
* fix: export first-boot env so eks-gateway-publish child sees it
k3s-first-boot.sh and mulga-eks-state-report.sh sourced /etc/spinifex-eks/
first-boot.env but never exported the vars, so the eks-gateway-publish child
process inherited an empty environment and exited '--account-id is required'.
The control-plane VM came up healthy but never published its bootstrap
envelopes, so the cluster never reached ACTIVE. Wrap the source in set -a /
set +a so the KEY=value lines export to child processes.
* fix: make awsgw start --host override actually bind
The 'awsgw start --host' override printed 'Overwriting awsgw host' but the
assignment was commented out, so AWSGW always bound the spinifex.toml host
(a specific WAN IP, e.g. 192.168.0.10). A system instance (EKS K3s control
plane, internal LB agent) has a mgmt-only NIC on-link to the br-mgmt IP with
no route to the WAN IP, so its SigV4 POSTs to the gateway were refused and the
cluster never bootstrapped. Honour --host (e2e passes 0.0.0.0:9999) so AWSGW
listens on all interfaces incl. br-mgmt; the serving cert already SANs every
local interface IP, and WAN clients still reach it.
* test(eks): install kubectl on e2e VM so KubectlGetNodes runs
The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full
aws eks get-token -> kubectl -> apiserver -> token-webhook -> gateway broker
-> STS verify path — the only end-to-end exercise of the token-review broker.
The e2e binaries run on the VM, where kubectl was absent, so
harness.NewKubectl t.Skip'd it and the path went unvalidated. Install a
checksum-verified kubectl (pinned to the k3s server version) on the VM,
gated on e2e-eks.
Internal-scheme LBs on single-node have no dedicated mgmt-bind IP on
br-mgmt, so resolveMgmtRoute falls back to routing the agent to the
advertised WAN IP via the mgmt NIC. That advertised IP lives on a
different bridge than the agent's mgmt NIC, and the cross-bridge return
path never delivers — the agent never heartbeats and the LB hangs in
provisioning (normal single-node dodges this entirely via dev_networking
slirp at 10.0.2.2; bare-metal external-mode is the first real exercise of
this path).
Point the agent at the br-mgmt host IP, which shares its mgmt L2 — no
cross-bridge hop — mirroring the multi-node datapath. Requires AWSGW to
listen on br-mgmt (bind 0.0.0.0); the server cert already SANs that IP.
Internal-scheme LBs only leave provisioning once the lb-agent heartbeats
AWSGW, which requires the mgmt return route resolved by resolveMgmtRoute
(MgmtRoute{Gateway,Target}, or MgmtBridgeIP+AdvertiseIP on single-node).
When none resolve, the route was silently skipped and the LB hung in
provisioning until the client timed out, with no signal as to why.
Log which precondition is empty on the internal skip path, and mark the
LB failed instead of provisioning when no return route exists, so the
broken data-plane surfaces immediately.
InitBuckets was passed the OVN chassis count as the replica factor. When the
chassis count exceeds the NATS node count (single-node bare-metal: 1 node but
2 SBDB chassis), CreateKeyValue requested more replicas than JetStream could
satisfy, the create-or-open fallback returned "bucket not found", and vpcd
crash-looped — cascading to RunInstances ServerInternal. Create at 1 replica
like STS/IAM; the daemon's deferred upgradeJetStreamReplicas raises all KV_*
streams to cluster size when the cluster grows.
* feat(eks): webhook observability — console log sink + per-request TokenReview decision
* fix(eks): wire token-webhook authenticator into generated k3s apiserver config
The generated /etc/rancher/k3s/config.yaml overrides the AMI-baked
config.yaml.skel, but its kube-apiserver-arg list omitted the
authentication-token-webhook-config-file arg. The running apiserver
therefore never loaded the token webhook, so bearer tokens minted by
'aws eks get-token' 401'd without the webhook ever being invoked.
* fix(eks): echo TokenReview audiences so api-audiences apiserver accepts the principal
With --api-audiences set, kube-apiserver sends spec.audiences in the
TokenReview and discards an authenticated=true response that does not
confirm a matching audience. The webhook returned an empty
status.audiences, so a valid IAM principal surfaced as a 401. Echo the
request's audiences on success.
* fix(eks): pin token-webhook version v1 so apiserver accepts the response
kube-apiserver --authentication-token-webhook-version defaults to
v1beta1, but the eks-token-webhook responds with an
authentication.k8s.io/v1 TokenReview. The apiserver decodes the response
against the expected GVK and rejects the mismatch, surfacing a valid
principal as a 401. Pin v1 to match the webhook.
Bare-metal e2e produced no artifact, so RCA fell back to truncated
log-failed output. Tee bootstrap + e2e output to test-baremetal.log,
generate junit-baremetal.xml, run e2e-analyze to emit analysis.md, and
upload the bundle as e2e-baremetal-artifacts-<run_id>.
aws eks list-clusters returned InternalErrorException for a few seconds after a
daemon restart, while describe-cluster (by name) succeeded. During the
post-restart window the per-account KV stream's Raft group has no leader yet, so
kv.Keys() (an ordered consumer over the warming stream) fails with a transient
NATS error. ListClusters wrapped it as a fmt.Errorf string the daemon's
ValidErrorCode cannot map, so it sanitized to ServerInternal — a misleading 500
that tells AWS clients the request is non-retryable.
Classify transient NATS conditions (no-responders, timeout, no-stream-response,
connection-closed) and return a retryable ServiceUnavailable (503) instead;
genuine faults still pass through wrapped. describe-cluster survived only
because kv.Get on the same bucket is cheaper than the consumer kv.Keys spins up.
* feat(iam,awsgw): OIDC provider CRUD + IRSA discovery endpoints (6e)
Complete the AWS-SDK-facing half of IRSA:
- IAM Create/Get/List/DeleteOpenIDConnectProvider, backed by the per-account
oidc-providers registry that STS AssumeRoleWithWebIdentity already reads to
authorise an issuer. Create writes the key STS checks; the provider ARN is
the scheme-stripped issuer per AWS convention.
- awsgw serves the per-cluster OIDC discovery document
(/.well-known/openid-configuration) and JWKS (/keys) as unauthenticated
public endpoints, mounted ahead of the SigV4 surface in their own route
group. The discovery issuer is the cluster's stored OIDCIssuer, byte-exact
with the apiserver iss claim; JWKS is served from the EKS account bucket.
Unit + route tests cover CRUD happy/idempotent/not-found/invalid-url, the
ARN<->issuer round-trip, discovery issuer/jwks_uri equality, JWKS bytes, 404s,
and the no-auth bypass. make preflight green.
* fix(awsgw): route anonymous AssumeRoleWithWebIdentity past SigV4 (6e)
AssumeRoleWithWebIdentity is the IRSA credential-bootstrap call: the AWS SDK
sends it unsigned because the caller holds no AWS credentials, only a projected
ServiceAccount JWT in the body. The gateway ran the SigV4 middleware first and
rejected it with MissingAuthenticationToken, so the true anonymous path never
reached the handler.
Add anonymousSTSInterceptor, a pre-auth middleware mounted ahead of the SigV4
surface. For an unsigned POST it body-peeks the Action and, for the
anonymousSTSActions set, dispatches straight to STS_Request; STS_Request now
skips SigV4-caller resolution for those actions (the handler authenticates the
JWT and gates on the role's web-identity trust policy). Signed requests carry an
Authorization header and pass through untouched, off the body-buffering path.
Surfaced by the 6e live exit gate: a real in-pod assume-role-with-web-identity
would otherwise have been rejected before reaching token validation.
* feat(eks): nodegroup provisioning + K3s agent join (6d)
CreateNodegroup launches N customer-owned worker EC2 from the unified
eks-node AMI in agent role; each joins the cluster via K3S_URL + K3S_TOKEN
baked into cloud-init (first-boot.env SPINIFEX_K3S_ROLE=agent + agent.env).
Workers are customer-owned and DescribeInstances-visible (no ManagedBy,
no mgmt NIC) via the normal RunInstances path, matching real EKS.
K3S_URL targets the control-plane ENI's in-VPC private IP on :6443 — the
cluster NLB DNS endpoint is not resolvable in-VPC yet; switch to the NLB
endpoint on :443 once authoritative in-VPC DNS lands. K3S_TOKEN is the
decrypted cluster node-token persisted at server bootstrap.
Describe/List/Update(scaling)/Delete round out CRUD. Cluster<->nodegroup
SG ingress rules (apiserver 6443, flannel VXLAN 8472/udp, kubelet 10250,
intra-nodegroup) authorized idempotently. UpdateNodegroupVersion deferred.
* feat: gate EKS cluster health via NATS state reports
The K3s apiserver binds the VPC node-ip, unreachable from the host-side
spinifex daemon, so the reconciler cannot probe /healthz directly. Add a
control-plane state reporter that publishes {healthz, node_count} over the
management NATS bus the daemon already shares.
- eks-node server role runs mulga-eks-state-report as an OpenRC service,
publishing eks.state.{account}.{cluster}.server every 30s.
- ClusterReconciler subscribes to that subject, gates CREATING -> ACTIVE on
a healthy report, and records Ready node_count into ClusterMeta.
- node_count counts Ready nodes only; k3s retains terminated workers as
NotReady until pruned, so a raw count never drops on scale-down.
The iso job mounts a kernel-global devtmpfs under the workspace
(.cache/rootfs/dev). A killed job skips its EXIT-trap unmount and leaks
the bind; the verify job (same ci-single runner) then ran a bare
'rm -rf $WORKSPACE/*' that could traverse it and wipe the host /dev.
Detach any mount under the workspace before deleting and add
--one-file-system so rm never crosses a mount boundary. Same backstop
added to the iso job's own cleanup.
A killed ubuntu system-image build can leave `mount --bind /dev` under
its build dir. The e2e Clean Workspace step then ran
`sudo rm -rf "$GITHUB_WORKSPACE"/*` as root, traversed the leaked bind,
and deleted the host's /dev (incl. /dev/null), wedging the runner.
- e2e.yml: umount anything under the workspace (deepest-first) before
deleting, and pass `rm --one-file-system` so rm can never cross a mount
boundary into host /dev. Hard backstop, applied to all 3 Clean Workspace
steps.
- build-system-image.sh: recursive lazy unmount in cleanup() detaches the
chroot binds even when busy, and mark the build subtree rprivate so the
/dev bind can't propagate into the host namespace.
* fix: switch EKS datastore to embedded etcd + fix Alpine cloud-init IMDS route
Flip the K3s control-plane datastore from SQLite/kine back to embedded etcd
(cluster-init: true, etcd-expose-metrics: true). The earlier abandonment
premise — that etcd's per-commit fsync stalls past the apiserver's 5s deadline
on the viperblock-backed root volume — was wrong: etcd inits fine on an
unchanged viperblock (apiserver ready 25-30s, clusters reach ACTIVE). The real
blocker was a network-config regression, fixed here.
Alpine's cloud-init eni renderer crashes (KeyError: 'gateway') on the
gateway-less IMDS on-link route, which failed the whole network-config render
so the static mgmt NIC never came up (no NATS bootstrap, no /healthz). Fixes:
- generateNetworkConfig: add emitIMDSRoute bool; selectNetworkConfigForFamily
passes false for Alpine so the eni renderer never sees the route.
- Deliver the IMDS route out-of-band via a persistent /etc/local.d script +
OpenRC local service. Resolve the VPC NIC via its default route at runtime
(ip route show default), not a hardcoded name: the persistent-net rename to
vpc0 doesn't apply on the Alpine AMI, so the NIC is eth0 at first boot.
- EKS path bakes the same route into buildK3sUserData's own write_files/runcmd;
the generic template Alpine block fires only when no #cloud-config user-data,
avoiding a duplicate write_files key (yaml.safe_load keeps last, silently
dropping a block).
- eks-node image: config.yaml.skel adds cluster-init; comments updated.
* fix: surface CreateCluster step failures at ERROR + map subnet faults; add auth-mode/delete test cases
CreateCluster wrapped its step failures in bare fmt.Errorf that the request
surface collapses to an opaque ServerInternal ("contact AWS re:Post"), so the
real cause never reached the operator. Add logCreateErr to log each step's
precise cause at ERROR before returning, and map a subnet-resolve failure
(caller-supplied parameter) to InvalidParameterValue instead of a retryable
ServerInternal the SDK pointlessly retries. (mulga-siv-165.4)
Tests (mulga-siv-165.15): reject authenticationMode=API_AND_CONFIG_MAP (only
the CONFIG_MAP value was covered); DeleteCluster on a nonexistent cluster via
the full-deps path returns ResourceNotFound (the shim path short-circuits to
ServiceUnavailable before the meta lookup).
* fix: hide EKS control-plane VMs from customer DescribeInstances
EKS K3s control-plane VMs are owned by the customer account (their ENI
lives in the customer VPC for the NLB target), so unlike LB microVMs
(owned by the system account) they leaked into customer
ec2:DescribeInstances. The spinifex:managed-by tag only filtered the UI
Nodes page, not the EC2 API.
Add a caller-aware exclusion: skip any ManagedBy-tagged VM unless the
caller is root. Preserves operator visibility, matches LB behaviour, and
leaves future customer-owned worker nodes (no ManagedBy) visible.
* docs: correct stale SQLite datastore comments in eks-node image (now embedded etcd)
reloadNginx started nginx via cmd.CombinedOutput(). nginx daemonizes, and the
rendered NLB config uses `error_log stderr`, so the forked master and workers
inherit and hold this command's stderr open. CombinedOutput's Wait blocks on
stderr EOF, which the long-lived daemon never sends — so the call never returns,
wedging the agent's synchronous tick() before it applies the nginx engine or
reports health. NLB targets stayed at 0/N until the e2e health wait timed out,
while the data plane (nginx itself) ran fine — so manual smoke tests passed.
ALB is unaffected: haproxy -D redirects its stdio to /dev/null on daemonize, so
CombinedOutput sees EOF and returns.
Capture nginx start/reload stdio to a temp file and use cmd.Run(), so Wait
blocks only on the short-lived foreground process that exits once daemonized.
Surfaced via the LB microVM guest console now collected in CI.
NLB target health stalls at 0/N in CI while the data plane works locally.
The lb-agent's nginx engine activation, any nginx -t/reload error, and its
active-probe results are logged to the LB microVM serial console — which the
host journal never captures and the GetConsoleOutput API path reports as
NotFound, so the failure is undiagnosable from CI artifacts.
- e2e.yml: pull /run/spinifex/console-*.log off each node in the single- and
multi-node collect steps (files persist past VM teardown) into lb-console/.
- lbagent healthcheck: log probe failures at Warn (was Debug) so unreachable
targets surface at the agent's default level on the captured console.
E2E NLB suites timed out at "0/2 healthy" while ALB suites on the same
target instances went healthy in ~5s. The host daemon journal shows the
NLB agent reporting an empty server list (no probe results at all), which
happens when the lb-agent never enters nginx mode — i.e. `nginx -t`/start
fails closed inside the LB microVM, so the agent never probes and reports
nothing. The data plane itself is verified working; this is purely the
active-health-check pathway failing to converge in CI.
- lbagent: pre-create nginx runtime prefix dirs (pid dir + /var/lib/nginx,
/var/lib/nginx/tmp, /var/log/nginx) before `nginx -t`, mirroring
reloadHAProxy's socket-dir pre-create. A stripped Alpine initramfs may
lack these, dead-ending the whole NLB health path.
- build-microvm-image.sh: assert the nginx binary and ngx_stream_module.so
are present in the image and fail the build loudly, instead of surfacing
as an opaque e2e health timeout.
- e2e: capture the LB VM serial console to the artifact bundle on NLB
health-wait failure. The agent's engine activation and any `reload nginx:`
error land on the guest console, which the host journal never sees.
* feat(elbv2): NLB L4 data plane on nginx stream (mulga-siv-952)
Network load balancers render an nginx `stream` config (TCP/UDP/TLS/
TCP_UDP) instead of HAProxy, which cannot load-balance UDP in any
release. The gateway tags GetLBConfig with Engine=nginx; the LB agent
dispatches to the nginx engine (config/pid/cert paths + reload) and
skips HAProxy stats polling. ALB stays on HAProxy. nginx is added to
the microvm image.
* docs(elbv2): note NLB runs nginx stream, ALB runs HAProxy (mulga-siv-952)
* fix(elbv2): load nginx stream module for NLB data plane
Alpine's `nginx` apk is built without the stream module — it ships
separately as `nginx-mod-stream`. Without it, the generated NLB config
fails `nginx -t` with [emerg] unknown directive "stream", so the agent
never applies it. Add the package to the microvm image and emit a
`load_module .../ngx_stream_module.so;` directive at the top of the
generated config.
* fix(elbv2-ui): allow single-subnet LB + full NLB protocols in listener edit
Create-LB form required min 2 subnets while the CLI/backend accept 1;
relax to min(1) to match. Listener add/edit only offered HTTP/HTTPS —
add the NLB protocol set (TCP, UDP, TLS, TCP_UDP), driven by LB type,
with cert/ssl-policy fields gated on secure protocols (HTTPS/TLS).
* feat(elbv2): active target health checks for nginx NLBs
nginx stream has no active upstream probing, so NLB targets never
advanced past initial — DescribeTargetHealth stayed initial forever and
WaitForTargetsHealthy timed out for every NLB. GetLBConfig now delivers
per-target HealthTarget specs (daemon-computed server name, probe
address, protocol/path) for network LBs; the lb-agent probes each target
every tick (TCP dial or HTTP/HTTPS GET) and reports UP/DOWN via the
existing heartbeat path, so the daemon's threshold hysteresis advances
targets to healthy. ALBs are unchanged (HAProxy stats).
Adds an Internal_NLB_UDP e2e suite gating on WaitForTargetsHealthy plus a
UDP datagram round-trip through the VIP.
* test: cover Sprint 3b ENI hot-plug surface
Sprint 3b's gateway + daemon + KV-helper additions landed without unit
tests. dev's diff-coverage gate flipped to 28.5% as a result. This adds
the three test files needed to bring the diff coverage of the ENI
hot-plug surface back above the 60% threshold.
- handlers/ec2/vpc/eni_hotplug_helpers_test.go: GetENIRecord, UpdateENI,
FindENIByAttachment — happy paths, not-found, account scoping,
round-trip of every new ENIRecord field.
- gateway/ec2/vpc/network_interface_hotplug_test.go: Attach + Detach
validation, no-responders, NATS roundtrip with command capture,
daemon error / malformed responses, attachment-resolver branches.
- daemon/daemon_handlers_eni_test.go: eniHotplugErrorCode table,
publishENIHotplugEvent (nil + publish), full handleAttach /
handleDetach matrix (nil data, not-running, ENI/attachment not found,
wrong owner, success with stub QMP + NATS event assertion, hot-plug
failure + KV rollback assertion).
vm.SetHotPlugTestSeams is added as an exported test seam following the
same pattern as utils.SetSudoCommandForTest so daemon-package tests can
swap newDeviceController + eniPipelineSleep without reassigning vm-package
unexported vars.
* feat(vm): wire real OVS tap into ENI hot-plug pipeline
Sprint 3c: replace the stubTapAdd/stubOVSAddPort/stubTapDel/stubOVSDelPort
placeholders with the shared NetworkPlumber. SetupTap creates the kernel tap
and attaches it to br-int with the OVN iface-id binding in one atomic step;
CleanupTap reverses both on rollback and detach. Nil-plumber managers stay a
noop, mirroring setupExtraENINICs.
Adds fake-plumber coverage: tap-on-br-int spec assertion, tap-setup failure
aborts before any QMP call, and QMP/detach paths both invoke CleanupTap.
The previous new-subnet egress baseline used the default VPC, whose main
route table is prebaked with 0.0.0.0/0 -> IGW at bootstrap and reconciled by
the vpcd drift loop, so a subnet created in it is reachable regardless of the
create-time gap — the test passed and never flagged the bug.
Replace it with runNewVPCEgressBaseline: a brand-new VPC where the IGW route
is added to the MAIN route table BEFORE the public subnet is created. The
subnet then implicitly joins a main RT that already carries the route, but
CreateSubnet does not recompute per-subnet egress, so the subnet's OVN egress
LRP is never installed and the instance has no datapath to the IGW. With an
open SG, routing is the only barrier, so the instance is unreachable — red
until CreateSubnet triggers a per-subnet egress recompute.
PoolMode-gated like runVPCSubnetE2E (fresh VPC needs OVN IPAM for a public
IP). runVPCSubnetE2E doesn't catch this: it associates a custom RT then adds
the route, so the route mutation enumerates the subnet and installs the LRP.
Add baseline tests that confirm two fresh-install regressions the existing
suite masks (it authorizes tcp/22 on the default SG before every SSH probe):
- default SG default-deny gates external reach of instances in the
default subnet (SG, not routing)
- a newly-created public subnet never gets its OVN IGW egress LRP, so it
has no route to the VPC IGW despite MapPublicIpOnLaunch
Plus a positive test: instances sharing the default SG can communicate via
ICMP (same-host single-node, cross-host multinode).
Single-node baselines run after TestClusterStatsCLI (0-VM gate); multinode
after TestMultinodePreflight (fixture init). Instances are self-cleaning so
they don't perturb the singleton-VM ordering.
Remove the qemu hostfwd SSH fallback from the harness (InstancePublicSSHHost,
GuestSSHEndpoint) - it bypasses the OVN datapath and masks these regressions.
Endpoints now t.Fatal when no public IP is assigned.
* feat(ui): NLB support + protocol/type-dependent create-LB fields
The Create Load Balancer page was hardcoded to ALB. Add a load-balancer
Type selector (application/network) and switch the listener + target-group
form options on the selected type, mirroring the server's validation:
- ALB: HTTP/HTTPS listeners and target groups.
- NLB: TCP/UDP/TLS/TCP_UDP listeners and target groups; no security groups.
- Secure protocols (HTTPS/TLS) reveal a certificate selector; HTTPS also a
security-policy selector. Cert is required for both; SSL policy is HTTPS-only.
- The default target-group protocol + health-check protocol stay in sync with
the listener (TCP health checks omit HTTP path/matcher).
- The wizard mutation now sends the chosen Type and threads cert/SSL-policy
through CreateListener (previously hardcoded Type=application and dropped
certificates).
* feat(ui): NLB-aware target groups and LB detail
Standalone Create Target Group offers the full protocol set
(HTTP/HTTPS/TCP/UDP/TLS/TCP_UDP) and syncs the health check to TCP for L4
groups, dropping path/matcher from the request and CLI preview. Hide the
Security groups tab on the load balancer detail page for network LBs.
* fix(admin): write ~/.aws credentials 0600 via atomic temp+rename
UpdateAWSINIFile ended with ini.SaveTo, which uses os.Create (0644 &
~umask) and left ~/.aws/credentials world-readable. Render to a 0600
temp file and rename into place so secrets are never exposed.
* fix(eks): purge OIDC signing key on zeroize; pin account bucket History=1
ZeroizeClusterOIDCKey used kv.Delete (a tombstone) which leaves prior
revisions of the encrypted PEM in KV history. Use kv.Purge to drop the
key's whole history. Decouple the account bucket's History from the
schema version (KVBucketEKSAccountHistory=1) so a version bump can't
silently widen history and resurrect ciphertext.
* fix(reconcile): get-or-create leader bucket in AcquireLeader
CreateKeyValue returns 'stream name already in use' once the bucket
exists, so every reconcile after the first hit the 60s deadline and
never elected a leader. Attach to the existing bucket first, create
only when genuinely absent.
* fix(eks): drop registry entry on lease-held instead of phantom holder
RunClusterReconciler returned (nil, nil) when another node held the
lease; Spawn recorded a holder with a nil release, so Has() stayed true
and a later Spawn no-op'd — the cluster was never re-reconciled until
daemon restart. Return ErrLeaseHeld so Spawn drops the entry and retries
after the holder's TTL lapses.
* refactor: tidy EKS cluster state, dedup, and not-found error wiring
- listClusterNames: O(n) map dedup instead of O(n^2) slices.Contains
- drop dead store *Store field + NewStore call from EKSServiceImpl
- ClusterMeta.LastHealthProbe: drop misleading omitzero tag (stdlib
encoding/json ignores it; json/v2 only) and document why it always
serializes
- clarify ErrorResourceNotFound is the ACM code; EKS uses
ErrorEKSResourceNotFound (ResourceNotFoundException)
* test: EKS 6b pagination, restart-resume, and bootstrap bind hardening
- T7: ListClusters NextToken/MaxResults paging + out-of-range clamp
- T4: SpawnRegisteredReconcilers resumes CREATING/ACTIVE on restart,
skips terminal clusters, no-ops when deps absent
- T-quality: replace sleep-based subscription binding with polling
NumSubscriptions, poll KV for the one-shot drop, and assert Run
tears down its subscriptions on return
T1 (restjson wire shape), T5 (JWKS kty mismatch), and T6
(resolveGatewayHost branches) were already covered.
* test(reconcile): cover AcquireLeader get-or-create election
First acquire creates the bucket and wins; a second acquire while the
lock is held attaches to the existing bucket and loses cleanly (the
get-or-create fix, not a CreateKeyValue dead-end); after release a later
acquire takes over.
* feat(ui): HTTPS listeners with ACM certificate + SSL policy
Expose HTTPS in the load balancer listener UI. The ListenerForm gains a
HTTPS protocol option that reveals a certificate selector (backed by a new
ACM ListCertificates query), an inline certificate import dialog
(ImportCertificate), and an SSL security-policy selector (DescribeSSLPolicies).
CreateListener/ModifyListener now carry Certificates + SslPolicy for HTTPS;
the listeners table summarises the policy and certificate per HTTPS listener.
Adds an ACM client (getAcmClient) and @aws-sdk/client-acm. Server, gateway
and ACM endpoints already exist (epic phases A + B); this is frontend-only.
* fix: ACM JSON timestamps, react-compiler watch reactivity, scroll-chaining
- gateway/acm: serialise responses with jsonutil.BuildJSON so time.Time
fields emit epoch seconds (AWS JSON 1.1 wire format). encoding/json
emitted RFC3339 strings the JS SDK rejected with 'Expected real number',
crashing the LB detail Listeners tab once any cert exists.
- frontend: replace watch() with useWatch across forms so the React
Compiler tracks the subscription and re-renders on field change
(HTTPS listener cert/policy fields now reveal). Add toggle regression
test for listener-form.
- frontend: scope overscroll-none to html,body instead of *, restoring
wheel scroll-chaining over nested scroll containers (sidebar, tables,
dialogs).
* feat(eks): unify eks-server + eks-agent into one eks-node AMI
Collapse the two K3s AMIs into a single spinifex-eks-node image whose role
(server vs agent) is chosen per-instance at first boot, halving build time,
storage, and CVE-rotation surface.
- scripts/images/eks-node/: superset manifest + merged setup.sh; all role
services baked but only crond + the new eks-node-role selector enabled.
- eks-node-role first-boot selector reads SPINIFEX_K3S_ROLE, rc-update adds +
starts the matching services (server: k3s + eks-token-webhook + k3s-first-boot
publisher; agent: k3s-agent), records /etc/spinifex-eks/role, self-disables.
Paths are env-overridable; eks-node-role_test.sh covers all five cases.
- mulga-eks-etcd-snapshot.sh early-exits on agent nodes.
- k3s_server_vm.go seeds SPINIFEX_K3S_ROLE=server into first-boot.env.
- Makefile/dev-install: build-eks-node-image / import-eks-node-image replace
the server+agent targets.
- Delete scripts/images/eks-server + eks-agent. The launcher resolves the AMI
by the spinifex:managed-by=eks tag, so running clusters (which hold the
resolved AMI ID) are unaffected.
AMI bake + cluster boot are operational integration steps, not yet run.
* fix(eks): order eks-node-role after cloud-config/cloud-final
The first-boot role selector reads /etc/spinifex-eks/first-boot.env, which
cloud-init delivers via write_files in the cloud-config/cloud-final stages.
The selector only depended on net+cgroups, so on a fast boot it could start
before write_files landed and read a missing role env. Add an after-ordering
on cloud-config + cloud-final so the env file is always present first.
* feat(eks): 6c AccessEntry CRUD + access-policy association
Replace the nine notImpl() AccessEntry/AccessPolicy stubs with real
KV-backed handlers (Q9, API auth mode):
- AccessEntry CRUD persisted at clusters/{name}/access-entries/{hash};
the principal ARN is hashed into the key (ARNs carry ':', illegal in
NATS KV keys) while the record keeps the plaintext ARN.
- AssociateAccessPolicy/Disassociate/ListAssociated against the four
supported AWS-managed policies; unsupported policy ARN or bad access
scope -> InvalidParameterValue.
- ListAccessPolicies returns the static supported catalogue.
- Cluster-existence gate -> ResourceNotFoundException; duplicate create
-> ResourceInUseException; node-type entries rejected until 6d.
Updates the daemon routing test for the now-implemented handlers.
* feat(eks): plumb caller principal ARN + bootstrap-creator-admin
Thread the caller's resolved IAM principal ARN from the gateway to the
EKS control plane via a new X-Principal-ARN NATS header, so CreateCluster
can mint the bootstrapClusterCreatorAdminPermissions AccessEntry (Q9):
- utils.NATSRequest takes optional extra headers; PrincipalARNHeader +
PrincipalARNFromMsg added.
- Gateway resolves the caller ARN best-effort (buildCallerARN) and passes
it down the EKS route table; only CreateCluster consumes it.
- CreateCluster signature gains callerPrincipalARN; default-true bootstrap
flag mints a system:masters entry keyed by the caller's exact ARN (the
token webhook resolves the same ARN from STS). '' degrades gracefully.
- Dedicated daemon handler reads both headers.
* feat(eks): IAM token webhook + STS verify-over-NATS + apiserver wiring
The eks-token-webhook (baked into the eks-server AMI) authenticates kubectl
requests that present an `aws eks get-token` bearer token: decode the presigned
STS GetCallerIdentity URL, verify it, resolve the principal ARN to its cluster
AccessEntry, and answer the apiserver TokenReview with the mapped username +
groups.
STS is hosted in-process in the awsgw service and is not otherwise on the bus,
so awsgw now serves an `eks.VerifyToken` NATS subject that wraps
VerifyPresignedGetCallerIdentity (the cluster name is forwarded so STS enforces
the x-k8s-aws-id anti-replay pin). The webhook reads the AccessEntry KV directly
over its own NATS connection.
Apiserver wiring: the k3s config skeleton sets
authentication-token-webhook-config-file; the webhook writes its own self-signed
serving cert + the apiserver kubeconfig at startup (ordered `before k3s`, with a
bounded wait in the k3s start_pre so the apiserver never loads a missing file).
* fix(eks): omit empty spec from token webhook TokenReview response
* fix(sts): presigned GetCallerIdentity must hash empty body, not UNSIGNED-PAYLOAD
VerifyPresignedGetCallerIdentity built the SigV4 canonical request with
"UNSIGNED-PAYLOAD" as the payload hash. That is an S3-only convention; botocore
and the aws-sdk presign non-S3 requests (sts:GetCallerIdentity, the shape `aws
eks get-token` produces) with the SHA256 of the empty string. Every real
get-token therefore failed signature verification with InvalidIdentityToken,
breaking the entire EKS IAM token-auth flow end to end.
Use the empty-string SHA256. The existing hand-rolled presign tests masked the
bug by signing with the same wrong constant; they now sign correctly, and a new
test presigns with the real aws-sdk-go v4 signer (the same canonicalisation the
CLI uses) so this can't regress silently.
Verified live: a real `aws eks get-token --cluster-name dev1` now resolves to
the caller principal over the awsgw eks.VerifyToken surface; an otherwise-valid
token presented for a different cluster is still rejected (anti-replay intact).
* fix(eks): match access-entry routes on escaped path so slashed principal ARNs resolve
IAM principal ARNs (user/admin, role/foo) embed a '/' that the AWS CLI
percent-encodes in the request path. The dispatcher matched r.URL.Path
(decoded), so %2F became a literal '/' and split the ARN across the
([^/]+) capture, dropping every describe/update/delete-access-entry and
associate/disassociate/list-access-policy call to InvalidActionException.
Match r.URL.EscapedPath() so the encoded ARN stays one segment, then
url.PathUnescape each captured param before it reaches the handler.
* test(eks): unit-cover access-entry scope validation, AWS converters, store guards
Ansible orchestrates the whole stack (spinifex + predastore + viperblock +
OVN + tofu hosts) and its spec/beads live in mulga. Moving it to top-level
mulga/ansible/ ends the per-change submodule-pointer-bump friction.
- git rm -r scripts/ansible/
- Makefile: drop ansible-dev-* / ansible-cluster-bootstrap targets + .PHONY
The tree lands at mulga/ansible/ (flat copy, Option B). History pivot:
tag feat/uplift-ansible-snapshot-pre-move marks the pre-move HEAD.
Plan: docs/development/improvements/ansible-uplift-to-mulga.md (in mulga).
An unresolvable certificate ARN was accepted into a listener record and
only failed later at config-render time, which aborted updateStoredConfig
wholesale and silently froze the load balancer's data-plane convergence.
Validate every certificate ARN against the ACM store (existence + account
ownership) in CreateListener, ModifyListener and AddListenerCertificates,
returning CertificateNotFound up front. Skipped when the ACM store is
unavailable, matching the nil-safe resolve path.
Resolve listener certificate ARNs to PEM via the ACM store (shared
JetStream KV, no cross-service hop), render HAProxy 'ssl crt' for secure
frontends, and deliver the combined cert+chain+key PEM to the LB agent
alongside the config. ConfigHash folds in cert content so rotation
triggers a reload; the agent writes each cert 0600 (path-constrained to
its cert dir) before the config + reload.
Phase B of the ELBv2 HTTPS epic (mulga-siv-180).
Josh Siviour