Align the top of the README with mulgadc.com: new headline and intro,
rewritten "What is Spinifex?", a new "Three ways to deploy" section
(Neocloud / on-premise / edge) and a new "AWS compatibility" service
matrix (EC2, EBS, S3, VPC, IAM, ALB/NLB, EKS, ECR, ECS available; RDS
Q3 2026) linking the GitHub releases. Refresh the architecture prose and
Key Features, add Deploy and AWS-services nav links, and swap banner.svg
for the ember-on-ink mini-stack design matching platform.svg.
Rebuild the message-driven architecture diagram in the ember-on-ink
theme to match platform.svg: ember gradient NATS bus, ember-tinted
gateway and daemon cards, neutral clients and backend cards. Content
unchanged (clients → gateway → NATS → ec2/ebs/s3/vpc/iam daemons →
local backends). De-slop the copy and refresh the img alt text.
Replace the platform diagram with a static SVG matching the mulgadc.com
front-page stack diagram: ember accent, ink panel, the portable-stack
box (Neocloud / on-prem / edge) wrapping the Spinifex AWS-compatible
service layer, over standard Linux on CPU/GPU hardware. Self-contained
and GitHub-safe (presentation attributes only). Refresh the img alt text.
The publish-eks-ami matrix fans out four legs on the single persistent
ci-single runner, all reusing one workspace root. go work init is
create-only and checkout never scrubs the root go.work, so legs after the
first fail with 'go.work already exists'. Remove any stale go.work before
init so each leg re-inits cleanly.
Expand the publish workflow to build the two Ubuntu NVIDIA GPU variants
alongside the Alpine eks-node/ecs-agent AMIs. The image build+publish
scripts are already manifest-driven and Ubuntu-aware; only the workflow
lacked the GPU legs.
Refactor the single-job body into a matrix over the four image manifests,
resolved by a new 'variant' input (all|cpu|gpu) so the slow 16G GPU legs
can be skipped when only the CPU AMIs need republishing. GPU images do not
fit a RAM-backed /tmp, so set SYSTEM_IMAGE_BUILD_DIR onto the runner's
on-disk _temp; legs run serially (max-parallel 1) on the single ci runner.
* ci: GPU e2e workflow on ci-gpu self-hosted runner
Runs tests/e2e/gpu (EC2/ECS/EKS GPU passthrough) against the persistent,
hand-bootstrapped wattle shared host, driven over SSH by a banksia runner
labelled ci-gpu (matches the ci-single/ci-multi convention) — no runner
installed on the shared box.
- runs-on [self-hosted, ci-gpu]; concurrency ci-gpu, cancel-in-progress
false (single A1000 = single-flight, queue not stomp)
- workflow_dispatch + GPU-path filter (not every push)
- manual-work mutex gate (~/.ci-gpu-paused / CI_GPU_PAUSED) so hands-on GPU
testing blocks CI
- if: always() cleanup terminates launched GPU instances; never resets the
box (no setup-ovn.sh/dev-reset/cluster-reset)
- drives the control plane with spx + the already-configured aws spinifex
profile directly (no wrapper scripts, nothing to install on wattle)
* ci(gpu-e2e): advisory AMI preflight, fail only if zero GPU AMIs
The suite is self-guarding — each test skips when its AMI is absent. The
hard-fail on the base ubuntu GPU AMI blocked the whole run even when the
ECS GPU AMI (the only one imported) could execute. Inventory the AMIs and
fail only when none are present.
* ci(gpu-e2e): self-provision base GPU AMI, hard-require it in preflight
The GPU E2E workflow was green-because-skipped: with no GPU AMIs imported
the self-guarding suite skipped every test, passing as a false green. Make
the raw-EC2 GPU passthrough test the always-on baseline.
- Add an import-if-absent setup step that pulls the prebuilt base GPU image
from the catalog by name (ubuntu-26.04-nvidia-gpu-x86_64), naming the AMI
exactly what tests/e2e/gpu filters for. Idempotent; caches on wattle's
persistent loop-fs. Runs spx under sg spinifex (local storage write).
- Preflight now hard-requires the base AMI (fail, not skip) so a silent
import failure can't pass as green; ECS/EKS node AMIs stay opt-in warn.
- Reconcile the local build path: ubuntu-gpu-nvidia.conf AMI_NAME and
launch-gpu.sh's nvidia default now use the same canonical name, so a
locally built+imported image also satisfies the suite.
* ci(gpu-e2e): pass --ami-name on base GPU import so the AMI name matches
spx admin images import --name selects the catalog entry but names the
resulting AMI generically (ami-<distro>-<version>-<arch>), so the import
landed as ami-ubuntu-26.04-x86_64 and the tests/e2e/gpu name filter never
matched. Pass --ami-name ubuntu-26.04-nvidia-gpu-x86_64 explicitly (same
flag build-system-image.sh uses) and poll briefly for availability.
* fix(network,vm): reconcile auto-assigned public-IP NAT + scale VFIO QMP timeout
Auto-assigned public IPs were never carried through the network reconciler.
applyEIPs ran AddEIP + ensureGuestPortDatapath for user EIPs only; auto-assigned
IPs (intent.Ports[].PublicIP) got egress exemption but no NAT re-assert and no
gatewayLRP->guest convergence. Their dnat_and_snat was created once at
RunInstances and never touched, so a vpcd/OVN rebuild dropped it and the
guest-port hop never converged: inbound DNATs at the gateway but never reaches
the guest (TCP/SSH dark) while ICMP is answered by the gateway LR (ping OK).
floatingIPSpecs now folds every auto-assigned/ELB public IP into the same
AddEIP + guest-port convergence path as user EIPs (deduped by private IP; a user
EIP wins), making the datapath self-healing across reconcile ticks and deploys.
qmpGreetingTimeout for VFIO guests now scales with guest RAM (base 30s + 8s/GiB,
floored at 180s, capped at 600s) instead of a flat 180s, so a large guest is not
SIGKILLed mid-pin while the whole guest RAM is IOMMU-locked before QMP answers.
* fix(vm): scale QMP handshake command deadline for VFIO launches
The greeting timeout scaling alone did not fix the VFIO launch SIGKILL: a GPU
guest sends its QMP greeting before the RAM pin completes, then cannot service
qmp_capabilities until the pin finishes. sendQMPCommand hardcoded a 30s response
deadline, so the handshake decode timed out (~43s for a 16 GiB g5.xlarge) and
the launch cleanup SIGKILLed a QEMU that would have come up.
Parameterize the command deadline (sendQMPCommandWithTimeout); the handshake now
uses the RAM-scaled qmpGreetingTimeout (base 30s + 8s/GiB, floor 180s, cap 600s)
for qmp_capabilities. Other callers keep the 30s default.
* test(gpu-e2e): share one GPU instance across passthrough subtests to avoid mid-suite IP recycle
Every tofu-based e2e workflow set TOFU_LOG to a /tmp path outside
ARTIFACT_DIR. ci-fmt.sh's quiet_run helper appends all tofu init/apply/
destroy output to that file, but the Teardown step (which runs the
destroy) always ran AFTER the existing upload-artifact step, so the
full log — including destroy-time failures — was silently discarded
on every run.
Add an if: always() upload-artifact step after Teardown in each
tofu-based job (e2e.yml, e2e-suites.yml, e2e-nightly.yml, e2e-ddil.yml)
that uploads $TOFU_LOG directly. Since quiet_run appends both apply and
destroy output to the same file, one upload after Teardown now
captures everything, recoverable even when destroy itself fails.
Shipping this log to Elasticsearch/OTLP is deferred to mulga-5n7fe,
which will consume this artifact.
* feat(otelsetup): OTel init, resource attrs, trace-stamping slog handler
Env-gated on OTEL_EXPORTER_OTLP_ENDPOINT: with no endpoint only the W3C
propagator is installed and globals stay no-op, so instrumented binaries
ship everywhere at zero cost. Resource carries service/host/mulga.env/
mulga.source plus OTEL_RESOURCE_ATTRIBUTES (ci.run_id in CI).
* feat(otelsetup): wire telemetry init into all spx services
Every spx service start command installs the trace-stamping JSON slog
default and the OTel providers via initTelemetry, flushed on shutdown.
Gateway's own slog setup now routes through otelsetup so its level
handling keeps trace correlation.
* feat(gateway): HTTP tracing middleware on awsgw and daemon cluster API
Server span per request with W3C traceparent extraction; awsgw spans are
renamed to the resolved SigV4 service.Action post-auth with account and
region attributes. Request log lines carry trace_id via context.
* feat(nats): propagate W3C trace context over command subjects
NATSRequestCtx injects traceparent beside X-Account-ID and opens a client
span per hop; ServeNATSRequest opens a consumer span joining the producer
trace, with a Ctx variant for handlers that thread context. RunInstances
is plumbed end-to-end (gateway dispatch through placement fan-out to the
daemon handler) and viperblockd ebs.mount/unmount join inbound traces.
* feat(systemd): source optional /etc/spinifex/telemetry.env in service units
Lets operators enable OTLP export (OTEL_EXPORTER_OTLP_ENDPOINT,
MULGA_ENV, MULGA_SOURCE) by dropping a single env file, with no unit
edits. File is optional; absent means telemetry stays disabled.
* feat(ci): export OTel telemetry from E2E envs to the central Elastic sink
All E2E workflows set SPINIFEX_OTLP_ENDPOINT (repo variable override,
default wattle) plus ci.run_id/ci.workflow resource attributes, so
bootstrap-install.sh writes telemetry.env on every node. Teardown runs
flush-telemetry.sh before the final tofu destroy so buffered spans and
metrics reach the sink.
* feat(otelsetup): request metrics and Go runtime metrics via OTLP
mulga.requests counter + mulga.request.duration histogram recorded by the
same middleware that opens spans: awsgw carries the resolved SigV4 action,
the daemon cluster API uses the chi route pattern, and NATS handlers record
per-topic at the subscribe chokepoints (node names stripped to keep
cardinality flat). Go runtime metrics start inside the same env gate.
* feat(daemon): deep spans for QMP commands, VM launch phases, EBS ops
Single chokepoint span per QMP command in sendQMPCommand covering all
callsites; vm.launch phase children (mount_volumes, start_qemu, qmp_connect,
persist_state); ec2.cmd consumer span with traceparent extraction plus
start/stop/reboot/terminate child ops; ebs.mount/unmount spans in
viperblockd with volume attributes.
* feat(telemetry): thread request context through NATS, object store and VM paths
Migrate utils.NATSRequest call sites to NATSRequestCtx (241 sites; W3C
traceparent now flows gateway -> daemon on every request), add GatherCtx and
migrate all Gather call sites, inject traceparent on manual EKS publishes.
ObjectStore interface is ctx-first with aws-sdk WithContext calls and an
otelhttp transport, so predastore spans join the caller trace. NATS consumer
spans record error status when the response envelope carries an error.
vm.Manager and the ec2 handler service interfaces take ctx-first so vm.launch
and QMP spans join the request trace; detached background work uses explicit
context.Background(). Hot-path slog calls use *Context variants for trace_id
binding.
* feat(telemetry): suppress health-check, heartbeat and unrooted trace noise
Health probes record metrics only (no span per poll); the QMP heartbeat
poller marks its context so query-status ticks stop rooting a trace each.
rootSampler honors MULGA_ROOT_TRACE_RATIO (ParentBased ratio, default 1.0)
and the predastore unit sets 0.05: guest block I/O roots are sampled while
request-linked spans are always kept.
* feat(otel): export deployment.environment resource attribute
Elastic APM maps it to service.environment, enabling the native environment
selector across the APM UI (Services, Traces, Dependencies).
* fix(telemetry): stop objectstore rooting method-only HTTP transactions
Background S3 calls (state sync, cleanup loops) carried no caller span, so
otelhttp rooted a trace per request and APM surfaced thousands of parentless
'HTTP GET/PUT/DELETE' transactions with expected-404 probes counted as
failures. Emit the client span only when the request context already carries
one, matching the viperblock backend. Bump viperblock/predastore to branch
heads so CI builds also pick up the equivalent span filter and root sampler.
* feat(telemetry): per-VM guest metrics via dedicated QMP socket + qmp-collector
Goanna Phase 1 producer, operator-consumed first. Each VM gets a second
-qmp chardev (qmp-telemetry-<id>.sock) plus a sidecar discovery file
carrying instance/account/vcpus/taps/monitoring-tier; the new
qmp-collector service polls query-cpus-fast (vCPU thread /proc jiffies),
query-blockstats, query-balloon and host tap sysfs counters, and
publishes CloudWatch-mappable goanna_ec2_* series to NATS
metrics.ec2.<instance-id>. A queue-group bridge in the same process
forwards the series to the node-local OTLP receiver so they reach the
operator sink until Goanna consumes the subjects. The collector never
touches the manager's control QMP socket; telemetry failures never
block a launch.
* fix(qmp-collector): dial NATS via DialTarget, drop ProtectProc, fix meta race
The collector connected to the raw NATS host (0.0.0.0), failing TLS
verification against a cert valid for 127.0.0.1; route through
admin.DialTarget like the other daemons.
ProtectProc=invisible hid the QEMU vCPU threads (dumpable=0) that the
collector must read from /proc/<tid>/stat, so every tick failed and
re-primed without ever publishing. Drop it to default.
Surface collect failures at Warn so silent metric loss is visible, drop
per-tick publish to Debug, and read InstanceID under the lock in run()
to fix a race with updateMeta on ENI hot-plug.
* chore(deps): re-pin predastore + viperblock to dev-landed commits
* feat(eks): multinode control-plane networking, IAM, and CoreDNS parity
Integrates the EKS multinode fix set for the managed-control-plane topology
so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS
EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the
node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it.
EKS:
- Route apiserver egress (exec/logs/webhooks) through konnectivity
(apiserver-network-proxy): CP servers run konnectivity-server behind the
cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod
traffic rides the agent-initiated reverse tunnel instead of an unroutable
CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every
replica (HA-correct).
- Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay
flannel mesh SG rule authorized at cluster launch with ordered teardown.
- Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay.
- Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master
tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity.
- Spread nodegroup workers across hosts; addon readiness falls back on
workload rollout when gvks are absent.
- elbv2: allow the proxy_protocol_v2.enabled target-group attribute.
EBS:
- Move volume attachment state to state.json so a live nbdkit VB cannot
clobber it; route mounted-volume state updates through the live VB with a
retry for in-use marks racing the mount responder; keep the boot volume
in-use across stop/crash unmount.
* fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM
missingOrchestrationDeps gated on s.deps.IAM, but production wires only the
lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only).
The gate therefore reported missing=[IAM] on every node forever, so
CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even
though the IAM service was fully up via the provider.
Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else
deps.IAMProvider()) the nodegroup/role code already uses — so the protective
check still rejects a node with no IAM service but passes when the provider
supplies one. Add a regression test that sets only IAMProvider and asserts the
gate is satisfied; the existing nil-both test still rejects.
* fix(eks): let eks-token-webhook use IMDS creds when static keys absent
The CP VM launches on an instance profile, so buildK3sUserData omits
EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys
before eksgw.New could fall back to the IMDS SDK chain, so the webhook
aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver
crashed on the missing --authentication-token-webhook-config-file file,
stalling the cluster at CREATING until it timed out to FAILED.
Drop the static-key precondition (matching the sibling eks-gateway-publish
and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/
cluster gates.
* fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane
The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt,
which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a
169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing
169.254.169.254 and steering IMDS off the data NIC away from the per-tap
br-imds datapath, where the host RST it. eks-gateway-publish then failed to
sign requests and no bootstrap artifacts were ever published, so clusters sat
CREATING until they timed out to FAILED.
Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address
or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI)
are unaffected and IMDS falls through the data NIC's default route to the
br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net
deletes any default route on a NIC declared NIC<n>_DEFAULT=0.
* fix(eks): grant eks:WebhookTokenReview to the control-plane instance role
The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway
broker at POST /clusters/{name}/token-review, signing with the control-plane
VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes
against eks:WebhookTokenReview, but the role's inline policy granted only
eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every
relay and the apiserver returned 401 to all external kubectl.
Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile
re-asserts the inline policy on every control-plane instance-profile ensure, and
the gateway evaluates authorization per request, so existing clusters converge
without a rebuild once the shared role is updated.
Cell 19 bootstraps with --external-mode=nat plus the CI static pool as
the public wan pool and runs the natuplink suite, covering the Tier-1
transit datapath and Tier-2 EIP ingress. The run step exits 0 when
natuplink.test is absent from the tests tar so scheduled runs against
branches predating the suite stay green.
Add a second build+publish step to the EKS AMI workflow so the same
manual run also builds the ecs-agent system AMI and uploads its qcow2 +
.sha256 sidecar to R2. Reuses the shared checkout, Go, and tooling setup;
renames the workflow to cover both node images.
* docs: add ECS (Elastic Container Service) guide + register ecs-quickstart workbook
Add a Container Services -> ECS doc mirroring the EKS/ECR shape: concepts
(cluster, task definition, task, service, container instances via the
ECS-AMI EC2 launch type), an aws ecs / console / Terraform quickstart,
task IAM roles via the credential endpoint, awsvpc networking, and the
honest v1 limitations (no service discovery, awslogs discarded, no rolling
deployments, no capacity providers, no distinct execution role).
Convert the ecs-quickstart workbook README to the doc-page format
(frontmatter + overview/prerequisites/instructions/troubleshooting) so it
renders as a Terraform Workbooks entry alongside the EKS workbooks.
* docs: add node-image prerequisite banner + infra diagram to EKS and ECS pages
Add a GitHub-style [!IMPORTANT] banner to the EKS and ECS prerequisites
flagging the required node-image import (eks-node / spinifex-ecs-node),
mirroring the install page's WAN-bridge banner, each with a describe-images
verification command.
Add a basic Spinifex topology diagram (SVG) to each Overview, in the
tf-nginx-alb visual style: the EKS managed control plane + node-group
workers, and the ECS ALB + container instances with the gateway/agent
and task-credential path.
The per-run ARTIFACT_DIR lives in /tmp, outside GITHUB_WORKSPACE, so a
re-run on the same self-hosted runner reused a prior attempt's append-only
suite-status.txt. The 'Fail job if any suite failed' gate's -s check then
failed the job even when the surviving attempt passed with rc=0.
Wipe ARTIFACT_DIR in Clean Workspace and gate solely on the per-attempt
steps.e2e.outputs.rc (fail-closed: an empty rc still fails).
docker export of a container leaks its overlay mount on Docker's
overlay2 driver — the mount survives docker rm, so the layer dirs are
never reaped except by a daemon restart. The lb-agent microVM build ran
docker export on every e2e job across ~9 runners, accumulating ~69G of
orphaned overlay2 layers on the CI host (disk hit 94%).
Build the rootfs with `docker buildx --output type=local` from a small
generated Dockerfile instead, which writes the image filesystem to a
directory and releases all mounts — nothing leaks. Keep the podman path
(its store is unaffected). Tighten the e2e cleanup image prune to
dangling-only so the alpine base stays cached instead of re-pulling.
* test(e2e): add targeted suite selector + shadow-mode CI step
select-suites maps a diff to the e2e suites it must run: changed files →
services (manifest path/additional_paths) → consumer closure over
depends_on vs publishes/subscribes → suites whose covers intersect.
Special cases force-all (infra/harness/workflow, ≥3 services, submodule
bumps) or a lone suite dir; empty diffs fall back to default suites. An
AST scan of each suite's harness.Ensure* call graph derives per-test
service coverage and emits an optional -test.run subset (RUN_PATTERN).
Wired into e2e.yml as a non-gating 'Select Suites (shadow)' job that
logs the computed selection to the run summary so predicted vs actual
can be compared before the selector drives SUITES.
* ci(e2e): add dispatch-only per-suite job blocks grouped by env
Restructure the looped e2e flow into env-grouped provision -> matrix
suites -> teardown blocks so each suite surfaces as its own node in the
Actions graph instead of a single green tick:
single_provision -> single_suite [single,cert,lb,eks] -> single_teardown
multi_provision -> multi_suite [multinode] -> multi_teardown
Provision resolves the VM/cluster and ships test binaries, emitting WAN
IP + ssh key path as job outputs; each matrix leg runs one suite over
SSH and uploads its own junit/analysis artifacts; teardown (if:always)
reuses the persisted tofu workspace on the same runner to destroy.
workflow_dispatch only -- runs alongside the gating e2e.yml for
validation and posts no commit status.
* test(e2e): add manifest drift lint (fixture-create + NATS subject)
Bead 5 of the targeted-suite-selection plan. Two ratcheted drift guards in
tests/e2e/manifest-lint, wired into make preflight and the spinifex.yml PR job:
- Fixture lint: AST-flags direct resource-create AWS calls
(c.EC2.RunInstances, c.ELBv2.CreateLoadBalancer, ...) in suite test files,
steering them through harness.Ensure* fixtures. ~104 pre-existing call sites
are captured in a checked-in baseline; the lint fails only on NEW creates.
Intentional create-path tests opt out with an inline // e2e:allow-create.
- Subject lint: extracts NATS subject literals from the natsSub{} registry and
literal Subscribe/Publish/Request args, cross-referenced wildcard-aware
against service-interfaces.yaml. Code subjects missing from the manifest fail
(ratcheted); manifest subjects with no static reference warn only (many are
produced dynamically). Fixed real drift surfaced on first run: declared the
five spinifex.cluster.shutdown.* phase subjects under spinifex_daemon.
manifest-check is now CI-enforced alongside manifest-lint (previously
preflight/local-only).
* feat: run-scoped e2e teardown sweep
Tag every Ensure* fixture resource with e2e:run=<run-id> when
SPINIFEX_E2E_RUN_ID is set, and add a teardown.test suite that sweeps
everything carrying that tag at end of run — reclaiming resources leaked
by a crashed suite, an interrupted run, or a persistent cluster.
- harness/sweep.go: tagRunResources (EC2 CreateTags) + tagRunELB (ELBv2
AddTags) + SweepRunResources in dependency order (LB, instances, NAT,
snapshots, volumes, images, SGs, subnets, key pairs). Blank run id
no-ops both tagging and sweeping, so the sweep can never touch an
untagged or foreign resource.
- harness/fixtures.go: stamp the run tag on each created resource.
- tests/e2e/teardown: thin suite that runs the sweep; skips without a
cluster or run id, logs the report, never fails the run.
- e2e-suites.yml: build + ship teardown.test, pass the run id into suite
legs, sweep on the VM in both teardown jobs before tofu destroy.
* ci(e2e-nightly): run Go harness suites instead of bash run-*.sh
Swap nightly cells 1-3 (single), 8-10 (real-multi) and 18 (reboot) to the
compiled <suite>.test binaries, mirroring e2e.yml: SPINIFEX_E2E=1 with
SPINIFEX_MODE single|multinode, per-suite logs to ARTIFACT_DIR, then
go-junit-report + e2e-analyze + upload-artifact. Adds a spinifex checkout for
the composite actions and pulls /run/spinifex/console-*.log so LB microVM
(lb-agent) state is captured on failure. Cells 17 (tofu) and 20 (baremetal)
stay on bash — no Go suite exists.
* ci(e2e-nightly): cell-17 RCA analytics + cell-20 cosmetic alignment
run-tofu-examples-e2e.sh now emits go test -v markers per workbook so the
cell-17 step's tee'd test-tofu.log flows through the same go-junit-report ->
e2e-analyze -> upload-artifact chain as the Go suites, producing junit-tofu.xml
and an analysis bundle for RCA. Align e2e-baremetal.yml (cell 20) ARTIFACT_DIR,
analyze title, and artifact name with the matrix cells so CI-RCA discovers its
bundle uniformly.
* ci(e2e-nightly): wire host journal + VM consoles into RCA bundles
Collect now emits the canonical spinifex-journal.log (concat of all nodes)
that the e2e-analyze bundler slices into each failure's journal.log, so
elbv2-service/vpcd/daemon lines in the failure window travel with the bundle
(previously the journal uploaded as spinifex-journal-node*.log, a name the
bundler never reads). Rename the console capture to vm-console-node* and note
it also carries guest cloud-init/sshd consoles, covering the cell-3 internal-LB,
cell-18 reboot-datapath, and cell-9/10 guest-ssh blind spots.
* ci(e2e): self-heal runner /dev + --one-file-system on workspace clean
Clean Workspace now detaches any leaked iso-builder rootfs mount, recreates
/dev/null if a prior build clobbered it, and uses rm --one-file-system so the
clean can't descend into a still-mounted devtmpfs and delete host device nodes.
Guards both the matrix cells and the bare-metal job against the host-wide
/dev/null wipe that fails go build with 'open /dev/null: no such file or
directory'.
* ci(eks-ami): build on self-hosted ci-single-1 runner
GitHub-hosted ubuntu-24.04 cannot run the libguestfs appliance network:
passt fails (PID file denied under its AppArmor confine / privilege drop)
and the SLIRP fallback has no outbound, so virt-customize --network dies.
The self-hosted ci-single-1 runner has /dev/kvm and real networking, so
the existing libguestfs build works unmodified.
* fix(build-system-image): add --no-cache, force fresh build on publish
The <10min cached-raw skip avoids duplicate work for concurrent e2e
jobs, but on the persistent publish runner it could republish a stale
image from a prior or concurrent run. Add --no-cache to bypass the skip
and have publish-system-image.sh pass it on every build.
* ci(eks-ami): target ci-single label (no ci-single-1 label exists)
The self-hosted runners are named ci-single/-2/-3 but all share the
single label 'ci-single'; runs-on matches labels, not names, so
ci-single-1 never schedules.
* ci(eks-ami): disable Go cache on self-hosted runner
setup-go's cache-save (Post Setup Go) re-uploaded the Go cache to the
Actions backend on every run, taking ~2.5min vs the 36s build. The
self-hosted runner already keeps the cache on local disk, so cache:false.
* Revert "fix(network): prime host neigh with gateway port MAC on centralised EIP attach"
This reverts commit f7446c69c6.
* fix(network): round-robin external pool cursor so a released IP is not reused instantly
The static external pool handed out the lowest free address and freed IPs
immediately on release, so a just-released EIP was reassigned within ~0.6s. A
recycled EIP front-end is unreachable from the same host: the kernel still
holds the prior owner's ARP binding (or a negative cache after the neigh
flush) and OVN does not answer the re-ARP in time, so the neighbour entry
settles to FAILED (EHOSTUNREACH). This surfaced as EKS TestEKS/KubectlGetNodes
failing in single-node CI once #315 routed kubectl through the cluster NLB's
external EIP. Dev never hit it — it uses DHCP-random addresses.
Add a persisted per-pool cursor: allocation resumes one past the last address
handed out and wraps at RangeEnd, so a freed IP is not reused until the cursor
cycles the whole range. Empty cursor (fresh pool / older KV record) starts at
RangeStart, preserving lowest-free order for the first cycle — backward
compatible, no migration.
* ci: drop e2e-eks from default e2e suites
EKS data-plane subtest (TestEKS/KubectlGetNodes) is a persistent flake on
single-node CI — a freshly-attached centralised EIP is intermittently
unreachable from the host (no route to host to the cluster endpoint), which
fails the whole E2E run and blocks unrelated suites. Remove e2e-eks from the
default SUITES so PR/push runs no longer gate on it. The e2e-eks target and
the workflow_dispatch suites input are unchanged, so EKS stays runnable on
demand and via make e2e-all.
* feat(eks): publish eks-node AMI to R2 + spx catalogue entry
Add a spinifex-eks-node catalogue entry so operators install the EKS
node image with a single command:
spx admin images import --name spinifex-eks-node
The entry pulls the Alpine+K3s image from iso.mulgadc.com, verifies its
sha256, and stamps spinifex:managed-by=eks — the tag the control-plane
and node-group launchers resolve the boot AMI by, so no extra config is
needed.
Add scripts/publish-system-image.sh (+ make publish-eks-node-image) to
convert the built raw to a compressed qcow2, emit a checksum sidecar,
and upload both to the R2 bucket backing iso.mulgadc.com/system-ami via
the S3 API.
* ci(eks): add manual workflow to build + publish eks-node AMI to R2
Build the eks-node AMI on the runner (libguestfs) and publish via
publish-system-image.sh using the existing CF_R2_* secrets. Manual
workflow_dispatch with a dry-run input. Fix the publish script's default
bucket to spinifex-iso (the bucket iso.mulgadc.com is served from).
EKS create-cluster fails deterministically on the multinode job with
ServiceUnavailableException: the EKS KV is a clustered (Raft) JetStream
stream and create-cluster needs an elected leader, but multinode
JetStream clustering isn't reliably forming yet (see mulga-omq/z2z/5yg).
EKS is validated single-node only at this stage.
Drop eks from the multinode build+run suite filters (alongside the
existing reboot/single exclusions) so the multinode job reflects what's
actually supported. Single-node continues to run the full eks suite.
Make e2e-baremetal.yml a reusable workflow (workflow_call + a ref input)
so the nightly and release pipelines can invoke the real PXE-reimaged
hardware suite, which previously only ran on manual dispatch.
Nightly: add a baremetal job (conceptual cell 20) that runs every
weeknight on the existing cron, or on dispatch when 20 is selected. It
tests dev on schedule, matching GIT_BRANCH. Sits outside the tofu matrix
(different runner, provisioning, 120m timeout) and shares the
e2e-baremetal-cluster concurrency group, so all baremetal runs serialise
on the single physical node.
Release: a release builds and uploads an ISO to R2 but never booted that
ISO/squashfs path on hardware. Add a baremetal job (needs: release) and
hard-gate on it: promotion moves to a dedicated promote job
needs:[verify, baremetal] (with a matching mark-failed job on failure),
and the iso job's upload gate now waits for both verify and baremetal to
conclude success before pushing to R2. Bump the iso job timeout to 180m
since it now waits on the parallel ~2h baremetal run.
build-microvm-image required depmod (kmod) as a mandatory tool to
generate modules.dep. Self-hosted CI runners don't ship kmod and can't
reliably apt-install it, so the inline 'apt-get install kmod' steps added
to e2e/e2e-nightly/e2e-ddil were unreliable.
modules.dep is an optimisation, not a boot requirement: the guest init's
load_mod() (build/microvm/init.sh) falls back to insmod-by-path when
modprobe can't resolve a module. So:
- Drop depmod from the mandatory-tool preflight.
- Run depmod only when present; still assert a non-empty modules.dep in
that case to keep catching the trixie-kmod empty-dep bug.
- Warn and skip when depmod is absent (guest uses insmod fallback).
- Remove the apt-get kmod install steps from all three workflows.
Gold-image bake still installs kmod (provision.sh), so production images
get a proper modules.dep; only kmod-less CI runners take the fallback.
* ci(e2e): install kmod (depmod) before microVM image build
#305 made depmod a hard requirement in build-microvm-image.sh, but the
self-hosted ci-single/ci-multi runners don't ship the kmod package, so the
Build Install Artifacts step aborts with 'required tool not found: depmod'
before any E2E suite runs. Add an idempotent kmod install ahead of each
Build Install Artifacts step (single/multi/eks jobs); the gold-image builder
bakes kmod in too, so this no-ops once runners pick up the new image.
* ci: install kmod (depmod) in nightly + ddil too
e2e-nightly and e2e-ddil call the same build-install-artifacts.sh (-> make
build-microvm-image) on the same self-hosted ci-single/ci-multi runners as
e2e.yml, so they hit the identical 'depmod not found' abort. Add the same
idempotent kmod guard before Build Install Artifacts in both.
release distro builds run on GitHub-hosted ubuntu (kmod present); e2e-baremetal
runs on ci-baremetal which already has it (green today) — neither needs the
guard.
* test(eks): EKS control-plane e2e suite (Phase 1, API-level)
Add tests/e2e/eks/eks_test.go driving the EKS lifecycle against awsgw:
CreateCluster -> ACTIVE (endpoint + CA asserted), kubeconfig artifact
assembly, AccessEntry CRUD, get-token presign decode, DeleteCluster -> gone.
Harness: EKS SDK client on AWSClient + WaitForEKSCluster{Active,Deleted}
pollers. Wire e2e-eks into the Makefile and the single-node CI suite list.
* test(eks): Phase 2 kubectl get-nodes data-plane gate
Add the reachability shim (harness/eks_mgmt.go) that reads the control-plane
VM's br-mgmt IP from the cluster KV meta over NATS, plus a kubectl wrapper
(harness/kubectl.go) that skips when kubectl is absent. KubectlGetNodes points
kubectl at https://<mgmtIP>:6443 (insecure-skip-tls-verify; cert SANs the NLB
DNS, not the mgmt IP) with an aws eks get-token exec block and polls until the
control-plane node is Ready — the get-token webhook auth regression gate.
IRSA pod + the real NLB/Route53 passthrough are deferred to a follow-up.
* test(eks): tolerate doubled NotFound code + dial loopback NATS
DeleteCluster detection matched the SDK constant ResourceNotFoundException,
but awsgw emits ResourceNotFoundExceptionException on the wire — match the
NotFound code by substring so deletion is detected immediately instead of
timing out. The mgmt-IP shim dialed NATS.Host verbatim (0.0.0.0:4222), which
the serving cert does not SAN; rewrite the wildcard bind to loopback.
Validated live: all 6 subtests green, kubectl get nodes shows the
control-plane node Ready via the get-token webhook auth path.
* feat: build system images with libguestfs (no host block devices)
Replace the qemu-nbd + sudo mount + chroot + /dev-bind flow in
build-system-image.sh with libguestfs (virt-customize / virt-resize /
guestfish). Every operation runs in an isolated userspace appliance VM with
its own kernel, so the build touches no host block device, no host mount, and
needs no sudo — removing the failure class that wiped the host's /dev on shared
CI runners (a leaked /dev bind traversed by rm -rf).
Applies to all distros: alpine resizes via guestfish e2fsck+resize2fs, ubuntu
via virt-resize; customization (packages, binaries, files, setup script,
service enable, cache cleanup) is one ordered virt-customize invocation.
LIBGUESTFS_BACKEND=direct (no libvirt on runners).
Wire the eks-node AMI into the single-node e2e job: the runner builds the
device-free raw, ships it to the VM, and the VM imports it (predastore/NATS are
VM-only) so the eks suite's CreateCluster has an AMI to boot.
* feat(eks): bootstrap + state-report via AWS gateway broker
Converge the EKS control-plane → host channel onto the ELBv2 lb-agent model:
the K3s server VM publishes its bootstrap envelopes and periodic state reports
via SigV4-signed HTTPS POST to the AWS gateway, instead of dialing core NATS
directly. A new gateway action (PublishInternal, POST
/clusters/{name}/internal-publish) relays the payload verbatim onto the
existing eks.bus.* / eks.state.* subjects, so the daemon-side subscriber, KV
persistence/encryption, and reconciler are unchanged.
This removes the VM's need to reach NATS at all, so NATS can stay
cluster-internal behind its bind/firewall/mTLS controls — the prior direct
path broke when NATS was not VM-reachable (e2e CreateCluster timeout) and was
DDIL-fragile (fire-and-forget nats pub, no ack/retry).
- gateway/eks: PublishInternal broker + route; closed-set channel/kind validation
- cmd/eks-gateway-publish: one-shot SigV4 publisher (retry-until-ack) baked into
the eks-node AMI; replaces the nats-cli publisher in k3s-first-boot.sh and
mulga-eks-state-report.sh
- handlers/eks + daemon: hand the K3s server VM the mgmt-reachable gateway URL +
system (Predastore) SigV4 creds + gateway CA instead of NATS URL/token/CA;
new resolveSystemGatewayBaseURL (mgmt-bridge-first)
- eks-node image: build/install eks-gateway-publish, drop nats-cli download
* feat: migrate eks-token-webhook to the AWS gateway broker
The TokenReview webhook dialed core NATS directly (STS verify + AccessEntry
KV read). On a mgmt-only control-plane VM with NATS bound off-subnet that
fails, so the webhook exited at startup, the apiserver token-webhook
kubeconfig was never written, and the apiserver crash-looped — the cluster
never reached ACTIVE.
Converge it onto the same gateway-broker model as bootstrap/state-report and
ELBv2's lb-agent: the webhook SigV4-signs an HTTPS POST {accountId,token} to
/clusters/{name}/token-review, and the gateway runs the verify + lookup
host-side and returns the resolved identity. The VM now speaks no core NATS.
- internal/eksgw: shared on-VM SigV4 client (also adopted by
eks-gateway-publish).
- handlers/eks/token_review.go: Authenticate decision core (moved from the
webhook binary) + ResolveTokenReview host-side wiring.
- gateway/eks token-review route + WebhookTokenReview handler.
- webhook: drop NATS, relay via eksgw.Client; broker fault -> 503 (retryable)
not a silent deny.
- initd: export EKS_GATEWAY_* / EKS_ACCESS_* creds, not SPINIFEX_NATS_*.
* fix: build eks-node helpers with GOFIPS140; cover token-review broker
eks-gateway-publish blank-imports internal/fipsboot, whose init() panics
unless the binary is built with GOFIPS140=v1.0.0. The eks-node AMI build
omitted the flag, so the binary paniced at first boot right after k3s came
up — k3s-first-boot (set -eu) died before publishing the bootstrap token and
the cluster never reached ACTIVE. Build both eks-node helpers with
GOFIPS140=v1.0.0, matching cmd/lb-agent.
Add NATS-backed unit tests for ResolveTokenReview and the WebhookTokenReview
gateway handler (verify responder + account KV) to cover the new broker path.
* fix: export first-boot env so eks-gateway-publish child sees it
k3s-first-boot.sh and mulga-eks-state-report.sh sourced /etc/spinifex-eks/
first-boot.env but never exported the vars, so the eks-gateway-publish child
process inherited an empty environment and exited '--account-id is required'.
The control-plane VM came up healthy but never published its bootstrap
envelopes, so the cluster never reached ACTIVE. Wrap the source in set -a /
set +a so the KEY=value lines export to child processes.
* fix: make awsgw start --host override actually bind
The 'awsgw start --host' override printed 'Overwriting awsgw host' but the
assignment was commented out, so AWSGW always bound the spinifex.toml host
(a specific WAN IP, e.g. 192.168.0.10). A system instance (EKS K3s control
plane, internal LB agent) has a mgmt-only NIC on-link to the br-mgmt IP with
no route to the WAN IP, so its SigV4 POSTs to the gateway were refused and the
cluster never bootstrapped. Honour --host (e2e passes 0.0.0.0:9999) so AWSGW
listens on all interfaces incl. br-mgmt; the serving cert already SANs every
local interface IP, and WAN clients still reach it.
* test(eks): install kubectl on e2e VM so KubectlGetNodes runs
The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full
aws eks get-token -> kubectl -> apiserver -> token-webhook -> gateway broker
-> STS verify path — the only end-to-end exercise of the token-review broker.
The e2e binaries run on the VM, where kubectl was absent, so
harness.NewKubectl t.Skip'd it and the path went unvalidated. Install a
checksum-verified kubectl (pinned to the k3s server version) on the VM,
gated on e2e-eks.
Bare-metal e2e produced no artifact, so RCA fell back to truncated
log-failed output. Tee bootstrap + e2e output to test-baremetal.log,
generate junit-baremetal.xml, run e2e-analyze to emit analysis.md, and
upload the bundle as e2e-baremetal-artifacts-<run_id>.
The iso job mounts a kernel-global devtmpfs under the workspace
(.cache/rootfs/dev). A killed job skips its EXIT-trap unmount and leaks
the bind; the verify job (same ci-single runner) then ran a bare
'rm -rf $WORKSPACE/*' that could traverse it and wipe the host /dev.
Detach any mount under the workspace before deleting and add
--one-file-system so rm never crosses a mount boundary. Same backstop
added to the iso job's own cleanup.
A killed ubuntu system-image build can leave `mount --bind /dev` under
its build dir. The e2e Clean Workspace step then ran
`sudo rm -rf "$GITHUB_WORKSPACE"/*` as root, traversed the leaked bind,
and deleted the host's /dev (incl. /dev/null), wedging the runner.
- e2e.yml: umount anything under the workspace (deepest-first) before
deleting, and pass `rm --one-file-system` so rm can never cross a mount
boundary into host /dev. Hard backstop, applied to all 3 Clean Workspace
steps.
- build-system-image.sh: recursive lazy unmount in cleanup() detaches the
chroot binds even when busy, and mark the build subtree rprivate so the
/dev bind can't propagate into the host namespace.
NLB target health stalls at 0/N in CI while the data plane works locally.
The lb-agent's nginx engine activation, any nginx -t/reload error, and its
active-probe results are logged to the LB microVM serial console — which the
host journal never captures and the GetConsoleOutput API path reports as
NotFound, so the failure is undiagnosable from CI artifacts.
- e2e.yml: pull /run/spinifex/console-*.log off each node in the single- and
multi-node collect steps (files persist past VM teardown) into lb-console/.
- lbagent healthcheck: log probe failures at Warn (was Debug) so unreachable
targets surface at the agent's default level on the captured console.
Ben Duncan