Both jobs filtered SUITES by exclusion, so any suite named in neither
deny-list ran in both jobs. That is how lb came to run twice per e2e run,
and it makes adding a suite silently fan it out everywhere.
Each job now names the suites it is eligible to run. Selection is still
driven by SUITES, so this is behaviour-preserving for the current default:
single/iam/cert on the single-node job, multinode/lb/cert on the multi-node
job, exactly as before. eks, ecs and storagegrowth are listed as eligible on
the single-node job but are not yet in the default SUITES.
Both e2e jobs filter SUITES by exclusion, and neither excluded lb, so the
suite ran twice per e2e run: 354s on single and 654s on multi. Balancing
across instances on one node is not a deployment mode we support, and
ELBv2 proper requires two AZs, so the single-node run was redundant.
Drops ~354s from the single-node job, which is the gate critical path.
Runs the in-process integration suite first, on the ci-single runner host
rather than inside a provisioned VM, so a broken control plane is caught
before either live job pays the Spinifex install. Both e2e_single and
e2e_multi now require it, and e2e_status treats a skipped integration job as
a failure so the gating status cannot go green on a gate that never ran.
Drops the shadow-mode suite selector: targeted per-diff suite selection is
not the direction being pursued, and leaving a non-functional shadow job in
the gating workflow only invited it to be mistaken for a live gate.
Add e2e_es_summary, a self-hosted job that aggregates suite-status.txt
from both e2e legs and POSTs one doc per run to the Elastic sink, keyed
by labels.ci_run_id, so Kibana can answer "did run X pass, and where
are its artifacts" without hand-aggregating traces-apm*/logs-apm.error*.
Runs on ci-single (LAN route to the sink) since e2e_status stays on
GitHub-hosted ubuntu-26.04 for the PR status-report check. State is
computed with the same single_ok/multi_ok logic as e2e_status so the
two can never disagree.
The Build/Lint/Race jobs checked out viperblock and predastore at a hardcoded
ref: dev and overlaid them with a go workspace, so the go.mod pins were ignored
and a spinifex change depending on an unmerged sibling change could never
compile in CI. Resolve each sibling to a branch of the same name when it exists
on that repo, else dev, mirroring the e2e path in
scripts/tofu-cluster/build-install-artifacts.sh so unit CI and e2e CI agree.
The publish-eks-ami matrix fans out four legs on the single persistent
ci-single runner, all reusing one workspace root. go work init is
create-only and checkout never scrubs the root go.work, so legs after the
first fail with 'go.work already exists'. Remove any stale go.work before
init so each leg re-inits cleanly.
Expand the publish workflow to build the two Ubuntu NVIDIA GPU variants
alongside the Alpine eks-node/ecs-agent AMIs. The image build+publish
scripts are already manifest-driven and Ubuntu-aware; only the workflow
lacked the GPU legs.
Refactor the single-job body into a matrix over the four image manifests,
resolved by a new 'variant' input (all|cpu|gpu) so the slow 16G GPU legs
can be skipped when only the CPU AMIs need republishing. GPU images do not
fit a RAM-backed /tmp, so set SYSTEM_IMAGE_BUILD_DIR onto the runner's
on-disk _temp; legs run serially (max-parallel 1) on the single ci runner.
* ci: GPU e2e workflow on ci-gpu self-hosted runner
Runs tests/e2e/gpu (EC2/ECS/EKS GPU passthrough) against the persistent,
hand-bootstrapped wattle shared host, driven over SSH by a banksia runner
labelled ci-gpu (matches the ci-single/ci-multi convention) — no runner
installed on the shared box.
- runs-on [self-hosted, ci-gpu]; concurrency ci-gpu, cancel-in-progress
false (single A1000 = single-flight, queue not stomp)
- workflow_dispatch + GPU-path filter (not every push)
- manual-work mutex gate (~/.ci-gpu-paused / CI_GPU_PAUSED) so hands-on GPU
testing blocks CI
- if: always() cleanup terminates launched GPU instances; never resets the
box (no setup-ovn.sh/dev-reset/cluster-reset)
- drives the control plane with spx + the already-configured aws spinifex
profile directly (no wrapper scripts, nothing to install on wattle)
* ci(gpu-e2e): advisory AMI preflight, fail only if zero GPU AMIs
The suite is self-guarding — each test skips when its AMI is absent. The
hard-fail on the base ubuntu GPU AMI blocked the whole run even when the
ECS GPU AMI (the only one imported) could execute. Inventory the AMIs and
fail only when none are present.
* ci(gpu-e2e): self-provision base GPU AMI, hard-require it in preflight
The GPU E2E workflow was green-because-skipped: with no GPU AMIs imported
the self-guarding suite skipped every test, passing as a false green. Make
the raw-EC2 GPU passthrough test the always-on baseline.
- Add an import-if-absent setup step that pulls the prebuilt base GPU image
from the catalog by name (ubuntu-26.04-nvidia-gpu-x86_64), naming the AMI
exactly what tests/e2e/gpu filters for. Idempotent; caches on wattle's
persistent loop-fs. Runs spx under sg spinifex (local storage write).
- Preflight now hard-requires the base AMI (fail, not skip) so a silent
import failure can't pass as green; ECS/EKS node AMIs stay opt-in warn.
- Reconcile the local build path: ubuntu-gpu-nvidia.conf AMI_NAME and
launch-gpu.sh's nvidia default now use the same canonical name, so a
locally built+imported image also satisfies the suite.
* ci(gpu-e2e): pass --ami-name on base GPU import so the AMI name matches
spx admin images import --name selects the catalog entry but names the
resulting AMI generically (ami-<distro>-<version>-<arch>), so the import
landed as ami-ubuntu-26.04-x86_64 and the tests/e2e/gpu name filter never
matched. Pass --ami-name ubuntu-26.04-nvidia-gpu-x86_64 explicitly (same
flag build-system-image.sh uses) and poll briefly for availability.
* fix(network,vm): reconcile auto-assigned public-IP NAT + scale VFIO QMP timeout
Auto-assigned public IPs were never carried through the network reconciler.
applyEIPs ran AddEIP + ensureGuestPortDatapath for user EIPs only; auto-assigned
IPs (intent.Ports[].PublicIP) got egress exemption but no NAT re-assert and no
gatewayLRP->guest convergence. Their dnat_and_snat was created once at
RunInstances and never touched, so a vpcd/OVN rebuild dropped it and the
guest-port hop never converged: inbound DNATs at the gateway but never reaches
the guest (TCP/SSH dark) while ICMP is answered by the gateway LR (ping OK).
floatingIPSpecs now folds every auto-assigned/ELB public IP into the same
AddEIP + guest-port convergence path as user EIPs (deduped by private IP; a user
EIP wins), making the datapath self-healing across reconcile ticks and deploys.
qmpGreetingTimeout for VFIO guests now scales with guest RAM (base 30s + 8s/GiB,
floored at 180s, capped at 600s) instead of a flat 180s, so a large guest is not
SIGKILLed mid-pin while the whole guest RAM is IOMMU-locked before QMP answers.
* fix(vm): scale QMP handshake command deadline for VFIO launches
The greeting timeout scaling alone did not fix the VFIO launch SIGKILL: a GPU
guest sends its QMP greeting before the RAM pin completes, then cannot service
qmp_capabilities until the pin finishes. sendQMPCommand hardcoded a 30s response
deadline, so the handshake decode timed out (~43s for a 16 GiB g5.xlarge) and
the launch cleanup SIGKILLed a QEMU that would have come up.
Parameterize the command deadline (sendQMPCommandWithTimeout); the handshake now
uses the RAM-scaled qmpGreetingTimeout (base 30s + 8s/GiB, floor 180s, cap 600s)
for qmp_capabilities. Other callers keep the 30s default.
* test(gpu-e2e): share one GPU instance across passthrough subtests to avoid mid-suite IP recycle
Every tofu-based e2e workflow set TOFU_LOG to a /tmp path outside
ARTIFACT_DIR. ci-fmt.sh's quiet_run helper appends all tofu init/apply/
destroy output to that file, but the Teardown step (which runs the
destroy) always ran AFTER the existing upload-artifact step, so the
full log — including destroy-time failures — was silently discarded
on every run.
Add an if: always() upload-artifact step after Teardown in each
tofu-based job (e2e.yml, e2e-suites.yml, e2e-nightly.yml, e2e-ddil.yml)
that uploads $TOFU_LOG directly. Since quiet_run appends both apply and
destroy output to the same file, one upload after Teardown now
captures everything, recoverable even when destroy itself fails.
Shipping this log to Elasticsearch/OTLP is deferred to mulga-5n7fe,
which will consume this artifact.
* feat(otelsetup): OTel init, resource attrs, trace-stamping slog handler
Env-gated on OTEL_EXPORTER_OTLP_ENDPOINT: with no endpoint only the W3C
propagator is installed and globals stay no-op, so instrumented binaries
ship everywhere at zero cost. Resource carries service/host/mulga.env/
mulga.source plus OTEL_RESOURCE_ATTRIBUTES (ci.run_id in CI).
* feat(otelsetup): wire telemetry init into all spx services
Every spx service start command installs the trace-stamping JSON slog
default and the OTel providers via initTelemetry, flushed on shutdown.
Gateway's own slog setup now routes through otelsetup so its level
handling keeps trace correlation.
* feat(gateway): HTTP tracing middleware on awsgw and daemon cluster API
Server span per request with W3C traceparent extraction; awsgw spans are
renamed to the resolved SigV4 service.Action post-auth with account and
region attributes. Request log lines carry trace_id via context.
* feat(nats): propagate W3C trace context over command subjects
NATSRequestCtx injects traceparent beside X-Account-ID and opens a client
span per hop; ServeNATSRequest opens a consumer span joining the producer
trace, with a Ctx variant for handlers that thread context. RunInstances
is plumbed end-to-end (gateway dispatch through placement fan-out to the
daemon handler) and viperblockd ebs.mount/unmount join inbound traces.
* feat(systemd): source optional /etc/spinifex/telemetry.env in service units
Lets operators enable OTLP export (OTEL_EXPORTER_OTLP_ENDPOINT,
MULGA_ENV, MULGA_SOURCE) by dropping a single env file, with no unit
edits. File is optional; absent means telemetry stays disabled.
* feat(ci): export OTel telemetry from E2E envs to the central Elastic sink
All E2E workflows set SPINIFEX_OTLP_ENDPOINT (repo variable override,
default wattle) plus ci.run_id/ci.workflow resource attributes, so
bootstrap-install.sh writes telemetry.env on every node. Teardown runs
flush-telemetry.sh before the final tofu destroy so buffered spans and
metrics reach the sink.
* feat(otelsetup): request metrics and Go runtime metrics via OTLP
mulga.requests counter + mulga.request.duration histogram recorded by the
same middleware that opens spans: awsgw carries the resolved SigV4 action,
the daemon cluster API uses the chi route pattern, and NATS handlers record
per-topic at the subscribe chokepoints (node names stripped to keep
cardinality flat). Go runtime metrics start inside the same env gate.
* feat(daemon): deep spans for QMP commands, VM launch phases, EBS ops
Single chokepoint span per QMP command in sendQMPCommand covering all
callsites; vm.launch phase children (mount_volumes, start_qemu, qmp_connect,
persist_state); ec2.cmd consumer span with traceparent extraction plus
start/stop/reboot/terminate child ops; ebs.mount/unmount spans in
viperblockd with volume attributes.
* feat(telemetry): thread request context through NATS, object store and VM paths
Migrate utils.NATSRequest call sites to NATSRequestCtx (241 sites; W3C
traceparent now flows gateway -> daemon on every request), add GatherCtx and
migrate all Gather call sites, inject traceparent on manual EKS publishes.
ObjectStore interface is ctx-first with aws-sdk WithContext calls and an
otelhttp transport, so predastore spans join the caller trace. NATS consumer
spans record error status when the response envelope carries an error.
vm.Manager and the ec2 handler service interfaces take ctx-first so vm.launch
and QMP spans join the request trace; detached background work uses explicit
context.Background(). Hot-path slog calls use *Context variants for trace_id
binding.
* feat(telemetry): suppress health-check, heartbeat and unrooted trace noise
Health probes record metrics only (no span per poll); the QMP heartbeat
poller marks its context so query-status ticks stop rooting a trace each.
rootSampler honors MULGA_ROOT_TRACE_RATIO (ParentBased ratio, default 1.0)
and the predastore unit sets 0.05: guest block I/O roots are sampled while
request-linked spans are always kept.
* feat(otel): export deployment.environment resource attribute
Elastic APM maps it to service.environment, enabling the native environment
selector across the APM UI (Services, Traces, Dependencies).
* fix(telemetry): stop objectstore rooting method-only HTTP transactions
Background S3 calls (state sync, cleanup loops) carried no caller span, so
otelhttp rooted a trace per request and APM surfaced thousands of parentless
'HTTP GET/PUT/DELETE' transactions with expected-404 probes counted as
failures. Emit the client span only when the request context already carries
one, matching the viperblock backend. Bump viperblock/predastore to branch
heads so CI builds also pick up the equivalent span filter and root sampler.
* feat(telemetry): per-VM guest metrics via dedicated QMP socket + qmp-collector
Goanna Phase 1 producer, operator-consumed first. Each VM gets a second
-qmp chardev (qmp-telemetry-<id>.sock) plus a sidecar discovery file
carrying instance/account/vcpus/taps/monitoring-tier; the new
qmp-collector service polls query-cpus-fast (vCPU thread /proc jiffies),
query-blockstats, query-balloon and host tap sysfs counters, and
publishes CloudWatch-mappable goanna_ec2_* series to NATS
metrics.ec2.<instance-id>. A queue-group bridge in the same process
forwards the series to the node-local OTLP receiver so they reach the
operator sink until Goanna consumes the subjects. The collector never
touches the manager's control QMP socket; telemetry failures never
block a launch.
* fix(qmp-collector): dial NATS via DialTarget, drop ProtectProc, fix meta race
The collector connected to the raw NATS host (0.0.0.0), failing TLS
verification against a cert valid for 127.0.0.1; route through
admin.DialTarget like the other daemons.
ProtectProc=invisible hid the QEMU vCPU threads (dumpable=0) that the
collector must read from /proc/<tid>/stat, so every tick failed and
re-primed without ever publishing. Drop it to default.
Surface collect failures at Warn so silent metric loss is visible, drop
per-tick publish to Debug, and read InstanceID under the lock in run()
to fix a race with updateMeta on ENI hot-plug.
* chore(deps): re-pin predastore + viperblock to dev-landed commits
Cell 19 bootstraps with --external-mode=nat plus the CI static pool as
the public wan pool and runs the natuplink suite, covering the Tier-1
transit datapath and Tier-2 EIP ingress. The run step exits 0 when
natuplink.test is absent from the tests tar so scheduled runs against
branches predating the suite stay green.
Add a second build+publish step to the EKS AMI workflow so the same
manual run also builds the ecs-agent system AMI and uploads its qcow2 +
.sha256 sidecar to R2. Reuses the shared checkout, Go, and tooling setup;
renames the workflow to cover both node images.
The per-run ARTIFACT_DIR lives in /tmp, outside GITHUB_WORKSPACE, so a
re-run on the same self-hosted runner reused a prior attempt's append-only
suite-status.txt. The 'Fail job if any suite failed' gate's -s check then
failed the job even when the surviving attempt passed with rc=0.
Wipe ARTIFACT_DIR in Clean Workspace and gate solely on the per-attempt
steps.e2e.outputs.rc (fail-closed: an empty rc still fails).
docker export of a container leaks its overlay mount on Docker's
overlay2 driver — the mount survives docker rm, so the layer dirs are
never reaped except by a daemon restart. The lb-agent microVM build ran
docker export on every e2e job across ~9 runners, accumulating ~69G of
orphaned overlay2 layers on the CI host (disk hit 94%).
Build the rootfs with `docker buildx --output type=local` from a small
generated Dockerfile instead, which writes the image filesystem to a
directory and releases all mounts — nothing leaks. Keep the podman path
(its store is unaffected). Tighten the e2e cleanup image prune to
dangling-only so the alpine base stays cached instead of re-pulling.
* test(e2e): add targeted suite selector + shadow-mode CI step
select-suites maps a diff to the e2e suites it must run: changed files →
services (manifest path/additional_paths) → consumer closure over
depends_on vs publishes/subscribes → suites whose covers intersect.
Special cases force-all (infra/harness/workflow, ≥3 services, submodule
bumps) or a lone suite dir; empty diffs fall back to default suites. An
AST scan of each suite's harness.Ensure* call graph derives per-test
service coverage and emits an optional -test.run subset (RUN_PATTERN).
Wired into e2e.yml as a non-gating 'Select Suites (shadow)' job that
logs the computed selection to the run summary so predicted vs actual
can be compared before the selector drives SUITES.
* ci(e2e): add dispatch-only per-suite job blocks grouped by env
Restructure the looped e2e flow into env-grouped provision -> matrix
suites -> teardown blocks so each suite surfaces as its own node in the
Actions graph instead of a single green tick:
single_provision -> single_suite [single,cert,lb,eks] -> single_teardown
multi_provision -> multi_suite [multinode] -> multi_teardown
Provision resolves the VM/cluster and ships test binaries, emitting WAN
IP + ssh key path as job outputs; each matrix leg runs one suite over
SSH and uploads its own junit/analysis artifacts; teardown (if:always)
reuses the persisted tofu workspace on the same runner to destroy.
workflow_dispatch only -- runs alongside the gating e2e.yml for
validation and posts no commit status.
* test(e2e): add manifest drift lint (fixture-create + NATS subject)
Bead 5 of the targeted-suite-selection plan. Two ratcheted drift guards in
tests/e2e/manifest-lint, wired into make preflight and the spinifex.yml PR job:
- Fixture lint: AST-flags direct resource-create AWS calls
(c.EC2.RunInstances, c.ELBv2.CreateLoadBalancer, ...) in suite test files,
steering them through harness.Ensure* fixtures. ~104 pre-existing call sites
are captured in a checked-in baseline; the lint fails only on NEW creates.
Intentional create-path tests opt out with an inline // e2e:allow-create.
- Subject lint: extracts NATS subject literals from the natsSub{} registry and
literal Subscribe/Publish/Request args, cross-referenced wildcard-aware
against service-interfaces.yaml. Code subjects missing from the manifest fail
(ratcheted); manifest subjects with no static reference warn only (many are
produced dynamically). Fixed real drift surfaced on first run: declared the
five spinifex.cluster.shutdown.* phase subjects under spinifex_daemon.
manifest-check is now CI-enforced alongside manifest-lint (previously
preflight/local-only).
* feat: run-scoped e2e teardown sweep
Tag every Ensure* fixture resource with e2e:run=<run-id> when
SPINIFEX_E2E_RUN_ID is set, and add a teardown.test suite that sweeps
everything carrying that tag at end of run — reclaiming resources leaked
by a crashed suite, an interrupted run, or a persistent cluster.
- harness/sweep.go: tagRunResources (EC2 CreateTags) + tagRunELB (ELBv2
AddTags) + SweepRunResources in dependency order (LB, instances, NAT,
snapshots, volumes, images, SGs, subnets, key pairs). Blank run id
no-ops both tagging and sweeping, so the sweep can never touch an
untagged or foreign resource.
- harness/fixtures.go: stamp the run tag on each created resource.
- tests/e2e/teardown: thin suite that runs the sweep; skips without a
cluster or run id, logs the report, never fails the run.
- e2e-suites.yml: build + ship teardown.test, pass the run id into suite
legs, sweep on the VM in both teardown jobs before tofu destroy.
* ci(e2e-nightly): run Go harness suites instead of bash run-*.sh
Swap nightly cells 1-3 (single), 8-10 (real-multi) and 18 (reboot) to the
compiled <suite>.test binaries, mirroring e2e.yml: SPINIFEX_E2E=1 with
SPINIFEX_MODE single|multinode, per-suite logs to ARTIFACT_DIR, then
go-junit-report + e2e-analyze + upload-artifact. Adds a spinifex checkout for
the composite actions and pulls /run/spinifex/console-*.log so LB microVM
(lb-agent) state is captured on failure. Cells 17 (tofu) and 20 (baremetal)
stay on bash — no Go suite exists.
* ci(e2e-nightly): cell-17 RCA analytics + cell-20 cosmetic alignment
run-tofu-examples-e2e.sh now emits go test -v markers per workbook so the
cell-17 step's tee'd test-tofu.log flows through the same go-junit-report ->
e2e-analyze -> upload-artifact chain as the Go suites, producing junit-tofu.xml
and an analysis bundle for RCA. Align e2e-baremetal.yml (cell 20) ARTIFACT_DIR,
analyze title, and artifact name with the matrix cells so CI-RCA discovers its
bundle uniformly.
* ci(e2e-nightly): wire host journal + VM consoles into RCA bundles
Collect now emits the canonical spinifex-journal.log (concat of all nodes)
that the e2e-analyze bundler slices into each failure's journal.log, so
elbv2-service/vpcd/daemon lines in the failure window travel with the bundle
(previously the journal uploaded as spinifex-journal-node*.log, a name the
bundler never reads). Rename the console capture to vm-console-node* and note
it also carries guest cloud-init/sshd consoles, covering the cell-3 internal-LB,
cell-18 reboot-datapath, and cell-9/10 guest-ssh blind spots.
* ci(e2e): self-heal runner /dev + --one-file-system on workspace clean
Clean Workspace now detaches any leaked iso-builder rootfs mount, recreates
/dev/null if a prior build clobbered it, and uses rm --one-file-system so the
clean can't descend into a still-mounted devtmpfs and delete host device nodes.
Guards both the matrix cells and the bare-metal job against the host-wide
/dev/null wipe that fails go build with 'open /dev/null: no such file or
directory'.
* ci(eks-ami): build on self-hosted ci-single-1 runner
GitHub-hosted ubuntu-24.04 cannot run the libguestfs appliance network:
passt fails (PID file denied under its AppArmor confine / privilege drop)
and the SLIRP fallback has no outbound, so virt-customize --network dies.
The self-hosted ci-single-1 runner has /dev/kvm and real networking, so
the existing libguestfs build works unmodified.
* fix(build-system-image): add --no-cache, force fresh build on publish
The <10min cached-raw skip avoids duplicate work for concurrent e2e
jobs, but on the persistent publish runner it could republish a stale
image from a prior or concurrent run. Add --no-cache to bypass the skip
and have publish-system-image.sh pass it on every build.
* ci(eks-ami): target ci-single label (no ci-single-1 label exists)
The self-hosted runners are named ci-single/-2/-3 but all share the
single label 'ci-single'; runs-on matches labels, not names, so
ci-single-1 never schedules.
* ci(eks-ami): disable Go cache on self-hosted runner
setup-go's cache-save (Post Setup Go) re-uploaded the Go cache to the
Actions backend on every run, taking ~2.5min vs the 36s build. The
self-hosted runner already keeps the cache on local disk, so cache:false.
* Revert "fix(network): prime host neigh with gateway port MAC on centralised EIP attach"
This reverts commit f7446c69c6.
* fix(network): round-robin external pool cursor so a released IP is not reused instantly
The static external pool handed out the lowest free address and freed IPs
immediately on release, so a just-released EIP was reassigned within ~0.6s. A
recycled EIP front-end is unreachable from the same host: the kernel still
holds the prior owner's ARP binding (or a negative cache after the neigh
flush) and OVN does not answer the re-ARP in time, so the neighbour entry
settles to FAILED (EHOSTUNREACH). This surfaced as EKS TestEKS/KubectlGetNodes
failing in single-node CI once #315 routed kubectl through the cluster NLB's
external EIP. Dev never hit it — it uses DHCP-random addresses.
Add a persisted per-pool cursor: allocation resumes one past the last address
handed out and wraps at RangeEnd, so a freed IP is not reused until the cursor
cycles the whole range. Empty cursor (fresh pool / older KV record) starts at
RangeStart, preserving lowest-free order for the first cycle — backward
compatible, no migration.
* ci: drop e2e-eks from default e2e suites
EKS data-plane subtest (TestEKS/KubectlGetNodes) is a persistent flake on
single-node CI — a freshly-attached centralised EIP is intermittently
unreachable from the host (no route to host to the cluster endpoint), which
fails the whole E2E run and blocks unrelated suites. Remove e2e-eks from the
default SUITES so PR/push runs no longer gate on it. The e2e-eks target and
the workflow_dispatch suites input are unchanged, so EKS stays runnable on
demand and via make e2e-all.
* feat(eks): publish eks-node AMI to R2 + spx catalogue entry
Add a spinifex-eks-node catalogue entry so operators install the EKS
node image with a single command:
spx admin images import --name spinifex-eks-node
The entry pulls the Alpine+K3s image from iso.mulgadc.com, verifies its
sha256, and stamps spinifex:managed-by=eks — the tag the control-plane
and node-group launchers resolve the boot AMI by, so no extra config is
needed.
Add scripts/publish-system-image.sh (+ make publish-eks-node-image) to
convert the built raw to a compressed qcow2, emit a checksum sidecar,
and upload both to the R2 bucket backing iso.mulgadc.com/system-ami via
the S3 API.
* ci(eks): add manual workflow to build + publish eks-node AMI to R2
Build the eks-node AMI on the runner (libguestfs) and publish via
publish-system-image.sh using the existing CF_R2_* secrets. Manual
workflow_dispatch with a dry-run input. Fix the publish script's default
bucket to spinifex-iso (the bucket iso.mulgadc.com is served from).
EKS create-cluster fails deterministically on the multinode job with
ServiceUnavailableException: the EKS KV is a clustered (Raft) JetStream
stream and create-cluster needs an elected leader, but multinode
JetStream clustering isn't reliably forming yet (see mulga-omq/z2z/5yg).
EKS is validated single-node only at this stage.
Drop eks from the multinode build+run suite filters (alongside the
existing reboot/single exclusions) so the multinode job reflects what's
actually supported. Single-node continues to run the full eks suite.
Make e2e-baremetal.yml a reusable workflow (workflow_call + a ref input)
so the nightly and release pipelines can invoke the real PXE-reimaged
hardware suite, which previously only ran on manual dispatch.
Nightly: add a baremetal job (conceptual cell 20) that runs every
weeknight on the existing cron, or on dispatch when 20 is selected. It
tests dev on schedule, matching GIT_BRANCH. Sits outside the tofu matrix
(different runner, provisioning, 120m timeout) and shares the
e2e-baremetal-cluster concurrency group, so all baremetal runs serialise
on the single physical node.
Release: a release builds and uploads an ISO to R2 but never booted that
ISO/squashfs path on hardware. Add a baremetal job (needs: release) and
hard-gate on it: promotion moves to a dedicated promote job
needs:[verify, baremetal] (with a matching mark-failed job on failure),
and the iso job's upload gate now waits for both verify and baremetal to
conclude success before pushing to R2. Bump the iso job timeout to 180m
since it now waits on the parallel ~2h baremetal run.
build-microvm-image required depmod (kmod) as a mandatory tool to
generate modules.dep. Self-hosted CI runners don't ship kmod and can't
reliably apt-install it, so the inline 'apt-get install kmod' steps added
to e2e/e2e-nightly/e2e-ddil were unreliable.
modules.dep is an optimisation, not a boot requirement: the guest init's
load_mod() (build/microvm/init.sh) falls back to insmod-by-path when
modprobe can't resolve a module. So:
- Drop depmod from the mandatory-tool preflight.
- Run depmod only when present; still assert a non-empty modules.dep in
that case to keep catching the trixie-kmod empty-dep bug.
- Warn and skip when depmod is absent (guest uses insmod fallback).
- Remove the apt-get kmod install steps from all three workflows.
Gold-image bake still installs kmod (provision.sh), so production images
get a proper modules.dep; only kmod-less CI runners take the fallback.
* ci(e2e): install kmod (depmod) before microVM image build
#305 made depmod a hard requirement in build-microvm-image.sh, but the
self-hosted ci-single/ci-multi runners don't ship the kmod package, so the
Build Install Artifacts step aborts with 'required tool not found: depmod'
before any E2E suite runs. Add an idempotent kmod install ahead of each
Build Install Artifacts step (single/multi/eks jobs); the gold-image builder
bakes kmod in too, so this no-ops once runners pick up the new image.
* ci: install kmod (depmod) in nightly + ddil too
e2e-nightly and e2e-ddil call the same build-install-artifacts.sh (-> make
build-microvm-image) on the same self-hosted ci-single/ci-multi runners as
e2e.yml, so they hit the identical 'depmod not found' abort. Add the same
idempotent kmod guard before Build Install Artifacts in both.
release distro builds run on GitHub-hosted ubuntu (kmod present); e2e-baremetal
runs on ci-baremetal which already has it (green today) — neither needs the
guard.
* test(eks): EKS control-plane e2e suite (Phase 1, API-level)
Add tests/e2e/eks/eks_test.go driving the EKS lifecycle against awsgw:
CreateCluster -> ACTIVE (endpoint + CA asserted), kubeconfig artifact
assembly, AccessEntry CRUD, get-token presign decode, DeleteCluster -> gone.
Harness: EKS SDK client on AWSClient + WaitForEKSCluster{Active,Deleted}
pollers. Wire e2e-eks into the Makefile and the single-node CI suite list.
* test(eks): Phase 2 kubectl get-nodes data-plane gate
Add the reachability shim (harness/eks_mgmt.go) that reads the control-plane
VM's br-mgmt IP from the cluster KV meta over NATS, plus a kubectl wrapper
(harness/kubectl.go) that skips when kubectl is absent. KubectlGetNodes points
kubectl at https://<mgmtIP>:6443 (insecure-skip-tls-verify; cert SANs the NLB
DNS, not the mgmt IP) with an aws eks get-token exec block and polls until the
control-plane node is Ready — the get-token webhook auth regression gate.
IRSA pod + the real NLB/Route53 passthrough are deferred to a follow-up.
* test(eks): tolerate doubled NotFound code + dial loopback NATS
DeleteCluster detection matched the SDK constant ResourceNotFoundException,
but awsgw emits ResourceNotFoundExceptionException on the wire — match the
NotFound code by substring so deletion is detected immediately instead of
timing out. The mgmt-IP shim dialed NATS.Host verbatim (0.0.0.0:4222), which
the serving cert does not SAN; rewrite the wildcard bind to loopback.
Validated live: all 6 subtests green, kubectl get nodes shows the
control-plane node Ready via the get-token webhook auth path.
* feat: build system images with libguestfs (no host block devices)
Replace the qemu-nbd + sudo mount + chroot + /dev-bind flow in
build-system-image.sh with libguestfs (virt-customize / virt-resize /
guestfish). Every operation runs in an isolated userspace appliance VM with
its own kernel, so the build touches no host block device, no host mount, and
needs no sudo — removing the failure class that wiped the host's /dev on shared
CI runners (a leaked /dev bind traversed by rm -rf).
Applies to all distros: alpine resizes via guestfish e2fsck+resize2fs, ubuntu
via virt-resize; customization (packages, binaries, files, setup script,
service enable, cache cleanup) is one ordered virt-customize invocation.
LIBGUESTFS_BACKEND=direct (no libvirt on runners).
Wire the eks-node AMI into the single-node e2e job: the runner builds the
device-free raw, ships it to the VM, and the VM imports it (predastore/NATS are
VM-only) so the eks suite's CreateCluster has an AMI to boot.
* feat(eks): bootstrap + state-report via AWS gateway broker
Converge the EKS control-plane → host channel onto the ELBv2 lb-agent model:
the K3s server VM publishes its bootstrap envelopes and periodic state reports
via SigV4-signed HTTPS POST to the AWS gateway, instead of dialing core NATS
directly. A new gateway action (PublishInternal, POST
/clusters/{name}/internal-publish) relays the payload verbatim onto the
existing eks.bus.* / eks.state.* subjects, so the daemon-side subscriber, KV
persistence/encryption, and reconciler are unchanged.
This removes the VM's need to reach NATS at all, so NATS can stay
cluster-internal behind its bind/firewall/mTLS controls — the prior direct
path broke when NATS was not VM-reachable (e2e CreateCluster timeout) and was
DDIL-fragile (fire-and-forget nats pub, no ack/retry).
- gateway/eks: PublishInternal broker + route; closed-set channel/kind validation
- cmd/eks-gateway-publish: one-shot SigV4 publisher (retry-until-ack) baked into
the eks-node AMI; replaces the nats-cli publisher in k3s-first-boot.sh and
mulga-eks-state-report.sh
- handlers/eks + daemon: hand the K3s server VM the mgmt-reachable gateway URL +
system (Predastore) SigV4 creds + gateway CA instead of NATS URL/token/CA;
new resolveSystemGatewayBaseURL (mgmt-bridge-first)
- eks-node image: build/install eks-gateway-publish, drop nats-cli download
* feat: migrate eks-token-webhook to the AWS gateway broker
The TokenReview webhook dialed core NATS directly (STS verify + AccessEntry
KV read). On a mgmt-only control-plane VM with NATS bound off-subnet that
fails, so the webhook exited at startup, the apiserver token-webhook
kubeconfig was never written, and the apiserver crash-looped — the cluster
never reached ACTIVE.
Converge it onto the same gateway-broker model as bootstrap/state-report and
ELBv2's lb-agent: the webhook SigV4-signs an HTTPS POST {accountId,token} to
/clusters/{name}/token-review, and the gateway runs the verify + lookup
host-side and returns the resolved identity. The VM now speaks no core NATS.
- internal/eksgw: shared on-VM SigV4 client (also adopted by
eks-gateway-publish).
- handlers/eks/token_review.go: Authenticate decision core (moved from the
webhook binary) + ResolveTokenReview host-side wiring.
- gateway/eks token-review route + WebhookTokenReview handler.
- webhook: drop NATS, relay via eksgw.Client; broker fault -> 503 (retryable)
not a silent deny.
- initd: export EKS_GATEWAY_* / EKS_ACCESS_* creds, not SPINIFEX_NATS_*.
* fix: build eks-node helpers with GOFIPS140; cover token-review broker
eks-gateway-publish blank-imports internal/fipsboot, whose init() panics
unless the binary is built with GOFIPS140=v1.0.0. The eks-node AMI build
omitted the flag, so the binary paniced at first boot right after k3s came
up — k3s-first-boot (set -eu) died before publishing the bootstrap token and
the cluster never reached ACTIVE. Build both eks-node helpers with
GOFIPS140=v1.0.0, matching cmd/lb-agent.
Add NATS-backed unit tests for ResolveTokenReview and the WebhookTokenReview
gateway handler (verify responder + account KV) to cover the new broker path.
* fix: export first-boot env so eks-gateway-publish child sees it
k3s-first-boot.sh and mulga-eks-state-report.sh sourced /etc/spinifex-eks/
first-boot.env but never exported the vars, so the eks-gateway-publish child
process inherited an empty environment and exited '--account-id is required'.
The control-plane VM came up healthy but never published its bootstrap
envelopes, so the cluster never reached ACTIVE. Wrap the source in set -a /
set +a so the KEY=value lines export to child processes.
* fix: make awsgw start --host override actually bind
The 'awsgw start --host' override printed 'Overwriting awsgw host' but the
assignment was commented out, so AWSGW always bound the spinifex.toml host
(a specific WAN IP, e.g. 192.168.0.10). A system instance (EKS K3s control
plane, internal LB agent) has a mgmt-only NIC on-link to the br-mgmt IP with
no route to the WAN IP, so its SigV4 POSTs to the gateway were refused and the
cluster never bootstrapped. Honour --host (e2e passes 0.0.0.0:9999) so AWSGW
listens on all interfaces incl. br-mgmt; the serving cert already SANs every
local interface IP, and WAN clients still reach it.
* test(eks): install kubectl on e2e VM so KubectlGetNodes runs
The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full
aws eks get-token -> kubectl -> apiserver -> token-webhook -> gateway broker
-> STS verify path — the only end-to-end exercise of the token-review broker.
The e2e binaries run on the VM, where kubectl was absent, so
harness.NewKubectl t.Skip'd it and the path went unvalidated. Install a
checksum-verified kubectl (pinned to the k3s server version) on the VM,
gated on e2e-eks.
Bare-metal e2e produced no artifact, so RCA fell back to truncated
log-failed output. Tee bootstrap + e2e output to test-baremetal.log,
generate junit-baremetal.xml, run e2e-analyze to emit analysis.md, and
upload the bundle as e2e-baremetal-artifacts-<run_id>.
The iso job mounts a kernel-global devtmpfs under the workspace
(.cache/rootfs/dev). A killed job skips its EXIT-trap unmount and leaks
the bind; the verify job (same ci-single runner) then ran a bare
'rm -rf $WORKSPACE/*' that could traverse it and wipe the host /dev.
Detach any mount under the workspace before deleting and add
--one-file-system so rm never crosses a mount boundary. Same backstop
added to the iso job's own cleanup.
A killed ubuntu system-image build can leave `mount --bind /dev` under
its build dir. The e2e Clean Workspace step then ran
`sudo rm -rf "$GITHUB_WORKSPACE"/*` as root, traversed the leaked bind,
and deleted the host's /dev (incl. /dev/null), wedging the runner.
- e2e.yml: umount anything under the workspace (deepest-first) before
deleting, and pass `rm --one-file-system` so rm can never cross a mount
boundary into host /dev. Hard backstop, applied to all 3 Clean Workspace
steps.
- build-system-image.sh: recursive lazy unmount in cleanup() detaches the
chroot binds even when busy, and mark the build subtree rprivate so the
/dev bind can't propagate into the host namespace.
NLB target health stalls at 0/N in CI while the data plane works locally.
The lb-agent's nginx engine activation, any nginx -t/reload error, and its
active-probe results are logged to the LB microVM serial console — which the
host journal never captures and the GetConsoleOutput API path reports as
NotFound, so the failure is undiagnosable from CI artifacts.
- e2e.yml: pull /run/spinifex/console-*.log off each node in the single- and
multi-node collect steps (files persist past VM teardown) into lb-console/.
- lbagent healthcheck: log probe failures at Warn (was Debug) so unreachable
targets surface at the agent's default level on the captured console.
Two bugs in afd63ed9:
1. `sudo mkdir` makes /tmp/spinifex-diag root-owned 0755. The
subsequent `> /tmp/spinifex-diag/tcpdump.log` is performed by
the calling tf-user shell BEFORE sudo gets the command, failing
with Permission denied. action shell runs with `set -e` so the
step aborts.
2. `sudo nohup ... &; echo $!` captures the wrong PID — `$!` is the
backgrounded sudo, which may exit when its target dies, leaving
stale tcpdump.pid.
Fixes:
- chmod 0777 on /tmp/spinifex-diag after creating it (next steps'
redirects from tf-user shell now work).
- Wrap the tcpdump launch in `sudo bash -c '...'` so backgrounding
and `$!` happen inside the sudo'd shell — `$!` is the actual
tcpdump PID (nohup execs, sudo bash holds it). Captures stdin
to /dev/null, stdout/stderr to /tmp/spinifex-diag/tcpdump.log.
- sleep 2 (was 1) — gives tcpdump time to fully init pcap header.
- Hard fail (exit 1) if PID is missing or process dead, so the
step's failure is visible instead of a silent green.
bash -n verified.
Mulga-siv-167 Phase 1. Adds evidence-capture for the LB
single-host return-path drop class:
e2e_single only:
- New step 'Start Packet Capture' between Bootstrap and Run E2E
Tests. SSHes to test VM and starts background tcpdump on br-wan
capturing TCP/22/80/443/9999 + ARP into /tmp/spinifex-diag/br-wan.pcap.
PID stored at /tmp/spinifex-diag/tcpdump.pid.
All three jobs (single, pseudo-multi, real-multi):
- Collect Diagnostics: kill tcpdump (single only — PID file
guarded) -INT for graceful flush, falls through to -KILL after 5s.
- Adds journal-spinifex-full.log (no -n truncation) alongside
the existing -n 10000 capped one.
- Adds bridge fdb show, bridge link, and hairpin_mode/learning/flood
for every br-wan brif — needed to confirm/disconfirm the
hairpin-loss hypothesis in the plan doc.
- Adds 'chmod -R a+r /tmp/spinifex-diag' before tar to ensure pcap
(created by sudo tcpdump as root) is readable by tf-user.
bash -n verified on all four rendered scripts. YAML parses.
Zero new gold-image deps (tcpdump, bridge present in base image).
Adds a Collect Diagnostics + Upload Diagnostics step pair to all three
E2E jobs (single, pseudo-multi, real-multi). Runs before Teardown,
if: always() so green runs produce baselines too. SSHes to test
VM(s), dumps:
- journalctl for spinifex-*, ovn-*, openvswitch-*
- ovs-vsctl/ofctl/appctl: bridge state, br-int + br-ex flows,
dpctl flow + conntrack tables
- ovn-nbctl/sbctl: show, Port_Binding, Chassis, NAT, Logical_Router
- ovn-appctl ct-zone-list (LR -> ct_zone mapping)
- ip addr/route/neigh, iptables-save, nft list, dmesg, virsh list
Tarred, scp'd back, uploaded as e2e-diag-{single,pseudo,multi}-<run>
artifact (14-day retention).
OVS-native ovs-appctl dpctl/dump-conntrack — no new gold-image deps.
Multi-node job writes script to RUNNER_TEMP then ssh-stdin's it
per-node (avoids YAML/heredoc indent traps inside for-loop). bash -n
verified on all three rendered scripts.
Unblocks siv-111 debug loop: bash LB driver does not emit an
analysis bundle on failure, so post-W2 recurrences have been blind.
Workflow-level capture is hypothesis-free.
joshsiv-mulga