The release workflow granted contents: write to every job, including the distro builds and the self-hosted verify job, which only read. Only release, promote, mark-failed and iso create or edit the GitHub release, so they alone get write and the rest default to read.
setup-go caches by default, and a release or dev prerelease run restores caches that any dev or main run saved. A poisoned module cache from one of those runs would then flow into the shipped binaries, so both release builds now start from a clean cache.
* feat(oci): separate using an instance principal from creating one
The allocator needs OCI credentials at runtime, and a node with none forms,
passes every health check, then refuses every launch wanting a public address.
Instance principal is the credential route that puts no key material on a node,
but it needs a dynamic group and a policy, and those are tenancy-root resources
the compartment-scoped user the rest of the configuration runs as cannot create.
enable_instance_principal conflated the two, so the better credential route was
only reachable by a tenancy admin. instance_principal now takes off, adopt or
create: adopt selects the credential and creates nothing, because the matching
rule covers the whole compartment and so names no instance to reference.
setup-identity.sh owns the create case, with its own state. validate-topology.sh
destroys its state at the end of every run, so holding tenancy resources there
would let a teardown delete the tenancy's policy.
Also package install-node.sh and smoke-test.sh, so a cluster can be formed from
a curl | bash install with no checkout. smoke-test.sh is not optional: formation
scp's it from its own directory for the post-formation verification.
* feat(oci): take the credential handoff from an operator hook, and widen the workflow name
An API-key deployment needs a credential on each node, and nothing in this
repository can supply one: a key in user-data is readable from instance metadata
for the life of the instance, and this host runs other people's guests. A manual
install does not survive either, because a topology destroys itself on success.
So the harness calls an operator-owned executable after formation and before the
pool, the only window where a node has /etc/spinifex and has not started the
allocator. Instance principal needs no handoff, so the hook is skipped there.
The workflow is renamed for the vendors it will cover rather than the first one.
Nothing is orphaned in the Actions UI: it has never been on the default branch
and has never run, so there is no history attached to the old filename.
* docs(oci): the API-key path needs the config file, not just the PEM
oci_config_file names an ordinary OCI SDK config and nothing creates it, so a node
holding only the key forms and then fails every allocation with no stated cause.
* fix(ci): the runner context is not available in a job-level env block
A workflow referencing it there is rejected before any job starts, which is why
the rename produced a failed run attributed to the push. The reference had never
been validated: a dispatch-only workflow that is never dispatched is never parsed.
* fix(deps): bump containerd to v2.4.1 for GO-2026-6597
The image-pull DoS via crafted OCI index graph amplification is reachable from the ECS agent's containerd puller, so the vulnerability scan fails preflight on v2.4.0. A patch release with no API change.
* fix(e2e): probe guest public addresses from outside the cloud's own network
A VCN does not reflect its own public addresses back into itself, so a node asking for one gets nothing however well the datapath works. The workbook driver runs on the first node, so every assertion that curls or SSHes a guest's public address failed on OCI against a cluster that was serving those addresses correctly to the internet.
The driver now takes E2E_PUBLIC_PROXY, a SOCKS5 proxy used for public addresses only, and validate-topology.sh supplies one with a remote dynamic forward on the connection that launches the driver. Egress is then the host running the validation, which is outside the VCN and is where a customer sits. Unset leaves every other platform on a direct probe, which is already correct there.
Also adds --version TAG for a repeatable install, separates the workflow's auth choice from its external-pool choice so an API key can run with the allocator live, and captures allocator diagnostics before the teardown destroys them.
* ci: read the Go version from go.mod in the release workflows
Both pinned 1.27.0 by hand while every other workflow reads the file, so the toolchain they build with drifts from the one the module declares the day go.mod moves. release.yml was already inconsistent with itself: its second setup-go step reads spinifex/go.mod.
* feat: publish the OCI topology results as tables on the run page
A green tick on one job with 4,000 lines of log behind it is confidence for
whoever wrote the harness and nothing to anybody else. Someone assessing whether
Spinifex works on their cloud should be able to see that it formed and served
guests on a single VM, on three VMs and on bare metal without opening a log.
record now takes <gate> <status> [detail] and writes results.tsv beside the
human block, so a gate's status is a field the script sets rather than a regex
over prose that reads fine and matches wrong. record_workbooks turns the
driver's own go-test-shaped lines into workbooks.tsv, and runs on the failing
branch too: which four passed is most of what the fifth failing means.
e2e-oci.sh --summary-only renders both tables to stdout and to
GITHUB_STEP_SUMMARY, and the workflow calls it under always() before the sweep,
because the sweep destroys the state directories the tables are read from. A
skipped gate gets its own mark, and a topology that never ran says so: the one
way a results table can do real harm is a reader taking an absent row for a
passing one.
* docs(oci): cut the deploy-by-hand appendix, and validate the config for real
The appendix was 444 of 1014 lines restating an install the Terraform already
does, which made the document read as though deploying by hand were a normal
route. It is not: the configuration builds the VCN, the volumes and both VNICs
per node and renders the cloud-init, in the same order every time. What is
genuinely OCI-specific is four things, each of which already has a section, so
the replacement names those and sends everyone else to the single-node and
multi-node install guides.
Verify end to end told the reader to ping a guest's public address from the
host, which cannot work: a VCN does not route its own public addresses back into
itself. That is the finding this branch is built on and the document was
asserting the opposite. It now says where to probe from and why, and carries the
allocator's `resolved the external VNIC` check, which existed only in the cut
appendix and is the one line proving the credentials and compartment are right.
`spx config validate` is not a command -- spx has admin, get, service, top and
version -- so that line could only ever fail and its output and status were both
discarded. It is replaced by parsing the TOML, which is what was wanted: the
remap and the pool append are textual edits, and a broken result would otherwise
surface as every service failing to start with nothing saying why. The remap now
checks its own substitution count for the same reason, since re.sub with no
match rewrites the file unchanged and exits 0.
Publication of the run-page tables moves to one owner. GITHUB_STEP_SUMMARY is
per step, so appending from the run as well as the publish step would have put
two copies of every table on the run page.
* fix(oci): check the credential hook before building anything
The hook is a path from a CI variable, and whether it exists on the runner is
knowable before Terraform runs. Checked where it was used, a wrong path cost the
apply, the install and the formation first -- forty minutes and a bare-metal bill
to be told a file is missing. Nothing about the check depends on the apply, so it
moves to the top, where --dry-run reaches it too.
* fix(oci): the summary must say what was built and what was not tested
The first CI run headed its tables "Published release `latest channel`" over a
build of a feature branch, which is the one error in a results table that would
mislead a reader who trusted it. The header now states a tree build as a tree
build, and names how the allocator authenticated.
A run with no external pool says so in the header rather than leaving it to a
SKIPPED row further down. Public-address allocation is the claim a green table is
most likely to be read as making, and it is exactly the claim such a run cannot
support. Both flags are passed to the publish step too, so the summary rendered
after a failure describes the same run as the one that failed.
Raise the dev prerelease prune from five to twenty. A pinned dev tag is how an
external evaluator installs a specific build, and five is about a week of them --
short enough that the tag someone was handed stops resolving while they are still
using it.
* docs(oci): the workbook table is a snapshot, and the nightly is the current answer
It is dated against a build five releases of work ago, and the nightly now runs
the same workbooks on all three topologies and publishes its own table. Say which
one to read rather than letting a stale table be taken as current.
* ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file
A workflow's displayed name is captured when GitHub first registers it and is not
refreshed afterwards. Both OCI entries were registered from a run that could not
be parsed -- a job-level env block referencing runner.temp -- so both list as
their path, which reads as a missing name: key and is not one. The 2026-10-02 run
was green on all three topologies and the entry still read as a path, with
updated_at unchanged from the broken first run, so fixing the file is necessary
and not sufficient.
A new path is a new registration, and its first run reads a valid file. Dropping
-nightly is also more honest: there is no schedule: here, because GitHub
registers one only from the default branch, so a cron could never run dev and a
feature branch as separate nightlies.
* Revert "ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file"
The rename makes the workflow undispatchable. `gh workflow run` resolves a
filename against the default branch, and the raw dispatch endpoint returns 404
for a path GitHub has not registered -- checked twice, several minutes apart, so
it is not an indexing delay. The old name works only because it is already
registered, and an unrunnable workflow is a worse problem than a mislabelled one.
The sidebar label needs a fresh registration, which a rename cannot provide from
a feature branch. It corrects itself when the file reaches main, where GitHub
re-reads it; the alternative is deleting the dead registrations' runs, which is an
operator's call.
* fix(ci): never publish the credential hook's output as an artifact
The hook is the operator's own executable, and the one thing known about it is
that it handles a private key. Its stdout and stderr are captured to
credential-hook.log, which the artifact's *.log glob was collecting into a
downloadable archive with a fourteen-day retention.
Excluded rather than trusted to have been quiet. A hook that prints its key once
while someone is debugging it should not thereby publish it.
* fix(oci): the hairpin probe measured nothing on one node, and claimed too much
With a single host, HOSTS[0] and HOSTS[-1] are the same address, so the node
connected to itself -- which always succeeds. It reported "present" on vm-single
and bm for that reason and not as a measurement. It now needs two nodes and says
so otherwise.
The claim it carried was also wrong. "A VCN does not reflect its own public
addresses back into itself" was read off a workstation run whose
node_client_cidr_allow_list was narrowed to one office address, which makes a
node's own public address a non-permitted source: the refusal that follows is
identical to an absent hairpin. With the committed 0.0.0.0/0 default the probe
reports the hairpin present between two nodes, so the original reading was
confounded and is retracted.
None of this changes the remedy. A customer reaches a guest from outside the VCN,
which is where the proxy's egress is, and probing from there is correct whichever
of the two causes is in play -- 15 of 15 workbooks pass through it on all three
topologies. What changes is what the documentation may assert: the guide now names
both causes as indistinguishable rather than stating one as fact.
* docs(oci): a reader could not run the first command, because nothing told them to clone
The guide's first code block was `cd scripts/terraform/oci-spx`, into a directory
no step had obtained. It also never mentioned the SSH keypair the driver refuses to
start without, nor that the configuration is not on main or in v1.21.0 yet, so all
three had to be inferred from the deploy failing.
Restructured so the deploy comes first and reads as four steps and one command:
get the configuration, choose one node or three, write the inputs, run it. The
credential decision stays before step 4, because it changes an argument there and a
node without one forms, looks healthy, and then refuses every launch that wants a
public address. Everything else -- why OCI, sizing, the architecture, the IAM
policy, quotas, the oci CLI, what Terraform builds, every variable -- moves under
Reference, unchanged in substance.
Adds what a successful run prints, naming `allocator ready` as the line to look for:
every line above it appears on a node that cannot allocate, and that is the one
failure the run does not announce.
Also drops an empty verdict.txt committed from a local run. The directory the guide
now tells customers to work in should not ship one run's artifacts.
* fix: stop the e2e analyser inventing failures and let a detach outlast its seal
go-junit-report emits <error message="No test result found"> for a === RUN with
no matching result line, and attaches the test's own captured log as the body.
The analyser counted those as failures, so the baremetal cell reported 25
failures against a junit file recording 1, with log output reading as
assertions. Separate the marker from a real error in both the parse and the
roll-up, count the unresolved cases and say so in the summary.
The gateway gave a detach 30s while the daemon budgets 180s for the ebs.unmount
seal and 180s more for the QMP unplug chain, so any detach whose seal replayed a
WAL was reported as ServerInternal while it went on to succeed.
Also switch four nightly job gates from always() to !cancelled(): always()
overrides cancellation, so a cancelled run kept starting jobs.
* fix: serialise the e2e workflows on one group and retry a dropped libvirt
e2e.yml had no concurrency block at all, so every push to dev could provision
alongside every other push and alongside the nightly. All four workflows that
build on these hypervisors now share e2e-hypervisors, queued rather than
cancelled. A per-environment group is not expressible: GitHub picks the group
before it picks a runner, and the environment is a property of the runner.
tofu apply also failed outright on a transient libvirt connect, which fails a
45-minute job in 90 seconds and reports it as a test failure. retry_run retries
only when that attempt's own output names a connection failure, so a real
failure still fails once.
Also collect the unfiltered journal from the matrix cells and the node journal
from cell-21: the unit filter drops the kernel and OOM lines that explain why
one node behaved unlike its siblings, and a passing run's journal is what makes
a failing one readable.
* fix: build the baremetal junit from the unfiltered suite log
run-baremetal-e2e.sh streams the full suite output to suite-output.log and a
grep-filtered summary to its stdout, which the workflow tees into
test-baremetal.log. That filter anchors --- PASS at column 0 and Go indents
every subtest result by four spaces, so the junit was generated from a log with
no subtest results in it: go-junit-report saw each === RUN with no outcome and
emitted <error message="No test result found">.
17 of cell-28's 39 testcases and 93 of cell-20's 120 were that. Regenerating
cell-28 from suite-output.log gives 39 tests, 0 failures, 0 errors.
* fix: converge released EIPs instead of trusting a fire-and-forget teardown
A disassociate leaves the ENI running, so the orphan sweep's owning-port test
never fires for it. The teardown then rests entirely on vpc.delete-nat, which is
core NATS and is lost whenever vpcd is restarting — and the intent KV watch is
UpdatesOnly, so the same window hides the write that would have driven a repair
pass. The dnat_and_snat row and its host /32 route to a released public address
survived until something else happened to touch them, which for a live ENI was
nothing at all.
Give the sweep the second test it was missing: AddEIP is the only writer of
dnat_and_snat, so the floating IPs intent asks for are the complete set of
addresses that may hold one, and a row for any other address is stale however
alive its port. Ports and external IPs travel together in LiveEIPs, both unioned
from the start-of-pass snapshot and the fresh re-read, so a mid-pass launch is
still spared. The host unbind now runs even when the row had already gone, which
is the state a crash between the OVN delete and the unbind leaves.
Run that sweep on the startup pass too. Startup is the pass that follows the
blind window, so skipping it deferred repair of exactly the changes that were
missed. The port-group and ENI-port sweeps stay gated there for the in-flight
create they were written for; the EIP sweep re-reads intent immediately before
deciding and abandons the whole sweep if that read fails, which is the same
protection by a stronger route.
* fix: keep a release verify off the hypervisors an e2e run is using
release.yml is the fifth workflow that provisions on the ci-single pool and had
no concurrency block at all, so a tag push could land a tofu apply on the same
libvirt host a nightly cell was mid-provision on. The group goes on the verify
job rather than the workflow: the build jobs touch no hypervisor and should not
queue behind a nightly.
* fix: stop cell-27 handing a 50-minute suite a 50-minute job
The rds suite is budgeted 50m in suite-sets.sh and cell-27 took the matrix
default of 50 minutes for the whole job, so provisioning, install, the suite and
teardown had to fit inside the suite's allowance alone. It never could: the last
measured run spent 6 minutes reaching the suite and was killed 44 minutes into
it, mid-TestAutomatedBackups. A job timeout reports "The operation was canceled"
and nothing else, which is why this cell has no verdict in any nightly rather
than a failure anyone could read.
cell-27 gets 75 minutes, and the select job now refuses to start a run whose job
budget does not clear its suite budget plus a setup allowance. The inner timeout
firing first is the property that matters: go test says where it stopped and
what failed, and the job timeout says nothing at all. cell-29 already had the
same fix applied by hand; nothing stopped the next suite from repeating it.
* fix: converge host EIP plumbing against intent, not against the NAT row
The released-EIP sweep landed and did not fire. Its premise was that a stranded
dnat_and_snat row is what a stranded host route can be found by, and on the path
that actually strands one there is no row left. AddEIP scrubs the predecessor
row sharing the private IP before installing the new one, so when reconcile
re-added the instance's auto-assigned address it silently deleted the released
EIP's row — and applyEIPs runs ahead of the sweep, which then walked a table
with nothing stale in it while the /32 route and its proxy-ARP stayed up.
Give the sweep the host's own inventory. ListEIPIngress reads the EIPs back from
the FORWARD accepts EnsureEIPIngress stamps with its own comment, so a binding is
found by what installed it rather than by a row some other path is free to
delete. Anything plumbed that intent no longer asks for is torn down, which also
covers a crash between the OVN delete and the unbind. A failed inventory fails
the pass: an empty list and an unreadable one are indistinguishable, and reading
the second as the first is a sweep that reports success having done nothing.
Close the window inside the pass too. The predecessor scrub now reads the row it
is about to drop and releases that address's host state, so the strand does not
exist between applyEIPs and the sweep rather than merely being repaired a step
later.
* fix: retry the release verify's libvirt calls like every other job does
The verify job drives the same libvirt provider on the same hypervisors as the
e2e cells, where a dropped connection under concurrent load is the steady state
and is retried. Here it was a hard failure that reads as a broken release. It
was skipped earlier only because the job checks out mulga and not spinifex, so
the shared helpers were not on disk — a sparse checkout of .github/scripts costs
nothing and makes apply and destroy behave like their e2e counterparts.
* fix: do not report a rebooting DB instance available on the pre-reboot agent's health
RegisterDBInstance stamped a fresh LastSeen while leaving Agent.EngineHealth at
the value the previous agent process reported. After a reboot the new agent
registers before it has probed anything, so the record read as a healthy engine
with a beat newer than the transition, and the reconciler moved the instance to
available while PostgreSQL was still starting up. A client that connects the
moment the status turns green then gets "the database system is starting up".
A register now discards the previous process's verdict and records when this
agent process started. The restart reconciler requires that registration to be
newer than the transition, which also closes the window where a beat lands
between the engine being stopped and the VM going down.
* test: record the credential kubectl sent when an EKS call comes back Unauthorized
The IRSA pod subtest's first kubectl is the gateway-CA configmap, which lands
before the pod dumps are registered, so a 401 there left no evidence at all. A
verbose read-only probe reports whether an Authorization header was sent, which
separates a token the apiserver rejected from one the exec plugin never made.
* fix: report a starting engine promptly, and budget CI jobs from the suite
The agent waited a full heartbeat interval before its first beat, so a restart
could not be observed as finished for up to 30 seconds after the engine was
already serving. Beat at startup instead, and report every 5 seconds until the
engine has answered once — the control plane is actively waiting on that beat,
and the steady-state cadence it hands back is for a database already serving.
The rds suite is the one most sensitive to what else the hypervisor is doing:
its DB VMs serialise behind a 4 GiB semaphore, so its wall clock is set by how
long it waits for the budget. The same pass ran 45m beside four nightly cells
and overran 50m beside nine, panicking with two tests still running. Raise its
budget to 60m.
That budget was then larger than the jobs meant to contain it. A job cancelled
on timeout reports only "The operation was canceled" and takes the artifacts
with it, so the inner `go test` timeout must always fire first — and it could
not for rds on e2e.yml, nor for storagefault or diskperf, which have asked for
90m and 180m inside a 60m job since they were added. Both workflows now resolve
their job budget from suite-sets.sh rather than carrying a literal: e2e.yml
from the largest requested suite, e2e-suites.yml per matrix leg, so a leg is
bounded by its own suite and not by the longest one beside it. The default push
set resolves to exactly the timeouts both workflows had before.
* fix: check a permutation cell's job budget against the suites it runs
The guard only saw cells that name a suite, so the nine that run a topology's
nightly subset were never checked against it — the same class of mistake the
guard exists to catch, in the majority of the cells. Bound each by the longest
suite in its subset: max() and not sum(), because whichever suite hangs must
panic on its own timeout first, and a sum would demand a budget the set has
never come close to using. Every cell clears it today.
* fix: report why a PostgreSQL engine would not start, and stop failing a stop that landed
Three defects, all found by the first rds pass that ever ran deep enough to
reach them.
A wait that saw an instance reach `failed` called t.Fatalf from inside the
polling condition, and testify runs that condition on its own goroutine. The
Goexit killed the tick and left the collector empty, which reads as the
condition being satisfied — so the wait returned successfully on an instance it
had just declared broken, logged "reached status available" about one that was
failed, and let the test carry on against it. Six conditions across four suites
call Fatal or require the same way. EventuallyErr now runs its condition on the
test's goroutine, which makes all six behave as written.
A DB VM stop has two drivers: the API call and the reconciler resuming one whose
caller it assumes has died. When they overlap the loser is told the VM is not in
a state to be stopped — which is the state the stop was heading for — and the
customer's StopDBInstance returned InvalidDBInstanceState for a stop that had
just completed. Both paths now treat a refused command the way they already
treat an unanswered one: it says nothing about the VM, so the fleet view
settles it.
PostgreSQL had no equivalent of the engine log tail MariaDB has carried since it
was added, so a postmaster that refuses a setting produced "engine did not
respond on /run/postgresql:5432" and nothing else — on a guest with no SSH,
whose console records only that pg_ctl could not start the server. The probe now
quotes the tail of the log the image already directs the postmaster to, and the
reboot and modify paths carry the agent's reason into the failure the way the
create path always has. The reason reaches the customer through StatusInfos.
The rds suite's budget goes to 90m and cell-27's job to 100m. Its wall clock is
what the DB-VM queue costs rather than what the tests do — single tests logged
11m, 22m and 26m waiting for budget — and the pass above still had three of
seventeen tests running at 60m.
* fix: keep a DB VM recoverable when its agent cannot start, and flush the volume before a snapshot
An rds-agent that failed to construct exited, so a transient IMDS credential
resolution took the guest down for good: rds-datadir and rds-init cascade on the
missing handoff, the engine never starts, and the control plane sees only silence
until it declares the instance failed ten minutes later. Construction is retried
like every other boot step instead.
OpenRC skips start_post when the start itself fails, which is the one case with
no diagnostic at all — the console carried nothing but "died". The log mirror
moves out of the handoff wait so a failed start uses it too.
A snapshot quiesced the engine and nothing else. The engine repairs its own files
from the WAL, but postgresql.conf sits beside them and it does not: a restore
came up on a torn tail and the postmaster stopped on a syntax error at the
unterminated include hook. The quiesce now flushes the whole data mount, and the
hook is renamed into place rather than appended, so a boot that finds a half
written one replaces it instead of appending after its open quote.
* fix: flush the config cloud-init wrote before a reset can take it
A DB instance rebooted seconds after it went available came back with no
RDS_GATEWAY_URL, so rds-agent could not reach the control plane and the engine
never started. cloud-init writes agent.env once, per-instance, and never again;
a reboot is a QMP system_reset with no guest shutdown, so the file was still
dirty page cache when the machine reset under it and the next boot read nothing.
The handoff wait now syncs, which is the point at which the agent has proved it
read that file.
* fix: stop dropping an EIP teardown published while vpcd is restarting
DisassociateAddress publishes vpc.delete-nat fire-and-forget. On run
34046316024 vpcd disconnected from NATS at 03:51:26.73 and its replacement
subscribed at 03:51:27.09; the publish landed at 03:51:26.92, in the gap, and
went nowhere. ReleaseAddress then returned the address to the pool two minutes
later with its /32 host route still delivering to the guest that had held it.
vpc.delete-nat now uses request-reply like vpc.add-nat, and retries only on
no-responders — which returns immediately, so the cost is paid solely when
nothing is listening. A teardown that still fails is logged as an error naming
the address rather than lost silently. Neither is fatal to the caller: the
reconciler's orphan sweep remains the backstop.
vpc.add-nat on the EIP path stays fire-and-forget. The shared helper blocks it
on the flows barrier for up to 45s, and associating an address should not
inherit that.
Two reconcile passes in that run had the address absent from intent and did not
sweep its host route, and the journal cannot say why: the sweep logged only when
it acted, so a route it kept was indistinguishable from one it never saw. It now
logs the bindings it found, kept and wanted.
* ci: render one run-level table for the e2e nightly
Reading a nightly meant opening nineteen jobs, and the Actions UI does not
separate a cell that failed from one cancelled out from under it — a
distinction every investigation of this suite has needed, and one the run
34046316024 analysis had to reach for the jobs API to get.
A final job, needs: every cell and if: always(), renders cell, topology,
result, duration, runner and per-cell test counts into the step summary, with
run-level totals, the three slowest cells and the failing test names.
Result, duration and runner come from the jobs API so a cell that died before
writing junit still gets a row. Counts come from a new junit-only artifact per
cell rather than the full cell artifact, so the roll-up does not pull nineteen
journals it has no use for.
Verified against run 34046316024: 14 success, 2 cancelled, 1 failure, with the
two cancellations shown as cancelled rather than folded into failure.
* fix: assert EC2's actual EIP semantics, and probe EKS auth on an authenticated path
natuplink asserted that an instance's auto-assigned public IP is still plumbed
after an EIP associated to the same instance is torn down. EC2 does the
opposite: associating an EIP releases the auto-assigned address back into the
pool, and it cannot be reused. Our datapath already does that — run 34059879666
shows 192.168.0.72's ingress removed at 07:09:44.000, 36ms before .73's was
installed — so the test was asserting behaviour neither AWS nor we have.
It now asserts the release at the point it happens, and that no public delivery
remains after the teardown. This surfaced only once the teardown itself started
working: the run's other assertions passed, including .73's removal, which was
the failure before.
Our records still disagree with our datapath — the ENI keeps the released
address, which is why termination tears the old one down twice and never the
current one. That is the remaining half of mulga-7zc9w and is not fixed here;
it needs an IPAM release on the associate path.
Separately, AuthDiagnostics read /version, which is anonymously readable, so its
200 was consistent with both branches it claimed to distinguish. It now runs
auth whoami, reads kube-system, and invokes the credential plugin directly,
reporting exit code, stderr and whether a token was produced — never the token,
since CI logs outlive it.
* test: capture per-guest state when a spread guest never answers SSH
TestMultinodeSpread's only failure dump is dumpPlacementNATDiag, which collects
OVN, OVS, ARP and route state aimed at the NAT and EIP half of the test. The
failure it actually hits is a guest that never accepted SSH — run 34040324968,
the guest on node3 at 10.100.2.5 — and for that it recorded no console, no port
binding and no tap state. There was no way to tell from the artifact whether the
guest booted at all.
Now dumps, per guest: the console, the NB logical switch port, the SB port
binding, the OVS interface backing its tap, and br-int's port list. Probed on
every node rather than the expected one, because a port bound on the wrong
chassis is exactly what a hosting-node-only dump cannot show.
Covers all three guests, not just the failing one, so a working guest is
available as the control.
* ci: sample hypervisor load either side of every cell
Eight runner names map onto three hypervisors, so cells that never share a
runner still share a host: when cell 10's SSH wait expired on run 34040324968,
cells 22, 17, 18 and 28 were concurrently live, two of them multi-node. Nothing
in the artifacts measured host load, which left co-tenancy as an explanation
that could be neither confirmed nor ruled out.
Records loadavg, MemAvailable, SwapFree, the cpu line carrying steal, and the
QEMU process count before and after each cell, into the cell artifact.
* ci: note that main's missing concurrency block is why the nightly runs ungrouped
Three refs carry three different groupings: main has no concurrency block at
all, dev uses e2e-nightly, this branch uses e2e-hypervisors. Scheduled runs load
the workflow from the default branch, so every cron nightly runs ungrouped and
cannot queue against anything — which is how a dispatch of 34050630627 ran
alongside the scheduled 34046316024 and cost it cells 24 and 27.
e2e-hypervisors is the right group and deliberately broader than dev's, since
e2e.yml, e2e-suites.yml and e2e-ddil.yml draw on the same runner pools. Nothing
on a feature branch can fix this; the block has to reach main.
Make e2e-baremetal.yml a reusable workflow (workflow_call + a ref input)
so the nightly and release pipelines can invoke the real PXE-reimaged
hardware suite, which previously only ran on manual dispatch.
Nightly: add a baremetal job (conceptual cell 20) that runs every
weeknight on the existing cron, or on dispatch when 20 is selected. It
tests dev on schedule, matching GIT_BRANCH. Sits outside the tofu matrix
(different runner, provisioning, 120m timeout) and shares the
e2e-baremetal-cluster concurrency group, so all baremetal runs serialise
on the single physical node.
Release: a release builds and uploads an ISO to R2 but never booted that
ISO/squashfs path on hardware. Add a baremetal job (needs: release) and
hard-gate on it: promotion moves to a dedicated promote job
needs:[verify, baremetal] (with a matching mark-failed job on failure),
and the iso job's upload gate now waits for both verify and baremetal to
conclude success before pushing to R2. Bump the iso job timeout to 180m
since it now waits on the parallel ~2h baremetal run.
The iso job mounts a kernel-global devtmpfs under the workspace
(.cache/rootfs/dev). A killed job skips its EXIT-trap unmount and leaks
the bind; the verify job (same ci-single runner) then ran a bare
'rm -rf $WORKSPACE/*' that could traverse it and wipe the host /dev.
Detach any mount under the workspace before deleting and add
--one-file-system so rm never crosses a mount boundary. Same backstop
added to the iso job's own cleanup.
* feat(vm): direct-boot foundations + config/input extensions + image build pipeline
vm package (mulga-siv-62):
- Add KernelImage, Initrd, KernelCmdline, FwCfg/FwCfgEntry to vm.Config
- Emit -kernel/-initrd/-append/-fw_cfg args when fields set
- Relax drive invariant: accept kernel image in place of drives
- Add devices.go: IsMMIO predicate, NetDevice/BlkDevice/RngDevice helpers
- Refactor lifecycle.go + network_helpers.go call sites to use helpers
- shutdown.go: os.Remove fw_cfg tmpfiles after QEMU exits
- Add devices_test.go, microvm_test.go; extend vm_test.go
config/input extensions (mulga-siv-63):
- Add MicrovmConfig{ELBv2Enabled, ImagePath} to DaemonConfig
- ImagePath falls back to /usr/share/spinifex/microvm when empty
- Add commented [microvm] section to spinifex.toml template
- Add DirectBoot bool + NICs []NICConfig to SystemInstanceInput
image build pipeline (mulga-siv-64):
- scripts/build-microvm-image.sh: Alpine chroot build producing vmlinuz + initramfs.cpio.gz
- build/microvm/init.sh: fw_cfg-driven PID 1 init (mounts, modprobe, NIC config, lb-agent-env, ca-cert)
- build/microvm/inittab: BusyBox init with lb-agent respawn
- make build-microvm-image target
- setup.sh: install microvm/* to /usr/share/spinifex/microvm/
- .gitignore: exclude built vmlinuz + initramfs.cpio.gz artifacts
* feat(daemon): direct-boot launch path for microvm system instances
When SystemInstanceInput.DirectBoot=true:
- Skip GetAMIConfig and GenerateVolumes
- Build vm.Config directly (no buildBaseVMConfig, no PCIe root ports)
- MachineType=microvm,x-option-roms=off,pic=off,pit=off,rtc=on,acpi=on,isa-serial=on
- Resolve kernel/initrd from Microvm.ImagePath config field
- Serialise NICs to three fw_cfg tmpfiles (netcfg, lb-agent-env, ca-cert)
- Validate exactly one NICConfig.IsDefault=true before writing netcfg blob
- Shutdown path cleans up tmpfiles via existing Config.FwCfg iteration
PC-machine path (DirectBoot=false) unchanged.
Closes mulga-siv-65
* feat(elbv2): wire microvm.elbv2_enabled flag into ELBv2 service
When ELBv2Enabled=true:
- Drop lbVMUserData cloud-config; emit KEY=value LBAgentEnv blob instead
- Set SystemInstanceInput.DirectBoot=true; omit ImageID
- Populate NICs []NICConfig (index 0=primary ENI IsDefault, 1=mgmt, 2+=extra ENIs)
- Validate exactly one IsDefault NIC at service layer
When ELBv2Enabled=false: existing PC-machine path unchanged.
Adds buildLBAgentEnv, buildMicrovmNICs, subnetCIDRForIP, subnetGatewayIP helpers.
Tests in arn_test.go cover both flag-on and flag-off paths.
Closes mulga-siv-66
* test(daemon,vm): add unit tests for direct-boot helpers to meet diff coverage threshold
* feat(daemon): Phase 1 perf instrumentation + e2e runner microvm flag
- Log t1→t2 boot timing via slog on the direct-boot path
- measureQEMURSSMiB helper reads /proc/<pid>/status (tested)
- Gate constants: phase1BootGateMs=500, phase1RSSGateMiB=25
- run-lb-e2e.sh: --microvm flag, dual-path CI annotation, density test note
Closes mulga-siv-67
* fix(build): remove busybox-initscripts — not in Alpine 3.20, not needed
* fix(microvm): end-to-end fixes for direct-boot LB path
Serial output was silent (x-option-roms=off suppresses BIOS serial init);
fixed by removing it from the machine string and using -serial chardev:console0
to attach to the machine-created ISA serial device instead of adding a
duplicate isa-serial device (which would create ttyS1 while kernel uses ttyS0).
virtio-net interfaces were invisible to the kernel because QEMU's
auto-kernel-cmdline does not append virtio_mmio.device= params when -append
is used. Fixed by generating explicit virtio_mmio.device=0x200@0xfebNNNNN:N
params for each NIC in KernelCmdline, and modprobing virtio_mmio + virtio_net
in init.sh.
sys.micro bumped to 512 MiB — 128 MiB caused kernel OOM during initramfs
decompression (~68 MiB uncompressed).
BusyBox init ::respawn: entries do not source or export env files; fixed
inittab to use set -a; . /etc/conf.d/lb-agent; set +a before exec.
wireLBAgentConfig was documented as reading the CA cert but never set
elbv2Service.CACert; lb-agent TLS handshakes to AWSGW failed with
x509: certificate signed by unknown authority. Fixed by reading
d.config.NATS.CACert (same local CA that signs the AWSGW server cert)
and assigning to elbv2Service.CACert so writeFwCfgBlobs writes a
non-empty cacert blob into fw_cfg.
Adds vm_test TestExecute_MicrovmFileChardev and updates lifecycle_test
to assert SerialSocket is intentionally empty on direct-boot VMs.
* fix(build): auto-detect docker/podman when apk unavailable for microvm image build
* fix(build): create /dev/console and /dev/null as proper device nodes in initramfs
tar extraction without root silently demotes device files to 0-byte regular
files (mknod requires CAP_MKNOD). The kernel opens /dev/console when wiring
init's stdio to the serial console; a regular file there causes all init output
to be silently discarded to RAM — the VM appears stuck at 'Run /init as init
process' with no subsequent console output and lb-agent never heartbeats.
Fix: after rootfs population, wipe /dev and recreate console (c 5 1) and
null (c 1 3) via sudo mknod. devtmpfs provides all other devices at runtime
once init.sh mounts it.
* fix(microvm): fix mgmt NIC routing for all LB schemes
Two bugs prevented lb-agent from reaching the AWSGW via the mgmt interface:
1. RouteDst/RouteVia swapped — resolveMgmtRoute returns (gateway, target) but
buildMicrovmNICs assigned them as (Dst=gateway, Via=target), producing
'ip route add 10.15.8.1 via 192.168.1.33' instead of the correct
'ip route add 192.168.1.33 via 10.15.8.1'.
2. NIC[1] MAC and CIDR never filled — the daemon allocates mgmt IP/MAC
after buildMicrovmNICs returns, but didn't inject them into input.NICs[1]
before calling writeFwCfgBlobs, leaving NIC1_MAC= empty in the netcfg
blob and skipping interface setup in init.sh entirely.
Also extends mgmt routing to internet-facing scheme in the microvm path:
VPC SNAT is not reliable across all VPC topologies for control-plane
heartbeats; mgmt is always a direct L2 path to the host.
* feat(microvm): Phase 2 kernel stripping + boot optimisation
- cpu: switch to host,-pmu (disable PMU — not needed in microvm guest)
- machine type: make isa-serial configurable via microvm.isa_serial config
field; default true (on) preserves console log path
- build: strip unused kernel modules (USB, IDE, ATA, GPU, media, input,
HID, bluetooth, staging, sound, wireless) from initramfs before cpio
- build: add Phase 2 artifact size gate — fail if vmlinuz + initramfs.cpio.gz
exceeds MICROVM_ARTIFACT_MAX_MiB (default 50 MiB); override via env var
during development
- perf: add phase2BootGateMs=300 and phase2ArtifactGateMiB=50 constants
with canary tests to catch unintentional gate changes
- tests: cover buildMicrovmNICs nil-VPC and extra-ENI paths
- tests: cover microvmMachineType nil/true/false branches
* fix(build): use find-based virtio+fw_cfg module keep-list in initramfs strip
The directory-only keep-list (drivers/virtio, drivers/firmware) missed
virtio_net.ko which lives in drivers/net/, virtio_blk in drivers/block/,
and virtio-rng in drivers/char/hw_random/. Switch to find-based approach:
collect all virtio*.ko* and *fw_cfg*.ko* regardless of subdirectory, then
restore them into their original relative paths under kernel/.
Artifacts now: 42 MiB (vmlinuz 11 MiB + initramfs 31 MiB), gate passes.
* fix(build): keep failover modules in initramfs strip
virtio_net depends on net_failover/failover for live-migration support.
Stripping them left virtio_net with unresolved symbols
(net_failover_create/destroy), so the module failed to load and the
guest came up with only lo. init.sh then skipped every NIC ("no
interface found for MAC ..."), leaving lb-agent with no network path
to AWSGW and the LB stuck in provisioning.
Add *failover*.ko* to the keep-list. Image size unchanged at 42 MiB
(under the 50 MiB Phase 2 gate). ELBv2 e2e now reaches active and
2/2 targets healthy for both ALB and NLB internal LBs.
* fix(daemon): restore slog default after viperblock.New mutates it
viperblock.New unconditionally calls vb.SetDebug, which calls
slog.SetDefault with a TextHandler at LevelError — globally silencing
every slog.Info and slog.Warn in the caller process for the rest of
its life. Discovered while validating microvm Phase 2 (mulga-siv-68):
the daemon stopped emitting Info logs the moment the first customer
VM with volumes launched, hiding direct-boot timing,
LaunchSystemInstance completed, and CreateLoadBalancer completed
lines that we need for boot-time tuning.
Workaround in newViperblock: re-install an Info-level TextHandler on
os.Stderr after each viperblock.New call. Also drop the three explicit
vb.SetDebug(false) calls in prepareRootVolume / prepareEFIVolume /
prepareCloudInitVolume — they were no-ops (viperblock.New already
runs SetDebug internally) and each one re-triggered the global
mutation we are trying to undo.
Proper fix is in viperblock itself — tracked as mulga-siv-70: SetDebug
should not call slog.SetDefault. Until then, this restore call is the
shortest path to getting the daemon's observability back so we can
measure t1_to_t2_ms and tune Phase 2.
Preflight note: golangci-lint not available on this dev host, so the
project preflight target cannot run end-to-end. Verified with
go vet ./spinifex/handlers/ec2/instance/... (clean) and
go test ./spinifex/handlers/ec2/instance/... (pass).
* perf(microvm): hit Phase 2 boot/memory gates
t1→t2 hypervisor: 3224 ms → 264 ms (12×). 128 MiB guest now boots
cleanly without OOM, reverting the temporary 512 MiB bump.
Root causes:
1. build-microvm-image.sh staged the entire chroot into the cpio
without dropping /boot, embedding a duplicate vmlinuz, Alpine's
stock initramfs-virt, System.map, and a "boot → ." symlink loop
(~25 MiB). Decompression peak exceeded 128 MiB. Strip /boot after
the kernel is copied out, before the cpio. Initramfs drops from
31 MiB → 11 MiB (53 MiB → 28 MiB uncompressed); total artifact
42 MiB → 22 MiB.
2. vm/lifecycle.go burned 3 s of fixed sleep on every direct-boot
launch: 2 s nbdkit pre-exec wait (no nbdkit on direct-boot) and
1 s post-fork settle (QEMU binds QMP within ~10 ms when there is
no firmware, ROM, or block layer). Both extracted to
directBoot-aware helpers — nbdkitPreExecWait collapses to 0, and
qemuStartSettleWait to 50 ms.
3. KernelCmdline only carried `console=ttyS0 virtio_mmio.device=…`.
Added boot-time perf flags: quiet loglevel=3 (silence printk over
115200-baud isa-serial), mitigations=off (KVM guest on trusted
host, single-tenant LB workload), tsc=reliable, no_timer_check,
reboot=t, i8042.{nopnp,noaux,nokbd} (microvm has no PS/2).
4. init.sh dropped modprobe virtio_mmio (built-in:
CONFIG_VIRTIO_MMIO=y) and tightened polling: fw_cfg/iface waits go
from 10 × 200 ms (up to 2 s) to 5 × 20 ms (up to 100 ms).
Phase 2 gates:
- t1→t2 ≤ 300 ms ✅ 264 ms / 281 ms
- artifact ≤ 50 MiB ✅ 22 MiB (gate already in build script)
- ELBv2 e2e green ✅ 32/33 (pre-existing client-VM failure,
unrelated to microvm path)
* build(microvm): wire build into deploy + distro + release pipeline
Adds install-microvm Makefile target (with grouped-target staleness
detection on vmlinuz/initramfs.cpio.gz), wires it into dev-install.sh
and reset-dev-env.sh, makes 'make deploy' reinstall artifacts when
present, and makes 'make distro-amd64' stage build/microvm/* into the
release tarball staging dir. arm64 distro stays unchanged (no aarch64
microvm artifact yet — Phase 4).
Release workflow gains a setup-go step and a 'make build-microvm-image'
step before 'make distro-amd64' on the amd64 runner so the resulting
tarball includes the kernel + initramfs without a manual copy.
Precursor for the Phase 3 PC-machine LB removal (mulga-siv-69).
Closes mulga-siv-71.
* build(ansible): include microvm artifacts in build role tarball
Adds a build-microvm-image task to the ansible build role and stages
build/microvm/* into the role's release tarball so the install role's
setup.sh run picks them up via INSTALL_SPINIFEX_TARBALL. Skipped
gracefully when the build script is absent (older branches without
direct-boot LB wiring).
* feat(config): enable microvm direct-boot LB path by default
Uncomments the [nodes.<node>.daemon.microvm] section in the generated
spinifex.toml template and sets elbv2_enabled = true so all new
clusters bring up ELBv2 LBs via the direct-boot microvm path. Operators
can opt back to the legacy PC-machine path by flipping the flag to
false in the rendered config.
Required for end-to-end CI on this branch to exercise the direct-boot
path. Existing clusters' configs are unchanged (the template only
affects fresh 'spx admin init' renders).
* fix(microvm): bootstrap apk.static when no host rootfs tool exists
Self-hosted CI runners and the GitHub-hosted ubuntu-24.04 image often
have none of apk, docker, or podman installed. apk-tools is not in the
Debian 13 (trixie) repositories. Without one of these the rootfs build
fails up front with 'no rootfs build tool available'.
When all three are absent, fetch apk-tools-static from the configured
Alpine mirror, extract it to a tempdir, and prepend it to PATH so the
existing native-apk strategy below picks it up transparently. Requires
only curl + tar, both of which are present on every CI runner that can
already check out source.
No-op when apk, docker, or podman is already present.
Unblocks https://github.com/mulgadc/spinifex/actions/runs/25988830244
* revert: drop apk.static bootstrap in build-microvm-image
Reverts 0bd76b7. Self-hosted runners will get docker.io installed
out of band; existing Strategy 2 (docker run alpine:3.20) handles
the rootfs build without host sudo or unsigned package install.
* docs(workbook): drop lb-alpine import for microvm direct-boot LB
LBs now launch via direct-boot QEMU microvm using kernel + initramfs
bundled at /usr/share/spinifex/microvm/, so the legacy lb-alpine system
AMI import is no longer required.
* fix(daemon): own tap devices by daemon user so QEMU can attach without CAP_NET_ADMIN
When the daemon runs non-root (production: User=spinifex-daemon), child
QEMU processes inherit that uid but lack CAP_NET_ADMIN. The kernel's tun
attach check then requires the tap owner uid to match the caller's euid.
Without the `user` flag, `ip tuntap add` defaults to root:root and qemu's
TUNSETIFF fails with EPERM ("could not configure /dev/net/tun: Operation
not permitted").
Surfaced by the e2e-nightly cell-17 microvm LB launch on the ci-pseudo
runner where the kernel enforces strictly; legacy EC2 launches on
ci-single runners happened to attach successfully despite the same code
path. Adding `user $current group $gid` to `ip tuntap add` ensures the
tap is owned by whatever uid the daemon (and its qemu children) run as.
Root daemons skip the flag — they have CAP_NET_ADMIN ambient.
* fix(e2e): redirect lb probe chatter to stderr so stalls are visible
test_http_from / test_tcp_from echoed progress to stdout, which the
callers captured into *_OK vars via $(...). Result: 6+ minute retry
loops ran silently in CI, and the "$ok" comparison never matched
"true" because the captured value contained the full multi-line
chatter. Reachability pass/fail counters were also lost in the
subshell.
Redirect helper output to stderr, leave only the true/false flag on
stdout, and move pass/fail to the caller so they actually count.
* fix(microvm): don't /32 AdvertiseIP via mgmt for internet-facing single-node
buildMicrovmNICs overrode resolveMgmtRoute's deliberate empty return
for the single-node internet-facing case by falling back to
(MgmtBridgeIP, AdvertiseIP). That installed a /32 route to AdvertiseIP
via the mgmt NIC inside the LB microvm. On single-node where the
AWSGW host IP equals the WAN client IP, the /32 also catches reply
traffic for inbound HTTP/TCP to the LB's floating IP — replies leave
via the mgmt NIC with src=data-NIC IP, the host kernel drops the
asymmetric flow, and same-chassis ingress hangs.
resolveMgmtRoute already encodes the correct policy:
- multi-node any scheme → explicit MgmtRoute{Gateway,Target}
- single-node internal → fallback to (MgmtBridgeIP, AdvertiseIP)
- single-node internet-facing → empty (reach AWSGW via VPC SNAT)
Honor it. Matches the legacy lbVMUserData path, whose own comment
flagged the stolen-WAN-return-path symptom this introduced.
Test covers all three branches.
* feat(ansible): dev-deploy playbook for in-place artifact swap
Fills the gap between ansible-dev-install (skips when already installed)
and ansible-dev-reset (full teardown + reinstall + smoketest). Mirrors
Makefile `deploy` target: rebuild + replace binaries and microvm
artifacts on a running node, restart spinifex.target, wait for awsgw.
Skips setup.sh (no user/dir/systemd-unit changes) and smoketest so
iteration is fast. Use ansible-dev-reset when changing systemd units,
helper scripts, logrotate, or setup.sh itself.
Adds:
- roles/deploy/tasks/main.yml — install spx + nbdkit plugin + microvm
artifacts, with plugindir auto-detection via `nbdkit --dump-config`.
- playbooks/dev-deploy.yml — chains build + deploy roles.
- Makefile ansible-dev-deploy target.
- README playbook table + "when to use which" guidance.
* chore(microvm): remove legacy LB Alpine image build pipeline
Phase 3 microvm removal — strip the PC-machine LB image build
artifacts that direct-boot no longer needs. The microvm initramfs
already carries lb-agent + HAProxy via build-microvm-image, so the
lb.conf manifest, lb-setup.sh chroot script, build-lb-image target,
dev-install / reset-dev-env import steps, and ansible LB-image tasks
all become dead.
build-system-image stays — other manifests (e.g. GPU images) still
use it.
* refactor(microvm): remove legacy PC-machine LB boot path
Phase 3 — direct-boot becomes the only ELBv2 LB launch path.
- Drop SystemInstanceInput.DirectBoot / ImageID / UserData (every LB
caller set DirectBoot=true; legacy cloud-init userdata is gone).
- Drop ELBv2ServiceImpl.lbVMUserData, systemAMI* fields, SetSystemAMIFunc,
getSystemAMI, and the lazy-AMI wiring in daemon.startCluster.
- Collapse CreateLoadBalancer to a single direct-boot launch input.
- Collapse daemon.LaunchSystemInstance: drop AMI validation, ImageID
on RunInstancesInput, UserData base64 encode, GenerateVolumes branch.
buildDirectBootConfig hardcodes /usr/share/spinifex/microvm and
isa-serial=on (microvmMachineType no longer takes a flag).
- Drop MicrovmConfig from config.DaemonConfig and the
[daemon.microvm] section from the spinifex.toml template.
- Prune dead tests (lbVMUserData_*, PCMachine_Unchanged, AMI resolver
paths) and refresh stale comments that still mentioned lbVMUserData.
ARM hosts have no LB launch path until Phase 4 adds direct-boot on
virt — CreateLoadBalancer on aarch64 fails naturally at QEMU exec.
* docs: drop lb-alpine catalog entry from cluster setup guide
LB system instances no longer use an AMI — direct-boot microvm
artifacts ship in the spinifex package.
* feat(ui): remove LB image import banner + LB-AMI gating
Direct-boot microvm artifacts ship in the spinifex package, so the
LB system AMI no longer exists — the "Load balancer image not imported"
banner and the Create-button disable on the LB list / wizard pages
can never fire. Strip:
- lb-image-missing-banner.tsx component
- hasLbImage / isLbImage / LB_IMAGE_NAME / LB_MANAGED_BY_VALUE from
system-managed.ts (isSystemManagedImage stays — still used by the
EC2 Images pages to hide system AMIs)
- Banner gating + ec2ImagesQueryOptions imports from the LB list and
wizard pages
- The corresponding tests that asserted the banner appears
* revert: restore lb.conf + lb-setup.sh for mulga install pipeline
Phase 3 microvm removal (commit 6313d74) deleted these too early — the
mulga install/iso artifact builders (scripts/tofu-cluster/build-install
-artifacts.sh, scripts/tofu-iso/build-iso-artifacts.sh) still invoke
scripts/build-system-image.sh scripts/images/lb.conf, breaking e2e with
"ERROR: Manifest not found: scripts/images/lb.conf".
Restore both files verbatim so the legacy pipeline keeps building until
the mulga-side migration to microvm-only artifacts lands post-merge.
- workflows: add 'dev' to push/PR triggers (spinifex.yml, ui.yml, e2e.yml)
- workflows: drop "Resolve Mulga Branch" fallback step; mulga is single-branch
so checkout@v6 uses the default branch implicitly
- dependabot: route gomod/github-actions/npm PRs to dev
- diff-coverage.sh: pick origin/dev as base for feat branches, origin/main for dev
- iac/proxmox-template: default git_branch to dev for cluster builds
Refs docs/development/improvements/reintroduce-dev-branch.md
Julian Sommer