38 Commits

Author	SHA1	Message	Date
Ben Duncan	c6841ce1f0	feat(oci): separate using an instance principal from creating one (#1157) ... * feat(oci): separate using an instance principal from creating one The allocator needs OCI credentials at runtime, and a node with none forms, passes every health check, then refuses every launch wanting a public address. Instance principal is the credential route that puts no key material on a node, but it needs a dynamic group and a policy, and those are tenancy-root resources the compartment-scoped user the rest of the configuration runs as cannot create. enable_instance_principal conflated the two, so the better credential route was only reachable by a tenancy admin. instance_principal now takes off, adopt or create: adopt selects the credential and creates nothing, because the matching rule covers the whole compartment and so names no instance to reference. setup-identity.sh owns the create case, with its own state. validate-topology.sh destroys its state at the end of every run, so holding tenancy resources there would let a teardown delete the tenancy's policy. Also package install-node.sh and smoke-test.sh, so a cluster can be formed from a curl | bash install with no checkout. smoke-test.sh is not optional: formation scp's it from its own directory for the post-formation verification. * feat(oci): take the credential handoff from an operator hook, and widen the workflow name An API-key deployment needs a credential on each node, and nothing in this repository can supply one: a key in user-data is readable from instance metadata for the life of the instance, and this host runs other people's guests. A manual install does not survive either, because a topology destroys itself on success. So the harness calls an operator-owned executable after formation and before the pool, the only window where a node has /etc/spinifex and has not started the allocator. Instance principal needs no handoff, so the hook is skipped there. The workflow is renamed for the vendors it will cover rather than the first one. Nothing is orphaned in the Actions UI: it has never been on the default branch and has never run, so there is no history attached to the old filename. * docs(oci): the API-key path needs the config file, not just the PEM oci_config_file names an ordinary OCI SDK config and nothing creates it, so a node holding only the key forms and then fails every allocation with no stated cause. * fix(ci): the runner context is not available in a job-level env block A workflow referencing it there is rejected before any job starts, which is why the rename produced a failed run attributed to the push. The reference had never been validated: a dispatch-only workflow that is never dispatched is never parsed. * fix(deps): bump containerd to v2.4.1 for GO-2026-6597 The image-pull DoS via crafted OCI index graph amplification is reachable from the ECS agent's containerd puller, so the vulnerability scan fails preflight on v2.4.0. A patch release with no API change. * fix(e2e): probe guest public addresses from outside the cloud's own network A VCN does not reflect its own public addresses back into itself, so a node asking for one gets nothing however well the datapath works. The workbook driver runs on the first node, so every assertion that curls or SSHes a guest's public address failed on OCI against a cluster that was serving those addresses correctly to the internet. The driver now takes E2E_PUBLIC_PROXY, a SOCKS5 proxy used for public addresses only, and validate-topology.sh supplies one with a remote dynamic forward on the connection that launches the driver. Egress is then the host running the validation, which is outside the VCN and is where a customer sits. Unset leaves every other platform on a direct probe, which is already correct there. Also adds --version TAG for a repeatable install, separates the workflow's auth choice from its external-pool choice so an API key can run with the allocator live, and captures allocator diagnostics before the teardown destroys them. * ci: read the Go version from go.mod in the release workflows Both pinned 1.27.0 by hand while every other workflow reads the file, so the toolchain they build with drifts from the one the module declares the day go.mod moves. release.yml was already inconsistent with itself: its second setup-go step reads spinifex/go.mod. * feat: publish the OCI topology results as tables on the run page A green tick on one job with 4,000 lines of log behind it is confidence for whoever wrote the harness and nothing to anybody else. Someone assessing whether Spinifex works on their cloud should be able to see that it formed and served guests on a single VM, on three VMs and on bare metal without opening a log. record now takes <gate> <status> [detail] and writes results.tsv beside the human block, so a gate's status is a field the script sets rather than a regex over prose that reads fine and matches wrong. record_workbooks turns the driver's own go-test-shaped lines into workbooks.tsv, and runs on the failing branch too: which four passed is most of what the fifth failing means. e2e-oci.sh --summary-only renders both tables to stdout and to GITHUB_STEP_SUMMARY, and the workflow calls it under always() before the sweep, because the sweep destroys the state directories the tables are read from. A skipped gate gets its own mark, and a topology that never ran says so: the one way a results table can do real harm is a reader taking an absent row for a passing one. * docs(oci): cut the deploy-by-hand appendix, and validate the config for real The appendix was 444 of 1014 lines restating an install the Terraform already does, which made the document read as though deploying by hand were a normal route. It is not: the configuration builds the VCN, the volumes and both VNICs per node and renders the cloud-init, in the same order every time. What is genuinely OCI-specific is four things, each of which already has a section, so the replacement names those and sends everyone else to the single-node and multi-node install guides. Verify end to end told the reader to ping a guest's public address from the host, which cannot work: a VCN does not route its own public addresses back into itself. That is the finding this branch is built on and the document was asserting the opposite. It now says where to probe from and why, and carries the allocator's `resolved the external VNIC` check, which existed only in the cut appendix and is the one line proving the credentials and compartment are right. `spx config validate` is not a command -- spx has admin, get, service, top and version -- so that line could only ever fail and its output and status were both discarded. It is replaced by parsing the TOML, which is what was wanted: the remap and the pool append are textual edits, and a broken result would otherwise surface as every service failing to start with nothing saying why. The remap now checks its own substitution count for the same reason, since re.sub with no match rewrites the file unchanged and exits 0. Publication of the run-page tables moves to one owner. GITHUB_STEP_SUMMARY is per step, so appending from the run as well as the publish step would have put two copies of every table on the run page. * fix(oci): check the credential hook before building anything The hook is a path from a CI variable, and whether it exists on the runner is knowable before Terraform runs. Checked where it was used, a wrong path cost the apply, the install and the formation first -- forty minutes and a bare-metal bill to be told a file is missing. Nothing about the check depends on the apply, so it moves to the top, where --dry-run reaches it too. * fix(oci): the summary must say what was built and what was not tested The first CI run headed its tables "Published release `latest channel`" over a build of a feature branch, which is the one error in a results table that would mislead a reader who trusted it. The header now states a tree build as a tree build, and names how the allocator authenticated. A run with no external pool says so in the header rather than leaving it to a SKIPPED row further down. Public-address allocation is the claim a green table is most likely to be read as making, and it is exactly the claim such a run cannot support. Both flags are passed to the publish step too, so the summary rendered after a failure describes the same run as the one that failed. Raise the dev prerelease prune from five to twenty. A pinned dev tag is how an external evaluator installs a specific build, and five is about a week of them -- short enough that the tag someone was handed stops resolving while they are still using it. * docs(oci): the workbook table is a snapshot, and the nightly is the current answer It is dated against a build five releases of work ago, and the nightly now runs the same workbooks on all three topologies and publishes its own table. Say which one to read rather than letting a stale table be taken as current. * ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file A workflow's displayed name is captured when GitHub first registers it and is not refreshed afterwards. Both OCI entries were registered from a run that could not be parsed -- a job-level env block referencing runner.temp -- so both list as their path, which reads as a missing name: key and is not one. The 2026-10-02 run was green on all three topologies and the entry still read as a path, with updated_at unchanged from the broken first run, so fixing the file is necessary and not sufficient. A new path is a new registration, and its first run reads a valid file. Dropping -nightly is also more honest: there is no schedule: here, because GitHub registers one only from the default branch, so a cron could never run dev and a feature branch as separate nightlies. * Revert "ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file" The rename makes the workflow undispatchable. `gh workflow run` resolves a filename against the default branch, and the raw dispatch endpoint returns 404 for a path GitHub has not registered -- checked twice, several minutes apart, so it is not an indexing delay. The old name works only because it is already registered, and an unrunnable workflow is a worse problem than a mislabelled one. The sidebar label needs a fresh registration, which a rename cannot provide from a feature branch. It corrects itself when the file reaches main, where GitHub re-reads it; the alternative is deleting the dead registrations' runs, which is an operator's call. * fix(ci): never publish the credential hook's output as an artifact The hook is the operator's own executable, and the one thing known about it is that it handles a private key. Its stdout and stderr are captured to credential-hook.log, which the artifact's *.log glob was collecting into a downloadable archive with a fourteen-day retention. Excluded rather than trusted to have been quiet. A hook that prints its key once while someone is debugging it should not thereby publish it. * fix(oci): the hairpin probe measured nothing on one node, and claimed too much With a single host, HOSTS[0] and HOSTS[-1] are the same address, so the node connected to itself -- which always succeeds. It reported "present" on vm-single and bm for that reason and not as a measurement. It now needs two nodes and says so otherwise. The claim it carried was also wrong. "A VCN does not reflect its own public addresses back into itself" was read off a workstation run whose node_client_cidr_allow_list was narrowed to one office address, which makes a node's own public address a non-permitted source: the refusal that follows is identical to an absent hairpin. With the committed 0.0.0.0/0 default the probe reports the hairpin present between two nodes, so the original reading was confounded and is retracted. None of this changes the remedy. A customer reaches a guest from outside the VCN, which is where the proxy's egress is, and probing from there is correct whichever of the two causes is in play -- 15 of 15 workbooks pass through it on all three topologies. What changes is what the documentation may assert: the guide now names both causes as indistinguishable rather than stating one as fact. * docs(oci): a reader could not run the first command, because nothing told them to clone The guide's first code block was `cd scripts/terraform/oci-spx`, into a directory no step had obtained. It also never mentioned the SSH keypair the driver refuses to start without, nor that the configuration is not on main or in v1.21.0 yet, so all three had to be inferred from the deploy failing. Restructured so the deploy comes first and reads as four steps and one command: get the configuration, choose one node or three, write the inputs, run it. The credential decision stays before step 4, because it changes an argument there and a node without one forms, looks healthy, and then refuses every launch that wants a public address. Everything else -- why OCI, sizing, the architecture, the IAM policy, quotas, the oci CLI, what Terraform builds, every variable -- moves under Reference, unchanged in substance. Adds what a successful run prints, naming `allocator ready` as the line to look for: every line above it appears on a node that cannot allocate, and that is the one failure the run does not announce. Also drops an empty verdict.txt committed from a local run. The directory the guide now tells customers to work in should not ship one run's artifacts.	2026-10-04 09:04:55 -05:00
Ben Duncan	18853cef31	fix: close the four defects keeping the e2e nightly red (#1007) ... * fix: stop the e2e analyser inventing failures and let a detach outlast its seal go-junit-report emits <error message="No test result found"> for a === RUN with no matching result line, and attaches the test's own captured log as the body. The analyser counted those as failures, so the baremetal cell reported 25 failures against a junit file recording 1, with log output reading as assertions. Separate the marker from a real error in both the parse and the roll-up, count the unresolved cases and say so in the summary. The gateway gave a detach 30s while the daemon budgets 180s for the ebs.unmount seal and 180s more for the QMP unplug chain, so any detach whose seal replayed a WAL was reported as ServerInternal while it went on to succeed. Also switch four nightly job gates from always() to !cancelled(): always() overrides cancellation, so a cancelled run kept starting jobs. * fix: serialise the e2e workflows on one group and retry a dropped libvirt e2e.yml had no concurrency block at all, so every push to dev could provision alongside every other push and alongside the nightly. All four workflows that build on these hypervisors now share e2e-hypervisors, queued rather than cancelled. A per-environment group is not expressible: GitHub picks the group before it picks a runner, and the environment is a property of the runner. tofu apply also failed outright on a transient libvirt connect, which fails a 45-minute job in 90 seconds and reports it as a test failure. retry_run retries only when that attempt's own output names a connection failure, so a real failure still fails once. Also collect the unfiltered journal from the matrix cells and the node journal from cell-21: the unit filter drops the kernel and OOM lines that explain why one node behaved unlike its siblings, and a passing run's journal is what makes a failing one readable. * fix: build the baremetal junit from the unfiltered suite log run-baremetal-e2e.sh streams the full suite output to suite-output.log and a grep-filtered summary to its stdout, which the workflow tees into test-baremetal.log. That filter anchors --- PASS at column 0 and Go indents every subtest result by four spaces, so the junit was generated from a log with no subtest results in it: go-junit-report saw each === RUN with no outcome and emitted <error message="No test result found">. 17 of cell-28's 39 testcases and 93 of cell-20's 120 were that. Regenerating cell-28 from suite-output.log gives 39 tests, 0 failures, 0 errors. * fix: converge released EIPs instead of trusting a fire-and-forget teardown A disassociate leaves the ENI running, so the orphan sweep's owning-port test never fires for it. The teardown then rests entirely on vpc.delete-nat, which is core NATS and is lost whenever vpcd is restarting — and the intent KV watch is UpdatesOnly, so the same window hides the write that would have driven a repair pass. The dnat_and_snat row and its host /32 route to a released public address survived until something else happened to touch them, which for a live ENI was nothing at all. Give the sweep the second test it was missing: AddEIP is the only writer of dnat_and_snat, so the floating IPs intent asks for are the complete set of addresses that may hold one, and a row for any other address is stale however alive its port. Ports and external IPs travel together in LiveEIPs, both unioned from the start-of-pass snapshot and the fresh re-read, so a mid-pass launch is still spared. The host unbind now runs even when the row had already gone, which is the state a crash between the OVN delete and the unbind leaves. Run that sweep on the startup pass too. Startup is the pass that follows the blind window, so skipping it deferred repair of exactly the changes that were missed. The port-group and ENI-port sweeps stay gated there for the in-flight create they were written for; the EIP sweep re-reads intent immediately before deciding and abandons the whole sweep if that read fails, which is the same protection by a stronger route. * fix: keep a release verify off the hypervisors an e2e run is using release.yml is the fifth workflow that provisions on the ci-single pool and had no concurrency block at all, so a tag push could land a tofu apply on the same libvirt host a nightly cell was mid-provision on. The group goes on the verify job rather than the workflow: the build jobs touch no hypervisor and should not queue behind a nightly. * fix: stop cell-27 handing a 50-minute suite a 50-minute job The rds suite is budgeted 50m in suite-sets.sh and cell-27 took the matrix default of 50 minutes for the whole job, so provisioning, install, the suite and teardown had to fit inside the suite's allowance alone. It never could: the last measured run spent 6 minutes reaching the suite and was killed 44 minutes into it, mid-TestAutomatedBackups. A job timeout reports "The operation was canceled" and nothing else, which is why this cell has no verdict in any nightly rather than a failure anyone could read. cell-27 gets 75 minutes, and the select job now refuses to start a run whose job budget does not clear its suite budget plus a setup allowance. The inner timeout firing first is the property that matters: go test says where it stopped and what failed, and the job timeout says nothing at all. cell-29 already had the same fix applied by hand; nothing stopped the next suite from repeating it. * fix: converge host EIP plumbing against intent, not against the NAT row The released-EIP sweep landed and did not fire. Its premise was that a stranded dnat_and_snat row is what a stranded host route can be found by, and on the path that actually strands one there is no row left. AddEIP scrubs the predecessor row sharing the private IP before installing the new one, so when reconcile re-added the instance's auto-assigned address it silently deleted the released EIP's row — and applyEIPs runs ahead of the sweep, which then walked a table with nothing stale in it while the /32 route and its proxy-ARP stayed up. Give the sweep the host's own inventory. ListEIPIngress reads the EIPs back from the FORWARD accepts EnsureEIPIngress stamps with its own comment, so a binding is found by what installed it rather than by a row some other path is free to delete. Anything plumbed that intent no longer asks for is torn down, which also covers a crash between the OVN delete and the unbind. A failed inventory fails the pass: an empty list and an unreadable one are indistinguishable, and reading the second as the first is a sweep that reports success having done nothing. Close the window inside the pass too. The predecessor scrub now reads the row it is about to drop and releases that address's host state, so the strand does not exist between applyEIPs and the sweep rather than merely being repaired a step later. * fix: retry the release verify's libvirt calls like every other job does The verify job drives the same libvirt provider on the same hypervisors as the e2e cells, where a dropped connection under concurrent load is the steady state and is retried. Here it was a hard failure that reads as a broken release. It was skipped earlier only because the job checks out mulga and not spinifex, so the shared helpers were not on disk — a sparse checkout of .github/scripts costs nothing and makes apply and destroy behave like their e2e counterparts. * fix: do not report a rebooting DB instance available on the pre-reboot agent's health RegisterDBInstance stamped a fresh LastSeen while leaving Agent.EngineHealth at the value the previous agent process reported. After a reboot the new agent registers before it has probed anything, so the record read as a healthy engine with a beat newer than the transition, and the reconciler moved the instance to available while PostgreSQL was still starting up. A client that connects the moment the status turns green then gets "the database system is starting up". A register now discards the previous process's verdict and records when this agent process started. The restart reconciler requires that registration to be newer than the transition, which also closes the window where a beat lands between the engine being stopped and the VM going down. * test: record the credential kubectl sent when an EKS call comes back Unauthorized The IRSA pod subtest's first kubectl is the gateway-CA configmap, which lands before the pod dumps are registered, so a 401 there left no evidence at all. A verbose read-only probe reports whether an Authorization header was sent, which separates a token the apiserver rejected from one the exec plugin never made. * fix: report a starting engine promptly, and budget CI jobs from the suite The agent waited a full heartbeat interval before its first beat, so a restart could not be observed as finished for up to 30 seconds after the engine was already serving. Beat at startup instead, and report every 5 seconds until the engine has answered once — the control plane is actively waiting on that beat, and the steady-state cadence it hands back is for a database already serving. The rds suite is the one most sensitive to what else the hypervisor is doing: its DB VMs serialise behind a 4 GiB semaphore, so its wall clock is set by how long it waits for the budget. The same pass ran 45m beside four nightly cells and overran 50m beside nine, panicking with two tests still running. Raise its budget to 60m. That budget was then larger than the jobs meant to contain it. A job cancelled on timeout reports only "The operation was canceled" and takes the artifacts with it, so the inner `go test` timeout must always fire first — and it could not for rds on e2e.yml, nor for storagefault or diskperf, which have asked for 90m and 180m inside a 60m job since they were added. Both workflows now resolve their job budget from suite-sets.sh rather than carrying a literal: e2e.yml from the largest requested suite, e2e-suites.yml per matrix leg, so a leg is bounded by its own suite and not by the longest one beside it. The default push set resolves to exactly the timeouts both workflows had before. * fix: check a permutation cell's job budget against the suites it runs The guard only saw cells that name a suite, so the nine that run a topology's nightly subset were never checked against it — the same class of mistake the guard exists to catch, in the majority of the cells. Bound each by the longest suite in its subset: max() and not sum(), because whichever suite hangs must panic on its own timeout first, and a sum would demand a budget the set has never come close to using. Every cell clears it today. * fix: report why a PostgreSQL engine would not start, and stop failing a stop that landed Three defects, all found by the first rds pass that ever ran deep enough to reach them. A wait that saw an instance reach `failed` called t.Fatalf from inside the polling condition, and testify runs that condition on its own goroutine. The Goexit killed the tick and left the collector empty, which reads as the condition being satisfied — so the wait returned successfully on an instance it had just declared broken, logged "reached status available" about one that was failed, and let the test carry on against it. Six conditions across four suites call Fatal or require the same way. EventuallyErr now runs its condition on the test's goroutine, which makes all six behave as written. A DB VM stop has two drivers: the API call and the reconciler resuming one whose caller it assumes has died. When they overlap the loser is told the VM is not in a state to be stopped — which is the state the stop was heading for — and the customer's StopDBInstance returned InvalidDBInstanceState for a stop that had just completed. Both paths now treat a refused command the way they already treat an unanswered one: it says nothing about the VM, so the fleet view settles it. PostgreSQL had no equivalent of the engine log tail MariaDB has carried since it was added, so a postmaster that refuses a setting produced "engine did not respond on /run/postgresql:5432" and nothing else — on a guest with no SSH, whose console records only that pg_ctl could not start the server. The probe now quotes the tail of the log the image already directs the postmaster to, and the reboot and modify paths carry the agent's reason into the failure the way the create path always has. The reason reaches the customer through StatusInfos. The rds suite's budget goes to 90m and cell-27's job to 100m. Its wall clock is what the DB-VM queue costs rather than what the tests do — single tests logged 11m, 22m and 26m waiting for budget — and the pass above still had three of seventeen tests running at 60m. * fix: keep a DB VM recoverable when its agent cannot start, and flush the volume before a snapshot An rds-agent that failed to construct exited, so a transient IMDS credential resolution took the guest down for good: rds-datadir and rds-init cascade on the missing handoff, the engine never starts, and the control plane sees only silence until it declares the instance failed ten minutes later. Construction is retried like every other boot step instead. OpenRC skips start_post when the start itself fails, which is the one case with no diagnostic at all — the console carried nothing but "died". The log mirror moves out of the handoff wait so a failed start uses it too. A snapshot quiesced the engine and nothing else. The engine repairs its own files from the WAL, but postgresql.conf sits beside them and it does not: a restore came up on a torn tail and the postmaster stopped on a syntax error at the unterminated include hook. The quiesce now flushes the whole data mount, and the hook is renamed into place rather than appended, so a boot that finds a half written one replaces it instead of appending after its open quote. * fix: flush the config cloud-init wrote before a reset can take it A DB instance rebooted seconds after it went available came back with no RDS_GATEWAY_URL, so rds-agent could not reach the control plane and the engine never started. cloud-init writes agent.env once, per-instance, and never again; a reboot is a QMP system_reset with no guest shutdown, so the file was still dirty page cache when the machine reset under it and the next boot read nothing. The handoff wait now syncs, which is the point at which the agent has proved it read that file. * fix: stop dropping an EIP teardown published while vpcd is restarting DisassociateAddress publishes vpc.delete-nat fire-and-forget. On run 34046316024 vpcd disconnected from NATS at 03:51:26.73 and its replacement subscribed at 03:51:27.09; the publish landed at 03:51:26.92, in the gap, and went nowhere. ReleaseAddress then returned the address to the pool two minutes later with its /32 host route still delivering to the guest that had held it. vpc.delete-nat now uses request-reply like vpc.add-nat, and retries only on no-responders — which returns immediately, so the cost is paid solely when nothing is listening. A teardown that still fails is logged as an error naming the address rather than lost silently. Neither is fatal to the caller: the reconciler's orphan sweep remains the backstop. vpc.add-nat on the EIP path stays fire-and-forget. The shared helper blocks it on the flows barrier for up to 45s, and associating an address should not inherit that. Two reconcile passes in that run had the address absent from intent and did not sweep its host route, and the journal cannot say why: the sweep logged only when it acted, so a route it kept was indistinguishable from one it never saw. It now logs the bindings it found, kept and wanted. * ci: render one run-level table for the e2e nightly Reading a nightly meant opening nineteen jobs, and the Actions UI does not separate a cell that failed from one cancelled out from under it — a distinction every investigation of this suite has needed, and one the run 34046316024 analysis had to reach for the jobs API to get. A final job, needs: every cell and if: always(), renders cell, topology, result, duration, runner and per-cell test counts into the step summary, with run-level totals, the three slowest cells and the failing test names. Result, duration and runner come from the jobs API so a cell that died before writing junit still gets a row. Counts come from a new junit-only artifact per cell rather than the full cell artifact, so the roll-up does not pull nineteen journals it has no use for. Verified against run 34046316024: 14 success, 2 cancelled, 1 failure, with the two cancellations shown as cancelled rather than folded into failure. * fix: assert EC2's actual EIP semantics, and probe EKS auth on an authenticated path natuplink asserted that an instance's auto-assigned public IP is still plumbed after an EIP associated to the same instance is torn down. EC2 does the opposite: associating an EIP releases the auto-assigned address back into the pool, and it cannot be reused. Our datapath already does that — run 34059879666 shows 192.168.0.72's ingress removed at 07:09:44.000, 36ms before .73's was installed — so the test was asserting behaviour neither AWS nor we have. It now asserts the release at the point it happens, and that no public delivery remains after the teardown. This surfaced only once the teardown itself started working: the run's other assertions passed, including .73's removal, which was the failure before. Our records still disagree with our datapath — the ENI keeps the released address, which is why termination tears the old one down twice and never the current one. That is the remaining half of mulga-7zc9w and is not fixed here; it needs an IPAM release on the associate path. Separately, AuthDiagnostics read /version, which is anonymously readable, so its 200 was consistent with both branches it claimed to distinguish. It now runs auth whoami, reads kube-system, and invokes the credential plugin directly, reporting exit code, stderr and whether a token was produced — never the token, since CI logs outlive it. * test: capture per-guest state when a spread guest never answers SSH TestMultinodeSpread's only failure dump is dumpPlacementNATDiag, which collects OVN, OVS, ARP and route state aimed at the NAT and EIP half of the test. The failure it actually hits is a guest that never accepted SSH — run 34040324968, the guest on node3 at 10.100.2.5 — and for that it recorded no console, no port binding and no tap state. There was no way to tell from the artifact whether the guest booted at all. Now dumps, per guest: the console, the NB logical switch port, the SB port binding, the OVS interface backing its tap, and br-int's port list. Probed on every node rather than the expected one, because a port bound on the wrong chassis is exactly what a hosting-node-only dump cannot show. Covers all three guests, not just the failing one, so a working guest is available as the control. * ci: sample hypervisor load either side of every cell Eight runner names map onto three hypervisors, so cells that never share a runner still share a host: when cell 10's SSH wait expired on run 34040324968, cells 22, 17, 18 and 28 were concurrently live, two of them multi-node. Nothing in the artifacts measured host load, which left co-tenancy as an explanation that could be neither confirmed nor ruled out. Records loadavg, MemAvailable, SwapFree, the cpu line carrying steal, and the QEMU process count before and after each cell, into the cell artifact. * ci: note that main's missing concurrency block is why the nightly runs ungrouped Three refs carry three different groupings: main has no concurrency block at all, dev uses e2e-nightly, this branch uses e2e-hypervisors. Scheduled runs load the workflow from the default branch, so every cron nightly runs ungrouped and cannot queue against anything — which is how a dispatch of 34050630627 ran alongside the scheduled 34046316024 and cost it cells 24 and 27. e2e-hypervisors is the right group and deliberately broader than dev's, since e2e.yml, e2e-suites.yml and e2e-ddil.yml draw on the same runner pools. Nothing on a feature branch can fix this; the block has to reach main.	2026-09-06 16:53:32 -05:00
Julian Sommer	e36c8f6ba8	chore: update to go 1.27	2026-08-20 10:37:52 +10:00
Julian Sommer	c99b91bca5	fix: add publish system ami job to release	2026-08-17 09:56:30 +10:00
Josh Siviour	9102588e5a	chore: bump Go toolchain to 1.26.6 (#793)	2026-08-14 09:29:03 +10:00
Julian Sommer	7dfadd4576	fix: add bluebottle to scripts	2026-08-12 10:37:28 +10:00
Julian Sommer	5363d25ef5	fix: add northstar to scripts	2026-08-12 10:25:36 +10:00
dependabot[bot]	5dc945a6df	chore: bump the github-actions group with 2 updates (#552) ... Bumps the github-actions group with 2 updates: [actions/setup-go](https://github.com/actions/setup-go) and [actions/setup-node](https://github.com/actions/setup-node). Updates `actions/setup-go` from 6 to 7 - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](https://github.com/actions/setup-go/compare/v6...v7) Updates `actions/setup-node` from 6 to 7 - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](https://github.com/actions/setup-node/compare/v6...v7) --- updated-dependencies: - dependency-name: actions/setup-go dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: actions/setup-node dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2026-07-21 15:49:37 +10:00
Julian Sommer	47604f43dc	chore: go 1.26.5	2026-07-09 09:43:32 +10:00
dependabot[bot]	79e881f1c8	chore: bump actions/checkout from 6 to 7 in the github-actions group (#405)	2026-06-23 09:11:02 +10:00
Julian Sommer	51706938e8	chore: use ubuntu-26.04 runners	2026-06-19 16:04:09 +10:00
Julian Sommer	c6c26c04e2	chore: trim workflow comments	2026-06-12 11:04:46 +10:00
Josh Siviour	a4817b1afc	ci: run baremetal E2E nightly + hard-gate releases on it (#306) ... Make e2e-baremetal.yml a reusable workflow (workflow_call + a ref input) so the nightly and release pipelines can invoke the real PXE-reimaged hardware suite, which previously only ran on manual dispatch. Nightly: add a baremetal job (conceptual cell 20) that runs every weeknight on the existing cron, or on dispatch when 20 is selected. It tests dev on schedule, matching GIT_BRANCH. Sits outside the tofu matrix (different runner, provisioning, 120m timeout) and shares the e2e-baremetal-cluster concurrency group, so all baremetal runs serialise on the single physical node. Release: a release builds and uploads an ISO to R2 but never booted that ISO/squashfs path on hardware. Add a baremetal job (needs: release) and hard-gate on it: promotion moves to a dedicated promote job needs:[verify, baremetal] (with a matching mark-failed job on failure), and the iso job's upload gate now waits for both verify and baremetal to conclude success before pushing to R2. Bump the iso job timeout to 180m since it now waits on the parallel ~2h baremetal run.	2026-06-07 22:14:21 +10:00
joshsiv-mulga	3bb90c9fd3	fix(ci): guard release Clean Workspace against leaked iso-builder devtmpfs ... The iso job mounts a kernel-global devtmpfs under the workspace (.cache/rootfs/dev). A killed job skips its EXIT-trap unmount and leaks the bind; the verify job (same ci-single runner) then ran a bare 'rm -rf $WORKSPACE/*' that could traverse it and wipe the host /dev. Detach any mount under the workspace before deleting and add --one-file-system so rm never crosses a mount boundary. Same backstop added to the iso job's own cleanup.	2026-06-05 22:58:07 +10:00
tomnewton-mulga	6be3ccbb23	fix: unify .iso and .iso.sha filenames	2026-06-05 17:07:26 +10:00
tomnewton-mulga	158b3a25c5	fix: updated release.yml for clean iso uploads	2026-06-05 16:48:40 +10:00
Julian Sommer	9508aaacce	chore: bump go 1.26.4	2026-06-03 08:59:25 +10:00
dependabot[bot]	ef097bf4ab	chore: bump actions/setup-go from 5 to 6 in the github-actions group (#251)	2026-05-26 09:09:24 +10:00
tomnewton-mulga	e8ca596083	fix: secret for CF URL endpoint	2026-05-19 10:45:37 +10:00
tomnewton-mulga	01d9fa858f	feat: iso release flow (#229)	2026-05-19 09:07:26 +10:00
Josh Siviour	4ae2a4ac53	feat(elbv2): QEMU microvm direct-boot for LB system instances (#228) ... * feat(vm): direct-boot foundations + config/input extensions + image build pipeline vm package (mulga-siv-62): - Add KernelImage, Initrd, KernelCmdline, FwCfg/FwCfgEntry to vm.Config - Emit -kernel/-initrd/-append/-fw_cfg args when fields set - Relax drive invariant: accept kernel image in place of drives - Add devices.go: IsMMIO predicate, NetDevice/BlkDevice/RngDevice helpers - Refactor lifecycle.go + network_helpers.go call sites to use helpers - shutdown.go: os.Remove fw_cfg tmpfiles after QEMU exits - Add devices_test.go, microvm_test.go; extend vm_test.go config/input extensions (mulga-siv-63): - Add MicrovmConfig{ELBv2Enabled, ImagePath} to DaemonConfig - ImagePath falls back to /usr/share/spinifex/microvm when empty - Add commented [microvm] section to spinifex.toml template - Add DirectBoot bool + NICs []NICConfig to SystemInstanceInput image build pipeline (mulga-siv-64): - scripts/build-microvm-image.sh: Alpine chroot build producing vmlinuz + initramfs.cpio.gz - build/microvm/init.sh: fw_cfg-driven PID 1 init (mounts, modprobe, NIC config, lb-agent-env, ca-cert) - build/microvm/inittab: BusyBox init with lb-agent respawn - make build-microvm-image target - setup.sh: install microvm/* to /usr/share/spinifex/microvm/ - .gitignore: exclude built vmlinuz + initramfs.cpio.gz artifacts * feat(daemon): direct-boot launch path for microvm system instances When SystemInstanceInput.DirectBoot=true: - Skip GetAMIConfig and GenerateVolumes - Build vm.Config directly (no buildBaseVMConfig, no PCIe root ports) - MachineType=microvm,x-option-roms=off,pic=off,pit=off,rtc=on,acpi=on,isa-serial=on - Resolve kernel/initrd from Microvm.ImagePath config field - Serialise NICs to three fw_cfg tmpfiles (netcfg, lb-agent-env, ca-cert) - Validate exactly one NICConfig.IsDefault=true before writing netcfg blob - Shutdown path cleans up tmpfiles via existing Config.FwCfg iteration PC-machine path (DirectBoot=false) unchanged. Closes mulga-siv-65 * feat(elbv2): wire microvm.elbv2_enabled flag into ELBv2 service When ELBv2Enabled=true: - Drop lbVMUserData cloud-config; emit KEY=value LBAgentEnv blob instead - Set SystemInstanceInput.DirectBoot=true; omit ImageID - Populate NICs []NICConfig (index 0=primary ENI IsDefault, 1=mgmt, 2+=extra ENIs) - Validate exactly one IsDefault NIC at service layer When ELBv2Enabled=false: existing PC-machine path unchanged. Adds buildLBAgentEnv, buildMicrovmNICs, subnetCIDRForIP, subnetGatewayIP helpers. Tests in arn_test.go cover both flag-on and flag-off paths. Closes mulga-siv-66 * test(daemon,vm): add unit tests for direct-boot helpers to meet diff coverage threshold * feat(daemon): Phase 1 perf instrumentation + e2e runner microvm flag - Log t1→t2 boot timing via slog on the direct-boot path - measureQEMURSSMiB helper reads /proc/<pid>/status (tested) - Gate constants: phase1BootGateMs=500, phase1RSSGateMiB=25 - run-lb-e2e.sh: --microvm flag, dual-path CI annotation, density test note Closes mulga-siv-67 * fix(build): remove busybox-initscripts — not in Alpine 3.20, not needed * fix(microvm): end-to-end fixes for direct-boot LB path Serial output was silent (x-option-roms=off suppresses BIOS serial init); fixed by removing it from the machine string and using -serial chardev:console0 to attach to the machine-created ISA serial device instead of adding a duplicate isa-serial device (which would create ttyS1 while kernel uses ttyS0). virtio-net interfaces were invisible to the kernel because QEMU's auto-kernel-cmdline does not append virtio_mmio.device= params when -append is used. Fixed by generating explicit virtio_mmio.device=0x200@0xfebNNNNN:N params for each NIC in KernelCmdline, and modprobing virtio_mmio + virtio_net in init.sh. sys.micro bumped to 512 MiB — 128 MiB caused kernel OOM during initramfs decompression (~68 MiB uncompressed). BusyBox init ::respawn: entries do not source or export env files; fixed inittab to use set -a; . /etc/conf.d/lb-agent; set +a before exec. wireLBAgentConfig was documented as reading the CA cert but never set elbv2Service.CACert; lb-agent TLS handshakes to AWSGW failed with x509: certificate signed by unknown authority. Fixed by reading d.config.NATS.CACert (same local CA that signs the AWSGW server cert) and assigning to elbv2Service.CACert so writeFwCfgBlobs writes a non-empty cacert blob into fw_cfg. Adds vm_test TestExecute_MicrovmFileChardev and updates lifecycle_test to assert SerialSocket is intentionally empty on direct-boot VMs. * fix(build): auto-detect docker/podman when apk unavailable for microvm image build * fix(build): create /dev/console and /dev/null as proper device nodes in initramfs tar extraction without root silently demotes device files to 0-byte regular files (mknod requires CAP_MKNOD). The kernel opens /dev/console when wiring init's stdio to the serial console; a regular file there causes all init output to be silently discarded to RAM — the VM appears stuck at 'Run /init as init process' with no subsequent console output and lb-agent never heartbeats. Fix: after rootfs population, wipe /dev and recreate console (c 5 1) and null (c 1 3) via sudo mknod. devtmpfs provides all other devices at runtime once init.sh mounts it. * fix(microvm): fix mgmt NIC routing for all LB schemes Two bugs prevented lb-agent from reaching the AWSGW via the mgmt interface: 1. RouteDst/RouteVia swapped — resolveMgmtRoute returns (gateway, target) but buildMicrovmNICs assigned them as (Dst=gateway, Via=target), producing 'ip route add 10.15.8.1 via 192.168.1.33' instead of the correct 'ip route add 192.168.1.33 via 10.15.8.1'. 2. NIC[1] MAC and CIDR never filled — the daemon allocates mgmt IP/MAC after buildMicrovmNICs returns, but didn't inject them into input.NICs[1] before calling writeFwCfgBlobs, leaving NIC1_MAC= empty in the netcfg blob and skipping interface setup in init.sh entirely. Also extends mgmt routing to internet-facing scheme in the microvm path: VPC SNAT is not reliable across all VPC topologies for control-plane heartbeats; mgmt is always a direct L2 path to the host. * feat(microvm): Phase 2 kernel stripping + boot optimisation - cpu: switch to host,-pmu (disable PMU — not needed in microvm guest) - machine type: make isa-serial configurable via microvm.isa_serial config field; default true (on) preserves console log path - build: strip unused kernel modules (USB, IDE, ATA, GPU, media, input, HID, bluetooth, staging, sound, wireless) from initramfs before cpio - build: add Phase 2 artifact size gate — fail if vmlinuz + initramfs.cpio.gz exceeds MICROVM_ARTIFACT_MAX_MiB (default 50 MiB); override via env var during development - perf: add phase2BootGateMs=300 and phase2ArtifactGateMiB=50 constants with canary tests to catch unintentional gate changes - tests: cover buildMicrovmNICs nil-VPC and extra-ENI paths - tests: cover microvmMachineType nil/true/false branches * fix(build): use find-based virtio+fw_cfg module keep-list in initramfs strip The directory-only keep-list (drivers/virtio, drivers/firmware) missed virtio_net.ko which lives in drivers/net/, virtio_blk in drivers/block/, and virtio-rng in drivers/char/hw_random/. Switch to find-based approach: collect all virtio*.ko* and *fw_cfg*.ko* regardless of subdirectory, then restore them into their original relative paths under kernel/. Artifacts now: 42 MiB (vmlinuz 11 MiB + initramfs 31 MiB), gate passes. * fix(build): keep failover modules in initramfs strip virtio_net depends on net_failover/failover for live-migration support. Stripping them left virtio_net with unresolved symbols (net_failover_create/destroy), so the module failed to load and the guest came up with only lo. init.sh then skipped every NIC ("no interface found for MAC ..."), leaving lb-agent with no network path to AWSGW and the LB stuck in provisioning. Add *failover*.ko* to the keep-list. Image size unchanged at 42 MiB (under the 50 MiB Phase 2 gate). ELBv2 e2e now reaches active and 2/2 targets healthy for both ALB and NLB internal LBs. * fix(daemon): restore slog default after viperblock.New mutates it viperblock.New unconditionally calls vb.SetDebug, which calls slog.SetDefault with a TextHandler at LevelError — globally silencing every slog.Info and slog.Warn in the caller process for the rest of its life. Discovered while validating microvm Phase 2 (mulga-siv-68): the daemon stopped emitting Info logs the moment the first customer VM with volumes launched, hiding direct-boot timing, LaunchSystemInstance completed, and CreateLoadBalancer completed lines that we need for boot-time tuning. Workaround in newViperblock: re-install an Info-level TextHandler on os.Stderr after each viperblock.New call. Also drop the three explicit vb.SetDebug(false) calls in prepareRootVolume / prepareEFIVolume / prepareCloudInitVolume — they were no-ops (viperblock.New already runs SetDebug internally) and each one re-triggered the global mutation we are trying to undo. Proper fix is in viperblock itself — tracked as mulga-siv-70: SetDebug should not call slog.SetDefault. Until then, this restore call is the shortest path to getting the daemon's observability back so we can measure t1_to_t2_ms and tune Phase 2. Preflight note: golangci-lint not available on this dev host, so the project preflight target cannot run end-to-end. Verified with go vet ./spinifex/handlers/ec2/instance/... (clean) and go test ./spinifex/handlers/ec2/instance/... (pass). * perf(microvm): hit Phase 2 boot/memory gates t1→t2 hypervisor: 3224 ms → 264 ms (12×). 128 MiB guest now boots cleanly without OOM, reverting the temporary 512 MiB bump. Root causes: 1. build-microvm-image.sh staged the entire chroot into the cpio without dropping /boot, embedding a duplicate vmlinuz, Alpine's stock initramfs-virt, System.map, and a "boot → ." symlink loop (~25 MiB). Decompression peak exceeded 128 MiB. Strip /boot after the kernel is copied out, before the cpio. Initramfs drops from 31 MiB → 11 MiB (53 MiB → 28 MiB uncompressed); total artifact 42 MiB → 22 MiB. 2. vm/lifecycle.go burned 3 s of fixed sleep on every direct-boot launch: 2 s nbdkit pre-exec wait (no nbdkit on direct-boot) and 1 s post-fork settle (QEMU binds QMP within ~10 ms when there is no firmware, ROM, or block layer). Both extracted to directBoot-aware helpers — nbdkitPreExecWait collapses to 0, and qemuStartSettleWait to 50 ms. 3. KernelCmdline only carried `console=ttyS0 virtio_mmio.device=…`. Added boot-time perf flags: quiet loglevel=3 (silence printk over 115200-baud isa-serial), mitigations=off (KVM guest on trusted host, single-tenant LB workload), tsc=reliable, no_timer_check, reboot=t, i8042.{nopnp,noaux,nokbd} (microvm has no PS/2). 4. init.sh dropped modprobe virtio_mmio (built-in: CONFIG_VIRTIO_MMIO=y) and tightened polling: fw_cfg/iface waits go from 10 × 200 ms (up to 2 s) to 5 × 20 ms (up to 100 ms). Phase 2 gates: - t1→t2 ≤ 300 ms ✅ 264 ms / 281 ms - artifact ≤ 50 MiB ✅ 22 MiB (gate already in build script) - ELBv2 e2e green ✅ 32/33 (pre-existing client-VM failure, unrelated to microvm path) * build(microvm): wire build into deploy + distro + release pipeline Adds install-microvm Makefile target (with grouped-target staleness detection on vmlinuz/initramfs.cpio.gz), wires it into dev-install.sh and reset-dev-env.sh, makes 'make deploy' reinstall artifacts when present, and makes 'make distro-amd64' stage build/microvm/* into the release tarball staging dir. arm64 distro stays unchanged (no aarch64 microvm artifact yet — Phase 4). Release workflow gains a setup-go step and a 'make build-microvm-image' step before 'make distro-amd64' on the amd64 runner so the resulting tarball includes the kernel + initramfs without a manual copy. Precursor for the Phase 3 PC-machine LB removal (mulga-siv-69). Closes mulga-siv-71. * build(ansible): include microvm artifacts in build role tarball Adds a build-microvm-image task to the ansible build role and stages build/microvm/* into the role's release tarball so the install role's setup.sh run picks them up via INSTALL_SPINIFEX_TARBALL. Skipped gracefully when the build script is absent (older branches without direct-boot LB wiring). * feat(config): enable microvm direct-boot LB path by default Uncomments the [nodes.<node>.daemon.microvm] section in the generated spinifex.toml template and sets elbv2_enabled = true so all new clusters bring up ELBv2 LBs via the direct-boot microvm path. Operators can opt back to the legacy PC-machine path by flipping the flag to false in the rendered config. Required for end-to-end CI on this branch to exercise the direct-boot path. Existing clusters' configs are unchanged (the template only affects fresh 'spx admin init' renders). * fix(microvm): bootstrap apk.static when no host rootfs tool exists Self-hosted CI runners and the GitHub-hosted ubuntu-24.04 image often have none of apk, docker, or podman installed. apk-tools is not in the Debian 13 (trixie) repositories. Without one of these the rootfs build fails up front with 'no rootfs build tool available'. When all three are absent, fetch apk-tools-static from the configured Alpine mirror, extract it to a tempdir, and prepend it to PATH so the existing native-apk strategy below picks it up transparently. Requires only curl + tar, both of which are present on every CI runner that can already check out source. No-op when apk, docker, or podman is already present. Unblocks https://github.com/mulgadc/spinifex/actions/runs/25988830244 * revert: drop apk.static bootstrap in build-microvm-image Reverts 0bd76b7. Self-hosted runners will get docker.io installed out of band; existing Strategy 2 (docker run alpine:3.20) handles the rootfs build without host sudo or unsigned package install. * docs(workbook): drop lb-alpine import for microvm direct-boot LB LBs now launch via direct-boot QEMU microvm using kernel + initramfs bundled at /usr/share/spinifex/microvm/, so the legacy lb-alpine system AMI import is no longer required. * fix(daemon): own tap devices by daemon user so QEMU can attach without CAP_NET_ADMIN When the daemon runs non-root (production: User=spinifex-daemon), child QEMU processes inherit that uid but lack CAP_NET_ADMIN. The kernel's tun attach check then requires the tap owner uid to match the caller's euid. Without the `user` flag, `ip tuntap add` defaults to root:root and qemu's TUNSETIFF fails with EPERM ("could not configure /dev/net/tun: Operation not permitted"). Surfaced by the e2e-nightly cell-17 microvm LB launch on the ci-pseudo runner where the kernel enforces strictly; legacy EC2 launches on ci-single runners happened to attach successfully despite the same code path. Adding `user $current group $gid` to `ip tuntap add` ensures the tap is owned by whatever uid the daemon (and its qemu children) run as. Root daemons skip the flag — they have CAP_NET_ADMIN ambient. * fix(e2e): redirect lb probe chatter to stderr so stalls are visible test_http_from / test_tcp_from echoed progress to stdout, which the callers captured into *_OK vars via $(...). Result: 6+ minute retry loops ran silently in CI, and the "$ok" comparison never matched "true" because the captured value contained the full multi-line chatter. Reachability pass/fail counters were also lost in the subshell. Redirect helper output to stderr, leave only the true/false flag on stdout, and move pass/fail to the caller so they actually count. * fix(microvm): don't /32 AdvertiseIP via mgmt for internet-facing single-node buildMicrovmNICs overrode resolveMgmtRoute's deliberate empty return for the single-node internet-facing case by falling back to (MgmtBridgeIP, AdvertiseIP). That installed a /32 route to AdvertiseIP via the mgmt NIC inside the LB microvm. On single-node where the AWSGW host IP equals the WAN client IP, the /32 also catches reply traffic for inbound HTTP/TCP to the LB's floating IP — replies leave via the mgmt NIC with src=data-NIC IP, the host kernel drops the asymmetric flow, and same-chassis ingress hangs. resolveMgmtRoute already encodes the correct policy: - multi-node any scheme → explicit MgmtRoute{Gateway,Target} - single-node internal → fallback to (MgmtBridgeIP, AdvertiseIP) - single-node internet-facing → empty (reach AWSGW via VPC SNAT) Honor it. Matches the legacy lbVMUserData path, whose own comment flagged the stolen-WAN-return-path symptom this introduced. Test covers all three branches. * feat(ansible): dev-deploy playbook for in-place artifact swap Fills the gap between ansible-dev-install (skips when already installed) and ansible-dev-reset (full teardown + reinstall + smoketest). Mirrors Makefile `deploy` target: rebuild + replace binaries and microvm artifacts on a running node, restart spinifex.target, wait for awsgw. Skips setup.sh (no user/dir/systemd-unit changes) and smoketest so iteration is fast. Use ansible-dev-reset when changing systemd units, helper scripts, logrotate, or setup.sh itself. Adds: - roles/deploy/tasks/main.yml — install spx + nbdkit plugin + microvm artifacts, with plugindir auto-detection via `nbdkit --dump-config`. - playbooks/dev-deploy.yml — chains build + deploy roles. - Makefile ansible-dev-deploy target. - README playbook table + "when to use which" guidance. * chore(microvm): remove legacy LB Alpine image build pipeline Phase 3 microvm removal — strip the PC-machine LB image build artifacts that direct-boot no longer needs. The microvm initramfs already carries lb-agent + HAProxy via build-microvm-image, so the lb.conf manifest, lb-setup.sh chroot script, build-lb-image target, dev-install / reset-dev-env import steps, and ansible LB-image tasks all become dead. build-system-image stays — other manifests (e.g. GPU images) still use it. * refactor(microvm): remove legacy PC-machine LB boot path Phase 3 — direct-boot becomes the only ELBv2 LB launch path. - Drop SystemInstanceInput.DirectBoot / ImageID / UserData (every LB caller set DirectBoot=true; legacy cloud-init userdata is gone). - Drop ELBv2ServiceImpl.lbVMUserData, systemAMI* fields, SetSystemAMIFunc, getSystemAMI, and the lazy-AMI wiring in daemon.startCluster. - Collapse CreateLoadBalancer to a single direct-boot launch input. - Collapse daemon.LaunchSystemInstance: drop AMI validation, ImageID on RunInstancesInput, UserData base64 encode, GenerateVolumes branch. buildDirectBootConfig hardcodes /usr/share/spinifex/microvm and isa-serial=on (microvmMachineType no longer takes a flag). - Drop MicrovmConfig from config.DaemonConfig and the [daemon.microvm] section from the spinifex.toml template. - Prune dead tests (lbVMUserData_*, PCMachine_Unchanged, AMI resolver paths) and refresh stale comments that still mentioned lbVMUserData. ARM hosts have no LB launch path until Phase 4 adds direct-boot on virt — CreateLoadBalancer on aarch64 fails naturally at QEMU exec. * docs: drop lb-alpine catalog entry from cluster setup guide LB system instances no longer use an AMI — direct-boot microvm artifacts ship in the spinifex package. * feat(ui): remove LB image import banner + LB-AMI gating Direct-boot microvm artifacts ship in the spinifex package, so the LB system AMI no longer exists — the "Load balancer image not imported" banner and the Create-button disable on the LB list / wizard pages can never fire. Strip: - lb-image-missing-banner.tsx component - hasLbImage / isLbImage / LB_IMAGE_NAME / LB_MANAGED_BY_VALUE from system-managed.ts (isSystemManagedImage stays — still used by the EC2 Images pages to hide system AMIs) - Banner gating + ec2ImagesQueryOptions imports from the LB list and wizard pages - The corresponding tests that asserted the banner appears * revert: restore lb.conf + lb-setup.sh for mulga install pipeline Phase 3 microvm removal (commit 6313d74) deleted these too early — the mulga install/iso artifact builders (scripts/tofu-cluster/build-install -artifacts.sh, scripts/tofu-iso/build-iso-artifacts.sh) still invoke scripts/build-system-image.sh scripts/images/lb.conf, breaking e2e with "ERROR: Manifest not found: scripts/images/lb.conf". Restore both files verbatim so the legacy pipeline keeps building until the mulga-side migration to microvm-only artifacts lands post-merge.	2026-05-18 15:48:04 +10:00
Julian Sommer	84278b794f	chore: reintroduce dev branch in CI + scripts ... - workflows: add 'dev' to push/PR triggers (spinifex.yml, ui.yml, e2e.yml) - workflows: drop "Resolve Mulga Branch" fallback step; mulga is single-branch so checkout@v6 uses the default branch implicitly - dependabot: route gomod/github-actions/npm PRs to dev - diff-coverage.sh: pick origin/dev as base for feat branches, origin/main for dev - iac/proxmox-template: default git_branch to dev for cluster builds Refs docs/development/improvements/reintroduce-dev-branch.md	2026-04-30 11:50:13 +10:00
Julian Sommer	1ca8621e16	fix: allow mulga feat branch	2026-04-21 11:14:29 +10:00
Julian Sommer	68ae17dcf1	chore: rename app-id to client-id	2026-04-15 15:03:16 +10:00
Julian Sommer	5b39efb078	fix: make e2e use env vars	2026-04-15 13:40:22 +10:00
Julian Sommer	7cf0783122	fix: set dev tags as prerelease	2026-04-14 13:29:05 +10:00
dependabot[bot]	36da673ecd	chore: bump the github-actions group with 2 updates (#133)	2026-04-14 08:59:01 +10:00
Julian Sommer	dda4575888	fix: use binary install for single node e2e (#111)	2026-04-02 16:22:41 +11:00
Julian Sommer	7c6f03f0d0	chore: update action versions	2026-03-26 10:45:02 +10:00
Julian Sommer	b7678ab7c5	feat: fix silent failures across spinifex (P2 audit H-25 through H-43) ... - PID file failures now prevent service start instead of logging and continuing - Status() methods use PID-based health checks instead of returning empty string - NewResourceManager returns error instead of silently defaulting to 8GB - Crash recovery re-allocates resources before relaunch - restoreInstances stops after failed resource allocation - Cluster manager bind errors propagated to Start() - NATS disconnect/reconnect logged at Warn level - VPC CIDR fallback logs warning about overbroad rule - vpcd respond() handles json.Marshal failure - Network cleanup failures logged instead of silently discarded - fmt.Println replaced with slog.Debug in NATS service - Signature mismatch no longer logs expected signature - extractDiskImagePath uses errors.Is for EOF - validateDiskImagePath checks file command error - os.Stat uncommented in admin image import - net.SplitHostPort errors checked - Gateway response write errors logged - vpcd OVN NB address uses leader IP on non-primary nodes - Release workflow parallelized with native arm64 runner	2026-03-19 13:37:41 +10:00
Julian Sommer	11730be685	fix: locate release assets by filename instead of hardcoded path	2026-03-19 12:13:28 +10:00
Julian Sommer	fafc104c30	feat: add release verification job with draft-then-promote flow	2026-03-19 10:52:39 +10:00
Julian Sommer	58e6817efe	chore: bump docker build action ver	2026-03-19 10:19:02 +10:00
Julian Sommer	9c430eea07	fix: release workflow	2026-03-19 08:17:30 +10:00
Julian Sommer	7d43d0f782	hopefully last rename	2026-03-18 13:24:45 +10:00
Ben Duncan	55186d95f2	refactor: rename CI/CD workflows from hive to spinifex ... - Rename hive.yml → spinifex.yml - Update workflow names, path triggers, working directories - Update release.yml tarball references - Update e2e.yml test paths - Update ui.yml frontend paths	2026-03-18 01:09:02 +00:00
Julian Sommer	48bbe423df	fix: complete termination lifecycle for failed instances	2026-03-18 09:53:48 +10:00
Ben Duncan	a458a28406	new binary mode and systemd process	2026-03-16 11:57:52 +00:00