mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-22 12:16:52 +00:00
Ben Duncan
021f439bab
* feat: recover instances automatically from a node that stopped heartbeating A node that dies hard takes its guests with it, and nothing brings them back until an operator brings that node back. Every daemon now runs the same small reconciler: it reads the instance records, selects desired-running instances whose owner is stale, and relaunches them here. The claim is a CAS on Status.LastNode and that is the whole of the coordination. Several survivors race one instance, exactly one revision wins, and the losers find a record naming a live owner and stop considering it. A claimant that dies mid-launch leaves the record naming a node that is itself stale, which is the same candidate the next pass derives again, so there is no lease to expire and nothing to reconcile by hand. Ownership is not permission to write. The volume leases still decide whether the guest can move, and VolumeMounter.Mount runs before startQEMU, so an attempt that loses the volume race has touched no tap, no OVS port and no OVN binding. A storage failure is not a host failure and wants the opposite response. A guest paused by werror=stop publishes Health.IOErrorSince, and a record carrying it is refused: the volumes are the same objects in the same store from anywhere, so relaunching moves the pause rather than ending it. A node whose own object store is not answering stands down entirely, which separates a store broken here from a store broken everywhere without either having to be diagnosed. A launch that does not complete backs this node off that instance, doubling to a cap, because nothing on the mount path can yet say which of the two it hit. Recovery is off by default and inert below three nodes. It also waits until it has seen its peers live, or five minutes, so the first node back from a coordinated shutdown does not recover guests from nodes that are still booting. * test: prove instance recovery moves a host failure and refuses a storage failure The e2e half of automatic instance recovery, asserted from both sides because a reconciler that cannot tell the two failures apart passes one and fails the other. InstanceAutoRecovery takes a node down and requires the guest back on exactly one survivor, identified by the qemu process rather than by what the API reports. InstanceRecoveryRefusesStorageFault stops predastore cluster-wide first, so the guest is paused by werror=stop when its node goes away, and requires that nothing moves. Also fixes the settle window, which re-evaluated every pass and so held a daemon back for its first five minutes over exactly the failure the reconciler exists to handle. It now latches once every peer has been seen live. harness.InstanceHostingNodes replaces the single-node lookup so a caller can assert there is exactly one host rather than assume it, and skip a node it has deliberately taken down. * fix: make automatic instance recovery a cluster-wide setting Recovery was read from the per-node Config, so it had to be written three times on a three-node cluster, once inside each [nodes.<name>] table. A top-level [recovery] table — the obvious thing to write, and what the e2e suite writes — was parsed into nothing and the daemon reported recovery disabled on every node with the setting plainly in the file. Cluster-wide is also the honest shape. A node with recovery off is not opting out of anything: recovery is what survivors do to a dead node's instances, so the setting belongs where there is one of it. * test: drive real I/O into the storage-fault case, and freeze the store rather than stop it Measured on a three-node cluster: predastore stopped cluster-wide for twelve minutes against idle guests paused none of them. Viperblock serves what it has cached and an idle guest issues nothing that reaches the object store, so the test asserted that nothing moved in a run where nothing was ever faulted. The victim now launches with cloud-init user-data that writes to its root volume continuously. User-data rather than SSH because a routed-NAT cell gives guests no public address, so a test that had to log in could not run on half of them. The fault is SIGSTOP rather than a unit stop, for the reason the storagefault suite already gives: a frozen process holds its connections open and answers nothing. Under that fault and that load the first BLOCK_IO_ERROR arrived 4m47s later, so the pause budget is now ten minutes rather than three. * test: give instance recovery its own suite and its own nightly cell Both tests take a node away and one of them freezes predastore cluster-wide, so anything sharing the environment reports that outage as its own failure — the same argument storagefault already makes for being its own package. They are also slow by nature: a dead owner has to be seen stale twice, a guest has to boot, and the storage case waits for a pause measured at 4m47s and then holds four minutes to prove nothing moved. Roughly twenty minutes on top of a permutation cell budgeted at thirty-five. Cell 31 is three-node and static, matching storagefault's, with a 75-minute job budget over a 60-minute suite timeout. The suite is eligible on multi-node environments and deliberately not in the nightly permutation set. * fix: give a guest long enough to open a volume the object store is slow to serve QEMU opens every drive before its monitor answers a greeting, so the QMP greeting deadline is really a deadline on opening the guest's volumes. At 30s it was shorter than a cold viperblock open costs on a cluster that has lost a node, where every object read is served degraded: measured at 35s with one node of three away. The launch killed the guest five seconds short of ready and then did it again on every retry, which is a recovery that can never complete. A network-backed drive now gets its own deadline for the same reason VFIO already had one — synchronous work before the monitor answers — and the instancerecovery suite is what found it. Also make the suite's predastore thaw best effort per node. It runs after a node has been taken away, where there is no process to signal, and a node it cannot reach must not stop it restoring the rest. * docs: name the two newest cells in the dispatch input The cells input is the operator's index of what a number means, and it stopped at 29 while 30 and 31 were added. * test: prove the guest is writing before freezing the store in the recovery suite The storage-fault test froze the object store as soon as RunInstances reported running, which is when QEMU has launched rather than when the guest has booted. On the nightly cluster the freeze landed while the guest was still in UEFI, so the only drive it could reach was the EFI varstore — and pflash has no werror=stop to pause on, so the guest never paused and the test reported a product failure it had caused itself. The load driver now prints to the serial console after each pass it has written and synced, and the test waits for that before freezing. The console is read through GetConsoleOutput, so this still needs no login and still runs on cells whose guests have no public address. * feat: recover instances automatically with nothing to configure The switch is gone. An instance whose host has failed is down whether or not recovery is enabled, so the only outcome a disabled reconciler can produce is leaving it down — which is the state recovery exists to end, not a safer one to fall back to. Against that, the switch cost a cluster that can be configured into never recovering anything, with no symptom until a node dies. This feature already hit that once: a top-level table parsed into nothing and every node reported recovery off with the setting plainly in the file. What makes recovery inert is now only facts about the cluster — no cluster config, no JetStream, or fewer than three nodes — and each says in the log which one applied. The e2e no longer writes config or restarts daemons: it waits for every node to report the reconciler active on a cluster nothing configured, so reintroducing a switch fails there as well as in the unit tests. * feat: stop retrying a recovery that is never going to work Every failed attempt used to go to the same exponential backoff and stay there. An instance nothing could launch was retried every fifteen minutes forever, still observed-running on a node that is gone, with no account of it anywhere the customer or an operator would look. A failure is now classified and counted. Two classes are read from what the launch actually reports: the node it was taken from still holds a volume lease, which clears on that lease's clock and is retried on a faster one, and this node has no room, which another survivor may. There is no terminal class, because nothing on this path can prove one — the launch collapses everything else into ServerInternal, so a verdict of "never going to work" would be a guess, and the guess that abandons a recoverable instance is the expensive one. The budget bounds the unknown case instead. Every class counts against it, since a class that did not would be the old unbounded retry under a narrower condition. Exhausting it leaves the instance where AWS leaves one whose host failed and could not be brought back: stopped, carrying a StateReason that names the cause and the last error, one StartInstances from running again. Writing the desired state down is also what makes one node's decision cluster-wide, because selection reads desired state — the same property the claim already has. A record another node has since claimed is never touched, so a node exhausting its attempts cannot stop a guest that is now running elsewhere. * fix: give a guest the time for every cost it pays before its monitor answers A GPU instance booting from a network-backed root volume pays both costs: QEMU opens every drive across the object store and then pins the whole of guest RAM, one after the other, all before the monitor answers. The deadline took whichever was larger, so such a guest was given less time than it needs. That the two floors are both 180s is why nothing has been seen yet — the VFIO branch has always returned at least the network-drive deadline. It is the same defect either way, and the symptom when it lands is a GPU instance SIGKILLed on every attempt while its plain sibling launches, which reads as a GPU fault rather than as a deadline. The costs are now added, the plain QEMU startup is counted once rather than replaced, and the cap covers both floors together so it cannot clip a guest before either cost has been waited out. * test: assert the AWS-visible arc of a host failure end to end A customer cannot see which host an instance is on, and AWS never tells them. What they can see is DescribeInstanceStatus, so that is where a recovery has to be provable, and the suite never looked at it. Three frames are now required, in order: running with system-status ok before anything is broken, system-status impaired while the owning node is gone, and ok again once a survivor is running it. The middle frame is the one worth having — the window opens when the owner's heartbeat goes stale, which is strictly before any survivor may claim, and closes when the relaunch registers, so it is bounded rather than lucky. Reporting ok throughout would be the more comfortable lie. The availability zone is compared across the first and last frames as well. A recovery that crossed one would break every placement promise made to the customer, and nothing else in the suite would have noticed. * test: prove a half-healthy node fences itself and is never a second writer The two existing recovery cases take a node away by stopping its services, which is the failure this feature handles least interestingly: a node that is gone has no opinion about who owns its volumes. The one that matters is the node that is still up, still running QEMU, still answering its own local clients, and simply cannot reach consensus. That is the state in which two nodes can each believe they own a volume, and the only thing between that and a corrupted disk is that the partitioned node gives up first, on a clock it can evaluate without reaching anything. So this drops NATS on 4222 and 4248 between one node and its peers and nothing else, then asserts the ordering the safety argument rests on rather than the outcome it produces. A watch samples every node every three seconds for the whole run, including the heal, and fails on any single sample naming two hosts. Two guests writing one volume through one object store is data loss already under way, not a race that resolves, and no point-in-time assertion can see one that appeared between its own samples. The injector takes its own nft table rather than touching inet spinifex_filter, because a test that flushed the node's real firewall would leave it either unfirewalled or misfirewalled with no sign of which, and a separate table at priority -300 heals in one atomic delete. A transient systemd timer removes it independently, so a cancelled workflow does not leave a node cut off from its cluster until somebody notices. It proves SSH still works and proves a peer port is genuinely unreachable before anything is asserted, since a rule set that installed cleanly and dropped nothing would read as the system tolerating a partition. The suite budget goes to 90m and cell 31 to 105m to fit the third case. * fix: move an instance's boot state, volumes, logical port and address with it A record-driven launch assumes the node reading the record is the node that wrote it. On a daemon restart that holds. On a recovery onto a survivor it does not, and five things follow from it — one that stops a moved instance booting at all, and four that stop it running anywhere. A load balancer never comes back from a host failure. Its boot configuration is a set of files under the runtime directory and the record holds their paths, so the survivor restores paths that exist only on the node that died. The hook that rebuilds them was reachable from Restore alone, and the cross-node path goes straight to vmMgr.Run — so QEMU is handed a netcfg, an agent env and a CA cert that are not there. The gateway URL baked into that env names the creating node's management bridge as well, which the rebuild corrects because it rebuilds from whichever node is running it. Firing the hook on both paths fixes both, and a rebuild that fails now refuses the launch: a guest that boots without its network configuration is worse than one that does not boot, because the records would say it recovered. The export outlives the guest and holds the volume lease. A returning node is healthy, so it renews that lease indefinitely and the new owner never acquires: the guest is stopped here and refused there, recovery spends its attempt budget on a volume that was never going to be free, and gives up on an instance that was never unrecoverable. So the export is given up too, over a new node-addressed request that tears it down exactly as the fence does. Deliberately not an unmount — an unmount seals, this node's block map is the stale one, and sealing publishes it over the winner's. Unlike the fence it deletes the lease key rather than leaving it to expire, because the lease is still this node's to hand back. The tap outlives the guest, and the tap is what claims the instance's OVN logical port. The binding names one chassis, so two nodes offering the same iface-id do not split the traffic — they contend, and the address answers from whichever won last. It also blocks the layer above: the external IP's host ingress is pruned on the evidence that the owning guest sits on another chassis, and that stays false while this node is still claiming the port. The management address outlives the guest as a cluster-wide reservation, and the release was unconditional. br-mgmt is one flat L2 segment, so the node an instance left was freeing an address still in use and offering it to the next instance anywhere. Only the node an entry names may release it now, with the takeover recorded on arrival so the node actually running the instance can still give the address back. An entry written before the node field existed names nobody and stays releasable, because there is no other holder to collide with. The fence path already cleaned up the tap, so the release fix repairs that path too. * test: prove a load balancer keeps its name, its address and its traffic when its host dies Takes away the node running an internet-facing ALB and asserts the four things a customer of one actually holds: the name keeps resolving, the address behind it does not move, requests through the name are answered again without anybody being called, and the backends are healthy again from the recovered load balancer's point of view. The backends are placed around the load balancer rather than beside it. A load balancer lands wherever a node answers its queue group, so its node is discovered and the backends are relaunched until both are elsewhere; one on the node about to be taken away would be recovered too, and the run could not then tell an ALB that came back serving from one that came back with nothing to serve. DNS is asked of a node's own northstar over the wire, and the HTTP probes resolve through the same server, so the claim under test is what this cluster answers for a name rather than what the machine running the tests was configured to forward to. A watch samples every surviving node every five seconds for the whole run, because an answer that went missing or pointed elsewhere for one reconcile pass is the outage, and no single sample reaches it. The returning node then has to give up what it no longer owns: no second copy of the guest, and no tap still claiming the load balancer's logical port, since OVN binds a port to one chassis and a second claimant contends for it rather than splitting the traffic. The last assertion is that the name still serves with that node back in the cluster. * docs: name the new cell in the nightly's cell list * test: stop the disk-load guest from filling the console it is judged by The load generator announced every write, once a second, for as long as the test ran. The console is a ring buffer this suite reads back, so an hour of that pushes out everything explaining a failure, including the kernel's own account of the I/O error the test exists to produce. Announce twenty times, then keep writing quietly. * docs: say what the OCI guide has been proven on, and what a host failure does The guide read as a support statement for OCI rather than as an account of one tested path, promised seven steps and then gave eight, and said nothing about what happens to a guest or its public address when the node under it fails. State the evidence, state that the nightly guards the datapath and not the cloud, and give the failure behaviour as a table with the reason codes a customer will actually see. Also say plainly what the volume lease is: one running guest and a bounded, logged window, not a write barrier the object store enforces. Someone sizing a workload against this should read that before they read the table. * docs: say why the mount classifier refuses to retry a lease another node holds It reads as an omission — a lease held elsewhere is the most transient refusal there is. It is not. This classifier drives the restore path, where the holder is a node that is running the guest, so retrying would relaunch against a live owner. Recovery wants the opposite and reads the refusal itself. * feat: claim a recovered guest's public address in line, and verify it arrived A recovered instance was reported running before anything moved its public address off the node that died, so for up to fifteen seconds the records said recovered and the address answered nothing. The recovery path now runs the affinity pass itself, which is idempotent and is the same pass the ticker runs. Two things the pass could not do before. It abandoned every remaining address on the first API failure, though a throttle or a denial is per-object and the guests behind the others are just as dark. And it trusted the move's own response, which is OCI agreeing to the request rather than OCI having done it, so an accepted move that never landed read as success. It now re-reads what OCI delivers here and reports the difference. An address that will not arrive makes the guest a system-status failure with the time it went dark, the same shape AWS uses for a host-level fault. The guest passes every check it can be asked about itself, which is exactly why reporting it healthy was the worst available answer. * feat: give a recovery attempt one trace across all three of its phases The volumes opening, QEMU starting and its monitor answering already trace themselves, but nothing tied them to the attempt that asked for them: three orphan traces per recovery, no way to ask which phase a slow one spent its time in, and nothing naming the node the guest came from. A span over the attempt makes them its children, and the reconciler's own log lines carry its trace ID because the handler stamps it from the context. A failed attempt records the fault and the attempt count on the span, so the budget running out is readable as a sequence rather than as twelve unrelated warnings. * fix: finish a fence once its writer finally exits A kill that did not take effect was logged, counted and dropped. The export stayed registered and the lease stayed held, which is correct while the writer lives and wrong forever after it dies: a lease a healthy node keeps renewing never expires, so the volume could not start anywhere. The failure is now re-reported every 30 s until it resolves, and the teardown completes as soon as the writer is confirmed gone. Only a delivered kill is waited on — a refused one says nothing about what became of the writer. * test: stop judging a detached stop or terminate on a transient state Both tests read the instance state immediately after the ack, but the transition runs on its own goroutine, so the read races it in both directions: too early sees running, too late sees the final state. The race gate caught the first of those on a loaded runner. * fix: replicate the volume lease and dirty buckets across the cluster Both were created with the default single replica, so both lived on one node. That node holds the answer to "who may write this volume" and the marker every mount reads, and losing it takes volume leasing and mounting down cluster-wide. A partitioned node was fenced correctly and then no survivor could relaunch its guest: every attempt failed on "read dirty mark: no responders available", for ten minutes, until the budget ran out. The replica count is now the cluster's node count, as the gateway's own buckets already do. Existing buckets are raised in place rather than requiring a wipe. The e2e diagnostics that should have said this are fixed too. The fence and the surrender are logged by the storage service, not the daemon, so the assertions read the wrong journal; and Eventuallyf evaluates its message at the call site, so a ten-minute wait was explained by a log that stopped before it began.
1620 lines
75 KiB
YAML
1620 lines
75 KiB
YAML
name: E2E Nightly
|
|
|
|
# Triggered nightly by a Cloudflare Worker that dispatches this workflow once on
|
|
# `dev` and once on `main`, rather than by a `schedule:` here. GitHub registers a
|
|
# schedule only from the default branch, so a cron in this file can never run
|
|
# `main` and `dev` as separate nightlies, and it would always execute `main`'s
|
|
# copy of this file — leaving a fix to the workflow itself untestable until it
|
|
# had already shipped. A dispatch takes both the file and the source from its ref.
|
|
on:
|
|
workflow_dispatch:
|
|
inputs:
|
|
cells:
|
|
description: "Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection, 30 routed NAT on three nodes, 31 instance recovery on three nodes, 32 load balancer recovery on three nodes"
|
|
required: false
|
|
|
|
permissions:
|
|
contents: read
|
|
actions: read
|
|
|
|
# Every cell builds its cluster on the same five hypervisors, so two runs in
|
|
# flight at once compete for their disks and their libvirt guests. Queue the
|
|
# second rather than cancelling it: a dispatch is usually a verification run
|
|
# and discarding it loses the result someone is waiting on.
|
|
#
|
|
# The group is shared with e2e-suites.yml and e2e-ddil.yml, which draw on the
|
|
# same runner pools. e2e.yml is deliberately outside it: a single-cluster push
|
|
# run is cheap enough to overlap, and grouping it stalled every push behind a
|
|
# multi-hour nightly.
|
|
concurrency:
|
|
group: e2e-hypervisors
|
|
cancel-in-progress: false
|
|
|
|
env:
|
|
GIT_BRANCH: ${{ github.ref_name }}
|
|
UBUNTU_GOLD_IMAGE: /mnt/disk1/gold-images/spinifex-image-builder-ubuntu-26.04.qcow2
|
|
SPINIFEX_OTLP_ENDPOINT: ${{ vars.SPINIFEX_OTLP_ENDPOINT || 'http://192.168.1.13:8200' }}
|
|
SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }}
|
|
|
|
jobs:
|
|
select:
|
|
runs-on: ubuntu-26.04
|
|
outputs:
|
|
cells: ${{ steps.pick.outputs.cells }}
|
|
any: ${{ steps.pick.outputs.any }}
|
|
steps:
|
|
- uses: actions/checkout@v7
|
|
with:
|
|
sparse-checkout: |
|
|
.github/e2e-cells.json
|
|
.github/scripts/suite-sets.sh
|
|
sparse-checkout-cone-mode: false
|
|
- id: pick
|
|
env:
|
|
CELLS: ${{ github.event_name == 'workflow_dispatch' && inputs.cells || '' }}
|
|
SETUP_ALLOWANCE_MIN: 10
|
|
run: |
|
|
if [ -z "$CELLS" ]; then
|
|
sel=$(jq -c . .github/e2e-cells.json)
|
|
else
|
|
sel=$(jq -c --arg c "$CELLS" \
|
|
'[.[] | . as $e | select(($c | split(",")) | index($e.cell|tostring))]' \
|
|
.github/e2e-cells.json)
|
|
fi
|
|
echo "cells=$sel" >> "$GITHUB_OUTPUT"
|
|
[ "$(jq 'length' <<<"$sel")" -gt 0 ] && any=true || any=false
|
|
echo "any=$any" >> "$GITHUB_OUTPUT"
|
|
echo "selected: $(jq -c '[.[].cell]' <<<"$sel")"
|
|
|
|
# The inner timeout must always fire before the outer one. go test
|
|
# reports where it stopped and the suite's own failure; a job timeout
|
|
# reports only "The operation was canceled". cell-27 ran a 50m rds
|
|
# suite inside a 50m job and could never finish either way.
|
|
. .github/scripts/suite-sets.sh
|
|
# Cells with no suite are filtered in jq, not with an empty-field test:
|
|
# tab is IFS whitespace, so read collapses a run of them and an empty
|
|
# middle column silently shifts every later one left.
|
|
bad=0
|
|
while IFS=$'\t' read -r cell suite budget; do
|
|
want="$(e2e_suite_timeout "$suite")"; want="${want%m}"
|
|
if [ "$budget" -lt $((want + SETUP_ALLOWANCE_MIN)) ]; then
|
|
echo "::error::cell-$cell gives suite '$suite' ${want}m inside a ${budget}m job. Raise timeout_minutes in .github/e2e-cells.json to at least $((want + SETUP_ALLOWANCE_MIN)), or lower the suite budget in suite-sets.sh." >&2
|
|
bad=1
|
|
fi
|
|
done < <(jq -r '.[]|select((.suite//"")!="")|[.cell,.suite,(.timeout_minutes//50)]|@tsv' <<<"$sel")
|
|
|
|
# A permutation cell names no suite and runs the nightly subset for its
|
|
# topology, so its bound is the longest of those. max() and not sum():
|
|
# whichever suite hangs must panic on its own timeout first, and a sum
|
|
# would demand a budget the set has never come close to using.
|
|
while IFS=$'\t' read -r cell topo budget; do
|
|
case "$topo" in
|
|
*multi) set -- ${E2E_SUITES_NIGHTLY_MULTI} ;;
|
|
*) set -- ${E2E_SUITES_NIGHTLY_SINGLE} ;;
|
|
esac
|
|
want=0
|
|
for suite in "$@"; do
|
|
t="$(e2e_suite_timeout "$suite")"; t="${t%m}"
|
|
[ "$t" -gt "$want" ] && want="$t"
|
|
done
|
|
if [ "$budget" -lt $((want + SETUP_ALLOWANCE_MIN)) ]; then
|
|
echo "::error::cell-$cell runs the $topo nightly subset, whose longest suite asks ${want}m, inside a ${budget}m job. Raise timeout_minutes in .github/e2e-cells.json to at least $((want + SETUP_ALLOWANCE_MIN))." >&2
|
|
bad=1
|
|
fi
|
|
done < <(jq -r '.[]|select((.suite//"")=="")|[.cell,.topo,(.timeout_minutes//50)]|@tsv' <<<"$sel")
|
|
[ "$bad" -eq 0 ]
|
|
|
|
matrix:
|
|
name: "cell-${{ matrix.cell }} (${{ matrix.topo }}/${{ matrix.net }}/${{
|
|
matrix.host }}${{ matrix.extra && format('/{0}', matrix.extra) || '' }})"
|
|
needs: select
|
|
if: needs.select.outputs.any == 'true'
|
|
strategy:
|
|
fail-fast: false
|
|
max-parallel: 3
|
|
matrix:
|
|
include: ${{ fromJSON(needs.select.outputs.cells) }}
|
|
runs-on: [self-hosted, "${{ matrix.runner }}"]
|
|
timeout-minutes: ${{ matrix.timeout_minutes || 50 }}
|
|
env:
|
|
TOFU_LOG: /tmp/e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}.log
|
|
ARTIFACT_DIR: /tmp/spinifex-e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}
|
|
SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=${{ matrix.cell }},ci.matrix=${{ matrix.topo }}/${{ matrix.net }}/${{ matrix.host }}
|
|
|
|
steps:
|
|
- name: Clean Workspace
|
|
run: |
|
|
sudo umount -R -l /var/cache/spinifex-iso-builder/rootfs 2>/dev/null || true
|
|
if [ ! -c /dev/null ]; then sudo rm -f /dev/null; sudo mknod -m 666 /dev/null c 1 3; fi
|
|
sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
|
|
|
|
- uses: actions/create-github-app-token@v3
|
|
id: app-token
|
|
with:
|
|
client-id: ${{ secrets.APP_ID }}
|
|
private-key: ${{ secrets.APP_PRIVATE_KEY }}
|
|
repositories: mulga
|
|
|
|
- uses: actions/checkout@v7
|
|
with:
|
|
repository: mulgadc/mulga
|
|
token: ${{ steps.app-token.outputs.token }}
|
|
sparse-checkout: scripts/tofu-cluster
|
|
path: mulga
|
|
|
|
- uses: actions/checkout@v7
|
|
with:
|
|
path: spinifex
|
|
|
|
# Twelve cells share one run id and every cell names its hosts
|
|
# node1/node2/node3, so ci.cell above separates them and ci.job_id here
|
|
# links each cell's telemetry to the job log that produced it. Runs
|
|
# before Bootstrap bakes SPINIFEX_OTEL_ATTRS into the VM's telemetry.env.
|
|
- name: Resolve CI job id
|
|
uses: ./spinifex/.github/actions/otel-job-id
|
|
with:
|
|
job-name: "cell-${{ matrix.cell }} (${{ matrix.topo }}/${{ matrix.net }}/${{ matrix.host }}${{ matrix.extra && format('/{0}', matrix.extra) || '' }})"
|
|
|
|
- name: Build Install Artifacts
|
|
working-directory: mulga/scripts/tofu-cluster
|
|
env:
|
|
MATRIX_SUITE: ${{ matrix.suite }}
|
|
run: |
|
|
# The script's default set is the nightly baseline. A service cell
|
|
# runs one suite outside it, and asks for that binary by name.
|
|
./build-install-artifacts.sh "${GIT_BRANCH}" \
|
|
"${RUNNER_TEMP}/spinifex-distro.tar.gz" \
|
|
"${RUNNER_TEMP}/setup.sh" \
|
|
"${RUNNER_TEMP}/spinifex-tests.tar" \
|
|
--extra-suites "${MATRIX_SUITE}"
|
|
|
|
# Every runner agent runs on one host, while their nodes sit on several, so
|
|
# this samples the runner's host and not the node's hypervisor. Kept for
|
|
# runner-side contention; the node's own capacity is sampled after.
|
|
- name: Sample runner-host load (before)
|
|
if: always()
|
|
run: |
|
|
mkdir -p "${ARTIFACT_DIR}"
|
|
{
|
|
echo "=== before: $(date -u +%Y-%m-%dT%H:%M:%SZ) runner=${RUNNER_NAME} host=$(hostname) ==="
|
|
echo "loadavg: $(cat /proc/loadavg)"
|
|
grep -E '^(MemTotal|MemAvailable|SwapFree):' /proc/meminfo || true
|
|
echo "cpu(steal is field 8): $(grep '^cpu ' /proc/stat || true)"
|
|
echo "running qemu: $(pgrep -c qemu-system-x86 || echo 0)"
|
|
} >> "${ARTIFACT_DIR}/hypervisor-load.txt"
|
|
|
|
- name: Provision VMs
|
|
working-directory: mulga/scripts/tofu-cluster
|
|
env:
|
|
MATRIX_HOST: ${{ matrix.host }}
|
|
run: |
|
|
export TF_VAR_git_branch="${GIT_BRANCH}"
|
|
|
|
# Match the family, not a pinned version: an exact match silently stops
|
|
# overriding the image when the matrix bumps release, dropping the cell
|
|
# back onto the debian gold.
|
|
TOFU_VAR_ARGS=""
|
|
case "$MATRIX_HOST" in
|
|
ubuntu-*) TOFU_VAR_ARGS="-var=host_image_override=${UBUNTU_GOLD_IMAGE}" ;;
|
|
esac
|
|
|
|
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
|
|
tofu init -input=false >> "$TOFU_LOG" 2>&1
|
|
tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
|
|
tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars ${TOFU_VAR_ARGS} -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
|
|
./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
|
|
# shellcheck disable=SC2086
|
|
retry_run "tofu apply" "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars ${TOFU_VAR_ARGS} -auto-approve -input=false
|
|
|
|
- name: Bootstrap via Binary Install
|
|
working-directory: mulga/scripts/tofu-cluster
|
|
env:
|
|
MATRIX_NET: ${{ matrix.net }}
|
|
run: |
|
|
if [ "$MATRIX_NET" = "nat" ]; then
|
|
# Routed-NAT mode with a static public pool: exercises both the
|
|
# Tier-1 transit datapath and the Tier-2 EIP ingress lane.
|
|
export SPX_EXTERNAL_MODE=nat
|
|
export SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}"
|
|
elif [ "$MATRIX_NET" = "dhcp" ]; then
|
|
unset SPX_EXTERNAL_POOL
|
|
export SPX_EXTERNAL_SOURCE=dhcp
|
|
else
|
|
export SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}"
|
|
fi
|
|
|
|
./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
|
|
"${RUNNER_TEMP}/setup.sh" \
|
|
"${RUNNER_TEMP}/spinifex-distro.tar.gz" \
|
|
--test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
|
|
|
|
- name: Run Single Node E2E
|
|
if: matrix.topo == 'single' && matrix.extra == '' && matrix.suite == ''
|
|
working-directory: mulga/scripts/tofu-cluster
|
|
run: |
|
|
set -euo pipefail
|
|
# Nightly runs a documented subset of the single-node set; the assert
|
|
# fails the cell if that subset ever names a suite this topology
|
|
# cannot fully exercise.
|
|
. "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
|
|
e2e_suites_assert_subset "e2e-nightly single" "${E2E_SUITES_NIGHTLY_SINGLE}" "${E2E_SUITES_SINGLE}"
|
|
|
|
INVENTORY=$(tofu output -json inventory)
|
|
WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
|
|
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
|
|
SSH_KEY="${SSH_KEY/#\~/$HOME}"
|
|
ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
|
|
"set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; FAIL=0; \
|
|
for SUITE in ${E2E_SUITES_NIGHTLY_SINGLE}; do \
|
|
echo \"::group::suite \$SUITE\"; \
|
|
date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
|
|
GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
|
|
ARTIFACT_DIR='${ARTIFACT_DIR}' \
|
|
./\$SUITE.test -test.v -test.timeout=30m \
|
|
2>&1 | tee '${ARTIFACT_DIR}/test-'\$SUITE'.log'; \
|
|
rc=\${PIPESTATUS[0]}; echo \"::endgroup::\"; \
|
|
if [ \$rc -ne 0 ]; then echo \"\$SUITE: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; FAIL=1; fi; \
|
|
done; exit \$FAIL"
|
|
|
|
- name: Run NAT Uplink E2E
|
|
if: matrix.extra == 'nat-single'
|
|
working-directory: mulga/scripts/tofu-cluster
|
|
# Skips cleanly (exit 0) when the tested branch predates the natuplink
|
|
# suite so scheduled runs against dev stay green until the merge lands.
|
|
run: |
|
|
INVENTORY=$(tofu output -json inventory)
|
|
WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
|
|
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
|
|
SSH_KEY="${SSH_KEY/#\~/$HOME}"
|
|
ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
|
|
"set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; \
|
|
if [ ! -f natuplink.test ]; then echo 'natuplink.test not in tests tar (branch predates suite) — skipping'; exit 0; fi; \
|
|
echo '::group::suite natuplink'; \
|
|
date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-natuplink.start'; \
|
|
GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
|
|
ARTIFACT_DIR='${ARTIFACT_DIR}' \
|
|
./natuplink.test -test.v -test.timeout=30m \
|
|
2>&1 | tee '${ARTIFACT_DIR}/test-natuplink.log'; \
|
|
rc=\${PIPESTATUS[0]}; echo '::endgroup::'; \
|
|
if [ \$rc -ne 0 ]; then echo 'natuplink: FAILED' >> '${ARTIFACT_DIR}/suite-status.txt'; fi; exit \$rc"
|
|
|
|
# natuplink proves a guest with a public IP reaches the internet. The two
|
|
# surfaces it never touches are a private subnet's isolation and its NAT
|
|
# gateway, and TestVPCEgressPaths is the authority on both — it skipped in
|
|
# nat mode until the fixture learned that routed NAT carries EIPs too.
|
|
- name: Run VPC Egress Paths in NAT mode
|
|
if: matrix.extra == 'nat-single'
|
|
working-directory: mulga/scripts/tofu-cluster
|
|
run: |
|
|
INVENTORY=$(tofu output -json inventory)
|
|
WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
|
|
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
|
|
SSH_KEY="${SSH_KEY/#\~/$HOME}"
|
|
ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
|
|
"set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; \
|
|
echo '::group::suite single/TestVPCEgressPaths'; \
|
|
date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-vpcegress-nat.start'; \
|
|
GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
|
|
ARTIFACT_DIR='${ARTIFACT_DIR}' \
|
|
./single.test -test.v -test.timeout=30m -test.run '^TestVPCEgressPaths\$' \
|
|
2>&1 | tee '${ARTIFACT_DIR}/test-vpcegress-nat.log'; \
|
|
rc=\${PIPESTATUS[0]}; echo '::endgroup::'; \
|
|
if [ \$rc -ne 0 ]; then echo 'vpcegress-nat: FAILED' >> '${ARTIFACT_DIR}/suite-status.txt'; fi; exit \$rc"
|
|
|
|
- name: Run Multi Node E2E
|
|
if: matrix.topo == 'real-multi' && matrix.suite == '' && matrix.extra == ''
|
|
working-directory: mulga/scripts/tofu-cluster
|
|
run: |
|
|
set -euo pipefail
|
|
# See the single-node step: same subset guard, multi-node set.
|
|
. "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
|
|
e2e_suites_assert_subset "e2e-nightly multi" "${E2E_SUITES_NIGHTLY_MULTI}" "${E2E_SUITES_MULTI}"
|
|
|
|
INVENTORY=$(tofu output -json inventory)
|
|
NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
|
|
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
|
|
SSH_KEY="${SSH_KEY/#\~/$HOME}"
|
|
|
|
NODES_CSV=""
|
|
NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
|
|
for i in $(seq 0 $((NODE_COUNT - 1))); do
|
|
ip=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
|
|
NODES_CSV="${NODES_CSV:+$NODES_CSV,}$ip"
|
|
done
|
|
|
|
ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
|
|
"set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; FAIL=0; \
|
|
for SUITE in ${E2E_SUITES_NIGHTLY_MULTI}; do \
|
|
echo \"::group::suite \$SUITE\"; \
|
|
date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
|
|
GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
|
|
SPINIFEX_NODES='${NODES_CSV}' SPINIFEX_NODE_IPS='${NODES_CSV}' \
|
|
SPINIFEX_SSH_USER=tf-user SPINIFEX_SSH_KEY=\$HOME/.ssh/tf-user-ap-southeast-2 \
|
|
ARTIFACT_DIR='${ARTIFACT_DIR}' \
|
|
./\$SUITE.test -test.v -test.timeout=30m \
|
|
2>&1 | tee '${ARTIFACT_DIR}/test-'\$SUITE'.log'; \
|
|
rc=\${PIPESTATUS[0]}; echo \"::endgroup::\"; \
|
|
if [ \$rc -ne 0 ]; then echo \"\$SUITE: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; FAIL=1; fi; \
|
|
done; exit \$FAIL"
|
|
|
|
# The same suite as the nat-single cell, run against three nodes: there it
|
|
# proves routed NAT works at all, here that each node delivers its own
|
|
# guests' public IPs and no others — which one node cannot show.
|
|
- name: Run NAT Uplink Multi E2E
|
|
if: matrix.extra == 'nat-multi'
|
|
working-directory: mulga/scripts/tofu-cluster
|
|
run: |
|
|
set -euo pipefail
|
|
INVENTORY=$(tofu output -json inventory)
|
|
NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
|
|
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
|
|
SSH_KEY="${SSH_KEY/#\~/$HOME}"
|
|
|
|
NODES_CSV=""
|
|
NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
|
|
for i in $(seq 0 $((NODE_COUNT - 1))); do
|
|
ip=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
|
|
NODES_CSV="${NODES_CSV:+$NODES_CSV,}$ip"
|
|
done
|
|
|
|
ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
|
|
"set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; \
|
|
if [ ! -f natuplink.test ]; then echo 'natuplink.test not in tests tar (branch predates suite) — skipping'; exit 0; fi; \
|
|
echo '::group::suite natuplink'; \
|
|
date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-natuplink.start'; \
|
|
GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
|
|
SPINIFEX_NODES='${NODES_CSV}' SPINIFEX_NODE_IPS='${NODES_CSV}' \
|
|
SPINIFEX_SSH_USER=tf-user SPINIFEX_SSH_KEY=\$HOME/.ssh/tf-user-ap-southeast-2 \
|
|
ARTIFACT_DIR='${ARTIFACT_DIR}' \
|
|
./natuplink.test -test.v -test.timeout=30m \
|
|
2>&1 | tee '${ARTIFACT_DIR}/test-natuplink.log'; \
|
|
rc=\${PIPESTATUS[0]}; echo '::endgroup::'; \
|
|
if [ \$rc -ne 0 ]; then echo 'natuplink: FAILED' >> '${ARTIFACT_DIR}/suite-status.txt'; fi; exit \$rc"
|
|
|
|
# The dedicated suite cells only. Resolved once here rather than in each
|
|
# step below, which would repeat the same three lines five times.
|
|
- name: Resolve VM SSH details
|
|
id: vm
|
|
if: matrix.suite != '' && matrix.topo == 'single'
|
|
working-directory: mulga/scripts/tofu-cluster
|
|
run: |
|
|
set -euo pipefail
|
|
INVENTORY=$(tofu output -json inventory)
|
|
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
|
|
echo "wan_ip=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')" >> "$GITHUB_OUTPUT"
|
|
echo "ssh_key=${SSH_KEY/#\~/$HOME}" >> "$GITHUB_OUTPUT"
|
|
|
|
# The eks suite's CreateCluster boots a control-plane VM from the eks-node
|
|
# AMI, so that AMI must be in the VM's image store before the suite runs.
|
|
# build-system-image.sh customizes with libguestfs (an isolated userspace
|
|
# appliance touching no host block device), so it is safe on the shared
|
|
# runner. The build needs source + network (runner only) and the import
|
|
# needs predastore/NATS (VM only), hence the split.
|
|
- name: Build eks-node AMI (libguestfs, runner)
|
|
if: matrix.suite == 'eks'
|
|
working-directory: spinifex
|
|
run: |
|
|
set -euo pipefail
|
|
# direct backend = no libvirt. Set here rather than in the script:
|
|
# it has to be in the environment the build itself runs in.
|
|
export LIBGUESTFS_BACKEND=direct
|
|
"${GITHUB_WORKSPACE}/spinifex/.github/scripts/ensure-libguestfs.sh"
|
|
make build-eks-node-image
|
|
|
|
- name: Import eks-node AMI on VM
|
|
if: matrix.suite == 'eks'
|
|
run: |
|
|
set -euo pipefail
|
|
eval "$(grep -E '^(ALPINE_VERSION|AMI_NAME|SYSTEM_TAG)=' \
|
|
spinifex/scripts/images/eks-node/manifest.conf)"
|
|
SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
|
|
VM=tf-user@${{ steps.vm.outputs.wan_ip }}
|
|
scp "${SSH_OPTS[@]}" \
|
|
/tmp/eks-node-image-build/eks-node-alpine.raw \
|
|
"$VM":/tmp/eks-node-alpine.raw
|
|
ssh "${SSH_OPTS[@]}" "$VM" \
|
|
"spx admin images import --file /tmp/eks-node-alpine.raw \
|
|
--distro alpine --version ${ALPINE_VERSION} --arch x86_64 \
|
|
--boot-mode bios --ami-name ${AMI_NAME} --tag ${SYSTEM_TAG}" >/dev/null 2>&1
|
|
|
|
# The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full
|
|
# `aws eks get-token` -> kubectl -> apiserver -> token-webhook -> gateway
|
|
# broker -> STS verify path. The e2e binaries run ON the VM, so kubectl
|
|
# must be on the VM's PATH or harness.NewKubectl skips the subtest and the
|
|
# cell reports green having never exercised the data plane. Pinned to the
|
|
# k3s server version (setup.sh K3S_VERSION) and checksum-verified.
|
|
- name: Install kubectl on VM
|
|
if: matrix.suite == 'eks'
|
|
run: |
|
|
set -euo pipefail
|
|
SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
|
|
ssh "${SSH_OPTS[@]}" tf-user@${{ steps.vm.outputs.wan_ip }} 'bash -s' <<'EOF'
|
|
set -euo pipefail
|
|
KVER=v1.32.5
|
|
case "$(uname -m)" in
|
|
x86_64) ARCH=amd64 ;;
|
|
aarch64) ARCH=arm64 ;;
|
|
*) echo "unsupported arch $(uname -m)" >&2; exit 1 ;;
|
|
esac
|
|
cd /tmp
|
|
curl -fsSLO "https://dl.k8s.io/release/${KVER}/bin/linux/${ARCH}/kubectl"
|
|
curl -fsSLO "https://dl.k8s.io/release/${KVER}/bin/linux/${ARCH}/kubectl.sha256"
|
|
echo "$(cat kubectl.sha256) kubectl" | sha256sum -c -
|
|
sudo install -m 0755 kubectl /usr/local/bin/kubectl
|
|
kubectl version --client
|
|
EOF
|
|
|
|
# RegisterContainerInstance boots the container instance from the ecs-node
|
|
# AMI, and resolveECSNodeAMI matches on the managed-by tag rather than the
|
|
# name, so the tag is not optional. The image directory is named for
|
|
# IMAGE_NAME in the manifest (ecs-agent), not for the make target.
|
|
- name: Build ecs-node AMI (libguestfs, runner)
|
|
if: matrix.suite == 'ecs'
|
|
working-directory: spinifex
|
|
run: |
|
|
set -euo pipefail
|
|
# direct backend = no libvirt. Set here rather than in the script:
|
|
# it has to be in the environment the build itself runs in.
|
|
export LIBGUESTFS_BACKEND=direct
|
|
"${GITHUB_WORKSPACE}/spinifex/.github/scripts/ensure-libguestfs.sh"
|
|
make build-ecs-node-image
|
|
|
|
- name: Import ecs-node AMI on VM
|
|
if: matrix.suite == 'ecs'
|
|
run: |
|
|
set -euo pipefail
|
|
eval "$(grep -E '^(ALPINE_VERSION|AMI_NAME|SYSTEM_TAG)=' \
|
|
spinifex/scripts/images/ecs-agent/manifest.conf)"
|
|
SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
|
|
VM=tf-user@${{ steps.vm.outputs.wan_ip }}
|
|
scp "${SSH_OPTS[@]}" \
|
|
/tmp/ecs-agent-image-build/ecs-agent-alpine.raw \
|
|
"$VM":/tmp/ecs-agent-alpine.raw
|
|
ssh "${SSH_OPTS[@]}" "$VM" \
|
|
"spx admin images import --file /tmp/ecs-agent-alpine.raw \
|
|
--distro alpine --version ${ALPINE_VERSION} --arch x86_64 \
|
|
--boot-mode bios --ami-name ${AMI_NAME} --tag ${SYSTEM_TAG}" >/dev/null 2>&1
|
|
|
|
# CreateDBInstance boots the DB VM from an RDS AMI, so both engines' images
|
|
# must be in the store before the suite runs. The suite's psql and mariadb
|
|
# clients need no step here: they run inside a client VM that installs them
|
|
# from its own user-data, because a private-only endpoint is unreachable
|
|
# from the runner and from the cluster host alike.
|
|
- name: Build RDS AMIs (libguestfs, runner)
|
|
if: matrix.suite == 'rds'
|
|
working-directory: spinifex
|
|
run: |
|
|
set -euo pipefail
|
|
# direct backend = no libvirt. Set here rather than in the script:
|
|
# it has to be in the environment the build itself runs in.
|
|
export LIBGUESTFS_BACKEND=direct
|
|
"${GITHUB_WORKSPACE}/spinifex/.github/scripts/ensure-libguestfs.sh"
|
|
for IMAGE in rds-postgres rds-mariadb; do
|
|
make "build-${IMAGE}-image"
|
|
done
|
|
|
|
- name: Import RDS AMIs on VM
|
|
if: matrix.suite == 'rds'
|
|
run: |
|
|
set -euo pipefail
|
|
SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
|
|
VM=tf-user@${{ steps.vm.outputs.wan_ip }}
|
|
for IMAGE in rds-postgres rds-mariadb; do
|
|
eval "$(grep -E '^(ALPINE_VERSION|AMI_NAME|SYSTEM_TAG)=' \
|
|
"spinifex/scripts/images/${IMAGE}/manifest.conf")"
|
|
# SYSTEM_TAG carries four tags, not one: managed-by marks it a system
|
|
# image, engine/engine-version are what an EngineVersion request
|
|
# resolves against, and the contract tag gates the data-volume format
|
|
# grant. --tag is repeatable, so expand rather than pass the whole
|
|
# string as a single value.
|
|
TAG_ARGS=""
|
|
for tag in ${SYSTEM_TAG}; do TAG_ARGS="${TAG_ARGS} --tag ${tag}"; done
|
|
scp "${SSH_OPTS[@]}" \
|
|
"/tmp/${IMAGE}-image-build/${IMAGE}-alpine.raw" \
|
|
"$VM":"/tmp/${IMAGE}-alpine.raw"
|
|
ssh "${SSH_OPTS[@]}" "$VM" \
|
|
"spx admin images import --file /tmp/${IMAGE}-alpine.raw \
|
|
--distro alpine --version ${ALPINE_VERSION} --arch x86_64 \
|
|
--boot-mode bios --ami-name ${AMI_NAME}${TAG_ARGS}" >/dev/null 2>&1
|
|
done
|
|
|
|
- name: Run Service Suite E2E
|
|
if: matrix.suite != '' && matrix.topo == 'single'
|
|
env:
|
|
SUITE: ${{ matrix.suite }}
|
|
run: |
|
|
set -euo pipefail
|
|
# Same guard as the permutation steps, against the full single-node
|
|
# set: a cell naming a suite this topology cannot fully exercise fails
|
|
# here rather than skipping half its assertions.
|
|
. "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
|
|
e2e_suites_assert_subset "e2e-nightly cell-${{ matrix.cell }}" "${SUITE}" "${E2E_SUITES_SINGLE}"
|
|
TMO="$(e2e_suite_timeout "${SUITE}")"
|
|
|
|
ssh -tt -o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}" \
|
|
tf-user@${{ steps.vm.outputs.wan_ip }} \
|
|
"set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; \
|
|
echo \"::group::suite ${SUITE}\"; \
|
|
date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-${SUITE}.start'; \
|
|
GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
|
|
ARTIFACT_DIR='${ARTIFACT_DIR}' \
|
|
./${SUITE}.test -test.v -test.timeout=${TMO} \
|
|
2>&1 | tee '${ARTIFACT_DIR}/test-${SUITE}.log'; \
|
|
rc=\${PIPESTATUS[0]}; echo \"::endgroup::\"; \
|
|
if [ \$rc -ne 0 ]; then echo \"${SUITE}: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; fi; \
|
|
exit \$rc"
|
|
|
|
# The multi-node twin of the step above. Separate rather than merged
|
|
# because the suite needs every node's address to reach the whole cluster,
|
|
# which the single-node path has no way to supply.
|
|
- name: Run Multi Suite E2E
|
|
if: matrix.suite != '' && matrix.topo == 'real-multi'
|
|
working-directory: mulga/scripts/tofu-cluster
|
|
env:
|
|
SUITE: ${{ matrix.suite }}
|
|
run: |
|
|
set -euo pipefail
|
|
. "${GITHUB_WORKSPACE}/spinifex/.github/scripts/suite-sets.sh"
|
|
e2e_suites_assert_subset "e2e-nightly cell-${{ matrix.cell }}" "${SUITE}" "${E2E_SUITES_MULTI}"
|
|
TMO="$(e2e_suite_timeout "${SUITE}")"
|
|
|
|
INVENTORY=$(tofu output -json inventory)
|
|
NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
|
|
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
|
|
SSH_KEY="${SSH_KEY/#\~/$HOME}"
|
|
|
|
NODES_CSV=""
|
|
NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
|
|
for i in $(seq 0 $((NODE_COUNT - 1))); do
|
|
ip=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
|
|
NODES_CSV="${NODES_CSV:+$NODES_CSV,}$ip"
|
|
done
|
|
|
|
ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
|
|
"set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; \
|
|
echo \"::group::suite ${SUITE}\"; \
|
|
date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-${SUITE}.start'; \
|
|
GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
|
|
SPINIFEX_NODES='${NODES_CSV}' SPINIFEX_NODE_IPS='${NODES_CSV}' \
|
|
SPINIFEX_SSH_USER=tf-user SPINIFEX_SSH_KEY=\$HOME/.ssh/tf-user-ap-southeast-2 \
|
|
ARTIFACT_DIR='${ARTIFACT_DIR}' \
|
|
./${SUITE}.test -test.v -test.timeout=${TMO} \
|
|
2>&1 | tee '${ARTIFACT_DIR}/test-${SUITE}.log'; \
|
|
rc=\${PIPESTATUS[0]}; echo \"::endgroup::\"; \
|
|
if [ \$rc -ne 0 ]; then echo \"${SUITE}: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; fi; \
|
|
exit \$rc"
|
|
|
|
- name: Import RDS PostgreSQL AMI
|
|
if: matrix.extra == 'tofu-examples'
|
|
working-directory: mulga/scripts/tofu-cluster
|
|
run: |
|
|
set -euo pipefail
|
|
INVENTORY=$(tofu output -json inventory)
|
|
WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
|
|
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
|
|
SSH_KEY="${SSH_KEY/#\~/$HOME}"
|
|
ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
|
|
"spx admin images import --name spinifex-rds-postgres"
|
|
|
|
- name: Run Tofu Workbooks
|
|
if: matrix.extra == 'tofu-examples'
|
|
working-directory: mulga/scripts/tofu-cluster
|
|
# The driver emits go test -v markers per workbook; teeing to
|
|
# test-tofu.log feeds the same go-junit-report -> e2e-analyze -> upload
|
|
# chain the Go suites use, so cell 17 produces an RCA bundle too.
|
|
run: |
|
|
set +e
|
|
mkdir -p "${ARTIFACT_DIR}"
|
|
INVENTORY=$(tofu output -json inventory)
|
|
WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
|
|
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
|
|
SSH_KEY="${SSH_KEY/#\~/$HOME}"
|
|
date -u +%Y-%m-%dT%H:%M:%SZ > "${ARTIFACT_DIR}/test-tofu.start"
|
|
ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
|
|
"cd ~/e2e-tests && ./run-tofu-examples-e2e.sh" \
|
|
2>&1 | tee "${ARTIFACT_DIR}/test-tofu.log"
|
|
rc=${PIPESTATUS[0]}
|
|
if [ $rc -ne 0 ]; then echo "tofu: FAILED" >> "${ARTIFACT_DIR}/suite-status.txt"; fi
|
|
exit $rc
|
|
|
|
- name: Run Reboot Resilience E2E
|
|
if: matrix.extra == 'reboot-single'
|
|
working-directory: mulga/scripts/tofu-cluster
|
|
# reboot.test runs from the runner (NOT via ssh-and-run) so it survives the
|
|
# systemctl-reboot-induced ssh disconnect mid-run.
|
|
run: |
|
|
set +e
|
|
mkdir -p "${ARTIFACT_DIR}"
|
|
INVENTORY=$(tofu output -json inventory)
|
|
WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
|
|
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
|
|
SSH_KEY="${SSH_KEY/#\~/$HOME}"
|
|
|
|
REBOOT_DIR="${RUNNER_TEMP}/reboot-e2e"
|
|
mkdir -p "$REBOOT_DIR"
|
|
tar xf "${RUNNER_TEMP}/spinifex-tests.tar" -C "$REBOOT_DIR"
|
|
chmod +x "$REBOOT_DIR/_bin/reboot.test"
|
|
|
|
SSH_OPTS=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i "$SSH_KEY")
|
|
CREDS=$(ssh "${SSH_OPTS[@]}" tf-user@${WAN_IP} 'cat ~/.aws/credentials')
|
|
AWS_AK=$(echo "$CREDS" | awk -F '= *' '/^aws_access_key_id/ {print $2; exit}')
|
|
AWS_SK=$(echo "$CREDS" | awk -F '= *' '/^aws_secret_access_key/ {print $2; exit}')
|
|
echo "::add-mask::${AWS_AK}"; echo "::add-mask::${AWS_SK}"
|
|
|
|
date -u +%Y-%m-%dT%H:%M:%SZ > "${ARTIFACT_DIR}/test-reboot.start"
|
|
GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
|
|
SPINIFEX_WAN_IP="${WAN_IP}" SPINIFEX_SSH_KEY="${SSH_KEY}" \
|
|
SPINIFEX_AWS_ENDPOINT="https://${WAN_IP}:9999" \
|
|
SPINIFEX_AWS_ACCESS_KEY_ID="${AWS_AK}" SPINIFEX_AWS_SECRET_ACCESS_KEY="${AWS_SK}" \
|
|
ARTIFACT_DIR="${ARTIFACT_DIR}" \
|
|
"$REBOOT_DIR/_bin/reboot.test" -test.v -test.timeout=30m \
|
|
2>&1 | tee "${ARTIFACT_DIR}/test-reboot.log"
|
|
rc=${PIPESTATUS[0]}
|
|
if [ $rc -ne 0 ]; then echo "reboot: FAILED" >> "${ARTIFACT_DIR}/suite-status.txt"; fi
|
|
exit $rc
|
|
|
|
- name: Sample runner-host load (after)
|
|
if: always()
|
|
run: |
|
|
mkdir -p "${ARTIFACT_DIR}"
|
|
{
|
|
echo "=== after: $(date -u +%Y-%m-%dT%H:%M:%SZ) runner=${RUNNER_NAME} host=$(hostname) ==="
|
|
echo "loadavg: $(cat /proc/loadavg)"
|
|
grep -E '^(MemTotal|MemAvailable|SwapFree):' /proc/meminfo || true
|
|
echo "cpu(steal is field 8): $(grep '^cpu ' /proc/stat || true)"
|
|
echo "running qemu: $(pgrep -c qemu-system-x86 || echo 0)"
|
|
} >> "${ARTIFACT_DIR}/hypervisor-load.txt"
|
|
|
|
# The node, not the runner's host, is what the guests are scheduled on, so
|
|
# its load is the figure that explains a starved guest. Never fails the
|
|
# cell: a node already gone is the case this is trying to describe.
|
|
- name: Sample node load (after)
|
|
if: always()
|
|
working-directory: mulga/scripts/tofu-cluster
|
|
run: |
|
|
set +e
|
|
mkdir -p "${ARTIFACT_DIR}"
|
|
INVENTORY=$(tofu output -json inventory 2>/dev/null)
|
|
NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip' 2>/dev/null)
|
|
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path' 2>/dev/null)
|
|
SSH_KEY="${SSH_KEY/#\~/$HOME}"
|
|
[ -n "$NODE1_IP" ] && [ "$NODE1_IP" != "null" ] || exit 0
|
|
[ -n "$SSH_KEY" ] && [ "$SSH_KEY" != "null" ] || exit 0
|
|
{
|
|
echo "=== node after: $(date -u +%Y-%m-%dT%H:%M:%SZ) ip=${NODE1_IP} ==="
|
|
ssh -o StrictHostKeyChecking=no -o BatchMode=yes -o ConnectTimeout=10 \
|
|
-i "$SSH_KEY" "tf-user@${NODE1_IP}" \
|
|
'echo "host: $(hostname)"; echo "loadavg: $(cat /proc/loadavg)";
|
|
grep -E "^(MemTotal|MemAvailable|SwapFree):" /proc/meminfo;
|
|
echo "cpu(steal is field 8): $(grep "^cpu " /proc/stat)";
|
|
echo "running qemu: $(pgrep -c qemu-system-x86 || echo 0)"' 2>&1
|
|
} >> "${ARTIFACT_DIR}/node-load.txt"
|
|
exit 0
|
|
|
|
- name: Collect e2e artifacts
|
|
if: always()
|
|
working-directory: mulga/scripts/tofu-cluster
|
|
run: |
|
|
set +e
|
|
mkdir -p "${ARTIFACT_DIR}"
|
|
INVENTORY=$(tofu output -json inventory 2>/dev/null) || { echo "no inventory; skipping"; exit 0; }
|
|
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path'); SSH_KEY="${SSH_KEY/#\~/$HOME}"
|
|
NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
|
|
NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
|
|
SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY")
|
|
|
|
ssh "${SSH_OPTS[@]}" tf-user@${NODE1_IP} "tar -C '${ARTIFACT_DIR}' -cf - . 2>/dev/null || true" \
|
|
| tar -C "${ARTIFACT_DIR}" -xf - 2>/dev/null
|
|
|
|
for i in $(seq 0 $((NODE_COUNT - 1))); do
|
|
IP=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip"); idx=$((i + 1))
|
|
mkdir -p "${ARTIFACT_DIR}/vm-console-node${idx}"
|
|
ssh "${SSH_OPTS[@]}" tf-user@${IP} 'sudo sh -c "cd /run/spinifex 2>/dev/null && tar -cf - console-*.log 2>/dev/null"' \
|
|
| tar -C "${ARTIFACT_DIR}/vm-console-node${idx}" -xf - 2>/dev/null
|
|
ssh "${SSH_OPTS[@]}" tf-user@${IP} 'sudo journalctl -u "spinifex-*" --no-pager --output=short-iso --since "60 min ago"' \
|
|
> "${ARTIFACT_DIR}/spinifex-journal-node${idx}.log" 2>/dev/null
|
|
ssh "${SSH_OPTS[@]}" tf-user@${IP} 'sudo journalctl -u openvswitch-ipsec.service --no-pager --output=short-iso --since "60 min ago"' \
|
|
> "${ARTIFACT_DIR}/openvswitch-ipsec-node${idx}.log" 2>/dev/null
|
|
# The unit filter above drops kernel and OOM messages, which are
|
|
# exactly what explains one node behaving unlike its siblings.
|
|
# Gzipped on the far side; the whole journal is far larger.
|
|
ssh "${SSH_OPTS[@]}" tf-user@${IP} 'sudo journalctl --no-pager --output=short-iso --since "90 min ago" | gzip -6' \
|
|
> "${ARTIFACT_DIR}/full-journal-node${idx}.log.gz" 2>/dev/null
|
|
# ovn-controller and ovs-vswitchd log to files, never to journald:
|
|
# without these a cross-chassis black-hole cannot be attributed to
|
|
# IPsec versus SB port binding.
|
|
mkdir -p "${ARTIFACT_DIR}/ovs-logs-node${idx}"
|
|
ssh "${SSH_OPTS[@]}" tf-user@${IP} 'sudo tar -C /var/log -cf - ovn openvswitch 2>/dev/null || true' \
|
|
| tar -C "${ARTIFACT_DIR}/ovs-logs-node${idx}" -xf - 2>/dev/null
|
|
done
|
|
|
|
cat "${ARTIFACT_DIR}"/spinifex-journal-node*.log \
|
|
> "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null || true
|
|
echo "collected: $(ls "${ARTIFACT_DIR}" 2>/dev/null | wc -l) entries"
|
|
|
|
# Whether br-mgmt is a per-host island or one L2 segment spanning the
|
|
# cluster decides whether a system instance can reach the gateway address
|
|
# it was baked with. Nothing in a failing run records which one was built,
|
|
# and the environment is destroyed before it can be asked.
|
|
- name: Collect management-fabric topology
|
|
if: always()
|
|
working-directory: mulga/scripts/tofu-cluster
|
|
run: |
|
|
set +e
|
|
mkdir -p "${ARTIFACT_DIR}"
|
|
OUT="${ARTIFACT_DIR}/mgmt-fabric.txt"
|
|
INVENTORY=$(tofu output -json inventory 2>/dev/null) || { echo "no inventory; skipping"; exit 0; }
|
|
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path'); SSH_KEY="${SSH_KEY/#\~/$HOME}"
|
|
NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
|
|
SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY")
|
|
|
|
declare -a WAN=() MGMT=()
|
|
for i in $(seq 0 $((NODE_COUNT - 1))); do
|
|
WAN[$i]=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
|
|
done
|
|
|
|
for i in $(seq 0 $((NODE_COUNT - 1))); do
|
|
idx=$((i + 1))
|
|
{
|
|
echo "=== node${idx} (${WAN[$i]}) ==="
|
|
ssh "${SSH_OPTS[@]}" tf-user@"${WAN[$i]}" \
|
|
'sudo ovs-vsctl list-ports br-mgmt 2>&1 | sed "s/^/ports: /"; \
|
|
ip -4 addr show br-mgmt 2>&1 | sed "s/^/addr: /"; \
|
|
sudo ovs-vsctl get Bridge br-mgmt fail_mode 2>&1 | sed "s/^/fail: /"'
|
|
echo
|
|
} >> "$OUT" 2>&1
|
|
MGMT[$i]=$(ssh "${SSH_OPTS[@]}" tf-user@"${WAN[$i]}" \
|
|
"ip -4 -o addr show br-mgmt 2>/dev/null | awk '{print \$4}' | cut -d/ -f1" 2>/dev/null)
|
|
done
|
|
|
|
# An island cluster reaches only its own address and that is correct;
|
|
# a spanned one must reach every peer. Either result is a finding, and
|
|
# a mixed result is the failure this matrix exists to catch.
|
|
{
|
|
echo "=== br-mgmt reachability (row = prober, column = target) ==="
|
|
printf '%-10s' "from\\to"
|
|
for i in $(seq 0 $((NODE_COUNT - 1))); do printf '%-18s' "node$((i+1)):${MGMT[$i]:-none}"; done
|
|
echo
|
|
for i in $(seq 0 $((NODE_COUNT - 1))); do
|
|
printf '%-10s' "node$((i+1))"
|
|
for j in $(seq 0 $((NODE_COUNT - 1))); do
|
|
if [ -z "${MGMT[$j]}" ]; then printf '%-18s' "-"; continue; fi
|
|
if ssh "${SSH_OPTS[@]}" tf-user@"${WAN[$i]}" \
|
|
"ping -c1 -W1 ${MGMT[$j]} >/dev/null 2>&1"; then
|
|
printf '%-18s' ok
|
|
else
|
|
printf '%-18s' FAIL
|
|
fi
|
|
done
|
|
echo
|
|
done
|
|
} >> "$OUT" 2>&1
|
|
|
|
cat "$OUT"
|
|
|
|
- name: Install go-junit-report
|
|
if: always()
|
|
run: |
|
|
go install github.com/jstemmer/go-junit-report/v2@latest
|
|
echo "$HOME/go/bin" >> "$GITHUB_PATH"
|
|
|
|
- name: Convert test logs to JUnit XML
|
|
if: always()
|
|
run: |
|
|
shopt -s nullglob
|
|
for LOG in "${ARTIFACT_DIR}"/test-*.log; do
|
|
BASE=$(basename "$LOG" .log); SUITE=${BASE#test-}
|
|
go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
|
|
done
|
|
|
|
# Tolerated on failure: the sink being unreachable must not turn a
|
|
# green test run red.
|
|
- name: Ship test results to Elasticsearch
|
|
if: always()
|
|
continue-on-error: true
|
|
uses: ./spinifex/.github/actions/junit-to-es
|
|
with:
|
|
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
|
|
cell: ${{ matrix.cell }}
|
|
|
|
- name: Analyze e2e failures
|
|
if: always()
|
|
uses: ./spinifex/.github/actions/e2e-analyze
|
|
with:
|
|
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
|
|
log-dir: ${{ env.ARTIFACT_DIR }}
|
|
title: "E2E failure analysis (cell ${{ matrix.cell }})"
|
|
|
|
- uses: actions/upload-artifact@v7
|
|
if: always()
|
|
with:
|
|
name: e2e-artifacts-cell-${{ matrix.cell }}-${{ github.run_id }}
|
|
path: ${{ env.ARTIFACT_DIR }}
|
|
if-no-files-found: warn
|
|
retention-days: 14
|
|
|
|
# The XML on its own, so the roll-up can read every cell's counts without
|
|
# pulling nineteen journals it has no use for.
|
|
- name: Upload junit only
|
|
if: always()
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: junit-cell-${{ matrix.cell }}-${{ github.run_id }}
|
|
path: ${{ env.ARTIFACT_DIR }}/junit-*.xml
|
|
if-no-files-found: ignore
|
|
retention-days: 14
|
|
|
|
# Gated like every other step, not just always(): a cell the dispatch
|
|
# deselected never checked out or ran tofu init, so destroying would fail
|
|
# on missing plugins and report the cell red for correctly doing nothing.
|
|
- name: Teardown
|
|
if: always()
|
|
working-directory: mulga/scripts/tofu-cluster
|
|
env:
|
|
MATRIX_HOST: ${{ matrix.host }}
|
|
run: |
|
|
# Must mirror the Provision step's override, or destroy runs against a
|
|
# different image path than apply did.
|
|
TOFU_VAR_ARGS=""
|
|
case "$MATRIX_HOST" in
|
|
ubuntu-*) TOFU_VAR_ARGS="-var=host_image_override=${UBUNTU_GOLD_IMAGE}" ;;
|
|
esac
|
|
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
|
|
tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
|
|
timeout 120 ./flush-telemetry.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
|
|
# shellcheck disable=SC2086
|
|
retry_run "tofu destroy" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars ${TOFU_VAR_ARGS} -auto-approve -input=false
|
|
./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
|
|
|
|
- name: Upload tofu log
|
|
if: always()
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: tofu-log-cell-${{ matrix.cell }}-${{ github.run_id }}
|
|
path: ${{ env.TOFU_LOG }}
|
|
if-no-files-found: ignore
|
|
retention-days: 14
|
|
|
|
# Bare-metal single-node suite. Doesn't fit the tofu matrix shape, so it
|
|
# runs as a sibling job via the reusable e2e-baremetal workflow. Gated like a
|
|
# matrix cell ("cell 20"): blank cells is the whole nightly, or name 20 to run
|
|
# it alone. A reusable call needs its own ref, since it cannot read GIT_BRANCH.
|
|
baremetal:
|
|
name: cell-20 (baremetal/pxe/real-hw)
|
|
if: >-
|
|
inputs.cells == '' ||
|
|
contains(format(',{0},', inputs.cells), ',20,')
|
|
permissions:
|
|
contents: read
|
|
statuses: write
|
|
uses: ./.github/workflows/e2e-baremetal.yml
|
|
with:
|
|
ref: ${{ github.ref_name }}
|
|
# A reusable-workflow call gets none of this workflow's context (env:
|
|
# above, matrix.cell) for free — cell-20 needs its own attrs string.
|
|
# There's no matrix leg here, just the one fixed baremetal cell.
|
|
otel-attrs: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=20,ci.matrix=baremetal/pxe/real-hw
|
|
secrets: inherit
|
|
|
|
# The guest disk-performance gate. Same bare-metal node as cell 20 and
|
|
# therefore queued behind it by the shared concurrency group, which is the
|
|
# point: perf numbers off a cell still running other suites are noise, and
|
|
# the availability assertions measure host responsiveness under a single
|
|
# tenant's load, which a co-tenant correctness suite would confound.
|
|
#
|
|
# It reimages the cluster first, like every other caller of that workflow, so
|
|
# it is a fresh node rather than whatever cell-20 left behind.
|
|
diskperf:
|
|
name: cell-28 (baremetal/diskperf)
|
|
needs: baremetal
|
|
if: >-
|
|
!cancelled() && (
|
|
inputs.cells == '' ||
|
|
contains(format(',{0},', inputs.cells), ',28,')
|
|
)
|
|
permissions:
|
|
contents: read
|
|
statuses: write
|
|
uses: ./.github/workflows/e2e-baremetal.yml
|
|
with:
|
|
ref: ${{ github.ref_name }}
|
|
cell: "28"
|
|
suites: diskperf
|
|
otel-attrs: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=28,ci.matrix=baremetal/pxe/diskperf
|
|
secrets: inherit
|
|
|
|
# ISO cells. Unlike 1-19 these need no tofu: the harnesses run QEMU directly
|
|
# on the runner. The nodes are therefore L1, which keeps the instances they
|
|
# launch at L2 and their benchmark numbers comparable between the two cells.
|
|
#
|
|
# Serialised against each other because the guest MACs are fixed
|
|
# (52:54:00:12:...): stable MACs are what make DHCP discovery work, and two
|
|
# concurrent runs would put duplicates on the rack LAN, which is a confusing
|
|
# failure for everyone on it, not just CI.
|
|
iso-single:
|
|
name: cell-21 (iso/dhcp/single)
|
|
if: >-
|
|
inputs.cells == '' ||
|
|
contains(format(',{0},', inputs.cells), ',21,')
|
|
runs-on: [self-hosted, ci-single]
|
|
# ISO build, install, boot and pool re-form is ~35 of these on its own,
|
|
# before the workload step imports an AMI and launches an instance.
|
|
timeout-minutes: 90
|
|
concurrency:
|
|
group: iso-qemu-rack
|
|
cancel-in-progress: false
|
|
env:
|
|
CI_WAN_MODE: dhcp
|
|
CI_PASSWORD: ${{ secrets.BM_CI_PASSWORD }}
|
|
# What a real node gets. The harness default of 4G is a dev-box size, and
|
|
# importing an 820 MiB AMI on top of the running stack does not fit in it.
|
|
MEM: 8G
|
|
SMP: 8
|
|
# .230 is consumed by the OVN router's SNAT, leaving .231-.234 for
|
|
# instances. Disjoint from cell-22's pool so a run that dies holding its
|
|
# SNAT address cannot collide with the next cell.
|
|
EXT_POOL_START: 192.168.0.230
|
|
EXT_POOL_END: 192.168.0.234
|
|
EXT_GATEWAY: 192.168.1.1
|
|
EXT_PREFIX_LEN: 23
|
|
# Pins where the benchmark writes, so collecting and removing it later are
|
|
# both exact. Left to itself the harness picks a timestamped name.
|
|
BENCH_OUT: /tmp/cell21-bench-${{ github.run_id }}
|
|
steps:
|
|
- name: Clean Workspace
|
|
run: |
|
|
sudo umount -R -l /var/cache/spinifex-iso-builder/rootfs 2>/dev/null || true
|
|
sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
|
|
|
|
- name: Configure work directories
|
|
run: |
|
|
E2E_ROOT=/mnt/nvme/e2e
|
|
mkdir -p "$E2E_ROOT/work" 2>/dev/null ||
|
|
{ sudo mkdir -p "$E2E_ROOT/work" && sudo chown "$(id -u):$(id -g)" "$E2E_ROOT" "$E2E_ROOT/work"; }
|
|
echo "TMPDIR=$E2E_ROOT/work" >> "$GITHUB_ENV"
|
|
|
|
- uses: actions/create-github-app-token@v3
|
|
id: app-token
|
|
with:
|
|
client-id: ${{ secrets.APP_ID }}
|
|
private-key: ${{ secrets.APP_PRIVATE_KEY }}
|
|
repositories: mulga
|
|
|
|
- uses: actions/checkout@v7
|
|
with:
|
|
repository: mulgadc/mulga
|
|
token: ${{ steps.app-token.outputs.token }}
|
|
path: mulga
|
|
|
|
- uses: actions/checkout@v7
|
|
with:
|
|
ref: ${{ env.GIT_BRANCH }}
|
|
path: mulga/spinifex
|
|
|
|
- name: Resolve sibling refs
|
|
id: siblings
|
|
env:
|
|
BRANCH: ${{ env.GIT_BRANCH }}
|
|
run: bash .github/resolve-sibling-refs.sh
|
|
working-directory: mulga/spinifex
|
|
|
|
- uses: actions/checkout@v7
|
|
with:
|
|
repository: mulgadc/viperblock
|
|
ref: ${{ steps.siblings.outputs.viperblock }}
|
|
path: mulga/viperblock
|
|
|
|
- uses: actions/checkout@v7
|
|
with:
|
|
repository: mulgadc/predastore
|
|
ref: ${{ steps.siblings.outputs.predastore }}
|
|
path: mulga/predastore
|
|
|
|
- uses: actions/checkout@v7
|
|
with:
|
|
repository: mulgadc/northstar
|
|
ref: ${{ steps.siblings.outputs.northstar }}
|
|
path: mulga/northstar
|
|
|
|
- uses: actions/checkout@v7
|
|
with:
|
|
repository: mulgadc/bluebottle
|
|
ref: ${{ steps.siblings.outputs.bluebottle }}
|
|
path: mulga/bluebottle
|
|
|
|
# Assert the host can do what the cell assumes, before spending 40
|
|
# minutes finding out it cannot. A checklist in a plan doc goes stale;
|
|
# this does not.
|
|
- name: Preflight
|
|
run: |
|
|
set -euo pipefail
|
|
mkdir -p "$TMPDIR"
|
|
ip link show "${BRIDGE:-br-wan}" >/dev/null ||
|
|
{ echo "::error::bridge br-wan not found on this runner"; exit 1; }
|
|
grep -qE "^allow +${BRIDGE:-br-wan}$" /etc/qemu/bridge.conf ||
|
|
{ echo "::error::br-wan not permitted in /etc/qemu/bridge.conf"; exit 1; }
|
|
command -v sshpass >/dev/null ||
|
|
{ echo "::error::sshpass not installed"; exit 1; }
|
|
free_mb=$(free -m | awk '/^Mem:/ {print $7}')
|
|
[ "$free_mb" -ge 8192 ] ||
|
|
{ echo "::error::need 8 GiB available, have ${free_mb} MiB"; exit 1; }
|
|
# A 40G image on tmpfs competes with the 8 GiB the VM itself needs.
|
|
fstype=$(df --output=fstype "$TMPDIR" | tail -1 | tr -d ' ')
|
|
[ "$fstype" != tmpfs ] ||
|
|
{ echo "::error::$TMPDIR is tmpfs — the node disk would be in RAM"; exit 1; }
|
|
df -h "$TMPDIR"
|
|
avail_gb=$(df -BG --output=avail "$TMPDIR" | tail -1 | tr -dc '0-9')
|
|
[ "$avail_gb" -ge 60 ] ||
|
|
{ echo "::error::need 60 GiB on $TMPDIR, have ${avail_gb} GiB"; exit 1; }
|
|
|
|
- name: Build artefacts
|
|
working-directory: mulga/scripts/iso-builder
|
|
run: make build-rootfs build-squashfs build-initramfs
|
|
|
|
# One key for both halves of the cell. test-workload.sh SSHes to the node
|
|
# AND registers the same key as the EC2 key pair it uses to reach the
|
|
# instances, so the key test-smoke.sh installs has to be that one — its
|
|
# default ~/.ssh/id_*.pub would leave the workload step locked out.
|
|
- name: Prepare SSH key
|
|
run: |
|
|
set -euo pipefail
|
|
mkdir -p "$HOME/.ssh" && chmod 700 "$HOME/.ssh"
|
|
# Two keys, because the guest key must not share a name with the one
|
|
# smoke-test.sh registers from the node: the node holds that private
|
|
# half, and IMDS caches material by key name for five minutes, so
|
|
# re-importing over it is invisible to the guests.
|
|
for k in spinifex-key spinifex-workload-key; do
|
|
[ -f "$HOME/.ssh/$k" ] ||
|
|
ssh-keygen -t ed25519 -N "" -f "$HOME/.ssh/$k" -C "$k-ci"
|
|
done
|
|
|
|
# Boots the ISO, installs unattended, reads the lease off the serial
|
|
# console, runs the in-guest smoke test, and installs the key for the
|
|
# next step.
|
|
- name: Install and smoke test
|
|
id: smoke
|
|
working-directory: mulga/scripts/iso-builder
|
|
run: |
|
|
set -euo pipefail
|
|
# KEEP_NODE: the harness owns the only VM there is and tears it down
|
|
# on exit, so without this the workload step below has no node left.
|
|
KEEP_NODE=1 SSH_IDENTITY="$HOME/.ssh/spinifex-key.pub" \
|
|
./test/test-smoke.sh | tee /tmp/smoke-$GITHUB_RUN_ID.log
|
|
ip=$(sed -n 's/^\[smoke\] node address: //p' /tmp/smoke-$GITHUB_RUN_ID.log | tail -1)
|
|
[ -n "$ip" ] || { echo "::error::smoke test did not report a node address"; exit 1; }
|
|
echo "node_ip=$ip" >> "$GITHUB_OUTPUT"
|
|
|
|
# The customer path: AMI, key, VPC, instance, SSH in. --benchmark records
|
|
# host and guest disk numbers so a regression shows up as a number rather
|
|
# than a story.
|
|
- name: Workload and benchmark
|
|
working-directory: mulga/scripts/iso-builder
|
|
run: |
|
|
set -euo pipefail
|
|
cat > /tmp/cell21.env <<EOF
|
|
NODES=(${{ steps.smoke.outputs.node_ip }})
|
|
NODE_NAMES=(spinifex-ci)
|
|
SSH_USER=spinifex
|
|
SSH_KEY_PATH=$HOME/.ssh/spinifex-workload-key
|
|
# SSH_KEY_PATH reaches the guests, SSH_IDENTITY reaches the node, and
|
|
# they are deliberately different keys. Without the second, node SSH
|
|
# falls back to the runner's default identity, which the node has
|
|
# never seen.
|
|
SSH_IDENTITY=$HOME/.ssh/spinifex-key
|
|
EXT_POOL_START=$EXT_POOL_START
|
|
EXT_POOL_END=$EXT_POOL_END
|
|
EXT_GATEWAY=$EXT_GATEWAY
|
|
EXT_PREFIX_LEN=$EXT_PREFIX_LEN
|
|
EOF
|
|
./test/test-workload.sh --env /tmp/cell21.env --benchmark
|
|
|
|
# The node's own journal, which nothing else preserves: the serial log
|
|
# stops at login and the harness owns the only node there is. Pulled
|
|
# before teardown and never fatal, so a passing night keeps it too.
|
|
- name: Collect node journal
|
|
if: always() && steps.smoke.outputs.node_ip != ''
|
|
env:
|
|
NODE_IP: ${{ steps.smoke.outputs.node_ip }}
|
|
run: |
|
|
mkdir -p /tmp/cell21-artifacts-$GITHUB_RUN_ID
|
|
timeout 180 ssh -o BatchMode=yes -o StrictHostKeyChecking=no \
|
|
-o UserKnownHostsFile=/dev/null -o ConnectTimeout=10 \
|
|
-i "$HOME/.ssh/spinifex-key" "spinifex@${NODE_IP}" \
|
|
'sudo journalctl -b --no-pager --output=short-iso | gzip -6' \
|
|
> "/tmp/cell21-artifacts-$GITHUB_RUN_ID/node-journal.log.gz" ||
|
|
echo "::warning::could not pull the node journal from ${NODE_IP}"
|
|
|
|
- name: Collect logs
|
|
if: always()
|
|
run: |
|
|
mkdir -p /tmp/cell21-artifacts-$GITHUB_RUN_ID
|
|
cp /tmp/smoke-$GITHUB_RUN_ID.log /tmp/cell21-artifacts-$GITHUB_RUN_ID/ ||
|
|
echo "::warning::no smoke log to collect"
|
|
# fio JSON and the summary table. Warned about rather than swallowed:
|
|
# a benchmark that ran and produced nothing is worth knowing about.
|
|
if [ -d "$BENCH_OUT" ]; then
|
|
cp -r "$BENCH_OUT" /tmp/cell21-artifacts-$GITHUB_RUN_ID/benchmark
|
|
else
|
|
echo "::warning::no benchmark results at $BENCH_OUT"
|
|
fi
|
|
# The guest's own view. Without it a node that dies mid-test is only
|
|
# visible as the harness command that stopped producing output.
|
|
serial=$(sed -n 's/^\[smoke\] serial log preserved: //p' \
|
|
/tmp/smoke-$GITHUB_RUN_ID.log | tail -1)
|
|
[ -n "$serial" ] && cp "$serial" /tmp/cell21-artifacts-$GITHUB_RUN_ID/serial.log || true
|
|
node=$(sed -n 's/^\[smoke\] node logs preserved: //p' \
|
|
/tmp/smoke-$GITHUB_RUN_ID.log | tail -1)
|
|
[ -n "$node" ] && cp "$node" /tmp/cell21-artifacts-$GITHUB_RUN_ID/node.log || true
|
|
# QEMU's stderr no longer reaches the step log: it is redirected so a
|
|
# kept-alive VM cannot hold the step's stdout open.
|
|
qemu=$(sed -n 's/^\[smoke\] qemu log preserved: //p' \
|
|
/tmp/smoke-$GITHUB_RUN_ID.log | tail -1)
|
|
[ -n "$qemu" ] && cp "$qemu" /tmp/cell21-artifacts-$GITHUB_RUN_ID/qemu.log || true
|
|
|
|
- uses: actions/upload-artifact@v7
|
|
if: always()
|
|
with:
|
|
name: e2e-artifacts-cell-21-${{ github.run_id }}
|
|
path: /tmp/cell21-artifacts-${{ github.run_id }}
|
|
if-no-files-found: warn
|
|
# Matches the predastore benchmark in cell-23: a night's numbers are
|
|
# only useful against the nights before them.
|
|
retention-days: 90
|
|
|
|
- name: Teardown
|
|
if: always()
|
|
working-directory: mulga/scripts/iso-builder
|
|
run: |
|
|
# No env file means the workload step never ran, so there is nothing
|
|
# of its to clean up.
|
|
if [ -f /tmp/cell21.env ]; then
|
|
./test/test-workload.sh --env /tmp/cell21.env --cleanup || true
|
|
fi
|
|
sudo pkill -f spinifex-smoke-test || true
|
|
rm -rf /tmp/cell21-artifacts-$GITHUB_RUN_ID /tmp/cell21.env \
|
|
/tmp/smoke-$GITHUB_RUN_ID.log "$BENCH_OUT"
|
|
# Runs before BENCH_OUT existed left a timestamped directory behind
|
|
# every night. Sweep anything older than a day.
|
|
find /tmp -maxdepth 1 -name 'spinifex-benchmark-*' -type d -mmin +1440 \
|
|
-exec rm -rf {} + 2>/dev/null || true
|
|
# KEEP_NODE leaves the node's disk and serial log behind for us.
|
|
sleep 2
|
|
rm -f "$TMPDIR"/tmp.*.qcow2 "$TMPDIR"/tmp.*.log "$TMPDIR"/tmp.*-node.log
|
|
|
|
iso-multi:
|
|
name: cell-22 (iso/dhcp/three-node)
|
|
# After cell-21 rather than beside it: same concurrency group, and running
|
|
# them together would need 32 GiB on one box.
|
|
needs: [iso-single]
|
|
if: >-
|
|
!cancelled() && (
|
|
inputs.cells == '' ||
|
|
contains(format(',{0},', inputs.cells), ',22,') )
|
|
runs-on: [self-hosted, ci-multi]
|
|
timeout-minutes: 90
|
|
concurrency:
|
|
group: iso-qemu-rack
|
|
cancel-in-progress: false
|
|
env:
|
|
MEM: 8G
|
|
SMP: 8
|
|
PASSWORD: ${{ secrets.BM_CI_PASSWORD }}
|
|
EXT_POOL_START: 192.168.0.235
|
|
EXT_POOL_END: 192.168.0.239
|
|
EXT_GATEWAY: 192.168.1.1
|
|
EXT_PREFIX_LEN: 23
|
|
steps:
|
|
- name: Clean Workspace
|
|
run: |
|
|
sudo umount -R -l /var/cache/spinifex-iso-builder/rootfs 2>/dev/null || true
|
|
sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
|
|
|
|
- name: Configure work directories
|
|
run: |
|
|
E2E_ROOT=/mnt/nvme/e2e
|
|
mkdir -p "$E2E_ROOT/work" 2>/dev/null ||
|
|
{ sudo mkdir -p "$E2E_ROOT/work" && sudo chown "$(id -u):$(id -g)" "$E2E_ROOT" "$E2E_ROOT/work"; }
|
|
echo "TMPDIR=$E2E_ROOT/work" >> "$GITHUB_ENV"
|
|
# Pinned so the per-node serial logs can be collected afterwards, and
|
|
# kept beside TMPDIR rather than in /tmp: it also holds nine 40G node
|
|
# disks, which is what the tmpfs assertion below is about.
|
|
echo "WORK_DIR=$E2E_ROOT/work/cell22-$GITHUB_RUN_ID" >> "$GITHUB_ENV"
|
|
|
|
- uses: actions/create-github-app-token@v3
|
|
id: app-token
|
|
with:
|
|
client-id: ${{ secrets.APP_ID }}
|
|
private-key: ${{ secrets.APP_PRIVATE_KEY }}
|
|
repositories: mulga
|
|
|
|
- uses: actions/checkout@v7
|
|
with:
|
|
repository: mulgadc/mulga
|
|
token: ${{ steps.app-token.outputs.token }}
|
|
path: mulga
|
|
|
|
- uses: actions/checkout@v7
|
|
with:
|
|
ref: ${{ env.GIT_BRANCH }}
|
|
path: mulga/spinifex
|
|
|
|
- name: Resolve sibling refs
|
|
id: siblings
|
|
env:
|
|
BRANCH: ${{ env.GIT_BRANCH }}
|
|
run: bash .github/resolve-sibling-refs.sh
|
|
working-directory: mulga/spinifex
|
|
|
|
- uses: actions/checkout@v7
|
|
with:
|
|
repository: mulgadc/viperblock
|
|
ref: ${{ steps.siblings.outputs.viperblock }}
|
|
path: mulga/viperblock
|
|
|
|
- uses: actions/checkout@v7
|
|
with:
|
|
repository: mulgadc/predastore
|
|
ref: ${{ steps.siblings.outputs.predastore }}
|
|
path: mulga/predastore
|
|
|
|
- uses: actions/checkout@v7
|
|
with:
|
|
repository: mulgadc/northstar
|
|
ref: ${{ steps.siblings.outputs.northstar }}
|
|
path: mulga/northstar
|
|
|
|
- uses: actions/checkout@v7
|
|
with:
|
|
repository: mulgadc/bluebottle
|
|
ref: ${{ steps.siblings.outputs.bluebottle }}
|
|
path: mulga/bluebottle
|
|
|
|
# 3 x 8 GiB plus 9 sparse 40G disks, before the instances the cluster
|
|
# launches on top.
|
|
- name: Preflight
|
|
run: |
|
|
set -euo pipefail
|
|
mkdir -p "$TMPDIR"
|
|
free_mb=$(free -m | awk '/^Mem:/ {print $7}')
|
|
[ "$free_mb" -ge 26624 ] ||
|
|
{ echo "::error::need 26 GiB available, have ${free_mb} MiB"; exit 1; }
|
|
fstype=$(df --output=fstype "$TMPDIR" | tail -1 | tr -d ' ')
|
|
[ "$fstype" != tmpfs ] ||
|
|
{ echo "::error::$TMPDIR is tmpfs — nine node disks would be in RAM"; exit 1; }
|
|
df -h "$TMPDIR"
|
|
avail_gb=$(df -BG --output=avail "$TMPDIR" | tail -1 | tr -dc '0-9')
|
|
[ "$avail_gb" -ge 120 ] ||
|
|
{ echo "::error::need 120 GiB on $TMPDIR ($fstype), have ${avail_gb} GiB"; exit 1; }
|
|
|
|
- name: Build artefacts
|
|
working-directory: mulga/scripts/iso-builder
|
|
run: make build-rootfs build-squashfs build-initramfs
|
|
|
|
# Four phases: install three nodes, convert to one cluster, assert it
|
|
# formed (including the external pool), then pull node 1's power and
|
|
# assert the survivors hold quorum.
|
|
#
|
|
# Phase 5 (test-workload.sh, as cell-21 runs it) is not wired yet: the
|
|
# nodes are still on user-mode NAT and reachable only through the
|
|
# harness's own port forwards, and test-cluster.sh does not open the
|
|
# formation port. Both land with mulga-yxtda and the bridged WAN mode;
|
|
# until then this cell proves the cluster forms and survives, not that a
|
|
# customer can use it.
|
|
- name: Install, form and verify the cluster
|
|
working-directory: mulga/scripts/iso-builder
|
|
run: |
|
|
set -euo pipefail
|
|
./test/test-cluster.sh 2>&1 | tee /tmp/cell22-$GITHUB_RUN_ID.log
|
|
|
|
- name: Collect logs
|
|
if: always()
|
|
run: |
|
|
mkdir -p /tmp/cell22-artifacts-$GITHUB_RUN_ID
|
|
cp /tmp/cell22-$GITHUB_RUN_ID.log /tmp/cell22-artifacts-$GITHUB_RUN_ID/ ||
|
|
echo "::warning::no cluster log to collect"
|
|
# Serial only. The same directory holds the nodes' disk images, which
|
|
# are gigabytes and must not go into an artifact.
|
|
cp "$WORK_DIR"/*-serial.log /tmp/cell22-artifacts-$GITHUB_RUN_ID/ 2>/dev/null ||
|
|
echo "no per-node serial logs — the run passed and cleaned up after itself"
|
|
|
|
- uses: actions/upload-artifact@v7
|
|
if: always()
|
|
with:
|
|
name: e2e-artifacts-cell-22-${{ github.run_id }}
|
|
path: /tmp/cell22-artifacts-${{ github.run_id }}
|
|
if-no-files-found: warn
|
|
retention-days: 90
|
|
|
|
- name: Teardown
|
|
if: always()
|
|
run: |
|
|
sudo pkill -f spinifex-cluster-test || true
|
|
rm -rf /tmp/cell22-artifacts-$GITHUB_RUN_ID /tmp/cell22-$GITHUB_RUN_ID.log \
|
|
"$WORK_DIR"
|
|
|
|
# Predastore throughput ("cell 23"). Predastore has no runners of its own and
|
|
# spinifex depends on it, so the benchmark lives here and checks predastore
|
|
# out as a sibling.
|
|
#
|
|
# needs: is the whole point. A throughput benchmark sharing a host with a cell
|
|
# measures the neighbour, and ci-single and ci-multi both resolve to
|
|
# bottlebrush, so "a different label" would not have separated them. needs: on
|
|
# a matrix job waits for every leg, so this starts on a quiet machine.
|
|
#
|
|
# !cancelled() rather than always(): a failed cell must not silently skip the
|
|
# benchmark and leave a green-looking night with no number in it, but always()
|
|
# also overrides cancellation and keeps the run starting jobs after a cancel.
|
|
performance:
|
|
name: cell-23 (predastore/throughput)
|
|
needs: [matrix, baremetal, iso-single, iso-multi]
|
|
if: >-
|
|
!cancelled() && (
|
|
inputs.cells == '' ||
|
|
contains(format(',{0},', inputs.cells), ',23,') )
|
|
runs-on: [self-hosted, ci-single]
|
|
timeout-minutes: 60
|
|
env:
|
|
# Two-minute samples. The 30s smoke preset is too short to compare nights.
|
|
PERF_PRESET: compare
|
|
steps:
|
|
- name: Clean Workspace
|
|
run: sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
|
|
|
|
- name: Configure work directories
|
|
run: |
|
|
E2E_ROOT=/mnt/nvme/e2e
|
|
mkdir -p "$E2E_ROOT/work" 2>/dev/null ||
|
|
{ sudo mkdir -p "$E2E_ROOT/work" && sudo chown "$(id -u):$(id -g)" "$E2E_ROOT" "$E2E_ROOT/work"; }
|
|
echo "TMPDIR=$E2E_ROOT/work" >> "$GITHUB_ENV"
|
|
echo "PERF_RESULTS_ROOT=$E2E_ROOT/predastore-perf-$GITHUB_RUN_ID" >> "$GITHUB_ENV"
|
|
|
|
- name: Checkout Spinifex
|
|
uses: actions/checkout@v7
|
|
with:
|
|
path: spinifex
|
|
|
|
# Which predastore branch to measure: the same-named branch when one
|
|
# exists, dev otherwise. Matches how the e2e path already picks siblings,
|
|
# so a cross-repo change is benchmarked as a set.
|
|
- name: Resolve sibling refs
|
|
id: siblings
|
|
env:
|
|
BRANCH: ${{ github.ref_name }}
|
|
run: bash .github/resolve-sibling-refs.sh
|
|
working-directory: spinifex
|
|
|
|
- name: Checkout Predastore
|
|
uses: actions/checkout@v7
|
|
with:
|
|
repository: mulgadc/predastore
|
|
ref: ${{ steps.siblings.outputs.predastore }}
|
|
path: predastore
|
|
|
|
# The benchmark works in $TMPDIR and predastore refuses writes below a 5%
|
|
# free-space watermark. Without this the symptom is several hundred
|
|
# "storage pool is at capacity" lines from warp and no statement of how
|
|
# much space there actually was.
|
|
#
|
|
# TMPDIR is pointed at the runner's disk above, and the tmpfs assertion is
|
|
# the one that matters: a storage benchmark on a RAM-backed filesystem
|
|
# measures memcpy. It would report a number that looks excellent, sit in
|
|
# the baseline, and never notice a disk regression.
|
|
- name: Preflight
|
|
run: |
|
|
set -euo pipefail
|
|
mkdir -p "$TMPDIR"
|
|
fstype=$(df --output=fstype "$TMPDIR" | tail -1 | tr -d ' ')
|
|
case "$fstype" in
|
|
tmpfs|ramfs)
|
|
echo "::error::$TMPDIR is $fstype — a storage benchmark there measures RAM"
|
|
exit 1 ;;
|
|
esac
|
|
# Work dirs from runs that were killed before their cleanup trap ran.
|
|
# -mtime +0 so a concurrent run's directory is never in scope.
|
|
find "$TMPDIR" -maxdepth 1 -name 'predastore-e2e-performance.*' \
|
|
-type d -mtime +0 -exec rm -rf {} + 2>/dev/null || true
|
|
df -h "$TMPDIR"
|
|
avail_gb=$(df -BG --output=avail "$TMPDIR" | tail -1 | tr -dc '0-9')
|
|
pct_used=$(df --output=pcent "$TMPDIR" | tail -1 | tr -dc '0-9')
|
|
if [ "$pct_used" -ge 90 ] || [ "$avail_gb" -lt 20 ]; then
|
|
echo "::error::$TMPDIR ($fstype) is ${pct_used}% full with ${avail_gb} GiB free."
|
|
echo "::error::The benchmark needs headroom above predastore's 5% watermark."
|
|
exit 1
|
|
fi
|
|
|
|
# PERF_PORT_OFFSET defaults to 10000, so the benchmark's own nodes do not
|
|
# collide with a cluster left on the default ports.
|
|
- name: Run throughput benchmark
|
|
working-directory: predastore
|
|
run: |
|
|
set -euo pipefail
|
|
# Into the results directory, so the artifact reads on its own rather
|
|
# than needing the job log it was printed into. The benchmark only
|
|
# ever creates this directory, so a log placed here first survives.
|
|
mkdir -p "$PERF_RESULTS_ROOT"
|
|
make e2e-performance 2>&1 | tee "$PERF_RESULTS_ROOT/e2e-performance.log"
|
|
|
|
# Report-only. The baseline comparison and the -10% gate are mulga-wpjsx;
|
|
# until the run-to-run spread is known from real nights, a threshold would
|
|
# be a guess, and a gate that cries wolf gets muted.
|
|
- uses: actions/upload-artifact@v7
|
|
if: always()
|
|
with:
|
|
name: predastore-perf-${{ github.run_id }}
|
|
path: ${{ env.PERF_RESULTS_ROOT }}
|
|
if-no-files-found: warn
|
|
retention-days: 90
|
|
|
|
- name: Clean up results
|
|
if: always()
|
|
run: rm -rf "$PERF_RESULTS_ROOT"
|
|
|
|
# Fault injection, not throughput. Cell-23 asks how fast predastore is when
|
|
# every node is healthy; this asks whether it survives one that is not.
|
|
#
|
|
# After cell-23 rather than beside it: both want a quiet single runner, and a
|
|
# benchmark sharing a machine with a fault injection run measures the wrong
|
|
# thing. The port offset keeps them apart even so.
|
|
stress:
|
|
name: cell-24 (predastore/fault-injection)
|
|
needs: [matrix, baremetal, iso-single, iso-multi, performance]
|
|
if: >-
|
|
!cancelled() && (
|
|
inputs.cells == '' ||
|
|
contains(format(',{0},', inputs.cells), ',24,') )
|
|
runs-on: [self-hosted, ci-single]
|
|
# The thirteen scenarios run serially and are ~35 minutes of work on their
|
|
# own, measured from the per-scenario predastore matrix, before checkout
|
|
# and build. 30 could not have passed whatever the disk did.
|
|
timeout-minutes: 75
|
|
env:
|
|
# Freezes a follower. STRESS_HOST=leader is a known failure: a gate keeps
|
|
# the frozen node first in its meta read order and pays the full client
|
|
# timeout per key, so listing slows in proportion to object count.
|
|
STRESS_HOST: follower
|
|
# Clear of cell-23's default 10000, so the two never collide on a runner
|
|
# that lets them overlap.
|
|
STRESS_PORT_OFFSET: "20000"
|
|
steps:
|
|
- name: Clean Workspace
|
|
run: sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/*
|
|
|
|
# STRESS_WORK_ROOT is set as well as TMPDIR because the harness ignores
|
|
# TMPDIR for the scenarios that need the most disk: large-object and
|
|
# multipart-upload default to $HOME/.cache/predastore-e2e, on the theory
|
|
# that TMPDIR is commonly tmpfs. Here it is a 938G nvme, so pointing them
|
|
# at it is both correct and the only path with room.
|
|
- name: Configure work directories
|
|
run: |
|
|
E2E_ROOT=/mnt/nvme/e2e
|
|
for d in work stress-work; do
|
|
mkdir -p "$E2E_ROOT/$d" 2>/dev/null ||
|
|
{ sudo mkdir -p "$E2E_ROOT/$d" && sudo chown "$(id -u):$(id -g)" "$E2E_ROOT" "$E2E_ROOT/$d"; }
|
|
done
|
|
echo "TMPDIR=$E2E_ROOT/work" >> "$GITHUB_ENV"
|
|
echo "STRESS_WORK_ROOT=$E2E_ROOT/stress-work" >> "$GITHUB_ENV"
|
|
echo "STRESS_RESULTS_ROOT=$E2E_ROOT/predastore-stress-$GITHUB_RUN_ID" >> "$GITHUB_ENV"
|
|
|
|
- name: Checkout Spinifex
|
|
uses: actions/checkout@v7
|
|
with:
|
|
path: spinifex
|
|
|
|
- name: Resolve sibling refs
|
|
id: siblings
|
|
env:
|
|
BRANCH: ${{ github.ref_name }}
|
|
run: bash .github/resolve-sibling-refs.sh
|
|
working-directory: spinifex
|
|
|
|
- name: Checkout Predastore
|
|
uses: actions/checkout@v7
|
|
with:
|
|
repository: mulgadc/predastore
|
|
ref: ${{ steps.siblings.outputs.predastore }}
|
|
path: predastore
|
|
|
|
# A frozen node is indistinguishable from a slow one, so a run on a
|
|
# loaded or nearly full disk fails as a timeout and reads as a real
|
|
# fault. The checks are the cheap way to tell those apart afterwards.
|
|
- name: Preflight
|
|
run: |
|
|
set -euo pipefail
|
|
mkdir -p "$TMPDIR" "$STRESS_WORK_ROOT"
|
|
for d in "$TMPDIR" "$STRESS_WORK_ROOT" "$HOME/.cache/predastore-e2e"; do
|
|
[ -d "$d" ] || continue
|
|
find "$d" -maxdepth 1 -name 'predastore-e2e-stress.*' \
|
|
-type d -mtime +0 -exec rm -rf {} + 2>/dev/null || true
|
|
done
|
|
# Both paths, because they can be different filesystems and the one
|
|
# holding the large objects is not the one TMPDIR names. large-object
|
|
# needs 2GiB and 4GiB sharded with parity, multipart 1GiBx8.
|
|
rc=0
|
|
for d in "$TMPDIR" "$STRESS_WORK_ROOT"; do
|
|
df -h "$d"
|
|
avail_gb=$(df -BG --output=avail "$d" | tail -1 | tr -dc '0-9')
|
|
if [ "$avail_gb" -lt 40 ]; then
|
|
echo "::error::$d has ${avail_gb} GiB free; the run needs headroom above predastore's 5% watermark and room for the 1GiBx8 multipart case."
|
|
rc=1
|
|
fi
|
|
done
|
|
exit "$rc"
|
|
|
|
# Leftovers from a run killed before its cleanup trap: a surviving node
|
|
# holds its ports and fails this run somewhere unrelated.
|
|
#
|
|
# Under the host benchmark lock, because this pattern matches every
|
|
# predastore node on the runner and the predastore repository runs the
|
|
# same harnesses on the same machine. Holding the lock is what makes
|
|
# "still running" mean "stale" rather than "somebody else's benchmark":
|
|
# on 2026-09-16 this step killed a predastore run 32s into its workload.
|
|
- name: Clear stale nodes
|
|
run: |
|
|
flock -w 3600 "${PREDA_BENCH_LOCK:-/var/lock/predastore-bench.lock}" \
|
|
pkill -KILL -f '/bin/s3d -config' || true
|
|
|
|
- name: Run fault injection
|
|
working-directory: predastore
|
|
run: |
|
|
set -euo pipefail
|
|
# events.txt carries the timeline and the FAIL line, but not the build
|
|
# output or anything Warp wrote before the harness took over. The
|
|
# harness only creates this directory, so a log placed here survives.
|
|
mkdir -p "$STRESS_RESULTS_ROOT"
|
|
make e2e-stress 2>&1 | tee "$STRESS_RESULTS_ROOT/e2e-stress.log"
|
|
|
|
- uses: actions/upload-artifact@v7
|
|
if: always()
|
|
with:
|
|
name: predastore-stress-${{ github.run_id }}
|
|
path: ${{ env.STRESS_RESULTS_ROOT }}
|
|
if-no-files-found: warn
|
|
retention-days: 90
|
|
|
|
# Under the lock for the same reason as "Clear stale nodes": by the time
|
|
# this runs the harness has released it, so another repository's benchmark
|
|
# can already have a cluster of its own up.
|
|
- name: Clean up
|
|
if: always()
|
|
run: |
|
|
flock -w 3600 "${PREDA_BENCH_LOCK:-/var/lock/predastore-bench.lock}" \
|
|
pkill -KILL -f '/bin/s3d -config' || true
|
|
rm -rf "$STRESS_RESULTS_ROOT"
|
|
|
|
# Reading a nightly meant opening every cell job, and the Actions UI does not
|
|
# separate a cell that failed from one cancelled out from under it. needs:
|
|
# every cell so this runs last; always() so a red run is the one it describes.
|
|
summary:
|
|
name: run summary
|
|
needs:
|
|
[
|
|
select,
|
|
matrix,
|
|
baremetal,
|
|
diskperf,
|
|
iso-single,
|
|
iso-multi,
|
|
performance,
|
|
stress,
|
|
]
|
|
if: always()
|
|
runs-on: ubuntu-26.04
|
|
steps:
|
|
- uses: actions/checkout@v7
|
|
with:
|
|
sparse-checkout: .github/scripts/e2e-run-summary.py
|
|
sparse-checkout-cone-mode: false
|
|
|
|
- name: Collect junit
|
|
uses: actions/download-artifact@v8
|
|
with:
|
|
pattern: junit-cell-*-${{ github.run_id }}
|
|
path: junit
|
|
continue-on-error: true
|
|
|
|
- name: Write run summary
|
|
env:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
JUNIT_ROOT: junit
|
|
run: python3 .github/scripts/e2e-run-summary.py
|