GIT_BRANCH was hardcoded to dev, so workflow_dispatch runs ignored the
ref chosen in the UI. Use github.ref_name for dispatch (and any non-cron
trigger), keep dev as the explicit target for scheduled cron runs.
Add a step before the E2E run that ensures the Ubuntu 24.04 noble cloud
image is cached at BM_ISO_CACHE. Downloads from ubuntu.com if missing,
prints cached size if present. Passes path via BM_UBUNTU_IMAGE to
bm-bootstrap.sh which SCPs it to the node for --file import.
* test(ddil): flip Scenarios A/B/C to real assertions
Replaces the t.Skip stubs landed in #204 with the assertions described in
docs/development/improvements/ddil-e2e-test-harness.md §3:
- A NATSKill — KillNATS on node3, assert ModeStandalone within 30s,
/local/instances still serves on the isolated node, majority /health
on node1+node2 stays up, witness counters on all three nodes keep
advancing, snapshot survives reconnect.
- B DaemonRestartWithoutNATS — KillNATS + RestartDaemonOnly, assert
ModeStandalone within 30s of restart, recovered instances present
with live QEMU PIDs, cluster rejoin after StartNATS. Strict pre/post
PID equality dropped — see file comment; recovery + a non-zero PID
give the same regression coverage without coupling to the daemon
unit's KillMode.
- C CleanPartition — PartitionNode peer-IP DROP, majority daemons keep
serving /health and /local/instances, isolated node3 reports
ModeStandalone with /local/* responsive, witness counter on the
partitioned side keeps advancing, heal converges and snapshot is
preserved.
Supporting changes:
- harness/daemon_client.go — daemonPort 8443→4432; 8443 is predastore,
the deployed daemon ClusterManager binds [nodes.<id>.daemon].host.
The Phase-1 stub used a placeholder that would have blocked any
/local/* assertion against the real cluster.
- harness/witness.go — export WitnessVM.Terminate so scenarios can
clean up their counter VMs from t.Cleanup.
- scenarios/main_test.go — requireLiveCluster + launchWitnesses
helpers to keep the three flipped scenarios tight.
- TEST_COVERAGE.md — A/B/C status flipped SKIPPED → ENABLED;
TestCoverageDrift continues to enforce alignment with the package.
Validation: built with -tags=e2e, dry-run scenarios pass, coverage
drift passes, preflight clean. workflow_dispatch e2e-ddil.yml against
tofu cluster is the next step.
* fix(ddil): countDropRules — single iptables -S call + filter
Phase 1's countDropRules ran `iptables -S INPUT -S OUTPUT` to grep DROPs
across both chains, but iptables -S takes a single optional chain — the
second -S errors to stderr, the combined SSH buffer poisons the grep
input, and Atoi fails ("ip" is what we saw on tofu-cluster). Hit by
TestHarnessSmoke and every flipped scenario (A/B/C) via AssertCleanState.
Switch to one `iptables -S` (lists every chain in iptables-save format),
filter for INPUT/OUTPUT lines, count `-j DROP`. `2>/dev/null` silences
sudo/iptables stderr so a noisy MOTD or warning cannot leak into stdin.
Error path now also surfaces the raw SSH output for the next time this
breaks somewhere unexpected.
* ci(e2e-ddil): export spinifex AWS creds for witness SDK
The witness EC2 client (harness/witness.go NewWitness) calls
session.NewSession with no static credentials, so the AWS SDK falls
through its default chain — env vars → shared profile → IAM role —
and on the CI runner none of those exist. Result on the live cluster:
"NoCredentialProviders: no valid providers in chain. Deprecated."
launches every Scenario A/B/C attempt straight to t.Fatal at
launchWitnesses.
spx admin init writes [spinifex] aws_access_key_id /
aws_secret_access_key to the operator's ~/.aws/credentials on node1
during bootstrap. Pull them over SSH and export as
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY so the SDK's env-var step
resolves before it gets to the empty profile/IAM checks. Hard-fail
the step if either field is missing — better than silently watching
every scenario t.Fatal on the same line.
* fix(ddil): CI green-up — instance-type discovery, gotestsum, EC2 envelope
Three fixes to take the DDIL E2E suite from "scenarios flipped" to actually
green on the tofu cluster. mulga-siv-56.
1. Witness auto-discovers the smallest registered instance type
(DescribeInstanceTypes, sort by memory then vCPUs). The hard-coded
t2.micro default returned InvalidInstanceType on every Scenario A/B/C
launch because the cluster doesn't register that type. DDIL_WITNESS_-
INSTANCE_TYPE stays as an operator override.
2. e2e-ddil.yml uses gotestsum so the Actions UI shows per-test PASS/FAIL/
SKIP lines instead of a JSON wall. results.json artifact preserved for
future flake-rate dashboarding.
3. GenerateEC2ErrorResponse emits the real EC2 query-API envelope
<Response><Errors><Error>... instead of the IAM-shaped <ErrorResponse>.
aws-sdk-go v1's ec2query handler rejects the latter with
SerializationError and drops the code, leaving Go callers without an
awserr.Code() to inspect. New SDK round-trip test in gateway_test.go
guards the regression.
* ci(e2e-ddil): install gotestsum from prebuilt tarball
`go install gotest.tools/gotestsum@v1.12.0` failed on the self-hosted
runner because v1.12.0 pulls golang.org/x/tools v0.19.0, whose
tokeninternal static-assert pegs a narrow Go window (~1.22). Both older
and newer runner Go versions trip `invalid array length -delta * delta`.
A prebuilt linux_amd64 binary sidesteps toolchain coupling entirely
without changing the gotestsum version pin.
* ci(e2e-ddil): collect cluster logs on failure
Witness `findHost` is currently failing with "could not locate QEMU host
for i-XXX across 3 nodes" — daemon reports running but no QEMU fork is
visible on any node. Cluster gets destroyed during teardown, so logs
have to be pulled before that runs.
Adds a failure-only step that ssh's each node and bundles
journalctl spinifex-*, process listings, and state dirs into the
ddil-results artifact for offline triage.
* fix(ddil): support tap-mode QEMU networking in witness harness
findHost no longer extracts hostfwd port — production launches go through
auto-allocated ENI on the OVS bridge and emit `-netdev tap,...`, never
hostfwd. Witness was timing out across all 3 nodes.
Discover guest IP via DescribeInstances.PrivateIpAddress (populated by
PrepareRunInstances on ENI auto-create) and tunnel SSH through the host
node to guestIP:22. Host routes to the tap bridge subnet, so the relay
works without runner-side bridge routing.
WitnessVM.SSHPort → WitnessVM.GuestIP. hostfwdPortRE removed.
Closes mulga-siv-57.
* fix(ddil): SSH witness via public IP, drop unreachable tunnel path
tofu-cluster bootstraps with --external-mode=pool which forces
dev_networking=false → no QEMU hostfwd dev NIC and no host route to the
ENI's PrivateIpAddress (tap on OVS br-int, 172.31.0.0/16 invisible to
host kernel). Previous tunnel-via-host fix could find the host but
couldn't dial the guest.
Switch to the path shell e2e already uses: default subnet has
MapPublicIpOnLaunch=true, daemon auto-allocates an EIP from the external
pool, and the runner reaches that IP via the same WAN subnet as the
cluster nodes. No host relay required, survives Scenario C (partition
only DROPs peer cluster IPs, not the runner's source IP).
Workflow now scp's ~/.ssh/spinifex-key from the primary node (the
counterpart to the public material bootstrap-install.sh imports) and
exports DDIL_GUEST_SSH_KEY so the harness can auth as ubuntu against
cloud-init's authorized_keys.
WitnessVM.GuestIP → WitnessVM.PublicIP. hostSigner / dialHost removed.
Continues mulga-siv-57.
* fix(ddil): tunnel guest SSH through host node
EIP from MapPublicIpOnLaunch sits on the cluster's external pool subnet,
which is OVN-routed — reachable from any cluster node, not from the
off-cluster runner sitting on the DDIL_NODES WAN subnet.
Restore the host-jump pattern: SSH the hosting node (already known from
findHost), dial publicIP:22 over that connection, run a guest SSH
handshake over the tunnel. Symmetric with how shell e2e reaches guests
from inside a node.
Continues mulga-siv-57.
* fix(ddil): wait for witness sshd before baseline read
Previous run reached the guest EIP via the host tunnel but hit
'Connection refused' — sshd hadn't bound yet. waitForRunning only
checks EC2 state, which flips on QEMU start, not cloud-init completion.
The OVN dnat_and_snat rule for the auto-allocated EIP is also written
asynchronously, so the tunnel can land before either layer is ready.
awaitGuestSSH polls the host-tunnelled SSH handshake to publicIP:22 for
up to 90s, retrying on transient transport errors (refused/reset/EOF/
timeout/no-route) and failing fast on auth errors so a bad key surfaces
immediately. Called once after resolvePublicIP, before the baseline
counter read.
Continues mulga-siv-57.
* fix(ddil): case-insensitive transient SSH error match
awaitGuestSSH's retry matcher used lowercase substrings, but x/crypto/ssh
emits "ssh: rejected: connect failed (Connection refused)" with capital
C — strings.Contains returned false and the helper bailed on first
attempt instead of polling. Last run failed in 49s, not the 90s budget.
Lowercase the error before matching, and add "connect failed" as the
canonical signal for "jump host dial through tunnel failed".
Continues mulga-siv-57.
* fix(ddil): default guest SSH user to ec2-user
Shell E2E SSHes the same ami-ubuntu-* images as ec2-user
(tests/e2e/run-multinode-e2e.sh:650). DDIL harness defaulted to ubuntu
and hit 'unable to authenticate, attempted methods [none publickey]' —
authorized_keys is provisioned under the AMI default account, which is
ec2-user, not ubuntu.
Continues mulga-siv-57.
* fix(ddil): wait for witness-counter.service to seed /var/lib/counter
awaitGuestSSH passed at sshd-up, but cloud-init's runcmd that enables
witness-counter.service runs in the final cloud-init stage — after sshd
binds. Baseline read landed in the window where sshd accepts but
ExecStartPre hasn't created /var/lib/counter:
cat: /var/lib/counter: No such file or directory
Collapse awaitGuestSSH + readCounter into awaitBaseline that polls the
full path (tunnel + SSH handshake + cat) and retries on the transport
errors plus "no such file or directory". Single 2-minute budget covers
cloud-init final stage on tofu-cluster hardware.
Continues mulga-siv-57.
* fix(systemd): drop Wants=spinifex-nats from daemon unit
Wants= reactivates a stopped unit on `systemctl restart`, so a deliberately
killed spinifex-nats was getting pulled back up the moment the daemon
restarted — masking the DDIL Tier 1 standalone-startup path and breaking
the Scenario B assertion (mulga-siv-58). Cold-boot coverage stays intact:
spinifex.target's Wants= activates every spinifex unit on boot. After=
remains for ordering and is non-blocking when NATS is absent.
scripts/test-ddil-phase1.sh hit the same trap and worked around it with a
runtime drop-in plus sentinel file; the fix here makes that workaround
unnecessary for any caller that restarts the daemon while NATS is stopped.
* feat(daemon): peer-health probe drives standalone mode on partition
Mode() is now derived from two atomic signals — natsConnected (NATS client
TCP link, flipped by the existing disconnect/reconnect callbacks) and
peersReachable (any peer daemon's /health responded in the most recent
probe cycle). Reports cluster iff both are up; standalone otherwise.
The peer-health probe is the missing signal for DDIL Scenario C: an iptables
DROP between peer IPs blocks NATS cluster-routing traffic but leaves each
node's local NATS client TCP connection healthy, so nats.Conn.IsConnected()
keeps returning true and onNATSDisconnect never fires. The probe dials each
peer's /health over the cluster network with a 2s timeout on a 2s ticker;
worst-case detection of total partition is ~6s on a 3-node cluster, well
inside the 30s Scenario C budget. Single-node clusters keep peersReachable
permanently true (nothing to probe) so the derived mode flips to cluster as
soon as the local NATS connect succeeds.
The probe runs from startLocal so the partition signal exists even before
(or in the absence of) a successful NATS connect. It does not touch NATS or
JetStream, so the §1e Tier 1 invariant asserted by
assertNoClusterServicesInitialised is unaffected.
Tests:
- TestMode_RequiresBothSignals — every nats×peers combination.
- TestProbePeersOnce_FlipsOnHealthChange — deterministic probe-only test.
- TestProbePeerHealth_RejectsBadAddr — fails closed on missing addresses.
- TestMonitorPeerReachability_EndToEnd — ticker-driven false→true.
Closes mulga-siv-58.
* fix(daemon): peer probe falls back to local daemon port
`spx admin init` only renders `[nodes.<self>.daemon]` in spinifex.toml;
remote-node blocks carry just `host = "<ip>"` with no daemon sub-table.
So `peer.Daemon.Host` was always empty in a deployed cluster and the
probe failed closed, leaving peersReachable permanently false.
Tofu-cluster run 25665158964 surfaced this as a Scenario A regression:
after StartNATS, natsConnected flipped true on reconnect but Mode()
stayed standalone because the derived signal required peersReachable.
`a_nats_kill_test.go:75: node3 did not return to cluster mode … (still
reporting another mode)`.
Add peerDaemonPort(p) that prefers an explicit peer Daemon.Host port
when present and otherwise reuses the local daemon's port (clusters are
symmetric — every node binds the same port the template renders). New
TestPeerDaemonPort_FallbackToSelf covers both branches plus the no-port-
available case; TestProbePeerHealth_RejectsBadAddr is updated to clear
self config so it still asserts the fail-closed path.
* fix(daemon): SIGTERM preserves VMs + KillMode=process on the unit
DDIL Scenario B asserts that `systemctl restart spinifex-daemon`
preserves every guest VM so the new daemon can reattach via the
persisted local state file. The previous setupShutdown SIGTERM handler
called vmMgr.StopAll(), gracefully QMP-powering-down every guest before
the daemon exited — ~90s wall-clock and zero surviving QEMU on the
other side. CI run 25666369604 surfaced this as
`recovered instance i-... has no live QEMU PID`.
Two changes restore the Tier 1 contract:
- setupShutdown no longer calls StopAll. It still flips d.shuttingDown
so crash/restart handlers bail out, persists state, unsubscribes from
NATS, and exits. VMs are only stopped when coordinated cluster
shutdown drains them — handleShutdownDrain still calls StopAll and
flips the flag before the SIGTERM path runs. Operator-visible
behaviour change: `systemctl stop spinifex-daemon` leaves guest VMs
running. To stop VMs, use the cluster-shutdown topics or terminate
via the AWS API.
- The systemd unit gains `KillMode=process` so systemd only signals the
daemon's main PID. With the default control-group mode, QEMU
children were killed when systemd tore down the daemon's cgroup —
defeating the SIGTERM change above. `TimeoutStopSec=15` bounds the
new fast-exit grace window.
Also nil-guards natsConn.Close in the shutdown path: when a daemon
restarts while NATS is unreachable (Scenario B), natsConn is nil and
the close would panic.
Mulga-siv-58 (reopened).
* fix(vm): QEMU stdout/stderr to shared log file (SIGPIPE survives daemon restart)
QEMU stdout/stderr were piped to daemon-side scanner goroutines.
When the daemon exits (e.g. `systemctl restart spinifex-daemon`),
the pipe read ends close; QEMU's next write triggers SIGPIPE and
the guest dies — defeating DDIL Scenario B's "daemon restart
preserves running instances" guarantee.
Open `<runtimeDir>/qemu.log` and assign it to cmd.Stdout/cmd.Stderr
instead. The FD is dup'd into QEMU at fork, so closing it on the
daemon side after Start() leaves QEMU writing into the file
indefinitely. POSIX O_APPEND makes concurrent QEMU writes
interleave atomically per line, so a single shared log is safe and
avoids per-instance clutter in RuntimeDir.
Trade-off: QEMU lines no longer land in journalctl -u
spinifex-daemon. Operators inspect <runtimeDir>/qemu.log directly
(path logged at instance start under the qemu_log field).
mulga-siv-58 (DDIL Scenario B iteration 4)
* fix(daemon): pin XDG_RUNTIME_DIR=/run/spinifex; revert SIGPIPE workaround
Root cause of Scenario B's "no live QEMU PID" was not SIGPIPE — QEMU
ignores SIGPIPE (SIG_IGN set early in QEMU init). It was systemd's
`PrivateTmp=yes` on the daemon unit combined with `utils.pidPath()`
falling through to `os.TempDir()` (=/tmp):
- spinifex-daemon HOME=/var/lib/spinifex/spinifex, so
`$HOME/spinifex` (=/var/lib/spinifex/spinifex/spinifex) does not
exist and pidPath() returns /tmp.
- Every daemon start gets a fresh private /tmp namespace. QEMU stays
in the OLD namespace after the daemon dies (mount namespaces aren't
changed by reparenting). The new daemon's /tmp is empty.
- `/local/instances` reads `<pidPath>/<id>.pid`; in the new namespace
the file doesn't exist, so PID=0.
Fix: set `Environment=XDG_RUNTIME_DIR=/run/spinifex` on the daemon
unit. /run/spinifex is host-wide tmpfs (declared via tmpfiles.d,
already in ReadWritePaths) and survives daemon restart. pidPath()
short-circuits on XDG_RUNTIME_DIR, so QEMU pidfiles, QMP sockets, and
RuntimeDir() all resolve to the stable path. Both daemon incarnations
and QEMU children see the same files.
Revert iteration 4's pipe→file QEMU log change (commit 8a9a6f9f).
SIGPIPE was never killing QEMU, and the file-redirect dropped QEMU
lines from `journalctl -u spinifex-daemon` for no gain.
mulga-siv-58 (DDIL Scenario B iteration 5)
* feat(daemon): siv-60 heal reconcile — phase 1 (probe-heal trigger) (#211)
* feat(daemon): wire reconcileOnHeal on peer-probe heal edge (siv-60 phase 1)
Scenario C heal triggered zero resync: the local NATS client stayed
connected to its loopback server throughout the partition, so
onNATSReconnect never fired. The peer-probe false→true transition was
the only heal signal but had no resync action wired to it.
Adds reconcileOnHeal, called from both edges and coalesced via a
reconciling atomic.Bool so the two callbacks landing on the same heal
don't double-push. For phase 1 the body is the existing WriteState
(re-asserts our per-node row in the instance-state KV). Phase 2 will
extend it to pull KV → merge → push for the stopped/terminated buckets
and re-fire per-node bootstrap.
Plan: docs/development/feature/ddil-heal-reconcile.md
* feat(daemon): siv-60 phase 2 — idempotent default-VPC bootstrap re-fire on heal
Calls ensureDefaultVPCInfrastructure() from reconcileOnHeal after the
WriteState push. Idempotent: each branch (default VPC lookup, IGW
attach, default-SG create, default route insert) checks for existing
infrastructure before creating. Covers the failure mode where
startCluster ran during partition and the initial creates returned
errors against an unreachable NATS — without a heal-time retry the
cluster would be stuck without a default VPC until the next daemon
restart.
Scope note: the bead's "pull KV → merge → push" assumed the daemon
cached peer-node running state, stopped instances, terminated
instances, and VPC/IGW/SG state locally. The codebase audit (see plan
doc) confirms it doesn't — every consumer reads its KV bucket fresh
per request, and vmMgr only tracks VMs running on this node. The
bootstrap re-fire is the one heal-time action that has a real
in-memory target. Watchdog sweep is unnecessary because
vm.Manager.CrashHandler is event-driven from cmd.Wait().
* fix(daemon): drop ensureDefaultVPCInfrastructure from reconcileOnHeal (siv-60 follow-up) (#212)
* fix(daemon): drop ensureDefaultVPCInfrastructure from reconcileOnHeal
DDIL Scenario C regressed at the witness-SSH step after siv-60
landed in #211. Root cause: the first peer-probe tick on multi-node
startup flips peersReachable false→true (its zero value is false), so
reconcileOnHeal fires concurrently with startCluster's own
ensureDefaultVPCInfrastructure call. The Describe→Create→Attach
sequence is not cluster-wide singleton, so multiple daemons + double
calls per daemon raced and produced duplicate IGW attachments,
corrupting default-VPC routing — which broke the witness VM's tunnel
SSH path once Scenario C's partition was installed.
Phase 1 (the heal trigger + coalescing) is the meaningful fix and
stays. Phase 2's bootstrap re-fire needs leader election or a
debounce against the startup-time path before being re-introduced;
leaving an inline NOTE so the next attempt has the context.
Failed run: actions/runs/25706585637 (Scenario C, attempts 1+2).
Last green DDIL run: 25671044714 (pre-merge, SHA 40e3d847).
* fix(daemon): init peersReachable=true to suppress startup heal edge
DDIL Scenario C kept flaking at attempt-1 even after dropping the
bootstrap re-fire. Remaining trigger: peersReachable starts as the
zero value (false) on multi-node clusters, and the first probePeersOnce
tick flips it false→true. That counts as a "heal" edge under siv-60
phase 1, so reconcileOnHeal fires at boot — concurrent with
startCluster's bootstrap — perturbing timing in a way that the
harness's Scenario C exposes (witness sshd handshake EOF mid-test).
Initialise peersReachable optimistically. Mode() also requires
natsConnected (starts false), so the optimistic peer init can never
falsely report cluster mode — but it does eliminate the spurious
startup edge. Real partition→heal transitions still fire the trigger
exactly as designed: true→false on partition, false→true on heal.
---------
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
Separate from normal e2e. Checks out spinifex, checks out mulga scripts
(sparse: iso-builder + baremetal), builds spx + test tar via
bm-build-artifacts.sh, builds PXE artifacts (squashfs/initrd/vmlinuz)
via iso-builder with persistent cache on banksia, then runs
run-baremetal-e2e.sh against the single bare-metal node.
* feat: add GPU discovery package (mulga-992)
Adds spinifex/gpu package with sysfs-based GPU enumeration, IOMMU group
member lookup, and a PCI device ID model table. Vendor-specific enrichment
via nvidia-smi is attempted opportunistically and falls back to the built-in
table when unavailable. All discovery and IOMMU logic is covered by unit tests
using mock sysfs trees.
* feat: add VFIO bind/unbind and GPU pool manager (mulga-993)
Adds gpu.BindVFIO/UnbindVFIO using sysfs driver_override for per-device
driver switching, and gpu.Manager which owns the claim/release lifecycle
for the GPU pool including full IOMMU group binding, rollback on partial
failure, and marking entries unavailable when rebind fails after VM
termination.
* feat: add G5 instance type generator with GPU model support (mulga-994)
- Add GPUModel type, NVIDIAa10g constant, and GPUModelForVendorDevice lookup
- Add generateGPUTypes to emit g5.* types with GpuInfo populated
- Add IsGPUType helper for consumers filtering GPU-capable types
- Update DetectAndGenerate to accept gpuModels []GPUModel parameter
* feat: integrate GPU passthrough into daemon lifecycle (mulga-995)
- Add gpu_passthrough config flag; discover GPUs and build manager at startup
- Gate g5 instance types on GPU pool availability via canAllocateCount
- Claim GPU (VFIO bind) in LaunchInstance; inject vfio-pci QEMU device
- Release GPU (VFIO unbind, rebind to host driver) in stopInstance
- Re-claim GPU on daemon restart when QEMU process is still alive
- Add UseUEFI / OVMF support for x86_64 VMs (optional, falls back to SeaBIOS)
- Fix Claim to use stored OriginalDriver when device is already vfio-pci bound
* feat: add GPU fields to NodeStatus and spx top nodes (mulga-996)
- Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes)
- Populate from gpuManager in handleNodeStatus
- Show GPU (used/total) column in spx top nodes; "-" when no GPUs present
* feat: Ubuntu 24.04 GPU AMI build pipeline (mulga-997)
Extends build-system-image.sh to support Ubuntu alongside Alpine.
Key differences handled: partition table (parted resizepart p1),
bind mounts for DKMS (/proc /sys /dev), apt vs apk, systemctl vs
rc-update. DISTRO defaults to "alpine" for backward compatibility.
Adds ubuntu-gpu.conf manifest and ubuntu-gpu-setup.sh chroot script
that installs nvidia-headless-550-server with DKMS pre-built against
a pinned linux-image-generic kernel, blacklists nouveau, and prevents
unattended-upgrades from invalidating the pre-built module.
* Fixed GPU passthrough and added test script
* feat: always-probe GPU at startup, surface capability in NodeStatusResponse
- Add gpuProbeResult struct and probeGPU() which runs at daemon startup
regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU
groups, and vfio_pci module presence.
- Store gpuProbe on Daemon struct; split NewDaemon into probe (always)
and activate (opt-in) phases.
- Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse
so operators can see hardware state before enabling passthrough.
- Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but
passthrough not active, 'used/total' = passthrough active.
* feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001)
- Export GenerateGPUTypes for use by the reload path
- Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place
(preserves references held by instanceService) and updates NATS
subscriptions without full daemon restart
- Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP
in a context-aware loop; setupShutdown no longer intercepts SIGHUP
- Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal,
handles false→true (re-probe + activate) and true→false (refuse if
instances running, otherwise deactivate) transitions
* feat: spx admin gpu status/enable/disable commands (mulga-1000)
- admin.SetGPUPassthrough: Go implementation of the idempotent TOML write
(replaces the inline Python heredoc in spx-test-gpu.sh)
- spx admin gpu status: shows GPU hardware, IOMMU/vfio state, passthrough
activation, and advertised g5 instance types via NATS NodeStatusResponse
- spx admin gpu enable: checks prerequisites, writes TOML, sends SIGHUP,
polls until daemon confirms GPUPassthrough=true
- spx admin gpu disable: refuses while GPU instances are running, otherwise
writes TOML, sends SIGHUP, polls until confirmed
- spx-test-gpu.sh: replace Python TOML heredoc with spx admin gpu enable
* fixed admin
* fix: filter Intel iGPU from passthrough pool, use lspci for consumer GPU names
Discover() now skips non-NVIDIA/AMD vendors so Intel iGPUs never enter
the passthrough pool. enrichFromLspci() uses the system PCI ID database
via lspci -vmm to populate the model name for any card not in the
hardcoded table, without needing nvidia-smi or driver tools.
* fix: grant CAP_SYS_ADMIN and vfio device access to spinifex-daemon
GPU passthrough requires writing to /sys/bus/pci/drivers/vfio-pci/unbind
on instance stop. The kernel gates this on CAP_SYS_ADMIN regardless of
file permissions. QEMU subprocesses also need char-vfio rw access to
open /dev/vfio/N for the passed-through device.
* fixing gpu bining permissions
* working state of gpu passthrough on phyiscal box
* installer grub image fixes
* made gpu-passthrough more user facing
* fixed banner
* fixed banner
* banner fix
* gpu passthrough docs
* preflight tweak
* vpcd fixes to set br-wan to promisc for public IP DHCP processes
* fixed vpcd br-wan promisc and veth pair
* ssh keygen fix
* fixes after running /review-branch
* feat: re-wire GPU claim and device injection after vm.Manager refactor
GPU passthrough claim (VFIO bind) and QEMU device injection were lost
when dev's vm.Manager refactor (#185/#186) moved StartInstance/
CreateQMPClient into vm/lifecycle.go. Re-add both in their correct
new homes:
- daemon_handlers_instance.go: claim GPU before vmMgr.Run() in both
RunInstances and StartStoppedInstance handlers; release on failure
- vm/lifecycle.go startQEMU(): inject vfio-pci device with x-vga flag
derived from instance.GPUXVGAEnabled when GPUPCIAddress is set
* fix: Create spinifex user ~/.ssh and keys in firstboot
---------
Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
Scheduled runs were using github.ref_name, which equals the default branch
(main) — so the nightly was testing main. Hardcode GIT_BRANCH=dev so the
build-install-artifacts.sh + tofu provisioning target dev's HEAD instead.
GitHub always loads scheduled-workflow files from the default branch, so
this change must stay in sync on main for the schedule to use it.
- workflows: add 'dev' to push/PR triggers (spinifex.yml, ui.yml, e2e.yml)
- workflows: drop "Resolve Mulga Branch" fallback step; mulga is single-branch
so checkout@v6 uses the default branch implicitly
- dependabot: route gomod/github-actions/npm PRs to dev
- diff-coverage.sh: pick origin/dev as base for feat branches, origin/main for dev
- iac/proxmox-template: default git_branch to dev for cluster builds
Refs docs/development/improvements/reintroduce-dev-branch.md
* preflight tweak
* fix: pin gateway LRP to link-local, end intra-cluster ARP collision
Every VPC's IGW gateway-LRP claimed pool.GatewayIP || pool.RangeStart on
attach. Multi-VPC clusters all GARP'd the same .160 with three different
deterministic MACs on the shared br-ext localnet, flapping the upstream
ARP cache and dropping 100% of inbound traffic to dnat_and_snat external
IPs (mulga-siv-26).
handleIGWAttach and reconcileIGW now unconditionally set the gateway
router port to 169.254.0.1/30. The default route's OutputPort already
pins the LRP, so the WAN nexthop need not be on-subnet. Pool IPs stay
allocated only for per-VM dnat_and_snat external addresses, which are
unique by construction.
Self-heal: RetrofitAllGatewayPortNetworks runs at startup and rewrites
stale Networks in place via the new UpdateLogicalRouterPort path. Needed
because Reconcile gates IGW work on "ext switch missing", so a cluster
shipped with the buggy CIDR never re-enters reconcileIGW.
Regression guards: TestTopologyHandler_IGWAttach_MultiVPC_NoIPCollision
and TestReconcileIGW_RewritesStaleGatewayPortNetworks. Existing tests
updated to assert link-local and OutputPort=&gwPortName. Log field
gateway_ip renamed to lrp_network to signal the semantic shift.
* fix: rewrite utils.HashMAC to SHA-256 / 46-bit output
The old 24-bit rolling hash with literal class prefix had birthday-paradox
collision probability of 1% at ~580 ids and 50% at 4096 — well within the
range of a single deployment's resource population (LRPs, ENIs, dev/mgmt
NICs). On a shared L2 segment, two clusters could trivially produce
duplicate MACs, breaking neighbour discovery and DHCP.
Switch HashMAC to SHA-256 across the entire input, taking the low 6 bytes
and forcing the IEEE 802 reserved bits (bit0=0 unicast, bit1=1 LAA) on the
first octet. Output entropy goes from 24 bits to 46 bits — 1% collision
boundary moves from ~580 to ~1.2M ids.
Drop the redundant prefix arg: it was no longer the literal output prefix
under the new scheme, just folded into the hash input. Class disambiguation
moves to the caller, where it's explicit. Three of four callers had nothing
to disambiguate (eni-, subnet-, gw-vpc- ids are already unique); dev and
mgmt NICs share instance.ID and now compose "dev:" / "mgmt:" tags.
Also consolidate vpcd.generateMAC, which was a local duplicate of the
broken algorithm rather than a HashMAC wrapper, into a HashMAC delegation.
Plan: docs/development/bugs/vpcd-mac-collisions-shared-l2.md
Bead: mulga-siv-27
* fix: pin HashMAC first octet to 0x02
Previous impl forced only the IEEE 802 reserved bits (bit0=0 unicast,
bit1=1 LAA) on the first octet, leaving the upper 6 bits hash-derived.
Technically valid LAA unicast, but produced MACs like 4a:.. and fe:..
that visually encroach on vendor OUI space and tripped CI/ops checks
expecting all generated MACs to start with 02:.
Pin byte 0 to literal 0x02; hash bytes 1-5 from sha256[0:5]. Output
entropy drops from 46 to 40 bits — birthday-paradox 1% collision at
~150k ids, still 256x the old 24-bit bound (~580).
* chore(e2e): dump cross-chassis OVN state on guest SSH failure
Adds chassis registrations, port_binding claims, gateway_chassis rows,
ovs-vsctl topology, Geneve tunnel state, and br-int tunnel flows to
dump_guest_ssh_diagnostics so we can pinpoint where the cross-chassis
dataplane stalls (cr-gw chassis vs remote chassis with the VM bound).
* fix: gate vpcd startup reconcile behind NATS KV leader election
N vpcds in a multi-node cluster all hit Get-then-Create on Logical_Router
with no atomicity, producing duplicate rows that ovn-nbctl rejects with
"Multiple logical routers named '...'. Use a UUID."
Add AcquireReconcileLeader using NATS KV CAS (kv.Create on a single key
in spinifex-vpcd-reconcile bucket). Only the elected vpcd runs Reconcile,
ReconcileFromKV, and the two retrofit passes — others Subscribe and rely
on the vpcd-workers queue group for runtime events. MaxAge=60s on the
bucket caps recovery time when a leader crashes mid-reconcile.
Falls through (elected=true) when KV is unavailable so first-boot does
not deadlock on a missing JetStream.
mulga-siv-29
* fix(elbv2): force mgmt host route for internal-scheme LBs (mulga-siv-30)
Single-node CI e2e Phase 3b ALB internal hangs in 'provisioning' for 270s
because the lb-agent inside the ALB VM has no return path to AWSGW.
The gateway-URL decision tree in daemon.wireLBAgentConfig picks advertiseIP
for single-node and deliberately skips adding the /32 mgmt host route — the
WAN-return-path concern in the comment is real but applies only to
internet-facing LBs, which have an EIP and an OVN dnat_and_snat pair that
gives reply traffic a way home. Internal LBs have no EIP / no SNAT, so when
the agent dials advertiseIP the reply targets the VM's private subnet IP from
the host's WAN with no route back.
Make lbVMUserData scheme-aware. New helper resolveMgmtRoute returns the
existing (MgmtRouteGateway, MgmtRouteTarget) when set (multi-node case) and
falls back to (MgmtBridgeIP, AdvertiseIP) when scheme is internal — only
internal LBs need the /32 route, so internet-facing single-node still gets
no bootcmd and the host-initiated WAN path stays clean.
Daemon now exposes MgmtBridgeIP and AdvertiseIP unconditionally so the
fallback has the inputs it needs.
* fix(daemon): stub sudoCommand in TestSetupComputeNode_ValidatesArgs (mulga-siv-31)
The test was calling real SetupComputeNode("chassis-test", ...) which shelled
out via sudoCommand to set external_ids:system-id=chassis-test on the host's
live OVS. On a dev box with OVS installed this overwrote the persistent UUID
(owned by openvswitch-switch and re-applied only on reboot), so vpcd's
discoverChassis filtered out the legitimate SBDB chassis row as 'stale local'
and crashlooped — taking out all VPC ops including ALB ENI provisioning.
Bind sudoCommand to a var so tests can swap it. Stub it in the test to return
exec.Command("/bin/false") and assert SetupComputeNode returns an error. The
test's intent (signature + error-path coverage) is preserved; nothing on the
host is touched.
* chore(vpcd): log DHCP Acquire concurrency + duration (mulga-siv-32)
Diagnostics-only change to surface whether nightly e2e cell-3 / cell-8
DORA timeouts correlate with concurrent Acquire calls on the same
bridge. Tracks per-bridge in-flight count and wallclock duration of each
Acquire; logs WARN on failure with both fields, DEBUG on success.
No behavioural change.
* ci(e2e-nightly): dump cluster journals on failure (mulga-siv-32)
run-multinode-e2e.sh dumps node journals via its own EXIT trap, but
run-cert-e2e.sh / run-lb-e2e.sh / reboot-e2e do not. When those scripts
fail the cluster is destroyed in Teardown without surfacing any daemon
logs, blocking root-cause analysis.
Add a workflow step that runs after the test scripts but before
Teardown when a previous step failed, ssh'ing to each node and pulling
the last 200 lines of every spinifex-* unit. Best-effort: skips
gracefully when inventory or nodes are unreachable.
Diagnostic-only; no behavioural change to the test path.
* fix(setup): mask dhcpcd.service to stop it racing vpcd's DHCP client (mulga-siv-32)
Debian 13 (Trixie) auto-enables a standalone dhcpcd.service when
dhcpcd-base is installed. It binds br-wan and races vpcd's in-process
nclient4 client for upstream OFFERs, draining the lab DHCP pool with
duplicate leases and causing intermittent DORA timeouts in the e2e
public-IP path (worst on cell-8 real-multi/dhcp/debian-13).
The ISO installer already masks the unit at install time
(cmd/installer/install/install.go:316). The binary install path used by
e2e nightly missed it, so debian hosts installed via setup.sh ran with
both DHCP clients live. Mirror the ISO mask here.
Ubuntu hosts are unaffected (dhcpcd-base does not auto-enable
dhcpcd.service on Noble).
* ci(e2e-nightly): also dump dhcpcd.service state on failure (mulga-siv-32)
Confirms or rules out the dhcpcd-vs-vpcd race hypothesis. Pulls the
dhcpcd.service journal alongside the spinifex-* units, plus
systemctl is-enabled / is-active and the DHCP client port (UDP 68)
listener so we can see directly whether dhcpcd is running and bound
to br-wan when an Acquire times out.
---------
Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>
* feat: scaffold DDIL E2E harness (cluster, SSH, daemon client)
Phase 1 of the DDIL E2E test harness (mulga-siv-5.1). Adds the
tests/e2e/ddil/ package tree with the three harness primitives required
by every sibling scenario/fault bead:
- harness/cluster.go — Cluster, Node, ClusterFromEnv, NodeFromEnv
- harness/ssh.go — typed SSH transport backed by golang.org/x/crypto/ssh,
interface so dry-run and tests can stub it
- harness/daemon_client.go — thin HTTPS wrapper over /local/status,
/local/instances (placeholder types until daemon-local-autonomy lands),
plus /health for scaffolding validation
Also adds doc.go and README.md (run instructions + env-var contract). All
harness files carry //go:build e2e so default builds skip them.
Plan: docs/development/improvements/ddil-e2e-test-harness.md §0, §1.
* feat: add DDIL harness fault injection and link profiles
Phase 2 of the DDIL E2E test harness (mulga-siv-5.2). Adds the fault
injection and netem primitives consumed by Scenarios A–F:
- harness/fault.go — PartitionNode (per-peer iptables DROP with
orchestrator-SSH sanity check), HealNode, KillNATS, StartNATS,
RestartDaemonOnly, WaitForMode (polls /local/status for typed
DaemonMode enum)
- harness/netem.go — ApplyNetem (tc qdisc replace, idempotent),
ClearNetem (idempotent clear)
- harness/profiles.go — LinkProfile struct plus LAN, WAN, LTEDegraded,
SATCOM, HFData, Flapping constants (unvalidated approximations per
plan §2 table; validation tracking lives in TEST_COVERAGE.md)
All files carry //go:build e2e. Flapping profile is defined but
ApplyNetem rejects it — flap orchestration lands with the scenarios
that need it.
Plan: docs/development/improvements/ddil-e2e-test-harness.md §1, §2.
* feat: add DDIL harness witness VM helpers
LaunchWitnessVM launches a counter VM via the cluster's AWS gateway,
locates its QEMU host via ps, and captures a baseline counter reading.
ReadCounter/AssertProgressed tunnel SSH through the host's QEMU
hostfwd port to verify the workload progressed across a fault. README
documents the new DDIL_WITNESS_* and DDIL_GUEST_SSH_* env vars.
* feat: add DDIL harness snapshot and state assertions
TakeSnapshot / AssertPreserved compare /local/instances pre and post
fault, failing if IDs disappear or a running/pending instance regresses
to a terminal state. LocalStateRevision / AssertMonotonic read the
local state file revision from /local/status so scenarios can detect
the daemon rewinding its persisted state.
* feat: add DDIL harness reset, retry, and quarantine
ResetAllNodes brings every node back to a clean baseline (iptables -F,
tc netem clear on any interface carrying one, spinifex-nats and
spinifex-daemon started) then waits for cluster health — daemon /health
200 plus NATS routez peer count matching len(nodes)-1, mirroring the
shell suite's verify_nats_cluster predicate.
AssertCleanState is a pre-scenario baseline check that recovers a dirty
state via ResetAllNodes instead of failing outright, since a dirty
baseline usually means a previous scenario's cleanup was skipped.
Run wraps each scenario with quarantine handling (DDIL_QUARANTINED
skip) and a single reset-and-retry on failure. First-attempt failures
are not hidden; the retry is a diagnostic aid rather than a flake mask.
bd: mulga-siv-5.5
* feat: DDIL scenarios skeleton (6x t.Skip + TestMain + coverage drift)
Phase 1 of mulga-siv-5 (DDIL E2E harness): all six scenarios land as
TestScenario<L>_... stubs that t.Skipf on their hardening-epic dependency.
TestMain installs a SIGINT/SIGTERM trap that ResetAllNodes before
exiting; DDIL_DRY_RUN=1 short-circuits cluster setup and logs the planned
scenarios. TestCoverageDrift AST-parses sibling *_test.go files and
reconciles them against the DDIL table in TEST_COVERAGE.md so Phase 2/3
flips cannot silently diverge doc from code. TestHarnessSmoke covers the
AssertCleanState + ResetAllNodes round trip for the "full run on real
cluster" acceptance bullet.
* feat: add DDIL E2E CI workflow (workflow_dispatch only)
Phase 1 CI wiring for mulga-siv-5. Mirrors e2e_real_multi's
provision/bootstrap/teardown shape against the same ci-multi runner and
tofu-cluster, but runs the Go harness on the runner (outside the
cluster) so its SSH path survives the peer-traffic iptables rules that
Scenario C installs on a cluster node. Triggers are workflow_dispatch
only — nightly cron and on-main-merge come in mulga-siv-5.8 once ≥4
scenarios are asserting. results.json (go test -json) uploads as an
artifact so flake-rate visibility can be built on gh run view +
structured output, per the design doc.
Julian Sommer