This reverts commit f9c9f0b38d.
#580 switched setup.sh to read the northstar :53 listener and zones from
northstar.toml, so the resolved stage now defers at install time (before
formation) and must be re-run afterward. Only firstboot re-runs it; the
manual setup.sh + spx admin init install and CI never do, so those hosts
get no host DNS and cluster-zone names (ELB/EKS) fail to resolve from the node.
Reverting restores the pre-#580 behaviour: systemd-resolved is configured
at install time from the WAN IP, with no post-formation re-run needed. This
reintroduces the original gap where resolvconf hosts get no host DNS. The
proper fix configures host DNS from spx admin init/join for both resolver
managers (mulga-x30yv).
* feat: add OVSDB RAFT cluster primitives to setup-ovn.sh
Add --db-cluster-local-addr / --db-cluster-remote-addr flags so a
management node can run the OVN NB/SB databases as a native OVSDB RAFT
cluster instead of a single non-clustered ovn-central. When a local addr
is given, write a shared OVN_CTL_OPTS to /etc/default/ovn-central (NB
RAFT 6643 / SB RAFT 6644) and restart the per-DB units directly, since
the ovn-central aggregator is ExecStart=/bin/true and won't restart its
children. Empty remote addr creates the cluster (init node); a set remote
addr joins it. NB/SB client set-connection runs only on the create node
(RAFT replicates it cluster-wide).
The single-node --management path is unchanged when no cluster addrs are
supplied, so dev boxes and existing single-node clusters keep working.
--ovn-remote already passes through verbatim, so a comma-separated SB
endpoint list works for compute nodes; documented in usage.
* feat: OVN client multi-endpoint failover for RAFT clusters
Accept a comma-separated OVN NB/SB endpoint list so vpcd can target a
clustered DB. config.ParseEndpoints splits the list (trimming whitespace,
dropping empties); live.go Connect passes one libovsdb WithEndpoint per
endpoint, so the client tries them in order, fails over on disconnect, and
follows RAFT leader redirects. Single-endpoint strings still yield one
endpoint, so single-node dev is unchanged.
ovn-sbctl --db= already accepts the raw comma list, so the SB CLI
consumers (discoverChassis, gateway_claim) need no change.
* feat: write clustered OVN NB/SB endpoints across the RAFT quorum
spx admin init/join now emit comma-separated OVN NB/SB endpoint lists for
the first three quorum nodes (formation.BuildOVNDBAddrs) instead of loopback
or a single leader address, so vpcd fails over across the cluster.
firstboot defers OVN bring-up to the provisioning controller when it owns
formation (SkipFormation): a standalone single-node .db would otherwise block
the controller's clustered create/join.
* test: add multinode OVN RAFT cluster + failover e2e; document clustered topology
ovn_raft_test.go asserts NB/SB cluster/status reports a 3-server quorum with
one leader on every DB node, then stops the NB leader's ovn-central and proves
an NB write still lands via a surviving endpoint.
Docs: vpc-networking gains the multi-endpoint vpcd config and a RAFT
cluster-status debugging section; network-connections lists the 6643/6644 RAFT
ports.
Add SkipFormation bool to firstboot.Config. When set, the cluster
formation block (spx admin init/join) is replaced with a no-op echo,
leaving formation to the provisioning controller.
Wired through: autoinstall reads SPINIFEX_SKIP_FORMATION=1 from env,
install.go passes it to firstboot.Config. spinifex-init.sh parses the
kernel cmdline param and exports it. boot-handler.go hardcodes
SPINIFEX_SKIP_FORMATION=1 so all bare-metal CI nodes skip formation —
bm-bootstrap.sh always runs spx admin init/join with the correct
--nodes N from nodes.json, enabling proper multi-node formation.
In the PXE path SPINIFEX_INSTALL_CALLBACK arrives on the iPXE kernel
cmdline but was never threaded into the firstboot script, so firstboot
never curled the controller's /done endpoint. The flag stayed on disk,
the node re-entered the install loop on the next PXE boot.
Fix: read SPINIFEX_INSTALL_CALLBACK from the live-environment env in
toFirstbootConfig and embed a best-effort curl into the generated
firstboot script, fired once after the success marker is written.
Interactive and USB-stick installs are unaffected (env var absent
-> callbackBlock is empty, no curl injected).
* feat: add GPU discovery package (mulga-992)
Adds spinifex/gpu package with sysfs-based GPU enumeration, IOMMU group
member lookup, and a PCI device ID model table. Vendor-specific enrichment
via nvidia-smi is attempted opportunistically and falls back to the built-in
table when unavailable. All discovery and IOMMU logic is covered by unit tests
using mock sysfs trees.
* feat: add VFIO bind/unbind and GPU pool manager (mulga-993)
Adds gpu.BindVFIO/UnbindVFIO using sysfs driver_override for per-device
driver switching, and gpu.Manager which owns the claim/release lifecycle
for the GPU pool including full IOMMU group binding, rollback on partial
failure, and marking entries unavailable when rebind fails after VM
termination.
* feat: add G5 instance type generator with GPU model support (mulga-994)
- Add GPUModel type, NVIDIAa10g constant, and GPUModelForVendorDevice lookup
- Add generateGPUTypes to emit g5.* types with GpuInfo populated
- Add IsGPUType helper for consumers filtering GPU-capable types
- Update DetectAndGenerate to accept gpuModels []GPUModel parameter
* feat: integrate GPU passthrough into daemon lifecycle (mulga-995)
- Add gpu_passthrough config flag; discover GPUs and build manager at startup
- Gate g5 instance types on GPU pool availability via canAllocateCount
- Claim GPU (VFIO bind) in LaunchInstance; inject vfio-pci QEMU device
- Release GPU (VFIO unbind, rebind to host driver) in stopInstance
- Re-claim GPU on daemon restart when QEMU process is still alive
- Add UseUEFI / OVMF support for x86_64 VMs (optional, falls back to SeaBIOS)
- Fix Claim to use stored OriginalDriver when device is already vfio-pci bound
* feat: add GPU fields to NodeStatus and spx top nodes (mulga-996)
- Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes)
- Populate from gpuManager in handleNodeStatus
- Show GPU (used/total) column in spx top nodes; "-" when no GPUs present
* feat: Ubuntu 24.04 GPU AMI build pipeline (mulga-997)
Extends build-system-image.sh to support Ubuntu alongside Alpine.
Key differences handled: partition table (parted resizepart p1),
bind mounts for DKMS (/proc /sys /dev), apt vs apk, systemctl vs
rc-update. DISTRO defaults to "alpine" for backward compatibility.
Adds ubuntu-gpu.conf manifest and ubuntu-gpu-setup.sh chroot script
that installs nvidia-headless-550-server with DKMS pre-built against
a pinned linux-image-generic kernel, blacklists nouveau, and prevents
unattended-upgrades from invalidating the pre-built module.
* Fixed GPU passthrough and added test script
* feat: always-probe GPU at startup, surface capability in NodeStatusResponse
- Add gpuProbeResult struct and probeGPU() which runs at daemon startup
regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU
groups, and vfio_pci module presence.
- Store gpuProbe on Daemon struct; split NewDaemon into probe (always)
and activate (opt-in) phases.
- Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse
so operators can see hardware state before enabling passthrough.
- Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but
passthrough not active, 'used/total' = passthrough active.
* feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001)
- Export GenerateGPUTypes for use by the reload path
- Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place
(preserves references held by instanceService) and updates NATS
subscriptions without full daemon restart
- Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP
in a context-aware loop; setupShutdown no longer intercepts SIGHUP
- Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal,
handles false→true (re-probe + activate) and true→false (refuse if
instances running, otherwise deactivate) transitions
* feat: spx admin gpu status/enable/disable commands (mulga-1000)
- admin.SetGPUPassthrough: Go implementation of the idempotent TOML write
(replaces the inline Python heredoc in spx-test-gpu.sh)
- spx admin gpu status: shows GPU hardware, IOMMU/vfio state, passthrough
activation, and advertised g5 instance types via NATS NodeStatusResponse
- spx admin gpu enable: checks prerequisites, writes TOML, sends SIGHUP,
polls until daemon confirms GPUPassthrough=true
- spx admin gpu disable: refuses while GPU instances are running, otherwise
writes TOML, sends SIGHUP, polls until confirmed
- spx-test-gpu.sh: replace Python TOML heredoc with spx admin gpu enable
* fixed admin
* fix: filter Intel iGPU from passthrough pool, use lspci for consumer GPU names
Discover() now skips non-NVIDIA/AMD vendors so Intel iGPUs never enter
the passthrough pool. enrichFromLspci() uses the system PCI ID database
via lspci -vmm to populate the model name for any card not in the
hardcoded table, without needing nvidia-smi or driver tools.
* fix: grant CAP_SYS_ADMIN and vfio device access to spinifex-daemon
GPU passthrough requires writing to /sys/bus/pci/drivers/vfio-pci/unbind
on instance stop. The kernel gates this on CAP_SYS_ADMIN regardless of
file permissions. QEMU subprocesses also need char-vfio rw access to
open /dev/vfio/N for the passed-through device.
* fixing gpu bining permissions
* working state of gpu passthrough on phyiscal box
* installer grub image fixes
* made gpu-passthrough more user facing
* fixed banner
* fixed banner
* banner fix
* gpu passthrough docs
* preflight tweak
* vpcd fixes to set br-wan to promisc for public IP DHCP processes
* fixed vpcd br-wan promisc and veth pair
* ssh keygen fix
* fixes after running /review-branch
* feat: re-wire GPU claim and device injection after vm.Manager refactor
GPU passthrough claim (VFIO bind) and QEMU device injection were lost
when dev's vm.Manager refactor (#185/#186) moved StartInstance/
CreateQMPClient into vm/lifecycle.go. Re-add both in their correct
new homes:
- daemon_handlers_instance.go: claim GPU before vmMgr.Run() in both
RunInstances and StartStoppedInstance handlers; release on failure
- vm/lifecycle.go startQEMU(): inject vfio-pci device with x-vga flag
derived from instance.GPUXVGAEnabled when GPUPCIAddress is set
* fix: Create spinifex user ~/.ssh and keys in firstboot
---------
Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
Extract all inline systemd unit and drop-in content from install.go and
firstboot.go into a dedicated cmd/installer/systemd package for editability.
Bash scripts remain inline in firstboot.go.
Also copies the GRUB splash image from the live ISO into the installed
system's /boot/grub/ and sets GRUB_BACKGROUND so the branded background
appears in the post-install GRUB, not just the installer GRUB.
Remove the duplicated chown/ownership/symlink block from the firstboot
script. setup.sh (called with INSTALL_SPINIFEX_TARBALL) already handles
service users, directory ownership, sudoers, systemd unit installation,
and enabling spinifex.target — the same steps as the online curl|bash
install.
Firstboot now follows the documented install flow directly:
1. setup.sh --tarball (air-gapped equivalent of curl|bash)
2. setup-ovn.sh --management
3. spx admin init
4. systemctl start spinifex.target
Also reduce wait-for-nats.sh timeout from 90s to 30s per reviewer feedback.
- partUUID: error on empty blkid output instead of writing UUID= to fstab
- installBootloader: fail if both EFI and BIOS targets fail; track efiErr
- machine-id fallback: write at mode 0600 so systemd can commit on first boot
- Run: defer unmount so partitions are always cleaned up on failure
- copyRootfs: verify critical paths exist after rsync before continuing
- resolveNIC: skip virtual/bridge interfaces (docker0, virbr0, ovs-system)
- EjectAndReboot: use syscall.Reboot instead of exec(reboot) which needs D-Bus
- ui.Run: return error on TTY open failure instead of silent fallback
- firstboot script: trap EXIT to always self-disable, preventing retry loops
- enableUnit/enableBannerUnit: deduplicate into enableSystemdUnit; check Remove
- runAdminBanner: promote service restart failure to slog.Error
- parseNodeConf: log non-ENOENT read failures
- Add spinifex login user password via chpasswd in install.go (same
password as root, collected once in TUI)
- Copy /root/.aws/ to /home/spinifex/.aws/ in firstboot so the operator
can use the AWS CLI without sudo after first boot
- Fix getty drop-ins to be instance-specific (tty1/ttyS0 only), leaving
tty2+ immediately available as rescue terminals
- Remove autologin — node now requires password at login prompt
- Pre-start OVS/OVN in firstboot and poll NB DB up to 120s to eliminate
setup-ovn.sh timeout races on slow physical hardware
- Tighten per-service ownership in firstboot (each service user owns only
its own /var/lib/spinifex subdirectory)
- Add LAN bridge as a non-critical systemd unit so a missing LAN cable
can never stall network-online.target or firstboot
- Write dhcpcd.conf with short br-lan timeout; add networking.service
drop-in to treat exit 1 as success for optional interface failures
- Disable IPv6 on management bridges to silence dhcpcd journal noise
- Fix WAN/LAN enter-key and ESC navigation logic in TUI
- Fix LAN NIC cursor to skip WAN NIC correctly
- Replace single network screen with separate WAN and LAN screens
- WAN/LAN IPs now live on Linux bridges (br-wan, br-lan) over the physical
NICs, not on the NICs directly — makes OVS/OVN veth attachment SSH-safe
- Add DHCP/static toggle on both screens (DHCP is the default)
- Add DNS nameserver input for static mode (comma-separated, default hint
1.1.1.1 / 8.8.8.8 shown as placeholder)
- Detect WiFi NICs via /sys/class/net/<name>/wireless; show SSID and
password fields when a WiFi adapter is selected
- LAN screen shows WAN NIC greyed/unavailable; cursor skips over it
- LAN screen only shown when 2+ NICs are detected
- Geneve encap IP uses br-lan address when LAN NIC present, else br-wan
- Remove --macvlan/--wan-iface from firstboot; setup-ovn.sh auto-detects
br-wan as a Linux bridge and uses veth mode
- Pin both WAN and LAN NIC names to MAC via a single udev rules file
- node.conf writes MANAGEMENT_IFACE=br-wan (bridge, not physical NIC)
- banner runs After=spinifex.target so services are settled before banner writes
- switch systemctl restart → try-restart (safe if target not yet active)
- append banner to /etc/motd with sentinel instead of overwriting, preserving
the Debian disclaimer; sentinel ensures re-runs replace cleanly
- remove /etc/issue write — getty shows it pre-login which is wrong
- add getty@.service.d drop-in so all getty instances wait for banner before
showing login prompt; prevents service messages scrolling during login
- fix Makefile to prefer locally built spx/installer binaries over system PATH
- add `spx admin banner --boot-check` subcommand to spx CLI
- reads management IP from the NIC at runtime (not from node.conf)
- if IP has changed, updates /etc/spinifex/node.conf and restarts spinifex.target
- writes banner to /etc/issue (pre-login console) and /etc/motd (SSH MOTD)
- banner instructs user to cat ~/.aws/credentials (standard AWS CLI location)
- update firstboot.go to use spx-based banner unit; remove writeBannerScript
and old spinifex-banner.sh entirely
- new banner unit runs Before=spinifex.target so IP is current before services start
- chown /etc/spinifex and /var/lib/spinifex to spinifex:spinifex after
spx admin init so services running as the spinifex user can read configs
- add Environment=HOME=/root to firstboot unit to suppress $HOME warning
- increase wait-for-nats.sh timeout from 30s to 90s for slow startup
- banner now writes to both /dev/tty1 and /etc/motd (visible over SSH)
- add systemd.show_status=1 to GRUB cmdline for per-service boot feedback
- add --removable to grub-install EFI so fallback path EFI/BOOT/BOOTX64.EFI
is written (required for UTM and bare metal without NVRAM boot entries)
- add root password screen and manual NIC entry to installer TUI
- tee spx admin init output to /root/spinifex-credentials.txt with 0600 perms
Adds cmd/installer with three packages:
- ui: huh-based TUI collecting disk, network, node, and cluster config
- install: sequential install steps (partition, debootstrap, chroot, bootloader)
- firstboot: oneshot systemd service that runs setup-ovn and spx admin init/join on first real boot
Supports both single-node init and multi-node join flows.
Julian Sommer