mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-23 04:36:40 +00:00
69404b38ce
* fix(installer): emit kernel cmdline to ttyS0 alongside tty0 (mulga-siv-33)
GRUB_CMDLINE_LINUX in /etc/default/grub only listed console=tty0, so the
installed kernel printed nothing on serial after the GRUB menu. Headless
installs (CI, racks with serial-only IPMI, remote-dev SSH-to-QEMU) saw a
black screen forever and had no path to debug boot failures.
Append console=ttyS0,115200 — the last `console=` on the cmdline becomes
the system console, so ttyS0 wins on serial-only hardware while tty0
still receives output on local displays.
Plan: docs/development/bugs/iso-public-ip-boot-race.md
* fix(ipam): widen vpc.dhcp.acquire retry budget to 60 s (mulga-siv-34)
ObtainDHCPLease retried only 5 attempts × 1 s before returning a hard
error. On a cold ISO boot, NATS comes up well before vpcd's reconcile
leader-election + dhcp.Manager subscribe (typical 5–30 s, pathological
~60 s during full reconcile), so daemon's NewExternalIPAM races vpcd's
dhcp.acquire subscription and fails permanently — d.externalIPAM stays
nil for the daemon's lifetime, daemon_handlers_instance.go silently
skips all auto-public-IP allocation, and instances launch with no
public IP, no dnat_and_snat, and no NAT event ever published.
Bump maxAttempts to 60 with 1 s spacing; expose both as package vars
so tests can shrink the budget for fast timeout-path coverage. Pure
widening of the existing retry loop — no behaviour change on the
success path, only the worst-case wallclock when vpcd never comes up.
The dominant cause of slow vpcd starts (mulga-siv-33: systemd-networkd
not enabled in the ISO → veth pair vanishes on reboot → vpcd
crashloops) is fixed alongside this. A 60 s budget fits comfortably
above all observed healthy-startup times.
Plan: docs/development/bugs/iso-public-ip-boot-race.md
* fix(setup-ovn): declare Linux WAN bridge as NetDev so networkd-managed veth resolves Bridge= on reboot (mulga-siv-33)
The veth-mode reboot persistence drop-ins added in mulga-998.b were
incomplete: setup-ovn.sh wrote `15-spinifex-veth-wan.network` with
`Bridge=br-wan`, but never declared br-wan itself to networkd.
br-wan is created by `networking.service` (ifupdown via
`/etc/network/interfaces`, generated by `install.go writeNetworkConfig`),
not networkd. networkd's `Bridge=` directive requires the target to be a
known NetDev — when it isn't, networkd silently logs
`br-wan NetDev could not be found, ignoring assignment` and the veth
ends up with no master. vpcd's bridge-mode sanity check then fails on
every reboot.
Fix: also write `14-spinifex-br-wan.netdev` declaring br-wan as
`Kind=bridge`. networkd matches the existing kernel bridge by name+kind
(the `Failed to create netdev: File exists` log is harmless), and
`Bridge=br-wan` resolves on next boot. ifupdown still owns DHCP and the
IP — networkd just needs the NetDev declaration to enslave the veth.
The write is gated to avoid clobbering operator-managed networkd config:
if any other `.netdev` under `/etc/systemd/network`,
`/run/systemd/network`, or `/usr/lib/systemd/network` already declares
the WAN bridge by name, the write is skipped with a log line pointing
at the operator file. Cleanup hook also removes the new file when
switching away from veth mode.
Verified end-to-end via QEMU repro: clean install + first boot + reboot,
veth-wan-br master persists across reboots, vpcd starts cleanly.
* fix(vpcd/dhcp): enable AF_PACKET promisc on bridge for DORA (mulga-siv-33)
Real-world DHCP servers commonly ignore the BOOTP BROADCAST flag and
reply with a unicast OFFER addressed to the synthetic per-tenant chaddr
(e.g. 02:f3:...). A Linux bridge floods unknown unicast to slave ports
but does NOT pass it up to the bridge netdev unless IFF_PROMISC is set,
so vpcd's AF_PACKET socket on br-wan misses the OFFER and DORA times
out at 15s ("unable to receive an offer: context deadline exceeded").
Dev/CI didn't hit this:
- CI uses spx_wan_mode=static — no DORA via vpcd.
- QEMU dev DHCP servers (SLIRP/dnsmasq) honor the BROADCAST flag, so
their replies are L2 broadcast which the bridge always passes up.
Replace the three nclient4.New(iface, ...) call sites in Acquire/Renew/
Release with nclient4.NewWithConn(...) wrapping a custom AF_PACKET
DGRAM conn that calls SetPromiscuous(true). Promisc is per-socket
(per-DORA, ~15s) and refcounted by the kernel — auto-clears on close.
* revert(vpcd/dhcp): drop AF_PACKET promisc wrapper, restore nclient4.New
ecfbd4f swapped nclient4.New for openPromiscBridgeConn + NewWithConn to
catch unicast OFFER replies that bypass the bridge netdev. On dev hosts
where upstream honors the BOOTP BROADCAST flag, the wrapper regressed
DORA — public-IP allocation failed where main works. Reverting until
the promisc path is validated against both broadcast- and unicast-OFFER
servers.
* fix(setup-ovn): mark br-int/br-ext Unmanaged for systemd-networkd (mulga-siv-37)
Fresh ISO install on Trixie+systemd-networkd left the OVS internal netdevs
admin-down: networkd has no matching .network so it tags them as unmanaged
but never brings them up. OVS dataplane works but ovn-controller flow
programming and any tooling that probes link state misbehaves.
Drop /etc/systemd/network/05-spinifex-ovs-internal.network with
Unmanaged=yes so OVS itself owns the link state — `ovs-vsctl add-br`
already runs the netdev up at create time.
* fix(vpcd): allocate gateway LRP IP from pool in centralized NAT (mulga-siv-36)
In veth/macvlan modes the gateway LRP carries all egress, so it must hold
a WAN-subnet IP — otherwise the upstream router silently drops ARP for
the WAN nexthop (RFC 826) and the default route never resolves. The
hardcoded 169.254.0.1/30 was correct for distributed NAT but regressed
centralized NAT (introduced under mulga-siv-26).
Add gw_lrp_range_start/end to ExternalPool. In centralized mode at IGW
attach the gateway LRP gets a fresh IP from that range, persisted as
external_ids:spinifex:gateway_ip on the LRP for idempotent reconcile and
retrofit. Direct mode keeps link-local. Per-VM EIP allocator skips the
gw range; ValidatePoolConfig rejects overlap with the EIP range.
Adds TestTopologyHandler_IGWAttach_CentralizedNATAllocatesGwLrpIP and
extends RetrofitAllGatewayPortNetworks to migrate stale link-local LRPs
to allocated pool IPs at startup.
* fix(vpcd): drop link-local fallback, auto-derive gw_lrp_range (mulga-siv-36)
In centralized NAT (veth/macvlan) the gateway LRP must hold a WAN-subnet
IP — link-local cannot ARP-resolve the upstream nexthop, so the upstream
router silently drops egress. Hardware deployment confirmed the previous
fallback to 169.254.0.1/30 was always reached when gw_lrp_range was not
explicitly configured.
Auto-derive a 16-IP range at the top of the WAN subnet (broadcast - 16
.. broadcast - 1) when gw_lrp_range is unset; shift below RangeStart on
overlap with the per-VM EIP range; clamp inside the subnet; guard the
upstream gateway IP. IGW attach now errors out instead of falling back
to link-local when allocation fails — surfaces misconfiguration loudly.
Adds focused unit tests for gwLrpRange, ValidatePoolConfig and
nextAvailableExternalIP gw-lrp skip; updates the multi-VPC and
PoolWithGatewayIP regression tests.
* feat(vpcd): DHCP-acquire gateway LRP IPs (mulga-siv-38)
Per-VPC OVN gateway LRPs now obtain their WAN-subnet IP via vpc.dhcp.acquire
when the pool is source="dhcp", instead of carving from a static gw_lrp_range.
The upstream DHCP server becomes the single arbiter of LAN address space; a
cluster grows by N nodes without operator-side range carving in spinifex.toml.
- Consolidate ObtainDHCPLease/ReleaseDHCPLease into the dhcp package as
RequestAcquire/RequestRelease so vpcd and the daemon-side ExternalIPAM
share one helper. handlers/ec2/vpc keeps its existing ENI/EIP DHCP path.
- TopologyHandler dispatches by pool source: DHCP pools call DHCPManager
via NATS (client-id "gw-lrp-<vpcId>", idempotent on retry); static pools
keep the siv-36 gw_lrp_range / auto-derive allocator. Direct mode and
no-pool deployments still use link-local.
- IGW detach issues vpc.dhcp.release so the upstream server reclaims the
IP. Best-effort — server-side expiry is the safety net.
- Tests: TestTopologyHandler_IGWAttach_DHCPGwLrpIP (3 VPCs share one DHCP
pool, distinct IPs, idempotent attach, release on detach),
IGWAttach_DHCPAcquireFailureAborts (no link-local fallback when DHCP
errors), TestPrefixFromMask. Existing static / auto-derive tests
preserved.
* fix(vpcd): subscribe DHCPManager before reconcile to fix boot race (mulga-siv-38)
DHCPManager.Subscribe ran AFTER ReconcileFromKV / RetrofitAllGatewayPortNetworks,
so retrofit's vpc.dhcp.acquire requests for existing IGWs hit "no responders" and
burned the 60-attempt × 1 s cold-boot retry budget per VPC before failing. On a
dev box with one bootstrap VPC the symptom is a 60 s vpcd startup hang followed
by "allocate gw LRP IP failed", and the bootstrap IGW topology missing from OVN
until the next reconcile pass.
Move dhcpManager.Subscribe ahead of the two reconcile passes so the subscription
is live before any topology code can fan out an acquire.
* fix(vpcd): also subscribe DHCPManager before bootstrap reconcile pass
Previous fix moved DHCPManager.Subscribe ahead of ReconcileFromKV but
missed Pass 1 (the bootstrap-config Reconcile that runs even earlier).
On a dev box with a source="dhcp" pool and a bootstrap IGW, that earlier
pass still hit "no responders" and burned the 60-retry cold-boot budget
before the Manager came up. Move both reconcile passes after Subscribe.
* fix(vpcd): RFC 2131 §4.1 DORA exponential backoff (mulga-siv-39)
DHCPManager.Acquire previously did one DISCOVER and gave up after 15s
because the outer ctx.WithTimeout matched the per-attempt timeout —
nclient4's WithRetry(3) could never fire. Real DHCP servers (consumer
routers, ISC dhcpd, dnsmasq) silently drop OFFERs under load; the spec
mandates retransmission with exponential backoff, and a single lost
DISCOVER was killing legitimate boots.
Move retransmission into DHCPManager.acquireWithBackoff with a
4s/8s/16s/32s schedule (±1s jitter, ~60s worst case). NClient4Client
becomes single-attempt; manager owns retry policy. Default
acquireTimeout lifts from 15s to 90s (sum of schedule + slack).
* fix(vpcd): retry NewDHCPManager on JetStream-not-ready (mulga-siv-40)
NewDHCPManager creates the spinifex-dhcp-leases KV bucket via
utils.GetOrCreateKVBucket → js.CreateKeyValue with no retry. After the
mulga-siv-38 launchService reorder, this runs before any other JetStream
user, so on multi-node cold-boot vpcd hits "JetStream system temporarily
unavailable" and exits before cluster quorum forms. Pseudo-multinode CI
calls start_node_services sequentially, so node2's vpcd failure aborts
the script before node3 ever launches and quorum then never forms.
Wrap NewDHCPManager in newDHCPManagerWithRetry (500ms→10s, 5 min cap,
ctx-cancellable) mirroring daemon.initJetStream so vpcd waits as
patiently as the daemon for the cluster to come up.
* fix(daemon): idempotent DetachVolume QMP path (mulga-siv-41)
DetachVolume's QMP sequence was not retry-safe. First call: device_del
succeeds, blockdev-del fails with "Node nbd-vol-X is in use" because NBD
client teardown / in-flight I/O still hold the block node. Handler bails
ServerInternal "leaving volume state intact" but the guest device is
actually gone. AWS-CLI retries: device_del returns DeviceNotFound,
handler bails again. After 2 retries aws cli exits 254 — Single-Node
Phase 8 Step 3 reproducibly fails this way.
Phase 1: treat DeviceNotFound on device_del as already-detached, log and
continue. Detach is now idempotent across retries.
Phase 2: tryBlockdevDel polls blockdev-del up to 20×detachDelay on "in
use" errors so the I/O drain can complete instead of failing the whole
detach. Other QMP errors fall through immediately as before.
732 lines
26 KiB
Go
732 lines
26 KiB
Go
/*
|
|
Copyright © 2026 Mulga Defense Corporation
|
|
|
|
This program is free software: you can redistribute it and/or modify
|
|
it under the terms of the GNU Affero General Public License as published by
|
|
the Free Software Foundation, either version 3 of the License, or
|
|
(at your option) any later version.
|
|
|
|
This program is distributed in the hope that it will be useful,
|
|
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
GNU Affero General Public License for more details.
|
|
|
|
You should have received a copy of the GNU Affero General Public License
|
|
along with this program. If not, see <https://www.gnu.org/licenses/>.
|
|
*/
|
|
|
|
package install
|
|
|
|
import (
|
|
"bufio"
|
|
"fmt"
|
|
"io"
|
|
"log/slog"
|
|
"os"
|
|
"os/exec"
|
|
"path/filepath"
|
|
"strings"
|
|
"syscall"
|
|
"time"
|
|
|
|
"github.com/mulgadc/spinifex/cmd/installer/firstboot"
|
|
"github.com/mulgadc/spinifex/cmd/installer/systemd"
|
|
)
|
|
|
|
const (
|
|
mountRoot = "/mnt/spinifex-install"
|
|
efiPart = mountRoot + "/boot/efi"
|
|
)
|
|
|
|
// Run executes all installation steps in order. It is intentionally sequential
|
|
// and explicit — each step is visible in logs so failures are easy to diagnose.
|
|
func Run(cfg *Config) error {
|
|
// The live environment may not have /sbin or /usr/sbin in PATH. Set it
|
|
// explicitly so exec.Command's LookPath finds system binaries like grub-install.
|
|
_ = os.Setenv("PATH", "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin")
|
|
|
|
// Unmount unconditionally on exit so a failed step never leaves partitions
|
|
// mounted in the live environment, which would cause a retry to double-mount.
|
|
defer func() {
|
|
_ = run("umount", efiPart)
|
|
_ = run("umount", mountRoot)
|
|
}()
|
|
|
|
steps := []struct {
|
|
name string
|
|
fn func() error
|
|
}{
|
|
{"partition disk", func() error { return partitionDisk(cfg.Disk) }},
|
|
{"format partitions", func() error { return formatPartitions(cfg.Disk) }},
|
|
{"mount partitions", func() error { return mountPartitions(cfg.Disk) }},
|
|
{"copy rootfs", copyRootfs},
|
|
{"write fstab", func() error { return writeFstab(cfg.Disk) }},
|
|
{"install spinifex", func() error { return installSpinifex(cfg) }},
|
|
{"write network config", func() error { return writeNetworkConfig(cfg) }},
|
|
{"write firstboot service", func() error { return firstboot.Write(mountRoot, cfg.toFirstbootConfig()) }},
|
|
{"install bootloader", func() error { return installBootloader(cfg.Disk) }},
|
|
{"install CA cert", func() error { return installCACert(cfg) }},
|
|
}
|
|
|
|
for _, s := range steps {
|
|
slog.Info("installer", "step", s.name)
|
|
if err := s.fn(); err != nil {
|
|
return fmt.Errorf("step %q: %w", s.name, err)
|
|
}
|
|
}
|
|
|
|
slog.Info("installation complete")
|
|
promptRemoveUSB()
|
|
return reboot()
|
|
}
|
|
|
|
func partitionDisk(disk string) error {
|
|
// GPT table with three partitions:
|
|
// p1: 1MiB BIOS Boot Partition — required for grub-install i386-pc on GPT
|
|
// p2: 512MiB EFI System Partition
|
|
// p3: remainder as root (ext4)
|
|
if err := run("parted", "--script", disk,
|
|
"mklabel", "gpt",
|
|
"mkpart", "bios_boot", "1MiB", "2MiB",
|
|
"set", "1", "bios_grub", "on",
|
|
"mkpart", "ESP", "fat32", "2MiB", "514MiB",
|
|
"set", "2", "esp", "on",
|
|
"mkpart", "root", "ext4", "514MiB", "100%",
|
|
); err != nil {
|
|
return err
|
|
}
|
|
// Force the kernel to re-read the partition table and wait for udev to
|
|
// create the partition device nodes. Without this, mkfs.fat in the next
|
|
// step may race and fail with "No such file or directory" on /dev/sda2 —
|
|
// the kernel has accepted the new layout but /dev hasn't been populated
|
|
// yet. Trixie's udev seems slower at this than Bookworm's was.
|
|
return waitForPartitions(disk)
|
|
}
|
|
|
|
// waitForPartitions ensures the EFI and root partition device nodes exist
|
|
// after parted creates them. It runs partprobe (kernel re-read) and
|
|
// udevadm settle (wait for queued events), then polls /dev for the files.
|
|
func waitForPartitions(disk string) error {
|
|
// Best-effort: partprobe failure isn't fatal — udev may still pick up
|
|
// the change from the BLKRRPART ioctl that parted itself issued.
|
|
if err := run("partprobe", disk); err != nil {
|
|
slog.Warn("partprobe failed, continuing", "disk", disk, "err", err)
|
|
}
|
|
if err := run("udevadm", "settle", "--timeout=10"); err != nil {
|
|
slog.Warn("udevadm settle failed, continuing", "err", err)
|
|
}
|
|
efi, root := partitionPaths(disk)
|
|
deadline := time.Now().Add(15 * time.Second)
|
|
for _, part := range []string{efi, root} {
|
|
for {
|
|
if _, err := os.Stat(part); err == nil {
|
|
break
|
|
}
|
|
if time.Now().After(deadline) {
|
|
return fmt.Errorf("partition device %s did not appear within timeout — kernel/udev did not pick up new partition table", part)
|
|
}
|
|
time.Sleep(200 * time.Millisecond)
|
|
}
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func formatPartitions(disk string) error {
|
|
efi, root := partitionPaths(disk)
|
|
if err := run("mkfs.fat", "-F32", efi); err != nil {
|
|
return err
|
|
}
|
|
return run("mkfs.ext4", "-F", root)
|
|
}
|
|
|
|
func mountPartitions(disk string) error {
|
|
_, root := partitionPaths(disk)
|
|
if err := os.MkdirAll(mountRoot, 0o755); err != nil {
|
|
return err
|
|
}
|
|
if err := run("mount", root, mountRoot); err != nil {
|
|
return err
|
|
}
|
|
if err := os.MkdirAll(efiPart, 0o755); err != nil {
|
|
return err
|
|
}
|
|
efi, _ := partitionPaths(disk)
|
|
return run("mount", efi, efiPart)
|
|
}
|
|
|
|
// copyRootfs copies the live squashfs environment onto the target disk using
|
|
// rsync. This is the air-gapped alternative to debootstrap — all packages are
|
|
// already embedded in the ISO so no network access is required.
|
|
func copyRootfs() error {
|
|
if err := run("rsync", "-aHAX", "--delete", "--info=progress2",
|
|
"--exclude=/proc",
|
|
"--exclude=/sys",
|
|
"--exclude=/dev",
|
|
"--exclude=/run",
|
|
"--exclude=/tmp",
|
|
"--exclude=/cdrom",
|
|
"--exclude=/mnt",
|
|
"--exclude=/etc/machine-id",
|
|
"--exclude=/var/lib/dbus/",
|
|
"--exclude=/etc/openvswitch/",
|
|
"--exclude=/var/lib/openvswitch/",
|
|
"--exclude=/var/lib/dhcpcd/",
|
|
"--exclude=/lost+found",
|
|
"--exclude=/boot/efi",
|
|
"/", mountRoot+"/",
|
|
); err != nil {
|
|
return err
|
|
}
|
|
|
|
// Verify critical paths exist before proceeding. rsync exits 0 on ENOSPC
|
|
// for individual file writes on some filesystems, which would produce a
|
|
// partial rootfs that boots into a panic.
|
|
critical := []string{
|
|
filepath.Join(mountRoot, "bin/bash"),
|
|
filepath.Join(mountRoot, "lib/systemd/systemd"),
|
|
filepath.Join(mountRoot, "usr/local/bin/spx"),
|
|
}
|
|
for _, p := range critical {
|
|
if _, err := os.Stat(p); err != nil {
|
|
return fmt.Errorf("copyRootfs: critical path missing after rsync (%s): %w", p, err)
|
|
}
|
|
}
|
|
|
|
// rsync skips excluded paths entirely — recreate the virtual filesystem
|
|
// mount points that systemd expects to exist on the installed system.
|
|
mountPoints := []struct {
|
|
path string
|
|
mode os.FileMode
|
|
}{
|
|
{"proc", 0o555},
|
|
{"sys", 0o555},
|
|
{"dev", 0o755},
|
|
{"run", 0o755},
|
|
{"run/lock", 0o1777},
|
|
{"tmp", 0o1777},
|
|
{"mnt", 0o755},
|
|
}
|
|
for _, mp := range mountPoints {
|
|
if err := os.MkdirAll(filepath.Join(mountRoot, mp.path), mp.mode); err != nil {
|
|
return fmt.Errorf("create mountpoint /%s: %w", mp.path, err)
|
|
}
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func installSpinifex(cfg *Config) error {
|
|
// The rootfs copy already contains spx and spinifex-installer at
|
|
// /usr/local/bin/ — no binary copy needed. Regenerate machine-specific
|
|
// identity files so each installed node is unique.
|
|
|
|
// Bind-mount /dev, /proc, /sys into the chroot so PAM (chpasswd) and
|
|
// other chroot commands work correctly. Trixie's PAM requires /proc and
|
|
// /dev/urandom for password hashing.
|
|
if err := bindChrootMounts(); err != nil {
|
|
return err
|
|
}
|
|
defer unbindChrootMounts()
|
|
|
|
// Generate a unique machine-id from the kernel CSPRNG. We do not use
|
|
// systemd-machine-id-setup in the chroot: it reads the SMBIOS UUID via
|
|
// the bind-mounted /sys, which is identical on cloned VMs or hosts with
|
|
// absent/zeroed DMI data, and falls back to non-unique sources (MAC,
|
|
// hostname) when that also fails — producing the same ID on every node.
|
|
// /proc/sys/kernel/random/uuid is always unique and requires no chroot.
|
|
machineIDPath := filepath.Join(mountRoot, "etc/machine-id")
|
|
rawUUID, err := os.ReadFile("/proc/sys/kernel/random/uuid")
|
|
if err != nil {
|
|
return fmt.Errorf("installSpinifex: read kernel uuid: %w", err)
|
|
}
|
|
machineID := strings.ReplaceAll(strings.TrimSpace(string(rawUUID)), "-", "") + "\n"
|
|
if err := os.WriteFile(machineIDPath, []byte(machineID), 0o444); err != nil {
|
|
return fmt.Errorf("installSpinifex: write machine-id: %w", err)
|
|
}
|
|
// dbus mirrors /etc/machine-id; remove it so it is re-created from the
|
|
// new ID on first boot rather than carrying a stale value.
|
|
_ = os.Remove(filepath.Join(mountRoot, "var/lib/dbus/machine-id"))
|
|
|
|
// Hostname.
|
|
hostnamePath := filepath.Join(mountRoot, "etc/hostname")
|
|
if err := os.WriteFile(hostnamePath, []byte(cfg.Hostname+"\n"), 0o644); err != nil {
|
|
return err
|
|
}
|
|
|
|
// /etc/hosts entry for the hostname.
|
|
hosts := fmt.Sprintf("127.0.0.1\tlocalhost\n127.0.1.1\t%s\n", cfg.Hostname)
|
|
if err := os.WriteFile(filepath.Join(mountRoot, "etc/hosts"), []byte(hosts), 0o644); err != nil {
|
|
return err
|
|
}
|
|
|
|
// Set root + spinifex passwords. We invoke chpasswd from the LIVE
|
|
// installer (not via `chroot`), passing the target root with -R and
|
|
// forcing -c YESCRYPT. This deliberately bypasses PAM:
|
|
// * `chroot ... chpasswd` uses /etc/pam.d/chpasswd → common-password →
|
|
// pam_unix.so with the "obscure" option, which can return
|
|
// "Authentication token manipulation error" inside a chroot for
|
|
// reasons that are awkward to diagnose (audit subsystem, locked
|
|
// shadow entries from useradd, etc.).
|
|
// * -c YESCRYPT tells chpasswd to hash locally with libcrypt and write
|
|
// directly to <root>/etc/shadow — no PAM stack involved. YESCRYPT
|
|
// matches Trixie's ENCRYPT_METHOD so subsequent logins use the same
|
|
// algorithm.
|
|
// * -R <root> opens the target's passwd/shadow directly; no bind
|
|
// mounts of /dev/urandom or /proc are needed for the password step.
|
|
if cfg.RootPassword != "" {
|
|
if err := setShadowPassword("root", cfg.RootPassword); err != nil {
|
|
return fmt.Errorf("set root password: %w", err)
|
|
}
|
|
// The spinifex account is the default interactive login on the
|
|
// node (console + SSH). Root SSH is disabled, so this is the sole
|
|
// remote entry point. The user itself is created at rootfs build
|
|
// time (build-rootfs.sh) — here we just set its password.
|
|
if err := setShadowPassword("spinifex", cfg.RootPassword); err != nil {
|
|
return fmt.Errorf("set spinifex password: %w", err)
|
|
}
|
|
}
|
|
|
|
// Write /etc/spinifex/node.conf — read at runtime by spx admin banner
|
|
// to look up the current IP dynamically (handles IP changes after install).
|
|
// MANAGEMENT_IFACE is the bridge (br-wan), not the physical NIC.
|
|
// MANAGEMENT_IP is empty for DHCP — banner's --boot-check fills it in at boot.
|
|
nodeConf := fmt.Sprintf("MANAGEMENT_IP=%s\nMANAGEMENT_IFACE=br-wan\nNODE_HOSTNAME=%s\n",
|
|
cfg.WANAddress, cfg.Hostname)
|
|
confDir := filepath.Join(mountRoot, "etc/spinifex")
|
|
if err := os.MkdirAll(confDir, 0o755); err != nil {
|
|
return err
|
|
}
|
|
if err := os.WriteFile(filepath.Join(confDir, "node.conf"), []byte(nodeConf), 0o644); err != nil {
|
|
return err
|
|
}
|
|
|
|
// dhcpcd.conf: only apply a short DHCP timeout on the optional LAN bridge.
|
|
// br-wan keeps the dhcpcd default (30 s) so it reliably gets a lease even
|
|
// on physical switches that take time to come up. br-lan is brought up by
|
|
// spinifex-lan-bridge.service (non-critical), so failing fast there is fine.
|
|
// Writing this here (after copyRootfs) ensures the installed system always
|
|
// has the correct config regardless of what the rootfs image contains.
|
|
dhcpcdConf := "# Generated by Spinifex installer\ninterface br-lan\ntimeout 10\n"
|
|
if err := os.WriteFile(filepath.Join(mountRoot, "etc/dhcpcd.conf"), []byte(dhcpcdConf), 0o644); err != nil {
|
|
return err
|
|
}
|
|
|
|
// Mask dhcpcd.service so only ifupdown controls DHCP on the bridges.
|
|
// Trixie's dhcpcd-base ships a standalone dhcpcd.service that races with
|
|
// ifupdown's own dhcpcd invocations, causing duplicate leases or IP flapping.
|
|
if err := maskSystemdUnit(mountRoot, "dhcpcd.service"); err != nil {
|
|
slog.Warn("installSpinifex: failed to mask dhcpcd.service", "err", err)
|
|
}
|
|
|
|
return systemd.WriteNetworkingDropIn(mountRoot)
|
|
}
|
|
|
|
func writeNetworkConfig(cfg *Config) error {
|
|
// IPs live on Linux bridges (br-wan, br-lan), not on the physical NICs.
|
|
// This lets OVN/OVS wire into the bridges non-destructively via veth pairs
|
|
// without ever moving the IP off a live interface (SSH-safe).
|
|
//
|
|
// Only br-wan uses `auto` so networking.service brings it up at boot.
|
|
// br-lan deliberately omits the `auto` stanza — it is brought up by
|
|
// spinifex-lan-bridge.service *after* network-online.target, so a missing
|
|
// LAN cable or slow switch can never block networking.service or firstboot.
|
|
var b strings.Builder
|
|
b.WriteString("# Generated by Spinifex installer\nsource /etc/network/interfaces.d/*\n\nauto lo\niface lo inet loopback\n\n")
|
|
|
|
writeBridge := func(nicIface, bridgeName string, dhcp bool, addr, mask, gw string, dns []string, ssid, wifiPass string, hasGateway, autoStart bool) {
|
|
comment := strings.ToUpper(bridgeName[3:]) // "br-wan" → "WAN", "br-lan" → "LAN"
|
|
fmt.Fprintf(&b, "# %s NIC\nauto %s\niface %s inet manual\n", comment, nicIface, nicIface)
|
|
if ssid != "" {
|
|
fmt.Fprintf(&b, " wpa-ssid %s\n wpa-psk %s\n", ssid, wifiPass)
|
|
}
|
|
if !dhcp {
|
|
for _, ns := range dns {
|
|
ns = strings.TrimSpace(ns)
|
|
if ns != "" {
|
|
fmt.Fprintf(&b, " dns-nameservers %s\n", ns)
|
|
}
|
|
}
|
|
}
|
|
b.WriteString("\n")
|
|
|
|
// Bridge stanza.
|
|
if autoStart {
|
|
fmt.Fprintf(&b, "# %s bridge\nauto %s\n", comment, bridgeName)
|
|
} else {
|
|
fmt.Fprintf(&b, "# %s bridge — brought up by spinifex-lan-bridge.service\n", comment)
|
|
}
|
|
if dhcp {
|
|
fmt.Fprintf(&b, "iface %s inet dhcp\n", bridgeName)
|
|
} else {
|
|
fmt.Fprintf(&b, "iface %s inet static\n address %s\n netmask %s\n", bridgeName, addr, mask)
|
|
if hasGateway && gw != "" {
|
|
fmt.Fprintf(&b, " gateway %s\n", gw)
|
|
}
|
|
}
|
|
fmt.Fprintf(&b, " bridge_ports %s\n bridge_stp off\n bridge_fd 0\n bridge_maxwait 0\n\n", nicIface)
|
|
}
|
|
|
|
writeBridge(cfg.WANInterface, "br-wan", cfg.WANDHCPMode,
|
|
cfg.WANAddress, cfg.WANMask, cfg.WANGateway, cfg.WANDNS,
|
|
cfg.WANWiFiSSID, cfg.WANWiFiPass, true, true)
|
|
|
|
if cfg.LANInterface != "" {
|
|
writeBridge(cfg.LANInterface, "br-lan", cfg.LANDHCPMode,
|
|
cfg.LANAddress, cfg.LANMask, "", cfg.LANDNS,
|
|
cfg.LANWiFiSSID, cfg.LANWiFiPass, false, false)
|
|
}
|
|
|
|
if err := os.WriteFile(filepath.Join(mountRoot, "etc/network/interfaces"), []byte(b.String()), 0o644); err != nil {
|
|
return err
|
|
}
|
|
|
|
// Write a non-critical systemd unit for br-lan so it comes up after
|
|
// network-online.target without blocking the boot path.
|
|
if cfg.LANInterface != "" {
|
|
if err := systemd.WriteLANBridgeUnit(mountRoot); err != nil {
|
|
return fmt.Errorf("lan bridge unit: %w", err)
|
|
}
|
|
}
|
|
|
|
// Disable IPv6 on the bridge interfaces. We only use IPv4 for management
|
|
// and OVN tunnels; without this, dhcpcd logs "no IPv6 routers available"
|
|
// errors and the boot journal is noisy.
|
|
bridges := []string{"br-wan"}
|
|
if cfg.LANInterface != "" {
|
|
bridges = append(bridges, "br-lan")
|
|
}
|
|
var sysctl strings.Builder
|
|
sysctl.WriteString("# Generated by Spinifex installer — IPv6 disabled on management bridges\n")
|
|
for _, br := range bridges {
|
|
fmt.Fprintf(&sysctl, "net.ipv6.conf.%s.disable_ipv6=1\n", br)
|
|
}
|
|
sysctlDir := filepath.Join(mountRoot, "etc/sysctl.d")
|
|
if err := os.MkdirAll(sysctlDir, 0o755); err != nil {
|
|
return err
|
|
}
|
|
if err := os.WriteFile(filepath.Join(sysctlDir, "99-spinifex-network.conf"), []byte(sysctl.String()), 0o644); err != nil {
|
|
return err
|
|
}
|
|
|
|
// Pin each NIC name to its MAC address via udev so the installed system
|
|
// always uses the same interface name regardless of probe order.
|
|
udevDir := filepath.Join(mountRoot, "etc/udev/rules.d")
|
|
if err := os.MkdirAll(udevDir, 0o755); err != nil {
|
|
return err
|
|
}
|
|
var udevRules strings.Builder
|
|
for _, iface := range []string{cfg.WANInterface, cfg.LANInterface} {
|
|
if iface == "" {
|
|
continue
|
|
}
|
|
mac, err := os.ReadFile("/sys/class/net/" + iface + "/address")
|
|
if err != nil {
|
|
slog.Warn("writeNetworkConfig: could not read NIC MAC, skipping udev pin", "iface", iface, "err", err)
|
|
continue
|
|
}
|
|
fmt.Fprintf(&udevRules, "SUBSYSTEM==\"net\", ACTION==\"add\", ATTR{address}==\"%s\", NAME=\"%s\"\n",
|
|
strings.TrimSpace(string(mac)), iface)
|
|
}
|
|
if udevRules.Len() > 0 {
|
|
return os.WriteFile(filepath.Join(udevDir, "70-spinifex-net.rules"), []byte(udevRules.String()), 0o644)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func installBootloader(disk string) error {
|
|
// grub-install runs in the live environment (not chroot) using the
|
|
// grub-pc-bin and grub-efi-amd64-bin packages already present on the ISO.
|
|
// --boot-directory points at the installed system's /boot.
|
|
bootDir := filepath.Join(mountRoot, "boot")
|
|
efiDir := filepath.Join(mountRoot, "boot", "efi")
|
|
|
|
efiErr := run("grub-install",
|
|
"--target=x86_64-efi",
|
|
"--efi-directory="+efiDir,
|
|
"--boot-directory="+bootDir,
|
|
"--bootloader-id=spinifex",
|
|
"--removable",
|
|
"--recheck",
|
|
)
|
|
if efiErr != nil {
|
|
slog.Warn("installBootloader: EFI install failed", "err", efiErr)
|
|
}
|
|
if biosErr := run("grub-install",
|
|
"--target=i386-pc",
|
|
"--boot-directory="+bootDir,
|
|
"--recheck",
|
|
disk,
|
|
); biosErr != nil {
|
|
if efiErr != nil {
|
|
// Both targets failed — the system will not boot.
|
|
return fmt.Errorf("both bootloader targets failed (EFI: %v; BIOS: %w)", efiErr, biosErr)
|
|
}
|
|
return biosErr
|
|
}
|
|
copySplashImage(mountRoot)
|
|
copyGrubFont(mountRoot)
|
|
|
|
// Both consoles listed: tty0 stays primary for local install, ttyS0 mirrors
|
|
// kernel output to serial so headless installs (CI, racks with serial-only
|
|
// IPMI, remote-dev SSH-to-QEMU) see boot messages. The last `console=` on
|
|
// the cmdline becomes the system console, so ttyS0 wins on serial-only
|
|
// hardware while tty0 still receives output for local display.
|
|
grubDefault := `GRUB_DEFAULT=0
|
|
GRUB_TIMEOUT=5
|
|
GRUB_DISTRIBUTOR=Spinifex
|
|
GRUB_CMDLINE_LINUX_DEFAULT=""
|
|
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200 systemd.show_status=1"
|
|
`
|
|
|
|
if err := os.WriteFile(filepath.Join(mountRoot, "etc/default/grub"), []byte(grubDefault), 0o644); err != nil {
|
|
return fmt.Errorf("write /etc/default/grub: %w", err)
|
|
}
|
|
|
|
// Mirror the ISO grub.cfg graphical block exactly so the installed GRUB menu
|
|
// looks identical to the installer menu. gfxterm MUST be activated before
|
|
// serial is appended — background_image silently does nothing in text mode.
|
|
// Using exec tail so update-grub includes everything from line 3 as raw GRUB config.
|
|
grubTheme := `#!/bin/sh
|
|
exec tail -n +3 $0
|
|
insmod all_video
|
|
insmod font
|
|
if loadfont /boot/grub/fonts/unicode.pf2; then
|
|
set gfxmode=auto
|
|
insmod gfxterm
|
|
terminal_output gfxterm
|
|
fi
|
|
insmod serial
|
|
if serial --unit=0 --speed=115200 --timeout=1; then
|
|
terminal_input --append serial
|
|
terminal_output --append serial
|
|
fi
|
|
|
|
# --- Branding ---
|
|
insmod png
|
|
set theme=/boot/grub/theme.txt
|
|
export theme
|
|
`
|
|
if err := os.WriteFile(filepath.Join(mountRoot, "etc/grub.d/06_spinifex"), []byte(grubTheme), 0o755); err != nil {
|
|
return fmt.Errorf("write /etc/grub.d/06_spinifex: %w", err)
|
|
}
|
|
|
|
if err := bindChrootMounts(); err != nil {
|
|
return err
|
|
}
|
|
defer unbindChrootMounts()
|
|
return run("chroot", mountRoot, "update-grub")
|
|
}
|
|
|
|
func installCACert(cfg *Config) error {
|
|
if !cfg.HasCACert || cfg.CACert == "" {
|
|
return nil
|
|
}
|
|
certPath := filepath.Join(mountRoot, "usr/local/share/ca-certificates/spinifex-ca.crt")
|
|
if err := os.WriteFile(certPath, []byte(cfg.CACert), 0o644); err != nil {
|
|
return err
|
|
}
|
|
if err := bindChrootMounts(); err != nil {
|
|
return err
|
|
}
|
|
defer unbindChrootMounts()
|
|
return run("chroot", mountRoot, "update-ca-certificates")
|
|
}
|
|
|
|
// promptRemoveUSB prints a removal reminder and waits up to 10 seconds for
|
|
// the user to press Enter before rebooting. Reading from os.Stdin works because
|
|
// spinifex-init redirects the installer's stdin from $CONSOLE_DEV.
|
|
func promptRemoveUSB() {
|
|
fmt.Println("\n\033[1mInstallation complete.\033[0m")
|
|
fmt.Println("Remove the USB drive, then press Enter to reboot (auto-rebooting in 10 seconds)...")
|
|
|
|
done := make(chan struct{})
|
|
go func() {
|
|
scanner := bufio.NewScanner(os.Stdin)
|
|
scanner.Scan()
|
|
close(done)
|
|
}()
|
|
|
|
select {
|
|
case <-done:
|
|
case <-time.After(10 * time.Second):
|
|
}
|
|
}
|
|
|
|
func reboot() error {
|
|
// sync filesystems before reboot so nothing is lost.
|
|
_ = run("sync")
|
|
// Use the kernel syscall directly — the live environment runs spinifex-init
|
|
// as PID 1 (not systemd), so the reboot(8) utility fails trying to reach D-Bus.
|
|
return syscall.Reboot(syscall.LINUX_REBOOT_CMD_RESTART)
|
|
}
|
|
|
|
// toFirstbootConfig maps installer Config to the firstboot package's Config.
|
|
func (c *Config) toFirstbootConfig() firstboot.Config {
|
|
// Geneve encap IP: use LAN bridge IP when a dedicated LAN NIC is present,
|
|
// otherwise fall back to WAN bridge IP. Empty for DHCP — setup-ovn.sh
|
|
// auto-detects the IP from the default route at boot in that case.
|
|
encapIP := c.WANAddress
|
|
if c.LANInterface != "" && c.LANAddress != "" {
|
|
encapIP = c.LANAddress
|
|
}
|
|
return firstboot.Config{
|
|
Hostname: c.Hostname,
|
|
EncapIP: encapIP,
|
|
ClusterRole: c.ClusterRole,
|
|
JoinAddr: c.JoinAddr,
|
|
Email: c.Email,
|
|
}
|
|
}
|
|
|
|
// writeFstab writes /etc/fstab on the installed system using partition UUIDs so
|
|
// the root filesystem is mounted read-write at boot and the EFI partition is
|
|
// mounted at /boot/efi.
|
|
func writeFstab(disk string) error {
|
|
efi, root := partitionPaths(disk)
|
|
rootUUID, err := partUUID(root)
|
|
if err != nil {
|
|
return fmt.Errorf("get root UUID: %w", err)
|
|
}
|
|
efiUUID, err := partUUID(efi)
|
|
if err != nil {
|
|
return fmt.Errorf("get EFI UUID: %w", err)
|
|
}
|
|
fstab := fmt.Sprintf("# /etc/fstab — generated by Spinifex installer\nUUID=%s / ext4 errors=remount-ro 0 1\nUUID=%s /boot/efi vfat umask=0077 0 1\n",
|
|
rootUUID, efiUUID)
|
|
return os.WriteFile(filepath.Join(mountRoot, "etc/fstab"), []byte(fstab), 0o644)
|
|
}
|
|
|
|
func partUUID(dev string) (string, error) {
|
|
out, err := exec.Command("blkid", "-s", "UUID", "-o", "value", dev).Output()
|
|
if err != nil {
|
|
return "", fmt.Errorf("blkid %s: %w", dev, err)
|
|
}
|
|
uuid := strings.TrimSpace(string(out))
|
|
if uuid == "" {
|
|
return "", fmt.Errorf("blkid returned no UUID for %s — partition may not have a filesystem yet", dev)
|
|
}
|
|
return uuid, nil
|
|
}
|
|
|
|
// partitionPaths returns the EFI and root partition device paths for a given
|
|
// disk. p1 is the BIOS Boot Partition (no filesystem), p2 is EFI, p3 is root.
|
|
// Handles both /dev/sdX (→ /dev/sdX2, /dev/sdX3) and /dev/nvmeXnY
|
|
// (→ /dev/nvmeXnYp2, /dev/nvmeXnYp3).
|
|
func partitionPaths(disk string) (efi, root string) {
|
|
// NVMe devices use a 'p' separator before the partition number.
|
|
if len(disk) > 0 && disk[len(disk)-1] >= '0' && disk[len(disk)-1] <= '9' {
|
|
return disk + "p2", disk + "p3"
|
|
}
|
|
return disk + "2", disk + "3"
|
|
}
|
|
|
|
// copySplashImage copies the GRUB splash (embedded in the squashfs at build time by
|
|
// inject-bins.sh) into the installed system so the post-install GRUB shows the same
|
|
// branded background as the installer GRUB. Non-fatal — missing source is logged and skipped.
|
|
func copySplashImage(root string) {
|
|
const src = "/usr/share/spinifex/grub-splash.png"
|
|
in, err := os.Open(src)
|
|
if err != nil {
|
|
slog.Warn("copySplashImage: splash not found, skipping", "path", src)
|
|
return
|
|
}
|
|
defer in.Close()
|
|
|
|
dstDir := filepath.Join(root, "boot/grub")
|
|
if err := os.MkdirAll(dstDir, 0o755); err != nil {
|
|
slog.Warn("copySplashImage: cannot create boot/grub dir", "err", err)
|
|
return
|
|
}
|
|
out, err := os.OpenFile(filepath.Join(dstDir, "splash.png"), os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644)
|
|
if err != nil {
|
|
slog.Warn("copySplashImage: cannot open destination", "err", err)
|
|
return
|
|
}
|
|
defer out.Close()
|
|
if _, err := io.Copy(out, in); err != nil {
|
|
slog.Warn("copySplashImage: copy failed", "err", err)
|
|
}
|
|
}
|
|
|
|
// copyGrubFont copies the unicode font into the installed system's
|
|
// /boot/grub/fonts/ so the loadfont path in 05_spinifex resolves at boot.
|
|
// grub-install does not copy fonts; we mirror what build-iso.sh does.
|
|
func copyGrubFont(root string) {
|
|
const src = "/usr/share/grub/unicode.pf2"
|
|
in, err := os.Open(src)
|
|
if err != nil {
|
|
slog.Warn("copyGrubFont: font not found, graphical GRUB may not work", "path", src)
|
|
return
|
|
}
|
|
defer in.Close()
|
|
|
|
dstDir := filepath.Join(root, "boot/grub/fonts")
|
|
if err := os.MkdirAll(dstDir, 0o755); err != nil {
|
|
slog.Warn("copyGrubFont: cannot create fonts dir", "err", err)
|
|
return
|
|
}
|
|
out, err := os.OpenFile(filepath.Join(dstDir, "unicode.pf2"), os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644)
|
|
if err != nil {
|
|
slog.Warn("copyGrubFont: cannot open destination", "err", err)
|
|
return
|
|
}
|
|
defer out.Close()
|
|
if _, err := io.Copy(out, in); err != nil {
|
|
slog.Warn("copyGrubFont: copy failed", "err", err)
|
|
}
|
|
}
|
|
|
|
// maskSystemdUnit creates a symlink to /dev/null for the given unit, which is
|
|
// the standard way to permanently disable a unit so systemd will never start it.
|
|
func maskSystemdUnit(root, unit string) error {
|
|
unitPath := filepath.Join(root, "etc/systemd/system", unit)
|
|
_ = os.Remove(unitPath)
|
|
return os.Symlink("/dev/null", unitPath)
|
|
}
|
|
|
|
// setShadowPassword sets a Unix password on the installed system without
|
|
// going through PAM. See the long comment in installSpinifex for the
|
|
// rationale.
|
|
func setShadowPassword(user, password string) error {
|
|
cmd := exec.Command("chpasswd", "-c", "YESCRYPT", "-R", mountRoot)
|
|
cmd.Stdin = strings.NewReader(user + ":" + password)
|
|
cmd.Stdout = os.Stdout
|
|
cmd.Stderr = os.Stderr
|
|
return cmd.Run()
|
|
}
|
|
|
|
// chrootMountPaths lists virtual filesystems to bind-mount into the chroot.
|
|
// Order matters: unbind in reverse.
|
|
var chrootMountPaths = []string{"dev", "proc", "sys"}
|
|
|
|
// bindChrootMounts bind-mounts /dev, /proc, and /sys into the installed rootfs
|
|
// so chroot commands (chpasswd, systemd-machine-id-setup, update-grub) can
|
|
// access hardware, process info, and entropy sources. Idempotent — already-
|
|
// mounted paths are skipped.
|
|
func bindChrootMounts() error {
|
|
for _, m := range chrootMountPaths {
|
|
dst := filepath.Join(mountRoot, m)
|
|
if err := os.MkdirAll(dst, 0o755); err != nil {
|
|
return fmt.Errorf("create chroot mountpoint /%s: %w", m, err)
|
|
}
|
|
if err := run("mount", "--bind", "/"+m, dst); err != nil {
|
|
return fmt.Errorf("bind-mount /%s into chroot: %w", m, err)
|
|
}
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// unbindChrootMounts unmounts the virtual filesystems in reverse order.
|
|
// Errors are logged but not returned — this is best-effort cleanup.
|
|
func unbindChrootMounts() {
|
|
for i := len(chrootMountPaths) - 1; i >= 0; i-- {
|
|
_ = run("umount", filepath.Join(mountRoot, chrootMountPaths[i]))
|
|
}
|
|
}
|
|
|
|
func run(name string, args ...string) error {
|
|
cmd := exec.Command(name, args...)
|
|
cmd.Stdout = os.Stdout
|
|
cmd.Stderr = os.Stderr
|
|
return cmd.Run()
|
|
}
|