fix: keep needrestart from restarting the Spinifex, OVS and OVN units

Guest QEMU runs in spinifex-daemon's cgroup, so needrestart charged QEMU's
replaced libraries to the daemon and restarted it after every dpkg run. The
restart never cleared them, so an unattended upgrade ran the unit into its
start limit and left the control plane down.

setup.sh now writes /etc/needrestart/conf.d/90-spinifex.conf, deselecting
spinifex-, ovs-, ovsdb-, ovn- and openvswitch- units. needrestart still lists
them, so restarting after an upgrade stays the operator's choice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

scripts/README.md

@@ -6,7 +6,7 @@ Node installation, dev-environment lifecycle, guest image builds, and verificati
 
 | Script | What it does |
 |---|---|
-| `setup.sh` | The production binary installer, served from `install.mulgadc.com`. Installs deps, users, sudoers, the host firewall policy, files, systemd units, logrotate and udev rules, and loads `openvswitch`, `geneve`, `vhost_net` and `tun`, listing them in `/etc/modules-load.d/spinifex.conf` for later boots. It masks firewalld wherever it is installed, since the Spinifex nftables policy is the only supported node firewall. An unset `--firewall` keeps the mode recorded in `/etc/spinifex/firewall/mode`, and otherwise arms the policy when firewalld was active. Where systemd-resolved is installed but not running, resolvconf is absent and NetworkManager runs, it hands host DNS to resolved: it saves `/etc/resolv.conf` once to `/var/lib/spinifex-reset/resolv.conf.orig`, links `/etc/resolv.conf` to the resolved stub, and stops unless resolved has upstream servers and the package repository host still resolves. It links `/usr/bin/spx` to `/usr/local/bin/spx` where sudo's PATH leaves out `/usr/local/bin`, and takes the nbdkit plugin directory from `nbdkit --dump-config`, stopping when nbdkit cannot say. `SETUP_STAGES` runs a subset; `ISO_BUILD=1` runs it inside the ISO builder's chroot. |
+| `setup.sh` | The production binary installer, served from `install.mulgadc.com`. Installs deps, users, sudoers, the host firewall policy, files, systemd units, logrotate and udev rules, and loads `openvswitch`, `geneve`, `vhost_net` and `tun`, listing them in `/etc/modules-load.d/spinifex.conf` for later boots. It masks firewalld wherever it is installed, since the Spinifex nftables policy is the only supported node firewall. It writes a needrestart drop-in so an apt upgrade never restarts the Spinifex, OVS or OVN units; restarting them after an upgrade is left to the operator. An unset `--firewall` keeps the mode recorded in `/etc/spinifex/firewall/mode`, and otherwise arms the policy when firewalld was active. Where systemd-resolved is installed but not running, resolvconf is absent and NetworkManager runs, it hands host DNS to resolved: it saves `/etc/resolv.conf` once to `/var/lib/spinifex-reset/resolv.conf.orig`, links `/etc/resolv.conf` to the resolved stub, and stops unless resolved has upstream servers and the package repository host still resolves. It links `/usr/bin/spx` to `/usr/local/bin/spx` where sudo's PATH leaves out `/usr/local/bin`, and takes the nbdkit plugin directory from `nbdkit --dump-config`, stopping when nbdkit cannot say. `SETUP_STAGES` runs a subset; `ISO_BUILD=1` runs it inside the ISO builder's chroot. |
 | `setup-ovn.sh` | Bootstraps a compute node for OVN VPC networking: installs OVN/OVS, creates `br-int`, configures the WAN bridge (bridged, DHCP, or routed-NAT uplink), sets the chassis identity, and applies overlay sysctls. It runs OVS and OVN as root, setting `OVS_USER_ID` and `OVN_USER_ID` where `/etc/sysconfig` holds them, and hooks the socket-permission helper onto `ovs-vswitchd`, `ovn-controller`, `openvswitch-ipsec` and, on EL, `ovn-northd`. `--management` also starts the OVN central services, writes their `ovn-ctl` flags to `/etc/default/ovn-central` (Debian) or `/etc/sysconfig/ovn-northd` (EL), and records the role in `/etc/spinifex/ovn-role`. With clustered databases it returns only once this node's NB and SB members are back in their RAFT clusters under a known leader, and fails after `OVN_RAFT_REJOIN_WAIT` seconds (default 60) otherwise, so reruns one node at a time never take two members of a cluster down. A later run without `--management` refuses while that file, or the `OVN_CTL_OPTS` line older nodes carry in `/etc/default/ovn-central`, records the role. It adds `maxsize 100M` to the OVS and OVN logrotate files, plus `rotate 5` where the file sets no count, and rotates as root, matching the daemons. On a host whose IPsec stack is libreswan it enables, stops and masks `openvswitch-ipsec.service` and `ipsec.service` for the daemon to unmask when IPsec is on, unless `openvswitch-ipsec.service` is already running; elsewhere it enables and starts `openvswitch-ipsec.service`, unless the daemon has masked it to keep IPsec off. The links it creates persist through systemd-networkd where that is enabled, and otherwise through `/etc/spinifex/hostnet.json`, which `spinifex-hostnet.service` (`spx admin hostnet apply`) replays at boot and on every OVS start, starting `ovn-controller` again when an OVS stop took it down; the first run records the choice in `/etc/spinifex/hostnet-backend`. Where NetworkManager runs, it leaves the links Spinifex owns unmanaged. |
 | `install-node.sh` | Forms a multi-node cluster across hosts that are already running Spinifex: rebuilds the OVN database in clustered form, runs `spx admin init` on the first host and `spx admin join` on the rest, then verifies. Before touching any host it refuses hosts of different OS families (`distro_os_family`) or `spx` versions, and `--ipsec on` when any host uses libreswan. The scripted form of [install-multi-node](../docs/install/install-multi-node/README.md). Destroys the joining nodes' logical network state and crypto — see its `--help`. |
 | `dev-install.sh` | Full local dev setup through the production installer: builds from source, packs a tarball, runs `setup.sh`, initialises the cluster, and starts services under systemd. |

scripts/setup.sh

@@ -16,8 +16,8 @@
 #   VERBOSE                    Set to 1 to echo "[setup] <stage>" before each top-level step.
 #   SETUP_STAGES               Comma-separated subset of stages to run:
 #                                deps, aws, dns, users, sudoers, firewall, timesync, files,
-#                                directories, env, systemd, sysctl, modules, logrotate,
-#                                udev, fixown, migrations
+#                                directories, env, systemd, sysctl, needrestart, modules,
+#                                logrotate, udev, fixown, migrations
 #                              Unset = run every stage appropriate for the current mode.
 
 set -e
@@ -1895,6 +1895,28 @@ SYSCTL
     info "Kernel tunables installed (net.core.rmem_max/wmem_max = 16MB)"
 }
 
+# --- Keep needrestart off the Spinifex, OVS and OVN units ---
+# Guest QEMU runs in spinifex-daemon's cgroup, so needrestart charges QEMU's
+# replaced libraries to the daemon and restarts it after every dpkg run, which
+# never clears them and ends at the unit's start limit. Restarting the stack
+# after an upgrade is the operator's call; needrestart still lists the units.
+install_needrestart_conf() {
+    stage "installing needrestart override"
+    local dir="${DISTRO_ROOT:-}/etc/needrestart/conf.d"
+    $SUDO install -d "$dir"
+    $SUDO tee "$dir/90-spinifex.conf" > /dev/null << 'NEEDRESTART'
+# Managed by spinifex setup.sh — edits are overwritten.
+# Never restart these automatically; restart them by hand when it suits.
+$nrconf{override_rc}{qr(^spinifex-)} = 0;
+$nrconf{override_rc}{qr(^ovs-)} = 0;
+$nrconf{override_rc}{qr(^ovsdb-)} = 0;
+$nrconf{override_rc}{qr(^ovn-)} = 0;
+$nrconf{override_rc}{qr(^openvswitch-)} = 0;
+NEEDRESTART
+    $SUDO chmod 0644 "$dir/90-spinifex.conf"
+    info "needrestart will not restart the spinifex, OVS or OVN units"
+}
+
 # --- Install logrotate ---
 install_logrotate() {
     stage "installing logrotate config"
@@ -2089,6 +2111,7 @@ main() {
     stage_enabled fixown     && fix_file_ownership
     stage_enabled systemd    && install_systemd
     stage_enabled sysctl     && install_sysctl
+    stage_enabled needrestart && install_needrestart_conf
     stage_enabled modules    && install_kernel_modules
     stage_enabled logrotate  && install_logrotate
     stage_enabled udev       && install_udev

scripts/setup_host_test.sh

@@ -1,6 +1,6 @@
 #!/bin/sh
 # setup_host_test.sh — tests for setup.sh's DNS handover to systemd-resolved,
-# its chrony drop-in and its nbdkit plugin directory.
+# its chrony and needrestart drop-ins and its nbdkit plugin directory.
 #
 # setup.sh is sourced with INSTALL_SPINIFEX_LIB_ONLY=1 so main never runs. PATH
 # holds stub commands and links to a few real tools only, so no host resolver,
@@ -17,10 +17,10 @@ fails=0
 pass() { echo "  ok   - $1"; }
 fail() { echo "  FAIL - $1"; fails=$((fails + 1)); }
 
-echo "-- setup.sh host DNS, chrony and nbdkit"
+echo "-- setup.sh host DNS, chrony, needrestart and nbdkit"
 
 mkdir -p "$TMP/sys"
-for tool in bash awk sed grep readlink basename dirname cp ln mkdir cat head tail wc tee rm chmod; do
+for tool in bash awk sed grep readlink basename dirname cp ln mkdir cat head tail wc tee rm chmod install; do
     ln -s "$(command -v "$tool")" "$TMP/sys/$tool"
 done
 
@@ -249,6 +249,50 @@ case "$RC:$OUT" in
     *) fail "no chrony: rc=$RC: $OUT" ;;
 esac
 
+# --------------------------------------------------------------------------
+# needrestart override.
+# --------------------------------------------------------------------------
+# restarts <conf> <unit> — how needrestart would treat <unit> under <conf>:
+# "no" when an override_rc pattern deselects it, "yes" otherwise.
+restarts() {
+    perl -e 'our %nrconf; do $ARGV[0]; die $@ if $@;
+        for my $re (keys %{$nrconf{override_rc}}) {
+            if ($ARGV[1] =~ $re) { print $nrconf{override_rc}{$re} ? "yes" : "no"; exit }
+        }
+        print "yes"' "$1" "$2"
+}
+
+host
+run install_needrestart_conf
+conf="$TMP/root/etc/needrestart/conf.d/90-spinifex.conf"
+if [ "$RC" != 0 ] || [ ! -f "$conf" ]; then
+    fail "needrestart: rc=$RC: $OUT"
+elif ! command -v perl > /dev/null; then
+    fail "needrestart: perl is not installed, so the override cannot be checked"
+else
+    wrong=""
+    for unit in spinifex-daemon.service spinifex-viperblock.service ovs-vswitchd.service \
+        ovsdb-server.service ovn-controller.service openvswitch-switch.service; do
+        [ "$(restarts "$conf" "$unit")" = no ] || wrong="$wrong $unit"
+    done
+    for unit in ssh.service rsyslog.service polkit.service; do
+        [ "$(restarts "$conf" "$unit")" = yes ] || wrong="$wrong $unit"
+    done
+    if [ -n "$wrong" ]; then
+        fail "needrestart override treats these wrongly:$wrong"
+    else
+        pass "needrestart keeps off the stack's units and still restarts everything else"
+    fi
+fi
+
+before=$(cat "$conf" 2>/dev/null)
+run install_needrestart_conf
+if [ "$RC" = 0 ] && [ "$(cat "$conf" 2>/dev/null)" = "$before" ]; then
+    pass "needrestart override is unchanged by a second install"
+else
+    fail "needrestart override changed on re-run: rc=$RC: $OUT"
+fi
+
 # --------------------------------------------------------------------------
 # nbdkit plugin directory.
 # --------------------------------------------------------------------------

scripts/uninstall-spx.sh

@@ -29,8 +29,9 @@
 #
 # /usr/local/bin/spx, the nbdkit plugin, /usr/local/{share,lib}/spinifex,
 # /usr/share/spinifex, every spinifex systemd unit and slice, the eight
-# spinifex-* service users, the sudoers grant, the sysctl/logrotate/chrony/udev
-# drop-ins, the host firewall policy and its loaded nft table, the OVS bridges
+# spinifex-* service users, the sudoers grant, the sysctl, logrotate, chrony,
+# needrestart and udev drop-ins, the host firewall policy and its loaded nft
+# table, the OVS bridges
 # and OVN databases, the Spinifex resolved drop-in, and the Spinifex CA from the
 # host trust store. Where setup.sh turned on systemd-resolved, it restores
 # /etc/resolv.conf and turns resolved off.
@@ -457,6 +458,7 @@ rm_path /etc/sysctl.d/*spinifex* \
         /etc/sudoers.d/spinifex-network \
         /etc/chrony/conf.d/spinifex.conf \
         /etc/chrony.d/spinifex.conf \
+        /etc/needrestart/conf.d/90-spinifex.conf \
         /etc/logrotate.d/spinifex \
         /etc/tmpfiles.d/spinifex.conf \
         /etc/udev/rules.d/*spinifex* \
@@ -615,6 +617,7 @@ if ! $DRY_RUN; then
              /usr/share/spinifex "$ETC_DIR" "$LOG_DIR" \
              /etc/sudoers.d/spinifex-network /etc/logrotate.d/spinifex /etc/tmpfiles.d/spinifex.conf \
              /etc/chrony/conf.d/spinifex.conf /etc/chrony.d/spinifex.conf "$RESOLV_CONF_SAVED" \
+             /etc/needrestart/conf.d/90-spinifex.conf \
              "$RESOLVED_DROPIN"; do
         exists "$p" && LEFT+=("$p")
     done