gsadmin/spinifex
1
0
Fork 0
mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-23 20:57:02 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: graceful guest drain on systemctl stop / host shutdown (#412)

Browse Source 
* fix: drop removed predastore s3.Config.Compaction from admin_test

The predastore submodule bump removed the Compaction field from s3.Config.
Parse the rendered [compaction] block into a local struct so the template
round-trip test no longer references the deleted field.

* feat: graceful guest drain on systemctl stop / host shutdown

systemctl stop spinifex.target tore down nbdkit/viperblock while guest QEMU
was still writing, corrupting guest filesystems. The ordered drain existed
only over NATS via cluster shutdown, never on the systemd/host path.

Add spx admin node drain --local (GATE+DRAIN against the local node, leaving
STORAGE/PERSIST/INFRA to systemd), wired as the ExecStop of a new
spinifex-shutdown.service oneshot ordered After= all services so it stops
first and drains guests while every service is still up. DDIL preserved: the
oneshot fires only on target/host shutdown, not on a daemon restart.

Harden the viperblockd SIGTERM handler to flush the WAL but skip reaping an
nbdkit a guest is still attached to. Add a units ordering invariant.

* fix: ExecStop drain timeout needs duration unit (120 -> 120s)

The spinifex-shutdown.service ExecStop passed --timeout=120, but spx admin
node drain's --timeout is a cobra Duration flag and rejects a bare integer
(time: missing unit in duration "120"). The drain exited 1 without running,
so a systemctl stop tore nbdkit out from under the running guest and corrupted
its ext4 (the exact failure this feature prevents). Pass 120s.

* fix: nbdkitInUse read ss state from correct column + cover it

ss -H rows are '<netid> <state> ...', so the state is field[1], not field[0]
(field[0] is 'u_str'). The old index never matched ESTAB, so an in-use nbdkit
read as idle and the SIGTERM path would reap it under a live guest. Add tests
exercising a real connected unix socket, an absent socket, and TCP transport.
This commit is contained in:
Josh Siviour
2026-06-24 13:31:14 +10:00
committed by GitHub
parent f45c97215a
commit 9e333c8a6c
9 changed files with 333 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
build/systemd/spinifex-shutdown.service
+18
View File
@@ -0,0 +1,18 @@
[Unit]
Description=Spinifex graceful guest drain
PartOf=spinifex.target
# After= all services so it stops FIRST on target/host shutdown: the drain runs
# while nats/predastore/viperblock/daemon are still up (QMP, unmount, WAL flush).
After=spinifex-nats.service spinifex-predastore.service spinifex-viperblock.service spinifex-daemon.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/true
# ExecStop only runs when the target/host goes down, not on a single-service
# restart — so a daemon restart still leaves guests running (DDIL).
ExecStop=/usr/local/bin/spx admin node drain --local --timeout=120s
TimeoutStopSec=180
[Install]
WantedBy=spinifex.target
build/systemd/spinifex.target
+1 -1
View File
@@ -2,7 +2,7 @@
Description=Spinifex Infrastructure Platform
After=network-online.target
Wants=network-online.target
Wants=spinifex-nats.service spinifex-predastore.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service openvswitch-switch.service ovn-controller.service
Wants=spinifex-nats.service spinifex-predastore.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service spinifex-shutdown.service openvswitch-switch.service ovn-controller.service
[Install]
WantedBy=multi-user.target