spinifex/docs/admin/host-lifecycle

title, description, category, tags, resources

title	description	category	tags	resources
Host and Guest Lifecycle	The contract between Spinifex host service lifecycle events and running guest EC2 instances.	Admin	lifecycle shutdown systemd guests	title url Spinifex Repository https://github.com/mulgadc/spinifex title url Updating Spinifex /docs/admin/update

Host and Guest Lifecycle

The contract between Spinifex host service lifecycle events and running guest EC2 instances.

Table of Contents

• Overview
• The Contract
• Older Releases: systemctl stop spinifex.target
• Why Guests Outlive the Daemon
• Checking State
• Maintenance Runbooks

---

Overview

spinifex.target groups every Spinifex service on a node: spinifex-nats, spinifex-predastore, spinifex-northstar, spinifex-viperblock, spinifex-daemon, spinifex-awsgw, spinifex-vpcd, spinifex-ui, spinifex-qmp-collector, spinifex-shutdown, and the spinifex-nats-watchdog timer. A guest EC2 instance is a qemu-system-x86_64 process launched by spinifex-daemon — it is not itself a systemd unit, and it is deliberately allowed to keep running after the service that launched it stops.

The one idea to hold onto: guests outlive the daemon by design, so what happens to a running guest depends entirely on which lifecycle event triggered the stop, not on whether "Spinifex" as a whole looks like it is still running.

The Contract

Event	What happens to running guests	Is storage available afterwards	What the operator must do
Host reboot / poweroff / halt	Drained gracefully — QMP system_powerdown, volumes unmounted, WAL flushed — by spinifex-shutdown.service's ExecStop	n/a, the host is going down	Nothing extra
systemctl restart spinifex.target	Survive. Not rebooted. Reattached over QMP by Manager.Restore when the daemon comes back	Yes, within seconds	Nothing extra — this is the supported way to pick up a config change
systemctl stop spinifex.target	Drained the same as a host shutdown, because nothing guarantees the stack comes back on its own	No, not until the stack is started again	Nothing extra — but see Older Releases if the node predates this behaviour
systemctl restart spinifex-daemon.service (daemon only)	Survive via KillMode=process. Reattached by Manager.Restore	Yes, throughout	Nothing extra
Daemon crash / SIGKILL	Survive	Yes, throughout — the crash only takes down the daemon	Nothing — crash recovery reattaches automatically on the next daemon start
Host power loss	Die with the host. No drain is possible	No, until the host boots and services start again	Expect Manager.Restore to run crash recovery on boot. Guest filesystems may need an fsck the same as after any unclean power cut
spx admin cluster shutdown	Stopped in the DRAIN phase, before storage (STORAGE, PERSIST) and control-plane (INFRA) services stop	n/a, the cluster is being taken down	Nothing extra — this is the correct command for taking a whole cluster down

Older Releases: systemctl stop spinifex.target

Warning

Releases from v1.15.0 up to the release carrying this page do NOT drain on a plain systemctl stop spinifex.target. Running guests get no signal at all — no QMP system_powerdown, no SIGTERM. Predastore and viperblock stop underneath the guest, leaving it with no data path, while QEMU keeps running. Drives are launched with cache=none,werror=report,rerror=report, so I/O errors are reported into the guest instead of pausing it — the guest keeps running and keeps believing it has a disk. Anything written after storage goes away is lost, and a guest filesystem that takes I/O errors on its journal can be left inconsistent.

Cause: on those releases spinifex-shutdown.service's ExecStop ran spx admin node drain --local --timeout=120s --only-if-host-stopping, which drains only when systemctl is-system-running reports the literal state stopping. That is true for a real reboot/poweroff but not for a plain systemctl stop spinifex.target — systemd reports running for both a plain stop and the stop half of a restart, so the gate could not tell them apart. It skipped the drain and logged Host is not shutting down; skipping guest drain. The gate now also reads systemd's pending job list, which does distinguish the two.

The fastest way to check is spx admin upgrade --dry-run, which reports every unit's status against the one shipped in the running binary — including spinifex-shutdown.service — without changing anything. See Checking for Unit Drift. A node reported stale or missing for that unit predates this fix; reconcile it with sudo spx admin upgrade --units-only --yes (this only rewrites the unit and runs daemon-reload — it does not restart anything or drain guests itself).

To check by hand instead, look at the unit directly:

systemctl cat spinifex-shutdown.service | grep ExecStop

--unless-restarting drains on a plain stop; --only-if-host-stopping does not. On a node still showing --only-if-host-stopping, drain guests explicitly before stopping the target.

1. Stop the instances first, via the AWS-compatible API:

   aws ec2 stop-instances --instance-ids i-0123456789abcdef0
   

   Or run the drain command by hand on the node — invoked without a gate flag it drains unconditionally:

   sudo spx admin node drain --local
   

2. Confirm no guests are left running:

   ps auxw | grep qemu-system
   

3. Only then stop the target:

   sudo systemctl stop spinifex.target
   

This does not apply to systemctl restart spinifex.target — the restart row in the contract table above is unaffected, since the same gate is what makes guests survive a restart intact.

Why Guests Outlive the Daemon

KillMode=process is set on spinifex-daemon.service and spinifex-viperblock.service. It tells systemd to signal only the unit's main process and leave everything else in its cgroup alone, so guest QEMU processes and any nbdkit backend still serving a guest survive a restart of the service that spawned them. This is deliberate: a control-plane restart or upgrade must never interrupt a customer workload just because the daemon that launched it needed to bounce.

On the way back up, Manager.Restore reconnects to each surviving guest over QMP rather than relaunching it, so the guest is never rebooted by a daemon or target restart.

This is what makes the restart row in the contract table safe, and it holds only as long as storage comes back with the daemon — a daemon-only restart never takes predastore or viperblock down, and a target restart brings them back within seconds. It does not hold for an event that leaves storage down indefinitely, which is exactly the deviation described above.

Checking State

# Service status for the whole stack
systemctl status spinifex.target

# Is systemd mid-shutdown, or just running?
systemctl is-system-running

# Any guest QEMU processes still alive
ps auxw | grep qemu-system

# Guest state as Spinifex sees it (node, status, health)
spx get vms

# EC2-API view of instance state
aws ec2 describe-instances

# Confirm whether the last target stop actually drained guests
journalctl -u spinifex-shutdown -b
# Look for: "Host is not shutting down; skipping guest drain."

Maintenance Runbooks

Apply a Config Change

1. Edit /etc/spinifex/spinifex.toml.

2. Apply it:

   sudo systemctl restart spinifex.target
   

Guests survive this restart — they are not rebooted, and storage returns within seconds. This is the supported way to pick up a config change; see the restart row in The Contract.

Take a Node Down for Hardware Maintenance

1. Stop or reboot the host — both drain running guests first:

   sudo systemctl stop spinifex.target   # taking the node fully out of service
   sudo reboot                           # or, for a reboot
   

2. Confirm nothing is left running:

   ps auxw | grep qemu-system
   

The drain is automatic on both paths. Stop the instances yourself first only if you want them to stay stopped when the node returns, or if the node still runs an older build — see Older Releases above.