Josh Siviour
4dca2c7b59
* fix: drain guests on a plain target stop, not just restart
`systemctl is-system-running` cannot tell a plain `systemctl stop
spinifex.target` apart from the stop half of a restart -- both report
`running`, and the old `--only-if-host-stopping` gate skipped the drain
for both. That left guests running with no QMP shutdown and no volume
unmount while predastore/viperblock stopped underneath them, so a real
stop discarded any writes still in flight.
Add a second signal: `systemctl list-jobs` shows a pending START job for
spinifex.target during a restart (the target's stop leg resolves
instantly, since it owns no processes) or a pending RESTART job for
spinifex-shutdown.service itself when that unit is restarted directly
(PartOf propagates target-to-member only). Renamed the flag to
`--unless-restarting` to match: drain on a real shutdown or a plain
stop, skip only when one of those jobs shows a restart is already
queued. Any step systemctl can't answer fails toward draining, and a
skipped drain now warns loudly (naming each instance still running) so
an operator sees the risk in the journal instead of discovering it via
`ps`. `--only-if-host-stopping` stays as a deprecated, still-functional
alias so a partial upgrade between unit file and binary doesn't break.
* fix: retry the local drain GATE/DRAIN ACK instead of failing on first miss
A drain that fails at GATE is indistinguishable from a skipped drain from
the guest's perspective: it keeps running while storage goes away
underneath it. Stopping spinifex.target shortly after a restart hit exactly
this: the daemon had not finished re-subscribing to NATS, so the ACK
request came back with zero subscribers and NATS bounced it back near-
instantly ("no responders"), which is why the failure logged in about a
second despite --timeout=120s. collectLocalShutdownACK now retries with
backoff bounded by the phase timeout instead of failing on the first bounce,
and an exhausted retry budget logs at ERROR and names the guests still
running via a severity-aware refactor of warnIfGuestsLeftRunning, so a
failed drain is unmistakable in the journal rather than silent.
* fix: reap idle nbdkit concurrently and re-derive viperblock's stop budget
KillMode=process made viperblockd the sole reaper of nbdkit on SIGTERM, but
shutdownVolumes reaped them one at a time and TimeoutStopSec was never
re-derived for that change. With several mounted volumes the serial loop
blew past the 30s budget, systemd SIGKILLed viperblockd mid-loop, and
whichever nbdkit hadn't been reaped yet was orphaned.
Fan the per-volume utils.KillProcess calls out concurrently so the reap
costs one worst-case nbdkit, not N of them summed, and raise
TimeoutStopSec to 150s to strictly exceed that single-volume worst case
(120s KillProcess grace + nbdkit's own 20s drain + headroom). Add an
invariant asserting the two stay in that relationship, and a test proving
the reap is actually concurrent (fails against the old serial loop).
* docs: document the host and guest lifecycle contract
States what happens to running guests for every host lifecycle event: reboot, target restart, target stop, daemon restart, daemon crash, power loss and a coordinated cluster shutdown. Records the older-release behaviour where a plain target stop left guests with no data path, and how to tell which behaviour a node has.
4.4 KiB
title, description, category, tags, resources
| title | description | category | tags | resources | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Updating Spinifex | Upgrade an existing Spinifex installation to the latest release. | Admin |
|
|
Updating Spinifex
Upgrade an existing Spinifex installation to the latest release.
Table of Contents
Overview
Updating Spinifex is the same command used to install it. The installer detects an existing installation, downloads the latest binary and runs any pending configuration migrations before restarting services.
For operators who want to review migrations before they are applied, a manual upgrade path is also supported.
Warning
Object storage is exempt when upgrading from v1.15.0 or earlier to v1.16.0. Predastore's configuration schema and on-disk layout changed with the object storage cutover in v1.16.0, and no migration converts an installation from before it.
spx admin upgradewill not report anything pending forpredastore.toml, and updating the binary over such an installation leaves Predastore unable to start. Those clusters have to be re-initialised from scratch, which discards their stored objects — export anything you need first. Upgrades between v1.16.0 and later releases are unaffected.
Instructions
Step 1. Re-run the Installer
curl -fsSL https://install.mulgadc.com | bash
That's it. The installer will:
- Download and install the latest Spinifex binary.
- Reinstall systemd units so new services are picked up.
- Run any pending configuration migrations automatically (equivalent to
spx admin upgrade --yes). - Restart
spinifex.targetif the services were already running.
Step 2. Verify
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
If this returns a list of instance types, your upgrade is complete.
Manual Upgrade
If you prefer to review pending migrations before they are applied, Spinifex supports running spx admin init to allow you to verify config file migrations.
Step 1. Install the New Binary Without Running Migrations
curl -fsSL https://install.mulgadc.com | INSTALL_SPINIFEX_SKIP_MIGRATE=1 bash
The installer will download the new binary and reinstall systemd units, but will not apply any configuration migrations.
Step 2. Review Pending Migrations
sudo spx admin upgrade
The command prints the current version of each config file, the migrations that would be applied, and a from → to description for each. It then prompts for confirmation before making any changes. Answer n to abort without touching your config.
Step 3. Apply Migrations
When you are ready, answer y at the prompt, or re-run with --yes to apply non-interactively:
sudo spx admin upgrade --yes
Step 4. Restart Services
Migrations modify config files on disk but do not restart running services. Apply the new config with:
sudo systemctl restart spinifex.target
A restart preserves any running guests — they are not rebooted, and storage returns within seconds. See Host and Guest Lifecycle for the full contract.
Troubleshooting
No Pending Config Migrations
No pending config migrations.
Your config is already at the latest version. Nothing to do.
No Spinifex Installation Found
No Spinifex installation found at /etc/spinifex
Run 'spx admin init' first.
spx admin upgrade requires an initialized installation. If this is a fresh host, follow the Single-Node Install guide instead.
Migration Failure
If a migration fails, the installer and spx admin upgrade exit non-zero and leave the config in its prior state where possible. Review the error output, then re-run sudo spx admin upgrade once the underlying issue is resolved.
Services Did Not Pick Up New Config
Migrations edit config files on disk but the running daemons continue to use the config they loaded at start-up. Restart with:
sudo systemctl restart spinifex.target
A restart preserves any running guests — they are not rebooted, and storage returns within seconds. See Host and Guest Lifecycle for the full contract.