Overwriting /usr/local/bin/spx while spinifex.target is running does not move
the services onto it. Each keeps executing the replaced inode until it
restarts, so a node serves a mixture of two builds, and because the handlers
are NATS queue-group workers the skew shows up as roughly one request in three
behaving like the old build — an intermittent fault rather than a broken node.
The ansible deploy path already detects and remediates this, but only there:
an out-of-band binary copy leaves nothing to report it, and on env19 that
state went unnoticed for a full day. spx admin preflight now reports it, so an
operator has a command to run and a script has a non-zero exit to gate on.
Detection follows the deploy path's: a (deleted) marker on /proc/<pid>/exe, a
target path that no longer resolves, or a path that resolves to a different
inode than the one being executed.
* feat: reconcile systemd unit files on upgrade
Unit files were written once at install time by install_systemd() in
scripts/setup.sh and never re-asserted. A node upgraded by swapping the
spx binary kept whatever units it was installed with, so any fix that
lives in a unit rather than in code never reached it.
This is not theoretical. A node whose spinifex-viperblock.service
predates KillMode=process defaults to KillMode=control-group, so on a
target stop TimeoutStopSec expires mid-flush and systemd cgroup-SIGKILLs
every nbdkit while guest QEMU survives in the daemon's own cgroup: the
guest keeps running with no storage path.
Units are now stamped with a spinifex-unit-version marker, embedded in
the binary via a generated file, and reconciled by spx admin upgrade.
Missing or version-stale units are replaced after a timestamped backup;
a unit at the current version whose content differs is reported as
operator-modified and never overwritten. Drop-ins under <unit>.d are
untouched, so systemctl edit overrides survive.
Nothing is restarted. A daemon-reload is enough for KillMode and
TimeoutStopSec to apply to the next stop, so drift is corrected without
disturbing a running guest.
Adds --dry-run, --units-only and --skip-units to spx admin upgrade.
* test: cover systemd reconcile error paths and the admin upgrade CLI flow
Diff coverage on the unit-migration PR sat at 45.8%, below the 60% gate.
Adds tests for reconcile.go's error/edge paths (unreadable unit, backup
and write failures, daemon-reload failure, malformed version markers,
downgrade-is-conflict) and drives admin_upgrade.go's CLI flow directly
via an injectable exit and a decision matrix extracted into
validateUpgradeFlags/upgradeSummary for testability.
AMI metadata moved to ebsmetadata documents and the legacy
ami-<id>/config.json read path was removed rather than migrated, so an
AMI imported before the change is invisible to the control plane after
upgrading. Nothing reports it as pending, because the gap is in stored
data rather than in a config file. Warn about it alongside the existing
object-storage exemption, and say how to check and recover.
Also document the binary-swap trap: replacing spx under a running
target leaves services on the replaced inode, and because handlers are
queue-group workers a skewed node answers a fraction of requests with
the old behaviour, which reads as an intermittent fault.
* fix: drain guests on a plain target stop, not just restart
`systemctl is-system-running` cannot tell a plain `systemctl stop
spinifex.target` apart from the stop half of a restart -- both report
`running`, and the old `--only-if-host-stopping` gate skipped the drain
for both. That left guests running with no QMP shutdown and no volume
unmount while predastore/viperblock stopped underneath them, so a real
stop discarded any writes still in flight.
Add a second signal: `systemctl list-jobs` shows a pending START job for
spinifex.target during a restart (the target's stop leg resolves
instantly, since it owns no processes) or a pending RESTART job for
spinifex-shutdown.service itself when that unit is restarted directly
(PartOf propagates target-to-member only). Renamed the flag to
`--unless-restarting` to match: drain on a real shutdown or a plain
stop, skip only when one of those jobs shows a restart is already
queued. Any step systemctl can't answer fails toward draining, and a
skipped drain now warns loudly (naming each instance still running) so
an operator sees the risk in the journal instead of discovering it via
`ps`. `--only-if-host-stopping` stays as a deprecated, still-functional
alias so a partial upgrade between unit file and binary doesn't break.
* fix: retry the local drain GATE/DRAIN ACK instead of failing on first miss
A drain that fails at GATE is indistinguishable from a skipped drain from
the guest's perspective: it keeps running while storage goes away
underneath it. Stopping spinifex.target shortly after a restart hit exactly
this: the daemon had not finished re-subscribing to NATS, so the ACK
request came back with zero subscribers and NATS bounced it back near-
instantly ("no responders"), which is why the failure logged in about a
second despite --timeout=120s. collectLocalShutdownACK now retries with
backoff bounded by the phase timeout instead of failing on the first bounce,
and an exhausted retry budget logs at ERROR and names the guests still
running via a severity-aware refactor of warnIfGuestsLeftRunning, so a
failed drain is unmistakable in the journal rather than silent.
* fix: reap idle nbdkit concurrently and re-derive viperblock's stop budget
KillMode=process made viperblockd the sole reaper of nbdkit on SIGTERM, but
shutdownVolumes reaped them one at a time and TimeoutStopSec was never
re-derived for that change. With several mounted volumes the serial loop
blew past the 30s budget, systemd SIGKILLed viperblockd mid-loop, and
whichever nbdkit hadn't been reaped yet was orphaned.
Fan the per-volume utils.KillProcess calls out concurrently so the reap
costs one worst-case nbdkit, not N of them summed, and raise
TimeoutStopSec to 150s to strictly exceed that single-volume worst case
(120s KillProcess grace + nbdkit's own 20s drain + headroom). Add an
invariant asserting the two stay in that relationship, and a test proving
the reap is actually concurrent (fails against the old serial loop).
* docs: document the host and guest lifecycle contract
States what happens to running guests for every host lifecycle event: reboot, target restart, target stop, daemon restart, daemon crash, power loss and a coordinated cluster shutdown. Records the older-release behaviour where a plain target stop left guests with no data path, and how to tell which behaviour a node has.
The port tables were wrong in both directions: inter-node traffic is QUIC over
UDP rather than TCP, and the shard range 9991-9993 no longer exists. Ports are
unique within a host rather than across the cluster, so every machine uses the
same three, and a single-node install opens only the S3 port because colocated
nodes talk over an in-process pipe.
The reference architectures scripted a move of node data out of a directory
layout that no longer exists, which would have broken any install following
them. The update guide now says object storage has no migration path.
Josh Siviour