Multi-node install
- Lead with the fact that the guide builds a three-node cluster, and say
how to go beyond three. The callout sits inside ## Overview because the
docs site discards everything above the first section heading.
- Replace the OVN-only sizing prose with a table covering all three
distributed layers, and move the OVN control-plane detail into the VPC
networking guide, linked from here.
- Add minimum and recommended hardware.
- Move the object storage layout to the end as background; it was the
second thing a reader hit.
- Collapse the ISO conversion notes from a section into one skippable
box, now that --force is unconditional in Step 4 and the formation
port needs no manual intervention.
- Expand verification with spx get nodes and spx top nodes, including
what to look for in each.
- Drop the install-node.sh shortcut (internal tooling) and the
"adding a node later" section (no good story yet).
Corrections found while writing
- Servers 4+ were described as running no database. They run predastore
and viperblock like every other node; only the OVN database cluster is
capped at three. Fixed in four places.
- Step 4 never needed the firewall down: spx admin init opens the
formation port itself for the formation window. Only Step 3's
peer-scoped OVN ports are affected.
- `nft list table inet spinifex_filter | grep spinifex_peers` could never
match — the peer list is an nft define, expanded at load time. Anyone
following it would have seen empty output and assumed the worst.
- Disambiguate "database": with predastore, NATS and OVN all in play it
needed qualifying throughout.
- Align the example output region with the AWS_REGION the guide exports.
Cross-doc links
- Rewrite relative and category-prefixed links to the canonical
/docs/<slug> form. The site publishes docs flat, so ../foo/README.md
and /docs/category/foo both 404.
- Point the six links to unpublished docs at the source repo instead.
- Fix four fragments written against GitHub's anchor algorithm.
* docs: bring doc titles and descriptions into SEO length limits
Rewrite the frontmatter title and description of every published doc so the
rendered <title> lands in 50-60 characters (including the site suffix) and the
meta description lands in 150-160 characters.
* docs: move SEO titles to a seoTitle field and restore display titles
The frontmatter title renders as the page H1 and the sidebar label, so tuning it
for the title tag shortened nothing and lengthened the visible navigation. Put
the search-facing string in seoTitle and give title back its original value.
Overwriting /usr/local/bin/spx while spinifex.target is running does not move
the services onto it. Each keeps executing the replaced inode until it
restarts, so a node serves a mixture of two builds, and because the handlers
are NATS queue-group workers the skew shows up as roughly one request in three
behaving like the old build — an intermittent fault rather than a broken node.
The ansible deploy path already detects and remediates this, but only there:
an out-of-band binary copy leaves nothing to report it, and on env19 that
state went unnoticed for a full day. spx admin preflight now reports it, so an
operator has a command to run and a script has a non-zero exit to gate on.
Detection follows the deploy path's: a (deleted) marker on /proc/<pid>/exe, a
target path that no longer resolves, or a path that resolves to a different
inode than the one being executed.
* feat: reconcile systemd unit files on upgrade
Unit files were written once at install time by install_systemd() in
scripts/setup.sh and never re-asserted. A node upgraded by swapping the
spx binary kept whatever units it was installed with, so any fix that
lives in a unit rather than in code never reached it.
This is not theoretical. A node whose spinifex-viperblock.service
predates KillMode=process defaults to KillMode=control-group, so on a
target stop TimeoutStopSec expires mid-flush and systemd cgroup-SIGKILLs
every nbdkit while guest QEMU survives in the daemon's own cgroup: the
guest keeps running with no storage path.
Units are now stamped with a spinifex-unit-version marker, embedded in
the binary via a generated file, and reconciled by spx admin upgrade.
Missing or version-stale units are replaced after a timestamped backup;
a unit at the current version whose content differs is reported as
operator-modified and never overwritten. Drop-ins under <unit>.d are
untouched, so systemctl edit overrides survive.
Nothing is restarted. A daemon-reload is enough for KillMode and
TimeoutStopSec to apply to the next stop, so drift is corrected without
disturbing a running guest.
Adds --dry-run, --units-only and --skip-units to spx admin upgrade.
* test: cover systemd reconcile error paths and the admin upgrade CLI flow
Diff coverage on the unit-migration PR sat at 45.8%, below the 60% gate.
Adds tests for reconcile.go's error/edge paths (unreadable unit, backup
and write failures, daemon-reload failure, malformed version markers,
downgrade-is-conflict) and drives admin_upgrade.go's CLI flow directly
via an injectable exit and a decision matrix extracted into
validateUpgradeFlags/upgradeSummary for testability.
AMI metadata moved to ebsmetadata documents and the legacy
ami-<id>/config.json read path was removed rather than migrated, so an
AMI imported before the change is invisible to the control plane after
upgrading. Nothing reports it as pending, because the gap is in stored
data rather than in a config file. Warn about it alongside the existing
object-storage exemption, and say how to check and recover.
Also document the binary-swap trap: replacing spx under a running
target leaves services on the replaced inode, and because handlers are
queue-group workers a skewed node answers a fraction of requests with
the old behaviour, which reads as an intermittent fault.
* fix: drain guests on a plain target stop, not just restart
`systemctl is-system-running` cannot tell a plain `systemctl stop
spinifex.target` apart from the stop half of a restart -- both report
`running`, and the old `--only-if-host-stopping` gate skipped the drain
for both. That left guests running with no QMP shutdown and no volume
unmount while predastore/viperblock stopped underneath them, so a real
stop discarded any writes still in flight.
Add a second signal: `systemctl list-jobs` shows a pending START job for
spinifex.target during a restart (the target's stop leg resolves
instantly, since it owns no processes) or a pending RESTART job for
spinifex-shutdown.service itself when that unit is restarted directly
(PartOf propagates target-to-member only). Renamed the flag to
`--unless-restarting` to match: drain on a real shutdown or a plain
stop, skip only when one of those jobs shows a restart is already
queued. Any step systemctl can't answer fails toward draining, and a
skipped drain now warns loudly (naming each instance still running) so
an operator sees the risk in the journal instead of discovering it via
`ps`. `--only-if-host-stopping` stays as a deprecated, still-functional
alias so a partial upgrade between unit file and binary doesn't break.
* fix: retry the local drain GATE/DRAIN ACK instead of failing on first miss
A drain that fails at GATE is indistinguishable from a skipped drain from
the guest's perspective: it keeps running while storage goes away
underneath it. Stopping spinifex.target shortly after a restart hit exactly
this: the daemon had not finished re-subscribing to NATS, so the ACK
request came back with zero subscribers and NATS bounced it back near-
instantly ("no responders"), which is why the failure logged in about a
second despite --timeout=120s. collectLocalShutdownACK now retries with
backoff bounded by the phase timeout instead of failing on the first bounce,
and an exhausted retry budget logs at ERROR and names the guests still
running via a severity-aware refactor of warnIfGuestsLeftRunning, so a
failed drain is unmistakable in the journal rather than silent.
* fix: reap idle nbdkit concurrently and re-derive viperblock's stop budget
KillMode=process made viperblockd the sole reaper of nbdkit on SIGTERM, but
shutdownVolumes reaped them one at a time and TimeoutStopSec was never
re-derived for that change. With several mounted volumes the serial loop
blew past the 30s budget, systemd SIGKILLed viperblockd mid-loop, and
whichever nbdkit hadn't been reaped yet was orphaned.
Fan the per-volume utils.KillProcess calls out concurrently so the reap
costs one worst-case nbdkit, not N of them summed, and raise
TimeoutStopSec to 150s to strictly exceed that single-volume worst case
(120s KillProcess grace + nbdkit's own 20s drain + headroom). Add an
invariant asserting the two stay in that relationship, and a test proving
the reap is actually concurrent (fails against the old serial loop).
* docs: document the host and guest lifecycle contract
States what happens to running guests for every host lifecycle event: reboot, target restart, target stop, daemon restart, daemon crash, power loss and a coordinated cluster shutdown. Records the older-release behaviour where a plain target stop left guests with no data path, and how to tell which behaviour a node has.
The port tables were wrong in both directions: inter-node traffic is QUIC over
UDP rather than TCP, and the shard range 9991-9993 no longer exists. Ports are
unique within a host rather than across the cluster, so every machine uses the
same three, and a single-node install opens only the S3 port because colocated
nodes talk over an in-process pipe.
The reference architectures scripted a move of node data out of a directory
layout that no longer exists, which would have broken any install following
them. The update guide now says object storage has no migration path.
Julian Sommer