4 Commits

Author	SHA1	Message	Date
Ben Duncan	8b3c706ea3	docs: rework the multi-node install guide, fix cross-doc links ... Multi-node install - Lead with the fact that the guide builds a three-node cluster, and say how to go beyond three. The callout sits inside ## Overview because the docs site discards everything above the first section heading. - Replace the OVN-only sizing prose with a table covering all three distributed layers, and move the OVN control-plane detail into the VPC networking guide, linked from here. - Add minimum and recommended hardware. - Move the object storage layout to the end as background; it was the second thing a reader hit. - Collapse the ISO conversion notes from a section into one skippable box, now that --force is unconditional in Step 4 and the formation port needs no manual intervention. - Expand verification with spx get nodes and spx top nodes, including what to look for in each. - Drop the install-node.sh shortcut (internal tooling) and the "adding a node later" section (no good story yet). Corrections found while writing - Servers 4+ were described as running no database. They run predastore and viperblock like every other node; only the OVN database cluster is capped at three. Fixed in four places. - Step 4 never needed the firewall down: spx admin init opens the formation port itself for the formation window. Only Step 3's peer-scoped OVN ports are affected. - `nft list table inet spinifex_filter | grep spinifex_peers` could never match — the peer list is an nft define, expanded at load time. Anyone following it would have seen empty output and assumed the worst. - Disambiguate "database": with predastore, NATS and OVN all in play it needed qualifying throughout. - Align the example output region with the AWS_REGION the guide exports. Cross-doc links - Rewrite relative and category-prefixed links to the canonical /docs/<slug> form. The site publishes docs flat, so ../foo/README.md and /docs/category/foo both 404. - Point the six links to unpublished docs at the source repo instead. - Fix four fragments written against GitHub's anchor algorithm.	2026-08-21 14:14:28 -05:00
Josh Siviour	2fe885441b	fix(shutdown): scope a local drain to its own node and stop counting failed phases as done (#780) ... * fix(shutdown): scope a local drain to its own node and stop counting failed phases as done A node's spinifex-shutdown.service publishes its GATE and DRAIN requests to the shared spinifex.cluster.shutdown.* subjects, which every daemon subscribes to as a fan-out. The node filter only decided which ACK the stopping node waited for, so by the time it applied, every other node had already stopped its gateway, UI and VPC daemon and powered down its guests. Stopping one node took down every guest in the cluster. Requests now carry the node they apply to, and a daemon drops one addressed elsewhere without ACKing. A cluster shutdown leaves the field empty, so it still reaches every node. The STORAGE phase also ran 'pkill -f nbdkit', which matches every nbdkit on the host: another node's backends on a shared host, and a live guest's backend that survived a daemon restart under KillMode=process. Cleanup now sweeps this node's own nbdkit-vol-*.pid files and signals a PID only while /proc still agrees it is nbdkit, so a recycled PID is never hit. Finally, the coordinator counted errored ACKs toward its per-phase quorum, so a node that failed to drain reported N/N and the run advanced into STORAGE and PERSIST with guests still running. Only clean ACKs count now, and any errored one aborts the phase unless --force. * test(shutdown): cover the phase gate, every handler's target check and the real proc paths Diff coverage came in at 52.9% against a 60% threshold. The gaps were the per-phase target checks beyond GATE, the real procComm and signalProcess bodies that every cleanup test stubs out, and the coordinator's abort branch, which sat inside runClusterShutdown where no unit test could reach it. The abort decision moves into shutdownPhaseOutcome, a pure function returning the completed count and the message to print, so the rule that an errored node does not count toward the quorum is testable on its own rather than only reachable through a full cluster shutdown.	2026-08-13 11:50:13 +10:00
Josh Siviour	6f4aa495e6	feat: reconcile systemd unit files on upgrade (#771) ... * feat: reconcile systemd unit files on upgrade Unit files were written once at install time by install_systemd() in scripts/setup.sh and never re-asserted. A node upgraded by swapping the spx binary kept whatever units it was installed with, so any fix that lives in a unit rather than in code never reached it. This is not theoretical. A node whose spinifex-viperblock.service predates KillMode=process defaults to KillMode=control-group, so on a target stop TimeoutStopSec expires mid-flush and systemd cgroup-SIGKILLs every nbdkit while guest QEMU survives in the daemon's own cgroup: the guest keeps running with no storage path. Units are now stamped with a spinifex-unit-version marker, embedded in the binary via a generated file, and reconciled by spx admin upgrade. Missing or version-stale units are replaced after a timestamped backup; a unit at the current version whose content differs is reported as operator-modified and never overwritten. Drop-ins under <unit>.d are untouched, so systemctl edit overrides survive. Nothing is restarted. A daemon-reload is enough for KillMode and TimeoutStopSec to apply to the next stop, so drift is corrected without disturbing a running guest. Adds --dry-run, --units-only and --skip-units to spx admin upgrade. * test: cover systemd reconcile error paths and the admin upgrade CLI flow Diff coverage on the unit-migration PR sat at 45.8%, below the 60% gate. Adds tests for reconcile.go's error/edge paths (unreadable unit, backup and write failures, daemon-reload failure, malformed version markers, downgrade-is-conflict) and drives admin_upgrade.go's CLI flow directly via an injectable exit and a decision matrix extracted into validateUpgradeFlags/upgradeSummary for testability.	2026-08-12 22:29:48 +10:00
Josh Siviour	4dca2c7b59	fix: drain guests on a plain systemctl stop spinifex.target (#766) ... * fix: drain guests on a plain target stop, not just restart `systemctl is-system-running` cannot tell a plain `systemctl stop spinifex.target` apart from the stop half of a restart -- both report `running`, and the old `--only-if-host-stopping` gate skipped the drain for both. That left guests running with no QMP shutdown and no volume unmount while predastore/viperblock stopped underneath them, so a real stop discarded any writes still in flight. Add a second signal: `systemctl list-jobs` shows a pending START job for spinifex.target during a restart (the target's stop leg resolves instantly, since it owns no processes) or a pending RESTART job for spinifex-shutdown.service itself when that unit is restarted directly (PartOf propagates target-to-member only). Renamed the flag to `--unless-restarting` to match: drain on a real shutdown or a plain stop, skip only when one of those jobs shows a restart is already queued. Any step systemctl can't answer fails toward draining, and a skipped drain now warns loudly (naming each instance still running) so an operator sees the risk in the journal instead of discovering it via `ps`. `--only-if-host-stopping` stays as a deprecated, still-functional alias so a partial upgrade between unit file and binary doesn't break. * fix: retry the local drain GATE/DRAIN ACK instead of failing on first miss A drain that fails at GATE is indistinguishable from a skipped drain from the guest's perspective: it keeps running while storage goes away underneath it. Stopping spinifex.target shortly after a restart hit exactly this: the daemon had not finished re-subscribing to NATS, so the ACK request came back with zero subscribers and NATS bounced it back near- instantly ("no responders"), which is why the failure logged in about a second despite --timeout=120s. collectLocalShutdownACK now retries with backoff bounded by the phase timeout instead of failing on the first bounce, and an exhausted retry budget logs at ERROR and names the guests still running via a severity-aware refactor of warnIfGuestsLeftRunning, so a failed drain is unmistakable in the journal rather than silent. * fix: reap idle nbdkit concurrently and re-derive viperblock's stop budget KillMode=process made viperblockd the sole reaper of nbdkit on SIGTERM, but shutdownVolumes reaped them one at a time and TimeoutStopSec was never re-derived for that change. With several mounted volumes the serial loop blew past the 30s budget, systemd SIGKILLed viperblockd mid-loop, and whichever nbdkit hadn't been reaped yet was orphaned. Fan the per-volume utils.KillProcess calls out concurrently so the reap costs one worst-case nbdkit, not N of them summed, and raise TimeoutStopSec to 150s to strictly exceed that single-volume worst case (120s KillProcess grace + nbdkit's own 20s drain + headroom). Add an invariant asserting the two stay in that relationship, and a test proving the reap is actually concurrent (fails against the old serial loop). * docs: document the host and guest lifecycle contract States what happens to running guests for every host lifecycle event: reboot, target restart, target stop, daemon restart, daemon crash, power loss and a coordinated cluster shutdown. Records the older-release behaviour where a plain target stop left guests with no data path, and how to tell which behaviour a node has.	2026-08-12 12:39:38 +10:00