mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
Josh Siviour
2fe885441b
* fix(shutdown): scope a local drain to its own node and stop counting failed phases as done A node's spinifex-shutdown.service publishes its GATE and DRAIN requests to the shared spinifex.cluster.shutdown.* subjects, which every daemon subscribes to as a fan-out. The node filter only decided which ACK the stopping node waited for, so by the time it applied, every other node had already stopped its gateway, UI and VPC daemon and powered down its guests. Stopping one node took down every guest in the cluster. Requests now carry the node they apply to, and a daemon drops one addressed elsewhere without ACKing. A cluster shutdown leaves the field empty, so it still reaches every node. The STORAGE phase also ran 'pkill -f nbdkit', which matches every nbdkit on the host: another node's backends on a shared host, and a live guest's backend that survived a daemon restart under KillMode=process. Cleanup now sweeps this node's own nbdkit-vol-*.pid files and signals a PID only while /proc still agrees it is nbdkit, so a recycled PID is never hit. Finally, the coordinator counted errored ACKs toward its per-phase quorum, so a node that failed to drain reported N/N and the run advanced into STORAGE and PERSIST with guests still running. Only clean ACKs count now, and any errored one aborts the phase unless --force. * test(shutdown): cover the phase gate, every handler's target check and the real proc paths Diff coverage came in at 52.9% against a 60% threshold. The gaps were the per-phase target checks beyond GATE, the real procComm and signalProcess bodies that every cleanup test stubs out, and the coordinator's abort branch, which sat inside runClusterShutdown where no unit test could reach it. The abort decision moves into shutdownPhaseOutcome, a pure function returning the completed count and the message to print, so the rule that an errored node does not count toward the quorum is testable on its own rather than only reachable through a full cluster shutdown.
Spinifex Documentation
This directory contains the Spinifex documentation. This is available in individual markdown files, and a generated website to aid navigation and browsing documentation at: https://docs.mulgadc.com
The AWS model operation coverage report is generated from Spinifex's dispatch tables and the pinned AWS API models. Run make aws-model-coverage to regenerate and print it.