mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
Ben Duncan
1afc7fa8aa
* fix: retry a recovery relaunch that lost a race with a starting dependency A guest drain-stopped by a coordinated cluster shutdown could be lost for good. The relaunch ran before JetStream was serving, the volume lease claim timed out, and the failure was recorded as permanent. The backoff this needed already existed. relaunchWithRetry retries five times, but only on ErrMountRetryable, and mountErrRetryable recognised exactly two viperblock state-load sentinels. A lease claim returning context deadline exceeded matched neither, so the mount was reported as permanently failed and the backoff never engaged once. - Classify a lease store that cannot answer as ErrLeaseStoreUnavailable, and include it in mountErrRetryable. A lease held by another node is deliberately excluded: that is an answer, not an outage. - Probe readiness rather than connectedness. checkViperblockReady tested natsConn.IsConnected(), which is true against a server whose JetStream is still starting; it now calls AccountInfo with a timeout. - Reconsider a latched failure. RetryRecoveryFailed relaunches instances parked in error by a dependency-unavailable reason, once the backing store is confirmed ready, for a bounded number of passes, then leaves them alone. Only mount-state and launch failures qualify; a failed reconnect or pre-relaunch hook can mean the guest itself needs looking at. Closes the e2e gap that made this feel covered. The node-failure tests stop the service units directly so guests keep running and recovery is a QMP re-attach to a live process. Nothing stopped a guest, so nothing relaunched one from a sealed volume against dependencies that were still starting. TestMultinodeClusterRestartResumesGuests does, including the wait for every spx process to exit that makes it a real restart rather than a stop that returned early. Beads mulga-v7kp4, mulga-kyb1d. * fix: stop every spx process by unit in the cluster-restart helper The first CI run of TestMultinodeClusterRestartResumesGuests failed at the process wait with "node1 still has 2 spx process(es)", and the same thing happened by hand on a prod node. Every service is the same binary, and the stop was driven from a hard-coded list. StopNode names seven units and omits spinifex-northstar and spinifex-qmp-collector, which is exactly the two survivors. The unit name now comes from the surviving process's cgroup, so the set that stops is whoever is still running the binary and the list cannot go stale when a service is added. StopNode itself is left alone: its list is deliberate for the hard outage it simulates, where the target's drain must not fire. Also marks recovery_retry_test.go as //test:in-package — the retry policy is unexported and the tests drive it through the seams restore_test.go uses. * fix: stop spinifex.target in the cluster-restart helper, not the units The second CI run reformed NATS to 2 peers of 3: node3's NATS was stopped and never started again. StopNode stops the service units directly and leaves spinifex.target active. That is right for the hard outage it simulates and wrong for a real shutdown — the target then reads as satisfied while its units are dead, and the later start raced a stop job that was still queued. The helper now runs what the operator procedure and update-nodes.sh run: stop spinifex.target, wait for every spx process to exit, and additionally wait for systemd to have no spinifex job left queued before starting. That last wait is the one that closes this race. * fix: wait for the right NATS peer count in the cluster-restart helper WaitNATSPeers counts remote peers, so a healthy three-node cluster reports two. Asking for three could never be satisfied, and the run that hit it had in fact restarted all three NATS correctly.