fix: stop lbagent heartbeats on positive LB deletion, and actually enable the JetStream watchdog (#643)

* fix(lbagent): stop heartbeating a load balancer once it is confirmed deleted

A LoadBalancerNotFound response is the gateway's positive confirmation that
the store has no record for the LB, not a connectivity failure — unlike a
dial error, timeout, or 5xx, which leave its existence unknown and must keep
retrying. Past the startup registration grace window (or after any prior
successful heartbeat), it can no longer be the record's own replication
race, so the agent now stops rather than retrying forever and filling the
log with heartbeat noise after teardown.

* fix(nats): enable the JetStream ENOSPC-latch watchdog timer

nats-server disables JetStream process-wide on ENOSPC (server/jetstream.go
handleOutOfSpace -> ShutdownJetStream) and never re-enables it itself —
that decision is nats-server's own internal state, not spinifex's, so a
restart is the only way back. The spinifex-nats-watchdog.timer + service
already probe /healthz?js-enabled-only=true and try-restart on a sustained
failure, but the timer's WantedBy=timers.target never took effect: neither
setup.sh nor firstboot enabled it, so the watchdog shipped inert and a full
disk needed a manual restart forever. Enable --now it alongside the rest of
the unit set.
This commit is contained in:
Josh Siviour
2026-08-02 08:21:46 +10:00
committed by GitHub
parent a3fb4cccec
commit dad5d702e1
5 changed files with 222 additions and 8 deletions
+4
View File
@@ -667,6 +667,10 @@ EOF
fi
$SUDO systemctl daemon-reload
$SUDO systemctl enable spinifex.target
# The unit file alone does not self-activate (WantedBy=timers.target only
# takes effect once enabled) — without this the JetStream ENOSPC-latch
# watchdog is inert and a full disk requires a manual restart forever.
$SUDO systemctl enable --now spinifex-nats-watchdog.timer
info "Systemd units installed and enabled (per-service users)"
}