mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
Josh Siviour
ab1b1bacf6
When NATS disables JetStream at runtime (e.g. a filesystem permission error flushing stream state) it keeps the process alive and serves KV as read-only. That is not a process failure, so spinifex-nats.service's Restart=on-failure never fires, and KV-backed startup (awsgw IAM bucket init) stalls for minutes until an operator restarts NATS by hand. Add a JetStream liveness watchdog: - build/scripts/nats-js-watchdog.sh probes the NATS monitoring endpoint (http://127.0.0.1:8222/healthz?js-enabled-only=true). Only when the service is active and JS fails every sample of a short multi-sample window (so transient startup/quorum 503s are ignored) does it systemctl try-restart spinifex-nats. - spinifex-nats-watchdog.{service,timer} run it as a oneshot every 30s, with a 60s boot delay so cluster quorum settles first. - spinifex.target Wants= the timer so it starts with the platform. The NATS-internal permission race itself is a server quirk outside spinifex's control; this restores service automatically when it recurs.