The watchdog probed only healthz?js-enabled-only=true, which reports the
server-wide JetStream state. A per-message-block write error is recorded on
the block and returned on every later write, and nothing clears it short of
a restart -- the server-wide state is never touched, so healthz stays 200
while writes fail indefinitely after the disk recovers.
spx admin node js-probe round-trips a canary key through the cluster-state
bucket, which is the only operation that observes the latch. It probes that
bucket rather than one of its own because a dedicated bucket is a separate
stream that can be placed elsewhere and stay writable.
When NATS disables JetStream at runtime (e.g. a filesystem permission error
flushing stream state) it keeps the process alive and serves KV as read-only.
That is not a process failure, so spinifex-nats.service's Restart=on-failure
never fires, and KV-backed startup (awsgw IAM bucket init) stalls for minutes
until an operator restarts NATS by hand.
Add a JetStream liveness watchdog:
- build/scripts/nats-js-watchdog.sh probes the NATS monitoring endpoint
(http://127.0.0.1:8222/healthz?js-enabled-only=true). Only when the service
is active and JS fails every sample of a short multi-sample window (so
transient startup/quorum 503s are ignored) does it systemctl try-restart
spinifex-nats.
- spinifex-nats-watchdog.{service,timer} run it as a oneshot every 30s, with a
60s boot delay so cluster quorum settles first.
- spinifex.target Wants= the timer so it starts with the platform.
The NATS-internal permission race itself is a server quirk outside spinifex's
control; this restores service automatically when it recurs.
Remove the duplicated chown/ownership/symlink block from the firstboot
script. setup.sh (called with INSTALL_SPINIFEX_TARBALL) already handles
service users, directory ownership, sudoers, systemd unit installation,
and enabling spinifex.target — the same steps as the online curl|bash
install.
Firstboot now follows the documented install flow directly:
1. setup.sh --tarball (air-gapped equivalent of curl|bash)
2. setup-ovn.sh --management
3. spx admin init
4. systemctl start spinifex.target
Also reduce wait-for-nats.sh timeout from 90s to 30s per reviewer feedback.
- chown /etc/spinifex and /var/lib/spinifex to spinifex:spinifex after
spx admin init so services running as the spinifex user can read configs
- add Environment=HOME=/root to firstboot unit to suppress $HOME warning
- increase wait-for-nats.sh timeout from 30s to 90s for slow startup
- banner now writes to both /dev/tty1 and /etc/motd (visible over SSH)
- add systemd.show_status=1 to GRUB cmdline for per-service boot feedback
- add --removable to grub-install EFI so fallback path EFI/BOOT/BOOTX64.EFI
is written (required for UTM and bare metal without NVRAM boot entries)
- add root password screen and manual NIC entry to installer TUI
- tee spx admin init output to /root/spinifex-credentials.txt with 0600 perms
Josh Siviour