The watchdog's is-active guard cannot fire against spinifex-nats.service
(Type=simple reports active the instant the process forks, long before
JetStream replays its state), so it restarted a NATS that was still
recovering. Separately, any non-zero exit from spx js-probe was read as a
JetStream write failure, so a broken spx binary read as a cluster-wide
outage instead of an unrunnable tool.
Replaces the is-active guard with an age gate on
ExecMainStartTimestampMonotonic, adds a post-restart cooldown and a
restart-count escalation that gives up rather than looping, and teaches the
watchdog to treat js-probe exit 2 (and any exit code that isn't 0 or 1) as
inconclusive rather than as proof of failure. Bounds the unit's own start
with TimeoutStartSec=120, matching the existing TimeoutStopSec derivation.
Remove the duplicated chown/ownership/symlink block from the firstboot
script. setup.sh (called with INSTALL_SPINIFEX_TARBALL) already handles
service users, directory ownership, sudoers, systemd unit installation,
and enabling spinifex.target — the same steps as the online curl|bash
install.
Firstboot now follows the documented install flow directly:
1. setup.sh --tarball (air-gapped equivalent of curl|bash)
2. setup-ovn.sh --management
3. spx admin init
4. systemctl start spinifex.target
Also reduce wait-for-nats.sh timeout from 90s to 30s per reviewer feedback.
- chown /etc/spinifex and /var/lib/spinifex to spinifex:spinifex after
spx admin init so services running as the spinifex user can read configs
- add Environment=HOME=/root to firstboot unit to suppress $HOME warning
- increase wait-for-nats.sh timeout from 30s to 90s for slow startup
- banner now writes to both /dev/tty1 and /etc/motd (visible over SSH)
- add systemd.show_status=1 to GRUB cmdline for per-service boot feedback
- add --removable to grub-install EFI so fallback path EFI/BOOT/BOOTX64.EFI
is written (required for UTM and bare metal without NVRAM boot entries)
- add root password screen and manual NIC entry to installer TUI
- tee spx admin init output to /root/spinifex-credentials.txt with 0600 perms
Josh Siviour