mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
Josh Siviour
22171c44ab
* fix(vm): durably record terminate even when local state persistence fails finalizeTerminated bailed out of WriteTerminatedInstance whenever the transition to terminated hit a persistence-only failure (e.g. ENOSPC writing local state), even though the in-memory status had already reached terminated. That left the instance owned by nothing: StuckTerminateReaper skips it (no longer shutting-down) and TerminatedTeardownReaper never sees it (never reached the terminated KV bucket). Only an operator daemon restart recovered it. WriteTerminatedInstance is a JetStream KV write, independent of local disk space, so keep going into it whenever the in-memory status did reach terminated, distinguishing that case from a genuine invalid/raced transition. Once the disk has space again, teardown now completes on its own via the existing idempotent TerminatedTeardownReaper. * fix(systemd): cap per-service memory under spinifex-system.slice Every service under spinifex.slice ran with MemoryMax=infinity / MemoryHigh=infinity, so one runaway service could exhaust the whole slice budget and the slice-level OOM would then reap an arbitrary victim instead of the offender. Add per-unit MemoryHigh/MemoryMax as percentages of physical RAM (so the same units scale from a dev box to bare metal). spinifex-daemon.service still hosts every guest QEMU process in its own cgroup (no per-instance slice yet), so it gets by far the largest allowance, just under the slice's own 80%/90%, to avoid starving VM launches. predastore is the legitimately memory-hungry service and gets generous headroom above its idle footprint. The remaining infra services are small, single-purpose Go processes and get a shared conservative bound. * fix(viperblockd): remove local volume directory on ebs.delete DeleteVolume deleted the S3 data and notified viperblockd, but the ebs.delete handler only stopped the volume's goroutines, killed nbdkit, and removed the NBD socket — it never removed the local WAL/checkpoint cache directory under BaseDir. The only path that removed it (VB.Close's RemoveLocalFiles) only ran from the unmount handler's seal step, which -efi volumes never reach (they carry no durable data to seal) and which a main volume also skips whenever a prior seal failed (e.g. disk full, deliberately keeping local files for recovery). Since terminate always unmounts before deleting, and delete is permanent, retention had no remaining purpose by the time ebs.delete ran. Unconditionally remove BaseDir/<volume> in the ebs.delete handler regardless of mount-tracking state, closing the leak for both the root volume and its -efi companion. * fix(viperblockd): validate volume names before removing local state ebs.delete unmarshalled the volume name straight off the wire and handed it to os.RemoveAll: an empty name collapsed to BaseDir itself, and ".." components could escape it entirely. localVolumeDir now rejects an empty/invalid name or one that resolves outside BaseDir before any removal, and ebs.delete returns Success: false with an error instead of silently skipping. Applied the same gate to ebs.mount and ebs.config, the other two handlers that turn a wire-supplied volume name into a BaseDir-relative path. * chore: pin viperblock to the merged concurrent-open corruption fix The ansible build role builds with GOWORK=off, so the deployed binary links whatever go.mod pins rather than the go.work overlay. Move the pin to the revision carrying the per-volume flock and the WAL/state write fixes.