* feat: reconcile systemd unit files on upgrade
Unit files were written once at install time by install_systemd() in
scripts/setup.sh and never re-asserted. A node upgraded by swapping the
spx binary kept whatever units it was installed with, so any fix that
lives in a unit rather than in code never reached it.
This is not theoretical. A node whose spinifex-viperblock.service
predates KillMode=process defaults to KillMode=control-group, so on a
target stop TimeoutStopSec expires mid-flush and systemd cgroup-SIGKILLs
every nbdkit while guest QEMU survives in the daemon's own cgroup: the
guest keeps running with no storage path.
Units are now stamped with a spinifex-unit-version marker, embedded in
the binary via a generated file, and reconciled by spx admin upgrade.
Missing or version-stale units are replaced after a timestamped backup;
a unit at the current version whose content differs is reported as
operator-modified and never overwritten. Drop-ins under <unit>.d are
untouched, so systemctl edit overrides survive.
Nothing is restarted. A daemon-reload is enough for KillMode and
TimeoutStopSec to apply to the next stop, so drift is corrected without
disturbing a running guest.
Adds --dry-run, --units-only and --skip-units to spx admin upgrade.
* test: cover systemd reconcile error paths and the admin upgrade CLI flow
Diff coverage on the unit-migration PR sat at 45.8%, below the 60% gate.
Adds tests for reconcile.go's error/edge paths (unreadable unit, backup
and write failures, daemon-reload failure, malformed version markers,
downgrade-is-conflict) and drives admin_upgrade.go's CLI flow directly
via an injectable exit and a decision matrix extracted into
validateUpgradeFlags/upgradeSummary for testability.
* feat(daemon): live MemAvailable admission gate + fixed-services reserve scope
Admission previously charged only guest -m against a flat reserve, blind to the
co-located storage VMs, nbdkit, and daemon stack — so it greenlit launches that
overcommit real RAM and OOM-kill guests on memory-tight hosts.
Add a second admission gate that clamps the accounting-derived launch count by
live /proc/meminfo MemAvailable (which already nets reclaimable cache and sees
every untracked consumer): admitted = min(accounting, (MemAvailable-reserve)/
guestMem). Reads live per admission, fails open on a /proc read error so a
transient fault can't wedge launches, and is disabled by SPINIFEX_ADMISSION_LIVE_MEM=0.
Re-scope the host reserve to the fixed infrastructure tier only; per-instance
nbdkit is no longer expected to fit inside it (explicit per-instance charge is a
follow-up; the live gate covers it implicitly in the meantime).
Pure helpers (liveMemCount, parseMemAvailableKB, liveMemAdmissionEnabled) are
unit-tested; liveMemGate covered via an injected reader.
* feat(daemon): charge nbdkit per-instance in admission accounting
An instance's admission memory charge is now its guest -m plus the nbdkit
processes backing its volumes (one per attached volume), not guest -m alone.
The static reserve no longer has to absorb per-instance nbdkit, which was
unsizable and the root of the host-memory-overcommit OOM.
Per-volume charge is measured per class (main 768 MiB: 128 cache + plugin
working set + in-flight I/O; aux 96 MiB: process floor, 0 cache), env-tunable
via SPINIFEX_NBDKIT_MAIN_MIB / SPINIFEX_NBDKIT_AUX_MIB. Default layout is one
main + two aux volumes (root + cloudinit + efi) ~= 0.94 GB/instance. Applied at
every accounting site through instanceMemChargeMiB so charge and release never
drift; the live MemAvailable gate backstops block-device-mapping volumes the
static count misses.
* feat(vm): three-tier guest OOM ladder + spinifex-guests.slice
Replace the flat oom_score_adj=500 on every guest QEMU with a ManagedBy-derived
ladder: user guests (ManagedBy=='') stay +500 and are reaped first under host
memory pressure; system instances (ELBv2 LB-VMs, EKS control-plane nodes,
ManagedBy!='') score 0, ranking above user guests but below the infra tier
(-500). Killing a system instance degrades the data path or cluster control
plane for many user VMs, so it must outlive them.
Add spinifex-guests.slice as a sibling of spinifex-system under spinifex.slice —
the parent tier for the per-instance leaves that land with RG-7. Guest QEMU
placement into the slice is deferred to the per-instance work.
* refactor(systemd): lean unit comments to RG-n tags + RG-9/RG-11 invariants
Strip the multi-paragraph rationale novels from spinifex-daemon.service (48->7
comment lines), spinifex-vpcd.service (29->8) and spinifex.slice (15->5),
replacing them with settings + terse # RG-n clause tags. Remove plan-doc, bead
and CI-run references from comments (project policy). Directives are byte-for-byte
unchanged — comment-only.
Add RG-9/RG-11 invariant tests asserting per-tier confinement (locked-down
storage/control tier drops all caps + NoNewPrivileges=yes; daemon keeps exactly
CAP_SYS_ADMIN+CAP_DAC_OVERRIDE and the four-entry DeviceAllow; vpcd is the
documented network-tier exception) and the lean-comment budget + no-plan-ref rule.
* feat(systemd): bound spinifex to a resource-limited slice
Add a spinifex.slice parent ceiling (MemoryHigh=80%, MemoryMax=90%,
MemorySwapMax=0) with a spinifex-system.slice child holding the seven
control/infra services, so a runaway hits the slice instead of the host.
A system.slice drop-in (MemoryMin=1G, CPUWeight=300) reserves RAM and CPU
for sshd/journald/the operator, keeping the box reachable when spinifex is
maxed.
The drop-in is generated in setup.sh install_systemd() rather than shipped
as a staged file, since the packaging globs flatten the systemd/ dir; this
mirrors the existing sshd-keygen drop-in pattern. The two .slice files flow
through every staging path unchanged.
Within the single effective cgroup, a slice MemoryMax OOM picks the highest
oom_score victim — guest QEMU (+500) before any service (-500) — so a slice
OOM degrades by killing one guest while the daemon, storage tier, NATS, and
the host all survive.
* feat(setup): provision 8G host swapfile + vm.swappiness=10
Add a setup_swap stage that creates an 8G /swapfile, enables it, persists it
to fstab, and writes vm.swappiness=10 so the kernel only swaps under real
pressure. Backs spinifex.slice MemorySwapMax so the slice can spill to swap
for graceful degradation instead of OOMing a guest. Idempotent; skipped under
ISO_BUILD (firstboot provisions swap on ISO hosts). Reverses the prior swap=0
assumption.
Josh Siviour