* feat: accept and surface gp3 Throughput on CreateVolume/DescribeVolumes
Honors input.Throughput (125-1000 MiB/s, default 125 when omitted) on
CreateVolume, persists it to VolumeMetadata, and surfaces it via
DescribeVolumes (getVolumeByID), mirroring the Iops handling shipped
in mulga-siv-380. Pins viperblock to the commit that adds the
VolumeMetadata.Throughput field.
Zero Throughput on a read volume means it predates this field; it is
omitted from the response rather than surfaced as a misleading 0
MiB/s, matching the existing IOPS convention.
* chore: pin viperblock to the merged gp3 Throughput commit
* fix: stop viper AutomaticEnv from shadowing predastore's own env vars
Viper resolves AutomaticEnv before explicit BindEnv, so predastore's
bare "config-path", "host" and "base-path" viper keys auto-derived to
SPINIFEX_CONFIG_PATH, SPINIFEX_HOST and SPINIFEX_BASE_PATH -
coincidentally the same names used elsewhere (the global cluster
config path, and root's global --host override). Setting
SPINIFEX_CONFIG_PATH for the cluster config (needed by
derivePredastoreBind) silently made predastore parse the cluster
config as its own S3 config, breaking auth for every signed request.
Rename predastore's host/base-path/config-path viper keys to
"predastore-"-prefixed keys whose AutomaticEnv-derived name is
identical to their explicit BindEnv target, so no shadow is possible
regardless of what else is set in the environment.
* style: trim predastore env-binding comments to the 3-line limit
* ci: skip building unused E2E suite binaries in e2e.yml's single/multi jobs
These two jobs immediately rebuild suite-aware binaries after Build
Install Artifacts and never consume the tar's _bin contents, so the
cross-compiled suite build was pure waste. build-install-artifacts.sh
gains an optional --skip-e2e-build flag (positional contract unchanged);
e2e-nightly.yml and the other callers keep building the full set.
* refactor: replace hand-rolled IMDS client with AWS SDK v2
internal/imdscreds, cmd/ecs-agent's metadata fetch and its credentials
provider used a hand-written IMDSv2 token/GET flow with manual RFC3339
expiry parsing. A malformed Expiration silently fell back to "never
expires" instead of failing.
Replace it with feature/ec2/imds, credentials/ec2rolecreds and
aws.CredentialsCache, following the internal/gwsign/signer.go
precedent. An unparseable Expiration is now a hard error from Fetch,
covered by TestFetch_BadExpirationErrors. Output shapes to callers are
unchanged, and IMDS base URLs ending in /latest keep working.
* refactor: derive predastore bind config from spinifex.toml in spx itself
predastore-start.sh awk-parsed spinifex.toml for the bind host/port and
node_id, exported them as env vars, then exec'd spx. Move that
derivation into predastoreStartCmd via derivePredastoreBind, which
reads the raw (non-DIAL-normalized) nodes.<node>.predastore section,
defaulting node_id to -1 (co-located) when the key is absent, same as
the script. An explicit --host/--port/--node-id flag or
SPINIFEX_PREDASTORE_* env var still overrides the derived value.
Delete the now-unused script and point the systemd unit's ExecStart
directly at spx service predastore start, matching the
spinifex-viperblock.service precedent.
* feat: close IMDS and vm_adapters trace-linking gaps
Wrap the IMDS mux with otelsetup.HTTPMiddleware (outermost, ahead of
rejectForwarded) so guest metadata requests parent their DescribeInstances/
Attribute NATS fan-out instead of rooting a trace per call, including
requests the SSRF guard rejects.
Route daemon/vm_adapters.go's ebs.mount/ebs.unmount NATS requests through a
new ebsRequestWithTrace helper that opens a client span and injects it into
the message headers, so viperblockd's consumer spans join the producer trace
instead of rooting their own.
* feat: trace the ebs.delete request from the instance cleaner
* fix: preserve running guests across target restart, harden state write
spinifex-shutdown.service's ExecStop fired on any spinifex.target stop,
including the stop half of `systemctl restart spinifex.target`, draining
every guest via QMP even though the host never went down. Gate the drain
behind `systemctl is-system-running` == "stopping", which only holds once
systemd is genuinely unwinding into shutdown.target (reboot/poweroff), so
a target restart or plain `systemctl stop` now leaves QEMU/nbdkit running
(KillMode=process) for the daemon to reconnect to on the way back up.
Daemon.WriteState returned as soon as the local state file write failed,
skipping the independent JetStream KV write entirely. A crash in that
window left a live VM with no durable record anywhere. Attempt the KV
write regardless of local disk health, then still surface the local
write error to callers, mirroring finalizeTerminated's handling of the
same fault class on the terminate path.
* fix: gate the shutdown drain via an spx flag, not a shell script
Replace the shutdown-drain-gate.sh gate with a --only-if-host-stopping
flag on `spx admin node drain`. The unit now points at /usr/local/bin/spx
directly (already required for the node to function) instead of a script
installed under the data directory, which a partial rebuild of that
directory could silently drop without ever failing ExecStop. The gate
logic also becomes a plain Go function with injectable state lookup, so
it gets real table-driven coverage instead of only a live reboot.
hostIsStopping() drains only on a literal "stopping" from `systemctl
is-system-running`. Every other recognized state (running/degraded/...)
and any unrecognized output skip the drain; a fully unreachable systemctl
fails toward draining, since a skipped drain on a real shutdown hard-kills
guests and loses unflushed writes, while a spurious drain on a restart
only costs an extra graceful stop/relaunch cycle. Run by hand without the
flag, the command still drains unconditionally as it always has.
* refactor: split the shutdown-gate comments to sit with the code they explain
* fix(vm): durably record terminate even when local state persistence fails
finalizeTerminated bailed out of WriteTerminatedInstance whenever the
transition to terminated hit a persistence-only failure (e.g. ENOSPC writing
local state), even though the in-memory status had already reached
terminated. That left the instance owned by nothing: StuckTerminateReaper
skips it (no longer shutting-down) and TerminatedTeardownReaper never sees
it (never reached the terminated KV bucket). Only an operator daemon restart
recovered it.
WriteTerminatedInstance is a JetStream KV write, independent of local disk
space, so keep going into it whenever the in-memory status did reach
terminated, distinguishing that case from a genuine invalid/raced
transition. Once the disk has space again, teardown now completes on its
own via the existing idempotent TerminatedTeardownReaper.
* fix(systemd): cap per-service memory under spinifex-system.slice
Every service under spinifex.slice ran with MemoryMax=infinity /
MemoryHigh=infinity, so one runaway service could exhaust the whole slice
budget and the slice-level OOM would then reap an arbitrary victim instead
of the offender.
Add per-unit MemoryHigh/MemoryMax as percentages of physical RAM (so the
same units scale from a dev box to bare metal). spinifex-daemon.service
still hosts every guest QEMU process in its own cgroup (no per-instance
slice yet), so it gets by far the largest allowance, just under the slice's
own 80%/90%, to avoid starving VM launches. predastore is the legitimately
memory-hungry service and gets generous headroom above its idle footprint.
The remaining infra services are small, single-purpose Go processes and get
a shared conservative bound.
* fix(viperblockd): remove local volume directory on ebs.delete
DeleteVolume deleted the S3 data and notified viperblockd, but the ebs.delete
handler only stopped the volume's goroutines, killed nbdkit, and removed the
NBD socket — it never removed the local WAL/checkpoint cache directory under
BaseDir. The only path that removed it (VB.Close's RemoveLocalFiles) only
ran from the unmount handler's seal step, which -efi volumes never reach
(they carry no durable data to seal) and which a main volume also skips
whenever a prior seal failed (e.g. disk full, deliberately keeping local
files for recovery). Since terminate always unmounts before deleting, and
delete is permanent, retention had no remaining purpose by the time
ebs.delete ran.
Unconditionally remove BaseDir/<volume> in the ebs.delete handler regardless
of mount-tracking state, closing the leak for both the root volume and its
-efi companion.
* fix(viperblockd): validate volume names before removing local state
ebs.delete unmarshalled the volume name straight off the wire and
handed it to os.RemoveAll: an empty name collapsed to BaseDir itself,
and ".." components could escape it entirely. localVolumeDir now
rejects an empty/invalid name or one that resolves outside BaseDir
before any removal, and ebs.delete returns Success: false with an
error instead of silently skipping. Applied the same gate to
ebs.mount and ebs.config, the other two handlers that turn a
wire-supplied volume name into a BaseDir-relative path.
* chore: pin viperblock to the merged concurrent-open corruption fix
The ansible build role builds with GOWORK=off, so the deployed binary links
whatever go.mod pins rather than the go.work overlay. Move the pin to the
revision carrying the per-volume flock and the WAL/state write fixes.
* fix(vpc): close the route-table and ENI teardown leaks
DeleteVpc validated subnets and non-default security groups but never
non-main route tables, so it deleted the VPC out from under them and left
records naming a VPC that no longer resolves. It now returns
DependencyViolation, alongside the existing checks and before any cascade,
and still cascades the main route table on its own.
DeleteSubnet left route table associations naming the deleted subnet.
DeleteRouteTable blocks on any non-main association carrying a subnet ID,
and that subnet could never be deleted again to clear it, so the table was
pinned forever. Associations are now cleared before the subnet record goes,
leaving a mid-way failure retryable. routetable imports vpc, so the record
cannot be typed here; only the associations key is rewritten, and every
other field is passed through untouched.
DeleteLoadBalancer detached an ENI and then re-read it to delete, where a
lagging R3 replica could report it still in use. The failure was logged at
WARN and the LB record deleted anyway, orphaning the ENI and its security
group. DetachAndDeleteENI now reads once and carries the detach revision
into the delete via LastRevision, retrying the whole flow from a fresh read
on a conflict, bounded at 3 attempts. The IPAM release happens after the
delete, not before, so a retry cannot hand the same IP back twice. When
ENIs survive, the LB record is kept naming them and the call returns
DependencyViolation rather than dropping the only reference to the leak.
Instance termination routes through the same call, and both ENI paths now
distinguish a real delete from an already-absent one instead of logging a
deletion that did not happen.
TestRLC5_NoOrphanedRouteTableAfterSubnetThenVpcTeardown covers the ordering
that defeated the per-operation tests: subnet first, then VPC.
* fix(elbv2): route SetSubnets ENI teardown through the atomic detach+delete
SetSubnets detached every ENI in one loop, then deleted the removed subnets'
ENIs with force=false in another. That is the same shape as the
DeleteLoadBalancer leak: the delete re-reads the ENI, a lagging replica can
still show it attached, and the delete fails as in-use while the LB moves on
without it.
Removed subnets now go through DetachAndDeleteENI, which carries the detach
revision into the delete under one read. ENIs that survive the reshape are
still detached on their own, since they are re-attached to the relaunched VM.
isProductionLayout() keyed off /etc/spinifex alone. A node reset legitimately
removes that directory, so spx silently fell back to the dev layout and rebuilt
the whole node — configs, keys, nats.conf — under the invoking user's home while
the systemd units kept reading /etc/spinifex. Cluster formation reported success
and every service then failed to start.
Detection now also accepts the installed systemd target, which setup.sh writes
and a reset never touches, so the layout survives the reset that caused this.
init and join no longer mkdir the production config dir either. Only setup.sh
can rebuild the per-service subtree and its ownership, so recreating the top
level would still leave a node that inits cleanly and dies at service start.
They now refuse with the remedy instead.
/var/log/spinifex is 0775 root:spinifex so each service user can write its own
log. logrotate refuses to rotate inside a group-writable directory unless the
config says which user to rotate as, so every entry was skipped and the logs
grew without bound.
Verified on env13: before, logrotate -d reported 'skipping
/var/log/spinifex/nats.log because parent directory has insecure permissions'
and nothing had ever rotated. After, the debug run is clean and forced
rotations produce nats.log.1 and a compressed nats.log.2.gz.
* fix(gateway): log the two SigV4 paths that reject silently
Three code paths answer SignatureDoesNotMatch, but only the Verify mismatch
logged anything. A skewed request timestamp and an unsupported service in the
credential scope both returned the same wire error with no log line at all, so
they were indistinguishable from a canonicalisation defect after the fact.
Both now log at Warn. The skew path records the client and server timestamps
plus the window, which is what separates clock drift from a signing bug.
Adds signingTime, since Parse rejects a skewed request before returning a
SignedRequest, leaving the raw header or presigned query arg as the only copy
of the timestamp available to log.
* fix(gateway): log the malformed-envelope rejection too
The parse-error default branch answers IncompleteSignature and was the last
auth rejection path with no log line, so a malformed Authorization header,
unsupported algorithm or missing content hash left no record of which
validation stage rejected the request. The parse error names it.
shutdownVolumes deliberately spares nbdkit that a guest is still writing
through, because tearing one out mid-write corrupts that guest's filesystem.
The unit left KillMode at the default, so the cgroup SIGTERMed those children
anyway and SIGKILLed them at TimeoutStopSec -- causing both a 30s stop stall
and the corruption the skip exists to prevent.
KillMode=process matches spinifex-daemon, where guest QEMU already survives a
restart for DDIL reattach. Idle nbdkit is still reaped explicitly.
Measured on env12 with a guest attached: control-group stopped in 30.08s with
'Failed with result timeout' and SIGKILLed nbdkit, leaving QEMU running with
zero ESTAB nbd sockets. KillMode=process stops sub-second, both nbdkit
survive, and the guest keeps its device.
* fix(ec2): drain the attached volume before CreateImage reads a live checkpoint
CreateImage against a running instance (--no-reboot) read the volume's
checkpoint directly from predastore without draining in-flight writes
first, unlike CreateSnapshot. A checkpoint captured before a guest-
triggered write (e.g. growpart rewriting the GPT after a root volume
is launched larger than the base image) could be stale, producing an
AMI that boots to a UEFI shell instead of the OS.
Extracts the existing snapshot-package drain logic into a shared,
reusable DrainVolume so both live-snapshot call sites route through
the same drain-before-read guarantee instead of being able to diverge.
* test(e2e): boot-verify CreateImage AMIs instead of trusting State=available
describe-images reporting State=available is not evidence an AMI
boots — a well-formed but unbootable AMI (stuck at the UEFI shell)
still registers cleanly. Extend the CreateImage e2e test to launch a
probe instance from each produced AMI (running-path and stopped-path)
and require a real SSH login, with console-dump-on-failure wired up
for diagnosis.
* fix(images): sparsify the Alpine guestfish .raw before conversion
The Alpine build path grows the qcow2 in place via qemu-img resize +
guestfish resize2fs, which can leave metadata scattered across the
newly-grown region. virt-sparsify zeroes the guest's unused blocks
before the raw conversion so they collapse into real holes, matching
the Ubuntu virt-resize path's sparseness.
DescribeInstances silently returned an empty Reservations list for an
instance ID that never existed, so callers could not distinguish "gone"
from "never existed". Mirror the two-pass found-set pattern from
DescribeSecurityGroups, but only assert absence when the NATS fan-out and
both stopped/terminated KV bucket queries are provably complete — a
node timing out during the fan-out must never produce a false NotFound
for an instance that actually exists on that node.
The assertion is split into a new DescribeInstancesChecked, used only by
the gateway's customer-facing action. The plain DescribeInstances stays
silent for internal callers (IMDS instance lookup, the RDS instance-state
resolver, the EKS control-plane reconciler) that treat "not found" as
"not currently present" rather than a fault.
The watchdog probed only healthz?js-enabled-only=true, which reports the
server-wide JetStream state. A per-message-block write error is recorded on
the block and returned on every later write, and nothing clears it short of
a restart -- the server-wide state is never touched, so healthz stays 200
while writes fail indefinitely after the disk recovers.
spx admin node js-probe round-trips a canary key through the cluster-state
bucket, which is the only operation that observes the latch. It probes that
bucket rather than one of its own because a dedicated bucket is a separate
stream that can be placed elsewhere and stay writable.
* fix: stop passing nbdkit S3 credentials as literal argv
access_key/secret_key were interpolated into the nbdkit plugin argv,
which is world-readable via /proc/<pid>/cmdline for the life of the
process. Credentials now travel via cmd.Env (VB_ACCESS_KEY/
VB_SECRET_KEY) instead. Requires a viperblock nbdkit plugin build that
reads these environment variables; inert against an older plugin
until that pin moves.
* chore: pin viperblock to the VB_ACCESS_KEY/VB_SECRET_KEY fallback
The spinifex half sets these env vars on the nbdkit child, but the
deploy builds with GOWORK=off, so viperblock's matching read of them
only ships once go.mod points at 8a4787a.
* fix(lbagent): stop heartbeating a load balancer once it is confirmed deleted
A LoadBalancerNotFound response is the gateway's positive confirmation that
the store has no record for the LB, not a connectivity failure — unlike a
dial error, timeout, or 5xx, which leave its existence unknown and must keep
retrying. Past the startup registration grace window (or after any prior
successful heartbeat), it can no longer be the record's own replication
race, so the agent now stops rather than retrying forever and filling the
log with heartbeat noise after teardown.
* fix(nats): enable the JetStream ENOSPC-latch watchdog timer
nats-server disables JetStream process-wide on ENOSPC (server/jetstream.go
handleOutOfSpace -> ShutdownJetStream) and never re-enables it itself —
that decision is nats-server's own internal state, not spinifex's, so a
restart is the only way back. The spinifex-nats-watchdog.timer + service
already probe /healthz?js-enabled-only=true and try-restart on a sustained
failure, but the timer's WantedBy=timers.target never took effect: neither
setup.sh nor firstboot enabled it, so the watchdog shipped inert and a full
disk needed a manual restart forever. Enable --now it alongside the rest of
the unit set.
* fix: bound EKS DELETING re-drive with backoff and a terminal give-up
reapCluster re-drove purgeClusterInfra on every GC tick with no attempt
tracking, so a permanently-failing teardown (e.g. an unretriable
DependencyViolation) was retried forever. Track attempt count and last
attempt time on ClusterMeta, back off exponentially between re-drives,
and give up after maxDeleteReapAttempts with a single error-level log.
Status stays DELETING on give-up so the cluster's infra remains tracked
and billable pending operator intervention, preserving the ADR-0006 §6
billable-before-sweep invariant.
* fix: create the EKS e2e node role independently in each subtest
runEBSCSIVolume string-built the worker node role ARN without ever
creating it, only passing because runIRSAPod's t.Cleanup discarded
DeleteRole's error while the role was still attached to its instance
profile, leaving it behind for EBSCSIVolume to reuse by naming
convention. Extract ensureNodeRole and call it independently from both
subtests, detaching the role from its instance profile before deleting
it and asserting that error instead of discarding it.
* fix: reclaim CSI-provisioned EBS volumes on EKS cluster delete
purgeClusterInfra tore down the control plane, nodegroups, and
networking but never looked at EBS volumes, so any volume the
in-cluster CSI driver had provisioned for a PVC was left behind
untracked and unbilled for. Add a report-only reclaim step that scans
for volumes tagged with the cluster's CSI owner tag and logs them for
operator follow-up.
Volumes are never auto-deleted: a PV's reclaim policy (Delete vs
Retain) is a Kubernetes-side property the CSI driver does not tag onto
the EC2 volume, so there is no safe way to tell from tags alone
whether deleting a given volume is intended. This mirrors the
volume-leak reaper's existing mark-and-alarm policy for the same
reason.
The cluster-owner tag requires the EBS CSI controller's
--k8s-tag-cluster-id flag, added here to the controller manifest;
without it a CSI volume carries no cluster-scoped tag and reclaim
would have to guess across every cluster sharing the account.
* fix: persist the last purge error on a wedged DELETING cluster
DeleteReapExhausted told an operator a cluster's teardown had given
up, but not why: nothing recorded the failing purge's error.
Persist it on every failed re-drive, not only once the reaper
exhausts its attempts, so it is visible on a still-retrying cluster
too. Status stays DELETING throughout, preserving the billing
invariant that untorn-down infra remains tracked.
* fix: make the EKS e2e suite confirm its CSI volume is actually deleted
The EBSCSIVolume subtest deleted its PVC on cleanup but never
confirmed the underlying volume was reclaimed, so a CSI driver that
silently failed to delete it would leak a volume with nobody
noticing. The suite provisions and owns this volume, so unlike
production teardown it can safely wait on its StorageClass's Delete
reclaim policy: capture the CSI volumeHandle while the PVC is bound,
then have cleanup poll EC2 for that exact volume to disappear before
the addon/nodegroup that run the CSI controller are torn down.
* test: gate scripts/images/ shell suites + shellcheck in CI
The eks-node/rds-postgres *_test.sh suites and shellcheck only ever ran
by hand: spinifex.yml filters pushes on Go-only paths and no job
invoked them, so a broken helper could land unnoticed. Add a
`make test-images` target and a path-filtered image-scripts.yml
workflow so both run automatically on scripts/images/** and images/**
changes, kept separate from preflight so image-asset churn doesn't run
on every Go contributor's commit. Also clears the shellcheck findings
that would have started this gate red (SC1090 on a runtime-resolved
source path, SC2154 x2 on eval-assigned variables shellcheck can't trace).
* test: pin vCPU quota gate ordering in RequestSpotInstances
Nothing asserted that RequestSpotInstances runs the vCPU quota gate
before dispatching capacity or launching a VM, so a regression that
dropped or reordered EnforceLaunch would pass silently. Add an
ordering test using a real quota.Service (embedded JetStream KV) and
real NATS responders standing in for the capacity and launch daemons,
recording a global call sequence and asserting the gate's index
precedes both. Verified by removing the gate: the first quota-KV read
then comes from the post-launch charge instead, landing after capacity
dispatch and launch, and the ordering assertion fails as expected.
* fix(systemd): give every unit an explicit TimeoutStopSec
9 of 12 spinifex systemd units had no TimeoutStopSec and inherited
systemd's 90s default, which is either far more or far less time than
their own SIGTERM handling needs. Size each one to what its shutdown
path actually does: predastore matches viperblock's 30s WAL-drain
budget since viperblock flushes through it (After= already sequences
viperblock's stop first; this is the backstop if that's ever bypassed),
spinifex-ui covers its 30s internal shutdown context plus margin for
long-lived streaming proxy connections, nats covers its own
JetStream-shutdown/raft-stepdown/client-drain signal handler, and
services with no SIGTERM handler at all (awsgw) get a short bound
since they exit immediately regardless.
Extends the existing unit-file invariant test with a check that every
.service file declares TimeoutStopSec, so a newly added unit that
forgets one fails CI instead of silently falling back to 90s.
* fix(vm): reconcile QEMU processes with no instance record on restore
classifyRestoredInstances only walks Manager.Snapshot(), so a
qemu-system process with no instance record at all -- a crash
mid-launch, a lost KV write, a stale process surviving a wiped state
file -- was never handed to killOrphanedQEMU and stayed permanently
invisible to recovery.
Add a host-process scan that diffs live qemu-system PIDs (matched via
/proc/<pid>/comm) against the PIDs claimed by known instances' own
-pidfile files. It only reports what it finds, never kills: this
daemon cannot distinguish "record not loaded yet" or "owned by another
daemon/tenant on this host" from a genuine orphan, and killing a
running customer VM to fix an invisibility gap is the wrong trade.
Runs at the end of Restore, once every known instance has been
classified and relaunched, so a live process still unaccounted for at
that point genuinely has no record anywhere.
* fix(vm): distinguish recordless QEMU orphans by pidfile ownership
A recordless qemu-system process with a stale spinifex pidfile is
positively this daemon's own orphan; one with no pidfile at all could
belong to another tenant on a shared host. Log the two cases distinctly
instead of one blanket warning, so an operator can tell them apart.
Also sandbox XDG_RUNTIME_DIR in the shared daemon test harness so
Restore()-driven tests never read this box's real runtime directory.
* fix(spinifexui): treat ErrServerClosed as a clean shutdown, not a failure
Serve returns http.ErrServerClosed after Shutdown is called deliberately
(the SIGTERM handler above does exactly that), but Start propagated it
as an error, making systemd log a graceful stop as exit code 1 and
Result: exit-code -- indistinguishable from a real crash.
Query-protocol services (EC2, IAM, STS...) get their action from the
SigV4 Action= form param before dispatch, so traceActionEnricher can
rename the span and set aws.action/aws.service from that context.
REST-JSON services (EKS, ECS, ACM, tagging, ECR, bedrock...) only
resolve their action inside their own dispatcher, well after that
middleware runs, so the enrichment never fired for them.
checkPolicyResource is the one call every dispatcher already routes
its resolved (service, action) pair through, so move the enrichment
there via a shared recordResolvedAction helper instead of duplicating
it per REST-JSON handler.
BytesFreed counted logical object sizes deleted from the S3 namespace,
but predastore only reclaims that disk space later via background
compaction. The CLI printed "freed X GiB" immediately on delete, which
made a compaction outage that never reclaimed the space invisible.
Rename to BytesDeleted and reword the CLI output to say the bytes are
marked for deletion and reclaimed by background compaction, not freed
immediately.
* fix(ec2): rename ambiguous 'code' slog key to firmwarePath
The EFI variable store prep log used 'code' for the OVMF firmware
path, which reads as an error/status code in log search. Rename to
firmwarePath to avoid confusion.
* fix(spinifexui): write pid file to a service-owned BaseDir
spinifexui.Start() called the legacy utils.WritePidFile, which has no
MkdirAll and falls back to $HOME/spinifex. Under the spinifex-ui unit's
ProtectHome=yes this directory is neither guaranteed to exist nor
writable. Add Config.BaseDir, switch to utils.WritePidFileTo (which
does MkdirAll) for Start/Stop/Status, wire it through the CLI as a
spinifex-ui-base-dir flag/SPINIFEX_UI_BASE_DIR env var, provision
/var/lib/spinifex/spinifex-ui in setup.sh, and add it to the unit's
ReadWritePaths.
* test(viperblockd): replace fixed sleep with a readiness poll
TestEBSConfigQueueGroup_DetachedOpenFails slept 500ms before assuming
launchService's ebs.config subscriber was bound, then raced a hardcoded
5s request deadline. Poll with a cheap malformed request (rejected at
json.Unmarshal, before the slow S3 path) until the subscriber answers,
then issue the real request with room for openVolumeVB's S3 backoff.
* fix(e2e): capture GPU worker console log on test failure
Wire harness.OnFailure into TestEKSGPUPodExposure to dump the GPU
worker's serial console via the established EC2 GetConsoleOutput path,
matching lb_test.go's captureLBConsoleOnFailure. A driver or
device-plugin hang after boot only ever surfaces on the guest console.
* fix(ec2): render empty Describe result containers and add requestId
xmlutil.BuildXML omits a nil slice's XML container entirely but renders
an empty one for a non-nil empty slice, so every EC2 Describe with zero
results (DescribeKeyPairs, DescribeVolumes, DescribeInstances, etc.)
came back as a bare "<FooResponse></FooResponse>" instead of AWS's
"<FooResponse><requestId/><fooSet/></FooResponse>". botocore parses that
body as having no result key at all, so the CLI silently prints nothing
and exits 0 — this broke the ansible key-import guard that gates
cluster bootstrap on describe-key-pairs' output.
Fix normalizes nil result slices to empty ones and synthesizes a
requestId in the single marshal chokepoint shared by every EC2 action
(marshalEC2Response, used by both ec2Handler and ec2HandlerWithReq),
rather than patching each handler individually.
* fix(ec2): return InvalidKeyPair.NotFound / InvalidSnapshot.NotFound for missing explicit IDs
DescribeKeyPairs and DescribeSnapshots silently returned an empty list
when the caller named a specific, nonexistent KeyName/KeyPairId or
SnapshotId, instead of erroring like real AWS (and like the existing
DescribeSecurityGroups/DescribeImages/DescribeVolumes/DescribeSubnets
handlers already do). botocore treats the empty-list response as
success, so ansible's cluster-post key-import gate — which checks
`describe-key-pairs --key-names spinifex-key` for a nonzero exit code —
never fires, and spinifex-key is never imported during bootstrap.
Mirrors the two-pass found-set pattern already used by
DescribeSecurityGroups: build the set of names/IDs actually present in
the filtered results, then error if any explicitly requested name/ID
is absent. An unfiltered list or a --filters query that matches
nothing is unaffected and still returns an empty list with no error.
DescribeInstances has the same gap but needs a design decision on how
a NotFound check interacts with the fan-out's existing partial-result
leniency, so it is left for separate follow-up.
TanStack router-plugin <1.168.21 sorted routes by coercing route nodes
to strings, so output order followed filesystem-walk order and flipped
depending on which generator version a contributor's node_modules
resolved. Pin the version so it cannot drift, commit the sorted output
from the fixed generator, and fail CI if a build produces a different
route tree than the one committed.
On a zero-worker or GPU-only cluster no node tolerates the DaemonSet,
so it has zero desired pods. The apiserver then has no tunnel to
kube-system pods (including metrics-server), v1beta1.metrics.k8s.io
never goes Available, and namespace GC hangs on the kubernetes
finalizer for every namespace, not just ones using metrics.
Add the same CriticalAddonsOnly + nvidia.com/gpu tolerations already
used by every other control-plane-critical addon in this repo
(aws-load-balancer-controller, ebs-csi-driver, argocd) so the agent
also schedules onto the control plane.
A tool whose child dies by signal can still exit 0 through go tool, so
govulncheck printed 'ok' after being OOM-killed and preflight passed
without a vulnerability scan having run. Route lint, govulncheck and
nilaway through scripts/run-gate.sh, which fails on a non-zero exit or
on signal-death output.
RegisterTaskDefinition let any caller with ecs:RegisterTaskDefinition name
an arbitrary taskRoleArn/executionRoleArn, and RunTask/StartTask/CreateService/
UpdateService could launch tasks under a previously registered definition's
roles, all without holding iam:PassRole on those roles. Mirror the EC2
gateway's PassRoleChecker closure pattern to close both paths.
ClaimLBName's crash-orphan reclaim treated any owner with no persisted
LB record as abandoned, but a legitimate in-flight create has no record
yet either during the gap between claiming the name and PutLoadBalancer.
Two concurrent creates for the same name could both win.
Track a claim timestamp and only allow reclaim once it exceeds a TTL,
so a fresh in-flight claim blocks a second claimer like a live LB would,
while a claim abandoned by a genuine crash is still reclaimed.
gateway.ErrorHandler unconditionally overwrote every error with
ErrorLookup's static per-code text, discarding both a caller's own
formatted message and any per-service distinction for wire codes
shared across services (e.g. ACM and EKS both use
ResourceInUseException). ResolveErrorDetail now also returns a
message when the producing call site used awserrors.Errorf, and
LookupErrorMessage resolves service-scoped wording before falling
back to the global default. DeleteVpc's DependencyViolation call
sites now name the blocking subnet or security group.
* fix: scope golangci-lint cache per worktree
golangci-lint's default cache under ~/.cache/golangci-lint is keyed by
module path rather than worktree path, so parallel worktrees of this
module collide and surface each other's lint issues, including from
worktrees that have since been deleted.
* fix: correct eks-node-gpu catalog description to match agent-only image
The GPU node image never installs eks-token-webhook or a server/agent
role selector, per scripts/images/eks-node-gpu/manifest.conf; it ships
ecr-credential-provider only and is agent-only because GPU nodes never
run the EKS control plane.
instanceCleanerAdapter.DeleteVolumes only called DetachVolumeOnTerminate for
a still-attached Boot volume, assuming shutdownAndUnmount's Unmount already
cleared a non-Boot volume's attachment. That assumption breaks whenever the
unmount seal fails (Unmount swallows the error) or never runs at all, as on
the stuck-terminate force-complete path. A leaked volume then stays in-use
with no recovery path once the instance record is deleted: detach-volume
--force fails InvalidInstanceID.NotFound and delete-volume fails
VolumeInUse. Call the same explicit, idempotent detach for every
non-DeleteOnTermination volume regardless of Boot.
DescribeImages hardcoded every AMI as "available" regardless of import
progress, so a failed backend import that never reached the flush was
indistinguishable from a complete one. Project the AMI's own State,
treating empty as available so pre-existing AMIs registered before the
field existed stay visible. CreateImageFromInstance, RegisterImage, and
CopyImage now set State explicitly since each completes synchronously
before the config write.
* fix(vm): report ENOSPC to the guest on data volumes instead of pausing the VM
QEMU's default werror=enospc pauses the whole VM when a data volume's
backend runs out of space. The boot drive already avoids this via
werror=report,rerror=report; data volumes had no on-error policy at
all in either the cold-boot -device or the hotplug device_add.
Set werror=report,rerror=report on the virtio-blk-pci device itself,
since a data volume's blockdev-add NBD node has no BlockBackend to
carry it the way -drive does for the boot drive.
* fix(vm): stamp ShuttingDownAt even when the state transition write fails
transitionWithPrecheck returned before stampShuttingDownAt whenever
TransitionState failed, even though the in-memory status had already
moved to the target. On a full disk this leaves ShuttingDownAt zero,
and StuckTerminateReaper.Sweep skips any VM whose ShuttingDownAt is
zero — so the backstop meant to bound a wedged terminate is disabled
by exactly the condition it exists to catch.
Stamp on the persistence-failure branch too, using the same helper
and locking as the success path, while still returning the original
error to the caller. The raced branch (TransitionState failed and the
in-memory status never reached target) is left unstamped, since the
target state was never actually entered there.
Start registers startCluster as a member of shutdownWg and then waits on
that group. startCluster's tail waited on the same group, so it could not
return until the counter it was waiting on reached zero, and the counter
could not reach zero until it returned.
The SIGTERM handler completed and logged "Shutdown complete" in ~20ms, but
the process never exited: it sat until systemd's TimeoutStopSec=15 and was
SIGKILLed, on every stop. Because the member stop chain is serialized,
that 15s also gated viperblock, predastore and nats, which each stop in
milliseconds once reached.
The duplicate setupShutdown in the same tail registered a second
signal.Notify and a second handler goroutine, which is where the
duplicated shutdown log lines and NATS unsubscribe pass came from.
Drop both. Start already installs the handler and owns the single wait.
The invariant test pins each helper to one call site, since a second call
site is the defect rather than a symptom of it.
Five manager tests waited on a fake DHCP counter and then read the KV
store. Fake.Acquire increments that counter before Manager.doAcquire
persists the entry, so the assertion ran in the gap between the two and
read the pre-acquire address whenever the write lost the race.
Idle machines always won the race, which is why this only ever failed
under load: the last three Build and Test failures on dev were this test
twice, on a diff that never touched the package.
Make the persisted KV state the barrier, matching the shape already used
by TestManagerStartKeepsNewLeaseWhenStaleReleaseFails, and keep the
counter as a plain assertion once the write is observed.
* fix(ec2): release post-launch ENIs on instance terminate
Terminate released only the ENI recorded at RunInstances time. Attachments
made after boot — ECS awsvpc task ENIs, or any direct AttachNetworkInterface
— are written to the spinifex-vpc-enis KV and never reach instance.ENIId, so
both terminate paths walked straight past them. The survivors held their
security group, blocking the SG, subnet and VPC deletes behind
DependencyViolation.
Both paths now sweep the KV for records naming the instance, independent of
the scalar. The sweep is additive, so an instance with no secondary ENIs
behaves exactly as before, and best-effort per ENI so one failure neither
aborts the rest nor blocks convergence on re-run.
DeleteOnTermination was advertised by DescribeNetworkInterfaces but backed by
nothing; ENIRecord now carries it, defaulting to true, and false detaches
without deleting.
* test(ec2): cover ENI release branches
Add table-driven and NATS-backed tests for the terminate-time ENI
sweep: DetachENI/ForceDeleteInstanceENI failure tolerance,
DeleteOnTermination skip, multi-ENI happy path, and
ListInstanceENIs error mapping on both the daemon and instance
service adapters.
viperblock#64 stops WriteAtCtx returning ErrNoSpace on predastore's nearfull
pressure signal, which rides on a successful write and means slow down rather
than stop. Without this, spx refused every write once the store fell below 15%
free, 10 points above the watermark at which predastore actually rejects.
Needed by spx itself: image import runs in-process against the module-cache
build, so the plugin-side fix alone left the control plane on the old
behaviour.
Six merged viperblock changes behind one repin: local files kept when the
WAL cannot be uploaded, stranded WAL generations retried after a failed
chunk upload, the flush-covers-acked-writes invariant pinned, the nbdkit
plugin's shutdown drain bounded at 20s, seal receipts written after a clean
seal, and FLUSH fsyncing the WAL before it reports success.
The seal receipts pair with the consumer already on dev, and the 20s drain
bound pairs with TimeoutStopSec=30 on spinifex-viperblock.service.
The unmount path decided whether to seal by stat-ing BaseDir/<volume>, but
the nbdkit plugin deletes exactly that directory as the last step of a
successful seal. So the fallback seal was unreachable whenever the plugin
exited cleanly, and a WARN claiming there was no local state fired on every
healthy unmount.
Branch on the receipt the plugin now leaves instead. Local state present
still means the plugin's seal failed or was cut short, so the fallback runs
and is finally reachable; a receipt means the plugin sealed and cleaned up;
neither keeps the original WARN, now meaning what it says.
The aux-volume check moves out of volumeNeedsSeal and to the front of the
chain: split out but left second, an -efi volume with local state would
newly reach a seal it has always been excluded from.
Stale receipts are cleared at mount rather than matched by PID at unmount,
so nothing depends on nbdkit's pid surviving to plugin load.
In-use nbdkit children are SIGTERMed with this cgroup and run the plugin's
shutdown drain. That drain is now bounded at 20s in the plugin itself, so
30s here lets the plugin's own bound fire first and keeps SIGKILL
exceptional. At the 90s default a slow or unreachable backend stalled the
whole target-stop transaction.
Picks up viperblock #57, which routes the JSON slog handler to stderr so
the nbdkit plugin's logs reach journald — nbdkit repoints a plugin's fd 1
at /dev/null, so the whole volume seal path was discarded. Also picks up
#56, removing a per-call allocation from the per-block cache instrument.
No spinifex code depends on either change: spx does not import
viperblock/telemetry. The bump is needed because the build role compares
the viperblock source behind spx (module cache, GOWORK=off) against the
one behind the nbdkit plugin (the submodule tree) and fails on a mismatch.
lb-agent configured no logger at all, so it fell back to slog's default
text handler. That folds attributes into the message, leaving the cause of
a failure trapped in a string: 'Heartbeat failed err="gateway returned
400"' arrived greppable but never queryable.
No OTLP export: the agent's only egress is the gateway it is watching, so
the host-side serial console remains the path that survives a blackout.
Init bolts the OTLP bridge onto whatever slog default is installed at the
time. SetupRoutes then called SetDefaultJSONLogger purely to set a level,
and that bare slog.SetDefault replaced the fanout with a stdout-only
handler, so the gateway went dark to the sink for the rest of the process.
One nightly cell shipped 9 app-log docs, all within 30ms of startup, while
its journal held 574 request lines.
Backs the stdout handler with a LevelVar and adds SetLevel, so verbosity
changes no longer touch the handler chain.
* feat(gateway): log the canonical request and action on signature mismatch
A SignatureDoesNotMatch told us nothing beyond err="signature mismatch":
the canonical request the server reconstructed was never logged, and the
APM trace is a bare "POST /" because auth fails before the dispatcher
parses the action. An intermittent mismatch was therefore undiagnosable.
Logs the server-side canonical request, with the session token masked, plus
the recovered action. Warn rather than Debug because the failures worth
catching never reproduce on demand; the path is already rate-limited.
* chore(deps): bump predastore for SignedRequest.CanonicalRequest
Picks up the exported canonical request the mismatch logging needs.
Render summed FailCount across suites and printed "No failures across any
suite" whenever the total was zero, without asking whether any test had run.
A cell that dies during provisioning emits junit with no testcases at all, so
it took the clean path. Nightly run 30467638631 cell 20 failed with "install
callback never fired after 3 attempts; PXE never triggered install" and its
analysis.md read "No failures across any suite" — absence of evidence rendered
as positive evidence.
Zero testcases now renders a warning naming the layer above the suites. A run
with tests and no failures is unchanged.
Cells 2, 3, 9 and 10 ran on spinifex-image-builder-ubuntu-24.04.qcow2, built
2026-04-20 — five weeks before openvswitch-ipsec and strongswan-charon were
added to the image builder. Those cells therefore had no OVN IPsec support and
ran intra-AZ Geneve in plaintext. Every other workflow is already on 26.04.
The host_image_override guards now match ubuntu-* rather than a pinned version.
An exact match is what let this drift go unnoticed: bumping the matrix release
silently stops overriding the image and drops the cell onto the debian gold.
* fix(e2e): fail TestMultinodeIPSec when a cluster that requested IPsec has none
An empty ipsec_encapsulation was treated as "bootstrapped without IPsec" and
skipped. On ubuntu-24.04 openvswitch-ipsec.service is inactive at daemon start,
so the enable guard correctly refuses to flip the flag and the cluster runs
plaintext intra-AZ Geneve — but the test then skipped and the cell reported
green, masking the gap on 2 of 3 multinode cells.
Cross-check network.ipsec_enabled from the node config: an explicit false still
skips, anything else fails with the plaintext condition named.
* fix(e2e): read only the ipsec_enabled value, not the whole config line
Detail() rendered "ipsec_enabled=ipsec_enabled = true" because the raw grep
line was logged as the value. Take the first token after "=" so a trailing
comment cannot be read as the value either, and compare exactly rather than
by substring.
Josh Siviour