Files
spinifex/go.mod
T
Josh Siviour 22171c44ab fix: close the storage-exhaustion recovery gaps (durable terminate, per-service memory caps, local volume leak) (#653)
* fix(vm): durably record terminate even when local state persistence fails

finalizeTerminated bailed out of WriteTerminatedInstance whenever the
transition to terminated hit a persistence-only failure (e.g. ENOSPC writing
local state), even though the in-memory status had already reached
terminated. That left the instance owned by nothing: StuckTerminateReaper
skips it (no longer shutting-down) and TerminatedTeardownReaper never sees
it (never reached the terminated KV bucket). Only an operator daemon restart
recovered it.

WriteTerminatedInstance is a JetStream KV write, independent of local disk
space, so keep going into it whenever the in-memory status did reach
terminated, distinguishing that case from a genuine invalid/raced
transition. Once the disk has space again, teardown now completes on its
own via the existing idempotent TerminatedTeardownReaper.

* fix(systemd): cap per-service memory under spinifex-system.slice

Every service under spinifex.slice ran with MemoryMax=infinity /
MemoryHigh=infinity, so one runaway service could exhaust the whole slice
budget and the slice-level OOM would then reap an arbitrary victim instead
of the offender.

Add per-unit MemoryHigh/MemoryMax as percentages of physical RAM (so the
same units scale from a dev box to bare metal). spinifex-daemon.service
still hosts every guest QEMU process in its own cgroup (no per-instance
slice yet), so it gets by far the largest allowance, just under the slice's
own 80%/90%, to avoid starving VM launches. predastore is the legitimately
memory-hungry service and gets generous headroom above its idle footprint.
The remaining infra services are small, single-purpose Go processes and get
a shared conservative bound.

* fix(viperblockd): remove local volume directory on ebs.delete

DeleteVolume deleted the S3 data and notified viperblockd, but the ebs.delete
handler only stopped the volume's goroutines, killed nbdkit, and removed the
NBD socket — it never removed the local WAL/checkpoint cache directory under
BaseDir. The only path that removed it (VB.Close's RemoveLocalFiles) only
ran from the unmount handler's seal step, which -efi volumes never reach
(they carry no durable data to seal) and which a main volume also skips
whenever a prior seal failed (e.g. disk full, deliberately keeping local
files for recovery). Since terminate always unmounts before deleting, and
delete is permanent, retention had no remaining purpose by the time
ebs.delete ran.

Unconditionally remove BaseDir/<volume> in the ebs.delete handler regardless
of mount-tracking state, closing the leak for both the root volume and its
-efi companion.

* fix(viperblockd): validate volume names before removing local state

ebs.delete unmarshalled the volume name straight off the wire and
handed it to os.RemoveAll: an empty name collapsed to BaseDir itself,
and ".." components could escape it entirely. localVolumeDir now
rejects an empty/invalid name or one that resolves outside BaseDir
before any removal, and ebs.delete returns Success: false with an
error instead of silently skipping. Applied the same gate to
ebs.mount and ebs.config, the other two handlers that turn a
wire-supplied volume name into a BaseDir-relative path.

* chore: pin viperblock to the merged concurrent-open corruption fix

The ansible build role builds with GOWORK=off, so the deployed binary links
whatever go.mod pins rather than the go.work overlay. Move the pin to the
revision carrying the per-volume flock and the WAL/state write fixes.
2026-08-02 18:58:34 +10:00

224 lines
11 KiB
Modula-2

module github.com/mulgadc/spinifex
go 1.26.5
require (
github.com/aws/aws-sdk-go v1.55.8
github.com/aws/aws-sdk-go-v2 v1.43.2
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14
github.com/aws/aws-sdk-go-v2/config v1.32.31
github.com/aws/aws-sdk-go-v2/credentials v1.19.30
github.com/cenkalti/backoff/v4 v4.3.0
github.com/charmbracelet/bubbles v1.0.0
github.com/charmbracelet/bubbletea v1.3.10
github.com/charmbracelet/lipgloss v1.1.0
github.com/containerd/containerd/v2 v2.3.3
github.com/distribution/reference v0.6.0
github.com/go-chi/chi/v5 v5.3.1
github.com/golang-jwt/jwt/v5 v5.3.1
github.com/google/uuid v1.6.0
github.com/insomniacslk/dhcp v0.0.0-20260407060928-11b94ed970f2
github.com/klauspost/cpuid/v2 v2.4.0
github.com/miekg/dns v1.1.72
github.com/mulgadc/northstar v1.14.0
github.com/mulgadc/predastore v1.14.1-0.20260730221229-608939ef2bb4
github.com/mulgadc/viperblock v1.14.1-0.20260802084514-ed52f6a0e4c0
github.com/nats-io/nats-server/v2 v2.14.3
github.com/nats-io/nats.go v1.52.0
github.com/opencontainers/runtime-spec v1.3.0
github.com/ovn-kubernetes/libovsdb v0.8.1
github.com/pelletier/go-toml/v2 v2.4.3
github.com/pterm/pterm v0.12.83
github.com/spf13/cobra v1.10.2
github.com/spf13/viper v1.21.0
github.com/stretchr/testify v1.11.1
go.opentelemetry.io/contrib/bridges/otelslog v0.19.0
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.69.0
go.opentelemetry.io/contrib/instrumentation/runtime v0.69.0
go.opentelemetry.io/otel v1.44.0
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.20.0
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.44.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.44.0
go.opentelemetry.io/otel/log v0.20.0
go.opentelemetry.io/otel/metric v1.44.0
go.opentelemetry.io/otel/sdk v1.44.0
go.opentelemetry.io/otel/sdk/log v0.20.0
go.opentelemetry.io/otel/sdk/metric v1.44.0
go.opentelemetry.io/otel/trace v1.44.0
go.uber.org/goleak v1.3.0
go.yaml.in/yaml/v3 v3.0.4
golang.org/x/crypto v0.54.0
golang.org/x/sync v0.22.0
golang.org/x/sys v0.47.0
gopkg.in/ini.v1 v1.67.3
)
require (
atomicgo.dev/cursor v0.2.0 // indirect
atomicgo.dev/keyboard v0.2.9 // indirect
atomicgo.dev/schedule v0.1.0 // indirect
github.com/Microsoft/go-winio v0.6.3-0.20251027160822-ad3df93bed29 // indirect
github.com/Microsoft/hcsshim v0.15.0-rc.1 // indirect
github.com/antithesishq/antithesis-sdk-go v0.7.0 // indirect
github.com/armon/go-metrics v0.4.1 // indirect
github.com/atotto/clipboard v0.1.4 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.31 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.33 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.33 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.32 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.14 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.24 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.33 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.32 // indirect
github.com/aws/aws-sdk-go-v2/service/s3 v1.106.0 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.5.0 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.33.0 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.0 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.45.0 // indirect
github.com/aws/smithy-go v1.27.5 // indirect
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/boltdb/bolt v1.3.1 // indirect
github.com/buraksezer/consistent v0.10.0 // indirect
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/cenkalti/hub v1.0.2 // indirect
github.com/cenkalti/rpc2 v1.0.5 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/charmbracelet/colorprofile v0.4.1 // indirect
github.com/charmbracelet/x/ansi v0.11.6 // indirect
github.com/charmbracelet/x/cellbuf v0.0.15 // indirect
github.com/charmbracelet/x/term v0.2.2 // indirect
github.com/clipperhouse/displaywidth v0.9.0 // indirect
github.com/clipperhouse/stringish v0.1.1 // indirect
github.com/clipperhouse/uax29/v2 v2.7.0 // indirect
github.com/containerd/cgroups/v3 v3.1.3 // indirect
github.com/containerd/console v1.0.5 // indirect
github.com/containerd/containerd/api v1.11.1 // indirect
github.com/containerd/continuity v0.5.0 // indirect
github.com/containerd/errdefs v1.0.0 // indirect
github.com/containerd/errdefs/pkg v0.3.0 // indirect
github.com/containerd/fifo v1.1.0 // indirect
github.com/containerd/log v0.1.0 // indirect
github.com/containerd/platforms v1.0.0-rc.4 // indirect
github.com/containerd/plugin v1.1.0 // indirect
github.com/containerd/ttrpc v1.2.8 // indirect
github.com/containerd/typeurl/v2 v2.2.3 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/dgraph-io/badger/v4 v4.9.5 // indirect
github.com/dgraph-io/ristretto/v2 v2.4.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
github.com/fatih/color v1.19.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/fsnotify/fsnotify v1.10.1 // indirect
github.com/gabriel-vasile/mimetype v1.4.13 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-playground/validator/v10 v10.30.1 // indirect
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
github.com/google/flatbuffers v25.12.19+incompatible // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/gookit/color v1.6.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.29.0 // indirect
github.com/hashicorp/go-hclog v1.6.3 // indirect
github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
github.com/hashicorp/go-metrics v0.5.4 // indirect
github.com/hashicorp/go-msgpack/v2 v2.1.5 // indirect
github.com/hashicorp/golang-lru v1.0.2 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/hashicorp/raft v1.7.3 // indirect
github.com/hashicorp/raft-boltdb/v2 v2.3.1 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/josharian/native v1.1.0 // indirect
github.com/klauspost/compress v1.18.6 // indirect
github.com/klauspost/reedsolomon v1.14.1 // indirect
github.com/leodido/go-urn v1.4.0 // indirect
github.com/lithammer/fuzzysearch v1.1.8 // indirect
github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-localereader v0.0.1 // indirect
github.com/mattn/go-runewidth v0.0.21 // indirect
github.com/mdlayher/packet v1.1.2 // indirect
github.com/mdlayher/socket v0.5.1 // indirect
github.com/minio/crc64nvme v1.1.1 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/moby/locker v1.0.1 // indirect
github.com/moby/sys/capability v0.4.0 // indirect
github.com/moby/sys/mountinfo v0.7.2 // indirect
github.com/moby/sys/sequential v0.6.0 // indirect
github.com/moby/sys/signal v0.7.1 // indirect
github.com/moby/sys/user v0.4.0 // indirect
github.com/moby/sys/userns v0.1.0 // indirect
github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 // indirect
github.com/muesli/cancelreader v0.2.2 // indirect
github.com/muesli/termenv v0.16.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/nats-io/jwt/v2 v2.8.2 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/opencontainers/runtime-tools v0.9.1-0.20251114084447-edf4cb3d2116 // indirect
github.com/pierrec/lz4/v4 v4.1.18 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/client_golang v1.23.2 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.67.5 // indirect
github.com/prometheus/procfs v0.20.1 // indirect
github.com/quic-go/quic-go v0.61.0 // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/sagikazarmark/locafero v0.12.0 // indirect
github.com/sirupsen/logrus v1.9.4 // indirect
github.com/spf13/afero v1.15.0 // indirect
github.com/spf13/cast v1.10.0 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
github.com/u-root/uio v0.0.0-20230220225925-ffce2a382923 // indirect
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
go.etcd.io/bbolt v1.4.3 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.44.0 // indirect
go.opentelemetry.io/proto/otlp v1.10.0 // indirect
go.uber.org/nilaway v0.0.0-20260721205819-4a0f7653488a // indirect
go.yaml.in/yaml/v2 v2.4.4 // indirect
golang.org/x/exp/typeparams v0.0.0-20260611194520-c48552f49976 // indirect
golang.org/x/mod v0.37.0 // indirect
golang.org/x/net v0.56.0 // indirect
golang.org/x/telemetry v0.0.0-20260625142307-59b4966ccb57 // indirect
golang.org/x/term v0.45.0 // indirect
golang.org/x/text v0.40.0 // indirect
golang.org/x/time v0.15.0 // indirect
golang.org/x/tools v0.47.0 // indirect
golang.org/x/vuln v1.4.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260618152121-87f3d3e198d3 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260618152121-87f3d3e198d3 // indirect
google.golang.org/grpc v1.82.1 // indirect
google.golang.org/protobuf v1.36.12-0.20260120151049-f2248ac996af // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
sigs.k8s.io/yaml v1.6.0 // indirect
tags.cncf.io/container-device-interface v1.1.0 // indirect
tags.cncf.io/container-device-interface/specs-go v1.1.0 // indirect
)
// go.opencensus.io -> grpc@v1.33.2 drags in the pre-split monolithic genproto,
// which still ships google.golang.org/genproto/googleapis/rpc/status and so
// collides with the split genproto/googleapis/rpc module that containerd/ttrpc
// needs ("ambiguous import"). The main module never imports the monolith, so a
// plain require is dropped by `go mod tidy`; pin it via replace to a post-split
// version (no rpc/status) so the fix survives tidy. Remove if grpc/opencensus
// stop requiring the monolith.
replace google.golang.org/genproto => google.golang.org/genproto v0.0.0-20260622175928-b703f567277d
tool (
go.uber.org/nilaway/cmd/nilaway
golang.org/x/vuln/cmd/govulncheck
)