gsadmin/spinifex
1
0
Fork 0
mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-15 09:01:31 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
d52245001e33556bb7ecfca754b1a9112928e17c
spinifex/go.mod
T
Josh Siviour d52245001e feat: Ochre vector store (tenant pgvector, embeddings, ingestion, operator teardown) (#788) 
* feat: add /v1/embeddings gateway adapter

* feat: Ochre vector store pgvector backend, KV index registry, orchestration

Adds handlers_ochrevector: the VectorBackend interface + pgx/v5 impl
with per-account schema/role isolation enforced via SET LOCAL ROLE +
search_path inside every transaction, the KV index registry
(CREATING/READY/DELETING/STALE), and the create/delete/list/reconcile
orchestration tying them together with crash-safe rollback and
re-entrant recovery of stuck records.

* feat: singleton appliance-ensure state machine for ochre vector store

Adds the encrypted control-plane credential store and re-entrant
Ensure state machine for the platform pgvector appliance (D2): a
fixed-key KV claim mirroring Registry.Reserve elects exactly one
winner to launch the appliance via an injected ApplianceLauncher,
losers wait or resume a stale crashed provision, and the master
password is generated once, stored only AES-256-GCM encrypted, and
decrypted solely inside Connect to build a pgxBackend.

* feat: Ochre vector store async ingestion (chunker, job store, worker)

Adds Stage 3: a deterministic recursive chunk splitter, a per-account
ingestion job-record store mirroring the index registry, the
transactional per-document replace on the backend, and the
IngestService worker that lists a predastore source, chunks and
embeds each object, and replaces its rows idempotently.

* feat: Ochre vector store similarity query and metadata filter grammar

Adds VectorBackend.Query (cosine kNN, k default/cap, iterative HNSW
scan under a filter) and a Bedrock-KB-shaped metadata Filter AST that
compiles to parameterized SQL over jsonb, closing off arbitrary SQL by
allowlisting jsonb keys and always binding values as params. Also
fixes RunJob reporting READY on a job that ingested nothing regardless
of failure cause, and stamps SourceSpec.Metadata onto ingested rows so
the new filters have data to match against.

* feat: Ochre vector store request/response API, VectorService, and CLI

Adds the daemon-facing VectorService interface (index CRUD, ingest,
describe job, query), its server implementation over the existing
registry/backend/embedder, a NATS-forwarding client, and the
`spx admin ochre vector` operator CLI. Query resolves and embeds
against each index's own pinned model; ingest stamps the same
model/dimension onto every source spec server-side. accountID is
always taken from the caller context, never the request payload.

Daemon subscription wiring and manifest registration are a later
change; this is pure request/response plumbing with no construction
into daemon.go or awsgw.go.

* feat: wire Ochre vector store VectorService into the daemon

Constructs the platform Postgres appliance and VectorService behind a new
OchreVector config section that defaults off, so existing daemons are
unaffected. Adds a real RDS-backed ApplianceLauncher that provisions the
appliance over rds.CreateDBInstance/DescribeDBInstances, registers the six
ochre.vector.* NATS subjects mirroring the bedrock endpoint service pattern,
and degrades to a disabled (nil) service on any boot-time appliance failure
rather than blocking daemon startup.

* fix: fix Ochre vector store boot ordering and self-register subjects

startOchreVector ran before subscribeAll, so its appliance launch fired
rds.CreateDBInstance/DescribeDBInstances requests with no responder yet
registered, timing out the launch on a first-enable boot. It also ran
synchronously, blocking daemon startup up to the appliance launch timeout.

Move it to a background goroutine started after subscribeAll returns, and
have it register its own ochre.vector.* subjects once the VectorService is
actually constructed, via a new registerNatsSubs helper shared with
subscribeAll's boot-time table. Guard d.natsSubscriptions with d.mu, since
this background registration can now race the shutdown path's
unsubscribe-all sweep.

* feat: bake pgvector into the shared rds-postgres image

* fix: reach the ochre appliance over a routed daemon port on its ENI IP

The daemon dialled the RDS appliance over rec.Endpoint, a vanity spx3.net
hostname the control-plane host can neither resolve nor route to, so a healthy
appliance was never reachable and the vector store disabled itself.

Resolve the appliance's customer-ENI private IP read-only via
ec2.DescribeNetworkInterfaces filtered on the spinifex:rds-db-instance tag, dial
that IP for the DSN instead of the hostname, and give the daemon its own routed
ENI in the appliance's subnet bound to an OVS host port so the IP is reachable.
Retry appliance startup with bounded backoff instead of hard-disabling on the
first transient.

* fix: install pgvector at bootstrap for the Ochre vector appliance

The daemon connects to the appliance as its NOSUPERUSER master, and the
image's pgvector is not a trusted extension, so CREATE EXTENSION vector
failed with permission denied. rds-init now installs it via the bootstrap
superuser, gated on the appliance's fixed master name so tenant instances
are unaffected.

* fix: grant the Ochre appliance master CREATE on its database

The daemon's EnsureAccount runs CREATE SCHEMA per account as the NOSUPERUSER
master, which does not own the postgres database, so bootstrap now grants it
CREATE on the database alongside the pgvector install. Both are gated to the
appliance master, so tenant instances are unaffected.

* fix: make the Ochre appliance's pgvector reachable under per-account isolation

The daemon connects as a NOSUPERUSER master and runs every account operation
under SET ROLE with a search_path pinned to the account schema. Two gaps blocked
that against a real non-superuser Postgres: the master could not SET ROLE into
the roles it creates, and the account role could not see the vector type.

Bootstrap now grants the appliance master createrole_self_grant and installs
pgvector into a read-only extensions schema granted to PUBLIC; the backend
carries that schema on each account search_path so the vector type resolves
without weakening per-account write isolation. All appliance-scoped.

* fix: prune stale VPC host port before installing the daemon appliance port

A replaced Ochre appliance is handed the same subnet IP, so a new daemon ENI
mints a fresh vhp- port at an address a superseded port still holds. The old
port survives at the OS/OVS layer even after its ENI record is wiped, so two
interfaces claim one address and the kernel can route to the dead one
(EHOSTUNREACH). installVPCHostPort now prunes any other vhp- port holding the
target address before installing.

* fix: gate VM restore's stale-socket kill on backing-store readiness

On daemon start, a live VM whose NBD sockets fail to dial was force-killed as an
orphan. While predastore/viperblock is still coming up after a restart those
sockets are transiently unreachable, so a healthy VM (the Ochre appliance was
the observed victim) was destroyed and its RDS record flipped to failed. The
kill now only fires when the backing store is confirmed up; otherwise the live
QEMU is kept and QMP reconnected. Genuine orphan reaping under a healthy store
is unchanged.

* fix: drive ochre ingest jobs from a daemon-side sweep scheduler

* feat: add operator teardown verb for the ochre platform appliance

Adds Appliance.Teardown to remove the singleton platform Postgres
appliance (RDS instance, host port, KV record), an admin-only NATS
subject and daemon handler to drive it, and `spx admin ochre appliance
teardown --confirm` to invoke it without exposing the action through
the tenant awsgw surface.

* fix: register ochre appliance teardown before Connect so it survives a broken appliance

* fix: purge ochre index and job metadata on appliance teardown

* test: mark ochre in-package test files for the external-test gate
2026-08-13 15:54:48 +10:00

229 lines
11 KiB
Modula-2
Raw Blame History

module github.com/mulgadc/spinifex
go 1.26.5
require (
github.com/aws/aws-sdk-go v1.55.8
github.com/aws/aws-sdk-go-v2 v1.43.5
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.16
github.com/aws/aws-sdk-go-v2/config v1.32.33
github.com/aws/aws-sdk-go-v2/credentials v1.19.34
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.35
github.com/cenkalti/backoff/v4 v4.3.0
github.com/charmbracelet/bubbles v1.0.0
github.com/charmbracelet/bubbletea v1.3.10
github.com/charmbracelet/lipgloss v1.1.0
github.com/containerd/containerd/v2 v2.3.3
github.com/distribution/reference v0.6.0
github.com/go-chi/chi/v5 v5.3.1
github.com/golang-jwt/jwt/v5 v5.3.1
github.com/google/uuid v1.6.0
github.com/insomniacslk/dhcp v0.0.0-20260407060928-11b94ed970f2
github.com/klauspost/cpuid/v2 v2.4.0
github.com/miekg/dns v1.1.72
github.com/mulgadc/bluebottle v1.16.1-0.20260812012137-683e88e465e3
github.com/mulgadc/northstar v1.16.1-0.20260812012604-5d7a622ad059
github.com/mulgadc/predastore v1.16.1-0.20260813022738-0b4392b43cae
github.com/mulgadc/viperblock v1.16.1-0.20260812124023-7e0df2323c75
github.com/nats-io/nats-server/v2 v2.14.4
github.com/nats-io/nats.go v1.52.0
github.com/opencontainers/runtime-spec v1.3.0
github.com/ovn-kubernetes/libovsdb v0.8.1
github.com/pelletier/go-toml/v2 v2.4.3
github.com/pterm/pterm v0.12.83
github.com/spf13/cobra v1.10.2
github.com/spf13/viper v1.21.0
github.com/stretchr/testify v1.11.1
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0
go.opentelemetry.io/otel v1.45.0
go.opentelemetry.io/otel/metric v1.45.0
go.opentelemetry.io/otel/sdk v1.45.0
go.opentelemetry.io/otel/trace v1.45.0
go.uber.org/goleak v1.3.1-0.20260602155401-5f791e3ef83b
go.yaml.in/yaml/v3 v3.0.5
golang.org/x/crypto v0.54.0
golang.org/x/sync v0.22.0
golang.org/x/sys v0.47.0
gopkg.in/ini.v1 v1.67.3
)
require (
atomicgo.dev/cursor v0.2.0 // indirect
atomicgo.dev/keyboard v0.2.9 // indirect
atomicgo.dev/schedule v0.1.0 // indirect
github.com/Microsoft/go-winio v0.6.3-0.20251027160822-ad3df93bed29 // indirect
github.com/Microsoft/hcsshim v0.15.0-rc.1 // indirect
github.com/antithesishq/antithesis-sdk-go v0.7.2-default-no-op // indirect
github.com/armon/go-metrics v0.4.1 // indirect
github.com/atotto/clipboard v0.1.4 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.35 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.35 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.36 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.15 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.28 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.35 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.36 // indirect
github.com/aws/aws-sdk-go-v2/service/s3 v1.107.0 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.5.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.33.4 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.45.4 // indirect
github.com/aws/smithy-go v1.27.7 // indirect
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/boltdb/bolt v1.3.1 // indirect
github.com/buraksezer/consistent v0.10.0 // indirect
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/cenkalti/hub v1.0.2 // indirect
github.com/cenkalti/rpc2 v1.0.5 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/charmbracelet/colorprofile v0.4.1 // indirect
github.com/charmbracelet/x/ansi v0.11.6 // indirect
github.com/charmbracelet/x/cellbuf v0.0.15 // indirect
github.com/charmbracelet/x/term v0.2.2 // indirect
github.com/clipperhouse/displaywidth v0.9.0 // indirect
github.com/clipperhouse/stringish v0.1.1 // indirect
github.com/clipperhouse/uax29/v2 v2.7.0 // indirect
github.com/containerd/cgroups/v3 v3.1.3 // indirect
github.com/containerd/console v1.0.5 // indirect
github.com/containerd/containerd/api v1.11.1 // indirect
github.com/containerd/continuity v0.5.0 // indirect
github.com/containerd/errdefs v1.0.0 // indirect
github.com/containerd/errdefs/pkg v0.3.0 // indirect
github.com/containerd/fifo v1.1.0 // indirect
github.com/containerd/log v0.1.0 // indirect
github.com/containerd/platforms v1.0.0-rc.4 // indirect
github.com/containerd/plugin v1.1.0 // indirect
github.com/containerd/ttrpc v1.2.8 // indirect
github.com/containerd/typeurl/v2 v2.2.3 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/dgraph-io/badger/v4 v4.9.6 // indirect
github.com/dgraph-io/ristretto/v2 v2.4.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
github.com/fatih/color v1.19.0 // indirect
github.com/felixge/httpsnoop v1.1.0 // indirect
github.com/fsnotify/fsnotify v1.10.1 // indirect
github.com/gabriel-vasile/mimetype v1.4.13 // indirect
github.com/go-logr/logr v1.4.4 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-playground/validator/v10 v10.30.1 // indirect
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
github.com/google/flatbuffers v25.12.19+incompatible // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/gookit/color v1.6.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.29.0 // indirect
github.com/hashicorp/go-hclog v1.6.3 // indirect
github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
github.com/hashicorp/go-metrics v0.5.4 // indirect
github.com/hashicorp/go-msgpack/v2 v2.1.5 // indirect
github.com/hashicorp/golang-lru v1.0.2 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/hashicorp/raft v1.7.3 // indirect
github.com/hashicorp/raft-boltdb/v2 v2.3.1 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/pgx/v5 v5.10.0 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/josharian/native v1.1.0 // indirect
github.com/klauspost/compress v1.19.0 // indirect
github.com/klauspost/reedsolomon v1.14.1 // indirect
github.com/leodido/go-urn v1.4.0 // indirect
github.com/lithammer/fuzzysearch v1.1.8 // indirect
github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-localereader v0.0.1 // indirect
github.com/mattn/go-runewidth v0.0.21 // indirect
github.com/mdlayher/packet v1.1.2 // indirect
github.com/mdlayher/socket v0.5.1 // indirect
github.com/minio/crc64nvme v1.1.1 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/moby/locker v1.0.1 // indirect
github.com/moby/sys/capability v0.4.0 // indirect
github.com/moby/sys/mountinfo v0.7.2 // indirect
github.com/moby/sys/sequential v0.6.0 // indirect
github.com/moby/sys/signal v0.7.1 // indirect
github.com/moby/sys/user v0.4.0 // indirect
github.com/moby/sys/userns v0.1.0 // indirect
github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 // indirect
github.com/muesli/cancelreader v0.2.2 // indirect
github.com/muesli/termenv v0.16.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/nats-io/jwt/v2 v2.8.2 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/opencontainers/runtime-tools v0.9.1-0.20251114084447-edf4cb3d2116 // indirect
github.com/pierrec/lz4/v4 v4.1.18 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/client_golang v1.23.2 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.67.5 // indirect
github.com/prometheus/procfs v0.20.1 // indirect
github.com/quic-go/quic-go v0.61.0 // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/sagikazarmark/locafero v0.12.0 // indirect
github.com/sirupsen/logrus v1.9.4 // indirect
github.com/spf13/afero v1.15.0 // indirect
github.com/spf13/cast v1.10.0 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
github.com/tidwall/btree v1.8.1 // indirect
github.com/u-root/uio v0.0.0-20230220225925-ffce2a382923 // indirect
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
go.etcd.io/bbolt v1.4.3 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/bridges/otelslog v0.20.0 // indirect
go.opentelemetry.io/contrib/instrumentation/runtime v0.70.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.21.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.45.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.45.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.45.0 // indirect
go.opentelemetry.io/otel/log v0.21.0 // indirect
go.opentelemetry.io/otel/sdk/log v0.21.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.45.0 // indirect
go.opentelemetry.io/proto/otlp v1.11.0 // indirect
go.uber.org/nilaway v0.0.0-20260721205819-4a0f7653488a // indirect
go.yaml.in/yaml/v2 v2.4.4 // indirect
golang.org/x/exp/typeparams v0.0.0-20260611194520-c48552f49976 // indirect
golang.org/x/mod v0.38.0 // indirect
golang.org/x/net v0.57.0 // indirect
golang.org/x/telemetry v0.0.0-20260708182218-49f421fb7959 // indirect
golang.org/x/term v0.45.0 // indirect
golang.org/x/text v0.40.0 // indirect
golang.org/x/time v0.15.0 // indirect
golang.org/x/tools v0.48.0 // indirect
golang.org/x/vuln v1.6.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260803160001-6ac0973c030d // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260803160001-6ac0973c030d // indirect
google.golang.org/grpc v1.83.0 // indirect
google.golang.org/protobuf v1.36.12-0.20260120151049-f2248ac996af // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
sigs.k8s.io/yaml v1.6.0 // indirect
tags.cncf.io/container-device-interface v1.1.0 // indirect
tags.cncf.io/container-device-interface/specs-go v1.1.0 // indirect
)
// go.opencensus.io -> grpc@v1.33.2 drags in the pre-split monolithic genproto,
// which still ships google.golang.org/genproto/googleapis/rpc/status and so
// collides with the split genproto/googleapis/rpc module that containerd/ttrpc
// needs ("ambiguous import"). The main module never imports the monolith, so a
// plain require is dropped by `go mod tidy`; pin it via replace to a post-split
// version (no rpc/status) so the fix survives tidy. Remove if grpc/opencensus
// stop requiring the monolith.
replace google.golang.org/genproto => google.golang.org/genproto v0.0.0-20260622175928-b703f567277d
tool (
go.uber.org/nilaway/cmd/nilaway
golang.org/x/vuln/cmd/govulncheck
)
Reference in New Issue View Git Blame Copy Permalink