mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 09:01:31 +00:00
feat: Ochre vector store (tenant pgvector, embeddings, ingestion, operator teardown) (#788)
* feat: add /v1/embeddings gateway adapter * feat: Ochre vector store pgvector backend, KV index registry, orchestration Adds handlers_ochrevector: the VectorBackend interface + pgx/v5 impl with per-account schema/role isolation enforced via SET LOCAL ROLE + search_path inside every transaction, the KV index registry (CREATING/READY/DELETING/STALE), and the create/delete/list/reconcile orchestration tying them together with crash-safe rollback and re-entrant recovery of stuck records. * feat: singleton appliance-ensure state machine for ochre vector store Adds the encrypted control-plane credential store and re-entrant Ensure state machine for the platform pgvector appliance (D2): a fixed-key KV claim mirroring Registry.Reserve elects exactly one winner to launch the appliance via an injected ApplianceLauncher, losers wait or resume a stale crashed provision, and the master password is generated once, stored only AES-256-GCM encrypted, and decrypted solely inside Connect to build a pgxBackend. * feat: Ochre vector store async ingestion (chunker, job store, worker) Adds Stage 3: a deterministic recursive chunk splitter, a per-account ingestion job-record store mirroring the index registry, the transactional per-document replace on the backend, and the IngestService worker that lists a predastore source, chunks and embeds each object, and replaces its rows idempotently. * feat: Ochre vector store similarity query and metadata filter grammar Adds VectorBackend.Query (cosine kNN, k default/cap, iterative HNSW scan under a filter) and a Bedrock-KB-shaped metadata Filter AST that compiles to parameterized SQL over jsonb, closing off arbitrary SQL by allowlisting jsonb keys and always binding values as params. Also fixes RunJob reporting READY on a job that ingested nothing regardless of failure cause, and stamps SourceSpec.Metadata onto ingested rows so the new filters have data to match against. * feat: Ochre vector store request/response API, VectorService, and CLI Adds the daemon-facing VectorService interface (index CRUD, ingest, describe job, query), its server implementation over the existing registry/backend/embedder, a NATS-forwarding client, and the `spx admin ochre vector` operator CLI. Query resolves and embeds against each index's own pinned model; ingest stamps the same model/dimension onto every source spec server-side. accountID is always taken from the caller context, never the request payload. Daemon subscription wiring and manifest registration are a later change; this is pure request/response plumbing with no construction into daemon.go or awsgw.go. * feat: wire Ochre vector store VectorService into the daemon Constructs the platform Postgres appliance and VectorService behind a new OchreVector config section that defaults off, so existing daemons are unaffected. Adds a real RDS-backed ApplianceLauncher that provisions the appliance over rds.CreateDBInstance/DescribeDBInstances, registers the six ochre.vector.* NATS subjects mirroring the bedrock endpoint service pattern, and degrades to a disabled (nil) service on any boot-time appliance failure rather than blocking daemon startup. * fix: fix Ochre vector store boot ordering and self-register subjects startOchreVector ran before subscribeAll, so its appliance launch fired rds.CreateDBInstance/DescribeDBInstances requests with no responder yet registered, timing out the launch on a first-enable boot. It also ran synchronously, blocking daemon startup up to the appliance launch timeout. Move it to a background goroutine started after subscribeAll returns, and have it register its own ochre.vector.* subjects once the VectorService is actually constructed, via a new registerNatsSubs helper shared with subscribeAll's boot-time table. Guard d.natsSubscriptions with d.mu, since this background registration can now race the shutdown path's unsubscribe-all sweep. * feat: bake pgvector into the shared rds-postgres image * fix: reach the ochre appliance over a routed daemon port on its ENI IP The daemon dialled the RDS appliance over rec.Endpoint, a vanity spx3.net hostname the control-plane host can neither resolve nor route to, so a healthy appliance was never reachable and the vector store disabled itself. Resolve the appliance's customer-ENI private IP read-only via ec2.DescribeNetworkInterfaces filtered on the spinifex:rds-db-instance tag, dial that IP for the DSN instead of the hostname, and give the daemon its own routed ENI in the appliance's subnet bound to an OVS host port so the IP is reachable. Retry appliance startup with bounded backoff instead of hard-disabling on the first transient. * fix: install pgvector at bootstrap for the Ochre vector appliance The daemon connects to the appliance as its NOSUPERUSER master, and the image's pgvector is not a trusted extension, so CREATE EXTENSION vector failed with permission denied. rds-init now installs it via the bootstrap superuser, gated on the appliance's fixed master name so tenant instances are unaffected. * fix: grant the Ochre appliance master CREATE on its database The daemon's EnsureAccount runs CREATE SCHEMA per account as the NOSUPERUSER master, which does not own the postgres database, so bootstrap now grants it CREATE on the database alongside the pgvector install. Both are gated to the appliance master, so tenant instances are unaffected. * fix: make the Ochre appliance's pgvector reachable under per-account isolation The daemon connects as a NOSUPERUSER master and runs every account operation under SET ROLE with a search_path pinned to the account schema. Two gaps blocked that against a real non-superuser Postgres: the master could not SET ROLE into the roles it creates, and the account role could not see the vector type. Bootstrap now grants the appliance master createrole_self_grant and installs pgvector into a read-only extensions schema granted to PUBLIC; the backend carries that schema on each account search_path so the vector type resolves without weakening per-account write isolation. All appliance-scoped. * fix: prune stale VPC host port before installing the daemon appliance port A replaced Ochre appliance is handed the same subnet IP, so a new daemon ENI mints a fresh vhp- port at an address a superseded port still holds. The old port survives at the OS/OVS layer even after its ENI record is wiped, so two interfaces claim one address and the kernel can route to the dead one (EHOSTUNREACH). installVPCHostPort now prunes any other vhp- port holding the target address before installing. * fix: gate VM restore's stale-socket kill on backing-store readiness On daemon start, a live VM whose NBD sockets fail to dial was force-killed as an orphan. While predastore/viperblock is still coming up after a restart those sockets are transiently unreachable, so a healthy VM (the Ochre appliance was the observed victim) was destroyed and its RDS record flipped to failed. The kill now only fires when the backing store is confirmed up; otherwise the live QEMU is kept and QMP reconnected. Genuine orphan reaping under a healthy store is unchanged. * fix: drive ochre ingest jobs from a daemon-side sweep scheduler * feat: add operator teardown verb for the ochre platform appliance Adds Appliance.Teardown to remove the singleton platform Postgres appliance (RDS instance, host port, KV record), an admin-only NATS subject and daemon handler to drive it, and `spx admin ochre appliance teardown --confirm` to invoke it without exposing the action through the tenant awsgw surface. * fix: register ochre appliance teardown before Connect so it survives a broken appliance * fix: purge ochre index and job metadata on appliance teardown * test: mark ochre in-package test files for the external-test gate
This commit is contained in:
Josh Siviour
@@ -10,8 +10,10 @@ IMAGE_SIZE="2G"
|
||||
# icu-data-full is deliberate: the cluster uses an ICU locale provider (musl has
|
||||
# no libc locales) and stock english-only data would reject other collations.
|
||||
# blkid is a hard requirement — without it rds-datadir refuses to run rather
|
||||
# than risk formatting a customer's datadir.
|
||||
APK_PACKAGES="postgresql18 postgresql18-client postgresql18-contrib postgresql18-openrc postgresql-common-openrc icu-data-full ca-certificates cloud-utils-growpart e2fsprogs-extra xfsprogs blkid openrc"
|
||||
# than risk formatting a customer's datadir. postgresql-pgvector ships the vector
|
||||
# extension for the Ochre appliance; it stays inert on tenant DBs until a
|
||||
# CREATE EXTENSION runs, so baking it into the shared image is harmless.
|
||||
APK_PACKAGES="postgresql18 postgresql18-client postgresql18-contrib postgresql-pgvector postgresql18-openrc postgresql-common-openrc icu-data-full ca-certificates cloud-utils-growpart e2fsprogs-extra xfsprogs blkid openrc"
|
||||
|
||||
# rds-agent is the only built binary (static, FIPS boot like the host build).
|
||||
BUILD_COMMANDS="CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -ldflags \"-s -w\" -o bin/rds-agent ./cmd/rds-agent"
|
||||
|
||||
@@ -85,6 +85,11 @@ MOUNTS_FILE="${RDS_MOUNTS_FILE:-/proc/mounts}"
|
||||
# AWS, so a customer's `GRANT rds_superuser TO ...` works here too.
|
||||
MASTER_GROUP_ROLE="rds_superuser"
|
||||
|
||||
# The Ochre vector store appliance is the one platform (non-tenant) consumer of
|
||||
# this image, provisioned with this fixed master name. Tenant instances take a
|
||||
# different master, so the pgvector install below is scoped to the appliance.
|
||||
OCHRE_APPLIANCE_MASTER="ochre_vector_admin"
|
||||
|
||||
# NAMEDATALEN-1: the server truncates a longer identifier, so the database
|
||||
# created would not be the one the customer asked for.
|
||||
MAX_DB_NAME_LEN=63
|
||||
@@ -152,6 +157,11 @@ load_handoff() {
|
||||
;;
|
||||
esac
|
||||
|
||||
# Scopes the platform pgvector install to the appliance; a tenant master
|
||||
# takes a different name and skips it.
|
||||
RDS_IS_OCHRE_APPLIANCE=0
|
||||
[ "${_master_lower}" = "${OCHRE_APPLIANCE_MASTER}" ] && RDS_IS_OCHRE_APPLIANCE=1
|
||||
|
||||
# The same rule the control plane applies, for the same reason the reserved
|
||||
# name is re-checked: an unusable name reaches initdb, fails at CREATE
|
||||
# DATABASE and spends the one-shot password on the way. psql quotes the
|
||||
@@ -496,6 +506,9 @@ bootstrap_master() {
|
||||
if [ "${_rc}" -eq 0 ] && [ -n "${RDS_DB_NAME}" ]; then
|
||||
create_database || _rc=$?
|
||||
fi
|
||||
if [ "${_rc}" -eq 0 ]; then
|
||||
create_platform_extensions || _rc=$?
|
||||
fi
|
||||
|
||||
# `set -e` is suspended for a function called from an AND-OR list, so the
|
||||
# stop status is captured rather than propagated. A failed stop must also not
|
||||
@@ -631,6 +644,23 @@ CREATE DATABASE :"dbname" OWNER :"master";
|
||||
SQL
|
||||
}
|
||||
|
||||
# pgvector is baked in but left uninstalled so tenant databases stay free of it.
|
||||
# The appliance master is NOSUPERUSER, so the bootstrap superuser installs the
|
||||
# extension into a read-only extensions schema and grants the master the database
|
||||
# and role privileges its daemon needs to create a per-account schema and role.
|
||||
# Scoped to the appliance master alone; tenant masters skip this entirely.
|
||||
create_platform_extensions() {
|
||||
[ "${RDS_IS_OCHRE_APPLIANCE:-0}" = "1" ] || return 0
|
||||
log "configuring pgvector and platform privileges for the Ochre vector appliance"
|
||||
psql_bootstrap <<SQL
|
||||
CREATE SCHEMA IF NOT EXISTS extensions;
|
||||
CREATE EXTENSION IF NOT EXISTS vector SCHEMA extensions;
|
||||
GRANT USAGE ON SCHEMA extensions TO PUBLIC;
|
||||
GRANT CREATE ON DATABASE postgres TO "${OCHRE_APPLIANCE_MASTER}";
|
||||
ALTER ROLE "${OCHRE_APPLIANCE_MASTER}" SET createrole_self_grant = 'set, inherit';
|
||||
SQL
|
||||
}
|
||||
|
||||
# Drops the master password from the handoff once applied, so it is not left
|
||||
# readable for the life of the boot. The rest is tmpfs and goes at reboot.
|
||||
consume_secrets() {
|
||||
|
||||
@@ -313,6 +313,15 @@ if run_ok "initialize"; then
|
||||
&& pass "initialize: password passed through the environment" || fail "initialize: password not in psql env"
|
||||
grep -q 'CREATE DATABASE' "${PSQL_CALLS}" \
|
||||
&& pass "initialize: initial database created" || fail "initialize: no CREATE DATABASE"
|
||||
grep -q 'CREATE EXTENSION' "${PSQL_CALLS}" \
|
||||
&& fail "initialize: a tenant instance installed a platform extension" \
|
||||
|| pass "initialize: tenant instance installs no platform extension"
|
||||
grep -q 'GRANT CREATE ON DATABASE' "${PSQL_CALLS}" \
|
||||
&& fail "initialize: a tenant instance was granted database CREATE" \
|
||||
|| pass "initialize: tenant instance gets no platform database grant"
|
||||
grep -q 'createrole_self_grant' "${PSQL_CALLS}" \
|
||||
&& fail "initialize: a tenant master got createrole_self_grant" \
|
||||
|| pass "initialize: tenant master gets no createrole_self_grant"
|
||||
|
||||
# The master role is administrative but not a PostgreSQL superuser: a
|
||||
# superuser reaches outside the database (COPY FROM PROGRAM, pg_read_file,
|
||||
@@ -369,6 +378,29 @@ if run_ok "attach"; then
|
||||
&& pass "attach: TLS survives a handoff without a new cert" || fail "attach: TLS turned off"
|
||||
fi
|
||||
|
||||
# --- Case 3b: the Ochre appliance master installs pgvector ---
|
||||
# No DBName, as the appliance is launched: create_database is skipped and the
|
||||
# gated extension install is the only platform-specific bootstrap step.
|
||||
reset_state
|
||||
MASTER_USER=ochre_vector_admin
|
||||
write_handoff initialize 's3cr3t' ''
|
||||
write_parameters
|
||||
if run_ok "ochre-appliance"; then
|
||||
grep -q 'CREATE EXTENSION IF NOT EXISTS vector SCHEMA extensions' "${PSQL_CALLS}" \
|
||||
&& pass "ochre-appliance: pgvector installed into the extensions schema" \
|
||||
|| fail "ochre-appliance: pgvector not installed into the extensions schema"
|
||||
grep -q 'GRANT USAGE ON SCHEMA extensions TO PUBLIC' "${PSQL_CALLS}" \
|
||||
&& pass "ochre-appliance: extensions schema readable by account roles" \
|
||||
|| fail "ochre-appliance: extensions schema not granted to PUBLIC"
|
||||
grep -q 'GRANT CREATE ON DATABASE postgres TO "ochre_vector_admin"' "${PSQL_CALLS}" \
|
||||
&& pass "ochre-appliance: master granted CREATE on the database" \
|
||||
|| fail "ochre-appliance: master not granted CREATE on the database"
|
||||
grep -q 'createrole_self_grant' "${PSQL_CALLS}" \
|
||||
&& pass "ochre-appliance: master can set into the roles it creates" \
|
||||
|| fail "ochre-appliance: master missing createrole_self_grant"
|
||||
fi
|
||||
unset MASTER_USER
|
||||
|
||||
# --- Case 3a: a parameter group cannot log the master password ---
|
||||
# The customer's parameters are installed before the master role is applied and
|
||||
# the bootstrap postmaster does not override config_file, so log_statement = all
|
||||
|
||||
Reference in New Issue
Block a user