gsadmin/spinifex
1
0
Fork 0
mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-15 17:11:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
1e00425bb302e56e97a4eaeb389dc0cdc59641b8
spinifex/docs/service-interfaces.yaml
T
Josh Siviour d52245001e feat: Ochre vector store (tenant pgvector, embeddings, ingestion, operator teardown) (#788) 
* feat: add /v1/embeddings gateway adapter

* feat: Ochre vector store pgvector backend, KV index registry, orchestration

Adds handlers_ochrevector: the VectorBackend interface + pgx/v5 impl
with per-account schema/role isolation enforced via SET LOCAL ROLE +
search_path inside every transaction, the KV index registry
(CREATING/READY/DELETING/STALE), and the create/delete/list/reconcile
orchestration tying them together with crash-safe rollback and
re-entrant recovery of stuck records.

* feat: singleton appliance-ensure state machine for ochre vector store

Adds the encrypted control-plane credential store and re-entrant
Ensure state machine for the platform pgvector appliance (D2): a
fixed-key KV claim mirroring Registry.Reserve elects exactly one
winner to launch the appliance via an injected ApplianceLauncher,
losers wait or resume a stale crashed provision, and the master
password is generated once, stored only AES-256-GCM encrypted, and
decrypted solely inside Connect to build a pgxBackend.

* feat: Ochre vector store async ingestion (chunker, job store, worker)

Adds Stage 3: a deterministic recursive chunk splitter, a per-account
ingestion job-record store mirroring the index registry, the
transactional per-document replace on the backend, and the
IngestService worker that lists a predastore source, chunks and
embeds each object, and replaces its rows idempotently.

* feat: Ochre vector store similarity query and metadata filter grammar

Adds VectorBackend.Query (cosine kNN, k default/cap, iterative HNSW
scan under a filter) and a Bedrock-KB-shaped metadata Filter AST that
compiles to parameterized SQL over jsonb, closing off arbitrary SQL by
allowlisting jsonb keys and always binding values as params. Also
fixes RunJob reporting READY on a job that ingested nothing regardless
of failure cause, and stamps SourceSpec.Metadata onto ingested rows so
the new filters have data to match against.

* feat: Ochre vector store request/response API, VectorService, and CLI

Adds the daemon-facing VectorService interface (index CRUD, ingest,
describe job, query), its server implementation over the existing
registry/backend/embedder, a NATS-forwarding client, and the
`spx admin ochre vector` operator CLI. Query resolves and embeds
against each index's own pinned model; ingest stamps the same
model/dimension onto every source spec server-side. accountID is
always taken from the caller context, never the request payload.

Daemon subscription wiring and manifest registration are a later
change; this is pure request/response plumbing with no construction
into daemon.go or awsgw.go.

* feat: wire Ochre vector store VectorService into the daemon

Constructs the platform Postgres appliance and VectorService behind a new
OchreVector config section that defaults off, so existing daemons are
unaffected. Adds a real RDS-backed ApplianceLauncher that provisions the
appliance over rds.CreateDBInstance/DescribeDBInstances, registers the six
ochre.vector.* NATS subjects mirroring the bedrock endpoint service pattern,
and degrades to a disabled (nil) service on any boot-time appliance failure
rather than blocking daemon startup.

* fix: fix Ochre vector store boot ordering and self-register subjects

startOchreVector ran before subscribeAll, so its appliance launch fired
rds.CreateDBInstance/DescribeDBInstances requests with no responder yet
registered, timing out the launch on a first-enable boot. It also ran
synchronously, blocking daemon startup up to the appliance launch timeout.

Move it to a background goroutine started after subscribeAll returns, and
have it register its own ochre.vector.* subjects once the VectorService is
actually constructed, via a new registerNatsSubs helper shared with
subscribeAll's boot-time table. Guard d.natsSubscriptions with d.mu, since
this background registration can now race the shutdown path's
unsubscribe-all sweep.

* feat: bake pgvector into the shared rds-postgres image

* fix: reach the ochre appliance over a routed daemon port on its ENI IP

The daemon dialled the RDS appliance over rec.Endpoint, a vanity spx3.net
hostname the control-plane host can neither resolve nor route to, so a healthy
appliance was never reachable and the vector store disabled itself.

Resolve the appliance's customer-ENI private IP read-only via
ec2.DescribeNetworkInterfaces filtered on the spinifex:rds-db-instance tag, dial
that IP for the DSN instead of the hostname, and give the daemon its own routed
ENI in the appliance's subnet bound to an OVS host port so the IP is reachable.
Retry appliance startup with bounded backoff instead of hard-disabling on the
first transient.

* fix: install pgvector at bootstrap for the Ochre vector appliance

The daemon connects to the appliance as its NOSUPERUSER master, and the
image's pgvector is not a trusted extension, so CREATE EXTENSION vector
failed with permission denied. rds-init now installs it via the bootstrap
superuser, gated on the appliance's fixed master name so tenant instances
are unaffected.

* fix: grant the Ochre appliance master CREATE on its database

The daemon's EnsureAccount runs CREATE SCHEMA per account as the NOSUPERUSER
master, which does not own the postgres database, so bootstrap now grants it
CREATE on the database alongside the pgvector install. Both are gated to the
appliance master, so tenant instances are unaffected.

* fix: make the Ochre appliance's pgvector reachable under per-account isolation

The daemon connects as a NOSUPERUSER master and runs every account operation
under SET ROLE with a search_path pinned to the account schema. Two gaps blocked
that against a real non-superuser Postgres: the master could not SET ROLE into
the roles it creates, and the account role could not see the vector type.

Bootstrap now grants the appliance master createrole_self_grant and installs
pgvector into a read-only extensions schema granted to PUBLIC; the backend
carries that schema on each account search_path so the vector type resolves
without weakening per-account write isolation. All appliance-scoped.

* fix: prune stale VPC host port before installing the daemon appliance port

A replaced Ochre appliance is handed the same subnet IP, so a new daemon ENI
mints a fresh vhp- port at an address a superseded port still holds. The old
port survives at the OS/OVS layer even after its ENI record is wiped, so two
interfaces claim one address and the kernel can route to the dead one
(EHOSTUNREACH). installVPCHostPort now prunes any other vhp- port holding the
target address before installing.

* fix: gate VM restore's stale-socket kill on backing-store readiness

On daemon start, a live VM whose NBD sockets fail to dial was force-killed as an
orphan. While predastore/viperblock is still coming up after a restart those
sockets are transiently unreachable, so a healthy VM (the Ochre appliance was
the observed victim) was destroyed and its RDS record flipped to failed. The
kill now only fires when the backing store is confirmed up; otherwise the live
QEMU is kept and QMP reconnected. Genuine orphan reaping under a healthy store
is unchanged.

* fix: drive ochre ingest jobs from a daemon-side sweep scheduler

* feat: add operator teardown verb for the ochre platform appliance

Adds Appliance.Teardown to remove the singleton platform Postgres
appliance (RDS instance, host port, KV record), an admin-only NATS
subject and daemon handler to drive it, and `spx admin ochre appliance
teardown --confirm` to invoke it without exposing the action through
the tenant awsgw surface.

* fix: register ochre appliance teardown before Connect so it survives a broken appliance

* fix: purge ochre index and job metadata on appliance teardown

* test: mark ochre in-package test files for the external-test gate
2026-08-13 15:54:48 +10:00

402 lines
13 KiB
YAML
Raw Blame History

# Service interface manifest.
#
# Source of truth for "which services produce/consume which NATS subjects"
# and "which e2e suites cover which services". Consumed by
# tests/e2e/select-suites (Bead 4) to derive targeted suite selection from
# a PR diff.
#
# See docs/development/improvements/e2e-targeted-suite-selection.md.
#
# Editing rules
# -------------
# - subscribes / publishes: subjects this service receives / sends. Both
# lists use the literal subject string, or a "<prefix>.*" wildcard for
# pattern subscriptions (e.g. "ebs.snapshot.*").
# - depends_on: subjects this service requires from peers. Used by the
# selector to compute "if X changes, which suites might break".
# - Subjects must match what is actually referenced in source. The lint
# in Bead 5 greps `"(ec2|vpc|ebs|s3|iam|elbv2|spinifex)\.[a-zA-Z._-]+"`
# and fails if any subject is absent from this manifest.
version: 1
services:
awsgw:
path: spinifex/services/awsgw
# TLS frontend, SigV4 auth. Pure router: HTTP → NATS request/response.
# No subscribes; only publishes (via NATSRequest).
subscribes: []
publishes:
- "ec2.*"
- "elbv2.*"
- "acm.*"
- "iam.*"
- "rds.*"
depends_on:
- "ec2.*"
- "elbv2.*"
- "acm.*"
- "iam.*"
- "rds.*"
spinifex_daemon:
path: spinifex/services/spinifex
# Orchestrator. Owns most ec2.* + all elbv2.* + iam event publication.
# Translates VPC / EBS / S3 operations into vpc.* / ebs.* / s3.* and
# forwards to vpcd / viperblockd / predastore.
#
# Source layout: spinifex/services/spinifex (entry), spinifex/daemon
# (subscribe registration), spinifex/handlers/{ec2,elbv2,iam} (per-
# subject handler bodies). Selector treats all four paths as belonging
# to this service via the `additional_paths` field.
additional_paths:
- spinifex/daemon
- spinifex/handlers/ec2
- spinifex/handlers/elbv2
- spinifex/handlers/acm
- spinifex/handlers/iam
- spinifex/handlers/rds
- spinifex/handlers/ochrevector
subscribes:
# EC2 instance lifecycle.
- "ec2.RunInstances"
- "ec2.DescribeInstances"
- "ec2.DescribeStoppedInstances"
- "ec2.DescribeTerminatedInstances"
- "ec2.DescribeInstanceAttribute"
- "ec2.DescribeInstanceStatus"
- "ec2.DescribeInstanceTypes"
- "ec2.ModifyInstanceAttribute"
- "ec2.ModifyVolume"
- "ec2.start"
- "ec2.terminate"
- "ec2.launch"
- "ec2.cmd.*"
# EC2 VPC / network ops handled by daemon before delegation to vpcd.
- "ec2.CreateVpc"
- "ec2.DescribeVpcs"
- "ec2.CreateSubnet"
- "ec2.CreateInternetGateway"
- "ec2.CreateKeyPair"
- "ec2.CreateImage"
- "ec2.DescribePlacementGroups"
# Cluster coordination subjects.
- "ec2.ReserveClusterNode"
- "ec2.FinalizeClusterInstances"
- "spinifex.node.status"
- "spinifex.node.vms"
- "spinifex.nodes.discover"
- "spinifex.cluster.shutdown.progress"
# Coordinated cluster-shutdown phases (fan-out, no queue group).
- "spinifex.cluster.shutdown.gate"
- "spinifex.cluster.shutdown.drain"
- "spinifex.cluster.shutdown.storage"
- "spinifex.cluster.shutdown.persist"
- "spinifex.cluster.shutdown.infra"
- "spinifex.storage.config"
# ELBv2 control plane (handlers/elbv2/service_nats.go topic map).
- "elbv2.CreateLoadBalancer"
- "elbv2.DeleteLoadBalancer"
- "elbv2.DescribeLoadBalancers"
- "elbv2.CreateTargetGroup"
- "elbv2.DeleteTargetGroup"
- "elbv2.DescribeTargetGroups"
- "elbv2.RegisterTargets"
- "elbv2.DeregisterTargets"
- "elbv2.DescribeTargetHealth"
- "elbv2.CreateListener"
- "elbv2.DeleteListener"
- "elbv2.DescribeListeners"
# ACM-alike certificate store (handlers/acm/service_nats.go topic map).
- "acm.ImportCertificate"
- "acm.DescribeCertificate"
- "acm.ListCertificates"
- "acm.DeleteCertificate"
- "elbv2.AddListenerCertificates"
- "elbv2.RemoveListenerCertificates"
- "elbv2.DescribeListenerCertificates"
- "elbv2.DescribeSSLPolicies"
# RDS control plane (handlers/rds/subjects.go). One wildcard rather than
# 30 literals: every action subject is registered from a constant, so a
# per-subject list would be a hand-mirror of that file with nothing
# checking it.
- "rds.*"
# Layer-2 agent bus: register and health are wildcard subscriptions over
# {account}.{db-instance}, so the subject is built at runtime and only the
# prefix is declarable.
- "rds.bus.*"
# Ochre vector store tenant surface (handlers/ochrevector/api.go),
# gated behind config.OchreVectorConfig.Enabled (default off).
- "ochre.vector.createIndex"
- "ochre.vector.deleteIndex"
- "ochre.vector.listIndexes"
- "ochre.vector.ingest"
- "ochre.vector.describeJob"
- "ochre.vector.query"
publishes:
# → vpcd
- "vpc.create"
- "vpc.delete"
- "vpc.create-subnet"
- "vpc.delete-subnet"
- "vpc.create-sg"
- "vpc.delete-sg"
- "vpc.update-sg"
- "vpc.create-port"
- "vpc.delete-port"
- "vpc.update-port-sgs"
- "vpc.igw-attach"
- "vpc.igw-detach"
- "vpc.add-nat-gateway"
- "vpc.delete-nat-gateway"
- "vpc.add-nat"
- "vpc.delete-nat"
- "vpc.dhcp.acquire"
- "vpc.dhcp.release"
# → viperblockd
- "ebs.mount"
- "ebs.unmount"
- "ebs.delete"
- "ebs.sync"
- "ebs.snapshot.*"
- "ebs.config"
- "ebs.config.*"
# → predastore (S3 / AMI / keypair store)
- "s3.*"
# IAM lifecycle event
- "iam.account.created"
# → the in-guest rds-agent: set-password and the like, published on the
# per-instance command subject and answered on its reply subject.
- "rds.bus.*"
depends_on:
- "vpc.*"
- "ebs.*"
- "s3.*"
vpcd:
path: spinifex/vpcd
# OVN-backed VPC / subnet / SG / NAT / IGW data plane.
subscribes:
- "vpc.create"
- "vpc.delete"
- "vpc.create-subnet"
- "vpc.delete-subnet"
- "vpc.create-sg"
- "vpc.delete-sg"
- "vpc.update-sg"
- "vpc.create-port"
- "vpc.delete-port"
- "vpc.update-port-sgs"
- "vpc.igw-attach"
- "vpc.igw-detach"
- "vpc.add-nat-gateway"
- "vpc.delete-nat-gateway"
- "vpc.add-nat"
- "vpc.delete-nat"
- "vpc.dhcp.acquire"
- "vpc.dhcp.release"
publishes:
- "vpc.port-status"
- "vpc.dhcp.lease-changed"
- "vpc.dhcp.owner-check"
depends_on: []
viperblockd:
path: spinifex/services/viperblockd
# EBS data plane. Backed by the viperblock storage engine in the
# mulga/viperblock submodule. Submodule pointer bump → selector forces
# "run all" (contract changes can be invisible at the spinifex source
# level).
subscribes:
- "ebs.mount"
- "ebs.unmount"
- "ebs.delete"
- "ebs.sync"
- "ebs.snapshot.*"
- "ebs.config"
- "ebs.config.*"
publishes:
- "ebs.mount.response"
- "ebs.unmount.response"
depends_on: []
predastore:
path: spinifex/services/predastore
# S3 / AMI / keypair storage. External submodule under mulga/predastore;
# this dir contains only the spinifex-side adapter. Submodule pointer
# bump → selector forces "run all".
subscribes:
- "s3.*"
publishes: []
depends_on: []
nats:
path: spinifex/services/nats
# NATS broker bootstrap / cluster config. Not a NATS pub/sub
# participant. Any change here is infra-path; selector forces
# "run all".
subscribes: []
publishes: []
depends_on: []
spinifexui:
path: spinifex/services/spinifexui
# Web UI. Talks to awsgw over HTTP, never NATS. Diffs limited to this
# dir do not need any e2e suite to run.
subscribes: []
publishes: []
depends_on: []
covers_no_suite: true
# Fixture → service mapping. Each Ensure* helper in
# tests/e2e/harness/fixtures.go exercises this set of services; the
# Bead 4 selector AST-scans test files for harness.Ensure* call sites
# and resolves coverage via this block.
fixtures:
EnsureKeyPair:
services: [spinifex_daemon, predastore]
EnsureAMI:
services: [spinifex_daemon, predastore, viperblockd]
EnsureDefaultVPC:
services: [vpcd, spinifex_daemon]
EnsureSubnet:
services: [vpcd, spinifex_daemon]
EnsureSG:
services: [vpcd, spinifex_daemon]
EnsureInstance:
services: [spinifex_daemon, viperblockd, vpcd]
EnsureVolume:
services: [spinifex_daemon, viperblockd]
EnsureSnapshot:
services: [spinifex_daemon, viperblockd, predastore]
EnsureNATGateway:
services: [vpcd, spinifex_daemon]
EnsureLoadBalancer:
services: [spinifex_daemon, awsgw]
suites:
e2e-cert:
path: tests/e2e/cert
covers: [awsgw, spinifex_daemon, vpcd, predastore]
e2e-lb:
path: tests/e2e/lb
covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
# Exercises full ELBv2 control plane (handlers/elbv2) + data plane
# (LB agent on the system VM). ELBv2 is part of spinifex_daemon in
# this manifest; revisit if it splits into a standalone service.
e2e-single:
path: tests/e2e/single
covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
# Full single-node lifecycle: VPC, AMI, keypair, instance, volume,
# snapshot, IAM. Currently runs all phases sequentially under one
# `TestSingleNode`. Bead 3 (single-fixture-migration) splits this
# into per-test fixture-driven coverage so the selector can pick
# subtests, not just whole suites.
e2e-iam:
path: tests/e2e/iam
covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
# "Everything IAM": the IAM/STS control plane plus the instance-identity
# tests (instance-profile association + IMDS), which boot guest VMs — OVN
# via vpcd, AMI via predastore, root volume via viperblockd, lifecycle via
# spinifex_daemon. Single-mode, VM-booting leg.
e2e-ecr:
path: tests/e2e/ecr
covers: [awsgw, spinifex_daemon, predastore]
# ECR control plane (handlers/ecr, gateway/ecr*) + OCI data plane: blobs and
# manifests in predastore, control + auth bridge in awsgw, KV via
# spinifex_daemon. docker/crane/skopeo data-plane subtests skip when the
# client binary or registry-host resolution is absent.
e2e-bedrock:
path: tests/e2e/bedrock
covers: [awsgw]
# Ochre (AWS-Bedrock) control plane: static catalog list/get served
# straight from awsgw (gateway/bedrock.go, gateway/bedrock/) — no NATS
# hop to spinifex_daemon. Converse (Anthropic-direct / self-host vLLM)
# subtests are gated on backend availability and skip by default.
e2e-ecs:
path: tests/e2e/ecs
covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
# ECS data plane end-to-end: cluster + a real container instance launched
# from the spinifex-ecs-node AMI (boots, registers over the gateway, runs
# tasks through containerd), standalone tasks in awsvpc/bridge/host modes,
# and an awsvpc service fronted by an ELBv2 target group. Boots a guest VM,
# so vpcd (OVN + awsvpc ENI), viperblockd (root volume), predastore (AMI)
# are all in the path.
e2e-rds:
path: tests/e2e/rds
covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
# RDS create/describe/connect: CreateDBInstance boots a dual-NIC DB VM from
# the engine AMI (predastore) with a Viperblock data volume (viperblockd) and
# a customer ENI (vpcd), and the reconciler flips it to available on the
# in-guest agent's first healthy heartbeat over awsgw. Every test deletes the
# instances it creates, so the suite needs no gate beyond SPINIFEX_E2E.
e2e-ddil:
path: tests/e2e/ddil
covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
# DDIL = degraded / disconnected / intermittent / low-bandwidth.
# Exercises the full daemon stack under network failure injection;
# any service touching the wire path is in scope.
# Suites below are bash-only today. Entries activate as the Go ports
# land (Bead 2 fixtures unblock new-shape ports for multinode etc.).
#
# e2e-multinode:
# path: spinifex/tests/e2e/multinode
# covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
# e2e-reboot:
# path: spinifex/tests/e2e/reboot
# covers: [spinifex_daemon, viperblockd]
# e2e-tofu:
# path: spinifex/tests/e2e/tofu
# covers: [awsgw, spinifex_daemon, vpcd]
# e2e-bm:
# path: spinifex/tests/e2e/baremetal
# covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
#
# e2e-natuplink activates once CI can provision an external_mode=nat
# node (setup-ovn.sh --nat-uplink + spx admin init --external-mode=nat).
# e2e-natuplink:
# path: spinifex/tests/e2e/natuplink
# covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
# Special-case rules consumed by the selector (Bead 4). Hard-coded in
# the tool, listed here for visibility.
#
# infra_paths_force_all:
# - spinifex/services/nats/**
# - spinifex/tests/e2e/harness/**
# - spinifex/tests/e2e/lib/**
# - spinifex/.github/workflows/e2e-go.yml
# - spinifex/.github/scripts/**
# - spinifex/Makefile
# - mulga/scripts/tofu-cluster/**
#
# submodule_force_all:
# - mulga/predastore (submodule pointer)
# - mulga/viperblock (submodule pointer)
#
# many_services_force_all_threshold: 3 # ≥ N services touched → run all
Reference in New Issue View Git Blame Copy Permalink