applyPorts refuses a port whose SG policy cannot be programmed, but applyEIPs
still ran ensureGuestPortDatapath for every public-IP guest it skipped. GuestPortUp
reports false, not an error, for a port that does not exist, so each refused port
burned the full 45s deadline and ~9 ovn-controller recomputes before giving up.
* feat: give a reconcile pass its own revisit deadline, and convert the RDS loop
The reconciler assumed every loop was a pure function of the buckets it
watches. Quota and DNS are; RDS, EKS and ECS are not. They also wait on an
EC2 VM reaching running, an endpoint answering, and an agent falling silent,
none of which write to KV, so a watch never sees them.
Converting RDS on the watch alone would have traded traffic for detection
latency: an agent going silent writes nothing, so a dead database would have
been noticed on the 5 minute resync rather than within 15s of its heartbeat
expiring.
Reconcile and ReconcileKey now return a revisit deadline, clamped to the
resync. The RDS reconciler returns the earliest across the fleet: the old
interval for a transitional record, the instant the heartbeat expires for a
settled one, the resolve timeout for a snapshot stuck in creating, and
nothing at all for a region with no databases.
* fix: keep a deadline while a dark instance runs down its failure grace
revisitFor measured a settled instance against its heartbeat expiry, and
returned that difference directly. Once the beat expired the difference went
non-positive, which the loop reads as no deadline at all.
That is the window the detector lives in. Between the beat expiring and the
grace closing nothing writes to KV, so there was nothing left to schedule the
pass that does the failing, and a dead database would have waited for the
resync — slower than the ticker this replaced.
The deadline is now the beat expiry while it is still ahead, and whatever
remains of the failure grace once it is not. A failed instance asks for
nothing: it is terminal, and recovery arrives as a heartbeat, which is a write
the watch already delivers. Already due but still not failed is the degraded
reading, dark agent over a live VM, so it retries at the old interval rather
than immediately.
* feat: gate the Ochre console behind a runtime flag
Add OchreEnabled to the spinifexui service config and /api/config
body, defaulting to false so the console and every /bedrock/* route
stay hidden until a node explicitly turns it on.
* feat: wire the Ochre flag through to the spinifex-ui service unit
Add a spinifex-ui-ochre-enabled flag (SPINIFEX_UI_OCHRE_ENABLED, off
by default) so the console flag can actually be set on a deployed
node, and bump the service unit so the reconciler picks it up.
* feat: add knowledge base and data source create/delete to the console
Ochre's KB pages were read-only. Adds create + delete for knowledge
bases and their S3 data sources, mirroring the guardrail CRUD pattern.
* feat: note data source is added after KB creation
Users expect to pick a data source at create time; the create form now
says up front that S3 data sources are attached from the detail page
after the knowledge base exists, then ingested separately.
applyPorts dropped any security group whose OVN port group was missing and
carried on. When that removed the last one, the ENI's logical switch port was
created belonging to no port group, so no SG ACL applied — including the
priority 900/800 default-denies, which are attached per port group — and the
port was fully unrestricted until a later pass healed it.
Fail closed instead: a port whose policy cannot be programmed in full is
refused and reported unconverged, so the drift loop retries it once applySGs
catches up. The update path is skipped too, leaving a live guest on a stale
policy rather than stripping it bare.
EnsurePort and SetPortSecurityGroups get the same guard for an empty SG list.
Every ENI record written by CreateNetworkInterface has at least one SG, but the
field is omitempty with no migration, so a legacy row would otherwise yield an
unrestricted port.
The vCPU sweep ran on a 30s ticker and called the account-filtered
DescribeInstances fan-out once per account per pass, costing
passes x accounts x (nodes + 2) and measured at ~62,000 instance
describes a day, none of it user traffic and none of it tracking how
often anything actually changed.
The account is on the instance record and vCPUs derive from the instance
type, which is spec, so the whole sweep is one read of the record space.
The loop moves onto reconciler.Run: a change to one instance recomputes
that one account, and the resync sweeps every account as the backstop.
The completeness flag has to stay, for a different reason than the one it
was written for. nats.go hardcodes AllowDirect on KV buckets, so a get at
R3 may be served by a lagging replica, and ChargeLaunch increments the
counter while reconcile overwrites it absolutely. A scan that misses a
just-charged launch would lower the counter and lift the cap. The lower
is now written only while the counter still stands at the revision read
before the scan started, so a charge landing mid-scan wins and the scan
is dropped for the next pass.
Two behaviours change with the source. A failed sweep was a per-account
describe error the pass continued past; it is now one read covering every
account, so a failure moves no counter at all. And an account holding
nothing is absent from the scan rather than reporting empty, so the
caller's account list is what zeroes it.
The retype gate and the volume delete guard still use the fan-out; only
the sweep comes off it.
The loop watches KV buckets and runs a whole-set recompute when anything
in them changes, discarding the update it already holds. A caller whose
work is per resource can now set ReconcileKey and be handed the key that
changed, so a change to one instance costs one instance's work.
Reconcile stays required. A per-key queue only converges if something
behind it covers what the queue never learned: a key deleted while the
watch was down is enqueued by nothing, because nobody observes an
absence. Keeping the whole-set pass as the startup pass and the resync
backstop is what makes that real, and it makes the change additive — a
caller that sets only Reconcile behaves exactly as before.
KeyFor maps an update onto the caller's unit of work, which is not
always the KV key: fifty instances changing in one account are one
recompute. Reporting ok=false falls back to a whole-set pass, which is
the honest answer for a delete — a tombstone carries no value, so a
caller whose work key lives in the value cannot name it.
A dropped and re-established watch queues the whole set rather than any
key, since UpdatesOnly hides the gap. One worker drains the queue, and a
key taken off it is no longer pending, so a change arriving mid-reconcile
is queued again rather than folded into the pass already running.
* feat: decide instance set membership from the record, not the key prefix
* feat: cut instance records over to one key per instance
The record space becomes the only copy. Which set an instance belongs to
was structural before this — a stopped instance sat at instance.<id> and a
running one inside the node.<nodeID> blob, so it could not be wrong — and
is now a predicate over the record, decided in one place rather than at
each call site.
The old key spaces are frozen, not deleted. A migration that removed them
would make the crossing unrollbackable, and the node blob is the only
record of what a node was running that a build predating the cutover can
read. instance.<id> and terminated.<id> drain on the delete path.
The per-node key survives as a presence marker at its own prefix. Reads
need it because a node with no instances and a node the cluster has no
record of both scan empty, and only one of them may replace what the node
has locally. It carries no instances, so a state change no longer rewrites
the node's whole running set.
Claiming a stopped instance becomes a compare-and-set. It cannot wait for
a later change: with one key holding an instance for its whole life, a
claim that deletes is deleting the instance it is about to start. The
winner clears the stop intent and leaves LastNode alone, so an instance
whose launch never happens is still recoverable by the node that ran it.
A build predating the cutover now refuses to open a bucket stamped past
what it understands, rather than reading the keys it knows and writing
back a view assembled from half a key space.
* fix: carry node ownership out of the blob key during the cutover migration
The node.<id> key carried two facts in its name: that the node exists, and
which instances are its. The 4->5 migration moved the first and not the
second.
A blob member's own LastNode is empty, because the key was the owner, and so
is last_node on the record copied forward from it. After the cutover
ownership is read off the record, so an unowned record is one no node lists
- and with the presence marker seeded, LoadRunningState returns found=true
with an empty set, which restore adopts over the instances the node is
actually running.
The migration now stamps last_node from the blob's key onto every record in
it that has no owner, and leaves a record that already names one alone.
Found on a live cluster before crossing it. The test helpers built blob
members with LastNode already set, which production blobs never have; the
new tests seed them the way the bucket actually holds them.
* feat: add guardrail create, edit and delete to the console
Adds CRUD for Bedrock guardrails alongside the existing read-only view:
create and edit forms covering topic, content, word, PII, regex and
contextual-grounding policies, a delete confirmation flow, and a
create-version action, all built on the same real-SDK client pattern
already used for guardrail reads and runtime testing.
* fix: mount the guardrail edit form instead of no-opping the link
The guardrail detail route file dot-nested under the edit route
(detail as parent, edit as child), but the detail page renders no
outlet, so navigating to the edit URL matched the route without ever
mounting the edit form. Renaming the detail route file to an index
sibling of edit fixes the mount without changing any full path.
* feat: render predastore admin_port and consult /readyz in health probe
Spinifex now allocates predastore's admin listener port (8660, after the
existing gate/blob/meta bases) and renders admin_port into both the
single-node and multi-node predastore.toml templates. The daemon's health
probe consults /readyz over that port on the two weak paths (single-host
gate fallback and hosts running no meta node), falling back to the prior
behavior when no admin port is configured or the listener refuses.
* build: pin predastore to the release carrying Host.AdminPort
The admin port field this branch reads is new in predastore, so CI could not
compile against the pinned version: go.work hid the gap locally by resolving
predastore to the checkout instead of the module.
* feat: add admin Model Access console page
Lets an admin grant or revoke a provider-tier model for a tenant
account from the console, cross-referencing the model catalog with
that account's current grants via the existing signed-proxy actions.
* feat: replace free-text account id with a real account picker
Sources the Model Access account selector from /admin/ListAccounts
via a new signed JSON admin-API fetch helper, sorted by name and
excluding the global account. Falls back to a validated 12-digit
free-text account id when enumeration is unauthorized or empty, since
that call requires a long-lived IAM user in the super-admin account.
* feat: mirror each node's running set onto the i/<id> key space
* refactor: re-key instance records onto a dot separator
A reconciler Filter is a NATS subject filter, where "*" matches one
dot-delimited token. "i/<id>" is a single token, so "i/*" matches nothing
and a watch over the record space would have had to take the whole bucket
with ">" — waking on every key rather than on instance state, which is
worse than the "node.*" it replaces. "i.<id>" is two tokens, so "i.*" is
the filter the DNS reconcile can narrow to at the cutover.
The space stays disjoint from the three it replaces for the same reason
the slash did: List and DeletePrefix match a plain string prefix, and
"instance.i-1" does not begin "i.".
Each bucket gains a re-key step that moves "i/<id>" to "i.<id>" and
deletes the old key. Deleting is safe under the shadow rule: a node still
writing slashes also reads them, so what it loses is a mirror, and a
missing mirror falls back to the key it mirrors rather than erroring.
A pair of tests asserts both halves — that "i.*" delivers a record write
and that "i/*" does not — so the reasoning that chose the separator is
checked rather than trusted.
* test: benchmark what the node blob costs per state change
The case for splitting the blob is that one instance changing state
rewrites every instance on the node, and that cost grows with N while the
record's does not. That was asserted rather than measured.
Two benchmarks measure both sides on a record sized from env19, which
averages around 10KB terminated: marshalling the blob at N of 1, 10, 50
and 100, and marshalling one record. Each reports the bytes it produced
alongside the time, so the amplification is readable from the output
rather than needing a second calculation.
There is no assertion, because there is no threshold worth failing a
build on. Benchmarks compile in CI and run only when asked, which is what
a measurement wants — a test that cannot fail would run every pass and
gate nothing.
* fix: cover the embedder cold-start window with retry and Retry-After
Guardrail and knowledge-base embed calls hard-failed with a raw 503
whenever the embedding endpoint's connection was refused mid-relaunch,
instead of riding out the short window until it accepted connections.
embedBatch now retries a bounded number of times on connection-refused
or connection-reset, and any resulting 503 carries a Retry-After header
so callers back off instead of hammering the endpoint. A background
warm-up probe gates the embed path for statically pinned endpoints,
logging once when the endpoint first becomes reachable. Guardrail and
embedder infra logs now carry service/action labels for easier triage.
Fail-closed guardrail semantics are unchanged: an unavailable embedder
still denies unverified content, this only changes how quickly it
recovers once the endpoint is reachable again.
* fix: re-verify a persisted READY endpoint's liveness before resolving it
DynamicEndpointResolver's StateReady branch trusted the KV record's base
URL unconditionally. After an awsgw restart the record can still say
READY while the co-resident engine on that VM is mid-rebind, so the
resolver handed callers an address nothing was listening on yet.
describeAndEnsure now re-probes a READY member's own readiness route
before trusting it, short-TTL cached so a warm model still resolves at
no extra cost per request. A member that is not answering falls through
to the same not-ready path as a launch already in flight, so guardrail
and embed callers get the existing retryable answer instead of dialing
a dead address. This applies at the resolver seam, so it covers every
co-served member, not just the embedder.
The spinifex-ui reverse proxy copied only the decoded URL path and dropped
RawPath, so a percent-encoded path segment (a ':' in a bedrock modelId) was
re-encoded before reaching the gateway. The gateway then canonicalised a
different path than the one the browser SDK signed, and SigV4 rejected the
request with a signature mismatch — surfacing as an unparseable '<?xml'
deserialization error in the console Playground.
Carry RawPath through the rewrite so the escaped path arrives byte-identical.
Also route the auth, cluster-unavailable, and throttle error emitters through a
shared jsonErrorService helper so the bedrock family receives AWS JSON 1.1
errors the SDK can parse, and stamp the signed service onto the request before
verification so a rejection renders in the client's format.
* feat: add Ochre console Playground page
A Converse chat surface with model picker, system prompt, inference params,
optional guardrail, per-turn tokens, raw request/response JSON, and copy-as-CLI.
The transport lives in one module so a later ConverseStream swap edits a single
file; the message list appends and mutates the last turn in place. A cold model
degrades to a retryable warming-up turn instead of an error.
* feat: pin the playground composer, send on Enter, render markdown
The chat composer sat below the message list in page flow, so it slid down as
the conversation grew and the reader had to scroll to reach it. Bound the
chat + composer to a fixed-height column: the message pane scrolls internally
and autoscrolls to the newest turn while the composer stays pinned.
Enter now sends (Shift+Enter inserts a newline), and assistant replies render
markdown — lists, inline code, code blocks, and links — instead of raw text.
Adds the accessors for the per-resource instance record key space, over both
buckets that carry instance records. Nothing calls them yet.
The prefix is disjoint from the three it replaces: List and DeletePrefix match
on a plain string prefix rather than on NATS subject tokens, so i/ shares a
bucket with instance., node. and terminated. without colliding with any of them.
mutateVM and listVMs become generic so the record accessors reuse them rather
than carrying a second copy.
InstanceRecord is resource.Object[InstanceSpec, InstanceStatus], the persisted
form of an instance under the per-resource key space the re-key introduces.
Nothing reads or writes it yet: the boundary is settled and tested before the
keys move onto it.
The wrappers around the EBS and ENI request slices are dropped on the way in
and rebuilt on the way out, so the record does not carry a mutex.
Instance and Reservation are carried whole. They are SDK types returned
verbatim by DescribeInstances and hold both halves in one object, so the
boundary does not hold there and the comment says so.
Object[Spec, Status] separates what an operator asked for from what a node
observed, with the identity and lifecycle bookkeeping common to both.
Generation is bumped only through MutateSpec so NeedsSync means something, and
ObserveGeneration records the generation the node acted on rather than the
current one, so a spec change landing mid-reconcile is not marked done.
No ARN or tag helpers: the arn package already builds ARNs per service, and the
tag helpers in utils already cover the map every record carries.
StateReason was cleared by ModifyInstanceAttribute, so the API wrote a field
the node owns. Move the clear onto the launch path.
ExtraHostfwd held the requested guest port and the bound host port in one map,
with zero standing in for both 'not bound yet' and a real value. Split it into
HostfwdPorts and HostfwdPortMap, rebuilt per launch, and fold legacy records on
decode.
The command an instance was last handled with was stored on the record, but
only stop_instance and delete_instance were ever read back. Replace them with
DesiredState and DeletionTimestamp, which say what the read sites actually
mean, and drop the other twelve booleans.
Legacy records are folded on decode so an operator-stopped instance is not
relaunched on upgrade.
A staged self-host model is granted to every account without a per-account
grant, making the admin import the only v1 gate. Wrap the access resolver so
every gate opens together; the admin catalog reports self-host state
weights-first. Per-account grants remain as a dormant seam.
Operator-only catalog page listing every model with modality, streaming,
price, VRAM/instance, co-serve group, and availability with a fix-affordance
hint per reason. Reads the ListOchreCatalog admin action, mirroring the
existing admin-service query pattern.
Expose per-model availability and reason on an operator-only ListOchreCatalog
action for the console, keeping the tenant ListFoundationModels omission
intact. Remove the unbuilt provider-direct entry from the v1 catalog.
The server queues the API audit advisory after the reply the caller
waits on, so subscribing once the bucket exists could still catch the
create the first call made. Watch from the start and account for it.
The handler records its request point after it replies, so asserting as
soon as Request returns could read the metric before it landed. Poll for
the outcome the way the sibling drain test does.
The 40ms apply-params timeout was shared by the tests that expect the
agent to answer and those that expect it not to, so a parallel coverage
run could miss the reply. Only the unreachable-agent cases keep the
short budget.
* feat: semantic topic detection for guardrails via embedding similarity
Guardrail topic policy previously only blocked on an exact word-boundary
match against a denied topic's name or listed examples, so a paraphrase or
synonym for a blocked topic sailed straight through. This adds an
embedding-similarity check: each denied topic's name, definition, and
examples are embedded and cached, incoming text is embedded and compared by
max cosine similarity against those phrase vectors, and a topic blocks when
that similarity clears a conservative threshold. The literal exact-match
check still runs first and always blocks on its own. If no embedder is
configured, or an embed call fails, the check falls back to the existing
literal match alone rather than ever failing open.
* fix: fail closed when guardrail topic embedding is unavailable
The topic policy's semantic check silently fell back to its literal-only
matcher whenever the embedder was configured but temporarily unreachable
(for example during model cold-start), so a paraphrase of a denied topic
could pass through unblocked while the embedding endpoint was down. Now that
case is distinguished from an unconfigured embedder: if literal matching
already caught the request, that block still wins outright. Otherwise, when
the embedder is wired but the embed call itself fails, the guardrail check
returns an error and the caller aborts the request rather than letting
unverified content through.
* feat: converge DNS on change rather than on a timer
The drift backstop now watches the four stores dnsDesiredSet reads: instance state, ELBv2 load balancers, and the per-account EKS and RDS buckets. Only the trigger changes; the leader election, computeBatch and the publish path are untouched.
The interval becomes the backstop for drift the buckets cannot report, such as a record edited directly in the zone object, rather than the path by which a lifecycle event reaches DNS.
* test: cover the DNS watch wiring and reconciler run path
Julian Sommer