The vCPU sweep ran on a 30s ticker and called the account-filtered
DescribeInstances fan-out once per account per pass, costing
passes x accounts x (nodes + 2) and measured at ~62,000 instance
describes a day, none of it user traffic and none of it tracking how
often anything actually changed.
The account is on the instance record and vCPUs derive from the instance
type, which is spec, so the whole sweep is one read of the record space.
The loop moves onto reconciler.Run: a change to one instance recomputes
that one account, and the resync sweeps every account as the backstop.
The completeness flag has to stay, for a different reason than the one it
was written for. nats.go hardcodes AllowDirect on KV buckets, so a get at
R3 may be served by a lagging replica, and ChargeLaunch increments the
counter while reconcile overwrites it absolutely. A scan that misses a
just-charged launch would lower the counter and lift the cap. The lower
is now written only while the counter still stands at the revision read
before the scan started, so a charge landing mid-scan wins and the scan
is dropped for the next pass.
Two behaviours change with the source. A failed sweep was a per-account
describe error the pass continued past; it is now one read covering every
account, so a failure moves no counter at all. And an account holding
nothing is absent from the scan rather than reporting empty, so the
caller's account list is what zeroes it.
The retype gate and the volume delete guard still use the fan-out; only
the sweep comes off it.
The loop watches KV buckets and runs a whole-set recompute when anything
in them changes, discarding the update it already holds. A caller whose
work is per resource can now set ReconcileKey and be handed the key that
changed, so a change to one instance costs one instance's work.
Reconcile stays required. A per-key queue only converges if something
behind it covers what the queue never learned: a key deleted while the
watch was down is enqueued by nothing, because nobody observes an
absence. Keeping the whole-set pass as the startup pass and the resync
backstop is what makes that real, and it makes the change additive — a
caller that sets only Reconcile behaves exactly as before.
KeyFor maps an update onto the caller's unit of work, which is not
always the KV key: fifty instances changing in one account are one
recompute. Reporting ok=false falls back to a whole-set pass, which is
the honest answer for a delete — a tombstone carries no value, so a
caller whose work key lives in the value cannot name it.
A dropped and re-established watch queues the whole set rather than any
key, since UpdatesOnly hides the gap. One worker drains the queue, and a
key taken off it is no longer pending, so a change arriving mid-reconcile
is queued again rather than folded into the pass already running.
* feat: decide instance set membership from the record, not the key prefix
* feat: cut instance records over to one key per instance
The record space becomes the only copy. Which set an instance belongs to
was structural before this — a stopped instance sat at instance.<id> and a
running one inside the node.<nodeID> blob, so it could not be wrong — and
is now a predicate over the record, decided in one place rather than at
each call site.
The old key spaces are frozen, not deleted. A migration that removed them
would make the crossing unrollbackable, and the node blob is the only
record of what a node was running that a build predating the cutover can
read. instance.<id> and terminated.<id> drain on the delete path.
The per-node key survives as a presence marker at its own prefix. Reads
need it because a node with no instances and a node the cluster has no
record of both scan empty, and only one of them may replace what the node
has locally. It carries no instances, so a state change no longer rewrites
the node's whole running set.
Claiming a stopped instance becomes a compare-and-set. It cannot wait for
a later change: with one key holding an instance for its whole life, a
claim that deletes is deleting the instance it is about to start. The
winner clears the stop intent and leaves LastNode alone, so an instance
whose launch never happens is still recoverable by the node that ran it.
A build predating the cutover now refuses to open a bucket stamped past
what it understands, rather than reading the keys it knows and writing
back a view assembled from half a key space.
* fix: carry node ownership out of the blob key during the cutover migration
The node.<id> key carried two facts in its name: that the node exists, and
which instances are its. The 4->5 migration moved the first and not the
second.
A blob member's own LastNode is empty, because the key was the owner, and so
is last_node on the record copied forward from it. After the cutover
ownership is read off the record, so an unowned record is one no node lists
- and with the presence marker seeded, LoadRunningState returns found=true
with an empty set, which restore adopts over the instances the node is
actually running.
The migration now stamps last_node from the blob's key onto every record in
it that has no owner, and leaves a record that already names one alone.
Found on a live cluster before crossing it. The test helpers built blob
members with LastNode already set, which production blobs never have; the
new tests seed them the way the bucket actually holds them.
* feat: add guardrail create, edit and delete to the console
Adds CRUD for Bedrock guardrails alongside the existing read-only view:
create and edit forms covering topic, content, word, PII, regex and
contextual-grounding policies, a delete confirmation flow, and a
create-version action, all built on the same real-SDK client pattern
already used for guardrail reads and runtime testing.
* fix: mount the guardrail edit form instead of no-opping the link
The guardrail detail route file dot-nested under the edit route
(detail as parent, edit as child), but the detail page renders no
outlet, so navigating to the edit URL matched the route without ever
mounting the edit form. Renaming the detail route file to an index
sibling of edit fixes the mount without changing any full path.
* feat: render predastore admin_port and consult /readyz in health probe
Spinifex now allocates predastore's admin listener port (8660, after the
existing gate/blob/meta bases) and renders admin_port into both the
single-node and multi-node predastore.toml templates. The daemon's health
probe consults /readyz over that port on the two weak paths (single-host
gate fallback and hosts running no meta node), falling back to the prior
behavior when no admin port is configured or the listener refuses.
* build: pin predastore to the release carrying Host.AdminPort
The admin port field this branch reads is new in predastore, so CI could not
compile against the pinned version: go.work hid the gap locally by resolving
predastore to the checkout instead of the module.
* feat: add admin Model Access console page
Lets an admin grant or revoke a provider-tier model for a tenant
account from the console, cross-referencing the model catalog with
that account's current grants via the existing signed-proxy actions.
* feat: replace free-text account id with a real account picker
Sources the Model Access account selector from /admin/ListAccounts
via a new signed JSON admin-API fetch helper, sorted by name and
excluding the global account. Falls back to a validated 12-digit
free-text account id when enumeration is unauthorized or empty, since
that call requires a long-lived IAM user in the super-admin account.
* feat: mirror each node's running set onto the i/<id> key space
* refactor: re-key instance records onto a dot separator
A reconciler Filter is a NATS subject filter, where "*" matches one
dot-delimited token. "i/<id>" is a single token, so "i/*" matches nothing
and a watch over the record space would have had to take the whole bucket
with ">" — waking on every key rather than on instance state, which is
worse than the "node.*" it replaces. "i.<id>" is two tokens, so "i.*" is
the filter the DNS reconcile can narrow to at the cutover.
The space stays disjoint from the three it replaces for the same reason
the slash did: List and DeletePrefix match a plain string prefix, and
"instance.i-1" does not begin "i.".
Each bucket gains a re-key step that moves "i/<id>" to "i.<id>" and
deletes the old key. Deleting is safe under the shadow rule: a node still
writing slashes also reads them, so what it loses is a mirror, and a
missing mirror falls back to the key it mirrors rather than erroring.
A pair of tests asserts both halves — that "i.*" delivers a record write
and that "i/*" does not — so the reasoning that chose the separator is
checked rather than trusted.
* test: benchmark what the node blob costs per state change
The case for splitting the blob is that one instance changing state
rewrites every instance on the node, and that cost grows with N while the
record's does not. That was asserted rather than measured.
Two benchmarks measure both sides on a record sized from env19, which
averages around 10KB terminated: marshalling the blob at N of 1, 10, 50
and 100, and marshalling one record. Each reports the bytes it produced
alongside the time, so the amplification is readable from the output
rather than needing a second calculation.
There is no assertion, because there is no threshold worth failing a
build on. Benchmarks compile in CI and run only when asked, which is what
a measurement wants — a test that cannot fail would run every pass and
gate nothing.
* fix: cover the embedder cold-start window with retry and Retry-After
Guardrail and knowledge-base embed calls hard-failed with a raw 503
whenever the embedding endpoint's connection was refused mid-relaunch,
instead of riding out the short window until it accepted connections.
embedBatch now retries a bounded number of times on connection-refused
or connection-reset, and any resulting 503 carries a Retry-After header
so callers back off instead of hammering the endpoint. A background
warm-up probe gates the embed path for statically pinned endpoints,
logging once when the endpoint first becomes reachable. Guardrail and
embedder infra logs now carry service/action labels for easier triage.
Fail-closed guardrail semantics are unchanged: an unavailable embedder
still denies unverified content, this only changes how quickly it
recovers once the endpoint is reachable again.
* fix: re-verify a persisted READY endpoint's liveness before resolving it
DynamicEndpointResolver's StateReady branch trusted the KV record's base
URL unconditionally. After an awsgw restart the record can still say
READY while the co-resident engine on that VM is mid-rebind, so the
resolver handed callers an address nothing was listening on yet.
describeAndEnsure now re-probes a READY member's own readiness route
before trusting it, short-TTL cached so a warm model still resolves at
no extra cost per request. A member that is not answering falls through
to the same not-ready path as a launch already in flight, so guardrail
and embed callers get the existing retryable answer instead of dialing
a dead address. This applies at the resolver seam, so it covers every
co-served member, not just the embedder.
The spinifex-ui reverse proxy copied only the decoded URL path and dropped
RawPath, so a percent-encoded path segment (a ':' in a bedrock modelId) was
re-encoded before reaching the gateway. The gateway then canonicalised a
different path than the one the browser SDK signed, and SigV4 rejected the
request with a signature mismatch — surfacing as an unparseable '<?xml'
deserialization error in the console Playground.
Carry RawPath through the rewrite so the escaped path arrives byte-identical.
Also route the auth, cluster-unavailable, and throttle error emitters through a
shared jsonErrorService helper so the bedrock family receives AWS JSON 1.1
errors the SDK can parse, and stamp the signed service onto the request before
verification so a rejection renders in the client's format.
* feat: add Ochre console Playground page
A Converse chat surface with model picker, system prompt, inference params,
optional guardrail, per-turn tokens, raw request/response JSON, and copy-as-CLI.
The transport lives in one module so a later ConverseStream swap edits a single
file; the message list appends and mutates the last turn in place. A cold model
degrades to a retryable warming-up turn instead of an error.
* feat: pin the playground composer, send on Enter, render markdown
The chat composer sat below the message list in page flow, so it slid down as
the conversation grew and the reader had to scroll to reach it. Bound the
chat + composer to a fixed-height column: the message pane scrolls internally
and autoscrolls to the newest turn while the composer stays pinned.
Enter now sends (Shift+Enter inserts a newline), and assistant replies render
markdown — lists, inline code, code blocks, and links — instead of raw text.
Adds the accessors for the per-resource instance record key space, over both
buckets that carry instance records. Nothing calls them yet.
The prefix is disjoint from the three it replaces: List and DeletePrefix match
on a plain string prefix rather than on NATS subject tokens, so i/ shares a
bucket with instance., node. and terminated. without colliding with any of them.
mutateVM and listVMs become generic so the record accessors reuse them rather
than carrying a second copy.
InstanceRecord is resource.Object[InstanceSpec, InstanceStatus], the persisted
form of an instance under the per-resource key space the re-key introduces.
Nothing reads or writes it yet: the boundary is settled and tested before the
keys move onto it.
The wrappers around the EBS and ENI request slices are dropped on the way in
and rebuilt on the way out, so the record does not carry a mutex.
Instance and Reservation are carried whole. They are SDK types returned
verbatim by DescribeInstances and hold both halves in one object, so the
boundary does not hold there and the comment says so.
Object[Spec, Status] separates what an operator asked for from what a node
observed, with the identity and lifecycle bookkeeping common to both.
Generation is bumped only through MutateSpec so NeedsSync means something, and
ObserveGeneration records the generation the node acted on rather than the
current one, so a spec change landing mid-reconcile is not marked done.
No ARN or tag helpers: the arn package already builds ARNs per service, and the
tag helpers in utils already cover the map every record carries.
StateReason was cleared by ModifyInstanceAttribute, so the API wrote a field
the node owns. Move the clear onto the launch path.
ExtraHostfwd held the requested guest port and the bound host port in one map,
with zero standing in for both 'not bound yet' and a real value. Split it into
HostfwdPorts and HostfwdPortMap, rebuilt per launch, and fold legacy records on
decode.
The command an instance was last handled with was stored on the record, but
only stop_instance and delete_instance were ever read back. Replace them with
DesiredState and DeletionTimestamp, which say what the read sites actually
mean, and drop the other twelve booleans.
Legacy records are folded on decode so an operator-stopped instance is not
relaunched on upgrade.
A staged self-host model is granted to every account without a per-account
grant, making the admin import the only v1 gate. Wrap the access resolver so
every gate opens together; the admin catalog reports self-host state
weights-first. Per-account grants remain as a dormant seam.
Operator-only catalog page listing every model with modality, streaming,
price, VRAM/instance, co-serve group, and availability with a fix-affordance
hint per reason. Reads the ListOchreCatalog admin action, mirroring the
existing admin-service query pattern.
Expose per-model availability and reason on an operator-only ListOchreCatalog
action for the console, keeping the tenant ListFoundationModels omission
intact. Remove the unbuilt provider-direct entry from the v1 catalog.
The server queues the API audit advisory after the reply the caller
waits on, so subscribing once the bucket exists could still catch the
create the first call made. Watch from the start and account for it.
The handler records its request point after it replies, so asserting as
soon as Request returns could read the metric before it landed. Poll for
the outcome the way the sibling drain test does.
The 40ms apply-params timeout was shared by the tests that expect the
agent to answer and those that expect it not to, so a parallel coverage
run could miss the reply. Only the unreachable-agent cases keep the
short budget.
* feat: semantic topic detection for guardrails via embedding similarity
Guardrail topic policy previously only blocked on an exact word-boundary
match against a denied topic's name or listed examples, so a paraphrase or
synonym for a blocked topic sailed straight through. This adds an
embedding-similarity check: each denied topic's name, definition, and
examples are embedded and cached, incoming text is embedded and compared by
max cosine similarity against those phrase vectors, and a topic blocks when
that similarity clears a conservative threshold. The literal exact-match
check still runs first and always blocks on its own. If no embedder is
configured, or an embed call fails, the check falls back to the existing
literal match alone rather than ever failing open.
* fix: fail closed when guardrail topic embedding is unavailable
The topic policy's semantic check silently fell back to its literal-only
matcher whenever the embedder was configured but temporarily unreachable
(for example during model cold-start), so a paraphrase of a denied topic
could pass through unblocked while the embedding endpoint was down. Now that
case is distinguished from an unconfigured embedder: if literal matching
already caught the request, that block still wins outright. Otherwise, when
the embedder is wired but the embed call itself fails, the guardrail check
returns an error and the caller aborts the request rather than letting
unverified content through.
* feat: converge DNS on change rather than on a timer
The drift backstop now watches the four stores dnsDesiredSet reads: instance state, ELBv2 load balancers, and the per-account EKS and RDS buckets. Only the trigger changes; the leader election, computeBatch and the publish path are untouched.
The interval becomes the backstop for drift the buckets cannot report, such as a record edited directly in the zone object, rather than the path by which a lifecycle event reaches DNS.
* test: cover the DNS watch wiring and reconciler run path
ListUsers, ListRoles, ListInstanceProfiles and ListGroupsForUser built their
results from a nil slice when the filtering loop matched nothing. xmlutil
drops a nil slice entirely rather than emitting an empty element, so each
response omitted a member its output shape marks required.
Each operation's empty-bucket shortcut already returned an empty non-nil
slice, so only the filtering path was affected. Start those slices empty and
non-nil too.
ListAttachedGroupPolicies, ListAttachedRolePolicies, ListPolicies and
ListAttachedUserPolicies declare no required output members, so their nil
slices are legal and are left alone.
DescribeImages projected BootMode through aws.String unconditionally, so an
AMI recording none emitted "", which is outside the model enum. RegisterImage
takes BootMode as an optional input, so this is the normal path for an AMI
registered without one, not stale data. Set the member only when present.
ListAccessKeys and ListGroups built their results from a nil slice when
nothing matched. xmlutil drops a nil slice entirely rather than emitting an
empty element, so the response lost a member the model marks required. Start
both slices empty and non-nil.
Integration conformance drops from 18 violations to 15; the remainder are the
EC2 AccessDenied error-code findings, which are unrelated.
The conformance subtest asserting the clamp ran against a provider holding
no volumes, so len(resp.Volumes) <= MaxListResults held whatever PageSize
did. Removing the clamp branch left the suite green.
Add a unit test on PageSize covering above the cap, at it, below it, and at
or below zero, and give the conformance subtests a fixture so the claim they
keep, that an over-cap request returns a page rather than an error, can fail.
buildInstanceStatus read vm.EC2StateCodes directly, so a StateError instance
surfaced the non-AWS name "error" while DescribeInstances returned "stopped"
for the same instance.
Ubuntu 22.04 was accepted by detect_os but the runtime apt list has never
resolved on jammy, which ships dhcpcd5 rather than dhcpcd-base. No CI cell
or E2E matrix installs on it, and the install docs already claim Ubuntu
26.04 and Debian 13 only, so drop the claim instead of carrying a
jammy-conditional package branch for an untested suite.
Watch is used zero times in spinifex against 33 ticker loops. This adds the primitive on Bucket, which already owns the memoised handle, plus the watch/debounce/resync/re-establish loop, so each converted caller changes only its trigger.
Watches are established before the first pass: a change landing between the two would otherwise wait for the next resync, since the pass that would have seen it ran before the watch existed.
dependabot[bot]