* fix: build the EC2 DNS desired set from the instance records so stale records prune
* fix: keep DNS records for stopped instances in the reconcile desired set
The desired set filtered on Running, but the lifecycle publish deliberately
does not withdraw a stopped instance's records: stop and start retain the
address, so they retain the name. With EC2 pruning enabled, absence from the
desired set became a deletion, so a stopped instance would have lost both its
names until the next publish.
Both sides now read one predicate, InstanceRetainsRecords, which replaces
needsDNSWithdrawal and lives next to the names it governs. Records carrying a
DeletionTimestamp are skipped as well, so a re-upsert cannot race the
withdrawal the terminate path has already published.
* refactor: wake the vpcd drift loop on intent changes, not a fixed 5-minute ticker
DriftLoop was the last ticker-driven reconcile loop in the event-driven
reconciliation work. It runs on reconciler.Run now, watching the eight KV
buckets LoadIntentFromKV reads, with DriftInterval kept as the resync
backstop and the existing backoff ladder returned as the pass deadline.
Repair latency for a dropped vpc.* event drops from a uniform 0-300s to
the debounce plus a 5s floor. The floor is new and deliberate: a pass
loads eight buckets and scans the OVN NB DB, so without it a launch storm
would turn a five-minute scan into a continuous one.
The buckets are enumerated rather than fixed so watching never creates
one: the EC2 handlers own them and set their history depth.
The bootstrap ReconcileApplyOnly pass stays the startup scan. The loop's
own startup pass does no work and returns the seed deadline instead, so a
clean boot still waits a full interval and a failed one still requeues at
5s.
* fix: keep the intent watchers up when a bucket lookup blips
A per-bucket resolve failure was skipped, which drops the bucket out of
the enumerated set and has the reconciler tear its watcher down until the
next resync. Only a genuine ErrBucketNotFound is a skip now; anything else
fails the enumeration, which is what tells the caller to leave the
watchers it already has in place.
* fix: measure the drift pass floor from the end of a pass, not its start
Stamping before the scan means a pass slower than the floor leaves no gap
at all behind it. The floor is meant to be the minimum time between
passes, so it is stamped once the scan returns.
* test: wait for the intent watcher before the write it must catch
* fix: follow ListObjectsV2 continuation tokens on every production caller
Six ListObjectsV2 call sites (DeleteSnapshot, DescribeSnapshots, EKS
etcd-snapshot selection, DescribeTags, and both key-pair listings) issued a
single request and never read IsTruncated or NextContinuationToken. This was
latent while predastore ignored max-keys and always answered IsTruncated:
false; predastore#117 pages listings at 1000 keys, turning a truncated
listing beyond that into a silently wrong answer rather than a short one.
DeleteSnapshot in particular could delete 1000 objects, report success, and
orphan the rest, and the EKS restore path could resolve to a stale snapshot
if the newest keys fell past the first page.
Add objectstore.ListAll, a shared helper that drains a listing to
exhaustion, accumulating Contents and CommonPrefixes across pages and
erroring instead of returning a partial result if the server claims
truncation with no token to resume from or never finishes within a bounded
page budget. All six call sites, plus the ebsfake test double with the same
shape, now use it. Bump the predastore pin to dev head ff6a1dc4 so a real
cluster picks up the server-side pagination fix this depends on.
* fix: use external test packages and exclude test fakes from diff coverage
test-package-check requires new test files to bind to the exported API so
internals can change without rewriting tests. Move list_test.go and
delete_snapshot_pagination_test.go to their _test packages, adding a small
export_test.go for the one unexported constant a test needs to assert
against. Simplify list_test.go's pagingStore to embed MemoryObjectStore
instead of hand-stubbing unused methods.
Exclude spinifex/testutil from diff-coverage.sh: those files are fakes and
harnesses driven only from other packages' _test.go files, so go test
records them as zero-covered under the package they live in however hard
another package's tests exercise them, which is not a real coverage gap.
* refactor: drive the ECS scheduler from the bus and deadlines, not four tickers
The scheduler ran four tickers — service reconcile every 10s, heartbeat reaper
every 30s, stopped-task sweep every 60s, instance-role converge every 5m — and
the 10s one rewrote every service record whether or not anything changed.
A KV watch is the wrong tool here: the account bucket takes a heartbeat write
per instance per interval and the key is one NATS token, so no filter can
exclude them. Instead the service reconcile is triggered from the task-state
subscription, which is already separate from the heartbeat one at the bus
layer, and the three timed jobs return their own deadlines.
The service write becomes conditional so an idle service is no longer churned,
and a mutex keeps the two loops from overlapping, which the four tickers got
from sharing one goroutine.
* fix: wake the ECS service reconcile from the path the agent actually uses
The agent stopped publishing to ecs.bus when it moved onto the gateway, so
nothing publishes those subjects any more: the scheduler's task-state
subscription is dead in production and an in-process trigger off it would
never have fired on a live cluster.
A task state report now arrives as SubmitTaskStateChange on whichever worker
serves the call, so that path publishes a payload-free wake on the cluster's
service-reconcile subject and the leader — usually another node — subscribes
to it.
The pass also names its own deadline rather than trusting the wake to arrive:
a service with a task still in flight asks to be revisited at the old 10s
cadence, and only a settled fleet asks for nothing.
listDocuments issued a single ListObjectsV2 and iterated result.Contents.
It never read IsTruncated or NextContinuationToken, so above the server's
page limit the listing returned a prefix of the truth and reported success.
Every caller of ListVolumes, ListVolumesStrict, ListAMIs and ListAMIsStrict
inherited it. The strict variants are the worse case: their whole premise
is that a document they did not see is not evidence of absence, and a
truncated page gives them exactly that with no error to notice.
Key collection now walks the prefix to exhaustion before any document is
fetched. A page that claims truncation without a token to resume from is an
error rather than a short answer, since it is the one truncation a client
cannot honour, and a page count bounds a server that keeps claiming there
is more so it fails instead of hanging.
Nil keys are dropped while collecting rather than carried as empty slots
through the fetch, which removes the empty flag from the result struct.
The loop polled every 30s for two things it was already being told about: the
bootstrap artifacts are keys in the account bucket a watch can see land, and the
control plane's health arrives over core NATS into a field the reconciler read
only on the next tick.
It now runs on reconciler.Run over a watch of the account bucket, plus a new
Config.Trigger fed by the state-report subscription, and returns a revisit
deadline for what elapsed time alone decides: the create timeout, and the instant
the newest report goes stale. A degraded cluster keeps the old interval rather
than restating the three recovery ladders' grace and backoff arithmetic in a
second place.
The trigger only fires on a report that changes health. The control plane
publishes on its own timer regardless, so forwarding every report would have been
the same 30s tick under another name.
applyPorts refuses a port whose SG policy cannot be programmed, but applyEIPs
still ran ensureGuestPortDatapath for every public-IP guest it skipped. GuestPortUp
reports false, not an error, for a port that does not exist, so each refused port
burned the full 45s deadline and ~9 ovn-controller recomputes before giving up.
* feat: give a reconcile pass its own revisit deadline, and convert the RDS loop
The reconciler assumed every loop was a pure function of the buckets it
watches. Quota and DNS are; RDS, EKS and ECS are not. They also wait on an
EC2 VM reaching running, an endpoint answering, and an agent falling silent,
none of which write to KV, so a watch never sees them.
Converting RDS on the watch alone would have traded traffic for detection
latency: an agent going silent writes nothing, so a dead database would have
been noticed on the 5 minute resync rather than within 15s of its heartbeat
expiring.
Reconcile and ReconcileKey now return a revisit deadline, clamped to the
resync. The RDS reconciler returns the earliest across the fleet: the old
interval for a transitional record, the instant the heartbeat expires for a
settled one, the resolve timeout for a snapshot stuck in creating, and
nothing at all for a region with no databases.
* fix: keep a deadline while a dark instance runs down its failure grace
revisitFor measured a settled instance against its heartbeat expiry, and
returned that difference directly. Once the beat expired the difference went
non-positive, which the loop reads as no deadline at all.
That is the window the detector lives in. Between the beat expiring and the
grace closing nothing writes to KV, so there was nothing left to schedule the
pass that does the failing, and a dead database would have waited for the
resync — slower than the ticker this replaced.
The deadline is now the beat expiry while it is still ahead, and whatever
remains of the failure grace once it is not. A failed instance asks for
nothing: it is terminal, and recovery arrives as a heartbeat, which is a write
the watch already delivers. Already due but still not failed is the degraded
reading, dark agent over a live VM, so it retries at the old interval rather
than immediately.
* feat: gate the Ochre console behind a runtime flag
Add OchreEnabled to the spinifexui service config and /api/config
body, defaulting to false so the console and every /bedrock/* route
stay hidden until a node explicitly turns it on.
* feat: wire the Ochre flag through to the spinifex-ui service unit
Add a spinifex-ui-ochre-enabled flag (SPINIFEX_UI_OCHRE_ENABLED, off
by default) so the console flag can actually be set on a deployed
node, and bump the service unit so the reconciler picks it up.
* feat: add knowledge base and data source create/delete to the console
Ochre's KB pages were read-only. Adds create + delete for knowledge
bases and their S3 data sources, mirroring the guardrail CRUD pattern.
* feat: note data source is added after KB creation
Users expect to pick a data source at create time; the create form now
says up front that S3 data sources are attached from the detail page
after the knowledge base exists, then ingested separately.
applyPorts dropped any security group whose OVN port group was missing and
carried on. When that removed the last one, the ENI's logical switch port was
created belonging to no port group, so no SG ACL applied — including the
priority 900/800 default-denies, which are attached per port group — and the
port was fully unrestricted until a later pass healed it.
Fail closed instead: a port whose policy cannot be programmed in full is
refused and reported unconverged, so the drift loop retries it once applySGs
catches up. The update path is skipped too, leaving a live guest on a stale
policy rather than stripping it bare.
EnsurePort and SetPortSecurityGroups get the same guard for an empty SG list.
Every ENI record written by CreateNetworkInterface has at least one SG, but the
field is omitempty with no migration, so a legacy row would otherwise yield an
unrestricted port.
The vCPU sweep ran on a 30s ticker and called the account-filtered
DescribeInstances fan-out once per account per pass, costing
passes x accounts x (nodes + 2) and measured at ~62,000 instance
describes a day, none of it user traffic and none of it tracking how
often anything actually changed.
The account is on the instance record and vCPUs derive from the instance
type, which is spec, so the whole sweep is one read of the record space.
The loop moves onto reconciler.Run: a change to one instance recomputes
that one account, and the resync sweeps every account as the backstop.
The completeness flag has to stay, for a different reason than the one it
was written for. nats.go hardcodes AllowDirect on KV buckets, so a get at
R3 may be served by a lagging replica, and ChargeLaunch increments the
counter while reconcile overwrites it absolutely. A scan that misses a
just-charged launch would lower the counter and lift the cap. The lower
is now written only while the counter still stands at the revision read
before the scan started, so a charge landing mid-scan wins and the scan
is dropped for the next pass.
Two behaviours change with the source. A failed sweep was a per-account
describe error the pass continued past; it is now one read covering every
account, so a failure moves no counter at all. And an account holding
nothing is absent from the scan rather than reporting empty, so the
caller's account list is what zeroes it.
The retype gate and the volume delete guard still use the fan-out; only
the sweep comes off it.
The loop watches KV buckets and runs a whole-set recompute when anything
in them changes, discarding the update it already holds. A caller whose
work is per resource can now set ReconcileKey and be handed the key that
changed, so a change to one instance costs one instance's work.
Reconcile stays required. A per-key queue only converges if something
behind it covers what the queue never learned: a key deleted while the
watch was down is enqueued by nothing, because nobody observes an
absence. Keeping the whole-set pass as the startup pass and the resync
backstop is what makes that real, and it makes the change additive — a
caller that sets only Reconcile behaves exactly as before.
KeyFor maps an update onto the caller's unit of work, which is not
always the KV key: fifty instances changing in one account are one
recompute. Reporting ok=false falls back to a whole-set pass, which is
the honest answer for a delete — a tombstone carries no value, so a
caller whose work key lives in the value cannot name it.
A dropped and re-established watch queues the whole set rather than any
key, since UpdatesOnly hides the gap. One worker drains the queue, and a
key taken off it is no longer pending, so a change arriving mid-reconcile
is queued again rather than folded into the pass already running.
* feat: decide instance set membership from the record, not the key prefix
* feat: cut instance records over to one key per instance
The record space becomes the only copy. Which set an instance belongs to
was structural before this — a stopped instance sat at instance.<id> and a
running one inside the node.<nodeID> blob, so it could not be wrong — and
is now a predicate over the record, decided in one place rather than at
each call site.
The old key spaces are frozen, not deleted. A migration that removed them
would make the crossing unrollbackable, and the node blob is the only
record of what a node was running that a build predating the cutover can
read. instance.<id> and terminated.<id> drain on the delete path.
The per-node key survives as a presence marker at its own prefix. Reads
need it because a node with no instances and a node the cluster has no
record of both scan empty, and only one of them may replace what the node
has locally. It carries no instances, so a state change no longer rewrites
the node's whole running set.
Claiming a stopped instance becomes a compare-and-set. It cannot wait for
a later change: with one key holding an instance for its whole life, a
claim that deletes is deleting the instance it is about to start. The
winner clears the stop intent and leaves LastNode alone, so an instance
whose launch never happens is still recoverable by the node that ran it.
A build predating the cutover now refuses to open a bucket stamped past
what it understands, rather than reading the keys it knows and writing
back a view assembled from half a key space.
* fix: carry node ownership out of the blob key during the cutover migration
The node.<id> key carried two facts in its name: that the node exists, and
which instances are its. The 4->5 migration moved the first and not the
second.
A blob member's own LastNode is empty, because the key was the owner, and so
is last_node on the record copied forward from it. After the cutover
ownership is read off the record, so an unowned record is one no node lists
- and with the presence marker seeded, LoadRunningState returns found=true
with an empty set, which restore adopts over the instances the node is
actually running.
The migration now stamps last_node from the blob's key onto every record in
it that has no owner, and leaves a record that already names one alone.
Found on a live cluster before crossing it. The test helpers built blob
members with LastNode already set, which production blobs never have; the
new tests seed them the way the bucket actually holds them.
* feat: add guardrail create, edit and delete to the console
Adds CRUD for Bedrock guardrails alongside the existing read-only view:
create and edit forms covering topic, content, word, PII, regex and
contextual-grounding policies, a delete confirmation flow, and a
create-version action, all built on the same real-SDK client pattern
already used for guardrail reads and runtime testing.
* fix: mount the guardrail edit form instead of no-opping the link
The guardrail detail route file dot-nested under the edit route
(detail as parent, edit as child), but the detail page renders no
outlet, so navigating to the edit URL matched the route without ever
mounting the edit form. Renaming the detail route file to an index
sibling of edit fixes the mount without changing any full path.
* feat: render predastore admin_port and consult /readyz in health probe
Spinifex now allocates predastore's admin listener port (8660, after the
existing gate/blob/meta bases) and renders admin_port into both the
single-node and multi-node predastore.toml templates. The daemon's health
probe consults /readyz over that port on the two weak paths (single-host
gate fallback and hosts running no meta node), falling back to the prior
behavior when no admin port is configured or the listener refuses.
* build: pin predastore to the release carrying Host.AdminPort
The admin port field this branch reads is new in predastore, so CI could not
compile against the pinned version: go.work hid the gap locally by resolving
predastore to the checkout instead of the module.
* feat: add admin Model Access console page
Lets an admin grant or revoke a provider-tier model for a tenant
account from the console, cross-referencing the model catalog with
that account's current grants via the existing signed-proxy actions.
* feat: replace free-text account id with a real account picker
Sources the Model Access account selector from /admin/ListAccounts
via a new signed JSON admin-API fetch helper, sorted by name and
excluding the global account. Falls back to a validated 12-digit
free-text account id when enumeration is unauthorized or empty, since
that call requires a long-lived IAM user in the super-admin account.
* feat: mirror each node's running set onto the i/<id> key space
* refactor: re-key instance records onto a dot separator
A reconciler Filter is a NATS subject filter, where "*" matches one
dot-delimited token. "i/<id>" is a single token, so "i/*" matches nothing
and a watch over the record space would have had to take the whole bucket
with ">" — waking on every key rather than on instance state, which is
worse than the "node.*" it replaces. "i.<id>" is two tokens, so "i.*" is
the filter the DNS reconcile can narrow to at the cutover.
The space stays disjoint from the three it replaces for the same reason
the slash did: List and DeletePrefix match a plain string prefix, and
"instance.i-1" does not begin "i.".
Each bucket gains a re-key step that moves "i/<id>" to "i.<id>" and
deletes the old key. Deleting is safe under the shadow rule: a node still
writing slashes also reads them, so what it loses is a mirror, and a
missing mirror falls back to the key it mirrors rather than erroring.
A pair of tests asserts both halves — that "i.*" delivers a record write
and that "i/*" does not — so the reasoning that chose the separator is
checked rather than trusted.
* test: benchmark what the node blob costs per state change
The case for splitting the blob is that one instance changing state
rewrites every instance on the node, and that cost grows with N while the
record's does not. That was asserted rather than measured.
Two benchmarks measure both sides on a record sized from env19, which
averages around 10KB terminated: marshalling the blob at N of 1, 10, 50
and 100, and marshalling one record. Each reports the bytes it produced
alongside the time, so the amplification is readable from the output
rather than needing a second calculation.
There is no assertion, because there is no threshold worth failing a
build on. Benchmarks compile in CI and run only when asked, which is what
a measurement wants — a test that cannot fail would run every pass and
gate nothing.
* fix: cover the embedder cold-start window with retry and Retry-After
Guardrail and knowledge-base embed calls hard-failed with a raw 503
whenever the embedding endpoint's connection was refused mid-relaunch,
instead of riding out the short window until it accepted connections.
embedBatch now retries a bounded number of times on connection-refused
or connection-reset, and any resulting 503 carries a Retry-After header
so callers back off instead of hammering the endpoint. A background
warm-up probe gates the embed path for statically pinned endpoints,
logging once when the endpoint first becomes reachable. Guardrail and
embedder infra logs now carry service/action labels for easier triage.
Fail-closed guardrail semantics are unchanged: an unavailable embedder
still denies unverified content, this only changes how quickly it
recovers once the endpoint is reachable again.
* fix: re-verify a persisted READY endpoint's liveness before resolving it
DynamicEndpointResolver's StateReady branch trusted the KV record's base
URL unconditionally. After an awsgw restart the record can still say
READY while the co-resident engine on that VM is mid-rebind, so the
resolver handed callers an address nothing was listening on yet.
describeAndEnsure now re-probes a READY member's own readiness route
before trusting it, short-TTL cached so a warm model still resolves at
no extra cost per request. A member that is not answering falls through
to the same not-ready path as a launch already in flight, so guardrail
and embed callers get the existing retryable answer instead of dialing
a dead address. This applies at the resolver seam, so it covers every
co-served member, not just the embedder.
The spinifex-ui reverse proxy copied only the decoded URL path and dropped
RawPath, so a percent-encoded path segment (a ':' in a bedrock modelId) was
re-encoded before reaching the gateway. The gateway then canonicalised a
different path than the one the browser SDK signed, and SigV4 rejected the
request with a signature mismatch — surfacing as an unparseable '<?xml'
deserialization error in the console Playground.
Carry RawPath through the rewrite so the escaped path arrives byte-identical.
Also route the auth, cluster-unavailable, and throttle error emitters through a
shared jsonErrorService helper so the bedrock family receives AWS JSON 1.1
errors the SDK can parse, and stamp the signed service onto the request before
verification so a rejection renders in the client's format.
* feat: add Ochre console Playground page
A Converse chat surface with model picker, system prompt, inference params,
optional guardrail, per-turn tokens, raw request/response JSON, and copy-as-CLI.
The transport lives in one module so a later ConverseStream swap edits a single
file; the message list appends and mutates the last turn in place. A cold model
degrades to a retryable warming-up turn instead of an error.
* feat: pin the playground composer, send on Enter, render markdown
The chat composer sat below the message list in page flow, so it slid down as
the conversation grew and the reader had to scroll to reach it. Bound the
chat + composer to a fixed-height column: the message pane scrolls internally
and autoscrolls to the newest turn while the composer stays pinned.
Enter now sends (Shift+Enter inserts a newline), and assistant replies render
markdown — lists, inline code, code blocks, and links — instead of raw text.
Adds the accessors for the per-resource instance record key space, over both
buckets that carry instance records. Nothing calls them yet.
The prefix is disjoint from the three it replaces: List and DeletePrefix match
on a plain string prefix rather than on NATS subject tokens, so i/ shares a
bucket with instance., node. and terminated. without colliding with any of them.
mutateVM and listVMs become generic so the record accessors reuse them rather
than carrying a second copy.
InstanceRecord is resource.Object[InstanceSpec, InstanceStatus], the persisted
form of an instance under the per-resource key space the re-key introduces.
Nothing reads or writes it yet: the boundary is settled and tested before the
keys move onto it.
The wrappers around the EBS and ENI request slices are dropped on the way in
and rebuilt on the way out, so the record does not carry a mutex.
Instance and Reservation are carried whole. They are SDK types returned
verbatim by DescribeInstances and hold both halves in one object, so the
boundary does not hold there and the comment says so.
Object[Spec, Status] separates what an operator asked for from what a node
observed, with the identity and lifecycle bookkeeping common to both.
Generation is bumped only through MutateSpec so NeedsSync means something, and
ObserveGeneration records the generation the node acted on rather than the
current one, so a spec change landing mid-reconcile is not marked done.
No ARN or tag helpers: the arn package already builds ARNs per service, and the
tag helpers in utils already cover the map every record carries.
StateReason was cleared by ModifyInstanceAttribute, so the API wrote a field
the node owns. Move the clear onto the launch path.
ExtraHostfwd held the requested guest port and the bound host port in one map,
with zero standing in for both 'not bound yet' and a real value. Split it into
HostfwdPorts and HostfwdPortMap, rebuilt per launch, and fold legacy records on
decode.
The command an instance was last handled with was stored on the record, but
only stop_instance and delete_instance were ever read back. Replace them with
DesiredState and DeletionTimestamp, which say what the read sites actually
mean, and drop the other twelve booleans.
Legacy records are folded on decode so an operator-stopped instance is not
relaunched on upgrade.
A staged self-host model is granted to every account without a per-account
grant, making the admin import the only v1 gate. Wrap the access resolver so
every gate opens together; the admin catalog reports self-host state
weights-first. Per-account grants remain as a dormant seam.
Operator-only catalog page listing every model with modality, streaming,
price, VRAM/instance, co-serve group, and availability with a fix-affordance
hint per reason. Reads the ListOchreCatalog admin action, mirroring the
existing admin-service query pattern.
Expose per-model availability and reason on an operator-only ListOchreCatalog
action for the console, keeping the tenant ListFoundationModels omission
intact. Remove the unbuilt provider-direct entry from the v1 catalog.
Julian Sommer