A refused launch and an idle GPU could coexist indefinitely: capacity said no
while a device sat under a model nobody was calling. Ensure now makes room once
before giving up, choosing the least recently used endpoint that is safe to
stop — READY, nothing in flight, not pinned, past minResidency, and on this
node, since GPU capacity is node-local and stopping an endpoint elsewhere frees
nothing here.
The eviction goes through Delete, so the device is released by the same
DRAINING -> terminate -> release path an operator delete takes rather than
being taken from underneath a running process. Nothing evictable returns the
original ModelNotReadyException unchanged: capacity contention breaches no
quota and must not claim to.
minResidency defaults to 5m so two models cannot thrash each other with a full
cold boot per call. Beyond that the deployment is under-provisioned, and the
eviction rate is what says so.
Also adds coldStartWait (OCHRE_COLD_START_WAIT), default 0. At zero the
fail-fast contract is unchanged and costs no extra describe. Above zero the
resolver polls inside its existing singleflight, so a burst of cold callers
shares one wait and one launch. The default stays 0 because the measurement
behind it has not changed: cold start is minutes, and no client retry policy
spans that.
A launched model held its GPU until an operator deleted the endpoint. On a
single-GPU deployment that made every other model in the catalog permanently
unreachable, with no path to recovery that did not involve a human.
The reaper scrapes each READY endpoint's vLLM Prometheus /metrics and reads
idleness from the serving process itself rather than from gateway-reported
activity, which keeps the daemon the single writer of endpoint state and works
unchanged with any number of gateways. Idle means both scheduler queues are
empty and the completed-request counter has not advanced since the previous
sweep: the counter clause is what catches a request that started and finished
entirely between two sweeps.
Two rules bound the reclaim. An endpoint is never reaped within idleTTL of
readyAt, so it cannot be taken back between reaching READY and the request that
launched it arriving. A failed scrape is unknown, never idle; five consecutive
failures terminate and relaunch, so a wedged VM is not mistaken for a quiet one
and left holding a device.
Observations are persisted on the record rather than kept in the reaper's
memory: a leader change must not reset the idle clock, and the eviction
decision that follows in the next change runs on whichever replica the queue
group picked. A sweep writes only when what it saw actually changed.
The inference path resolved self-host models from a static pinned map and
never asked for a launch, so on any deployment that had not pinned an endpoint
by hand a staged model was advertised, accepted by the access gate, and then
rejected by every invoke forever with no VM ever started.
Add a resolver over the endpoint registry, satisfying the existing
EndpointResolver interface so the provider and router layers are unchanged.
It lives in handlers/bedrock rather than gateway/bedrock because that package
cannot import this one; restating the subjects and payloads gateway-side would
be two definitions of one wire contract, free to drift.
Resolution order keeps the escape hatch first: OCHRE_VLLM_ENDPOINTS wins, then
a short-TTL cache of READY base URLs, then describe. STARTING and DRAINING do
not resolve and do not re-request, since a launch or a teardown is already in
flight. ABSENT requests one and reports not-resolved, which the invoke paths
turn into a retryable ModelNotReadyException — the cold-start contract.
Concurrent cold calls for one model collapse through singleflight, so a burst
costs one describe and one ensure rather than a pair per request. Describe and
ensure errors propagate rather than being flattened into a cold start, so a
dead control plane stays distinguishable from a model that is merely starting,
and the daemon's own refusals keep their AWS codes.
finalizeTerminated treated a refused terminated -> terminated transition as a
failure, but reaching the state we were driving to is convergence: the VM GC
had reconciled the same instance while terminateCleanup was running.
The caller pays for that error. Observed on wattle: an Ochre endpoint delete
wrote DRAINING, lost this race, and returned before the DRAINING -> ABSENT
write, so the VM was gone and the GPU released while the record stayed
DRAINING. Delete only accepted READY, so no retry could clear it and the model
was unreachable until the KV was edited by hand. ELB, EKS and RDS system
instances race the same GC.
So converge in the vm layer when the instance did reach terminated, keeping
the refusal for one that did not, and let Delete resume from DRAINING: a
teardown that failed partway has to be retryable.
* feat(bedrock): gate Ochre models behind per-account access grants
Self-host models were invokable by any authenticated account: tieredCatalog
returned them unconditionally and no runtime router checked the caller.
GetFoundationModel also described models the caller could not use.
Add a bedrock-model-access JetStream KV bucket holding presence-only grants,
resolved through a new AccessResolver. Access is deny-by-default: a gateway
with no store configured falls back to DenyAllAccessResolver, so an
unconfigured deployment serves nothing rather than everything. The system
account bypasses grants, matching how quotas exempt it.
All four runtime routers share one grantedCatalogEntry gate so they cannot
drift on who may invoke what. List and describe agree: an ungranted model is
absent from ListFoundationModels and reported as ResourceNotFoundException by
GetFoundationModel, so the error cannot confirm a model exists.
Administration lands on two surfaces: Grant/Revoke/ListModelAccess actions on
the spinifex gateway service for the UI, and `spx admin ochre access` for
operators and provisioning, which opens the bucket over NATS directly rather
than requiring a SigV4 client the CLI does not have.
* feat(bedrock): seed the admin account's model grants at gateway start
Deny-by-default left a fresh install with a catalog nobody could see, and the
first cut filled that gap from ansible — which is a dev tool, so any install
that never ran it came up unusable.
Seed from the product instead: on start awsgw grants the platform admin
account the full catalog, once, behind a marker key in the same bucket. The
marker makes it a bootstrap step rather than a policy: tenant accounts still
begin with no access, and an operator who revokes a grant does not have it
restored by the next restart. Grants are written before the marker so a crash
midway repeats the idempotent grants rather than leaving a half-seeded
account, and the marker is a conditional create so only the first node to
reach it need win.
Seeding is best-effort: a KV failure logs and leaves the gateway serving,
and the next start retries because the marker is written only on success.
* test(bedrock): cover the model-access CLI and gateway admin actions
Route the access CLI runners through the same loadConfigAndConnectFn /
ochreExit seam the weights runners use, so grant, revoke and list can be
driven against an embedded JetStream server, and add gateway tests for the
grant admin actions including the non-admin denial.
* feat(images): add a vllm-serving AMI as a stacked mkosi profile
LaunchServingVM resolves its AMI by the tags spinifex:managed-by=bedrock and
spinifex:bedrock-role=vllm-serving. No image carried either, so every Ensure
failed at resolveServingAMI. The only GPU image ships the NVIDIA driver and
Docker but no Python, no vLLM and no serving unit, so tagging it would have
moved the failure from admission to first boot.
Add a vllm profile that stacks onto gpu-nvidia and owns the whole guest side
of the launcher contract: the runtime, vllm-serve.service, the weights mount
and the tags. vLLM is baked into a uv-managed venv rather than pulled as a
container at boot — a runtime docker pull of vllm/vllm-openai is known to
stall this platform's write plane, and mkosi's unprivileged build has no
daemon to bake an image from anyway.
Also bring images/mkosi.profiles/ under make test-images. Its scripts have no
.sh suffix, so the existing name glob skipped exactly the files worth linting;
sources are found by shebang instead.
* fix(images): make the vllm postinst survive the build sandbox
Two failures found by actually running the build.
uv installs to /usr/local/bin, which mkosi's postinst PATH need not carry, so
`command -v uv` reported a good install as missing. Call it by absolute path,
and check it the same way.
`vllm --version` initialises device config and dies with "Failed to infer
device type" on a builder with no GPU, so it could never have verified
anything. Assert the pinned version through package metadata instead, which
also catches a version mismatch rather than merely a broken entrypoint.
* fix(images): pin the serving AMI datasource and mirror cloud-init to the console
A serving VM boots through LaunchSystemInstance, so it comes up dual-NIC with a
fw_cfg netcfg blob and takes its user-data solely from IMDS — the same shape as
the EKS control plane. That path carries a cloud-init datasource pin; the vllm
profile never got one, and the base config deliberately leaves datasource_list
unpinned, so the serving image was the only system VM booting without it.
Reuse the EKS pin rather than copying it. The 99- prefix is load-bearing:
read_conf_d() applies cloud.cfg.d in REVERSE and mergemanydict() is
first-source-wins, so a 90- name loses to cloud-init's own 90_dpkg.cfg ('_'
sorts above '-') and the pin is discarded in silence. The postinst asserts the
MERGED datasource_list via cloud-init's own read_conf_with_confd rather than
checking the file exists, because it existed at 90- too and did nothing there.
Also mirror every cloud-init stage to the serial console. A serving VM is
headless, and a guest that wedges before handover never surrenders
/var/log/cloud-init-output.log, so the console get-console-output reads is the
only channel an operator has.
The images_test.go case covers the two tags resolveServingAMI filters on; it
matches on those alone and never on name, so a typo hides the image entirely.
* fix(images): make the dhcpcd data-NIC lease genuinely one-shot
dhcp_oneshot() promised to configure the link and exit "leaving no daemon
to fight cloud-init", but the dhcpcd branch passed only -q, which is quiet
rather than one-shot. Every Ubuntu image takes that branch, so a dhcpcd
daemon survived on the data NIC holding the control socket.
cloud-init's Ec2 datasource then runs its own ephemeral dhcpcd on the same
NIC. Instead of spawning, dhcpcd forwards the invocation to the resident
daemon as a control command and exits 0, and --dumplease returns the
resident daemon's lease, so discovery looks successful. Cleanup then waits
for a pid file at /run/dhcpcd/<iface>-4.pid that no process ever wrote,
spinning its full 30000 x 10ms budget: init-local stalls for exactly 300s
on every system-instance boot.
Adding -1 exits once the interface is configured and -p keeps the address
and routes across that exit. Both are needed; -1 alone tears the link back
down. The test now models the real clients — the stub stays resident unless
given the flag that makes its client exit, and the dhcpcd branch is forced
by dropping udhcpc from PATH rather than left to whether the host happens
to have /usr/share/udhcpc/default.script.
* fix(images): serve without a CUDA toolkit and stop vllm-serve crash-looping
vLLM picks FlashInfer for top-k/top-p sampling, and FlashInfer JIT-compiles
that kernel on first use. The serving image ships the driver and the
container toolkit but no nvcc, so the sampler warmup died with "Could not
find nvcc and default cuda_home='/usr/local/cuda' doesn't exist" minutes
into a GPU cold start, after the model had already loaded.
VLLM_USE_FLASHINFER_SAMPLER=0 selects vLLM's native top-k/top-p sampler,
which is the same operation with no CUDA toolchain at runtime. Everything
else on the serving path is already precompiled in the wheels: the attention
backend resolves to FLASH_ATTN and CUDA graph capture completes, both
without nvcc. Baking FlashInfer's AOT modules is the alternative if the
fused kernel is ever wanted, and it keeps the toolchain in the builder
rather than the appliance.
The unit had Restart=on-failure with no bound, so an unfixable start failure
restarted forever and buried its own first traceback under identical copies
on the serial console — the only view an operator has of a headless guest.
StartLimitIntervalSec/StartLimitBurst make it report instead.
The postinst asserts the installed unit carries the sampler setting, for the
same reason the datasource pin is asserted there: the alternative is finding
out at runtime, on a GPU, five minutes in.
Also corrects the shared datasource pin's header, which still claimed it
installs as 90-spinifex-ec2-imds.cfg. It is installed at 99- by the vllm
profile precisely because a 90- name loses the reverse-sorted, first-source-
wins cloud.cfg.d merge to cloud-init's own 90_dpkg.cfg.
* fix(bedrock): raise the serving-endpoint readiness bound to 15 minutes
The 5 minute bound was chosen before any cold start had been measured, and
its own comment asked to be corrected once a model actually ran on wattle.
It now has: Llama 3.2 1B on an RTX A1000 reached "Application startup
complete" 257.6s after launch, leaving ~42s of margin on the smallest model
this platform can serve, with a cold torch.compile accounting for 26.3s of
it. A bigger card and a bigger model are unmeasured.
The two failure modes are not symmetric. Too low tears down an endpoint that
was about to come up and reports it as a failed launch; too high only makes
a genuinely stuck launch take longer to give up. So the bound is set well
clear of the single data point rather than close to it.
The CLI's own wait moves to 20 minutes to stay above it. The daemon starts
its clock when the VM is launched, up to a minute after the CLI starts its
own, so leaving both at 15 would expire the client first and report a bare
"timed out" in place of the daemon's abort, which carries the reason.
* fix: give the Ochre daemon a routed port in the Bedrock system VPC
The readiness probe and every InvokeModel call dial a serving VM's
system-VPC address from inside the daemon, but that subnet lives in OVN
on br-int and the host has no route to it, so a healthy VM never reached
READY.
Each daemon now owns an ENI in the system VPC's private subnet, bound to
an OVS internal port on br-int by iface-id, exactly as ovn-controller
binds a guest tap. The ENI address is installed at the subnet's prefix
length, which is what supplies the connected route. BaseURL is unchanged:
the address it already carried is now reachable.
Describe-or-create keyed on a per-node description, since ENIs carry no
tag filter, so a restart or a second endpoint reuses one address instead
of leaking one per launch. The port is ensured before anything else the
launch creates: a VM the daemon cannot dial holds a GPU for the whole
readiness window before unwinding.
Both actions carried no scope, so their policy check evaluated against "*".
Matching is pattern-to-value, so a statement whose Resource is a specific ARN
never matches "*" — a Deny fencing an instance blocked the scoped
ModifyDBInstance and DeleteDBInstance but not a snapshot of the same instance.
The principal could snapshot the fenced instance, restore it under a name the
Deny does not cover, set a master password of their choosing and read the data.
Each is scoped to its source: the instance for a create, the snapshot for a
restore, which breaks the chain at its first step. Both identifiers are required
on their action and ResourceARN already falls back to "*" when absent, so this
narrows where the identifier is present and is a no-op where it is not.
Evaluating both ARNs per action, for AWS parity, is deliberately left out: it
turns the scope into a slice across all 13 scoped entries and touches the shared
dispatcher call site.
* feat: add spx admin ochre endpoint ensure/describe/list/delete
The daemon subscribes all four bedrock.endpoint.* subjects, but nothing in
production calls their NATS client, so the serving-endpoint lifecycle could
not be exercised at all in a running system. This gives an operator a direct
way to drive it.
ensure --wait polls describe until READY and reports the elapsed time, since
the daemon replies STARTING as soon as it claims the model and the launch
outlives the request. That is the measured cold start.
Also corrects the weights stage tokenizer rule in COMMANDS.md, which still
described both tokenizer files as required rather than either.
* test: cover the ochre endpoint Run wrappers' connect and exit paths
* fix: treat an empty endpoint bucket as no endpoints, not an error
ListEndpoints was the only kvutil.Keys caller not handling ErrNoKeysFound,
so 'ochre endpoint list' failed on a cluster that had never launched one.
weights stage created the snapshot at the viperblock layer only, writing
the block checkpoint and config.json but not the metadata.json that
CreateVolume resolves a SnapshotId through. The endpoint launcher clones
the weights volume via ec2.CreateVolume, so every launch failed with
InvalidSnapshot.NotFound against a snapshot that was present on disk.
The GPU model table was only ever used to report VRAM, so wrong or missing
entries were cosmetic. Bedrock's capacity check is the first code that rejects
on VRAM, which turns every latent data error into an admission decision.
Correct seven entries verified against pci.ids — most seriously 1002:7448,
labelled Instinct MI300X at 192GB when it is a Radeon Pro W7900 at 48GB, a
4x over-report that admits a model which then OOMs at weight load. Record the
smallest capacity where one PCI ID spans several cards, since under-reporting
refuses an endpoint an operator can see and override while over-reporting
fails later and looks like a vLLM bug.
Widen the table so common datacenter, workstation and consumer cards resolve
without an operator-supplied override, and apply GPUModelOverrides to the pool
devices as well as the advertised model, so the override reaches the code that
actually gates. An override with no memory_mib is left alone rather than
wiping discovered VRAM, since the config cannot distinguish it from a
deliberate zero.
MemoryMiB 0 stays the "undiscoverable" sentinel, but checkCapacity no longer
counts it as a device with no VRAM: an otherwise-free device now reports which
card is unknown and how to fix it.
Josh Siviour