mariadbd has no time_zone startup option — the name only exists for SET — so
the resolved parameter group aborted the server while it parsed defaults, before
it opened the datadir. No pidfile was ever written and the reconciler failed
every instance at the 20-minute engine-health deadline.
ParameterSpec gains an option-file spelling, used only when rendering the
include. time_zone is the one setting that needs it, as default_time_zone, which
resolves @@global.time_zone to the same value. The customer-facing name is
unchanged: the API still reports time_zone and a live apply still issues
SET GLOBAL time_zone. Names read back out of a generated file return to the
catalog's namespace first, so a startup spelling cannot read as an unknown name
and report a pending restart nothing could clear.
setup.sh now feeds the catalog's resolved defaults to the image's own mariadbd
at build time, so a name the server refuses fails the build rather than every
instance built from it.
The rds suite gains three MariaDB tests. TestMariaDBInstance carries one
instance from create through a TLS connection, a password rotation, an attached
parameter group's dynamic and static edits, a stop/start and a class change,
re-asserting both overrides after every transition: an instance that came back
on catalog defaults is healthy, available and running a configuration its owner
did not choose, which no status reports. TestMariaDBSnapshotRestore proves the
snapshot carries MariaDB's own engine, version and port and that the restore
holds the rows as of the moment it was taken. TestMariaDBValidation boots
nothing and covers the unpinned versions, mysql, the cross-family group in both
directions and the implicit default.mariadb11.8.
The master password carries ' and \ at create and at rotation, the two
characters ValidateMasterUserPassword permits and the client cannot quote for
us, so rds-init's escaping and the agent's ALTER USER are under test against a
live server rather than a fixture.
The shared client VM installs mariadb-client alongside psql and opens 3306 as
well as 5432. The MariaDB client helpers avoid --ssl, which the 11.x client
deprecates: its notice would land in the same stream as the query's output,
which the guest exec folds together.
COMMANDS.md now covers both engines in one section, with a per-engine table of
pins, ports, families and limits, the mysql-is-not-an-alias note, the enumerated
MariaDB master grant, the InnoDB-only snapshot guarantee for non-transactional
tables, and the family-must-match rule. The Terraform quickstart stays
explicitly PostgreSQL and names what a MariaDB run would change.
Registers MariaDB in the engine registry now that its image, bootstrap and
guest implementation exist, so a create naming it resolves the MariaDB AMI
rather than failing validation. default.mariadb11.8 joins default.postgres18
in DescribeDBParameterGroups, as AWS reports one default group per engine.
mysql stays rejected and is deliberately not aliased onto MariaDB.
The crash-consistent snapshot event becomes engine-aware. Its old wording
promised recovery from a write-ahead log, which is false for a MariaDB
database holding Aria or MyISAM tables: those have no redo log and can be
torn by a snapshot taken without a quiesce, which is exactly the case that
event reports.
Adds mariadbEngine alongside the PostgreSQL one: a three-stage probe that
separates a live-but-unreachable server from an absent one, ALTER USER password
rotation on stdin behind a session logging guard, a dynamic SET GLOBAL apply
with rollback of the installed file, file-comparison pending-restart against a
serving copy, last-known-good restore, service stop and restart, and BACKUP
STAGE quiesce on a held session with a bounded lock_wait_timeout.
The parameter-file machinery, service actions, quiesce hold bookkeeping and
client session are extracted so both engines share them, and a contract suite
runs the same cases against both through the interfaces. MariaDB is not
registered in the agent's engine table yet, so construction still refuses.
Adds scripts/images/rds-mariadb/: the manifest with every mariadb package pinned =~11.8, setup.sh, the rds-datadir and rds-agent services, and the static include drop-in that points MariaDB at the generated configuration on the data volume. The AMI carries engine=mariadb, engine-version=11.8 and the data-volume contract tag, so resolveEngineAMI can select it.
The include is named zz- rather than 00-: MariaDB walks an include directory in byte order and takes the last occurrence of a setting, so a digit prefix would sort ahead of Alpine's packaged mariadb-server.cnf and let a distribution default beat a platform-owned one. setup.sh asserts the resulting order against the real directory at build time. The include target is baked empty as well, because MariaDB treats an includedir it cannot open as a fatal defaults-parsing error and a boot with no data volume would otherwise lose the client alongside the server.
The packaged drop-in also sets skip-networking, which would leave the endpoint resolving and the health probe passing while nothing on the customer ENI could connect. setup.sh disables it at the source and asserts no configuration file still does.
The mariadb service is deliberately not enabled: the packaged service only warns about an empty datadir rather than refusing, so an engine enabled before the bootstrap exists would launch mysqld_safe against an unbootstrapped volume.
rds-datadir moves to scripts/images/common/ rather than being copied. Volume selection, format authorization and mount are already engine-neutral; the mount point and owning user are the only per-engine facts and now come from each preset's own service. The script fails closed when either is unset, so no image can inherit another engine's layout by omission.
MariaDB 11.8 as an Engine with port 3306, its own reserved accounts,
an 80-character username limit and a 64-character database name limit,
alongside a curated parameter catalog of the same shape and size as the
PostgreSQL one. The size-derived defaults follow RDS's own formulas:
innodb_buffer_pool_size at three quarters of class memory and
max_connections on {DBInstanceClassMemory/12582880}, both with class
ceilings. Platform-owned settings are absent from the catalog rather
than present-and-unmodifiable.
It is deliberately not registered in engines. The image and guest
implementation land first, so no create can resolve an AMI nothing
builds.
The generic catalog machinery had two PostgreSQL assumptions baked into
it that a second engine cannot share. Boolean spellings are now carried
on the spec, because MariaDB refuses yes and no where PostgreSQL accepts
them, and the per-parameter string rules move onto a Validate hook, so
the timezone and datestyle checks live with PostgreSQL and sql_mode and
time_zone live with MariaDB.
Another engine's datadir mounts cleanly and reads as uninitialised to
rds-init, which would initialise beside the customer's data and serve an
empty database that passes every health probe while the real data sits
unreferenced.
rds-init now writes the engine to ${DATA_MOUNT}/.spinifex-rds/engine
before the datadir is inspected, backfills it on attach for volumes
predating it, and refuses a stamp that disagrees.
Define the composite engine interface and a newEngine() factory keyed by a per-engine layout table, so the agent's defaults, health probe and command implementation all come from the engine the image bakes rather than from PostgreSQL-shaped constants. Rename PGBin/PGData/PGUser to EngineBinDir/EngineDataDir/EngineUser, delete PGIsReady, and give engineProbe an injected state function with the pg_isready exit-code mapping moved into the PostgreSQL implementation.
setup.sh now stamps /etc/spinifex-rds/engine and buildAgentUserData emits RDS_ENGINE. The agent compares the two after registering and refuses to run when they disagree, reporting unhealthy before the bootstrap handoff is written so rds-init never unblocks and no datadir is touched.
DBName was passed through unvalidated, so a name such as my-db was accepted
at CreateDBInstance and only quoted into safety by psql inside the guest.
Engine gains a validateDBName rule and ValidateDBName, and rds-init re-checks
the same rule in load_handoff before initdb, where a failure would clear the
datadir and spend the one-shot master password.
PostgreSQL caps the name at NAMEDATALEN-1.
Every path that binds a parameter group to an instance — create, modify,
snapshot restore, the deferred apply and group propagation — bottoms out
in resolveGroupParameters, which resolved against PostgreSQL whatever
the instance ran. It now takes the instance's engine and rejects a group
whose family is not that engine's with InvalidParameterCombination,
naming the group, its family and the family it needs. Each caller passes
the engine it already holds.
ModifyDBParameterGroup and DescribeDBParameters resolve the engine from
the target group's own family before validating or listing values, so a
parameter of one engine cannot be stored into another engine's group and
fail later against whichever instance next attached it.
The unrecognised-family rejection lists every supported family instead
of asserting postgres18 is the only one. An omitted family still takes
postgres18.
Completes the control-plane half of the engine seam ahead of a second
engine. Engine gains maxUsernameLen, its own parameter table and its own
cross-parameter checks, and the catalog accessors become methods on it,
so no path resolves parameters without naming an engine.
PostgreSQL's table, size-derived formulas and combination checks move to
engine_postgres.go. engines is also indexed by parameter-group family,
for the callers that hold only a family string. EnginePostgres() is
deleted: the guest's reserved-username re-check now resolves the engine
through LookupEngine once at agent startup.
No behavior change.
* feat: add guardrail CRUD store and ARN support to bedrock gateway
Stage 1 of Ochre Guardrails: control-plane store for CreateGuardrail,
CreateGuardrailVersion, GetGuardrail, ListGuardrails, UpdateGuardrail,
and DeleteGuardrail, backed by a JetStream KV bucket keyed per account.
Records keep the full policy config so later stages can build the
filter engine and inference enforcement on top without re-deriving it.
* feat: add deterministic guardrail filter engine and ApplyGuardrail op
Adds the pure wordPolicy/sensitiveInformationPolicy filter (regex-detectable
PII plus a starter profanity list), scaffolds content/topic/contextualGrounding
as NONE, and exposes it all through the bedrock-runtime ApplyGuardrail route.
* feat: enforce guardrails on Converse and ConverseStream
Wire GuardrailConfig enforcement into Router.Converse and
ConverseStream via a shared enforceGuardrail helper (resolve, load,
apply the deterministic filter engine). INPUT blocks skip the
backend entirely; OUTPUT blocks or ANONYMIZE redactions rewrite the
completion before it reaches the caller. ConverseStream buffers
delta text and assesses it at block/message-stop boundaries rather
than streaming unassessed model output. Both self-host and
Anthropic-direct backends are covered automatically since the
enforcement sits above Router's per-backend dispatch. Requests
without a GuardrailConfig are unaffected. An unresolvable or foreign
guardrail fails closed with ResourceNotFoundException.
* feat: enforce guardrails on InvokeModel and InvokeModelWithResponseStream
Adds the Invoke half of Ochre guardrails (plan §5), reusing the Stage 3
enforceGuardrail helper and Stage 2 filter engine unchanged. Guardrail
selection comes from the X-Amzn-Bedrock-Guardrail{Identifier,Version}
headers. Since InvokeModel bodies are family-native/opaque at the
Router, text is extracted/reinjected per family (llama self-host,
anthropic) in the new guardrail_invoke.go. INPUT is checked before the
backend is ever called (fail closed on an unresolved guardrail, no
backend call); OUTPUT is checked after, handling block/anonymize.
InvokeModelWithResponseStream buffers the inner stream and assesses
once at end-of-stream via a guardrailInvokeStreamSource decorator,
mirroring Stage 3's guardrailStreamSource.
* fix: route ApplyGuardrail through bedrock-runtime and use guardrail-specific RNF text
The service discriminator only promoted /model/... paths from bedrock to
bedrock-runtime, so ApplyGuardrail's singular /guardrail/.../apply route fell
through to the control-plane handler and was rejected. Extend the
discriminator to also promote /guardrail/..., which can't collide with the
plural /guardrails control-plane CRUD routes.
Also give guardrail-not-found paths their own ResourceNotFoundException
message instead of the generic "could not resolve the foundation model"
wording shared with model-lookup failures.
CI builds spinifex in a workspace over viperblock and predastore only, so
northstar resolves from the proxy. Its v1.15.0 log sanitiser predates the
attribute.Value migration and does not compile against otel/log v0.21.0.
The log API moved bodies and attributes onto attribute.Value in v0.21.0, so the
otelslog bridge and the sibling log sanitisers have to move in the same window.
No spinifex source change is needed; the workspace resolves the migrated
predastore and viperblock.
* feat: parameterise bedrock endpoint create/delete by account and pinned
Widens EnsureEndpointInput/DescribeEndpointInput/DeleteEndpointInput
with AccountID and (Ensure-only) Pinned, and keys the daemon's
Ensure/Describe/Delete on the resolved account instead of the
hardcoded GlobalAccountID. An empty AccountID still resolves to
GlobalAccountID and Pinned defaults to false, so every existing
caller (resolver, reaper, evictForCapacity, the NATS gateway client)
is unchanged. This is stage 1 of provisioned throughput: it reuses
the existing launch/capacity/readiness machinery for a dedicated,
account-scoped, pinned endpoint without adding a parallel surface.
* feat(bedrock): add provisioned-throughput CRUD and PT ARN
Implements CreateProvisionedModelThroughput, Get, List, Update, and
Delete over a new bedrock-provisioned KV store, plus the
provisioned-model ARN format/parse. Create rejects provider-hosted
models and requests a pinned, account-scoped endpoint from the daemon;
Get derives status live from the pinned endpoint's state; Update
honours name changes only and rejects a served-model swap; Delete
tears down the pinned endpoint before removing the record.
EndpointProvisioner is declared in gateway_bedrock with primitive-typed
methods (handlers_bedrock already imports gateway_bedrock, so it can't
be imported back for its EndpointService interface directly); a small
adapter in handlers_bedrock bridges the two, always setting Pinned:true
and the caller's real account on every Ensure.
Does not yet wire a PT ARN into the Converse/InvokeModel inference
path (next stage).
* feat(bedrock): accept a provisioned-throughput ARN on Converse/InvokeModel
Translates a PT ARN before resolve: the gateway inference entry parses it,
loads the bedrock-provisioned record, and resolves against the pinned
endpoint for the commitment's own account rather than the GlobalAccountID
shorthand every bare modelId uses. Adds EndpointResolver.EndpointForAccount
(gateway-side interface, implemented by DynamicEndpointResolver) so the seam
stays import-cycle-safe, and threads it through vllm.go/invoke_llama.go's
resolve call sites. A foreign-account or unknown PT ARN reads as
ResourceNotFoundException, same as any other unresolvable model identifier.
* fix(bedrock): accept a provisioned-throughput ARN on streaming inference
ConverseStream and InvokeModelWithResponseStream built their router/adapter
with a nil ProvisionedStore and the GlobalAccountID-shorthand provider, so a
PT ARN was silently never translated on the streaming path even though the
non-stream Converse/InvokeModel siblings already handled it. Thread the
gateway's provisioned store through both entry points and give
Router.ConverseStream / InvokeStreamRouter.InvokeModelWithResponseStream the
same translate-before-resolve treatment Converse/InvokeModel already have.
* fix(bedrock): surface pinned endpoints in the operator CLI
Service.List only ever enumerated the GlobalAccountID key prefix, and
'endpoint describe' always looked up GlobalAccountID too, so a pinned,
account-scoped endpoint held a GPU while reading ABSENT and missing from
list. List now scans every account's records, and describe gains an
optional --account flag (default empty, preserving GlobalAccountID
resolution) to look up a specific account's endpoint. list/describe both
render the endpoint's account and pinned status.
* fix(ochre): self-heal weights stage when the recorded snapshot is gone
runStageWeights's idempotency check trusted a matching source URI without
verifying the snapshot it names still exists, so a host rebuild or volume
GC that lost the snapshot left stage permanently short-circuited on a
useless KV record until an operator ran 'weights remove' first. Inject a
weightsSnapshotChecker, verify liveness before the no-op return, and fall
through to re-materialise when the snapshot is gone. 'weights list' uses
the same checker to mark dangling entries so an operator can see which
staged models can actually serve without waiting for endpoint launch to
fail.
* docs: note weights stage self-heal and list STATUS column
Adds resolveServingAMI test cases: single match, multiple matches
selected deterministically by newest CreationDate independent of
describe-order, zero matches, and nil/empty ImageId entries skipped.
A binary-swap deploy (make deploy or a manual spx copy) ships a new binary
whose host-config prerequisites — the privileged helper scripts and their
sudoers grant, placed only by setup.sh / setup-ovn.sh — it does not ship.
The node looks healthy and the gap surfaces later as a cryptic launch
failure or a silent security regression.
Add a preflight package that checks the host against what this build
expects, comparing asset content, not existence, so a stale-but-present
asset is caught too. The managed helpers install from quoted heredocs, so
their on-disk bytes equal the heredoc body verbatim; the binary carries
the canonical sha256s and compares.
- spinifex/preflight: CheckHost / CheckHostAt, content hash of the two
managed helpers and a NOPASSWD grant-line check for the sysctl helper.
- Generated canonicalHashes plus a drift guard that re-extracts the
heredocs and fails if the committed manifest no longer matches.
- Three surfaces: spx admin preflight, a make deploy gate
(SKIP_ASSET_PREFLIGHT=1 to bypass), and a non-fatal daemon boot warning.
Release reported "nothing was claimed" as an error, and the terminated-instance
teardown reaper persisted that as the gpu teardown mark. The mark could then
never reach done, so the record was never purged and the GC re-drove it every
two minutes for as long as it existed, failing identically each time because
the condition being retried was that the work had already been done.
Stop-then-terminate is ordinary: the stop releases the GPU and the later
terminate finds nothing left to release. Two instances on wattle had been in
this state for days, at 2251 and 640 retries, logging an ERROR naming the
node's only GPU and falsely claiming it might need a manual rebind.
Release now returns a matchable ErrNoGPUClaimed and the teardown cleaner reads
it as success, which is what the reaper already documents its cleaners to do.
The sentinel stays an error rather than becoming nil so the launch-rollback
path can still tell that its claim/release pair is out of step, and so a real
vfio rebind failure - which clears Available, the health signal the Bedrock
capacity gate reads - stays distinct from the benign case.
checkCapacity read only PoolEntry.Available, but that field tracks device
health: Release clears it when a vfio rebind fails, so a device that cannot
be handed back is not offered again. Occupancy is the separate InstanceID
field, which PoolEntry documents as empty when free, and gpu.Manager.Claim
tests both before taking a device.
Reading Available alone let a GPU already held by a running serving VM admit.
Ensure never took its capacity refusal, so evictForCapacity was never called
and LRU eviction never fired: on a single-GPU node the launch instead
proceeded to STARTING and failed later with InsufficientInstanceCapacity,
with no eviction attempted and no reason logged.
The existing table had a case named 'already claimed' that set Available to
false, which is not the shape Claim leaves behind. That encoded the wrong
model and is why the gap went unnoticed, so it is renamed to what it really
covers and the real claimed-device shapes are added alongside.
Also stop daemonGPUSnapshotter capturing d.gpuManager by value.
applyGPUConfig swaps that pointer on a passthrough toggle and re-points
daemonGPUClaimer but not the snapshotter, which would leave bedrock reading
an orphaned pool that sees no claims and admits everything.
The catalog carried exactly one self-host entry, so two models could never
contend for a GPU: an Ensure for the model already running returns the
existing record before reaching the capacity check. LRU eviction was
therefore built and unit-tested but unable to fire in production.
Llama 3.2 3B Instruct is a real AWS-published Bedrock model ID, so it keeps
the drop-in-SDK promise, and it is a genuine quality tier above the 1B model
rather than a fixture added only to unblock a test.
Staying inside the Meta family is deliberate. InvokeRouter dispatches
self-host requests to llamaInvokeAdapter on tier alone, so a model of another
family would be served with Llama's native request and response schema
instead of being refused. A new test pins that assumption to the catalog so
it fails loudly rather than silently mis-serving.
Sizing for the RTX A1000: 3.21B params at bf16 is roughly 6124 MiB of weights
against 8188 MiB of VRAM, so --enforce-eager skips CUDA graph capture to free
several hundred MiB and --max-model-len is half the 1B entry's. fp8 is not an
option on this GPU: GA107 is compute capability 8.6, and fp8 needs sm_89.
These figures are estimates to be corrected against a live reading.
Weights gating is now measurably per-entry rather than all-or-nothing, which
only became observable once two self-host entries could disagree.
Three gaps found while verifying idle reclaim on live hardware.
The reclaim log line passed a time.Duration, which slog renders as an int64
nanosecond count in JSON: the line read idleFor:135000000000 rather than 2m15s,
in journalctl and in the metrics sink alike. It now logs an integer second
count under a unit-suffixed field, which is both readable and still
aggregatable — a string would have cost the latter. The same shape appears at
around ten other sites across the fleet and is tracked separately.
The endpoint command's help still told operators the gateway could not request
endpoints itself, which stopped being true when dynamic resolution landed. It
now describes the lifecycle that actually runs, and says what the manual
commands are still for.
describe showed nothing the idle sweep observed, so an operator asking why an
endpoint was or was not reclaimed had to read the KV. It now reports in-flight
count, last activity, idle duration, pinned status and consecutive scrape
failures for a READY endpoint, and nothing for any other state, since no other
state is swept.
LastActive moves onto EndpointRecord so the idle clock, the LRU eviction order
and this output all read one definition rather than three.
A refused launch and an idle GPU could coexist indefinitely: capacity said no
while a device sat under a model nobody was calling. Ensure now makes room once
before giving up, choosing the least recently used endpoint that is safe to
stop — READY, nothing in flight, not pinned, past minResidency, and on this
node, since GPU capacity is node-local and stopping an endpoint elsewhere frees
nothing here.
The eviction goes through Delete, so the device is released by the same
DRAINING -> terminate -> release path an operator delete takes rather than
being taken from underneath a running process. Nothing evictable returns the
original ModelNotReadyException unchanged: capacity contention breaches no
quota and must not claim to.
minResidency defaults to 5m so two models cannot thrash each other with a full
cold boot per call. Beyond that the deployment is under-provisioned, and the
eviction rate is what says so.
Also adds coldStartWait (OCHRE_COLD_START_WAIT), default 0. At zero the
fail-fast contract is unchanged and costs no extra describe. Above zero the
resolver polls inside its existing singleflight, so a burst of cold callers
shares one wait and one launch. The default stays 0 because the measurement
behind it has not changed: cold start is minutes, and no client retry policy
spans that.
A launched model held its GPU until an operator deleted the endpoint. On a
single-GPU deployment that made every other model in the catalog permanently
unreachable, with no path to recovery that did not involve a human.
The reaper scrapes each READY endpoint's vLLM Prometheus /metrics and reads
idleness from the serving process itself rather than from gateway-reported
activity, which keeps the daemon the single writer of endpoint state and works
unchanged with any number of gateways. Idle means both scheduler queues are
empty and the completed-request counter has not advanced since the previous
sweep: the counter clause is what catches a request that started and finished
entirely between two sweeps.
Two rules bound the reclaim. An endpoint is never reaped within idleTTL of
readyAt, so it cannot be taken back between reaching READY and the request that
launched it arriving. A failed scrape is unknown, never idle; five consecutive
failures terminate and relaunch, so a wedged VM is not mistaken for a quiet one
and left holding a device.
Observations are persisted on the record rather than kept in the reaper's
memory: a leader change must not reset the idle clock, and the eviction
decision that follows in the next change runs on whichever replica the queue
group picked. A sweep writes only when what it saw actually changed.
The inference path resolved self-host models from a static pinned map and
never asked for a launch, so on any deployment that had not pinned an endpoint
by hand a staged model was advertised, accepted by the access gate, and then
rejected by every invoke forever with no VM ever started.
Add a resolver over the endpoint registry, satisfying the existing
EndpointResolver interface so the provider and router layers are unchanged.
It lives in handlers/bedrock rather than gateway/bedrock because that package
cannot import this one; restating the subjects and payloads gateway-side would
be two definitions of one wire contract, free to drift.
Resolution order keeps the escape hatch first: OCHRE_VLLM_ENDPOINTS wins, then
a short-TTL cache of READY base URLs, then describe. STARTING and DRAINING do
not resolve and do not re-request, since a launch or a teardown is already in
flight. ABSENT requests one and reports not-resolved, which the invoke paths
turn into a retryable ModelNotReadyException — the cold-start contract.
Concurrent cold calls for one model collapse through singleflight, so a burst
costs one describe and one ensure rather than a pair per request. Describe and
ensure errors propagate rather than being flattened into a cold start, so a
dead control plane stays distinguishable from a model that is merely starting,
and the daemon's own refusals keep their AWS codes.
finalizeTerminated treated a refused terminated -> terminated transition as a
failure, but reaching the state we were driving to is convergence: the VM GC
had reconciled the same instance while terminateCleanup was running.
The caller pays for that error. Observed on wattle: an Ochre endpoint delete
wrote DRAINING, lost this race, and returned before the DRAINING -> ABSENT
write, so the VM was gone and the GPU released while the record stayed
DRAINING. Delete only accepted READY, so no retry could clear it and the model
was unreachable until the KV was edited by hand. ELB, EKS and RDS system
instances race the same GC.
So converge in the vm layer when the instance did reach terminated, keeping
the refusal for one that did not, and let Delete resume from DRAINING: a
teardown that failed partway has to be retryable.
* feat(bedrock): gate Ochre models behind per-account access grants
Self-host models were invokable by any authenticated account: tieredCatalog
returned them unconditionally and no runtime router checked the caller.
GetFoundationModel also described models the caller could not use.
Add a bedrock-model-access JetStream KV bucket holding presence-only grants,
resolved through a new AccessResolver. Access is deny-by-default: a gateway
with no store configured falls back to DenyAllAccessResolver, so an
unconfigured deployment serves nothing rather than everything. The system
account bypasses grants, matching how quotas exempt it.
All four runtime routers share one grantedCatalogEntry gate so they cannot
drift on who may invoke what. List and describe agree: an ungranted model is
absent from ListFoundationModels and reported as ResourceNotFoundException by
GetFoundationModel, so the error cannot confirm a model exists.
Administration lands on two surfaces: Grant/Revoke/ListModelAccess actions on
the spinifex gateway service for the UI, and `spx admin ochre access` for
operators and provisioning, which opens the bucket over NATS directly rather
than requiring a SigV4 client the CLI does not have.
* feat(bedrock): seed the admin account's model grants at gateway start
Deny-by-default left a fresh install with a catalog nobody could see, and the
first cut filled that gap from ansible — which is a dev tool, so any install
that never ran it came up unusable.
Seed from the product instead: on start awsgw grants the platform admin
account the full catalog, once, behind a marker key in the same bucket. The
marker makes it a bootstrap step rather than a policy: tenant accounts still
begin with no access, and an operator who revokes a grant does not have it
restored by the next restart. Grants are written before the marker so a crash
midway repeats the idempotent grants rather than leaving a half-seeded
account, and the marker is a conditional create so only the first node to
reach it need win.
Seeding is best-effort: a KV failure logs and leaves the gateway serving,
and the next start retries because the marker is written only on success.
* test(bedrock): cover the model-access CLI and gateway admin actions
Route the access CLI runners through the same loadConfigAndConnectFn /
ochreExit seam the weights runners use, so grant, revoke and list can be
driven against an embedded JetStream server, and add gateway tests for the
grant admin actions including the non-admin denial.
* feat(images): add a vllm-serving AMI as a stacked mkosi profile
LaunchServingVM resolves its AMI by the tags spinifex:managed-by=bedrock and
spinifex:bedrock-role=vllm-serving. No image carried either, so every Ensure
failed at resolveServingAMI. The only GPU image ships the NVIDIA driver and
Docker but no Python, no vLLM and no serving unit, so tagging it would have
moved the failure from admission to first boot.
Add a vllm profile that stacks onto gpu-nvidia and owns the whole guest side
of the launcher contract: the runtime, vllm-serve.service, the weights mount
and the tags. vLLM is baked into a uv-managed venv rather than pulled as a
container at boot — a runtime docker pull of vllm/vllm-openai is known to
stall this platform's write plane, and mkosi's unprivileged build has no
daemon to bake an image from anyway.
Also bring images/mkosi.profiles/ under make test-images. Its scripts have no
.sh suffix, so the existing name glob skipped exactly the files worth linting;
sources are found by shebang instead.
* fix(images): make the vllm postinst survive the build sandbox
Two failures found by actually running the build.
uv installs to /usr/local/bin, which mkosi's postinst PATH need not carry, so
`command -v uv` reported a good install as missing. Call it by absolute path,
and check it the same way.
`vllm --version` initialises device config and dies with "Failed to infer
device type" on a builder with no GPU, so it could never have verified
anything. Assert the pinned version through package metadata instead, which
also catches a version mismatch rather than merely a broken entrypoint.
* fix(images): pin the serving AMI datasource and mirror cloud-init to the console
A serving VM boots through LaunchSystemInstance, so it comes up dual-NIC with a
fw_cfg netcfg blob and takes its user-data solely from IMDS — the same shape as
the EKS control plane. That path carries a cloud-init datasource pin; the vllm
profile never got one, and the base config deliberately leaves datasource_list
unpinned, so the serving image was the only system VM booting without it.
Reuse the EKS pin rather than copying it. The 99- prefix is load-bearing:
read_conf_d() applies cloud.cfg.d in REVERSE and mergemanydict() is
first-source-wins, so a 90- name loses to cloud-init's own 90_dpkg.cfg ('_'
sorts above '-') and the pin is discarded in silence. The postinst asserts the
MERGED datasource_list via cloud-init's own read_conf_with_confd rather than
checking the file exists, because it existed at 90- too and did nothing there.
Also mirror every cloud-init stage to the serial console. A serving VM is
headless, and a guest that wedges before handover never surrenders
/var/log/cloud-init-output.log, so the console get-console-output reads is the
only channel an operator has.
The images_test.go case covers the two tags resolveServingAMI filters on; it
matches on those alone and never on name, so a typo hides the image entirely.
* fix(images): make the dhcpcd data-NIC lease genuinely one-shot
dhcp_oneshot() promised to configure the link and exit "leaving no daemon
to fight cloud-init", but the dhcpcd branch passed only -q, which is quiet
rather than one-shot. Every Ubuntu image takes that branch, so a dhcpcd
daemon survived on the data NIC holding the control socket.
cloud-init's Ec2 datasource then runs its own ephemeral dhcpcd on the same
NIC. Instead of spawning, dhcpcd forwards the invocation to the resident
daemon as a control command and exits 0, and --dumplease returns the
resident daemon's lease, so discovery looks successful. Cleanup then waits
for a pid file at /run/dhcpcd/<iface>-4.pid that no process ever wrote,
spinning its full 30000 x 10ms budget: init-local stalls for exactly 300s
on every system-instance boot.
Adding -1 exits once the interface is configured and -p keeps the address
and routes across that exit. Both are needed; -1 alone tears the link back
down. The test now models the real clients — the stub stays resident unless
given the flag that makes its client exit, and the dhcpcd branch is forced
by dropping udhcpc from PATH rather than left to whether the host happens
to have /usr/share/udhcpc/default.script.
* fix(images): serve without a CUDA toolkit and stop vllm-serve crash-looping
vLLM picks FlashInfer for top-k/top-p sampling, and FlashInfer JIT-compiles
that kernel on first use. The serving image ships the driver and the
container toolkit but no nvcc, so the sampler warmup died with "Could not
find nvcc and default cuda_home='/usr/local/cuda' doesn't exist" minutes
into a GPU cold start, after the model had already loaded.
VLLM_USE_FLASHINFER_SAMPLER=0 selects vLLM's native top-k/top-p sampler,
which is the same operation with no CUDA toolchain at runtime. Everything
else on the serving path is already precompiled in the wheels: the attention
backend resolves to FLASH_ATTN and CUDA graph capture completes, both
without nvcc. Baking FlashInfer's AOT modules is the alternative if the
fused kernel is ever wanted, and it keeps the toolchain in the builder
rather than the appliance.
The unit had Restart=on-failure with no bound, so an unfixable start failure
restarted forever and buried its own first traceback under identical copies
on the serial console — the only view an operator has of a headless guest.
StartLimitIntervalSec/StartLimitBurst make it report instead.
The postinst asserts the installed unit carries the sampler setting, for the
same reason the datasource pin is asserted there: the alternative is finding
out at runtime, on a GPU, five minutes in.
Also corrects the shared datasource pin's header, which still claimed it
installs as 90-spinifex-ec2-imds.cfg. It is installed at 99- by the vllm
profile precisely because a 90- name loses the reverse-sorted, first-source-
wins cloud.cfg.d merge to cloud-init's own 90_dpkg.cfg.
* fix(bedrock): raise the serving-endpoint readiness bound to 15 minutes
The 5 minute bound was chosen before any cold start had been measured, and
its own comment asked to be corrected once a model actually ran on wattle.
It now has: Llama 3.2 1B on an RTX A1000 reached "Application startup
complete" 257.6s after launch, leaving ~42s of margin on the smallest model
this platform can serve, with a cold torch.compile accounting for 26.3s of
it. A bigger card and a bigger model are unmeasured.
The two failure modes are not symmetric. Too low tears down an endpoint that
was about to come up and reports it as a failed launch; too high only makes
a genuinely stuck launch take longer to give up. So the bound is set well
clear of the single data point rather than close to it.
The CLI's own wait moves to 20 minutes to stay above it. The daemon starts
its clock when the VM is launched, up to a minute after the CLI starts its
own, so leaving both at 15 would expire the client first and report a bare
"timed out" in place of the daemon's abort, which carries the reason.
* fix: give the Ochre daemon a routed port in the Bedrock system VPC
The readiness probe and every InvokeModel call dial a serving VM's
system-VPC address from inside the daemon, but that subnet lives in OVN
on br-int and the host has no route to it, so a healthy VM never reached
READY.
Each daemon now owns an ENI in the system VPC's private subnet, bound to
an OVS internal port on br-int by iface-id, exactly as ovn-controller
binds a guest tap. The ENI address is installed at the subnet's prefix
length, which is what supplies the connected route. BaseURL is unchanged:
the address it already carried is now reachable.
Describe-or-create keyed on a per-node description, since ENIs carry no
tag filter, so a restart or a second endpoint reuses one address instead
of leaking one per launch. The port is ensured before anything else the
launch creates: a VM the daemon cannot dial holds a GPU for the whole
readiness window before unwinding.
Julian Sommer