Both actions carried no scope, so their policy check evaluated against "*".
Matching is pattern-to-value, so a statement whose Resource is a specific ARN
never matches "*" — a Deny fencing an instance blocked the scoped
ModifyDBInstance and DeleteDBInstance but not a snapshot of the same instance.
The principal could snapshot the fenced instance, restore it under a name the
Deny does not cover, set a master password of their choosing and read the data.
Each is scoped to its source: the instance for a create, the snapshot for a
restore, which breaks the chain at its first step. Both identifiers are required
on their action and ResourceARN already falls back to "*" when absent, so this
narrows where the identifier is present and is a no-op where it is not.
Evaluating both ARNs per action, for AWS parity, is deliberately left out: it
turns the scope into a slice across all 13 scoped entries and touches the shared
dispatcher call site.
* feat: add spx admin ochre endpoint ensure/describe/list/delete
The daemon subscribes all four bedrock.endpoint.* subjects, but nothing in
production calls their NATS client, so the serving-endpoint lifecycle could
not be exercised at all in a running system. This gives an operator a direct
way to drive it.
ensure --wait polls describe until READY and reports the elapsed time, since
the daemon replies STARTING as soon as it claims the model and the launch
outlives the request. That is the measured cold start.
Also corrects the weights stage tokenizer rule in COMMANDS.md, which still
described both tokenizer files as required rather than either.
* test: cover the ochre endpoint Run wrappers' connect and exit paths
* fix: treat an empty endpoint bucket as no endpoints, not an error
ListEndpoints was the only kvutil.Keys caller not handling ErrNoKeysFound,
so 'ochre endpoint list' failed on a cluster that had never launched one.
weights stage created the snapshot at the viperblock layer only, writing
the block checkpoint and config.json but not the metadata.json that
CreateVolume resolves a SnapshotId through. The endpoint launcher clones
the weights volume via ec2.CreateVolume, so every launch failed with
InvalidSnapshot.NotFound against a snapshot that was present on disk.
The GPU model table was only ever used to report VRAM, so wrong or missing
entries were cosmetic. Bedrock's capacity check is the first code that rejects
on VRAM, which turns every latent data error into an admission decision.
Correct seven entries verified against pci.ids — most seriously 1002:7448,
labelled Instinct MI300X at 192GB when it is a Radeon Pro W7900 at 48GB, a
4x over-report that admits a model which then OOMs at weight load. Record the
smallest capacity where one PCI ID spans several cards, since under-reporting
refuses an endpoint an operator can see and override while over-reporting
fails later and looks like a vLLM bug.
Widen the table so common datacenter, workstation and consumer cards resolve
without an operator-supplied override, and apply GPUModelOverrides to the pool
devices as well as the advertised model, so the override reaches the code that
actually gates. An override with no memory_mib is left alone rather than
wiping discovered VRAM, since the config cannot distinguish it from a
deliberate zero.
MemoryMiB 0 stays the "undiscoverable" sentinel, but checkCapacity no longer
counts it as a device with no VRAM: an otherwise-free device now reports which
card is unknown and how to fix it.
* fix: make the predastore readiness gate a real readiness probe
checkPredastoreReady only did a TCP dial + close, so a listening socket
satisfied it well before predastore could serve TLS. In CI this passed
in 170 microseconds, 0.72s after predastore opened its listener, and
recovery started against a backend that wasn't serving yet — predastore
logged the probe itself as a TLS handshake EOF.
Replace the dial with an HTTPS GET against the configured predastore
endpoint on a package-level client (hoisted out of the 2s poll loop),
with a per-request 3s context timeout. Any HTTP response counts as
ready, including 403 from this deliberately unsigned request, which is
exactly what an unauthenticated S3 GET returns. Only a transport or TLS
failure means not-ready.
This narrows the race window and removes the spurious TLS-handshake-EOF
log lines predastore emits on every daemon start, but it does not close
the window on its own: predastore can serve TLS before its storage
backend answers reads correctly. The load-bearing fix is the viperblock
VBState read retry landing separately.
* style: trim the predastore readiness comments to three lines
GetPasswordData returned an empty string both when the guest had not emitted a
password yet and when it never could, because the launch key pair was ED25519
and cannot perform PKCS#1 v1.5 encryption. The two are indistinguishable, so a
caller in the second case polls forever with no signal.
An empty result on a Windows instance whose launch key is not RSA now returns
InvalidKeyPair.Type naming the cause and the fix. Every uncertainty keeps the
old empty result: a populated blob always wins, Linux is never judged, and an
unresolvable key is not treated as evidence of the wrong type.
get-password-data --priv-launch-key requires PEM, but ssh-keygen writes the
OpenSSH container by default, so even an explicitly-RSA key pair failed with
"Unable to decrypt password data using provided private key file" — which
reads like a padding problem rather than a format one. AWS returns PKCS#1 PEM
for RSA key pairs. OpenSSH reads that format for SSH either way.
The create-key-pair UI gains a key-type selector, defaulting to the rsa the
page already sent, with a hint that RSA is required for Windows password
retrieval.
The API default stays ed25519. Changing it is a separate decision.
ResourceNotFoundException is a shared wire code, so Bedrock aliases the EKS
error constant and inherited its message: a caller asking for a model was
told to list its EKS clusters. Register Bedrock overrides in the existing
errorLookupByService table, under both service keys since the gateway splits
bedrock from bedrock-runtime on the /model/ path prefix.
PutModelInvocationLoggingConfiguration also accepted embeddingDataDeliveryEnabled
and dropped it: the internal LoggingConfig had no such field, so the flag never
survived to Get. Carry it through both translation directions and fold it into
the bucket-required check, since embedding delivery with no bucket is as
undeliverable as text or image.
cloudbase-init has no IMDSv2 token support in any release, so a Windows
instance launched without an explicit MetadataOptions block 401s on every
metadata GET and never reaches its main plugin stage. Derive the launch
default from the resolved image's platform: Windows gets optional, every
other platform keeps required, and an explicit request value always wins.
* feat: add spx admin ochre weights stage/list/remove
Give operators a supported path to stage a self-hosted model's
weights: point at a predastore S3 prefix holding ordinary Hugging
Face files, and stage materialises them into a viperblock volume,
snapshots it, and records the source URI and snapshot ID in the
bedrock-weights KV bucket.
stage refuses before downloading anything if the model is not a
self-host catalog entry, or if the prefix is missing any required
file. Re-staging the same source is a no-op; a different source
replaces the entry and reports the previous snapshot for separate
reclamation.
WeightsStore gains GetWeights and a source-URI field on its record
to support this. LookupServingSpec (catalog.go) is reused unchanged
to validate --model-id.
* fix: unify ServingSpec after rebase onto dev's daemon-launcher spec
Rebasing onto dev pulled in mulga-vmfng.7.5's daemon-side ServingSpec
and LookupServingSpec, textually duplicating the ochre weights stage
CLI's own versions in catalog.go. Collapse both into one type carrying
ModelID and VLLMArgs, and one three-value LookupServingSpec so callers
distinguish an unknown model (found=false) from a provider-tier one
(selfHost=false). Update the daemon launcher's call site accordingly.
* test: cover ochre weights stage/list/remove command behaviour
Splits runOchreWeightsStage/List/Remove into testable decision functions
(runStageWeights, listWeightsOutput, removeWeights) separate from cobra and
NATS wiring, and makes mkfs.ext4 execution and the connect/exit steps
swappable so the full command flow can be driven against an embedded
JetStream server instead of live infrastructure. Also relaxes the tokenizer
file requirement to accept either tokenizer.json or tokenizer.model, since
canonical Llama 3.x repos ship only the former at the prefix root.
* fix: resolve mkfs.ext4 from sbin when it is absent from PATH
e2fsprogs installs mkfs.ext4 into sbin, which a non-login shell omits, so
weights staging failed on an unprivileged operator's otherwise fine host.
* fix: size the weights volume to a whole GiB so the tail block fits
viperblock rejects a write whose block overruns VolumeSize, so importing an
image of unaligned length failed on its last block. Rounding up also makes
the volume's byte size and its manifest SizeGiB describe the same volume.
Name the volume in the import error so a part-written import is traceable.
DescribeImages skipped any ami-* prefix whose config.json could not be
fetched, read, or parsed, logging each at DEBUG. At the daemon's default
INFO level the result was an empty Images list and a completion line
reading count=0, indistinguishable from a healthy empty registry.
Promote the three fault branches plus the body-close failure to WARN,
leave NoSuchKey at DEBUG since a prefix with no config is a legitimate
non-AMI directory, and report prefixes examined alongside images
returned so a mismatch is visible without raising the log level.
SetServiceOwnership chowned per-service directories to service users
without propagating a missing user, so spx admin init reported success
while leaving /etc/spinifex/northstar root-owned 0600. ChownRecursive
and SetServiceOwnership now return errors, failures are aggregated
across every service rather than stopping at the first, and
finalizeNodeSetup prints the unapplied paths instead of a success mark.
The per-tap datapath only matched IP, on the assumption that a guest reaches
169.254.169.254 through its default gateway. A guest that treats 169.254.0.0/16
as on-link resolves the address itself instead, and nothing on br-imds, br-int
or in OVN answered that ARP, so it could not reach IMDS at all.
Windows does exactly this, which left cloudbase-init timing out on every
metadata request and skipping the rest of its plugin stage.
Install a per-tap ARP responder for each captured address under the existing
cookie, replying with the gateway MAC the egress flow already presents.
The viperblock start command applies --s3-host, --s3-bucket and --s3-region over
spinifex.toml whenever viper reports a non-empty value. s3-bucket and s3-region
carried non-empty defaults, so the override fired on every start even though the
service unit passes neither flag nor environment variable, and the region read
from the config was replaced with ap-southeast-2.
Any cluster deployed to another region could not launch an instance: viperblockd
signed its S3 requests with ap-southeast-2, predastore verified them against its
own region and rejected them as AuthorizationHeaderMalformed, the volume mount
failed and RunInstances ended in launch_failed. The bare-metal e2e cell, which
installs with --region bare-metal, failed every instance-backed test this way.
Default all three to empty, matching s3-host, so an unset flag leaves the
configured value alone while an explicit flag or SPINIFEX_VIPERBLOCK_* still
wins. The predastore.toml template also pinned the main bucket to a literal
ap-southeast-2 while the top-level key and the northstar bucket both rendered
the configured region; it now renders it too.
Introduces handlers/bedrock, the daemon-side owner of vLLM serving VM
placement and lifecycle. State lives in a new JetStream KV bucket
(bedrock-endpoints) keyed {accountID}/{modelID}, cycling through
ABSENT -> STARTING -> READY -> DRAINING -> ABSENT. The daemon is the
sole KV writer; the gateway (or any other caller) requests transitions
over NATS via bedrock.endpoint.{ensure,describe,list,delete}, wired
into subscribeAll exactly as handlers/eks registers its own subjects.
Ensure is idempotent and single-flighted at two layers: a local
per-key mutex collapses concurrent same-process callers, and a
JetStream KV Create/CAS on the record's generation collapses races
across daemon replicas, so a burst of concurrent Ensure calls for one
model launches exactly one VM either way. Capacity is admission-checked
against gpu.Manager's free devices before a launch is claimed, so a
refusal never leaves a stuck STARTING record. Readiness is an explicit
poll of the serving VM's /v1/models endpoint until 200 or
startupTimeout, never a sleep.
Serving VMs boot via the system-instance path against a dedicated
system VPC, and follow the Phase-0 weights resolution: a snapshot
staged out-of-band is COW-cloned per endpoint into a data volume the
VM attaches post-boot, decoupled from the boot AMI.
Adds the .7.6 metering slice on top of .7.4's invocation stream: a
second durable consumer (ochre-usage-metering) aggregates per-account
token counts and external cost into a new bedrock-usage KV bucket,
keyed {accountID}/{modelID}/{period}. RequestID dedupe (an atomic
create-if-absent marker in bedrock-usage-dedupe) makes redelivery
count-neutral rather than double-billing a customer.
Prices are integer micro-USD per million tokens (D12): in-tree
catalog defaults with KV overrides in bedrock-prices, mirroring the
serving-spec shape from .7.3. Self-hosted models always resolve a
known zero; a model with neither an override nor a default resolves
Known=false, which the usage counter surfaces as a sticky CostUnknown
flag distinct from a genuine zero-cost period.
handlers_quota gains two Bedrock dimensions, both defaulting to
disabled independently of the block's existing Enabled toggle:
tokens-per-month (a check-against-history hard cap reading the
stream-fed counter, tolerant of a few seconds of lag) and
requests-per-minute (a local per-account token bucket enforced
immediately in-process, with a periodic KV sync for cross-node
observability only — enforcement never round-trips). Both are wired
into BedrockRuntime_Request ahead of every Converse/InvokeModel
variant.
* feat(ec2): serve the Windows administrator password via GetPasswordData
Adds the GetPasswordData EC2 API, mirroring the existing GetConsoleOutput
path: a per-instance NATS subscription, a daemon handler, and a gateway
shim registered in ec2Actions. The guest emits its encrypted administrator
password to the serial console at first boot; the daemon scans the whole
console log (not just the AWS 64KB tail, since the password would scroll
off a long-running instance) for the last <Password>...</Password> block
and returns its base64 ciphertext unchanged. Spinifex never handles key
material and never sees the plaintext password.
onInstanceUpHook's per-instance topic binding is refactored from two
hand-rolled subscribe-then-rollback blocks into a loop over the topic set,
so adding this third topic doesn't nest the rollback logic further and a
failure partway through still unwinds every topic already bound in that
call.
* test(ec2): cover the GetPasswordData gateway shim and remaining daemon branches
The gateway shim had no test file, leaving diff coverage below the
preflight threshold. Adds GetPasswordData_test.go covering input
validation, the happy path against a stubbed NATS responder, the
ErrNoResponders and ErrTimeout mappings to InvalidInstanceID.NotFound, a
daemon error payload, and a malformed JSON response. The request timeout
is now a package variable so the timeout test doesn't have to wait out
the real 5 seconds.
Also adds daemon-side coverage for the missing-InstanceId, malformed
input payload, and cross-account ownership-denied branches of
handleEC2GetPasswordData.
DescribeImages and DescribeInstances only ever populated PlatformDetails,
which is a billing string covering every OS. The AWS SDKs and the Terraform
provider key off the separate Platform field, which is either "windows" or
absent, so a Windows AMI read as Linux.
Derive Platform from PlatformDetails rather than persisting a new AMI field,
so already-imported Windows AMIs are fixed without a re-import.
The v4 to v5 topology migration runs as root under `spx admin upgrade`,
but predastore runs as spinifex-storage with an empty capability
bounding set. Two paths left it locked out of its own state.
The migration created the cluster data dir as root:root 0750, so
spinifex-storage could not traverse into the node dirs the rename had
placed under it, and clusterrun could not create new ones. Separately,
the raft recovery that follows each state replica move reopens badger
and writes a fresh snapshot as root, leaving root-owned files inside a
node dir that the rename had otherwise left correctly owned.
Both now take their ownership from the data being moved rather than
from a service user name, so a dev install running as the invoking user
needs no special case. The data dir chown is unconditional so a re-run
repairs a dir left behind by an earlier failed run, and the tree chown
runs even when there was nothing to recover, since that path still
opens the stores.
* feat: record every Bedrock invocation to JetStream, deliver to S3 and logs
Publishes an InvocationRecord for every Converse/InvokeModel call (streaming
or not) to a durable JetStream stream, then fans it out via a delivery
consumer to the account's own predastore bucket (body text, gated per
account by PutModelInvocationLoggingConfiguration) and a metadata-only log
line. Publish failures increment a drop counter instead of failing the
invocation. Streaming pumps record on every exit path via defer, including
client disconnect, and Close() now cancels the upstream request outright so
a disconnected client actually frees backend generation. vLLM streams
estimate output tokens from content-delta counts when real usage isn't
available yet, flagged usage_estimated; Anthropic always reports real usage.
* fix: deliver invocation records to S3 regardless of body, replicate stream
deliver() was gating the S3 write itself on InputText/OutputText being
non-empty, so an account with a destination bucket configured but body
delivery left off got nothing — the delivery flags already gate the body
fields upstream in Record(), and shouldn't gate delivery a second time.
EnsureInvocationStream defaulted to a single replica on a multi-node
cluster, unlike its sibling KV stores, so losing one node lost billing and
audit records without the control plane itself going down. Also renames
the awsgw predastore client from ecrStore to objStore now that it backs
both the ECR registry and Bedrock invocation delivery.
* feat: in-tree serving specs + KV weights override for ochre catalog
Extend catalogEntry with MinVRAMMiB, InstanceType and VLLMArgs so a
self-host model's serving requirements ship with the catalog rather
than living only in an operator's head. Add WeightsStore, a
deployment-scoped (not account-scoped) JetStream KV override in a new
bedrock-weights bucket that resolves a model's staged weights snapshot
ID, following the same shape as the existing CredentialStore. It is
plumbed into tieredCatalog and GetFoundationModel through a
package-level resolver (SetWeightsResolver/currentWeightsResolver),
since both are called from gateway/bedrock.go's fixed-arity route
table, which is out of scope here.
A self-host model with no resolvable weights snapshot is not
servable and must not be advertised: tieredCatalog filters it out and
GetFoundationModel returns ResourceNotFoundException for it.
Replace the meta.llama3-70b-instruct-v1:0 self-host entry, which
needs roughly 140 GB of VRAM in fp16 and has never actually run, with
meta.llama3-2-1b-instruct-v1:0, a real AWS-published Bedrock model
ID. At bf16 its 1.24B params are roughly 2.5 GiB of weights, well
inside wattle's single RTX A1000 (8188 MiB); MinVRAMMiB is set to
4096 to budget headroom for CUDA context, activations and KV cache on
top of that. This is an estimate, not a measurement off wattle -
.7.5's launcher should correct it against a real nvidia-smi reading
once the model is actually running there.
lookupCatalogEntry, used by the untiered invoke/converse routing
path, is left unchanged and still ignores weights resolvability;
gating that path is out of scope for this bead.
* fix: separate weights resolve errors from not-found in ochre catalog
An internal fault (e.g. a JetStream outage) reaching the KV weights
resolver was being collapsed into the same "not servable" verdict as
a clean not-found: tieredCatalog would silently unadvertise every
self-host model, and GetFoundationModel would return
ResourceNotFoundException for a model that does exist but whose
resolver is temporarily broken. Both call sites now distinguish the
two: a resolve error is logged and surfaced as its own error (mapped
to ServerInternal at the AWS error boundary), and only a clean
not-resolvable result yields ResourceNotFoundException.
tieredCatalog's signature changes to return an error alongside the
entries so ListFoundationModels can propagate a resolve fault instead
of silently returning a thinned-out list.
Also raises the self-host entry's gpu-memory-utilization from 0.5 to
0.6 (roughly 4913 MiB of vLLM's own pool) and MinVRAMMiB from 4096 to
5120 to match: at 0.5, weights (~2.5 GiB) left under 1.5 GiB for KV
cache and activations at max-model-len=8192, likely too tight for any
real concurrency. The two numbers are meant to agree: MinVRAMMiB is
the admission-gate floor, gpu-memory-utilization is what tells vLLM
to actually claim that much on an 8188 MiB device.
Comments trimmed to the repo's 3-line limit and stripped of plan/bead
references; the VRAM reasoning and model-choice rationale live in the
commit body and the plan doc instead.
Threading the weights resolver through as an explicit parameter
(mirroring CredentialResolver) rather than a package-level global was
considered per review. Doing so needs edits to gateway/bedrock.go
(bedrockRouteHandler's signature, both route closures, the
Bedrock_Request call site, a new bedrockWeightsResolver() method) and
gateway/gateway.go (a new BedrockWeights field on GatewayConfig).
Both are outside gateway/bedrock/ and are explicitly out of scope for
this change, owned by a parallel workstream already modifying the
same route table. The global resolver stays, now with the
error-handling fix above.
* fix: stop e2e GetFoundationModel regressing on the new weights gate
TestGetFoundationModel's "known model" subtest asserted on
meta.llama3-2-1b-instruct-v1:0 unconditionally, but that entry is now
gated on a weights snapshot staged in the bedrock-weights KV bucket.
On any real deployment that bucket starts empty, so the subtest would
fail with ResourceNotFoundException outside of OCHRE_E2E_SELFHOST.
The "known model" assertion now targets the Anthropic entry, which
resolves unconditionally and keeps testing what the subtest was
actually there for: a known model round-trips, an unknown one is
ResourceNotFoundException. The self-host assertion moves to a new
TestGetFoundationModelSelfHost, gated behind OCHRE_E2E_SELFHOST=1
alongside TestConverseSelfHost, where weights are expected staged.
tests/integration/bedrock_test.go's TestListFoundationModels already
wires a weights-resolver stub before asserting the self-host entry is
present, so it does not regress the same way.
* test: correct a stale assertion message on the ochre catalog weights gate
joshsiv-mulga