mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 09:01:31 +00:00
Josh Siviour
261d5d505e
* feat: in-tree serving specs + KV weights override for ochre catalog Extend catalogEntry with MinVRAMMiB, InstanceType and VLLMArgs so a self-host model's serving requirements ship with the catalog rather than living only in an operator's head. Add WeightsStore, a deployment-scoped (not account-scoped) JetStream KV override in a new bedrock-weights bucket that resolves a model's staged weights snapshot ID, following the same shape as the existing CredentialStore. It is plumbed into tieredCatalog and GetFoundationModel through a package-level resolver (SetWeightsResolver/currentWeightsResolver), since both are called from gateway/bedrock.go's fixed-arity route table, which is out of scope here. A self-host model with no resolvable weights snapshot is not servable and must not be advertised: tieredCatalog filters it out and GetFoundationModel returns ResourceNotFoundException for it. Replace the meta.llama3-70b-instruct-v1:0 self-host entry, which needs roughly 140 GB of VRAM in fp16 and has never actually run, with meta.llama3-2-1b-instruct-v1:0, a real AWS-published Bedrock model ID. At bf16 its 1.24B params are roughly 2.5 GiB of weights, well inside wattle's single RTX A1000 (8188 MiB); MinVRAMMiB is set to 4096 to budget headroom for CUDA context, activations and KV cache on top of that. This is an estimate, not a measurement off wattle - .7.5's launcher should correct it against a real nvidia-smi reading once the model is actually running there. lookupCatalogEntry, used by the untiered invoke/converse routing path, is left unchanged and still ignores weights resolvability; gating that path is out of scope for this bead. * fix: separate weights resolve errors from not-found in ochre catalog An internal fault (e.g. a JetStream outage) reaching the KV weights resolver was being collapsed into the same "not servable" verdict as a clean not-found: tieredCatalog would silently unadvertise every self-host model, and GetFoundationModel would return ResourceNotFoundException for a model that does exist but whose resolver is temporarily broken. Both call sites now distinguish the two: a resolve error is logged and surfaced as its own error (mapped to ServerInternal at the AWS error boundary), and only a clean not-resolvable result yields ResourceNotFoundException. tieredCatalog's signature changes to return an error alongside the entries so ListFoundationModels can propagate a resolve fault instead of silently returning a thinned-out list. Also raises the self-host entry's gpu-memory-utilization from 0.5 to 0.6 (roughly 4913 MiB of vLLM's own pool) and MinVRAMMiB from 4096 to 5120 to match: at 0.5, weights (~2.5 GiB) left under 1.5 GiB for KV cache and activations at max-model-len=8192, likely too tight for any real concurrency. The two numbers are meant to agree: MinVRAMMiB is the admission-gate floor, gpu-memory-utilization is what tells vLLM to actually claim that much on an 8188 MiB device. Comments trimmed to the repo's 3-line limit and stripped of plan/bead references; the VRAM reasoning and model-choice rationale live in the commit body and the plan doc instead. Threading the weights resolver through as an explicit parameter (mirroring CredentialResolver) rather than a package-level global was considered per review. Doing so needs edits to gateway/bedrock.go (bedrockRouteHandler's signature, both route closures, the Bedrock_Request call site, a new bedrockWeightsResolver() method) and gateway/gateway.go (a new BedrockWeights field on GatewayConfig). Both are outside gateway/bedrock/ and are explicitly out of scope for this change, owned by a parallel workstream already modifying the same route table. The global resolver stays, now with the error-handling fix above. * fix: stop e2e GetFoundationModel regressing on the new weights gate TestGetFoundationModel's "known model" subtest asserted on meta.llama3-2-1b-instruct-v1:0 unconditionally, but that entry is now gated on a weights snapshot staged in the bedrock-weights KV bucket. On any real deployment that bucket starts empty, so the subtest would fail with ResourceNotFoundException outside of OCHRE_E2E_SELFHOST. The "known model" assertion now targets the Anthropic entry, which resolves unconditionally and keeps testing what the subtest was actually there for: a known model round-trips, an unknown one is ResourceNotFoundException. The self-host assertion moves to a new TestGetFoundationModelSelfHost, gated behind OCHRE_E2E_SELFHOST=1 alongside TestConverseSelfHost, where weights are expected staged. tests/integration/bedrock_test.go's TestListFoundationModels already wires a weights-resolver stub before asserting the self-host entry is present, so it does not regress the same way. * test: correct a stale assertion message on the ochre catalog weights gate