mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
Josh Siviour
261d5d505e
* feat: in-tree serving specs + KV weights override for ochre catalog Extend catalogEntry with MinVRAMMiB, InstanceType and VLLMArgs so a self-host model's serving requirements ship with the catalog rather than living only in an operator's head. Add WeightsStore, a deployment-scoped (not account-scoped) JetStream KV override in a new bedrock-weights bucket that resolves a model's staged weights snapshot ID, following the same shape as the existing CredentialStore. It is plumbed into tieredCatalog and GetFoundationModel through a package-level resolver (SetWeightsResolver/currentWeightsResolver), since both are called from gateway/bedrock.go's fixed-arity route table, which is out of scope here. A self-host model with no resolvable weights snapshot is not servable and must not be advertised: tieredCatalog filters it out and GetFoundationModel returns ResourceNotFoundException for it. Replace the meta.llama3-70b-instruct-v1:0 self-host entry, which needs roughly 140 GB of VRAM in fp16 and has never actually run, with meta.llama3-2-1b-instruct-v1:0, a real AWS-published Bedrock model ID. At bf16 its 1.24B params are roughly 2.5 GiB of weights, well inside wattle's single RTX A1000 (8188 MiB); MinVRAMMiB is set to 4096 to budget headroom for CUDA context, activations and KV cache on top of that. This is an estimate, not a measurement off wattle - .7.5's launcher should correct it against a real nvidia-smi reading once the model is actually running there. lookupCatalogEntry, used by the untiered invoke/converse routing path, is left unchanged and still ignores weights resolvability; gating that path is out of scope for this bead. * fix: separate weights resolve errors from not-found in ochre catalog An internal fault (e.g. a JetStream outage) reaching the KV weights resolver was being collapsed into the same "not servable" verdict as a clean not-found: tieredCatalog would silently unadvertise every self-host model, and GetFoundationModel would return ResourceNotFoundException for a model that does exist but whose resolver is temporarily broken. Both call sites now distinguish the two: a resolve error is logged and surfaced as its own error (mapped to ServerInternal at the AWS error boundary), and only a clean not-resolvable result yields ResourceNotFoundException. tieredCatalog's signature changes to return an error alongside the entries so ListFoundationModels can propagate a resolve fault instead of silently returning a thinned-out list. Also raises the self-host entry's gpu-memory-utilization from 0.5 to 0.6 (roughly 4913 MiB of vLLM's own pool) and MinVRAMMiB from 4096 to 5120 to match: at 0.5, weights (~2.5 GiB) left under 1.5 GiB for KV cache and activations at max-model-len=8192, likely too tight for any real concurrency. The two numbers are meant to agree: MinVRAMMiB is the admission-gate floor, gpu-memory-utilization is what tells vLLM to actually claim that much on an 8188 MiB device. Comments trimmed to the repo's 3-line limit and stripped of plan/bead references; the VRAM reasoning and model-choice rationale live in the commit body and the plan doc instead. Threading the weights resolver through as an explicit parameter (mirroring CredentialResolver) rather than a package-level global was considered per review. Doing so needs edits to gateway/bedrock.go (bedrockRouteHandler's signature, both route closures, the Bedrock_Request call site, a new bedrockWeightsResolver() method) and gateway/gateway.go (a new BedrockWeights field on GatewayConfig). Both are outside gateway/bedrock/ and are explicitly out of scope for this change, owned by a parallel workstream already modifying the same route table. The global resolver stays, now with the error-handling fix above. * fix: stop e2e GetFoundationModel regressing on the new weights gate TestGetFoundationModel's "known model" subtest asserted on meta.llama3-2-1b-instruct-v1:0 unconditionally, but that entry is now gated on a weights snapshot staged in the bedrock-weights KV bucket. On any real deployment that bucket starts empty, so the subtest would fail with ResourceNotFoundException outside of OCHRE_E2E_SELFHOST. The "known model" assertion now targets the Anthropic entry, which resolves unconditionally and keeps testing what the subtest was actually there for: a known model round-trips, an unknown one is ResourceNotFoundException. The self-host assertion moves to a new TestGetFoundationModelSelfHost, gated behind OCHRE_E2E_SELFHOST=1 alongside TestConverseSelfHost, where weights are expected staged. tests/integration/bedrock_test.go's TestListFoundationModels already wires a weights-resolver stub before asserting the self-host entry is present, so it does not regress the same way. * test: correct a stale assertion message on the ochre catalog weights gate
69 lines
2.7 KiB
Go
69 lines
2.7 KiB
Go
//go:build integration
|
|
|
|
package integration
|
|
|
|
import (
|
|
"context"
|
|
"testing"
|
|
|
|
"github.com/aws/aws-sdk-go/aws"
|
|
"github.com/aws/aws-sdk-go/service/bedrock"
|
|
gateway_bedrock "github.com/mulgadc/spinifex/spinifex/gateway/bedrock"
|
|
"github.com/stretchr/testify/assert"
|
|
"github.com/stretchr/testify/require"
|
|
)
|
|
|
|
// modelSelfHostLlama and modelAnthropic mirror the static Phase-1 catalog in
|
|
// gateway/bedrock/catalog.go — see tests/e2e/bedrock/bedrock_test.go, whose
|
|
// TestListFoundationModels this ports.
|
|
const (
|
|
modelSelfHostLlama = "meta.llama3-2-1b-instruct-v1:0"
|
|
modelAnthropic = "anthropic.claude-3-5-sonnet-20240620-v1:0"
|
|
)
|
|
|
|
// integrationWeightsStub resolves a weights snapshot only for
|
|
// modelSelfHostLlama, standing in for the KV override that gates a self-host
|
|
// catalog entry on having somewhere to serve it from.
|
|
type integrationWeightsStub struct{}
|
|
|
|
func (integrationWeightsStub) Resolve(_ context.Context, modelID string) (string, bool, error) {
|
|
if modelID != modelSelfHostLlama {
|
|
return "", false, nil
|
|
}
|
|
return "snap-stub", true, nil
|
|
}
|
|
|
|
// TestListFoundationModels ports tests/e2e/bedrock/bedrock_test.go's
|
|
// TestListFoundationModels. ListFoundationModels is pure gateway-side static
|
|
// catalog logic (gateway/bedrock/catalog.go) with no NATS/daemon hop at all —
|
|
// no guest, no daemon wiring needed — so it round-trips through the real
|
|
// gateway HTTP/SigV4 path exactly like the live test, without a live cluster.
|
|
func TestListFoundationModels(t *testing.T) {
|
|
// Not t.Parallel(): ListFoundationModels/GetFoundationModel read the
|
|
// weights resolver through a package-level setter (StartGateway builds a
|
|
// GatewayConfig directly, with no field for it), so this test owns that
|
|
// process-wide state for its duration.
|
|
gateway_bedrock.SetWeightsResolver(integrationWeightsStub{})
|
|
t.Cleanup(func() { gateway_bedrock.SetWeightsResolver(nil) })
|
|
|
|
gw := StartGateway(t)
|
|
bedrockCli := gw.BedrockClient(t)
|
|
|
|
out, err := bedrockCli.ListFoundationModels(&bedrock.ListFoundationModelsInput{})
|
|
require.NoError(t, err, "list-foundation-models")
|
|
|
|
ids := make(map[string]*bedrock.FoundationModelSummary, len(out.ModelSummaries))
|
|
for _, m := range out.ModelSummaries {
|
|
ids[aws.StringValue(m.ModelId)] = m
|
|
}
|
|
|
|
assert.Contains(t, ids, modelSelfHostLlama, "self-host model must be advertised when its weights resolve")
|
|
|
|
// This tier configures no BedrockCredentials resolver, so the Anthropic
|
|
// entry is never resolvable and must not be advertised — the mirror image
|
|
// of the live test's "only assert when present" check (which tolerates
|
|
// either state depending on live cluster credential configuration).
|
|
_, anthropicPresent := ids[modelAnthropic]
|
|
assert.False(t, anthropicPresent, "anthropic model advertised without a resolvable credential")
|
|
}
|