Align the top of the README with mulgadc.com: new headline and intro,
rewritten "What is Spinifex?", a new "Three ways to deploy" section
(Neocloud / on-premise / edge) and a new "AWS compatibility" service
matrix (EC2, EBS, S3, VPC, IAM, ALB/NLB, EKS, ECR, ECS available; RDS
Q3 2026) linking the GitHub releases. Refresh the architecture prose and
Key Features, add Deploy and AWS-services nav links, and swap banner.svg
for the ember-on-ink mini-stack design matching platform.svg.
Rebuild the message-driven architecture diagram in the ember-on-ink
theme to match platform.svg: ember gradient NATS bus, ember-tinted
gateway and daemon cards, neutral clients and backend cards. Content
unchanged (clients → gateway → NATS → ec2/ebs/s3/vpc/iam daemons →
local backends). De-slop the copy and refresh the img alt text.
Replace the platform diagram with a static SVG matching the mulgadc.com
front-page stack diagram: ember accent, ink panel, the portable-stack
box (Neocloud / on-prem / edge) wrapping the Spinifex AWS-compatible
service layer, over standard Linux on CPU/GPU hardware. Self-contained
and GitHub-safe (presentation attributes only). Refresh the img alt text.
* fix(admin): round the AMI root volume up so the image fits
`spx admin images import` sized the root volume with integer division, which
floors: a 4.94 GiB image registered a 4 GiB volume. The image is then copied
into a volume too small to hold it, is truncated, and the guest boots with no
root partition — it stalls on the root device until systemd gives up, and
nothing anywhere reports an undersized volume.
floorVolumeSizeToAMI already exists to prevent exactly this ("Under-sizing
hangs the guest in dracut"), but it raises the caller's requested size to the
AMI's recorded VolumeSizeGiB — the same floored value — so it inherited the
mistake instead of catching it.
This stayed hidden because every current system image is an exact multiple of
1 GiB, where flooring and rounding up agree. It surfaced on a 5306470400-byte
image, which would not boot.
Round up instead, via a helper so the boundaries are testable: an exact
multiple must not gain a spare GiB, and one byte over must.
* fix(eks): default a nodegroup's root volume to 20 GiB, as AWS does
CreateNodegroup stored diskSize verbatim, so omitting it persisted 0 and
the launch skipped its block device mapping entirely, dropping the worker
onto whatever volume the AMI itself declares.
That was harmless while the node AMIs baked generous virtual disks — the
GPU image declares 16 GiB for ~2 GiB of content, so the fall-through
happened to leave plenty of room. It stops being harmless as soon as an
image is sized to its contents: the worker then boots onto a volume with
barely a spare GiB and cannot pull the container images it exists to run.
Sizing the node is the nodegroup's decision either way, not a property
the image should be able to dictate by accident.
20 GiB matches the AWS EKS managed-nodegroup default. It is resolved at
create rather than at launch so DescribeNodegroup reports the size the
node actually gets.
TestCreateNodegroup_NoDiskSizeOmitsBlockDeviceMapping asserted the old
fall-through and is replaced by its inverse; the explicit-size case is
already covered by TestCreateNodegroup_DiskSizePropagatesToBlockDeviceMapping.
* fix(ecs): size a container instance's root volume to 30 GiB by default
ProvisionCapacity built its RunInstances input without a block device
mapping, so a container instance inherited whatever volume its AMI
declares. EC2 has no default of its own to fall back on — the AMI is the
only thing that decides — and a node AMI is sized to hold the image, which
leaves the agent no room for the container images it exists to pull.
30 GiB mirrors what the AWS ECS-optimized AMI declares. ProvisionCapacity
is spinifex's own API rather than an AWS one, so DiskSize is added to its
input for callers that want to size a cluster explicitly, defaulting when
omitted the same way InstanceType and Count already do.
* fix(images): pull the Ubuntu catalog images from releases, not a pinned daily
Both Ubuntu catalog entries pointed at a dated build under the codename
dailies tree, which Canonical prunes as builds age out. That path now 404s,
so "spx admin images import --name ubuntu-26.04-x86_64" fails outright for
x86_64 and arm64 alike — the entries had rotted with nothing to catch it.
Release builds are retained rather than pruned, so the pin stays resolvable
while keeping a specific, reproducible build. The filenames under releases/
carry the version instead of the codename, which matters because the sums
entry is matched on the URL's basename.
CI is unaffected and could not have caught this: it fetches resolute/current/,
a symlink that never rots, so nothing exercised the pinned path.
Debian pins a dated directory too but is fine — those builds are retained,
and its entries still resolve.
Phase 8 of TestNATUplink checked the second VPC router's lr-nat-list with a
single non-polling assert; the igw-default-snat row commits a beat after the
gateway LRP appears, so the read raced the OVN commit and intermittently saw
an empty list (flaky CI red). Wrap it in EventuallyErr, matching the sibling
LRP and host-route polls in the same phase.
AddSNAT logged only on the idempotent-skip path, so the first install was
silent and RCA could not tell a late SNAT from a missing one. Emit an install
log keyed by router + vpc_cidr + external_ip.
Server-cert IP SANs were built solely from live interface enumeration
(DiscoverLocalIPs) plus bindIP, so the canonical mgmt-bridge IP entered
the SAN set only when br-mgmt happened to be up at cert-generation time.
On a host where br-mgmt is down when the cert is minted, 10.15.8.1 was
silently dropped and the EKS control-plane publish to https://10.15.8.1
failed cert verify, leaving the cluster stuck CREATING.
Pin config.DefaultMgmtBridgeIP into the extra SANs for both the init and
join cert paths so publish succeeds regardless of interface state at
generation time. Add a regression test asserting the SAN is present when
br-mgmt is not a live interface.
Add an IRSAPod subtest that applies a real pod wired for IRSA — a projected
ServiceAccount token (aud sts.amazonaws.com) plus the AWS_ROLE_ARN /
AWS_WEB_IDENTITY_TOKEN_FILE / AWS_ENDPOINT_URL_STS / AWS_CA_BUNDLE env the
addon-sync agent injects — and asserts the pod's in-cluster
aws sts get-caller-identity resolves to the assumed IRSA role. This closes the
gap left by IRSAWebIdentity, which only exchanges the token host-side.
Extract registerOIDCRole (shared with IRSAWebIdentity); the pod runs on a
1-node worker nodegroup (the control-plane node is CriticalAddonsOnly-tainted)
and reaches awsgw STS over the worker subnet NAT.
The build script derives BUILD_DIR from SYSTEM_IMAGE_BUILD_DIR (the AMI
publish workflow points it off tmpfs so the 16G Ubuntu GPU raw fits), but
the publish step hardcoded /tmp and could not find the freshly built raw.
GPU legs failed 'Raw image not found'; Alpine legs silently reused stale
pre-existing /tmp raws on the persistent runner. Mirror the build script's
derivation so publish looks where the build actually wrote.
The publish-eks-ami matrix fans out four legs on the single persistent
ci-single runner, all reusing one workspace root. go work init is
create-only and checkout never scrubs the root go.work, so legs after the
first fail with 'go.work already exists'. Remove any stale go.work before
init so each leg re-inits cleanly.
Expand the publish workflow to build the two Ubuntu NVIDIA GPU variants
alongside the Alpine eks-node/ecs-agent AMIs. The image build+publish
scripts are already manifest-driven and Ubuntu-aware; only the workflow
lacked the GPU legs.
Refactor the single-job body into a matrix over the four image manifests,
resolved by a new 'variant' input (all|cpu|gpu) so the slow 16G GPU legs
can be skipped when only the CPU AMIs need republishing. GPU images do not
fit a RAM-backed /tmp, so set SYSTEM_IMAGE_BUILD_DIR onto the runner's
on-disk _temp; legs run serially (max-parallel 1) on the single ci runner.
The nodegroup label was only written to K3S_NODE_LABEL in agent.env, but
k3s has no such env var, so non-GPU workers never received the label.
waitWorkersReady buckets Ready nodes by that label (added in #511), so
the affected nodegroups never counted toward their ACTIVE transition and
stayed CREATING until timeout.
Ride the label on a real k3s node-label config drop-in
(15-nodegroup-label.yaml), delivered via cloud-init user-data. Drop the
inert K3S_NODE_LABEL from agent.env and the initd export.
WaitForInstanceState now aborts immediately with the recorded StateReason
message when it observes a reaped state (shutting-down/terminated) while
waiting for a live target, instead of burning the full 5-minute timeout on
an instance that can never reach it.
MarkFailed takes a context and records the failure on the active span via
span.RecordError so a reaped launch surfaces in the APM error stream,
correlated by trace id. Only the LaunchRunInstances path carries a live
span; other callers pass context.Background() and the record is a no-op.
Surfaced by RCA of E2E run 29258398516 (TestAccountScoping/Step2).
TestNATGateway flaked with 100% egress loss: the tenant VPC gateway router's
OVN MAC_Binding had no entry for its upstream default-route nexthop. OVN
resolves that nexthop via lazy dynamic ARP on first egress; if that ARP is
lost during gateway bring-up, SNAT'd packets die in lr_in_arp_resolve and no
reply returns. MAC_Binding is per (datapath, logical_port, ip), so a sibling
VPC router resolving the same nexthop does not help.
Deterministically seed a static MAC binding for the nexthop at IGW attach. The
nexthop MAC is the host's own default-gateway MAC (the host shares the physical
uplink), read from ip neigh and written via ovn-nbctl static-mac-binding-add
(northd programs it into SB MAC_Binding, GC-safe). Host layer L0 owns the
shell-out per ADR-0006 S2; igw calls it via an injected hook. Best-effort and
idempotent: a missing MAC leaves dynamic ARP as the fallback, and it re-runs on
every AttachIGW so a binding lost to a later flap self-heals.
utils.GetOrCreateKVBucket hardcoded Replicas=1, so the ~38 handlers that
create their KV buckets lazily (VPC/subnet/ENI/EIP/ELBv2/ECS/ECR/ACM/
natgw/snapshot/placementgroup/IPAM/account-settings/dhcp/static-pool …)
created them at R1 on a multi-node cluster. upgradeJetStreamReplicas runs
once at startup, so any bucket created lazily after boot stayed R1
permanently — and an R1 stream's single Raft replica is transiently
leaderless during formation/join/rebalance, surfacing as
ErrNoResponders/ErrTimeout on KV ops.
Add a package-level default replica count that the daemon sets once at
boot (startCluster, after initJetStream) to len(clusterConfig.Nodes),
before any handler creates a bucket. GetOrCreateKVBucket now creates at
that default, so lazily-created buckets are born quorate. The one-shot
startup UpdateReplicas remains as a safety net for pre-existing R1
buckets from older deploys.
Zero call-site churn: the default flows through the existing
GetOrCreateKVBucketWithReplicas path already used by eks/ecs/iam.
TestSpawnRegisteredReconcilers_ResumesNonTerminal flaked two ways. First,
the shared fixture meta stamps a weeks-old CreatedAt, so a resumed CREATING
reconciler instantly trips the CREATE timeout, marks the cluster FAILED and
deregisters before the registry.Has assertion runs — the test was racing a
self-removal. Stamp a recent CreatedAt so the reconciler stays CREATING.
Second, SpawnRegisteredReconcilers ran a one-shot JetStream enumeration
(KeyValueStoreNames + kv.Keys), which is eventually consistent: under load a
just-committed cluster can be missing and never resumed, silently on a real
daemon. Re-scan until the observed CREATING/ACTIVE count stops growing, capped
and short-backed-off; Spawn is idempotent so re-runs only pick up stragglers.
discoverSingleNodeIPs seeds 127.0.0.1 then appends net.InterfaceAddrs()
in undefined order, so WANHost=nodeIPs[0] was loopback and the guest
gateway URL helpers fell through to the first non-loopback interface
addr — an arbitrary co-tenant bridge on multi-bridge shared hosts
(wattle: br-lan-env1..23, br-wan), leaving guest VMs unable to reach the
advertised gateway to register.
Resolve WANHost from spinifex.toml nodes.<node>.advertise (falling back
to that node's host), mirroring awsgwBindIP's toml scan; fall back to the
first non-loopback discovered addr only when config is unreadable. This
removes the need for the SPINIFEX_WAN_IP override on shared hosts.
The crash-supervisor auto-restart path (qmpHeartbeat -> HandleCrash ->
MaybeRestart -> startQEMU -> newQMPClientWithHandshake -> dialQMPWithRetry)
already unlinks the stale QMP socket and retries a refused/absent connect. But
a freshly relaunched QEMU can accept the connect then reset the connection
mid-greeting while it initialises, surfacing 'connection reset by peer'
immediately after 'QEMU started successfully' as a ServerInternal.
Add ECONNRESET and EPIPE to isTransientDialError so the dial is retried until
QEMU services the monitor, closing the auto-restart QMP-connect race.
Add non-streaming InvokeModel to bedrock-runtime. Unlike Converse's unified
schema, InvokeModel passes model-provider-native bodies through, so route by
model family: Anthropic Claude bodies forward to the Messages API (drop the
Bedrock-only anthropic_version, inject model, return verbatim); Meta Llama
bodies translate to/from the self-host vLLM OpenAI completions endpoint.
Raw-body dispatch bypasses the JSON response path (WriteRawResponse). Reuses
the Phase 1 credential + endpoint resolvers and upstream-status mapping.
Unit-tested (routing, body rewrite, translation, error classes) plus gated
InvokeModel E2E.
Both services already run otelsetup.Init and ship logs/metrics but never
opened a span, so they were invisible in the Kibana APM service map.
qmp-collector now opens a root span per per-instance poll tick (marks
error on QMP collect/publish failure); spinifex-ui now wraps its HTTP
handler chain with otelsetup.HTTPMiddleware and its reverse-proxy
transport with otelhttp, so UI requests get a server span and proxied
calls to awsgw/predastore carry traceparent instead of stopping the
trace at the UI hop.
nats and eks are intentionally left uninstrumented in this pass: nats is
the vendored message-broker process with no message-level hook to attach
spans to (the traffic it carries is already traced end-to-end at its
producer and consumer endpoints); eks request/response calls already
produce spans today under service.name=spinifex-daemon since the
handlers run in that process, and its async reconciler/state-machine
goroutines need a separate span design across a much larger surface.
Skip ERROR log when a subscription is already invalid
(nats.ErrBadSubscription) during graceful shutdown; log at debug
instead. Cosmetic cleanup race, no functional impact.
golangci-lint canonicalheader flagged 6 lowercase header literals in the
Ochre/Bedrock gateway (anthropic.go, anthropic_test.go, vllm.go). Canonical
form is wire-equivalent; canonicalise to X-Api-Key, Anthropic-Version and
Content-Type to restore CI lint.
* feat(bedrock): Ochre runtime MVP — Converse over self-host + Anthropic-direct
Add the AWS-Bedrock-compatible gateway services (bedrock, bedrock-runtime)
following the EKS REST-JSON pattern: ListFoundationModels / GetFoundationModel
over a static connectivity-tiered catalog, and non-streaming Converse routed
per modelId to one of two provider backends — self-hosted vLLM (OpenAI wire)
and Anthropic-direct (Anthropic Messages wire).
Per-account provider API keys are a master-key-encrypted JetStream KV secret
with an optional platform default; the advertised catalog gates external
models on a resolvable credential. Self-host endpoints are pinned via config
(OCHRE_VLLM_ENDPOINTS) behind an EndpointResolver seam.
Adds gateway/bedrock/ (catalog, provider router, Anthropic + vLLM adapters,
credential store), awserrors REST-JSON codes, and the bedrock E2E suite
(list/get light; Converse gated on live backends). Daemon/NATS control plane
and dynamic endpoint lifecycle are deferred to Phase 3.
* test(bedrock): cover dispatch, router, credentials KV + endpoint config
Add unit coverage for the Ochre Phase 1 diff: Bedrock_Request /
BedrockRuntime_Request end-to-end dispatch, provider router, JSON
response/error writers, upstream-status mapping, the credential-store KV
round-trip, the Anthropic bound-provider path, and parseBedrockEndpoints.
Lifts PR diff coverage above the 60% gate.
* ci: GPU e2e workflow on ci-gpu self-hosted runner
Runs tests/e2e/gpu (EC2/ECS/EKS GPU passthrough) against the persistent,
hand-bootstrapped wattle shared host, driven over SSH by a banksia runner
labelled ci-gpu (matches the ci-single/ci-multi convention) — no runner
installed on the shared box.
- runs-on [self-hosted, ci-gpu]; concurrency ci-gpu, cancel-in-progress
false (single A1000 = single-flight, queue not stomp)
- workflow_dispatch + GPU-path filter (not every push)
- manual-work mutex gate (~/.ci-gpu-paused / CI_GPU_PAUSED) so hands-on GPU
testing blocks CI
- if: always() cleanup terminates launched GPU instances; never resets the
box (no setup-ovn.sh/dev-reset/cluster-reset)
- drives the control plane with spx + the already-configured aws spinifex
profile directly (no wrapper scripts, nothing to install on wattle)
* ci(gpu-e2e): advisory AMI preflight, fail only if zero GPU AMIs
The suite is self-guarding — each test skips when its AMI is absent. The
hard-fail on the base ubuntu GPU AMI blocked the whole run even when the
ECS GPU AMI (the only one imported) could execute. Inventory the AMIs and
fail only when none are present.
* ci(gpu-e2e): self-provision base GPU AMI, hard-require it in preflight
The GPU E2E workflow was green-because-skipped: with no GPU AMIs imported
the self-guarding suite skipped every test, passing as a false green. Make
the raw-EC2 GPU passthrough test the always-on baseline.
- Add an import-if-absent setup step that pulls the prebuilt base GPU image
from the catalog by name (ubuntu-26.04-nvidia-gpu-x86_64), naming the AMI
exactly what tests/e2e/gpu filters for. Idempotent; caches on wattle's
persistent loop-fs. Runs spx under sg spinifex (local storage write).
- Preflight now hard-requires the base AMI (fail, not skip) so a silent
import failure can't pass as green; ECS/EKS node AMIs stay opt-in warn.
- Reconcile the local build path: ubuntu-gpu-nvidia.conf AMI_NAME and
launch-gpu.sh's nvidia default now use the same canonical name, so a
locally built+imported image also satisfies the suite.
* ci(gpu-e2e): pass --ami-name on base GPU import so the AMI name matches
spx admin images import --name selects the catalog entry but names the
resulting AMI generically (ami-<distro>-<version>-<arch>), so the import
landed as ami-ubuntu-26.04-x86_64 and the tests/e2e/gpu name filter never
matched. Pass --ami-name ubuntu-26.04-nvidia-gpu-x86_64 explicitly (same
flag build-system-image.sh uses) and poll briefly for availability.
* fix(network,vm): reconcile auto-assigned public-IP NAT + scale VFIO QMP timeout
Auto-assigned public IPs were never carried through the network reconciler.
applyEIPs ran AddEIP + ensureGuestPortDatapath for user EIPs only; auto-assigned
IPs (intent.Ports[].PublicIP) got egress exemption but no NAT re-assert and no
gatewayLRP->guest convergence. Their dnat_and_snat was created once at
RunInstances and never touched, so a vpcd/OVN rebuild dropped it and the
guest-port hop never converged: inbound DNATs at the gateway but never reaches
the guest (TCP/SSH dark) while ICMP is answered by the gateway LR (ping OK).
floatingIPSpecs now folds every auto-assigned/ELB public IP into the same
AddEIP + guest-port convergence path as user EIPs (deduped by private IP; a user
EIP wins), making the datapath self-healing across reconcile ticks and deploys.
qmpGreetingTimeout for VFIO guests now scales with guest RAM (base 30s + 8s/GiB,
floored at 180s, capped at 600s) instead of a flat 180s, so a large guest is not
SIGKILLed mid-pin while the whole guest RAM is IOMMU-locked before QMP answers.
* fix(vm): scale QMP handshake command deadline for VFIO launches
The greeting timeout scaling alone did not fix the VFIO launch SIGKILL: a GPU
guest sends its QMP greeting before the RAM pin completes, then cannot service
qmp_capabilities until the pin finishes. sendQMPCommand hardcoded a 30s response
deadline, so the handshake decode timed out (~43s for a 16 GiB g5.xlarge) and
the launch cleanup SIGKILLed a QEMU that would have come up.
Parameterize the command deadline (sendQMPCommandWithTimeout); the handshake now
uses the RAM-scaled qmpGreetingTimeout (base 30s + 8s/GiB, floor 180s, cap 600s)
for qmp_capabilities. Other callers keep the 30s default.
* test(gpu-e2e): share one GPU instance across passthrough subtests to avoid mid-suite IP recycle
Recycling both the floating and private IP onto a new ENI left the ingress
datapath targeting the terminated instance, so the successor was unreachable
on its EIP while IMDS stayed up.
- AddEIP re-points the dnat_and_snat row when the owning ENI changed even
though the external+logical IP are unchanged, keyed on the portable
spinifex:logical_port external-id (centralised mode leaves the native
LogicalPort column empty). The genuine same-ENI case still skips row churn.
- Flush the SB MAC_Binding for a private IP on ENI teardown and successor
bind so a reused address re-resolves to the live owner's MAC, via a new
MACBindingFlusher backed by ovn-sbctl.
- Reconciler prunes orphan dnat_and_snat rows whose stamped owning ENI is
absent from intent (leaked by lost vpc.delete-nat across dead VPCs).
- IMDS tap responder suspends retries for an ENI whose record is gone,
stopping the 15s reconcile churn.
* test(daemon): cover GPU inventory in GetNodes and GPU attachment in GetVMs
Adds unit tests for buildGPUInventory/handleNodeStatus (whole-GPU claimed/
free, MIG slices claimed/free, no-GPU node) and resolveVMGPU/handleNodeVMs
(whole-GPU attachment, MIG attachment, no attachment). Production code for
these paths (types.GPUInfo/GPUSliceInfo/VMGPUInfo, buildGPUInventory,
resolveVMGPU) and EC2 DescribeInstanceTypes GpuInfo.Gpus population
(instancetypes.GenerateGPUTypes/GenerateMIGTypes/IsGPUType) already landed
via the daemon GPU-admission work and were already test-covered for part C.
* feat(ui): shared GpuInstanceTypeSelect grouped instance-type picker
Reusable Select wrapper for EC2/EKS/ECS instance-type pickers: groups
Standard vs GPU-Accelerated types via structural GpuInfo.Gpus detection
(AWS SDK shape, no name heuristics), annotates GPU items with VRAM from
MemoryInfo.SizeInMiB and an optional per-type availability count, and
falls back to a single ungrouped list when no GPU types are present.
* feat(ui): EC2 GPU inventory, instance GPU card, shared picker in run-instances
* feat(ui): EKS nodegroup GPU picker + AMI/taint note
Add-nodegroup form now uses the shared GpuInstanceTypeSelect, grouping
and VRAM-annotating GPU instance types instead of a free-text field.
Selecting a GPU type surfaces a note that the GPU node AMI is
auto-selected and the nodegroup is tainted
nvidia.com/gpu=present:NoSchedule by default. Nodegroup rows badge GPU
nodegroups, derived from the primary instance type since
NodegroupRecord does not expose a GPU flag over the API.
* feat(ui): ECS GPU capacity picker, task-def GPU count, task gpuIds
* feat(eks): expose nodegroup GPU signal in Describe/ListNodegroups + UI badge
* chore(ui): rebuild embedded console dist for GPU UI
Bump github.com/mulgadc/viperblock to v1.11.1-0.20260713024608-eb4a376becc8
(dev tip through viperblock #34): bounded parallel chunk upload + coalesced
checkpoint (edou7 sec 7 items 2 and 9), sweeping the require up from #32 and
picking up the #33 write backpressure on the way. The nbdkit plugin the VM
write plane uses is built from the submodule source; this bump aligns the
viperblockd library that spx links.
Julian Sommer