The GPU model table was only ever used to report VRAM, so wrong or missing
entries were cosmetic. Bedrock's capacity check is the first code that rejects
on VRAM, which turns every latent data error into an admission decision.
Correct seven entries verified against pci.ids — most seriously 1002:7448,
labelled Instinct MI300X at 192GB when it is a Radeon Pro W7900 at 48GB, a
4x over-report that admits a model which then OOMs at weight load. Record the
smallest capacity where one PCI ID spans several cards, since under-reporting
refuses an endpoint an operator can see and override while over-reporting
fails later and looks like a vLLM bug.
Widen the table so common datacenter, workstation and consumer cards resolve
without an operator-supplied override, and apply GPUModelOverrides to the pool
devices as well as the advertised model, so the override reaches the code that
actually gates. An override with no memory_mib is left alone rather than
wiping discovered VRAM, since the config cannot distinguish it from a
deliberate zero.
MemoryMiB 0 stays the "undiscoverable" sentinel, but checkCapacity no longer
counts it as a device with no VRAM: an otherwise-free device now reports which
card is unknown and how to fix it.
* fix: make the predastore readiness gate a real readiness probe
checkPredastoreReady only did a TCP dial + close, so a listening socket
satisfied it well before predastore could serve TLS. In CI this passed
in 170 microseconds, 0.72s after predastore opened its listener, and
recovery started against a backend that wasn't serving yet — predastore
logged the probe itself as a TLS handshake EOF.
Replace the dial with an HTTPS GET against the configured predastore
endpoint on a package-level client (hoisted out of the 2s poll loop),
with a per-request 3s context timeout. Any HTTP response counts as
ready, including 403 from this deliberately unsigned request, which is
exactly what an unauthenticated S3 GET returns. Only a transport or TLS
failure means not-ready.
This narrows the race window and removes the spurious TLS-handshake-EOF
log lines predastore emits on every daemon start, but it does not close
the window on its own: predastore can serve TLS before its storage
backend answers reads correctly. The load-bearing fix is the viperblock
VBState read retry landing separately.
* style: trim the predastore readiness comments to three lines
GetPasswordData returned an empty string both when the guest had not emitted a
password yet and when it never could, because the launch key pair was ED25519
and cannot perform PKCS#1 v1.5 encryption. The two are indistinguishable, so a
caller in the second case polls forever with no signal.
An empty result on a Windows instance whose launch key is not RSA now returns
InvalidKeyPair.Type naming the cause and the fix. Every uncertainty keeps the
old empty result: a populated blob always wins, Linux is never judged, and an
unresolvable key is not treated as evidence of the wrong type.
get-password-data --priv-launch-key requires PEM, but ssh-keygen writes the
OpenSSH container by default, so even an explicitly-RSA key pair failed with
"Unable to decrypt password data using provided private key file" — which
reads like a padding problem rather than a format one. AWS returns PKCS#1 PEM
for RSA key pairs. OpenSSH reads that format for SSH either way.
The create-key-pair UI gains a key-type selector, defaulting to the rsa the
page already sent, with a hint that RSA is required for Windows password
retrieval.
The API default stays ed25519. Changing it is a separate decision.
ResourceNotFoundException is a shared wire code, so Bedrock aliases the EKS
error constant and inherited its message: a caller asking for a model was
told to list its EKS clusters. Register Bedrock overrides in the existing
errorLookupByService table, under both service keys since the gateway splits
bedrock from bedrock-runtime on the /model/ path prefix.
PutModelInvocationLoggingConfiguration also accepted embeddingDataDeliveryEnabled
and dropped it: the internal LoggingConfig had no such field, so the flag never
survived to Get. Carry it through both translation directions and fold it into
the bucket-required check, since embedding delivery with no bucket is as
undeliverable as text or image.
cloudbase-init has no IMDSv2 token support in any release, so a Windows
instance launched without an explicit MetadataOptions block 401s on every
metadata GET and never reaches its main plugin stage. Derive the launch
default from the resolved image's platform: Windows gets optional, every
other platform keeps required, and an explicit request value always wins.
* feat: add spx admin ochre weights stage/list/remove
Give operators a supported path to stage a self-hosted model's
weights: point at a predastore S3 prefix holding ordinary Hugging
Face files, and stage materialises them into a viperblock volume,
snapshots it, and records the source URI and snapshot ID in the
bedrock-weights KV bucket.
stage refuses before downloading anything if the model is not a
self-host catalog entry, or if the prefix is missing any required
file. Re-staging the same source is a no-op; a different source
replaces the entry and reports the previous snapshot for separate
reclamation.
WeightsStore gains GetWeights and a source-URI field on its record
to support this. LookupServingSpec (catalog.go) is reused unchanged
to validate --model-id.
* fix: unify ServingSpec after rebase onto dev's daemon-launcher spec
Rebasing onto dev pulled in mulga-vmfng.7.5's daemon-side ServingSpec
and LookupServingSpec, textually duplicating the ochre weights stage
CLI's own versions in catalog.go. Collapse both into one type carrying
ModelID and VLLMArgs, and one three-value LookupServingSpec so callers
distinguish an unknown model (found=false) from a provider-tier one
(selfHost=false). Update the daemon launcher's call site accordingly.
* test: cover ochre weights stage/list/remove command behaviour
Splits runOchreWeightsStage/List/Remove into testable decision functions
(runStageWeights, listWeightsOutput, removeWeights) separate from cobra and
NATS wiring, and makes mkfs.ext4 execution and the connect/exit steps
swappable so the full command flow can be driven against an embedded
JetStream server instead of live infrastructure. Also relaxes the tokenizer
file requirement to accept either tokenizer.json or tokenizer.model, since
canonical Llama 3.x repos ship only the former at the prefix root.
* fix: resolve mkfs.ext4 from sbin when it is absent from PATH
e2fsprogs installs mkfs.ext4 into sbin, which a non-login shell omits, so
weights staging failed on an unprivileged operator's otherwise fine host.
* fix: size the weights volume to a whole GiB so the tail block fits
viperblock rejects a write whose block overruns VolumeSize, so importing an
image of unaligned length failed on its last block. Rounding up also makes
the volume's byte size and its manifest SizeGiB describe the same volume.
Name the volume in the import error so a part-written import is traceable.
DescribeImages skipped any ami-* prefix whose config.json could not be
fetched, read, or parsed, logging each at DEBUG. At the daemon's default
INFO level the result was an empty Images list and a completion line
reading count=0, indistinguishable from a healthy empty registry.
Promote the three fault branches plus the body-close failure to WARN,
leave NoSuchKey at DEBUG since a prefix with no config is a legitimate
non-AMI directory, and report prefixes examined alongside images
returned so a mismatch is visible without raising the log level.
SetServiceOwnership chowned per-service directories to service users
without propagating a missing user, so spx admin init reported success
while leaving /etc/spinifex/northstar root-owned 0600. ChownRecursive
and SetServiceOwnership now return errors, failures are aggregated
across every service rather than stopping at the first, and
finalizeNodeSetup prints the unapplied paths instead of a success mark.
The per-tap datapath only matched IP, on the assumption that a guest reaches
169.254.169.254 through its default gateway. A guest that treats 169.254.0.0/16
as on-link resolves the address itself instead, and nothing on br-imds, br-int
or in OVN answered that ARP, so it could not reach IMDS at all.
Windows does exactly this, which left cloudbase-init timing out on every
metadata request and skipping the rest of its plugin stage.
Install a per-tap ARP responder for each captured address under the existing
cookie, replying with the gateway MAC the egress flow already presents.
Introduces handlers/bedrock, the daemon-side owner of vLLM serving VM
placement and lifecycle. State lives in a new JetStream KV bucket
(bedrock-endpoints) keyed {accountID}/{modelID}, cycling through
ABSENT -> STARTING -> READY -> DRAINING -> ABSENT. The daemon is the
sole KV writer; the gateway (or any other caller) requests transitions
over NATS via bedrock.endpoint.{ensure,describe,list,delete}, wired
into subscribeAll exactly as handlers/eks registers its own subjects.
Ensure is idempotent and single-flighted at two layers: a local
per-key mutex collapses concurrent same-process callers, and a
JetStream KV Create/CAS on the record's generation collapses races
across daemon replicas, so a burst of concurrent Ensure calls for one
model launches exactly one VM either way. Capacity is admission-checked
against gpu.Manager's free devices before a launch is claimed, so a
refusal never leaves a stuck STARTING record. Readiness is an explicit
poll of the serving VM's /v1/models endpoint until 200 or
startupTimeout, never a sleep.
Serving VMs boot via the system-instance path against a dedicated
system VPC, and follow the Phase-0 weights resolution: a snapshot
staged out-of-band is COW-cloned per endpoint into a data volume the
VM attaches post-boot, decoupled from the boot AMI.
Adds the .7.6 metering slice on top of .7.4's invocation stream: a
second durable consumer (ochre-usage-metering) aggregates per-account
token counts and external cost into a new bedrock-usage KV bucket,
keyed {accountID}/{modelID}/{period}. RequestID dedupe (an atomic
create-if-absent marker in bedrock-usage-dedupe) makes redelivery
count-neutral rather than double-billing a customer.
Prices are integer micro-USD per million tokens (D12): in-tree
catalog defaults with KV overrides in bedrock-prices, mirroring the
serving-spec shape from .7.3. Self-hosted models always resolve a
known zero; a model with neither an override nor a default resolves
Known=false, which the usage counter surfaces as a sticky CostUnknown
flag distinct from a genuine zero-cost period.
handlers_quota gains two Bedrock dimensions, both defaulting to
disabled independently of the block's existing Enabled toggle:
tokens-per-month (a check-against-history hard cap reading the
stream-fed counter, tolerant of a few seconds of lag) and
requests-per-minute (a local per-account token bucket enforced
immediately in-process, with a periodic KV sync for cross-node
observability only — enforcement never round-trips). Both are wired
into BedrockRuntime_Request ahead of every Converse/InvokeModel
variant.
* feat(ec2): serve the Windows administrator password via GetPasswordData
Adds the GetPasswordData EC2 API, mirroring the existing GetConsoleOutput
path: a per-instance NATS subscription, a daemon handler, and a gateway
shim registered in ec2Actions. The guest emits its encrypted administrator
password to the serial console at first boot; the daemon scans the whole
console log (not just the AWS 64KB tail, since the password would scroll
off a long-running instance) for the last <Password>...</Password> block
and returns its base64 ciphertext unchanged. Spinifex never handles key
material and never sees the plaintext password.
onInstanceUpHook's per-instance topic binding is refactored from two
hand-rolled subscribe-then-rollback blocks into a loop over the topic set,
so adding this third topic doesn't nest the rollback logic further and a
failure partway through still unwinds every topic already bound in that
call.
* test(ec2): cover the GetPasswordData gateway shim and remaining daemon branches
The gateway shim had no test file, leaving diff coverage below the
preflight threshold. Adds GetPasswordData_test.go covering input
validation, the happy path against a stubbed NATS responder, the
ErrNoResponders and ErrTimeout mappings to InvalidInstanceID.NotFound, a
daemon error payload, and a malformed JSON response. The request timeout
is now a package variable so the timeout test doesn't have to wait out
the real 5 seconds.
Also adds daemon-side coverage for the missing-InstanceId, malformed
input payload, and cross-account ownership-denied branches of
handleEC2GetPasswordData.
DescribeImages and DescribeInstances only ever populated PlatformDetails,
which is a billing string covering every OS. The AWS SDKs and the Terraform
provider key off the separate Platform field, which is either "windows" or
absent, so a Windows AMI read as Linux.
Derive Platform from PlatformDetails rather than persisting a new AMI field,
so already-imported Windows AMIs are fixed without a re-import.
The v4 to v5 topology migration runs as root under `spx admin upgrade`,
but predastore runs as spinifex-storage with an empty capability
bounding set. Two paths left it locked out of its own state.
The migration created the cluster data dir as root:root 0750, so
spinifex-storage could not traverse into the node dirs the rename had
placed under it, and clusterrun could not create new ones. Separately,
the raft recovery that follows each state replica move reopens badger
and writes a fresh snapshot as root, leaving root-owned files inside a
node dir that the rename had otherwise left correctly owned.
Both now take their ownership from the data being moved rather than
from a service user name, so a dev install running as the invoking user
needs no special case. The data dir chown is unconditional so a re-run
repairs a dir left behind by an earlier failed run, and the tree chown
runs even when there was nothing to recover, since that path still
opens the stores.
* feat: record every Bedrock invocation to JetStream, deliver to S3 and logs
Publishes an InvocationRecord for every Converse/InvokeModel call (streaming
or not) to a durable JetStream stream, then fans it out via a delivery
consumer to the account's own predastore bucket (body text, gated per
account by PutModelInvocationLoggingConfiguration) and a metadata-only log
line. Publish failures increment a drop counter instead of failing the
invocation. Streaming pumps record on every exit path via defer, including
client disconnect, and Close() now cancels the upstream request outright so
a disconnected client actually frees backend generation. vLLM streams
estimate output tokens from content-delta counts when real usage isn't
available yet, flagged usage_estimated; Anthropic always reports real usage.
* fix: deliver invocation records to S3 regardless of body, replicate stream
deliver() was gating the S3 write itself on InputText/OutputText being
non-empty, so an account with a destination bucket configured but body
delivery left off got nothing — the delivery flags already gate the body
fields upstream in Record(), and shouldn't gate delivery a second time.
EnsureInvocationStream defaulted to a single replica on a multi-node
cluster, unlike its sibling KV stores, so losing one node lost billing and
audit records without the control plane itself going down. Also renames
the awsgw predastore client from ecrStore to objStore now that it backs
both the ECR registry and Bedrock invocation delivery.
* feat: in-tree serving specs + KV weights override for ochre catalog
Extend catalogEntry with MinVRAMMiB, InstanceType and VLLMArgs so a
self-host model's serving requirements ship with the catalog rather
than living only in an operator's head. Add WeightsStore, a
deployment-scoped (not account-scoped) JetStream KV override in a new
bedrock-weights bucket that resolves a model's staged weights snapshot
ID, following the same shape as the existing CredentialStore. It is
plumbed into tieredCatalog and GetFoundationModel through a
package-level resolver (SetWeightsResolver/currentWeightsResolver),
since both are called from gateway/bedrock.go's fixed-arity route
table, which is out of scope here.
A self-host model with no resolvable weights snapshot is not
servable and must not be advertised: tieredCatalog filters it out and
GetFoundationModel returns ResourceNotFoundException for it.
Replace the meta.llama3-70b-instruct-v1:0 self-host entry, which
needs roughly 140 GB of VRAM in fp16 and has never actually run, with
meta.llama3-2-1b-instruct-v1:0, a real AWS-published Bedrock model
ID. At bf16 its 1.24B params are roughly 2.5 GiB of weights, well
inside wattle's single RTX A1000 (8188 MiB); MinVRAMMiB is set to
4096 to budget headroom for CUDA context, activations and KV cache on
top of that. This is an estimate, not a measurement off wattle -
.7.5's launcher should correct it against a real nvidia-smi reading
once the model is actually running there.
lookupCatalogEntry, used by the untiered invoke/converse routing
path, is left unchanged and still ignores weights resolvability;
gating that path is out of scope for this bead.
* fix: separate weights resolve errors from not-found in ochre catalog
An internal fault (e.g. a JetStream outage) reaching the KV weights
resolver was being collapsed into the same "not servable" verdict as
a clean not-found: tieredCatalog would silently unadvertise every
self-host model, and GetFoundationModel would return
ResourceNotFoundException for a model that does exist but whose
resolver is temporarily broken. Both call sites now distinguish the
two: a resolve error is logged and surfaced as its own error (mapped
to ServerInternal at the AWS error boundary), and only a clean
not-resolvable result yields ResourceNotFoundException.
tieredCatalog's signature changes to return an error alongside the
entries so ListFoundationModels can propagate a resolve fault instead
of silently returning a thinned-out list.
Also raises the self-host entry's gpu-memory-utilization from 0.5 to
0.6 (roughly 4913 MiB of vLLM's own pool) and MinVRAMMiB from 4096 to
5120 to match: at 0.5, weights (~2.5 GiB) left under 1.5 GiB for KV
cache and activations at max-model-len=8192, likely too tight for any
real concurrency. The two numbers are meant to agree: MinVRAMMiB is
the admission-gate floor, gpu-memory-utilization is what tells vLLM
to actually claim that much on an 8188 MiB device.
Comments trimmed to the repo's 3-line limit and stripped of plan/bead
references; the VRAM reasoning and model-choice rationale live in the
commit body and the plan doc instead.
Threading the weights resolver through as an explicit parameter
(mirroring CredentialResolver) rather than a package-level global was
considered per review. Doing so needs edits to gateway/bedrock.go
(bedrockRouteHandler's signature, both route closures, the
Bedrock_Request call site, a new bedrockWeightsResolver() method) and
gateway/gateway.go (a new BedrockWeights field on GatewayConfig).
Both are outside gateway/bedrock/ and are explicitly out of scope for
this change, owned by a parallel workstream already modifying the
same route table. The global resolver stays, now with the
error-handling fix above.
* fix: stop e2e GetFoundationModel regressing on the new weights gate
TestGetFoundationModel's "known model" subtest asserted on
meta.llama3-2-1b-instruct-v1:0 unconditionally, but that entry is now
gated on a weights snapshot staged in the bedrock-weights KV bucket.
On any real deployment that bucket starts empty, so the subtest would
fail with ResourceNotFoundException outside of OCHRE_E2E_SELFHOST.
The "known model" assertion now targets the Anthropic entry, which
resolves unconditionally and keeps testing what the subtest was
actually there for: a known model round-trips, an unknown one is
ResourceNotFoundException. The self-host assertion moves to a new
TestGetFoundationModelSelfHost, gated behind OCHRE_E2E_SELFHOST=1
alongside TestConverseSelfHost, where weights are expected staged.
tests/integration/bedrock_test.go's TestListFoundationModels already
wires a weights-resolver stub before asserting the self-host entry is
present, so it does not regress the same way.
* test: correct a stale assertion message on the ochre catalog weights gate
* match new predastore config surface area
* fix(migrate): carry existing installs onto the predastore v5 topology
The cluster topology refactor breaks an install three ways at once. The old
config carries a top-level host key, which the new schema binds to the [[host]]
table array, so TOML rejects the document and the daemon dies before it reads
any topology. Node state moved from distributed/{db,nodes}/node-N to a per-host
data directory keyed by a node id that is now unique across roles. And because
object metadata records which node holds each shard, the shard nodes had to
keep their ids, so the state replicas took new ones and every replica's
persisted raft configuration stopped listing it.
predastore.toml gains a 4 to 5 migration covering all three: it rewrites the
tables, relocates whichever node directories exist on this machine, and
recovers each relocated state replica onto the new member set. The rewrite is
line-oriented rather than a re-render from the template, so buckets, auth,
rate limits and iam settings an operator edited survive.
spinifex.toml gains a 3 to 4 migration renaming the predastore node_id to
host_id, which predastore-start.sh greps to select the host this machine runs.
The predastore migration lives in its own package rather than in migrate,
which every handler that stamps a KV bucket version imports; reaching the
cluster runtime from there would pull raft, badger and the QUIC client into all
of them. admin_upgrade.go blank-imports it, so only the binary that runs the
upgrade links it, and a test pins both registrations.
* test(predastore): cover the service runtime, gateway and shutdown paths
The service wrapper's new seams had no unit tests: buildRuntime, serve and
the errgroup Start runs the cluster and the S3 gateway under. CI's diff
coverage gate measured the PR at 34.6% against a 60% threshold as a result.
Cover every buildRuntime error wrap and its host selection, both ways the
gateway can fail serve, and the full lifecycle: Start brings the nodes and
the gateway up together, Shutdown drains both by cancelling the context
they share. Topologies are written per test as real config, with node ids
from a process-wide counter — the pipe transport's listener registry is
global, and this package's integration tests hold a fixture daemon on the
low ids for the life of the test binary.
setup.sh aborts twice on resolute. qemu-kvm is a virtual package with two
providers there, so apt refuses to resolve it; the token was redundant anyway
since $QEMU_PACKAGES already installs qemu-system-x86/arm and spinifex execs
the qemu-system-* binaries directly.
sudo-rs, Ubuntu's default sudo since 25.10, then rejects the sudoers file: it
implements no pam_session Defaults setting, and forbids a wildcard inside a
command argument. pam_session is now emitted only for classic sudo, and the
sysctl grant names a fixed-verb helper instead of a wildcard.
The helper is narrower than the rule it replaces: it accepts one interface, a
key restricted to rp_filter or accept_local, and a value restricted to 0 or 1,
where the old grant permitted any net.ipv4.conf key at any value. Granting
sysctl with an unconstrained argument was rejected as root-equivalent.
Adds scripts/lint-apt-packages.sh and an Apt Packages workflow so the runtime
package list is resolved against each supported suite's real indices in CI.
Spinifex rejected HttpTokens=optional and served 401 to every untokened
GET, which is stricter than AWS, whose own default is optional. That made
Windows guests impossible: cloudbase-init is the only realistic Windows
bootstrap agent and its EC2Service has no IMDSv2 token support in any
released version, so a Windows instance never received its hostname,
injected password or user-data.
HttpTokens is now settable at launch and via ModifyInstanceMetadataOptions,
accepting optional or required. The launch default stays required, so an
instance that does not ask for IMDSv1 never gets it. Every other
metadata-option field keeps its existing rejection.
The untokened path is reachable without credentials, so the per-instance
lookup is cached with a short TTL; without that a guest could drive
unbounded DescribeInstances fan-outs. A valid token short-circuits before
the lookup, so compliant IMDSv2 clients are unaffected.
* feat: spread HA control-plane placement across distinct AZs first
SchedulableHosts now orders eligible nodes by AZ round-robin (one host
per AZ before repeating any AZ) instead of raw fan-out arrival order.
ReserveSpreadNodes takes a strict order-preserving prefix of that list,
so an N-way HA reservation lands on N distinct AZs whenever that many
exist, degrading to today's node-only spread when AZ data is uniform
or absent. No change to the placement-group wire format.
* feat: honour ClientRequestToken idempotency for EKS CreateCluster
Mirrors the EC2 RunInstances ClientToken pattern with a TTL JetStream
KV bucket scoped per EKSServiceImpl instance. Same token and identical
params replays the in-progress/created cluster; same token with
different params returns IdempotentParameterMismatch; a distinct token
reusing the same cluster name still hits the existing atomic
name-claim and returns ResourceInUse.
* style: trim EKS spread comments to the 3-line max and drop a bead reference
* fix: bump goleak past the elided-frames stack-parse panic
goleak v1.3.0's stack parser treats a "...N frames elided..." line as
a function name and panics when it can't parse one. Go elides frames
on deep/recursive stacks, which is rare but load-dependent, so this
surfaced as an intermittent panic inside goleak itself (not a real
leak) during TestDrainVolume_DispatchedGoroutineDoesNotLeak under a
full parallel preflight run. Upstream fixed this in goleak@b62053b,
unreleased as a tagged version, so this pins to the latest commit.
* test: guard goleak against panicking on elided-frame stacks
* style: trim goleak regression test comments to the 3-line max
* feat: add segscan storage oracle for the real-predastore integration tier
Adds spinifex/testutil/segscanoracle, which copies a fixture node data dir
and execs the mulga umbrella repo's scripts/segscan (a separate Go module
that cannot be imported) to assert live/dead-tombstoned/dead-orphan byte
accounting against real predastore writes. Kept off the default PR path: a
new segscanoracle build tag excludes the test from make test-integration,
and a separate segscan-oracle.yml workflow (push to main/dev + manual
dispatch only, mirroring e2e.yml's non-gating trigger) checks out
scripts/segscan rather than adding that checkout cost to spinifex.yml.
* test: cover segscanoracle's pure logic to satisfy diff coverage
segscanoracle has no build tag and always compiles into the default
build, but nothing in that build exercised it, so every new line
counted as uncovered against the 60% diff-coverage gate. Adds unit
tests for JSON decoding (including the malformed-payload path), the
Locate resolution order, and the directory-copy routine, plus a
throwaway stand-in module (not the real scripts/segscan) to exercise
build+exec+decode end to end.
Pulls decodeReport out of Run and resolveSegscanDir out of Locate so
the pure logic is testable without a *testing.T or a real binary.
* feat: accept and surface gp3 Throughput on CreateVolume/DescribeVolumes
Honors input.Throughput (125-1000 MiB/s, default 125 when omitted) on
CreateVolume, persists it to VolumeMetadata, and surfaces it via
DescribeVolumes (getVolumeByID), mirroring the Iops handling shipped
in mulga-siv-380. Pins viperblock to the commit that adds the
VolumeMetadata.Throughput field.
Zero Throughput on a read volume means it predates this field; it is
omitted from the response rather than surfaced as a misleading 0
MiB/s, matching the existing IOPS convention.
* chore: pin viperblock to the merged gp3 Throughput commit
* feat: close IMDS and vm_adapters trace-linking gaps
Wrap the IMDS mux with otelsetup.HTTPMiddleware (outermost, ahead of
rejectForwarded) so guest metadata requests parent their DescribeInstances/
Attribute NATS fan-out instead of rooting a trace per call, including
requests the SSRF guard rejects.
Route daemon/vm_adapters.go's ebs.mount/ebs.unmount NATS requests through a
new ebsRequestWithTrace helper that opens a client span and injects it into
the message headers, so viperblockd's consumer spans join the producer trace
instead of rooting their own.
* feat: trace the ebs.delete request from the instance cleaner
* fix: preserve running guests across target restart, harden state write
spinifex-shutdown.service's ExecStop fired on any spinifex.target stop,
including the stop half of `systemctl restart spinifex.target`, draining
every guest via QMP even though the host never went down. Gate the drain
behind `systemctl is-system-running` == "stopping", which only holds once
systemd is genuinely unwinding into shutdown.target (reboot/poweroff), so
a target restart or plain `systemctl stop` now leaves QEMU/nbdkit running
(KillMode=process) for the daemon to reconnect to on the way back up.
Daemon.WriteState returned as soon as the local state file write failed,
skipping the independent JetStream KV write entirely. A crash in that
window left a live VM with no durable record anywhere. Attempt the KV
write regardless of local disk health, then still surface the local
write error to callers, mirroring finalizeTerminated's handling of the
same fault class on the terminate path.
* fix: gate the shutdown drain via an spx flag, not a shell script
Replace the shutdown-drain-gate.sh gate with a --only-if-host-stopping
flag on `spx admin node drain`. The unit now points at /usr/local/bin/spx
directly (already required for the node to function) instead of a script
installed under the data directory, which a partial rebuild of that
directory could silently drop without ever failing ExecStop. The gate
logic also becomes a plain Go function with injectable state lookup, so
it gets real table-driven coverage instead of only a live reboot.
hostIsStopping() drains only on a literal "stopping" from `systemctl
is-system-running`. Every other recognized state (running/degraded/...)
and any unrecognized output skip the drain; a fully unreachable systemctl
fails toward draining, since a skipped drain on a real shutdown hard-kills
guests and loses unflushed writes, while a spurious drain on a restart
only costs an extra graceful stop/relaunch cycle. Run by hand without the
flag, the command still drains unconditionally as it always has.
* refactor: split the shutdown-gate comments to sit with the code they explain
* fix(vm): durably record terminate even when local state persistence fails
finalizeTerminated bailed out of WriteTerminatedInstance whenever the
transition to terminated hit a persistence-only failure (e.g. ENOSPC writing
local state), even though the in-memory status had already reached
terminated. That left the instance owned by nothing: StuckTerminateReaper
skips it (no longer shutting-down) and TerminatedTeardownReaper never sees
it (never reached the terminated KV bucket). Only an operator daemon restart
recovered it.
WriteTerminatedInstance is a JetStream KV write, independent of local disk
space, so keep going into it whenever the in-memory status did reach
terminated, distinguishing that case from a genuine invalid/raced
transition. Once the disk has space again, teardown now completes on its
own via the existing idempotent TerminatedTeardownReaper.
* fix(systemd): cap per-service memory under spinifex-system.slice
Every service under spinifex.slice ran with MemoryMax=infinity /
MemoryHigh=infinity, so one runaway service could exhaust the whole slice
budget and the slice-level OOM would then reap an arbitrary victim instead
of the offender.
Add per-unit MemoryHigh/MemoryMax as percentages of physical RAM (so the
same units scale from a dev box to bare metal). spinifex-daemon.service
still hosts every guest QEMU process in its own cgroup (no per-instance
slice yet), so it gets by far the largest allowance, just under the slice's
own 80%/90%, to avoid starving VM launches. predastore is the legitimately
memory-hungry service and gets generous headroom above its idle footprint.
The remaining infra services are small, single-purpose Go processes and get
a shared conservative bound.
* fix(viperblockd): remove local volume directory on ebs.delete
DeleteVolume deleted the S3 data and notified viperblockd, but the ebs.delete
handler only stopped the volume's goroutines, killed nbdkit, and removed the
NBD socket — it never removed the local WAL/checkpoint cache directory under
BaseDir. The only path that removed it (VB.Close's RemoveLocalFiles) only
ran from the unmount handler's seal step, which -efi volumes never reach
(they carry no durable data to seal) and which a main volume also skips
whenever a prior seal failed (e.g. disk full, deliberately keeping local
files for recovery). Since terminate always unmounts before deleting, and
delete is permanent, retention had no remaining purpose by the time
ebs.delete ran.
Unconditionally remove BaseDir/<volume> in the ebs.delete handler regardless
of mount-tracking state, closing the leak for both the root volume and its
-efi companion.
* fix(viperblockd): validate volume names before removing local state
ebs.delete unmarshalled the volume name straight off the wire and
handed it to os.RemoveAll: an empty name collapsed to BaseDir itself,
and ".." components could escape it entirely. localVolumeDir now
rejects an empty/invalid name or one that resolves outside BaseDir
before any removal, and ebs.delete returns Success: false with an
error instead of silently skipping. Applied the same gate to
ebs.mount and ebs.config, the other two handlers that turn a
wire-supplied volume name into a BaseDir-relative path.
* chore: pin viperblock to the merged concurrent-open corruption fix
The ansible build role builds with GOWORK=off, so the deployed binary links
whatever go.mod pins rather than the go.work overlay. Move the pin to the
revision carrying the per-volume flock and the WAL/state write fixes.
* fix(vpc): close the route-table and ENI teardown leaks
DeleteVpc validated subnets and non-default security groups but never
non-main route tables, so it deleted the VPC out from under them and left
records naming a VPC that no longer resolves. It now returns
DependencyViolation, alongside the existing checks and before any cascade,
and still cascades the main route table on its own.
DeleteSubnet left route table associations naming the deleted subnet.
DeleteRouteTable blocks on any non-main association carrying a subnet ID,
and that subnet could never be deleted again to clear it, so the table was
pinned forever. Associations are now cleared before the subnet record goes,
leaving a mid-way failure retryable. routetable imports vpc, so the record
cannot be typed here; only the associations key is rewritten, and every
other field is passed through untouched.
DeleteLoadBalancer detached an ENI and then re-read it to delete, where a
lagging R3 replica could report it still in use. The failure was logged at
WARN and the LB record deleted anyway, orphaning the ENI and its security
group. DetachAndDeleteENI now reads once and carries the detach revision
into the delete via LastRevision, retrying the whole flow from a fresh read
on a conflict, bounded at 3 attempts. The IPAM release happens after the
delete, not before, so a retry cannot hand the same IP back twice. When
ENIs survive, the LB record is kept naming them and the call returns
DependencyViolation rather than dropping the only reference to the leak.
Instance termination routes through the same call, and both ENI paths now
distinguish a real delete from an already-absent one instead of logging a
deletion that did not happen.
TestRLC5_NoOrphanedRouteTableAfterSubnetThenVpcTeardown covers the ordering
that defeated the per-operation tests: subnet first, then VPC.
* fix(elbv2): route SetSubnets ENI teardown through the atomic detach+delete
SetSubnets detached every ENI in one loop, then deleted the removed subnets'
ENIs with force=false in another. That is the same shape as the
DeleteLoadBalancer leak: the delete re-reads the ENI, a lagging replica can
still show it attached, and the delete fails as in-use while the LB moves on
without it.
Removed subnets now go through DetachAndDeleteENI, which carries the detach
revision into the delete under one read. ENIs that survive the reshape are
still detached on their own, since they are re-attached to the relaunched VM.
* fix(gateway): log the two SigV4 paths that reject silently
Three code paths answer SignatureDoesNotMatch, but only the Verify mismatch
logged anything. A skewed request timestamp and an unsupported service in the
credential scope both returned the same wire error with no log line at all, so
they were indistinguishable from a canonicalisation defect after the fact.
Both now log at Warn. The skew path records the client and server timestamps
plus the window, which is what separates clock drift from a signing bug.
Adds signingTime, since Parse rejects a skewed request before returning a
SignedRequest, leaving the raw header or presigned query arg as the only copy
of the timestamp available to log.
* fix(gateway): log the malformed-envelope rejection too
The parse-error default branch answers IncompleteSignature and was the last
auth rejection path with no log line, so a malformed Authorization header,
unsupported algorithm or missing content hash left no record of which
validation stage rejected the request. The parse error names it.
shutdownVolumes deliberately spares nbdkit that a guest is still writing
through, because tearing one out mid-write corrupts that guest's filesystem.
The unit left KillMode at the default, so the cgroup SIGTERMed those children
anyway and SIGKILLed them at TimeoutStopSec -- causing both a 30s stop stall
and the corruption the skip exists to prevent.
KillMode=process matches spinifex-daemon, where guest QEMU already survives a
restart for DDIL reattach. Idle nbdkit is still reaped explicitly.
Measured on env12 with a guest attached: control-group stopped in 30.08s with
'Failed with result timeout' and SIGKILLed nbdkit, leaving QEMU running with
zero ESTAB nbd sockets. KillMode=process stops sub-second, both nbdkit
survive, and the guest keeps its device.
* fix(ec2): drain the attached volume before CreateImage reads a live checkpoint
CreateImage against a running instance (--no-reboot) read the volume's
checkpoint directly from predastore without draining in-flight writes
first, unlike CreateSnapshot. A checkpoint captured before a guest-
triggered write (e.g. growpart rewriting the GPT after a root volume
is launched larger than the base image) could be stale, producing an
AMI that boots to a UEFI shell instead of the OS.
Extracts the existing snapshot-package drain logic into a shared,
reusable DrainVolume so both live-snapshot call sites route through
the same drain-before-read guarantee instead of being able to diverge.
* test(e2e): boot-verify CreateImage AMIs instead of trusting State=available
describe-images reporting State=available is not evidence an AMI
boots — a well-formed but unbootable AMI (stuck at the UEFI shell)
still registers cleanly. Extend the CreateImage e2e test to launch a
probe instance from each produced AMI (running-path and stopped-path)
and require a real SSH login, with console-dump-on-failure wired up
for diagnosis.
* fix(images): sparsify the Alpine guestfish .raw before conversion
The Alpine build path grows the qcow2 in place via qemu-img resize +
guestfish resize2fs, which can leave metadata scattered across the
newly-grown region. virt-sparsify zeroes the guest's unused blocks
before the raw conversion so they collapse into real holes, matching
the Ubuntu virt-resize path's sparseness.
DescribeInstances silently returned an empty Reservations list for an
instance ID that never existed, so callers could not distinguish "gone"
from "never existed". Mirror the two-pass found-set pattern from
DescribeSecurityGroups, but only assert absence when the NATS fan-out and
both stopped/terminated KV bucket queries are provably complete — a
node timing out during the fan-out must never produce a false NotFound
for an instance that actually exists on that node.
The assertion is split into a new DescribeInstancesChecked, used only by
the gateway's customer-facing action. The plain DescribeInstances stays
silent for internal callers (IMDS instance lookup, the RDS instance-state
resolver, the EKS control-plane reconciler) that treat "not found" as
"not currently present" rather than a fault.
* fix: stop passing nbdkit S3 credentials as literal argv
access_key/secret_key were interpolated into the nbdkit plugin argv,
which is world-readable via /proc/<pid>/cmdline for the life of the
process. Credentials now travel via cmd.Env (VB_ACCESS_KEY/
VB_SECRET_KEY) instead. Requires a viperblock nbdkit plugin build that
reads these environment variables; inert against an older plugin
until that pin moves.
* chore: pin viperblock to the VB_ACCESS_KEY/VB_SECRET_KEY fallback
The spinifex half sets these env vars on the nbdkit child, but the
deploy builds with GOWORK=off, so viperblock's matching read of them
only ships once go.mod points at 8a4787a.
* fix(lbagent): stop heartbeating a load balancer once it is confirmed deleted
A LoadBalancerNotFound response is the gateway's positive confirmation that
the store has no record for the LB, not a connectivity failure — unlike a
dial error, timeout, or 5xx, which leave its existence unknown and must keep
retrying. Past the startup registration grace window (or after any prior
successful heartbeat), it can no longer be the record's own replication
race, so the agent now stops rather than retrying forever and filling the
log with heartbeat noise after teardown.
* fix(nats): enable the JetStream ENOSPC-latch watchdog timer
nats-server disables JetStream process-wide on ENOSPC (server/jetstream.go
handleOutOfSpace -> ShutdownJetStream) and never re-enables it itself —
that decision is nats-server's own internal state, not spinifex's, so a
restart is the only way back. The spinifex-nats-watchdog.timer + service
already probe /healthz?js-enabled-only=true and try-restart on a sustained
failure, but the timer's WantedBy=timers.target never took effect: neither
setup.sh nor firstboot enabled it, so the watchdog shipped inert and a full
disk needed a manual restart forever. Enable --now it alongside the rest of
the unit set.
* fix: bound EKS DELETING re-drive with backoff and a terminal give-up
reapCluster re-drove purgeClusterInfra on every GC tick with no attempt
tracking, so a permanently-failing teardown (e.g. an unretriable
DependencyViolation) was retried forever. Track attempt count and last
attempt time on ClusterMeta, back off exponentially between re-drives,
and give up after maxDeleteReapAttempts with a single error-level log.
Status stays DELETING on give-up so the cluster's infra remains tracked
and billable pending operator intervention, preserving the ADR-0006 §6
billable-before-sweep invariant.
* fix: create the EKS e2e node role independently in each subtest
runEBSCSIVolume string-built the worker node role ARN without ever
creating it, only passing because runIRSAPod's t.Cleanup discarded
DeleteRole's error while the role was still attached to its instance
profile, leaving it behind for EBSCSIVolume to reuse by naming
convention. Extract ensureNodeRole and call it independently from both
subtests, detaching the role from its instance profile before deleting
it and asserting that error instead of discarding it.
* fix: reclaim CSI-provisioned EBS volumes on EKS cluster delete
purgeClusterInfra tore down the control plane, nodegroups, and
networking but never looked at EBS volumes, so any volume the
in-cluster CSI driver had provisioned for a PVC was left behind
untracked and unbilled for. Add a report-only reclaim step that scans
for volumes tagged with the cluster's CSI owner tag and logs them for
operator follow-up.
Volumes are never auto-deleted: a PV's reclaim policy (Delete vs
Retain) is a Kubernetes-side property the CSI driver does not tag onto
the EC2 volume, so there is no safe way to tell from tags alone
whether deleting a given volume is intended. This mirrors the
volume-leak reaper's existing mark-and-alarm policy for the same
reason.
The cluster-owner tag requires the EBS CSI controller's
--k8s-tag-cluster-id flag, added here to the controller manifest;
without it a CSI volume carries no cluster-scoped tag and reclaim
would have to guess across every cluster sharing the account.
* fix: persist the last purge error on a wedged DELETING cluster
DeleteReapExhausted told an operator a cluster's teardown had given
up, but not why: nothing recorded the failing purge's error.
Persist it on every failed re-drive, not only once the reaper
exhausts its attempts, so it is visible on a still-retrying cluster
too. Status stays DELETING throughout, preserving the billing
invariant that untorn-down infra remains tracked.
* fix: make the EKS e2e suite confirm its CSI volume is actually deleted
The EBSCSIVolume subtest deleted its PVC on cleanup but never
confirmed the underlying volume was reclaimed, so a CSI driver that
silently failed to delete it would leak a volume with nobody
noticing. The suite provisions and owns this volume, so unlike
production teardown it can safely wait on its StorageClass's Delete
reclaim policy: capture the CSI volumeHandle while the PVC is bound,
then have cleanup poll EC2 for that exact volume to disappear before
the addon/nodegroup that run the CSI controller are torn down.
* test: gate scripts/images/ shell suites + shellcheck in CI
The eks-node/rds-postgres *_test.sh suites and shellcheck only ever ran
by hand: spinifex.yml filters pushes on Go-only paths and no job
invoked them, so a broken helper could land unnoticed. Add a
`make test-images` target and a path-filtered image-scripts.yml
workflow so both run automatically on scripts/images/** and images/**
changes, kept separate from preflight so image-asset churn doesn't run
on every Go contributor's commit. Also clears the shellcheck findings
that would have started this gate red (SC1090 on a runtime-resolved
source path, SC2154 x2 on eval-assigned variables shellcheck can't trace).
* test: pin vCPU quota gate ordering in RequestSpotInstances
Nothing asserted that RequestSpotInstances runs the vCPU quota gate
before dispatching capacity or launching a VM, so a regression that
dropped or reordered EnforceLaunch would pass silently. Add an
ordering test using a real quota.Service (embedded JetStream KV) and
real NATS responders standing in for the capacity and launch daemons,
recording a global call sequence and asserting the gate's index
precedes both. Verified by removing the gate: the first quota-KV read
then comes from the post-launch charge instead, landing after capacity
dispatch and launch, and the ordering assertion fails as expected.
* fix(systemd): give every unit an explicit TimeoutStopSec
9 of 12 spinifex systemd units had no TimeoutStopSec and inherited
systemd's 90s default, which is either far more or far less time than
their own SIGTERM handling needs. Size each one to what its shutdown
path actually does: predastore matches viperblock's 30s WAL-drain
budget since viperblock flushes through it (After= already sequences
viperblock's stop first; this is the backstop if that's ever bypassed),
spinifex-ui covers its 30s internal shutdown context plus margin for
long-lived streaming proxy connections, nats covers its own
JetStream-shutdown/raft-stepdown/client-drain signal handler, and
services with no SIGTERM handler at all (awsgw) get a short bound
since they exit immediately regardless.
Extends the existing unit-file invariant test with a check that every
.service file declares TimeoutStopSec, so a newly added unit that
forgets one fails CI instead of silently falling back to 90s.
* fix(vm): reconcile QEMU processes with no instance record on restore
classifyRestoredInstances only walks Manager.Snapshot(), so a
qemu-system process with no instance record at all -- a crash
mid-launch, a lost KV write, a stale process surviving a wiped state
file -- was never handed to killOrphanedQEMU and stayed permanently
invisible to recovery.
Add a host-process scan that diffs live qemu-system PIDs (matched via
/proc/<pid>/comm) against the PIDs claimed by known instances' own
-pidfile files. It only reports what it finds, never kills: this
daemon cannot distinguish "record not loaded yet" or "owned by another
daemon/tenant on this host" from a genuine orphan, and killing a
running customer VM to fix an invisibility gap is the wrong trade.
Runs at the end of Restore, once every known instance has been
classified and relaunched, so a live process still unaccounted for at
that point genuinely has no record anywhere.
* fix(vm): distinguish recordless QEMU orphans by pidfile ownership
A recordless qemu-system process with a stale spinifex pidfile is
positively this daemon's own orphan; one with no pidfile at all could
belong to another tenant on a shared host. Log the two cases distinctly
instead of one blanket warning, so an operator can tell them apart.
Also sandbox XDG_RUNTIME_DIR in the shared daemon test harness so
Restore()-driven tests never read this box's real runtime directory.
* fix(spinifexui): treat ErrServerClosed as a clean shutdown, not a failure
Serve returns http.ErrServerClosed after Shutdown is called deliberately
(the SIGTERM handler above does exactly that), but Start propagated it
as an error, making systemd log a graceful stop as exit code 1 and
Result: exit-code -- indistinguishable from a real crash.
Query-protocol services (EC2, IAM, STS...) get their action from the
SigV4 Action= form param before dispatch, so traceActionEnricher can
rename the span and set aws.action/aws.service from that context.
REST-JSON services (EKS, ECS, ACM, tagging, ECR, bedrock...) only
resolve their action inside their own dispatcher, well after that
middleware runs, so the enrichment never fired for them.
checkPolicyResource is the one call every dispatcher already routes
its resolved (service, action) pair through, so move the enrichment
there via a shared recordResolvedAction helper instead of duplicating
it per REST-JSON handler.
BytesFreed counted logical object sizes deleted from the S3 namespace,
but predastore only reclaims that disk space later via background
compaction. The CLI printed "freed X GiB" immediately on delete, which
made a compaction outage that never reclaimed the space invisible.
Rename to BytesDeleted and reword the CLI output to say the bytes are
marked for deletion and reclaimed by background compaction, not freed
immediately.
* fix(ec2): rename ambiguous 'code' slog key to firmwarePath
The EFI variable store prep log used 'code' for the OVMF firmware
path, which reads as an error/status code in log search. Rename to
firmwarePath to avoid confusion.
* fix(spinifexui): write pid file to a service-owned BaseDir
spinifexui.Start() called the legacy utils.WritePidFile, which has no
MkdirAll and falls back to $HOME/spinifex. Under the spinifex-ui unit's
ProtectHome=yes this directory is neither guaranteed to exist nor
writable. Add Config.BaseDir, switch to utils.WritePidFileTo (which
does MkdirAll) for Start/Stop/Status, wire it through the CLI as a
spinifex-ui-base-dir flag/SPINIFEX_UI_BASE_DIR env var, provision
/var/lib/spinifex/spinifex-ui in setup.sh, and add it to the unit's
ReadWritePaths.
* test(viperblockd): replace fixed sleep with a readiness poll
TestEBSConfigQueueGroup_DetachedOpenFails slept 500ms before assuming
launchService's ebs.config subscriber was bound, then raced a hardcoded
5s request deadline. Poll with a cheap malformed request (rejected at
json.Unmarshal, before the slow S3 path) until the subscriber answers,
then issue the real request with room for openVolumeVB's S3 backoff.
* fix(e2e): capture GPU worker console log on test failure
Wire harness.OnFailure into TestEKSGPUPodExposure to dump the GPU
worker's serial console via the established EC2 GetConsoleOutput path,
matching lb_test.go's captureLBConsoleOnFailure. A driver or
device-plugin hang after boot only ever surfaces on the guest console.
* fix(ec2): render empty Describe result containers and add requestId
xmlutil.BuildXML omits a nil slice's XML container entirely but renders
an empty one for a non-nil empty slice, so every EC2 Describe with zero
results (DescribeKeyPairs, DescribeVolumes, DescribeInstances, etc.)
came back as a bare "<FooResponse></FooResponse>" instead of AWS's
"<FooResponse><requestId/><fooSet/></FooResponse>". botocore parses that
body as having no result key at all, so the CLI silently prints nothing
and exits 0 — this broke the ansible key-import guard that gates
cluster bootstrap on describe-key-pairs' output.
Fix normalizes nil result slices to empty ones and synthesizes a
requestId in the single marshal chokepoint shared by every EC2 action
(marshalEC2Response, used by both ec2Handler and ec2HandlerWithReq),
rather than patching each handler individually.
* fix(ec2): return InvalidKeyPair.NotFound / InvalidSnapshot.NotFound for missing explicit IDs
DescribeKeyPairs and DescribeSnapshots silently returned an empty list
when the caller named a specific, nonexistent KeyName/KeyPairId or
SnapshotId, instead of erroring like real AWS (and like the existing
DescribeSecurityGroups/DescribeImages/DescribeVolumes/DescribeSubnets
handlers already do). botocore treats the empty-list response as
success, so ansible's cluster-post key-import gate — which checks
`describe-key-pairs --key-names spinifex-key` for a nonzero exit code —
never fires, and spinifex-key is never imported during bootstrap.
Mirrors the two-pass found-set pattern already used by
DescribeSecurityGroups: build the set of names/IDs actually present in
the filtered results, then error if any explicitly requested name/ID
is absent. An unfiltered list or a --filters query that matches
nothing is unaffected and still returns an empty list with no error.
DescribeInstances has the same gap but needs a design decision on how
a NotFound check interacts with the fan-out's existing partial-result
leniency, so it is left for separate follow-up.
TanStack router-plugin <1.168.21 sorted routes by coercing route nodes
to strings, so output order followed filesystem-walk order and flipped
depending on which generator version a contributor's node_modules
resolved. Pin the version so it cannot drift, commit the sorted output
from the fixed generator, and fail CI if a build produces a different
route tree than the one committed.
RegisterTaskDefinition let any caller with ecs:RegisterTaskDefinition name
an arbitrary taskRoleArn/executionRoleArn, and RunTask/StartTask/CreateService/
UpdateService could launch tasks under a previously registered definition's
roles, all without holding iam:PassRole on those roles. Mirror the EC2
gateway's PassRoleChecker closure pattern to close both paths.
ClaimLBName's crash-orphan reclaim treated any owner with no persisted
LB record as abandoned, but a legitimate in-flight create has no record
yet either during the gap between claiming the name and PutLoadBalancer.
Two concurrent creates for the same name could both win.
Track a claim timestamp and only allow reclaim once it exceeds a TTL,
so a fresh in-flight claim blocks a second claimer like a live LB would,
while a claim abandoned by a genuine crash is still reclaimed.
Josh Siviour