The RDS customer endpoint ENI is configured statically in the guest, but OVN still attached dhcpv4_options to its logical switch port. The guest's cloud-init dhcpcd then took an OVN DHCP lease on that NIC and flushed the static address at the 3600s lease expiry, roughly an hour after every boot. Thread a SuppressDHCP flag from the customer-ENI creation through the ENI record, port event, and PortSpec, and gate the dhcpv4_options assignment at both create sites on it. The primary NIC keeps DHCP for IMDS; a delete and recreate never re-adds DHCP because the reconciler intent carries the flag too.
Multi-node install
- Lead with the fact that the guide builds a three-node cluster, and say
how to go beyond three. The callout sits inside ## Overview because the
docs site discards everything above the first section heading.
- Replace the OVN-only sizing prose with a table covering all three
distributed layers, and move the OVN control-plane detail into the VPC
networking guide, linked from here.
- Add minimum and recommended hardware.
- Move the object storage layout to the end as background; it was the
second thing a reader hit.
- Collapse the ISO conversion notes from a section into one skippable
box, now that --force is unconditional in Step 4 and the formation
port needs no manual intervention.
- Expand verification with spx get nodes and spx top nodes, including
what to look for in each.
- Drop the install-node.sh shortcut (internal tooling) and the
"adding a node later" section (no good story yet).
Corrections found while writing
- Servers 4+ were described as running no database. They run predastore
and viperblock like every other node; only the OVN database cluster is
capped at three. Fixed in four places.
- Step 4 never needed the firewall down: spx admin init opens the
formation port itself for the formation window. Only Step 3's
peer-scoped OVN ports are affected.
- `nft list table inet spinifex_filter | grep spinifex_peers` could never
match — the peer list is an nft define, expanded at load time. Anyone
following it would have seen empty output and assumed the worst.
- Disambiguate "database": with predastore, NATS and OVN all in play it
needed qualifying throughout.
- Align the example output region with the AWS_REGION the guide exports.
Cross-doc links
- Rewrite relative and category-prefixed links to the canonical
/docs/<slug> form. The site publishes docs flat, so ../foo/README.md
and /docs/category/foo both 404.
- Point the six links to unpublished docs at the source repo instead.
- Fix four fragments written against GitHub's anchor algorithm.
* feat: implement EC2 MonitorInstances and UnmonitorInstances
The telemetry collector already supports both EC2 monitoring tiers and
already switches a running poller between them when its discovery file
changes, but nothing could make that change: Monitoring.Enabled was only
ever written at launch, and both API actions returned InvalidAction.
Route through the owning daemon over ec2.cmd.<id> rather than the
spinifex-workers queue group, so an instance running on any node is
reachable; no responders falls back to the shared stopped store, where the
write is a CAS so a racing start-claim cannot resurrect a deleted record.
Also report Monitoring on DescribeInstances. It is derived in
ProjectInstance from the same field the collector's period reads, so a
describe cannot disagree with the tier actually being collected and
instances launched before this change report disabled without a backfill.
* fix: wake the telemetry poller when its monitoring tier changes
poller.run re-read its period inside the tick loop, so a reset waited out
the interval being replaced: moving a 300s instance to the detailed tier
took up to five minutes to have any effect, which reads as the API call
not working. Found verifying monitor-instances on env13.
updateMeta now signals a buffered retier channel when PeriodSeconds moves
and run selects on it alongside the ticker. The signal only resets the
ticker rather than forcing a tick, so a published delta still spans the
period its batch declares. The send is non-blocking and the channel
coalesces, so a slow poller cannot stall the reconcile loop.
* fix: join the daemon host-port ENI to the appliance security group
The customer ENI's self-referencing SG drops all ingress except its own
members, so the daemon's host-port ENI could open TCP but never complete a
postgres handshake. Carry the appliance's security groups onto the daemon ENI
at create, and re-associate an already-created one in place.
* fix: complete teardown GC when a volume's metadata doc is already gone
DeleteVolumeOnTerminate and DetachVolumeOnTerminate propagated the
NoSuchKey error from a missing volume metadata doc as a hard failure, so
the teardown reaper backed off forever and never purged the terminated
record. A missing metadata doc means the volume is already gone, so treat
it as success via objectstore.IsNoSuchKeyError at both call sites, with a
matching tolerance in the instance-cleaner adapter.
* fix: statically address the RDS customer ENI so it survives the DHCP lease
The customer ENI was configured via a one-shot udhcpc with no renewal
against OVN's 3600s lease, so the DB endpoint went address-less an hour
after every boot. Resolve the customer subnet's prefix and gateway at
launch and emit the ENI as a static NIC (CIDR + gateway, never the
default route), mirroring the mgmt NIC, so it no longer depends on the
lease being renewed. The guest static branch now installs the same
source-based return-path policy a non-default DHCP NIC gets.
* fix: delete the OVN logical switch port when an ENI is deleted
An ENI delete published vpc.delete-port fire-and-forget, so a lost or slow
message let the ENI record and its IP free through with the logical switch
port still in OVN. A recreate onto the same IP then had two ports carrying
it and the router latched the stale MAC, blackholing the datapath. Publish
the delete synchronously so the port is confirmed gone before reuse, and
make DeletePort idempotent so an already-absent port is a no-op.
Both the endpoint ENI and the management NIC carry the rds-db-instance tag,
so a first-match pick non-deterministically dialed the unreachable management
NIC. Select the endpoint ENI by its RDS description, falling back to any usable
tagged NIC.
The ELBv2 LB microVM boots with -m 128 (sys.micro). After kernel
reservations the 37.7 MiB initramfs — 45% of it the Go 1.27 lb-agent —
leaves under 512 KiB of headroom, so the guest panics with 'Initramfs
unpacking failed' before init and lb-agent never heartbeats.
Bump sys.micro to 256 MiB so the initramfs unpacks with margin, and add
an uncompressed-initramfs gate to build-microvm-image.sh (the compressed
gate was blind to the guest-RAM footprint) so a future bump fails the
build loudly instead of panicking in production.
A class-change ModifyDBInstance is the intended remedy for a failed DB
instance, but applyPendingModifications applied the parameter group through the
live instance agent first. A dead agent returned ErrCommandUnreachable and
aborted the modify before replaceInstanceVM could run, so the one recovery
lever the API offers could not run for the case that needs it most.
The replacement VM's fresh agent applies the resolved set itself on boot, so
the pre-replace apply against the old dead agent is redundant for a class
change. Tolerate ErrCommandUnreachable there: resolve the class-correct set
(a KV read, no agent), persist it for the fresh agent, and skip only the dead
agent's apply. Parameter-group-only modifies still fail on a dead agent.
* feat: gate self-host inference behind capacity-based ThrottlingException
Add a process-scoped concurrency limiter keyed by (servingAccountID,
modelID) with capacity = catalog MaxConcurrency x max(ModelUnits, 1),
and apply it to every self-host Converse/ConverseStream/InvokeModel/
InvokeModelWithResponseStream path. Over-capacity requests reject
immediately with ThrottlingException instead of queuing inside vLLM,
matching real Bedrock. Managed (Anthropic) providers are untouched.
* test: mark admission_test as in-package for the unexported limiter
* test: cover capacity admission gate and throttling
* feat: back up the ochre pgvector schema to object storage
Adds account-scoped pg_dump/restore of the per-account kb_<accountID>
schema to predastore, gzip-compressed and keyed under the account's
own prefix. Backup shells out to pg_dump (--no-owner --no-privileges
--clean --if-exists) behind a PgDumper seam; restore runs psql then
repairs the account role's grants via EnsureAccount/RegrantAccount
since the dump carries no ownership. Exposed as
`spx admin ochre vector backup/restore --account <id>`.
* test: cover ochre vector backup dump/restore and error paths
* feat: co-served GPU model bundles (catalog, lifecycle, embed/rerank routing)
Serve the generative LLM, embedder and reranker as one co-resident GPU
bundle: EMBEDDING/RERANK catalog modalities + co-serve group, bundle-aware
admission (one GPU claim sized to summed member VRAM), single-flight one-VM
launch with a member->port EndpointRecord, DynamicEndpointResolver member
routing. Ochrevector embed/rerank/token-introspection resolve through the
resolver (static single-endpoint map retired); rerank degrades to plain KNN on
absent/failure. Public InvokeModel-embeddings and top-level Rerank deferred.
* feat: co-serve vLLM+TEI bundle image with per-engine readiness
Replace the single-model vllm-serve unit with a generic bedrock-serve@
systemd template driven by per-member BEDROCK_* env files, dispatching to
vLLM or text-embeddings-router by engine and binding each member to its own
port and weights device. TEI serves directly on its serve port; readiness
probes each member on the route its engine answers (vLLM /v1/models, TEI
/health) with the engine family carried end-to-end from the catalog.
* fix: serve nomic embedder correctly in the co-serve bundle
Bound embeddings requests to the TEI client batch limit so a large
document's chunks are split across requests instead of 422ing. Serve
nomic with explicit mean pooling, since the flat weights layout omits
1_Pooling/config.json and TEI would otherwise default to CLS. Add
bge-base-en-v1.5 as a standalone alternative embedder.
* fix: make the bedrock reaper family-aware and pin-safe
A pinned endpoint is now exempt from terminate-and-relaunch on scrape
failure, mirroring how it is already exempt from idle reclaim, so a
transient unreachability can no longer tear down operator- or
commitment-managed serving.
A bundle with no generative vLLM member exposes no vLLM load metrics, so
it is liveness-probed on /health and kept warm rather than scraped for
idleness on a route it does not serve and reaped in a loop.
* test: lock in co-serve group-member convergence to one VM
* fix: pin self-host HF repo/revision in catalog and stop nested config flattening
Carry HFRepo/HFRevision per self-host catalog entry; weights pull defaults
--hf-repo and --revision from them so a bare pull cannot silently grab a
regressed upstream main (the nomic v5 config.json duplicate-field break).
Restrict fixed config filenames to the repo root so a nested config.json
(sentence-transformers 1_Pooling/) is dropped instead of flattening onto the
model config, and refuse any two source paths that collide at one dest key.
* fix: let endpoint delete clear an account-scoped pinned record and report honestly
A bare 'endpoint delete' resolved the Global key, which never held a pinned,
account-scoped record, then printed 'torn down' over the idempotent no-op while
the record survived. Add --account to the delete command so an operator can
target the account 'endpoint list' shows, and return DeleteEndpointOutput.Removed
so the CLI names a real teardown versus a lookup that found nothing and points
at --account. A gone VM already tolerates NotFound, so a scoped delete clears a
record whose instance was terminated out of band.
Real Bedrock emits contentBlockStart only for toolUse blocks; text flows
messageStart -> contentBlockDelta -> contentBlockStop -> messageStop. An
empty start:{} union violates Smithy's exactly-one-member rule and breaks
strict SDK stream consumers (boto3 errors, the AWS Go SDK stalls).
The drift loop tests launched DriftLoop with a bare go statement and never
joined it, so shrinkDriftTiming's cleanup wrote DriftInterval and
driftBackoffBase while the loop was still reading them in nextDriftBackoff.
Cleanups run LIFO, so registering the join after shrinkDriftTiming makes it
run first and closes the window.
Go 1.27's encoding/json v2 decoder stops at the closing brace and leaves
Encoder.Encode's trailing newline unread. net.Pipe is unbuffered and
synchronous, so the writer blocks on that undelivered byte while the peer
blocks writing its reply, deadlocking on the first QMP command.
TestTryBlockdevDel hung until the 180s package timeout. A real unix socket
is kernel-buffered and matches production QMP transport, which is already
the pattern newMockQMPClientWithEvents uses.
The replacement lease is persisted before releaseStaleUpstream runs, so
the KV predicate did not prove the release had completed and the count
assertion could observe zero under race-detector scheduling.
The RDS reconciler and the ECS scheduler call InitLeaderBucket on every
tick, and it created before attaching. CreateKeyValue against a bucket that
already exists is only a no-op when the stored config matches the requested
one; otherwise JetStream answers 10058 and the caller silently falls back to
an attach.
On a multi-node cluster it never matches. The stored buckets are replicated
while the RDS and ECS calls ask for the default replica count, so every tick
billed an error. That is the whole of the 14-18% JetStream API error rate
seen on env1 and env19, and it landed entirely on the meta leader because
that is the node that serves STREAM.CREATE. Single-node clusters matched and
so sat at 0%, which is why this never showed up in CI.
Attaching first removes the per-tick create outright, so it no longer matters
whether the configs agree. EKS already passed a replica count and so did not
error, but it made the same wasted call every tick and is changed with them;
its replica count now applies to a genuine first creation, which is the only
time it was ever read.
The regression test asserts on the API traffic rather than the return value.
Both orders reach the same handle, so idempotency cannot tell them apart, and
on a single-node test server the old order raised no error at all - it just
issued the create. Watching for the STREAM.CREATE advisory catches it there.
Adds per-account resolution of the [quota] limits from a KV override record,
an admin surface and CLI to set one, RDS and load-balancer dimensions, and an
e2e suite that drives every gate on single and multi node.
Also fixes external IPAM initialising without retry: a node that lost a race
against JetStream convergence silently served the disabled EIP stub for the
life of the process, answering cluster-wide with an empty address list.
Julian Sommer