The port tables were wrong in both directions: inter-node traffic is QUIC over
UDP rather than TCP, and the shard range 9991-9993 no longer exists. Ports are
unique within a host rather than across the cluster, so every machine uses the
same three, and a single-node install opens only the S3 port because colocated
nodes talk over an in-process pipe.
The reference architectures scripted a move of node data out of a directory
layout that no longer exists, which would have broken any install following
them. The update guide now says object storage has no migration path.
The fixture wrote a config the strict decoder rejects and built its cluster
through clusterrun, so it now renders a v1 TOML, loads it through LoadConfig and
serves under Run. Loading the real file means the fixture trips over the same
validation an install does. Its data directory holds node-<id> directly, and it
drains on cancellation rather than being abandoned at exit.
The runtime tests drove buildRuntime, serve and the old on-disk layout, all of
which predastore owns now, so they go. What replaces them covers what spinifex
still decides: the flag merge and its precedence, a host id or gate node the
config does not have, and the pid file behind Stop and Status.
Two fixes to earlier commits in this series: the host id conversion moves behind
its own guard, and the node status probes use WaitGroup.Go.
* fix(ochre): self-heal weights stage when the recorded snapshot is gone
runStageWeights's idempotency check trusted a matching source URI without
verifying the snapshot it names still exists, so a host rebuild or volume
GC that lost the snapshot left stage permanently short-circuited on a
useless KV record until an operator ran 'weights remove' first. Inject a
weightsSnapshotChecker, verify liveness before the no-op return, and fall
through to re-materialise when the snapshot is gone. 'weights list' uses
the same checker to mark dangling entries so an operator can see which
staged models can actually serve without waiting for endpoint launch to
fail.
* docs: note weights stage self-heal and list STATUS column
* feat(images): add a vllm-serving AMI as a stacked mkosi profile
LaunchServingVM resolves its AMI by the tags spinifex:managed-by=bedrock and
spinifex:bedrock-role=vllm-serving. No image carried either, so every Ensure
failed at resolveServingAMI. The only GPU image ships the NVIDIA driver and
Docker but no Python, no vLLM and no serving unit, so tagging it would have
moved the failure from admission to first boot.
Add a vllm profile that stacks onto gpu-nvidia and owns the whole guest side
of the launcher contract: the runtime, vllm-serve.service, the weights mount
and the tags. vLLM is baked into a uv-managed venv rather than pulled as a
container at boot — a runtime docker pull of vllm/vllm-openai is known to
stall this platform's write plane, and mkosi's unprivileged build has no
daemon to bake an image from anyway.
Also bring images/mkosi.profiles/ under make test-images. Its scripts have no
.sh suffix, so the existing name glob skipped exactly the files worth linting;
sources are found by shebang instead.
* fix(images): make the vllm postinst survive the build sandbox
Two failures found by actually running the build.
uv installs to /usr/local/bin, which mkosi's postinst PATH need not carry, so
`command -v uv` reported a good install as missing. Call it by absolute path,
and check it the same way.
`vllm --version` initialises device config and dies with "Failed to infer
device type" on a builder with no GPU, so it could never have verified
anything. Assert the pinned version through package metadata instead, which
also catches a version mismatch rather than merely a broken entrypoint.
* fix(images): pin the serving AMI datasource and mirror cloud-init to the console
A serving VM boots through LaunchSystemInstance, so it comes up dual-NIC with a
fw_cfg netcfg blob and takes its user-data solely from IMDS — the same shape as
the EKS control plane. That path carries a cloud-init datasource pin; the vllm
profile never got one, and the base config deliberately leaves datasource_list
unpinned, so the serving image was the only system VM booting without it.
Reuse the EKS pin rather than copying it. The 99- prefix is load-bearing:
read_conf_d() applies cloud.cfg.d in REVERSE and mergemanydict() is
first-source-wins, so a 90- name loses to cloud-init's own 90_dpkg.cfg ('_'
sorts above '-') and the pin is discarded in silence. The postinst asserts the
MERGED datasource_list via cloud-init's own read_conf_with_confd rather than
checking the file exists, because it existed at 90- too and did nothing there.
Also mirror every cloud-init stage to the serial console. A serving VM is
headless, and a guest that wedges before handover never surrenders
/var/log/cloud-init-output.log, so the console get-console-output reads is the
only channel an operator has.
The images_test.go case covers the two tags resolveServingAMI filters on; it
matches on those alone and never on name, so a typo hides the image entirely.
* fix(images): make the dhcpcd data-NIC lease genuinely one-shot
dhcp_oneshot() promised to configure the link and exit "leaving no daemon
to fight cloud-init", but the dhcpcd branch passed only -q, which is quiet
rather than one-shot. Every Ubuntu image takes that branch, so a dhcpcd
daemon survived on the data NIC holding the control socket.
cloud-init's Ec2 datasource then runs its own ephemeral dhcpcd on the same
NIC. Instead of spawning, dhcpcd forwards the invocation to the resident
daemon as a control command and exits 0, and --dumplease returns the
resident daemon's lease, so discovery looks successful. Cleanup then waits
for a pid file at /run/dhcpcd/<iface>-4.pid that no process ever wrote,
spinning its full 30000 x 10ms budget: init-local stalls for exactly 300s
on every system-instance boot.
Adding -1 exits once the interface is configured and -p keeps the address
and routes across that exit. Both are needed; -1 alone tears the link back
down. The test now models the real clients — the stub stays resident unless
given the flag that makes its client exit, and the dhcpcd branch is forced
by dropping udhcpc from PATH rather than left to whether the host happens
to have /usr/share/udhcpc/default.script.
* fix(images): serve without a CUDA toolkit and stop vllm-serve crash-looping
vLLM picks FlashInfer for top-k/top-p sampling, and FlashInfer JIT-compiles
that kernel on first use. The serving image ships the driver and the
container toolkit but no nvcc, so the sampler warmup died with "Could not
find nvcc and default cuda_home='/usr/local/cuda' doesn't exist" minutes
into a GPU cold start, after the model had already loaded.
VLLM_USE_FLASHINFER_SAMPLER=0 selects vLLM's native top-k/top-p sampler,
which is the same operation with no CUDA toolchain at runtime. Everything
else on the serving path is already precompiled in the wheels: the attention
backend resolves to FLASH_ATTN and CUDA graph capture completes, both
without nvcc. Baking FlashInfer's AOT modules is the alternative if the
fused kernel is ever wanted, and it keeps the toolchain in the builder
rather than the appliance.
The unit had Restart=on-failure with no bound, so an unfixable start failure
restarted forever and buried its own first traceback under identical copies
on the serial console — the only view an operator has of a headless guest.
StartLimitIntervalSec/StartLimitBurst make it report instead.
The postinst asserts the installed unit carries the sampler setting, for the
same reason the datasource pin is asserted there: the alternative is finding
out at runtime, on a GPU, five minutes in.
Also corrects the shared datasource pin's header, which still claimed it
installs as 90-spinifex-ec2-imds.cfg. It is installed at 99- by the vllm
profile precisely because a 90- name loses the reverse-sorted, first-source-
wins cloud.cfg.d merge to cloud-init's own 90_dpkg.cfg.
* fix(bedrock): raise the serving-endpoint readiness bound to 15 minutes
The 5 minute bound was chosen before any cold start had been measured, and
its own comment asked to be corrected once a model actually ran on wattle.
It now has: Llama 3.2 1B on an RTX A1000 reached "Application startup
complete" 257.6s after launch, leaving ~42s of margin on the smallest model
this platform can serve, with a cold torch.compile accounting for 26.3s of
it. A bigger card and a bigger model are unmeasured.
The two failure modes are not symmetric. Too low tears down an endpoint that
was about to come up and reports it as a failed launch; too high only makes
a genuinely stuck launch take longer to give up. So the bound is set well
clear of the single data point rather than close to it.
The CLI's own wait moves to 20 minutes to stay above it. The daemon starts
its clock when the VM is launched, up to a minute after the CLI starts its
own, so leaving both at 15 would expire the client first and report a bare
"timed out" in place of the daemon's abort, which carries the reason.
* fix: give the Ochre daemon a routed port in the Bedrock system VPC
The readiness probe and every InvokeModel call dial a serving VM's
system-VPC address from inside the daemon, but that subnet lives in OVN
on br-int and the host has no route to it, so a healthy VM never reached
READY.
Each daemon now owns an ENI in the system VPC's private subnet, bound to
an OVS internal port on br-int by iface-id, exactly as ovn-controller
binds a guest tap. The ENI address is installed at the subnet's prefix
length, which is what supplies the connected route. BaseURL is unchanged:
the address it already carried is now reachable.
Describe-or-create keyed on a per-node description, since ENIs carry no
tag filter, so a restart or a second endpoint reuses one address instead
of leaking one per launch. The port is ensured before anything else the
launch creates: a VM the daemon cannot dial holds a GPU for the whole
readiness window before unwinding.
* feat: add spx admin ochre endpoint ensure/describe/list/delete
The daemon subscribes all four bedrock.endpoint.* subjects, but nothing in
production calls their NATS client, so the serving-endpoint lifecycle could
not be exercised at all in a running system. This gives an operator a direct
way to drive it.
ensure --wait polls describe until READY and reports the elapsed time, since
the daemon replies STARTING as soon as it claims the model and the launch
outlives the request. That is the measured cold start.
Also corrects the weights stage tokenizer rule in COMMANDS.md, which still
described both tokenizer files as required rather than either.
* test: cover the ochre endpoint Run wrappers' connect and exit paths
* fix: treat an empty endpoint bucket as no endpoints, not an error
ListEndpoints was the only kvutil.Keys caller not handling ErrNoKeysFound,
so 'ochre endpoint list' failed on a cluster that had never launched one.
* feat: add spx admin ochre weights stage/list/remove
Give operators a supported path to stage a self-hosted model's
weights: point at a predastore S3 prefix holding ordinary Hugging
Face files, and stage materialises them into a viperblock volume,
snapshots it, and records the source URI and snapshot ID in the
bedrock-weights KV bucket.
stage refuses before downloading anything if the model is not a
self-host catalog entry, or if the prefix is missing any required
file. Re-staging the same source is a no-op; a different source
replaces the entry and reports the previous snapshot for separate
reclamation.
WeightsStore gains GetWeights and a source-URI field on its record
to support this. LookupServingSpec (catalog.go) is reused unchanged
to validate --model-id.
* fix: unify ServingSpec after rebase onto dev's daemon-launcher spec
Rebasing onto dev pulled in mulga-vmfng.7.5's daemon-side ServingSpec
and LookupServingSpec, textually duplicating the ochre weights stage
CLI's own versions in catalog.go. Collapse both into one type carrying
ModelID and VLLMArgs, and one three-value LookupServingSpec so callers
distinguish an unknown model (found=false) from a provider-tier one
(selfHost=false). Update the daemon launcher's call site accordingly.
* test: cover ochre weights stage/list/remove command behaviour
Splits runOchreWeightsStage/List/Remove into testable decision functions
(runStageWeights, listWeightsOutput, removeWeights) separate from cobra and
NATS wiring, and makes mkfs.ext4 execution and the connect/exit steps
swappable so the full command flow can be driven against an embedded
JetStream server instead of live infrastructure. Also relaxes the tokenizer
file requirement to accept either tokenizer.json or tokenizer.model, since
canonical Llama 3.x repos ship only the former at the prefix root.
* fix: resolve mkfs.ext4 from sbin when it is absent from PATH
e2fsprogs installs mkfs.ext4 into sbin, which a non-login shell omits, so
weights staging failed on an unprivileged operator's otherwise fine host.
* fix: size the weights volume to a whole GiB so the tail block fits
viperblock rejects a write whose block overruns VolumeSize, so importing an
image of unaligned length failed on its last block. Rounding up also makes
the volume's byte size and its manifest SizeGiB describe the same volume.
Name the volume in the import error so a part-written import is traceable.
* match new predastore config surface area
* fix(migrate): carry existing installs onto the predastore v5 topology
The cluster topology refactor breaks an install three ways at once. The old
config carries a top-level host key, which the new schema binds to the [[host]]
table array, so TOML rejects the document and the daemon dies before it reads
any topology. Node state moved from distributed/{db,nodes}/node-N to a per-host
data directory keyed by a node id that is now unique across roles. And because
object metadata records which node holds each shard, the shard nodes had to
keep their ids, so the state replicas took new ones and every replica's
persisted raft configuration stopped listing it.
predastore.toml gains a 4 to 5 migration covering all three: it rewrites the
tables, relocates whichever node directories exist on this machine, and
recovers each relocated state replica onto the new member set. The rewrite is
line-oriented rather than a re-render from the template, so buckets, auth,
rate limits and iam settings an operator edited survive.
spinifex.toml gains a 3 to 4 migration renaming the predastore node_id to
host_id, which predastore-start.sh greps to select the host this machine runs.
The predastore migration lives in its own package rather than in migrate,
which every handler that stamps a KV bucket version imports; reaching the
cluster runtime from there would pull raft, badger and the QUIC client into all
of them. admin_upgrade.go blank-imports it, so only the binary that runs the
upgrade links it, and a test pins both registrations.
* test(predastore): cover the service runtime, gateway and shutdown paths
The service wrapper's new seams had no unit tests: buildRuntime, serve and
the errgroup Start runs the cluster and the S3 gateway under. CI's diff
coverage gate measured the PR at 34.6% against a 60% threshold as a result.
Cover every buildRuntime error wrap and its host selection, both ways the
gateway can fail serve, and the full lifecycle: Start brings the nodes and
the gateway up together, Shutdown drains both by cancelling the context
they share. Topologies are written per test as real config, with node ids
from a process-wide counter — the pipe transport's listener registry is
global, and this package's integration tests hold a fixture daemon on the
low ids for the life of the test binary.
setup.sh aborts twice on resolute. qemu-kvm is a virtual package with two
providers there, so apt refuses to resolve it; the token was redundant anyway
since $QEMU_PACKAGES already installs qemu-system-x86/arm and spinifex execs
the qemu-system-* binaries directly.
sudo-rs, Ubuntu's default sudo since 25.10, then rejects the sudoers file: it
implements no pam_session Defaults setting, and forbids a wildcard inside a
command argument. pam_session is now emitted only for classic sudo, and the
sysctl grant names a fixed-verb helper instead of a wildcard.
The helper is narrower than the rule it replaces: it accepts one interface, a
key restricted to rp_filter or accept_local, and a value restricted to 0 or 1,
where the old grant permitted any net.ipv4.conf key at any value. Granting
sysctl with an unconstrained argument was rejected as root-equivalent.
Adds scripts/lint-apt-packages.sh and an Apt Packages workflow so the runtime
package list is resolved against each supported suite's real indices in CI.
Spinifex rejected HttpTokens=optional and served 401 to every untokened
GET, which is stricter than AWS, whose own default is optional. That made
Windows guests impossible: cloudbase-init is the only realistic Windows
bootstrap agent and its EC2Service has no IMDSv2 token support in any
released version, so a Windows instance never received its hostname,
injected password or user-data.
HttpTokens is now settable at launch and via ModifyInstanceMetadataOptions,
accepting optional or required. The launch default stays required, so an
instance that does not ask for IMDSv1 never gets it. Every other
metadata-option field keeps its existing rejection.
The untokened path is reachable without credentials, so the per-instance
lookup is cached with a short TTL; without that a guest could drive
unbounded DescribeInstances fan-outs. A valid token short-circuits before
the lookup, so compliant IMDSv2 clients are unaffected.
* feat: accept and surface gp3 Throughput on CreateVolume/DescribeVolumes
Honors input.Throughput (125-1000 MiB/s, default 125 when omitted) on
CreateVolume, persists it to VolumeMetadata, and surfaces it via
DescribeVolumes (getVolumeByID), mirroring the Iops handling shipped
in mulga-siv-380. Pins viperblock to the commit that adds the
VolumeMetadata.Throughput field.
Zero Throughput on a read volume means it predates this field; it is
omitted from the response rather than surfaced as a misleading 0
MiB/s, matching the existing IOPS convention.
* chore: pin viperblock to the merged gp3 Throughput commit
* ci: skip building unused E2E suite binaries in e2e.yml's single/multi jobs
These two jobs immediately rebuild suite-aware binaries after Build
Install Artifacts and never consume the tar's _bin contents, so the
cross-compiled suite build was pure waste. build-install-artifacts.sh
gains an optional --skip-e2e-build flag (positional contract unchanged);
e2e-nightly.yml and the other callers keep building the full set.
* refactor: replace hand-rolled IMDS client with AWS SDK v2
internal/imdscreds, cmd/ecs-agent's metadata fetch and its credentials
provider used a hand-written IMDSv2 token/GET flow with manual RFC3339
expiry parsing. A malformed Expiration silently fell back to "never
expires" instead of failing.
Replace it with feature/ec2/imds, credentials/ec2rolecreds and
aws.CredentialsCache, following the internal/gwsign/signer.go
precedent. An unparseable Expiration is now a hard error from Fetch,
covered by TestFetch_BadExpirationErrors. Output shapes to callers are
unchanged, and IMDS base URLs ending in /latest keep working.
* refactor: derive predastore bind config from spinifex.toml in spx itself
predastore-start.sh awk-parsed spinifex.toml for the bind host/port and
node_id, exported them as env vars, then exec'd spx. Move that
derivation into predastoreStartCmd via derivePredastoreBind, which
reads the raw (non-DIAL-normalized) nodes.<node>.predastore section,
defaulting node_id to -1 (co-located) when the key is absent, same as
the script. An explicit --host/--port/--node-id flag or
SPINIFEX_PREDASTORE_* env var still overrides the derived value.
Delete the now-unused script and point the systemd unit's ExecStart
directly at spx service predastore start, matching the
spinifex-viperblock.service precedent.
* fix(dhcp): release stranded leases and label them with the owning node
Two paths dropped a lease from KV without returning it to the upstream
server, so a source="dhcp" pool leaked addresses permanently.
Manager.Start deleted an expired entry outright. Dropping the entry alone
abandons the binding rather than returning it, and a server that ages
leases by liveness holds the address for as long as anything answers ARP
for it. Adoption now sends a bounded, best-effort RELEASE first, skipping
leases with no stored offer/ack because nclient4 cannot rebuild those.
Release-by-IP answered SUCCESS when the IP was absent from KV, telling
the caller the address was freed when nothing went on the wire. It now
returns an error. Every ReleaseIP call site logs and continues, so this
surfaces the strand without wedging teardown.
Leases were also unattributable upstream: chaddr is a derived HashMAC and
option 61 carries that same MAC, so no field identified the host that took
the lease. Option 12 is now "<node>-<client-id>" and option 60 is
"mulga-spinifex/<node>", giving the lease table a per-lease identity and a
per-host column to group on. HashMAC is untouched — encoding the node
there would cut per-node entropy and reverse a documented decision.
Observed live: 20 office-LAN addresses held by one node's orphaned OVN
gateway LRPs, with no lease records and no renewals.
* fix: rebind the datapath when a DHCP lease moves to a new address
A lease re-issued on a different IP was persisted to KV with no notification
of any kind. doAcquire replaced the lease after an expiry and doRenew accepted
whatever a rebind answered with, while the gateway LRP's address is written
once at attach and never revisited. Each such event burned one address
permanently: the old one squatted by OVN's ARP responder with no renewer and
no DHCPRELEASE, the new one renewed forever attached to nothing.
Route both paths through applyLease, which releases the superseded address and
invokes an IPChangeHook so the datapath can follow. vpcd registers a hook that
moves the gateway LRP's networks and gateway_ip stamp, re-runs the routed leg
so the VPC SNAT and host ingress route follow, and waits on the flows barrier.
EIP, ENI-public and NATGW addresses are recorded outside vpcd in
DescribeAddresses and DescribeInstances, so the hook reports those rather than
repointing a datapath the control plane no longer agrees with. The release half
still applies to every purpose, so the leak itself is closed either way.
A failed rebind does not fail the lease loop: the new address is held
regardless, and dropping it would strand both.
* fix: reconcile API-visible records when their DHCP lease moves
An EIP or auto-assigned public IP whose lease was re-issued elsewhere left
DescribeAddresses and DescribeNetworkInterfaces advertising an address vpcd had
already released. The datapath kept the old dnat_and_snat too, so the address
was dead on both sides while the upstream server was free to hand it to
someone else.
vpcd cannot fix this alone: it holds the lease, but the records naming those
addresses live daemon-side. Add vpc.dhcp.lease-changed, which vpcd requests and
the daemon serves, dispatching to the service owning the moved address.
RebindPublicIP and RebindENIPublicIP commit the new external IP through
vpc.add-nat first — AddEIP scrubs the predecessor row keyed on the logical IP,
so one commit is enough — and only then move the record, so a NAT failure never
leaves a record advertising an address that does not route.
Both refuse to move a record naming neither address, so a stale request cannot
overwrite an allocation the lease does not own, and both are idempotent under
duplicate delivery.
A failed or unanswered reconcile propagates back to the lease loop rather than
reading as success. vpc.dhcp.lease-expired, listed in the service manifest but
never implemented, is replaced by the subject that now exists.
* fix: dedupe retried AllocateAddress so one call takes one address
EC2 AllocateAddress carries no ClientToken, and vpcd's DORA ladder can
outlast the AWS client read timeout, so every SDK-level retry minted a
fresh allocation ID and drew another upstream DHCP lease. The caller kept
one EIP and the rest stayed leased with no record naming them, so nothing
ever released them.
Retries are now identified by the SDK's amz-sdk-invocation-id, which is
stable across attempts of one call. The gateway lifts it off the request,
NATSRequest forwards it as a header, and the daemon puts it back on the
context, so the EIP service sees the same token every attempt. singleflight
joins a retry to the call still in flight — the common case — and a TTL'd
KV record answers retries that arrive after the first call completed.
* fix: reclaim DHCP leases whose owner is gone and stop dropping expired ones
Three leaks that all end the same way: an address stays leased upstream with
nothing renewing it on purpose and nothing that will ever release it.
Manager.Start deleted any lease that expired while vpcd was down, after
releasing it. The resource it was taken for still existed, so the datapath kept
forwarding on an address the upstream server now considers free — a
duplicate-IP conflict waiting for the server to reissue it — and nothing ever
re-acquired. Expired leases now get a renewal loop like any other, which
re-DORAs on its first pass; the existing IP-change hook rebinds the datapath
when the new address differs. The reaffirm RENEW is skipped for these, since a
lease the server has already aged out will only fail it.
Nothing reconciled leases against the resources that justify them, so a
resource deleted out from under its lease left the address renewed forever.
ReapOrphans sweeps them: vpcd asks the daemon over vpc.dhcp.owner-check
whether the EIP, ENI or VPC still exists, and releases the lease when it does
not. A gateway lease also has a router port to remove, so DetachIGW runs
before the address goes back — releasing it while OVN still answers ARP for it
invites the same conflict. Anything the daemon cannot resolve is reported
unknown and kept: an unreachable daemon must never read as a fleet-wide
deletion.
The acquire ladder ran to ~62s, just past the 60s read timeout botocore
defaults to, which is what turned one AllocateAddress into two leases. It now
totals 40s with a 45s budget, and the DORA RPC 50s, so the caller hears the
outcome instead of giving up and retrying. This is a mitigation, not a fix, for
callers that send no retry token: botocore emits no amz-sdk-invocation-id, so
the dedupe added earlier cannot see their retries at all.
Fixes mulga-j19gj, mulga-9exe6, mulga-7qyzj.
* feat: add spx get dhcp-leases so held external addresses are visible
The upstream leases behind a source="dhcp" external pool lived in a per-AZ KV
bucket with no read path, so working out which addresses were held, by what,
and whether the owner still existed meant writing a NATS client or reading
journals. That was the main obstacle to diagnosing every leak in this series.
The command lists each lease with its purpose, VPC, address, MAC, holding node
and remaining time, and resolves the owner over vpc.dhcp.owner-check so an
orphan is visible as such. --orphans narrows the output to leaks;
--skip-owner-check drops the lookup when only the addresses matter.
Fixes mulga-655iu.
* fix(vpcd): reconcile gateway ports onto their lease address at startup
A gateway LRP configured from a static pool keeps its address when the pool is
switched to source="dhcp", so the VPC forwards on an address nothing leases
while the lease it holds goes unused. No lease event ever fires to correct it.
vpcd now sweeps every gw-lrp lease on startup and rebinds the port to the
address the lease actually holds. RebindGatewayIP is idempotent, so a port
already in agreement costs one OVSDB read.
* test: cover dhcp lease owner-check handlers and the lease CLI
The owner-check path decides whether the reaper releases an address, so every
inconclusive branch is asserted to report unknown rather than gone.
* fix(dhcp): make the outcome of a stale lease release observable
A stale RELEASE was logged only when the transport failed. A lease with no raw
offer/ack was skipped in silence, and a RELEASE the server ignored looked
identical to one it honoured, so an address left bound upstream could not be
traced to either cause.
The skip now warns, and a successful release logs the chaddr it used, which is
what a server matches a RELEASE against.
* fix(dhcp): take the RELEASE chaddr from the lease being released
A server matches a RELEASE to a binding on chaddr, and the release path
re-resolved it live. For a UseIfaceMAC lease whose interface MAC had changed
since, that sends an address the binding never had: the server drops the RELEASE
and says nothing, so the address stays held with no renewer.
The chaddr now comes from the lease record, which carries what took the lease
out, falling back to live resolution only when the record has none.
* fix: fan out certificate re-import to referencing load balancers
Re-importing new material under an existing ACM ARN previously updated
only the stored CertRecord; no load balancer referencing that ARN ever
re-rendered its config, so HAProxy kept serving the old leaf until it
expired while the API reported success.
Add an InUseBy reverse index on CertRecord, maintained via snapshot-diff
around every listener create/modify/delete and cert attach/detach in
handlers/elbv2. On re-import, ACMServiceImpl invokes a fan-out hook
(wired in daemon.go to avoid an import cycle) that re-runs
updateStoredConfig for every load balancer in the ARN's InUseBy set,
mirroring the existing target-group fan-out. lbagent is unchanged: it
already polls ConfigHash and reloads on change.
* fix: make InUseBy index updates CAS-safe under concurrent writers
AddInUseBy/RemoveInUseBy previously did a plain get/mutate/put, which
loses concurrent writers silently: two listeners on different load
balancers attaching the same certificate at once could drop one
entry with no error. Because InUseBy is the sole mechanism that fans
a renewed certificate out to the data plane, a lost entry means that
load balancer never re-renders and its certificate silently expires
in HAProxy while ACM reports it as healthy.
Switch both to a JetStream KV revision-based compare-and-swap loop
with jittered backoff, retrying on a concurrent writer instead of
overwriting it. Replace the old concurrency test, which only
asserted no errors/panics and so could not detect a lost index
entry, with one that asserts exact final state across N concurrent
writers.
* fix: encrypt ACM certificate private keys at rest
CertRecord.PrivateKey was persisted to the spinifex-acm JetStream KV
bucket as plaintext PEM. Encrypt it with AES-256-GCM via the existing
handlers_iam.EncryptSecret/DecryptSecret helpers, keyed by master.key.
Store.PutCert encrypts when a master key is set; GetCert/ListCerts
decrypt, falling back to a narrowly-bounded legacy-plaintext read (only
when the stored value is unambiguously a PEM private key block) so
pre-existing records keep working and are re-encrypted on next write.
A Store never given a key (ELBv2's independent read-only acmStore)
behaves exactly as before — encryption is opt-in per Store instance via
the new SetMasterKey method, so no other package needs to change.
NewACMServiceImplWithNATS now requires a master key and fails
construction without one; daemon.go loads master.key inside the
initServiceWithRetry closure so a missing key fails daemon startup
after the retry window instead of leaving the service permanently
degraded.
* fix: wire ELBv2's ACM store with the deployment master key
ELBv2 constructed its own handlers_acm.Store without a master key, so
once ACM started encrypting CertRecord.PrivateKey at rest, ELBv2 could
no longer resolve listener certificates: it read back ciphertext where
it expected PEM. Require NewStore(ctx, nc, masterKey) to always take a
key so an unkeyed Store can no longer be constructed, thread the
deployment master key through NewELBv2ServiceImplWithNATS and the
daemon's ELBv2 init closure (mirroring the ACM closure), and make ACM
store construction failure fatal instead of silently degrading every
HTTPS listener.
* feat(acm): tenant Private CA with x509 name constraints
Adds an independent tenant root CA (separate from the platform CA) that
signs leaf certificates only within its baked-in permitted domains.
Name constraints (PermittedDNSDomains, excluded IP ranges, excluded
email/URI) are frozen at creation and drift is rejected rather than
silently regenerated, so a device's trust store is never invalidated
without an explicit operator action.
* docs(acm): record OpenSSL verification of the empty email/URI exclusions
Confirms the zero-length ExcludedEmailAddresses/ExcludedURIDomains
entries parse cleanly and enforce as expected under OpenSSL 3.5.6, not
just Go's own verifier, since this CA is meant for device trust stores.
* feat: implement RequestCertificate with PRIVATE_CA managed issuance
Adds the ACM RequestCertificate API surface: mode derivation
(PROVIDER_API/MANUAL_TXT/PRIVATE_CA), synchronous PRIVATE_CA issuance via an
injected CertAuthority, and correctly-shaped PENDING_VALIDATION records for
the deferred modes. Extends CertRecord with issuance/validation/renewal
state, wires the gateway and daemon NATS dispatch, and updates COMMANDS.md.
* feat: wire tenant CA into the running spinifex daemon
Add LoadTenantCA as a load-only entry point (never creates), refactor
LoadOrCreateTenantCA to share the load path. Add 'spx admin cert
create-tenant-ca' to create the tenant CA with explicit permitted
domains, idempotent unless --regenerate is passed. Wire tenant CA
loading into daemon startup: missing tenant CA logs a warning and
degrades PRIVATE_CA issuance only, it does not block startup like the
master key does. Fold tenant CA paths into admin's CA path helpers,
keeping filenames distinct from the platform CA (tenant-ca.pem/.key
vs ca.pem/.key) so the two roots can never be confused.
Also give ACM's DeleteCertificate in-use refusal its own awserrors
alias instead of reusing the EKS-labelled one directly, and correct
COMMANDS.md to reflect that the tagging endpoints are implemented.
* fix: derive ACM validation mode from the requested domain, not a global flag
Mode derivation tested cfg.Northstar.DefaultDomain != "", which says only that
northstar is configured, never whether it is authoritative for the domain being
requested. Any deployment with a default domain set therefore returned
MANUAL_TXT for every request, including domains hosted nowhere, with a TXT
record the caller could not publish.
Resolve the requested domain against northstar's actual hosted zones, and put
MANUAL_TXT last. A name-constrained tenant CA is a definite local capability the
operator declared explicitly, whereas MANUAL_TXT assumes the domain is publicly
delegated so a public CA can resolve the challenge - which is never true of an
internal northstar zone. A tenant CA covering every requested domain now wins,
and a configured DNS provider still takes precedence over the manual path so
operators with a real domain are not downgraded to a private certificate.
* feat: renew PRIVATE_CA certificates automatically before they expire
Managed issuance signed a leaf once and never renewed it. With 90-day leaves
every certificate silently expired a quarter after the first apply, which is
the treadmill managed issuance exists to remove.
Renewal is scoped to PRIVATE_CA: re-signing from the tenant root is fully
offline, so this needs no ACME client and no network access. The window is
proportional - two thirds of the certificate's own lifetime, floored at a
minimum - so it self-tunes as issuance lifetimes change rather than degenerating
into 'renew immediately, forever' the way a fixed window would. A deterministic
per-ARN jitter keeps certificates issued in one apply from all renewing in the
same minute.
Reissue keeps the certificate ARN and generates a fresh leaf key, then pushes
through the ELBv2 fan-out. The push is mandatory rather than incidental:
GetLBConfig returns stored configuration and never re-resolves certificate
material, so a renewal that did not push would leave HAProxy serving the old
leaf until something unrelated changed the config hash.
A per-certificate lease keyed on the ARN means one node drives a given renewal
without serialising all renewals behind a single leader. Certificates whose
domains have fallen outside the tenant CA's name constraints record a failure
reason instead of retrying in a tight loop.
The renewal scan reads metadata only and never decrypts stored private keys;
the lease is a raw compare-and-swap that leaves the encrypted key untouched.
* fix: carry handler error messages over the daemon NATS transport
handleNATSRequest responded with the sanitized error code alone, so any
handler error without a registered code collapsed to a bare ServerInternal
and its reason was visible only in the daemon log. utils.ServeNATSRequestCtx
has always preserved the message; the two implementations had drifted.
Adds respondWithServiceError and routes the 16 service-error sites through
it, plus transport-level tests pinning that the message survives for an
uncoded error, that a wrapped coded error keeps its real code, and that a
bare code carries no message.
* feat: implement ACM GetCertificate and stop decrypting keys on read paths
GetCertificate was unimplemented end to end: it now returns the PEM
certificate body and chain for an owned ARN, ResourceNotFound for an unknown
or cross-account ARN, and RequestInProgressException for a certificate that
has a record but no body yet, wired through the service interface, the NATS
client, the daemon subscription and the gateway action map.
Store gains GetCertMetadata alongside GetCert, and ListCerts becomes
ListCertMetadata. Every read that answers a question about a certificate
rather than serving it - ownership checks, Describe, Delete, tag reads,
GetCertificate and listing - now leaves PrivateKey empty instead of running an
AES-GCM open per record to produce plaintext key material no caller reads.
* fix: reconcile the ACM master-key wiring with dev's new tests
The InUseBy and cert fan-out tests that landed on dev construct the ACM and
ELBv2 services directly, and both constructors now take the deployment master
key. Pass one shared key in the fan-out test - two services over the same
bucket disagreeing on the key is the exact failure the wiring exists to stop.
A Store is never unkeyed now, so the two test helpers collapse into one.
* fix: rebuild the ACM InUseBy index at daemon startup
The index that drives certificate fan-out is written only by listener
mutations, so a listener whose certificates have not been touched since it
was created has no entry. A certificate with an empty index then neither
fans out on renewal nor trips DeleteCertificate's in-use guard, and both
failures are silent: the renewal reports success while the load balancer
keeps serving the old leaf, and a certificate a live listener depends on
can be deleted.
Reconcile the index from the persisted listener records once at startup.
Walking every certificate rather than only the referenced ones makes it a
rebuild rather than a backfill, so it also drops entries naming a load
balancer that no longer exists. Writing only differences keeps it
idempotent, and the index writes are already CAS-guarded, so the nodes of
a cluster reconciling the same bucket converge.
* fix: deliver listener SNI certificates to HAProxy
Add/RemoveListenerCertificates persisted the listener record and then
returned, so the config was never re-rendered and lbagent had nothing to
poll for. Re-render through reloadListenerLB, the same helper the rule
mutations use.
That alone changed nothing, because the renderer dropped SNI certificates:
resolveFrontendCert returned only the default certificate and the bind line
emitted a single crt, with the PEM path keyed on the listener so a second
certificate had nowhere to live. Render every attached certificate instead,
default first so it stays the no-SNI fallback, each with its own path.
HAProxy accepts crt more than once on a bind line and selects by SNI.
The default certificate keeps its existing filename so an upgrade does not
move a deployed PEM, and the nginx renderer keeps the single-certificate
resolver it still relies on.
* fix: prune stale TLS certs from the lb-agent cert dir
A delivered PEM is cert + chain + private key. writeCertFiles only ever
wrote, so every certificate detached from a listener left its key on the
load balancer VM's disk indefinitely. Prune after a successful reload,
scoped to .pem files in the cert dir's immediate entries.
* fix: serve every attached certificate on the nginx NLB engine
nginx allows one ssl_certificate per key type in a server block, and the
NLB config is a stream block where server_name does not exist. Select by
SNI instead: give ssl_certificate a variable and map $ssl_server_name to
each certificate's path, keyed on the leaf's DNS SANs. A single-certificate
listener keeps a literal path, since a variable costs a per-handshake read.
The cert-path helpers were HAProxy-only by hard-coding lbagent.CertDir, so
they now take the dir and both engines share one multi-cert resolver.
* fix: emit ValidationMethod in DomainValidationOptions
DescribeCertificate returned each domain validation entry without a
ValidationMethod, so the AWS provider had nothing to compare against
aws_acm_certificate.validation_method. Emit DNS for the DNS-based modes.
PRIVATE_CA stays empty on purpose: it validates nothing, and real ACM omits
the method on a private certificate, where the provider reads it as NONE
whatever the request asked for.
* feat(bedrock): Ochre runtime MVP — Converse over self-host + Anthropic-direct
Add the AWS-Bedrock-compatible gateway services (bedrock, bedrock-runtime)
following the EKS REST-JSON pattern: ListFoundationModels / GetFoundationModel
over a static connectivity-tiered catalog, and non-streaming Converse routed
per modelId to one of two provider backends — self-hosted vLLM (OpenAI wire)
and Anthropic-direct (Anthropic Messages wire).
Per-account provider API keys are a master-key-encrypted JetStream KV secret
with an optional platform default; the advertised catalog gates external
models on a resolvable credential. Self-host endpoints are pinned via config
(OCHRE_VLLM_ENDPOINTS) behind an EndpointResolver seam.
Adds gateway/bedrock/ (catalog, provider router, Anthropic + vLLM adapters,
credential store), awserrors REST-JSON codes, and the bedrock E2E suite
(list/get light; Converse gated on live backends). Daemon/NATS control plane
and dynamic endpoint lifecycle are deferred to Phase 3.
* test(bedrock): cover dispatch, router, credentials KV + endpoint config
Add unit coverage for the Ochre Phase 1 diff: Bedrock_Request /
BedrockRuntime_Request end-to-end dispatch, provider router, JSON
response/error writers, upstream-status mapping, the credential-store KV
round-trip, the Anthropic bound-provider path, and parseBedrockEndpoints.
Lifts PR diff coverage above the 60% gate.
Three related nat-mode fixes:
- ura01: nat-transit pool now emits gw_lrp_range_start/end. admin init
sets 100.127.0.16-.254 (239 VPC gateways) instead of the auto-derived
top-16 (~15). Adds GwLrpRange* to PoolData + the spinifex.toml template
so operators can widen any pool.
- 724mm: detectPoolMode returns true only for external_mode=pool. nat and
disabled clusters now skip Phase 8b/8d, which otherwise failed on
AllocateAddress UnsupportedOperation.
- pnz2i: document that system instances (ECS/EKS/LB agents) are
unsupported in nat v1 (require pool mode) in the vpc-networking README
and COMMANDS.md; document the gw_lrp_range_* fields.
* fix(eks): write etcd-snapshot.env at CP launch so snapshots actually run
* test(eks): cover resolveSystemPredastoreURL and buildEKSServiceDeps
Diff-coverage gate for 852f61d9 was unmet (48.8% < 60%) because the
mgmt-bridge branches and EKSServiceDeps assembly had no direct test.
* feat(eks): restore-snapshot DR path (fresh CP + cluster-reset restore + NLB re-point)
* fix(eks): make restore-snapshot fail-safe (validate snapshot, unwind on partial failure, honest status)
* fix(eks): sign etcd-snapshot uploads with deployment region (was 403ing)
* fix(eks): apply recovery directive on DR-seed first boot + single-node replacement host
* fix(daemon): log NATS handler failures at Error (were only on the trace span)
* feat(network): add routed uplink mode, Routed wiring, and NAT egress rule helper
* feat(external): route gateway LRP over transit pool and install VPC SNAT in routed NAT mode
* feat(vpcd): support bridge_mode=nat (routed uplink, transit pool, egress rules)
* feat(daemon): register EIP handlers unconditionally with disabled-mode stub; seed default subnet per external mode
* feat(admin): support --external-mode=nat and suggest it for non-bridgeable uplinks
* feat(scripts): add --nat-uplink routed mode to setup-ovn.sh
* test(e2e): add routed-NAT single-node natuplink suite
Covers the external_mode=nat lane end to end on a nat-initialized node:
host transit wiring (veth, br-ext port, ip_forward, iptables egress
rules), spinifex.toml contents, disabled EIP surface (empty
describe-addresses, UnsupportedOperation on allocate), default subnet
MapPublicIpOnLaunch=false, OVN gateway LRP + VPC SNAT on the transit
/24, guest outbound egress proven via a serial-console probe (nat mode
has no inbound path), and unique transit gateway IPs across VPCs.
Suite self-gates on external_mode=nat; registered as e2e-natuplink and
in SUITES_BUILD so it compiles in CI. Manifest entry stays commented
until CI can provision a nat-mode environment.
* feat(ovn): Address_Set CRUD and NAT exempted_ext_ips column
Adds the exempted_ext_ips column (optional strong Address_Set ref) to the
nbdb NAT model, Address_Set ensure/get with wait-op create semantics, and an
in-place SetNATExemptedExtIPs update on the client interface, live client,
and mock. Test NB schema gains the column and covers insert-resolves-UUID,
converge, clear, and missing-NAT paths.
* feat(policy): exempt address set on routed SNAT/EIP rows
Adds WithSNATExemptSet so routed mode stamps exempted_ext_ips (the
spinifex_nat_exempt Address_Set: transit /24 plus operator extras) on IGW
snat and EIP dnat_and_snat rows, keeping VM replies to host-initiated flows
un-NATted. AddSNAT gains an idempotency guard keyed on (router, VPC CIDR)
with stale-IP replacement, and both AddSNAT and the AddEIP idempotent-skip
path patch the ref in place on pre-existing rows. The set is a singleton
strong ref and is never deleted. Non-routed modes are unchanged.
* feat(host): VPC ingress route primitives for routed-NAT mode
EnsureVPCIngressRoute / RemoveVPCIngressRoute manage the host route that
sends VPC-CIDR traffic into OVN via the gateway LRP transit IP on the
spx-nat veth. Replace/del semantics keep both idempotent; a missing route
on removal is not an error.
* feat(external): routed ingress leg on IGW attach/detach, reconcile-safe
AttachIGW's routed leg now also installs the host VPC ingress route via
RoutedIngressHooks, and re-runs on the already-attached path so the
reconcile loop restores host routes (and patches legacy SNAT exempt refs)
after a reboot — OVN state survives restarts but host routes do not.
DetachIGW removes the route alongside the SNAT. vpcd wires the hooks and
the spinifex_nat_exempt set (transit /24 + [network] nat_exempt_cidrs,
validated nat-mode-only) exclusively in routed-NAT mode; pool/veth/direct
pass nil hooks and are unchanged.
* test(natuplink): Tier 1 host-ingress phase; docs for jumpbox pattern
New Phase 7 asserts the automatic ingress tier: host route to the VPC CIDR
via the gateway LRP transit IP, exempted_ext_ips ref on the SNAT row
pointing at spinifex_nat_exempt (containing the transit /24), and — after
opening the SG from the transit net (default-closed, AWS parity) — ping and
an sshd TCP handshake against the instance private IP from the host, which
proves the un-SNATted return path. Phase 8 additionally waits for the
second VPC's ingress route to appear on IGW attach. vpc-networking README
documents the jumpbox pattern, SG note, and nat_exempt_cidrs.
* feat(config,admin): public wan pool alongside nat-transit
Restructure admin config generation from single-pool scalars to a Pools
slice so nat mode can emit both the transit pool and an optional public
wan pool. nat mode now accepts --external-pool/--external-source (plus
--external-bind-bridge/--gateway-ip when a public pool is requested),
validated by the shared resolvePublicPoolFlags helper also used by pool
mode. Transit pool always renders first so the IGW allocator picks it.
Formation wire format unchanged (nat mode is single-node only).
* feat(daemon): enable EIP service and auto-assign in nat mode with public pool
External IPAM (and with it the real EIP service, RunInstances public IP
auto-assign, and MapPublicIpOnLaunch on default subnets) now activates
whenever public pools exist: pool mode as before, plus nat mode when a
wan pool rides alongside the transit segment. The transit pool is
filtered out of IPAM — its addresses are gateway-LRP plumbing, never
allocatable EIPs. Transit-only nat configs keep the disabled EIP stub.
* feat(host): per-EIP ingress primitives for routed-NAT mode
DetectUplinkFor parses ip route get to find the interface and source IP
facing the pool gateway (re-run per ensure: WiFi/cellular uplinks move).
EnsureEIPIngress steers an EIP into OVN with a /32 route via the VPC
gateway LRP transit IP (src-hinted so host-local traffic still DNATs),
answers ARP for the EIP with a proxy neighbor entry (L3 only, same MAC,
WiFi-safe), sends gratuitous ARP, and adds per-EIP FORWARD accepts for
DROP-policy hosts. RemoveEIPIngress mirrors; missing state tolerated.
* feat(policy,vpcd): host EIP binder option on NATManager
WithHostEIPBinder injects routed-mode host plumbing hooks. AddEIP binds
after priming on both the fresh-add and idempotent-skip paths (host
routes and proxy-ARP are volatile even when the OVN row survives, so
reconcile re-ensures them after reboot); DeleteEIP unbinds best-effort.
The next hop comes from the new gatewayPortIP helper reading the gateway
LRP's stamped spinifex:gateway_ip (Networks fallback), with the ext-ID
key now an exported policy const shared with the external package.
vpcd selects the IGW pool by name instead of index — with two pools in
nat mode the gateway allocator must stay on the transit pool — and wires
the binder over host.EnsureEIPIngress/RemoveEIPIngress when a public
pool is present. EnsureEIPIngress gains an uplink hint so dhcp-sourced
pools (gateway unknown until lease) proxy-ARP on their bind bridge.
* feat(dhcp): interface-MAC lease variant with MAC-keyed-router detection
dhcp_mac = "derived" (default) | "interface" on external pools. The
interface variant leases with the bind interface's own MAC — WiFi/WWAN
uplinks drop frames with foreign source MACs, so derived chaddr MACs
never complete DORA there. Option 61 stays the lease discriminator.
UseIfaceMAC threads AllocateRequest → NATS wire → nclient4 → lease and
persists through KV so restart re-acquires repeat it.
Routers that key leases on MAC and ignore client-id would hand every
lease the same IP; the manager detects an ACK matching the interface's
own address or another client-id's live lease, releases the colliding
lease, and hard-errors advising a static range instead.
admin init auto-writes dhcp_mac="interface" for nat-mode public dhcp
pools bound to non-bridgeable uplinks (wlan/wwan/ppp).
* test(natuplink)+docs: Tier 2 EIP ingress e2e phases and public-pool docs
The natuplink suite now detects a public (non-transit) pool in spinifex.toml
and switches lanes: Tier-1-only configs keep the EIP-disabled and
no-public-IP assertions; public-pool configs assert the EIP surface is
enabled, the default subnet has MapPublicIpOnLaunch=true, instances
auto-assign public IPs, and a new Phase 9 exercises the full EIP ingress
lifecycle — host /32 route via the gateway LRP, proxy-ARP on the uplink,
per-EIP FORWARD accepts, exempt-stamped dnat_and_snat, TCP handshake to the
EIP, reconcile replay across a vpcd restart, and teardown on disassociate.
Docs: vpc-networking README gains a public-pool section for routed-NAT nodes
(init flags, dhcp_mac=interface behaviour, host-to-EIP caveat) and an updated
mode comparison; COMMANDS.md notes nat-mode EIP support behind a public pool.
* fix(setup): whitelist iptables, neigh sysctl, and arping for spinifex-vpcd
Routed-NAT mode shells out to iptables for the transit masquerade and
per-EIP FORWARD accepts, tunes proxy_delay, and sends gratuitous ARP for
host-delivered EIPs. Without these sudoers entries vpcd crash-loops with
"command not allowed" on nat-mode nodes.
* fix(ovn): scope DeleteStaticRoute prefix match to the target router
Every VPC router carries a 0.0.0.0/0 static route, so an unscoped cache
match could grab another router's row and fail with a referential
integrity violation, wedging IGW attach on multi-VPC nodes. Match only
rows referenced by the target router, mirroring FindStaticRoute. Mock
updated to the same semantics, with a regression test covering drift
replacement across two routers.
* fix(reconcile): always replay AttachIGW so host ingress state is re-ensured
applyIGWs skipped AttachIGW whenever the gateway switch port existed in
actual state, so the already-attached re-ensure path never ran and
routed-NAT host ingress routes lost on reboot were never reinstalled.
AttachIGW is idempotent; run it unconditionally.
* fix(vpcd): resolve duplicate-CIDR host ingress route collisions, bootstrap VPC wins
Every account's default VPC shares 172.31.0.0/16, so Tier-1 host routes
collide on the single host route table and the last IGW attach silently
stole the route (route replace). Pass vpcID and the gateway LRP IP
through RoutedIngressHooks; the bootstrap VPC always installs its route,
other VPCs only install when the CIDR is free and only remove a route
their own gateway IP holds. Adds host.VPCIngressRouteVia to read the
current route holder.
* ci(e2e): add nightly cell 19 for routed-NAT single-node suite
Cell 19 bootstraps with --external-mode=nat plus the CI static pool as
the public wan pool and runs the natuplink suite, covering the Tier-1
transit datapath and Tier-2 EIP ingress. The run step exits 0 when
natuplink.test is absent from the tests tar so scheduled runs against
branches predating the suite stay green.
* fix(admin): nat-mode init summary reflects configured public pool
The summary printed 'outbound-only — no public IPs/EIPs' even when
--external-pool/--external-source configured a public wan pool. Print
the pool range (or dhcp bind) and note EIPs are enabled.
* fix(eip,vpc): thread ctx through disabled EIP stub and NAT call sites
Reconcile the routed-NAT branch with dev's ctx-first service APIs after
rebase: DisabledEIPService and its tests take ctx to satisfy the EIPService
interface, the vm_adapters public-IP resolver asserts the ctx-signed
AssociatedPublicIPForInstance, and the default-subnet mapping test threads
ctx into DescribeSubnets.
* feat(otelsetup): OTel init, resource attrs, trace-stamping slog handler
Env-gated on OTEL_EXPORTER_OTLP_ENDPOINT: with no endpoint only the W3C
propagator is installed and globals stay no-op, so instrumented binaries
ship everywhere at zero cost. Resource carries service/host/mulga.env/
mulga.source plus OTEL_RESOURCE_ATTRIBUTES (ci.run_id in CI).
* feat(otelsetup): wire telemetry init into all spx services
Every spx service start command installs the trace-stamping JSON slog
default and the OTel providers via initTelemetry, flushed on shutdown.
Gateway's own slog setup now routes through otelsetup so its level
handling keeps trace correlation.
* feat(gateway): HTTP tracing middleware on awsgw and daemon cluster API
Server span per request with W3C traceparent extraction; awsgw spans are
renamed to the resolved SigV4 service.Action post-auth with account and
region attributes. Request log lines carry trace_id via context.
* feat(nats): propagate W3C trace context over command subjects
NATSRequestCtx injects traceparent beside X-Account-ID and opens a client
span per hop; ServeNATSRequest opens a consumer span joining the producer
trace, with a Ctx variant for handlers that thread context. RunInstances
is plumbed end-to-end (gateway dispatch through placement fan-out to the
daemon handler) and viperblockd ebs.mount/unmount join inbound traces.
* feat(systemd): source optional /etc/spinifex/telemetry.env in service units
Lets operators enable OTLP export (OTEL_EXPORTER_OTLP_ENDPOINT,
MULGA_ENV, MULGA_SOURCE) by dropping a single env file, with no unit
edits. File is optional; absent means telemetry stays disabled.
* feat(ci): export OTel telemetry from E2E envs to the central Elastic sink
All E2E workflows set SPINIFEX_OTLP_ENDPOINT (repo variable override,
default wattle) plus ci.run_id/ci.workflow resource attributes, so
bootstrap-install.sh writes telemetry.env on every node. Teardown runs
flush-telemetry.sh before the final tofu destroy so buffered spans and
metrics reach the sink.
* feat(otelsetup): request metrics and Go runtime metrics via OTLP
mulga.requests counter + mulga.request.duration histogram recorded by the
same middleware that opens spans: awsgw carries the resolved SigV4 action,
the daemon cluster API uses the chi route pattern, and NATS handlers record
per-topic at the subscribe chokepoints (node names stripped to keep
cardinality flat). Go runtime metrics start inside the same env gate.
* feat(daemon): deep spans for QMP commands, VM launch phases, EBS ops
Single chokepoint span per QMP command in sendQMPCommand covering all
callsites; vm.launch phase children (mount_volumes, start_qemu, qmp_connect,
persist_state); ec2.cmd consumer span with traceparent extraction plus
start/stop/reboot/terminate child ops; ebs.mount/unmount spans in
viperblockd with volume attributes.
* feat(telemetry): thread request context through NATS, object store and VM paths
Migrate utils.NATSRequest call sites to NATSRequestCtx (241 sites; W3C
traceparent now flows gateway -> daemon on every request), add GatherCtx and
migrate all Gather call sites, inject traceparent on manual EKS publishes.
ObjectStore interface is ctx-first with aws-sdk WithContext calls and an
otelhttp transport, so predastore spans join the caller trace. NATS consumer
spans record error status when the response envelope carries an error.
vm.Manager and the ec2 handler service interfaces take ctx-first so vm.launch
and QMP spans join the request trace; detached background work uses explicit
context.Background(). Hot-path slog calls use *Context variants for trace_id
binding.
* feat(telemetry): suppress health-check, heartbeat and unrooted trace noise
Health probes record metrics only (no span per poll); the QMP heartbeat
poller marks its context so query-status ticks stop rooting a trace each.
rootSampler honors MULGA_ROOT_TRACE_RATIO (ParentBased ratio, default 1.0)
and the predastore unit sets 0.05: guest block I/O roots are sampled while
request-linked spans are always kept.
* feat(otel): export deployment.environment resource attribute
Elastic APM maps it to service.environment, enabling the native environment
selector across the APM UI (Services, Traces, Dependencies).
* fix(telemetry): stop objectstore rooting method-only HTTP transactions
Background S3 calls (state sync, cleanup loops) carried no caller span, so
otelhttp rooted a trace per request and APM surfaced thousands of parentless
'HTTP GET/PUT/DELETE' transactions with expected-404 probes counted as
failures. Emit the client span only when the request context already carries
one, matching the viperblock backend. Bump viperblock/predastore to branch
heads so CI builds also pick up the equivalent span filter and root sampler.
* feat(telemetry): per-VM guest metrics via dedicated QMP socket + qmp-collector
Goanna Phase 1 producer, operator-consumed first. Each VM gets a second
-qmp chardev (qmp-telemetry-<id>.sock) plus a sidecar discovery file
carrying instance/account/vcpus/taps/monitoring-tier; the new
qmp-collector service polls query-cpus-fast (vCPU thread /proc jiffies),
query-blockstats, query-balloon and host tap sysfs counters, and
publishes CloudWatch-mappable goanna_ec2_* series to NATS
metrics.ec2.<instance-id>. A queue-group bridge in the same process
forwards the series to the node-local OTLP receiver so they reach the
operator sink until Goanna consumes the subjects. The collector never
touches the manager's control QMP socket; telemetry failures never
block a launch.
* fix(qmp-collector): dial NATS via DialTarget, drop ProtectProc, fix meta race
The collector connected to the raw NATS host (0.0.0.0), failing TLS
verification against a cert valid for 127.0.0.1; route through
admin.DialTarget like the other daemons.
ProtectProc=invisible hid the QEMU vCPU threads (dumpable=0) that the
collector must read from /proc/<tid>/stat, so every tick failed and
re-primed without ever publishing. Drop it to default.
Surface collect failures at Warn so silent metric loss is visible, drop
per-tick publish to Debug, and read InstanceID under the lock in run()
to fix a race with updateMeta on ENI hot-plug.
* chore(deps): re-pin predastore + viperblock to dev-landed commits
* feat(gpu): add H200/RTX Pro 6000 SE models and fix MIG host resource allocation
- Fix H100 SXM/PCIe device ID labels swapped in models.go (10de:2330=SXM, 10de:2331=PCIe)
- Add H200 SXM (10de:2335) to knownModels, computeModels, and migCapableModels
- Add NVIDIArtxPro6000BlackwellSE to definitions.go mapped to g7e family (matching AWS g7e instance type)
- Add MIGHostResources() policy table: 4 vCPUs / 32 GiB host RAM per compute slice, covering all known NVIDIA MIG profiles
- GenerateMIGTypes now emits non-zero vCPU/memory for EC2 instance type descriptors
* fix(daemon): enforce CPU/memory limits for MIG instance types
Whole-GPU passthrough remains bypass-only (gpuManager.Claim is the sole gate).
MIG types now fall through to canAllocateCount so host CPU and RAM are checked
before launch — prevents over-committing a host where one physical GPU backs
multiple concurrent slices, each with non-trivial vCPU/memory allocation.
* feat: Add snapshot image promotion flow
* fix: Small changes to viperblock/spinifex so snapshot captures delta properly and instances can boot from snapshot AMIs
* fix: test coverage and more consistent prefix check
* fixed testing
* more tests
* fix: added new graceful process killer in utils, bumped NATS timeout for testing
* fix: complete at-rest encryption launch + volume lifecycle (#361)
* chore: bump viperblock to encrypted-snapshot VolumeUUID nonce fix
Pull viperblock 1044c2a (feat/snapshot-uuid-nonce-fix) so E2E builds against
the fix for the encrypted AMI snapshot metadata tag-verify failure
(cipher: message authentication failed) on instance launch. Pairs with
viperblock PR #22.
* fix: forward viperblock master key to nbdkit mount path
viperblockd loaded the master key but never passed it to the nbdkit
plugin it spawns for volume mounts, so encrypted boot volumes opened
in cleartext and failed LoadState with an encryption-state mismatch.
Forward cfg.EncryptionKeyFile into NBDKitConfig and emit the plugin's
encryption_key_file= argument when set.
* chore: bump viperblock to fresh-volume VolumeUUID mint fix
* chore: bump viperblock to snapshot-link LoadState ordering fix
* fix: route encrypted-volume config writes through viperblock keyholder
config.json is a sealed VBState whose AES-GCM tag and StateSeqNum-derived
nonce can only be advanced by the master-key holder that owns the volume.
The EC2 volume handler wrote it directly to S3, so mergeVolumeConfig had to
refuse encrypted volumes (rewriting would strip the tag) — bricking
UpdateVolumeState (attach in-use / detach available) and ModifyVolume.
Route encrypted-volume config updates through viperblockd, mirroring the
ebs.snapshot idiom: per-volume ebs.config.{volumeID} hits the owning node's
live VB (the single StateSeqNum writer); on ErrNoResponders the ebs.config
queue group opens the detached volume exclusively and reseals. Cleartext
volumes keep the direct-S3 fast path. No crypto leaves viperblockd.
mulga-siv-323
* fix: unwrap encryption envelope when snapshot handler reads config.json
CreateSnapshot and snapshotInUseByVolumes decoded config.json straight into
a VBState. For encrypted volumes config.json is an at-rest envelope
{payload, authtag}; a raw decode yields a zero VBState (SizeGiB==0), so
CreateSnapshot 500s on the size guard and the in-use scan silently drops
encrypted volumes' snapshots. Unwrap via viperblock.StateBody first,
mirroring the volume handler's mergeVolumeConfig path.
mulga-siv-324
* test: cover viperblockd ebs.config handlers
Unit/integration coverage for applyConfigUpdate, makeConfigUpdateHandler
(ebs.config.{volumeID}), and the ebs.config queue-group fallback: grow-only
size reconcile, live-VB apply, invalid JSON, apply errors, and the
detached-volume open path. Clears the diff-coverage gate (68.8%).
* chore: pin viperblock to dev tip (#22 merged)
* fix(viperblockd): seal volume block-map to predastore on detach + durability e2es (#371)
* feat: cloud-init EC2 datasource (#411)
* feat: Add snapshot image promotion flow
* fix: Small changes to viperblock/spinifex so snapshot captures delta properly and instances can boot from snapshot AMIs
* fix: test coverage and more consistent prefix check
* merge changes
* chore: remove dead SnapshotSub field and SIGTERM snapshot remnants from viperblockd
SnapshotSub was added during the NATS/SIGTERM snapshot attempts but is never
set post-rebase. Removes the field from MountedVolume and the two nil-checked
unsubscribe blocks. Re-adds the nbd import which is still needed for NBDKitConfig.
* fix: post-rebase gofmt fixes and remove dead snapshot handler code
Split gofmt-squashed lines in go.mod, vm_adapters.go, nbd_test.go,
viperblockd.go, volumes.go, and volume_durability_test.go. Removes the
dead unix-socket snapshot handler tests from viperblockd_handlers_test.go.
* test: add viperblock snapshot S3 coverage tests
Exercises snapshotVolume, snapshotRunningVolume, and snapshotStoppedVolume
past the nil-config guard using an httptest server that returns 403.
The AWS SDK credential check fails synchronously without network I/O,
keeping the tests fast while covering the S3Config construction, master key
load, and viperblock.New code paths. Brings diff coverage from 45% to 69.9%.
* fix: cloneAMIToVolume reads config via format-agnostic path
LoadStateRequest expects a viperblock binary envelope (encrypted),
but CreateImageFromInstance writes the AMI config as plain JSON via
putAMIConfig. Replace LoadStateRequest with s.amiLoader.GetAMIConfig
which uses viperblock.StateBody to handle both formats, matching the
path used by ListImages and all other AMI readers.
* fix: pin viperblock to feat/snapshot-ami pseudo-version for LoadLiveCheckpoint
go.mod referenced v1.10.0 (released from dev) which pre-dates LoadLiveCheckpoint.
CI builds without go.work so it resolved to the published version and failed.
Pin to the snapshot-ami HEAD pseudo-version until the branch is merged and tagged.
* fix: replace indexOf with slices.Index, revert e2e runner and viperblock version
- nbd_test.go: remove local indexOf helper, use slices.Index from stdlib
- e2e-suites.yml: revert ubuntu-latest back to ubuntu-26.04 with correct formatting
- go.mod: revert viperblock back to v1.10.0 (will resolve once viperblock PR merges)
* fix: remove pseudo-version pin from go.sum
* feat: drain viperblock via snapshot socket before CreateSnapshot
Before LoadLiveCheckpoint(), dial {data_dir}/viperblock/{vol}/snapshot.sock
and wait for DrainToBackend() to complete (30s timeout). This ensures WAL
chunks and the live checkpoint are current in S3 before the snapshot VB
reads them. If the socket is absent the instance is stopped and Close()
already drained.
fix: socket path used s.config.WalDir which is empty in the daemon config;
changed to s.config.DataDir + "/viperblock" to match the path the NBD
plugin uses when creating the socket.
* feat: wire snapshot/AMI UI actions and add PromoteImage NATS endpoint
- Register as AMI from snapshot detail (RegisterImage with BDM pointing at snapshot)
- Deregister from image detail (DeregisterImage with navigate-away on success)
- Promote to System from image detail (admin-only, irreversible via UI)
- NATS subject spinifex.image.promote wired through gateway → daemon → existing admin.PromoteSystemImage
- signedFetch extended with optional params to support action-keyed extra fields
* refactor: address PR review on PromoteImage types and subscription ordering
- Merge PromoteImageInput/PromoteImageOutput into PromoteImageOpts/PromoteImageResult
(add JSON tags; drop the duplicate pair)
- Inline the input variable in gateway/spx/promote_image.go
- Move spinifex.image.promote subscription to the spinifex.* group in daemon.go
* test: cover snapshotVolume/Running/Stopped error paths for diff coverage
Add encryption-key-load-error and LoadState-failure tests for the three
new viperblock snapshot paths (snapshotVolume in snapshot service,
snapshotRunningVolume and snapshotStoppedVolume in image service).
Diff coverage: 58.8% → 65.5%.
---------
Co-authored-by: Josh Siviour <mrk@mulgadc.com>
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
brynmailer-mdc