Expand the publish workflow to build the two Ubuntu NVIDIA GPU variants
alongside the Alpine eks-node/ecs-agent AMIs. The image build+publish
scripts are already manifest-driven and Ubuntu-aware; only the workflow
lacked the GPU legs.
Refactor the single-job body into a matrix over the four image manifests,
resolved by a new 'variant' input (all|cpu|gpu) so the slow 16G GPU legs
can be skipped when only the CPU AMIs need republishing. GPU images do not
fit a RAM-backed /tmp, so set SYSTEM_IMAGE_BUILD_DIR onto the runner's
on-disk _temp; legs run serially (max-parallel 1) on the single ci runner.
The nodegroup label was only written to K3S_NODE_LABEL in agent.env, but
k3s has no such env var, so non-GPU workers never received the label.
waitWorkersReady buckets Ready nodes by that label (added in #511), so
the affected nodegroups never counted toward their ACTIVE transition and
stayed CREATING until timeout.
Ride the label on a real k3s node-label config drop-in
(15-nodegroup-label.yaml), delivered via cloud-init user-data. Drop the
inert K3S_NODE_LABEL from agent.env and the initd export.
WaitForInstanceState now aborts immediately with the recorded StateReason
message when it observes a reaped state (shutting-down/terminated) while
waiting for a live target, instead of burning the full 5-minute timeout on
an instance that can never reach it.
MarkFailed takes a context and records the failure on the active span via
span.RecordError so a reaped launch surfaces in the APM error stream,
correlated by trace id. Only the LaunchRunInstances path carries a live
span; other callers pass context.Background() and the record is a no-op.
Surfaced by RCA of E2E run 29258398516 (TestAccountScoping/Step2).
TestNATGateway flaked with 100% egress loss: the tenant VPC gateway router's
OVN MAC_Binding had no entry for its upstream default-route nexthop. OVN
resolves that nexthop via lazy dynamic ARP on first egress; if that ARP is
lost during gateway bring-up, SNAT'd packets die in lr_in_arp_resolve and no
reply returns. MAC_Binding is per (datapath, logical_port, ip), so a sibling
VPC router resolving the same nexthop does not help.
Deterministically seed a static MAC binding for the nexthop at IGW attach. The
nexthop MAC is the host's own default-gateway MAC (the host shares the physical
uplink), read from ip neigh and written via ovn-nbctl static-mac-binding-add
(northd programs it into SB MAC_Binding, GC-safe). Host layer L0 owns the
shell-out per ADR-0006 S2; igw calls it via an injected hook. Best-effort and
idempotent: a missing MAC leaves dynamic ARP as the fallback, and it re-runs on
every AttachIGW so a binding lost to a later flap self-heals.
utils.GetOrCreateKVBucket hardcoded Replicas=1, so the ~38 handlers that
create their KV buckets lazily (VPC/subnet/ENI/EIP/ELBv2/ECS/ECR/ACM/
natgw/snapshot/placementgroup/IPAM/account-settings/dhcp/static-pool …)
created them at R1 on a multi-node cluster. upgradeJetStreamReplicas runs
once at startup, so any bucket created lazily after boot stayed R1
permanently — and an R1 stream's single Raft replica is transiently
leaderless during formation/join/rebalance, surfacing as
ErrNoResponders/ErrTimeout on KV ops.
Add a package-level default replica count that the daemon sets once at
boot (startCluster, after initJetStream) to len(clusterConfig.Nodes),
before any handler creates a bucket. GetOrCreateKVBucket now creates at
that default, so lazily-created buckets are born quorate. The one-shot
startup UpdateReplicas remains as a safety net for pre-existing R1
buckets from older deploys.
Zero call-site churn: the default flows through the existing
GetOrCreateKVBucketWithReplicas path already used by eks/ecs/iam.
TestSpawnRegisteredReconcilers_ResumesNonTerminal flaked two ways. First,
the shared fixture meta stamps a weeks-old CreatedAt, so a resumed CREATING
reconciler instantly trips the CREATE timeout, marks the cluster FAILED and
deregisters before the registry.Has assertion runs — the test was racing a
self-removal. Stamp a recent CreatedAt so the reconciler stays CREATING.
Second, SpawnRegisteredReconcilers ran a one-shot JetStream enumeration
(KeyValueStoreNames + kv.Keys), which is eventually consistent: under load a
just-committed cluster can be missing and never resumed, silently on a real
daemon. Re-scan until the observed CREATING/ACTIVE count stops growing, capped
and short-backed-off; Spawn is idempotent so re-runs only pick up stragglers.
discoverSingleNodeIPs seeds 127.0.0.1 then appends net.InterfaceAddrs()
in undefined order, so WANHost=nodeIPs[0] was loopback and the guest
gateway URL helpers fell through to the first non-loopback interface
addr — an arbitrary co-tenant bridge on multi-bridge shared hosts
(wattle: br-lan-env1..23, br-wan), leaving guest VMs unable to reach the
advertised gateway to register.
Resolve WANHost from spinifex.toml nodes.<node>.advertise (falling back
to that node's host), mirroring awsgwBindIP's toml scan; fall back to the
first non-loopback discovered addr only when config is unreadable. This
removes the need for the SPINIFEX_WAN_IP override on shared hosts.
The crash-supervisor auto-restart path (qmpHeartbeat -> HandleCrash ->
MaybeRestart -> startQEMU -> newQMPClientWithHandshake -> dialQMPWithRetry)
already unlinks the stale QMP socket and retries a refused/absent connect. But
a freshly relaunched QEMU can accept the connect then reset the connection
mid-greeting while it initialises, surfacing 'connection reset by peer'
immediately after 'QEMU started successfully' as a ServerInternal.
Add ECONNRESET and EPIPE to isTransientDialError so the dial is retried until
QEMU services the monitor, closing the auto-restart QMP-connect race.
Add non-streaming InvokeModel to bedrock-runtime. Unlike Converse's unified
schema, InvokeModel passes model-provider-native bodies through, so route by
model family: Anthropic Claude bodies forward to the Messages API (drop the
Bedrock-only anthropic_version, inject model, return verbatim); Meta Llama
bodies translate to/from the self-host vLLM OpenAI completions endpoint.
Raw-body dispatch bypasses the JSON response path (WriteRawResponse). Reuses
the Phase 1 credential + endpoint resolvers and upstream-status mapping.
Unit-tested (routing, body rewrite, translation, error classes) plus gated
InvokeModel E2E.
Both services already run otelsetup.Init and ship logs/metrics but never
opened a span, so they were invisible in the Kibana APM service map.
qmp-collector now opens a root span per per-instance poll tick (marks
error on QMP collect/publish failure); spinifex-ui now wraps its HTTP
handler chain with otelsetup.HTTPMiddleware and its reverse-proxy
transport with otelhttp, so UI requests get a server span and proxied
calls to awsgw/predastore carry traceparent instead of stopping the
trace at the UI hop.
nats and eks are intentionally left uninstrumented in this pass: nats is
the vendored message-broker process with no message-level hook to attach
spans to (the traffic it carries is already traced end-to-end at its
producer and consumer endpoints); eks request/response calls already
produce spans today under service.name=spinifex-daemon since the
handlers run in that process, and its async reconciler/state-machine
goroutines need a separate span design across a much larger surface.
Skip ERROR log when a subscription is already invalid
(nats.ErrBadSubscription) during graceful shutdown; log at debug
instead. Cosmetic cleanup race, no functional impact.
golangci-lint canonicalheader flagged 6 lowercase header literals in the
Ochre/Bedrock gateway (anthropic.go, anthropic_test.go, vllm.go). Canonical
form is wire-equivalent; canonicalise to X-Api-Key, Anthropic-Version and
Content-Type to restore CI lint.
* feat(bedrock): Ochre runtime MVP — Converse over self-host + Anthropic-direct
Add the AWS-Bedrock-compatible gateway services (bedrock, bedrock-runtime)
following the EKS REST-JSON pattern: ListFoundationModels / GetFoundationModel
over a static connectivity-tiered catalog, and non-streaming Converse routed
per modelId to one of two provider backends — self-hosted vLLM (OpenAI wire)
and Anthropic-direct (Anthropic Messages wire).
Per-account provider API keys are a master-key-encrypted JetStream KV secret
with an optional platform default; the advertised catalog gates external
models on a resolvable credential. Self-host endpoints are pinned via config
(OCHRE_VLLM_ENDPOINTS) behind an EndpointResolver seam.
Adds gateway/bedrock/ (catalog, provider router, Anthropic + vLLM adapters,
credential store), awserrors REST-JSON codes, and the bedrock E2E suite
(list/get light; Converse gated on live backends). Daemon/NATS control plane
and dynamic endpoint lifecycle are deferred to Phase 3.
* test(bedrock): cover dispatch, router, credentials KV + endpoint config
Add unit coverage for the Ochre Phase 1 diff: Bedrock_Request /
BedrockRuntime_Request end-to-end dispatch, provider router, JSON
response/error writers, upstream-status mapping, the credential-store KV
round-trip, the Anthropic bound-provider path, and parseBedrockEndpoints.
Lifts PR diff coverage above the 60% gate.
* ci: GPU e2e workflow on ci-gpu self-hosted runner
Runs tests/e2e/gpu (EC2/ECS/EKS GPU passthrough) against the persistent,
hand-bootstrapped wattle shared host, driven over SSH by a banksia runner
labelled ci-gpu (matches the ci-single/ci-multi convention) — no runner
installed on the shared box.
- runs-on [self-hosted, ci-gpu]; concurrency ci-gpu, cancel-in-progress
false (single A1000 = single-flight, queue not stomp)
- workflow_dispatch + GPU-path filter (not every push)
- manual-work mutex gate (~/.ci-gpu-paused / CI_GPU_PAUSED) so hands-on GPU
testing blocks CI
- if: always() cleanup terminates launched GPU instances; never resets the
box (no setup-ovn.sh/dev-reset/cluster-reset)
- drives the control plane with spx + the already-configured aws spinifex
profile directly (no wrapper scripts, nothing to install on wattle)
* ci(gpu-e2e): advisory AMI preflight, fail only if zero GPU AMIs
The suite is self-guarding — each test skips when its AMI is absent. The
hard-fail on the base ubuntu GPU AMI blocked the whole run even when the
ECS GPU AMI (the only one imported) could execute. Inventory the AMIs and
fail only when none are present.
* ci(gpu-e2e): self-provision base GPU AMI, hard-require it in preflight
The GPU E2E workflow was green-because-skipped: with no GPU AMIs imported
the self-guarding suite skipped every test, passing as a false green. Make
the raw-EC2 GPU passthrough test the always-on baseline.
- Add an import-if-absent setup step that pulls the prebuilt base GPU image
from the catalog by name (ubuntu-26.04-nvidia-gpu-x86_64), naming the AMI
exactly what tests/e2e/gpu filters for. Idempotent; caches on wattle's
persistent loop-fs. Runs spx under sg spinifex (local storage write).
- Preflight now hard-requires the base AMI (fail, not skip) so a silent
import failure can't pass as green; ECS/EKS node AMIs stay opt-in warn.
- Reconcile the local build path: ubuntu-gpu-nvidia.conf AMI_NAME and
launch-gpu.sh's nvidia default now use the same canonical name, so a
locally built+imported image also satisfies the suite.
* ci(gpu-e2e): pass --ami-name on base GPU import so the AMI name matches
spx admin images import --name selects the catalog entry but names the
resulting AMI generically (ami-<distro>-<version>-<arch>), so the import
landed as ami-ubuntu-26.04-x86_64 and the tests/e2e/gpu name filter never
matched. Pass --ami-name ubuntu-26.04-nvidia-gpu-x86_64 explicitly (same
flag build-system-image.sh uses) and poll briefly for availability.
* fix(network,vm): reconcile auto-assigned public-IP NAT + scale VFIO QMP timeout
Auto-assigned public IPs were never carried through the network reconciler.
applyEIPs ran AddEIP + ensureGuestPortDatapath for user EIPs only; auto-assigned
IPs (intent.Ports[].PublicIP) got egress exemption but no NAT re-assert and no
gatewayLRP->guest convergence. Their dnat_and_snat was created once at
RunInstances and never touched, so a vpcd/OVN rebuild dropped it and the
guest-port hop never converged: inbound DNATs at the gateway but never reaches
the guest (TCP/SSH dark) while ICMP is answered by the gateway LR (ping OK).
floatingIPSpecs now folds every auto-assigned/ELB public IP into the same
AddEIP + guest-port convergence path as user EIPs (deduped by private IP; a user
EIP wins), making the datapath self-healing across reconcile ticks and deploys.
qmpGreetingTimeout for VFIO guests now scales with guest RAM (base 30s + 8s/GiB,
floored at 180s, capped at 600s) instead of a flat 180s, so a large guest is not
SIGKILLed mid-pin while the whole guest RAM is IOMMU-locked before QMP answers.
* fix(vm): scale QMP handshake command deadline for VFIO launches
The greeting timeout scaling alone did not fix the VFIO launch SIGKILL: a GPU
guest sends its QMP greeting before the RAM pin completes, then cannot service
qmp_capabilities until the pin finishes. sendQMPCommand hardcoded a 30s response
deadline, so the handshake decode timed out (~43s for a 16 GiB g5.xlarge) and
the launch cleanup SIGKILLed a QEMU that would have come up.
Parameterize the command deadline (sendQMPCommandWithTimeout); the handshake now
uses the RAM-scaled qmpGreetingTimeout (base 30s + 8s/GiB, floor 180s, cap 600s)
for qmp_capabilities. Other callers keep the 30s default.
* test(gpu-e2e): share one GPU instance across passthrough subtests to avoid mid-suite IP recycle
Recycling both the floating and private IP onto a new ENI left the ingress
datapath targeting the terminated instance, so the successor was unreachable
on its EIP while IMDS stayed up.
- AddEIP re-points the dnat_and_snat row when the owning ENI changed even
though the external+logical IP are unchanged, keyed on the portable
spinifex:logical_port external-id (centralised mode leaves the native
LogicalPort column empty). The genuine same-ENI case still skips row churn.
- Flush the SB MAC_Binding for a private IP on ENI teardown and successor
bind so a reused address re-resolves to the live owner's MAC, via a new
MACBindingFlusher backed by ovn-sbctl.
- Reconciler prunes orphan dnat_and_snat rows whose stamped owning ENI is
absent from intent (leaked by lost vpc.delete-nat across dead VPCs).
- IMDS tap responder suspends retries for an ENI whose record is gone,
stopping the 15s reconcile churn.
* test(daemon): cover GPU inventory in GetNodes and GPU attachment in GetVMs
Adds unit tests for buildGPUInventory/handleNodeStatus (whole-GPU claimed/
free, MIG slices claimed/free, no-GPU node) and resolveVMGPU/handleNodeVMs
(whole-GPU attachment, MIG attachment, no attachment). Production code for
these paths (types.GPUInfo/GPUSliceInfo/VMGPUInfo, buildGPUInventory,
resolveVMGPU) and EC2 DescribeInstanceTypes GpuInfo.Gpus population
(instancetypes.GenerateGPUTypes/GenerateMIGTypes/IsGPUType) already landed
via the daemon GPU-admission work and were already test-covered for part C.
* feat(ui): shared GpuInstanceTypeSelect grouped instance-type picker
Reusable Select wrapper for EC2/EKS/ECS instance-type pickers: groups
Standard vs GPU-Accelerated types via structural GpuInfo.Gpus detection
(AWS SDK shape, no name heuristics), annotates GPU items with VRAM from
MemoryInfo.SizeInMiB and an optional per-type availability count, and
falls back to a single ungrouped list when no GPU types are present.
* feat(ui): EC2 GPU inventory, instance GPU card, shared picker in run-instances
* feat(ui): EKS nodegroup GPU picker + AMI/taint note
Add-nodegroup form now uses the shared GpuInstanceTypeSelect, grouping
and VRAM-annotating GPU instance types instead of a free-text field.
Selecting a GPU type surfaces a note that the GPU node AMI is
auto-selected and the nodegroup is tainted
nvidia.com/gpu=present:NoSchedule by default. Nodegroup rows badge GPU
nodegroups, derived from the primary instance type since
NodegroupRecord does not expose a GPU flag over the API.
* feat(ui): ECS GPU capacity picker, task-def GPU count, task gpuIds
* feat(eks): expose nodegroup GPU signal in Describe/ListNodegroups + UI badge
* chore(ui): rebuild embedded console dist for GPU UI
Bump github.com/mulgadc/viperblock to v1.11.1-0.20260713024608-eb4a376becc8
(dev tip through viperblock #34): bounded parallel chunk upload + coalesced
checkpoint (edou7 sec 7 items 2 and 9), sweeping the require up from #32 and
picking up the #33 write backpressure on the way. The nbdkit plugin the VM
write plane uses is built from the submodule source; this bump aligns the
viperblockd library that spx links.
* fix(daemon): gate GPU instance admission on cpu/mem and GPU-slot availability
canAllocateLocked early-returned count for whole-GPU (non-MIG) types,
bypassing the vCPU/memory admission calc and the live MemAvailable gate.
canAllocateCount already had a requiresGPU/availGPU branch, but every
production caller passed requiresGPU=false, making it dead code.
GPU instance types (whole-GPU and MIG) now flow through the same
cpu/mem + live-mem gate as every other type, with GPU-slot count wired
in as an additional hard constraint from gpuManager.Available(). This
also gives MIG a real up-front GPU-slice gate instead of only failing
late at Claim.
* fix(vm): release leaked GPU pool slot on crash-to-error and add reconciler backstop
HandleCrash freed vCPU/memory via deallocateResources but never called
InstanceCleaner.ReleaseGPU, since it bypasses the stop/terminate cleanup
chain that normally carries that call. A successful restart reuses the
still-set GPUAttachments, so that path is untouched, but when restarts
are exhausted, the instance type is unknown, there isn't enough capacity
to restart, or the relaunch itself fails, the instance settles
permanently in StateError still holding its GPU pool entry with nothing
to reclaim it.
Release the GPU and clear GPUAttachments on each of those terminal
paths. Add a GPU-pool reconciler mirroring the existing ENI reconciler,
wired into the daemon's reconcile loop, that periodically frees any pool
entry whose instance is absent or has settled into a non-running state,
as a backstop for a node-down mid-cascade or any other gap in the
synchronous release.
* fix(eks): keep derived ELBv2 names under 32 chars and surface CREATE_FAILED reason
Long cluster names made the derived "eks-<name>[-cp|-konn]" NLB/target-group
names exceed the ELBv2 32-char limit, failing cluster creation asynchronously
with no visible cause. safeELBv2Name now deterministically truncates the name
and appends a stable hash of the full cluster name when needed, so the same
name is derived on every call (create, lookup, delete). DescribeCluster also
now surfaces ClusterMeta.StatusReason as a Health.Issues entry when a cluster
is FAILED, instead of leaving callers to guess why.
* fix(eks): plumb per-nodegroup Ready-node counts through state reports
The on-VM state-report script now emits nodegroup_ready, a per-nodegroup Ready
count keyed by the eks.amazonaws.com/nodegroup label value, alongside the
existing cluster-wide node_count. ServerStateReport, the ClusterReconciler's
observe step, and SetClusterHealthState now carry this map through to
ClusterMeta.NodegroupNodeCounts, so per-nodegroup consumers (wired in the next
commit) can gate on a nodegroup's own workers instead of the cluster total.
* fix(eks): scope nodegroup Ready-gate per nodegroup and retry failed worker launches
waitWorkersReady compared the cluster-wide Ready-node count against a
nodegroup's baseline+desired, so in a multi-nodegroup cluster one nodegroup's
Ready workers could mask another whose own workers never registered. It now
reads meta.NodegroupNodeCounts[ng], scoped to nodes carrying the
eks.amazonaws.com/nodegroup=<name> label.
Separately, launchNodegroupInfra never retried a failed worker launch (a
transient QMP timeout or host capacity pressure), leaving the nodegroup stuck
CREATING with zero workers. The new launchWorkersUntilDesired wrapper relaunches
only the shortfall on failure, logging each retry, until the desired count is
reached or workerLaunchRetryTimeout elapses.
Both fixes are covered by regression tests: a two-nodegroup scenario where only
one has Ready workers, a transient-failure-then-success relaunch, and an
exhausted-retry-budget CREATE_FAILED path. Not covered here: a worker that dies
after the nodegroup is already ACTIVE — that needs an ongoing reconciler, which
this change does not add.
* fix(eks): tolerate GPU NoSchedule taint in konnectivity-agent DaemonSet
A GPU-only worker is tainted nvidia.com/gpu=present:NoSchedule, so the
konnectivity-agent DaemonSet (no tolerations) showed DESIRED=0 there and
kubectl logs/exec against pods on that node failed "No agent available".
Adds an Exists toleration for nvidia.com/gpu:NoSchedule.
Live-cluster validation (DESIRED count, kubectl exec) is deferred — no cluster
available to test against right now.
* fix: tolerate nvidia.com/gpu taint on ebs-csi-node and coredns DaemonSets so GPU workers get storage and DNS
A re-driven DeleteCluster whose managed control-plane VPC was already gone
(EC2 InvalidVpcID.NotFound) kept looping DependencyViolation: the internal
cp-vpc state record (ClusterMeta.ManagedCPVPC) was never cleared unless the
*entire* purge succeeded, so any unrelated flaky step retried CP-VPC/SG API
calls against a stale VpcId forever instead of converging.
purgeClusterInfra now clears ManagedCPVPC (via a new CAS-based
ClearClusterManagedCPVPC) as soon as DeleteClusterCPVPC + its SG cleanup both
succeed, independent of whatever else in that run still fails.
DeleteClusterCPVPC also now republishes vpc.delete for the CP VPC on every
call — including the already-gone fast path, falling back to the persisted
VpcId when the live tag-indexed describe can no longer name it — so vpcd's
topology manager gets another chance to GC an orphaned logical
router/subnets/SubnetEgressPriorityDrop policies if the original delete's
fire-and-forget event was ever lost.
handleRelease cancelled the per-lease renewal goroutine but deleted the
KV entry without waiting for it to exit. An in-flight reaffirm/renew Put
could land after the Delete, re-creating the lease and leaking a KV
record. Add a per-loop done channel and block on it after cancel so any
pending Put completes before the delete, making DrainAll's release path
deterministically synchronous.
* feat(eks): expose NVIDIA GPUs to pods via CDI device plugin
GPU nodegroups now label workers nvidia.com/gpu.present=true and taint them
nvidia.com/gpu=present:NoSchedule (via a k3s config.yaml.d drop-in, since k3s
has no K3S_NODE_TAINT env), and stage a hidden nvidia-device-plugin addon that
addon-sync renders into the K3s auto-deploy dir. The device plugin runs with
--device-list-strategy=cdi-cri and is gated to GPU nodes by nodeSelector +
toleration, advertising nvidia.com/gpu allocatable so pods land on GPU nodes
and get the device injected via the CDI spec baked by mulga-cdi-generate.
Relies on containerd 2.0.5 default enable_cdi=true (no config fork).
* fix(vm): scale QMP greeting deadline for VFIO passthrough guests
A GPU passthrough guest (-device vfio-pci + large -m) must lock and DMA-map
its entire guest RAM through the IOMMU before QEMU's QMP monitor answers,
which exceeds the fixed 30s greeting deadline and gets the VM SIGKILLed at
launch. Parameterize the greeting timeout (NewQMPClientWithGreetingTimeout)
and scale it to 180s for VMs with GPU attachments; plain VMs keep the 30s
default and reconnects against an already-running QEMU keep it too.
* fix(eks): register GPU workers reliably and expose GPUs to pods
Three fixes let a GPU nodegroup worker boot, register, and advertise its GPU:
- k3s-agent.service gains Delegate=yes (plus KillMode/TasksMax/limits and
overlay/br_netfilter modprobe): k3s embeds containerd+kubelet in-process, so
on cgroup-v2 the kubelet cannot create /kubepods without a delegated subtree
and ContainerManager.Start fails before Node registration.
- The GPU node label rides a k3s config drop-in (20-gpu.yaml node-label +
node-taint) instead of K3S_NODE_LABEL, which k3s ignores; without the label
the nvidia-device-plugin nodeSelector never matches.
- Raise the nodegroup ready timeout to 20m for the ~7min GPU first boot.
Adds the GPU e2e (VPC + NAT-GW egress harness) and a slog-restore unit test.
* fix(eks): install ecr-credential-provider on the GPU node AMI
The GPU worker's kubelet is started (from nodegroup userdata) with
--image-credential-provider-config + --image-credential-provider-bin-dir=
/usr/local/bin, but the eks-node-gpu AMI never built or installed the
ecr-credential-provider binary. kubelet treats a referenced-but-missing
provider binary as a fatal RegisterCredentialProviderPlugins error, so the
node crash-loops and never registers (while the Alpine eks-node worker, which
ships the binary, joins fine). Build + install it in the GPU manifest; the
server-only binaries stay omitted since GPU nodes are agent-only.
Also tee k3s-agent stderr (and its embedded kubelet klog) to the serial
console: Ubuntu logs only to journald, which is off-box-invisible on a worker
in an isolated customer VPC, so a kubelet that fails to register left no trace.
* fix(eks): give the nvidia-device-plugin its own GPU access
The CDI-mode device plugin (--device-list-strategy=cdi-cri) must run
nvml.Init and generate the CDI spec in its own container, but the DS had
no GPU wiring, so nvml.Init crash-looped and nvidia.com/gpu was never
advertised. Run the DS under the k3s-auto-configured nvidia runtimeClass
with NVIDIA_VISIBLE_DEVICES/CAPABILITIES=all, the host root mounted
read-only at /driver-root so it can locate the driver libs + ld.so.cache,
and /var/run/cdi mounted writable so the generated spec is visible to
containerd. No enable_cdi drop-in is needed on k3s v1.32.5 / containerd
2.0.5. GPU workload pods still need only resources.limits nvidia.com/gpu.
Live-validated end-to-end on the single-GPU host (RTX A1000): baked
control-plane AMI, GPU nodegroup, nvidia.com/gpu allocatable, nvidia-smi
pod, zero manual intervention.
* test(ec2): drop obsolete restoreSlogDefault test
Rebased onto dev, which dropped the restoreSlogDefault workaround in #506
(viperblock no longer mutates the global slog). The test referenced the
now-deleted helper and no longer compiles. Real GPU/registration diff is
independently covered (diff coverage 95.3%).
When no br-mgmt bridge is present (supported single-node config where system
instances reach AWSGW via the advertise IP), LaunchSystemInstance correctly
skips mgmt-tap creation but the NIC list still carried the mgmt slot, so
buildNICNetdevs emitted a QEMU netdev/device for a tap that was never created
and QEMU aborted (could not configure /dev/net/tun: Operation not permitted),
failing every CreateLoadBalancer/CreateCluster. Skip the mgmt NIC (index 1)
when its MAC is unset across netdevs, kernel-cmdline mmio slots, and the guest
fw_cfg netcfg blob; multi-node (br-mgmt present) is unchanged.
- 4ncdg: NATSRequest now round-trips the AWS error Message, not just the
sanitized Code. Server sends GenerateErrorPayloadWithMessage(code,
err.Error()); the client returns the Message when present so the CLI prints
the real reason (e.g. "snapshot not found; refusing to restore") instead of
a bare "ServerInternal". One code path fixes every NATS-driven command.
- fbmhl: the UI node view rendered raw host totals (vCPU:80, 0/125.5GB),
ignoring the daemon reserve. It now shows schedulable = total - reserved,
the capacity a tenant can actually consume. reserved_vcpu/reserved_mem_gb
added to the NodeInfo type (already present in the API payload).
Three related nat-mode fixes:
- ura01: nat-transit pool now emits gw_lrp_range_start/end. admin init
sets 100.127.0.16-.254 (239 VPC gateways) instead of the auto-derived
top-16 (~15). Adds GwLrpRange* to PoolData + the spinifex.toml template
so operators can widen any pool.
- 724mm: detectPoolMode returns true only for external_mode=pool. nat and
disabled clusters now skip Phase 8b/8d, which otherwise failed on
AllocateAddress UnsupportedOperation.
- pnz2i: document that system instances (ECS/EKS/LB agents) are
unsupported in nat v1 (require pool mode) in the vpc-networking README
and COMMANDS.md; document the gw_lrp_range_* fields.
* fix(eks): write etcd-snapshot.env at CP launch so snapshots actually run
* test(eks): cover resolveSystemPredastoreURL and buildEKSServiceDeps
Diff-coverage gate for 852f61d9 was unmet (48.8% < 60%) because the
mgmt-bridge branches and EKSServiceDeps assembly had no direct test.
* feat(eks): restore-snapshot DR path (fresh CP + cluster-reset restore + NLB re-point)
* fix(eks): make restore-snapshot fail-safe (validate snapshot, unwind on partial failure, honest status)
* fix(eks): sign etcd-snapshot uploads with deployment region (was 403ing)
* fix(eks): apply recovery directive on DR-seed first boot + single-node replacement host
* fix(daemon): log NATS handler failures at Error (were only on the trace span)
* feat(gpu): Ubuntu NVIDIA GPU node image + GPU AMI selection
Epic A of GPU passthrough to EKS pods / ECS task containers.
- Add gpu-vendor AMI tag constants + GPUVendorForType/IsGPUTypeName helpers
(instance-family prefix -> vendor from knownGPUModels).
- EKS + ECS worker AMI resolution branches to a GPU-tagged node image for GPU
instance types; no fallback to the driverless image (hard error instead).
- Exclude gpu-vendor-tagged AMIs from the plain (non-GPU) lookup so a newer GPU
AMI cannot hijack ordinary nodes (managed-by tag is shared; filters have no
negation, so exclude client-side).
- NodegroupRecord caches GPUEnabled/GPUVendor from InstanceTypes.
- Catalog entries spinifex-eks-node-gpu / spinifex-ecs-node-gpu (dual-tagged).
- New Ubuntu 26.04 GPU node images (eks-node-gpu, ecs-agent-gpu): reuse the
proven DKMS -server driver recipe (drop Docker CE + dev payload), add
nvidia-container-toolkit, OpenRC->systemd port of the worker/agent service
subset, udev EBS by-id bridge, nvidia modules-load, first-boot CDI generation.
- build-system-image.sh: SYSTEM_TAG supports multiple space-separated tags.
* build: allow SYSTEM_IMAGE_BUILD_DIR override for image build dir
Ubuntu GPU images are 16G; a RAM-backed /tmp tmpfs is too small to hold the
qcow2 + expanded raw. Default stays /tmp; SYSTEM_IMAGE_BUILD_DIR redirects the
build onto a real disk.
An ELBv2 LB whose system VM launches but whose in-guest lb-agent never
delivers a first heartbeat previously stayed in provisioning forever with
an empty StateReason: promotion to active happens only on the first
heartbeat and the only provisioning->failed transitions were at
create/launch time. A single wedged LB in the multi-node e2e lb suite
leaked its listener/target-group and cascaded into ResourceInUse and
DependencyViolation on VPC teardown.
- Add a lifecycle reaper that marks failed any LB still provisioning past
lbProvisioningTimeout (4m) with no heartbeat, reusing markLBFailed, and
populate a provisioning StateReason so a stuck LB is self-describing.
Started once at daemon startup; stopped on service Close.
- Harness: add ErrLBProvisioningTimeout and treat a provisioning-timeout
as retryable in createActiveLB; register idempotent LB/listener teardown
up-front so no exit path leaks into the shared VPC.
- IMDS: bounded retry+backoff on the instance-fact and user-data lookups so
a transient gather timeout under multi-node load does not starve the
lb-agent bootstrap.
The coordinated host-shutdown DRAIN phase transitioned every running VM to
StateStopped and migrated it to the operator-stopped shared KV bucket, evicting
it from node-local running state. On the next boot Restore only loads local
running state and treated every StateStopped instance as user-stopped, so
previously-running VMs never relaunched after a host reboot.
Gate on the existing operator-intent signal Attributes.StopInstance (set only by
StopInstances, cleared by StartInstance): stopOne keeps drain-stopped VMs in
local running state instead of migrating them, and Restore relaunches
drain-stopped VMs while operator-stopped ones stay stopped. markUnschedulable
now stamps StopInstance so capacity-demoted instances are not relaunched.
Julian Sommer