Expand the publish workflow to build the two Ubuntu NVIDIA GPU variants
alongside the Alpine eks-node/ecs-agent AMIs. The image build+publish
scripts are already manifest-driven and Ubuntu-aware; only the workflow
lacked the GPU legs.
Refactor the single-job body into a matrix over the four image manifests,
resolved by a new 'variant' input (all|cpu|gpu) so the slow 16G GPU legs
can be skipped when only the CPU AMIs need republishing. GPU images do not
fit a RAM-backed /tmp, so set SYSTEM_IMAGE_BUILD_DIR onto the runner's
on-disk _temp; legs run serially (max-parallel 1) on the single ci runner.
The nodegroup label was only written to K3S_NODE_LABEL in agent.env, but
k3s has no such env var, so non-GPU workers never received the label.
waitWorkersReady buckets Ready nodes by that label (added in #511), so
the affected nodegroups never counted toward their ACTIVE transition and
stayed CREATING until timeout.
Ride the label on a real k3s node-label config drop-in
(15-nodegroup-label.yaml), delivered via cloud-init user-data. Drop the
inert K3S_NODE_LABEL from agent.env and the initd export.
WaitForInstanceState now aborts immediately with the recorded StateReason
message when it observes a reaped state (shutting-down/terminated) while
waiting for a live target, instead of burning the full 5-minute timeout on
an instance that can never reach it.
MarkFailed takes a context and records the failure on the active span via
span.RecordError so a reaped launch surfaces in the APM error stream,
correlated by trace id. Only the LaunchRunInstances path carries a live
span; other callers pass context.Background() and the record is a no-op.
Surfaced by RCA of E2E run 29258398516 (TestAccountScoping/Step2).
TestNATGateway flaked with 100% egress loss: the tenant VPC gateway router's
OVN MAC_Binding had no entry for its upstream default-route nexthop. OVN
resolves that nexthop via lazy dynamic ARP on first egress; if that ARP is
lost during gateway bring-up, SNAT'd packets die in lr_in_arp_resolve and no
reply returns. MAC_Binding is per (datapath, logical_port, ip), so a sibling
VPC router resolving the same nexthop does not help.
Deterministically seed a static MAC binding for the nexthop at IGW attach. The
nexthop MAC is the host's own default-gateway MAC (the host shares the physical
uplink), read from ip neigh and written via ovn-nbctl static-mac-binding-add
(northd programs it into SB MAC_Binding, GC-safe). Host layer L0 owns the
shell-out per ADR-0006 S2; igw calls it via an injected hook. Best-effort and
idempotent: a missing MAC leaves dynamic ARP as the fallback, and it re-runs on
every AttachIGW so a binding lost to a later flap self-heals.
utils.GetOrCreateKVBucket hardcoded Replicas=1, so the ~38 handlers that
create their KV buckets lazily (VPC/subnet/ENI/EIP/ELBv2/ECS/ECR/ACM/
natgw/snapshot/placementgroup/IPAM/account-settings/dhcp/static-pool …)
created them at R1 on a multi-node cluster. upgradeJetStreamReplicas runs
once at startup, so any bucket created lazily after boot stayed R1
permanently — and an R1 stream's single Raft replica is transiently
leaderless during formation/join/rebalance, surfacing as
ErrNoResponders/ErrTimeout on KV ops.
Add a package-level default replica count that the daemon sets once at
boot (startCluster, after initJetStream) to len(clusterConfig.Nodes),
before any handler creates a bucket. GetOrCreateKVBucket now creates at
that default, so lazily-created buckets are born quorate. The one-shot
startup UpdateReplicas remains as a safety net for pre-existing R1
buckets from older deploys.
Zero call-site churn: the default flows through the existing
GetOrCreateKVBucketWithReplicas path already used by eks/ecs/iam.
TestSpawnRegisteredReconcilers_ResumesNonTerminal flaked two ways. First,
the shared fixture meta stamps a weeks-old CreatedAt, so a resumed CREATING
reconciler instantly trips the CREATE timeout, marks the cluster FAILED and
deregisters before the registry.Has assertion runs — the test was racing a
self-removal. Stamp a recent CreatedAt so the reconciler stays CREATING.
Second, SpawnRegisteredReconcilers ran a one-shot JetStream enumeration
(KeyValueStoreNames + kv.Keys), which is eventually consistent: under load a
just-committed cluster can be missing and never resumed, silently on a real
daemon. Re-scan until the observed CREATING/ACTIVE count stops growing, capped
and short-backed-off; Spawn is idempotent so re-runs only pick up stragglers.
discoverSingleNodeIPs seeds 127.0.0.1 then appends net.InterfaceAddrs()
in undefined order, so WANHost=nodeIPs[0] was loopback and the guest
gateway URL helpers fell through to the first non-loopback interface
addr — an arbitrary co-tenant bridge on multi-bridge shared hosts
(wattle: br-lan-env1..23, br-wan), leaving guest VMs unable to reach the
advertised gateway to register.
Resolve WANHost from spinifex.toml nodes.<node>.advertise (falling back
to that node's host), mirroring awsgwBindIP's toml scan; fall back to the
first non-loopback discovered addr only when config is unreadable. This
removes the need for the SPINIFEX_WAN_IP override on shared hosts.
The crash-supervisor auto-restart path (qmpHeartbeat -> HandleCrash ->
MaybeRestart -> startQEMU -> newQMPClientWithHandshake -> dialQMPWithRetry)
already unlinks the stale QMP socket and retries a refused/absent connect. But
a freshly relaunched QEMU can accept the connect then reset the connection
mid-greeting while it initialises, surfacing 'connection reset by peer'
immediately after 'QEMU started successfully' as a ServerInternal.
Add ECONNRESET and EPIPE to isTransientDialError so the dial is retried until
QEMU services the monitor, closing the auto-restart QMP-connect race.
Add non-streaming InvokeModel to bedrock-runtime. Unlike Converse's unified
schema, InvokeModel passes model-provider-native bodies through, so route by
model family: Anthropic Claude bodies forward to the Messages API (drop the
Bedrock-only anthropic_version, inject model, return verbatim); Meta Llama
bodies translate to/from the self-host vLLM OpenAI completions endpoint.
Raw-body dispatch bypasses the JSON response path (WriteRawResponse). Reuses
the Phase 1 credential + endpoint resolvers and upstream-status mapping.
Unit-tested (routing, body rewrite, translation, error classes) plus gated
InvokeModel E2E.
Both services already run otelsetup.Init and ship logs/metrics but never
opened a span, so they were invisible in the Kibana APM service map.
qmp-collector now opens a root span per per-instance poll tick (marks
error on QMP collect/publish failure); spinifex-ui now wraps its HTTP
handler chain with otelsetup.HTTPMiddleware and its reverse-proxy
transport with otelhttp, so UI requests get a server span and proxied
calls to awsgw/predastore carry traceparent instead of stopping the
trace at the UI hop.
nats and eks are intentionally left uninstrumented in this pass: nats is
the vendored message-broker process with no message-level hook to attach
spans to (the traffic it carries is already traced end-to-end at its
producer and consumer endpoints); eks request/response calls already
produce spans today under service.name=spinifex-daemon since the
handlers run in that process, and its async reconciler/state-machine
goroutines need a separate span design across a much larger surface.
Skip ERROR log when a subscription is already invalid
(nats.ErrBadSubscription) during graceful shutdown; log at debug
instead. Cosmetic cleanup race, no functional impact.
golangci-lint canonicalheader flagged 6 lowercase header literals in the
Ochre/Bedrock gateway (anthropic.go, anthropic_test.go, vllm.go). Canonical
form is wire-equivalent; canonicalise to X-Api-Key, Anthropic-Version and
Content-Type to restore CI lint.
* feat(bedrock): Ochre runtime MVP — Converse over self-host + Anthropic-direct
Add the AWS-Bedrock-compatible gateway services (bedrock, bedrock-runtime)
following the EKS REST-JSON pattern: ListFoundationModels / GetFoundationModel
over a static connectivity-tiered catalog, and non-streaming Converse routed
per modelId to one of two provider backends — self-hosted vLLM (OpenAI wire)
and Anthropic-direct (Anthropic Messages wire).
Per-account provider API keys are a master-key-encrypted JetStream KV secret
with an optional platform default; the advertised catalog gates external
models on a resolvable credential. Self-host endpoints are pinned via config
(OCHRE_VLLM_ENDPOINTS) behind an EndpointResolver seam.
Adds gateway/bedrock/ (catalog, provider router, Anthropic + vLLM adapters,
credential store), awserrors REST-JSON codes, and the bedrock E2E suite
(list/get light; Converse gated on live backends). Daemon/NATS control plane
and dynamic endpoint lifecycle are deferred to Phase 3.
* test(bedrock): cover dispatch, router, credentials KV + endpoint config
Add unit coverage for the Ochre Phase 1 diff: Bedrock_Request /
BedrockRuntime_Request end-to-end dispatch, provider router, JSON
response/error writers, upstream-status mapping, the credential-store KV
round-trip, the Anthropic bound-provider path, and parseBedrockEndpoints.
Lifts PR diff coverage above the 60% gate.
* ci: GPU e2e workflow on ci-gpu self-hosted runner
Runs tests/e2e/gpu (EC2/ECS/EKS GPU passthrough) against the persistent,
hand-bootstrapped wattle shared host, driven over SSH by a banksia runner
labelled ci-gpu (matches the ci-single/ci-multi convention) — no runner
installed on the shared box.
- runs-on [self-hosted, ci-gpu]; concurrency ci-gpu, cancel-in-progress
false (single A1000 = single-flight, queue not stomp)
- workflow_dispatch + GPU-path filter (not every push)
- manual-work mutex gate (~/.ci-gpu-paused / CI_GPU_PAUSED) so hands-on GPU
testing blocks CI
- if: always() cleanup terminates launched GPU instances; never resets the
box (no setup-ovn.sh/dev-reset/cluster-reset)
- drives the control plane with spx + the already-configured aws spinifex
profile directly (no wrapper scripts, nothing to install on wattle)
* ci(gpu-e2e): advisory AMI preflight, fail only if zero GPU AMIs
The suite is self-guarding — each test skips when its AMI is absent. The
hard-fail on the base ubuntu GPU AMI blocked the whole run even when the
ECS GPU AMI (the only one imported) could execute. Inventory the AMIs and
fail only when none are present.
* ci(gpu-e2e): self-provision base GPU AMI, hard-require it in preflight
The GPU E2E workflow was green-because-skipped: with no GPU AMIs imported
the self-guarding suite skipped every test, passing as a false green. Make
the raw-EC2 GPU passthrough test the always-on baseline.
- Add an import-if-absent setup step that pulls the prebuilt base GPU image
from the catalog by name (ubuntu-26.04-nvidia-gpu-x86_64), naming the AMI
exactly what tests/e2e/gpu filters for. Idempotent; caches on wattle's
persistent loop-fs. Runs spx under sg spinifex (local storage write).
- Preflight now hard-requires the base AMI (fail, not skip) so a silent
import failure can't pass as green; ECS/EKS node AMIs stay opt-in warn.
- Reconcile the local build path: ubuntu-gpu-nvidia.conf AMI_NAME and
launch-gpu.sh's nvidia default now use the same canonical name, so a
locally built+imported image also satisfies the suite.
* ci(gpu-e2e): pass --ami-name on base GPU import so the AMI name matches
spx admin images import --name selects the catalog entry but names the
resulting AMI generically (ami-<distro>-<version>-<arch>), so the import
landed as ami-ubuntu-26.04-x86_64 and the tests/e2e/gpu name filter never
matched. Pass --ami-name ubuntu-26.04-nvidia-gpu-x86_64 explicitly (same
flag build-system-image.sh uses) and poll briefly for availability.
* fix(network,vm): reconcile auto-assigned public-IP NAT + scale VFIO QMP timeout
Auto-assigned public IPs were never carried through the network reconciler.
applyEIPs ran AddEIP + ensureGuestPortDatapath for user EIPs only; auto-assigned
IPs (intent.Ports[].PublicIP) got egress exemption but no NAT re-assert and no
gatewayLRP->guest convergence. Their dnat_and_snat was created once at
RunInstances and never touched, so a vpcd/OVN rebuild dropped it and the
guest-port hop never converged: inbound DNATs at the gateway but never reaches
the guest (TCP/SSH dark) while ICMP is answered by the gateway LR (ping OK).
floatingIPSpecs now folds every auto-assigned/ELB public IP into the same
AddEIP + guest-port convergence path as user EIPs (deduped by private IP; a user
EIP wins), making the datapath self-healing across reconcile ticks and deploys.
qmpGreetingTimeout for VFIO guests now scales with guest RAM (base 30s + 8s/GiB,
floored at 180s, capped at 600s) instead of a flat 180s, so a large guest is not
SIGKILLed mid-pin while the whole guest RAM is IOMMU-locked before QMP answers.
* fix(vm): scale QMP handshake command deadline for VFIO launches
The greeting timeout scaling alone did not fix the VFIO launch SIGKILL: a GPU
guest sends its QMP greeting before the RAM pin completes, then cannot service
qmp_capabilities until the pin finishes. sendQMPCommand hardcoded a 30s response
deadline, so the handshake decode timed out (~43s for a 16 GiB g5.xlarge) and
the launch cleanup SIGKILLed a QEMU that would have come up.
Parameterize the command deadline (sendQMPCommandWithTimeout); the handshake now
uses the RAM-scaled qmpGreetingTimeout (base 30s + 8s/GiB, floor 180s, cap 600s)
for qmp_capabilities. Other callers keep the 30s default.
* test(gpu-e2e): share one GPU instance across passthrough subtests to avoid mid-suite IP recycle
Recycling both the floating and private IP onto a new ENI left the ingress
datapath targeting the terminated instance, so the successor was unreachable
on its EIP while IMDS stayed up.
- AddEIP re-points the dnat_and_snat row when the owning ENI changed even
though the external+logical IP are unchanged, keyed on the portable
spinifex:logical_port external-id (centralised mode leaves the native
LogicalPort column empty). The genuine same-ENI case still skips row churn.
- Flush the SB MAC_Binding for a private IP on ENI teardown and successor
bind so a reused address re-resolves to the live owner's MAC, via a new
MACBindingFlusher backed by ovn-sbctl.
- Reconciler prunes orphan dnat_and_snat rows whose stamped owning ENI is
absent from intent (leaked by lost vpc.delete-nat across dead VPCs).
- IMDS tap responder suspends retries for an ENI whose record is gone,
stopping the 15s reconcile churn.
* test(daemon): cover GPU inventory in GetNodes and GPU attachment in GetVMs
Adds unit tests for buildGPUInventory/handleNodeStatus (whole-GPU claimed/
free, MIG slices claimed/free, no-GPU node) and resolveVMGPU/handleNodeVMs
(whole-GPU attachment, MIG attachment, no attachment). Production code for
these paths (types.GPUInfo/GPUSliceInfo/VMGPUInfo, buildGPUInventory,
resolveVMGPU) and EC2 DescribeInstanceTypes GpuInfo.Gpus population
(instancetypes.GenerateGPUTypes/GenerateMIGTypes/IsGPUType) already landed
via the daemon GPU-admission work and were already test-covered for part C.
* feat(ui): shared GpuInstanceTypeSelect grouped instance-type picker
Reusable Select wrapper for EC2/EKS/ECS instance-type pickers: groups
Standard vs GPU-Accelerated types via structural GpuInfo.Gpus detection
(AWS SDK shape, no name heuristics), annotates GPU items with VRAM from
MemoryInfo.SizeInMiB and an optional per-type availability count, and
falls back to a single ungrouped list when no GPU types are present.
* feat(ui): EC2 GPU inventory, instance GPU card, shared picker in run-instances
* feat(ui): EKS nodegroup GPU picker + AMI/taint note
Add-nodegroup form now uses the shared GpuInstanceTypeSelect, grouping
and VRAM-annotating GPU instance types instead of a free-text field.
Selecting a GPU type surfaces a note that the GPU node AMI is
auto-selected and the nodegroup is tainted
nvidia.com/gpu=present:NoSchedule by default. Nodegroup rows badge GPU
nodegroups, derived from the primary instance type since
NodegroupRecord does not expose a GPU flag over the API.
* feat(ui): ECS GPU capacity picker, task-def GPU count, task gpuIds
* feat(eks): expose nodegroup GPU signal in Describe/ListNodegroups + UI badge
* chore(ui): rebuild embedded console dist for GPU UI
Bump github.com/mulgadc/viperblock to v1.11.1-0.20260713024608-eb4a376becc8
(dev tip through viperblock #34): bounded parallel chunk upload + coalesced
checkpoint (edou7 sec 7 items 2 and 9), sweeping the require up from #32 and
picking up the #33 write backpressure on the way. The nbdkit plugin the VM
write plane uses is built from the submodule source; this bump aligns the
viperblockd library that spx links.
* fix(daemon): gate GPU instance admission on cpu/mem and GPU-slot availability
canAllocateLocked early-returned count for whole-GPU (non-MIG) types,
bypassing the vCPU/memory admission calc and the live MemAvailable gate.
canAllocateCount already had a requiresGPU/availGPU branch, but every
production caller passed requiresGPU=false, making it dead code.
GPU instance types (whole-GPU and MIG) now flow through the same
cpu/mem + live-mem gate as every other type, with GPU-slot count wired
in as an additional hard constraint from gpuManager.Available(). This
also gives MIG a real up-front GPU-slice gate instead of only failing
late at Claim.
* fix(vm): release leaked GPU pool slot on crash-to-error and add reconciler backstop
HandleCrash freed vCPU/memory via deallocateResources but never called
InstanceCleaner.ReleaseGPU, since it bypasses the stop/terminate cleanup
chain that normally carries that call. A successful restart reuses the
still-set GPUAttachments, so that path is untouched, but when restarts
are exhausted, the instance type is unknown, there isn't enough capacity
to restart, or the relaunch itself fails, the instance settles
permanently in StateError still holding its GPU pool entry with nothing
to reclaim it.
Release the GPU and clear GPUAttachments on each of those terminal
paths. Add a GPU-pool reconciler mirroring the existing ENI reconciler,
wired into the daemon's reconcile loop, that periodically frees any pool
entry whose instance is absent or has settled into a non-running state,
as a backstop for a node-down mid-cascade or any other gap in the
synchronous release.
* fix(eks): keep derived ELBv2 names under 32 chars and surface CREATE_FAILED reason
Long cluster names made the derived "eks-<name>[-cp|-konn]" NLB/target-group
names exceed the ELBv2 32-char limit, failing cluster creation asynchronously
with no visible cause. safeELBv2Name now deterministically truncates the name
and appends a stable hash of the full cluster name when needed, so the same
name is derived on every call (create, lookup, delete). DescribeCluster also
now surfaces ClusterMeta.StatusReason as a Health.Issues entry when a cluster
is FAILED, instead of leaving callers to guess why.
* fix(eks): plumb per-nodegroup Ready-node counts through state reports
The on-VM state-report script now emits nodegroup_ready, a per-nodegroup Ready
count keyed by the eks.amazonaws.com/nodegroup label value, alongside the
existing cluster-wide node_count. ServerStateReport, the ClusterReconciler's
observe step, and SetClusterHealthState now carry this map through to
ClusterMeta.NodegroupNodeCounts, so per-nodegroup consumers (wired in the next
commit) can gate on a nodegroup's own workers instead of the cluster total.
* fix(eks): scope nodegroup Ready-gate per nodegroup and retry failed worker launches
waitWorkersReady compared the cluster-wide Ready-node count against a
nodegroup's baseline+desired, so in a multi-nodegroup cluster one nodegroup's
Ready workers could mask another whose own workers never registered. It now
reads meta.NodegroupNodeCounts[ng], scoped to nodes carrying the
eks.amazonaws.com/nodegroup=<name> label.
Separately, launchNodegroupInfra never retried a failed worker launch (a
transient QMP timeout or host capacity pressure), leaving the nodegroup stuck
CREATING with zero workers. The new launchWorkersUntilDesired wrapper relaunches
only the shortfall on failure, logging each retry, until the desired count is
reached or workerLaunchRetryTimeout elapses.
Both fixes are covered by regression tests: a two-nodegroup scenario where only
one has Ready workers, a transient-failure-then-success relaunch, and an
exhausted-retry-budget CREATE_FAILED path. Not covered here: a worker that dies
after the nodegroup is already ACTIVE — that needs an ongoing reconciler, which
this change does not add.
* fix(eks): tolerate GPU NoSchedule taint in konnectivity-agent DaemonSet
A GPU-only worker is tainted nvidia.com/gpu=present:NoSchedule, so the
konnectivity-agent DaemonSet (no tolerations) showed DESIRED=0 there and
kubectl logs/exec against pods on that node failed "No agent available".
Adds an Exists toleration for nvidia.com/gpu:NoSchedule.
Live-cluster validation (DESIRED count, kubectl exec) is deferred — no cluster
available to test against right now.
* fix: tolerate nvidia.com/gpu taint on ebs-csi-node and coredns DaemonSets so GPU workers get storage and DNS
A re-driven DeleteCluster whose managed control-plane VPC was already gone
(EC2 InvalidVpcID.NotFound) kept looping DependencyViolation: the internal
cp-vpc state record (ClusterMeta.ManagedCPVPC) was never cleared unless the
*entire* purge succeeded, so any unrelated flaky step retried CP-VPC/SG API
calls against a stale VpcId forever instead of converging.
purgeClusterInfra now clears ManagedCPVPC (via a new CAS-based
ClearClusterManagedCPVPC) as soon as DeleteClusterCPVPC + its SG cleanup both
succeed, independent of whatever else in that run still fails.
DeleteClusterCPVPC also now republishes vpc.delete for the CP VPC on every
call — including the already-gone fast path, falling back to the persisted
VpcId when the live tag-indexed describe can no longer name it — so vpcd's
topology manager gets another chance to GC an orphaned logical
router/subnets/SubnetEgressPriorityDrop policies if the original delete's
fire-and-forget event was ever lost.
handleRelease cancelled the per-lease renewal goroutine but deleted the
KV entry without waiting for it to exit. An in-flight reaffirm/renew Put
could land after the Delete, re-creating the lease and leaking a KV
record. Add a per-loop done channel and block on it after cancel so any
pending Put completes before the delete, making DrainAll's release path
deterministically synchronous.
* feat(eks): expose NVIDIA GPUs to pods via CDI device plugin
GPU nodegroups now label workers nvidia.com/gpu.present=true and taint them
nvidia.com/gpu=present:NoSchedule (via a k3s config.yaml.d drop-in, since k3s
has no K3S_NODE_TAINT env), and stage a hidden nvidia-device-plugin addon that
addon-sync renders into the K3s auto-deploy dir. The device plugin runs with
--device-list-strategy=cdi-cri and is gated to GPU nodes by nodeSelector +
toleration, advertising nvidia.com/gpu allocatable so pods land on GPU nodes
and get the device injected via the CDI spec baked by mulga-cdi-generate.
Relies on containerd 2.0.5 default enable_cdi=true (no config fork).
* fix(vm): scale QMP greeting deadline for VFIO passthrough guests
A GPU passthrough guest (-device vfio-pci + large -m) must lock and DMA-map
its entire guest RAM through the IOMMU before QEMU's QMP monitor answers,
which exceeds the fixed 30s greeting deadline and gets the VM SIGKILLed at
launch. Parameterize the greeting timeout (NewQMPClientWithGreetingTimeout)
and scale it to 180s for VMs with GPU attachments; plain VMs keep the 30s
default and reconnects against an already-running QEMU keep it too.
* fix(eks): register GPU workers reliably and expose GPUs to pods
Three fixes let a GPU nodegroup worker boot, register, and advertise its GPU:
- k3s-agent.service gains Delegate=yes (plus KillMode/TasksMax/limits and
overlay/br_netfilter modprobe): k3s embeds containerd+kubelet in-process, so
on cgroup-v2 the kubelet cannot create /kubepods without a delegated subtree
and ContainerManager.Start fails before Node registration.
- The GPU node label rides a k3s config drop-in (20-gpu.yaml node-label +
node-taint) instead of K3S_NODE_LABEL, which k3s ignores; without the label
the nvidia-device-plugin nodeSelector never matches.
- Raise the nodegroup ready timeout to 20m for the ~7min GPU first boot.
Adds the GPU e2e (VPC + NAT-GW egress harness) and a slog-restore unit test.
* fix(eks): install ecr-credential-provider on the GPU node AMI
The GPU worker's kubelet is started (from nodegroup userdata) with
--image-credential-provider-config + --image-credential-provider-bin-dir=
/usr/local/bin, but the eks-node-gpu AMI never built or installed the
ecr-credential-provider binary. kubelet treats a referenced-but-missing
provider binary as a fatal RegisterCredentialProviderPlugins error, so the
node crash-loops and never registers (while the Alpine eks-node worker, which
ships the binary, joins fine). Build + install it in the GPU manifest; the
server-only binaries stay omitted since GPU nodes are agent-only.
Also tee k3s-agent stderr (and its embedded kubelet klog) to the serial
console: Ubuntu logs only to journald, which is off-box-invisible on a worker
in an isolated customer VPC, so a kubelet that fails to register left no trace.
* fix(eks): give the nvidia-device-plugin its own GPU access
The CDI-mode device plugin (--device-list-strategy=cdi-cri) must run
nvml.Init and generate the CDI spec in its own container, but the DS had
no GPU wiring, so nvml.Init crash-looped and nvidia.com/gpu was never
advertised. Run the DS under the k3s-auto-configured nvidia runtimeClass
with NVIDIA_VISIBLE_DEVICES/CAPABILITIES=all, the host root mounted
read-only at /driver-root so it can locate the driver libs + ld.so.cache,
and /var/run/cdi mounted writable so the generated spec is visible to
containerd. No enable_cdi drop-in is needed on k3s v1.32.5 / containerd
2.0.5. GPU workload pods still need only resources.limits nvidia.com/gpu.
Live-validated end-to-end on the single-GPU host (RTX A1000): baked
control-plane AMI, GPU nodegroup, nvidia.com/gpu allocatable, nvidia-smi
pod, zero manual intervention.
* test(ec2): drop obsolete restoreSlogDefault test
Rebased onto dev, which dropped the restoreSlogDefault workaround in #506
(viperblock no longer mutates the global slog). The test referenced the
now-deleted helper and no longer compiles. Real GPU/registration diff is
independently covered (diff coverage 95.3%).
When no br-mgmt bridge is present (supported single-node config where system
instances reach AWSGW via the advertise IP), LaunchSystemInstance correctly
skips mgmt-tap creation but the NIC list still carried the mgmt slot, so
buildNICNetdevs emitted a QEMU netdev/device for a tap that was never created
and QEMU aborted (could not configure /dev/net/tun: Operation not permitted),
failing every CreateLoadBalancer/CreateCluster. Skip the mgmt NIC (index 1)
when its MAC is unset across netdevs, kernel-cmdline mmio slots, and the guest
fw_cfg netcfg blob; multi-node (br-mgmt present) is unchanged.
- 4ncdg: NATSRequest now round-trips the AWS error Message, not just the
sanitized Code. Server sends GenerateErrorPayloadWithMessage(code,
err.Error()); the client returns the Message when present so the CLI prints
the real reason (e.g. "snapshot not found; refusing to restore") instead of
a bare "ServerInternal". One code path fixes every NATS-driven command.
- fbmhl: the UI node view rendered raw host totals (vCPU:80, 0/125.5GB),
ignoring the daemon reserve. It now shows schedulable = total - reserved,
the capacity a tenant can actually consume. reserved_vcpu/reserved_mem_gb
added to the NodeInfo type (already present in the API payload).
Three related nat-mode fixes:
- ura01: nat-transit pool now emits gw_lrp_range_start/end. admin init
sets 100.127.0.16-.254 (239 VPC gateways) instead of the auto-derived
top-16 (~15). Adds GwLrpRange* to PoolData + the spinifex.toml template
so operators can widen any pool.
- 724mm: detectPoolMode returns true only for external_mode=pool. nat and
disabled clusters now skip Phase 8b/8d, which otherwise failed on
AllocateAddress UnsupportedOperation.
- pnz2i: document that system instances (ECS/EKS/LB agents) are
unsupported in nat v1 (require pool mode) in the vpc-networking README
and COMMANDS.md; document the gw_lrp_range_* fields.
* fix(eks): write etcd-snapshot.env at CP launch so snapshots actually run
* test(eks): cover resolveSystemPredastoreURL and buildEKSServiceDeps
Diff-coverage gate for 852f61d9 was unmet (48.8% < 60%) because the
mgmt-bridge branches and EKSServiceDeps assembly had no direct test.
* feat(eks): restore-snapshot DR path (fresh CP + cluster-reset restore + NLB re-point)
* fix(eks): make restore-snapshot fail-safe (validate snapshot, unwind on partial failure, honest status)
* fix(eks): sign etcd-snapshot uploads with deployment region (was 403ing)
* fix(eks): apply recovery directive on DR-seed first boot + single-node replacement host
* fix(daemon): log NATS handler failures at Error (were only on the trace span)
* feat(gpu): Ubuntu NVIDIA GPU node image + GPU AMI selection
Epic A of GPU passthrough to EKS pods / ECS task containers.
- Add gpu-vendor AMI tag constants + GPUVendorForType/IsGPUTypeName helpers
(instance-family prefix -> vendor from knownGPUModels).
- EKS + ECS worker AMI resolution branches to a GPU-tagged node image for GPU
instance types; no fallback to the driverless image (hard error instead).
- Exclude gpu-vendor-tagged AMIs from the plain (non-GPU) lookup so a newer GPU
AMI cannot hijack ordinary nodes (managed-by tag is shared; filters have no
negation, so exclude client-side).
- NodegroupRecord caches GPUEnabled/GPUVendor from InstanceTypes.
- Catalog entries spinifex-eks-node-gpu / spinifex-ecs-node-gpu (dual-tagged).
- New Ubuntu 26.04 GPU node images (eks-node-gpu, ecs-agent-gpu): reuse the
proven DKMS -server driver recipe (drop Docker CE + dev payload), add
nvidia-container-toolkit, OpenRC->systemd port of the worker/agent service
subset, udev EBS by-id bridge, nvidia modules-load, first-boot CDI generation.
- build-system-image.sh: SYSTEM_TAG supports multiple space-separated tags.
* build: allow SYSTEM_IMAGE_BUILD_DIR override for image build dir
Ubuntu GPU images are 16G; a RAM-backed /tmp tmpfs is too small to hold the
qcow2 + expanded raw. Default stays /tmp; SYSTEM_IMAGE_BUILD_DIR redirects the
build onto a real disk.
An ELBv2 LB whose system VM launches but whose in-guest lb-agent never
delivers a first heartbeat previously stayed in provisioning forever with
an empty StateReason: promotion to active happens only on the first
heartbeat and the only provisioning->failed transitions were at
create/launch time. A single wedged LB in the multi-node e2e lb suite
leaked its listener/target-group and cascaded into ResourceInUse and
DependencyViolation on VPC teardown.
- Add a lifecycle reaper that marks failed any LB still provisioning past
lbProvisioningTimeout (4m) with no heartbeat, reusing markLBFailed, and
populate a provisioning StateReason so a stuck LB is self-describing.
Started once at daemon startup; stopped on service Close.
- Harness: add ErrLBProvisioningTimeout and treat a provisioning-timeout
as retryable in createActiveLB; register idempotent LB/listener teardown
up-front so no exit path leaks into the shared VPC.
- IMDS: bounded retry+backoff on the instance-fact and user-data lookups so
a transient gather timeout under multi-node load does not starve the
lb-agent bootstrap.
The coordinated host-shutdown DRAIN phase transitioned every running VM to
StateStopped and migrated it to the operator-stopped shared KV bucket, evicting
it from node-local running state. On the next boot Restore only loads local
running state and treated every StateStopped instance as user-stopped, so
previously-running VMs never relaunched after a host reboot.
Gate on the existing operator-intent signal Attributes.StopInstance (set only by
StopInstances, cleared by StartInstance): stopOne keeps drain-stopped VMs in
local running state instead of migrating them, and Restore relaunches
drain-stopped VMs while operator-stopped ones stay stopped. markUnschedulable
now stamps StopInstance so capacity-demoted instances are not relaunched.
Close four pre-existing lifecycle races surfaced during env19 hardening.
Volume seal/reaper (42hbm, c7ce2): the ebs.unmount handler now seals the
volume before removing it from MountedVolumes, so a retry after a
timed-out-but-completed seal reliably reports NotFound. EBSUnMountResponse
carries a NotFound flag and unmountResponseError treats it as seal success,
so state reconciles to available/detached instead of leaking. The volume
leak reaper reconciles the attachment record (never the data) for volumes
whose owning instance is terminated, so GC can delete them.
Stopped-instance CAS (ozlbq, wlmea): add UpdateStoppedInstance, a
mutate-callback backed by casUpdate(createIfAbsent=false), and migrate
TagStoppedInstance and ModifyInstanceAttribute off the resurrect-prone
Load->mutate->WriteStoppedInstance path. A tag/attribute write racing a
winning start-claim now fails cleanly instead of re-creating the deleted
entry. StartStoppedInstance returns IncorrectInstanceState (not
InvalidInstanceID.NotFound) when the stopped entry is absent but the
instance is present locally.
Every tofu-based e2e workflow set TOFU_LOG to a /tmp path outside
ARTIFACT_DIR. ci-fmt.sh's quiet_run helper appends all tofu init/apply/
destroy output to that file, but the Teardown step (which runs the
destroy) always ran AFTER the existing upload-artifact step, so the
full log — including destroy-time failures — was silently discarded
on every run.
Add an if: always() upload-artifact step after Teardown in each
tofu-based job (e2e.yml, e2e-suites.yml, e2e-nightly.yml, e2e-ddil.yml)
that uploads $TOFU_LOG directly. Since quiet_run appends both apply and
destroy output to the same file, one upload after Teardown now
captures everything, recoverable even when destroy itself fails.
Shipping this log to Elasticsearch/OTLP is deferred to mulga-5n7fe,
which will consume this artifact.
Init now installs an sdk/log LoggerProvider inside the existing
exportEnabled() gate, sharing the same resource as the tracer/meter (incl.
ci.run_id via OTEL_RESOURCE_ATTRIBUTES) and joining lp.Shutdown into the
returned shutdown func. The process default slog handler is fanned out to
both the existing stdout/journald handler and an otelslog bridge, so app
logs export over OTLP with trace correlation while journald output is
unchanged. With no OTLP endpoint configured Init remains a no-op: no
LoggerProvider is installed and the default logger is left untouched.
PR #487 turned attach/detach flaky: reconnectQMP dialed a fresh QMP
client before closing the old connection. QEMU's monitor is a
single-client `server,nowait` listener, so while the old conn holds the
slot the redial gets no greeting, blocks ~30s, and fails, leaving
q.Dead permanently true and every later attach/detach ServerInternal.
Fix the ordering (close, then dial via dialQMPWithRetry to absorb the
brief post-close ECONNREFUSED window).
waitForDeviceDeletedEvent marked q.Dead on any decode error, including
a benign read-deadline timeout — routine, since DEVICE_DELETED is
frequently pre-swallowed by device_del's own reply loop. The
underlying connection survives a timeout, but encoding/json.Decoder
permanently caches read errors on itself, so on a timeout the Decoder
is swapped for a fresh one over the same conn instead of forcing a
full QMP reconnect; q.Dead is now only set on a genuine decode error.
Also: an AttachVolume re-attach to the same instance with a different
device now returns VolumeInUse (matching AWS) instead of
InvalidParameterValue.
* feat(ec2,ebs): harden volume attach/detach + instance lifecycle state machine
Batch of related EBS/instance-lifecycle correctness fixes verified together.
- u94vs: add compare-and-swap (casUpdate/casPut via nats KV Update revision)
to the multi-writer lifecycle KV sites (WriteTerminatedInstance,
WriteStoppedInstance, WriteClusterShutdown); make the teardown-reaper
read-modify-write atomic via a merge callback so retries never re-run
cleaner side effects.
- yi2ed: add ClaimStoppedInstance, a revision-checked delete-as-claim on the
stopped bucket performed before Allocate/Insert/Run, so a forwarded start
and a local fallback cannot double-start one instance onto the same volume.
- siv-473: make AttachVolume idempotent — return attached success when the
volume is already attached to the requesting instance; a conflicting
requested device returns InvalidParameterValue.
- siv-462: reconnect now re-asserts in-use for all attached volumes (boot and
non-boot data volumes), not just boot, so a daemon restart no longer leaves
stateful-pod data volumes split-brain; EFI pflash is skipped.
- 9w2mv: wait for the QMP DEVICE_DELETED event (bounded) before blockdev-del
instead of a blind sleep, so a slow guest unplug no longer exhausts retries.
- soyah: poll (50ms) instead of fixed 1s sleeps in KillProcess and the
viperblockd nbdkit confirm-running path, cutting volume-lifecycle latency.
- siv-481: a cross-node start forward that times out now falls back to a local
start; this is safe because the yi2ed claim is the single serialization point.
* fix(eks): create per-account and leader KV buckets with cluster replica count
The EKS per-account KV bucket and the leader lease bucket were created at
Replicas=1 and never upgraded, since they're created lazily after
upgradeJetStreamReplicas runs at startup. On multi-node clusters this leaves
their single-replica Raft group transiently leaderless during formation,
causing KV timeouts that surface as a deterministic 503 on ListClusters.
Thread clusterSize through GetOrCreateAccountBucket/InitLeaderBucket (backed
by a new utils.GetOrCreateKVBucketWithReplicas) so both buckets are created
at max(clusterSize,1) replicas, matching every other node-count-scaled
JetStream stream.
A fresh cluster or SB re-bootstrap could leave a node's ovn-controller with
its OVSDB client latched on stale Southbound RAFT cluster state, looping on
"clustered database server has stale data" and silently no longer realising
new Port_Bindings. Cross-host CP data-NICs then never bind: no DHCP, no IMDS,
degraded boot. A manual `systemctl restart ovn-controller` was the only
recovery. Root cause is deterministic: setup-ovn.sh restarts ovn-controller
per node as each RAFT member forms, so a controller started for an early
member races the SB membership changes that later joins trigger.
Add three layers of recovery (the ansible bring-up settle pass that fixes this
at the source lives in the mulga repo):
- Per-host wedge watchdog (vpcd/ovn_controller_watchdog.go): not leader-gated
since the wedge is per chassis. Polls SB connection-status; after a stale
threshold issues sb-cluster-state-reset, throttled by a cooldown. Decision
logic takes an injected clock for deterministic tests.
- Reconcile-loop escalation: ensureGatewayClaimed and ensureGuestPortDatapath
escalate to one sb-cluster-state-reset after repeated recompute misses only
when the SB is not connected — a recompute cannot clear a stale-SB wedge.
- Prober primitives SBConnectionState/ResetSBClusterState on GatewayClaimProber
plus the GatewayClaimVerifier interface extension.
Also fix the awsgw OVN health probe, which passed a slashed `-t` socket path
that never resolves and so always reported ovn-controller not_running. Probe
connection-status instead: OVNControllerUp is now true only when connected, so
healthy nodes read ok and a real wedge correctly reads not_running.
The EIP "verify unreachable after disassociate" step polls runSSHQuiet
inside EventuallyErr(90s), but a wedged ssh made os/exec Wait block for
14 minutes — EventuallyErr cannot interrupt an in-flight CombinedOutput,
so one probe consumed the whole 30-min test-binary budget and panicked.
The panic skipped t.Cleanup, leaking the EIP and NAT-gateway VMs, which
held single-node capacity and starved the lb suite's sys.micro launch
(the daemon unsubscribes the full topic, so NATS returns no-responders).
Bound the ssh child at the process layer with exec.CommandContext (30s
deadline) plus WaitDelay, returning a descriptive timeout error. This is
version- and phase-independent, unlike ssh's own ConnectTimeout. 30s is
safe for all callers (worst legit call is ~15s) and well under the 90s
poll budget, so a wedged probe now yields a clean bounded failure with
cleanup intact.
* feat(network): add routed uplink mode, Routed wiring, and NAT egress rule helper
* feat(external): route gateway LRP over transit pool and install VPC SNAT in routed NAT mode
* feat(vpcd): support bridge_mode=nat (routed uplink, transit pool, egress rules)
* feat(daemon): register EIP handlers unconditionally with disabled-mode stub; seed default subnet per external mode
* feat(admin): support --external-mode=nat and suggest it for non-bridgeable uplinks
* feat(scripts): add --nat-uplink routed mode to setup-ovn.sh
* test(e2e): add routed-NAT single-node natuplink suite
Covers the external_mode=nat lane end to end on a nat-initialized node:
host transit wiring (veth, br-ext port, ip_forward, iptables egress
rules), spinifex.toml contents, disabled EIP surface (empty
describe-addresses, UnsupportedOperation on allocate), default subnet
MapPublicIpOnLaunch=false, OVN gateway LRP + VPC SNAT on the transit
/24, guest outbound egress proven via a serial-console probe (nat mode
has no inbound path), and unique transit gateway IPs across VPCs.
Suite self-gates on external_mode=nat; registered as e2e-natuplink and
in SUITES_BUILD so it compiles in CI. Manifest entry stays commented
until CI can provision a nat-mode environment.
* feat(ovn): Address_Set CRUD and NAT exempted_ext_ips column
Adds the exempted_ext_ips column (optional strong Address_Set ref) to the
nbdb NAT model, Address_Set ensure/get with wait-op create semantics, and an
in-place SetNATExemptedExtIPs update on the client interface, live client,
and mock. Test NB schema gains the column and covers insert-resolves-UUID,
converge, clear, and missing-NAT paths.
* feat(policy): exempt address set on routed SNAT/EIP rows
Adds WithSNATExemptSet so routed mode stamps exempted_ext_ips (the
spinifex_nat_exempt Address_Set: transit /24 plus operator extras) on IGW
snat and EIP dnat_and_snat rows, keeping VM replies to host-initiated flows
un-NATted. AddSNAT gains an idempotency guard keyed on (router, VPC CIDR)
with stale-IP replacement, and both AddSNAT and the AddEIP idempotent-skip
path patch the ref in place on pre-existing rows. The set is a singleton
strong ref and is never deleted. Non-routed modes are unchanged.
* feat(host): VPC ingress route primitives for routed-NAT mode
EnsureVPCIngressRoute / RemoveVPCIngressRoute manage the host route that
sends VPC-CIDR traffic into OVN via the gateway LRP transit IP on the
spx-nat veth. Replace/del semantics keep both idempotent; a missing route
on removal is not an error.
* feat(external): routed ingress leg on IGW attach/detach, reconcile-safe
AttachIGW's routed leg now also installs the host VPC ingress route via
RoutedIngressHooks, and re-runs on the already-attached path so the
reconcile loop restores host routes (and patches legacy SNAT exempt refs)
after a reboot — OVN state survives restarts but host routes do not.
DetachIGW removes the route alongside the SNAT. vpcd wires the hooks and
the spinifex_nat_exempt set (transit /24 + [network] nat_exempt_cidrs,
validated nat-mode-only) exclusively in routed-NAT mode; pool/veth/direct
pass nil hooks and are unchanged.
* test(natuplink): Tier 1 host-ingress phase; docs for jumpbox pattern
New Phase 7 asserts the automatic ingress tier: host route to the VPC CIDR
via the gateway LRP transit IP, exempted_ext_ips ref on the SNAT row
pointing at spinifex_nat_exempt (containing the transit /24), and — after
opening the SG from the transit net (default-closed, AWS parity) — ping and
an sshd TCP handshake against the instance private IP from the host, which
proves the un-SNATted return path. Phase 8 additionally waits for the
second VPC's ingress route to appear on IGW attach. vpc-networking README
documents the jumpbox pattern, SG note, and nat_exempt_cidrs.
* feat(config,admin): public wan pool alongside nat-transit
Restructure admin config generation from single-pool scalars to a Pools
slice so nat mode can emit both the transit pool and an optional public
wan pool. nat mode now accepts --external-pool/--external-source (plus
--external-bind-bridge/--gateway-ip when a public pool is requested),
validated by the shared resolvePublicPoolFlags helper also used by pool
mode. Transit pool always renders first so the IGW allocator picks it.
Formation wire format unchanged (nat mode is single-node only).
* feat(daemon): enable EIP service and auto-assign in nat mode with public pool
External IPAM (and with it the real EIP service, RunInstances public IP
auto-assign, and MapPublicIpOnLaunch on default subnets) now activates
whenever public pools exist: pool mode as before, plus nat mode when a
wan pool rides alongside the transit segment. The transit pool is
filtered out of IPAM — its addresses are gateway-LRP plumbing, never
allocatable EIPs. Transit-only nat configs keep the disabled EIP stub.
* feat(host): per-EIP ingress primitives for routed-NAT mode
DetectUplinkFor parses ip route get to find the interface and source IP
facing the pool gateway (re-run per ensure: WiFi/cellular uplinks move).
EnsureEIPIngress steers an EIP into OVN with a /32 route via the VPC
gateway LRP transit IP (src-hinted so host-local traffic still DNATs),
answers ARP for the EIP with a proxy neighbor entry (L3 only, same MAC,
WiFi-safe), sends gratuitous ARP, and adds per-EIP FORWARD accepts for
DROP-policy hosts. RemoveEIPIngress mirrors; missing state tolerated.
* feat(policy,vpcd): host EIP binder option on NATManager
WithHostEIPBinder injects routed-mode host plumbing hooks. AddEIP binds
after priming on both the fresh-add and idempotent-skip paths (host
routes and proxy-ARP are volatile even when the OVN row survives, so
reconcile re-ensures them after reboot); DeleteEIP unbinds best-effort.
The next hop comes from the new gatewayPortIP helper reading the gateway
LRP's stamped spinifex:gateway_ip (Networks fallback), with the ext-ID
key now an exported policy const shared with the external package.
vpcd selects the IGW pool by name instead of index — with two pools in
nat mode the gateway allocator must stay on the transit pool — and wires
the binder over host.EnsureEIPIngress/RemoveEIPIngress when a public
pool is present. EnsureEIPIngress gains an uplink hint so dhcp-sourced
pools (gateway unknown until lease) proxy-ARP on their bind bridge.
* feat(dhcp): interface-MAC lease variant with MAC-keyed-router detection
dhcp_mac = "derived" (default) | "interface" on external pools. The
interface variant leases with the bind interface's own MAC — WiFi/WWAN
uplinks drop frames with foreign source MACs, so derived chaddr MACs
never complete DORA there. Option 61 stays the lease discriminator.
UseIfaceMAC threads AllocateRequest → NATS wire → nclient4 → lease and
persists through KV so restart re-acquires repeat it.
Routers that key leases on MAC and ignore client-id would hand every
lease the same IP; the manager detects an ACK matching the interface's
own address or another client-id's live lease, releases the colliding
lease, and hard-errors advising a static range instead.
admin init auto-writes dhcp_mac="interface" for nat-mode public dhcp
pools bound to non-bridgeable uplinks (wlan/wwan/ppp).
* test(natuplink)+docs: Tier 2 EIP ingress e2e phases and public-pool docs
The natuplink suite now detects a public (non-transit) pool in spinifex.toml
and switches lanes: Tier-1-only configs keep the EIP-disabled and
no-public-IP assertions; public-pool configs assert the EIP surface is
enabled, the default subnet has MapPublicIpOnLaunch=true, instances
auto-assign public IPs, and a new Phase 9 exercises the full EIP ingress
lifecycle — host /32 route via the gateway LRP, proxy-ARP on the uplink,
per-EIP FORWARD accepts, exempt-stamped dnat_and_snat, TCP handshake to the
EIP, reconcile replay across a vpcd restart, and teardown on disassociate.
Docs: vpc-networking README gains a public-pool section for routed-NAT nodes
(init flags, dhcp_mac=interface behaviour, host-to-EIP caveat) and an updated
mode comparison; COMMANDS.md notes nat-mode EIP support behind a public pool.
* fix(setup): whitelist iptables, neigh sysctl, and arping for spinifex-vpcd
Routed-NAT mode shells out to iptables for the transit masquerade and
per-EIP FORWARD accepts, tunes proxy_delay, and sends gratuitous ARP for
host-delivered EIPs. Without these sudoers entries vpcd crash-loops with
"command not allowed" on nat-mode nodes.
* fix(ovn): scope DeleteStaticRoute prefix match to the target router
Every VPC router carries a 0.0.0.0/0 static route, so an unscoped cache
match could grab another router's row and fail with a referential
integrity violation, wedging IGW attach on multi-VPC nodes. Match only
rows referenced by the target router, mirroring FindStaticRoute. Mock
updated to the same semantics, with a regression test covering drift
replacement across two routers.
* fix(reconcile): always replay AttachIGW so host ingress state is re-ensured
applyIGWs skipped AttachIGW whenever the gateway switch port existed in
actual state, so the already-attached re-ensure path never ran and
routed-NAT host ingress routes lost on reboot were never reinstalled.
AttachIGW is idempotent; run it unconditionally.
* fix(vpcd): resolve duplicate-CIDR host ingress route collisions, bootstrap VPC wins
Every account's default VPC shares 172.31.0.0/16, so Tier-1 host routes
collide on the single host route table and the last IGW attach silently
stole the route (route replace). Pass vpcID and the gateway LRP IP
through RoutedIngressHooks; the bootstrap VPC always installs its route,
other VPCs only install when the CIDR is free and only remove a route
their own gateway IP holds. Adds host.VPCIngressRouteVia to read the
current route holder.
* ci(e2e): add nightly cell 19 for routed-NAT single-node suite
Cell 19 bootstraps with --external-mode=nat plus the CI static pool as
the public wan pool and runs the natuplink suite, covering the Tier-1
transit datapath and Tier-2 EIP ingress. The run step exits 0 when
natuplink.test is absent from the tests tar so scheduled runs against
branches predating the suite stay green.
* fix(admin): nat-mode init summary reflects configured public pool
The summary printed 'outbound-only — no public IPs/EIPs' even when
--external-pool/--external-source configured a public wan pool. Print
the pool range (or dhcp bind) and note EIPs are enabled.
* fix(eip,vpc): thread ctx through disabled EIP stub and NAT call sites
Reconcile the routed-NAT branch with dev's ctx-first service APIs after
rebase: DisabledEIPService and its tests take ctx to satisfy the EIPService
interface, the vm_adapters public-IP resolver asserts the ctx-signed
AssociatedPublicIPForInstance, and the default-subnet mapping test threads
ctx into DescribeSubnets.
* feat(otelsetup): OTel init, resource attrs, trace-stamping slog handler
Env-gated on OTEL_EXPORTER_OTLP_ENDPOINT: with no endpoint only the W3C
propagator is installed and globals stay no-op, so instrumented binaries
ship everywhere at zero cost. Resource carries service/host/mulga.env/
mulga.source plus OTEL_RESOURCE_ATTRIBUTES (ci.run_id in CI).
* feat(otelsetup): wire telemetry init into all spx services
Every spx service start command installs the trace-stamping JSON slog
default and the OTel providers via initTelemetry, flushed on shutdown.
Gateway's own slog setup now routes through otelsetup so its level
handling keeps trace correlation.
* feat(gateway): HTTP tracing middleware on awsgw and daemon cluster API
Server span per request with W3C traceparent extraction; awsgw spans are
renamed to the resolved SigV4 service.Action post-auth with account and
region attributes. Request log lines carry trace_id via context.
* feat(nats): propagate W3C trace context over command subjects
NATSRequestCtx injects traceparent beside X-Account-ID and opens a client
span per hop; ServeNATSRequest opens a consumer span joining the producer
trace, with a Ctx variant for handlers that thread context. RunInstances
is plumbed end-to-end (gateway dispatch through placement fan-out to the
daemon handler) and viperblockd ebs.mount/unmount join inbound traces.
* feat(systemd): source optional /etc/spinifex/telemetry.env in service units
Lets operators enable OTLP export (OTEL_EXPORTER_OTLP_ENDPOINT,
MULGA_ENV, MULGA_SOURCE) by dropping a single env file, with no unit
edits. File is optional; absent means telemetry stays disabled.
* feat(ci): export OTel telemetry from E2E envs to the central Elastic sink
All E2E workflows set SPINIFEX_OTLP_ENDPOINT (repo variable override,
default wattle) plus ci.run_id/ci.workflow resource attributes, so
bootstrap-install.sh writes telemetry.env on every node. Teardown runs
flush-telemetry.sh before the final tofu destroy so buffered spans and
metrics reach the sink.
* feat(otelsetup): request metrics and Go runtime metrics via OTLP
mulga.requests counter + mulga.request.duration histogram recorded by the
same middleware that opens spans: awsgw carries the resolved SigV4 action,
the daemon cluster API uses the chi route pattern, and NATS handlers record
per-topic at the subscribe chokepoints (node names stripped to keep
cardinality flat). Go runtime metrics start inside the same env gate.
* feat(daemon): deep spans for QMP commands, VM launch phases, EBS ops
Single chokepoint span per QMP command in sendQMPCommand covering all
callsites; vm.launch phase children (mount_volumes, start_qemu, qmp_connect,
persist_state); ec2.cmd consumer span with traceparent extraction plus
start/stop/reboot/terminate child ops; ebs.mount/unmount spans in
viperblockd with volume attributes.
* feat(telemetry): thread request context through NATS, object store and VM paths
Migrate utils.NATSRequest call sites to NATSRequestCtx (241 sites; W3C
traceparent now flows gateway -> daemon on every request), add GatherCtx and
migrate all Gather call sites, inject traceparent on manual EKS publishes.
ObjectStore interface is ctx-first with aws-sdk WithContext calls and an
otelhttp transport, so predastore spans join the caller trace. NATS consumer
spans record error status when the response envelope carries an error.
vm.Manager and the ec2 handler service interfaces take ctx-first so vm.launch
and QMP spans join the request trace; detached background work uses explicit
context.Background(). Hot-path slog calls use *Context variants for trace_id
binding.
* feat(telemetry): suppress health-check, heartbeat and unrooted trace noise
Health probes record metrics only (no span per poll); the QMP heartbeat
poller marks its context so query-status ticks stop rooting a trace each.
rootSampler honors MULGA_ROOT_TRACE_RATIO (ParentBased ratio, default 1.0)
and the predastore unit sets 0.05: guest block I/O roots are sampled while
request-linked spans are always kept.
* feat(otel): export deployment.environment resource attribute
Elastic APM maps it to service.environment, enabling the native environment
selector across the APM UI (Services, Traces, Dependencies).
* fix(telemetry): stop objectstore rooting method-only HTTP transactions
Background S3 calls (state sync, cleanup loops) carried no caller span, so
otelhttp rooted a trace per request and APM surfaced thousands of parentless
'HTTP GET/PUT/DELETE' transactions with expected-404 probes counted as
failures. Emit the client span only when the request context already carries
one, matching the viperblock backend. Bump viperblock/predastore to branch
heads so CI builds also pick up the equivalent span filter and root sampler.
* feat(telemetry): per-VM guest metrics via dedicated QMP socket + qmp-collector
Goanna Phase 1 producer, operator-consumed first. Each VM gets a second
-qmp chardev (qmp-telemetry-<id>.sock) plus a sidecar discovery file
carrying instance/account/vcpus/taps/monitoring-tier; the new
qmp-collector service polls query-cpus-fast (vCPU thread /proc jiffies),
query-blockstats, query-balloon and host tap sysfs counters, and
publishes CloudWatch-mappable goanna_ec2_* series to NATS
metrics.ec2.<instance-id>. A queue-group bridge in the same process
forwards the series to the node-local OTLP receiver so they reach the
operator sink until Goanna consumes the subjects. The collector never
touches the manager's control QMP socket; telemetry failures never
block a launch.
* fix(qmp-collector): dial NATS via DialTarget, drop ProtectProc, fix meta race
The collector connected to the raw NATS host (0.0.0.0), failing TLS
verification against a cert valid for 127.0.0.1; route through
admin.DialTarget like the other daemons.
ProtectProc=invisible hid the QEMU vCPU threads (dumpable=0) that the
collector must read from /proc/<tid>/stat, so every tick failed and
re-primed without ever publishing. Drop it to default.
Surface collect failures at Warn so silent metric loss is visible, drop
per-tick publish to Debug, and read InstanceID under the lock in run()
to fix a race with updateMeta on ENI hot-plug.
* chore(deps): re-pin predastore + viperblock to dev-landed commits
* feat(eks): reconciler in-place control-plane restart on sustained degradation
Phase 1 of EKS CP recovery (siv-243). The ACTIVE-branch reconciler only
recorded health; a CP VM that died (OOM/host crash) left the cluster wedged
ACTIVE-but-unreachable with no recovery path. Add a CPInstanceControl surface
and bounded in-place restart: once an ACTIVE cluster's CP stays unhealthy past
a grace window, StartInstance is issued (backoff-spaced, capped) when the
instance is stopped/error, re-mounting the same root volume so embedded etcd
survives. Degradation stays AWS-faithful ACTIVE + a ClusterHealth issue rather
than a non-AWS DEGRADED status enum. cpControl nil keeps the reflect-only path.
Wiring of a real CPInstanceControl in the daemon and the error-state
manager-eviction fix are the remaining Phase-1 integration steps; Phase 2
(etcd snapshot cadence + restore-snapshot) is tracked separately.
* feat(eks): wire control-plane restart into the reconciler via instance service
Threads a CPInstanceControl into the per-cluster reconciler (siv-243 Phase 1).
cpControlAdapter binds the daemon instance service + account: InstanceState
reads the CP's EC2 state via DescribeInstances, StartInstance restarts a stopped
CP via StartStoppedInstance, which loads from the shared stopped store and
re-mounts the same root volume so embedded etcd survives. Wired as
EKSServiceDeps.CPControl = d.instanceService and passed through spawnReconciler.
Recovers a cleanly-stopped control plane co-located with the reconciler leader.
Remaining: DescribeInstances is node-local so a cross-node stopped CP needs an
aggregating describe; and an error/crashed CP (not in the stopped store) needs a
dedicated error-state restart path — both tracked as Phase-1 follow-ups.
* feat(eks): cross-node CP describe + error-state control-plane restart
The reconciler's control-plane recovery was wired to the node-local instance
service: DescribeInstances only saw the leader's node and StartStoppedInstance
only handled a cleanly-stopped CP from the shared KV. A crashed/OOM CP ends in
error state, still resident in its owning node's manager map (not the stopped
store), so recovery never reached it — the actual incident in the recovery plan.
Wire EKSServiceDeps.CPControl to a NATS-backed eksCPControl. DescribeInstances
fans out across every host (gateway scatter-gather) so the reconciler observes a
CP on any node. RecoverInstance drives the gateway StartInstances two-step:
ec2.cmd.<id> recovers a live error/running owner in place (the owner stays
subscribed through the crash; vmMgr.Start relaunches error->pending, re-mounting
the etcd root volume), falling back to ec2.start shared-KV rehydration for a
stopped CP.
The CP VM is ManagedBy=EKS, hidden from customer DescribeInstances; relax that
hide so the owning account resolves its own managed CP when targeting it by
explicit instance ID (bulk listings still hide it). EC2APIState already maps the
internal error state to the AWS closed-enum stopped, which cpRestartable accepts.
* fix(eks): reconcile CP as system account, not customer
The control-plane VMs are launched under the system account
(admin.SystemAccountID()), not the customer account that owns the cluster
record. spawnReconciler was binding cpControlAdapter to the customer account,
so InstanceState/RecoverInstance ran as the customer — which cannot see or own
its own cluster's system-owned CP VMs. DescribeInstances returned zero and the
reconciler logged 'control-plane instance not found', never restarting a wedged
CP. Bind the adapter to the system account instead.
Revert the earlier DescribeInstances managed-VM relaxation: it was based on the
mistaken assumption that CP VMs are customer-owned. Visibility is resolved by
the system-account adapter, so the explicit-ID escape hatch is dead code.
* fix(vm): pre-create mgmt tap on disk-boot restart path
The direct-boot branch pre-creates the management tap via NetworkPlumber.SetupTap
(owned by the daemon euid), but the disk-boot/else branch only appended the QEMU
tap,id=mgmt0 device without creating the tap. The daemon runs as a non-root user
without CAP_NET_ADMIN, so QEMU can only attach a tap it already owns; on the
disk-boot path QEMU tried to create /dev/net/tun itself and failed with
'Operation not permitted', failing every stopped-instance restart (StartInstances
returned ServerInternal). This blocked EKS control-plane recovery: the reconciler
found the stopped CP and issued the restart, but the VM never booted.
Call SetupTap for the mgmt tap in the disk-boot branch, mirroring direct-boot.
* feat(eks): recover all HA control-plane members, not just primary
The reconciler restarted only ControlPlaneInstanceID (the primary CP
member), so on an HA cluster the other server VMs stayed stopped and
etcd never regained quorum (1/3) — apiserver healthz stayed fail even
after recovery ran. maybeRecoverControlPlane now iterates every member
in ControlPlaneNodes (falling back to the scalar ID for single-node
clusters), restarting each stopped/error member per pass while
preserving the grace window, attempt cap, and backoff gates.
Refs mulga-siv-509
* fix(eks): keep cloud-init polling IMDS so primary CP gets its user-data
The eks-node system VM's user-data is delivered only via IMDS (the
NoCloud seed was retired in #411). IMDS (169.254.169.254) is reachable
only once mulga-mgmt-net brings up the data NIC via DHCP and the per-tap
br-imds datapath reconciles, which on the first-launched (primary) CP
can lag cloud-init's stock 120s probe window. With the default max_wait
cloud-init falls back to DataSourceNone, permanently losing the launch
user-data (no SPINIFEX_K3S_ROLE -> no k3s server) and wedging the cluster
in CREATING.
Bake a cloud.cfg.d drop-in that pins DataSourceEc2 and raises max_wait to
600s so cloud-init keeps polling until the datapath is ready instead of
falling through to None. Worst case a slow boot, never a lost role.
* fix(eks): warm IMDS instance-role creds before first SigV4 sign
On IMDS-credentials mode the on-VM SigV4 helpers (eks-token-webhook,
eks-gateway-publish, mulga-eks-state-report) sign via the SDK default
chain, which resolves in-VM IMDS instance-role creds. The per-tap
br-imds datapath lags VM boot by minutes (observed ~264s), and the SDK's
IMDS client fast-fails each probe (250ms dial), so early Retrieve calls
return 'context deadline exceeded' / 'no EC2 IMDS role found'.
eks-gateway-publish only retries for 30x5s=150s, shorter than the
datapath warm time, so it exhausts its budget and drops the
bootstrap/state publish -> cluster hangs in CREATING.
The credential chain is sound (the CP role trusts ec2.amazonaws.com and
AssumeRoleForInstance evaluates that principal); the failure is timing.
Add gwsign.Signer.EnsureCredentials and block in eksgw.New until the
instance-role creds first retrieve (2s interval, 10m ceiling aligned
with cloud-init max_wait), so every helper's first signed request
already holds creds instead of racing a cold datapath. Static-key mode
is unaffected.
* fix(eks): retry CP data-NIC DHCP until cross-host datapath is up
A single ~16s DHCP one-shot on the primary data NIC lost the race against
the cross-host OVN datapath (logical-port binding + per-tap reconcile on the
remote placement host), stranding the NIC IP-less so the Ec2 datasource never
reached IMDS and the CP member wedged. Add dhcp_acquire: retry the one-shot on
a budget aligned with cloud-init's Ec2 max_wait (600s), confirming an inet
address before success. Each attempt still exits leaving no daemon, so
cloud-init's later ENI management is unaffected.
* fix(eks): drop reconciler registry entry when Run self-exits
ReconcilerRegistry.Spawn only cleaned up its holder on ctx cancel
(Stop/StopAll). When a ClusterReconciler's Run returned on its own — a
terminal cluster state or a lost lease — nothing cancelled the ctx, so
the holder leaked and a later same-name create silently no-opped at the
head of Spawn, never reconciling to ACTIVE.
Thread a finished channel from RunClusterReconciler's Run goroutine back
to Spawn's watcher, which now cleans up on whichever fires first: an
external cancel or Run exiting. On self-exit it deletes its own holder
and releases the lease, so a fresh same-name create re-spawns and
reconciles.
* fix(eks): run mulga-mgmt-net in boot runlevel before cloud-init-local
CP user-data is delivered only via IMDS since the NoCloud seed was retired
(#411), so cloud-init DataSourceEc2 must reach 169.254.169.254. mulga-mgmt-net
DHCPs the data NIC that makes IMDS reachable, but it was added to the default
runlevel and only declared `before cloud-init` (the network-stage service).
cloud-init-local therefore ran its init-local Ec2 crawl first with no data-NIC
route and burned its whole probe window (~4 min) before mulga-mgmt-net brought
eth0 up, regressing CP bringup from ~1m50s to ~5m54s.
Add mulga-mgmt-net to the boot runlevel in setup.sh instead of default, drop it
from ENABLE_SERVICES, and order it `before cloud-init-local` so the data NIC and
IMDS route are up before the init-local crawl. Keeps the IMDS-only user-data
path; the raised Ec2 max_wait stays as a safety net for host br-imds datapath lag.
* fix(vm): refresh LaunchTime on crash-restart so watchdog spares relaunch
RestartCrashedInstance re-launched a crashed VM without refreshing
Instance.LaunchTime. The pending watchdog measures now-LaunchTime against
PendingWatchdogTimeout, so the stale original launch time tripped
launch_timeout on the next tick and terminated the crash-restart
mid-flight. Refresh LaunchTime under the state lock, mirroring the
daemon-restart restore path (restore.go), so the relaunch gets a fresh
timeout window. Adds a regression test asserting LaunchTime advances.
* feat(eks): reconcile control-plane member count after a lost member
The cluster reconciler self-heals a stopped/error control-plane VM in
place but never replaced a terminated/gone member, so an HA cluster ran
below haControlPlaneCount permanently. Add a member-count reconcile pass
that provisions one replacement CP joining the surviving etcd quorum.
- CPProvisioner + EKSServiceImpl.ProvisionReplacementCP: replay the
persisted create template on a free host, the single-node analogue of
the spread join branch, with rotating creds re-derived.
- ControlPlaneTemplate persisted on ClusterMeta at create (per-node and
rotating-cred fields cleared). SwapControlPlaneMember CAS swaps the dead
node for the replacement and refreshes the [0] scalar mirrors.
- maybeReplaceControlPlaneMember fires only when no member is restartable
and one is truly lost, a healthy majority survives, bounded by grace,
backoff and an attempt cap; placement excludes live-member hosts.
- Replacement prunes the superseded member from etcd once joined:
k3s-first-boot deletes the stale Node (EKS_ETCD_PRUNE_PEER_IP) so quorum
width returns to N. json tags added to K3sServerInput (now persisted).
* feat(eks): in-guest control-plane self-diagnosis on healthz fail
When a control-plane apiserver reports unhealthy, the reconciler only
saw healthz=fail with no cause, and the guest has no on-demand command
channel (serial root login is password-gated, no guest agent). Make the
CP self-diagnose and relay the cause through the one-way state report.
- mulga-eks-state-report.sh: diagnose() collects failing /readyz?verbose
subchecks, isolated /healthz/etcd reachability, and etcd-db-dir free
space; on fail write a verbose block to the host-captured serial
console and add a JSON-safe reason to the state payload.
- ServerStateReport: add Reason (omitempty; empty from a healthy CP or
an older AMI predating the field).
- cluster_reconciler observe(): surface the reason in the health issue.
- Tests: POSIX unit test for the agent (stubbed kubectl/df); Go tests
for the reason-bearing and terse health-issue paths.
* feat(eks): 15-min etcd snapshot cadence alongside nightly
Nightly alone is too coarse a recovery point for a young cluster. Install the
same mulga-eks-etcd-snapshot script into a second Alpine crond periodic dir
(/etc/periodic/15min) so it also runs on a ~15-minute RPO window; the script
picks its cadence tier from the dir it is invoked from ($0 sniff) and encodes
the tier in the snapshot key (etcd-{tier}-{ts}.snap), so a predastore S3
lifecycle rule can retain the frequent rolling window shallow and the nightly
copies deep. Server-role only (agents exit early); no restore-path change.
Add a self-contained POSIX unit test (stub k3s/curl/logger, symlinked crond
dirs to exercise the real $0 tier sniff) covering agent no-op, frequent + daily
key prefixes, local-snapshot prune, and quiet no-op without creds.
* feat(eks): auto-reform wedged etcd quorum via boot-time cluster-reset
An HA k3s control plane does not self-heal after a simultaneous full-CP
restart: embedded etcd never reforms quorum (etcd:unreachable sustained),
and in-place VM restarts cannot fix it. Add a last-resort escalation that
drives the documented k3s cluster-reset recovery, pull-based so the host
never needs an inbound channel into the guest.
Reconciler: maybeReformEtcdQuorum escalates only when every CP member is
VM-running but the health issue carries etcd:unreachable, the cluster is
HA, and a grace window has elapsed. It sets per-member recovery directives
(seed = cluster-reset, others = wipe-rejoin), epoch-bumped and keyed by
instance ID, then restarts the members. Bounded by grace/backoff/attempts;
a recovered cluster clears the directives once. Enabled via
WithEtcdResetRecovery alongside the existing CP control wiring.
Directive plane: per-member KV directive {epoch, action, snapshot} with
store helpers, an EKS service Get/Set pair over NATS-RPC, and an
internal-recovery gateway route (system-cred carve-out, same as
internal-addons). The CP server instance role gains eks:GetRecoveryDirective.
Guest: mulga-eks-k3s-recovery is a pre-k3s oneshot (server role only) that
fetches its directive through the AWS gateway (eks-gateway-fetch -resource
recovery), epoch-guards one-shot application, then runs
`k3s server --cluster-reset` (optionally restoring a predastore snapshot)
or wipes the local etcd datastore to rejoin the reset seed. Wired into the
image manifest, setup.sh perms, and the eks-node-role selector.
Tests: reconciler escalation gates + recovery clear, directive store epoch
increment + service validation, and a POSIX unit test for the guest agent.
* fix(eks): reboot running CP members to apply etcd cluster-reset directive
The etcd quorum-reformation escalation drove StartInstance on the seed and
followers, but the reset gate only fires when every member is VM-running. The
StartInstances path rejects a running instance with IncorrectInstanceState, so
the guest never re-entered its boot sequence and the cluster-reset/wipe-rejoin
directive was never applied — the wedged control plane stayed wedged.
Reboot the running members (QMP system_reset) instead, re-entering the boot
sequence so the on-VM recovery agent applies the directive. Adds RebootInstance
to the CP control surface (interface, adapter, NATS-backed daemon impl) and
switches maybeReformEtcdQuorum to it; the stopped/error in-place restart path
keeps StartInstance.
* fix(eks): graceful stop for etcd reset recovery
The etcd quorum-reformation reset path forced running CP members through a
hard QMP system_reset to re-enter the boot sequence and apply their recovery
directives. A hard reset skips a clean unmount: on the next boot e2fsck
cleared the recovery and k3s runlevel entries (dangling inodes), so the
cluster-reset never ran and k3s was unstartable — the reset defeated recovery.
Replace the reboot with a graceful StopInstance (QMP system_powerdown, clean
unmount, SIGKILL fallback). The existing in-place restart path
(maybeRecoverControlPlane) then boots each stopped member clean on a later
tick, and the on-VM recovery agent applies its directive against an intact
filesystem. Reuses existing lifecycle machinery; the stop also tolerates a
wedged QMP client via the SIGKILL fallback.
Swaps RebootInstance -> StopInstance across CPInstanceControl,
cpInstanceController, the cpControlAdapter, and the NATS-backed eksCPControl
(gateway RebootInstances -> StopInstances), plus the reset/recovery test fakes.
* fix(vm): retry QMP dial and unlink stale socket on relaunch
A QEMU killed with SIGKILL (crash or etcd-reset restart) leaves its
qmp-<id>.sock inode behind. On relaunch the startup os.Stat probe and the
QMP dial both raced that dead listener, so a single dial returned
'connection refused': StartStoppedInstance then reported ServerInternal and
left the instance KV-stopped even though the guest booted fully. The wedged
member stayed invisible to the reconciler's cpLive gate, stalling recovery.
Unlink the stale socket before launch so the inode that appears is the fresh
QEMU's, and retry the QMP connect with a short backoff to cover the window
where the socket is bound but not yet listening. Handshake failures are not
retried — a listener answered, so the error surfaces immediately.
* feat(eks-ui): surface cluster Health.Issues on the EKS console
DescribeCluster is AWS-faithful (Status stays ACTIVE, degradation lives in
Health.Issues), but the cluster detail page only rendered the status badge, so
a degraded control plane still showed as ACTIVE with no visible signal. Add a
destructive health badge next to the status and a Health row that lists the
issue messages, mirroring the addons tab. Healthy clusters read OK.
* chore: rebuild ui
* fix(viperblockd): stop mount-path daemon VB uploader to prevent checkpoint clobber
The daemon-side VB created in the ebs.mount handler is stored in
MountedVolume.VB for the whole mount, and its background chunk uploader was
only stopped at daemon shutdown. That state-only VB ran DrainToBackend every
30s, overwriting the live checkpoint the nbdkit plugin VB (the data-path
writer) keeps updating, leaving SeqNums inconsistent with sealed chunk
ciphertext and causing transient AEAD auth failures on cloned encrypted
volumes. Stop its uploader right after New, matching the other short-lived VB
call sites.
Also bump viperblock to the atomic chunk-ObjectID allocation fix.
* build(deps): pin viperblock to landed AEAD concurrency fix
Bump github.com/mulgadc/viperblock to the dev commit b08bea6 that
eliminates "cipher: message authentication failed" on encrypted volumes
under concurrent read-modify-write + background chunk drain. Removes the
local-path replace used while the fix was in flight.
* fix(viperblock): stop chunk-uploader goroutine leak at all VB call sites
viperblock.New() unconditionally starts a background chunk-uploader
goroutine (30 s interval) that only VB.Close() stops. Seven call sites
created short-lived or state-only VB instances without calling Close(),
leaking one uploader goroutine per instance. On a single node the leaked
goroutines exhaust memory (predastore RSS grew ~1.3 GB/5 min at idle
post-single-suite), causing liveMemGate to reject launches and the lb
suite to fail.
The leaked uploaders are also the primary cause of the AEAD auth
failures (mulga-be2ev): a leaked VB from prepareRootVolume holds the
clone-point BlockLookup and overwrites the correct live checkpoint in S3
every 30 s, leaving SeqNums inconsistent with sealed chunk ciphertext.
Fix all seven sites:
- instance/service_impl.go prepareRootVolume: defer StopChunkUploader +
StopWALSyncer after successful viperblock.New (mirrors the EFI path's
intent; full Close is unsafe here because cloneAMIToVolume already
called SaveBlockState + RemoveLocalFiles)
- viperblockd.go ebs.delete handler: add StopChunkUploader alongside
existing StopWALSyncer
- viperblockd.go ebs.unmount handler: same
- viperblockd.go shutdownVolumes: same
- snapshot/service_impl.go snapshotVolume: defer both stops after New
- volume/service_impl.go CreateVolume: defer both stops after New
- image/service_impl.go snapshotRunningVolume: defer both stops after New
- image/service_impl.go snapshotStoppedVolume: defer both stops after New
Closes mulga-0zzci
Closes mulga-ji5fi
* fix(deps): bump viperblock to 01a7e686 (AEAD regression fix)
The prior pin (6af89855) carried the snapshot-AMI AEAD regression that
corrupts block storage on restore. Point at the merged viperblock dev fix.
---------
Co-authored-by: joshsiv-mulga <mrk@mulgadc.com>
Failed LBs previously reported an empty State.Reason, so e2e failures
read 'entered terminal state failed:' with no cause. Record the cause
on every failed transition — launch error (e.g. insufficient sys.micro
capacity), missing system credentials, missing internal mgmt return
route, and async-launch panic — persist it as StateReason, and map it
to LoadBalancerState.Reason in describe responses. Cleared when the
first heartbeat transitions the LB to active.
The internal lb subtests raced the previous subtest's sys.micro VM
deallocation: fixed 15s sleeps were not enough on capacity-tight
single nodes, so the LB entered terminal state failed (insufficient
capacity, then nats no-responders after the daemon unsubscribed the
full topic).
Replace the sleeps with createActiveLB: create LB + listener, wait for
active via the new non-fatal harness WaitForLBActiveErr, and on a
terminal failed state (wraps ErrLBTerminalFailed) tear down fully and
retry with backoff, up to 3 attempts.
go.mod referenced v1.10.0 which pre-dates LoadLiveCheckpoint. CI builds
without go.work so e2e-build failed to compile. Pin to viperblock dev tip
6af8985 (feat/snapshot-ami merged) until v1.11.0 is tagged.
* feat(gpu): add H200/RTX Pro 6000 SE models and fix MIG host resource allocation
- Fix H100 SXM/PCIe device ID labels swapped in models.go (10de:2330=SXM, 10de:2331=PCIe)
- Add H200 SXM (10de:2335) to knownModels, computeModels, and migCapableModels
- Add NVIDIArtxPro6000BlackwellSE to definitions.go mapped to g7e family (matching AWS g7e instance type)
- Add MIGHostResources() policy table: 4 vCPUs / 32 GiB host RAM per compute slice, covering all known NVIDIA MIG profiles
- GenerateMIGTypes now emits non-zero vCPU/memory for EC2 instance type descriptors
* fix(daemon): enforce CPU/memory limits for MIG instance types
Whole-GPU passthrough remains bypass-only (gpuManager.Claim is the sole gate).
MIG types now fall through to canAllocateCount so host CPU and RAM are checked
before launch — prevents over-committing a host where one physical GPU backs
multiple concurrent slices, each with non-trivial vCPU/memory allocation.
* feat: Add snapshot image promotion flow
* fix: Small changes to viperblock/spinifex so snapshot captures delta properly and instances can boot from snapshot AMIs
* fix: test coverage and more consistent prefix check
* fixed testing
* more tests
* fix: added new graceful process killer in utils, bumped NATS timeout for testing
* fix: complete at-rest encryption launch + volume lifecycle (#361)
* chore: bump viperblock to encrypted-snapshot VolumeUUID nonce fix
Pull viperblock 1044c2a (feat/snapshot-uuid-nonce-fix) so E2E builds against
the fix for the encrypted AMI snapshot metadata tag-verify failure
(cipher: message authentication failed) on instance launch. Pairs with
viperblock PR #22.
* fix: forward viperblock master key to nbdkit mount path
viperblockd loaded the master key but never passed it to the nbdkit
plugin it spawns for volume mounts, so encrypted boot volumes opened
in cleartext and failed LoadState with an encryption-state mismatch.
Forward cfg.EncryptionKeyFile into NBDKitConfig and emit the plugin's
encryption_key_file= argument when set.
* chore: bump viperblock to fresh-volume VolumeUUID mint fix
* chore: bump viperblock to snapshot-link LoadState ordering fix
* fix: route encrypted-volume config writes through viperblock keyholder
config.json is a sealed VBState whose AES-GCM tag and StateSeqNum-derived
nonce can only be advanced by the master-key holder that owns the volume.
The EC2 volume handler wrote it directly to S3, so mergeVolumeConfig had to
refuse encrypted volumes (rewriting would strip the tag) — bricking
UpdateVolumeState (attach in-use / detach available) and ModifyVolume.
Route encrypted-volume config updates through viperblockd, mirroring the
ebs.snapshot idiom: per-volume ebs.config.{volumeID} hits the owning node's
live VB (the single StateSeqNum writer); on ErrNoResponders the ebs.config
queue group opens the detached volume exclusively and reseals. Cleartext
volumes keep the direct-S3 fast path. No crypto leaves viperblockd.
mulga-siv-323
* fix: unwrap encryption envelope when snapshot handler reads config.json
CreateSnapshot and snapshotInUseByVolumes decoded config.json straight into
a VBState. For encrypted volumes config.json is an at-rest envelope
{payload, authtag}; a raw decode yields a zero VBState (SizeGiB==0), so
CreateSnapshot 500s on the size guard and the in-use scan silently drops
encrypted volumes' snapshots. Unwrap via viperblock.StateBody first,
mirroring the volume handler's mergeVolumeConfig path.
mulga-siv-324
* test: cover viperblockd ebs.config handlers
Unit/integration coverage for applyConfigUpdate, makeConfigUpdateHandler
(ebs.config.{volumeID}), and the ebs.config queue-group fallback: grow-only
size reconcile, live-VB apply, invalid JSON, apply errors, and the
detached-volume open path. Clears the diff-coverage gate (68.8%).
* chore: pin viperblock to dev tip (#22 merged)
* fix(viperblockd): seal volume block-map to predastore on detach + durability e2es (#371)
* feat: cloud-init EC2 datasource (#411)
* feat: Add snapshot image promotion flow
* fix: Small changes to viperblock/spinifex so snapshot captures delta properly and instances can boot from snapshot AMIs
* fix: test coverage and more consistent prefix check
* merge changes
* chore: remove dead SnapshotSub field and SIGTERM snapshot remnants from viperblockd
SnapshotSub was added during the NATS/SIGTERM snapshot attempts but is never
set post-rebase. Removes the field from MountedVolume and the two nil-checked
unsubscribe blocks. Re-adds the nbd import which is still needed for NBDKitConfig.
* fix: post-rebase gofmt fixes and remove dead snapshot handler code
Split gofmt-squashed lines in go.mod, vm_adapters.go, nbd_test.go,
viperblockd.go, volumes.go, and volume_durability_test.go. Removes the
dead unix-socket snapshot handler tests from viperblockd_handlers_test.go.
* test: add viperblock snapshot S3 coverage tests
Exercises snapshotVolume, snapshotRunningVolume, and snapshotStoppedVolume
past the nil-config guard using an httptest server that returns 403.
The AWS SDK credential check fails synchronously without network I/O,
keeping the tests fast while covering the S3Config construction, master key
load, and viperblock.New code paths. Brings diff coverage from 45% to 69.9%.
* fix: cloneAMIToVolume reads config via format-agnostic path
LoadStateRequest expects a viperblock binary envelope (encrypted),
but CreateImageFromInstance writes the AMI config as plain JSON via
putAMIConfig. Replace LoadStateRequest with s.amiLoader.GetAMIConfig
which uses viperblock.StateBody to handle both formats, matching the
path used by ListImages and all other AMI readers.
* fix: pin viperblock to feat/snapshot-ami pseudo-version for LoadLiveCheckpoint
go.mod referenced v1.10.0 (released from dev) which pre-dates LoadLiveCheckpoint.
CI builds without go.work so it resolved to the published version and failed.
Pin to the snapshot-ami HEAD pseudo-version until the branch is merged and tagged.
* fix: replace indexOf with slices.Index, revert e2e runner and viperblock version
- nbd_test.go: remove local indexOf helper, use slices.Index from stdlib
- e2e-suites.yml: revert ubuntu-latest back to ubuntu-26.04 with correct formatting
- go.mod: revert viperblock back to v1.10.0 (will resolve once viperblock PR merges)
* fix: remove pseudo-version pin from go.sum
* feat: drain viperblock via snapshot socket before CreateSnapshot
Before LoadLiveCheckpoint(), dial {data_dir}/viperblock/{vol}/snapshot.sock
and wait for DrainToBackend() to complete (30s timeout). This ensures WAL
chunks and the live checkpoint are current in S3 before the snapshot VB
reads them. If the socket is absent the instance is stopped and Close()
already drained.
fix: socket path used s.config.WalDir which is empty in the daemon config;
changed to s.config.DataDir + "/viperblock" to match the path the NBD
plugin uses when creating the socket.
* feat: wire snapshot/AMI UI actions and add PromoteImage NATS endpoint
- Register as AMI from snapshot detail (RegisterImage with BDM pointing at snapshot)
- Deregister from image detail (DeregisterImage with navigate-away on success)
- Promote to System from image detail (admin-only, irreversible via UI)
- NATS subject spinifex.image.promote wired through gateway → daemon → existing admin.PromoteSystemImage
- signedFetch extended with optional params to support action-keyed extra fields
* refactor: address PR review on PromoteImage types and subscription ordering
- Merge PromoteImageInput/PromoteImageOutput into PromoteImageOpts/PromoteImageResult
(add JSON tags; drop the duplicate pair)
- Inline the input variable in gateway/spx/promote_image.go
- Move spinifex.image.promote subscription to the spinifex.* group in daemon.go
* test: cover snapshotVolume/Running/Stopped error paths for diff coverage
Add encryption-key-load-error and LoadState-failure tests for the three
new viperblock snapshot paths (snapshotVolume in snapshot service,
snapshotRunningVolume and snapshotStoppedVolume in image service).
Diff coverage: 58.8% → 65.5%.
---------
Co-authored-by: Josh Siviour <mrk@mulgadc.com>
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
* fix: ECS StopTask reaps container before releasing capacity
StopTask marked a task lastStatus=STOPPED and released its capacity/ENI in
the control plane, but nothing ever told the ecs-agent to kill the container.
The container kept holding its host ports, so DescribeTasks reported a STOPPED
task whose container was still RUNNING, and a replacement task placed on the
freed capacity that rebound the same host port hit EADDRINUSE.
Wire the agent-driven cooperative stop the design intended (ecs-v1.md:165):
- Add a per-instance stop inbox (StopsPrefix/StopKey) and bus.StopDirective.
- requestStopTask marks DesiredStatus=STOPPED and posts a stop directive,
leaving lastStatus/capacity/ENI/targets untouched. StopTask, DeleteService,
and service scale-in use it; the heartbeat reaper and cluster/instance
teardown keep the immediate forceStopTask since no live agent can reap.
- PollAssignments returns pending stops; the agent lists containers by
mulga.ecs.taskID label, Removes them (kill+delete, freeing ports/netns),
then reports STOPPED. recordTaskState performs the single capacity release
on the agent's STOPPED report and reclaims the stop inbox.
- listServiceTasks excludes DesiredStatus=STOPPED tasks so a cooperatively
stopped task drops out of the service's running total immediately.
DescribeTasks now keeps the task RUNNING with desiredStatus=STOPPED until the
container is actually reaped, matching AWS ordering and eliminating the port
race.
* test: cover StopTask reap paths — poll stop-inbox drain, unplaced force-stop, agent stopTask edges
* ci: retrigger PR checks
- RegisterTaskDefinition rejects container secrets[] (InvalidParameterException)
and warns on unsupported log drivers instead of silently dropping them.
- Capture logConfiguration; only host-side json-file is collected, other drivers
warn at register and reach the agent for parity. Document json-file retrieval.
- Plumb executionRoleArn taskdef -> assign -> agent; the agent assumes it to
authorize ECR pulls, falling back to the instance role when unset.
- Add deployment objects with a health-gated rolling update honoring
deploymentConfiguration minimumHealthyPercent/maximumPercent, plus a deployment
circuit breaker with optional rollback to the last-good task definition.
EnsureLogicalSwitch/Router/PortGroup returned a client-side named-uuid on the
fresh-insert path but the real server uuid when the row already existed, so a
caller storing the returned uuid then doing Where(uuid) would miss. Resolve the
persisted uuid from the monitored cache after insert and add an explicit created
flag; topology callers now branch on created instead of uuid equality.
ReplaceACLs deleted and recreated the whole ACL set every reconcile, minting new
row uuids even for an unchanged SG and forcing ovn-northd flow recompute on every
drift tick. Diff current vs desired ACLs and no-op when unchanged (live + mock).
* test(network): in-process OVN NB reconcile harness
Add ovntest.StartNB: an in-process OVN Northbound DB backed by libovsdb's
pure-Go OVSDB server (no ovsdb-server/ovn-northd C daemons) plus a vendored
NB schema subset matching nbdb. Real LiveClient connects over a unix socket,
so reconcile scenarios exercise intent generation, reconcile wiring, LiveClient
and OVSDB transaction semantics in milliseconds without VMs.
Add live reconcile scenarios asserting NB rows and idempotency:
- VPC + subnet + SG + ENI from a hand-built IntentState
- the same seeded through JetStream KV via LoadIntentFromKV (front-to-back)
- IGW attach + drop gate + public-instance exemption (Logical_Router_Policy)
The idempotency snapshot normalizes UUID and pointer churn so it asserts
logical stability; datapath and NB->SB translation stay in the e2e suites.
* test: add orphan-prune and kitchen-sink golden live NB scenarios
Extend the in-process OVN reconcile harness with the two P0 completeness
scenarios. TestScenario_OrphanPrune_Live locks the teardown path and the
ReconcileApplyOnly startup-race guard: apply-only must not prune, full
Reconcile must, and re-pruning is idempotent. TestScenario_KitchenSinkGolden_Live
reconciles a maximal intent exercising every applyX path (VPC, public+private
subnets, two ENIs, SG with tenant rules, IGW, associated EIP, NAT gateway,
IGW/NATGW reroutes, drop gates) and asserts the normalized NB snapshot against
a checked-in golden, mechanically catching an unwired or mis-wired resource
type. Regenerate the golden with UPDATE_GOLDEN=1 go test.
* fix: repoint DB-node ovn-northd at RAFT member list; extend add-nat deadline
Two related fixes for multi-node OVN clusters:
- setup-ovn.sh (mulga-siv-504): in the clustered --management branch, point
ovn-northd's client NB/SB connections at the full RAFT member list derived
from OVN_REMOTE, so the active northd follows the leader instead of dialing
its local member only. Without this, northd stops advancing SB_Global.nb_cfg
once leadership moves off the node, wedging the ovn-nbctl --wait=hv barrier.
Guarded on a real member list so the standalone single-DB-node path is
unchanged.
- utils/nats.go (mulga-js-206): raise the add-nat request deadline from 10s to
45s so the client no longer rolls back before the server-side --wait=hv flows
barrier (30s) completes.
* fix: stop ovn-northd restart from tearing down split NB/SB ovsdb servers
The packaged ovn-northd.service ExecStop runs 'ovn-ctl stop_northd' without
--ovn-manage-ovsdb=no, so restarting northd to apply OVN_CTL_OPTS also stops the
separately-managed NB/SB ovsdb-server units. On a clean clustered bring-up this
races the freshly-started DBs and kills them, so setup-ovn's wait for the NB
socket times out and the RAFT create/join aborts. Install an ovn-northd.service
drop-in overriding ExecStop with --ovn-manage-ovsdb=no so northd never touches
the DB lifecycle (mulga-siv-504).
* test(e2e): capture SNAT datapath on NAT GW egress failure
TestNATGateway captured nothing when the private-VM egress probe failed, so
the baremetal nightly's intermittent 100% packet loss left no OVN/OVS state in
the failure bundle to root-cause it. Register harness.OnFailure before the
egress-verify poll to dump the SNAT datapath (OVN show, NAT rules, port
bindings, br-int flows, conntrack, ARP) keyed on the NAT GW external IP,
matching the eip/vpc datapath tests.
Also add a br-ext localnet flow capture to dumpDatapathState: br-int flows miss
the egress hop where SNAT'd traffic leaves for the physical uplink, which is
exactly where an egress black-hole would show.
* test(e2e): add OVN pipeline diagnostics to NAT GW egress capture
Capture the gateway router's per-stage logical-flow packet counts
(ovn-sbctl --stats lflow-list), MAC_Binding, static routes and the VM's
logical switch port alongside the existing OVS/conntrack/ARP dumps. When
the SNAT flow reads n_packets=0 while the guest transmits, these pinpoint
which pipeline stage drops the packet upstream of lr_out_snat.
* test(e2e): dump full lflow-list + datapath ct/flows for egress RCA
The IP-filtered lflow dump hides the default-drop ACL flow (its match
carries no IP), so a pre-SNAT egress drop at ls_out_acl only showed the
allow-flows at n_packets=0, never the deny that ate the packets. Replace
it with a full ovn-sbctl --stats lflow-list, echoed so it survives in the
CI stdout log. Swap the conntrack capture to ovs-appctl dpctl/dump-conntrack
(the standalone conntrack binary is absent on the baremetal runner) and add
dpctl/dump-flows -m for kernel megaflow hit counts.
* fix: claim gateway cr-port on every reconcile miss and stop masking launch failures
ensureGatewayClaimed nudged ovn-controller recompute only once then polled
until timeout, so a fresh VPC bring-up or chassis flap that fired the single
nudge too early left the gateway chassisredirect port permanently unclaimed.
Recompute on every miss instead, mirroring ensureGuestPortDatapath.
aggregateResults and distributeInstancesSpread masked any node launch failure
(including the resulting 5-minute launch timeout) as InsufficientInstance
Capacity. Surface the real node error when one exists; keep the capacity code
only for genuine capacity shortfalls.
* feat(eks): multinode control-plane networking, IAM, and CoreDNS parity
Integrates the EKS multinode fix set for the managed-control-plane topology
so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS
EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the
node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it.
EKS:
- Route apiserver egress (exec/logs/webhooks) through konnectivity
(apiserver-network-proxy): CP servers run konnectivity-server behind the
cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod
traffic rides the agent-initiated reverse tunnel instead of an unroutable
CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every
replica (HA-correct).
- Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay
flannel mesh SG rule authorized at cluster launch with ordered teardown.
- Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay.
- Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master
tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity.
- Spread nodegroup workers across hosts; addon readiness falls back on
workload rollout when gvks are absent.
- elbv2: allow the proxy_protocol_v2.enabled target-group attribute.
EBS:
- Move volume attachment state to state.json so a live nbdkit VB cannot
clobber it; route mounted-volume state updates through the live VB with a
retry for in-use marks racing the mount responder; keep the boot volume
in-use across stop/crash unmount.
* fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM
missingOrchestrationDeps gated on s.deps.IAM, but production wires only the
lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only).
The gate therefore reported missing=[IAM] on every node forever, so
CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even
though the IAM service was fully up via the provider.
Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else
deps.IAMProvider()) the nodegroup/role code already uses — so the protective
check still rejects a node with no IAM service but passes when the provider
supplies one. Add a regression test that sets only IAMProvider and asserts the
gate is satisfied; the existing nil-both test still rejects.
* fix(eks): let eks-token-webhook use IMDS creds when static keys absent
The CP VM launches on an instance profile, so buildK3sUserData omits
EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys
before eksgw.New could fall back to the IMDS SDK chain, so the webhook
aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver
crashed on the missing --authentication-token-webhook-config-file file,
stalling the cluster at CREATING until it timed out to FAILED.
Drop the static-key precondition (matching the sibling eks-gateway-publish
and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/
cluster gates.
* fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane
The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt,
which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a
169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing
169.254.169.254 and steering IMDS off the data NIC away from the per-tap
br-imds datapath, where the host RST it. eks-gateway-publish then failed to
sign requests and no bootstrap artifacts were ever published, so clusters sat
CREATING until they timed out to FAILED.
Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address
or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI)
are unaffected and IMDS falls through the data NIC's default route to the
br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net
deletes any default route on a NIC declared NIC<n>_DEFAULT=0.
* fix(eks): grant eks:WebhookTokenReview to the control-plane instance role
The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway
broker at POST /clusters/{name}/token-review, signing with the control-plane
VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes
against eks:WebhookTokenReview, but the role's inline policy granted only
eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every
relay and the apiserver returned 401 to all external kubectl.
Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile
re-asserts the inline policy on every control-plane instance-profile ensure, and
the gateway evaluates authorization per request, so existing clusters converge
without a rebuild once the shared role is updated.
* feat: add OVSDB RAFT cluster primitives to setup-ovn.sh
Add --db-cluster-local-addr / --db-cluster-remote-addr flags so a
management node can run the OVN NB/SB databases as a native OVSDB RAFT
cluster instead of a single non-clustered ovn-central. When a local addr
is given, write a shared OVN_CTL_OPTS to /etc/default/ovn-central (NB
RAFT 6643 / SB RAFT 6644) and restart the per-DB units directly, since
the ovn-central aggregator is ExecStart=/bin/true and won't restart its
children. Empty remote addr creates the cluster (init node); a set remote
addr joins it. NB/SB client set-connection runs only on the create node
(RAFT replicates it cluster-wide).
The single-node --management path is unchanged when no cluster addrs are
supplied, so dev boxes and existing single-node clusters keep working.
--ovn-remote already passes through verbatim, so a comma-separated SB
endpoint list works for compute nodes; documented in usage.
* feat: OVN client multi-endpoint failover for RAFT clusters
Accept a comma-separated OVN NB/SB endpoint list so vpcd can target a
clustered DB. config.ParseEndpoints splits the list (trimming whitespace,
dropping empties); live.go Connect passes one libovsdb WithEndpoint per
endpoint, so the client tries them in order, fails over on disconnect, and
follows RAFT leader redirects. Single-endpoint strings still yield one
endpoint, so single-node dev is unchanged.
ovn-sbctl --db= already accepts the raw comma list, so the SB CLI
consumers (discoverChassis, gateway_claim) need no change.
* feat: write clustered OVN NB/SB endpoints across the RAFT quorum
spx admin init/join now emit comma-separated OVN NB/SB endpoint lists for
the first three quorum nodes (formation.BuildOVNDBAddrs) instead of loopback
or a single leader address, so vpcd fails over across the cluster.
firstboot defers OVN bring-up to the provisioning controller when it owns
formation (SkipFormation): a standalone single-node .db would otherwise block
the controller's clustered create/join.
* test: add multinode OVN RAFT cluster + failover e2e; document clustered topology
ovn_raft_test.go asserts NB/SB cluster/status reports a 3-server quorum with
one leader on every DB node, then stops the NB leader's ovn-central and proves
an NB write still lands via a surviving endpoint.
Docs: vpc-networking gains the multi-endpoint vpcd config and a RAFT
cluster-status debugging section; network-connections lists the 6643/6644 RAFT
ports.
vpcd's libovsdb NB client was a single long-lived connection with no
reconnect logic. After the NB server restarted (host reboot, service
restart) every vpcd holding a dead/half-open socket never re-established
it, so OVN-write handlers blocked until the caller timeout and CreateVpc
returned ServerInternal intermittently (queue-distributed across vpcd).
Add WithInactivityCheck to the OVSDB client: a periodic Echo probe
detects the dead peer and triggers reconnect against the configured
endpoint, with infinite backoff so a daemon keeps retrying across a long
outage. Bound the SG subscriber handlers with a timeout context so a
stuck OVN op frees the worker instead of hanging on context.Background.
NLBs hard-rejected customer security groups and always minted a managed SG that
auto-opened listener ports, so aws-load-balancer-controller (which assigns SGs to
type=external NLBs) got a 400 and looped. Accept up to 5 customer SGs on an NLB:
when supplied they replace the managed SG on the ENIs and the caller owns the
listener-port rules (CreateListener already no-ops its auto-open when there is no
managed SG); without them the managed-SG default is unchanged. SetSecurityGroups
follows AWS create-time semantics — replace SGs on an NLB created with SGs, reject
adding them to one created without.
CreateVolume hardcoded iops to 3000, silently dropping a caller-supplied Iops,
and rejected non-gp3 VolumeType with the generic InvalidParameterValue. Honor
input.Iops within the gp3 envelope (3000 baseline, 500 IOPS/GiB, 16000 ceiling)
and return the volume-type-specific UnknownVolumeType for unsupported types.
Closes EBS CSI API gaps #3-4 (siv-380); gp3 Throughput (gap #2) deferred.
* feat: IMDS-capable gateway signer for lb-agent + EKS clients
Add internal/gwsign, a SigV4 signer that draws credentials from either
fixed static keys or the AWS SDK default credential chain. In a guest VM
the chain falls back to the in-VM IMDS endpoint, yielding scoped,
auto-rotating instance-role credentials; a session token is signed as
X-Amz-Security-Token so the gateway can verify temporary (ASIA) creds.
Wire it into the lb-agent (lbagent.Agent) and the EKS on-VM client
(internal/eksgw). When static keys are absent the clients now sign with
IMDS credentials instead of failing; when present, behaviour is
unchanged (static path). This is the client-side foundation for dropping
long-lived secrets from VM user-data.
Credentials are retrieved per request so rotated IMDS creds take effect
without an agent restart.
* feat: IMDS instance-role creds for EKS server + LB VMs (Phase B/C)
Attach a Spinifex-managed system instance profile to the k3s control-plane VM
and the elbv2 LB VM so both authenticate to the gateway with scoped, rotating
IMDS instance-role credentials instead of a long-lived static secret baked into
user-data.
- Add a shared handlers/iam.EnsureSystemInstanceProfile that find-or-creates a
role (EC2 trust) plus its instance profile and asserts an inline permission
policy. The gateway enforces per-action authz on assumed-role principals, so
the roles carry exactly the actions each VM calls:
- EKS server: eks:PublishInternal, eks:ListInternalAddons.
- LB agent: elasticloadbalancing:LBAgentHeartbeat, GetLBConfig.
- Plumb IamInstanceProfileArn through sysinstance.SystemInstanceInput and both
daemon launch paths (BootAMI via PrepareRunInstances->RunInstance; direct-boot
microVM via RunInstance), which persist it so the in-VM IMDS endpoint serves
the role.
- Drop the baked secrets when a profile is attached: omit EKS_ACCESS_KEY/
EKS_SECRET_KEY from the k3s first-boot env and LB_ACCESS_KEY/LB_SECRET_KEY from
the lb-agent fw_cfg env; relax k3s server validation to accept profile-only;
drop the EKS key requires from the first-boot and state-report scripts.
- Both verticals fall back to baked static creds when IAM is unwired, so an
absent master key never blocks a launch.
* fix: resolve IMDS instance-role creds under the instance account for system VMs
Three boot-path defects kept the LB VM's lb-agent from authenticating to the
gateway with IMDS-served instance-role creds; each produced a silent empty IMDS
role list and a heartbeat failing with "unexpected empty EC2 IMDS role list".
Found and fixed against env19.
- Build the system IAM ensurer lazily on the first system-VM launch instead of
eagerly at daemon startup, where it raced the NATS KV backend and fell back to
baked static creds permanently. ELBv2 and the EKS deps now hold an IAMProvider
resolved at launch time.
- Create the LB instance profile in the system account the LB VM runs under
(utils.GlobalAccountID), not the LB-owner account. IMDS resolves the profile
under the instance's account, so a customer-account role was invisible to it.
- Record the attached instance's owning account on the ENI (InstanceOwnerId,
mirroring AWS Attachment.InstanceOwnerId) and resolve the instance and every
IAM/STS lookup under it. The LB ENI lives in the customer account while the
system LB VM and its role live in the system account; keying the whole IMDS
chain off the ENI account left the instance invisible and the role lookup
missing. VPC leaves (subnet/VPC/SG CIDR) keep using the ENI account.
A fresh LB on env19 then authenticated end-to-end: IMDS serves a non-empty role
list plus ASIA creds, and the gateway accepts the heartbeat as an assumed-role
principal with no static keys in user-data.
* test(e2e): assert LB VM authenticates via IMDS instance-role creds
The LB suite's targets-healthy assertion already depends on the lb-agent
authenticating to the gateway, and with static keys dropped from user-data that
auth is IMDS-only. Add an explicit check that a recent gateway authentication
for the LB used an assumed-role (ASIA) principal, so a regression to the
instance-account IMDS resolution fails with a clear message instead of a
mysterious datapath timeout. The LB VM is a system-account instance invisible to
the customer AWS API, so the assertion observes the gateway journal on the
cluster nodes; it is time-bounded to avoid the stale-console-log false-fail and
skips when no node is SSH-reachable.
Cell 19 bootstraps with --external-mode=nat plus the CI static pool as
the public wan pool and runs the natuplink suite, covering the Tier-1
transit datapath and Tier-2 EIP ingress. The run step exits 0 when
natuplink.test is absent from the tests tar so scheduled runs against
branches predating the suite stay green.
Add a second build+publish step to the EKS AMI workflow so the same
manual run also builds the ecs-agent system AMI and uploads its qcow2 +
.sha256 sidecar to R2. Reuses the shared checkout, Go, and tooling setup;
renames the workflow to cover both node images.
Flip the k3s server apiserver to anonymous-auth=false. The only
consumer of the anonymous path was mulga-eks-state-report's /healthz
probe; switch it to an authenticated kubectl get --raw=/healthz through
the node admin kubeconfig (an unauthenticated curl now returns 401).
The NLB target group uses a TCP health check, and the host-side
reconciler rides the authenticated NATS state-report, so neither
depends on the anonymous apiserver path.
* fix: free host-netns :80 for ECS bridge/host tasks via cred-endpoint DNAT
bridge and host network-mode tasks share the VM host netns where the
ecs-agent credential endpoint bound 169.254.170.2:80, so a task binding
0.0.0.0:80 hit EADDRINUSE and crash-looped. Bind the listener on the high
proxy port 51679 and DNAT 169.254.170.2:80 to it (OUTPUT for host/bridge
task traffic, PREROUTING for the awsvpc credential veth), keeping the dst
IP on the endpoint so no route_localnet is needed. Nothing holds a socket
on :80, so host/bridge tasks can bind it.
e2e runs nginx on :80 in all three network modes; the settle guard catches
a regression where a crash-looping :80 bind flaps RUNNING then STOPPED.
* test: validate ECS host-netns :80 binding via bridge task
The bridge subtest now runs nginx on :80 to prove the cred-endpoint DNAT
frees the host netns (pre-fix it crash-looped on EADDRINUSE). host mode
keeps a no-port command: it shares the same host netns, so two :80 tasks
cannot coexist on one node (one :80 per host, as on AWS), and StopTask
does not synchronously reap the prior container's port.
When NATS disables JetStream at runtime (e.g. a filesystem permission error
flushing stream state) it keeps the process alive and serves KV as read-only.
That is not a process failure, so spinifex-nats.service's Restart=on-failure
never fires, and KV-backed startup (awsgw IAM bucket init) stalls for minutes
until an operator restarts NATS by hand.
Add a JetStream liveness watchdog:
- build/scripts/nats-js-watchdog.sh probes the NATS monitoring endpoint
(http://127.0.0.1:8222/healthz?js-enabled-only=true). Only when the service
is active and JS fails every sample of a short multi-sample window (so
transient startup/quorum 503s are ignored) does it systemctl try-restart
spinifex-nats.
- spinifex-nats-watchdog.{service,timer} run it as a oneshot every 30s, with a
60s boot delay so cluster quorum settles first.
- spinifex.target Wants= the timer so it starts with the platform.
The NATS-internal permission race itself is a server quirk outside spinifex's
control; this restores service automatically when it recurs.
The in-cluster AWS LoadBalancer Controller provisions its own ALBs
(k8s-toc-*) and security groups (k8s-traffic-toc-*) in the customer VPC
via the awsgw ELBv2/EC2 APIs. Spinifex never tracks these, so cluster
teardown left them orphaned: DeleteSecurityGroup / DeleteVpc then failed
with DependencyViolation and the cluster wedged in DELETING.
purgeClusterInfra now reaps them by the AWS-standard ownership tag
elbv2.k8s.aws/cluster=<name>, scoped to the customer VPC, after the
cluster NLB teardown and before the SG / VPC block:
- ReapLBCLoadBalancers lists account LBs, filters to the customer VPC,
confirms the ownership tag via DescribeTags, and deletes by ARN.
- DeleteClusterSGs deletes any VPC SG carrying the ownership tag (reusing
the DependencyViolation retry budget), in addition to the named cluster
SGs, after rules are revoked.
- DescribeTags is added to the narrow nlbProvisioner interface.
Errors join teardownErrs so a failed reap keeps the cluster in DELETING
for a retry rather than completing with leaked, VPC-pinning infra.
TerminateWorkerInstances routes terminates via the per-instance ec2.cmd.<id>
topic, whose subscription is torn down when a VM stops. A stopped (incl.
stopped+wedged, volume remount broken) worker therefore has no responder, and
the old code treated ErrNoResponders as already-gone and returned nil. The
worker was still in shared KV with its ENI intact, so the nodegroup record was
deleted while the ENI kept pinning the customer subnet/VPC, wedging DeleteCluster
in DELETING on DependencyViolation until a manual ec2 terminate-instances.
On ErrNoResponders, fall back to the ec2.terminate queue group (mirroring the
gateway TerminateInstances path), which runs TerminateStoppedInstance against
shared KV — deleting the worker's volumes, IP, and ENI without a running QEMU.
NotFound or no responder is idempotent success; any other error surfaces so the
teardown backstop retries.
TerminateInstances marks the EC2 record terminated and drives node-local
teardown, but when that teardown runs on a node other than the one hosting
QEMU (per-instance subscription lost, request falls back to the ec2.terminate
broadcast), the process survives and keeps holding its OVN logical port / IP,
blocking rebuilds.
Add OrphanQEMUReaper (node-local, ScopeNodeLocal): it inspects only the
terminated KV bucket and reaps a live process found via the instance's local
PID file. Running, pending, and stopped instances are never touched. Whichever
node holds the orphan PID file is the one that reaps it; every other node's
lookup misses and is a no-op.
Add utils.ProcessAlive (signal-0 liveness) and utils.ForceKillProcess
(SIGKILL + bounded wait), guard shutdownAndUnmount's force-kill on liveness,
and register the reaper in the daemon GC loop.
reconnectInstance re-asserted the instance running state but never the boot
volume's in-use state, so after a daemon restart describe-instances showed the
VM running while describe-volumes showed its root volume available — a
split-brain that makes start-instances attempt to double-launch a live QEMU.
Extract the launch path's boot-volume promotion into markBootVolumesInUse and
call it from reconnectInstance so launch and recovery keep volume state
consistent. Add a reconnect unit test asserting the boot volume is marked
in-use and non-boot volumes are left to their own attach flow.
The daemon-side in-use gate trusts persisted volume State/AttachedInstance,
which the attach/detach lifecycle can leave desynced: a running root volume
was observed with State:"" and AttachedInstance:"" while the instance's
BlockDeviceMappings still listed it, so DeleteVolume passed the gate and
removed the durable backing under a live QEMU.
Add an authoritative guard at the gateway DeleteVolume path: fan out the
strict DescribeInstances survey and refuse with VolumeInUse when any
non-terminated instance's BlockDeviceMappings reference the volume. Fail
closed on an incomplete cluster view rather than delete against a partial
survey. Terminate-time cleanup uses the daemon volume service directly and is
unaffected.
Drives the ECS data plane against the local awsgw: a customer VPC plus a real
container instance launched from the spinifex-ecs-node AMI, then standalone
tasks in all three network modes and an awsvpc service fronted by an ALB target
group. Adds an ECS client to the harness, a Makefile target, and a
service-interfaces suite entry.
bridge/host tasks share the host netns where the agent credential endpoint owns
169.254.170.2:80, so they run a no-port command and assert placement + mode;
awsvpc and the ELB service prove real :80 serving. waitTaskRunning re-checks
after a settle so a transient RUNNING cannot pass.
Wire crash detection and host-resource signals through to the AWS-faithful
DescribeInstanceStatus API and the spx CLI, closing out the EC2 health
monitoring / self-healing work.
Phase C — QMP health monitoring:
- qmpHeartbeat now tracks consecutive query-status failures; at
QMPMaxConsecutiveFailures it checks process liveness, triggering HandleCrash
on a dead QEMU and marking an alive-but-wedged process impaired.
- InstanceHealthState gains QMPConsecutiveFailures, LastQMPSuccess, ImpairedSince.
Phase E — DescribeInstanceStatus mapping:
- buildInstanceStatus maps running VMs to ok / impaired (QMP-unresponsive,
with ImpairedSince) / initializing (2-min launch grace), non-running to
not-applicable.
- SystemStatus reflects node memory pressure via ResourceManager
HostUnderMemoryPressure (live MemAvailable below the host reserve).
Phase C2 — observability:
- spx get vms gains HEALTH and CRASHES columns derived from vmHealthLabel.
Tests cover crash/QMP helpers, status mapping, memory-pressure system status,
and the health label.
On startup the agent had no reconcile pass: Run registered, heartbeat, and
immediately polled assignments with an in-memory dedup map. After a restart the
gateway re-delivers the instance's assignments and the agent re-ran them,
orphaning containers still running from before and double-starting tasks.
Add a reconcile() pass that runs once before polling: it lists the runtime's
containers, and for each task still running in this cluster re-registers its IAM
credentials, re-attaches the exit-wait, and refreshes RUNNING on the gateway.
The adopted task set seeds pollAssignments so re-delivered assignments are acked
but not re-run.
- runtime: add a label-agnostic List(ctx) []Container to the Runner seam, with a
containerd implementation and fake support.
- taskLabels: stamp credID / taskRoleArn / eniMac (when set) so a container is
self-describing enough to re-register creds and tear down its netns on adoption.
- pollAssignments takes the seed dedup map.
Best-effort: a nil runner or a List error degrades to today's poll+run behaviour.
* docs: add ECS (Elastic Container Service) guide + register ecs-quickstart workbook
Add a Container Services -> ECS doc mirroring the EKS/ECR shape: concepts
(cluster, task definition, task, service, container instances via the
ECS-AMI EC2 launch type), an aws ecs / console / Terraform quickstart,
task IAM roles via the credential endpoint, awsvpc networking, and the
honest v1 limitations (no service discovery, awslogs discarded, no rolling
deployments, no capacity providers, no distinct execution role).
Convert the ecs-quickstart workbook README to the doc-page format
(frontmatter + overview/prerequisites/instructions/troubleshooting) so it
renders as a Terraform Workbooks entry alongside the EKS workbooks.
* docs: add node-image prerequisite banner + infra diagram to EKS and ECS pages
Add a GitHub-style [!IMPORTANT] banner to the EKS and ECS prerequisites
flagging the required node-image import (eks-node / spinifex-ecs-node),
mirroring the install page's WAN-bridge banner, each with a describe-images
verification command.
Add a basic Spinifex topology diagram (SVG) to each Overview, in the
tf-nginx-alb visual style: the EKS managed control plane + node-group
workers, and the ECS ALB + container instances with the gateway/agent
and task-credential path.
End-to-end ECS stack against the Spinifex gateway: cluster, task definition
with a task role, container instances from the spinifex-ecs-node AMI, and an
awsvpc service fronted by an ALB target group. Unlike the console's provision-
capacity action, the workbook spells out the two things a Spinifex container
instance needs that AWS hides: a LAN-reachable gateway URL and the gateway CA,
both seeded into cloud-init user-data; instance creds come from IMDS via the
ecsInstanceRole profile. Live-verified apply (HTTP 200 through the ALB) and
destroy round-trip.
The agent heartbeats by periodically re-registering through the gateway
(RegisterContainerInstance). After a control-plane restart, the heartbeat
reaper marks instances DRAINING for the missed-heartbeat window, but the
re-register path only refreshed LastSeen and never restored the status, so a
reaped instance stayed DRAINING forever even though its agent was healthy and
re-registering.
Mark reaper-induced DRAINING with a Reaped flag and restore the instance to
ACTIVE on the next re-register, clearing the flag. An operator
UpdateContainerInstancesState drain clears Reaped, so an intentional drain is
never undone by the live agent.
* feat(ecs): provision EC2 container-instance capacity via IMDS instance-role
Adds a ProvisionCapacity gateway action that launches spinifex-ecs-node
EC2 instances which register as ECS container instances, fixing
RESOURCE:placement when a cluster has no capacity.
Agent control-plane now SigV4-signs gateway calls with instance-role
credentials read from IMDS (no static keys seeded in user-data). The
gateway already verifies STS/temp creds, so the instance role carries an
ecs:* policy + ec2 trust; ensureECSInstanceProfile find-or-creates the
ecsInstanceRole instance profile attached at launch.
Backend resolves the spinifex-ecs-node AMI by spinifex:managed-by=ecs
tag, renders keyless user-data, and launches via the customer
RunInstances path. Frontend adds a hand-signed JSON-1.1 ProvisionCapacity
call, a provision-capacity dialog on the container-instances tab, and the
ECS console section nav + system-image gating.
* feat: auto-assign EIPs to awsvpc service tasks; add ECS task/service detail pages
Wire AssignPublicIp=ENABLED through the service reconciler so awsvpc service
tasks automatically allocate and associate an Elastic IP on the RUNNING
transition, and release it on STOP. assignTaskPublicIP/releaseTaskPublicIP
mirror the target-group registration hooks in recordTaskState and forceStopTask;
the public IP and allocation ID are persisted on the task record and surfaced in
DescribeTasks attachments as publicIPv4Address.
Add AWS-parity ECS task and service detail pages showing private and public
endpoints, container state, and networking configuration, reachable by clicking
a task or service in the cluster tabs.
* test: cover natsEIPManager allocate/associate/release over NATS
ListTaskDefinitions appended every revision regardless of the requested
status, so deregistered (INACTIVE) task definitions never dropped off the
default listing and the console could not clear them. Filter by status,
defaulting to ACTIVE to match AWS.
Add an ECS console mirroring the EKS/ECR route shape: a single ECS nav
entry, cluster list and detail with Services/Tasks/Task Definitions/
Container Instances tabs, plus create and teardown paths.
Adds three full-page wizards to drive container deploys end-to-end:
Register Task Definition, Run Task, and Create Service. RunTask surfaces
a 200-with-failures placement result as an error instead of silently
navigating away.
Includes the @aws-sdk/client-ecs client singleton, ecs queries and
mutations, zod form schemas, and co-located tests.
STOPPED task records were never pruned: ListTasks enumerated every task a
cluster ever ran and the per-account KV grew unbounded. Add a leader-gated sweep
tick to the scheduler cron (alongside the reaper/reconcile) that deletes task
records STOPPED longer than a 1h retention window — keeping a just-stopped task
describable for DescribeTasks / UI exit reason, matching AWS's STOPPED window.
A task missing its StoppedAt timestamp is never pruned (defensive). Single-writer
(leader only), reuses the reap() bucket walk.
Wire the four NotImplemented ECS teardown actions so an IaC stack and the
console can reverse a deployment:
- DeleteCluster cascades: force-stops every task (releasing ENIs, deregistering
TG targets, returning capacity), then sweeps the cluster's whole KV subtree.
Unlike AWS it cascades rather than rejecting a non-empty cluster, so terraform
destroy round-trips regardless of teardown ordering.
- DeregisterTaskDefinition marks a specific family:revision INACTIVE (kept
describable); a bare family is rejected, matching AWS.
- DeregisterContainerInstance removes an instance, force-stopping its tasks when
Force is set and rejecting a non-empty instance otherwise.
- UpdateContainerInstancesState sets ACTIVE/DRAINING; draining force-stops the
instance's service-owned tasks so the reconciler relaunches them elsewhere.
Reuses the existing forceStopTask primitive and the reaper drain pattern; adds a
deleteKeysWithPrefix KV helper and INACTIVE status constants.
Serve task-role credentials to containers the AWS-faithful way: an HTTP
endpoint at 169.254.170.2:80 inside the ecs-agent (host VM netns) serving
short-lived credentials at /v2/credentials/{credID}. Containers find it via
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI, injected per container.
The agent mints credentials by SigV4-signing sts:AssumeRole(taskRoleARN)
against the gateway with its instance-role credentials, mirroring
internal/ecrauth. This reconciles ecs-v1 Q4/Q10 (STS over NATS + scheduler
JWT assertion) with the 4f rewire that moved the agent entirely onto the
gateway: the agent has no NATS connection and already holds instance creds,
so it assumes the task role directly. Credentials are cached per credID and
refreshed near expiry.
Each awsvpc task netns gets a credentials veth into the host netns and a
host-routed /32 to the endpoint, keeping it off the task ENI's VPC route.
The scheduler now persists taskRoleArn on the task definition and carries it
in the assign payload so the endpoint activates end to end.
* fix: serialize EBS hot-plug per instance to prevent PCIe port race
Concurrent AttachVolume calls to one instance both allocated the lowest free
hot-plug PCIe root port via nextHotplugEBSPort (which reads live QEMU state),
then issued QMP device_add on the same hotplug-ebs bus. The second device_add
failed with 'PCI: slot 0 function 0 already occupied by virtio-blk-pci',
surfacing as nats:timeout -> ServerInternal and a CSI retry-storm
('retry quota exceeded'); volumes never attached.
Add a per-instance attachMu and hold it across the whole AttachVolume and
DetachVolume critical sections, mirroring the ENI hot-plug serialization.
The mutex is non-persisted and shared via the Manager's stable *VM.
* fix: allocate EBS hot-plug PCIe port from in-memory accounting
AttachVolume derived the free PCIe hot-plug root port from a live
query-block scan (nextHotplugEBSPort), extracting the bus with a
hotplug-ebs(\d+) regex on each device's QDev. Real QEMU reports a
hot-plugged blockdev by id (/machine/peripheral/vdisk-vol-xxx/
virtio-backend), with no hotplug-ebs component, so the regex never
matched, the used set stayed empty, and the allocator always returned
port 1. Only the first EBS volume per instance attached; every later
attach collided on "PCI: slot 0 function 0 already occupied by
virtio-blk-pci", rolling back the blockdev, killing NBDKit, and
surfacing as nats: timeout / ServerInternal to the CSI driver, which
then exhausted its retry quota.
Allocate the port from authoritative in-memory accounting instead:
record the chosen port as EBSRequest.HotplugPort (persisted, so it
survives a daemon restart) and pick the lowest unclaimed port with
freeHotplugEBSPort over the instance's recorded requests. Hold
attachMu across allocate->device_add so concurrent CSI attaches cannot
pick the same port. DetachVolume already removes the request, freeing
the port. Drops the query-block round-trip from the attach path.
* feat(ecsgw): on-VM SigV4 client for the ECS agent→gateway channel
Adds internal/ecsgw, the twin of internal/eksgw: a TLS + SigV4 (service "ecs")
client the ecs-agent will use to reach the AWS gateway over HTTPS instead of
connecting to host NATS directly. Calls are AWS JSON 1.1 with an X-Amz-Target
action; the gateway relays onto ecs.* host-side, keeping NATS cluster-internal.
* feat(ecs): SubmitTaskStateChange gateway action → recordTaskState
Wires the AWS-API SubmitTaskStateChange action end to end (gateway action →
ecs.SubmitTaskStateChange NATS → daemon → Service.SubmitTaskStateChange) so a
gateway-routed agent reports task state without touching the NATS bus. The
handler maps the SDK input onto the same bus.TaskState shape and converges on
recordTaskState; the account is authoritative from the SigV4 caller, so an
instance cannot report state for another account's task.
* feat(ecs): durable assign KV inbox + PollAssignments gateway action
Scheduler writes task assigns into a per-instance KV inbox
(clusters/{cluster}/assignments/{instance}/{task}) instead of a
fire-and-forget core-NATS publish, and a new PollAssignments gateway
action lets a gateway-routed agent drain its inbox over TLS+SigV4.
At-least-once: an unacked assign is re-delivered on the next poll;
the agent acks accepted assigns (AckTaskIDs) and STOPPED transitions
reclaim stale inbox entries. The Poll response carries bus.Assign with
an RFC3339 time, so it is encoded with encoding/json (RawJSONActions)
rather than the jsonutil epoch-second marshaler.
* feat(ecs-agent): rewire control-plane off NATS onto the AWS gateway
The agent no longer connects to NATS. It registers, heartbeats, reports
task state, and receives assignments entirely through the AWS gateway
over TLS+SigV4, using seeded IAM credentials — keeping the NATS bus
host-internal and closing the trust-boundary gap (a tenant container
instance must not hold a shared bus token).
- controlPlane seam over ecsgw.Client replaces the NATS publisher:
Register (RegisterContainerInstance), SubmitTaskState
(SubmitTaskStateChange), PollAssignments.
- Heartbeat folds into a periodic idempotent re-register (refreshes
LastSeen); bus.Heartbeat is dropped from the agent.
- subscribeAssign (NATS subscribe) becomes a poll loop: drain the
inbox, dispatch each assign once, ack accepted taskIDs on the next
poll (at-least-once).
- config: ECS_ACCESS_KEY / ECS_SECRET_KEY / ECS_POLL_INTERVAL added,
ECS_NATS_URL removed; cloud-init initd seeds the IAM creds.
Tests move from the NATS publisher fake to a controlPlane fake.
* feat(ecs): cloud-init cred-seed + ecs-node bring-up script
ECS container instances are plain EC2 booted from the spinifex-ecs-node
AMI (AWS-faithful: no ECS-specific launch API). This renders the
cloud-init user-data that seeds /etc/spinifex-ecs/agent.env (gateway
URL/CA/region/cluster + IAM creds) and the gateway CA, then runs the
instance. The agent registers, heartbeats, polls assignments, and
reports task state entirely through the AWS gateway over TLS+SigV4 —
never NATS.
The guest dials the customer-facing gateway address (network advertise
IP), not the mgmt bridge, mirroring the EKS GatewayURL/AddonGatewayURL
split. Seeds the system key for now; scoped per-instance creds are a
follow-up (4g task-role endpoint).
* fix(ecs): reconcile dev merge — restore #418 ENI release + netns wiring
The 'Merge branch dev' resolution dropped code where Sprint 4d (#418,
awsvpc ENI-per-task) and the gateway control-plane rewrite (#419) touch
the same paths:
- recordTaskState STOPPED branch: reclaimTaskENI was clobbered by
reclaimAssignInbox; both are needed (release the task ENI AND drain the
KV inbox). Restores TestRecordTaskState_Awsvpc_ReleasesENIOnStopOnce.
- ecs-agent Agent: the netns field, its init, and the setupTaskNetns/
teardownTaskNetns wrappers were lost when the controlPlane Agent struct
won the merge, leaving taskrun.go calling undefined methods.
- eni_test assign assertion read the assign over NATS; #419 delivers it via
the durable KV inbox, so it now polls via PollAssignments.
- taskrun_test netns cases used the removed historyPub fake; switched to
fakeCP to match the controlPlane seam.
* feat(ecs): service controller + ELBv2 target-group registration (Sprint 4f)
Land the ECS Service controller on top of the 4e scheduler/RunTask flow:
- CreateService/UpdateService/DeleteService/DescribeServices/ListServices.
REPLICA only; DAEMON and serviceRegistries rejected in v1 (Q15).
- StartTask (explicit placement) and StopTask.
- Leader-driven replica reconciliation: a 10s scheduler tick converges each
service's running task count on desiredCount, launching the shortfall via
RunTask (tasks tagged group=service:{name}) and force-stopping the surplus.
- ELBv2 target-group registration: the scheduler registers a task's awsvpc ENI
IP on the RUNNING transition and deregisters on STOPPED via the existing
elbv2.RegisterTargets/DeregisterTargets NATS handlers (Q8, single-writer).
- forceStopTask consolidates the control-plane stop (STOPPED state, capacity +
ENI release, inbox drain, TG deregister); reused by StopTask, scale-in, and
the heartbeat reaper (the reaper now deregisters targets per Q13).
- Wire the 7 new actions through the gateway action map, NATS adapter, and
daemon subscriptions; add ServiceNotFoundException.
* fix(ecs-agent): anonymous pull for non-ECR images + normalize image refs
The agent minted an ECR authorization token via IMDS for every image, so
public docker.io refs failed with 'IMDS returned empty role name' on hosts
without an instance role. Gate ECR auth on .dkr.ecr. refs and pull all other
registries anonymously.
Bare image strings (nginx:alpine) were handed to containerd unmodified, which
parsed them as dummy://nginx:alpine and rejected the pull. Normalize refs to
their fully-qualified form (docker.io/library/nginx:alpine, :latest default)
before Pull and GetImage.
Adds StartTask/StopTask service-controller coverage.
* fix: address ECS 4f live-verification findings
ecs-node-bringup.sh defaulted GATEWAY_IP to the mgmt bridge (10.15.8.1), which
a VPC guest cannot route to — the agent could not reach the gateway. Default to
the host's WAN-bridge IP, which guests reach by NATing out through the WAN.
A target group not forwarded to by any listener reported its targets stuck in
'initial' forever (no health checker advances a detached TG). DescribeTargetHealth
now reports such targets as 'unused' with reason Target.NotInUse, matching AWS.
* feat(ecsgw): on-VM SigV4 client for the ECS agent→gateway channel
Adds internal/ecsgw, the twin of internal/eksgw: a TLS + SigV4 (service "ecs")
client the ecs-agent will use to reach the AWS gateway over HTTPS instead of
connecting to host NATS directly. Calls are AWS JSON 1.1 with an X-Amz-Target
action; the gateway relays onto ecs.* host-side, keeping NATS cluster-internal.
* feat(ecs): SubmitTaskStateChange gateway action → recordTaskState
Wires the AWS-API SubmitTaskStateChange action end to end (gateway action →
ecs.SubmitTaskStateChange NATS → daemon → Service.SubmitTaskStateChange) so a
gateway-routed agent reports task state without touching the NATS bus. The
handler maps the SDK input onto the same bus.TaskState shape and converges on
recordTaskState; the account is authoritative from the SigV4 caller, so an
instance cannot report state for another account's task.
* feat(ecs): durable assign KV inbox + PollAssignments gateway action
Scheduler writes task assigns into a per-instance KV inbox
(clusters/{cluster}/assignments/{instance}/{task}) instead of a
fire-and-forget core-NATS publish, and a new PollAssignments gateway
action lets a gateway-routed agent drain its inbox over TLS+SigV4.
At-least-once: an unacked assign is re-delivered on the next poll;
the agent acks accepted assigns (AckTaskIDs) and STOPPED transitions
reclaim stale inbox entries. The Poll response carries bus.Assign with
an RFC3339 time, so it is encoded with encoding/json (RawJSONActions)
rather than the jsonutil epoch-second marshaler.
* feat(ecs-agent): rewire control-plane off NATS onto the AWS gateway
The agent no longer connects to NATS. It registers, heartbeats, reports
task state, and receives assignments entirely through the AWS gateway
over TLS+SigV4, using seeded IAM credentials — keeping the NATS bus
host-internal and closing the trust-boundary gap (a tenant container
instance must not hold a shared bus token).
- controlPlane seam over ecsgw.Client replaces the NATS publisher:
Register (RegisterContainerInstance), SubmitTaskState
(SubmitTaskStateChange), PollAssignments.
- Heartbeat folds into a periodic idempotent re-register (refreshes
LastSeen); bus.Heartbeat is dropped from the agent.
- subscribeAssign (NATS subscribe) becomes a poll loop: drain the
inbox, dispatch each assign once, ack accepted taskIDs on the next
poll (at-least-once).
- config: ECS_ACCESS_KEY / ECS_SECRET_KEY / ECS_POLL_INTERVAL added,
ECS_NATS_URL removed; cloud-init initd seeds the IAM creds.
Tests move from the NATS publisher fake to a controlPlane fake.
* feat(ecs): cloud-init cred-seed + ecs-node bring-up script
ECS container instances are plain EC2 booted from the spinifex-ecs-node
AMI (AWS-faithful: no ECS-specific launch API). This renders the
cloud-init user-data that seeds /etc/spinifex-ecs/agent.env (gateway
URL/CA/region/cluster + IAM creds) and the gateway CA, then runs the
instance. The agent registers, heartbeats, polls assignments, and
reports task state entirely through the AWS gateway over TLS+SigV4 —
never NATS.
The guest dials the customer-facing gateway address (network advertise
IP), not the mgmt bridge, mirroring the EKS GatewayURL/AddonGatewayURL
split. Seeds the system key for now; scoped per-instance creds are a
follow-up (4g task-role endpoint).
* fix(ecs): reconcile dev merge — restore #418 ENI release + netns wiring
The 'Merge branch dev' resolution dropped code where Sprint 4d (#418,
awsvpc ENI-per-task) and the gateway control-plane rewrite (#419) touch
the same paths:
- recordTaskState STOPPED branch: reclaimTaskENI was clobbered by
reclaimAssignInbox; both are needed (release the task ENI AND drain the
KV inbox). Restores TestRecordTaskState_Awsvpc_ReleasesENIOnStopOnce.
- ecs-agent Agent: the netns field, its init, and the setupTaskNetns/
teardownTaskNetns wrappers were lost when the controlPlane Agent struct
won the merge, leaving taskrun.go calling undefined methods.
- eni_test assign assertion read the assign over NATS; #419 delivers it via
the durable KV inbox, so it now polls via PollAssignments.
- taskrun_test netns cases used the removed historyPub fake; switched to
fakeCP to match the controlPlane seam.
* feat(ecs): awsvpc ENI-per-task control plane (Sprint 4d)
Scheduler owns the task-ENI lifecycle for awsvpc mode: allocate from the
task subnet on placement, hot-plug onto the container instance, plumb ENI
identity through the assign payload + task record, and detach+delete on
STOPPED (graceful + reaper, idempotent leak guard). bridge/host unchanged.
* feat(ecs-agent): awsvpc guest-side task netns from hot-plugged ENI
The agent builds a per-task network namespace for awsvpc tasks: it finds the
hot-plugged ENI NIC by MAC, moves it into /var/run/netns/ecs-<taskID>, and DHCPs
the OVN-assigned ENI IP. The container then joins that netns via the OCI network
namespace path; bridge/host tasks keep the host (VM) netns as before.
* build(ecs-node): add iproute2 for awsvpc task netns
The ecs-agent builds awsvpc task network namespaces with `ip netns`, which
busybox's `ip` applet does not implement. Install iproute2 in the ecs-node AMI;
udhcpc (for the in-netns ENI DHCP lease) is already provided by busybox.
PR #411 (cloud-init EC2 datasource) dropped the cloud-init ISO that
created the ec2-user account and baked the harness SSH key; guests now
boot the stock Ubuntu gold AMI (default user ubuntu) bootstrapped via the
Ec2/IMDS datasource. eni_reconcile_test.go was the lone single-suite test
still SSHing as ec2-user, so TestENIHotplugReconcile failed publickey at
the first SSH. Align it with every other single-suite test.
* feat(eni): hot-plug reconciler + orphan sweep (Sprint 3d)
Add eniReconciler, a node-local vm.Reaper that converges ENI hot-plug
state across a daemon restart or crash: KV ENIRecord <-> in-memory PCIe
slot map <-> guest query-pci device list. Resolves records stuck in
attaching/detaching, re-adopts the slot map after restart, and reaps
orphan QMP devices and slot entries.
A staleness gate (new ENIRecord.AttachmentStateAt + 30s threshold) keeps
the reconciler off records a live attach/detach handler is mid-pipeline
on. Wired into the existing GarbageCollector backstop.
* test(eni): unit-cover hot-plug reconciler vm helpers
* test(eni): e2e reconciler convergence across daemon restart (Sprint 3e)
StopNode now stops spinifex-nats.service, so WaitNATSPeers polling that
node's dead 127.0.0.1:8222 monitor port fails with curl exit 7 and never
converges. Add WithSkipNodes to exclude the stopped node from the poll;
TestMultinodeNodeFailure skips node2.
Add the ECS v1 scheduler control loop and end-to-end RunTask flow:
- handlers/ecs: per-account KV-backed Service (clusters, task definitions,
container instances, tasks), bin-pack/spread/random placement with CAS
capacity reservation, and a leader-elected Scheduler that owns the Layer-2
bus subscriptions (register/heartbeat/task-state) plus a heartbeat reaper.
- gateway/ecs: wire CreateCluster, Describe/ListClusters,
Register/Describe/ListTaskDefinitions, Register/Describe/ListContainerInstances,
RunTask, Describe/ListTasks onto the daemon over NATS.
- daemon: construct the Service, start the Scheduler goroutine, subscribe the
ECS action subjects.
- ecs-agent: subscribe the per-instance assign subject, pull and run each
container via containerd, and report task state back on the bus.
Adds the spinifex-ecs-node AMI variant: alpine 3.21.7 + containerd + runc +
cni-plugins + ca-certificates with the ecs-agent binary baked to
/usr/local/bin and three OpenRC services wired in dependency order
(ecs-ca-install -> containerd -> ecs-agent).
- manifest.conf drives build-system-image.sh (no host mounts).
- ecs-ca-install.initd: first-boot oneshot installing the cloud-init
delivered gateway CA into the system trust; no-op when absent.
- ecs-agent.initd: sources /etc/spinifex-ecs/agent.env, backgrounds the
agent after net + containerd.
- 10-bridge.conflist: CNI bridge for task networking (Sprint 4d).
- setup.sh: exec bits, /opt/cni/bin symlink, content-store dirs, ttyS0
console reorder + fast boot menu.
- Makefile: build-ecs-node-image / import-ecs-node-image targets.
* feat(ecs): ecs-agent skeleton + containerd image pull (Sprint 4c)
Add the ecs-agent binary that runs inside an ECS container instance: it
resolves host identity from IMDS, registers on the Layer-2 NATS bus, and
heartbeats while alive. Image pulls go through a containerd-backed
ImagePuller behind an interface, with an ECR resolver that mints
GetAuthorizationToken creds from the instance role.
Extract the IMDS-credential and ECR-auth flow out of
ecr-credential-provider into internal/imdscreds and internal/ecrauth so
the agent and the kubelet provider share one implementation.
Add spinifex/handlers/ecs/bus as the single source of truth for Layer-2
subject builders and wire payloads, shared by the agent now and the
scheduler in Sprint 4e.
* build: pin genproto via replace to fix containerd/v2 ambiguous import
go.opencensus.io -> grpc@v1.33.2 drags the pre-split monolithic genproto
into the module graph; it still ships googleapis/rpc/status, which collides
with the split genproto/googleapis/rpc module that containerd/ttrpc needs.
The main module never imports the monolith, so a plain require bump is
dropped by 'go mod tidy'. Pin it with a replace to a post-split version so
the fix survives tidy and contributors can run 'go mod tidy' freely.
* fix(ecs): agent boot failsafes from runtime verify
- Build the ECR gateway HTTP client lazily on first image pull instead of
at boot. A missing/malformed gateway CA no longer kills the agent before
it can register + heartbeat, matching the best-effort containerd path.
- Actively probe containerd with IsServing in runtime.New. The client dials
lazily, so a dead socket previously slipped past the boot-time
'containerd unavailable' warn and only surfaced on the first pull.
- Make the heartbeat interval configurable via ECS_HEARTBEAT_INTERVAL
(default 30s) for test/verify and slow-link tuning.
* fix: drop removed predastore s3.Config.Compaction from admin_test
The predastore submodule bump removed the Compaction field from s3.Config.
Parse the rendered [compaction] block into a local struct so the template
round-trip test no longer references the deleted field.
* feat: graceful guest drain on systemctl stop / host shutdown
systemctl stop spinifex.target tore down nbdkit/viperblock while guest QEMU
was still writing, corrupting guest filesystems. The ordered drain existed
only over NATS via cluster shutdown, never on the systemd/host path.
Add spx admin node drain --local (GATE+DRAIN against the local node, leaving
STORAGE/PERSIST/INFRA to systemd), wired as the ExecStop of a new
spinifex-shutdown.service oneshot ordered After= all services so it stops
first and drains guests while every service is still up. DDIL preserved: the
oneshot fires only on target/host shutdown, not on a daemon restart.
Harden the viperblockd SIGTERM handler to flush the WAL but skip reaping an
nbdkit a guest is still attached to. Add a units ordering invariant.
* fix: ExecStop drain timeout needs duration unit (120 -> 120s)
The spinifex-shutdown.service ExecStop passed --timeout=120, but spx admin
node drain's --timeout is a cobra Duration flag and rejects a bare integer
(time: missing unit in duration "120"). The drain exited 1 without running,
so a systemctl stop tore nbdkit out from under the running guest and corrupted
its ext4 (the exact failure this feature prevents). Pass 120s.
* fix: nbdkitInUse read ss state from correct column + cover it
ss -H rows are '<netid> <state> ...', so the state is field[1], not field[0]
(field[0] is 'u_str'). The old index never matched ESTAB, so an in-use nbdkit
read as idle and the SIGTERM path would reap it under a live guest. Add tests
exercising a real connected unix socket, an absent socket, and TCP transport.
* feat: add predastore compaction E2E
Inject a short compaction interval into test-cluster predastore config via a
new --predastore-compaction-interval flag on admin init/join, gated so
production output stays byte-identical when unset. Add a harness helper that
drives write-then-delete churn against predastore S3, brackets it with
du-over-SSH usage, and asserts end-of-churn usage settles below 70% of the
churn peak; wire it into the multinode suite as TestPredastoreCompaction.
* fix: compaction E2E gate writes churn to its own bucket
The reclaim gate signed PutObject against the config-defined system
'predastore' bucket, which belongs to account 000000000000 with only a
static [[auth]] grant. The harness admin identity is a different IAM
account, so predastore returned 403 AccessDenied cross-account.
Create a per-run bucket the admin identity owns (AdministratorAccess)
and churn into it. Objects still erasure-code into distributed/nodes/
node-* so the du-based reclaim measurement is unchanged. The gateway
(:9999) does not proxy S3, so the client stays predastore-direct.
* test: run predastore compaction gate before destructive node tests
The compaction reclaim gate was declared last in the multinode suite, so it
measured shard-store usage after NodeFailure/NodeRecovery had torn down and
restarted node2. NodeRecovery only gates on NATS/gateway/EC2 readiness, not
predastore, so node2's shard store was still settling and the du baseline came
back empty. Move the test before the destructive node tests so it runs against
a pristine, fully-converged cluster primed by the volume tests.
* fix: race-free SSH output capture + empty-tolerant du probe
The compaction reclaim gate crashed with ParseInt("") when node1's du
probe returned empty combined output. Harden the probe and the SSH layer:
- ssh.Run used one bytes.Buffer for both session.Stdout and session.Stderr,
which x/crypto/ssh copies in two concurrent goroutines — a data race on a
non-concurrent-safe type. Switch to session.CombinedOutput, which guards the
combined stream with a mutex-wrapped singleWriter.
- shardStoreUsageBytes now terminates the remote command with a sentinel echo.
A missing sentinel is reported as output lost in SSH retrieval; a present
sentinel with no number is treated as a 0 measurement (anomalous, logged)
instead of crashing the gate.
The sentinel lets the next run distinguish a transport-layer drop from a
genuinely empty remote measurement.
* test(e2e): replace predastore du-reclaim gate with object lifecycle contract
The multinode compaction gate measured shard-store disk usage over SSH (du)
and asserted reclaim after churn. It could never pass: segments only seal at
the 4 GiB roll threshold, so test-scale churn produces no compactable segment,
and the du probe read 0 bytes before and after. Reclaim correctness is already
covered deterministically by predastore's store-level unit tests.
Delete AssertPredastoreCompaction and its du/churn/poll/sentinel machinery,
the harness unit tests that exercised them, and the multinode TestPredastore
Compaction entry. Add a single-node TestPredastoreObjectLifecycle that checks
the user-visible contract: an object is readable after PutObject, gone after
DeleteObject (GET returns NoSuchKey and ListObjectsV2 is empty), and the store
keeps serving fresh writes. newPredastoreS3 is retained, refactored to take a
host so it works outside the multinode cluster type.
* test(e2e): drop orphaned multinode TestPredastoreCompaction
Complete the du-reclaim -> object-lifecycle refactor: remove the
multinode TestPredastoreCompaction caller (and its now-unused context
and harness imports) that still referenced the deleted
harness.AssertPredastoreCompaction, which broke e2e-build. Wire the
single-node TestPredastoreObjectLifecycle entry that replaces it.
* docs: add Containers section with EKS and ECR service pages
New docs/containers/{eks,ecr} pages covering each service via AWS CLI, the Spinifex console, and Terraform, plus the requisite VPC/IGW/NAT/SG/IAM setup. EKS page embeds console screenshots. Cross-links the EKS Terraform workbooks.
* docs(eks): rework Terraform workbooks into a capability ladder
Rework the three EKS workbooks into a coherent ladder and publish them to
the Containers section. All tiers are single- or three-node capable via
node_desired_size and serve a shared Spinifex-themed demo app from ECR,
replacing nginxdemos/hello.
- demo-app: new std-lib Go web app (themed page reporting pod/node/cluster/
region + hit counter; optional /data persistence), distroless image.
- eks-quickstart: node_desired_size var, ECR repo + themed app over NodePort.
- eks-https-ingress (was eks-addons-access): HTTPS via the aws-load-balancer-
controller addon + ACM, public/private VPC with NAT, managed-ingress=false.
- eks-gitops-argocd (was eks-https-ingress): Argo CD addon syncs the app from
a git repo, aws-ebs-csi-driver PVC (Viperblock-backed) for state, access
entry RBAC, HTTPS Ingress carried over.
Pending live e2e verification (image build/push, apply, ALB + PVC bind).
* fix(ecr): implement ListTagsForResource (empty tags) for client parity
The AWS provider calls ecr:ListTagsForResource on every repository read, which
returned 501 NotImplementedException and broke 'tofu apply' of any workbook that
creates an aws_ecr_repository. Implement it to return an empty tag set (resource
tagging stays a deferred feature) so reads succeed. Drop the tags block from the
three EKS workbooks' ECR repos so the provider doesn't call the still-stubbed
TagResource on create.
* feat(eks): pull from internal ECR via kubelet credential provider
Worker nodes could not pull custom images from the internal Spinifex ECR
registry: the eks-node had no DNS for the registry host, no containerd CA
trust, no port rewrite to :9999, and no pull credentials. Only public
images worked.
Add cmd/ecr-credential-provider, a kubelet exec credential provider that
fetches the node IAM-role credentials from IMDS, calls GetAuthorizationToken
against the gateway ECR endpoint, and returns the minted AWS:<jwt> as Basic
auth for the registry host. Thread the gateway URL, account, region, and
internal suffix into the agent userdata, which now writes the gateway CA,
a registries.yaml (CA trust + :9999 endpoint), the kubelet
credential-provider config and config.yaml.d drop-in, and an /etc/hosts
entry mapping the registry host to the gateway IP. Ship the provider binary
in the eks-node AMI.
Also remove a stray paragraph from the admin-cli README.
* fix: dial EKS ECR registry mirror by gateway IP
cloud-init manage_etc_hosts rewrites /etc/hosts from the alpine template on
every boot, wiping the registry-host mapping appended via bootcmd, so worker
nodes could not resolve the internal ECR host and pulls failed with no such
host. Point the k3s registries.yaml mirror endpoint and tls config at the
gateway IP instead (the gateway cert carries an IP SAN), keeping the pod image
ref port-less and dropping the DNS leg until northstar lands. Fall back to the
hostname endpoint when the gateway is not an IP.
* fix: attach node-role instance profile to EKS workers for ECR
EKS workers launched with no IAM instance profile, so IMDS served no role and
the ECR credential provider could not mint a registry token, leaving internal
ECR pulls failing with 401. Real EKS implicitly creates an instance profile for
the node role; do the same here. Wire a KV-backed IAM service into the EKS deps
and, at worker launch, find-or-create an instance profile named for the node
role, attach the role, and pass it to RunInstances so IMDS exposes the role to
the credential provider.
* feat: use mulga logo in EKS demo app
* feat: add ACM certificate tag support (List/Add/RemoveTags)
Implements ListTagsForCertificate, AddTagsToCertificate and RemoveTagsFromCertificate across the store, service, NATS, daemon and gateway layers. Import now persists tags. Fixes terraform aws_acm_certificate InvalidActionException on ListTagsForCertificate.
* test: cover ACM tag gateway and NATS wrapper paths
* feat: support EKS nodegroup tags (Tag/Untag/ListTags)
Extends EKS tag ops to back nodegroup ARNs alongside cluster ARNs. Nodegroup create-time tags persist on the record and echo via DescribeNodegroup, giving the terraform-aws provider a clean default_tags round-trip. Fixes aws_eks_node_group TagResource 501 NotImplementedException.
* fix: provision EKS ALB ingress via node credentials
The aws-load-balancer-controller addon could not provision an ALB. Three fixes:
- TLS: the controller's per-service ec2/elbv2/acm SDK clients install their own
BaseEndpoint resolver and build an HTTP client that bypasses AWS_CA_BUNDLE,
falling back to the Go system trust pool and rejecting the gateway cert as
'unknown authority'. Add SSL_CERT_FILE so the gateway CA joins the system pool.
- Credentials: an empty AWS_ROLE_ARN beside the web-identity token file
half-arms the SDK web-identity provider, which then signs anonymously
(MissingAuthenticationToken). Inject the IRSA env only when a service-account
role ARN is staged; otherwise omit it so the default chain falls back to the
node instance-profile credentials served over IMDS.
- Authorization: the node role lacked the ec2/elbv2 actions the controller
needs. The eks-https-ingress workbook now attaches an AWSLoadBalancerController
managed policy to the node role (CreatePolicy + AttachRolePolicy).
* fix: add force_delete to EKS workbook demo ECR repos
tofu destroy of an EKS workbook left the demo ECR repository behind: it
still holds the pushed demo image, so DeleteRepository (force=false)
returns RepositoryNotEmptyException. Add force_delete = true to each
workbook's aws_ecr_repository.demo so teardown removes the repo with its
images.
* fix: force-delete EKS control-plane ENIs on teardown
TerminateK3sServerVM deleted the CP ENI with a plain force=false
DeleteNetworkInterface. eniIsLiveAttachment reports InUse whenever the
record's Status/InstanceId/AttachmentId are set, so when the K3s VM is
gone but the ENI record's attachment fields were never cleared the delete
returned InvalidNetworkInterface.InUse permanently and EKSDeletingReaper
re-drove the same non-idempotent call every 90s, wedging the cluster in
DELETING.
Teardown owns its CP ENIs and has already terminated the VM, so the
attachment is authoritatively dead. Replace the InUse-retry budget loop
with detach-then-delete (detachAndDeleteServerENI): DetachENI clears the
stale attachment fields, then DeleteNetworkInterface passes the in-use
guard. Both calls tolerate an already-gone ENI (NotFound) so a race with
the async instance-terminate cascade resolves to idempotent success.
purgeClusterInfra's private-endpoint ENI path reuses the same helper.
* fix: detach+delete worker ENIs on EKS nodegroup teardown
A terminated nodegroup worker could leave its primary ENI in
spinifex-vpc-enis as in-use/attached, and a dangling ENI throws
DependencyViolation on subnet/VPC delete, blocking tofu destroy. Two
gaps:
TerminateWorkerInstances did a local-only vmMgr.Get and silently skipped
any worker owned by another node, stranding both the VM and its ENI on
multi-node clusters. Route the terminate per-instance via ec2.cmd.<id>
(mirroring the gateway TerminateInstances path) so the owning daemon runs
the full teardown — detach + force-delete the primary ENI and clear its
record. No responder / NotFound payload is treated as already-gone so a
retried DeleteNodegroup stays idempotent.
deleteInstanceENI (stopped-instance terminate) issued a force=false
DeleteNetworkInterface with no prior detach: a stale in-use attachment
made it fail InvalidNetworkInterface.InUse, leaving the ENI record behind
after the instance was already written to the terminated bucket. Detach
first, then delete, tolerating NotFound.
* fix: heal volume State drift on detach/terminate teardown
A detach/terminate could leave a volume's State empty instead of
'available'. DeleteVolume gated on State=='available' exactly, so it
returned VolumeInUse for an unattached volume nothing was using, leaving
it undeletable and blocking stack teardown. DescribeVolumes cosmetically
rendered an empty State as 'available', masking the drift.
- DeleteVolume now treats an empty State with no attachment as deletable
(an attached volume still returns VolumeInUse).
- DescribeVolumes derives an empty State from the attachment (attached ->
in-use, unattached -> available) instead of blindly rendering
available, so an empty-but-attached volume is no longer hidden.
- UpdateVolumeState normalizes an unattached volume with an empty state to
available, so a detach writeback that omits the state cannot strand the
volume in drift.
* fix: break all SG cross-references on EKS teardown
DeleteClusterSGs broke the cp<->nodegroup cycle too narrowly. It revoked
only ingress (IpPermissions), never egress (IpPermissionsEgress), so a
cross-reference via an egress rule kept the SG pinned on
DependencyViolation. It also resolved only the two named cluster SGs, so
any OTHER SG in the VPC referencing them — the default SG, the LBC/ALB SG,
the private-endpoint SG, user/TF SGs — was never revoked and pinned the
referenced SG on delete, leaving the VPC stuck.
Enumerate every non-default SG in the VPC and revoke BOTH ingress and
egress on each before deleting, breaking any cycle order-independently,
then delete only the cluster-owned SGs (referrer SGs are deleted by their
own teardown). The default SG is left untouched. Since DeleteClusterSGs
runs against both the customer VPC and the managed control-plane VPC, the
broadened sweep covers both teardown paths.
* feat: support TargetGroup ProtocolVersion in elbv2
CreateTargetGroup now stores ProtocolVersion (defaulting HTTP/HTTPS target
groups to HTTP1) and DescribeTargetGroups/Create output return it. Without
this the aws-load-balancer-controller sees a perpetual immutable-field
mismatch (desired HTTP1 vs empty) and recreates the target group every
reconcile, looping on DuplicateTargetGroupName once a shared ALB IngressGroup
reconciles more than one target group. NLB protocols (TCP/UDP/TLS) keep it
omitted to match AWS.
* fix: omit IRSA env from ebs-csi addon when no role staged
The ebs-csi controller manifest hardcoded AWS_ROLE_ARN={{SERVICE_ACCOUNT_ROLE_ARN}}
plus the web-identity token volume. With no service-account role staged the
placeholder rendered empty, half-arming the SDK web-identity provider, which
panics 'role ARN is not set' and CrashLoops the controller, leaving PVCs
Pending. Convert to the conditional {{IRSA_ENV}}/{{IRSA_VOLUME_MOUNT}}/{{IRSA_VOLUME}}
markers like the LBC bundle so the IRSA env and token volume are omitted when
no role is staged, falling back to node instance-profile credentials over IMDS.
The EBS-specific AWS_EC2_ENDPOINT/AWS_ENDPOINT_URL_EC2 stay in both modes.
Requires an eks-node AMI rebake to take effect.
* docs: route gitops-argocd workbook through a shared ALB and expose ArgoCD UI
Fold the demo app and ArgoCD ingresses onto one ALB via an LBC IngressGroup
with host routing (app./argocd. on a wildcard cert) instead of one ALB per
ingress, add a NodePort + HTTPS Ingress for the ArgoCD UI, and document the
access steps. Add the node-role LBC and EBS-CSI IAM policies the controllers
need (Spinifex wires controller creds at the node level, not IRSA) and an
ignore_changes block for the cluster's access_config and
vpc_config security_group_ids, which Spinifex DescribeCluster does not echo
back, so the provider stops trying to replace the cluster on every plan.
* fix: stop addon-sync substituting IRSA markers named in comments
The ebs-csi controller manifest's header comment spelled out the literal
{{IRSA_ENV}} / {{IRSA_VOLUME_MOUNT}} / {{IRSA_VOLUME}} marker tokens. addon-sync
matched each marker by substring, so the comment lines were replaced with the
env/mount blocks, injecting a bare YAML sequence between two documents. K3s'
auto-deploy then rejected the whole file (ParseManifestFailed: cannot unmarshal
array into unstructured), so the ebs-csi Addon CR never applied and the EKS
addon hung in CREATING. Reword the comment to not contain the literal tokens,
and anchor the marker substitution to lines that are exactly the marker
(whitespace aside) so a future comment mentioning a marker can't retrigger it.
* fix: disable K3s local-storage on EKS clusters in parity mode
In AWS LB Controller parity mode (BuiltinIngress=false) the K3s server
disabled traefik and servicelb but left local-storage enabled. Its local-path
provisioner ships a StorageClass marked default, which races the EBS CSI
ebs-gp3 default StorageClass — two defaults, so dynamic PVCs bind
nondeterministically. EKS has no local-path, so disable local-storage too and
let ebs-gp3 be the sole default. The built-in-ingress path is unchanged.
* fix: re-encrypt HTTPS target groups in haproxy ALB backends
An ALB target group with backend-protocol HTTPS (e.g. argocd-server, which
serves TLS on its own port) rendered a plaintext haproxy server line. HAProxy's
own check then spoke HTTP to a TLS port, failed, and marked the server DOWN,
returning 503 — even though the lb-agent's separate HTTPS health probe reported
the target healthy, so describe-target-health masked the fault.
Emit check-ssl ssl verify none on the server line for HTTPS target groups so
both the check and proxied traffic re-encrypt to the backend.
* fix: reconnect wedged QMP client on next command
A timed-out QMP decode under host contention left the cached client's
shared json.Decoder mid-message, wedging every later command and breaking
EBS AttachVolume/DetachVolume for the VM's lifetime. Mark the client Dead
on encode/decode error and redial the retained socket path (re-running the
qmp_capabilities handshake in place) on the next send.
* fix: round-trip security-group rule descriptions
SG ingress/egress rules dropped the per-rule Description on store and
describe. The AWS Load Balancer Controller tags the worker-node SG rules it
manages with a UserIdGroupPair description and identifies them by that value
to revoke them on teardown. Without the round-trip the controller cannot
match its rules, never revokes them, and the backend security group stays
referenced — DeleteSecurityGroup returns DependencyViolation forever,
hanging the ingress finalizer and tofu destroy with orphaned security groups.
Persist Description on SGRule (excluded from sgRuleKey so it is metadata, not
identity) and render it back through describe.
* fix: reclaim customer-VPC nodegroup SGs on managed-CP-VPC teardown
In the managed control-plane VPC topology, launchNodegroupInfra creates the
cluster control-plane and nodegroup security groups in the customer VPC for
worker<->control-plane networking, but purgeClusterInfra only reclaimed the
security groups in the managed CP VPC. The customer-VPC SGs orphaned,
cross-referenced each other, and pinned the customer VPC with
DependencyViolation, hanging tofu destroy of the VPC. Reclaim them in the
managed-CP-VPC branch too, matching the legacy customer-VPC branch.
* fix: reclaim ALB EIP when system VM already gone on terminate
TerminateSystemInstance only released an internet-facing ALB's Elastic IP on
the local path, where the VM is still present in the manager. When the VM was
already gone (health-reaped, crashed, or raced by ENI teardown), it took the
remote/no-owner path and returned idempotent success without reconciling the
EIP, leaking the wan-pool address still associated to a deleted ENI.
Add EIPServiceImpl.ReleaseAddressByInstanceID, an idempotent account-agnostic
scan that disassociates then releases any EIP recorded against an instance, and
call it as a backstop in TerminateSystemInstance for both branches.
* fix: cascade-prune demo PVC via Argo CD finalizer in eks-gitops workbook
The demo Application carried no resources finalizer, so tofu destroy removed
the Application CR without pruning the synced PVC. The PVC survived, the cluster
was destroyed, and the Delete-reclaim PV's EBS volume orphaned. Add the
resources-finalizer.argocd.argoproj.io finalizer so deleting the Application
cascades to the PVC and ebs-csi reaps the volume before cluster teardown.
* feat: raw-IP access + documented Argo CD creds in eks-gitops workbook
The shared ALB host-routed everything, so the raw ALB IP 404'd in a browser
without an /etc/hosts entry. Make the app the hostless catch-all on :443 and add
a hostless Argo CD listener on :8443, so https://<alb-ip>/ serves the app and
https://<alb-ip>:8443/ serves the UI with no DNS. Surface the admin-password
command as a tofu output and document the credential + IP URLs in the README.
* fix: reclaim ALB EIP on the owning-node terminate path
The EIP backstop lived only in TerminateSystemInstance, but the ALB
delete path goes serveSystemTerminateInstance -> terminateSystemInstanceLocal
directly, bypassing it, and releaseSystemInstanceEIP relied on the VM's
cached allocation which is often empty. Extract reclaimSystemInstanceEIP
(authoritative KV reconcile via ReleaseAddressByInstanceID) and call it
from terminateSystemInstanceLocal as well as the no-owner backstop, so an
internet-facing ALB's EIP is released on every terminate route.
* chore(eks): drop coredns from the managed addon catalog
Cluster DNS ships as the baked node-local coredns-mulga DaemonSet, not as
a managed addon — there is no bundled coredns manifest, so CreateAddon
coredns would stage but fail VM-side render. Remove the phantom catalog
entry so DescribeAddonVersions and CreateAddon no longer advertise it;
repoint the addon tests that used coredns as their sample to argocd.
* feat(ui): group EKS and ECR under a Containers nav section
Fold the separate EKS and ECR sidebar groups into one Containers group
with EKS (clusters) and ECR (repositories) entries.
* fix(ui): satisfy strict null checks in ECR component tests
Index mock.calls via optional chaining and narrow the getAllByRole result
before clicking, clearing the TS2532/TS2345 errors that failed pnpm fix.
* refactor(eks): hide internal fixture addons from the public catalog
Add an AddonSpec.Hidden flag and mark spinifex-noop hidden: it stays
creatable (lookupAddon finds it, so the delivery-transport e2e still drives
it) but no longer surfaces in the unfiltered DescribeAddonVersions listing.
* refactor(eks): remove K3s built-in ingress; LBC-only
Drop the BuiltinIngress option and its managed-ingress tag opt-in. traefik,
servicelb, and local-storage are now always disabled on EKS nodes (AWS
parity — Service type=LoadBalancer and Ingress are the Load Balancer
Controller's job). Removes the EKS_DEFER_TRAEFIK userdata flag, the k3s.initd
traefik .skip staging, and the maybe_enable_traefik deferred-enable loop in
the state reporter.
* fix(ui): clear lint debt in VPC-wizard mutations and CIDR validation
Parallelize the independent subnet-create and route-table-association loops
in the create-VPC mutation with Promise.all (each task records its resource
in the rollback list before any further call), and switch the CIDR-validation
regex to named capture groups. Clears the no-await-in-loop and
prefer-named-capture-group lint errors.
* fix: carry gateway port in ECR registry endpoints; rename mulga.internal -> spinifex.internal
The ECR registry is served on the gateway port (:9999), but proxyEndpoint and
repositoryUri were emitted port-less, so docker login/tag/push dialed :443 and
failed. Derive the port from the awsgw listen host and append it to the registry
host (omitted for 443). Also rename the default internal DNS suffix.
* docs: drop verbose ECR :9999 host note
* docs: fix literal-asterisk emphasis (single-* not rendered)
* docs: brace-wrap ${REGISTRY}:latest (zsh :l modifier ate the tag); document Docker CA trust
* feat: default ECR registry URIs to the gateway host (no-DNS interim)
The registry authenticates by token, not hostname, so the per-account
<acct>.dkr.ecr name is cosmetic. Advertise the gateway's reachable bind host:port
(already a cert SAN) in repositoryUri/proxyEndpoint so docker login/tag/push need
no DNS; the parity host still works server-side where DNS exists.
* fix: serve ECR registry on a DNS-free gateway host end to end
Advertise the gateway's AdvertiseIP as the ECR registry host when the AWSGW
bind is unspecified (0.0.0.0/::), so repository_url resolves to a reachable
host:9999 carried in the server cert SANs rather than the unresolvable
per-account parity name. Docker push then dials it directly with no DNS.
Mirror the same host on EKS workers: the registries.yaml mirror keys and the
credential provider matchImages now cover the port-less parity name, the
port-ful parity name, and the gateway IP:9999, all redirecting to the dialed
gateway endpoint. A port-ful pull from the API-returned repository_url matches
the mirror instead of bypassing it into a failed DNS lookup.
Net: docker push host, pod image host, and the API's repository_url are the
same value, with no DNS or /etc/hosts.
* docs: resolve eks-https Ingress ALB DNS name to its public IP (no DNS)
* docs: resolve eks-gitops Ingress ALB DNS name to its public IP (no DNS)
* docs: remind to run aws eks update-kubeconfig before kubectl in EKS workbooks
* feat(ecr): return OperationNotSupportedException for image-scanning actions
The ECR image-scan control-plane actions resolved to the generic
NotImplemented 501 stub, indistinguishable from unbuilt features and
surfaced as a server fault. mulga runs no vulnerability scanner, so the
scan surface now returns OperationNotSupportedException (400):
- awserrors: add OperationNotSupportedException (400) ECR code.
- ecrapi: ScanningNotSupported handler; repoint the four image-scan
actions and register the three registry-scanning actions at it.
- spinifexui: Scan tab on the repository detail page renders a static
'scanning not supported' panel rather than a request that errors.
* refactor(config): default internal DNS suffix to spinifex.internal
Rename the AWS-parity internal DNS suffix from mulga.internal to
spinifex.internal across the config default, cert SAN builder, ECR
endpoint/audience construction, and their tests. Single runtime knob is
config.DefaultAWSInternalSuffix; no DNS backend exists yet, so this is a
naming change with no functional impact.
The ECR auth bridge minted ES256 tokens from a key that was created once and
never rotated. Add a background rotator (ecr-v1 Q3): mint a fresh key every 30
days, retain the previous keys for 24h so in-flight 12h tokens stay verifiable,
then prune. The active key is now the newest by KV creation time (not the
lexically-greatest kid) so a minted key takes over and all nodes converge. The
Issuer and Verifier became hot-swappable so the rotator updates the live
gateway as keys roll.
* feat: bake aws-ebs-csi-driver addon + mdev by-id bridge
Add the aws-ebs-csi-driver entry (v1.40.1) to the eks addon catalog and
ship its baked manifests (RBAC, IRSA-wired controller, node DaemonSet,
default gp3 StorageClass) under the eks-node image.
The node plugin locates EBS volumes via /dev/disk/by-id, which busybox
mdev does not populate. Add mulga-ebs-byid: an mdev rule + helper that
symlinks nvme-Amazon_Elastic_Block_Store_<serial> from the virtio-blk
serial, plus a boot oneshot that points the kernel hotplug helper at
mdev and coldplugs volumes present at boot.
* test: e2e EBS CSI volume — PVC bind + mount + reschedule survival
Add a TestEKS/EBSCSIVolume subtest that installs the aws-ebs-csi-driver
managed addon bound to an IRSA role, provisions a default gp3 PVC + pod,
and asserts a marker written by the first pod survives a reschedule onto a
fresh pod backed by the same PVC. Exercises the full data path:
CreateVolume(Viperblock)/AttachVolume virtio-blk hotplug with serial set to
the volume-id, mdev /dev/disk/by-id symlink, and ext4 mount.
* feat: eks-node provider-id + topology/instance-type labels for EBS CSI
mulga-eks-provider-id boot oneshot writes a K3s config.yaml.d drop-in
from IMDS before K3s starts: kubelet provider-id plus the region, zone
and instance-type node labels. The aws-ebs-csi-driver node plugin reads
this metadata for NodeGetInfo (instance id from providerID, region/zone
from topology labels, instance type for the volume-limit family lookup).
k3s runs no AWS cloud-provider so none of it is set otherwise, which left
the node plugin's metadata init failing, NodeGetInfo panicking in
IsNitroInstanceType, and the CSINode carrying no topology key. Wire the
service into the image manifest (INSTALL_FILES + ENABLE_SERVICES) and
chmod it in setup.
* test: e2e EBS CSI controller IRSA policy + failure diagnostics
awsgw enforces IAM on assumed-role EC2 calls, so the controller's
CreateVolume/AttachVolume need a permission policy on the IRSA role.
Attach a customer-managed EC2 policy (AWS-managed ARNs are opaque and
grant nothing) covering the EBS CSI action set; CreateVolume then
succeeds. Add an OnFailure dump of the CSI control path (provisioner and
plugin logs, pvc/pod describe, events, csinode) to the EBSCSIVolume
subtest so provisioning/attach failures are diagnosable post-teardown.
* feat: EBS CSI obeys control-plane taint; e2e worker nodegroup
Drop the CriticalAddonsOnly/NoExecute tolerations from the EBS CSI
controller + node manifests so the driver runs only on customer worker
nodes, never the system control-plane VM. The e2e creates a 1-node
worker nodegroup and the writer/reader pods drop their CP tolerations so
the customer-owned volume attaches in the customer space.
* fix: EBS CSI attach — decouple hotplug PCIe port from device name; iothread rollback
virtio-blk-pci hot-plug bus was derived from the AWS device-name letter
(/dev/sd[f-p]); the EBS CSI driver sends arbitrary names (/dev/xvdaa) that
map to no slot, so device_add fell back to pcie.0 — which rejects hot-plug —
and the attach hung. A failed attach also left the per-volume iothread object
behind, so every retry died on object-add 'duplicate property'.
nextHotplugEBSPort allocates the lowest free hotplug-ebs port from live
query-block state, independent of the device name. delIothreadBestEffort
removes the iothread on every attach rollback path.
* fix: EBS CSI node discovery — un-shadow mdev by-id rule, mint nvme link
Alpine eks-node runs busybox mdev (no eudev), which stops at the first
matching /etc/mdev.conf rule. setup.sh appended the mulga-ebs-byid rule after
the stock 'vd[a-z].* -> persistent-storage' line, so it never fired on
hot-attach: only /dev/disk/by-id/virtio-<serial> was created, while the
upstream aws-ebs-csi-driver resolves nvme-Amazon_Elastic_Block_Store_<volid>.
Swap the stock vd* rule in place to route through mulga-ebs-byid, which now
delegates to persistent-storage (preserving stock links) before minting the
nvme-style link the node plugin expects.
* test: EKS e2e — NAT Gateway egress for private worker nodegroup
The customer-VPC worker is private; AttachIGW programs no SNAT (mirrors AWS:
an IGW serves only public-IP instances), so an IGW-only fixture left the worker
unable to reach the public NLB endpoint or pull public CSI images — the
nodegroup never joined.
createWorkerEgress now builds the real topology: an IGW-fronted public subnet
hosting a NAT Gateway, with the worker subnet default-routed to it. Associating
the worker route table triggers the subnet-wide SNAT egress reroute. Teardown
drains the NAT Gateway before releasing the public subnet + EIP it pins.
The OCI /v2 registry auto-created a repository on first push (ensureRepo).
AWS ECR does not: a push or PutImage to a repository that was never created
with CreateRepository is rejected. Replace ensureRepo with requireRepo, which
returns OCI NAME_UNKNOWN on the data plane and RepositoryNotFoundException on
the JSON PutImage path. Enforcement sits at the single push choke point
(startUpload + StoreManifest).
* test(e2e): add ECR end-to-end suite (control plane + docker/crane/skopeo)
New tests/e2e/ecr/ suite plus harness support:
- harness: ECR SDK client, control-plane helpers (repo CRUD, get-login-password,
describe-images), and docker/crane/skopeo CLI wrappers that skip when the
binary or registry-host resolution is absent.
- suite: control-plane CRUD + policy + lifecycle round-trip; docker login/push/
pull; crane append/pull and skopeo copy exercising the Bearer /v2/token flow.
- registered e2e-ecr in the Makefile + service manifest.
Predastore capacity/load leg is a skipped placeholder pending mulga-siv-359/356.
* test(e2e): carry awsgw port in ECR registry host for dev clusters
Dev clusters bind awsgw on a non-standard port (e.g. 9999); production uses
:443. ECRRegistryHost now appends SPINIFEX_ECR_PORT when set so docker/crane/
skopeo address the right endpoint, and RequireRegistryResolves strips the port
before DNS lookup.
getManifest read only the first Accept header line via r.Header.Get, so
clients that send each acceptable manifest media type as a separate Accept
header (skopeo) never matched the stored type and got 406 MANIFEST_INVALID.
Join all Accept header values before negotiation.
Add the Docker Registry v2 token endpoint (GET /v2/token) and switch the
/v2 401 challenge from Basic to a Bearer realm pointing at it. The endpoint
verifies the caller's existing AWS:<jwt> credential and re-mints a fresh
full-TTL token, providing the token-refresh path for OCI clients (docker,
crane, skopeo). The auth bridge still accepts Basic AWS:<jwt> directly.
Stamp the dash-stripped volume-id (vol+17hex = 20 bytes, the virtio-blk
serial limit) as the hotplug disk serial so the EBS CSI node plugin can
locate the device via /dev/disk/by-id and match lsblk SERIAL.
* feat: ECR background lifecycle expiry sweep
Add a gateway-side periodic sweep (LifecycleSweeper) that applies each repo's
stored lifecycle policy: it evaluates the shared rule engine against the repo's
current images and deletes the expired set through the existing registry GC
reclaim path (DeleteImage -> reclaimManifest). It runs in the gateway because
only the gateway holds a Registry (object store + meta); the daemon owns the KV
but cannot reclaim predastore bytes. Launched on the STS janitor's lifetime
context in awsgw, enumerating ACTIVE IAM accounts.
Per repo: skip when no policy; evaluate; then before deleting, re-read the
current tag set and skip any image whose tags changed since evaluation
(optimistic-concurrency guard against concurrent push/re-tag). Cross-repo blob
mount safety is provided by reclaimManifest/referencedDigests. Per-item errors
are logged and skipped so one bad repo never aborts the cycle.
* test: cover lifecycle sweeper Run loop and active-account filter
Add tests for LifecycleSweeper.Run ticker loop, NewLifecycleSweeper
interval fallback, and sweepRepo error branches. Extract accountLister
interface so activeAccountIDs is unit-testable. Raises diff coverage
on the sweep changes from 53.4% to 76.9%.
* feat: ECR lifecycle policies (CRUD + evaluation engine + preview)
Implement PutLifecyclePolicy/GetLifecyclePolicy/DeleteLifecyclePolicy and
StartLifecyclePolicyPreview/GetLifecyclePolicyPreview, replacing the
NotImplemented stubs. A pure rule-evaluation engine parses the AWS
lifecycle-policy JSON and, given a repo's images, returns the ordered set that
would expire; each image is claimed by at most one rule (lowest rulePriority
first). Preview is served synchronously and deletes nothing.
Lifecycle-policy documents persist as a per-repo KV doc (repos/<name>/lifecycle)
over the ecr.lifecycle.* NATS subjects, mirroring the repo-policy storage stack,
and cascade on DeleteRepository. The console gains a Lifecycle tab with a policy
editor and a preview table over the saved policy.
The background expiry sweep that applies the engine and deletes images via GC is
deferred to mulga-siv-387.
* feat: client-side JSON validation in ECR policy editors
Swap the raw Textarea in the repository-policy and lifecycle-policy editors for
the shared JsonEditor (blur-time pretty-print) and gate Save on isValidJson,
showing an inline error for malformed JSON — matching the IAM/EKS editors. The
backend remains the source of truth for schema validation.
Honor, persist and echo imageTagMutability on CreateRepository and
DescribeRepositories instead of the hardcoded MUTABLE, and implement
PutImageTagMutability to flip a repository between MUTABLE and IMMUTABLE.
Enforce immutability at the shared StoreManifest chokepoint: on an
IMMUTABLE repo, re-tagging an existing tag onto a different digest is
rejected before any write (OCI 409 TAG_IMMUTABLE; JSON PutImage maps to
ImageTagAlreadyExistsException). Same-digest re-pushes, new tags and
digest references are still allowed.
Console: migrate the repository detail page to Summary/Images/Permissions
tabs and add a tag-immutability toggle on the Summary tab.
DELETE /v2/{name}/manifests/{digest} was a 202 no-op. Now delete-by-digest
removes the manifest record, deletes the manifest object, and sweeps the
account-wide blob pool: each child blob no longer referenced by any live
manifest is reclaimed, while shared blobs survive.
- DeleteImage (digest path) captures ChildDigests, deletes the manifest object,
then reclaimManifest() drops orphaned blobs via referencedDigests() (a
mark-and-sweep over the live manifest set across the account's repos).
- OCI deleteManifest by-digest wired to DeleteImage (real work + 202; 404 on
unknown). Tag delete stays KV-only so untagged images persist (AWS-faithful).
- Predastore reclaim is best-effort: a failure leaks bytes but never undoes the
logical delete.
Serve the five image-management JSON 1.1 actions inline in ECR_Request,
reusing the gateway-side OCI Registry for manifest bytes:
- ListImages / DescribeImages enumerate images by tag + digest (KV meta).
- BatchGetImage returns verbatim manifest bytes with Q14 partial-failure
semantics (cap 100, digest wins over tag, 200 + failures array).
- PutImage stores a manifest + tag via a shared StoreManifest core that the
OCI manifest PUT now delegates to.
- BatchDeleteImage removes tag pointers / manifest meta (blob GC deferred).
Adds DeleteManifestMeta across the MetaStore relay chain and ECR image
error codes.
Add the two missing repository-lifecycle actions, served inline in
ECR_Request (like DescribeRepositories) since both return the repository
ARN/URI built from the gateway region + internal suffix.
CreateRepository validates the name, guards cross-account registryId,
rejects an existing repo (RepositoryAlreadyExistsException), and writes
the per-account KV meta (predastore bucket stays lazy on first push).
DeleteRepository rejects a non-empty repo without force
(RepositoryNotEmptyException) and cascades the KV records (meta, policy,
tags, manifests). Predastore blob GC is deferred to a separate pass.
Adds MetaStore DeleteRepo + ListManifests with the matching daemon-relayed
subjects, service methods, and NATS client, plus two new awserrors codes.
The per-run ARTIFACT_DIR lives in /tmp, outside GITHUB_WORKSPACE, so a
re-run on the same self-hosted runner reused a prior attempt's append-only
suite-status.txt. The 'Fail job if any suite failed' gate's -s check then
failed the job even when the surviving attempt passed with rc=0.
Wipe ARTIFACT_DIR in Clean Workspace and gate solely on the per-attempt
steps.e2e.outputs.rc (fail-closed: an empty rc still fails).
CreateTags/DeleteTags wrote only the standalone tag store, which
DescribeImages does not read (it reads AMI metadata), so tag mutations on
an AMI were invisible — DeleteTags appeared to be a silent no-op. Mirror
tag add/remove into the owning AMI config (ownership-checked; absent or
foreign AMIs skipped) so DescribeImages reflects them, matching the
existing subnet/vpc mirror.
* feat(ecr): auth bridge + GetAuthorizationToken JWT (Sprint 2c)
Mint a self-contained ES256 JWT from GetAuthorizationToken and verify it on
the /v2 OCI registry via a new ECRAuthBridgeMiddleware (Bearer | Basic AWS:jwt).
Signing keys live in the cluster-replicated awsgw-keys KV bucket under
jwt-signing/{kid}, encrypted with the IAM master key.
The registry is now scoped per-request to the token's account (replacing the
fixed bootstrap account); bucket-ensure is cached per account. Cross-account
token/host mismatch is rejected 403; unauthenticated /v2 access gets a 401
Bearer challenge.
* fix(ecr): use Basic auth challenge on /v2 (not Bearer token-endpoint flow)
ECR authenticates the OCI registry with Basic AWS:<token>; a Bearer realm
made docker chase an unimplemented /v2/token endpoint and fail login.
* feat(ui): inline EKS cluster IAM role creation in wizard
* feat(ui): show trust policy + attached managed policy in EKS role dialog
Render the trust policy with PolicyDocumentViewer (pretty-printed, safe
fallback) instead of a raw read-only textarea, matching the IAM detail
pages. Surface the attached AmazonEKSClusterPolicy ARN so users can see
exactly which permissions the cluster role grants.
* fix(ec2): stamp subnet AvailabilityZoneId for single-AZ ALB resolution
DescribeSubnets omitted AvailabilityZoneId, so the AWS Load Balancer
Controller's DescribeAvailabilityZones-by-ZoneId lookup resolved no AZ
info ("cannot resolve AZ info for AZs: []") and never built the ALB
model. Stamp the single zone-id on every subnet and share it with
DescribeAvailabilityZones through one SingleZoneID constant so the two
responses can never drift.
* feat(eks): add aws-load-balancer-controller v2.8.1 addon bundle
Baked manifests for the LBC managed addon (CRDs, controller Deployment/
RBAC/webhooks/Service, IngressClass alb, IRSA render markers). Two
wirings make it work against the spinifex gateway:
- --aws-api-endpoints carries sts= alongside elasticloadbalancing/ec2/
acm. The aws-sdk-go v1 credential bootstrap resolves the STS client
from the session endpoint resolver, not AWS_ENDPOINT_URL_STS, so the
web-identity AssumeRoleWithWebIdentity must be pointed at the gateway
here or it falls back to real AWS STS.
- --feature-gates=ALBSingleSubnet=true drops the ALB minimum subnet
count from two to one so an ALB provisions in a single-AZ deployment.
* feat(eks): render IRSA into managed addons + node-minted webhook cert
Extend the addon-sync renderer to inject the projected-token / role-arn /
gateway STS-endpoint + CA block into managed-addon pod specs via opt-in
markers, and substitute cluster-name/region/vpc-id/gateway-endpoint/CA
placeholders. Add cmd/eks-webhook-cert: the eks-node AMI has no openssl,
so this mints (then reuses) a self-signed ECDSA-P256 serving cert for the
admission webhook, cached on disk so the render is byte-stable across
ticks. Wire both into the image build (manifest.conf) plus INSTALL_FILES
for the LBC bundle.
* feat(eks): surface cluster VPC id as EKS_VPC_ID for the LB controller
Thread the control-plane VPC id through K3sServerInput.VpcID into the
first-boot EKS_VPC_ID env so the in-cluster aws-load-balancer-controller
can pass --aws-vpc-id to the gateway elbv2/ec2 handlers.
* fix(eks): point the LB controller at the customer VPC + customer-facing gateway
The aws-load-balancer-controller ran with two control-plane-scoped values
that customer worker pods cannot use:
- --aws-vpc-id was set from the managed control-plane VPC (cpRefs.VpcID),
which the gateway's DescribeVpcs does not return for customer calls, so
backend-SG auto-create failed with InvalidVpcID.NotFound. Source it from
meta.ResourcesVpcConfig.VpcId, the customer cluster VPC where the workers
and their subnets live and where the ALB is provisioned.
- The addon render baked the mgmt gateway endpoint (EKS_GATEWAY_URL /
SystemGatewayURL) into pod specs, unreachable from worker pods. Add a
distinct EKS_ADDON_GATEWAY_URL (the customer-facing GatewayBaseURL, same
host as the OIDC issuer) and substitute it for {{GATEWAY_ENDPOINT}} in
managed addon manifests. EKS_GATEWAY_URL stays mgmt for the CP's own
bootstrap, state-report and token-webhook paths.
* fix(elbv2): resolve forward target group from ForwardConfig
listenerActionFromSDK read only the legacy flat Action.TargetGroupArn, so a
forward action carrying its target group via ForwardConfig.TargetGroups (the
shape the aws-load-balancer-controller emits) left the action target empty and
CreateRule/CreateListener rejected it with MissingParameter. Flatten the first
target group from ForwardConfig when the flat field is absent.
* fix(ec2): mirror CreateTags/DeleteTags into subnet/vpc records
The generic S3 tag store and the per-resource KV records are disjoint:
DescribeSubnets/DescribeVpcs read tags from their KV record, while
CreateTags only wrote the generic store. Subnet tags applied after
create were therefore invisible to tag-filtered describes, breaking
aws-load-balancer-controller subnet auto-discovery (kubernetes.io/role/elb).
Write through CreateTags/DeleteTags to the owning subnet/vpc record so
tag-filtered describes observe them. Resource types the VPC service does
not own are skipped; the generic store stays their record of truth.
* test(daemon): pin synthetic capacity in host-dependent allocate tests
TestCanAllocate_CountEdgeCases/Mixed_instance_types and
TestHandleEC2DescribeInstanceTypes/NoFilterReturnsSupportedRegardlessOfAllocation
called rm.allocate against live-detected host capacity. allocate gates on
readMemAvailableGB (live /proc MemAvailable), so a box already running VMs
fails the allocation spuriously, breaking the preflight gate locally.
Pin hostVCPU/hostMemGB/reserved and nil readMemAvailableGB to a synthetic
host, matching the sibling Capacity_decreases_after_allocation subtest.
* fix(eks): route apiserver->pod traffic via agent tunnel (egress-selector cluster)
In the managed-CP topology the apiserver VM sits in the system CP VPC with
no route to the worker pod network in the unpeered customer VPC. k3s default
egress-selector-mode=agent tunnels only apiserver->kubelet, so kubectl
logs/exec work but apiserver->pod traffic (admission webhooks, aggregated
APIs) is attempted directly and times out -- the aws-load-balancer-controller
webhooks fail with context deadline exceeded, blocking Service/Ingress
creation cluster-wide.
Set egress-selector-mode=cluster so apiserver->pod/service traffic routes
through the agent's outbound tunnel, selected by the node hosting the
destination, without requiring inbound CP->worker reachability.
* feat: bump aws-load-balancer-controller addon bundle to v2.11.0
v2.8.1 predates the ALBSingleSubnet feature gate, so single-AZ spinifex
clusters fail Ingress reconcile (subnet resolver dedups to 1 < required 2)
and never provision an ALB. v2.11.0 honors --feature-gates=ALBSingleSubnet=true.
Replace the baked bundle: new CRDs (ingressclassparams + targetgroupbindings),
controller manifest, and image tag v2.11.0, re-applying the spinifex deltas
(IRSA SA annotation, node-minted webhook serving-cert secret, gateway-CA
secret, gateway aws-api-endpoints, caBundle on every webhook clientConfig,
alb IngressClass). Pick up the new ClusterRole configmaps rule. Drop v2.8.1.
* fix: point LBC at the gateway via AWS_ENDPOINT_URL (sdk-v2)
LBC v2.11 uses aws-sdk-go-v2, which keys --aws-api-endpoints on SDK
service IDs (e.g. 'Elastic Load Balancing v2'), not the legacy lowercase
names the bundle passed. Those overrides were silently ignored, so every
non-STS call (ec2/elbv2/acm) went to real AWS and failed TLS against the
gateway CA -- no ALB could provision.
Inject AWS_ENDPOINT_URL in the shared IRSA env block (sdk-v2 global
base-endpoint override; AWS_ENDPOINT_URL_STS still takes precedence for
STS) and drop the dead --aws-api-endpoints flag from the controller args.
* fix: synthesize taggable ARN for default listener rule
Gateway DescribeRules returned the synthetic IsDefault rule with a null
RuleArn. LBC read the arn-less default rule after creating a listener and
issued DescribeTags with an empty resource ARN, which the gateway rejected
with 400 MissingParameter, aborting reconcile.
Set a deterministic RuleArn on the default rule, resolve it back to its
parent listener in DescribeTags (empty tags) and DescribeRules by-arn.
Also disable shield/waf/wafv2 in the LBC bundle to drop unsupported calls.
* feat: inject ELB subnets into alb IngressClassParams for single-AZ ALB
LBC's tag auto-discovery never threads the ALBSingleSubnet feature gate, so a
single-AZ cluster (subnets sharing one AZ) dedups to 1<2 subnets and aborts
Ingress reconcile. Only the explicit-subnet path honors the gate.
Resolve the cluster's ELB-eligible subnets, dedup to one per AZ, and surface
them as EKS_ELB_SUBNET_IDS. The addon sync renders them into the alb
IngressClassParams .spec.subnets.ids so every Ingress takes the explicit path.
An empty set falls back to auto-discovery, preserving the native UX.
* fix: route LBC service clients to gateway via --aws-api-endpoints, link IngressClassParams
Live verification showed two gaps in the LBC addon bundle:
LBC installs a per-client BaseEndpoint resolver, so the global AWS_ENDPOINT_URL
is ignored for the ec2/elbv2/acm clients (calls hit real AWS, failing TLS).
Redirect them with --aws-api-endpoints keyed on the aws-sdk-go-v2 service IDs
(EC2, "Elastic Load Balancing v2", ACM, STS) rather than the legacy lowercase
names the flag never matched. Drop the misleading AWS_ENDPOINT_URL env; the
web-identity STS client still resolves via AWS_ENDPOINT_URL_STS.
The injected IngressClassParams was orphaned: the alb IngressClass had no
.spec.parameters, so LBC fell back to subnet auto-discovery (which fails on a
single-AZ cluster). Link the IngressClass to the IngressClassParams so the
explicit-subnet path is taken.
* fix: stamp EKS-parity providerID on worker nodes for LBC ip targets
k3s registers workers with providerID k3s://<name>, which the AWS Load
Balancer Controller rejects (providerID invalid for EC2 instances) when
resolving nodes for ip-mode target registration. Resolve the instance-id and
AZ from IMDSv2 at agent boot and set --kubelet-arg=provider-id=aws:///<az>/<id>
so LBC can map the node to its EC2 instance. On any IMDS miss the agent keeps
the default providerID and still starts.
* fix(elbv2): honor SubnetMappings in CreateLoadBalancer
LBC supplies ALB subnets via SubnetMappings, not the plain Subnets list.
createLoadBalancer only read input.Subnets, so ENI allocation was skipped,
VpcId/AZ stayed empty, and the HAProxy data-plane VM never launched — the
ALB went active but could not serve traffic. Add flattenSubnetIDs to read
both sources and reuse it in createLoadBalancer and desiredSubnetSet.
* fix(vm): capture GCInterval before spawning GC goroutine
TestStart_SweepsImmediatelyThenStopsOnCtxCancel mutates the package global
GCInterval and restores it in a defer, racing the Start goroutine's read at
time.NewTicker(GCInterval) under -race. Read GCInterval on the caller's
goroutine into a local before spawning so the global access is ordered with
the test's defer; the goroutine touches only the captured local.
* feat: implement OCI Distribution v2 blob and manifest storage
Replace the /v2/* 501 stub with a real path-parsing dispatcher and
content-addressable storage for the ECR registry surface:
- Blob upload (monolithic + chunked PATCH), HEAD/GET, cross-repo mount,
and cancel, with server-side sha256 verification on finalize.
- Manifest PUT with type-aware shallow validation (image manifest blob
existence; image index child-manifest existence + mediaType match),
byte-for-byte GET with stored Content-Type, HEAD, and tag delete.
- Tag list and account-scoped catalog.
Blob and manifest bytes stream straight to predastore; repo, tag,
manifest and in-progress-upload metadata live in a per-account
JetStream KV bucket with optimistic-concurrency CAS on uploads. Chunks
are buffered to a temp upload object and reassembled on finalize, since
predastore exposes no append primitive through the shared object-store
interface. Adds HeadObject to the objectstore abstraction.
Refs: mulgadc/mulga#73
* refactor(ecr): route registry metadata through NATS daemon handlers
The gateway no longer talks JetStream KV directly. The daemon owns the
per-account KV via a MetaService bound to ecr.* request/reply subjects;
the gateway is a NATS client (NATSMetaStore). Blob and manifest bytes
still stream gateway-to-predastore directly.
Also fixes four registry review findings:
- finishUpload: clean up the temp object and upload record on a blob
store failure so the uuid is reusable and no partial blob lingers.
- patchUpload: commit the appended bytes and advanced hash state under a
single KV CAS keyed per write, so a racing PATCH is rejected (409)
rather than corrupting the assembled bytes; the verified digest always
matches the stored bytes.
- cancelUpload: delete both the temp bytes and the record, 204 on
success, 404 on unknown uuid, no dead branch.
- getManifest: return MANIFEST_INVALID (not MANIFEST_UNKNOWN) for a 406
Accept media-type mismatch.
Refs: mulgadc/mulga#73
* fix(ecr): provision account predastore bucket on demand
OCI blob/manifest writes target ecr-{accountID}, which nothing created,
so every push failed with a backend 403. Add idempotent EnsureBucket to
the object store (HeadBucket -> CreateBucket, swallowing already-owned)
and ensure it once per process at the top of the registry ServeHTTP, so
ECR storage appears transparently like AWS-managed ECR.
* fix(ecr): sanitize ':' from predastore blob/manifest object keys
predastore mis-canonicalizes SigV4 for object keys containing ':', so
writes/reads of content-addressable keys (sha256:<hex>) were rejected
403 (reproducible with the aws CLI too). Route BlobKey and ManifestKey
digests through DigestToken so the object key carries no ':' separator.
Implement the registry host-routing middleware so /v2/* requests addressed
at {accountID}.dkr.ecr.{region}.{suffix} carry the parsed target account and
region on the request context; non-registry hosts pass through unchanged.
Add cluster-wide AWS-parity config (aws.region, aws.internal_suffix) with
defaults, and append the AWS-parity ECR SANs (exact control-plane host plus
*.dkr wildcard) to the awsgw server certificate during cert generation.
Refs: mulgadc/mulga#78
* feat: scaffold ECR control-plane + OCI registry gateway with 501 stubs
Add the ECR HTTP surface on awsgw as compiling skeletons that return
NotImplemented, so later work can call into the packages against a fixed
contract.
- gateway/ecrapi: ECR control plane (AWS JSON 1.1, X-Amz-Target). Full action
namespace registered; every action resolves to a shared 501 stub.
- gateway/ecr: OCI Distribution v2 (/v2/*). Catch-all 501 with the OCI error
envelope; GET /v2/ is live for the registry version-check probe.
- gateway/hostmatch: no-op host-routing middleware skeleton for the registry
host group.
- handlers/ecr/storage: per-account bucket + object-key layout constants.
- gateway wiring: ecr in supportedServices, dispatch case, JSON 1.1 error
branch, and /v2 mounted outside the SigV4 group.
* test: cover ECR scaffolding packages for diff-coverage gate
In-package tests for gateway_ecr, gateway_ecrapi, and handlers/ecr so the new
code reports coverage under its own package profile. Diff coverage 76.5%.
The kubernetes provider in the eks-quickstart, eks-addons-access and
eks-https-ingress workbooks was configured from the managed cluster
resource (host = aws_eks_cluster.this.endpoint). On destroy the endpoint
resolved empty, the provider fell back to localhost:80, and refreshing
kubernetes_deployment_v1.hello / kubernetes_service_v1.hello failed with
'dial tcp [::1]:80: connect: connection refused'.
Move the kubernetes provider and the hello Deployment + Service into a
nested workloads/ root module with its own state. It reads cluster
identity from the parent via terraform_remote_state and the live
endpoint/CA from a data aws_eks_cluster source, so the provider is only
ever configured while the cluster exists. Apply parent then workloads;
destroy workloads then parent.
purgeClusterInfra stranded the cluster security groups two ways, leaving
the nodegroup/control-plane SGs (which live in the customer VPC) behind to
pin aws_vpc destroy on DependencyViolation:
* an early return before the SG cleanup block skipped DeleteClusterSGs
whenever any earlier best-effort step (NLB / ENI / VM / EIP) errored;
* DeleteClusterSGs failures were warn-and-swallowed, so a nodegroup SG
still referenced by a terminating worker's ENI leaked while the cluster
completed deletion — no reaper retry ever fired.
Surface DeleteClusterSGs / DeleteClusterCPVPC / DeleteClusterIGW failures
into teardownErrs and gate the KV sweep on them, so a leaked VPC-pinning SG
keeps the cluster DELETING for a retry. Add a bounded DependencyViolation
retry to the SG delete (mirroring deleteENIAwaitingTerminate) to ride out
the worker-ENI-detach race in the common case.
Fixes the manual 'revoke rules + delete SGs before the VPC will destroy'
teardown step.
* fix: persist instance TagSpecifications in RunInstance
RunInstance built the ec2.Instance metadata but never read
input.TagSpecifications, so DescribeInstances returned untagged
instances and tag filters matched nothing. EKS node groups discover
their workers by the spinifex:eks-cluster / spinifex:eks-nodegroup
tags, so scale-up convergence and worker reconciliation depended on
tags that were silently dropped.
Apply instance-scoped TagSpecifications onto the built instance so the
tags surface through DescribeInstances and tag filters.
* fix: serialize nodegroup reconcile via CAS to prevent scale overshoot
UpdateNodegroupConfig is dispatched on a queue group, so two
overlapping requests (e.g. an SDK retry mid-flight) each read
current = len(InstanceIDs) and each launched desired-current workers,
overshooting the target (1 -> 5 for desired=3) and stranding untracked
VMs via last-writer-wins on the unconditional PutNodegroupRecord.
Make every durable nodegroup-record mutation a compare-and-swap on the
KV revision. Scale-up launches one worker at a time and CAS-appends it
only while the record is below the live ScalingDesired, terminating any
surplus VM launched into an already-full record. Scale-down terminates
before the CAS truncation so a crash or retry leaves reclaimable dead
IDs rather than orphaned VMs. A concurrent regression test asserts two
overlapping scale-ups converge to exactly desired under -race.
* fix: return securityGroupRuleSet from AuthorizeSecurityGroup ingress/egress
The AWS provider's aws_vpc_security_group_ingress_rule reads
SecurityGroupRules[0].SecurityGroupRuleId on create; an empty ruleset
panics the plugin (Plugin did not respond), though the rule applies
server-side. Echo the created rules via sgRuleToSecurityGroupRule, the
same shape DescribeSecurityGroupRules already returns.
* fix: surface InsufficientInstanceCapacity from nodegroup scale, not 500
launchWorkersCAS wrapped the bare awserror code via fmt.Errorf, so the
gateway ErrorHandler's exact-match against ErrorLookup missed and remapped
to InternalError (500). An out-of-capacity scale now surfaces the real
InsufficientInstanceCapacity code (400); opaque internal failures still
wrap and stay 500. Adds awserrors.HasErrorCode for the membership test.
* fix: cut EKS node boot ~50s — drop cloud-final deadlock + syslinux menu
Worker/server user-data runcmd ended with `[ rc-service, local, start ]`,
which runs inside cloud-final while the OpenRC `local` service is ordered
after it — a deadlock broken only by OpenRC's ~50s start timeout, stalling
the k3s join. Run the IMDS route script directly instead and keep
`rc-update add local default` for reboot persistence.
setup.sh: cut the SYSLINUX boot-menu countdown 10s->~1s (kept nonzero so the
menu stays interruptible over serial).
Validated live on env19 CP: boot menu 1s, no `waiting for cloud-final`
stall, apiserver ready ~66s (was ~109s+).
mulga-siv-332
* feat: add argocd managed addon + bake bundle into eks-node AMI
The addon catalog advertised coredns and aws-load-balancer-controller, but
the only baked manifest shipped in the eks-node AMI was the spinifex-noop
fixture. Installing any real addon failed with 'no baked bundle for version'
once the VM-side renderer found nothing on disk.
Add argocd v3.0.23 as a real, demonstrable managed addon: catalog entry (no
IRSA), a flattened namespaced install manifest baked into the AMI, and the
manifest.conf bake line. The eks-addons-access workbook switches from coredns
(redundant on K3s, never baked) to argocd.
The baked bundle is generated from upstream install.yaml via kubectl kustomize
with namespace: argocd plus a prepended Namespace, since upstream is
namespace-agnostic and K3s auto-deploy applies with no -n flag.
* fix(eks): run CoreDNS as a node-local DaemonSet for worker DNS
k3s ships CoreDNS as a single-replica Deployment that tolerates the CP taint
and strands itself on the control-plane node. In the managed-CP topology the
worker<->CP pod overlay cannot cross the VPC boundary, so worker pods cannot
reach a CP-resident CoreDNS and lose all cluster + external DNS.
Skip the bundled CoreDNS (coredns.yaml.skip, staged in k3s.initd like traefik)
and deploy a Mulga-owned variant: a DaemonSet (one pod per node) plus kube-dns
internalTrafficPolicy=Local, so every node resolves via its own CoreDNS. Immune
to the cross-VPC overlay gap and topology-agnostic (single-node and managed-CP
both work). Image-side only; no handler change.
* fix(eks): revoke cross-referencing SG rules before delete on teardown
DeleteClusterSGs deleted the control-plane and nodegroup SGs directly, but they
cross-reference each other (EnsureNodegroupSGRules), and AWS refuses to delete a
security group still referenced by another SG's rule. Both deletes failed with
DependencyViolation, the SGs leaked, and the VPC could not be deleted (tofu
destroy hung on DeleteVpc).
Resolve both SG IDs, revoke all ingress on each to break the cp<->ng cycle, then
delete both. Adds RevokeSecurityGroupIngress to sgProvisioner (already on
VPCServiceImpl) and revokeAllSGIngress, with a regression test that models AWS's
DependencyViolation.
* fix(eks): advertise NLB front-end so worker pods reach the apiserver ClusterIP
In the managed-control-plane topology the apiserver advertised its Set-B
node-ip into the in-cluster `kubernetes` Endpoints. That IP lives in the
unpeered CP VPC and is unreachable from worker pods, so every worker pod
using the apiserver ClusterIP (CoreDNS kubernetes-plugin sync, controllers,
argocd redis secret-init) failed.
- k3s_server_vm.go: set advertise-address to the worker-reachable NLB
front-end (PrivateEndpointIP, else EndpointIP); both are already cert SANs.
- nlb.go: add a second :6443 listener on the cluster NLB (same CP target
group) so PrivateEndpointIP:6443 — the advertised endpoint — is served.
- security_groups.go: admit :6443 on the private-endpoint SG.
Validated on a live managed-CP cluster: worker CoreDNS /ready 200, all argocd
pods Running, kubectl logs on worker pods working.
The DisassociateAccessPolicy route regex terminated at
/access-entries/{principalArn}/access-policies, but the AWS EKS REST API
sends the policy ARN as the final path segment:
DELETE /clusters/{name}/access-entries/{principalArn}/access-policies/{policyArn}
The route never matched, so the gateway returned InvalidActionException
(seen as a 400 on 'tofu destroy' of aws_eks_access_policy_association).
The handler also read policyArn from the request body, which AWS leaves
empty for this DELETE.
Capture policyArn as a third path param and pass it through to the input,
dropping the body read. Update route and wrapper tests accordingly.
The host clock needs an NTP daemon; without one it drifts and SigV4 request
signatures fall outside the skew window, kicking the web console back to /login.
Mirrors the ISO rootfs (iso-builder packages.list) so both host-install paths
ship the same software. Guest VMs track the host via kvm-clock.
Bug: mulga-siv-335
* fix: preserve terminated-instance describe-visibility window in GC reaper
The TerminatedTeardownReaper purged a terminated record the moment it drove
the instance's pending NAT/OVN teardown to done. Normal terminations always
arrive NAT/OVN-pending, so the first 2m GC sweep completed and purged them
immediately, removing the instance from DescribeInstances before clients could
observe state=terminated — an AWS-compat divergence that flaked e2e-single
TestRunInstancesMultiCount.
Stamp TerminatedAt on transition and purge a completed record only once it is
older than a 15m visibility window; fresher records are left to the terminated
bucket's 1h TTL, keeping the instance describable as terminated. The ADR-0003
§3 purge-on-completion invariants still hold (their seeded records predate the
window). Closes mulga-siv-328.
* test(e2e): poll intra-SG east-west ping until L2 converges
TestSameSGComms fired a single un-retried 'ping -c 3 -W 2' immediately after
the source instance's SSH became reachable, but the ICMP target is the
destination's east-west data plane. Two instances boot in parallel and the
destination guest NIC / OVN port flows come up several seconds after the EC2
running transition the launch waits on. When the probe wins that race the
source kernel reports 'Destination Host Unreachable' (ARP unresolved), not an
ACL drop, and the test flaked.
Replace the one-shot ping with pingConverged, a bounded poll that retries
until 0% packet loss or 45s. A genuine self-ingress/datapath break still
stays at 100% loss for the full window and fails, so coverage is preserved.
<textx="24"y="30"font-family="'Segoe UI', system-ui, sans-serif"font-size="11"fill="#a1a1aa"letter-spacing="2.5"font-weight="600">ECS · EC2 LAUNCH TYPE ON SPINIFEX</text>
<textx="876"y="30"font-family="'SF Mono', ui-monospace, monospace"font-size="10"fill="#71717a"text-anchor="end">container instances from spinifex-ecs-node AMI · awsvpc + ALB</text>
<textx="450"y="410"font-family="'Segoe UI', sans-serif"font-size="9.5"fill="#71717a"text-anchor="middle">No ECS launch API — you add capacity by booting EC2 instances from the spinifex-ecs-node AMI (the console's "Provision capacity" wraps RunInstances)</text>
<textx="450"y="426"font-family="'Segoe UI', sans-serif"font-size="9.5"fill="#71717a"text-anchor="middle">The service keeps the desired task count running and registers each task's ENI IP with the ALB target group</text>
<textx="450"y="206"font-family="'Segoe UI', sans-serif"font-size="9.5"fill="#7ee787"text-anchor="middle">managed node group · workers join the control plane</text>
<textx="450"y="404"font-family="'Segoe UI', sans-serif"font-size="9.5"fill="#71717a"text-anchor="middle">You provide the VPC, subnets, and IAM roles · Spinifex provisions the control plane, node-group VMs, and security groups</text>
<textx="450"y="420"font-family="'Segoe UI', sans-serif"font-size="9.5"fill="#71717a"text-anchor="middle">Node groups always boot the registered eks-node image · workers need egress (IGW / NAT) to pull images</text>
# Manual-work mutex gate — see setup note 4 above. Checked BEFORE any
# SSH to wattle so a paused run touches the box zero times.
- name:Manual-work mutex gate
id:gate
env:
REPO_VAR_PAUSED:${{ vars.CI_GPU_PAUSED }}
run:|
set -euo pipefail
PAUSED=false
if [ -f "$HOME/.ci-gpu-paused" ]; then
PAUSED=true
echo "::warning::ci-gpu runner paused via ~/.ci-gpu-paused sentinel on the runner — skipping this run (manual GPU work in progress on wattle). Remove the file to resume CI."
fi
if [ "$(echo "${REPO_VAR_PAUSED:-}" | tr '[:upper:]' '[:lower:]')" = "true" ]; then
PAUSED=true
echo "::warning::ci-gpu paused via repo/environment variable CI_GPU_PAUSED=true — skipping this run."
fi
echo "paused=${PAUSED}" >> "$GITHUB_OUTPUT"
- name:Paused — nothing to do
if:steps.gate.outputs.paused == 'true'
run:echo "ci-gpu is paused; job exits cleanly without touching wattle. See gate step above for the reason."
echo "::error::GPU e2e suite failed (rc=${{ steps.e2e.outputs.rc }}) — see analysis above; raw ES/APM traces at ${SPINIFEX_OTLP_ENDPOINT} (ci.run_id=${{ github.run_id }})"
@@ -192,6 +192,10 @@ While Spinifex is architected and implemented by experienced engineers, we lever
This hybrid approach ensures Spinifex benefits from both proven engineering expertise and modern development acceleration, while maintaining the quality and reliability standards required for production infrastructure.
## Trademarks
"AWS", "Amazon Web Services", and all related service names (EC2, EBS, S3, VPC, IAM, and others) are trademarks of Amazon.com, Inc. or its affiliates. Mulga Defense Corporation and Spinifex are independent and not affiliated with, endorsed by, or sponsored by Amazon.com, Inc. or Amazon Web Services, Inc. References to AWS services describe interoperability and compatibility only.
## License
Spinifex is open source under the [GNU Affero General Public License v3.0](LICENSE). You're free to use, modify, and deploy it anywhere you need reliable infrastructure without depending on centralized cloud platforms.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Julian Sommer
