Expand the publish workflow to build the two Ubuntu NVIDIA GPU variants
alongside the Alpine eks-node/ecs-agent AMIs. The image build+publish
scripts are already manifest-driven and Ubuntu-aware; only the workflow
lacked the GPU legs.
Refactor the single-job body into a matrix over the four image manifests,
resolved by a new 'variant' input (all|cpu|gpu) so the slow 16G GPU legs
can be skipped when only the CPU AMIs need republishing. GPU images do not
fit a RAM-backed /tmp, so set SYSTEM_IMAGE_BUILD_DIR onto the runner's
on-disk _temp; legs run serially (max-parallel 1) on the single ci runner.
The nodegroup label was only written to K3S_NODE_LABEL in agent.env, but
k3s has no such env var, so non-GPU workers never received the label.
waitWorkersReady buckets Ready nodes by that label (added in #511), so
the affected nodegroups never counted toward their ACTIVE transition and
stayed CREATING until timeout.
Ride the label on a real k3s node-label config drop-in
(15-nodegroup-label.yaml), delivered via cloud-init user-data. Drop the
inert K3S_NODE_LABEL from agent.env and the initd export.
WaitForInstanceState now aborts immediately with the recorded StateReason
message when it observes a reaped state (shutting-down/terminated) while
waiting for a live target, instead of burning the full 5-minute timeout on
an instance that can never reach it.
MarkFailed takes a context and records the failure on the active span via
span.RecordError so a reaped launch surfaces in the APM error stream,
correlated by trace id. Only the LaunchRunInstances path carries a live
span; other callers pass context.Background() and the record is a no-op.
Surfaced by RCA of E2E run 29258398516 (TestAccountScoping/Step2).
TestNATGateway flaked with 100% egress loss: the tenant VPC gateway router's
OVN MAC_Binding had no entry for its upstream default-route nexthop. OVN
resolves that nexthop via lazy dynamic ARP on first egress; if that ARP is
lost during gateway bring-up, SNAT'd packets die in lr_in_arp_resolve and no
reply returns. MAC_Binding is per (datapath, logical_port, ip), so a sibling
VPC router resolving the same nexthop does not help.
Deterministically seed a static MAC binding for the nexthop at IGW attach. The
nexthop MAC is the host's own default-gateway MAC (the host shares the physical
uplink), read from ip neigh and written via ovn-nbctl static-mac-binding-add
(northd programs it into SB MAC_Binding, GC-safe). Host layer L0 owns the
shell-out per ADR-0006 S2; igw calls it via an injected hook. Best-effort and
idempotent: a missing MAC leaves dynamic ARP as the fallback, and it re-runs on
every AttachIGW so a binding lost to a later flap self-heals.
utils.GetOrCreateKVBucket hardcoded Replicas=1, so the ~38 handlers that
create their KV buckets lazily (VPC/subnet/ENI/EIP/ELBv2/ECS/ECR/ACM/
natgw/snapshot/placementgroup/IPAM/account-settings/dhcp/static-pool …)
created them at R1 on a multi-node cluster. upgradeJetStreamReplicas runs
once at startup, so any bucket created lazily after boot stayed R1
permanently — and an R1 stream's single Raft replica is transiently
leaderless during formation/join/rebalance, surfacing as
ErrNoResponders/ErrTimeout on KV ops.
Add a package-level default replica count that the daemon sets once at
boot (startCluster, after initJetStream) to len(clusterConfig.Nodes),
before any handler creates a bucket. GetOrCreateKVBucket now creates at
that default, so lazily-created buckets are born quorate. The one-shot
startup UpdateReplicas remains as a safety net for pre-existing R1
buckets from older deploys.
Zero call-site churn: the default flows through the existing
GetOrCreateKVBucketWithReplicas path already used by eks/ecs/iam.
TestSpawnRegisteredReconcilers_ResumesNonTerminal flaked two ways. First,
the shared fixture meta stamps a weeks-old CreatedAt, so a resumed CREATING
reconciler instantly trips the CREATE timeout, marks the cluster FAILED and
deregisters before the registry.Has assertion runs — the test was racing a
self-removal. Stamp a recent CreatedAt so the reconciler stays CREATING.
Second, SpawnRegisteredReconcilers ran a one-shot JetStream enumeration
(KeyValueStoreNames + kv.Keys), which is eventually consistent: under load a
just-committed cluster can be missing and never resumed, silently on a real
daemon. Re-scan until the observed CREATING/ACTIVE count stops growing, capped
and short-backed-off; Spawn is idempotent so re-runs only pick up stragglers.
discoverSingleNodeIPs seeds 127.0.0.1 then appends net.InterfaceAddrs()
in undefined order, so WANHost=nodeIPs[0] was loopback and the guest
gateway URL helpers fell through to the first non-loopback interface
addr — an arbitrary co-tenant bridge on multi-bridge shared hosts
(wattle: br-lan-env1..23, br-wan), leaving guest VMs unable to reach the
advertised gateway to register.
Resolve WANHost from spinifex.toml nodes.<node>.advertise (falling back
to that node's host), mirroring awsgwBindIP's toml scan; fall back to the
first non-loopback discovered addr only when config is unreadable. This
removes the need for the SPINIFEX_WAN_IP override on shared hosts.
The crash-supervisor auto-restart path (qmpHeartbeat -> HandleCrash ->
MaybeRestart -> startQEMU -> newQMPClientWithHandshake -> dialQMPWithRetry)
already unlinks the stale QMP socket and retries a refused/absent connect. But
a freshly relaunched QEMU can accept the connect then reset the connection
mid-greeting while it initialises, surfacing 'connection reset by peer'
immediately after 'QEMU started successfully' as a ServerInternal.
Add ECONNRESET and EPIPE to isTransientDialError so the dial is retried until
QEMU services the monitor, closing the auto-restart QMP-connect race.
Add non-streaming InvokeModel to bedrock-runtime. Unlike Converse's unified
schema, InvokeModel passes model-provider-native bodies through, so route by
model family: Anthropic Claude bodies forward to the Messages API (drop the
Bedrock-only anthropic_version, inject model, return verbatim); Meta Llama
bodies translate to/from the self-host vLLM OpenAI completions endpoint.
Raw-body dispatch bypasses the JSON response path (WriteRawResponse). Reuses
the Phase 1 credential + endpoint resolvers and upstream-status mapping.
Unit-tested (routing, body rewrite, translation, error classes) plus gated
InvokeModel E2E.
Both services already run otelsetup.Init and ship logs/metrics but never
opened a span, so they were invisible in the Kibana APM service map.
qmp-collector now opens a root span per per-instance poll tick (marks
error on QMP collect/publish failure); spinifex-ui now wraps its HTTP
handler chain with otelsetup.HTTPMiddleware and its reverse-proxy
transport with otelhttp, so UI requests get a server span and proxied
calls to awsgw/predastore carry traceparent instead of stopping the
trace at the UI hop.
nats and eks are intentionally left uninstrumented in this pass: nats is
the vendored message-broker process with no message-level hook to attach
spans to (the traffic it carries is already traced end-to-end at its
producer and consumer endpoints); eks request/response calls already
produce spans today under service.name=spinifex-daemon since the
handlers run in that process, and its async reconciler/state-machine
goroutines need a separate span design across a much larger surface.
Skip ERROR log when a subscription is already invalid
(nats.ErrBadSubscription) during graceful shutdown; log at debug
instead. Cosmetic cleanup race, no functional impact.
golangci-lint canonicalheader flagged 6 lowercase header literals in the
Ochre/Bedrock gateway (anthropic.go, anthropic_test.go, vllm.go). Canonical
form is wire-equivalent; canonicalise to X-Api-Key, Anthropic-Version and
Content-Type to restore CI lint.
* feat(bedrock): Ochre runtime MVP — Converse over self-host + Anthropic-direct
Add the AWS-Bedrock-compatible gateway services (bedrock, bedrock-runtime)
following the EKS REST-JSON pattern: ListFoundationModels / GetFoundationModel
over a static connectivity-tiered catalog, and non-streaming Converse routed
per modelId to one of two provider backends — self-hosted vLLM (OpenAI wire)
and Anthropic-direct (Anthropic Messages wire).
Per-account provider API keys are a master-key-encrypted JetStream KV secret
with an optional platform default; the advertised catalog gates external
models on a resolvable credential. Self-host endpoints are pinned via config
(OCHRE_VLLM_ENDPOINTS) behind an EndpointResolver seam.
Adds gateway/bedrock/ (catalog, provider router, Anthropic + vLLM adapters,
credential store), awserrors REST-JSON codes, and the bedrock E2E suite
(list/get light; Converse gated on live backends). Daemon/NATS control plane
and dynamic endpoint lifecycle are deferred to Phase 3.
* test(bedrock): cover dispatch, router, credentials KV + endpoint config
Add unit coverage for the Ochre Phase 1 diff: Bedrock_Request /
BedrockRuntime_Request end-to-end dispatch, provider router, JSON
response/error writers, upstream-status mapping, the credential-store KV
round-trip, the Anthropic bound-provider path, and parseBedrockEndpoints.
Lifts PR diff coverage above the 60% gate.
* ci: GPU e2e workflow on ci-gpu self-hosted runner
Runs tests/e2e/gpu (EC2/ECS/EKS GPU passthrough) against the persistent,
hand-bootstrapped wattle shared host, driven over SSH by a banksia runner
labelled ci-gpu (matches the ci-single/ci-multi convention) — no runner
installed on the shared box.
- runs-on [self-hosted, ci-gpu]; concurrency ci-gpu, cancel-in-progress
false (single A1000 = single-flight, queue not stomp)
- workflow_dispatch + GPU-path filter (not every push)
- manual-work mutex gate (~/.ci-gpu-paused / CI_GPU_PAUSED) so hands-on GPU
testing blocks CI
- if: always() cleanup terminates launched GPU instances; never resets the
box (no setup-ovn.sh/dev-reset/cluster-reset)
- drives the control plane with spx + the already-configured aws spinifex
profile directly (no wrapper scripts, nothing to install on wattle)
* ci(gpu-e2e): advisory AMI preflight, fail only if zero GPU AMIs
The suite is self-guarding — each test skips when its AMI is absent. The
hard-fail on the base ubuntu GPU AMI blocked the whole run even when the
ECS GPU AMI (the only one imported) could execute. Inventory the AMIs and
fail only when none are present.
* ci(gpu-e2e): self-provision base GPU AMI, hard-require it in preflight
The GPU E2E workflow was green-because-skipped: with no GPU AMIs imported
the self-guarding suite skipped every test, passing as a false green. Make
the raw-EC2 GPU passthrough test the always-on baseline.
- Add an import-if-absent setup step that pulls the prebuilt base GPU image
from the catalog by name (ubuntu-26.04-nvidia-gpu-x86_64), naming the AMI
exactly what tests/e2e/gpu filters for. Idempotent; caches on wattle's
persistent loop-fs. Runs spx under sg spinifex (local storage write).
- Preflight now hard-requires the base AMI (fail, not skip) so a silent
import failure can't pass as green; ECS/EKS node AMIs stay opt-in warn.
- Reconcile the local build path: ubuntu-gpu-nvidia.conf AMI_NAME and
launch-gpu.sh's nvidia default now use the same canonical name, so a
locally built+imported image also satisfies the suite.
* ci(gpu-e2e): pass --ami-name on base GPU import so the AMI name matches
spx admin images import --name selects the catalog entry but names the
resulting AMI generically (ami-<distro>-<version>-<arch>), so the import
landed as ami-ubuntu-26.04-x86_64 and the tests/e2e/gpu name filter never
matched. Pass --ami-name ubuntu-26.04-nvidia-gpu-x86_64 explicitly (same
flag build-system-image.sh uses) and poll briefly for availability.
* fix(network,vm): reconcile auto-assigned public-IP NAT + scale VFIO QMP timeout
Auto-assigned public IPs were never carried through the network reconciler.
applyEIPs ran AddEIP + ensureGuestPortDatapath for user EIPs only; auto-assigned
IPs (intent.Ports[].PublicIP) got egress exemption but no NAT re-assert and no
gatewayLRP->guest convergence. Their dnat_and_snat was created once at
RunInstances and never touched, so a vpcd/OVN rebuild dropped it and the
guest-port hop never converged: inbound DNATs at the gateway but never reaches
the guest (TCP/SSH dark) while ICMP is answered by the gateway LR (ping OK).
floatingIPSpecs now folds every auto-assigned/ELB public IP into the same
AddEIP + guest-port convergence path as user EIPs (deduped by private IP; a user
EIP wins), making the datapath self-healing across reconcile ticks and deploys.
qmpGreetingTimeout for VFIO guests now scales with guest RAM (base 30s + 8s/GiB,
floored at 180s, capped at 600s) instead of a flat 180s, so a large guest is not
SIGKILLed mid-pin while the whole guest RAM is IOMMU-locked before QMP answers.
* fix(vm): scale QMP handshake command deadline for VFIO launches
The greeting timeout scaling alone did not fix the VFIO launch SIGKILL: a GPU
guest sends its QMP greeting before the RAM pin completes, then cannot service
qmp_capabilities until the pin finishes. sendQMPCommand hardcoded a 30s response
deadline, so the handshake decode timed out (~43s for a 16 GiB g5.xlarge) and
the launch cleanup SIGKILLed a QEMU that would have come up.
Parameterize the command deadline (sendQMPCommandWithTimeout); the handshake now
uses the RAM-scaled qmpGreetingTimeout (base 30s + 8s/GiB, floor 180s, cap 600s)
for qmp_capabilities. Other callers keep the 30s default.
* test(gpu-e2e): share one GPU instance across passthrough subtests to avoid mid-suite IP recycle
Recycling both the floating and private IP onto a new ENI left the ingress
datapath targeting the terminated instance, so the successor was unreachable
on its EIP while IMDS stayed up.
- AddEIP re-points the dnat_and_snat row when the owning ENI changed even
though the external+logical IP are unchanged, keyed on the portable
spinifex:logical_port external-id (centralised mode leaves the native
LogicalPort column empty). The genuine same-ENI case still skips row churn.
- Flush the SB MAC_Binding for a private IP on ENI teardown and successor
bind so a reused address re-resolves to the live owner's MAC, via a new
MACBindingFlusher backed by ovn-sbctl.
- Reconciler prunes orphan dnat_and_snat rows whose stamped owning ENI is
absent from intent (leaked by lost vpc.delete-nat across dead VPCs).
- IMDS tap responder suspends retries for an ENI whose record is gone,
stopping the 15s reconcile churn.
* test(daemon): cover GPU inventory in GetNodes and GPU attachment in GetVMs
Adds unit tests for buildGPUInventory/handleNodeStatus (whole-GPU claimed/
free, MIG slices claimed/free, no-GPU node) and resolveVMGPU/handleNodeVMs
(whole-GPU attachment, MIG attachment, no attachment). Production code for
these paths (types.GPUInfo/GPUSliceInfo/VMGPUInfo, buildGPUInventory,
resolveVMGPU) and EC2 DescribeInstanceTypes GpuInfo.Gpus population
(instancetypes.GenerateGPUTypes/GenerateMIGTypes/IsGPUType) already landed
via the daemon GPU-admission work and were already test-covered for part C.
* feat(ui): shared GpuInstanceTypeSelect grouped instance-type picker
Reusable Select wrapper for EC2/EKS/ECS instance-type pickers: groups
Standard vs GPU-Accelerated types via structural GpuInfo.Gpus detection
(AWS SDK shape, no name heuristics), annotates GPU items with VRAM from
MemoryInfo.SizeInMiB and an optional per-type availability count, and
falls back to a single ungrouped list when no GPU types are present.
* feat(ui): EC2 GPU inventory, instance GPU card, shared picker in run-instances
* feat(ui): EKS nodegroup GPU picker + AMI/taint note
Add-nodegroup form now uses the shared GpuInstanceTypeSelect, grouping
and VRAM-annotating GPU instance types instead of a free-text field.
Selecting a GPU type surfaces a note that the GPU node AMI is
auto-selected and the nodegroup is tainted
nvidia.com/gpu=present:NoSchedule by default. Nodegroup rows badge GPU
nodegroups, derived from the primary instance type since
NodegroupRecord does not expose a GPU flag over the API.
* feat(ui): ECS GPU capacity picker, task-def GPU count, task gpuIds
* feat(eks): expose nodegroup GPU signal in Describe/ListNodegroups + UI badge
* chore(ui): rebuild embedded console dist for GPU UI
Bump github.com/mulgadc/viperblock to v1.11.1-0.20260713024608-eb4a376becc8
(dev tip through viperblock #34): bounded parallel chunk upload + coalesced
checkpoint (edou7 sec 7 items 2 and 9), sweeping the require up from #32 and
picking up the #33 write backpressure on the way. The nbdkit plugin the VM
write plane uses is built from the submodule source; this bump aligns the
viperblockd library that spx links.
* fix(daemon): gate GPU instance admission on cpu/mem and GPU-slot availability
canAllocateLocked early-returned count for whole-GPU (non-MIG) types,
bypassing the vCPU/memory admission calc and the live MemAvailable gate.
canAllocateCount already had a requiresGPU/availGPU branch, but every
production caller passed requiresGPU=false, making it dead code.
GPU instance types (whole-GPU and MIG) now flow through the same
cpu/mem + live-mem gate as every other type, with GPU-slot count wired
in as an additional hard constraint from gpuManager.Available(). This
also gives MIG a real up-front GPU-slice gate instead of only failing
late at Claim.
* fix(vm): release leaked GPU pool slot on crash-to-error and add reconciler backstop
HandleCrash freed vCPU/memory via deallocateResources but never called
InstanceCleaner.ReleaseGPU, since it bypasses the stop/terminate cleanup
chain that normally carries that call. A successful restart reuses the
still-set GPUAttachments, so that path is untouched, but when restarts
are exhausted, the instance type is unknown, there isn't enough capacity
to restart, or the relaunch itself fails, the instance settles
permanently in StateError still holding its GPU pool entry with nothing
to reclaim it.
Release the GPU and clear GPUAttachments on each of those terminal
paths. Add a GPU-pool reconciler mirroring the existing ENI reconciler,
wired into the daemon's reconcile loop, that periodically frees any pool
entry whose instance is absent or has settled into a non-running state,
as a backstop for a node-down mid-cascade or any other gap in the
synchronous release.
* fix(eks): keep derived ELBv2 names under 32 chars and surface CREATE_FAILED reason
Long cluster names made the derived "eks-<name>[-cp|-konn]" NLB/target-group
names exceed the ELBv2 32-char limit, failing cluster creation asynchronously
with no visible cause. safeELBv2Name now deterministically truncates the name
and appends a stable hash of the full cluster name when needed, so the same
name is derived on every call (create, lookup, delete). DescribeCluster also
now surfaces ClusterMeta.StatusReason as a Health.Issues entry when a cluster
is FAILED, instead of leaving callers to guess why.
* fix(eks): plumb per-nodegroup Ready-node counts through state reports
The on-VM state-report script now emits nodegroup_ready, a per-nodegroup Ready
count keyed by the eks.amazonaws.com/nodegroup label value, alongside the
existing cluster-wide node_count. ServerStateReport, the ClusterReconciler's
observe step, and SetClusterHealthState now carry this map through to
ClusterMeta.NodegroupNodeCounts, so per-nodegroup consumers (wired in the next
commit) can gate on a nodegroup's own workers instead of the cluster total.
* fix(eks): scope nodegroup Ready-gate per nodegroup and retry failed worker launches
waitWorkersReady compared the cluster-wide Ready-node count against a
nodegroup's baseline+desired, so in a multi-nodegroup cluster one nodegroup's
Ready workers could mask another whose own workers never registered. It now
reads meta.NodegroupNodeCounts[ng], scoped to nodes carrying the
eks.amazonaws.com/nodegroup=<name> label.
Separately, launchNodegroupInfra never retried a failed worker launch (a
transient QMP timeout or host capacity pressure), leaving the nodegroup stuck
CREATING with zero workers. The new launchWorkersUntilDesired wrapper relaunches
only the shortfall on failure, logging each retry, until the desired count is
reached or workerLaunchRetryTimeout elapses.
Both fixes are covered by regression tests: a two-nodegroup scenario where only
one has Ready workers, a transient-failure-then-success relaunch, and an
exhausted-retry-budget CREATE_FAILED path. Not covered here: a worker that dies
after the nodegroup is already ACTIVE — that needs an ongoing reconciler, which
this change does not add.
* fix(eks): tolerate GPU NoSchedule taint in konnectivity-agent DaemonSet
A GPU-only worker is tainted nvidia.com/gpu=present:NoSchedule, so the
konnectivity-agent DaemonSet (no tolerations) showed DESIRED=0 there and
kubectl logs/exec against pods on that node failed "No agent available".
Adds an Exists toleration for nvidia.com/gpu:NoSchedule.
Live-cluster validation (DESIRED count, kubectl exec) is deferred — no cluster
available to test against right now.
* fix: tolerate nvidia.com/gpu taint on ebs-csi-node and coredns DaemonSets so GPU workers get storage and DNS
A re-driven DeleteCluster whose managed control-plane VPC was already gone
(EC2 InvalidVpcID.NotFound) kept looping DependencyViolation: the internal
cp-vpc state record (ClusterMeta.ManagedCPVPC) was never cleared unless the
*entire* purge succeeded, so any unrelated flaky step retried CP-VPC/SG API
calls against a stale VpcId forever instead of converging.
purgeClusterInfra now clears ManagedCPVPC (via a new CAS-based
ClearClusterManagedCPVPC) as soon as DeleteClusterCPVPC + its SG cleanup both
succeed, independent of whatever else in that run still fails.
DeleteClusterCPVPC also now republishes vpc.delete for the CP VPC on every
call — including the already-gone fast path, falling back to the persisted
VpcId when the live tag-indexed describe can no longer name it — so vpcd's
topology manager gets another chance to GC an orphaned logical
router/subnets/SubnetEgressPriorityDrop policies if the original delete's
fire-and-forget event was ever lost.
handleRelease cancelled the per-lease renewal goroutine but deleted the
KV entry without waiting for it to exit. An in-flight reaffirm/renew Put
could land after the Delete, re-creating the lease and leaking a KV
record. Add a per-loop done channel and block on it after cancel so any
pending Put completes before the delete, making DrainAll's release path
deterministically synchronous.
* feat(eks): expose NVIDIA GPUs to pods via CDI device plugin
GPU nodegroups now label workers nvidia.com/gpu.present=true and taint them
nvidia.com/gpu=present:NoSchedule (via a k3s config.yaml.d drop-in, since k3s
has no K3S_NODE_TAINT env), and stage a hidden nvidia-device-plugin addon that
addon-sync renders into the K3s auto-deploy dir. The device plugin runs with
--device-list-strategy=cdi-cri and is gated to GPU nodes by nodeSelector +
toleration, advertising nvidia.com/gpu allocatable so pods land on GPU nodes
and get the device injected via the CDI spec baked by mulga-cdi-generate.
Relies on containerd 2.0.5 default enable_cdi=true (no config fork).
* fix(vm): scale QMP greeting deadline for VFIO passthrough guests
A GPU passthrough guest (-device vfio-pci + large -m) must lock and DMA-map
its entire guest RAM through the IOMMU before QEMU's QMP monitor answers,
which exceeds the fixed 30s greeting deadline and gets the VM SIGKILLed at
launch. Parameterize the greeting timeout (NewQMPClientWithGreetingTimeout)
and scale it to 180s for VMs with GPU attachments; plain VMs keep the 30s
default and reconnects against an already-running QEMU keep it too.
* fix(eks): register GPU workers reliably and expose GPUs to pods
Three fixes let a GPU nodegroup worker boot, register, and advertise its GPU:
- k3s-agent.service gains Delegate=yes (plus KillMode/TasksMax/limits and
overlay/br_netfilter modprobe): k3s embeds containerd+kubelet in-process, so
on cgroup-v2 the kubelet cannot create /kubepods without a delegated subtree
and ContainerManager.Start fails before Node registration.
- The GPU node label rides a k3s config drop-in (20-gpu.yaml node-label +
node-taint) instead of K3S_NODE_LABEL, which k3s ignores; without the label
the nvidia-device-plugin nodeSelector never matches.
- Raise the nodegroup ready timeout to 20m for the ~7min GPU first boot.
Adds the GPU e2e (VPC + NAT-GW egress harness) and a slog-restore unit test.
* fix(eks): install ecr-credential-provider on the GPU node AMI
The GPU worker's kubelet is started (from nodegroup userdata) with
--image-credential-provider-config + --image-credential-provider-bin-dir=
/usr/local/bin, but the eks-node-gpu AMI never built or installed the
ecr-credential-provider binary. kubelet treats a referenced-but-missing
provider binary as a fatal RegisterCredentialProviderPlugins error, so the
node crash-loops and never registers (while the Alpine eks-node worker, which
ships the binary, joins fine). Build + install it in the GPU manifest; the
server-only binaries stay omitted since GPU nodes are agent-only.
Also tee k3s-agent stderr (and its embedded kubelet klog) to the serial
console: Ubuntu logs only to journald, which is off-box-invisible on a worker
in an isolated customer VPC, so a kubelet that fails to register left no trace.
* fix(eks): give the nvidia-device-plugin its own GPU access
The CDI-mode device plugin (--device-list-strategy=cdi-cri) must run
nvml.Init and generate the CDI spec in its own container, but the DS had
no GPU wiring, so nvml.Init crash-looped and nvidia.com/gpu was never
advertised. Run the DS under the k3s-auto-configured nvidia runtimeClass
with NVIDIA_VISIBLE_DEVICES/CAPABILITIES=all, the host root mounted
read-only at /driver-root so it can locate the driver libs + ld.so.cache,
and /var/run/cdi mounted writable so the generated spec is visible to
containerd. No enable_cdi drop-in is needed on k3s v1.32.5 / containerd
2.0.5. GPU workload pods still need only resources.limits nvidia.com/gpu.
Live-validated end-to-end on the single-GPU host (RTX A1000): baked
control-plane AMI, GPU nodegroup, nvidia.com/gpu allocatable, nvidia-smi
pod, zero manual intervention.
* test(ec2): drop obsolete restoreSlogDefault test
Rebased onto dev, which dropped the restoreSlogDefault workaround in #506
(viperblock no longer mutates the global slog). The test referenced the
now-deleted helper and no longer compiles. Real GPU/registration diff is
independently covered (diff coverage 95.3%).
When no br-mgmt bridge is present (supported single-node config where system
instances reach AWSGW via the advertise IP), LaunchSystemInstance correctly
skips mgmt-tap creation but the NIC list still carried the mgmt slot, so
buildNICNetdevs emitted a QEMU netdev/device for a tap that was never created
and QEMU aborted (could not configure /dev/net/tun: Operation not permitted),
failing every CreateLoadBalancer/CreateCluster. Skip the mgmt NIC (index 1)
when its MAC is unset across netdevs, kernel-cmdline mmio slots, and the guest
fw_cfg netcfg blob; multi-node (br-mgmt present) is unchanged.
- 4ncdg: NATSRequest now round-trips the AWS error Message, not just the
sanitized Code. Server sends GenerateErrorPayloadWithMessage(code,
err.Error()); the client returns the Message when present so the CLI prints
the real reason (e.g. "snapshot not found; refusing to restore") instead of
a bare "ServerInternal". One code path fixes every NATS-driven command.
- fbmhl: the UI node view rendered raw host totals (vCPU:80, 0/125.5GB),
ignoring the daemon reserve. It now shows schedulable = total - reserved,
the capacity a tenant can actually consume. reserved_vcpu/reserved_mem_gb
added to the NodeInfo type (already present in the API payload).
Three related nat-mode fixes:
- ura01: nat-transit pool now emits gw_lrp_range_start/end. admin init
sets 100.127.0.16-.254 (239 VPC gateways) instead of the auto-derived
top-16 (~15). Adds GwLrpRange* to PoolData + the spinifex.toml template
so operators can widen any pool.
- 724mm: detectPoolMode returns true only for external_mode=pool. nat and
disabled clusters now skip Phase 8b/8d, which otherwise failed on
AllocateAddress UnsupportedOperation.
- pnz2i: document that system instances (ECS/EKS/LB agents) are
unsupported in nat v1 (require pool mode) in the vpc-networking README
and COMMANDS.md; document the gw_lrp_range_* fields.
* fix(eks): write etcd-snapshot.env at CP launch so snapshots actually run
* test(eks): cover resolveSystemPredastoreURL and buildEKSServiceDeps
Diff-coverage gate for 852f61d9 was unmet (48.8% < 60%) because the
mgmt-bridge branches and EKSServiceDeps assembly had no direct test.
* feat(eks): restore-snapshot DR path (fresh CP + cluster-reset restore + NLB re-point)
* fix(eks): make restore-snapshot fail-safe (validate snapshot, unwind on partial failure, honest status)
* fix(eks): sign etcd-snapshot uploads with deployment region (was 403ing)
* fix(eks): apply recovery directive on DR-seed first boot + single-node replacement host
* fix(daemon): log NATS handler failures at Error (were only on the trace span)
* feat(gpu): Ubuntu NVIDIA GPU node image + GPU AMI selection
Epic A of GPU passthrough to EKS pods / ECS task containers.
- Add gpu-vendor AMI tag constants + GPUVendorForType/IsGPUTypeName helpers
(instance-family prefix -> vendor from knownGPUModels).
- EKS + ECS worker AMI resolution branches to a GPU-tagged node image for GPU
instance types; no fallback to the driverless image (hard error instead).
- Exclude gpu-vendor-tagged AMIs from the plain (non-GPU) lookup so a newer GPU
AMI cannot hijack ordinary nodes (managed-by tag is shared; filters have no
negation, so exclude client-side).
- NodegroupRecord caches GPUEnabled/GPUVendor from InstanceTypes.
- Catalog entries spinifex-eks-node-gpu / spinifex-ecs-node-gpu (dual-tagged).
- New Ubuntu 26.04 GPU node images (eks-node-gpu, ecs-agent-gpu): reuse the
proven DKMS -server driver recipe (drop Docker CE + dev payload), add
nvidia-container-toolkit, OpenRC->systemd port of the worker/agent service
subset, udev EBS by-id bridge, nvidia modules-load, first-boot CDI generation.
- build-system-image.sh: SYSTEM_TAG supports multiple space-separated tags.
* build: allow SYSTEM_IMAGE_BUILD_DIR override for image build dir
Ubuntu GPU images are 16G; a RAM-backed /tmp tmpfs is too small to hold the
qcow2 + expanded raw. Default stays /tmp; SYSTEM_IMAGE_BUILD_DIR redirects the
build onto a real disk.
An ELBv2 LB whose system VM launches but whose in-guest lb-agent never
delivers a first heartbeat previously stayed in provisioning forever with
an empty StateReason: promotion to active happens only on the first
heartbeat and the only provisioning->failed transitions were at
create/launch time. A single wedged LB in the multi-node e2e lb suite
leaked its listener/target-group and cascaded into ResourceInUse and
DependencyViolation on VPC teardown.
- Add a lifecycle reaper that marks failed any LB still provisioning past
lbProvisioningTimeout (4m) with no heartbeat, reusing markLBFailed, and
populate a provisioning StateReason so a stuck LB is self-describing.
Started once at daemon startup; stopped on service Close.
- Harness: add ErrLBProvisioningTimeout and treat a provisioning-timeout
as retryable in createActiveLB; register idempotent LB/listener teardown
up-front so no exit path leaks into the shared VPC.
- IMDS: bounded retry+backoff on the instance-fact and user-data lookups so
a transient gather timeout under multi-node load does not starve the
lb-agent bootstrap.
The coordinated host-shutdown DRAIN phase transitioned every running VM to
StateStopped and migrated it to the operator-stopped shared KV bucket, evicting
it from node-local running state. On the next boot Restore only loads local
running state and treated every StateStopped instance as user-stopped, so
previously-running VMs never relaunched after a host reboot.
Gate on the existing operator-intent signal Attributes.StopInstance (set only by
StopInstances, cleared by StartInstance): stopOne keeps drain-stopped VMs in
local running state instead of migrating them, and Restore relaunches
drain-stopped VMs while operator-stopped ones stay stopped. markUnschedulable
now stamps StopInstance so capacity-demoted instances are not relaunched.
Close four pre-existing lifecycle races surfaced during env19 hardening.
Volume seal/reaper (42hbm, c7ce2): the ebs.unmount handler now seals the
volume before removing it from MountedVolumes, so a retry after a
timed-out-but-completed seal reliably reports NotFound. EBSUnMountResponse
carries a NotFound flag and unmountResponseError treats it as seal success,
so state reconciles to available/detached instead of leaking. The volume
leak reaper reconciles the attachment record (never the data) for volumes
whose owning instance is terminated, so GC can delete them.
Stopped-instance CAS (ozlbq, wlmea): add UpdateStoppedInstance, a
mutate-callback backed by casUpdate(createIfAbsent=false), and migrate
TagStoppedInstance and ModifyInstanceAttribute off the resurrect-prone
Load->mutate->WriteStoppedInstance path. A tag/attribute write racing a
winning start-claim now fails cleanly instead of re-creating the deleted
entry. StartStoppedInstance returns IncorrectInstanceState (not
InvalidInstanceID.NotFound) when the stopped entry is absent but the
instance is present locally.
Every tofu-based e2e workflow set TOFU_LOG to a /tmp path outside
ARTIFACT_DIR. ci-fmt.sh's quiet_run helper appends all tofu init/apply/
destroy output to that file, but the Teardown step (which runs the
destroy) always ran AFTER the existing upload-artifact step, so the
full log — including destroy-time failures — was silently discarded
on every run.
Add an if: always() upload-artifact step after Teardown in each
tofu-based job (e2e.yml, e2e-suites.yml, e2e-nightly.yml, e2e-ddil.yml)
that uploads $TOFU_LOG directly. Since quiet_run appends both apply and
destroy output to the same file, one upload after Teardown now
captures everything, recoverable even when destroy itself fails.
Shipping this log to Elasticsearch/OTLP is deferred to mulga-5n7fe,
which will consume this artifact.
Init now installs an sdk/log LoggerProvider inside the existing
exportEnabled() gate, sharing the same resource as the tracer/meter (incl.
ci.run_id via OTEL_RESOURCE_ATTRIBUTES) and joining lp.Shutdown into the
returned shutdown func. The process default slog handler is fanned out to
both the existing stdout/journald handler and an otelslog bridge, so app
logs export over OTLP with trace correlation while journald output is
unchanged. With no OTLP endpoint configured Init remains a no-op: no
LoggerProvider is installed and the default logger is left untouched.
PR #487 turned attach/detach flaky: reconnectQMP dialed a fresh QMP
client before closing the old connection. QEMU's monitor is a
single-client `server,nowait` listener, so while the old conn holds the
slot the redial gets no greeting, blocks ~30s, and fails, leaving
q.Dead permanently true and every later attach/detach ServerInternal.
Fix the ordering (close, then dial via dialQMPWithRetry to absorb the
brief post-close ECONNREFUSED window).
waitForDeviceDeletedEvent marked q.Dead on any decode error, including
a benign read-deadline timeout — routine, since DEVICE_DELETED is
frequently pre-swallowed by device_del's own reply loop. The
underlying connection survives a timeout, but encoding/json.Decoder
permanently caches read errors on itself, so on a timeout the Decoder
is swapped for a fresh one over the same conn instead of forcing a
full QMP reconnect; q.Dead is now only set on a genuine decode error.
Also: an AttachVolume re-attach to the same instance with a different
device now returns VolumeInUse (matching AWS) instead of
InvalidParameterValue.
* feat(ec2,ebs): harden volume attach/detach + instance lifecycle state machine
Batch of related EBS/instance-lifecycle correctness fixes verified together.
- u94vs: add compare-and-swap (casUpdate/casPut via nats KV Update revision)
to the multi-writer lifecycle KV sites (WriteTerminatedInstance,
WriteStoppedInstance, WriteClusterShutdown); make the teardown-reaper
read-modify-write atomic via a merge callback so retries never re-run
cleaner side effects.
- yi2ed: add ClaimStoppedInstance, a revision-checked delete-as-claim on the
stopped bucket performed before Allocate/Insert/Run, so a forwarded start
and a local fallback cannot double-start one instance onto the same volume.
- siv-473: make AttachVolume idempotent — return attached success when the
volume is already attached to the requesting instance; a conflicting
requested device returns InvalidParameterValue.
- siv-462: reconnect now re-asserts in-use for all attached volumes (boot and
non-boot data volumes), not just boot, so a daemon restart no longer leaves
stateful-pod data volumes split-brain; EFI pflash is skipped.
- 9w2mv: wait for the QMP DEVICE_DELETED event (bounded) before blockdev-del
instead of a blind sleep, so a slow guest unplug no longer exhausts retries.
- soyah: poll (50ms) instead of fixed 1s sleeps in KillProcess and the
viperblockd nbdkit confirm-running path, cutting volume-lifecycle latency.
- siv-481: a cross-node start forward that times out now falls back to a local
start; this is safe because the yi2ed claim is the single serialization point.
* fix(eks): create per-account and leader KV buckets with cluster replica count
The EKS per-account KV bucket and the leader lease bucket were created at
Replicas=1 and never upgraded, since they're created lazily after
upgradeJetStreamReplicas runs at startup. On multi-node clusters this leaves
their single-replica Raft group transiently leaderless during formation,
causing KV timeouts that surface as a deterministic 503 on ListClusters.
Thread clusterSize through GetOrCreateAccountBucket/InitLeaderBucket (backed
by a new utils.GetOrCreateKVBucketWithReplicas) so both buckets are created
at max(clusterSize,1) replicas, matching every other node-count-scaled
JetStream stream.
A fresh cluster or SB re-bootstrap could leave a node's ovn-controller with
its OVSDB client latched on stale Southbound RAFT cluster state, looping on
"clustered database server has stale data" and silently no longer realising
new Port_Bindings. Cross-host CP data-NICs then never bind: no DHCP, no IMDS,
degraded boot. A manual `systemctl restart ovn-controller` was the only
recovery. Root cause is deterministic: setup-ovn.sh restarts ovn-controller
per node as each RAFT member forms, so a controller started for an early
member races the SB membership changes that later joins trigger.
Add three layers of recovery (the ansible bring-up settle pass that fixes this
at the source lives in the mulga repo):
- Per-host wedge watchdog (vpcd/ovn_controller_watchdog.go): not leader-gated
since the wedge is per chassis. Polls SB connection-status; after a stale
threshold issues sb-cluster-state-reset, throttled by a cooldown. Decision
logic takes an injected clock for deterministic tests.
- Reconcile-loop escalation: ensureGatewayClaimed and ensureGuestPortDatapath
escalate to one sb-cluster-state-reset after repeated recompute misses only
when the SB is not connected — a recompute cannot clear a stale-SB wedge.
- Prober primitives SBConnectionState/ResetSBClusterState on GatewayClaimProber
plus the GatewayClaimVerifier interface extension.
Also fix the awsgw OVN health probe, which passed a slashed `-t` socket path
that never resolves and so always reported ovn-controller not_running. Probe
connection-status instead: OVNControllerUp is now true only when connected, so
healthy nodes read ok and a real wedge correctly reads not_running.
The EIP "verify unreachable after disassociate" step polls runSSHQuiet
inside EventuallyErr(90s), but a wedged ssh made os/exec Wait block for
14 minutes — EventuallyErr cannot interrupt an in-flight CombinedOutput,
so one probe consumed the whole 30-min test-binary budget and panicked.
The panic skipped t.Cleanup, leaking the EIP and NAT-gateway VMs, which
held single-node capacity and starved the lb suite's sys.micro launch
(the daemon unsubscribes the full topic, so NATS returns no-responders).
Bound the ssh child at the process layer with exec.CommandContext (30s
deadline) plus WaitDelay, returning a descriptive timeout error. This is
version- and phase-independent, unlike ssh's own ConnectTimeout. 30s is
safe for all callers (worst legit call is ~15s) and well under the 90s
poll budget, so a wedged probe now yields a clean bounded failure with
cleanup intact.
* feat(network): add routed uplink mode, Routed wiring, and NAT egress rule helper
* feat(external): route gateway LRP over transit pool and install VPC SNAT in routed NAT mode
* feat(vpcd): support bridge_mode=nat (routed uplink, transit pool, egress rules)
* feat(daemon): register EIP handlers unconditionally with disabled-mode stub; seed default subnet per external mode
* feat(admin): support --external-mode=nat and suggest it for non-bridgeable uplinks
* feat(scripts): add --nat-uplink routed mode to setup-ovn.sh
* test(e2e): add routed-NAT single-node natuplink suite
Covers the external_mode=nat lane end to end on a nat-initialized node:
host transit wiring (veth, br-ext port, ip_forward, iptables egress
rules), spinifex.toml contents, disabled EIP surface (empty
describe-addresses, UnsupportedOperation on allocate), default subnet
MapPublicIpOnLaunch=false, OVN gateway LRP + VPC SNAT on the transit
/24, guest outbound egress proven via a serial-console probe (nat mode
has no inbound path), and unique transit gateway IPs across VPCs.
Suite self-gates on external_mode=nat; registered as e2e-natuplink and
in SUITES_BUILD so it compiles in CI. Manifest entry stays commented
until CI can provision a nat-mode environment.
* feat(ovn): Address_Set CRUD and NAT exempted_ext_ips column
Adds the exempted_ext_ips column (optional strong Address_Set ref) to the
nbdb NAT model, Address_Set ensure/get with wait-op create semantics, and an
in-place SetNATExemptedExtIPs update on the client interface, live client,
and mock. Test NB schema gains the column and covers insert-resolves-UUID,
converge, clear, and missing-NAT paths.
* feat(policy): exempt address set on routed SNAT/EIP rows
Adds WithSNATExemptSet so routed mode stamps exempted_ext_ips (the
spinifex_nat_exempt Address_Set: transit /24 plus operator extras) on IGW
snat and EIP dnat_and_snat rows, keeping VM replies to host-initiated flows
un-NATted. AddSNAT gains an idempotency guard keyed on (router, VPC CIDR)
with stale-IP replacement, and both AddSNAT and the AddEIP idempotent-skip
path patch the ref in place on pre-existing rows. The set is a singleton
strong ref and is never deleted. Non-routed modes are unchanged.
* feat(host): VPC ingress route primitives for routed-NAT mode
EnsureVPCIngressRoute / RemoveVPCIngressRoute manage the host route that
sends VPC-CIDR traffic into OVN via the gateway LRP transit IP on the
spx-nat veth. Replace/del semantics keep both idempotent; a missing route
on removal is not an error.
* feat(external): routed ingress leg on IGW attach/detach, reconcile-safe
AttachIGW's routed leg now also installs the host VPC ingress route via
RoutedIngressHooks, and re-runs on the already-attached path so the
reconcile loop restores host routes (and patches legacy SNAT exempt refs)
after a reboot — OVN state survives restarts but host routes do not.
DetachIGW removes the route alongside the SNAT. vpcd wires the hooks and
the spinifex_nat_exempt set (transit /24 + [network] nat_exempt_cidrs,
validated nat-mode-only) exclusively in routed-NAT mode; pool/veth/direct
pass nil hooks and are unchanged.
* test(natuplink): Tier 1 host-ingress phase; docs for jumpbox pattern
New Phase 7 asserts the automatic ingress tier: host route to the VPC CIDR
via the gateway LRP transit IP, exempted_ext_ips ref on the SNAT row
pointing at spinifex_nat_exempt (containing the transit /24), and — after
opening the SG from the transit net (default-closed, AWS parity) — ping and
an sshd TCP handshake against the instance private IP from the host, which
proves the un-SNATted return path. Phase 8 additionally waits for the
second VPC's ingress route to appear on IGW attach. vpc-networking README
documents the jumpbox pattern, SG note, and nat_exempt_cidrs.
* feat(config,admin): public wan pool alongside nat-transit
Restructure admin config generation from single-pool scalars to a Pools
slice so nat mode can emit both the transit pool and an optional public
wan pool. nat mode now accepts --external-pool/--external-source (plus
--external-bind-bridge/--gateway-ip when a public pool is requested),
validated by the shared resolvePublicPoolFlags helper also used by pool
mode. Transit pool always renders first so the IGW allocator picks it.
Formation wire format unchanged (nat mode is single-node only).
* feat(daemon): enable EIP service and auto-assign in nat mode with public pool
External IPAM (and with it the real EIP service, RunInstances public IP
auto-assign, and MapPublicIpOnLaunch on default subnets) now activates
whenever public pools exist: pool mode as before, plus nat mode when a
wan pool rides alongside the transit segment. The transit pool is
filtered out of IPAM — its addresses are gateway-LRP plumbing, never
allocatable EIPs. Transit-only nat configs keep the disabled EIP stub.
* feat(host): per-EIP ingress primitives for routed-NAT mode
DetectUplinkFor parses ip route get to find the interface and source IP
facing the pool gateway (re-run per ensure: WiFi/cellular uplinks move).
EnsureEIPIngress steers an EIP into OVN with a /32 route via the VPC
gateway LRP transit IP (src-hinted so host-local traffic still DNATs),
answers ARP for the EIP with a proxy neighbor entry (L3 only, same MAC,
WiFi-safe), sends gratuitous ARP, and adds per-EIP FORWARD accepts for
DROP-policy hosts. RemoveEIPIngress mirrors; missing state tolerated.
* feat(policy,vpcd): host EIP binder option on NATManager
WithHostEIPBinder injects routed-mode host plumbing hooks. AddEIP binds
after priming on both the fresh-add and idempotent-skip paths (host
routes and proxy-ARP are volatile even when the OVN row survives, so
reconcile re-ensures them after reboot); DeleteEIP unbinds best-effort.
The next hop comes from the new gatewayPortIP helper reading the gateway
LRP's stamped spinifex:gateway_ip (Networks fallback), with the ext-ID
key now an exported policy const shared with the external package.
vpcd selects the IGW pool by name instead of index — with two pools in
nat mode the gateway allocator must stay on the transit pool — and wires
the binder over host.EnsureEIPIngress/RemoveEIPIngress when a public
pool is present. EnsureEIPIngress gains an uplink hint so dhcp-sourced
pools (gateway unknown until lease) proxy-ARP on their bind bridge.
* feat(dhcp): interface-MAC lease variant with MAC-keyed-router detection
dhcp_mac = "derived" (default) | "interface" on external pools. The
interface variant leases with the bind interface's own MAC — WiFi/WWAN
uplinks drop frames with foreign source MACs, so derived chaddr MACs
never complete DORA there. Option 61 stays the lease discriminator.
UseIfaceMAC threads AllocateRequest → NATS wire → nclient4 → lease and
persists through KV so restart re-acquires repeat it.
Routers that key leases on MAC and ignore client-id would hand every
lease the same IP; the manager detects an ACK matching the interface's
own address or another client-id's live lease, releases the colliding
lease, and hard-errors advising a static range instead.
admin init auto-writes dhcp_mac="interface" for nat-mode public dhcp
pools bound to non-bridgeable uplinks (wlan/wwan/ppp).
* test(natuplink)+docs: Tier 2 EIP ingress e2e phases and public-pool docs
The natuplink suite now detects a public (non-transit) pool in spinifex.toml
and switches lanes: Tier-1-only configs keep the EIP-disabled and
no-public-IP assertions; public-pool configs assert the EIP surface is
enabled, the default subnet has MapPublicIpOnLaunch=true, instances
auto-assign public IPs, and a new Phase 9 exercises the full EIP ingress
lifecycle — host /32 route via the gateway LRP, proxy-ARP on the uplink,
per-EIP FORWARD accepts, exempt-stamped dnat_and_snat, TCP handshake to the
EIP, reconcile replay across a vpcd restart, and teardown on disassociate.
Docs: vpc-networking README gains a public-pool section for routed-NAT nodes
(init flags, dhcp_mac=interface behaviour, host-to-EIP caveat) and an updated
mode comparison; COMMANDS.md notes nat-mode EIP support behind a public pool.
* fix(setup): whitelist iptables, neigh sysctl, and arping for spinifex-vpcd
Routed-NAT mode shells out to iptables for the transit masquerade and
per-EIP FORWARD accepts, tunes proxy_delay, and sends gratuitous ARP for
host-delivered EIPs. Without these sudoers entries vpcd crash-loops with
"command not allowed" on nat-mode nodes.
* fix(ovn): scope DeleteStaticRoute prefix match to the target router
Every VPC router carries a 0.0.0.0/0 static route, so an unscoped cache
match could grab another router's row and fail with a referential
integrity violation, wedging IGW attach on multi-VPC nodes. Match only
rows referenced by the target router, mirroring FindStaticRoute. Mock
updated to the same semantics, with a regression test covering drift
replacement across two routers.
* fix(reconcile): always replay AttachIGW so host ingress state is re-ensured
applyIGWs skipped AttachIGW whenever the gateway switch port existed in
actual state, so the already-attached re-ensure path never ran and
routed-NAT host ingress routes lost on reboot were never reinstalled.
AttachIGW is idempotent; run it unconditionally.
* fix(vpcd): resolve duplicate-CIDR host ingress route collisions, bootstrap VPC wins
Every account's default VPC shares 172.31.0.0/16, so Tier-1 host routes
collide on the single host route table and the last IGW attach silently
stole the route (route replace). Pass vpcID and the gateway LRP IP
through RoutedIngressHooks; the bootstrap VPC always installs its route,
other VPCs only install when the CIDR is free and only remove a route
their own gateway IP holds. Adds host.VPCIngressRouteVia to read the
current route holder.
* ci(e2e): add nightly cell 19 for routed-NAT single-node suite
Cell 19 bootstraps with --external-mode=nat plus the CI static pool as
the public wan pool and runs the natuplink suite, covering the Tier-1
transit datapath and Tier-2 EIP ingress. The run step exits 0 when
natuplink.test is absent from the tests tar so scheduled runs against
branches predating the suite stay green.
* fix(admin): nat-mode init summary reflects configured public pool
The summary printed 'outbound-only — no public IPs/EIPs' even when
--external-pool/--external-source configured a public wan pool. Print
the pool range (or dhcp bind) and note EIPs are enabled.
* fix(eip,vpc): thread ctx through disabled EIP stub and NAT call sites
Reconcile the routed-NAT branch with dev's ctx-first service APIs after
rebase: DisabledEIPService and its tests take ctx to satisfy the EIPService
interface, the vm_adapters public-IP resolver asserts the ctx-signed
AssociatedPublicIPForInstance, and the default-subnet mapping test threads
ctx into DescribeSubnets.
* feat(otelsetup): OTel init, resource attrs, trace-stamping slog handler
Env-gated on OTEL_EXPORTER_OTLP_ENDPOINT: with no endpoint only the W3C
propagator is installed and globals stay no-op, so instrumented binaries
ship everywhere at zero cost. Resource carries service/host/mulga.env/
mulga.source plus OTEL_RESOURCE_ATTRIBUTES (ci.run_id in CI).
* feat(otelsetup): wire telemetry init into all spx services
Every spx service start command installs the trace-stamping JSON slog
default and the OTel providers via initTelemetry, flushed on shutdown.
Gateway's own slog setup now routes through otelsetup so its level
handling keeps trace correlation.
* feat(gateway): HTTP tracing middleware on awsgw and daemon cluster API
Server span per request with W3C traceparent extraction; awsgw spans are
renamed to the resolved SigV4 service.Action post-auth with account and
region attributes. Request log lines carry trace_id via context.
* feat(nats): propagate W3C trace context over command subjects
NATSRequestCtx injects traceparent beside X-Account-ID and opens a client
span per hop; ServeNATSRequest opens a consumer span joining the producer
trace, with a Ctx variant for handlers that thread context. RunInstances
is plumbed end-to-end (gateway dispatch through placement fan-out to the
daemon handler) and viperblockd ebs.mount/unmount join inbound traces.
* feat(systemd): source optional /etc/spinifex/telemetry.env in service units
Lets operators enable OTLP export (OTEL_EXPORTER_OTLP_ENDPOINT,
MULGA_ENV, MULGA_SOURCE) by dropping a single env file, with no unit
edits. File is optional; absent means telemetry stays disabled.
* feat(ci): export OTel telemetry from E2E envs to the central Elastic sink
All E2E workflows set SPINIFEX_OTLP_ENDPOINT (repo variable override,
default wattle) plus ci.run_id/ci.workflow resource attributes, so
bootstrap-install.sh writes telemetry.env on every node. Teardown runs
flush-telemetry.sh before the final tofu destroy so buffered spans and
metrics reach the sink.
* feat(otelsetup): request metrics and Go runtime metrics via OTLP
mulga.requests counter + mulga.request.duration histogram recorded by the
same middleware that opens spans: awsgw carries the resolved SigV4 action,
the daemon cluster API uses the chi route pattern, and NATS handlers record
per-topic at the subscribe chokepoints (node names stripped to keep
cardinality flat). Go runtime metrics start inside the same env gate.
* feat(daemon): deep spans for QMP commands, VM launch phases, EBS ops
Single chokepoint span per QMP command in sendQMPCommand covering all
callsites; vm.launch phase children (mount_volumes, start_qemu, qmp_connect,
persist_state); ec2.cmd consumer span with traceparent extraction plus
start/stop/reboot/terminate child ops; ebs.mount/unmount spans in
viperblockd with volume attributes.
* feat(telemetry): thread request context through NATS, object store and VM paths
Migrate utils.NATSRequest call sites to NATSRequestCtx (241 sites; W3C
traceparent now flows gateway -> daemon on every request), add GatherCtx and
migrate all Gather call sites, inject traceparent on manual EKS publishes.
ObjectStore interface is ctx-first with aws-sdk WithContext calls and an
otelhttp transport, so predastore spans join the caller trace. NATS consumer
spans record error status when the response envelope carries an error.
vm.Manager and the ec2 handler service interfaces take ctx-first so vm.launch
and QMP spans join the request trace; detached background work uses explicit
context.Background(). Hot-path slog calls use *Context variants for trace_id
binding.
* feat(telemetry): suppress health-check, heartbeat and unrooted trace noise
Health probes record metrics only (no span per poll); the QMP heartbeat
poller marks its context so query-status ticks stop rooting a trace each.
rootSampler honors MULGA_ROOT_TRACE_RATIO (ParentBased ratio, default 1.0)
and the predastore unit sets 0.05: guest block I/O roots are sampled while
request-linked spans are always kept.
* feat(otel): export deployment.environment resource attribute
Elastic APM maps it to service.environment, enabling the native environment
selector across the APM UI (Services, Traces, Dependencies).
* fix(telemetry): stop objectstore rooting method-only HTTP transactions
Background S3 calls (state sync, cleanup loops) carried no caller span, so
otelhttp rooted a trace per request and APM surfaced thousands of parentless
'HTTP GET/PUT/DELETE' transactions with expected-404 probes counted as
failures. Emit the client span only when the request context already carries
one, matching the viperblock backend. Bump viperblock/predastore to branch
heads so CI builds also pick up the equivalent span filter and root sampler.
* feat(telemetry): per-VM guest metrics via dedicated QMP socket + qmp-collector
Goanna Phase 1 producer, operator-consumed first. Each VM gets a second
-qmp chardev (qmp-telemetry-<id>.sock) plus a sidecar discovery file
carrying instance/account/vcpus/taps/monitoring-tier; the new
qmp-collector service polls query-cpus-fast (vCPU thread /proc jiffies),
query-blockstats, query-balloon and host tap sysfs counters, and
publishes CloudWatch-mappable goanna_ec2_* series to NATS
metrics.ec2.<instance-id>. A queue-group bridge in the same process
forwards the series to the node-local OTLP receiver so they reach the
operator sink until Goanna consumes the subjects. The collector never
touches the manager's control QMP socket; telemetry failures never
block a launch.
* fix(qmp-collector): dial NATS via DialTarget, drop ProtectProc, fix meta race
The collector connected to the raw NATS host (0.0.0.0), failing TLS
verification against a cert valid for 127.0.0.1; route through
admin.DialTarget like the other daemons.
ProtectProc=invisible hid the QEMU vCPU threads (dumpable=0) that the
collector must read from /proc/<tid>/stat, so every tick failed and
re-primed without ever publishing. Drop it to default.
Surface collect failures at Warn so silent metric loss is visible, drop
per-tick publish to Debug, and read InstanceID under the lock in run()
to fix a race with updateMeta on ENI hot-plug.
* chore(deps): re-pin predastore + viperblock to dev-landed commits
* feat(eks): reconciler in-place control-plane restart on sustained degradation
Phase 1 of EKS CP recovery (siv-243). The ACTIVE-branch reconciler only
recorded health; a CP VM that died (OOM/host crash) left the cluster wedged
ACTIVE-but-unreachable with no recovery path. Add a CPInstanceControl surface
and bounded in-place restart: once an ACTIVE cluster's CP stays unhealthy past
a grace window, StartInstance is issued (backoff-spaced, capped) when the
instance is stopped/error, re-mounting the same root volume so embedded etcd
survives. Degradation stays AWS-faithful ACTIVE + a ClusterHealth issue rather
than a non-AWS DEGRADED status enum. cpControl nil keeps the reflect-only path.
Wiring of a real CPInstanceControl in the daemon and the error-state
manager-eviction fix are the remaining Phase-1 integration steps; Phase 2
(etcd snapshot cadence + restore-snapshot) is tracked separately.
* feat(eks): wire control-plane restart into the reconciler via instance service
Threads a CPInstanceControl into the per-cluster reconciler (siv-243 Phase 1).
cpControlAdapter binds the daemon instance service + account: InstanceState
reads the CP's EC2 state via DescribeInstances, StartInstance restarts a stopped
CP via StartStoppedInstance, which loads from the shared stopped store and
re-mounts the same root volume so embedded etcd survives. Wired as
EKSServiceDeps.CPControl = d.instanceService and passed through spawnReconciler.
Recovers a cleanly-stopped control plane co-located with the reconciler leader.
Remaining: DescribeInstances is node-local so a cross-node stopped CP needs an
aggregating describe; and an error/crashed CP (not in the stopped store) needs a
dedicated error-state restart path — both tracked as Phase-1 follow-ups.
* feat(eks): cross-node CP describe + error-state control-plane restart
The reconciler's control-plane recovery was wired to the node-local instance
service: DescribeInstances only saw the leader's node and StartStoppedInstance
only handled a cleanly-stopped CP from the shared KV. A crashed/OOM CP ends in
error state, still resident in its owning node's manager map (not the stopped
store), so recovery never reached it — the actual incident in the recovery plan.
Wire EKSServiceDeps.CPControl to a NATS-backed eksCPControl. DescribeInstances
fans out across every host (gateway scatter-gather) so the reconciler observes a
CP on any node. RecoverInstance drives the gateway StartInstances two-step:
ec2.cmd.<id> recovers a live error/running owner in place (the owner stays
subscribed through the crash; vmMgr.Start relaunches error->pending, re-mounting
the etcd root volume), falling back to ec2.start shared-KV rehydration for a
stopped CP.
The CP VM is ManagedBy=EKS, hidden from customer DescribeInstances; relax that
hide so the owning account resolves its own managed CP when targeting it by
explicit instance ID (bulk listings still hide it). EC2APIState already maps the
internal error state to the AWS closed-enum stopped, which cpRestartable accepts.
* fix(eks): reconcile CP as system account, not customer
The control-plane VMs are launched under the system account
(admin.SystemAccountID()), not the customer account that owns the cluster
record. spawnReconciler was binding cpControlAdapter to the customer account,
so InstanceState/RecoverInstance ran as the customer — which cannot see or own
its own cluster's system-owned CP VMs. DescribeInstances returned zero and the
reconciler logged 'control-plane instance not found', never restarting a wedged
CP. Bind the adapter to the system account instead.
Revert the earlier DescribeInstances managed-VM relaxation: it was based on the
mistaken assumption that CP VMs are customer-owned. Visibility is resolved by
the system-account adapter, so the explicit-ID escape hatch is dead code.
* fix(vm): pre-create mgmt tap on disk-boot restart path
The direct-boot branch pre-creates the management tap via NetworkPlumber.SetupTap
(owned by the daemon euid), but the disk-boot/else branch only appended the QEMU
tap,id=mgmt0 device without creating the tap. The daemon runs as a non-root user
without CAP_NET_ADMIN, so QEMU can only attach a tap it already owns; on the
disk-boot path QEMU tried to create /dev/net/tun itself and failed with
'Operation not permitted', failing every stopped-instance restart (StartInstances
returned ServerInternal). This blocked EKS control-plane recovery: the reconciler
found the stopped CP and issued the restart, but the VM never booted.
Call SetupTap for the mgmt tap in the disk-boot branch, mirroring direct-boot.
* feat(eks): recover all HA control-plane members, not just primary
The reconciler restarted only ControlPlaneInstanceID (the primary CP
member), so on an HA cluster the other server VMs stayed stopped and
etcd never regained quorum (1/3) — apiserver healthz stayed fail even
after recovery ran. maybeRecoverControlPlane now iterates every member
in ControlPlaneNodes (falling back to the scalar ID for single-node
clusters), restarting each stopped/error member per pass while
preserving the grace window, attempt cap, and backoff gates.
Refs mulga-siv-509
* fix(eks): keep cloud-init polling IMDS so primary CP gets its user-data
The eks-node system VM's user-data is delivered only via IMDS (the
NoCloud seed was retired in #411). IMDS (169.254.169.254) is reachable
only once mulga-mgmt-net brings up the data NIC via DHCP and the per-tap
br-imds datapath reconciles, which on the first-launched (primary) CP
can lag cloud-init's stock 120s probe window. With the default max_wait
cloud-init falls back to DataSourceNone, permanently losing the launch
user-data (no SPINIFEX_K3S_ROLE -> no k3s server) and wedging the cluster
in CREATING.
Bake a cloud.cfg.d drop-in that pins DataSourceEc2 and raises max_wait to
600s so cloud-init keeps polling until the datapath is ready instead of
falling through to None. Worst case a slow boot, never a lost role.
* fix(eks): warm IMDS instance-role creds before first SigV4 sign
On IMDS-credentials mode the on-VM SigV4 helpers (eks-token-webhook,
eks-gateway-publish, mulga-eks-state-report) sign via the SDK default
chain, which resolves in-VM IMDS instance-role creds. The per-tap
br-imds datapath lags VM boot by minutes (observed ~264s), and the SDK's
IMDS client fast-fails each probe (250ms dial), so early Retrieve calls
return 'context deadline exceeded' / 'no EC2 IMDS role found'.
eks-gateway-publish only retries for 30x5s=150s, shorter than the
datapath warm time, so it exhausts its budget and drops the
bootstrap/state publish -> cluster hangs in CREATING.
The credential chain is sound (the CP role trusts ec2.amazonaws.com and
AssumeRoleForInstance evaluates that principal); the failure is timing.
Add gwsign.Signer.EnsureCredentials and block in eksgw.New until the
instance-role creds first retrieve (2s interval, 10m ceiling aligned
with cloud-init max_wait), so every helper's first signed request
already holds creds instead of racing a cold datapath. Static-key mode
is unaffected.
* fix(eks): retry CP data-NIC DHCP until cross-host datapath is up
A single ~16s DHCP one-shot on the primary data NIC lost the race against
the cross-host OVN datapath (logical-port binding + per-tap reconcile on the
remote placement host), stranding the NIC IP-less so the Ec2 datasource never
reached IMDS and the CP member wedged. Add dhcp_acquire: retry the one-shot on
a budget aligned with cloud-init's Ec2 max_wait (600s), confirming an inet
address before success. Each attempt still exits leaving no daemon, so
cloud-init's later ENI management is unaffected.
* fix(eks): drop reconciler registry entry when Run self-exits
ReconcilerRegistry.Spawn only cleaned up its holder on ctx cancel
(Stop/StopAll). When a ClusterReconciler's Run returned on its own — a
terminal cluster state or a lost lease — nothing cancelled the ctx, so
the holder leaked and a later same-name create silently no-opped at the
head of Spawn, never reconciling to ACTIVE.
Thread a finished channel from RunClusterReconciler's Run goroutine back
to Spawn's watcher, which now cleans up on whichever fires first: an
external cancel or Run exiting. On self-exit it deletes its own holder
and releases the lease, so a fresh same-name create re-spawns and
reconciles.
* fix(eks): run mulga-mgmt-net in boot runlevel before cloud-init-local
CP user-data is delivered only via IMDS since the NoCloud seed was retired
(#411), so cloud-init DataSourceEc2 must reach 169.254.169.254. mulga-mgmt-net
DHCPs the data NIC that makes IMDS reachable, but it was added to the default
runlevel and only declared `before cloud-init` (the network-stage service).
cloud-init-local therefore ran its init-local Ec2 crawl first with no data-NIC
route and burned its whole probe window (~4 min) before mulga-mgmt-net brought
eth0 up, regressing CP bringup from ~1m50s to ~5m54s.
Add mulga-mgmt-net to the boot runlevel in setup.sh instead of default, drop it
from ENABLE_SERVICES, and order it `before cloud-init-local` so the data NIC and
IMDS route are up before the init-local crawl. Keeps the IMDS-only user-data
path; the raised Ec2 max_wait stays as a safety net for host br-imds datapath lag.
* fix(vm): refresh LaunchTime on crash-restart so watchdog spares relaunch
RestartCrashedInstance re-launched a crashed VM without refreshing
Instance.LaunchTime. The pending watchdog measures now-LaunchTime against
PendingWatchdogTimeout, so the stale original launch time tripped
launch_timeout on the next tick and terminated the crash-restart
mid-flight. Refresh LaunchTime under the state lock, mirroring the
daemon-restart restore path (restore.go), so the relaunch gets a fresh
timeout window. Adds a regression test asserting LaunchTime advances.
* feat(eks): reconcile control-plane member count after a lost member
The cluster reconciler self-heals a stopped/error control-plane VM in
place but never replaced a terminated/gone member, so an HA cluster ran
below haControlPlaneCount permanently. Add a member-count reconcile pass
that provisions one replacement CP joining the surviving etcd quorum.
- CPProvisioner + EKSServiceImpl.ProvisionReplacementCP: replay the
persisted create template on a free host, the single-node analogue of
the spread join branch, with rotating creds re-derived.
- ControlPlaneTemplate persisted on ClusterMeta at create (per-node and
rotating-cred fields cleared). SwapControlPlaneMember CAS swaps the dead
node for the replacement and refreshes the [0] scalar mirrors.
- maybeReplaceControlPlaneMember fires only when no member is restartable
and one is truly lost, a healthy majority survives, bounded by grace,
backoff and an attempt cap; placement excludes live-member hosts.
- Replacement prunes the superseded member from etcd once joined:
k3s-first-boot deletes the stale Node (EKS_ETCD_PRUNE_PEER_IP) so quorum
width returns to N. json tags added to K3sServerInput (now persisted).
* feat(eks): in-guest control-plane self-diagnosis on healthz fail
When a control-plane apiserver reports unhealthy, the reconciler only
saw healthz=fail with no cause, and the guest has no on-demand command
channel (serial root login is password-gated, no guest agent). Make the
CP self-diagnose and relay the cause through the one-way state report.
- mulga-eks-state-report.sh: diagnose() collects failing /readyz?verbose
subchecks, isolated /healthz/etcd reachability, and etcd-db-dir free
space; on fail write a verbose block to the host-captured serial
console and add a JSON-safe reason to the state payload.
- ServerStateReport: add Reason (omitempty; empty from a healthy CP or
an older AMI predating the field).
- cluster_reconciler observe(): surface the reason in the health issue.
- Tests: POSIX unit test for the agent (stubbed kubectl/df); Go tests
for the reason-bearing and terse health-issue paths.
* feat(eks): 15-min etcd snapshot cadence alongside nightly
Nightly alone is too coarse a recovery point for a young cluster. Install the
same mulga-eks-etcd-snapshot script into a second Alpine crond periodic dir
(/etc/periodic/15min) so it also runs on a ~15-minute RPO window; the script
picks its cadence tier from the dir it is invoked from ($0 sniff) and encodes
the tier in the snapshot key (etcd-{tier}-{ts}.snap), so a predastore S3
lifecycle rule can retain the frequent rolling window shallow and the nightly
copies deep. Server-role only (agents exit early); no restore-path change.
Add a self-contained POSIX unit test (stub k3s/curl/logger, symlinked crond
dirs to exercise the real $0 tier sniff) covering agent no-op, frequent + daily
key prefixes, local-snapshot prune, and quiet no-op without creds.
* feat(eks): auto-reform wedged etcd quorum via boot-time cluster-reset
An HA k3s control plane does not self-heal after a simultaneous full-CP
restart: embedded etcd never reforms quorum (etcd:unreachable sustained),
and in-place VM restarts cannot fix it. Add a last-resort escalation that
drives the documented k3s cluster-reset recovery, pull-based so the host
never needs an inbound channel into the guest.
Reconciler: maybeReformEtcdQuorum escalates only when every CP member is
VM-running but the health issue carries etcd:unreachable, the cluster is
HA, and a grace window has elapsed. It sets per-member recovery directives
(seed = cluster-reset, others = wipe-rejoin), epoch-bumped and keyed by
instance ID, then restarts the members. Bounded by grace/backoff/attempts;
a recovered cluster clears the directives once. Enabled via
WithEtcdResetRecovery alongside the existing CP control wiring.
Directive plane: per-member KV directive {epoch, action, snapshot} with
store helpers, an EKS service Get/Set pair over NATS-RPC, and an
internal-recovery gateway route (system-cred carve-out, same as
internal-addons). The CP server instance role gains eks:GetRecoveryDirective.
Guest: mulga-eks-k3s-recovery is a pre-k3s oneshot (server role only) that
fetches its directive through the AWS gateway (eks-gateway-fetch -resource
recovery), epoch-guards one-shot application, then runs
`k3s server --cluster-reset` (optionally restoring a predastore snapshot)
or wipes the local etcd datastore to rejoin the reset seed. Wired into the
image manifest, setup.sh perms, and the eks-node-role selector.
Tests: reconciler escalation gates + recovery clear, directive store epoch
increment + service validation, and a POSIX unit test for the guest agent.
* fix(eks): reboot running CP members to apply etcd cluster-reset directive
The etcd quorum-reformation escalation drove StartInstance on the seed and
followers, but the reset gate only fires when every member is VM-running. The
StartInstances path rejects a running instance with IncorrectInstanceState, so
the guest never re-entered its boot sequence and the cluster-reset/wipe-rejoin
directive was never applied — the wedged control plane stayed wedged.
Reboot the running members (QMP system_reset) instead, re-entering the boot
sequence so the on-VM recovery agent applies the directive. Adds RebootInstance
to the CP control surface (interface, adapter, NATS-backed daemon impl) and
switches maybeReformEtcdQuorum to it; the stopped/error in-place restart path
keeps StartInstance.
* fix(eks): graceful stop for etcd reset recovery
The etcd quorum-reformation reset path forced running CP members through a
hard QMP system_reset to re-enter the boot sequence and apply their recovery
directives. A hard reset skips a clean unmount: on the next boot e2fsck
cleared the recovery and k3s runlevel entries (dangling inodes), so the
cluster-reset never ran and k3s was unstartable — the reset defeated recovery.
Replace the reboot with a graceful StopInstance (QMP system_powerdown, clean
unmount, SIGKILL fallback). The existing in-place restart path
(maybeRecoverControlPlane) then boots each stopped member clean on a later
tick, and the on-VM recovery agent applies its directive against an intact
filesystem. Reuses existing lifecycle machinery; the stop also tolerates a
wedged QMP client via the SIGKILL fallback.
Swaps RebootInstance -> StopInstance across CPInstanceControl,
cpInstanceController, the cpControlAdapter, and the NATS-backed eksCPControl
(gateway RebootInstances -> StopInstances), plus the reset/recovery test fakes.
* fix(vm): retry QMP dial and unlink stale socket on relaunch
A QEMU killed with SIGKILL (crash or etcd-reset restart) leaves its
qmp-<id>.sock inode behind. On relaunch the startup os.Stat probe and the
QMP dial both raced that dead listener, so a single dial returned
'connection refused': StartStoppedInstance then reported ServerInternal and
left the instance KV-stopped even though the guest booted fully. The wedged
member stayed invisible to the reconciler's cpLive gate, stalling recovery.
Unlink the stale socket before launch so the inode that appears is the fresh
QEMU's, and retry the QMP connect with a short backoff to cover the window
where the socket is bound but not yet listening. Handshake failures are not
retried — a listener answered, so the error surfaces immediately.
* feat(eks-ui): surface cluster Health.Issues on the EKS console
DescribeCluster is AWS-faithful (Status stays ACTIVE, degradation lives in
Health.Issues), but the cluster detail page only rendered the status badge, so
a degraded control plane still showed as ACTIVE with no visible signal. Add a
destructive health badge next to the status and a Health row that lists the
issue messages, mirroring the addons tab. Healthy clusters read OK.
* chore: rebuild ui
* fix(viperblockd): stop mount-path daemon VB uploader to prevent checkpoint clobber
The daemon-side VB created in the ebs.mount handler is stored in
MountedVolume.VB for the whole mount, and its background chunk uploader was
only stopped at daemon shutdown. That state-only VB ran DrainToBackend every
30s, overwriting the live checkpoint the nbdkit plugin VB (the data-path
writer) keeps updating, leaving SeqNums inconsistent with sealed chunk
ciphertext and causing transient AEAD auth failures on cloned encrypted
volumes. Stop its uploader right after New, matching the other short-lived VB
call sites.
Also bump viperblock to the atomic chunk-ObjectID allocation fix.
* build(deps): pin viperblock to landed AEAD concurrency fix
Bump github.com/mulgadc/viperblock to the dev commit b08bea6 that
eliminates "cipher: message authentication failed" on encrypted volumes
under concurrent read-modify-write + background chunk drain. Removes the
local-path replace used while the fix was in flight.
* fix(viperblock): stop chunk-uploader goroutine leak at all VB call sites
viperblock.New() unconditionally starts a background chunk-uploader
goroutine (30 s interval) that only VB.Close() stops. Seven call sites
created short-lived or state-only VB instances without calling Close(),
leaking one uploader goroutine per instance. On a single node the leaked
goroutines exhaust memory (predastore RSS grew ~1.3 GB/5 min at idle
post-single-suite), causing liveMemGate to reject launches and the lb
suite to fail.
The leaked uploaders are also the primary cause of the AEAD auth
failures (mulga-be2ev): a leaked VB from prepareRootVolume holds the
clone-point BlockLookup and overwrites the correct live checkpoint in S3
every 30 s, leaving SeqNums inconsistent with sealed chunk ciphertext.
Fix all seven sites:
- instance/service_impl.go prepareRootVolume: defer StopChunkUploader +
StopWALSyncer after successful viperblock.New (mirrors the EFI path's
intent; full Close is unsafe here because cloneAMIToVolume already
called SaveBlockState + RemoveLocalFiles)
- viperblockd.go ebs.delete handler: add StopChunkUploader alongside
existing StopWALSyncer
- viperblockd.go ebs.unmount handler: same
- viperblockd.go shutdownVolumes: same
- snapshot/service_impl.go snapshotVolume: defer both stops after New
- volume/service_impl.go CreateVolume: defer both stops after New
- image/service_impl.go snapshotRunningVolume: defer both stops after New
- image/service_impl.go snapshotStoppedVolume: defer both stops after New
Closes mulga-0zzci
Closes mulga-ji5fi
* fix(deps): bump viperblock to 01a7e686 (AEAD regression fix)
The prior pin (6af89855) carried the snapshot-AMI AEAD regression that
corrupts block storage on restore. Point at the merged viperblock dev fix.
---------
Co-authored-by: joshsiv-mulga <mrk@mulgadc.com>
Failed LBs previously reported an empty State.Reason, so e2e failures
read 'entered terminal state failed:' with no cause. Record the cause
on every failed transition — launch error (e.g. insufficient sys.micro
capacity), missing system credentials, missing internal mgmt return
route, and async-launch panic — persist it as StateReason, and map it
to LoadBalancerState.Reason in describe responses. Cleared when the
first heartbeat transitions the LB to active.
The internal lb subtests raced the previous subtest's sys.micro VM
deallocation: fixed 15s sleeps were not enough on capacity-tight
single nodes, so the LB entered terminal state failed (insufficient
capacity, then nats no-responders after the daemon unsubscribed the
full topic).
Replace the sleeps with createActiveLB: create LB + listener, wait for
active via the new non-fatal harness WaitForLBActiveErr, and on a
terminal failed state (wraps ErrLBTerminalFailed) tear down fully and
retry with backoff, up to 3 attempts.
go.mod referenced v1.10.0 which pre-dates LoadLiveCheckpoint. CI builds
without go.work so e2e-build failed to compile. Pin to viperblock dev tip
6af8985 (feat/snapshot-ami merged) until v1.11.0 is tagged.
* feat(gpu): add H200/RTX Pro 6000 SE models and fix MIG host resource allocation
- Fix H100 SXM/PCIe device ID labels swapped in models.go (10de:2330=SXM, 10de:2331=PCIe)
- Add H200 SXM (10de:2335) to knownModels, computeModels, and migCapableModels
- Add NVIDIArtxPro6000BlackwellSE to definitions.go mapped to g7e family (matching AWS g7e instance type)
- Add MIGHostResources() policy table: 4 vCPUs / 32 GiB host RAM per compute slice, covering all known NVIDIA MIG profiles
- GenerateMIGTypes now emits non-zero vCPU/memory for EC2 instance type descriptors
* fix(daemon): enforce CPU/memory limits for MIG instance types
Whole-GPU passthrough remains bypass-only (gpuManager.Claim is the sole gate).
MIG types now fall through to canAllocateCount so host CPU and RAM are checked
before launch — prevents over-committing a host where one physical GPU backs
multiple concurrent slices, each with non-trivial vCPU/memory allocation.
* feat: Add snapshot image promotion flow
* fix: Small changes to viperblock/spinifex so snapshot captures delta properly and instances can boot from snapshot AMIs
* fix: test coverage and more consistent prefix check
* fixed testing
* more tests
* fix: added new graceful process killer in utils, bumped NATS timeout for testing
* fix: complete at-rest encryption launch + volume lifecycle (#361)
* chore: bump viperblock to encrypted-snapshot VolumeUUID nonce fix
Pull viperblock 1044c2a (feat/snapshot-uuid-nonce-fix) so E2E builds against
the fix for the encrypted AMI snapshot metadata tag-verify failure
(cipher: message authentication failed) on instance launch. Pairs with
viperblock PR #22.
* fix: forward viperblock master key to nbdkit mount path
viperblockd loaded the master key but never passed it to the nbdkit
plugin it spawns for volume mounts, so encrypted boot volumes opened
in cleartext and failed LoadState with an encryption-state mismatch.
Forward cfg.EncryptionKeyFile into NBDKitConfig and emit the plugin's
encryption_key_file= argument when set.
* chore: bump viperblock to fresh-volume VolumeUUID mint fix
* chore: bump viperblock to snapshot-link LoadState ordering fix
* fix: route encrypted-volume config writes through viperblock keyholder
config.json is a sealed VBState whose AES-GCM tag and StateSeqNum-derived
nonce can only be advanced by the master-key holder that owns the volume.
The EC2 volume handler wrote it directly to S3, so mergeVolumeConfig had to
refuse encrypted volumes (rewriting would strip the tag) — bricking
UpdateVolumeState (attach in-use / detach available) and ModifyVolume.
Route encrypted-volume config updates through viperblockd, mirroring the
ebs.snapshot idiom: per-volume ebs.config.{volumeID} hits the owning node's
live VB (the single StateSeqNum writer); on ErrNoResponders the ebs.config
queue group opens the detached volume exclusively and reseals. Cleartext
volumes keep the direct-S3 fast path. No crypto leaves viperblockd.
mulga-siv-323
* fix: unwrap encryption envelope when snapshot handler reads config.json
CreateSnapshot and snapshotInUseByVolumes decoded config.json straight into
a VBState. For encrypted volumes config.json is an at-rest envelope
{payload, authtag}; a raw decode yields a zero VBState (SizeGiB==0), so
CreateSnapshot 500s on the size guard and the in-use scan silently drops
encrypted volumes' snapshots. Unwrap via viperblock.StateBody first,
mirroring the volume handler's mergeVolumeConfig path.
mulga-siv-324
* test: cover viperblockd ebs.config handlers
Unit/integration coverage for applyConfigUpdate, makeConfigUpdateHandler
(ebs.config.{volumeID}), and the ebs.config queue-group fallback: grow-only
size reconcile, live-VB apply, invalid JSON, apply errors, and the
detached-volume open path. Clears the diff-coverage gate (68.8%).
* chore: pin viperblock to dev tip (#22 merged)
* fix(viperblockd): seal volume block-map to predastore on detach + durability e2es (#371)
* feat: cloud-init EC2 datasource (#411)
* feat: Add snapshot image promotion flow
* fix: Small changes to viperblock/spinifex so snapshot captures delta properly and instances can boot from snapshot AMIs
* fix: test coverage and more consistent prefix check
* merge changes
* chore: remove dead SnapshotSub field and SIGTERM snapshot remnants from viperblockd
SnapshotSub was added during the NATS/SIGTERM snapshot attempts but is never
set post-rebase. Removes the field from MountedVolume and the two nil-checked
unsubscribe blocks. Re-adds the nbd import which is still needed for NBDKitConfig.
* fix: post-rebase gofmt fixes and remove dead snapshot handler code
Split gofmt-squashed lines in go.mod, vm_adapters.go, nbd_test.go,
viperblockd.go, volumes.go, and volume_durability_test.go. Removes the
dead unix-socket snapshot handler tests from viperblockd_handlers_test.go.
* test: add viperblock snapshot S3 coverage tests
Exercises snapshotVolume, snapshotRunningVolume, and snapshotStoppedVolume
past the nil-config guard using an httptest server that returns 403.
The AWS SDK credential check fails synchronously without network I/O,
keeping the tests fast while covering the S3Config construction, master key
load, and viperblock.New code paths. Brings diff coverage from 45% to 69.9%.
* fix: cloneAMIToVolume reads config via format-agnostic path
LoadStateRequest expects a viperblock binary envelope (encrypted),
but CreateImageFromInstance writes the AMI config as plain JSON via
putAMIConfig. Replace LoadStateRequest with s.amiLoader.GetAMIConfig
which uses viperblock.StateBody to handle both formats, matching the
path used by ListImages and all other AMI readers.
* fix: pin viperblock to feat/snapshot-ami pseudo-version for LoadLiveCheckpoint
go.mod referenced v1.10.0 (released from dev) which pre-dates LoadLiveCheckpoint.
CI builds without go.work so it resolved to the published version and failed.
Pin to the snapshot-ami HEAD pseudo-version until the branch is merged and tagged.
* fix: replace indexOf with slices.Index, revert e2e runner and viperblock version
- nbd_test.go: remove local indexOf helper, use slices.Index from stdlib
- e2e-suites.yml: revert ubuntu-latest back to ubuntu-26.04 with correct formatting
- go.mod: revert viperblock back to v1.10.0 (will resolve once viperblock PR merges)
* fix: remove pseudo-version pin from go.sum
* feat: drain viperblock via snapshot socket before CreateSnapshot
Before LoadLiveCheckpoint(), dial {data_dir}/viperblock/{vol}/snapshot.sock
and wait for DrainToBackend() to complete (30s timeout). This ensures WAL
chunks and the live checkpoint are current in S3 before the snapshot VB
reads them. If the socket is absent the instance is stopped and Close()
already drained.
fix: socket path used s.config.WalDir which is empty in the daemon config;
changed to s.config.DataDir + "/viperblock" to match the path the NBD
plugin uses when creating the socket.
* feat: wire snapshot/AMI UI actions and add PromoteImage NATS endpoint
- Register as AMI from snapshot detail (RegisterImage with BDM pointing at snapshot)
- Deregister from image detail (DeregisterImage with navigate-away on success)
- Promote to System from image detail (admin-only, irreversible via UI)
- NATS subject spinifex.image.promote wired through gateway → daemon → existing admin.PromoteSystemImage
- signedFetch extended with optional params to support action-keyed extra fields
* refactor: address PR review on PromoteImage types and subscription ordering
- Merge PromoteImageInput/PromoteImageOutput into PromoteImageOpts/PromoteImageResult
(add JSON tags; drop the duplicate pair)
- Inline the input variable in gateway/spx/promote_image.go
- Move spinifex.image.promote subscription to the spinifex.* group in daemon.go
* test: cover snapshotVolume/Running/Stopped error paths for diff coverage
Add encryption-key-load-error and LoadState-failure tests for the three
new viperblock snapshot paths (snapshotVolume in snapshot service,
snapshotRunningVolume and snapshotStoppedVolume in image service).
Diff coverage: 58.8% → 65.5%.
---------
Co-authored-by: Josh Siviour <mrk@mulgadc.com>
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
* fix: ECS StopTask reaps container before releasing capacity
StopTask marked a task lastStatus=STOPPED and released its capacity/ENI in
the control plane, but nothing ever told the ecs-agent to kill the container.
The container kept holding its host ports, so DescribeTasks reported a STOPPED
task whose container was still RUNNING, and a replacement task placed on the
freed capacity that rebound the same host port hit EADDRINUSE.
Wire the agent-driven cooperative stop the design intended (ecs-v1.md:165):
- Add a per-instance stop inbox (StopsPrefix/StopKey) and bus.StopDirective.
- requestStopTask marks DesiredStatus=STOPPED and posts a stop directive,
leaving lastStatus/capacity/ENI/targets untouched. StopTask, DeleteService,
and service scale-in use it; the heartbeat reaper and cluster/instance
teardown keep the immediate forceStopTask since no live agent can reap.
- PollAssignments returns pending stops; the agent lists containers by
mulga.ecs.taskID label, Removes them (kill+delete, freeing ports/netns),
then reports STOPPED. recordTaskState performs the single capacity release
on the agent's STOPPED report and reclaims the stop inbox.
- listServiceTasks excludes DesiredStatus=STOPPED tasks so a cooperatively
stopped task drops out of the service's running total immediately.
DescribeTasks now keeps the task RUNNING with desiredStatus=STOPPED until the
container is actually reaped, matching AWS ordering and eliminating the port
race.
* test: cover StopTask reap paths — poll stop-inbox drain, unplaced force-stop, agent stopTask edges
* ci: retrigger PR checks
- RegisterTaskDefinition rejects container secrets[] (InvalidParameterException)
and warns on unsupported log drivers instead of silently dropping them.
- Capture logConfiguration; only host-side json-file is collected, other drivers
warn at register and reach the agent for parity. Document json-file retrieval.
- Plumb executionRoleArn taskdef -> assign -> agent; the agent assumes it to
authorize ECR pulls, falling back to the instance role when unset.
- Add deployment objects with a health-gated rolling update honoring
deploymentConfiguration minimumHealthyPercent/maximumPercent, plus a deployment
circuit breaker with optional rollback to the last-good task definition.
EnsureLogicalSwitch/Router/PortGroup returned a client-side named-uuid on the
fresh-insert path but the real server uuid when the row already existed, so a
caller storing the returned uuid then doing Where(uuid) would miss. Resolve the
persisted uuid from the monitored cache after insert and add an explicit created
flag; topology callers now branch on created instead of uuid equality.
ReplaceACLs deleted and recreated the whole ACL set every reconcile, minting new
row uuids even for an unchanged SG and forcing ovn-northd flow recompute on every
drift tick. Diff current vs desired ACLs and no-op when unchanged (live + mock).
* test(network): in-process OVN NB reconcile harness
Add ovntest.StartNB: an in-process OVN Northbound DB backed by libovsdb's
pure-Go OVSDB server (no ovsdb-server/ovn-northd C daemons) plus a vendored
NB schema subset matching nbdb. Real LiveClient connects over a unix socket,
so reconcile scenarios exercise intent generation, reconcile wiring, LiveClient
and OVSDB transaction semantics in milliseconds without VMs.
Add live reconcile scenarios asserting NB rows and idempotency:
- VPC + subnet + SG + ENI from a hand-built IntentState
- the same seeded through JetStream KV via LoadIntentFromKV (front-to-back)
- IGW attach + drop gate + public-instance exemption (Logical_Router_Policy)
The idempotency snapshot normalizes UUID and pointer churn so it asserts
logical stability; datapath and NB->SB translation stay in the e2e suites.
* test: add orphan-prune and kitchen-sink golden live NB scenarios
Extend the in-process OVN reconcile harness with the two P0 completeness
scenarios. TestScenario_OrphanPrune_Live locks the teardown path and the
ReconcileApplyOnly startup-race guard: apply-only must not prune, full
Reconcile must, and re-pruning is idempotent. TestScenario_KitchenSinkGolden_Live
reconciles a maximal intent exercising every applyX path (VPC, public+private
subnets, two ENIs, SG with tenant rules, IGW, associated EIP, NAT gateway,
IGW/NATGW reroutes, drop gates) and asserts the normalized NB snapshot against
a checked-in golden, mechanically catching an unwired or mis-wired resource
type. Regenerate the golden with UPDATE_GOLDEN=1 go test.
* fix: repoint DB-node ovn-northd at RAFT member list; extend add-nat deadline
Two related fixes for multi-node OVN clusters:
- setup-ovn.sh (mulga-siv-504): in the clustered --management branch, point
ovn-northd's client NB/SB connections at the full RAFT member list derived
from OVN_REMOTE, so the active northd follows the leader instead of dialing
its local member only. Without this, northd stops advancing SB_Global.nb_cfg
once leadership moves off the node, wedging the ovn-nbctl --wait=hv barrier.
Guarded on a real member list so the standalone single-DB-node path is
unchanged.
- utils/nats.go (mulga-js-206): raise the add-nat request deadline from 10s to
45s so the client no longer rolls back before the server-side --wait=hv flows
barrier (30s) completes.
* fix: stop ovn-northd restart from tearing down split NB/SB ovsdb servers
The packaged ovn-northd.service ExecStop runs 'ovn-ctl stop_northd' without
--ovn-manage-ovsdb=no, so restarting northd to apply OVN_CTL_OPTS also stops the
separately-managed NB/SB ovsdb-server units. On a clean clustered bring-up this
races the freshly-started DBs and kills them, so setup-ovn's wait for the NB
socket times out and the RAFT create/join aborts. Install an ovn-northd.service
drop-in overriding ExecStop with --ovn-manage-ovsdb=no so northd never touches
the DB lifecycle (mulga-siv-504).
* test(e2e): capture SNAT datapath on NAT GW egress failure
TestNATGateway captured nothing when the private-VM egress probe failed, so
the baremetal nightly's intermittent 100% packet loss left no OVN/OVS state in
the failure bundle to root-cause it. Register harness.OnFailure before the
egress-verify poll to dump the SNAT datapath (OVN show, NAT rules, port
bindings, br-int flows, conntrack, ARP) keyed on the NAT GW external IP,
matching the eip/vpc datapath tests.
Also add a br-ext localnet flow capture to dumpDatapathState: br-int flows miss
the egress hop where SNAT'd traffic leaves for the physical uplink, which is
exactly where an egress black-hole would show.
* test(e2e): add OVN pipeline diagnostics to NAT GW egress capture
Capture the gateway router's per-stage logical-flow packet counts
(ovn-sbctl --stats lflow-list), MAC_Binding, static routes and the VM's
logical switch port alongside the existing OVS/conntrack/ARP dumps. When
the SNAT flow reads n_packets=0 while the guest transmits, these pinpoint
which pipeline stage drops the packet upstream of lr_out_snat.
* test(e2e): dump full lflow-list + datapath ct/flows for egress RCA
The IP-filtered lflow dump hides the default-drop ACL flow (its match
carries no IP), so a pre-SNAT egress drop at ls_out_acl only showed the
allow-flows at n_packets=0, never the deny that ate the packets. Replace
it with a full ovn-sbctl --stats lflow-list, echoed so it survives in the
CI stdout log. Swap the conntrack capture to ovs-appctl dpctl/dump-conntrack
(the standalone conntrack binary is absent on the baremetal runner) and add
dpctl/dump-flows -m for kernel megaflow hit counts.
* fix: claim gateway cr-port on every reconcile miss and stop masking launch failures
ensureGatewayClaimed nudged ovn-controller recompute only once then polled
until timeout, so a fresh VPC bring-up or chassis flap that fired the single
nudge too early left the gateway chassisredirect port permanently unclaimed.
Recompute on every miss instead, mirroring ensureGuestPortDatapath.
aggregateResults and distributeInstancesSpread masked any node launch failure
(including the resulting 5-minute launch timeout) as InsufficientInstance
Capacity. Surface the real node error when one exists; keep the capacity code
only for genuine capacity shortfalls.
* feat(eks): multinode control-plane networking, IAM, and CoreDNS parity
Integrates the EKS multinode fix set for the managed-control-plane topology
so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS
EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the
node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it.
EKS:
- Route apiserver egress (exec/logs/webhooks) through konnectivity
(apiserver-network-proxy): CP servers run konnectivity-server behind the
cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod
traffic rides the agent-initiated reverse tunnel instead of an unroutable
CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every
replica (HA-correct).
- Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay
flannel mesh SG rule authorized at cluster launch with ordered teardown.
- Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay.
- Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master
tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity.
- Spread nodegroup workers across hosts; addon readiness falls back on
workload rollout when gvks are absent.
- elbv2: allow the proxy_protocol_v2.enabled target-group attribute.
EBS:
- Move volume attachment state to state.json so a live nbdkit VB cannot
clobber it; route mounted-volume state updates through the live VB with a
retry for in-use marks racing the mount responder; keep the boot volume
in-use across stop/crash unmount.
* fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM
missingOrchestrationDeps gated on s.deps.IAM, but production wires only the
lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only).
The gate therefore reported missing=[IAM] on every node forever, so
CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even
though the IAM service was fully up via the provider.
Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else
deps.IAMProvider()) the nodegroup/role code already uses — so the protective
check still rejects a node with no IAM service but passes when the provider
supplies one. Add a regression test that sets only IAMProvider and asserts the
gate is satisfied; the existing nil-both test still rejects.
* fix(eks): let eks-token-webhook use IMDS creds when static keys absent
The CP VM launches on an instance profile, so buildK3sUserData omits
EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys
before eksgw.New could fall back to the IMDS SDK chain, so the webhook
aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver
crashed on the missing --authentication-token-webhook-config-file file,
stalling the cluster at CREATING until it timed out to FAILED.
Drop the static-key precondition (matching the sibling eks-gateway-publish
and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/
cluster gates.
* fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane
The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt,
which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a
169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing
169.254.169.254 and steering IMDS off the data NIC away from the per-tap
br-imds datapath, where the host RST it. eks-gateway-publish then failed to
sign requests and no bootstrap artifacts were ever published, so clusters sat
CREATING until they timed out to FAILED.
Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address
or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI)
are unaffected and IMDS falls through the data NIC's default route to the
br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net
deletes any default route on a NIC declared NIC<n>_DEFAULT=0.
* fix(eks): grant eks:WebhookTokenReview to the control-plane instance role
The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway
broker at POST /clusters/{name}/token-review, signing with the control-plane
VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes
against eks:WebhookTokenReview, but the role's inline policy granted only
eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every
relay and the apiserver returned 401 to all external kubectl.
Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile
re-asserts the inline policy on every control-plane instance-profile ensure, and
the gateway evaluates authorization per request, so existing clusters converge
without a rebuild once the shared role is updated.
* feat: add OVSDB RAFT cluster primitives to setup-ovn.sh
Add --db-cluster-local-addr / --db-cluster-remote-addr flags so a
management node can run the OVN NB/SB databases as a native OVSDB RAFT
cluster instead of a single non-clustered ovn-central. When a local addr
is given, write a shared OVN_CTL_OPTS to /etc/default/ovn-central (NB
RAFT 6643 / SB RAFT 6644) and restart the per-DB units directly, since
the ovn-central aggregator is ExecStart=/bin/true and won't restart its
children. Empty remote addr creates the cluster (init node); a set remote
addr joins it. NB/SB client set-connection runs only on the create node
(RAFT replicates it cluster-wide).
The single-node --management path is unchanged when no cluster addrs are
supplied, so dev boxes and existing single-node clusters keep working.
--ovn-remote already passes through verbatim, so a comma-separated SB
endpoint list works for compute nodes; documented in usage.
* feat: OVN client multi-endpoint failover for RAFT clusters
Accept a comma-separated OVN NB/SB endpoint list so vpcd can target a
clustered DB. config.ParseEndpoints splits the list (trimming whitespace,
dropping empties); live.go Connect passes one libovsdb WithEndpoint per
endpoint, so the client tries them in order, fails over on disconnect, and
follows RAFT leader redirects. Single-endpoint strings still yield one
endpoint, so single-node dev is unchanged.
ovn-sbctl --db= already accepts the raw comma list, so the SB CLI
consumers (discoverChassis, gateway_claim) need no change.
* feat: write clustered OVN NB/SB endpoints across the RAFT quorum
spx admin init/join now emit comma-separated OVN NB/SB endpoint lists for
the first three quorum nodes (formation.BuildOVNDBAddrs) instead of loopback
or a single leader address, so vpcd fails over across the cluster.
firstboot defers OVN bring-up to the provisioning controller when it owns
formation (SkipFormation): a standalone single-node .db would otherwise block
the controller's clustered create/join.
* test: add multinode OVN RAFT cluster + failover e2e; document clustered topology
ovn_raft_test.go asserts NB/SB cluster/status reports a 3-server quorum with
one leader on every DB node, then stops the NB leader's ovn-central and proves
an NB write still lands via a surviving endpoint.
Docs: vpc-networking gains the multi-endpoint vpcd config and a RAFT
cluster-status debugging section; network-connections lists the 6643/6644 RAFT
ports.
vpcd's libovsdb NB client was a single long-lived connection with no
reconnect logic. After the NB server restarted (host reboot, service
restart) every vpcd holding a dead/half-open socket never re-established
it, so OVN-write handlers blocked until the caller timeout and CreateVpc
returned ServerInternal intermittently (queue-distributed across vpcd).
Add WithInactivityCheck to the OVSDB client: a periodic Echo probe
detects the dead peer and triggers reconnect against the configured
endpoint, with infinite backoff so a daemon keeps retrying across a long
outage. Bound the SG subscriber handlers with a timeout context so a
stuck OVN op frees the worker instead of hanging on context.Background.
NLBs hard-rejected customer security groups and always minted a managed SG that
auto-opened listener ports, so aws-load-balancer-controller (which assigns SGs to
type=external NLBs) got a 400 and looped. Accept up to 5 customer SGs on an NLB:
when supplied they replace the managed SG on the ENIs and the caller owns the
listener-port rules (CreateListener already no-ops its auto-open when there is no
managed SG); without them the managed-SG default is unchanged. SetSecurityGroups
follows AWS create-time semantics — replace SGs on an NLB created with SGs, reject
adding them to one created without.
CreateVolume hardcoded iops to 3000, silently dropping a caller-supplied Iops,
and rejected non-gp3 VolumeType with the generic InvalidParameterValue. Honor
input.Iops within the gp3 envelope (3000 baseline, 500 IOPS/GiB, 16000 ceiling)
and return the volume-type-specific UnknownVolumeType for unsupported types.
Closes EBS CSI API gaps #3-4 (siv-380); gp3 Throughput (gap #2) deferred.
* feat: IMDS-capable gateway signer for lb-agent + EKS clients
Add internal/gwsign, a SigV4 signer that draws credentials from either
fixed static keys or the AWS SDK default credential chain. In a guest VM
the chain falls back to the in-VM IMDS endpoint, yielding scoped,
auto-rotating instance-role credentials; a session token is signed as
X-Amz-Security-Token so the gateway can verify temporary (ASIA) creds.
Wire it into the lb-agent (lbagent.Agent) and the EKS on-VM client
(internal/eksgw). When static keys are absent the clients now sign with
IMDS credentials instead of failing; when present, behaviour is
unchanged (static path). This is the client-side foundation for dropping
long-lived secrets from VM user-data.
Credentials are retrieved per request so rotated IMDS creds take effect
without an agent restart.
* feat: IMDS instance-role creds for EKS server + LB VMs (Phase B/C)
Attach a Spinifex-managed system instance profile to the k3s control-plane VM
and the elbv2 LB VM so both authenticate to the gateway with scoped, rotating
IMDS instance-role credentials instead of a long-lived static secret baked into
user-data.
- Add a shared handlers/iam.EnsureSystemInstanceProfile that find-or-creates a
role (EC2 trust) plus its instance profile and asserts an inline permission
policy. The gateway enforces per-action authz on assumed-role principals, so
the roles carry exactly the actions each VM calls:
- EKS server: eks:PublishInternal, eks:ListInternalAddons.
- LB agent: elasticloadbalancing:LBAgentHeartbeat, GetLBConfig.
- Plumb IamInstanceProfileArn through sysinstance.SystemInstanceInput and both
daemon launch paths (BootAMI via PrepareRunInstances->RunInstance; direct-boot
microVM via RunInstance), which persist it so the in-VM IMDS endpoint serves
the role.
- Drop the baked secrets when a profile is attached: omit EKS_ACCESS_KEY/
EKS_SECRET_KEY from the k3s first-boot env and LB_ACCESS_KEY/LB_SECRET_KEY from
the lb-agent fw_cfg env; relax k3s server validation to accept profile-only;
drop the EKS key requires from the first-boot and state-report scripts.
- Both verticals fall back to baked static creds when IAM is unwired, so an
absent master key never blocks a launch.
* fix: resolve IMDS instance-role creds under the instance account for system VMs
Three boot-path defects kept the LB VM's lb-agent from authenticating to the
gateway with IMDS-served instance-role creds; each produced a silent empty IMDS
role list and a heartbeat failing with "unexpected empty EC2 IMDS role list".
Found and fixed against env19.
- Build the system IAM ensurer lazily on the first system-VM launch instead of
eagerly at daemon startup, where it raced the NATS KV backend and fell back to
baked static creds permanently. ELBv2 and the EKS deps now hold an IAMProvider
resolved at launch time.
- Create the LB instance profile in the system account the LB VM runs under
(utils.GlobalAccountID), not the LB-owner account. IMDS resolves the profile
under the instance's account, so a customer-account role was invisible to it.
- Record the attached instance's owning account on the ENI (InstanceOwnerId,
mirroring AWS Attachment.InstanceOwnerId) and resolve the instance and every
IAM/STS lookup under it. The LB ENI lives in the customer account while the
system LB VM and its role live in the system account; keying the whole IMDS
chain off the ENI account left the instance invisible and the role lookup
missing. VPC leaves (subnet/VPC/SG CIDR) keep using the ENI account.
A fresh LB on env19 then authenticated end-to-end: IMDS serves a non-empty role
list plus ASIA creds, and the gateway accepts the heartbeat as an assumed-role
principal with no static keys in user-data.
* test(e2e): assert LB VM authenticates via IMDS instance-role creds
The LB suite's targets-healthy assertion already depends on the lb-agent
authenticating to the gateway, and with static keys dropped from user-data that
auth is IMDS-only. Add an explicit check that a recent gateway authentication
for the LB used an assumed-role (ASIA) principal, so a regression to the
instance-account IMDS resolution fails with a clear message instead of a
mysterious datapath timeout. The LB VM is a system-account instance invisible to
the customer AWS API, so the assertion observes the gateway journal on the
cluster nodes; it is time-bounded to avoid the stale-console-log false-fail and
skips when no node is SSH-reachable.
Cell 19 bootstraps with --external-mode=nat plus the CI static pool as
the public wan pool and runs the natuplink suite, covering the Tier-1
transit datapath and Tier-2 EIP ingress. The run step exits 0 when
natuplink.test is absent from the tests tar so scheduled runs against
branches predating the suite stay green.
Add a second build+publish step to the EKS AMI workflow so the same
manual run also builds the ecs-agent system AMI and uploads its qcow2 +
.sha256 sidecar to R2. Reuses the shared checkout, Go, and tooling setup;
renames the workflow to cover both node images.
Flip the k3s server apiserver to anonymous-auth=false. The only
consumer of the anonymous path was mulga-eks-state-report's /healthz
probe; switch it to an authenticated kubectl get --raw=/healthz through
the node admin kubeconfig (an unauthenticated curl now returns 401).
The NLB target group uses a TCP health check, and the host-side
reconciler rides the authenticated NATS state-report, so neither
depends on the anonymous apiserver path.
* fix: free host-netns :80 for ECS bridge/host tasks via cred-endpoint DNAT
bridge and host network-mode tasks share the VM host netns where the
ecs-agent credential endpoint bound 169.254.170.2:80, so a task binding
0.0.0.0:80 hit EADDRINUSE and crash-looped. Bind the listener on the high
proxy port 51679 and DNAT 169.254.170.2:80 to it (OUTPUT for host/bridge
task traffic, PREROUTING for the awsvpc credential veth), keeping the dst
IP on the endpoint so no route_localnet is needed. Nothing holds a socket
on :80, so host/bridge tasks can bind it.
e2e runs nginx on :80 in all three network modes; the settle guard catches
a regression where a crash-looping :80 bind flaps RUNNING then STOPPED.
* test: validate ECS host-netns :80 binding via bridge task
The bridge subtest now runs nginx on :80 to prove the cred-endpoint DNAT
frees the host netns (pre-fix it crash-looped on EADDRINUSE). host mode
keeps a no-port command: it shares the same host netns, so two :80 tasks
cannot coexist on one node (one :80 per host, as on AWS), and StopTask
does not synchronously reap the prior container's port.
When NATS disables JetStream at runtime (e.g. a filesystem permission error
flushing stream state) it keeps the process alive and serves KV as read-only.
That is not a process failure, so spinifex-nats.service's Restart=on-failure
never fires, and KV-backed startup (awsgw IAM bucket init) stalls for minutes
until an operator restarts NATS by hand.
Add a JetStream liveness watchdog:
- build/scripts/nats-js-watchdog.sh probes the NATS monitoring endpoint
(http://127.0.0.1:8222/healthz?js-enabled-only=true). Only when the service
is active and JS fails every sample of a short multi-sample window (so
transient startup/quorum 503s are ignored) does it systemctl try-restart
spinifex-nats.
- spinifex-nats-watchdog.{service,timer} run it as a oneshot every 30s, with a
60s boot delay so cluster quorum settles first.
- spinifex.target Wants= the timer so it starts with the platform.
The NATS-internal permission race itself is a server quirk outside spinifex's
control; this restores service automatically when it recurs.
The in-cluster AWS LoadBalancer Controller provisions its own ALBs
(k8s-toc-*) and security groups (k8s-traffic-toc-*) in the customer VPC
via the awsgw ELBv2/EC2 APIs. Spinifex never tracks these, so cluster
teardown left them orphaned: DeleteSecurityGroup / DeleteVpc then failed
with DependencyViolation and the cluster wedged in DELETING.
purgeClusterInfra now reaps them by the AWS-standard ownership tag
elbv2.k8s.aws/cluster=<name>, scoped to the customer VPC, after the
cluster NLB teardown and before the SG / VPC block:
- ReapLBCLoadBalancers lists account LBs, filters to the customer VPC,
confirms the ownership tag via DescribeTags, and deletes by ARN.
- DeleteClusterSGs deletes any VPC SG carrying the ownership tag (reusing
the DependencyViolation retry budget), in addition to the named cluster
SGs, after rules are revoked.
- DescribeTags is added to the narrow nlbProvisioner interface.
Errors join teardownErrs so a failed reap keeps the cluster in DELETING
for a retry rather than completing with leaked, VPC-pinning infra.
TerminateWorkerInstances routes terminates via the per-instance ec2.cmd.<id>
topic, whose subscription is torn down when a VM stops. A stopped (incl.
stopped+wedged, volume remount broken) worker therefore has no responder, and
the old code treated ErrNoResponders as already-gone and returned nil. The
worker was still in shared KV with its ENI intact, so the nodegroup record was
deleted while the ENI kept pinning the customer subnet/VPC, wedging DeleteCluster
in DELETING on DependencyViolation until a manual ec2 terminate-instances.
On ErrNoResponders, fall back to the ec2.terminate queue group (mirroring the
gateway TerminateInstances path), which runs TerminateStoppedInstance against
shared KV — deleting the worker's volumes, IP, and ENI without a running QEMU.
NotFound or no responder is idempotent success; any other error surfaces so the
teardown backstop retries.
TerminateInstances marks the EC2 record terminated and drives node-local
teardown, but when that teardown runs on a node other than the one hosting
QEMU (per-instance subscription lost, request falls back to the ec2.terminate
broadcast), the process survives and keeps holding its OVN logical port / IP,
blocking rebuilds.
Add OrphanQEMUReaper (node-local, ScopeNodeLocal): it inspects only the
terminated KV bucket and reaps a live process found via the instance's local
PID file. Running, pending, and stopped instances are never touched. Whichever
node holds the orphan PID file is the one that reaps it; every other node's
lookup misses and is a no-op.
Add utils.ProcessAlive (signal-0 liveness) and utils.ForceKillProcess
(SIGKILL + bounded wait), guard shutdownAndUnmount's force-kill on liveness,
and register the reaper in the daemon GC loop.
reconnectInstance re-asserted the instance running state but never the boot
volume's in-use state, so after a daemon restart describe-instances showed the
VM running while describe-volumes showed its root volume available — a
split-brain that makes start-instances attempt to double-launch a live QEMU.
Extract the launch path's boot-volume promotion into markBootVolumesInUse and
call it from reconnectInstance so launch and recovery keep volume state
consistent. Add a reconnect unit test asserting the boot volume is marked
in-use and non-boot volumes are left to their own attach flow.
The daemon-side in-use gate trusts persisted volume State/AttachedInstance,
which the attach/detach lifecycle can leave desynced: a running root volume
was observed with State:"" and AttachedInstance:"" while the instance's
BlockDeviceMappings still listed it, so DeleteVolume passed the gate and
removed the durable backing under a live QEMU.
Add an authoritative guard at the gateway DeleteVolume path: fan out the
strict DescribeInstances survey and refuse with VolumeInUse when any
non-terminated instance's BlockDeviceMappings reference the volume. Fail
closed on an incomplete cluster view rather than delete against a partial
survey. Terminate-time cleanup uses the daemon volume service directly and is
unaffected.
Drives the ECS data plane against the local awsgw: a customer VPC plus a real
container instance launched from the spinifex-ecs-node AMI, then standalone
tasks in all three network modes and an awsvpc service fronted by an ALB target
group. Adds an ECS client to the harness, a Makefile target, and a
service-interfaces suite entry.
bridge/host tasks share the host netns where the agent credential endpoint owns
169.254.170.2:80, so they run a no-port command and assert placement + mode;
awsvpc and the ELB service prove real :80 serving. waitTaskRunning re-checks
after a settle so a transient RUNNING cannot pass.
Wire crash detection and host-resource signals through to the AWS-faithful
DescribeInstanceStatus API and the spx CLI, closing out the EC2 health
monitoring / self-healing work.
Phase C — QMP health monitoring:
- qmpHeartbeat now tracks consecutive query-status failures; at
QMPMaxConsecutiveFailures it checks process liveness, triggering HandleCrash
on a dead QEMU and marking an alive-but-wedged process impaired.
- InstanceHealthState gains QMPConsecutiveFailures, LastQMPSuccess, ImpairedSince.
Phase E — DescribeInstanceStatus mapping:
- buildInstanceStatus maps running VMs to ok / impaired (QMP-unresponsive,
with ImpairedSince) / initializing (2-min launch grace), non-running to
not-applicable.
- SystemStatus reflects node memory pressure via ResourceManager
HostUnderMemoryPressure (live MemAvailable below the host reserve).
Phase C2 — observability:
- spx get vms gains HEALTH and CRASHES columns derived from vmHealthLabel.
Tests cover crash/QMP helpers, status mapping, memory-pressure system status,
and the health label.
On startup the agent had no reconcile pass: Run registered, heartbeat, and
immediately polled assignments with an in-memory dedup map. After a restart the
gateway re-delivers the instance's assignments and the agent re-ran them,
orphaning containers still running from before and double-starting tasks.
Add a reconcile() pass that runs once before polling: it lists the runtime's
containers, and for each task still running in this cluster re-registers its IAM
credentials, re-attaches the exit-wait, and refreshes RUNNING on the gateway.
The adopted task set seeds pollAssignments so re-delivered assignments are acked
but not re-run.
- runtime: add a label-agnostic List(ctx) []Container to the Runner seam, with a
containerd implementation and fake support.
- taskLabels: stamp credID / taskRoleArn / eniMac (when set) so a container is
self-describing enough to re-register creds and tear down its netns on adoption.
- pollAssignments takes the seed dedup map.
Best-effort: a nil runner or a List error degrades to today's poll+run behaviour.
* docs: add ECS (Elastic Container Service) guide + register ecs-quickstart workbook
Add a Container Services -> ECS doc mirroring the EKS/ECR shape: concepts
(cluster, task definition, task, service, container instances via the
ECS-AMI EC2 launch type), an aws ecs / console / Terraform quickstart,
task IAM roles via the credential endpoint, awsvpc networking, and the
honest v1 limitations (no service discovery, awslogs discarded, no rolling
deployments, no capacity providers, no distinct execution role).
Convert the ecs-quickstart workbook README to the doc-page format
(frontmatter + overview/prerequisites/instructions/troubleshooting) so it
renders as a Terraform Workbooks entry alongside the EKS workbooks.
* docs: add node-image prerequisite banner + infra diagram to EKS and ECS pages
Add a GitHub-style [!IMPORTANT] banner to the EKS and ECS prerequisites
flagging the required node-image import (eks-node / spinifex-ecs-node),
mirroring the install page's WAN-bridge banner, each with a describe-images
verification command.
Add a basic Spinifex topology diagram (SVG) to each Overview, in the
tf-nginx-alb visual style: the EKS managed control plane + node-group
workers, and the ECS ALB + container instances with the gateway/agent
and task-credential path.
End-to-end ECS stack against the Spinifex gateway: cluster, task definition
with a task role, container instances from the spinifex-ecs-node AMI, and an
awsvpc service fronted by an ALB target group. Unlike the console's provision-
capacity action, the workbook spells out the two things a Spinifex container
instance needs that AWS hides: a LAN-reachable gateway URL and the gateway CA,
both seeded into cloud-init user-data; instance creds come from IMDS via the
ecsInstanceRole profile. Live-verified apply (HTTP 200 through the ALB) and
destroy round-trip.
The agent heartbeats by periodically re-registering through the gateway
(RegisterContainerInstance). After a control-plane restart, the heartbeat
reaper marks instances DRAINING for the missed-heartbeat window, but the
re-register path only refreshed LastSeen and never restored the status, so a
reaped instance stayed DRAINING forever even though its agent was healthy and
re-registering.
Mark reaper-induced DRAINING with a Reaped flag and restore the instance to
ACTIVE on the next re-register, clearing the flag. An operator
UpdateContainerInstancesState drain clears Reaped, so an intentional drain is
never undone by the live agent.
* feat(ecs): provision EC2 container-instance capacity via IMDS instance-role
Adds a ProvisionCapacity gateway action that launches spinifex-ecs-node
EC2 instances which register as ECS container instances, fixing
RESOURCE:placement when a cluster has no capacity.
Agent control-plane now SigV4-signs gateway calls with instance-role
credentials read from IMDS (no static keys seeded in user-data). The
gateway already verifies STS/temp creds, so the instance role carries an
ecs:* policy + ec2 trust; ensureECSInstanceProfile find-or-creates the
ecsInstanceRole instance profile attached at launch.
Backend resolves the spinifex-ecs-node AMI by spinifex:managed-by=ecs
tag, renders keyless user-data, and launches via the customer
RunInstances path. Frontend adds a hand-signed JSON-1.1 ProvisionCapacity
call, a provision-capacity dialog on the container-instances tab, and the
ECS console section nav + system-image gating.
* feat: auto-assign EIPs to awsvpc service tasks; add ECS task/service detail pages
Wire AssignPublicIp=ENABLED through the service reconciler so awsvpc service
tasks automatically allocate and associate an Elastic IP on the RUNNING
transition, and release it on STOP. assignTaskPublicIP/releaseTaskPublicIP
mirror the target-group registration hooks in recordTaskState and forceStopTask;
the public IP and allocation ID are persisted on the task record and surfaced in
DescribeTasks attachments as publicIPv4Address.
Add AWS-parity ECS task and service detail pages showing private and public
endpoints, container state, and networking configuration, reachable by clicking
a task or service in the cluster tabs.
* test: cover natsEIPManager allocate/associate/release over NATS
ListTaskDefinitions appended every revision regardless of the requested
status, so deregistered (INACTIVE) task definitions never dropped off the
default listing and the console could not clear them. Filter by status,
defaulting to ACTIVE to match AWS.
Add an ECS console mirroring the EKS/ECR route shape: a single ECS nav
entry, cluster list and detail with Services/Tasks/Task Definitions/
Container Instances tabs, plus create and teardown paths.
Adds three full-page wizards to drive container deploys end-to-end:
Register Task Definition, Run Task, and Create Service. RunTask surfaces
a 200-with-failures placement result as an error instead of silently
navigating away.
Includes the @aws-sdk/client-ecs client singleton, ecs queries and
mutations, zod form schemas, and co-located tests.
STOPPED task records were never pruned: ListTasks enumerated every task a
cluster ever ran and the per-account KV grew unbounded. Add a leader-gated sweep
tick to the scheduler cron (alongside the reaper/reconcile) that deletes task
records STOPPED longer than a 1h retention window — keeping a just-stopped task
describable for DescribeTasks / UI exit reason, matching AWS's STOPPED window.
A task missing its StoppedAt timestamp is never pruned (defensive). Single-writer
(leader only), reuses the reap() bucket walk.
Wire the four NotImplemented ECS teardown actions so an IaC stack and the
console can reverse a deployment:
- DeleteCluster cascades: force-stops every task (releasing ENIs, deregistering
TG targets, returning capacity), then sweeps the cluster's whole KV subtree.
Unlike AWS it cascades rather than rejecting a non-empty cluster, so terraform
destroy round-trips regardless of teardown ordering.
- DeregisterTaskDefinition marks a specific family:revision INACTIVE (kept
describable); a bare family is rejected, matching AWS.
- DeregisterContainerInstance removes an instance, force-stopping its tasks when
Force is set and rejecting a non-empty instance otherwise.
- UpdateContainerInstancesState sets ACTIVE/DRAINING; draining force-stops the
instance's service-owned tasks so the reconciler relaunches them elsewhere.
Reuses the existing forceStopTask primitive and the reaper drain pattern; adds a
deleteKeysWithPrefix KV helper and INACTIVE status constants.
Serve task-role credentials to containers the AWS-faithful way: an HTTP
endpoint at 169.254.170.2:80 inside the ecs-agent (host VM netns) serving
short-lived credentials at /v2/credentials/{credID}. Containers find it via
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI, injected per container.
The agent mints credentials by SigV4-signing sts:AssumeRole(taskRoleARN)
against the gateway with its instance-role credentials, mirroring
internal/ecrauth. This reconciles ecs-v1 Q4/Q10 (STS over NATS + scheduler
JWT assertion) with the 4f rewire that moved the agent entirely onto the
gateway: the agent has no NATS connection and already holds instance creds,
so it assumes the task role directly. Credentials are cached per credID and
refreshed near expiry.
Each awsvpc task netns gets a credentials veth into the host netns and a
host-routed /32 to the endpoint, keeping it off the task ENI's VPC route.
The scheduler now persists taskRoleArn on the task definition and carries it
in the assign payload so the endpoint activates end to end.
* fix: serialize EBS hot-plug per instance to prevent PCIe port race
Concurrent AttachVolume calls to one instance both allocated the lowest free
hot-plug PCIe root port via nextHotplugEBSPort (which reads live QEMU state),
then issued QMP device_add on the same hotplug-ebs bus. The second device_add
failed with 'PCI: slot 0 function 0 already occupied by virtio-blk-pci',
surfacing as nats:timeout -> ServerInternal and a CSI retry-storm
('retry quota exceeded'); volumes never attached.
Add a per-instance attachMu and hold it across the whole AttachVolume and
DetachVolume critical sections, mirroring the ENI hot-plug serialization.
The mutex is non-persisted and shared via the Manager's stable *VM.
* fix: allocate EBS hot-plug PCIe port from in-memory accounting
AttachVolume derived the free PCIe hot-plug root port from a live
query-block scan (nextHotplugEBSPort), extracting the bus with a
hotplug-ebs(\d+) regex on each device's QDev. Real QEMU reports a
hot-plugged blockdev by id (/machine/peripheral/vdisk-vol-xxx/
virtio-backend), with no hotplug-ebs component, so the regex never
matched, the used set stayed empty, and the allocator always returned
port 1. Only the first EBS volume per instance attached; every later
attach collided on "PCI: slot 0 function 0 already occupied by
virtio-blk-pci", rolling back the blockdev, killing NBDKit, and
surfacing as nats: timeout / ServerInternal to the CSI driver, which
then exhausted its retry quota.
Allocate the port from authoritative in-memory accounting instead:
record the chosen port as EBSRequest.HotplugPort (persisted, so it
survives a daemon restart) and pick the lowest unclaimed port with
freeHotplugEBSPort over the instance's recorded requests. Hold
attachMu across allocate->device_add so concurrent CSI attaches cannot
pick the same port. DetachVolume already removes the request, freeing
the port. Drops the query-block round-trip from the attach path.
* feat(ecsgw): on-VM SigV4 client for the ECS agent→gateway channel
Adds internal/ecsgw, the twin of internal/eksgw: a TLS + SigV4 (service "ecs")
client the ecs-agent will use to reach the AWS gateway over HTTPS instead of
connecting to host NATS directly. Calls are AWS JSON 1.1 with an X-Amz-Target
action; the gateway relays onto ecs.* host-side, keeping NATS cluster-internal.
* feat(ecs): SubmitTaskStateChange gateway action → recordTaskState
Wires the AWS-API SubmitTaskStateChange action end to end (gateway action →
ecs.SubmitTaskStateChange NATS → daemon → Service.SubmitTaskStateChange) so a
gateway-routed agent reports task state without touching the NATS bus. The
handler maps the SDK input onto the same bus.TaskState shape and converges on
recordTaskState; the account is authoritative from the SigV4 caller, so an
instance cannot report state for another account's task.
* feat(ecs): durable assign KV inbox + PollAssignments gateway action
Scheduler writes task assigns into a per-instance KV inbox
(clusters/{cluster}/assignments/{instance}/{task}) instead of a
fire-and-forget core-NATS publish, and a new PollAssignments gateway
action lets a gateway-routed agent drain its inbox over TLS+SigV4.
At-least-once: an unacked assign is re-delivered on the next poll;
the agent acks accepted assigns (AckTaskIDs) and STOPPED transitions
reclaim stale inbox entries. The Poll response carries bus.Assign with
an RFC3339 time, so it is encoded with encoding/json (RawJSONActions)
rather than the jsonutil epoch-second marshaler.
* feat(ecs-agent): rewire control-plane off NATS onto the AWS gateway
The agent no longer connects to NATS. It registers, heartbeats, reports
task state, and receives assignments entirely through the AWS gateway
over TLS+SigV4, using seeded IAM credentials — keeping the NATS bus
host-internal and closing the trust-boundary gap (a tenant container
instance must not hold a shared bus token).
- controlPlane seam over ecsgw.Client replaces the NATS publisher:
Register (RegisterContainerInstance), SubmitTaskState
(SubmitTaskStateChange), PollAssignments.
- Heartbeat folds into a periodic idempotent re-register (refreshes
LastSeen); bus.Heartbeat is dropped from the agent.
- subscribeAssign (NATS subscribe) becomes a poll loop: drain the
inbox, dispatch each assign once, ack accepted taskIDs on the next
poll (at-least-once).
- config: ECS_ACCESS_KEY / ECS_SECRET_KEY / ECS_POLL_INTERVAL added,
ECS_NATS_URL removed; cloud-init initd seeds the IAM creds.
Tests move from the NATS publisher fake to a controlPlane fake.
* feat(ecs): cloud-init cred-seed + ecs-node bring-up script
ECS container instances are plain EC2 booted from the spinifex-ecs-node
AMI (AWS-faithful: no ECS-specific launch API). This renders the
cloud-init user-data that seeds /etc/spinifex-ecs/agent.env (gateway
URL/CA/region/cluster + IAM creds) and the gateway CA, then runs the
instance. The agent registers, heartbeats, polls assignments, and
reports task state entirely through the AWS gateway over TLS+SigV4 —
never NATS.
The guest dials the customer-facing gateway address (network advertise
IP), not the mgmt bridge, mirroring the EKS GatewayURL/AddonGatewayURL
split. Seeds the system key for now; scoped per-instance creds are a
follow-up (4g task-role endpoint).
* fix(ecs): reconcile dev merge — restore #418 ENI release + netns wiring
The 'Merge branch dev' resolution dropped code where Sprint 4d (#418,
awsvpc ENI-per-task) and the gateway control-plane rewrite (#419) touch
the same paths:
- recordTaskState STOPPED branch: reclaimTaskENI was clobbered by
reclaimAssignInbox; both are needed (release the task ENI AND drain the
KV inbox). Restores TestRecordTaskState_Awsvpc_ReleasesENIOnStopOnce.
- ecs-agent Agent: the netns field, its init, and the setupTaskNetns/
teardownTaskNetns wrappers were lost when the controlPlane Agent struct
won the merge, leaving taskrun.go calling undefined methods.
- eni_test assign assertion read the assign over NATS; #419 delivers it via
the durable KV inbox, so it now polls via PollAssignments.
- taskrun_test netns cases used the removed historyPub fake; switched to
fakeCP to match the controlPlane seam.
* feat(ecs): service controller + ELBv2 target-group registration (Sprint 4f)
Land the ECS Service controller on top of the 4e scheduler/RunTask flow:
- CreateService/UpdateService/DeleteService/DescribeServices/ListServices.
REPLICA only; DAEMON and serviceRegistries rejected in v1 (Q15).
- StartTask (explicit placement) and StopTask.
- Leader-driven replica reconciliation: a 10s scheduler tick converges each
service's running task count on desiredCount, launching the shortfall via
RunTask (tasks tagged group=service:{name}) and force-stopping the surplus.
- ELBv2 target-group registration: the scheduler registers a task's awsvpc ENI
IP on the RUNNING transition and deregisters on STOPPED via the existing
elbv2.RegisterTargets/DeregisterTargets NATS handlers (Q8, single-writer).
- forceStopTask consolidates the control-plane stop (STOPPED state, capacity +
ENI release, inbox drain, TG deregister); reused by StopTask, scale-in, and
the heartbeat reaper (the reaper now deregisters targets per Q13).
- Wire the 7 new actions through the gateway action map, NATS adapter, and
daemon subscriptions; add ServiceNotFoundException.
* fix(ecs-agent): anonymous pull for non-ECR images + normalize image refs
The agent minted an ECR authorization token via IMDS for every image, so
public docker.io refs failed with 'IMDS returned empty role name' on hosts
without an instance role. Gate ECR auth on .dkr.ecr. refs and pull all other
registries anonymously.
Bare image strings (nginx:alpine) were handed to containerd unmodified, which
parsed them as dummy://nginx:alpine and rejected the pull. Normalize refs to
their fully-qualified form (docker.io/library/nginx:alpine, :latest default)
before Pull and GetImage.
Adds StartTask/StopTask service-controller coverage.
* fix: address ECS 4f live-verification findings
ecs-node-bringup.sh defaulted GATEWAY_IP to the mgmt bridge (10.15.8.1), which
a VPC guest cannot route to — the agent could not reach the gateway. Default to
the host's WAN-bridge IP, which guests reach by NATing out through the WAN.
A target group not forwarded to by any listener reported its targets stuck in
'initial' forever (no health checker advances a detached TG). DescribeTargetHealth
now reports such targets as 'unused' with reason Target.NotInUse, matching AWS.
* feat(ecsgw): on-VM SigV4 client for the ECS agent→gateway channel
Adds internal/ecsgw, the twin of internal/eksgw: a TLS + SigV4 (service "ecs")
client the ecs-agent will use to reach the AWS gateway over HTTPS instead of
connecting to host NATS directly. Calls are AWS JSON 1.1 with an X-Amz-Target
action; the gateway relays onto ecs.* host-side, keeping NATS cluster-internal.
* feat(ecs): SubmitTaskStateChange gateway action → recordTaskState
Wires the AWS-API SubmitTaskStateChange action end to end (gateway action →
ecs.SubmitTaskStateChange NATS → daemon → Service.SubmitTaskStateChange) so a
gateway-routed agent reports task state without touching the NATS bus. The
handler maps the SDK input onto the same bus.TaskState shape and converges on
recordTaskState; the account is authoritative from the SigV4 caller, so an
instance cannot report state for another account's task.
* feat(ecs): durable assign KV inbox + PollAssignments gateway action
Scheduler writes task assigns into a per-instance KV inbox
(clusters/{cluster}/assignments/{instance}/{task}) instead of a
fire-and-forget core-NATS publish, and a new PollAssignments gateway
action lets a gateway-routed agent drain its inbox over TLS+SigV4.
At-least-once: an unacked assign is re-delivered on the next poll;
the agent acks accepted assigns (AckTaskIDs) and STOPPED transitions
reclaim stale inbox entries. The Poll response carries bus.Assign with
an RFC3339 time, so it is encoded with encoding/json (RawJSONActions)
rather than the jsonutil epoch-second marshaler.
* feat(ecs-agent): rewire control-plane off NATS onto the AWS gateway
The agent no longer connects to NATS. It registers, heartbeats, reports
task state, and receives assignments entirely through the AWS gateway
over TLS+SigV4, using seeded IAM credentials — keeping the NATS bus
host-internal and closing the trust-boundary gap (a tenant container
instance must not hold a shared bus token).
- controlPlane seam over ecsgw.Client replaces the NATS publisher:
Register (RegisterContainerInstance), SubmitTaskState
(SubmitTaskStateChange), PollAssignments.
- Heartbeat folds into a periodic idempotent re-register (refreshes
LastSeen); bus.Heartbeat is dropped from the agent.
- subscribeAssign (NATS subscribe) becomes a poll loop: drain the
inbox, dispatch each assign once, ack accepted taskIDs on the next
poll (at-least-once).
- config: ECS_ACCESS_KEY / ECS_SECRET_KEY / ECS_POLL_INTERVAL added,
ECS_NATS_URL removed; cloud-init initd seeds the IAM creds.
Tests move from the NATS publisher fake to a controlPlane fake.
* feat(ecs): cloud-init cred-seed + ecs-node bring-up script
ECS container instances are plain EC2 booted from the spinifex-ecs-node
AMI (AWS-faithful: no ECS-specific launch API). This renders the
cloud-init user-data that seeds /etc/spinifex-ecs/agent.env (gateway
URL/CA/region/cluster + IAM creds) and the gateway CA, then runs the
instance. The agent registers, heartbeats, polls assignments, and
reports task state entirely through the AWS gateway over TLS+SigV4 —
never NATS.
The guest dials the customer-facing gateway address (network advertise
IP), not the mgmt bridge, mirroring the EKS GatewayURL/AddonGatewayURL
split. Seeds the system key for now; scoped per-instance creds are a
follow-up (4g task-role endpoint).
* fix(ecs): reconcile dev merge — restore #418 ENI release + netns wiring
The 'Merge branch dev' resolution dropped code where Sprint 4d (#418,
awsvpc ENI-per-task) and the gateway control-plane rewrite (#419) touch
the same paths:
- recordTaskState STOPPED branch: reclaimTaskENI was clobbered by
reclaimAssignInbox; both are needed (release the task ENI AND drain the
KV inbox). Restores TestRecordTaskState_Awsvpc_ReleasesENIOnStopOnce.
- ecs-agent Agent: the netns field, its init, and the setupTaskNetns/
teardownTaskNetns wrappers were lost when the controlPlane Agent struct
won the merge, leaving taskrun.go calling undefined methods.
- eni_test assign assertion read the assign over NATS; #419 delivers it via
the durable KV inbox, so it now polls via PollAssignments.
- taskrun_test netns cases used the removed historyPub fake; switched to
fakeCP to match the controlPlane seam.
* feat(ecs): awsvpc ENI-per-task control plane (Sprint 4d)
Scheduler owns the task-ENI lifecycle for awsvpc mode: allocate from the
task subnet on placement, hot-plug onto the container instance, plumb ENI
identity through the assign payload + task record, and detach+delete on
STOPPED (graceful + reaper, idempotent leak guard). bridge/host unchanged.
* feat(ecs-agent): awsvpc guest-side task netns from hot-plugged ENI
The agent builds a per-task network namespace for awsvpc tasks: it finds the
hot-plugged ENI NIC by MAC, moves it into /var/run/netns/ecs-<taskID>, and DHCPs
the OVN-assigned ENI IP. The container then joins that netns via the OCI network
namespace path; bridge/host tasks keep the host (VM) netns as before.
* build(ecs-node): add iproute2 for awsvpc task netns
The ecs-agent builds awsvpc task network namespaces with `ip netns`, which
busybox's `ip` applet does not implement. Install iproute2 in the ecs-node AMI;
udhcpc (for the in-netns ENI DHCP lease) is already provided by busybox.
PR #411 (cloud-init EC2 datasource) dropped the cloud-init ISO that
created the ec2-user account and baked the harness SSH key; guests now
boot the stock Ubuntu gold AMI (default user ubuntu) bootstrapped via the
Ec2/IMDS datasource. eni_reconcile_test.go was the lone single-suite test
still SSHing as ec2-user, so TestENIHotplugReconcile failed publickey at
the first SSH. Align it with every other single-suite test.
* feat(eni): hot-plug reconciler + orphan sweep (Sprint 3d)
Add eniReconciler, a node-local vm.Reaper that converges ENI hot-plug
state across a daemon restart or crash: KV ENIRecord <-> in-memory PCIe
slot map <-> guest query-pci device list. Resolves records stuck in
attaching/detaching, re-adopts the slot map after restart, and reaps
orphan QMP devices and slot entries.
A staleness gate (new ENIRecord.AttachmentStateAt + 30s threshold) keeps
the reconciler off records a live attach/detach handler is mid-pipeline
on. Wired into the existing GarbageCollector backstop.
* test(eni): unit-cover hot-plug reconciler vm helpers
* test(eni): e2e reconciler convergence across daemon restart (Sprint 3e)
StopNode now stops spinifex-nats.service, so WaitNATSPeers polling that
node's dead 127.0.0.1:8222 monitor port fails with curl exit 7 and never
converges. Add WithSkipNodes to exclude the stopped node from the poll;
TestMultinodeNodeFailure skips node2.
Add the ECS v1 scheduler control loop and end-to-end RunTask flow:
- handlers/ecs: per-account KV-backed Service (clusters, task definitions,
container instances, tasks), bin-pack/spread/random placement with CAS
capacity reservation, and a leader-elected Scheduler that owns the Layer-2
bus subscriptions (register/heartbeat/task-state) plus a heartbeat reaper.
- gateway/ecs: wire CreateCluster, Describe/ListClusters,
Register/Describe/ListTaskDefinitions, Register/Describe/ListContainerInstances,
RunTask, Describe/ListTasks onto the daemon over NATS.
- daemon: construct the Service, start the Scheduler goroutine, subscribe the
ECS action subjects.
- ecs-agent: subscribe the per-instance assign subject, pull and run each
container via containerd, and report task state back on the bus.
Adds the spinifex-ecs-node AMI variant: alpine 3.21.7 + containerd + runc +
cni-plugins + ca-certificates with the ecs-agent binary baked to
/usr/local/bin and three OpenRC services wired in dependency order
(ecs-ca-install -> containerd -> ecs-agent).
- manifest.conf drives build-system-image.sh (no host mounts).
- ecs-ca-install.initd: first-boot oneshot installing the cloud-init
delivered gateway CA into the system trust; no-op when absent.
- ecs-agent.initd: sources /etc/spinifex-ecs/agent.env, backgrounds the
agent after net + containerd.
- 10-bridge.conflist: CNI bridge for task networking (Sprint 4d).
- setup.sh: exec bits, /opt/cni/bin symlink, content-store dirs, ttyS0
console reorder + fast boot menu.
- Makefile: build-ecs-node-image / import-ecs-node-image targets.
* feat(ecs): ecs-agent skeleton + containerd image pull (Sprint 4c)
Add the ecs-agent binary that runs inside an ECS container instance: it
resolves host identity from IMDS, registers on the Layer-2 NATS bus, and
heartbeats while alive. Image pulls go through a containerd-backed
ImagePuller behind an interface, with an ECR resolver that mints
GetAuthorizationToken creds from the instance role.
Extract the IMDS-credential and ECR-auth flow out of
ecr-credential-provider into internal/imdscreds and internal/ecrauth so
the agent and the kubelet provider share one implementation.
Add spinifex/handlers/ecs/bus as the single source of truth for Layer-2
subject builders and wire payloads, shared by the agent now and the
scheduler in Sprint 4e.
* build: pin genproto via replace to fix containerd/v2 ambiguous import
go.opencensus.io -> grpc@v1.33.2 drags the pre-split monolithic genproto
into the module graph; it still ships googleapis/rpc/status, which collides
with the split genproto/googleapis/rpc module that containerd/ttrpc needs.
The main module never imports the monolith, so a plain require bump is
dropped by 'go mod tidy'. Pin it with a replace to a post-split version so
the fix survives tidy and contributors can run 'go mod tidy' freely.
* fix(ecs): agent boot failsafes from runtime verify
- Build the ECR gateway HTTP client lazily on first image pull instead of
at boot. A missing/malformed gateway CA no longer kills the agent before
it can register + heartbeat, matching the best-effort containerd path.
- Actively probe containerd with IsServing in runtime.New. The client dials
lazily, so a dead socket previously slipped past the boot-time
'containerd unavailable' warn and only surfaced on the first pull.
- Make the heartbeat interval configurable via ECS_HEARTBEAT_INTERVAL
(default 30s) for test/verify and slow-link tuning.
* fix: drop removed predastore s3.Config.Compaction from admin_test
The predastore submodule bump removed the Compaction field from s3.Config.
Parse the rendered [compaction] block into a local struct so the template
round-trip test no longer references the deleted field.
* feat: graceful guest drain on systemctl stop / host shutdown
systemctl stop spinifex.target tore down nbdkit/viperblock while guest QEMU
was still writing, corrupting guest filesystems. The ordered drain existed
only over NATS via cluster shutdown, never on the systemd/host path.
Add spx admin node drain --local (GATE+DRAIN against the local node, leaving
STORAGE/PERSIST/INFRA to systemd), wired as the ExecStop of a new
spinifex-shutdown.service oneshot ordered After= all services so it stops
first and drains guests while every service is still up. DDIL preserved: the
oneshot fires only on target/host shutdown, not on a daemon restart.
Harden the viperblockd SIGTERM handler to flush the WAL but skip reaping an
nbdkit a guest is still attached to. Add a units ordering invariant.
* fix: ExecStop drain timeout needs duration unit (120 -> 120s)
The spinifex-shutdown.service ExecStop passed --timeout=120, but spx admin
node drain's --timeout is a cobra Duration flag and rejects a bare integer
(time: missing unit in duration "120"). The drain exited 1 without running,
so a systemctl stop tore nbdkit out from under the running guest and corrupted
its ext4 (the exact failure this feature prevents). Pass 120s.
* fix: nbdkitInUse read ss state from correct column + cover it
ss -H rows are '<netid> <state> ...', so the state is field[1], not field[0]
(field[0] is 'u_str'). The old index never matched ESTAB, so an in-use nbdkit
read as idle and the SIGTERM path would reap it under a live guest. Add tests
exercising a real connected unix socket, an absent socket, and TCP transport.
* feat: add predastore compaction E2E
Inject a short compaction interval into test-cluster predastore config via a
new --predastore-compaction-interval flag on admin init/join, gated so
production output stays byte-identical when unset. Add a harness helper that
drives write-then-delete churn against predastore S3, brackets it with
du-over-SSH usage, and asserts end-of-churn usage settles below 70% of the
churn peak; wire it into the multinode suite as TestPredastoreCompaction.
* fix: compaction E2E gate writes churn to its own bucket
The reclaim gate signed PutObject against the config-defined system
'predastore' bucket, which belongs to account 000000000000 with only a
static [[auth]] grant. The harness admin identity is a different IAM
account, so predastore returned 403 AccessDenied cross-account.
Create a per-run bucket the admin identity owns (AdministratorAccess)
and churn into it. Objects still erasure-code into distributed/nodes/
node-* so the du-based reclaim measurement is unchanged. The gateway
(:9999) does not proxy S3, so the client stays predastore-direct.
* test: run predastore compaction gate before destructive node tests
The compaction reclaim gate was declared last in the multinode suite, so it
measured shard-store usage after NodeFailure/NodeRecovery had torn down and
restarted node2. NodeRecovery only gates on NATS/gateway/EC2 readiness, not
predastore, so node2's shard store was still settling and the du baseline came
back empty. Move the test before the destructive node tests so it runs against
a pristine, fully-converged cluster primed by the volume tests.
* fix: race-free SSH output capture + empty-tolerant du probe
The compaction reclaim gate crashed with ParseInt("") when node1's du
probe returned empty combined output. Harden the probe and the SSH layer:
- ssh.Run used one bytes.Buffer for both session.Stdout and session.Stderr,
which x/crypto/ssh copies in two concurrent goroutines — a data race on a
non-concurrent-safe type. Switch to session.CombinedOutput, which guards the
combined stream with a mutex-wrapped singleWriter.
- shardStoreUsageBytes now terminates the remote command with a sentinel echo.
A missing sentinel is reported as output lost in SSH retrieval; a present
sentinel with no number is treated as a 0 measurement (anomalous, logged)
instead of crashing the gate.
The sentinel lets the next run distinguish a transport-layer drop from a
genuinely empty remote measurement.
* test(e2e): replace predastore du-reclaim gate with object lifecycle contract
The multinode compaction gate measured shard-store disk usage over SSH (du)
and asserted reclaim after churn. It could never pass: segments only seal at
the 4 GiB roll threshold, so test-scale churn produces no compactable segment,
and the du probe read 0 bytes before and after. Reclaim correctness is already
covered deterministically by predastore's store-level unit tests.
Delete AssertPredastoreCompaction and its du/churn/poll/sentinel machinery,
the harness unit tests that exercised them, and the multinode TestPredastore
Compaction entry. Add a single-node TestPredastoreObjectLifecycle that checks
the user-visible contract: an object is readable after PutObject, gone after
DeleteObject (GET returns NoSuchKey and ListObjectsV2 is empty), and the store
keeps serving fresh writes. newPredastoreS3 is retained, refactored to take a
host so it works outside the multinode cluster type.
* test(e2e): drop orphaned multinode TestPredastoreCompaction
Complete the du-reclaim -> object-lifecycle refactor: remove the
multinode TestPredastoreCompaction caller (and its now-unused context
and harness imports) that still referenced the deleted
harness.AssertPredastoreCompaction, which broke e2e-build. Wire the
single-node TestPredastoreObjectLifecycle entry that replaces it.
* docs: add Containers section with EKS and ECR service pages
New docs/containers/{eks,ecr} pages covering each service via AWS CLI, the Spinifex console, and Terraform, plus the requisite VPC/IGW/NAT/SG/IAM setup. EKS page embeds console screenshots. Cross-links the EKS Terraform workbooks.
* docs(eks): rework Terraform workbooks into a capability ladder
Rework the three EKS workbooks into a coherent ladder and publish them to
the Containers section. All tiers are single- or three-node capable via
node_desired_size and serve a shared Spinifex-themed demo app from ECR,
replacing nginxdemos/hello.
- demo-app: new std-lib Go web app (themed page reporting pod/node/cluster/
region + hit counter; optional /data persistence), distroless image.
- eks-quickstart: node_desired_size var, ECR repo + themed app over NodePort.
- eks-https-ingress (was eks-addons-access): HTTPS via the aws-load-balancer-
controller addon + ACM, public/private VPC with NAT, managed-ingress=false.
- eks-gitops-argocd (was eks-https-ingress): Argo CD addon syncs the app from
a git repo, aws-ebs-csi-driver PVC (Viperblock-backed) for state, access
entry RBAC, HTTPS Ingress carried over.
Pending live e2e verification (image build/push, apply, ALB + PVC bind).
* fix(ecr): implement ListTagsForResource (empty tags) for client parity
The AWS provider calls ecr:ListTagsForResource on every repository read, which
returned 501 NotImplementedException and broke 'tofu apply' of any workbook that
creates an aws_ecr_repository. Implement it to return an empty tag set (resource
tagging stays a deferred feature) so reads succeed. Drop the tags block from the
three EKS workbooks' ECR repos so the provider doesn't call the still-stubbed
TagResource on create.
* feat(eks): pull from internal ECR via kubelet credential provider
Worker nodes could not pull custom images from the internal Spinifex ECR
registry: the eks-node had no DNS for the registry host, no containerd CA
trust, no port rewrite to :9999, and no pull credentials. Only public
images worked.
Add cmd/ecr-credential-provider, a kubelet exec credential provider that
fetches the node IAM-role credentials from IMDS, calls GetAuthorizationToken
against the gateway ECR endpoint, and returns the minted AWS:<jwt> as Basic
auth for the registry host. Thread the gateway URL, account, region, and
internal suffix into the agent userdata, which now writes the gateway CA,
a registries.yaml (CA trust + :9999 endpoint), the kubelet
credential-provider config and config.yaml.d drop-in, and an /etc/hosts
entry mapping the registry host to the gateway IP. Ship the provider binary
in the eks-node AMI.
Also remove a stray paragraph from the admin-cli README.
* fix: dial EKS ECR registry mirror by gateway IP
cloud-init manage_etc_hosts rewrites /etc/hosts from the alpine template on
every boot, wiping the registry-host mapping appended via bootcmd, so worker
nodes could not resolve the internal ECR host and pulls failed with no such
host. Point the k3s registries.yaml mirror endpoint and tls config at the
gateway IP instead (the gateway cert carries an IP SAN), keeping the pod image
ref port-less and dropping the DNS leg until northstar lands. Fall back to the
hostname endpoint when the gateway is not an IP.
* fix: attach node-role instance profile to EKS workers for ECR
EKS workers launched with no IAM instance profile, so IMDS served no role and
the ECR credential provider could not mint a registry token, leaving internal
ECR pulls failing with 401. Real EKS implicitly creates an instance profile for
the node role; do the same here. Wire a KV-backed IAM service into the EKS deps
and, at worker launch, find-or-create an instance profile named for the node
role, attach the role, and pass it to RunInstances so IMDS exposes the role to
the credential provider.
* feat: use mulga logo in EKS demo app
* feat: add ACM certificate tag support (List/Add/RemoveTags)
Implements ListTagsForCertificate, AddTagsToCertificate and RemoveTagsFromCertificate across the store, service, NATS, daemon and gateway layers. Import now persists tags. Fixes terraform aws_acm_certificate InvalidActionException on ListTagsForCertificate.
* test: cover ACM tag gateway and NATS wrapper paths
* feat: support EKS nodegroup tags (Tag/Untag/ListTags)
Extends EKS tag ops to back nodegroup ARNs alongside cluster ARNs. Nodegroup create-time tags persist on the record and echo via DescribeNodegroup, giving the terraform-aws provider a clean default_tags round-trip. Fixes aws_eks_node_group TagResource 501 NotImplementedException.
* fix: provision EKS ALB ingress via node credentials
The aws-load-balancer-controller addon could not provision an ALB. Three fixes:
- TLS: the controller's per-service ec2/elbv2/acm SDK clients install their own
BaseEndpoint resolver and build an HTTP client that bypasses AWS_CA_BUNDLE,
falling back to the Go system trust pool and rejecting the gateway cert as
'unknown authority'. Add SSL_CERT_FILE so the gateway CA joins the system pool.
- Credentials: an empty AWS_ROLE_ARN beside the web-identity token file
half-arms the SDK web-identity provider, which then signs anonymously
(MissingAuthenticationToken). Inject the IRSA env only when a service-account
role ARN is staged; otherwise omit it so the default chain falls back to the
node instance-profile credentials served over IMDS.
- Authorization: the node role lacked the ec2/elbv2 actions the controller
needs. The eks-https-ingress workbook now attaches an AWSLoadBalancerController
managed policy to the node role (CreatePolicy + AttachRolePolicy).
* fix: add force_delete to EKS workbook demo ECR repos
tofu destroy of an EKS workbook left the demo ECR repository behind: it
still holds the pushed demo image, so DeleteRepository (force=false)
returns RepositoryNotEmptyException. Add force_delete = true to each
workbook's aws_ecr_repository.demo so teardown removes the repo with its
images.
* fix: force-delete EKS control-plane ENIs on teardown
TerminateK3sServerVM deleted the CP ENI with a plain force=false
DeleteNetworkInterface. eniIsLiveAttachment reports InUse whenever the
record's Status/InstanceId/AttachmentId are set, so when the K3s VM is
gone but the ENI record's attachment fields were never cleared the delete
returned InvalidNetworkInterface.InUse permanently and EKSDeletingReaper
re-drove the same non-idempotent call every 90s, wedging the cluster in
DELETING.
Teardown owns its CP ENIs and has already terminated the VM, so the
attachment is authoritatively dead. Replace the InUse-retry budget loop
with detach-then-delete (detachAndDeleteServerENI): DetachENI clears the
stale attachment fields, then DeleteNetworkInterface passes the in-use
guard. Both calls tolerate an already-gone ENI (NotFound) so a race with
the async instance-terminate cascade resolves to idempotent success.
purgeClusterInfra's private-endpoint ENI path reuses the same helper.
* fix: detach+delete worker ENIs on EKS nodegroup teardown
A terminated nodegroup worker could leave its primary ENI in
spinifex-vpc-enis as in-use/attached, and a dangling ENI throws
DependencyViolation on subnet/VPC delete, blocking tofu destroy. Two
gaps:
TerminateWorkerInstances did a local-only vmMgr.Get and silently skipped
any worker owned by another node, stranding both the VM and its ENI on
multi-node clusters. Route the terminate per-instance via ec2.cmd.<id>
(mirroring the gateway TerminateInstances path) so the owning daemon runs
the full teardown — detach + force-delete the primary ENI and clear its
record. No responder / NotFound payload is treated as already-gone so a
retried DeleteNodegroup stays idempotent.
deleteInstanceENI (stopped-instance terminate) issued a force=false
DeleteNetworkInterface with no prior detach: a stale in-use attachment
made it fail InvalidNetworkInterface.InUse, leaving the ENI record behind
after the instance was already written to the terminated bucket. Detach
first, then delete, tolerating NotFound.
* fix: heal volume State drift on detach/terminate teardown
A detach/terminate could leave a volume's State empty instead of
'available'. DeleteVolume gated on State=='available' exactly, so it
returned VolumeInUse for an unattached volume nothing was using, leaving
it undeletable and blocking stack teardown. DescribeVolumes cosmetically
rendered an empty State as 'available', masking the drift.
- DeleteVolume now treats an empty State with no attachment as deletable
(an attached volume still returns VolumeInUse).
- DescribeVolumes derives an empty State from the attachment (attached ->
in-use, unattached -> available) instead of blindly rendering
available, so an empty-but-attached volume is no longer hidden.
- UpdateVolumeState normalizes an unattached volume with an empty state to
available, so a detach writeback that omits the state cannot strand the
volume in drift.
* fix: break all SG cross-references on EKS teardown
DeleteClusterSGs broke the cp<->nodegroup cycle too narrowly. It revoked
only ingress (IpPermissions), never egress (IpPermissionsEgress), so a
cross-reference via an egress rule kept the SG pinned on
DependencyViolation. It also resolved only the two named cluster SGs, so
any OTHER SG in the VPC referencing them — the default SG, the LBC/ALB SG,
the private-endpoint SG, user/TF SGs — was never revoked and pinned the
referenced SG on delete, leaving the VPC stuck.
Enumerate every non-default SG in the VPC and revoke BOTH ingress and
egress on each before deleting, breaking any cycle order-independently,
then delete only the cluster-owned SGs (referrer SGs are deleted by their
own teardown). The default SG is left untouched. Since DeleteClusterSGs
runs against both the customer VPC and the managed control-plane VPC, the
broadened sweep covers both teardown paths.
* feat: support TargetGroup ProtocolVersion in elbv2
CreateTargetGroup now stores ProtocolVersion (defaulting HTTP/HTTPS target
groups to HTTP1) and DescribeTargetGroups/Create output return it. Without
this the aws-load-balancer-controller sees a perpetual immutable-field
mismatch (desired HTTP1 vs empty) and recreates the target group every
reconcile, looping on DuplicateTargetGroupName once a shared ALB IngressGroup
reconciles more than one target group. NLB protocols (TCP/UDP/TLS) keep it
omitted to match AWS.
* fix: omit IRSA env from ebs-csi addon when no role staged
The ebs-csi controller manifest hardcoded AWS_ROLE_ARN={{SERVICE_ACCOUNT_ROLE_ARN}}
plus the web-identity token volume. With no service-account role staged the
placeholder rendered empty, half-arming the SDK web-identity provider, which
panics 'role ARN is not set' and CrashLoops the controller, leaving PVCs
Pending. Convert to the conditional {{IRSA_ENV}}/{{IRSA_VOLUME_MOUNT}}/{{IRSA_VOLUME}}
markers like the LBC bundle so the IRSA env and token volume are omitted when
no role is staged, falling back to node instance-profile credentials over IMDS.
The EBS-specific AWS_EC2_ENDPOINT/AWS_ENDPOINT_URL_EC2 stay in both modes.
Requires an eks-node AMI rebake to take effect.
* docs: route gitops-argocd workbook through a shared ALB and expose ArgoCD UI
Fold the demo app and ArgoCD ingresses onto one ALB via an LBC IngressGroup
with host routing (app./argocd. on a wildcard cert) instead of one ALB per
ingress, add a NodePort + HTTPS Ingress for the ArgoCD UI, and document the
access steps. Add the node-role LBC and EBS-CSI IAM policies the controllers
need (Spinifex wires controller creds at the node level, not IRSA) and an
ignore_changes block for the cluster's access_config and
vpc_config security_group_ids, which Spinifex DescribeCluster does not echo
back, so the provider stops trying to replace the cluster on every plan.
* fix: stop addon-sync substituting IRSA markers named in comments
The ebs-csi controller manifest's header comment spelled out the literal
{{IRSA_ENV}} / {{IRSA_VOLUME_MOUNT}} / {{IRSA_VOLUME}} marker tokens. addon-sync
matched each marker by substring, so the comment lines were replaced with the
env/mount blocks, injecting a bare YAML sequence between two documents. K3s'
auto-deploy then rejected the whole file (ParseManifestFailed: cannot unmarshal
array into unstructured), so the ebs-csi Addon CR never applied and the EKS
addon hung in CREATING. Reword the comment to not contain the literal tokens,
and anchor the marker substitution to lines that are exactly the marker
(whitespace aside) so a future comment mentioning a marker can't retrigger it.
* fix: disable K3s local-storage on EKS clusters in parity mode
In AWS LB Controller parity mode (BuiltinIngress=false) the K3s server
disabled traefik and servicelb but left local-storage enabled. Its local-path
provisioner ships a StorageClass marked default, which races the EBS CSI
ebs-gp3 default StorageClass — two defaults, so dynamic PVCs bind
nondeterministically. EKS has no local-path, so disable local-storage too and
let ebs-gp3 be the sole default. The built-in-ingress path is unchanged.
* fix: re-encrypt HTTPS target groups in haproxy ALB backends
An ALB target group with backend-protocol HTTPS (e.g. argocd-server, which
serves TLS on its own port) rendered a plaintext haproxy server line. HAProxy's
own check then spoke HTTP to a TLS port, failed, and marked the server DOWN,
returning 503 — even though the lb-agent's separate HTTPS health probe reported
the target healthy, so describe-target-health masked the fault.
Emit check-ssl ssl verify none on the server line for HTTPS target groups so
both the check and proxied traffic re-encrypt to the backend.
* fix: reconnect wedged QMP client on next command
A timed-out QMP decode under host contention left the cached client's
shared json.Decoder mid-message, wedging every later command and breaking
EBS AttachVolume/DetachVolume for the VM's lifetime. Mark the client Dead
on encode/decode error and redial the retained socket path (re-running the
qmp_capabilities handshake in place) on the next send.
* fix: round-trip security-group rule descriptions
SG ingress/egress rules dropped the per-rule Description on store and
describe. The AWS Load Balancer Controller tags the worker-node SG rules it
manages with a UserIdGroupPair description and identifies them by that value
to revoke them on teardown. Without the round-trip the controller cannot
match its rules, never revokes them, and the backend security group stays
referenced — DeleteSecurityGroup returns DependencyViolation forever,
hanging the ingress finalizer and tofu destroy with orphaned security groups.
Persist Description on SGRule (excluded from sgRuleKey so it is metadata, not
identity) and render it back through describe.
* fix: reclaim customer-VPC nodegroup SGs on managed-CP-VPC teardown
In the managed control-plane VPC topology, launchNodegroupInfra creates the
cluster control-plane and nodegroup security groups in the customer VPC for
worker<->control-plane networking, but purgeClusterInfra only reclaimed the
security groups in the managed CP VPC. The customer-VPC SGs orphaned,
cross-referenced each other, and pinned the customer VPC with
DependencyViolation, hanging tofu destroy of the VPC. Reclaim them in the
managed-CP-VPC branch too, matching the legacy customer-VPC branch.
* fix: reclaim ALB EIP when system VM already gone on terminate
TerminateSystemInstance only released an internet-facing ALB's Elastic IP on
the local path, where the VM is still present in the manager. When the VM was
already gone (health-reaped, crashed, or raced by ENI teardown), it took the
remote/no-owner path and returned idempotent success without reconciling the
EIP, leaking the wan-pool address still associated to a deleted ENI.
Add EIPServiceImpl.ReleaseAddressByInstanceID, an idempotent account-agnostic
scan that disassociates then releases any EIP recorded against an instance, and
call it as a backstop in TerminateSystemInstance for both branches.
* fix: cascade-prune demo PVC via Argo CD finalizer in eks-gitops workbook
The demo Application carried no resources finalizer, so tofu destroy removed
the Application CR without pruning the synced PVC. The PVC survived, the cluster
was destroyed, and the Delete-reclaim PV's EBS volume orphaned. Add the
resources-finalizer.argocd.argoproj.io finalizer so deleting the Application
cascades to the PVC and ebs-csi reaps the volume before cluster teardown.
* feat: raw-IP access + documented Argo CD creds in eks-gitops workbook
The shared ALB host-routed everything, so the raw ALB IP 404'd in a browser
without an /etc/hosts entry. Make the app the hostless catch-all on :443 and add
a hostless Argo CD listener on :8443, so https://<alb-ip>/ serves the app and
https://<alb-ip>:8443/ serves the UI with no DNS. Surface the admin-password
command as a tofu output and document the credential + IP URLs in the README.
* fix: reclaim ALB EIP on the owning-node terminate path
The EIP backstop lived only in TerminateSystemInstance, but the ALB
delete path goes serveSystemTerminateInstance -> terminateSystemInstanceLocal
directly, bypassing it, and releaseSystemInstanceEIP relied on the VM's
cached allocation which is often empty. Extract reclaimSystemInstanceEIP
(authoritative KV reconcile via ReleaseAddressByInstanceID) and call it
from terminateSystemInstanceLocal as well as the no-owner backstop, so an
internet-facing ALB's EIP is released on every terminate route.
* chore(eks): drop coredns from the managed addon catalog
Cluster DNS ships as the baked node-local coredns-mulga DaemonSet, not as
a managed addon — there is no bundled coredns manifest, so CreateAddon
coredns would stage but fail VM-side render. Remove the phantom catalog
entry so DescribeAddonVersions and CreateAddon no longer advertise it;
repoint the addon tests that used coredns as their sample to argocd.
* feat(ui): group EKS and ECR under a Containers nav section
Fold the separate EKS and ECR sidebar groups into one Containers group
with EKS (clusters) and ECR (repositories) entries.
* fix(ui): satisfy strict null checks in ECR component tests
Index mock.calls via optional chaining and narrow the getAllByRole result
before clicking, clearing the TS2532/TS2345 errors that failed pnpm fix.
* refactor(eks): hide internal fixture addons from the public catalog
Add an AddonSpec.Hidden flag and mark spinifex-noop hidden: it stays
creatable (lookupAddon finds it, so the delivery-transport e2e still drives
it) but no longer surfaces in the unfiltered DescribeAddonVersions listing.
* refactor(eks): remove K3s built-in ingress; LBC-only
Drop the BuiltinIngress option and its managed-ingress tag opt-in. traefik,
servicelb, and local-storage are now always disabled on EKS nodes (AWS
parity — Service type=LoadBalancer and Ingress are the Load Balancer
Controller's job). Removes the EKS_DEFER_TRAEFIK userdata flag, the k3s.initd
traefik .skip staging, and the maybe_enable_traefik deferred-enable loop in
the state reporter.
* fix(ui): clear lint debt in VPC-wizard mutations and CIDR validation
Parallelize the independent subnet-create and route-table-association loops
in the create-VPC mutation with Promise.all (each task records its resource
in the rollback list before any further call), and switch the CIDR-validation
regex to named capture groups. Clears the no-await-in-loop and
prefer-named-capture-group lint errors.
* fix: carry gateway port in ECR registry endpoints; rename mulga.internal -> spinifex.internal
The ECR registry is served on the gateway port (:9999), but proxyEndpoint and
repositoryUri were emitted port-less, so docker login/tag/push dialed :443 and
failed. Derive the port from the awsgw listen host and append it to the registry
host (omitted for 443). Also rename the default internal DNS suffix.
* docs: drop verbose ECR :9999 host note
* docs: fix literal-asterisk emphasis (single-* not rendered)
* docs: brace-wrap ${REGISTRY}:latest (zsh :l modifier ate the tag); document Docker CA trust
* feat: default ECR registry URIs to the gateway host (no-DNS interim)
The registry authenticates by token, not hostname, so the per-account
<acct>.dkr.ecr name is cosmetic. Advertise the gateway's reachable bind host:port
(already a cert SAN) in repositoryUri/proxyEndpoint so docker login/tag/push need
no DNS; the parity host still works server-side where DNS exists.
* fix: serve ECR registry on a DNS-free gateway host end to end
Advertise the gateway's AdvertiseIP as the ECR registry host when the AWSGW
bind is unspecified (0.0.0.0/::), so repository_url resolves to a reachable
host:9999 carried in the server cert SANs rather than the unresolvable
per-account parity name. Docker push then dials it directly with no DNS.
Mirror the same host on EKS workers: the registries.yaml mirror keys and the
credential provider matchImages now cover the port-less parity name, the
port-ful parity name, and the gateway IP:9999, all redirecting to the dialed
gateway endpoint. A port-ful pull from the API-returned repository_url matches
the mirror instead of bypassing it into a failed DNS lookup.
Net: docker push host, pod image host, and the API's repository_url are the
same value, with no DNS or /etc/hosts.
* docs: resolve eks-https Ingress ALB DNS name to its public IP (no DNS)
* docs: resolve eks-gitops Ingress ALB DNS name to its public IP (no DNS)
* docs: remind to run aws eks update-kubeconfig before kubectl in EKS workbooks
* feat(ecr): return OperationNotSupportedException for image-scanning actions
The ECR image-scan control-plane actions resolved to the generic
NotImplemented 501 stub, indistinguishable from unbuilt features and
surfaced as a server fault. mulga runs no vulnerability scanner, so the
scan surface now returns OperationNotSupportedException (400):
- awserrors: add OperationNotSupportedException (400) ECR code.
- ecrapi: ScanningNotSupported handler; repoint the four image-scan
actions and register the three registry-scanning actions at it.
- spinifexui: Scan tab on the repository detail page renders a static
'scanning not supported' panel rather than a request that errors.
* refactor(config): default internal DNS suffix to spinifex.internal
Rename the AWS-parity internal DNS suffix from mulga.internal to
spinifex.internal across the config default, cert SAN builder, ECR
endpoint/audience construction, and their tests. Single runtime knob is
config.DefaultAWSInternalSuffix; no DNS backend exists yet, so this is a
naming change with no functional impact.
The ECR auth bridge minted ES256 tokens from a key that was created once and
never rotated. Add a background rotator (ecr-v1 Q3): mint a fresh key every 30
days, retain the previous keys for 24h so in-flight 12h tokens stay verifiable,
then prune. The active key is now the newest by KV creation time (not the
lexically-greatest kid) so a minted key takes over and all nodes converge. The
Issuer and Verifier became hot-swappable so the rotator updates the live
gateway as keys roll.
* feat: bake aws-ebs-csi-driver addon + mdev by-id bridge
Add the aws-ebs-csi-driver entry (v1.40.1) to the eks addon catalog and
ship its baked manifests (RBAC, IRSA-wired controller, node DaemonSet,
default gp3 StorageClass) under the eks-node image.
The node plugin locates EBS volumes via /dev/disk/by-id, which busybox
mdev does not populate. Add mulga-ebs-byid: an mdev rule + helper that
symlinks nvme-Amazon_Elastic_Block_Store_<serial> from the virtio-blk
serial, plus a boot oneshot that points the kernel hotplug helper at
mdev and coldplugs volumes present at boot.
* test: e2e EBS CSI volume — PVC bind + mount + reschedule survival
Add a TestEKS/EBSCSIVolume subtest that installs the aws-ebs-csi-driver
managed addon bound to an IRSA role, provisions a default gp3 PVC + pod,
and asserts a marker written by the first pod survives a reschedule onto a
fresh pod backed by the same PVC. Exercises the full data path:
CreateVolume(Viperblock)/AttachVolume virtio-blk hotplug with serial set to
the volume-id, mdev /dev/disk/by-id symlink, and ext4 mount.
* feat: eks-node provider-id + topology/instance-type labels for EBS CSI
mulga-eks-provider-id boot oneshot writes a K3s config.yaml.d drop-in
from IMDS before K3s starts: kubelet provider-id plus the region, zone
and instance-type node labels. The aws-ebs-csi-driver node plugin reads
this metadata for NodeGetInfo (instance id from providerID, region/zone
from topology labels, instance type for the volume-limit family lookup).
k3s runs no AWS cloud-provider so none of it is set otherwise, which left
the node plugin's metadata init failing, NodeGetInfo panicking in
IsNitroInstanceType, and the CSINode carrying no topology key. Wire the
service into the image manifest (INSTALL_FILES + ENABLE_SERVICES) and
chmod it in setup.
* test: e2e EBS CSI controller IRSA policy + failure diagnostics
awsgw enforces IAM on assumed-role EC2 calls, so the controller's
CreateVolume/AttachVolume need a permission policy on the IRSA role.
Attach a customer-managed EC2 policy (AWS-managed ARNs are opaque and
grant nothing) covering the EBS CSI action set; CreateVolume then
succeeds. Add an OnFailure dump of the CSI control path (provisioner and
plugin logs, pvc/pod describe, events, csinode) to the EBSCSIVolume
subtest so provisioning/attach failures are diagnosable post-teardown.
* feat: EBS CSI obeys control-plane taint; e2e worker nodegroup
Drop the CriticalAddonsOnly/NoExecute tolerations from the EBS CSI
controller + node manifests so the driver runs only on customer worker
nodes, never the system control-plane VM. The e2e creates a 1-node
worker nodegroup and the writer/reader pods drop their CP tolerations so
the customer-owned volume attaches in the customer space.
* fix: EBS CSI attach — decouple hotplug PCIe port from device name; iothread rollback
virtio-blk-pci hot-plug bus was derived from the AWS device-name letter
(/dev/sd[f-p]); the EBS CSI driver sends arbitrary names (/dev/xvdaa) that
map to no slot, so device_add fell back to pcie.0 — which rejects hot-plug —
and the attach hung. A failed attach also left the per-volume iothread object
behind, so every retry died on object-add 'duplicate property'.
nextHotplugEBSPort allocates the lowest free hotplug-ebs port from live
query-block state, independent of the device name. delIothreadBestEffort
removes the iothread on every attach rollback path.
* fix: EBS CSI node discovery — un-shadow mdev by-id rule, mint nvme link
Alpine eks-node runs busybox mdev (no eudev), which stops at the first
matching /etc/mdev.conf rule. setup.sh appended the mulga-ebs-byid rule after
the stock 'vd[a-z].* -> persistent-storage' line, so it never fired on
hot-attach: only /dev/disk/by-id/virtio-<serial> was created, while the
upstream aws-ebs-csi-driver resolves nvme-Amazon_Elastic_Block_Store_<volid>.
Swap the stock vd* rule in place to route through mulga-ebs-byid, which now
delegates to persistent-storage (preserving stock links) before minting the
nvme-style link the node plugin expects.
* test: EKS e2e — NAT Gateway egress for private worker nodegroup
The customer-VPC worker is private; AttachIGW programs no SNAT (mirrors AWS:
an IGW serves only public-IP instances), so an IGW-only fixture left the worker
unable to reach the public NLB endpoint or pull public CSI images — the
nodegroup never joined.
createWorkerEgress now builds the real topology: an IGW-fronted public subnet
hosting a NAT Gateway, with the worker subnet default-routed to it. Associating
the worker route table triggers the subnet-wide SNAT egress reroute. Teardown
drains the NAT Gateway before releasing the public subnet + EIP it pins.
The OCI /v2 registry auto-created a repository on first push (ensureRepo).
AWS ECR does not: a push or PutImage to a repository that was never created
with CreateRepository is rejected. Replace ensureRepo with requireRepo, which
returns OCI NAME_UNKNOWN on the data plane and RepositoryNotFoundException on
the JSON PutImage path. Enforcement sits at the single push choke point
(startUpload + StoreManifest).
* test(e2e): add ECR end-to-end suite (control plane + docker/crane/skopeo)
New tests/e2e/ecr/ suite plus harness support:
- harness: ECR SDK client, control-plane helpers (repo CRUD, get-login-password,
describe-images), and docker/crane/skopeo CLI wrappers that skip when the
binary or registry-host resolution is absent.
- suite: control-plane CRUD + policy + lifecycle round-trip; docker login/push/
pull; crane append/pull and skopeo copy exercising the Bearer /v2/token flow.
- registered e2e-ecr in the Makefile + service manifest.
Predastore capacity/load leg is a skipped placeholder pending mulga-siv-359/356.
* test(e2e): carry awsgw port in ECR registry host for dev clusters
Dev clusters bind awsgw on a non-standard port (e.g. 9999); production uses
:443. ECRRegistryHost now appends SPINIFEX_ECR_PORT when set so docker/crane/
skopeo address the right endpoint, and RequireRegistryResolves strips the port
before DNS lookup.
getManifest read only the first Accept header line via r.Header.Get, so
clients that send each acceptable manifest media type as a separate Accept
header (skopeo) never matched the stored type and got 406 MANIFEST_INVALID.
Join all Accept header values before negotiation.
Add the Docker Registry v2 token endpoint (GET /v2/token) and switch the
/v2 401 challenge from Basic to a Bearer realm pointing at it. The endpoint
verifies the caller's existing AWS:<jwt> credential and re-mints a fresh
full-TTL token, providing the token-refresh path for OCI clients (docker,
crane, skopeo). The auth bridge still accepts Basic AWS:<jwt> directly.
Stamp the dash-stripped volume-id (vol+17hex = 20 bytes, the virtio-blk
serial limit) as the hotplug disk serial so the EBS CSI node plugin can
locate the device via /dev/disk/by-id and match lsblk SERIAL.
* feat: ECR background lifecycle expiry sweep
Add a gateway-side periodic sweep (LifecycleSweeper) that applies each repo's
stored lifecycle policy: it evaluates the shared rule engine against the repo's
current images and deletes the expired set through the existing registry GC
reclaim path (DeleteImage -> reclaimManifest). It runs in the gateway because
only the gateway holds a Registry (object store + meta); the daemon owns the KV
but cannot reclaim predastore bytes. Launched on the STS janitor's lifetime
context in awsgw, enumerating ACTIVE IAM accounts.
Per repo: skip when no policy; evaluate; then before deleting, re-read the
current tag set and skip any image whose tags changed since evaluation
(optimistic-concurrency guard against concurrent push/re-tag). Cross-repo blob
mount safety is provided by reclaimManifest/referencedDigests. Per-item errors
are logged and skipped so one bad repo never aborts the cycle.
* test: cover lifecycle sweeper Run loop and active-account filter
Add tests for LifecycleSweeper.Run ticker loop, NewLifecycleSweeper
interval fallback, and sweepRepo error branches. Extract accountLister
interface so activeAccountIDs is unit-testable. Raises diff coverage
on the sweep changes from 53.4% to 76.9%.
* feat: ECR lifecycle policies (CRUD + evaluation engine + preview)
Implement PutLifecyclePolicy/GetLifecyclePolicy/DeleteLifecyclePolicy and
StartLifecyclePolicyPreview/GetLifecyclePolicyPreview, replacing the
NotImplemented stubs. A pure rule-evaluation engine parses the AWS
lifecycle-policy JSON and, given a repo's images, returns the ordered set that
would expire; each image is claimed by at most one rule (lowest rulePriority
first). Preview is served synchronously and deletes nothing.
Lifecycle-policy documents persist as a per-repo KV doc (repos/<name>/lifecycle)
over the ecr.lifecycle.* NATS subjects, mirroring the repo-policy storage stack,
and cascade on DeleteRepository. The console gains a Lifecycle tab with a policy
editor and a preview table over the saved policy.
The background expiry sweep that applies the engine and deletes images via GC is
deferred to mulga-siv-387.
* feat: client-side JSON validation in ECR policy editors
Swap the raw Textarea in the repository-policy and lifecycle-policy editors for
the shared JsonEditor (blur-time pretty-print) and gate Save on isValidJson,
showing an inline error for malformed JSON — matching the IAM/EKS editors. The
backend remains the source of truth for schema validation.
Honor, persist and echo imageTagMutability on CreateRepository and
DescribeRepositories instead of the hardcoded MUTABLE, and implement
PutImageTagMutability to flip a repository between MUTABLE and IMMUTABLE.
Enforce immutability at the shared StoreManifest chokepoint: on an
IMMUTABLE repo, re-tagging an existing tag onto a different digest is
rejected before any write (OCI 409 TAG_IMMUTABLE; JSON PutImage maps to
ImageTagAlreadyExistsException). Same-digest re-pushes, new tags and
digest references are still allowed.
Console: migrate the repository detail page to Summary/Images/Permissions
tabs and add a tag-immutability toggle on the Summary tab.
DELETE /v2/{name}/manifests/{digest} was a 202 no-op. Now delete-by-digest
removes the manifest record, deletes the manifest object, and sweeps the
account-wide blob pool: each child blob no longer referenced by any live
manifest is reclaimed, while shared blobs survive.
- DeleteImage (digest path) captures ChildDigests, deletes the manifest object,
then reclaimManifest() drops orphaned blobs via referencedDigests() (a
mark-and-sweep over the live manifest set across the account's repos).
- OCI deleteManifest by-digest wired to DeleteImage (real work + 202; 404 on
unknown). Tag delete stays KV-only so untagged images persist (AWS-faithful).
- Predastore reclaim is best-effort: a failure leaks bytes but never undoes the
logical delete.
Serve the five image-management JSON 1.1 actions inline in ECR_Request,
reusing the gateway-side OCI Registry for manifest bytes:
- ListImages / DescribeImages enumerate images by tag + digest (KV meta).
- BatchGetImage returns verbatim manifest bytes with Q14 partial-failure
semantics (cap 100, digest wins over tag, 200 + failures array).
- PutImage stores a manifest + tag via a shared StoreManifest core that the
OCI manifest PUT now delegates to.
- BatchDeleteImage removes tag pointers / manifest meta (blob GC deferred).
Adds DeleteManifestMeta across the MetaStore relay chain and ECR image
error codes.
Add the two missing repository-lifecycle actions, served inline in
ECR_Request (like DescribeRepositories) since both return the repository
ARN/URI built from the gateway region + internal suffix.
CreateRepository validates the name, guards cross-account registryId,
rejects an existing repo (RepositoryAlreadyExistsException), and writes
the per-account KV meta (predastore bucket stays lazy on first push).
DeleteRepository rejects a non-empty repo without force
(RepositoryNotEmptyException) and cascades the KV records (meta, policy,
tags, manifests). Predastore blob GC is deferred to a separate pass.
Adds MetaStore DeleteRepo + ListManifests with the matching daemon-relayed
subjects, service methods, and NATS client, plus two new awserrors codes.
The per-run ARTIFACT_DIR lives in /tmp, outside GITHUB_WORKSPACE, so a
re-run on the same self-hosted runner reused a prior attempt's append-only
suite-status.txt. The 'Fail job if any suite failed' gate's -s check then
failed the job even when the surviving attempt passed with rc=0.
Wipe ARTIFACT_DIR in Clean Workspace and gate solely on the per-attempt
steps.e2e.outputs.rc (fail-closed: an empty rc still fails).
CreateTags/DeleteTags wrote only the standalone tag store, which
DescribeImages does not read (it reads AMI metadata), so tag mutations on
an AMI were invisible — DeleteTags appeared to be a silent no-op. Mirror
tag add/remove into the owning AMI config (ownership-checked; absent or
foreign AMIs skipped) so DescribeImages reflects them, matching the
existing subnet/vpc mirror.
* feat(ecr): auth bridge + GetAuthorizationToken JWT (Sprint 2c)
Mint a self-contained ES256 JWT from GetAuthorizationToken and verify it on
the /v2 OCI registry via a new ECRAuthBridgeMiddleware (Bearer | Basic AWS:jwt).
Signing keys live in the cluster-replicated awsgw-keys KV bucket under
jwt-signing/{kid}, encrypted with the IAM master key.
The registry is now scoped per-request to the token's account (replacing the
fixed bootstrap account); bucket-ensure is cached per account. Cross-account
token/host mismatch is rejected 403; unauthenticated /v2 access gets a 401
Bearer challenge.
* fix(ecr): use Basic auth challenge on /v2 (not Bearer token-endpoint flow)
ECR authenticates the OCI registry with Basic AWS:<token>; a Bearer realm
made docker chase an unimplemented /v2/token endpoint and fail login.
* feat(ui): inline EKS cluster IAM role creation in wizard
* feat(ui): show trust policy + attached managed policy in EKS role dialog
Render the trust policy with PolicyDocumentViewer (pretty-printed, safe
fallback) instead of a raw read-only textarea, matching the IAM detail
pages. Surface the attached AmazonEKSClusterPolicy ARN so users can see
exactly which permissions the cluster role grants.
* fix(ec2): stamp subnet AvailabilityZoneId for single-AZ ALB resolution
DescribeSubnets omitted AvailabilityZoneId, so the AWS Load Balancer
Controller's DescribeAvailabilityZones-by-ZoneId lookup resolved no AZ
info ("cannot resolve AZ info for AZs: []") and never built the ALB
model. Stamp the single zone-id on every subnet and share it with
DescribeAvailabilityZones through one SingleZoneID constant so the two
responses can never drift.
* feat(eks): add aws-load-balancer-controller v2.8.1 addon bundle
Baked manifests for the LBC managed addon (CRDs, controller Deployment/
RBAC/webhooks/Service, IngressClass alb, IRSA render markers). Two
wirings make it work against the spinifex gateway:
- --aws-api-endpoints carries sts= alongside elasticloadbalancing/ec2/
acm. The aws-sdk-go v1 credential bootstrap resolves the STS client
from the session endpoint resolver, not AWS_ENDPOINT_URL_STS, so the
web-identity AssumeRoleWithWebIdentity must be pointed at the gateway
here or it falls back to real AWS STS.
- --feature-gates=ALBSingleSubnet=true drops the ALB minimum subnet
count from two to one so an ALB provisions in a single-AZ deployment.
* feat(eks): render IRSA into managed addons + node-minted webhook cert
Extend the addon-sync renderer to inject the projected-token / role-arn /
gateway STS-endpoint + CA block into managed-addon pod specs via opt-in
markers, and substitute cluster-name/region/vpc-id/gateway-endpoint/CA
placeholders. Add cmd/eks-webhook-cert: the eks-node AMI has no openssl,
so this mints (then reuses) a self-signed ECDSA-P256 serving cert for the
admission webhook, cached on disk so the render is byte-stable across
ticks. Wire both into the image build (manifest.conf) plus INSTALL_FILES
for the LBC bundle.
* feat(eks): surface cluster VPC id as EKS_VPC_ID for the LB controller
Thread the control-plane VPC id through K3sServerInput.VpcID into the
first-boot EKS_VPC_ID env so the in-cluster aws-load-balancer-controller
can pass --aws-vpc-id to the gateway elbv2/ec2 handlers.
* fix(eks): point the LB controller at the customer VPC + customer-facing gateway
The aws-load-balancer-controller ran with two control-plane-scoped values
that customer worker pods cannot use:
- --aws-vpc-id was set from the managed control-plane VPC (cpRefs.VpcID),
which the gateway's DescribeVpcs does not return for customer calls, so
backend-SG auto-create failed with InvalidVpcID.NotFound. Source it from
meta.ResourcesVpcConfig.VpcId, the customer cluster VPC where the workers
and their subnets live and where the ALB is provisioned.
- The addon render baked the mgmt gateway endpoint (EKS_GATEWAY_URL /
SystemGatewayURL) into pod specs, unreachable from worker pods. Add a
distinct EKS_ADDON_GATEWAY_URL (the customer-facing GatewayBaseURL, same
host as the OIDC issuer) and substitute it for {{GATEWAY_ENDPOINT}} in
managed addon manifests. EKS_GATEWAY_URL stays mgmt for the CP's own
bootstrap, state-report and token-webhook paths.
* fix(elbv2): resolve forward target group from ForwardConfig
listenerActionFromSDK read only the legacy flat Action.TargetGroupArn, so a
forward action carrying its target group via ForwardConfig.TargetGroups (the
shape the aws-load-balancer-controller emits) left the action target empty and
CreateRule/CreateListener rejected it with MissingParameter. Flatten the first
target group from ForwardConfig when the flat field is absent.
* fix(ec2): mirror CreateTags/DeleteTags into subnet/vpc records
The generic S3 tag store and the per-resource KV records are disjoint:
DescribeSubnets/DescribeVpcs read tags from their KV record, while
CreateTags only wrote the generic store. Subnet tags applied after
create were therefore invisible to tag-filtered describes, breaking
aws-load-balancer-controller subnet auto-discovery (kubernetes.io/role/elb).
Write through CreateTags/DeleteTags to the owning subnet/vpc record so
tag-filtered describes observe them. Resource types the VPC service does
not own are skipped; the generic store stays their record of truth.
* test(daemon): pin synthetic capacity in host-dependent allocate tests
TestCanAllocate_CountEdgeCases/Mixed_instance_types and
TestHandleEC2DescribeInstanceTypes/NoFilterReturnsSupportedRegardlessOfAllocation
called rm.allocate against live-detected host capacity. allocate gates on
readMemAvailableGB (live /proc MemAvailable), so a box already running VMs
fails the allocation spuriously, breaking the preflight gate locally.
Pin hostVCPU/hostMemGB/reserved and nil readMemAvailableGB to a synthetic
host, matching the sibling Capacity_decreases_after_allocation subtest.
* fix(eks): route apiserver->pod traffic via agent tunnel (egress-selector cluster)
In the managed-CP topology the apiserver VM sits in the system CP VPC with
no route to the worker pod network in the unpeered customer VPC. k3s default
egress-selector-mode=agent tunnels only apiserver->kubelet, so kubectl
logs/exec work but apiserver->pod traffic (admission webhooks, aggregated
APIs) is attempted directly and times out -- the aws-load-balancer-controller
webhooks fail with context deadline exceeded, blocking Service/Ingress
creation cluster-wide.
Set egress-selector-mode=cluster so apiserver->pod/service traffic routes
through the agent's outbound tunnel, selected by the node hosting the
destination, without requiring inbound CP->worker reachability.
* feat: bump aws-load-balancer-controller addon bundle to v2.11.0
v2.8.1 predates the ALBSingleSubnet feature gate, so single-AZ spinifex
clusters fail Ingress reconcile (subnet resolver dedups to 1 < required 2)
and never provision an ALB. v2.11.0 honors --feature-gates=ALBSingleSubnet=true.
Replace the baked bundle: new CRDs (ingressclassparams + targetgroupbindings),
controller manifest, and image tag v2.11.0, re-applying the spinifex deltas
(IRSA SA annotation, node-minted webhook serving-cert secret, gateway-CA
secret, gateway aws-api-endpoints, caBundle on every webhook clientConfig,
alb IngressClass). Pick up the new ClusterRole configmaps rule. Drop v2.8.1.
* fix: point LBC at the gateway via AWS_ENDPOINT_URL (sdk-v2)
LBC v2.11 uses aws-sdk-go-v2, which keys --aws-api-endpoints on SDK
service IDs (e.g. 'Elastic Load Balancing v2'), not the legacy lowercase
names the bundle passed. Those overrides were silently ignored, so every
non-STS call (ec2/elbv2/acm) went to real AWS and failed TLS against the
gateway CA -- no ALB could provision.
Inject AWS_ENDPOINT_URL in the shared IRSA env block (sdk-v2 global
base-endpoint override; AWS_ENDPOINT_URL_STS still takes precedence for
STS) and drop the dead --aws-api-endpoints flag from the controller args.
* fix: synthesize taggable ARN for default listener rule
Gateway DescribeRules returned the synthetic IsDefault rule with a null
RuleArn. LBC read the arn-less default rule after creating a listener and
issued DescribeTags with an empty resource ARN, which the gateway rejected
with 400 MissingParameter, aborting reconcile.
Set a deterministic RuleArn on the default rule, resolve it back to its
parent listener in DescribeTags (empty tags) and DescribeRules by-arn.
Also disable shield/waf/wafv2 in the LBC bundle to drop unsupported calls.
* feat: inject ELB subnets into alb IngressClassParams for single-AZ ALB
LBC's tag auto-discovery never threads the ALBSingleSubnet feature gate, so a
single-AZ cluster (subnets sharing one AZ) dedups to 1<2 subnets and aborts
Ingress reconcile. Only the explicit-subnet path honors the gate.
Resolve the cluster's ELB-eligible subnets, dedup to one per AZ, and surface
them as EKS_ELB_SUBNET_IDS. The addon sync renders them into the alb
IngressClassParams .spec.subnets.ids so every Ingress takes the explicit path.
An empty set falls back to auto-discovery, preserving the native UX.
* fix: route LBC service clients to gateway via --aws-api-endpoints, link IngressClassParams
Live verification showed two gaps in the LBC addon bundle:
LBC installs a per-client BaseEndpoint resolver, so the global AWS_ENDPOINT_URL
is ignored for the ec2/elbv2/acm clients (calls hit real AWS, failing TLS).
Redirect them with --aws-api-endpoints keyed on the aws-sdk-go-v2 service IDs
(EC2, "Elastic Load Balancing v2", ACM, STS) rather than the legacy lowercase
names the flag never matched. Drop the misleading AWS_ENDPOINT_URL env; the
web-identity STS client still resolves via AWS_ENDPOINT_URL_STS.
The injected IngressClassParams was orphaned: the alb IngressClass had no
.spec.parameters, so LBC fell back to subnet auto-discovery (which fails on a
single-AZ cluster). Link the IngressClass to the IngressClassParams so the
explicit-subnet path is taken.
* fix: stamp EKS-parity providerID on worker nodes for LBC ip targets
k3s registers workers with providerID k3s://<name>, which the AWS Load
Balancer Controller rejects (providerID invalid for EC2 instances) when
resolving nodes for ip-mode target registration. Resolve the instance-id and
AZ from IMDSv2 at agent boot and set --kubelet-arg=provider-id=aws:///<az>/<id>
so LBC can map the node to its EC2 instance. On any IMDS miss the agent keeps
the default providerID and still starts.
* fix(elbv2): honor SubnetMappings in CreateLoadBalancer
LBC supplies ALB subnets via SubnetMappings, not the plain Subnets list.
createLoadBalancer only read input.Subnets, so ENI allocation was skipped,
VpcId/AZ stayed empty, and the HAProxy data-plane VM never launched — the
ALB went active but could not serve traffic. Add flattenSubnetIDs to read
both sources and reuse it in createLoadBalancer and desiredSubnetSet.
* fix(vm): capture GCInterval before spawning GC goroutine
TestStart_SweepsImmediatelyThenStopsOnCtxCancel mutates the package global
GCInterval and restores it in a defer, racing the Start goroutine's read at
time.NewTicker(GCInterval) under -race. Read GCInterval on the caller's
goroutine into a local before spawning so the global access is ordered with
the test's defer; the goroutine touches only the captured local.
* feat: implement OCI Distribution v2 blob and manifest storage
Replace the /v2/* 501 stub with a real path-parsing dispatcher and
content-addressable storage for the ECR registry surface:
- Blob upload (monolithic + chunked PATCH), HEAD/GET, cross-repo mount,
and cancel, with server-side sha256 verification on finalize.
- Manifest PUT with type-aware shallow validation (image manifest blob
existence; image index child-manifest existence + mediaType match),
byte-for-byte GET with stored Content-Type, HEAD, and tag delete.
- Tag list and account-scoped catalog.
Blob and manifest bytes stream straight to predastore; repo, tag,
manifest and in-progress-upload metadata live in a per-account
JetStream KV bucket with optimistic-concurrency CAS on uploads. Chunks
are buffered to a temp upload object and reassembled on finalize, since
predastore exposes no append primitive through the shared object-store
interface. Adds HeadObject to the objectstore abstraction.
Refs: mulgadc/mulga#73
* refactor(ecr): route registry metadata through NATS daemon handlers
The gateway no longer talks JetStream KV directly. The daemon owns the
per-account KV via a MetaService bound to ecr.* request/reply subjects;
the gateway is a NATS client (NATSMetaStore). Blob and manifest bytes
still stream gateway-to-predastore directly.
Also fixes four registry review findings:
- finishUpload: clean up the temp object and upload record on a blob
store failure so the uuid is reusable and no partial blob lingers.
- patchUpload: commit the appended bytes and advanced hash state under a
single KV CAS keyed per write, so a racing PATCH is rejected (409)
rather than corrupting the assembled bytes; the verified digest always
matches the stored bytes.
- cancelUpload: delete both the temp bytes and the record, 204 on
success, 404 on unknown uuid, no dead branch.
- getManifest: return MANIFEST_INVALID (not MANIFEST_UNKNOWN) for a 406
Accept media-type mismatch.
Refs: mulgadc/mulga#73
* fix(ecr): provision account predastore bucket on demand
OCI blob/manifest writes target ecr-{accountID}, which nothing created,
so every push failed with a backend 403. Add idempotent EnsureBucket to
the object store (HeadBucket -> CreateBucket, swallowing already-owned)
and ensure it once per process at the top of the registry ServeHTTP, so
ECR storage appears transparently like AWS-managed ECR.
* fix(ecr): sanitize ':' from predastore blob/manifest object keys
predastore mis-canonicalizes SigV4 for object keys containing ':', so
writes/reads of content-addressable keys (sha256:<hex>) were rejected
403 (reproducible with the aws CLI too). Route BlobKey and ManifestKey
digests through DigestToken so the object key carries no ':' separator.
Implement the registry host-routing middleware so /v2/* requests addressed
at {accountID}.dkr.ecr.{region}.{suffix} carry the parsed target account and
region on the request context; non-registry hosts pass through unchanged.
Add cluster-wide AWS-parity config (aws.region, aws.internal_suffix) with
defaults, and append the AWS-parity ECR SANs (exact control-plane host plus
*.dkr wildcard) to the awsgw server certificate during cert generation.
Refs: mulgadc/mulga#78
* feat: scaffold ECR control-plane + OCI registry gateway with 501 stubs
Add the ECR HTTP surface on awsgw as compiling skeletons that return
NotImplemented, so later work can call into the packages against a fixed
contract.
- gateway/ecrapi: ECR control plane (AWS JSON 1.1, X-Amz-Target). Full action
namespace registered; every action resolves to a shared 501 stub.
- gateway/ecr: OCI Distribution v2 (/v2/*). Catch-all 501 with the OCI error
envelope; GET /v2/ is live for the registry version-check probe.
- gateway/hostmatch: no-op host-routing middleware skeleton for the registry
host group.
- handlers/ecr/storage: per-account bucket + object-key layout constants.
- gateway wiring: ecr in supportedServices, dispatch case, JSON 1.1 error
branch, and /v2 mounted outside the SigV4 group.
* test: cover ECR scaffolding packages for diff-coverage gate
In-package tests for gateway_ecr, gateway_ecrapi, and handlers/ecr so the new
code reports coverage under its own package profile. Diff coverage 76.5%.
The kubernetes provider in the eks-quickstart, eks-addons-access and
eks-https-ingress workbooks was configured from the managed cluster
resource (host = aws_eks_cluster.this.endpoint). On destroy the endpoint
resolved empty, the provider fell back to localhost:80, and refreshing
kubernetes_deployment_v1.hello / kubernetes_service_v1.hello failed with
'dial tcp [::1]:80: connect: connection refused'.
Move the kubernetes provider and the hello Deployment + Service into a
nested workloads/ root module with its own state. It reads cluster
identity from the parent via terraform_remote_state and the live
endpoint/CA from a data aws_eks_cluster source, so the provider is only
ever configured while the cluster exists. Apply parent then workloads;
destroy workloads then parent.
purgeClusterInfra stranded the cluster security groups two ways, leaving
the nodegroup/control-plane SGs (which live in the customer VPC) behind to
pin aws_vpc destroy on DependencyViolation:
* an early return before the SG cleanup block skipped DeleteClusterSGs
whenever any earlier best-effort step (NLB / ENI / VM / EIP) errored;
* DeleteClusterSGs failures were warn-and-swallowed, so a nodegroup SG
still referenced by a terminating worker's ENI leaked while the cluster
completed deletion — no reaper retry ever fired.
Surface DeleteClusterSGs / DeleteClusterCPVPC / DeleteClusterIGW failures
into teardownErrs and gate the KV sweep on them, so a leaked VPC-pinning SG
keeps the cluster DELETING for a retry. Add a bounded DependencyViolation
retry to the SG delete (mirroring deleteENIAwaitingTerminate) to ride out
the worker-ENI-detach race in the common case.
Fixes the manual 'revoke rules + delete SGs before the VPC will destroy'
teardown step.
* fix: persist instance TagSpecifications in RunInstance
RunInstance built the ec2.Instance metadata but never read
input.TagSpecifications, so DescribeInstances returned untagged
instances and tag filters matched nothing. EKS node groups discover
their workers by the spinifex:eks-cluster / spinifex:eks-nodegroup
tags, so scale-up convergence and worker reconciliation depended on
tags that were silently dropped.
Apply instance-scoped TagSpecifications onto the built instance so the
tags surface through DescribeInstances and tag filters.
* fix: serialize nodegroup reconcile via CAS to prevent scale overshoot
UpdateNodegroupConfig is dispatched on a queue group, so two
overlapping requests (e.g. an SDK retry mid-flight) each read
current = len(InstanceIDs) and each launched desired-current workers,
overshooting the target (1 -> 5 for desired=3) and stranding untracked
VMs via last-writer-wins on the unconditional PutNodegroupRecord.
Make every durable nodegroup-record mutation a compare-and-swap on the
KV revision. Scale-up launches one worker at a time and CAS-appends it
only while the record is below the live ScalingDesired, terminating any
surplus VM launched into an already-full record. Scale-down terminates
before the CAS truncation so a crash or retry leaves reclaimable dead
IDs rather than orphaned VMs. A concurrent regression test asserts two
overlapping scale-ups converge to exactly desired under -race.
* fix: return securityGroupRuleSet from AuthorizeSecurityGroup ingress/egress
The AWS provider's aws_vpc_security_group_ingress_rule reads
SecurityGroupRules[0].SecurityGroupRuleId on create; an empty ruleset
panics the plugin (Plugin did not respond), though the rule applies
server-side. Echo the created rules via sgRuleToSecurityGroupRule, the
same shape DescribeSecurityGroupRules already returns.
* fix: surface InsufficientInstanceCapacity from nodegroup scale, not 500
launchWorkersCAS wrapped the bare awserror code via fmt.Errorf, so the
gateway ErrorHandler's exact-match against ErrorLookup missed and remapped
to InternalError (500). An out-of-capacity scale now surfaces the real
InsufficientInstanceCapacity code (400); opaque internal failures still
wrap and stay 500. Adds awserrors.HasErrorCode for the membership test.
* fix: cut EKS node boot ~50s — drop cloud-final deadlock + syslinux menu
Worker/server user-data runcmd ended with `[ rc-service, local, start ]`,
which runs inside cloud-final while the OpenRC `local` service is ordered
after it — a deadlock broken only by OpenRC's ~50s start timeout, stalling
the k3s join. Run the IMDS route script directly instead and keep
`rc-update add local default` for reboot persistence.
setup.sh: cut the SYSLINUX boot-menu countdown 10s->~1s (kept nonzero so the
menu stays interruptible over serial).
Validated live on env19 CP: boot menu 1s, no `waiting for cloud-final`
stall, apiserver ready ~66s (was ~109s+).
mulga-siv-332
* feat: add argocd managed addon + bake bundle into eks-node AMI
The addon catalog advertised coredns and aws-load-balancer-controller, but
the only baked manifest shipped in the eks-node AMI was the spinifex-noop
fixture. Installing any real addon failed with 'no baked bundle for version'
once the VM-side renderer found nothing on disk.
Add argocd v3.0.23 as a real, demonstrable managed addon: catalog entry (no
IRSA), a flattened namespaced install manifest baked into the AMI, and the
manifest.conf bake line. The eks-addons-access workbook switches from coredns
(redundant on K3s, never baked) to argocd.
The baked bundle is generated from upstream install.yaml via kubectl kustomize
with namespace: argocd plus a prepended Namespace, since upstream is
namespace-agnostic and K3s auto-deploy applies with no -n flag.
* fix(eks): run CoreDNS as a node-local DaemonSet for worker DNS
k3s ships CoreDNS as a single-replica Deployment that tolerates the CP taint
and strands itself on the control-plane node. In the managed-CP topology the
worker<->CP pod overlay cannot cross the VPC boundary, so worker pods cannot
reach a CP-resident CoreDNS and lose all cluster + external DNS.
Skip the bundled CoreDNS (coredns.yaml.skip, staged in k3s.initd like traefik)
and deploy a Mulga-owned variant: a DaemonSet (one pod per node) plus kube-dns
internalTrafficPolicy=Local, so every node resolves via its own CoreDNS. Immune
to the cross-VPC overlay gap and topology-agnostic (single-node and managed-CP
both work). Image-side only; no handler change.
* fix(eks): revoke cross-referencing SG rules before delete on teardown
DeleteClusterSGs deleted the control-plane and nodegroup SGs directly, but they
cross-reference each other (EnsureNodegroupSGRules), and AWS refuses to delete a
security group still referenced by another SG's rule. Both deletes failed with
DependencyViolation, the SGs leaked, and the VPC could not be deleted (tofu
destroy hung on DeleteVpc).
Resolve both SG IDs, revoke all ingress on each to break the cp<->ng cycle, then
delete both. Adds RevokeSecurityGroupIngress to sgProvisioner (already on
VPCServiceImpl) and revokeAllSGIngress, with a regression test that models AWS's
DependencyViolation.
* fix(eks): advertise NLB front-end so worker pods reach the apiserver ClusterIP
In the managed-control-plane topology the apiserver advertised its Set-B
node-ip into the in-cluster `kubernetes` Endpoints. That IP lives in the
unpeered CP VPC and is unreachable from worker pods, so every worker pod
using the apiserver ClusterIP (CoreDNS kubernetes-plugin sync, controllers,
argocd redis secret-init) failed.
- k3s_server_vm.go: set advertise-address to the worker-reachable NLB
front-end (PrivateEndpointIP, else EndpointIP); both are already cert SANs.
- nlb.go: add a second :6443 listener on the cluster NLB (same CP target
group) so PrivateEndpointIP:6443 — the advertised endpoint — is served.
- security_groups.go: admit :6443 on the private-endpoint SG.
Validated on a live managed-CP cluster: worker CoreDNS /ready 200, all argocd
pods Running, kubectl logs on worker pods working.
The DisassociateAccessPolicy route regex terminated at
/access-entries/{principalArn}/access-policies, but the AWS EKS REST API
sends the policy ARN as the final path segment:
DELETE /clusters/{name}/access-entries/{principalArn}/access-policies/{policyArn}
The route never matched, so the gateway returned InvalidActionException
(seen as a 400 on 'tofu destroy' of aws_eks_access_policy_association).
The handler also read policyArn from the request body, which AWS leaves
empty for this DELETE.
Capture policyArn as a third path param and pass it through to the input,
dropping the body read. Update route and wrapper tests accordingly.
The host clock needs an NTP daemon; without one it drifts and SigV4 request
signatures fall outside the skew window, kicking the web console back to /login.
Mirrors the ISO rootfs (iso-builder packages.list) so both host-install paths
ship the same software. Guest VMs track the host via kvm-clock.
Bug: mulga-siv-335
* fix: preserve terminated-instance describe-visibility window in GC reaper
The TerminatedTeardownReaper purged a terminated record the moment it drove
the instance's pending NAT/OVN teardown to done. Normal terminations always
arrive NAT/OVN-pending, so the first 2m GC sweep completed and purged them
immediately, removing the instance from DescribeInstances before clients could
observe state=terminated — an AWS-compat divergence that flaked e2e-single
TestRunInstancesMultiCount.
Stamp TerminatedAt on transition and purge a completed record only once it is
older than a 15m visibility window; fresher records are left to the terminated
bucket's 1h TTL, keeping the instance describable as terminated. The ADR-0003
§3 purge-on-completion invariants still hold (their seeded records predate the
window). Closes mulga-siv-328.
* test(e2e): poll intra-SG east-west ping until L2 converges
TestSameSGComms fired a single un-retried 'ping -c 3 -W 2' immediately after
the source instance's SSH became reachable, but the ICMP target is the
destination's east-west data plane. Two instances boot in parallel and the
destination guest NIC / OVN port flows come up several seconds after the EC2
running transition the launch waits on. When the probe wins that race the
source kernel reports 'Destination Host Unreachable' (ARP unresolved), not an
ACL drop, and the test flaked.
Replace the one-shot ping with pingConverged, a bounded poll that retries
until 0% packet loss or 45s. A genuine self-ingress/datapath break still
stays at 100% loss for the full window and fails, so coverage is preserved.
* feat: join nodegroup workers via cluster NLB endpoint
Under the Set-B managed-CP-VPC topology the control plane runs in the
system-account CP VPC while nodegroup workers launch in the customer
VPC. The agent user-data set K3S_URL to the in-VPC control-plane ENI IP
(:6443), which sits in a different, unpeered VPC and is unreachable, so
the join path was dead.
Point K3S_URL at the published cluster NLB endpoint (meta.Endpoint,
https://<frontend-ip>:443, the :443->:6443 listener kubectl already
uses). Guard CreateNodegroup on a non-empty endpoint.
mulga-siv-298
* feat: gate nodegroup ACTIVE on workers registering Ready
launchNodegroupInfra flipped the nodegroup to ACTIVE the moment
RunInstances returned, so a worker that booted but never joined left the
nodegroup falsely ACTIVE with no usable capacity.
Gate ACTIVE on the cluster's Ready-node count (meta.NodeCount, refreshed
by the cluster reconciler from the CP NATS state report — Ready nodes
only) rising by the desired worker count over the create-time baseline.
On timeout the nodegroup is marked CREATE_FAILED with its instance IDs
retained for the reclaim path. Mirrors the CP reconciler's
healthy-observe gating. Bounded by injectable timeout/poll knobs.
mulga-siv-299
* feat(eks): private endpoint access via cross-account Set A ENI
Make EndpointPrivateAccess real: in-VPC workers and kubectl reach the
control plane through a customer-VPC (Set A) resident ENI on the cluster
NLB's LB VM, removing the public-NLB hairpin and the NAT-GW egress path.
Teardown: the Set A ENI is an extra cross-account NIC on the LB VM, not
its primary, so the instance-terminate cascade never reclaims it. Detach
(store-clear) the stale attachment before deleting, so private clusters
delete cleanly instead of looping on InvalidNetworkInterface.InUse.
Refs: mulga-siv-301
* fix(vpc): honor SecurityGroupRuleIds in RevokeSecurityGroup ingress/egress
Modern Terraform (aws_vpc_security_group_ingress_rule) revokes per-rule
resources by SecurityGroupRuleId with empty IpPermissions. The handlers
only read IpPermissions, so they returned success while removing nothing;
the surviving SG-reference rule then pinned both SGs undeletable
(DependencyViolation), deadlocking destroy of mutually-referencing SGs.
Resolve SecurityGroupRuleIds to their stored rules and remove them
(malformed -> InvalidSecurityGroupRuleId.Malformed, absent -> NotFound).
Refs: mulga-siv-321
* fix(vpc): reap the main route table on DeleteVpc
CreateVpc auto-creates a main route table and DeleteRouteTable refuses to
delete a main RT (AWS-faithful), so nothing reclaimed it — every deleted
VPC leaked one orphaned main RT in the rtb KV bucket. DeleteVpc now reaps
the VPC's main route table before deleting the VPC record.
Refs: mulga-siv-322
* fix: make AttachRolePolicy and AssociateRouteTable atomic via KV CAS
Concurrent writes to a single role / route table record used a blind
read-modify-Put with no optimistic-concurrency guard, so parallel writers
racing from the same revision clobbered each other and lost updates.
Verified on env19: three parallel policy attachments to one node role kept
only one; two parallel subnet associations to one route table kept only
one (the provider then timed out waiting for state 'associated').
Route the mutating paths (AttachRolePolicy/DetachRolePolicy,
AssociateRouteTable/DisassociateRouteTable) through a revision-guarded
Update + bounded retry, matching the existing accountCounterBucket CAS
idiom. Add concurrent regression tests covering both lost updates.
* docs: add EKS Terraform workbooks (quickstart, addons-access, https-ingress)
Three OpenTofu workbooks demonstrating EKS on Spinifex, each ending in a
browsable load-balanced demo app:
- eks-quickstart: minimal single-node cluster + NodePort demo
- eks-addons-access: CoreDNS managed addon + access-entry RBAC
- eks-https-ingress: 3-node cluster behind an HTTPS ALB terminating TLS
with an ACM-imported self-signed certificate
Each covers the Spinifex-specific endpoints, the auto-managed worker SG
NodePort pattern, and the omitted addon_version constraint.
* fix: idempotent Delete* across VPC family + volume (RLC1)
Delete handlers now return empty success when the resource is already
absent (nats.ErrKeyNotFound), instead of *NotFound. Transient read
errors still map to ServerInternal. Covers DeleteVpc, DeleteSubnet,
DeleteSecurityGroup, DeleteNetworkInterface, DeleteInternetGateway,
DeleteNatGateway, DeleteRouteTable and DeleteVolume.
Makes destroy retries converge instead of failing mid-graph, removing a
source of orphaned/zombie resources between deploys. Adds a
TestRLC1_*DeleteIdempotentOnAbsent invariant per package and updates the
daemon delegation table; supersedes the old absent->NotFound tests.
* feat: prune orphan ENI LSPs in reconcile applyPorts (ADR-0003 §4)
applyPorts was create-or-update only, so a guest LSP whose
spinifex:eni_id no longer maps to an intent ENI survived instance
terminate and VM-host reinstall — the orphan-LSP capacity-drift root
cause. It now takes pruneOrphans (mirroring applySGs): on a full
reconcile it lists LSPs and tears down each managed ENI port absent from
intent via the composed topology.DeletePort cascade (clears port-group
memberships, then deletes the LSP). The apply-only startup path does not
prune, preserving in-flight ports until subscribers converge.
Adds ListLogicalSwitchPorts to the OVN client (live + mock). Invariant
TestRLC5_OrphanLSPPrunedAfterReconcile guards ADR-0003 §4; a startup-race
test guards the apply-only path.
* feat: idempotent + tracked instance terminate (ADR-0003 §1/§2)
Make terminate idempotent across all three layers and break the
un-terminable-ENI deadlock, and record per-dependency teardown so the
GC backstop can later tell a fully-reaped instance from a dangling one.
- vm.Manager.Terminate: absent / already-terminated / shutting-down now
return nil instead of NotFound / invalid-transition (ADR-0003 §2).
- VM.Teardown map + teardown.go marks: sync steps stamped Done/Failed via
cleaner error returns; fire-and-forget OVN LSP and NAT rule stamped
Pending for the reconciler / GC reaper (ADR-0003 §1).
- InstanceCleaner: DeleteVolumes / ReleasePublicIP / DetachAndDeleteENI /
RemoveFromPlacementGroup / ReleaseGPU now return error.
- vpc ForceDeleteInstanceENI bypasses the in-use guard for the owning
instance, breaking the failed-detach deadlock; plain delete still
rejects an in-use ENI held by another instance.
- gateway TerminateInstances: KV-health-gated idempotent absence (a failed
terminated-bucket query does not fabricate success, ADR-0003 §3).
- Invariants: TestRLC1_TerminateIdempotentOnAbsent, TestRLC4 teardown-map
stamping + failed-dependency, TestRLC_ENINonDeadlock.
* feat: reconciler GC backstop framework + terminated-teardown reaper (ADR-0003 §3)
Add the shared reality->desired garbage-collection framework (inverse of
Restore) and its first reaper, closing the gap 295.3 deferred: nothing
retried pending/failed teardown or purged the terminated record when its
dependents reached done.
- vm/gc.go: GarbageCollector + Reaper interface (Class/Scope/Sweep).
KV-health gated — holds and alarms when the desired-state cannot be
trusted, never reaps against it. ClusterWide reapers run only on the
elected leader (optional elector); NodeLocal run on every node. Start
sweeps immediately then every GCInterval.
- vm/teardown_reaper.go: TerminatedTeardownReaper (node-local). Scans this
node's terminated records, re-drives each not-done dependent through the
idempotent cleaner, and purges the record once all reach done. Records
already complete on arrival are left to the bucket's 1h TTL.
- StateStore gains DeleteTerminatedInstance (JetStream impl already existed).
- daemon: wires the GC after the pending watchdog, gated on a cheap
JetStream AccountInfo health probe.
- Cluster OVN-LSP arm already lands via the reconciler drift prune (295.2).
- Invariants: TestRLC5_GCRetriesPendingTeardownToDoneThenPurges,
TestRLC5_GCNeverPurgesIncompleteRecord, TestRLC5_GCHoldsWhenKVDegraded,
TestRLC5_GCSkipsOtherNodesRecords.
* feat: VPC-family dependency guards (ADR-0004 rule #3)
DeleteSubnet blocks while a live ENI attachment (a resident instance)
remains; available/orphan ENIs do not pin it. DeleteNetworkInterface
guards on the structured attachment (eniIsLiveAttachment), not a bare
in-use string. DeleteNatGateway blocks while a live route table forwards
to it and disassociates its EIP keeping the allocation (AWS parity);
CreateNatGateway now marks the EIP associated so the disassociate is
real and a second NAT GW cannot reuse the allocation.
Binding invariants: TestRLC3 dependency-guard completeness across the
subnet/ENI/NAT-gateway handlers.
* feat: EBS volume data-safety GC override (ADR-0005)
The shared GC backstop reaps actual-desired for every other class; for
volumes that default is forbidden because they carry customer data. Add
VolumeLeakReaper (node-local, rides the 295.5 GC): it marks a volume left
attached to a definitively-gone instance as orphaned (a non-destructive
tag) and alarms, but NEVER issues a delete. The daemon feeds it only this
node's terminated instances whose volumes-teardown failed, so the shared
predastore scan can never false-mark another node's live-instance volume.
DeleteVolume idempotency + the detach/snapshot guards + the
DeleteOnTermination teardown retry already landed (295.1/295.3/295.5).
Binding invariants: TestRLC5 (GC marks+alarms, never deletes data),
TestRLC3 (detach-before-delete).
* test(elbv2): lock ADR-0002 §3 live-only TG in-use guard (mulga-siv-295.4)
* feat(eks): idempotent DeleteCluster on absent + ADR-0006 lifecycle invariants (mulga-siv-295.8)
DeleteCluster returns empty success when the cluster meta is already swept
(Common Resource Lifecycle Contract #1), so tofu destroy retries converge.
Adds TestRLC1 (idempotent delete), TestRLC2 (billable-before-KV-sweep ordering)
and TestRLC3 (NLB target-group no-orphan, riding ADR-0002/172).
* fix(eks): GC reaper terminates orphan control-plane VM after cluster delete (mulga-siv-294)
A normal DeleteCluster can leave a running control-plane VM behind (terminate
completes async; a daemon restart re-launches it from saved running-state) after
the cluster meta is swept — a billable orphan with no owner. Add EKSBillableReaper
on the 0003 GC backstop: a running managed-by=eks CP VM whose cluster meta is
definitively gone is terminated. The CP ENI now carries the customer-account tag
so a system-account VM can be mapped back to its cluster meta; the reaper never
reaps on uncertainty (ENI gone/untagged or meta present).
* feat: reclaim orphan EKS NLB + NAT-GW EIP in billable reaper
The billable GC reaper terminated an orphan control-plane VM but left the
cluster's NAT-gateway EIP and NLB front-end NAT behind when the meta was
gone. After a successful terminate, run the name/tag-driven DeleteClusterNLB
+ DeleteClusterCPVPC teardown (idempotent, meta-independent) so the billable
NAT-GW EIP and NLB NAT are reclaimed too. ADR-0006 §5.
* fix: persist NAT-GW & route-table tags and reorder CP-VPC teardown to reclaim EKS NAT-GW EIP
CreateNatGateway and CreateRouteTable both dropped input.TagSpecifications,
so the tag-driven DeleteClusterCPVPC teardown could not discover the cluster
NAT gateway or its private route table. The explicit DeleteNatGateway +
ReleaseAddress were skipped and the billable NAT-GW EIP leaked under the
system account on every clean DeleteCluster.
Three fixes:
- CreateNatGateway persists ExtractTags(TagSpecifications, natgateway).
- CreateRouteTable persists ExtractTags(TagSpecifications, route-table).
- DeleteClusterCPVPC tears down route tables before the NAT gateway, so the
rule #3 route-ref guard (DependencyViolation / InvalidAddress.Locked) is
not tripped and the EIP releases.
Adds tag-persistence regression tests for both create paths and a teardown
ordering invariant (TestRLC5). Validated live: NAT-GW EIP released cleanly,
no DependencyViolation/Locked.
* fix: route EKS control-plane VM terminate cross-node + DELETING teardown backstop
Multi-node EKS cluster delete wedged in DELETING and stranded the billable
NAT-GW EIP. Two root causes, both invisible on a single-node box:
Cross-node terminate was never routed. TerminateSystemInstance was node-local,
so terminating a CP VM owned by another node returned NotFound (treated as
idempotent) and the teardown deleted a still-attached ENI
(InvalidNetworkInterface.InUse). That error short-circuited purgeClusterInfra
before the managed CP-VPC teardown that releases the NAT-GW EIP. Split
TerminateSystemInstance into a local body plus a routing wrapper that issues
system.TerminateInstance.<id> to the owning node. Bind that subscription for
every system-managed VM (elbv2 + eks) via the OnInstanceUp funnel, so CP VMs
placed on the coordinator's own node (and recovered after a daemon restart) are
reachable too, not only remotely-launched ones.
No DELETING backstop. A failed synchronous DeleteCluster left the cluster
DELETING with nothing to re-drive it. Add EKSDeletingReaper (vm.Reaper, class
eks-deleting) that re-runs purgeClusterInfra for clusters wedged past a min-age,
gated by the per-cluster reconciler leader lease. Stamp ClusterMeta.DeletingSince
on the transition into DELETING.
TerminateK3sServerVM additionally waits out any residual ENI-InUse with a bounded
budget as defense-in-depth.
Validated live on env19 3-node: first delete-cluster converges in 29s, both
remote CP VMs terminated cross-node, NAT-GW EIP released, backstop never fires.
* fix(network): gate gateway reconcile on real datapath reachability
Post-reboot the external OVN datapath can stay dead while every
control-plane signal is green: the chassisredirect SB binding is claimed
(up=true), NAT rows are present and the host neigh is primed, yet
ovn-controller leaves stale gateway/localnet flows so EIP north-south and
the VM->gateway heartbeat forward nothing. ensureGatewayClaimed only
nudges recompute when the binding is unclaimed, so a claimed-but-dead
datapath is never detected or repaired.
Add ensureGatewayDatapath: after the SB claim converges, probe the
gateway LRP IP (OVN answers ICMP natively, no guest or security-group
dependency), and on a miss force one ovn-controller recompute and
re-probe until a short deadline. Reuses the existing recompute nudge and
no-ops when no verifier is wired or no gateway IP resolves.
* fix: self-heal external uplink after reboot boot-race
Post-reboot, ovs-vswitchd/ovn-controller order after the passive
network.target rather than network-online.target, so they start
concurrently with systemd-networkd. Under a heavy boot the WAN-glue veth
(veth-wan-ovs) can come up admin-down before networkd brings it up,
leaving br-wan with no carrier and the external datapath dead while every
control-plane signal reconciles green.
Broaden the datapath health-gate remediation from recompute-only to
RepairDatapath: re-assert the veth admin state (covers the admin-down
link, which a flow recompute cannot revive) then recompute (covers a
stale gateway ofport). Idempotent; the veth ops are skipped in
physical-uplink mode where the device is absent.
Also capture ip -d link (veth admin/carrier) and ovs-ofctl show br-ext
(ofport) in the reboot diag bundle to disambiguate the two failure shapes
from CI artifacts.
* fix: restore AWS-faithful Delete* NotFound; move idempotency to teardown callers (mulga-siv-306)
RLC1 (f9218982) made EC2/VPC/EBS Delete* handlers return success on an absent
resource, breaking AWS compatibility and the cross-account isolation signal (a
cross-account delete reads as absent -> success instead of *.NotFound).
Restore the AWS contract at the API while keeping lifecycle convergence:
- Handlers return canonical *.NotFound on absent: DeleteVpc, DeleteSubnet,
DeleteSecurityGroup, DeleteInternetGateway, DeleteNatGateway, DeleteRouteTable,
DeleteVolume, and the public DeleteNetworkInterface (ForceDeleteInstanceENI
keeps its idempotent force path, ADR-0003 section 2).
- Add awserrors.IsNotFound; destroy orchestration (instance terminate, EKS
CP-VPC teardown, EKS IGW/SG cleanup, k3s rollback, ELBv2 ENI/SG teardown)
tolerates NotFound so re-issued/double-targeted deletes converge.
- Flip the RLC1 unit invariants to assert *.NotFound on absent; revert the
daemon round-trip cases; add IsNotFound + CP-VPC convergence tests.
ELBv2 deletes and TerminateInstances stay idempotent (AWS itself is).
* fix(network): gate ingress EIP datapath on guest SB Port_Binding convergence
TestMultinodeGuestSSH intermittently times out (Connection timed out, not
#355's reset-by-peer) connecting to a fresh trio instance's public IP after
the suite churns instances across all three chassis. The instance is running
and AddEIP has installed the DNAT row, flushed/primed the host neigh, and
barriered the gateway-chassis flow — but none of that proves the
gatewayLRP->guest geneve hop: until the guest ENI's southbound Port_Binding is
up on its hosting chassis the cross-node flow is missing and the DNAT-translated
packet blackholes.
Add ensureGuestPortDatapath, the guest-side analog of ensureGatewayDatapath:
after AddEIP, probe the guest LSP's SB Port_Binding (ovn-sbctl up column); on a
miss force one ovn-controller recompute (binds the port / installs the remote
flow) and re-probe until a short deadline. Recompute, not RepairDatapath: this
is flow/binding convergence, not a WAN-uplink fault. No-op when no verifier is
wired or the EIP carries no guest port. spec.PortName is populated for every
associated EIP regardless of NAT mode.
Extends GatewayClaimVerifier with GuestPortUp and wires the gate into
applyEIPs so it runs in the continuous reconcile loop on every node.
* Revert "fix(network): gate ingress EIP datapath on guest SB Port_Binding convergence"
This reverts commit 728b1a9cec6eaf5c7ef5e9eda4469f239b4d8fc0.
* fix(network): key EIP DNAT delete on external IP, not logical IP
A dnat_and_snat row's identity on the VPC gateway router is its external
IP. The logical (private) IP is not a stable key: the EIP pool round-robins
and instance private IPs are reused as instances recycle, so the same
logical IP is held by different EIPs over time.
DeleteEIP deleted the row by logical IP, so a stale or retried delete for a
freed EIP clobbered whichever live EIP now held that recycled private IP --
the next owner's row vanished with no re-add and its public IP went dark
(SSH SYN never left br-wan: Connection timed out). Lifecycle-resilience
teardown/GC retries amplified this by firing vpc.delete-nat repeatedly.
Key the delete on the external IP via DeleteNATByExternalIP and skip when
no external IP is supplied (no EIP means no dnat_and_snat row). A stale
delete now targets the freed EIP's own external IP (ErrNATNotFound no-op)
and can no longer touch the live row.
* refactor: relax DeleteNatGateway to AWS-faithful
Remove the route-reference dependency guard (checkNatGatewayRouteRefs) so a
NAT gateway deletes even while route tables still forward to it, matching AWS
(the route blackholes). The reconciler's loadNATGWs drops the SNAT once the
record leaves the active bucket, so a forwarding route is not a delete-blocking
dependent. EIP is still disassociated on delete (no leak).
* test: invert NAT-GW route-ref invariant to AWS-faithful
TestRLC3_NatGatewayBlocksWhileRouted asserted DependencyViolation while a
route forwarded to the NAT gateway, which is AWS-infaithful. Replace with
TestRLC3_NatGatewayDeletesWhileRouted: delete must succeed whether or not a
live route still targets it (the route blackholes; the reconciler drops the
SNAT). Per ADR-0004 §2.
* fix: tear down NAT-GW SNAT on delete (restore delete-time fan-out)
The guard commit (53ed55a9) dropped publishDeleteEventsForNatGateway on the
assumption a routed NAT GW could never be deleted. With the route-ref guard
removed, deleting a NAT GW while routes still target it left the SNAT in
place (applyNATGWs is create-only, so the reconciler never prunes it) and
egress kept working — TestNATGateway's post-teardown 'lost internet' check
failed. Restore the fan-out: on delete, scan route tables and publish
vpc.delete-nat-gateway per associated subnet so egress stops, while the
route entry survives as a blackhole (AWS-faithful).
* fix: probe EIP datapath in reboot self-heal, not the gateway LRP IP
The post-reboot gateway datapath self-heal probed the gateway LRP IP for
liveness, which OVN answers natively at the router — so it stayed green even
when the EIP NAT pipeline was stranded (ofproto/trace dropped EIP traffic at
table 0). RepairDatapath never fired and TestRebootResilience timed out, with
the drift loop re-probing the same false-green every cycle.
Prefer probing an associated EIP: force a fresh ARP of the dnat_and_snat
external IP (flush neigh, ping to trigger, read the resolved MAC). OVN answers
ARP for the external IP from the gateway chassis independent of the guest, so a
resolved MAC proves the WAN uplink forwards and the NAT flows are installed —
the signal the LRP IP cannot give. Fall back to the LRP IP for VPCs with no
EIP. The existing drift-loop cadence now re-heals a later flow strand.
* fix: lb-agent heartbeats over mgmt port, not the WAN/EIP datapath
The lb-agent heartbeat to awsgw is control-plane and must survive a host
reboot that strands the OVN/EIP north-south data plane. Customer
(internet-facing) lb-agents heartbeated over the WAN GatewayURL, riding the
EIP datapath, so after a reboot the ALB never reported healthy until the
datapath self-healed. System LBs already heartbeated over the on-link
br-mgmt NIC; this extends that path to every lb-agent.
buildLBAgentEnv now always uses mgmtGatewayURL (on-link MgmtBridgeIP, the
mgmt NIC the lb-agent VM already carries), falling back to the WAN URL only
when no mgmt bridge exists. Drops the account-id branch.
* fix: re-heal guest ENI SB Port_Binding post-reboot so EIP ingress converges
Post-reboot every EIP in a VPC goes dark north-south: the DNAT row, host
neigh and gateway-chassis flow are all in place (ARP to the EIP resolves)
but the gatewayLRP->guest flow is missing until ovn-controller binds the
guest ENI's SB Port_Binding, so the DNAT-translated packet blackholes and
the public IP is unreachable on an otherwise-running instance.
Restore the guest-port convergence gate in applyEIPs: probe the guest LSP's
SB Port_Binding up column; on a miss force an ovn-controller recompute and
re-probe until a deadline. Unlike the earlier reverted form, recompute on
every miss, not once: after a host reboot the guest tap is replumbed several
seconds after the reconcile runs, so a single early nudge fires before the
port exists and never binds it. The window is widened to span guest boot.
This is the southbound analog of the gateway datapath self-heal; the EIP
ARP probe cannot detect it because the gateway chassis answers ARP for the
external IP even while the guest delivery flow is absent.
* fix(network): force post-reboot EIP recompute on guest-port bind, startup-only
The reboot suite passed only by accident: the pre-#351 gateway-claim verifier
polled the always-chassis-less gateway LRP, so it fired an ovn-controller
inc-engine/recompute on every reconcile. That spurious recompute was load-bearing
- it reinstalled the centralised gateway DNAT flows that ovn-controller's
incremental engine misses on a cold (post-reboot) start. #351 correctly switched
the check to the chassisredirect port, eliminating the every-cycle recompute, and
the EIP datapath went dark post-reboot with every control-plane signal green.
Restore the kick, scoped to the post-reboot startup path: applyEIPs now gates a
single recompute on the guest ENI's SB Port_Binding chassis claim - the reliable
signal the tap is back, unlike the up column which stays stale-true across a
reboot. Drift never recomputes, preserving the #351 fix for freshly attached
EIPs. Probe renamed GuestPortUp -> GuestPortBound (chassis, not up).
* Revert "fix(network): force post-reboot EIP recompute on guest-port bind, startup-only"
This reverts commit 792e318211ef03efbe24febeb470749d88c0b03c.
* test(e2e/reboot): capture dataplane diag on ALB serve failure
The post-reboot ALB-serve assertion (8.6) failed with no flow/ARP evidence:
the OVN/OVS dataplane dumps only ran on the 8.5 targets-unhealthy path, but
8.5 passes on the daemon's own health view while the north-south EIP datapath
is dark, so the serve failure (8.6) left nothing to localise the drop.
On a serve failure, capture dumpALBProbe/dumpOVNDataplane/dumpOVNClaimState/
dumpHeartbeatDiagnostics/dumpAppDiagnostics before failing. Add to the OVN
dataplane dump: dump-flows br-ext and fdb/show br-ext (north-south bridge,
previously only br-int), and a real ARP probe that flushes the neigh entry
first so primeReachability's pre-seeded reachable entry can no longer mask an
unanswered ARP. Pure diagnostics; no production change.
* test(e2e/reboot): decisive datapath diag — untruncate br-int, real-ingress ofproto/trace, conntrack + tap capture
* test(e2e/reboot): add south->north return-path ofproto/trace to localize egress drop
* test(e2e/reboot): derive LRP MAC from realized flow + remediation experiment (recompute vs ovn-controller restart)
* test(e2e/reboot): dump persisted SB MAC_Binding + external MACs; flush experiments (conntrack vs MAC_Binding) to isolate recompute-resistant return-path drop
* test(e2e/reboot): untruncate Port_Binding + enumerate DGP/chassisredirect claim state (does cr-gw row exist + resident chassis)
* test(e2e/reboot): dump NB source of return-path drop (lflow stage, NAT external_mac/options, LR policy, LSP port_security)
* fix: exempt public-IP instances from subnet egress drop gate (mulga-siv-311)
Post-reboot reconcile installs the priority-1100 subnet egress drop gate for an
IGW-attached subnet that has no 0.0.0.0/0 route, but nothing punched the subnet's
public-IP instances through it. lr_in_policy runs before lr_out un-DNAT/SNAT, so an
inbound connection's reply still carries its private source when it hits the gate and
is dropped. The ALB/EIP datapath goes dark south->north post-reboot while NAT, ARP,
chassis and guest-port binding all read green; the pre-reboot path survives only
because the event-driven gate never evaluates a subnet created after IGW attach.
Install the existing priority-1200 /32 reroute (above the gate) for every public-IP
instance during reconcile. Reroute only: the instance's dnat_and_snat already supplies
SNAT, unlike EnsureSystemInstanceEgress which also adds a plain snat. Public IPs come
from two disjoint sources -- auto-assigned/ELB addresses on the ENI record and user
EIPs in the EIP bucket -- so drive the exemption from both intent.Ports (new
PortSpec.PublicIP) and intent.EIPs.
* refactor: scope public-IP drop-gate exemption to gated subnets
Only install the priority-1200 reroute when the instance's subnet actually carries
a drop gate. Routed subnets egress via their priority-1000 reroute and need no
exemption, so bounding on gate presence keeps the change inert for IGW-routed and
NATGW-routed subnets (no redundant or overriding policy). Adds a negative test.
* refactor: drop unused RemoveEIPInstanceEgress
No production caller; the exemption is install-only in reconcile. Removed the
interface method and impl rather than leave dead API surface.
* test: dump guest console + OVN/datapath diag on SG-to-SG client SSH timeout
TestSGToSGDatapath waited for client-vm SSH via EventuallyErr and reported
only 'exit status 255' on timeout — no guest-side or datapath state. When the
fresh public-IP VM is unreachable for the full window the failure was
undiagnosable from CI artifacts.
Swap the wait to the non-fatal trySSHReady probe and, on timeout, call
DumpVPCFlowDiagnostics (guest console tail, ovn-nbctl/sbctl show, NAT rows,
br-int flows / conntrack / host ARP keyed on the public + private IP) before
Fatal, mirroring vpc_test/baseline_test. Captures persist to the artifact dir
for the Stage 2 analyzer bundle.
* fix: ownership-scope public-IP teardown to survive IP recycling
A live instance's dnat_and_snat rule, host ARP, and IPAM lease were torn down
by the delayed garbage-collection teardown of a different, long-dead instance
that previously held the same recycled public IP. The 2-min vm/gc
instance-teardown reaper re-runs ReleasePublicIP on a stale terminated record;
because the external IPAM round-robins a small pool, the IP had already been
reassigned to a live instance, and both teardown primitives acted on the IP by
value with no ownership check:
- policy.DeleteEIP deleted the dnat_and_snat row by external IP alone,
discarding the logical IP it receives.
- StaticPoolAllocator.Release freed the pool entry by IP value.
Scope both to the current owner so a stale or duplicated teardown for a
recycled IP is a no-op:
- DeleteEIP looks up the row by external IP and skips the delete (and the ARP
flush) when its logical IP no longer matches, mirroring AddEIP's idempotency
lookup. Defeats both external- and logical-IP reuse races.
- Allocator.Release / ExternalIPAM.ReleaseIP take an ownerENIID; the static
allocator no-ops a release whose ENI differs from the live allocation, and
treats an already-freed owner-scoped release as idempotent. Teardown and
rollback callers pass their ENI; user-driven EIP release passes none.
Proven via E2E single TestSGToSGDatapath (run 27515293563): client-vm SSH
black-holed after its NAT was deleted by the dead instance's GC sweep.
* fix(eks): single-flight DeleteCluster teardown per cluster
SDK retries of one DeleteCluster get fanned across nodes by the worker
queue group, so purgeClusterInfra raced the same cluster on multiple
nodes: duplicate ENI/NLB teardown, duplicate TargetGroupNotFound
deregisters, and a real double-release risk on the billable NAT-GW EIP.
Idempotency kept it correct but noisy.
Add a dedicated per-cluster teardown lease (teardownLeaderKey =
reconcilerLeaderKey + "/teardown"), distinct from the reconciler lease
so a delete never contends with the still-live reconciler that holds the
reconciler lease until it observes DELETING and exits. DeleteCluster
acquires it after the NotFound-idempotency check: the loser returns the
cluster as DELETING (AWS-async semantics) without running the teardown;
the winner drives purgeClusterInfra and releases. The DELETING backstop
reaper shares the same lease, so at most one purgeClusterInfra runs per
cluster cluster-wide.
A nil leaderKV (single-node) skips gating, so local behavior is
unchanged.
* fix: env-driven SSH-ready budget for slow baremetal guests
Add SPINIFEX_SSH_READY_TIMEOUT override for the single-node SSH-handshake
probes (default unchanged at 3m). Baremetal boots q35 guests slower (OVMF +
NBD-backed disks) than CI loop devices and intermittently misses the 3m
window; the harness can now widen it to 5m without affecting shared-runner
e2e. Applied at all single-node probe sites; failure messages now report the
actual budget.
* fix: shared external switch to stop per-VPC localnet ARP collision
Per-VPC external switches each attached an untagged localnet port to the same
physical network (br-ext). OVN cannot keep multiple untagged localnets on one
physical network L2-separate, so with two or more concurrent public-IP VPCs ARP
for a public IP landed on the wrong VPC's switch and the owning gateway router's
ARP responder never fired, leaving the public IP unreachable.
Move to a single shared external switch (provider-network pattern): one switch
with one localnet on network_name=external. Each VPC's gateway LRP attaches via
its own router-type switch port; per-VPC NAT isolation is unchanged. AttachIGW
ensures the shared switch idempotently and is migration-safe; DetachIGW removes
only the per-VPC port + LRP and preserves the shared switch. DeleteVPC removes
the per-VPC gateway switch port from the shared switch so it does not dangle.
Reconcile detects attachment via the gateway LRP since the shared switch carries
no vpc_id.
* fix: owner-scope DeleteEIP by ENI port to survive identical pair recycling
A stale or GC-re-emitted vpc.delete-nat for a terminated instance tore down a
live instance's dnat_and_snat rule (and flushed the host ARP) when the public
IP and private IP both recycled, together, to that live instance in the same
VPC. The (external_ip, logical_ip) pair-key matches under identical recycling,
so the existing logical-IP guard fell through.
Stamp the owning ENI port (spinifex:logical_port) on the dnat_and_snat row at
AddEIP, thread the port through handleDeleteNAT -> DetachEIP -> DeleteEIP, and
skip the delete when the row's stamped port differs from the incoming one. The
IPAM release was already owner-scoped; this mirrors that guard at the NAT layer
so any stale delete for a recycled pair is a no-op. Empty port preserves the
prior pair-key behaviour.
* chore: bump viperblock to encrypted-snapshot VolumeUUID nonce fix
Pull viperblock 1044c2a (feat/snapshot-uuid-nonce-fix) so E2E builds against
the fix for the encrypted AMI snapshot metadata tag-verify failure
(cipher: message authentication failed) on instance launch. Pairs with
viperblock PR #22.
* fix: forward viperblock master key to nbdkit mount path
viperblockd loaded the master key but never passed it to the nbdkit
plugin it spawns for volume mounts, so encrypted boot volumes opened
in cleartext and failed LoadState with an encryption-state mismatch.
Forward cfg.EncryptionKeyFile into NBDKitConfig and emit the plugin's
encryption_key_file= argument when set.
* chore: bump viperblock to fresh-volume VolumeUUID mint fix
* chore: bump viperblock to snapshot-link LoadState ordering fix
* fix: route encrypted-volume config writes through viperblock keyholder
config.json is a sealed VBState whose AES-GCM tag and StateSeqNum-derived
nonce can only be advanced by the master-key holder that owns the volume.
The EC2 volume handler wrote it directly to S3, so mergeVolumeConfig had to
refuse encrypted volumes (rewriting would strip the tag) — bricking
UpdateVolumeState (attach in-use / detach available) and ModifyVolume.
Route encrypted-volume config updates through viperblockd, mirroring the
ebs.snapshot idiom: per-volume ebs.config.{volumeID} hits the owning node's
live VB (the single StateSeqNum writer); on ErrNoResponders the ebs.config
queue group opens the detached volume exclusively and reseals. Cleartext
volumes keep the direct-S3 fast path. No crypto leaves viperblockd.
mulga-siv-323
* fix: unwrap encryption envelope when snapshot handler reads config.json
CreateSnapshot and snapshotInUseByVolumes decoded config.json straight into
a VBState. For encrypted volumes config.json is an at-rest envelope
{payload, authtag}; a raw decode yields a zero VBState (SizeGiB==0), so
CreateSnapshot 500s on the size guard and the in-use scan silently drops
encrypted volumes' snapshots. Unwrap via viperblock.StateBody first,
mirroring the volume handler's mergeVolumeConfig path.
mulga-siv-324
* test: cover viperblockd ebs.config handlers
Unit/integration coverage for applyConfigUpdate, makeConfigUpdateHandler
(ebs.config.{volumeID}), and the ebs.config queue-group fallback: grow-only
size reconcile, live-VB apply, invalid JSON, apply errors, and the
detached-volume open path. Clears the diff-coverage gate (68.8%).
* chore: pin viperblock to dev tip (#22 merged)
docker export of a container leaks its overlay mount on Docker's
overlay2 driver — the mount survives docker rm, so the layer dirs are
never reaped except by a daemon restart. The lb-agent microVM build ran
docker export on every e2e job across ~9 runners, accumulating ~69G of
orphaned overlay2 layers on the CI host (disk hit 94%).
Build the rootfs with `docker buildx --output type=local` from a small
generated Dockerfile instead, which writes the image filesystem to a
directory and releases all mounts — nothing leaks. Keep the podman path
(its store is unaffected). Tighten the e2e cleanup image prune to
dangling-only so the alpine base stays cached instead of re-pulling.
CI e2e-build compiles the spinifex module with GOWORK=off, resolving the
go.mod pins instead of the go.work local submodule redirect. v1.7.0 of both
modules predates symbols spinifex now calls (viperblock.StateBody,
predastore auth.ParseRoleARN), so the build failed with undefined-symbol
errors that go.work masked locally. Pin both to current dev-branch
pseudo-versions until a coordinated release is cut.
* feat: enable viperblock at-rest encryption by default on fresh installs
Fresh installs left nodes.<node>.viperblock.encryption_key_file empty, which
viperblockd treats as legacy cleartext mode. The encryption machinery existed
but the install path never generated a key or set the field.
Provision a cluster-wide viperblock master key in all three fresh-install
flows and render its path into spinifex.toml:
- admin init (single node): generate the key locally.
- admin init --multi-node (leader): generate once, save locally, and
distribute via the formation server (mirrors IAM master key distribution).
- admin join (joiner): receive the shared key from the leader rather than
minting its own, so volumes sealed on any node open everywhere. Lenient on
absence — an older leader returns no key, so the joiner warns and stays
cleartext instead of failing the join.
The key file is omitted from the template when unset, so upgrades with no
existing key keep running cleartext.
* fix: make viperblock encryption key group-readable for both readers
The key is loaded by two services per node — viperblockd (spinifex-viperblock)
and the awsgw EC2 handlers (spinifex-gw), both in the spinifex group — so it
must be group-readable, not owner-only like the per-node predastore key.
The previous commit generated the key but never registered its path with the
deployment ownership machinery, leaving /etc/spinifex/viperblock as root:root
0700 (non-traversable) and the key as root:root 0600. The daemon failed at
startup with "stat ... permission denied".
Mirror master.key: setup.sh creates /etc/spinifex/viperblock (0750), and
SetServiceOwnership chowns the dir to spinifex-viperblock and promotes the key
to root:spinifex 0640.
* fix: read viperblock config.json through StateBody for encrypted volumes
The metadata layer parsed config.json with json.Unmarshal/NewDecoder directly,
which fails on encrypted volumes/AMIs now that viperblock wraps VBState in a
valid-JSON envelope ({"v":1,"payload":...,"authtag":...}). DescribeImages
silently skipped encrypted AMIs (count=0 → "no AMIs available"), bricking the
whole launch path.
Route every config.json read through viperblock.StateBody, which unwraps the
envelope (no key needed — the metadata is authenticated, not encrypted) and is
a no-op for unencrypted volumes:
- image: DescribeImages, getVolumeConfig, amiNameExists, GetAMIConfig
- volume: getVolumeConfigAndEncryption, and mergeVolumeConfig (so the
refuse-to-merge-encrypted guard sees the real VBState instead of clobbering
the envelope with a tag-less blob)
- admin: readAMIConfig, readVolumeConfig
* feat(eks): VM-side managed-addon delivery transport
Stage CreateAddon manifests in KV host-side; the control-plane VM pulls the
staged set through the AWS gateway (signed GET /clusters/{name}/internal-addons/
{accountId}), renders the baked bundles into the K3s auto-deploy dir, and
reports per-addon delivery status on a dedicated eks.addon.*.status channel that
the cluster reconciler CASes onto AWS-visible AddonStatus (CREATING->ACTIVE).
- gateway: ListInternalAddons route + ListStagedAddonManifests NATS action
- handlers: addon status reconciler (nextAddonStatus), staging installer
- eks-node AMI: eks-gateway-fetch binary + mulga-eks-addon-sync agent
- agent renders idempotently (cmp before replace) so K3s auto-deploy does not
re-apply every tick and clear the Addon CR .status.gvks readiness signal
- agent mirrors per-tick delivery outcomes to the serial console for host-side
observability
* fix(eks): provision cluster NLB synchronously so the endpoint carries the real front-end IP
The cluster NLB front-end IP is assigned by the async LB-VM launch
(launchLBVMAsync), but EnsureClusterNLB read it back synchronously right after
CreateLoadBalancer returned — losing the race and falling back to the synthetic
<lb>.elb.spinifex.local DNS name. That name is unresolvable (no Route53) and is
not in the apiserver serving-cert SAN, so kubectl against the cluster never
connects.
Split the elbv2 create: the public CreateLoadBalancer stays async (267.4 — an
inline multi-minute VM boot head-of-line-blocks the gateway responder), and a
new CreateLoadBalancerSync drives the data-plane launch inline and returns the
LB carrying its front-end address. The EKS cluster-create worker runs off the
gateway responder, so it uses the sync entry and reads the IP from the return
value instead of racing persisted state. No polling.
- elbv2: factor provisionLBDataPlane out of launchLBVMAsync; add CreateLoadBalancerSync
- eks: ensureClusterLB uses the sync entry, sets FrontendIP from the result,
drops the read-back; EnsureClusterNLB fails loud (ErrClusterNLBFrontendIPUnavailable)
when no front-end IP is allocated rather than shipping the DNS-name fallback
- eks: drop the DNS-name endpoint fallback; the endpoint is always the front-end IP
* feat: auto-wire IGW for internet-facing EKS cluster endpoint
An internet-facing cluster NLB allocates an external front-end IP and the
OVN dnat_and_snat for it, but the external switch + localnet + gateway LRP
that make that IP answerable on the wire are built only on IGW attach. The
control-plane NLB is a system instance, so ensure an IGW on cluster build
and reclaim it on teardown.
EnsureClusterIGW (idempotent, reuses a customer IGW untouched) runs before
EnsureClusterNLB when the endpoint is public; DeleteClusterIGW (ownership-
scoped) removes only the cluster-owned IGW in purgeClusterInfra, before the
VPC delete. New IGW dep wired from the daemon.
* fix(eks): tear down infra on failed cluster launch to stop LB EIP leak
A CreateCluster that fails during its background launch flipped the meta
to FAILED but performed no teardown, relying on a later same-name create
to reclaim and purge the infra. EKS cluster names are unique per create,
so that reclaim never fires: the internet-facing load-balancer VM keeps
running with its allocated+associated dnat_and_snat EIP indefinitely.
Over repeated create/delete cycles the EIP pool fills with allocations
bound to terminated instances, so a later allocation's advertised
endpoint and its live OVN NAT diverge onto different pool addresses.
- failClusterLaunch now runs a best-effort purgeClusterInfra on the
in-memory meta (freshest refs even when the failing step was the meta
write itself) before MarkClusterFailed. purgeClusterInfra is
idempotent, so a later reclaim or DeleteCluster re-purge is a no-op;
the FAILED meta is retained for observability.
- Daemon releaseSystemInstanceEIP extracted and called from both
TerminateSystemInstance and cleanupFailedSystemInstance, so a launch
that fails after AssociateAddress releases the eipService allocation
instead of leaking it (MarkFailed's ReleasePublicIP only knows the
externalIPAM path).
* feat: EKS managed control-plane VPC (AWS Set B) composed from EC2 APIs
CreateCluster now auto-builds the managed control-plane VPC under the
system account, the spinifex analogue of AWS EKS's hidden managed-account
VPC: a public subnet for the internet-facing NLB, private subnet(s) for the
k3s control-plane VM(s), and a NAT gateway giving the private CP egress for
image pulls. It is composed from the real EC2 VPC-family APIs
(CreateVpc/Subnet, IGW, RouteTable/Route, NatGateway), so per-subnet egress
gating is wired by the existing topology subscribers off the route-table
events those APIs publish — no direct OVN mutation.
This resolves the mulga-siv-292 root cause: the internet-facing NLB and the
private CP were crammed into one customer subnet with no IGW route, so both
were treated as private and dropped by the lr_in_policy 1100 rule, killing
the LB-VM return leg and lb-agent heartbeat. The CP only survived via a
per-instance 1200 reroute bandaid, now removed — the topology expresses
egress instead.
Placement is uniform single/multi-node (CP always in the private subnet).
The system LB VM heartbeats over the mgmt bridge (SystemGatewayURL) rather
than the WAN. Managed-VPC teardown reclaims NAT GW + EIP, route tables,
subnets, IGW, and the VPC in dependency order. Customer subnetIds (Set A /
worker side) remain consumed-not-created and are deferred to the worker
round.
* fix(eks): bake cluster-owner account into CP-VM EKS_ACCOUNT_ID
The managed control-plane VPC (Set B) launches the K3s control-plane VMs and
their ENIs under the system (infra) account, but the on-VM agents — bootstrap
publish, state report, token-review, and add-on fetch — must key on the
cluster-OWNER (customer) account that owns the cluster meta, namespaces, and
AccessEntry KV. Add ClusterAccountID to K3sServerInput and bake it into the
VM's EKS_ACCOUNT_ID, decoupling it from the infra AccountID the VM launches
under. Without this the VM reported state and fetched add-ons against the
system account, so the cluster never observed a healthy control plane (stuck
CREATING, null CA) and kubectl/add-on delivery timed out.
Also refresh the e2e suite doc comment to describe the Set B topology and the
data-plane subtests (kubectl + managed-addon delivery) that now run.
* fix(eks): explicitly delete addon objects on unstage
K3s does not delete the applied resources when an auto-deploy manifest file is
removed from the server manifests dir (documented behaviour). The addon-sync
GC pass relied on that, so after DeleteAddon the rendered manifest was dropped
but its Namespace/ConfigMap stayed standing — the e2e AddonDelivery GC wait
timed out. Copy the manifest aside, drop the original so K3s stops tracking it,
then kubectl delete the objects (--wait=false to avoid finalizer stalls).
* fix(eks): surface CREATING block reason at Info, not Debug
A cluster stuck in CREATING logged its per-tick bootstrap/health block
reason at Debug, invisible at the default level, so a create that ran to
the 15m timeout FAILED gave no diagnosable cause. Bump both reasons to
Info; they fire only during the CREATING window.
* feat(iam): round-trip AWS-managed policy ARNs for stock EKS tooling
Stock terraform-aws-eks / eksctl attach AWS-managed policies
(AmazonEKSClusterPolicy, AmazonEKSWorkerNodePolicy, AmazonEKS_CNI_Policy,
...) that Spinifex does not provision. AttachRolePolicy rejected them
NoSuchEntity and ListAttachedRolePolicies dropped them, breaking the
node/cluster role round-trip DescribeNodegroup relies on. Accept and
store managed ARNs opaquely (no backing document), deriving the policy
name from the ARN on read.
* feat(eks): echo cluster tags + store-only Tag/Untag/ListTags
DescribeCluster never returned create-time tags and ClusterMeta did not
store them, so a stock terraform-aws provider reconciling default_tags
saw perpetual drift and issued TagResource on every apply — which itself
returned NotImplemented. Store tags on the cluster record, echo them from
DescribeCluster, and back the three tag ops store-only against the record
(cluster ARNs only) so the default_tags round-trip is clean.
* fix(eks): persist nodegroup workers incrementally + boot reclaim sweep
launchWorkers only wrote the worker IDs to the record after the whole
launch loop, so a daemon crash mid-loop stranded already-launched workers
under a record that still showed empty InstanceIDs — nothing could reclaim
them. Persist each worker before launching the next, and add a boot-time
reclaim sweep (SpawnRegisteredReconcilers) that terminates recorded
workers for any nodegroup left CREATING by a crashed launcher, or
CREATE_FAILED still holding workers, settling it to CREATE_FAILED with
InstanceIDs cleared.
* fix: plumb query params into EKS gateway dispatcher
EKS_Request never passed the query string to per-action handlers, so
UntagResource (DELETE /tags/{arn}?tagKeys=k1&tagKeys=k2) saw an empty
TagKeys and silently no-op'd. Fold url.Values into the JSON body when
the body is empty so the handler unmarshal sees the keys.
* feat(iam): resolve AWS-managed EKS policy grants for authorization
Assumed-role/user authorization (checkPolicy -> Get{Role,User}Policies ->
EvaluateAccess) could not honour AWS-managed policy grants: Spinifex stores
no document for managed ARNs, so stock EKS roles whose permissions come
entirely from managed policies (node role: AmazonEKSWorkerNodePolicy +
AmazonEC2ContainerRegistryReadOnly + AmazonEKS_CNI_Policy) evaluated to an
empty grant set and were denied every call.
Add a builtin registry of well-known AWS-managed policy documents, resolved
via a shared resolveAttachedPolicy helper used by both GetRolePolicies and
GetUserPolicies. Unmodeled managed ARNs resolve to no grant (fail-closed
deny) rather than erroring the whole request; customer-managed ARNs still
fail closed when unresolvable.
* test(e2e): add targeted suite selector + shadow-mode CI step
select-suites maps a diff to the e2e suites it must run: changed files →
services (manifest path/additional_paths) → consumer closure over
depends_on vs publishes/subscribes → suites whose covers intersect.
Special cases force-all (infra/harness/workflow, ≥3 services, submodule
bumps) or a lone suite dir; empty diffs fall back to default suites. An
AST scan of each suite's harness.Ensure* call graph derives per-test
service coverage and emits an optional -test.run subset (RUN_PATTERN).
Wired into e2e.yml as a non-gating 'Select Suites (shadow)' job that
logs the computed selection to the run summary so predicted vs actual
can be compared before the selector drives SUITES.
* ci(e2e): add dispatch-only per-suite job blocks grouped by env
Restructure the looped e2e flow into env-grouped provision -> matrix
suites -> teardown blocks so each suite surfaces as its own node in the
Actions graph instead of a single green tick:
single_provision -> single_suite [single,cert,lb,eks] -> single_teardown
multi_provision -> multi_suite [multinode] -> multi_teardown
Provision resolves the VM/cluster and ships test binaries, emitting WAN
IP + ssh key path as job outputs; each matrix leg runs one suite over
SSH and uploads its own junit/analysis artifacts; teardown (if:always)
reuses the persisted tofu workspace on the same runner to destroy.
workflow_dispatch only -- runs alongside the gating e2e.yml for
validation and posts no commit status.
* test(e2e): add manifest drift lint (fixture-create + NATS subject)
Bead 5 of the targeted-suite-selection plan. Two ratcheted drift guards in
tests/e2e/manifest-lint, wired into make preflight and the spinifex.yml PR job:
- Fixture lint: AST-flags direct resource-create AWS calls
(c.EC2.RunInstances, c.ELBv2.CreateLoadBalancer, ...) in suite test files,
steering them through harness.Ensure* fixtures. ~104 pre-existing call sites
are captured in a checked-in baseline; the lint fails only on NEW creates.
Intentional create-path tests opt out with an inline // e2e:allow-create.
- Subject lint: extracts NATS subject literals from the natsSub{} registry and
literal Subscribe/Publish/Request args, cross-referenced wildcard-aware
against service-interfaces.yaml. Code subjects missing from the manifest fail
(ratcheted); manifest subjects with no static reference warn only (many are
produced dynamically). Fixed real drift surfaced on first run: declared the
five spinifex.cluster.shutdown.* phase subjects under spinifex_daemon.
manifest-check is now CI-enforced alongside manifest-lint (previously
preflight/local-only).
* feat: run-scoped e2e teardown sweep
Tag every Ensure* fixture resource with e2e:run=<run-id> when
SPINIFEX_E2E_RUN_ID is set, and add a teardown.test suite that sweeps
everything carrying that tag at end of run — reclaiming resources leaked
by a crashed suite, an interrupted run, or a persistent cluster.
- harness/sweep.go: tagRunResources (EC2 CreateTags) + tagRunELB (ELBv2
AddTags) + SweepRunResources in dependency order (LB, instances, NAT,
snapshots, volumes, images, SGs, subnets, key pairs). Blank run id
no-ops both tagging and sweeping, so the sweep can never touch an
untagged or foreign resource.
- harness/fixtures.go: stamp the run tag on each created resource.
- tests/e2e/teardown: thin suite that runs the sweep; skips without a
cluster or run id, logs the report, never fails the run.
- e2e-suites.yml: build + ship teardown.test, pass the run id into suite
legs, sweep on the VM in both teardown jobs before tofu destroy.
The gateway-claim verifier polled the SB Port_Binding for the distributed
gateway port LRP (gw-<vpc>), which is always chassis-less by OVN design — the
chassis claim lands on the chassisredirect port cr-gw-<vpc>. So
ensureGatewayClaimed always read "unclaimed", fired ovn-appctl
inc-engine/recompute every reconcile cycle, and logged a spurious "gateway SB
chassis claim did not converge" error after its 30s budget. The repeated forced
recomputes rebuilt the entire OVN flow table every cycle, intermittently
darkening freshly attached EIP datapaths and flaking the single-node e2e suite
(observed: every br-int EIP flow at duration~10s with n_packets=0).
Check the chassisredirect Port_Binding instead. The nudge now fires only on a
genuinely unclaimed redirect — the real reboot race #339 targets — rather than
on every healthy reconcile.
- topology: add GatewayChassisRedirectPort(vpcID) -> "cr-gw-<vpc>"
- reconcile/apply: pass the cr- port to ensureGatewayClaimed
- reconcile/host: doc the verifier checks the chassisredirect binding
- test: capture the queried port; assert the cr- port is checked, no nudge
TestMultinodeGuestSSH ran with t.Parallel(), so it resumed immediately
after the sequential bucket and probed trio[0] — which CrossNodeOps
stop/starts but only waits to reach state "running", not for sshd to
answer. The guest was often still booting, producing
kex_exchange_identification resets and 2m timeouts that flaked CI red.
Drop t.Parallel() and declare the test last in the sequential bucket so
it runs after CrossNodeOps + NodeFailure/Recovery + Spread have
restabilised the cluster, giving every trio member time to settle and
removing concurrent VM churn.
CreateLoadBalancer is async: InstanceID is only set when the LB VM's QEMU
process starts, ~600ms after the create burst. The reactive
updateStoredConfig calls from CreateListener/RegisterTargets all no-op
during that window (InstanceID empty), and nothing rebuilds config once
the VM launches. The lb-agent then heartbeats, gets an empty ConfigHash,
never sees it change, and probes zero backends -> e2e 0/2 healthy.
Build the data-plane config on the provisioning->active heartbeat
transition instead: that is the one point where InstanceID is guaranteed
set and the agent is alive to consume config. The InstanceID guard in
updateStoredConfig stays as a cheap no-data-plane skip, no longer the
primary gate.
DHCP-source external-pool leases are DHCPRELEASEd only on explicit
instance/EIP terminate. Manager.Stop() preserves leases for
adopt-on-reboot, so env teardown/reset kills vpcd and destroys the
per-AZ lease KV without releasing — stranding every live lease
(instance ENIs, EIP allocations, per-VPC LRP gateways) on the upstream
DHCP server until TTL. Across CI churn with fresh client-ids the
upstream scope fills and new launches get no public IP.
Add Manager.DrainAll + a vpc.dhcp.drain RPC and a standalone
'spx admin cluster drain-dhcp' command that releases every held lease.
Wired into both teardown paths (single-node teardown role and
multi-node cluster-teardown.yml) before services stop. Stop() is
unchanged, so normal restarts still adopt leases on reboot.
* fix(ec2): fail RunInstances when public-IP auto-assign fails
PrepareRunInstances swallowed an AllocateIP failure for a
MapPublicIpOnLaunch subnet (logged WARN, fell through), so the instance
booted running with no public IP. With no qemu-hostfwd fallback such an
instance is unreachable. Mirror the adjacent AddNAT-failure path: detach
and delete the auto-created ENI, deallocate capacity, drop the instance,
and surface InsufficientAddressCapacity.
* test(daemon): deterministically pick a fitting 2-vCPU type in capacity filter test
CapacityFilterRespectsAllocation selected the first 2-vCPU type by Go
map-iteration order, intermittently picking a type near the
schedulable-memory boundary the allocator then rejected (~3/5 failures in
isolation on small hosts). Select the smallest-memory 2-vCPU type from the
capacity-aware set so it always fits.
CreateLoadBalancer's PutLoadBalancer-failure path only released the name
claim, orphaning the ENIs and managed front-end SG already created. Extract
the ENI+SG+claim teardown into rollbackLBInfra and call it from both the
ENI-create-failure and PutLoadBalancer-failure paths.
CreateListener persisted the listener record before authorizing the NLB
managed-SG port and reloading config; a failure in either left an orphan
listener record (and, after a config failure, a dangling SG authorization).
Add rollbackListener to revoke the port (when already opened) and
cascade-delete the listener on every post-persist failure.
Also wait for the async worker launch in the two CreateNodegroup disk-size
tests so they observe the launch under the record-before-launch contract.
* fix: make platform create handlers async to stop responder starvation
EKS CreateCluster/CreateNodegroup and ELBv2 CreateLoadBalancer ran their
full VM/control-plane launch synchronously inside the NATS subscription
callback on the shared spinifex-workers queue. A multi-minute create
head-of-line blocked the worker, so system.LaunchInstance/TerminateInstance
found no responder within the gateway timeout and failed with
'nats: no responders available' (mulga-siv-267.4).
Each create now runs validation + the atomic name claim synchronously,
snapshots the CREATING/provisioning output, then spawns the slow launch on
a per-impl launchWG and returns immediately. Launch failures mark the
record FAILED/CREATE_FAILED so 267.3 reclaim makes a retry idempotent. The
service interfaces are shared by the gateway NATS clients and the daemon
impls, so the async split lives inside the impl method and leaves the
interface, handleNATSRequest, and all daemon handlers unchanged.
The host-pinned system.LaunchInstance/TerminateInstance responders now
dispatch each request on its own goroutine (systemDispatchWg) so a slow
boot, which must respond after it completes, no longer blocks the subject.
ELBv2 keeps ENI creation synchronous (cheap VPC calls that yield the
AZ/VpcId the create response needs); only the LB-VM boot goes async,
folding InstanceID/IPs back into the record for a later Describe.
* fix: keep node-targeted launch subscription live under capacity pressure
The node-targeted (unicast) system.LaunchInstance.{type}.{node} subject was
torn down whenever the node filled, same as the queue (anycast) subject. For
the queue subject that is correct — NATS reroutes the launch to a node with
room. For the node-targeted subject there is no other node: a committed spread
reservation (EKS HA control-plane leg) that targets the now-unsubscribed node
gets 'nats: no responders available' instead of a capacity decision.
Keep the node-targeted subscription live for any type the node can host;
enforce capacity at launch time via ResourceManager.allocate, which already
rejects an over-capacity launch atomically. A reserved launch always reaches a
responder and, when the slot is still good, succeeds.
launchWorkers built RunInstancesInput with no BlockDeviceMappings, so
parseVolumeParams applied the 4GB root-volume default and the nodegroup
DiskSize was silently ignored. Wire rec.DiskSize into a BlockDeviceMapping
when set.
Run 27254808154 showed TestRebootResilience fail with ovn-controller on
the live chassis claiming NO local Port_Bindings post-reboot: ENIs stuck
on a dead chassis UUID, cr-gw chassis empty, NAT external_mac empty —
while NB intent was correct and taps carried the right iface-id. The SB
Chassis _uuid churned across the reboot (old UUID gone, new one created)
while the SB DB persisted, orphaning every claim.
The current artifacts cannot pin WHY the new chassis won't reclaim: the
ovn-controller vlog lives under /var/log/ovn, not journald, so it is
absent from the systemd-journal capture.
Add:
- phase5Snapshot: capture chassis identity PRE-reboot (system-id, SB
Chassis/Chassis_Private, get-connection) to ovn-pre-chassis.txt as the
churn baseline.
- dumpOVNClaimState: at post-reboot failure, capture the same chassis
state POST-reboot plus ovn-controller connection-status/debug, the
gw/eni Port_Binding claim detail, ovn-central-vs-ovn-controller boot
ordering, and tails of ovn-controller.log / ovn-northd.log →
diag-ovn-claim.txt.
Disambiguates chassis-identity churn from an SB/NB startup-ordering race
on the next reboot-suite run. Diagnostic only; no behavior change.
* feat: atomic create idempotency across EKS, ELBv2 and EC2 ClientToken
Every create handler took a non-atomic Get-check then a blind Put, so a
gateway/NATS retry that spawned a second concurrent handler could pass the
check in both and double-launch + leak VMs (267 failure-mode C).
Make each create claim its identity key atomically before any launching work:
- EKS CreateCluster: claim the cluster meta via kv.Create; a prior FAILED
attempt is reclaimed via a CREATING CAS so exactly one concurrent retry wins.
purgeClusterInfra gains a deleteMeta flag so reclaim purges infra while
keeping its claim key.
- EKS CreateNodegroup: claim the nodegroup record before SG provisioning or
worker launch; duplicates return ResourceInUse without provisioning.
- ELBv2 CreateLoadBalancer: account-scoped name claim (lbname.<acct>.<name>)
taken before ENI/VM work, released on every failure path and in
DeleteLoadBalancer; a name claim whose owner lbID has no live record is a
crashed-create orphan and is reclaimed.
- EC2 RunInstances: ClientToken dedup over a dedicated TTL bucket with a
two-phase claim (in-flight -> done) and reservation replay; a token reused
with different params returns IdempotentParameterMismatch.
Race tests under -race assert exactly one owner and a single launch for
concurrent same-name creates on all paths.
* test: cover ClientToken idempotency; extract orchestration for testability
Extract the claim->replay->launch->finalize/abort flow out of RunInstances
into runInstancesWithClientToken(store, ..., launch func) so the idempotency
path is unit-testable without a live gateway/NATS. RunInstances now just
resolves the store, hashes params, and passes a launch closure.
Make the in-flight poll timeout/step package vars so a duplicate-caller
wait-timeout is deterministically testable.
Adds tests: replay-skips-launch, owner-launches-once-then-replay,
launch-failure-aborts, param-mismatch maps to AWS error, getClientTokenStore
binds once, and in-flight wait-timeout.
After a host reboot the Northbound gateway_chassis intent recovers correctly
(SetGatewayChassis re-asserts it idempotently), but ovn-controller can leave the
Southbound Port_Binding for the gateway router port unclaimed (chassis == []).
With no claimed binding it installs no logical->physical flows for the centralised
gateway redirect, so every floating IP on the VPC is dark: ofproto/trace drops the
FIP ingress at br-int table 0, host->FIP is 100% packet loss, and the ALB lb-agent
heartbeat times out until the 5-min health window expires (TestRebootResilience).
rebindGatewayChassis only asserted NB intent. Now, after SetGatewayChassis, the
reconciler verifies the SB Port_Binding has a non-empty chassis and, if not, nudges
the local ovn-controller to recompute (inc-engine/recompute), polling until the
claim converges or a bounded deadline elapses — logging an error rather than
blocking reconcile. SB read + recompute are exec-only (ovn-sbctl/ovn-appctl) and
live in network/host behind a reconcile.GatewayClaimVerifier interface, wired in
vpcd; reconcile stays NB-pure and mockable. No NB object create/delete (ADR-0006 S5).
Verified by run 27251769281's diag-ovn-flows.txt (SB chassis empty + table-0 drop).
Validation completes on the next reboot-suite nightly (single-node self-heals too
fast to reproduce locally).
* test(reboot): capture heartbeat-path diagnostics on post-reboot health timeout
When post-reboot ALB targets fail to go healthy, the lb-agent's heartbeat to the
gateway times out with 'context deadline exceeded while awaiting headers' — a
symptom that alone cannot distinguish a network-level drop from an app/gateway
stall. Replace the fatal WaitForTargetsHealthy with a non-fatal poll that, on
timeout, tcpdumps br-wan for the heartbeat 5-tuple + ARP for the ALB floating IP
and pulls the lb-agent console + awsgw journal, then fails. Disambiguates the
layer before any fix (mulga-siv-265).
* test(reboot): re-aim heartbeat capture + add OVN dataplane flow-dump
Run 27250238163 proved the post-reboot failure is network-level (all 3 floating
IPs 100% packet-loss from host) but NOT neigh/NAT priming — neigh is primed and
NAT rows are present, only br-int forwarding is dead. The prior capture watched
br-wan for the ALB floating IP, while the real lb-agent heartbeat posts to the
gateway port on the host's advertise interface, so the L4 classifier could never
see the heartbeat and defaulted regardless.
- heartbeat tcpdump now watches the gateway port on `any` (real path) plus a
secondary br-wan floating-IP capture for north-south; classifier keys off
gateway-port L4 presence.
- new dumpOVNDataplane captures ovs-vsctl show, ovs-ofctl dump-flows br-int,
ovn-sbctl chassis + Port_Binding, dnat_and_snat NAT rows, host neigh, and a
host->floating-IP ping + ofproto/trace, to pin the flow-restore step.
- failure path dumps diag-ovn-flows.txt alongside diag-heartbeat.txt.
Test-only, build-tagged e2e; no production code.
CreateCluster set the control-plane VM/ENI/spread-group refs into the cluster
meta but first persisted them to KV only after allocateClusterEgressIP. A
failure between placement and that persist left the live sys.medium CP VMs
unrecorded, so neither DeleteCluster nor the FAILED-cluster reclaim could reach
them and they leaked, reserving host capacity.
Persist the meta immediately after placeControlPlane returns. The reclaim path
keys off the persisted ControlPlaneNodes + ControlPlaneSpreadGroup, so this also
frees reserved spread capacity on teardown.
A retried DeleteCluster (or the egress-delete cascade) can release the
hidden-pool allocation before the release step runs, so ReleaseAddress
returns InvalidAllocationID.NotFound. That was treated as a fatal teardown
error, so the cluster wedged in DELETING forever and its KV meta plus
downstream resources leaked. Swallow the NotFound variants as idempotent
success — mirroring the 165.17 ENI-teardown fix — so the SG + KV sweep
proceeds. A genuine release failure still wedges DELETING for retry.
* feat(eks): node-targetable control-plane VM (HA spread transport prereq)
Groundwork for HA control-plane placement (mulga-siv-231.7.1): make the k3s
control-plane VM placeable on a specific Spinifex host.
- Add a sys.medium instance type (2 vCPU / 4 GB) and boot the control-plane VM
as it instead of t3.medium. A sys.* type keeps the CP VM out of customer
DescribeInstanceTypes and makes the daemon register the node-targeted
system.LaunchInstance.{type}.{nodeID} subject (only sys.* types get it).
- Add K3sServerInput.TargetNodeID and a LaunchSystemInstanceOnNode method on
the launcher: empty/local node launches in-process (unchanged single-CP
path); any other node dispatches over the node-targeted subject, where the
receiving daemon runs the launch and owns the VM.
The spread orchestrator (reserve N hosts, launch per host, verify, rollback,
<3-host fallback) lands separately; CreateCluster still launches locally
(TargetNodeID empty), so behaviour is unchanged.
* feat(eks): HA control-plane spread orchestrator
Place N=3 k3s server VMs across distinct Spinifex hosts via a per-cluster
spread placement group, with all-or-nothing rollback and single-CP fallback.
- placeControlPlane: capacity fan-out (HostScheduler) -> >=3-host gate ->
ensure spread group (system account) -> ReserveSpreadNodes(Min=Max=3) ->
parallel per-node LaunchK3sServerVM(TargetNodeID) -> node-vms placement
verify -> FinalizeSpreadInstances. Any launch/verify/finalize failure
terminates launched VMs, releases the reservation, drops the group.
- <3 schedulable hosts or a lost reservation race falls back to a single
local control-plane VM (prior behaviour).
- ClusterMeta gains ControlPlaneNodes[] + ControlPlaneSpreadGroup; [0] is
mirrored into the scalar ControlPlane* fields for back-compat. DeleteCluster
tears down every CP VM and the spread group.
- Wire PlacementGroup + Scheduler into EKSServiceDeps + orchestration-deps gate.
* fix(eks): size HA control-plane spread on raw host headroom
SchedulableHosts matched the CP type by name in node.status's per-type
capacity list, but that list omits sys.* types (system types are hidden
from customers). The CP VM is sys.medium, so the match never hit and
SchedulableHosts returned 0 on every node — placeControlPlane always fell
back to a single CP, making the >=3-host spread unreachable in production.
System VMs consume host vCPU/memory like any guest, so size the fit against
each node's raw schedulable headroom (Total - Reserved - Alloc) and the
type's footprint instead. Adds instancetypes.SpecForSystemType for the
sys.* vCPU/memory spec and a pure nodeFitsSystemInstance helper.
* feat(eks): k3s server-join role for HA control-plane quorum
Servers 2..N now boot in join mode: server: https://<first-server>:6443 +
a shared cluster token, without cluster-init, so they join the first server's
embedded-etcd quorum instead of each cluster-initing an independent datastore.
- K3sServerInput gains JoinToken (shared cluster secret, set on every server)
and ServerURL (set => join server, no cluster-init, server-join role).
- buildK3sUserData branches cluster-init vs server:, emits token:, and selects
SPINIFEX_K3S_ROLE server vs server-join.
- launchControlPlaneSpread launches nodes[0] first (cluster-init), derives the
join URL from its ENI IP, then launches the rest in parallel as join servers;
a first-server failure skips the joins and rolls back.
- CreateCluster mints a per-cluster crypto-random token (GenerateK3sClusterToken)
threaded into the placement template.
- eks-node-role.sh adds a server-join case: token-webhook + k3s + state-report,
but not k3s-first-boot (the first server already publishes the bootstrap
artifacts; a join re-publish only races the bootstrap bus).
* fix: open etcd-peer/kubelet ingress on EKS control-plane SG for HA join
Join servers (231.7.2) register over the supervisor port 6443 but their
embedded etcd could not peer with the quorum: the control-plane SG only
admitted 6443 from the VPC CIDR, omitting etcd client+peer 2379-2380 and
kubelet 10250 between control-plane nodes. A single cluster-init server never
exercises those ports, so the gap stayed hidden until multi-server HA — the
join registered but its etcd never replicated the raft log, leaving the node
perpetually NotReady.
Add EnsureControlPlaneHAIngress with self-referencing (cpSG->cpSG) rules for
etcd 2379-2380 and kubelet 10250, called from CreateCluster after
EnsureControlPlaneIngress. Verified live on a 3-node cluster: join servers
reach Ready and form a 3-member etcd quorum that survives loss of the original
first server.
* feat: register all control-plane ENI IPs as EKS cluster NLB targets
Register every HA control-plane server ENI IP in the cluster NLB target
group, not just the primary. The TG already health-checks :6443, so a
dead member drains and kubectl/apiserver traffic fails over to surviving
etcd-quorum members. Add RegisterClusterTargets batch helper; the single
RegisterClusterTarget now delegates to it.
Associated Elastic IPs lost external connectivity after a host reboot:
vpcd's reconcile re-applied the dnat_and_snat rule without the ENI MAC,
collapsing it to the centralised shape (external_mac/logical_port unset),
and the daemon's relaunch republish only fired for auto-assigned public
IPs, never for store-backed EIPs.
- EIPRecord gains MacAddress, captured at AssociateAddress and cleared at
DisassociateAddress, so reconcile can re-apply the distributed shape
without re-querying the ENI.
- loadEIPs feeds rec.MacAddress into the reconcile spec; policy.AddEIP
self-heals a stale centralised row when given a MAC.
- AssociatedPublicIPForInstance lets the relaunch hook resolve an
associated EIP from the store and republish vpc.add-nat with the MAC.
* feat(eks): node-targetable control-plane VM (HA spread transport prereq)
Groundwork for HA control-plane placement (mulga-siv-231.7.1): make the k3s
control-plane VM placeable on a specific Spinifex host.
- Add a sys.medium instance type (2 vCPU / 4 GB) and boot the control-plane VM
as it instead of t3.medium. A sys.* type keeps the CP VM out of customer
DescribeInstanceTypes and makes the daemon register the node-targeted
system.LaunchInstance.{type}.{nodeID} subject (only sys.* types get it).
- Add K3sServerInput.TargetNodeID and a LaunchSystemInstanceOnNode method on
the launcher: empty/local node launches in-process (unchanged single-CP
path); any other node dispatches over the node-targeted subject, where the
receiving daemon runs the launch and owns the VM.
The spread orchestrator (reserve N hosts, launch per host, verify, rollback,
<3-host fallback) lands separately; CreateCluster still launches locally
(TargetNodeID empty), so behaviour is unchanged.
* feat(eks): HA control-plane spread orchestrator
Place N=3 k3s server VMs across distinct Spinifex hosts via a per-cluster
spread placement group, with all-or-nothing rollback and single-CP fallback.
- placeControlPlane: capacity fan-out (HostScheduler) -> >=3-host gate ->
ensure spread group (system account) -> ReserveSpreadNodes(Min=Max=3) ->
parallel per-node LaunchK3sServerVM(TargetNodeID) -> node-vms placement
verify -> FinalizeSpreadInstances. Any launch/verify/finalize failure
terminates launched VMs, releases the reservation, drops the group.
- <3 schedulable hosts or a lost reservation race falls back to a single
local control-plane VM (prior behaviour).
- ClusterMeta gains ControlPlaneNodes[] + ControlPlaneSpreadGroup; [0] is
mirrored into the scalar ControlPlane* fields for back-compat. DeleteCluster
tears down every CP VM and the spread group.
- Wire PlacementGroup + Scheduler into EKSServiceDeps + orchestration-deps gate.
* fix(eks): size HA control-plane spread on raw host headroom
SchedulableHosts matched the CP type by name in node.status's per-type
capacity list, but that list omits sys.* types (system types are hidden
from customers). The CP VM is sys.medium, so the match never hit and
SchedulableHosts returned 0 on every node — placeControlPlane always fell
back to a single CP, making the >=3-host spread unreachable in production.
System VMs consume host vCPU/memory like any guest, so size the fit against
each node's raw schedulable headroom (Total - Reserved - Alloc) and the
type's footprint instead. Adds instancetypes.SpecForSystemType for the
sys.* vCPU/memory spec and a pure nodeFitsSystemInstance helper.
* fix: strip duplicate lb- prefix from haproxy socket and cert paths
LoadBalancerID, ListenerID and friends already carry their type prefix
(lb-/lst-), so the stats-socket, frontend cert, and HAProxyManager
config/pid filename templates double-prefixed: lb-lb-<id>.sock,
lb-lb-<id>-<lst>.pem, alb-lb-<id>.cfg.
The socket path is computed independently on both sides (gateway render
in buildHAProxyConfig, agent poll in lbagent.New); any divergence in the
lb- prefix convention between the two leaves the agent polling a socket
HAProxy never bound, spamming 'HAProxy stats unavailable: no such file'
while the data plane serves fine. Drop the redundant literal prefix so
both sides resolve to <id>.sock and the alb-/nginx cert paths match the
delivered config.
* refactor: remove unused HAProxyManager
HAProxyManager was never constructed (NewHAProxyManager had no callers).
The live ALB data plane runs in the lb-agent via reloadHAProxy with the
config generated daemon-side and shipped; this per-daemon process manager
was dead code. Remove the type, its methods, tests, and now-unused
imports (log/slog, os/exec, sync, syscall).
* ci(e2e-nightly): run Go harness suites instead of bash run-*.sh
Swap nightly cells 1-3 (single), 8-10 (real-multi) and 18 (reboot) to the
compiled <suite>.test binaries, mirroring e2e.yml: SPINIFEX_E2E=1 with
SPINIFEX_MODE single|multinode, per-suite logs to ARTIFACT_DIR, then
go-junit-report + e2e-analyze + upload-artifact. Adds a spinifex checkout for
the composite actions and pulls /run/spinifex/console-*.log so LB microVM
(lb-agent) state is captured on failure. Cells 17 (tofu) and 20 (baremetal)
stay on bash — no Go suite exists.
* ci(e2e-nightly): cell-17 RCA analytics + cell-20 cosmetic alignment
run-tofu-examples-e2e.sh now emits go test -v markers per workbook so the
cell-17 step's tee'd test-tofu.log flows through the same go-junit-report ->
e2e-analyze -> upload-artifact chain as the Go suites, producing junit-tofu.xml
and an analysis bundle for RCA. Align e2e-baremetal.yml (cell 20) ARTIFACT_DIR,
analyze title, and artifact name with the matrix cells so CI-RCA discovers its
bundle uniformly.
* ci(e2e-nightly): wire host journal + VM consoles into RCA bundles
Collect now emits the canonical spinifex-journal.log (concat of all nodes)
that the e2e-analyze bundler slices into each failure's journal.log, so
elbv2-service/vpcd/daemon lines in the failure window travel with the bundle
(previously the journal uploaded as spinifex-journal-node*.log, a name the
bundler never reads). Rename the console capture to vm-console-node* and note
it also carries guest cloud-init/sshd consoles, covering the cell-3 internal-LB,
cell-18 reboot-datapath, and cell-9/10 guest-ssh blind spots.
* ci(e2e): self-heal runner /dev + --one-file-system on workspace clean
Clean Workspace now detaches any leaked iso-builder rootfs mount, recreates
/dev/null if a prior build clobbered it, and uses rm --one-file-system so the
clean can't descend into a still-mounted devtmpfs and delete host device nodes.
Guards both the matrix cells and the bare-metal job against the host-wide
/dev/null wipe that fails go build with 'open /dev/null: no such file or
directory'.
The LB VM's external-pool floating-IP NAT (dnat_and_snat external_ip -> LB
VPC IP) was reaped only conditionally on teardown: DeleteNetworkInterface
skips EIP-owned addresses (defers to the EIP lifecycle), and that lifecycle's
reap runs in the async TerminateSystemInstance goroutine gated on the EIP
association. So repeated create/delete cycles accumulated stale rules that
collapsed onto the reused LB VIP, making OVN SNAT/un-DNAT ambiguous and
dropping the lb-agent heartbeat return path.
DeleteLoadBalancer now publishes vpc.delete-nat deterministically from the LB
record (external=AvailZones public IP, logical=VPCIP). DeleteEIP matches on
external+logical and is idempotent, so redundant reaps from other paths are
harmless. Internal LBs (no floating IP) are a no-op.
* fix: re-prime EIP reachability on AddEIP idempotent-skip path
AddEIP early-returned on the idempotent-skip path (OVN DNAT row already
matches) before emitting GARP and priming the host neighbour. Stop->start
and daemon reboot-recovery re-attach the same EIP with an unchanged row, so
the host neigh entry / OVS datapath never got re-primed and the EIP/ALB-VIP
stayed dark until the kernel ARP entry timed out (60-300s). inject-garp is
unsupported on some chassis, making neighPrime the only working remediation
— exactly the step the skip bypassed.
Extract garp + neighPrime/flush into primeReachability and run it on the
skip path as well as after a fresh AddNAT. The FlowsBarrier still fires only
on row churn. Resolves cell-9/10 TestMultinodeGuestSSH and cell-18
TestRebootResilience nightly failures.
* refactor: remove dead inject-garp GARP emitter from EIP path
ovn-appctl inject-garp is not a valid command on the OVN build Spinifex
deploys and tests against — every AddEIP GARP attempt failed with
'inject-garp is not a valid command' and fell through to the host neigh
prime/flush, which is the only remediation that works. The GARP hook was a
guaranteed-failing shell-out plus a misleading WARN on every EIP attach.
Delete host.InjectGARP, the GARPEmitter type, WithGARPEmitter option, the
garp field + its m.garp() call in primeReachability, and the vpcd
injectGARPEmitter wiring. host.ReplaceNeigh (distributed) and host.FlushNeigh
(centralised) already cover both NAT shapes and become the sole EIP
reachability path. Reword the neigh hook docs that framed themselves as an
inject-garp-independent fallback. Prune GARP-only tests; move the shared
recordingRunner helper into neigh_test.go.
* feat(daemon): live MemAvailable admission gate + fixed-services reserve scope
Admission previously charged only guest -m against a flat reserve, blind to the
co-located storage VMs, nbdkit, and daemon stack — so it greenlit launches that
overcommit real RAM and OOM-kill guests on memory-tight hosts.
Add a second admission gate that clamps the accounting-derived launch count by
live /proc/meminfo MemAvailable (which already nets reclaimable cache and sees
every untracked consumer): admitted = min(accounting, (MemAvailable-reserve)/
guestMem). Reads live per admission, fails open on a /proc read error so a
transient fault can't wedge launches, and is disabled by SPINIFEX_ADMISSION_LIVE_MEM=0.
Re-scope the host reserve to the fixed infrastructure tier only; per-instance
nbdkit is no longer expected to fit inside it (explicit per-instance charge is a
follow-up; the live gate covers it implicitly in the meantime).
Pure helpers (liveMemCount, parseMemAvailableKB, liveMemAdmissionEnabled) are
unit-tested; liveMemGate covered via an injected reader.
* feat(daemon): charge nbdkit per-instance in admission accounting
An instance's admission memory charge is now its guest -m plus the nbdkit
processes backing its volumes (one per attached volume), not guest -m alone.
The static reserve no longer has to absorb per-instance nbdkit, which was
unsizable and the root of the host-memory-overcommit OOM.
Per-volume charge is measured per class (main 768 MiB: 128 cache + plugin
working set + in-flight I/O; aux 96 MiB: process floor, 0 cache), env-tunable
via SPINIFEX_NBDKIT_MAIN_MIB / SPINIFEX_NBDKIT_AUX_MIB. Default layout is one
main + two aux volumes (root + cloudinit + efi) ~= 0.94 GB/instance. Applied at
every accounting site through instanceMemChargeMiB so charge and release never
drift; the live MemAvailable gate backstops block-device-mapping volumes the
static count misses.
* feat(vm): three-tier guest OOM ladder + spinifex-guests.slice
Replace the flat oom_score_adj=500 on every guest QEMU with a ManagedBy-derived
ladder: user guests (ManagedBy=='') stay +500 and are reaped first under host
memory pressure; system instances (ELBv2 LB-VMs, EKS control-plane nodes,
ManagedBy!='') score 0, ranking above user guests but below the infra tier
(-500). Killing a system instance degrades the data path or cluster control
plane for many user VMs, so it must outlive them.
Add spinifex-guests.slice as a sibling of spinifex-system under spinifex.slice —
the parent tier for the per-instance leaves that land with RG-7. Guest QEMU
placement into the slice is deferred to the per-instance work.
* refactor(systemd): lean unit comments to RG-n tags + RG-9/RG-11 invariants
Strip the multi-paragraph rationale novels from spinifex-daemon.service (48->7
comment lines), spinifex-vpcd.service (29->8) and spinifex.slice (15->5),
replacing them with settings + terse # RG-n clause tags. Remove plan-doc, bead
and CI-run references from comments (project policy). Directives are byte-for-byte
unchanged — comment-only.
Add RG-9/RG-11 invariant tests asserting per-tier confinement (locked-down
storage/control tier drops all caps + NoNewPrivileges=yes; daemon keeps exactly
CAP_SYS_ADMIN+CAP_DAC_OVERRIDE and the four-entry DeviceAllow; vpcd is the
documented network-tier exception) and the lean-comment budget + no-plan-ref rule.
Project internal StateError onto AWS stopped (code 80) in DescribeInstances
so SDK/UI clients no longer break on the non-AWS 'error' state label.
Route start-instances through the per-instance ec2.cmd topic first so a
crash-pinned StateError instance is restarted by its live owner node, whose
StartInstance handler now accepts StateError (Manager.Start transitions
error->pending before relaunch). Fall back to the ec2.start queue-group path
on ErrNoResponders for genuinely stopped instances in the shared KV.
* ci(eks-ami): build on self-hosted ci-single-1 runner
GitHub-hosted ubuntu-24.04 cannot run the libguestfs appliance network:
passt fails (PID file denied under its AppArmor confine / privilege drop)
and the SLIRP fallback has no outbound, so virt-customize --network dies.
The self-hosted ci-single-1 runner has /dev/kvm and real networking, so
the existing libguestfs build works unmodified.
* fix(build-system-image): add --no-cache, force fresh build on publish
The <10min cached-raw skip avoids duplicate work for concurrent e2e
jobs, but on the persistent publish runner it could republish a stale
image from a prior or concurrent run. Add --no-cache to bypass the skip
and have publish-system-image.sh pass it on every build.
* ci(eks-ami): target ci-single label (no ci-single-1 label exists)
The self-hosted runners are named ci-single/-2/-3 but all share the
single label 'ci-single'; runs-on matches labels, not names, so
ci-single-1 never schedules.
* ci(eks-ami): disable Go cache on self-hosted runner
setup-go's cache-save (Post Setup Go) re-uploaded the Go cache to the
Actions backend on every run, taking ~2.5min vs the 36s build. The
self-hosted runner already keeps the cache on local disk, so cache:false.
* feat(systemd): bound spinifex to a resource-limited slice
Add a spinifex.slice parent ceiling (MemoryHigh=80%, MemoryMax=90%,
MemorySwapMax=0) with a spinifex-system.slice child holding the seven
control/infra services, so a runaway hits the slice instead of the host.
A system.slice drop-in (MemoryMin=1G, CPUWeight=300) reserves RAM and CPU
for sshd/journald/the operator, keeping the box reachable when spinifex is
maxed.
The drop-in is generated in setup.sh install_systemd() rather than shipped
as a staged file, since the packaging globs flatten the systemd/ dir; this
mirrors the existing sshd-keygen drop-in pattern. The two .slice files flow
through every staging path unchanged.
Within the single effective cgroup, a slice MemoryMax OOM picks the highest
oom_score victim — guest QEMU (+500) before any service (-500) — so a slice
OOM degrades by killing one guest while the daemon, storage tier, NATS, and
the host all survive.
* feat(setup): provision 8G host swapfile + vm.swappiness=10
Add a setup_swap stage that creates an 8G /swapfile, enables it, persists it
to fstab, and writes vm.swappiness=10 so the kernel only swaps under real
pressure. Backs spinifex.slice MemorySwapMax so the slice can spill to swap
for graceful degradation instead of OOMing a guest. Idempotent; skipped under
ISO_BUILD (firstboot provisions swap on ISO hosts). Reverses the prior swap=0
assumption.
* Revert "fix(network): prime host neigh with gateway port MAC on centralised EIP attach"
This reverts commit f7446c69c6.
* fix(network): round-robin external pool cursor so a released IP is not reused instantly
The static external pool handed out the lowest free address and freed IPs
immediately on release, so a just-released EIP was reassigned within ~0.6s. A
recycled EIP front-end is unreachable from the same host: the kernel still
holds the prior owner's ARP binding (or a negative cache after the neigh
flush) and OVN does not answer the re-ARP in time, so the neighbour entry
settles to FAILED (EHOSTUNREACH). This surfaced as EKS TestEKS/KubectlGetNodes
failing in single-node CI once #315 routed kubectl through the cluster NLB's
external EIP. Dev never hit it — it uses DHCP-random addresses.
Add a persisted per-pool cursor: allocation resumes one past the last address
handed out and wraps at RangeEnd, so a freed IP is not reused until the cursor
cycles the whole range. Empty cursor (fresh pool / older KV record) starts at
RangeStart, preserving lowest-free order for the first cycle — backward
compatible, no migration.
* ci: drop e2e-eks from default e2e suites
EKS data-plane subtest (TestEKS/KubectlGetNodes) is a persistent flake on
single-node CI — a freshly-attached centralised EIP is intermittently
unreachable from the host (no route to host to the cluster endpoint), which
fails the whole E2E run and blocks unrelated suites. Remove e2e-eks from the
default SUITES so PR/push runs no longer gate on it. The e2e-eks target and
the workflow_dispatch suites input are unchanged, so EKS stays runnable on
demand and via make e2e-all.
Centralised-mode EIP attach only flushed the host neighbour entry and relied
on OVN re-answering the host's ARP. On OVN builds whose ovn-controller lacks
inject-garp, no node answers a same-chassis recycled IP's re-ARP, so the entry
decays to FAILED and the external IP is unreachable ("no route to host"). This
sank the EKS e2e suite: the cluster NLB front-end EIP never resolved on br-wan,
so kubectl could not reach the published apiserver endpoint.
Mirror the distributed-mode prime: resolve the VPC gateway router port MAC (the
L2 owner of a centralised EIP on the WAN segment) and program the host
neighbour entry directly, falling back to a flush only when no MAC resolves.
* feat: un-stub EKS Addon API with managed-addon state machine
Wire the 6 EKS Addon ops (List/DescribeVersions/Create/Describe/Update/
Delete) to a per-cluster managed-addon state machine in the per-account KV.
- addon_catalog.go: static air-gapped catalog (aws-load-balancer-controller,
coredns) with version validation.
- addon_state.go: AddonRecord + AddonStatus, KV CRUD + CAS helpers; list skips
staged-manifest sub-keys; delete sweeps record + manifest.
- addon_installer.go: transport-agnostic AddonInstaller; stagingInstaller
stages a manifest descriptor in KV for the (future) VM-side delivery slice.
- addons.go: op impls + AWS-shape converters; installer failure flips
CREATE_FAILED. Status stays CREATING until delivery confirms ACTIVE (private
apiserver is host-unreachable, so delivery must be VM-side).
- service_impl.go: drop notImpl stubs; add AddonInstaller dep (nil-defaults to
staging installer).
DeleteClusterPrefix already sweeps clusters/{name}/ so addon keys clean up on
cluster delete.
* fix: route ListAddons at GET /clusters/{name}/addons
ListAddons was wired at GET /addons with no cluster param, so the AWS CLI's
GET /clusters/{name}/addons hit no route (InvalidActionException) and the
wrapper couldn't pass a cluster to the service. Match the real EKS path and
thread the cluster name through.
Recycled WAN EIPs stayed unreachable from the host (kubectl to a cluster
endpoint: "no route to host", host neigh FAILED) on OVN builds whose
ovn-controller lacks the inject-garp appctl. OVN advertises a recycled
external IP's new MAC only via the GARP it emits on LSP binding-chassis
migration; a same-chassis rebind emits none, and the neigh flush added for
recycle reachability then waits on an ARP reply no node sends.
In distributed mode the external_mac is known at attach time, so program the
host neighbour entry directly (ip neigh replace <ip> lladdr <mac> nud
reachable) instead of flushing. Inbound traffic refreshes the entry from
there. Centralized attach (no external_mac) and all detaches keep the flush.
Adds host.ReplaceNeigh + policy.NeighPrimer hook wired in vpcd alongside the
existing flusher.
Workloads need a working ingress/servicelb path before the
aws-load-balancer-controller add-on lands. Flip builtinIngressEnabled to
default ON; opt out per-cluster for AWS parity with the CreateCluster tag
spinifex.io/managed-ingress=false. Interim divergence — revert to
default-off when the controller add-on ships.
Recycled WAN external IPs stayed unreachable for the kernel ARP timeout
(60-300s) when ovn-controller lacks the inject-garp appctl: the GARP refresh
after AddEIP is then a no-op, leaving the host neighbour cache pointed at the
prior owner's MAC.
Add host.FlushNeigh (ip neigh flush) and a policy.NeighFlusher hook wired
into AddEIP and DeleteEIP, so a recycled IP re-resolves L2 against the current
owner independent of inject-garp. DeleteEIP now takes the external IP (the
vpc.delete-nat event already carries it) to flush on release.
* feat(eks): public/private endpoint access drives NLB scheme + IP:port reachability
Parse resourcesVpcConfig.endpointPublicAccess/endpointPrivateAccess/
publicAccessCidrs (defaults public=true, private=false, cidrs=0.0.0.0/0) and
reject the both-disabled combination. Public access provisions an
internet-facing NLB and publishes its external frontend IP as the cluster
endpoint; private access keeps the NLB internal and uses the VPC frontend IP.
Inject the resolved endpoint IP into the k3s apiserver tls-san so the
published IP:443 endpoint serves a cert valid for verified TLS, and surface
the access fields in DescribeCluster. The e2e suite now connects via the
verified endpoint with TLS verification on, dropping the insecure mgmt-IP
shim.
* fix(eks): authorize NLB->apiserver ingress on control-plane SG
The cluster control-plane security group was created with empty ingress, so
the NLB's backing LB VM could not reach the apiserver on 6443 from inside the
VPC. The target group stayed unhealthy and the published endpoint never
served. Authorize tcp/6443 on the control-plane SG from the VPC CIDR after the
SGs are created; public exposure remains gated at the NLB front-end by
publicAccessCidrs, so this only opens the in-VPC LB->apiserver hop.
Extend SubnetVPCResolver with GetVPCCIDR so the service can resolve the VPC
CIDR for the rule.
* feat(eks): thread publicAccessCidrs to NLB SetLoadBalancerIngressCIDRs
Wire the cluster's publicAccessCidrs (parsed from CreateCluster
ResourcesVpcConfig) through EnsureClusterNLB to the ELBv2
SetLoadBalancerIngressCIDRs override so a public cluster endpoint can be
narrowed below the wide-open scheme default. The override only fires for
an internet-facing NLB whose CIDRs restrict below 0.0.0.0/0; internal
NLBs and the default keep the LB's out-of-the-box ingress.
NLB ENIs fell back to the VPC default security group, whose intra-SG-only
ingress made the OVN ACL drop inbound listener traffic at the front-end VIP
even with a healthy target group. NLBs reject customer SGs, so mint an internal
managed SG per network LB instead: attach it to every LB ENI at create, open
each listener port on it from the resolved client CIDRs at CreateListener,
close ports on DeleteListener, and delete the SG on DeleteLoadBalancer.
Default ingress CIDRs are scheme-based (internet-facing: 0.0.0.0/0; internal:
the VPC CIDR). SetLoadBalancerIngressCIDRs overrides them and re-authorizes
existing listeners, providing the hook for EKS publicAccessCidrs. ALBs are
unchanged.
* feat(eks): publish eks-node AMI to R2 + spx catalogue entry
Add a spinifex-eks-node catalogue entry so operators install the EKS
node image with a single command:
spx admin images import --name spinifex-eks-node
The entry pulls the Alpine+K3s image from iso.mulgadc.com, verifies its
sha256, and stamps spinifex:managed-by=eks — the tag the control-plane
and node-group launchers resolve the boot AMI by, so no extra config is
needed.
Add scripts/publish-system-image.sh (+ make publish-eks-node-image) to
convert the built raw to a compressed qcow2, emit a checksum sidecar,
and upload both to the R2 bucket backing iso.mulgadc.com/system-ami via
the S3 API.
* ci(eks): add manual workflow to build + publish eks-node AMI to R2
Build the eks-node AMI on the runner (libguestfs) and publish via
publish-system-image.sh using the existing CF_R2_* secrets. Manual
workflow_dispatch with a dry-run input. Fix the publish script's default
bucket to spinifex-iso (the bucket iso.mulgadc.com is served from).
The LB system microVM stripped its module tree to virtio + fw_cfg +
failover, but init.sh only loaded qemu_fw_cfg and virtio_net. virtio_net
depends on net_failover -> failover; modprobe resolves that chain via
modules.dep, and load_mod()'s insmod-by-path fallback loads a single .ko
with no dependency resolution.
When the image is built on a kmod-less host (depmod skipped, no
modules.dep) the chain can't be resolved either way, so virtio_net fails
to load, no NIC appears ('no interface found for MAC'), the lb-agent has
no network, heartbeats never reach AWSGW and every LB hangs in
provisioning until the 5m e2e timeout. Baremetal runners ship kmod so
modules.dep is generated and the chain resolves — which is why the LB
suite passed there but failed on normal CI.
Load failover -> net_failover -> virtio_net explicitly bottom-up so the
NIC comes up regardless of whether modules.dep exists.
EKS create-cluster fails deterministically on the multinode job with
ServiceUnavailableException: the EKS KV is a clustered (Raft) JetStream
stream and create-cluster needs an elected leader, but multinode
JetStream clustering isn't reliably forming yet (see mulga-omq/z2z/5yg).
EKS is validated single-node only at this stage.
Drop eks from the multinode build+run suite filters (alongside the
existing reboot/single exclusions) so the multinode job reflects what's
actually supported. Single-node continues to run the full eks suite.
* feat(elbv2): SetSubnets — full add+remove LB subnet re-homing
Add the ELBv2 SetSubnets operation across the interface, NATS subject,
daemon handler/subscription, and gateway route/forwarder.
Re-homing is a full add+remove: subnets new to the LB get a fresh managed
ENI, dropped subnets have their ENI deleted, then the backing system VM is
relaunched on the new ENI set. The system-instance launcher exposes only
Launch/Terminate (no live ENI hotplug for system VMs), so applying new
subnets relaunches the LB VM — a brief data-plane interruption. The record
is mutated only after the new ENIs exist and the VM is relaunched.
Extract a shared launchLBVM helper from CreateLoadBalancer so create and
SetSubnets boot the LB VM identically. Idempotent on no-op; record-only
fallback when no VPC service is wired.
Completes the elbv2 mutation-API surface (SetSubnets was the last gap).
* feat(tagging): resourcegroupstaggingapi GetResources for controller discovery
Add the Resource Groups Tagging API as a new gateway service (SigV4 scope
\"tagging\", AWS JSON 1.1 / X-Amz-Target, mirroring ACM). GetResources is a
gateway-side aggregator over the existing elbv2 and ec2 tag stores — no new
NATS subject or daemon handler.
It lists the account's load balancers and target groups (attaching tags via
batched DescribeTags) plus EC2 resources from the tag store, then applies the
request's ResourceTypeFilters (service-wide or fully-qualified) and TagFilters
(AND across filters, OR within values). Results are ARN-sorted with opaque
offset pagination.
This is how aws-load-balancer-controller v2 discovers and adopts the elbv2
resources it manages (filtering on its cluster/stack tags); DescribeLoadBalancers
has no tag-filter field, so RGT GetResources is the required path.
* fix(eks): emit canonical exception code, no doubled suffix
GenerateEKSErrorResponse appended \"Exception\" unconditionally, so codes that
already carry it (e.g. awserrors.ErrorEKSResourceNotFound =
\"ResourceNotFoundException\") serialized as ResourceNotFoundExceptionException.
SDK clients comparing eks.ErrCodeResourceNotFoundException then fail to match.
Append the suffix idempotently. Also covers the many other awserrors codes that
already end in \"Exception\" and flow through the shared JSON error envelope
(EKS, ACM, tagging).
* refactor(tagging): split GetResources fetch from orchestration for testing
Introduce a resourceLister seam: GetResources wires a NATS-backed lister, while
getResources (parse + tag-filter + sort + paginate) and buildEC2Mappings (tag
grouping + ARN synthesis) are now pure and unit-tested without a live backend.
No behaviour change.
* feat(eks): per-cluster opt-in K3s built-in ingress + enforce parity default
The runtime-rendered /etc/rancher/k3s/config.yaml emitted no disable: block, so
traefik + servicelb actually ran on every cluster — the AMI config.yaml.skel
that disables them is a never-applied boot fallback (cloud-init overrides it).
Render the disable block by default so traefik + servicelb are off for AWS
parity (Service type=LoadBalancer / Ingress are the AWS Load Balancer
Controller's job). A cluster created with tag spinifex.io/managed-ingress=true
keeps the K3s built-ins for interim in-VPC app exposure before the controller
add-on lands. The flag is derived into ClusterMeta.BuiltinIngress at
CreateCluster and threaded onto K3sServerInput.
* feat(ui): subnet editor on load balancer detail page (SetSubnets)
Add a Subnets tab to the LB detail page backed by the elbv2 SetSubnets
API. The editor mirrors the security-groups editor (checkbox list of the
VPC's subnets, dirty-tracking, Reset/Save) and warns that applying a
change re-homes the load balancer by relaunching its system VM, causing
a brief data-plane interruption.
* fix(elbv2): SetSubnets must detach ENIs before delete/relaunch + roll back on terminate failure
TerminateSystemInstance tears down the LB VM but leaves its ENIs marked
in-use, so the relaunch path could neither delete removed-subnet ENIs
(InvalidNetworkInterface.InUse leak) nor re-attach retained ENIs to the
relaunched VM (attach failed InUse, dropping the retained subnet's data
plane). Detach the existing ENIs explicitly after terminate — mirroring
DeleteLoadBalancer — before deleting removed ones and relaunching.
Also roll back the freshly-created added-subnet ENIs when the terminate
step itself fails, so a failed relaunch no longer leaks ENIs (amplified
by SDK retries).
* fix(daemon): re-arm system-instance terminate subscription on recovery
system.TerminateInstance.{id} was bound only by the system.LaunchInstance
handler at first launch, so it was lost across a daemon restart: recovery
reconnects to the surviving QEMU (or relaunches) but never re-bound the
subject, leaving system.TerminateInstance.{id} with no responder. Any LB
that predated the restart could then never be torn down — SetSubnets
relaunch and DeleteLoadBalancer both failed with 'no responders'.
Re-arm it in onInstanceUpHook, which fires on both the relaunch and the
reconnect-to-surviving-QEMU recovery paths, for ManagedByELBv2 VMs; drop
it in onInstanceDownHook. Split subscribeSystemTerminate into a locked
helper so the hook (which already holds d.mu) can reuse it.
Make e2e-baremetal.yml a reusable workflow (workflow_call + a ref input)
so the nightly and release pipelines can invoke the real PXE-reimaged
hardware suite, which previously only ran on manual dispatch.
Nightly: add a baremetal job (conceptual cell 20) that runs every
weeknight on the existing cron, or on dispatch when 20 is selected. It
tests dev on schedule, matching GIT_BRANCH. Sits outside the tofu matrix
(different runner, provisioning, 120m timeout) and shares the
e2e-baremetal-cluster concurrency group, so all baremetal runs serialise
on the single physical node.
Release: a release builds and uploads an ISO to R2 but never booted that
ISO/squashfs path on hardware. Add a baremetal job (needs: release) and
hard-gate on it: promotion moves to a dedicated promote job
needs:[verify, baremetal] (with a matching mark-failed job on failure),
and the iso job's upload gate now waits for both verify and baremetal to
conclude success before pushing to R2. Bump the iso job timeout to 180m
since it now waits on the parallel ~2h baremetal run.
build-microvm-image required depmod (kmod) as a mandatory tool to
generate modules.dep. Self-hosted CI runners don't ship kmod and can't
reliably apt-install it, so the inline 'apt-get install kmod' steps added
to e2e/e2e-nightly/e2e-ddil were unreliable.
modules.dep is an optimisation, not a boot requirement: the guest init's
load_mod() (build/microvm/init.sh) falls back to insmod-by-path when
modprobe can't resolve a module. So:
- Drop depmod from the mandatory-tool preflight.
- Run depmod only when present; still assert a non-empty modules.dep in
that case to keep catching the trixie-kmod empty-dep bug.
- Warn and skip when depmod is absent (guest uses insmod fallback).
- Remove the apt-get kmod install steps from all three workflows.
Gold-image bake still installs kmod (provision.sh), so production images
get a proper modules.dep; only kmod-less CI runners take the fallback.
* ci(e2e): install kmod (depmod) before microVM image build
#305 made depmod a hard requirement in build-microvm-image.sh, but the
self-hosted ci-single/ci-multi runners don't ship the kmod package, so the
Build Install Artifacts step aborts with 'required tool not found: depmod'
before any E2E suite runs. Add an idempotent kmod install ahead of each
Build Install Artifacts step (single/multi/eks jobs); the gold-image builder
bakes kmod in too, so this no-ops once runners pick up the new image.
* ci: install kmod (depmod) in nightly + ddil too
e2e-nightly and e2e-ddil call the same build-install-artifacts.sh (-> make
build-microvm-image) on the same self-hosted ci-single/ci-multi runners as
e2e.yml, so they hit the identical 'depmod not found' abort. Add the same
idempotent kmod guard before Build Install Artifacts in both.
release distro builds run on GitHub-hosted ubuntu (kmod present); e2e-baremetal
runs on ci-baremetal which already has it (green today) — neither needs the
guard.
* feat(elbv2): localize internal-LB heartbeat failures for baremetal debug
Add diagnostics so a stuck-in-provisioning internal LB can be pinned to
one of three points on the heartbeat path:
- lb-agent failure log now carries the gateway target + region, so the
microVM serial console shows whether the heartbeat hit a transport
error (never reached AWSGW) or an HTTP-level rejection.
- AWSGW LBAgentHeartbeat logs receipt (debug) and warns on the
previously-silent not-found/account-mismatch reject path, so the
journal distinguishes 'heartbeat arrived but cannot transition' from
'heartbeat never arrived'.
* test(e2e/lb): fail fast on stuck internal LB + probe fw_cfg on microvm boot
- lb suite: gate the remaining internal subtests on Internal_ALB. The
internal LBs share one mgmt-return datapath and fixture, so if the
first never reaches active the rest time out identically — skip them
instead of burning ~5m per subtest (~20m saved on a known-bad run).
- microvm init.sh: widen the fw_cfg netcfg poll window and, on the
not-available path, dump DMI / loaded modules / qemu_fw_cfg by_name
listing / dmesg fw_cfg to the serial console before init exits, so a
boot panic is diagnosable from the captured console log.
* fix(microvm): insmod fallback for fw_cfg + fail build on bad modules.dep
Baremetal-built microVM images shipped an unusable modules.dep (the build
host's depmod errors were swallowed), so busybox modprobe in the guest init
silently no-oped, qemu_fw_cfg never loaded, /sys/firmware/qemu_fw_cfg was
absent, init could not read opt/spinifex/netcfg, and the LB microVM panicked
before lb-agent ran — leaving internal LBs stuck in provisioning.
init.sh: load_mod() tries modprobe, then falls back to insmod by explicit
path (.ko/.ko.gz) which needs no modules.dep, so the guest boots regardless
of which host built the image.
build-microvm-image.sh: require depmod, drop the error swallow, and assert
modules.dep is non-empty after depmod so a boot-broken image fails the build
loudly instead of silently.
* fix(microvm): decompress kernel modules before depmod
Debian trixie's kmod 34 depmod silently emits an empty modules.dep when fed
Alpine's gzip-compressed (.ko.gz) modules — reproduced: 928 modules in, 0-byte
modules.dep out. The guest's busybox modprobe then no-ops, qemu_fw_cfg never
loads, and internal LB microVMs panic before lb-agent runs (stuck provisioning
on bare metal; GitHub-hosted runners ship a kmod that reads .gz, so normal CI
was unaffected).
Decompress kept modules to plain .ko after the keep-list strip, before depmod.
Host depmod then produces a usable map regardless of its gzip support, and the
guest loads modules without a decompressor.
* feat(e2e): capacity-aware scheduling for dense single-node runs
Denominate daemon schedulable capacity in physical cores instead of
logical threads, and auto-serialise e2e launches when the cluster is at
capacity, so a single hyperthreaded node stops over-packing guests.
Daemon: NewResourceManager sizes hostVCPU from physicalCoreCount()
(distinct physical-id:core-id pairs in /proc/cpuinfo) rather than
runtime.NumCPU(). Falls back to logical CPUs when topology is absent
(non-Linux, container, ARM cpuinfo without core id). On a hyperthreaded
box this halves the over-pack: with the unchanged 2 vCPU reserve the
i5-12600H schedules 10 vCPU (12 cores - 2) instead of 14, keeping the
boot-spike CPU within real cores.
Harness: capacityRetryEC2 decorates the EC2 client and, on
InsufficientInstanceCapacity, waits and retries RunInstances instead of
failing — turning the daemon admission gate into back-pressure so
concurrent subtests queue. AWSClient.EC2 is now the wrapped
ec2iface.EC2API; EC2Conf keeps the raw *ec2.EC2 for .Config access.
Adds unit tests for parsePhysicalCores (hyperthreaded/hybrid/dual-socket/
fallback) and the retry decorator (retry-then-succeed, non-capacity
passthrough, deadline).
* fix(daemon): SPINIFEX_HOST_VCPU override so small CI hosts can start
Denominating capacity by physical cores starves tiny hosts: a 2-physical-
core CI runner leaves 0 schedulable after the default 2-vCPU reserve, so
NewResourceManager refuses to start and every daemon-constructing unit test
fails. The previous runtime.NumCPU() count (4 logical on the same runner)
incidentally cleared the reserve floor.
Add a SPINIFEX_HOST_VCPU escape hatch (symmetric with the existing
SPINIFEX_RESERVED_* overrides) that pins the detected core count for hosts
whose /proc/cpuinfo topology misreports, and pin it to 4 in the daemon
TestMain so unit tests are deterministic regardless of runner topology.
Reserve stays at the default 2.
* test(eks): EKS control-plane e2e suite (Phase 1, API-level)
Add tests/e2e/eks/eks_test.go driving the EKS lifecycle against awsgw:
CreateCluster -> ACTIVE (endpoint + CA asserted), kubeconfig artifact
assembly, AccessEntry CRUD, get-token presign decode, DeleteCluster -> gone.
Harness: EKS SDK client on AWSClient + WaitForEKSCluster{Active,Deleted}
pollers. Wire e2e-eks into the Makefile and the single-node CI suite list.
* test(eks): Phase 2 kubectl get-nodes data-plane gate
Add the reachability shim (harness/eks_mgmt.go) that reads the control-plane
VM's br-mgmt IP from the cluster KV meta over NATS, plus a kubectl wrapper
(harness/kubectl.go) that skips when kubectl is absent. KubectlGetNodes points
kubectl at https://<mgmtIP>:6443 (insecure-skip-tls-verify; cert SANs the NLB
DNS, not the mgmt IP) with an aws eks get-token exec block and polls until the
control-plane node is Ready — the get-token webhook auth regression gate.
IRSA pod + the real NLB/Route53 passthrough are deferred to a follow-up.
* test(eks): tolerate doubled NotFound code + dial loopback NATS
DeleteCluster detection matched the SDK constant ResourceNotFoundException,
but awsgw emits ResourceNotFoundExceptionException on the wire — match the
NotFound code by substring so deletion is detected immediately instead of
timing out. The mgmt-IP shim dialed NATS.Host verbatim (0.0.0.0:4222), which
the serving cert does not SAN; rewrite the wildcard bind to loopback.
Validated live: all 6 subtests green, kubectl get nodes shows the
control-plane node Ready via the get-token webhook auth path.
* feat: build system images with libguestfs (no host block devices)
Replace the qemu-nbd + sudo mount + chroot + /dev-bind flow in
build-system-image.sh with libguestfs (virt-customize / virt-resize /
guestfish). Every operation runs in an isolated userspace appliance VM with
its own kernel, so the build touches no host block device, no host mount, and
needs no sudo — removing the failure class that wiped the host's /dev on shared
CI runners (a leaked /dev bind traversed by rm -rf).
Applies to all distros: alpine resizes via guestfish e2fsck+resize2fs, ubuntu
via virt-resize; customization (packages, binaries, files, setup script,
service enable, cache cleanup) is one ordered virt-customize invocation.
LIBGUESTFS_BACKEND=direct (no libvirt on runners).
Wire the eks-node AMI into the single-node e2e job: the runner builds the
device-free raw, ships it to the VM, and the VM imports it (predastore/NATS are
VM-only) so the eks suite's CreateCluster has an AMI to boot.
* feat(eks): bootstrap + state-report via AWS gateway broker
Converge the EKS control-plane → host channel onto the ELBv2 lb-agent model:
the K3s server VM publishes its bootstrap envelopes and periodic state reports
via SigV4-signed HTTPS POST to the AWS gateway, instead of dialing core NATS
directly. A new gateway action (PublishInternal, POST
/clusters/{name}/internal-publish) relays the payload verbatim onto the
existing eks.bus.* / eks.state.* subjects, so the daemon-side subscriber, KV
persistence/encryption, and reconciler are unchanged.
This removes the VM's need to reach NATS at all, so NATS can stay
cluster-internal behind its bind/firewall/mTLS controls — the prior direct
path broke when NATS was not VM-reachable (e2e CreateCluster timeout) and was
DDIL-fragile (fire-and-forget nats pub, no ack/retry).
- gateway/eks: PublishInternal broker + route; closed-set channel/kind validation
- cmd/eks-gateway-publish: one-shot SigV4 publisher (retry-until-ack) baked into
the eks-node AMI; replaces the nats-cli publisher in k3s-first-boot.sh and
mulga-eks-state-report.sh
- handlers/eks + daemon: hand the K3s server VM the mgmt-reachable gateway URL +
system (Predastore) SigV4 creds + gateway CA instead of NATS URL/token/CA;
new resolveSystemGatewayBaseURL (mgmt-bridge-first)
- eks-node image: build/install eks-gateway-publish, drop nats-cli download
* feat: migrate eks-token-webhook to the AWS gateway broker
The TokenReview webhook dialed core NATS directly (STS verify + AccessEntry
KV read). On a mgmt-only control-plane VM with NATS bound off-subnet that
fails, so the webhook exited at startup, the apiserver token-webhook
kubeconfig was never written, and the apiserver crash-looped — the cluster
never reached ACTIVE.
Converge it onto the same gateway-broker model as bootstrap/state-report and
ELBv2's lb-agent: the webhook SigV4-signs an HTTPS POST {accountId,token} to
/clusters/{name}/token-review, and the gateway runs the verify + lookup
host-side and returns the resolved identity. The VM now speaks no core NATS.
- internal/eksgw: shared on-VM SigV4 client (also adopted by
eks-gateway-publish).
- handlers/eks/token_review.go: Authenticate decision core (moved from the
webhook binary) + ResolveTokenReview host-side wiring.
- gateway/eks token-review route + WebhookTokenReview handler.
- webhook: drop NATS, relay via eksgw.Client; broker fault -> 503 (retryable)
not a silent deny.
- initd: export EKS_GATEWAY_* / EKS_ACCESS_* creds, not SPINIFEX_NATS_*.
* fix: build eks-node helpers with GOFIPS140; cover token-review broker
eks-gateway-publish blank-imports internal/fipsboot, whose init() panics
unless the binary is built with GOFIPS140=v1.0.0. The eks-node AMI build
omitted the flag, so the binary paniced at first boot right after k3s came
up — k3s-first-boot (set -eu) died before publishing the bootstrap token and
the cluster never reached ACTIVE. Build both eks-node helpers with
GOFIPS140=v1.0.0, matching cmd/lb-agent.
Add NATS-backed unit tests for ResolveTokenReview and the WebhookTokenReview
gateway handler (verify responder + account KV) to cover the new broker path.
* fix: export first-boot env so eks-gateway-publish child sees it
k3s-first-boot.sh and mulga-eks-state-report.sh sourced /etc/spinifex-eks/
first-boot.env but never exported the vars, so the eks-gateway-publish child
process inherited an empty environment and exited '--account-id is required'.
The control-plane VM came up healthy but never published its bootstrap
envelopes, so the cluster never reached ACTIVE. Wrap the source in set -a /
set +a so the KEY=value lines export to child processes.
* fix: make awsgw start --host override actually bind
The 'awsgw start --host' override printed 'Overwriting awsgw host' but the
assignment was commented out, so AWSGW always bound the spinifex.toml host
(a specific WAN IP, e.g. 192.168.0.10). A system instance (EKS K3s control
plane, internal LB agent) has a mgmt-only NIC on-link to the br-mgmt IP with
no route to the WAN IP, so its SigV4 POSTs to the gateway were refused and the
cluster never bootstrapped. Honour --host (e2e passes 0.0.0.0:9999) so AWSGW
listens on all interfaces incl. br-mgmt; the serving cert already SANs every
local interface IP, and WAN clients still reach it.
* test(eks): install kubectl on e2e VM so KubectlGetNodes runs
The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full
aws eks get-token -> kubectl -> apiserver -> token-webhook -> gateway broker
-> STS verify path — the only end-to-end exercise of the token-review broker.
The e2e binaries run on the VM, where kubectl was absent, so
harness.NewKubectl t.Skip'd it and the path went unvalidated. Install a
checksum-verified kubectl (pinned to the k3s server version) on the VM,
gated on e2e-eks.
Internal-scheme LBs on single-node have no dedicated mgmt-bind IP on
br-mgmt, so resolveMgmtRoute falls back to routing the agent to the
advertised WAN IP via the mgmt NIC. That advertised IP lives on a
different bridge than the agent's mgmt NIC, and the cross-bridge return
path never delivers — the agent never heartbeats and the LB hangs in
provisioning (normal single-node dodges this entirely via dev_networking
slirp at 10.0.2.2; bare-metal external-mode is the first real exercise of
this path).
Point the agent at the br-mgmt host IP, which shares its mgmt L2 — no
cross-bridge hop — mirroring the multi-node datapath. Requires AWSGW to
listen on br-mgmt (bind 0.0.0.0); the server cert already SANs that IP.
Internal-scheme LBs only leave provisioning once the lb-agent heartbeats
AWSGW, which requires the mgmt return route resolved by resolveMgmtRoute
(MgmtRoute{Gateway,Target}, or MgmtBridgeIP+AdvertiseIP on single-node).
When none resolve, the route was silently skipped and the LB hung in
provisioning until the client timed out, with no signal as to why.
Log which precondition is empty on the internal skip path, and mark the
LB failed instead of provisioning when no return route exists, so the
broken data-plane surfaces immediately.
InitBuckets was passed the OVN chassis count as the replica factor. When the
chassis count exceeds the NATS node count (single-node bare-metal: 1 node but
2 SBDB chassis), CreateKeyValue requested more replicas than JetStream could
satisfy, the create-or-open fallback returned "bucket not found", and vpcd
crash-looped — cascading to RunInstances ServerInternal. Create at 1 replica
like STS/IAM; the daemon's deferred upgradeJetStreamReplicas raises all KV_*
streams to cluster size when the cluster grows.
* feat(eks): webhook observability — console log sink + per-request TokenReview decision
* fix(eks): wire token-webhook authenticator into generated k3s apiserver config
The generated /etc/rancher/k3s/config.yaml overrides the AMI-baked
config.yaml.skel, but its kube-apiserver-arg list omitted the
authentication-token-webhook-config-file arg. The running apiserver
therefore never loaded the token webhook, so bearer tokens minted by
'aws eks get-token' 401'd without the webhook ever being invoked.
* fix(eks): echo TokenReview audiences so api-audiences apiserver accepts the principal
With --api-audiences set, kube-apiserver sends spec.audiences in the
TokenReview and discards an authenticated=true response that does not
confirm a matching audience. The webhook returned an empty
status.audiences, so a valid IAM principal surfaced as a 401. Echo the
request's audiences on success.
* fix(eks): pin token-webhook version v1 so apiserver accepts the response
kube-apiserver --authentication-token-webhook-version defaults to
v1beta1, but the eks-token-webhook responds with an
authentication.k8s.io/v1 TokenReview. The apiserver decodes the response
against the expected GVK and rejects the mismatch, surfacing a valid
principal as a 401. Pin v1 to match the webhook.
Bare-metal e2e produced no artifact, so RCA fell back to truncated
log-failed output. Tee bootstrap + e2e output to test-baremetal.log,
generate junit-baremetal.xml, run e2e-analyze to emit analysis.md, and
upload the bundle as e2e-baremetal-artifacts-<run_id>.
aws eks list-clusters returned InternalErrorException for a few seconds after a
daemon restart, while describe-cluster (by name) succeeded. During the
post-restart window the per-account KV stream's Raft group has no leader yet, so
kv.Keys() (an ordered consumer over the warming stream) fails with a transient
NATS error. ListClusters wrapped it as a fmt.Errorf string the daemon's
ValidErrorCode cannot map, so it sanitized to ServerInternal — a misleading 500
that tells AWS clients the request is non-retryable.
Classify transient NATS conditions (no-responders, timeout, no-stream-response,
connection-closed) and return a retryable ServiceUnavailable (503) instead;
genuine faults still pass through wrapped. describe-cluster survived only
because kv.Get on the same bucket is cheaper than the consumer kv.Keys spins up.
* feat(iam,awsgw): OIDC provider CRUD + IRSA discovery endpoints (6e)
Complete the AWS-SDK-facing half of IRSA:
- IAM Create/Get/List/DeleteOpenIDConnectProvider, backed by the per-account
oidc-providers registry that STS AssumeRoleWithWebIdentity already reads to
authorise an issuer. Create writes the key STS checks; the provider ARN is
the scheme-stripped issuer per AWS convention.
- awsgw serves the per-cluster OIDC discovery document
(/.well-known/openid-configuration) and JWKS (/keys) as unauthenticated
public endpoints, mounted ahead of the SigV4 surface in their own route
group. The discovery issuer is the cluster's stored OIDCIssuer, byte-exact
with the apiserver iss claim; JWKS is served from the EKS account bucket.
Unit + route tests cover CRUD happy/idempotent/not-found/invalid-url, the
ARN<->issuer round-trip, discovery issuer/jwks_uri equality, JWKS bytes, 404s,
and the no-auth bypass. make preflight green.
* fix(awsgw): route anonymous AssumeRoleWithWebIdentity past SigV4 (6e)
AssumeRoleWithWebIdentity is the IRSA credential-bootstrap call: the AWS SDK
sends it unsigned because the caller holds no AWS credentials, only a projected
ServiceAccount JWT in the body. The gateway ran the SigV4 middleware first and
rejected it with MissingAuthenticationToken, so the true anonymous path never
reached the handler.
Add anonymousSTSInterceptor, a pre-auth middleware mounted ahead of the SigV4
surface. For an unsigned POST it body-peeks the Action and, for the
anonymousSTSActions set, dispatches straight to STS_Request; STS_Request now
skips SigV4-caller resolution for those actions (the handler authenticates the
JWT and gates on the role's web-identity trust policy). Signed requests carry an
Authorization header and pass through untouched, off the body-buffering path.
Surfaced by the 6e live exit gate: a real in-pod assume-role-with-web-identity
would otherwise have been rejected before reaching token validation.
* feat(eks): nodegroup provisioning + K3s agent join (6d)
CreateNodegroup launches N customer-owned worker EC2 from the unified
eks-node AMI in agent role; each joins the cluster via K3S_URL + K3S_TOKEN
baked into cloud-init (first-boot.env SPINIFEX_K3S_ROLE=agent + agent.env).
Workers are customer-owned and DescribeInstances-visible (no ManagedBy,
no mgmt NIC) via the normal RunInstances path, matching real EKS.
K3S_URL targets the control-plane ENI's in-VPC private IP on :6443 — the
cluster NLB DNS endpoint is not resolvable in-VPC yet; switch to the NLB
endpoint on :443 once authoritative in-VPC DNS lands. K3S_TOKEN is the
decrypted cluster node-token persisted at server bootstrap.
Describe/List/Update(scaling)/Delete round out CRUD. Cluster<->nodegroup
SG ingress rules (apiserver 6443, flannel VXLAN 8472/udp, kubelet 10250,
intra-nodegroup) authorized idempotently. UpdateNodegroupVersion deferred.
* feat: gate EKS cluster health via NATS state reports
The K3s apiserver binds the VPC node-ip, unreachable from the host-side
spinifex daemon, so the reconciler cannot probe /healthz directly. Add a
control-plane state reporter that publishes {healthz, node_count} over the
management NATS bus the daemon already shares.
- eks-node server role runs mulga-eks-state-report as an OpenRC service,
publishing eks.state.{account}.{cluster}.server every 30s.
- ClusterReconciler subscribes to that subject, gates CREATING -> ACTIVE on
a healthy report, and records Ready node_count into ClusterMeta.
- node_count counts Ready nodes only; k3s retains terminated workers as
NotReady until pruned, so a raw count never drops on scale-down.
The iso job mounts a kernel-global devtmpfs under the workspace
(.cache/rootfs/dev). A killed job skips its EXIT-trap unmount and leaks
the bind; the verify job (same ci-single runner) then ran a bare
'rm -rf $WORKSPACE/*' that could traverse it and wipe the host /dev.
Detach any mount under the workspace before deleting and add
--one-file-system so rm never crosses a mount boundary. Same backstop
added to the iso job's own cleanup.
A killed ubuntu system-image build can leave `mount --bind /dev` under
its build dir. The e2e Clean Workspace step then ran
`sudo rm -rf "$GITHUB_WORKSPACE"/*` as root, traversed the leaked bind,
and deleted the host's /dev (incl. /dev/null), wedging the runner.
- e2e.yml: umount anything under the workspace (deepest-first) before
deleting, and pass `rm --one-file-system` so rm can never cross a mount
boundary into host /dev. Hard backstop, applied to all 3 Clean Workspace
steps.
- build-system-image.sh: recursive lazy unmount in cleanup() detaches the
chroot binds even when busy, and mark the build subtree rprivate so the
/dev bind can't propagate into the host namespace.
* fix: switch EKS datastore to embedded etcd + fix Alpine cloud-init IMDS route
Flip the K3s control-plane datastore from SQLite/kine back to embedded etcd
(cluster-init: true, etcd-expose-metrics: true). The earlier abandonment
premise — that etcd's per-commit fsync stalls past the apiserver's 5s deadline
on the viperblock-backed root volume — was wrong: etcd inits fine on an
unchanged viperblock (apiserver ready 25-30s, clusters reach ACTIVE). The real
blocker was a network-config regression, fixed here.
Alpine's cloud-init eni renderer crashes (KeyError: 'gateway') on the
gateway-less IMDS on-link route, which failed the whole network-config render
so the static mgmt NIC never came up (no NATS bootstrap, no /healthz). Fixes:
- generateNetworkConfig: add emitIMDSRoute bool; selectNetworkConfigForFamily
passes false for Alpine so the eni renderer never sees the route.
- Deliver the IMDS route out-of-band via a persistent /etc/local.d script +
OpenRC local service. Resolve the VPC NIC via its default route at runtime
(ip route show default), not a hardcoded name: the persistent-net rename to
vpc0 doesn't apply on the Alpine AMI, so the NIC is eth0 at first boot.
- EKS path bakes the same route into buildK3sUserData's own write_files/runcmd;
the generic template Alpine block fires only when no #cloud-config user-data,
avoiding a duplicate write_files key (yaml.safe_load keeps last, silently
dropping a block).
- eks-node image: config.yaml.skel adds cluster-init; comments updated.
* fix: surface CreateCluster step failures at ERROR + map subnet faults; add auth-mode/delete test cases
CreateCluster wrapped its step failures in bare fmt.Errorf that the request
surface collapses to an opaque ServerInternal ("contact AWS re:Post"), so the
real cause never reached the operator. Add logCreateErr to log each step's
precise cause at ERROR before returning, and map a subnet-resolve failure
(caller-supplied parameter) to InvalidParameterValue instead of a retryable
ServerInternal the SDK pointlessly retries. (mulga-siv-165.4)
Tests (mulga-siv-165.15): reject authenticationMode=API_AND_CONFIG_MAP (only
the CONFIG_MAP value was covered); DeleteCluster on a nonexistent cluster via
the full-deps path returns ResourceNotFound (the shim path short-circuits to
ServiceUnavailable before the meta lookup).
* fix: hide EKS control-plane VMs from customer DescribeInstances
EKS K3s control-plane VMs are owned by the customer account (their ENI
lives in the customer VPC for the NLB target), so unlike LB microVMs
(owned by the system account) they leaked into customer
ec2:DescribeInstances. The spinifex:managed-by tag only filtered the UI
Nodes page, not the EC2 API.
Add a caller-aware exclusion: skip any ManagedBy-tagged VM unless the
caller is root. Preserves operator visibility, matches LB behaviour, and
leaves future customer-owned worker nodes (no ManagedBy) visible.
* docs: correct stale SQLite datastore comments in eks-node image (now embedded etcd)
reloadNginx started nginx via cmd.CombinedOutput(). nginx daemonizes, and the
rendered NLB config uses `error_log stderr`, so the forked master and workers
inherit and hold this command's stderr open. CombinedOutput's Wait blocks on
stderr EOF, which the long-lived daemon never sends — so the call never returns,
wedging the agent's synchronous tick() before it applies the nginx engine or
reports health. NLB targets stayed at 0/N until the e2e health wait timed out,
while the data plane (nginx itself) ran fine — so manual smoke tests passed.
ALB is unaffected: haproxy -D redirects its stdio to /dev/null on daemonize, so
CombinedOutput sees EOF and returns.
Capture nginx start/reload stdio to a temp file and use cmd.Run(), so Wait
blocks only on the short-lived foreground process that exits once daemonized.
Surfaced via the LB microVM guest console now collected in CI.
NLB target health stalls at 0/N in CI while the data plane works locally.
The lb-agent's nginx engine activation, any nginx -t/reload error, and its
active-probe results are logged to the LB microVM serial console — which the
host journal never captures and the GetConsoleOutput API path reports as
NotFound, so the failure is undiagnosable from CI artifacts.
- e2e.yml: pull /run/spinifex/console-*.log off each node in the single- and
multi-node collect steps (files persist past VM teardown) into lb-console/.
- lbagent healthcheck: log probe failures at Warn (was Debug) so unreachable
targets surface at the agent's default level on the captured console.
E2E NLB suites timed out at "0/2 healthy" while ALB suites on the same
target instances went healthy in ~5s. The host daemon journal shows the
NLB agent reporting an empty server list (no probe results at all), which
happens when the lb-agent never enters nginx mode — i.e. `nginx -t`/start
fails closed inside the LB microVM, so the agent never probes and reports
nothing. The data plane itself is verified working; this is purely the
active-health-check pathway failing to converge in CI.
- lbagent: pre-create nginx runtime prefix dirs (pid dir + /var/lib/nginx,
/var/lib/nginx/tmp, /var/log/nginx) before `nginx -t`, mirroring
reloadHAProxy's socket-dir pre-create. A stripped Alpine initramfs may
lack these, dead-ending the whole NLB health path.
- build-microvm-image.sh: assert the nginx binary and ngx_stream_module.so
are present in the image and fail the build loudly, instead of surfacing
as an opaque e2e health timeout.
- e2e: capture the LB VM serial console to the artifact bundle on NLB
health-wait failure. The agent's engine activation and any `reload nginx:`
error land on the guest console, which the host journal never sees.
* feat(elbv2): NLB L4 data plane on nginx stream (mulga-siv-952)
Network load balancers render an nginx `stream` config (TCP/UDP/TLS/
TCP_UDP) instead of HAProxy, which cannot load-balance UDP in any
release. The gateway tags GetLBConfig with Engine=nginx; the LB agent
dispatches to the nginx engine (config/pid/cert paths + reload) and
skips HAProxy stats polling. ALB stays on HAProxy. nginx is added to
the microvm image.
* docs(elbv2): note NLB runs nginx stream, ALB runs HAProxy (mulga-siv-952)
* fix(elbv2): load nginx stream module for NLB data plane
Alpine's `nginx` apk is built without the stream module — it ships
separately as `nginx-mod-stream`. Without it, the generated NLB config
fails `nginx -t` with [emerg] unknown directive "stream", so the agent
never applies it. Add the package to the microvm image and emit a
`load_module .../ngx_stream_module.so;` directive at the top of the
generated config.
* fix(elbv2-ui): allow single-subnet LB + full NLB protocols in listener edit
Create-LB form required min 2 subnets while the CLI/backend accept 1;
relax to min(1) to match. Listener add/edit only offered HTTP/HTTPS —
add the NLB protocol set (TCP, UDP, TLS, TCP_UDP), driven by LB type,
with cert/ssl-policy fields gated on secure protocols (HTTPS/TLS).
* feat(elbv2): active target health checks for nginx NLBs
nginx stream has no active upstream probing, so NLB targets never
advanced past initial — DescribeTargetHealth stayed initial forever and
WaitForTargetsHealthy timed out for every NLB. GetLBConfig now delivers
per-target HealthTarget specs (daemon-computed server name, probe
address, protocol/path) for network LBs; the lb-agent probes each target
every tick (TCP dial or HTTP/HTTPS GET) and reports UP/DOWN via the
existing heartbeat path, so the daemon's threshold hysteresis advances
targets to healthy. ALBs are unchanged (HAProxy stats).
Adds an Internal_NLB_UDP e2e suite gating on WaitForTargetsHealthy plus a
UDP datagram round-trip through the VIP.
* test: cover Sprint 3b ENI hot-plug surface
Sprint 3b's gateway + daemon + KV-helper additions landed without unit
tests. dev's diff-coverage gate flipped to 28.5% as a result. This adds
the three test files needed to bring the diff coverage of the ENI
hot-plug surface back above the 60% threshold.
- handlers/ec2/vpc/eni_hotplug_helpers_test.go: GetENIRecord, UpdateENI,
FindENIByAttachment — happy paths, not-found, account scoping,
round-trip of every new ENIRecord field.
- gateway/ec2/vpc/network_interface_hotplug_test.go: Attach + Detach
validation, no-responders, NATS roundtrip with command capture,
daemon error / malformed responses, attachment-resolver branches.
- daemon/daemon_handlers_eni_test.go: eniHotplugErrorCode table,
publishENIHotplugEvent (nil + publish), full handleAttach /
handleDetach matrix (nil data, not-running, ENI/attachment not found,
wrong owner, success with stub QMP + NATS event assertion, hot-plug
failure + KV rollback assertion).
vm.SetHotPlugTestSeams is added as an exported test seam following the
same pattern as utils.SetSudoCommandForTest so daemon-package tests can
swap newDeviceController + eniPipelineSleep without reassigning vm-package
unexported vars.
* feat(vm): wire real OVS tap into ENI hot-plug pipeline
Sprint 3c: replace the stubTapAdd/stubOVSAddPort/stubTapDel/stubOVSDelPort
placeholders with the shared NetworkPlumber. SetupTap creates the kernel tap
and attaches it to br-int with the OVN iface-id binding in one atomic step;
CleanupTap reverses both on rollback and detach. Nil-plumber managers stay a
noop, mirroring setupExtraENINICs.
Adds fake-plumber coverage: tap-on-br-int spec assertion, tap-setup failure
aborts before any QMP call, and QMP/detach paths both invoke CleanupTap.
The previous new-subnet egress baseline used the default VPC, whose main
route table is prebaked with 0.0.0.0/0 -> IGW at bootstrap and reconciled by
the vpcd drift loop, so a subnet created in it is reachable regardless of the
create-time gap — the test passed and never flagged the bug.
Replace it with runNewVPCEgressBaseline: a brand-new VPC where the IGW route
is added to the MAIN route table BEFORE the public subnet is created. The
subnet then implicitly joins a main RT that already carries the route, but
CreateSubnet does not recompute per-subnet egress, so the subnet's OVN egress
LRP is never installed and the instance has no datapath to the IGW. With an
open SG, routing is the only barrier, so the instance is unreachable — red
until CreateSubnet triggers a per-subnet egress recompute.
PoolMode-gated like runVPCSubnetE2E (fresh VPC needs OVN IPAM for a public
IP). runVPCSubnetE2E doesn't catch this: it associates a custom RT then adds
the route, so the route mutation enumerates the subnet and installs the LRP.
Add baseline tests that confirm two fresh-install regressions the existing
suite masks (it authorizes tcp/22 on the default SG before every SSH probe):
- default SG default-deny gates external reach of instances in the
default subnet (SG, not routing)
- a newly-created public subnet never gets its OVN IGW egress LRP, so it
has no route to the VPC IGW despite MapPublicIpOnLaunch
Plus a positive test: instances sharing the default SG can communicate via
ICMP (same-host single-node, cross-host multinode).
Single-node baselines run after TestClusterStatsCLI (0-VM gate); multinode
after TestMultinodePreflight (fixture init). Instances are self-cleaning so
they don't perturb the singleton-VM ordering.
Remove the qemu hostfwd SSH fallback from the harness (InstancePublicSSHHost,
GuestSSHEndpoint) - it bypasses the OVN datapath and masks these regressions.
Endpoints now t.Fatal when no public IP is assigned.
* feat(ui): NLB support + protocol/type-dependent create-LB fields
The Create Load Balancer page was hardcoded to ALB. Add a load-balancer
Type selector (application/network) and switch the listener + target-group
form options on the selected type, mirroring the server's validation:
- ALB: HTTP/HTTPS listeners and target groups.
- NLB: TCP/UDP/TLS/TCP_UDP listeners and target groups; no security groups.
- Secure protocols (HTTPS/TLS) reveal a certificate selector; HTTPS also a
security-policy selector. Cert is required for both; SSL policy is HTTPS-only.
- The default target-group protocol + health-check protocol stay in sync with
the listener (TCP health checks omit HTTP path/matcher).
- The wizard mutation now sends the chosen Type and threads cert/SSL-policy
through CreateListener (previously hardcoded Type=application and dropped
certificates).
* feat(ui): NLB-aware target groups and LB detail
Standalone Create Target Group offers the full protocol set
(HTTP/HTTPS/TCP/UDP/TLS/TCP_UDP) and syncs the health check to TCP for L4
groups, dropping path/matcher from the request and CLI preview. Hide the
Security groups tab on the load balancer detail page for network LBs.
* fix(admin): write ~/.aws credentials 0600 via atomic temp+rename
UpdateAWSINIFile ended with ini.SaveTo, which uses os.Create (0644 &
~umask) and left ~/.aws/credentials world-readable. Render to a 0600
temp file and rename into place so secrets are never exposed.
* fix(eks): purge OIDC signing key on zeroize; pin account bucket History=1
ZeroizeClusterOIDCKey used kv.Delete (a tombstone) which leaves prior
revisions of the encrypted PEM in KV history. Use kv.Purge to drop the
key's whole history. Decouple the account bucket's History from the
schema version (KVBucketEKSAccountHistory=1) so a version bump can't
silently widen history and resurrect ciphertext.
* fix(reconcile): get-or-create leader bucket in AcquireLeader
CreateKeyValue returns 'stream name already in use' once the bucket
exists, so every reconcile after the first hit the 60s deadline and
never elected a leader. Attach to the existing bucket first, create
only when genuinely absent.
* fix(eks): drop registry entry on lease-held instead of phantom holder
RunClusterReconciler returned (nil, nil) when another node held the
lease; Spawn recorded a holder with a nil release, so Has() stayed true
and a later Spawn no-op'd — the cluster was never re-reconciled until
daemon restart. Return ErrLeaseHeld so Spawn drops the entry and retries
after the holder's TTL lapses.
* refactor: tidy EKS cluster state, dedup, and not-found error wiring
- listClusterNames: O(n) map dedup instead of O(n^2) slices.Contains
- drop dead store *Store field + NewStore call from EKSServiceImpl
- ClusterMeta.LastHealthProbe: drop misleading omitzero tag (stdlib
encoding/json ignores it; json/v2 only) and document why it always
serializes
- clarify ErrorResourceNotFound is the ACM code; EKS uses
ErrorEKSResourceNotFound (ResourceNotFoundException)
* test: EKS 6b pagination, restart-resume, and bootstrap bind hardening
- T7: ListClusters NextToken/MaxResults paging + out-of-range clamp
- T4: SpawnRegisteredReconcilers resumes CREATING/ACTIVE on restart,
skips terminal clusters, no-ops when deps absent
- T-quality: replace sleep-based subscription binding with polling
NumSubscriptions, poll KV for the one-shot drop, and assert Run
tears down its subscriptions on return
T1 (restjson wire shape), T5 (JWKS kty mismatch), and T6
(resolveGatewayHost branches) were already covered.
* test(reconcile): cover AcquireLeader get-or-create election
First acquire creates the bucket and wins; a second acquire while the
lock is held attaches to the existing bucket and loses cleanly (the
get-or-create fix, not a CreateKeyValue dead-end); after release a later
acquire takes over.
* feat(ui): HTTPS listeners with ACM certificate + SSL policy
Expose HTTPS in the load balancer listener UI. The ListenerForm gains a
HTTPS protocol option that reveals a certificate selector (backed by a new
ACM ListCertificates query), an inline certificate import dialog
(ImportCertificate), and an SSL security-policy selector (DescribeSSLPolicies).
CreateListener/ModifyListener now carry Certificates + SslPolicy for HTTPS;
the listeners table summarises the policy and certificate per HTTPS listener.
Adds an ACM client (getAcmClient) and @aws-sdk/client-acm. Server, gateway
and ACM endpoints already exist (epic phases A + B); this is frontend-only.
* fix: ACM JSON timestamps, react-compiler watch reactivity, scroll-chaining
- gateway/acm: serialise responses with jsonutil.BuildJSON so time.Time
fields emit epoch seconds (AWS JSON 1.1 wire format). encoding/json
emitted RFC3339 strings the JS SDK rejected with 'Expected real number',
crashing the LB detail Listeners tab once any cert exists.
- frontend: replace watch() with useWatch across forms so the React
Compiler tracks the subscription and re-renders on field change
(HTTPS listener cert/policy fields now reveal). Add toggle regression
test for listener-form.
- frontend: scope overscroll-none to html,body instead of *, restoring
wheel scroll-chaining over nested scroll containers (sidebar, tables,
dialogs).
* feat(eks): unify eks-server + eks-agent into one eks-node AMI
Collapse the two K3s AMIs into a single spinifex-eks-node image whose role
(server vs agent) is chosen per-instance at first boot, halving build time,
storage, and CVE-rotation surface.
- scripts/images/eks-node/: superset manifest + merged setup.sh; all role
services baked but only crond + the new eks-node-role selector enabled.
- eks-node-role first-boot selector reads SPINIFEX_K3S_ROLE, rc-update adds +
starts the matching services (server: k3s + eks-token-webhook + k3s-first-boot
publisher; agent: k3s-agent), records /etc/spinifex-eks/role, self-disables.
Paths are env-overridable; eks-node-role_test.sh covers all five cases.
- mulga-eks-etcd-snapshot.sh early-exits on agent nodes.
- k3s_server_vm.go seeds SPINIFEX_K3S_ROLE=server into first-boot.env.
- Makefile/dev-install: build-eks-node-image / import-eks-node-image replace
the server+agent targets.
- Delete scripts/images/eks-server + eks-agent. The launcher resolves the AMI
by the spinifex:managed-by=eks tag, so running clusters (which hold the
resolved AMI ID) are unaffected.
AMI bake + cluster boot are operational integration steps, not yet run.
* fix(eks): order eks-node-role after cloud-config/cloud-final
The first-boot role selector reads /etc/spinifex-eks/first-boot.env, which
cloud-init delivers via write_files in the cloud-config/cloud-final stages.
The selector only depended on net+cgroups, so on a fast boot it could start
before write_files landed and read a missing role env. Add an after-ordering
on cloud-config + cloud-final so the env file is always present first.
* feat(eks): 6c AccessEntry CRUD + access-policy association
Replace the nine notImpl() AccessEntry/AccessPolicy stubs with real
KV-backed handlers (Q9, API auth mode):
- AccessEntry CRUD persisted at clusters/{name}/access-entries/{hash};
the principal ARN is hashed into the key (ARNs carry ':', illegal in
NATS KV keys) while the record keeps the plaintext ARN.
- AssociateAccessPolicy/Disassociate/ListAssociated against the four
supported AWS-managed policies; unsupported policy ARN or bad access
scope -> InvalidParameterValue.
- ListAccessPolicies returns the static supported catalogue.
- Cluster-existence gate -> ResourceNotFoundException; duplicate create
-> ResourceInUseException; node-type entries rejected until 6d.
Updates the daemon routing test for the now-implemented handlers.
* feat(eks): plumb caller principal ARN + bootstrap-creator-admin
Thread the caller's resolved IAM principal ARN from the gateway to the
EKS control plane via a new X-Principal-ARN NATS header, so CreateCluster
can mint the bootstrapClusterCreatorAdminPermissions AccessEntry (Q9):
- utils.NATSRequest takes optional extra headers; PrincipalARNHeader +
PrincipalARNFromMsg added.
- Gateway resolves the caller ARN best-effort (buildCallerARN) and passes
it down the EKS route table; only CreateCluster consumes it.
- CreateCluster signature gains callerPrincipalARN; default-true bootstrap
flag mints a system:masters entry keyed by the caller's exact ARN (the
token webhook resolves the same ARN from STS). '' degrades gracefully.
- Dedicated daemon handler reads both headers.
* feat(eks): IAM token webhook + STS verify-over-NATS + apiserver wiring
The eks-token-webhook (baked into the eks-server AMI) authenticates kubectl
requests that present an `aws eks get-token` bearer token: decode the presigned
STS GetCallerIdentity URL, verify it, resolve the principal ARN to its cluster
AccessEntry, and answer the apiserver TokenReview with the mapped username +
groups.
STS is hosted in-process in the awsgw service and is not otherwise on the bus,
so awsgw now serves an `eks.VerifyToken` NATS subject that wraps
VerifyPresignedGetCallerIdentity (the cluster name is forwarded so STS enforces
the x-k8s-aws-id anti-replay pin). The webhook reads the AccessEntry KV directly
over its own NATS connection.
Apiserver wiring: the k3s config skeleton sets
authentication-token-webhook-config-file; the webhook writes its own self-signed
serving cert + the apiserver kubeconfig at startup (ordered `before k3s`, with a
bounded wait in the k3s start_pre so the apiserver never loads a missing file).
* fix(eks): omit empty spec from token webhook TokenReview response
* fix(sts): presigned GetCallerIdentity must hash empty body, not UNSIGNED-PAYLOAD
VerifyPresignedGetCallerIdentity built the SigV4 canonical request with
"UNSIGNED-PAYLOAD" as the payload hash. That is an S3-only convention; botocore
and the aws-sdk presign non-S3 requests (sts:GetCallerIdentity, the shape `aws
eks get-token` produces) with the SHA256 of the empty string. Every real
get-token therefore failed signature verification with InvalidIdentityToken,
breaking the entire EKS IAM token-auth flow end to end.
Use the empty-string SHA256. The existing hand-rolled presign tests masked the
bug by signing with the same wrong constant; they now sign correctly, and a new
test presigns with the real aws-sdk-go v4 signer (the same canonicalisation the
CLI uses) so this can't regress silently.
Verified live: a real `aws eks get-token --cluster-name dev1` now resolves to
the caller principal over the awsgw eks.VerifyToken surface; an otherwise-valid
token presented for a different cluster is still rejected (anti-replay intact).
* fix(eks): match access-entry routes on escaped path so slashed principal ARNs resolve
IAM principal ARNs (user/admin, role/foo) embed a '/' that the AWS CLI
percent-encodes in the request path. The dispatcher matched r.URL.Path
(decoded), so %2F became a literal '/' and split the ARN across the
([^/]+) capture, dropping every describe/update/delete-access-entry and
associate/disassociate/list-access-policy call to InvalidActionException.
Match r.URL.EscapedPath() so the encoded ARN stays one segment, then
url.PathUnescape each captured param before it reaches the handler.
* test(eks): unit-cover access-entry scope validation, AWS converters, store guards
Ansible orchestrates the whole stack (spinifex + predastore + viperblock +
OVN + tofu hosts) and its spec/beads live in mulga. Moving it to top-level
mulga/ansible/ ends the per-change submodule-pointer-bump friction.
- git rm -r scripts/ansible/
- Makefile: drop ansible-dev-* / ansible-cluster-bootstrap targets + .PHONY
The tree lands at mulga/ansible/ (flat copy, Option B). History pivot:
tag feat/uplift-ansible-snapshot-pre-move marks the pre-move HEAD.
Plan: docs/development/improvements/ansible-uplift-to-mulga.md (in mulga).
An unresolvable certificate ARN was accepted into a listener record and
only failed later at config-render time, which aborted updateStoredConfig
wholesale and silently froze the load balancer's data-plane convergence.
Validate every certificate ARN against the ACM store (existence + account
ownership) in CreateListener, ModifyListener and AddListenerCertificates,
returning CertificateNotFound up front. Skipped when the ACM store is
unavailable, matching the nil-safe resolve path.
Resolve listener certificate ARNs to PEM via the ACM store (shared
JetStream KV, no cross-service hop), render HAProxy 'ssl crt' for secure
frontends, and deliver the combined cert+chain+key PEM to the LB agent
alongside the config. ConfigHash folds in cert content so rotation
triggers a reload; the agent writes each cert 0600 (path-constrained to
its cert dir) before the config + reload.
Phase B of the ELBv2 HTTPS epic (mulga-siv-180).
Stand up acm as its own AWS JSON 1.1 gateway service wiring the BYO-cert
import/store path ELBv2 HTTPS listeners need. ImportCertificate validates
tls.X509KeyPair, parses the leaf for domain/serial/expiry/key-algorithm,
mints arn:aws:acm:<region>:<acct>:certificate/<uuid>, and stores PEM (cert,
chain, key) in JetStream KV. DescribeCertificate/ListCertificates/
DeleteCertificate round-trip; access is account-scoped.
New gateway dispatch by X-Amz-Target (CertificateManager.<Action>); errors
share the AWS JSON 1.1 envelope with EKS. Wired through gateway/acm ops,
handlers/acm (service iface, store, NATS client), daemon handlers + acm.*
subscriptions, and service-interfaces.yaml.
Out of scope (future, same epic): RequestCertificate + DNS validation,
managed renewal, Get/ExportCertificate, tagging, ACM-PCA.
* feat(eks): cluster_state KV helpers — ClusterMeta + status CAS + prefix sweep
Stage 1 of Sprint 6b CreateCluster lifecycle. Introduces the first
read/write helpers against the eks-account-{accountID} bucket so later
stages (oidc keypair, NLB wiring, cluster reconciler) can persist state
without re-inventing CAS/sweep logic.
- ClusterMeta struct holds the DescribeCluster source of truth (status,
endpoint, OIDC issuer, CA, VPC config, control-plane VM + ENI, NLB ARN).
- ClusterStatus enum mirrors the AWS EKS string set verbatim.
- PutClusterMeta is unconditional (CreateCluster path); GetClusterMeta
returns ErrClusterNotFound on key absence so the service boundary can
map to ResourceNotFoundException.
- SetClusterStatus is revision-checked with bounded CAS retries so the
reconciler and CreateCluster body cannot wedge each other.
- DeleteClusterPrefix sweeps every clusters/{name}/* key; called from
DeleteCluster after the OIDC private key is zeroized.
Tests cover round-trip, missing/corrupt blob, idempotent status set,
DELETING transition from every prior state, concurrent revision bump
recovery, sibling-cluster preservation, empty-bucket no-op.
Plan: docs/development/feature/eks-6b-create-cluster.md
* feat(eks): per-cluster ECDSA-P256 OIDC keypair generation + zeroize
Stage 2 of Sprint 6b. Lays the at-rest primitives the CreateCluster body
will call before launching the K3s server VM. Pure crypto helpers — no
daemon, no launcher, no NATS bus.
- GenerateClusterOIDCKeypair mints an ECDSA-P256 keypair per cluster
(matches eks-v1.md Q8), PEM-marshals the private as PKCS8,
envelope-encrypts it via handlers_iam.EncryptSecret (AES-256-GCM,
AwsGW masterKey), writes the ciphertext to OIDCSigningKeyKey, and
publishes the public JWKS document to OIDCJWKSKey. Returns the JWKS
bytes so CreateCluster can hand them straight to user-data without
a follow-up KV read.
- JWKS shape mirrors what handlers_sts.JWKS already decodes for IRSA
verification (RFC 7517 EC P-256, kid = base64url SHA-256 of SPKI DER,
alg ES256, use sig). Types are defined locally to avoid a
handlers_eks → handlers_sts → handlers_eks import cycle; JSON wire
compat is the contract.
- LoadClusterOIDCPrivateKey is the recovery-path read for future
stages (reconciler restart, restore-snapshot tooling). Validates
PKCS8 parses to an ECDSA P-256 *PrivateKey and rejects anything else.
- ZeroizeClusterOIDCKey overwrites the encrypted blob with zeros of
the same length, then deletes. DeleteCluster calls this *first* so a
crash mid-teardown leaves no recoverable key material. Idempotent
on missing key — supports retry semantics.
Tests cover JWKS shape (RFC 7517 EC P-256 with 32-byte SEC1
coordinates), per-cluster key uniqueness, round-trip private-key
decrypt matches JWKS public, wrong-master-key fails, missing-key
ErrClusterNotFound mapping, zeroize idempotency, JWKS survives
zeroize (carries no secret material), padCoord left-pad correctness.
Plan: docs/development/feature/eks-6b-create-cluster.md
* feat(eks): cluster SG + NLB ensure/delete helpers
Stage 3 of Sprint 6b CreateCluster lifecycle. Adds the customer-account
SG provisioning + per-cluster NLB plumbing the CreateCluster body will
call between OIDC keypair (Stage 2) and K3s VM launch (Stage 4). Pure
interface-call helpers — no daemon, no KV, no NATS bus.
- EnsureClusterSGs creates (or returns) eks-cluster-{name}-control-plane-sg
and eks-cluster-{name}-nodegroup-sg in the customer VPC, tagged
spinifex:managed-by=eks + spinifex:eks-cluster=<name> + spinifex:eks-role.
Idempotent on group-name + vpc-id pair. DeleteClusterSGs is best-effort
with continue-on-error sweep.
- EnsureClusterNLB provisions the eks-<name> NLB (internal scheme), the
eks-<name>-cp target group (TCP, port 6443, target-type ip per eks-v1.md
Q13), and the TCP :443 forward listener. Returns ClusterNLB
{LoadBalancerArn, TargetGroupArn, ListenerArn, DNSName} so CreateCluster
can persist every ARN in ClusterMeta. Idempotent on cluster name;
resource-name length validated against the AWS 32-char cap so a too-long
cluster name fails with a clear error instead of a generic
InvalidParameterValue downstream.
- RegisterClusterTarget + DeregisterClusterTarget bind the K3s server VM
ENI IP to the TG on :6443. Stage 4 calls Register once the VM has its
ENI IP; DeleteCluster calls Deregister before VM termination.
- DeleteClusterNLB tears down NLB + TG. DeleteLoadBalancer already
cascades listener cleanup inside the ELBv2 impl. LB sweep error takes
precedence so DeleteCluster surfaces the more visible failure first
while the TG sweep still runs.
Narrow sgProvisioner + nlbProvisioner interfaces in-package give tests
hand-rolled recorder fakes; the production VPCService + ELBv2Service
satisfy them structurally so no adapter is needed at Stage 6 wire-up.
Tests cover fresh-creates with right names/tags/ports, idempotent reuse,
mixed-existence (control-plane exists, nodegroup created), name-too-long
rejection, empty-inputs rejection, missing-resource no-op on delete, and
first-error-surfaced sweep continuation on delete.
Plan: docs/development/feature/eks-6b-create-cluster.md
* feat(ec2): RunInstances honors pre-created NetworkInterfaceId
Closes the AWS-parity gap where RunInstances always auto-created a
primary ENI even when the caller specified
NetworkInterfaces[0].NetworkInterfaceId. The EKS Sprint 6b CreateCluster
launcher (Stage 4) needs to pre-create the K3s server VM's ENI so the
ENI can be tagged/owned distinctly from the launcher's instance; the
existing path silently created a second ENI and left the caller's
unattached.
Changes:
- ENICreator interface gains GetENI(accountID, eniID) → *ENIInfo. ENIInfo
is a translated struct (SubnetID, VpcID, PrivateIp, MAC, Status,
SecurityGroupIDs) so handlers/ec2/instance stays free of an import on
handlers/ec2/vpc.ENIRecord.
- daemon/vpc_adapter.go gains daemonENICreator.GetENI translating
VPCServiceImpl.GetENIRecord into ENIInfo.
- PrepareRunInstances inspects NetworkInterfaces[0].NetworkInterfaceId
after the existing subnet/SG lift block. If non-empty, the loop
dispatches to a new attachPreCreatedENI helper:
- Looks up the ENI under accountID.
- Rejects with InvalidNetworkInterface.InUse if Status == "in-use".
- Calls AttachENI to mark the ENI in-use and bind it to the new VM.
- Populates instance.ENIId / ENIMac and ec2.Instance network fields
(PrivateIp, SubnetId, VpcId, SecurityGroups, NetworkInterfaces[0]).
- Skips the auto-create + public-IP-auto-assign paths; the caller is
expected to manage public IP out-of-band.
- Auto-create path is left untouched for every other code path: callers
that pass SubnetId without a NetworkInterfaceId still get a fresh ENI
with the existing semantics.
Tests cover the four shapes: happy path (auto-create not called, attach
called once, instance populated), in-use ENI rejected with the canonical
AWS error code, lookup error surfaces verbatim, attach error surfaces
verbatim.
* feat(eks): K3s control-plane VM launcher with cloud-init user-data
LaunchK3sServerVM resolves the eks-server AMI by name, pre-creates the
customer-account control-plane ENI in the supplied subnet with the
control-plane SG, renders cloud-init user-data (first-boot env, OIDC
private key, K3s config.yaml with tls-san + service-account args), then
calls RunInstances with NetworkInterfaces[0].NetworkInterfaceId pointing
at the pre-created ENI. RunInstances failures roll back the ENI so the
caller does not leak a customer-account resource.
TerminateK3sServerVM tears the instance and ENI down for DeleteCluster;
missing identifiers are a no-op so retries stay safe.
ENI + VM both live in the customer account (deviation from plan §7 which
specified a SystemAccount-owned VM); cross-account ENI attach is deferred
to a follow-up bead.
* feat(eks): NATS bootstrap subscriber + cluster reconciler + leader lease
NATSBootstrap subscribes to the four single-shot subjects the K3s server VM
publishes (k3s-bootstrap-token, k3s-admin-kubeconfig, k3s-oidc-jwks, k3s-ca)
on eks.bus.{accountID}.{clusterName}.{kind}. Token and admin kubeconfig are
envelope-encrypted via handlers_iam.EncryptSecret and persisted to KV. The
published JWKS is cross-checked against the controller-generated keypair by
kid and rejected on mismatch. The CA lands on meta.CertificateAuthorityB64
via the new SetClusterCertificateAuthority CAS updater.
ClusterReconciler holds a leader lease on spinifex-eks-leader/{accountID}/
{clusterName} (60s TTL, 20s refresh) and drives:
- CREATING -> ACTIVE once the four bootstrap KV artifacts are present and
NLB /healthz responds 200.
- ACTIVE -> probe /healthz on the configured interval, warn on failure.
- DELETING / FAILED -> exit (DeleteCluster body owns teardown).
Loss of the leader lease (CAS conflict or different holderID) returns
ErrReconcilerLeaseLost so the caller can re-acquire from another node.
Tested with embedded JetStream + a stub HTTPDoer; race-clean.
* feat(eks): OIDC issuer URL builder + reconciler registry
ClusterOIDCIssuer renders the {gatewayBaseURL}/oidc/eks/{region}/{accountID}/
{clusterName} URL kube-apiserver advertises as service-account-issuer and
that STS / external IRSA validators dereference. Validates URL safety on the
path segments so a hostile cluster name cannot inject scheme/path fragments.
ReconcilerRegistry tracks per-(accountID, clusterName) reconciler goroutines
with idempotent Spawn (drop duplicate spawn calls), Stop (cancel one + wait
for release), and StopAll (graceful daemon shutdown). RunClusterReconciler
is the production SpawnReconcilerFn: constructs a ClusterReconciler,
acquires the leader lease, runs Run in a goroutine. Both helpers consumed
by the upcoming service_impl CreateCluster body.
* feat(eks): CreateCluster/DescribeCluster/ListClusters/DeleteCluster bodies
Wires the EKSServiceImpl orchestration deps (VPC SG/K3s/Subnet, NLB,
Instance launcher, AMI resolver, master key, gateway base URL, region,
holder ID, reconciler registry) and implements the four lifecycle
methods:
- CreateCluster: validate → CREATING meta → SGs → NLB → OIDC issuer +
keypair → K3s server VM (cloud-init carries OIDC private PEM) →
register ENI IP with NLB target group → spawn NATS bootstrap
subscriber + per-cluster reconciler via the registry.
- DescribeCluster: KV meta read → eks.Cluster shape, with
ErrClusterNotFound translated to ResourceNotFoundException at the
service boundary.
- ListClusters: KV key-prefix scan + sorted maxResults/nextToken
pagination.
- DeleteCluster: SetClusterStatus DELETING → zeroize OIDC key → stop
reconciler via registry → deregister NLB target + delete NLB →
terminate K3s VM → delete SGs → delete cluster KV prefix.
Daemon wiring:
- buildEKSServiceDeps loads MasterKey from BaseDir/config/master.key
best-effort, resolves the gateway base URL via the same precedence the
LB-agent block applies later, and passes VPCService + ELBv2Service +
ImageService directly while adapting InstanceServiceImpl via
daemonEKSInstanceLauncher (RunInstances passes through; Terminate-
Instances loops vmMgr.Get + StopOrTerminateInstance in-process).
- SubnetVPCResolver is satisfied by daemonEKSSubnetResolver wrapping
VPCService.GetSubnet.
- Startup spawns reconcilers for every persisted CREATING/ACTIVE cluster
via SpawnRegisteredReconcilers; shutdown calls eksService.Shutdown so
every per-cluster goroutine releases its lease before exit.
Test surface:
- service_impl_test: TestEKSServiceImpl_ClusterLifecycleShimMode covers
the four lifecycle methods when constructed via the back-compat shim
(CreateCluster/DeleteCluster → ServerInternal, DescribeCluster →
ResourceNotFound on empty bucket, ListClusters → empty list).
- daemon_handlers_eks_test: drop the four lifecycle subjects from the
NotImplemented routing loop; handler-side coverage now lives in
handlers/eks.
- awserrors: add ResourceNotFound (404) for the DescribeCluster path.
* fix(eks): load master.key from config dir not service data dir
eks_deps read master.key from {BaseDir}/config/master.key, but the daemon
systemd unit sets SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex (a per-service
data dir). admin init writes master.key to the shared config dir, so the
daemon looked in a path that only resolved when a {BaseDir}/config symlink
existed — awsgw has one, the daemon's nested data dir does not. Result:
MasterKey=nil, depsReadyForOrchestration() false, CreateCluster/DeleteCluster
collapse to ServerInternal.
master.key is a config artifact written next to spinifex.toml. Read it from
filepath.Dir(d.configPath), matching how the daemon already resolves
predastore.toml and TLS certs. No symlink or provisioning change needed.
* fix(eks): register + resolve eks-server AMI via tag, add import --ami-name
CreateCluster's K3s launch failed 'eks: AMI eks-server not found' on every
clean install. Two defects:
1. spx admin images import had no way to name a locally-built appliance AMI:
--name is a catalog lookup (rejects non-catalog values) and the AMI name
was hardcoded to ami-{distro}-{version}-{arch}. build-system-image.sh ran
'import --name $AMI_NAME' for eks-server/eks-agent/GPU images, which all
failed 'Image name not found in available images'.
2. The launcher resolved DescribeImages by name=='eks-server', a name the
pipeline could never produce.
Add --ami-name to override the registered AMI name on --file imports (fixes
eks-server, eks-agent, and GPU appliance imports). Resolve the EKS AMI by the
spinifex:managed-by=eks tag the pipeline already stamps, newest-wins on
multiple matches. Tag-based resolution survives the planned server+agent →
single EKS AMI unify untouched (role chosen per-instance at launch).
build-system-image.sh now passes --ami-name; Makefile gains
import-eks-server-image (IMPORT=1) and dev-install.sh imports the AMI when
DEV_INSTALL_EKS=1.
* fix(eks): give K3s server VM NATS token + CA so bootstrap can publish
The K3s control-plane VM was launched with NATSURL=ConnectedUrl() (loopback,
unreachable from the VM) and no auth material, while spinifex NATS requires a
shared token over TLS. The VM could not connect to NATS, so it never published
its four one-shot bootstrap messages and the cluster stayed in CREATING.
Hand the VM the daemon's shared NATS token + CA PEM and a VM-reachable NATS URL
(mirrors the ELBv2 LB-agent shared-credential pattern; spinifex has no nkeys
hierarchy to scope per-principal creds against).
- K3sServerInput: drop dead NATSCredsContent; add NATSToken + NATSCACert,
required by validateK3sServerInput. user-data writes SPINIFEX_NATS_TOKEN,
SPINIFEX_NATS_CA + the CA file; first-boot.env tightened to 0600.
- EKSServiceDeps + CreateCluster: pass NATSURL/NATSToken/NATSCACert.
- buildEKSServiceDeps: read CA from config.NATS.CACert; extract
resolveGatewayHost(); add resolveNATSURL() (same reachable host, NATS port).
- k3s-first-boot.sh: build nats CLI args with --token + --tlsca.
* fix(eks): retry transient bootstrap persists, fail cluster on permanent error
The NATS bootstrap subscriber tore down on the first persist error and the
cluster sat in CREATING forever. Core-NATS delivers a publication once, so a
transient KV blip during persist was unrecoverable and a malformed payload
(bad JWKS kid, non-base64 CA, empty envelope) silently stranded the cluster.
- persistWithRetry rides out transient persist failures in-handler (3 attempts,
200ms backoff) instead of unsubscribing and waiting for a re-publish that
never comes.
- errBootstrapPermanent classifies validation failures (malformed envelope,
empty fields, JWKS kid mismatch, non-base64 CA) as non-retryable so they
return immediately.
- spawnBootstrap marks the cluster FAILED (with reason) when Run exits with a
non-context error, so DescribeCluster surfaces the cause and DeleteCluster
can reclaim resources.
- MarkClusterFailed transitions to FAILED+StatusReason only from CREATING, so a
late bootstrap error cannot clobber a concurrent delete or healthy cluster.
- casUpdateMeta extracts the revision-checked RMW loop shared by the meta
setters.
* fix(eks): resume NATS bootstrap for CREATING clusters on daemon restart
SpawnRegisteredReconcilers re-spawned only the reconciler on boot, never the
bootstrap subscriber. A daemon restart while a cluster was CREATING (K3s VM
still booting) lost the subscriptions; the VM published its four one-shot
core-NATS messages to nobody and the cluster hung in CREATING forever.
- BootstrapPendingKinds reports which artifacts (token, kubeconfig, CA) are
still missing from KV. JWKS is excluded: it persists no artifact and
re-waiting on a once-published subject would deadlock — its cross-check is
best-effort across a restart.
- RunForKinds subscribes to a subject subset (nil = all four); Run delegates to
it. subjectHandlers maps each kind to its handler and rejects unknown kinds.
- SpawnRegisteredReconcilers resumes bootstrap for CREATING clusters, waiting
only on the missing artifacts, so a mid-CREATING restart still drives the
cluster to ACTIVE.
* fix(eks): DeleteCluster must not wipe KV or report success on teardown failure
DeleteCluster ran zeroize, NLB teardown, and VM termination as warn-and-continue
then unconditionally swept the per-cluster KV prefix and returned success. A
failed NLB or VM teardown leaked billable resources with no owning record (the
meta holding their ARNs/IDs was erased) and told the operator the cluster was
gone. Zeroize — a stated security guarantee of no recoverable key material —
had its failure ignored too.
- Zeroize, NLB delete, and VM terminate are now blocking: their failures
accumulate (errors.Join) and abort the call before the KV sweep, leaving the
cluster DELETING and returning the error so a retry can still reach the ARNs.
- KV prefix sweep runs only after NLB + VM teardown confirm.
- SG cleanup stays best-effort (plan step 5) — a leaked SG strands no owning
record — and target deregister stays best-effort since DeleteClusterNLB tears
down the whole NLB + target group.
* fix(eks): persist NLB + control-plane ARNs mid-CreateCluster so failures don't leak
CreateCluster set meta.NLBArn / target-group / instance / ENI in memory and
persisted them only in the final PutClusterMeta, after LaunchK3sServerVM. Any
failure between EnsureClusterNLB and that persist left the NLB + target group +
listener + backing LB VM created but unrecorded; markFailed did no rollback and
DeleteCluster keys teardown off the persisted ARNs, so the resources leaked.
- Persist meta immediately after EnsureClusterNLB (records NLB/TG ARNs) and
again after LaunchK3sServerVM (records instance + ENI), before the next
fallible step.
- A failed create now leaves a FAILED meta carrying the resource references, so
the DeleteCluster teardown path (165.9) reclaims every NLB/TG/listener/VM.
- Tests: NLB-arn-persisted-before-later-failure, failed-create-then-delete
reclaims NLB, and a CreateCluster happy path. Shared eksServiceFixture
extracted from the delete-cluster fixture.
* fix(sts): parse path-based EKS OIDC issuer so IRSA accepts 6b cluster tokens
ParseEKSIssuerURL required an `oidc.eks.` host prefix and an exactly-2-segment
path (AWS hosted-EKS shape), but ClusterOIDCIssuer emits — and awsgw serves —
https://{gateway-host}/oidc/eks/{region}/{accountID}/{clusterName}. Every 6b
cluster's iss claim was rejected with "issuer host must start with oidc.eks",
so AssumeRoleWithWebIdentity would fail all IRSA tokens once 6e serves them.
Reconcile on the path-based shape, which is what is actually reachable on
bare-metal/on-prem (no oidc.eks.* vhost or DNS exists). Host trust comes from
the IAM OIDC-provider registration (the full issuer string is the registry key)
and the JWKS signature check, not a host pattern — so the parser drops the host
prefix rule and instead pins accountID/clusterName at fixed positions in a
strict 5-segment /oidc/eks/{region}/{accountID}/{clusterName} path. A crafted
iss with extra or missing segments still fails closed.
Adds a cross-component round-trip test asserting a ClusterOIDCIssuer-built iss
parses back to the same (accountID, clusterName).
* fix(eks): reclaim FAILED clusters on retry + cluster-scoped exists/not-found errors
A failed CreateCluster left a FAILED ClusterMeta that poisoned retries: the
preflight returned ErrorELBv2TargetGroupInUse ("The target group is currently
in use...") for a cluster-exists condition, and because that maps to a
retryable code the SDK's auto-retry surfaced ResourceInUseException instead of
the original failure. FAILED clusters were also a dead state — visible to
neither a clean recreate nor a clear lifecycle.
- Add EKS-specific ResourceInUseException / ResourceNotFoundException codes with
cluster-scoped messages; CreateCluster's exists check and DescribeCluster /
DeleteCluster not-found now use them instead of the ELBv2 / EC2 codes.
- CreateCluster reclaims a FAILED cluster: it purges the prior attempt's
recorded resources and clears KV before recreating, so a retry (including the
SDK auto-retry) is idempotent rather than blocked. Teardown failure surfaces
rather than recreating on top of orphans.
- Extract purgeClusterInfra, shared by DeleteCluster and the reclaim path
(blocking zeroize/NLB/VM, best-effort SG, KV sweep only after teardown
confirms).
- FAILED clusters remain observable via Describe (status=FAILED) and List — no
state where create blocks but describe/list show nothing.
* fix(eks): fail CREATING clusters that exceed create timeout
A per-cluster reconciler that never observes a ready control plane (bootstrap
artifacts missing or healthz unreachable) previously stayed CREATING forever.
Add a configurable create timeout (default 15m): once meta.CreatedAt is older
than the deadline and the cluster is still not ready, the reconciler marks it
FAILED with a reason, exiting the loop. Within the deadline it stays CREATING.
* fix(eks): surface ACTIVE cluster health in describe-cluster
The reconciler only slog.Warn'd a failing /healthz probe for an ACTIVE
cluster, so describe-cluster kept reporting a dead control plane as healthy.
Add HealthIssue + LastHealthProbe to ClusterMeta, write them from the ACTIVE
reconcile branch (recording the failure reason, clearing on recovery), and map
a non-empty HealthIssue to a ClusterHealth ClusterUnreachable issue in
describe-cluster.
* test(eks): add plan-mandated CreateCluster/DeleteCluster service cases
service_impl_test only covered shim mode. Add the four cases the plan's Testing
section requires: CONFIG_MAP auth mode rejected (and API accepted), missing
subnetIds rejected, DescribeCluster 404 with full deps wired (past the shim
short-circuit), and the DeleteCluster zeroize-before-teardown ordering proven
by forcing a teardown failure and asserting the OIDC key is already wiped.
* refactor(eks): return OIDC private-key PEM from generator, drop re-read
CreateCluster discarded GenerateClusterOIDCKeypair's return then did a second
KV get + AES-GCM decrypt via loadOIDCPrivateKeyPEM to re-derive the same PEM the
generator already had in hand. Return the plaintext PEM (alongside the JWKS)
from the generator and use it directly; remove loadOIDCPrivateKeyPEM and the now
-unused crypto/x509 + encoding/pem imports in service_impl.
* fix(eks): make CreateCluster/DeleteCluster failures diagnosable
Unmet orchestration deps and a missing eks-server AMI both collapsed to a bare
ServerInternal with no request-time log, leaving the client a 'contact AWS
re:Post' message and zero cause. Enumerate the specific missing deps and log
them at ERROR, returning ServiceUnavailable (an operator/config fault, not an
unrecoverable internal one). Add an ErrEKSServerAMINotFound sentinel so a
missing control-plane AMI is logged and surfaced as ServiceUnavailable too.
* fix(eks): serialize REST-JSON responses via jsonutil so the AWS SDK can parse them
WriteJSONResponse marshaled aws-sdk-go *Output structs with encoding/json,
which ignores their locationName tags and emits Go PascalCase keys. EKS
clients parse restjson on the lowercase locationName, so every field of
list/describe/create/delete responses was silently dropped (empty endpoint,
null identity.oidc.issuer). Marshal via jsonutil.BuildJSON — the same
marshaler the SDK uses for restjson bodies — which honors locationName.
Adds wire-shape tests asserting lowercase keys present and PascalCase absent.
* feat(eks): launch K3s server VM via boot-agnostic system-instance path
The EKS K3s control-plane VM launched through the plain tenant RunInstances
path, so it had only its VPC ENI and no management-bridge NIC -- no route to
the daemon's NATS endpoint. It never published its bootstrap subjects and the
cluster stayed CREATING.
Generalize LaunchSystemInstance to a boot-agnostic contract supporting both
system-instance boot styles:
- BootDirect: direct-boot microVM (fw_cfg blobs) -- ELBv2 LB agent
- BootAMI: AMI boot + cloud-init seed (q35) -- EKS K3s server VM
The launch contract moves to a neutral handlers/sysinstance package so EKS can
launch system VMs without an eks->elbv2 dependency; handlers/elbv2 re-exports
the types as aliases. The BootAMI branch mirrors the daemon's
Prepare -> Insert -> Launch split and allocates the mgmt NIC on the VM record
between Insert and Launch, so the cloud-init network-config renders a static
mgmt0 interface. The EKS NATSURL is now sourced from the mgmt-bridge IP (the
on-link address the VM's mgmt NIC can reach), falling back to the gateway-host
resolution when no mgmt bridge exists.
Verified: K3s VM now boots from the eks-server AMI under q35 with a mgmt0 NIC
(10.15.8.x), reachable from the host over br-mgmt, with NATS on-link at the
bridge IP. No AMI rebuild required.
* fix(eks): make DeleteCluster K3s teardown idempotent so it can't wedge
TerminateK3sServerVM treated every DeleteNetworkInterface error -- and every
TerminateSystemInstance error -- as blocking. Because instance terminate is
async, the immediate ENI delete races it: the first attempt fails
InvalidNetworkInterface.InUse, and once the instance-terminate cascade releases
the ENI every retry fails InvalidNetworkInterfaceID.NotFound. Both joined into
purgeClusterInfra's teardownErrs gate, which short-circuits before the SG sweep
and KV prefix delete -- so once the ENI was gone the cluster was stuck DELETING
permanently, leaking its control-plane + nodegroup SGs and KV meta.
Classify the teardown errors instead of blanket-propagating:
- ENI InvalidNetworkInterfaceID.NotFound / InvalidNetworkInterface.NotFound
-> idempotent success (the cascade or a prior retry already deleted it);
must not block the downstream SG + KV sweep.
- ENI InvalidNetworkInterface.InUse (VM still terminating) and any other
error -> retryable; surfaced so the cluster stays DELETING and the
reconciler retries once the instance releases the ENI.
- Instance already gone -> idempotent. TerminateSystemInstance now returns a
sysinstance.ErrSystemInstanceNotFound sentinel that the EKS path tolerates
via errors.Is, so a retried delete after the VM drained completes.
Add awserrors.IsErrorCode(err, code) as the shared delete-error classifier
(substring match through %w wrapping) and route the lone ad-hoc strings.Contains
guard in daemon/vm_adapters.go through it.
Verified: delete-cluster on three clusters left wedged DELETING / FAILED (ENI +
VM already gone) all complete -- clusters erased, SGs + ENIs reclaimed, no leak.
* fix(eks): gate ACTIVE on verified JWKS cross-check, not pre-seeded key
The controller pre-seeds OIDCJWKSKey at create time, so its presence
proved nothing about the running cluster's actual signing key. A cluster
could reach ACTIVE — and start signing IRSA tokens — before the K3s VM's
published JWKS was ever cross-checked against the generated keypair.
Add an OIDCJWKSVerifiedKey marker written by persistJWKS only after
assertJWKSMatch passes. The reconciler's ACTIVE gate and
BootstrapPendingKinds now key off the marker, not the pre-seeded JWKS.
assertJWKSMatch is map-based and also rejects kty mismatch.
* fix(eks): collapse duplicated gateway-host switch into resolveGatewayHost
startCluster reimplemented the 5-case gateway-host precedence inline,
duplicating resolveGatewayHost (used by the OIDC issuer and EKS NATS URL).
If one drifted, the OIDC issuer host embedded in cluster tokens would
diverge from the host LB VMs use to reach the daemon -> IRSA iss mismatch
and cluster-wide token rejection.
startCluster now calls resolveGatewayHost and derives the multi-node
mgmt host route (mgmtRouteVia) from the resolved host. resolveGatewayHost
carries the canonical case enumeration; a table test pins all five
branches plus the case-1 skip guards and the no-host fallthrough.
* fix(eks): make eks-server VM boot diagnosable + fix first-boot publisher crash
Three issues found while diagnosing why K3s clusters never leave CREATING:
1. setup.sh: the stock Alpine cloud image lists `console=tty0` LAST in
default_kernel_opts, so Linux binds /dev/console to the framebuffer and the
host-side serial capture sees only kernel dmesg — OpenRC, cloud-init, and
k3s-first-boot output were invisible. Reorder so console=ttyS0 is last.
2. k3s-first-boot.initd: command_background was "no" while output_log/error_log
were set; start-stop-daemon rejects --stdout/--stderr without --background
("Failed to start k3s-first-boot"), so the bootstrap publisher never ran and
the cluster wedged in CREATING. Background it (the publisher waits up to 5min
on K3s /healthz and must not block the runlevel anyway) and route its output
to the serial console, which the orchestrator captures (the VM has no login).
3. k3s-first-boot.sh: on the /healthz timeout, dump the tail of /var/log/k3s.log
to the console so the apiserver's own failure is visible host-side.
* fix(eks): seed K3s apiserver a public OIDC key, not the private key
buildK3sUserData pointed both kube-apiserver service-account args at the same
private-key file. --service-account-key-file requires a PUBLIC key — k8s
ParsePublicKeysPEM rejects a private-key PEM with "data does not contain any
valid RSA or ECDSA public keys" and the apiserver crash-loops, so K3s never
reaches /healthz, first-boot never publishes, and the cluster wedges in
CREATING.
Add PublicKeyPEMFromPrivate (PKIX public PEM from the PKCS#8 private key) and
seed it to a separate /etc/rancher/k3s/oidc-signing-key.pub.pem. Point
service-account-key-file at the public file; keep service-account-signing-key-file
on the private file. Runtime-verified: the apiserver crash is gone and K3s now
starts and schedules system pods.
* feat(eks): egress-only SNAT + DNS for K3s control-plane VM
The K3s server VM (a hidden system instance) could not pull system-pod
images: it had no internet egress and no working resolver, so
coredns/traefik/metrics-server never started and /healthz timed out.
Egress: give the single control-plane VM egress-only internet, confined
to its /32, with no inbound exposure and no customer-visible resource.
- policy/route.go: SystemInstanceEgressPriority (1200, above the drop
gate) + AddSystemInstanceEgress/Delete — a reroute scoped to
ip4.src == <vmIP>/32 so peers in a shared subnet are unaffected.
- policy/nat.go: AddSystemInstanceSNAT/Delete — a plain snat (not
dnat_and_snat, so no inbound path), idempotent on its pool external IP.
- external/igw.go: L5 EnsureSystemInstanceEgress/Remove resolves the
gateway nexthop and attaches the reroute + snat (mirrors the NAT-GW
egress path; ADR-0006 S1/S4/S5 preserved).
- subscribers: vpc.add/delete-system-egress topics + handlers.
- handlers/eks: allocate a hidden (ManagedBy=eks) pool address on
CreateCluster and publish add; release + publish delete on teardown.
EIP dep wired through EKSServiceDeps + the daemon.
DNS: the eks-server AMI is Alpine/dhcpcd and /etc/resolv.conf is a
dangling symlink, so DHCP-supplied resolvers are never persisted. Write
a real /etc/resolv.conf from the K3s VM cloud-config via bootcmd (not
write_files — a write_files entry follows the dead symlink, fails, and
aborts the whole block, dropping first-boot.env + the k3s config).
* feat: EKS 6b single-node SQLite pivot + bootstrap-envelope fixes
Switch the K3s server VM from embedded etcd to embedded SQLite (kine)
for the single-node v1, so CreateCluster reaches a healthy control plane.
etcd's per-commit fsync stalls past the apiserver deadline on viperblock's
deferred-durability block path; SQLite tolerates it.
- k3s server: omit cluster-init (selects SQLite/kine), add anonymous-auth=true
so the reconciler's anonymous /healthz NLB probe returns 200.
- first-boot readiness gate: anonymous /healthz (401 under CIS anonymous-auth
=false) -> authenticated 'kubectl get --raw /readyz'.
- first-boot publish: emit a JSON BootstrapEnvelope for all four subjects
(token, admin-kubeconfig, oidc-jwks, ca); jwks and ca were absent before.
- nats pub: --stdin -> --force-stdin (nats CLI 0.4.0).
- manifest: add jq (builds the bootstrap-message JSON envelopes).
* feat: EKS reconciler probes control-plane mgmt IP until DNS lands
The host-side cluster reconciler cannot resolve or route to the VPC-internal
NLB DNS endpoint (no authoritative DNS / Eclipso yet), so CreateCluster timed
out in CREATING and was marked FAILED even though the control plane was healthy.
Probe the control-plane VM's br-mgmt NIC address directly — the one apiserver
endpoint reachable from the daemon. Capture the mgmt IP from the launch result
(SystemInstanceOutput.MgmtIP -> K3sServerOutput.MgmtIP -> ClusterMeta.
ControlPlaneMgmtIP) and build the reconciler health URL as
https://<mgmtIP>:6443/healthz. The probe client already sets InsecureSkipVerify,
so the mgmt IP need not be in the apiserver cert SAN.
Workaround until authoritative DNS + a host-reachable NLB front-end land, at
which point spawnReconciler reverts to the NLB endpoint.
* fix(elbv2): make resource deletes idempotent on absent
DeleteLoadBalancer, DeleteTargetGroup, DeleteListener and DeleteRule
returned a NotFound error when the target was already gone. Real AWS
ELBv2 deletes are idempotent (success on absent), so terraform-provider-aws
fails a destroy step on retry. Return an empty success output instead,
mirroring the KV store which already tolerates absent keys.
Adds TestRLC1_ELBv2DeleteIdempotentOnAbsent enforcing the contract across
the ELBv2 delete surface; converts the three _NotFound delete tests to
assert idempotent success.
Refs mulga-siv-172 (idempotency clause; cascade-composition clause stays open).
* feat: ELBv2 AddTags/RemoveTags, SetSecurityGroups, SetIpAddressType
- O(1) GetListenerByArn via ARN-parse short-ID lookup + ARN equality check
- persist tags on listeners/rules; seed at create time
- AddTags/RemoveTags across LB, target group, listener, rule resources
- SetSecurityGroups (ALB only) reattaches SGs to each ENI via live port update
- SetIpAddressType (ipv4-only, idempotent)
- wire all four ops through gateway, NATS client, and daemon handlers
- frontend: editable Tags tab on LB/TG detail, Security groups tab on LB detail
* feat(elbv2): redirect listener/rule actions
- add redirect action type; parse/store/read-back RedirectActionConfig
- validate status code + render-safe fields (strip AWS #{...} placeholders)
- accept redirect and fixed-response on listener rules (not just defaults)
- HAProxy: http-request redirect scheme (HTTP->HTTPS fast path) or rebuilt
location with placeholder->fetch translation; synthetic per-rule backends
* refactor: dedupe ELBv2 attribute handlers and harden store lookups
Collapse the four near-identical Modify/Describe attribute handlers (target
group and load balancer mirror pairs) onto generic helpers in both the service
impl and the gateway, and table-drive the mirror-pair tests over a TG/LB kind.
- service_impl_attributes.go: generic attrResource/modifyResourceAttributes/
describeResourceAttributes plus sdkAttrsIn/sdkAttrsOut; folds in nil-input
guards, explicit empty-accountID rejection, fresh-string output, and a
no-op skip-persist for unchanged Modify calls.
- gateway/elbv2/attributes.go: generic attributeOp + shared validateAttrArn,
replacing four per-endpoint files.
- types.go: default attribute sets are now package-level maps cloned on access
instead of rebuilt per Describe; document the access_logs empty-string defaults.
- store.go: drop the stale 'linear scan' docstring and log ARN mismatches on
GetLoadBalancerByArn/GetTargetGroupByArn instead of silently returning nil.
Adds cross-account isolation, wrong-account no-mutation, and empty-string-clear
coverage.
* feat: support ip target type in ELBv2 RegisterTargets
RegisterTargets resolved every target ID through an instance ENI lookup, so
ip-type target groups silently no-op'd with an empty PrivateIP and never
received traffic.
- types.go: add TargetTypeInstance/TargetTypeIP constants.
- CreateTargetGroup: default to instance; reject target types other than
instance/ip with InvalidParameterValue.
- RegisterTargets: resolve PrivateIP per target type via resolveRegisteredTargetIP
— ip targets use the supplied IP directly (validated with net.ParseIP),
instance targets keep the ENI lookup and reject IP-shaped IDs.
* fix: cascade rule deletion in ELBv2 DeleteLoadBalancer
DeleteLoadBalancer deleted listeners via store.DeleteListener directly, which
bypasses the rule cascade in the DeleteListener handler. Orphan rules survived
LB teardown and pinned their target groups as ResourceInUse, leaving the TG
permanently undeletable and breaking tofu destroy on multinode.
Extract deleteListenerCascade (rules then listener record) and call it from both
DeleteListener and the DeleteLoadBalancer cleanup loop.
Regression: TestDeleteLoadBalancer_CascadesRuleDeletion.
* fix: harden ELBv2 delete lifecycle against orphaned cascades
- DeleteLoadBalancer: fail-fast on listener-list/cascade error and run the
rule cascade before backing-artifact teardown, so a transient failure
leaves the record intact for retry instead of orphaning rules.
- DeleteTargetGroup: in-use scan skips listeners/rules whose owning load
balancer no longer exists, so a partial-teardown orphan can't pin a target
group as ResourceInUse permanently.
- DeleteRule: propagate the listener lookup error instead of discarding it,
so a failed HAProxy reload is no longer reported as success.
- Add lifecycle invariants for no-orphan completeness, no store-bypass
cascade, and target-group deletability after LB teardown.
- Drop plan/bead references from comments.
* test: add NATS e2e round-trips for new ELBv2 ops
Cover the client→NATS→handler marshalling path that unit tests skip:
tags lifecycle (AddTags/DescribeTags/RemoveTags), target-group and
load-balancer attribute round-trips, redirect-action default listener,
ip target-type registration, and SetSecurityGroups/SetIpAddressType.
* fix(elbv2): correct misleading InvalidConfigurationRequest message
The error code is shared by five unrelated conditions (dualstack
rejection in SetIpAddressType, cross-listener rule moves, unsupported
rule action types, and the two genuine NLB security-group cases), but
its canonical message claimed security groups were unsupported on
network load balancers. Callers hitting it from SetIpAddressType or the
rule paths got a message about security groups and NLBs that did not
match their request.
Neutralize to a generic, condition-agnostic message. The AWS error code
is unchanged.
* style(ui): oxfmt security-groups-editor test
agent.go signed gateway requests with aws-sdk-go v1's v4.Signer. Switch to
predastore/auth.SignReq (a thin wrapper over aws-sdk-go-v2's v4.Signer) so the
monorepo runs a single signer version and drops the last direct dependency on
aws-sdk-go v1's signer subpackage.
- Replace the *v4.Signer field with the raw access/secret keys.
- signedPost computes the hex SHA-256 payload hash and calls auth.SignReq.
- Add TestSignedPost_ProducesVerifiableSignature: the signed request parses and
verifies server-side via auth.ParseReq/Verify with a matching body hash.
Shared-ingress ALBs use a fixed-response default action (HTTP 404) with
host-header rules forwarding to per-app target groups. Two gaps broke this:
- The HAProxy generator read DefaultActions[0].TargetGroupArn unconditionally,
so a fixed-response default (empty ARN) rendered `default_backend bk_`, a
dangling reference that made HAProxy fail to start — the listener never bound
:80 and the LB served no traffic. Store and render the fixed-response config
as a synthetic backend (`http-request return status N ...`), with status,
content-type, and body validated before interpolation.
- ListenerAction dropped the fixed-response config on create/modify, so
DescribeListeners read back an empty action and tofu saw a perpetual diff.
Preserve it through CreateListener/ModifyListener and the SDK conversion.
- DescribeTags returned InvalidParameterValue for listener-rule ARNs; resolve
the rule and return an (untagged) descriptor instead.
Adds generator tests for the fixed-response default (with host-header rules)
and unsafe-body rejection, plus DescribeTags listener-rule coverage.
sudo mknod created real /dev/console and /dev/null char-device nodes on the
host under $CHROOT_DIR. Real device nodes on the host filesystem are a footgun:
a stale-image cleanup, tmp reaper, or an empty $CHROOT_DIR (which would redirect
the recursive /dev removal at the host root) could clobber host /dev/null and
force a reboot, breaking CI servers.
Write the two static nodes the kernel needs into the cpio archive inside a
fakeroot session instead. mknod/cpio see faked device entries and emit real
char-device records into the archive, while the host filesystem never gets an
actual node. init mounts devtmpfs over /dev at runtime for everything else.
Also guard $CHROOT_DIR against empty/unexpected values before any destructive
removal, and add fakeroot to the required-tools check.
After PR #270 the VPC-LR-wide 0.0.0.0/0 -> IGW static route is no longer
installed implicitly at attach-internet-gateway. Subnets whose effective
route table lacks an internet-bound route get an OVN drop policy at
SubnetEgressPriorityDrop (1100), which kills LB target reply traffic
before un-DNAT can fire and stalls the LB internal-client probe until
job timeout.
Add the standard AWS pattern: describe-route-tables -> create-route
0.0.0.0/0 -> IGW. This fires vpc.add-igw-route + vpc.ungate-subnet-egress,
removing the drop policy and installing a reroute policy at
SubnetEgressPriorityIGW (1000).
The W2 inject-garp fix landed in 72a6a59d adds a shellout to
`ovn-appctl -t ovn-controller inject-garp <port>` via the L0
execRunner, which prefixes `sudo` when not running as root.
spinifex-vpcd runs as user spinifex-vpcd, so every emission was
hitting `sudo: a password is required` and silently returning a
WARN that nobody read until W5 diag capture surfaced the journal.
Net: W2 has been a complete no-op since it shipped. Every
recycled-EIP scenario has continued to leave stale upstream ARP.
Evidence: diag bundle from run 26813235295 — 12/12 GARP attempts
denied by sudo, 0 successful emissions; pcap shows runner ARP for
recycled EIP .232 stuck on the prior VPC's gw chassisredirect MAC
(02:75:d9:ef:d8:4b) while OVN NB has handed .232 to a new VPC
with a different gw MAC; pcap also shows runner sending 174 SYNs
to recycled EIP .233 with 0 SYN+ACKs received.
Fix: add /usr/bin/ovn-appctl to the spinifex-vpcd NOPASSWD grant.
Trivial; ovn-appctl is already trusted via the existing NB/SB
read-write grants.
Follow-ups (separate change): tighten host.InjectGARP error
classification so sudo-deny is loud (Error or fatal) rather than
silently warn-and-continue; add a boot-time self-check that
verifies sudo permissions for every host-tool the daemon will
shell out to.
Two bugs in afd63ed9:
1. `sudo mkdir` makes /tmp/spinifex-diag root-owned 0755. The
subsequent `> /tmp/spinifex-diag/tcpdump.log` is performed by
the calling tf-user shell BEFORE sudo gets the command, failing
with Permission denied. action shell runs with `set -e` so the
step aborts.
2. `sudo nohup ... &; echo $!` captures the wrong PID — `$!` is the
backgrounded sudo, which may exit when its target dies, leaving
stale tcpdump.pid.
Fixes:
- chmod 0777 on /tmp/spinifex-diag after creating it (next steps'
redirects from tf-user shell now work).
- Wrap the tcpdump launch in `sudo bash -c '...'` so backgrounding
and `$!` happen inside the sudo'd shell — `$!` is the actual
tcpdump PID (nohup execs, sudo bash holds it). Captures stdin
to /dev/null, stdout/stderr to /tmp/spinifex-diag/tcpdump.log.
- sleep 2 (was 1) — gives tcpdump time to fully init pcap header.
- Hard fail (exit 1) if PID is missing or process dead, so the
step's failure is visible instead of a silent green.
bash -n verified.
Mulga-siv-167 Phase 1. Adds evidence-capture for the LB
single-host return-path drop class:
e2e_single only:
- New step 'Start Packet Capture' between Bootstrap and Run E2E
Tests. SSHes to test VM and starts background tcpdump on br-wan
capturing TCP/22/80/443/9999 + ARP into /tmp/spinifex-diag/br-wan.pcap.
PID stored at /tmp/spinifex-diag/tcpdump.pid.
All three jobs (single, pseudo-multi, real-multi):
- Collect Diagnostics: kill tcpdump (single only — PID file
guarded) -INT for graceful flush, falls through to -KILL after 5s.
- Adds journal-spinifex-full.log (no -n truncation) alongside
the existing -n 10000 capped one.
- Adds bridge fdb show, bridge link, and hairpin_mode/learning/flood
for every br-wan brif — needed to confirm/disconfirm the
hairpin-loss hypothesis in the plan doc.
- Adds 'chmod -R a+r /tmp/spinifex-diag' before tar to ensure pcap
(created by sudo tcpdump as root) is readable by tf-user.
bash -n verified on all four rendered scripts. YAML parses.
Zero new gold-image deps (tcpdump, bridge present in base image).
Adds a Collect Diagnostics + Upload Diagnostics step pair to all three
E2E jobs (single, pseudo-multi, real-multi). Runs before Teardown,
if: always() so green runs produce baselines too. SSHes to test
VM(s), dumps:
- journalctl for spinifex-*, ovn-*, openvswitch-*
- ovs-vsctl/ofctl/appctl: bridge state, br-int + br-ex flows,
dpctl flow + conntrack tables
- ovn-nbctl/sbctl: show, Port_Binding, Chassis, NAT, Logical_Router
- ovn-appctl ct-zone-list (LR -> ct_zone mapping)
- ip addr/route/neigh, iptables-save, nft list, dmesg, virsh list
Tarred, scp'd back, uploaded as e2e-diag-{single,pseudo,multi}-<run>
artifact (14-day retention).
OVS-native ovs-appctl dpctl/dump-conntrack — no new gold-image deps.
Multi-node job writes script to RUNNER_TEMP then ssh-stdin's it
per-node (avoids YAML/heredoc indent traps inside for-loop). bash -n
verified on all three rendered scripts.
Unblocks siv-111 debug loop: bash LB driver does not emit an
analysis bundle on failure, so post-W2 recurrences have been blind.
Workflow-level capture is hypothesis-free.
ovn-controller emits gratuitous ARPs automatically when a NAT-bound LSP
migrates chassis, but not when an external IP is reassigned to a different
LR whose chassisredirect MAC differs from the prior occupant's. In CI
single-host runs the WAN pool slice is small enough that the same EIP
recycles across VPCs within the 60-300s kernel ARP timeout, so upstream
caches keep pointing at the dead chassis-redirect MAC and packets to the
new VM never arrive.
Fix: add a post-AddEIP hook that runs `ovn-appctl -t ovn-controller
inject-garp <port>` (OVN 22.09+). Port is the EIP's LSP in distributed
NAT or "cr-gw-<vpcID>" in centralized NAT. Best-effort — failure logs a
warning and the NAT install still succeeds.
- network/host/garp.go: new L0 shell-out helper InjectGARP.
- network/policy/nat.go: GARPEmitter option type, called after the
flows-ready barrier in AddEIP.
- vpcd: wire the production emitter that derives the right OVN port
from the EIPSpec.
- invariants: add ovn-appctl to the host-tool ban list so future
callers are forced into network/host/.
* feat(elbv2): ListenerRules CRUD with HAProxy ACL routing
Implement CreateRule, ModifyRule, DeleteRule, DescribeRules, and
SetRulePriorities for ALB listeners. Rules support host-header,
path-pattern, http-header, http-request-method, source-ip, and
query-string condition fields. Actions limited to forward (single
target group); redirect and fixed-response deferred.
HAProxy frontends emit per-rule ACLs with `use_backend ... if <acls>`
ordered by priority. Wildcards in host/path map to HAProxy match
modes (`-m beg/end/sub/str`). Condition values are validated at
store-time and re-validated at render-time to prevent config
injection. Target groups referenced by rule actions are protected
against deletion. Deleting a listener cascades its rules.
Frontend adds a Rules tab beneath each listener row showing rules
with conditions and target groups, plus add/edit/delete dialogs
with field-typed condition inputs.
mulga-950
* test(elbv2): rule CRUD e2e + render coverage for all condition fields
Adds Internal_ALB_ListenerRules subtest exercising CreateRule, ModifyRule,
DeleteRule, and DescribeRules over a real ALB with HAProxy. Two TGs with
disjoint backends (app#0 -> tgA, app#1 -> tgB) let the probe responder
identify which TG actually served the request, so path-pattern routing
can be asserted without inspecting internal state.
Extends app-userdata.sh with alpha/ and beta/ sub-paths and extends
client-userdata.sh trigger to accept optional http_path and host fields
so probes can target specific paths and Host headers. Both changes are
backward-compatible: existing callers default to / with no Host override.
Adds unit-test coverage for every condition-field render path
(http-header, http-request-method, query-string with and without key,
source-ip) plus error-path validation tests for CreateRule, ModifyRule,
DeleteRule, DescribeRules, and SetRulePriorities. Lifts diff coverage
above the 60% gate.
* test(elbv2): wait for tgB healthy after rule creation, not before
Spinifex only health-checks a target group once it is referenced by a
listener default action or a rule action. The previous ordering waited
for tgB healthy before CreateRule wired it, so health checks never ran
and the wait timed out. Move the WaitForTargetsHealthy(tgB) call after
createPathRule.
* fix(elbv2): include rule-action TGs in TargetGroupsForLB
healthChecker uses Store.TargetGroupsForLB to map HAProxy server
statuses back to TargetGroupRecord entries. The previous implementation
walked only listener DefaultActions, so a target group referenced only
by a CreateRule action was invisible to the health checker — HAProxy
probed its backends but the resulting health never propagated to the
store. DescribeTargetHealth reported the targets as 'initial' forever.
Extend TargetGroupsForLB to walk rule actions per listener as well.
Adds a unit test that wires a TG via CreateRule and asserts the
resolver returns it.
* test(elbv2): fix wait-for-rule logic to poll away from default backend
waitForPathRouted required the caller to name the expected new host, but
at the post-CreateRule wait point we only know the default host (appA);
the rule target hostname (appB) is discovered by the subsequent probe.
The previous code passed appA as both "new" and "old", so the wait
checked "see appA" against "/alpha/" responses — which never matched
because the rule had successfully routed appA out.
Replace with two narrower helpers: waitForPathRoutedAway (poll until
host X stops appearing) and waitForPathRoutedTo (poll until host Y
appears). Update the three call sites to use the variant that matches
what the caller actually knows at that point.
* fix(elbv2): IAM-style error envelope + ModifyTargetGroup + listener attr stubs
ELBv2 success responses use the IAM-style XML envelope (<ErrorResponse>),
but error responses were emitted with the EC2 envelope (<Response>),
which aws-sdk-go-v2 cannot deserialize for ELBv2 and surfaces as
"UnknownError: UnknownError" — defeating provider-side InvalidAction
ignores (e.g. terraform-provider-aws v5.100's DescribeCapacityReservation
swallow). Route elasticloadbalancing errors through GenerateIAMErrorResponse.
Add ModifyTargetGroup end-to-end (interface, impl, NATS shim, daemon
handler, gateway dispatch) so terraform-provider-aws can set health-check
fields after CreateTargetGroup without infinite drift. Persist
HealthCheckEnabled on TargetGroupRecord and emit it in tgRecordToSDK so
the provider sees the configured value across refresh cycles.
Stub DescribeListenerAttributes / ModifyListenerAttributes with
hand-rolled input/output structs (aws-sdk-go v1.55 lacks these types).
Returns empty attribute list — sufficient for default-config workflows.
* test(elbv2): cover ModifyTargetGroup + listener attribute stubs
Extend handler tests with all-fields, invalid path/matcher, wrong-account
ModifyTargetGroup cases plus HealthCheckEnabled assertion on
CreateTargetGroup. Add gateway-layer validation tests for the new
DescribeListenerAttributes, ModifyListenerAttributes, and ModifyTargetGroup
dispatchers. Raises diff coverage from 58.8% to 63.7%.
Lifts diff-coverage on PR #276 from 37.5% to 63.5% (threshold 60%) so
CI's Check Diff Coverage step passes.
- utils: LoadViperblockMasterKey — empty path, happy path + cache reuse,
load error, and bad-perm failure-not-cached.
- handlers/ec2/image: clusterEncryptionEnabled — nil config, empty key
file, masterkey load error (warn + false), and key-loaded (true).
- handlers/ec2/volume:
- GetVolumeConfig surfaces "failed to unmarshal config" when the body
is neither a valid VBState nor a wrapper.
- mergeVolumeConfig refuses to re-marshal an encrypted VBState (would
strip the AES-GCM tag and brick the volume).
- CreateVolume returns ServerInternal when EncryptionKeyFile is set
but unreadable.
VolumeMetadata.IsEncrypted is being removed from viperblock — it was
hardcoded false everywhere in spinifex and misrepresented the per-volume
encryption status to AWS SDK callers. The authoritative flag is now
VBState.EncryptionEnabled, persisted by viperblock at New() when
MasterKey != nil.
- volume CreateVolume drops the dead IsEncrypted struct member and
surfaces Encrypted=(mkey != nil) on the EC2 response.
- volume getVolumeByID now decodes the full VBState (falling back to a
wrapper-only blob for legacy pre-encryption volumes) so it can read
state.EncryptionEnabled directly.
- snapshot CreateSnapshot pulls Encrypted from the source volume's
VBState.EncryptionEnabled (already loaded one level up).
- image synthesizeRootBlockDeviceMapping takes the cluster encryption
posture (mkey != nil) since AMI metadata carries no per-image flag;
cached masterkey load keeps the hot path cheap.
- Test fixtures drop the now-defunct IsEncrypted lines; the volume
fixture that asserts Encrypted=true switches to a full VBState seed
with EncryptionEnabled=true.
LoadStateRequest can fail for many reasons — HMAC integrity, key
mismatch, transient backend 5xx, JSON parse failure. The previous
fall-through treated every error as "volume doesn't exist, clone from
AMI", which would overwrite the legitimate encrypted volume's
config.json with AMI base-map state on a tamper or transient backend
fault. Discriminate os.ErrNotExist explicitly and abort otherwise.
Adds EncryptionKeyFile to ViperblockConfig (cluster config) and
viperblockd.Config. viperblockd loads the master key once at
launchService start via masterkey.LoadShared and passes
MasterKey + EncryptionEnabled into the VB constructed for every
mounted volume.
AWS handlers (volume.CreateVolume, instance.newViperblock,
image.snapshotStoppedVolume) and the spx admin AMI-import path now
resolve the key via utils.LoadViperblockMasterKey — a process-wide
cache keyed by path so the per-VB lookup is one map hit after the
first load.
volume.mergeVolumeConfig: switch from json.Unmarshal to json.Decoder
so a trailing AES-GCM tag cannot route the body to the wrapper-only
path, and refuse outright when state.EncryptionEnabled — re-marshaling
an encrypted VBState here would strip the tag and brick the volume.
cmd/spinifex viperblock CLI grows a --encryption-key-file flag (env
SPINIFEX_VIPERBLOCK_ENCRYPTION_KEY_FILE) under a distinct viper key
so it does not collide with predastore's identically-named flag.
* feat(eks): eks-server + eks-agent AMI manifests + builder INSTALL_FILES
Manifest-driven AMI build for the two EKS image variants (eks-v1.md Q12).
Lifted forward from cs-eks-6b so the K3s boot path can be smoke-tested in
isolation before CreateCluster orchestration lands.
Builder:
- build-system-image.sh gains optional INSTALL_FILES (src:dst pairs,
mode 0644, parent dirs auto-created). Symmetric with INSTALL_BINARIES.
- ENABLE_SERVICES moves after INSTALL_FILES + SETUP_SCRIPT so init scripts
the manifest references are on disk before rc-update enables them.
eks-server image (Alpine 3.21.7, q35, 4G):
- K3s v1.32.5+k3s1 (downloaded at bake time, SHA256 verified against the
upstream sha256sum-amd64.txt trust anchor).
- nats-cli v0.4.0 (not in Alpine main/community — pulled from upstream
release zip; unzip installed + purged in same setup step).
- OpenRC init scripts: k3s, eks-token-webhook, k3s-first-boot.
- Cron periodics under /etc/periodic/{15min,daily}/ for state-report +
etcd-snapshot.
- k3s-first-boot publishes node-token + admin kubeconfig over NATS on
first successful K3s healthz, then self-disables (sentinel + rc-update
del).
- config.yaml.skel without webhook-token-auth — wiring it before the real
webhook lands (cs-eks-6c) would block anonymous probes against /healthz.
Orchestrator drops the live config + restarts k3s.
eks-agent image (Alpine 3.21.7, q35, 2G):
- Same pinned K3s release for server/agent lock-step.
- k3s-agent OpenRC service refuses to start without K3S_URL+K3S_TOKEN —
fail-closed pre-orchestration.
eks-token-webhook stub (cmd/eks-token-webhook/main.go):
- 503 + Retry-After:30 on every path. Statically linked. Lets the AMI
build succeed and kube-apiserver boot cleanly. Replaced wholesale by
cs-eks-6c.
Init system divergence from spec wording: Alpine ships OpenRC, not
systemd. Unit files use .initd extensions and cron periodics instead of
.service / .timer. Bead-level intent (auto-enable on boot, periodic
execution) preserved.
Smoke-tested:
- eks-server: builds 300M qcow2; qemu q35 boot reaches K3s apiserver on
6443 within ~120s; HTTPS probes return valid k8s Status JSON (the 401
is K3s v1.32 default RBAC behaviour against anonymous, not a regression).
- eks-agent: builds 280M qcow2; qemu boot mounts root cleanly.
* feat(eks): make targets for eks-server + eks-agent AMI builds
- build-system-image gains subdir-manifest fallback: tries
scripts/images/$(IMAGE).conf first, then scripts/images/$(IMAGE)/manifest.conf
so the new EKS layout works alongside the existing flat lb manifest.
- build-eks-server-image / build-eks-agent-image — explicit convenience
targets that delegate to the generic one.
* feat(sts): EKS Q4 STS extensions — interface + JWKS reader + stubs
Lays the structural slice of AssumeRoleWithWebIdentity + presigned-URL
GetCallerIdentity verify for IRSA. The verify-path bodies (JWT signature
check against fetched JWKS, claims validation, Federated-principal trust
policy evaluation, SigV4 presigned-URL canonical-request reconstruction)
land in follow-up commits on this branch.
Adds:
- handlers/sts/oidc_jwks.go — JWK/JWKS types, ParseEKSIssuerURL (defensive
iss parsing), FetchClusterJWKS (KV-backed read from
eks-account-{accountID}/clusters/{name}/oidc-jwks.json via the EKS store
helpers).
- handlers/sts/oidc_jwks_test.go — unit tests over a real JetStream
test fixture.
- handlers/sts/assume_role_with_web_identity.go — interface entry, stub
returns ErrorNotImplemented.
- handlers/sts/presigned_url_verify.go — PresignedCallerIdentity type +
VerifyPresignedGetCallerIdentity stub.
- gateway/sts/AssumeRoleWithWebIdentity.go — dispatch shell.
Modifies:
- handlers/sts/service.go — STSService interface gains
AssumeRoleWithWebIdentity + VerifyPresignedGetCallerIdentity.
- gateway/sts.go — registers AssumeRoleWithWebIdentity in stsActions +
stsSkipPolicyCheck (anonymous; trust-policy-gated in handler).
- gateway/sts_request_test.go, gateway/sts/handler_test.go,
gateway/auth_test.go — stub implementations updated to satisfy
extended STSService interface.
Anonymous routing for AssumeRoleWithWebIdentity (eks-v1.md Q5 mux rework)
lands in Sprint 6e — for now the action is registered on the standard STS
gateway and rejects non-SigV4 requests upstream.
Refs: cs-eks-sts-q4 (mulga monorepo bead)
* feat(sts): AssumeRoleWithWebIdentity + presigned GetCallerIdentity verify
Implements EKS Q4 STS extensions for IRSA + EKS token-webhook flows.
AssumeRoleWithWebIdentity:
- ES256 JWT verification against per-cluster JWKS (eks-account-{accountID}
KV bucket), keyed by issuer host-path.
- Federated trust-policy evaluator (explicit-deny-wins, exact-match
Principal.Federated, fail-closed on Principal: "*").
- Condition support scoped to StringEquals on {iss}:sub (exact) and
{iss}:aud (set-membership). All other operators rejected at validator.
- OIDC provider registry check against iam-account-{roleAccountID}; v1
does not support cross-account registration.
- New awserrors: InvalidIdentityToken, IDPRejectedClaim,
IDPCommunicationError.
VerifyPresignedGetCallerIdentity:
- Hand-rolled SigV4 canonical-request reconstruction (no sdk dep).
- Mandatory X-K8s-Aws-Id in SignedHeaders + value pinned to
expectedClusterName during canonicalisation — cross-cluster replay
surfaces as a signature mismatch.
- AKID-prefix dispatch (ASIA → session credential, AKIA → IAM user).
- Strict envelope checks: algorithm, credential scope service/terminator,
X-Amz-Date / credential-scope date agreement, expiry within
X-Amz-Expires, 5-minute wire skew.
Tests:
- 14 web-identity scenarios (happy paths, tampering, expiry, wrong aud,
unknown kid, HS256 rejection, federated-wildcard fail-closed,
explicit-deny-wins, provider-not-registered).
- 12 presigned-URL scenarios including cross-cluster replay defence,
missing-x-k8s-aws-id, tampered query, bad service in credential, AKIA
+ ASIA happy paths.
- Cross-cluster anti-replay invariant test (rejects matrix of variant
cluster names — case-different, prefix, suffix).
- AWS SigV4 known-vector test for deriveSigningKey.
Gateway:
- TODO documents that AssumeRoleWithWebIdentity is currently dispatched
on the SigV4-gated path; a pre-auth route is needed so a projected
ServiceAccount token alone can authenticate the caller.
IAM:
- Relaxes trust-policy Condition validator: now permitted only when
Action is exactly [sts:AssumeRoleWithWebIdentity] and operator is
StringEquals. Preserves loud failure for sts:AssumeRole.
* feat(gpu): add NVIDIA MIG multi-instance GPU support
Partitions MIG-capable GPUs (A100, A30, H100) into isolated slices exposed
as mdev devices instead of whole-GPU VFIO passthrough. VMs attach via
-device vfio-pci,sysfsdev=<mdev-path> using the GPUAttachment discriminated
union (PCIAddress for whole-GPU, MdevPath for MIG).
Key changes:
- gpu: new MIGProfile/MIGInstance types, IsMIGCapable model map, mig.go ops
(EnableMIGMode, DisableMIGMode, CreateInstances, ListInstances,
DestroyAllInstances, enrichMdevPaths)
- gpu.Manager: Claim() returns (*GPUDevice, *MIGInstance, error); AddMIGInstances,
ReclaimByMdev for restart recovery; MarkMIGFailed per-slice
- daemon: buildGPUPool helper partitions devices into whole-GPU vs MIG pools;
MIGProfile config field selects profile per node or per GPU override
- vm.VM: GPUAttachments []gpu.GPUAttachment replaces GPUPCIAddresses []string
- admin CLI: spx admin gpu mig status/enable/disable subcommands with
admin.SetMIGProfile for spinifex.toml writes
- docs: COMMANDS.md updated with three new MIG management command rows
* test(gpu): unit tests for MIG parsers and pool manager
Cover all pure parsing functions (parseMIGMemory, isCapacityExhaustedError,
extractProfileFields/GIFields, parseMIGProfiles, parseMIGInstances,
parseCreatedGIID/CIID) with realistic nvidia-smi fixture output.
Manager MIG tests cover AddMIGInstances, Claim/Release on mdev-backed entries,
MarkMIGFailed, ReclaimByMdev, and interaction with whole-GPU MarkFailed — all
using fake mdev directories in t.TempDir() so no GPU hardware is required.
* feat(gpu): dynamic MIG profile-per-request with on-demand GI/CI lifecycle
Replaces static config-driven MIG pre-carving with a dynamic model where
the MIG profile is specified per RunInstances call via mig.<profile>
instance type names (e.g. mig.1g.10gb) that mirror nvidia-smi nomenclature
directly.
- Manager.Claim(instanceID, profileName): fast path claims a pre-carved
slice; slow path dequeues a free MIG GPU, releases the lock, calls
nvidia-smi mig -cgi/-cci to carve all slices, then reclaims the lock to
commit and return the first slice
- Manager.Release: last release on a dynamically-carved GPU calls
DestroyAllInstances and returns the GPU to freeMIGGPUs, allowing a
different profile on the next claim
- AddMIGGPU(dev): registers a fresh MIG GPU for dynamic allocation;
AddMIGInstances (restart recovery) keeps its no-destroy semantics
- GenerateMIGTypes: produces mig.<profile> InstanceTypeInfo entries from
gpu.ListProfiles at daemon startup; IsMIGType/MIGProfileFromType let
callers inspect these names
- buildGPUPool: calls ListInstances for recovery, AddMIGGPU for fresh
GPUs, ListProfiles to collect MIG profile specs for type generation;
removes setupMIGInstances and migProfileForDevice
- GPUClaimer.Claim gains profileName param; daemonGPUClaimer, service_impl
call sites, and fakeGPUClaimer all updated
* fix(gpu): correct mdev parent detection and make --profile optional for mig enable
enrichMdevPaths was using os.Readlink on "parent_bus_0" (a non-existent
sysfs attribute) and then on ".." (a directory, not a symlink) — both
would fail on real NVIDIA hardware, breaking CreateInstances and
ListInstances on daemon restart recovery. Fix: read the mdev UUID's own
symlink target, which always encodes the parent GPU's PCI address in its
path (e.g. .../0000:01:00.0/<uuid>).
Make basePath injectable in enrichMdevPaths so the fix is covered by a
new set of unit tests using temp-dir symlinks (TestEnrichMdevPaths_*).
Remove the MarkFlagRequired("profile") from "spx admin gpu mig enable":
Phase 2 is fully dynamic — the daemon carves slices on demand at
RunInstances time. Without --profile, the command just enables MIG mode
and destroys any pre-carved slices so the daemon starts with a free GPU.
With --profile, behaviour is unchanged (pre-carve a static pool).
* chore: writeups for smci and arcee
* test: add coverage for MIG exec, manager slow-path, instancetypes and admin
- Add mig_exec_test.go with fake nvidia-smi harness covering Enable/Disable MIGMode,
IsMIGModeEnabled, DestroyAllInstances, ListProfiles, CreateInstances, ListInstances
- Extend manager_mig_test.go with slow-path Claim/Release tests using fake nvidia-smi
- Add generate_test.go coverage for IsMIGType, MIGProfileFromType, GenerateMIGTypes
- Add admin_test.go coverage for SetMIGProfile (create, update, idempotent, error paths)
- Make mdevBasePath a var to allow sysfs injection in tests
- Add json tags to GPUAttachment fields to satisfy musttag linter
Diff coverage: 75.6% (threshold: 60%).
* fixed gpu passthrough doc
* Revert "fixed gpu passthrough doc"
This reverts commit 1e20f5ab13.
* feat: Added RTX 6000 GPU MIG support and fixed docs
* feat: expose GPU/MIG inventory in GetNodes and GetVMs admin API
Add Manager.Snapshot() for lock-safe pool inspection. Extend
NodeStatusResponse with GPUs []GPUInfo (per-physical-GPU inventory
with per-slice detail for MIG) and VMInfo with GPU *VMGPUInfo
(model, VRAM, profile, mdev path for the attached GPU). Populate
both in handleNodeStatus and handleNodeVMs via buildGPUInventory,
buildPoolLookup, and resolveVMGPU helpers.
* feat: surface GPU info on EC2 instance detail page (mulga-new-03)
Admin users see a GPU card (model, VRAM, attachment type, MIG profile/mdev
path or PCI address) populated from the GetVMs admin API. Adds formatVRAMMiB
helper and VMGPUInfo/GPUInfo/GPUSliceInfo types to the frontend.
* feat: GPU columns + inventory card on Nodes page (mulga-new-04)
Adds GPU column to VMsTable (model/VRAM or MIG profile) and ResourceTable
(alloc/total GPUs). Section 4 GPU Inventory card shows per-node GPU state with
per-slice MIG drill-down. Also adds total_gpus/alloc_gpus to NodeInfo type.
* feat: grouped GPU-aware instance type picker in Run Instances (mulga-new-05)
Splits the flat instance type dropdown into Standard and GPU-Accelerated groups.
GPU items show total VRAM alongside the type name. Selecting a GPU type surfaces
an info line with GPU name and VRAM.
* testing image builder scripts
* fixed nvidia build script
* more descriptive errors for image build script
* more fixes
* more fixes
* fix: surface PCI address in whole-GPU VMGPUInfo and rebuild frontend dist
Add PCIAddress to VMGPUInfo, populate it in resolveVMGPU for whole-GPU
passthrough, and fix the instance detail page which was incorrectly
showing instance type in the PCI address field. Rebuilt frontend dist
to include GPU UI changes (inventory card, resource columns, GPU-aware
instance type picker).
* fix: improve kernel version detection in nvidia image build script
Fall back to /lib/modules when /boot/vmlinuz-* is absent, and add
diagnostic output to help debug kernel detection failures.
* fixing nvidia image build script
* fix ami name issue
* correct kernel versions
* another fix
* image build works, added docker and nvidia toolkit for QoL
* fix: add gpg
* fixing amd image builder
* add universe
* another fix
* nvm
* fix docker user group issues
* make rocm land where it is supposed to
* fix docker groups
* feat: add Ubuntu 26.04 GPU AMI catalog entries and fix build script parted detection
Add ubuntu-26.04-nvidia-gpu-x86_64 and ubuntu-26.04-amd-gpu-x86_64 to AvailableImages,
hosted on iso.mulgadc.com/system-ami/ with SHA256 checksums. Fix build-system-image.sh
to detect parted in /usr/sbin when not on PATH. Suppress gosec false positives in DHCP
manager (cancel stored in leaseLoop) and promisc ioctl (standard syscall uintptr pattern).
* Updated passthrough doc regarding GPU images
* fix: remove orphaned nolint directives (golangci-lint v2.12.2 no longer flags G115/G118)
* chore: remove stale spinifex@192.168.3.107 directory
Scripts were accidentally committed with a node hostname as the path
prefix during initial testing. Correct copies live in scripts/images/.
* feat(eks): gateway + handler scaffolding for EKS control plane
Lands the EKS service contract — gateway REST-JSON dispatcher, handler
interface + stub impl, JetStream KV bucket layout, and daemon wiring —
ahead of the K3s control-plane bodies.
Every action currently returns NotImplemented (HTTP 501 + JSON
NotImplementedException) so downstream work can call into a stable
contract while real bodies land sprint by sprint.
- Gateway: 34-action REST-JSON routing table under svc=eks, per-action
closures dispatch to handlers_eks.NATSEKSService. ErrorHandler /
writeClusterUnavailable / writeThrottleError gain EKS JSON branches so
clients see application/x-amz-json-1.1 envelopes.
- Handler: EKSService interface + NATS wrapper + stub impl with
var _ EKSService compile checks on both implementations.
- KV: eks-account-{accountID} lazy per-account bucket factory plus the
shared spinifex-eks-leader bucket (TTL 60s, History 1) bootstrapped
at daemon start. Key-path helpers cover the full Q2 layout
(cluster meta, nodegroups, access entries, OIDC providers + signing
key + JWKS, admin kubeconfig, K3s node-token, events).
- Daemon: 34 eks.<Action> subscriptions on the spinifex-workers queue
group, init via initServiceWithRetry, mirrors the ELBv2 layout.
- awserrors: new ErrorNotImplemented constant (HTTP 501).
- tags: ManagedByEKS constant for downstream SG/NLB/VM tagging.
* test(eks): cover scaffolding wrappers + daemon NATS handlers
Adds two test files closing the diff-coverage gap on the EKS
scaffolding commit:
- spinifex/gateway/eks/wrappers_test.go drives every per-action
gateway wrapper (cluster, nodegroup, access, addons, oidc, tags —
34 actions, 50+ subtest invocations) through an embedded NATS
server with a `eks.>` catch-all stub responder. Covers the
wrappers themselves plus handlers/eks/service_nats.go via the
NewNATSEKSService construction inside each wrapper. Bad-JSON
branches in unmarshalIfBody are also covered.
- spinifex/daemon/daemon_handlers_eks_test.go constructs a minimal
Daemon{eksService: real impl} and dispatches each of the 34
handleEKS* handlers via embedded NATS, asserting the NotImplemented
error code rides back through handleNATSRequest.
Lifts diff coverage from 27.3% to 62.6%, above the 60% threshold.
* fix(network): IGW default route AWS-parity via per-subnet LR policy
ADR-0006's IGWManager.AttachIGW installed a VPC-router-wide 0.0.0.0/0
static route at attach time, violating AWS semantics: every subnet on
the LR saw egress to the IGW regardless of its route table. Private
subnets that should be IGW-isolated were not.
Move IGW egress lifecycle onto the route-table API. CreateRoute(IGW,
0.0.0.0/0) emits vpc.add-igw-route per associated subnet; the network
subscriber installs an OVN Logical_Router_Policy matching
`inport == "rtr-<subnetID>" && ip4.dst == 0.0.0.0/0` with action
`reroute` via the IGW gateway port. DeleteRoute / DisassociateRouteTable
/ ReplaceRouteTableAssociation tear it down. AssociateRouteTable emits
add events so the terraform ordering (CreateRoute -> Associate) is
honored.
Additions:
- nbdb.LogicalRouterPolicy + Logical_Router_Policy registration
- ovn.Client {Add,Delete,Find,List}LogicalRouterPolicy + live + mock
- policy.RouteManager.{Add,Delete}SubnetEgress with priority constants
- external.IGWManager.{Ensure,Remove}SubnetEgress resolving nexthop +
output port from the gateway state
- subscribers handleAddIGWRoute / handleDeleteIGWRoute on
vpc.{add,delete}-igw-route
- routetable publishIGWRouteEvents / *ForAssociation / *ForSubnet
Test updates:
- network/policy: AddSubnetEgress idempotency, drift-replace, per-subnet
separation, DeleteSubnetEgress
- network/external: AttachIGW must NOT install a router-wide default
route; EnsureSubnetEgress installs the per-subnet policy with the
resolved nexthop (link-local fallback + allocator override paths)
- handlers/ec2/routetable: CreateRoute(IGW), Associate, DeleteRoute,
Disassociate publish events with the expected subnetID + igwID
- ovn/nbdb: table-count + Logical_Router_Policy presence
- subscribers: TestSubscribe_RegistersAllTopics covers the two new topics
* fix(network): NATGW per-subnet egress + DeleteRoute event symmetry
handlers/ec2/routetable:
- DeleteRoute now publishes vpc.delete-nat-gateway per associated subnet
(mirror of the IGW DeleteRoute path) so the subscriber tears down both
the SNAT rule and the per-subnet egress policy.
- publishNatGatewayEvent* signatures carry SubnetId + DestinationCidr so
the subscriber can install the matching Logical_Router_Policy.
network/external:
- IGWManager: EnsureNATGatewaySubnetEgress / RemoveNATGatewaySubnetEgress
install/remove a per-subnet LR policy at priority SubnetEgressPriorityNATGW.
Same nexthop and output port as the IGW priority sibling; lower priority
so an IGW route wins when both apply to the same subnet.
- EnsureSubnetEgress delegates to ensureSubnetEgressAtPriority for shared
validation, lookup, and slog tagging.
network/subscribers:
- handleAddNATGateway: after SNAT install, parse DestinationCidr (defaults
to 0.0.0.0/0) and call EnsureNATGatewaySubnetEgress. Without this the LR
has no route to egress NAT'd packets and private-subnet traffic is
dropped (E2E Phase 8d regression).
- handleDeleteNATGateway: remove the per-subnet policy FIRST, then detach
SNAT, so packets are never routed unmasqueraded after SNAT removal.
- NATGatewayEvent gains SubnetId + DestinationCidr.
Tests:
- policy.AddSubnetEgress validation + drift-replace + propagation paths.
- mock LR-policy methods reject missing routers and tolerate dangling UUIDs.
- subscribers cover NATGW add/delete with invalid CIDR, bad JSON, missing
router, IGW-route bad JSON + invalid CIDR. Full end-to-end add+delete
via NATS asserting policy lifecycle.
- IGW manager covers RemoveSubnetEgress / RemoveNATGatewaySubnetEgress
empty-arg rejection + delete error propagation.
- routetable: DeleteRoute(NATGW) publishes per associated subnet.
* fix(network): exclude VPC CIDR from per-subnet egress reroute predicate
In-VPC traffic (e.g. bastion -> private SSH) was being intercepted by the
NATGW per-subnet egress LR policy on `ip4.dst == 0.0.0.0/0` and forwarded
out the gateway with SNAT, breaking peer-subnet connectivity. Append
`&& ip4.dst != <vpcCIDR>` to the match so in-VPC destinations skip the
reroute and follow the directly-connected route. Apply to both IGW (1000)
and NATGW (900) priorities. CIDR is read from the LR ExternalIDs label
populated at VPC create.
DeleteSubnetEgress takes the same excludes so the match string is
identical on cleanup. igwManager.vpcExcludeCIDRs fetches from the router
each call; lookup failure returns nil and the policy installs without
exclusions (preserves prior behaviour as fallback).
* fix(routetable): publish per-subnet events for implicit main-RT membership
E2E run 26620082197 RCA: subnets with no explicit RT association inherit
the VPC main route table per AWS semantics, but publishIGWRouteEvents and
publishNatGatewayEvents iterated only record.Associations and skipped
Main:true/SubnetId:"" sentinels. Main RT carries exactly one such
sentinel, so CreateRoute(igw)/CreateRoute(natgw) on the main RT fanned
out to ZERO subnets — subscriber never called EnsureSubnetEgress, OVN
LR policy never installed, ping 8.8.8.8 from default-VPC VMs dropped at
the VPC router. Caused TestVPCSubnetE2E, TestNATGateway,
TestMultinodeSpread/NATGatewayInternet to fail.
Add subnetsImplicitlyOnMainRT helper that enumerates VPC subnets
without explicit non-main associations, and fan publish to them when
record.IsMain. Apply to all three publish helpers (IGW, NATGW,
NATGW-delete).
Symmetric fixes for effective-RT changes:
- AssociateRouteTable: subnet leaves implicit main → emit deletes for
main RT's routes before adds for explicit RT
- DisassociateRouteTable: subnet returns to implicit main → emit adds
for main RT's routes after deletes for departing RT
* chore(e2e): capture per-LR lr-route-list/lr-policy-list/lr-nat-list
ovn-nbctl show does not dump static routes or policies. Adds three
per-LR captures so RCA can verify policy/route install state after
the IGW per-subnet refactor.
* chore(e2e): tcpdump br-wan + neigh/fdb/conntrack capture during tests
EIP egress regression on feat/igw-route-table-aws-parity shows SNAT
packets present (n_packets>0) but pings still time out. Post-teardown
ovn.* snapshots can't tell whether replies actually arrived on br-wan
or got dropped at L2 / by upstream router.
start-net-capture.sh: nohup tcpdump on br-wan filtered to 192.168.0.0/23
(EIP pool) + ARP for same range, 200-byte snaplen, 64 MiB cap. Pair:
capture-net-diag.sh stops it via /tmp/spinifex-e2e-tcpdump.pid and copies
wan.pcap into net-diag/.
Also added neigh/bridge-fdb/conntrack dumps to capture-net-diag.sh so
L2/L3 binding state is visible alongside the existing OVN dumps.
Wired both single-VM and multinode workflows to start capture before
the e2e run; teardown step picks up the pcap automatically.
* fix(reconcile): rebuild IGW/NATGW per-subnet egress from KV
Publisher fires vpc.add-igw-route during default-VPC bootstrap before the
subscriber daemon attaches to the topic. NATS core pub/sub drops the
message, leaving the LR without a 0.0.0.0/0 policy and off-subnet egress
broken on the affected VPC for the lifetime of the process. Reproduced
in CI: TestVPCSubnetE2E bastion EIP could ARP and DNS its on-LAN
neighbour but never reach 8.8.8.8.
LoadIntentFromKV now fans every IGW or NAT-gateway RT route out over
its explicit + implicit-main subnets and the reconcile pass invokes
EnsureSubnetEgress / EnsureNATGatewaySubnetEgress for each one. Same
shape as publishIGWRouteEvents so the bootstrap and the runtime event
path produce identical OVN state.
* fix(routes): serialize OVN egress wiring via request-reply
vpc.igw-attach, vpc.add-igw-route and vpc.add-nat-gateway were fire-and-forget
Publish, racing the subscriber's AttachIGW barrier. AttachInternetGateway
returned before vpcd finished wiring the OVN gateway LRP, so a follow-up
CreateRoute(IGW) published an add-igw-route event that the subscriber
processed in a parallel goroutine — installing a Logical_Router_Policy with a
nexthop the LRP didn't yet expose. northd compiled an inert reroute that
never recovered, leaving the bastion's egress traffic blackholed (no packets
dst 8.8.8.8 from the failing VPC's EIP even though local DNS succeeded).
Switch the three add-* publishers to utils.RequestEvent. The subscriber ACKs
once OVN state is committed, so the gateway handler can't issue subsequent
route events against a half-wired router. delete-* paths stay Publish —
teardown ordering doesn't matter.
handleAddNATGateway now calls respond() at every exit so requesters don't
hang. Route-table tests gain an ackSub wrapper (async subscribe + auto-Reply)
so request-based publishers complete during unit tests.
* Revert "fix(routes): serialize OVN egress wiring via request-reply"
This reverts commit bf6f77d5bc8b21f72ed5867175d5a67c010066c8.
* fix(igw): restore default static route in AttachIGW/DetachIGW
OVN 25.03 LR pipeline runs lr_in_ip_routing (table 15) BEFORE lr_in_policy
(table 17). When this branch swapped the per-VPC default static route for
the per-subnet Logical_Router_Policy reroute installed by EnsureSubnetEgress,
external destinations started dropping at routing priority-0 before the
policy table ran. Reproduced locally: ovn-trace stops at table 15 drop, ping
8.8.8.8 from inside a fresh-VPC bastion produced zero packets on br-wan
until `ovn-nbctl lr-route-add <router> 0.0.0.0/0 <wan-gw> <gw-port>` was
issued manually — after which the same ping succeeded immediately. dev still
green because dev calls AddDefaultRoute on AttachIGW.
Restore the original AddDefaultRoute / DeleteDefaultRoute calls; keep the
per-subnet EnsureSubnetEgress machinery intact so future per-subnet egress
gating (DROP policies for subnets whose RT lacks a 0.0.0.0/0 route) can
layer on top. Flip the two TestAttachIGW assertions that pinned the
"AttachIGW must not install a default static route" expectation.
Pairs with revert of bf6f77d5 in the previous commit (request-reply
serialization treated a non-existent race; real bug was missing routing
fallback).
* feat(network): per-subnet egress drop gating for AWS RT parity
VPC LRs carry a router-wide 0.0.0.0/0 default static route while an IGW is
attached (lr_in_ip_routing runs before lr_in_policy on OVN 25.03, so per-
subnet reroute alone is insufficient). Subnets whose effective route table
lacks 0.0.0.0/0 were previously able to egress via that default route,
violating AWS semantics.
Install an LR_Policy DROP at SubnetEgressPriorityDrop=1100 (above IGW=1000
and NATGW=900) for unrouted subnets. Match excludes VPC CIDR, link-local
(169.254/16) and multicast (224/4) so intra-VPC, DHCP/metadata and
multicast still flow.
Wiring:
- policy.RouteManager gains AddSubnetEgressDrop / DeleteSubnetEgressDrop.
- external.IGWManager gains EnsureSubnetEgressDrop / RemoveSubnetEgressDrop.
- New events: SubnetEgressGateEvent / SubnetEgressUngateEvent on topics
vpc.gate-subnet-egress / vpc.ungate-subnet-egress.
- routetable handler publishes gate/ungate alongside existing add/delete
IGW/NATGW route events on every CreateRoute/DeleteRoute/Associate/
Disassociate/Replace mutation, based on subnet's recomputed effective RT.
- reconciler bootstrap path emits DropGates intent for subnets in IGW-
attached VPCs whose effective RT lacks 0.0.0.0/0.
Tests cover primitive idempotency, drift recovery, event publishing per
mutation surface, implicit-main fan-out, and intent computation.
* fix(igw): fan out subnet egress gate decisions on IGW attach/detach
The reconciler computes per-subnet DROP policies for any subnet in an
IGW-attached VPC whose effective route table lacks 0.0.0.0/0, but those
decisions were only re-evaluated by the periodic drift tick (every
5 min). AttachInternetGateway / DetachInternetGateway now publish
gate/ungate events for every subnet in the VPC immediately, closing
the window where a freshly-attached IGW could leave unrouted subnets
unprotected.
Wiring: IGWServiceImpl gains a nil-safe GatePublisher hook satisfied
by RouteTableServiceImpl. Daemon installs the hook once both services
exist. RouteTableServiceImpl.PublishGateDecisionsForVPC enumerates the
VPC's subnets and reuses publishSubnetEgressGateDecision per subnet.
Also fixes the LB e2e suite, which built VPC + IGW + subnet without
ever creating a 0.0.0.0/0 → IGW route — relying on the pre-DROP-gate
behaviour where unrouted subnets still had silent egress via the
router-wide static route. The fixture now installs the explicit
public route on the main RT (AWS-faithful semantics for a public
subnet).
* fix(igw): skip gate fan-out on attach to avoid bootstrap race
AttachInternetGateway fan-out races with the default-VPC bootstrap path
in ensureDefaultVPCInfrastructureFor, which attaches the IGW and
immediately publishes a 0.0.0.0/0 → IGW CreateRoute event. The two
events land on distinct NATS subjects (vpc.gate-subnet-egress vs
vpc.ungate-subnet-egress) with no cross-subject ordering guarantee.
When ungate arrived first the subscriber became a no-op and the
subsequent gate installed a permanent DROP on the default subnet,
killing the return path for guest SSH probes
(TestMultinodeGuestSSH connect timeout to the instance public IP).
Detach fan-out is retained — no follow-up CreateRoute fires after
DetachInternetGateway so the path is race-free.
Subnets without a default route still get DROPped during the
reconciler's periodic drift pass (loadSubnetDropGates). The window is
re-opened to ≤5 min on user-driven AttachInternetGateway, which is
the original behaviour pre-mulga-siv-163. A follow-up bead can close
that gap properly by converting the gate/ungate subjects into a
single recompute trigger that re-reads effective RT state.
* fix(vpc): prevent duplicate main route table from bootstrap race
Concurrent multi-node daemons running EnsureDefaultVPC for the admin
account (bootstrap-pinned VpcId) each minted a fresh random rtb-ID and
Put it with IsMain=true. The KV ended up with two main route table
records for the same VPC.
CreateRoute's post-write gate recompute (per-subnet egress drop gating)
then resolved effectiveRouteTable non-deterministically across the two
IsMain=true records; when the orphan (route-less) RT won, the gate
decision became "no default egress" and vpc.gate-subnet-egress fired.
The subscriber installed a Priority-1100 DROP on the default subnet's
egress, killing the return path for guest SSH probes
(TestMultinodeGuestSSH connect timeout to 192.168.0.161 in run
26699045536).
Two-part fix:
1. createMainRouteTable scans rtbKV for an existing IsMain=true record
on the same VPC before Put-ing a new one. Race window narrows from
"always duplicates under concurrent bootstrap" to a sub-ms TOCTOU
between scan and Put.
2. effectiveRouteTable / mainRouteTable use a deterministic preferMain
tiebreak when multiple IsMain=true records exist: more routes >
oldest CreatedAt > smaller RouteTableId. Defensive — any duplicate
that slips through (cross-AZ restore, manual KV edit, residual race)
still produces consistent egress gate decisions instead of flapping
on KV scan order.
* fix(policy): atomic ACL replacement for SG mutations
Add ovn.Client.ReplaceACLs — detach + delete existing ACL rows and
attach freshly-created rows in one OVSDB transaction. EnsureSG and
UpdateSG now funnel through ReplaceACLs instead of ClearACLs +
AddACLs, closing the mid-flight window where the port group held
zero ACLs and tenant LSP traffic defaulted to drop (observable on
ICMP and on TCP SYNs that miss conntrack).
Mock client exposes ClearACLCalls / ReplaceACLCalls counters so
tests can assert atomicity. New TestSGManager_UpdateSG_AtomicReplace
fails if SG mutations ever fall back to the split path.
* feat(elbv2): implement ModifyListener
Adds AWS-compatible ModifyListener to the spinifex ELBv2 stack so
listeners can be edited in place — port, protocol, and default
actions — without the current delete+create dance that briefly drops
the listener from HAProxy.
- Handler interface, impl, and NATS wrapper get ModifyListener;
validation mirrors CreateListener (LB type ↔ protocol, dup-port
excluding self, target-group existence + protocol compatibility).
- Daemon subscribes to elbv2.ModifyListener on the spinifex-workers
queue and a new gateway action dispatches it.
- Frontend exposes a per-row edit button that pre-fills a dialog from
the existing listener and calls ModifyListenerCommand, replacing
the "Listeners cannot be edited" note.
- Unit + NATS round-trip + frontend tests cover the new flow.
Cert / SslPolicy / Alpn handling stays out of scope and lands with
HTTPS listener support.
* test(e2e): cover ModifyListener in lb harness
Adds an Internal_ALB_ModifyListener subtest to TestLoadBalancer that
exercises in-place listener edits end-to-end:
- Provisions tgA + tgB sharing the same app instances.
- Brings up an internal ALB with HTTP:80 → tgA, baselines round-robin.
- ModifyListener port=8090, asserts DescribeListeners shows the new
port and that probes on 8090 still round-robin — proves HAProxy is
reconciled without dropping the listener (the regression mulga-944
fixes; the frontend previously delete+created on every edit).
- ModifyListener DefaultActions → tgB, asserts traffic still flows.
Also factors a probeAtPort helper so the order JSON can target any
listener port, and updates TEST_COVERAGE.md with phase 3e.
* fix(ui): widen listener add/edit dialog so content stays inside the box
AlertDialogContent defaults to max-w-xs (20rem) which clipped the listener
form on the right edge. Use a 32rem cap with grid-cols-[minmax(0,1fr)] +
min-w-0 on the form so CSS grid tracks shrink instead of expanding to
max-content.
* test: cover ModifyListener + system instance dispatch
Lifts ModifyListener diff coverage by exercising nil/empty arn paths,
NLB protocol acceptance + rejection, protocol-only revalidation against
existing target groups, and the launcher-nats wire format error paths
(missing input, missing instance type, decode failures, no responder,
default timeout fallback).
Adds a new spinifex/daemon/daemon_system_dispatch_test.go covering
splitSubjectTail edge cases, respondWithSystem{Launch,Terminate}{Output,
OK,Error} envelope shape, handler entry points for invalid JSON and
missing instance ID, and subscribeSystemTerminate nil-conn + idempotent
re-subscribe behaviour.
Fixes the flaky TestHandleEC2DescribeInstanceTypes/NoFilter… subtest by
selecting a 2-vCPU instance type that also fits schedulable memory, so
hosts with smaller schedulableMem (CI 4 vCPU / ~13 GiB) don't pick a
16 GiB-mem type that fails allocation.
* test(e2e): wait for tgB after listener swap, not before
Spinifex health-checks only target groups wired into a listener's
HAProxy config. Waiting for tgB to report healthy before
modifyListenerDefaultTG attaches it to the listener is impossible —
the agent never probes its targets, so they stay in initial and the
2m EventuallyErr trips.
Drop the pre-swap tgB wait. After modifyListenerDefaultTG, replace
the fixed 3s sleep with WaitForTargetsHealthy on tgB (post-swap),
which is now the listener's DefaultAction.
* test(e2e): derive LB name from ARN in lbENI/deleteLB ENI filter
The ENI description filter was building the LB-name token from a
hard-coded "alb"/"nlb" template (e.g. `ELB app/lb-e2e-alb-int/<id>`),
which only matched LBs whose name happened to be `lb-e2e-alb-*` or
`lb-e2e-nlb-*`. The ModifyListener suite names its LB `lb-e2e-mod`,
so lbENI's EventuallyErr could never find the ENI and tripped after
30s. deleteLB had the same bug: sysInstanceID stayed empty and
WaitForENICleanup waited for a non-existent filter, allowing the
next suite's createLB to race the sys.micro VM reclaim.
Parse the LB name straight out of lb.ARN (penultimate path segment,
same shape createLB already relies on) so the filter matches any
caller-chosen name. Drop the now-redundant scheme arg from lbENI.
* test(e2e): poll for listener serving on new port after ModifyListener
The post-modifyListenerPort settle was a hard 3s sleep, but lb-agent
polls for new HAProxy config on a 5s tick. After the API returns, the
agent has typically not yet detected the config-hash change, so
HAProxy is still bound to the old port — every probe to the new port
RSTs and probeAtPort sees 0/0 successful.
Replace the sleep with waitForListenerServing: a polling loop that
fires 2 probes via the shared trigger client every 2s until at least
one returns a parsable instance_id, with a 60s ceiling. Factor the
trigger-and-fetch out of probeAtPort into probeOnce so the polling
helper can reuse it without the require.* asserts.
* feat(instancetypes): emit cross-vendor sibling families on x86_64 hosts
Each x86_64 cpuGeneration's family list is now extended at generate time
with its cross-vendor counterpart family (Intel t3 ↔ AMD t3a,
c5 ↔ c5a, m6i ↔ m6a, etc.) via the new vendorSiblingFamily map in
definitions.go. Sibling families share an identical vCPU/memory shape
per size (same instanceFamilyDef size tables) and the same x86_64 arch
tag, so any host can schedule either vendor's request without changes
to canAllocate or the subscription path.
ResourceManager.updateInstanceSubscriptions iterates the type catalog
unchanged; it now picks up both vendors automatically and subscribes to
ec2.RunInstances.t3.* AND ec2.RunInstances.t3a.* on the same node.
RunInstances for either family load-balances across mixed Intel + AMD
clusters via the existing spinifex-workers queue group.
ARM (Graviton t4g/c?g/m?g/r?g) and legacy Intel-only families (t2, c4,
m4, r4) have no cross-vendor sibling and are unaffected.
Bead: mulga-siv-157 (vendor aliasing portion). sys.* NATS fan-out for
ALB-VMs is a separate change in the same plan.
* feat(daemon,elbv2): NATS fan-out for system instance launch/terminate
ELBv2 ALB-VMs (sys.micro) previously launched via a direct Go method call
from ELBv2ServiceImpl into the local Daemon, so every ALB-VM piled on
whichever node handled the upstream CreateLoadBalancer call. Mixed
clusters tripped 'insufficient capacity for sys.micro' once that node's
host-reserve trimmed schedulable vCPU below the next launch.
Route system-instance lifecycle through a dedicated NATS subject root:
system.LaunchInstance.{type} (queue group spinifex-workers)
system.LaunchInstance.{type}.{nodeID} (targeted)
system.TerminateInstance.{instanceID} (owning-node only)
ResourceManager.updateInstanceSubscriptions now treats sys.* types under
the new subject root with a separate handler hook (systemHandler) and
the same capacity-aware subscribe/unsubscribe behaviour customer types
already have — sys.micro subscriptions drop when the host saturates,
naturally re-distributing load.
The daemon side keeps LaunchSystemInstance / TerminateSystemInstance as
local methods; the new handleSystemLaunchInstance / handleSystemTerminate
handlers decode the JSON envelope, invoke the local method, and reply
with output or error. The terminate subscription is bound on launch
success so future terminate requests reach the owning node, and dropped
on terminate success.
ELBv2ServiceImpl now consumes a NATSSystemInstanceLauncher (handlers/elbv2/
system_instance_launcher_nats.go) instead of a direct daemon pointer, so
any node in the cluster can carry an ALB-VM.
SystemInstanceInput / Output / NICConfig / ExtraENIInput get explicit
JSON tags because the wire crosses NATS.
Tests cover JSON round-trip, remote-error propagation, terminate routing,
and no-responder failure mode.
Bead: mulga-siv-157 (sys.* fan-out portion completes the bead).
* fix(gateway/ec2): fan out DescribeInstanceAttribute across daemons
DescribeInstanceAttribute was queue-grouped on `spinifex-workers`, so each
request landed on one daemon at random. The handler only consults per-daemon
state (vmMgr / stoppedStore), so 1/N requests succeeded and the rest returned
InvalidInstanceID.NotFound. Terraform's AWS provider refreshes aws_instance
with three attribute reads per cycle, which on a 5-node cluster fails ~99% of
the time.
Move the daemon subscription into the fan-out block (no queue group) and
rewrite the gateway to mirror DescribeInstances: inbox + N replies + drop
NotFound. The aggregator returns the first parsed success, falls back to
NotFound only when every daemon confirms absence, and propagates deterministic
client errors (InvalidParameterValue, MissingParameter) when no daemon
succeeds.
Tests:
- New: MultipleNodes_OwnerWins, AllNodesNotFound, ClientErrorPropagates,
NoResponders, SingleNode (rewritten without queue group).
- daemon_handlers_test.go: drop the `spinifex-workers` queue-group string from
the ten DescribeInstanceAttribute subscriptions so the tests match the new
production wiring.
* fix(ec2): DescribeInstanceTypes returns supported set, not capacity-gated
The no-filter DescribeInstanceTypes response was filtered to types with
at least one free slot. AWS-compatible callers (e.g. the Terraform AWS
provider's post-RunInstances metadata lookup) expect a stable supported
set: when a launch consumes the last free slot of its type, the next
DescribeInstanceTypes for that type must still return it, or the apply
fails with "reading EC2 Instance Type (X): empty result".
Split the API into two ResourceManager methods so capacity semantics
stay strict for the capacity=true filter (used for cluster aggregation)
while default callers get the supported answer:
- GetAvailableInstanceTypeInfos(showCapacity bool) — unchanged; gates on
remaining schedulable CPU/memory and (for GPU types) free GPU pool
slots. Used when capacity=true is requested.
- GetSupportedInstanceTypeInfos() — new; lists every loaded instance
type once, regardless of current allocation or GPU pool state. Still
skips system types ("sys.*") and entries with incomplete CPU/memory
metadata.
The handler picks between the two based on the capacity filter. The
gateway aggregator (DescribeInstanceTypes.go) needs no change — it
already deduplicates by type name when capacity=false, so requesting a
specific name now resolves from whichever node supports it.
Tests:
- Daemon unit tests for the new method cover the three filter rules
(no free slots, system/incomplete skip, GPU without manager).
- Handler tests assert the supported path is hit by default and the
capacity-gated path is hit only when capacity=true.
- The NATS-driven integration test is split into two subtests: one
proves the no-filter response is invariant under allocation, the
other proves capacity=true still gates on remaining resources.
* fix: persist API device name in volume metadata on AttachVolume
aws_volume_attachment polls DescribeVolumes with
attachment.device=/dev/sd[f-p], the API-form name. Spinifex was
persisting the in-guest virtio path (/dev/vd*) via UpdateVolumeState
after AttachVolume, so the filter rejected every attached volume and
the post-attach wait loop failed with "couldn't find resource".
UpdateVolumeState now receives the API-form device name and the
daemon's AttachVolume response echoes DeviceName instead of
GuestDevice. BlockDeviceMappings continues to carry the guest path
(mulga-599 / PR #55) since UpdateGuestDeviceNames re-applies that
convention on every Launch/Start; only the volume-metadata path
diverges to the API form for the AWS-spec round-trip.
Adds fakeVolumeStateUpdater to manager_deps_test.go and a regression
test that asserts the API-form name flows through
UpdateVolumeState while BDM keeps the guest path.
Resolves mulga-siv-154.
* refactor(network/external): introduce Allocator interface + StaticPoolAllocator
Carve external IP allocation out of handlers/ec2/vpc/external_ipam.go into
network/external/static_pool.go behind a typed Allocator interface. The KV
bucket name, schema, CAS retry count, and gateway-IP reservation semantics
are unchanged so existing static-pool data carries over with no manual
migration.
ExternalIPAM stays in handlers/ec2/vpc (L5 cleanup deferred per plan) but
becomes a thin facade that constructs a StaticPoolAllocator and delegates
Allocate / Release / GetPoolRecord through the new interface. Q4 will plug
DHCPPoolAllocator into the same Allocator slot.
Plan: docs/development/improvements/external-pool-dhcp-source.md (Q1).
* feat(network/external/dhcp): package skeleton — Client + Fake + KV store
New network/external/dhcp/ package, restored from the structure deleted
in Slice C (commit 7ff013a) but rehomed under the ADR-0006 layer tree:
- client.go Client interface + AcquireRequest + Lease (with
RenewAt/RebindAt/ExpiresAt helpers).
- nclient4.go NClient4Client — AF_PACKET DORA / Renew / Release
backed by insomniacslk/dhcp/dhcpv4/nclient4.
- fake.go In-memory test fake with per-step hooks for fault
injection and acquire/renew/release call counters.
- kv.go Store over the per-AZ spinifex-dhcp-leases-<az> KV
bucket: Put / Get / Delete / List. Schema matches the
plan (absolute timestamps for acquired/t1/t2/expiry).
- *_test.go Lease-timer arithmetic, fake lifecycle, nclient4 input
validation, KV roundtrip + version-key isolation.
Not yet wired into vpcd, ExternalIPAM, or any allocator — Q3/Q4 plug
it in. Plan: docs/development/improvements/external-pool-dhcp-source.md (Q2).
* feat(network/external/dhcp): DHCPManager + daemon-side NATSClient
Adds the per-vpcd DHCPManager that owns active lease state: KV scan on
startup (re-issues RENEW for non-expired entries, drops the rest),
per-lease renewal goroutines with jittered T1/T2/expiry timers and
post-expiry re-DORA, and NATS request/reply subscribers on
vpc.dhcp.acquire / vpc.dhcp.release. Acquire is idempotent on client-id,
coalesced via singleflight so concurrent retries do not produce
duplicate wire DORAs.
Adds the daemon-side dhcp.NATSClient wrapper (RequestAcquire /
RequestRelease) that ExternalIPAM will route DHCP-sourced allocations
through in Q4. Still no allocator wiring — vpcd does not yet construct
the manager.
* feat(network/external/dhcp): wire DHCP allocators; re-allow source=dhcp
Adds DHCPGatewayLRPAllocator and DHCPPoolAllocator, the two consumers
of the Q3 DHCPManager. The gateway-LRP allocator runs in-process inside
vpcd and DORAs once per VPC (client-id `dhcp-gw-lrp-<vpcID>`, used in
centralized NAT). The pool allocator runs daemon-side and routes
per-EIP / ENI / instance acquisitions through vpcd over the restored
vpc.dhcp.acquire / vpc.dhcp.release topics.
ExternalIPAM now dispatches per-pool: static pools keep the existing
StaticPoolAllocator + KV bucket; DHCP-sourced pools attach a
DHCPPoolAllocator via EnableDHCP. vpcd constructs the dhcp.Manager
when any pool has Source="dhcp" and selects the DHCPGatewayLRPAllocator
for IGWManager; everything else keeps StaticRangeAllocator /
LinkLocalAllocator.
Config validator re-allows source="dhcp", requires bind_bridge, and
rejects range_* / gw_lrp_range_* keys on DHCP pools. Unknown sources
are rejected with the supported list. Allocator.Release uses a
server-side IP→client-id lookup so the existing (pool, ip) signature
keeps working for both static and DHCP pools.
Plan: docs/development/improvements/external-pool-dhcp-source.md (Q4)
* feat(spx admin init): restore --external-source=dhcp + --external-bind-bridge
Source defaults to dhcp when --external-pool is omitted; static remains the
default when a pool range is supplied. Adds PoolBindBridge propagation through
ConfigSettings + formation.NetworkConfig + spinifex.toml template so vpcd's
DHCPManager finds the bridge to DORA on across single-node and joiner paths.
* fix(network/external/dhcp): derive chaddr from client-id when HWAddr unset
DHCPGatewayLRPAllocator and DHCPPoolAllocator (first acquire) own no NIC
and pass an empty HWAddr to the manager. nclient4 rejected the request
with "dhcp acquire: hw_addr is required", breaking nightly cells 1/3/8/10
at AttachIGW. Derive the chaddr from the client-id via utils.HashMAC so
the upstream DHCP server sees a stable 02:xx:xx:xx:xx:xx MAC per AWS
identifier, matching the OVN-side MAC scheme operators already grep for.
* fix(network/external/dhcp): enable IFF_PROMISC on bind_bridge across DORA
Nightly cell 1 timed out at OFFER step with "no matching response packet
received". Upstream router replied unicast to the derived 02:xx chaddr
which the Linux bridge had never learned, so the kernel dropped the
frame before nclient4's AF_PACKET socket could see it.
Reference-counted IFF_PROMISC toggle around every Acquire/Renew/Release
so overlapping leases on the same bridge share one flag flip and the
last release restores prior state. Soft-fails to a warn log when
CAP_NET_ADMIN is unavailable — operator may already have promisc set.
* fix(network/external/dhcp): RFC 2131 retransmission — retry DISCOVER 3×
NClient4 was configured with WithRetry(1) + WithTimeout(60s), meaning a
single DISCOVER per Acquire. When the first frame is lost (STP convergence
on a freshly-up bridge, upstream packet loss) the call blocks the full
60s and fails. Nightly cell 1 hit this at startup: gw-lrp Acquire for two
VPCs timed out back-to-back before any e2e step ran.
Switch to WithRetry(3) with a 5s per-attempt seed; nclient4 doubles the
deadline on each retransmission (5s, 10s, 20s) and sends a fresh DISCOVER
each round. Total budget ~35s, three frames on the wire — recoverable
without the higher layer noticing.
* fix(network/external): three CI-blocking extdhcp bugs
DHCP-source pools failed e2e-nightly cells 1/3/8/10 (net: dhcp) after
the restore-dhcp-source branch landed. Three independent root causes:
* gateway nexthop dropped: GatewayIPAllocator.Allocate had no return
channel for the upstream router learned from lease.Routers. igw fell
back to link-local 169.254.0.2, unreachable on the WAN subnet, so
instance egress and SSH-in died silently. Widen the interface to
return nexthop; DHCP impl pulls Routers[0]; empty Routers is now a
hard error. Static impl returns pool.Gateway as before.
* DORA budget shrunk 6x: WithTimeout(5s) x WithRetry(3) gave a 15 s
wallclock for the full DORA, no outer backoff. Restore the deleted
dhcp_manager.go schedule [4s, 8s, 16s, 32s] under a 90 s budget with
+/- 1 s per-attempt jitter; drop nclient4Retries to 1 so the leaf
stays single-shot and the budget arithmetic lives in one place.
* every vpcd subscribed to vpc.dhcp.acquire: plain nc.Subscribe meant
every vpcd in real-multi cells DORA'd in parallel with the same
derived chaddr, causing upstream-server NAK / lease leak. Switch to
QueueSubscribe with per-AZ queue "vpcd-dhcp-workers-<az>".
Plan: docs/development/bugs/extdhcp-pool-fixes.md (mulga).
Bead: mulga-siv-150.
* feat(gpu): add NVIDIA MIG multi-instance GPU support
Partitions MIG-capable GPUs (A100, A30, H100) into isolated slices exposed
as mdev devices instead of whole-GPU VFIO passthrough. VMs attach via
-device vfio-pci,sysfsdev=<mdev-path> using the GPUAttachment discriminated
union (PCIAddress for whole-GPU, MdevPath for MIG).
Key changes:
- gpu: new MIGProfile/MIGInstance types, IsMIGCapable model map, mig.go ops
(EnableMIGMode, DisableMIGMode, CreateInstances, ListInstances,
DestroyAllInstances, enrichMdevPaths)
- gpu.Manager: Claim() returns (*GPUDevice, *MIGInstance, error); AddMIGInstances,
ReclaimByMdev for restart recovery; MarkMIGFailed per-slice
- daemon: buildGPUPool helper partitions devices into whole-GPU vs MIG pools;
MIGProfile config field selects profile per node or per GPU override
- vm.VM: GPUAttachments []gpu.GPUAttachment replaces GPUPCIAddresses []string
- admin CLI: spx admin gpu mig status/enable/disable subcommands with
admin.SetMIGProfile for spinifex.toml writes
- docs: COMMANDS.md updated with three new MIG management command rows
* test(gpu): unit tests for MIG parsers and pool manager
Cover all pure parsing functions (parseMIGMemory, isCapacityExhaustedError,
extractProfileFields/GIFields, parseMIGProfiles, parseMIGInstances,
parseCreatedGIID/CIID) with realistic nvidia-smi fixture output.
Manager MIG tests cover AddMIGInstances, Claim/Release on mdev-backed entries,
MarkMIGFailed, ReclaimByMdev, and interaction with whole-GPU MarkFailed — all
using fake mdev directories in t.TempDir() so no GPU hardware is required.
* feat(gpu): dynamic MIG profile-per-request with on-demand GI/CI lifecycle
Replaces static config-driven MIG pre-carving with a dynamic model where
the MIG profile is specified per RunInstances call via mig.<profile>
instance type names (e.g. mig.1g.10gb) that mirror nvidia-smi nomenclature
directly.
- Manager.Claim(instanceID, profileName): fast path claims a pre-carved
slice; slow path dequeues a free MIG GPU, releases the lock, calls
nvidia-smi mig -cgi/-cci to carve all slices, then reclaims the lock to
commit and return the first slice
- Manager.Release: last release on a dynamically-carved GPU calls
DestroyAllInstances and returns the GPU to freeMIGGPUs, allowing a
different profile on the next claim
- AddMIGGPU(dev): registers a fresh MIG GPU for dynamic allocation;
AddMIGInstances (restart recovery) keeps its no-destroy semantics
- GenerateMIGTypes: produces mig.<profile> InstanceTypeInfo entries from
gpu.ListProfiles at daemon startup; IsMIGType/MIGProfileFromType let
callers inspect these names
- buildGPUPool: calls ListInstances for recovery, AddMIGGPU for fresh
GPUs, ListProfiles to collect MIG profile specs for type generation;
removes setupMIGInstances and migProfileForDevice
- GPUClaimer.Claim gains profileName param; daemonGPUClaimer, service_impl
call sites, and fakeGPUClaimer all updated
* fix(gpu): correct mdev parent detection and make --profile optional for mig enable
enrichMdevPaths was using os.Readlink on "parent_bus_0" (a non-existent
sysfs attribute) and then on ".." (a directory, not a symlink) — both
would fail on real NVIDIA hardware, breaking CreateInstances and
ListInstances on daemon restart recovery. Fix: read the mdev UUID's own
symlink target, which always encodes the parent GPU's PCI address in its
path (e.g. .../0000:01:00.0/<uuid>).
Make basePath injectable in enrichMdevPaths so the fix is covered by a
new set of unit tests using temp-dir symlinks (TestEnrichMdevPaths_*).
Remove the MarkFlagRequired("profile") from "spx admin gpu mig enable":
Phase 2 is fully dynamic — the daemon carves slices on demand at
RunInstances time. Without --profile, the command just enables MIG mode
and destroys any pre-carved slices so the daemon starts with a free GPU.
With --profile, behaviour is unchanged (pre-carve a static pool).
* chore: writeups for smci and arcee
* test: add coverage for MIG exec, manager slow-path, instancetypes and admin
- Add mig_exec_test.go with fake nvidia-smi harness covering Enable/Disable MIGMode,
IsMIGModeEnabled, DestroyAllInstances, ListProfiles, CreateInstances, ListInstances
- Extend manager_mig_test.go with slow-path Claim/Release tests using fake nvidia-smi
- Add generate_test.go coverage for IsMIGType, MIGProfileFromType, GenerateMIGTypes
- Add admin_test.go coverage for SetMIGProfile (create, update, idempotent, error paths)
- Make mdevBasePath a var to allow sysfs injection in tests
- Add json tags to GPUAttachment fields to satisfy musttag linter
Diff coverage: 75.6% (threshold: 60%).
* fixed gpu passthrough doc
* Revert "fixed gpu passthrough doc"
This reverts commit 1e20f5ab13.
* feat: ENI hot-plug Sprint 3a scaffolding
Lays the Sprint 3a Tier 0 groundwork from
docs/development/feature/enihotplug-v1.md so the live QMP pipeline in 3b
can plug in without further plumbing changes.
- types.ENIRequests: per-VM PCIe slot allocator state (free-list +
eniID→slot reverse map), mirrors EBSRequests pattern.
- instancetypes.MaxENIsForType / HotPlugENISlotsForType: instance-type →
ENI cap lookup with explicit-type-then-family-then-default precedence.
- vm.DeviceController interface + QMPDeviceController (real) +
StubDeviceController (in-memory): unit-testable QMP device_add /
device_del / netdev_add / netdev_del surface. QueryPCI body lands in
Sprint 3b alongside the live response parser.
- buildBaseVMConfig now accepts the instance type and emits two distinct
PCIe root-port pools at boot: hotplug-ebs{1..11} (fixed, /dev/sd[f-p])
and hotplug-eni{1..N} (per-type, N = MaxENIs - 1).
- volumes.go bus id renamed hotplug%d → hotplug-ebs%d; query-block QDev
regex narrowed to hotplug-ebs(\d+) so future virtio-net devices on
hotplug-eni ports cannot collide with the EBS ordering.
- restore + ResetNodeLocalState reset ENIRequests.Mu alongside the EBS
mutex so restored VMs start with a fresh allocator.
Closes mulga-cs-enihotplug-3a.
* feat: ENI hot-plug Sprint 3b — live QMP pipeline + AttachENI wiring
Lands the Sprint 3b Tier 1 deliverables from
docs/development/feature/enihotplug-v1.md so AttachNetworkInterface /
DetachNetworkInterface AWS APIs drive the host-side QMP pipeline against
a running VM.
- vm/qmp_device.go: real QueryPCI parser. Walks the per-bus query-pci
response and recurses into pci_bridge.devices so virtio-net devices
attached under hotplug-eni{N} root ports surface in the flat result.
- vm/eni_hotplug.go (new): HotPlugENI / HotUnplugENI on Manager. 4-step
pipeline (tap → ovs-add-port → netdev_add → device_add + query-pci
poll) with full rollback on step-N failure. Tap + OVS calls are
stubbed in Sprint 3b; Sprint 3c drops in real NetworkPlumber /
ovs-vsctl invocations against the same call-site interface.
- vm/eni_hotplug_test.go (new): unit tests with StubDeviceController —
success path, rollback on each step, AttachmentLimitExceeded,
idempotent re-attach, non-running rejection, force-detach semantics,
concurrent serialization, query-pci timeout.
- types/ec2.go: EC2InstanceCommand gains AttachENI / DetachENI attrs +
AttachENIData / DetachENIData payloads, mirroring the AttachVolume
command surface.
- handlers/ec2/vpc/eni.go: ENIRecord gains AttachmentStatus,
HotPlugSlot, LastAttachError, DetachInFlight, DetachForce fields per
plan §State machine. New helpers: GetENIRecord, UpdateENI (CAS
patcher), FindENIByAttachment (used by DetachNetworkInterface lookup
by attachmentId).
- gateway/ec2/vpc/{Attach,Detach}NetworkInterface.go (new): AWS SDK
request decoders that route via the per-instance ec2.cmd.{id}
channel; DetachNetworkInterface resolves attachmentId →
instanceId via DescribeNetworkInterfaces first.
- gateway/ec2.go: registers the two new EC2 actions in the dispatcher.
- daemon/daemon_handlers.go: handleEC2Events dispatches the two new
attribute flags to per-instance handlers.
- daemon/daemon_handlers_eni.go (new): composes vpcService.AttachENI
(KV) with vmMgr.HotPlugENI (QMP) and publishes
vpc.eni-hotplug.{attached,detached}.{instanceID} on success. KV
state machine: available → attaching → attached → detaching →
available. On QMP failure: KV rolls back to available with
LastAttachError persisted for DescribeNetworkInterfaces visibility.
- vm/errors.go: ErrENINotAttached, ErrENIPipelineTimeout,
ErrQMPUnavailable.
The Sprint 3c bead lands the real tap + ovs-vsctl wiring on the same
pipeline; 3d adds the reconciler that converges the attaching /
detaching KV states left behind by a daemon crash mid-pipeline.
Closes mulga-cs-enihotplug-3b.
* Update ec2.go
* Update ec2.go
Lays the Sprint 3a Tier 0 groundwork from
docs/development/feature/enihotplug-v1.md so the live QMP pipeline in 3b
can plug in without further plumbing changes.
- types.ENIRequests: per-VM PCIe slot allocator state (free-list +
eniID→slot reverse map), mirrors EBSRequests pattern.
- instancetypes.MaxENIsForType / HotPlugENISlotsForType: instance-type →
ENI cap lookup with explicit-type-then-family-then-default precedence.
- vm.DeviceController interface + QMPDeviceController (real) +
StubDeviceController (in-memory): unit-testable QMP device_add /
device_del / netdev_add / netdev_del surface. QueryPCI body lands in
Sprint 3b alongside the live response parser.
- buildBaseVMConfig now accepts the instance type and emits two distinct
PCIe root-port pools at boot: hotplug-ebs{1..11} (fixed, /dev/sd[f-p])
and hotplug-eni{1..N} (per-type, N = MaxENIs - 1).
- volumes.go bus id renamed hotplug%d → hotplug-ebs%d; query-block QDev
regex narrowed to hotplug-ebs(\d+) so future virtio-net devices on
hotplug-eni ports cannot collide with the EBS ordering.
- restore + ResetNodeLocalState reset ENIRequests.Mu alongside the EBS
mutex so restored VMs start with a fresh allocator.
Closes mulga-cs-enihotplug-3a.
* refactor(vpcd): remove macvlan bridge mode
Macvlan was a single-NIC SSH-safe hack that predated veth mode and was
never selected by any active config path. Deletion is Phase 0 item 1 of
the network layer redesign (mulga-siv-125.1).
Removed:
- BridgeModeMacvlan const + setMacvlanMAC + ifaceIsMacvlan probe
- Macvlan MAC alignment block in launchService (only fired on dead path)
- Macvlan branch in detectBridgeMode + matching tests
- Macvlan duplicates of IGW/AddNAT centralized-NAT tests (veth versions
already cover the same path)
- --macvlan flag + macvlan case branch in setup-ovn.sh
Refactored:
- useCentralizedNAT() now == BridgeModeVeth (was != BridgeModeDirect);
verifyBridgeMode already rejects empty/unknown at startup
- Three IGW-pool tests now explicitly WithBridgeMode(BridgeModeVeth)
(previously relied on empty-mode defaulting to centralized)
Kept intentionally:
- Deprecation notes in docs/compute/vpc-networking/README.md
- Legacy macvlan-interface cleanup in reset-dev-env.sh + ansible
teardown role (migration for operator systems with stale spx-ext-*
interfaces from pre-veth installs)
* feat(vpcd): resolve external CIDR at startup with bounded retry
Block service start until the WAN bridge has an IPv4 address, with a 30s
retry window. Phase 0 item 2 of the network layer redesign
(mulga-siv-125.1). Guards the boot race where vpcd starts before
systemd-networkd finishes DHCP on br-wan, or before netplan applies a
static config.
Skipped when ExternalMode is empty (overlay-only deployments).
Forward-compatible with ADR-0006: the new resolveExternalCIDR /
externalCIDRFromBridge helpers become the backing implementation of
HostWiring.ExternalCIDR() once the network/host package lands in Phase 1.
Until then the resolved value is only used to fail-start on missing
uplink.
* fix(vpcd): cover external CIDR startup paths
Extract launchService's CIDR-resolve block to ensureExternalCIDRReady
helper so it's unit-testable, and add direct tests for
externalCIDRFromBridge (loopback + unknown-interface) to exercise the
original literal body. Closes the diff-coverage gap on Phase 0 commit
15b564d which left these paths unreachable from unit tests.
* refactor(network/ovn): extract L1 OVN client to network/ovn package
Move OVNClient interface, LiveOVNClient, ACLSpec, ErrNATNotFound and
namedUUID helper from services/vpcd to a new spinifex/network/ovn
package. The mock relocates to spinifex/network/ovn/mock as a
sub-package. Storage fields on the mock become exported so vpcd tests
can continue to seed/inspect state across the new package boundary.
services/vpcd retains thin type-alias and var-alias shims so existing
callers compile unchanged; Phase 2 removes those shims as vpcd is torn
down.
Adds -coverpkg=./spinifex/... to test-cover so cross-package test
exercise of mock methods continues to count toward coverage now that
the mock lives outside vpcd.
First sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md
(bead mulga-siv-125.2). Pure refactor; no functional change. All
existing tests pass; preflight green.
* feat(network/topology): TopologyManager interface + vpcd adapter
Add spinifex/network/topology with Manager interface, VPC/Subnet/Port
specs, and OVN object naming helpers covering the L2 stable contract
(§7 of the redesign plan). Phase 1 keeps existing in-tree OVN names
(rtr-, ext-port-, sg_) and notes the §7.3 targets for a later
reconciliation migration.
TopologyHandler in services/vpcd grows EnsureVPC / DeleteVPC /
EnsureSubnet / DeleteSubnet / EnsurePort / DeletePort /
SetPortSecurityGroups methods that mirror the corresponding NATS
handlers' OVN ops with Spec inputs instead of nats.Msg envelopes,
satisfying topology.Manager via a compile-time check. Existing handlers
are unchanged; the duplication is short-lived (Phase 2 collapses them).
Smoke tests in topology_manager_test.go cover the idempotent create
path, delete cascade, and declarative SG membership move. Deliverable:
TopologyManager.EnsureVPC/Subnet/Port callable; all existing tests
green.
Second sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md
(bead mulga-siv-125.2).
* feat(network/host): L0 HostWiring interface + physical/veth impls
Phase 1 deliverable for mulga-siv-125.2. Introduces network/host (L0 of
ADR-0006): UplinkMode enum, Wiring interface, and Physical/Veth
implementations with injectable Runner + InterfaceReader for testability.
Not yet wired into daemon/vpcd — Phase 2 replaces daemon.SetupComputeNode
and vpcd's resolveExternalCIDR with calls through this package.
* feat(network/policy): L3 SecurityGroup/NAT/Route managers
Introduce network/policy as the data-plane policy layer (ADR-0006 L3).
Three managers, all backed by network/ovn:
- SecurityGroupManager: clear+add ACL replacement on an existing port
group; port group lifecycle stays with L2 (topology.Manager).
- NATManager: bound to one NATMode at construction (derived from
host.UplinkMode). Distributed sets logical_port/external_mac on EIP
dnat_and_snat rules; centralised leaves them unset. Per-EIP, NAT GW,
and IGW-default SNAT entrypoints; all deletes idempotent.
- RouteManager: AddStaticRoute is read-then-decide so retries no longer
duplicate rows (closing the long-standing non-idempotency bug in
vpcd/topology.go's IGW default-route path).
ACL priority constants and infrastructure ACL builders move from
vpcd/{security,acl}.go into policy/acl.go with their semantics preserved
(default-deny 900/800 logged, DHCP-allow 1050, tenant 1000). No vpcd
wiring changes here — additive only; the reconciler (bead .3.2) and
network/external (bead .3.4) will consume these managers.
bead: mulga-siv-125.3.1
* feat(network/external): L5 IGW/EIP/NATGW managers
Pure additive — new package, no topology.Manager changes, no rewiring of
existing vpcd handlers. IGWManager builds the external switch + localnet +
gateway LRP + default route + chassis HA, with a flows barrier callers can
inject (production: ovn-nbctl --wait=hv sync). EIP/NATGW are thin L5
wrappers over policy.NATManager. GatewayIPAllocator interface lets the
DHCP-backed allocator stay in vpcd until the DHCP-mgr removal bead lands;
LinkLocalAllocator + StaticRangeAllocator ship here.
* feat(network/reconcile): L6 intent-actual reconciler replaces 5-pass startup
Build spinifex/spinifex/network/reconcile/ package per Phase 2.2 of the
network layer redesign. Reads desired state from NATS KV filtered by
vpc.AZ (with legacy empty-AZ matches-all rule), scans actual state from
OVN NB DB, diffs and applies with VPC → Subnet → Port → SG → IGW →
EIP/NATGW topological ordering.
Atomic port-group join via CreateLogicalSwitchPortInGroups closes the
racy gap the legacy scanENIPortMembership pass used to revive 30s later.
A 5-minute drift goroutine (leader-gated via JetStream KV CAS) replaces
the old 30s ReconcileSGsLoop.
Adds AZ field to VPCRecord stamped at create time. Empty AZ on pre-Phase
2.2 records matches every local AZ, so no migration pass is required.
Replaces and removes the 5-pass startup sequence in services/vpcd:
- Reconcile + ReconcileFromKV
- ReconcileSGsOnce + ReconcileSGsLoop
- RetrofitAllExternalLocalnetOptions
- RetrofitAllGatewayPortNetworks
IGW fresh-create continues to flow through the legacy NATS handler path
(handleIGWAttach) so the dhcpManager wiring stays intact until Phase
2.3; the reconciler only rebinds gateway-chassis on existing IGW
topology.
Closes mulga-siv-125.3.2.
* refactor(network/topology): promote live Manager impl to network/topology
Move the topology.Manager implementation out of services/vpcd/topology_manager.go
into network/topology/manager_live.go as a standalone liveManager constructed
via NewLiveManager(ovn.Client, opts...). Inputs that varied per-handler
(DNS server resolution from the external pool) are now passed as functional
options (WithDNSServer).
TopologyHandler still satisfies topology.Manager but its methods delegate
to a cached liveManager. The shim stays until the rest of mulga-siv-129
routes NATS subscribers through topology.Manager directly.
Also exports SubnetGatewayCIDR + FormatDNSServerList from network/topology
so the reconciler in network/reconcile can share helpers with the live
manager (audit pass to follow per plan §6).
Test cases ported one-for-one from services/vpcd/topology_manager_test.go
to network/topology/manager_live_test.go; the vpcd-side test file is
deleted. New WithDNSServer test asserts the option propagates into
DHCPOptions rows.
Refs mulga-siv-129 (Phase 2.6 vpcd subscriber migration, Slice 1 of N).
* refactor(vpcd): collapse SG subscribers onto network/{topology,policy} managers
Phase 2.6 / mulga-siv-129 slice 2. Rewrites the vpc.create-sg /
vpc.update-sg / vpc.delete-sg NATS subscribers in services/vpcd as
thin wrappers over topology.Manager (port group lifecycle) and
policy.SecurityGroupManager (ACL programming). Deletes the ~1000
lines of duplicated ACL builder and provisioning logic — its
authoritative copy lives in network/policy.
Surface additions:
- ovn.ErrPortGroupNotFound sentinel; mock + live GetPortGroup wrap
not-found via fmt.Errorf("%w: ...") so callers can use errors.Is.
- topology.Manager grows EnsureSGPortGroup / DeleteSGPortGroup
(resolves the apply.go TODO from §2.1 of the redesign plan).
liveManager implements them with GetPortGroup-then-Create /
GetPortGroup-then-Clear+Delete idempotency.
Behaviour delta: the old provisionSG rolled back the OVN port group
on AddACLs failure ("all-or-nothing"). The new path lets the port
group survive and relies on network/reconcile to retry EnsureSG on
its next sweep. The all-or-nothing fail-fast test cases are dropped;
ACL emission semantics are covered by network/policy/sg_test.go.
* refactor(vpcd): migrate VPC/Subnet/ENI subscribers to topology.Manager
Phase 2.6 / mulga-siv-129 slice 3. Rewrites vpc.create / vpc.delete /
vpc.create-subnet / vpc.delete-subnet / vpc.create-port /
vpc.delete-port / vpc.update-port-sgs NATS subscribers as thin
wrappers (≤30 lines each) over topology.Manager. Subscriber bodies
now: parse event → translate to spec → delegate to manager.
Deletes:
- vpcd.subnetGateway (and helpers_test.go + topology_test.go's
TestSubnetGateway block); parity coverage lives in
network/topology.TestSubnetGatewayCIDR.
- vpcd.reconcilePortSGs (the live manager already exposes the same
diff-then-transact via reconcilePortSGs on the L2 side).
- vpcd.TopologyHandler.dnsServer; lmDNSServer (passed via
topology.WithDNSServer to the live manager) is the live path.
Behaviour delta: handlers now strict-parse PrivateIpAddress /
MacAddress / CidrBlock at the NATS boundary and reject malformed
payloads instead of degrading to "no DHCP options". The reconciler
guarantees that field validity is upheld upstream of the publish.
* refactor(network/reconcile): drive SG port groups via topology.Manager
applySGs now calls topology.Manager.EnsureSGPortGroup /
DeleteSGPortGroupByName instead of reaching directly into the OVN client.
Reconcile.Config gains a Topology field (required); the production caller
in services/vpcd passes the TopologyHandler shim, the reconcile package
tests construct a fresh topology.NewLiveManager bound to the mock OVN
client.
DeleteSGPortGroupByName is added to the topology.Manager interface for the
reconciler's orphan-removal path, which iterates raw OVN port-group names
without a groupID round-trip. The existing DeleteSGPortGroup(groupID)
delegates to it.
Resolves the TODO at apply.go:137 carried over from slice 2 of
mulga-siv-129.
* feat(network/policy): add FlowsBarrier option to NATManager
NATManager.AddEIP needs to block on `ovn-nbctl --wait=hv sync` after
writing the dnat_and_snat row so the NATS reply only returns once every
chassis has the rewrite flow. The legacy vpc.add-nat handler does this at
the tail of handleAddNAT; the manager has lacked it since slice 2.1.
NewNATManager grows a functional-options tail (Option / WithFlowsBarrier).
Defaults to a no-op for tests; production wiring in services/vpcd passes
waitForFlowsHV. AddEIP now invokes the barrier after AddNAT, logging at
warn level on failure to match the legacy handler's _ = waitForFlowsHV()
swallowing semantics.
Pre-work for slice 4 of mulga-siv-129 — without the barrier the NAT EIP
subscriber migration would lose the post-write flow-install gate.
* refactor(vpcd): migrate NAT GW subscribers to policy.NATManager
handleAddNATGateway / handleDeleteNATGateway now decode the NATS event
and delegate to policy.NATManager.{AddNATGateway,DeleteNATGateway}. The
manager carries identical OVN logic — same snat type, same ExternalIDs
(spinifex:vpc_id + spinifex:nat_gateway_id) — so existing
TestTopologyHandler_AddNATGateway / DeleteNATGateway suites stay green.
Lazy-constructed NATManager added on TopologyHandler (natmOnce). Mode is
derived from the bridge mode (veth → centralized, direct → distributed)
and the FlowsBarrier is bound to waitForFlowsHV for parity with the
legacy EIP path's tail barrier (used by the next slice, vpc.add-nat).
Slice 4 of mulga-siv-129, sub-step b.
* refactor(vpcd): migrate EIP NAT subscribers to policy.NATManager
handleAddNAT / handleDeleteNAT now decode the NATS event and delegate to
policy.NATManager.{AddEIP,DeleteEIP}. The behavioural contract is
preserved end-to-end:
- Idempotency-skip on a matching dnat_and_snat row (same VPC + LogicalIP,
plus MAC + LogicalPort under distributed NAT) moves into AddEIP so
duplicate vpc.add-nat publishes never tear down the rule the previous
invocation just installed (mulga-siv-124).
- Stale-rule cleanup across every router stays in AddEIP.
- Post-write waitForFlowsHV barrier is now driven by the manager via the
FlowsBarrier option wired in services/vpcd (slice 4a).
- DeleteEIP returns nil on ErrNATNotFound, matching the handler's
fire-and-forget "already absent = success" semantic.
The vpcd "errors" import drops out with the inline error-comparison.
Slice 4 of mulga-siv-129, sub-step c.
* refactor(vpcd): migrate IGW subscribers to external.IGWManager (partial)
handleIGWAttach now branches at decode time:
- Centralised NAT + DHCP-sourced pool keeps using attachIGWLegacy because
the gateway-LRP IP comes from vpc.dhcp.acquire, and that DHCP client is
the vpcd-local manager that bead mulga-siv-125.3.3 will retire.
- All other shapes (distributed NAT, static-range pool, no pool)
delegate to external.IGWManager.AttachIGW. The manager mirrors every
step the legacy handler ran — external switch + localnet port +
gateway LRP + switch-side router peer + default route + chassis HA +
waitForFlowsHV tail barrier — so observable behaviour is unchanged.
handleIGWDetach drops its inline OVN walk entirely; both paths exit
through external.IGWManager.DetachIGW followed by releaseGatewayLRPLease.
The lease release stays in vpcd because the upstream client lives in
services/vpcd/dhcp; calling it after every detach is safe because the
upstream server expires unknown leases on its own.
Lazy-constructed IGWManager added on TopologyHandler (igwmOnce). Pool,
allocator (StaticRangeAllocator), chassis, NAT mode, NAT manager, route
manager, and FlowsBarrier are all derived from the existing
TopologyHandler state.
The DHCP-coupled legacy branch retains its full inline OVN logic; it
will disappear in slice-after-125.3.3 follow-on. Slice 4 of
mulga-siv-129, sub-step d.
* feat(ec2/instance): honor AssociatePublicIpAddress per-NIC override
RunInstances --network-interfaces "AssociatePublicIpAddress=true|false"
was silently ignored, only subnet.MapPublicIpOnLaunch gated public IP
assignment. AWS contract is per-NIC override wins.
Extend TestPrepareRunInstances_PublicIPAutoAssigned to table-driven
3 cases: subnet true no-override, subnet false override true, subnet
true override false.
* refactor(config): drop upstream-DHCP config surface (static WAN pool only)
Hard-error in LoadConfig on legacy keys: [network] external_dhcp,
[nodes.<n>.vpcd] dhcp_bind_bridge, and [[network.external_pools]]
source = "dhcp". Add pool range validation (must fit inside the
gateway/prefix_len CIDR; per-cluster non-overlap).
Strip ExternalDHCP from config.NetworkConfig + formation.NetworkConfig
+ admin.ConfigSettings, and DhcpBindBridge from config.VPCDConfig +
admin.ConfigSettings + vpcd.Config. vpcd.ExternalPoolConfig.IsDHCP()
short-circuits to false; the field and downstream DHCP branches in
external_ipam / topology.go remain for Slice C to delete together with
dhcp_manager.
cmd/spinifex/cmd/admin.go: drop useExternalDHCP path and the
detectedDhcpBindBridge helper; require --external-pool when
--external-mode=pool; auto-detect default no longer infers source=dhcp.
service.go's wan_bridge legacy-key rejection now advises removing the
key entirely (vpcd auto-detects br-wan), not renaming to
dhcp_bind_bridge.
Two topology_test cases that exercise the DHCP gateway-LRP path are
skipped — Slice C deletes attachIGWLegacy and these tests together.
* refactor(vpcd): delete dhcp_manager + DHCP gateway-LRP path (Slice C)
Removes upstream-DHCP-client gateway-LRP allocation in favor of static
WAN-pool allocation (StaticRangeAllocator). Part of bead mulga-siv-125.3.3.
- delete services/vpcd/dhcp_manager.go + dhcp/ subpackage (~865 lines)
- delete allocateGatewayLRPIPViaDHCP + releaseGatewayLRPLease paths
- drop ExternalPoolConfig.Source + DhcpBindBridge + IsDHCP() fields
- drop ExternalIPAllocation lease fields (Expiry/ServerID/HWAddr) and
ExternalIPAMRecord.Source
- drop NewExternalIPAM nc *nats.Conn argument; update all 4 call sites
- rename verifyBridgeMode local dhcpBindBridge -> wanBridge; update error
strings and tests
Slice D (delete attachIGWLegacy + dispatcher) intentionally deferred.
* refactor(vpcd): delete attachIGWLegacy + IsDHCP stub (Slice D)
Closes mulga-siv-129 slice 4d. Part of bead mulga-siv-125.3.3.
handleIGWAttach now unconditionally delegates to external.IGWManager.
attachIGWLegacy (~207 lines, ~5 OVN write steps + chassis scheduling)
deleted along with its dispatcher branch in handleIGWAttach. The
ExternalPoolConfig.IsDHCP() dead stub is gone.
No code paths or NATS subjects regress: the static-pool / distributed
branches were already going through external.IGWManager since Slice 4d
(6719dc3).
* feat(ipam): Purpose field on every IPAM record (Slice B)
Part of bead mulga-siv-125.3.3.
Adds a Purpose enum (handlers/ec2/vpc/purpose.go) tagging every IP
allocation with its semantic role (eni-primary, eni-public, eip,
igw-lrp, natgw-external, …) so multi-VPC clusters can reclaim/audit
by (owner, purpose).
External IPAM (spinifex-external-ipam, v1→v2):
- ExternalIPAllocation.Type → .Purpose; legacy values mapped via
LegacyExternalTypeToPurpose (gateway→igw-lrp, auto_assign→eni-public,
elastic_ip→eip). Unknown legacy values warn and become "unknown".
- AllocateIP / AllocateFromPool / allocateFromPool param rename
allocType→purpose. All call sites pass Purpose* constants.
- migrate/002_external_ipam_purpose.go: idempotent CAS-retry migration
with v1/v2-union record shape so partial migrations resume cleanly.
Internal IPAM (spinifex-vpc-ipam, v1→v2):
- IPAMRecord.Allocated []string → []IPEntry{IP, Purpose, OwnerID}.
- AllocateIP gains purpose+ownerID args; ReleaseIP unchanged (still
keyed on IP).
- AllocatedIPs returns []IPEntry.
- eni.go reorders eniId generation before AllocateIP so the owning
resource ID lands on the new entry.
- migrate/003_ipam_purpose.go: cross-bucket backfill that reads the
ENI bucket via the new RunKVWithJetStream path. Orphaned IPs
(no owning ENI) warn and become Purpose=unknown.
Migrate framework:
- KVContext gains an optional JetStream field; new
Registry.RunKVWithJetStream method threads it through. RunKV
signature unchanged — all other callers are unaffected.
Tests:
- ipam_test.go: AllocateIP signature updates; AllocatedIPs now asserts
IP / Purpose / OwnerID on the returned entries.
- external_ipam_test.go: replaces "auto_assign"/"elastic_ip" literals
with PurposeENIPublic/PurposeEIP and asserts alloc.Purpose.
- migrate/002 + 003 tests: schema-detect, idempotent rerun,
no-JetStream fallback, empty bucket.
* refactor(daemon): tombstone obsolete spinifex-dhcp-leases KV bucket
The dhcp_manager removed in Phase 2.3 (bead mulga-siv-125.3.3) owned the
spinifex-dhcp-leases KV bucket. The migrate framework only handles
per-bucket schema version bumps, not bucket deletion, so the obsolete
bucket would linger on upgraded clusters.
Add a one-shot tombstone after initJetStream that deletes the bucket if
present. DeleteKVBucketIfExists treats ErrBucketNotFound and
ErrStreamNotFound as success, so the call is idempotent across restarts.
* feat(config): add network.ipsec_enabled cluster flag (default true)
Add IPSecEnabled to NetworkConfig in both the cluster TOML schema and
the formation handshake payload. Default true so AWS-parity intra-AZ
encryption is on by default. Operators can flip false in
[network] for trusted single-rack lab topologies.
Phase 2.5 (bead mulga-siv-125.3.5) consumes the flag from admin init
(strongSwan provisioning), formation server (propagates to joiners),
and daemon startup (ovs-vsctl ipsec_encapsulation toggle).
* feat(admin): IPsec peer cert generator signed by cluster CA
Add GenerateIPSecPeerCert + IPSecCertPaths in spinifex/admin so admin
init (and the future admin add-node path) can issue per-node IKEv2 peer
credentials signed by the existing cluster CA. The cert carries
id-kp-ipsecIKE (RFC 4945) so strongSwan accepts it as an IKEv2 peer
credential when ovs-monitor-ipsec materialises tunnel configs.
Cluster CA is reused directly — no intermediate strongSwan PKI today;
when intermediate CAs land that becomes a separate workstream.
The cert/key pair lives under {configDir}/ipsec/peer.{pem,key}; the
daemon points OVS Open_vSwitch.other_config (certificate, private_key,
ca_cert) at these paths when network.ipsec_enabled is true.
* feat(admin): wire IPsec peer cert into admin init + join paths
Add --ipsec bool flag (default true) on adminInitCmd. Single-node init
and the multi-node leader path both emit an IPsec peer cert after the
existing CA+server-cert generation. The multi-node formation server
propagates the IPSecEnabled flag to joining nodes via NetworkConfig;
admin join honours it and emits its own peer cert during the same
cert-issuance step that already produces the server cert.
spinifex.toml template gains an unconditional [network] block carrying
ipsec_enabled so the value is visible to operators on every install
regardless of whether external networking is configured. The daemon
reads it via cluster config (NetworkConfig.IPSecEnabled).
Bead mulga-siv-125.3.5.
* build: install openvswitch-ipsec for OVN native IPsec
openvswitch-ipsec provides ovs-monitor-ipsec, which reads the
Open_vSwitch.other_config cert/key/CA pointers and materialises
strongSwan configs for every Geneve tunnel managed by ovn-controller.
Pulls in strongswan + strongswan-pki as transitive deps.
Required by bead mulga-siv-125.3.5.
* feat(daemon): enable OVN native IPsec on intra-AZ Geneve at startup
When network.ipsec_enabled is true (default), the daemon points the
local OVS DB other_config at the per-node IPsec peer cert that admin
init / admin join provisioned (configDir/ipsec/peer.{pem,key} + the
cluster CA at configDir/ca.pem), then sets
other_config:ipsec_encapsulation=true.
ovs-monitor-ipsec (shipped by the openvswitch-ipsec package) consumes
those pointers and materialises strongSwan configs for every Geneve
tunnel ovn-controller programs — including OVN-IC inter-AZ tunnels
when federation lands in Phase 3a.
Missing cert files log a warning and leave Geneve plaintext; operators
upgrading an older cluster reissue certs via spx admin to close the
gap.
Bead mulga-siv-125.3.5.
* test(e2e): multinode IPsec wiring + xfrm SAs + ESP capture
TestMultinodeIPSec asserts:
- OVS DB on every node carries certificate/private_key/ca_cert pointers
and other_config:ipsec_encapsulation=true (what daemon.enableOVNIPSec
writes).
- ip xfrm state on every node has at least one AEAD SA in AES-GCM mode
(rfc4106(gcm(aes)) or gcm(aes)) within 90s.
- tcpdump catches ESP (ip proto 50) on the underlay (best-effort —
logged only; geneve may be idle during the 5s capture window).
Skips cleanly when the cluster was bootstrapped with --ipsec=false
(no ipsec_encapsulation key on first node).
Read-only over SSH; runs in the parallel bucket alongside the other
non-mutating multinode tests.
* fix(igw): write attachment State="attached" for AWS parity
Reconciler intent loader (network/reconcile/intent.go::loadIGWs) filters
IGW records with `State == "attached"`, but AttachInternetGateway was
writing `State = "available"`. The reconciler therefore never picked up
any attached IGW — `intent_igws=0` on every pass — so the OVN external
switch + gateway LRP + WAN bridge wiring built by IGWManager.AttachIGW
never ran for default-VPC IGWs (whose live `vpc.igw-attach` publish at
daemon bootstrap is lost — vpcd subscriber not yet wired).
Net effect on single-node e2e: every SSH probe to 192.168.x public IPs
timed out (TestSSHProbe, TestModifyInstanceAttribute, TestRebootInstance,
TestNATGateway, TestSGToSGDatapath) because SNAT/DNAT rules on the VPC
router had no path to the external network.
Match AWS InternetGatewayAttachment.State semantics:
- Attach: State="attached"
- Detach: State="detached"
Detach value is internal — recordToEC2 gates Attachments on VpcId != "",
so the detached State is never surfaced to API consumers.
Bead: mulga-siv-131
* fix(reconcile): call AttachIGW when IGW topology missing
applyIGWs previously deferred to the NATS handler when an intent IGW
had no external switch in OVN. When the vpc.igw-attach event is lost
(bootstrap-time default-VPC IGW published before vpcd subscriber is
ready), deferral never resolves and VM public-IP traffic dies.
Call IGWManager.AttachIGW directly — it is idempotent (short-circuits
when the external switch already exists), so the reconciler converges
default-VPC IGWs on first pass without duplicating work on subsequent
passes.
Closes mulga-siv-132.
* fix(test): align NewExternalIPAM call with 2-arg signature
Rebase onto dev pulled in daemon_system_instance_test.go which used
the legacy 3-arg NewExternalIPAM(jsNC, js, pools). The feature branch
already narrowed the signature to (js, pools) via the upstream-DHCP
config drop; align the test with the current signature.
* fix(vpcd): EnsureSubnet must EnsureVPC first to avoid NATS race
vpc.create-vpc and vpc.create-subnet are independent NATS events with
no inter-event ordering guarantee. Tenant bootstrap on iam.account.created
publishes both within <1ms, so the subnet handler's
CreateLogicalRouterPort can race the parent VPC's CreateLogicalRouter and
fail with "logical router not found". The cascade fails CreateNetworkInterface
which fails RunInstances which surfaces as InsufficientInstanceCapacity to
the SDK — masking the underlying topology race.
handleSubnetCreate now calls EnsureVPC(VPCID-only) before EnsureSubnet.
EnsureVPC short-circuits on existing LR (idempotent), so the race-winning
vpc.create-vpc handler's full CIDR/VNI ExternalIDs are preserved; if subnet
wins the race the LR is created with bare ExternalIDs (cosmetic) and the
follow-on vpc.create-vpc short-circuits.
Adds TestIntegration_SubnetBeforeVPC regression covering both orderings.
Closes mulga-siv-133.
* feat(ansible): require --external-pool, drop NAT, forward EXTRA_VARS
spx admin init now rejects --external-mode=nat and demands --external-pool
when mode=pool. The ansible init role silently skipped pool flags when the
range was unset, so installs fell over inside the spx validator. Make
targets also swallowed `-e` (make's env-override flag), so operators could
not override pool start/end without editing defaults.yml.
- vars/defaults.yml: rename nat_gateway_ip → external_gateway_ip; add
spinifex_external_no_external opt-out; document the spx flag contract.
- roles/init/tasks/main.yml: drop NAT branch; assert pool range upfront
with a fail_msg pointing at EXTRA_VARS / defaults.yml; emit gateway,
prefix-len, gateway-ip only when explicitly set (let spx auto-detect);
honour --no-external.
- playbooks/dev-reset.yml: rename captured var; coerce legacy
external_mode=nat to pool with a warn; tighten gateway regex with \b.
- Makefile: ansible-dev-* targets now forward EXTRA_VARS="key=val ..."
through --extra-vars so `make` callers can override without editing
defaults.yml.
- scripts/ansible/README.md: document pool requirement, both invocation
styles (direct -e vs make EXTRA_VARS), NAT removal.
* fix(ipsec): guard ipsec_encapsulation on ovs-monitor-ipsec liveness
enableOVNIPSec set OVS cert pointers + ipsec_encapsulation=true without
ensuring ovs-monitor-ipsec.service was running. Without that daemon, OVS
holds the cert pointers unused, strongSwan never gets IKE config, and
tunnel traffic ovn-controller marks for encryption gets silently
dropped. TestMultinodeIPSec fails on `node1 xfrm has no AEAD SAs`.
- systemd: spinifex-daemon Wants=/After= ovs-monitor-ipsec.service
- daemon: ensureOVSMonitorIPSecActive — systemctl enable --now +
is-active poll; refuses to flip ipsec_encapsulation if daemon is dead
- daemon: single-node short-circuit — no peers means no tunnels means
the flag is at best inert and at worst load-bearing-silent-drop
- ci: capture-net-diag.sh + workflow steps to snapshot OVS/OVN/xfrm
state on failure, fan out across all 3 multinode nodes
Refs: mulga-siv-136
* test(e2e): sticky-skip downstream SSH tests when probe fails
When TestSSHProbe times out (3 min, typically because of mulga-siv-134
datapath bug), four downstream tests each re-probe SSH on the same
broken datapath and time out identically — single-node job runs ~24 min
instead of ~12 min, with the same error logged five times.
First waitForSSHReady timeout now sets a package-level atomic flag;
subsequent waitForSSHReady / trySSHReady / requireSSHHealthy calls skip
immediately. Preserves the original red signal on TestSSHProbe and on
any test that genuinely needed SSH (Skip ≠ Pass). Bool variant
(trySSHReady) consults but does not set the flag — its caller already
treats False as advisory warning, not fatal.
TestSGToSGDatapath has its own EventuallyErr SSH wait (not via
waitForSSHReady) and needs an explicit requireSSHHealthy(t) gate at
the top of runSGToSGDatapath. TestNATGateway already calls
waitForSSHReady first so the gate fires naturally.
Refs: mulga-siv-137
* fix(ipsec): use Debian unit name openvswitch-ipsec.service
CI artifact shows `Unit ovs-monitor-ipsec.service could not be found`
on every node. Debian's openvswitch-ipsec package ships the unit as
openvswitch-ipsec.service (the binary inside is ovs-monitor-ipsec, but
systemd's name follows the package). The previous commit guarded on
the wrong unit name, so:
- spinifex-daemon.service Wants=/After= silently no-op'd (missing unit)
- enableOVNIPSec's `systemctl enable --now ovs-monitor-ipsec.service`
failed, daemon logged a warning and continued without flipping
ipsec_encapsulation=true
- TestMultinodeIPSec skipped with `node1 reports ""` because OVS DB
never received the cert pointers
Rename across all 5 spots (daemon, systemd unit, test, CI diag script,
workflow journal capture) to match the Debian package layout.
Note: the package itself still must be present on the CI VM image —
that's tracked in the mulga repo (image-builder provision.sh +
bootstrap-install.sh shim).
Refs: mulga-siv-136
* fix(e2e): authorize default SG tcp/22 + icmp in single-node fixture
admin init leaves the default SG closed (AWS parity); the Go single suite
never ported the Phase-5 SG-ingress block that run-e2e.sh has had since day
one, so every test runner SSH/ping probe hits the OVN port-group ACL drop.
- ec2helpers: add AuthorizeICMPIngress + EnsureDefaultSGOpen helpers.
- single/main_test: open default SG once per process in pkgFixOnce.Do.
- witness: extend ensureDefaultSGSSHIngress to authorize ICMP too.
- run-e2e.sh: append matching ICMP authorize next to tcp/22 block.
mulga-siv-134
* fix(network/reconcile): key DHCP_Options lookup by subnet_id not CIDR
Multi-tenant default VPCs share the same subnet CIDR (172.31.0.0/20) per
account. The existing FindDHCPOptionsByCIDR gate at apply.go:107 found
the first account's DHCP_Options row and skipped creation for every
subsequent subnet with the same CIDR. Those subnets' ports then attached
nothing (FindDHCPOptionsByExternalID misses on subnet_id), VMs got no
DHCPv4 lease, and SSH/ICMP into them silently failed even though SG
ACLs and NAT were correct.
Switch the gate to FindDHCPOptionsByExternalID keyed on spinifex:subnet_id
so each subnet owns its own row, matching the attach lookup at line 203.
Verified: account 000000000001 default VPC RunInstances now produces an
LSP with dhcpv4_options populated, VM cloud-init reports a 172.31.0.5/20
lease + default route via 172.31.0.1, and ping + ssh TCP handshake to
the EIP succeed end-to-end. Stale DHCP_Options row for the prior dead
subnet was wiped manually; reconciler recreated the correct row for the
live subnet on its next pass.
* fix(ipsec): enable openvswitch-ipsec at provision time, drop daemon write
Provisioning now runs `systemctl enable --now openvswitch-ipsec.service`
alongside openvswitch-switch and ovn-controller. The spinifex-daemon
process drops its `enable --now` shell-out and only polls is-active.
Sudoers gains one read-only line:
spinifex-daemon ALL=(root) NOPASSWD: /usr/bin/systemctl is-active openvswitch-ipsec.service
This mirrors the existing vpcd ovn-controller is-active precedent and
removes the daemon's need for systemd-write capability. Aligns with
least-privilege service account pattern: provision enables services,
daemon only reads status.
If the unit is inactive at runtime, the daemon refuses to flip
ipsec_encapsulation=true (silent-drop guard from mulga-siv-136) and
surfaces a clear error pointing at scripts/setup-ovn.sh.
* chore: unblock preflight on feat/network-layer-redesign
Two pre-existing failures blocked preflight on the branch:
- govulncheck flagged golang.org/x/net@v0.54.0 (GO-2026-5026, ASCII Punycode
rejection bug in idna.ToASCII); reached via utils.DownloadFileWithProgress.
Bump to v0.55.0; go mod tidy carries x/sys to v0.45.0.
- TestTopologyHandler_SubnetCreate_RouterPortFails asserted subnet create
fails when the parent VPC router is missing. That contract changed in
aed7b2e (mulga-siv-133), which made handleSubnetCreate call EnsureVPC
idempotently before EnsureSubnet to avoid a NATS publish-order race.
Delete the now-impossible failure case.
* refactor(network/ovn): relocate nbdb model package from services/vpcd
Phase 4 / mulga-siv-125.6 Slice A. Pure import rename, no logic change.
The OVN Northbound DB model (LogicalSwitch, LogicalRouter, DHCPOptions,
etc.) lived under services/vpcd/nbdb/ because that was the original home
of every OVN caller. After Phase 1+2 the L1 client (network/ovn) and all
L3/L5 managers (network/policy, network/external, network/topology,
network/reconcile) became the primary callers — 8 importers in network/*
vs 2 in services/vpcd/. The directory move puts the schema model where
its consumers are, and unblocks deleting services/vpcd/ entirely once
the subscriber/launcher relocations land (Slices B–D).
* refactor(vpcd): delete dead retrofit/reconcileIGW helpers + bridge-mode bleed
Phase 4 / mulga-siv-125.6 Slice B.
Production path no longer touches any of:
- RetrofitAllGatewayPortNetworks / RetrofitAllExternalLocalnetOptions —
the 5-pass startup leftovers that network/reconcile (Phase 2.2) made
redundant. Nothing called them outside tests.
- reconcileIGW + its allocateGatewayLRPIP / gwLrpRange helpers + the
ipv4ToUint32 / uint32ToIPv4 utilities — superseded by
external.IGWManager (Phase 2.4) which carries its own pool logic.
- ensureLocalnetOptions / ensureGatewayPortNetworks /
expectedGatewayPortNetwork — read-before-write retrofit shims for the
same code paths.
- generateMAC — only callers were reconcileIGW + its tests.
useCentralizedNAT (the bridge-mode bleed flagged in plan §2.2) is
replaced by a policy.NATMode value injected via WithNATMode(...). The
mode is resolved once in vpcd.go from host.UplinkMode and threaded into
both the topology-handler lazy managers and the reconciler stack, so
"is this veth?" is no longer a question topology.go can ask. The
TopologyHandler defaults to NATModeDistributed when WithNATMode is
omitted, matching the previous default.
Drops services/vpcd/topology.go from 1220 LOC to 693, deletes ~700 LOC
of obsolete tests (TestGenerateMAC, TestEnsureLocalnetOptions_*,
TestRetrofitAllExternalLocalnetOptions_FixesClearedOptions,
TestReconcileIGW_*, TestGwLrpRange), and frees the nbdb + utils +
net imports in topology.go.
Stale "phase 4 migrates those onto the new managers" comment in vpcd.go
fixed in passing — the subscribers were migrated in Phase 2.6
(mulga-siv-129); only the file-relocation parts of Phase 4 remain.
Stale ensureLocalnetOptions / ensureGatewayPortNetworks references in
network/ovn/{live.go,mock/mock.go} doc comments also cleaned up.
* refactor(network): move NATS VPC subscribers to network/subscribers
Phase 4 Slice C (mulga-siv-125.6). Promotes the NATS event handlers out of
services/vpcd into a new spinifex/network/subscribers package. Subscriber
holds the topology/policy/external managers directly — the lazy *Once
adapter shim (topology_manager.go) is deleted along with vpcd's local
event-struct copies.
launchService now constructs topology.LiveManager eagerly with a DNS
callback derived from external pools (preserving the unscoped-pool lookup
old TopologyHandler.lmDNSServer did), builds every manager up front, then
hands them to subscribers.New + Subscribe. reconcile.New receives
topology.Manager directly instead of through the adapter.
Bridge-mode constants and waitForFlowsHV move into vpcd.go alongside the
launcher; they were left behind as topology.go's last responsibilities.
Handler-layer tests (topology_test.go, security_test.go, integration_test.go)
are dropped: substance is covered by network/topology, network/policy,
network/external manager tests. subscribers/subscribers_test.go adds
focused coverage: missing-dep rejection, topic registration smoke, and
bad-JSON envelope per topic.
services/vpcd is now down to the launcher + the ovn/mock compat aliases
(removed in Slice D when the launcher promotes to cmd/vpcd/).
* refactor(vpcd): relocate services/vpcd → spinifex/vpcd; drop dead aliases
Phase 4 Slice D (mulga-siv-125.6). Final cleanup of the network-redesign
package layout. services/vpcd/ disappears entirely; the launcher remains
an in-process Service impl, just rehomed under spinifex/vpcd/ (no separate
binary — spx vpcd start stays the cobra entrypoint).
- Delete services/vpcd/mock_ovn.go + ovn.go (compat aliases now dead post
Slice C; only callsite was launchService's NewLiveOVNClient, inlined to
ovn.NewLiveClient).
- git-rename services/vpcd/vpcd.go + vpcd_test.go to spinifex/vpcd/.
- Retarget three importers (spinifex/service/service.go +
service_test.go, cmd/spinifex/cmd/service.go) to the new path.
- Fix stale services/vpcd / TopologyHandler comments in network/reconcile/
+ network/external/types.go + spinifex/vpcd/vpcd.go.
- service-interfaces.yaml vpcd path updated.
- Plan doc §13/§14/§16 + §9.2/§19 updated: drop cmd/vpcd/main.go (was
speculative; never built) in favour of in-process Service impl at
spinifex/vpcd/vpcd.go. Plan §16 Created list now lists subscribers/.
Phase 4 plan items 1+2 (remove services/vpcd, entrypoint location)
satisfied. Items 3+4 remain blocked on mulga-siv-125.4 + 125.5 (bundled
join token, plan archive).
* fix(ovn): install openvswitch-ipsec + strongswan explicitly
ovs-monitor-ipsec.service fails to start without an IKE daemon. The
openvswitch-ipsec package only Recommends strongswan, so any apt
config that drops Recommends (or a base image with the recommends
already skipped) leaves the service in a Restart loop with 'IKE
daemon is not installed'. Hoist both packages into setup-ovn.sh so
provisioning is the single source of truth — the bootstrap shim in
mulga is no longer needed.
* test(network): pin ADR-0006 invariants to executable suite
Add static-analysis + behavioural tests that bind every safety and
liveness clause in ADR-0006 (S1-S9, L1-L3) to a named Go test. Failure
output quotes the ADR clause verbatim and points at the offending
file:line so future debugging sessions can't quietly bypass a layer
contract to make a test green.
Static-analysis suite (spinifex/network/invariants/):
- S1 layers_test.go: parses `go list -json`; rejects upward layer
imports and layers pulling from cross-cutter packages
- S2 host_containment_test.go: AST-detects ovs-vsctl / ovn-nbctl /
ovs-ofctl shell-out outside network/host/
- S3 nat_mode_test.go: rejects .UplinkMode() / NATModeFromUplinkMode()
calls outside the init-time resolver locations
- S4 names_test.go: AST-detects OVN object name construction (concat,
fmt.Sprintf, strings.Join) outside topology/names.go
- S5 policy_no_create_test.go: rejects CreateLogical{Switch,Router,...}
and DeleteLogical{...} calls inside network/policy/
- S6 federation_isolation_test.go: rejects TransitSwitch /
TransitRouterPort references and ovn-ic literals outside
network/federation/ (pre-protects the layer before it lands)
- S8 ipsec_test.go: rejects strongswan / xfrm / IKEv2 / libreswan /
openswan imports anywhere in network/
- S9 reconciler_singleton_test.go: rejects Reconcile…Loop and
Reconcile…Pass<n> function names anywhere in network/
- S7 nats_subjects_test.go: scans for `vpc.<verb>` subjects missing
the {azID} segment; calls t.Skip with the gap report so every test
run prints the work-to-do without breaking CI. Flip to t.Fatalf
when the cluster-wide subject rename ships.
Behavioural suite (spinifex/network/reconcile/liveness_test.go):
- L1 reconciler converges: scaled-intent (N=1,3,8) double-Reconcile
property test; asserts second pass is a no-op
- L2 drift bound: structural check that DriftInterval is positive and
≤30 min support window; full fake-clock behavioural test deferred
- L3 federation re-enables: t.Skip stub pointing at the missing
network/federation/ package
Fixes the one production S4 violation the new walker found:
network/external/igw.go was constructing "gw-port-"+vpcID inline.
Adds topology.GatewaySwitchPort(vpcID) helper and retargets both
AttachIGW and DetachIGW callsites through L2.
Plan + per-clause mechanic + phasing in
docs/development/improvements/network-invariants.md.
Refs: mulga-siv-140
* fix(ovn): install strongswan-charon (not strongswan metapackage)
The strongswan metapackage's '\''charon-systemd | strongswan-charon'\''
alternative resolves to charon-systemd by default, which ships
/usr/sbin/charon-systemd but not the legacy /usr/sbin/ipsec helper or
/usr/lib/ipsec/charon binary that ovs-monitor-ipsec --ike-daemon=strongswan
shells out to. Install strongswan-charon explicitly to pull
strongswan-starter (legacy ipsec controller) + the matching charon binary,
matching what openvswitch-ipsec.service expects.
Also reset-failed + restart openvswitch-ipsec.service after enable to
recover from the deb post-install starting the service before its
dependencies finished unpacking.
* chore: add CODEOWNERS — joshsiv-mulga gates network/ + vpcd/
Fall-through reviewer for any change is the maintainer.
network/ and vpcd/ are explicitly listed so the matching branch
protection rule ("Require review from Code Owners") routes those
PRs to joshsiv-mulga. Those trees are governed by ADR-0006
invariant tests (spinifex/network/invariants/); CODEOWNERS adds a
human review gate on top of the mechanical fence.
Refs: mulga-siv-141
* chore: CODEOWNERS fallthrough → @mulgadc/mulgadc-engineers team
Fallthrough reviewer should be the team, not a single maintainer, so
any engineer can approve PRs outside the explicitly-gated paths.
network/ and vpcd/ keep joshsiv-mulga as the required reviewer per
ADR-0006 invariant ownership.
Refs: mulga-siv-141
* feat(ci): add test_run workflow input for -test.run filter
Pass an optional regex through to each suite's '\''go test -test.run'\'' so
dispatches can target a single test (e.g. '\''^TestMultinodeIPSec$'\'') without
running the full multinode suite. Threaded through single, multinode, and
reboot suite invocations; empty input preserves current behaviour.
* fix(daemon): flip NB_Global.ipsec=true on management node
Setting Open_vSwitch.other_config:ipsec_encapsulation=true on each
chassis is necessary but not sufficient for OVN native IPsec — without
NB_Global.ipsec=true, ovn-controller never adds options:remote_name to
the Geneve tunnel ports, so ovs-monitor-ipsec sees no IPsec-enabled
tunnels and the kernel xfrm tables stay empty. After enabling
ipsec_encapsulation locally, also write NB_Global.ipsec=true when the
local ovnnb_db socket is present (management node only); workers skip
silently — one writer is enough for a cluster-wide flag.
Adds a scoped sudoers entry pinning the daemon to that exact ovn-nbctl
invocation so the grant can't be reused for other NB mutations.
* fix(ovn): grant charon AppArmor read on /etc/spinifex/ipsec
The strongswan-charon profile only allows reads from /etc/ipsec.*,
/etc/strongswan.*, and a small fixed allowlist. ovs-monitor-ipsec writes
the per-tunnel strongSwan config with absolute paths to our peer cert +
key under /etc/spinifex/ipsec/, so charon hits 'Permission denied'
loading them (it runs as uid 0 but AppArmor still blocks) and no xfrm
SAs come up. Drop a read grant for /etc/spinifex/ipsec/** into the
profile's local override hook + reload.
* fix(ovn): pin OVS system-id to hostname for IPsec cert identity match
ovs-monitor-ipsec uses chassis-id as the IKEv2 `@<name>` peer identity in
the strongSwan config it generates. Our per-node IPsec peer cert
(admin.GenerateIPSecPeerCert) carries the node name as CN + dnsName SAN.
With chassis-id left at the package-generated UUID, charon validates
`@<UUID>` against a cert with dnsName=<hostname> — mismatch returns
AUTHENTICATION_FAILED and no xfrm SAs are negotiated.
Pin system-id to hostname (which equals the node name on Mulga clusters)
by writing both the on-disk source of truth (/etc/openvswitch/system-id.conf,
read by ovs-ctl on boot) and the running OVS DB value. Writing only one
would drift on the next reboot — the original "do not override" comment
was right about that hazard, just wrong about the resolution.
Cross-link the invariant in admin/ipsec.go: changing the cert identity
without updating setup-ovn.sh (or vice versa) breaks IKE silently.
Refs: mulga-siv-136
* fix(ovn): take --node-name flag so chassis-id matches cluster identity
CI hostnames are bead-themed (env20-casuarina) and don't match the
cluster node name (node1/node2) baked into the IPsec peer cert dnsName
SAN at `spx admin init/join --node NAME`. Pinning system-id to
`hostname -s` produced env20-casuarina; charon then validated
`@node1` against dnsName=env20-casuarina and IKE failed with
AUTHENTICATION_FAILED.
Add a --node-name=NAME flag to setup-ovn.sh, default to hostname for
manual/dev invocations, and update the ansible install role to pass
{{ spinifex_node_name }}. The matching bootstrap-install.sh change
lives in mulga.
Refs: mulga-siv-136
* refactor(daemon): relocate L0 helpers to network/host + widen ADR-0006 S2
Phase 4 Slice E of mulga-siv-125.6. Cleans up spinifex/daemon/network.go
per the audit and widens the S2 invariant to forbid OVS/OVN shell-outs
in spinifex/daemon/ (daemon is an L0 consumer; S2 applies).
Moved to network/host/:
- OVSNetworkPlumber + SetupTap/CleanupTap → host.OVSPlumber (host.NewOVSPlumber)
- GetBridgeIPv4 → host.GetBridgeIPv4
- OVNHealthStatus + CheckOVNHealth → host.OVNHealth + host.HealthStatus
- daemon/ipsec.go OVS/OVN ops → host.SetIPSecCertPaths, host.EnableIPSecEncapsulation, host.SetNBGlobalIPSec
Moved to utils:
- sudoCommand → utils.SudoCommand + utils.SetSudoCommandForTest test seam
(cross-package reassign would trip the reassign linter; helper keeps
the indirection inside utils).
Moved to vm:
- generateMgmtMAC → vm.GenerateMgmtMAC (pairs with vm.GenerateDevMAC)
Deleted (dead):
- SetupComputeNode (replaced by network/host/bridges.go:ensureBridges +
scripts/setup-ovn.sh:572-579)
- EnsureDataRoute
- findInterfaceByIP
- daemon/network.go + dead tests in daemon/network_test.go (duplicates of
vm package tests retained; mgmt MAC test moved to vm/lifecycle_test.go)
S2 invariant (network/invariants/host_containment_test.go) now walks
both spinifex/network/ and spinifex/daemon/, skipping network/host/.
* fix(ovn): broaden AppArmor read grant to /etc/spinifex (covers CA cert)
The earlier override granted reads on /etc/spinifex/ipsec/** only, which
covers the peer cert + key but NOT the cluster CA at /etc/spinifex/ca.pem.
charon then can't load the CA, can't validate the peer cert chain, and
surfaces the failure as `no trusted RSA public key found for '<peer>'`
even though identity matching now succeeds — IKE_AUTH fails with
AUTHENTICATION_FAILED and no xfrm SAs negotiate.
Grant /etc/spinifex/** instead (covers ca.pem + ipsec/peer.{pem,key})
and always rewrite the override so re-runs replace the older narrower
rule.
Refs: mulga-siv-136
* fix(ipsec): symlink cluster CA into charon's default cacerts dir
ovs-monitor-ipsec writes per-conn `cacert=/etc/spinifex/ca.pem` references
in the strongSwan config, but strongSwan 6.0 only loads CAs declared in
standalone `ca` sections or files placed in /etc/ipsec.d/cacerts at
startup. Without this, charon authenticates the local side fine but
rejects every inbound peer cert as `no trusted RSA public key found for
'<peer>'`, AUTH_FAILED, and no xfrm SAs negotiate.
Do the symlink in admin.GenerateIPSecPeerCert (right after writing the
peer cert) — setup-ovn.sh runs before `spx admin init/join` creates
/etc/spinifex/ca.pem, so the wiring can't live there. Use a stage-and-
rename pattern so concurrent test invocations don't race on the symlink.
/etc/ipsec.d/cacerts is on charon's default AppArmor allowlist so no
profile change needed.
Refs: mulga-siv-136
* chore(setup-ovn): drop apt-get install_packages block
OVS/OVN/openvswitch-ipsec/strongswan-charon are baked into the CI gold
image via mulga's image-builder/scripts/provision.sh (commits 116bb05,
a9b3a9c). Keeping a runtime apt-get install path here pulled CI through
apt-cacher-ng on every bootstrap and tripped on intermittent "Message
has been manipulated" InRelease cache corruption (env21/env20 runs).
Drop the install block. Gold-image rebuild is the canonical place to
add new system packages. If a package is missing, downstream ovs/ovn
commands fail loudly during setup-ovn.sh's later steps — clearer signal
than silently re-installing through a flaky proxy.
setup.sh already supports INSTALL_SPINIFEX_SKIP_APT=1 (passed by mulga's
bootstrap-install.sh:237) so this removes the last runtime apt-get call
on the CI provisioning path.
* fix(ovn): gate system-id pin on explicit --node-name
setup-ovn.sh used to fall back to `hostname -s` when --node-name wasn't
passed, and then unconditionally wrote that value to
/etc/openvswitch/system-id.conf + ovs-vsctl external_ids:system-id.
On CI single-node, the gold image already registered a chassis with
name=<UUID>; setup-ovn.sh then rewrote system-id to the bead-themed
hostname (env10-banksia). vpcd's discoverChassis treats any chassis row
whose name != local system-id but hostname == local hostname as a stale
local chassis and skips it, so vpcd reported "no OVN chassis registered
in SBDB" and refused to start, cascading into the e2e + e2e-go failures
on run 26430839508.
Drop the hostname fallback. If --node-name is given, pin both the
on-disk file and the running OVS DB to it (callers that care about
IPsec cert identity matching still get it). If no --node-name is given,
leave the existing system-id alone — that's always safe whether the
value came from a gold-image UUID, bootstrap-install.sh writing nodeN
before this script runs, or a manual ansible value.
* fix(network/reconcile): skip orphan prune on startup; drift owns GC
The vpcd startup reconcile races against the daemon's EnsureDefaultVPC:
the daemon writes SG KV rows + publishes vpc.create-sg on each node in
parallel with vpcd starting up. The leader-gated reconcile can load
intent before any SG row is visible on its local JetStream replica
while subscribers on peer nodes are concurrently creating port groups
from the same vpc.create-sg events. A startup orphan-port-group sweep
then deletes managed sg_* port groups that intent hasn't caught up to.
CI run 26434124790 trace:
node2 reconcile/lock: elected holder=env20-bottlebrush
node2 reconcile: starting intent_sgs=0
node2 reconcile/apply: removed orphan port group pg=sg_161c70ce23e704a2a
node2 reconcile/apply: removed orphan port group pg=sg_a0d078650a3150391
node2 reconcile/apply: removed orphan port group pg=sg_54b564b374e9162bb
Every subsequent RunInstances → vpc.create-port failed with
"port group not found" for ~3 minutes until the 5-minute drift tick
recreated the port groups. Tests give up after 60s; six multinode
suites cascade-fail on InsufficientInstanceCapacity.
Split the reconciler into Reconcile (full prune, drift loop) and
ReconcileApplyOnly (no prune, startup). applySGs takes a pruneOrphans
bool. vpcd startup calls ReconcileApplyOnly; drift loop unchanged.
Legitimate orphans get garbage-collected on the next drift tick — at
worst 5 minutes of lingering rows, no production impact.
Add TestReconcile_ApplyOnlyKeepsOrphanPortGroup to lock the contract.
* fix(ipsec): trigger charon rereadcacerts after CA symlink install
strongSwan scans /etc/ipsec.d/cacerts only at charon startup. The unit
openvswitch-ipsec.service is enabled at provision time and starts before
admin init writes the cluster CA symlink, so charon ends up running with
zero CAs loaded. Every IKE_AUTH then fails with `no trusted RSA public
key found for '<peer>'` and no SAs negotiate — the failure mode exposed
by TestMultinodeIPSec.
Append a stroke rereadcacerts call at the end of
installCAIntoCharonTrustStore so an already-running charon picks up the
freshly-placed CA without a service restart. Failure is swallowed at
debug — a missing/stopped charon at admin-init time is expected and a
later startup will scan cacerts naturally.
* refactor(network/host): move IPSec orchestrator from daemon (ADR-0006 S8)
ADR-0006 S8: "IPSec is OVN-native only. No layer implements custom IKEv2,
XFRM rules, or direct strongSwan management. IPSec SA lifecycle is delegated
entirely to OVN native IPSec and is invisible above L0."
The enableOVNIPSec orchestrator and its ensureOVSMonitorIPSecActive helper
lived in daemon/ipsec.go but composed L0 primitives (SetIPSecCertPaths,
EnableIPSecEncapsulation, SetNBGlobalIPSec) that already lived in
network/host. The orchestrator now lives alongside its callees as
host.EnableOVNIPSec, taking configPath and clusterConfig from the daemon at
the call site. Helper and test-override package vars (systemctlActiveTimeout,
ovnNBSocketPath) move with it; tests are translated to call EnableOVNIPSec
directly rather than via the Daemon struct.
Closes mulga-siv-142
* refactor(daemon): use topology.Port for OVN port names (ADR-0006 S4)
ADR-0006 S4 forbids any layer above L2 from constructing OVN object
names. Three daemon sites formatted "port-" + ENIId inline; route them
through topology.Port instead.
Closes mulga-siv-143
* refactor: use topology.Port for OVN port names in handlers + vm (ADR-0006 S4)
Replace inline "port-"+eniID formatting across five sites (eip, instance x2,
vpc/eni, vm/network_helpers) with topology.Port. OVSIfaceID stays as a thin
wrapper to preserve the OVN binding contract comment and existing call sites.
Closes mulga-siv-144
* test(network/invariants): widen scan roots to daemon/handlers/vm/testutil/utils (ADR-0006)
Widen ADR-0006 invariant scan roots so drift in upstack consumers surfaces
in CI. Backlog of violations in widened roots was drained by
mulga-siv-142/-143/-144; this commit makes the wider scan permanent.
- S2 (host_containment): add vm/ alongside network/ + daemon/
- S3 (nat_mode): add daemon/ alongside network/ + vpcd/
- S4 (names): add daemon/, handlers/, vm/ alongside
network/ + vpcd/; add narrow AWS-ID exemption
(vpc-cidr-assoc-) so AWS resource IDs are not
misclassified as OVN object names
- S6 (federation): add daemon/, handlers/ alongside network/;
convert single-root walk to slice
- S7 (nats_subjects): add daemon/, handlers/, testutil/, utils/ —
stays t.Skip pending cluster-wide subject
rename; gap report now lists 58 subjects
CODEOWNERS: add /spinifex/network/invariants/ rule so widening or
narrowing the scan needs explicit owner review.
Closes mulga-siv-145
Refs: audit-adr-0006.md
* chore(e2e): disable IPsec in pseudo multi-node test
Pseudo multi-node runs three simulated nodes as unprivileged tf-user.
The charon trust-store install in 904b45e writes /etc/ipsec.d/cacerts,
which fails with EACCES under a non-root caller and aborts admin init.
Disable --ipsec for this suite — loopback-only nodes don't exercise
real Geneve encryption anyway. Real-multi-node + single-node e2e
(both run admin init via sudo) continue to cover the IPsec path.
* fix(ovn): atomic Ensure* primitives prevent duplicate logical entities
OVN NB has no unique constraint on Name. The previous
GetLogicalRouter→CreateLogicalRouter sequence in EnsureVPC was a TOCTOU
race: when vpc.create on one node and the defensive vpc.create-subnet
EnsureVPC call on another both observed absence, both inserted, yielding
two routers per VPC. The orphan held DNAT references blocking EIP
detach, and DNAT/flow installation landing on the orphan caused
TestMultinodeGuestSSH SSH timeouts.
Add EnsureLogicalRouter / EnsureLogicalSwitch / EnsurePortGroup to the
libovsdb client. Each builds a single OVSDB transaction combining a
wait-op (Until=equal, Rows=[]) gated on absence of a Name match with the
insert. The server serialises the wait+insert atomically; concurrent
callers see exactly one survivor and the rest fall through to a re-fetch
that returns the canonical row.
Switch all production callers (topology.EnsureVPC / EnsureSubnet /
EnsureSGPortGroup, external.AttachIGW, reconcile.applyVPCs /
applySubnets) to the Ensure* variants. Bare Create* remains available to
the L1 wrapper.
Add invariant test TestS4_OVNNamedCreatesUseEnsure (AST walk) to block
future reintroduction of the bare Create* call sites outside
network/ovn.
Mock implements Ensure* under its existing mutex so the concurrency
tests (50/30/30 goroutines for Router/Switch/PortGroup) prove a single
UUID survives.
Refs: mulga-siv-146
* fix(ovn): wait-op rows must be non-empty; flip to NotEqual predicate
The Ensure* primitives sent a wait-op with `Rows: []ovsdb.Row{}` plus
`Until: ==` (RFC 7047 §5.2.6 absence test). The libovsdb Operation
struct tags Rows as `json:"rows,omitempty"`, so the empty slice was
dropped on the wire and ovsdb-server rejected every transaction with
"Required 'rows' member is missing", breaking reconciler bootstrap and
every downstream create-port lookup ("logical switch X not found").
Flip the predicate to the equivalent `Until: !=` formulation with a
single-row projection containing the Name we want absent:
no row → projection [] != [{name}] → predicate holds
row exists → projection [{name}] == [{name}] → predicate fails
Same atomic guarantee, JSON now well-formed. Mock tests can't catch
this — only a real ovsdb-server validates the envelope.
* chore: remove generateMAC wrapper
* chore: remove all bead / plan references
* fix(network/policy): wait for chassis flows after AddNATGateway
AddEIP fires m.barrier() to block until ovn-northd compiles NB→SB and every
chassis installs the new dnat_and_snat flows; AddNATGateway returned right
after the NB commit. So `aws ec2 create-route … --nat-gateway-id` would
publish vpc.add-nat-gateway, the SNAT row would land in NB, and the API
call would succeed before the gateway chassis had programmed the SNAT flow
for the private subnet.
Phase 8d's 10×5s=50s ping budget races that flow-install latency on slow
CI hosts (last seen run 26455602615 single-node, identical flake on
unmodified dev run 26434711010 — pre-existing, not introduced by Ensure*).
Mirror AddEIP: call m.barrier() after AddNAT returns. Add unit test
TestNATManager_AddNATGateway_FlowsBarrier_Fires covering the new path.
* chore: remove more plan refs
* chore: remove GenerateIPSecPeerCertIfEnabled
* chore: remove dupliate subnetGatewayCIDR function
* chore(network): trim superfluous comments across network/ tree
Collapse multi-paragraph godocs to 1-2 sentences, drop WHAT-restatement
comments, remove section dividers grouping <10 items, and cut prose
narration in tests. ADR clause references, one-line WHY comments for
non-obvious OVN/OVSDB/AWS/NATS semantics, and exported-symbol godoc are
preserved. No behavioural changes.
Packages: invariants, topology, host, external, ovn, reconcile, policy,
subscribers.
* fix(igw): restore attachment State="available" for AWS parity
AWS InternetGatewayAttachment.State is "available" when an IGW is
attached to a VPC, per the AWS SDK doc ("the state is available when
attached to a VPC; otherwise, this value is not returned"). Commit
b7505f8 wrote "attached"/"detached" to align with the reconciler
filter, which inverted AWS parity.
Fix the producer to write "available" again and update the consumer:
- handlers/ec2/igw: AttachInternetGateway and DetachInternetGateway
both persist State="available"; IGWRecord comment narrowed.
- network/reconcile/intent.loadIGWs: filter accepts "available" (was
"attached"); attachment is identified by VpcId != "".
- tests updated: igw service_impl_test, routetable service_impl_test
fixture, reconcile intent_test fixture.
* Update spinifex.toml
* refactor(subscribers): route EIP/NATGW through L5 wrappers (ADR-0006 §S1)
Subscriber previously called policy.NATManager directly for EIP and NAT
Gateway operations, bypassing the L5 wrappers (EIPManager, NATGWManager)
that existed on paper. ADR-0006 §S1 requires every cross-layer call to
pass through the immediate lower neighbor's typed interface; IGW already
followed this pattern.
Wire EIPManager and NATGWManager into the Subscriber struct, construct
them in vpcd alongside IGWManager, and swap the four direct-L3 call
sites (AddEIP, DeleteEIP, AddNATGateway, DeleteNATGateway) onto the
L5 surface (AttachEIP, DetachEIP, AttachNATGateway, DetachNATGateway).
AttachEIP now fires the flows-barrier, giving callers the same
"reachable on return" guarantee IGW already had.
The reconciler still calls policy.NATManager directly; that's a
cross-cutter and a separate concern.
* test(reboot): port 8MB WAL pad workaround from bash run-reboot-e2e.sh
App VMs' user-data writes (unit file + index.html) sit in the on-host
WAL until WriteWALToChunk crosses the 4MB threshold. Host reboot before
that loses them — systemd then refuses to load the zero-byte unit and
the responder never starts. Mirror the bash driver's dd+sync pad so the
WAL flushes pre-reboot. Drop when the upstream WAL fix lands.
* test(e2e): dump OVN/OVS/SG state on Phase 8d NATGW failure
Phase 8d's existing failure path only prints lr-nat-list and
lr-route-list. CI artifacts are not uploaded when run-e2e.sh exits
nonzero, so debugging the NATGW regression requires inline diagnostics
in the journal. Expand the dump to include nb/sb show, chassis bindings,
port_group ACLs, br-int flows filtered to the NATGW IP/private subnet,
host routes, and the private VM's own ip/route/neigh state.
* fix(ovn,reconcile): scope NAT deletes to router; NATGW snat per associated subnet
Two compounding bugs caused Phase 8d NATGW E2E flake (~60% CI failure):
1. ovn.DeleteNAT/DeleteNATByExternalIP searched all NAT rows globally and
only mutated the named router afterwards. With AWS subnet CIDRs repeating
across VPCs, (type, logicalIP/externalIP) is not globally unique — the
delete picked a NAT row owned by a different router, the mutate was a
no-op, and the row delete failed with "remaining reference(s)", leaving
the original router's NAT orphaned. Both calls now resolve the LR first
and filter NAT lookups to lr.NAT UUIDs. Mock client mirrors the scoping.
2. reconcile.loadNATGWs used the NATGW's home (public) subnet CIDR as
SubnetCIDR, producing an snat for the wrong CIDR that collided with the
correct event-published snat. loadNATGWs now walks route tables and
emits one NATGWSpec per (NATGW, associated private subnet) pair,
mirroring publishNatGatewayEventsForAssociation. Intent map keyed by
natgwID|subnetCIDR so multi-subnet NATGWs produce distinct specs.
Tests:
- nat_test.go: cross-router same-logicalIP isolation for DeleteEIP and
DeleteNATGateway
- intent_test.go: NATGW snat uses associated private subnet CIDR, not the
NATGW home subnet; NATGW with no associations produces no spec
---------
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
* chore(ansible): add dev-status playbook (read-only health doctor)
New `dev-status` role + playbook + Makefile target. Read-only snapshot of
spinifex.target, failed units, port listeners, OVN/OVS state, gateway
round-trip, AMI/keypair/volume/instance counts, and build-vs-install drift
for spx + nbdkit plugin + microvm artifacts. Pairs with dev-preflight as
the post-install / running-system diagnostic.
Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.1.
* chore(ansible): add dev-logs playbook (bug-report bundle)
New `logs` role + playbook + Makefile target. Captures journalctl for
spinifex-* / ovn-* / openvswitch-switch / qemu / nbdkit, copies
/var/log/spinifex/, snapshots systemctl status + networking (ip/ss/ovs/
ovn show) + spx/plugin/microvm version manifest + dmesg, then bundles
into /tmp/spinifex-logs-<host>-<ts>.tar.gz owned by the invoking user.
Tail mode deliberately omitted — devs should `journalctl -u 'spinifex-*' -f`
directly; ansible captures stdout and is the wrong tool for follow.
Overrides: -e logs_since=10min, -e logs_output_dir=/tmp,
-e logs_include_dmesg=false.
Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.2.
* chore(ansible): add dev-snapshot / dev-restore (state checkpoint)
New snapshot + restore roles + playbooks + Makefile targets.
dev-snapshot quiesces the full spinifex+OVN+OVS stack in dependency
order (spinifex.target → ovn-controller → ovn-central →
openvswitch-switch), kills lingering QEMU VMs, then tars /etc/spinifex,
/var/lib/spinifex (viperblock volumes included), /var/lib/ovn (NB+SB
DBs), and /etc/openvswitch/conf.db into /var/lib/spinifex-snapshots/
<name>.tar.gz with a sibling .manifest, restarts services in reverse,
and waits for the AWS gateway to come back.
dev-restore inverses: stop services, wipe the same path set, untar the
named bundle, copy /etc/spinifex/ca.pem back into the system trust
store and rebuild the CA bundle (the snapshot doesn't include
/usr/local/share/ca-certificates/spinifex-ca.crt), daemon-reload,
restart in reverse order.
Both refuse to run without -e snapshot_name=<name>, validate the name
matches [A-Za-z0-9._-]+, and produce honest summaries when the gateway
fails to come back (does not mask other ongoing issues).
Tested on dev box: snapshot/restore of 2.4 GiB state (large viperblock
volumes) completes cleanly with no data drift; services all return to
active.
Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.3.
* fix(ansible): wrap aws/spx calls in sg spinifex to fix group inheritance
Adds scripts/ansible/scripts/aws-as-spinifex.sh and spx-as-spinifex.sh
helpers that re-enter the command under `sg spinifex -c '...'` with
AWS_PROFILE=spinifex preset. Swaps every AWS / spx admin task across
the status, init, deploy, smoketest, snapshot, and restore roles to
use these via spinifex_aws_cli / spinifex_spx_cli vars.
Root cause (mulga-siv-92): ansible's `become_user` is a no-op when the
target user equals the calling user, so sudo never runs and
supplementary groups are inherited from the operator's invoking shell
rather than re-initialised via initgroups(3). On shells that predate
setup.sh's spinifex group add, the spinifex group is missing and
/etc/spinifex/ca.pem (drwxr-x--- root:spinifex, the AWS CLI's
ca_bundle) is unreadable — aws cli fails with SSL Errno 13. The
helpers re-enter the command in a shell that explicitly assumes the
spinifex group via sg(1), bypassing the issue regardless of how
ansible-playbook is invoked.
Side benefit: drops 50 lines of repeated `become: true / become_user /
environment: AWS_PROFILE` boilerplate across the roles.
Closes mulga-siv-92.
* chore(ansible): add dev-version playbook (build vs install drift)
New `version` role + playbook + Makefile target. Compares sha256 of
built artifacts (bin/spx, viperblock/lib/nbdkit-viperblock-plugin.so,
build/microvm/{vmlinuz,initramfs.cpio.gz}) against their installed
counterparts (/usr/local/bin/spx, plugindir, /usr/share/spinifex/
microvm/*). Prints per-artifact drift verdict plus sub-repo HEAD
short-revs (spinifex/viperblock/predastore) for context. Closes with
either "DRIFT detected — run: make ansible-dev-deploy" or "No drift".
Pure read-only. Faster than dev-status (no AWS/OVN/port probing).
Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.9.
* chore(ansible): add missing roles/deploy/meta/main.yml
All other roles have meta/main.yml; deploy was the lone outlier.
Consistency fix. Closes mulga-siv-86.10.
* chore(ansible): add dev-vpc playbook (idempotent dev VPC fixture)
New `vpc` role + playbook + Makefile target. Idempotent
present/absent dispatch creates a deterministic spinifex-dev-vpc with
public subnet (10.99.1.0/24, map-public-ip-on-launch), internet
gateway, route table, and default route. Hooks into smoketest's launch
path — smoketest's existing
Name=map-public-ip-on-launch,Values=true filter now picks it up
instead of falling through to the "no subnet → skip launch" branch.
Idempotency: each step describe-then-create, looking up by Name tag
where possible. Re-running on a converged VPC is changed=0, skipped=5
(the create tasks).
Workaround for mulga-siv-94 (gateway silently drops
--tag-specifications and no-ops create-tags for route tables): role
falls back to "RT associated with our subnet" lookup so tag absence
doesn't break idempotency. create-tags task is kept as a no-op-now
forward-looking step for when the gateway gap is closed.
Teardown (-e dev_vpc_state=absent) enumerates ALL non-main route
tables in the VPC (not just the tagged one), so orphans from prior
runs that landed before the fallback was added get cleaned up too.
Tested end-to-end against josh-banksia dev gateway: create / re-run
(idempotent) / teardown all converge cleanly.
Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.4.
<textx="24"y="30"font-family="'Segoe UI', system-ui, sans-serif"font-size="11"fill="#a1a1aa"letter-spacing="2.5"font-weight="600">ECS · EC2 LAUNCH TYPE ON SPINIFEX</text>
<textx="876"y="30"font-family="'SF Mono', ui-monospace, monospace"font-size="10"fill="#71717a"text-anchor="end">container instances from spinifex-ecs-node AMI · awsvpc + ALB</text>
<textx="450"y="410"font-family="'Segoe UI', sans-serif"font-size="9.5"fill="#71717a"text-anchor="middle">No ECS launch API — you add capacity by booting EC2 instances from the spinifex-ecs-node AMI (the console's "Provision capacity" wraps RunInstances)</text>
<textx="450"y="426"font-family="'Segoe UI', sans-serif"font-size="9.5"fill="#71717a"text-anchor="middle">The service keeps the desired task count running and registers each task's ENI IP with the ALB target group</text>
<textx="450"y="206"font-family="'Segoe UI', sans-serif"font-size="9.5"fill="#7ee787"text-anchor="middle">managed node group · workers join the control plane</text>
<textx="450"y="404"font-family="'Segoe UI', sans-serif"font-size="9.5"fill="#71717a"text-anchor="middle">You provide the VPC, subnets, and IAM roles · Spinifex provisions the control plane, node-group VMs, and security groups</text>
<textx="450"y="420"font-family="'Segoe UI', sans-serif"font-size="9.5"fill="#71717a"text-anchor="middle">Node groups always boot the registered eks-node image · workers need egress (IGW / NAT) to pull images</text>
# Manual-work mutex gate — see setup note 4 above. Checked BEFORE any
# SSH to wattle so a paused run touches the box zero times.
- name:Manual-work mutex gate
id:gate
env:
REPO_VAR_PAUSED:${{ vars.CI_GPU_PAUSED }}
run:|
set -euo pipefail
PAUSED=false
if [ -f "$HOME/.ci-gpu-paused" ]; then
PAUSED=true
echo "::warning::ci-gpu runner paused via ~/.ci-gpu-paused sentinel on the runner — skipping this run (manual GPU work in progress on wattle). Remove the file to resume CI."
fi
if [ "$(echo "${REPO_VAR_PAUSED:-}" | tr '[:upper:]' '[:lower:]')" = "true" ]; then
PAUSED=true
echo "::warning::ci-gpu paused via repo/environment variable CI_GPU_PAUSED=true — skipping this run."
fi
echo "paused=${PAUSED}" >> "$GITHUB_OUTPUT"
- name:Paused — nothing to do
if:steps.gate.outputs.paused == 'true'
run:echo "ci-gpu is paused; job exits cleanly without touching wattle. See gate step above for the reason."
echo "::error::GPU e2e suite failed (rc=${{ steps.e2e.outputs.rc }}) — see analysis above; raw ES/APM traces at ${SPINIFEX_OTLP_ENDPOINT} (ci.run_id=${{ github.run_id }})"
@@ -192,6 +192,10 @@ While Spinifex is architected and implemented by experienced engineers, we lever
This hybrid approach ensures Spinifex benefits from both proven engineering expertise and modern development acceleration, while maintaining the quality and reliability standards required for production infrastructure.
## Trademarks
"AWS", "Amazon Web Services", and all related service names (EC2, EBS, S3, VPC, IAM, and others) are trademarks of Amazon.com, Inc. or its affiliates. Mulga Defense Corporation and Spinifex are independent and not affiliated with, endorsed by, or sponsored by Amazon.com, Inc. or Amazon Web Services, Inc. References to AWS services describe interoperability and compatibility only.
## License
Spinifex is open source under the [GNU Affero General Public License v3.0](LICENSE). You're free to use, modify, and deploy it anywhere you need reliable infrastructure without depending on centralized cloud platforms.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Julian Sommer
