* fix(eks): GPU node AMI declares uefi boot mode
The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but
the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under
legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at
firmware with zero serial console output.
* fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs
Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to
initialize under the closed/proprietary kernel module: dmesg shows
RmInitAdapter failing with "requires use of the NVIDIA open kernel
modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to
nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia
recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no
separate open variant (userspace tools only), so it is unchanged.
* fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons
A cluster with a GPU-only nodegroup has no untainted node: the CP
carries CriticalAddonsOnly (EKS parity keeps user workloads off it)
and every worker carries nvidia.com/gpu. Without tolerations for both,
ebs-csi-controller and aws-load-balancer-controller have nowhere to
schedule and stay Pending forever. Real EKS never needs this since its
control plane isn't a schedulable node, so the stock upstream
manifests carry no such toleration.
* fix(images): give the libguestfs appliance a working resolver
The guest's /etc/resolv.conf symlinks to systemd-resolved's stub
(127.0.0.53), which isn't running inside the appliance chroot, so DNS
is dead until package installs need it. Point it at a real resolver
before package installs, then restore the stub symlink afterward so
the shipped AMI still gets its DNS from systemd-resolved at boot like
a stock cloud image.
* fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind
A GPU's IOMMU group can include an upstream PCIe bridge when the
platform lacks ACS isolation between them. vfio-pci refuses to bind
non-endpoint bridge devices, so Claim/ReclaimByAddress must filter
bridge-class (0x06) members out of the group before touching driver
bindings, rather than failing the whole claim or leaving the bridge
rebound incorrectly.
* fix(ecs): GPU node AMI declares uefi boot mode
The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the
Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS
no OVMF/pflash firmware is attached, so the guest hangs at firmware with
zero serial console output.
Same defect and same fix as the eks-node-gpu entry: both images are built
by the same script from the same stock Ubuntu cloudimg, so both were
EFI-only and only one was corrected.
* fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch
ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary
kernel module. NVIDIA ships no proprietary module for Blackwell (GB202)
and later, so ECS GPU container-instances could not bind those cards on
any driver branch. It is the third verbatim copy of the routine and was
missed when the other two were fixed.
Also pins the driver branch across all three copies. The branch was
resolved at build time via apt-cache search | sort -rn | head -1, which
sorts on the number in the package name — but that number is a branch
alias, not a driver version (nvidia-dkms-535-server installs 580.126.20),
so the ordering key need not match what installs. Resolving also let
identical source produce different drivers on different days.
The pin lives in setup.sh rather than manifest.conf because manifest
variables are not passed into the guest: the setup script is uploaded and
executed by virt-customize --run, with no environment carried across.
* fix(eks): tolerate CP and GPU taints in argocd addon
All six argocd Deployments and its StatefulSet carried no tolerations. A
cluster with a GPU-only nodegroup has no untainted node to place them on:
the control plane carries CriticalAddonsOnly and every worker carries
nvidia.com/gpu, so the pods stay Pending forever.
Same defect the ebs-csi-controller and load-balancer-controller addons
already had. Real EKS never hits it because its control plane is not a
schedulable node, so the stock upstream manifests carry no such
toleration and the divergence is ours to maintain.
---------
Co-authored-by: joshsiv-mulga <mrk@mulgadc.com>
* fix: detach before delete on instance terminate to stop root-volume leak
TerminateStoppedInstance and the running-instance terminate path both
deleted a DeleteOnTermination root volume without first clearing its
attachment, so DeleteVolume's in-use guard rejected it and the error was
swallowed (stopped path) or only alarmed (running path), leaking the
volume permanently.
Add VolumeServiceImpl.DeleteVolumeOnTerminate, which clears the stale
attachment via UpdateVolumeState (terminate implies detach) before
deleting, and route both terminate paths through it. Stamp
Teardown[TeardownVolumes] on the stopped path so a delete failure is
visible to the volume leak reaper instead of being lost.
* fix: detach non-DeleteOnTermination root on terminate; cover teardown self-heal
A DeleteOnTermination=false root volume stayed attached to the terminated
instance on both terminate paths: Stop's Unmount deliberately skips Boot
volumes, so the root kept its attachment across stop and neither terminate
path detached it, stranding it attached to an instance that no longer
exists. Add VolumeServiceImpl.DetachVolumeOnTerminate (metadata-only
attachment clear, no delete) and route the DeleteOnTermination=false Boot
volume through it on both paths, matching AWS terminate-implies-detach.
Also add a regression test proving the TerminatedTeardownReaper re-drives a
failed volumes-teardown through the now-detach-first cleaner and self-heals
the leak, so a transient delete failure at terminate does not strand a
volume permanently.
* fix: bound detach chain and reconcile vanished-QEMU wedge
DetachVolume gains a bounded aggregate context deadline over the QMP
hot-unplug chain and a confirmed-dead-QEMU short-circuit that skips the
QMP steps (which would only hang on the wedged process) and seals
directly, so a detach can no longer pin attachMu forever when nbdkit or
QEMU die mid-flight.
OrphanQEMUReaper now also finalizes shutting-down instances whose QEMU
has vanished: a terminate that wedged downstream and never reached
finalizeTerminated is driven to terminated, with outstanding teardown
stamped failed so TerminatedTeardownReaper completes it idempotently.
Gated on a dead process, so a live in-progress terminate is never
stolen.
* feat: add delete-authorized stuck-terminate backstop
A terminate that wedges with QEMU still alive is invisible to the
OrphanQEMUReaper dead-QEMU reconcile and would sit shutting-down forever.
StuckTerminateReaper force-completes it: past stuckTerminateTimeout it
force-kills the wedged process, reclaims DeleteOnTermination volume space
through the cleaner, stamps remaining teardown for TerminatedTeardownReaper,
and drives the record to terminated.
It is the one reaper explicitly authorized to delete volume data, but only
for a DeleteOnTermination volume on an already-terminating instance — a far
narrower licence than ADR-0005 §3 denies VolumeLeakReaper, which stays
mark-and-alarm-only. A new ShuttingDownAt timestamp, stamped once on the
transition, bounds how long a terminate may wedge; a missing timestamp reads
as not-yet-stuck so the backstop never acts on an unbounded record.
TestNPassOverwrite detaches after every pass, so viperblock's drain and GC
only ever run at reopen — it cannot show whether GC reclaims superseded
chunks while a volume stays open, and its serial write rate never nears the
predastore rate limiter's burst. TestContinuousChurn attaches once and drives
concurrent 4 MiB-aligned raw overwrites of per-writer regions for several
minutes without detaching, so a sustained superseded-chunk backlog builds
under one long open, spanning multiple GC ticks.
The workload is bounded and self-contained: each dd writes one chunk and the
clock is re-checked after every write, the loop runs as root under a single
sudo bash with a process-group kill trap, and it is wrapped in a guest-side
timeout. Root ownership is load-bearing — the writers open a raw device, so
only a root-owned process group can reap them; a best-effort guest pkill runs
before detach as a backstop so a stray writer cannot wedge the detach.
Picks up viperblock 3dcfbb7 (#49): the chunk GC sweep now runs on a
dedicated goroutine instead of sharing the chunk-uploader select, so
sustained overwrite churn no longer starves GC and wedges the volume at
the internal space cap.
This reverts commit f9c9f0b38d.
#580 switched setup.sh to read the northstar :53 listener and zones from
northstar.toml, so the resolved stage now defers at install time (before
formation) and must be re-run afterward. Only firstboot re-runs it; the
manual setup.sh + spx admin init install and CI never do, so those hosts
get no host DNS and cluster-zone names (ELB/EKS) fail to resolve from the node.
Reverting restores the pre-#580 behaviour: systemd-resolved is configured
at install time from the WAN IP, with no post-formation re-run needed. This
reintroduces the original gap where resolvconf hosts get no host DNS. The
proper fix configures host DNS from spx admin init/join for both resolver
managers (mulga-x30yv).
A freshly launched LB agent begins heartbeating the instant the VM boots,
but the gateway node fielding that first heartbeat may not yet have the LB
record — it is still replicating from the node that created it — so the
gateway answers 400 LoadBalancerNotFound. The IMDS credential chain is
likewise not always warm on the first tick. Both self-heal within a few
ticks, yet every early failure was logged at ERROR, flooding the console
telemetry with startup noise (46 such lines across 8 agents in one run).
Bound these to a registration grace window: before any heartbeat has ever
succeeded, and only within the window, a failure logs at INFO. Past the
window with no success — the fingerprint of a genuinely blackholed agent —
or after any prior success, a failure logs at ERROR as before, so the
serial-console telemetry still surfaces a real blackhole. The only change
for a persistent failure is a bounded delay before it escalates.
* fix: make the seal-failure unmount test deterministic
The test forced the seal to fail by pointing S3Host at an unreachable
endpoint, so the handler only returned once the S3 client had exhausted
its jittered backoff. Measured handler time spanned 1.8s to 5.5s against
a 5s request deadline, which failed 2-3 runs in 10 on an idle machine and
worse on a loaded CI runner.
Config gains a sealVolume seam, nil meaning the real seal, and the test
injects an immediate failure instead. It now runs in 0.53s, 50 for 50,
and still asserts what it was written to assert -- plus that the handler
actually attempted the seal, which the old form could not distinguish.
* chore: bump viperblock to dev tip (GC observability + scratch sweep)
Picks up viperblock #47 (every chunk GC sweep logs its outcome at Info)
and #48 (TestMain reclaims scratch dirs abandoned by a panicking run).
The tier ran inside the Build and Test job, serialised ahead of test-cover, and
again in preflight. Neither placement bought signal that a parallel job does not:
it provisions nothing and finishes in seconds, so the only effect was wall time
on the two paths developers wait for most.
It stays PR-blocking as its own job, which is what distinguishes it from
e2e.yml's front gate — that one is push-triggered on a self-hosted runner and so
never gates a PR before merge.
MgmtIPAllocator allocated from a process-local map, but br-mgmt is
trunked onto a shared physical NIC making the /24 one flat L2 segment
across the whole cluster. Every node independently allocated from
.10, causing collisions and ARP-race losers being blackholed.
Rewrite the allocator on the spinifex-cluster-state NATS KV bucket
using compare-and-swap, keyed by mgmt-ipam.<subnet>, modeled on IPAM.
Bind the KV handle in startCluster (not startLocal) to respect the
DDIL boundary. Serve already-allocated instance IDs from the local
cache when KV is unhealthy, but refuse new allocations until it
recovers. Rebuild now reconciles this node's running VMs into the
cluster-wide record instead of just refreshing the local cache.
* feat(config): enable viperblock chunk GC in rendered node configs
The gc_enabled toggle was plumbed through every VB construction site but
nothing ever set it. GCEnabled is a *bool that every consumer treats as
off when nil, and the config template rendered shardwal while omitting
gc_enabled, so no deployed node ever carried the key and chunk GC was off
everywhere by construction. Superseded chunks were never reclaimed under
overwrite churn.
Render gc_enabled = true alongside shardwal, and cover it with a test that
loads the rendered file through config.LoadConfig rather than matching the
template text. A string match would still pass if the value failed to parse
into a non-nil pointer, which is the exact way the setting reaches its
consumers.
Existing deployments keep the old rendering until their config is
regenerated; the template governs install and bootstrap only.
* chore: pin viperblock to the build with the cross-process GC snapshot marker
Enabling chunk GC in rendered node configs is only safe once the sweeping
engine can see a snapshot taken by another process. viperblock 3b56d61 adds the
per-volume snapshot marker that closes that window, and 2a34add fixes the
concurrent-stop panic in the background goroutine lifecycle that GC's own ticker
would otherwise widen. Pin both before turning GC on.
LaunchRunInstances is a best-effort batch launcher that reports failure only
through VM state: volume preparation, GPU claim and vmMgr.Run errors each call
MarkFailed and continue. launchAMISystemInstance ignored that and always
returned a SystemInstanceOutput, so an EKS control-plane server, ELB or Bedrock
serving VM whose launch failed handed its caller an instance ID for a VM that
was already being torn down.
Check the instance reached running state before returning. Cleanup stays with
the MarkFailed chain LaunchRunInstances has already run.
buildDrives emitted every non-boot data volume as a bare -drive with no
id= and no node-name=, so a stop/start (or reboot/resize) relaunch left
the volume anonymous: device_del/blockdev-del had nothing to address,
QEMU kept the NBD socket open, and DetachVolume stalled through
nbdkit's full kill grace before giving up.
Add -blockdev support to the QEMU config builder and use it for
non-boot, non-EFI volumes, sharing the node-name/device-id/iothread-id
naming and virtio-blk-pci device arguments with AttachVolume's hot-plug
path so cold-boot and hot-attach can no longer drift apart. A data
volume with no persisted HotplugPort now gets one allocated and written
back during buildDrives instead of landing on an unplugged bus.
Also raise unmountSealTimeout past viperblockd's kill grace (it runs
the seal after the SIGKILL, so matching the grace meant an unmount
that used the whole grace could never be observed as successful), and
bound the e2e harness's forced teardown detach by the teardown
deadline so a slow detach is reported as slow rather than as an
unexplained leak.
* test: assert the gateway->vpcd->OVN crossing in the integration tier
The tier already ran the real VPCServiceImpl for the AWS-facing half, but
nothing consumed the vpc.* events it published: the SG topics were answered
by a canned {"success":true} and the fire-and-forget topology topics were
dropped. Both sides of that wire were tested only against stubs, so the join
was asserted nowhere below the live e2e tier.
Wire a real subscribers.Subscriber over a real in-process OVN Northbound DB
(ovntest.StartNB), opt-in per test so only tests that assert OVN state pay
the startup cost. StartDaemonLite gains WithRealVPCD to suppress its stubs,
since a stub and the subscriber both answering one request-reply would race.
Scope is the crossing, not NB row shapes — those stay with the reconcile
scenario tests. Everything host-dependent (ovn-controller, OVS, taps, netns,
nftables, packet paths) remains the live tier's claim.
* fix: make the integration tier re-runnable and quiet the ensure probe
Two defects the vpcd/OVN work surfaced.
The tier was green at -count=1 and red package-wide at -count=2. Test
accounts were keyed by t.Name(), which is unique among tests but repeats
across passes in one TestMain, so a second pass inherited the first pass's
JetStream namespace — including an ECR signing key sealed with a master key
StartGateway regenerates per call. Key each account off a counter instead.
TestRunInstances_ClientTokenIdempotency then collided with itself for the
same reason: its dedup store outlives a pass, so its fixed ClientToken made
the second pass's opening launch look like a replay. Suffix the tokens per
invocation.
ensureNamedRowOps' wait-op is built to fail fast when the row it guards
already exists, which is the ordinary idempotent path, but transactOps
reported it as "OVSDB operation failed" at ERROR. Every re-ensure emitted
one, drowning real faults in the log sink. Classify that one result as the
probe outcome it is and log it at Debug.
* test: cover transactOps' failed-result paths
The classifier split one log line into two branches and neither was
exercised: transactOps sat at 31% because EnsureLogicalRouter's cache
lookup absorbs a re-ensure before it ever transacts, so no unit test
reached a failed operation result.
Drive both kinds directly against a real NB — the guard probe finding an
existing row, and two same-named inserts violating the schema's name
index — since only the second is a fault and they now log at different
levels. transactOps goes to 94%; the remainder is the transport-error
return, which needs a broken connection to reach.
* feat(ci): extend per-cell telemetry scoping to every e2e workflow
ci.cell only reached e2e-nightly and e2e-suites, so a run of e2e, e2e-ddil
or gpu-e2e still landed in the sink as one undifferentiated bucket. Stamp it
on the remaining workflows: single/multi on e2e's two provisioning jobs, ddil
and gpu on their single-job workflows.
The best-effort numeric job id lookup moves out of e2e-nightly into a shared
composite action, so the four other provisioning jobs get ci.job_id without
copying the gh/jq incantation. The lookup now matches on env.JOB_NAME rather
than interpolating the name into the jq filter, which broke on names carrying
quotes or non-ASCII separators, and no-ops when telemetry is disabled.
gpu-e2e gets ci.cell but deliberately no job id lookup: nothing in that job
bakes the attrs into a daemon, so the API call would be paid for an env var
no exporter reads.
* fix(ci): define e2e suite topology eligibility once
Three workflows carried three different answers to which suites a topology
can run. e2e-suites and e2e-nightly ran lb on their single-node leg, where
every internet-facing subtest skips for want of a peer, so a fraction of the
suite reported as a green lb run. e2e-suites' multi leg ran multinode alone,
dropping the lb and cert coverage that only a multi-node env can complete —
so full lb coverage never ran there at all.
.github/scripts/suite-sets.sh is now the single definition, sourced by all
three. e2e.yml's two allow-list regexes derive from it, e2e-suites derives
both matrices from it (multi_provision gains the suite-matrix step
single_provision already had), and e2e-nightly's deliberately narrower
per-cell lists are declared there too, guarded by an assert that fails the
cell if a subset ever names a suite its topology cannot fully exercise.
The nightly stays narrow on purpose: its cells have a 35-minute budget and
exist to prove each install/network/host-OS permutation boots and serves.
Suite breadth is the PR gate's job and already runs on every push. That is
now stated in one place rather than implied by a third divergent list.
* test(volume): assert chunk GC stays wired on in CreateVolume
GCEnabled was zero-valued at every viperblock construction site for a long
time and no test noticed. It is not part of viperblock.VBState, so it never
round-trips through a saved config.json and cannot be observed by reading
volume state back.
Extract the viperblock.VB assembly out of CreateVolume into buildVBConfig so
the GC wiring is assertable directly against the returned config, without a
live viperblock or S3 backend. The config is returned by pointer because
viperblock.VB carries atomic.Uint64 fields that vet's copylocks check
rightly refuses to see copied.
* test(e2e): snapshot backend byte accounting before volume purge
DeleteVolume purges a volume's entire backend prefix, including any chunks a
leak left behind, before anything measures it. That is the same masking that
hid the storage-growth leak in production: a passing E2E cannot fail on
backend growth when the evidence is deleted first.
Snapshot the volume's backend chunk footprint in EnsureVolume's cleanup
closure, immediately before DeleteVolume, so every volume the shared teardown
path purges is measured rather than silently discarded. The snapshot is
always taken and logged; the hard ratio check additionally runs for callers
that registered a guest-written-bytes baseline, since most callers never
attach or write to the volume they create and so have no baseline to compare
against.
The tolerance is the RS(2,1) erasure floor widened for per-chunk header and
tail-chunk overhead. It is sized to distinguish unbounded growth from a
footprint settling at the floor, not to catch bounded residual garbage.
The check is non-fatal inside the cleanup callback: t.Fatalf's runtime.Goexit
would abort the DeleteVolume call queued behind it and turn a reported leak
into a real one.
* test(e2e): add memory-constrained concurrent-write durability regression
The existing durability tests write one 4MiB sentinel, sequentially, on a
comfortably sized guest, which is the exact inverse of the known-open
data-loss bug: acked writes read back zero-filled after a cold reread when a
guest is memory-starved during concurrent writeback. No happy-path test can
reach it, because the fault lives in the intersection of memory pressure,
write concurrency and cache-cold reread.
Add a suite that encodes those three conditions together: a guest discovered
at or below the working set, N parallel large writers launched as background
jobs in a single command so they genuinely overlap, swap disabled, then sync
and drop_caches before rereading. Assertions are per stream rather than
aggregate so a failure names the losing writer instead of leaving whoever
reads it to re-derive which of N writers lost data from one combined digest.
GuestExec gains a timeout-parameterised variant. Its fixed two-minute default
is sized for sub-second payloads, and a multi-gigabyte write to a
network-backed volume would exceed it, so the suite would have failed by
timeout while looking like it had caught the corruption. The budget scales
with the working set, and a deadline is reported distinctly from a data
mismatch so the two red states are never confused.
The suite is dispatchable but deliberately absent from the default SUITES:
the bug it reproduces is still open, so it is expected to fail and must not
sit in a merge-gating path. It is registered in the workflow allow-list so it
stays discoverable rather than being buried behind a skip.
Also record on the live IAM instance-profile test why its assertions were not
moved down to the integration tier, so the trace is not repeated.
* test(e2e): drop backend byte accounting, assert a node-wide delta instead
The accounting helper listed the internal predastore bucket to attribute
bytes per volume. That listing is refused unconditionally: predastore
authorises bucket access by same-account ownership, and the bucket belongs
to the service account, so a test caller gets 403 no matter what it holds.
Granting the test a service credential would invert the trust boundary to
buy a measurement, which is the wrong trade.
Replace it with what the S3 surface actually permits: sample node-wide
allocated bytes either side of the write and bound the delta against the
bytes the guest wrote. Coarser, but it measures something real rather than
depending on a call that cannot succeed.
The tolerance moves into the suite that uses it. It encodes RS(2,1)'s 1.5x
expansion floor plus headroom; a cluster running the recommended RS(3,2)
has a 1.667x floor and only ~12% of margin left, so the constant needs
revisiting if the encoding changes.
* revert: drop memdurability e2e suite
The suite asserted the guest read back what it wrote after a
memory-constrained concurrent write, but the assertion could not fail for
the reason it claimed: it compared the guest's own read against the
guest's own write with the page cache still warm, so it passed whether or
not the data reached the backend. A test that is green on a build with a
known-open durability bug is worse than no test — it advertises coverage
that does not exist.
Removes the suite and its Makefile/workflow wiring, plus
DiscoverInstanceTypeAtLeastMemory, which had no other caller.
GuestExecTimeout stays: GuestExec now delegates to it, and the old
two-minute hard cap it replaced was a latent trap for any long guest
command.
* test(e2e): add partial-block concurrent-write durability suite
Replaces the removed memdurability suite, which passed on a build that
still carried the bug it claimed to test.
The engine reaches its read-modify-write path only for a partial block;
a fully covered block takes a full-overwrite branch that reads nothing
and can lose nothing. The old workload wrote through ext4 with 1 MiB
buffers, which the filesystem coalesced into aligned full-block
requests: block-layer tracing recorded zero unaligned and zero sub-4096
requests, so the path under test never ran, and writers 4 MiB apart
could not collide on a block even in principle.
This suite writes the shape the defect needs. Two writers issue
concurrent O_DIRECT writes into disjoint halves of the same 4096-byte
blocks on a raw, unformatted device, stamping every half with the round
number, then drop the page cache and reread with O_DIRECT asserting
every half carries the final round. One half reading back at an earlier
round is the signature of the lost update.
The old suite also could not fail even had it reached the path: its
write phase ended in 'wait; sync; sha256sum' and reported sha256sum's
exit status, while bare wait returns 0 unconditionally, so writer errors
were invisible and a backend ENOSPC presented as a pass. Here each
writer's exit status and exact byte count are asserted before any
durability claim is made.
Working set is 8 MiB rather than 2048 MiB, and the volume is deleted on
teardown; the predecessor left roughly 3 GiB of backend per run and
twice drove a node under the nearfull watermark.
Dispatchable but not in the default suite list until a green run and a
demonstrated fail-first against a pre-fix engine exist.
* test(e2e): let the fixture own the working-set volume delete
The suite deleted the volume in its own cleanup and the fixture deleted
it again on close, so every run ended with an InvalidVolume.NotFound
from the second attempt. Detach stays — DeleteVolume against a still
attached volume is what leaks volumes on teardown — and the delete is
left to the fixture so the volume is released exactly once.
* test(e2e): gate merges on the partial-block durability suite
Verified on one node with the same test binary and workload, varying
only the engine: 1758 of 8192 halves reverted a generation against the
pre-fix build, 0 of 8192 against the fixed one. That is the fail-first
evidence the suite needed before it could be trusted to gate anything.
Adding it to the default suite list rather than leaving it dispatchable
is the point. Its predecessor sat outside every path that runs, which
is how it survived long enough to be discovered green on a build that
still lost data.
* test(integration): cover the ECR data plane in the front gate
The live ecr/ suite drove docker, crane and skopeo against a real
registry, yet none of it boots a guest: it needs a registry, an object
store and an HTTP client. The in-process gateway already serves the OCI
Distribution v2 surface on an ephemeral port with a memory-backed object
store, so these run here in 0.18s instead of behind full provisioning.
Covers the three claims the live tests made beyond authorization, which
is already down here: a push/pull round trip asserting blobs and
manifests read back byte-identical, refusal to auto-create a repository
on push, and retagging an existing manifest without disturbing the
original tag.
Content is compared rather than status codes alone. A registry that
stores a blob under the wrong digest, truncates it, or serves a
different one answers 200 to every request in the sequence.
The capacity/load leg deliberately stays live: it exists to put real
parallel pressure on a real predastore, and a memory-backed object store
cannot make that claim.
* fix(e2e): make volume teardown release the volume before deleting it
Fixture cleanups drain LIFO, so a suite that creates its instance before its
volume tears the volume down first, while it is still attached. DeleteVolume
answered VolumeInUse, nothing retried, and the volume was stranded on the node.
Reordering the chain cannot fix this — registration order has no way to express
the instance/volume dependency. Instead teardown is now self-sufficient for any
creation order: it force-detaches, waits for the volume to reach available, and
only then deletes. Instance teardown waits for terminated for the same reason,
since TerminateInstances returns as soon as the request is accepted.
Two things kept the leak invisible. A failing cleanup was logged and forgotten,
so a run stayed green with resources left behind; failures are now reported as
leaks, failing the bound test or surfacing through Close() for TestMain to fail
the run. And a panic in one cleanup aborted the whole chain, stranding every
resource behind it; each callback is now isolated.
RegisterVolumeTeardown and the fixture had grown separate teardown paths, only
one of which detached. Both now share it.
Re-creating a nodegroup of the same name could never reach ACTIVE. The
Ready-gate took its baseline from the live Ready count for that name, so
a terraform replace after CREATE_FAILED — which deletes then re-creates
— captured the previous incarnation's workers, demanded old+new, then
watched the count fall as the old workers were terminated. The target
was unreachable by construction.
Observed on env19 with a perfectly healthy cluster: three new workers
all joined and the create still failed with
reports 3 Ready nodes, want >= 5 (baseline 2 + 3 workers)
and each retry raised the bar again (baseline 3 => want 6), so the only
way out was untainting the resource by hand.
A create means every worker under that name is new, so the gate is just
"desired Ready nodes carrying this nodegroup label" — baseline 0. The
trade-off is stated in the comment: a stale incarnation's Ready nodes
can briefly inflate the count and satisfy the gate early, but they go
NotReady within about a minute of termination, whereas baselining on
them fails every replacement permanently.
The test models the real sequence — delete, stale Ready count still
reported, re-create — and fails without the fix with the same shape as
the field error.
* fix(dhcp): derive nclient4 read deadline from the caller's context
nclient4 races its own read deadline against the caller's context and the
shorter one ends the attempt. Acquire and Renew passed a fixed 5s, so every
window in Manager's 4s/8s/16s/32s retransmission schedule past the first was
truncated to 5s: the client listened for ~19s of its 90s budget and a server
that went quiet for longer was never waited out.
Measured in the sink, the split was total — attempt 1 (jittered under 5s)
always failed with the caller's context deadline, attempts 2 to 4 always
failed with nclient4's ErrNoResponse — and only 27% of clients that lost
attempt 1 were rescued by attempt 2.
Track the context's remaining time instead, keeping the configured value as
the fallback its doc comment already describes.
* fix(dhcp): keep the socket deadline strictly behind the caller's
Setting the nclient4 read deadline to exactly the context's remaining time
makes the two expire together, so a timed-out attempt reports either
context.DeadlineExceeded or nclient4's ErrNoResponse depending on which
goroutine wins. Observed live on a settled bridge: three attempts reported the
context deadline and the fourth reported ErrNoResponse, which reads as a
packet-matching fault rather than the plain timeout it is.
Give the socket a second of grace so ctx.Done() is always what fires.
* fix(dhcp): encode option 61 with its hardware-type byte
The client identifier was sent as a bare string, so the server read the
identifier's first character as the hardware type — "dhcp-gw-lrp-vpc-x"
arrived as type 100 ('d') with the identifier truncated. Since that is not
Ethernet, the server recorded no hardware address, and the upstream lease
table listed our leases with a MAC of all zeroes.
Send the chaddr typed as Ethernet, falling back to a type-0 identifier when
no hardware address is available. RELEASE now encodes identically, since the
server matches a release to a lease by client identifier.
Option 61 no longer carries the readable id, so restore it via hostname and
vendor class, which were specified but never populated. dhcptest builds its
options through the same helper rather than its own copy, which had drifted
into reproducing the broken encoding.
* fix(dhcptest): derive the probe MAC the way vpcd does
The probe carried its own copy of the MAC derivation, still on the 24-bit
02:00:00:xx:xx:xx hash that vpcd moved off when widening it to a SHA-256
prefix. So --spinifex-id leased under a MAC the daemon would never use:
"gateway-wan1" resolved to 02:00:00:7c:b7:b6 here and 02:54:df:71:22:5c in
vpcd, which makes the probe unable to reproduce a daemon lease at all.
Call DeriveMAC instead and drop the local copy.
The nightly matrix runs every cell under a single ci.run_id, and each cell
names its hosts node1/node2/node3, so telemetry from all cells collapses into
one undifferentiated bucket in the ES sink. Root-causing a single failed cell
means aggregating across every other cell in the run.
Stamp ci.cell and ci.matrix alongside the existing ci.run_id. Workflow-level
env cannot reference the matrix context, so the attrs are overridden in each
matrixed job's own env block, leaving the workflow-level value as the
non-matrix fallback.
The numeric GitHub job id is not exposed in any context (github.job is the
YAML key, not the id), so resolve it best-effort via the jobs API and append
it if found. The lookup never fails the job.
e2e-suites can only distinguish single from multi: daemon telemetry is baked
into the VM at provision time, before the suite matrix exists.
Baremetal shipped no telemetry at all — neither the endpoint nor the attrs
crossed the workflow_call boundary. Pass them through as an explicit input so
the cell reaches the sink for the first time.
* fix(eks-node): stop the DHCP MTU overriding the pinned link MTU
Pinning eth0 to 1320 was never enough. This image runs dhcpcd, which
honours DHCP option 26 and stamps the advertised MTU onto every route it
installs:
default via 10.32.101.1 dev eth0 proto dhcp src 10.32.101.7 \
metric 1002 mtu 1442
Linux prefers a route's MTU metric over the device MTU, so the node kept
advertising an MSS derived from 1442 while the real path MTU through the
NAT gateway is 1408. Small packets pass and full-MSS inbound segments —
a TLS ServerHello with its certificate chain, image layers — are
silently dropped. It presents as `net/http: TLS handshake timeout` on
image pulls, not as anything MTU-shaped, and it only affects the host
netns, so containerd pulls fail while pod traffic sized off cni0 is
fine. Per-destination PMTU exceptions expiring in and out of the route
cache made it look intermittent and host-specific.
Three changes, innermost first:
- dhcpcd.conf gets `nooption interface_mtu`, so no lease can stamp a
route MTU. mulga-vpc-mtu is the single source of truth for the MTU
of these nodes.
- mulga-vpc-mtu re-stamps any route that already carries a different
MTU, covering an image built before this fix or a dhcpcd already
running. Routes with no MTU metric inherit the device MTU and are
left alone.
- the subnet DHCP option drops 1442 to 1408. 1442 is geneve-only and
ignores the IPsec ESP overhead on the NAT egress path, so it
overstates the usable MTU for every guest, not just these nodes.
Also corrects the header comment, which claimed Alpine's udhcpc ignores
option 26 — this image runs dhcpcd, which honours it. That wrong premise
is why the route metric was never considered.
Adds a stubbed-`ip` test covering the rewrite, the dev token that
`ip route show dev X` omits, idempotency, and routes without an MTU
metric.
* fix(eks): sweep the CP VPC by identity so teardown can finish
DeleteClusterCPVPC finds the managed control-plane VPC's subnets and SGs
by role tag. A create that fails partway — the first control-plane
server failing to launch, say — leaves resources behind that never got
tagged, so the sweep cannot see them.
DeleteVpc rejects while any subnet or non-default SG remains, and
teardown is re-driven on a timer, so the cluster sits in DELETING
forever retrying the same doomed delete every couple of minutes:
eks-deleting: re-driving wedged DELETING teardown cluster=toc
eks-deleting: re-drive teardown failed
err="delete managed CP VPC: eks: delete cp vpc vpc-...: DependencyViolation"
Nothing converges: the cluster name cannot be reused and terraform's
delete wait can never pass. Observed on env19, wedged for ~11 hours
until this change went in, at which point the next re-drive cleared it.
After the tagged sweep, clear anything still in the VPC scoped by
vpc-id: subnets, and non-default SGs (the default SG goes with the VPC
cascade). Teardown had no SG surface, so CPVPCDeps gains an optional SG
provisioner — nil is tolerated, since only teardown uses it.
* chore: pin viperblock + predastore to the merged storage fixes
viperblock f8cfd77 (#42) makes read-modify-write atomic per block,
removing the lost update that silently reverted a 4096-byte block to a
previous generation of itself. Confirmed as the env19 corruption cause
by an A/B: the pre-fix build corrupted image layers within ~15 minutes
(failed to extract layer / invalid tar header / unpigz: corrupted) while
the fixed build ran 5+ hours clean, absorbing 6407 same-block RMW
conflicts on the worker root disks.
predastore c0a90d2 (#63) batches FSM.Restore. Without it, any predastore
restart on a store whose metadata set outgrew one badger transaction
fails with "Txn is too big to fit into one request" on every node at
once, taking the metadata plane — and with it the AMI catalogue and
cluster creation — down permanently.
Both were verified in the built artifact rather than from go.mod alone:
go version -m reports the new pseudo-versions, and the rmw_conflicts
counter plus the batched-restore error strings are present in the
binary. Pinning by go.mod while the submodule pointer says otherwise is
how an earlier analysis ended up reading dead code.
* test: record the corrected DHCP MTU in the kitchen-sink golden
The NB golden pins the DHCP options published per subnet, including the
advertised MTU. Dropping the subnet option from 1442 to 1408 changes
what the reconciler writes, so the golden diverged and the live scenario
test failed — the test doing exactly its job.
Regenerated with UPDATE_GOLDEN=1. The diff is only mtu:1442 -> mtu:1408
on the two subnet DHCP records; nothing else in the snapshot moves.
* test: add in-process integration test tier
Roughly half of the behavioural tests under tests/e2e/ assert control-plane
API shapes, authz and error codes that need no live environment, yet they can
only run against a tofu-provisioned environment on a scarce self-hosted
runner. This adds the tier they can move down to.
The real gateway router runs in-process: SetupRoutes() is a pure function of
GatewayConfig, so it serves from an httptest server over embedded NATS
JetStream with real IAM/STS services and genuine SigV4 verification. Only the
daemon side of the wire is faked -- the ec2.*/ebs.* NATS subjects a live node
would answer are stubbed per test.
Nothing is provisioned and no fixed ports are bound, so the tier runs
concurrently on a shared build host and is safe as a CI front gate ahead of
the live suites. It is build-tagged and kept out of preflight, which stays
unit-only.
* test(integration): add IAM/STS client accessors and port the zero-stub authz tests
Adds IAMClient/STSClient plus ClientsWithCreds/ClientsWithSessionCreds so a
test can act as a principal other than the seeded root user, and stubs the
node-discovery subject in StartGateway so call paths that trigger discovery
resolve immediately instead of riding out the fan-out timeout.
Ports the four enforcement tests that guard on ec2:DescribeRegions, a
gateway-local action, so they need no daemon-side stub at all.
* test(integration): port remaining IAM/STS authz tests from e2e
Ports TestIAMUserCRUD, TestIAMAccessKeyLifecycle, TestIAMUserAuthentication,
TestIAMPolicyCRUD, TestIAMPolicyAttachmentEnforcement, TestIAMPolicyLifecycle,
TestIAMGroupsLifecycle, TestIAMRolesAndProfiles and
TestSTSAssumeRoleAndGetCallerIdentity from tests/e2e/iam/ into the in-process
integration tier. Each test now self-bootstraps its own users/policies/roles
instead of relying on shared suite-level state, since every integration test
gets its own gateway and embedded NATS. TestIAMRolesAndProfiles stubs
ec2.IamProfileAssociation.describe because DeleteInstanceProfile always
counts live associations over NATS regardless of which guard actually fires.
TestIAMCleanup is not ported: it only swept a shared long-lived E2E
environment, and every ordering/dependency assertion it made is already
covered by TestIAMPolicyLifecycle, TestIAMGroupsLifecycle and
TestIAMRolesAndProfiles's own delete-conflict guards.
* test(e2e): remove IAM/STS authz tests migrated to the integration tier
All 13 signed-request authz tests plus TestIAMCleanup now have faithful,
mutation-proven equivalents in tests/integration/ — the gateway dispatches
IAM/STS actions directly with no daemon hop, so the in-process tier is a
faithful and far cheaper substitute. The iam/ e2e suite now runs only
TestIAMInstanceProfileAssociation and TestIMDS.
Drop the now fully-dead iam_helpers_test.go alongside the six deleted test
files, and remove harness.IAMGroupARN/IAMDeleteGroupBestEffort, which had no
remaining caller anywhere in the repo once iam_groups_test.go was gone.
* feat(integration): daemon-lite harness for key/tags/routetable/vpc
Adds DaemonLite, an in-process stand-in for a live spinifex daemon that
subscribes the real key, tags, route table, and VPC/subnet/SG/IGW service
implementations directly to their NATS subjects, backed by memory and
embedded-JetStream stores. Ports TestKeyPairs, TestTagManagement,
TestRouteTableValidation, and TestReplaceRouteConvergence from
tests/e2e/single so the integration tier drives genuine daemon-side
business logic instead of StubSubject canned replies.
vpcd (OVN) is out of scope, so its "vpc.create-sg"/"vpc.delete-sg"
acknowledgements are stubbed with a fixed success reply — the SG record
itself is still written by the real service impl before that event fires,
so the stub only unblocks the synchronous round-trip rather than
replacing tested behavior.
* ci(e2e): gate live E2E jobs behind the in-process integration tier
Runs the in-process integration suite first, on the ci-single runner host
rather than inside a provisioned VM, so a broken control plane is caught
before either live job pays the Spinifex install. Both e2e_single and
e2e_multi now require it, and e2e_status treats a skipped integration job as
a failure so the gating status cannot go green on a gate that never ran.
Drops the shadow-mode suite selector: targeted per-diff suite selection is
not the direction being pursued, and leaving a non-functional shadow job in
the gating workflow only invited it to be mistaken for a live gate.
* test(e2e): remove single/ tests superseded by daemon-lite ports
TestKeyPairs, TestTagManagement, TestRouteTableValidation, and
TestReplaceRouteConvergence now have proven-equivalent ports in
tests/integration/ (mutation testing confirmed parity). Delete the
e2e originals and their runX helpers, along with keypair_test.go's
and tags_test.go's file-local helpers left with no caller.
generateImportPubKey is shared by other single/ tests so it moves to
helpers_test.go instead of being deleted. Slims the manifest-lint
baseline for the removed create-call entries.
* ci: run the lb e2e suite on multinode only
Both e2e jobs filter SUITES by exclusion, and neither excluded lb, so the
suite ran twice per e2e run: 354s on single and 654s on multi. Balancing
across instances on one node is not a deployment mode we support, and
ELBv2 proper requires two AZs, so the single-node run was redundant.
Drops ~354s from the single-node job, which is the gate critical path.
* test(integration): port 3 of 7 ECR tests to the in-process tier
Wires the gateway's ECR control plane (repository/policy/lifecycle
metadata over an in-process daemon-lite NATS subscriber, blob and
manifest bytes via a memory object store, JWT issuer/verifier keyed off
the existing IAM master key) into the integration harness, and ports
TestECRControlPlane, TestECRGetLoginPassword, and
TestECRIAMAuthorization_AssumedRolePullOnly from tests/e2e/ecr — each
proven equivalent by mutation (evaluatePrincipalPolicy forced to allow,
and the repo-describe path forced to not-found) before its E2E original
was removed.
The other 4 authorization tests (PullOnly, ReadOnly,
DetachPolicyDeniesImmediately, DeactivatedKeyReturns401) stay on the E2E
tier: AttachUserPolicy lacks the AWS-managed-ARN existence-check bypass
that AttachRolePolicy/AttachGroupPolicy already have, so attaching
AmazonEC2ContainerRegistryPullOnly/ReadOnly straight to a user fails
NoSuchEntity in both tiers alike. Left untouched pending a fix to that
shared IAM code.
* fix(iam): accept AWS-managed policy ARNs on AttachUserPolicy
AttachUserPolicy called getPolicyByARN unconditionally, so attaching an
AWS-managed ARN to a user returned NoSuchEntity. AttachRolePolicy and
AttachGroupPolicy already skip that lookup for AWS-managed ARNs and store
them opaquely; the user path was missed when that guard was introduced.
ListAttachedUserPolicies had the mirror of the same gap: it resolved every
ARN through getPolicyByARN and silently dropped the ones it could not find,
so a managed ARN would disappear from the listing rather than round-trip.
Evaluation already handled these correctly via resolveAttachedPolicy, so
only the write and list paths needed the guard.
* test(integration): port remaining 4 ECR authorization tests
Ports TestECRIAMAuthorization_PullOnly, _ReadOnly,
_DetachPolicyDeniesImmediately, and _DeactivatedKeyReturns401 from
tests/e2e/ecr now that AttachUserPolicy accepts AWS-managed policy ARNs.
Each was proven equivalent by mutation before its E2E original was
removed: PullOnly/ReadOnly/DetachPolicyDeniesImmediately fail when
evaluatePrincipalPolicy is forced to always allow, and
DeactivatedKeyReturns401 fails when the access-key-active check in
resolveECRLongLivedPrincipal is bypassed. All mutants were reverted and
a clean run confirmed before committing.
tests/e2e/ecr/iam_authorization_test.go is now fully removed — every
test it held has an in-process equivalent, and its helpers had no other
callers in the package.
* ci: select e2e suites by allow-list instead of deny-list
Both jobs filtered SUITES by exclusion, so any suite named in neither
deny-list ran in both jobs. That is how lb came to run twice per e2e run,
and it makes adding a suite silently fan it out everywhere.
Each job now names the suites it is eligible to run. Selection is still
driven by SUITES, so this is behaviour-preserving for the current default:
single/iam/cert on the single-node job, multinode/lb/cert on the multi-node
job, exactly as before. eks, ecs and storagegrowth are listed as eligible on
the single-node job but are not yet in the default SUITES.
* test(e2e): delete seven tests with no coverage
TestFinalClusterStats and TestInstanceLaunch assert nothing beyond
what other callers of their shared helpers already imply. The
multinode baseline, volume, guest-SSH, and preflight tests duplicate
single-node coverage or exercise per-guest behaviour that isn't
cluster-specific. TestMultinodeInstanceDistribution can't fail: three
independent RunInstances calls with no placement group make a strict
spread assertion flaky and a loose one meaningless.
Preflight checks (/dev/kvm writability, peer SSH reachability) move
into requireMultiNodeFixture so the package fixture singleton still
fails fast with a clear message instead of disappearing along with
the dedicated test. The remaining callers of needInstanceTrio
(CrossNodeOps, CrossNodeGateway, NodeFailure, VolumeDurability) still
trigger the shared trio launch on their own, so removing the
distribution and volume-lifecycle tests doesn't leave it unbuilt.
Also slims the manifest-lint baseline entry for the deleted
multinode volume test.
* fix(e2e): require full 3-node spread in TestMultinodeSpread
SpreadPlacement passed at unique>=3, warned via t.Logf at unique==2,
and only failed below that. A 33% spread failure (two instances
colocated) passed with a log line. The 243.6s NAT-egress scenario that
follows only proves what it claims to if the private trio actually
landed on three distinct chassis, so that premise needs to be a hard
assertion, not a warning.
* refactor(e2e): drop the unused spread-assertion harness helpers
AssertSpreadAcrossNodes and CountInstancesPerNode lost their only caller when
the instance-distribution test was removed. Beyond being dead, the assertion
helper documented minNodes=1 as an acceptable logging-only mode inherited from
the bash driver, which is the pattern that let a placement test pass while
every instance landed on one node.
InstanceHostingNode and its ps helpers stay; five multinode tests use them.
* fix(ec2): stop leaked goroutines on prepareEFIVolume error paths
newViperblock starts the chunk uploader and WAL syncer unconditionally,
but prepareEFIVolume only released them on the happy path via Close().
Six error returns between acquisition and Close left both goroutines
running. Add a defer immediately after acquisition, matching the
sibling prepareRootVolume; Close() remains the durability boundary.
* test(predastore): add real predastore fixture for integration tests
Extract predastore_integration_test.go's inline daemon startup into a
reusable testutil/predastore package so tests/integration can point
DaemonLite-style wiring at a real S3-compatible backend instead of an
unreachable host. Kept as its own leaf package rather than folding into
the shared testutil package: it transitively imports predastore's
quicclient, whose package-level connection pool starts a permanent
goroutine at init that would trip goleak checks in any of the ~80
other packages that import testutil for unrelated NATS helpers.
* test(integration): migrate CreateVolume to assert real predastore persistence
Add StartVolumeDaemonLite, an opt-in DaemonLite variant wired to the
real predastore fixture, and migrate TestCreateVolume_PersistsToRealPredastore
off memory-backed DaemonLite so CreateVolume/DescribeVolumes exercise
viperblock's real S3 backend end to end instead of stopping at input
validation.
* test(integration): add real storage-state oracle for CreateVolume
DescribeVolumes only proves the handler's own read path agrees with its
own write path, both via the same deserialization; it cannot catch a bug
in a field it never re-derives (e.g. a unit-conversion error in the bytes
handed to viperblock, since VolumeMetadata.SizeGiB is copied from the
request directly). vbscan reads a volume's persisted config.json and
chunk objects straight over S3, independent of any spinifex handler code,
so TestCreateVolume_PersistsToRealPredastore can assert on what viperblock
actually wrote rather than only on what the handler claims.
vbscan is a local reimplementation of scripts/vbrefscan's read-only scan
rather than an import: spinifex cannot depend on mulga, the module that
contains it. It only reimplements the config/chunk-listing half of
vbrefscan — the checkpoint-based reference-integrity half needs
viperblock.ParseBlockCheckpointBytes, which does not exist in the
viperblock version spinifex currently depends on (v1.12.1) and is also
moot for this volume: CreateVolume never calls SaveLiveCheckpoint, so a
freshly created volume has no checkpoint to scan regardless of version.
* refactor(test): move the predastore daemon fixture out of the coverage scope
test-cover runs go test ./spinifex/..., so anything under spinifex/ is
compiled and measured. The daemon fixture cannot be exercised by unit
tests, so it reported a coverage gap no unit test could close. The e2e
harness already avoids this by living under tests/; the fixture now
does the same. vbscan stays put, being ordinary library code with its
own unit tests.
* test(integration): migrate RunInstances multi-count expansion off live guests
RunInstances with MinCount/MaxCount=2 expands into a single per-node
targeted launch (distributeInstances in gateway/ec2/instance/placement.go)
before the daemon-facing NATS hop, so the real gateway router plus a
stubbed daemon response is sufficient to prove it — no throwaway VM is
needed to exercise the expansion logic itself.
The live single-node E2E test's WaitForInstanceState/TerminateInstances
sequence was teardown hygiene for the real guests it booted, not an
assertion of the expansion behavior, so it is dropped rather than
emulated. The migrated test additionally asserts the per-node launch
request itself carries MinCount=MaxCount=2, proving the count actually
propagates rather than the launch succeeding by coincidence or via the
zero-daemon degenerate path in placement.go.
* docs: drop a doc reference to the deleted live test
* test(integration): migrate launch-template RunInstances expansion off live guests
RunInstances --launch-template resolves ImageId/InstanceType from the
stored template entirely gateway-side (expandLaunchTemplate, before
validation/placement/the daemon-facing NATS hop), so proving it forwards
the resolved fields never needed a real guest boot. Ports the assertion
to tests/integration via a new opt-in StartLaunchTemplateDaemonLite (real
LaunchTemplateServiceImpl over embedded JetStream) plus a direct
subscription on the per-node launch subject to capture what the gateway
actually dispatched, rather than only the client-visible response.
The live test keeps its CRUD/versioning lifecycle (create/modify/delete
version, tag-filtered describe) — none of it touches a guest — and drops
only the RunInstances-from-template + wait-for-running + terminate tail,
whose only job was waiting out a real boot to read back fields already
proven resolved.
* test(integration): migrate KV-only account-scoping checks off live tenants
Ports the KeyPair/VPC-Subnet/IGW-EIGW/AccountSettings cross-account
isolation assertions from tests/e2e/single/account_test.go to the
in-process integration tier. Each of these resources resolves ownership
from a plain account-scoped KV key (utils.AccountKey(accountID, id), or
an account-keyed settings record) with no vm.Manager or real guest
involved, so DaemonLite now also wires the real EgressOnlyIGWServiceImpl
and AccountSettingsServiceImpl alongside the existing VPC/IGW/Key
services. Two independently minted tenant accounts (via
IAMService.CreateAccount/CreateUser/CreateAccessKey/CreatePolicy/
AttachUserPolicy, in-process — the same calls runAccountCreate makes
without the CLI/stdout round-trip) replace the live tier's carousel so
isolation is checked between real, separately owned accounts.
Left live: instance scoping (Step2) and volume scoping (Step3) —
checkInstanceOwnership, the actual cross-account gate for stop/start/
reboot/terminate/attach-volume, runs daemon-side only after a real
vm.Manager lookup, so it cannot be exercised without a live guest.
Snapshot scoping (Step5) needs a real backing volume for CreateSnapshot
(viperblock, out of DaemonLite's scope), so it stays live entirely rather
than split off only its two KV-only assertions. Global-resources (Step9)
carries no account branching in the handlers at all, so asserting its
equality across two accounts wouldn't exercise anything a fake node
responder didn't already trivially satisfy — not migrated.
* test(integration): migrate launch-template CRUD lifecycle off live guests
TestLaunchTemplates booted zero guests: its RunInstances coverage had
already moved to TestRunInstances_LaunchTemplateExpansion, leaving only
template CRUD/versioning assertions occupying a full live-cluster slot.
Port the remaining create/describe/version/modify/delete lifecycle,
including the SourceVersion merge and a nested timestamp round-trip, into
TestLaunchTemplates_CRUDLifecycle against the in-process integration
gateway.
* refactor(tests): move spot instance lifecycle coverage to integration tier
RequestSpotInstances is a mock over on-demand RunInstances (no bidding,
interruption, or reclamation), so booting real VMs in the e2e tier proved
nothing spot-specific. Migrate the SIR CRUD lifecycle (request, describe,
cancel, close-on-terminate, insufficient-capacity error) to a NATS-level
integration test that exercises the real gateway orchestration and daemon
service impl without a live guest. Real-VM-boot assertions are already
covered by TestRunInstances_MultiCountExpansion and the e2e baseline tests.
* fix(tests): share one embedded NATS server across the integration package
ClientTokenStore in spinifex/gateway/ec2/instance is a process-wide
sync.Once singleton bound to whichever *nats.Conn first initialises it.
Every integration test booted its own embedded NATS server, so the first
RunInstances-calling test captured the singleton, and any later test
inherited a KV handle bound to that server's now-closed connection.
TestMain now boots one embedded NATS+JetStream server for the whole
package; every StartGateway call connects into its own dynamically
provisioned NATS account (accountAuthenticator in shared_nats.go), giving
each test a private JetStream namespace despite the shared server, and
none of the per-test connections are closed until every test has finished
- whichever one the singleton latches onto stays live for the run.
Also wire the suite into the required PR gate: it was previously only
invoked from e2e.yml's push-triggered (main/dev), self-hosted "Integration
(front gate)" job, so a regression here never blocked a PR. The whole
package runs in single-digit seconds, so it now runs inline in
`make preflight` and as its own step in spinifex.yml's build_and_test job.
* test(integration): migrate serial-console access checks off live guests
Enable/Disable/GetSerialConsoleAccessEnabled is pure gateway+account-settings
logic with no guest involved, so it moves to the in-process integration tier
against the real AccountSettingsServiceImpl wired into DaemonLite.
* test(integration): migrate DescribeImages round-trip off live guests
runImage's AMI describe round-trip needs no real cluster: it registers an
AMI against a hand-seeded snapshot via the real ImageServiceImpl, backed by
a new StartImageDaemonLite helper wiring RegisterImage/DescribeImages onto an
in-memory object store.
* test(integration): migrate predastore object lifecycle off live guests
runPredastoreObjectLifecycle only needs a real predastore daemon, not a
tofu-provisioned cluster: it now runs against tests/fixtures/predastore's
in-process fixture, covering the same create/put/get/delete/list contract.
* test(integration): migrate ListFoundationModels catalog check off live cluster
ListFoundationModels is pure gateway-side static catalog logic with no
NATS/daemon hop, so it moves to the integration tier via a new BedrockClient
harness accessor; this tier configures no credential resolver, so the
Anthropic entry's absence is asserted explicitly rather than left optional.
* test(integration): cover RunInstances client-token idempotency
Proves replaying the same ClientToken through the real gateway returns the original reservation without a second daemon dispatch, and a different token launches independently.
* test(integration): cover RunInstances launch-template expansion
Proves version selection (omitted Version resolves to the template default, explicit Version selects that version) and field precedence (explicit RunInstances parameters override the template, unset fields inherit it) through the real gateway.
* test(integration): cover RunInstances malformed-input rejection
Proves a malformed ImageId and a malformed targeted-launch CapacityReservationId are rejected before any NATS dispatch, with no daemon stub registered so a regression that let either through would fail fast instead of hanging.
* test(integration): cover RunInstances iam:PassRole enforcement
Proves a caller granted ec2:RunInstances but not iam:PassRole on an instance profile's role is denied, and the identical session succeeds once PassRole is granted for that exact role ARN.
* test(integration): cover RunInstances placement-group strategy routing
Adds a placement-group daemon-lite wiring a real PlacementGroupServiceImpl over NATS, then proves spread strategy assigns exactly one instance per node while cluster strategy pins the whole launch to the single highest-capacity node.
* test(integration): cover RunInstances capacity distribution across nodes
Proves the default (non-placement-group) path packs a multi-instance launch across nodes proportionally to spare capacity rather than exhausting one node first.
* test(e2e): merge SameSGComms and SGToSGDatapath around one client/target pair
Both booted their own instance pair to check SG-to-SG datapath enforcement
and same-default-SG east-west connectivity. Merging them around one shared
client/target pair cuts four boots to two, and converting the inline
harness.Step sequence into named t.Run stages (PortGroupMembership,
AllowedTraffic, DeniedTraffic, RevokeAndVerifyBlocked,
ReauthorizeAndVerifyRestored, SameSGComms) gives each its own pass/fail
attribution instead of one opaque test result.
SameSGComms now runs last and joins the default SG onto each ENI via
ModifyNetworkInterfaceAttribute only after every SG-to-SG allow/deny
assertion has already recorded its own result, since the default SG's
self-reference rule admits all traffic between co-members and would
otherwise defeat the denied-traffic check.
* test(e2e): merge default-SG reachability, guest DNS, and egress enforcement
DefaultSGReachabilityBaseline, GuestDNSResolution, and SecurityGroupEgress
each booted their own instance to check one facet of "a guest is reachable
only when policy says so": ingress deny/authorize, resolver + WAN egress,
and egress ACL enforcement. Merging them around one shared guest cuts
three boots to one, with named t.Run stages (Deny, Authorize, Resolver,
WANEgress, RevokeEgress/RestoreEgress) giving each its own attribution.
The egress revoke/restore round-trip now mutates this scenario's own
dedicated SG instead of the shared default SG the rest of the suite
depends on, isolating its egress-rule mutation from every other test.
* fix(ci): single-quote e2e test_run regex across the ssh boundary
* fix(e2e): anchor SG egress-enforcement probe to a WAN IP, not the gateway
* test(e2e): merge single-node guest churn tests into one durability scenario
Combines TestSSHProbe, TestENIHotplug, TestENIHotplugReconcile,
TestVolumeDurability, TestModifyInstanceAttribute, and TestRebootInstance
into TestGuestChurnDurability, a single shared-guest scenario that
re-verifies a data-integrity sentinel sha256 after every perturbation
stage (ENI hotplug, daemon restart, ENI detach, instance-type resize,
stop/start, reboot). This proves the sentinel survives the full churn
sequence rather than just one isolated stop/start.
* fix(vpc): clear subnet drop gate when installing egress reroute
A subnet with an attached IGW but no 0.0.0.0/0 route carries a
priority-1100 drop gate on the VPC router. When a route later lands
(NATGW reroute at 900, IGW reroute at 1000) the reroute is installed but
the drop gate was never removed, so the higher-priority gate wins and
black-holes the subnet's egress in lr_in_policy before it reaches routing
or SNAT. This left NAT-gateway private-subnet egress with 100% packet
loss: the SNAT stage saw zero packets and no conntrack was committed.
Remove the subnet's drop gate for the same prefix once a reroute is
installed. Idempotent and prefix-scoped, so a specific-prefix reroute is
a no-op and leaves the default gate intact.
* test(e2e): merge VPC subnet, NAT gateway, and EIP single-node tests
Merges the former TestVPCSubnetE2E, TestNewVPCEgressBaseline,
TestNATGateway, and TestInstanceEIP into one staged scenario,
TestVPCEgressPaths, around one scenario-owned VPC and its two guests,
cutting five guest boots to three.
The public guest serves PublicSubnetEgress, then the NAT Gateway
bastion hop, then (last, since association rewrites its public IP)
the EIP flip. The private guest serves the NAT Gateway path, launched
directly into the same VPC's already-existing private subnet instead
of a dedicated one. RouteBeforeSubnet keeps its own second VPC and
instance since it exercises the implicit main-route-table code path,
a different one than the explicit route table this scenario's own VPC
uses for its public subnet — collapsing the two would drop coverage of
whichever path got merged away.
The NAT Gateway up/down flip now repeats for an env-tunable round
count (SPINIFEX_VPCEGRESS_NAT_ROUNDS, default 1), matching every other
merged single-node scenario.
* test: TEMPORARY mutation-check — revert a795c2b2 drop-gate clear
Reverts the RemoveSubnetEgressDrop call in ensureSubnetEgressAtPriority
to confirm TestVPCEgressPaths/Round1/NATGatewayUp actually detects the
regression it guards against. This commit will be reverted immediately
after the live run confirms a failure.
* test: TEMPORARY mutation-check v2 — neutralize RemoveSubnetEgressDrop
Restores the real a795c2b2 fix in ensureSubnetEgressAtPriority (the
first mutation attempt only reverted this call site and was masked by
the independent gate-recompute path in routetable service_impl.go's
CreateRoute, which also clears the drop gate on every 0.0.0.0/0 route
add and predates a795c2b2). Neutralizing RemoveSubnetEgressDrop itself
disables gate-clearing regardless of caller, reproducing the literal
regression. Will be reverted immediately after live verification.
* Revert "test: TEMPORARY mutation-check v2 — neutralize RemoveSubnetEgressDrop"
Mutation-check confirmed: neutralizing RemoveSubnetEgressDrop reproduced
a795c2b2's regression exactly (stale priority-1100 drop gate outranking
the priority-1000 IGW reroute, confirmed via OVN flow dump; PublicSubnetEgress
failed with an SSH handshake timeout to the public instance). Restoring
the real fix; igw.go is now byte-identical to a795c2b2.
* fix(e2e): stop pruning shared multinode SSH key pem mid-run
The multinode package fixture cached Fixture.Artifacts once, computed
from harness.ArtifactDir(t, env) against whichever top-level Test*
happened to win the pkgFixOnce race. That helper both names the
directory after that one test and registers a t.Cleanup that removes
it once that test passes. needKeyPair wrote the shared EC2 key pair's
pem into that same directory, so as soon as the winning test finished,
its own artifact-dir cleanup deleted the pem every later test in the
package still needed to SSH into fresh guests, even though the
underlying EC2 key pair itself was untouched (its DeleteKeyPair
cleanup already correctly drains at process Close() in TestMain).
Fix: give Fixture an ArtifactDir(t) method — the same per-call pattern
already used by the gpu/single/storagegrowth/iam fixtures — so each
diagnostic dump site recomputes its own dir against the CURRENT test.
The shared key pair pem now lives under Fixture.TmpDir, which is
genuinely package-scoped and only removed once in TestMain after
m.Run() completes.
* refactor(e2e): parallelize LB e2e scenarios on multinode clusters
Each of the 8 TestLoadBalancer scenarios already stands up its own
dedicated LB + target group (ModifyListener/ListenerRules mutate their
LB's listener in place, so it can never be shared). The bottleneck was
running them serially: each teardown made createActiveLB back off
waiting for the previous LB's sys.micro slot to be reclaimed before the
next one could stand up.
On multinode clusters (peer != "", i.e. the 3-node cluster this suite
targets) there's spare capacity to stand up all 8 dedicated LBs at once,
so scenarios now run as t.Parallel() subtests instead of serial
create/teardown/backoff. Since each scenario owns an independent LB,
one never reaching active only fails that scenario — it no longer
cascades into five sequential 5-minute timeouts, so the serial fail-fast
gate is skipped in this mode; concurrency itself bounds the worst case
to a single timeout window.
Single-node dev boxes (no peer) keep the original serial path,
including the fail-fast gate, since they lack the spare capacity for
concurrent LB standup. No subtest body, assertion, or call signature
changed — only the dispatch/gating logic in TestLoadBalancer.
* fix(ci): gate e2e jobs on the suites they actually run
* fix(e2e): serialize internet-facing LB subtests among themselves
Live multinode verification showed InternetFacing_ALB and
InternetFacing_NLB running concurrently each clobber the other's
public-LB DNS record mid-registration, so neither ever resolves within
the 90s wait. The old serial suite never had two internet-facing LBs
alive at once, so this never surfaced.
Drop t.Parallel() from the three internet-facing subtests so their
register+resolve windows never overlap each other, while leaving the
internal subtests parallel as before. They may still run alongside the
internal group, which resolves internal (not public) DNS and isn't
affected.
* fix(e2e): give each ALB LB scenario its own security group
Concurrent internal subtests all created ALBs with no explicit
SecurityGroups, so each ALB's ENI fell back to the one shared VPC
default SG. One scenario authorizing/reconfiguring its own listener
racing another mutating the same SG made Internal_ALB_ModifyListener
intermittently fail to serve on the modified port under parallel load.
Each ALB scenario now creates a dedicated SG (createScenarioSG),
authorized only for the ports that scenario's listener(s) use, and
passes it to CreateLoadBalancer. The shared app-instance/probe-client
SG is untouched, since SG ingress is enforced at the destination ENI
regardless of the source ENI's SG. NLBs are unaffected: they already
get an auto-managed per-LB SG when none is supplied.
* test(e2e): annotate intentional create-path calls for manifest-lint
The vpcegress and per-scenario LB security-group tests create raw AWS
resources on purpose — the egress datapath, the route-before-subnet
ordering, and per-scenario SG isolation are the subjects under test, so
they cannot route through a shared Ensure* fixture. Mark each with
// e2e:allow-create and slim the baseline of entries for the removed
single-node network tests.
* test(e2e): restore live spot terminate-close and cancel-running coverage
The integration-tier spot test calls CloseForInstance directly and infers
cancel-keeps-running from the SIR's InstanceId, because DaemonLite has no
vm.Manager to drive the real TerminateInstances -> daemon teardown cleaner
-> RemoveFromSpotRequest -> CloseForInstance chain or to observe a real
VM's running state. Add a thin live e2e test that drives both terminal
transitions through a real backing VM: cancel one spot-backed instance and
observe it stays running, terminate the other and observe its SIR reach
closed through the genuine async delegation.
* fix(network): refresh orphan-EIP prune liveness to spare mid-pass launches
pruneOrphanEIPs listed OVN NAT rows live but matched them against the intent
snapshot captured at the start of the reconcile pass. Because the apply phase
can block for tens of seconds on guest-port datapath waits, a guest launched
during the pass had a live dnat_and_snat row absent from the stale snapshot, so
the prune swept it and blackholed the guest's auto-assigned public IP until the
next pass re-added it — an intermittent multinode SSH-dark flake.
Add an optional FreshIntent re-read callback (wired in vpcd to LoadIntentFromKV)
and union its ports into the live set before pruning, so a mid-pass launch
counts as live. Skip the prune entirely when the re-read fails rather than risk
a false sweep against a snapshot known to be stale. Self-healing and idempotent:
a genuine orphan absent from both snapshot and fresh read is still pruned next
drift. Single-node behaviour is unchanged (callback nil in tests).
* feat: plumb GCEnabled config through to every VB construction site
viperblock chunk GC is gated behind VB.GCEnabled, but spinifex never
set it, so every deployed node ran with GC off by construction
regardless of which viperblock binary shipped. Add a
[viperblock] gc_enabled config key (default false, mirrors shardwal)
and thread it through to the three places spinifex builds a VB: the
nbdkit plugin it spawns per mounted volume (via a new gc_enabled
plugin param), viperblockd's detached-volume opens, and the volume
service's CreateVolume.
* chore: repin viperblock to GC-enable lineage
The gc-enable plumbing references viperblock.VB.GCEnabled, which is
absent from the previously pinned v1.12.1. Repin to the feat/nbd-gc-enable
commit so a standalone (GOWORK=off) build — the path the ansible deploy
uses — resolves the field, and spx plus the nbdkit plugin build from the
same GC-bearing viperblock lineage.
* fix(storagegrowth): stop launching a nano for image-pull repro
ensureImagePullInstance shared harness.DiscoverNanoInstanceType with the
synthetic npass suite. A 512MB guest OOM-thrashes containerd before any
disk-driven ENOSPC and conflates the storage leak with guest-memory
pressure. Add a package-local imagePullInstanceType helper: an exact
override via SPINIFEX_STORAGEGROWTH_INSTANCE_TYPE, otherwise the smallest
advertised type at or above SPINIFEX_STORAGEGROWTH_MIN_MEM_MIB (default
16384 MiB). npass keeps using the nano selector unchanged.
* feat: report guest block write errors so backend ENOSPC reaches the guest
Set werror=report and rerror=report on the guest boot drive so a viperblock
ENOSPC (or any other backend write failure) is surfaced to the guest as a
real I/O error instead of QEMU silently pausing the VM. Without this the
guest kernel never learns its disk is out of space, so it can't fail
writes cleanly or let an operator react.
* chore: repin predastore and viperblock to the storage-capacity-safety lineage
Drop the local-path replace for predastore (a test shortcut) and pin both
predastore and viperblock to their feat/storage-capacity-safety commits, so
a standalone (GOWORK=off) build resolves the free-space watermark, ENOSPC
classification, and backpressure plumbing that spinifex's werror/rerror
change depends on.
* chore: bump viperblock to 054a44a3f0f5
Realigns spx's linked viperblock with the submodule working tree, which the
build role's source-drift gate compares against.
* chore: repin predastore and viperblock to their dev lineage
Both were pinned to pseudo-versions off feat/storage-capacity-safety. That work
has now landed on dev in each repo, and both PRs were squash-merged, so the
branch commits those pins named are unreachable from any branch.
Repin to the dev commits carrying the same code: predastore 5699329 and
viperblock 35dd9f2.
Northstar carried working northstar.dns.* instruments but shipped no
telemetry, so its dashboard was empty and the only northstar data in
Elasticsearch was journald scraped under the systemd unit name.
Two gaps, both outside the instrumentation. northstarStartCmd never called
initTelemetry, unlike every other service command, so the instruments
recorded into a no-op provider. The unit also did not source
/etc/spinifex/telemetry.env, which every other application unit reads for
OTEL_EXPORTER_OTLP_ENDPOINT, MULGA_ENV and MULGA_SOURCE — so even an
initialised SDK would have had no endpoint and no env label.
Add an invariant covering the second gap: any unit launching an spx service
must source the telemetry drop-in. A service missing it starts healthy and
silently reports nothing, which only surfaces when someone notices an empty
dashboard. Verified the test fails against the unpatched unit.
Add e2e_es_summary, a self-hosted job that aggregates suite-status.txt
from both e2e legs and POSTs one doc per run to the Elastic sink, keyed
by labels.ci_run_id, so Kibana can answer "did run X pass, and where
are its artifacts" without hand-aggregating traces-apm*/logs-apm.error*.
Runs on ci-single (LAN route to the sink) since e2e_status stays on
GitHub-hosted ubuntu-26.04 for the PR status-report check. State is
computed with the same single_ok/multi_ok logic as e2e_status so the
two can never disagree.
* feat(bedrock): Ochre streaming — ConverseStream + InvokeModelWithResponseStream
Adds the two streaming bedrock-runtime operations behind a shared
eventstream.Encoder frame writer: ConverseStream reframes vLLM/Anthropic SSE
into the normalized Bedrock event taxonomy (messageStart -> contentBlock*
-> messageStop -> metadata), InvokeModelWithResponseStream wraps each
provider-native chunk in a PayloadPart frame. Fixes otelsetup's
statusRecorder to implement Flusher/Unwrap so flushes actually reach the
socket through the middleware chain, which streaming depends on.
* fix(bedrock): keep vLLM ConverseStream taxonomy ordered on edge cases
Route contentBlockStop/messageStop through an idempotent ensureStarted so a
usage-only chunk arriving before any finish_reason no longer emits metadata
ahead of messageStop, and a zero-content stream no longer emits stop events
without a preceding messageStart. Regression tests for both.
* fix(bedrock): route /model/ data-plane calls to bedrock-runtime
AWS SDKs sign bedrock-runtime with the SigV4 signing name "bedrock", the
same as the control plane; the two services are separated only by endpoint
hostname. The gateway serves one endpoint and routed purely on the signing
name, so every /model/{id}/converse*|invoke* call landed in Bedrock_Request
(control plane) and returned InvalidAction — no standard SDK/CLI could reach
any bedrock-runtime endpoint.
Disambiguate by path in GetService: a "bedrock"-scoped request to a /model/
path (data-plane-exclusive) resolves to bedrock-runtime. Control-plane paths
(/foundation-models, /custom-models, /model-invocation-job) are unaffected.
* test(bedrock): satisfy testifylint go-require/empty in streaming tests
Use assert (guarded where a value is dereferenced afterward) instead of
require inside httptest handler closures, and assert.Empty for empty-string
checks, per dev's stricter testifylint config.
The Build/Lint/Race jobs checked out viperblock and predastore at a hardcoded
ref: dev and overlaid them with a go workspace, so the go.mod pins were ignored
and a spinifex change depending on an unmerged sibling change could never
compile in CI. Resolve each sibling to a branch of the same name when it exists
on that repo, else dev, mirroring the e2e path in
scripts/tofu-cluster/build-install-artifacts.sh so unit CI and e2e CI agree.
Treat the ECR registry JWT as a signed identity pointer rather than a
permission grant. Every /v2/* request now rehydrates the token's access
key against current IAM/STS state, maps the OCI route to an ecr: action
and repository ARN, and runs the identity-policy evaluator before
dispatch. PullOnly/ReadOnly principals can no longer push, overwrite
tags, or delete manifests, and key/session/user/role/account/policy
revocation takes effect on the next request instead of at token expiry.
Cross-repository blob mounts now require source-read permission and
proof the digest is referenced by a manifest in the named source repo,
closing an account-wide blob-fishing gap.
A CreateLoadBalancerSync whose data-plane launch failed returned without
rolling back the NLB managed security group. The orphaned SG sat in the EKS
managed control-plane VPC and blocked DeleteVpc with DependencyViolation
indefinitely, wedging the cluster in DELETING. The failure was also flattened
to ServerInternal, discarding the only useful diagnostic.
The sync path now unwinds fully on launch failure (ENIs, managed SG, name
claim, record) and maps the launch error onto its real cause: capacity
shortfalls pass through as their own code, anything else stays ServerInternal.
The async path deliberately keeps its failed record — the caller holds the ARN
and can reclaim it via DeleteLoadBalancer.
The image-pull storage-growth workload HEADed <volID>/checkpoints/blocks.live.bin
with the operator box's gateway identity, which has no S3 read grant on the
bucket and was refused 403 on every sample, so checkpoint_bytes read 0 across a
whole run. It now HEADs the object on the node over PeerSSH using predastore's
own [[auth]] credentials from predastore.toml (root-owned, so read via
passwordless sudo; the CA is readable through the spinifex group). The
credentials are read into shell locals, used inline for one aws s3api
head-object, then unset — never echoed, never leaving the node. A probe failure
that is not a legitimate 404 voids the sample (checkpoint_bytes -1 plus a probe
error) rather than reporting a false zero, so a dead instrument can never look
like no growth. The dead operator-side S3 client path is removed.
Align the top of the README with mulgadc.com: new headline and intro,
rewritten "What is Spinifex?", a new "Three ways to deploy" section
(Neocloud / on-premise / edge) and a new "AWS compatibility" service
matrix (EC2, EBS, S3, VPC, IAM, ALB/NLB, EKS, ECR, ECS available; RDS
Q3 2026) linking the GitHub releases. Refresh the architecture prose and
Key Features, add Deploy and AWS-services nav links, and swap banner.svg
for the ember-on-ink mini-stack design matching platform.svg.
Rebuild the message-driven architecture diagram in the ember-on-ink
theme to match platform.svg: ember gradient NATS bus, ember-tinted
gateway and daemon cards, neutral clients and backend cards. Content
unchanged (clients → gateway → NATS → ec2/ebs/s3/vpc/iam daemons →
local backends). De-slop the copy and refresh the img alt text.
Replace the platform diagram with a static SVG matching the mulgadc.com
front-page stack diagram: ember accent, ink panel, the portable-stack
box (Neocloud / on-prem / edge) wrapping the Spinifex AWS-compatible
service layer, over standard Linux on CPU/GPU hardware. Self-contained
and GitHub-safe (presentation attributes only). Refresh the img alt text.
* fix(admin): round the AMI root volume up so the image fits
`spx admin images import` sized the root volume with integer division, which
floors: a 4.94 GiB image registered a 4 GiB volume. The image is then copied
into a volume too small to hold it, is truncated, and the guest boots with no
root partition — it stalls on the root device until systemd gives up, and
nothing anywhere reports an undersized volume.
floorVolumeSizeToAMI already exists to prevent exactly this ("Under-sizing
hangs the guest in dracut"), but it raises the caller's requested size to the
AMI's recorded VolumeSizeGiB — the same floored value — so it inherited the
mistake instead of catching it.
This stayed hidden because every current system image is an exact multiple of
1 GiB, where flooring and rounding up agree. It surfaced on a 5306470400-byte
image, which would not boot.
Round up instead, via a helper so the boundaries are testable: an exact
multiple must not gain a spare GiB, and one byte over must.
* fix(eks): default a nodegroup's root volume to 20 GiB, as AWS does
CreateNodegroup stored diskSize verbatim, so omitting it persisted 0 and
the launch skipped its block device mapping entirely, dropping the worker
onto whatever volume the AMI itself declares.
That was harmless while the node AMIs baked generous virtual disks — the
GPU image declares 16 GiB for ~2 GiB of content, so the fall-through
happened to leave plenty of room. It stops being harmless as soon as an
image is sized to its contents: the worker then boots onto a volume with
barely a spare GiB and cannot pull the container images it exists to run.
Sizing the node is the nodegroup's decision either way, not a property
the image should be able to dictate by accident.
20 GiB matches the AWS EKS managed-nodegroup default. It is resolved at
create rather than at launch so DescribeNodegroup reports the size the
node actually gets.
TestCreateNodegroup_NoDiskSizeOmitsBlockDeviceMapping asserted the old
fall-through and is replaced by its inverse; the explicit-size case is
already covered by TestCreateNodegroup_DiskSizePropagatesToBlockDeviceMapping.
* fix(ecs): size a container instance's root volume to 30 GiB by default
ProvisionCapacity built its RunInstances input without a block device
mapping, so a container instance inherited whatever volume its AMI
declares. EC2 has no default of its own to fall back on — the AMI is the
only thing that decides — and a node AMI is sized to hold the image, which
leaves the agent no room for the container images it exists to pull.
30 GiB mirrors what the AWS ECS-optimized AMI declares. ProvisionCapacity
is spinifex's own API rather than an AWS one, so DiskSize is added to its
input for callers that want to size a cluster explicitly, defaulting when
omitted the same way InstanceType and Count already do.
* fix(images): pull the Ubuntu catalog images from releases, not a pinned daily
Both Ubuntu catalog entries pointed at a dated build under the codename
dailies tree, which Canonical prunes as builds age out. That path now 404s,
so "spx admin images import --name ubuntu-26.04-x86_64" fails outright for
x86_64 and arm64 alike — the entries had rotted with nothing to catch it.
Release builds are retained rather than pruned, so the pin stays resolvable
while keeping a specific, reproducible build. The filenames under releases/
carry the version instead of the codename, which matters because the sums
entry is matched on the URL's basename.
CI is unaffected and could not have caught this: it fetches resolute/current/,
a symlink that never rots, so nothing exercised the pinned path.
Debian pins a dated directory too but is fine — those builds are retained,
and its entries still resolve.
Pulls a container image inside a guest onto a viperblock volume and samples
guest used bytes, host backing-store bytes, nbdkit RSS and the live
checkpoint size as it goes, so backing-store cost can be fitted against the
bytes the guest actually wrote.
Complements the N-pass overwrite workload, which rewrites one extent and so
pins the block map size — that shape cannot show whether cost grows in
proportion to bytes written or faster than them.
Redirects both dockerd's data-root and containerd's root onto the target
volume. Redirecting data-root alone leaves containerd extracting layers into
/var/lib/containerd on the AMI's small root disk, which fails with ENOSPC
while the target volume sits empty.
The storagegrowth gate called ResolveCACert unconditionally before
building the AWS client, even though newAWSClient itself skips CA
resolution entirely when SPINIFEX_AWS_INSECURE=1. That blocked the
documented runner-resident remote path (env12 and similar clusters
with no local CA cert on disk) and the skip message falsely claimed
no node was running when the real cause was a missing CA. Skip the
CA check under SPINIFEX_AWS_INSECURE=1 and state the actual cause
plus the escape hatch in the skip message.
Rewrites one logical extent N times through full attach/format-write/detach
cycles, so the guest-visible footprint stays constant while total bytes written
scales with N. That shape isolates the leak where viperblock mints chunk objects
under unique monotonic keys and never deletes the superseded ones: a sequential
write-once supersedes nothing and so cannot exercise it.
The suite drives the workload only and deliberately does not measure the
backend. Measuring on-disk bytes from here would mean either a host du (which
cannot separate live from dead bytes, and so cannot distinguish this leak from
the unrelated one where an overwritten key's old extent is orphaned without a
tombstone) or a dependency on tooling outside this repository, which would stop
the suite building standalone. It instead reports what it did to a caller-named
result file and asserts only what a guest can observe: a sha256 round-trip
through a fresh mount per pass, and a footprint that stays constant across them.
The volume is retained on request, because deleting it removes the evidence:
DeleteVolume enumerates the bucket by prefix and drops every key under it,
including the leaked chunks.
Phase 8 of TestNATUplink checked the second VPC router's lr-nat-list with a
single non-polling assert; the igw-default-snat row commits a beat after the
gateway LRP appears, so the read raced the OVN commit and intermittently saw
an empty list (flaky CI red). Wrap it in EventuallyErr, matching the sibling
LRP and host-route polls in the same phase.
AddSNAT logged only on the idempotent-skip path, so the first install was
silent and RCA could not tell a late SNAT from a missing one. Emit an install
log keyed by router + vpc_cidr + external_ip.
Server-cert IP SANs were built solely from live interface enumeration
(DiscoverLocalIPs) plus bindIP, so the canonical mgmt-bridge IP entered
the SAN set only when br-mgmt happened to be up at cert-generation time.
On a host where br-mgmt is down when the cert is minted, 10.15.8.1 was
silently dropped and the EKS control-plane publish to https://10.15.8.1
failed cert verify, leaving the cluster stuck CREATING.
Pin config.DefaultMgmtBridgeIP into the extra SANs for both the init and
join cert paths so publish succeeds regardless of interface state at
generation time. Add a regression test asserting the SAN is present when
br-mgmt is not a live interface.
Add an IRSAPod subtest that applies a real pod wired for IRSA — a projected
ServiceAccount token (aud sts.amazonaws.com) plus the AWS_ROLE_ARN /
AWS_WEB_IDENTITY_TOKEN_FILE / AWS_ENDPOINT_URL_STS / AWS_CA_BUNDLE env the
addon-sync agent injects — and asserts the pod's in-cluster
aws sts get-caller-identity resolves to the assumed IRSA role. This closes the
gap left by IRSAWebIdentity, which only exchanges the token host-side.
Extract registerOIDCRole (shared with IRSAWebIdentity); the pod runs on a
1-node worker nodegroup (the control-plane node is CriticalAddonsOnly-tainted)
and reaches awsgw STS over the worker subnet NAT.
The build script derives BUILD_DIR from SYSTEM_IMAGE_BUILD_DIR (the AMI
publish workflow points it off tmpfs so the 16G Ubuntu GPU raw fits), but
the publish step hardcoded /tmp and could not find the freshly built raw.
GPU legs failed 'Raw image not found'; Alpine legs silently reused stale
pre-existing /tmp raws on the persistent runner. Mirror the build script's
derivation so publish looks where the build actually wrote.
The publish-eks-ami matrix fans out four legs on the single persistent
ci-single runner, all reusing one workspace root. go work init is
create-only and checkout never scrubs the root go.work, so legs after the
first fail with 'go.work already exists'. Remove any stale go.work before
init so each leg re-inits cleanly.
Expand the publish workflow to build the two Ubuntu NVIDIA GPU variants
alongside the Alpine eks-node/ecs-agent AMIs. The image build+publish
scripts are already manifest-driven and Ubuntu-aware; only the workflow
lacked the GPU legs.
Refactor the single-job body into a matrix over the four image manifests,
resolved by a new 'variant' input (all|cpu|gpu) so the slow 16G GPU legs
can be skipped when only the CPU AMIs need republishing. GPU images do not
fit a RAM-backed /tmp, so set SYSTEM_IMAGE_BUILD_DIR onto the runner's
on-disk _temp; legs run serially (max-parallel 1) on the single ci runner.
The nodegroup label was only written to K3S_NODE_LABEL in agent.env, but
k3s has no such env var, so non-GPU workers never received the label.
waitWorkersReady buckets Ready nodes by that label (added in #511), so
the affected nodegroups never counted toward their ACTIVE transition and
stayed CREATING until timeout.
Ride the label on a real k3s node-label config drop-in
(15-nodegroup-label.yaml), delivered via cloud-init user-data. Drop the
inert K3S_NODE_LABEL from agent.env and the initd export.
WaitForInstanceState now aborts immediately with the recorded StateReason
message when it observes a reaped state (shutting-down/terminated) while
waiting for a live target, instead of burning the full 5-minute timeout on
an instance that can never reach it.
MarkFailed takes a context and records the failure on the active span via
span.RecordError so a reaped launch surfaces in the APM error stream,
correlated by trace id. Only the LaunchRunInstances path carries a live
span; other callers pass context.Background() and the record is a no-op.
Surfaced by RCA of E2E run 29258398516 (TestAccountScoping/Step2).
TestNATGateway flaked with 100% egress loss: the tenant VPC gateway router's
OVN MAC_Binding had no entry for its upstream default-route nexthop. OVN
resolves that nexthop via lazy dynamic ARP on first egress; if that ARP is
lost during gateway bring-up, SNAT'd packets die in lr_in_arp_resolve and no
reply returns. MAC_Binding is per (datapath, logical_port, ip), so a sibling
VPC router resolving the same nexthop does not help.
Deterministically seed a static MAC binding for the nexthop at IGW attach. The
nexthop MAC is the host's own default-gateway MAC (the host shares the physical
uplink), read from ip neigh and written via ovn-nbctl static-mac-binding-add
(northd programs it into SB MAC_Binding, GC-safe). Host layer L0 owns the
shell-out per ADR-0006 S2; igw calls it via an injected hook. Best-effort and
idempotent: a missing MAC leaves dynamic ARP as the fallback, and it re-runs on
every AttachIGW so a binding lost to a later flap self-heals.
utils.GetOrCreateKVBucket hardcoded Replicas=1, so the ~38 handlers that
create their KV buckets lazily (VPC/subnet/ENI/EIP/ELBv2/ECS/ECR/ACM/
natgw/snapshot/placementgroup/IPAM/account-settings/dhcp/static-pool …)
created them at R1 on a multi-node cluster. upgradeJetStreamReplicas runs
once at startup, so any bucket created lazily after boot stayed R1
permanently — and an R1 stream's single Raft replica is transiently
leaderless during formation/join/rebalance, surfacing as
ErrNoResponders/ErrTimeout on KV ops.
Add a package-level default replica count that the daemon sets once at
boot (startCluster, after initJetStream) to len(clusterConfig.Nodes),
before any handler creates a bucket. GetOrCreateKVBucket now creates at
that default, so lazily-created buckets are born quorate. The one-shot
startup UpdateReplicas remains as a safety net for pre-existing R1
buckets from older deploys.
Zero call-site churn: the default flows through the existing
GetOrCreateKVBucketWithReplicas path already used by eks/ecs/iam.
TestSpawnRegisteredReconcilers_ResumesNonTerminal flaked two ways. First,
the shared fixture meta stamps a weeks-old CreatedAt, so a resumed CREATING
reconciler instantly trips the CREATE timeout, marks the cluster FAILED and
deregisters before the registry.Has assertion runs — the test was racing a
self-removal. Stamp a recent CreatedAt so the reconciler stays CREATING.
Second, SpawnRegisteredReconcilers ran a one-shot JetStream enumeration
(KeyValueStoreNames + kv.Keys), which is eventually consistent: under load a
just-committed cluster can be missing and never resumed, silently on a real
daemon. Re-scan until the observed CREATING/ACTIVE count stops growing, capped
and short-backed-off; Spawn is idempotent so re-runs only pick up stragglers.
discoverSingleNodeIPs seeds 127.0.0.1 then appends net.InterfaceAddrs()
in undefined order, so WANHost=nodeIPs[0] was loopback and the guest
gateway URL helpers fell through to the first non-loopback interface
addr — an arbitrary co-tenant bridge on multi-bridge shared hosts
(wattle: br-lan-env1..23, br-wan), leaving guest VMs unable to reach the
advertised gateway to register.
Resolve WANHost from spinifex.toml nodes.<node>.advertise (falling back
to that node's host), mirroring awsgwBindIP's toml scan; fall back to the
first non-loopback discovered addr only when config is unreadable. This
removes the need for the SPINIFEX_WAN_IP override on shared hosts.
The crash-supervisor auto-restart path (qmpHeartbeat -> HandleCrash ->
MaybeRestart -> startQEMU -> newQMPClientWithHandshake -> dialQMPWithRetry)
already unlinks the stale QMP socket and retries a refused/absent connect. But
a freshly relaunched QEMU can accept the connect then reset the connection
mid-greeting while it initialises, surfacing 'connection reset by peer'
immediately after 'QEMU started successfully' as a ServerInternal.
Add ECONNRESET and EPIPE to isTransientDialError so the dial is retried until
QEMU services the monitor, closing the auto-restart QMP-connect race.
Add non-streaming InvokeModel to bedrock-runtime. Unlike Converse's unified
schema, InvokeModel passes model-provider-native bodies through, so route by
model family: Anthropic Claude bodies forward to the Messages API (drop the
Bedrock-only anthropic_version, inject model, return verbatim); Meta Llama
bodies translate to/from the self-host vLLM OpenAI completions endpoint.
Raw-body dispatch bypasses the JSON response path (WriteRawResponse). Reuses
the Phase 1 credential + endpoint resolvers and upstream-status mapping.
Unit-tested (routing, body rewrite, translation, error classes) plus gated
InvokeModel E2E.
Both services already run otelsetup.Init and ship logs/metrics but never
opened a span, so they were invisible in the Kibana APM service map.
qmp-collector now opens a root span per per-instance poll tick (marks
error on QMP collect/publish failure); spinifex-ui now wraps its HTTP
handler chain with otelsetup.HTTPMiddleware and its reverse-proxy
transport with otelhttp, so UI requests get a server span and proxied
calls to awsgw/predastore carry traceparent instead of stopping the
trace at the UI hop.
nats and eks are intentionally left uninstrumented in this pass: nats is
the vendored message-broker process with no message-level hook to attach
spans to (the traffic it carries is already traced end-to-end at its
producer and consumer endpoints); eks request/response calls already
produce spans today under service.name=spinifex-daemon since the
handlers run in that process, and its async reconciler/state-machine
goroutines need a separate span design across a much larger surface.
Skip ERROR log when a subscription is already invalid
(nats.ErrBadSubscription) during graceful shutdown; log at debug
instead. Cosmetic cleanup race, no functional impact.
golangci-lint canonicalheader flagged 6 lowercase header literals in the
Ochre/Bedrock gateway (anthropic.go, anthropic_test.go, vllm.go). Canonical
form is wire-equivalent; canonicalise to X-Api-Key, Anthropic-Version and
Content-Type to restore CI lint.
* feat(bedrock): Ochre runtime MVP — Converse over self-host + Anthropic-direct
Add the AWS-Bedrock-compatible gateway services (bedrock, bedrock-runtime)
following the EKS REST-JSON pattern: ListFoundationModels / GetFoundationModel
over a static connectivity-tiered catalog, and non-streaming Converse routed
per modelId to one of two provider backends — self-hosted vLLM (OpenAI wire)
and Anthropic-direct (Anthropic Messages wire).
Per-account provider API keys are a master-key-encrypted JetStream KV secret
with an optional platform default; the advertised catalog gates external
models on a resolvable credential. Self-host endpoints are pinned via config
(OCHRE_VLLM_ENDPOINTS) behind an EndpointResolver seam.
Adds gateway/bedrock/ (catalog, provider router, Anthropic + vLLM adapters,
credential store), awserrors REST-JSON codes, and the bedrock E2E suite
(list/get light; Converse gated on live backends). Daemon/NATS control plane
and dynamic endpoint lifecycle are deferred to Phase 3.
* test(bedrock): cover dispatch, router, credentials KV + endpoint config
Add unit coverage for the Ochre Phase 1 diff: Bedrock_Request /
BedrockRuntime_Request end-to-end dispatch, provider router, JSON
response/error writers, upstream-status mapping, the credential-store KV
round-trip, the Anthropic bound-provider path, and parseBedrockEndpoints.
Lifts PR diff coverage above the 60% gate.
* ci: GPU e2e workflow on ci-gpu self-hosted runner
Runs tests/e2e/gpu (EC2/ECS/EKS GPU passthrough) against the persistent,
hand-bootstrapped wattle shared host, driven over SSH by a banksia runner
labelled ci-gpu (matches the ci-single/ci-multi convention) — no runner
installed on the shared box.
- runs-on [self-hosted, ci-gpu]; concurrency ci-gpu, cancel-in-progress
false (single A1000 = single-flight, queue not stomp)
- workflow_dispatch + GPU-path filter (not every push)
- manual-work mutex gate (~/.ci-gpu-paused / CI_GPU_PAUSED) so hands-on GPU
testing blocks CI
- if: always() cleanup terminates launched GPU instances; never resets the
box (no setup-ovn.sh/dev-reset/cluster-reset)
- drives the control plane with spx + the already-configured aws spinifex
profile directly (no wrapper scripts, nothing to install on wattle)
* ci(gpu-e2e): advisory AMI preflight, fail only if zero GPU AMIs
The suite is self-guarding — each test skips when its AMI is absent. The
hard-fail on the base ubuntu GPU AMI blocked the whole run even when the
ECS GPU AMI (the only one imported) could execute. Inventory the AMIs and
fail only when none are present.
* ci(gpu-e2e): self-provision base GPU AMI, hard-require it in preflight
The GPU E2E workflow was green-because-skipped: with no GPU AMIs imported
the self-guarding suite skipped every test, passing as a false green. Make
the raw-EC2 GPU passthrough test the always-on baseline.
- Add an import-if-absent setup step that pulls the prebuilt base GPU image
from the catalog by name (ubuntu-26.04-nvidia-gpu-x86_64), naming the AMI
exactly what tests/e2e/gpu filters for. Idempotent; caches on wattle's
persistent loop-fs. Runs spx under sg spinifex (local storage write).
- Preflight now hard-requires the base AMI (fail, not skip) so a silent
import failure can't pass as green; ECS/EKS node AMIs stay opt-in warn.
- Reconcile the local build path: ubuntu-gpu-nvidia.conf AMI_NAME and
launch-gpu.sh's nvidia default now use the same canonical name, so a
locally built+imported image also satisfies the suite.
* ci(gpu-e2e): pass --ami-name on base GPU import so the AMI name matches
spx admin images import --name selects the catalog entry but names the
resulting AMI generically (ami-<distro>-<version>-<arch>), so the import
landed as ami-ubuntu-26.04-x86_64 and the tests/e2e/gpu name filter never
matched. Pass --ami-name ubuntu-26.04-nvidia-gpu-x86_64 explicitly (same
flag build-system-image.sh uses) and poll briefly for availability.
* fix(network,vm): reconcile auto-assigned public-IP NAT + scale VFIO QMP timeout
Auto-assigned public IPs were never carried through the network reconciler.
applyEIPs ran AddEIP + ensureGuestPortDatapath for user EIPs only; auto-assigned
IPs (intent.Ports[].PublicIP) got egress exemption but no NAT re-assert and no
gatewayLRP->guest convergence. Their dnat_and_snat was created once at
RunInstances and never touched, so a vpcd/OVN rebuild dropped it and the
guest-port hop never converged: inbound DNATs at the gateway but never reaches
the guest (TCP/SSH dark) while ICMP is answered by the gateway LR (ping OK).
floatingIPSpecs now folds every auto-assigned/ELB public IP into the same
AddEIP + guest-port convergence path as user EIPs (deduped by private IP; a user
EIP wins), making the datapath self-healing across reconcile ticks and deploys.
qmpGreetingTimeout for VFIO guests now scales with guest RAM (base 30s + 8s/GiB,
floored at 180s, capped at 600s) instead of a flat 180s, so a large guest is not
SIGKILLed mid-pin while the whole guest RAM is IOMMU-locked before QMP answers.
* fix(vm): scale QMP handshake command deadline for VFIO launches
The greeting timeout scaling alone did not fix the VFIO launch SIGKILL: a GPU
guest sends its QMP greeting before the RAM pin completes, then cannot service
qmp_capabilities until the pin finishes. sendQMPCommand hardcoded a 30s response
deadline, so the handshake decode timed out (~43s for a 16 GiB g5.xlarge) and
the launch cleanup SIGKILLed a QEMU that would have come up.
Parameterize the command deadline (sendQMPCommandWithTimeout); the handshake now
uses the RAM-scaled qmpGreetingTimeout (base 30s + 8s/GiB, floor 180s, cap 600s)
for qmp_capabilities. Other callers keep the 30s default.
* test(gpu-e2e): share one GPU instance across passthrough subtests to avoid mid-suite IP recycle
Recycling both the floating and private IP onto a new ENI left the ingress
datapath targeting the terminated instance, so the successor was unreachable
on its EIP while IMDS stayed up.
- AddEIP re-points the dnat_and_snat row when the owning ENI changed even
though the external+logical IP are unchanged, keyed on the portable
spinifex:logical_port external-id (centralised mode leaves the native
LogicalPort column empty). The genuine same-ENI case still skips row churn.
- Flush the SB MAC_Binding for a private IP on ENI teardown and successor
bind so a reused address re-resolves to the live owner's MAC, via a new
MACBindingFlusher backed by ovn-sbctl.
- Reconciler prunes orphan dnat_and_snat rows whose stamped owning ENI is
absent from intent (leaked by lost vpc.delete-nat across dead VPCs).
- IMDS tap responder suspends retries for an ENI whose record is gone,
stopping the 15s reconcile churn.
* test(daemon): cover GPU inventory in GetNodes and GPU attachment in GetVMs
Adds unit tests for buildGPUInventory/handleNodeStatus (whole-GPU claimed/
free, MIG slices claimed/free, no-GPU node) and resolveVMGPU/handleNodeVMs
(whole-GPU attachment, MIG attachment, no attachment). Production code for
these paths (types.GPUInfo/GPUSliceInfo/VMGPUInfo, buildGPUInventory,
resolveVMGPU) and EC2 DescribeInstanceTypes GpuInfo.Gpus population
(instancetypes.GenerateGPUTypes/GenerateMIGTypes/IsGPUType) already landed
via the daemon GPU-admission work and were already test-covered for part C.
* feat(ui): shared GpuInstanceTypeSelect grouped instance-type picker
Reusable Select wrapper for EC2/EKS/ECS instance-type pickers: groups
Standard vs GPU-Accelerated types via structural GpuInfo.Gpus detection
(AWS SDK shape, no name heuristics), annotates GPU items with VRAM from
MemoryInfo.SizeInMiB and an optional per-type availability count, and
falls back to a single ungrouped list when no GPU types are present.
* feat(ui): EC2 GPU inventory, instance GPU card, shared picker in run-instances
* feat(ui): EKS nodegroup GPU picker + AMI/taint note
Add-nodegroup form now uses the shared GpuInstanceTypeSelect, grouping
and VRAM-annotating GPU instance types instead of a free-text field.
Selecting a GPU type surfaces a note that the GPU node AMI is
auto-selected and the nodegroup is tainted
nvidia.com/gpu=present:NoSchedule by default. Nodegroup rows badge GPU
nodegroups, derived from the primary instance type since
NodegroupRecord does not expose a GPU flag over the API.
* feat(ui): ECS GPU capacity picker, task-def GPU count, task gpuIds
* feat(eks): expose nodegroup GPU signal in Describe/ListNodegroups + UI badge
* chore(ui): rebuild embedded console dist for GPU UI
Bump github.com/mulgadc/viperblock to v1.11.1-0.20260713024608-eb4a376becc8
(dev tip through viperblock #34): bounded parallel chunk upload + coalesced
checkpoint (edou7 sec 7 items 2 and 9), sweeping the require up from #32 and
picking up the #33 write backpressure on the way. The nbdkit plugin the VM
write plane uses is built from the submodule source; this bump aligns the
viperblockd library that spx links.
* fix(daemon): gate GPU instance admission on cpu/mem and GPU-slot availability
canAllocateLocked early-returned count for whole-GPU (non-MIG) types,
bypassing the vCPU/memory admission calc and the live MemAvailable gate.
canAllocateCount already had a requiresGPU/availGPU branch, but every
production caller passed requiresGPU=false, making it dead code.
GPU instance types (whole-GPU and MIG) now flow through the same
cpu/mem + live-mem gate as every other type, with GPU-slot count wired
in as an additional hard constraint from gpuManager.Available(). This
also gives MIG a real up-front GPU-slice gate instead of only failing
late at Claim.
* fix(vm): release leaked GPU pool slot on crash-to-error and add reconciler backstop
HandleCrash freed vCPU/memory via deallocateResources but never called
InstanceCleaner.ReleaseGPU, since it bypasses the stop/terminate cleanup
chain that normally carries that call. A successful restart reuses the
still-set GPUAttachments, so that path is untouched, but when restarts
are exhausted, the instance type is unknown, there isn't enough capacity
to restart, or the relaunch itself fails, the instance settles
permanently in StateError still holding its GPU pool entry with nothing
to reclaim it.
Release the GPU and clear GPUAttachments on each of those terminal
paths. Add a GPU-pool reconciler mirroring the existing ENI reconciler,
wired into the daemon's reconcile loop, that periodically frees any pool
entry whose instance is absent or has settled into a non-running state,
as a backstop for a node-down mid-cascade or any other gap in the
synchronous release.
* fix(eks): keep derived ELBv2 names under 32 chars and surface CREATE_FAILED reason
Long cluster names made the derived "eks-<name>[-cp|-konn]" NLB/target-group
names exceed the ELBv2 32-char limit, failing cluster creation asynchronously
with no visible cause. safeELBv2Name now deterministically truncates the name
and appends a stable hash of the full cluster name when needed, so the same
name is derived on every call (create, lookup, delete). DescribeCluster also
now surfaces ClusterMeta.StatusReason as a Health.Issues entry when a cluster
is FAILED, instead of leaving callers to guess why.
* fix(eks): plumb per-nodegroup Ready-node counts through state reports
The on-VM state-report script now emits nodegroup_ready, a per-nodegroup Ready
count keyed by the eks.amazonaws.com/nodegroup label value, alongside the
existing cluster-wide node_count. ServerStateReport, the ClusterReconciler's
observe step, and SetClusterHealthState now carry this map through to
ClusterMeta.NodegroupNodeCounts, so per-nodegroup consumers (wired in the next
commit) can gate on a nodegroup's own workers instead of the cluster total.
* fix(eks): scope nodegroup Ready-gate per nodegroup and retry failed worker launches
waitWorkersReady compared the cluster-wide Ready-node count against a
nodegroup's baseline+desired, so in a multi-nodegroup cluster one nodegroup's
Ready workers could mask another whose own workers never registered. It now
reads meta.NodegroupNodeCounts[ng], scoped to nodes carrying the
eks.amazonaws.com/nodegroup=<name> label.
Separately, launchNodegroupInfra never retried a failed worker launch (a
transient QMP timeout or host capacity pressure), leaving the nodegroup stuck
CREATING with zero workers. The new launchWorkersUntilDesired wrapper relaunches
only the shortfall on failure, logging each retry, until the desired count is
reached or workerLaunchRetryTimeout elapses.
Both fixes are covered by regression tests: a two-nodegroup scenario where only
one has Ready workers, a transient-failure-then-success relaunch, and an
exhausted-retry-budget CREATE_FAILED path. Not covered here: a worker that dies
after the nodegroup is already ACTIVE — that needs an ongoing reconciler, which
this change does not add.
* fix(eks): tolerate GPU NoSchedule taint in konnectivity-agent DaemonSet
A GPU-only worker is tainted nvidia.com/gpu=present:NoSchedule, so the
konnectivity-agent DaemonSet (no tolerations) showed DESIRED=0 there and
kubectl logs/exec against pods on that node failed "No agent available".
Adds an Exists toleration for nvidia.com/gpu:NoSchedule.
Live-cluster validation (DESIRED count, kubectl exec) is deferred — no cluster
available to test against right now.
* fix: tolerate nvidia.com/gpu taint on ebs-csi-node and coredns DaemonSets so GPU workers get storage and DNS
A re-driven DeleteCluster whose managed control-plane VPC was already gone
(EC2 InvalidVpcID.NotFound) kept looping DependencyViolation: the internal
cp-vpc state record (ClusterMeta.ManagedCPVPC) was never cleared unless the
*entire* purge succeeded, so any unrelated flaky step retried CP-VPC/SG API
calls against a stale VpcId forever instead of converging.
purgeClusterInfra now clears ManagedCPVPC (via a new CAS-based
ClearClusterManagedCPVPC) as soon as DeleteClusterCPVPC + its SG cleanup both
succeed, independent of whatever else in that run still fails.
DeleteClusterCPVPC also now republishes vpc.delete for the CP VPC on every
call — including the already-gone fast path, falling back to the persisted
VpcId when the live tag-indexed describe can no longer name it — so vpcd's
topology manager gets another chance to GC an orphaned logical
router/subnets/SubnetEgressPriorityDrop policies if the original delete's
fire-and-forget event was ever lost.
handleRelease cancelled the per-lease renewal goroutine but deleted the
KV entry without waiting for it to exit. An in-flight reaffirm/renew Put
could land after the Delete, re-creating the lease and leaking a KV
record. Add a per-loop done channel and block on it after cancel so any
pending Put completes before the delete, making DrainAll's release path
deterministically synchronous.
* feat(eks): expose NVIDIA GPUs to pods via CDI device plugin
GPU nodegroups now label workers nvidia.com/gpu.present=true and taint them
nvidia.com/gpu=present:NoSchedule (via a k3s config.yaml.d drop-in, since k3s
has no K3S_NODE_TAINT env), and stage a hidden nvidia-device-plugin addon that
addon-sync renders into the K3s auto-deploy dir. The device plugin runs with
--device-list-strategy=cdi-cri and is gated to GPU nodes by nodeSelector +
toleration, advertising nvidia.com/gpu allocatable so pods land on GPU nodes
and get the device injected via the CDI spec baked by mulga-cdi-generate.
Relies on containerd 2.0.5 default enable_cdi=true (no config fork).
* fix(vm): scale QMP greeting deadline for VFIO passthrough guests
A GPU passthrough guest (-device vfio-pci + large -m) must lock and DMA-map
its entire guest RAM through the IOMMU before QEMU's QMP monitor answers,
which exceeds the fixed 30s greeting deadline and gets the VM SIGKILLed at
launch. Parameterize the greeting timeout (NewQMPClientWithGreetingTimeout)
and scale it to 180s for VMs with GPU attachments; plain VMs keep the 30s
default and reconnects against an already-running QEMU keep it too.
* fix(eks): register GPU workers reliably and expose GPUs to pods
Three fixes let a GPU nodegroup worker boot, register, and advertise its GPU:
- k3s-agent.service gains Delegate=yes (plus KillMode/TasksMax/limits and
overlay/br_netfilter modprobe): k3s embeds containerd+kubelet in-process, so
on cgroup-v2 the kubelet cannot create /kubepods without a delegated subtree
and ContainerManager.Start fails before Node registration.
- The GPU node label rides a k3s config drop-in (20-gpu.yaml node-label +
node-taint) instead of K3S_NODE_LABEL, which k3s ignores; without the label
the nvidia-device-plugin nodeSelector never matches.
- Raise the nodegroup ready timeout to 20m for the ~7min GPU first boot.
Adds the GPU e2e (VPC + NAT-GW egress harness) and a slog-restore unit test.
* fix(eks): install ecr-credential-provider on the GPU node AMI
The GPU worker's kubelet is started (from nodegroup userdata) with
--image-credential-provider-config + --image-credential-provider-bin-dir=
/usr/local/bin, but the eks-node-gpu AMI never built or installed the
ecr-credential-provider binary. kubelet treats a referenced-but-missing
provider binary as a fatal RegisterCredentialProviderPlugins error, so the
node crash-loops and never registers (while the Alpine eks-node worker, which
ships the binary, joins fine). Build + install it in the GPU manifest; the
server-only binaries stay omitted since GPU nodes are agent-only.
Also tee k3s-agent stderr (and its embedded kubelet klog) to the serial
console: Ubuntu logs only to journald, which is off-box-invisible on a worker
in an isolated customer VPC, so a kubelet that fails to register left no trace.
* fix(eks): give the nvidia-device-plugin its own GPU access
The CDI-mode device plugin (--device-list-strategy=cdi-cri) must run
nvml.Init and generate the CDI spec in its own container, but the DS had
no GPU wiring, so nvml.Init crash-looped and nvidia.com/gpu was never
advertised. Run the DS under the k3s-auto-configured nvidia runtimeClass
with NVIDIA_VISIBLE_DEVICES/CAPABILITIES=all, the host root mounted
read-only at /driver-root so it can locate the driver libs + ld.so.cache,
and /var/run/cdi mounted writable so the generated spec is visible to
containerd. No enable_cdi drop-in is needed on k3s v1.32.5 / containerd
2.0.5. GPU workload pods still need only resources.limits nvidia.com/gpu.
Live-validated end-to-end on the single-GPU host (RTX A1000): baked
control-plane AMI, GPU nodegroup, nvidia.com/gpu allocatable, nvidia-smi
pod, zero manual intervention.
* test(ec2): drop obsolete restoreSlogDefault test
Rebased onto dev, which dropped the restoreSlogDefault workaround in #506
(viperblock no longer mutates the global slog). The test referenced the
now-deleted helper and no longer compiles. Real GPU/registration diff is
independently covered (diff coverage 95.3%).
When no br-mgmt bridge is present (supported single-node config where system
instances reach AWSGW via the advertise IP), LaunchSystemInstance correctly
skips mgmt-tap creation but the NIC list still carried the mgmt slot, so
buildNICNetdevs emitted a QEMU netdev/device for a tap that was never created
and QEMU aborted (could not configure /dev/net/tun: Operation not permitted),
failing every CreateLoadBalancer/CreateCluster. Skip the mgmt NIC (index 1)
when its MAC is unset across netdevs, kernel-cmdline mmio slots, and the guest
fw_cfg netcfg blob; multi-node (br-mgmt present) is unchanged.
- 4ncdg: NATSRequest now round-trips the AWS error Message, not just the
sanitized Code. Server sends GenerateErrorPayloadWithMessage(code,
err.Error()); the client returns the Message when present so the CLI prints
the real reason (e.g. "snapshot not found; refusing to restore") instead of
a bare "ServerInternal". One code path fixes every NATS-driven command.
- fbmhl: the UI node view rendered raw host totals (vCPU:80, 0/125.5GB),
ignoring the daemon reserve. It now shows schedulable = total - reserved,
the capacity a tenant can actually consume. reserved_vcpu/reserved_mem_gb
added to the NodeInfo type (already present in the API payload).
Three related nat-mode fixes:
- ura01: nat-transit pool now emits gw_lrp_range_start/end. admin init
sets 100.127.0.16-.254 (239 VPC gateways) instead of the auto-derived
top-16 (~15). Adds GwLrpRange* to PoolData + the spinifex.toml template
so operators can widen any pool.
- 724mm: detectPoolMode returns true only for external_mode=pool. nat and
disabled clusters now skip Phase 8b/8d, which otherwise failed on
AllocateAddress UnsupportedOperation.
- pnz2i: document that system instances (ECS/EKS/LB agents) are
unsupported in nat v1 (require pool mode) in the vpc-networking README
and COMMANDS.md; document the gw_lrp_range_* fields.
* fix(eks): write etcd-snapshot.env at CP launch so snapshots actually run
* test(eks): cover resolveSystemPredastoreURL and buildEKSServiceDeps
Diff-coverage gate for 852f61d9 was unmet (48.8% < 60%) because the
mgmt-bridge branches and EKSServiceDeps assembly had no direct test.
* feat(eks): restore-snapshot DR path (fresh CP + cluster-reset restore + NLB re-point)
* fix(eks): make restore-snapshot fail-safe (validate snapshot, unwind on partial failure, honest status)
* fix(eks): sign etcd-snapshot uploads with deployment region (was 403ing)
* fix(eks): apply recovery directive on DR-seed first boot + single-node replacement host
* fix(daemon): log NATS handler failures at Error (were only on the trace span)
* feat(gpu): Ubuntu NVIDIA GPU node image + GPU AMI selection
Epic A of GPU passthrough to EKS pods / ECS task containers.
- Add gpu-vendor AMI tag constants + GPUVendorForType/IsGPUTypeName helpers
(instance-family prefix -> vendor from knownGPUModels).
- EKS + ECS worker AMI resolution branches to a GPU-tagged node image for GPU
instance types; no fallback to the driverless image (hard error instead).
- Exclude gpu-vendor-tagged AMIs from the plain (non-GPU) lookup so a newer GPU
AMI cannot hijack ordinary nodes (managed-by tag is shared; filters have no
negation, so exclude client-side).
- NodegroupRecord caches GPUEnabled/GPUVendor from InstanceTypes.
- Catalog entries spinifex-eks-node-gpu / spinifex-ecs-node-gpu (dual-tagged).
- New Ubuntu 26.04 GPU node images (eks-node-gpu, ecs-agent-gpu): reuse the
proven DKMS -server driver recipe (drop Docker CE + dev payload), add
nvidia-container-toolkit, OpenRC->systemd port of the worker/agent service
subset, udev EBS by-id bridge, nvidia modules-load, first-boot CDI generation.
- build-system-image.sh: SYSTEM_TAG supports multiple space-separated tags.
* build: allow SYSTEM_IMAGE_BUILD_DIR override for image build dir
Ubuntu GPU images are 16G; a RAM-backed /tmp tmpfs is too small to hold the
qcow2 + expanded raw. Default stays /tmp; SYSTEM_IMAGE_BUILD_DIR redirects the
build onto a real disk.
An ELBv2 LB whose system VM launches but whose in-guest lb-agent never
delivers a first heartbeat previously stayed in provisioning forever with
an empty StateReason: promotion to active happens only on the first
heartbeat and the only provisioning->failed transitions were at
create/launch time. A single wedged LB in the multi-node e2e lb suite
leaked its listener/target-group and cascaded into ResourceInUse and
DependencyViolation on VPC teardown.
- Add a lifecycle reaper that marks failed any LB still provisioning past
lbProvisioningTimeout (4m) with no heartbeat, reusing markLBFailed, and
populate a provisioning StateReason so a stuck LB is self-describing.
Started once at daemon startup; stopped on service Close.
- Harness: add ErrLBProvisioningTimeout and treat a provisioning-timeout
as retryable in createActiveLB; register idempotent LB/listener teardown
up-front so no exit path leaks into the shared VPC.
- IMDS: bounded retry+backoff on the instance-fact and user-data lookups so
a transient gather timeout under multi-node load does not starve the
lb-agent bootstrap.
The coordinated host-shutdown DRAIN phase transitioned every running VM to
StateStopped and migrated it to the operator-stopped shared KV bucket, evicting
it from node-local running state. On the next boot Restore only loads local
running state and treated every StateStopped instance as user-stopped, so
previously-running VMs never relaunched after a host reboot.
Gate on the existing operator-intent signal Attributes.StopInstance (set only by
StopInstances, cleared by StartInstance): stopOne keeps drain-stopped VMs in
local running state instead of migrating them, and Restore relaunches
drain-stopped VMs while operator-stopped ones stay stopped. markUnschedulable
now stamps StopInstance so capacity-demoted instances are not relaunched.
Close four pre-existing lifecycle races surfaced during env19 hardening.
Volume seal/reaper (42hbm, c7ce2): the ebs.unmount handler now seals the
volume before removing it from MountedVolumes, so a retry after a
timed-out-but-completed seal reliably reports NotFound. EBSUnMountResponse
carries a NotFound flag and unmountResponseError treats it as seal success,
so state reconciles to available/detached instead of leaking. The volume
leak reaper reconciles the attachment record (never the data) for volumes
whose owning instance is terminated, so GC can delete them.
Stopped-instance CAS (ozlbq, wlmea): add UpdateStoppedInstance, a
mutate-callback backed by casUpdate(createIfAbsent=false), and migrate
TagStoppedInstance and ModifyInstanceAttribute off the resurrect-prone
Load->mutate->WriteStoppedInstance path. A tag/attribute write racing a
winning start-claim now fails cleanly instead of re-creating the deleted
entry. StartStoppedInstance returns IncorrectInstanceState (not
InvalidInstanceID.NotFound) when the stopped entry is absent but the
instance is present locally.
Every tofu-based e2e workflow set TOFU_LOG to a /tmp path outside
ARTIFACT_DIR. ci-fmt.sh's quiet_run helper appends all tofu init/apply/
destroy output to that file, but the Teardown step (which runs the
destroy) always ran AFTER the existing upload-artifact step, so the
full log — including destroy-time failures — was silently discarded
on every run.
Add an if: always() upload-artifact step after Teardown in each
tofu-based job (e2e.yml, e2e-suites.yml, e2e-nightly.yml, e2e-ddil.yml)
that uploads $TOFU_LOG directly. Since quiet_run appends both apply and
destroy output to the same file, one upload after Teardown now
captures everything, recoverable even when destroy itself fails.
Shipping this log to Elasticsearch/OTLP is deferred to mulga-5n7fe,
which will consume this artifact.
Init now installs an sdk/log LoggerProvider inside the existing
exportEnabled() gate, sharing the same resource as the tracer/meter (incl.
ci.run_id via OTEL_RESOURCE_ATTRIBUTES) and joining lp.Shutdown into the
returned shutdown func. The process default slog handler is fanned out to
both the existing stdout/journald handler and an otelslog bridge, so app
logs export over OTLP with trace correlation while journald output is
unchanged. With no OTLP endpoint configured Init remains a no-op: no
LoggerProvider is installed and the default logger is left untouched.
PR #487 turned attach/detach flaky: reconnectQMP dialed a fresh QMP
client before closing the old connection. QEMU's monitor is a
single-client `server,nowait` listener, so while the old conn holds the
slot the redial gets no greeting, blocks ~30s, and fails, leaving
q.Dead permanently true and every later attach/detach ServerInternal.
Fix the ordering (close, then dial via dialQMPWithRetry to absorb the
brief post-close ECONNREFUSED window).
waitForDeviceDeletedEvent marked q.Dead on any decode error, including
a benign read-deadline timeout — routine, since DEVICE_DELETED is
frequently pre-swallowed by device_del's own reply loop. The
underlying connection survives a timeout, but encoding/json.Decoder
permanently caches read errors on itself, so on a timeout the Decoder
is swapped for a fresh one over the same conn instead of forcing a
full QMP reconnect; q.Dead is now only set on a genuine decode error.
Also: an AttachVolume re-attach to the same instance with a different
device now returns VolumeInUse (matching AWS) instead of
InvalidParameterValue.
* feat(ec2,ebs): harden volume attach/detach + instance lifecycle state machine
Batch of related EBS/instance-lifecycle correctness fixes verified together.
- u94vs: add compare-and-swap (casUpdate/casPut via nats KV Update revision)
to the multi-writer lifecycle KV sites (WriteTerminatedInstance,
WriteStoppedInstance, WriteClusterShutdown); make the teardown-reaper
read-modify-write atomic via a merge callback so retries never re-run
cleaner side effects.
- yi2ed: add ClaimStoppedInstance, a revision-checked delete-as-claim on the
stopped bucket performed before Allocate/Insert/Run, so a forwarded start
and a local fallback cannot double-start one instance onto the same volume.
- siv-473: make AttachVolume idempotent — return attached success when the
volume is already attached to the requesting instance; a conflicting
requested device returns InvalidParameterValue.
- siv-462: reconnect now re-asserts in-use for all attached volumes (boot and
non-boot data volumes), not just boot, so a daemon restart no longer leaves
stateful-pod data volumes split-brain; EFI pflash is skipped.
- 9w2mv: wait for the QMP DEVICE_DELETED event (bounded) before blockdev-del
instead of a blind sleep, so a slow guest unplug no longer exhausts retries.
- soyah: poll (50ms) instead of fixed 1s sleeps in KillProcess and the
viperblockd nbdkit confirm-running path, cutting volume-lifecycle latency.
- siv-481: a cross-node start forward that times out now falls back to a local
start; this is safe because the yi2ed claim is the single serialization point.
* fix(eks): create per-account and leader KV buckets with cluster replica count
The EKS per-account KV bucket and the leader lease bucket were created at
Replicas=1 and never upgraded, since they're created lazily after
upgradeJetStreamReplicas runs at startup. On multi-node clusters this leaves
their single-replica Raft group transiently leaderless during formation,
causing KV timeouts that surface as a deterministic 503 on ListClusters.
Thread clusterSize through GetOrCreateAccountBucket/InitLeaderBucket (backed
by a new utils.GetOrCreateKVBucketWithReplicas) so both buckets are created
at max(clusterSize,1) replicas, matching every other node-count-scaled
JetStream stream.
A fresh cluster or SB re-bootstrap could leave a node's ovn-controller with
its OVSDB client latched on stale Southbound RAFT cluster state, looping on
"clustered database server has stale data" and silently no longer realising
new Port_Bindings. Cross-host CP data-NICs then never bind: no DHCP, no IMDS,
degraded boot. A manual `systemctl restart ovn-controller` was the only
recovery. Root cause is deterministic: setup-ovn.sh restarts ovn-controller
per node as each RAFT member forms, so a controller started for an early
member races the SB membership changes that later joins trigger.
Add three layers of recovery (the ansible bring-up settle pass that fixes this
at the source lives in the mulga repo):
- Per-host wedge watchdog (vpcd/ovn_controller_watchdog.go): not leader-gated
since the wedge is per chassis. Polls SB connection-status; after a stale
threshold issues sb-cluster-state-reset, throttled by a cooldown. Decision
logic takes an injected clock for deterministic tests.
- Reconcile-loop escalation: ensureGatewayClaimed and ensureGuestPortDatapath
escalate to one sb-cluster-state-reset after repeated recompute misses only
when the SB is not connected — a recompute cannot clear a stale-SB wedge.
- Prober primitives SBConnectionState/ResetSBClusterState on GatewayClaimProber
plus the GatewayClaimVerifier interface extension.
Also fix the awsgw OVN health probe, which passed a slashed `-t` socket path
that never resolves and so always reported ovn-controller not_running. Probe
connection-status instead: OVNControllerUp is now true only when connected, so
healthy nodes read ok and a real wedge correctly reads not_running.
The EIP "verify unreachable after disassociate" step polls runSSHQuiet
inside EventuallyErr(90s), but a wedged ssh made os/exec Wait block for
14 minutes — EventuallyErr cannot interrupt an in-flight CombinedOutput,
so one probe consumed the whole 30-min test-binary budget and panicked.
The panic skipped t.Cleanup, leaking the EIP and NAT-gateway VMs, which
held single-node capacity and starved the lb suite's sys.micro launch
(the daemon unsubscribes the full topic, so NATS returns no-responders).
Bound the ssh child at the process layer with exec.CommandContext (30s
deadline) plus WaitDelay, returning a descriptive timeout error. This is
version- and phase-independent, unlike ssh's own ConnectTimeout. 30s is
safe for all callers (worst legit call is ~15s) and well under the 90s
poll budget, so a wedged probe now yields a clean bounded failure with
cleanup intact.
* feat(network): add routed uplink mode, Routed wiring, and NAT egress rule helper
* feat(external): route gateway LRP over transit pool and install VPC SNAT in routed NAT mode
* feat(vpcd): support bridge_mode=nat (routed uplink, transit pool, egress rules)
* feat(daemon): register EIP handlers unconditionally with disabled-mode stub; seed default subnet per external mode
* feat(admin): support --external-mode=nat and suggest it for non-bridgeable uplinks
* feat(scripts): add --nat-uplink routed mode to setup-ovn.sh
* test(e2e): add routed-NAT single-node natuplink suite
Covers the external_mode=nat lane end to end on a nat-initialized node:
host transit wiring (veth, br-ext port, ip_forward, iptables egress
rules), spinifex.toml contents, disabled EIP surface (empty
describe-addresses, UnsupportedOperation on allocate), default subnet
MapPublicIpOnLaunch=false, OVN gateway LRP + VPC SNAT on the transit
/24, guest outbound egress proven via a serial-console probe (nat mode
has no inbound path), and unique transit gateway IPs across VPCs.
Suite self-gates on external_mode=nat; registered as e2e-natuplink and
in SUITES_BUILD so it compiles in CI. Manifest entry stays commented
until CI can provision a nat-mode environment.
* feat(ovn): Address_Set CRUD and NAT exempted_ext_ips column
Adds the exempted_ext_ips column (optional strong Address_Set ref) to the
nbdb NAT model, Address_Set ensure/get with wait-op create semantics, and an
in-place SetNATExemptedExtIPs update on the client interface, live client,
and mock. Test NB schema gains the column and covers insert-resolves-UUID,
converge, clear, and missing-NAT paths.
* feat(policy): exempt address set on routed SNAT/EIP rows
Adds WithSNATExemptSet so routed mode stamps exempted_ext_ips (the
spinifex_nat_exempt Address_Set: transit /24 plus operator extras) on IGW
snat and EIP dnat_and_snat rows, keeping VM replies to host-initiated flows
un-NATted. AddSNAT gains an idempotency guard keyed on (router, VPC CIDR)
with stale-IP replacement, and both AddSNAT and the AddEIP idempotent-skip
path patch the ref in place on pre-existing rows. The set is a singleton
strong ref and is never deleted. Non-routed modes are unchanged.
* feat(host): VPC ingress route primitives for routed-NAT mode
EnsureVPCIngressRoute / RemoveVPCIngressRoute manage the host route that
sends VPC-CIDR traffic into OVN via the gateway LRP transit IP on the
spx-nat veth. Replace/del semantics keep both idempotent; a missing route
on removal is not an error.
* feat(external): routed ingress leg on IGW attach/detach, reconcile-safe
AttachIGW's routed leg now also installs the host VPC ingress route via
RoutedIngressHooks, and re-runs on the already-attached path so the
reconcile loop restores host routes (and patches legacy SNAT exempt refs)
after a reboot — OVN state survives restarts but host routes do not.
DetachIGW removes the route alongside the SNAT. vpcd wires the hooks and
the spinifex_nat_exempt set (transit /24 + [network] nat_exempt_cidrs,
validated nat-mode-only) exclusively in routed-NAT mode; pool/veth/direct
pass nil hooks and are unchanged.
* test(natuplink): Tier 1 host-ingress phase; docs for jumpbox pattern
New Phase 7 asserts the automatic ingress tier: host route to the VPC CIDR
via the gateway LRP transit IP, exempted_ext_ips ref on the SNAT row
pointing at spinifex_nat_exempt (containing the transit /24), and — after
opening the SG from the transit net (default-closed, AWS parity) — ping and
an sshd TCP handshake against the instance private IP from the host, which
proves the un-SNATted return path. Phase 8 additionally waits for the
second VPC's ingress route to appear on IGW attach. vpc-networking README
documents the jumpbox pattern, SG note, and nat_exempt_cidrs.
* feat(config,admin): public wan pool alongside nat-transit
Restructure admin config generation from single-pool scalars to a Pools
slice so nat mode can emit both the transit pool and an optional public
wan pool. nat mode now accepts --external-pool/--external-source (plus
--external-bind-bridge/--gateway-ip when a public pool is requested),
validated by the shared resolvePublicPoolFlags helper also used by pool
mode. Transit pool always renders first so the IGW allocator picks it.
Formation wire format unchanged (nat mode is single-node only).
* feat(daemon): enable EIP service and auto-assign in nat mode with public pool
External IPAM (and with it the real EIP service, RunInstances public IP
auto-assign, and MapPublicIpOnLaunch on default subnets) now activates
whenever public pools exist: pool mode as before, plus nat mode when a
wan pool rides alongside the transit segment. The transit pool is
filtered out of IPAM — its addresses are gateway-LRP plumbing, never
allocatable EIPs. Transit-only nat configs keep the disabled EIP stub.
* feat(host): per-EIP ingress primitives for routed-NAT mode
DetectUplinkFor parses ip route get to find the interface and source IP
facing the pool gateway (re-run per ensure: WiFi/cellular uplinks move).
EnsureEIPIngress steers an EIP into OVN with a /32 route via the VPC
gateway LRP transit IP (src-hinted so host-local traffic still DNATs),
answers ARP for the EIP with a proxy neighbor entry (L3 only, same MAC,
WiFi-safe), sends gratuitous ARP, and adds per-EIP FORWARD accepts for
DROP-policy hosts. RemoveEIPIngress mirrors; missing state tolerated.
* feat(policy,vpcd): host EIP binder option on NATManager
WithHostEIPBinder injects routed-mode host plumbing hooks. AddEIP binds
after priming on both the fresh-add and idempotent-skip paths (host
routes and proxy-ARP are volatile even when the OVN row survives, so
reconcile re-ensures them after reboot); DeleteEIP unbinds best-effort.
The next hop comes from the new gatewayPortIP helper reading the gateway
LRP's stamped spinifex:gateway_ip (Networks fallback), with the ext-ID
key now an exported policy const shared with the external package.
vpcd selects the IGW pool by name instead of index — with two pools in
nat mode the gateway allocator must stay on the transit pool — and wires
the binder over host.EnsureEIPIngress/RemoveEIPIngress when a public
pool is present. EnsureEIPIngress gains an uplink hint so dhcp-sourced
pools (gateway unknown until lease) proxy-ARP on their bind bridge.
* feat(dhcp): interface-MAC lease variant with MAC-keyed-router detection
dhcp_mac = "derived" (default) | "interface" on external pools. The
interface variant leases with the bind interface's own MAC — WiFi/WWAN
uplinks drop frames with foreign source MACs, so derived chaddr MACs
never complete DORA there. Option 61 stays the lease discriminator.
UseIfaceMAC threads AllocateRequest → NATS wire → nclient4 → lease and
persists through KV so restart re-acquires repeat it.
Routers that key leases on MAC and ignore client-id would hand every
lease the same IP; the manager detects an ACK matching the interface's
own address or another client-id's live lease, releases the colliding
lease, and hard-errors advising a static range instead.
admin init auto-writes dhcp_mac="interface" for nat-mode public dhcp
pools bound to non-bridgeable uplinks (wlan/wwan/ppp).
* test(natuplink)+docs: Tier 2 EIP ingress e2e phases and public-pool docs
The natuplink suite now detects a public (non-transit) pool in spinifex.toml
and switches lanes: Tier-1-only configs keep the EIP-disabled and
no-public-IP assertions; public-pool configs assert the EIP surface is
enabled, the default subnet has MapPublicIpOnLaunch=true, instances
auto-assign public IPs, and a new Phase 9 exercises the full EIP ingress
lifecycle — host /32 route via the gateway LRP, proxy-ARP on the uplink,
per-EIP FORWARD accepts, exempt-stamped dnat_and_snat, TCP handshake to the
EIP, reconcile replay across a vpcd restart, and teardown on disassociate.
Docs: vpc-networking README gains a public-pool section for routed-NAT nodes
(init flags, dhcp_mac=interface behaviour, host-to-EIP caveat) and an updated
mode comparison; COMMANDS.md notes nat-mode EIP support behind a public pool.
* fix(setup): whitelist iptables, neigh sysctl, and arping for spinifex-vpcd
Routed-NAT mode shells out to iptables for the transit masquerade and
per-EIP FORWARD accepts, tunes proxy_delay, and sends gratuitous ARP for
host-delivered EIPs. Without these sudoers entries vpcd crash-loops with
"command not allowed" on nat-mode nodes.
* fix(ovn): scope DeleteStaticRoute prefix match to the target router
Every VPC router carries a 0.0.0.0/0 static route, so an unscoped cache
match could grab another router's row and fail with a referential
integrity violation, wedging IGW attach on multi-VPC nodes. Match only
rows referenced by the target router, mirroring FindStaticRoute. Mock
updated to the same semantics, with a regression test covering drift
replacement across two routers.
* fix(reconcile): always replay AttachIGW so host ingress state is re-ensured
applyIGWs skipped AttachIGW whenever the gateway switch port existed in
actual state, so the already-attached re-ensure path never ran and
routed-NAT host ingress routes lost on reboot were never reinstalled.
AttachIGW is idempotent; run it unconditionally.
* fix(vpcd): resolve duplicate-CIDR host ingress route collisions, bootstrap VPC wins
Every account's default VPC shares 172.31.0.0/16, so Tier-1 host routes
collide on the single host route table and the last IGW attach silently
stole the route (route replace). Pass vpcID and the gateway LRP IP
through RoutedIngressHooks; the bootstrap VPC always installs its route,
other VPCs only install when the CIDR is free and only remove a route
their own gateway IP holds. Adds host.VPCIngressRouteVia to read the
current route holder.
* ci(e2e): add nightly cell 19 for routed-NAT single-node suite
Cell 19 bootstraps with --external-mode=nat plus the CI static pool as
the public wan pool and runs the natuplink suite, covering the Tier-1
transit datapath and Tier-2 EIP ingress. The run step exits 0 when
natuplink.test is absent from the tests tar so scheduled runs against
branches predating the suite stay green.
* fix(admin): nat-mode init summary reflects configured public pool
The summary printed 'outbound-only — no public IPs/EIPs' even when
--external-pool/--external-source configured a public wan pool. Print
the pool range (or dhcp bind) and note EIPs are enabled.
* fix(eip,vpc): thread ctx through disabled EIP stub and NAT call sites
Reconcile the routed-NAT branch with dev's ctx-first service APIs after
rebase: DisabledEIPService and its tests take ctx to satisfy the EIPService
interface, the vm_adapters public-IP resolver asserts the ctx-signed
AssociatedPublicIPForInstance, and the default-subnet mapping test threads
ctx into DescribeSubnets.
* feat(otelsetup): OTel init, resource attrs, trace-stamping slog handler
Env-gated on OTEL_EXPORTER_OTLP_ENDPOINT: with no endpoint only the W3C
propagator is installed and globals stay no-op, so instrumented binaries
ship everywhere at zero cost. Resource carries service/host/mulga.env/
mulga.source plus OTEL_RESOURCE_ATTRIBUTES (ci.run_id in CI).
* feat(otelsetup): wire telemetry init into all spx services
Every spx service start command installs the trace-stamping JSON slog
default and the OTel providers via initTelemetry, flushed on shutdown.
Gateway's own slog setup now routes through otelsetup so its level
handling keeps trace correlation.
* feat(gateway): HTTP tracing middleware on awsgw and daemon cluster API
Server span per request with W3C traceparent extraction; awsgw spans are
renamed to the resolved SigV4 service.Action post-auth with account and
region attributes. Request log lines carry trace_id via context.
* feat(nats): propagate W3C trace context over command subjects
NATSRequestCtx injects traceparent beside X-Account-ID and opens a client
span per hop; ServeNATSRequest opens a consumer span joining the producer
trace, with a Ctx variant for handlers that thread context. RunInstances
is plumbed end-to-end (gateway dispatch through placement fan-out to the
daemon handler) and viperblockd ebs.mount/unmount join inbound traces.
* feat(systemd): source optional /etc/spinifex/telemetry.env in service units
Lets operators enable OTLP export (OTEL_EXPORTER_OTLP_ENDPOINT,
MULGA_ENV, MULGA_SOURCE) by dropping a single env file, with no unit
edits. File is optional; absent means telemetry stays disabled.
* feat(ci): export OTel telemetry from E2E envs to the central Elastic sink
All E2E workflows set SPINIFEX_OTLP_ENDPOINT (repo variable override,
default wattle) plus ci.run_id/ci.workflow resource attributes, so
bootstrap-install.sh writes telemetry.env on every node. Teardown runs
flush-telemetry.sh before the final tofu destroy so buffered spans and
metrics reach the sink.
* feat(otelsetup): request metrics and Go runtime metrics via OTLP
mulga.requests counter + mulga.request.duration histogram recorded by the
same middleware that opens spans: awsgw carries the resolved SigV4 action,
the daemon cluster API uses the chi route pattern, and NATS handlers record
per-topic at the subscribe chokepoints (node names stripped to keep
cardinality flat). Go runtime metrics start inside the same env gate.
* feat(daemon): deep spans for QMP commands, VM launch phases, EBS ops
Single chokepoint span per QMP command in sendQMPCommand covering all
callsites; vm.launch phase children (mount_volumes, start_qemu, qmp_connect,
persist_state); ec2.cmd consumer span with traceparent extraction plus
start/stop/reboot/terminate child ops; ebs.mount/unmount spans in
viperblockd with volume attributes.
* feat(telemetry): thread request context through NATS, object store and VM paths
Migrate utils.NATSRequest call sites to NATSRequestCtx (241 sites; W3C
traceparent now flows gateway -> daemon on every request), add GatherCtx and
migrate all Gather call sites, inject traceparent on manual EKS publishes.
ObjectStore interface is ctx-first with aws-sdk WithContext calls and an
otelhttp transport, so predastore spans join the caller trace. NATS consumer
spans record error status when the response envelope carries an error.
vm.Manager and the ec2 handler service interfaces take ctx-first so vm.launch
and QMP spans join the request trace; detached background work uses explicit
context.Background(). Hot-path slog calls use *Context variants for trace_id
binding.
* feat(telemetry): suppress health-check, heartbeat and unrooted trace noise
Health probes record metrics only (no span per poll); the QMP heartbeat
poller marks its context so query-status ticks stop rooting a trace each.
rootSampler honors MULGA_ROOT_TRACE_RATIO (ParentBased ratio, default 1.0)
and the predastore unit sets 0.05: guest block I/O roots are sampled while
request-linked spans are always kept.
* feat(otel): export deployment.environment resource attribute
Elastic APM maps it to service.environment, enabling the native environment
selector across the APM UI (Services, Traces, Dependencies).
* fix(telemetry): stop objectstore rooting method-only HTTP transactions
Background S3 calls (state sync, cleanup loops) carried no caller span, so
otelhttp rooted a trace per request and APM surfaced thousands of parentless
'HTTP GET/PUT/DELETE' transactions with expected-404 probes counted as
failures. Emit the client span only when the request context already carries
one, matching the viperblock backend. Bump viperblock/predastore to branch
heads so CI builds also pick up the equivalent span filter and root sampler.
* feat(telemetry): per-VM guest metrics via dedicated QMP socket + qmp-collector
Goanna Phase 1 producer, operator-consumed first. Each VM gets a second
-qmp chardev (qmp-telemetry-<id>.sock) plus a sidecar discovery file
carrying instance/account/vcpus/taps/monitoring-tier; the new
qmp-collector service polls query-cpus-fast (vCPU thread /proc jiffies),
query-blockstats, query-balloon and host tap sysfs counters, and
publishes CloudWatch-mappable goanna_ec2_* series to NATS
metrics.ec2.<instance-id>. A queue-group bridge in the same process
forwards the series to the node-local OTLP receiver so they reach the
operator sink until Goanna consumes the subjects. The collector never
touches the manager's control QMP socket; telemetry failures never
block a launch.
* fix(qmp-collector): dial NATS via DialTarget, drop ProtectProc, fix meta race
The collector connected to the raw NATS host (0.0.0.0), failing TLS
verification against a cert valid for 127.0.0.1; route through
admin.DialTarget like the other daemons.
ProtectProc=invisible hid the QEMU vCPU threads (dumpable=0) that the
collector must read from /proc/<tid>/stat, so every tick failed and
re-primed without ever publishing. Drop it to default.
Surface collect failures at Warn so silent metric loss is visible, drop
per-tick publish to Debug, and read InstanceID under the lock in run()
to fix a race with updateMeta on ENI hot-plug.
* chore(deps): re-pin predastore + viperblock to dev-landed commits
* feat(eks): reconciler in-place control-plane restart on sustained degradation
Phase 1 of EKS CP recovery (siv-243). The ACTIVE-branch reconciler only
recorded health; a CP VM that died (OOM/host crash) left the cluster wedged
ACTIVE-but-unreachable with no recovery path. Add a CPInstanceControl surface
and bounded in-place restart: once an ACTIVE cluster's CP stays unhealthy past
a grace window, StartInstance is issued (backoff-spaced, capped) when the
instance is stopped/error, re-mounting the same root volume so embedded etcd
survives. Degradation stays AWS-faithful ACTIVE + a ClusterHealth issue rather
than a non-AWS DEGRADED status enum. cpControl nil keeps the reflect-only path.
Wiring of a real CPInstanceControl in the daemon and the error-state
manager-eviction fix are the remaining Phase-1 integration steps; Phase 2
(etcd snapshot cadence + restore-snapshot) is tracked separately.
* feat(eks): wire control-plane restart into the reconciler via instance service
Threads a CPInstanceControl into the per-cluster reconciler (siv-243 Phase 1).
cpControlAdapter binds the daemon instance service + account: InstanceState
reads the CP's EC2 state via DescribeInstances, StartInstance restarts a stopped
CP via StartStoppedInstance, which loads from the shared stopped store and
re-mounts the same root volume so embedded etcd survives. Wired as
EKSServiceDeps.CPControl = d.instanceService and passed through spawnReconciler.
Recovers a cleanly-stopped control plane co-located with the reconciler leader.
Remaining: DescribeInstances is node-local so a cross-node stopped CP needs an
aggregating describe; and an error/crashed CP (not in the stopped store) needs a
dedicated error-state restart path — both tracked as Phase-1 follow-ups.
* feat(eks): cross-node CP describe + error-state control-plane restart
The reconciler's control-plane recovery was wired to the node-local instance
service: DescribeInstances only saw the leader's node and StartStoppedInstance
only handled a cleanly-stopped CP from the shared KV. A crashed/OOM CP ends in
error state, still resident in its owning node's manager map (not the stopped
store), so recovery never reached it — the actual incident in the recovery plan.
Wire EKSServiceDeps.CPControl to a NATS-backed eksCPControl. DescribeInstances
fans out across every host (gateway scatter-gather) so the reconciler observes a
CP on any node. RecoverInstance drives the gateway StartInstances two-step:
ec2.cmd.<id> recovers a live error/running owner in place (the owner stays
subscribed through the crash; vmMgr.Start relaunches error->pending, re-mounting
the etcd root volume), falling back to ec2.start shared-KV rehydration for a
stopped CP.
The CP VM is ManagedBy=EKS, hidden from customer DescribeInstances; relax that
hide so the owning account resolves its own managed CP when targeting it by
explicit instance ID (bulk listings still hide it). EC2APIState already maps the
internal error state to the AWS closed-enum stopped, which cpRestartable accepts.
* fix(eks): reconcile CP as system account, not customer
The control-plane VMs are launched under the system account
(admin.SystemAccountID()), not the customer account that owns the cluster
record. spawnReconciler was binding cpControlAdapter to the customer account,
so InstanceState/RecoverInstance ran as the customer — which cannot see or own
its own cluster's system-owned CP VMs. DescribeInstances returned zero and the
reconciler logged 'control-plane instance not found', never restarting a wedged
CP. Bind the adapter to the system account instead.
Revert the earlier DescribeInstances managed-VM relaxation: it was based on the
mistaken assumption that CP VMs are customer-owned. Visibility is resolved by
the system-account adapter, so the explicit-ID escape hatch is dead code.
* fix(vm): pre-create mgmt tap on disk-boot restart path
The direct-boot branch pre-creates the management tap via NetworkPlumber.SetupTap
(owned by the daemon euid), but the disk-boot/else branch only appended the QEMU
tap,id=mgmt0 device without creating the tap. The daemon runs as a non-root user
without CAP_NET_ADMIN, so QEMU can only attach a tap it already owns; on the
disk-boot path QEMU tried to create /dev/net/tun itself and failed with
'Operation not permitted', failing every stopped-instance restart (StartInstances
returned ServerInternal). This blocked EKS control-plane recovery: the reconciler
found the stopped CP and issued the restart, but the VM never booted.
Call SetupTap for the mgmt tap in the disk-boot branch, mirroring direct-boot.
* feat(eks): recover all HA control-plane members, not just primary
The reconciler restarted only ControlPlaneInstanceID (the primary CP
member), so on an HA cluster the other server VMs stayed stopped and
etcd never regained quorum (1/3) — apiserver healthz stayed fail even
after recovery ran. maybeRecoverControlPlane now iterates every member
in ControlPlaneNodes (falling back to the scalar ID for single-node
clusters), restarting each stopped/error member per pass while
preserving the grace window, attempt cap, and backoff gates.
Refs mulga-siv-509
* fix(eks): keep cloud-init polling IMDS so primary CP gets its user-data
The eks-node system VM's user-data is delivered only via IMDS (the
NoCloud seed was retired in #411). IMDS (169.254.169.254) is reachable
only once mulga-mgmt-net brings up the data NIC via DHCP and the per-tap
br-imds datapath reconciles, which on the first-launched (primary) CP
can lag cloud-init's stock 120s probe window. With the default max_wait
cloud-init falls back to DataSourceNone, permanently losing the launch
user-data (no SPINIFEX_K3S_ROLE -> no k3s server) and wedging the cluster
in CREATING.
Bake a cloud.cfg.d drop-in that pins DataSourceEc2 and raises max_wait to
600s so cloud-init keeps polling until the datapath is ready instead of
falling through to None. Worst case a slow boot, never a lost role.
* fix(eks): warm IMDS instance-role creds before first SigV4 sign
On IMDS-credentials mode the on-VM SigV4 helpers (eks-token-webhook,
eks-gateway-publish, mulga-eks-state-report) sign via the SDK default
chain, which resolves in-VM IMDS instance-role creds. The per-tap
br-imds datapath lags VM boot by minutes (observed ~264s), and the SDK's
IMDS client fast-fails each probe (250ms dial), so early Retrieve calls
return 'context deadline exceeded' / 'no EC2 IMDS role found'.
eks-gateway-publish only retries for 30x5s=150s, shorter than the
datapath warm time, so it exhausts its budget and drops the
bootstrap/state publish -> cluster hangs in CREATING.
The credential chain is sound (the CP role trusts ec2.amazonaws.com and
AssumeRoleForInstance evaluates that principal); the failure is timing.
Add gwsign.Signer.EnsureCredentials and block in eksgw.New until the
instance-role creds first retrieve (2s interval, 10m ceiling aligned
with cloud-init max_wait), so every helper's first signed request
already holds creds instead of racing a cold datapath. Static-key mode
is unaffected.
* fix(eks): retry CP data-NIC DHCP until cross-host datapath is up
A single ~16s DHCP one-shot on the primary data NIC lost the race against
the cross-host OVN datapath (logical-port binding + per-tap reconcile on the
remote placement host), stranding the NIC IP-less so the Ec2 datasource never
reached IMDS and the CP member wedged. Add dhcp_acquire: retry the one-shot on
a budget aligned with cloud-init's Ec2 max_wait (600s), confirming an inet
address before success. Each attempt still exits leaving no daemon, so
cloud-init's later ENI management is unaffected.
* fix(eks): drop reconciler registry entry when Run self-exits
ReconcilerRegistry.Spawn only cleaned up its holder on ctx cancel
(Stop/StopAll). When a ClusterReconciler's Run returned on its own — a
terminal cluster state or a lost lease — nothing cancelled the ctx, so
the holder leaked and a later same-name create silently no-opped at the
head of Spawn, never reconciling to ACTIVE.
Thread a finished channel from RunClusterReconciler's Run goroutine back
to Spawn's watcher, which now cleans up on whichever fires first: an
external cancel or Run exiting. On self-exit it deletes its own holder
and releases the lease, so a fresh same-name create re-spawns and
reconciles.
* fix(eks): run mulga-mgmt-net in boot runlevel before cloud-init-local
CP user-data is delivered only via IMDS since the NoCloud seed was retired
(#411), so cloud-init DataSourceEc2 must reach 169.254.169.254. mulga-mgmt-net
DHCPs the data NIC that makes IMDS reachable, but it was added to the default
runlevel and only declared `before cloud-init` (the network-stage service).
cloud-init-local therefore ran its init-local Ec2 crawl first with no data-NIC
route and burned its whole probe window (~4 min) before mulga-mgmt-net brought
eth0 up, regressing CP bringup from ~1m50s to ~5m54s.
Add mulga-mgmt-net to the boot runlevel in setup.sh instead of default, drop it
from ENABLE_SERVICES, and order it `before cloud-init-local` so the data NIC and
IMDS route are up before the init-local crawl. Keeps the IMDS-only user-data
path; the raised Ec2 max_wait stays as a safety net for host br-imds datapath lag.
* fix(vm): refresh LaunchTime on crash-restart so watchdog spares relaunch
RestartCrashedInstance re-launched a crashed VM without refreshing
Instance.LaunchTime. The pending watchdog measures now-LaunchTime against
PendingWatchdogTimeout, so the stale original launch time tripped
launch_timeout on the next tick and terminated the crash-restart
mid-flight. Refresh LaunchTime under the state lock, mirroring the
daemon-restart restore path (restore.go), so the relaunch gets a fresh
timeout window. Adds a regression test asserting LaunchTime advances.
* feat(eks): reconcile control-plane member count after a lost member
The cluster reconciler self-heals a stopped/error control-plane VM in
place but never replaced a terminated/gone member, so an HA cluster ran
below haControlPlaneCount permanently. Add a member-count reconcile pass
that provisions one replacement CP joining the surviving etcd quorum.
- CPProvisioner + EKSServiceImpl.ProvisionReplacementCP: replay the
persisted create template on a free host, the single-node analogue of
the spread join branch, with rotating creds re-derived.
- ControlPlaneTemplate persisted on ClusterMeta at create (per-node and
rotating-cred fields cleared). SwapControlPlaneMember CAS swaps the dead
node for the replacement and refreshes the [0] scalar mirrors.
- maybeReplaceControlPlaneMember fires only when no member is restartable
and one is truly lost, a healthy majority survives, bounded by grace,
backoff and an attempt cap; placement excludes live-member hosts.
- Replacement prunes the superseded member from etcd once joined:
k3s-first-boot deletes the stale Node (EKS_ETCD_PRUNE_PEER_IP) so quorum
width returns to N. json tags added to K3sServerInput (now persisted).
* feat(eks): in-guest control-plane self-diagnosis on healthz fail
When a control-plane apiserver reports unhealthy, the reconciler only
saw healthz=fail with no cause, and the guest has no on-demand command
channel (serial root login is password-gated, no guest agent). Make the
CP self-diagnose and relay the cause through the one-way state report.
- mulga-eks-state-report.sh: diagnose() collects failing /readyz?verbose
subchecks, isolated /healthz/etcd reachability, and etcd-db-dir free
space; on fail write a verbose block to the host-captured serial
console and add a JSON-safe reason to the state payload.
- ServerStateReport: add Reason (omitempty; empty from a healthy CP or
an older AMI predating the field).
- cluster_reconciler observe(): surface the reason in the health issue.
- Tests: POSIX unit test for the agent (stubbed kubectl/df); Go tests
for the reason-bearing and terse health-issue paths.
* feat(eks): 15-min etcd snapshot cadence alongside nightly
Nightly alone is too coarse a recovery point for a young cluster. Install the
same mulga-eks-etcd-snapshot script into a second Alpine crond periodic dir
(/etc/periodic/15min) so it also runs on a ~15-minute RPO window; the script
picks its cadence tier from the dir it is invoked from ($0 sniff) and encodes
the tier in the snapshot key (etcd-{tier}-{ts}.snap), so a predastore S3
lifecycle rule can retain the frequent rolling window shallow and the nightly
copies deep. Server-role only (agents exit early); no restore-path change.
Add a self-contained POSIX unit test (stub k3s/curl/logger, symlinked crond
dirs to exercise the real $0 tier sniff) covering agent no-op, frequent + daily
key prefixes, local-snapshot prune, and quiet no-op without creds.
* feat(eks): auto-reform wedged etcd quorum via boot-time cluster-reset
An HA k3s control plane does not self-heal after a simultaneous full-CP
restart: embedded etcd never reforms quorum (etcd:unreachable sustained),
and in-place VM restarts cannot fix it. Add a last-resort escalation that
drives the documented k3s cluster-reset recovery, pull-based so the host
never needs an inbound channel into the guest.
Reconciler: maybeReformEtcdQuorum escalates only when every CP member is
VM-running but the health issue carries etcd:unreachable, the cluster is
HA, and a grace window has elapsed. It sets per-member recovery directives
(seed = cluster-reset, others = wipe-rejoin), epoch-bumped and keyed by
instance ID, then restarts the members. Bounded by grace/backoff/attempts;
a recovered cluster clears the directives once. Enabled via
WithEtcdResetRecovery alongside the existing CP control wiring.
Directive plane: per-member KV directive {epoch, action, snapshot} with
store helpers, an EKS service Get/Set pair over NATS-RPC, and an
internal-recovery gateway route (system-cred carve-out, same as
internal-addons). The CP server instance role gains eks:GetRecoveryDirective.
Guest: mulga-eks-k3s-recovery is a pre-k3s oneshot (server role only) that
fetches its directive through the AWS gateway (eks-gateway-fetch -resource
recovery), epoch-guards one-shot application, then runs
`k3s server --cluster-reset` (optionally restoring a predastore snapshot)
or wipes the local etcd datastore to rejoin the reset seed. Wired into the
image manifest, setup.sh perms, and the eks-node-role selector.
Tests: reconciler escalation gates + recovery clear, directive store epoch
increment + service validation, and a POSIX unit test for the guest agent.
* fix(eks): reboot running CP members to apply etcd cluster-reset directive
The etcd quorum-reformation escalation drove StartInstance on the seed and
followers, but the reset gate only fires when every member is VM-running. The
StartInstances path rejects a running instance with IncorrectInstanceState, so
the guest never re-entered its boot sequence and the cluster-reset/wipe-rejoin
directive was never applied — the wedged control plane stayed wedged.
Reboot the running members (QMP system_reset) instead, re-entering the boot
sequence so the on-VM recovery agent applies the directive. Adds RebootInstance
to the CP control surface (interface, adapter, NATS-backed daemon impl) and
switches maybeReformEtcdQuorum to it; the stopped/error in-place restart path
keeps StartInstance.
* fix(eks): graceful stop for etcd reset recovery
The etcd quorum-reformation reset path forced running CP members through a
hard QMP system_reset to re-enter the boot sequence and apply their recovery
directives. A hard reset skips a clean unmount: on the next boot e2fsck
cleared the recovery and k3s runlevel entries (dangling inodes), so the
cluster-reset never ran and k3s was unstartable — the reset defeated recovery.
Replace the reboot with a graceful StopInstance (QMP system_powerdown, clean
unmount, SIGKILL fallback). The existing in-place restart path
(maybeRecoverControlPlane) then boots each stopped member clean on a later
tick, and the on-VM recovery agent applies its directive against an intact
filesystem. Reuses existing lifecycle machinery; the stop also tolerates a
wedged QMP client via the SIGKILL fallback.
Swaps RebootInstance -> StopInstance across CPInstanceControl,
cpInstanceController, the cpControlAdapter, and the NATS-backed eksCPControl
(gateway RebootInstances -> StopInstances), plus the reset/recovery test fakes.
* fix(vm): retry QMP dial and unlink stale socket on relaunch
A QEMU killed with SIGKILL (crash or etcd-reset restart) leaves its
qmp-<id>.sock inode behind. On relaunch the startup os.Stat probe and the
QMP dial both raced that dead listener, so a single dial returned
'connection refused': StartStoppedInstance then reported ServerInternal and
left the instance KV-stopped even though the guest booted fully. The wedged
member stayed invisible to the reconciler's cpLive gate, stalling recovery.
Unlink the stale socket before launch so the inode that appears is the fresh
QEMU's, and retry the QMP connect with a short backoff to cover the window
where the socket is bound but not yet listening. Handshake failures are not
retried — a listener answered, so the error surfaces immediately.
* feat(eks-ui): surface cluster Health.Issues on the EKS console
DescribeCluster is AWS-faithful (Status stays ACTIVE, degradation lives in
Health.Issues), but the cluster detail page only rendered the status badge, so
a degraded control plane still showed as ACTIVE with no visible signal. Add a
destructive health badge next to the status and a Health row that lists the
issue messages, mirroring the addons tab. Healthy clusters read OK.
* chore: rebuild ui
* fix(viperblockd): stop mount-path daemon VB uploader to prevent checkpoint clobber
The daemon-side VB created in the ebs.mount handler is stored in
MountedVolume.VB for the whole mount, and its background chunk uploader was
only stopped at daemon shutdown. That state-only VB ran DrainToBackend every
30s, overwriting the live checkpoint the nbdkit plugin VB (the data-path
writer) keeps updating, leaving SeqNums inconsistent with sealed chunk
ciphertext and causing transient AEAD auth failures on cloned encrypted
volumes. Stop its uploader right after New, matching the other short-lived VB
call sites.
Also bump viperblock to the atomic chunk-ObjectID allocation fix.
* build(deps): pin viperblock to landed AEAD concurrency fix
Bump github.com/mulgadc/viperblock to the dev commit b08bea6 that
eliminates "cipher: message authentication failed" on encrypted volumes
under concurrent read-modify-write + background chunk drain. Removes the
local-path replace used while the fix was in flight.
* fix(viperblock): stop chunk-uploader goroutine leak at all VB call sites
viperblock.New() unconditionally starts a background chunk-uploader
goroutine (30 s interval) that only VB.Close() stops. Seven call sites
created short-lived or state-only VB instances without calling Close(),
leaking one uploader goroutine per instance. On a single node the leaked
goroutines exhaust memory (predastore RSS grew ~1.3 GB/5 min at idle
post-single-suite), causing liveMemGate to reject launches and the lb
suite to fail.
The leaked uploaders are also the primary cause of the AEAD auth
failures (mulga-be2ev): a leaked VB from prepareRootVolume holds the
clone-point BlockLookup and overwrites the correct live checkpoint in S3
every 30 s, leaving SeqNums inconsistent with sealed chunk ciphertext.
Fix all seven sites:
- instance/service_impl.go prepareRootVolume: defer StopChunkUploader +
StopWALSyncer after successful viperblock.New (mirrors the EFI path's
intent; full Close is unsafe here because cloneAMIToVolume already
called SaveBlockState + RemoveLocalFiles)
- viperblockd.go ebs.delete handler: add StopChunkUploader alongside
existing StopWALSyncer
- viperblockd.go ebs.unmount handler: same
- viperblockd.go shutdownVolumes: same
- snapshot/service_impl.go snapshotVolume: defer both stops after New
- volume/service_impl.go CreateVolume: defer both stops after New
- image/service_impl.go snapshotRunningVolume: defer both stops after New
- image/service_impl.go snapshotStoppedVolume: defer both stops after New
Closes mulga-0zzci
Closes mulga-ji5fi
* fix(deps): bump viperblock to 01a7e686 (AEAD regression fix)
The prior pin (6af89855) carried the snapshot-AMI AEAD regression that
corrupts block storage on restore. Point at the merged viperblock dev fix.
---------
Co-authored-by: joshsiv-mulga <mrk@mulgadc.com>
Failed LBs previously reported an empty State.Reason, so e2e failures
read 'entered terminal state failed:' with no cause. Record the cause
on every failed transition — launch error (e.g. insufficient sys.micro
capacity), missing system credentials, missing internal mgmt return
route, and async-launch panic — persist it as StateReason, and map it
to LoadBalancerState.Reason in describe responses. Cleared when the
first heartbeat transitions the LB to active.
The internal lb subtests raced the previous subtest's sys.micro VM
deallocation: fixed 15s sleeps were not enough on capacity-tight
single nodes, so the LB entered terminal state failed (insufficient
capacity, then nats no-responders after the daemon unsubscribed the
full topic).
Replace the sleeps with createActiveLB: create LB + listener, wait for
active via the new non-fatal harness WaitForLBActiveErr, and on a
terminal failed state (wraps ErrLBTerminalFailed) tear down fully and
retry with backoff, up to 3 attempts.
go.mod referenced v1.10.0 which pre-dates LoadLiveCheckpoint. CI builds
without go.work so e2e-build failed to compile. Pin to viperblock dev tip
6af8985 (feat/snapshot-ami merged) until v1.11.0 is tagged.
* feat(gpu): add H200/RTX Pro 6000 SE models and fix MIG host resource allocation
- Fix H100 SXM/PCIe device ID labels swapped in models.go (10de:2330=SXM, 10de:2331=PCIe)
- Add H200 SXM (10de:2335) to knownModels, computeModels, and migCapableModels
- Add NVIDIArtxPro6000BlackwellSE to definitions.go mapped to g7e family (matching AWS g7e instance type)
- Add MIGHostResources() policy table: 4 vCPUs / 32 GiB host RAM per compute slice, covering all known NVIDIA MIG profiles
- GenerateMIGTypes now emits non-zero vCPU/memory for EC2 instance type descriptors
* fix(daemon): enforce CPU/memory limits for MIG instance types
Whole-GPU passthrough remains bypass-only (gpuManager.Claim is the sole gate).
MIG types now fall through to canAllocateCount so host CPU and RAM are checked
before launch — prevents over-committing a host where one physical GPU backs
multiple concurrent slices, each with non-trivial vCPU/memory allocation.
* feat: Add snapshot image promotion flow
* fix: Small changes to viperblock/spinifex so snapshot captures delta properly and instances can boot from snapshot AMIs
* fix: test coverage and more consistent prefix check
* fixed testing
* more tests
* fix: added new graceful process killer in utils, bumped NATS timeout for testing
* fix: complete at-rest encryption launch + volume lifecycle (#361)
* chore: bump viperblock to encrypted-snapshot VolumeUUID nonce fix
Pull viperblock 1044c2a (feat/snapshot-uuid-nonce-fix) so E2E builds against
the fix for the encrypted AMI snapshot metadata tag-verify failure
(cipher: message authentication failed) on instance launch. Pairs with
viperblock PR #22.
* fix: forward viperblock master key to nbdkit mount path
viperblockd loaded the master key but never passed it to the nbdkit
plugin it spawns for volume mounts, so encrypted boot volumes opened
in cleartext and failed LoadState with an encryption-state mismatch.
Forward cfg.EncryptionKeyFile into NBDKitConfig and emit the plugin's
encryption_key_file= argument when set.
* chore: bump viperblock to fresh-volume VolumeUUID mint fix
* chore: bump viperblock to snapshot-link LoadState ordering fix
* fix: route encrypted-volume config writes through viperblock keyholder
config.json is a sealed VBState whose AES-GCM tag and StateSeqNum-derived
nonce can only be advanced by the master-key holder that owns the volume.
The EC2 volume handler wrote it directly to S3, so mergeVolumeConfig had to
refuse encrypted volumes (rewriting would strip the tag) — bricking
UpdateVolumeState (attach in-use / detach available) and ModifyVolume.
Route encrypted-volume config updates through viperblockd, mirroring the
ebs.snapshot idiom: per-volume ebs.config.{volumeID} hits the owning node's
live VB (the single StateSeqNum writer); on ErrNoResponders the ebs.config
queue group opens the detached volume exclusively and reseals. Cleartext
volumes keep the direct-S3 fast path. No crypto leaves viperblockd.
mulga-siv-323
* fix: unwrap encryption envelope when snapshot handler reads config.json
CreateSnapshot and snapshotInUseByVolumes decoded config.json straight into
a VBState. For encrypted volumes config.json is an at-rest envelope
{payload, authtag}; a raw decode yields a zero VBState (SizeGiB==0), so
CreateSnapshot 500s on the size guard and the in-use scan silently drops
encrypted volumes' snapshots. Unwrap via viperblock.StateBody first,
mirroring the volume handler's mergeVolumeConfig path.
mulga-siv-324
* test: cover viperblockd ebs.config handlers
Unit/integration coverage for applyConfigUpdate, makeConfigUpdateHandler
(ebs.config.{volumeID}), and the ebs.config queue-group fallback: grow-only
size reconcile, live-VB apply, invalid JSON, apply errors, and the
detached-volume open path. Clears the diff-coverage gate (68.8%).
* chore: pin viperblock to dev tip (#22 merged)
* fix(viperblockd): seal volume block-map to predastore on detach + durability e2es (#371)
* feat: cloud-init EC2 datasource (#411)
* feat: Add snapshot image promotion flow
* fix: Small changes to viperblock/spinifex so snapshot captures delta properly and instances can boot from snapshot AMIs
* fix: test coverage and more consistent prefix check
* merge changes
* chore: remove dead SnapshotSub field and SIGTERM snapshot remnants from viperblockd
SnapshotSub was added during the NATS/SIGTERM snapshot attempts but is never
set post-rebase. Removes the field from MountedVolume and the two nil-checked
unsubscribe blocks. Re-adds the nbd import which is still needed for NBDKitConfig.
* fix: post-rebase gofmt fixes and remove dead snapshot handler code
Split gofmt-squashed lines in go.mod, vm_adapters.go, nbd_test.go,
viperblockd.go, volumes.go, and volume_durability_test.go. Removes the
dead unix-socket snapshot handler tests from viperblockd_handlers_test.go.
* test: add viperblock snapshot S3 coverage tests
Exercises snapshotVolume, snapshotRunningVolume, and snapshotStoppedVolume
past the nil-config guard using an httptest server that returns 403.
The AWS SDK credential check fails synchronously without network I/O,
keeping the tests fast while covering the S3Config construction, master key
load, and viperblock.New code paths. Brings diff coverage from 45% to 69.9%.
* fix: cloneAMIToVolume reads config via format-agnostic path
LoadStateRequest expects a viperblock binary envelope (encrypted),
but CreateImageFromInstance writes the AMI config as plain JSON via
putAMIConfig. Replace LoadStateRequest with s.amiLoader.GetAMIConfig
which uses viperblock.StateBody to handle both formats, matching the
path used by ListImages and all other AMI readers.
* fix: pin viperblock to feat/snapshot-ami pseudo-version for LoadLiveCheckpoint
go.mod referenced v1.10.0 (released from dev) which pre-dates LoadLiveCheckpoint.
CI builds without go.work so it resolved to the published version and failed.
Pin to the snapshot-ami HEAD pseudo-version until the branch is merged and tagged.
* fix: replace indexOf with slices.Index, revert e2e runner and viperblock version
- nbd_test.go: remove local indexOf helper, use slices.Index from stdlib
- e2e-suites.yml: revert ubuntu-latest back to ubuntu-26.04 with correct formatting
- go.mod: revert viperblock back to v1.10.0 (will resolve once viperblock PR merges)
* fix: remove pseudo-version pin from go.sum
* feat: drain viperblock via snapshot socket before CreateSnapshot
Before LoadLiveCheckpoint(), dial {data_dir}/viperblock/{vol}/snapshot.sock
and wait for DrainToBackend() to complete (30s timeout). This ensures WAL
chunks and the live checkpoint are current in S3 before the snapshot VB
reads them. If the socket is absent the instance is stopped and Close()
already drained.
fix: socket path used s.config.WalDir which is empty in the daemon config;
changed to s.config.DataDir + "/viperblock" to match the path the NBD
plugin uses when creating the socket.
* feat: wire snapshot/AMI UI actions and add PromoteImage NATS endpoint
- Register as AMI from snapshot detail (RegisterImage with BDM pointing at snapshot)
- Deregister from image detail (DeregisterImage with navigate-away on success)
- Promote to System from image detail (admin-only, irreversible via UI)
- NATS subject spinifex.image.promote wired through gateway → daemon → existing admin.PromoteSystemImage
- signedFetch extended with optional params to support action-keyed extra fields
* refactor: address PR review on PromoteImage types and subscription ordering
- Merge PromoteImageInput/PromoteImageOutput into PromoteImageOpts/PromoteImageResult
(add JSON tags; drop the duplicate pair)
- Inline the input variable in gateway/spx/promote_image.go
- Move spinifex.image.promote subscription to the spinifex.* group in daemon.go
* test: cover snapshotVolume/Running/Stopped error paths for diff coverage
Add encryption-key-load-error and LoadState-failure tests for the three
new viperblock snapshot paths (snapshotVolume in snapshot service,
snapshotRunningVolume and snapshotStoppedVolume in image service).
Diff coverage: 58.8% → 65.5%.
---------
Co-authored-by: Josh Siviour <mrk@mulgadc.com>
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
* fix: ECS StopTask reaps container before releasing capacity
StopTask marked a task lastStatus=STOPPED and released its capacity/ENI in
the control plane, but nothing ever told the ecs-agent to kill the container.
The container kept holding its host ports, so DescribeTasks reported a STOPPED
task whose container was still RUNNING, and a replacement task placed on the
freed capacity that rebound the same host port hit EADDRINUSE.
Wire the agent-driven cooperative stop the design intended (ecs-v1.md:165):
- Add a per-instance stop inbox (StopsPrefix/StopKey) and bus.StopDirective.
- requestStopTask marks DesiredStatus=STOPPED and posts a stop directive,
leaving lastStatus/capacity/ENI/targets untouched. StopTask, DeleteService,
and service scale-in use it; the heartbeat reaper and cluster/instance
teardown keep the immediate forceStopTask since no live agent can reap.
- PollAssignments returns pending stops; the agent lists containers by
mulga.ecs.taskID label, Removes them (kill+delete, freeing ports/netns),
then reports STOPPED. recordTaskState performs the single capacity release
on the agent's STOPPED report and reclaims the stop inbox.
- listServiceTasks excludes DesiredStatus=STOPPED tasks so a cooperatively
stopped task drops out of the service's running total immediately.
DescribeTasks now keeps the task RUNNING with desiredStatus=STOPPED until the
container is actually reaped, matching AWS ordering and eliminating the port
race.
* test: cover StopTask reap paths — poll stop-inbox drain, unplaced force-stop, agent stopTask edges
* ci: retrigger PR checks
- RegisterTaskDefinition rejects container secrets[] (InvalidParameterException)
and warns on unsupported log drivers instead of silently dropping them.
- Capture logConfiguration; only host-side json-file is collected, other drivers
warn at register and reach the agent for parity. Document json-file retrieval.
- Plumb executionRoleArn taskdef -> assign -> agent; the agent assumes it to
authorize ECR pulls, falling back to the instance role when unset.
- Add deployment objects with a health-gated rolling update honoring
deploymentConfiguration minimumHealthyPercent/maximumPercent, plus a deployment
circuit breaker with optional rollback to the last-good task definition.
EnsureLogicalSwitch/Router/PortGroup returned a client-side named-uuid on the
fresh-insert path but the real server uuid when the row already existed, so a
caller storing the returned uuid then doing Where(uuid) would miss. Resolve the
persisted uuid from the monitored cache after insert and add an explicit created
flag; topology callers now branch on created instead of uuid equality.
ReplaceACLs deleted and recreated the whole ACL set every reconcile, minting new
row uuids even for an unchanged SG and forcing ovn-northd flow recompute on every
drift tick. Diff current vs desired ACLs and no-op when unchanged (live + mock).
* test(network): in-process OVN NB reconcile harness
Add ovntest.StartNB: an in-process OVN Northbound DB backed by libovsdb's
pure-Go OVSDB server (no ovsdb-server/ovn-northd C daemons) plus a vendored
NB schema subset matching nbdb. Real LiveClient connects over a unix socket,
so reconcile scenarios exercise intent generation, reconcile wiring, LiveClient
and OVSDB transaction semantics in milliseconds without VMs.
Add live reconcile scenarios asserting NB rows and idempotency:
- VPC + subnet + SG + ENI from a hand-built IntentState
- the same seeded through JetStream KV via LoadIntentFromKV (front-to-back)
- IGW attach + drop gate + public-instance exemption (Logical_Router_Policy)
The idempotency snapshot normalizes UUID and pointer churn so it asserts
logical stability; datapath and NB->SB translation stay in the e2e suites.
* test: add orphan-prune and kitchen-sink golden live NB scenarios
Extend the in-process OVN reconcile harness with the two P0 completeness
scenarios. TestScenario_OrphanPrune_Live locks the teardown path and the
ReconcileApplyOnly startup-race guard: apply-only must not prune, full
Reconcile must, and re-pruning is idempotent. TestScenario_KitchenSinkGolden_Live
reconciles a maximal intent exercising every applyX path (VPC, public+private
subnets, two ENIs, SG with tenant rules, IGW, associated EIP, NAT gateway,
IGW/NATGW reroutes, drop gates) and asserts the normalized NB snapshot against
a checked-in golden, mechanically catching an unwired or mis-wired resource
type. Regenerate the golden with UPDATE_GOLDEN=1 go test.
* fix: repoint DB-node ovn-northd at RAFT member list; extend add-nat deadline
Two related fixes for multi-node OVN clusters:
- setup-ovn.sh (mulga-siv-504): in the clustered --management branch, point
ovn-northd's client NB/SB connections at the full RAFT member list derived
from OVN_REMOTE, so the active northd follows the leader instead of dialing
its local member only. Without this, northd stops advancing SB_Global.nb_cfg
once leadership moves off the node, wedging the ovn-nbctl --wait=hv barrier.
Guarded on a real member list so the standalone single-DB-node path is
unchanged.
- utils/nats.go (mulga-js-206): raise the add-nat request deadline from 10s to
45s so the client no longer rolls back before the server-side --wait=hv flows
barrier (30s) completes.
* fix: stop ovn-northd restart from tearing down split NB/SB ovsdb servers
The packaged ovn-northd.service ExecStop runs 'ovn-ctl stop_northd' without
--ovn-manage-ovsdb=no, so restarting northd to apply OVN_CTL_OPTS also stops the
separately-managed NB/SB ovsdb-server units. On a clean clustered bring-up this
races the freshly-started DBs and kills them, so setup-ovn's wait for the NB
socket times out and the RAFT create/join aborts. Install an ovn-northd.service
drop-in overriding ExecStop with --ovn-manage-ovsdb=no so northd never touches
the DB lifecycle (mulga-siv-504).
* test(e2e): capture SNAT datapath on NAT GW egress failure
TestNATGateway captured nothing when the private-VM egress probe failed, so
the baremetal nightly's intermittent 100% packet loss left no OVN/OVS state in
the failure bundle to root-cause it. Register harness.OnFailure before the
egress-verify poll to dump the SNAT datapath (OVN show, NAT rules, port
bindings, br-int flows, conntrack, ARP) keyed on the NAT GW external IP,
matching the eip/vpc datapath tests.
Also add a br-ext localnet flow capture to dumpDatapathState: br-int flows miss
the egress hop where SNAT'd traffic leaves for the physical uplink, which is
exactly where an egress black-hole would show.
* test(e2e): add OVN pipeline diagnostics to NAT GW egress capture
Capture the gateway router's per-stage logical-flow packet counts
(ovn-sbctl --stats lflow-list), MAC_Binding, static routes and the VM's
logical switch port alongside the existing OVS/conntrack/ARP dumps. When
the SNAT flow reads n_packets=0 while the guest transmits, these pinpoint
which pipeline stage drops the packet upstream of lr_out_snat.
* test(e2e): dump full lflow-list + datapath ct/flows for egress RCA
The IP-filtered lflow dump hides the default-drop ACL flow (its match
carries no IP), so a pre-SNAT egress drop at ls_out_acl only showed the
allow-flows at n_packets=0, never the deny that ate the packets. Replace
it with a full ovn-sbctl --stats lflow-list, echoed so it survives in the
CI stdout log. Swap the conntrack capture to ovs-appctl dpctl/dump-conntrack
(the standalone conntrack binary is absent on the baremetal runner) and add
dpctl/dump-flows -m for kernel megaflow hit counts.
* fix: claim gateway cr-port on every reconcile miss and stop masking launch failures
ensureGatewayClaimed nudged ovn-controller recompute only once then polled
until timeout, so a fresh VPC bring-up or chassis flap that fired the single
nudge too early left the gateway chassisredirect port permanently unclaimed.
Recompute on every miss instead, mirroring ensureGuestPortDatapath.
aggregateResults and distributeInstancesSpread masked any node launch failure
(including the resulting 5-minute launch timeout) as InsufficientInstance
Capacity. Surface the real node error when one exists; keep the capacity code
only for genuine capacity shortfalls.
* feat(eks): multinode control-plane networking, IAM, and CoreDNS parity
Integrates the EKS multinode fix set for the managed-control-plane topology
so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS
EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the
node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it.
EKS:
- Route apiserver egress (exec/logs/webhooks) through konnectivity
(apiserver-network-proxy): CP servers run konnectivity-server behind the
cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod
traffic rides the agent-initiated reverse tunnel instead of an unroutable
CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every
replica (HA-correct).
- Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay
flannel mesh SG rule authorized at cluster launch with ordered teardown.
- Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay.
- Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master
tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity.
- Spread nodegroup workers across hosts; addon readiness falls back on
workload rollout when gvks are absent.
- elbv2: allow the proxy_protocol_v2.enabled target-group attribute.
EBS:
- Move volume attachment state to state.json so a live nbdkit VB cannot
clobber it; route mounted-volume state updates through the live VB with a
retry for in-use marks racing the mount responder; keep the boot volume
in-use across stop/crash unmount.
* fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM
missingOrchestrationDeps gated on s.deps.IAM, but production wires only the
lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only).
The gate therefore reported missing=[IAM] on every node forever, so
CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even
though the IAM service was fully up via the provider.
Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else
deps.IAMProvider()) the nodegroup/role code already uses — so the protective
check still rejects a node with no IAM service but passes when the provider
supplies one. Add a regression test that sets only IAMProvider and asserts the
gate is satisfied; the existing nil-both test still rejects.
* fix(eks): let eks-token-webhook use IMDS creds when static keys absent
The CP VM launches on an instance profile, so buildK3sUserData omits
EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys
before eksgw.New could fall back to the IMDS SDK chain, so the webhook
aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver
crashed on the missing --authentication-token-webhook-config-file file,
stalling the cluster at CREATING until it timed out to FAILED.
Drop the static-key precondition (matching the sibling eks-gateway-publish
and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/
cluster gates.
* fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane
The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt,
which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a
169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing
169.254.169.254 and steering IMDS off the data NIC away from the per-tap
br-imds datapath, where the host RST it. eks-gateway-publish then failed to
sign requests and no bootstrap artifacts were ever published, so clusters sat
CREATING until they timed out to FAILED.
Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address
or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI)
are unaffected and IMDS falls through the data NIC's default route to the
br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net
deletes any default route on a NIC declared NIC<n>_DEFAULT=0.
* fix(eks): grant eks:WebhookTokenReview to the control-plane instance role
The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway
broker at POST /clusters/{name}/token-review, signing with the control-plane
VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes
against eks:WebhookTokenReview, but the role's inline policy granted only
eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every
relay and the apiserver returned 401 to all external kubectl.
Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile
re-asserts the inline policy on every control-plane instance-profile ensure, and
the gateway evaluates authorization per request, so existing clusters converge
without a rebuild once the shared role is updated.
* feat: add OVSDB RAFT cluster primitives to setup-ovn.sh
Add --db-cluster-local-addr / --db-cluster-remote-addr flags so a
management node can run the OVN NB/SB databases as a native OVSDB RAFT
cluster instead of a single non-clustered ovn-central. When a local addr
is given, write a shared OVN_CTL_OPTS to /etc/default/ovn-central (NB
RAFT 6643 / SB RAFT 6644) and restart the per-DB units directly, since
the ovn-central aggregator is ExecStart=/bin/true and won't restart its
children. Empty remote addr creates the cluster (init node); a set remote
addr joins it. NB/SB client set-connection runs only on the create node
(RAFT replicates it cluster-wide).
The single-node --management path is unchanged when no cluster addrs are
supplied, so dev boxes and existing single-node clusters keep working.
--ovn-remote already passes through verbatim, so a comma-separated SB
endpoint list works for compute nodes; documented in usage.
* feat: OVN client multi-endpoint failover for RAFT clusters
Accept a comma-separated OVN NB/SB endpoint list so vpcd can target a
clustered DB. config.ParseEndpoints splits the list (trimming whitespace,
dropping empties); live.go Connect passes one libovsdb WithEndpoint per
endpoint, so the client tries them in order, fails over on disconnect, and
follows RAFT leader redirects. Single-endpoint strings still yield one
endpoint, so single-node dev is unchanged.
ovn-sbctl --db= already accepts the raw comma list, so the SB CLI
consumers (discoverChassis, gateway_claim) need no change.
* feat: write clustered OVN NB/SB endpoints across the RAFT quorum
spx admin init/join now emit comma-separated OVN NB/SB endpoint lists for
the first three quorum nodes (formation.BuildOVNDBAddrs) instead of loopback
or a single leader address, so vpcd fails over across the cluster.
firstboot defers OVN bring-up to the provisioning controller when it owns
formation (SkipFormation): a standalone single-node .db would otherwise block
the controller's clustered create/join.
* test: add multinode OVN RAFT cluster + failover e2e; document clustered topology
ovn_raft_test.go asserts NB/SB cluster/status reports a 3-server quorum with
one leader on every DB node, then stops the NB leader's ovn-central and proves
an NB write still lands via a surviving endpoint.
Docs: vpc-networking gains the multi-endpoint vpcd config and a RAFT
cluster-status debugging section; network-connections lists the 6643/6644 RAFT
ports.
vpcd's libovsdb NB client was a single long-lived connection with no
reconnect logic. After the NB server restarted (host reboot, service
restart) every vpcd holding a dead/half-open socket never re-established
it, so OVN-write handlers blocked until the caller timeout and CreateVpc
returned ServerInternal intermittently (queue-distributed across vpcd).
Add WithInactivityCheck to the OVSDB client: a periodic Echo probe
detects the dead peer and triggers reconnect against the configured
endpoint, with infinite backoff so a daemon keeps retrying across a long
outage. Bound the SG subscriber handlers with a timeout context so a
stuck OVN op frees the worker instead of hanging on context.Background.
NLBs hard-rejected customer security groups and always minted a managed SG that
auto-opened listener ports, so aws-load-balancer-controller (which assigns SGs to
type=external NLBs) got a 400 and looped. Accept up to 5 customer SGs on an NLB:
when supplied they replace the managed SG on the ENIs and the caller owns the
listener-port rules (CreateListener already no-ops its auto-open when there is no
managed SG); without them the managed-SG default is unchanged. SetSecurityGroups
follows AWS create-time semantics — replace SGs on an NLB created with SGs, reject
adding them to one created without.
CreateVolume hardcoded iops to 3000, silently dropping a caller-supplied Iops,
and rejected non-gp3 VolumeType with the generic InvalidParameterValue. Honor
input.Iops within the gp3 envelope (3000 baseline, 500 IOPS/GiB, 16000 ceiling)
and return the volume-type-specific UnknownVolumeType for unsupported types.
Closes EBS CSI API gaps #3-4 (siv-380); gp3 Throughput (gap #2) deferred.
* feat: IMDS-capable gateway signer for lb-agent + EKS clients
Add internal/gwsign, a SigV4 signer that draws credentials from either
fixed static keys or the AWS SDK default credential chain. In a guest VM
the chain falls back to the in-VM IMDS endpoint, yielding scoped,
auto-rotating instance-role credentials; a session token is signed as
X-Amz-Security-Token so the gateway can verify temporary (ASIA) creds.
Wire it into the lb-agent (lbagent.Agent) and the EKS on-VM client
(internal/eksgw). When static keys are absent the clients now sign with
IMDS credentials instead of failing; when present, behaviour is
unchanged (static path). This is the client-side foundation for dropping
long-lived secrets from VM user-data.
Credentials are retrieved per request so rotated IMDS creds take effect
without an agent restart.
* feat: IMDS instance-role creds for EKS server + LB VMs (Phase B/C)
Attach a Spinifex-managed system instance profile to the k3s control-plane VM
and the elbv2 LB VM so both authenticate to the gateway with scoped, rotating
IMDS instance-role credentials instead of a long-lived static secret baked into
user-data.
- Add a shared handlers/iam.EnsureSystemInstanceProfile that find-or-creates a
role (EC2 trust) plus its instance profile and asserts an inline permission
policy. The gateway enforces per-action authz on assumed-role principals, so
the roles carry exactly the actions each VM calls:
- EKS server: eks:PublishInternal, eks:ListInternalAddons.
- LB agent: elasticloadbalancing:LBAgentHeartbeat, GetLBConfig.
- Plumb IamInstanceProfileArn through sysinstance.SystemInstanceInput and both
daemon launch paths (BootAMI via PrepareRunInstances->RunInstance; direct-boot
microVM via RunInstance), which persist it so the in-VM IMDS endpoint serves
the role.
- Drop the baked secrets when a profile is attached: omit EKS_ACCESS_KEY/
EKS_SECRET_KEY from the k3s first-boot env and LB_ACCESS_KEY/LB_SECRET_KEY from
the lb-agent fw_cfg env; relax k3s server validation to accept profile-only;
drop the EKS key requires from the first-boot and state-report scripts.
- Both verticals fall back to baked static creds when IAM is unwired, so an
absent master key never blocks a launch.
* fix: resolve IMDS instance-role creds under the instance account for system VMs
Three boot-path defects kept the LB VM's lb-agent from authenticating to the
gateway with IMDS-served instance-role creds; each produced a silent empty IMDS
role list and a heartbeat failing with "unexpected empty EC2 IMDS role list".
Found and fixed against env19.
- Build the system IAM ensurer lazily on the first system-VM launch instead of
eagerly at daemon startup, where it raced the NATS KV backend and fell back to
baked static creds permanently. ELBv2 and the EKS deps now hold an IAMProvider
resolved at launch time.
- Create the LB instance profile in the system account the LB VM runs under
(utils.GlobalAccountID), not the LB-owner account. IMDS resolves the profile
under the instance's account, so a customer-account role was invisible to it.
- Record the attached instance's owning account on the ENI (InstanceOwnerId,
mirroring AWS Attachment.InstanceOwnerId) and resolve the instance and every
IAM/STS lookup under it. The LB ENI lives in the customer account while the
system LB VM and its role live in the system account; keying the whole IMDS
chain off the ENI account left the instance invisible and the role lookup
missing. VPC leaves (subnet/VPC/SG CIDR) keep using the ENI account.
A fresh LB on env19 then authenticated end-to-end: IMDS serves a non-empty role
list plus ASIA creds, and the gateway accepts the heartbeat as an assumed-role
principal with no static keys in user-data.
* test(e2e): assert LB VM authenticates via IMDS instance-role creds
The LB suite's targets-healthy assertion already depends on the lb-agent
authenticating to the gateway, and with static keys dropped from user-data that
auth is IMDS-only. Add an explicit check that a recent gateway authentication
for the LB used an assumed-role (ASIA) principal, so a regression to the
instance-account IMDS resolution fails with a clear message instead of a
mysterious datapath timeout. The LB VM is a system-account instance invisible to
the customer AWS API, so the assertion observes the gateway journal on the
cluster nodes; it is time-bounded to avoid the stale-console-log false-fail and
skips when no node is SSH-reachable.
Cell 19 bootstraps with --external-mode=nat plus the CI static pool as
the public wan pool and runs the natuplink suite, covering the Tier-1
transit datapath and Tier-2 EIP ingress. The run step exits 0 when
natuplink.test is absent from the tests tar so scheduled runs against
branches predating the suite stay green.
Add a second build+publish step to the EKS AMI workflow so the same
manual run also builds the ecs-agent system AMI and uploads its qcow2 +
.sha256 sidecar to R2. Reuses the shared checkout, Go, and tooling setup;
renames the workflow to cover both node images.
Flip the k3s server apiserver to anonymous-auth=false. The only
consumer of the anonymous path was mulga-eks-state-report's /healthz
probe; switch it to an authenticated kubectl get --raw=/healthz through
the node admin kubeconfig (an unauthenticated curl now returns 401).
The NLB target group uses a TCP health check, and the host-side
reconciler rides the authenticated NATS state-report, so neither
depends on the anonymous apiserver path.
* fix: free host-netns :80 for ECS bridge/host tasks via cred-endpoint DNAT
bridge and host network-mode tasks share the VM host netns where the
ecs-agent credential endpoint bound 169.254.170.2:80, so a task binding
0.0.0.0:80 hit EADDRINUSE and crash-looped. Bind the listener on the high
proxy port 51679 and DNAT 169.254.170.2:80 to it (OUTPUT for host/bridge
task traffic, PREROUTING for the awsvpc credential veth), keeping the dst
IP on the endpoint so no route_localnet is needed. Nothing holds a socket
on :80, so host/bridge tasks can bind it.
e2e runs nginx on :80 in all three network modes; the settle guard catches
a regression where a crash-looping :80 bind flaps RUNNING then STOPPED.
* test: validate ECS host-netns :80 binding via bridge task
The bridge subtest now runs nginx on :80 to prove the cred-endpoint DNAT
frees the host netns (pre-fix it crash-looped on EADDRINUSE). host mode
keeps a no-port command: it shares the same host netns, so two :80 tasks
cannot coexist on one node (one :80 per host, as on AWS), and StopTask
does not synchronously reap the prior container's port.
When NATS disables JetStream at runtime (e.g. a filesystem permission error
flushing stream state) it keeps the process alive and serves KV as read-only.
That is not a process failure, so spinifex-nats.service's Restart=on-failure
never fires, and KV-backed startup (awsgw IAM bucket init) stalls for minutes
until an operator restarts NATS by hand.
Add a JetStream liveness watchdog:
- build/scripts/nats-js-watchdog.sh probes the NATS monitoring endpoint
(http://127.0.0.1:8222/healthz?js-enabled-only=true). Only when the service
is active and JS fails every sample of a short multi-sample window (so
transient startup/quorum 503s are ignored) does it systemctl try-restart
spinifex-nats.
- spinifex-nats-watchdog.{service,timer} run it as a oneshot every 30s, with a
60s boot delay so cluster quorum settles first.
- spinifex.target Wants= the timer so it starts with the platform.
The NATS-internal permission race itself is a server quirk outside spinifex's
control; this restores service automatically when it recurs.
The in-cluster AWS LoadBalancer Controller provisions its own ALBs
(k8s-toc-*) and security groups (k8s-traffic-toc-*) in the customer VPC
via the awsgw ELBv2/EC2 APIs. Spinifex never tracks these, so cluster
teardown left them orphaned: DeleteSecurityGroup / DeleteVpc then failed
with DependencyViolation and the cluster wedged in DELETING.
purgeClusterInfra now reaps them by the AWS-standard ownership tag
elbv2.k8s.aws/cluster=<name>, scoped to the customer VPC, after the
cluster NLB teardown and before the SG / VPC block:
- ReapLBCLoadBalancers lists account LBs, filters to the customer VPC,
confirms the ownership tag via DescribeTags, and deletes by ARN.
- DeleteClusterSGs deletes any VPC SG carrying the ownership tag (reusing
the DependencyViolation retry budget), in addition to the named cluster
SGs, after rules are revoked.
- DescribeTags is added to the narrow nlbProvisioner interface.
Errors join teardownErrs so a failed reap keeps the cluster in DELETING
for a retry rather than completing with leaked, VPC-pinning infra.
TerminateWorkerInstances routes terminates via the per-instance ec2.cmd.<id>
topic, whose subscription is torn down when a VM stops. A stopped (incl.
stopped+wedged, volume remount broken) worker therefore has no responder, and
the old code treated ErrNoResponders as already-gone and returned nil. The
worker was still in shared KV with its ENI intact, so the nodegroup record was
deleted while the ENI kept pinning the customer subnet/VPC, wedging DeleteCluster
in DELETING on DependencyViolation until a manual ec2 terminate-instances.
On ErrNoResponders, fall back to the ec2.terminate queue group (mirroring the
gateway TerminateInstances path), which runs TerminateStoppedInstance against
shared KV — deleting the worker's volumes, IP, and ENI without a running QEMU.
NotFound or no responder is idempotent success; any other error surfaces so the
teardown backstop retries.
TerminateInstances marks the EC2 record terminated and drives node-local
teardown, but when that teardown runs on a node other than the one hosting
QEMU (per-instance subscription lost, request falls back to the ec2.terminate
broadcast), the process survives and keeps holding its OVN logical port / IP,
blocking rebuilds.
Add OrphanQEMUReaper (node-local, ScopeNodeLocal): it inspects only the
terminated KV bucket and reaps a live process found via the instance's local
PID file. Running, pending, and stopped instances are never touched. Whichever
node holds the orphan PID file is the one that reaps it; every other node's
lookup misses and is a no-op.
Add utils.ProcessAlive (signal-0 liveness) and utils.ForceKillProcess
(SIGKILL + bounded wait), guard shutdownAndUnmount's force-kill on liveness,
and register the reaper in the daemon GC loop.
reconnectInstance re-asserted the instance running state but never the boot
volume's in-use state, so after a daemon restart describe-instances showed the
VM running while describe-volumes showed its root volume available — a
split-brain that makes start-instances attempt to double-launch a live QEMU.
Extract the launch path's boot-volume promotion into markBootVolumesInUse and
call it from reconnectInstance so launch and recovery keep volume state
consistent. Add a reconnect unit test asserting the boot volume is marked
in-use and non-boot volumes are left to their own attach flow.
The daemon-side in-use gate trusts persisted volume State/AttachedInstance,
which the attach/detach lifecycle can leave desynced: a running root volume
was observed with State:"" and AttachedInstance:"" while the instance's
BlockDeviceMappings still listed it, so DeleteVolume passed the gate and
removed the durable backing under a live QEMU.
Add an authoritative guard at the gateway DeleteVolume path: fan out the
strict DescribeInstances survey and refuse with VolumeInUse when any
non-terminated instance's BlockDeviceMappings reference the volume. Fail
closed on an incomplete cluster view rather than delete against a partial
survey. Terminate-time cleanup uses the daemon volume service directly and is
unaffected.
Drives the ECS data plane against the local awsgw: a customer VPC plus a real
container instance launched from the spinifex-ecs-node AMI, then standalone
tasks in all three network modes and an awsvpc service fronted by an ALB target
group. Adds an ECS client to the harness, a Makefile target, and a
service-interfaces suite entry.
bridge/host tasks share the host netns where the agent credential endpoint owns
169.254.170.2:80, so they run a no-port command and assert placement + mode;
awsvpc and the ELB service prove real :80 serving. waitTaskRunning re-checks
after a settle so a transient RUNNING cannot pass.
Wire crash detection and host-resource signals through to the AWS-faithful
DescribeInstanceStatus API and the spx CLI, closing out the EC2 health
monitoring / self-healing work.
Phase C — QMP health monitoring:
- qmpHeartbeat now tracks consecutive query-status failures; at
QMPMaxConsecutiveFailures it checks process liveness, triggering HandleCrash
on a dead QEMU and marking an alive-but-wedged process impaired.
- InstanceHealthState gains QMPConsecutiveFailures, LastQMPSuccess, ImpairedSince.
Phase E — DescribeInstanceStatus mapping:
- buildInstanceStatus maps running VMs to ok / impaired (QMP-unresponsive,
with ImpairedSince) / initializing (2-min launch grace), non-running to
not-applicable.
- SystemStatus reflects node memory pressure via ResourceManager
HostUnderMemoryPressure (live MemAvailable below the host reserve).
Phase C2 — observability:
- spx get vms gains HEALTH and CRASHES columns derived from vmHealthLabel.
Tests cover crash/QMP helpers, status mapping, memory-pressure system status,
and the health label.
On startup the agent had no reconcile pass: Run registered, heartbeat, and
immediately polled assignments with an in-memory dedup map. After a restart the
gateway re-delivers the instance's assignments and the agent re-ran them,
orphaning containers still running from before and double-starting tasks.
Add a reconcile() pass that runs once before polling: it lists the runtime's
containers, and for each task still running in this cluster re-registers its IAM
credentials, re-attaches the exit-wait, and refreshes RUNNING on the gateway.
The adopted task set seeds pollAssignments so re-delivered assignments are acked
but not re-run.
- runtime: add a label-agnostic List(ctx) []Container to the Runner seam, with a
containerd implementation and fake support.
- taskLabels: stamp credID / taskRoleArn / eniMac (when set) so a container is
self-describing enough to re-register creds and tear down its netns on adoption.
- pollAssignments takes the seed dedup map.
Best-effort: a nil runner or a List error degrades to today's poll+run behaviour.
* docs: add ECS (Elastic Container Service) guide + register ecs-quickstart workbook
Add a Container Services -> ECS doc mirroring the EKS/ECR shape: concepts
(cluster, task definition, task, service, container instances via the
ECS-AMI EC2 launch type), an aws ecs / console / Terraform quickstart,
task IAM roles via the credential endpoint, awsvpc networking, and the
honest v1 limitations (no service discovery, awslogs discarded, no rolling
deployments, no capacity providers, no distinct execution role).
Convert the ecs-quickstart workbook README to the doc-page format
(frontmatter + overview/prerequisites/instructions/troubleshooting) so it
renders as a Terraform Workbooks entry alongside the EKS workbooks.
* docs: add node-image prerequisite banner + infra diagram to EKS and ECS pages
Add a GitHub-style [!IMPORTANT] banner to the EKS and ECS prerequisites
flagging the required node-image import (eks-node / spinifex-ecs-node),
mirroring the install page's WAN-bridge banner, each with a describe-images
verification command.
Add a basic Spinifex topology diagram (SVG) to each Overview, in the
tf-nginx-alb visual style: the EKS managed control plane + node-group
workers, and the ECS ALB + container instances with the gateway/agent
and task-credential path.
End-to-end ECS stack against the Spinifex gateway: cluster, task definition
with a task role, container instances from the spinifex-ecs-node AMI, and an
awsvpc service fronted by an ALB target group. Unlike the console's provision-
capacity action, the workbook spells out the two things a Spinifex container
instance needs that AWS hides: a LAN-reachable gateway URL and the gateway CA,
both seeded into cloud-init user-data; instance creds come from IMDS via the
ecsInstanceRole profile. Live-verified apply (HTTP 200 through the ALB) and
destroy round-trip.
The agent heartbeats by periodically re-registering through the gateway
(RegisterContainerInstance). After a control-plane restart, the heartbeat
reaper marks instances DRAINING for the missed-heartbeat window, but the
re-register path only refreshed LastSeen and never restored the status, so a
reaped instance stayed DRAINING forever even though its agent was healthy and
re-registering.
Mark reaper-induced DRAINING with a Reaped flag and restore the instance to
ACTIVE on the next re-register, clearing the flag. An operator
UpdateContainerInstancesState drain clears Reaped, so an intentional drain is
never undone by the live agent.
* feat(ecs): provision EC2 container-instance capacity via IMDS instance-role
Adds a ProvisionCapacity gateway action that launches spinifex-ecs-node
EC2 instances which register as ECS container instances, fixing
RESOURCE:placement when a cluster has no capacity.
Agent control-plane now SigV4-signs gateway calls with instance-role
credentials read from IMDS (no static keys seeded in user-data). The
gateway already verifies STS/temp creds, so the instance role carries an
ecs:* policy + ec2 trust; ensureECSInstanceProfile find-or-creates the
ecsInstanceRole instance profile attached at launch.
Backend resolves the spinifex-ecs-node AMI by spinifex:managed-by=ecs
tag, renders keyless user-data, and launches via the customer
RunInstances path. Frontend adds a hand-signed JSON-1.1 ProvisionCapacity
call, a provision-capacity dialog on the container-instances tab, and the
ECS console section nav + system-image gating.
* feat: auto-assign EIPs to awsvpc service tasks; add ECS task/service detail pages
Wire AssignPublicIp=ENABLED through the service reconciler so awsvpc service
tasks automatically allocate and associate an Elastic IP on the RUNNING
transition, and release it on STOP. assignTaskPublicIP/releaseTaskPublicIP
mirror the target-group registration hooks in recordTaskState and forceStopTask;
the public IP and allocation ID are persisted on the task record and surfaced in
DescribeTasks attachments as publicIPv4Address.
Add AWS-parity ECS task and service detail pages showing private and public
endpoints, container state, and networking configuration, reachable by clicking
a task or service in the cluster tabs.
* test: cover natsEIPManager allocate/associate/release over NATS
ListTaskDefinitions appended every revision regardless of the requested
status, so deregistered (INACTIVE) task definitions never dropped off the
default listing and the console could not clear them. Filter by status,
defaulting to ACTIVE to match AWS.
Add an ECS console mirroring the EKS/ECR route shape: a single ECS nav
entry, cluster list and detail with Services/Tasks/Task Definitions/
Container Instances tabs, plus create and teardown paths.
Adds three full-page wizards to drive container deploys end-to-end:
Register Task Definition, Run Task, and Create Service. RunTask surfaces
a 200-with-failures placement result as an error instead of silently
navigating away.
Includes the @aws-sdk/client-ecs client singleton, ecs queries and
mutations, zod form schemas, and co-located tests.
STOPPED task records were never pruned: ListTasks enumerated every task a
cluster ever ran and the per-account KV grew unbounded. Add a leader-gated sweep
tick to the scheduler cron (alongside the reaper/reconcile) that deletes task
records STOPPED longer than a 1h retention window — keeping a just-stopped task
describable for DescribeTasks / UI exit reason, matching AWS's STOPPED window.
A task missing its StoppedAt timestamp is never pruned (defensive). Single-writer
(leader only), reuses the reap() bucket walk.
Wire the four NotImplemented ECS teardown actions so an IaC stack and the
console can reverse a deployment:
- DeleteCluster cascades: force-stops every task (releasing ENIs, deregistering
TG targets, returning capacity), then sweeps the cluster's whole KV subtree.
Unlike AWS it cascades rather than rejecting a non-empty cluster, so terraform
destroy round-trips regardless of teardown ordering.
- DeregisterTaskDefinition marks a specific family:revision INACTIVE (kept
describable); a bare family is rejected, matching AWS.
- DeregisterContainerInstance removes an instance, force-stopping its tasks when
Force is set and rejecting a non-empty instance otherwise.
- UpdateContainerInstancesState sets ACTIVE/DRAINING; draining force-stops the
instance's service-owned tasks so the reconciler relaunches them elsewhere.
Reuses the existing forceStopTask primitive and the reaper drain pattern; adds a
deleteKeysWithPrefix KV helper and INACTIVE status constants.
Serve task-role credentials to containers the AWS-faithful way: an HTTP
endpoint at 169.254.170.2:80 inside the ecs-agent (host VM netns) serving
short-lived credentials at /v2/credentials/{credID}. Containers find it via
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI, injected per container.
The agent mints credentials by SigV4-signing sts:AssumeRole(taskRoleARN)
against the gateway with its instance-role credentials, mirroring
internal/ecrauth. This reconciles ecs-v1 Q4/Q10 (STS over NATS + scheduler
JWT assertion) with the 4f rewire that moved the agent entirely onto the
gateway: the agent has no NATS connection and already holds instance creds,
so it assumes the task role directly. Credentials are cached per credID and
refreshed near expiry.
Each awsvpc task netns gets a credentials veth into the host netns and a
host-routed /32 to the endpoint, keeping it off the task ENI's VPC route.
The scheduler now persists taskRoleArn on the task definition and carries it
in the assign payload so the endpoint activates end to end.
* fix: serialize EBS hot-plug per instance to prevent PCIe port race
Concurrent AttachVolume calls to one instance both allocated the lowest free
hot-plug PCIe root port via nextHotplugEBSPort (which reads live QEMU state),
then issued QMP device_add on the same hotplug-ebs bus. The second device_add
failed with 'PCI: slot 0 function 0 already occupied by virtio-blk-pci',
surfacing as nats:timeout -> ServerInternal and a CSI retry-storm
('retry quota exceeded'); volumes never attached.
Add a per-instance attachMu and hold it across the whole AttachVolume and
DetachVolume critical sections, mirroring the ENI hot-plug serialization.
The mutex is non-persisted and shared via the Manager's stable *VM.
* fix: allocate EBS hot-plug PCIe port from in-memory accounting
AttachVolume derived the free PCIe hot-plug root port from a live
query-block scan (nextHotplugEBSPort), extracting the bus with a
hotplug-ebs(\d+) regex on each device's QDev. Real QEMU reports a
hot-plugged blockdev by id (/machine/peripheral/vdisk-vol-xxx/
virtio-backend), with no hotplug-ebs component, so the regex never
matched, the used set stayed empty, and the allocator always returned
port 1. Only the first EBS volume per instance attached; every later
attach collided on "PCI: slot 0 function 0 already occupied by
virtio-blk-pci", rolling back the blockdev, killing NBDKit, and
surfacing as nats: timeout / ServerInternal to the CSI driver, which
then exhausted its retry quota.
Allocate the port from authoritative in-memory accounting instead:
record the chosen port as EBSRequest.HotplugPort (persisted, so it
survives a daemon restart) and pick the lowest unclaimed port with
freeHotplugEBSPort over the instance's recorded requests. Hold
attachMu across allocate->device_add so concurrent CSI attaches cannot
pick the same port. DetachVolume already removes the request, freeing
the port. Drops the query-block round-trip from the attach path.
* feat(ecsgw): on-VM SigV4 client for the ECS agent→gateway channel
Adds internal/ecsgw, the twin of internal/eksgw: a TLS + SigV4 (service "ecs")
client the ecs-agent will use to reach the AWS gateway over HTTPS instead of
connecting to host NATS directly. Calls are AWS JSON 1.1 with an X-Amz-Target
action; the gateway relays onto ecs.* host-side, keeping NATS cluster-internal.
* feat(ecs): SubmitTaskStateChange gateway action → recordTaskState
Wires the AWS-API SubmitTaskStateChange action end to end (gateway action →
ecs.SubmitTaskStateChange NATS → daemon → Service.SubmitTaskStateChange) so a
gateway-routed agent reports task state without touching the NATS bus. The
handler maps the SDK input onto the same bus.TaskState shape and converges on
recordTaskState; the account is authoritative from the SigV4 caller, so an
instance cannot report state for another account's task.
* feat(ecs): durable assign KV inbox + PollAssignments gateway action
Scheduler writes task assigns into a per-instance KV inbox
(clusters/{cluster}/assignments/{instance}/{task}) instead of a
fire-and-forget core-NATS publish, and a new PollAssignments gateway
action lets a gateway-routed agent drain its inbox over TLS+SigV4.
At-least-once: an unacked assign is re-delivered on the next poll;
the agent acks accepted assigns (AckTaskIDs) and STOPPED transitions
reclaim stale inbox entries. The Poll response carries bus.Assign with
an RFC3339 time, so it is encoded with encoding/json (RawJSONActions)
rather than the jsonutil epoch-second marshaler.
* feat(ecs-agent): rewire control-plane off NATS onto the AWS gateway
The agent no longer connects to NATS. It registers, heartbeats, reports
task state, and receives assignments entirely through the AWS gateway
over TLS+SigV4, using seeded IAM credentials — keeping the NATS bus
host-internal and closing the trust-boundary gap (a tenant container
instance must not hold a shared bus token).
- controlPlane seam over ecsgw.Client replaces the NATS publisher:
Register (RegisterContainerInstance), SubmitTaskState
(SubmitTaskStateChange), PollAssignments.
- Heartbeat folds into a periodic idempotent re-register (refreshes
LastSeen); bus.Heartbeat is dropped from the agent.
- subscribeAssign (NATS subscribe) becomes a poll loop: drain the
inbox, dispatch each assign once, ack accepted taskIDs on the next
poll (at-least-once).
- config: ECS_ACCESS_KEY / ECS_SECRET_KEY / ECS_POLL_INTERVAL added,
ECS_NATS_URL removed; cloud-init initd seeds the IAM creds.
Tests move from the NATS publisher fake to a controlPlane fake.
* feat(ecs): cloud-init cred-seed + ecs-node bring-up script
ECS container instances are plain EC2 booted from the spinifex-ecs-node
AMI (AWS-faithful: no ECS-specific launch API). This renders the
cloud-init user-data that seeds /etc/spinifex-ecs/agent.env (gateway
URL/CA/region/cluster + IAM creds) and the gateway CA, then runs the
instance. The agent registers, heartbeats, polls assignments, and
reports task state entirely through the AWS gateway over TLS+SigV4 —
never NATS.
The guest dials the customer-facing gateway address (network advertise
IP), not the mgmt bridge, mirroring the EKS GatewayURL/AddonGatewayURL
split. Seeds the system key for now; scoped per-instance creds are a
follow-up (4g task-role endpoint).
* fix(ecs): reconcile dev merge — restore #418 ENI release + netns wiring
The 'Merge branch dev' resolution dropped code where Sprint 4d (#418,
awsvpc ENI-per-task) and the gateway control-plane rewrite (#419) touch
the same paths:
- recordTaskState STOPPED branch: reclaimTaskENI was clobbered by
reclaimAssignInbox; both are needed (release the task ENI AND drain the
KV inbox). Restores TestRecordTaskState_Awsvpc_ReleasesENIOnStopOnce.
- ecs-agent Agent: the netns field, its init, and the setupTaskNetns/
teardownTaskNetns wrappers were lost when the controlPlane Agent struct
won the merge, leaving taskrun.go calling undefined methods.
- eni_test assign assertion read the assign over NATS; #419 delivers it via
the durable KV inbox, so it now polls via PollAssignments.
- taskrun_test netns cases used the removed historyPub fake; switched to
fakeCP to match the controlPlane seam.
* feat(ecs): service controller + ELBv2 target-group registration (Sprint 4f)
Land the ECS Service controller on top of the 4e scheduler/RunTask flow:
- CreateService/UpdateService/DeleteService/DescribeServices/ListServices.
REPLICA only; DAEMON and serviceRegistries rejected in v1 (Q15).
- StartTask (explicit placement) and StopTask.
- Leader-driven replica reconciliation: a 10s scheduler tick converges each
service's running task count on desiredCount, launching the shortfall via
RunTask (tasks tagged group=service:{name}) and force-stopping the surplus.
- ELBv2 target-group registration: the scheduler registers a task's awsvpc ENI
IP on the RUNNING transition and deregisters on STOPPED via the existing
elbv2.RegisterTargets/DeregisterTargets NATS handlers (Q8, single-writer).
- forceStopTask consolidates the control-plane stop (STOPPED state, capacity +
ENI release, inbox drain, TG deregister); reused by StopTask, scale-in, and
the heartbeat reaper (the reaper now deregisters targets per Q13).
- Wire the 7 new actions through the gateway action map, NATS adapter, and
daemon subscriptions; add ServiceNotFoundException.
* fix(ecs-agent): anonymous pull for non-ECR images + normalize image refs
The agent minted an ECR authorization token via IMDS for every image, so
public docker.io refs failed with 'IMDS returned empty role name' on hosts
without an instance role. Gate ECR auth on .dkr.ecr. refs and pull all other
registries anonymously.
Bare image strings (nginx:alpine) were handed to containerd unmodified, which
parsed them as dummy://nginx:alpine and rejected the pull. Normalize refs to
their fully-qualified form (docker.io/library/nginx:alpine, :latest default)
before Pull and GetImage.
Adds StartTask/StopTask service-controller coverage.
* fix: address ECS 4f live-verification findings
ecs-node-bringup.sh defaulted GATEWAY_IP to the mgmt bridge (10.15.8.1), which
a VPC guest cannot route to — the agent could not reach the gateway. Default to
the host's WAN-bridge IP, which guests reach by NATing out through the WAN.
A target group not forwarded to by any listener reported its targets stuck in
'initial' forever (no health checker advances a detached TG). DescribeTargetHealth
now reports such targets as 'unused' with reason Target.NotInUse, matching AWS.
* feat(ecsgw): on-VM SigV4 client for the ECS agent→gateway channel
Adds internal/ecsgw, the twin of internal/eksgw: a TLS + SigV4 (service "ecs")
client the ecs-agent will use to reach the AWS gateway over HTTPS instead of
connecting to host NATS directly. Calls are AWS JSON 1.1 with an X-Amz-Target
action; the gateway relays onto ecs.* host-side, keeping NATS cluster-internal.
* feat(ecs): SubmitTaskStateChange gateway action → recordTaskState
Wires the AWS-API SubmitTaskStateChange action end to end (gateway action →
ecs.SubmitTaskStateChange NATS → daemon → Service.SubmitTaskStateChange) so a
gateway-routed agent reports task state without touching the NATS bus. The
handler maps the SDK input onto the same bus.TaskState shape and converges on
recordTaskState; the account is authoritative from the SigV4 caller, so an
instance cannot report state for another account's task.
* feat(ecs): durable assign KV inbox + PollAssignments gateway action
Scheduler writes task assigns into a per-instance KV inbox
(clusters/{cluster}/assignments/{instance}/{task}) instead of a
fire-and-forget core-NATS publish, and a new PollAssignments gateway
action lets a gateway-routed agent drain its inbox over TLS+SigV4.
At-least-once: an unacked assign is re-delivered on the next poll;
the agent acks accepted assigns (AckTaskIDs) and STOPPED transitions
reclaim stale inbox entries. The Poll response carries bus.Assign with
an RFC3339 time, so it is encoded with encoding/json (RawJSONActions)
rather than the jsonutil epoch-second marshaler.
* feat(ecs-agent): rewire control-plane off NATS onto the AWS gateway
The agent no longer connects to NATS. It registers, heartbeats, reports
task state, and receives assignments entirely through the AWS gateway
over TLS+SigV4, using seeded IAM credentials — keeping the NATS bus
host-internal and closing the trust-boundary gap (a tenant container
instance must not hold a shared bus token).
- controlPlane seam over ecsgw.Client replaces the NATS publisher:
Register (RegisterContainerInstance), SubmitTaskState
(SubmitTaskStateChange), PollAssignments.
- Heartbeat folds into a periodic idempotent re-register (refreshes
LastSeen); bus.Heartbeat is dropped from the agent.
- subscribeAssign (NATS subscribe) becomes a poll loop: drain the
inbox, dispatch each assign once, ack accepted taskIDs on the next
poll (at-least-once).
- config: ECS_ACCESS_KEY / ECS_SECRET_KEY / ECS_POLL_INTERVAL added,
ECS_NATS_URL removed; cloud-init initd seeds the IAM creds.
Tests move from the NATS publisher fake to a controlPlane fake.
* feat(ecs): cloud-init cred-seed + ecs-node bring-up script
ECS container instances are plain EC2 booted from the spinifex-ecs-node
AMI (AWS-faithful: no ECS-specific launch API). This renders the
cloud-init user-data that seeds /etc/spinifex-ecs/agent.env (gateway
URL/CA/region/cluster + IAM creds) and the gateway CA, then runs the
instance. The agent registers, heartbeats, polls assignments, and
reports task state entirely through the AWS gateway over TLS+SigV4 —
never NATS.
The guest dials the customer-facing gateway address (network advertise
IP), not the mgmt bridge, mirroring the EKS GatewayURL/AddonGatewayURL
split. Seeds the system key for now; scoped per-instance creds are a
follow-up (4g task-role endpoint).
* fix(ecs): reconcile dev merge — restore #418 ENI release + netns wiring
The 'Merge branch dev' resolution dropped code where Sprint 4d (#418,
awsvpc ENI-per-task) and the gateway control-plane rewrite (#419) touch
the same paths:
- recordTaskState STOPPED branch: reclaimTaskENI was clobbered by
reclaimAssignInbox; both are needed (release the task ENI AND drain the
KV inbox). Restores TestRecordTaskState_Awsvpc_ReleasesENIOnStopOnce.
- ecs-agent Agent: the netns field, its init, and the setupTaskNetns/
teardownTaskNetns wrappers were lost when the controlPlane Agent struct
won the merge, leaving taskrun.go calling undefined methods.
- eni_test assign assertion read the assign over NATS; #419 delivers it via
the durable KV inbox, so it now polls via PollAssignments.
- taskrun_test netns cases used the removed historyPub fake; switched to
fakeCP to match the controlPlane seam.
* feat(ecs): awsvpc ENI-per-task control plane (Sprint 4d)
Scheduler owns the task-ENI lifecycle for awsvpc mode: allocate from the
task subnet on placement, hot-plug onto the container instance, plumb ENI
identity through the assign payload + task record, and detach+delete on
STOPPED (graceful + reaper, idempotent leak guard). bridge/host unchanged.
* feat(ecs-agent): awsvpc guest-side task netns from hot-plugged ENI
The agent builds a per-task network namespace for awsvpc tasks: it finds the
hot-plugged ENI NIC by MAC, moves it into /var/run/netns/ecs-<taskID>, and DHCPs
the OVN-assigned ENI IP. The container then joins that netns via the OCI network
namespace path; bridge/host tasks keep the host (VM) netns as before.
* build(ecs-node): add iproute2 for awsvpc task netns
The ecs-agent builds awsvpc task network namespaces with `ip netns`, which
busybox's `ip` applet does not implement. Install iproute2 in the ecs-node AMI;
udhcpc (for the in-netns ENI DHCP lease) is already provided by busybox.
PR #411 (cloud-init EC2 datasource) dropped the cloud-init ISO that
created the ec2-user account and baked the harness SSH key; guests now
boot the stock Ubuntu gold AMI (default user ubuntu) bootstrapped via the
Ec2/IMDS datasource. eni_reconcile_test.go was the lone single-suite test
still SSHing as ec2-user, so TestENIHotplugReconcile failed publickey at
the first SSH. Align it with every other single-suite test.
* feat(eni): hot-plug reconciler + orphan sweep (Sprint 3d)
Add eniReconciler, a node-local vm.Reaper that converges ENI hot-plug
state across a daemon restart or crash: KV ENIRecord <-> in-memory PCIe
slot map <-> guest query-pci device list. Resolves records stuck in
attaching/detaching, re-adopts the slot map after restart, and reaps
orphan QMP devices and slot entries.
A staleness gate (new ENIRecord.AttachmentStateAt + 30s threshold) keeps
the reconciler off records a live attach/detach handler is mid-pipeline
on. Wired into the existing GarbageCollector backstop.
* test(eni): unit-cover hot-plug reconciler vm helpers
* test(eni): e2e reconciler convergence across daemon restart (Sprint 3e)
StopNode now stops spinifex-nats.service, so WaitNATSPeers polling that
node's dead 127.0.0.1:8222 monitor port fails with curl exit 7 and never
converges. Add WithSkipNodes to exclude the stopped node from the poll;
TestMultinodeNodeFailure skips node2.
Add the ECS v1 scheduler control loop and end-to-end RunTask flow:
- handlers/ecs: per-account KV-backed Service (clusters, task definitions,
container instances, tasks), bin-pack/spread/random placement with CAS
capacity reservation, and a leader-elected Scheduler that owns the Layer-2
bus subscriptions (register/heartbeat/task-state) plus a heartbeat reaper.
- gateway/ecs: wire CreateCluster, Describe/ListClusters,
Register/Describe/ListTaskDefinitions, Register/Describe/ListContainerInstances,
RunTask, Describe/ListTasks onto the daemon over NATS.
- daemon: construct the Service, start the Scheduler goroutine, subscribe the
ECS action subjects.
- ecs-agent: subscribe the per-instance assign subject, pull and run each
container via containerd, and report task state back on the bus.
Adds the spinifex-ecs-node AMI variant: alpine 3.21.7 + containerd + runc +
cni-plugins + ca-certificates with the ecs-agent binary baked to
/usr/local/bin and three OpenRC services wired in dependency order
(ecs-ca-install -> containerd -> ecs-agent).
- manifest.conf drives build-system-image.sh (no host mounts).
- ecs-ca-install.initd: first-boot oneshot installing the cloud-init
delivered gateway CA into the system trust; no-op when absent.
- ecs-agent.initd: sources /etc/spinifex-ecs/agent.env, backgrounds the
agent after net + containerd.
- 10-bridge.conflist: CNI bridge for task networking (Sprint 4d).
- setup.sh: exec bits, /opt/cni/bin symlink, content-store dirs, ttyS0
console reorder + fast boot menu.
- Makefile: build-ecs-node-image / import-ecs-node-image targets.
* feat(ecs): ecs-agent skeleton + containerd image pull (Sprint 4c)
Add the ecs-agent binary that runs inside an ECS container instance: it
resolves host identity from IMDS, registers on the Layer-2 NATS bus, and
heartbeats while alive. Image pulls go through a containerd-backed
ImagePuller behind an interface, with an ECR resolver that mints
GetAuthorizationToken creds from the instance role.
Extract the IMDS-credential and ECR-auth flow out of
ecr-credential-provider into internal/imdscreds and internal/ecrauth so
the agent and the kubelet provider share one implementation.
Add spinifex/handlers/ecs/bus as the single source of truth for Layer-2
subject builders and wire payloads, shared by the agent now and the
scheduler in Sprint 4e.
* build: pin genproto via replace to fix containerd/v2 ambiguous import
go.opencensus.io -> grpc@v1.33.2 drags the pre-split monolithic genproto
into the module graph; it still ships googleapis/rpc/status, which collides
with the split genproto/googleapis/rpc module that containerd/ttrpc needs.
The main module never imports the monolith, so a plain require bump is
dropped by 'go mod tidy'. Pin it with a replace to a post-split version so
the fix survives tidy and contributors can run 'go mod tidy' freely.
* fix(ecs): agent boot failsafes from runtime verify
- Build the ECR gateway HTTP client lazily on first image pull instead of
at boot. A missing/malformed gateway CA no longer kills the agent before
it can register + heartbeat, matching the best-effort containerd path.
- Actively probe containerd with IsServing in runtime.New. The client dials
lazily, so a dead socket previously slipped past the boot-time
'containerd unavailable' warn and only surfaced on the first pull.
- Make the heartbeat interval configurable via ECS_HEARTBEAT_INTERVAL
(default 30s) for test/verify and slow-link tuning.
* fix: drop removed predastore s3.Config.Compaction from admin_test
The predastore submodule bump removed the Compaction field from s3.Config.
Parse the rendered [compaction] block into a local struct so the template
round-trip test no longer references the deleted field.
* feat: graceful guest drain on systemctl stop / host shutdown
systemctl stop spinifex.target tore down nbdkit/viperblock while guest QEMU
was still writing, corrupting guest filesystems. The ordered drain existed
only over NATS via cluster shutdown, never on the systemd/host path.
Add spx admin node drain --local (GATE+DRAIN against the local node, leaving
STORAGE/PERSIST/INFRA to systemd), wired as the ExecStop of a new
spinifex-shutdown.service oneshot ordered After= all services so it stops
first and drains guests while every service is still up. DDIL preserved: the
oneshot fires only on target/host shutdown, not on a daemon restart.
Harden the viperblockd SIGTERM handler to flush the WAL but skip reaping an
nbdkit a guest is still attached to. Add a units ordering invariant.
* fix: ExecStop drain timeout needs duration unit (120 -> 120s)
The spinifex-shutdown.service ExecStop passed --timeout=120, but spx admin
node drain's --timeout is a cobra Duration flag and rejects a bare integer
(time: missing unit in duration "120"). The drain exited 1 without running,
so a systemctl stop tore nbdkit out from under the running guest and corrupted
its ext4 (the exact failure this feature prevents). Pass 120s.
* fix: nbdkitInUse read ss state from correct column + cover it
ss -H rows are '<netid> <state> ...', so the state is field[1], not field[0]
(field[0] is 'u_str'). The old index never matched ESTAB, so an in-use nbdkit
read as idle and the SIGTERM path would reap it under a live guest. Add tests
exercising a real connected unix socket, an absent socket, and TCP transport.
* feat: add predastore compaction E2E
Inject a short compaction interval into test-cluster predastore config via a
new --predastore-compaction-interval flag on admin init/join, gated so
production output stays byte-identical when unset. Add a harness helper that
drives write-then-delete churn against predastore S3, brackets it with
du-over-SSH usage, and asserts end-of-churn usage settles below 70% of the
churn peak; wire it into the multinode suite as TestPredastoreCompaction.
* fix: compaction E2E gate writes churn to its own bucket
The reclaim gate signed PutObject against the config-defined system
'predastore' bucket, which belongs to account 000000000000 with only a
static [[auth]] grant. The harness admin identity is a different IAM
account, so predastore returned 403 AccessDenied cross-account.
Create a per-run bucket the admin identity owns (AdministratorAccess)
and churn into it. Objects still erasure-code into distributed/nodes/
node-* so the du-based reclaim measurement is unchanged. The gateway
(:9999) does not proxy S3, so the client stays predastore-direct.
* test: run predastore compaction gate before destructive node tests
The compaction reclaim gate was declared last in the multinode suite, so it
measured shard-store usage after NodeFailure/NodeRecovery had torn down and
restarted node2. NodeRecovery only gates on NATS/gateway/EC2 readiness, not
predastore, so node2's shard store was still settling and the du baseline came
back empty. Move the test before the destructive node tests so it runs against
a pristine, fully-converged cluster primed by the volume tests.
* fix: race-free SSH output capture + empty-tolerant du probe
The compaction reclaim gate crashed with ParseInt("") when node1's du
probe returned empty combined output. Harden the probe and the SSH layer:
- ssh.Run used one bytes.Buffer for both session.Stdout and session.Stderr,
which x/crypto/ssh copies in two concurrent goroutines — a data race on a
non-concurrent-safe type. Switch to session.CombinedOutput, which guards the
combined stream with a mutex-wrapped singleWriter.
- shardStoreUsageBytes now terminates the remote command with a sentinel echo.
A missing sentinel is reported as output lost in SSH retrieval; a present
sentinel with no number is treated as a 0 measurement (anomalous, logged)
instead of crashing the gate.
The sentinel lets the next run distinguish a transport-layer drop from a
genuinely empty remote measurement.
* test(e2e): replace predastore du-reclaim gate with object lifecycle contract
The multinode compaction gate measured shard-store disk usage over SSH (du)
and asserted reclaim after churn. It could never pass: segments only seal at
the 4 GiB roll threshold, so test-scale churn produces no compactable segment,
and the du probe read 0 bytes before and after. Reclaim correctness is already
covered deterministically by predastore's store-level unit tests.
Delete AssertPredastoreCompaction and its du/churn/poll/sentinel machinery,
the harness unit tests that exercised them, and the multinode TestPredastore
Compaction entry. Add a single-node TestPredastoreObjectLifecycle that checks
the user-visible contract: an object is readable after PutObject, gone after
DeleteObject (GET returns NoSuchKey and ListObjectsV2 is empty), and the store
keeps serving fresh writes. newPredastoreS3 is retained, refactored to take a
host so it works outside the multinode cluster type.
* test(e2e): drop orphaned multinode TestPredastoreCompaction
Complete the du-reclaim -> object-lifecycle refactor: remove the
multinode TestPredastoreCompaction caller (and its now-unused context
and harness imports) that still referenced the deleted
harness.AssertPredastoreCompaction, which broke e2e-build. Wire the
single-node TestPredastoreObjectLifecycle entry that replaces it.
* docs: add Containers section with EKS and ECR service pages
New docs/containers/{eks,ecr} pages covering each service via AWS CLI, the Spinifex console, and Terraform, plus the requisite VPC/IGW/NAT/SG/IAM setup. EKS page embeds console screenshots. Cross-links the EKS Terraform workbooks.
* docs(eks): rework Terraform workbooks into a capability ladder
Rework the three EKS workbooks into a coherent ladder and publish them to
the Containers section. All tiers are single- or three-node capable via
node_desired_size and serve a shared Spinifex-themed demo app from ECR,
replacing nginxdemos/hello.
- demo-app: new std-lib Go web app (themed page reporting pod/node/cluster/
region + hit counter; optional /data persistence), distroless image.
- eks-quickstart: node_desired_size var, ECR repo + themed app over NodePort.
- eks-https-ingress (was eks-addons-access): HTTPS via the aws-load-balancer-
controller addon + ACM, public/private VPC with NAT, managed-ingress=false.
- eks-gitops-argocd (was eks-https-ingress): Argo CD addon syncs the app from
a git repo, aws-ebs-csi-driver PVC (Viperblock-backed) for state, access
entry RBAC, HTTPS Ingress carried over.
Pending live e2e verification (image build/push, apply, ALB + PVC bind).
* fix(ecr): implement ListTagsForResource (empty tags) for client parity
The AWS provider calls ecr:ListTagsForResource on every repository read, which
returned 501 NotImplementedException and broke 'tofu apply' of any workbook that
creates an aws_ecr_repository. Implement it to return an empty tag set (resource
tagging stays a deferred feature) so reads succeed. Drop the tags block from the
three EKS workbooks' ECR repos so the provider doesn't call the still-stubbed
TagResource on create.
* feat(eks): pull from internal ECR via kubelet credential provider
Worker nodes could not pull custom images from the internal Spinifex ECR
registry: the eks-node had no DNS for the registry host, no containerd CA
trust, no port rewrite to :9999, and no pull credentials. Only public
images worked.
Add cmd/ecr-credential-provider, a kubelet exec credential provider that
fetches the node IAM-role credentials from IMDS, calls GetAuthorizationToken
against the gateway ECR endpoint, and returns the minted AWS:<jwt> as Basic
auth for the registry host. Thread the gateway URL, account, region, and
internal suffix into the agent userdata, which now writes the gateway CA,
a registries.yaml (CA trust + :9999 endpoint), the kubelet
credential-provider config and config.yaml.d drop-in, and an /etc/hosts
entry mapping the registry host to the gateway IP. Ship the provider binary
in the eks-node AMI.
Also remove a stray paragraph from the admin-cli README.
* fix: dial EKS ECR registry mirror by gateway IP
cloud-init manage_etc_hosts rewrites /etc/hosts from the alpine template on
every boot, wiping the registry-host mapping appended via bootcmd, so worker
nodes could not resolve the internal ECR host and pulls failed with no such
host. Point the k3s registries.yaml mirror endpoint and tls config at the
gateway IP instead (the gateway cert carries an IP SAN), keeping the pod image
ref port-less and dropping the DNS leg until northstar lands. Fall back to the
hostname endpoint when the gateway is not an IP.
* fix: attach node-role instance profile to EKS workers for ECR
EKS workers launched with no IAM instance profile, so IMDS served no role and
the ECR credential provider could not mint a registry token, leaving internal
ECR pulls failing with 401. Real EKS implicitly creates an instance profile for
the node role; do the same here. Wire a KV-backed IAM service into the EKS deps
and, at worker launch, find-or-create an instance profile named for the node
role, attach the role, and pass it to RunInstances so IMDS exposes the role to
the credential provider.
* feat: use mulga logo in EKS demo app
* feat: add ACM certificate tag support (List/Add/RemoveTags)
Implements ListTagsForCertificate, AddTagsToCertificate and RemoveTagsFromCertificate across the store, service, NATS, daemon and gateway layers. Import now persists tags. Fixes terraform aws_acm_certificate InvalidActionException on ListTagsForCertificate.
* test: cover ACM tag gateway and NATS wrapper paths
* feat: support EKS nodegroup tags (Tag/Untag/ListTags)
Extends EKS tag ops to back nodegroup ARNs alongside cluster ARNs. Nodegroup create-time tags persist on the record and echo via DescribeNodegroup, giving the terraform-aws provider a clean default_tags round-trip. Fixes aws_eks_node_group TagResource 501 NotImplementedException.
* fix: provision EKS ALB ingress via node credentials
The aws-load-balancer-controller addon could not provision an ALB. Three fixes:
- TLS: the controller's per-service ec2/elbv2/acm SDK clients install their own
BaseEndpoint resolver and build an HTTP client that bypasses AWS_CA_BUNDLE,
falling back to the Go system trust pool and rejecting the gateway cert as
'unknown authority'. Add SSL_CERT_FILE so the gateway CA joins the system pool.
- Credentials: an empty AWS_ROLE_ARN beside the web-identity token file
half-arms the SDK web-identity provider, which then signs anonymously
(MissingAuthenticationToken). Inject the IRSA env only when a service-account
role ARN is staged; otherwise omit it so the default chain falls back to the
node instance-profile credentials served over IMDS.
- Authorization: the node role lacked the ec2/elbv2 actions the controller
needs. The eks-https-ingress workbook now attaches an AWSLoadBalancerController
managed policy to the node role (CreatePolicy + AttachRolePolicy).
* fix: add force_delete to EKS workbook demo ECR repos
tofu destroy of an EKS workbook left the demo ECR repository behind: it
still holds the pushed demo image, so DeleteRepository (force=false)
returns RepositoryNotEmptyException. Add force_delete = true to each
workbook's aws_ecr_repository.demo so teardown removes the repo with its
images.
* fix: force-delete EKS control-plane ENIs on teardown
TerminateK3sServerVM deleted the CP ENI with a plain force=false
DeleteNetworkInterface. eniIsLiveAttachment reports InUse whenever the
record's Status/InstanceId/AttachmentId are set, so when the K3s VM is
gone but the ENI record's attachment fields were never cleared the delete
returned InvalidNetworkInterface.InUse permanently and EKSDeletingReaper
re-drove the same non-idempotent call every 90s, wedging the cluster in
DELETING.
Teardown owns its CP ENIs and has already terminated the VM, so the
attachment is authoritatively dead. Replace the InUse-retry budget loop
with detach-then-delete (detachAndDeleteServerENI): DetachENI clears the
stale attachment fields, then DeleteNetworkInterface passes the in-use
guard. Both calls tolerate an already-gone ENI (NotFound) so a race with
the async instance-terminate cascade resolves to idempotent success.
purgeClusterInfra's private-endpoint ENI path reuses the same helper.
* fix: detach+delete worker ENIs on EKS nodegroup teardown
A terminated nodegroup worker could leave its primary ENI in
spinifex-vpc-enis as in-use/attached, and a dangling ENI throws
DependencyViolation on subnet/VPC delete, blocking tofu destroy. Two
gaps:
TerminateWorkerInstances did a local-only vmMgr.Get and silently skipped
any worker owned by another node, stranding both the VM and its ENI on
multi-node clusters. Route the terminate per-instance via ec2.cmd.<id>
(mirroring the gateway TerminateInstances path) so the owning daemon runs
the full teardown — detach + force-delete the primary ENI and clear its
record. No responder / NotFound payload is treated as already-gone so a
retried DeleteNodegroup stays idempotent.
deleteInstanceENI (stopped-instance terminate) issued a force=false
DeleteNetworkInterface with no prior detach: a stale in-use attachment
made it fail InvalidNetworkInterface.InUse, leaving the ENI record behind
after the instance was already written to the terminated bucket. Detach
first, then delete, tolerating NotFound.
* fix: heal volume State drift on detach/terminate teardown
A detach/terminate could leave a volume's State empty instead of
'available'. DeleteVolume gated on State=='available' exactly, so it
returned VolumeInUse for an unattached volume nothing was using, leaving
it undeletable and blocking stack teardown. DescribeVolumes cosmetically
rendered an empty State as 'available', masking the drift.
- DeleteVolume now treats an empty State with no attachment as deletable
(an attached volume still returns VolumeInUse).
- DescribeVolumes derives an empty State from the attachment (attached ->
in-use, unattached -> available) instead of blindly rendering
available, so an empty-but-attached volume is no longer hidden.
- UpdateVolumeState normalizes an unattached volume with an empty state to
available, so a detach writeback that omits the state cannot strand the
volume in drift.
* fix: break all SG cross-references on EKS teardown
DeleteClusterSGs broke the cp<->nodegroup cycle too narrowly. It revoked
only ingress (IpPermissions), never egress (IpPermissionsEgress), so a
cross-reference via an egress rule kept the SG pinned on
DependencyViolation. It also resolved only the two named cluster SGs, so
any OTHER SG in the VPC referencing them — the default SG, the LBC/ALB SG,
the private-endpoint SG, user/TF SGs — was never revoked and pinned the
referenced SG on delete, leaving the VPC stuck.
Enumerate every non-default SG in the VPC and revoke BOTH ingress and
egress on each before deleting, breaking any cycle order-independently,
then delete only the cluster-owned SGs (referrer SGs are deleted by their
own teardown). The default SG is left untouched. Since DeleteClusterSGs
runs against both the customer VPC and the managed control-plane VPC, the
broadened sweep covers both teardown paths.
* feat: support TargetGroup ProtocolVersion in elbv2
CreateTargetGroup now stores ProtocolVersion (defaulting HTTP/HTTPS target
groups to HTTP1) and DescribeTargetGroups/Create output return it. Without
this the aws-load-balancer-controller sees a perpetual immutable-field
mismatch (desired HTTP1 vs empty) and recreates the target group every
reconcile, looping on DuplicateTargetGroupName once a shared ALB IngressGroup
reconciles more than one target group. NLB protocols (TCP/UDP/TLS) keep it
omitted to match AWS.
* fix: omit IRSA env from ebs-csi addon when no role staged
The ebs-csi controller manifest hardcoded AWS_ROLE_ARN={{SERVICE_ACCOUNT_ROLE_ARN}}
plus the web-identity token volume. With no service-account role staged the
placeholder rendered empty, half-arming the SDK web-identity provider, which
panics 'role ARN is not set' and CrashLoops the controller, leaving PVCs
Pending. Convert to the conditional {{IRSA_ENV}}/{{IRSA_VOLUME_MOUNT}}/{{IRSA_VOLUME}}
markers like the LBC bundle so the IRSA env and token volume are omitted when
no role is staged, falling back to node instance-profile credentials over IMDS.
The EBS-specific AWS_EC2_ENDPOINT/AWS_ENDPOINT_URL_EC2 stay in both modes.
Requires an eks-node AMI rebake to take effect.
* docs: route gitops-argocd workbook through a shared ALB and expose ArgoCD UI
Fold the demo app and ArgoCD ingresses onto one ALB via an LBC IngressGroup
with host routing (app./argocd. on a wildcard cert) instead of one ALB per
ingress, add a NodePort + HTTPS Ingress for the ArgoCD UI, and document the
access steps. Add the node-role LBC and EBS-CSI IAM policies the controllers
need (Spinifex wires controller creds at the node level, not IRSA) and an
ignore_changes block for the cluster's access_config and
vpc_config security_group_ids, which Spinifex DescribeCluster does not echo
back, so the provider stops trying to replace the cluster on every plan.
* fix: stop addon-sync substituting IRSA markers named in comments
The ebs-csi controller manifest's header comment spelled out the literal
{{IRSA_ENV}} / {{IRSA_VOLUME_MOUNT}} / {{IRSA_VOLUME}} marker tokens. addon-sync
matched each marker by substring, so the comment lines were replaced with the
env/mount blocks, injecting a bare YAML sequence between two documents. K3s'
auto-deploy then rejected the whole file (ParseManifestFailed: cannot unmarshal
array into unstructured), so the ebs-csi Addon CR never applied and the EKS
addon hung in CREATING. Reword the comment to not contain the literal tokens,
and anchor the marker substitution to lines that are exactly the marker
(whitespace aside) so a future comment mentioning a marker can't retrigger it.
* fix: disable K3s local-storage on EKS clusters in parity mode
In AWS LB Controller parity mode (BuiltinIngress=false) the K3s server
disabled traefik and servicelb but left local-storage enabled. Its local-path
provisioner ships a StorageClass marked default, which races the EBS CSI
ebs-gp3 default StorageClass — two defaults, so dynamic PVCs bind
nondeterministically. EKS has no local-path, so disable local-storage too and
let ebs-gp3 be the sole default. The built-in-ingress path is unchanged.
* fix: re-encrypt HTTPS target groups in haproxy ALB backends
An ALB target group with backend-protocol HTTPS (e.g. argocd-server, which
serves TLS on its own port) rendered a plaintext haproxy server line. HAProxy's
own check then spoke HTTP to a TLS port, failed, and marked the server DOWN,
returning 503 — even though the lb-agent's separate HTTPS health probe reported
the target healthy, so describe-target-health masked the fault.
Emit check-ssl ssl verify none on the server line for HTTPS target groups so
both the check and proxied traffic re-encrypt to the backend.
* fix: reconnect wedged QMP client on next command
A timed-out QMP decode under host contention left the cached client's
shared json.Decoder mid-message, wedging every later command and breaking
EBS AttachVolume/DetachVolume for the VM's lifetime. Mark the client Dead
on encode/decode error and redial the retained socket path (re-running the
qmp_capabilities handshake in place) on the next send.
* fix: round-trip security-group rule descriptions
SG ingress/egress rules dropped the per-rule Description on store and
describe. The AWS Load Balancer Controller tags the worker-node SG rules it
manages with a UserIdGroupPair description and identifies them by that value
to revoke them on teardown. Without the round-trip the controller cannot
match its rules, never revokes them, and the backend security group stays
referenced — DeleteSecurityGroup returns DependencyViolation forever,
hanging the ingress finalizer and tofu destroy with orphaned security groups.
Persist Description on SGRule (excluded from sgRuleKey so it is metadata, not
identity) and render it back through describe.
* fix: reclaim customer-VPC nodegroup SGs on managed-CP-VPC teardown
In the managed control-plane VPC topology, launchNodegroupInfra creates the
cluster control-plane and nodegroup security groups in the customer VPC for
worker<->control-plane networking, but purgeClusterInfra only reclaimed the
security groups in the managed CP VPC. The customer-VPC SGs orphaned,
cross-referenced each other, and pinned the customer VPC with
DependencyViolation, hanging tofu destroy of the VPC. Reclaim them in the
managed-CP-VPC branch too, matching the legacy customer-VPC branch.
* fix: reclaim ALB EIP when system VM already gone on terminate
TerminateSystemInstance only released an internet-facing ALB's Elastic IP on
the local path, where the VM is still present in the manager. When the VM was
already gone (health-reaped, crashed, or raced by ENI teardown), it took the
remote/no-owner path and returned idempotent success without reconciling the
EIP, leaking the wan-pool address still associated to a deleted ENI.
Add EIPServiceImpl.ReleaseAddressByInstanceID, an idempotent account-agnostic
scan that disassociates then releases any EIP recorded against an instance, and
call it as a backstop in TerminateSystemInstance for both branches.
* fix: cascade-prune demo PVC via Argo CD finalizer in eks-gitops workbook
The demo Application carried no resources finalizer, so tofu destroy removed
the Application CR without pruning the synced PVC. The PVC survived, the cluster
was destroyed, and the Delete-reclaim PV's EBS volume orphaned. Add the
resources-finalizer.argocd.argoproj.io finalizer so deleting the Application
cascades to the PVC and ebs-csi reaps the volume before cluster teardown.
* feat: raw-IP access + documented Argo CD creds in eks-gitops workbook
The shared ALB host-routed everything, so the raw ALB IP 404'd in a browser
without an /etc/hosts entry. Make the app the hostless catch-all on :443 and add
a hostless Argo CD listener on :8443, so https://<alb-ip>/ serves the app and
https://<alb-ip>:8443/ serves the UI with no DNS. Surface the admin-password
command as a tofu output and document the credential + IP URLs in the README.
* fix: reclaim ALB EIP on the owning-node terminate path
The EIP backstop lived only in TerminateSystemInstance, but the ALB
delete path goes serveSystemTerminateInstance -> terminateSystemInstanceLocal
directly, bypassing it, and releaseSystemInstanceEIP relied on the VM's
cached allocation which is often empty. Extract reclaimSystemInstanceEIP
(authoritative KV reconcile via ReleaseAddressByInstanceID) and call it
from terminateSystemInstanceLocal as well as the no-owner backstop, so an
internet-facing ALB's EIP is released on every terminate route.
* chore(eks): drop coredns from the managed addon catalog
Cluster DNS ships as the baked node-local coredns-mulga DaemonSet, not as
a managed addon — there is no bundled coredns manifest, so CreateAddon
coredns would stage but fail VM-side render. Remove the phantom catalog
entry so DescribeAddonVersions and CreateAddon no longer advertise it;
repoint the addon tests that used coredns as their sample to argocd.
* feat(ui): group EKS and ECR under a Containers nav section
Fold the separate EKS and ECR sidebar groups into one Containers group
with EKS (clusters) and ECR (repositories) entries.
* fix(ui): satisfy strict null checks in ECR component tests
Index mock.calls via optional chaining and narrow the getAllByRole result
before clicking, clearing the TS2532/TS2345 errors that failed pnpm fix.
* refactor(eks): hide internal fixture addons from the public catalog
Add an AddonSpec.Hidden flag and mark spinifex-noop hidden: it stays
creatable (lookupAddon finds it, so the delivery-transport e2e still drives
it) but no longer surfaces in the unfiltered DescribeAddonVersions listing.
* refactor(eks): remove K3s built-in ingress; LBC-only
Drop the BuiltinIngress option and its managed-ingress tag opt-in. traefik,
servicelb, and local-storage are now always disabled on EKS nodes (AWS
parity — Service type=LoadBalancer and Ingress are the Load Balancer
Controller's job). Removes the EKS_DEFER_TRAEFIK userdata flag, the k3s.initd
traefik .skip staging, and the maybe_enable_traefik deferred-enable loop in
the state reporter.
* fix(ui): clear lint debt in VPC-wizard mutations and CIDR validation
Parallelize the independent subnet-create and route-table-association loops
in the create-VPC mutation with Promise.all (each task records its resource
in the rollback list before any further call), and switch the CIDR-validation
regex to named capture groups. Clears the no-await-in-loop and
prefer-named-capture-group lint errors.
* fix: carry gateway port in ECR registry endpoints; rename mulga.internal -> spinifex.internal
The ECR registry is served on the gateway port (:9999), but proxyEndpoint and
repositoryUri were emitted port-less, so docker login/tag/push dialed :443 and
failed. Derive the port from the awsgw listen host and append it to the registry
host (omitted for 443). Also rename the default internal DNS suffix.
* docs: drop verbose ECR :9999 host note
* docs: fix literal-asterisk emphasis (single-* not rendered)
* docs: brace-wrap ${REGISTRY}:latest (zsh :l modifier ate the tag); document Docker CA trust
* feat: default ECR registry URIs to the gateway host (no-DNS interim)
The registry authenticates by token, not hostname, so the per-account
<acct>.dkr.ecr name is cosmetic. Advertise the gateway's reachable bind host:port
(already a cert SAN) in repositoryUri/proxyEndpoint so docker login/tag/push need
no DNS; the parity host still works server-side where DNS exists.
* fix: serve ECR registry on a DNS-free gateway host end to end
Advertise the gateway's AdvertiseIP as the ECR registry host when the AWSGW
bind is unspecified (0.0.0.0/::), so repository_url resolves to a reachable
host:9999 carried in the server cert SANs rather than the unresolvable
per-account parity name. Docker push then dials it directly with no DNS.
Mirror the same host on EKS workers: the registries.yaml mirror keys and the
credential provider matchImages now cover the port-less parity name, the
port-ful parity name, and the gateway IP:9999, all redirecting to the dialed
gateway endpoint. A port-ful pull from the API-returned repository_url matches
the mirror instead of bypassing it into a failed DNS lookup.
Net: docker push host, pod image host, and the API's repository_url are the
same value, with no DNS or /etc/hosts.
* docs: resolve eks-https Ingress ALB DNS name to its public IP (no DNS)
* docs: resolve eks-gitops Ingress ALB DNS name to its public IP (no DNS)
* docs: remind to run aws eks update-kubeconfig before kubectl in EKS workbooks
* feat(ecr): return OperationNotSupportedException for image-scanning actions
The ECR image-scan control-plane actions resolved to the generic
NotImplemented 501 stub, indistinguishable from unbuilt features and
surfaced as a server fault. mulga runs no vulnerability scanner, so the
scan surface now returns OperationNotSupportedException (400):
- awserrors: add OperationNotSupportedException (400) ECR code.
- ecrapi: ScanningNotSupported handler; repoint the four image-scan
actions and register the three registry-scanning actions at it.
- spinifexui: Scan tab on the repository detail page renders a static
'scanning not supported' panel rather than a request that errors.
* refactor(config): default internal DNS suffix to spinifex.internal
Rename the AWS-parity internal DNS suffix from mulga.internal to
spinifex.internal across the config default, cert SAN builder, ECR
endpoint/audience construction, and their tests. Single runtime knob is
config.DefaultAWSInternalSuffix; no DNS backend exists yet, so this is a
naming change with no functional impact.
The ECR auth bridge minted ES256 tokens from a key that was created once and
never rotated. Add a background rotator (ecr-v1 Q3): mint a fresh key every 30
days, retain the previous keys for 24h so in-flight 12h tokens stay verifiable,
then prune. The active key is now the newest by KV creation time (not the
lexically-greatest kid) so a minted key takes over and all nodes converge. The
Issuer and Verifier became hot-swappable so the rotator updates the live
gateway as keys roll.
* feat: bake aws-ebs-csi-driver addon + mdev by-id bridge
Add the aws-ebs-csi-driver entry (v1.40.1) to the eks addon catalog and
ship its baked manifests (RBAC, IRSA-wired controller, node DaemonSet,
default gp3 StorageClass) under the eks-node image.
The node plugin locates EBS volumes via /dev/disk/by-id, which busybox
mdev does not populate. Add mulga-ebs-byid: an mdev rule + helper that
symlinks nvme-Amazon_Elastic_Block_Store_<serial> from the virtio-blk
serial, plus a boot oneshot that points the kernel hotplug helper at
mdev and coldplugs volumes present at boot.
* test: e2e EBS CSI volume — PVC bind + mount + reschedule survival
Add a TestEKS/EBSCSIVolume subtest that installs the aws-ebs-csi-driver
managed addon bound to an IRSA role, provisions a default gp3 PVC + pod,
and asserts a marker written by the first pod survives a reschedule onto a
fresh pod backed by the same PVC. Exercises the full data path:
CreateVolume(Viperblock)/AttachVolume virtio-blk hotplug with serial set to
the volume-id, mdev /dev/disk/by-id symlink, and ext4 mount.
* feat: eks-node provider-id + topology/instance-type labels for EBS CSI
mulga-eks-provider-id boot oneshot writes a K3s config.yaml.d drop-in
from IMDS before K3s starts: kubelet provider-id plus the region, zone
and instance-type node labels. The aws-ebs-csi-driver node plugin reads
this metadata for NodeGetInfo (instance id from providerID, region/zone
from topology labels, instance type for the volume-limit family lookup).
k3s runs no AWS cloud-provider so none of it is set otherwise, which left
the node plugin's metadata init failing, NodeGetInfo panicking in
IsNitroInstanceType, and the CSINode carrying no topology key. Wire the
service into the image manifest (INSTALL_FILES + ENABLE_SERVICES) and
chmod it in setup.
* test: e2e EBS CSI controller IRSA policy + failure diagnostics
awsgw enforces IAM on assumed-role EC2 calls, so the controller's
CreateVolume/AttachVolume need a permission policy on the IRSA role.
Attach a customer-managed EC2 policy (AWS-managed ARNs are opaque and
grant nothing) covering the EBS CSI action set; CreateVolume then
succeeds. Add an OnFailure dump of the CSI control path (provisioner and
plugin logs, pvc/pod describe, events, csinode) to the EBSCSIVolume
subtest so provisioning/attach failures are diagnosable post-teardown.
* feat: EBS CSI obeys control-plane taint; e2e worker nodegroup
Drop the CriticalAddonsOnly/NoExecute tolerations from the EBS CSI
controller + node manifests so the driver runs only on customer worker
nodes, never the system control-plane VM. The e2e creates a 1-node
worker nodegroup and the writer/reader pods drop their CP tolerations so
the customer-owned volume attaches in the customer space.
* fix: EBS CSI attach — decouple hotplug PCIe port from device name; iothread rollback
virtio-blk-pci hot-plug bus was derived from the AWS device-name letter
(/dev/sd[f-p]); the EBS CSI driver sends arbitrary names (/dev/xvdaa) that
map to no slot, so device_add fell back to pcie.0 — which rejects hot-plug —
and the attach hung. A failed attach also left the per-volume iothread object
behind, so every retry died on object-add 'duplicate property'.
nextHotplugEBSPort allocates the lowest free hotplug-ebs port from live
query-block state, independent of the device name. delIothreadBestEffort
removes the iothread on every attach rollback path.
* fix: EBS CSI node discovery — un-shadow mdev by-id rule, mint nvme link
Alpine eks-node runs busybox mdev (no eudev), which stops at the first
matching /etc/mdev.conf rule. setup.sh appended the mulga-ebs-byid rule after
the stock 'vd[a-z].* -> persistent-storage' line, so it never fired on
hot-attach: only /dev/disk/by-id/virtio-<serial> was created, while the
upstream aws-ebs-csi-driver resolves nvme-Amazon_Elastic_Block_Store_<volid>.
Swap the stock vd* rule in place to route through mulga-ebs-byid, which now
delegates to persistent-storage (preserving stock links) before minting the
nvme-style link the node plugin expects.
* test: EKS e2e — NAT Gateway egress for private worker nodegroup
The customer-VPC worker is private; AttachIGW programs no SNAT (mirrors AWS:
an IGW serves only public-IP instances), so an IGW-only fixture left the worker
unable to reach the public NLB endpoint or pull public CSI images — the
nodegroup never joined.
createWorkerEgress now builds the real topology: an IGW-fronted public subnet
hosting a NAT Gateway, with the worker subnet default-routed to it. Associating
the worker route table triggers the subnet-wide SNAT egress reroute. Teardown
drains the NAT Gateway before releasing the public subnet + EIP it pins.
The OCI /v2 registry auto-created a repository on first push (ensureRepo).
AWS ECR does not: a push or PutImage to a repository that was never created
with CreateRepository is rejected. Replace ensureRepo with requireRepo, which
returns OCI NAME_UNKNOWN on the data plane and RepositoryNotFoundException on
the JSON PutImage path. Enforcement sits at the single push choke point
(startUpload + StoreManifest).
* test(e2e): add ECR end-to-end suite (control plane + docker/crane/skopeo)
New tests/e2e/ecr/ suite plus harness support:
- harness: ECR SDK client, control-plane helpers (repo CRUD, get-login-password,
describe-images), and docker/crane/skopeo CLI wrappers that skip when the
binary or registry-host resolution is absent.
- suite: control-plane CRUD + policy + lifecycle round-trip; docker login/push/
pull; crane append/pull and skopeo copy exercising the Bearer /v2/token flow.
- registered e2e-ecr in the Makefile + service manifest.
Predastore capacity/load leg is a skipped placeholder pending mulga-siv-359/356.
* test(e2e): carry awsgw port in ECR registry host for dev clusters
Dev clusters bind awsgw on a non-standard port (e.g. 9999); production uses
:443. ECRRegistryHost now appends SPINIFEX_ECR_PORT when set so docker/crane/
skopeo address the right endpoint, and RequireRegistryResolves strips the port
before DNS lookup.
getManifest read only the first Accept header line via r.Header.Get, so
clients that send each acceptable manifest media type as a separate Accept
header (skopeo) never matched the stored type and got 406 MANIFEST_INVALID.
Join all Accept header values before negotiation.
Add the Docker Registry v2 token endpoint (GET /v2/token) and switch the
/v2 401 challenge from Basic to a Bearer realm pointing at it. The endpoint
verifies the caller's existing AWS:<jwt> credential and re-mints a fresh
full-TTL token, providing the token-refresh path for OCI clients (docker,
crane, skopeo). The auth bridge still accepts Basic AWS:<jwt> directly.
Stamp the dash-stripped volume-id (vol+17hex = 20 bytes, the virtio-blk
serial limit) as the hotplug disk serial so the EBS CSI node plugin can
locate the device via /dev/disk/by-id and match lsblk SERIAL.
* feat: ECR background lifecycle expiry sweep
Add a gateway-side periodic sweep (LifecycleSweeper) that applies each repo's
stored lifecycle policy: it evaluates the shared rule engine against the repo's
current images and deletes the expired set through the existing registry GC
reclaim path (DeleteImage -> reclaimManifest). It runs in the gateway because
only the gateway holds a Registry (object store + meta); the daemon owns the KV
but cannot reclaim predastore bytes. Launched on the STS janitor's lifetime
context in awsgw, enumerating ACTIVE IAM accounts.
Per repo: skip when no policy; evaluate; then before deleting, re-read the
current tag set and skip any image whose tags changed since evaluation
(optimistic-concurrency guard against concurrent push/re-tag). Cross-repo blob
mount safety is provided by reclaimManifest/referencedDigests. Per-item errors
are logged and skipped so one bad repo never aborts the cycle.
* test: cover lifecycle sweeper Run loop and active-account filter
Add tests for LifecycleSweeper.Run ticker loop, NewLifecycleSweeper
interval fallback, and sweepRepo error branches. Extract accountLister
interface so activeAccountIDs is unit-testable. Raises diff coverage
on the sweep changes from 53.4% to 76.9%.
* feat: ECR lifecycle policies (CRUD + evaluation engine + preview)
Implement PutLifecyclePolicy/GetLifecyclePolicy/DeleteLifecyclePolicy and
StartLifecyclePolicyPreview/GetLifecyclePolicyPreview, replacing the
NotImplemented stubs. A pure rule-evaluation engine parses the AWS
lifecycle-policy JSON and, given a repo's images, returns the ordered set that
would expire; each image is claimed by at most one rule (lowest rulePriority
first). Preview is served synchronously and deletes nothing.
Lifecycle-policy documents persist as a per-repo KV doc (repos/<name>/lifecycle)
over the ecr.lifecycle.* NATS subjects, mirroring the repo-policy storage stack,
and cascade on DeleteRepository. The console gains a Lifecycle tab with a policy
editor and a preview table over the saved policy.
The background expiry sweep that applies the engine and deletes images via GC is
deferred to mulga-siv-387.
* feat: client-side JSON validation in ECR policy editors
Swap the raw Textarea in the repository-policy and lifecycle-policy editors for
the shared JsonEditor (blur-time pretty-print) and gate Save on isValidJson,
showing an inline error for malformed JSON — matching the IAM/EKS editors. The
backend remains the source of truth for schema validation.
Honor, persist and echo imageTagMutability on CreateRepository and
DescribeRepositories instead of the hardcoded MUTABLE, and implement
PutImageTagMutability to flip a repository between MUTABLE and IMMUTABLE.
Enforce immutability at the shared StoreManifest chokepoint: on an
IMMUTABLE repo, re-tagging an existing tag onto a different digest is
rejected before any write (OCI 409 TAG_IMMUTABLE; JSON PutImage maps to
ImageTagAlreadyExistsException). Same-digest re-pushes, new tags and
digest references are still allowed.
Console: migrate the repository detail page to Summary/Images/Permissions
tabs and add a tag-immutability toggle on the Summary tab.
DELETE /v2/{name}/manifests/{digest} was a 202 no-op. Now delete-by-digest
removes the manifest record, deletes the manifest object, and sweeps the
account-wide blob pool: each child blob no longer referenced by any live
manifest is reclaimed, while shared blobs survive.
- DeleteImage (digest path) captures ChildDigests, deletes the manifest object,
then reclaimManifest() drops orphaned blobs via referencedDigests() (a
mark-and-sweep over the live manifest set across the account's repos).
- OCI deleteManifest by-digest wired to DeleteImage (real work + 202; 404 on
unknown). Tag delete stays KV-only so untagged images persist (AWS-faithful).
- Predastore reclaim is best-effort: a failure leaks bytes but never undoes the
logical delete.
Serve the five image-management JSON 1.1 actions inline in ECR_Request,
reusing the gateway-side OCI Registry for manifest bytes:
- ListImages / DescribeImages enumerate images by tag + digest (KV meta).
- BatchGetImage returns verbatim manifest bytes with Q14 partial-failure
semantics (cap 100, digest wins over tag, 200 + failures array).
- PutImage stores a manifest + tag via a shared StoreManifest core that the
OCI manifest PUT now delegates to.
- BatchDeleteImage removes tag pointers / manifest meta (blob GC deferred).
Adds DeleteManifestMeta across the MetaStore relay chain and ECR image
error codes.
Add the two missing repository-lifecycle actions, served inline in
ECR_Request (like DescribeRepositories) since both return the repository
ARN/URI built from the gateway region + internal suffix.
CreateRepository validates the name, guards cross-account registryId,
rejects an existing repo (RepositoryAlreadyExistsException), and writes
the per-account KV meta (predastore bucket stays lazy on first push).
DeleteRepository rejects a non-empty repo without force
(RepositoryNotEmptyException) and cascades the KV records (meta, policy,
tags, manifests). Predastore blob GC is deferred to a separate pass.
Adds MetaStore DeleteRepo + ListManifests with the matching daemon-relayed
subjects, service methods, and NATS client, plus two new awserrors codes.
The per-run ARTIFACT_DIR lives in /tmp, outside GITHUB_WORKSPACE, so a
re-run on the same self-hosted runner reused a prior attempt's append-only
suite-status.txt. The 'Fail job if any suite failed' gate's -s check then
failed the job even when the surviving attempt passed with rc=0.
Wipe ARTIFACT_DIR in Clean Workspace and gate solely on the per-attempt
steps.e2e.outputs.rc (fail-closed: an empty rc still fails).
CreateTags/DeleteTags wrote only the standalone tag store, which
DescribeImages does not read (it reads AMI metadata), so tag mutations on
an AMI were invisible — DeleteTags appeared to be a silent no-op. Mirror
tag add/remove into the owning AMI config (ownership-checked; absent or
foreign AMIs skipped) so DescribeImages reflects them, matching the
existing subnet/vpc mirror.
* feat(ecr): auth bridge + GetAuthorizationToken JWT (Sprint 2c)
Mint a self-contained ES256 JWT from GetAuthorizationToken and verify it on
the /v2 OCI registry via a new ECRAuthBridgeMiddleware (Bearer | Basic AWS:jwt).
Signing keys live in the cluster-replicated awsgw-keys KV bucket under
jwt-signing/{kid}, encrypted with the IAM master key.
The registry is now scoped per-request to the token's account (replacing the
fixed bootstrap account); bucket-ensure is cached per account. Cross-account
token/host mismatch is rejected 403; unauthenticated /v2 access gets a 401
Bearer challenge.
* fix(ecr): use Basic auth challenge on /v2 (not Bearer token-endpoint flow)
ECR authenticates the OCI registry with Basic AWS:<token>; a Bearer realm
made docker chase an unimplemented /v2/token endpoint and fail login.
* feat(ui): inline EKS cluster IAM role creation in wizard
* feat(ui): show trust policy + attached managed policy in EKS role dialog
Render the trust policy with PolicyDocumentViewer (pretty-printed, safe
fallback) instead of a raw read-only textarea, matching the IAM detail
pages. Surface the attached AmazonEKSClusterPolicy ARN so users can see
exactly which permissions the cluster role grants.
* fix(ec2): stamp subnet AvailabilityZoneId for single-AZ ALB resolution
DescribeSubnets omitted AvailabilityZoneId, so the AWS Load Balancer
Controller's DescribeAvailabilityZones-by-ZoneId lookup resolved no AZ
info ("cannot resolve AZ info for AZs: []") and never built the ALB
model. Stamp the single zone-id on every subnet and share it with
DescribeAvailabilityZones through one SingleZoneID constant so the two
responses can never drift.
* feat(eks): add aws-load-balancer-controller v2.8.1 addon bundle
Baked manifests for the LBC managed addon (CRDs, controller Deployment/
RBAC/webhooks/Service, IngressClass alb, IRSA render markers). Two
wirings make it work against the spinifex gateway:
- --aws-api-endpoints carries sts= alongside elasticloadbalancing/ec2/
acm. The aws-sdk-go v1 credential bootstrap resolves the STS client
from the session endpoint resolver, not AWS_ENDPOINT_URL_STS, so the
web-identity AssumeRoleWithWebIdentity must be pointed at the gateway
here or it falls back to real AWS STS.
- --feature-gates=ALBSingleSubnet=true drops the ALB minimum subnet
count from two to one so an ALB provisions in a single-AZ deployment.
* feat(eks): render IRSA into managed addons + node-minted webhook cert
Extend the addon-sync renderer to inject the projected-token / role-arn /
gateway STS-endpoint + CA block into managed-addon pod specs via opt-in
markers, and substitute cluster-name/region/vpc-id/gateway-endpoint/CA
placeholders. Add cmd/eks-webhook-cert: the eks-node AMI has no openssl,
so this mints (then reuses) a self-signed ECDSA-P256 serving cert for the
admission webhook, cached on disk so the render is byte-stable across
ticks. Wire both into the image build (manifest.conf) plus INSTALL_FILES
for the LBC bundle.
* feat(eks): surface cluster VPC id as EKS_VPC_ID for the LB controller
Thread the control-plane VPC id through K3sServerInput.VpcID into the
first-boot EKS_VPC_ID env so the in-cluster aws-load-balancer-controller
can pass --aws-vpc-id to the gateway elbv2/ec2 handlers.
* fix(eks): point the LB controller at the customer VPC + customer-facing gateway
The aws-load-balancer-controller ran with two control-plane-scoped values
that customer worker pods cannot use:
- --aws-vpc-id was set from the managed control-plane VPC (cpRefs.VpcID),
which the gateway's DescribeVpcs does not return for customer calls, so
backend-SG auto-create failed with InvalidVpcID.NotFound. Source it from
meta.ResourcesVpcConfig.VpcId, the customer cluster VPC where the workers
and their subnets live and where the ALB is provisioned.
- The addon render baked the mgmt gateway endpoint (EKS_GATEWAY_URL /
SystemGatewayURL) into pod specs, unreachable from worker pods. Add a
distinct EKS_ADDON_GATEWAY_URL (the customer-facing GatewayBaseURL, same
host as the OIDC issuer) and substitute it for {{GATEWAY_ENDPOINT}} in
managed addon manifests. EKS_GATEWAY_URL stays mgmt for the CP's own
bootstrap, state-report and token-webhook paths.
* fix(elbv2): resolve forward target group from ForwardConfig
listenerActionFromSDK read only the legacy flat Action.TargetGroupArn, so a
forward action carrying its target group via ForwardConfig.TargetGroups (the
shape the aws-load-balancer-controller emits) left the action target empty and
CreateRule/CreateListener rejected it with MissingParameter. Flatten the first
target group from ForwardConfig when the flat field is absent.
* fix(ec2): mirror CreateTags/DeleteTags into subnet/vpc records
The generic S3 tag store and the per-resource KV records are disjoint:
DescribeSubnets/DescribeVpcs read tags from their KV record, while
CreateTags only wrote the generic store. Subnet tags applied after
create were therefore invisible to tag-filtered describes, breaking
aws-load-balancer-controller subnet auto-discovery (kubernetes.io/role/elb).
Write through CreateTags/DeleteTags to the owning subnet/vpc record so
tag-filtered describes observe them. Resource types the VPC service does
not own are skipped; the generic store stays their record of truth.
* test(daemon): pin synthetic capacity in host-dependent allocate tests
TestCanAllocate_CountEdgeCases/Mixed_instance_types and
TestHandleEC2DescribeInstanceTypes/NoFilterReturnsSupportedRegardlessOfAllocation
called rm.allocate against live-detected host capacity. allocate gates on
readMemAvailableGB (live /proc MemAvailable), so a box already running VMs
fails the allocation spuriously, breaking the preflight gate locally.
Pin hostVCPU/hostMemGB/reserved and nil readMemAvailableGB to a synthetic
host, matching the sibling Capacity_decreases_after_allocation subtest.
* fix(eks): route apiserver->pod traffic via agent tunnel (egress-selector cluster)
In the managed-CP topology the apiserver VM sits in the system CP VPC with
no route to the worker pod network in the unpeered customer VPC. k3s default
egress-selector-mode=agent tunnels only apiserver->kubelet, so kubectl
logs/exec work but apiserver->pod traffic (admission webhooks, aggregated
APIs) is attempted directly and times out -- the aws-load-balancer-controller
webhooks fail with context deadline exceeded, blocking Service/Ingress
creation cluster-wide.
Set egress-selector-mode=cluster so apiserver->pod/service traffic routes
through the agent's outbound tunnel, selected by the node hosting the
destination, without requiring inbound CP->worker reachability.
* feat: bump aws-load-balancer-controller addon bundle to v2.11.0
v2.8.1 predates the ALBSingleSubnet feature gate, so single-AZ spinifex
clusters fail Ingress reconcile (subnet resolver dedups to 1 < required 2)
and never provision an ALB. v2.11.0 honors --feature-gates=ALBSingleSubnet=true.
Replace the baked bundle: new CRDs (ingressclassparams + targetgroupbindings),
controller manifest, and image tag v2.11.0, re-applying the spinifex deltas
(IRSA SA annotation, node-minted webhook serving-cert secret, gateway-CA
secret, gateway aws-api-endpoints, caBundle on every webhook clientConfig,
alb IngressClass). Pick up the new ClusterRole configmaps rule. Drop v2.8.1.
* fix: point LBC at the gateway via AWS_ENDPOINT_URL (sdk-v2)
LBC v2.11 uses aws-sdk-go-v2, which keys --aws-api-endpoints on SDK
service IDs (e.g. 'Elastic Load Balancing v2'), not the legacy lowercase
names the bundle passed. Those overrides were silently ignored, so every
non-STS call (ec2/elbv2/acm) went to real AWS and failed TLS against the
gateway CA -- no ALB could provision.
Inject AWS_ENDPOINT_URL in the shared IRSA env block (sdk-v2 global
base-endpoint override; AWS_ENDPOINT_URL_STS still takes precedence for
STS) and drop the dead --aws-api-endpoints flag from the controller args.
* fix: synthesize taggable ARN for default listener rule
Gateway DescribeRules returned the synthetic IsDefault rule with a null
RuleArn. LBC read the arn-less default rule after creating a listener and
issued DescribeTags with an empty resource ARN, which the gateway rejected
with 400 MissingParameter, aborting reconcile.
Set a deterministic RuleArn on the default rule, resolve it back to its
parent listener in DescribeTags (empty tags) and DescribeRules by-arn.
Also disable shield/waf/wafv2 in the LBC bundle to drop unsupported calls.
* feat: inject ELB subnets into alb IngressClassParams for single-AZ ALB
LBC's tag auto-discovery never threads the ALBSingleSubnet feature gate, so a
single-AZ cluster (subnets sharing one AZ) dedups to 1<2 subnets and aborts
Ingress reconcile. Only the explicit-subnet path honors the gate.
Resolve the cluster's ELB-eligible subnets, dedup to one per AZ, and surface
them as EKS_ELB_SUBNET_IDS. The addon sync renders them into the alb
IngressClassParams .spec.subnets.ids so every Ingress takes the explicit path.
An empty set falls back to auto-discovery, preserving the native UX.
* fix: route LBC service clients to gateway via --aws-api-endpoints, link IngressClassParams
Live verification showed two gaps in the LBC addon bundle:
LBC installs a per-client BaseEndpoint resolver, so the global AWS_ENDPOINT_URL
is ignored for the ec2/elbv2/acm clients (calls hit real AWS, failing TLS).
Redirect them with --aws-api-endpoints keyed on the aws-sdk-go-v2 service IDs
(EC2, "Elastic Load Balancing v2", ACM, STS) rather than the legacy lowercase
names the flag never matched. Drop the misleading AWS_ENDPOINT_URL env; the
web-identity STS client still resolves via AWS_ENDPOINT_URL_STS.
The injected IngressClassParams was orphaned: the alb IngressClass had no
.spec.parameters, so LBC fell back to subnet auto-discovery (which fails on a
single-AZ cluster). Link the IngressClass to the IngressClassParams so the
explicit-subnet path is taken.
* fix: stamp EKS-parity providerID on worker nodes for LBC ip targets
k3s registers workers with providerID k3s://<name>, which the AWS Load
Balancer Controller rejects (providerID invalid for EC2 instances) when
resolving nodes for ip-mode target registration. Resolve the instance-id and
AZ from IMDSv2 at agent boot and set --kubelet-arg=provider-id=aws:///<az>/<id>
so LBC can map the node to its EC2 instance. On any IMDS miss the agent keeps
the default providerID and still starts.
* fix(elbv2): honor SubnetMappings in CreateLoadBalancer
LBC supplies ALB subnets via SubnetMappings, not the plain Subnets list.
createLoadBalancer only read input.Subnets, so ENI allocation was skipped,
VpcId/AZ stayed empty, and the HAProxy data-plane VM never launched — the
ALB went active but could not serve traffic. Add flattenSubnetIDs to read
both sources and reuse it in createLoadBalancer and desiredSubnetSet.
* fix(vm): capture GCInterval before spawning GC goroutine
TestStart_SweepsImmediatelyThenStopsOnCtxCancel mutates the package global
GCInterval and restores it in a defer, racing the Start goroutine's read at
time.NewTicker(GCInterval) under -race. Read GCInterval on the caller's
goroutine into a local before spawning so the global access is ordered with
the test's defer; the goroutine touches only the captured local.
* feat: implement OCI Distribution v2 blob and manifest storage
Replace the /v2/* 501 stub with a real path-parsing dispatcher and
content-addressable storage for the ECR registry surface:
- Blob upload (monolithic + chunked PATCH), HEAD/GET, cross-repo mount,
and cancel, with server-side sha256 verification on finalize.
- Manifest PUT with type-aware shallow validation (image manifest blob
existence; image index child-manifest existence + mediaType match),
byte-for-byte GET with stored Content-Type, HEAD, and tag delete.
- Tag list and account-scoped catalog.
Blob and manifest bytes stream straight to predastore; repo, tag,
manifest and in-progress-upload metadata live in a per-account
JetStream KV bucket with optimistic-concurrency CAS on uploads. Chunks
are buffered to a temp upload object and reassembled on finalize, since
predastore exposes no append primitive through the shared object-store
interface. Adds HeadObject to the objectstore abstraction.
Refs: mulgadc/mulga#73
* refactor(ecr): route registry metadata through NATS daemon handlers
The gateway no longer talks JetStream KV directly. The daemon owns the
per-account KV via a MetaService bound to ecr.* request/reply subjects;
the gateway is a NATS client (NATSMetaStore). Blob and manifest bytes
still stream gateway-to-predastore directly.
Also fixes four registry review findings:
- finishUpload: clean up the temp object and upload record on a blob
store failure so the uuid is reusable and no partial blob lingers.
- patchUpload: commit the appended bytes and advanced hash state under a
single KV CAS keyed per write, so a racing PATCH is rejected (409)
rather than corrupting the assembled bytes; the verified digest always
matches the stored bytes.
- cancelUpload: delete both the temp bytes and the record, 204 on
success, 404 on unknown uuid, no dead branch.
- getManifest: return MANIFEST_INVALID (not MANIFEST_UNKNOWN) for a 406
Accept media-type mismatch.
Refs: mulgadc/mulga#73
* fix(ecr): provision account predastore bucket on demand
OCI blob/manifest writes target ecr-{accountID}, which nothing created,
so every push failed with a backend 403. Add idempotent EnsureBucket to
the object store (HeadBucket -> CreateBucket, swallowing already-owned)
and ensure it once per process at the top of the registry ServeHTTP, so
ECR storage appears transparently like AWS-managed ECR.
* fix(ecr): sanitize ':' from predastore blob/manifest object keys
predastore mis-canonicalizes SigV4 for object keys containing ':', so
writes/reads of content-addressable keys (sha256:<hex>) were rejected
403 (reproducible with the aws CLI too). Route BlobKey and ManifestKey
digests through DigestToken so the object key carries no ':' separator.
Implement the registry host-routing middleware so /v2/* requests addressed
at {accountID}.dkr.ecr.{region}.{suffix} carry the parsed target account and
region on the request context; non-registry hosts pass through unchanged.
Add cluster-wide AWS-parity config (aws.region, aws.internal_suffix) with
defaults, and append the AWS-parity ECR SANs (exact control-plane host plus
*.dkr wildcard) to the awsgw server certificate during cert generation.
Refs: mulgadc/mulga#78
* feat: scaffold ECR control-plane + OCI registry gateway with 501 stubs
Add the ECR HTTP surface on awsgw as compiling skeletons that return
NotImplemented, so later work can call into the packages against a fixed
contract.
- gateway/ecrapi: ECR control plane (AWS JSON 1.1, X-Amz-Target). Full action
namespace registered; every action resolves to a shared 501 stub.
- gateway/ecr: OCI Distribution v2 (/v2/*). Catch-all 501 with the OCI error
envelope; GET /v2/ is live for the registry version-check probe.
- gateway/hostmatch: no-op host-routing middleware skeleton for the registry
host group.
- handlers/ecr/storage: per-account bucket + object-key layout constants.
- gateway wiring: ecr in supportedServices, dispatch case, JSON 1.1 error
branch, and /v2 mounted outside the SigV4 group.
* test: cover ECR scaffolding packages for diff-coverage gate
In-package tests for gateway_ecr, gateway_ecrapi, and handlers/ecr so the new
code reports coverage under its own package profile. Diff coverage 76.5%.
The kubernetes provider in the eks-quickstart, eks-addons-access and
eks-https-ingress workbooks was configured from the managed cluster
resource (host = aws_eks_cluster.this.endpoint). On destroy the endpoint
resolved empty, the provider fell back to localhost:80, and refreshing
kubernetes_deployment_v1.hello / kubernetes_service_v1.hello failed with
'dial tcp [::1]:80: connect: connection refused'.
Move the kubernetes provider and the hello Deployment + Service into a
nested workloads/ root module with its own state. It reads cluster
identity from the parent via terraform_remote_state and the live
endpoint/CA from a data aws_eks_cluster source, so the provider is only
ever configured while the cluster exists. Apply parent then workloads;
destroy workloads then parent.
purgeClusterInfra stranded the cluster security groups two ways, leaving
the nodegroup/control-plane SGs (which live in the customer VPC) behind to
pin aws_vpc destroy on DependencyViolation:
* an early return before the SG cleanup block skipped DeleteClusterSGs
whenever any earlier best-effort step (NLB / ENI / VM / EIP) errored;
* DeleteClusterSGs failures were warn-and-swallowed, so a nodegroup SG
still referenced by a terminating worker's ENI leaked while the cluster
completed deletion — no reaper retry ever fired.
Surface DeleteClusterSGs / DeleteClusterCPVPC / DeleteClusterIGW failures
into teardownErrs and gate the KV sweep on them, so a leaked VPC-pinning SG
keeps the cluster DELETING for a retry. Add a bounded DependencyViolation
retry to the SG delete (mirroring deleteENIAwaitingTerminate) to ride out
the worker-ENI-detach race in the common case.
Fixes the manual 'revoke rules + delete SGs before the VPC will destroy'
teardown step.
* fix: persist instance TagSpecifications in RunInstance
RunInstance built the ec2.Instance metadata but never read
input.TagSpecifications, so DescribeInstances returned untagged
instances and tag filters matched nothing. EKS node groups discover
their workers by the spinifex:eks-cluster / spinifex:eks-nodegroup
tags, so scale-up convergence and worker reconciliation depended on
tags that were silently dropped.
Apply instance-scoped TagSpecifications onto the built instance so the
tags surface through DescribeInstances and tag filters.
* fix: serialize nodegroup reconcile via CAS to prevent scale overshoot
UpdateNodegroupConfig is dispatched on a queue group, so two
overlapping requests (e.g. an SDK retry mid-flight) each read
current = len(InstanceIDs) and each launched desired-current workers,
overshooting the target (1 -> 5 for desired=3) and stranding untracked
VMs via last-writer-wins on the unconditional PutNodegroupRecord.
Make every durable nodegroup-record mutation a compare-and-swap on the
KV revision. Scale-up launches one worker at a time and CAS-appends it
only while the record is below the live ScalingDesired, terminating any
surplus VM launched into an already-full record. Scale-down terminates
before the CAS truncation so a crash or retry leaves reclaimable dead
IDs rather than orphaned VMs. A concurrent regression test asserts two
overlapping scale-ups converge to exactly desired under -race.
* fix: return securityGroupRuleSet from AuthorizeSecurityGroup ingress/egress
The AWS provider's aws_vpc_security_group_ingress_rule reads
SecurityGroupRules[0].SecurityGroupRuleId on create; an empty ruleset
panics the plugin (Plugin did not respond), though the rule applies
server-side. Echo the created rules via sgRuleToSecurityGroupRule, the
same shape DescribeSecurityGroupRules already returns.
* fix: surface InsufficientInstanceCapacity from nodegroup scale, not 500
launchWorkersCAS wrapped the bare awserror code via fmt.Errorf, so the
gateway ErrorHandler's exact-match against ErrorLookup missed and remapped
to InternalError (500). An out-of-capacity scale now surfaces the real
InsufficientInstanceCapacity code (400); opaque internal failures still
wrap and stay 500. Adds awserrors.HasErrorCode for the membership test.
* fix: cut EKS node boot ~50s — drop cloud-final deadlock + syslinux menu
Worker/server user-data runcmd ended with `[ rc-service, local, start ]`,
which runs inside cloud-final while the OpenRC `local` service is ordered
after it — a deadlock broken only by OpenRC's ~50s start timeout, stalling
the k3s join. Run the IMDS route script directly instead and keep
`rc-update add local default` for reboot persistence.
setup.sh: cut the SYSLINUX boot-menu countdown 10s->~1s (kept nonzero so the
menu stays interruptible over serial).
Validated live on env19 CP: boot menu 1s, no `waiting for cloud-final`
stall, apiserver ready ~66s (was ~109s+).
mulga-siv-332
* feat: add argocd managed addon + bake bundle into eks-node AMI
The addon catalog advertised coredns and aws-load-balancer-controller, but
the only baked manifest shipped in the eks-node AMI was the spinifex-noop
fixture. Installing any real addon failed with 'no baked bundle for version'
once the VM-side renderer found nothing on disk.
Add argocd v3.0.23 as a real, demonstrable managed addon: catalog entry (no
IRSA), a flattened namespaced install manifest baked into the AMI, and the
manifest.conf bake line. The eks-addons-access workbook switches from coredns
(redundant on K3s, never baked) to argocd.
The baked bundle is generated from upstream install.yaml via kubectl kustomize
with namespace: argocd plus a prepended Namespace, since upstream is
namespace-agnostic and K3s auto-deploy applies with no -n flag.
* fix(eks): run CoreDNS as a node-local DaemonSet for worker DNS
k3s ships CoreDNS as a single-replica Deployment that tolerates the CP taint
and strands itself on the control-plane node. In the managed-CP topology the
worker<->CP pod overlay cannot cross the VPC boundary, so worker pods cannot
reach a CP-resident CoreDNS and lose all cluster + external DNS.
Skip the bundled CoreDNS (coredns.yaml.skip, staged in k3s.initd like traefik)
and deploy a Mulga-owned variant: a DaemonSet (one pod per node) plus kube-dns
internalTrafficPolicy=Local, so every node resolves via its own CoreDNS. Immune
to the cross-VPC overlay gap and topology-agnostic (single-node and managed-CP
both work). Image-side only; no handler change.
* fix(eks): revoke cross-referencing SG rules before delete on teardown
DeleteClusterSGs deleted the control-plane and nodegroup SGs directly, but they
cross-reference each other (EnsureNodegroupSGRules), and AWS refuses to delete a
security group still referenced by another SG's rule. Both deletes failed with
DependencyViolation, the SGs leaked, and the VPC could not be deleted (tofu
destroy hung on DeleteVpc).
Resolve both SG IDs, revoke all ingress on each to break the cp<->ng cycle, then
delete both. Adds RevokeSecurityGroupIngress to sgProvisioner (already on
VPCServiceImpl) and revokeAllSGIngress, with a regression test that models AWS's
DependencyViolation.
* fix(eks): advertise NLB front-end so worker pods reach the apiserver ClusterIP
In the managed-control-plane topology the apiserver advertised its Set-B
node-ip into the in-cluster `kubernetes` Endpoints. That IP lives in the
unpeered CP VPC and is unreachable from worker pods, so every worker pod
using the apiserver ClusterIP (CoreDNS kubernetes-plugin sync, controllers,
argocd redis secret-init) failed.
- k3s_server_vm.go: set advertise-address to the worker-reachable NLB
front-end (PrivateEndpointIP, else EndpointIP); both are already cert SANs.
- nlb.go: add a second :6443 listener on the cluster NLB (same CP target
group) so PrivateEndpointIP:6443 — the advertised endpoint — is served.
- security_groups.go: admit :6443 on the private-endpoint SG.
Validated on a live managed-CP cluster: worker CoreDNS /ready 200, all argocd
pods Running, kubectl logs on worker pods working.
The DisassociateAccessPolicy route regex terminated at
/access-entries/{principalArn}/access-policies, but the AWS EKS REST API
sends the policy ARN as the final path segment:
DELETE /clusters/{name}/access-entries/{principalArn}/access-policies/{policyArn}
The route never matched, so the gateway returned InvalidActionException
(seen as a 400 on 'tofu destroy' of aws_eks_access_policy_association).
The handler also read policyArn from the request body, which AWS leaves
empty for this DELETE.
Capture policyArn as a third path param and pass it through to the input,
dropping the body read. Update route and wrapper tests accordingly.
The host clock needs an NTP daemon; without one it drifts and SigV4 request
signatures fall outside the skew window, kicking the web console back to /login.
Mirrors the ISO rootfs (iso-builder packages.list) so both host-install paths
ship the same software. Guest VMs track the host via kvm-clock.
Bug: mulga-siv-335
* fix: preserve terminated-instance describe-visibility window in GC reaper
The TerminatedTeardownReaper purged a terminated record the moment it drove
the instance's pending NAT/OVN teardown to done. Normal terminations always
arrive NAT/OVN-pending, so the first 2m GC sweep completed and purged them
immediately, removing the instance from DescribeInstances before clients could
observe state=terminated — an AWS-compat divergence that flaked e2e-single
TestRunInstancesMultiCount.
Stamp TerminatedAt on transition and purge a completed record only once it is
older than a 15m visibility window; fresher records are left to the terminated
bucket's 1h TTL, keeping the instance describable as terminated. The ADR-0003
§3 purge-on-completion invariants still hold (their seeded records predate the
window). Closes mulga-siv-328.
* test(e2e): poll intra-SG east-west ping until L2 converges
TestSameSGComms fired a single un-retried 'ping -c 3 -W 2' immediately after
the source instance's SSH became reachable, but the ICMP target is the
destination's east-west data plane. Two instances boot in parallel and the
destination guest NIC / OVN port flows come up several seconds after the EC2
running transition the launch waits on. When the probe wins that race the
source kernel reports 'Destination Host Unreachable' (ARP unresolved), not an
ACL drop, and the test flaked.
Replace the one-shot ping with pingConverged, a bounded poll that retries
until 0% packet loss or 45s. A genuine self-ingress/datapath break still
stays at 100% loss for the full window and fails, so coverage is preserved.
* feat: join nodegroup workers via cluster NLB endpoint
Under the Set-B managed-CP-VPC topology the control plane runs in the
system-account CP VPC while nodegroup workers launch in the customer
VPC. The agent user-data set K3S_URL to the in-VPC control-plane ENI IP
(:6443), which sits in a different, unpeered VPC and is unreachable, so
the join path was dead.
Point K3S_URL at the published cluster NLB endpoint (meta.Endpoint,
https://<frontend-ip>:443, the :443->:6443 listener kubectl already
uses). Guard CreateNodegroup on a non-empty endpoint.
mulga-siv-298
* feat: gate nodegroup ACTIVE on workers registering Ready
launchNodegroupInfra flipped the nodegroup to ACTIVE the moment
RunInstances returned, so a worker that booted but never joined left the
nodegroup falsely ACTIVE with no usable capacity.
Gate ACTIVE on the cluster's Ready-node count (meta.NodeCount, refreshed
by the cluster reconciler from the CP NATS state report — Ready nodes
only) rising by the desired worker count over the create-time baseline.
On timeout the nodegroup is marked CREATE_FAILED with its instance IDs
retained for the reclaim path. Mirrors the CP reconciler's
healthy-observe gating. Bounded by injectable timeout/poll knobs.
mulga-siv-299
* feat(eks): private endpoint access via cross-account Set A ENI
Make EndpointPrivateAccess real: in-VPC workers and kubectl reach the
control plane through a customer-VPC (Set A) resident ENI on the cluster
NLB's LB VM, removing the public-NLB hairpin and the NAT-GW egress path.
Teardown: the Set A ENI is an extra cross-account NIC on the LB VM, not
its primary, so the instance-terminate cascade never reclaims it. Detach
(store-clear) the stale attachment before deleting, so private clusters
delete cleanly instead of looping on InvalidNetworkInterface.InUse.
Refs: mulga-siv-301
* fix(vpc): honor SecurityGroupRuleIds in RevokeSecurityGroup ingress/egress
Modern Terraform (aws_vpc_security_group_ingress_rule) revokes per-rule
resources by SecurityGroupRuleId with empty IpPermissions. The handlers
only read IpPermissions, so they returned success while removing nothing;
the surviving SG-reference rule then pinned both SGs undeletable
(DependencyViolation), deadlocking destroy of mutually-referencing SGs.
Resolve SecurityGroupRuleIds to their stored rules and remove them
(malformed -> InvalidSecurityGroupRuleId.Malformed, absent -> NotFound).
Refs: mulga-siv-321
* fix(vpc): reap the main route table on DeleteVpc
CreateVpc auto-creates a main route table and DeleteRouteTable refuses to
delete a main RT (AWS-faithful), so nothing reclaimed it — every deleted
VPC leaked one orphaned main RT in the rtb KV bucket. DeleteVpc now reaps
the VPC's main route table before deleting the VPC record.
Refs: mulga-siv-322
* fix: make AttachRolePolicy and AssociateRouteTable atomic via KV CAS
Concurrent writes to a single role / route table record used a blind
read-modify-Put with no optimistic-concurrency guard, so parallel writers
racing from the same revision clobbered each other and lost updates.
Verified on env19: three parallel policy attachments to one node role kept
only one; two parallel subnet associations to one route table kept only
one (the provider then timed out waiting for state 'associated').
Route the mutating paths (AttachRolePolicy/DetachRolePolicy,
AssociateRouteTable/DisassociateRouteTable) through a revision-guarded
Update + bounded retry, matching the existing accountCounterBucket CAS
idiom. Add concurrent regression tests covering both lost updates.
* docs: add EKS Terraform workbooks (quickstart, addons-access, https-ingress)
Three OpenTofu workbooks demonstrating EKS on Spinifex, each ending in a
browsable load-balanced demo app:
- eks-quickstart: minimal single-node cluster + NodePort demo
- eks-addons-access: CoreDNS managed addon + access-entry RBAC
- eks-https-ingress: 3-node cluster behind an HTTPS ALB terminating TLS
with an ACM-imported self-signed certificate
Each covers the Spinifex-specific endpoints, the auto-managed worker SG
NodePort pattern, and the omitted addon_version constraint.
* fix: idempotent Delete* across VPC family + volume (RLC1)
Delete handlers now return empty success when the resource is already
absent (nats.ErrKeyNotFound), instead of *NotFound. Transient read
errors still map to ServerInternal. Covers DeleteVpc, DeleteSubnet,
DeleteSecurityGroup, DeleteNetworkInterface, DeleteInternetGateway,
DeleteNatGateway, DeleteRouteTable and DeleteVolume.
Makes destroy retries converge instead of failing mid-graph, removing a
source of orphaned/zombie resources between deploys. Adds a
TestRLC1_*DeleteIdempotentOnAbsent invariant per package and updates the
daemon delegation table; supersedes the old absent->NotFound tests.
* feat: prune orphan ENI LSPs in reconcile applyPorts (ADR-0003 §4)
applyPorts was create-or-update only, so a guest LSP whose
spinifex:eni_id no longer maps to an intent ENI survived instance
terminate and VM-host reinstall — the orphan-LSP capacity-drift root
cause. It now takes pruneOrphans (mirroring applySGs): on a full
reconcile it lists LSPs and tears down each managed ENI port absent from
intent via the composed topology.DeletePort cascade (clears port-group
memberships, then deletes the LSP). The apply-only startup path does not
prune, preserving in-flight ports until subscribers converge.
Adds ListLogicalSwitchPorts to the OVN client (live + mock). Invariant
TestRLC5_OrphanLSPPrunedAfterReconcile guards ADR-0003 §4; a startup-race
test guards the apply-only path.
* feat: idempotent + tracked instance terminate (ADR-0003 §1/§2)
Make terminate idempotent across all three layers and break the
un-terminable-ENI deadlock, and record per-dependency teardown so the
GC backstop can later tell a fully-reaped instance from a dangling one.
- vm.Manager.Terminate: absent / already-terminated / shutting-down now
return nil instead of NotFound / invalid-transition (ADR-0003 §2).
- VM.Teardown map + teardown.go marks: sync steps stamped Done/Failed via
cleaner error returns; fire-and-forget OVN LSP and NAT rule stamped
Pending for the reconciler / GC reaper (ADR-0003 §1).
- InstanceCleaner: DeleteVolumes / ReleasePublicIP / DetachAndDeleteENI /
RemoveFromPlacementGroup / ReleaseGPU now return error.
- vpc ForceDeleteInstanceENI bypasses the in-use guard for the owning
instance, breaking the failed-detach deadlock; plain delete still
rejects an in-use ENI held by another instance.
- gateway TerminateInstances: KV-health-gated idempotent absence (a failed
terminated-bucket query does not fabricate success, ADR-0003 §3).
- Invariants: TestRLC1_TerminateIdempotentOnAbsent, TestRLC4 teardown-map
stamping + failed-dependency, TestRLC_ENINonDeadlock.
* feat: reconciler GC backstop framework + terminated-teardown reaper (ADR-0003 §3)
Add the shared reality->desired garbage-collection framework (inverse of
Restore) and its first reaper, closing the gap 295.3 deferred: nothing
retried pending/failed teardown or purged the terminated record when its
dependents reached done.
- vm/gc.go: GarbageCollector + Reaper interface (Class/Scope/Sweep).
KV-health gated — holds and alarms when the desired-state cannot be
trusted, never reaps against it. ClusterWide reapers run only on the
elected leader (optional elector); NodeLocal run on every node. Start
sweeps immediately then every GCInterval.
- vm/teardown_reaper.go: TerminatedTeardownReaper (node-local). Scans this
node's terminated records, re-drives each not-done dependent through the
idempotent cleaner, and purges the record once all reach done. Records
already complete on arrival are left to the bucket's 1h TTL.
- StateStore gains DeleteTerminatedInstance (JetStream impl already existed).
- daemon: wires the GC after the pending watchdog, gated on a cheap
JetStream AccountInfo health probe.
- Cluster OVN-LSP arm already lands via the reconciler drift prune (295.2).
- Invariants: TestRLC5_GCRetriesPendingTeardownToDoneThenPurges,
TestRLC5_GCNeverPurgesIncompleteRecord, TestRLC5_GCHoldsWhenKVDegraded,
TestRLC5_GCSkipsOtherNodesRecords.
* feat: VPC-family dependency guards (ADR-0004 rule #3)
DeleteSubnet blocks while a live ENI attachment (a resident instance)
remains; available/orphan ENIs do not pin it. DeleteNetworkInterface
guards on the structured attachment (eniIsLiveAttachment), not a bare
in-use string. DeleteNatGateway blocks while a live route table forwards
to it and disassociates its EIP keeping the allocation (AWS parity);
CreateNatGateway now marks the EIP associated so the disassociate is
real and a second NAT GW cannot reuse the allocation.
Binding invariants: TestRLC3 dependency-guard completeness across the
subnet/ENI/NAT-gateway handlers.
* feat: EBS volume data-safety GC override (ADR-0005)
The shared GC backstop reaps actual-desired for every other class; for
volumes that default is forbidden because they carry customer data. Add
VolumeLeakReaper (node-local, rides the 295.5 GC): it marks a volume left
attached to a definitively-gone instance as orphaned (a non-destructive
tag) and alarms, but NEVER issues a delete. The daemon feeds it only this
node's terminated instances whose volumes-teardown failed, so the shared
predastore scan can never false-mark another node's live-instance volume.
DeleteVolume idempotency + the detach/snapshot guards + the
DeleteOnTermination teardown retry already landed (295.1/295.3/295.5).
Binding invariants: TestRLC5 (GC marks+alarms, never deletes data),
TestRLC3 (detach-before-delete).
* test(elbv2): lock ADR-0002 §3 live-only TG in-use guard (mulga-siv-295.4)
* feat(eks): idempotent DeleteCluster on absent + ADR-0006 lifecycle invariants (mulga-siv-295.8)
DeleteCluster returns empty success when the cluster meta is already swept
(Common Resource Lifecycle Contract #1), so tofu destroy retries converge.
Adds TestRLC1 (idempotent delete), TestRLC2 (billable-before-KV-sweep ordering)
and TestRLC3 (NLB target-group no-orphan, riding ADR-0002/172).
* fix(eks): GC reaper terminates orphan control-plane VM after cluster delete (mulga-siv-294)
A normal DeleteCluster can leave a running control-plane VM behind (terminate
completes async; a daemon restart re-launches it from saved running-state) after
the cluster meta is swept — a billable orphan with no owner. Add EKSBillableReaper
on the 0003 GC backstop: a running managed-by=eks CP VM whose cluster meta is
definitively gone is terminated. The CP ENI now carries the customer-account tag
so a system-account VM can be mapped back to its cluster meta; the reaper never
reaps on uncertainty (ENI gone/untagged or meta present).
* feat: reclaim orphan EKS NLB + NAT-GW EIP in billable reaper
The billable GC reaper terminated an orphan control-plane VM but left the
cluster's NAT-gateway EIP and NLB front-end NAT behind when the meta was
gone. After a successful terminate, run the name/tag-driven DeleteClusterNLB
+ DeleteClusterCPVPC teardown (idempotent, meta-independent) so the billable
NAT-GW EIP and NLB NAT are reclaimed too. ADR-0006 §5.
* fix: persist NAT-GW & route-table tags and reorder CP-VPC teardown to reclaim EKS NAT-GW EIP
CreateNatGateway and CreateRouteTable both dropped input.TagSpecifications,
so the tag-driven DeleteClusterCPVPC teardown could not discover the cluster
NAT gateway or its private route table. The explicit DeleteNatGateway +
ReleaseAddress were skipped and the billable NAT-GW EIP leaked under the
system account on every clean DeleteCluster.
Three fixes:
- CreateNatGateway persists ExtractTags(TagSpecifications, natgateway).
- CreateRouteTable persists ExtractTags(TagSpecifications, route-table).
- DeleteClusterCPVPC tears down route tables before the NAT gateway, so the
rule #3 route-ref guard (DependencyViolation / InvalidAddress.Locked) is
not tripped and the EIP releases.
Adds tag-persistence regression tests for both create paths and a teardown
ordering invariant (TestRLC5). Validated live: NAT-GW EIP released cleanly,
no DependencyViolation/Locked.
* fix: route EKS control-plane VM terminate cross-node + DELETING teardown backstop
Multi-node EKS cluster delete wedged in DELETING and stranded the billable
NAT-GW EIP. Two root causes, both invisible on a single-node box:
Cross-node terminate was never routed. TerminateSystemInstance was node-local,
so terminating a CP VM owned by another node returned NotFound (treated as
idempotent) and the teardown deleted a still-attached ENI
(InvalidNetworkInterface.InUse). That error short-circuited purgeClusterInfra
before the managed CP-VPC teardown that releases the NAT-GW EIP. Split
TerminateSystemInstance into a local body plus a routing wrapper that issues
system.TerminateInstance.<id> to the owning node. Bind that subscription for
every system-managed VM (elbv2 + eks) via the OnInstanceUp funnel, so CP VMs
placed on the coordinator's own node (and recovered after a daemon restart) are
reachable too, not only remotely-launched ones.
No DELETING backstop. A failed synchronous DeleteCluster left the cluster
DELETING with nothing to re-drive it. Add EKSDeletingReaper (vm.Reaper, class
eks-deleting) that re-runs purgeClusterInfra for clusters wedged past a min-age,
gated by the per-cluster reconciler leader lease. Stamp ClusterMeta.DeletingSince
on the transition into DELETING.
TerminateK3sServerVM additionally waits out any residual ENI-InUse with a bounded
budget as defense-in-depth.
Validated live on env19 3-node: first delete-cluster converges in 29s, both
remote CP VMs terminated cross-node, NAT-GW EIP released, backstop never fires.
* fix(network): gate gateway reconcile on real datapath reachability
Post-reboot the external OVN datapath can stay dead while every
control-plane signal is green: the chassisredirect SB binding is claimed
(up=true), NAT rows are present and the host neigh is primed, yet
ovn-controller leaves stale gateway/localnet flows so EIP north-south and
the VM->gateway heartbeat forward nothing. ensureGatewayClaimed only
nudges recompute when the binding is unclaimed, so a claimed-but-dead
datapath is never detected or repaired.
Add ensureGatewayDatapath: after the SB claim converges, probe the
gateway LRP IP (OVN answers ICMP natively, no guest or security-group
dependency), and on a miss force one ovn-controller recompute and
re-probe until a short deadline. Reuses the existing recompute nudge and
no-ops when no verifier is wired or no gateway IP resolves.
* fix: self-heal external uplink after reboot boot-race
Post-reboot, ovs-vswitchd/ovn-controller order after the passive
network.target rather than network-online.target, so they start
concurrently with systemd-networkd. Under a heavy boot the WAN-glue veth
(veth-wan-ovs) can come up admin-down before networkd brings it up,
leaving br-wan with no carrier and the external datapath dead while every
control-plane signal reconciles green.
Broaden the datapath health-gate remediation from recompute-only to
RepairDatapath: re-assert the veth admin state (covers the admin-down
link, which a flow recompute cannot revive) then recompute (covers a
stale gateway ofport). Idempotent; the veth ops are skipped in
physical-uplink mode where the device is absent.
Also capture ip -d link (veth admin/carrier) and ovs-ofctl show br-ext
(ofport) in the reboot diag bundle to disambiguate the two failure shapes
from CI artifacts.
* fix: restore AWS-faithful Delete* NotFound; move idempotency to teardown callers (mulga-siv-306)
RLC1 (f9218982) made EC2/VPC/EBS Delete* handlers return success on an absent
resource, breaking AWS compatibility and the cross-account isolation signal (a
cross-account delete reads as absent -> success instead of *.NotFound).
Restore the AWS contract at the API while keeping lifecycle convergence:
- Handlers return canonical *.NotFound on absent: DeleteVpc, DeleteSubnet,
DeleteSecurityGroup, DeleteInternetGateway, DeleteNatGateway, DeleteRouteTable,
DeleteVolume, and the public DeleteNetworkInterface (ForceDeleteInstanceENI
keeps its idempotent force path, ADR-0003 section 2).
- Add awserrors.IsNotFound; destroy orchestration (instance terminate, EKS
CP-VPC teardown, EKS IGW/SG cleanup, k3s rollback, ELBv2 ENI/SG teardown)
tolerates NotFound so re-issued/double-targeted deletes converge.
- Flip the RLC1 unit invariants to assert *.NotFound on absent; revert the
daemon round-trip cases; add IsNotFound + CP-VPC convergence tests.
ELBv2 deletes and TerminateInstances stay idempotent (AWS itself is).
* fix(network): gate ingress EIP datapath on guest SB Port_Binding convergence
TestMultinodeGuestSSH intermittently times out (Connection timed out, not
#355's reset-by-peer) connecting to a fresh trio instance's public IP after
the suite churns instances across all three chassis. The instance is running
and AddEIP has installed the DNAT row, flushed/primed the host neigh, and
barriered the gateway-chassis flow — but none of that proves the
gatewayLRP->guest geneve hop: until the guest ENI's southbound Port_Binding is
up on its hosting chassis the cross-node flow is missing and the DNAT-translated
packet blackholes.
Add ensureGuestPortDatapath, the guest-side analog of ensureGatewayDatapath:
after AddEIP, probe the guest LSP's SB Port_Binding (ovn-sbctl up column); on a
miss force one ovn-controller recompute (binds the port / installs the remote
flow) and re-probe until a short deadline. Recompute, not RepairDatapath: this
is flow/binding convergence, not a WAN-uplink fault. No-op when no verifier is
wired or the EIP carries no guest port. spec.PortName is populated for every
associated EIP regardless of NAT mode.
Extends GatewayClaimVerifier with GuestPortUp and wires the gate into
applyEIPs so it runs in the continuous reconcile loop on every node.
* Revert "fix(network): gate ingress EIP datapath on guest SB Port_Binding convergence"
This reverts commit 728b1a9cec6eaf5c7ef5e9eda4469f239b4d8fc0.
* fix(network): key EIP DNAT delete on external IP, not logical IP
A dnat_and_snat row's identity on the VPC gateway router is its external
IP. The logical (private) IP is not a stable key: the EIP pool round-robins
and instance private IPs are reused as instances recycle, so the same
logical IP is held by different EIPs over time.
DeleteEIP deleted the row by logical IP, so a stale or retried delete for a
freed EIP clobbered whichever live EIP now held that recycled private IP --
the next owner's row vanished with no re-add and its public IP went dark
(SSH SYN never left br-wan: Connection timed out). Lifecycle-resilience
teardown/GC retries amplified this by firing vpc.delete-nat repeatedly.
Key the delete on the external IP via DeleteNATByExternalIP and skip when
no external IP is supplied (no EIP means no dnat_and_snat row). A stale
delete now targets the freed EIP's own external IP (ErrNATNotFound no-op)
and can no longer touch the live row.
* refactor: relax DeleteNatGateway to AWS-faithful
Remove the route-reference dependency guard (checkNatGatewayRouteRefs) so a
NAT gateway deletes even while route tables still forward to it, matching AWS
(the route blackholes). The reconciler's loadNATGWs drops the SNAT once the
record leaves the active bucket, so a forwarding route is not a delete-blocking
dependent. EIP is still disassociated on delete (no leak).
* test: invert NAT-GW route-ref invariant to AWS-faithful
TestRLC3_NatGatewayBlocksWhileRouted asserted DependencyViolation while a
route forwarded to the NAT gateway, which is AWS-infaithful. Replace with
TestRLC3_NatGatewayDeletesWhileRouted: delete must succeed whether or not a
live route still targets it (the route blackholes; the reconciler drops the
SNAT). Per ADR-0004 §2.
* fix: tear down NAT-GW SNAT on delete (restore delete-time fan-out)
The guard commit (53ed55a9) dropped publishDeleteEventsForNatGateway on the
assumption a routed NAT GW could never be deleted. With the route-ref guard
removed, deleting a NAT GW while routes still target it left the SNAT in
place (applyNATGWs is create-only, so the reconciler never prunes it) and
egress kept working — TestNATGateway's post-teardown 'lost internet' check
failed. Restore the fan-out: on delete, scan route tables and publish
vpc.delete-nat-gateway per associated subnet so egress stops, while the
route entry survives as a blackhole (AWS-faithful).
* fix: probe EIP datapath in reboot self-heal, not the gateway LRP IP
The post-reboot gateway datapath self-heal probed the gateway LRP IP for
liveness, which OVN answers natively at the router — so it stayed green even
when the EIP NAT pipeline was stranded (ofproto/trace dropped EIP traffic at
table 0). RepairDatapath never fired and TestRebootResilience timed out, with
the drift loop re-probing the same false-green every cycle.
Prefer probing an associated EIP: force a fresh ARP of the dnat_and_snat
external IP (flush neigh, ping to trigger, read the resolved MAC). OVN answers
ARP for the external IP from the gateway chassis independent of the guest, so a
resolved MAC proves the WAN uplink forwards and the NAT flows are installed —
the signal the LRP IP cannot give. Fall back to the LRP IP for VPCs with no
EIP. The existing drift-loop cadence now re-heals a later flow strand.
* fix: lb-agent heartbeats over mgmt port, not the WAN/EIP datapath
The lb-agent heartbeat to awsgw is control-plane and must survive a host
reboot that strands the OVN/EIP north-south data plane. Customer
(internet-facing) lb-agents heartbeated over the WAN GatewayURL, riding the
EIP datapath, so after a reboot the ALB never reported healthy until the
datapath self-healed. System LBs already heartbeated over the on-link
br-mgmt NIC; this extends that path to every lb-agent.
buildLBAgentEnv now always uses mgmtGatewayURL (on-link MgmtBridgeIP, the
mgmt NIC the lb-agent VM already carries), falling back to the WAN URL only
when no mgmt bridge exists. Drops the account-id branch.
* fix: re-heal guest ENI SB Port_Binding post-reboot so EIP ingress converges
Post-reboot every EIP in a VPC goes dark north-south: the DNAT row, host
neigh and gateway-chassis flow are all in place (ARP to the EIP resolves)
but the gatewayLRP->guest flow is missing until ovn-controller binds the
guest ENI's SB Port_Binding, so the DNAT-translated packet blackholes and
the public IP is unreachable on an otherwise-running instance.
Restore the guest-port convergence gate in applyEIPs: probe the guest LSP's
SB Port_Binding up column; on a miss force an ovn-controller recompute and
re-probe until a deadline. Unlike the earlier reverted form, recompute on
every miss, not once: after a host reboot the guest tap is replumbed several
seconds after the reconcile runs, so a single early nudge fires before the
port exists and never binds it. The window is widened to span guest boot.
This is the southbound analog of the gateway datapath self-heal; the EIP
ARP probe cannot detect it because the gateway chassis answers ARP for the
external IP even while the guest delivery flow is absent.
* fix(network): force post-reboot EIP recompute on guest-port bind, startup-only
The reboot suite passed only by accident: the pre-#351 gateway-claim verifier
polled the always-chassis-less gateway LRP, so it fired an ovn-controller
inc-engine/recompute on every reconcile. That spurious recompute was load-bearing
- it reinstalled the centralised gateway DNAT flows that ovn-controller's
incremental engine misses on a cold (post-reboot) start. #351 correctly switched
the check to the chassisredirect port, eliminating the every-cycle recompute, and
the EIP datapath went dark post-reboot with every control-plane signal green.
Restore the kick, scoped to the post-reboot startup path: applyEIPs now gates a
single recompute on the guest ENI's SB Port_Binding chassis claim - the reliable
signal the tap is back, unlike the up column which stays stale-true across a
reboot. Drift never recomputes, preserving the #351 fix for freshly attached
EIPs. Probe renamed GuestPortUp -> GuestPortBound (chassis, not up).
* Revert "fix(network): force post-reboot EIP recompute on guest-port bind, startup-only"
This reverts commit 792e318211ef03efbe24febeb470749d88c0b03c.
* test(e2e/reboot): capture dataplane diag on ALB serve failure
The post-reboot ALB-serve assertion (8.6) failed with no flow/ARP evidence:
the OVN/OVS dataplane dumps only ran on the 8.5 targets-unhealthy path, but
8.5 passes on the daemon's own health view while the north-south EIP datapath
is dark, so the serve failure (8.6) left nothing to localise the drop.
On a serve failure, capture dumpALBProbe/dumpOVNDataplane/dumpOVNClaimState/
dumpHeartbeatDiagnostics/dumpAppDiagnostics before failing. Add to the OVN
dataplane dump: dump-flows br-ext and fdb/show br-ext (north-south bridge,
previously only br-int), and a real ARP probe that flushes the neigh entry
first so primeReachability's pre-seeded reachable entry can no longer mask an
unanswered ARP. Pure diagnostics; no production change.
* test(e2e/reboot): decisive datapath diag — untruncate br-int, real-ingress ofproto/trace, conntrack + tap capture
* test(e2e/reboot): add south->north return-path ofproto/trace to localize egress drop
* test(e2e/reboot): derive LRP MAC from realized flow + remediation experiment (recompute vs ovn-controller restart)
* test(e2e/reboot): dump persisted SB MAC_Binding + external MACs; flush experiments (conntrack vs MAC_Binding) to isolate recompute-resistant return-path drop
* test(e2e/reboot): untruncate Port_Binding + enumerate DGP/chassisredirect claim state (does cr-gw row exist + resident chassis)
* test(e2e/reboot): dump NB source of return-path drop (lflow stage, NAT external_mac/options, LR policy, LSP port_security)
* fix: exempt public-IP instances from subnet egress drop gate (mulga-siv-311)
Post-reboot reconcile installs the priority-1100 subnet egress drop gate for an
IGW-attached subnet that has no 0.0.0.0/0 route, but nothing punched the subnet's
public-IP instances through it. lr_in_policy runs before lr_out un-DNAT/SNAT, so an
inbound connection's reply still carries its private source when it hits the gate and
is dropped. The ALB/EIP datapath goes dark south->north post-reboot while NAT, ARP,
chassis and guest-port binding all read green; the pre-reboot path survives only
because the event-driven gate never evaluates a subnet created after IGW attach.
Install the existing priority-1200 /32 reroute (above the gate) for every public-IP
instance during reconcile. Reroute only: the instance's dnat_and_snat already supplies
SNAT, unlike EnsureSystemInstanceEgress which also adds a plain snat. Public IPs come
from two disjoint sources -- auto-assigned/ELB addresses on the ENI record and user
EIPs in the EIP bucket -- so drive the exemption from both intent.Ports (new
PortSpec.PublicIP) and intent.EIPs.
* refactor: scope public-IP drop-gate exemption to gated subnets
Only install the priority-1200 reroute when the instance's subnet actually carries
a drop gate. Routed subnets egress via their priority-1000 reroute and need no
exemption, so bounding on gate presence keeps the change inert for IGW-routed and
NATGW-routed subnets (no redundant or overriding policy). Adds a negative test.
* refactor: drop unused RemoveEIPInstanceEgress
No production caller; the exemption is install-only in reconcile. Removed the
interface method and impl rather than leave dead API surface.
* test: dump guest console + OVN/datapath diag on SG-to-SG client SSH timeout
TestSGToSGDatapath waited for client-vm SSH via EventuallyErr and reported
only 'exit status 255' on timeout — no guest-side or datapath state. When the
fresh public-IP VM is unreachable for the full window the failure was
undiagnosable from CI artifacts.
Swap the wait to the non-fatal trySSHReady probe and, on timeout, call
DumpVPCFlowDiagnostics (guest console tail, ovn-nbctl/sbctl show, NAT rows,
br-int flows / conntrack / host ARP keyed on the public + private IP) before
Fatal, mirroring vpc_test/baseline_test. Captures persist to the artifact dir
for the Stage 2 analyzer bundle.
* fix: ownership-scope public-IP teardown to survive IP recycling
A live instance's dnat_and_snat rule, host ARP, and IPAM lease were torn down
by the delayed garbage-collection teardown of a different, long-dead instance
that previously held the same recycled public IP. The 2-min vm/gc
instance-teardown reaper re-runs ReleasePublicIP on a stale terminated record;
because the external IPAM round-robins a small pool, the IP had already been
reassigned to a live instance, and both teardown primitives acted on the IP by
value with no ownership check:
- policy.DeleteEIP deleted the dnat_and_snat row by external IP alone,
discarding the logical IP it receives.
- StaticPoolAllocator.Release freed the pool entry by IP value.
Scope both to the current owner so a stale or duplicated teardown for a
recycled IP is a no-op:
- DeleteEIP looks up the row by external IP and skips the delete (and the ARP
flush) when its logical IP no longer matches, mirroring AddEIP's idempotency
lookup. Defeats both external- and logical-IP reuse races.
- Allocator.Release / ExternalIPAM.ReleaseIP take an ownerENIID; the static
allocator no-ops a release whose ENI differs from the live allocation, and
treats an already-freed owner-scoped release as idempotent. Teardown and
rollback callers pass their ENI; user-driven EIP release passes none.
Proven via E2E single TestSGToSGDatapath (run 27515293563): client-vm SSH
black-holed after its NAT was deleted by the dead instance's GC sweep.
* fix(eks): single-flight DeleteCluster teardown per cluster
SDK retries of one DeleteCluster get fanned across nodes by the worker
queue group, so purgeClusterInfra raced the same cluster on multiple
nodes: duplicate ENI/NLB teardown, duplicate TargetGroupNotFound
deregisters, and a real double-release risk on the billable NAT-GW EIP.
Idempotency kept it correct but noisy.
Add a dedicated per-cluster teardown lease (teardownLeaderKey =
reconcilerLeaderKey + "/teardown"), distinct from the reconciler lease
so a delete never contends with the still-live reconciler that holds the
reconciler lease until it observes DELETING and exits. DeleteCluster
acquires it after the NotFound-idempotency check: the loser returns the
cluster as DELETING (AWS-async semantics) without running the teardown;
the winner drives purgeClusterInfra and releases. The DELETING backstop
reaper shares the same lease, so at most one purgeClusterInfra runs per
cluster cluster-wide.
A nil leaderKV (single-node) skips gating, so local behavior is
unchanged.
* fix: env-driven SSH-ready budget for slow baremetal guests
Add SPINIFEX_SSH_READY_TIMEOUT override for the single-node SSH-handshake
probes (default unchanged at 3m). Baremetal boots q35 guests slower (OVMF +
NBD-backed disks) than CI loop devices and intermittently misses the 3m
window; the harness can now widen it to 5m without affecting shared-runner
e2e. Applied at all single-node probe sites; failure messages now report the
actual budget.
* fix: shared external switch to stop per-VPC localnet ARP collision
Per-VPC external switches each attached an untagged localnet port to the same
physical network (br-ext). OVN cannot keep multiple untagged localnets on one
physical network L2-separate, so with two or more concurrent public-IP VPCs ARP
for a public IP landed on the wrong VPC's switch and the owning gateway router's
ARP responder never fired, leaving the public IP unreachable.
Move to a single shared external switch (provider-network pattern): one switch
with one localnet on network_name=external. Each VPC's gateway LRP attaches via
its own router-type switch port; per-VPC NAT isolation is unchanged. AttachIGW
ensures the shared switch idempotently and is migration-safe; DetachIGW removes
only the per-VPC port + LRP and preserves the shared switch. DeleteVPC removes
the per-VPC gateway switch port from the shared switch so it does not dangle.
Reconcile detects attachment via the gateway LRP since the shared switch carries
no vpc_id.
* fix: owner-scope DeleteEIP by ENI port to survive identical pair recycling
A stale or GC-re-emitted vpc.delete-nat for a terminated instance tore down a
live instance's dnat_and_snat rule (and flushed the host ARP) when the public
IP and private IP both recycled, together, to that live instance in the same
VPC. The (external_ip, logical_ip) pair-key matches under identical recycling,
so the existing logical-IP guard fell through.
Stamp the owning ENI port (spinifex:logical_port) on the dnat_and_snat row at
AddEIP, thread the port through handleDeleteNAT -> DetachEIP -> DeleteEIP, and
skip the delete when the row's stamped port differs from the incoming one. The
IPAM release was already owner-scoped; this mirrors that guard at the NAT layer
so any stale delete for a recycled pair is a no-op. Empty port preserves the
prior pair-key behaviour.
* chore: bump viperblock to encrypted-snapshot VolumeUUID nonce fix
Pull viperblock 1044c2a (feat/snapshot-uuid-nonce-fix) so E2E builds against
the fix for the encrypted AMI snapshot metadata tag-verify failure
(cipher: message authentication failed) on instance launch. Pairs with
viperblock PR #22.
* fix: forward viperblock master key to nbdkit mount path
viperblockd loaded the master key but never passed it to the nbdkit
plugin it spawns for volume mounts, so encrypted boot volumes opened
in cleartext and failed LoadState with an encryption-state mismatch.
Forward cfg.EncryptionKeyFile into NBDKitConfig and emit the plugin's
encryption_key_file= argument when set.
* chore: bump viperblock to fresh-volume VolumeUUID mint fix
* chore: bump viperblock to snapshot-link LoadState ordering fix
* fix: route encrypted-volume config writes through viperblock keyholder
config.json is a sealed VBState whose AES-GCM tag and StateSeqNum-derived
nonce can only be advanced by the master-key holder that owns the volume.
The EC2 volume handler wrote it directly to S3, so mergeVolumeConfig had to
refuse encrypted volumes (rewriting would strip the tag) — bricking
UpdateVolumeState (attach in-use / detach available) and ModifyVolume.
Route encrypted-volume config updates through viperblockd, mirroring the
ebs.snapshot idiom: per-volume ebs.config.{volumeID} hits the owning node's
live VB (the single StateSeqNum writer); on ErrNoResponders the ebs.config
queue group opens the detached volume exclusively and reseals. Cleartext
volumes keep the direct-S3 fast path. No crypto leaves viperblockd.
mulga-siv-323
* fix: unwrap encryption envelope when snapshot handler reads config.json
CreateSnapshot and snapshotInUseByVolumes decoded config.json straight into
a VBState. For encrypted volumes config.json is an at-rest envelope
{payload, authtag}; a raw decode yields a zero VBState (SizeGiB==0), so
CreateSnapshot 500s on the size guard and the in-use scan silently drops
encrypted volumes' snapshots. Unwrap via viperblock.StateBody first,
mirroring the volume handler's mergeVolumeConfig path.
mulga-siv-324
* test: cover viperblockd ebs.config handlers
Unit/integration coverage for applyConfigUpdate, makeConfigUpdateHandler
(ebs.config.{volumeID}), and the ebs.config queue-group fallback: grow-only
size reconcile, live-VB apply, invalid JSON, apply errors, and the
detached-volume open path. Clears the diff-coverage gate (68.8%).
* chore: pin viperblock to dev tip (#22 merged)
docker export of a container leaks its overlay mount on Docker's
overlay2 driver — the mount survives docker rm, so the layer dirs are
never reaped except by a daemon restart. The lb-agent microVM build ran
docker export on every e2e job across ~9 runners, accumulating ~69G of
orphaned overlay2 layers on the CI host (disk hit 94%).
Build the rootfs with `docker buildx --output type=local` from a small
generated Dockerfile instead, which writes the image filesystem to a
directory and releases all mounts — nothing leaks. Keep the podman path
(its store is unaffected). Tighten the e2e cleanup image prune to
dangling-only so the alpine base stays cached instead of re-pulling.
CI e2e-build compiles the spinifex module with GOWORK=off, resolving the
go.mod pins instead of the go.work local submodule redirect. v1.7.0 of both
modules predates symbols spinifex now calls (viperblock.StateBody,
predastore auth.ParseRoleARN), so the build failed with undefined-symbol
errors that go.work masked locally. Pin both to current dev-branch
pseudo-versions until a coordinated release is cut.
* feat: enable viperblock at-rest encryption by default on fresh installs
Fresh installs left nodes.<node>.viperblock.encryption_key_file empty, which
viperblockd treats as legacy cleartext mode. The encryption machinery existed
but the install path never generated a key or set the field.
Provision a cluster-wide viperblock master key in all three fresh-install
flows and render its path into spinifex.toml:
- admin init (single node): generate the key locally.
- admin init --multi-node (leader): generate once, save locally, and
distribute via the formation server (mirrors IAM master key distribution).
- admin join (joiner): receive the shared key from the leader rather than
minting its own, so volumes sealed on any node open everywhere. Lenient on
absence — an older leader returns no key, so the joiner warns and stays
cleartext instead of failing the join.
The key file is omitted from the template when unset, so upgrades with no
existing key keep running cleartext.
* fix: make viperblock encryption key group-readable for both readers
The key is loaded by two services per node — viperblockd (spinifex-viperblock)
and the awsgw EC2 handlers (spinifex-gw), both in the spinifex group — so it
must be group-readable, not owner-only like the per-node predastore key.
The previous commit generated the key but never registered its path with the
deployment ownership machinery, leaving /etc/spinifex/viperblock as root:root
0700 (non-traversable) and the key as root:root 0600. The daemon failed at
startup with "stat ... permission denied".
Mirror master.key: setup.sh creates /etc/spinifex/viperblock (0750), and
SetServiceOwnership chowns the dir to spinifex-viperblock and promotes the key
to root:spinifex 0640.
* fix: read viperblock config.json through StateBody for encrypted volumes
The metadata layer parsed config.json with json.Unmarshal/NewDecoder directly,
which fails on encrypted volumes/AMIs now that viperblock wraps VBState in a
valid-JSON envelope ({"v":1,"payload":...,"authtag":...}). DescribeImages
silently skipped encrypted AMIs (count=0 → "no AMIs available"), bricking the
whole launch path.
Route every config.json read through viperblock.StateBody, which unwraps the
envelope (no key needed — the metadata is authenticated, not encrypted) and is
a no-op for unencrypted volumes:
- image: DescribeImages, getVolumeConfig, amiNameExists, GetAMIConfig
- volume: getVolumeConfigAndEncryption, and mergeVolumeConfig (so the
refuse-to-merge-encrypted guard sees the real VBState instead of clobbering
the envelope with a tag-less blob)
- admin: readAMIConfig, readVolumeConfig
* feat(eks): VM-side managed-addon delivery transport
Stage CreateAddon manifests in KV host-side; the control-plane VM pulls the
staged set through the AWS gateway (signed GET /clusters/{name}/internal-addons/
{accountId}), renders the baked bundles into the K3s auto-deploy dir, and
reports per-addon delivery status on a dedicated eks.addon.*.status channel that
the cluster reconciler CASes onto AWS-visible AddonStatus (CREATING->ACTIVE).
- gateway: ListInternalAddons route + ListStagedAddonManifests NATS action
- handlers: addon status reconciler (nextAddonStatus), staging installer
- eks-node AMI: eks-gateway-fetch binary + mulga-eks-addon-sync agent
- agent renders idempotently (cmp before replace) so K3s auto-deploy does not
re-apply every tick and clear the Addon CR .status.gvks readiness signal
- agent mirrors per-tick delivery outcomes to the serial console for host-side
observability
* fix(eks): provision cluster NLB synchronously so the endpoint carries the real front-end IP
The cluster NLB front-end IP is assigned by the async LB-VM launch
(launchLBVMAsync), but EnsureClusterNLB read it back synchronously right after
CreateLoadBalancer returned — losing the race and falling back to the synthetic
<lb>.elb.spinifex.local DNS name. That name is unresolvable (no Route53) and is
not in the apiserver serving-cert SAN, so kubectl against the cluster never
connects.
Split the elbv2 create: the public CreateLoadBalancer stays async (267.4 — an
inline multi-minute VM boot head-of-line-blocks the gateway responder), and a
new CreateLoadBalancerSync drives the data-plane launch inline and returns the
LB carrying its front-end address. The EKS cluster-create worker runs off the
gateway responder, so it uses the sync entry and reads the IP from the return
value instead of racing persisted state. No polling.
- elbv2: factor provisionLBDataPlane out of launchLBVMAsync; add CreateLoadBalancerSync
- eks: ensureClusterLB uses the sync entry, sets FrontendIP from the result,
drops the read-back; EnsureClusterNLB fails loud (ErrClusterNLBFrontendIPUnavailable)
when no front-end IP is allocated rather than shipping the DNS-name fallback
- eks: drop the DNS-name endpoint fallback; the endpoint is always the front-end IP
* feat: auto-wire IGW for internet-facing EKS cluster endpoint
An internet-facing cluster NLB allocates an external front-end IP and the
OVN dnat_and_snat for it, but the external switch + localnet + gateway LRP
that make that IP answerable on the wire are built only on IGW attach. The
control-plane NLB is a system instance, so ensure an IGW on cluster build
and reclaim it on teardown.
EnsureClusterIGW (idempotent, reuses a customer IGW untouched) runs before
EnsureClusterNLB when the endpoint is public; DeleteClusterIGW (ownership-
scoped) removes only the cluster-owned IGW in purgeClusterInfra, before the
VPC delete. New IGW dep wired from the daemon.
* fix(eks): tear down infra on failed cluster launch to stop LB EIP leak
A CreateCluster that fails during its background launch flipped the meta
to FAILED but performed no teardown, relying on a later same-name create
to reclaim and purge the infra. EKS cluster names are unique per create,
so that reclaim never fires: the internet-facing load-balancer VM keeps
running with its allocated+associated dnat_and_snat EIP indefinitely.
Over repeated create/delete cycles the EIP pool fills with allocations
bound to terminated instances, so a later allocation's advertised
endpoint and its live OVN NAT diverge onto different pool addresses.
- failClusterLaunch now runs a best-effort purgeClusterInfra on the
in-memory meta (freshest refs even when the failing step was the meta
write itself) before MarkClusterFailed. purgeClusterInfra is
idempotent, so a later reclaim or DeleteCluster re-purge is a no-op;
the FAILED meta is retained for observability.
- Daemon releaseSystemInstanceEIP extracted and called from both
TerminateSystemInstance and cleanupFailedSystemInstance, so a launch
that fails after AssociateAddress releases the eipService allocation
instead of leaking it (MarkFailed's ReleasePublicIP only knows the
externalIPAM path).
* feat: EKS managed control-plane VPC (AWS Set B) composed from EC2 APIs
CreateCluster now auto-builds the managed control-plane VPC under the
system account, the spinifex analogue of AWS EKS's hidden managed-account
VPC: a public subnet for the internet-facing NLB, private subnet(s) for the
k3s control-plane VM(s), and a NAT gateway giving the private CP egress for
image pulls. It is composed from the real EC2 VPC-family APIs
(CreateVpc/Subnet, IGW, RouteTable/Route, NatGateway), so per-subnet egress
gating is wired by the existing topology subscribers off the route-table
events those APIs publish — no direct OVN mutation.
This resolves the mulga-siv-292 root cause: the internet-facing NLB and the
private CP were crammed into one customer subnet with no IGW route, so both
were treated as private and dropped by the lr_in_policy 1100 rule, killing
the LB-VM return leg and lb-agent heartbeat. The CP only survived via a
per-instance 1200 reroute bandaid, now removed — the topology expresses
egress instead.
Placement is uniform single/multi-node (CP always in the private subnet).
The system LB VM heartbeats over the mgmt bridge (SystemGatewayURL) rather
than the WAN. Managed-VPC teardown reclaims NAT GW + EIP, route tables,
subnets, IGW, and the VPC in dependency order. Customer subnetIds (Set A /
worker side) remain consumed-not-created and are deferred to the worker
round.
* fix(eks): bake cluster-owner account into CP-VM EKS_ACCOUNT_ID
The managed control-plane VPC (Set B) launches the K3s control-plane VMs and
their ENIs under the system (infra) account, but the on-VM agents — bootstrap
publish, state report, token-review, and add-on fetch — must key on the
cluster-OWNER (customer) account that owns the cluster meta, namespaces, and
AccessEntry KV. Add ClusterAccountID to K3sServerInput and bake it into the
VM's EKS_ACCOUNT_ID, decoupling it from the infra AccountID the VM launches
under. Without this the VM reported state and fetched add-ons against the
system account, so the cluster never observed a healthy control plane (stuck
CREATING, null CA) and kubectl/add-on delivery timed out.
Also refresh the e2e suite doc comment to describe the Set B topology and the
data-plane subtests (kubectl + managed-addon delivery) that now run.
* fix(eks): explicitly delete addon objects on unstage
K3s does not delete the applied resources when an auto-deploy manifest file is
removed from the server manifests dir (documented behaviour). The addon-sync
GC pass relied on that, so after DeleteAddon the rendered manifest was dropped
but its Namespace/ConfigMap stayed standing — the e2e AddonDelivery GC wait
timed out. Copy the manifest aside, drop the original so K3s stops tracking it,
then kubectl delete the objects (--wait=false to avoid finalizer stalls).
* fix(eks): surface CREATING block reason at Info, not Debug
A cluster stuck in CREATING logged its per-tick bootstrap/health block
reason at Debug, invisible at the default level, so a create that ran to
the 15m timeout FAILED gave no diagnosable cause. Bump both reasons to
Info; they fire only during the CREATING window.
* feat(iam): round-trip AWS-managed policy ARNs for stock EKS tooling
Stock terraform-aws-eks / eksctl attach AWS-managed policies
(AmazonEKSClusterPolicy, AmazonEKSWorkerNodePolicy, AmazonEKS_CNI_Policy,
...) that Spinifex does not provision. AttachRolePolicy rejected them
NoSuchEntity and ListAttachedRolePolicies dropped them, breaking the
node/cluster role round-trip DescribeNodegroup relies on. Accept and
store managed ARNs opaquely (no backing document), deriving the policy
name from the ARN on read.
* feat(eks): echo cluster tags + store-only Tag/Untag/ListTags
DescribeCluster never returned create-time tags and ClusterMeta did not
store them, so a stock terraform-aws provider reconciling default_tags
saw perpetual drift and issued TagResource on every apply — which itself
returned NotImplemented. Store tags on the cluster record, echo them from
DescribeCluster, and back the three tag ops store-only against the record
(cluster ARNs only) so the default_tags round-trip is clean.
* fix(eks): persist nodegroup workers incrementally + boot reclaim sweep
launchWorkers only wrote the worker IDs to the record after the whole
launch loop, so a daemon crash mid-loop stranded already-launched workers
under a record that still showed empty InstanceIDs — nothing could reclaim
them. Persist each worker before launching the next, and add a boot-time
reclaim sweep (SpawnRegisteredReconcilers) that terminates recorded
workers for any nodegroup left CREATING by a crashed launcher, or
CREATE_FAILED still holding workers, settling it to CREATE_FAILED with
InstanceIDs cleared.
* fix: plumb query params into EKS gateway dispatcher
EKS_Request never passed the query string to per-action handlers, so
UntagResource (DELETE /tags/{arn}?tagKeys=k1&tagKeys=k2) saw an empty
TagKeys and silently no-op'd. Fold url.Values into the JSON body when
the body is empty so the handler unmarshal sees the keys.
* feat(iam): resolve AWS-managed EKS policy grants for authorization
Assumed-role/user authorization (checkPolicy -> Get{Role,User}Policies ->
EvaluateAccess) could not honour AWS-managed policy grants: Spinifex stores
no document for managed ARNs, so stock EKS roles whose permissions come
entirely from managed policies (node role: AmazonEKSWorkerNodePolicy +
AmazonEC2ContainerRegistryReadOnly + AmazonEKS_CNI_Policy) evaluated to an
empty grant set and were denied every call.
Add a builtin registry of well-known AWS-managed policy documents, resolved
via a shared resolveAttachedPolicy helper used by both GetRolePolicies and
GetUserPolicies. Unmodeled managed ARNs resolve to no grant (fail-closed
deny) rather than erroring the whole request; customer-managed ARNs still
fail closed when unresolvable.
* test(e2e): add targeted suite selector + shadow-mode CI step
select-suites maps a diff to the e2e suites it must run: changed files →
services (manifest path/additional_paths) → consumer closure over
depends_on vs publishes/subscribes → suites whose covers intersect.
Special cases force-all (infra/harness/workflow, ≥3 services, submodule
bumps) or a lone suite dir; empty diffs fall back to default suites. An
AST scan of each suite's harness.Ensure* call graph derives per-test
service coverage and emits an optional -test.run subset (RUN_PATTERN).
Wired into e2e.yml as a non-gating 'Select Suites (shadow)' job that
logs the computed selection to the run summary so predicted vs actual
can be compared before the selector drives SUITES.
* ci(e2e): add dispatch-only per-suite job blocks grouped by env
Restructure the looped e2e flow into env-grouped provision -> matrix
suites -> teardown blocks so each suite surfaces as its own node in the
Actions graph instead of a single green tick:
single_provision -> single_suite [single,cert,lb,eks] -> single_teardown
multi_provision -> multi_suite [multinode] -> multi_teardown
Provision resolves the VM/cluster and ships test binaries, emitting WAN
IP + ssh key path as job outputs; each matrix leg runs one suite over
SSH and uploads its own junit/analysis artifacts; teardown (if:always)
reuses the persisted tofu workspace on the same runner to destroy.
workflow_dispatch only -- runs alongside the gating e2e.yml for
validation and posts no commit status.
* test(e2e): add manifest drift lint (fixture-create + NATS subject)
Bead 5 of the targeted-suite-selection plan. Two ratcheted drift guards in
tests/e2e/manifest-lint, wired into make preflight and the spinifex.yml PR job:
- Fixture lint: AST-flags direct resource-create AWS calls
(c.EC2.RunInstances, c.ELBv2.CreateLoadBalancer, ...) in suite test files,
steering them through harness.Ensure* fixtures. ~104 pre-existing call sites
are captured in a checked-in baseline; the lint fails only on NEW creates.
Intentional create-path tests opt out with an inline // e2e:allow-create.
- Subject lint: extracts NATS subject literals from the natsSub{} registry and
literal Subscribe/Publish/Request args, cross-referenced wildcard-aware
against service-interfaces.yaml. Code subjects missing from the manifest fail
(ratcheted); manifest subjects with no static reference warn only (many are
produced dynamically). Fixed real drift surfaced on first run: declared the
five spinifex.cluster.shutdown.* phase subjects under spinifex_daemon.
manifest-check is now CI-enforced alongside manifest-lint (previously
preflight/local-only).
* feat: run-scoped e2e teardown sweep
Tag every Ensure* fixture resource with e2e:run=<run-id> when
SPINIFEX_E2E_RUN_ID is set, and add a teardown.test suite that sweeps
everything carrying that tag at end of run — reclaiming resources leaked
by a crashed suite, an interrupted run, or a persistent cluster.
- harness/sweep.go: tagRunResources (EC2 CreateTags) + tagRunELB (ELBv2
AddTags) + SweepRunResources in dependency order (LB, instances, NAT,
snapshots, volumes, images, SGs, subnets, key pairs). Blank run id
no-ops both tagging and sweeping, so the sweep can never touch an
untagged or foreign resource.
- harness/fixtures.go: stamp the run tag on each created resource.
- tests/e2e/teardown: thin suite that runs the sweep; skips without a
cluster or run id, logs the report, never fails the run.
- e2e-suites.yml: build + ship teardown.test, pass the run id into suite
legs, sweep on the VM in both teardown jobs before tofu destroy.
The gateway-claim verifier polled the SB Port_Binding for the distributed
gateway port LRP (gw-<vpc>), which is always chassis-less by OVN design — the
chassis claim lands on the chassisredirect port cr-gw-<vpc>. So
ensureGatewayClaimed always read "unclaimed", fired ovn-appctl
inc-engine/recompute every reconcile cycle, and logged a spurious "gateway SB
chassis claim did not converge" error after its 30s budget. The repeated forced
recomputes rebuilt the entire OVN flow table every cycle, intermittently
darkening freshly attached EIP datapaths and flaking the single-node e2e suite
(observed: every br-int EIP flow at duration~10s with n_packets=0).
Check the chassisredirect Port_Binding instead. The nudge now fires only on a
genuinely unclaimed redirect — the real reboot race #339 targets — rather than
on every healthy reconcile.
- topology: add GatewayChassisRedirectPort(vpcID) -> "cr-gw-<vpc>"
- reconcile/apply: pass the cr- port to ensureGatewayClaimed
- reconcile/host: doc the verifier checks the chassisredirect binding
- test: capture the queried port; assert the cr- port is checked, no nudge
TestMultinodeGuestSSH ran with t.Parallel(), so it resumed immediately
after the sequential bucket and probed trio[0] — which CrossNodeOps
stop/starts but only waits to reach state "running", not for sshd to
answer. The guest was often still booting, producing
kex_exchange_identification resets and 2m timeouts that flaked CI red.
Drop t.Parallel() and declare the test last in the sequential bucket so
it runs after CrossNodeOps + NodeFailure/Recovery + Spread have
restabilised the cluster, giving every trio member time to settle and
removing concurrent VM churn.
CreateLoadBalancer is async: InstanceID is only set when the LB VM's QEMU
process starts, ~600ms after the create burst. The reactive
updateStoredConfig calls from CreateListener/RegisterTargets all no-op
during that window (InstanceID empty), and nothing rebuilds config once
the VM launches. The lb-agent then heartbeats, gets an empty ConfigHash,
never sees it change, and probes zero backends -> e2e 0/2 healthy.
Build the data-plane config on the provisioning->active heartbeat
transition instead: that is the one point where InstanceID is guaranteed
set and the agent is alive to consume config. The InstanceID guard in
updateStoredConfig stays as a cheap no-data-plane skip, no longer the
primary gate.
DHCP-source external-pool leases are DHCPRELEASEd only on explicit
instance/EIP terminate. Manager.Stop() preserves leases for
adopt-on-reboot, so env teardown/reset kills vpcd and destroys the
per-AZ lease KV without releasing — stranding every live lease
(instance ENIs, EIP allocations, per-VPC LRP gateways) on the upstream
DHCP server until TTL. Across CI churn with fresh client-ids the
upstream scope fills and new launches get no public IP.
Add Manager.DrainAll + a vpc.dhcp.drain RPC and a standalone
'spx admin cluster drain-dhcp' command that releases every held lease.
Wired into both teardown paths (single-node teardown role and
multi-node cluster-teardown.yml) before services stop. Stop() is
unchanged, so normal restarts still adopt leases on reboot.
* fix(ec2): fail RunInstances when public-IP auto-assign fails
PrepareRunInstances swallowed an AllocateIP failure for a
MapPublicIpOnLaunch subnet (logged WARN, fell through), so the instance
booted running with no public IP. With no qemu-hostfwd fallback such an
instance is unreachable. Mirror the adjacent AddNAT-failure path: detach
and delete the auto-created ENI, deallocate capacity, drop the instance,
and surface InsufficientAddressCapacity.
* test(daemon): deterministically pick a fitting 2-vCPU type in capacity filter test
CapacityFilterRespectsAllocation selected the first 2-vCPU type by Go
map-iteration order, intermittently picking a type near the
schedulable-memory boundary the allocator then rejected (~3/5 failures in
isolation on small hosts). Select the smallest-memory 2-vCPU type from the
capacity-aware set so it always fits.
CreateLoadBalancer's PutLoadBalancer-failure path only released the name
claim, orphaning the ENIs and managed front-end SG already created. Extract
the ENI+SG+claim teardown into rollbackLBInfra and call it from both the
ENI-create-failure and PutLoadBalancer-failure paths.
CreateListener persisted the listener record before authorizing the NLB
managed-SG port and reloading config; a failure in either left an orphan
listener record (and, after a config failure, a dangling SG authorization).
Add rollbackListener to revoke the port (when already opened) and
cascade-delete the listener on every post-persist failure.
Also wait for the async worker launch in the two CreateNodegroup disk-size
tests so they observe the launch under the record-before-launch contract.
* fix: make platform create handlers async to stop responder starvation
EKS CreateCluster/CreateNodegroup and ELBv2 CreateLoadBalancer ran their
full VM/control-plane launch synchronously inside the NATS subscription
callback on the shared spinifex-workers queue. A multi-minute create
head-of-line blocked the worker, so system.LaunchInstance/TerminateInstance
found no responder within the gateway timeout and failed with
'nats: no responders available' (mulga-siv-267.4).
Each create now runs validation + the atomic name claim synchronously,
snapshots the CREATING/provisioning output, then spawns the slow launch on
a per-impl launchWG and returns immediately. Launch failures mark the
record FAILED/CREATE_FAILED so 267.3 reclaim makes a retry idempotent. The
service interfaces are shared by the gateway NATS clients and the daemon
impls, so the async split lives inside the impl method and leaves the
interface, handleNATSRequest, and all daemon handlers unchanged.
The host-pinned system.LaunchInstance/TerminateInstance responders now
dispatch each request on its own goroutine (systemDispatchWg) so a slow
boot, which must respond after it completes, no longer blocks the subject.
ELBv2 keeps ENI creation synchronous (cheap VPC calls that yield the
AZ/VpcId the create response needs); only the LB-VM boot goes async,
folding InstanceID/IPs back into the record for a later Describe.
* fix: keep node-targeted launch subscription live under capacity pressure
The node-targeted (unicast) system.LaunchInstance.{type}.{node} subject was
torn down whenever the node filled, same as the queue (anycast) subject. For
the queue subject that is correct — NATS reroutes the launch to a node with
room. For the node-targeted subject there is no other node: a committed spread
reservation (EKS HA control-plane leg) that targets the now-unsubscribed node
gets 'nats: no responders available' instead of a capacity decision.
Keep the node-targeted subscription live for any type the node can host;
enforce capacity at launch time via ResourceManager.allocate, which already
rejects an over-capacity launch atomically. A reserved launch always reaches a
responder and, when the slot is still good, succeeds.
launchWorkers built RunInstancesInput with no BlockDeviceMappings, so
parseVolumeParams applied the 4GB root-volume default and the nodegroup
DiskSize was silently ignored. Wire rec.DiskSize into a BlockDeviceMapping
when set.
Run 27254808154 showed TestRebootResilience fail with ovn-controller on
the live chassis claiming NO local Port_Bindings post-reboot: ENIs stuck
on a dead chassis UUID, cr-gw chassis empty, NAT external_mac empty —
while NB intent was correct and taps carried the right iface-id. The SB
Chassis _uuid churned across the reboot (old UUID gone, new one created)
while the SB DB persisted, orphaning every claim.
The current artifacts cannot pin WHY the new chassis won't reclaim: the
ovn-controller vlog lives under /var/log/ovn, not journald, so it is
absent from the systemd-journal capture.
Add:
- phase5Snapshot: capture chassis identity PRE-reboot (system-id, SB
Chassis/Chassis_Private, get-connection) to ovn-pre-chassis.txt as the
churn baseline.
- dumpOVNClaimState: at post-reboot failure, capture the same chassis
state POST-reboot plus ovn-controller connection-status/debug, the
gw/eni Port_Binding claim detail, ovn-central-vs-ovn-controller boot
ordering, and tails of ovn-controller.log / ovn-northd.log →
diag-ovn-claim.txt.
Disambiguates chassis-identity churn from an SB/NB startup-ordering race
on the next reboot-suite run. Diagnostic only; no behavior change.
* feat: atomic create idempotency across EKS, ELBv2 and EC2 ClientToken
Every create handler took a non-atomic Get-check then a blind Put, so a
gateway/NATS retry that spawned a second concurrent handler could pass the
check in both and double-launch + leak VMs (267 failure-mode C).
Make each create claim its identity key atomically before any launching work:
- EKS CreateCluster: claim the cluster meta via kv.Create; a prior FAILED
attempt is reclaimed via a CREATING CAS so exactly one concurrent retry wins.
purgeClusterInfra gains a deleteMeta flag so reclaim purges infra while
keeping its claim key.
- EKS CreateNodegroup: claim the nodegroup record before SG provisioning or
worker launch; duplicates return ResourceInUse without provisioning.
- ELBv2 CreateLoadBalancer: account-scoped name claim (lbname.<acct>.<name>)
taken before ENI/VM work, released on every failure path and in
DeleteLoadBalancer; a name claim whose owner lbID has no live record is a
crashed-create orphan and is reclaimed.
- EC2 RunInstances: ClientToken dedup over a dedicated TTL bucket with a
two-phase claim (in-flight -> done) and reservation replay; a token reused
with different params returns IdempotentParameterMismatch.
Race tests under -race assert exactly one owner and a single launch for
concurrent same-name creates on all paths.
* test: cover ClientToken idempotency; extract orchestration for testability
Extract the claim->replay->launch->finalize/abort flow out of RunInstances
into runInstancesWithClientToken(store, ..., launch func) so the idempotency
path is unit-testable without a live gateway/NATS. RunInstances now just
resolves the store, hashes params, and passes a launch closure.
Make the in-flight poll timeout/step package vars so a duplicate-caller
wait-timeout is deterministically testable.
Adds tests: replay-skips-launch, owner-launches-once-then-replay,
launch-failure-aborts, param-mismatch maps to AWS error, getClientTokenStore
binds once, and in-flight wait-timeout.
After a host reboot the Northbound gateway_chassis intent recovers correctly
(SetGatewayChassis re-asserts it idempotently), but ovn-controller can leave the
Southbound Port_Binding for the gateway router port unclaimed (chassis == []).
With no claimed binding it installs no logical->physical flows for the centralised
gateway redirect, so every floating IP on the VPC is dark: ofproto/trace drops the
FIP ingress at br-int table 0, host->FIP is 100% packet loss, and the ALB lb-agent
heartbeat times out until the 5-min health window expires (TestRebootResilience).
rebindGatewayChassis only asserted NB intent. Now, after SetGatewayChassis, the
reconciler verifies the SB Port_Binding has a non-empty chassis and, if not, nudges
the local ovn-controller to recompute (inc-engine/recompute), polling until the
claim converges or a bounded deadline elapses — logging an error rather than
blocking reconcile. SB read + recompute are exec-only (ovn-sbctl/ovn-appctl) and
live in network/host behind a reconcile.GatewayClaimVerifier interface, wired in
vpcd; reconcile stays NB-pure and mockable. No NB object create/delete (ADR-0006 S5).
Verified by run 27251769281's diag-ovn-flows.txt (SB chassis empty + table-0 drop).
Validation completes on the next reboot-suite nightly (single-node self-heals too
fast to reproduce locally).
* test(reboot): capture heartbeat-path diagnostics on post-reboot health timeout
When post-reboot ALB targets fail to go healthy, the lb-agent's heartbeat to the
gateway times out with 'context deadline exceeded while awaiting headers' — a
symptom that alone cannot distinguish a network-level drop from an app/gateway
stall. Replace the fatal WaitForTargetsHealthy with a non-fatal poll that, on
timeout, tcpdumps br-wan for the heartbeat 5-tuple + ARP for the ALB floating IP
and pulls the lb-agent console + awsgw journal, then fails. Disambiguates the
layer before any fix (mulga-siv-265).
* test(reboot): re-aim heartbeat capture + add OVN dataplane flow-dump
Run 27250238163 proved the post-reboot failure is network-level (all 3 floating
IPs 100% packet-loss from host) but NOT neigh/NAT priming — neigh is primed and
NAT rows are present, only br-int forwarding is dead. The prior capture watched
br-wan for the ALB floating IP, while the real lb-agent heartbeat posts to the
gateway port on the host's advertise interface, so the L4 classifier could never
see the heartbeat and defaulted regardless.
- heartbeat tcpdump now watches the gateway port on `any` (real path) plus a
secondary br-wan floating-IP capture for north-south; classifier keys off
gateway-port L4 presence.
- new dumpOVNDataplane captures ovs-vsctl show, ovs-ofctl dump-flows br-int,
ovn-sbctl chassis + Port_Binding, dnat_and_snat NAT rows, host neigh, and a
host->floating-IP ping + ofproto/trace, to pin the flow-restore step.
- failure path dumps diag-ovn-flows.txt alongside diag-heartbeat.txt.
Test-only, build-tagged e2e; no production code.
CreateCluster set the control-plane VM/ENI/spread-group refs into the cluster
meta but first persisted them to KV only after allocateClusterEgressIP. A
failure between placement and that persist left the live sys.medium CP VMs
unrecorded, so neither DeleteCluster nor the FAILED-cluster reclaim could reach
them and they leaked, reserving host capacity.
Persist the meta immediately after placeControlPlane returns. The reclaim path
keys off the persisted ControlPlaneNodes + ControlPlaneSpreadGroup, so this also
frees reserved spread capacity on teardown.
A retried DeleteCluster (or the egress-delete cascade) can release the
hidden-pool allocation before the release step runs, so ReleaseAddress
returns InvalidAllocationID.NotFound. That was treated as a fatal teardown
error, so the cluster wedged in DELETING forever and its KV meta plus
downstream resources leaked. Swallow the NotFound variants as idempotent
success — mirroring the 165.17 ENI-teardown fix — so the SG + KV sweep
proceeds. A genuine release failure still wedges DELETING for retry.
* feat(eks): node-targetable control-plane VM (HA spread transport prereq)
Groundwork for HA control-plane placement (mulga-siv-231.7.1): make the k3s
control-plane VM placeable on a specific Spinifex host.
- Add a sys.medium instance type (2 vCPU / 4 GB) and boot the control-plane VM
as it instead of t3.medium. A sys.* type keeps the CP VM out of customer
DescribeInstanceTypes and makes the daemon register the node-targeted
system.LaunchInstance.{type}.{nodeID} subject (only sys.* types get it).
- Add K3sServerInput.TargetNodeID and a LaunchSystemInstanceOnNode method on
the launcher: empty/local node launches in-process (unchanged single-CP
path); any other node dispatches over the node-targeted subject, where the
receiving daemon runs the launch and owns the VM.
The spread orchestrator (reserve N hosts, launch per host, verify, rollback,
<3-host fallback) lands separately; CreateCluster still launches locally
(TargetNodeID empty), so behaviour is unchanged.
* feat(eks): HA control-plane spread orchestrator
Place N=3 k3s server VMs across distinct Spinifex hosts via a per-cluster
spread placement group, with all-or-nothing rollback and single-CP fallback.
- placeControlPlane: capacity fan-out (HostScheduler) -> >=3-host gate ->
ensure spread group (system account) -> ReserveSpreadNodes(Min=Max=3) ->
parallel per-node LaunchK3sServerVM(TargetNodeID) -> node-vms placement
verify -> FinalizeSpreadInstances. Any launch/verify/finalize failure
terminates launched VMs, releases the reservation, drops the group.
- <3 schedulable hosts or a lost reservation race falls back to a single
local control-plane VM (prior behaviour).
- ClusterMeta gains ControlPlaneNodes[] + ControlPlaneSpreadGroup; [0] is
mirrored into the scalar ControlPlane* fields for back-compat. DeleteCluster
tears down every CP VM and the spread group.
- Wire PlacementGroup + Scheduler into EKSServiceDeps + orchestration-deps gate.
* fix(eks): size HA control-plane spread on raw host headroom
SchedulableHosts matched the CP type by name in node.status's per-type
capacity list, but that list omits sys.* types (system types are hidden
from customers). The CP VM is sys.medium, so the match never hit and
SchedulableHosts returned 0 on every node — placeControlPlane always fell
back to a single CP, making the >=3-host spread unreachable in production.
System VMs consume host vCPU/memory like any guest, so size the fit against
each node's raw schedulable headroom (Total - Reserved - Alloc) and the
type's footprint instead. Adds instancetypes.SpecForSystemType for the
sys.* vCPU/memory spec and a pure nodeFitsSystemInstance helper.
* feat(eks): k3s server-join role for HA control-plane quorum
Servers 2..N now boot in join mode: server: https://<first-server>:6443 +
a shared cluster token, without cluster-init, so they join the first server's
embedded-etcd quorum instead of each cluster-initing an independent datastore.
- K3sServerInput gains JoinToken (shared cluster secret, set on every server)
and ServerURL (set => join server, no cluster-init, server-join role).
- buildK3sUserData branches cluster-init vs server:, emits token:, and selects
SPINIFEX_K3S_ROLE server vs server-join.
- launchControlPlaneSpread launches nodes[0] first (cluster-init), derives the
join URL from its ENI IP, then launches the rest in parallel as join servers;
a first-server failure skips the joins and rolls back.
- CreateCluster mints a per-cluster crypto-random token (GenerateK3sClusterToken)
threaded into the placement template.
- eks-node-role.sh adds a server-join case: token-webhook + k3s + state-report,
but not k3s-first-boot (the first server already publishes the bootstrap
artifacts; a join re-publish only races the bootstrap bus).
* fix: open etcd-peer/kubelet ingress on EKS control-plane SG for HA join
Join servers (231.7.2) register over the supervisor port 6443 but their
embedded etcd could not peer with the quorum: the control-plane SG only
admitted 6443 from the VPC CIDR, omitting etcd client+peer 2379-2380 and
kubelet 10250 between control-plane nodes. A single cluster-init server never
exercises those ports, so the gap stayed hidden until multi-server HA — the
join registered but its etcd never replicated the raft log, leaving the node
perpetually NotReady.
Add EnsureControlPlaneHAIngress with self-referencing (cpSG->cpSG) rules for
etcd 2379-2380 and kubelet 10250, called from CreateCluster after
EnsureControlPlaneIngress. Verified live on a 3-node cluster: join servers
reach Ready and form a 3-member etcd quorum that survives loss of the original
first server.
* feat: register all control-plane ENI IPs as EKS cluster NLB targets
Register every HA control-plane server ENI IP in the cluster NLB target
group, not just the primary. The TG already health-checks :6443, so a
dead member drains and kubectl/apiserver traffic fails over to surviving
etcd-quorum members. Add RegisterClusterTargets batch helper; the single
RegisterClusterTarget now delegates to it.
Associated Elastic IPs lost external connectivity after a host reboot:
vpcd's reconcile re-applied the dnat_and_snat rule without the ENI MAC,
collapsing it to the centralised shape (external_mac/logical_port unset),
and the daemon's relaunch republish only fired for auto-assigned public
IPs, never for store-backed EIPs.
- EIPRecord gains MacAddress, captured at AssociateAddress and cleared at
DisassociateAddress, so reconcile can re-apply the distributed shape
without re-querying the ENI.
- loadEIPs feeds rec.MacAddress into the reconcile spec; policy.AddEIP
self-heals a stale centralised row when given a MAC.
- AssociatedPublicIPForInstance lets the relaunch hook resolve an
associated EIP from the store and republish vpc.add-nat with the MAC.
* feat(eks): node-targetable control-plane VM (HA spread transport prereq)
Groundwork for HA control-plane placement (mulga-siv-231.7.1): make the k3s
control-plane VM placeable on a specific Spinifex host.
- Add a sys.medium instance type (2 vCPU / 4 GB) and boot the control-plane VM
as it instead of t3.medium. A sys.* type keeps the CP VM out of customer
DescribeInstanceTypes and makes the daemon register the node-targeted
system.LaunchInstance.{type}.{nodeID} subject (only sys.* types get it).
- Add K3sServerInput.TargetNodeID and a LaunchSystemInstanceOnNode method on
the launcher: empty/local node launches in-process (unchanged single-CP
path); any other node dispatches over the node-targeted subject, where the
receiving daemon runs the launch and owns the VM.
The spread orchestrator (reserve N hosts, launch per host, verify, rollback,
<3-host fallback) lands separately; CreateCluster still launches locally
(TargetNodeID empty), so behaviour is unchanged.
* feat(eks): HA control-plane spread orchestrator
Place N=3 k3s server VMs across distinct Spinifex hosts via a per-cluster
spread placement group, with all-or-nothing rollback and single-CP fallback.
- placeControlPlane: capacity fan-out (HostScheduler) -> >=3-host gate ->
ensure spread group (system account) -> ReserveSpreadNodes(Min=Max=3) ->
parallel per-node LaunchK3sServerVM(TargetNodeID) -> node-vms placement
verify -> FinalizeSpreadInstances. Any launch/verify/finalize failure
terminates launched VMs, releases the reservation, drops the group.
- <3 schedulable hosts or a lost reservation race falls back to a single
local control-plane VM (prior behaviour).
- ClusterMeta gains ControlPlaneNodes[] + ControlPlaneSpreadGroup; [0] is
mirrored into the scalar ControlPlane* fields for back-compat. DeleteCluster
tears down every CP VM and the spread group.
- Wire PlacementGroup + Scheduler into EKSServiceDeps + orchestration-deps gate.
* fix(eks): size HA control-plane spread on raw host headroom
SchedulableHosts matched the CP type by name in node.status's per-type
capacity list, but that list omits sys.* types (system types are hidden
from customers). The CP VM is sys.medium, so the match never hit and
SchedulableHosts returned 0 on every node — placeControlPlane always fell
back to a single CP, making the >=3-host spread unreachable in production.
System VMs consume host vCPU/memory like any guest, so size the fit against
each node's raw schedulable headroom (Total - Reserved - Alloc) and the
type's footprint instead. Adds instancetypes.SpecForSystemType for the
sys.* vCPU/memory spec and a pure nodeFitsSystemInstance helper.
* fix: strip duplicate lb- prefix from haproxy socket and cert paths
LoadBalancerID, ListenerID and friends already carry their type prefix
(lb-/lst-), so the stats-socket, frontend cert, and HAProxyManager
config/pid filename templates double-prefixed: lb-lb-<id>.sock,
lb-lb-<id>-<lst>.pem, alb-lb-<id>.cfg.
The socket path is computed independently on both sides (gateway render
in buildHAProxyConfig, agent poll in lbagent.New); any divergence in the
lb- prefix convention between the two leaves the agent polling a socket
HAProxy never bound, spamming 'HAProxy stats unavailable: no such file'
while the data plane serves fine. Drop the redundant literal prefix so
both sides resolve to <id>.sock and the alb-/nginx cert paths match the
delivered config.
* refactor: remove unused HAProxyManager
HAProxyManager was never constructed (NewHAProxyManager had no callers).
The live ALB data plane runs in the lb-agent via reloadHAProxy with the
config generated daemon-side and shipped; this per-daemon process manager
was dead code. Remove the type, its methods, tests, and now-unused
imports (log/slog, os/exec, sync, syscall).
* ci(e2e-nightly): run Go harness suites instead of bash run-*.sh
Swap nightly cells 1-3 (single), 8-10 (real-multi) and 18 (reboot) to the
compiled <suite>.test binaries, mirroring e2e.yml: SPINIFEX_E2E=1 with
SPINIFEX_MODE single|multinode, per-suite logs to ARTIFACT_DIR, then
go-junit-report + e2e-analyze + upload-artifact. Adds a spinifex checkout for
the composite actions and pulls /run/spinifex/console-*.log so LB microVM
(lb-agent) state is captured on failure. Cells 17 (tofu) and 20 (baremetal)
stay on bash — no Go suite exists.
* ci(e2e-nightly): cell-17 RCA analytics + cell-20 cosmetic alignment
run-tofu-examples-e2e.sh now emits go test -v markers per workbook so the
cell-17 step's tee'd test-tofu.log flows through the same go-junit-report ->
e2e-analyze -> upload-artifact chain as the Go suites, producing junit-tofu.xml
and an analysis bundle for RCA. Align e2e-baremetal.yml (cell 20) ARTIFACT_DIR,
analyze title, and artifact name with the matrix cells so CI-RCA discovers its
bundle uniformly.
* ci(e2e-nightly): wire host journal + VM consoles into RCA bundles
Collect now emits the canonical spinifex-journal.log (concat of all nodes)
that the e2e-analyze bundler slices into each failure's journal.log, so
elbv2-service/vpcd/daemon lines in the failure window travel with the bundle
(previously the journal uploaded as spinifex-journal-node*.log, a name the
bundler never reads). Rename the console capture to vm-console-node* and note
it also carries guest cloud-init/sshd consoles, covering the cell-3 internal-LB,
cell-18 reboot-datapath, and cell-9/10 guest-ssh blind spots.
* ci(e2e): self-heal runner /dev + --one-file-system on workspace clean
Clean Workspace now detaches any leaked iso-builder rootfs mount, recreates
/dev/null if a prior build clobbered it, and uses rm --one-file-system so the
clean can't descend into a still-mounted devtmpfs and delete host device nodes.
Guards both the matrix cells and the bare-metal job against the host-wide
/dev/null wipe that fails go build with 'open /dev/null: no such file or
directory'.
The LB VM's external-pool floating-IP NAT (dnat_and_snat external_ip -> LB
VPC IP) was reaped only conditionally on teardown: DeleteNetworkInterface
skips EIP-owned addresses (defers to the EIP lifecycle), and that lifecycle's
reap runs in the async TerminateSystemInstance goroutine gated on the EIP
association. So repeated create/delete cycles accumulated stale rules that
collapsed onto the reused LB VIP, making OVN SNAT/un-DNAT ambiguous and
dropping the lb-agent heartbeat return path.
DeleteLoadBalancer now publishes vpc.delete-nat deterministically from the LB
record (external=AvailZones public IP, logical=VPCIP). DeleteEIP matches on
external+logical and is idempotent, so redundant reaps from other paths are
harmless. Internal LBs (no floating IP) are a no-op.
* fix: re-prime EIP reachability on AddEIP idempotent-skip path
AddEIP early-returned on the idempotent-skip path (OVN DNAT row already
matches) before emitting GARP and priming the host neighbour. Stop->start
and daemon reboot-recovery re-attach the same EIP with an unchanged row, so
the host neigh entry / OVS datapath never got re-primed and the EIP/ALB-VIP
stayed dark until the kernel ARP entry timed out (60-300s). inject-garp is
unsupported on some chassis, making neighPrime the only working remediation
— exactly the step the skip bypassed.
Extract garp + neighPrime/flush into primeReachability and run it on the
skip path as well as after a fresh AddNAT. The FlowsBarrier still fires only
on row churn. Resolves cell-9/10 TestMultinodeGuestSSH and cell-18
TestRebootResilience nightly failures.
* refactor: remove dead inject-garp GARP emitter from EIP path
ovn-appctl inject-garp is not a valid command on the OVN build Spinifex
deploys and tests against — every AddEIP GARP attempt failed with
'inject-garp is not a valid command' and fell through to the host neigh
prime/flush, which is the only remediation that works. The GARP hook was a
guaranteed-failing shell-out plus a misleading WARN on every EIP attach.
Delete host.InjectGARP, the GARPEmitter type, WithGARPEmitter option, the
garp field + its m.garp() call in primeReachability, and the vpcd
injectGARPEmitter wiring. host.ReplaceNeigh (distributed) and host.FlushNeigh
(centralised) already cover both NAT shapes and become the sole EIP
reachability path. Reword the neigh hook docs that framed themselves as an
inject-garp-independent fallback. Prune GARP-only tests; move the shared
recordingRunner helper into neigh_test.go.
* feat(daemon): live MemAvailable admission gate + fixed-services reserve scope
Admission previously charged only guest -m against a flat reserve, blind to the
co-located storage VMs, nbdkit, and daemon stack — so it greenlit launches that
overcommit real RAM and OOM-kill guests on memory-tight hosts.
Add a second admission gate that clamps the accounting-derived launch count by
live /proc/meminfo MemAvailable (which already nets reclaimable cache and sees
every untracked consumer): admitted = min(accounting, (MemAvailable-reserve)/
guestMem). Reads live per admission, fails open on a /proc read error so a
transient fault can't wedge launches, and is disabled by SPINIFEX_ADMISSION_LIVE_MEM=0.
Re-scope the host reserve to the fixed infrastructure tier only; per-instance
nbdkit is no longer expected to fit inside it (explicit per-instance charge is a
follow-up; the live gate covers it implicitly in the meantime).
Pure helpers (liveMemCount, parseMemAvailableKB, liveMemAdmissionEnabled) are
unit-tested; liveMemGate covered via an injected reader.
* feat(daemon): charge nbdkit per-instance in admission accounting
An instance's admission memory charge is now its guest -m plus the nbdkit
processes backing its volumes (one per attached volume), not guest -m alone.
The static reserve no longer has to absorb per-instance nbdkit, which was
unsizable and the root of the host-memory-overcommit OOM.
Per-volume charge is measured per class (main 768 MiB: 128 cache + plugin
working set + in-flight I/O; aux 96 MiB: process floor, 0 cache), env-tunable
via SPINIFEX_NBDKIT_MAIN_MIB / SPINIFEX_NBDKIT_AUX_MIB. Default layout is one
main + two aux volumes (root + cloudinit + efi) ~= 0.94 GB/instance. Applied at
every accounting site through instanceMemChargeMiB so charge and release never
drift; the live MemAvailable gate backstops block-device-mapping volumes the
static count misses.
* feat(vm): three-tier guest OOM ladder + spinifex-guests.slice
Replace the flat oom_score_adj=500 on every guest QEMU with a ManagedBy-derived
ladder: user guests (ManagedBy=='') stay +500 and are reaped first under host
memory pressure; system instances (ELBv2 LB-VMs, EKS control-plane nodes,
ManagedBy!='') score 0, ranking above user guests but below the infra tier
(-500). Killing a system instance degrades the data path or cluster control
plane for many user VMs, so it must outlive them.
Add spinifex-guests.slice as a sibling of spinifex-system under spinifex.slice —
the parent tier for the per-instance leaves that land with RG-7. Guest QEMU
placement into the slice is deferred to the per-instance work.
* refactor(systemd): lean unit comments to RG-n tags + RG-9/RG-11 invariants
Strip the multi-paragraph rationale novels from spinifex-daemon.service (48->7
comment lines), spinifex-vpcd.service (29->8) and spinifex.slice (15->5),
replacing them with settings + terse # RG-n clause tags. Remove plan-doc, bead
and CI-run references from comments (project policy). Directives are byte-for-byte
unchanged — comment-only.
Add RG-9/RG-11 invariant tests asserting per-tier confinement (locked-down
storage/control tier drops all caps + NoNewPrivileges=yes; daemon keeps exactly
CAP_SYS_ADMIN+CAP_DAC_OVERRIDE and the four-entry DeviceAllow; vpcd is the
documented network-tier exception) and the lean-comment budget + no-plan-ref rule.
Project internal StateError onto AWS stopped (code 80) in DescribeInstances
so SDK/UI clients no longer break on the non-AWS 'error' state label.
Route start-instances through the per-instance ec2.cmd topic first so a
crash-pinned StateError instance is restarted by its live owner node, whose
StartInstance handler now accepts StateError (Manager.Start transitions
error->pending before relaunch). Fall back to the ec2.start queue-group path
on ErrNoResponders for genuinely stopped instances in the shared KV.
* ci(eks-ami): build on self-hosted ci-single-1 runner
GitHub-hosted ubuntu-24.04 cannot run the libguestfs appliance network:
passt fails (PID file denied under its AppArmor confine / privilege drop)
and the SLIRP fallback has no outbound, so virt-customize --network dies.
The self-hosted ci-single-1 runner has /dev/kvm and real networking, so
the existing libguestfs build works unmodified.
* fix(build-system-image): add --no-cache, force fresh build on publish
The <10min cached-raw skip avoids duplicate work for concurrent e2e
jobs, but on the persistent publish runner it could republish a stale
image from a prior or concurrent run. Add --no-cache to bypass the skip
and have publish-system-image.sh pass it on every build.
* ci(eks-ami): target ci-single label (no ci-single-1 label exists)
The self-hosted runners are named ci-single/-2/-3 but all share the
single label 'ci-single'; runs-on matches labels, not names, so
ci-single-1 never schedules.
* ci(eks-ami): disable Go cache on self-hosted runner
setup-go's cache-save (Post Setup Go) re-uploaded the Go cache to the
Actions backend on every run, taking ~2.5min vs the 36s build. The
self-hosted runner already keeps the cache on local disk, so cache:false.
* feat(systemd): bound spinifex to a resource-limited slice
Add a spinifex.slice parent ceiling (MemoryHigh=80%, MemoryMax=90%,
MemorySwapMax=0) with a spinifex-system.slice child holding the seven
control/infra services, so a runaway hits the slice instead of the host.
A system.slice drop-in (MemoryMin=1G, CPUWeight=300) reserves RAM and CPU
for sshd/journald/the operator, keeping the box reachable when spinifex is
maxed.
The drop-in is generated in setup.sh install_systemd() rather than shipped
as a staged file, since the packaging globs flatten the systemd/ dir; this
mirrors the existing sshd-keygen drop-in pattern. The two .slice files flow
through every staging path unchanged.
Within the single effective cgroup, a slice MemoryMax OOM picks the highest
oom_score victim — guest QEMU (+500) before any service (-500) — so a slice
OOM degrades by killing one guest while the daemon, storage tier, NATS, and
the host all survive.
* feat(setup): provision 8G host swapfile + vm.swappiness=10
Add a setup_swap stage that creates an 8G /swapfile, enables it, persists it
to fstab, and writes vm.swappiness=10 so the kernel only swaps under real
pressure. Backs spinifex.slice MemorySwapMax so the slice can spill to swap
for graceful degradation instead of OOMing a guest. Idempotent; skipped under
ISO_BUILD (firstboot provisions swap on ISO hosts). Reverses the prior swap=0
assumption.
* Revert "fix(network): prime host neigh with gateway port MAC on centralised EIP attach"
This reverts commit f7446c69c6.
* fix(network): round-robin external pool cursor so a released IP is not reused instantly
The static external pool handed out the lowest free address and freed IPs
immediately on release, so a just-released EIP was reassigned within ~0.6s. A
recycled EIP front-end is unreachable from the same host: the kernel still
holds the prior owner's ARP binding (or a negative cache after the neigh
flush) and OVN does not answer the re-ARP in time, so the neighbour entry
settles to FAILED (EHOSTUNREACH). This surfaced as EKS TestEKS/KubectlGetNodes
failing in single-node CI once #315 routed kubectl through the cluster NLB's
external EIP. Dev never hit it — it uses DHCP-random addresses.
Add a persisted per-pool cursor: allocation resumes one past the last address
handed out and wraps at RangeEnd, so a freed IP is not reused until the cursor
cycles the whole range. Empty cursor (fresh pool / older KV record) starts at
RangeStart, preserving lowest-free order for the first cycle — backward
compatible, no migration.
* ci: drop e2e-eks from default e2e suites
EKS data-plane subtest (TestEKS/KubectlGetNodes) is a persistent flake on
single-node CI — a freshly-attached centralised EIP is intermittently
unreachable from the host (no route to host to the cluster endpoint), which
fails the whole E2E run and blocks unrelated suites. Remove e2e-eks from the
default SUITES so PR/push runs no longer gate on it. The e2e-eks target and
the workflow_dispatch suites input are unchanged, so EKS stays runnable on
demand and via make e2e-all.
Centralised-mode EIP attach only flushed the host neighbour entry and relied
on OVN re-answering the host's ARP. On OVN builds whose ovn-controller lacks
inject-garp, no node answers a same-chassis recycled IP's re-ARP, so the entry
decays to FAILED and the external IP is unreachable ("no route to host"). This
sank the EKS e2e suite: the cluster NLB front-end EIP never resolved on br-wan,
so kubectl could not reach the published apiserver endpoint.
Mirror the distributed-mode prime: resolve the VPC gateway router port MAC (the
L2 owner of a centralised EIP on the WAN segment) and program the host
neighbour entry directly, falling back to a flush only when no MAC resolves.
* feat: un-stub EKS Addon API with managed-addon state machine
Wire the 6 EKS Addon ops (List/DescribeVersions/Create/Describe/Update/
Delete) to a per-cluster managed-addon state machine in the per-account KV.
- addon_catalog.go: static air-gapped catalog (aws-load-balancer-controller,
coredns) with version validation.
- addon_state.go: AddonRecord + AddonStatus, KV CRUD + CAS helpers; list skips
staged-manifest sub-keys; delete sweeps record + manifest.
- addon_installer.go: transport-agnostic AddonInstaller; stagingInstaller
stages a manifest descriptor in KV for the (future) VM-side delivery slice.
- addons.go: op impls + AWS-shape converters; installer failure flips
CREATE_FAILED. Status stays CREATING until delivery confirms ACTIVE (private
apiserver is host-unreachable, so delivery must be VM-side).
- service_impl.go: drop notImpl stubs; add AddonInstaller dep (nil-defaults to
staging installer).
DeleteClusterPrefix already sweeps clusters/{name}/ so addon keys clean up on
cluster delete.
* fix: route ListAddons at GET /clusters/{name}/addons
ListAddons was wired at GET /addons with no cluster param, so the AWS CLI's
GET /clusters/{name}/addons hit no route (InvalidActionException) and the
wrapper couldn't pass a cluster to the service. Match the real EKS path and
thread the cluster name through.
Recycled WAN EIPs stayed unreachable from the host (kubectl to a cluster
endpoint: "no route to host", host neigh FAILED) on OVN builds whose
ovn-controller lacks the inject-garp appctl. OVN advertises a recycled
external IP's new MAC only via the GARP it emits on LSP binding-chassis
migration; a same-chassis rebind emits none, and the neigh flush added for
recycle reachability then waits on an ARP reply no node sends.
In distributed mode the external_mac is known at attach time, so program the
host neighbour entry directly (ip neigh replace <ip> lladdr <mac> nud
reachable) instead of flushing. Inbound traffic refreshes the entry from
there. Centralized attach (no external_mac) and all detaches keep the flush.
Adds host.ReplaceNeigh + policy.NeighPrimer hook wired in vpcd alongside the
existing flusher.
Workloads need a working ingress/servicelb path before the
aws-load-balancer-controller add-on lands. Flip builtinIngressEnabled to
default ON; opt out per-cluster for AWS parity with the CreateCluster tag
spinifex.io/managed-ingress=false. Interim divergence — revert to
default-off when the controller add-on ships.
Recycled WAN external IPs stayed unreachable for the kernel ARP timeout
(60-300s) when ovn-controller lacks the inject-garp appctl: the GARP refresh
after AddEIP is then a no-op, leaving the host neighbour cache pointed at the
prior owner's MAC.
Add host.FlushNeigh (ip neigh flush) and a policy.NeighFlusher hook wired
into AddEIP and DeleteEIP, so a recycled IP re-resolves L2 against the current
owner independent of inject-garp. DeleteEIP now takes the external IP (the
vpc.delete-nat event already carries it) to flush on release.
* feat(eks): public/private endpoint access drives NLB scheme + IP:port reachability
Parse resourcesVpcConfig.endpointPublicAccess/endpointPrivateAccess/
publicAccessCidrs (defaults public=true, private=false, cidrs=0.0.0.0/0) and
reject the both-disabled combination. Public access provisions an
internet-facing NLB and publishes its external frontend IP as the cluster
endpoint; private access keeps the NLB internal and uses the VPC frontend IP.
Inject the resolved endpoint IP into the k3s apiserver tls-san so the
published IP:443 endpoint serves a cert valid for verified TLS, and surface
the access fields in DescribeCluster. The e2e suite now connects via the
verified endpoint with TLS verification on, dropping the insecure mgmt-IP
shim.
* fix(eks): authorize NLB->apiserver ingress on control-plane SG
The cluster control-plane security group was created with empty ingress, so
the NLB's backing LB VM could not reach the apiserver on 6443 from inside the
VPC. The target group stayed unhealthy and the published endpoint never
served. Authorize tcp/6443 on the control-plane SG from the VPC CIDR after the
SGs are created; public exposure remains gated at the NLB front-end by
publicAccessCidrs, so this only opens the in-VPC LB->apiserver hop.
Extend SubnetVPCResolver with GetVPCCIDR so the service can resolve the VPC
CIDR for the rule.
* feat(eks): thread publicAccessCidrs to NLB SetLoadBalancerIngressCIDRs
Wire the cluster's publicAccessCidrs (parsed from CreateCluster
ResourcesVpcConfig) through EnsureClusterNLB to the ELBv2
SetLoadBalancerIngressCIDRs override so a public cluster endpoint can be
narrowed below the wide-open scheme default. The override only fires for
an internet-facing NLB whose CIDRs restrict below 0.0.0.0/0; internal
NLBs and the default keep the LB's out-of-the-box ingress.
NLB ENIs fell back to the VPC default security group, whose intra-SG-only
ingress made the OVN ACL drop inbound listener traffic at the front-end VIP
even with a healthy target group. NLBs reject customer SGs, so mint an internal
managed SG per network LB instead: attach it to every LB ENI at create, open
each listener port on it from the resolved client CIDRs at CreateListener,
close ports on DeleteListener, and delete the SG on DeleteLoadBalancer.
Default ingress CIDRs are scheme-based (internet-facing: 0.0.0.0/0; internal:
the VPC CIDR). SetLoadBalancerIngressCIDRs overrides them and re-authorizes
existing listeners, providing the hook for EKS publicAccessCidrs. ALBs are
unchanged.
* feat(eks): publish eks-node AMI to R2 + spx catalogue entry
Add a spinifex-eks-node catalogue entry so operators install the EKS
node image with a single command:
spx admin images import --name spinifex-eks-node
The entry pulls the Alpine+K3s image from iso.mulgadc.com, verifies its
sha256, and stamps spinifex:managed-by=eks — the tag the control-plane
and node-group launchers resolve the boot AMI by, so no extra config is
needed.
Add scripts/publish-system-image.sh (+ make publish-eks-node-image) to
convert the built raw to a compressed qcow2, emit a checksum sidecar,
and upload both to the R2 bucket backing iso.mulgadc.com/system-ami via
the S3 API.
* ci(eks): add manual workflow to build + publish eks-node AMI to R2
Build the eks-node AMI on the runner (libguestfs) and publish via
publish-system-image.sh using the existing CF_R2_* secrets. Manual
workflow_dispatch with a dry-run input. Fix the publish script's default
bucket to spinifex-iso (the bucket iso.mulgadc.com is served from).
The LB system microVM stripped its module tree to virtio + fw_cfg +
failover, but init.sh only loaded qemu_fw_cfg and virtio_net. virtio_net
depends on net_failover -> failover; modprobe resolves that chain via
modules.dep, and load_mod()'s insmod-by-path fallback loads a single .ko
with no dependency resolution.
When the image is built on a kmod-less host (depmod skipped, no
modules.dep) the chain can't be resolved either way, so virtio_net fails
to load, no NIC appears ('no interface found for MAC'), the lb-agent has
no network, heartbeats never reach AWSGW and every LB hangs in
provisioning until the 5m e2e timeout. Baremetal runners ship kmod so
modules.dep is generated and the chain resolves — which is why the LB
suite passed there but failed on normal CI.
Load failover -> net_failover -> virtio_net explicitly bottom-up so the
NIC comes up regardless of whether modules.dep exists.
EKS create-cluster fails deterministically on the multinode job with
ServiceUnavailableException: the EKS KV is a clustered (Raft) JetStream
stream and create-cluster needs an elected leader, but multinode
JetStream clustering isn't reliably forming yet (see mulga-omq/z2z/5yg).
EKS is validated single-node only at this stage.
Drop eks from the multinode build+run suite filters (alongside the
existing reboot/single exclusions) so the multinode job reflects what's
actually supported. Single-node continues to run the full eks suite.
* feat(elbv2): SetSubnets — full add+remove LB subnet re-homing
Add the ELBv2 SetSubnets operation across the interface, NATS subject,
daemon handler/subscription, and gateway route/forwarder.
Re-homing is a full add+remove: subnets new to the LB get a fresh managed
ENI, dropped subnets have their ENI deleted, then the backing system VM is
relaunched on the new ENI set. The system-instance launcher exposes only
Launch/Terminate (no live ENI hotplug for system VMs), so applying new
subnets relaunches the LB VM — a brief data-plane interruption. The record
is mutated only after the new ENIs exist and the VM is relaunched.
Extract a shared launchLBVM helper from CreateLoadBalancer so create and
SetSubnets boot the LB VM identically. Idempotent on no-op; record-only
fallback when no VPC service is wired.
Completes the elbv2 mutation-API surface (SetSubnets was the last gap).
* feat(tagging): resourcegroupstaggingapi GetResources for controller discovery
Add the Resource Groups Tagging API as a new gateway service (SigV4 scope
\"tagging\", AWS JSON 1.1 / X-Amz-Target, mirroring ACM). GetResources is a
gateway-side aggregator over the existing elbv2 and ec2 tag stores — no new
NATS subject or daemon handler.
It lists the account's load balancers and target groups (attaching tags via
batched DescribeTags) plus EC2 resources from the tag store, then applies the
request's ResourceTypeFilters (service-wide or fully-qualified) and TagFilters
(AND across filters, OR within values). Results are ARN-sorted with opaque
offset pagination.
This is how aws-load-balancer-controller v2 discovers and adopts the elbv2
resources it manages (filtering on its cluster/stack tags); DescribeLoadBalancers
has no tag-filter field, so RGT GetResources is the required path.
* fix(eks): emit canonical exception code, no doubled suffix
GenerateEKSErrorResponse appended \"Exception\" unconditionally, so codes that
already carry it (e.g. awserrors.ErrorEKSResourceNotFound =
\"ResourceNotFoundException\") serialized as ResourceNotFoundExceptionException.
SDK clients comparing eks.ErrCodeResourceNotFoundException then fail to match.
Append the suffix idempotently. Also covers the many other awserrors codes that
already end in \"Exception\" and flow through the shared JSON error envelope
(EKS, ACM, tagging).
* refactor(tagging): split GetResources fetch from orchestration for testing
Introduce a resourceLister seam: GetResources wires a NATS-backed lister, while
getResources (parse + tag-filter + sort + paginate) and buildEC2Mappings (tag
grouping + ARN synthesis) are now pure and unit-tested without a live backend.
No behaviour change.
* feat(eks): per-cluster opt-in K3s built-in ingress + enforce parity default
The runtime-rendered /etc/rancher/k3s/config.yaml emitted no disable: block, so
traefik + servicelb actually ran on every cluster — the AMI config.yaml.skel
that disables them is a never-applied boot fallback (cloud-init overrides it).
Render the disable block by default so traefik + servicelb are off for AWS
parity (Service type=LoadBalancer / Ingress are the AWS Load Balancer
Controller's job). A cluster created with tag spinifex.io/managed-ingress=true
keeps the K3s built-ins for interim in-VPC app exposure before the controller
add-on lands. The flag is derived into ClusterMeta.BuiltinIngress at
CreateCluster and threaded onto K3sServerInput.
* feat(ui): subnet editor on load balancer detail page (SetSubnets)
Add a Subnets tab to the LB detail page backed by the elbv2 SetSubnets
API. The editor mirrors the security-groups editor (checkbox list of the
VPC's subnets, dirty-tracking, Reset/Save) and warns that applying a
change re-homes the load balancer by relaunching its system VM, causing
a brief data-plane interruption.
* fix(elbv2): SetSubnets must detach ENIs before delete/relaunch + roll back on terminate failure
TerminateSystemInstance tears down the LB VM but leaves its ENIs marked
in-use, so the relaunch path could neither delete removed-subnet ENIs
(InvalidNetworkInterface.InUse leak) nor re-attach retained ENIs to the
relaunched VM (attach failed InUse, dropping the retained subnet's data
plane). Detach the existing ENIs explicitly after terminate — mirroring
DeleteLoadBalancer — before deleting removed ones and relaunching.
Also roll back the freshly-created added-subnet ENIs when the terminate
step itself fails, so a failed relaunch no longer leaks ENIs (amplified
by SDK retries).
* fix(daemon): re-arm system-instance terminate subscription on recovery
system.TerminateInstance.{id} was bound only by the system.LaunchInstance
handler at first launch, so it was lost across a daemon restart: recovery
reconnects to the surviving QEMU (or relaunches) but never re-bound the
subject, leaving system.TerminateInstance.{id} with no responder. Any LB
that predated the restart could then never be torn down — SetSubnets
relaunch and DeleteLoadBalancer both failed with 'no responders'.
Re-arm it in onInstanceUpHook, which fires on both the relaunch and the
reconnect-to-surviving-QEMU recovery paths, for ManagedByELBv2 VMs; drop
it in onInstanceDownHook. Split subscribeSystemTerminate into a locked
helper so the hook (which already holds d.mu) can reuse it.
Make e2e-baremetal.yml a reusable workflow (workflow_call + a ref input)
so the nightly and release pipelines can invoke the real PXE-reimaged
hardware suite, which previously only ran on manual dispatch.
Nightly: add a baremetal job (conceptual cell 20) that runs every
weeknight on the existing cron, or on dispatch when 20 is selected. It
tests dev on schedule, matching GIT_BRANCH. Sits outside the tofu matrix
(different runner, provisioning, 120m timeout) and shares the
e2e-baremetal-cluster concurrency group, so all baremetal runs serialise
on the single physical node.
Release: a release builds and uploads an ISO to R2 but never booted that
ISO/squashfs path on hardware. Add a baremetal job (needs: release) and
hard-gate on it: promotion moves to a dedicated promote job
needs:[verify, baremetal] (with a matching mark-failed job on failure),
and the iso job's upload gate now waits for both verify and baremetal to
conclude success before pushing to R2. Bump the iso job timeout to 180m
since it now waits on the parallel ~2h baremetal run.
build-microvm-image required depmod (kmod) as a mandatory tool to
generate modules.dep. Self-hosted CI runners don't ship kmod and can't
reliably apt-install it, so the inline 'apt-get install kmod' steps added
to e2e/e2e-nightly/e2e-ddil were unreliable.
modules.dep is an optimisation, not a boot requirement: the guest init's
load_mod() (build/microvm/init.sh) falls back to insmod-by-path when
modprobe can't resolve a module. So:
- Drop depmod from the mandatory-tool preflight.
- Run depmod only when present; still assert a non-empty modules.dep in
that case to keep catching the trixie-kmod empty-dep bug.
- Warn and skip when depmod is absent (guest uses insmod fallback).
- Remove the apt-get kmod install steps from all three workflows.
Gold-image bake still installs kmod (provision.sh), so production images
get a proper modules.dep; only kmod-less CI runners take the fallback.
* ci(e2e): install kmod (depmod) before microVM image build
#305 made depmod a hard requirement in build-microvm-image.sh, but the
self-hosted ci-single/ci-multi runners don't ship the kmod package, so the
Build Install Artifacts step aborts with 'required tool not found: depmod'
before any E2E suite runs. Add an idempotent kmod install ahead of each
Build Install Artifacts step (single/multi/eks jobs); the gold-image builder
bakes kmod in too, so this no-ops once runners pick up the new image.
* ci: install kmod (depmod) in nightly + ddil too
e2e-nightly and e2e-ddil call the same build-install-artifacts.sh (-> make
build-microvm-image) on the same self-hosted ci-single/ci-multi runners as
e2e.yml, so they hit the identical 'depmod not found' abort. Add the same
idempotent kmod guard before Build Install Artifacts in both.
release distro builds run on GitHub-hosted ubuntu (kmod present); e2e-baremetal
runs on ci-baremetal which already has it (green today) — neither needs the
guard.
* feat(elbv2): localize internal-LB heartbeat failures for baremetal debug
Add diagnostics so a stuck-in-provisioning internal LB can be pinned to
one of three points on the heartbeat path:
- lb-agent failure log now carries the gateway target + region, so the
microVM serial console shows whether the heartbeat hit a transport
error (never reached AWSGW) or an HTTP-level rejection.
- AWSGW LBAgentHeartbeat logs receipt (debug) and warns on the
previously-silent not-found/account-mismatch reject path, so the
journal distinguishes 'heartbeat arrived but cannot transition' from
'heartbeat never arrived'.
* test(e2e/lb): fail fast on stuck internal LB + probe fw_cfg on microvm boot
- lb suite: gate the remaining internal subtests on Internal_ALB. The
internal LBs share one mgmt-return datapath and fixture, so if the
first never reaches active the rest time out identically — skip them
instead of burning ~5m per subtest (~20m saved on a known-bad run).
- microvm init.sh: widen the fw_cfg netcfg poll window and, on the
not-available path, dump DMI / loaded modules / qemu_fw_cfg by_name
listing / dmesg fw_cfg to the serial console before init exits, so a
boot panic is diagnosable from the captured console log.
* fix(microvm): insmod fallback for fw_cfg + fail build on bad modules.dep
Baremetal-built microVM images shipped an unusable modules.dep (the build
host's depmod errors were swallowed), so busybox modprobe in the guest init
silently no-oped, qemu_fw_cfg never loaded, /sys/firmware/qemu_fw_cfg was
absent, init could not read opt/spinifex/netcfg, and the LB microVM panicked
before lb-agent ran — leaving internal LBs stuck in provisioning.
init.sh: load_mod() tries modprobe, then falls back to insmod by explicit
path (.ko/.ko.gz) which needs no modules.dep, so the guest boots regardless
of which host built the image.
build-microvm-image.sh: require depmod, drop the error swallow, and assert
modules.dep is non-empty after depmod so a boot-broken image fails the build
loudly instead of silently.
* fix(microvm): decompress kernel modules before depmod
Debian trixie's kmod 34 depmod silently emits an empty modules.dep when fed
Alpine's gzip-compressed (.ko.gz) modules — reproduced: 928 modules in, 0-byte
modules.dep out. The guest's busybox modprobe then no-ops, qemu_fw_cfg never
loads, and internal LB microVMs panic before lb-agent runs (stuck provisioning
on bare metal; GitHub-hosted runners ship a kmod that reads .gz, so normal CI
was unaffected).
Decompress kept modules to plain .ko after the keep-list strip, before depmod.
Host depmod then produces a usable map regardless of its gzip support, and the
guest loads modules without a decompressor.
* feat(e2e): capacity-aware scheduling for dense single-node runs
Denominate daemon schedulable capacity in physical cores instead of
logical threads, and auto-serialise e2e launches when the cluster is at
capacity, so a single hyperthreaded node stops over-packing guests.
Daemon: NewResourceManager sizes hostVCPU from physicalCoreCount()
(distinct physical-id:core-id pairs in /proc/cpuinfo) rather than
runtime.NumCPU(). Falls back to logical CPUs when topology is absent
(non-Linux, container, ARM cpuinfo without core id). On a hyperthreaded
box this halves the over-pack: with the unchanged 2 vCPU reserve the
i5-12600H schedules 10 vCPU (12 cores - 2) instead of 14, keeping the
boot-spike CPU within real cores.
Harness: capacityRetryEC2 decorates the EC2 client and, on
InsufficientInstanceCapacity, waits and retries RunInstances instead of
failing — turning the daemon admission gate into back-pressure so
concurrent subtests queue. AWSClient.EC2 is now the wrapped
ec2iface.EC2API; EC2Conf keeps the raw *ec2.EC2 for .Config access.
Adds unit tests for parsePhysicalCores (hyperthreaded/hybrid/dual-socket/
fallback) and the retry decorator (retry-then-succeed, non-capacity
passthrough, deadline).
* fix(daemon): SPINIFEX_HOST_VCPU override so small CI hosts can start
Denominating capacity by physical cores starves tiny hosts: a 2-physical-
core CI runner leaves 0 schedulable after the default 2-vCPU reserve, so
NewResourceManager refuses to start and every daemon-constructing unit test
fails. The previous runtime.NumCPU() count (4 logical on the same runner)
incidentally cleared the reserve floor.
Add a SPINIFEX_HOST_VCPU escape hatch (symmetric with the existing
SPINIFEX_RESERVED_* overrides) that pins the detected core count for hosts
whose /proc/cpuinfo topology misreports, and pin it to 4 in the daemon
TestMain so unit tests are deterministic regardless of runner topology.
Reserve stays at the default 2.
* test(eks): EKS control-plane e2e suite (Phase 1, API-level)
Add tests/e2e/eks/eks_test.go driving the EKS lifecycle against awsgw:
CreateCluster -> ACTIVE (endpoint + CA asserted), kubeconfig artifact
assembly, AccessEntry CRUD, get-token presign decode, DeleteCluster -> gone.
Harness: EKS SDK client on AWSClient + WaitForEKSCluster{Active,Deleted}
pollers. Wire e2e-eks into the Makefile and the single-node CI suite list.
* test(eks): Phase 2 kubectl get-nodes data-plane gate
Add the reachability shim (harness/eks_mgmt.go) that reads the control-plane
VM's br-mgmt IP from the cluster KV meta over NATS, plus a kubectl wrapper
(harness/kubectl.go) that skips when kubectl is absent. KubectlGetNodes points
kubectl at https://<mgmtIP>:6443 (insecure-skip-tls-verify; cert SANs the NLB
DNS, not the mgmt IP) with an aws eks get-token exec block and polls until the
control-plane node is Ready — the get-token webhook auth regression gate.
IRSA pod + the real NLB/Route53 passthrough are deferred to a follow-up.
* test(eks): tolerate doubled NotFound code + dial loopback NATS
DeleteCluster detection matched the SDK constant ResourceNotFoundException,
but awsgw emits ResourceNotFoundExceptionException on the wire — match the
NotFound code by substring so deletion is detected immediately instead of
timing out. The mgmt-IP shim dialed NATS.Host verbatim (0.0.0.0:4222), which
the serving cert does not SAN; rewrite the wildcard bind to loopback.
Validated live: all 6 subtests green, kubectl get nodes shows the
control-plane node Ready via the get-token webhook auth path.
* feat: build system images with libguestfs (no host block devices)
Replace the qemu-nbd + sudo mount + chroot + /dev-bind flow in
build-system-image.sh with libguestfs (virt-customize / virt-resize /
guestfish). Every operation runs in an isolated userspace appliance VM with
its own kernel, so the build touches no host block device, no host mount, and
needs no sudo — removing the failure class that wiped the host's /dev on shared
CI runners (a leaked /dev bind traversed by rm -rf).
Applies to all distros: alpine resizes via guestfish e2fsck+resize2fs, ubuntu
via virt-resize; customization (packages, binaries, files, setup script,
service enable, cache cleanup) is one ordered virt-customize invocation.
LIBGUESTFS_BACKEND=direct (no libvirt on runners).
Wire the eks-node AMI into the single-node e2e job: the runner builds the
device-free raw, ships it to the VM, and the VM imports it (predastore/NATS are
VM-only) so the eks suite's CreateCluster has an AMI to boot.
* feat(eks): bootstrap + state-report via AWS gateway broker
Converge the EKS control-plane → host channel onto the ELBv2 lb-agent model:
the K3s server VM publishes its bootstrap envelopes and periodic state reports
via SigV4-signed HTTPS POST to the AWS gateway, instead of dialing core NATS
directly. A new gateway action (PublishInternal, POST
/clusters/{name}/internal-publish) relays the payload verbatim onto the
existing eks.bus.* / eks.state.* subjects, so the daemon-side subscriber, KV
persistence/encryption, and reconciler are unchanged.
This removes the VM's need to reach NATS at all, so NATS can stay
cluster-internal behind its bind/firewall/mTLS controls — the prior direct
path broke when NATS was not VM-reachable (e2e CreateCluster timeout) and was
DDIL-fragile (fire-and-forget nats pub, no ack/retry).
- gateway/eks: PublishInternal broker + route; closed-set channel/kind validation
- cmd/eks-gateway-publish: one-shot SigV4 publisher (retry-until-ack) baked into
the eks-node AMI; replaces the nats-cli publisher in k3s-first-boot.sh and
mulga-eks-state-report.sh
- handlers/eks + daemon: hand the K3s server VM the mgmt-reachable gateway URL +
system (Predastore) SigV4 creds + gateway CA instead of NATS URL/token/CA;
new resolveSystemGatewayBaseURL (mgmt-bridge-first)
- eks-node image: build/install eks-gateway-publish, drop nats-cli download
* feat: migrate eks-token-webhook to the AWS gateway broker
The TokenReview webhook dialed core NATS directly (STS verify + AccessEntry
KV read). On a mgmt-only control-plane VM with NATS bound off-subnet that
fails, so the webhook exited at startup, the apiserver token-webhook
kubeconfig was never written, and the apiserver crash-looped — the cluster
never reached ACTIVE.
Converge it onto the same gateway-broker model as bootstrap/state-report and
ELBv2's lb-agent: the webhook SigV4-signs an HTTPS POST {accountId,token} to
/clusters/{name}/token-review, and the gateway runs the verify + lookup
host-side and returns the resolved identity. The VM now speaks no core NATS.
- internal/eksgw: shared on-VM SigV4 client (also adopted by
eks-gateway-publish).
- handlers/eks/token_review.go: Authenticate decision core (moved from the
webhook binary) + ResolveTokenReview host-side wiring.
- gateway/eks token-review route + WebhookTokenReview handler.
- webhook: drop NATS, relay via eksgw.Client; broker fault -> 503 (retryable)
not a silent deny.
- initd: export EKS_GATEWAY_* / EKS_ACCESS_* creds, not SPINIFEX_NATS_*.
* fix: build eks-node helpers with GOFIPS140; cover token-review broker
eks-gateway-publish blank-imports internal/fipsboot, whose init() panics
unless the binary is built with GOFIPS140=v1.0.0. The eks-node AMI build
omitted the flag, so the binary paniced at first boot right after k3s came
up — k3s-first-boot (set -eu) died before publishing the bootstrap token and
the cluster never reached ACTIVE. Build both eks-node helpers with
GOFIPS140=v1.0.0, matching cmd/lb-agent.
Add NATS-backed unit tests for ResolveTokenReview and the WebhookTokenReview
gateway handler (verify responder + account KV) to cover the new broker path.
* fix: export first-boot env so eks-gateway-publish child sees it
k3s-first-boot.sh and mulga-eks-state-report.sh sourced /etc/spinifex-eks/
first-boot.env but never exported the vars, so the eks-gateway-publish child
process inherited an empty environment and exited '--account-id is required'.
The control-plane VM came up healthy but never published its bootstrap
envelopes, so the cluster never reached ACTIVE. Wrap the source in set -a /
set +a so the KEY=value lines export to child processes.
* fix: make awsgw start --host override actually bind
The 'awsgw start --host' override printed 'Overwriting awsgw host' but the
assignment was commented out, so AWSGW always bound the spinifex.toml host
(a specific WAN IP, e.g. 192.168.0.10). A system instance (EKS K3s control
plane, internal LB agent) has a mgmt-only NIC on-link to the br-mgmt IP with
no route to the WAN IP, so its SigV4 POSTs to the gateway were refused and the
cluster never bootstrapped. Honour --host (e2e passes 0.0.0.0:9999) so AWSGW
listens on all interfaces incl. br-mgmt; the serving cert already SANs every
local interface IP, and WAN clients still reach it.
* test(eks): install kubectl on e2e VM so KubectlGetNodes runs
The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full
aws eks get-token -> kubectl -> apiserver -> token-webhook -> gateway broker
-> STS verify path — the only end-to-end exercise of the token-review broker.
The e2e binaries run on the VM, where kubectl was absent, so
harness.NewKubectl t.Skip'd it and the path went unvalidated. Install a
checksum-verified kubectl (pinned to the k3s server version) on the VM,
gated on e2e-eks.
Internal-scheme LBs on single-node have no dedicated mgmt-bind IP on
br-mgmt, so resolveMgmtRoute falls back to routing the agent to the
advertised WAN IP via the mgmt NIC. That advertised IP lives on a
different bridge than the agent's mgmt NIC, and the cross-bridge return
path never delivers — the agent never heartbeats and the LB hangs in
provisioning (normal single-node dodges this entirely via dev_networking
slirp at 10.0.2.2; bare-metal external-mode is the first real exercise of
this path).
Point the agent at the br-mgmt host IP, which shares its mgmt L2 — no
cross-bridge hop — mirroring the multi-node datapath. Requires AWSGW to
listen on br-mgmt (bind 0.0.0.0); the server cert already SANs that IP.
Internal-scheme LBs only leave provisioning once the lb-agent heartbeats
AWSGW, which requires the mgmt return route resolved by resolveMgmtRoute
(MgmtRoute{Gateway,Target}, or MgmtBridgeIP+AdvertiseIP on single-node).
When none resolve, the route was silently skipped and the LB hung in
provisioning until the client timed out, with no signal as to why.
Log which precondition is empty on the internal skip path, and mark the
LB failed instead of provisioning when no return route exists, so the
broken data-plane surfaces immediately.
InitBuckets was passed the OVN chassis count as the replica factor. When the
chassis count exceeds the NATS node count (single-node bare-metal: 1 node but
2 SBDB chassis), CreateKeyValue requested more replicas than JetStream could
satisfy, the create-or-open fallback returned "bucket not found", and vpcd
crash-looped — cascading to RunInstances ServerInternal. Create at 1 replica
like STS/IAM; the daemon's deferred upgradeJetStreamReplicas raises all KV_*
streams to cluster size when the cluster grows.
* feat(eks): webhook observability — console log sink + per-request TokenReview decision
* fix(eks): wire token-webhook authenticator into generated k3s apiserver config
The generated /etc/rancher/k3s/config.yaml overrides the AMI-baked
config.yaml.skel, but its kube-apiserver-arg list omitted the
authentication-token-webhook-config-file arg. The running apiserver
therefore never loaded the token webhook, so bearer tokens minted by
'aws eks get-token' 401'd without the webhook ever being invoked.
* fix(eks): echo TokenReview audiences so api-audiences apiserver accepts the principal
With --api-audiences set, kube-apiserver sends spec.audiences in the
TokenReview and discards an authenticated=true response that does not
confirm a matching audience. The webhook returned an empty
status.audiences, so a valid IAM principal surfaced as a 401. Echo the
request's audiences on success.
* fix(eks): pin token-webhook version v1 so apiserver accepts the response
kube-apiserver --authentication-token-webhook-version defaults to
v1beta1, but the eks-token-webhook responds with an
authentication.k8s.io/v1 TokenReview. The apiserver decodes the response
against the expected GVK and rejects the mismatch, surfacing a valid
principal as a 401. Pin v1 to match the webhook.
Bare-metal e2e produced no artifact, so RCA fell back to truncated
log-failed output. Tee bootstrap + e2e output to test-baremetal.log,
generate junit-baremetal.xml, run e2e-analyze to emit analysis.md, and
upload the bundle as e2e-baremetal-artifacts-<run_id>.
aws eks list-clusters returned InternalErrorException for a few seconds after a
daemon restart, while describe-cluster (by name) succeeded. During the
post-restart window the per-account KV stream's Raft group has no leader yet, so
kv.Keys() (an ordered consumer over the warming stream) fails with a transient
NATS error. ListClusters wrapped it as a fmt.Errorf string the daemon's
ValidErrorCode cannot map, so it sanitized to ServerInternal — a misleading 500
that tells AWS clients the request is non-retryable.
Classify transient NATS conditions (no-responders, timeout, no-stream-response,
connection-closed) and return a retryable ServiceUnavailable (503) instead;
genuine faults still pass through wrapped. describe-cluster survived only
because kv.Get on the same bucket is cheaper than the consumer kv.Keys spins up.
* feat(iam,awsgw): OIDC provider CRUD + IRSA discovery endpoints (6e)
Complete the AWS-SDK-facing half of IRSA:
- IAM Create/Get/List/DeleteOpenIDConnectProvider, backed by the per-account
oidc-providers registry that STS AssumeRoleWithWebIdentity already reads to
authorise an issuer. Create writes the key STS checks; the provider ARN is
the scheme-stripped issuer per AWS convention.
- awsgw serves the per-cluster OIDC discovery document
(/.well-known/openid-configuration) and JWKS (/keys) as unauthenticated
public endpoints, mounted ahead of the SigV4 surface in their own route
group. The discovery issuer is the cluster's stored OIDCIssuer, byte-exact
with the apiserver iss claim; JWKS is served from the EKS account bucket.
Unit + route tests cover CRUD happy/idempotent/not-found/invalid-url, the
ARN<->issuer round-trip, discovery issuer/jwks_uri equality, JWKS bytes, 404s,
and the no-auth bypass. make preflight green.
* fix(awsgw): route anonymous AssumeRoleWithWebIdentity past SigV4 (6e)
AssumeRoleWithWebIdentity is the IRSA credential-bootstrap call: the AWS SDK
sends it unsigned because the caller holds no AWS credentials, only a projected
ServiceAccount JWT in the body. The gateway ran the SigV4 middleware first and
rejected it with MissingAuthenticationToken, so the true anonymous path never
reached the handler.
Add anonymousSTSInterceptor, a pre-auth middleware mounted ahead of the SigV4
surface. For an unsigned POST it body-peeks the Action and, for the
anonymousSTSActions set, dispatches straight to STS_Request; STS_Request now
skips SigV4-caller resolution for those actions (the handler authenticates the
JWT and gates on the role's web-identity trust policy). Signed requests carry an
Authorization header and pass through untouched, off the body-buffering path.
Surfaced by the 6e live exit gate: a real in-pod assume-role-with-web-identity
would otherwise have been rejected before reaching token validation.
* feat(eks): nodegroup provisioning + K3s agent join (6d)
CreateNodegroup launches N customer-owned worker EC2 from the unified
eks-node AMI in agent role; each joins the cluster via K3S_URL + K3S_TOKEN
baked into cloud-init (first-boot.env SPINIFEX_K3S_ROLE=agent + agent.env).
Workers are customer-owned and DescribeInstances-visible (no ManagedBy,
no mgmt NIC) via the normal RunInstances path, matching real EKS.
K3S_URL targets the control-plane ENI's in-VPC private IP on :6443 — the
cluster NLB DNS endpoint is not resolvable in-VPC yet; switch to the NLB
endpoint on :443 once authoritative in-VPC DNS lands. K3S_TOKEN is the
decrypted cluster node-token persisted at server bootstrap.
Describe/List/Update(scaling)/Delete round out CRUD. Cluster<->nodegroup
SG ingress rules (apiserver 6443, flannel VXLAN 8472/udp, kubelet 10250,
intra-nodegroup) authorized idempotently. UpdateNodegroupVersion deferred.
* feat: gate EKS cluster health via NATS state reports
The K3s apiserver binds the VPC node-ip, unreachable from the host-side
spinifex daemon, so the reconciler cannot probe /healthz directly. Add a
control-plane state reporter that publishes {healthz, node_count} over the
management NATS bus the daemon already shares.
- eks-node server role runs mulga-eks-state-report as an OpenRC service,
publishing eks.state.{account}.{cluster}.server every 30s.
- ClusterReconciler subscribes to that subject, gates CREATING -> ACTIVE on
a healthy report, and records Ready node_count into ClusterMeta.
- node_count counts Ready nodes only; k3s retains terminated workers as
NotReady until pruned, so a raw count never drops on scale-down.
The iso job mounts a kernel-global devtmpfs under the workspace
(.cache/rootfs/dev). A killed job skips its EXIT-trap unmount and leaks
the bind; the verify job (same ci-single runner) then ran a bare
'rm -rf $WORKSPACE/*' that could traverse it and wipe the host /dev.
Detach any mount under the workspace before deleting and add
--one-file-system so rm never crosses a mount boundary. Same backstop
added to the iso job's own cleanup.
A killed ubuntu system-image build can leave `mount --bind /dev` under
its build dir. The e2e Clean Workspace step then ran
`sudo rm -rf "$GITHUB_WORKSPACE"/*` as root, traversed the leaked bind,
and deleted the host's /dev (incl. /dev/null), wedging the runner.
- e2e.yml: umount anything under the workspace (deepest-first) before
deleting, and pass `rm --one-file-system` so rm can never cross a mount
boundary into host /dev. Hard backstop, applied to all 3 Clean Workspace
steps.
- build-system-image.sh: recursive lazy unmount in cleanup() detaches the
chroot binds even when busy, and mark the build subtree rprivate so the
/dev bind can't propagate into the host namespace.
* fix: switch EKS datastore to embedded etcd + fix Alpine cloud-init IMDS route
Flip the K3s control-plane datastore from SQLite/kine back to embedded etcd
(cluster-init: true, etcd-expose-metrics: true). The earlier abandonment
premise — that etcd's per-commit fsync stalls past the apiserver's 5s deadline
on the viperblock-backed root volume — was wrong: etcd inits fine on an
unchanged viperblock (apiserver ready 25-30s, clusters reach ACTIVE). The real
blocker was a network-config regression, fixed here.
Alpine's cloud-init eni renderer crashes (KeyError: 'gateway') on the
gateway-less IMDS on-link route, which failed the whole network-config render
so the static mgmt NIC never came up (no NATS bootstrap, no /healthz). Fixes:
- generateNetworkConfig: add emitIMDSRoute bool; selectNetworkConfigForFamily
passes false for Alpine so the eni renderer never sees the route.
- Deliver the IMDS route out-of-band via a persistent /etc/local.d script +
OpenRC local service. Resolve the VPC NIC via its default route at runtime
(ip route show default), not a hardcoded name: the persistent-net rename to
vpc0 doesn't apply on the Alpine AMI, so the NIC is eth0 at first boot.
- EKS path bakes the same route into buildK3sUserData's own write_files/runcmd;
the generic template Alpine block fires only when no #cloud-config user-data,
avoiding a duplicate write_files key (yaml.safe_load keeps last, silently
dropping a block).
- eks-node image: config.yaml.skel adds cluster-init; comments updated.
* fix: surface CreateCluster step failures at ERROR + map subnet faults; add auth-mode/delete test cases
CreateCluster wrapped its step failures in bare fmt.Errorf that the request
surface collapses to an opaque ServerInternal ("contact AWS re:Post"), so the
real cause never reached the operator. Add logCreateErr to log each step's
precise cause at ERROR before returning, and map a subnet-resolve failure
(caller-supplied parameter) to InvalidParameterValue instead of a retryable
ServerInternal the SDK pointlessly retries. (mulga-siv-165.4)
Tests (mulga-siv-165.15): reject authenticationMode=API_AND_CONFIG_MAP (only
the CONFIG_MAP value was covered); DeleteCluster on a nonexistent cluster via
the full-deps path returns ResourceNotFound (the shim path short-circuits to
ServiceUnavailable before the meta lookup).
* fix: hide EKS control-plane VMs from customer DescribeInstances
EKS K3s control-plane VMs are owned by the customer account (their ENI
lives in the customer VPC for the NLB target), so unlike LB microVMs
(owned by the system account) they leaked into customer
ec2:DescribeInstances. The spinifex:managed-by tag only filtered the UI
Nodes page, not the EC2 API.
Add a caller-aware exclusion: skip any ManagedBy-tagged VM unless the
caller is root. Preserves operator visibility, matches LB behaviour, and
leaves future customer-owned worker nodes (no ManagedBy) visible.
* docs: correct stale SQLite datastore comments in eks-node image (now embedded etcd)
reloadNginx started nginx via cmd.CombinedOutput(). nginx daemonizes, and the
rendered NLB config uses `error_log stderr`, so the forked master and workers
inherit and hold this command's stderr open. CombinedOutput's Wait blocks on
stderr EOF, which the long-lived daemon never sends — so the call never returns,
wedging the agent's synchronous tick() before it applies the nginx engine or
reports health. NLB targets stayed at 0/N until the e2e health wait timed out,
while the data plane (nginx itself) ran fine — so manual smoke tests passed.
ALB is unaffected: haproxy -D redirects its stdio to /dev/null on daemonize, so
CombinedOutput sees EOF and returns.
Capture nginx start/reload stdio to a temp file and use cmd.Run(), so Wait
blocks only on the short-lived foreground process that exits once daemonized.
Surfaced via the LB microVM guest console now collected in CI.
NLB target health stalls at 0/N in CI while the data plane works locally.
The lb-agent's nginx engine activation, any nginx -t/reload error, and its
active-probe results are logged to the LB microVM serial console — which the
host journal never captures and the GetConsoleOutput API path reports as
NotFound, so the failure is undiagnosable from CI artifacts.
- e2e.yml: pull /run/spinifex/console-*.log off each node in the single- and
multi-node collect steps (files persist past VM teardown) into lb-console/.
- lbagent healthcheck: log probe failures at Warn (was Debug) so unreachable
targets surface at the agent's default level on the captured console.
E2E NLB suites timed out at "0/2 healthy" while ALB suites on the same
target instances went healthy in ~5s. The host daemon journal shows the
NLB agent reporting an empty server list (no probe results at all), which
happens when the lb-agent never enters nginx mode — i.e. `nginx -t`/start
fails closed inside the LB microVM, so the agent never probes and reports
nothing. The data plane itself is verified working; this is purely the
active-health-check pathway failing to converge in CI.
- lbagent: pre-create nginx runtime prefix dirs (pid dir + /var/lib/nginx,
/var/lib/nginx/tmp, /var/log/nginx) before `nginx -t`, mirroring
reloadHAProxy's socket-dir pre-create. A stripped Alpine initramfs may
lack these, dead-ending the whole NLB health path.
- build-microvm-image.sh: assert the nginx binary and ngx_stream_module.so
are present in the image and fail the build loudly, instead of surfacing
as an opaque e2e health timeout.
- e2e: capture the LB VM serial console to the artifact bundle on NLB
health-wait failure. The agent's engine activation and any `reload nginx:`
error land on the guest console, which the host journal never sees.
* feat(elbv2): NLB L4 data plane on nginx stream (mulga-siv-952)
Network load balancers render an nginx `stream` config (TCP/UDP/TLS/
TCP_UDP) instead of HAProxy, which cannot load-balance UDP in any
release. The gateway tags GetLBConfig with Engine=nginx; the LB agent
dispatches to the nginx engine (config/pid/cert paths + reload) and
skips HAProxy stats polling. ALB stays on HAProxy. nginx is added to
the microvm image.
* docs(elbv2): note NLB runs nginx stream, ALB runs HAProxy (mulga-siv-952)
* fix(elbv2): load nginx stream module for NLB data plane
Alpine's `nginx` apk is built without the stream module — it ships
separately as `nginx-mod-stream`. Without it, the generated NLB config
fails `nginx -t` with [emerg] unknown directive "stream", so the agent
never applies it. Add the package to the microvm image and emit a
`load_module .../ngx_stream_module.so;` directive at the top of the
generated config.
* fix(elbv2-ui): allow single-subnet LB + full NLB protocols in listener edit
Create-LB form required min 2 subnets while the CLI/backend accept 1;
relax to min(1) to match. Listener add/edit only offered HTTP/HTTPS —
add the NLB protocol set (TCP, UDP, TLS, TCP_UDP), driven by LB type,
with cert/ssl-policy fields gated on secure protocols (HTTPS/TLS).
* feat(elbv2): active target health checks for nginx NLBs
nginx stream has no active upstream probing, so NLB targets never
advanced past initial — DescribeTargetHealth stayed initial forever and
WaitForTargetsHealthy timed out for every NLB. GetLBConfig now delivers
per-target HealthTarget specs (daemon-computed server name, probe
address, protocol/path) for network LBs; the lb-agent probes each target
every tick (TCP dial or HTTP/HTTPS GET) and reports UP/DOWN via the
existing heartbeat path, so the daemon's threshold hysteresis advances
targets to healthy. ALBs are unchanged (HAProxy stats).
Adds an Internal_NLB_UDP e2e suite gating on WaitForTargetsHealthy plus a
UDP datagram round-trip through the VIP.
* test: cover Sprint 3b ENI hot-plug surface
Sprint 3b's gateway + daemon + KV-helper additions landed without unit
tests. dev's diff-coverage gate flipped to 28.5% as a result. This adds
the three test files needed to bring the diff coverage of the ENI
hot-plug surface back above the 60% threshold.
- handlers/ec2/vpc/eni_hotplug_helpers_test.go: GetENIRecord, UpdateENI,
FindENIByAttachment — happy paths, not-found, account scoping,
round-trip of every new ENIRecord field.
- gateway/ec2/vpc/network_interface_hotplug_test.go: Attach + Detach
validation, no-responders, NATS roundtrip with command capture,
daemon error / malformed responses, attachment-resolver branches.
- daemon/daemon_handlers_eni_test.go: eniHotplugErrorCode table,
publishENIHotplugEvent (nil + publish), full handleAttach /
handleDetach matrix (nil data, not-running, ENI/attachment not found,
wrong owner, success with stub QMP + NATS event assertion, hot-plug
failure + KV rollback assertion).
vm.SetHotPlugTestSeams is added as an exported test seam following the
same pattern as utils.SetSudoCommandForTest so daemon-package tests can
swap newDeviceController + eniPipelineSleep without reassigning vm-package
unexported vars.
* feat(vm): wire real OVS tap into ENI hot-plug pipeline
Sprint 3c: replace the stubTapAdd/stubOVSAddPort/stubTapDel/stubOVSDelPort
placeholders with the shared NetworkPlumber. SetupTap creates the kernel tap
and attaches it to br-int with the OVN iface-id binding in one atomic step;
CleanupTap reverses both on rollback and detach. Nil-plumber managers stay a
noop, mirroring setupExtraENINICs.
Adds fake-plumber coverage: tap-on-br-int spec assertion, tap-setup failure
aborts before any QMP call, and QMP/detach paths both invoke CleanupTap.
The previous new-subnet egress baseline used the default VPC, whose main
route table is prebaked with 0.0.0.0/0 -> IGW at bootstrap and reconciled by
the vpcd drift loop, so a subnet created in it is reachable regardless of the
create-time gap — the test passed and never flagged the bug.
Replace it with runNewVPCEgressBaseline: a brand-new VPC where the IGW route
is added to the MAIN route table BEFORE the public subnet is created. The
subnet then implicitly joins a main RT that already carries the route, but
CreateSubnet does not recompute per-subnet egress, so the subnet's OVN egress
LRP is never installed and the instance has no datapath to the IGW. With an
open SG, routing is the only barrier, so the instance is unreachable — red
until CreateSubnet triggers a per-subnet egress recompute.
PoolMode-gated like runVPCSubnetE2E (fresh VPC needs OVN IPAM for a public
IP). runVPCSubnetE2E doesn't catch this: it associates a custom RT then adds
the route, so the route mutation enumerates the subnet and installs the LRP.
Add baseline tests that confirm two fresh-install regressions the existing
suite masks (it authorizes tcp/22 on the default SG before every SSH probe):
- default SG default-deny gates external reach of instances in the
default subnet (SG, not routing)
- a newly-created public subnet never gets its OVN IGW egress LRP, so it
has no route to the VPC IGW despite MapPublicIpOnLaunch
Plus a positive test: instances sharing the default SG can communicate via
ICMP (same-host single-node, cross-host multinode).
Single-node baselines run after TestClusterStatsCLI (0-VM gate); multinode
after TestMultinodePreflight (fixture init). Instances are self-cleaning so
they don't perturb the singleton-VM ordering.
Remove the qemu hostfwd SSH fallback from the harness (InstancePublicSSHHost,
GuestSSHEndpoint) - it bypasses the OVN datapath and masks these regressions.
Endpoints now t.Fatal when no public IP is assigned.
* feat(ui): NLB support + protocol/type-dependent create-LB fields
The Create Load Balancer page was hardcoded to ALB. Add a load-balancer
Type selector (application/network) and switch the listener + target-group
form options on the selected type, mirroring the server's validation:
- ALB: HTTP/HTTPS listeners and target groups.
- NLB: TCP/UDP/TLS/TCP_UDP listeners and target groups; no security groups.
- Secure protocols (HTTPS/TLS) reveal a certificate selector; HTTPS also a
security-policy selector. Cert is required for both; SSL policy is HTTPS-only.
- The default target-group protocol + health-check protocol stay in sync with
the listener (TCP health checks omit HTTP path/matcher).
- The wizard mutation now sends the chosen Type and threads cert/SSL-policy
through CreateListener (previously hardcoded Type=application and dropped
certificates).
* feat(ui): NLB-aware target groups and LB detail
Standalone Create Target Group offers the full protocol set
(HTTP/HTTPS/TCP/UDP/TLS/TCP_UDP) and syncs the health check to TCP for L4
groups, dropping path/matcher from the request and CLI preview. Hide the
Security groups tab on the load balancer detail page for network LBs.
* fix(admin): write ~/.aws credentials 0600 via atomic temp+rename
UpdateAWSINIFile ended with ini.SaveTo, which uses os.Create (0644 &
~umask) and left ~/.aws/credentials world-readable. Render to a 0600
temp file and rename into place so secrets are never exposed.
* fix(eks): purge OIDC signing key on zeroize; pin account bucket History=1
ZeroizeClusterOIDCKey used kv.Delete (a tombstone) which leaves prior
revisions of the encrypted PEM in KV history. Use kv.Purge to drop the
key's whole history. Decouple the account bucket's History from the
schema version (KVBucketEKSAccountHistory=1) so a version bump can't
silently widen history and resurrect ciphertext.
* fix(reconcile): get-or-create leader bucket in AcquireLeader
CreateKeyValue returns 'stream name already in use' once the bucket
exists, so every reconcile after the first hit the 60s deadline and
never elected a leader. Attach to the existing bucket first, create
only when genuinely absent.
* fix(eks): drop registry entry on lease-held instead of phantom holder
RunClusterReconciler returned (nil, nil) when another node held the
lease; Spawn recorded a holder with a nil release, so Has() stayed true
and a later Spawn no-op'd — the cluster was never re-reconciled until
daemon restart. Return ErrLeaseHeld so Spawn drops the entry and retries
after the holder's TTL lapses.
* refactor: tidy EKS cluster state, dedup, and not-found error wiring
- listClusterNames: O(n) map dedup instead of O(n^2) slices.Contains
- drop dead store *Store field + NewStore call from EKSServiceImpl
- ClusterMeta.LastHealthProbe: drop misleading omitzero tag (stdlib
encoding/json ignores it; json/v2 only) and document why it always
serializes
- clarify ErrorResourceNotFound is the ACM code; EKS uses
ErrorEKSResourceNotFound (ResourceNotFoundException)
* test: EKS 6b pagination, restart-resume, and bootstrap bind hardening
- T7: ListClusters NextToken/MaxResults paging + out-of-range clamp
- T4: SpawnRegisteredReconcilers resumes CREATING/ACTIVE on restart,
skips terminal clusters, no-ops when deps absent
- T-quality: replace sleep-based subscription binding with polling
NumSubscriptions, poll KV for the one-shot drop, and assert Run
tears down its subscriptions on return
T1 (restjson wire shape), T5 (JWKS kty mismatch), and T6
(resolveGatewayHost branches) were already covered.
* test(reconcile): cover AcquireLeader get-or-create election
First acquire creates the bucket and wins; a second acquire while the
lock is held attaches to the existing bucket and loses cleanly (the
get-or-create fix, not a CreateKeyValue dead-end); after release a later
acquire takes over.
* feat(ui): HTTPS listeners with ACM certificate + SSL policy
Expose HTTPS in the load balancer listener UI. The ListenerForm gains a
HTTPS protocol option that reveals a certificate selector (backed by a new
ACM ListCertificates query), an inline certificate import dialog
(ImportCertificate), and an SSL security-policy selector (DescribeSSLPolicies).
CreateListener/ModifyListener now carry Certificates + SslPolicy for HTTPS;
the listeners table summarises the policy and certificate per HTTPS listener.
Adds an ACM client (getAcmClient) and @aws-sdk/client-acm. Server, gateway
and ACM endpoints already exist (epic phases A + B); this is frontend-only.
* fix: ACM JSON timestamps, react-compiler watch reactivity, scroll-chaining
- gateway/acm: serialise responses with jsonutil.BuildJSON so time.Time
fields emit epoch seconds (AWS JSON 1.1 wire format). encoding/json
emitted RFC3339 strings the JS SDK rejected with 'Expected real number',
crashing the LB detail Listeners tab once any cert exists.
- frontend: replace watch() with useWatch across forms so the React
Compiler tracks the subscription and re-renders on field change
(HTTPS listener cert/policy fields now reveal). Add toggle regression
test for listener-form.
- frontend: scope overscroll-none to html,body instead of *, restoring
wheel scroll-chaining over nested scroll containers (sidebar, tables,
dialogs).
* feat(eks): unify eks-server + eks-agent into one eks-node AMI
Collapse the two K3s AMIs into a single spinifex-eks-node image whose role
(server vs agent) is chosen per-instance at first boot, halving build time,
storage, and CVE-rotation surface.
- scripts/images/eks-node/: superset manifest + merged setup.sh; all role
services baked but only crond + the new eks-node-role selector enabled.
- eks-node-role first-boot selector reads SPINIFEX_K3S_ROLE, rc-update adds +
starts the matching services (server: k3s + eks-token-webhook + k3s-first-boot
publisher; agent: k3s-agent), records /etc/spinifex-eks/role, self-disables.
Paths are env-overridable; eks-node-role_test.sh covers all five cases.
- mulga-eks-etcd-snapshot.sh early-exits on agent nodes.
- k3s_server_vm.go seeds SPINIFEX_K3S_ROLE=server into first-boot.env.
- Makefile/dev-install: build-eks-node-image / import-eks-node-image replace
the server+agent targets.
- Delete scripts/images/eks-server + eks-agent. The launcher resolves the AMI
by the spinifex:managed-by=eks tag, so running clusters (which hold the
resolved AMI ID) are unaffected.
AMI bake + cluster boot are operational integration steps, not yet run.
* fix(eks): order eks-node-role after cloud-config/cloud-final
The first-boot role selector reads /etc/spinifex-eks/first-boot.env, which
cloud-init delivers via write_files in the cloud-config/cloud-final stages.
The selector only depended on net+cgroups, so on a fast boot it could start
before write_files landed and read a missing role env. Add an after-ordering
on cloud-config + cloud-final so the env file is always present first.
* feat(eks): 6c AccessEntry CRUD + access-policy association
Replace the nine notImpl() AccessEntry/AccessPolicy stubs with real
KV-backed handlers (Q9, API auth mode):
- AccessEntry CRUD persisted at clusters/{name}/access-entries/{hash};
the principal ARN is hashed into the key (ARNs carry ':', illegal in
NATS KV keys) while the record keeps the plaintext ARN.
- AssociateAccessPolicy/Disassociate/ListAssociated against the four
supported AWS-managed policies; unsupported policy ARN or bad access
scope -> InvalidParameterValue.
- ListAccessPolicies returns the static supported catalogue.
- Cluster-existence gate -> ResourceNotFoundException; duplicate create
-> ResourceInUseException; node-type entries rejected until 6d.
Updates the daemon routing test for the now-implemented handlers.
* feat(eks): plumb caller principal ARN + bootstrap-creator-admin
Thread the caller's resolved IAM principal ARN from the gateway to the
EKS control plane via a new X-Principal-ARN NATS header, so CreateCluster
can mint the bootstrapClusterCreatorAdminPermissions AccessEntry (Q9):
- utils.NATSRequest takes optional extra headers; PrincipalARNHeader +
PrincipalARNFromMsg added.
- Gateway resolves the caller ARN best-effort (buildCallerARN) and passes
it down the EKS route table; only CreateCluster consumes it.
- CreateCluster signature gains callerPrincipalARN; default-true bootstrap
flag mints a system:masters entry keyed by the caller's exact ARN (the
token webhook resolves the same ARN from STS). '' degrades gracefully.
- Dedicated daemon handler reads both headers.
* feat(eks): IAM token webhook + STS verify-over-NATS + apiserver wiring
The eks-token-webhook (baked into the eks-server AMI) authenticates kubectl
requests that present an `aws eks get-token` bearer token: decode the presigned
STS GetCallerIdentity URL, verify it, resolve the principal ARN to its cluster
AccessEntry, and answer the apiserver TokenReview with the mapped username +
groups.
STS is hosted in-process in the awsgw service and is not otherwise on the bus,
so awsgw now serves an `eks.VerifyToken` NATS subject that wraps
VerifyPresignedGetCallerIdentity (the cluster name is forwarded so STS enforces
the x-k8s-aws-id anti-replay pin). The webhook reads the AccessEntry KV directly
over its own NATS connection.
Apiserver wiring: the k3s config skeleton sets
authentication-token-webhook-config-file; the webhook writes its own self-signed
serving cert + the apiserver kubeconfig at startup (ordered `before k3s`, with a
bounded wait in the k3s start_pre so the apiserver never loads a missing file).
* fix(eks): omit empty spec from token webhook TokenReview response
* fix(sts): presigned GetCallerIdentity must hash empty body, not UNSIGNED-PAYLOAD
VerifyPresignedGetCallerIdentity built the SigV4 canonical request with
"UNSIGNED-PAYLOAD" as the payload hash. That is an S3-only convention; botocore
and the aws-sdk presign non-S3 requests (sts:GetCallerIdentity, the shape `aws
eks get-token` produces) with the SHA256 of the empty string. Every real
get-token therefore failed signature verification with InvalidIdentityToken,
breaking the entire EKS IAM token-auth flow end to end.
Use the empty-string SHA256. The existing hand-rolled presign tests masked the
bug by signing with the same wrong constant; they now sign correctly, and a new
test presigns with the real aws-sdk-go v4 signer (the same canonicalisation the
CLI uses) so this can't regress silently.
Verified live: a real `aws eks get-token --cluster-name dev1` now resolves to
the caller principal over the awsgw eks.VerifyToken surface; an otherwise-valid
token presented for a different cluster is still rejected (anti-replay intact).
* fix(eks): match access-entry routes on escaped path so slashed principal ARNs resolve
IAM principal ARNs (user/admin, role/foo) embed a '/' that the AWS CLI
percent-encodes in the request path. The dispatcher matched r.URL.Path
(decoded), so %2F became a literal '/' and split the ARN across the
([^/]+) capture, dropping every describe/update/delete-access-entry and
associate/disassociate/list-access-policy call to InvalidActionException.
Match r.URL.EscapedPath() so the encoded ARN stays one segment, then
url.PathUnescape each captured param before it reaches the handler.
* test(eks): unit-cover access-entry scope validation, AWS converters, store guards
Ansible orchestrates the whole stack (spinifex + predastore + viperblock +
OVN + tofu hosts) and its spec/beads live in mulga. Moving it to top-level
mulga/ansible/ ends the per-change submodule-pointer-bump friction.
- git rm -r scripts/ansible/
- Makefile: drop ansible-dev-* / ansible-cluster-bootstrap targets + .PHONY
The tree lands at mulga/ansible/ (flat copy, Option B). History pivot:
tag feat/uplift-ansible-snapshot-pre-move marks the pre-move HEAD.
Plan: docs/development/improvements/ansible-uplift-to-mulga.md (in mulga).
An unresolvable certificate ARN was accepted into a listener record and
only failed later at config-render time, which aborted updateStoredConfig
wholesale and silently froze the load balancer's data-plane convergence.
Validate every certificate ARN against the ACM store (existence + account
ownership) in CreateListener, ModifyListener and AddListenerCertificates,
returning CertificateNotFound up front. Skipped when the ACM store is
unavailable, matching the nil-safe resolve path.
Resolve listener certificate ARNs to PEM via the ACM store (shared
JetStream KV, no cross-service hop), render HAProxy 'ssl crt' for secure
frontends, and deliver the combined cert+chain+key PEM to the LB agent
alongside the config. ConfigHash folds in cert content so rotation
triggers a reload; the agent writes each cert 0600 (path-constrained to
its cert dir) before the config + reload.
Phase B of the ELBv2 HTTPS epic (mulga-siv-180).
Stand up acm as its own AWS JSON 1.1 gateway service wiring the BYO-cert
import/store path ELBv2 HTTPS listeners need. ImportCertificate validates
tls.X509KeyPair, parses the leaf for domain/serial/expiry/key-algorithm,
mints arn:aws:acm:<region>:<acct>:certificate/<uuid>, and stores PEM (cert,
chain, key) in JetStream KV. DescribeCertificate/ListCertificates/
DeleteCertificate round-trip; access is account-scoped.
New gateway dispatch by X-Amz-Target (CertificateManager.<Action>); errors
share the AWS JSON 1.1 envelope with EKS. Wired through gateway/acm ops,
handlers/acm (service iface, store, NATS client), daemon handlers + acm.*
subscriptions, and service-interfaces.yaml.
Out of scope (future, same epic): RequestCertificate + DNS validation,
managed renewal, Get/ExportCertificate, tagging, ACM-PCA.
* feat(eks): cluster_state KV helpers — ClusterMeta + status CAS + prefix sweep
Stage 1 of Sprint 6b CreateCluster lifecycle. Introduces the first
read/write helpers against the eks-account-{accountID} bucket so later
stages (oidc keypair, NLB wiring, cluster reconciler) can persist state
without re-inventing CAS/sweep logic.
- ClusterMeta struct holds the DescribeCluster source of truth (status,
endpoint, OIDC issuer, CA, VPC config, control-plane VM + ENI, NLB ARN).
- ClusterStatus enum mirrors the AWS EKS string set verbatim.
- PutClusterMeta is unconditional (CreateCluster path); GetClusterMeta
returns ErrClusterNotFound on key absence so the service boundary can
map to ResourceNotFoundException.
- SetClusterStatus is revision-checked with bounded CAS retries so the
reconciler and CreateCluster body cannot wedge each other.
- DeleteClusterPrefix sweeps every clusters/{name}/* key; called from
DeleteCluster after the OIDC private key is zeroized.
Tests cover round-trip, missing/corrupt blob, idempotent status set,
DELETING transition from every prior state, concurrent revision bump
recovery, sibling-cluster preservation, empty-bucket no-op.
Plan: docs/development/feature/eks-6b-create-cluster.md
* feat(eks): per-cluster ECDSA-P256 OIDC keypair generation + zeroize
Stage 2 of Sprint 6b. Lays the at-rest primitives the CreateCluster body
will call before launching the K3s server VM. Pure crypto helpers — no
daemon, no launcher, no NATS bus.
- GenerateClusterOIDCKeypair mints an ECDSA-P256 keypair per cluster
(matches eks-v1.md Q8), PEM-marshals the private as PKCS8,
envelope-encrypts it via handlers_iam.EncryptSecret (AES-256-GCM,
AwsGW masterKey), writes the ciphertext to OIDCSigningKeyKey, and
publishes the public JWKS document to OIDCJWKSKey. Returns the JWKS
bytes so CreateCluster can hand them straight to user-data without
a follow-up KV read.
- JWKS shape mirrors what handlers_sts.JWKS already decodes for IRSA
verification (RFC 7517 EC P-256, kid = base64url SHA-256 of SPKI DER,
alg ES256, use sig). Types are defined locally to avoid a
handlers_eks → handlers_sts → handlers_eks import cycle; JSON wire
compat is the contract.
- LoadClusterOIDCPrivateKey is the recovery-path read for future
stages (reconciler restart, restore-snapshot tooling). Validates
PKCS8 parses to an ECDSA P-256 *PrivateKey and rejects anything else.
- ZeroizeClusterOIDCKey overwrites the encrypted blob with zeros of
the same length, then deletes. DeleteCluster calls this *first* so a
crash mid-teardown leaves no recoverable key material. Idempotent
on missing key — supports retry semantics.
Tests cover JWKS shape (RFC 7517 EC P-256 with 32-byte SEC1
coordinates), per-cluster key uniqueness, round-trip private-key
decrypt matches JWKS public, wrong-master-key fails, missing-key
ErrClusterNotFound mapping, zeroize idempotency, JWKS survives
zeroize (carries no secret material), padCoord left-pad correctness.
Plan: docs/development/feature/eks-6b-create-cluster.md
* feat(eks): cluster SG + NLB ensure/delete helpers
Stage 3 of Sprint 6b CreateCluster lifecycle. Adds the customer-account
SG provisioning + per-cluster NLB plumbing the CreateCluster body will
call between OIDC keypair (Stage 2) and K3s VM launch (Stage 4). Pure
interface-call helpers — no daemon, no KV, no NATS bus.
- EnsureClusterSGs creates (or returns) eks-cluster-{name}-control-plane-sg
and eks-cluster-{name}-nodegroup-sg in the customer VPC, tagged
spinifex:managed-by=eks + spinifex:eks-cluster=<name> + spinifex:eks-role.
Idempotent on group-name + vpc-id pair. DeleteClusterSGs is best-effort
with continue-on-error sweep.
- EnsureClusterNLB provisions the eks-<name> NLB (internal scheme), the
eks-<name>-cp target group (TCP, port 6443, target-type ip per eks-v1.md
Q13), and the TCP :443 forward listener. Returns ClusterNLB
{LoadBalancerArn, TargetGroupArn, ListenerArn, DNSName} so CreateCluster
can persist every ARN in ClusterMeta. Idempotent on cluster name;
resource-name length validated against the AWS 32-char cap so a too-long
cluster name fails with a clear error instead of a generic
InvalidParameterValue downstream.
- RegisterClusterTarget + DeregisterClusterTarget bind the K3s server VM
ENI IP to the TG on :6443. Stage 4 calls Register once the VM has its
ENI IP; DeleteCluster calls Deregister before VM termination.
- DeleteClusterNLB tears down NLB + TG. DeleteLoadBalancer already
cascades listener cleanup inside the ELBv2 impl. LB sweep error takes
precedence so DeleteCluster surfaces the more visible failure first
while the TG sweep still runs.
Narrow sgProvisioner + nlbProvisioner interfaces in-package give tests
hand-rolled recorder fakes; the production VPCService + ELBv2Service
satisfy them structurally so no adapter is needed at Stage 6 wire-up.
Tests cover fresh-creates with right names/tags/ports, idempotent reuse,
mixed-existence (control-plane exists, nodegroup created), name-too-long
rejection, empty-inputs rejection, missing-resource no-op on delete, and
first-error-surfaced sweep continuation on delete.
Plan: docs/development/feature/eks-6b-create-cluster.md
* feat(ec2): RunInstances honors pre-created NetworkInterfaceId
Closes the AWS-parity gap where RunInstances always auto-created a
primary ENI even when the caller specified
NetworkInterfaces[0].NetworkInterfaceId. The EKS Sprint 6b CreateCluster
launcher (Stage 4) needs to pre-create the K3s server VM's ENI so the
ENI can be tagged/owned distinctly from the launcher's instance; the
existing path silently created a second ENI and left the caller's
unattached.
Changes:
- ENICreator interface gains GetENI(accountID, eniID) → *ENIInfo. ENIInfo
is a translated struct (SubnetID, VpcID, PrivateIp, MAC, Status,
SecurityGroupIDs) so handlers/ec2/instance stays free of an import on
handlers/ec2/vpc.ENIRecord.
- daemon/vpc_adapter.go gains daemonENICreator.GetENI translating
VPCServiceImpl.GetENIRecord into ENIInfo.
- PrepareRunInstances inspects NetworkInterfaces[0].NetworkInterfaceId
after the existing subnet/SG lift block. If non-empty, the loop
dispatches to a new attachPreCreatedENI helper:
- Looks up the ENI under accountID.
- Rejects with InvalidNetworkInterface.InUse if Status == "in-use".
- Calls AttachENI to mark the ENI in-use and bind it to the new VM.
- Populates instance.ENIId / ENIMac and ec2.Instance network fields
(PrivateIp, SubnetId, VpcId, SecurityGroups, NetworkInterfaces[0]).
- Skips the auto-create + public-IP-auto-assign paths; the caller is
expected to manage public IP out-of-band.
- Auto-create path is left untouched for every other code path: callers
that pass SubnetId without a NetworkInterfaceId still get a fresh ENI
with the existing semantics.
Tests cover the four shapes: happy path (auto-create not called, attach
called once, instance populated), in-use ENI rejected with the canonical
AWS error code, lookup error surfaces verbatim, attach error surfaces
verbatim.
* feat(eks): K3s control-plane VM launcher with cloud-init user-data
LaunchK3sServerVM resolves the eks-server AMI by name, pre-creates the
customer-account control-plane ENI in the supplied subnet with the
control-plane SG, renders cloud-init user-data (first-boot env, OIDC
private key, K3s config.yaml with tls-san + service-account args), then
calls RunInstances with NetworkInterfaces[0].NetworkInterfaceId pointing
at the pre-created ENI. RunInstances failures roll back the ENI so the
caller does not leak a customer-account resource.
TerminateK3sServerVM tears the instance and ENI down for DeleteCluster;
missing identifiers are a no-op so retries stay safe.
ENI + VM both live in the customer account (deviation from plan §7 which
specified a SystemAccount-owned VM); cross-account ENI attach is deferred
to a follow-up bead.
* feat(eks): NATS bootstrap subscriber + cluster reconciler + leader lease
NATSBootstrap subscribes to the four single-shot subjects the K3s server VM
publishes (k3s-bootstrap-token, k3s-admin-kubeconfig, k3s-oidc-jwks, k3s-ca)
on eks.bus.{accountID}.{clusterName}.{kind}. Token and admin kubeconfig are
envelope-encrypted via handlers_iam.EncryptSecret and persisted to KV. The
published JWKS is cross-checked against the controller-generated keypair by
kid and rejected on mismatch. The CA lands on meta.CertificateAuthorityB64
via the new SetClusterCertificateAuthority CAS updater.
ClusterReconciler holds a leader lease on spinifex-eks-leader/{accountID}/
{clusterName} (60s TTL, 20s refresh) and drives:
- CREATING -> ACTIVE once the four bootstrap KV artifacts are present and
NLB /healthz responds 200.
- ACTIVE -> probe /healthz on the configured interval, warn on failure.
- DELETING / FAILED -> exit (DeleteCluster body owns teardown).
Loss of the leader lease (CAS conflict or different holderID) returns
ErrReconcilerLeaseLost so the caller can re-acquire from another node.
Tested with embedded JetStream + a stub HTTPDoer; race-clean.
* feat(eks): OIDC issuer URL builder + reconciler registry
ClusterOIDCIssuer renders the {gatewayBaseURL}/oidc/eks/{region}/{accountID}/
{clusterName} URL kube-apiserver advertises as service-account-issuer and
that STS / external IRSA validators dereference. Validates URL safety on the
path segments so a hostile cluster name cannot inject scheme/path fragments.
ReconcilerRegistry tracks per-(accountID, clusterName) reconciler goroutines
with idempotent Spawn (drop duplicate spawn calls), Stop (cancel one + wait
for release), and StopAll (graceful daemon shutdown). RunClusterReconciler
is the production SpawnReconcilerFn: constructs a ClusterReconciler,
acquires the leader lease, runs Run in a goroutine. Both helpers consumed
by the upcoming service_impl CreateCluster body.
* feat(eks): CreateCluster/DescribeCluster/ListClusters/DeleteCluster bodies
Wires the EKSServiceImpl orchestration deps (VPC SG/K3s/Subnet, NLB,
Instance launcher, AMI resolver, master key, gateway base URL, region,
holder ID, reconciler registry) and implements the four lifecycle
methods:
- CreateCluster: validate → CREATING meta → SGs → NLB → OIDC issuer +
keypair → K3s server VM (cloud-init carries OIDC private PEM) →
register ENI IP with NLB target group → spawn NATS bootstrap
subscriber + per-cluster reconciler via the registry.
- DescribeCluster: KV meta read → eks.Cluster shape, with
ErrClusterNotFound translated to ResourceNotFoundException at the
service boundary.
- ListClusters: KV key-prefix scan + sorted maxResults/nextToken
pagination.
- DeleteCluster: SetClusterStatus DELETING → zeroize OIDC key → stop
reconciler via registry → deregister NLB target + delete NLB →
terminate K3s VM → delete SGs → delete cluster KV prefix.
Daemon wiring:
- buildEKSServiceDeps loads MasterKey from BaseDir/config/master.key
best-effort, resolves the gateway base URL via the same precedence the
LB-agent block applies later, and passes VPCService + ELBv2Service +
ImageService directly while adapting InstanceServiceImpl via
daemonEKSInstanceLauncher (RunInstances passes through; Terminate-
Instances loops vmMgr.Get + StopOrTerminateInstance in-process).
- SubnetVPCResolver is satisfied by daemonEKSSubnetResolver wrapping
VPCService.GetSubnet.
- Startup spawns reconcilers for every persisted CREATING/ACTIVE cluster
via SpawnRegisteredReconcilers; shutdown calls eksService.Shutdown so
every per-cluster goroutine releases its lease before exit.
Test surface:
- service_impl_test: TestEKSServiceImpl_ClusterLifecycleShimMode covers
the four lifecycle methods when constructed via the back-compat shim
(CreateCluster/DeleteCluster → ServerInternal, DescribeCluster →
ResourceNotFound on empty bucket, ListClusters → empty list).
- daemon_handlers_eks_test: drop the four lifecycle subjects from the
NotImplemented routing loop; handler-side coverage now lives in
handlers/eks.
- awserrors: add ResourceNotFound (404) for the DescribeCluster path.
* fix(eks): load master.key from config dir not service data dir
eks_deps read master.key from {BaseDir}/config/master.key, but the daemon
systemd unit sets SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex (a per-service
data dir). admin init writes master.key to the shared config dir, so the
daemon looked in a path that only resolved when a {BaseDir}/config symlink
existed — awsgw has one, the daemon's nested data dir does not. Result:
MasterKey=nil, depsReadyForOrchestration() false, CreateCluster/DeleteCluster
collapse to ServerInternal.
master.key is a config artifact written next to spinifex.toml. Read it from
filepath.Dir(d.configPath), matching how the daemon already resolves
predastore.toml and TLS certs. No symlink or provisioning change needed.
* fix(eks): register + resolve eks-server AMI via tag, add import --ami-name
CreateCluster's K3s launch failed 'eks: AMI eks-server not found' on every
clean install. Two defects:
1. spx admin images import had no way to name a locally-built appliance AMI:
--name is a catalog lookup (rejects non-catalog values) and the AMI name
was hardcoded to ami-{distro}-{version}-{arch}. build-system-image.sh ran
'import --name $AMI_NAME' for eks-server/eks-agent/GPU images, which all
failed 'Image name not found in available images'.
2. The launcher resolved DescribeImages by name=='eks-server', a name the
pipeline could never produce.
Add --ami-name to override the registered AMI name on --file imports (fixes
eks-server, eks-agent, and GPU appliance imports). Resolve the EKS AMI by the
spinifex:managed-by=eks tag the pipeline already stamps, newest-wins on
multiple matches. Tag-based resolution survives the planned server+agent →
single EKS AMI unify untouched (role chosen per-instance at launch).
build-system-image.sh now passes --ami-name; Makefile gains
import-eks-server-image (IMPORT=1) and dev-install.sh imports the AMI when
DEV_INSTALL_EKS=1.
* fix(eks): give K3s server VM NATS token + CA so bootstrap can publish
The K3s control-plane VM was launched with NATSURL=ConnectedUrl() (loopback,
unreachable from the VM) and no auth material, while spinifex NATS requires a
shared token over TLS. The VM could not connect to NATS, so it never published
its four one-shot bootstrap messages and the cluster stayed in CREATING.
Hand the VM the daemon's shared NATS token + CA PEM and a VM-reachable NATS URL
(mirrors the ELBv2 LB-agent shared-credential pattern; spinifex has no nkeys
hierarchy to scope per-principal creds against).
- K3sServerInput: drop dead NATSCredsContent; add NATSToken + NATSCACert,
required by validateK3sServerInput. user-data writes SPINIFEX_NATS_TOKEN,
SPINIFEX_NATS_CA + the CA file; first-boot.env tightened to 0600.
- EKSServiceDeps + CreateCluster: pass NATSURL/NATSToken/NATSCACert.
- buildEKSServiceDeps: read CA from config.NATS.CACert; extract
resolveGatewayHost(); add resolveNATSURL() (same reachable host, NATS port).
- k3s-first-boot.sh: build nats CLI args with --token + --tlsca.
* fix(eks): retry transient bootstrap persists, fail cluster on permanent error
The NATS bootstrap subscriber tore down on the first persist error and the
cluster sat in CREATING forever. Core-NATS delivers a publication once, so a
transient KV blip during persist was unrecoverable and a malformed payload
(bad JWKS kid, non-base64 CA, empty envelope) silently stranded the cluster.
- persistWithRetry rides out transient persist failures in-handler (3 attempts,
200ms backoff) instead of unsubscribing and waiting for a re-publish that
never comes.
- errBootstrapPermanent classifies validation failures (malformed envelope,
empty fields, JWKS kid mismatch, non-base64 CA) as non-retryable so they
return immediately.
- spawnBootstrap marks the cluster FAILED (with reason) when Run exits with a
non-context error, so DescribeCluster surfaces the cause and DeleteCluster
can reclaim resources.
- MarkClusterFailed transitions to FAILED+StatusReason only from CREATING, so a
late bootstrap error cannot clobber a concurrent delete or healthy cluster.
- casUpdateMeta extracts the revision-checked RMW loop shared by the meta
setters.
* fix(eks): resume NATS bootstrap for CREATING clusters on daemon restart
SpawnRegisteredReconcilers re-spawned only the reconciler on boot, never the
bootstrap subscriber. A daemon restart while a cluster was CREATING (K3s VM
still booting) lost the subscriptions; the VM published its four one-shot
core-NATS messages to nobody and the cluster hung in CREATING forever.
- BootstrapPendingKinds reports which artifacts (token, kubeconfig, CA) are
still missing from KV. JWKS is excluded: it persists no artifact and
re-waiting on a once-published subject would deadlock — its cross-check is
best-effort across a restart.
- RunForKinds subscribes to a subject subset (nil = all four); Run delegates to
it. subjectHandlers maps each kind to its handler and rejects unknown kinds.
- SpawnRegisteredReconcilers resumes bootstrap for CREATING clusters, waiting
only on the missing artifacts, so a mid-CREATING restart still drives the
cluster to ACTIVE.
* fix(eks): DeleteCluster must not wipe KV or report success on teardown failure
DeleteCluster ran zeroize, NLB teardown, and VM termination as warn-and-continue
then unconditionally swept the per-cluster KV prefix and returned success. A
failed NLB or VM teardown leaked billable resources with no owning record (the
meta holding their ARNs/IDs was erased) and told the operator the cluster was
gone. Zeroize — a stated security guarantee of no recoverable key material —
had its failure ignored too.
- Zeroize, NLB delete, and VM terminate are now blocking: their failures
accumulate (errors.Join) and abort the call before the KV sweep, leaving the
cluster DELETING and returning the error so a retry can still reach the ARNs.
- KV prefix sweep runs only after NLB + VM teardown confirm.
- SG cleanup stays best-effort (plan step 5) — a leaked SG strands no owning
record — and target deregister stays best-effort since DeleteClusterNLB tears
down the whole NLB + target group.
* fix(eks): persist NLB + control-plane ARNs mid-CreateCluster so failures don't leak
CreateCluster set meta.NLBArn / target-group / instance / ENI in memory and
persisted them only in the final PutClusterMeta, after LaunchK3sServerVM. Any
failure between EnsureClusterNLB and that persist left the NLB + target group +
listener + backing LB VM created but unrecorded; markFailed did no rollback and
DeleteCluster keys teardown off the persisted ARNs, so the resources leaked.
- Persist meta immediately after EnsureClusterNLB (records NLB/TG ARNs) and
again after LaunchK3sServerVM (records instance + ENI), before the next
fallible step.
- A failed create now leaves a FAILED meta carrying the resource references, so
the DeleteCluster teardown path (165.9) reclaims every NLB/TG/listener/VM.
- Tests: NLB-arn-persisted-before-later-failure, failed-create-then-delete
reclaims NLB, and a CreateCluster happy path. Shared eksServiceFixture
extracted from the delete-cluster fixture.
* fix(sts): parse path-based EKS OIDC issuer so IRSA accepts 6b cluster tokens
ParseEKSIssuerURL required an `oidc.eks.` host prefix and an exactly-2-segment
path (AWS hosted-EKS shape), but ClusterOIDCIssuer emits — and awsgw serves —
https://{gateway-host}/oidc/eks/{region}/{accountID}/{clusterName}. Every 6b
cluster's iss claim was rejected with "issuer host must start with oidc.eks",
so AssumeRoleWithWebIdentity would fail all IRSA tokens once 6e serves them.
Reconcile on the path-based shape, which is what is actually reachable on
bare-metal/on-prem (no oidc.eks.* vhost or DNS exists). Host trust comes from
the IAM OIDC-provider registration (the full issuer string is the registry key)
and the JWKS signature check, not a host pattern — so the parser drops the host
prefix rule and instead pins accountID/clusterName at fixed positions in a
strict 5-segment /oidc/eks/{region}/{accountID}/{clusterName} path. A crafted
iss with extra or missing segments still fails closed.
Adds a cross-component round-trip test asserting a ClusterOIDCIssuer-built iss
parses back to the same (accountID, clusterName).
* fix(eks): reclaim FAILED clusters on retry + cluster-scoped exists/not-found errors
A failed CreateCluster left a FAILED ClusterMeta that poisoned retries: the
preflight returned ErrorELBv2TargetGroupInUse ("The target group is currently
in use...") for a cluster-exists condition, and because that maps to a
retryable code the SDK's auto-retry surfaced ResourceInUseException instead of
the original failure. FAILED clusters were also a dead state — visible to
neither a clean recreate nor a clear lifecycle.
- Add EKS-specific ResourceInUseException / ResourceNotFoundException codes with
cluster-scoped messages; CreateCluster's exists check and DescribeCluster /
DeleteCluster not-found now use them instead of the ELBv2 / EC2 codes.
- CreateCluster reclaims a FAILED cluster: it purges the prior attempt's
recorded resources and clears KV before recreating, so a retry (including the
SDK auto-retry) is idempotent rather than blocked. Teardown failure surfaces
rather than recreating on top of orphans.
- Extract purgeClusterInfra, shared by DeleteCluster and the reclaim path
(blocking zeroize/NLB/VM, best-effort SG, KV sweep only after teardown
confirms).
- FAILED clusters remain observable via Describe (status=FAILED) and List — no
state where create blocks but describe/list show nothing.
* fix(eks): fail CREATING clusters that exceed create timeout
A per-cluster reconciler that never observes a ready control plane (bootstrap
artifacts missing or healthz unreachable) previously stayed CREATING forever.
Add a configurable create timeout (default 15m): once meta.CreatedAt is older
than the deadline and the cluster is still not ready, the reconciler marks it
FAILED with a reason, exiting the loop. Within the deadline it stays CREATING.
* fix(eks): surface ACTIVE cluster health in describe-cluster
The reconciler only slog.Warn'd a failing /healthz probe for an ACTIVE
cluster, so describe-cluster kept reporting a dead control plane as healthy.
Add HealthIssue + LastHealthProbe to ClusterMeta, write them from the ACTIVE
reconcile branch (recording the failure reason, clearing on recovery), and map
a non-empty HealthIssue to a ClusterHealth ClusterUnreachable issue in
describe-cluster.
* test(eks): add plan-mandated CreateCluster/DeleteCluster service cases
service_impl_test only covered shim mode. Add the four cases the plan's Testing
section requires: CONFIG_MAP auth mode rejected (and API accepted), missing
subnetIds rejected, DescribeCluster 404 with full deps wired (past the shim
short-circuit), and the DeleteCluster zeroize-before-teardown ordering proven
by forcing a teardown failure and asserting the OIDC key is already wiped.
* refactor(eks): return OIDC private-key PEM from generator, drop re-read
CreateCluster discarded GenerateClusterOIDCKeypair's return then did a second
KV get + AES-GCM decrypt via loadOIDCPrivateKeyPEM to re-derive the same PEM the
generator already had in hand. Return the plaintext PEM (alongside the JWKS)
from the generator and use it directly; remove loadOIDCPrivateKeyPEM and the now
-unused crypto/x509 + encoding/pem imports in service_impl.
* fix(eks): make CreateCluster/DeleteCluster failures diagnosable
Unmet orchestration deps and a missing eks-server AMI both collapsed to a bare
ServerInternal with no request-time log, leaving the client a 'contact AWS
re:Post' message and zero cause. Enumerate the specific missing deps and log
them at ERROR, returning ServiceUnavailable (an operator/config fault, not an
unrecoverable internal one). Add an ErrEKSServerAMINotFound sentinel so a
missing control-plane AMI is logged and surfaced as ServiceUnavailable too.
* fix(eks): serialize REST-JSON responses via jsonutil so the AWS SDK can parse them
WriteJSONResponse marshaled aws-sdk-go *Output structs with encoding/json,
which ignores their locationName tags and emits Go PascalCase keys. EKS
clients parse restjson on the lowercase locationName, so every field of
list/describe/create/delete responses was silently dropped (empty endpoint,
null identity.oidc.issuer). Marshal via jsonutil.BuildJSON — the same
marshaler the SDK uses for restjson bodies — which honors locationName.
Adds wire-shape tests asserting lowercase keys present and PascalCase absent.
* feat(eks): launch K3s server VM via boot-agnostic system-instance path
The EKS K3s control-plane VM launched through the plain tenant RunInstances
path, so it had only its VPC ENI and no management-bridge NIC -- no route to
the daemon's NATS endpoint. It never published its bootstrap subjects and the
cluster stayed CREATING.
Generalize LaunchSystemInstance to a boot-agnostic contract supporting both
system-instance boot styles:
- BootDirect: direct-boot microVM (fw_cfg blobs) -- ELBv2 LB agent
- BootAMI: AMI boot + cloud-init seed (q35) -- EKS K3s server VM
The launch contract moves to a neutral handlers/sysinstance package so EKS can
launch system VMs without an eks->elbv2 dependency; handlers/elbv2 re-exports
the types as aliases. The BootAMI branch mirrors the daemon's
Prepare -> Insert -> Launch split and allocates the mgmt NIC on the VM record
between Insert and Launch, so the cloud-init network-config renders a static
mgmt0 interface. The EKS NATSURL is now sourced from the mgmt-bridge IP (the
on-link address the VM's mgmt NIC can reach), falling back to the gateway-host
resolution when no mgmt bridge exists.
Verified: K3s VM now boots from the eks-server AMI under q35 with a mgmt0 NIC
(10.15.8.x), reachable from the host over br-mgmt, with NATS on-link at the
bridge IP. No AMI rebuild required.
* fix(eks): make DeleteCluster K3s teardown idempotent so it can't wedge
TerminateK3sServerVM treated every DeleteNetworkInterface error -- and every
TerminateSystemInstance error -- as blocking. Because instance terminate is
async, the immediate ENI delete races it: the first attempt fails
InvalidNetworkInterface.InUse, and once the instance-terminate cascade releases
the ENI every retry fails InvalidNetworkInterfaceID.NotFound. Both joined into
purgeClusterInfra's teardownErrs gate, which short-circuits before the SG sweep
and KV prefix delete -- so once the ENI was gone the cluster was stuck DELETING
permanently, leaking its control-plane + nodegroup SGs and KV meta.
Classify the teardown errors instead of blanket-propagating:
- ENI InvalidNetworkInterfaceID.NotFound / InvalidNetworkInterface.NotFound
-> idempotent success (the cascade or a prior retry already deleted it);
must not block the downstream SG + KV sweep.
- ENI InvalidNetworkInterface.InUse (VM still terminating) and any other
error -> retryable; surfaced so the cluster stays DELETING and the
reconciler retries once the instance releases the ENI.
- Instance already gone -> idempotent. TerminateSystemInstance now returns a
sysinstance.ErrSystemInstanceNotFound sentinel that the EKS path tolerates
via errors.Is, so a retried delete after the VM drained completes.
Add awserrors.IsErrorCode(err, code) as the shared delete-error classifier
(substring match through %w wrapping) and route the lone ad-hoc strings.Contains
guard in daemon/vm_adapters.go through it.
Verified: delete-cluster on three clusters left wedged DELETING / FAILED (ENI +
VM already gone) all complete -- clusters erased, SGs + ENIs reclaimed, no leak.
* fix(eks): gate ACTIVE on verified JWKS cross-check, not pre-seeded key
The controller pre-seeds OIDCJWKSKey at create time, so its presence
proved nothing about the running cluster's actual signing key. A cluster
could reach ACTIVE — and start signing IRSA tokens — before the K3s VM's
published JWKS was ever cross-checked against the generated keypair.
Add an OIDCJWKSVerifiedKey marker written by persistJWKS only after
assertJWKSMatch passes. The reconciler's ACTIVE gate and
BootstrapPendingKinds now key off the marker, not the pre-seeded JWKS.
assertJWKSMatch is map-based and also rejects kty mismatch.
* fix(eks): collapse duplicated gateway-host switch into resolveGatewayHost
startCluster reimplemented the 5-case gateway-host precedence inline,
duplicating resolveGatewayHost (used by the OIDC issuer and EKS NATS URL).
If one drifted, the OIDC issuer host embedded in cluster tokens would
diverge from the host LB VMs use to reach the daemon -> IRSA iss mismatch
and cluster-wide token rejection.
startCluster now calls resolveGatewayHost and derives the multi-node
mgmt host route (mgmtRouteVia) from the resolved host. resolveGatewayHost
carries the canonical case enumeration; a table test pins all five
branches plus the case-1 skip guards and the no-host fallthrough.
* fix(eks): make eks-server VM boot diagnosable + fix first-boot publisher crash
Three issues found while diagnosing why K3s clusters never leave CREATING:
1. setup.sh: the stock Alpine cloud image lists `console=tty0` LAST in
default_kernel_opts, so Linux binds /dev/console to the framebuffer and the
host-side serial capture sees only kernel dmesg — OpenRC, cloud-init, and
k3s-first-boot output were invisible. Reorder so console=ttyS0 is last.
2. k3s-first-boot.initd: command_background was "no" while output_log/error_log
were set; start-stop-daemon rejects --stdout/--stderr without --background
("Failed to start k3s-first-boot"), so the bootstrap publisher never ran and
the cluster wedged in CREATING. Background it (the publisher waits up to 5min
on K3s /healthz and must not block the runlevel anyway) and route its output
to the serial console, which the orchestrator captures (the VM has no login).
3. k3s-first-boot.sh: on the /healthz timeout, dump the tail of /var/log/k3s.log
to the console so the apiserver's own failure is visible host-side.
* fix(eks): seed K3s apiserver a public OIDC key, not the private key
buildK3sUserData pointed both kube-apiserver service-account args at the same
private-key file. --service-account-key-file requires a PUBLIC key — k8s
ParsePublicKeysPEM rejects a private-key PEM with "data does not contain any
valid RSA or ECDSA public keys" and the apiserver crash-loops, so K3s never
reaches /healthz, first-boot never publishes, and the cluster wedges in
CREATING.
Add PublicKeyPEMFromPrivate (PKIX public PEM from the PKCS#8 private key) and
seed it to a separate /etc/rancher/k3s/oidc-signing-key.pub.pem. Point
service-account-key-file at the public file; keep service-account-signing-key-file
on the private file. Runtime-verified: the apiserver crash is gone and K3s now
starts and schedules system pods.
* feat(eks): egress-only SNAT + DNS for K3s control-plane VM
The K3s server VM (a hidden system instance) could not pull system-pod
images: it had no internet egress and no working resolver, so
coredns/traefik/metrics-server never started and /healthz timed out.
Egress: give the single control-plane VM egress-only internet, confined
to its /32, with no inbound exposure and no customer-visible resource.
- policy/route.go: SystemInstanceEgressPriority (1200, above the drop
gate) + AddSystemInstanceEgress/Delete — a reroute scoped to
ip4.src == <vmIP>/32 so peers in a shared subnet are unaffected.
- policy/nat.go: AddSystemInstanceSNAT/Delete — a plain snat (not
dnat_and_snat, so no inbound path), idempotent on its pool external IP.
- external/igw.go: L5 EnsureSystemInstanceEgress/Remove resolves the
gateway nexthop and attaches the reroute + snat (mirrors the NAT-GW
egress path; ADR-0006 S1/S4/S5 preserved).
- subscribers: vpc.add/delete-system-egress topics + handlers.
- handlers/eks: allocate a hidden (ManagedBy=eks) pool address on
CreateCluster and publish add; release + publish delete on teardown.
EIP dep wired through EKSServiceDeps + the daemon.
DNS: the eks-server AMI is Alpine/dhcpcd and /etc/resolv.conf is a
dangling symlink, so DHCP-supplied resolvers are never persisted. Write
a real /etc/resolv.conf from the K3s VM cloud-config via bootcmd (not
write_files — a write_files entry follows the dead symlink, fails, and
aborts the whole block, dropping first-boot.env + the k3s config).
* feat: EKS 6b single-node SQLite pivot + bootstrap-envelope fixes
Switch the K3s server VM from embedded etcd to embedded SQLite (kine)
for the single-node v1, so CreateCluster reaches a healthy control plane.
etcd's per-commit fsync stalls past the apiserver deadline on viperblock's
deferred-durability block path; SQLite tolerates it.
- k3s server: omit cluster-init (selects SQLite/kine), add anonymous-auth=true
so the reconciler's anonymous /healthz NLB probe returns 200.
- first-boot readiness gate: anonymous /healthz (401 under CIS anonymous-auth
=false) -> authenticated 'kubectl get --raw /readyz'.
- first-boot publish: emit a JSON BootstrapEnvelope for all four subjects
(token, admin-kubeconfig, oidc-jwks, ca); jwks and ca were absent before.
- nats pub: --stdin -> --force-stdin (nats CLI 0.4.0).
- manifest: add jq (builds the bootstrap-message JSON envelopes).
* feat: EKS reconciler probes control-plane mgmt IP until DNS lands
The host-side cluster reconciler cannot resolve or route to the VPC-internal
NLB DNS endpoint (no authoritative DNS / Eclipso yet), so CreateCluster timed
out in CREATING and was marked FAILED even though the control plane was healthy.
Probe the control-plane VM's br-mgmt NIC address directly — the one apiserver
endpoint reachable from the daemon. Capture the mgmt IP from the launch result
(SystemInstanceOutput.MgmtIP -> K3sServerOutput.MgmtIP -> ClusterMeta.
ControlPlaneMgmtIP) and build the reconciler health URL as
https://<mgmtIP>:6443/healthz. The probe client already sets InsecureSkipVerify,
so the mgmt IP need not be in the apiserver cert SAN.
Workaround until authoritative DNS + a host-reachable NLB front-end land, at
which point spawnReconciler reverts to the NLB endpoint.
* fix(elbv2): make resource deletes idempotent on absent
DeleteLoadBalancer, DeleteTargetGroup, DeleteListener and DeleteRule
returned a NotFound error when the target was already gone. Real AWS
ELBv2 deletes are idempotent (success on absent), so terraform-provider-aws
fails a destroy step on retry. Return an empty success output instead,
mirroring the KV store which already tolerates absent keys.
Adds TestRLC1_ELBv2DeleteIdempotentOnAbsent enforcing the contract across
the ELBv2 delete surface; converts the three _NotFound delete tests to
assert idempotent success.
Refs mulga-siv-172 (idempotency clause; cascade-composition clause stays open).
* feat: ELBv2 AddTags/RemoveTags, SetSecurityGroups, SetIpAddressType
- O(1) GetListenerByArn via ARN-parse short-ID lookup + ARN equality check
- persist tags on listeners/rules; seed at create time
- AddTags/RemoveTags across LB, target group, listener, rule resources
- SetSecurityGroups (ALB only) reattaches SGs to each ENI via live port update
- SetIpAddressType (ipv4-only, idempotent)
- wire all four ops through gateway, NATS client, and daemon handlers
- frontend: editable Tags tab on LB/TG detail, Security groups tab on LB detail
* feat(elbv2): redirect listener/rule actions
- add redirect action type; parse/store/read-back RedirectActionConfig
- validate status code + render-safe fields (strip AWS #{...} placeholders)
- accept redirect and fixed-response on listener rules (not just defaults)
- HAProxy: http-request redirect scheme (HTTP->HTTPS fast path) or rebuilt
location with placeholder->fetch translation; synthetic per-rule backends
* refactor: dedupe ELBv2 attribute handlers and harden store lookups
Collapse the four near-identical Modify/Describe attribute handlers (target
group and load balancer mirror pairs) onto generic helpers in both the service
impl and the gateway, and table-drive the mirror-pair tests over a TG/LB kind.
- service_impl_attributes.go: generic attrResource/modifyResourceAttributes/
describeResourceAttributes plus sdkAttrsIn/sdkAttrsOut; folds in nil-input
guards, explicit empty-accountID rejection, fresh-string output, and a
no-op skip-persist for unchanged Modify calls.
- gateway/elbv2/attributes.go: generic attributeOp + shared validateAttrArn,
replacing four per-endpoint files.
- types.go: default attribute sets are now package-level maps cloned on access
instead of rebuilt per Describe; document the access_logs empty-string defaults.
- store.go: drop the stale 'linear scan' docstring and log ARN mismatches on
GetLoadBalancerByArn/GetTargetGroupByArn instead of silently returning nil.
Adds cross-account isolation, wrong-account no-mutation, and empty-string-clear
coverage.
* feat: support ip target type in ELBv2 RegisterTargets
RegisterTargets resolved every target ID through an instance ENI lookup, so
ip-type target groups silently no-op'd with an empty PrivateIP and never
received traffic.
- types.go: add TargetTypeInstance/TargetTypeIP constants.
- CreateTargetGroup: default to instance; reject target types other than
instance/ip with InvalidParameterValue.
- RegisterTargets: resolve PrivateIP per target type via resolveRegisteredTargetIP
— ip targets use the supplied IP directly (validated with net.ParseIP),
instance targets keep the ENI lookup and reject IP-shaped IDs.
* fix: cascade rule deletion in ELBv2 DeleteLoadBalancer
DeleteLoadBalancer deleted listeners via store.DeleteListener directly, which
bypasses the rule cascade in the DeleteListener handler. Orphan rules survived
LB teardown and pinned their target groups as ResourceInUse, leaving the TG
permanently undeletable and breaking tofu destroy on multinode.
Extract deleteListenerCascade (rules then listener record) and call it from both
DeleteListener and the DeleteLoadBalancer cleanup loop.
Regression: TestDeleteLoadBalancer_CascadesRuleDeletion.
* fix: harden ELBv2 delete lifecycle against orphaned cascades
- DeleteLoadBalancer: fail-fast on listener-list/cascade error and run the
rule cascade before backing-artifact teardown, so a transient failure
leaves the record intact for retry instead of orphaning rules.
- DeleteTargetGroup: in-use scan skips listeners/rules whose owning load
balancer no longer exists, so a partial-teardown orphan can't pin a target
group as ResourceInUse permanently.
- DeleteRule: propagate the listener lookup error instead of discarding it,
so a failed HAProxy reload is no longer reported as success.
- Add lifecycle invariants for no-orphan completeness, no store-bypass
cascade, and target-group deletability after LB teardown.
- Drop plan/bead references from comments.
* test: add NATS e2e round-trips for new ELBv2 ops
Cover the client→NATS→handler marshalling path that unit tests skip:
tags lifecycle (AddTags/DescribeTags/RemoveTags), target-group and
load-balancer attribute round-trips, redirect-action default listener,
ip target-type registration, and SetSecurityGroups/SetIpAddressType.
* fix(elbv2): correct misleading InvalidConfigurationRequest message
The error code is shared by five unrelated conditions (dualstack
rejection in SetIpAddressType, cross-listener rule moves, unsupported
rule action types, and the two genuine NLB security-group cases), but
its canonical message claimed security groups were unsupported on
network load balancers. Callers hitting it from SetIpAddressType or the
rule paths got a message about security groups and NLBs that did not
match their request.
Neutralize to a generic, condition-agnostic message. The AWS error code
is unchanged.
* style(ui): oxfmt security-groups-editor test
agent.go signed gateway requests with aws-sdk-go v1's v4.Signer. Switch to
predastore/auth.SignReq (a thin wrapper over aws-sdk-go-v2's v4.Signer) so the
monorepo runs a single signer version and drops the last direct dependency on
aws-sdk-go v1's signer subpackage.
- Replace the *v4.Signer field with the raw access/secret keys.
- signedPost computes the hex SHA-256 payload hash and calls auth.SignReq.
- Add TestSignedPost_ProducesVerifiableSignature: the signed request parses and
verifies server-side via auth.ParseReq/Verify with a matching body hash.
Shared-ingress ALBs use a fixed-response default action (HTTP 404) with
host-header rules forwarding to per-app target groups. Two gaps broke this:
- The HAProxy generator read DefaultActions[0].TargetGroupArn unconditionally,
so a fixed-response default (empty ARN) rendered `default_backend bk_`, a
dangling reference that made HAProxy fail to start — the listener never bound
:80 and the LB served no traffic. Store and render the fixed-response config
as a synthetic backend (`http-request return status N ...`), with status,
content-type, and body validated before interpolation.
- ListenerAction dropped the fixed-response config on create/modify, so
DescribeListeners read back an empty action and tofu saw a perpetual diff.
Preserve it through CreateListener/ModifyListener and the SDK conversion.
- DescribeTags returned InvalidParameterValue for listener-rule ARNs; resolve
the rule and return an (untagged) descriptor instead.
Adds generator tests for the fixed-response default (with host-header rules)
and unsafe-body rejection, plus DescribeTags listener-rule coverage.
sudo mknod created real /dev/console and /dev/null char-device nodes on the
host under $CHROOT_DIR. Real device nodes on the host filesystem are a footgun:
a stale-image cleanup, tmp reaper, or an empty $CHROOT_DIR (which would redirect
the recursive /dev removal at the host root) could clobber host /dev/null and
force a reboot, breaking CI servers.
Write the two static nodes the kernel needs into the cpio archive inside a
fakeroot session instead. mknod/cpio see faked device entries and emit real
char-device records into the archive, while the host filesystem never gets an
actual node. init mounts devtmpfs over /dev at runtime for everything else.
Also guard $CHROOT_DIR against empty/unexpected values before any destructive
removal, and add fakeroot to the required-tools check.
Daemons are <tspanfill="#F87171"font-weight="600">stateless</tspan> — scale horizontally by starting more.
No etcd, no Kubernetes, no external control plane. Systemd units, a NATS cluster, and your hardware.
</text>
</g>
</svg>
<svgxmlns="http://www.w3.org/2000/svg"width="1020"height="770"viewBox="0 0 1020 770"role="img"aria-label="Spinifex message-driven architecture: standard AWS clients call the AWS Gateway over HTTPS with SigV4, which publishes to a NATS bus; stateless ec2/ebs/s3/vpc/iam daemons answer and drive local backends (QEMU/KVM, Viperblock, Predastore, OVN/OVS, identity store).">
<textx="48.0"y="697.0"font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif"font-size="12"fill="#5B636C"font-weight="400"><tspanfill="#EE4B12"font-weight="700">Queue groups</tspan><tspanfill="#5B636C"> load-balance ops across daemon replicas. </tspan><tspanfill="#EE4B12"font-weight="700">Fan-out</tspan><tspanfill="#5B636C"> (no queue group) dispatches describe calls to every node.</tspan></text>
<textx="48.0"y="720.0"font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif"font-size="12"fill="#5B636C"font-weight="400"><tspanfill="#5B636C">Daemons are </tspan><tspanfill="#EE4B12"font-weight="700">stateless</tspan><tspanfill="#5B636C">: scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Systemd units, a NATS cluster, and your hardware.</tspan></text>
<textx="24"y="30"font-family="'Segoe UI', system-ui, sans-serif"font-size="11"fill="#a1a1aa"letter-spacing="2.5"font-weight="600">ECS · EC2 LAUNCH TYPE ON SPINIFEX</text>
<textx="876"y="30"font-family="'SF Mono', ui-monospace, monospace"font-size="10"fill="#71717a"text-anchor="end">container instances from spinifex-ecs-node AMI · awsvpc + ALB</text>
<textx="450"y="410"font-family="'Segoe UI', sans-serif"font-size="9.5"fill="#71717a"text-anchor="middle">No ECS launch API — you add capacity by booting EC2 instances from the spinifex-ecs-node AMI (the console's "Provision capacity" wraps RunInstances)</text>
<textx="450"y="426"font-family="'Segoe UI', sans-serif"font-size="9.5"fill="#71717a"text-anchor="middle">The service keeps the desired task count running and registers each task's ENI IP with the ALB target group</text>
<textx="450"y="206"font-family="'Segoe UI', sans-serif"font-size="9.5"fill="#7ee787"text-anchor="middle">managed node group · workers join the control plane</text>
<textx="450"y="404"font-family="'Segoe UI', sans-serif"font-size="9.5"fill="#71717a"text-anchor="middle">You provide the VPC, subnets, and IAM roles · Spinifex provisions the control plane, node-group VMs, and security groups</text>
<textx="450"y="420"font-family="'Segoe UI', sans-serif"font-size="9.5"fill="#71717a"text-anchor="middle">Node groups always boot the registered eks-node image · workers need egress (IGW / NAT) to pull images</text>
<textx="450"y="58"font-family="'Segoe UI', system-ui, sans-serif"font-size="20"font-weight="600"fill="#f5f5f5"text-anchor="middle">AWS-style workflows, on your own stack</text>
<svgxmlns="http://www.w3.org/2000/svg"width="1000"height="664"viewBox="0 0 1000 664"role="img"aria-label="The Spinifex stack: applications and AWS tooling on top, the Spinifex AWS-compatible service layer, running on standard Linux on CPU/GPU hardware, deployable to Neocloud, on-prem or edge.">
<textx="136.0"y="249.8"font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace"font-size="11"fill="#9AA1A8"font-weight="400"text-anchor="start">Runs the same on a single node, cluster, edge, or air-gapped rack.</text>
echo"::error::${label} names suite '${suite}', which is not eligible for that topology (eligible: ${eligible}). Fix the list in .github/scripts/suite-sets.sh." >&2
description:"spinifex ref to checkout/test (default: triggering ref)"
required:false
type:string
default:""
otel-attrs:
description:"OTEL_RESOURCE_ATTRIBUTES value for this run, e.g. ci.run_id=123,ci.cell=20 (caller-specific, so it's an input rather than a workflow-level default)"
required:false
type:string
default:""
permissions:
contents:read
@@ -9,10 +21,14 @@ permissions:
env:
GIT_BRANCH:${{ github.ref_name }}
# Same central ES/APM sink as the other four e2e workflows. Bare-metal had
# no default here before — this cell's telemetry never reached the sink at
# all (not just missing ci.cell), see bm-bootstrap.sh's telemetry.env step.
# Manual-work mutex gate — see setup note 4 above. Checked BEFORE any
# SSH to wattle so a paused run touches the box zero times.
- name:Manual-work mutex gate
id:gate
env:
REPO_VAR_PAUSED:${{ vars.CI_GPU_PAUSED }}
run:|
set -euo pipefail
PAUSED=false
if [ -f "$HOME/.ci-gpu-paused" ]; then
PAUSED=true
echo "::warning::ci-gpu runner paused via ~/.ci-gpu-paused sentinel on the runner — skipping this run (manual GPU work in progress on wattle). Remove the file to resume CI."
fi
if [ "$(echo "${REPO_VAR_PAUSED:-}" | tr '[:upper:]' '[:lower:]')" = "true" ]; then
PAUSED=true
echo "::warning::ci-gpu paused via repo/environment variable CI_GPU_PAUSED=true — skipping this run."
fi
echo "paused=${PAUSED}" >> "$GITHUB_OUTPUT"
- name:Paused — nothing to do
if:steps.gate.outputs.paused == 'true'
run:echo "ci-gpu is paused; job exits cleanly without touching wattle. See gate step above for the reason."
echo "::error::GPU e2e suite failed (rc=${{ steps.e2e.outputs.rc }}) — see analysis above; raw ES/APM traces at ${SPINIFEX_OTLP_ENDPOINT} (ci.run_id=${{ github.run_id }})"
<img src=".github/assets/banner.svg" alt="Spinifex: the open, AWS-compatible cloud you run yourself. EC2, EBS, S3, VPC, and IAM on your infrastructure, at the edge, or on a Neocloud." width="900">
</p>
<p align="center">
@@ -11,6 +11,8 @@
<p align="center">
<a href="#what-is-spinifex">What is Spinifex?</a> ·
# Spinifex: An Open Source AWS-Compatible Stack for Bare-Metal, Edge, and On-Prem Deployments
# Spinifex: the open, AWS-compatible cloud you run yourself
Spinifex developed by [Mulga Defense Corporation](https://mulgadc.com) is an open source infrastructure platform that brings the core services of AWS—like EC2, VPC, EBS, and S3—to environments where running in the cloud isn't an option. Whether you're deploying to edge sites, private data-centers, or need to operate in low-connectivity or highly contested environments, Spinifex gives you AWS-style workflows on your own hardware.
Spinifex, built by [Mulga](https://mulgadc.com), recreates the AWS services your
software already uses (EC2, EBS, S3, VPC, IAM) on infrastructure you control.
Move workloads off the hyperscalers, run AI at a fraction of the cost, or take
the cloud to the edge, without rewriting a line. The exact build you ship to AWS,
Spinifex serves on your own hardware: same APIs, same Terraform, same SDKs.
## What is Spinifex?
Spinifex replicates essential AWS primitives—virtual machines, block volumes, and object storage—using lightweight, self-contained components that are easy to deploy and integrate.
Spinifex is an open-source infrastructure platform that recreates the AWS service
surface (EC2, EBS, S3, VPC, IAM) on bare-metal, edge, and on-premises
environments. Run cloud-native software without a hyperscaler.
It's designed for developers and operators who need:
Most AWS alternatives wrap an API and call it a cloud. We rebuilt the engineering
underneath: distributed object storage with erasure coding, block storage built
to survive failure, and compute on bare metal. That depth is the reason your
software runs unchanged, instead of rewritten.
- Cloud-like infrastructure without the cloud
- Minimal dependencies, full control
- Drop-in compatibility with tools like the AWS CLI, SDKs, and Terraform
- A secure cloud environment you control and own the entire hardware, network and software stack
It's built for teams that need:
You can run Spinifex on a few servers in a rack, a field site, or anywhere centralized cloud services aren't feasible.
- The AWS tooling they already use (AWS CLI, SDKs, Terraform), with nothing to rewrite
- Full control of the stack: your hardware, your network, your data, your keys
- A cloud that keeps running offline, through disconnection, with no external control plane
- An open core (AGPL-3.0), auditable and yours to keep, with no lock-in
You change *where* your software runs, not *what* it is.
## The Platform
<p align="center">
<img src=".github/assets/platform.svg" alt="The Mulga Platform — Hardware to Applications" width="900">
<img src=".github/assets/platform.svg" alt="The Spinifex stack: your apps and AWS tooling on top, the Spinifex AWS-compatible service layer, running on standard Linux on CPU and GPU hardware, deployable to Neocloud, on-premise, or the edge." width="900">
</p>
From commodity hardware up to unmodified AWS tooling, every layer is replaceable and yours to own.
## Three ways to deploy
One AWS-compatible surface, three deployment shapes. Pick the one that matches
your reality; the platform on top is the same.
### Neocloud
Lift and shift onto a partner Neocloud. Move workloads off the hyperscalers and
onto our Neocloud partner ecosystem without rewriting them. GPU capacity
(H100 / H200 / B200), cheaper, available now. Change an endpoint, keep the
software.
### On-premise
Bring your own hardware and host Spinifex in your own data centre. A real
multi-node HA cluster integrated with your storage and networking, with full
control of stack, data, and jurisdiction. A predictable bill, no egress
surprises, an auditable open-source core.
### Edge
Cloud where the cloud can't reach. Air-gapped sites, vehicles, vessels,
factories, and clinics. Compute next to the data, running through disconnection,
on hardware you own. The same AWS APIs your software already uses.
## AWS compatibility
Speak the AWS API surface, natively. The AWS SDKs, AWS CLI, and Terraform:
everything you deploy on AWS deploys on Spinifex unchanged. At the edge,
on-premise, or on a partner Neocloud.
| Service | What it is | Status |
| --- | --- | --- |
| **EC2** | Compute | Available |
| **EBS** | Block storage | Available |
| **S3** | Object storage | Available |
| **VPC** | Networking | Available |
| **IAM** | Identity & auth | Available |
| **ALB / NLB** | Load balancers | Available |
| **EKS** | Kubernetes | Available |
| **ECR** | Container registry | Available |
| **ECS** | Container service | Available |
| **RDS** | Databases | Q3 2026 |
Roadmap items (Q3 2026) ship under the same AWS API surface. Code written for AWS
today keeps working the moment they land.
[Track what's shipped in the release notes](https://github.com/mulgadc/spinifex/releases).
## Core Components
### Spinifex (Compute Service – EC2 Alternative)
@@ -75,19 +136,23 @@ Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs
<img src=".github/assets/architecture.svg" alt="Spinifex message-driven architecture: standard AWS clients call the AWS Gateway over HTTPS with SigV4, which publishes to a NATS bus answered by stateless ec2/ebs/s3/vpc/iam daemons over local backends." width="900">
</p>
Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless — scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Deep dive: **[`docs/DESIGN.md`](docs/DESIGN.md)**.
Every AWS API call is authenticated at the gateway, published to a NATS subject,
and answered by whichever daemon claims it. Daemons are stateless: scale
horizontally by starting more. No etcd, no Kubernetes, no external control plane.
Just systemd units, a NATS cluster, and your hardware. Deep dive:
**[`docs/DESIGN.md`](docs/DESIGN.md)**.
## Key Features
AWS-Compatible Interfaces – Provision infrastructure with awscli, Terraform, or SDKs you already use.
-Designed for Control – Run on your own terms, whether that's in a datacenter, remote site, or sensitive environment.
-Minimal Dependencies – Each service is standalone and avoids complex orchestration layers.
-Works Offline – No reliance on centralized cloud services or external networks.
- Open Source – Licensed under AGPL3.0. Fork it, extend it, or deploy it as-is.
- **AWS-compatible APIs.** Use the AWS CLI, SDKs, and Terraform you already know. Repoint your endpoint and ship; nothing to rewrite.
- **Zero cloud dependency.** Runs entirely on your hardware. No phone-home, no control plane, no external authority. Works fully offline.
-**Bare-metal compute.** QEMU-based with direct hardware access. No hypervisor tax, no abstraction overhead.
-**Built-in storage.** Block and object storage included: NVMe caching, Reed-Solomon erasure coding, and replication out of the box.
-**Edge-first architecture.** Designed for disconnected, contested, and resource-constrained environments from day one.
-**Open core, no lock-in.** AGPL-3.0 with a commercial option. Inspect it, modify it, deploy it. The platform is yours to keep.
## Installation
@@ -192,6 +257,10 @@ While Spinifex is architected and implemented by experienced engineers, we lever
This hybrid approach ensures Spinifex benefits from both proven engineering expertise and modern development acceleration, while maintaining the quality and reliability standards required for production infrastructure.
## Trademarks
"AWS", "Amazon Web Services", and all related service names (EC2, EBS, S3, VPC, IAM, and others) are trademarks of Amazon.com, Inc. or its affiliates. Mulga Defense Corporation and Spinifex are independent and not affiliated with, endorsed by, or sponsored by Amazon.com, Inc. or Amazon Web Services, Inc. References to AWS services describe interoperability and compatibility only.
## License
Spinifex is open source under the [GNU Affero General Public License v3.0](LICENSE). You're free to use, modify, and deploy it anywhere you need reliable infrastructure without depending on centralized cloud platforms.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
tomnewton-mulga
