Viperblock's ordered extent index pulls in tidwall/btree, so the bump needs
the matching go.sum entry or the build fails resolving a transitive import.
Spinifex imports github.com/mulgadc/predastore/clusterrun, but go.mod still
pinned the pre-merge pseudo-version 608939ef2bb4, which predates that package.
Off-workspace builds (CI, which has no go.work) fell back to resolving the
import at @latest — v1.14.0 — and failed.
* fix: bump goleak past the elided-frames stack-parse panic
goleak v1.3.0's stack parser treats a "...N frames elided..." line as
a function name and panics when it can't parse one. Go elides frames
on deep/recursive stacks, which is rare but load-dependent, so this
surfaced as an intermittent panic inside goleak itself (not a real
leak) during TestDrainVolume_DispatchedGoroutineDoesNotLeak under a
full parallel preflight run. Upstream fixed this in goleak@b62053b,
unreleased as a tagged version, so this pins to the latest commit.
* test: guard goleak against panicking on elided-frame stacks
* style: trim goleak regression test comments to the 3-line max
* feat: accept and surface gp3 Throughput on CreateVolume/DescribeVolumes
Honors input.Throughput (125-1000 MiB/s, default 125 when omitted) on
CreateVolume, persists it to VolumeMetadata, and surfaces it via
DescribeVolumes (getVolumeByID), mirroring the Iops handling shipped
in mulga-siv-380. Pins viperblock to the commit that adds the
VolumeMetadata.Throughput field.
Zero Throughput on a read volume means it predates this field; it is
omitted from the response rather than surfaced as a misleading 0
MiB/s, matching the existing IOPS convention.
* chore: pin viperblock to the merged gp3 Throughput commit
* ci: skip building unused E2E suite binaries in e2e.yml's single/multi jobs
These two jobs immediately rebuild suite-aware binaries after Build
Install Artifacts and never consume the tar's _bin contents, so the
cross-compiled suite build was pure waste. build-install-artifacts.sh
gains an optional --skip-e2e-build flag (positional contract unchanged);
e2e-nightly.yml and the other callers keep building the full set.
* refactor: replace hand-rolled IMDS client with AWS SDK v2
internal/imdscreds, cmd/ecs-agent's metadata fetch and its credentials
provider used a hand-written IMDSv2 token/GET flow with manual RFC3339
expiry parsing. A malformed Expiration silently fell back to "never
expires" instead of failing.
Replace it with feature/ec2/imds, credentials/ec2rolecreds and
aws.CredentialsCache, following the internal/gwsign/signer.go
precedent. An unparseable Expiration is now a hard error from Fetch,
covered by TestFetch_BadExpirationErrors. Output shapes to callers are
unchanged, and IMDS base URLs ending in /latest keep working.
* refactor: derive predastore bind config from spinifex.toml in spx itself
predastore-start.sh awk-parsed spinifex.toml for the bind host/port and
node_id, exported them as env vars, then exec'd spx. Move that
derivation into predastoreStartCmd via derivePredastoreBind, which
reads the raw (non-DIAL-normalized) nodes.<node>.predastore section,
defaulting node_id to -1 (co-located) when the key is absent, same as
the script. An explicit --host/--port/--node-id flag or
SPINIFEX_PREDASTORE_* env var still overrides the derived value.
Delete the now-unused script and point the systemd unit's ExecStart
directly at spx service predastore start, matching the
spinifex-viperblock.service precedent.
* fix(vm): durably record terminate even when local state persistence fails
finalizeTerminated bailed out of WriteTerminatedInstance whenever the
transition to terminated hit a persistence-only failure (e.g. ENOSPC writing
local state), even though the in-memory status had already reached
terminated. That left the instance owned by nothing: StuckTerminateReaper
skips it (no longer shutting-down) and TerminatedTeardownReaper never sees
it (never reached the terminated KV bucket). Only an operator daemon restart
recovered it.
WriteTerminatedInstance is a JetStream KV write, independent of local disk
space, so keep going into it whenever the in-memory status did reach
terminated, distinguishing that case from a genuine invalid/raced
transition. Once the disk has space again, teardown now completes on its
own via the existing idempotent TerminatedTeardownReaper.
* fix(systemd): cap per-service memory under spinifex-system.slice
Every service under spinifex.slice ran with MemoryMax=infinity /
MemoryHigh=infinity, so one runaway service could exhaust the whole slice
budget and the slice-level OOM would then reap an arbitrary victim instead
of the offender.
Add per-unit MemoryHigh/MemoryMax as percentages of physical RAM (so the
same units scale from a dev box to bare metal). spinifex-daemon.service
still hosts every guest QEMU process in its own cgroup (no per-instance
slice yet), so it gets by far the largest allowance, just under the slice's
own 80%/90%, to avoid starving VM launches. predastore is the legitimately
memory-hungry service and gets generous headroom above its idle footprint.
The remaining infra services are small, single-purpose Go processes and get
a shared conservative bound.
* fix(viperblockd): remove local volume directory on ebs.delete
DeleteVolume deleted the S3 data and notified viperblockd, but the ebs.delete
handler only stopped the volume's goroutines, killed nbdkit, and removed the
NBD socket — it never removed the local WAL/checkpoint cache directory under
BaseDir. The only path that removed it (VB.Close's RemoveLocalFiles) only
ran from the unmount handler's seal step, which -efi volumes never reach
(they carry no durable data to seal) and which a main volume also skips
whenever a prior seal failed (e.g. disk full, deliberately keeping local
files for recovery). Since terminate always unmounts before deleting, and
delete is permanent, retention had no remaining purpose by the time
ebs.delete ran.
Unconditionally remove BaseDir/<volume> in the ebs.delete handler regardless
of mount-tracking state, closing the leak for both the root volume and its
-efi companion.
* fix(viperblockd): validate volume names before removing local state
ebs.delete unmarshalled the volume name straight off the wire and
handed it to os.RemoveAll: an empty name collapsed to BaseDir itself,
and ".." components could escape it entirely. localVolumeDir now
rejects an empty/invalid name or one that resolves outside BaseDir
before any removal, and ebs.delete returns Success: false with an
error instead of silently skipping. Applied the same gate to
ebs.mount and ebs.config, the other two handlers that turn a
wire-supplied volume name into a BaseDir-relative path.
* chore: pin viperblock to the merged concurrent-open corruption fix
The ansible build role builds with GOWORK=off, so the deployed binary links
whatever go.mod pins rather than the go.work overlay. Move the pin to the
revision carrying the per-volume flock and the WAL/state write fixes.
* fix: stop passing nbdkit S3 credentials as literal argv
access_key/secret_key were interpolated into the nbdkit plugin argv,
which is world-readable via /proc/<pid>/cmdline for the life of the
process. Credentials now travel via cmd.Env (VB_ACCESS_KEY/
VB_SECRET_KEY) instead. Requires a viperblock nbdkit plugin build that
reads these environment variables; inert against an older plugin
until that pin moves.
* chore: pin viperblock to the VB_ACCESS_KEY/VB_SECRET_KEY fallback
The spinifex half sets these env vars on the nbdkit child, but the
deploy builds with GOWORK=off, so viperblock's matching read of them
only ships once go.mod points at 8a4787a.
DescribeImages hardcoded every AMI as "available" regardless of import
progress, so a failed backend import that never reached the flush was
indistinguishable from a complete one. Project the AMI's own State,
treating empty as available so pre-existing AMIs registered before the
field existed stay visible. CreateImageFromInstance, RegisterImage, and
CopyImage now set State explicitly since each completes synchronously
before the config write.
viperblock#64 stops WriteAtCtx returning ErrNoSpace on predastore's nearfull
pressure signal, which rides on a successful write and means slow down rather
than stop. Without this, spx refused every write once the store fell below 15%
free, 10 points above the watermark at which predastore actually rejects.
Needed by spx itself: image import runs in-process against the module-cache
build, so the plugin-side fix alone left the control plane on the old
behaviour.
Six merged viperblock changes behind one repin: local files kept when the
WAL cannot be uploaded, stranded WAL generations retried after a failed
chunk upload, the flush-covers-acked-writes invariant pinned, the nbdkit
plugin's shutdown drain bounded at 20s, seal receipts written after a clean
seal, and FLUSH fsyncing the WAL before it reports success.
The seal receipts pair with the consumer already on dev, and the 20s drain
bound pairs with TimeoutStopSec=30 on spinifex-viperblock.service.
Picks up viperblock #57, which routes the JSON slog handler to stderr so
the nbdkit plugin's logs reach journald — nbdkit repoints a plugin's fd 1
at /dev/null, so the whole volume seal path was discarded. Also picks up
#56, removing a per-call allocation from the per-block cache instrument.
No spinifex code depends on either change: spx does not import
viperblock/telemetry. The bump is needed because the build role compares
the viperblock source behind spx (module cache, GOWORK=off) against the
one behind the nbdkit plugin (the submodule tree) and fails on a mismatch.
* feat(gateway): log the canonical request and action on signature mismatch
A SignatureDoesNotMatch told us nothing beyond err="signature mismatch":
the canonical request the server reconstructed was never logged, and the
APM trace is a bare "POST /" because auth fails before the dispatcher
parses the action. An intermittent mismatch was therefore undiagnosable.
Logs the server-side canonical request, with the session token masked, plus
the recovered action. Warn rather than Debug because the failures worth
catching never reproduce on demand; the path is already rate-limited.
* chore(deps): bump predastore for SignedRequest.CanonicalRequest
Picks up the exported canonical request the mismatch logging needs.
* fix(test): reclaim fixture scratch dirs stranded in /tmp
The shared predastore fixture and the integration suite's JetStream store both
outlive any individual test, so neither can use t.TempDir() - the first test to
finish would delete files the daemon still has open. Both used os.MkdirTemp
with nothing removing the directory afterwards, so every run stranded one, and
a run killed by a timeout panic or the OOM killer stranded it regardless. /tmp
is a tmpfs on the dev boxes, so the leak is charged to RAM and swap, and
swapped tmpfs pages have no backing store to be dropped to.
Add a shared sweep that reclaims a directory only when it matches both the
prefix and an age no live run can reach. The NATS store also has a clean
shutdown point, so TestMain now removes it outright and the sweep is left to
cover the killed-run case. The predastore daemon runs until the process exits
and has no such point, so the sweep is its only mechanism.
* chore: repin viperblock to the dev tip carrying the scratch-dir fix
Test-only upstream change, so the build output is unchanged - this keeps
go.mod aligned with the submodule pointer rather than delivering a fix.
Picks up viperblock 3dcfbb7 (#49): the chunk GC sweep now runs on a
dedicated goroutine instead of sharing the chunk-uploader select, so
sustained overwrite churn no longer starves GC and wedges the volume at
the internal space cap.
* fix: make the seal-failure unmount test deterministic
The test forced the seal to fail by pointing S3Host at an unreachable
endpoint, so the handler only returned once the S3 client had exhausted
its jittered backoff. Measured handler time spanned 1.8s to 5.5s against
a 5s request deadline, which failed 2-3 runs in 10 on an idle machine and
worse on a loaded CI runner.
Config gains a sealVolume seam, nil meaning the real seal, and the test
injects an immediate failure instead. It now runs in 0.53s, 50 for 50,
and still asserts what it was written to assert -- plus that the handler
actually attempted the seal, which the old form could not distinguish.
* chore: bump viperblock to dev tip (GC observability + scratch sweep)
Picks up viperblock #47 (every chunk GC sweep logs its outcome at Info)
and #48 (TestMain reclaims scratch dirs abandoned by a panicking run).
* feat(config): enable viperblock chunk GC in rendered node configs
The gc_enabled toggle was plumbed through every VB construction site but
nothing ever set it. GCEnabled is a *bool that every consumer treats as
off when nil, and the config template rendered shardwal while omitting
gc_enabled, so no deployed node ever carried the key and chunk GC was off
everywhere by construction. Superseded chunks were never reclaimed under
overwrite churn.
Render gc_enabled = true alongside shardwal, and cover it with a test that
loads the rendered file through config.LoadConfig rather than matching the
template text. A string match would still pass if the value failed to parse
into a non-nil pointer, which is the exact way the setting reaches its
consumers.
Existing deployments keep the old rendering until their config is
regenerated; the template governs install and bootstrap only.
* chore: pin viperblock to the build with the cross-process GC snapshot marker
Enabling chunk GC in rendered node configs is only safe once the sweeping
engine can see a snapshot taken by another process. viperblock 3b56d61 adds the
per-volume snapshot marker that closes that window, and 2a34add fixes the
concurrent-stop panic in the background goroutine lifecycle that GC's own ticker
would otherwise widen. Pin both before turning GC on.
* fix(eks-node): stop the DHCP MTU overriding the pinned link MTU
Pinning eth0 to 1320 was never enough. This image runs dhcpcd, which
honours DHCP option 26 and stamps the advertised MTU onto every route it
installs:
default via 10.32.101.1 dev eth0 proto dhcp src 10.32.101.7 \
metric 1002 mtu 1442
Linux prefers a route's MTU metric over the device MTU, so the node kept
advertising an MSS derived from 1442 while the real path MTU through the
NAT gateway is 1408. Small packets pass and full-MSS inbound segments —
a TLS ServerHello with its certificate chain, image layers — are
silently dropped. It presents as `net/http: TLS handshake timeout` on
image pulls, not as anything MTU-shaped, and it only affects the host
netns, so containerd pulls fail while pod traffic sized off cni0 is
fine. Per-destination PMTU exceptions expiring in and out of the route
cache made it look intermittent and host-specific.
Three changes, innermost first:
- dhcpcd.conf gets `nooption interface_mtu`, so no lease can stamp a
route MTU. mulga-vpc-mtu is the single source of truth for the MTU
of these nodes.
- mulga-vpc-mtu re-stamps any route that already carries a different
MTU, covering an image built before this fix or a dhcpcd already
running. Routes with no MTU metric inherit the device MTU and are
left alone.
- the subnet DHCP option drops 1442 to 1408. 1442 is geneve-only and
ignores the IPsec ESP overhead on the NAT egress path, so it
overstates the usable MTU for every guest, not just these nodes.
Also corrects the header comment, which claimed Alpine's udhcpc ignores
option 26 — this image runs dhcpcd, which honours it. That wrong premise
is why the route metric was never considered.
Adds a stubbed-`ip` test covering the rewrite, the dev token that
`ip route show dev X` omits, idempotency, and routes without an MTU
metric.
* fix(eks): sweep the CP VPC by identity so teardown can finish
DeleteClusterCPVPC finds the managed control-plane VPC's subnets and SGs
by role tag. A create that fails partway — the first control-plane
server failing to launch, say — leaves resources behind that never got
tagged, so the sweep cannot see them.
DeleteVpc rejects while any subnet or non-default SG remains, and
teardown is re-driven on a timer, so the cluster sits in DELETING
forever retrying the same doomed delete every couple of minutes:
eks-deleting: re-driving wedged DELETING teardown cluster=toc
eks-deleting: re-drive teardown failed
err="delete managed CP VPC: eks: delete cp vpc vpc-...: DependencyViolation"
Nothing converges: the cluster name cannot be reused and terraform's
delete wait can never pass. Observed on env19, wedged for ~11 hours
until this change went in, at which point the next re-drive cleared it.
After the tagged sweep, clear anything still in the VPC scoped by
vpc-id: subnets, and non-default SGs (the default SG goes with the VPC
cascade). Teardown had no SG surface, so CPVPCDeps gains an optional SG
provisioner — nil is tolerated, since only teardown uses it.
* chore: pin viperblock + predastore to the merged storage fixes
viperblock f8cfd77 (#42) makes read-modify-write atomic per block,
removing the lost update that silently reverted a 4096-byte block to a
previous generation of itself. Confirmed as the env19 corruption cause
by an A/B: the pre-fix build corrupted image layers within ~15 minutes
(failed to extract layer / invalid tar header / unpigz: corrupted) while
the fixed build ran 5+ hours clean, absorbing 6407 same-block RMW
conflicts on the worker root disks.
predastore c0a90d2 (#63) batches FSM.Restore. Without it, any predastore
restart on a store whose metadata set outgrew one badger transaction
fails with "Txn is too big to fit into one request" on every node at
once, taking the metadata plane — and with it the AMI catalogue and
cluster creation — down permanently.
Both were verified in the built artifact rather than from go.mod alone:
go version -m reports the new pseudo-versions, and the rmw_conflicts
counter plus the batched-restore error strings are present in the
binary. Pinning by go.mod while the submodule pointer says otherwise is
how an earlier analysis ended up reading dead code.
* test: record the corrected DHCP MTU in the kitchen-sink golden
The NB golden pins the DHCP options published per subnet, including the
advertised MTU. Dropping the subnet option from 1442 to 1408 changes
what the reconciler writes, so the golden diverged and the live scenario
test failed — the test doing exactly its job.
Regenerated with UPDATE_GOLDEN=1. The diff is only mtu:1442 -> mtu:1408
on the two subnet DHCP records; nothing else in the snapshot moves.
* feat: plumb GCEnabled config through to every VB construction site
viperblock chunk GC is gated behind VB.GCEnabled, but spinifex never
set it, so every deployed node ran with GC off by construction
regardless of which viperblock binary shipped. Add a
[viperblock] gc_enabled config key (default false, mirrors shardwal)
and thread it through to the three places spinifex builds a VB: the
nbdkit plugin it spawns per mounted volume (via a new gc_enabled
plugin param), viperblockd's detached-volume opens, and the volume
service's CreateVolume.
* chore: repin viperblock to GC-enable lineage
The gc-enable plumbing references viperblock.VB.GCEnabled, which is
absent from the previously pinned v1.12.1. Repin to the feat/nbd-gc-enable
commit so a standalone (GOWORK=off) build — the path the ansible deploy
uses — resolves the field, and spx plus the nbdkit plugin build from the
same GC-bearing viperblock lineage.
* fix(storagegrowth): stop launching a nano for image-pull repro
ensureImagePullInstance shared harness.DiscoverNanoInstanceType with the
synthetic npass suite. A 512MB guest OOM-thrashes containerd before any
disk-driven ENOSPC and conflates the storage leak with guest-memory
pressure. Add a package-local imagePullInstanceType helper: an exact
override via SPINIFEX_STORAGEGROWTH_INSTANCE_TYPE, otherwise the smallest
advertised type at or above SPINIFEX_STORAGEGROWTH_MIN_MEM_MIB (default
16384 MiB). npass keeps using the nano selector unchanged.
* feat: report guest block write errors so backend ENOSPC reaches the guest
Set werror=report and rerror=report on the guest boot drive so a viperblock
ENOSPC (or any other backend write failure) is surfaced to the guest as a
real I/O error instead of QEMU silently pausing the VM. Without this the
guest kernel never learns its disk is out of space, so it can't fail
writes cleanly or let an operator react.
* chore: repin predastore and viperblock to the storage-capacity-safety lineage
Drop the local-path replace for predastore (a test shortcut) and pin both
predastore and viperblock to their feat/storage-capacity-safety commits, so
a standalone (GOWORK=off) build resolves the free-space watermark, ENOSPC
classification, and backpressure plumbing that spinifex's werror/rerror
change depends on.
* chore: bump viperblock to 054a44a3f0f5
Realigns spx's linked viperblock with the submodule working tree, which the
build role's source-drift gate compares against.
* chore: repin predastore and viperblock to their dev lineage
Both were pinned to pseudo-versions off feat/storage-capacity-safety. That work
has now landed on dev in each repo, and both PRs were squash-merged, so the
branch commits those pins named are unreachable from any branch.
Repin to the dev commits carrying the same code: predastore 5699329 and
viperblock 35dd9f2.
* feat(bedrock): Ochre streaming — ConverseStream + InvokeModelWithResponseStream
Adds the two streaming bedrock-runtime operations behind a shared
eventstream.Encoder frame writer: ConverseStream reframes vLLM/Anthropic SSE
into the normalized Bedrock event taxonomy (messageStart -> contentBlock*
-> messageStop -> metadata), InvokeModelWithResponseStream wraps each
provider-native chunk in a PayloadPart frame. Fixes otelsetup's
statusRecorder to implement Flusher/Unwrap so flushes actually reach the
socket through the middleware chain, which streaming depends on.
* fix(bedrock): keep vLLM ConverseStream taxonomy ordered on edge cases
Route contentBlockStop/messageStop through an idempotent ensureStarted so a
usage-only chunk arriving before any finish_reason no longer emits metadata
ahead of messageStop, and a zero-content stream no longer emits stop events
without a preceding messageStart. Regression tests for both.
* fix(bedrock): route /model/ data-plane calls to bedrock-runtime
AWS SDKs sign bedrock-runtime with the SigV4 signing name "bedrock", the
same as the control plane; the two services are separated only by endpoint
hostname. The gateway serves one endpoint and routed purely on the signing
name, so every /model/{id}/converse*|invoke* call landed in Bedrock_Request
(control plane) and returned InvalidAction — no standard SDK/CLI could reach
any bedrock-runtime endpoint.
Disambiguate by path in GetService: a "bedrock"-scoped request to a /model/
path (data-plane-exclusive) resolves to bedrock-runtime. Control-plane paths
(/foundation-models, /custom-models, /model-invocation-job) are unaffected.
* test(bedrock): satisfy testifylint go-require/empty in streaming tests
Use assert (guarded where a value is dereferenced afterward) instead of
require inside httptest handler closures, and assert.Empty for empty-string
checks, per dev's stricter testifylint config.
Bump github.com/mulgadc/viperblock to v1.11.1-0.20260713024608-eb4a376becc8
(dev tip through viperblock #34): bounded parallel chunk upload + coalesced
checkpoint (edou7 sec 7 items 2 and 9), sweeping the require up from #32 and
picking up the #33 write backpressure on the way. The nbdkit plugin the VM
write plane uses is built from the submodule source; this bump aligns the
viperblockd library that spx links.
Init now installs an sdk/log LoggerProvider inside the existing
exportEnabled() gate, sharing the same resource as the tracer/meter (incl.
ci.run_id via OTEL_RESOURCE_ATTRIBUTES) and joining lp.Shutdown into the
returned shutdown func. The process default slog handler is fanned out to
both the existing stdout/journald handler and an otelslog bridge, so app
logs export over OTLP with trace correlation while journald output is
unchanged. With no OTLP endpoint configured Init remains a no-op: no
LoggerProvider is installed and the default logger is left untouched.
* feat(otelsetup): OTel init, resource attrs, trace-stamping slog handler
Env-gated on OTEL_EXPORTER_OTLP_ENDPOINT: with no endpoint only the W3C
propagator is installed and globals stay no-op, so instrumented binaries
ship everywhere at zero cost. Resource carries service/host/mulga.env/
mulga.source plus OTEL_RESOURCE_ATTRIBUTES (ci.run_id in CI).
* feat(otelsetup): wire telemetry init into all spx services
Every spx service start command installs the trace-stamping JSON slog
default and the OTel providers via initTelemetry, flushed on shutdown.
Gateway's own slog setup now routes through otelsetup so its level
handling keeps trace correlation.
* feat(gateway): HTTP tracing middleware on awsgw and daemon cluster API
Server span per request with W3C traceparent extraction; awsgw spans are
renamed to the resolved SigV4 service.Action post-auth with account and
region attributes. Request log lines carry trace_id via context.
* feat(nats): propagate W3C trace context over command subjects
NATSRequestCtx injects traceparent beside X-Account-ID and opens a client
span per hop; ServeNATSRequest opens a consumer span joining the producer
trace, with a Ctx variant for handlers that thread context. RunInstances
is plumbed end-to-end (gateway dispatch through placement fan-out to the
daemon handler) and viperblockd ebs.mount/unmount join inbound traces.
* feat(systemd): source optional /etc/spinifex/telemetry.env in service units
Lets operators enable OTLP export (OTEL_EXPORTER_OTLP_ENDPOINT,
MULGA_ENV, MULGA_SOURCE) by dropping a single env file, with no unit
edits. File is optional; absent means telemetry stays disabled.
* feat(ci): export OTel telemetry from E2E envs to the central Elastic sink
All E2E workflows set SPINIFEX_OTLP_ENDPOINT (repo variable override,
default wattle) plus ci.run_id/ci.workflow resource attributes, so
bootstrap-install.sh writes telemetry.env on every node. Teardown runs
flush-telemetry.sh before the final tofu destroy so buffered spans and
metrics reach the sink.
* feat(otelsetup): request metrics and Go runtime metrics via OTLP
mulga.requests counter + mulga.request.duration histogram recorded by the
same middleware that opens spans: awsgw carries the resolved SigV4 action,
the daemon cluster API uses the chi route pattern, and NATS handlers record
per-topic at the subscribe chokepoints (node names stripped to keep
cardinality flat). Go runtime metrics start inside the same env gate.
* feat(daemon): deep spans for QMP commands, VM launch phases, EBS ops
Single chokepoint span per QMP command in sendQMPCommand covering all
callsites; vm.launch phase children (mount_volumes, start_qemu, qmp_connect,
persist_state); ec2.cmd consumer span with traceparent extraction plus
start/stop/reboot/terminate child ops; ebs.mount/unmount spans in
viperblockd with volume attributes.
* feat(telemetry): thread request context through NATS, object store and VM paths
Migrate utils.NATSRequest call sites to NATSRequestCtx (241 sites; W3C
traceparent now flows gateway -> daemon on every request), add GatherCtx and
migrate all Gather call sites, inject traceparent on manual EKS publishes.
ObjectStore interface is ctx-first with aws-sdk WithContext calls and an
otelhttp transport, so predastore spans join the caller trace. NATS consumer
spans record error status when the response envelope carries an error.
vm.Manager and the ec2 handler service interfaces take ctx-first so vm.launch
and QMP spans join the request trace; detached background work uses explicit
context.Background(). Hot-path slog calls use *Context variants for trace_id
binding.
* feat(telemetry): suppress health-check, heartbeat and unrooted trace noise
Health probes record metrics only (no span per poll); the QMP heartbeat
poller marks its context so query-status ticks stop rooting a trace each.
rootSampler honors MULGA_ROOT_TRACE_RATIO (ParentBased ratio, default 1.0)
and the predastore unit sets 0.05: guest block I/O roots are sampled while
request-linked spans are always kept.
* feat(otel): export deployment.environment resource attribute
Elastic APM maps it to service.environment, enabling the native environment
selector across the APM UI (Services, Traces, Dependencies).
* fix(telemetry): stop objectstore rooting method-only HTTP transactions
Background S3 calls (state sync, cleanup loops) carried no caller span, so
otelhttp rooted a trace per request and APM surfaced thousands of parentless
'HTTP GET/PUT/DELETE' transactions with expected-404 probes counted as
failures. Emit the client span only when the request context already carries
one, matching the viperblock backend. Bump viperblock/predastore to branch
heads so CI builds also pick up the equivalent span filter and root sampler.
* feat(telemetry): per-VM guest metrics via dedicated QMP socket + qmp-collector
Goanna Phase 1 producer, operator-consumed first. Each VM gets a second
-qmp chardev (qmp-telemetry-<id>.sock) plus a sidecar discovery file
carrying instance/account/vcpus/taps/monitoring-tier; the new
qmp-collector service polls query-cpus-fast (vCPU thread /proc jiffies),
query-blockstats, query-balloon and host tap sysfs counters, and
publishes CloudWatch-mappable goanna_ec2_* series to NATS
metrics.ec2.<instance-id>. A queue-group bridge in the same process
forwards the series to the node-local OTLP receiver so they reach the
operator sink until Goanna consumes the subjects. The collector never
touches the manager's control QMP socket; telemetry failures never
block a launch.
* fix(qmp-collector): dial NATS via DialTarget, drop ProtectProc, fix meta race
The collector connected to the raw NATS host (0.0.0.0), failing TLS
verification against a cert valid for 127.0.0.1; route through
admin.DialTarget like the other daemons.
ProtectProc=invisible hid the QEMU vCPU threads (dumpable=0) that the
collector must read from /proc/<tid>/stat, so every tick failed and
re-primed without ever publishing. Drop it to default.
Surface collect failures at Warn so silent metric loss is visible, drop
per-tick publish to Debug, and read InstanceID under the lock in run()
to fix a race with updateMeta on ENI hot-plug.
* chore(deps): re-pin predastore + viperblock to dev-landed commits
* feat(eks): reconciler in-place control-plane restart on sustained degradation
Phase 1 of EKS CP recovery (siv-243). The ACTIVE-branch reconciler only
recorded health; a CP VM that died (OOM/host crash) left the cluster wedged
ACTIVE-but-unreachable with no recovery path. Add a CPInstanceControl surface
and bounded in-place restart: once an ACTIVE cluster's CP stays unhealthy past
a grace window, StartInstance is issued (backoff-spaced, capped) when the
instance is stopped/error, re-mounting the same root volume so embedded etcd
survives. Degradation stays AWS-faithful ACTIVE + a ClusterHealth issue rather
than a non-AWS DEGRADED status enum. cpControl nil keeps the reflect-only path.
Wiring of a real CPInstanceControl in the daemon and the error-state
manager-eviction fix are the remaining Phase-1 integration steps; Phase 2
(etcd snapshot cadence + restore-snapshot) is tracked separately.
* feat(eks): wire control-plane restart into the reconciler via instance service
Threads a CPInstanceControl into the per-cluster reconciler (siv-243 Phase 1).
cpControlAdapter binds the daemon instance service + account: InstanceState
reads the CP's EC2 state via DescribeInstances, StartInstance restarts a stopped
CP via StartStoppedInstance, which loads from the shared stopped store and
re-mounts the same root volume so embedded etcd survives. Wired as
EKSServiceDeps.CPControl = d.instanceService and passed through spawnReconciler.
Recovers a cleanly-stopped control plane co-located with the reconciler leader.
Remaining: DescribeInstances is node-local so a cross-node stopped CP needs an
aggregating describe; and an error/crashed CP (not in the stopped store) needs a
dedicated error-state restart path — both tracked as Phase-1 follow-ups.
* feat(eks): cross-node CP describe + error-state control-plane restart
The reconciler's control-plane recovery was wired to the node-local instance
service: DescribeInstances only saw the leader's node and StartStoppedInstance
only handled a cleanly-stopped CP from the shared KV. A crashed/OOM CP ends in
error state, still resident in its owning node's manager map (not the stopped
store), so recovery never reached it — the actual incident in the recovery plan.
Wire EKSServiceDeps.CPControl to a NATS-backed eksCPControl. DescribeInstances
fans out across every host (gateway scatter-gather) so the reconciler observes a
CP on any node. RecoverInstance drives the gateway StartInstances two-step:
ec2.cmd.<id> recovers a live error/running owner in place (the owner stays
subscribed through the crash; vmMgr.Start relaunches error->pending, re-mounting
the etcd root volume), falling back to ec2.start shared-KV rehydration for a
stopped CP.
The CP VM is ManagedBy=EKS, hidden from customer DescribeInstances; relax that
hide so the owning account resolves its own managed CP when targeting it by
explicit instance ID (bulk listings still hide it). EC2APIState already maps the
internal error state to the AWS closed-enum stopped, which cpRestartable accepts.
* fix(eks): reconcile CP as system account, not customer
The control-plane VMs are launched under the system account
(admin.SystemAccountID()), not the customer account that owns the cluster
record. spawnReconciler was binding cpControlAdapter to the customer account,
so InstanceState/RecoverInstance ran as the customer — which cannot see or own
its own cluster's system-owned CP VMs. DescribeInstances returned zero and the
reconciler logged 'control-plane instance not found', never restarting a wedged
CP. Bind the adapter to the system account instead.
Revert the earlier DescribeInstances managed-VM relaxation: it was based on the
mistaken assumption that CP VMs are customer-owned. Visibility is resolved by
the system-account adapter, so the explicit-ID escape hatch is dead code.
* fix(vm): pre-create mgmt tap on disk-boot restart path
The direct-boot branch pre-creates the management tap via NetworkPlumber.SetupTap
(owned by the daemon euid), but the disk-boot/else branch only appended the QEMU
tap,id=mgmt0 device without creating the tap. The daemon runs as a non-root user
without CAP_NET_ADMIN, so QEMU can only attach a tap it already owns; on the
disk-boot path QEMU tried to create /dev/net/tun itself and failed with
'Operation not permitted', failing every stopped-instance restart (StartInstances
returned ServerInternal). This blocked EKS control-plane recovery: the reconciler
found the stopped CP and issued the restart, but the VM never booted.
Call SetupTap for the mgmt tap in the disk-boot branch, mirroring direct-boot.
* feat(eks): recover all HA control-plane members, not just primary
The reconciler restarted only ControlPlaneInstanceID (the primary CP
member), so on an HA cluster the other server VMs stayed stopped and
etcd never regained quorum (1/3) — apiserver healthz stayed fail even
after recovery ran. maybeRecoverControlPlane now iterates every member
in ControlPlaneNodes (falling back to the scalar ID for single-node
clusters), restarting each stopped/error member per pass while
preserving the grace window, attempt cap, and backoff gates.
Refs mulga-siv-509
* fix(eks): keep cloud-init polling IMDS so primary CP gets its user-data
The eks-node system VM's user-data is delivered only via IMDS (the
NoCloud seed was retired in #411). IMDS (169.254.169.254) is reachable
only once mulga-mgmt-net brings up the data NIC via DHCP and the per-tap
br-imds datapath reconciles, which on the first-launched (primary) CP
can lag cloud-init's stock 120s probe window. With the default max_wait
cloud-init falls back to DataSourceNone, permanently losing the launch
user-data (no SPINIFEX_K3S_ROLE -> no k3s server) and wedging the cluster
in CREATING.
Bake a cloud.cfg.d drop-in that pins DataSourceEc2 and raises max_wait to
600s so cloud-init keeps polling until the datapath is ready instead of
falling through to None. Worst case a slow boot, never a lost role.
* fix(eks): warm IMDS instance-role creds before first SigV4 sign
On IMDS-credentials mode the on-VM SigV4 helpers (eks-token-webhook,
eks-gateway-publish, mulga-eks-state-report) sign via the SDK default
chain, which resolves in-VM IMDS instance-role creds. The per-tap
br-imds datapath lags VM boot by minutes (observed ~264s), and the SDK's
IMDS client fast-fails each probe (250ms dial), so early Retrieve calls
return 'context deadline exceeded' / 'no EC2 IMDS role found'.
eks-gateway-publish only retries for 30x5s=150s, shorter than the
datapath warm time, so it exhausts its budget and drops the
bootstrap/state publish -> cluster hangs in CREATING.
The credential chain is sound (the CP role trusts ec2.amazonaws.com and
AssumeRoleForInstance evaluates that principal); the failure is timing.
Add gwsign.Signer.EnsureCredentials and block in eksgw.New until the
instance-role creds first retrieve (2s interval, 10m ceiling aligned
with cloud-init max_wait), so every helper's first signed request
already holds creds instead of racing a cold datapath. Static-key mode
is unaffected.
* fix(eks): retry CP data-NIC DHCP until cross-host datapath is up
A single ~16s DHCP one-shot on the primary data NIC lost the race against
the cross-host OVN datapath (logical-port binding + per-tap reconcile on the
remote placement host), stranding the NIC IP-less so the Ec2 datasource never
reached IMDS and the CP member wedged. Add dhcp_acquire: retry the one-shot on
a budget aligned with cloud-init's Ec2 max_wait (600s), confirming an inet
address before success. Each attempt still exits leaving no daemon, so
cloud-init's later ENI management is unaffected.
* fix(eks): drop reconciler registry entry when Run self-exits
ReconcilerRegistry.Spawn only cleaned up its holder on ctx cancel
(Stop/StopAll). When a ClusterReconciler's Run returned on its own — a
terminal cluster state or a lost lease — nothing cancelled the ctx, so
the holder leaked and a later same-name create silently no-opped at the
head of Spawn, never reconciling to ACTIVE.
Thread a finished channel from RunClusterReconciler's Run goroutine back
to Spawn's watcher, which now cleans up on whichever fires first: an
external cancel or Run exiting. On self-exit it deletes its own holder
and releases the lease, so a fresh same-name create re-spawns and
reconciles.
* fix(eks): run mulga-mgmt-net in boot runlevel before cloud-init-local
CP user-data is delivered only via IMDS since the NoCloud seed was retired
(#411), so cloud-init DataSourceEc2 must reach 169.254.169.254. mulga-mgmt-net
DHCPs the data NIC that makes IMDS reachable, but it was added to the default
runlevel and only declared `before cloud-init` (the network-stage service).
cloud-init-local therefore ran its init-local Ec2 crawl first with no data-NIC
route and burned its whole probe window (~4 min) before mulga-mgmt-net brought
eth0 up, regressing CP bringup from ~1m50s to ~5m54s.
Add mulga-mgmt-net to the boot runlevel in setup.sh instead of default, drop it
from ENABLE_SERVICES, and order it `before cloud-init-local` so the data NIC and
IMDS route are up before the init-local crawl. Keeps the IMDS-only user-data
path; the raised Ec2 max_wait stays as a safety net for host br-imds datapath lag.
* fix(vm): refresh LaunchTime on crash-restart so watchdog spares relaunch
RestartCrashedInstance re-launched a crashed VM without refreshing
Instance.LaunchTime. The pending watchdog measures now-LaunchTime against
PendingWatchdogTimeout, so the stale original launch time tripped
launch_timeout on the next tick and terminated the crash-restart
mid-flight. Refresh LaunchTime under the state lock, mirroring the
daemon-restart restore path (restore.go), so the relaunch gets a fresh
timeout window. Adds a regression test asserting LaunchTime advances.
* feat(eks): reconcile control-plane member count after a lost member
The cluster reconciler self-heals a stopped/error control-plane VM in
place but never replaced a terminated/gone member, so an HA cluster ran
below haControlPlaneCount permanently. Add a member-count reconcile pass
that provisions one replacement CP joining the surviving etcd quorum.
- CPProvisioner + EKSServiceImpl.ProvisionReplacementCP: replay the
persisted create template on a free host, the single-node analogue of
the spread join branch, with rotating creds re-derived.
- ControlPlaneTemplate persisted on ClusterMeta at create (per-node and
rotating-cred fields cleared). SwapControlPlaneMember CAS swaps the dead
node for the replacement and refreshes the [0] scalar mirrors.
- maybeReplaceControlPlaneMember fires only when no member is restartable
and one is truly lost, a healthy majority survives, bounded by grace,
backoff and an attempt cap; placement excludes live-member hosts.
- Replacement prunes the superseded member from etcd once joined:
k3s-first-boot deletes the stale Node (EKS_ETCD_PRUNE_PEER_IP) so quorum
width returns to N. json tags added to K3sServerInput (now persisted).
* feat(eks): in-guest control-plane self-diagnosis on healthz fail
When a control-plane apiserver reports unhealthy, the reconciler only
saw healthz=fail with no cause, and the guest has no on-demand command
channel (serial root login is password-gated, no guest agent). Make the
CP self-diagnose and relay the cause through the one-way state report.
- mulga-eks-state-report.sh: diagnose() collects failing /readyz?verbose
subchecks, isolated /healthz/etcd reachability, and etcd-db-dir free
space; on fail write a verbose block to the host-captured serial
console and add a JSON-safe reason to the state payload.
- ServerStateReport: add Reason (omitempty; empty from a healthy CP or
an older AMI predating the field).
- cluster_reconciler observe(): surface the reason in the health issue.
- Tests: POSIX unit test for the agent (stubbed kubectl/df); Go tests
for the reason-bearing and terse health-issue paths.
* feat(eks): 15-min etcd snapshot cadence alongside nightly
Nightly alone is too coarse a recovery point for a young cluster. Install the
same mulga-eks-etcd-snapshot script into a second Alpine crond periodic dir
(/etc/periodic/15min) so it also runs on a ~15-minute RPO window; the script
picks its cadence tier from the dir it is invoked from ($0 sniff) and encodes
the tier in the snapshot key (etcd-{tier}-{ts}.snap), so a predastore S3
lifecycle rule can retain the frequent rolling window shallow and the nightly
copies deep. Server-role only (agents exit early); no restore-path change.
Add a self-contained POSIX unit test (stub k3s/curl/logger, symlinked crond
dirs to exercise the real $0 tier sniff) covering agent no-op, frequent + daily
key prefixes, local-snapshot prune, and quiet no-op without creds.
* feat(eks): auto-reform wedged etcd quorum via boot-time cluster-reset
An HA k3s control plane does not self-heal after a simultaneous full-CP
restart: embedded etcd never reforms quorum (etcd:unreachable sustained),
and in-place VM restarts cannot fix it. Add a last-resort escalation that
drives the documented k3s cluster-reset recovery, pull-based so the host
never needs an inbound channel into the guest.
Reconciler: maybeReformEtcdQuorum escalates only when every CP member is
VM-running but the health issue carries etcd:unreachable, the cluster is
HA, and a grace window has elapsed. It sets per-member recovery directives
(seed = cluster-reset, others = wipe-rejoin), epoch-bumped and keyed by
instance ID, then restarts the members. Bounded by grace/backoff/attempts;
a recovered cluster clears the directives once. Enabled via
WithEtcdResetRecovery alongside the existing CP control wiring.
Directive plane: per-member KV directive {epoch, action, snapshot} with
store helpers, an EKS service Get/Set pair over NATS-RPC, and an
internal-recovery gateway route (system-cred carve-out, same as
internal-addons). The CP server instance role gains eks:GetRecoveryDirective.
Guest: mulga-eks-k3s-recovery is a pre-k3s oneshot (server role only) that
fetches its directive through the AWS gateway (eks-gateway-fetch -resource
recovery), epoch-guards one-shot application, then runs
`k3s server --cluster-reset` (optionally restoring a predastore snapshot)
or wipes the local etcd datastore to rejoin the reset seed. Wired into the
image manifest, setup.sh perms, and the eks-node-role selector.
Tests: reconciler escalation gates + recovery clear, directive store epoch
increment + service validation, and a POSIX unit test for the guest agent.
* fix(eks): reboot running CP members to apply etcd cluster-reset directive
The etcd quorum-reformation escalation drove StartInstance on the seed and
followers, but the reset gate only fires when every member is VM-running. The
StartInstances path rejects a running instance with IncorrectInstanceState, so
the guest never re-entered its boot sequence and the cluster-reset/wipe-rejoin
directive was never applied — the wedged control plane stayed wedged.
Reboot the running members (QMP system_reset) instead, re-entering the boot
sequence so the on-VM recovery agent applies the directive. Adds RebootInstance
to the CP control surface (interface, adapter, NATS-backed daemon impl) and
switches maybeReformEtcdQuorum to it; the stopped/error in-place restart path
keeps StartInstance.
* fix(eks): graceful stop for etcd reset recovery
The etcd quorum-reformation reset path forced running CP members through a
hard QMP system_reset to re-enter the boot sequence and apply their recovery
directives. A hard reset skips a clean unmount: on the next boot e2fsck
cleared the recovery and k3s runlevel entries (dangling inodes), so the
cluster-reset never ran and k3s was unstartable — the reset defeated recovery.
Replace the reboot with a graceful StopInstance (QMP system_powerdown, clean
unmount, SIGKILL fallback). The existing in-place restart path
(maybeRecoverControlPlane) then boots each stopped member clean on a later
tick, and the on-VM recovery agent applies its directive against an intact
filesystem. Reuses existing lifecycle machinery; the stop also tolerates a
wedged QMP client via the SIGKILL fallback.
Swaps RebootInstance -> StopInstance across CPInstanceControl,
cpInstanceController, the cpControlAdapter, and the NATS-backed eksCPControl
(gateway RebootInstances -> StopInstances), plus the reset/recovery test fakes.
* fix(vm): retry QMP dial and unlink stale socket on relaunch
A QEMU killed with SIGKILL (crash or etcd-reset restart) leaves its
qmp-<id>.sock inode behind. On relaunch the startup os.Stat probe and the
QMP dial both raced that dead listener, so a single dial returned
'connection refused': StartStoppedInstance then reported ServerInternal and
left the instance KV-stopped even though the guest booted fully. The wedged
member stayed invisible to the reconciler's cpLive gate, stalling recovery.
Unlink the stale socket before launch so the inode that appears is the fresh
QEMU's, and retry the QMP connect with a short backoff to cover the window
where the socket is bound but not yet listening. Handshake failures are not
retried — a listener answered, so the error surfaces immediately.
* feat(eks-ui): surface cluster Health.Issues on the EKS console
DescribeCluster is AWS-faithful (Status stays ACTIVE, degradation lives in
Health.Issues), but the cluster detail page only rendered the status badge, so
a degraded control plane still showed as ACTIVE with no visible signal. Add a
destructive health badge next to the status and a Health row that lists the
issue messages, mirroring the addons tab. Healthy clusters read OK.
* chore: rebuild ui
* fix(viperblockd): stop mount-path daemon VB uploader to prevent checkpoint clobber
The daemon-side VB created in the ebs.mount handler is stored in
MountedVolume.VB for the whole mount, and its background chunk uploader was
only stopped at daemon shutdown. That state-only VB ran DrainToBackend every
30s, overwriting the live checkpoint the nbdkit plugin VB (the data-path
writer) keeps updating, leaving SeqNums inconsistent with sealed chunk
ciphertext and causing transient AEAD auth failures on cloned encrypted
volumes. Stop its uploader right after New, matching the other short-lived VB
call sites.
Also bump viperblock to the atomic chunk-ObjectID allocation fix.
* build(deps): pin viperblock to landed AEAD concurrency fix
Bump github.com/mulgadc/viperblock to the dev commit b08bea6 that
eliminates "cipher: message authentication failed" on encrypted volumes
under concurrent read-modify-write + background chunk drain. Removes the
local-path replace used while the fix was in flight.
* fix(viperblock): stop chunk-uploader goroutine leak at all VB call sites
viperblock.New() unconditionally starts a background chunk-uploader
goroutine (30 s interval) that only VB.Close() stops. Seven call sites
created short-lived or state-only VB instances without calling Close(),
leaking one uploader goroutine per instance. On a single node the leaked
goroutines exhaust memory (predastore RSS grew ~1.3 GB/5 min at idle
post-single-suite), causing liveMemGate to reject launches and the lb
suite to fail.
The leaked uploaders are also the primary cause of the AEAD auth
failures (mulga-be2ev): a leaked VB from prepareRootVolume holds the
clone-point BlockLookup and overwrites the correct live checkpoint in S3
every 30 s, leaving SeqNums inconsistent with sealed chunk ciphertext.
Fix all seven sites:
- instance/service_impl.go prepareRootVolume: defer StopChunkUploader +
StopWALSyncer after successful viperblock.New (mirrors the EFI path's
intent; full Close is unsafe here because cloneAMIToVolume already
called SaveBlockState + RemoveLocalFiles)
- viperblockd.go ebs.delete handler: add StopChunkUploader alongside
existing StopWALSyncer
- viperblockd.go ebs.unmount handler: same
- viperblockd.go shutdownVolumes: same
- snapshot/service_impl.go snapshotVolume: defer both stops after New
- volume/service_impl.go CreateVolume: defer both stops after New
- image/service_impl.go snapshotRunningVolume: defer both stops after New
- image/service_impl.go snapshotStoppedVolume: defer both stops after New
Closes mulga-0zzci
Closes mulga-ji5fi
* fix(deps): bump viperblock to 01a7e686 (AEAD regression fix)
The prior pin (6af89855) carried the snapshot-AMI AEAD regression that
corrupts block storage on restore. Point at the merged viperblock dev fix.
---------
Co-authored-by: joshsiv-mulga <mrk@mulgadc.com>
go.mod referenced v1.10.0 which pre-dates LoadLiveCheckpoint. CI builds
without go.work so e2e-build failed to compile. Pin to viperblock dev tip
6af8985 (feat/snapshot-ami merged) until v1.11.0 is tagged.
vpcd's libovsdb NB client was a single long-lived connection with no
reconnect logic. After the NB server restarted (host reboot, service
restart) every vpcd holding a dead/half-open socket never re-established
it, so OVN-write handlers blocked until the caller timeout and CreateVpc
returned ServerInternal intermittently (queue-distributed across vpcd).
Add WithInactivityCheck to the OVSDB client: a periodic Echo probe
detects the dead peer and triggers reconnect against the configured
endpoint, with infinite backoff so a daemon keeps retrying across a long
outage. Bound the SG subscriber handlers with a timeout context so a
stuck OVN op frees the worker instead of hanging on context.Background.
* feat: IMDS-capable gateway signer for lb-agent + EKS clients
Add internal/gwsign, a SigV4 signer that draws credentials from either
fixed static keys or the AWS SDK default credential chain. In a guest VM
the chain falls back to the in-VM IMDS endpoint, yielding scoped,
auto-rotating instance-role credentials; a session token is signed as
X-Amz-Security-Token so the gateway can verify temporary (ASIA) creds.
Wire it into the lb-agent (lbagent.Agent) and the EKS on-VM client
(internal/eksgw). When static keys are absent the clients now sign with
IMDS credentials instead of failing; when present, behaviour is
unchanged (static path). This is the client-side foundation for dropping
long-lived secrets from VM user-data.
Credentials are retrieved per request so rotated IMDS creds take effect
without an agent restart.
* feat: IMDS instance-role creds for EKS server + LB VMs (Phase B/C)
Attach a Spinifex-managed system instance profile to the k3s control-plane VM
and the elbv2 LB VM so both authenticate to the gateway with scoped, rotating
IMDS instance-role credentials instead of a long-lived static secret baked into
user-data.
- Add a shared handlers/iam.EnsureSystemInstanceProfile that find-or-creates a
role (EC2 trust) plus its instance profile and asserts an inline permission
policy. The gateway enforces per-action authz on assumed-role principals, so
the roles carry exactly the actions each VM calls:
- EKS server: eks:PublishInternal, eks:ListInternalAddons.
- LB agent: elasticloadbalancing:LBAgentHeartbeat, GetLBConfig.
- Plumb IamInstanceProfileArn through sysinstance.SystemInstanceInput and both
daemon launch paths (BootAMI via PrepareRunInstances->RunInstance; direct-boot
microVM via RunInstance), which persist it so the in-VM IMDS endpoint serves
the role.
- Drop the baked secrets when a profile is attached: omit EKS_ACCESS_KEY/
EKS_SECRET_KEY from the k3s first-boot env and LB_ACCESS_KEY/LB_SECRET_KEY from
the lb-agent fw_cfg env; relax k3s server validation to accept profile-only;
drop the EKS key requires from the first-boot and state-report scripts.
- Both verticals fall back to baked static creds when IAM is unwired, so an
absent master key never blocks a launch.
* fix: resolve IMDS instance-role creds under the instance account for system VMs
Three boot-path defects kept the LB VM's lb-agent from authenticating to the
gateway with IMDS-served instance-role creds; each produced a silent empty IMDS
role list and a heartbeat failing with "unexpected empty EC2 IMDS role list".
Found and fixed against env19.
- Build the system IAM ensurer lazily on the first system-VM launch instead of
eagerly at daemon startup, where it raced the NATS KV backend and fell back to
baked static creds permanently. ELBv2 and the EKS deps now hold an IAMProvider
resolved at launch time.
- Create the LB instance profile in the system account the LB VM runs under
(utils.GlobalAccountID), not the LB-owner account. IMDS resolves the profile
under the instance's account, so a customer-account role was invisible to it.
- Record the attached instance's owning account on the ENI (InstanceOwnerId,
mirroring AWS Attachment.InstanceOwnerId) and resolve the instance and every
IAM/STS lookup under it. The LB ENI lives in the customer account while the
system LB VM and its role live in the system account; keying the whole IMDS
chain off the ENI account left the instance invisible and the role lookup
missing. VPC leaves (subnet/VPC/SG CIDR) keep using the ENI account.
A fresh LB on env19 then authenticated end-to-end: IMDS serves a non-empty role
list plus ASIA creds, and the gateway accepts the heartbeat as an assumed-role
principal with no static keys in user-data.
* test(e2e): assert LB VM authenticates via IMDS instance-role creds
The LB suite's targets-healthy assertion already depends on the lb-agent
authenticating to the gateway, and with static keys dropped from user-data that
auth is IMDS-only. Add an explicit check that a recent gateway authentication
for the LB used an assumed-role (ASIA) principal, so a regression to the
instance-account IMDS resolution fails with a clear message instead of a
mysterious datapath timeout. The LB VM is a system-account instance invisible to
the customer AWS API, so the assertion observes the gateway journal on the
cluster nodes; it is time-bounded to avoid the stale-console-log false-fail and
skips when no node is SSH-reachable.
Julian Sommer