* feat: render predastore templates with the cluster and S3 planes split
The templates set one bind_addr per host, which put raft and blob wherever
S3 was published. A host now binds the cluster plane on the address peers
dial and the gate carries its own bind_addr for the S3 API, so the installer
can hand S3 the WAN without exposing replication with it.
The single-node template also drops to one node of each role. Extra blob
nodes only hash-partition keys across more badger indexes on the same drive,
and extra meta replicas add fsyncs for a quorum that dies with the machine.
* docs: regenerate the AWS model operation coverage table
* fix: repin predastore to the merged dev commit
Rework the frontmatter titles and descriptions for the five Identity
docs so each leads with the AWS concept term users actually search, and
add contextual in-prose links between them. Only the roles and instance
profiles page currently earns search impressions, so it now links to all
four siblings and each sibling links back into the cluster.
The branch imports predastore/pkg/auth, predastore/pkg/ratelimit and the
root predastore package, none of which exist in the pinned v1.15.0. Every
build that goes through the go.work workspace resolved those from the
sibling checkout and passed; the e2e binary build on the CI runner uses a
bare checkout governed by go.mod alone, and failed to resolve them.
The port tables were wrong in both directions: inter-node traffic is QUIC over
UDP rather than TCP, and the shard range 9991-9993 no longer exists. Ports are
unique within a host rather than across the cluster, so every machine uses the
same three, and a single-node install opens only the S3 port because colocated
nodes talk over an in-process pipe.
The reference architectures scripted a move of node data out of a directory
layout that no longer exists, which would have broken any install following
them. The update guide now says object storage has no migration path.
The fixture wrote a config the strict decoder rejects and built its cluster
through clusterrun, so it now renders a v1 TOML, loads it through LoadConfig and
serves under Run. Loading the real file means the fixture trips over the same
validation an install does. Its data directory holds node-<id> directly, and it
drains on cancellation rather than being abandoned at exit.
The runtime tests drove buildRuntime, serve and the old on-disk layout, all of
which predastore owns now, so they go. What replaces them covers what spinifex
still decides: the flag merge and its precedence, a host id or gate node the
config does not have, and the pid file behind Stop and Status.
Two fixes to earlier commits in this series: the host id conversion moves behind
its own guard, and the node status probes use WaitGroup.Go.
Hosts carry their nodes under [[host.node]] now, and a node's address is its
host's addr plus its own port rather than a single host address it shared. The
gate belongs to neither storage table, so it is matched explicitly and an
unknown role still warns.
The reported vocabulary follows predastore's: db_nodes and shard_nodes become
meta_nodes and blob_nodes through the daemon, the gateway and the UI. Bucket
type is gone from predastore entirely, so the field goes with it.
The facade owns every node of a host, the S3 gate among them, so the service no
longer builds a cluster runtime or a gateway of its own. Waiting for a leader,
draining the gateway and the shutdown grace all moved behind Run; pprof and the
base path have no equivalent and are gone.
spinifex.toml stays authoritative for the S3 endpoint: the host entry is
settled from the service flags before Run, and the gate is found by role rather
than by position. A host id is now required, so init writes one for single-node
installs too, where the topology was previously left implicit.
BasePath survives as the pid directory alone. Stop and Status now read the same
value start writes, which they did not before.
The decoder is strict now, so every key the reorganised predastore no longer
reads is a startup failure rather than an ignored line. Both templates move to
nested [[host.node]] tables with the gate, blob and meta roles, portless
bind_addr and addr, singular [[bucket]], and the host TLS and encryption-key
paths the schema requires. version resets to 1 with the format.
A gate node is what serves the S3 API, so a host without one answers nothing.
Ports are unique within a host rather than across the cluster: gate 8443, blob
from 6660, meta from 7660. Colocated nodes bind no socket and still need one.
The daemon dialled a /status endpoint on each meta node to report which held
raft leadership. That endpoint lived in predastore's s3db package; after the
transport collapse the meta nodes speak rpc over QUIC and the in-process pipe
and expose no HTTP surface, so the probe dialled a dead port.
NodeStatusResponse.PredastoreRole is retained and left empty. Every consumer
already guards on the empty string, so the CLI and the UI simply omit the
badge, and restoring the probe later needs no contract change.
The migration rewrote a v4 topology into v5 and recovered raft state through
predastore's clusterrun package, which no longer exists. The on-disk layout it
relocated data into has moved too: node directories are now node-<id> under the
host root, with no host-N level. Nothing migrates a pre-cutover install; it is
re-initialised from scratch.
node-reset.sh keeps directories by design, so it now removes the predastore
cluster tree outright rather than leaving the old skeleton standing.
* feat: add guardrail CRUD store and ARN support to bedrock gateway
Stage 1 of Ochre Guardrails: control-plane store for CreateGuardrail,
CreateGuardrailVersion, GetGuardrail, ListGuardrails, UpdateGuardrail,
and DeleteGuardrail, backed by a JetStream KV bucket keyed per account.
Records keep the full policy config so later stages can build the
filter engine and inference enforcement on top without re-deriving it.
* feat: add deterministic guardrail filter engine and ApplyGuardrail op
Adds the pure wordPolicy/sensitiveInformationPolicy filter (regex-detectable
PII plus a starter profanity list), scaffolds content/topic/contextualGrounding
as NONE, and exposes it all through the bedrock-runtime ApplyGuardrail route.
* feat: enforce guardrails on Converse and ConverseStream
Wire GuardrailConfig enforcement into Router.Converse and
ConverseStream via a shared enforceGuardrail helper (resolve, load,
apply the deterministic filter engine). INPUT blocks skip the
backend entirely; OUTPUT blocks or ANONYMIZE redactions rewrite the
completion before it reaches the caller. ConverseStream buffers
delta text and assesses it at block/message-stop boundaries rather
than streaming unassessed model output. Both self-host and
Anthropic-direct backends are covered automatically since the
enforcement sits above Router's per-backend dispatch. Requests
without a GuardrailConfig are unaffected. An unresolvable or foreign
guardrail fails closed with ResourceNotFoundException.
* feat: enforce guardrails on InvokeModel and InvokeModelWithResponseStream
Adds the Invoke half of Ochre guardrails (plan §5), reusing the Stage 3
enforceGuardrail helper and Stage 2 filter engine unchanged. Guardrail
selection comes from the X-Amzn-Bedrock-Guardrail{Identifier,Version}
headers. Since InvokeModel bodies are family-native/opaque at the
Router, text is extracted/reinjected per family (llama self-host,
anthropic) in the new guardrail_invoke.go. INPUT is checked before the
backend is ever called (fail closed on an unresolved guardrail, no
backend call); OUTPUT is checked after, handling block/anonymize.
InvokeModelWithResponseStream buffers the inner stream and assesses
once at end-of-stream via a guardrailInvokeStreamSource decorator,
mirroring Stage 3's guardrailStreamSource.
* fix: route ApplyGuardrail through bedrock-runtime and use guardrail-specific RNF text
The service discriminator only promoted /model/... paths from bedrock to
bedrock-runtime, so ApplyGuardrail's singular /guardrail/.../apply route fell
through to the control-plane handler and was rejected. Extend the
discriminator to also promote /guardrail/..., which can't collide with the
plural /guardrails control-plane CRUD routes.
Also give guardrail-not-found paths their own ResourceNotFoundException
message instead of the generic "could not resolve the foundation model"
wording shared with model-lookup failures.
CI builds spinifex in a workspace over viperblock and predastore only, so
northstar resolves from the proxy. Its v1.15.0 log sanitiser predates the
attribute.Value migration and does not compile against otel/log v0.21.0.
The log API moved bodies and attributes onto attribute.Value in v0.21.0, so the
otelslog bridge and the sibling log sanitisers have to move in the same window.
No spinifex source change is needed; the workspace resolves the migrated
predastore and viperblock.
* feat: parameterise bedrock endpoint create/delete by account and pinned
Widens EnsureEndpointInput/DescribeEndpointInput/DeleteEndpointInput
with AccountID and (Ensure-only) Pinned, and keys the daemon's
Ensure/Describe/Delete on the resolved account instead of the
hardcoded GlobalAccountID. An empty AccountID still resolves to
GlobalAccountID and Pinned defaults to false, so every existing
caller (resolver, reaper, evictForCapacity, the NATS gateway client)
is unchanged. This is stage 1 of provisioned throughput: it reuses
the existing launch/capacity/readiness machinery for a dedicated,
account-scoped, pinned endpoint without adding a parallel surface.
* feat(bedrock): add provisioned-throughput CRUD and PT ARN
Implements CreateProvisionedModelThroughput, Get, List, Update, and
Delete over a new bedrock-provisioned KV store, plus the
provisioned-model ARN format/parse. Create rejects provider-hosted
models and requests a pinned, account-scoped endpoint from the daemon;
Get derives status live from the pinned endpoint's state; Update
honours name changes only and rejects a served-model swap; Delete
tears down the pinned endpoint before removing the record.
EndpointProvisioner is declared in gateway_bedrock with primitive-typed
methods (handlers_bedrock already imports gateway_bedrock, so it can't
be imported back for its EndpointService interface directly); a small
adapter in handlers_bedrock bridges the two, always setting Pinned:true
and the caller's real account on every Ensure.
Does not yet wire a PT ARN into the Converse/InvokeModel inference
path (next stage).
* feat(bedrock): accept a provisioned-throughput ARN on Converse/InvokeModel
Translates a PT ARN before resolve: the gateway inference entry parses it,
loads the bedrock-provisioned record, and resolves against the pinned
endpoint for the commitment's own account rather than the GlobalAccountID
shorthand every bare modelId uses. Adds EndpointResolver.EndpointForAccount
(gateway-side interface, implemented by DynamicEndpointResolver) so the seam
stays import-cycle-safe, and threads it through vllm.go/invoke_llama.go's
resolve call sites. A foreign-account or unknown PT ARN reads as
ResourceNotFoundException, same as any other unresolvable model identifier.
* fix(bedrock): accept a provisioned-throughput ARN on streaming inference
ConverseStream and InvokeModelWithResponseStream built their router/adapter
with a nil ProvisionedStore and the GlobalAccountID-shorthand provider, so a
PT ARN was silently never translated on the streaming path even though the
non-stream Converse/InvokeModel siblings already handled it. Thread the
gateway's provisioned store through both entry points and give
Router.ConverseStream / InvokeStreamRouter.InvokeModelWithResponseStream the
same translate-before-resolve treatment Converse/InvokeModel already have.
* fix(bedrock): surface pinned endpoints in the operator CLI
Service.List only ever enumerated the GlobalAccountID key prefix, and
'endpoint describe' always looked up GlobalAccountID too, so a pinned,
account-scoped endpoint held a GPU while reading ABSENT and missing from
list. List now scans every account's records, and describe gains an
optional --account flag (default empty, preserving GlobalAccountID
resolution) to look up a specific account's endpoint. list/describe both
render the endpoint's account and pinned status.
* fix(ochre): self-heal weights stage when the recorded snapshot is gone
runStageWeights's idempotency check trusted a matching source URI without
verifying the snapshot it names still exists, so a host rebuild or volume
GC that lost the snapshot left stage permanently short-circuited on a
useless KV record until an operator ran 'weights remove' first. Inject a
weightsSnapshotChecker, verify liveness before the no-op return, and fall
through to re-materialise when the snapshot is gone. 'weights list' uses
the same checker to mark dangling entries so an operator can see which
staged models can actually serve without waiting for endpoint launch to
fail.
* docs: note weights stage self-heal and list STATUS column
Adds resolveServingAMI test cases: single match, multiple matches
selected deterministically by newest CreationDate independent of
describe-order, zero matches, and nil/empty ImageId entries skipped.
A binary-swap deploy (make deploy or a manual spx copy) ships a new binary
whose host-config prerequisites — the privileged helper scripts and their
sudoers grant, placed only by setup.sh / setup-ovn.sh — it does not ship.
The node looks healthy and the gap surfaces later as a cryptic launch
failure or a silent security regression.
Add a preflight package that checks the host against what this build
expects, comparing asset content, not existence, so a stale-but-present
asset is caught too. The managed helpers install from quoted heredocs, so
their on-disk bytes equal the heredoc body verbatim; the binary carries
the canonical sha256s and compares.
- spinifex/preflight: CheckHost / CheckHostAt, content hash of the two
managed helpers and a NOPASSWD grant-line check for the sysctl helper.
- Generated canonicalHashes plus a drift guard that re-extracts the
heredocs and fails if the committed manifest no longer matches.
- Three surfaces: spx admin preflight, a make deploy gate
(SKIP_ASSET_PREFLIGHT=1 to bypass), and a non-fatal daemon boot warning.
Release reported "nothing was claimed" as an error, and the terminated-instance
teardown reaper persisted that as the gpu teardown mark. The mark could then
never reach done, so the record was never purged and the GC re-drove it every
two minutes for as long as it existed, failing identically each time because
the condition being retried was that the work had already been done.
Stop-then-terminate is ordinary: the stop releases the GPU and the later
terminate finds nothing left to release. Two instances on wattle had been in
this state for days, at 2251 and 640 retries, logging an ERROR naming the
node's only GPU and falsely claiming it might need a manual rebind.
Release now returns a matchable ErrNoGPUClaimed and the teardown cleaner reads
it as success, which is what the reaper already documents its cleaners to do.
The sentinel stays an error rather than becoming nil so the launch-rollback
path can still tell that its claim/release pair is out of step, and so a real
vfio rebind failure - which clears Available, the health signal the Bedrock
capacity gate reads - stays distinct from the benign case.
checkCapacity read only PoolEntry.Available, but that field tracks device
health: Release clears it when a vfio rebind fails, so a device that cannot
be handed back is not offered again. Occupancy is the separate InstanceID
field, which PoolEntry documents as empty when free, and gpu.Manager.Claim
tests both before taking a device.
Reading Available alone let a GPU already held by a running serving VM admit.
Ensure never took its capacity refusal, so evictForCapacity was never called
and LRU eviction never fired: on a single-GPU node the launch instead
proceeded to STARTING and failed later with InsufficientInstanceCapacity,
with no eviction attempted and no reason logged.
The existing table had a case named 'already claimed' that set Available to
false, which is not the shape Claim leaves behind. That encoded the wrong
model and is why the gap went unnoticed, so it is renamed to what it really
covers and the real claimed-device shapes are added alongside.
Also stop daemonGPUSnapshotter capturing d.gpuManager by value.
applyGPUConfig swaps that pointer on a passthrough toggle and re-points
daemonGPUClaimer but not the snapshotter, which would leave bedrock reading
an orphaned pool that sees no claims and admits everything.
The catalog carried exactly one self-host entry, so two models could never
contend for a GPU: an Ensure for the model already running returns the
existing record before reaching the capacity check. LRU eviction was
therefore built and unit-tested but unable to fire in production.
Llama 3.2 3B Instruct is a real AWS-published Bedrock model ID, so it keeps
the drop-in-SDK promise, and it is a genuine quality tier above the 1B model
rather than a fixture added only to unblock a test.
Staying inside the Meta family is deliberate. InvokeRouter dispatches
self-host requests to llamaInvokeAdapter on tier alone, so a model of another
family would be served with Llama's native request and response schema
instead of being refused. A new test pins that assumption to the catalog so
it fails loudly rather than silently mis-serving.
Sizing for the RTX A1000: 3.21B params at bf16 is roughly 6124 MiB of weights
against 8188 MiB of VRAM, so --enforce-eager skips CUDA graph capture to free
several hundred MiB and --max-model-len is half the 1B entry's. fp8 is not an
option on this GPU: GA107 is compute capability 8.6, and fp8 needs sm_89.
These figures are estimates to be corrected against a live reading.
Weights gating is now measurably per-entry rather than all-or-nothing, which
only became observable once two self-host entries could disagree.
Three gaps found while verifying idle reclaim on live hardware.
The reclaim log line passed a time.Duration, which slog renders as an int64
nanosecond count in JSON: the line read idleFor:135000000000 rather than 2m15s,
in journalctl and in the metrics sink alike. It now logs an integer second
count under a unit-suffixed field, which is both readable and still
aggregatable — a string would have cost the latter. The same shape appears at
around ten other sites across the fleet and is tracked separately.
The endpoint command's help still told operators the gateway could not request
endpoints itself, which stopped being true when dynamic resolution landed. It
now describes the lifecycle that actually runs, and says what the manual
commands are still for.
describe showed nothing the idle sweep observed, so an operator asking why an
endpoint was or was not reclaimed had to read the KV. It now reports in-flight
count, last activity, idle duration, pinned status and consecutive scrape
failures for a READY endpoint, and nothing for any other state, since no other
state is swept.
LastActive moves onto EndpointRecord so the idle clock, the LRU eviction order
and this output all read one definition rather than three.
A refused launch and an idle GPU could coexist indefinitely: capacity said no
while a device sat under a model nobody was calling. Ensure now makes room once
before giving up, choosing the least recently used endpoint that is safe to
stop — READY, nothing in flight, not pinned, past minResidency, and on this
node, since GPU capacity is node-local and stopping an endpoint elsewhere frees
nothing here.
The eviction goes through Delete, so the device is released by the same
DRAINING -> terminate -> release path an operator delete takes rather than
being taken from underneath a running process. Nothing evictable returns the
original ModelNotReadyException unchanged: capacity contention breaches no
quota and must not claim to.
minResidency defaults to 5m so two models cannot thrash each other with a full
cold boot per call. Beyond that the deployment is under-provisioned, and the
eviction rate is what says so.
Also adds coldStartWait (OCHRE_COLD_START_WAIT), default 0. At zero the
fail-fast contract is unchanged and costs no extra describe. Above zero the
resolver polls inside its existing singleflight, so a burst of cold callers
shares one wait and one launch. The default stays 0 because the measurement
behind it has not changed: cold start is minutes, and no client retry policy
spans that.
A launched model held its GPU until an operator deleted the endpoint. On a
single-GPU deployment that made every other model in the catalog permanently
unreachable, with no path to recovery that did not involve a human.
The reaper scrapes each READY endpoint's vLLM Prometheus /metrics and reads
idleness from the serving process itself rather than from gateway-reported
activity, which keeps the daemon the single writer of endpoint state and works
unchanged with any number of gateways. Idle means both scheduler queues are
empty and the completed-request counter has not advanced since the previous
sweep: the counter clause is what catches a request that started and finished
entirely between two sweeps.
Two rules bound the reclaim. An endpoint is never reaped within idleTTL of
readyAt, so it cannot be taken back between reaching READY and the request that
launched it arriving. A failed scrape is unknown, never idle; five consecutive
failures terminate and relaunch, so a wedged VM is not mistaken for a quiet one
and left holding a device.
Observations are persisted on the record rather than kept in the reaper's
memory: a leader change must not reset the idle clock, and the eviction
decision that follows in the next change runs on whichever replica the queue
group picked. A sweep writes only when what it saw actually changed.
The inference path resolved self-host models from a static pinned map and
never asked for a launch, so on any deployment that had not pinned an endpoint
by hand a staged model was advertised, accepted by the access gate, and then
rejected by every invoke forever with no VM ever started.
Add a resolver over the endpoint registry, satisfying the existing
EndpointResolver interface so the provider and router layers are unchanged.
It lives in handlers/bedrock rather than gateway/bedrock because that package
cannot import this one; restating the subjects and payloads gateway-side would
be two definitions of one wire contract, free to drift.
Resolution order keeps the escape hatch first: OCHRE_VLLM_ENDPOINTS wins, then
a short-TTL cache of READY base URLs, then describe. STARTING and DRAINING do
not resolve and do not re-request, since a launch or a teardown is already in
flight. ABSENT requests one and reports not-resolved, which the invoke paths
turn into a retryable ModelNotReadyException — the cold-start contract.
Concurrent cold calls for one model collapse through singleflight, so a burst
costs one describe and one ensure rather than a pair per request. Describe and
ensure errors propagate rather than being flattened into a cold start, so a
dead control plane stays distinguishable from a model that is merely starting,
and the daemon's own refusals keep their AWS codes.
finalizeTerminated treated a refused terminated -> terminated transition as a
failure, but reaching the state we were driving to is convergence: the VM GC
had reconciled the same instance while terminateCleanup was running.
The caller pays for that error. Observed on wattle: an Ochre endpoint delete
wrote DRAINING, lost this race, and returned before the DRAINING -> ABSENT
write, so the VM was gone and the GPU released while the record stayed
DRAINING. Delete only accepted READY, so no retry could clear it and the model
was unreachable until the KV was edited by hand. ELB, EKS and RDS system
instances race the same GC.
So converge in the vm layer when the instance did reach terminated, keeping
the refusal for one that did not, and let Delete resume from DRAINING: a
teardown that failed partway has to be retryable.
* feat(bedrock): gate Ochre models behind per-account access grants
Self-host models were invokable by any authenticated account: tieredCatalog
returned them unconditionally and no runtime router checked the caller.
GetFoundationModel also described models the caller could not use.
Add a bedrock-model-access JetStream KV bucket holding presence-only grants,
resolved through a new AccessResolver. Access is deny-by-default: a gateway
with no store configured falls back to DenyAllAccessResolver, so an
unconfigured deployment serves nothing rather than everything. The system
account bypasses grants, matching how quotas exempt it.
All four runtime routers share one grantedCatalogEntry gate so they cannot
drift on who may invoke what. List and describe agree: an ungranted model is
absent from ListFoundationModels and reported as ResourceNotFoundException by
GetFoundationModel, so the error cannot confirm a model exists.
Administration lands on two surfaces: Grant/Revoke/ListModelAccess actions on
the spinifex gateway service for the UI, and `spx admin ochre access` for
operators and provisioning, which opens the bucket over NATS directly rather
than requiring a SigV4 client the CLI does not have.
* feat(bedrock): seed the admin account's model grants at gateway start
Deny-by-default left a fresh install with a catalog nobody could see, and the
first cut filled that gap from ansible — which is a dev tool, so any install
that never ran it came up unusable.
Seed from the product instead: on start awsgw grants the platform admin
account the full catalog, once, behind a marker key in the same bucket. The
marker makes it a bootstrap step rather than a policy: tenant accounts still
begin with no access, and an operator who revokes a grant does not have it
restored by the next restart. Grants are written before the marker so a crash
midway repeats the idempotent grants rather than leaving a half-seeded
account, and the marker is a conditional create so only the first node to
reach it need win.
Seeding is best-effort: a KV failure logs and leaves the gateway serving,
and the next start retries because the marker is written only on success.
* test(bedrock): cover the model-access CLI and gateway admin actions
Route the access CLI runners through the same loadConfigAndConnectFn /
ochreExit seam the weights runners use, so grant, revoke and list can be
driven against an embedded JetStream server, and add gateway tests for the
grant admin actions including the non-admin denial.
* feat(images): add a vllm-serving AMI as a stacked mkosi profile
LaunchServingVM resolves its AMI by the tags spinifex:managed-by=bedrock and
spinifex:bedrock-role=vllm-serving. No image carried either, so every Ensure
failed at resolveServingAMI. The only GPU image ships the NVIDIA driver and
Docker but no Python, no vLLM and no serving unit, so tagging it would have
moved the failure from admission to first boot.
Add a vllm profile that stacks onto gpu-nvidia and owns the whole guest side
of the launcher contract: the runtime, vllm-serve.service, the weights mount
and the tags. vLLM is baked into a uv-managed venv rather than pulled as a
container at boot — a runtime docker pull of vllm/vllm-openai is known to
stall this platform's write plane, and mkosi's unprivileged build has no
daemon to bake an image from anyway.
Also bring images/mkosi.profiles/ under make test-images. Its scripts have no
.sh suffix, so the existing name glob skipped exactly the files worth linting;
sources are found by shebang instead.
* fix(images): make the vllm postinst survive the build sandbox
Two failures found by actually running the build.
uv installs to /usr/local/bin, which mkosi's postinst PATH need not carry, so
`command -v uv` reported a good install as missing. Call it by absolute path,
and check it the same way.
`vllm --version` initialises device config and dies with "Failed to infer
device type" on a builder with no GPU, so it could never have verified
anything. Assert the pinned version through package metadata instead, which
also catches a version mismatch rather than merely a broken entrypoint.
* fix(images): pin the serving AMI datasource and mirror cloud-init to the console
A serving VM boots through LaunchSystemInstance, so it comes up dual-NIC with a
fw_cfg netcfg blob and takes its user-data solely from IMDS — the same shape as
the EKS control plane. That path carries a cloud-init datasource pin; the vllm
profile never got one, and the base config deliberately leaves datasource_list
unpinned, so the serving image was the only system VM booting without it.
Reuse the EKS pin rather than copying it. The 99- prefix is load-bearing:
read_conf_d() applies cloud.cfg.d in REVERSE and mergemanydict() is
first-source-wins, so a 90- name loses to cloud-init's own 90_dpkg.cfg ('_'
sorts above '-') and the pin is discarded in silence. The postinst asserts the
MERGED datasource_list via cloud-init's own read_conf_with_confd rather than
checking the file exists, because it existed at 90- too and did nothing there.
Also mirror every cloud-init stage to the serial console. A serving VM is
headless, and a guest that wedges before handover never surrenders
/var/log/cloud-init-output.log, so the console get-console-output reads is the
only channel an operator has.
The images_test.go case covers the two tags resolveServingAMI filters on; it
matches on those alone and never on name, so a typo hides the image entirely.
* fix(images): make the dhcpcd data-NIC lease genuinely one-shot
dhcp_oneshot() promised to configure the link and exit "leaving no daemon
to fight cloud-init", but the dhcpcd branch passed only -q, which is quiet
rather than one-shot. Every Ubuntu image takes that branch, so a dhcpcd
daemon survived on the data NIC holding the control socket.
cloud-init's Ec2 datasource then runs its own ephemeral dhcpcd on the same
NIC. Instead of spawning, dhcpcd forwards the invocation to the resident
daemon as a control command and exits 0, and --dumplease returns the
resident daemon's lease, so discovery looks successful. Cleanup then waits
for a pid file at /run/dhcpcd/<iface>-4.pid that no process ever wrote,
spinning its full 30000 x 10ms budget: init-local stalls for exactly 300s
on every system-instance boot.
Adding -1 exits once the interface is configured and -p keeps the address
and routes across that exit. Both are needed; -1 alone tears the link back
down. The test now models the real clients — the stub stays resident unless
given the flag that makes its client exit, and the dhcpcd branch is forced
by dropping udhcpc from PATH rather than left to whether the host happens
to have /usr/share/udhcpc/default.script.
* fix(images): serve without a CUDA toolkit and stop vllm-serve crash-looping
vLLM picks FlashInfer for top-k/top-p sampling, and FlashInfer JIT-compiles
that kernel on first use. The serving image ships the driver and the
container toolkit but no nvcc, so the sampler warmup died with "Could not
find nvcc and default cuda_home='/usr/local/cuda' doesn't exist" minutes
into a GPU cold start, after the model had already loaded.
VLLM_USE_FLASHINFER_SAMPLER=0 selects vLLM's native top-k/top-p sampler,
which is the same operation with no CUDA toolchain at runtime. Everything
else on the serving path is already precompiled in the wheels: the attention
backend resolves to FLASH_ATTN and CUDA graph capture completes, both
without nvcc. Baking FlashInfer's AOT modules is the alternative if the
fused kernel is ever wanted, and it keeps the toolchain in the builder
rather than the appliance.
The unit had Restart=on-failure with no bound, so an unfixable start failure
restarted forever and buried its own first traceback under identical copies
on the serial console — the only view an operator has of a headless guest.
StartLimitIntervalSec/StartLimitBurst make it report instead.
The postinst asserts the installed unit carries the sampler setting, for the
same reason the datasource pin is asserted there: the alternative is finding
out at runtime, on a GPU, five minutes in.
Also corrects the shared datasource pin's header, which still claimed it
installs as 90-spinifex-ec2-imds.cfg. It is installed at 99- by the vllm
profile precisely because a 90- name loses the reverse-sorted, first-source-
wins cloud.cfg.d merge to cloud-init's own 90_dpkg.cfg.
* fix(bedrock): raise the serving-endpoint readiness bound to 15 minutes
The 5 minute bound was chosen before any cold start had been measured, and
its own comment asked to be corrected once a model actually ran on wattle.
It now has: Llama 3.2 1B on an RTX A1000 reached "Application startup
complete" 257.6s after launch, leaving ~42s of margin on the smallest model
this platform can serve, with a cold torch.compile accounting for 26.3s of
it. A bigger card and a bigger model are unmeasured.
The two failure modes are not symmetric. Too low tears down an endpoint that
was about to come up and reports it as a failed launch; too high only makes
a genuinely stuck launch take longer to give up. So the bound is set well
clear of the single data point rather than close to it.
The CLI's own wait moves to 20 minutes to stay above it. The daemon starts
its clock when the VM is launched, up to a minute after the CLI starts its
own, so leaving both at 15 would expire the client first and report a bare
"timed out" in place of the daemon's abort, which carries the reason.
* fix: give the Ochre daemon a routed port in the Bedrock system VPC
The readiness probe and every InvokeModel call dial a serving VM's
system-VPC address from inside the daemon, but that subnet lives in OVN
on br-int and the host has no route to it, so a healthy VM never reached
READY.
Each daemon now owns an ENI in the system VPC's private subnet, bound to
an OVS internal port on br-int by iface-id, exactly as ovn-controller
binds a guest tap. The ENI address is installed at the subnet's prefix
length, which is what supplies the connected route. BaseURL is unchanged:
the address it already carried is now reachable.
Describe-or-create keyed on a per-node description, since ENIs carry no
tag filter, so a restart or a second endpoint reuses one address instead
of leaking one per launch. The port is ensured before anything else the
launch creates: a VM the daemon cannot dial holds a GPU for the whole
readiness window before unwinding.
Both actions carried no scope, so their policy check evaluated against "*".
Matching is pattern-to-value, so a statement whose Resource is a specific ARN
never matches "*" — a Deny fencing an instance blocked the scoped
ModifyDBInstance and DeleteDBInstance but not a snapshot of the same instance.
The principal could snapshot the fenced instance, restore it under a name the
Deny does not cover, set a master password of their choosing and read the data.
Each is scoped to its source: the instance for a create, the snapshot for a
restore, which breaks the chain at its first step. Both identifiers are required
on their action and ResourceARN already falls back to "*" when absent, so this
narrows where the identifier is present and is a no-op where it is not.
Evaluating both ARNs per action, for AWS parity, is deliberately left out: it
turns the scope into a slice across all 13 scoped entries and touches the shared
dispatcher call site.
* feat: add spx admin ochre endpoint ensure/describe/list/delete
The daemon subscribes all four bedrock.endpoint.* subjects, but nothing in
production calls their NATS client, so the serving-endpoint lifecycle could
not be exercised at all in a running system. This gives an operator a direct
way to drive it.
ensure --wait polls describe until READY and reports the elapsed time, since
the daemon replies STARTING as soon as it claims the model and the launch
outlives the request. That is the measured cold start.
Also corrects the weights stage tokenizer rule in COMMANDS.md, which still
described both tokenizer files as required rather than either.
* test: cover the ochre endpoint Run wrappers' connect and exit paths
* fix: treat an empty endpoint bucket as no endpoints, not an error
ListEndpoints was the only kvutil.Keys caller not handling ErrNoKeysFound,
so 'ochre endpoint list' failed on a cluster that had never launched one.
weights stage created the snapshot at the viperblock layer only, writing
the block checkpoint and config.json but not the metadata.json that
CreateVolume resolves a SnapshotId through. The endpoint launcher clones
the weights volume via ec2.CreateVolume, so every launch failed with
InvalidSnapshot.NotFound against a snapshot that was present on disk.
The GPU model table was only ever used to report VRAM, so wrong or missing
entries were cosmetic. Bedrock's capacity check is the first code that rejects
on VRAM, which turns every latent data error into an admission decision.
Correct seven entries verified against pci.ids — most seriously 1002:7448,
labelled Instinct MI300X at 192GB when it is a Radeon Pro W7900 at 48GB, a
4x over-report that admits a model which then OOMs at weight load. Record the
smallest capacity where one PCI ID spans several cards, since under-reporting
refuses an endpoint an operator can see and override while over-reporting
fails later and looks like a vLLM bug.
Widen the table so common datacenter, workstation and consumer cards resolve
without an operator-supplied override, and apply GPUModelOverrides to the pool
devices as well as the advertised model, so the override reaches the code that
actually gates. An override with no memory_mib is left alone rather than
wiping discovered VRAM, since the config cannot distinguish it from a
deliberate zero.
MemoryMiB 0 stays the "undiscoverable" sentinel, but checkCapacity no longer
counts it as a device with no VRAM: an otherwise-free device now reports which
card is unknown and how to fix it.
* fix: make the predastore readiness gate a real readiness probe
checkPredastoreReady only did a TCP dial + close, so a listening socket
satisfied it well before predastore could serve TLS. In CI this passed
in 170 microseconds, 0.72s after predastore opened its listener, and
recovery started against a backend that wasn't serving yet — predastore
logged the probe itself as a TLS handshake EOF.
Replace the dial with an HTTPS GET against the configured predastore
endpoint on a package-level client (hoisted out of the 2s poll loop),
with a per-request 3s context timeout. Any HTTP response counts as
ready, including 403 from this deliberately unsigned request, which is
exactly what an unauthenticated S3 GET returns. Only a transport or TLS
failure means not-ready.
This narrows the race window and removes the spurious TLS-handshake-EOF
log lines predastore emits on every daemon start, but it does not close
the window on its own: predastore can serve TLS before its storage
backend answers reads correctly. The load-bearing fix is the viperblock
VBState read retry landing separately.
* style: trim the predastore readiness comments to three lines
GetPasswordData returned an empty string both when the guest had not emitted a
password yet and when it never could, because the launch key pair was ED25519
and cannot perform PKCS#1 v1.5 encryption. The two are indistinguishable, so a
caller in the second case polls forever with no signal.
An empty result on a Windows instance whose launch key is not RSA now returns
InvalidKeyPair.Type naming the cause and the fix. Every uncertainty keeps the
old empty result: a populated blob always wins, Linux is never judged, and an
unresolvable key is not treated as evidence of the wrong type.
get-password-data --priv-launch-key requires PEM, but ssh-keygen writes the
OpenSSH container by default, so even an explicitly-RSA key pair failed with
"Unable to decrypt password data using provided private key file" — which
reads like a padding problem rather than a format one. AWS returns PKCS#1 PEM
for RSA key pairs. OpenSSH reads that format for SSH either way.
The create-key-pair UI gains a key-type selector, defaulting to the rsa the
page already sent, with a hint that RSA is required for Windows password
retrieval.
The API default stays ed25519. Changing it is a separate decision.
ResourceNotFoundException is a shared wire code, so Bedrock aliases the EKS
error constant and inherited its message: a caller asking for a model was
told to list its EKS clusters. Register Bedrock overrides in the existing
errorLookupByService table, under both service keys since the gateway splits
bedrock from bedrock-runtime on the /model/ path prefix.
PutModelInvocationLoggingConfiguration also accepted embeddingDataDeliveryEnabled
and dropped it: the internal LoggingConfig had no such field, so the flag never
survived to Get. Carry it through both translation directions and fold it into
the bucket-required check, since embedding delivery with no bucket is as
undeliverable as text or image.
cloudbase-init has no IMDSv2 token support in any release, so a Windows
instance launched without an explicit MetadataOptions block 401s on every
metadata GET and never reaches its main plugin stage. Derive the launch
default from the resolved image's platform: Windows gets optional, every
other platform keeps required, and an explicit request value always wins.
* feat: add spx admin ochre weights stage/list/remove
Give operators a supported path to stage a self-hosted model's
weights: point at a predastore S3 prefix holding ordinary Hugging
Face files, and stage materialises them into a viperblock volume,
snapshots it, and records the source URI and snapshot ID in the
bedrock-weights KV bucket.
stage refuses before downloading anything if the model is not a
self-host catalog entry, or if the prefix is missing any required
file. Re-staging the same source is a no-op; a different source
replaces the entry and reports the previous snapshot for separate
reclamation.
WeightsStore gains GetWeights and a source-URI field on its record
to support this. LookupServingSpec (catalog.go) is reused unchanged
to validate --model-id.
* fix: unify ServingSpec after rebase onto dev's daemon-launcher spec
Rebasing onto dev pulled in mulga-vmfng.7.5's daemon-side ServingSpec
and LookupServingSpec, textually duplicating the ochre weights stage
CLI's own versions in catalog.go. Collapse both into one type carrying
ModelID and VLLMArgs, and one three-value LookupServingSpec so callers
distinguish an unknown model (found=false) from a provider-tier one
(selfHost=false). Update the daemon launcher's call site accordingly.
* test: cover ochre weights stage/list/remove command behaviour
Splits runOchreWeightsStage/List/Remove into testable decision functions
(runStageWeights, listWeightsOutput, removeWeights) separate from cobra and
NATS wiring, and makes mkfs.ext4 execution and the connect/exit steps
swappable so the full command flow can be driven against an embedded
JetStream server instead of live infrastructure. Also relaxes the tokenizer
file requirement to accept either tokenizer.json or tokenizer.model, since
canonical Llama 3.x repos ship only the former at the prefix root.
* fix: resolve mkfs.ext4 from sbin when it is absent from PATH
e2fsprogs installs mkfs.ext4 into sbin, which a non-login shell omits, so
weights staging failed on an unprivileged operator's otherwise fine host.
* fix: size the weights volume to a whole GiB so the tail block fits
viperblock rejects a write whose block overruns VolumeSize, so importing an
image of unaligned length failed on its last block. Rounding up also makes
the volume's byte size and its manifest SizeGiB describe the same volume.
Name the volume in the import error so a part-written import is traceable.
DescribeImages skipped any ami-* prefix whose config.json could not be
fetched, read, or parsed, logging each at DEBUG. At the daemon's default
INFO level the result was an empty Images list and a completion line
reading count=0, indistinguishable from a healthy empty registry.
Promote the three fault branches plus the body-close failure to WARN,
leave NoSuchKey at DEBUG since a prefix with no config is a legitimate
non-AMI directory, and report prefixes examined alongside images
returned so a mismatch is visible without raising the log level.
SetServiceOwnership chowned per-service directories to service users
without propagating a missing user, so spx admin init reported success
while leaving /etc/spinifex/northstar root-owned 0600. ChownRecursive
and SetServiceOwnership now return errors, failures are aggregated
across every service rather than stopping at the first, and
finalizeNodeSetup prints the unapplied paths instead of a success mark.
The per-tap datapath only matched IP, on the assumption that a guest reaches
169.254.169.254 through its default gateway. A guest that treats 169.254.0.0/16
as on-link resolves the address itself instead, and nothing on br-imds, br-int
or in OVN answered that ARP, so it could not reach IMDS at all.
Windows does exactly this, which left cloudbase-init timing out on every
metadata request and skipping the rest of its plugin stage.
Install a per-tap ARP responder for each captured address under the existing
cookie, replying with the gateway MAC the egress flow already presents.
The viperblock start command applies --s3-host, --s3-bucket and --s3-region over
spinifex.toml whenever viper reports a non-empty value. s3-bucket and s3-region
carried non-empty defaults, so the override fired on every start even though the
service unit passes neither flag nor environment variable, and the region read
from the config was replaced with ap-southeast-2.
Any cluster deployed to another region could not launch an instance: viperblockd
signed its S3 requests with ap-southeast-2, predastore verified them against its
own region and rejected them as AuthorizationHeaderMalformed, the volume mount
failed and RunInstances ended in launch_failed. The bare-metal e2e cell, which
installs with --region bare-metal, failed every instance-backed test this way.
Default all three to empty, matching s3-host, so an unset flag leaves the
configured value alone while an explicit flag or SPINIFEX_VIPERBLOCK_* still
wins. The predastore.toml template also pinned the main bucket to a literal
ap-southeast-2 while the top-level key and the northstar bucket both rendered
the configured region; it now renders it too.
Introduces handlers/bedrock, the daemon-side owner of vLLM serving VM
placement and lifecycle. State lives in a new JetStream KV bucket
(bedrock-endpoints) keyed {accountID}/{modelID}, cycling through
ABSENT -> STARTING -> READY -> DRAINING -> ABSENT. The daemon is the
sole KV writer; the gateway (or any other caller) requests transitions
over NATS via bedrock.endpoint.{ensure,describe,list,delete}, wired
into subscribeAll exactly as handlers/eks registers its own subjects.
Ensure is idempotent and single-flighted at two layers: a local
per-key mutex collapses concurrent same-process callers, and a
JetStream KV Create/CAS on the record's generation collapses races
across daemon replicas, so a burst of concurrent Ensure calls for one
model launches exactly one VM either way. Capacity is admission-checked
against gpu.Manager's free devices before a launch is claimed, so a
refusal never leaves a stuck STARTING record. Readiness is an explicit
poll of the serving VM's /v1/models endpoint until 200 or
startupTimeout, never a sleep.
Serving VMs boot via the system-instance path against a dedicated
system VPC, and follow the Phase-0 weights resolution: a snapshot
staged out-of-band is COW-cloned per endpoint into a data volume the
VM attaches post-boot, decoupled from the boot AMI.
Adds the .7.6 metering slice on top of .7.4's invocation stream: a
second durable consumer (ochre-usage-metering) aggregates per-account
token counts and external cost into a new bedrock-usage KV bucket,
keyed {accountID}/{modelID}/{period}. RequestID dedupe (an atomic
create-if-absent marker in bedrock-usage-dedupe) makes redelivery
count-neutral rather than double-billing a customer.
Prices are integer micro-USD per million tokens (D12): in-tree
catalog defaults with KV overrides in bedrock-prices, mirroring the
serving-spec shape from .7.3. Self-hosted models always resolve a
known zero; a model with neither an override nor a default resolves
Known=false, which the usage counter surfaces as a sticky CostUnknown
flag distinct from a genuine zero-cost period.
handlers_quota gains two Bedrock dimensions, both defaulting to
disabled independently of the block's existing Enabled toggle:
tokens-per-month (a check-against-history hard cap reading the
stream-fed counter, tolerant of a few seconds of lag) and
requests-per-minute (a local per-account token bucket enforced
immediately in-process, with a periodic KV sync for cross-node
observability only — enforcement never round-trips). Both are wired
into BedrockRuntime_Request ahead of every Converse/InvokeModel
variant.
* feat(ec2): serve the Windows administrator password via GetPasswordData
Adds the GetPasswordData EC2 API, mirroring the existing GetConsoleOutput
path: a per-instance NATS subscription, a daemon handler, and a gateway
shim registered in ec2Actions. The guest emits its encrypted administrator
password to the serial console at first boot; the daemon scans the whole
console log (not just the AWS 64KB tail, since the password would scroll
off a long-running instance) for the last <Password>...</Password> block
and returns its base64 ciphertext unchanged. Spinifex never handles key
material and never sees the plaintext password.
onInstanceUpHook's per-instance topic binding is refactored from two
hand-rolled subscribe-then-rollback blocks into a loop over the topic set,
so adding this third topic doesn't nest the rollback logic further and a
failure partway through still unwinds every topic already bound in that
call.
* test(ec2): cover the GetPasswordData gateway shim and remaining daemon branches
The gateway shim had no test file, leaving diff coverage below the
preflight threshold. Adds GetPasswordData_test.go covering input
validation, the happy path against a stubbed NATS responder, the
ErrNoResponders and ErrTimeout mappings to InvalidInstanceID.NotFound, a
daemon error payload, and a malformed JSON response. The request timeout
is now a package variable so the timeout test doesn't have to wait out
the real 5 seconds.
Also adds daemon-side coverage for the missing-InstanceId, malformed
input payload, and cross-account ownership-denied branches of
handleEC2GetPasswordData.
DescribeImages and DescribeInstances only ever populated PlatformDetails,
which is a billing string covering every OS. The AWS SDKs and the Terraform
provider key off the separate Platform field, which is either "windows" or
absent, so a Windows AMI read as Linux.
Derive Platform from PlatformDetails rather than persisting a new AMI field,
so already-imported Windows AMIs are fixed without a re-import.
The v4 to v5 topology migration runs as root under `spx admin upgrade`,
but predastore runs as spinifex-storage with an empty capability
bounding set. Two paths left it locked out of its own state.
The migration created the cluster data dir as root:root 0750, so
spinifex-storage could not traverse into the node dirs the rename had
placed under it, and clusterrun could not create new ones. Separately,
the raft recovery that follows each state replica move reopens badger
and writes a fresh snapshot as root, leaving root-owned files inside a
node dir that the rename had otherwise left correctly owned.
Both now take their ownership from the data being moved rather than
from a service user name, so a dev install running as the invoking user
needs no special case. The data dir chown is unconditional so a re-run
repairs a dir left behind by an earlier failed run, and the tree chown
runs even when there was nothing to recover, since that path still
opens the stores.
* feat: record every Bedrock invocation to JetStream, deliver to S3 and logs
Publishes an InvocationRecord for every Converse/InvokeModel call (streaming
or not) to a durable JetStream stream, then fans it out via a delivery
consumer to the account's own predastore bucket (body text, gated per
account by PutModelInvocationLoggingConfiguration) and a metadata-only log
line. Publish failures increment a drop counter instead of failing the
invocation. Streaming pumps record on every exit path via defer, including
client disconnect, and Close() now cancels the upstream request outright so
a disconnected client actually frees backend generation. vLLM streams
estimate output tokens from content-delta counts when real usage isn't
available yet, flagged usage_estimated; Anthropic always reports real usage.
* fix: deliver invocation records to S3 regardless of body, replicate stream
deliver() was gating the S3 write itself on InputText/OutputText being
non-empty, so an account with a destination bucket configured but body
delivery left off got nothing — the delivery flags already gate the body
fields upstream in Record(), and shouldn't gate delivery a second time.
EnsureInvocationStream defaulted to a single replica on a multi-node
cluster, unlike its sibling KV stores, so losing one node lost billing and
audit records without the control plane itself going down. Also renames
the awsgw predastore client from ecrStore to objStore now that it backs
both the ECR registry and Bedrock invocation delivery.
* feat: in-tree serving specs + KV weights override for ochre catalog
Extend catalogEntry with MinVRAMMiB, InstanceType and VLLMArgs so a
self-host model's serving requirements ship with the catalog rather
than living only in an operator's head. Add WeightsStore, a
deployment-scoped (not account-scoped) JetStream KV override in a new
bedrock-weights bucket that resolves a model's staged weights snapshot
ID, following the same shape as the existing CredentialStore. It is
plumbed into tieredCatalog and GetFoundationModel through a
package-level resolver (SetWeightsResolver/currentWeightsResolver),
since both are called from gateway/bedrock.go's fixed-arity route
table, which is out of scope here.
A self-host model with no resolvable weights snapshot is not
servable and must not be advertised: tieredCatalog filters it out and
GetFoundationModel returns ResourceNotFoundException for it.
Replace the meta.llama3-70b-instruct-v1:0 self-host entry, which
needs roughly 140 GB of VRAM in fp16 and has never actually run, with
meta.llama3-2-1b-instruct-v1:0, a real AWS-published Bedrock model
ID. At bf16 its 1.24B params are roughly 2.5 GiB of weights, well
inside wattle's single RTX A1000 (8188 MiB); MinVRAMMiB is set to
4096 to budget headroom for CUDA context, activations and KV cache on
top of that. This is an estimate, not a measurement off wattle -
.7.5's launcher should correct it against a real nvidia-smi reading
once the model is actually running there.
lookupCatalogEntry, used by the untiered invoke/converse routing
path, is left unchanged and still ignores weights resolvability;
gating that path is out of scope for this bead.
* fix: separate weights resolve errors from not-found in ochre catalog
An internal fault (e.g. a JetStream outage) reaching the KV weights
resolver was being collapsed into the same "not servable" verdict as
a clean not-found: tieredCatalog would silently unadvertise every
self-host model, and GetFoundationModel would return
ResourceNotFoundException for a model that does exist but whose
resolver is temporarily broken. Both call sites now distinguish the
two: a resolve error is logged and surfaced as its own error (mapped
to ServerInternal at the AWS error boundary), and only a clean
not-resolvable result yields ResourceNotFoundException.
tieredCatalog's signature changes to return an error alongside the
entries so ListFoundationModels can propagate a resolve fault instead
of silently returning a thinned-out list.
Also raises the self-host entry's gpu-memory-utilization from 0.5 to
0.6 (roughly 4913 MiB of vLLM's own pool) and MinVRAMMiB from 4096 to
5120 to match: at 0.5, weights (~2.5 GiB) left under 1.5 GiB for KV
cache and activations at max-model-len=8192, likely too tight for any
real concurrency. The two numbers are meant to agree: MinVRAMMiB is
the admission-gate floor, gpu-memory-utilization is what tells vLLM
to actually claim that much on an 8188 MiB device.
Comments trimmed to the repo's 3-line limit and stripped of plan/bead
references; the VRAM reasoning and model-choice rationale live in the
commit body and the plan doc instead.
Threading the weights resolver through as an explicit parameter
(mirroring CredentialResolver) rather than a package-level global was
considered per review. Doing so needs edits to gateway/bedrock.go
(bedrockRouteHandler's signature, both route closures, the
Bedrock_Request call site, a new bedrockWeightsResolver() method) and
gateway/gateway.go (a new BedrockWeights field on GatewayConfig).
Both are outside gateway/bedrock/ and are explicitly out of scope for
this change, owned by a parallel workstream already modifying the
same route table. The global resolver stays, now with the
error-handling fix above.
* fix: stop e2e GetFoundationModel regressing on the new weights gate
TestGetFoundationModel's "known model" subtest asserted on
meta.llama3-2-1b-instruct-v1:0 unconditionally, but that entry is now
gated on a weights snapshot staged in the bedrock-weights KV bucket.
On any real deployment that bucket starts empty, so the subtest would
fail with ResourceNotFoundException outside of OCHRE_E2E_SELFHOST.
The "known model" assertion now targets the Anthropic entry, which
resolves unconditionally and keeps testing what the subtest was
actually there for: a known model round-trips, an unknown one is
ResourceNotFoundException. The self-host assertion moves to a new
TestGetFoundationModelSelfHost, gated behind OCHRE_E2E_SELFHOST=1
alongside TestConverseSelfHost, where weights are expected staged.
tests/integration/bedrock_test.go's TestListFoundationModels already
wires a weights-resolver stub before asserting the self-host entry is
present, so it does not regress the same way.
* test: correct a stale assertion message on the ochre catalog weights gate
Viperblock's ordered extent index pulls in tidwall/btree, so the bump needs
the matching go.sum entry or the build fails resolving a transitive import.
Spinifex imports github.com/mulgadc/predastore/clusterrun, but go.mod still
pinned the pre-merge pseudo-version 608939ef2bb4, which predates that package.
Off-workspace builds (CI, which has no go.work) fell back to resolving the
import at @latest — v1.14.0 — and failed.
* match new predastore config surface area
* fix(migrate): carry existing installs onto the predastore v5 topology
The cluster topology refactor breaks an install three ways at once. The old
config carries a top-level host key, which the new schema binds to the [[host]]
table array, so TOML rejects the document and the daemon dies before it reads
any topology. Node state moved from distributed/{db,nodes}/node-N to a per-host
data directory keyed by a node id that is now unique across roles. And because
object metadata records which node holds each shard, the shard nodes had to
keep their ids, so the state replicas took new ones and every replica's
persisted raft configuration stopped listing it.
predastore.toml gains a 4 to 5 migration covering all three: it rewrites the
tables, relocates whichever node directories exist on this machine, and
recovers each relocated state replica onto the new member set. The rewrite is
line-oriented rather than a re-render from the template, so buckets, auth,
rate limits and iam settings an operator edited survive.
spinifex.toml gains a 3 to 4 migration renaming the predastore node_id to
host_id, which predastore-start.sh greps to select the host this machine runs.
The predastore migration lives in its own package rather than in migrate,
which every handler that stamps a KV bucket version imports; reaching the
cluster runtime from there would pull raft, badger and the QUIC client into all
of them. admin_upgrade.go blank-imports it, so only the binary that runs the
upgrade links it, and a test pins both registrations.
* test(predastore): cover the service runtime, gateway and shutdown paths
The service wrapper's new seams had no unit tests: buildRuntime, serve and
the errgroup Start runs the cluster and the S3 gateway under. CI's diff
coverage gate measured the PR at 34.6% against a 60% threshold as a result.
Cover every buildRuntime error wrap and its host selection, both ways the
gateway can fail serve, and the full lifecycle: Start brings the nodes and
the gateway up together, Shutdown drains both by cancelling the context
they share. Topologies are written per test as real config, with node ids
from a process-wide counter — the pipe transport's listener registry is
global, and this package's integration tests hold a fixture daemon on the
low ids for the life of the test binary.
setup.sh aborts twice on resolute. qemu-kvm is a virtual package with two
providers there, so apt refuses to resolve it; the token was redundant anyway
since $QEMU_PACKAGES already installs qemu-system-x86/arm and spinifex execs
the qemu-system-* binaries directly.
sudo-rs, Ubuntu's default sudo since 25.10, then rejects the sudoers file: it
implements no pam_session Defaults setting, and forbids a wildcard inside a
command argument. pam_session is now emitted only for classic sudo, and the
sysctl grant names a fixed-verb helper instead of a wildcard.
The helper is narrower than the rule it replaces: it accepts one interface, a
key restricted to rp_filter or accept_local, and a value restricted to 0 or 1,
where the old grant permitted any net.ipv4.conf key at any value. Granting
sysctl with an unconstrained argument was rejected as root-equivalent.
Adds scripts/lint-apt-packages.sh and an Apt Packages workflow so the runtime
package list is resolved against each supported suite's real indices in CI.
Spinifex rejected HttpTokens=optional and served 401 to every untokened
GET, which is stricter than AWS, whose own default is optional. That made
Windows guests impossible: cloudbase-init is the only realistic Windows
bootstrap agent and its EC2Service has no IMDSv2 token support in any
released version, so a Windows instance never received its hostname,
injected password or user-data.
HttpTokens is now settable at launch and via ModifyInstanceMetadataOptions,
accepting optional or required. The launch default stays required, so an
instance that does not ask for IMDSv1 never gets it. Every other
metadata-option field keeps its existing rejection.
The untokened path is reachable without credentials, so the per-instance
lookup is cached with a short TTL; without that a guest could drive
unbounded DescribeInstances fan-outs. A valid token short-circuits before
the lookup, so compliant IMDSv2 clients are unaffected.
* feat: spread HA control-plane placement across distinct AZs first
SchedulableHosts now orders eligible nodes by AZ round-robin (one host
per AZ before repeating any AZ) instead of raw fan-out arrival order.
ReserveSpreadNodes takes a strict order-preserving prefix of that list,
so an N-way HA reservation lands on N distinct AZs whenever that many
exist, degrading to today's node-only spread when AZ data is uniform
or absent. No change to the placement-group wire format.
* feat: honour ClientRequestToken idempotency for EKS CreateCluster
Mirrors the EC2 RunInstances ClientToken pattern with a TTL JetStream
KV bucket scoped per EKSServiceImpl instance. Same token and identical
params replays the in-progress/created cluster; same token with
different params returns IdempotentParameterMismatch; a distinct token
reusing the same cluster name still hits the existing atomic
name-claim and returns ResourceInUse.
* style: trim EKS spread comments to the 3-line max and drop a bead reference
* fix: namespace remaining viper keys shared across service subcommands
predastore, nats and awsgw registered "port", "host", "debug", "tls-cert",
"tls-key", "encryption-key-file" and "node-id" as bare, unqualified viper
keys. viper.BindPFlag overwrites the same key's registered pflag object on
each call, so only the last subcommand registered in init() kept its CLI
flag live; the others read a foreign flag's (usually unset) value.
Bare keys are also vulnerable to AutomaticEnv: viper checks the
SPINIFEX_<KEY> derived name before an explicit BindEnv target, so a
same-named env var set for one subcommand also feeds viper.Get calls in
every other subcommand sharing that key, or a generic SPINIFEX_DEBUG-style
var could shadow all of them at once.
Namespace each affected key per subcommand ("predastore-port",
"nats-debug", "awsgw-host", etc.) following the pattern already used for
predastore-host/base-path/config-path and spinifex-ui-*, so each key's
AutomaticEnv-derived name matches its own BindEnv target. CLI flag names
and environment variable names are unchanged. nats's own bind-host flag
uses "nats-service-host" rather than "nats-host", since that key is already
bound by rootCmd's own cluster-wide --nats-host override.
* fix: bind viperblock flags through the command that declares them
s3-host, s3-bucket, s3-region and plugin-path are declared on viperblockCmd
but were looked up on predastoreCmd, which declares none of them. The lookup
returned nil, BindPFlag rejected it silently, and the discarded error hid it.
Both the flags and their declared defaults were invisible to viper, so
s3-bucket resolved "" rather than "predastore".
* fix: bump goleak past the elided-frames stack-parse panic
goleak v1.3.0's stack parser treats a "...N frames elided..." line as
a function name and panics when it can't parse one. Go elides frames
on deep/recursive stacks, which is rare but load-dependent, so this
surfaced as an intermittent panic inside goleak itself (not a real
leak) during TestDrainVolume_DispatchedGoroutineDoesNotLeak under a
full parallel preflight run. Upstream fixed this in goleak@b62053b,
unreleased as a tagged version, so this pins to the latest commit.
* test: guard goleak against panicking on elided-frame stacks
* style: trim goleak regression test comments to the 3-line max
* feat: add segscan storage oracle for the real-predastore integration tier
Adds spinifex/testutil/segscanoracle, which copies a fixture node data dir
and execs the mulga umbrella repo's scripts/segscan (a separate Go module
that cannot be imported) to assert live/dead-tombstoned/dead-orphan byte
accounting against real predastore writes. Kept off the default PR path: a
new segscanoracle build tag excludes the test from make test-integration,
and a separate segscan-oracle.yml workflow (push to main/dev + manual
dispatch only, mirroring e2e.yml's non-gating trigger) checks out
scripts/segscan rather than adding that checkout cost to spinifex.yml.
* test: cover segscanoracle's pure logic to satisfy diff coverage
segscanoracle has no build tag and always compiles into the default
build, but nothing in that build exercised it, so every new line
counted as uncovered against the 60% diff-coverage gate. Adds unit
tests for JSON decoding (including the malformed-payload path), the
Locate resolution order, and the directory-copy routine, plus a
throwaway stand-in module (not the real scripts/segscan) to exercise
build+exec+decode end to end.
Pulls decodeReport out of Run and resolveSegscanDir out of Locate so
the pure logic is testable without a *testing.T or a real binary.
* feat: accept and surface gp3 Throughput on CreateVolume/DescribeVolumes
Honors input.Throughput (125-1000 MiB/s, default 125 when omitted) on
CreateVolume, persists it to VolumeMetadata, and surfaces it via
DescribeVolumes (getVolumeByID), mirroring the Iops handling shipped
in mulga-siv-380. Pins viperblock to the commit that adds the
VolumeMetadata.Throughput field.
Zero Throughput on a read volume means it predates this field; it is
omitted from the response rather than surfaced as a misleading 0
MiB/s, matching the existing IOPS convention.
* chore: pin viperblock to the merged gp3 Throughput commit
* fix: stop viper AutomaticEnv from shadowing predastore's own env vars
Viper resolves AutomaticEnv before explicit BindEnv, so predastore's
bare "config-path", "host" and "base-path" viper keys auto-derived to
SPINIFEX_CONFIG_PATH, SPINIFEX_HOST and SPINIFEX_BASE_PATH -
coincidentally the same names used elsewhere (the global cluster
config path, and root's global --host override). Setting
SPINIFEX_CONFIG_PATH for the cluster config (needed by
derivePredastoreBind) silently made predastore parse the cluster
config as its own S3 config, breaking auth for every signed request.
Rename predastore's host/base-path/config-path viper keys to
"predastore-"-prefixed keys whose AutomaticEnv-derived name is
identical to their explicit BindEnv target, so no shadow is possible
regardless of what else is set in the environment.
* style: trim predastore env-binding comments to the 3-line limit
* ci: skip building unused E2E suite binaries in e2e.yml's single/multi jobs
These two jobs immediately rebuild suite-aware binaries after Build
Install Artifacts and never consume the tar's _bin contents, so the
cross-compiled suite build was pure waste. build-install-artifacts.sh
gains an optional --skip-e2e-build flag (positional contract unchanged);
e2e-nightly.yml and the other callers keep building the full set.
* refactor: replace hand-rolled IMDS client with AWS SDK v2
internal/imdscreds, cmd/ecs-agent's metadata fetch and its credentials
provider used a hand-written IMDSv2 token/GET flow with manual RFC3339
expiry parsing. A malformed Expiration silently fell back to "never
expires" instead of failing.
Replace it with feature/ec2/imds, credentials/ec2rolecreds and
aws.CredentialsCache, following the internal/gwsign/signer.go
precedent. An unparseable Expiration is now a hard error from Fetch,
covered by TestFetch_BadExpirationErrors. Output shapes to callers are
unchanged, and IMDS base URLs ending in /latest keep working.
* refactor: derive predastore bind config from spinifex.toml in spx itself
predastore-start.sh awk-parsed spinifex.toml for the bind host/port and
node_id, exported them as env vars, then exec'd spx. Move that
derivation into predastoreStartCmd via derivePredastoreBind, which
reads the raw (non-DIAL-normalized) nodes.<node>.predastore section,
defaulting node_id to -1 (co-located) when the key is absent, same as
the script. An explicit --host/--port/--node-id flag or
SPINIFEX_PREDASTORE_* env var still overrides the derived value.
Delete the now-unused script and point the systemd unit's ExecStart
directly at spx service predastore start, matching the
spinifex-viperblock.service precedent.
* feat: close IMDS and vm_adapters trace-linking gaps
Wrap the IMDS mux with otelsetup.HTTPMiddleware (outermost, ahead of
rejectForwarded) so guest metadata requests parent their DescribeInstances/
Attribute NATS fan-out instead of rooting a trace per call, including
requests the SSRF guard rejects.
Route daemon/vm_adapters.go's ebs.mount/ebs.unmount NATS requests through a
new ebsRequestWithTrace helper that opens a client span and injects it into
the message headers, so viperblockd's consumer spans join the producer trace
instead of rooting their own.
* feat: trace the ebs.delete request from the instance cleaner
* fix: preserve running guests across target restart, harden state write
spinifex-shutdown.service's ExecStop fired on any spinifex.target stop,
including the stop half of `systemctl restart spinifex.target`, draining
every guest via QMP even though the host never went down. Gate the drain
behind `systemctl is-system-running` == "stopping", which only holds once
systemd is genuinely unwinding into shutdown.target (reboot/poweroff), so
a target restart or plain `systemctl stop` now leaves QEMU/nbdkit running
(KillMode=process) for the daemon to reconnect to on the way back up.
Daemon.WriteState returned as soon as the local state file write failed,
skipping the independent JetStream KV write entirely. A crash in that
window left a live VM with no durable record anywhere. Attempt the KV
write regardless of local disk health, then still surface the local
write error to callers, mirroring finalizeTerminated's handling of the
same fault class on the terminate path.
* fix: gate the shutdown drain via an spx flag, not a shell script
Replace the shutdown-drain-gate.sh gate with a --only-if-host-stopping
flag on `spx admin node drain`. The unit now points at /usr/local/bin/spx
directly (already required for the node to function) instead of a script
installed under the data directory, which a partial rebuild of that
directory could silently drop without ever failing ExecStop. The gate
logic also becomes a plain Go function with injectable state lookup, so
it gets real table-driven coverage instead of only a live reboot.
hostIsStopping() drains only on a literal "stopping" from `systemctl
is-system-running`. Every other recognized state (running/degraded/...)
and any unrecognized output skip the drain; a fully unreachable systemctl
fails toward draining, since a skipped drain on a real shutdown hard-kills
guests and loses unflushed writes, while a spurious drain on a restart
only costs an extra graceful stop/relaunch cycle. Run by hand without the
flag, the command still drains unconditionally as it always has.
* refactor: split the shutdown-gate comments to sit with the code they explain
* fix(vm): durably record terminate even when local state persistence fails
finalizeTerminated bailed out of WriteTerminatedInstance whenever the
transition to terminated hit a persistence-only failure (e.g. ENOSPC writing
local state), even though the in-memory status had already reached
terminated. That left the instance owned by nothing: StuckTerminateReaper
skips it (no longer shutting-down) and TerminatedTeardownReaper never sees
it (never reached the terminated KV bucket). Only an operator daemon restart
recovered it.
WriteTerminatedInstance is a JetStream KV write, independent of local disk
space, so keep going into it whenever the in-memory status did reach
terminated, distinguishing that case from a genuine invalid/raced
transition. Once the disk has space again, teardown now completes on its
own via the existing idempotent TerminatedTeardownReaper.
* fix(systemd): cap per-service memory under spinifex-system.slice
Every service under spinifex.slice ran with MemoryMax=infinity /
MemoryHigh=infinity, so one runaway service could exhaust the whole slice
budget and the slice-level OOM would then reap an arbitrary victim instead
of the offender.
Add per-unit MemoryHigh/MemoryMax as percentages of physical RAM (so the
same units scale from a dev box to bare metal). spinifex-daemon.service
still hosts every guest QEMU process in its own cgroup (no per-instance
slice yet), so it gets by far the largest allowance, just under the slice's
own 80%/90%, to avoid starving VM launches. predastore is the legitimately
memory-hungry service and gets generous headroom above its idle footprint.
The remaining infra services are small, single-purpose Go processes and get
a shared conservative bound.
* fix(viperblockd): remove local volume directory on ebs.delete
DeleteVolume deleted the S3 data and notified viperblockd, but the ebs.delete
handler only stopped the volume's goroutines, killed nbdkit, and removed the
NBD socket — it never removed the local WAL/checkpoint cache directory under
BaseDir. The only path that removed it (VB.Close's RemoveLocalFiles) only
ran from the unmount handler's seal step, which -efi volumes never reach
(they carry no durable data to seal) and which a main volume also skips
whenever a prior seal failed (e.g. disk full, deliberately keeping local
files for recovery). Since terminate always unmounts before deleting, and
delete is permanent, retention had no remaining purpose by the time
ebs.delete ran.
Unconditionally remove BaseDir/<volume> in the ebs.delete handler regardless
of mount-tracking state, closing the leak for both the root volume and its
-efi companion.
* fix(viperblockd): validate volume names before removing local state
ebs.delete unmarshalled the volume name straight off the wire and
handed it to os.RemoveAll: an empty name collapsed to BaseDir itself,
and ".." components could escape it entirely. localVolumeDir now
rejects an empty/invalid name or one that resolves outside BaseDir
before any removal, and ebs.delete returns Success: false with an
error instead of silently skipping. Applied the same gate to
ebs.mount and ebs.config, the other two handlers that turn a
wire-supplied volume name into a BaseDir-relative path.
* chore: pin viperblock to the merged concurrent-open corruption fix
The ansible build role builds with GOWORK=off, so the deployed binary links
whatever go.mod pins rather than the go.work overlay. Move the pin to the
revision carrying the per-volume flock and the WAL/state write fixes.
* fix(vpc): close the route-table and ENI teardown leaks
DeleteVpc validated subnets and non-default security groups but never
non-main route tables, so it deleted the VPC out from under them and left
records naming a VPC that no longer resolves. It now returns
DependencyViolation, alongside the existing checks and before any cascade,
and still cascades the main route table on its own.
DeleteSubnet left route table associations naming the deleted subnet.
DeleteRouteTable blocks on any non-main association carrying a subnet ID,
and that subnet could never be deleted again to clear it, so the table was
pinned forever. Associations are now cleared before the subnet record goes,
leaving a mid-way failure retryable. routetable imports vpc, so the record
cannot be typed here; only the associations key is rewritten, and every
other field is passed through untouched.
DeleteLoadBalancer detached an ENI and then re-read it to delete, where a
lagging R3 replica could report it still in use. The failure was logged at
WARN and the LB record deleted anyway, orphaning the ENI and its security
group. DetachAndDeleteENI now reads once and carries the detach revision
into the delete via LastRevision, retrying the whole flow from a fresh read
on a conflict, bounded at 3 attempts. The IPAM release happens after the
delete, not before, so a retry cannot hand the same IP back twice. When
ENIs survive, the LB record is kept naming them and the call returns
DependencyViolation rather than dropping the only reference to the leak.
Instance termination routes through the same call, and both ENI paths now
distinguish a real delete from an already-absent one instead of logging a
deletion that did not happen.
TestRLC5_NoOrphanedRouteTableAfterSubnetThenVpcTeardown covers the ordering
that defeated the per-operation tests: subnet first, then VPC.
* fix(elbv2): route SetSubnets ENI teardown through the atomic detach+delete
SetSubnets detached every ENI in one loop, then deleted the removed subnets'
ENIs with force=false in another. That is the same shape as the
DeleteLoadBalancer leak: the delete re-reads the ENI, a lagging replica can
still show it attached, and the delete fails as in-use while the LB moves on
without it.
Removed subnets now go through DetachAndDeleteENI, which carries the detach
revision into the delete under one read. ENIs that survive the reshape are
still detached on their own, since they are re-attached to the relaunched VM.
isProductionLayout() keyed off /etc/spinifex alone. A node reset legitimately
removes that directory, so spx silently fell back to the dev layout and rebuilt
the whole node — configs, keys, nats.conf — under the invoking user's home while
the systemd units kept reading /etc/spinifex. Cluster formation reported success
and every service then failed to start.
Detection now also accepts the installed systemd target, which setup.sh writes
and a reset never touches, so the layout survives the reset that caused this.
init and join no longer mkdir the production config dir either. Only setup.sh
can rebuild the per-service subtree and its ownership, so recreating the top
level would still leave a node that inits cleanly and dies at service start.
They now refuse with the remedy instead.
/var/log/spinifex is 0775 root:spinifex so each service user can write its own
log. logrotate refuses to rotate inside a group-writable directory unless the
config says which user to rotate as, so every entry was skipped and the logs
grew without bound.
Verified on env13: before, logrotate -d reported 'skipping
/var/log/spinifex/nats.log because parent directory has insecure permissions'
and nothing had ever rotated. After, the debug run is clean and forced
rotations produce nats.log.1 and a compressed nats.log.2.gz.
* fix(gateway): log the two SigV4 paths that reject silently
Three code paths answer SignatureDoesNotMatch, but only the Verify mismatch
logged anything. A skewed request timestamp and an unsupported service in the
credential scope both returned the same wire error with no log line at all, so
they were indistinguishable from a canonicalisation defect after the fact.
Both now log at Warn. The skew path records the client and server timestamps
plus the window, which is what separates clock drift from a signing bug.
Adds signingTime, since Parse rejects a skewed request before returning a
SignedRequest, leaving the raw header or presigned query arg as the only copy
of the timestamp available to log.
* fix(gateway): log the malformed-envelope rejection too
The parse-error default branch answers IncompleteSignature and was the last
auth rejection path with no log line, so a malformed Authorization header,
unsupported algorithm or missing content hash left no record of which
validation stage rejected the request. The parse error names it.
shutdownVolumes deliberately spares nbdkit that a guest is still writing
through, because tearing one out mid-write corrupts that guest's filesystem.
The unit left KillMode at the default, so the cgroup SIGTERMed those children
anyway and SIGKILLed them at TimeoutStopSec -- causing both a 30s stop stall
and the corruption the skip exists to prevent.
KillMode=process matches spinifex-daemon, where guest QEMU already survives a
restart for DDIL reattach. Idle nbdkit is still reaped explicitly.
Measured on env12 with a guest attached: control-group stopped in 30.08s with
'Failed with result timeout' and SIGKILLed nbdkit, leaving QEMU running with
zero ESTAB nbd sockets. KillMode=process stops sub-second, both nbdkit
survive, and the guest keeps its device.
* fix(ec2): drain the attached volume before CreateImage reads a live checkpoint
CreateImage against a running instance (--no-reboot) read the volume's
checkpoint directly from predastore without draining in-flight writes
first, unlike CreateSnapshot. A checkpoint captured before a guest-
triggered write (e.g. growpart rewriting the GPT after a root volume
is launched larger than the base image) could be stale, producing an
AMI that boots to a UEFI shell instead of the OS.
Extracts the existing snapshot-package drain logic into a shared,
reusable DrainVolume so both live-snapshot call sites route through
the same drain-before-read guarantee instead of being able to diverge.
* test(e2e): boot-verify CreateImage AMIs instead of trusting State=available
describe-images reporting State=available is not evidence an AMI
boots — a well-formed but unbootable AMI (stuck at the UEFI shell)
still registers cleanly. Extend the CreateImage e2e test to launch a
probe instance from each produced AMI (running-path and stopped-path)
and require a real SSH login, with console-dump-on-failure wired up
for diagnosis.
* fix(images): sparsify the Alpine guestfish .raw before conversion
The Alpine build path grows the qcow2 in place via qemu-img resize +
guestfish resize2fs, which can leave metadata scattered across the
newly-grown region. virt-sparsify zeroes the guest's unused blocks
before the raw conversion so they collapse into real holes, matching
the Ubuntu virt-resize path's sparseness.
DescribeInstances silently returned an empty Reservations list for an
instance ID that never existed, so callers could not distinguish "gone"
from "never existed". Mirror the two-pass found-set pattern from
DescribeSecurityGroups, but only assert absence when the NATS fan-out and
both stopped/terminated KV bucket queries are provably complete — a
node timing out during the fan-out must never produce a false NotFound
for an instance that actually exists on that node.
The assertion is split into a new DescribeInstancesChecked, used only by
the gateway's customer-facing action. The plain DescribeInstances stays
silent for internal callers (IMDS instance lookup, the RDS instance-state
resolver, the EKS control-plane reconciler) that treat "not found" as
"not currently present" rather than a fault.
The watchdog probed only healthz?js-enabled-only=true, which reports the
server-wide JetStream state. A per-message-block write error is recorded on
the block and returned on every later write, and nothing clears it short of
a restart -- the server-wide state is never touched, so healthz stays 200
while writes fail indefinitely after the disk recovers.
spx admin node js-probe round-trips a canary key through the cluster-state
bucket, which is the only operation that observes the latch. It probes that
bucket rather than one of its own because a dedicated bucket is a separate
stream that can be placed elsewhere and stay writable.
* fix: stop passing nbdkit S3 credentials as literal argv
access_key/secret_key were interpolated into the nbdkit plugin argv,
which is world-readable via /proc/<pid>/cmdline for the life of the
process. Credentials now travel via cmd.Env (VB_ACCESS_KEY/
VB_SECRET_KEY) instead. Requires a viperblock nbdkit plugin build that
reads these environment variables; inert against an older plugin
until that pin moves.
* chore: pin viperblock to the VB_ACCESS_KEY/VB_SECRET_KEY fallback
The spinifex half sets these env vars on the nbdkit child, but the
deploy builds with GOWORK=off, so viperblock's matching read of them
only ships once go.mod points at 8a4787a.
* fix(lbagent): stop heartbeating a load balancer once it is confirmed deleted
A LoadBalancerNotFound response is the gateway's positive confirmation that
the store has no record for the LB, not a connectivity failure — unlike a
dial error, timeout, or 5xx, which leave its existence unknown and must keep
retrying. Past the startup registration grace window (or after any prior
successful heartbeat), it can no longer be the record's own replication
race, so the agent now stops rather than retrying forever and filling the
log with heartbeat noise after teardown.
* fix(nats): enable the JetStream ENOSPC-latch watchdog timer
nats-server disables JetStream process-wide on ENOSPC (server/jetstream.go
handleOutOfSpace -> ShutdownJetStream) and never re-enables it itself —
that decision is nats-server's own internal state, not spinifex's, so a
restart is the only way back. The spinifex-nats-watchdog.timer + service
already probe /healthz?js-enabled-only=true and try-restart on a sustained
failure, but the timer's WantedBy=timers.target never took effect: neither
setup.sh nor firstboot enabled it, so the watchdog shipped inert and a full
disk needed a manual restart forever. Enable --now it alongside the rest of
the unit set.
* fix: bound EKS DELETING re-drive with backoff and a terminal give-up
reapCluster re-drove purgeClusterInfra on every GC tick with no attempt
tracking, so a permanently-failing teardown (e.g. an unretriable
DependencyViolation) was retried forever. Track attempt count and last
attempt time on ClusterMeta, back off exponentially between re-drives,
and give up after maxDeleteReapAttempts with a single error-level log.
Status stays DELETING on give-up so the cluster's infra remains tracked
and billable pending operator intervention, preserving the ADR-0006 §6
billable-before-sweep invariant.
* fix: create the EKS e2e node role independently in each subtest
runEBSCSIVolume string-built the worker node role ARN without ever
creating it, only passing because runIRSAPod's t.Cleanup discarded
DeleteRole's error while the role was still attached to its instance
profile, leaving it behind for EBSCSIVolume to reuse by naming
convention. Extract ensureNodeRole and call it independently from both
subtests, detaching the role from its instance profile before deleting
it and asserting that error instead of discarding it.
* fix: reclaim CSI-provisioned EBS volumes on EKS cluster delete
purgeClusterInfra tore down the control plane, nodegroups, and
networking but never looked at EBS volumes, so any volume the
in-cluster CSI driver had provisioned for a PVC was left behind
untracked and unbilled for. Add a report-only reclaim step that scans
for volumes tagged with the cluster's CSI owner tag and logs them for
operator follow-up.
Volumes are never auto-deleted: a PV's reclaim policy (Delete vs
Retain) is a Kubernetes-side property the CSI driver does not tag onto
the EC2 volume, so there is no safe way to tell from tags alone
whether deleting a given volume is intended. This mirrors the
volume-leak reaper's existing mark-and-alarm policy for the same
reason.
The cluster-owner tag requires the EBS CSI controller's
--k8s-tag-cluster-id flag, added here to the controller manifest;
without it a CSI volume carries no cluster-scoped tag and reclaim
would have to guess across every cluster sharing the account.
* fix: persist the last purge error on a wedged DELETING cluster
DeleteReapExhausted told an operator a cluster's teardown had given
up, but not why: nothing recorded the failing purge's error.
Persist it on every failed re-drive, not only once the reaper
exhausts its attempts, so it is visible on a still-retrying cluster
too. Status stays DELETING throughout, preserving the billing
invariant that untorn-down infra remains tracked.
* fix: make the EKS e2e suite confirm its CSI volume is actually deleted
The EBSCSIVolume subtest deleted its PVC on cleanup but never
confirmed the underlying volume was reclaimed, so a CSI driver that
silently failed to delete it would leak a volume with nobody
noticing. The suite provisions and owns this volume, so unlike
production teardown it can safely wait on its StorageClass's Delete
reclaim policy: capture the CSI volumeHandle while the PVC is bound,
then have cleanup poll EC2 for that exact volume to disappear before
the addon/nodegroup that run the CSI controller are torn down.
* test: gate scripts/images/ shell suites + shellcheck in CI
The eks-node/rds-postgres *_test.sh suites and shellcheck only ever ran
by hand: spinifex.yml filters pushes on Go-only paths and no job
invoked them, so a broken helper could land unnoticed. Add a
`make test-images` target and a path-filtered image-scripts.yml
workflow so both run automatically on scripts/images/** and images/**
changes, kept separate from preflight so image-asset churn doesn't run
on every Go contributor's commit. Also clears the shellcheck findings
that would have started this gate red (SC1090 on a runtime-resolved
source path, SC2154 x2 on eval-assigned variables shellcheck can't trace).
* test: pin vCPU quota gate ordering in RequestSpotInstances
Nothing asserted that RequestSpotInstances runs the vCPU quota gate
before dispatching capacity or launching a VM, so a regression that
dropped or reordered EnforceLaunch would pass silently. Add an
ordering test using a real quota.Service (embedded JetStream KV) and
real NATS responders standing in for the capacity and launch daemons,
recording a global call sequence and asserting the gate's index
precedes both. Verified by removing the gate: the first quota-KV read
then comes from the post-launch charge instead, landing after capacity
dispatch and launch, and the ordering assertion fails as expected.
* fix(systemd): give every unit an explicit TimeoutStopSec
9 of 12 spinifex systemd units had no TimeoutStopSec and inherited
systemd's 90s default, which is either far more or far less time than
their own SIGTERM handling needs. Size each one to what its shutdown
path actually does: predastore matches viperblock's 30s WAL-drain
budget since viperblock flushes through it (After= already sequences
viperblock's stop first; this is the backstop if that's ever bypassed),
spinifex-ui covers its 30s internal shutdown context plus margin for
long-lived streaming proxy connections, nats covers its own
JetStream-shutdown/raft-stepdown/client-drain signal handler, and
services with no SIGTERM handler at all (awsgw) get a short bound
since they exit immediately regardless.
Extends the existing unit-file invariant test with a check that every
.service file declares TimeoutStopSec, so a newly added unit that
forgets one fails CI instead of silently falling back to 90s.
* fix(vm): reconcile QEMU processes with no instance record on restore
classifyRestoredInstances only walks Manager.Snapshot(), so a
qemu-system process with no instance record at all -- a crash
mid-launch, a lost KV write, a stale process surviving a wiped state
file -- was never handed to killOrphanedQEMU and stayed permanently
invisible to recovery.
Add a host-process scan that diffs live qemu-system PIDs (matched via
/proc/<pid>/comm) against the PIDs claimed by known instances' own
-pidfile files. It only reports what it finds, never kills: this
daemon cannot distinguish "record not loaded yet" or "owned by another
daemon/tenant on this host" from a genuine orphan, and killing a
running customer VM to fix an invisibility gap is the wrong trade.
Runs at the end of Restore, once every known instance has been
classified and relaunched, so a live process still unaccounted for at
that point genuinely has no record anywhere.
* fix(vm): distinguish recordless QEMU orphans by pidfile ownership
A recordless qemu-system process with a stale spinifex pidfile is
positively this daemon's own orphan; one with no pidfile at all could
belong to another tenant on a shared host. Log the two cases distinctly
instead of one blanket warning, so an operator can tell them apart.
Also sandbox XDG_RUNTIME_DIR in the shared daemon test harness so
Restore()-driven tests never read this box's real runtime directory.
* fix(spinifexui): treat ErrServerClosed as a clean shutdown, not a failure
Serve returns http.ErrServerClosed after Shutdown is called deliberately
(the SIGTERM handler above does exactly that), but Start propagated it
as an error, making systemd log a graceful stop as exit code 1 and
Result: exit-code -- indistinguishable from a real crash.
Query-protocol services (EC2, IAM, STS...) get their action from the
SigV4 Action= form param before dispatch, so traceActionEnricher can
rename the span and set aws.action/aws.service from that context.
REST-JSON services (EKS, ECS, ACM, tagging, ECR, bedrock...) only
resolve their action inside their own dispatcher, well after that
middleware runs, so the enrichment never fired for them.
checkPolicyResource is the one call every dispatcher already routes
its resolved (service, action) pair through, so move the enrichment
there via a shared recordResolvedAction helper instead of duplicating
it per REST-JSON handler.
BytesFreed counted logical object sizes deleted from the S3 namespace,
but predastore only reclaims that disk space later via background
compaction. The CLI printed "freed X GiB" immediately on delete, which
made a compaction outage that never reclaimed the space invisible.
Rename to BytesDeleted and reword the CLI output to say the bytes are
marked for deletion and reclaimed by background compaction, not freed
immediately.
* fix(ec2): rename ambiguous 'code' slog key to firmwarePath
The EFI variable store prep log used 'code' for the OVMF firmware
path, which reads as an error/status code in log search. Rename to
firmwarePath to avoid confusion.
* fix(spinifexui): write pid file to a service-owned BaseDir
spinifexui.Start() called the legacy utils.WritePidFile, which has no
MkdirAll and falls back to $HOME/spinifex. Under the spinifex-ui unit's
ProtectHome=yes this directory is neither guaranteed to exist nor
writable. Add Config.BaseDir, switch to utils.WritePidFileTo (which
does MkdirAll) for Start/Stop/Status, wire it through the CLI as a
spinifex-ui-base-dir flag/SPINIFEX_UI_BASE_DIR env var, provision
/var/lib/spinifex/spinifex-ui in setup.sh, and add it to the unit's
ReadWritePaths.
* test(viperblockd): replace fixed sleep with a readiness poll
TestEBSConfigQueueGroup_DetachedOpenFails slept 500ms before assuming
launchService's ebs.config subscriber was bound, then raced a hardcoded
5s request deadline. Poll with a cheap malformed request (rejected at
json.Unmarshal, before the slow S3 path) until the subscriber answers,
then issue the real request with room for openVolumeVB's S3 backoff.
* fix(e2e): capture GPU worker console log on test failure
Wire harness.OnFailure into TestEKSGPUPodExposure to dump the GPU
worker's serial console via the established EC2 GetConsoleOutput path,
matching lb_test.go's captureLBConsoleOnFailure. A driver or
device-plugin hang after boot only ever surfaces on the guest console.
* fix(ec2): render empty Describe result containers and add requestId
xmlutil.BuildXML omits a nil slice's XML container entirely but renders
an empty one for a non-nil empty slice, so every EC2 Describe with zero
results (DescribeKeyPairs, DescribeVolumes, DescribeInstances, etc.)
came back as a bare "<FooResponse></FooResponse>" instead of AWS's
"<FooResponse><requestId/><fooSet/></FooResponse>". botocore parses that
body as having no result key at all, so the CLI silently prints nothing
and exits 0 — this broke the ansible key-import guard that gates
cluster bootstrap on describe-key-pairs' output.
Fix normalizes nil result slices to empty ones and synthesizes a
requestId in the single marshal chokepoint shared by every EC2 action
(marshalEC2Response, used by both ec2Handler and ec2HandlerWithReq),
rather than patching each handler individually.
* fix(ec2): return InvalidKeyPair.NotFound / InvalidSnapshot.NotFound for missing explicit IDs
DescribeKeyPairs and DescribeSnapshots silently returned an empty list
when the caller named a specific, nonexistent KeyName/KeyPairId or
SnapshotId, instead of erroring like real AWS (and like the existing
DescribeSecurityGroups/DescribeImages/DescribeVolumes/DescribeSubnets
handlers already do). botocore treats the empty-list response as
success, so ansible's cluster-post key-import gate — which checks
`describe-key-pairs --key-names spinifex-key` for a nonzero exit code —
never fires, and spinifex-key is never imported during bootstrap.
Mirrors the two-pass found-set pattern already used by
DescribeSecurityGroups: build the set of names/IDs actually present in
the filtered results, then error if any explicitly requested name/ID
is absent. An unfiltered list or a --filters query that matches
nothing is unaffected and still returns an empty list with no error.
DescribeInstances has the same gap but needs a design decision on how
a NotFound check interacts with the fan-out's existing partial-result
leniency, so it is left for separate follow-up.
TanStack router-plugin <1.168.21 sorted routes by coercing route nodes
to strings, so output order followed filesystem-walk order and flipped
depending on which generator version a contributor's node_modules
resolved. Pin the version so it cannot drift, commit the sorted output
from the fixed generator, and fail CI if a build produces a different
route tree than the one committed.
On a zero-worker or GPU-only cluster no node tolerates the DaemonSet,
so it has zero desired pods. The apiserver then has no tunnel to
kube-system pods (including metrics-server), v1beta1.metrics.k8s.io
never goes Available, and namespace GC hangs on the kubernetes
finalizer for every namespace, not just ones using metrics.
Add the same CriticalAddonsOnly + nvidia.com/gpu tolerations already
used by every other control-plane-critical addon in this repo
(aws-load-balancer-controller, ebs-csi-driver, argocd) so the agent
also schedules onto the control plane.
A tool whose child dies by signal can still exit 0 through go tool, so
govulncheck printed 'ok' after being OOM-killed and preflight passed
without a vulnerability scan having run. Route lint, govulncheck and
nilaway through scripts/run-gate.sh, which fails on a non-zero exit or
on signal-death output.
RegisterTaskDefinition let any caller with ecs:RegisterTaskDefinition name
an arbitrary taskRoleArn/executionRoleArn, and RunTask/StartTask/CreateService/
UpdateService could launch tasks under a previously registered definition's
roles, all without holding iam:PassRole on those roles. Mirror the EC2
gateway's PassRoleChecker closure pattern to close both paths.
ClaimLBName's crash-orphan reclaim treated any owner with no persisted
LB record as abandoned, but a legitimate in-flight create has no record
yet either during the gap between claiming the name and PutLoadBalancer.
Two concurrent creates for the same name could both win.
Track a claim timestamp and only allow reclaim once it exceeds a TTL,
so a fresh in-flight claim blocks a second claimer like a live LB would,
while a claim abandoned by a genuine crash is still reclaimed.
gateway.ErrorHandler unconditionally overwrote every error with
ErrorLookup's static per-code text, discarding both a caller's own
formatted message and any per-service distinction for wire codes
shared across services (e.g. ACM and EKS both use
ResourceInUseException). ResolveErrorDetail now also returns a
message when the producing call site used awserrors.Errorf, and
LookupErrorMessage resolves service-scoped wording before falling
back to the global default. DeleteVpc's DependencyViolation call
sites now name the blocking subnet or security group.
* fix: scope golangci-lint cache per worktree
golangci-lint's default cache under ~/.cache/golangci-lint is keyed by
module path rather than worktree path, so parallel worktrees of this
module collide and surface each other's lint issues, including from
worktrees that have since been deleted.
* fix: correct eks-node-gpu catalog description to match agent-only image
The GPU node image never installs eks-token-webhook or a server/agent
role selector, per scripts/images/eks-node-gpu/manifest.conf; it ships
ecr-credential-provider only and is agent-only because GPU nodes never
run the EKS control plane.
instanceCleanerAdapter.DeleteVolumes only called DetachVolumeOnTerminate for
a still-attached Boot volume, assuming shutdownAndUnmount's Unmount already
cleared a non-Boot volume's attachment. That assumption breaks whenever the
unmount seal fails (Unmount swallows the error) or never runs at all, as on
the stuck-terminate force-complete path. A leaked volume then stays in-use
with no recovery path once the instance record is deleted: detach-volume
--force fails InvalidInstanceID.NotFound and delete-volume fails
VolumeInUse. Call the same explicit, idempotent detach for every
non-DeleteOnTermination volume regardless of Boot.
DescribeImages hardcoded every AMI as "available" regardless of import
progress, so a failed backend import that never reached the flush was
indistinguishable from a complete one. Project the AMI's own State,
treating empty as available so pre-existing AMIs registered before the
field existed stay visible. CreateImageFromInstance, RegisterImage, and
CopyImage now set State explicitly since each completes synchronously
before the config write.
* fix(vm): report ENOSPC to the guest on data volumes instead of pausing the VM
QEMU's default werror=enospc pauses the whole VM when a data volume's
backend runs out of space. The boot drive already avoids this via
werror=report,rerror=report; data volumes had no on-error policy at
all in either the cold-boot -device or the hotplug device_add.
Set werror=report,rerror=report on the virtio-blk-pci device itself,
since a data volume's blockdev-add NBD node has no BlockBackend to
carry it the way -drive does for the boot drive.
* fix(vm): stamp ShuttingDownAt even when the state transition write fails
transitionWithPrecheck returned before stampShuttingDownAt whenever
TransitionState failed, even though the in-memory status had already
moved to the target. On a full disk this leaves ShuttingDownAt zero,
and StuckTerminateReaper.Sweep skips any VM whose ShuttingDownAt is
zero — so the backstop meant to bound a wedged terminate is disabled
by exactly the condition it exists to catch.
Stamp on the persistence-failure branch too, using the same helper
and locking as the success path, while still returning the original
error to the caller. The raced branch (TransitionState failed and the
in-memory status never reached target) is left unstamped, since the
target state was never actually entered there.
Start registers startCluster as a member of shutdownWg and then waits on
that group. startCluster's tail waited on the same group, so it could not
return until the counter it was waiting on reached zero, and the counter
could not reach zero until it returned.
The SIGTERM handler completed and logged "Shutdown complete" in ~20ms, but
the process never exited: it sat until systemd's TimeoutStopSec=15 and was
SIGKILLed, on every stop. Because the member stop chain is serialized,
that 15s also gated viperblock, predastore and nats, which each stop in
milliseconds once reached.
The duplicate setupShutdown in the same tail registered a second
signal.Notify and a second handler goroutine, which is where the
duplicated shutdown log lines and NATS unsubscribe pass came from.
Drop both. Start already installs the handler and owns the single wait.
The invariant test pins each helper to one call site, since a second call
site is the defect rather than a symptom of it.
Five manager tests waited on a fake DHCP counter and then read the KV
store. Fake.Acquire increments that counter before Manager.doAcquire
persists the entry, so the assertion ran in the gap between the two and
read the pre-acquire address whenever the write lost the race.
Idle machines always won the race, which is why this only ever failed
under load: the last three Build and Test failures on dev were this test
twice, on a diff that never touched the package.
Make the persisted KV state the barrier, matching the shape already used
by TestManagerStartKeepsNewLeaseWhenStaleReleaseFails, and keep the
counter as a plain assertion once the write is observed.
* fix(ec2): release post-launch ENIs on instance terminate
Terminate released only the ENI recorded at RunInstances time. Attachments
made after boot — ECS awsvpc task ENIs, or any direct AttachNetworkInterface
— are written to the spinifex-vpc-enis KV and never reach instance.ENIId, so
both terminate paths walked straight past them. The survivors held their
security group, blocking the SG, subnet and VPC deletes behind
DependencyViolation.
Both paths now sweep the KV for records naming the instance, independent of
the scalar. The sweep is additive, so an instance with no secondary ENIs
behaves exactly as before, and best-effort per ENI so one failure neither
aborts the rest nor blocks convergence on re-run.
DeleteOnTermination was advertised by DescribeNetworkInterfaces but backed by
nothing; ENIRecord now carries it, defaulting to true, and false detaches
without deleting.
* test(ec2): cover ENI release branches
Add table-driven and NATS-backed tests for the terminate-time ENI
sweep: DetachENI/ForceDeleteInstanceENI failure tolerance,
DeleteOnTermination skip, multi-ENI happy path, and
ListInstanceENIs error mapping on both the daemon and instance
service adapters.
viperblock#64 stops WriteAtCtx returning ErrNoSpace on predastore's nearfull
pressure signal, which rides on a successful write and means slow down rather
than stop. Without this, spx refused every write once the store fell below 15%
free, 10 points above the watermark at which predastore actually rejects.
Needed by spx itself: image import runs in-process against the module-cache
build, so the plugin-side fix alone left the control plane on the old
behaviour.
Six merged viperblock changes behind one repin: local files kept when the
WAL cannot be uploaded, stranded WAL generations retried after a failed
chunk upload, the flush-covers-acked-writes invariant pinned, the nbdkit
plugin's shutdown drain bounded at 20s, seal receipts written after a clean
seal, and FLUSH fsyncing the WAL before it reports success.
The seal receipts pair with the consumer already on dev, and the 20s drain
bound pairs with TimeoutStopSec=30 on spinifex-viperblock.service.
The unmount path decided whether to seal by stat-ing BaseDir/<volume>, but
the nbdkit plugin deletes exactly that directory as the last step of a
successful seal. So the fallback seal was unreachable whenever the plugin
exited cleanly, and a WARN claiming there was no local state fired on every
healthy unmount.
Branch on the receipt the plugin now leaves instead. Local state present
still means the plugin's seal failed or was cut short, so the fallback runs
and is finally reachable; a receipt means the plugin sealed and cleaned up;
neither keeps the original WARN, now meaning what it says.
The aux-volume check moves out of volumeNeedsSeal and to the front of the
chain: split out but left second, an -efi volume with local state would
newly reach a seal it has always been excluded from.
Stale receipts are cleared at mount rather than matched by PID at unmount,
so nothing depends on nbdkit's pid surviving to plugin load.
In-use nbdkit children are SIGTERMed with this cgroup and run the plugin's
shutdown drain. That drain is now bounded at 20s in the plugin itself, so
30s here lets the plugin's own bound fire first and keeps SIGKILL
exceptional. At the 90s default a slow or unreachable backend stalled the
whole target-stop transaction.
Picks up viperblock #57, which routes the JSON slog handler to stderr so
the nbdkit plugin's logs reach journald — nbdkit repoints a plugin's fd 1
at /dev/null, so the whole volume seal path was discarded. Also picks up
#56, removing a per-call allocation from the per-block cache instrument.
No spinifex code depends on either change: spx does not import
viperblock/telemetry. The bump is needed because the build role compares
the viperblock source behind spx (module cache, GOWORK=off) against the
one behind the nbdkit plugin (the submodule tree) and fails on a mismatch.
lb-agent configured no logger at all, so it fell back to slog's default
text handler. That folds attributes into the message, leaving the cause of
a failure trapped in a string: 'Heartbeat failed err="gateway returned
400"' arrived greppable but never queryable.
No OTLP export: the agent's only egress is the gateway it is watching, so
the host-side serial console remains the path that survives a blackout.
Init bolts the OTLP bridge onto whatever slog default is installed at the
time. SetupRoutes then called SetDefaultJSONLogger purely to set a level,
and that bare slog.SetDefault replaced the fanout with a stdout-only
handler, so the gateway went dark to the sink for the rest of the process.
One nightly cell shipped 9 app-log docs, all within 30ms of startup, while
its journal held 574 request lines.
Backs the stdout handler with a LevelVar and adds SetLevel, so verbosity
changes no longer touch the handler chain.
* feat(gateway): log the canonical request and action on signature mismatch
A SignatureDoesNotMatch told us nothing beyond err="signature mismatch":
the canonical request the server reconstructed was never logged, and the
APM trace is a bare "POST /" because auth fails before the dispatcher
parses the action. An intermittent mismatch was therefore undiagnosable.
Logs the server-side canonical request, with the session token masked, plus
the recovered action. Warn rather than Debug because the failures worth
catching never reproduce on demand; the path is already rate-limited.
* chore(deps): bump predastore for SignedRequest.CanonicalRequest
Picks up the exported canonical request the mismatch logging needs.
Render summed FailCount across suites and printed "No failures across any
suite" whenever the total was zero, without asking whether any test had run.
A cell that dies during provisioning emits junit with no testcases at all, so
it took the clean path. Nightly run 30467638631 cell 20 failed with "install
callback never fired after 3 attempts; PXE never triggered install" and its
analysis.md read "No failures across any suite" — absence of evidence rendered
as positive evidence.
Zero testcases now renders a warning naming the layer above the suites. A run
with tests and no failures is unchanged.
Cells 2, 3, 9 and 10 ran on spinifex-image-builder-ubuntu-24.04.qcow2, built
2026-04-20 — five weeks before openvswitch-ipsec and strongswan-charon were
added to the image builder. Those cells therefore had no OVN IPsec support and
ran intra-AZ Geneve in plaintext. Every other workflow is already on 26.04.
The host_image_override guards now match ubuntu-* rather than a pinned version.
An exact match is what let this drift go unnoticed: bumping the matrix release
silently stops overriding the image and drops the cell onto the debian gold.
Cells 2, 3, 9 and 10 ran on spinifex-image-builder-ubuntu-24.04.qcow2, built
2026-04-20 — five weeks before openvswitch-ipsec and strongswan-charon were
added to the image builder. Those cells therefore had no OVN IPsec support and
ran intra-AZ Geneve in plaintext. Every other workflow is already on 26.04.
The host_image_override guards now match ubuntu-* rather than a pinned version.
An exact match is what let this drift go unnoticed: bumping the matrix release
silently stops overriding the image and drops the cell onto the debian gold.
* fix(e2e): fail TestMultinodeIPSec when a cluster that requested IPsec has none
An empty ipsec_encapsulation was treated as "bootstrapped without IPsec" and
skipped. On ubuntu-24.04 openvswitch-ipsec.service is inactive at daemon start,
so the enable guard correctly refuses to flip the flag and the cluster runs
plaintext intra-AZ Geneve — but the test then skipped and the cell reported
green, masking the gap on 2 of 3 multinode cells.
Cross-check network.ipsec_enabled from the node config: an explicit false still
skips, anything else fails with the plaintext condition named.
* fix(e2e): read only the ipsec_enabled value, not the whole config line
Detail() rendered "ipsec_enabled=ipsec_enabled = true" because the raw grep
line was logged as the value. Take the first token after "=" so a trailing
comment cannot be read as the value either, and compare exactly rather than
by substring.
Every daemon read-modify-wrote the same zone object in s3://northstar with no
mutual exclusion. A NATS queue group balances load, it does not serialise
distinct messages, so concurrent daemons lost each other's records and could
leave the object unparseable as TOML. Once unparseable it stayed that way: both
the writer and the drift reconciler must parse the zone before they can repair
it, so DNS changes stopped applying until the object was edited by hand.
The exposure did not need user activity. The drift backstop ran on every node,
unelected, on a 60s ticker, so an idle three-node cluster raced its own zone
once per tick. Five publisher families feed the same object, so ordinary
concurrent API calls collided too.
- Add a per-zone JetStream KV lock, following the CAS patterns already used by
the vpcd reconcile leader and the vCPU quota store. One key per zone, so
unrelated zones never block. The wait budget stays under requestTimeout so a
producer's ack is not lost to contention, and a write aborts rather than land
outside an expired lease.
- Rebuild a zone whose stored bytes do not parse instead of failing every write
forever. The reconciler reports a corrupt zone as absent, which suppresses
pruning while still publishing the full desired set. Detection matches the
decoder's error type so it works against the pinned northstar release.
- Elect one node per drift cycle, removing the idle-cluster self-race.
- Correct three comments claiming the queue group serialised zone writes; they
are why the hazard went unnoticed.
- Run the LB e2e internet-facing subtests in parallel. Their serialisation
workaround exempted internal LBs on the false premise that those use a
different DNS path; ELBChanges sets the same base-domain zone for both.
* fix: group-scope the OVS/OVN sockets and stop sudoing read-only probes
The OVS DB socket and both sets of ctl sockets were chmod 0666 by setup-ovn.sh,
with a systemd override re-applying it on every openvswitch-switch restart. Write
access to db.sock is full control of the local datapath, so this handed the
datapath to any account on the box. Both service users run with Group=spinifex,
so the access is now 0660 root:spinifex, /var/run/ovn is 0750, and a shared
helper re-applies it from ExecStartPost on openvswitch-switch and ovn-controller
alike — the latter is new, and matters because the ovn-controller ctl socket name
embeds the pid and is recreated on every start.
That also fixes the health probe reporting ovn-controller as not_running on a
healthy node. It shelled out to `sudo ovn-appctl`, which only spinifex-vpcd is
granted; as spinifex-daemon sudo demanded a password and the swallowed error read
as OVN being down. With the ctl socket group-owned the probe needs no privilege,
so health.go moves to a non-escalating Runner rather than the alternative of
granting the daemon a root-equivalent ovn-appctl rule to read a status.
ListIMDSTaps ran `list-ports` plus one `get` per patch port on vpcd's 15s tick,
so 1+N sudo invocations every tick, each one logging a pam_limits warning about
the core rlimit the unit deliberately pins to 0. It now issues a single JSON
query per pass, and the sudoers drop-in disables the PAM session for both service
users, which also leaves sudo children inheriting the unit's rlimits.
An absent iface-id on a patch port is now skipped with a warning instead of
aborting the pass: it is the only record of the full ENI, so such a port cannot
join the live set either way, and failing the pass stalled reconcile for every
healthy port behind it. A transient query failure still aborts.
* fix: group-own the OVS/OVN pidfiles too, and cover the ipsec ctl socket
Group-owning the ctl sockets was not enough for `ovn-appctl -t ovn-controller`:
it resolves the pid-named socket through the pidfile, and OVS opens that
O_RDWR to test the fcntl liveness lock, so a group-readable pidfile still fails
the probe with EACCES. The pidfiles now get the same 0660 root:spinifex, which
also tightens them from the 0644 they ship with.
openvswitch-ipsec joins the override list because setup-ovn.sh restarts it
after the sweep runs, leaving its ctl socket as the one file still at the
shipped mode.
Verified on the dev node: every socket and pidfile is 0660 root:spinifex and
survives a restart of all three units, an unprivileged user is refused, and
/health reports ovn-controller ok where the sudo probe read not_running.
* refactor: stop escalating the OVS/OVN socket clients
Now that setup-ovn.sh group-owns the control sockets, ovs-vsctl, ovs-appctl,
ovn-nbctl, ovn-sbctl, ovn-appctl and systemctl reach their daemons as the
service user. utils.NeedsPrivilege carries that policy and both command
builders consult it, so ip, iptables, dhcpcd, sysctl and arping still escalate
while the socket clients no longer do. Their sudoers grants are removed: each
took unrestricted args and all of them accept --log-file=PATH, so every one was
a root-equivalent grant handed out to run a query.
ovs-ofctl keeps its grant. It talks to a per-bridge <bridge>.mgmt socket that
ovs-vswitchd creates when the bridge appears, including bridges spinifex
creates at runtime, so the provisioning sweep cannot group-own it up front.
The NB and SB database sockets join the sweep for ovn-nbctl and ovn-sbctl.
vpcd carried its own sudoCommand copy that escalated unconditionally, bypassing
the policy entirely. It survived only because the grants existed and broke the
OVN flows-ready barrier the moment they were removed, so it now delegates to
utils.SudoCommand. TestRG10_SudoOnlyThroughThePolicy fails the build if another
package reintroduces a local copy.
Verified on the dev node: an instance launches, IMDS is served, the flows-ready
barrier completes, and no sudo denial appears anywhere in the journal.
* refactor: drop every spinifex-vpcd sudoers grant for ambient capabilities
vpcd held NOPASSWD grants for ip, iptables, sysctl and arping. The ip grant
alone was root-equivalent: sudoers cannot constrain its arguments, so
`sudo ip netns exec <ns> /bin/sh` was a root shell for the service account.
The unit already grants CAP_NET_ADMIN and CAP_NET_RAW ambiently, and the kernel
passes an ambient set to each child on exec, so those tools work as the service
user with no escalation at all. Verified on a live node: all sixteen ip
subcommands vpcd issues, plus iptables reads and writes, sysctl -w on the
net.ipv4.neigh tree, and arping, all succeed under exactly the unit's ambient
set. spinifex-vpcd now has no sudoers rules.
NeedsPrivilege had to become capability-aware to do this. spx is one binary
shared by both service units, and the daemon holds CAP_SYS_ADMIN and
CAP_DAC_OVERRIDE but no CAP_NET_ADMIN, so a static tool list cannot express
"vpcd runs ip directly, the daemon escalates it". The decision now reads CapAmb
from /proc/self/status once and matches it against the capability each
invocation needs. sysctl is resolved per key, since CAP_NET_ADMIN covers the
net.* trees and nothing else.
The dhcpcd grants go too: neither daemon invokes it. Only setup-ovn.sh, which
runs as the provisioning user, and the in-guest ecs-agent.
TestRG10_VPCDHoldsNoSudoersGrant pins both halves together, so a future grant
cannot be re-added without also explaining why the capability stopped working.
* chore: restore routeTree.gen.ts swept in by mistake
* fix: probe ovn-controller with ovn-appctl, not ovs-appctl
checkOVNController's first probe passed a bare -t ovn-controller to ovs-appctl,
which resolves it against OVS_RUNDIR — /var/run/openvswitch. ovn-controller's
socket and pidfile are in /var/run/ovn, so the probe could never succeed and
logged "ovn-controller.pid: open: No such file or directory" on every check.
Not a functional break: the next probe globs the full /var/run/ovn path and
succeeds, so the function still returned nil. It cost an exec per check and
made a healthy node look like it was failing an OVN liveness probe.
ovn-appctl defaults OVS_RUNDIR to /var/run/ovn, so the common case now matches
first. The glob and systemctl fallbacks stay for older layouts.
* fix(dhcp): release stranded leases and label them with the owning node
Two paths dropped a lease from KV without returning it to the upstream
server, so a source="dhcp" pool leaked addresses permanently.
Manager.Start deleted an expired entry outright. Dropping the entry alone
abandons the binding rather than returning it, and a server that ages
leases by liveness holds the address for as long as anything answers ARP
for it. Adoption now sends a bounded, best-effort RELEASE first, skipping
leases with no stored offer/ack because nclient4 cannot rebuild those.
Release-by-IP answered SUCCESS when the IP was absent from KV, telling
the caller the address was freed when nothing went on the wire. It now
returns an error. Every ReleaseIP call site logs and continues, so this
surfaces the strand without wedging teardown.
Leases were also unattributable upstream: chaddr is a derived HashMAC and
option 61 carries that same MAC, so no field identified the host that took
the lease. Option 12 is now "<node>-<client-id>" and option 60 is
"mulga-spinifex/<node>", giving the lease table a per-lease identity and a
per-host column to group on. HashMAC is untouched — encoding the node
there would cut per-node entropy and reverse a documented decision.
Observed live: 20 office-LAN addresses held by one node's orphaned OVN
gateway LRPs, with no lease records and no renewals.
* fix: rebind the datapath when a DHCP lease moves to a new address
A lease re-issued on a different IP was persisted to KV with no notification
of any kind. doAcquire replaced the lease after an expiry and doRenew accepted
whatever a rebind answered with, while the gateway LRP's address is written
once at attach and never revisited. Each such event burned one address
permanently: the old one squatted by OVN's ARP responder with no renewer and
no DHCPRELEASE, the new one renewed forever attached to nothing.
Route both paths through applyLease, which releases the superseded address and
invokes an IPChangeHook so the datapath can follow. vpcd registers a hook that
moves the gateway LRP's networks and gateway_ip stamp, re-runs the routed leg
so the VPC SNAT and host ingress route follow, and waits on the flows barrier.
EIP, ENI-public and NATGW addresses are recorded outside vpcd in
DescribeAddresses and DescribeInstances, so the hook reports those rather than
repointing a datapath the control plane no longer agrees with. The release half
still applies to every purpose, so the leak itself is closed either way.
A failed rebind does not fail the lease loop: the new address is held
regardless, and dropping it would strand both.
* fix: reconcile API-visible records when their DHCP lease moves
An EIP or auto-assigned public IP whose lease was re-issued elsewhere left
DescribeAddresses and DescribeNetworkInterfaces advertising an address vpcd had
already released. The datapath kept the old dnat_and_snat too, so the address
was dead on both sides while the upstream server was free to hand it to
someone else.
vpcd cannot fix this alone: it holds the lease, but the records naming those
addresses live daemon-side. Add vpc.dhcp.lease-changed, which vpcd requests and
the daemon serves, dispatching to the service owning the moved address.
RebindPublicIP and RebindENIPublicIP commit the new external IP through
vpc.add-nat first — AddEIP scrubs the predecessor row keyed on the logical IP,
so one commit is enough — and only then move the record, so a NAT failure never
leaves a record advertising an address that does not route.
Both refuse to move a record naming neither address, so a stale request cannot
overwrite an allocation the lease does not own, and both are idempotent under
duplicate delivery.
A failed or unanswered reconcile propagates back to the lease loop rather than
reading as success. vpc.dhcp.lease-expired, listed in the service manifest but
never implemented, is replaced by the subject that now exists.
* fix: dedupe retried AllocateAddress so one call takes one address
EC2 AllocateAddress carries no ClientToken, and vpcd's DORA ladder can
outlast the AWS client read timeout, so every SDK-level retry minted a
fresh allocation ID and drew another upstream DHCP lease. The caller kept
one EIP and the rest stayed leased with no record naming them, so nothing
ever released them.
Retries are now identified by the SDK's amz-sdk-invocation-id, which is
stable across attempts of one call. The gateway lifts it off the request,
NATSRequest forwards it as a header, and the daemon puts it back on the
context, so the EIP service sees the same token every attempt. singleflight
joins a retry to the call still in flight — the common case — and a TTL'd
KV record answers retries that arrive after the first call completed.
* fix: reclaim DHCP leases whose owner is gone and stop dropping expired ones
Three leaks that all end the same way: an address stays leased upstream with
nothing renewing it on purpose and nothing that will ever release it.
Manager.Start deleted any lease that expired while vpcd was down, after
releasing it. The resource it was taken for still existed, so the datapath kept
forwarding on an address the upstream server now considers free — a
duplicate-IP conflict waiting for the server to reissue it — and nothing ever
re-acquired. Expired leases now get a renewal loop like any other, which
re-DORAs on its first pass; the existing IP-change hook rebinds the datapath
when the new address differs. The reaffirm RENEW is skipped for these, since a
lease the server has already aged out will only fail it.
Nothing reconciled leases against the resources that justify them, so a
resource deleted out from under its lease left the address renewed forever.
ReapOrphans sweeps them: vpcd asks the daemon over vpc.dhcp.owner-check
whether the EIP, ENI or VPC still exists, and releases the lease when it does
not. A gateway lease also has a router port to remove, so DetachIGW runs
before the address goes back — releasing it while OVN still answers ARP for it
invites the same conflict. Anything the daemon cannot resolve is reported
unknown and kept: an unreachable daemon must never read as a fleet-wide
deletion.
The acquire ladder ran to ~62s, just past the 60s read timeout botocore
defaults to, which is what turned one AllocateAddress into two leases. It now
totals 40s with a 45s budget, and the DORA RPC 50s, so the caller hears the
outcome instead of giving up and retrying. This is a mitigation, not a fix, for
callers that send no retry token: botocore emits no amz-sdk-invocation-id, so
the dedupe added earlier cannot see their retries at all.
Fixes mulga-j19gj, mulga-9exe6, mulga-7qyzj.
* feat: add spx get dhcp-leases so held external addresses are visible
The upstream leases behind a source="dhcp" external pool lived in a per-AZ KV
bucket with no read path, so working out which addresses were held, by what,
and whether the owner still existed meant writing a NATS client or reading
journals. That was the main obstacle to diagnosing every leak in this series.
The command lists each lease with its purpose, VPC, address, MAC, holding node
and remaining time, and resolves the owner over vpc.dhcp.owner-check so an
orphan is visible as such. --orphans narrows the output to leaks;
--skip-owner-check drops the lookup when only the addresses matter.
Fixes mulga-655iu.
* fix(vpcd): reconcile gateway ports onto their lease address at startup
A gateway LRP configured from a static pool keeps its address when the pool is
switched to source="dhcp", so the VPC forwards on an address nothing leases
while the lease it holds goes unused. No lease event ever fires to correct it.
vpcd now sweeps every gw-lrp lease on startup and rebinds the port to the
address the lease actually holds. RebindGatewayIP is idempotent, so a port
already in agreement costs one OVSDB read.
* test: cover dhcp lease owner-check handlers and the lease CLI
The owner-check path decides whether the reaper releases an address, so every
inconclusive branch is asserted to report unknown rather than gone.
* fix(dhcp): make the outcome of a stale lease release observable
A stale RELEASE was logged only when the transport failed. A lease with no raw
offer/ack was skipped in silence, and a RELEASE the server ignored looked
identical to one it honoured, so an address left bound upstream could not be
traced to either cause.
The skip now warns, and a successful release logs the chaddr it used, which is
what a server matches a RELEASE against.
* fix(dhcp): take the RELEASE chaddr from the lease being released
A server matches a RELEASE to a binding on chaddr, and the release path
re-resolved it live. For a UseIfaceMAC lease whose interface MAC had changed
since, that sends an address the binding never had: the server drops the RELEASE
and says nothing, so the address stays held with no renewer.
The chaddr now comes from the lease record, which carries what took the lease
out, falling back to live resolution only when the record has none.
* fix(eks): GPU node AMI declares uefi boot mode
The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but
the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under
legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at
firmware with zero serial console output.
* fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs
Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to
initialize under the closed/proprietary kernel module: dmesg shows
RmInitAdapter failing with "requires use of the NVIDIA open kernel
modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to
nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia
recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no
separate open variant (userspace tools only), so it is unchanged.
* fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons
A cluster with a GPU-only nodegroup has no untainted node: the CP
carries CriticalAddonsOnly (EKS parity keeps user workloads off it)
and every worker carries nvidia.com/gpu. Without tolerations for both,
ebs-csi-controller and aws-load-balancer-controller have nowhere to
schedule and stay Pending forever. Real EKS never needs this since its
control plane isn't a schedulable node, so the stock upstream
manifests carry no such toleration.
* fix(images): give the libguestfs appliance a working resolver
The guest's /etc/resolv.conf symlinks to systemd-resolved's stub
(127.0.0.53), which isn't running inside the appliance chroot, so DNS
is dead until package installs need it. Point it at a real resolver
before package installs, then restore the stub symlink afterward so
the shipped AMI still gets its DNS from systemd-resolved at boot like
a stock cloud image.
* fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind
A GPU's IOMMU group can include an upstream PCIe bridge when the
platform lacks ACS isolation between them. vfio-pci refuses to bind
non-endpoint bridge devices, so Claim/ReclaimByAddress must filter
bridge-class (0x06) members out of the group before touching driver
bindings, rather than failing the whole claim or leaving the bridge
rebound incorrectly.
* fix(ecs): GPU node AMI declares uefi boot mode
The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the
Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS
no OVMF/pflash firmware is attached, so the guest hangs at firmware with
zero serial console output.
Same defect and same fix as the eks-node-gpu entry: both images are built
by the same script from the same stock Ubuntu cloudimg, so both were
EFI-only and only one was corrected.
* fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch
ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary
kernel module. NVIDIA ships no proprietary module for Blackwell (GB202)
and later, so ECS GPU container-instances could not bind those cards on
any driver branch. It is the third verbatim copy of the routine and was
missed when the other two were fixed.
Also pins the driver branch across all three copies. The branch was
resolved at build time via apt-cache search | sort -rn | head -1, which
sorts on the number in the package name — but that number is a branch
alias, not a driver version (nvidia-dkms-535-server installs 580.126.20),
so the ordering key need not match what installs. Resolving also let
identical source produce different drivers on different days.
The pin lives in setup.sh rather than manifest.conf because manifest
variables are not passed into the guest: the setup script is uploaded and
executed by virt-customize --run, with no environment carried across.
* fix(eks): tolerate CP and GPU taints in argocd addon
All six argocd Deployments and its StatefulSet carried no tolerations. A
cluster with a GPU-only nodegroup has no untainted node to place them on:
the control plane carries CriticalAddonsOnly and every worker carries
nvidia.com/gpu, so the pods stay Pending forever.
Same defect the ebs-csi-controller and load-balancer-controller addons
already had. Real EKS never hits it because its control plane is not a
schedulable node, so the stock upstream manifests carry no such
toleration and the divergence is ours to maintain.
* refactor(images): extract eks-node initd logic into init-agnostic helpers
The OpenRC runscripts carried their start_pre logic inline, so porting a
service to systemd meant re-deriving that logic under a distro change
rather than just rewriting the init wiring. Move it into /bin/sh helpers
that either init system can drive, keeping eks-node on Alpine/OpenRC.
k3s-prestart takes the restore-block guard, config.yaml derivation,
konnectivity-agent and CoreDNS staging, and the token-webhook wait.
konnectivity-server-prestart takes the CA wait and cert mint, plus the
server-count derivation that used to run at parse time. Both keep every
guard, bound and message as-is.
Values flow back the way the GPU image already does it: the helper writes
shell assignments to a /run file and the init system sources them, so
command_args is built in start_pre where it used to be a top-level
assignment.
k3s-agent-prestart moves to eks-node and becomes the single copy. The GPU
manifest claimed its helpers were reused verbatim from eks-node while
hand-copying this one; it now points at the shared file, so the two
cannot drift.
k3s-agent.initd still sources agent.env and exports itself rather than
delegating: OpenRC has no EnvironmentFile= equivalent, and the daemon
inherits only what the runscript exports, so a helper subprocess sourcing
it would not reach k3s.
The helpers run under set -eu, which the runscripts did not. The awk over
first-boot.env is guarded on the konnectivity manifest, not on the env
file, so a missing env file exits awk nonzero; without || true that turned
a warn-and-continue into a refusal to start k3s. Covered by a test that
fails on rc=2 if the guard is dropped.
* feat(images): build system images in a pinned mkosi container
Move the image-build toolchain into a container so a runner needs only
Docker. The toolchain cannot drift between runners, and the same build
runs identically on a developer's machine.
mkosi builds in a user namespace, which two of Docker's default
confinements block: the seccomp profile masks CLONE_NEWUSER, and the
docker-default AppArmor profile separately blocks unsharing the mount
namespace. Both are relaxed; neither is a privilege grant, as
capabilities stay dropped and no devices are exposed. mkosi also runs as
an ordinary user rather than root, since as root it takes a path that
needs real CAP_SYS_ADMIN.
Verified end to end: an Ubuntu 26.04 Format=disk image builds from the
Debian container and the artefact is a valid GPT disk, with the package
cache on a named volume cutting a forced rebuild from ~2min to ~34s.
* feat(images): publish system images from the build container
Convert mkosi's raw output to a compressed qcow2, write the .sha256
sidecar the spx catalog verifier matches, and upload both to R2 using the
same pinned toolchain as the build, so publishing needs nothing on the
host either. The toolchain image build moves to a shared helper rather
than being duplicated across the two entry points.
Publishing needs no relaxed confinement, unlike the build: qemu-img and
aws create no namespaces, so it keeps Docker's default seccomp and
AppArmor profiles.
The object path carries no version and the catalog resolves newest-wins,
so an upload overwrites the live image with no way back. Copy the current
object to a timestamped archive key first, making a bad publish a restore
rather than an unrecoverable loss.
* fix(images): build the image on the output's mount, not the cache volume
mkosi assembles the image in its workspace and moves the result to the
output directory. The workspace lived under the cache volume, which is a
different mount from the output bind mount even when both sit on one
filesystem, so the move failed EXDEV and mkosi fell back to copying —
requiring the finished image to exist twice at once.
That is invisible on a small image and fatal on a real one: a ~4G GPU
image needed ~8G to land and died with ENOSPC after a successful build.
Point the workspace at the output's own mount so the move is a rename
again and the image only exists once. The package cache stays on the
volume, which is the part worth persisting across runs.
* feat(images): add mkosi base config and gpu-nvidia profile
Establish images/ with a base mkosi.conf plus a gpu-nvidia profile,
replacing the throwaway smoke-test config. The base builds a bootable
UEFI disk; the profile adds the NVIDIA stack.
Most of ubuntu-gpu-nvidia-setup.sh does not carry over. Its kernel-version
hunt, the do_bootloader=no kernel-img.conf hack and the manual
update-initramfs all exist only because that pipeline chroots into a
prebuilt cloud image that already has a kernel. mkosi installs the kernel
and owns the bootloader, so the version is known rather than detected, and
the postinst asserts exactly one kernel instead of searching for it. The
static configuration (nouveau blacklist, unattended-upgrades blacklist,
cloud-init docker group) becomes files under mkosi.extra/ rather than
heredocs.
linux-image-generic drags in every linux-firmware split package, none of
it relevant to a headless VM guest; left at the default the modules initrd
embeds all of it into the UKI, which grew to ~880MB and overflowed the
512M ESP. The initrd module set is trimmed to mkosi-initrd's curated
default, which is a boot fix and not only a size one.
Verified in the built artefact, not just the build log: nvidia.ko.zst and
the four sibling modules under /usr/lib/modules/<kver>/updates/dkms, the
nouveau blacklist, and the nvidia runtime in docker's daemon.json.
* fix(images): unblock guest boot on wait-online and firstboot
A guest built from these images stalled twice on the way to a login prompt,
both times on a unit waiting for input that never arrives in a VM.
systemd-networkd-wait-online holds network-online.target until every managed
link is up, and docker.service pulls that target in with Wants=, so any image
carrying docker inherits the wait whether or not it wants it. A guest whose
links are slow or absent ran the unit to its full 120s default with
multi-user.target queued behind it. Override it to --any with a short timeout:
one link up is what a single-NIC guest actually needs.
systemd-firstboot separately prompts on the console for anything the image
does not already carry, before multi-user.target. With no timezone baked in,
boot stopped dead at "Please enter the new timezone name" and waited for a
keystroke. Setting locale, keymap and timezone at build time leaves firstboot
with nothing to ask.
Both are baked into the base rather than delivered as first-boot drop-ins:
an image built from scratch has no cloud-init to write them.
Measured on wattle with an A1000 passed through: login at 42.8s, down from
~210s, and the firstboot prompt is gone. nvidia-smi in the guest reports
"NVIDIA RTX A1000, 595.71.05, 8188 MiB".
Also widen the images gitignore to output-*/ so a scratch build cannot leave
multi-GB artefacts stageable.
* feat(images): make the base image reachable — cloud-init, sshd, console
An image built from scratch launched unreachable. The images this replaces
start from a stock Ubuntu cloud image and inherit cloud-init, sshd, sudo, a
network renderer, dbus and a serial console from it; mkosi installs only what
is listed, so none of it was there and nothing said so.
Guests get their SSH key and user-data from IMDS via cloud-init, so without
the package a launched instance has no authorized_keys at all — no error, just
no way in. It also made any cloud-init-delivered config inert, including the
gpu profile's docker-group drop-in.
Each addition is load-bearing on its own:
cloud-init consumes IMDS: keys, user-data, hostname, network
openssh-server the key needs a sshd to be installed into
netplan.io cloud-init's renderer on Ubuntu; with no renderer the NIC is
never configured, IMDS is never reached, and cloud-init
falls back to DataSourceNone
dbus systemd only Recommends it and mkosi installs no Recommends,
so systemctl could not reach the system bus — and the EKS and
ECS bootstraps drive guests with systemctl from user-data
lsb-release cloud-init shells out to it; without it every boot reports
"degraded" forever and the status stops being a usable signal
The kernel also needs console=ttyS0 baked in. The launcher gives every guest a
serial port and get-console-output reads it, but a kernel with no console=
writes to none of it: the console stopped after the UEFI bootloader and every
boot failure was invisible. ttyS0 goes last because Linux makes the final
console= the controlling /dev/console.
manage_etc_hosts is set because cloud-init renames the guest per launch but at
its default never writes a matching hosts entry, so sudo warned on every call.
datasource_list is deliberately NOT pinned: the guest identifies the platform
from the SMBIOS system UUID the launcher spoofs, ds-identify narrows to Ec2 on
its own, and pinning per image is the customisation this migration deletes.
Verified on wattle with an A1000 passed through: ssh in as ubuntu with
spinifex-key, cloud-id "aws", cloud-init status done (not degraded), no failed
units, nvidia-smi reports the A1000 on 595.71.05, docker active, boot 26.7s.
* feat(images): let the guest grow its root into the volume it was given
mkosi sizes the disk to its content, so the image ships with a root
partition that fills the disk to its last sector and ~1.4G free. Nothing
in it can expand that: cloud-init's growpart module is enabled, but the
growpart binary lives in cloud-guest-utils, which is not installed and is
not pulled in as a Recommends by anything in the package set. systemd-repart
is not an alternative — Ubuntu's systemd package carries no repart binary,
unit or definitions at all.
The result was a node handed a larger volume that it could only ever use
the first few GiB of, with the rest allocated and unreachable. That is
fatal for a worker whose job is to pull container images.
gdisk is deliberately left out despite being a cloud-init Recommends:
growpart resizes GPT through sfdisk, which arrives via the hard fdisk
dependency, and e2fsprogs brings the resize2fs that the resizefs module
runs afterwards.
Verified on wattle: the gpu-nvidia image launched with a 20G root reports
20G on df -h / (was 4.4G), via
"'/' resized: changed (/dev/vda2) from 4770582528 to 20936900096", with
nvidia-smi still reporting the A1000 and no failed units.
* feat(images): move eks-node-gpu and ecs-agent-gpu onto mkosi profiles
Adds the eks-common, eks-agent and ecs profiles, so both GPU node images
build from the composed base rather than a hand-authored tree, and the three
verbatim copies of install_nvidia_gpu_stack() collapse into gpu-nvidia alone.
Splits Docker out of gpu-nvidia into its own profile. Docker belongs to the
standalone GPU AMI, whose purpose is `docker run --gpus all`; the EKS and ECS
node images run k3s or containerd and reach the GPU through CDI. Fusing the
two was fatal for ECS rather than merely wasteful: gpu-nvidia installed
containerd.io from Docker's repository, which Conflicts/Replaces the Ubuntu
containerd the ecs profile needs, so gpu-nvidia + ecs could not resolve at
all. nvidia-container-toolkit stays in gpu-nvidia — every GPU image needs
nvidia-ctk to generate its CDI spec.
Closes three gaps the pilot left, each of which only shows up on a booted
guest: /etc/modules-load.d/nvidia.conf, without which nothing loads the
modules on a headless node and CDI generation finds no GPU;
mulga-cdi-generate.service, which no profile shipped; and an apt-mark hold on
the toolkit packages, which the unattended-upgrades blacklist does not cover
for a plain `apt upgrade`.
mkosi-build.sh gains --image, naming an output and expanding it to an ordered
profile list. The order is load-bearing: mkosi runs each profile's postinst in
the order the profiles are passed, so docker's nvidia-ctk wiring must follow
gpu-nvidia. Leaving that to callers meant an argument-order mistake produced a
healthy-looking image with no GPU wiring, so docker's postinst also fails
loudly when gpu-nvidia is selected but nvidia-ctk is absent.
Profiles reference build inputs where they already live via ExtraTrees=
instead of copying them, keeping one copy of the helpers shared with the
Alpine image. Go binaries are built on the host into a staging tree, since the
builder container carries no Go on purpose: go.mod is the single pin for the
toolchain and a second pin in the Dockerfile could disagree with it silently.
* fix(images): install systemd-resolved so the guest has a DNS resolver
mkosi installs only the packages named in Packages= plus their hard dependency
closure, and systemd-resolved is merely a Recommends of systemd. So it was never
installed: the image shipped a 0-byte /etc/resolv.conf that nothing populated,
because netplan renders to systemd-networkd and networkd alone does not write
resolv.conf. glibc then fell back to localhost and every lookup failed with
"[::1]:53 connection refused", which reads as a network fault rather than a
missing package.
A node that cannot resolve cannot pull a container image. The ECS GPU node
booted, registered as a container instance, then failed every task at image
pull. The EKS image carries the same defect but is masked by the cloud-init
bootcmd in nodegroup_userdata.go that writes resolv.conf at launch; the ECS
launch path has no equivalent, so only ECS exposed it.
* fix(images): name build output by image and never report a skipped build as success
Two defects in the build wrapper that let the wrong bytes reach a boot test.
Every composition wrote the same spinifex.raw, taken from the base config's
ImageId, so an artefact carried no record of which profiles produced it and an
EKS image was distinguishable from an ECS one only by which build ran last.
Passing --image-id names the output after the image, and as a side effect the
name lands in the guest's /usr/lib/os-release. Publishing correspondingly loses
its default input: --name is the published object and --image the local
artefact, and with nothing tying the two together a defaulted input meant
whatever raw happened to be in the output directory could be uploaded under the
name the caller asked for.
mkosi also skips the build whenever the output path merely exists. It does not
compare inputs, so it never notices an edited mkosi.conf or a restaged binary,
prints "Use --force to rebuild" and exits 0 — a stale image indistinguishable
from a fresh one. Builds are now always forced, and the run additionally asserts
the expected raw was written during this invocation rather than trusting the
exit status. The cache worth keeping is the package download volume, which
--force does not touch.
* feat: add gpu-amd mkosi profile for the ROCm AMI
Ports scripts/images/ubuntu-gpu-amd-setup.sh so ubuntu-gpu-amd builds under
mkosi as 'gpu-amd docker'. Without it the legacy scripts/images/ tree has to
stay alive for one remaining image.
Pins the ROCm repo by apt priority rather than only adding it: Ubuntu 26.04
universe ships its own vendored rocminfo, which unpinned apt preferred over
AMD's, mixing release trains within one image.
Uses a systemd unit for render/video group membership instead of a cloud-init
runcmd, because cloud-init's config merge takes a duplicated runcmd key
wholesale from one fragment and silently discards the other, which would have
dropped the docker profile's own usermod.
Runtime GPU verification is deferred — no AMD hardware is available.
* feat(images): add eks-server, ec2-node and CPU eks-node/ecs-node mkosi images
Ports the K3s server side of EKS off the legacy Alpine/OpenRC builder: nine
systemd units, a templated etcd-snapshot service with daily and 15-minute
timers, the addon/coredns/konnectivity assets, and the six server binaries.
konnectivity-server is not an in-repo package, so it gets a pinned host-side
clone-and-build rather than an entry in profile_binaries().
Splits ec2-node out of eks-common. mulga-ebs-byid mints the by-id links for any
attached volume and mulga-mgmt-net configures any multi-NIC system VM, so
neither belongs behind an EKS profile that a database image would have to
compose to reach them.
Start-at-boot moves into preset files. mkosi runs preset-all after every
postinst, so a postinst enable is discarded before the image is finished, and
with no policy file the fallthrough is enable — which built an image that ran a
control plane and a worker at once.
Installs the cloud-init datasource pin as 99- rather than 90-. cloud-init-base
ships its own 90_dpkg.cfg declaring datasource_list, read_conf_d sorts the
directory in reverse and the merge keeps the first source, so a 90- prefixed
file lost the key silently. The postinst now asserts the merged result.
* refactor(eks-node): make the role selector init-agnostic
eks-node-role and k3s-first-boot drove OpenRC directly, so on the systemd image
the selector enabled nothing and the node never started k3s. Both now dispatch
through svc_enable/svc_start/svc_disable on a detected init, with EKS_NODE_INIT
overriding detection so either branch is testable on a host that has only one.
Detection prefers rc-update over systemctl: the test suite stubs rc-update on
PATH, and preferring systemctl would have silently rerouted those cases to a
path they were not written for.
Enable and start stay two separate passes. The server path starts
mulga-eks-k3s-recovery before k3s so a recovery directive applies before the
apiserver comes up, and collapsing to systemctl enable --now would interleave
them.
The dispatch is duplicated in both scripts rather than shared: they are
single-file deliverables with no library-path convention between them.
Tests cover both branches across all three roles, including that server-join
still gets no k3s-first-boot, and that rc-update wins when both inits are
present.
* fix(eks-node): enqueue role-selected units instead of blocking on them
eks-node-role.service declares Before=k3s.service, and
mulga-eks-k3s-recovery.service declares After=eks-node-role.service, so a
blocking systemctl start deadlocks: systemd will not run the queued job while
the selector is still activating, and the selector cannot finish until that job
runs. Type=oneshot defaults TimeoutStartSec=infinity, so nothing breaks the tie
and the node wedges mid-boot with the role written but no control plane. On a
booted guest the selector sat in activating for 261s until the VM was torn down.
Start with --no-block on systemd. The ordering the blocking call was implicitly
providing is already declared between the units themselves, so systemd
sequences them correctly on its own. OpenRC runs the script directly rather
than scheduling it, so that branch keeps blocking and keeps surfacing failures
through set -e.
* fix: escape $ in konnectivity-server ExecStart so server-count survives
systemd expands $VAR/${VAR} in Exec*= from the unit's own environment before
the command runs, substituting an empty string for a name it does not know.
KONN_SERVER_COUNT is not a unit variable — konnectivity-server-prestart writes
it to /run/konnectivity-server-args.env, which the wrapper shell sources at
runtime — so systemd rewrote the flag to `--server-count=` before /bin/sh saw
it. konnectivity-server then died on flag parsing, which is why the unit logged
Started and produced nothing despite --logtostderr=true and journal+console,
and why the apiserver's cluster egress selector found no UDS for the life of
the cluster.
This is a port artefact. konnectivity-server.initd is flag-for-flag identical
and shares the same prestart script, but under OpenRC the shell was the only
thing reading ${KONN_SERVER_COUNT}, so the same text was correct.
Escape it as $$ to defer expansion to the wrapper shell, and add
lint_unit_expansions() to reject any unescaped $VAR/${VAR} in an Exec*= line of
a profile's units. The linter joins line continuations first, since the
offending reference sat mid-way through a wrapped multi-line ExecStart, and
runs on the host before the container starts so a bad unit fails the build in
seconds rather than at first boot. $(cmd) is deliberately not flagged: systemd
leaves it literal.
Verified on env12 with a rebuilt image: TestEKS/IRSAPod and
TestEKS/EBSCSIVolume, which both previously failed on `kubectl exec ... timed
out` through the egress tunnel, now pass, with zero UDS connect errors across
the run.
* fix: refresh the stale qcow2 mkosi-build leaves beside a fresh raw
mkosi only emits the raw. A qcow2 next to it came from a previous run's hand
conversion, and nothing about its name says which build it belongs to, so it
survives a rebuild and silently ships the old image to a node while the raw
beside it carries the new one. Observed on a rebuild that left the raw at 15:34
carrying a fix and the qcow2 at 12:45 carrying the pre-fix unit.
This is the same failure the existing freshness assertion guards against, one
file over, so extend that block rather than add a second check. Refresh rather
than delete: the qcow2 is what gets copied to a node and imported, so removing
it would push the manual conversion, and this trap, back onto the caller.
Convert via a temp name and rename so an interrupted run leaves the old file
rather than a half-written one that looks current.
* fix: resolve the awsvpc task DHCP client instead of hardcoding udhcpc
ecs-agent leases a task ENI's OVN-assigned address inside a per-task netns,
which systemd-networkd does not manage, so it shells out to a DHCP client. That
call was hardcoded to busybox udhcpc, which Alpine ships and Ubuntu does not.
`ip netns exec` exits 1 rather than 127 when it cannot exec the binary, so the
failure surfaced as an opaque "network setup failed" with the task stuck in
STOPPED, and it took every awsvpc-mode workload with it: TaskAwsvpc and
ServiceWithELB both failed on the mkosi image, the latter as runningCount=0
because its target group targets task ENI IPs.
Resolve the client at setup time, preferring udhcpc so Alpine behaviour is
unchanged and falling back to dhcpcd. Presence of the udhcpc binary is not
sufficient to select it: Debian and Ubuntu's busybox ship the applet without
/usr/share/udhcpc/default.script, and udhcpc then takes the lease and exits
nonzero having configured nothing. Same order and same reasoning as
dhcp_oneshot() in mulga-mgmt-net.sh. Both invocations stay one-shot, leaving no
daemon behind in the netns.
Install dhcpcd-base in base, because neither client was present in any profile.
That also repairs mulga-mgmt-net.sh, whose own udhcpc/dhcpcd fallback resolved
to nothing on Ubuntu and would have failed the first time a cross-host
control-plane data NIC needed it — the S5 path the step-6 plan notes has never
been exercised.
The DHCP-client choice is injected so the tests assert a named platform's
branch rather than whatever happens to be installed on the machine running
them; without that the same test exercises different code on an Alpine CI box
and an Ubuntu dev box.
* refactor: retire the ec2-node profile, fold its helpers into the base image
ec2-node carried mulga-ebs-byid and mulga-mgmt-net, and every EKS composition
had to remember to list it — eks-common even guarded on its presence and
aborted the build without it. ECS did not list it, so the ECS image shipped
with no by-id symlinks for hot-attached volumes at all.
Neither helper is answerable at build time: any image may later be attached a
volume, and any image may later be launched as a control plane. Both are inert
when unused (mulga-ebs-byid is udev-driven; mulga-mgmt-net exits 0 when its
fw_cfg blob is absent), so the base image is where they belong. That deletes
the composition rule, the guard, and the ECS gap in one move.
Also drops ec2-node from the two eks image compositions in mkosi-build.sh.
Verified on freshly built spinifex-ecs-node and spinifex-eks-node raws: both
carry the helpers and the udev rule, mulga-ebs-byid is enabled via
multi-user.target.wants and mulga-mgmt-net via cloud-init.target.wants.
* fix(eks): force UseMTU=false on netplan units so DHCP renewal can't reinflate the pinned link
mulga-vpc-mtu only won the boot race by ordering: netplan renders
UseMTU=true (it does not inherit networkd's own false default), so a
DHCP lease renewal at T1 could silently re-apply the advertised MTU to
the link. usemtu_off() forces UseMTU=false via a runtime drop-in in
the systemd-documented [DHCPv4] section for every rendered unit and
reloads networkd, removing the dependency on boot ordering entirely.
assert_pinned() then verifies the link and its routes actually match
before exiting, so any regression is a loud boot-time failure instead
of a silent one. Both are no-ops on the Alpine/dhcpcd image, which has
no systemd-networkd.
* fix(images): state UseMTU=false under both the current section and the legacy alias
netplan renders the DHCP settings under [DHCP] while systemd documents
[DHCPv4]. A drop-in naming only one parses cleanly and does nothing if the
running systemd disagrees, which is the silent no-op the drop-in exists to
prevent.
* fix(ecs): install iptables on both ecs-node images, cover task-role credential fetch
credendpoint.go DNATs the ECS task credential endpoint via iptables; neither
the mkosi ecs profile nor the legacy Alpine manifest installed it, so the
endpoint silently failed to start and any task with a task role could not
fetch credentials while the instance still looked healthy.
Also fixes a latent bug in the e2e task-role trust policy (Service-principal-only,
which this platform's HTTPS AssumeRole path can never match) and adds a
TaskRoleCredentials subtest that fetches real credentials from inside a task
container, so a dead endpoint fails the suite instead of passing unnoticed.
* fix(images): ship preset files for base, eks-common and ecs start-at-boot
mkosi runs systemctl preset-all after every profile's postinst, discarding
whatever systemctl enable a postinst performed. eks-server and eks-agent
already carry preset files for this reason; base, eks-common and ecs still
relied on the default-enable fallthrough, which happens to match today but
is one upstream package change from reversing.
Adds 10-mulga-base.preset (mulga-ebs-byid, mulga-mgmt-net),
30-mulga-eks-common.preset (mulga-eks-provider-id, mulga-vpc-mtu) and
40-mulga-ecs.preset (containerd, ecs-ca-install, ecs-agent; also disables
cni-dhcp.service/socket, which containernetworking-plugins enables by
default with nothing to serve under the baked host-local IPAM conflist).
Removes the now-redundant systemctl enable calls from each postinst,
matching the eks-server/eks-agent precedent.
* fix(sts): attribute ecs-tasks.amazonaws.com to container-instance callers
The HTTPS AssumeRole path passed an empty principalSource, so matchServicePrincipal
denied every Service clause other than the IMDS-only EC2 one. A role carrying the
trust policy AWS documents for an ECS task role was unassumable by construction.
Service attribution is now derived from the authenticated SigV4 caller: an
assumed-role session of the account's ecsInstanceRole — the only credential the
in-guest agent holds — is attributed ecs-tasks.amazonaws.com, and only for roles
in its own account. Nothing in the request body contributes, so a task holding
its own task-role session cannot claim the service principal.
* fix(images): remove baked SSH host keys, pin Seed= for reproducibility
openssh-server's postinst bakes an identical host key pair into every
build, making every guest from one build mutually MITM-able; sshd-keygen
already regenerates keys on first boot (ConditionFirstBoot=yes), so the
baked pair is pure non-determinism with no purpose. Also pin Seed= so
systemd-repart stops assigning fresh partition UUIDs on every build.
Verified on a built spinifex-ecs-node image: no ssh_host_* files ship,
cloud-init still lists ssh in cloud_init_modules, and sshd-keygen.service
is wired via ssh.service.wants/ (redundant regeneration path). A two-build
comparison dropped content-differing files from 9 to 1 (only the UKI,
whose zstd internals remain sensitive to unrelated input changes).
---------
Co-authored-by: tomnewton-mulga <tom.newton@mulgadc.com>
* fix: fan out certificate re-import to referencing load balancers
Re-importing new material under an existing ACM ARN previously updated
only the stored CertRecord; no load balancer referencing that ARN ever
re-rendered its config, so HAProxy kept serving the old leaf until it
expired while the API reported success.
Add an InUseBy reverse index on CertRecord, maintained via snapshot-diff
around every listener create/modify/delete and cert attach/detach in
handlers/elbv2. On re-import, ACMServiceImpl invokes a fan-out hook
(wired in daemon.go to avoid an import cycle) that re-runs
updateStoredConfig for every load balancer in the ARN's InUseBy set,
mirroring the existing target-group fan-out. lbagent is unchanged: it
already polls ConfigHash and reloads on change.
* fix: make InUseBy index updates CAS-safe under concurrent writers
AddInUseBy/RemoveInUseBy previously did a plain get/mutate/put, which
loses concurrent writers silently: two listeners on different load
balancers attaching the same certificate at once could drop one
entry with no error. Because InUseBy is the sole mechanism that fans
a renewed certificate out to the data plane, a lost entry means that
load balancer never re-renders and its certificate silently expires
in HAProxy while ACM reports it as healthy.
Switch both to a JetStream KV revision-based compare-and-swap loop
with jittered backoff, retrying on a concurrent writer instead of
overwriting it. Replace the old concurrency test, which only
asserted no errors/panics and so could not detect a lost index
entry, with one that asserts exact final state across N concurrent
writers.
* fix: encrypt ACM certificate private keys at rest
CertRecord.PrivateKey was persisted to the spinifex-acm JetStream KV
bucket as plaintext PEM. Encrypt it with AES-256-GCM via the existing
handlers_iam.EncryptSecret/DecryptSecret helpers, keyed by master.key.
Store.PutCert encrypts when a master key is set; GetCert/ListCerts
decrypt, falling back to a narrowly-bounded legacy-plaintext read (only
when the stored value is unambiguously a PEM private key block) so
pre-existing records keep working and are re-encrypted on next write.
A Store never given a key (ELBv2's independent read-only acmStore)
behaves exactly as before — encryption is opt-in per Store instance via
the new SetMasterKey method, so no other package needs to change.
NewACMServiceImplWithNATS now requires a master key and fails
construction without one; daemon.go loads master.key inside the
initServiceWithRetry closure so a missing key fails daemon startup
after the retry window instead of leaving the service permanently
degraded.
* fix: wire ELBv2's ACM store with the deployment master key
ELBv2 constructed its own handlers_acm.Store without a master key, so
once ACM started encrypting CertRecord.PrivateKey at rest, ELBv2 could
no longer resolve listener certificates: it read back ciphertext where
it expected PEM. Require NewStore(ctx, nc, masterKey) to always take a
key so an unkeyed Store can no longer be constructed, thread the
deployment master key through NewELBv2ServiceImplWithNATS and the
daemon's ELBv2 init closure (mirroring the ACM closure), and make ACM
store construction failure fatal instead of silently degrading every
HTTPS listener.
* feat(acm): tenant Private CA with x509 name constraints
Adds an independent tenant root CA (separate from the platform CA) that
signs leaf certificates only within its baked-in permitted domains.
Name constraints (PermittedDNSDomains, excluded IP ranges, excluded
email/URI) are frozen at creation and drift is rejected rather than
silently regenerated, so a device's trust store is never invalidated
without an explicit operator action.
* docs(acm): record OpenSSL verification of the empty email/URI exclusions
Confirms the zero-length ExcludedEmailAddresses/ExcludedURIDomains
entries parse cleanly and enforce as expected under OpenSSL 3.5.6, not
just Go's own verifier, since this CA is meant for device trust stores.
* feat: implement RequestCertificate with PRIVATE_CA managed issuance
Adds the ACM RequestCertificate API surface: mode derivation
(PROVIDER_API/MANUAL_TXT/PRIVATE_CA), synchronous PRIVATE_CA issuance via an
injected CertAuthority, and correctly-shaped PENDING_VALIDATION records for
the deferred modes. Extends CertRecord with issuance/validation/renewal
state, wires the gateway and daemon NATS dispatch, and updates COMMANDS.md.
* feat: wire tenant CA into the running spinifex daemon
Add LoadTenantCA as a load-only entry point (never creates), refactor
LoadOrCreateTenantCA to share the load path. Add 'spx admin cert
create-tenant-ca' to create the tenant CA with explicit permitted
domains, idempotent unless --regenerate is passed. Wire tenant CA
loading into daemon startup: missing tenant CA logs a warning and
degrades PRIVATE_CA issuance only, it does not block startup like the
master key does. Fold tenant CA paths into admin's CA path helpers,
keeping filenames distinct from the platform CA (tenant-ca.pem/.key
vs ca.pem/.key) so the two roots can never be confused.
Also give ACM's DeleteCertificate in-use refusal its own awserrors
alias instead of reusing the EKS-labelled one directly, and correct
COMMANDS.md to reflect that the tagging endpoints are implemented.
* fix: derive ACM validation mode from the requested domain, not a global flag
Mode derivation tested cfg.Northstar.DefaultDomain != "", which says only that
northstar is configured, never whether it is authoritative for the domain being
requested. Any deployment with a default domain set therefore returned
MANUAL_TXT for every request, including domains hosted nowhere, with a TXT
record the caller could not publish.
Resolve the requested domain against northstar's actual hosted zones, and put
MANUAL_TXT last. A name-constrained tenant CA is a definite local capability the
operator declared explicitly, whereas MANUAL_TXT assumes the domain is publicly
delegated so a public CA can resolve the challenge - which is never true of an
internal northstar zone. A tenant CA covering every requested domain now wins,
and a configured DNS provider still takes precedence over the manual path so
operators with a real domain are not downgraded to a private certificate.
* feat: renew PRIVATE_CA certificates automatically before they expire
Managed issuance signed a leaf once and never renewed it. With 90-day leaves
every certificate silently expired a quarter after the first apply, which is
the treadmill managed issuance exists to remove.
Renewal is scoped to PRIVATE_CA: re-signing from the tenant root is fully
offline, so this needs no ACME client and no network access. The window is
proportional - two thirds of the certificate's own lifetime, floored at a
minimum - so it self-tunes as issuance lifetimes change rather than degenerating
into 'renew immediately, forever' the way a fixed window would. A deterministic
per-ARN jitter keeps certificates issued in one apply from all renewing in the
same minute.
Reissue keeps the certificate ARN and generates a fresh leaf key, then pushes
through the ELBv2 fan-out. The push is mandatory rather than incidental:
GetLBConfig returns stored configuration and never re-resolves certificate
material, so a renewal that did not push would leave HAProxy serving the old
leaf until something unrelated changed the config hash.
A per-certificate lease keyed on the ARN means one node drives a given renewal
without serialising all renewals behind a single leader. Certificates whose
domains have fallen outside the tenant CA's name constraints record a failure
reason instead of retrying in a tight loop.
The renewal scan reads metadata only and never decrypts stored private keys;
the lease is a raw compare-and-swap that leaves the encrypted key untouched.
* fix: carry handler error messages over the daemon NATS transport
handleNATSRequest responded with the sanitized error code alone, so any
handler error without a registered code collapsed to a bare ServerInternal
and its reason was visible only in the daemon log. utils.ServeNATSRequestCtx
has always preserved the message; the two implementations had drifted.
Adds respondWithServiceError and routes the 16 service-error sites through
it, plus transport-level tests pinning that the message survives for an
uncoded error, that a wrapped coded error keeps its real code, and that a
bare code carries no message.
* feat: implement ACM GetCertificate and stop decrypting keys on read paths
GetCertificate was unimplemented end to end: it now returns the PEM
certificate body and chain for an owned ARN, ResourceNotFound for an unknown
or cross-account ARN, and RequestInProgressException for a certificate that
has a record but no body yet, wired through the service interface, the NATS
client, the daemon subscription and the gateway action map.
Store gains GetCertMetadata alongside GetCert, and ListCerts becomes
ListCertMetadata. Every read that answers a question about a certificate
rather than serving it - ownership checks, Describe, Delete, tag reads,
GetCertificate and listing - now leaves PrivateKey empty instead of running an
AES-GCM open per record to produce plaintext key material no caller reads.
* fix: reconcile the ACM master-key wiring with dev's new tests
The InUseBy and cert fan-out tests that landed on dev construct the ACM and
ELBv2 services directly, and both constructors now take the deployment master
key. Pass one shared key in the fan-out test - two services over the same
bucket disagreeing on the key is the exact failure the wiring exists to stop.
A Store is never unkeyed now, so the two test helpers collapse into one.
* fix: rebuild the ACM InUseBy index at daemon startup
The index that drives certificate fan-out is written only by listener
mutations, so a listener whose certificates have not been touched since it
was created has no entry. A certificate with an empty index then neither
fans out on renewal nor trips DeleteCertificate's in-use guard, and both
failures are silent: the renewal reports success while the load balancer
keeps serving the old leaf, and a certificate a live listener depends on
can be deleted.
Reconcile the index from the persisted listener records once at startup.
Walking every certificate rather than only the referenced ones makes it a
rebuild rather than a backfill, so it also drops entries naming a load
balancer that no longer exists. Writing only differences keeps it
idempotent, and the index writes are already CAS-guarded, so the nodes of
a cluster reconciling the same bucket converge.
* fix: deliver listener SNI certificates to HAProxy
Add/RemoveListenerCertificates persisted the listener record and then
returned, so the config was never re-rendered and lbagent had nothing to
poll for. Re-render through reloadListenerLB, the same helper the rule
mutations use.
That alone changed nothing, because the renderer dropped SNI certificates:
resolveFrontendCert returned only the default certificate and the bind line
emitted a single crt, with the PEM path keyed on the listener so a second
certificate had nowhere to live. Render every attached certificate instead,
default first so it stays the no-SNI fallback, each with its own path.
HAProxy accepts crt more than once on a bind line and selects by SNI.
The default certificate keeps its existing filename so an upgrade does not
move a deployed PEM, and the nginx renderer keeps the single-certificate
resolver it still relies on.
* fix: prune stale TLS certs from the lb-agent cert dir
A delivered PEM is cert + chain + private key. writeCertFiles only ever
wrote, so every certificate detached from a listener left its key on the
load balancer VM's disk indefinitely. Prune after a successful reload,
scoped to .pem files in the cert dir's immediate entries.
* fix: serve every attached certificate on the nginx NLB engine
nginx allows one ssl_certificate per key type in a server block, and the
NLB config is a stream block where server_name does not exist. Select by
SNI instead: give ssl_certificate a variable and map $ssl_server_name to
each certificate's path, keyed on the leaf's DNS SANs. A single-certificate
listener keeps a literal path, since a variable costs a per-handshake read.
The cert-path helpers were HAProxy-only by hard-coding lbagent.CertDir, so
they now take the dir and both engines share one multi-cert resolver.
* fix: emit ValidationMethod in DomainValidationOptions
DescribeCertificate returned each domain validation entry without a
ValidationMethod, so the AWS provider had nothing to compare against
aws_acm_certificate.validation_method. Emit DNS for the DNS-based modes.
PRIVATE_CA stays empty on purpose: it validates nothing, and real ACM omits
the method on a private certificate, where the provider reads it as NONE
whatever the request asked for.
Persist and echo requiresCompatibilities on task definitions, so a Terraform round trip no longer reads null as drift and force-replaces the definition on every apply.
Report a task whose containers all exited while the ecs-agent was down. reconcile only re-adopted running containers, so the gateway held such a task RUNNING forever, its service never replaced it, and the load balancer target stayed unhealthy.
Single-node OVN RAFT bring-up gated only on the Northbound DB socket, then
restarted ovn-controller without waiting for the Southbound RAFT to elect a
leader, serve clients, and for northd to write the gateway logical flows. With
no join step to absorb the boot window the controller attached mid-election,
programmed a partial datapath (SNAT ct-commit without output/delivery) and
never reconverged, so NAT-GW egress and EIP ingress intermittently black-holed
on the single-node baremetal CI host.
Add an SB readiness gate before ovn-controller starts (SB reachable and, when
clustered, an elected leader), and after ovn-controller restart wait for its SB
connection then force inc-engine/recompute to rebuild all OpenFlow from the
converged Southbound. Applies to 1/2/3-node RAFT alike.
The failed state existed and was unreachable: nothing drove available →
failed, so DescribeDBInstances reported available for a database that had
been dead for a week. Add the failure classifier on top of the rds-4
reconciler.
An instance is failed when its system VM is not running AND its agent
heartbeat is stale — never either alone, since a stale beat under a live VM
is an agent fault rather than a dead database. The transition records a
timestamped reason that DescribeDBInstances and DescribeEvents can explain,
and a healthy heartbeat clears it again, which is the recovery the AMI's
in-guest Restart=on-failure and EC2's VM auto-restart provide underneath.
The recovery ladder and its fencing gate stay deferred to v1.1.
The failure clock is persisted rather than held in leader memory, so a
leader change does not restart the window, and it resets only on a healthy
heartbeat so a VM that boots and immediately wedges cannot mask a
persistent fault. Every lifecycle-owned state is skipped outright, which is
what keeps the classifier from racing rds-5a/rds-5b over the same VM.
A persisted heartbeat is judged against the stale window plus the persist
floor: beats are queue-group delivered, so the leader mostly sees them only
through the record, which a healthy agent refreshes no more often than
every five beats. The steady-state path is answered from the record alone,
so a healthy fleet costs no VM describe fan-outs.
* fix(test): reclaim fixture scratch dirs stranded in /tmp
The shared predastore fixture and the integration suite's JetStream store both
outlive any individual test, so neither can use t.TempDir() - the first test to
finish would delete files the daemon still has open. Both used os.MkdirTemp
with nothing removing the directory afterwards, so every run stranded one, and
a run killed by a timeout panic or the OOM killer stranded it regardless. /tmp
is a tmpfs on the dev boxes, so the leak is charged to RAM and swap, and
swapped tmpfs pages have no backing store to be dropped to.
Add a shared sweep that reclaims a directory only when it matches both the
prefix and an age no live run can reach. The NATS store also has a clean
shutdown point, so TestMain now removes it outright and the sweep is left to
cover the killed-run case. The predastore daemon runs until the process exits
and has no such point, so the sweep is its only mechanism.
* chore: repin viperblock to the dev tip carrying the scratch-dir fix
Test-only upstream change, so the build output is unchanged - this keeps
go.mod aligned with the submodule pointer rather than delivering a fix.
* fix(dhcp): retransmit within a DORA transaction instead of restarting it
nclient4's retryFn re-sends the same packet under the same xid with a
doubling read deadline, which is RFC 2131 4.1 retransmission. WithRetry(1)
runs that loop exactly once, so the client transmitted a single DISCOVER
per attempt and never retransmitted it. socketTimeout then handed that
lone transmission the caller's whole budget, so it sat silent for 4s,
then 8s, then 16s, then 32s.
Manager does retry, on that schedule, but each attempt builds a fresh
client and therefore a fresh xid. Every attempt is a new transaction
rather than a retransmission of the previous one, so an OFFER answering
an earlier exchange is discarded as belonging to an unknown transaction.
Send six transmissions per exchange starting from a 1s read, spanning a
63s ladder — longer than the widest window Manager schedules, so an
attempt ends on the caller's context rather than on an exhausted ladder.
socketTimeout now clamps to the caller's remaining budget instead of
consuming it in one silent read. Observed live recovering an exchange
whose OFFER arrived on the third transmission, which a single-shot
client would have missed.
Both allocation sites also stop replacing the allocator's error with a
bare InsufficientAddressCapacity. ResolveErrorCode unwraps, so a
genuinely exhausted static pool still surfaces that code while a DHCP
fault reports as itself. The flat mapping is why every failure on this
path read as a capacity problem regardless of cause.
* chore(dhcptest): add --broadcast so the probe can mirror vpcd's Acquire path
* fix(dhcp): resolve the chaddr on renew and release, not just on acquire
Acquire derived a chaddr when the request carried none. Renew and Release
read lease.HWAddr straight out of the store and handed it to nclient4, so
a lease persisted before the chaddr was carried renewed with all zeros in
chaddr and option 61 fell back to its untyped form. The lease then lands
upstream with no hardware address against it — the outcome option 61 was
fixed to stop, reached by a different route — and because the server sees
an unknown client each time, every renewal takes a new lease.
resolveHWAddr now serves all three paths and treats all-zeros as unusable
rather than merely empty: six zero bytes passes a width check yet is the
exact value that produces the unattributable entry.
* test(dhcp): cover chaddr resolution and the unmasked allocation errors
* fix(eks): GPU node AMI declares uefi boot mode
The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but
the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under
legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at
firmware with zero serial console output.
* fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs
Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to
initialize under the closed/proprietary kernel module: dmesg shows
RmInitAdapter failing with "requires use of the NVIDIA open kernel
modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to
nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia
recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no
separate open variant (userspace tools only), so it is unchanged.
* fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons
A cluster with a GPU-only nodegroup has no untainted node: the CP
carries CriticalAddonsOnly (EKS parity keeps user workloads off it)
and every worker carries nvidia.com/gpu. Without tolerations for both,
ebs-csi-controller and aws-load-balancer-controller have nowhere to
schedule and stay Pending forever. Real EKS never needs this since its
control plane isn't a schedulable node, so the stock upstream
manifests carry no such toleration.
* fix(images): give the libguestfs appliance a working resolver
The guest's /etc/resolv.conf symlinks to systemd-resolved's stub
(127.0.0.53), which isn't running inside the appliance chroot, so DNS
is dead until package installs need it. Point it at a real resolver
before package installs, then restore the stub symlink afterward so
the shipped AMI still gets its DNS from systemd-resolved at boot like
a stock cloud image.
* fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind
A GPU's IOMMU group can include an upstream PCIe bridge when the
platform lacks ACS isolation between them. vfio-pci refuses to bind
non-endpoint bridge devices, so Claim/ReclaimByAddress must filter
bridge-class (0x06) members out of the group before touching driver
bindings, rather than failing the whole claim or leaving the bridge
rebound incorrectly.
* fix(ecs): GPU node AMI declares uefi boot mode
The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the
Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS
no OVMF/pflash firmware is attached, so the guest hangs at firmware with
zero serial console output.
Same defect and same fix as the eks-node-gpu entry: both images are built
by the same script from the same stock Ubuntu cloudimg, so both were
EFI-only and only one was corrected.
* fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch
ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary
kernel module. NVIDIA ships no proprietary module for Blackwell (GB202)
and later, so ECS GPU container-instances could not bind those cards on
any driver branch. It is the third verbatim copy of the routine and was
missed when the other two were fixed.
Also pins the driver branch across all three copies. The branch was
resolved at build time via apt-cache search | sort -rn | head -1, which
sorts on the number in the package name — but that number is a branch
alias, not a driver version (nvidia-dkms-535-server installs 580.126.20),
so the ordering key need not match what installs. Resolving also let
identical source produce different drivers on different days.
The pin lives in setup.sh rather than manifest.conf because manifest
variables are not passed into the guest: the setup script is uploaded and
executed by virt-customize --run, with no environment carried across.
* fix(eks): tolerate CP and GPU taints in argocd addon
All six argocd Deployments and its StatefulSet carried no tolerations. A
cluster with a GPU-only nodegroup has no untainted node to place them on:
the control plane carries CriticalAddonsOnly and every worker carries
nvidia.com/gpu, so the pods stay Pending forever.
Same defect the ebs-csi-controller and load-balancer-controller addons
already had. Real EKS never hits it because its control plane is not a
schedulable node, so the stock upstream manifests carry no such
toleration and the divergence is ours to maintain.
---------
Co-authored-by: joshsiv-mulga <mrk@mulgadc.com>
* fix: detach before delete on instance terminate to stop root-volume leak
TerminateStoppedInstance and the running-instance terminate path both
deleted a DeleteOnTermination root volume without first clearing its
attachment, so DeleteVolume's in-use guard rejected it and the error was
swallowed (stopped path) or only alarmed (running path), leaking the
volume permanently.
Add VolumeServiceImpl.DeleteVolumeOnTerminate, which clears the stale
attachment via UpdateVolumeState (terminate implies detach) before
deleting, and route both terminate paths through it. Stamp
Teardown[TeardownVolumes] on the stopped path so a delete failure is
visible to the volume leak reaper instead of being lost.
* fix: detach non-DeleteOnTermination root on terminate; cover teardown self-heal
A DeleteOnTermination=false root volume stayed attached to the terminated
instance on both terminate paths: Stop's Unmount deliberately skips Boot
volumes, so the root kept its attachment across stop and neither terminate
path detached it, stranding it attached to an instance that no longer
exists. Add VolumeServiceImpl.DetachVolumeOnTerminate (metadata-only
attachment clear, no delete) and route the DeleteOnTermination=false Boot
volume through it on both paths, matching AWS terminate-implies-detach.
Also add a regression test proving the TerminatedTeardownReaper re-drives a
failed volumes-teardown through the now-detach-first cleaner and self-heals
the leak, so a transient delete failure at terminate does not strand a
volume permanently.
* fix: bound detach chain and reconcile vanished-QEMU wedge
DetachVolume gains a bounded aggregate context deadline over the QMP
hot-unplug chain and a confirmed-dead-QEMU short-circuit that skips the
QMP steps (which would only hang on the wedged process) and seals
directly, so a detach can no longer pin attachMu forever when nbdkit or
QEMU die mid-flight.
OrphanQEMUReaper now also finalizes shutting-down instances whose QEMU
has vanished: a terminate that wedged downstream and never reached
finalizeTerminated is driven to terminated, with outstanding teardown
stamped failed so TerminatedTeardownReaper completes it idempotently.
Gated on a dead process, so a live in-progress terminate is never
stolen.
* feat: add delete-authorized stuck-terminate backstop
A terminate that wedges with QEMU still alive is invisible to the
OrphanQEMUReaper dead-QEMU reconcile and would sit shutting-down forever.
StuckTerminateReaper force-completes it: past stuckTerminateTimeout it
force-kills the wedged process, reclaims DeleteOnTermination volume space
through the cleaner, stamps remaining teardown for TerminatedTeardownReaper,
and drives the record to terminated.
It is the one reaper explicitly authorized to delete volume data, but only
for a DeleteOnTermination volume on an already-terminating instance — a far
narrower licence than ADR-0005 §3 denies VolumeLeakReaper, which stays
mark-and-alarm-only. A new ShuttingDownAt timestamp, stamped once on the
transition, bounds how long a terminate may wedge; a missing timestamp reads
as not-yet-stuck so the backstop never acts on an unbounded record.
TestNPassOverwrite detaches after every pass, so viperblock's drain and GC
only ever run at reopen — it cannot show whether GC reclaims superseded
chunks while a volume stays open, and its serial write rate never nears the
predastore rate limiter's burst. TestContinuousChurn attaches once and drives
concurrent 4 MiB-aligned raw overwrites of per-writer regions for several
minutes without detaching, so a sustained superseded-chunk backlog builds
under one long open, spanning multiple GC ticks.
The workload is bounded and self-contained: each dd writes one chunk and the
clock is re-checked after every write, the loop runs as root under a single
sudo bash with a process-group kill trap, and it is wrapped in a guest-side
timeout. Root ownership is load-bearing — the writers open a raw device, so
only a root-owned process group can reap them; a best-effort guest pkill runs
before detach as a backstop so a stray writer cannot wedge the detach.
Picks up viperblock 3dcfbb7 (#49): the chunk GC sweep now runs on a
dedicated goroutine instead of sharing the chunk-uploader select, so
sustained overwrite churn no longer starves GC and wedges the volume at
the internal space cap.
This reverts commit f9c9f0b38d.
#580 switched setup.sh to read the northstar :53 listener and zones from
northstar.toml, so the resolved stage now defers at install time (before
formation) and must be re-run afterward. Only firstboot re-runs it; the
manual setup.sh + spx admin init install and CI never do, so those hosts
get no host DNS and cluster-zone names (ELB/EKS) fail to resolve from the node.
Reverting restores the pre-#580 behaviour: systemd-resolved is configured
at install time from the WAN IP, with no post-formation re-run needed. This
reintroduces the original gap where resolvconf hosts get no host DNS. The
proper fix configures host DNS from spx admin init/join for both resolver
managers (mulga-x30yv).
A freshly launched LB agent begins heartbeating the instant the VM boots,
but the gateway node fielding that first heartbeat may not yet have the LB
record — it is still replicating from the node that created it — so the
gateway answers 400 LoadBalancerNotFound. The IMDS credential chain is
likewise not always warm on the first tick. Both self-heal within a few
ticks, yet every early failure was logged at ERROR, flooding the console
telemetry with startup noise (46 such lines across 8 agents in one run).
Bound these to a registration grace window: before any heartbeat has ever
succeeded, and only within the window, a failure logs at INFO. Past the
window with no success — the fingerprint of a genuinely blackholed agent —
or after any prior success, a failure logs at ERROR as before, so the
serial-console telemetry still surfaces a real blackhole. The only change
for a persistent failure is a bounded delay before it escalates.
* fix: make the seal-failure unmount test deterministic
The test forced the seal to fail by pointing S3Host at an unreachable
endpoint, so the handler only returned once the S3 client had exhausted
its jittered backoff. Measured handler time spanned 1.8s to 5.5s against
a 5s request deadline, which failed 2-3 runs in 10 on an idle machine and
worse on a loaded CI runner.
Config gains a sealVolume seam, nil meaning the real seal, and the test
injects an immediate failure instead. It now runs in 0.53s, 50 for 50,
and still asserts what it was written to assert -- plus that the handler
actually attempted the seal, which the old form could not distinguish.
* chore: bump viperblock to dev tip (GC observability + scratch sweep)
Picks up viperblock #47 (every chunk GC sweep logs its outcome at Info)
and #48 (TestMain reclaims scratch dirs abandoned by a panicking run).
The tier ran inside the Build and Test job, serialised ahead of test-cover, and
again in preflight. Neither placement bought signal that a parallel job does not:
it provisions nothing and finishes in seconds, so the only effect was wall time
on the two paths developers wait for most.
It stays PR-blocking as its own job, which is what distinguishes it from
e2e.yml's front gate — that one is push-triggered on a self-hosted runner and so
never gates a PR before merge.
MgmtIPAllocator allocated from a process-local map, but br-mgmt is
trunked onto a shared physical NIC making the /24 one flat L2 segment
across the whole cluster. Every node independently allocated from
.10, causing collisions and ARP-race losers being blackholed.
Rewrite the allocator on the spinifex-cluster-state NATS KV bucket
using compare-and-swap, keyed by mgmt-ipam.<subnet>, modeled on IPAM.
Bind the KV handle in startCluster (not startLocal) to respect the
DDIL boundary. Serve already-allocated instance IDs from the local
cache when KV is unhealthy, but refuse new allocations until it
recovers. Rebuild now reconciles this node's running VMs into the
cluster-wide record instead of just refreshing the local cache.
* feat(config): enable viperblock chunk GC in rendered node configs
The gc_enabled toggle was plumbed through every VB construction site but
nothing ever set it. GCEnabled is a *bool that every consumer treats as
off when nil, and the config template rendered shardwal while omitting
gc_enabled, so no deployed node ever carried the key and chunk GC was off
everywhere by construction. Superseded chunks were never reclaimed under
overwrite churn.
Render gc_enabled = true alongside shardwal, and cover it with a test that
loads the rendered file through config.LoadConfig rather than matching the
template text. A string match would still pass if the value failed to parse
into a non-nil pointer, which is the exact way the setting reaches its
consumers.
Existing deployments keep the old rendering until their config is
regenerated; the template governs install and bootstrap only.
* chore: pin viperblock to the build with the cross-process GC snapshot marker
Enabling chunk GC in rendered node configs is only safe once the sweeping
engine can see a snapshot taken by another process. viperblock 3b56d61 adds the
per-volume snapshot marker that closes that window, and 2a34add fixes the
concurrent-stop panic in the background goroutine lifecycle that GC's own ticker
would otherwise widen. Pin both before turning GC on.
LaunchRunInstances is a best-effort batch launcher that reports failure only
through VM state: volume preparation, GPU claim and vmMgr.Run errors each call
MarkFailed and continue. launchAMISystemInstance ignored that and always
returned a SystemInstanceOutput, so an EKS control-plane server, ELB or Bedrock
serving VM whose launch failed handed its caller an instance ID for a VM that
was already being torn down.
Check the instance reached running state before returning. Cleanup stays with
the MarkFailed chain LaunchRunInstances has already run.
buildDrives emitted every non-boot data volume as a bare -drive with no
id= and no node-name=, so a stop/start (or reboot/resize) relaunch left
the volume anonymous: device_del/blockdev-del had nothing to address,
QEMU kept the NBD socket open, and DetachVolume stalled through
nbdkit's full kill grace before giving up.
Add -blockdev support to the QEMU config builder and use it for
non-boot, non-EFI volumes, sharing the node-name/device-id/iothread-id
naming and virtio-blk-pci device arguments with AttachVolume's hot-plug
path so cold-boot and hot-attach can no longer drift apart. A data
volume with no persisted HotplugPort now gets one allocated and written
back during buildDrives instead of landing on an unplugged bus.
Also raise unmountSealTimeout past viperblockd's kill grace (it runs
the seal after the SIGKILL, so matching the grace meant an unmount
that used the whole grace could never be observed as successful), and
bound the e2e harness's forced teardown detach by the teardown
deadline so a slow detach is reported as slow rather than as an
unexplained leak.
* test: assert the gateway->vpcd->OVN crossing in the integration tier
The tier already ran the real VPCServiceImpl for the AWS-facing half, but
nothing consumed the vpc.* events it published: the SG topics were answered
by a canned {"success":true} and the fire-and-forget topology topics were
dropped. Both sides of that wire were tested only against stubs, so the join
was asserted nowhere below the live e2e tier.
Wire a real subscribers.Subscriber over a real in-process OVN Northbound DB
(ovntest.StartNB), opt-in per test so only tests that assert OVN state pay
the startup cost. StartDaemonLite gains WithRealVPCD to suppress its stubs,
since a stub and the subscriber both answering one request-reply would race.
Scope is the crossing, not NB row shapes — those stay with the reconcile
scenario tests. Everything host-dependent (ovn-controller, OVS, taps, netns,
nftables, packet paths) remains the live tier's claim.
* fix: make the integration tier re-runnable and quiet the ensure probe
Two defects the vpcd/OVN work surfaced.
The tier was green at -count=1 and red package-wide at -count=2. Test
accounts were keyed by t.Name(), which is unique among tests but repeats
across passes in one TestMain, so a second pass inherited the first pass's
JetStream namespace — including an ECR signing key sealed with a master key
StartGateway regenerates per call. Key each account off a counter instead.
TestRunInstances_ClientTokenIdempotency then collided with itself for the
same reason: its dedup store outlives a pass, so its fixed ClientToken made
the second pass's opening launch look like a replay. Suffix the tokens per
invocation.
ensureNamedRowOps' wait-op is built to fail fast when the row it guards
already exists, which is the ordinary idempotent path, but transactOps
reported it as "OVSDB operation failed" at ERROR. Every re-ensure emitted
one, drowning real faults in the log sink. Classify that one result as the
probe outcome it is and log it at Debug.
* test: cover transactOps' failed-result paths
The classifier split one log line into two branches and neither was
exercised: transactOps sat at 31% because EnsureLogicalRouter's cache
lookup absorbs a re-ensure before it ever transacts, so no unit test
reached a failed operation result.
Drive both kinds directly against a real NB — the guard probe finding an
existing row, and two same-named inserts violating the schema's name
index — since only the second is a fault and they now log at different
levels. transactOps goes to 94%; the remainder is the transport-error
return, which needs a broken connection to reach.
* feat(ci): extend per-cell telemetry scoping to every e2e workflow
ci.cell only reached e2e-nightly and e2e-suites, so a run of e2e, e2e-ddil
or gpu-e2e still landed in the sink as one undifferentiated bucket. Stamp it
on the remaining workflows: single/multi on e2e's two provisioning jobs, ddil
and gpu on their single-job workflows.
The best-effort numeric job id lookup moves out of e2e-nightly into a shared
composite action, so the four other provisioning jobs get ci.job_id without
copying the gh/jq incantation. The lookup now matches on env.JOB_NAME rather
than interpolating the name into the jq filter, which broke on names carrying
quotes or non-ASCII separators, and no-ops when telemetry is disabled.
gpu-e2e gets ci.cell but deliberately no job id lookup: nothing in that job
bakes the attrs into a daemon, so the API call would be paid for an env var
no exporter reads.
* fix(ci): define e2e suite topology eligibility once
Three workflows carried three different answers to which suites a topology
can run. e2e-suites and e2e-nightly ran lb on their single-node leg, where
every internet-facing subtest skips for want of a peer, so a fraction of the
suite reported as a green lb run. e2e-suites' multi leg ran multinode alone,
dropping the lb and cert coverage that only a multi-node env can complete —
so full lb coverage never ran there at all.
.github/scripts/suite-sets.sh is now the single definition, sourced by all
three. e2e.yml's two allow-list regexes derive from it, e2e-suites derives
both matrices from it (multi_provision gains the suite-matrix step
single_provision already had), and e2e-nightly's deliberately narrower
per-cell lists are declared there too, guarded by an assert that fails the
cell if a subset ever names a suite its topology cannot fully exercise.
The nightly stays narrow on purpose: its cells have a 35-minute budget and
exist to prove each install/network/host-OS permutation boots and serves.
Suite breadth is the PR gate's job and already runs on every push. That is
now stated in one place rather than implied by a third divergent list.
* test(volume): assert chunk GC stays wired on in CreateVolume
GCEnabled was zero-valued at every viperblock construction site for a long
time and no test noticed. It is not part of viperblock.VBState, so it never
round-trips through a saved config.json and cannot be observed by reading
volume state back.
Extract the viperblock.VB assembly out of CreateVolume into buildVBConfig so
the GC wiring is assertable directly against the returned config, without a
live viperblock or S3 backend. The config is returned by pointer because
viperblock.VB carries atomic.Uint64 fields that vet's copylocks check
rightly refuses to see copied.
* test(e2e): snapshot backend byte accounting before volume purge
DeleteVolume purges a volume's entire backend prefix, including any chunks a
leak left behind, before anything measures it. That is the same masking that
hid the storage-growth leak in production: a passing E2E cannot fail on
backend growth when the evidence is deleted first.
Snapshot the volume's backend chunk footprint in EnsureVolume's cleanup
closure, immediately before DeleteVolume, so every volume the shared teardown
path purges is measured rather than silently discarded. The snapshot is
always taken and logged; the hard ratio check additionally runs for callers
that registered a guest-written-bytes baseline, since most callers never
attach or write to the volume they create and so have no baseline to compare
against.
The tolerance is the RS(2,1) erasure floor widened for per-chunk header and
tail-chunk overhead. It is sized to distinguish unbounded growth from a
footprint settling at the floor, not to catch bounded residual garbage.
The check is non-fatal inside the cleanup callback: t.Fatalf's runtime.Goexit
would abort the DeleteVolume call queued behind it and turn a reported leak
into a real one.
* test(e2e): add memory-constrained concurrent-write durability regression
The existing durability tests write one 4MiB sentinel, sequentially, on a
comfortably sized guest, which is the exact inverse of the known-open
data-loss bug: acked writes read back zero-filled after a cold reread when a
guest is memory-starved during concurrent writeback. No happy-path test can
reach it, because the fault lives in the intersection of memory pressure,
write concurrency and cache-cold reread.
Add a suite that encodes those three conditions together: a guest discovered
at or below the working set, N parallel large writers launched as background
jobs in a single command so they genuinely overlap, swap disabled, then sync
and drop_caches before rereading. Assertions are per stream rather than
aggregate so a failure names the losing writer instead of leaving whoever
reads it to re-derive which of N writers lost data from one combined digest.
GuestExec gains a timeout-parameterised variant. Its fixed two-minute default
is sized for sub-second payloads, and a multi-gigabyte write to a
network-backed volume would exceed it, so the suite would have failed by
timeout while looking like it had caught the corruption. The budget scales
with the working set, and a deadline is reported distinctly from a data
mismatch so the two red states are never confused.
The suite is dispatchable but deliberately absent from the default SUITES:
the bug it reproduces is still open, so it is expected to fail and must not
sit in a merge-gating path. It is registered in the workflow allow-list so it
stays discoverable rather than being buried behind a skip.
Also record on the live IAM instance-profile test why its assertions were not
moved down to the integration tier, so the trace is not repeated.
* test(e2e): drop backend byte accounting, assert a node-wide delta instead
The accounting helper listed the internal predastore bucket to attribute
bytes per volume. That listing is refused unconditionally: predastore
authorises bucket access by same-account ownership, and the bucket belongs
to the service account, so a test caller gets 403 no matter what it holds.
Granting the test a service credential would invert the trust boundary to
buy a measurement, which is the wrong trade.
Replace it with what the S3 surface actually permits: sample node-wide
allocated bytes either side of the write and bound the delta against the
bytes the guest wrote. Coarser, but it measures something real rather than
depending on a call that cannot succeed.
The tolerance moves into the suite that uses it. It encodes RS(2,1)'s 1.5x
expansion floor plus headroom; a cluster running the recommended RS(3,2)
has a 1.667x floor and only ~12% of margin left, so the constant needs
revisiting if the encoding changes.
* revert: drop memdurability e2e suite
The suite asserted the guest read back what it wrote after a
memory-constrained concurrent write, but the assertion could not fail for
the reason it claimed: it compared the guest's own read against the
guest's own write with the page cache still warm, so it passed whether or
not the data reached the backend. A test that is green on a build with a
known-open durability bug is worse than no test — it advertises coverage
that does not exist.
Removes the suite and its Makefile/workflow wiring, plus
DiscoverInstanceTypeAtLeastMemory, which had no other caller.
GuestExecTimeout stays: GuestExec now delegates to it, and the old
two-minute hard cap it replaced was a latent trap for any long guest
command.
* test(e2e): add partial-block concurrent-write durability suite
Replaces the removed memdurability suite, which passed on a build that
still carried the bug it claimed to test.
The engine reaches its read-modify-write path only for a partial block;
a fully covered block takes a full-overwrite branch that reads nothing
and can lose nothing. The old workload wrote through ext4 with 1 MiB
buffers, which the filesystem coalesced into aligned full-block
requests: block-layer tracing recorded zero unaligned and zero sub-4096
requests, so the path under test never ran, and writers 4 MiB apart
could not collide on a block even in principle.
This suite writes the shape the defect needs. Two writers issue
concurrent O_DIRECT writes into disjoint halves of the same 4096-byte
blocks on a raw, unformatted device, stamping every half with the round
number, then drop the page cache and reread with O_DIRECT asserting
every half carries the final round. One half reading back at an earlier
round is the signature of the lost update.
The old suite also could not fail even had it reached the path: its
write phase ended in 'wait; sync; sha256sum' and reported sha256sum's
exit status, while bare wait returns 0 unconditionally, so writer errors
were invisible and a backend ENOSPC presented as a pass. Here each
writer's exit status and exact byte count are asserted before any
durability claim is made.
Working set is 8 MiB rather than 2048 MiB, and the volume is deleted on
teardown; the predecessor left roughly 3 GiB of backend per run and
twice drove a node under the nearfull watermark.
Dispatchable but not in the default suite list until a green run and a
demonstrated fail-first against a pre-fix engine exist.
* test(e2e): let the fixture own the working-set volume delete
The suite deleted the volume in its own cleanup and the fixture deleted
it again on close, so every run ended with an InvalidVolume.NotFound
from the second attempt. Detach stays — DeleteVolume against a still
attached volume is what leaks volumes on teardown — and the delete is
left to the fixture so the volume is released exactly once.
* test(e2e): gate merges on the partial-block durability suite
Verified on one node with the same test binary and workload, varying
only the engine: 1758 of 8192 halves reverted a generation against the
pre-fix build, 0 of 8192 against the fixed one. That is the fail-first
evidence the suite needed before it could be trusted to gate anything.
Adding it to the default suite list rather than leaving it dispatchable
is the point. Its predecessor sat outside every path that runs, which
is how it survived long enough to be discovered green on a build that
still lost data.
* test(integration): cover the ECR data plane in the front gate
The live ecr/ suite drove docker, crane and skopeo against a real
registry, yet none of it boots a guest: it needs a registry, an object
store and an HTTP client. The in-process gateway already serves the OCI
Distribution v2 surface on an ephemeral port with a memory-backed object
store, so these run here in 0.18s instead of behind full provisioning.
Covers the three claims the live tests made beyond authorization, which
is already down here: a push/pull round trip asserting blobs and
manifests read back byte-identical, refusal to auto-create a repository
on push, and retagging an existing manifest without disturbing the
original tag.
Content is compared rather than status codes alone. A registry that
stores a blob under the wrong digest, truncates it, or serves a
different one answers 200 to every request in the sequence.
The capacity/load leg deliberately stays live: it exists to put real
parallel pressure on a real predastore, and a memory-backed object store
cannot make that claim.
* fix(e2e): make volume teardown release the volume before deleting it
Fixture cleanups drain LIFO, so a suite that creates its instance before its
volume tears the volume down first, while it is still attached. DeleteVolume
answered VolumeInUse, nothing retried, and the volume was stranded on the node.
Reordering the chain cannot fix this — registration order has no way to express
the instance/volume dependency. Instead teardown is now self-sufficient for any
creation order: it force-detaches, waits for the volume to reach available, and
only then deletes. Instance teardown waits for terminated for the same reason,
since TerminateInstances returns as soon as the request is accepted.
Two things kept the leak invisible. A failing cleanup was logged and forgotten,
so a run stayed green with resources left behind; failures are now reported as
leaks, failing the bound test or surfacing through Close() for TestMain to fail
the run. And a panic in one cleanup aborted the whole chain, stranding every
resource behind it; each callback is now isolated.
RegisterVolumeTeardown and the fixture had grown separate teardown paths, only
one of which detached. Both now share it.
Re-creating a nodegroup of the same name could never reach ACTIVE. The
Ready-gate took its baseline from the live Ready count for that name, so
a terraform replace after CREATE_FAILED — which deletes then re-creates
— captured the previous incarnation's workers, demanded old+new, then
watched the count fall as the old workers were terminated. The target
was unreachable by construction.
Observed on env19 with a perfectly healthy cluster: three new workers
all joined and the create still failed with
reports 3 Ready nodes, want >= 5 (baseline 2 + 3 workers)
and each retry raised the bar again (baseline 3 => want 6), so the only
way out was untainting the resource by hand.
A create means every worker under that name is new, so the gate is just
"desired Ready nodes carrying this nodegroup label" — baseline 0. The
trade-off is stated in the comment: a stale incarnation's Ready nodes
can briefly inflate the count and satisfy the gate early, but they go
NotReady within about a minute of termination, whereas baselining on
them fails every replacement permanently.
The test models the real sequence — delete, stale Ready count still
reported, re-create — and fails without the fix with the same shape as
the field error.
* fix(dhcp): derive nclient4 read deadline from the caller's context
nclient4 races its own read deadline against the caller's context and the
shorter one ends the attempt. Acquire and Renew passed a fixed 5s, so every
window in Manager's 4s/8s/16s/32s retransmission schedule past the first was
truncated to 5s: the client listened for ~19s of its 90s budget and a server
that went quiet for longer was never waited out.
Measured in the sink, the split was total — attempt 1 (jittered under 5s)
always failed with the caller's context deadline, attempts 2 to 4 always
failed with nclient4's ErrNoResponse — and only 27% of clients that lost
attempt 1 were rescued by attempt 2.
Track the context's remaining time instead, keeping the configured value as
the fallback its doc comment already describes.
* fix(dhcp): keep the socket deadline strictly behind the caller's
Setting the nclient4 read deadline to exactly the context's remaining time
makes the two expire together, so a timed-out attempt reports either
context.DeadlineExceeded or nclient4's ErrNoResponse depending on which
goroutine wins. Observed live on a settled bridge: three attempts reported the
context deadline and the fourth reported ErrNoResponse, which reads as a
packet-matching fault rather than the plain timeout it is.
Give the socket a second of grace so ctx.Done() is always what fires.
* fix(dhcp): encode option 61 with its hardware-type byte
The client identifier was sent as a bare string, so the server read the
identifier's first character as the hardware type — "dhcp-gw-lrp-vpc-x"
arrived as type 100 ('d') with the identifier truncated. Since that is not
Ethernet, the server recorded no hardware address, and the upstream lease
table listed our leases with a MAC of all zeroes.
Send the chaddr typed as Ethernet, falling back to a type-0 identifier when
no hardware address is available. RELEASE now encodes identically, since the
server matches a release to a lease by client identifier.
Option 61 no longer carries the readable id, so restore it via hostname and
vendor class, which were specified but never populated. dhcptest builds its
options through the same helper rather than its own copy, which had drifted
into reproducing the broken encoding.
* fix(dhcptest): derive the probe MAC the way vpcd does
The probe carried its own copy of the MAC derivation, still on the 24-bit
02:00:00:xx:xx:xx hash that vpcd moved off when widening it to a SHA-256
prefix. So --spinifex-id leased under a MAC the daemon would never use:
"gateway-wan1" resolved to 02:00:00:7c:b7:b6 here and 02:54:df:71:22:5c in
vpcd, which makes the probe unable to reproduce a daemon lease at all.
Call DeriveMAC instead and drop the local copy.
The nightly matrix runs every cell under a single ci.run_id, and each cell
names its hosts node1/node2/node3, so telemetry from all cells collapses into
one undifferentiated bucket in the ES sink. Root-causing a single failed cell
means aggregating across every other cell in the run.
Stamp ci.cell and ci.matrix alongside the existing ci.run_id. Workflow-level
env cannot reference the matrix context, so the attrs are overridden in each
matrixed job's own env block, leaving the workflow-level value as the
non-matrix fallback.
The numeric GitHub job id is not exposed in any context (github.job is the
YAML key, not the id), so resolve it best-effort via the jobs API and append
it if found. The lookup never fails the job.
e2e-suites can only distinguish single from multi: daemon telemetry is baked
into the VM at provision time, before the suite matrix exists.
Baremetal shipped no telemetry at all — neither the endpoint nor the attrs
crossed the workflow_call boundary. Pass them through as an explicit input so
the cell reaches the sink for the first time.
* fix(eks-node): stop the DHCP MTU overriding the pinned link MTU
Pinning eth0 to 1320 was never enough. This image runs dhcpcd, which
honours DHCP option 26 and stamps the advertised MTU onto every route it
installs:
default via 10.32.101.1 dev eth0 proto dhcp src 10.32.101.7 \
metric 1002 mtu 1442
Linux prefers a route's MTU metric over the device MTU, so the node kept
advertising an MSS derived from 1442 while the real path MTU through the
NAT gateway is 1408. Small packets pass and full-MSS inbound segments —
a TLS ServerHello with its certificate chain, image layers — are
silently dropped. It presents as `net/http: TLS handshake timeout` on
image pulls, not as anything MTU-shaped, and it only affects the host
netns, so containerd pulls fail while pod traffic sized off cni0 is
fine. Per-destination PMTU exceptions expiring in and out of the route
cache made it look intermittent and host-specific.
Three changes, innermost first:
- dhcpcd.conf gets `nooption interface_mtu`, so no lease can stamp a
route MTU. mulga-vpc-mtu is the single source of truth for the MTU
of these nodes.
- mulga-vpc-mtu re-stamps any route that already carries a different
MTU, covering an image built before this fix or a dhcpcd already
running. Routes with no MTU metric inherit the device MTU and are
left alone.
- the subnet DHCP option drops 1442 to 1408. 1442 is geneve-only and
ignores the IPsec ESP overhead on the NAT egress path, so it
overstates the usable MTU for every guest, not just these nodes.
Also corrects the header comment, which claimed Alpine's udhcpc ignores
option 26 — this image runs dhcpcd, which honours it. That wrong premise
is why the route metric was never considered.
Adds a stubbed-`ip` test covering the rewrite, the dev token that
`ip route show dev X` omits, idempotency, and routes without an MTU
metric.
* fix(eks): sweep the CP VPC by identity so teardown can finish
DeleteClusterCPVPC finds the managed control-plane VPC's subnets and SGs
by role tag. A create that fails partway — the first control-plane
server failing to launch, say — leaves resources behind that never got
tagged, so the sweep cannot see them.
DeleteVpc rejects while any subnet or non-default SG remains, and
teardown is re-driven on a timer, so the cluster sits in DELETING
forever retrying the same doomed delete every couple of minutes:
eks-deleting: re-driving wedged DELETING teardown cluster=toc
eks-deleting: re-drive teardown failed
err="delete managed CP VPC: eks: delete cp vpc vpc-...: DependencyViolation"
Nothing converges: the cluster name cannot be reused and terraform's
delete wait can never pass. Observed on env19, wedged for ~11 hours
until this change went in, at which point the next re-drive cleared it.
After the tagged sweep, clear anything still in the VPC scoped by
vpc-id: subnets, and non-default SGs (the default SG goes with the VPC
cascade). Teardown had no SG surface, so CPVPCDeps gains an optional SG
provisioner — nil is tolerated, since only teardown uses it.
* chore: pin viperblock + predastore to the merged storage fixes
viperblock f8cfd77 (#42) makes read-modify-write atomic per block,
removing the lost update that silently reverted a 4096-byte block to a
previous generation of itself. Confirmed as the env19 corruption cause
by an A/B: the pre-fix build corrupted image layers within ~15 minutes
(failed to extract layer / invalid tar header / unpigz: corrupted) while
the fixed build ran 5+ hours clean, absorbing 6407 same-block RMW
conflicts on the worker root disks.
predastore c0a90d2 (#63) batches FSM.Restore. Without it, any predastore
restart on a store whose metadata set outgrew one badger transaction
fails with "Txn is too big to fit into one request" on every node at
once, taking the metadata plane — and with it the AMI catalogue and
cluster creation — down permanently.
Both were verified in the built artifact rather than from go.mod alone:
go version -m reports the new pseudo-versions, and the rmw_conflicts
counter plus the batched-restore error strings are present in the
binary. Pinning by go.mod while the submodule pointer says otherwise is
how an earlier analysis ended up reading dead code.
* test: record the corrected DHCP MTU in the kitchen-sink golden
The NB golden pins the DHCP options published per subnet, including the
advertised MTU. Dropping the subnet option from 1442 to 1408 changes
what the reconciler writes, so the golden diverged and the live scenario
test failed — the test doing exactly its job.
Regenerated with UPDATE_GOLDEN=1. The diff is only mtu:1442 -> mtu:1408
on the two subnet DHCP records; nothing else in the snapshot moves.
* test: add in-process integration test tier
Roughly half of the behavioural tests under tests/e2e/ assert control-plane
API shapes, authz and error codes that need no live environment, yet they can
only run against a tofu-provisioned environment on a scarce self-hosted
runner. This adds the tier they can move down to.
The real gateway router runs in-process: SetupRoutes() is a pure function of
GatewayConfig, so it serves from an httptest server over embedded NATS
JetStream with real IAM/STS services and genuine SigV4 verification. Only the
daemon side of the wire is faked -- the ec2.*/ebs.* NATS subjects a live node
would answer are stubbed per test.
Nothing is provisioned and no fixed ports are bound, so the tier runs
concurrently on a shared build host and is safe as a CI front gate ahead of
the live suites. It is build-tagged and kept out of preflight, which stays
unit-only.
* test(integration): add IAM/STS client accessors and port the zero-stub authz tests
Adds IAMClient/STSClient plus ClientsWithCreds/ClientsWithSessionCreds so a
test can act as a principal other than the seeded root user, and stubs the
node-discovery subject in StartGateway so call paths that trigger discovery
resolve immediately instead of riding out the fan-out timeout.
Ports the four enforcement tests that guard on ec2:DescribeRegions, a
gateway-local action, so they need no daemon-side stub at all.
* test(integration): port remaining IAM/STS authz tests from e2e
Ports TestIAMUserCRUD, TestIAMAccessKeyLifecycle, TestIAMUserAuthentication,
TestIAMPolicyCRUD, TestIAMPolicyAttachmentEnforcement, TestIAMPolicyLifecycle,
TestIAMGroupsLifecycle, TestIAMRolesAndProfiles and
TestSTSAssumeRoleAndGetCallerIdentity from tests/e2e/iam/ into the in-process
integration tier. Each test now self-bootstraps its own users/policies/roles
instead of relying on shared suite-level state, since every integration test
gets its own gateway and embedded NATS. TestIAMRolesAndProfiles stubs
ec2.IamProfileAssociation.describe because DeleteInstanceProfile always
counts live associations over NATS regardless of which guard actually fires.
TestIAMCleanup is not ported: it only swept a shared long-lived E2E
environment, and every ordering/dependency assertion it made is already
covered by TestIAMPolicyLifecycle, TestIAMGroupsLifecycle and
TestIAMRolesAndProfiles's own delete-conflict guards.
* test(e2e): remove IAM/STS authz tests migrated to the integration tier
All 13 signed-request authz tests plus TestIAMCleanup now have faithful,
mutation-proven equivalents in tests/integration/ — the gateway dispatches
IAM/STS actions directly with no daemon hop, so the in-process tier is a
faithful and far cheaper substitute. The iam/ e2e suite now runs only
TestIAMInstanceProfileAssociation and TestIMDS.
Drop the now fully-dead iam_helpers_test.go alongside the six deleted test
files, and remove harness.IAMGroupARN/IAMDeleteGroupBestEffort, which had no
remaining caller anywhere in the repo once iam_groups_test.go was gone.
* feat(integration): daemon-lite harness for key/tags/routetable/vpc
Adds DaemonLite, an in-process stand-in for a live spinifex daemon that
subscribes the real key, tags, route table, and VPC/subnet/SG/IGW service
implementations directly to their NATS subjects, backed by memory and
embedded-JetStream stores. Ports TestKeyPairs, TestTagManagement,
TestRouteTableValidation, and TestReplaceRouteConvergence from
tests/e2e/single so the integration tier drives genuine daemon-side
business logic instead of StubSubject canned replies.
vpcd (OVN) is out of scope, so its "vpc.create-sg"/"vpc.delete-sg"
acknowledgements are stubbed with a fixed success reply — the SG record
itself is still written by the real service impl before that event fires,
so the stub only unblocks the synchronous round-trip rather than
replacing tested behavior.
* ci(e2e): gate live E2E jobs behind the in-process integration tier
Runs the in-process integration suite first, on the ci-single runner host
rather than inside a provisioned VM, so a broken control plane is caught
before either live job pays the Spinifex install. Both e2e_single and
e2e_multi now require it, and e2e_status treats a skipped integration job as
a failure so the gating status cannot go green on a gate that never ran.
Drops the shadow-mode suite selector: targeted per-diff suite selection is
not the direction being pursued, and leaving a non-functional shadow job in
the gating workflow only invited it to be mistaken for a live gate.
* test(e2e): remove single/ tests superseded by daemon-lite ports
TestKeyPairs, TestTagManagement, TestRouteTableValidation, and
TestReplaceRouteConvergence now have proven-equivalent ports in
tests/integration/ (mutation testing confirmed parity). Delete the
e2e originals and their runX helpers, along with keypair_test.go's
and tags_test.go's file-local helpers left with no caller.
generateImportPubKey is shared by other single/ tests so it moves to
helpers_test.go instead of being deleted. Slims the manifest-lint
baseline for the removed create-call entries.
* ci: run the lb e2e suite on multinode only
Both e2e jobs filter SUITES by exclusion, and neither excluded lb, so the
suite ran twice per e2e run: 354s on single and 654s on multi. Balancing
across instances on one node is not a deployment mode we support, and
ELBv2 proper requires two AZs, so the single-node run was redundant.
Drops ~354s from the single-node job, which is the gate critical path.
* test(integration): port 3 of 7 ECR tests to the in-process tier
Wires the gateway's ECR control plane (repository/policy/lifecycle
metadata over an in-process daemon-lite NATS subscriber, blob and
manifest bytes via a memory object store, JWT issuer/verifier keyed off
the existing IAM master key) into the integration harness, and ports
TestECRControlPlane, TestECRGetLoginPassword, and
TestECRIAMAuthorization_AssumedRolePullOnly from tests/e2e/ecr — each
proven equivalent by mutation (evaluatePrincipalPolicy forced to allow,
and the repo-describe path forced to not-found) before its E2E original
was removed.
The other 4 authorization tests (PullOnly, ReadOnly,
DetachPolicyDeniesImmediately, DeactivatedKeyReturns401) stay on the E2E
tier: AttachUserPolicy lacks the AWS-managed-ARN existence-check bypass
that AttachRolePolicy/AttachGroupPolicy already have, so attaching
AmazonEC2ContainerRegistryPullOnly/ReadOnly straight to a user fails
NoSuchEntity in both tiers alike. Left untouched pending a fix to that
shared IAM code.
* fix(iam): accept AWS-managed policy ARNs on AttachUserPolicy
AttachUserPolicy called getPolicyByARN unconditionally, so attaching an
AWS-managed ARN to a user returned NoSuchEntity. AttachRolePolicy and
AttachGroupPolicy already skip that lookup for AWS-managed ARNs and store
them opaquely; the user path was missed when that guard was introduced.
ListAttachedUserPolicies had the mirror of the same gap: it resolved every
ARN through getPolicyByARN and silently dropped the ones it could not find,
so a managed ARN would disappear from the listing rather than round-trip.
Evaluation already handled these correctly via resolveAttachedPolicy, so
only the write and list paths needed the guard.
* test(integration): port remaining 4 ECR authorization tests
Ports TestECRIAMAuthorization_PullOnly, _ReadOnly,
_DetachPolicyDeniesImmediately, and _DeactivatedKeyReturns401 from
tests/e2e/ecr now that AttachUserPolicy accepts AWS-managed policy ARNs.
Each was proven equivalent by mutation before its E2E original was
removed: PullOnly/ReadOnly/DetachPolicyDeniesImmediately fail when
evaluatePrincipalPolicy is forced to always allow, and
DeactivatedKeyReturns401 fails when the access-key-active check in
resolveECRLongLivedPrincipal is bypassed. All mutants were reverted and
a clean run confirmed before committing.
tests/e2e/ecr/iam_authorization_test.go is now fully removed — every
test it held has an in-process equivalent, and its helpers had no other
callers in the package.
* ci: select e2e suites by allow-list instead of deny-list
Both jobs filtered SUITES by exclusion, so any suite named in neither
deny-list ran in both jobs. That is how lb came to run twice per e2e run,
and it makes adding a suite silently fan it out everywhere.
Each job now names the suites it is eligible to run. Selection is still
driven by SUITES, so this is behaviour-preserving for the current default:
single/iam/cert on the single-node job, multinode/lb/cert on the multi-node
job, exactly as before. eks, ecs and storagegrowth are listed as eligible on
the single-node job but are not yet in the default SUITES.
* test(e2e): delete seven tests with no coverage
TestFinalClusterStats and TestInstanceLaunch assert nothing beyond
what other callers of their shared helpers already imply. The
multinode baseline, volume, guest-SSH, and preflight tests duplicate
single-node coverage or exercise per-guest behaviour that isn't
cluster-specific. TestMultinodeInstanceDistribution can't fail: three
independent RunInstances calls with no placement group make a strict
spread assertion flaky and a loose one meaningless.
Preflight checks (/dev/kvm writability, peer SSH reachability) move
into requireMultiNodeFixture so the package fixture singleton still
fails fast with a clear message instead of disappearing along with
the dedicated test. The remaining callers of needInstanceTrio
(CrossNodeOps, CrossNodeGateway, NodeFailure, VolumeDurability) still
trigger the shared trio launch on their own, so removing the
distribution and volume-lifecycle tests doesn't leave it unbuilt.
Also slims the manifest-lint baseline entry for the deleted
multinode volume test.
* fix(e2e): require full 3-node spread in TestMultinodeSpread
SpreadPlacement passed at unique>=3, warned via t.Logf at unique==2,
and only failed below that. A 33% spread failure (two instances
colocated) passed with a log line. The 243.6s NAT-egress scenario that
follows only proves what it claims to if the private trio actually
landed on three distinct chassis, so that premise needs to be a hard
assertion, not a warning.
* refactor(e2e): drop the unused spread-assertion harness helpers
AssertSpreadAcrossNodes and CountInstancesPerNode lost their only caller when
the instance-distribution test was removed. Beyond being dead, the assertion
helper documented minNodes=1 as an acceptable logging-only mode inherited from
the bash driver, which is the pattern that let a placement test pass while
every instance landed on one node.
InstanceHostingNode and its ps helpers stay; five multinode tests use them.
* fix(ec2): stop leaked goroutines on prepareEFIVolume error paths
newViperblock starts the chunk uploader and WAL syncer unconditionally,
but prepareEFIVolume only released them on the happy path via Close().
Six error returns between acquisition and Close left both goroutines
running. Add a defer immediately after acquisition, matching the
sibling prepareRootVolume; Close() remains the durability boundary.
* test(predastore): add real predastore fixture for integration tests
Extract predastore_integration_test.go's inline daemon startup into a
reusable testutil/predastore package so tests/integration can point
DaemonLite-style wiring at a real S3-compatible backend instead of an
unreachable host. Kept as its own leaf package rather than folding into
the shared testutil package: it transitively imports predastore's
quicclient, whose package-level connection pool starts a permanent
goroutine at init that would trip goleak checks in any of the ~80
other packages that import testutil for unrelated NATS helpers.
* test(integration): migrate CreateVolume to assert real predastore persistence
Add StartVolumeDaemonLite, an opt-in DaemonLite variant wired to the
real predastore fixture, and migrate TestCreateVolume_PersistsToRealPredastore
off memory-backed DaemonLite so CreateVolume/DescribeVolumes exercise
viperblock's real S3 backend end to end instead of stopping at input
validation.
* test(integration): add real storage-state oracle for CreateVolume
DescribeVolumes only proves the handler's own read path agrees with its
own write path, both via the same deserialization; it cannot catch a bug
in a field it never re-derives (e.g. a unit-conversion error in the bytes
handed to viperblock, since VolumeMetadata.SizeGiB is copied from the
request directly). vbscan reads a volume's persisted config.json and
chunk objects straight over S3, independent of any spinifex handler code,
so TestCreateVolume_PersistsToRealPredastore can assert on what viperblock
actually wrote rather than only on what the handler claims.
vbscan is a local reimplementation of scripts/vbrefscan's read-only scan
rather than an import: spinifex cannot depend on mulga, the module that
contains it. It only reimplements the config/chunk-listing half of
vbrefscan — the checkpoint-based reference-integrity half needs
viperblock.ParseBlockCheckpointBytes, which does not exist in the
viperblock version spinifex currently depends on (v1.12.1) and is also
moot for this volume: CreateVolume never calls SaveLiveCheckpoint, so a
freshly created volume has no checkpoint to scan regardless of version.
* refactor(test): move the predastore daemon fixture out of the coverage scope
test-cover runs go test ./spinifex/..., so anything under spinifex/ is
compiled and measured. The daemon fixture cannot be exercised by unit
tests, so it reported a coverage gap no unit test could close. The e2e
harness already avoids this by living under tests/; the fixture now
does the same. vbscan stays put, being ordinary library code with its
own unit tests.
* test(integration): migrate RunInstances multi-count expansion off live guests
RunInstances with MinCount/MaxCount=2 expands into a single per-node
targeted launch (distributeInstances in gateway/ec2/instance/placement.go)
before the daemon-facing NATS hop, so the real gateway router plus a
stubbed daemon response is sufficient to prove it — no throwaway VM is
needed to exercise the expansion logic itself.
The live single-node E2E test's WaitForInstanceState/TerminateInstances
sequence was teardown hygiene for the real guests it booted, not an
assertion of the expansion behavior, so it is dropped rather than
emulated. The migrated test additionally asserts the per-node launch
request itself carries MinCount=MaxCount=2, proving the count actually
propagates rather than the launch succeeding by coincidence or via the
zero-daemon degenerate path in placement.go.
* docs: drop a doc reference to the deleted live test
* test(integration): migrate launch-template RunInstances expansion off live guests
RunInstances --launch-template resolves ImageId/InstanceType from the
stored template entirely gateway-side (expandLaunchTemplate, before
validation/placement/the daemon-facing NATS hop), so proving it forwards
the resolved fields never needed a real guest boot. Ports the assertion
to tests/integration via a new opt-in StartLaunchTemplateDaemonLite (real
LaunchTemplateServiceImpl over embedded JetStream) plus a direct
subscription on the per-node launch subject to capture what the gateway
actually dispatched, rather than only the client-visible response.
The live test keeps its CRUD/versioning lifecycle (create/modify/delete
version, tag-filtered describe) — none of it touches a guest — and drops
only the RunInstances-from-template + wait-for-running + terminate tail,
whose only job was waiting out a real boot to read back fields already
proven resolved.
* test(integration): migrate KV-only account-scoping checks off live tenants
Ports the KeyPair/VPC-Subnet/IGW-EIGW/AccountSettings cross-account
isolation assertions from tests/e2e/single/account_test.go to the
in-process integration tier. Each of these resources resolves ownership
from a plain account-scoped KV key (utils.AccountKey(accountID, id), or
an account-keyed settings record) with no vm.Manager or real guest
involved, so DaemonLite now also wires the real EgressOnlyIGWServiceImpl
and AccountSettingsServiceImpl alongside the existing VPC/IGW/Key
services. Two independently minted tenant accounts (via
IAMService.CreateAccount/CreateUser/CreateAccessKey/CreatePolicy/
AttachUserPolicy, in-process — the same calls runAccountCreate makes
without the CLI/stdout round-trip) replace the live tier's carousel so
isolation is checked between real, separately owned accounts.
Left live: instance scoping (Step2) and volume scoping (Step3) —
checkInstanceOwnership, the actual cross-account gate for stop/start/
reboot/terminate/attach-volume, runs daemon-side only after a real
vm.Manager lookup, so it cannot be exercised without a live guest.
Snapshot scoping (Step5) needs a real backing volume for CreateSnapshot
(viperblock, out of DaemonLite's scope), so it stays live entirely rather
than split off only its two KV-only assertions. Global-resources (Step9)
carries no account branching in the handlers at all, so asserting its
equality across two accounts wouldn't exercise anything a fake node
responder didn't already trivially satisfy — not migrated.
* test(integration): migrate launch-template CRUD lifecycle off live guests
TestLaunchTemplates booted zero guests: its RunInstances coverage had
already moved to TestRunInstances_LaunchTemplateExpansion, leaving only
template CRUD/versioning assertions occupying a full live-cluster slot.
Port the remaining create/describe/version/modify/delete lifecycle,
including the SourceVersion merge and a nested timestamp round-trip, into
TestLaunchTemplates_CRUDLifecycle against the in-process integration
gateway.
* refactor(tests): move spot instance lifecycle coverage to integration tier
RequestSpotInstances is a mock over on-demand RunInstances (no bidding,
interruption, or reclamation), so booting real VMs in the e2e tier proved
nothing spot-specific. Migrate the SIR CRUD lifecycle (request, describe,
cancel, close-on-terminate, insufficient-capacity error) to a NATS-level
integration test that exercises the real gateway orchestration and daemon
service impl without a live guest. Real-VM-boot assertions are already
covered by TestRunInstances_MultiCountExpansion and the e2e baseline tests.
* fix(tests): share one embedded NATS server across the integration package
ClientTokenStore in spinifex/gateway/ec2/instance is a process-wide
sync.Once singleton bound to whichever *nats.Conn first initialises it.
Every integration test booted its own embedded NATS server, so the first
RunInstances-calling test captured the singleton, and any later test
inherited a KV handle bound to that server's now-closed connection.
TestMain now boots one embedded NATS+JetStream server for the whole
package; every StartGateway call connects into its own dynamically
provisioned NATS account (accountAuthenticator in shared_nats.go), giving
each test a private JetStream namespace despite the shared server, and
none of the per-test connections are closed until every test has finished
- whichever one the singleton latches onto stays live for the run.
Also wire the suite into the required PR gate: it was previously only
invoked from e2e.yml's push-triggered (main/dev), self-hosted "Integration
(front gate)" job, so a regression here never blocked a PR. The whole
package runs in single-digit seconds, so it now runs inline in
`make preflight` and as its own step in spinifex.yml's build_and_test job.
* test(integration): migrate serial-console access checks off live guests
Enable/Disable/GetSerialConsoleAccessEnabled is pure gateway+account-settings
logic with no guest involved, so it moves to the in-process integration tier
against the real AccountSettingsServiceImpl wired into DaemonLite.
* test(integration): migrate DescribeImages round-trip off live guests
runImage's AMI describe round-trip needs no real cluster: it registers an
AMI against a hand-seeded snapshot via the real ImageServiceImpl, backed by
a new StartImageDaemonLite helper wiring RegisterImage/DescribeImages onto an
in-memory object store.
* test(integration): migrate predastore object lifecycle off live guests
runPredastoreObjectLifecycle only needs a real predastore daemon, not a
tofu-provisioned cluster: it now runs against tests/fixtures/predastore's
in-process fixture, covering the same create/put/get/delete/list contract.
* test(integration): migrate ListFoundationModels catalog check off live cluster
ListFoundationModels is pure gateway-side static catalog logic with no
NATS/daemon hop, so it moves to the integration tier via a new BedrockClient
harness accessor; this tier configures no credential resolver, so the
Anthropic entry's absence is asserted explicitly rather than left optional.
* test(integration): cover RunInstances client-token idempotency
Proves replaying the same ClientToken through the real gateway returns the original reservation without a second daemon dispatch, and a different token launches independently.
* test(integration): cover RunInstances launch-template expansion
Proves version selection (omitted Version resolves to the template default, explicit Version selects that version) and field precedence (explicit RunInstances parameters override the template, unset fields inherit it) through the real gateway.
* test(integration): cover RunInstances malformed-input rejection
Proves a malformed ImageId and a malformed targeted-launch CapacityReservationId are rejected before any NATS dispatch, with no daemon stub registered so a regression that let either through would fail fast instead of hanging.
* test(integration): cover RunInstances iam:PassRole enforcement
Proves a caller granted ec2:RunInstances but not iam:PassRole on an instance profile's role is denied, and the identical session succeeds once PassRole is granted for that exact role ARN.
* test(integration): cover RunInstances placement-group strategy routing
Adds a placement-group daemon-lite wiring a real PlacementGroupServiceImpl over NATS, then proves spread strategy assigns exactly one instance per node while cluster strategy pins the whole launch to the single highest-capacity node.
* test(integration): cover RunInstances capacity distribution across nodes
Proves the default (non-placement-group) path packs a multi-instance launch across nodes proportionally to spare capacity rather than exhausting one node first.
* test(e2e): merge SameSGComms and SGToSGDatapath around one client/target pair
Both booted their own instance pair to check SG-to-SG datapath enforcement
and same-default-SG east-west connectivity. Merging them around one shared
client/target pair cuts four boots to two, and converting the inline
harness.Step sequence into named t.Run stages (PortGroupMembership,
AllowedTraffic, DeniedTraffic, RevokeAndVerifyBlocked,
ReauthorizeAndVerifyRestored, SameSGComms) gives each its own pass/fail
attribution instead of one opaque test result.
SameSGComms now runs last and joins the default SG onto each ENI via
ModifyNetworkInterfaceAttribute only after every SG-to-SG allow/deny
assertion has already recorded its own result, since the default SG's
self-reference rule admits all traffic between co-members and would
otherwise defeat the denied-traffic check.
* test(e2e): merge default-SG reachability, guest DNS, and egress enforcement
DefaultSGReachabilityBaseline, GuestDNSResolution, and SecurityGroupEgress
each booted their own instance to check one facet of "a guest is reachable
only when policy says so": ingress deny/authorize, resolver + WAN egress,
and egress ACL enforcement. Merging them around one shared guest cuts
three boots to one, with named t.Run stages (Deny, Authorize, Resolver,
WANEgress, RevokeEgress/RestoreEgress) giving each its own attribution.
The egress revoke/restore round-trip now mutates this scenario's own
dedicated SG instead of the shared default SG the rest of the suite
depends on, isolating its egress-rule mutation from every other test.
* fix(ci): single-quote e2e test_run regex across the ssh boundary
* fix(e2e): anchor SG egress-enforcement probe to a WAN IP, not the gateway
* test(e2e): merge single-node guest churn tests into one durability scenario
Combines TestSSHProbe, TestENIHotplug, TestENIHotplugReconcile,
TestVolumeDurability, TestModifyInstanceAttribute, and TestRebootInstance
into TestGuestChurnDurability, a single shared-guest scenario that
re-verifies a data-integrity sentinel sha256 after every perturbation
stage (ENI hotplug, daemon restart, ENI detach, instance-type resize,
stop/start, reboot). This proves the sentinel survives the full churn
sequence rather than just one isolated stop/start.
* fix(vpc): clear subnet drop gate when installing egress reroute
A subnet with an attached IGW but no 0.0.0.0/0 route carries a
priority-1100 drop gate on the VPC router. When a route later lands
(NATGW reroute at 900, IGW reroute at 1000) the reroute is installed but
the drop gate was never removed, so the higher-priority gate wins and
black-holes the subnet's egress in lr_in_policy before it reaches routing
or SNAT. This left NAT-gateway private-subnet egress with 100% packet
loss: the SNAT stage saw zero packets and no conntrack was committed.
Remove the subnet's drop gate for the same prefix once a reroute is
installed. Idempotent and prefix-scoped, so a specific-prefix reroute is
a no-op and leaves the default gate intact.
* test(e2e): merge VPC subnet, NAT gateway, and EIP single-node tests
Merges the former TestVPCSubnetE2E, TestNewVPCEgressBaseline,
TestNATGateway, and TestInstanceEIP into one staged scenario,
TestVPCEgressPaths, around one scenario-owned VPC and its two guests,
cutting five guest boots to three.
The public guest serves PublicSubnetEgress, then the NAT Gateway
bastion hop, then (last, since association rewrites its public IP)
the EIP flip. The private guest serves the NAT Gateway path, launched
directly into the same VPC's already-existing private subnet instead
of a dedicated one. RouteBeforeSubnet keeps its own second VPC and
instance since it exercises the implicit main-route-table code path,
a different one than the explicit route table this scenario's own VPC
uses for its public subnet — collapsing the two would drop coverage of
whichever path got merged away.
The NAT Gateway up/down flip now repeats for an env-tunable round
count (SPINIFEX_VPCEGRESS_NAT_ROUNDS, default 1), matching every other
merged single-node scenario.
* test: TEMPORARY mutation-check — revert a795c2b2 drop-gate clear
Reverts the RemoveSubnetEgressDrop call in ensureSubnetEgressAtPriority
to confirm TestVPCEgressPaths/Round1/NATGatewayUp actually detects the
regression it guards against. This commit will be reverted immediately
after the live run confirms a failure.
* test: TEMPORARY mutation-check v2 — neutralize RemoveSubnetEgressDrop
Restores the real a795c2b2 fix in ensureSubnetEgressAtPriority (the
first mutation attempt only reverted this call site and was masked by
the independent gate-recompute path in routetable service_impl.go's
CreateRoute, which also clears the drop gate on every 0.0.0.0/0 route
add and predates a795c2b2). Neutralizing RemoveSubnetEgressDrop itself
disables gate-clearing regardless of caller, reproducing the literal
regression. Will be reverted immediately after live verification.
* Revert "test: TEMPORARY mutation-check v2 — neutralize RemoveSubnetEgressDrop"
Mutation-check confirmed: neutralizing RemoveSubnetEgressDrop reproduced
a795c2b2's regression exactly (stale priority-1100 drop gate outranking
the priority-1000 IGW reroute, confirmed via OVN flow dump; PublicSubnetEgress
failed with an SSH handshake timeout to the public instance). Restoring
the real fix; igw.go is now byte-identical to a795c2b2.
* fix(e2e): stop pruning shared multinode SSH key pem mid-run
The multinode package fixture cached Fixture.Artifacts once, computed
from harness.ArtifactDir(t, env) against whichever top-level Test*
happened to win the pkgFixOnce race. That helper both names the
directory after that one test and registers a t.Cleanup that removes
it once that test passes. needKeyPair wrote the shared EC2 key pair's
pem into that same directory, so as soon as the winning test finished,
its own artifact-dir cleanup deleted the pem every later test in the
package still needed to SSH into fresh guests, even though the
underlying EC2 key pair itself was untouched (its DeleteKeyPair
cleanup already correctly drains at process Close() in TestMain).
Fix: give Fixture an ArtifactDir(t) method — the same per-call pattern
already used by the gpu/single/storagegrowth/iam fixtures — so each
diagnostic dump site recomputes its own dir against the CURRENT test.
The shared key pair pem now lives under Fixture.TmpDir, which is
genuinely package-scoped and only removed once in TestMain after
m.Run() completes.
* refactor(e2e): parallelize LB e2e scenarios on multinode clusters
Each of the 8 TestLoadBalancer scenarios already stands up its own
dedicated LB + target group (ModifyListener/ListenerRules mutate their
LB's listener in place, so it can never be shared). The bottleneck was
running them serially: each teardown made createActiveLB back off
waiting for the previous LB's sys.micro slot to be reclaimed before the
next one could stand up.
On multinode clusters (peer != "", i.e. the 3-node cluster this suite
targets) there's spare capacity to stand up all 8 dedicated LBs at once,
so scenarios now run as t.Parallel() subtests instead of serial
create/teardown/backoff. Since each scenario owns an independent LB,
one never reaching active only fails that scenario — it no longer
cascades into five sequential 5-minute timeouts, so the serial fail-fast
gate is skipped in this mode; concurrency itself bounds the worst case
to a single timeout window.
Single-node dev boxes (no peer) keep the original serial path,
including the fail-fast gate, since they lack the spare capacity for
concurrent LB standup. No subtest body, assertion, or call signature
changed — only the dispatch/gating logic in TestLoadBalancer.
* fix(ci): gate e2e jobs on the suites they actually run
* fix(e2e): serialize internet-facing LB subtests among themselves
Live multinode verification showed InternetFacing_ALB and
InternetFacing_NLB running concurrently each clobber the other's
public-LB DNS record mid-registration, so neither ever resolves within
the 90s wait. The old serial suite never had two internet-facing LBs
alive at once, so this never surfaced.
Drop t.Parallel() from the three internet-facing subtests so their
register+resolve windows never overlap each other, while leaving the
internal subtests parallel as before. They may still run alongside the
internal group, which resolves internal (not public) DNS and isn't
affected.
* fix(e2e): give each ALB LB scenario its own security group
Concurrent internal subtests all created ALBs with no explicit
SecurityGroups, so each ALB's ENI fell back to the one shared VPC
default SG. One scenario authorizing/reconfiguring its own listener
racing another mutating the same SG made Internal_ALB_ModifyListener
intermittently fail to serve on the modified port under parallel load.
Each ALB scenario now creates a dedicated SG (createScenarioSG),
authorized only for the ports that scenario's listener(s) use, and
passes it to CreateLoadBalancer. The shared app-instance/probe-client
SG is untouched, since SG ingress is enforced at the destination ENI
regardless of the source ENI's SG. NLBs are unaffected: they already
get an auto-managed per-LB SG when none is supplied.
* test(e2e): annotate intentional create-path calls for manifest-lint
The vpcegress and per-scenario LB security-group tests create raw AWS
resources on purpose — the egress datapath, the route-before-subnet
ordering, and per-scenario SG isolation are the subjects under test, so
they cannot route through a shared Ensure* fixture. Mark each with
// e2e:allow-create and slim the baseline of entries for the removed
single-node network tests.
* test(e2e): restore live spot terminate-close and cancel-running coverage
The integration-tier spot test calls CloseForInstance directly and infers
cancel-keeps-running from the SIR's InstanceId, because DaemonLite has no
vm.Manager to drive the real TerminateInstances -> daemon teardown cleaner
-> RemoveFromSpotRequest -> CloseForInstance chain or to observe a real
VM's running state. Add a thin live e2e test that drives both terminal
transitions through a real backing VM: cancel one spot-backed instance and
observe it stays running, terminate the other and observe its SIR reach
closed through the genuine async delegation.
* fix(network): refresh orphan-EIP prune liveness to spare mid-pass launches
pruneOrphanEIPs listed OVN NAT rows live but matched them against the intent
snapshot captured at the start of the reconcile pass. Because the apply phase
can block for tens of seconds on guest-port datapath waits, a guest launched
during the pass had a live dnat_and_snat row absent from the stale snapshot, so
the prune swept it and blackholed the guest's auto-assigned public IP until the
next pass re-added it — an intermittent multinode SSH-dark flake.
Add an optional FreshIntent re-read callback (wired in vpcd to LoadIntentFromKV)
and union its ports into the live set before pruning, so a mid-pass launch
counts as live. Skip the prune entirely when the re-read fails rather than risk
a false sweep against a snapshot known to be stale. Self-healing and idempotent:
a genuine orphan absent from both snapshot and fresh read is still pruned next
drift. Single-node behaviour is unchanged (callback nil in tests).
* feat: plumb GCEnabled config through to every VB construction site
viperblock chunk GC is gated behind VB.GCEnabled, but spinifex never
set it, so every deployed node ran with GC off by construction
regardless of which viperblock binary shipped. Add a
[viperblock] gc_enabled config key (default false, mirrors shardwal)
and thread it through to the three places spinifex builds a VB: the
nbdkit plugin it spawns per mounted volume (via a new gc_enabled
plugin param), viperblockd's detached-volume opens, and the volume
service's CreateVolume.
* chore: repin viperblock to GC-enable lineage
The gc-enable plumbing references viperblock.VB.GCEnabled, which is
absent from the previously pinned v1.12.1. Repin to the feat/nbd-gc-enable
commit so a standalone (GOWORK=off) build — the path the ansible deploy
uses — resolves the field, and spx plus the nbdkit plugin build from the
same GC-bearing viperblock lineage.
* fix(storagegrowth): stop launching a nano for image-pull repro
ensureImagePullInstance shared harness.DiscoverNanoInstanceType with the
synthetic npass suite. A 512MB guest OOM-thrashes containerd before any
disk-driven ENOSPC and conflates the storage leak with guest-memory
pressure. Add a package-local imagePullInstanceType helper: an exact
override via SPINIFEX_STORAGEGROWTH_INSTANCE_TYPE, otherwise the smallest
advertised type at or above SPINIFEX_STORAGEGROWTH_MIN_MEM_MIB (default
16384 MiB). npass keeps using the nano selector unchanged.
* feat: report guest block write errors so backend ENOSPC reaches the guest
Set werror=report and rerror=report on the guest boot drive so a viperblock
ENOSPC (or any other backend write failure) is surfaced to the guest as a
real I/O error instead of QEMU silently pausing the VM. Without this the
guest kernel never learns its disk is out of space, so it can't fail
writes cleanly or let an operator react.
* chore: repin predastore and viperblock to the storage-capacity-safety lineage
Drop the local-path replace for predastore (a test shortcut) and pin both
predastore and viperblock to their feat/storage-capacity-safety commits, so
a standalone (GOWORK=off) build resolves the free-space watermark, ENOSPC
classification, and backpressure plumbing that spinifex's werror/rerror
change depends on.
* chore: bump viperblock to 054a44a3f0f5
Realigns spx's linked viperblock with the submodule working tree, which the
build role's source-drift gate compares against.
* chore: repin predastore and viperblock to their dev lineage
Both were pinned to pseudo-versions off feat/storage-capacity-safety. That work
has now landed on dev in each repo, and both PRs were squash-merged, so the
branch commits those pins named are unreachable from any branch.
Repin to the dev commits carrying the same code: predastore 5699329 and
viperblock 35dd9f2.
Northstar carried working northstar.dns.* instruments but shipped no
telemetry, so its dashboard was empty and the only northstar data in
Elasticsearch was journald scraped under the systemd unit name.
Two gaps, both outside the instrumentation. northstarStartCmd never called
initTelemetry, unlike every other service command, so the instruments
recorded into a no-op provider. The unit also did not source
/etc/spinifex/telemetry.env, which every other application unit reads for
OTEL_EXPORTER_OTLP_ENDPOINT, MULGA_ENV and MULGA_SOURCE — so even an
initialised SDK would have had no endpoint and no env label.
Add an invariant covering the second gap: any unit launching an spx service
must source the telemetry drop-in. A service missing it starts healthy and
silently reports nothing, which only surfaces when someone notices an empty
dashboard. Verified the test fails against the unpatched unit.
Add e2e_es_summary, a self-hosted job that aggregates suite-status.txt
from both e2e legs and POSTs one doc per run to the Elastic sink, keyed
by labels.ci_run_id, so Kibana can answer "did run X pass, and where
are its artifacts" without hand-aggregating traces-apm*/logs-apm.error*.
Runs on ci-single (LAN route to the sink) since e2e_status stays on
GitHub-hosted ubuntu-26.04 for the PR status-report check. State is
computed with the same single_ok/multi_ok logic as e2e_status so the
two can never disagree.
* feat(bedrock): Ochre streaming — ConverseStream + InvokeModelWithResponseStream
Adds the two streaming bedrock-runtime operations behind a shared
eventstream.Encoder frame writer: ConverseStream reframes vLLM/Anthropic SSE
into the normalized Bedrock event taxonomy (messageStart -> contentBlock*
-> messageStop -> metadata), InvokeModelWithResponseStream wraps each
provider-native chunk in a PayloadPart frame. Fixes otelsetup's
statusRecorder to implement Flusher/Unwrap so flushes actually reach the
socket through the middleware chain, which streaming depends on.
* fix(bedrock): keep vLLM ConverseStream taxonomy ordered on edge cases
Route contentBlockStop/messageStop through an idempotent ensureStarted so a
usage-only chunk arriving before any finish_reason no longer emits metadata
ahead of messageStop, and a zero-content stream no longer emits stop events
without a preceding messageStart. Regression tests for both.
* fix(bedrock): route /model/ data-plane calls to bedrock-runtime
AWS SDKs sign bedrock-runtime with the SigV4 signing name "bedrock", the
same as the control plane; the two services are separated only by endpoint
hostname. The gateway serves one endpoint and routed purely on the signing
name, so every /model/{id}/converse*|invoke* call landed in Bedrock_Request
(control plane) and returned InvalidAction — no standard SDK/CLI could reach
any bedrock-runtime endpoint.
Disambiguate by path in GetService: a "bedrock"-scoped request to a /model/
path (data-plane-exclusive) resolves to bedrock-runtime. Control-plane paths
(/foundation-models, /custom-models, /model-invocation-job) are unaffected.
* test(bedrock): satisfy testifylint go-require/empty in streaming tests
Use assert (guarded where a value is dereferenced afterward) instead of
require inside httptest handler closures, and assert.Empty for empty-string
checks, per dev's stricter testifylint config.
The Build/Lint/Race jobs checked out viperblock and predastore at a hardcoded
ref: dev and overlaid them with a go workspace, so the go.mod pins were ignored
and a spinifex change depending on an unmerged sibling change could never
compile in CI. Resolve each sibling to a branch of the same name when it exists
on that repo, else dev, mirroring the e2e path in
scripts/tofu-cluster/build-install-artifacts.sh so unit CI and e2e CI agree.
Treat the ECR registry JWT as a signed identity pointer rather than a
permission grant. Every /v2/* request now rehydrates the token's access
key against current IAM/STS state, maps the OCI route to an ecr: action
and repository ARN, and runs the identity-policy evaluator before
dispatch. PullOnly/ReadOnly principals can no longer push, overwrite
tags, or delete manifests, and key/session/user/role/account/policy
revocation takes effect on the next request instead of at token expiry.
Cross-repository blob mounts now require source-read permission and
proof the digest is referenced by a manifest in the named source repo,
closing an account-wide blob-fishing gap.
A CreateLoadBalancerSync whose data-plane launch failed returned without
rolling back the NLB managed security group. The orphaned SG sat in the EKS
managed control-plane VPC and blocked DeleteVpc with DependencyViolation
indefinitely, wedging the cluster in DELETING. The failure was also flattened
to ServerInternal, discarding the only useful diagnostic.
The sync path now unwinds fully on launch failure (ENIs, managed SG, name
claim, record) and maps the launch error onto its real cause: capacity
shortfalls pass through as their own code, anything else stays ServerInternal.
The async path deliberately keeps its failed record — the caller holds the ARN
and can reclaim it via DeleteLoadBalancer.
The image-pull storage-growth workload HEADed <volID>/checkpoints/blocks.live.bin
with the operator box's gateway identity, which has no S3 read grant on the
bucket and was refused 403 on every sample, so checkpoint_bytes read 0 across a
whole run. It now HEADs the object on the node over PeerSSH using predastore's
own [[auth]] credentials from predastore.toml (root-owned, so read via
passwordless sudo; the CA is readable through the spinifex group). The
credentials are read into shell locals, used inline for one aws s3api
head-object, then unset — never echoed, never leaving the node. A probe failure
that is not a legitimate 404 voids the sample (checkpoint_bytes -1 plus a probe
error) rather than reporting a false zero, so a dead instrument can never look
like no growth. The dead operator-side S3 client path is removed.
Align the top of the README with mulgadc.com: new headline and intro,
rewritten "What is Spinifex?", a new "Three ways to deploy" section
(Neocloud / on-premise / edge) and a new "AWS compatibility" service
matrix (EC2, EBS, S3, VPC, IAM, ALB/NLB, EKS, ECR, ECS available; RDS
Q3 2026) linking the GitHub releases. Refresh the architecture prose and
Key Features, add Deploy and AWS-services nav links, and swap banner.svg
for the ember-on-ink mini-stack design matching platform.svg.
Rebuild the message-driven architecture diagram in the ember-on-ink
theme to match platform.svg: ember gradient NATS bus, ember-tinted
gateway and daemon cards, neutral clients and backend cards. Content
unchanged (clients → gateway → NATS → ec2/ebs/s3/vpc/iam daemons →
local backends). De-slop the copy and refresh the img alt text.
Replace the platform diagram with a static SVG matching the mulgadc.com
front-page stack diagram: ember accent, ink panel, the portable-stack
box (Neocloud / on-prem / edge) wrapping the Spinifex AWS-compatible
service layer, over standard Linux on CPU/GPU hardware. Self-contained
and GitHub-safe (presentation attributes only). Refresh the img alt text.
* fix(admin): round the AMI root volume up so the image fits
`spx admin images import` sized the root volume with integer division, which
floors: a 4.94 GiB image registered a 4 GiB volume. The image is then copied
into a volume too small to hold it, is truncated, and the guest boots with no
root partition — it stalls on the root device until systemd gives up, and
nothing anywhere reports an undersized volume.
floorVolumeSizeToAMI already exists to prevent exactly this ("Under-sizing
hangs the guest in dracut"), but it raises the caller's requested size to the
AMI's recorded VolumeSizeGiB — the same floored value — so it inherited the
mistake instead of catching it.
This stayed hidden because every current system image is an exact multiple of
1 GiB, where flooring and rounding up agree. It surfaced on a 5306470400-byte
image, which would not boot.
Round up instead, via a helper so the boundaries are testable: an exact
multiple must not gain a spare GiB, and one byte over must.
* fix(eks): default a nodegroup's root volume to 20 GiB, as AWS does
CreateNodegroup stored diskSize verbatim, so omitting it persisted 0 and
the launch skipped its block device mapping entirely, dropping the worker
onto whatever volume the AMI itself declares.
That was harmless while the node AMIs baked generous virtual disks — the
GPU image declares 16 GiB for ~2 GiB of content, so the fall-through
happened to leave plenty of room. It stops being harmless as soon as an
image is sized to its contents: the worker then boots onto a volume with
barely a spare GiB and cannot pull the container images it exists to run.
Sizing the node is the nodegroup's decision either way, not a property
the image should be able to dictate by accident.
20 GiB matches the AWS EKS managed-nodegroup default. It is resolved at
create rather than at launch so DescribeNodegroup reports the size the
node actually gets.
TestCreateNodegroup_NoDiskSizeOmitsBlockDeviceMapping asserted the old
fall-through and is replaced by its inverse; the explicit-size case is
already covered by TestCreateNodegroup_DiskSizePropagatesToBlockDeviceMapping.
* fix(ecs): size a container instance's root volume to 30 GiB by default
ProvisionCapacity built its RunInstances input without a block device
mapping, so a container instance inherited whatever volume its AMI
declares. EC2 has no default of its own to fall back on — the AMI is the
only thing that decides — and a node AMI is sized to hold the image, which
leaves the agent no room for the container images it exists to pull.
30 GiB mirrors what the AWS ECS-optimized AMI declares. ProvisionCapacity
is spinifex's own API rather than an AWS one, so DiskSize is added to its
input for callers that want to size a cluster explicitly, defaulting when
omitted the same way InstanceType and Count already do.
* fix(images): pull the Ubuntu catalog images from releases, not a pinned daily
Both Ubuntu catalog entries pointed at a dated build under the codename
dailies tree, which Canonical prunes as builds age out. That path now 404s,
so "spx admin images import --name ubuntu-26.04-x86_64" fails outright for
x86_64 and arm64 alike — the entries had rotted with nothing to catch it.
Release builds are retained rather than pruned, so the pin stays resolvable
while keeping a specific, reproducible build. The filenames under releases/
carry the version instead of the codename, which matters because the sums
entry is matched on the URL's basename.
CI is unaffected and could not have caught this: it fetches resolute/current/,
a symlink that never rots, so nothing exercised the pinned path.
Debian pins a dated directory too but is fine — those builds are retained,
and its entries still resolve.
Pulls a container image inside a guest onto a viperblock volume and samples
guest used bytes, host backing-store bytes, nbdkit RSS and the live
checkpoint size as it goes, so backing-store cost can be fitted against the
bytes the guest actually wrote.
Complements the N-pass overwrite workload, which rewrites one extent and so
pins the block map size — that shape cannot show whether cost grows in
proportion to bytes written or faster than them.
Redirects both dockerd's data-root and containerd's root onto the target
volume. Redirecting data-root alone leaves containerd extracting layers into
/var/lib/containerd on the AMI's small root disk, which fails with ENOSPC
while the target volume sits empty.
The storagegrowth gate called ResolveCACert unconditionally before
building the AWS client, even though newAWSClient itself skips CA
resolution entirely when SPINIFEX_AWS_INSECURE=1. That blocked the
documented runner-resident remote path (env12 and similar clusters
with no local CA cert on disk) and the skip message falsely claimed
no node was running when the real cause was a missing CA. Skip the
CA check under SPINIFEX_AWS_INSECURE=1 and state the actual cause
plus the escape hatch in the skip message.
Rewrites one logical extent N times through full attach/format-write/detach
cycles, so the guest-visible footprint stays constant while total bytes written
scales with N. That shape isolates the leak where viperblock mints chunk objects
under unique monotonic keys and never deletes the superseded ones: a sequential
write-once supersedes nothing and so cannot exercise it.
The suite drives the workload only and deliberately does not measure the
backend. Measuring on-disk bytes from here would mean either a host du (which
cannot separate live from dead bytes, and so cannot distinguish this leak from
the unrelated one where an overwritten key's old extent is orphaned without a
tombstone) or a dependency on tooling outside this repository, which would stop
the suite building standalone. It instead reports what it did to a caller-named
result file and asserts only what a guest can observe: a sha256 round-trip
through a fresh mount per pass, and a footprint that stays constant across them.
The volume is retained on request, because deleting it removes the evidence:
DeleteVolume enumerates the bucket by prefix and drops every key under it,
including the leaked chunks.
Phase 8 of TestNATUplink checked the second VPC router's lr-nat-list with a
single non-polling assert; the igw-default-snat row commits a beat after the
gateway LRP appears, so the read raced the OVN commit and intermittently saw
an empty list (flaky CI red). Wrap it in EventuallyErr, matching the sibling
LRP and host-route polls in the same phase.
AddSNAT logged only on the idempotent-skip path, so the first install was
silent and RCA could not tell a late SNAT from a missing one. Emit an install
log keyed by router + vpc_cidr + external_ip.
Server-cert IP SANs were built solely from live interface enumeration
(DiscoverLocalIPs) plus bindIP, so the canonical mgmt-bridge IP entered
the SAN set only when br-mgmt happened to be up at cert-generation time.
On a host where br-mgmt is down when the cert is minted, 10.15.8.1 was
silently dropped and the EKS control-plane publish to https://10.15.8.1
failed cert verify, leaving the cluster stuck CREATING.
Pin config.DefaultMgmtBridgeIP into the extra SANs for both the init and
join cert paths so publish succeeds regardless of interface state at
generation time. Add a regression test asserting the SAN is present when
br-mgmt is not a live interface.
Add an IRSAPod subtest that applies a real pod wired for IRSA — a projected
ServiceAccount token (aud sts.amazonaws.com) plus the AWS_ROLE_ARN /
AWS_WEB_IDENTITY_TOKEN_FILE / AWS_ENDPOINT_URL_STS / AWS_CA_BUNDLE env the
addon-sync agent injects — and asserts the pod's in-cluster
aws sts get-caller-identity resolves to the assumed IRSA role. This closes the
gap left by IRSAWebIdentity, which only exchanges the token host-side.
Extract registerOIDCRole (shared with IRSAWebIdentity); the pod runs on a
1-node worker nodegroup (the control-plane node is CriticalAddonsOnly-tainted)
and reaches awsgw STS over the worker subnet NAT.
The build script derives BUILD_DIR from SYSTEM_IMAGE_BUILD_DIR (the AMI
publish workflow points it off tmpfs so the 16G Ubuntu GPU raw fits), but
the publish step hardcoded /tmp and could not find the freshly built raw.
GPU legs failed 'Raw image not found'; Alpine legs silently reused stale
pre-existing /tmp raws on the persistent runner. Mirror the build script's
derivation so publish looks where the build actually wrote.
The publish-eks-ami matrix fans out four legs on the single persistent
ci-single runner, all reusing one workspace root. go work init is
create-only and checkout never scrubs the root go.work, so legs after the
first fail with 'go.work already exists'. Remove any stale go.work before
init so each leg re-inits cleanly.
Expand the publish workflow to build the two Ubuntu NVIDIA GPU variants
alongside the Alpine eks-node/ecs-agent AMIs. The image build+publish
scripts are already manifest-driven and Ubuntu-aware; only the workflow
lacked the GPU legs.
Refactor the single-job body into a matrix over the four image manifests,
resolved by a new 'variant' input (all|cpu|gpu) so the slow 16G GPU legs
can be skipped when only the CPU AMIs need republishing. GPU images do not
fit a RAM-backed /tmp, so set SYSTEM_IMAGE_BUILD_DIR onto the runner's
on-disk _temp; legs run serially (max-parallel 1) on the single ci runner.
The nodegroup label was only written to K3S_NODE_LABEL in agent.env, but
k3s has no such env var, so non-GPU workers never received the label.
waitWorkersReady buckets Ready nodes by that label (added in #511), so
the affected nodegroups never counted toward their ACTIVE transition and
stayed CREATING until timeout.
Ride the label on a real k3s node-label config drop-in
(15-nodegroup-label.yaml), delivered via cloud-init user-data. Drop the
inert K3S_NODE_LABEL from agent.env and the initd export.
WaitForInstanceState now aborts immediately with the recorded StateReason
message when it observes a reaped state (shutting-down/terminated) while
waiting for a live target, instead of burning the full 5-minute timeout on
an instance that can never reach it.
MarkFailed takes a context and records the failure on the active span via
span.RecordError so a reaped launch surfaces in the APM error stream,
correlated by trace id. Only the LaunchRunInstances path carries a live
span; other callers pass context.Background() and the record is a no-op.
Surfaced by RCA of E2E run 29258398516 (TestAccountScoping/Step2).
TestNATGateway flaked with 100% egress loss: the tenant VPC gateway router's
OVN MAC_Binding had no entry for its upstream default-route nexthop. OVN
resolves that nexthop via lazy dynamic ARP on first egress; if that ARP is
lost during gateway bring-up, SNAT'd packets die in lr_in_arp_resolve and no
reply returns. MAC_Binding is per (datapath, logical_port, ip), so a sibling
VPC router resolving the same nexthop does not help.
Deterministically seed a static MAC binding for the nexthop at IGW attach. The
nexthop MAC is the host's own default-gateway MAC (the host shares the physical
uplink), read from ip neigh and written via ovn-nbctl static-mac-binding-add
(northd programs it into SB MAC_Binding, GC-safe). Host layer L0 owns the
shell-out per ADR-0006 S2; igw calls it via an injected hook. Best-effort and
idempotent: a missing MAC leaves dynamic ARP as the fallback, and it re-runs on
every AttachIGW so a binding lost to a later flap self-heals.
utils.GetOrCreateKVBucket hardcoded Replicas=1, so the ~38 handlers that
create their KV buckets lazily (VPC/subnet/ENI/EIP/ELBv2/ECS/ECR/ACM/
natgw/snapshot/placementgroup/IPAM/account-settings/dhcp/static-pool …)
created them at R1 on a multi-node cluster. upgradeJetStreamReplicas runs
once at startup, so any bucket created lazily after boot stayed R1
permanently — and an R1 stream's single Raft replica is transiently
leaderless during formation/join/rebalance, surfacing as
ErrNoResponders/ErrTimeout on KV ops.
Add a package-level default replica count that the daemon sets once at
boot (startCluster, after initJetStream) to len(clusterConfig.Nodes),
before any handler creates a bucket. GetOrCreateKVBucket now creates at
that default, so lazily-created buckets are born quorate. The one-shot
startup UpdateReplicas remains as a safety net for pre-existing R1
buckets from older deploys.
Zero call-site churn: the default flows through the existing
GetOrCreateKVBucketWithReplicas path already used by eks/ecs/iam.
TestSpawnRegisteredReconcilers_ResumesNonTerminal flaked two ways. First,
the shared fixture meta stamps a weeks-old CreatedAt, so a resumed CREATING
reconciler instantly trips the CREATE timeout, marks the cluster FAILED and
deregisters before the registry.Has assertion runs — the test was racing a
self-removal. Stamp a recent CreatedAt so the reconciler stays CREATING.
Second, SpawnRegisteredReconcilers ran a one-shot JetStream enumeration
(KeyValueStoreNames + kv.Keys), which is eventually consistent: under load a
just-committed cluster can be missing and never resumed, silently on a real
daemon. Re-scan until the observed CREATING/ACTIVE count stops growing, capped
and short-backed-off; Spawn is idempotent so re-runs only pick up stragglers.
discoverSingleNodeIPs seeds 127.0.0.1 then appends net.InterfaceAddrs()
in undefined order, so WANHost=nodeIPs[0] was loopback and the guest
gateway URL helpers fell through to the first non-loopback interface
addr — an arbitrary co-tenant bridge on multi-bridge shared hosts
(wattle: br-lan-env1..23, br-wan), leaving guest VMs unable to reach the
advertised gateway to register.
Resolve WANHost from spinifex.toml nodes.<node>.advertise (falling back
to that node's host), mirroring awsgwBindIP's toml scan; fall back to the
first non-loopback discovered addr only when config is unreadable. This
removes the need for the SPINIFEX_WAN_IP override on shared hosts.
The crash-supervisor auto-restart path (qmpHeartbeat -> HandleCrash ->
MaybeRestart -> startQEMU -> newQMPClientWithHandshake -> dialQMPWithRetry)
already unlinks the stale QMP socket and retries a refused/absent connect. But
a freshly relaunched QEMU can accept the connect then reset the connection
mid-greeting while it initialises, surfacing 'connection reset by peer'
immediately after 'QEMU started successfully' as a ServerInternal.
Add ECONNRESET and EPIPE to isTransientDialError so the dial is retried until
QEMU services the monitor, closing the auto-restart QMP-connect race.
Add non-streaming InvokeModel to bedrock-runtime. Unlike Converse's unified
schema, InvokeModel passes model-provider-native bodies through, so route by
model family: Anthropic Claude bodies forward to the Messages API (drop the
Bedrock-only anthropic_version, inject model, return verbatim); Meta Llama
bodies translate to/from the self-host vLLM OpenAI completions endpoint.
Raw-body dispatch bypasses the JSON response path (WriteRawResponse). Reuses
the Phase 1 credential + endpoint resolvers and upstream-status mapping.
Unit-tested (routing, body rewrite, translation, error classes) plus gated
InvokeModel E2E.
Both services already run otelsetup.Init and ship logs/metrics but never
opened a span, so they were invisible in the Kibana APM service map.
qmp-collector now opens a root span per per-instance poll tick (marks
error on QMP collect/publish failure); spinifex-ui now wraps its HTTP
handler chain with otelsetup.HTTPMiddleware and its reverse-proxy
transport with otelhttp, so UI requests get a server span and proxied
calls to awsgw/predastore carry traceparent instead of stopping the
trace at the UI hop.
nats and eks are intentionally left uninstrumented in this pass: nats is
the vendored message-broker process with no message-level hook to attach
spans to (the traffic it carries is already traced end-to-end at its
producer and consumer endpoints); eks request/response calls already
produce spans today under service.name=spinifex-daemon since the
handlers run in that process, and its async reconciler/state-machine
goroutines need a separate span design across a much larger surface.
Skip ERROR log when a subscription is already invalid
(nats.ErrBadSubscription) during graceful shutdown; log at debug
instead. Cosmetic cleanup race, no functional impact.
golangci-lint canonicalheader flagged 6 lowercase header literals in the
Ochre/Bedrock gateway (anthropic.go, anthropic_test.go, vllm.go). Canonical
form is wire-equivalent; canonicalise to X-Api-Key, Anthropic-Version and
Content-Type to restore CI lint.
* feat(bedrock): Ochre runtime MVP — Converse over self-host + Anthropic-direct
Add the AWS-Bedrock-compatible gateway services (bedrock, bedrock-runtime)
following the EKS REST-JSON pattern: ListFoundationModels / GetFoundationModel
over a static connectivity-tiered catalog, and non-streaming Converse routed
per modelId to one of two provider backends — self-hosted vLLM (OpenAI wire)
and Anthropic-direct (Anthropic Messages wire).
Per-account provider API keys are a master-key-encrypted JetStream KV secret
with an optional platform default; the advertised catalog gates external
models on a resolvable credential. Self-host endpoints are pinned via config
(OCHRE_VLLM_ENDPOINTS) behind an EndpointResolver seam.
Adds gateway/bedrock/ (catalog, provider router, Anthropic + vLLM adapters,
credential store), awserrors REST-JSON codes, and the bedrock E2E suite
(list/get light; Converse gated on live backends). Daemon/NATS control plane
and dynamic endpoint lifecycle are deferred to Phase 3.
* test(bedrock): cover dispatch, router, credentials KV + endpoint config
Add unit coverage for the Ochre Phase 1 diff: Bedrock_Request /
BedrockRuntime_Request end-to-end dispatch, provider router, JSON
response/error writers, upstream-status mapping, the credential-store KV
round-trip, the Anthropic bound-provider path, and parseBedrockEndpoints.
Lifts PR diff coverage above the 60% gate.
* ci: GPU e2e workflow on ci-gpu self-hosted runner
Runs tests/e2e/gpu (EC2/ECS/EKS GPU passthrough) against the persistent,
hand-bootstrapped wattle shared host, driven over SSH by a banksia runner
labelled ci-gpu (matches the ci-single/ci-multi convention) — no runner
installed on the shared box.
- runs-on [self-hosted, ci-gpu]; concurrency ci-gpu, cancel-in-progress
false (single A1000 = single-flight, queue not stomp)
- workflow_dispatch + GPU-path filter (not every push)
- manual-work mutex gate (~/.ci-gpu-paused / CI_GPU_PAUSED) so hands-on GPU
testing blocks CI
- if: always() cleanup terminates launched GPU instances; never resets the
box (no setup-ovn.sh/dev-reset/cluster-reset)
- drives the control plane with spx + the already-configured aws spinifex
profile directly (no wrapper scripts, nothing to install on wattle)
* ci(gpu-e2e): advisory AMI preflight, fail only if zero GPU AMIs
The suite is self-guarding — each test skips when its AMI is absent. The
hard-fail on the base ubuntu GPU AMI blocked the whole run even when the
ECS GPU AMI (the only one imported) could execute. Inventory the AMIs and
fail only when none are present.
* ci(gpu-e2e): self-provision base GPU AMI, hard-require it in preflight
The GPU E2E workflow was green-because-skipped: with no GPU AMIs imported
the self-guarding suite skipped every test, passing as a false green. Make
the raw-EC2 GPU passthrough test the always-on baseline.
- Add an import-if-absent setup step that pulls the prebuilt base GPU image
from the catalog by name (ubuntu-26.04-nvidia-gpu-x86_64), naming the AMI
exactly what tests/e2e/gpu filters for. Idempotent; caches on wattle's
persistent loop-fs. Runs spx under sg spinifex (local storage write).
- Preflight now hard-requires the base AMI (fail, not skip) so a silent
import failure can't pass as green; ECS/EKS node AMIs stay opt-in warn.
- Reconcile the local build path: ubuntu-gpu-nvidia.conf AMI_NAME and
launch-gpu.sh's nvidia default now use the same canonical name, so a
locally built+imported image also satisfies the suite.
* ci(gpu-e2e): pass --ami-name on base GPU import so the AMI name matches
spx admin images import --name selects the catalog entry but names the
resulting AMI generically (ami-<distro>-<version>-<arch>), so the import
landed as ami-ubuntu-26.04-x86_64 and the tests/e2e/gpu name filter never
matched. Pass --ami-name ubuntu-26.04-nvidia-gpu-x86_64 explicitly (same
flag build-system-image.sh uses) and poll briefly for availability.
* fix(network,vm): reconcile auto-assigned public-IP NAT + scale VFIO QMP timeout
Auto-assigned public IPs were never carried through the network reconciler.
applyEIPs ran AddEIP + ensureGuestPortDatapath for user EIPs only; auto-assigned
IPs (intent.Ports[].PublicIP) got egress exemption but no NAT re-assert and no
gatewayLRP->guest convergence. Their dnat_and_snat was created once at
RunInstances and never touched, so a vpcd/OVN rebuild dropped it and the
guest-port hop never converged: inbound DNATs at the gateway but never reaches
the guest (TCP/SSH dark) while ICMP is answered by the gateway LR (ping OK).
floatingIPSpecs now folds every auto-assigned/ELB public IP into the same
AddEIP + guest-port convergence path as user EIPs (deduped by private IP; a user
EIP wins), making the datapath self-healing across reconcile ticks and deploys.
qmpGreetingTimeout for VFIO guests now scales with guest RAM (base 30s + 8s/GiB,
floored at 180s, capped at 600s) instead of a flat 180s, so a large guest is not
SIGKILLed mid-pin while the whole guest RAM is IOMMU-locked before QMP answers.
* fix(vm): scale QMP handshake command deadline for VFIO launches
The greeting timeout scaling alone did not fix the VFIO launch SIGKILL: a GPU
guest sends its QMP greeting before the RAM pin completes, then cannot service
qmp_capabilities until the pin finishes. sendQMPCommand hardcoded a 30s response
deadline, so the handshake decode timed out (~43s for a 16 GiB g5.xlarge) and
the launch cleanup SIGKILLed a QEMU that would have come up.
Parameterize the command deadline (sendQMPCommandWithTimeout); the handshake now
uses the RAM-scaled qmpGreetingTimeout (base 30s + 8s/GiB, floor 180s, cap 600s)
for qmp_capabilities. Other callers keep the 30s default.
* test(gpu-e2e): share one GPU instance across passthrough subtests to avoid mid-suite IP recycle
Recycling both the floating and private IP onto a new ENI left the ingress
datapath targeting the terminated instance, so the successor was unreachable
on its EIP while IMDS stayed up.
- AddEIP re-points the dnat_and_snat row when the owning ENI changed even
though the external+logical IP are unchanged, keyed on the portable
spinifex:logical_port external-id (centralised mode leaves the native
LogicalPort column empty). The genuine same-ENI case still skips row churn.
- Flush the SB MAC_Binding for a private IP on ENI teardown and successor
bind so a reused address re-resolves to the live owner's MAC, via a new
MACBindingFlusher backed by ovn-sbctl.
- Reconciler prunes orphan dnat_and_snat rows whose stamped owning ENI is
absent from intent (leaked by lost vpc.delete-nat across dead VPCs).
- IMDS tap responder suspends retries for an ENI whose record is gone,
stopping the 15s reconcile churn.
* test(daemon): cover GPU inventory in GetNodes and GPU attachment in GetVMs
Adds unit tests for buildGPUInventory/handleNodeStatus (whole-GPU claimed/
free, MIG slices claimed/free, no-GPU node) and resolveVMGPU/handleNodeVMs
(whole-GPU attachment, MIG attachment, no attachment). Production code for
these paths (types.GPUInfo/GPUSliceInfo/VMGPUInfo, buildGPUInventory,
resolveVMGPU) and EC2 DescribeInstanceTypes GpuInfo.Gpus population
(instancetypes.GenerateGPUTypes/GenerateMIGTypes/IsGPUType) already landed
via the daemon GPU-admission work and were already test-covered for part C.
* feat(ui): shared GpuInstanceTypeSelect grouped instance-type picker
Reusable Select wrapper for EC2/EKS/ECS instance-type pickers: groups
Standard vs GPU-Accelerated types via structural GpuInfo.Gpus detection
(AWS SDK shape, no name heuristics), annotates GPU items with VRAM from
MemoryInfo.SizeInMiB and an optional per-type availability count, and
falls back to a single ungrouped list when no GPU types are present.
* feat(ui): EC2 GPU inventory, instance GPU card, shared picker in run-instances
* feat(ui): EKS nodegroup GPU picker + AMI/taint note
Add-nodegroup form now uses the shared GpuInstanceTypeSelect, grouping
and VRAM-annotating GPU instance types instead of a free-text field.
Selecting a GPU type surfaces a note that the GPU node AMI is
auto-selected and the nodegroup is tainted
nvidia.com/gpu=present:NoSchedule by default. Nodegroup rows badge GPU
nodegroups, derived from the primary instance type since
NodegroupRecord does not expose a GPU flag over the API.
* feat(ui): ECS GPU capacity picker, task-def GPU count, task gpuIds
* feat(eks): expose nodegroup GPU signal in Describe/ListNodegroups + UI badge
* chore(ui): rebuild embedded console dist for GPU UI
Bump github.com/mulgadc/viperblock to v1.11.1-0.20260713024608-eb4a376becc8
(dev tip through viperblock #34): bounded parallel chunk upload + coalesced
checkpoint (edou7 sec 7 items 2 and 9), sweeping the require up from #32 and
picking up the #33 write backpressure on the way. The nbdkit plugin the VM
write plane uses is built from the submodule source; this bump aligns the
viperblockd library that spx links.
* fix(daemon): gate GPU instance admission on cpu/mem and GPU-slot availability
canAllocateLocked early-returned count for whole-GPU (non-MIG) types,
bypassing the vCPU/memory admission calc and the live MemAvailable gate.
canAllocateCount already had a requiresGPU/availGPU branch, but every
production caller passed requiresGPU=false, making it dead code.
GPU instance types (whole-GPU and MIG) now flow through the same
cpu/mem + live-mem gate as every other type, with GPU-slot count wired
in as an additional hard constraint from gpuManager.Available(). This
also gives MIG a real up-front GPU-slice gate instead of only failing
late at Claim.
* fix(vm): release leaked GPU pool slot on crash-to-error and add reconciler backstop
HandleCrash freed vCPU/memory via deallocateResources but never called
InstanceCleaner.ReleaseGPU, since it bypasses the stop/terminate cleanup
chain that normally carries that call. A successful restart reuses the
still-set GPUAttachments, so that path is untouched, but when restarts
are exhausted, the instance type is unknown, there isn't enough capacity
to restart, or the relaunch itself fails, the instance settles
permanently in StateError still holding its GPU pool entry with nothing
to reclaim it.
Release the GPU and clear GPUAttachments on each of those terminal
paths. Add a GPU-pool reconciler mirroring the existing ENI reconciler,
wired into the daemon's reconcile loop, that periodically frees any pool
entry whose instance is absent or has settled into a non-running state,
as a backstop for a node-down mid-cascade or any other gap in the
synchronous release.
* fix(eks): keep derived ELBv2 names under 32 chars and surface CREATE_FAILED reason
Long cluster names made the derived "eks-<name>[-cp|-konn]" NLB/target-group
names exceed the ELBv2 32-char limit, failing cluster creation asynchronously
with no visible cause. safeELBv2Name now deterministically truncates the name
and appends a stable hash of the full cluster name when needed, so the same
name is derived on every call (create, lookup, delete). DescribeCluster also
now surfaces ClusterMeta.StatusReason as a Health.Issues entry when a cluster
is FAILED, instead of leaving callers to guess why.
* fix(eks): plumb per-nodegroup Ready-node counts through state reports
The on-VM state-report script now emits nodegroup_ready, a per-nodegroup Ready
count keyed by the eks.amazonaws.com/nodegroup label value, alongside the
existing cluster-wide node_count. ServerStateReport, the ClusterReconciler's
observe step, and SetClusterHealthState now carry this map through to
ClusterMeta.NodegroupNodeCounts, so per-nodegroup consumers (wired in the next
commit) can gate on a nodegroup's own workers instead of the cluster total.
* fix(eks): scope nodegroup Ready-gate per nodegroup and retry failed worker launches
waitWorkersReady compared the cluster-wide Ready-node count against a
nodegroup's baseline+desired, so in a multi-nodegroup cluster one nodegroup's
Ready workers could mask another whose own workers never registered. It now
reads meta.NodegroupNodeCounts[ng], scoped to nodes carrying the
eks.amazonaws.com/nodegroup=<name> label.
Separately, launchNodegroupInfra never retried a failed worker launch (a
transient QMP timeout or host capacity pressure), leaving the nodegroup stuck
CREATING with zero workers. The new launchWorkersUntilDesired wrapper relaunches
only the shortfall on failure, logging each retry, until the desired count is
reached or workerLaunchRetryTimeout elapses.
Both fixes are covered by regression tests: a two-nodegroup scenario where only
one has Ready workers, a transient-failure-then-success relaunch, and an
exhausted-retry-budget CREATE_FAILED path. Not covered here: a worker that dies
after the nodegroup is already ACTIVE — that needs an ongoing reconciler, which
this change does not add.
* fix(eks): tolerate GPU NoSchedule taint in konnectivity-agent DaemonSet
A GPU-only worker is tainted nvidia.com/gpu=present:NoSchedule, so the
konnectivity-agent DaemonSet (no tolerations) showed DESIRED=0 there and
kubectl logs/exec against pods on that node failed "No agent available".
Adds an Exists toleration for nvidia.com/gpu:NoSchedule.
Live-cluster validation (DESIRED count, kubectl exec) is deferred — no cluster
available to test against right now.
* fix: tolerate nvidia.com/gpu taint on ebs-csi-node and coredns DaemonSets so GPU workers get storage and DNS
A re-driven DeleteCluster whose managed control-plane VPC was already gone
(EC2 InvalidVpcID.NotFound) kept looping DependencyViolation: the internal
cp-vpc state record (ClusterMeta.ManagedCPVPC) was never cleared unless the
*entire* purge succeeded, so any unrelated flaky step retried CP-VPC/SG API
calls against a stale VpcId forever instead of converging.
purgeClusterInfra now clears ManagedCPVPC (via a new CAS-based
ClearClusterManagedCPVPC) as soon as DeleteClusterCPVPC + its SG cleanup both
succeed, independent of whatever else in that run still fails.
DeleteClusterCPVPC also now republishes vpc.delete for the CP VPC on every
call — including the already-gone fast path, falling back to the persisted
VpcId when the live tag-indexed describe can no longer name it — so vpcd's
topology manager gets another chance to GC an orphaned logical
router/subnets/SubnetEgressPriorityDrop policies if the original delete's
fire-and-forget event was ever lost.
handleRelease cancelled the per-lease renewal goroutine but deleted the
KV entry without waiting for it to exit. An in-flight reaffirm/renew Put
could land after the Delete, re-creating the lease and leaking a KV
record. Add a per-loop done channel and block on it after cancel so any
pending Put completes before the delete, making DrainAll's release path
deterministically synchronous.
* feat(eks): expose NVIDIA GPUs to pods via CDI device plugin
GPU nodegroups now label workers nvidia.com/gpu.present=true and taint them
nvidia.com/gpu=present:NoSchedule (via a k3s config.yaml.d drop-in, since k3s
has no K3S_NODE_TAINT env), and stage a hidden nvidia-device-plugin addon that
addon-sync renders into the K3s auto-deploy dir. The device plugin runs with
--device-list-strategy=cdi-cri and is gated to GPU nodes by nodeSelector +
toleration, advertising nvidia.com/gpu allocatable so pods land on GPU nodes
and get the device injected via the CDI spec baked by mulga-cdi-generate.
Relies on containerd 2.0.5 default enable_cdi=true (no config fork).
* fix(vm): scale QMP greeting deadline for VFIO passthrough guests
A GPU passthrough guest (-device vfio-pci + large -m) must lock and DMA-map
its entire guest RAM through the IOMMU before QEMU's QMP monitor answers,
which exceeds the fixed 30s greeting deadline and gets the VM SIGKILLed at
launch. Parameterize the greeting timeout (NewQMPClientWithGreetingTimeout)
and scale it to 180s for VMs with GPU attachments; plain VMs keep the 30s
default and reconnects against an already-running QEMU keep it too.
* fix(eks): register GPU workers reliably and expose GPUs to pods
Three fixes let a GPU nodegroup worker boot, register, and advertise its GPU:
- k3s-agent.service gains Delegate=yes (plus KillMode/TasksMax/limits and
overlay/br_netfilter modprobe): k3s embeds containerd+kubelet in-process, so
on cgroup-v2 the kubelet cannot create /kubepods without a delegated subtree
and ContainerManager.Start fails before Node registration.
- The GPU node label rides a k3s config drop-in (20-gpu.yaml node-label +
node-taint) instead of K3S_NODE_LABEL, which k3s ignores; without the label
the nvidia-device-plugin nodeSelector never matches.
- Raise the nodegroup ready timeout to 20m for the ~7min GPU first boot.
Adds the GPU e2e (VPC + NAT-GW egress harness) and a slog-restore unit test.
* fix(eks): install ecr-credential-provider on the GPU node AMI
The GPU worker's kubelet is started (from nodegroup userdata) with
--image-credential-provider-config + --image-credential-provider-bin-dir=
/usr/local/bin, but the eks-node-gpu AMI never built or installed the
ecr-credential-provider binary. kubelet treats a referenced-but-missing
provider binary as a fatal RegisterCredentialProviderPlugins error, so the
node crash-loops and never registers (while the Alpine eks-node worker, which
ships the binary, joins fine). Build + install it in the GPU manifest; the
server-only binaries stay omitted since GPU nodes are agent-only.
Also tee k3s-agent stderr (and its embedded kubelet klog) to the serial
console: Ubuntu logs only to journald, which is off-box-invisible on a worker
in an isolated customer VPC, so a kubelet that fails to register left no trace.
* fix(eks): give the nvidia-device-plugin its own GPU access
The CDI-mode device plugin (--device-list-strategy=cdi-cri) must run
nvml.Init and generate the CDI spec in its own container, but the DS had
no GPU wiring, so nvml.Init crash-looped and nvidia.com/gpu was never
advertised. Run the DS under the k3s-auto-configured nvidia runtimeClass
with NVIDIA_VISIBLE_DEVICES/CAPABILITIES=all, the host root mounted
read-only at /driver-root so it can locate the driver libs + ld.so.cache,
and /var/run/cdi mounted writable so the generated spec is visible to
containerd. No enable_cdi drop-in is needed on k3s v1.32.5 / containerd
2.0.5. GPU workload pods still need only resources.limits nvidia.com/gpu.
Live-validated end-to-end on the single-GPU host (RTX A1000): baked
control-plane AMI, GPU nodegroup, nvidia.com/gpu allocatable, nvidia-smi
pod, zero manual intervention.
* test(ec2): drop obsolete restoreSlogDefault test
Rebased onto dev, which dropped the restoreSlogDefault workaround in #506
(viperblock no longer mutates the global slog). The test referenced the
now-deleted helper and no longer compiles. Real GPU/registration diff is
independently covered (diff coverage 95.3%).
When no br-mgmt bridge is present (supported single-node config where system
instances reach AWSGW via the advertise IP), LaunchSystemInstance correctly
skips mgmt-tap creation but the NIC list still carried the mgmt slot, so
buildNICNetdevs emitted a QEMU netdev/device for a tap that was never created
and QEMU aborted (could not configure /dev/net/tun: Operation not permitted),
failing every CreateLoadBalancer/CreateCluster. Skip the mgmt NIC (index 1)
when its MAC is unset across netdevs, kernel-cmdline mmio slots, and the guest
fw_cfg netcfg blob; multi-node (br-mgmt present) is unchanged.
- 4ncdg: NATSRequest now round-trips the AWS error Message, not just the
sanitized Code. Server sends GenerateErrorPayloadWithMessage(code,
err.Error()); the client returns the Message when present so the CLI prints
the real reason (e.g. "snapshot not found; refusing to restore") instead of
a bare "ServerInternal". One code path fixes every NATS-driven command.
- fbmhl: the UI node view rendered raw host totals (vCPU:80, 0/125.5GB),
ignoring the daemon reserve. It now shows schedulable = total - reserved,
the capacity a tenant can actually consume. reserved_vcpu/reserved_mem_gb
added to the NodeInfo type (already present in the API payload).
Three related nat-mode fixes:
- ura01: nat-transit pool now emits gw_lrp_range_start/end. admin init
sets 100.127.0.16-.254 (239 VPC gateways) instead of the auto-derived
top-16 (~15). Adds GwLrpRange* to PoolData + the spinifex.toml template
so operators can widen any pool.
- 724mm: detectPoolMode returns true only for external_mode=pool. nat and
disabled clusters now skip Phase 8b/8d, which otherwise failed on
AllocateAddress UnsupportedOperation.
- pnz2i: document that system instances (ECS/EKS/LB agents) are
unsupported in nat v1 (require pool mode) in the vpc-networking README
and COMMANDS.md; document the gw_lrp_range_* fields.
* fix(eks): write etcd-snapshot.env at CP launch so snapshots actually run
* test(eks): cover resolveSystemPredastoreURL and buildEKSServiceDeps
Diff-coverage gate for 852f61d9 was unmet (48.8% < 60%) because the
mgmt-bridge branches and EKSServiceDeps assembly had no direct test.
* feat(eks): restore-snapshot DR path (fresh CP + cluster-reset restore + NLB re-point)
* fix(eks): make restore-snapshot fail-safe (validate snapshot, unwind on partial failure, honest status)
* fix(eks): sign etcd-snapshot uploads with deployment region (was 403ing)
* fix(eks): apply recovery directive on DR-seed first boot + single-node replacement host
* fix(daemon): log NATS handler failures at Error (were only on the trace span)
* feat(gpu): Ubuntu NVIDIA GPU node image + GPU AMI selection
Epic A of GPU passthrough to EKS pods / ECS task containers.
- Add gpu-vendor AMI tag constants + GPUVendorForType/IsGPUTypeName helpers
(instance-family prefix -> vendor from knownGPUModels).
- EKS + ECS worker AMI resolution branches to a GPU-tagged node image for GPU
instance types; no fallback to the driverless image (hard error instead).
- Exclude gpu-vendor-tagged AMIs from the plain (non-GPU) lookup so a newer GPU
AMI cannot hijack ordinary nodes (managed-by tag is shared; filters have no
negation, so exclude client-side).
- NodegroupRecord caches GPUEnabled/GPUVendor from InstanceTypes.
- Catalog entries spinifex-eks-node-gpu / spinifex-ecs-node-gpu (dual-tagged).
- New Ubuntu 26.04 GPU node images (eks-node-gpu, ecs-agent-gpu): reuse the
proven DKMS -server driver recipe (drop Docker CE + dev payload), add
nvidia-container-toolkit, OpenRC->systemd port of the worker/agent service
subset, udev EBS by-id bridge, nvidia modules-load, first-boot CDI generation.
- build-system-image.sh: SYSTEM_TAG supports multiple space-separated tags.
* build: allow SYSTEM_IMAGE_BUILD_DIR override for image build dir
Ubuntu GPU images are 16G; a RAM-backed /tmp tmpfs is too small to hold the
qcow2 + expanded raw. Default stays /tmp; SYSTEM_IMAGE_BUILD_DIR redirects the
build onto a real disk.
An ELBv2 LB whose system VM launches but whose in-guest lb-agent never
delivers a first heartbeat previously stayed in provisioning forever with
an empty StateReason: promotion to active happens only on the first
heartbeat and the only provisioning->failed transitions were at
create/launch time. A single wedged LB in the multi-node e2e lb suite
leaked its listener/target-group and cascaded into ResourceInUse and
DependencyViolation on VPC teardown.
- Add a lifecycle reaper that marks failed any LB still provisioning past
lbProvisioningTimeout (4m) with no heartbeat, reusing markLBFailed, and
populate a provisioning StateReason so a stuck LB is self-describing.
Started once at daemon startup; stopped on service Close.
- Harness: add ErrLBProvisioningTimeout and treat a provisioning-timeout
as retryable in createActiveLB; register idempotent LB/listener teardown
up-front so no exit path leaks into the shared VPC.
- IMDS: bounded retry+backoff on the instance-fact and user-data lookups so
a transient gather timeout under multi-node load does not starve the
lb-agent bootstrap.
The coordinated host-shutdown DRAIN phase transitioned every running VM to
StateStopped and migrated it to the operator-stopped shared KV bucket, evicting
it from node-local running state. On the next boot Restore only loads local
running state and treated every StateStopped instance as user-stopped, so
previously-running VMs never relaunched after a host reboot.
Gate on the existing operator-intent signal Attributes.StopInstance (set only by
StopInstances, cleared by StartInstance): stopOne keeps drain-stopped VMs in
local running state instead of migrating them, and Restore relaunches
drain-stopped VMs while operator-stopped ones stay stopped. markUnschedulable
now stamps StopInstance so capacity-demoted instances are not relaunched.
Close four pre-existing lifecycle races surfaced during env19 hardening.
Volume seal/reaper (42hbm, c7ce2): the ebs.unmount handler now seals the
volume before removing it from MountedVolumes, so a retry after a
timed-out-but-completed seal reliably reports NotFound. EBSUnMountResponse
carries a NotFound flag and unmountResponseError treats it as seal success,
so state reconciles to available/detached instead of leaking. The volume
leak reaper reconciles the attachment record (never the data) for volumes
whose owning instance is terminated, so GC can delete them.
Stopped-instance CAS (ozlbq, wlmea): add UpdateStoppedInstance, a
mutate-callback backed by casUpdate(createIfAbsent=false), and migrate
TagStoppedInstance and ModifyInstanceAttribute off the resurrect-prone
Load->mutate->WriteStoppedInstance path. A tag/attribute write racing a
winning start-claim now fails cleanly instead of re-creating the deleted
entry. StartStoppedInstance returns IncorrectInstanceState (not
InvalidInstanceID.NotFound) when the stopped entry is absent but the
instance is present locally.
Every tofu-based e2e workflow set TOFU_LOG to a /tmp path outside
ARTIFACT_DIR. ci-fmt.sh's quiet_run helper appends all tofu init/apply/
destroy output to that file, but the Teardown step (which runs the
destroy) always ran AFTER the existing upload-artifact step, so the
full log — including destroy-time failures — was silently discarded
on every run.
Add an if: always() upload-artifact step after Teardown in each
tofu-based job (e2e.yml, e2e-suites.yml, e2e-nightly.yml, e2e-ddil.yml)
that uploads $TOFU_LOG directly. Since quiet_run appends both apply and
destroy output to the same file, one upload after Teardown now
captures everything, recoverable even when destroy itself fails.
Shipping this log to Elasticsearch/OTLP is deferred to mulga-5n7fe,
which will consume this artifact.
Init now installs an sdk/log LoggerProvider inside the existing
exportEnabled() gate, sharing the same resource as the tracer/meter (incl.
ci.run_id via OTEL_RESOURCE_ATTRIBUTES) and joining lp.Shutdown into the
returned shutdown func. The process default slog handler is fanned out to
both the existing stdout/journald handler and an otelslog bridge, so app
logs export over OTLP with trace correlation while journald output is
unchanged. With no OTLP endpoint configured Init remains a no-op: no
LoggerProvider is installed and the default logger is left untouched.
PR #487 turned attach/detach flaky: reconnectQMP dialed a fresh QMP
client before closing the old connection. QEMU's monitor is a
single-client `server,nowait` listener, so while the old conn holds the
slot the redial gets no greeting, blocks ~30s, and fails, leaving
q.Dead permanently true and every later attach/detach ServerInternal.
Fix the ordering (close, then dial via dialQMPWithRetry to absorb the
brief post-close ECONNREFUSED window).
waitForDeviceDeletedEvent marked q.Dead on any decode error, including
a benign read-deadline timeout — routine, since DEVICE_DELETED is
frequently pre-swallowed by device_del's own reply loop. The
underlying connection survives a timeout, but encoding/json.Decoder
permanently caches read errors on itself, so on a timeout the Decoder
is swapped for a fresh one over the same conn instead of forcing a
full QMP reconnect; q.Dead is now only set on a genuine decode error.
Also: an AttachVolume re-attach to the same instance with a different
device now returns VolumeInUse (matching AWS) instead of
InvalidParameterValue.
* feat(ec2,ebs): harden volume attach/detach + instance lifecycle state machine
Batch of related EBS/instance-lifecycle correctness fixes verified together.
- u94vs: add compare-and-swap (casUpdate/casPut via nats KV Update revision)
to the multi-writer lifecycle KV sites (WriteTerminatedInstance,
WriteStoppedInstance, WriteClusterShutdown); make the teardown-reaper
read-modify-write atomic via a merge callback so retries never re-run
cleaner side effects.
- yi2ed: add ClaimStoppedInstance, a revision-checked delete-as-claim on the
stopped bucket performed before Allocate/Insert/Run, so a forwarded start
and a local fallback cannot double-start one instance onto the same volume.
- siv-473: make AttachVolume idempotent — return attached success when the
volume is already attached to the requesting instance; a conflicting
requested device returns InvalidParameterValue.
- siv-462: reconnect now re-asserts in-use for all attached volumes (boot and
non-boot data volumes), not just boot, so a daemon restart no longer leaves
stateful-pod data volumes split-brain; EFI pflash is skipped.
- 9w2mv: wait for the QMP DEVICE_DELETED event (bounded) before blockdev-del
instead of a blind sleep, so a slow guest unplug no longer exhausts retries.
- soyah: poll (50ms) instead of fixed 1s sleeps in KillProcess and the
viperblockd nbdkit confirm-running path, cutting volume-lifecycle latency.
- siv-481: a cross-node start forward that times out now falls back to a local
start; this is safe because the yi2ed claim is the single serialization point.
* fix(eks): create per-account and leader KV buckets with cluster replica count
The EKS per-account KV bucket and the leader lease bucket were created at
Replicas=1 and never upgraded, since they're created lazily after
upgradeJetStreamReplicas runs at startup. On multi-node clusters this leaves
their single-replica Raft group transiently leaderless during formation,
causing KV timeouts that surface as a deterministic 503 on ListClusters.
Thread clusterSize through GetOrCreateAccountBucket/InitLeaderBucket (backed
by a new utils.GetOrCreateKVBucketWithReplicas) so both buckets are created
at max(clusterSize,1) replicas, matching every other node-count-scaled
JetStream stream.
A fresh cluster or SB re-bootstrap could leave a node's ovn-controller with
its OVSDB client latched on stale Southbound RAFT cluster state, looping on
"clustered database server has stale data" and silently no longer realising
new Port_Bindings. Cross-host CP data-NICs then never bind: no DHCP, no IMDS,
degraded boot. A manual `systemctl restart ovn-controller` was the only
recovery. Root cause is deterministic: setup-ovn.sh restarts ovn-controller
per node as each RAFT member forms, so a controller started for an early
member races the SB membership changes that later joins trigger.
Add three layers of recovery (the ansible bring-up settle pass that fixes this
at the source lives in the mulga repo):
- Per-host wedge watchdog (vpcd/ovn_controller_watchdog.go): not leader-gated
since the wedge is per chassis. Polls SB connection-status; after a stale
threshold issues sb-cluster-state-reset, throttled by a cooldown. Decision
logic takes an injected clock for deterministic tests.
- Reconcile-loop escalation: ensureGatewayClaimed and ensureGuestPortDatapath
escalate to one sb-cluster-state-reset after repeated recompute misses only
when the SB is not connected — a recompute cannot clear a stale-SB wedge.
- Prober primitives SBConnectionState/ResetSBClusterState on GatewayClaimProber
plus the GatewayClaimVerifier interface extension.
Also fix the awsgw OVN health probe, which passed a slashed `-t` socket path
that never resolves and so always reported ovn-controller not_running. Probe
connection-status instead: OVNControllerUp is now true only when connected, so
healthy nodes read ok and a real wedge correctly reads not_running.
The EIP "verify unreachable after disassociate" step polls runSSHQuiet
inside EventuallyErr(90s), but a wedged ssh made os/exec Wait block for
14 minutes — EventuallyErr cannot interrupt an in-flight CombinedOutput,
so one probe consumed the whole 30-min test-binary budget and panicked.
The panic skipped t.Cleanup, leaking the EIP and NAT-gateway VMs, which
held single-node capacity and starved the lb suite's sys.micro launch
(the daemon unsubscribes the full topic, so NATS returns no-responders).
Bound the ssh child at the process layer with exec.CommandContext (30s
deadline) plus WaitDelay, returning a descriptive timeout error. This is
version- and phase-independent, unlike ssh's own ConnectTimeout. 30s is
safe for all callers (worst legit call is ~15s) and well under the 90s
poll budget, so a wedged probe now yields a clean bounded failure with
cleanup intact.
* feat(network): add routed uplink mode, Routed wiring, and NAT egress rule helper
* feat(external): route gateway LRP over transit pool and install VPC SNAT in routed NAT mode
* feat(vpcd): support bridge_mode=nat (routed uplink, transit pool, egress rules)
* feat(daemon): register EIP handlers unconditionally with disabled-mode stub; seed default subnet per external mode
* feat(admin): support --external-mode=nat and suggest it for non-bridgeable uplinks
* feat(scripts): add --nat-uplink routed mode to setup-ovn.sh
* test(e2e): add routed-NAT single-node natuplink suite
Covers the external_mode=nat lane end to end on a nat-initialized node:
host transit wiring (veth, br-ext port, ip_forward, iptables egress
rules), spinifex.toml contents, disabled EIP surface (empty
describe-addresses, UnsupportedOperation on allocate), default subnet
MapPublicIpOnLaunch=false, OVN gateway LRP + VPC SNAT on the transit
/24, guest outbound egress proven via a serial-console probe (nat mode
has no inbound path), and unique transit gateway IPs across VPCs.
Suite self-gates on external_mode=nat; registered as e2e-natuplink and
in SUITES_BUILD so it compiles in CI. Manifest entry stays commented
until CI can provision a nat-mode environment.
* feat(ovn): Address_Set CRUD and NAT exempted_ext_ips column
Adds the exempted_ext_ips column (optional strong Address_Set ref) to the
nbdb NAT model, Address_Set ensure/get with wait-op create semantics, and an
in-place SetNATExemptedExtIPs update on the client interface, live client,
and mock. Test NB schema gains the column and covers insert-resolves-UUID,
converge, clear, and missing-NAT paths.
* feat(policy): exempt address set on routed SNAT/EIP rows
Adds WithSNATExemptSet so routed mode stamps exempted_ext_ips (the
spinifex_nat_exempt Address_Set: transit /24 plus operator extras) on IGW
snat and EIP dnat_and_snat rows, keeping VM replies to host-initiated flows
un-NATted. AddSNAT gains an idempotency guard keyed on (router, VPC CIDR)
with stale-IP replacement, and both AddSNAT and the AddEIP idempotent-skip
path patch the ref in place on pre-existing rows. The set is a singleton
strong ref and is never deleted. Non-routed modes are unchanged.
* feat(host): VPC ingress route primitives for routed-NAT mode
EnsureVPCIngressRoute / RemoveVPCIngressRoute manage the host route that
sends VPC-CIDR traffic into OVN via the gateway LRP transit IP on the
spx-nat veth. Replace/del semantics keep both idempotent; a missing route
on removal is not an error.
* feat(external): routed ingress leg on IGW attach/detach, reconcile-safe
AttachIGW's routed leg now also installs the host VPC ingress route via
RoutedIngressHooks, and re-runs on the already-attached path so the
reconcile loop restores host routes (and patches legacy SNAT exempt refs)
after a reboot — OVN state survives restarts but host routes do not.
DetachIGW removes the route alongside the SNAT. vpcd wires the hooks and
the spinifex_nat_exempt set (transit /24 + [network] nat_exempt_cidrs,
validated nat-mode-only) exclusively in routed-NAT mode; pool/veth/direct
pass nil hooks and are unchanged.
* test(natuplink): Tier 1 host-ingress phase; docs for jumpbox pattern
New Phase 7 asserts the automatic ingress tier: host route to the VPC CIDR
via the gateway LRP transit IP, exempted_ext_ips ref on the SNAT row
pointing at spinifex_nat_exempt (containing the transit /24), and — after
opening the SG from the transit net (default-closed, AWS parity) — ping and
an sshd TCP handshake against the instance private IP from the host, which
proves the un-SNATted return path. Phase 8 additionally waits for the
second VPC's ingress route to appear on IGW attach. vpc-networking README
documents the jumpbox pattern, SG note, and nat_exempt_cidrs.
* feat(config,admin): public wan pool alongside nat-transit
Restructure admin config generation from single-pool scalars to a Pools
slice so nat mode can emit both the transit pool and an optional public
wan pool. nat mode now accepts --external-pool/--external-source (plus
--external-bind-bridge/--gateway-ip when a public pool is requested),
validated by the shared resolvePublicPoolFlags helper also used by pool
mode. Transit pool always renders first so the IGW allocator picks it.
Formation wire format unchanged (nat mode is single-node only).
* feat(daemon): enable EIP service and auto-assign in nat mode with public pool
External IPAM (and with it the real EIP service, RunInstances public IP
auto-assign, and MapPublicIpOnLaunch on default subnets) now activates
whenever public pools exist: pool mode as before, plus nat mode when a
wan pool rides alongside the transit segment. The transit pool is
filtered out of IPAM — its addresses are gateway-LRP plumbing, never
allocatable EIPs. Transit-only nat configs keep the disabled EIP stub.
* feat(host): per-EIP ingress primitives for routed-NAT mode
DetectUplinkFor parses ip route get to find the interface and source IP
facing the pool gateway (re-run per ensure: WiFi/cellular uplinks move).
EnsureEIPIngress steers an EIP into OVN with a /32 route via the VPC
gateway LRP transit IP (src-hinted so host-local traffic still DNATs),
answers ARP for the EIP with a proxy neighbor entry (L3 only, same MAC,
WiFi-safe), sends gratuitous ARP, and adds per-EIP FORWARD accepts for
DROP-policy hosts. RemoveEIPIngress mirrors; missing state tolerated.
* feat(policy,vpcd): host EIP binder option on NATManager
WithHostEIPBinder injects routed-mode host plumbing hooks. AddEIP binds
after priming on both the fresh-add and idempotent-skip paths (host
routes and proxy-ARP are volatile even when the OVN row survives, so
reconcile re-ensures them after reboot); DeleteEIP unbinds best-effort.
The next hop comes from the new gatewayPortIP helper reading the gateway
LRP's stamped spinifex:gateway_ip (Networks fallback), with the ext-ID
key now an exported policy const shared with the external package.
vpcd selects the IGW pool by name instead of index — with two pools in
nat mode the gateway allocator must stay on the transit pool — and wires
the binder over host.EnsureEIPIngress/RemoveEIPIngress when a public
pool is present. EnsureEIPIngress gains an uplink hint so dhcp-sourced
pools (gateway unknown until lease) proxy-ARP on their bind bridge.
* feat(dhcp): interface-MAC lease variant with MAC-keyed-router detection
dhcp_mac = "derived" (default) | "interface" on external pools. The
interface variant leases with the bind interface's own MAC — WiFi/WWAN
uplinks drop frames with foreign source MACs, so derived chaddr MACs
never complete DORA there. Option 61 stays the lease discriminator.
UseIfaceMAC threads AllocateRequest → NATS wire → nclient4 → lease and
persists through KV so restart re-acquires repeat it.
Routers that key leases on MAC and ignore client-id would hand every
lease the same IP; the manager detects an ACK matching the interface's
own address or another client-id's live lease, releases the colliding
lease, and hard-errors advising a static range instead.
admin init auto-writes dhcp_mac="interface" for nat-mode public dhcp
pools bound to non-bridgeable uplinks (wlan/wwan/ppp).
* test(natuplink)+docs: Tier 2 EIP ingress e2e phases and public-pool docs
The natuplink suite now detects a public (non-transit) pool in spinifex.toml
and switches lanes: Tier-1-only configs keep the EIP-disabled and
no-public-IP assertions; public-pool configs assert the EIP surface is
enabled, the default subnet has MapPublicIpOnLaunch=true, instances
auto-assign public IPs, and a new Phase 9 exercises the full EIP ingress
lifecycle — host /32 route via the gateway LRP, proxy-ARP on the uplink,
per-EIP FORWARD accepts, exempt-stamped dnat_and_snat, TCP handshake to the
EIP, reconcile replay across a vpcd restart, and teardown on disassociate.
Docs: vpc-networking README gains a public-pool section for routed-NAT nodes
(init flags, dhcp_mac=interface behaviour, host-to-EIP caveat) and an updated
mode comparison; COMMANDS.md notes nat-mode EIP support behind a public pool.
* fix(setup): whitelist iptables, neigh sysctl, and arping for spinifex-vpcd
Routed-NAT mode shells out to iptables for the transit masquerade and
per-EIP FORWARD accepts, tunes proxy_delay, and sends gratuitous ARP for
host-delivered EIPs. Without these sudoers entries vpcd crash-loops with
"command not allowed" on nat-mode nodes.
* fix(ovn): scope DeleteStaticRoute prefix match to the target router
Every VPC router carries a 0.0.0.0/0 static route, so an unscoped cache
match could grab another router's row and fail with a referential
integrity violation, wedging IGW attach on multi-VPC nodes. Match only
rows referenced by the target router, mirroring FindStaticRoute. Mock
updated to the same semantics, with a regression test covering drift
replacement across two routers.
* fix(reconcile): always replay AttachIGW so host ingress state is re-ensured
applyIGWs skipped AttachIGW whenever the gateway switch port existed in
actual state, so the already-attached re-ensure path never ran and
routed-NAT host ingress routes lost on reboot were never reinstalled.
AttachIGW is idempotent; run it unconditionally.
* fix(vpcd): resolve duplicate-CIDR host ingress route collisions, bootstrap VPC wins
Every account's default VPC shares 172.31.0.0/16, so Tier-1 host routes
collide on the single host route table and the last IGW attach silently
stole the route (route replace). Pass vpcID and the gateway LRP IP
through RoutedIngressHooks; the bootstrap VPC always installs its route,
other VPCs only install when the CIDR is free and only remove a route
their own gateway IP holds. Adds host.VPCIngressRouteVia to read the
current route holder.
* ci(e2e): add nightly cell 19 for routed-NAT single-node suite
Cell 19 bootstraps with --external-mode=nat plus the CI static pool as
the public wan pool and runs the natuplink suite, covering the Tier-1
transit datapath and Tier-2 EIP ingress. The run step exits 0 when
natuplink.test is absent from the tests tar so scheduled runs against
branches predating the suite stay green.
* fix(admin): nat-mode init summary reflects configured public pool
The summary printed 'outbound-only — no public IPs/EIPs' even when
--external-pool/--external-source configured a public wan pool. Print
the pool range (or dhcp bind) and note EIPs are enabled.
* fix(eip,vpc): thread ctx through disabled EIP stub and NAT call sites
Reconcile the routed-NAT branch with dev's ctx-first service APIs after
rebase: DisabledEIPService and its tests take ctx to satisfy the EIPService
interface, the vm_adapters public-IP resolver asserts the ctx-signed
AssociatedPublicIPForInstance, and the default-subnet mapping test threads
ctx into DescribeSubnets.
* feat(otelsetup): OTel init, resource attrs, trace-stamping slog handler
Env-gated on OTEL_EXPORTER_OTLP_ENDPOINT: with no endpoint only the W3C
propagator is installed and globals stay no-op, so instrumented binaries
ship everywhere at zero cost. Resource carries service/host/mulga.env/
mulga.source plus OTEL_RESOURCE_ATTRIBUTES (ci.run_id in CI).
* feat(otelsetup): wire telemetry init into all spx services
Every spx service start command installs the trace-stamping JSON slog
default and the OTel providers via initTelemetry, flushed on shutdown.
Gateway's own slog setup now routes through otelsetup so its level
handling keeps trace correlation.
* feat(gateway): HTTP tracing middleware on awsgw and daemon cluster API
Server span per request with W3C traceparent extraction; awsgw spans are
renamed to the resolved SigV4 service.Action post-auth with account and
region attributes. Request log lines carry trace_id via context.
* feat(nats): propagate W3C trace context over command subjects
NATSRequestCtx injects traceparent beside X-Account-ID and opens a client
span per hop; ServeNATSRequest opens a consumer span joining the producer
trace, with a Ctx variant for handlers that thread context. RunInstances
is plumbed end-to-end (gateway dispatch through placement fan-out to the
daemon handler) and viperblockd ebs.mount/unmount join inbound traces.
* feat(systemd): source optional /etc/spinifex/telemetry.env in service units
Lets operators enable OTLP export (OTEL_EXPORTER_OTLP_ENDPOINT,
MULGA_ENV, MULGA_SOURCE) by dropping a single env file, with no unit
edits. File is optional; absent means telemetry stays disabled.
* feat(ci): export OTel telemetry from E2E envs to the central Elastic sink
All E2E workflows set SPINIFEX_OTLP_ENDPOINT (repo variable override,
default wattle) plus ci.run_id/ci.workflow resource attributes, so
bootstrap-install.sh writes telemetry.env on every node. Teardown runs
flush-telemetry.sh before the final tofu destroy so buffered spans and
metrics reach the sink.
* feat(otelsetup): request metrics and Go runtime metrics via OTLP
mulga.requests counter + mulga.request.duration histogram recorded by the
same middleware that opens spans: awsgw carries the resolved SigV4 action,
the daemon cluster API uses the chi route pattern, and NATS handlers record
per-topic at the subscribe chokepoints (node names stripped to keep
cardinality flat). Go runtime metrics start inside the same env gate.
* feat(daemon): deep spans for QMP commands, VM launch phases, EBS ops
Single chokepoint span per QMP command in sendQMPCommand covering all
callsites; vm.launch phase children (mount_volumes, start_qemu, qmp_connect,
persist_state); ec2.cmd consumer span with traceparent extraction plus
start/stop/reboot/terminate child ops; ebs.mount/unmount spans in
viperblockd with volume attributes.
* feat(telemetry): thread request context through NATS, object store and VM paths
Migrate utils.NATSRequest call sites to NATSRequestCtx (241 sites; W3C
traceparent now flows gateway -> daemon on every request), add GatherCtx and
migrate all Gather call sites, inject traceparent on manual EKS publishes.
ObjectStore interface is ctx-first with aws-sdk WithContext calls and an
otelhttp transport, so predastore spans join the caller trace. NATS consumer
spans record error status when the response envelope carries an error.
vm.Manager and the ec2 handler service interfaces take ctx-first so vm.launch
and QMP spans join the request trace; detached background work uses explicit
context.Background(). Hot-path slog calls use *Context variants for trace_id
binding.
* feat(telemetry): suppress health-check, heartbeat and unrooted trace noise
Health probes record metrics only (no span per poll); the QMP heartbeat
poller marks its context so query-status ticks stop rooting a trace each.
rootSampler honors MULGA_ROOT_TRACE_RATIO (ParentBased ratio, default 1.0)
and the predastore unit sets 0.05: guest block I/O roots are sampled while
request-linked spans are always kept.
* feat(otel): export deployment.environment resource attribute
Elastic APM maps it to service.environment, enabling the native environment
selector across the APM UI (Services, Traces, Dependencies).
* fix(telemetry): stop objectstore rooting method-only HTTP transactions
Background S3 calls (state sync, cleanup loops) carried no caller span, so
otelhttp rooted a trace per request and APM surfaced thousands of parentless
'HTTP GET/PUT/DELETE' transactions with expected-404 probes counted as
failures. Emit the client span only when the request context already carries
one, matching the viperblock backend. Bump viperblock/predastore to branch
heads so CI builds also pick up the equivalent span filter and root sampler.
* feat(telemetry): per-VM guest metrics via dedicated QMP socket + qmp-collector
Goanna Phase 1 producer, operator-consumed first. Each VM gets a second
-qmp chardev (qmp-telemetry-<id>.sock) plus a sidecar discovery file
carrying instance/account/vcpus/taps/monitoring-tier; the new
qmp-collector service polls query-cpus-fast (vCPU thread /proc jiffies),
query-blockstats, query-balloon and host tap sysfs counters, and
publishes CloudWatch-mappable goanna_ec2_* series to NATS
metrics.ec2.<instance-id>. A queue-group bridge in the same process
forwards the series to the node-local OTLP receiver so they reach the
operator sink until Goanna consumes the subjects. The collector never
touches the manager's control QMP socket; telemetry failures never
block a launch.
* fix(qmp-collector): dial NATS via DialTarget, drop ProtectProc, fix meta race
The collector connected to the raw NATS host (0.0.0.0), failing TLS
verification against a cert valid for 127.0.0.1; route through
admin.DialTarget like the other daemons.
ProtectProc=invisible hid the QEMU vCPU threads (dumpable=0) that the
collector must read from /proc/<tid>/stat, so every tick failed and
re-primed without ever publishing. Drop it to default.
Surface collect failures at Warn so silent metric loss is visible, drop
per-tick publish to Debug, and read InstanceID under the lock in run()
to fix a race with updateMeta on ENI hot-plug.
* chore(deps): re-pin predastore + viperblock to dev-landed commits
* feat(eks): reconciler in-place control-plane restart on sustained degradation
Phase 1 of EKS CP recovery (siv-243). The ACTIVE-branch reconciler only
recorded health; a CP VM that died (OOM/host crash) left the cluster wedged
ACTIVE-but-unreachable with no recovery path. Add a CPInstanceControl surface
and bounded in-place restart: once an ACTIVE cluster's CP stays unhealthy past
a grace window, StartInstance is issued (backoff-spaced, capped) when the
instance is stopped/error, re-mounting the same root volume so embedded etcd
survives. Degradation stays AWS-faithful ACTIVE + a ClusterHealth issue rather
than a non-AWS DEGRADED status enum. cpControl nil keeps the reflect-only path.
Wiring of a real CPInstanceControl in the daemon and the error-state
manager-eviction fix are the remaining Phase-1 integration steps; Phase 2
(etcd snapshot cadence + restore-snapshot) is tracked separately.
* feat(eks): wire control-plane restart into the reconciler via instance service
Threads a CPInstanceControl into the per-cluster reconciler (siv-243 Phase 1).
cpControlAdapter binds the daemon instance service + account: InstanceState
reads the CP's EC2 state via DescribeInstances, StartInstance restarts a stopped
CP via StartStoppedInstance, which loads from the shared stopped store and
re-mounts the same root volume so embedded etcd survives. Wired as
EKSServiceDeps.CPControl = d.instanceService and passed through spawnReconciler.
Recovers a cleanly-stopped control plane co-located with the reconciler leader.
Remaining: DescribeInstances is node-local so a cross-node stopped CP needs an
aggregating describe; and an error/crashed CP (not in the stopped store) needs a
dedicated error-state restart path — both tracked as Phase-1 follow-ups.
* feat(eks): cross-node CP describe + error-state control-plane restart
The reconciler's control-plane recovery was wired to the node-local instance
service: DescribeInstances only saw the leader's node and StartStoppedInstance
only handled a cleanly-stopped CP from the shared KV. A crashed/OOM CP ends in
error state, still resident in its owning node's manager map (not the stopped
store), so recovery never reached it — the actual incident in the recovery plan.
Wire EKSServiceDeps.CPControl to a NATS-backed eksCPControl. DescribeInstances
fans out across every host (gateway scatter-gather) so the reconciler observes a
CP on any node. RecoverInstance drives the gateway StartInstances two-step:
ec2.cmd.<id> recovers a live error/running owner in place (the owner stays
subscribed through the crash; vmMgr.Start relaunches error->pending, re-mounting
the etcd root volume), falling back to ec2.start shared-KV rehydration for a
stopped CP.
The CP VM is ManagedBy=EKS, hidden from customer DescribeInstances; relax that
hide so the owning account resolves its own managed CP when targeting it by
explicit instance ID (bulk listings still hide it). EC2APIState already maps the
internal error state to the AWS closed-enum stopped, which cpRestartable accepts.
* fix(eks): reconcile CP as system account, not customer
The control-plane VMs are launched under the system account
(admin.SystemAccountID()), not the customer account that owns the cluster
record. spawnReconciler was binding cpControlAdapter to the customer account,
so InstanceState/RecoverInstance ran as the customer — which cannot see or own
its own cluster's system-owned CP VMs. DescribeInstances returned zero and the
reconciler logged 'control-plane instance not found', never restarting a wedged
CP. Bind the adapter to the system account instead.
Revert the earlier DescribeInstances managed-VM relaxation: it was based on the
mistaken assumption that CP VMs are customer-owned. Visibility is resolved by
the system-account adapter, so the explicit-ID escape hatch is dead code.
* fix(vm): pre-create mgmt tap on disk-boot restart path
The direct-boot branch pre-creates the management tap via NetworkPlumber.SetupTap
(owned by the daemon euid), but the disk-boot/else branch only appended the QEMU
tap,id=mgmt0 device without creating the tap. The daemon runs as a non-root user
without CAP_NET_ADMIN, so QEMU can only attach a tap it already owns; on the
disk-boot path QEMU tried to create /dev/net/tun itself and failed with
'Operation not permitted', failing every stopped-instance restart (StartInstances
returned ServerInternal). This blocked EKS control-plane recovery: the reconciler
found the stopped CP and issued the restart, but the VM never booted.
Call SetupTap for the mgmt tap in the disk-boot branch, mirroring direct-boot.
* feat(eks): recover all HA control-plane members, not just primary
The reconciler restarted only ControlPlaneInstanceID (the primary CP
member), so on an HA cluster the other server VMs stayed stopped and
etcd never regained quorum (1/3) — apiserver healthz stayed fail even
after recovery ran. maybeRecoverControlPlane now iterates every member
in ControlPlaneNodes (falling back to the scalar ID for single-node
clusters), restarting each stopped/error member per pass while
preserving the grace window, attempt cap, and backoff gates.
Refs mulga-siv-509
* fix(eks): keep cloud-init polling IMDS so primary CP gets its user-data
The eks-node system VM's user-data is delivered only via IMDS (the
NoCloud seed was retired in #411). IMDS (169.254.169.254) is reachable
only once mulga-mgmt-net brings up the data NIC via DHCP and the per-tap
br-imds datapath reconciles, which on the first-launched (primary) CP
can lag cloud-init's stock 120s probe window. With the default max_wait
cloud-init falls back to DataSourceNone, permanently losing the launch
user-data (no SPINIFEX_K3S_ROLE -> no k3s server) and wedging the cluster
in CREATING.
Bake a cloud.cfg.d drop-in that pins DataSourceEc2 and raises max_wait to
600s so cloud-init keeps polling until the datapath is ready instead of
falling through to None. Worst case a slow boot, never a lost role.
* fix(eks): warm IMDS instance-role creds before first SigV4 sign
On IMDS-credentials mode the on-VM SigV4 helpers (eks-token-webhook,
eks-gateway-publish, mulga-eks-state-report) sign via the SDK default
chain, which resolves in-VM IMDS instance-role creds. The per-tap
br-imds datapath lags VM boot by minutes (observed ~264s), and the SDK's
IMDS client fast-fails each probe (250ms dial), so early Retrieve calls
return 'context deadline exceeded' / 'no EC2 IMDS role found'.
eks-gateway-publish only retries for 30x5s=150s, shorter than the
datapath warm time, so it exhausts its budget and drops the
bootstrap/state publish -> cluster hangs in CREATING.
The credential chain is sound (the CP role trusts ec2.amazonaws.com and
AssumeRoleForInstance evaluates that principal); the failure is timing.
Add gwsign.Signer.EnsureCredentials and block in eksgw.New until the
instance-role creds first retrieve (2s interval, 10m ceiling aligned
with cloud-init max_wait), so every helper's first signed request
already holds creds instead of racing a cold datapath. Static-key mode
is unaffected.
* fix(eks): retry CP data-NIC DHCP until cross-host datapath is up
A single ~16s DHCP one-shot on the primary data NIC lost the race against
the cross-host OVN datapath (logical-port binding + per-tap reconcile on the
remote placement host), stranding the NIC IP-less so the Ec2 datasource never
reached IMDS and the CP member wedged. Add dhcp_acquire: retry the one-shot on
a budget aligned with cloud-init's Ec2 max_wait (600s), confirming an inet
address before success. Each attempt still exits leaving no daemon, so
cloud-init's later ENI management is unaffected.
* fix(eks): drop reconciler registry entry when Run self-exits
ReconcilerRegistry.Spawn only cleaned up its holder on ctx cancel
(Stop/StopAll). When a ClusterReconciler's Run returned on its own — a
terminal cluster state or a lost lease — nothing cancelled the ctx, so
the holder leaked and a later same-name create silently no-opped at the
head of Spawn, never reconciling to ACTIVE.
Thread a finished channel from RunClusterReconciler's Run goroutine back
to Spawn's watcher, which now cleans up on whichever fires first: an
external cancel or Run exiting. On self-exit it deletes its own holder
and releases the lease, so a fresh same-name create re-spawns and
reconciles.
* fix(eks): run mulga-mgmt-net in boot runlevel before cloud-init-local
CP user-data is delivered only via IMDS since the NoCloud seed was retired
(#411), so cloud-init DataSourceEc2 must reach 169.254.169.254. mulga-mgmt-net
DHCPs the data NIC that makes IMDS reachable, but it was added to the default
runlevel and only declared `before cloud-init` (the network-stage service).
cloud-init-local therefore ran its init-local Ec2 crawl first with no data-NIC
route and burned its whole probe window (~4 min) before mulga-mgmt-net brought
eth0 up, regressing CP bringup from ~1m50s to ~5m54s.
Add mulga-mgmt-net to the boot runlevel in setup.sh instead of default, drop it
from ENABLE_SERVICES, and order it `before cloud-init-local` so the data NIC and
IMDS route are up before the init-local crawl. Keeps the IMDS-only user-data
path; the raised Ec2 max_wait stays as a safety net for host br-imds datapath lag.
* fix(vm): refresh LaunchTime on crash-restart so watchdog spares relaunch
RestartCrashedInstance re-launched a crashed VM without refreshing
Instance.LaunchTime. The pending watchdog measures now-LaunchTime against
PendingWatchdogTimeout, so the stale original launch time tripped
launch_timeout on the next tick and terminated the crash-restart
mid-flight. Refresh LaunchTime under the state lock, mirroring the
daemon-restart restore path (restore.go), so the relaunch gets a fresh
timeout window. Adds a regression test asserting LaunchTime advances.
* feat(eks): reconcile control-plane member count after a lost member
The cluster reconciler self-heals a stopped/error control-plane VM in
place but never replaced a terminated/gone member, so an HA cluster ran
below haControlPlaneCount permanently. Add a member-count reconcile pass
that provisions one replacement CP joining the surviving etcd quorum.
- CPProvisioner + EKSServiceImpl.ProvisionReplacementCP: replay the
persisted create template on a free host, the single-node analogue of
the spread join branch, with rotating creds re-derived.
- ControlPlaneTemplate persisted on ClusterMeta at create (per-node and
rotating-cred fields cleared). SwapControlPlaneMember CAS swaps the dead
node for the replacement and refreshes the [0] scalar mirrors.
- maybeReplaceControlPlaneMember fires only when no member is restartable
and one is truly lost, a healthy majority survives, bounded by grace,
backoff and an attempt cap; placement excludes live-member hosts.
- Replacement prunes the superseded member from etcd once joined:
k3s-first-boot deletes the stale Node (EKS_ETCD_PRUNE_PEER_IP) so quorum
width returns to N. json tags added to K3sServerInput (now persisted).
* feat(eks): in-guest control-plane self-diagnosis on healthz fail
When a control-plane apiserver reports unhealthy, the reconciler only
saw healthz=fail with no cause, and the guest has no on-demand command
channel (serial root login is password-gated, no guest agent). Make the
CP self-diagnose and relay the cause through the one-way state report.
- mulga-eks-state-report.sh: diagnose() collects failing /readyz?verbose
subchecks, isolated /healthz/etcd reachability, and etcd-db-dir free
space; on fail write a verbose block to the host-captured serial
console and add a JSON-safe reason to the state payload.
- ServerStateReport: add Reason (omitempty; empty from a healthy CP or
an older AMI predating the field).
- cluster_reconciler observe(): surface the reason in the health issue.
- Tests: POSIX unit test for the agent (stubbed kubectl/df); Go tests
for the reason-bearing and terse health-issue paths.
* feat(eks): 15-min etcd snapshot cadence alongside nightly
Nightly alone is too coarse a recovery point for a young cluster. Install the
same mulga-eks-etcd-snapshot script into a second Alpine crond periodic dir
(/etc/periodic/15min) so it also runs on a ~15-minute RPO window; the script
picks its cadence tier from the dir it is invoked from ($0 sniff) and encodes
the tier in the snapshot key (etcd-{tier}-{ts}.snap), so a predastore S3
lifecycle rule can retain the frequent rolling window shallow and the nightly
copies deep. Server-role only (agents exit early); no restore-path change.
Add a self-contained POSIX unit test (stub k3s/curl/logger, symlinked crond
dirs to exercise the real $0 tier sniff) covering agent no-op, frequent + daily
key prefixes, local-snapshot prune, and quiet no-op without creds.
* feat(eks): auto-reform wedged etcd quorum via boot-time cluster-reset
An HA k3s control plane does not self-heal after a simultaneous full-CP
restart: embedded etcd never reforms quorum (etcd:unreachable sustained),
and in-place VM restarts cannot fix it. Add a last-resort escalation that
drives the documented k3s cluster-reset recovery, pull-based so the host
never needs an inbound channel into the guest.
Reconciler: maybeReformEtcdQuorum escalates only when every CP member is
VM-running but the health issue carries etcd:unreachable, the cluster is
HA, and a grace window has elapsed. It sets per-member recovery directives
(seed = cluster-reset, others = wipe-rejoin), epoch-bumped and keyed by
instance ID, then restarts the members. Bounded by grace/backoff/attempts;
a recovered cluster clears the directives once. Enabled via
WithEtcdResetRecovery alongside the existing CP control wiring.
Directive plane: per-member KV directive {epoch, action, snapshot} with
store helpers, an EKS service Get/Set pair over NATS-RPC, and an
internal-recovery gateway route (system-cred carve-out, same as
internal-addons). The CP server instance role gains eks:GetRecoveryDirective.
Guest: mulga-eks-k3s-recovery is a pre-k3s oneshot (server role only) that
fetches its directive through the AWS gateway (eks-gateway-fetch -resource
recovery), epoch-guards one-shot application, then runs
`k3s server --cluster-reset` (optionally restoring a predastore snapshot)
or wipes the local etcd datastore to rejoin the reset seed. Wired into the
image manifest, setup.sh perms, and the eks-node-role selector.
Tests: reconciler escalation gates + recovery clear, directive store epoch
increment + service validation, and a POSIX unit test for the guest agent.
* fix(eks): reboot running CP members to apply etcd cluster-reset directive
The etcd quorum-reformation escalation drove StartInstance on the seed and
followers, but the reset gate only fires when every member is VM-running. The
StartInstances path rejects a running instance with IncorrectInstanceState, so
the guest never re-entered its boot sequence and the cluster-reset/wipe-rejoin
directive was never applied — the wedged control plane stayed wedged.
Reboot the running members (QMP system_reset) instead, re-entering the boot
sequence so the on-VM recovery agent applies the directive. Adds RebootInstance
to the CP control surface (interface, adapter, NATS-backed daemon impl) and
switches maybeReformEtcdQuorum to it; the stopped/error in-place restart path
keeps StartInstance.
* fix(eks): graceful stop for etcd reset recovery
The etcd quorum-reformation reset path forced running CP members through a
hard QMP system_reset to re-enter the boot sequence and apply their recovery
directives. A hard reset skips a clean unmount: on the next boot e2fsck
cleared the recovery and k3s runlevel entries (dangling inodes), so the
cluster-reset never ran and k3s was unstartable — the reset defeated recovery.
Replace the reboot with a graceful StopInstance (QMP system_powerdown, clean
unmount, SIGKILL fallback). The existing in-place restart path
(maybeRecoverControlPlane) then boots each stopped member clean on a later
tick, and the on-VM recovery agent applies its directive against an intact
filesystem. Reuses existing lifecycle machinery; the stop also tolerates a
wedged QMP client via the SIGKILL fallback.
Swaps RebootInstance -> StopInstance across CPInstanceControl,
cpInstanceController, the cpControlAdapter, and the NATS-backed eksCPControl
(gateway RebootInstances -> StopInstances), plus the reset/recovery test fakes.
* fix(vm): retry QMP dial and unlink stale socket on relaunch
A QEMU killed with SIGKILL (crash or etcd-reset restart) leaves its
qmp-<id>.sock inode behind. On relaunch the startup os.Stat probe and the
QMP dial both raced that dead listener, so a single dial returned
'connection refused': StartStoppedInstance then reported ServerInternal and
left the instance KV-stopped even though the guest booted fully. The wedged
member stayed invisible to the reconciler's cpLive gate, stalling recovery.
Unlink the stale socket before launch so the inode that appears is the fresh
QEMU's, and retry the QMP connect with a short backoff to cover the window
where the socket is bound but not yet listening. Handshake failures are not
retried — a listener answered, so the error surfaces immediately.
* feat(eks-ui): surface cluster Health.Issues on the EKS console
DescribeCluster is AWS-faithful (Status stays ACTIVE, degradation lives in
Health.Issues), but the cluster detail page only rendered the status badge, so
a degraded control plane still showed as ACTIVE with no visible signal. Add a
destructive health badge next to the status and a Health row that lists the
issue messages, mirroring the addons tab. Healthy clusters read OK.
* chore: rebuild ui
* fix(viperblockd): stop mount-path daemon VB uploader to prevent checkpoint clobber
The daemon-side VB created in the ebs.mount handler is stored in
MountedVolume.VB for the whole mount, and its background chunk uploader was
only stopped at daemon shutdown. That state-only VB ran DrainToBackend every
30s, overwriting the live checkpoint the nbdkit plugin VB (the data-path
writer) keeps updating, leaving SeqNums inconsistent with sealed chunk
ciphertext and causing transient AEAD auth failures on cloned encrypted
volumes. Stop its uploader right after New, matching the other short-lived VB
call sites.
Also bump viperblock to the atomic chunk-ObjectID allocation fix.
* build(deps): pin viperblock to landed AEAD concurrency fix
Bump github.com/mulgadc/viperblock to the dev commit b08bea6 that
eliminates "cipher: message authentication failed" on encrypted volumes
under concurrent read-modify-write + background chunk drain. Removes the
local-path replace used while the fix was in flight.
* fix(viperblock): stop chunk-uploader goroutine leak at all VB call sites
viperblock.New() unconditionally starts a background chunk-uploader
goroutine (30 s interval) that only VB.Close() stops. Seven call sites
created short-lived or state-only VB instances without calling Close(),
leaking one uploader goroutine per instance. On a single node the leaked
goroutines exhaust memory (predastore RSS grew ~1.3 GB/5 min at idle
post-single-suite), causing liveMemGate to reject launches and the lb
suite to fail.
The leaked uploaders are also the primary cause of the AEAD auth
failures (mulga-be2ev): a leaked VB from prepareRootVolume holds the
clone-point BlockLookup and overwrites the correct live checkpoint in S3
every 30 s, leaving SeqNums inconsistent with sealed chunk ciphertext.
Fix all seven sites:
- instance/service_impl.go prepareRootVolume: defer StopChunkUploader +
StopWALSyncer after successful viperblock.New (mirrors the EFI path's
intent; full Close is unsafe here because cloneAMIToVolume already
called SaveBlockState + RemoveLocalFiles)
- viperblockd.go ebs.delete handler: add StopChunkUploader alongside
existing StopWALSyncer
- viperblockd.go ebs.unmount handler: same
- viperblockd.go shutdownVolumes: same
- snapshot/service_impl.go snapshotVolume: defer both stops after New
- volume/service_impl.go CreateVolume: defer both stops after New
- image/service_impl.go snapshotRunningVolume: defer both stops after New
- image/service_impl.go snapshotStoppedVolume: defer both stops after New
Closes mulga-0zzci
Closes mulga-ji5fi
* fix(deps): bump viperblock to 01a7e686 (AEAD regression fix)
The prior pin (6af89855) carried the snapshot-AMI AEAD regression that
corrupts block storage on restore. Point at the merged viperblock dev fix.
---------
Co-authored-by: joshsiv-mulga <mrk@mulgadc.com>
Failed LBs previously reported an empty State.Reason, so e2e failures
read 'entered terminal state failed:' with no cause. Record the cause
on every failed transition — launch error (e.g. insufficient sys.micro
capacity), missing system credentials, missing internal mgmt return
route, and async-launch panic — persist it as StateReason, and map it
to LoadBalancerState.Reason in describe responses. Cleared when the
first heartbeat transitions the LB to active.
The internal lb subtests raced the previous subtest's sys.micro VM
deallocation: fixed 15s sleeps were not enough on capacity-tight
single nodes, so the LB entered terminal state failed (insufficient
capacity, then nats no-responders after the daemon unsubscribed the
full topic).
Replace the sleeps with createActiveLB: create LB + listener, wait for
active via the new non-fatal harness WaitForLBActiveErr, and on a
terminal failed state (wraps ErrLBTerminalFailed) tear down fully and
retry with backoff, up to 3 attempts.
go.mod referenced v1.10.0 which pre-dates LoadLiveCheckpoint. CI builds
without go.work so e2e-build failed to compile. Pin to viperblock dev tip
6af8985 (feat/snapshot-ami merged) until v1.11.0 is tagged.
* feat(gpu): add H200/RTX Pro 6000 SE models and fix MIG host resource allocation
- Fix H100 SXM/PCIe device ID labels swapped in models.go (10de:2330=SXM, 10de:2331=PCIe)
- Add H200 SXM (10de:2335) to knownModels, computeModels, and migCapableModels
- Add NVIDIArtxPro6000BlackwellSE to definitions.go mapped to g7e family (matching AWS g7e instance type)
- Add MIGHostResources() policy table: 4 vCPUs / 32 GiB host RAM per compute slice, covering all known NVIDIA MIG profiles
- GenerateMIGTypes now emits non-zero vCPU/memory for EC2 instance type descriptors
* fix(daemon): enforce CPU/memory limits for MIG instance types
Whole-GPU passthrough remains bypass-only (gpuManager.Claim is the sole gate).
MIG types now fall through to canAllocateCount so host CPU and RAM are checked
before launch — prevents over-committing a host where one physical GPU backs
multiple concurrent slices, each with non-trivial vCPU/memory allocation.
* feat: Add snapshot image promotion flow
* fix: Small changes to viperblock/spinifex so snapshot captures delta properly and instances can boot from snapshot AMIs
* fix: test coverage and more consistent prefix check
* fixed testing
* more tests
* fix: added new graceful process killer in utils, bumped NATS timeout for testing
* fix: complete at-rest encryption launch + volume lifecycle (#361)
* chore: bump viperblock to encrypted-snapshot VolumeUUID nonce fix
Pull viperblock 1044c2a (feat/snapshot-uuid-nonce-fix) so E2E builds against
the fix for the encrypted AMI snapshot metadata tag-verify failure
(cipher: message authentication failed) on instance launch. Pairs with
viperblock PR #22.
* fix: forward viperblock master key to nbdkit mount path
viperblockd loaded the master key but never passed it to the nbdkit
plugin it spawns for volume mounts, so encrypted boot volumes opened
in cleartext and failed LoadState with an encryption-state mismatch.
Forward cfg.EncryptionKeyFile into NBDKitConfig and emit the plugin's
encryption_key_file= argument when set.
* chore: bump viperblock to fresh-volume VolumeUUID mint fix
* chore: bump viperblock to snapshot-link LoadState ordering fix
* fix: route encrypted-volume config writes through viperblock keyholder
config.json is a sealed VBState whose AES-GCM tag and StateSeqNum-derived
nonce can only be advanced by the master-key holder that owns the volume.
The EC2 volume handler wrote it directly to S3, so mergeVolumeConfig had to
refuse encrypted volumes (rewriting would strip the tag) — bricking
UpdateVolumeState (attach in-use / detach available) and ModifyVolume.
Route encrypted-volume config updates through viperblockd, mirroring the
ebs.snapshot idiom: per-volume ebs.config.{volumeID} hits the owning node's
live VB (the single StateSeqNum writer); on ErrNoResponders the ebs.config
queue group opens the detached volume exclusively and reseals. Cleartext
volumes keep the direct-S3 fast path. No crypto leaves viperblockd.
mulga-siv-323
* fix: unwrap encryption envelope when snapshot handler reads config.json
CreateSnapshot and snapshotInUseByVolumes decoded config.json straight into
a VBState. For encrypted volumes config.json is an at-rest envelope
{payload, authtag}; a raw decode yields a zero VBState (SizeGiB==0), so
CreateSnapshot 500s on the size guard and the in-use scan silently drops
encrypted volumes' snapshots. Unwrap via viperblock.StateBody first,
mirroring the volume handler's mergeVolumeConfig path.
mulga-siv-324
* test: cover viperblockd ebs.config handlers
Unit/integration coverage for applyConfigUpdate, makeConfigUpdateHandler
(ebs.config.{volumeID}), and the ebs.config queue-group fallback: grow-only
size reconcile, live-VB apply, invalid JSON, apply errors, and the
detached-volume open path. Clears the diff-coverage gate (68.8%).
* chore: pin viperblock to dev tip (#22 merged)
* fix(viperblockd): seal volume block-map to predastore on detach + durability e2es (#371)
* feat: cloud-init EC2 datasource (#411)
* feat: Add snapshot image promotion flow
* fix: Small changes to viperblock/spinifex so snapshot captures delta properly and instances can boot from snapshot AMIs
* fix: test coverage and more consistent prefix check
* merge changes
* chore: remove dead SnapshotSub field and SIGTERM snapshot remnants from viperblockd
SnapshotSub was added during the NATS/SIGTERM snapshot attempts but is never
set post-rebase. Removes the field from MountedVolume and the two nil-checked
unsubscribe blocks. Re-adds the nbd import which is still needed for NBDKitConfig.
* fix: post-rebase gofmt fixes and remove dead snapshot handler code
Split gofmt-squashed lines in go.mod, vm_adapters.go, nbd_test.go,
viperblockd.go, volumes.go, and volume_durability_test.go. Removes the
dead unix-socket snapshot handler tests from viperblockd_handlers_test.go.
* test: add viperblock snapshot S3 coverage tests
Exercises snapshotVolume, snapshotRunningVolume, and snapshotStoppedVolume
past the nil-config guard using an httptest server that returns 403.
The AWS SDK credential check fails synchronously without network I/O,
keeping the tests fast while covering the S3Config construction, master key
load, and viperblock.New code paths. Brings diff coverage from 45% to 69.9%.
* fix: cloneAMIToVolume reads config via format-agnostic path
LoadStateRequest expects a viperblock binary envelope (encrypted),
but CreateImageFromInstance writes the AMI config as plain JSON via
putAMIConfig. Replace LoadStateRequest with s.amiLoader.GetAMIConfig
which uses viperblock.StateBody to handle both formats, matching the
path used by ListImages and all other AMI readers.
* fix: pin viperblock to feat/snapshot-ami pseudo-version for LoadLiveCheckpoint
go.mod referenced v1.10.0 (released from dev) which pre-dates LoadLiveCheckpoint.
CI builds without go.work so it resolved to the published version and failed.
Pin to the snapshot-ami HEAD pseudo-version until the branch is merged and tagged.
* fix: replace indexOf with slices.Index, revert e2e runner and viperblock version
- nbd_test.go: remove local indexOf helper, use slices.Index from stdlib
- e2e-suites.yml: revert ubuntu-latest back to ubuntu-26.04 with correct formatting
- go.mod: revert viperblock back to v1.10.0 (will resolve once viperblock PR merges)
* fix: remove pseudo-version pin from go.sum
* feat: drain viperblock via snapshot socket before CreateSnapshot
Before LoadLiveCheckpoint(), dial {data_dir}/viperblock/{vol}/snapshot.sock
and wait for DrainToBackend() to complete (30s timeout). This ensures WAL
chunks and the live checkpoint are current in S3 before the snapshot VB
reads them. If the socket is absent the instance is stopped and Close()
already drained.
fix: socket path used s.config.WalDir which is empty in the daemon config;
changed to s.config.DataDir + "/viperblock" to match the path the NBD
plugin uses when creating the socket.
* feat: wire snapshot/AMI UI actions and add PromoteImage NATS endpoint
- Register as AMI from snapshot detail (RegisterImage with BDM pointing at snapshot)
- Deregister from image detail (DeregisterImage with navigate-away on success)
- Promote to System from image detail (admin-only, irreversible via UI)
- NATS subject spinifex.image.promote wired through gateway → daemon → existing admin.PromoteSystemImage
- signedFetch extended with optional params to support action-keyed extra fields
* refactor: address PR review on PromoteImage types and subscription ordering
- Merge PromoteImageInput/PromoteImageOutput into PromoteImageOpts/PromoteImageResult
(add JSON tags; drop the duplicate pair)
- Inline the input variable in gateway/spx/promote_image.go
- Move spinifex.image.promote subscription to the spinifex.* group in daemon.go
* test: cover snapshotVolume/Running/Stopped error paths for diff coverage
Add encryption-key-load-error and LoadState-failure tests for the three
new viperblock snapshot paths (snapshotVolume in snapshot service,
snapshotRunningVolume and snapshotStoppedVolume in image service).
Diff coverage: 58.8% → 65.5%.
---------
Co-authored-by: Josh Siviour <mrk@mulgadc.com>
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
* fix: ECS StopTask reaps container before releasing capacity
StopTask marked a task lastStatus=STOPPED and released its capacity/ENI in
the control plane, but nothing ever told the ecs-agent to kill the container.
The container kept holding its host ports, so DescribeTasks reported a STOPPED
task whose container was still RUNNING, and a replacement task placed on the
freed capacity that rebound the same host port hit EADDRINUSE.
Wire the agent-driven cooperative stop the design intended (ecs-v1.md:165):
- Add a per-instance stop inbox (StopsPrefix/StopKey) and bus.StopDirective.
- requestStopTask marks DesiredStatus=STOPPED and posts a stop directive,
leaving lastStatus/capacity/ENI/targets untouched. StopTask, DeleteService,
and service scale-in use it; the heartbeat reaper and cluster/instance
teardown keep the immediate forceStopTask since no live agent can reap.
- PollAssignments returns pending stops; the agent lists containers by
mulga.ecs.taskID label, Removes them (kill+delete, freeing ports/netns),
then reports STOPPED. recordTaskState performs the single capacity release
on the agent's STOPPED report and reclaims the stop inbox.
- listServiceTasks excludes DesiredStatus=STOPPED tasks so a cooperatively
stopped task drops out of the service's running total immediately.
DescribeTasks now keeps the task RUNNING with desiredStatus=STOPPED until the
container is actually reaped, matching AWS ordering and eliminating the port
race.
* test: cover StopTask reap paths — poll stop-inbox drain, unplaced force-stop, agent stopTask edges
* ci: retrigger PR checks
- RegisterTaskDefinition rejects container secrets[] (InvalidParameterException)
and warns on unsupported log drivers instead of silently dropping them.
- Capture logConfiguration; only host-side json-file is collected, other drivers
warn at register and reach the agent for parity. Document json-file retrieval.
- Plumb executionRoleArn taskdef -> assign -> agent; the agent assumes it to
authorize ECR pulls, falling back to the instance role when unset.
- Add deployment objects with a health-gated rolling update honoring
deploymentConfiguration minimumHealthyPercent/maximumPercent, plus a deployment
circuit breaker with optional rollback to the last-good task definition.
EnsureLogicalSwitch/Router/PortGroup returned a client-side named-uuid on the
fresh-insert path but the real server uuid when the row already existed, so a
caller storing the returned uuid then doing Where(uuid) would miss. Resolve the
persisted uuid from the monitored cache after insert and add an explicit created
flag; topology callers now branch on created instead of uuid equality.
ReplaceACLs deleted and recreated the whole ACL set every reconcile, minting new
row uuids even for an unchanged SG and forcing ovn-northd flow recompute on every
drift tick. Diff current vs desired ACLs and no-op when unchanged (live + mock).
* test(network): in-process OVN NB reconcile harness
Add ovntest.StartNB: an in-process OVN Northbound DB backed by libovsdb's
pure-Go OVSDB server (no ovsdb-server/ovn-northd C daemons) plus a vendored
NB schema subset matching nbdb. Real LiveClient connects over a unix socket,
so reconcile scenarios exercise intent generation, reconcile wiring, LiveClient
and OVSDB transaction semantics in milliseconds without VMs.
Add live reconcile scenarios asserting NB rows and idempotency:
- VPC + subnet + SG + ENI from a hand-built IntentState
- the same seeded through JetStream KV via LoadIntentFromKV (front-to-back)
- IGW attach + drop gate + public-instance exemption (Logical_Router_Policy)
The idempotency snapshot normalizes UUID and pointer churn so it asserts
logical stability; datapath and NB->SB translation stay in the e2e suites.
* test: add orphan-prune and kitchen-sink golden live NB scenarios
Extend the in-process OVN reconcile harness with the two P0 completeness
scenarios. TestScenario_OrphanPrune_Live locks the teardown path and the
ReconcileApplyOnly startup-race guard: apply-only must not prune, full
Reconcile must, and re-pruning is idempotent. TestScenario_KitchenSinkGolden_Live
reconciles a maximal intent exercising every applyX path (VPC, public+private
subnets, two ENIs, SG with tenant rules, IGW, associated EIP, NAT gateway,
IGW/NATGW reroutes, drop gates) and asserts the normalized NB snapshot against
a checked-in golden, mechanically catching an unwired or mis-wired resource
type. Regenerate the golden with UPDATE_GOLDEN=1 go test.
* fix: repoint DB-node ovn-northd at RAFT member list; extend add-nat deadline
Two related fixes for multi-node OVN clusters:
- setup-ovn.sh (mulga-siv-504): in the clustered --management branch, point
ovn-northd's client NB/SB connections at the full RAFT member list derived
from OVN_REMOTE, so the active northd follows the leader instead of dialing
its local member only. Without this, northd stops advancing SB_Global.nb_cfg
once leadership moves off the node, wedging the ovn-nbctl --wait=hv barrier.
Guarded on a real member list so the standalone single-DB-node path is
unchanged.
- utils/nats.go (mulga-js-206): raise the add-nat request deadline from 10s to
45s so the client no longer rolls back before the server-side --wait=hv flows
barrier (30s) completes.
* fix: stop ovn-northd restart from tearing down split NB/SB ovsdb servers
The packaged ovn-northd.service ExecStop runs 'ovn-ctl stop_northd' without
--ovn-manage-ovsdb=no, so restarting northd to apply OVN_CTL_OPTS also stops the
separately-managed NB/SB ovsdb-server units. On a clean clustered bring-up this
races the freshly-started DBs and kills them, so setup-ovn's wait for the NB
socket times out and the RAFT create/join aborts. Install an ovn-northd.service
drop-in overriding ExecStop with --ovn-manage-ovsdb=no so northd never touches
the DB lifecycle (mulga-siv-504).
* test(e2e): capture SNAT datapath on NAT GW egress failure
TestNATGateway captured nothing when the private-VM egress probe failed, so
the baremetal nightly's intermittent 100% packet loss left no OVN/OVS state in
the failure bundle to root-cause it. Register harness.OnFailure before the
egress-verify poll to dump the SNAT datapath (OVN show, NAT rules, port
bindings, br-int flows, conntrack, ARP) keyed on the NAT GW external IP,
matching the eip/vpc datapath tests.
Also add a br-ext localnet flow capture to dumpDatapathState: br-int flows miss
the egress hop where SNAT'd traffic leaves for the physical uplink, which is
exactly where an egress black-hole would show.
* test(e2e): add OVN pipeline diagnostics to NAT GW egress capture
Capture the gateway router's per-stage logical-flow packet counts
(ovn-sbctl --stats lflow-list), MAC_Binding, static routes and the VM's
logical switch port alongside the existing OVS/conntrack/ARP dumps. When
the SNAT flow reads n_packets=0 while the guest transmits, these pinpoint
which pipeline stage drops the packet upstream of lr_out_snat.
* test(e2e): dump full lflow-list + datapath ct/flows for egress RCA
The IP-filtered lflow dump hides the default-drop ACL flow (its match
carries no IP), so a pre-SNAT egress drop at ls_out_acl only showed the
allow-flows at n_packets=0, never the deny that ate the packets. Replace
it with a full ovn-sbctl --stats lflow-list, echoed so it survives in the
CI stdout log. Swap the conntrack capture to ovs-appctl dpctl/dump-conntrack
(the standalone conntrack binary is absent on the baremetal runner) and add
dpctl/dump-flows -m for kernel megaflow hit counts.
* fix: claim gateway cr-port on every reconcile miss and stop masking launch failures
ensureGatewayClaimed nudged ovn-controller recompute only once then polled
until timeout, so a fresh VPC bring-up or chassis flap that fired the single
nudge too early left the gateway chassisredirect port permanently unclaimed.
Recompute on every miss instead, mirroring ensureGuestPortDatapath.
aggregateResults and distributeInstancesSpread masked any node launch failure
(including the resulting 5-minute launch timeout) as InsufficientInstance
Capacity. Surface the real node error when one exists; keep the capacity code
only for genuine capacity shortfalls.
* feat(eks): multinode control-plane networking, IAM, and CoreDNS parity
Integrates the EKS multinode fix set for the managed-control-plane topology
so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS
EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the
node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it.
EKS:
- Route apiserver egress (exec/logs/webhooks) through konnectivity
(apiserver-network-proxy): CP servers run konnectivity-server behind the
cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod
traffic rides the agent-initiated reverse tunnel instead of an unroutable
CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every
replica (HA-correct).
- Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay
flannel mesh SG rule authorized at cluster launch with ordered teardown.
- Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay.
- Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master
tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity.
- Spread nodegroup workers across hosts; addon readiness falls back on
workload rollout when gvks are absent.
- elbv2: allow the proxy_protocol_v2.enabled target-group attribute.
EBS:
- Move volume attachment state to state.json so a live nbdkit VB cannot
clobber it; route mounted-volume state updates through the live VB with a
retry for in-use marks racing the mount responder; keep the boot volume
in-use across stop/crash unmount.
* fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM
missingOrchestrationDeps gated on s.deps.IAM, but production wires only the
lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only).
The gate therefore reported missing=[IAM] on every node forever, so
CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even
though the IAM service was fully up via the provider.
Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else
deps.IAMProvider()) the nodegroup/role code already uses — so the protective
check still rejects a node with no IAM service but passes when the provider
supplies one. Add a regression test that sets only IAMProvider and asserts the
gate is satisfied; the existing nil-both test still rejects.
* fix(eks): let eks-token-webhook use IMDS creds when static keys absent
The CP VM launches on an instance profile, so buildK3sUserData omits
EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys
before eksgw.New could fall back to the IMDS SDK chain, so the webhook
aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver
crashed on the missing --authentication-token-webhook-config-file file,
stalling the cluster at CREATING until it timed out to FAILED.
Drop the static-key precondition (matching the sibling eks-gateway-publish
and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/
cluster gates.
* fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane
The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt,
which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a
169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing
169.254.169.254 and steering IMDS off the data NIC away from the per-tap
br-imds datapath, where the host RST it. eks-gateway-publish then failed to
sign requests and no bootstrap artifacts were ever published, so clusters sat
CREATING until they timed out to FAILED.
Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address
or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI)
are unaffected and IMDS falls through the data NIC's default route to the
br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net
deletes any default route on a NIC declared NIC<n>_DEFAULT=0.
* fix(eks): grant eks:WebhookTokenReview to the control-plane instance role
The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway
broker at POST /clusters/{name}/token-review, signing with the control-plane
VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes
against eks:WebhookTokenReview, but the role's inline policy granted only
eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every
relay and the apiserver returned 401 to all external kubectl.
Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile
re-asserts the inline policy on every control-plane instance-profile ensure, and
the gateway evaluates authorization per request, so existing clusters converge
without a rebuild once the shared role is updated.
* feat: add OVSDB RAFT cluster primitives to setup-ovn.sh
Add --db-cluster-local-addr / --db-cluster-remote-addr flags so a
management node can run the OVN NB/SB databases as a native OVSDB RAFT
cluster instead of a single non-clustered ovn-central. When a local addr
is given, write a shared OVN_CTL_OPTS to /etc/default/ovn-central (NB
RAFT 6643 / SB RAFT 6644) and restart the per-DB units directly, since
the ovn-central aggregator is ExecStart=/bin/true and won't restart its
children. Empty remote addr creates the cluster (init node); a set remote
addr joins it. NB/SB client set-connection runs only on the create node
(RAFT replicates it cluster-wide).
The single-node --management path is unchanged when no cluster addrs are
supplied, so dev boxes and existing single-node clusters keep working.
--ovn-remote already passes through verbatim, so a comma-separated SB
endpoint list works for compute nodes; documented in usage.
* feat: OVN client multi-endpoint failover for RAFT clusters
Accept a comma-separated OVN NB/SB endpoint list so vpcd can target a
clustered DB. config.ParseEndpoints splits the list (trimming whitespace,
dropping empties); live.go Connect passes one libovsdb WithEndpoint per
endpoint, so the client tries them in order, fails over on disconnect, and
follows RAFT leader redirects. Single-endpoint strings still yield one
endpoint, so single-node dev is unchanged.
ovn-sbctl --db= already accepts the raw comma list, so the SB CLI
consumers (discoverChassis, gateway_claim) need no change.
* feat: write clustered OVN NB/SB endpoints across the RAFT quorum
spx admin init/join now emit comma-separated OVN NB/SB endpoint lists for
the first three quorum nodes (formation.BuildOVNDBAddrs) instead of loopback
or a single leader address, so vpcd fails over across the cluster.
firstboot defers OVN bring-up to the provisioning controller when it owns
formation (SkipFormation): a standalone single-node .db would otherwise block
the controller's clustered create/join.
* test: add multinode OVN RAFT cluster + failover e2e; document clustered topology
ovn_raft_test.go asserts NB/SB cluster/status reports a 3-server quorum with
one leader on every DB node, then stops the NB leader's ovn-central and proves
an NB write still lands via a surviving endpoint.
Docs: vpc-networking gains the multi-endpoint vpcd config and a RAFT
cluster-status debugging section; network-connections lists the 6643/6644 RAFT
ports.
vpcd's libovsdb NB client was a single long-lived connection with no
reconnect logic. After the NB server restarted (host reboot, service
restart) every vpcd holding a dead/half-open socket never re-established
it, so OVN-write handlers blocked until the caller timeout and CreateVpc
returned ServerInternal intermittently (queue-distributed across vpcd).
Add WithInactivityCheck to the OVSDB client: a periodic Echo probe
detects the dead peer and triggers reconnect against the configured
endpoint, with infinite backoff so a daemon keeps retrying across a long
outage. Bound the SG subscriber handlers with a timeout context so a
stuck OVN op frees the worker instead of hanging on context.Background.
NLBs hard-rejected customer security groups and always minted a managed SG that
auto-opened listener ports, so aws-load-balancer-controller (which assigns SGs to
type=external NLBs) got a 400 and looped. Accept up to 5 customer SGs on an NLB:
when supplied they replace the managed SG on the ENIs and the caller owns the
listener-port rules (CreateListener already no-ops its auto-open when there is no
managed SG); without them the managed-SG default is unchanged. SetSecurityGroups
follows AWS create-time semantics — replace SGs on an NLB created with SGs, reject
adding them to one created without.
CreateVolume hardcoded iops to 3000, silently dropping a caller-supplied Iops,
and rejected non-gp3 VolumeType with the generic InvalidParameterValue. Honor
input.Iops within the gp3 envelope (3000 baseline, 500 IOPS/GiB, 16000 ceiling)
and return the volume-type-specific UnknownVolumeType for unsupported types.
Closes EBS CSI API gaps #3-4 (siv-380); gp3 Throughput (gap #2) deferred.
* feat: IMDS-capable gateway signer for lb-agent + EKS clients
Add internal/gwsign, a SigV4 signer that draws credentials from either
fixed static keys or the AWS SDK default credential chain. In a guest VM
the chain falls back to the in-VM IMDS endpoint, yielding scoped,
auto-rotating instance-role credentials; a session token is signed as
X-Amz-Security-Token so the gateway can verify temporary (ASIA) creds.
Wire it into the lb-agent (lbagent.Agent) and the EKS on-VM client
(internal/eksgw). When static keys are absent the clients now sign with
IMDS credentials instead of failing; when present, behaviour is
unchanged (static path). This is the client-side foundation for dropping
long-lived secrets from VM user-data.
Credentials are retrieved per request so rotated IMDS creds take effect
without an agent restart.
* feat: IMDS instance-role creds for EKS server + LB VMs (Phase B/C)
Attach a Spinifex-managed system instance profile to the k3s control-plane VM
and the elbv2 LB VM so both authenticate to the gateway with scoped, rotating
IMDS instance-role credentials instead of a long-lived static secret baked into
user-data.
- Add a shared handlers/iam.EnsureSystemInstanceProfile that find-or-creates a
role (EC2 trust) plus its instance profile and asserts an inline permission
policy. The gateway enforces per-action authz on assumed-role principals, so
the roles carry exactly the actions each VM calls:
- EKS server: eks:PublishInternal, eks:ListInternalAddons.
- LB agent: elasticloadbalancing:LBAgentHeartbeat, GetLBConfig.
- Plumb IamInstanceProfileArn through sysinstance.SystemInstanceInput and both
daemon launch paths (BootAMI via PrepareRunInstances->RunInstance; direct-boot
microVM via RunInstance), which persist it so the in-VM IMDS endpoint serves
the role.
- Drop the baked secrets when a profile is attached: omit EKS_ACCESS_KEY/
EKS_SECRET_KEY from the k3s first-boot env and LB_ACCESS_KEY/LB_SECRET_KEY from
the lb-agent fw_cfg env; relax k3s server validation to accept profile-only;
drop the EKS key requires from the first-boot and state-report scripts.
- Both verticals fall back to baked static creds when IAM is unwired, so an
absent master key never blocks a launch.
* fix: resolve IMDS instance-role creds under the instance account for system VMs
Three boot-path defects kept the LB VM's lb-agent from authenticating to the
gateway with IMDS-served instance-role creds; each produced a silent empty IMDS
role list and a heartbeat failing with "unexpected empty EC2 IMDS role list".
Found and fixed against env19.
- Build the system IAM ensurer lazily on the first system-VM launch instead of
eagerly at daemon startup, where it raced the NATS KV backend and fell back to
baked static creds permanently. ELBv2 and the EKS deps now hold an IAMProvider
resolved at launch time.
- Create the LB instance profile in the system account the LB VM runs under
(utils.GlobalAccountID), not the LB-owner account. IMDS resolves the profile
under the instance's account, so a customer-account role was invisible to it.
- Record the attached instance's owning account on the ENI (InstanceOwnerId,
mirroring AWS Attachment.InstanceOwnerId) and resolve the instance and every
IAM/STS lookup under it. The LB ENI lives in the customer account while the
system LB VM and its role live in the system account; keying the whole IMDS
chain off the ENI account left the instance invisible and the role lookup
missing. VPC leaves (subnet/VPC/SG CIDR) keep using the ENI account.
A fresh LB on env19 then authenticated end-to-end: IMDS serves a non-empty role
list plus ASIA creds, and the gateway accepts the heartbeat as an assumed-role
principal with no static keys in user-data.
* test(e2e): assert LB VM authenticates via IMDS instance-role creds
The LB suite's targets-healthy assertion already depends on the lb-agent
authenticating to the gateway, and with static keys dropped from user-data that
auth is IMDS-only. Add an explicit check that a recent gateway authentication
for the LB used an assumed-role (ASIA) principal, so a regression to the
instance-account IMDS resolution fails with a clear message instead of a
mysterious datapath timeout. The LB VM is a system-account instance invisible to
the customer AWS API, so the assertion observes the gateway journal on the
cluster nodes; it is time-bounded to avoid the stale-console-log false-fail and
skips when no node is SSH-reachable.
Cell 19 bootstraps with --external-mode=nat plus the CI static pool as
the public wan pool and runs the natuplink suite, covering the Tier-1
transit datapath and Tier-2 EIP ingress. The run step exits 0 when
natuplink.test is absent from the tests tar so scheduled runs against
branches predating the suite stay green.
Add a second build+publish step to the EKS AMI workflow so the same
manual run also builds the ecs-agent system AMI and uploads its qcow2 +
.sha256 sidecar to R2. Reuses the shared checkout, Go, and tooling setup;
renames the workflow to cover both node images.
Flip the k3s server apiserver to anonymous-auth=false. The only
consumer of the anonymous path was mulga-eks-state-report's /healthz
probe; switch it to an authenticated kubectl get --raw=/healthz through
the node admin kubeconfig (an unauthenticated curl now returns 401).
The NLB target group uses a TCP health check, and the host-side
reconciler rides the authenticated NATS state-report, so neither
depends on the anonymous apiserver path.
* fix: free host-netns :80 for ECS bridge/host tasks via cred-endpoint DNAT
bridge and host network-mode tasks share the VM host netns where the
ecs-agent credential endpoint bound 169.254.170.2:80, so a task binding
0.0.0.0:80 hit EADDRINUSE and crash-looped. Bind the listener on the high
proxy port 51679 and DNAT 169.254.170.2:80 to it (OUTPUT for host/bridge
task traffic, PREROUTING for the awsvpc credential veth), keeping the dst
IP on the endpoint so no route_localnet is needed. Nothing holds a socket
on :80, so host/bridge tasks can bind it.
e2e runs nginx on :80 in all three network modes; the settle guard catches
a regression where a crash-looping :80 bind flaps RUNNING then STOPPED.
* test: validate ECS host-netns :80 binding via bridge task
The bridge subtest now runs nginx on :80 to prove the cred-endpoint DNAT
frees the host netns (pre-fix it crash-looped on EADDRINUSE). host mode
keeps a no-port command: it shares the same host netns, so two :80 tasks
cannot coexist on one node (one :80 per host, as on AWS), and StopTask
does not synchronously reap the prior container's port.
When NATS disables JetStream at runtime (e.g. a filesystem permission error
flushing stream state) it keeps the process alive and serves KV as read-only.
That is not a process failure, so spinifex-nats.service's Restart=on-failure
never fires, and KV-backed startup (awsgw IAM bucket init) stalls for minutes
until an operator restarts NATS by hand.
Add a JetStream liveness watchdog:
- build/scripts/nats-js-watchdog.sh probes the NATS monitoring endpoint
(http://127.0.0.1:8222/healthz?js-enabled-only=true). Only when the service
is active and JS fails every sample of a short multi-sample window (so
transient startup/quorum 503s are ignored) does it systemctl try-restart
spinifex-nats.
- spinifex-nats-watchdog.{service,timer} run it as a oneshot every 30s, with a
60s boot delay so cluster quorum settles first.
- spinifex.target Wants= the timer so it starts with the platform.
The NATS-internal permission race itself is a server quirk outside spinifex's
control; this restores service automatically when it recurs.
The in-cluster AWS LoadBalancer Controller provisions its own ALBs
(k8s-toc-*) and security groups (k8s-traffic-toc-*) in the customer VPC
via the awsgw ELBv2/EC2 APIs. Spinifex never tracks these, so cluster
teardown left them orphaned: DeleteSecurityGroup / DeleteVpc then failed
with DependencyViolation and the cluster wedged in DELETING.
purgeClusterInfra now reaps them by the AWS-standard ownership tag
elbv2.k8s.aws/cluster=<name>, scoped to the customer VPC, after the
cluster NLB teardown and before the SG / VPC block:
- ReapLBCLoadBalancers lists account LBs, filters to the customer VPC,
confirms the ownership tag via DescribeTags, and deletes by ARN.
- DeleteClusterSGs deletes any VPC SG carrying the ownership tag (reusing
the DependencyViolation retry budget), in addition to the named cluster
SGs, after rules are revoked.
- DescribeTags is added to the narrow nlbProvisioner interface.
Errors join teardownErrs so a failed reap keeps the cluster in DELETING
for a retry rather than completing with leaked, VPC-pinning infra.
TerminateWorkerInstances routes terminates via the per-instance ec2.cmd.<id>
topic, whose subscription is torn down when a VM stops. A stopped (incl.
stopped+wedged, volume remount broken) worker therefore has no responder, and
the old code treated ErrNoResponders as already-gone and returned nil. The
worker was still in shared KV with its ENI intact, so the nodegroup record was
deleted while the ENI kept pinning the customer subnet/VPC, wedging DeleteCluster
in DELETING on DependencyViolation until a manual ec2 terminate-instances.
On ErrNoResponders, fall back to the ec2.terminate queue group (mirroring the
gateway TerminateInstances path), which runs TerminateStoppedInstance against
shared KV — deleting the worker's volumes, IP, and ENI without a running QEMU.
NotFound or no responder is idempotent success; any other error surfaces so the
teardown backstop retries.
TerminateInstances marks the EC2 record terminated and drives node-local
teardown, but when that teardown runs on a node other than the one hosting
QEMU (per-instance subscription lost, request falls back to the ec2.terminate
broadcast), the process survives and keeps holding its OVN logical port / IP,
blocking rebuilds.
Add OrphanQEMUReaper (node-local, ScopeNodeLocal): it inspects only the
terminated KV bucket and reaps a live process found via the instance's local
PID file. Running, pending, and stopped instances are never touched. Whichever
node holds the orphan PID file is the one that reaps it; every other node's
lookup misses and is a no-op.
Add utils.ProcessAlive (signal-0 liveness) and utils.ForceKillProcess
(SIGKILL + bounded wait), guard shutdownAndUnmount's force-kill on liveness,
and register the reaper in the daemon GC loop.
reconnectInstance re-asserted the instance running state but never the boot
volume's in-use state, so after a daemon restart describe-instances showed the
VM running while describe-volumes showed its root volume available — a
split-brain that makes start-instances attempt to double-launch a live QEMU.
Extract the launch path's boot-volume promotion into markBootVolumesInUse and
call it from reconnectInstance so launch and recovery keep volume state
consistent. Add a reconnect unit test asserting the boot volume is marked
in-use and non-boot volumes are left to their own attach flow.
The daemon-side in-use gate trusts persisted volume State/AttachedInstance,
which the attach/detach lifecycle can leave desynced: a running root volume
was observed with State:"" and AttachedInstance:"" while the instance's
BlockDeviceMappings still listed it, so DeleteVolume passed the gate and
removed the durable backing under a live QEMU.
Add an authoritative guard at the gateway DeleteVolume path: fan out the
strict DescribeInstances survey and refuse with VolumeInUse when any
non-terminated instance's BlockDeviceMappings reference the volume. Fail
closed on an incomplete cluster view rather than delete against a partial
survey. Terminate-time cleanup uses the daemon volume service directly and is
unaffected.
Drives the ECS data plane against the local awsgw: a customer VPC plus a real
container instance launched from the spinifex-ecs-node AMI, then standalone
tasks in all three network modes and an awsvpc service fronted by an ALB target
group. Adds an ECS client to the harness, a Makefile target, and a
service-interfaces suite entry.
bridge/host tasks share the host netns where the agent credential endpoint owns
169.254.170.2:80, so they run a no-port command and assert placement + mode;
awsvpc and the ELB service prove real :80 serving. waitTaskRunning re-checks
after a settle so a transient RUNNING cannot pass.
Wire crash detection and host-resource signals through to the AWS-faithful
DescribeInstanceStatus API and the spx CLI, closing out the EC2 health
monitoring / self-healing work.
Phase C — QMP health monitoring:
- qmpHeartbeat now tracks consecutive query-status failures; at
QMPMaxConsecutiveFailures it checks process liveness, triggering HandleCrash
on a dead QEMU and marking an alive-but-wedged process impaired.
- InstanceHealthState gains QMPConsecutiveFailures, LastQMPSuccess, ImpairedSince.
Phase E — DescribeInstanceStatus mapping:
- buildInstanceStatus maps running VMs to ok / impaired (QMP-unresponsive,
with ImpairedSince) / initializing (2-min launch grace), non-running to
not-applicable.
- SystemStatus reflects node memory pressure via ResourceManager
HostUnderMemoryPressure (live MemAvailable below the host reserve).
Phase C2 — observability:
- spx get vms gains HEALTH and CRASHES columns derived from vmHealthLabel.
Tests cover crash/QMP helpers, status mapping, memory-pressure system status,
and the health label.
On startup the agent had no reconcile pass: Run registered, heartbeat, and
immediately polled assignments with an in-memory dedup map. After a restart the
gateway re-delivers the instance's assignments and the agent re-ran them,
orphaning containers still running from before and double-starting tasks.
Add a reconcile() pass that runs once before polling: it lists the runtime's
containers, and for each task still running in this cluster re-registers its IAM
credentials, re-attaches the exit-wait, and refreshes RUNNING on the gateway.
The adopted task set seeds pollAssignments so re-delivered assignments are acked
but not re-run.
- runtime: add a label-agnostic List(ctx) []Container to the Runner seam, with a
containerd implementation and fake support.
- taskLabels: stamp credID / taskRoleArn / eniMac (when set) so a container is
self-describing enough to re-register creds and tear down its netns on adoption.
- pollAssignments takes the seed dedup map.
Best-effort: a nil runner or a List error degrades to today's poll+run behaviour.
* docs: add ECS (Elastic Container Service) guide + register ecs-quickstart workbook
Add a Container Services -> ECS doc mirroring the EKS/ECR shape: concepts
(cluster, task definition, task, service, container instances via the
ECS-AMI EC2 launch type), an aws ecs / console / Terraform quickstart,
task IAM roles via the credential endpoint, awsvpc networking, and the
honest v1 limitations (no service discovery, awslogs discarded, no rolling
deployments, no capacity providers, no distinct execution role).
Convert the ecs-quickstart workbook README to the doc-page format
(frontmatter + overview/prerequisites/instructions/troubleshooting) so it
renders as a Terraform Workbooks entry alongside the EKS workbooks.
* docs: add node-image prerequisite banner + infra diagram to EKS and ECS pages
Add a GitHub-style [!IMPORTANT] banner to the EKS and ECS prerequisites
flagging the required node-image import (eks-node / spinifex-ecs-node),
mirroring the install page's WAN-bridge banner, each with a describe-images
verification command.
Add a basic Spinifex topology diagram (SVG) to each Overview, in the
tf-nginx-alb visual style: the EKS managed control plane + node-group
workers, and the ECS ALB + container instances with the gateway/agent
and task-credential path.
End-to-end ECS stack against the Spinifex gateway: cluster, task definition
with a task role, container instances from the spinifex-ecs-node AMI, and an
awsvpc service fronted by an ALB target group. Unlike the console's provision-
capacity action, the workbook spells out the two things a Spinifex container
instance needs that AWS hides: a LAN-reachable gateway URL and the gateway CA,
both seeded into cloud-init user-data; instance creds come from IMDS via the
ecsInstanceRole profile. Live-verified apply (HTTP 200 through the ALB) and
destroy round-trip.
The agent heartbeats by periodically re-registering through the gateway
(RegisterContainerInstance). After a control-plane restart, the heartbeat
reaper marks instances DRAINING for the missed-heartbeat window, but the
re-register path only refreshed LastSeen and never restored the status, so a
reaped instance stayed DRAINING forever even though its agent was healthy and
re-registering.
Mark reaper-induced DRAINING with a Reaped flag and restore the instance to
ACTIVE on the next re-register, clearing the flag. An operator
UpdateContainerInstancesState drain clears Reaped, so an intentional drain is
never undone by the live agent.
* feat(ecs): provision EC2 container-instance capacity via IMDS instance-role
Adds a ProvisionCapacity gateway action that launches spinifex-ecs-node
EC2 instances which register as ECS container instances, fixing
RESOURCE:placement when a cluster has no capacity.
Agent control-plane now SigV4-signs gateway calls with instance-role
credentials read from IMDS (no static keys seeded in user-data). The
gateway already verifies STS/temp creds, so the instance role carries an
ecs:* policy + ec2 trust; ensureECSInstanceProfile find-or-creates the
ecsInstanceRole instance profile attached at launch.
Backend resolves the spinifex-ecs-node AMI by spinifex:managed-by=ecs
tag, renders keyless user-data, and launches via the customer
RunInstances path. Frontend adds a hand-signed JSON-1.1 ProvisionCapacity
call, a provision-capacity dialog on the container-instances tab, and the
ECS console section nav + system-image gating.
* feat: auto-assign EIPs to awsvpc service tasks; add ECS task/service detail pages
Wire AssignPublicIp=ENABLED through the service reconciler so awsvpc service
tasks automatically allocate and associate an Elastic IP on the RUNNING
transition, and release it on STOP. assignTaskPublicIP/releaseTaskPublicIP
mirror the target-group registration hooks in recordTaskState and forceStopTask;
the public IP and allocation ID are persisted on the task record and surfaced in
DescribeTasks attachments as publicIPv4Address.
Add AWS-parity ECS task and service detail pages showing private and public
endpoints, container state, and networking configuration, reachable by clicking
a task or service in the cluster tabs.
* test: cover natsEIPManager allocate/associate/release over NATS
ListTaskDefinitions appended every revision regardless of the requested
status, so deregistered (INACTIVE) task definitions never dropped off the
default listing and the console could not clear them. Filter by status,
defaulting to ACTIVE to match AWS.
Add an ECS console mirroring the EKS/ECR route shape: a single ECS nav
entry, cluster list and detail with Services/Tasks/Task Definitions/
Container Instances tabs, plus create and teardown paths.
Adds three full-page wizards to drive container deploys end-to-end:
Register Task Definition, Run Task, and Create Service. RunTask surfaces
a 200-with-failures placement result as an error instead of silently
navigating away.
Includes the @aws-sdk/client-ecs client singleton, ecs queries and
mutations, zod form schemas, and co-located tests.
STOPPED task records were never pruned: ListTasks enumerated every task a
cluster ever ran and the per-account KV grew unbounded. Add a leader-gated sweep
tick to the scheduler cron (alongside the reaper/reconcile) that deletes task
records STOPPED longer than a 1h retention window — keeping a just-stopped task
describable for DescribeTasks / UI exit reason, matching AWS's STOPPED window.
A task missing its StoppedAt timestamp is never pruned (defensive). Single-writer
(leader only), reuses the reap() bucket walk.
Wire the four NotImplemented ECS teardown actions so an IaC stack and the
console can reverse a deployment:
- DeleteCluster cascades: force-stops every task (releasing ENIs, deregistering
TG targets, returning capacity), then sweeps the cluster's whole KV subtree.
Unlike AWS it cascades rather than rejecting a non-empty cluster, so terraform
destroy round-trips regardless of teardown ordering.
- DeregisterTaskDefinition marks a specific family:revision INACTIVE (kept
describable); a bare family is rejected, matching AWS.
- DeregisterContainerInstance removes an instance, force-stopping its tasks when
Force is set and rejecting a non-empty instance otherwise.
- UpdateContainerInstancesState sets ACTIVE/DRAINING; draining force-stops the
instance's service-owned tasks so the reconciler relaunches them elsewhere.
Reuses the existing forceStopTask primitive and the reaper drain pattern; adds a
deleteKeysWithPrefix KV helper and INACTIVE status constants.
Serve task-role credentials to containers the AWS-faithful way: an HTTP
endpoint at 169.254.170.2:80 inside the ecs-agent (host VM netns) serving
short-lived credentials at /v2/credentials/{credID}. Containers find it via
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI, injected per container.
The agent mints credentials by SigV4-signing sts:AssumeRole(taskRoleARN)
against the gateway with its instance-role credentials, mirroring
internal/ecrauth. This reconciles ecs-v1 Q4/Q10 (STS over NATS + scheduler
JWT assertion) with the 4f rewire that moved the agent entirely onto the
gateway: the agent has no NATS connection and already holds instance creds,
so it assumes the task role directly. Credentials are cached per credID and
refreshed near expiry.
Each awsvpc task netns gets a credentials veth into the host netns and a
host-routed /32 to the endpoint, keeping it off the task ENI's VPC route.
The scheduler now persists taskRoleArn on the task definition and carries it
in the assign payload so the endpoint activates end to end.
* fix: serialize EBS hot-plug per instance to prevent PCIe port race
Concurrent AttachVolume calls to one instance both allocated the lowest free
hot-plug PCIe root port via nextHotplugEBSPort (which reads live QEMU state),
then issued QMP device_add on the same hotplug-ebs bus. The second device_add
failed with 'PCI: slot 0 function 0 already occupied by virtio-blk-pci',
surfacing as nats:timeout -> ServerInternal and a CSI retry-storm
('retry quota exceeded'); volumes never attached.
Add a per-instance attachMu and hold it across the whole AttachVolume and
DetachVolume critical sections, mirroring the ENI hot-plug serialization.
The mutex is non-persisted and shared via the Manager's stable *VM.
* fix: allocate EBS hot-plug PCIe port from in-memory accounting
AttachVolume derived the free PCIe hot-plug root port from a live
query-block scan (nextHotplugEBSPort), extracting the bus with a
hotplug-ebs(\d+) regex on each device's QDev. Real QEMU reports a
hot-plugged blockdev by id (/machine/peripheral/vdisk-vol-xxx/
virtio-backend), with no hotplug-ebs component, so the regex never
matched, the used set stayed empty, and the allocator always returned
port 1. Only the first EBS volume per instance attached; every later
attach collided on "PCI: slot 0 function 0 already occupied by
virtio-blk-pci", rolling back the blockdev, killing NBDKit, and
surfacing as nats: timeout / ServerInternal to the CSI driver, which
then exhausted its retry quota.
Allocate the port from authoritative in-memory accounting instead:
record the chosen port as EBSRequest.HotplugPort (persisted, so it
survives a daemon restart) and pick the lowest unclaimed port with
freeHotplugEBSPort over the instance's recorded requests. Hold
attachMu across allocate->device_add so concurrent CSI attaches cannot
pick the same port. DetachVolume already removes the request, freeing
the port. Drops the query-block round-trip from the attach path.
* feat(ecsgw): on-VM SigV4 client for the ECS agent→gateway channel
Adds internal/ecsgw, the twin of internal/eksgw: a TLS + SigV4 (service "ecs")
client the ecs-agent will use to reach the AWS gateway over HTTPS instead of
connecting to host NATS directly. Calls are AWS JSON 1.1 with an X-Amz-Target
action; the gateway relays onto ecs.* host-side, keeping NATS cluster-internal.
* feat(ecs): SubmitTaskStateChange gateway action → recordTaskState
Wires the AWS-API SubmitTaskStateChange action end to end (gateway action →
ecs.SubmitTaskStateChange NATS → daemon → Service.SubmitTaskStateChange) so a
gateway-routed agent reports task state without touching the NATS bus. The
handler maps the SDK input onto the same bus.TaskState shape and converges on
recordTaskState; the account is authoritative from the SigV4 caller, so an
instance cannot report state for another account's task.
* feat(ecs): durable assign KV inbox + PollAssignments gateway action
Scheduler writes task assigns into a per-instance KV inbox
(clusters/{cluster}/assignments/{instance}/{task}) instead of a
fire-and-forget core-NATS publish, and a new PollAssignments gateway
action lets a gateway-routed agent drain its inbox over TLS+SigV4.
At-least-once: an unacked assign is re-delivered on the next poll;
the agent acks accepted assigns (AckTaskIDs) and STOPPED transitions
reclaim stale inbox entries. The Poll response carries bus.Assign with
an RFC3339 time, so it is encoded with encoding/json (RawJSONActions)
rather than the jsonutil epoch-second marshaler.
* feat(ecs-agent): rewire control-plane off NATS onto the AWS gateway
The agent no longer connects to NATS. It registers, heartbeats, reports
task state, and receives assignments entirely through the AWS gateway
over TLS+SigV4, using seeded IAM credentials — keeping the NATS bus
host-internal and closing the trust-boundary gap (a tenant container
instance must not hold a shared bus token).
- controlPlane seam over ecsgw.Client replaces the NATS publisher:
Register (RegisterContainerInstance), SubmitTaskState
(SubmitTaskStateChange), PollAssignments.
- Heartbeat folds into a periodic idempotent re-register (refreshes
LastSeen); bus.Heartbeat is dropped from the agent.
- subscribeAssign (NATS subscribe) becomes a poll loop: drain the
inbox, dispatch each assign once, ack accepted taskIDs on the next
poll (at-least-once).
- config: ECS_ACCESS_KEY / ECS_SECRET_KEY / ECS_POLL_INTERVAL added,
ECS_NATS_URL removed; cloud-init initd seeds the IAM creds.
Tests move from the NATS publisher fake to a controlPlane fake.
* feat(ecs): cloud-init cred-seed + ecs-node bring-up script
ECS container instances are plain EC2 booted from the spinifex-ecs-node
AMI (AWS-faithful: no ECS-specific launch API). This renders the
cloud-init user-data that seeds /etc/spinifex-ecs/agent.env (gateway
URL/CA/region/cluster + IAM creds) and the gateway CA, then runs the
instance. The agent registers, heartbeats, polls assignments, and
reports task state entirely through the AWS gateway over TLS+SigV4 —
never NATS.
The guest dials the customer-facing gateway address (network advertise
IP), not the mgmt bridge, mirroring the EKS GatewayURL/AddonGatewayURL
split. Seeds the system key for now; scoped per-instance creds are a
follow-up (4g task-role endpoint).
* fix(ecs): reconcile dev merge — restore #418 ENI release + netns wiring
The 'Merge branch dev' resolution dropped code where Sprint 4d (#418,
awsvpc ENI-per-task) and the gateway control-plane rewrite (#419) touch
the same paths:
- recordTaskState STOPPED branch: reclaimTaskENI was clobbered by
reclaimAssignInbox; both are needed (release the task ENI AND drain the
KV inbox). Restores TestRecordTaskState_Awsvpc_ReleasesENIOnStopOnce.
- ecs-agent Agent: the netns field, its init, and the setupTaskNetns/
teardownTaskNetns wrappers were lost when the controlPlane Agent struct
won the merge, leaving taskrun.go calling undefined methods.
- eni_test assign assertion read the assign over NATS; #419 delivers it via
the durable KV inbox, so it now polls via PollAssignments.
- taskrun_test netns cases used the removed historyPub fake; switched to
fakeCP to match the controlPlane seam.
* feat(ecs): service controller + ELBv2 target-group registration (Sprint 4f)
Land the ECS Service controller on top of the 4e scheduler/RunTask flow:
- CreateService/UpdateService/DeleteService/DescribeServices/ListServices.
REPLICA only; DAEMON and serviceRegistries rejected in v1 (Q15).
- StartTask (explicit placement) and StopTask.
- Leader-driven replica reconciliation: a 10s scheduler tick converges each
service's running task count on desiredCount, launching the shortfall via
RunTask (tasks tagged group=service:{name}) and force-stopping the surplus.
- ELBv2 target-group registration: the scheduler registers a task's awsvpc ENI
IP on the RUNNING transition and deregisters on STOPPED via the existing
elbv2.RegisterTargets/DeregisterTargets NATS handlers (Q8, single-writer).
- forceStopTask consolidates the control-plane stop (STOPPED state, capacity +
ENI release, inbox drain, TG deregister); reused by StopTask, scale-in, and
the heartbeat reaper (the reaper now deregisters targets per Q13).
- Wire the 7 new actions through the gateway action map, NATS adapter, and
daemon subscriptions; add ServiceNotFoundException.
* fix(ecs-agent): anonymous pull for non-ECR images + normalize image refs
The agent minted an ECR authorization token via IMDS for every image, so
public docker.io refs failed with 'IMDS returned empty role name' on hosts
without an instance role. Gate ECR auth on .dkr.ecr. refs and pull all other
registries anonymously.
Bare image strings (nginx:alpine) were handed to containerd unmodified, which
parsed them as dummy://nginx:alpine and rejected the pull. Normalize refs to
their fully-qualified form (docker.io/library/nginx:alpine, :latest default)
before Pull and GetImage.
Adds StartTask/StopTask service-controller coverage.
* fix: address ECS 4f live-verification findings
ecs-node-bringup.sh defaulted GATEWAY_IP to the mgmt bridge (10.15.8.1), which
a VPC guest cannot route to — the agent could not reach the gateway. Default to
the host's WAN-bridge IP, which guests reach by NATing out through the WAN.
A target group not forwarded to by any listener reported its targets stuck in
'initial' forever (no health checker advances a detached TG). DescribeTargetHealth
now reports such targets as 'unused' with reason Target.NotInUse, matching AWS.
* feat(ecsgw): on-VM SigV4 client for the ECS agent→gateway channel
Adds internal/ecsgw, the twin of internal/eksgw: a TLS + SigV4 (service "ecs")
client the ecs-agent will use to reach the AWS gateway over HTTPS instead of
connecting to host NATS directly. Calls are AWS JSON 1.1 with an X-Amz-Target
action; the gateway relays onto ecs.* host-side, keeping NATS cluster-internal.
* feat(ecs): SubmitTaskStateChange gateway action → recordTaskState
Wires the AWS-API SubmitTaskStateChange action end to end (gateway action →
ecs.SubmitTaskStateChange NATS → daemon → Service.SubmitTaskStateChange) so a
gateway-routed agent reports task state without touching the NATS bus. The
handler maps the SDK input onto the same bus.TaskState shape and converges on
recordTaskState; the account is authoritative from the SigV4 caller, so an
instance cannot report state for another account's task.
* feat(ecs): durable assign KV inbox + PollAssignments gateway action
Scheduler writes task assigns into a per-instance KV inbox
(clusters/{cluster}/assignments/{instance}/{task}) instead of a
fire-and-forget core-NATS publish, and a new PollAssignments gateway
action lets a gateway-routed agent drain its inbox over TLS+SigV4.
At-least-once: an unacked assign is re-delivered on the next poll;
the agent acks accepted assigns (AckTaskIDs) and STOPPED transitions
reclaim stale inbox entries. The Poll response carries bus.Assign with
an RFC3339 time, so it is encoded with encoding/json (RawJSONActions)
rather than the jsonutil epoch-second marshaler.
* feat(ecs-agent): rewire control-plane off NATS onto the AWS gateway
The agent no longer connects to NATS. It registers, heartbeats, reports
task state, and receives assignments entirely through the AWS gateway
over TLS+SigV4, using seeded IAM credentials — keeping the NATS bus
host-internal and closing the trust-boundary gap (a tenant container
instance must not hold a shared bus token).
- controlPlane seam over ecsgw.Client replaces the NATS publisher:
Register (RegisterContainerInstance), SubmitTaskState
(SubmitTaskStateChange), PollAssignments.
- Heartbeat folds into a periodic idempotent re-register (refreshes
LastSeen); bus.Heartbeat is dropped from the agent.
- subscribeAssign (NATS subscribe) becomes a poll loop: drain the
inbox, dispatch each assign once, ack accepted taskIDs on the next
poll (at-least-once).
- config: ECS_ACCESS_KEY / ECS_SECRET_KEY / ECS_POLL_INTERVAL added,
ECS_NATS_URL removed; cloud-init initd seeds the IAM creds.
Tests move from the NATS publisher fake to a controlPlane fake.
* feat(ecs): cloud-init cred-seed + ecs-node bring-up script
ECS container instances are plain EC2 booted from the spinifex-ecs-node
AMI (AWS-faithful: no ECS-specific launch API). This renders the
cloud-init user-data that seeds /etc/spinifex-ecs/agent.env (gateway
URL/CA/region/cluster + IAM creds) and the gateway CA, then runs the
instance. The agent registers, heartbeats, polls assignments, and
reports task state entirely through the AWS gateway over TLS+SigV4 —
never NATS.
The guest dials the customer-facing gateway address (network advertise
IP), not the mgmt bridge, mirroring the EKS GatewayURL/AddonGatewayURL
split. Seeds the system key for now; scoped per-instance creds are a
follow-up (4g task-role endpoint).
* fix(ecs): reconcile dev merge — restore #418 ENI release + netns wiring
The 'Merge branch dev' resolution dropped code where Sprint 4d (#418,
awsvpc ENI-per-task) and the gateway control-plane rewrite (#419) touch
the same paths:
- recordTaskState STOPPED branch: reclaimTaskENI was clobbered by
reclaimAssignInbox; both are needed (release the task ENI AND drain the
KV inbox). Restores TestRecordTaskState_Awsvpc_ReleasesENIOnStopOnce.
- ecs-agent Agent: the netns field, its init, and the setupTaskNetns/
teardownTaskNetns wrappers were lost when the controlPlane Agent struct
won the merge, leaving taskrun.go calling undefined methods.
- eni_test assign assertion read the assign over NATS; #419 delivers it via
the durable KV inbox, so it now polls via PollAssignments.
- taskrun_test netns cases used the removed historyPub fake; switched to
fakeCP to match the controlPlane seam.
* feat(ecs): awsvpc ENI-per-task control plane (Sprint 4d)
Scheduler owns the task-ENI lifecycle for awsvpc mode: allocate from the
task subnet on placement, hot-plug onto the container instance, plumb ENI
identity through the assign payload + task record, and detach+delete on
STOPPED (graceful + reaper, idempotent leak guard). bridge/host unchanged.
* feat(ecs-agent): awsvpc guest-side task netns from hot-plugged ENI
The agent builds a per-task network namespace for awsvpc tasks: it finds the
hot-plugged ENI NIC by MAC, moves it into /var/run/netns/ecs-<taskID>, and DHCPs
the OVN-assigned ENI IP. The container then joins that netns via the OCI network
namespace path; bridge/host tasks keep the host (VM) netns as before.
* build(ecs-node): add iproute2 for awsvpc task netns
The ecs-agent builds awsvpc task network namespaces with `ip netns`, which
busybox's `ip` applet does not implement. Install iproute2 in the ecs-node AMI;
udhcpc (for the in-netns ENI DHCP lease) is already provided by busybox.
PR #411 (cloud-init EC2 datasource) dropped the cloud-init ISO that
created the ec2-user account and baked the harness SSH key; guests now
boot the stock Ubuntu gold AMI (default user ubuntu) bootstrapped via the
Ec2/IMDS datasource. eni_reconcile_test.go was the lone single-suite test
still SSHing as ec2-user, so TestENIHotplugReconcile failed publickey at
the first SSH. Align it with every other single-suite test.
* feat(eni): hot-plug reconciler + orphan sweep (Sprint 3d)
Add eniReconciler, a node-local vm.Reaper that converges ENI hot-plug
state across a daemon restart or crash: KV ENIRecord <-> in-memory PCIe
slot map <-> guest query-pci device list. Resolves records stuck in
attaching/detaching, re-adopts the slot map after restart, and reaps
orphan QMP devices and slot entries.
A staleness gate (new ENIRecord.AttachmentStateAt + 30s threshold) keeps
the reconciler off records a live attach/detach handler is mid-pipeline
on. Wired into the existing GarbageCollector backstop.
* test(eni): unit-cover hot-plug reconciler vm helpers
* test(eni): e2e reconciler convergence across daemon restart (Sprint 3e)
StopNode now stops spinifex-nats.service, so WaitNATSPeers polling that
node's dead 127.0.0.1:8222 monitor port fails with curl exit 7 and never
converges. Add WithSkipNodes to exclude the stopped node from the poll;
TestMultinodeNodeFailure skips node2.
Add the ECS v1 scheduler control loop and end-to-end RunTask flow:
- handlers/ecs: per-account KV-backed Service (clusters, task definitions,
container instances, tasks), bin-pack/spread/random placement with CAS
capacity reservation, and a leader-elected Scheduler that owns the Layer-2
bus subscriptions (register/heartbeat/task-state) plus a heartbeat reaper.
- gateway/ecs: wire CreateCluster, Describe/ListClusters,
Register/Describe/ListTaskDefinitions, Register/Describe/ListContainerInstances,
RunTask, Describe/ListTasks onto the daemon over NATS.
- daemon: construct the Service, start the Scheduler goroutine, subscribe the
ECS action subjects.
- ecs-agent: subscribe the per-instance assign subject, pull and run each
container via containerd, and report task state back on the bus.
Adds the spinifex-ecs-node AMI variant: alpine 3.21.7 + containerd + runc +
cni-plugins + ca-certificates with the ecs-agent binary baked to
/usr/local/bin and three OpenRC services wired in dependency order
(ecs-ca-install -> containerd -> ecs-agent).
- manifest.conf drives build-system-image.sh (no host mounts).
- ecs-ca-install.initd: first-boot oneshot installing the cloud-init
delivered gateway CA into the system trust; no-op when absent.
- ecs-agent.initd: sources /etc/spinifex-ecs/agent.env, backgrounds the
agent after net + containerd.
- 10-bridge.conflist: CNI bridge for task networking (Sprint 4d).
- setup.sh: exec bits, /opt/cni/bin symlink, content-store dirs, ttyS0
console reorder + fast boot menu.
- Makefile: build-ecs-node-image / import-ecs-node-image targets.
* feat(ecs): ecs-agent skeleton + containerd image pull (Sprint 4c)
Add the ecs-agent binary that runs inside an ECS container instance: it
resolves host identity from IMDS, registers on the Layer-2 NATS bus, and
heartbeats while alive. Image pulls go through a containerd-backed
ImagePuller behind an interface, with an ECR resolver that mints
GetAuthorizationToken creds from the instance role.
Extract the IMDS-credential and ECR-auth flow out of
ecr-credential-provider into internal/imdscreds and internal/ecrauth so
the agent and the kubelet provider share one implementation.
Add spinifex/handlers/ecs/bus as the single source of truth for Layer-2
subject builders and wire payloads, shared by the agent now and the
scheduler in Sprint 4e.
* build: pin genproto via replace to fix containerd/v2 ambiguous import
go.opencensus.io -> grpc@v1.33.2 drags the pre-split monolithic genproto
into the module graph; it still ships googleapis/rpc/status, which collides
with the split genproto/googleapis/rpc module that containerd/ttrpc needs.
The main module never imports the monolith, so a plain require bump is
dropped by 'go mod tidy'. Pin it with a replace to a post-split version so
the fix survives tidy and contributors can run 'go mod tidy' freely.
* fix(ecs): agent boot failsafes from runtime verify
- Build the ECR gateway HTTP client lazily on first image pull instead of
at boot. A missing/malformed gateway CA no longer kills the agent before
it can register + heartbeat, matching the best-effort containerd path.
- Actively probe containerd with IsServing in runtime.New. The client dials
lazily, so a dead socket previously slipped past the boot-time
'containerd unavailable' warn and only surfaced on the first pull.
- Make the heartbeat interval configurable via ECS_HEARTBEAT_INTERVAL
(default 30s) for test/verify and slow-link tuning.
* fix: drop removed predastore s3.Config.Compaction from admin_test
The predastore submodule bump removed the Compaction field from s3.Config.
Parse the rendered [compaction] block into a local struct so the template
round-trip test no longer references the deleted field.
* feat: graceful guest drain on systemctl stop / host shutdown
systemctl stop spinifex.target tore down nbdkit/viperblock while guest QEMU
was still writing, corrupting guest filesystems. The ordered drain existed
only over NATS via cluster shutdown, never on the systemd/host path.
Add spx admin node drain --local (GATE+DRAIN against the local node, leaving
STORAGE/PERSIST/INFRA to systemd), wired as the ExecStop of a new
spinifex-shutdown.service oneshot ordered After= all services so it stops
first and drains guests while every service is still up. DDIL preserved: the
oneshot fires only on target/host shutdown, not on a daemon restart.
Harden the viperblockd SIGTERM handler to flush the WAL but skip reaping an
nbdkit a guest is still attached to. Add a units ordering invariant.
* fix: ExecStop drain timeout needs duration unit (120 -> 120s)
The spinifex-shutdown.service ExecStop passed --timeout=120, but spx admin
node drain's --timeout is a cobra Duration flag and rejects a bare integer
(time: missing unit in duration "120"). The drain exited 1 without running,
so a systemctl stop tore nbdkit out from under the running guest and corrupted
its ext4 (the exact failure this feature prevents). Pass 120s.
* fix: nbdkitInUse read ss state from correct column + cover it
ss -H rows are '<netid> <state> ...', so the state is field[1], not field[0]
(field[0] is 'u_str'). The old index never matched ESTAB, so an in-use nbdkit
read as idle and the SIGTERM path would reap it under a live guest. Add tests
exercising a real connected unix socket, an absent socket, and TCP transport.
* feat: add predastore compaction E2E
Inject a short compaction interval into test-cluster predastore config via a
new --predastore-compaction-interval flag on admin init/join, gated so
production output stays byte-identical when unset. Add a harness helper that
drives write-then-delete churn against predastore S3, brackets it with
du-over-SSH usage, and asserts end-of-churn usage settles below 70% of the
churn peak; wire it into the multinode suite as TestPredastoreCompaction.
* fix: compaction E2E gate writes churn to its own bucket
The reclaim gate signed PutObject against the config-defined system
'predastore' bucket, which belongs to account 000000000000 with only a
static [[auth]] grant. The harness admin identity is a different IAM
account, so predastore returned 403 AccessDenied cross-account.
Create a per-run bucket the admin identity owns (AdministratorAccess)
and churn into it. Objects still erasure-code into distributed/nodes/
node-* so the du-based reclaim measurement is unchanged. The gateway
(:9999) does not proxy S3, so the client stays predastore-direct.
* test: run predastore compaction gate before destructive node tests
The compaction reclaim gate was declared last in the multinode suite, so it
measured shard-store usage after NodeFailure/NodeRecovery had torn down and
restarted node2. NodeRecovery only gates on NATS/gateway/EC2 readiness, not
predastore, so node2's shard store was still settling and the du baseline came
back empty. Move the test before the destructive node tests so it runs against
a pristine, fully-converged cluster primed by the volume tests.
* fix: race-free SSH output capture + empty-tolerant du probe
The compaction reclaim gate crashed with ParseInt("") when node1's du
probe returned empty combined output. Harden the probe and the SSH layer:
- ssh.Run used one bytes.Buffer for both session.Stdout and session.Stderr,
which x/crypto/ssh copies in two concurrent goroutines — a data race on a
non-concurrent-safe type. Switch to session.CombinedOutput, which guards the
combined stream with a mutex-wrapped singleWriter.
- shardStoreUsageBytes now terminates the remote command with a sentinel echo.
A missing sentinel is reported as output lost in SSH retrieval; a present
sentinel with no number is treated as a 0 measurement (anomalous, logged)
instead of crashing the gate.
The sentinel lets the next run distinguish a transport-layer drop from a
genuinely empty remote measurement.
* test(e2e): replace predastore du-reclaim gate with object lifecycle contract
The multinode compaction gate measured shard-store disk usage over SSH (du)
and asserted reclaim after churn. It could never pass: segments only seal at
the 4 GiB roll threshold, so test-scale churn produces no compactable segment,
and the du probe read 0 bytes before and after. Reclaim correctness is already
covered deterministically by predastore's store-level unit tests.
Delete AssertPredastoreCompaction and its du/churn/poll/sentinel machinery,
the harness unit tests that exercised them, and the multinode TestPredastore
Compaction entry. Add a single-node TestPredastoreObjectLifecycle that checks
the user-visible contract: an object is readable after PutObject, gone after
DeleteObject (GET returns NoSuchKey and ListObjectsV2 is empty), and the store
keeps serving fresh writes. newPredastoreS3 is retained, refactored to take a
host so it works outside the multinode cluster type.
* test(e2e): drop orphaned multinode TestPredastoreCompaction
Complete the du-reclaim -> object-lifecycle refactor: remove the
multinode TestPredastoreCompaction caller (and its now-unused context
and harness imports) that still referenced the deleted
harness.AssertPredastoreCompaction, which broke e2e-build. Wire the
single-node TestPredastoreObjectLifecycle entry that replaces it.
* docs: add Containers section with EKS and ECR service pages
New docs/containers/{eks,ecr} pages covering each service via AWS CLI, the Spinifex console, and Terraform, plus the requisite VPC/IGW/NAT/SG/IAM setup. EKS page embeds console screenshots. Cross-links the EKS Terraform workbooks.
* docs(eks): rework Terraform workbooks into a capability ladder
Rework the three EKS workbooks into a coherent ladder and publish them to
the Containers section. All tiers are single- or three-node capable via
node_desired_size and serve a shared Spinifex-themed demo app from ECR,
replacing nginxdemos/hello.
- demo-app: new std-lib Go web app (themed page reporting pod/node/cluster/
region + hit counter; optional /data persistence), distroless image.
- eks-quickstart: node_desired_size var, ECR repo + themed app over NodePort.
- eks-https-ingress (was eks-addons-access): HTTPS via the aws-load-balancer-
controller addon + ACM, public/private VPC with NAT, managed-ingress=false.
- eks-gitops-argocd (was eks-https-ingress): Argo CD addon syncs the app from
a git repo, aws-ebs-csi-driver PVC (Viperblock-backed) for state, access
entry RBAC, HTTPS Ingress carried over.
Pending live e2e verification (image build/push, apply, ALB + PVC bind).
* fix(ecr): implement ListTagsForResource (empty tags) for client parity
The AWS provider calls ecr:ListTagsForResource on every repository read, which
returned 501 NotImplementedException and broke 'tofu apply' of any workbook that
creates an aws_ecr_repository. Implement it to return an empty tag set (resource
tagging stays a deferred feature) so reads succeed. Drop the tags block from the
three EKS workbooks' ECR repos so the provider doesn't call the still-stubbed
TagResource on create.
* feat(eks): pull from internal ECR via kubelet credential provider
Worker nodes could not pull custom images from the internal Spinifex ECR
registry: the eks-node had no DNS for the registry host, no containerd CA
trust, no port rewrite to :9999, and no pull credentials. Only public
images worked.
Add cmd/ecr-credential-provider, a kubelet exec credential provider that
fetches the node IAM-role credentials from IMDS, calls GetAuthorizationToken
against the gateway ECR endpoint, and returns the minted AWS:<jwt> as Basic
auth for the registry host. Thread the gateway URL, account, region, and
internal suffix into the agent userdata, which now writes the gateway CA,
a registries.yaml (CA trust + :9999 endpoint), the kubelet
credential-provider config and config.yaml.d drop-in, and an /etc/hosts
entry mapping the registry host to the gateway IP. Ship the provider binary
in the eks-node AMI.
Also remove a stray paragraph from the admin-cli README.
* fix: dial EKS ECR registry mirror by gateway IP
cloud-init manage_etc_hosts rewrites /etc/hosts from the alpine template on
every boot, wiping the registry-host mapping appended via bootcmd, so worker
nodes could not resolve the internal ECR host and pulls failed with no such
host. Point the k3s registries.yaml mirror endpoint and tls config at the
gateway IP instead (the gateway cert carries an IP SAN), keeping the pod image
ref port-less and dropping the DNS leg until northstar lands. Fall back to the
hostname endpoint when the gateway is not an IP.
* fix: attach node-role instance profile to EKS workers for ECR
EKS workers launched with no IAM instance profile, so IMDS served no role and
the ECR credential provider could not mint a registry token, leaving internal
ECR pulls failing with 401. Real EKS implicitly creates an instance profile for
the node role; do the same here. Wire a KV-backed IAM service into the EKS deps
and, at worker launch, find-or-create an instance profile named for the node
role, attach the role, and pass it to RunInstances so IMDS exposes the role to
the credential provider.
* feat: use mulga logo in EKS demo app
* feat: add ACM certificate tag support (List/Add/RemoveTags)
Implements ListTagsForCertificate, AddTagsToCertificate and RemoveTagsFromCertificate across the store, service, NATS, daemon and gateway layers. Import now persists tags. Fixes terraform aws_acm_certificate InvalidActionException on ListTagsForCertificate.
* test: cover ACM tag gateway and NATS wrapper paths
* feat: support EKS nodegroup tags (Tag/Untag/ListTags)
Extends EKS tag ops to back nodegroup ARNs alongside cluster ARNs. Nodegroup create-time tags persist on the record and echo via DescribeNodegroup, giving the terraform-aws provider a clean default_tags round-trip. Fixes aws_eks_node_group TagResource 501 NotImplementedException.
* fix: provision EKS ALB ingress via node credentials
The aws-load-balancer-controller addon could not provision an ALB. Three fixes:
- TLS: the controller's per-service ec2/elbv2/acm SDK clients install their own
BaseEndpoint resolver and build an HTTP client that bypasses AWS_CA_BUNDLE,
falling back to the Go system trust pool and rejecting the gateway cert as
'unknown authority'. Add SSL_CERT_FILE so the gateway CA joins the system pool.
- Credentials: an empty AWS_ROLE_ARN beside the web-identity token file
half-arms the SDK web-identity provider, which then signs anonymously
(MissingAuthenticationToken). Inject the IRSA env only when a service-account
role ARN is staged; otherwise omit it so the default chain falls back to the
node instance-profile credentials served over IMDS.
- Authorization: the node role lacked the ec2/elbv2 actions the controller
needs. The eks-https-ingress workbook now attaches an AWSLoadBalancerController
managed policy to the node role (CreatePolicy + AttachRolePolicy).
* fix: add force_delete to EKS workbook demo ECR repos
tofu destroy of an EKS workbook left the demo ECR repository behind: it
still holds the pushed demo image, so DeleteRepository (force=false)
returns RepositoryNotEmptyException. Add force_delete = true to each
workbook's aws_ecr_repository.demo so teardown removes the repo with its
images.
* fix: force-delete EKS control-plane ENIs on teardown
TerminateK3sServerVM deleted the CP ENI with a plain force=false
DeleteNetworkInterface. eniIsLiveAttachment reports InUse whenever the
record's Status/InstanceId/AttachmentId are set, so when the K3s VM is
gone but the ENI record's attachment fields were never cleared the delete
returned InvalidNetworkInterface.InUse permanently and EKSDeletingReaper
re-drove the same non-idempotent call every 90s, wedging the cluster in
DELETING.
Teardown owns its CP ENIs and has already terminated the VM, so the
attachment is authoritatively dead. Replace the InUse-retry budget loop
with detach-then-delete (detachAndDeleteServerENI): DetachENI clears the
stale attachment fields, then DeleteNetworkInterface passes the in-use
guard. Both calls tolerate an already-gone ENI (NotFound) so a race with
the async instance-terminate cascade resolves to idempotent success.
purgeClusterInfra's private-endpoint ENI path reuses the same helper.
* fix: detach+delete worker ENIs on EKS nodegroup teardown
A terminated nodegroup worker could leave its primary ENI in
spinifex-vpc-enis as in-use/attached, and a dangling ENI throws
DependencyViolation on subnet/VPC delete, blocking tofu destroy. Two
gaps:
TerminateWorkerInstances did a local-only vmMgr.Get and silently skipped
any worker owned by another node, stranding both the VM and its ENI on
multi-node clusters. Route the terminate per-instance via ec2.cmd.<id>
(mirroring the gateway TerminateInstances path) so the owning daemon runs
the full teardown — detach + force-delete the primary ENI and clear its
record. No responder / NotFound payload is treated as already-gone so a
retried DeleteNodegroup stays idempotent.
deleteInstanceENI (stopped-instance terminate) issued a force=false
DeleteNetworkInterface with no prior detach: a stale in-use attachment
made it fail InvalidNetworkInterface.InUse, leaving the ENI record behind
after the instance was already written to the terminated bucket. Detach
first, then delete, tolerating NotFound.
* fix: heal volume State drift on detach/terminate teardown
A detach/terminate could leave a volume's State empty instead of
'available'. DeleteVolume gated on State=='available' exactly, so it
returned VolumeInUse for an unattached volume nothing was using, leaving
it undeletable and blocking stack teardown. DescribeVolumes cosmetically
rendered an empty State as 'available', masking the drift.
- DeleteVolume now treats an empty State with no attachment as deletable
(an attached volume still returns VolumeInUse).
- DescribeVolumes derives an empty State from the attachment (attached ->
in-use, unattached -> available) instead of blindly rendering
available, so an empty-but-attached volume is no longer hidden.
- UpdateVolumeState normalizes an unattached volume with an empty state to
available, so a detach writeback that omits the state cannot strand the
volume in drift.
* fix: break all SG cross-references on EKS teardown
DeleteClusterSGs broke the cp<->nodegroup cycle too narrowly. It revoked
only ingress (IpPermissions), never egress (IpPermissionsEgress), so a
cross-reference via an egress rule kept the SG pinned on
DependencyViolation. It also resolved only the two named cluster SGs, so
any OTHER SG in the VPC referencing them — the default SG, the LBC/ALB SG,
the private-endpoint SG, user/TF SGs — was never revoked and pinned the
referenced SG on delete, leaving the VPC stuck.
Enumerate every non-default SG in the VPC and revoke BOTH ingress and
egress on each before deleting, breaking any cycle order-independently,
then delete only the cluster-owned SGs (referrer SGs are deleted by their
own teardown). The default SG is left untouched. Since DeleteClusterSGs
runs against both the customer VPC and the managed control-plane VPC, the
broadened sweep covers both teardown paths.
* feat: support TargetGroup ProtocolVersion in elbv2
CreateTargetGroup now stores ProtocolVersion (defaulting HTTP/HTTPS target
groups to HTTP1) and DescribeTargetGroups/Create output return it. Without
this the aws-load-balancer-controller sees a perpetual immutable-field
mismatch (desired HTTP1 vs empty) and recreates the target group every
reconcile, looping on DuplicateTargetGroupName once a shared ALB IngressGroup
reconciles more than one target group. NLB protocols (TCP/UDP/TLS) keep it
omitted to match AWS.
* fix: omit IRSA env from ebs-csi addon when no role staged
The ebs-csi controller manifest hardcoded AWS_ROLE_ARN={{SERVICE_ACCOUNT_ROLE_ARN}}
plus the web-identity token volume. With no service-account role staged the
placeholder rendered empty, half-arming the SDK web-identity provider, which
panics 'role ARN is not set' and CrashLoops the controller, leaving PVCs
Pending. Convert to the conditional {{IRSA_ENV}}/{{IRSA_VOLUME_MOUNT}}/{{IRSA_VOLUME}}
markers like the LBC bundle so the IRSA env and token volume are omitted when
no role is staged, falling back to node instance-profile credentials over IMDS.
The EBS-specific AWS_EC2_ENDPOINT/AWS_ENDPOINT_URL_EC2 stay in both modes.
Requires an eks-node AMI rebake to take effect.
* docs: route gitops-argocd workbook through a shared ALB and expose ArgoCD UI
Fold the demo app and ArgoCD ingresses onto one ALB via an LBC IngressGroup
with host routing (app./argocd. on a wildcard cert) instead of one ALB per
ingress, add a NodePort + HTTPS Ingress for the ArgoCD UI, and document the
access steps. Add the node-role LBC and EBS-CSI IAM policies the controllers
need (Spinifex wires controller creds at the node level, not IRSA) and an
ignore_changes block for the cluster's access_config and
vpc_config security_group_ids, which Spinifex DescribeCluster does not echo
back, so the provider stops trying to replace the cluster on every plan.
* fix: stop addon-sync substituting IRSA markers named in comments
The ebs-csi controller manifest's header comment spelled out the literal
{{IRSA_ENV}} / {{IRSA_VOLUME_MOUNT}} / {{IRSA_VOLUME}} marker tokens. addon-sync
matched each marker by substring, so the comment lines were replaced with the
env/mount blocks, injecting a bare YAML sequence between two documents. K3s'
auto-deploy then rejected the whole file (ParseManifestFailed: cannot unmarshal
array into unstructured), so the ebs-csi Addon CR never applied and the EKS
addon hung in CREATING. Reword the comment to not contain the literal tokens,
and anchor the marker substitution to lines that are exactly the marker
(whitespace aside) so a future comment mentioning a marker can't retrigger it.
* fix: disable K3s local-storage on EKS clusters in parity mode
In AWS LB Controller parity mode (BuiltinIngress=false) the K3s server
disabled traefik and servicelb but left local-storage enabled. Its local-path
provisioner ships a StorageClass marked default, which races the EBS CSI
ebs-gp3 default StorageClass — two defaults, so dynamic PVCs bind
nondeterministically. EKS has no local-path, so disable local-storage too and
let ebs-gp3 be the sole default. The built-in-ingress path is unchanged.
* fix: re-encrypt HTTPS target groups in haproxy ALB backends
An ALB target group with backend-protocol HTTPS (e.g. argocd-server, which
serves TLS on its own port) rendered a plaintext haproxy server line. HAProxy's
own check then spoke HTTP to a TLS port, failed, and marked the server DOWN,
returning 503 — even though the lb-agent's separate HTTPS health probe reported
the target healthy, so describe-target-health masked the fault.
Emit check-ssl ssl verify none on the server line for HTTPS target groups so
both the check and proxied traffic re-encrypt to the backend.
* fix: reconnect wedged QMP client on next command
A timed-out QMP decode under host contention left the cached client's
shared json.Decoder mid-message, wedging every later command and breaking
EBS AttachVolume/DetachVolume for the VM's lifetime. Mark the client Dead
on encode/decode error and redial the retained socket path (re-running the
qmp_capabilities handshake in place) on the next send.
* fix: round-trip security-group rule descriptions
SG ingress/egress rules dropped the per-rule Description on store and
describe. The AWS Load Balancer Controller tags the worker-node SG rules it
manages with a UserIdGroupPair description and identifies them by that value
to revoke them on teardown. Without the round-trip the controller cannot
match its rules, never revokes them, and the backend security group stays
referenced — DeleteSecurityGroup returns DependencyViolation forever,
hanging the ingress finalizer and tofu destroy with orphaned security groups.
Persist Description on SGRule (excluded from sgRuleKey so it is metadata, not
identity) and render it back through describe.
* fix: reclaim customer-VPC nodegroup SGs on managed-CP-VPC teardown
In the managed control-plane VPC topology, launchNodegroupInfra creates the
cluster control-plane and nodegroup security groups in the customer VPC for
worker<->control-plane networking, but purgeClusterInfra only reclaimed the
security groups in the managed CP VPC. The customer-VPC SGs orphaned,
cross-referenced each other, and pinned the customer VPC with
DependencyViolation, hanging tofu destroy of the VPC. Reclaim them in the
managed-CP-VPC branch too, matching the legacy customer-VPC branch.
* fix: reclaim ALB EIP when system VM already gone on terminate
TerminateSystemInstance only released an internet-facing ALB's Elastic IP on
the local path, where the VM is still present in the manager. When the VM was
already gone (health-reaped, crashed, or raced by ENI teardown), it took the
remote/no-owner path and returned idempotent success without reconciling the
EIP, leaking the wan-pool address still associated to a deleted ENI.
Add EIPServiceImpl.ReleaseAddressByInstanceID, an idempotent account-agnostic
scan that disassociates then releases any EIP recorded against an instance, and
call it as a backstop in TerminateSystemInstance for both branches.
* fix: cascade-prune demo PVC via Argo CD finalizer in eks-gitops workbook
The demo Application carried no resources finalizer, so tofu destroy removed
the Application CR without pruning the synced PVC. The PVC survived, the cluster
was destroyed, and the Delete-reclaim PV's EBS volume orphaned. Add the
resources-finalizer.argocd.argoproj.io finalizer so deleting the Application
cascades to the PVC and ebs-csi reaps the volume before cluster teardown.
* feat: raw-IP access + documented Argo CD creds in eks-gitops workbook
The shared ALB host-routed everything, so the raw ALB IP 404'd in a browser
without an /etc/hosts entry. Make the app the hostless catch-all on :443 and add
a hostless Argo CD listener on :8443, so https://<alb-ip>/ serves the app and
https://<alb-ip>:8443/ serves the UI with no DNS. Surface the admin-password
command as a tofu output and document the credential + IP URLs in the README.
* fix: reclaim ALB EIP on the owning-node terminate path
The EIP backstop lived only in TerminateSystemInstance, but the ALB
delete path goes serveSystemTerminateInstance -> terminateSystemInstanceLocal
directly, bypassing it, and releaseSystemInstanceEIP relied on the VM's
cached allocation which is often empty. Extract reclaimSystemInstanceEIP
(authoritative KV reconcile via ReleaseAddressByInstanceID) and call it
from terminateSystemInstanceLocal as well as the no-owner backstop, so an
internet-facing ALB's EIP is released on every terminate route.
* chore(eks): drop coredns from the managed addon catalog
Cluster DNS ships as the baked node-local coredns-mulga DaemonSet, not as
a managed addon — there is no bundled coredns manifest, so CreateAddon
coredns would stage but fail VM-side render. Remove the phantom catalog
entry so DescribeAddonVersions and CreateAddon no longer advertise it;
repoint the addon tests that used coredns as their sample to argocd.
* feat(ui): group EKS and ECR under a Containers nav section
Fold the separate EKS and ECR sidebar groups into one Containers group
with EKS (clusters) and ECR (repositories) entries.
* fix(ui): satisfy strict null checks in ECR component tests
Index mock.calls via optional chaining and narrow the getAllByRole result
before clicking, clearing the TS2532/TS2345 errors that failed pnpm fix.
* refactor(eks): hide internal fixture addons from the public catalog
Add an AddonSpec.Hidden flag and mark spinifex-noop hidden: it stays
creatable (lookupAddon finds it, so the delivery-transport e2e still drives
it) but no longer surfaces in the unfiltered DescribeAddonVersions listing.
* refactor(eks): remove K3s built-in ingress; LBC-only
Drop the BuiltinIngress option and its managed-ingress tag opt-in. traefik,
servicelb, and local-storage are now always disabled on EKS nodes (AWS
parity — Service type=LoadBalancer and Ingress are the Load Balancer
Controller's job). Removes the EKS_DEFER_TRAEFIK userdata flag, the k3s.initd
traefik .skip staging, and the maybe_enable_traefik deferred-enable loop in
the state reporter.
* fix(ui): clear lint debt in VPC-wizard mutations and CIDR validation
Parallelize the independent subnet-create and route-table-association loops
in the create-VPC mutation with Promise.all (each task records its resource
in the rollback list before any further call), and switch the CIDR-validation
regex to named capture groups. Clears the no-await-in-loop and
prefer-named-capture-group lint errors.
* fix: carry gateway port in ECR registry endpoints; rename mulga.internal -> spinifex.internal
The ECR registry is served on the gateway port (:9999), but proxyEndpoint and
repositoryUri were emitted port-less, so docker login/tag/push dialed :443 and
failed. Derive the port from the awsgw listen host and append it to the registry
host (omitted for 443). Also rename the default internal DNS suffix.
* docs: drop verbose ECR :9999 host note
* docs: fix literal-asterisk emphasis (single-* not rendered)
* docs: brace-wrap ${REGISTRY}:latest (zsh :l modifier ate the tag); document Docker CA trust
* feat: default ECR registry URIs to the gateway host (no-DNS interim)
The registry authenticates by token, not hostname, so the per-account
<acct>.dkr.ecr name is cosmetic. Advertise the gateway's reachable bind host:port
(already a cert SAN) in repositoryUri/proxyEndpoint so docker login/tag/push need
no DNS; the parity host still works server-side where DNS exists.
* fix: serve ECR registry on a DNS-free gateway host end to end
Advertise the gateway's AdvertiseIP as the ECR registry host when the AWSGW
bind is unspecified (0.0.0.0/::), so repository_url resolves to a reachable
host:9999 carried in the server cert SANs rather than the unresolvable
per-account parity name. Docker push then dials it directly with no DNS.
Mirror the same host on EKS workers: the registries.yaml mirror keys and the
credential provider matchImages now cover the port-less parity name, the
port-ful parity name, and the gateway IP:9999, all redirecting to the dialed
gateway endpoint. A port-ful pull from the API-returned repository_url matches
the mirror instead of bypassing it into a failed DNS lookup.
Net: docker push host, pod image host, and the API's repository_url are the
same value, with no DNS or /etc/hosts.
* docs: resolve eks-https Ingress ALB DNS name to its public IP (no DNS)
* docs: resolve eks-gitops Ingress ALB DNS name to its public IP (no DNS)
* docs: remind to run aws eks update-kubeconfig before kubectl in EKS workbooks
* feat(ecr): return OperationNotSupportedException for image-scanning actions
The ECR image-scan control-plane actions resolved to the generic
NotImplemented 501 stub, indistinguishable from unbuilt features and
surfaced as a server fault. mulga runs no vulnerability scanner, so the
scan surface now returns OperationNotSupportedException (400):
- awserrors: add OperationNotSupportedException (400) ECR code.
- ecrapi: ScanningNotSupported handler; repoint the four image-scan
actions and register the three registry-scanning actions at it.
- spinifexui: Scan tab on the repository detail page renders a static
'scanning not supported' panel rather than a request that errors.
* refactor(config): default internal DNS suffix to spinifex.internal
Rename the AWS-parity internal DNS suffix from mulga.internal to
spinifex.internal across the config default, cert SAN builder, ECR
endpoint/audience construction, and their tests. Single runtime knob is
config.DefaultAWSInternalSuffix; no DNS backend exists yet, so this is a
naming change with no functional impact.
The ECR auth bridge minted ES256 tokens from a key that was created once and
never rotated. Add a background rotator (ecr-v1 Q3): mint a fresh key every 30
days, retain the previous keys for 24h so in-flight 12h tokens stay verifiable,
then prune. The active key is now the newest by KV creation time (not the
lexically-greatest kid) so a minted key takes over and all nodes converge. The
Issuer and Verifier became hot-swappable so the rotator updates the live
gateway as keys roll.
* feat: bake aws-ebs-csi-driver addon + mdev by-id bridge
Add the aws-ebs-csi-driver entry (v1.40.1) to the eks addon catalog and
ship its baked manifests (RBAC, IRSA-wired controller, node DaemonSet,
default gp3 StorageClass) under the eks-node image.
The node plugin locates EBS volumes via /dev/disk/by-id, which busybox
mdev does not populate. Add mulga-ebs-byid: an mdev rule + helper that
symlinks nvme-Amazon_Elastic_Block_Store_<serial> from the virtio-blk
serial, plus a boot oneshot that points the kernel hotplug helper at
mdev and coldplugs volumes present at boot.
* test: e2e EBS CSI volume — PVC bind + mount + reschedule survival
Add a TestEKS/EBSCSIVolume subtest that installs the aws-ebs-csi-driver
managed addon bound to an IRSA role, provisions a default gp3 PVC + pod,
and asserts a marker written by the first pod survives a reschedule onto a
fresh pod backed by the same PVC. Exercises the full data path:
CreateVolume(Viperblock)/AttachVolume virtio-blk hotplug with serial set to
the volume-id, mdev /dev/disk/by-id symlink, and ext4 mount.
* feat: eks-node provider-id + topology/instance-type labels for EBS CSI
mulga-eks-provider-id boot oneshot writes a K3s config.yaml.d drop-in
from IMDS before K3s starts: kubelet provider-id plus the region, zone
and instance-type node labels. The aws-ebs-csi-driver node plugin reads
this metadata for NodeGetInfo (instance id from providerID, region/zone
from topology labels, instance type for the volume-limit family lookup).
k3s runs no AWS cloud-provider so none of it is set otherwise, which left
the node plugin's metadata init failing, NodeGetInfo panicking in
IsNitroInstanceType, and the CSINode carrying no topology key. Wire the
service into the image manifest (INSTALL_FILES + ENABLE_SERVICES) and
chmod it in setup.
* test: e2e EBS CSI controller IRSA policy + failure diagnostics
awsgw enforces IAM on assumed-role EC2 calls, so the controller's
CreateVolume/AttachVolume need a permission policy on the IRSA role.
Attach a customer-managed EC2 policy (AWS-managed ARNs are opaque and
grant nothing) covering the EBS CSI action set; CreateVolume then
succeeds. Add an OnFailure dump of the CSI control path (provisioner and
plugin logs, pvc/pod describe, events, csinode) to the EBSCSIVolume
subtest so provisioning/attach failures are diagnosable post-teardown.
* feat: EBS CSI obeys control-plane taint; e2e worker nodegroup
Drop the CriticalAddonsOnly/NoExecute tolerations from the EBS CSI
controller + node manifests so the driver runs only on customer worker
nodes, never the system control-plane VM. The e2e creates a 1-node
worker nodegroup and the writer/reader pods drop their CP tolerations so
the customer-owned volume attaches in the customer space.
* fix: EBS CSI attach — decouple hotplug PCIe port from device name; iothread rollback
virtio-blk-pci hot-plug bus was derived from the AWS device-name letter
(/dev/sd[f-p]); the EBS CSI driver sends arbitrary names (/dev/xvdaa) that
map to no slot, so device_add fell back to pcie.0 — which rejects hot-plug —
and the attach hung. A failed attach also left the per-volume iothread object
behind, so every retry died on object-add 'duplicate property'.
nextHotplugEBSPort allocates the lowest free hotplug-ebs port from live
query-block state, independent of the device name. delIothreadBestEffort
removes the iothread on every attach rollback path.
* fix: EBS CSI node discovery — un-shadow mdev by-id rule, mint nvme link
Alpine eks-node runs busybox mdev (no eudev), which stops at the first
matching /etc/mdev.conf rule. setup.sh appended the mulga-ebs-byid rule after
the stock 'vd[a-z].* -> persistent-storage' line, so it never fired on
hot-attach: only /dev/disk/by-id/virtio-<serial> was created, while the
upstream aws-ebs-csi-driver resolves nvme-Amazon_Elastic_Block_Store_<volid>.
Swap the stock vd* rule in place to route through mulga-ebs-byid, which now
delegates to persistent-storage (preserving stock links) before minting the
nvme-style link the node plugin expects.
* test: EKS e2e — NAT Gateway egress for private worker nodegroup
The customer-VPC worker is private; AttachIGW programs no SNAT (mirrors AWS:
an IGW serves only public-IP instances), so an IGW-only fixture left the worker
unable to reach the public NLB endpoint or pull public CSI images — the
nodegroup never joined.
createWorkerEgress now builds the real topology: an IGW-fronted public subnet
hosting a NAT Gateway, with the worker subnet default-routed to it. Associating
the worker route table triggers the subnet-wide SNAT egress reroute. Teardown
drains the NAT Gateway before releasing the public subnet + EIP it pins.
The OCI /v2 registry auto-created a repository on first push (ensureRepo).
AWS ECR does not: a push or PutImage to a repository that was never created
with CreateRepository is rejected. Replace ensureRepo with requireRepo, which
returns OCI NAME_UNKNOWN on the data plane and RepositoryNotFoundException on
the JSON PutImage path. Enforcement sits at the single push choke point
(startUpload + StoreManifest).
* test(e2e): add ECR end-to-end suite (control plane + docker/crane/skopeo)
New tests/e2e/ecr/ suite plus harness support:
- harness: ECR SDK client, control-plane helpers (repo CRUD, get-login-password,
describe-images), and docker/crane/skopeo CLI wrappers that skip when the
binary or registry-host resolution is absent.
- suite: control-plane CRUD + policy + lifecycle round-trip; docker login/push/
pull; crane append/pull and skopeo copy exercising the Bearer /v2/token flow.
- registered e2e-ecr in the Makefile + service manifest.
Predastore capacity/load leg is a skipped placeholder pending mulga-siv-359/356.
* test(e2e): carry awsgw port in ECR registry host for dev clusters
Dev clusters bind awsgw on a non-standard port (e.g. 9999); production uses
:443. ECRRegistryHost now appends SPINIFEX_ECR_PORT when set so docker/crane/
skopeo address the right endpoint, and RequireRegistryResolves strips the port
before DNS lookup.
getManifest read only the first Accept header line via r.Header.Get, so
clients that send each acceptable manifest media type as a separate Accept
header (skopeo) never matched the stored type and got 406 MANIFEST_INVALID.
Join all Accept header values before negotiation.
Add the Docker Registry v2 token endpoint (GET /v2/token) and switch the
/v2 401 challenge from Basic to a Bearer realm pointing at it. The endpoint
verifies the caller's existing AWS:<jwt> credential and re-mints a fresh
full-TTL token, providing the token-refresh path for OCI clients (docker,
crane, skopeo). The auth bridge still accepts Basic AWS:<jwt> directly.
Stamp the dash-stripped volume-id (vol+17hex = 20 bytes, the virtio-blk
serial limit) as the hotplug disk serial so the EBS CSI node plugin can
locate the device via /dev/disk/by-id and match lsblk SERIAL.
* feat: ECR background lifecycle expiry sweep
Add a gateway-side periodic sweep (LifecycleSweeper) that applies each repo's
stored lifecycle policy: it evaluates the shared rule engine against the repo's
current images and deletes the expired set through the existing registry GC
reclaim path (DeleteImage -> reclaimManifest). It runs in the gateway because
only the gateway holds a Registry (object store + meta); the daemon owns the KV
but cannot reclaim predastore bytes. Launched on the STS janitor's lifetime
context in awsgw, enumerating ACTIVE IAM accounts.
Per repo: skip when no policy; evaluate; then before deleting, re-read the
current tag set and skip any image whose tags changed since evaluation
(optimistic-concurrency guard against concurrent push/re-tag). Cross-repo blob
mount safety is provided by reclaimManifest/referencedDigests. Per-item errors
are logged and skipped so one bad repo never aborts the cycle.
* test: cover lifecycle sweeper Run loop and active-account filter
Add tests for LifecycleSweeper.Run ticker loop, NewLifecycleSweeper
interval fallback, and sweepRepo error branches. Extract accountLister
interface so activeAccountIDs is unit-testable. Raises diff coverage
on the sweep changes from 53.4% to 76.9%.
* feat: ECR lifecycle policies (CRUD + evaluation engine + preview)
Implement PutLifecyclePolicy/GetLifecyclePolicy/DeleteLifecyclePolicy and
StartLifecyclePolicyPreview/GetLifecyclePolicyPreview, replacing the
NotImplemented stubs. A pure rule-evaluation engine parses the AWS
lifecycle-policy JSON and, given a repo's images, returns the ordered set that
would expire; each image is claimed by at most one rule (lowest rulePriority
first). Preview is served synchronously and deletes nothing.
Lifecycle-policy documents persist as a per-repo KV doc (repos/<name>/lifecycle)
over the ecr.lifecycle.* NATS subjects, mirroring the repo-policy storage stack,
and cascade on DeleteRepository. The console gains a Lifecycle tab with a policy
editor and a preview table over the saved policy.
The background expiry sweep that applies the engine and deletes images via GC is
deferred to mulga-siv-387.
* feat: client-side JSON validation in ECR policy editors
Swap the raw Textarea in the repository-policy and lifecycle-policy editors for
the shared JsonEditor (blur-time pretty-print) and gate Save on isValidJson,
showing an inline error for malformed JSON — matching the IAM/EKS editors. The
backend remains the source of truth for schema validation.
Honor, persist and echo imageTagMutability on CreateRepository and
DescribeRepositories instead of the hardcoded MUTABLE, and implement
PutImageTagMutability to flip a repository between MUTABLE and IMMUTABLE.
Enforce immutability at the shared StoreManifest chokepoint: on an
IMMUTABLE repo, re-tagging an existing tag onto a different digest is
rejected before any write (OCI 409 TAG_IMMUTABLE; JSON PutImage maps to
ImageTagAlreadyExistsException). Same-digest re-pushes, new tags and
digest references are still allowed.
Console: migrate the repository detail page to Summary/Images/Permissions
tabs and add a tag-immutability toggle on the Summary tab.
DELETE /v2/{name}/manifests/{digest} was a 202 no-op. Now delete-by-digest
removes the manifest record, deletes the manifest object, and sweeps the
account-wide blob pool: each child blob no longer referenced by any live
manifest is reclaimed, while shared blobs survive.
- DeleteImage (digest path) captures ChildDigests, deletes the manifest object,
then reclaimManifest() drops orphaned blobs via referencedDigests() (a
mark-and-sweep over the live manifest set across the account's repos).
- OCI deleteManifest by-digest wired to DeleteImage (real work + 202; 404 on
unknown). Tag delete stays KV-only so untagged images persist (AWS-faithful).
- Predastore reclaim is best-effort: a failure leaks bytes but never undoes the
logical delete.
Serve the five image-management JSON 1.1 actions inline in ECR_Request,
reusing the gateway-side OCI Registry for manifest bytes:
- ListImages / DescribeImages enumerate images by tag + digest (KV meta).
- BatchGetImage returns verbatim manifest bytes with Q14 partial-failure
semantics (cap 100, digest wins over tag, 200 + failures array).
- PutImage stores a manifest + tag via a shared StoreManifest core that the
OCI manifest PUT now delegates to.
- BatchDeleteImage removes tag pointers / manifest meta (blob GC deferred).
Adds DeleteManifestMeta across the MetaStore relay chain and ECR image
error codes.
Add the two missing repository-lifecycle actions, served inline in
ECR_Request (like DescribeRepositories) since both return the repository
ARN/URI built from the gateway region + internal suffix.
CreateRepository validates the name, guards cross-account registryId,
rejects an existing repo (RepositoryAlreadyExistsException), and writes
the per-account KV meta (predastore bucket stays lazy on first push).
DeleteRepository rejects a non-empty repo without force
(RepositoryNotEmptyException) and cascades the KV records (meta, policy,
tags, manifests). Predastore blob GC is deferred to a separate pass.
Adds MetaStore DeleteRepo + ListManifests with the matching daemon-relayed
subjects, service methods, and NATS client, plus two new awserrors codes.
The per-run ARTIFACT_DIR lives in /tmp, outside GITHUB_WORKSPACE, so a
re-run on the same self-hosted runner reused a prior attempt's append-only
suite-status.txt. The 'Fail job if any suite failed' gate's -s check then
failed the job even when the surviving attempt passed with rc=0.
Wipe ARTIFACT_DIR in Clean Workspace and gate solely on the per-attempt
steps.e2e.outputs.rc (fail-closed: an empty rc still fails).
CreateTags/DeleteTags wrote only the standalone tag store, which
DescribeImages does not read (it reads AMI metadata), so tag mutations on
an AMI were invisible — DeleteTags appeared to be a silent no-op. Mirror
tag add/remove into the owning AMI config (ownership-checked; absent or
foreign AMIs skipped) so DescribeImages reflects them, matching the
existing subnet/vpc mirror.
* feat(ecr): auth bridge + GetAuthorizationToken JWT (Sprint 2c)
Mint a self-contained ES256 JWT from GetAuthorizationToken and verify it on
the /v2 OCI registry via a new ECRAuthBridgeMiddleware (Bearer | Basic AWS:jwt).
Signing keys live in the cluster-replicated awsgw-keys KV bucket under
jwt-signing/{kid}, encrypted with the IAM master key.
The registry is now scoped per-request to the token's account (replacing the
fixed bootstrap account); bucket-ensure is cached per account. Cross-account
token/host mismatch is rejected 403; unauthenticated /v2 access gets a 401
Bearer challenge.
* fix(ecr): use Basic auth challenge on /v2 (not Bearer token-endpoint flow)
ECR authenticates the OCI registry with Basic AWS:<token>; a Bearer realm
made docker chase an unimplemented /v2/token endpoint and fail login.
* feat(ui): inline EKS cluster IAM role creation in wizard
* feat(ui): show trust policy + attached managed policy in EKS role dialog
Render the trust policy with PolicyDocumentViewer (pretty-printed, safe
fallback) instead of a raw read-only textarea, matching the IAM detail
pages. Surface the attached AmazonEKSClusterPolicy ARN so users can see
exactly which permissions the cluster role grants.
* fix(ec2): stamp subnet AvailabilityZoneId for single-AZ ALB resolution
DescribeSubnets omitted AvailabilityZoneId, so the AWS Load Balancer
Controller's DescribeAvailabilityZones-by-ZoneId lookup resolved no AZ
info ("cannot resolve AZ info for AZs: []") and never built the ALB
model. Stamp the single zone-id on every subnet and share it with
DescribeAvailabilityZones through one SingleZoneID constant so the two
responses can never drift.
* feat(eks): add aws-load-balancer-controller v2.8.1 addon bundle
Baked manifests for the LBC managed addon (CRDs, controller Deployment/
RBAC/webhooks/Service, IngressClass alb, IRSA render markers). Two
wirings make it work against the spinifex gateway:
- --aws-api-endpoints carries sts= alongside elasticloadbalancing/ec2/
acm. The aws-sdk-go v1 credential bootstrap resolves the STS client
from the session endpoint resolver, not AWS_ENDPOINT_URL_STS, so the
web-identity AssumeRoleWithWebIdentity must be pointed at the gateway
here or it falls back to real AWS STS.
- --feature-gates=ALBSingleSubnet=true drops the ALB minimum subnet
count from two to one so an ALB provisions in a single-AZ deployment.
* feat(eks): render IRSA into managed addons + node-minted webhook cert
Extend the addon-sync renderer to inject the projected-token / role-arn /
gateway STS-endpoint + CA block into managed-addon pod specs via opt-in
markers, and substitute cluster-name/region/vpc-id/gateway-endpoint/CA
placeholders. Add cmd/eks-webhook-cert: the eks-node AMI has no openssl,
so this mints (then reuses) a self-signed ECDSA-P256 serving cert for the
admission webhook, cached on disk so the render is byte-stable across
ticks. Wire both into the image build (manifest.conf) plus INSTALL_FILES
for the LBC bundle.
* feat(eks): surface cluster VPC id as EKS_VPC_ID for the LB controller
Thread the control-plane VPC id through K3sServerInput.VpcID into the
first-boot EKS_VPC_ID env so the in-cluster aws-load-balancer-controller
can pass --aws-vpc-id to the gateway elbv2/ec2 handlers.
* fix(eks): point the LB controller at the customer VPC + customer-facing gateway
The aws-load-balancer-controller ran with two control-plane-scoped values
that customer worker pods cannot use:
- --aws-vpc-id was set from the managed control-plane VPC (cpRefs.VpcID),
which the gateway's DescribeVpcs does not return for customer calls, so
backend-SG auto-create failed with InvalidVpcID.NotFound. Source it from
meta.ResourcesVpcConfig.VpcId, the customer cluster VPC where the workers
and their subnets live and where the ALB is provisioned.
- The addon render baked the mgmt gateway endpoint (EKS_GATEWAY_URL /
SystemGatewayURL) into pod specs, unreachable from worker pods. Add a
distinct EKS_ADDON_GATEWAY_URL (the customer-facing GatewayBaseURL, same
host as the OIDC issuer) and substitute it for {{GATEWAY_ENDPOINT}} in
managed addon manifests. EKS_GATEWAY_URL stays mgmt for the CP's own
bootstrap, state-report and token-webhook paths.
* fix(elbv2): resolve forward target group from ForwardConfig
listenerActionFromSDK read only the legacy flat Action.TargetGroupArn, so a
forward action carrying its target group via ForwardConfig.TargetGroups (the
shape the aws-load-balancer-controller emits) left the action target empty and
CreateRule/CreateListener rejected it with MissingParameter. Flatten the first
target group from ForwardConfig when the flat field is absent.
* fix(ec2): mirror CreateTags/DeleteTags into subnet/vpc records
The generic S3 tag store and the per-resource KV records are disjoint:
DescribeSubnets/DescribeVpcs read tags from their KV record, while
CreateTags only wrote the generic store. Subnet tags applied after
create were therefore invisible to tag-filtered describes, breaking
aws-load-balancer-controller subnet auto-discovery (kubernetes.io/role/elb).
Write through CreateTags/DeleteTags to the owning subnet/vpc record so
tag-filtered describes observe them. Resource types the VPC service does
not own are skipped; the generic store stays their record of truth.
* test(daemon): pin synthetic capacity in host-dependent allocate tests
TestCanAllocate_CountEdgeCases/Mixed_instance_types and
TestHandleEC2DescribeInstanceTypes/NoFilterReturnsSupportedRegardlessOfAllocation
called rm.allocate against live-detected host capacity. allocate gates on
readMemAvailableGB (live /proc MemAvailable), so a box already running VMs
fails the allocation spuriously, breaking the preflight gate locally.
Pin hostVCPU/hostMemGB/reserved and nil readMemAvailableGB to a synthetic
host, matching the sibling Capacity_decreases_after_allocation subtest.
* fix(eks): route apiserver->pod traffic via agent tunnel (egress-selector cluster)
In the managed-CP topology the apiserver VM sits in the system CP VPC with
no route to the worker pod network in the unpeered customer VPC. k3s default
egress-selector-mode=agent tunnels only apiserver->kubelet, so kubectl
logs/exec work but apiserver->pod traffic (admission webhooks, aggregated
APIs) is attempted directly and times out -- the aws-load-balancer-controller
webhooks fail with context deadline exceeded, blocking Service/Ingress
creation cluster-wide.
Set egress-selector-mode=cluster so apiserver->pod/service traffic routes
through the agent's outbound tunnel, selected by the node hosting the
destination, without requiring inbound CP->worker reachability.
* feat: bump aws-load-balancer-controller addon bundle to v2.11.0
v2.8.1 predates the ALBSingleSubnet feature gate, so single-AZ spinifex
clusters fail Ingress reconcile (subnet resolver dedups to 1 < required 2)
and never provision an ALB. v2.11.0 honors --feature-gates=ALBSingleSubnet=true.
Replace the baked bundle: new CRDs (ingressclassparams + targetgroupbindings),
controller manifest, and image tag v2.11.0, re-applying the spinifex deltas
(IRSA SA annotation, node-minted webhook serving-cert secret, gateway-CA
secret, gateway aws-api-endpoints, caBundle on every webhook clientConfig,
alb IngressClass). Pick up the new ClusterRole configmaps rule. Drop v2.8.1.
* fix: point LBC at the gateway via AWS_ENDPOINT_URL (sdk-v2)
LBC v2.11 uses aws-sdk-go-v2, which keys --aws-api-endpoints on SDK
service IDs (e.g. 'Elastic Load Balancing v2'), not the legacy lowercase
names the bundle passed. Those overrides were silently ignored, so every
non-STS call (ec2/elbv2/acm) went to real AWS and failed TLS against the
gateway CA -- no ALB could provision.
Inject AWS_ENDPOINT_URL in the shared IRSA env block (sdk-v2 global
base-endpoint override; AWS_ENDPOINT_URL_STS still takes precedence for
STS) and drop the dead --aws-api-endpoints flag from the controller args.
* fix: synthesize taggable ARN for default listener rule
Gateway DescribeRules returned the synthetic IsDefault rule with a null
RuleArn. LBC read the arn-less default rule after creating a listener and
issued DescribeTags with an empty resource ARN, which the gateway rejected
with 400 MissingParameter, aborting reconcile.
Set a deterministic RuleArn on the default rule, resolve it back to its
parent listener in DescribeTags (empty tags) and DescribeRules by-arn.
Also disable shield/waf/wafv2 in the LBC bundle to drop unsupported calls.
* feat: inject ELB subnets into alb IngressClassParams for single-AZ ALB
LBC's tag auto-discovery never threads the ALBSingleSubnet feature gate, so a
single-AZ cluster (subnets sharing one AZ) dedups to 1<2 subnets and aborts
Ingress reconcile. Only the explicit-subnet path honors the gate.
Resolve the cluster's ELB-eligible subnets, dedup to one per AZ, and surface
them as EKS_ELB_SUBNET_IDS. The addon sync renders them into the alb
IngressClassParams .spec.subnets.ids so every Ingress takes the explicit path.
An empty set falls back to auto-discovery, preserving the native UX.
* fix: route LBC service clients to gateway via --aws-api-endpoints, link IngressClassParams
Live verification showed two gaps in the LBC addon bundle:
LBC installs a per-client BaseEndpoint resolver, so the global AWS_ENDPOINT_URL
is ignored for the ec2/elbv2/acm clients (calls hit real AWS, failing TLS).
Redirect them with --aws-api-endpoints keyed on the aws-sdk-go-v2 service IDs
(EC2, "Elastic Load Balancing v2", ACM, STS) rather than the legacy lowercase
names the flag never matched. Drop the misleading AWS_ENDPOINT_URL env; the
web-identity STS client still resolves via AWS_ENDPOINT_URL_STS.
The injected IngressClassParams was orphaned: the alb IngressClass had no
.spec.parameters, so LBC fell back to subnet auto-discovery (which fails on a
single-AZ cluster). Link the IngressClass to the IngressClassParams so the
explicit-subnet path is taken.
* fix: stamp EKS-parity providerID on worker nodes for LBC ip targets
k3s registers workers with providerID k3s://<name>, which the AWS Load
Balancer Controller rejects (providerID invalid for EC2 instances) when
resolving nodes for ip-mode target registration. Resolve the instance-id and
AZ from IMDSv2 at agent boot and set --kubelet-arg=provider-id=aws:///<az>/<id>
so LBC can map the node to its EC2 instance. On any IMDS miss the agent keeps
the default providerID and still starts.
* fix(elbv2): honor SubnetMappings in CreateLoadBalancer
LBC supplies ALB subnets via SubnetMappings, not the plain Subnets list.
createLoadBalancer only read input.Subnets, so ENI allocation was skipped,
VpcId/AZ stayed empty, and the HAProxy data-plane VM never launched — the
ALB went active but could not serve traffic. Add flattenSubnetIDs to read
both sources and reuse it in createLoadBalancer and desiredSubnetSet.
* fix(vm): capture GCInterval before spawning GC goroutine
TestStart_SweepsImmediatelyThenStopsOnCtxCancel mutates the package global
GCInterval and restores it in a defer, racing the Start goroutine's read at
time.NewTicker(GCInterval) under -race. Read GCInterval on the caller's
goroutine into a local before spawning so the global access is ordered with
the test's defer; the goroutine touches only the captured local.
* feat: implement OCI Distribution v2 blob and manifest storage
Replace the /v2/* 501 stub with a real path-parsing dispatcher and
content-addressable storage for the ECR registry surface:
- Blob upload (monolithic + chunked PATCH), HEAD/GET, cross-repo mount,
and cancel, with server-side sha256 verification on finalize.
- Manifest PUT with type-aware shallow validation (image manifest blob
existence; image index child-manifest existence + mediaType match),
byte-for-byte GET with stored Content-Type, HEAD, and tag delete.
- Tag list and account-scoped catalog.
Blob and manifest bytes stream straight to predastore; repo, tag,
manifest and in-progress-upload metadata live in a per-account
JetStream KV bucket with optimistic-concurrency CAS on uploads. Chunks
are buffered to a temp upload object and reassembled on finalize, since
predastore exposes no append primitive through the shared object-store
interface. Adds HeadObject to the objectstore abstraction.
Refs: mulgadc/mulga#73
* refactor(ecr): route registry metadata through NATS daemon handlers
The gateway no longer talks JetStream KV directly. The daemon owns the
per-account KV via a MetaService bound to ecr.* request/reply subjects;
the gateway is a NATS client (NATSMetaStore). Blob and manifest bytes
still stream gateway-to-predastore directly.
Also fixes four registry review findings:
- finishUpload: clean up the temp object and upload record on a blob
store failure so the uuid is reusable and no partial blob lingers.
- patchUpload: commit the appended bytes and advanced hash state under a
single KV CAS keyed per write, so a racing PATCH is rejected (409)
rather than corrupting the assembled bytes; the verified digest always
matches the stored bytes.
- cancelUpload: delete both the temp bytes and the record, 204 on
success, 404 on unknown uuid, no dead branch.
- getManifest: return MANIFEST_INVALID (not MANIFEST_UNKNOWN) for a 406
Accept media-type mismatch.
Refs: mulgadc/mulga#73
* fix(ecr): provision account predastore bucket on demand
OCI blob/manifest writes target ecr-{accountID}, which nothing created,
so every push failed with a backend 403. Add idempotent EnsureBucket to
the object store (HeadBucket -> CreateBucket, swallowing already-owned)
and ensure it once per process at the top of the registry ServeHTTP, so
ECR storage appears transparently like AWS-managed ECR.
* fix(ecr): sanitize ':' from predastore blob/manifest object keys
predastore mis-canonicalizes SigV4 for object keys containing ':', so
writes/reads of content-addressable keys (sha256:<hex>) were rejected
403 (reproducible with the aws CLI too). Route BlobKey and ManifestKey
digests through DigestToken so the object key carries no ':' separator.
Implement the registry host-routing middleware so /v2/* requests addressed
at {accountID}.dkr.ecr.{region}.{suffix} carry the parsed target account and
region on the request context; non-registry hosts pass through unchanged.
Add cluster-wide AWS-parity config (aws.region, aws.internal_suffix) with
defaults, and append the AWS-parity ECR SANs (exact control-plane host plus
*.dkr wildcard) to the awsgw server certificate during cert generation.
Refs: mulgadc/mulga#78
* feat: scaffold ECR control-plane + OCI registry gateway with 501 stubs
Add the ECR HTTP surface on awsgw as compiling skeletons that return
NotImplemented, so later work can call into the packages against a fixed
contract.
- gateway/ecrapi: ECR control plane (AWS JSON 1.1, X-Amz-Target). Full action
namespace registered; every action resolves to a shared 501 stub.
- gateway/ecr: OCI Distribution v2 (/v2/*). Catch-all 501 with the OCI error
envelope; GET /v2/ is live for the registry version-check probe.
- gateway/hostmatch: no-op host-routing middleware skeleton for the registry
host group.
- handlers/ecr/storage: per-account bucket + object-key layout constants.
- gateway wiring: ecr in supportedServices, dispatch case, JSON 1.1 error
branch, and /v2 mounted outside the SigV4 group.
* test: cover ECR scaffolding packages for diff-coverage gate
In-package tests for gateway_ecr, gateway_ecrapi, and handlers/ecr so the new
code reports coverage under its own package profile. Diff coverage 76.5%.
The kubernetes provider in the eks-quickstart, eks-addons-access and
eks-https-ingress workbooks was configured from the managed cluster
resource (host = aws_eks_cluster.this.endpoint). On destroy the endpoint
resolved empty, the provider fell back to localhost:80, and refreshing
kubernetes_deployment_v1.hello / kubernetes_service_v1.hello failed with
'dial tcp [::1]:80: connect: connection refused'.
Move the kubernetes provider and the hello Deployment + Service into a
nested workloads/ root module with its own state. It reads cluster
identity from the parent via terraform_remote_state and the live
endpoint/CA from a data aws_eks_cluster source, so the provider is only
ever configured while the cluster exists. Apply parent then workloads;
destroy workloads then parent.
purgeClusterInfra stranded the cluster security groups two ways, leaving
the nodegroup/control-plane SGs (which live in the customer VPC) behind to
pin aws_vpc destroy on DependencyViolation:
* an early return before the SG cleanup block skipped DeleteClusterSGs
whenever any earlier best-effort step (NLB / ENI / VM / EIP) errored;
* DeleteClusterSGs failures were warn-and-swallowed, so a nodegroup SG
still referenced by a terminating worker's ENI leaked while the cluster
completed deletion — no reaper retry ever fired.
Surface DeleteClusterSGs / DeleteClusterCPVPC / DeleteClusterIGW failures
into teardownErrs and gate the KV sweep on them, so a leaked VPC-pinning SG
keeps the cluster DELETING for a retry. Add a bounded DependencyViolation
retry to the SG delete (mirroring deleteENIAwaitingTerminate) to ride out
the worker-ENI-detach race in the common case.
Fixes the manual 'revoke rules + delete SGs before the VPC will destroy'
teardown step.
* fix: persist instance TagSpecifications in RunInstance
RunInstance built the ec2.Instance metadata but never read
input.TagSpecifications, so DescribeInstances returned untagged
instances and tag filters matched nothing. EKS node groups discover
their workers by the spinifex:eks-cluster / spinifex:eks-nodegroup
tags, so scale-up convergence and worker reconciliation depended on
tags that were silently dropped.
Apply instance-scoped TagSpecifications onto the built instance so the
tags surface through DescribeInstances and tag filters.
* fix: serialize nodegroup reconcile via CAS to prevent scale overshoot
UpdateNodegroupConfig is dispatched on a queue group, so two
overlapping requests (e.g. an SDK retry mid-flight) each read
current = len(InstanceIDs) and each launched desired-current workers,
overshooting the target (1 -> 5 for desired=3) and stranding untracked
VMs via last-writer-wins on the unconditional PutNodegroupRecord.
Make every durable nodegroup-record mutation a compare-and-swap on the
KV revision. Scale-up launches one worker at a time and CAS-appends it
only while the record is below the live ScalingDesired, terminating any
surplus VM launched into an already-full record. Scale-down terminates
before the CAS truncation so a crash or retry leaves reclaimable dead
IDs rather than orphaned VMs. A concurrent regression test asserts two
overlapping scale-ups converge to exactly desired under -race.
* fix: return securityGroupRuleSet from AuthorizeSecurityGroup ingress/egress
The AWS provider's aws_vpc_security_group_ingress_rule reads
SecurityGroupRules[0].SecurityGroupRuleId on create; an empty ruleset
panics the plugin (Plugin did not respond), though the rule applies
server-side. Echo the created rules via sgRuleToSecurityGroupRule, the
same shape DescribeSecurityGroupRules already returns.
* fix: surface InsufficientInstanceCapacity from nodegroup scale, not 500
launchWorkersCAS wrapped the bare awserror code via fmt.Errorf, so the
gateway ErrorHandler's exact-match against ErrorLookup missed and remapped
to InternalError (500). An out-of-capacity scale now surfaces the real
InsufficientInstanceCapacity code (400); opaque internal failures still
wrap and stay 500. Adds awserrors.HasErrorCode for the membership test.
* fix: cut EKS node boot ~50s — drop cloud-final deadlock + syslinux menu
Worker/server user-data runcmd ended with `[ rc-service, local, start ]`,
which runs inside cloud-final while the OpenRC `local` service is ordered
after it — a deadlock broken only by OpenRC's ~50s start timeout, stalling
the k3s join. Run the IMDS route script directly instead and keep
`rc-update add local default` for reboot persistence.
setup.sh: cut the SYSLINUX boot-menu countdown 10s->~1s (kept nonzero so the
menu stays interruptible over serial).
Validated live on env19 CP: boot menu 1s, no `waiting for cloud-final`
stall, apiserver ready ~66s (was ~109s+).
mulga-siv-332
* feat: add argocd managed addon + bake bundle into eks-node AMI
The addon catalog advertised coredns and aws-load-balancer-controller, but
the only baked manifest shipped in the eks-node AMI was the spinifex-noop
fixture. Installing any real addon failed with 'no baked bundle for version'
once the VM-side renderer found nothing on disk.
Add argocd v3.0.23 as a real, demonstrable managed addon: catalog entry (no
IRSA), a flattened namespaced install manifest baked into the AMI, and the
manifest.conf bake line. The eks-addons-access workbook switches from coredns
(redundant on K3s, never baked) to argocd.
The baked bundle is generated from upstream install.yaml via kubectl kustomize
with namespace: argocd plus a prepended Namespace, since upstream is
namespace-agnostic and K3s auto-deploy applies with no -n flag.
* fix(eks): run CoreDNS as a node-local DaemonSet for worker DNS
k3s ships CoreDNS as a single-replica Deployment that tolerates the CP taint
and strands itself on the control-plane node. In the managed-CP topology the
worker<->CP pod overlay cannot cross the VPC boundary, so worker pods cannot
reach a CP-resident CoreDNS and lose all cluster + external DNS.
Skip the bundled CoreDNS (coredns.yaml.skip, staged in k3s.initd like traefik)
and deploy a Mulga-owned variant: a DaemonSet (one pod per node) plus kube-dns
internalTrafficPolicy=Local, so every node resolves via its own CoreDNS. Immune
to the cross-VPC overlay gap and topology-agnostic (single-node and managed-CP
both work). Image-side only; no handler change.
* fix(eks): revoke cross-referencing SG rules before delete on teardown
DeleteClusterSGs deleted the control-plane and nodegroup SGs directly, but they
cross-reference each other (EnsureNodegroupSGRules), and AWS refuses to delete a
security group still referenced by another SG's rule. Both deletes failed with
DependencyViolation, the SGs leaked, and the VPC could not be deleted (tofu
destroy hung on DeleteVpc).
Resolve both SG IDs, revoke all ingress on each to break the cp<->ng cycle, then
delete both. Adds RevokeSecurityGroupIngress to sgProvisioner (already on
VPCServiceImpl) and revokeAllSGIngress, with a regression test that models AWS's
DependencyViolation.
* fix(eks): advertise NLB front-end so worker pods reach the apiserver ClusterIP
In the managed-control-plane topology the apiserver advertised its Set-B
node-ip into the in-cluster `kubernetes` Endpoints. That IP lives in the
unpeered CP VPC and is unreachable from worker pods, so every worker pod
using the apiserver ClusterIP (CoreDNS kubernetes-plugin sync, controllers,
argocd redis secret-init) failed.
- k3s_server_vm.go: set advertise-address to the worker-reachable NLB
front-end (PrivateEndpointIP, else EndpointIP); both are already cert SANs.
- nlb.go: add a second :6443 listener on the cluster NLB (same CP target
group) so PrivateEndpointIP:6443 — the advertised endpoint — is served.
- security_groups.go: admit :6443 on the private-endpoint SG.
Validated on a live managed-CP cluster: worker CoreDNS /ready 200, all argocd
pods Running, kubectl logs on worker pods working.
The DisassociateAccessPolicy route regex terminated at
/access-entries/{principalArn}/access-policies, but the AWS EKS REST API
sends the policy ARN as the final path segment:
DELETE /clusters/{name}/access-entries/{principalArn}/access-policies/{policyArn}
The route never matched, so the gateway returned InvalidActionException
(seen as a 400 on 'tofu destroy' of aws_eks_access_policy_association).
The handler also read policyArn from the request body, which AWS leaves
empty for this DELETE.
Capture policyArn as a third path param and pass it through to the input,
dropping the body read. Update route and wrapper tests accordingly.
The host clock needs an NTP daemon; without one it drifts and SigV4 request
signatures fall outside the skew window, kicking the web console back to /login.
Mirrors the ISO rootfs (iso-builder packages.list) so both host-install paths
ship the same software. Guest VMs track the host via kvm-clock.
Bug: mulga-siv-335
* fix: preserve terminated-instance describe-visibility window in GC reaper
The TerminatedTeardownReaper purged a terminated record the moment it drove
the instance's pending NAT/OVN teardown to done. Normal terminations always
arrive NAT/OVN-pending, so the first 2m GC sweep completed and purged them
immediately, removing the instance from DescribeInstances before clients could
observe state=terminated — an AWS-compat divergence that flaked e2e-single
TestRunInstancesMultiCount.
Stamp TerminatedAt on transition and purge a completed record only once it is
older than a 15m visibility window; fresher records are left to the terminated
bucket's 1h TTL, keeping the instance describable as terminated. The ADR-0003
§3 purge-on-completion invariants still hold (their seeded records predate the
window). Closes mulga-siv-328.
* test(e2e): poll intra-SG east-west ping until L2 converges
TestSameSGComms fired a single un-retried 'ping -c 3 -W 2' immediately after
the source instance's SSH became reachable, but the ICMP target is the
destination's east-west data plane. Two instances boot in parallel and the
destination guest NIC / OVN port flows come up several seconds after the EC2
running transition the launch waits on. When the probe wins that race the
source kernel reports 'Destination Host Unreachable' (ARP unresolved), not an
ACL drop, and the test flaked.
Replace the one-shot ping with pingConverged, a bounded poll that retries
until 0% packet loss or 45s. A genuine self-ingress/datapath break still
stays at 100% loss for the full window and fails, so coverage is preserved.
Daemons are <tspanfill="#F87171"font-weight="600">stateless</tspan> — scale horizontally by starting more.
No etcd, no Kubernetes, no external control plane. Systemd units, a NATS cluster, and your hardware.
</text>
</g>
</svg>
<svgxmlns="http://www.w3.org/2000/svg"width="1020"height="770"viewBox="0 0 1020 770"role="img"aria-label="Spinifex message-driven architecture: standard AWS clients call the AWS Gateway over HTTPS with SigV4, which publishes to a NATS bus; stateless ec2/ebs/s3/vpc/iam daemons answer and drive local backends (QEMU/KVM, Viperblock, Predastore, OVN/OVS, identity store).">
<textx="48.0"y="697.0"font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif"font-size="12"fill="#5B636C"font-weight="400"><tspanfill="#EE4B12"font-weight="700">Queue groups</tspan><tspanfill="#5B636C"> load-balance ops across daemon replicas. </tspan><tspanfill="#EE4B12"font-weight="700">Fan-out</tspan><tspanfill="#5B636C"> (no queue group) dispatches describe calls to every node.</tspan></text>
<textx="48.0"y="720.0"font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif"font-size="12"fill="#5B636C"font-weight="400"><tspanfill="#5B636C">Daemons are </tspan><tspanfill="#EE4B12"font-weight="700">stateless</tspan><tspanfill="#5B636C">: scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Systemd units, a NATS cluster, and your hardware.</tspan></text>
<textx="24"y="30"font-family="'Segoe UI', system-ui, sans-serif"font-size="11"fill="#a1a1aa"letter-spacing="2.5"font-weight="600">ECS · EC2 LAUNCH TYPE ON SPINIFEX</text>
<textx="876"y="30"font-family="'SF Mono', ui-monospace, monospace"font-size="10"fill="#71717a"text-anchor="end">container instances from spinifex-ecs-node AMI · awsvpc + ALB</text>
<textx="450"y="410"font-family="'Segoe UI', sans-serif"font-size="9.5"fill="#71717a"text-anchor="middle">No ECS launch API — you add capacity by booting EC2 instances from the spinifex-ecs-node AMI (the console's "Provision capacity" wraps RunInstances)</text>
<textx="450"y="426"font-family="'Segoe UI', sans-serif"font-size="9.5"fill="#71717a"text-anchor="middle">The service keeps the desired task count running and registers each task's ENI IP with the ALB target group</text>
<textx="450"y="206"font-family="'Segoe UI', sans-serif"font-size="9.5"fill="#7ee787"text-anchor="middle">managed node group · workers join the control plane</text>
<textx="450"y="404"font-family="'Segoe UI', sans-serif"font-size="9.5"fill="#71717a"text-anchor="middle">You provide the VPC, subnets, and IAM roles · Spinifex provisions the control plane, node-group VMs, and security groups</text>
<textx="450"y="420"font-family="'Segoe UI', sans-serif"font-size="9.5"fill="#71717a"text-anchor="middle">Node groups always boot the registered eks-node image · workers need egress (IGW / NAT) to pull images</text>
<textx="450"y="58"font-family="'Segoe UI', system-ui, sans-serif"font-size="20"font-weight="600"fill="#f5f5f5"text-anchor="middle">AWS-style workflows, on your own stack</text>
<svgxmlns="http://www.w3.org/2000/svg"width="1000"height="664"viewBox="0 0 1000 664"role="img"aria-label="The Spinifex stack: applications and AWS tooling on top, the Spinifex AWS-compatible service layer, running on standard Linux on CPU/GPU hardware, deployable to Neocloud, on-prem or edge.">
<textx="136.0"y="249.8"font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace"font-size="11"fill="#9AA1A8"font-weight="400"text-anchor="start">Runs the same on a single node, cluster, edge, or air-gapped rack.</text>
echo"::error::${label} names suite '${suite}', which is not eligible for that topology (eligible: ${eligible}). Fix the list in .github/scripts/suite-sets.sh." >&2
description:"OTEL_RESOURCE_ATTRIBUTES value for this run, e.g. ci.run_id=123,ci.cell=20 (caller-specific, so it's an input rather than a workflow-level default)"
required:false
type:string
default:""
permissions:
contents:read
@@ -16,6 +21,10 @@ permissions:
env:
GIT_BRANCH:${{ github.ref_name }}
# Same central ES/APM sink as the other four e2e workflows. Bare-metal had
# no default here before — this cell's telemetry never reached the sink at
# all (not just missing ci.cell), see bm-bootstrap.sh's telemetry.env step.
# Manual-work mutex gate — see setup note 4 above. Checked BEFORE any
# SSH to wattle so a paused run touches the box zero times.
- name:Manual-work mutex gate
id:gate
env:
REPO_VAR_PAUSED:${{ vars.CI_GPU_PAUSED }}
run:|
set -euo pipefail
PAUSED=false
if [ -f "$HOME/.ci-gpu-paused" ]; then
PAUSED=true
echo "::warning::ci-gpu runner paused via ~/.ci-gpu-paused sentinel on the runner — skipping this run (manual GPU work in progress on wattle). Remove the file to resume CI."
fi
if [ "$(echo "${REPO_VAR_PAUSED:-}" | tr '[:upper:]' '[:lower:]')" = "true" ]; then
PAUSED=true
echo "::warning::ci-gpu paused via repo/environment variable CI_GPU_PAUSED=true — skipping this run."
fi
echo "paused=${PAUSED}" >> "$GITHUB_OUTPUT"
- name:Paused — nothing to do
if:steps.gate.outputs.paused == 'true'
run:echo "ci-gpu is paused; job exits cleanly without touching wattle. See gate step above for the reason."
echo "::error::GPU e2e suite failed (rc=${{ steps.e2e.outputs.rc }}) — see analysis above; raw ES/APM traces at ${SPINIFEX_OTLP_ENDPOINT} (ci.run_id=${{ github.run_id }})"
$(_Q)LOG_IGNORE=1AWS_MODEL_CONFORMANCE_MODE=$(AWS_MODEL_CONFORMANCE_MODE)AWS_MODEL_CONFORMANCE_REPORT=$(AWS_MODEL_CONFORMANCE_REPORT) go test -tags=integration -timeout 60s ./tests/integration/... $(_RACEQ)
<img src=".github/assets/banner.svg" alt="Spinifex: the open, AWS-compatible cloud you run yourself. EC2, EBS, S3, VPC, and IAM on your infrastructure, at the edge, or on a Neocloud." width="900">
# Spinifex: An Open Source AWS-Compatible Stack for Bare-Metal, Edge, and On-Prem Deployments
# Spinifex: the open, AWS-compatible cloud you run yourself
Spinifex developed by [Mulga Defense Corporation](https://mulgadc.com) is an open source infrastructure platform that brings the core services of AWS—like EC2, VPC, EBS, and S3—to environments where running in the cloud isn't an option. Whether you're deploying to edge sites, private data-centers, or need to operate in low-connectivity or highly contested environments, Spinifex gives you AWS-style workflows on your own hardware.
Spinifex, built by [Mulga](https://mulgadc.com), recreates the AWS services your software already uses (EC2, EBS, S3, VPC, IAM) on infrastructure you control. Move workloads off the hyperscalers, run AI at a fraction of the cost, or take the cloud to the edge, without rewriting a line. The exact build you ship to AWS, Spinifex serves on your own hardware: same APIs, same Terraform, same SDKs.
## What is Spinifex?
Spinifex replicates essential AWS primitives—virtual machines, block volumes, and object storage—using lightweight, self-contained components that are easy to deploy and integrate.
Spinifex is an open-source infrastructureplatform that recreates the AWS service surface (EC2, EBS, S3, VPC, IAM) on bare-metal, edge, and on-premises environments. Run cloud-native software without a hyperscaler.
It's designed for developers and operators who need:
Most AWS alternatives wrap an API and call it a cloud. We rebuilt the engineering underneath: distributed object storage with erasure coding, block storage built to survive failure, and compute on bare metal. That depth is the reason your software runs unchanged, instead of rewritten.
- Cloud-like infrastructure without the cloud
- Minimal dependencies, full control
- Drop-in compatibility with tools like the AWS CLI, SDKs, and Terraform
- A secure cloud environment you control and own the entire hardware, network and software stack
It's built for teams that need:
You can run Spinifex on a few servers in a rack, a field site, or anywhere centralized cloud services aren't feasible.
- The AWS tooling they already use (AWS CLI, SDKs, Terraform), with nothing to rewrite
- Full control of the stack: your hardware, your network, your data, your keys
- A cloud that keeps running offline, through disconnection, with no external control plane
- An open core (AGPL-3.0), auditable and yours to keep, with no lock-in
You change *where* your software runs, not *what* it is.
## The Platform
<p align="center">
<img src=".github/assets/platform.svg" alt="The Mulga Platform — Hardware to Applications" width="900">
<img src=".github/assets/platform.svg" alt="The Spinifex stack: your apps and AWS tooling on top, the Spinifex AWS-compatible service layer, running on standard Linux on CPU and GPU hardware, deployable to Neocloud, on-premise, or the edge." width="900">
</p>
From commodity hardware up to unmodified AWS tooling, every layer is replaceable and yours to own.
## Three Ways To Deploy
One AWS-compatible surface, three deployment shapes. Pick the one that matches your reality; the platform on top is the same.
### Neocloud
Lift and shift onto a partner Neocloud. Move workloads off the hyperscalers and onto our Neocloud partner ecosystem without rewriting them. GPU capacity (H100 / H200 / B200), cheaper, available now. Change an endpoint, keep the software.
### On-premise
Bring your own hardware and host Spinifex in your own data centre. A real multi-node HA cluster integrated with your storage and networking, with full control of stack, data, and jurisdiction. A predictable bill, no egress surprises, an auditable open-source core.
### Edge
Cloud where the cloud can't reach. Air-gapped sites, vehicles, vessels, factories, and clinics. Compute next to the data, running through disconnection, on hardware you own. The same AWS APIs your software already uses.
## AWS Compatibility
Speak the AWS API surface, natively. The AWS SDKs, AWS CLI, and Terraform: everything you deploy on AWS deploys on Spinifex unchanged. At the edge, on-premise, or on a partner Neocloud.
| Service | What it is | Status |
| --- | --- | --- |
| **EC2** | Compute | Available |
| **EBS** | Block Storage | Available |
| **S3** | Object Storage | Available |
| **VPC** | Networking | Available |
| **IAM** | Identity & Auth | Available |
| **ALB / NLB** | Load Balancers | Available |
| **EKS** | Kubernetes | Available |
| **ECR** | Container Registry | Available |
| **ECS** | Container Service | Available |
| **RDS** | Databases | Available |
| **Bedrock** | AI Deployment | Q3 2026 |
Roadmap items ship under the same AWS API surface. Code written for AWS today keeps working the moment they land. [Track what's shipped in the release notes](https://github.com/mulgadc/spinifex/releases).
## Core Components
### Spinifex (Compute Service – EC2 Alternative)
@@ -72,26 +112,34 @@ Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs
- Multipart uploads, streaming reads/writes
- Data redundancy with Reed-Solomon encoding
### Northstar (Authoritative DNS)
[Northstar](https://github.com/mulgadc/northstar) is a lightweight authoritative DNS server that handles both internal service discovery and public-facing DNS for a Spinifex cluster. Zones are human-readable TOML files, served from the local filesystem or synced from Predastore, keeping DNS configuration in the same object storage as the rest of your infrastructure.
- Authoritative DNS over UDP, TCP, and DNS-over-TLS
- Service discovery for cluster components via SRV records
- Zone files stored locally or in S3-compatible storage
<img src=".github/assets/architecture.svg" alt="Spinifex message-driven architecture: standard AWS clients call the AWS Gateway over HTTPS with SigV4, which publishes to a NATS bus answered by stateless ec2/ebs/s3/vpc/iam daemons over local backends." width="900">
</p>
Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless — scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Deep dive: **[`docs/DESIGN.md`](docs/DESIGN.md)**.
Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless: scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Just systemd units, a NATS cluster, and your hardware. Deep dive: **[`docs/DESIGN.md`](docs/DESIGN.md)**.
## Key Features
AWS-Compatible Interfaces – Provision infrastructure with awscli, Terraform, or SDKs you already use.
-Designed for Control – Run on your own terms, whether that's in a datacenter, remote site, or sensitive environment.
-Minimal Dependencies – Each service is standalone and avoids complex orchestration layers.
-Works Offline – No reliance on centralized cloud services or external networks.
- Open Source – Licensed under AGPL3.0. Fork it, extend it, or deploy it as-is.
- **AWS-compatible APIs.** Use the AWS CLI, SDKs, and Terraform you already know. Repoint your endpoint and ship; nothing to rewrite.
- **Zero cloud dependency.** Runs entirely on your hardware. No phone-home, no control plane, no external authority. Works fully offline.
-**Bare-metal compute.** QEMU-based with direct hardware access. No hypervisor tax, no abstraction overhead.
-**Built-in storage.** Block and object storage included: NVMe caching, Reed-Solomon erasure coding, and replication out of the box.
-**Edge-first architecture.** Designed for disconnected, contested, and resource-constrained environments from day one.
-**Open core, no lock-in.** AGPL-3.0 with a commercial option. Inspect it, modify it, deploy it. The platform is yours to keep.
## Installation
Installation requires an Ubuntu / Debian system. See the detailed documentation at [docs.mulgadc.com](https://docs.mulgadc.com) for maintaining and installing Spinifex.
Installation requires an Ubuntu 26.04 or Debian 13 system. See the detailed documentation at [docs.mulgadc.com](https://docs.mulgadc.com) for installing Spinifex.
### Bare Metal ISO
@@ -101,16 +149,11 @@ The recommended installation is a [bootable x86 installer](https://iso.mulgadc.c
curl -fLO https://iso.mulgadc.com/spinifex.iso
```
Follow the [USB install guide](https://docs.mulgadc.com/docs/install-usb) to write the ISO to USB and install on your hardware. The install guide walks through the full process.
Follow the [USB install guide](https://docs.mulgadc.com/docs/install-usb) to write the ISO to USB and install on your hardware.
### Single Node Install
The installation is straightforward to set up and running on a single node for testing purposes. Debian 13 is currently supported, additional Linux distributions are on the immediate roadmap.
>*Prerequisite:* Linux bridge for networking.
Spinifex requires a Linux bridge configured on the host for VM networking. See the [single-node install guide](https://docs.mulgadc.com/docs/install#prerequisites) prerequisites for setup details.
>*Prerequisite:* Spinifex requires a Linux bridge configured on the host for VM networking. See the [single-node install guide](https://docs.mulgadc.com/docs/install#prerequisites) for setup details.
- **[Northstar](https://github.com/mulgadc/northstar)** - Authoritative DNS server
Each component can be developed independently. See component-specific documentation for focused development guides.
@@ -149,10 +193,6 @@ Spinifex ships with a built-in web console — an optional alternative to the AW
The console is served by each node on port `3000` over TLS, and becomes available as soon as `spinifex.target` is up:
```bash
open https://YOUR_NODE_IP:3000
```
- **Same API, different surface.** Every action in the UI is the same AWS SigV4 call the CLI makes — so RBAC, audit trails, and IAM policies apply uniformly.
- **Single sign-on against your AWS credentials.** Log in with the access keys from `~/.aws/credentials` on the node where Spinifex is installed — no separate user database.
- **Self-hosted, works offline.** The UI is embedded in the Spinifex binary and served from the node itself. No external CDN, no analytics calls, no cloud dependency.
@@ -192,6 +232,10 @@ While Spinifex is architected and implemented by experienced engineers, we lever
This hybrid approach ensures Spinifex benefits from both proven engineering expertise and modern development acceleration, while maintaining the quality and reliability standards required for production infrastructure.
## Trademarks
"AWS", "Amazon Web Services", and all related service names (EC2, EBS, S3, VPC, IAM, and others) are trademarks of Amazon.com, Inc. or its affiliates. Mulga Defense Corporation and Spinifex are independent and not affiliated with, endorsed by, or sponsored by Amazon.com, Inc. or Amazon Web Services, Inc. References to AWS services describe interoperability and compatibility only.
## License
Spinifex is open source under the [GNU Affero General Public License v3.0](LICENSE). You're free to use, modify, and deploy it anywhere you need reliable infrastructure without depending on centralized cloud platforms.
retries:=flag.Int("retries",3,"Number of DORA attempts before giving up")
verbose:=flag.Bool("verbose",false,"Print full DHCP packet contents")
broadcast:=flag.Bool("broadcast",false,"Set the BOOTP broadcast flag, as vpcd's Acquire path does, so replies come back to ff:ff:ff:ff:ff:ff rather than unicast to the bridge MAC")
flag.Parse()
// Resolve the hardware address to use.
@@ -44,20 +46,25 @@ func main() {
switch{
case*spinifexID!="":
hwAddr,err=net.ParseMAC(generateMAC(*spinifexID))
// vpcd's own derivation, so the probe leases under the MAC the daemon
// would have used for this id.
hwAddr,err=dhcp.DeriveMAC(*spinifexID)
iferr!=nil{
log.Fatalf("generateMAC(%q): %v",*spinifexID,err)
slog.Error("Could not derive MAC","spinifex-id",*spinifexID,"error",err)
os.Exit(1)
}
fmt.Printf("spinifex virtual MAC for %q: %s\n",*spinifexID,hwAddr)
case*macStr!="":
hwAddr,err=net.ParseMAC(*macStr)
iferr!=nil{
log.Fatalf("parse --mac %q: %v",*macStr,err)
slog.Error("Could not parse --mac","mac",*macStr,"error",err)
os.Exit(1)
}
default:
hwAddr,err=ifaceMAC(*iface)
iferr!=nil{
log.Fatalf("get MAC for %s: %v",*iface,err)
slog.Error("Could not get MAC for interface","iface",*iface,"error",err)
os.Exit(1)
}
fmt.Printf("using real interface MAC: %s\n",hwAddr)
}
@@ -83,7 +90,7 @@ func main() {
if*promisc{
iferr:=setPromisc(*iface,true);err!=nil{
log.Printf("WARNING: failed to set promisc on %s: %v",*iface,err)
slog.Warn("Failed to set promisc on interface","iface",*iface,"error",err)
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Julian Sommer
