Proves a malformed ImageId and a malformed targeted-launch CapacityReservationId are rejected before any NATS dispatch, with no daemon stub registered so a regression that let either through would fail fast instead of hanging.
Proves version selection (omitted Version resolves to the template default, explicit Version selects that version) and field precedence (explicit RunInstances parameters override the template, unset fields inherit it) through the real gateway.
Proves replaying the same ClientToken through the real gateway returns the original reservation without a second daemon dispatch, and a different token launches independently.
ListFoundationModels is pure gateway-side static catalog logic with no
NATS/daemon hop, so it moves to the integration tier via a new BedrockClient
harness accessor; this tier configures no credential resolver, so the
Anthropic entry's absence is asserted explicitly rather than left optional.
runPredastoreObjectLifecycle only needs a real predastore daemon, not a
tofu-provisioned cluster: it now runs against tests/fixtures/predastore's
in-process fixture, covering the same create/put/get/delete/list contract.
runImage's AMI describe round-trip needs no real cluster: it registers an
AMI against a hand-seeded snapshot via the real ImageServiceImpl, backed by
a new StartImageDaemonLite helper wiring RegisterImage/DescribeImages onto an
in-memory object store.
Enable/Disable/GetSerialConsoleAccessEnabled is pure gateway+account-settings
logic with no guest involved, so it moves to the in-process integration tier
against the real AccountSettingsServiceImpl wired into DaemonLite.
ClientTokenStore in spinifex/gateway/ec2/instance is a process-wide
sync.Once singleton bound to whichever *nats.Conn first initialises it.
Every integration test booted its own embedded NATS server, so the first
RunInstances-calling test captured the singleton, and any later test
inherited a KV handle bound to that server's now-closed connection.
TestMain now boots one embedded NATS+JetStream server for the whole
package; every StartGateway call connects into its own dynamically
provisioned NATS account (accountAuthenticator in shared_nats.go), giving
each test a private JetStream namespace despite the shared server, and
none of the per-test connections are closed until every test has finished
- whichever one the singleton latches onto stays live for the run.
Also wire the suite into the required PR gate: it was previously only
invoked from e2e.yml's push-triggered (main/dev), self-hosted "Integration
(front gate)" job, so a regression here never blocked a PR. The whole
package runs in single-digit seconds, so it now runs inline in
`make preflight` and as its own step in spinifex.yml's build_and_test job.
RequestSpotInstances is a mock over on-demand RunInstances (no bidding,
interruption, or reclamation), so booting real VMs in the e2e tier proved
nothing spot-specific. Migrate the SIR CRUD lifecycle (request, describe,
cancel, close-on-terminate, insufficient-capacity error) to a NATS-level
integration test that exercises the real gateway orchestration and daemon
service impl without a live guest. Real-VM-boot assertions are already
covered by TestRunInstances_MultiCountExpansion and the e2e baseline tests.
TestLaunchTemplates booted zero guests: its RunInstances coverage had
already moved to TestRunInstances_LaunchTemplateExpansion, leaving only
template CRUD/versioning assertions occupying a full live-cluster slot.
Port the remaining create/describe/version/modify/delete lifecycle,
including the SourceVersion merge and a nested timestamp round-trip, into
TestLaunchTemplates_CRUDLifecycle against the in-process integration
gateway.
Ports the KeyPair/VPC-Subnet/IGW-EIGW/AccountSettings cross-account
isolation assertions from tests/e2e/single/account_test.go to the
in-process integration tier. Each of these resources resolves ownership
from a plain account-scoped KV key (utils.AccountKey(accountID, id), or
an account-keyed settings record) with no vm.Manager or real guest
involved, so DaemonLite now also wires the real EgressOnlyIGWServiceImpl
and AccountSettingsServiceImpl alongside the existing VPC/IGW/Key
services. Two independently minted tenant accounts (via
IAMService.CreateAccount/CreateUser/CreateAccessKey/CreatePolicy/
AttachUserPolicy, in-process — the same calls runAccountCreate makes
without the CLI/stdout round-trip) replace the live tier's carousel so
isolation is checked between real, separately owned accounts.
Left live: instance scoping (Step2) and volume scoping (Step3) —
checkInstanceOwnership, the actual cross-account gate for stop/start/
reboot/terminate/attach-volume, runs daemon-side only after a real
vm.Manager lookup, so it cannot be exercised without a live guest.
Snapshot scoping (Step5) needs a real backing volume for CreateSnapshot
(viperblock, out of DaemonLite's scope), so it stays live entirely rather
than split off only its two KV-only assertions. Global-resources (Step9)
carries no account branching in the handlers at all, so asserting its
equality across two accounts wouldn't exercise anything a fake node
responder didn't already trivially satisfy — not migrated.
RunInstances --launch-template resolves ImageId/InstanceType from the
stored template entirely gateway-side (expandLaunchTemplate, before
validation/placement/the daemon-facing NATS hop), so proving it forwards
the resolved fields never needed a real guest boot. Ports the assertion
to tests/integration via a new opt-in StartLaunchTemplateDaemonLite (real
LaunchTemplateServiceImpl over embedded JetStream) plus a direct
subscription on the per-node launch subject to capture what the gateway
actually dispatched, rather than only the client-visible response.
The live test keeps its CRUD/versioning lifecycle (create/modify/delete
version, tag-filtered describe) — none of it touches a guest — and drops
only the RunInstances-from-template + wait-for-running + terminate tail,
whose only job was waiting out a real boot to read back fields already
proven resolved.
RunInstances with MinCount/MaxCount=2 expands into a single per-node
targeted launch (distributeInstances in gateway/ec2/instance/placement.go)
before the daemon-facing NATS hop, so the real gateway router plus a
stubbed daemon response is sufficient to prove it — no throwaway VM is
needed to exercise the expansion logic itself.
The live single-node E2E test's WaitForInstanceState/TerminateInstances
sequence was teardown hygiene for the real guests it booted, not an
assertion of the expansion behavior, so it is dropped rather than
emulated. The migrated test additionally asserts the per-node launch
request itself carries MinCount=MaxCount=2, proving the count actually
propagates rather than the launch succeeding by coincidence or via the
zero-daemon degenerate path in placement.go.
test-cover runs go test ./spinifex/..., so anything under spinifex/ is
compiled and measured. The daemon fixture cannot be exercised by unit
tests, so it reported a coverage gap no unit test could close. The e2e
harness already avoids this by living under tests/; the fixture now
does the same. vbscan stays put, being ordinary library code with its
own unit tests.
DescribeVolumes only proves the handler's own read path agrees with its
own write path, both via the same deserialization; it cannot catch a bug
in a field it never re-derives (e.g. a unit-conversion error in the bytes
handed to viperblock, since VolumeMetadata.SizeGiB is copied from the
request directly). vbscan reads a volume's persisted config.json and
chunk objects straight over S3, independent of any spinifex handler code,
so TestCreateVolume_PersistsToRealPredastore can assert on what viperblock
actually wrote rather than only on what the handler claims.
vbscan is a local reimplementation of scripts/vbrefscan's read-only scan
rather than an import: spinifex cannot depend on mulga, the module that
contains it. It only reimplements the config/chunk-listing half of
vbrefscan — the checkpoint-based reference-integrity half needs
viperblock.ParseBlockCheckpointBytes, which does not exist in the
viperblock version spinifex currently depends on (v1.12.1) and is also
moot for this volume: CreateVolume never calls SaveLiveCheckpoint, so a
freshly created volume has no checkpoint to scan regardless of version.
Add StartVolumeDaemonLite, an opt-in DaemonLite variant wired to the
real predastore fixture, and migrate TestCreateVolume_PersistsToRealPredastore
off memory-backed DaemonLite so CreateVolume/DescribeVolumes exercise
viperblock's real S3 backend end to end instead of stopping at input
validation.
AssertSpreadAcrossNodes and CountInstancesPerNode lost their only caller when
the instance-distribution test was removed. Beyond being dead, the assertion
helper documented minNodes=1 as an acceptable logging-only mode inherited from
the bash driver, which is the pattern that let a placement test pass while
every instance landed on one node.
InstanceHostingNode and its ps helpers stay; five multinode tests use them.
SpreadPlacement passed at unique>=3, warned via t.Logf at unique==2,
and only failed below that. A 33% spread failure (two instances
colocated) passed with a log line. The 243.6s NAT-egress scenario that
follows only proves what it claims to if the private trio actually
landed on three distinct chassis, so that premise needs to be a hard
assertion, not a warning.
TestFinalClusterStats and TestInstanceLaunch assert nothing beyond
what other callers of their shared helpers already imply. The
multinode baseline, volume, guest-SSH, and preflight tests duplicate
single-node coverage or exercise per-guest behaviour that isn't
cluster-specific. TestMultinodeInstanceDistribution can't fail: three
independent RunInstances calls with no placement group make a strict
spread assertion flaky and a loose one meaningless.
Preflight checks (/dev/kvm writability, peer SSH reachability) move
into requireMultiNodeFixture so the package fixture singleton still
fails fast with a clear message instead of disappearing along with
the dedicated test. The remaining callers of needInstanceTrio
(CrossNodeOps, CrossNodeGateway, NodeFailure, VolumeDurability) still
trigger the shared trio launch on their own, so removing the
distribution and volume-lifecycle tests doesn't leave it unbuilt.
Also slims the manifest-lint baseline entry for the deleted
multinode volume test.
Ports TestECRIAMAuthorization_PullOnly, _ReadOnly,
_DetachPolicyDeniesImmediately, and _DeactivatedKeyReturns401 from
tests/e2e/ecr now that AttachUserPolicy accepts AWS-managed policy ARNs.
Each was proven equivalent by mutation before its E2E original was
removed: PullOnly/ReadOnly/DetachPolicyDeniesImmediately fail when
evaluatePrincipalPolicy is forced to always allow, and
DeactivatedKeyReturns401 fails when the access-key-active check in
resolveECRLongLivedPrincipal is bypassed. All mutants were reverted and
a clean run confirmed before committing.
tests/e2e/ecr/iam_authorization_test.go is now fully removed — every
test it held has an in-process equivalent, and its helpers had no other
callers in the package.
Wires the gateway's ECR control plane (repository/policy/lifecycle
metadata over an in-process daemon-lite NATS subscriber, blob and
manifest bytes via a memory object store, JWT issuer/verifier keyed off
the existing IAM master key) into the integration harness, and ports
TestECRControlPlane, TestECRGetLoginPassword, and
TestECRIAMAuthorization_AssumedRolePullOnly from tests/e2e/ecr — each
proven equivalent by mutation (evaluatePrincipalPolicy forced to allow,
and the repo-describe path forced to not-found) before its E2E original
was removed.
The other 4 authorization tests (PullOnly, ReadOnly,
DetachPolicyDeniesImmediately, DeactivatedKeyReturns401) stay on the E2E
tier: AttachUserPolicy lacks the AWS-managed-ARN existence-check bypass
that AttachRolePolicy/AttachGroupPolicy already have, so attaching
AmazonEC2ContainerRegistryPullOnly/ReadOnly straight to a user fails
NoSuchEntity in both tiers alike. Left untouched pending a fix to that
shared IAM code.
TestKeyPairs, TestTagManagement, TestRouteTableValidation, and
TestReplaceRouteConvergence now have proven-equivalent ports in
tests/integration/ (mutation testing confirmed parity). Delete the
e2e originals and their runX helpers, along with keypair_test.go's
and tags_test.go's file-local helpers left with no caller.
generateImportPubKey is shared by other single/ tests so it moves to
helpers_test.go instead of being deleted. Slims the manifest-lint
baseline for the removed create-call entries.
Adds DaemonLite, an in-process stand-in for a live spinifex daemon that
subscribes the real key, tags, route table, and VPC/subnet/SG/IGW service
implementations directly to their NATS subjects, backed by memory and
embedded-JetStream stores. Ports TestKeyPairs, TestTagManagement,
TestRouteTableValidation, and TestReplaceRouteConvergence from
tests/e2e/single so the integration tier drives genuine daemon-side
business logic instead of StubSubject canned replies.
vpcd (OVN) is out of scope, so its "vpc.create-sg"/"vpc.delete-sg"
acknowledgements are stubbed with a fixed success reply — the SG record
itself is still written by the real service impl before that event fires,
so the stub only unblocks the synchronous round-trip rather than
replacing tested behavior.
All 13 signed-request authz tests plus TestIAMCleanup now have faithful,
mutation-proven equivalents in tests/integration/ — the gateway dispatches
IAM/STS actions directly with no daemon hop, so the in-process tier is a
faithful and far cheaper substitute. The iam/ e2e suite now runs only
TestIAMInstanceProfileAssociation and TestIMDS.
Drop the now fully-dead iam_helpers_test.go alongside the six deleted test
files, and remove harness.IAMGroupARN/IAMDeleteGroupBestEffort, which had no
remaining caller anywhere in the repo once iam_groups_test.go was gone.
Ports TestIAMUserCRUD, TestIAMAccessKeyLifecycle, TestIAMUserAuthentication,
TestIAMPolicyCRUD, TestIAMPolicyAttachmentEnforcement, TestIAMPolicyLifecycle,
TestIAMGroupsLifecycle, TestIAMRolesAndProfiles and
TestSTSAssumeRoleAndGetCallerIdentity from tests/e2e/iam/ into the in-process
integration tier. Each test now self-bootstraps its own users/policies/roles
instead of relying on shared suite-level state, since every integration test
gets its own gateway and embedded NATS. TestIAMRolesAndProfiles stubs
ec2.IamProfileAssociation.describe because DeleteInstanceProfile always
counts live associations over NATS regardless of which guard actually fires.
TestIAMCleanup is not ported: it only swept a shared long-lived E2E
environment, and every ordering/dependency assertion it made is already
covered by TestIAMPolicyLifecycle, TestIAMGroupsLifecycle and
TestIAMRolesAndProfiles's own delete-conflict guards.
Adds IAMClient/STSClient plus ClientsWithCreds/ClientsWithSessionCreds so a
test can act as a principal other than the seeded root user, and stubs the
node-discovery subject in StartGateway so call paths that trigger discovery
resolve immediately instead of riding out the fan-out timeout.
Ports the four enforcement tests that guard on ec2:DescribeRegions, a
gateway-local action, so they need no daemon-side stub at all.
Roughly half of the behavioural tests under tests/e2e/ assert control-plane
API shapes, authz and error codes that need no live environment, yet they can
only run against a tofu-provisioned environment on a scarce self-hosted
runner. This adds the tier they can move down to.
The real gateway router runs in-process: SetupRoutes() is a pure function of
GatewayConfig, so it serves from an httptest server over embedded NATS
JetStream with real IAM/STS services and genuine SigV4 verification. Only the
daemon side of the wire is faked -- the ec2.*/ebs.* NATS subjects a live node
would answer are stubbed per test.
Nothing is provisioned and no fixed ports are bound, so the tier runs
concurrently on a shared build host and is safe as a CI front gate ahead of
the live suites. It is build-tagged and kept out of preflight, which stays
unit-only.
Treat the ECR registry JWT as a signed identity pointer rather than a
permission grant. Every /v2/* request now rehydrates the token's access
key against current IAM/STS state, maps the OCI route to an ecr: action
and repository ARN, and runs the identity-policy evaluator before
dispatch. PullOnly/ReadOnly principals can no longer push, overwrite
tags, or delete manifests, and key/session/user/role/account/policy
revocation takes effect on the next request instead of at token expiry.
Cross-repository blob mounts now require source-read permission and
proof the digest is referenced by a manifest in the named source repo,
closing an account-wide blob-fishing gap.
The image-pull storage-growth workload HEADed <volID>/checkpoints/blocks.live.bin
with the operator box's gateway identity, which has no S3 read grant on the
bucket and was refused 403 on every sample, so checkpoint_bytes read 0 across a
whole run. It now HEADs the object on the node over PeerSSH using predastore's
own [[auth]] credentials from predastore.toml (root-owned, so read via
passwordless sudo; the CA is readable through the spinifex group). The
credentials are read into shell locals, used inline for one aws s3api
head-object, then unset — never echoed, never leaving the node. A probe failure
that is not a legitimate 404 voids the sample (checkpoint_bytes -1 plus a probe
error) rather than reporting a false zero, so a dead instrument can never look
like no growth. The dead operator-side S3 client path is removed.
Pulls a container image inside a guest onto a viperblock volume and samples
guest used bytes, host backing-store bytes, nbdkit RSS and the live
checkpoint size as it goes, so backing-store cost can be fitted against the
bytes the guest actually wrote.
Complements the N-pass overwrite workload, which rewrites one extent and so
pins the block map size — that shape cannot show whether cost grows in
proportion to bytes written or faster than them.
Redirects both dockerd's data-root and containerd's root onto the target
volume. Redirecting data-root alone leaves containerd extracting layers into
/var/lib/containerd on the AMI's small root disk, which fails with ENOSPC
while the target volume sits empty.
The storagegrowth gate called ResolveCACert unconditionally before
building the AWS client, even though newAWSClient itself skips CA
resolution entirely when SPINIFEX_AWS_INSECURE=1. That blocked the
documented runner-resident remote path (env12 and similar clusters
with no local CA cert on disk) and the skip message falsely claimed
no node was running when the real cause was a missing CA. Skip the
CA check under SPINIFEX_AWS_INSECURE=1 and state the actual cause
plus the escape hatch in the skip message.
Rewrites one logical extent N times through full attach/format-write/detach
cycles, so the guest-visible footprint stays constant while total bytes written
scales with N. That shape isolates the leak where viperblock mints chunk objects
under unique monotonic keys and never deletes the superseded ones: a sequential
write-once supersedes nothing and so cannot exercise it.
The suite drives the workload only and deliberately does not measure the
backend. Measuring on-disk bytes from here would mean either a host du (which
cannot separate live from dead bytes, and so cannot distinguish this leak from
the unrelated one where an overwritten key's old extent is orphaned without a
tombstone) or a dependency on tooling outside this repository, which would stop
the suite building standalone. It instead reports what it did to a caller-named
result file and asserts only what a guest can observe: a sha256 round-trip
through a fresh mount per pass, and a footprint that stays constant across them.
The volume is retained on request, because deleting it removes the evidence:
DeleteVolume enumerates the bucket by prefix and drops every key under it,
including the leaked chunks.
Phase 8 of TestNATUplink checked the second VPC router's lr-nat-list with a
single non-polling assert; the igw-default-snat row commits a beat after the
gateway LRP appears, so the read raced the OVN commit and intermittently saw
an empty list (flaky CI red). Wrap it in EventuallyErr, matching the sibling
LRP and host-route polls in the same phase.
AddSNAT logged only on the idempotent-skip path, so the first install was
silent and RCA could not tell a late SNAT from a missing one. Emit an install
log keyed by router + vpc_cidr + external_ip.
Add an IRSAPod subtest that applies a real pod wired for IRSA — a projected
ServiceAccount token (aud sts.amazonaws.com) plus the AWS_ROLE_ARN /
AWS_WEB_IDENTITY_TOKEN_FILE / AWS_ENDPOINT_URL_STS / AWS_CA_BUNDLE env the
addon-sync agent injects — and asserts the pod's in-cluster
aws sts get-caller-identity resolves to the assumed IRSA role. This closes the
gap left by IRSAWebIdentity, which only exchanges the token host-side.
Extract registerOIDCRole (shared with IRSAWebIdentity); the pod runs on a
1-node worker nodegroup (the control-plane node is CriticalAddonsOnly-tainted)
and reaches awsgw STS over the worker subnet NAT.
WaitForInstanceState now aborts immediately with the recorded StateReason
message when it observes a reaped state (shutting-down/terminated) while
waiting for a live target, instead of burning the full 5-minute timeout on
an instance that can never reach it.
MarkFailed takes a context and records the failure on the active span via
span.RecordError so a reaped launch surfaces in the APM error stream,
correlated by trace id. Only the LaunchRunInstances path carries a live
span; other callers pass context.Background() and the record is a no-op.
Surfaced by RCA of E2E run 29258398516 (TestAccountScoping/Step2).
discoverSingleNodeIPs seeds 127.0.0.1 then appends net.InterfaceAddrs()
in undefined order, so WANHost=nodeIPs[0] was loopback and the guest
gateway URL helpers fell through to the first non-loopback interface
addr — an arbitrary co-tenant bridge on multi-bridge shared hosts
(wattle: br-lan-env1..23, br-wan), leaving guest VMs unable to reach the
advertised gateway to register.
Resolve WANHost from spinifex.toml nodes.<node>.advertise (falling back
to that node's host), mirroring awsgwBindIP's toml scan; fall back to the
first non-loopback discovered addr only when config is unreadable. This
removes the need for the SPINIFEX_WAN_IP override on shared hosts.
Add non-streaming InvokeModel to bedrock-runtime. Unlike Converse's unified
schema, InvokeModel passes model-provider-native bodies through, so route by
model family: Anthropic Claude bodies forward to the Messages API (drop the
Bedrock-only anthropic_version, inject model, return verbatim); Meta Llama
bodies translate to/from the self-host vLLM OpenAI completions endpoint.
Raw-body dispatch bypasses the JSON response path (WriteRawResponse). Reuses
the Phase 1 credential + endpoint resolvers and upstream-status mapping.
Unit-tested (routing, body rewrite, translation, error classes) plus gated
InvokeModel E2E.
* feat(bedrock): Ochre runtime MVP — Converse over self-host + Anthropic-direct
Add the AWS-Bedrock-compatible gateway services (bedrock, bedrock-runtime)
following the EKS REST-JSON pattern: ListFoundationModels / GetFoundationModel
over a static connectivity-tiered catalog, and non-streaming Converse routed
per modelId to one of two provider backends — self-hosted vLLM (OpenAI wire)
and Anthropic-direct (Anthropic Messages wire).
Per-account provider API keys are a master-key-encrypted JetStream KV secret
with an optional platform default; the advertised catalog gates external
models on a resolvable credential. Self-host endpoints are pinned via config
(OCHRE_VLLM_ENDPOINTS) behind an EndpointResolver seam.
Adds gateway/bedrock/ (catalog, provider router, Anthropic + vLLM adapters,
credential store), awserrors REST-JSON codes, and the bedrock E2E suite
(list/get light; Converse gated on live backends). Daemon/NATS control plane
and dynamic endpoint lifecycle are deferred to Phase 3.
* test(bedrock): cover dispatch, router, credentials KV + endpoint config
Add unit coverage for the Ochre Phase 1 diff: Bedrock_Request /
BedrockRuntime_Request end-to-end dispatch, provider router, JSON
response/error writers, upstream-status mapping, the credential-store KV
round-trip, the Anthropic bound-provider path, and parseBedrockEndpoints.
Lifts PR diff coverage above the 60% gate.
* ci: GPU e2e workflow on ci-gpu self-hosted runner
Runs tests/e2e/gpu (EC2/ECS/EKS GPU passthrough) against the persistent,
hand-bootstrapped wattle shared host, driven over SSH by a banksia runner
labelled ci-gpu (matches the ci-single/ci-multi convention) — no runner
installed on the shared box.
- runs-on [self-hosted, ci-gpu]; concurrency ci-gpu, cancel-in-progress
false (single A1000 = single-flight, queue not stomp)
- workflow_dispatch + GPU-path filter (not every push)
- manual-work mutex gate (~/.ci-gpu-paused / CI_GPU_PAUSED) so hands-on GPU
testing blocks CI
- if: always() cleanup terminates launched GPU instances; never resets the
box (no setup-ovn.sh/dev-reset/cluster-reset)
- drives the control plane with spx + the already-configured aws spinifex
profile directly (no wrapper scripts, nothing to install on wattle)
* ci(gpu-e2e): advisory AMI preflight, fail only if zero GPU AMIs
The suite is self-guarding — each test skips when its AMI is absent. The
hard-fail on the base ubuntu GPU AMI blocked the whole run even when the
ECS GPU AMI (the only one imported) could execute. Inventory the AMIs and
fail only when none are present.
* ci(gpu-e2e): self-provision base GPU AMI, hard-require it in preflight
The GPU E2E workflow was green-because-skipped: with no GPU AMIs imported
the self-guarding suite skipped every test, passing as a false green. Make
the raw-EC2 GPU passthrough test the always-on baseline.
- Add an import-if-absent setup step that pulls the prebuilt base GPU image
from the catalog by name (ubuntu-26.04-nvidia-gpu-x86_64), naming the AMI
exactly what tests/e2e/gpu filters for. Idempotent; caches on wattle's
persistent loop-fs. Runs spx under sg spinifex (local storage write).
- Preflight now hard-requires the base AMI (fail, not skip) so a silent
import failure can't pass as green; ECS/EKS node AMIs stay opt-in warn.
- Reconcile the local build path: ubuntu-gpu-nvidia.conf AMI_NAME and
launch-gpu.sh's nvidia default now use the same canonical name, so a
locally built+imported image also satisfies the suite.
* ci(gpu-e2e): pass --ami-name on base GPU import so the AMI name matches
spx admin images import --name selects the catalog entry but names the
resulting AMI generically (ami-<distro>-<version>-<arch>), so the import
landed as ami-ubuntu-26.04-x86_64 and the tests/e2e/gpu name filter never
matched. Pass --ami-name ubuntu-26.04-nvidia-gpu-x86_64 explicitly (same
flag build-system-image.sh uses) and poll briefly for availability.
* fix(network,vm): reconcile auto-assigned public-IP NAT + scale VFIO QMP timeout
Auto-assigned public IPs were never carried through the network reconciler.
applyEIPs ran AddEIP + ensureGuestPortDatapath for user EIPs only; auto-assigned
IPs (intent.Ports[].PublicIP) got egress exemption but no NAT re-assert and no
gatewayLRP->guest convergence. Their dnat_and_snat was created once at
RunInstances and never touched, so a vpcd/OVN rebuild dropped it and the
guest-port hop never converged: inbound DNATs at the gateway but never reaches
the guest (TCP/SSH dark) while ICMP is answered by the gateway LR (ping OK).
floatingIPSpecs now folds every auto-assigned/ELB public IP into the same
AddEIP + guest-port convergence path as user EIPs (deduped by private IP; a user
EIP wins), making the datapath self-healing across reconcile ticks and deploys.
qmpGreetingTimeout for VFIO guests now scales with guest RAM (base 30s + 8s/GiB,
floored at 180s, capped at 600s) instead of a flat 180s, so a large guest is not
SIGKILLed mid-pin while the whole guest RAM is IOMMU-locked before QMP answers.
* fix(vm): scale QMP handshake command deadline for VFIO launches
The greeting timeout scaling alone did not fix the VFIO launch SIGKILL: a GPU
guest sends its QMP greeting before the RAM pin completes, then cannot service
qmp_capabilities until the pin finishes. sendQMPCommand hardcoded a 30s response
deadline, so the handshake decode timed out (~43s for a 16 GiB g5.xlarge) and
the launch cleanup SIGKILLed a QEMU that would have come up.
Parameterize the command deadline (sendQMPCommandWithTimeout); the handshake now
uses the RAM-scaled qmpGreetingTimeout (base 30s + 8s/GiB, floor 180s, cap 600s)
for qmp_capabilities. Other callers keep the 30s default.
* test(gpu-e2e): share one GPU instance across passthrough subtests to avoid mid-suite IP recycle
* feat(eks): expose NVIDIA GPUs to pods via CDI device plugin
GPU nodegroups now label workers nvidia.com/gpu.present=true and taint them
nvidia.com/gpu=present:NoSchedule (via a k3s config.yaml.d drop-in, since k3s
has no K3S_NODE_TAINT env), and stage a hidden nvidia-device-plugin addon that
addon-sync renders into the K3s auto-deploy dir. The device plugin runs with
--device-list-strategy=cdi-cri and is gated to GPU nodes by nodeSelector +
toleration, advertising nvidia.com/gpu allocatable so pods land on GPU nodes
and get the device injected via the CDI spec baked by mulga-cdi-generate.
Relies on containerd 2.0.5 default enable_cdi=true (no config fork).
* fix(vm): scale QMP greeting deadline for VFIO passthrough guests
A GPU passthrough guest (-device vfio-pci + large -m) must lock and DMA-map
its entire guest RAM through the IOMMU before QEMU's QMP monitor answers,
which exceeds the fixed 30s greeting deadline and gets the VM SIGKILLed at
launch. Parameterize the greeting timeout (NewQMPClientWithGreetingTimeout)
and scale it to 180s for VMs with GPU attachments; plain VMs keep the 30s
default and reconnects against an already-running QEMU keep it too.
* fix(eks): register GPU workers reliably and expose GPUs to pods
Three fixes let a GPU nodegroup worker boot, register, and advertise its GPU:
- k3s-agent.service gains Delegate=yes (plus KillMode/TasksMax/limits and
overlay/br_netfilter modprobe): k3s embeds containerd+kubelet in-process, so
on cgroup-v2 the kubelet cannot create /kubepods without a delegated subtree
and ContainerManager.Start fails before Node registration.
- The GPU node label rides a k3s config drop-in (20-gpu.yaml node-label +
node-taint) instead of K3S_NODE_LABEL, which k3s ignores; without the label
the nvidia-device-plugin nodeSelector never matches.
- Raise the nodegroup ready timeout to 20m for the ~7min GPU first boot.
Adds the GPU e2e (VPC + NAT-GW egress harness) and a slog-restore unit test.
* fix(eks): install ecr-credential-provider on the GPU node AMI
The GPU worker's kubelet is started (from nodegroup userdata) with
--image-credential-provider-config + --image-credential-provider-bin-dir=
/usr/local/bin, but the eks-node-gpu AMI never built or installed the
ecr-credential-provider binary. kubelet treats a referenced-but-missing
provider binary as a fatal RegisterCredentialProviderPlugins error, so the
node crash-loops and never registers (while the Alpine eks-node worker, which
ships the binary, joins fine). Build + install it in the GPU manifest; the
server-only binaries stay omitted since GPU nodes are agent-only.
Also tee k3s-agent stderr (and its embedded kubelet klog) to the serial
console: Ubuntu logs only to journald, which is off-box-invisible on a worker
in an isolated customer VPC, so a kubelet that fails to register left no trace.
* fix(eks): give the nvidia-device-plugin its own GPU access
The CDI-mode device plugin (--device-list-strategy=cdi-cri) must run
nvml.Init and generate the CDI spec in its own container, but the DS had
no GPU wiring, so nvml.Init crash-looped and nvidia.com/gpu was never
advertised. Run the DS under the k3s-auto-configured nvidia runtimeClass
with NVIDIA_VISIBLE_DEVICES/CAPABILITIES=all, the host root mounted
read-only at /driver-root so it can locate the driver libs + ld.so.cache,
and /var/run/cdi mounted writable so the generated spec is visible to
containerd. No enable_cdi drop-in is needed on k3s v1.32.5 / containerd
2.0.5. GPU workload pods still need only resources.limits nvidia.com/gpu.
Live-validated end-to-end on the single-GPU host (RTX A1000): baked
control-plane AMI, GPU nodegroup, nvidia.com/gpu allocatable, nvidia-smi
pod, zero manual intervention.
* test(ec2): drop obsolete restoreSlogDefault test
Rebased onto dev, which dropped the restoreSlogDefault workaround in #506
(viperblock no longer mutates the global slog). The test referenced the
now-deleted helper and no longer compiles. Real GPU/registration diff is
independently covered (diff coverage 95.3%).
Three related nat-mode fixes:
- ura01: nat-transit pool now emits gw_lrp_range_start/end. admin init
sets 100.127.0.16-.254 (239 VPC gateways) instead of the auto-derived
top-16 (~15). Adds GwLrpRange* to PoolData + the spinifex.toml template
so operators can widen any pool.
- 724mm: detectPoolMode returns true only for external_mode=pool. nat and
disabled clusters now skip Phase 8b/8d, which otherwise failed on
AllocateAddress UnsupportedOperation.
- pnz2i: document that system instances (ECS/EKS/LB agents) are
unsupported in nat v1 (require pool mode) in the vpc-networking README
and COMMANDS.md; document the gw_lrp_range_* fields.
An ELBv2 LB whose system VM launches but whose in-guest lb-agent never
delivers a first heartbeat previously stayed in provisioning forever with
an empty StateReason: promotion to active happens only on the first
heartbeat and the only provisioning->failed transitions were at
create/launch time. A single wedged LB in the multi-node e2e lb suite
leaked its listener/target-group and cascaded into ResourceInUse and
DependencyViolation on VPC teardown.
- Add a lifecycle reaper that marks failed any LB still provisioning past
lbProvisioningTimeout (4m) with no heartbeat, reusing markLBFailed, and
populate a provisioning StateReason so a stuck LB is self-describing.
Started once at daemon startup; stopped on service Close.
- Harness: add ErrLBProvisioningTimeout and treat a provisioning-timeout
as retryable in createActiveLB; register idempotent LB/listener teardown
up-front so no exit path leaks into the shared VPC.
- IMDS: bounded retry+backoff on the instance-fact and user-data lookups so
a transient gather timeout under multi-node load does not starve the
lb-agent bootstrap.
The EIP "verify unreachable after disassociate" step polls runSSHQuiet
inside EventuallyErr(90s), but a wedged ssh made os/exec Wait block for
14 minutes — EventuallyErr cannot interrupt an in-flight CombinedOutput,
so one probe consumed the whole 30-min test-binary budget and panicked.
The panic skipped t.Cleanup, leaking the EIP and NAT-gateway VMs, which
held single-node capacity and starved the lb suite's sys.micro launch
(the daemon unsubscribes the full topic, so NATS returns no-responders).
Bound the ssh child at the process layer with exec.CommandContext (30s
deadline) plus WaitDelay, returning a descriptive timeout error. This is
version- and phase-independent, unlike ssh's own ConnectTimeout. 30s is
safe for all callers (worst legit call is ~15s) and well under the 90s
poll budget, so a wedged probe now yields a clean bounded failure with
cleanup intact.
joshsiv-mulga