Proves the default (non-placement-group) path packs a multi-instance launch across nodes proportionally to spare capacity rather than exhausting one node first.
Adds a placement-group daemon-lite wiring a real PlacementGroupServiceImpl over NATS, then proves spread strategy assigns exactly one instance per node while cluster strategy pins the whole launch to the single highest-capacity node.
Proves a caller granted ec2:RunInstances but not iam:PassRole on an instance profile's role is denied, and the identical session succeeds once PassRole is granted for that exact role ARN.
Proves a malformed ImageId and a malformed targeted-launch CapacityReservationId are rejected before any NATS dispatch, with no daemon stub registered so a regression that let either through would fail fast instead of hanging.
Proves version selection (omitted Version resolves to the template default, explicit Version selects that version) and field precedence (explicit RunInstances parameters override the template, unset fields inherit it) through the real gateway.
Proves replaying the same ClientToken through the real gateway returns the original reservation without a second daemon dispatch, and a different token launches independently.
ListFoundationModels is pure gateway-side static catalog logic with no
NATS/daemon hop, so it moves to the integration tier via a new BedrockClient
harness accessor; this tier configures no credential resolver, so the
Anthropic entry's absence is asserted explicitly rather than left optional.
runPredastoreObjectLifecycle only needs a real predastore daemon, not a
tofu-provisioned cluster: it now runs against tests/fixtures/predastore's
in-process fixture, covering the same create/put/get/delete/list contract.
runImage's AMI describe round-trip needs no real cluster: it registers an
AMI against a hand-seeded snapshot via the real ImageServiceImpl, backed by
a new StartImageDaemonLite helper wiring RegisterImage/DescribeImages onto an
in-memory object store.
Enable/Disable/GetSerialConsoleAccessEnabled is pure gateway+account-settings
logic with no guest involved, so it moves to the in-process integration tier
against the real AccountSettingsServiceImpl wired into DaemonLite.
ClientTokenStore in spinifex/gateway/ec2/instance is a process-wide
sync.Once singleton bound to whichever *nats.Conn first initialises it.
Every integration test booted its own embedded NATS server, so the first
RunInstances-calling test captured the singleton, and any later test
inherited a KV handle bound to that server's now-closed connection.
TestMain now boots one embedded NATS+JetStream server for the whole
package; every StartGateway call connects into its own dynamically
provisioned NATS account (accountAuthenticator in shared_nats.go), giving
each test a private JetStream namespace despite the shared server, and
none of the per-test connections are closed until every test has finished
- whichever one the singleton latches onto stays live for the run.
Also wire the suite into the required PR gate: it was previously only
invoked from e2e.yml's push-triggered (main/dev), self-hosted "Integration
(front gate)" job, so a regression here never blocked a PR. The whole
package runs in single-digit seconds, so it now runs inline in
`make preflight` and as its own step in spinifex.yml's build_and_test job.
RequestSpotInstances is a mock over on-demand RunInstances (no bidding,
interruption, or reclamation), so booting real VMs in the e2e tier proved
nothing spot-specific. Migrate the SIR CRUD lifecycle (request, describe,
cancel, close-on-terminate, insufficient-capacity error) to a NATS-level
integration test that exercises the real gateway orchestration and daemon
service impl without a live guest. Real-VM-boot assertions are already
covered by TestRunInstances_MultiCountExpansion and the e2e baseline tests.
TestLaunchTemplates booted zero guests: its RunInstances coverage had
already moved to TestRunInstances_LaunchTemplateExpansion, leaving only
template CRUD/versioning assertions occupying a full live-cluster slot.
Port the remaining create/describe/version/modify/delete lifecycle,
including the SourceVersion merge and a nested timestamp round-trip, into
TestLaunchTemplates_CRUDLifecycle against the in-process integration
gateway.
Ports the KeyPair/VPC-Subnet/IGW-EIGW/AccountSettings cross-account
isolation assertions from tests/e2e/single/account_test.go to the
in-process integration tier. Each of these resources resolves ownership
from a plain account-scoped KV key (utils.AccountKey(accountID, id), or
an account-keyed settings record) with no vm.Manager or real guest
involved, so DaemonLite now also wires the real EgressOnlyIGWServiceImpl
and AccountSettingsServiceImpl alongside the existing VPC/IGW/Key
services. Two independently minted tenant accounts (via
IAMService.CreateAccount/CreateUser/CreateAccessKey/CreatePolicy/
AttachUserPolicy, in-process — the same calls runAccountCreate makes
without the CLI/stdout round-trip) replace the live tier's carousel so
isolation is checked between real, separately owned accounts.
Left live: instance scoping (Step2) and volume scoping (Step3) —
checkInstanceOwnership, the actual cross-account gate for stop/start/
reboot/terminate/attach-volume, runs daemon-side only after a real
vm.Manager lookup, so it cannot be exercised without a live guest.
Snapshot scoping (Step5) needs a real backing volume for CreateSnapshot
(viperblock, out of DaemonLite's scope), so it stays live entirely rather
than split off only its two KV-only assertions. Global-resources (Step9)
carries no account branching in the handlers at all, so asserting its
equality across two accounts wouldn't exercise anything a fake node
responder didn't already trivially satisfy — not migrated.
RunInstances --launch-template resolves ImageId/InstanceType from the
stored template entirely gateway-side (expandLaunchTemplate, before
validation/placement/the daemon-facing NATS hop), so proving it forwards
the resolved fields never needed a real guest boot. Ports the assertion
to tests/integration via a new opt-in StartLaunchTemplateDaemonLite (real
LaunchTemplateServiceImpl over embedded JetStream) plus a direct
subscription on the per-node launch subject to capture what the gateway
actually dispatched, rather than only the client-visible response.
The live test keeps its CRUD/versioning lifecycle (create/modify/delete
version, tag-filtered describe) — none of it touches a guest — and drops
only the RunInstances-from-template + wait-for-running + terminate tail,
whose only job was waiting out a real boot to read back fields already
proven resolved.
RunInstances with MinCount/MaxCount=2 expands into a single per-node
targeted launch (distributeInstances in gateway/ec2/instance/placement.go)
before the daemon-facing NATS hop, so the real gateway router plus a
stubbed daemon response is sufficient to prove it — no throwaway VM is
needed to exercise the expansion logic itself.
The live single-node E2E test's WaitForInstanceState/TerminateInstances
sequence was teardown hygiene for the real guests it booted, not an
assertion of the expansion behavior, so it is dropped rather than
emulated. The migrated test additionally asserts the per-node launch
request itself carries MinCount=MaxCount=2, proving the count actually
propagates rather than the launch succeeding by coincidence or via the
zero-daemon degenerate path in placement.go.
test-cover runs go test ./spinifex/..., so anything under spinifex/ is
compiled and measured. The daemon fixture cannot be exercised by unit
tests, so it reported a coverage gap no unit test could close. The e2e
harness already avoids this by living under tests/; the fixture now
does the same. vbscan stays put, being ordinary library code with its
own unit tests.
DescribeVolumes only proves the handler's own read path agrees with its
own write path, both via the same deserialization; it cannot catch a bug
in a field it never re-derives (e.g. a unit-conversion error in the bytes
handed to viperblock, since VolumeMetadata.SizeGiB is copied from the
request directly). vbscan reads a volume's persisted config.json and
chunk objects straight over S3, independent of any spinifex handler code,
so TestCreateVolume_PersistsToRealPredastore can assert on what viperblock
actually wrote rather than only on what the handler claims.
vbscan is a local reimplementation of scripts/vbrefscan's read-only scan
rather than an import: spinifex cannot depend on mulga, the module that
contains it. It only reimplements the config/chunk-listing half of
vbrefscan — the checkpoint-based reference-integrity half needs
viperblock.ParseBlockCheckpointBytes, which does not exist in the
viperblock version spinifex currently depends on (v1.12.1) and is also
moot for this volume: CreateVolume never calls SaveLiveCheckpoint, so a
freshly created volume has no checkpoint to scan regardless of version.
Add StartVolumeDaemonLite, an opt-in DaemonLite variant wired to the
real predastore fixture, and migrate TestCreateVolume_PersistsToRealPredastore
off memory-backed DaemonLite so CreateVolume/DescribeVolumes exercise
viperblock's real S3 backend end to end instead of stopping at input
validation.
Ports TestECRIAMAuthorization_PullOnly, _ReadOnly,
_DetachPolicyDeniesImmediately, and _DeactivatedKeyReturns401 from
tests/e2e/ecr now that AttachUserPolicy accepts AWS-managed policy ARNs.
Each was proven equivalent by mutation before its E2E original was
removed: PullOnly/ReadOnly/DetachPolicyDeniesImmediately fail when
evaluatePrincipalPolicy is forced to always allow, and
DeactivatedKeyReturns401 fails when the access-key-active check in
resolveECRLongLivedPrincipal is bypassed. All mutants were reverted and
a clean run confirmed before committing.
tests/e2e/ecr/iam_authorization_test.go is now fully removed — every
test it held has an in-process equivalent, and its helpers had no other
callers in the package.
Wires the gateway's ECR control plane (repository/policy/lifecycle
metadata over an in-process daemon-lite NATS subscriber, blob and
manifest bytes via a memory object store, JWT issuer/verifier keyed off
the existing IAM master key) into the integration harness, and ports
TestECRControlPlane, TestECRGetLoginPassword, and
TestECRIAMAuthorization_AssumedRolePullOnly from tests/e2e/ecr — each
proven equivalent by mutation (evaluatePrincipalPolicy forced to allow,
and the repo-describe path forced to not-found) before its E2E original
was removed.
The other 4 authorization tests (PullOnly, ReadOnly,
DetachPolicyDeniesImmediately, DeactivatedKeyReturns401) stay on the E2E
tier: AttachUserPolicy lacks the AWS-managed-ARN existence-check bypass
that AttachRolePolicy/AttachGroupPolicy already have, so attaching
AmazonEC2ContainerRegistryPullOnly/ReadOnly straight to a user fails
NoSuchEntity in both tiers alike. Left untouched pending a fix to that
shared IAM code.
Adds DaemonLite, an in-process stand-in for a live spinifex daemon that
subscribes the real key, tags, route table, and VPC/subnet/SG/IGW service
implementations directly to their NATS subjects, backed by memory and
embedded-JetStream stores. Ports TestKeyPairs, TestTagManagement,
TestRouteTableValidation, and TestReplaceRouteConvergence from
tests/e2e/single so the integration tier drives genuine daemon-side
business logic instead of StubSubject canned replies.
vpcd (OVN) is out of scope, so its "vpc.create-sg"/"vpc.delete-sg"
acknowledgements are stubbed with a fixed success reply — the SG record
itself is still written by the real service impl before that event fires,
so the stub only unblocks the synchronous round-trip rather than
replacing tested behavior.
Ports TestIAMUserCRUD, TestIAMAccessKeyLifecycle, TestIAMUserAuthentication,
TestIAMPolicyCRUD, TestIAMPolicyAttachmentEnforcement, TestIAMPolicyLifecycle,
TestIAMGroupsLifecycle, TestIAMRolesAndProfiles and
TestSTSAssumeRoleAndGetCallerIdentity from tests/e2e/iam/ into the in-process
integration tier. Each test now self-bootstraps its own users/policies/roles
instead of relying on shared suite-level state, since every integration test
gets its own gateway and embedded NATS. TestIAMRolesAndProfiles stubs
ec2.IamProfileAssociation.describe because DeleteInstanceProfile always
counts live associations over NATS regardless of which guard actually fires.
TestIAMCleanup is not ported: it only swept a shared long-lived E2E
environment, and every ordering/dependency assertion it made is already
covered by TestIAMPolicyLifecycle, TestIAMGroupsLifecycle and
TestIAMRolesAndProfiles's own delete-conflict guards.
Adds IAMClient/STSClient plus ClientsWithCreds/ClientsWithSessionCreds so a
test can act as a principal other than the seeded root user, and stubs the
node-discovery subject in StartGateway so call paths that trigger discovery
resolve immediately instead of riding out the fan-out timeout.
Ports the four enforcement tests that guard on ec2:DescribeRegions, a
gateway-local action, so they need no daemon-side stub at all.
Roughly half of the behavioural tests under tests/e2e/ assert control-plane
API shapes, authz and error codes that need no live environment, yet they can
only run against a tofu-provisioned environment on a scarce self-hosted
runner. This adds the tier they can move down to.
The real gateway router runs in-process: SetupRoutes() is a pure function of
GatewayConfig, so it serves from an httptest server over embedded NATS
JetStream with real IAM/STS services and genuine SigV4 verification. Only the
daemon side of the wire is faked -- the ec2.*/ebs.* NATS subjects a live node
would answer are stubbed per test.
Nothing is provisioned and no fixed ports are bound, so the tier runs
concurrently on a shared build host and is safe as a CI front gate ahead of
the live suites. It is build-tagged and kept out of preflight, which stays
unit-only.
joshsiv-mulga