mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 09:01:31 +00:00
48749576c3
* test: add in-process integration test tier Roughly half of the behavioural tests under tests/e2e/ assert control-plane API shapes, authz and error codes that need no live environment, yet they can only run against a tofu-provisioned environment on a scarce self-hosted runner. This adds the tier they can move down to. The real gateway router runs in-process: SetupRoutes() is a pure function of GatewayConfig, so it serves from an httptest server over embedded NATS JetStream with real IAM/STS services and genuine SigV4 verification. Only the daemon side of the wire is faked -- the ec2.*/ebs.* NATS subjects a live node would answer are stubbed per test. Nothing is provisioned and no fixed ports are bound, so the tier runs concurrently on a shared build host and is safe as a CI front gate ahead of the live suites. It is build-tagged and kept out of preflight, which stays unit-only. * test(integration): add IAM/STS client accessors and port the zero-stub authz tests Adds IAMClient/STSClient plus ClientsWithCreds/ClientsWithSessionCreds so a test can act as a principal other than the seeded root user, and stubs the node-discovery subject in StartGateway so call paths that trigger discovery resolve immediately instead of riding out the fan-out timeout. Ports the four enforcement tests that guard on ec2:DescribeRegions, a gateway-local action, so they need no daemon-side stub at all. * test(integration): port remaining IAM/STS authz tests from e2e Ports TestIAMUserCRUD, TestIAMAccessKeyLifecycle, TestIAMUserAuthentication, TestIAMPolicyCRUD, TestIAMPolicyAttachmentEnforcement, TestIAMPolicyLifecycle, TestIAMGroupsLifecycle, TestIAMRolesAndProfiles and TestSTSAssumeRoleAndGetCallerIdentity from tests/e2e/iam/ into the in-process integration tier. Each test now self-bootstraps its own users/policies/roles instead of relying on shared suite-level state, since every integration test gets its own gateway and embedded NATS. TestIAMRolesAndProfiles stubs ec2.IamProfileAssociation.describe because DeleteInstanceProfile always counts live associations over NATS regardless of which guard actually fires. TestIAMCleanup is not ported: it only swept a shared long-lived E2E environment, and every ordering/dependency assertion it made is already covered by TestIAMPolicyLifecycle, TestIAMGroupsLifecycle and TestIAMRolesAndProfiles's own delete-conflict guards. * test(e2e): remove IAM/STS authz tests migrated to the integration tier All 13 signed-request authz tests plus TestIAMCleanup now have faithful, mutation-proven equivalents in tests/integration/ — the gateway dispatches IAM/STS actions directly with no daemon hop, so the in-process tier is a faithful and far cheaper substitute. The iam/ e2e suite now runs only TestIAMInstanceProfileAssociation and TestIMDS. Drop the now fully-dead iam_helpers_test.go alongside the six deleted test files, and remove harness.IAMGroupARN/IAMDeleteGroupBestEffort, which had no remaining caller anywhere in the repo once iam_groups_test.go was gone. * feat(integration): daemon-lite harness for key/tags/routetable/vpc Adds DaemonLite, an in-process stand-in for a live spinifex daemon that subscribes the real key, tags, route table, and VPC/subnet/SG/IGW service implementations directly to their NATS subjects, backed by memory and embedded-JetStream stores. Ports TestKeyPairs, TestTagManagement, TestRouteTableValidation, and TestReplaceRouteConvergence from tests/e2e/single so the integration tier drives genuine daemon-side business logic instead of StubSubject canned replies. vpcd (OVN) is out of scope, so its "vpc.create-sg"/"vpc.delete-sg" acknowledgements are stubbed with a fixed success reply — the SG record itself is still written by the real service impl before that event fires, so the stub only unblocks the synchronous round-trip rather than replacing tested behavior. * ci(e2e): gate live E2E jobs behind the in-process integration tier Runs the in-process integration suite first, on the ci-single runner host rather than inside a provisioned VM, so a broken control plane is caught before either live job pays the Spinifex install. Both e2e_single and e2e_multi now require it, and e2e_status treats a skipped integration job as a failure so the gating status cannot go green on a gate that never ran. Drops the shadow-mode suite selector: targeted per-diff suite selection is not the direction being pursued, and leaving a non-functional shadow job in the gating workflow only invited it to be mistaken for a live gate. * test(e2e): remove single/ tests superseded by daemon-lite ports TestKeyPairs, TestTagManagement, TestRouteTableValidation, and TestReplaceRouteConvergence now have proven-equivalent ports in tests/integration/ (mutation testing confirmed parity). Delete the e2e originals and their runX helpers, along with keypair_test.go's and tags_test.go's file-local helpers left with no caller. generateImportPubKey is shared by other single/ tests so it moves to helpers_test.go instead of being deleted. Slims the manifest-lint baseline for the removed create-call entries. * ci: run the lb e2e suite on multinode only Both e2e jobs filter SUITES by exclusion, and neither excluded lb, so the suite ran twice per e2e run: 354s on single and 654s on multi. Balancing across instances on one node is not a deployment mode we support, and ELBv2 proper requires two AZs, so the single-node run was redundant. Drops ~354s from the single-node job, which is the gate critical path. * test(integration): port 3 of 7 ECR tests to the in-process tier Wires the gateway's ECR control plane (repository/policy/lifecycle metadata over an in-process daemon-lite NATS subscriber, blob and manifest bytes via a memory object store, JWT issuer/verifier keyed off the existing IAM master key) into the integration harness, and ports TestECRControlPlane, TestECRGetLoginPassword, and TestECRIAMAuthorization_AssumedRolePullOnly from tests/e2e/ecr — each proven equivalent by mutation (evaluatePrincipalPolicy forced to allow, and the repo-describe path forced to not-found) before its E2E original was removed. The other 4 authorization tests (PullOnly, ReadOnly, DetachPolicyDeniesImmediately, DeactivatedKeyReturns401) stay on the E2E tier: AttachUserPolicy lacks the AWS-managed-ARN existence-check bypass that AttachRolePolicy/AttachGroupPolicy already have, so attaching AmazonEC2ContainerRegistryPullOnly/ReadOnly straight to a user fails NoSuchEntity in both tiers alike. Left untouched pending a fix to that shared IAM code. * fix(iam): accept AWS-managed policy ARNs on AttachUserPolicy AttachUserPolicy called getPolicyByARN unconditionally, so attaching an AWS-managed ARN to a user returned NoSuchEntity. AttachRolePolicy and AttachGroupPolicy already skip that lookup for AWS-managed ARNs and store them opaquely; the user path was missed when that guard was introduced. ListAttachedUserPolicies had the mirror of the same gap: it resolved every ARN through getPolicyByARN and silently dropped the ones it could not find, so a managed ARN would disappear from the listing rather than round-trip. Evaluation already handled these correctly via resolveAttachedPolicy, so only the write and list paths needed the guard. * test(integration): port remaining 4 ECR authorization tests Ports TestECRIAMAuthorization_PullOnly, _ReadOnly, _DetachPolicyDeniesImmediately, and _DeactivatedKeyReturns401 from tests/e2e/ecr now that AttachUserPolicy accepts AWS-managed policy ARNs. Each was proven equivalent by mutation before its E2E original was removed: PullOnly/ReadOnly/DetachPolicyDeniesImmediately fail when evaluatePrincipalPolicy is forced to always allow, and DeactivatedKeyReturns401 fails when the access-key-active check in resolveECRLongLivedPrincipal is bypassed. All mutants were reverted and a clean run confirmed before committing. tests/e2e/ecr/iam_authorization_test.go is now fully removed — every test it held has an in-process equivalent, and its helpers had no other callers in the package. * ci: select e2e suites by allow-list instead of deny-list Both jobs filtered SUITES by exclusion, so any suite named in neither deny-list ran in both jobs. That is how lb came to run twice per e2e run, and it makes adding a suite silently fan it out everywhere. Each job now names the suites it is eligible to run. Selection is still driven by SUITES, so this is behaviour-preserving for the current default: single/iam/cert on the single-node job, multinode/lb/cert on the multi-node job, exactly as before. eks, ecs and storagegrowth are listed as eligible on the single-node job but are not yet in the default SUITES. * test(e2e): delete seven tests with no coverage TestFinalClusterStats and TestInstanceLaunch assert nothing beyond what other callers of their shared helpers already imply. The multinode baseline, volume, guest-SSH, and preflight tests duplicate single-node coverage or exercise per-guest behaviour that isn't cluster-specific. TestMultinodeInstanceDistribution can't fail: three independent RunInstances calls with no placement group make a strict spread assertion flaky and a loose one meaningless. Preflight checks (/dev/kvm writability, peer SSH reachability) move into requireMultiNodeFixture so the package fixture singleton still fails fast with a clear message instead of disappearing along with the dedicated test. The remaining callers of needInstanceTrio (CrossNodeOps, CrossNodeGateway, NodeFailure, VolumeDurability) still trigger the shared trio launch on their own, so removing the distribution and volume-lifecycle tests doesn't leave it unbuilt. Also slims the manifest-lint baseline entry for the deleted multinode volume test. * fix(e2e): require full 3-node spread in TestMultinodeSpread SpreadPlacement passed at unique>=3, warned via t.Logf at unique==2, and only failed below that. A 33% spread failure (two instances colocated) passed with a log line. The 243.6s NAT-egress scenario that follows only proves what it claims to if the private trio actually landed on three distinct chassis, so that premise needs to be a hard assertion, not a warning. * refactor(e2e): drop the unused spread-assertion harness helpers AssertSpreadAcrossNodes and CountInstancesPerNode lost their only caller when the instance-distribution test was removed. Beyond being dead, the assertion helper documented minNodes=1 as an acceptable logging-only mode inherited from the bash driver, which is the pattern that let a placement test pass while every instance landed on one node. InstanceHostingNode and its ps helpers stay; five multinode tests use them. * fix(ec2): stop leaked goroutines on prepareEFIVolume error paths newViperblock starts the chunk uploader and WAL syncer unconditionally, but prepareEFIVolume only released them on the happy path via Close(). Six error returns between acquisition and Close left both goroutines running. Add a defer immediately after acquisition, matching the sibling prepareRootVolume; Close() remains the durability boundary. * test(predastore): add real predastore fixture for integration tests Extract predastore_integration_test.go's inline daemon startup into a reusable testutil/predastore package so tests/integration can point DaemonLite-style wiring at a real S3-compatible backend instead of an unreachable host. Kept as its own leaf package rather than folding into the shared testutil package: it transitively imports predastore's quicclient, whose package-level connection pool starts a permanent goroutine at init that would trip goleak checks in any of the ~80 other packages that import testutil for unrelated NATS helpers. * test(integration): migrate CreateVolume to assert real predastore persistence Add StartVolumeDaemonLite, an opt-in DaemonLite variant wired to the real predastore fixture, and migrate TestCreateVolume_PersistsToRealPredastore off memory-backed DaemonLite so CreateVolume/DescribeVolumes exercise viperblock's real S3 backend end to end instead of stopping at input validation. * test(integration): add real storage-state oracle for CreateVolume DescribeVolumes only proves the handler's own read path agrees with its own write path, both via the same deserialization; it cannot catch a bug in a field it never re-derives (e.g. a unit-conversion error in the bytes handed to viperblock, since VolumeMetadata.SizeGiB is copied from the request directly). vbscan reads a volume's persisted config.json and chunk objects straight over S3, independent of any spinifex handler code, so TestCreateVolume_PersistsToRealPredastore can assert on what viperblock actually wrote rather than only on what the handler claims. vbscan is a local reimplementation of scripts/vbrefscan's read-only scan rather than an import: spinifex cannot depend on mulga, the module that contains it. It only reimplements the config/chunk-listing half of vbrefscan — the checkpoint-based reference-integrity half needs viperblock.ParseBlockCheckpointBytes, which does not exist in the viperblock version spinifex currently depends on (v1.12.1) and is also moot for this volume: CreateVolume never calls SaveLiveCheckpoint, so a freshly created volume has no checkpoint to scan regardless of version. * refactor(test): move the predastore daemon fixture out of the coverage scope test-cover runs go test ./spinifex/..., so anything under spinifex/ is compiled and measured. The daemon fixture cannot be exercised by unit tests, so it reported a coverage gap no unit test could close. The e2e harness already avoids this by living under tests/; the fixture now does the same. vbscan stays put, being ordinary library code with its own unit tests. * test(integration): migrate RunInstances multi-count expansion off live guests RunInstances with MinCount/MaxCount=2 expands into a single per-node targeted launch (distributeInstances in gateway/ec2/instance/placement.go) before the daemon-facing NATS hop, so the real gateway router plus a stubbed daemon response is sufficient to prove it — no throwaway VM is needed to exercise the expansion logic itself. The live single-node E2E test's WaitForInstanceState/TerminateInstances sequence was teardown hygiene for the real guests it booted, not an assertion of the expansion behavior, so it is dropped rather than emulated. The migrated test additionally asserts the per-node launch request itself carries MinCount=MaxCount=2, proving the count actually propagates rather than the launch succeeding by coincidence or via the zero-daemon degenerate path in placement.go. * docs: drop a doc reference to the deleted live test * test(integration): migrate launch-template RunInstances expansion off live guests RunInstances --launch-template resolves ImageId/InstanceType from the stored template entirely gateway-side (expandLaunchTemplate, before validation/placement/the daemon-facing NATS hop), so proving it forwards the resolved fields never needed a real guest boot. Ports the assertion to tests/integration via a new opt-in StartLaunchTemplateDaemonLite (real LaunchTemplateServiceImpl over embedded JetStream) plus a direct subscription on the per-node launch subject to capture what the gateway actually dispatched, rather than only the client-visible response. The live test keeps its CRUD/versioning lifecycle (create/modify/delete version, tag-filtered describe) — none of it touches a guest — and drops only the RunInstances-from-template + wait-for-running + terminate tail, whose only job was waiting out a real boot to read back fields already proven resolved. * test(integration): migrate KV-only account-scoping checks off live tenants Ports the KeyPair/VPC-Subnet/IGW-EIGW/AccountSettings cross-account isolation assertions from tests/e2e/single/account_test.go to the in-process integration tier. Each of these resources resolves ownership from a plain account-scoped KV key (utils.AccountKey(accountID, id), or an account-keyed settings record) with no vm.Manager or real guest involved, so DaemonLite now also wires the real EgressOnlyIGWServiceImpl and AccountSettingsServiceImpl alongside the existing VPC/IGW/Key services. Two independently minted tenant accounts (via IAMService.CreateAccount/CreateUser/CreateAccessKey/CreatePolicy/ AttachUserPolicy, in-process — the same calls runAccountCreate makes without the CLI/stdout round-trip) replace the live tier's carousel so isolation is checked between real, separately owned accounts. Left live: instance scoping (Step2) and volume scoping (Step3) — checkInstanceOwnership, the actual cross-account gate for stop/start/ reboot/terminate/attach-volume, runs daemon-side only after a real vm.Manager lookup, so it cannot be exercised without a live guest. Snapshot scoping (Step5) needs a real backing volume for CreateSnapshot (viperblock, out of DaemonLite's scope), so it stays live entirely rather than split off only its two KV-only assertions. Global-resources (Step9) carries no account branching in the handlers at all, so asserting its equality across two accounts wouldn't exercise anything a fake node responder didn't already trivially satisfy — not migrated. * test(integration): migrate launch-template CRUD lifecycle off live guests TestLaunchTemplates booted zero guests: its RunInstances coverage had already moved to TestRunInstances_LaunchTemplateExpansion, leaving only template CRUD/versioning assertions occupying a full live-cluster slot. Port the remaining create/describe/version/modify/delete lifecycle, including the SourceVersion merge and a nested timestamp round-trip, into TestLaunchTemplates_CRUDLifecycle against the in-process integration gateway. * refactor(tests): move spot instance lifecycle coverage to integration tier RequestSpotInstances is a mock over on-demand RunInstances (no bidding, interruption, or reclamation), so booting real VMs in the e2e tier proved nothing spot-specific. Migrate the SIR CRUD lifecycle (request, describe, cancel, close-on-terminate, insufficient-capacity error) to a NATS-level integration test that exercises the real gateway orchestration and daemon service impl without a live guest. Real-VM-boot assertions are already covered by TestRunInstances_MultiCountExpansion and the e2e baseline tests. * fix(tests): share one embedded NATS server across the integration package ClientTokenStore in spinifex/gateway/ec2/instance is a process-wide sync.Once singleton bound to whichever *nats.Conn first initialises it. Every integration test booted its own embedded NATS server, so the first RunInstances-calling test captured the singleton, and any later test inherited a KV handle bound to that server's now-closed connection. TestMain now boots one embedded NATS+JetStream server for the whole package; every StartGateway call connects into its own dynamically provisioned NATS account (accountAuthenticator in shared_nats.go), giving each test a private JetStream namespace despite the shared server, and none of the per-test connections are closed until every test has finished - whichever one the singleton latches onto stays live for the run. Also wire the suite into the required PR gate: it was previously only invoked from e2e.yml's push-triggered (main/dev), self-hosted "Integration (front gate)" job, so a regression here never blocked a PR. The whole package runs in single-digit seconds, so it now runs inline in `make preflight` and as its own step in spinifex.yml's build_and_test job. * test(integration): migrate serial-console access checks off live guests Enable/Disable/GetSerialConsoleAccessEnabled is pure gateway+account-settings logic with no guest involved, so it moves to the in-process integration tier against the real AccountSettingsServiceImpl wired into DaemonLite. * test(integration): migrate DescribeImages round-trip off live guests runImage's AMI describe round-trip needs no real cluster: it registers an AMI against a hand-seeded snapshot via the real ImageServiceImpl, backed by a new StartImageDaemonLite helper wiring RegisterImage/DescribeImages onto an in-memory object store. * test(integration): migrate predastore object lifecycle off live guests runPredastoreObjectLifecycle only needs a real predastore daemon, not a tofu-provisioned cluster: it now runs against tests/fixtures/predastore's in-process fixture, covering the same create/put/get/delete/list contract. * test(integration): migrate ListFoundationModels catalog check off live cluster ListFoundationModels is pure gateway-side static catalog logic with no NATS/daemon hop, so it moves to the integration tier via a new BedrockClient harness accessor; this tier configures no credential resolver, so the Anthropic entry's absence is asserted explicitly rather than left optional. * test(integration): cover RunInstances client-token idempotency Proves replaying the same ClientToken through the real gateway returns the original reservation without a second daemon dispatch, and a different token launches independently. * test(integration): cover RunInstances launch-template expansion Proves version selection (omitted Version resolves to the template default, explicit Version selects that version) and field precedence (explicit RunInstances parameters override the template, unset fields inherit it) through the real gateway. * test(integration): cover RunInstances malformed-input rejection Proves a malformed ImageId and a malformed targeted-launch CapacityReservationId are rejected before any NATS dispatch, with no daemon stub registered so a regression that let either through would fail fast instead of hanging. * test(integration): cover RunInstances iam:PassRole enforcement Proves a caller granted ec2:RunInstances but not iam:PassRole on an instance profile's role is denied, and the identical session succeeds once PassRole is granted for that exact role ARN. * test(integration): cover RunInstances placement-group strategy routing Adds a placement-group daemon-lite wiring a real PlacementGroupServiceImpl over NATS, then proves spread strategy assigns exactly one instance per node while cluster strategy pins the whole launch to the single highest-capacity node. * test(integration): cover RunInstances capacity distribution across nodes Proves the default (non-placement-group) path packs a multi-instance launch across nodes proportionally to spare capacity rather than exhausting one node first. * test(e2e): merge SameSGComms and SGToSGDatapath around one client/target pair Both booted their own instance pair to check SG-to-SG datapath enforcement and same-default-SG east-west connectivity. Merging them around one shared client/target pair cuts four boots to two, and converting the inline harness.Step sequence into named t.Run stages (PortGroupMembership, AllowedTraffic, DeniedTraffic, RevokeAndVerifyBlocked, ReauthorizeAndVerifyRestored, SameSGComms) gives each its own pass/fail attribution instead of one opaque test result. SameSGComms now runs last and joins the default SG onto each ENI via ModifyNetworkInterfaceAttribute only after every SG-to-SG allow/deny assertion has already recorded its own result, since the default SG's self-reference rule admits all traffic between co-members and would otherwise defeat the denied-traffic check. * test(e2e): merge default-SG reachability, guest DNS, and egress enforcement DefaultSGReachabilityBaseline, GuestDNSResolution, and SecurityGroupEgress each booted their own instance to check one facet of "a guest is reachable only when policy says so": ingress deny/authorize, resolver + WAN egress, and egress ACL enforcement. Merging them around one shared guest cuts three boots to one, with named t.Run stages (Deny, Authorize, Resolver, WANEgress, RevokeEgress/RestoreEgress) giving each its own attribution. The egress revoke/restore round-trip now mutates this scenario's own dedicated SG instead of the shared default SG the rest of the suite depends on, isolating its egress-rule mutation from every other test. * fix(ci): single-quote e2e test_run regex across the ssh boundary * fix(e2e): anchor SG egress-enforcement probe to a WAN IP, not the gateway * test(e2e): merge single-node guest churn tests into one durability scenario Combines TestSSHProbe, TestENIHotplug, TestENIHotplugReconcile, TestVolumeDurability, TestModifyInstanceAttribute, and TestRebootInstance into TestGuestChurnDurability, a single shared-guest scenario that re-verifies a data-integrity sentinel sha256 after every perturbation stage (ENI hotplug, daemon restart, ENI detach, instance-type resize, stop/start, reboot). This proves the sentinel survives the full churn sequence rather than just one isolated stop/start. * fix(vpc): clear subnet drop gate when installing egress reroute A subnet with an attached IGW but no 0.0.0.0/0 route carries a priority-1100 drop gate on the VPC router. When a route later lands (NATGW reroute at 900, IGW reroute at 1000) the reroute is installed but the drop gate was never removed, so the higher-priority gate wins and black-holes the subnet's egress in lr_in_policy before it reaches routing or SNAT. This left NAT-gateway private-subnet egress with 100% packet loss: the SNAT stage saw zero packets and no conntrack was committed. Remove the subnet's drop gate for the same prefix once a reroute is installed. Idempotent and prefix-scoped, so a specific-prefix reroute is a no-op and leaves the default gate intact. * test(e2e): merge VPC subnet, NAT gateway, and EIP single-node tests Merges the former TestVPCSubnetE2E, TestNewVPCEgressBaseline, TestNATGateway, and TestInstanceEIP into one staged scenario, TestVPCEgressPaths, around one scenario-owned VPC and its two guests, cutting five guest boots to three. The public guest serves PublicSubnetEgress, then the NAT Gateway bastion hop, then (last, since association rewrites its public IP) the EIP flip. The private guest serves the NAT Gateway path, launched directly into the same VPC's already-existing private subnet instead of a dedicated one. RouteBeforeSubnet keeps its own second VPC and instance since it exercises the implicit main-route-table code path, a different one than the explicit route table this scenario's own VPC uses for its public subnet — collapsing the two would drop coverage of whichever path got merged away. The NAT Gateway up/down flip now repeats for an env-tunable round count (SPINIFEX_VPCEGRESS_NAT_ROUNDS, default 1), matching every other merged single-node scenario. * test: TEMPORARY mutation-check — reverta795c2b2drop-gate clear Reverts the RemoveSubnetEgressDrop call in ensureSubnetEgressAtPriority to confirm TestVPCEgressPaths/Round1/NATGatewayUp actually detects the regression it guards against. This commit will be reverted immediately after the live run confirms a failure. * test: TEMPORARY mutation-check v2 — neutralize RemoveSubnetEgressDrop Restores the reala795c2b2fix in ensureSubnetEgressAtPriority (the first mutation attempt only reverted this call site and was masked by the independent gate-recompute path in routetable service_impl.go's CreateRoute, which also clears the drop gate on every 0.0.0.0/0 route add and predatesa795c2b2). Neutralizing RemoveSubnetEgressDrop itself disables gate-clearing regardless of caller, reproducing the literal regression. Will be reverted immediately after live verification. * Revert "test: TEMPORARY mutation-check v2 — neutralize RemoveSubnetEgressDrop" Mutation-check confirmed: neutralizing RemoveSubnetEgressDrop reproduced a795c2b2's regression exactly (stale priority-1100 drop gate outranking the priority-1000 IGW reroute, confirmed via OVN flow dump; PublicSubnetEgress failed with an SSH handshake timeout to the public instance). Restoring the real fix; igw.go is now byte-identical toa795c2b2. * fix(e2e): stop pruning shared multinode SSH key pem mid-run The multinode package fixture cached Fixture.Artifacts once, computed from harness.ArtifactDir(t, env) against whichever top-level Test* happened to win the pkgFixOnce race. That helper both names the directory after that one test and registers a t.Cleanup that removes it once that test passes. needKeyPair wrote the shared EC2 key pair's pem into that same directory, so as soon as the winning test finished, its own artifact-dir cleanup deleted the pem every later test in the package still needed to SSH into fresh guests, even though the underlying EC2 key pair itself was untouched (its DeleteKeyPair cleanup already correctly drains at process Close() in TestMain). Fix: give Fixture an ArtifactDir(t) method — the same per-call pattern already used by the gpu/single/storagegrowth/iam fixtures — so each diagnostic dump site recomputes its own dir against the CURRENT test. The shared key pair pem now lives under Fixture.TmpDir, which is genuinely package-scoped and only removed once in TestMain after m.Run() completes. * refactor(e2e): parallelize LB e2e scenarios on multinode clusters Each of the 8 TestLoadBalancer scenarios already stands up its own dedicated LB + target group (ModifyListener/ListenerRules mutate their LB's listener in place, so it can never be shared). The bottleneck was running them serially: each teardown made createActiveLB back off waiting for the previous LB's sys.micro slot to be reclaimed before the next one could stand up. On multinode clusters (peer != "", i.e. the 3-node cluster this suite targets) there's spare capacity to stand up all 8 dedicated LBs at once, so scenarios now run as t.Parallel() subtests instead of serial create/teardown/backoff. Since each scenario owns an independent LB, one never reaching active only fails that scenario — it no longer cascades into five sequential 5-minute timeouts, so the serial fail-fast gate is skipped in this mode; concurrency itself bounds the worst case to a single timeout window. Single-node dev boxes (no peer) keep the original serial path, including the fail-fast gate, since they lack the spare capacity for concurrent LB standup. No subtest body, assertion, or call signature changed — only the dispatch/gating logic in TestLoadBalancer. * fix(ci): gate e2e jobs on the suites they actually run * fix(e2e): serialize internet-facing LB subtests among themselves Live multinode verification showed InternetFacing_ALB and InternetFacing_NLB running concurrently each clobber the other's public-LB DNS record mid-registration, so neither ever resolves within the 90s wait. The old serial suite never had two internet-facing LBs alive at once, so this never surfaced. Drop t.Parallel() from the three internet-facing subtests so their register+resolve windows never overlap each other, while leaving the internal subtests parallel as before. They may still run alongside the internal group, which resolves internal (not public) DNS and isn't affected. * fix(e2e): give each ALB LB scenario its own security group Concurrent internal subtests all created ALBs with no explicit SecurityGroups, so each ALB's ENI fell back to the one shared VPC default SG. One scenario authorizing/reconfiguring its own listener racing another mutating the same SG made Internal_ALB_ModifyListener intermittently fail to serve on the modified port under parallel load. Each ALB scenario now creates a dedicated SG (createScenarioSG), authorized only for the ports that scenario's listener(s) use, and passes it to CreateLoadBalancer. The shared app-instance/probe-client SG is untouched, since SG ingress is enforced at the destination ENI regardless of the source ENI's SG. NLBs are unaffected: they already get an auto-managed per-LB SG when none is supplied. * test(e2e): annotate intentional create-path calls for manifest-lint The vpcegress and per-scenario LB security-group tests create raw AWS resources on purpose — the egress datapath, the route-before-subnet ordering, and per-scenario SG isolation are the subjects under test, so they cannot route through a shared Ensure* fixture. Mark each with // e2e:allow-create and slim the baseline of entries for the removed single-node network tests. * test(e2e): restore live spot terminate-close and cancel-running coverage The integration-tier spot test calls CloseForInstance directly and infers cancel-keeps-running from the SIR's InstanceId, because DaemonLite has no vm.Manager to drive the real TerminateInstances -> daemon teardown cleaner -> RemoveFromSpotRequest -> CloseForInstance chain or to observe a real VM's running state. Add a thin live e2e test that drives both terminal transitions through a real backing VM: cancel one spot-backed instance and observe it stays running, terminate the other and observe its SIR reach closed through the genuine async delegation. * fix(network): refresh orphan-EIP prune liveness to spare mid-pass launches pruneOrphanEIPs listed OVN NAT rows live but matched them against the intent snapshot captured at the start of the reconcile pass. Because the apply phase can block for tens of seconds on guest-port datapath waits, a guest launched during the pass had a live dnat_and_snat row absent from the stale snapshot, so the prune swept it and blackholed the guest's auto-assigned public IP until the next pass re-added it — an intermittent multinode SSH-dark flake. Add an optional FreshIntent re-read callback (wired in vpcd to LoadIntentFromKV) and union its ports into the live set before pruning, so a mid-pass launch counts as live. Skip the prune entirely when the re-read fails rather than risk a false sweep against a snapshot known to be stale. Self-healing and idempotent: a genuine orphan absent from both snapshot and fresh read is still pruned next drift. Single-node behaviour is unchanged (callback nil in tests).
91 lines
3.7 KiB
Go
91 lines
3.7 KiB
Go
//go:build integration
|
|
|
|
package integration
|
|
|
|
import (
|
|
"encoding/json"
|
|
"fmt"
|
|
"testing"
|
|
|
|
"github.com/aws/aws-sdk-go/aws"
|
|
"github.com/aws/aws-sdk-go/service/ec2"
|
|
"github.com/mulgadc/spinifex/spinifex/types"
|
|
"github.com/nats-io/nats.go"
|
|
"github.com/stretchr/testify/require"
|
|
)
|
|
|
|
// TestRunInstances_MultiCountExpansion asserts that a RunInstances call with
|
|
// MinCount=MaxCount=2 returns 2 instances with non-empty InstanceIds. That
|
|
// expansion is gateway-side logic — distributeInstances in
|
|
// spinifex/gateway/ec2/instance/placement.go queries node capacity over NATS,
|
|
// computes how many instances to launch, and dispatches a single targeted
|
|
// launch per node — resolved entirely before the daemon-facing NATS hop, so a
|
|
// static stub of the daemon side is sufficient; no real guest is needed.
|
|
//
|
|
// The live test's WaitForInstanceState(running)/TerminateInstances/
|
|
// WaitForInstanceState(terminated) sequence is left behind: it is teardown
|
|
// hygiene for a real throwaway VM, not part of what this test asserts.
|
|
//
|
|
// To prove the expansion is genuinely exercised rather than short-circuited by
|
|
// the zero-daemon degenerate path (placement.go's queryNodeCapacity returning
|
|
// no eligible nodes, which also yields InsufficientInstanceCapacity — see
|
|
// placement.go:44), this test subscribes to the per-node launch subject
|
|
// directly instead of using StubSubject's static responder, and asserts the
|
|
// daemon actually received MinCount=MaxCount=2 on that targeted request.
|
|
func TestRunInstances_MultiCountExpansion(t *testing.T) {
|
|
gw := StartGateway(t)
|
|
|
|
const (
|
|
instanceType = "t3.micro"
|
|
nodeID = "node-1"
|
|
)
|
|
|
|
// A single node reporting capacity for 2 of instanceType — enough for
|
|
// distributeInstances to resolve MinCount=MaxCount=2 onto one node.
|
|
statusResp := mustMarshal(t, &types.NodeStatusResponse{
|
|
Node: nodeID,
|
|
InstanceTypes: []types.InstanceTypeCap{
|
|
{Name: instanceType, Available: 2},
|
|
},
|
|
})
|
|
gw.StubSubject(t, "spinifex.node.status", statusResp)
|
|
|
|
var gotMinCount, gotMaxCount int64
|
|
launchSubject := fmt.Sprintf("ec2.RunInstances.%s.%s", instanceType, nodeID)
|
|
sub, err := gw.NATSConn.Subscribe(launchSubject, func(msg *nats.Msg) {
|
|
var nodeInput ec2.RunInstancesInput
|
|
if jsonErr := json.Unmarshal(msg.Data, &nodeInput); jsonErr != nil {
|
|
t.Errorf("unmarshal per-node RunInstances request: %v", jsonErr)
|
|
return
|
|
}
|
|
gotMinCount = aws.Int64Value(nodeInput.MinCount)
|
|
gotMaxCount = aws.Int64Value(nodeInput.MaxCount)
|
|
_ = msg.Respond(mustMarshal(t, &ec2.Reservation{
|
|
ReservationId: aws.String("r-multi-count"),
|
|
Instances: []*ec2.Instance{
|
|
{InstanceId: aws.String("i-multi-count-1")},
|
|
{InstanceId: aws.String("i-multi-count-2")},
|
|
},
|
|
}))
|
|
})
|
|
require.NoError(t, err)
|
|
t.Cleanup(func() { _ = sub.Unsubscribe() })
|
|
|
|
out, err := gw.EC2Client(t).RunInstances(&ec2.RunInstancesInput{
|
|
ImageId: aws.String("ami-0123456789abcdef0"),
|
|
InstanceType: aws.String(instanceType),
|
|
MinCount: aws.Int64(2),
|
|
MaxCount: aws.Int64(2),
|
|
})
|
|
require.NoError(t, err, "run-instances --count 2")
|
|
require.Lenf(t, out.Instances, 2, "expected 2 instances from run-instances, got %d", len(out.Instances))
|
|
require.NotEmpty(t, aws.StringValue(out.Instances[0].InstanceId), "first sibling InstanceId empty")
|
|
require.NotEmpty(t, aws.StringValue(out.Instances[1].InstanceId), "second sibling InstanceId empty")
|
|
|
|
// Proves the gateway genuinely expanded the count onto the per-node
|
|
// launch, rather than the zero-daemon path (which never reaches this
|
|
// subject at all) or a daemon stub that ignores its input.
|
|
require.Equal(t, int64(2), gotMinCount, "gateway must dispatch MinCount=2 on the per-node launch")
|
|
require.Equal(t, int64(2), gotMaxCount, "gateway must dispatch MaxCount=2 on the per-node launch")
|
|
}
|