mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 09:01:31 +00:00
main
16 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Josh Siviour
|
f973b306c4 |
fix: serialise DNS zone writes behind a per-zone lock (#610)
Every daemon read-modify-wrote the same zone object in s3://northstar with no mutual exclusion. A NATS queue group balances load, it does not serialise distinct messages, so concurrent daemons lost each other's records and could leave the object unparseable as TOML. Once unparseable it stayed that way: both the writer and the drift reconciler must parse the zone before they can repair it, so DNS changes stopped applying until the object was edited by hand. The exposure did not need user activity. The drift backstop ran on every node, unelected, on a 60s ticker, so an idle three-node cluster raced its own zone once per tick. Five publisher families feed the same object, so ordinary concurrent API calls collided too. - Add a per-zone JetStream KV lock, following the CAS patterns already used by the vpcd reconcile leader and the vCPU quota store. One key per zone, so unrelated zones never block. The wait budget stays under requestTimeout so a producer's ack is not lost to contention, and a write aborts rather than land outside an expired lease. - Rebuild a zone whose stored bytes do not parse instead of failing every write forever. The reconciler reports a corrupt zone as absent, which suppresses pruning while still publishing the full desired set. Detection matches the decoder's error type so it works against the pinned northstar release. - Elect one node per drift cycle, removing the idle-cluster self-race. - Correct three comments claiming the queue group serialised zone writes; they are why the hazard went unnoticed. - Run the LB e2e internet-facing subtests in parallel. Their serialisation workaround exempted internal LBs on the false premise that those use a different DNS path; ELBChanges sets the same base-domain zone for both. |
||
|
Julian Sommer
|
4803469037 | fix: e2e test assertion | ||
|
Josh Siviour
|
48749576c3 |
test: rebalance E2E into in-process integration tier + shared-guest live scenarios (#556)
* test: add in-process integration test tier Roughly half of the behavioural tests under tests/e2e/ assert control-plane API shapes, authz and error codes that need no live environment, yet they can only run against a tofu-provisioned environment on a scarce self-hosted runner. This adds the tier they can move down to. The real gateway router runs in-process: SetupRoutes() is a pure function of GatewayConfig, so it serves from an httptest server over embedded NATS JetStream with real IAM/STS services and genuine SigV4 verification. Only the daemon side of the wire is faked -- the ec2.*/ebs.* NATS subjects a live node would answer are stubbed per test. Nothing is provisioned and no fixed ports are bound, so the tier runs concurrently on a shared build host and is safe as a CI front gate ahead of the live suites. It is build-tagged and kept out of preflight, which stays unit-only. * test(integration): add IAM/STS client accessors and port the zero-stub authz tests Adds IAMClient/STSClient plus ClientsWithCreds/ClientsWithSessionCreds so a test can act as a principal other than the seeded root user, and stubs the node-discovery subject in StartGateway so call paths that trigger discovery resolve immediately instead of riding out the fan-out timeout. Ports the four enforcement tests that guard on ec2:DescribeRegions, a gateway-local action, so they need no daemon-side stub at all. * test(integration): port remaining IAM/STS authz tests from e2e Ports TestIAMUserCRUD, TestIAMAccessKeyLifecycle, TestIAMUserAuthentication, TestIAMPolicyCRUD, TestIAMPolicyAttachmentEnforcement, TestIAMPolicyLifecycle, TestIAMGroupsLifecycle, TestIAMRolesAndProfiles and TestSTSAssumeRoleAndGetCallerIdentity from tests/e2e/iam/ into the in-process integration tier. Each test now self-bootstraps its own users/policies/roles instead of relying on shared suite-level state, since every integration test gets its own gateway and embedded NATS. TestIAMRolesAndProfiles stubs ec2.IamProfileAssociation.describe because DeleteInstanceProfile always counts live associations over NATS regardless of which guard actually fires. TestIAMCleanup is not ported: it only swept a shared long-lived E2E environment, and every ordering/dependency assertion it made is already covered by TestIAMPolicyLifecycle, TestIAMGroupsLifecycle and TestIAMRolesAndProfiles's own delete-conflict guards. * test(e2e): remove IAM/STS authz tests migrated to the integration tier All 13 signed-request authz tests plus TestIAMCleanup now have faithful, mutation-proven equivalents in tests/integration/ — the gateway dispatches IAM/STS actions directly with no daemon hop, so the in-process tier is a faithful and far cheaper substitute. The iam/ e2e suite now runs only TestIAMInstanceProfileAssociation and TestIMDS. Drop the now fully-dead iam_helpers_test.go alongside the six deleted test files, and remove harness.IAMGroupARN/IAMDeleteGroupBestEffort, which had no remaining caller anywhere in the repo once iam_groups_test.go was gone. * feat(integration): daemon-lite harness for key/tags/routetable/vpc Adds DaemonLite, an in-process stand-in for a live spinifex daemon that subscribes the real key, tags, route table, and VPC/subnet/SG/IGW service implementations directly to their NATS subjects, backed by memory and embedded-JetStream stores. Ports TestKeyPairs, TestTagManagement, TestRouteTableValidation, and TestReplaceRouteConvergence from tests/e2e/single so the integration tier drives genuine daemon-side business logic instead of StubSubject canned replies. vpcd (OVN) is out of scope, so its "vpc.create-sg"/"vpc.delete-sg" acknowledgements are stubbed with a fixed success reply — the SG record itself is still written by the real service impl before that event fires, so the stub only unblocks the synchronous round-trip rather than replacing tested behavior. * ci(e2e): gate live E2E jobs behind the in-process integration tier Runs the in-process integration suite first, on the ci-single runner host rather than inside a provisioned VM, so a broken control plane is caught before either live job pays the Spinifex install. Both e2e_single and e2e_multi now require it, and e2e_status treats a skipped integration job as a failure so the gating status cannot go green on a gate that never ran. Drops the shadow-mode suite selector: targeted per-diff suite selection is not the direction being pursued, and leaving a non-functional shadow job in the gating workflow only invited it to be mistaken for a live gate. * test(e2e): remove single/ tests superseded by daemon-lite ports TestKeyPairs, TestTagManagement, TestRouteTableValidation, and TestReplaceRouteConvergence now have proven-equivalent ports in tests/integration/ (mutation testing confirmed parity). Delete the e2e originals and their runX helpers, along with keypair_test.go's and tags_test.go's file-local helpers left with no caller. generateImportPubKey is shared by other single/ tests so it moves to helpers_test.go instead of being deleted. Slims the manifest-lint baseline for the removed create-call entries. * ci: run the lb e2e suite on multinode only Both e2e jobs filter SUITES by exclusion, and neither excluded lb, so the suite ran twice per e2e run: 354s on single and 654s on multi. Balancing across instances on one node is not a deployment mode we support, and ELBv2 proper requires two AZs, so the single-node run was redundant. Drops ~354s from the single-node job, which is the gate critical path. * test(integration): port 3 of 7 ECR tests to the in-process tier Wires the gateway's ECR control plane (repository/policy/lifecycle metadata over an in-process daemon-lite NATS subscriber, blob and manifest bytes via a memory object store, JWT issuer/verifier keyed off the existing IAM master key) into the integration harness, and ports TestECRControlPlane, TestECRGetLoginPassword, and TestECRIAMAuthorization_AssumedRolePullOnly from tests/e2e/ecr — each proven equivalent by mutation (evaluatePrincipalPolicy forced to allow, and the repo-describe path forced to not-found) before its E2E original was removed. The other 4 authorization tests (PullOnly, ReadOnly, DetachPolicyDeniesImmediately, DeactivatedKeyReturns401) stay on the E2E tier: AttachUserPolicy lacks the AWS-managed-ARN existence-check bypass that AttachRolePolicy/AttachGroupPolicy already have, so attaching AmazonEC2ContainerRegistryPullOnly/ReadOnly straight to a user fails NoSuchEntity in both tiers alike. Left untouched pending a fix to that shared IAM code. * fix(iam): accept AWS-managed policy ARNs on AttachUserPolicy AttachUserPolicy called getPolicyByARN unconditionally, so attaching an AWS-managed ARN to a user returned NoSuchEntity. AttachRolePolicy and AttachGroupPolicy already skip that lookup for AWS-managed ARNs and store them opaquely; the user path was missed when that guard was introduced. ListAttachedUserPolicies had the mirror of the same gap: it resolved every ARN through getPolicyByARN and silently dropped the ones it could not find, so a managed ARN would disappear from the listing rather than round-trip. Evaluation already handled these correctly via resolveAttachedPolicy, so only the write and list paths needed the guard. * test(integration): port remaining 4 ECR authorization tests Ports TestECRIAMAuthorization_PullOnly, _ReadOnly, _DetachPolicyDeniesImmediately, and _DeactivatedKeyReturns401 from tests/e2e/ecr now that AttachUserPolicy accepts AWS-managed policy ARNs. Each was proven equivalent by mutation before its E2E original was removed: PullOnly/ReadOnly/DetachPolicyDeniesImmediately fail when evaluatePrincipalPolicy is forced to always allow, and DeactivatedKeyReturns401 fails when the access-key-active check in resolveECRLongLivedPrincipal is bypassed. All mutants were reverted and a clean run confirmed before committing. tests/e2e/ecr/iam_authorization_test.go is now fully removed — every test it held has an in-process equivalent, and its helpers had no other callers in the package. * ci: select e2e suites by allow-list instead of deny-list Both jobs filtered SUITES by exclusion, so any suite named in neither deny-list ran in both jobs. That is how lb came to run twice per e2e run, and it makes adding a suite silently fan it out everywhere. Each job now names the suites it is eligible to run. Selection is still driven by SUITES, so this is behaviour-preserving for the current default: single/iam/cert on the single-node job, multinode/lb/cert on the multi-node job, exactly as before. eks, ecs and storagegrowth are listed as eligible on the single-node job but are not yet in the default SUITES. * test(e2e): delete seven tests with no coverage TestFinalClusterStats and TestInstanceLaunch assert nothing beyond what other callers of their shared helpers already imply. The multinode baseline, volume, guest-SSH, and preflight tests duplicate single-node coverage or exercise per-guest behaviour that isn't cluster-specific. TestMultinodeInstanceDistribution can't fail: three independent RunInstances calls with no placement group make a strict spread assertion flaky and a loose one meaningless. Preflight checks (/dev/kvm writability, peer SSH reachability) move into requireMultiNodeFixture so the package fixture singleton still fails fast with a clear message instead of disappearing along with the dedicated test. The remaining callers of needInstanceTrio (CrossNodeOps, CrossNodeGateway, NodeFailure, VolumeDurability) still trigger the shared trio launch on their own, so removing the distribution and volume-lifecycle tests doesn't leave it unbuilt. Also slims the manifest-lint baseline entry for the deleted multinode volume test. * fix(e2e): require full 3-node spread in TestMultinodeSpread SpreadPlacement passed at unique>=3, warned via t.Logf at unique==2, and only failed below that. A 33% spread failure (two instances colocated) passed with a log line. The 243.6s NAT-egress scenario that follows only proves what it claims to if the private trio actually landed on three distinct chassis, so that premise needs to be a hard assertion, not a warning. * refactor(e2e): drop the unused spread-assertion harness helpers AssertSpreadAcrossNodes and CountInstancesPerNode lost their only caller when the instance-distribution test was removed. Beyond being dead, the assertion helper documented minNodes=1 as an acceptable logging-only mode inherited from the bash driver, which is the pattern that let a placement test pass while every instance landed on one node. InstanceHostingNode and its ps helpers stay; five multinode tests use them. * fix(ec2): stop leaked goroutines on prepareEFIVolume error paths newViperblock starts the chunk uploader and WAL syncer unconditionally, but prepareEFIVolume only released them on the happy path via Close(). Six error returns between acquisition and Close left both goroutines running. Add a defer immediately after acquisition, matching the sibling prepareRootVolume; Close() remains the durability boundary. * test(predastore): add real predastore fixture for integration tests Extract predastore_integration_test.go's inline daemon startup into a reusable testutil/predastore package so tests/integration can point DaemonLite-style wiring at a real S3-compatible backend instead of an unreachable host. Kept as its own leaf package rather than folding into the shared testutil package: it transitively imports predastore's quicclient, whose package-level connection pool starts a permanent goroutine at init that would trip goleak checks in any of the ~80 other packages that import testutil for unrelated NATS helpers. * test(integration): migrate CreateVolume to assert real predastore persistence Add StartVolumeDaemonLite, an opt-in DaemonLite variant wired to the real predastore fixture, and migrate TestCreateVolume_PersistsToRealPredastore off memory-backed DaemonLite so CreateVolume/DescribeVolumes exercise viperblock's real S3 backend end to end instead of stopping at input validation. * test(integration): add real storage-state oracle for CreateVolume DescribeVolumes only proves the handler's own read path agrees with its own write path, both via the same deserialization; it cannot catch a bug in a field it never re-derives (e.g. a unit-conversion error in the bytes handed to viperblock, since VolumeMetadata.SizeGiB is copied from the request directly). vbscan reads a volume's persisted config.json and chunk objects straight over S3, independent of any spinifex handler code, so TestCreateVolume_PersistsToRealPredastore can assert on what viperblock actually wrote rather than only on what the handler claims. vbscan is a local reimplementation of scripts/vbrefscan's read-only scan rather than an import: spinifex cannot depend on mulga, the module that contains it. It only reimplements the config/chunk-listing half of vbrefscan — the checkpoint-based reference-integrity half needs viperblock.ParseBlockCheckpointBytes, which does not exist in the viperblock version spinifex currently depends on (v1.12.1) and is also moot for this volume: CreateVolume never calls SaveLiveCheckpoint, so a freshly created volume has no checkpoint to scan regardless of version. * refactor(test): move the predastore daemon fixture out of the coverage scope test-cover runs go test ./spinifex/..., so anything under spinifex/ is compiled and measured. The daemon fixture cannot be exercised by unit tests, so it reported a coverage gap no unit test could close. The e2e harness already avoids this by living under tests/; the fixture now does the same. vbscan stays put, being ordinary library code with its own unit tests. * test(integration): migrate RunInstances multi-count expansion off live guests RunInstances with MinCount/MaxCount=2 expands into a single per-node targeted launch (distributeInstances in gateway/ec2/instance/placement.go) before the daemon-facing NATS hop, so the real gateway router plus a stubbed daemon response is sufficient to prove it — no throwaway VM is needed to exercise the expansion logic itself. The live single-node E2E test's WaitForInstanceState/TerminateInstances sequence was teardown hygiene for the real guests it booted, not an assertion of the expansion behavior, so it is dropped rather than emulated. The migrated test additionally asserts the per-node launch request itself carries MinCount=MaxCount=2, proving the count actually propagates rather than the launch succeeding by coincidence or via the zero-daemon degenerate path in placement.go. * docs: drop a doc reference to the deleted live test * test(integration): migrate launch-template RunInstances expansion off live guests RunInstances --launch-template resolves ImageId/InstanceType from the stored template entirely gateway-side (expandLaunchTemplate, before validation/placement/the daemon-facing NATS hop), so proving it forwards the resolved fields never needed a real guest boot. Ports the assertion to tests/integration via a new opt-in StartLaunchTemplateDaemonLite (real LaunchTemplateServiceImpl over embedded JetStream) plus a direct subscription on the per-node launch subject to capture what the gateway actually dispatched, rather than only the client-visible response. The live test keeps its CRUD/versioning lifecycle (create/modify/delete version, tag-filtered describe) — none of it touches a guest — and drops only the RunInstances-from-template + wait-for-running + terminate tail, whose only job was waiting out a real boot to read back fields already proven resolved. * test(integration): migrate KV-only account-scoping checks off live tenants Ports the KeyPair/VPC-Subnet/IGW-EIGW/AccountSettings cross-account isolation assertions from tests/e2e/single/account_test.go to the in-process integration tier. Each of these resources resolves ownership from a plain account-scoped KV key (utils.AccountKey(accountID, id), or an account-keyed settings record) with no vm.Manager or real guest involved, so DaemonLite now also wires the real EgressOnlyIGWServiceImpl and AccountSettingsServiceImpl alongside the existing VPC/IGW/Key services. Two independently minted tenant accounts (via IAMService.CreateAccount/CreateUser/CreateAccessKey/CreatePolicy/ AttachUserPolicy, in-process — the same calls runAccountCreate makes without the CLI/stdout round-trip) replace the live tier's carousel so isolation is checked between real, separately owned accounts. Left live: instance scoping (Step2) and volume scoping (Step3) — checkInstanceOwnership, the actual cross-account gate for stop/start/ reboot/terminate/attach-volume, runs daemon-side only after a real vm.Manager lookup, so it cannot be exercised without a live guest. Snapshot scoping (Step5) needs a real backing volume for CreateSnapshot (viperblock, out of DaemonLite's scope), so it stays live entirely rather than split off only its two KV-only assertions. Global-resources (Step9) carries no account branching in the handlers at all, so asserting its equality across two accounts wouldn't exercise anything a fake node responder didn't already trivially satisfy — not migrated. * test(integration): migrate launch-template CRUD lifecycle off live guests TestLaunchTemplates booted zero guests: its RunInstances coverage had already moved to TestRunInstances_LaunchTemplateExpansion, leaving only template CRUD/versioning assertions occupying a full live-cluster slot. Port the remaining create/describe/version/modify/delete lifecycle, including the SourceVersion merge and a nested timestamp round-trip, into TestLaunchTemplates_CRUDLifecycle against the in-process integration gateway. * refactor(tests): move spot instance lifecycle coverage to integration tier RequestSpotInstances is a mock over on-demand RunInstances (no bidding, interruption, or reclamation), so booting real VMs in the e2e tier proved nothing spot-specific. Migrate the SIR CRUD lifecycle (request, describe, cancel, close-on-terminate, insufficient-capacity error) to a NATS-level integration test that exercises the real gateway orchestration and daemon service impl without a live guest. Real-VM-boot assertions are already covered by TestRunInstances_MultiCountExpansion and the e2e baseline tests. * fix(tests): share one embedded NATS server across the integration package ClientTokenStore in spinifex/gateway/ec2/instance is a process-wide sync.Once singleton bound to whichever *nats.Conn first initialises it. Every integration test booted its own embedded NATS server, so the first RunInstances-calling test captured the singleton, and any later test inherited a KV handle bound to that server's now-closed connection. TestMain now boots one embedded NATS+JetStream server for the whole package; every StartGateway call connects into its own dynamically provisioned NATS account (accountAuthenticator in shared_nats.go), giving each test a private JetStream namespace despite the shared server, and none of the per-test connections are closed until every test has finished - whichever one the singleton latches onto stays live for the run. Also wire the suite into the required PR gate: it was previously only invoked from e2e.yml's push-triggered (main/dev), self-hosted "Integration (front gate)" job, so a regression here never blocked a PR. The whole package runs in single-digit seconds, so it now runs inline in `make preflight` and as its own step in spinifex.yml's build_and_test job. * test(integration): migrate serial-console access checks off live guests Enable/Disable/GetSerialConsoleAccessEnabled is pure gateway+account-settings logic with no guest involved, so it moves to the in-process integration tier against the real AccountSettingsServiceImpl wired into DaemonLite. * test(integration): migrate DescribeImages round-trip off live guests runImage's AMI describe round-trip needs no real cluster: it registers an AMI against a hand-seeded snapshot via the real ImageServiceImpl, backed by a new StartImageDaemonLite helper wiring RegisterImage/DescribeImages onto an in-memory object store. * test(integration): migrate predastore object lifecycle off live guests runPredastoreObjectLifecycle only needs a real predastore daemon, not a tofu-provisioned cluster: it now runs against tests/fixtures/predastore's in-process fixture, covering the same create/put/get/delete/list contract. * test(integration): migrate ListFoundationModels catalog check off live cluster ListFoundationModels is pure gateway-side static catalog logic with no NATS/daemon hop, so it moves to the integration tier via a new BedrockClient harness accessor; this tier configures no credential resolver, so the Anthropic entry's absence is asserted explicitly rather than left optional. * test(integration): cover RunInstances client-token idempotency Proves replaying the same ClientToken through the real gateway returns the original reservation without a second daemon dispatch, and a different token launches independently. * test(integration): cover RunInstances launch-template expansion Proves version selection (omitted Version resolves to the template default, explicit Version selects that version) and field precedence (explicit RunInstances parameters override the template, unset fields inherit it) through the real gateway. * test(integration): cover RunInstances malformed-input rejection Proves a malformed ImageId and a malformed targeted-launch CapacityReservationId are rejected before any NATS dispatch, with no daemon stub registered so a regression that let either through would fail fast instead of hanging. * test(integration): cover RunInstances iam:PassRole enforcement Proves a caller granted ec2:RunInstances but not iam:PassRole on an instance profile's role is denied, and the identical session succeeds once PassRole is granted for that exact role ARN. * test(integration): cover RunInstances placement-group strategy routing Adds a placement-group daemon-lite wiring a real PlacementGroupServiceImpl over NATS, then proves spread strategy assigns exactly one instance per node while cluster strategy pins the whole launch to the single highest-capacity node. * test(integration): cover RunInstances capacity distribution across nodes Proves the default (non-placement-group) path packs a multi-instance launch across nodes proportionally to spare capacity rather than exhausting one node first. * test(e2e): merge SameSGComms and SGToSGDatapath around one client/target pair Both booted their own instance pair to check SG-to-SG datapath enforcement and same-default-SG east-west connectivity. Merging them around one shared client/target pair cuts four boots to two, and converting the inline harness.Step sequence into named t.Run stages (PortGroupMembership, AllowedTraffic, DeniedTraffic, RevokeAndVerifyBlocked, ReauthorizeAndVerifyRestored, SameSGComms) gives each its own pass/fail attribution instead of one opaque test result. SameSGComms now runs last and joins the default SG onto each ENI via ModifyNetworkInterfaceAttribute only after every SG-to-SG allow/deny assertion has already recorded its own result, since the default SG's self-reference rule admits all traffic between co-members and would otherwise defeat the denied-traffic check. * test(e2e): merge default-SG reachability, guest DNS, and egress enforcement DefaultSGReachabilityBaseline, GuestDNSResolution, and SecurityGroupEgress each booted their own instance to check one facet of "a guest is reachable only when policy says so": ingress deny/authorize, resolver + WAN egress, and egress ACL enforcement. Merging them around one shared guest cuts three boots to one, with named t.Run stages (Deny, Authorize, Resolver, WANEgress, RevokeEgress/RestoreEgress) giving each its own attribution. The egress revoke/restore round-trip now mutates this scenario's own dedicated SG instead of the shared default SG the rest of the suite depends on, isolating its egress-rule mutation from every other test. * fix(ci): single-quote e2e test_run regex across the ssh boundary * fix(e2e): anchor SG egress-enforcement probe to a WAN IP, not the gateway * test(e2e): merge single-node guest churn tests into one durability scenario Combines TestSSHProbe, TestENIHotplug, TestENIHotplugReconcile, TestVolumeDurability, TestModifyInstanceAttribute, and TestRebootInstance into TestGuestChurnDurability, a single shared-guest scenario that re-verifies a data-integrity sentinel sha256 after every perturbation stage (ENI hotplug, daemon restart, ENI detach, instance-type resize, stop/start, reboot). This proves the sentinel survives the full churn sequence rather than just one isolated stop/start. * fix(vpc): clear subnet drop gate when installing egress reroute A subnet with an attached IGW but no 0.0.0.0/0 route carries a priority-1100 drop gate on the VPC router. When a route later lands (NATGW reroute at 900, IGW reroute at 1000) the reroute is installed but the drop gate was never removed, so the higher-priority gate wins and black-holes the subnet's egress in lr_in_policy before it reaches routing or SNAT. This left NAT-gateway private-subnet egress with 100% packet loss: the SNAT stage saw zero packets and no conntrack was committed. Remove the subnet's drop gate for the same prefix once a reroute is installed. Idempotent and prefix-scoped, so a specific-prefix reroute is a no-op and leaves the default gate intact. * test(e2e): merge VPC subnet, NAT gateway, and EIP single-node tests Merges the former TestVPCSubnetE2E, TestNewVPCEgressBaseline, TestNATGateway, and TestInstanceEIP into one staged scenario, TestVPCEgressPaths, around one scenario-owned VPC and its two guests, cutting five guest boots to three. The public guest serves PublicSubnetEgress, then the NAT Gateway bastion hop, then (last, since association rewrites its public IP) the EIP flip. The private guest serves the NAT Gateway path, launched directly into the same VPC's already-existing private subnet instead of a dedicated one. RouteBeforeSubnet keeps its own second VPC and instance since it exercises the implicit main-route-table code path, a different one than the explicit route table this scenario's own VPC uses for its public subnet — collapsing the two would drop coverage of whichever path got merged away. The NAT Gateway up/down flip now repeats for an env-tunable round count (SPINIFEX_VPCEGRESS_NAT_ROUNDS, default 1), matching every other merged single-node scenario. * test: TEMPORARY mutation-check — revert |
||
|
Ben Duncan
|
28a876d17d | feat(dns): resolvable ALB/NLB + EKS DNS, forwarder hardening, cross-tenant reconcile (#532) | ||
|
Josh Siviour
|
e262ee0064 |
fix(elbv2): reap LBs wedged in provisioning + harden IMDS bootstrap (#496)
An ELBv2 LB whose system VM launches but whose in-guest lb-agent never delivers a first heartbeat previously stayed in provisioning forever with an empty StateReason: promotion to active happens only on the first heartbeat and the only provisioning->failed transitions were at create/launch time. A single wedged LB in the multi-node e2e lb suite leaked its listener/target-group and cascaded into ResourceInUse and DependencyViolation on VPC teardown. - Add a lifecycle reaper that marks failed any LB still provisioning past lbProvisioningTimeout (4m) with no heartbeat, reusing markLBFailed, and populate a provisioning StateReason so a stuck LB is self-describing. Started once at daemon startup; stopped on service Close. - Harness: add ErrLBProvisioningTimeout and treat a provisioning-timeout as retryable in createActiveLB; register idempotent LB/listener teardown up-front so no exit path leaks into the shared VPC. - IMDS: bounded retry+backoff on the instance-fact and user-data lookups so a transient gather timeout under multi-node load does not starve the lb-agent bootstrap. |
||
|
joshsiv-mulga
|
96909d0e9a |
fix(e2e): retry LB creation on sys.micro capacity reclaim race in lb suite
The internal lb subtests raced the previous subtest's sys.micro VM deallocation: fixed 15s sleeps were not enough on capacity-tight single nodes, so the LB entered terminal state failed (insufficient capacity, then nats no-responders after the daemon unsubscribed the full topic). Replace the sleeps with createActiveLB: create LB + listener, wait for active via the new non-fatal harness WaitForLBActiveErr, and on a terminal failed state (wraps ErrLBTerminalFailed) tear down fully and retry with backoff, up to 3 attempts. |
||
|
Josh Siviour
|
2d4f0fa94c |
feat: IMDS instance-role creds for lb-agent + EKS server VMs (siv-231.6) (#450)
* feat: IMDS-capable gateway signer for lb-agent + EKS clients Add internal/gwsign, a SigV4 signer that draws credentials from either fixed static keys or the AWS SDK default credential chain. In a guest VM the chain falls back to the in-VM IMDS endpoint, yielding scoped, auto-rotating instance-role credentials; a session token is signed as X-Amz-Security-Token so the gateway can verify temporary (ASIA) creds. Wire it into the lb-agent (lbagent.Agent) and the EKS on-VM client (internal/eksgw). When static keys are absent the clients now sign with IMDS credentials instead of failing; when present, behaviour is unchanged (static path). This is the client-side foundation for dropping long-lived secrets from VM user-data. Credentials are retrieved per request so rotated IMDS creds take effect without an agent restart. * feat: IMDS instance-role creds for EKS server + LB VMs (Phase B/C) Attach a Spinifex-managed system instance profile to the k3s control-plane VM and the elbv2 LB VM so both authenticate to the gateway with scoped, rotating IMDS instance-role credentials instead of a long-lived static secret baked into user-data. - Add a shared handlers/iam.EnsureSystemInstanceProfile that find-or-creates a role (EC2 trust) plus its instance profile and asserts an inline permission policy. The gateway enforces per-action authz on assumed-role principals, so the roles carry exactly the actions each VM calls: - EKS server: eks:PublishInternal, eks:ListInternalAddons. - LB agent: elasticloadbalancing:LBAgentHeartbeat, GetLBConfig. - Plumb IamInstanceProfileArn through sysinstance.SystemInstanceInput and both daemon launch paths (BootAMI via PrepareRunInstances->RunInstance; direct-boot microVM via RunInstance), which persist it so the in-VM IMDS endpoint serves the role. - Drop the baked secrets when a profile is attached: omit EKS_ACCESS_KEY/ EKS_SECRET_KEY from the k3s first-boot env and LB_ACCESS_KEY/LB_SECRET_KEY from the lb-agent fw_cfg env; relax k3s server validation to accept profile-only; drop the EKS key requires from the first-boot and state-report scripts. - Both verticals fall back to baked static creds when IAM is unwired, so an absent master key never blocks a launch. * fix: resolve IMDS instance-role creds under the instance account for system VMs Three boot-path defects kept the LB VM's lb-agent from authenticating to the gateway with IMDS-served instance-role creds; each produced a silent empty IMDS role list and a heartbeat failing with "unexpected empty EC2 IMDS role list". Found and fixed against env19. - Build the system IAM ensurer lazily on the first system-VM launch instead of eagerly at daemon startup, where it raced the NATS KV backend and fell back to baked static creds permanently. ELBv2 and the EKS deps now hold an IAMProvider resolved at launch time. - Create the LB instance profile in the system account the LB VM runs under (utils.GlobalAccountID), not the LB-owner account. IMDS resolves the profile under the instance's account, so a customer-account role was invisible to it. - Record the attached instance's owning account on the ENI (InstanceOwnerId, mirroring AWS Attachment.InstanceOwnerId) and resolve the instance and every IAM/STS lookup under it. The LB ENI lives in the customer account while the system LB VM and its role live in the system account; keying the whole IMDS chain off the ENI account left the instance invisible and the role lookup missing. VPC leaves (subnet/VPC/SG CIDR) keep using the ENI account. A fresh LB on env19 then authenticated end-to-end: IMDS serves a non-empty role list plus ASIA creds, and the gateway accepts the heartbeat as an assumed-role principal with no static keys in user-data. * test(e2e): assert LB VM authenticates via IMDS instance-role creds The LB suite's targets-healthy assertion already depends on the lb-agent authenticating to the gateway, and with static keys dropped from user-data that auth is IMDS-only. Add an explicit check that a recent gateway authentication for the LB used an assumed-role (ASIA) principal, so a regression to the instance-account IMDS resolution fails with a clear message instead of a mysterious datapath timeout. The LB VM is a system-account instance invisible to the customer AWS API, so the assertion observes the gateway journal on the cluster nodes; it is time-bounded to avoid the stale-console-log false-fail and skips when no node is SSH-reachable. |
||
|
Julian Sommer
|
2e9501524b | feat(e2e): more e2e coverage (#365) | ||
|
Julian Sommer
|
50b5495d26 | chore: more comment cleanup | ||
|
Josh Siviour
|
527308c342 |
fix(baremetal): empty modules.dep + capacity-aware single-node e2e (#305)
* feat(elbv2): localize internal-LB heartbeat failures for baremetal debug Add diagnostics so a stuck-in-provisioning internal LB can be pinned to one of three points on the heartbeat path: - lb-agent failure log now carries the gateway target + region, so the microVM serial console shows whether the heartbeat hit a transport error (never reached AWSGW) or an HTTP-level rejection. - AWSGW LBAgentHeartbeat logs receipt (debug) and warns on the previously-silent not-found/account-mismatch reject path, so the journal distinguishes 'heartbeat arrived but cannot transition' from 'heartbeat never arrived'. * test(e2e/lb): fail fast on stuck internal LB + probe fw_cfg on microvm boot - lb suite: gate the remaining internal subtests on Internal_ALB. The internal LBs share one mgmt-return datapath and fixture, so if the first never reaches active the rest time out identically — skip them instead of burning ~5m per subtest (~20m saved on a known-bad run). - microvm init.sh: widen the fw_cfg netcfg poll window and, on the not-available path, dump DMI / loaded modules / qemu_fw_cfg by_name listing / dmesg fw_cfg to the serial console before init exits, so a boot panic is diagnosable from the captured console log. * fix(microvm): insmod fallback for fw_cfg + fail build on bad modules.dep Baremetal-built microVM images shipped an unusable modules.dep (the build host's depmod errors were swallowed), so busybox modprobe in the guest init silently no-oped, qemu_fw_cfg never loaded, /sys/firmware/qemu_fw_cfg was absent, init could not read opt/spinifex/netcfg, and the LB microVM panicked before lb-agent ran — leaving internal LBs stuck in provisioning. init.sh: load_mod() tries modprobe, then falls back to insmod by explicit path (.ko/.ko.gz) which needs no modules.dep, so the guest boots regardless of which host built the image. build-microvm-image.sh: require depmod, drop the error swallow, and assert modules.dep is non-empty after depmod so a boot-broken image fails the build loudly instead of silently. * fix(microvm): decompress kernel modules before depmod Debian trixie's kmod 34 depmod silently emits an empty modules.dep when fed Alpine's gzip-compressed (.ko.gz) modules — reproduced: 928 modules in, 0-byte modules.dep out. The guest's busybox modprobe then no-ops, qemu_fw_cfg never loads, and internal LB microVMs panic before lb-agent runs (stuck provisioning on bare metal; GitHub-hosted runners ship a kmod that reads .gz, so normal CI was unaffected). Decompress kept modules to plain .ko after the keep-list strip, before depmod. Host depmod then produces a usable map regardless of its gzip support, and the guest loads modules without a decompressor. * feat(e2e): capacity-aware scheduling for dense single-node runs Denominate daemon schedulable capacity in physical cores instead of logical threads, and auto-serialise e2e launches when the cluster is at capacity, so a single hyperthreaded node stops over-packing guests. Daemon: NewResourceManager sizes hostVCPU from physicalCoreCount() (distinct physical-id:core-id pairs in /proc/cpuinfo) rather than runtime.NumCPU(). Falls back to logical CPUs when topology is absent (non-Linux, container, ARM cpuinfo without core id). On a hyperthreaded box this halves the over-pack: with the unchanged 2 vCPU reserve the i5-12600H schedules 10 vCPU (12 cores - 2) instead of 14, keeping the boot-spike CPU within real cores. Harness: capacityRetryEC2 decorates the EC2 client and, on InsufficientInstanceCapacity, waits and retries RunInstances instead of failing — turning the daemon admission gate into back-pressure so concurrent subtests queue. AWSClient.EC2 is now the wrapped ec2iface.EC2API; EC2Conf keeps the raw *ec2.EC2 for .Config access. Adds unit tests for parsePhysicalCores (hyperthreaded/hybrid/dual-socket/ fallback) and the retry decorator (retry-then-succeed, non-capacity passthrough, deadline). * fix(daemon): SPINIFEX_HOST_VCPU override so small CI hosts can start Denominating capacity by physical cores starves tiny hosts: a 2-physical- core CI runner leaves 0 schedulable after the default 2-vCPU reserve, so NewResourceManager refuses to start and every daemon-constructing unit test fails. The previous runtime.NumCPU() count (4 logical on the same runner) incidentally cleared the reserve floor. Add a SPINIFEX_HOST_VCPU escape hatch (symmetric with the existing SPINIFEX_RESERVED_* overrides) that pins the detected core count for hosts whose /proc/cpuinfo topology misreports, and pin it to 4 in the daemon TestMain so unit tests are deterministic regardless of runner topology. Reserve stays at the default 2. |
||
|
joshsiv-mulga
|
8e38faaede |
fix(elbv2): make NLB nginx health-check pathway robust in CI
E2E NLB suites timed out at "0/2 healthy" while ALB suites on the same target instances went healthy in ~5s. The host daemon journal shows the NLB agent reporting an empty server list (no probe results at all), which happens when the lb-agent never enters nginx mode — i.e. `nginx -t`/start fails closed inside the LB microVM, so the agent never probes and reports nothing. The data plane itself is verified working; this is purely the active-health-check pathway failing to converge in CI. - lbagent: pre-create nginx runtime prefix dirs (pid dir + /var/lib/nginx, /var/lib/nginx/tmp, /var/log/nginx) before `nginx -t`, mirroring reloadHAProxy's socket-dir pre-create. A stripped Alpine initramfs may lack these, dead-ending the whole NLB health path. - build-microvm-image.sh: assert the nginx binary and ngx_stream_module.so are present in the image and fail the build loudly, instead of surfacing as an opaque e2e health timeout. - e2e: capture the LB VM serial console to the artifact bundle on NLB health-wait failure. The agent's engine activation and any `reload nginx:` error land on the guest console, which the host journal never sees. |
||
|
Josh Siviour
|
554c5d96c7 |
feat(elbv2): NLB L4 data plane on nginx stream + UI parity (#295)
* feat(elbv2): NLB L4 data plane on nginx stream (mulga-siv-952) Network load balancers render an nginx `stream` config (TCP/UDP/TLS/ TCP_UDP) instead of HAProxy, which cannot load-balance UDP in any release. The gateway tags GetLBConfig with Engine=nginx; the LB agent dispatches to the nginx engine (config/pid/cert paths + reload) and skips HAProxy stats polling. ALB stays on HAProxy. nginx is added to the microvm image. * docs(elbv2): note NLB runs nginx stream, ALB runs HAProxy (mulga-siv-952) * fix(elbv2): load nginx stream module for NLB data plane Alpine's `nginx` apk is built without the stream module — it ships separately as `nginx-mod-stream`. Without it, the generated NLB config fails `nginx -t` with [emerg] unknown directive "stream", so the agent never applies it. Add the package to the microvm image and emit a `load_module .../ngx_stream_module.so;` directive at the top of the generated config. * fix(elbv2-ui): allow single-subnet LB + full NLB protocols in listener edit Create-LB form required min 2 subnets while the CLI/backend accept 1; relax to min(1) to match. Listener add/edit only offered HTTP/HTTPS — add the NLB protocol set (TCP, UDP, TLS, TCP_UDP), driven by LB type, with cert/ssl-policy fields gated on secure protocols (HTTPS/TLS). * feat(elbv2): active target health checks for nginx NLBs nginx stream has no active upstream probing, so NLB targets never advanced past initial — DescribeTargetHealth stayed initial forever and WaitForTargetsHealthy timed out for every NLB. GetLBConfig now delivers per-target HealthTarget specs (daemon-computed server name, probe address, protocol/path) for network LBs; the lb-agent probes each target every tick (TCP dial or HTTP/HTTPS GET) and reports UP/DOWN via the existing heartbeat path, so the daemon's threshold hysteresis advances targets to healthy. ALBs are unchanged (HAProxy stats). Adds an Internal_NLB_UDP e2e suite gating on WaitForTargetsHealthy plus a UDP datagram round-trip through the VIP. |
||
|
Josh Siviour
|
dd65a2dfa6 |
feat(elbv2): ListenerRules CRUD with HAProxy ACL routing (#275)
* feat(elbv2): ListenerRules CRUD with HAProxy ACL routing Implement CreateRule, ModifyRule, DeleteRule, DescribeRules, and SetRulePriorities for ALB listeners. Rules support host-header, path-pattern, http-header, http-request-method, source-ip, and query-string condition fields. Actions limited to forward (single target group); redirect and fixed-response deferred. HAProxy frontends emit per-rule ACLs with `use_backend ... if <acls>` ordered by priority. Wildcards in host/path map to HAProxy match modes (`-m beg/end/sub/str`). Condition values are validated at store-time and re-validated at render-time to prevent config injection. Target groups referenced by rule actions are protected against deletion. Deleting a listener cascades its rules. Frontend adds a Rules tab beneath each listener row showing rules with conditions and target groups, plus add/edit/delete dialogs with field-typed condition inputs. mulga-950 * test(elbv2): rule CRUD e2e + render coverage for all condition fields Adds Internal_ALB_ListenerRules subtest exercising CreateRule, ModifyRule, DeleteRule, and DescribeRules over a real ALB with HAProxy. Two TGs with disjoint backends (app#0 -> tgA, app#1 -> tgB) let the probe responder identify which TG actually served the request, so path-pattern routing can be asserted without inspecting internal state. Extends app-userdata.sh with alpha/ and beta/ sub-paths and extends client-userdata.sh trigger to accept optional http_path and host fields so probes can target specific paths and Host headers. Both changes are backward-compatible: existing callers default to / with no Host override. Adds unit-test coverage for every condition-field render path (http-header, http-request-method, query-string with and without key, source-ip) plus error-path validation tests for CreateRule, ModifyRule, DeleteRule, DescribeRules, and SetRulePriorities. Lifts diff coverage above the 60% gate. * test(elbv2): wait for tgB healthy after rule creation, not before Spinifex only health-checks a target group once it is referenced by a listener default action or a rule action. The previous ordering waited for tgB healthy before CreateRule wired it, so health checks never ran and the wait timed out. Move the WaitForTargetsHealthy(tgB) call after createPathRule. * fix(elbv2): include rule-action TGs in TargetGroupsForLB healthChecker uses Store.TargetGroupsForLB to map HAProxy server statuses back to TargetGroupRecord entries. The previous implementation walked only listener DefaultActions, so a target group referenced only by a CreateRule action was invisible to the health checker — HAProxy probed its backends but the resulting health never propagated to the store. DescribeTargetHealth reported the targets as 'initial' forever. Extend TargetGroupsForLB to walk rule actions per listener as well. Adds a unit test that wires a TG via CreateRule and asserts the resolver returns it. * test(elbv2): fix wait-for-rule logic to poll away from default backend waitForPathRouted required the caller to name the expected new host, but at the post-CreateRule wait point we only know the default host (appA); the rule target hostname (appB) is discovered by the subsequent probe. The previous code passed appA as both "new" and "old", so the wait checked "see appA" against "/alpha/" responses — which never matched because the rule had successfully routed appA out. Replace with two narrower helpers: waitForPathRoutedAway (poll until host X stops appearing) and waitForPathRoutedTo (poll until host Y appears). Update the three call sites to use the variant that matches what the caller actually knows at that point. * fix(elbv2): IAM-style error envelope + ModifyTargetGroup + listener attr stubs ELBv2 success responses use the IAM-style XML envelope (<ErrorResponse>), but error responses were emitted with the EC2 envelope (<Response>), which aws-sdk-go-v2 cannot deserialize for ELBv2 and surfaces as "UnknownError: UnknownError" — defeating provider-side InvalidAction ignores (e.g. terraform-provider-aws v5.100's DescribeCapacityReservation swallow). Route elasticloadbalancing errors through GenerateIAMErrorResponse. Add ModifyTargetGroup end-to-end (interface, impl, NATS shim, daemon handler, gateway dispatch) so terraform-provider-aws can set health-check fields after CreateTargetGroup without infinite drift. Persist HealthCheckEnabled on TargetGroupRecord and emit it in tgRecordToSDK so the provider sees the configured value across refresh cycles. Stub DescribeListenerAttributes / ModifyListenerAttributes with hand-rolled input/output structs (aws-sdk-go v1.55 lacks these types). Returns empty attribute list — sufficient for default-config workflows. * test(elbv2): cover ModifyTargetGroup + listener attribute stubs Extend handler tests with all-fields, invalid path/matcher, wrong-account ModifyTargetGroup cases plus HealthCheckEnabled assertion on CreateTargetGroup. Add gateway-layer validation tests for the new DescribeListenerAttributes, ModifyListenerAttributes, and ModifyTargetGroup dispatchers. Raises diff coverage from 58.8% to 63.7%. |
||
|
Josh Siviour
|
3415d9f3de |
fix(network): IGW route-table AWS parity + per-subnet egress gating (#270)
* fix(network): IGW default route AWS-parity via per-subnet LR policy
ADR-0006's IGWManager.AttachIGW installed a VPC-router-wide 0.0.0.0/0
static route at attach time, violating AWS semantics: every subnet on
the LR saw egress to the IGW regardless of its route table. Private
subnets that should be IGW-isolated were not.
Move IGW egress lifecycle onto the route-table API. CreateRoute(IGW,
0.0.0.0/0) emits vpc.add-igw-route per associated subnet; the network
subscriber installs an OVN Logical_Router_Policy matching
`inport == "rtr-<subnetID>" && ip4.dst == 0.0.0.0/0` with action
`reroute` via the IGW gateway port. DeleteRoute / DisassociateRouteTable
/ ReplaceRouteTableAssociation tear it down. AssociateRouteTable emits
add events so the terraform ordering (CreateRoute -> Associate) is
honored.
Additions:
- nbdb.LogicalRouterPolicy + Logical_Router_Policy registration
- ovn.Client {Add,Delete,Find,List}LogicalRouterPolicy + live + mock
- policy.RouteManager.{Add,Delete}SubnetEgress with priority constants
- external.IGWManager.{Ensure,Remove}SubnetEgress resolving nexthop +
output port from the gateway state
- subscribers handleAddIGWRoute / handleDeleteIGWRoute on
vpc.{add,delete}-igw-route
- routetable publishIGWRouteEvents / *ForAssociation / *ForSubnet
Test updates:
- network/policy: AddSubnetEgress idempotency, drift-replace, per-subnet
separation, DeleteSubnetEgress
- network/external: AttachIGW must NOT install a router-wide default
route; EnsureSubnetEgress installs the per-subnet policy with the
resolved nexthop (link-local fallback + allocator override paths)
- handlers/ec2/routetable: CreateRoute(IGW), Associate, DeleteRoute,
Disassociate publish events with the expected subnetID + igwID
- ovn/nbdb: table-count + Logical_Router_Policy presence
- subscribers: TestSubscribe_RegistersAllTopics covers the two new topics
* fix(network): NATGW per-subnet egress + DeleteRoute event symmetry
handlers/ec2/routetable:
- DeleteRoute now publishes vpc.delete-nat-gateway per associated subnet
(mirror of the IGW DeleteRoute path) so the subscriber tears down both
the SNAT rule and the per-subnet egress policy.
- publishNatGatewayEvent* signatures carry SubnetId + DestinationCidr so
the subscriber can install the matching Logical_Router_Policy.
network/external:
- IGWManager: EnsureNATGatewaySubnetEgress / RemoveNATGatewaySubnetEgress
install/remove a per-subnet LR policy at priority SubnetEgressPriorityNATGW.
Same nexthop and output port as the IGW priority sibling; lower priority
so an IGW route wins when both apply to the same subnet.
- EnsureSubnetEgress delegates to ensureSubnetEgressAtPriority for shared
validation, lookup, and slog tagging.
network/subscribers:
- handleAddNATGateway: after SNAT install, parse DestinationCidr (defaults
to 0.0.0.0/0) and call EnsureNATGatewaySubnetEgress. Without this the LR
has no route to egress NAT'd packets and private-subnet traffic is
dropped (E2E Phase 8d regression).
- handleDeleteNATGateway: remove the per-subnet policy FIRST, then detach
SNAT, so packets are never routed unmasqueraded after SNAT removal.
- NATGatewayEvent gains SubnetId + DestinationCidr.
Tests:
- policy.AddSubnetEgress validation + drift-replace + propagation paths.
- mock LR-policy methods reject missing routers and tolerate dangling UUIDs.
- subscribers cover NATGW add/delete with invalid CIDR, bad JSON, missing
router, IGW-route bad JSON + invalid CIDR. Full end-to-end add+delete
via NATS asserting policy lifecycle.
- IGW manager covers RemoveSubnetEgress / RemoveNATGatewaySubnetEgress
empty-arg rejection + delete error propagation.
- routetable: DeleteRoute(NATGW) publishes per associated subnet.
* fix(network): exclude VPC CIDR from per-subnet egress reroute predicate
In-VPC traffic (e.g. bastion -> private SSH) was being intercepted by the
NATGW per-subnet egress LR policy on `ip4.dst == 0.0.0.0/0` and forwarded
out the gateway with SNAT, breaking peer-subnet connectivity. Append
`&& ip4.dst != <vpcCIDR>` to the match so in-VPC destinations skip the
reroute and follow the directly-connected route. Apply to both IGW (1000)
and NATGW (900) priorities. CIDR is read from the LR ExternalIDs label
populated at VPC create.
DeleteSubnetEgress takes the same excludes so the match string is
identical on cleanup. igwManager.vpcExcludeCIDRs fetches from the router
each call; lookup failure returns nil and the policy installs without
exclusions (preserves prior behaviour as fallback).
* fix(routetable): publish per-subnet events for implicit main-RT membership
E2E run 26620082197 RCA: subnets with no explicit RT association inherit
the VPC main route table per AWS semantics, but publishIGWRouteEvents and
publishNatGatewayEvents iterated only record.Associations and skipped
Main:true/SubnetId:"" sentinels. Main RT carries exactly one such
sentinel, so CreateRoute(igw)/CreateRoute(natgw) on the main RT fanned
out to ZERO subnets — subscriber never called EnsureSubnetEgress, OVN
LR policy never installed, ping 8.8.8.8 from default-VPC VMs dropped at
the VPC router. Caused TestVPCSubnetE2E, TestNATGateway,
TestMultinodeSpread/NATGatewayInternet to fail.
Add subnetsImplicitlyOnMainRT helper that enumerates VPC subnets
without explicit non-main associations, and fan publish to them when
record.IsMain. Apply to all three publish helpers (IGW, NATGW,
NATGW-delete).
Symmetric fixes for effective-RT changes:
- AssociateRouteTable: subnet leaves implicit main → emit deletes for
main RT's routes before adds for explicit RT
- DisassociateRouteTable: subnet returns to implicit main → emit adds
for main RT's routes after deletes for departing RT
* chore(e2e): capture per-LR lr-route-list/lr-policy-list/lr-nat-list
ovn-nbctl show does not dump static routes or policies. Adds three
per-LR captures so RCA can verify policy/route install state after
the IGW per-subnet refactor.
* chore(e2e): tcpdump br-wan + neigh/fdb/conntrack capture during tests
EIP egress regression on feat/igw-route-table-aws-parity shows SNAT
packets present (n_packets>0) but pings still time out. Post-teardown
ovn.* snapshots can't tell whether replies actually arrived on br-wan
or got dropped at L2 / by upstream router.
start-net-capture.sh: nohup tcpdump on br-wan filtered to 192.168.0.0/23
(EIP pool) + ARP for same range, 200-byte snaplen, 64 MiB cap. Pair:
capture-net-diag.sh stops it via /tmp/spinifex-e2e-tcpdump.pid and copies
wan.pcap into net-diag/.
Also added neigh/bridge-fdb/conntrack dumps to capture-net-diag.sh so
L2/L3 binding state is visible alongside the existing OVN dumps.
Wired both single-VM and multinode workflows to start capture before
the e2e run; teardown step picks up the pcap automatically.
* fix(reconcile): rebuild IGW/NATGW per-subnet egress from KV
Publisher fires vpc.add-igw-route during default-VPC bootstrap before the
subscriber daemon attaches to the topic. NATS core pub/sub drops the
message, leaving the LR without a 0.0.0.0/0 policy and off-subnet egress
broken on the affected VPC for the lifetime of the process. Reproduced
in CI: TestVPCSubnetE2E bastion EIP could ARP and DNS its on-LAN
neighbour but never reach 8.8.8.8.
LoadIntentFromKV now fans every IGW or NAT-gateway RT route out over
its explicit + implicit-main subnets and the reconcile pass invokes
EnsureSubnetEgress / EnsureNATGatewaySubnetEgress for each one. Same
shape as publishIGWRouteEvents so the bootstrap and the runtime event
path produce identical OVN state.
* fix(routes): serialize OVN egress wiring via request-reply
vpc.igw-attach, vpc.add-igw-route and vpc.add-nat-gateway were fire-and-forget
Publish, racing the subscriber's AttachIGW barrier. AttachInternetGateway
returned before vpcd finished wiring the OVN gateway LRP, so a follow-up
CreateRoute(IGW) published an add-igw-route event that the subscriber
processed in a parallel goroutine — installing a Logical_Router_Policy with a
nexthop the LRP didn't yet expose. northd compiled an inert reroute that
never recovered, leaving the bastion's egress traffic blackholed (no packets
dst 8.8.8.8 from the failing VPC's EIP even though local DNS succeeded).
Switch the three add-* publishers to utils.RequestEvent. The subscriber ACKs
once OVN state is committed, so the gateway handler can't issue subsequent
route events against a half-wired router. delete-* paths stay Publish —
teardown ordering doesn't matter.
handleAddNATGateway now calls respond() at every exit so requesters don't
hang. Route-table tests gain an ackSub wrapper (async subscribe + auto-Reply)
so request-based publishers complete during unit tests.
* Revert "fix(routes): serialize OVN egress wiring via request-reply"
This reverts commit bf6f77d5bc8b21f72ed5867175d5a67c010066c8.
* fix(igw): restore default static route in AttachIGW/DetachIGW
OVN 25.03 LR pipeline runs lr_in_ip_routing (table 15) BEFORE lr_in_policy
(table 17). When this branch swapped the per-VPC default static route for
the per-subnet Logical_Router_Policy reroute installed by EnsureSubnetEgress,
external destinations started dropping at routing priority-0 before the
policy table ran. Reproduced locally: ovn-trace stops at table 15 drop, ping
8.8.8.8 from inside a fresh-VPC bastion produced zero packets on br-wan
until `ovn-nbctl lr-route-add <router> 0.0.0.0/0 <wan-gw> <gw-port>` was
issued manually — after which the same ping succeeded immediately. dev still
green because dev calls AddDefaultRoute on AttachIGW.
Restore the original AddDefaultRoute / DeleteDefaultRoute calls; keep the
per-subnet EnsureSubnetEgress machinery intact so future per-subnet egress
gating (DROP policies for subnets whose RT lacks a 0.0.0.0/0 route) can
layer on top. Flip the two TestAttachIGW assertions that pinned the
"AttachIGW must not install a default static route" expectation.
Pairs with revert of bf6f77d5 in the previous commit (request-reply
serialization treated a non-existent race; real bug was missing routing
fallback).
* feat(network): per-subnet egress drop gating for AWS RT parity
VPC LRs carry a router-wide 0.0.0.0/0 default static route while an IGW is
attached (lr_in_ip_routing runs before lr_in_policy on OVN 25.03, so per-
subnet reroute alone is insufficient). Subnets whose effective route table
lacks 0.0.0.0/0 were previously able to egress via that default route,
violating AWS semantics.
Install an LR_Policy DROP at SubnetEgressPriorityDrop=1100 (above IGW=1000
and NATGW=900) for unrouted subnets. Match excludes VPC CIDR, link-local
(169.254/16) and multicast (224/4) so intra-VPC, DHCP/metadata and
multicast still flow.
Wiring:
- policy.RouteManager gains AddSubnetEgressDrop / DeleteSubnetEgressDrop.
- external.IGWManager gains EnsureSubnetEgressDrop / RemoveSubnetEgressDrop.
- New events: SubnetEgressGateEvent / SubnetEgressUngateEvent on topics
vpc.gate-subnet-egress / vpc.ungate-subnet-egress.
- routetable handler publishes gate/ungate alongside existing add/delete
IGW/NATGW route events on every CreateRoute/DeleteRoute/Associate/
Disassociate/Replace mutation, based on subnet's recomputed effective RT.
- reconciler bootstrap path emits DropGates intent for subnets in IGW-
attached VPCs whose effective RT lacks 0.0.0.0/0.
Tests cover primitive idempotency, drift recovery, event publishing per
mutation surface, implicit-main fan-out, and intent computation.
* fix(igw): fan out subnet egress gate decisions on IGW attach/detach
The reconciler computes per-subnet DROP policies for any subnet in an
IGW-attached VPC whose effective route table lacks 0.0.0.0/0, but those
decisions were only re-evaluated by the periodic drift tick (every
5 min). AttachInternetGateway / DetachInternetGateway now publish
gate/ungate events for every subnet in the VPC immediately, closing
the window where a freshly-attached IGW could leave unrouted subnets
unprotected.
Wiring: IGWServiceImpl gains a nil-safe GatePublisher hook satisfied
by RouteTableServiceImpl. Daemon installs the hook once both services
exist. RouteTableServiceImpl.PublishGateDecisionsForVPC enumerates the
VPC's subnets and reuses publishSubnetEgressGateDecision per subnet.
Also fixes the LB e2e suite, which built VPC + IGW + subnet without
ever creating a 0.0.0.0/0 → IGW route — relying on the pre-DROP-gate
behaviour where unrouted subnets still had silent egress via the
router-wide static route. The fixture now installs the explicit
public route on the main RT (AWS-faithful semantics for a public
subnet).
* fix(igw): skip gate fan-out on attach to avoid bootstrap race
AttachInternetGateway fan-out races with the default-VPC bootstrap path
in ensureDefaultVPCInfrastructureFor, which attaches the IGW and
immediately publishes a 0.0.0.0/0 → IGW CreateRoute event. The two
events land on distinct NATS subjects (vpc.gate-subnet-egress vs
vpc.ungate-subnet-egress) with no cross-subject ordering guarantee.
When ungate arrived first the subscriber became a no-op and the
subsequent gate installed a permanent DROP on the default subnet,
killing the return path for guest SSH probes
(TestMultinodeGuestSSH connect timeout to the instance public IP).
Detach fan-out is retained — no follow-up CreateRoute fires after
DetachInternetGateway so the path is race-free.
Subnets without a default route still get DROPped during the
reconciler's periodic drift pass (loadSubnetDropGates). The window is
re-opened to ≤5 min on user-driven AttachInternetGateway, which is
the original behaviour pre-mulga-siv-163. A follow-up bead can close
that gap properly by converting the gate/ungate subjects into a
single recompute trigger that re-reads effective RT state.
* fix(vpc): prevent duplicate main route table from bootstrap race
Concurrent multi-node daemons running EnsureDefaultVPC for the admin
account (bootstrap-pinned VpcId) each minted a fresh random rtb-ID and
Put it with IsMain=true. The KV ended up with two main route table
records for the same VPC.
CreateRoute's post-write gate recompute (per-subnet egress drop gating)
then resolved effectiveRouteTable non-deterministically across the two
IsMain=true records; when the orphan (route-less) RT won, the gate
decision became "no default egress" and vpc.gate-subnet-egress fired.
The subscriber installed a Priority-1100 DROP on the default subnet's
egress, killing the return path for guest SSH probes
(TestMultinodeGuestSSH connect timeout to 192.168.0.161 in run
26699045536).
Two-part fix:
1. createMainRouteTable scans rtbKV for an existing IsMain=true record
on the same VPC before Put-ing a new one. Race window narrows from
"always duplicates under concurrent bootstrap" to a sub-ms TOCTOU
between scan and Put.
2. effectiveRouteTable / mainRouteTable use a deterministic preferMain
tiebreak when multiple IsMain=true records exist: more routes >
oldest CreatedAt > smaller RouteTableId. Defensive — any duplicate
that slips through (cross-AZ restore, manual KV edit, residual race)
still produces consistent egress gate decisions instead of flapping
on KV scan order.
* fix(policy): atomic ACL replacement for SG mutations
Add ovn.Client.ReplaceACLs — detach + delete existing ACL rows and
attach freshly-created rows in one OVSDB transaction. EnsureSG and
UpdateSG now funnel through ReplaceACLs instead of ClearACLs +
AddACLs, closing the mid-flight window where the port group held
zero ACLs and tenant LSP traffic defaulted to drop (observable on
ICMP and on TCP SYNs that miss conntrack).
Mock client exposes ClearACLCalls / ReplaceACLCalls counters so
tests can assert atomicity. New TestSGManager_UpdateSG_AtomicReplace
fails if SG mutations ever fall back to the split path.
|
||
|
Josh Siviour
|
287487a4f2 |
feat(elbv2): implement ModifyListener (#268)
* feat(elbv2): implement ModifyListener
Adds AWS-compatible ModifyListener to the spinifex ELBv2 stack so
listeners can be edited in place — port, protocol, and default
actions — without the current delete+create dance that briefly drops
the listener from HAProxy.
- Handler interface, impl, and NATS wrapper get ModifyListener;
validation mirrors CreateListener (LB type ↔ protocol, dup-port
excluding self, target-group existence + protocol compatibility).
- Daemon subscribes to elbv2.ModifyListener on the spinifex-workers
queue and a new gateway action dispatches it.
- Frontend exposes a per-row edit button that pre-fills a dialog from
the existing listener and calls ModifyListenerCommand, replacing
the "Listeners cannot be edited" note.
- Unit + NATS round-trip + frontend tests cover the new flow.
Cert / SslPolicy / Alpn handling stays out of scope and lands with
HTTPS listener support.
* test(e2e): cover ModifyListener in lb harness
Adds an Internal_ALB_ModifyListener subtest to TestLoadBalancer that
exercises in-place listener edits end-to-end:
- Provisions tgA + tgB sharing the same app instances.
- Brings up an internal ALB with HTTP:80 → tgA, baselines round-robin.
- ModifyListener port=8090, asserts DescribeListeners shows the new
port and that probes on 8090 still round-robin — proves HAProxy is
reconciled without dropping the listener (the regression mulga-944
fixes; the frontend previously delete+created on every edit).
- ModifyListener DefaultActions → tgB, asserts traffic still flows.
Also factors a probeAtPort helper so the order JSON can target any
listener port, and updates TEST_COVERAGE.md with phase 3e.
* fix(ui): widen listener add/edit dialog so content stays inside the box
AlertDialogContent defaults to max-w-xs (20rem) which clipped the listener
form on the right edge. Use a 32rem cap with grid-cols-[minmax(0,1fr)] +
min-w-0 on the form so CSS grid tracks shrink instead of expanding to
max-content.
* test: cover ModifyListener + system instance dispatch
Lifts ModifyListener diff coverage by exercising nil/empty arn paths,
NLB protocol acceptance + rejection, protocol-only revalidation against
existing target groups, and the launcher-nats wire format error paths
(missing input, missing instance type, decode failures, no responder,
default timeout fallback).
Adds a new spinifex/daemon/daemon_system_dispatch_test.go covering
splitSubjectTail edge cases, respondWithSystem{Launch,Terminate}{Output,
OK,Error} envelope shape, handler entry points for invalid JSON and
missing instance ID, and subscribeSystemTerminate nil-conn + idempotent
re-subscribe behaviour.
Fixes the flaky TestHandleEC2DescribeInstanceTypes/NoFilter… subtest by
selecting a 2-vCPU instance type that also fits schedulable memory, so
hosts with smaller schedulableMem (CI 4 vCPU / ~13 GiB) don't pick a
16 GiB-mem type that fails allocation.
* test(e2e): wait for tgB after listener swap, not before
Spinifex health-checks only target groups wired into a listener's
HAProxy config. Waiting for tgB to report healthy before
modifyListenerDefaultTG attaches it to the listener is impossible —
the agent never probes its targets, so they stay in initial and the
2m EventuallyErr trips.
Drop the pre-swap tgB wait. After modifyListenerDefaultTG, replace
the fixed 3s sleep with WaitForTargetsHealthy on tgB (post-swap),
which is now the listener's DefaultAction.
* test(e2e): derive LB name from ARN in lbENI/deleteLB ENI filter
The ENI description filter was building the LB-name token from a
hard-coded "alb"/"nlb" template (e.g. `ELB app/lb-e2e-alb-int/<id>`),
which only matched LBs whose name happened to be `lb-e2e-alb-*` or
`lb-e2e-nlb-*`. The ModifyListener suite names its LB `lb-e2e-mod`,
so lbENI's EventuallyErr could never find the ENI and tripped after
30s. deleteLB had the same bug: sysInstanceID stayed empty and
WaitForENICleanup waited for a non-existent filter, allowing the
next suite's createLB to race the sys.micro VM reclaim.
Parse the LB name straight out of lb.ARN (penultimate path segment,
same shape createLB already relies on) so the filter matches any
caller-chosen name. Drop the now-redundant scheme arg from lbENI.
* test(e2e): poll for listener serving on new port after ModifyListener
The post-modifyListenerPort settle was a hard 3s sleep, but lb-agent
polls for new HAProxy config on a 5s tick. After the API returns, the
agent has typically not yet detected the config-hash change, so
HAProxy is still bound to the old port — every probe to the new port
RSTs and probeAtPort sees 0/0 successful.
Replace the sleep with waitForListenerServing: a polling loop that
fires 2 probes via the shared trigger client every 2s until at least
one returns a parsable instance_id, with a 60s ceiling. Factor the
trigger-and-fetch out of probeAtPort into probeOnce so the polling
helper can reuse it without the require.* asserts.
|
||
|
Josh Siviour
|
050ac8f861 |
feat(e2e): Go E2E harness + single-node migration to Ensure*/Discover* + reboot suite (#241)
* feat(e2e): add Go test harness + cert scenario, port run-cert-e2e.sh
Lands step 1 of docs/development/improvements/e2e-go-harness.md:
shared harness primitives (env, wait, artifacts, cert, tls) plus the
TestCertIssuance scenario that supersedes run-cert-e2e.sh. All files
gated by //go:build e2e so default builds and `make preflight` are
unaffected. Bash script stays in place for one CI cycle before deletion.
* feat(e2e): port run-lb-e2e.sh to TestLoadBalancer Go scenario
Step 3 of docs/development/improvements/e2e-go-harness.md. Replaces 1246
lines of bash with a typed Go scenario exercising all 4 LB variants
(ALB/NLB × internet-facing/internal) against a shared VPC + app instance
pool. Internet-facing phase auto-skips when no peer node is configured;
internal phase always runs.
Adds shared harness primitives:
- harness/aws.go: AWS SDK client factory wired to Spinifex AWS gateway
- harness/lb.go: WaitForLBActive, WaitForTargetsHealthy, WaitForENICleanup,
WaitForInstanceRunning/Terminated, InstanceENI
- harness/traffic.go: HTTPRoundRobin, TCPRoundRobin, distribution asserts
- harness/ssh.go: minimal peer SSH wrapper shelling out to /usr/bin/ssh
(replaced by typed x/crypto/ssh client hoisted from DDIL in mulga-siv-74)
User-data scripts embedded via //go:embed under lb/testdata/.
Bash script stays in place for one CI cycle before deletion.
* fix(e2e): log client launch params + dump daemon log on InvalidInstanceType
Adds diagnostics for the LB internal client launch path. Logs the exact
InstanceType, AMI, subnet, key name, and user-data size right before
RunInstances, and registers a daemon-log dump on failure so we can
correlate the 400 InvalidInstanceType with what spinifex saw.
* refactor(e2e): serialise LB suites + share probe client across internal tests
Splits TestLoadBalancer into 4 sequential subtests (ALB/NLB ×
internet-facing/internal). Each suite creates its own TG + LB + listener,
runs its traffic test, and tears down before the next starts. Replaces
the per-suite throwaway client VM with a single long-lived probe server
launched at fixture setup; per-suite internal traffic tests GET
/probe?ip=...&proto=... against it instead of cold-booting a fresh client.
Drops peak instance count from 5 to 4 (2 app + 1 LB + 1 client), so the
internal phase passes on capacity-constrained dev nodes that previously
hit InvalidInstanceType (see mulga-siv-76, mulga-siv-77 for the underlying
spinifex bugs).
* feat(daemon): make hostReserve env-configurable for dev capacity squeeze
Adds SPINIFEX_RESERVED_VCPU and SPINIFEX_RESERVED_MEM_GB env-var overrides
read at daemon startup. Invalid values are logged and ignored so a typo
doesn't silently widen the reserve. Unblocks the LB E2E on capacity-tight
dev hosts (8 vCPU box: default 2-vCPU reserve + 3x t3.nano test load
leaves 0 free for the sys.micro LB VM). Lays groundwork for the planned
operator-tunable [capacity] config.
bead: mulga-siv-78
* fix(e2e): revert LB to per-suite client + fail-fast on terminal LB state
Reverts the shared probe-server VM that wouldn't come up reliably during
the InternalFacing suites — back to the proven bash pattern of launching
a fresh client VM per suite (one of ALB_IP / NLB_IP populated based on
the suite kind). Adds renderClientUserData + a conditional template so
each client probes only its target LB.
WaitForLBActive now bails immediately on terminal LB states (failed,
provisioning_failed) and emits a "still waiting" log line every 30s so
hangs are visible instead of silently burning the full 5-minute timeout.
* ci(e2e): add e2e-go.yml workflow + GOTESTSUM Makefile opt-in
Standalone workflow runs the new Go-based E2E harness on the existing
ci-single self-hosted runner — provisions a tofu VM, bootstraps spinifex,
installs gotestsum, runs each requested suite via the e2e Makefile, and
uploads per-test artifacts + JUnit XML + JSON streams as a build artifact.
Test summary parses the JUnit and emits ::error file=...:: annotations so
failures surface in the PR diff.
Gated to the feat/e2e-go-harness branch + workflow_dispatch so it doesn't
interfere with the existing e2e.yml while the Go conversion lands.
tests/e2e/Makefile gains a GOTESTSUM=1 opt-in: when set, the e2e target
runs gotestsum with --junitfile and --jsonfile rooted in ARTIFACT_DIR
under a suite-tagged filename derived from the first PKG component.
* feat(e2e): share probe client across both internal LB suites + SG fixes
- Replaces per-suite probe client with a single shared VM launched in the
fixture. Saves ~60s by avoiding a cold boot between the ALB and NLB
internal suites.
- Client cloud-init exposes a JSON trigger endpoint on :9090 in addition
to the http.server on :80 (results dir). Each suite POSTs
{proto, ip, count, outfile, http_port, tcp_port} to run a probe loop
against the LB private IP and writes the result file under /tmp/httpd
for the driver to fetch back.
- configureDefaultSG now uses the structured IpPermissions form (top-level
shortcut is silently ignored by vpcd — tracked separately) and opens
port 9090 in addition to 80 + 9000.
- deleteLB captures the underlying sys.micro VM id via the LB ENI before
delete, then polls DescribeLoadBalancers until LoadBalancerNotFound so
the next suite doesn't race the capacity reclaim.
- Internal_NLB sleeps 15s up front to let the ALB sys.micro deallocate
fully before its own createLB fires. DescribeLoadBalancers can report
the ALB gone before the vCPU/memory allocation is actually released on
capacity-tight dev hosts; without this settle, NLB's createLB trips the
sys.micro reserve and lands in terminal state=failed. Still cheaper
than cold-booting a per-suite client.
* ci(e2e): install Go on tofu VM before running Go E2E suites
The tofu-provisioned VM ships with the spx binary and bash test scripts
but no go toolchain, so the prior workflow's "Install gotestsum on VM"
step failed with `bash: line 2: go: command not found`. Add a curl-based
Go 1.26.3 install (matching go.mod) ahead of the gotestsum install and
update the test step's PATH so make e2e-* runs the freshly-installed
go binary.
* ci(e2e): bump gotestsum to @latest for Go 1.26 compatibility
gotestsum v1.12.0 pins golang.org/x/tools v0.19.0 which fails to compile
under Go 1.26 with:
invalid array length -delta * delta (constant -256 of type int64)
Switching to @latest pulls a newer x/tools that builds clean.
* ci(e2e): ship spinifex source to VM so make -C tests/e2e can run there
Single-node tofu VM only has the spx binary + bash test bundle at
~/e2e-tests; the Go e2e workflow was running 'make -C tests/e2e e2e-cert'
from $HOME/Development/mulga/spinifex which doesn't exist, producing:
make: *** No rule to make target 'e2e-cert'. Stop.
Add a checkout of the spinifex repo on the runner and rsync it (minus
.git, build/, bin/, node_modules) to the VM as
~/Development/mulga/spinifex before installing Go + gotestsum.
Interim solution; mulga-siv-80 tracks replacing the rsync with shipping
pre-built test binaries from the runner.
* ci(e2e-go): replace rsync with tar-over-ssh; VM lacks rsync
bootstrap-install.sh ships a minimal binary VM with no rsync; the
ship-source and collect-artifacts steps failed with
'rsync: command not found'. Stream a tarball over ssh instead — same
exclude list, no extra dependency on the VM.
* fix(e2e): discover single-node IPs + awsgw bind in LoadEnv
run-cert-e2e.sh single mode probes awsgw.host from spinifex.toml and
adds every non-loopback global IPv4 to the node list. The Go harness
defaulted to 127.0.0.1 only, so CI on a tofu VM (awsgw bound to the
LAN IP, not loopback) saw 'connection refused' on every TLS check.
Port the same discovery: ModeSingle now walks net.InterfaceAddrs for
NodeIPs and grep-parses spinifex.toml for the awsgw bind. Both stay
overridable via SPINIFEX_NODE_IPS / SPINIFEX_SERVICE_IPS.
* ci(e2e-go): pre-build test binaries on runner; drop go toolchain from VM
Replaces the source-tree + go + gotestsum payload that the e2e-go.yml
workflow previously shipped to the tofu CI VM with self-contained
test binaries built on the runner via `go test -c`. testdata for the
LB suite is already //go:embed'd into the binaries, so the bundle on
the VM is just `<suite>.test` files (cert: 14M, lb: 17M uncompressed).
Workflow changes
- actions/setup-go on runner.
- New step builds binaries via `make e2e-build SUITES_BUILD=...`.
- Install go-junit-report on runner; convert each VM-side test log to
junit-<suite>.xml after artifact pull so "Publish test summary" still
parses JUnit XML.
- Drop "Install Go + gotestsum on VM" entirely (saves ~60s + network).
- Drop source-shipping rsync/tar (saves ~30s + a lot of bandwidth).
- VM step runs `./<suite>.test -test.v -test.timeout=30m` directly.
Makefile changes
- New `e2e-build` target compiles each `SUITES_BUILD` entry into
`BIN_DIR/<suite>.test` via `go test -c -tags=e2e`. `clean` also
removes BIN_DIR.
Closes mulga-siv-80.
* ci(e2e-go): disable setup-go cache; upload dominates self-hosted run time
* refactor(e2e): hoist generic DDIL harness primitives to shared tests/e2e/harness
Step 1.5 of docs/development/improvements/e2e-go-harness.md. Promotes
Cluster/Node/SSH/DaemonClient/LocalStateRevision/ResetAllNodes/Witness
into the shared package so the LB, multinode, and BM scenarios in
steps 4-5 can build on the same primitives instead of duplicating SSH
+ cluster plumbing.
Moves
- ddil/harness/{cluster,ssh,daemon_client,state,reset,witness}.go
-> harness/ (plus the witness testdata embed dir).
- ddil/harness/{fault,netem,profiles,snapshot}.go -> ddil/fault/ as a
new package; fault.PartitionNode/HealNode/KillNATS/StartNATS/
RestartDaemonOnly/ApplyNetem/ClearNetem/LinkProfile*/TakeSnapshot
now take harness.Cluster / harness.Node / harness.SSH explicitly.
- ddil/harness/retry.go::Run retry-with-reset extracted as
harness.RetryWithReset in harness/wait.go; ddil/harness/quarantine.go
keeps the DDIL-only quarantine wrapper that calls into it.
- WaitForMode moved from fault.go to harness/daemon_client.go since
it polls daemon state and has no fault-injection concern.
- The pre-hoist harness/ssh.go (PeerSSH, subprocess ssh) is preserved
as harness/peer_ssh.go so the lb_test suite keeps working unchanged;
the typed x/crypto/ssh client from DDIL now owns harness/ssh.go.
- New harness/util.go exports ShellQuote (was unexported shellQuote in
netem.go) so both ddil/fault and reset.go/witness.go can use it.
Env var rename
- SPINIFEX_NODES / SPINIFEX_SSH_USER / SPINIFEX_SSH_KEY replace
DDIL_NODES / DDIL_SSH_USER / DDIL_SSH_KEY. Old names still honoured
with a stderr deprecation note (envWithLegacy) so the DDIL CI keeps
passing until .github/workflows/e2e-ddil.yml drops the old names.
DDIL_QUARANTINED stays DDIL-specific.
DDIL parity preserved — scenarios still call ddilh.Run for quarantine
semantics; the underlying retry-with-reset is the same code path,
just shared.
Closes mulga-siv-74.
* fix(e2e): open default SG :22 ingress via structured form before witness launch
The daemon currently drops the top-level shortcut AuthorizeSecurityGroupIngress
form silently (mulga-siv-79), so the default SG starts with no inbound :22 and
the witness EIP awaitBaseline tunnel times out under DDIL CI. Mirror the
workaround already present in lb_test by issuing the structured IpPermissions
form before LaunchWitnessVM. Idempotent on InvalidPermission.Duplicate so reruns
in the same cluster are safe.
* ci: render JUnit into job summary + collapsible groups in noisy steps
Add a reusable composite action (.github/actions/junit-report) that parses
a glob of JUnit XML files, writes a totals table + collapsible per-failure
details into $GITHUB_STEP_SUMMARY, and emits one ::error annotation per
failed/errored test case so failures surface in the run header instead of
being buried in step logs.
Apply the action to both e2e-go.yml and e2e-ddil.yml (e2e-ddil gotestsum
now also emits --junitfile so the action has input).
Same workflows also wrap noisy phases (tofu apply/destroy, scrub-env,
tar-over-ssh, per-suite test runs, cluster-log collection) in
::group::/::endgroup:: markers so the Actions UI collapses the chatter
by default and the human can expand only the failing phase. Tofu output
still tees to $TOFU_LOG so the artifact bundle is unchanged.
Future workflows (multinode, baremetal) can adopt the same pattern via
the README in .github/actions/junit-report.
* test(ddil): quarantine Scenario C until predastore §2 lands
CI run 26016840819 surfaced that ScenarioC_CleanPartition silently
depends on predastore-ddil-hardening §2 (write-quorum with deferred
parity repair). Root cause from node3 daemon log:
predastore: write udp ...->192.168.0.22:9991: sendmsg: operation not permitted
predastore: putObjectViaQUIC: put failed
spx QMP: BLOCK_IO_ERROR instanceId=<witness>
spx: StopOrTerminateInstance: Terminating
spx: Instance state transition running → shutting-down
The 'sendmsg: operation not permitted' is the EPERM Linux returns when
iptables drops an outbound packet — exactly the OUTPUT DROP rules
PartitionNode installs. The witness VM's block I/O is backed by
distributed predastore, so the same partition that severs node3↔peer
NATS traffic also severs the witness disk's cross-shard QUIC writes.
QEMU power-downs the guest after BLOCK_IO_ERROR, the daemon's
reconciler terminates the instance, and the next AssertProgressed dials
a sshd that's already gone (hence the 'handshake failed: EOF' the test
reports).
Quarantine by collapsing C to the same scenarioSkip stub D/E/F already
use, citing predastore-ddil-hardening §2 as the dependency. Flip C
back to ENABLED in the same PR that lands §2 — that's when the witness
disk can survive a peer partition without QEMU halting.
TEST_COVERAGE.md row C flipped ENABLED → SKIPPED; added a quarantine
note explaining the predastore dependency.
* ci: colored phase banners + quiet-on-success tofu output
Pretty up the live Actions log without flooding it. Three changes:
1. .github/scripts/ci-fmt.sh — sourceable bash helpers (banner, ok, bad,
warn, note, quiet_run) using ANSI colors that the Actions UI renders.
2. e2e-go.yml + e2e-ddil.yml — tofu phases switch back to quiet redirect
into $TOFU_LOG (the previous tee'd version regressed live noise vs
the original silent redirect). Each phase now prints one colored
ok/✗ line; on failure the tail of $TOFU_LOG dumps inside a single
collapsed group, so the live log stays scannable but the failing tail
is one click away.
3. junit-report render.py — compact summary: short status icons (✅/❌)
in a single column, no separate err column, single-line failure list
(truncated to 120 chars) instead of <details> blocks that wrap badly
on narrow screens.
Per-suite test runs still wrap in ::group:: with an inline cyan banner
so pass output collapses and the suite name is obvious when expanded.
* test(e2e): add harness.Phase / Step / Detail for human-readable test output
Test fixtures were emitting flat walls of t.Logf with every line prefixed
by the same `lb_test.go:61:` (file:line of the call site that the
t.Helper() chain bubbles up to) and forced 6-space indent. Hard to skim,
hard to find where one phase ends and the next begins.
Add three helpers in tests/e2e/harness/log.go:
Phase(t, fmt, ...) — flush-left colored banner via fmt.Println.
Bypasses t.Logf so the line lands without the
testing framework's indent/file:line prefix.
Pure visual structure; the diagnostic detail
stays on t.Logf.
Step(t, fmt, ...) — single-line sub-phase marker via t.Logf.
Keeps file:line attribution so a triager can
find the source of a progress message.
Detail(t, k, v, ...) — key=value resource IDs via t.Logf. Same
attribution rationale as Step — IDs are
diagnostic and want to land in JUnit XML
bucketed with the owning test.
Why the split: Phase is purely a banner; losing JUnit attribution costs
nothing because banners aren't diagnostic. Step/Detail carry triage
context (instance IDs, AMI, etc.) and a failing test's JUnit testcase
node should still contain them, so t.Logf attribution is kept.
ANSI colors are auto-disabled when stdout isn't a TTY and GITHUB_ACTIONS
isn't set, so piped local `go test -v` stays clean.
Apply to lb_test setupSharedFixture — five Phase banners replace the
implicit run of helper logs ("Discovering cluster capacity",
"Ensuring SSH key pair", "Creating shared VPC topology",
"Launching app instances", "Launching probe client") so the live CI log
reads as a story rather than a flat resource dump.
* ci(e2e-go): strip indent + file:line on display, propagate color env over SSH
Two display bugs spotted on the previous run:
1. Phase banners came out white instead of bold-cyan because the test
binary runs over SSH on the tofu VM, where neither GITHUB_ACTIONS
nor a TTY is present, so harness.log.go init() disables ANSI escapes.
Fix: export GITHUB_ACTIONS=true and FORCE_COLOR=1 in the remote env
so the helper keeps emitting color codes.
2. Every t.Logf line still rendered as
lb_test.go:61: instance type: t3.nano
(4-space indent + `<file>:<line>:` prefix) because t.Helper()
bubbles file:line to the call site. Fix: pipe the test binary's
stdout through `sed -E 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'`
so the display strips both. Tee to the artifact log happens BEFORE
the sed so raw output is preserved for triage; only the live UI
sees the cleaned form.
* feat(e2e-harness): extend harness for single-node Go suite port
Stage A of mulga-siv-89 — add the primitives the single-node run-e2e.sh
port consumes:
- AWSClient gains IAM + STS service handles, plus NewAWSClientWithCreds
for per-tenant account-scoping in Phase 8.
- awserror.go: ErrorCodeIs, AssertAWSError, ExpectError — Go equivalent
of the bash expect_error helper used throughout the negative phases.
- poll.go: WaitForInstance/Volume/Snapshot/Image state to any target,
with per-resource default timeouts tuned from the bash sleep loops.
- spxcli.go: thin shellouts for spx get nodes/vms, spx top nodes,
spx admin account create with output parsing for tenant-id/access-key.
- rawhttp.go: signed raw POST against the AWS gateway for the Phase 8
InvalidAction probe (gateway rejects on auth before action parsing).
- ovn.go: OvnNbctl/OvnSbctl wrappers + SkipIfNoOVN for Phase 8b-e
datapath checks; WaitForPortGroupMember handles propagation polls.
- ec2helpers.go: InstancePublicSSHHost (public-IP-or-qemu-hostfwd),
LsblkRootGiB, serial-console-access trio, DiscoverDefaultVPC,
idempotent AuthorizeSSHIngress.
- account.go: AccountCarousel maps tenant name -> scoped AWSClient,
built atop NewAWSClientWithCreds for the Phase 8 isolation tests.
Pure groundwork — no new tests, no production behaviour change. The
cert + lb scenarios still compile against the extended harness.
* ci(e2e-go): auto-cancel + reap orphans to prevent runner wedge
A cancelled run on the single-runner ci-single host has been leaving
orphan tofu/terraform procs (and stale tfstate locks) behind, which
caused the next push's tofu destroy to hang for 45m+ and saturate the
runner host until sshd refused new connections.
- Add `concurrency: e2e-go-${ref}, cancel-in-progress: true` so a new
push to feat/e2e-go-harness cancels the in-flight run instead of
stacking on the same self-hosted runner.
- New `Reap Orphan Procs` step at job start: SIGTERM then SIGKILL any
lingering `tofu`/`terraform` from the runner user, and unlink any
stale `.terraform.tfstate.lock.info`.
- Harden `Teardown`: drop `set -e` (scrub-env must always run as the
libvirt-level fallback), reap orphans before destroy, and wrap
`tofu destroy` in `timeout 600` so a wedged provider can't pin the
job to its 45m timeout.
* ci(e2e-go): drop push trigger, manual dispatch only
Pull the on.push branch trigger so the workflow only fires from
workflow_dispatch — matches the e2e-ddil.yml Phase-1 pattern. Push
trigger lifts again once the single-node Go port (mulga-siv-89) is
green end-to-end and run-e2e.sh is retired.
* test(e2e-single): port run-e2e.sh phases 1-4 to Go
Stage B of mulga-siv-89. Lands the TestSingleNode orchestrator + Fixture
that threads state between sequential phase subtests, plus phases:
- Phase 1: KVM check, AWS gateway TLS reachability, daemon NATS
readiness polling, region/AZ smoke.
- Phase 1b: spx get nodes / top nodes / get vms (single-node only).
- Phase 2: nano instance-type + arch discovery.
- Phase 2b: serial-console-access default/enable/disable round-trips.
- Phase 3: create-key-pair + import-key-pair lifecycle; test-key-1 PEM
kept for Phase 5 instance launch (cleanup deferred to teardown phase
in stage G).
- Phase 4: architecture-aware AMI discovery via name filter
ami-ubuntu-24.04-{x86_64,arm64}, falls back from spx import output.
Phase ordering is contractual (Phase 5+ depend on AMI, key pair, AZ).
detectPoolMode reads [network] external_mode from spinifex.toml so
stages F/G can skip pool-only phases on dev_networking fixtures.
No production code changed. cert + lb scenarios continue to compile.
* test(e2e-single): port run-e2e.sh phases 5-5e to Go
Stage C of mulga-siv-89 — the instance/volume/snapshot/image lifecycle
that everything else hangs off:
- Phase 5: run-instances (default VPC + default SG with tcp/22 opened
via idempotent harness.AuthorizeSSHIngress), wait running, resolve
root volume from BlockDeviceMappings.
- Phase 5a-pre: spx get vms now contains the running instance.
- Phase 5a: describe-instances metadata round-trip (type/key/AMI/BDMs).
- Phase 5a-ii: TCP wait, ssh id, lsblk cross-check vs API root volume
size (±1 GiB tolerance — lsblk rounds down), soft hostname check.
- Phase 5a-iii: get-console-output round-trip on InstanceId.
- Phase 5b: create-volume (10 GiB) → modify-volume (20 GiB) →
attach-volume → detach-volume → delete-volume, polling each state
transition through harness.WaitForVolumeState.
- Phase 5b-ii: describe-volume-status on the root volume.
- Phase 5c: create-snapshot → wait completed → copy-snapshot → both
visible → delete original → copy survives → delete copy.
- Phase 5d: AMI snapshot-backed launch verification via
DescribeImages.BlockDeviceMappings[].Ebs.SnapshotId. Predastore-side
config.json reads are TODO (no S3 client in harness yet).
- Phase 5e: create-image --no-reboot, wait available, capture backing
snapshot ID for Phase 9 cleanup.
Phase 7a (attach to stopped → error) stubbed in volume_test.go for
Stage D to fill once the lifecycle stop/start lands.
* test(e2e-single): port run-e2e.sh phases 5f-8 to Go
Stage D of the single-node Go port (mulga-siv-89). Adds four files
under tests/e2e/single/ covering Phase 5f (SG egress ACL enforcement),
Phase 6 (tag CRUD), Phases 7 / 7a / 7b / 7c-pre / 7c (stop, start,
attach-to-stopped negative, ModifyInstanceAttribute, reboot, multi-count
RunInstances), and Phase 8 negatives (18 sub-tests asserting AWS error
codes via harness.ExpectError / AssertAWSError / PostAWSAction).
Wired into TestSingleNode in main_test.go; phase ordering preserved
because Phase 7 chain mutates the shared instance state (stopped at end
of 7, .small after 7b, running after 7c-pre, untouched by 7c siblings)
and Phase 8 negatives need a running instance with the AMI/key created
earlier.
Removes the placeholder phase7a_AttachToStoppedError stub from
volume_test.go (real impl now in lifecycle_test.go).
run-e2e.sh stays in tree until step-7 cleanup per the parent plan.
* ci(e2e-go): wire e2e-single target + add single to build set
Lets the in-progress single-node Go port (mulga-siv-89, Stages A-D
landed) be exercised in CI via explicit `suites: e2e-single` dispatch
input ahead of Stage G flipping the workflow default.
Workflow default suites unchanged until Phase 9 teardown lands.
* test(e2e-single): port run-e2e.sh IAM 1-7 + Phase 8 account scoping to Go
Stage E of the single-node Go port (mulga-siv-89). Adds two files
under tests/e2e/single/:
- iam_test.go (903 lines): IAM Phases 1-7 — User CRUD, Access Key
Lifecycle, User Authentication (via harness.NewAWSClientWithCreds),
Policy CRUD (+ CreatePolicyVersion / SetDefaultPolicyVersion /
DeletePolicyVersion which bash lacks), Policy Attachment &
Enforcement (allow / deny / wildcard / root bypass), Policy Lifecycle
(DeleteConflict while attached), and idempotent Cleanup.
- account_test.go (1186 lines): Phase 8 EC2 Account Scoping — 11 Steps
(Account Setup, Instance / Volume / KeyPair / Snapshot / VPC-Subnet /
IGW-EIGW Scoping, Account Settings, Global Resources, Edge Cases,
Cleanup). Uses harness.AccountCarousel for the multi-profile
credential map and SpxAdminAccountCreate for admin account
bootstrap.
Fixture extended with IAM cross-phase state (IAMAdminAccount,
IAM{Alice,Bob,Charlie}{KeyID,Secret}) so Phase 7 cleanup is defensive
even when an intermediate IAM phase short-circuits. All 11 Phase 8
Steps share local state in the outer function; t.Cleanup mirrors
Step 11 LIFO so a mid-Step failure still tears down accounts.
Wired into TestSingleNode. run-e2e.sh stays in tree until step-7
cleanup per parent plan.
* fix(e2e-harness): use kebab-case is-default filter for DescribeVpcs
AWS EC2 DescribeVpcs filter names are kebab-case (is-default), and the
daemon's filter allowlist enforces that. The harness sent the camelCase
SDK field name (isDefault), so the daemon returned InvalidParameterValue
400 and DiscoverDefaultVPC fataled — leaving Phase 5 InstanceID empty
and cascading 16 downstream Single-Node E2E subtests to fail.
* test(e2e-single): port run-e2e.sh phases 8b-8e to Go
Stage F of the single-node Go port (mulga-siv-89). Adds three files
under tests/e2e/single/:
- vpc_test.go (553 lines): Phase 8b (pool-mode VPC public/private
subnet E2E — CreateVpc, dual subnets with MapPublicIpOnLaunch
assertions, IGW attach, custom route table + 0.0.0.0/0 route,
instance launch + SSH + external connectivity probe) and Phase 8c
(route table validation — main RT association, local + IGW route
presence, custom RT CRUD + association/disassociation, negative
delete-local + delete-main assertions).
- natgw_test.go (467 lines): Phase 8d (pool-mode NAT GW E2E —
CreateNatGateway with EIP, private subnet route via NAT, bastion-
hopped SSH into private VM, baseline egress drop / post-NAT egress
succeed / post-teardown egress lost). Inline waitForNATGatewayState
helper bounds the pending->available->deleted state machine via
harness.EventuallyErr.
- datapath_test.go (326 lines): Phase 8e (OVN-gated SG-to-SG datapath
— two SGs with UserIdGroupPair ingress rule, OVN port-group
membership + SB address-set wait loops, allowed traffic curl,
denied traffic nc, revoke->drop, re-add->restore).
Pool-mode gating via fix.PoolMode for 8b/8d; OVN gating via
harness.SkipIfNoOVN for 8e. Per-phase t.Cleanup LIFO ensures
teardown unwinds even on partial failure. Wired into TestSingleNode.
* test(e2e-single): port run-e2e.sh phases 9, 9a, 9b to Go
Stage G of the single-node Go port (mulga-siv-89) — final teardown +
post-teardown verification. Adds tests/e2e/single/teardown_test.go
(226 lines):
- phase9_Teardown: DeleteSnapshot(CustomAMISnapID) -> DeregisterImage
(CustomAMIID) -> TerminateInstances(InstanceID) + wait terminated ->
DeleteKeyPair(KeyName). Each call swallows *NotFound via
harness.ErrorCodeIs; unexpected errors logged as warnings so 9a/9b
always run.
- phase9a_VerifyTeardown: DescribeInstances must show terminated or
NotFound; DescribeImages must NOT include the deregistered AMI;
DescribeKeyPairs must NOT include the deleted key. Clears the
consumed Fixture fields.
- phase9b_FinalClusterStats: harness.SpxGetVMs must NOT list the
terminated InstanceID; orphan i- rows surface as soft Logf warnings
to avoid flakes when parallel suites share the node.
Wired into TestSingleNode. With this, mulga-siv-89's single-node Go
port (Stages A-G) is feature-complete; run-e2e.sh remains in tree
until a separate cleanup bead deletes it after CI burn-in.
* ci(e2e-go): unbuffer sed in test pipeline for live log streaming
GNU sed defaults to block-buffered stdout when stdout is a pipe
(GitHub Actions runner). Accumulates 4-64KB before flushing, so
test output appeared in sporadic batches unaligned to test
boundaries. -u flag forces line-buffered output so each go test -v
line surfaces in the Actions UI as it is produced.
Side effect: none. No process count change, no SSH session change,
no test execution change. Pure output flushing on the runner host.
* fix(e2e-single): persist key PEM across phases and skip unimplemented IAM ops
Three bugs blocked the Single-Node Go E2E suite after the DescribeVpcs
filter fix:
1. Phase 3 wrote the SSH PEM into `t.TempDir()`, which Go cleans up when
the Phase 3 subtest exits. Phases 5a-ii / 5a-iii / 7b / 7c-pre then
saw "Permission denied (publickey)" / "Identity file not accessible"
and the cascade left the instance in a state that broke 7c-pre's
`start-instances`. Move the PEM under a TestSingleNode-scoped
`fix.TmpDir` so it survives until the whole test exits.
2. IAM Phase 2's `require.NotEmpty(t, fix.IAMAdminAccount, "")` was
intended as a no-op (Phase 4 populates the field), but require.NotEmpty
fails on empty regardless of message. Removed the bogus check.
3. The daemon does not yet implement `UpdateUser` (IAM Phase 1) or the
policy-version family (IAM Phase 4): `CreatePolicyVersion`,
`ListPolicyVersions`, `SetDefaultPolicyVersion`, `DeletePolicyVersion`.
They return `InvalidAction` and abort the phase. Wrap each block in a
t.Run sub-test with t.Skip so the surrounding phase still runs and
the skips show up explicitly in the test report.
* test(e2e): parallelise hot phases + tune fast-path polls
Squash merge of feat/e2e-parallel-quickwins (mulga-siv-96). Four
quick wins from decomposing run-e2e.sh into Go phase subtests:
- Phase 8 negatives (single/negative_test.go): t.Parallel() inside
each of the 18 read-only error-path sub-tests. ~5-10x speedup.
- IAM Phase 4 policies (single/iam_test.go): EC2ReadOnly stays
seeded sequentially (populates fix.IAMAdminAccount); the other
four policies fan out as siblings under a wrapping
create_policies_parallel t.Run. ~5x on the Phase 4 sub-step.
- Volume + snapshot fast paths (single/{volume,snapshot}_test.go):
pass harness.WithPoll(500ms) on the five state transitions that
finish in <1s on local QEMU. Halves Phase 5b / 5c wall-clock.
- Cert per-IP loop (cert/cert_test.go): t.Parallel() inside the 4
inner t.Run(ip, ...) blocks. Cuts cert phase by len(nodes).
No harness changes, no Fixture changes, no test-execution-order
changes. mulga-siv-96.
* test(e2e-lb): resolve peer availability before shared fixture setup
setupSharedFixture builds a VPC + 2 app VMs + probe client (minutes
on capacity-tight dev hosts) and previously ran BEFORE the peer
pickup + ssh.Ping. On single-node, the InternetFacing_* subtests
would correctly t.Skip on peer == "" — but only after the fixture
work and once subtests started, so the wait appeared to be attached
to the first subtest name gotestsum printed.
Reorder: pickPeer + ssh.Ping first, log "internet-facing will skip"
immediately, then build the fixture for the internal subtests.
No behavior change for multi-node + peer-reachable; on single-node
the skip rationale surfaces before the fixture wait.
* test(e2e-single): extend SSH wait budget to 3m on slow boot paths
CI run 26034322018 timed out Phase5a_ii_SSH (60s budget) and
Phase8b_VPCSubnetE2E (2m budget) on the TCP-port-open poll while
sshd was still finishing pam/cloud-init. Bump all four wait sites
to 3m as a stop-gap:
- single/instance_test.go phase5aii_SSHProbe
- single/lifecycle_test.go waitForSSHReady (Phase 7b/7c/7c-pre)
- single/vpc_test.go phase8b_VPCSubnetE2E
- single/natgw_test.go phase8d bastion SSH wait
Proper fix tracked under mulga-siv-101 (hoist a harness.WaitForSSH
that probes the actual handshake, not just tcp/22 reachability).
* test(e2e-single): skip-gate IAM3/IAM5/IAM6 until daemon handlers land
CI run 26034322018 surfaced three IAM daemon gaps:
- IAM3_UserAuthentication: alice's freshly-created access key returns
403 from DescribeInstances — daemon doesn't honour scoped IAM key
signatures yet.
- IAM5_PolicyAttachmentEnforcement: cascades from IAM3 (alice can't
sign).
- IAM6_PolicyLifecycle: DetachUserPolicy returns 404 even on
attachments Phase 5 confirmed — daemon attachment ledger gap.
Skip-gate each phase with t.Skip + a pointer to mulga-siv-100 so the
remaining IAM coverage (IAM1/IAM2/IAM4/IAM7) keeps surfacing
regressions, and the suite as a whole doesn't fail on known gaps.
Flip to real assertions once the daemon handlers land.
* test(e2e-single): probe full SSH handshake (not just TCP port open)
CI run 26036025293 hit "Connection timed out during banner exchange"
on Phase5a-ii / 8b / 8d even with the 3m TCP-port-open budget: tcp/22
opens immediately, but sshd is still finishing pam/cloud-init, so the
first real runSSH call afterwards races the half-open daemon.
Switch waitForSSHReady to issue an actual ssh BatchMode + ConnectTimeout
handshake against the target, retrying until the daemon completes the
banner exchange. Drop the four TCP-only Eventually loops in favour of
the upgraded helper:
- instance_test.go phase5aii_SSHProbe
- lifecycle_test.go waitForSSHReady (Phase 7b / 7c-pre / 7c)
- vpc_test.go phase8b_VPCSubnetE2E
- natgw_test.go phase8d_NATGateway (bastion)
Signature: waitForSSHReady(t, host, port, keyPath). Backwards-compat
alias waitForSSHHandshake exposed for the instance_test caller. Proper
harness hoist tracked under mulga-siv-101.
* test(e2e-harness): dual-emit Step + Detail for live CI streaming
go test's testing framework buffers t.Logf output per-test and releases
it only on PASS/FAIL. Long subtests (Phase8Acct_AccountScoping ~115s,
Internal_NLB ~60s, Phase8b_VPCSubnetE2E 2-3m, Phase8d_NATGateway ~90s)
therefore went silent in CI for the entire subtest duration, even with
sed -u line-buffering in the workflow pipeline.
harness.Phase already wrote to os.Stdout directly for this reason.
Extend the pattern: have Step and Detail dual-emit — write the same
line to os.Stdout (streams live) AND t.Logf (keeps JUnit per-test
attribution). The duplicate on a failure is acceptable; both copies
carry identical content.
mulga-siv-103 (note: numbering skipped past siv-102 which was already
filed in the parent branch).
* test(e2e-harness): drop t.Logf duplicate from Step + Detail
The dual-emit (Fprintln + t.Logf) caused every Step/Detail line to
appear twice after long subtests: once live via stdout, once again
when the testing framework released its per-test buffer at
PASS/FAIL. Most visible after lb subtests (Internal_NLB ~60s) where
the bulk release dumped ~dozens of duplicated lines at once.
Drop the t.Logf copy. go-junit-report parses stdout between
`=== RUN` and `--- PASS/FAIL` markers, so per-test attribution in
JUnit XML is preserved without the duplicate.
Refs: mulga-siv-103
* test(e2e-harness): stream lb-waiter progress live in CI
WaitForLBActive, WaitForTargetsHealthy, WaitForENICleanup, and
WaitForInstanceRunning emitted progress via t.Logf, which the
testing framework buffers per-test. After the previous commit
dropped the dual-emit, that meant every poll loop went silent
until subtest end — visible in CI as 'nlb is active' hanging on
screen for the whole duration of the subsequent traffic check.
Route progress through harness.Step (direct stdout). Also emit a
'waiting for X' Step at the start of each helper so observers
know what the silent stretches actually represent.
Refs: mulga-siv-103
* test(e2e): dump VPC/IGW datapath diagnostics on Phase8b SSH timeout
Phase8b's 3min SSH timeout on a fresh-VPC VM (CI run 26037892507) is
the customer-visible surface of mulga-siv-105 — vpcd's handleIGWAttach
returns before ovn-northd compiles + the gateway chassis installs
flows. Two other cleanup-only bugs (siv-106 missing spinifex:cidr,
siv-107 dnat_and_snat double-delete) showed up in the same triage.
Keep the test hard-failing so the regression stays visible, but dump a
triage bundle on timeout so each occurrence is actionable:
- guest console output (last 4KB — boot + cloud-init state)
- ovn-nbctl show + ovn-sbctl show
- Logical_Router external_ids (confirms siv-106 reproduction)
- NAT rules (confirms siv-105 + siv-107 surface)
- Port_Binding chassis,up (confirms gateway port up state)
Diagnostics go straight to stdout (Step/Detail rationale) so they
stream live in CI rather than dumping at PASS/FAIL.
Refs: mulga-siv-105, mulga-siv-106, mulga-siv-107
* ci(junit-report): drop parent t.Run containers from tally + list skips with reasons
GitHub Actions test summary was counting Go's top-level Test* parent
plus its failing subtest as two distinct failures (the parent is just
a rollup of its children). The summary now de-dupes by dropping any
test case whose name is a strict prefix of another's, so the per-file
counts and the Failures block both match the actual leaf-test
outcome. CI run 26037892507 was reporting '2 failed' for single when
it should have been 1.
Also adds a collapsible Skipped section listing each skipped leaf test
with its t.Skipf call-site reason (the .go:<line>: line from the
CDATA). ANSI escapes from harness.Phase banners are stripped so the
reason is plain text. Reasons are useful triage signal — current run
has 9 skips spread across daemon gaps (mulga-siv-100) and pool-mode
gates (InternetFacing_*).
Refs: mulga-siv-89
* ci(e2e-go): propagate cancel to remote .test procs + nuke VM before destroy
Two related leaks made cancelled runs strand QEMU work on the libvirt host:
1. The long `ssh tf-user@VM "for SUITE in …; do ./$SUITE.test"` ran
without -tt, so when the runner cancelled the step the local ssh
died but sshd never sent SIGHUP through (no PTY). The remote
`.test` binary orphaned and kept running until its own 30m
`-test.timeout`, holding tmp dirs, NATS connections, and qcow2 file
handles.
2. Teardown went straight to `tofu destroy` without first killing any
stragglers on the VM, so destroy raced the .test's open file handles
on the root disk — sometimes successfully, sometimes leaving the
libvirt host with a half-released qcow2.
Fix:
- Allocate a PTY on the run-suites ssh (`-tt`) so sshd HUPs children
when the local ssh dies.
- Add a trap (EXIT INT TERM) in the run-suites step that ssh's back
to the VM and pkill's `/spinifex-e2e-bin/*.test` belt-and-braces.
- Mirror the same pkill at the top of Teardown so destroy runs against
a quiesced VM.
Both cleanups are scoped to the per-run wan_ip — safe to land while
multiple ci-single runner agents run concurrent jobs against different
tofu envs on the same host (we deliberately keep that parallelism).
Refs: runner host wedged 2026-05-19 (post-cancel orphans suspected;
no journal access yet to confirm).
* ci(e2e-go): scope orphan reap to GITHUB_WORKSPACE so it spares sibling runners
Reap Orphan Procs (job start) and the in-Teardown pkill block were using
`pkill -u $UID_SELF -x tofu/terraform` plus `find $HOME -maxdepth 6 -name
.terraform.tfstate.lock.info`. The single host runs 5+ self-hosted runners
all as `tf-user`, so every e2e-go run was SIGKILLing in-flight tofu from
sibling workflows (e2e.yml, e2e-ddil.yml, e2e-baremetal.yml, ...) and
deleting their state-lock files — exactly the kind of cross-runner
collateral the reap was meant to recover from.
Scope by `/proc/$pid/cwd` starting with $GITHUB_WORKSPACE; each runner's
workspace dir is unique on the host, so only this run's own tofu procs
match. Find for stale lockfiles is likewise scoped to $GITHUB_WORKSPACE.
* ci(e2e-go): bump setup-go v5->v6, upload-artifact v4->v7
GitHub now warns Node 20 actions are deprecated. e2e-go.yml lagged the
rest of the repo; bump to match versions already in use:
- actions/setup-go@v5 -> @v6 (matches spinifex.yml)
- actions/upload-artifact@v4 -> @v7 (matches release.yml, e2e-ddil.yml)
checkout@v6 and create-github-app-token@v3 are already current across
all workflows. The local junit-report action is `using: composite`
(shell-only), so the Node runtime change does not apply.
* fix(vpcd): add flows-ready barrier after IGW attach (mulga-siv-105)
handleIGWAttach wrote the external switch + gateway port + LRP + default
route to OVN NB and returned immediately, before ovn-northd had compiled
NB -> SB and before ovn-controller on the gateway chassis had installed
the resulting flows. A subsequent RunInstances -> vpc.add-nat could
complete while the datapath was still dark; the launched VM was then
unreachable on its public IP for whatever flow-install latency the
chassis happened to incur. Customer-visible as 'I just created a VPC,
launched a VM, and can't SSH in' for ~minutes.
Reproduced deterministically in CI run 26037892507 Phase8b: a VM in
a fresh VPC was unreachable at 192.168.0.202:22 for 3+ minutes, while
a VM at the same public IP in the warm default VPC (Phase8d) was
reachable in ~40 seconds.
Block at the end of handleIGWAttach on `ovn-nbctl --wait=hv sync`,
which bumps NB_Global.nb_cfg and waits for every connected chassis to
acknowledge the new sequence number — i.e. flows installed. Bounded by
--timeout=30; overruns log a WARN and continue (a 30s OVN stall is
already a deeper problem than failing VPC create would surface).
waitForFlowsHV is a package var so tests can stub it; in environments
without ovn-nbctl on PATH the shell-out fails fast and the WARN path
returns nil, so existing IGW tests stay green without a live OVN.
Refs: mulga-siv-105
* fix(vpcd): store spinifex:cidr on VPC create (mulga-siv-106)
handleVPCCreate built the OVN logical router with only spinifex:vpc_id
and spinifex:vni in ExternalIDs. Cleanup paths (handleIGWDetach + NAT
sweeps) read external_ids['spinifex:cidr'] to scope SNAT removal to
the VPC's own CIDR; when unset, they fell back to an overbroad
10.0.0.0/8 and emitted a WARN.
reconcileVPC already stored the same key at startup; the live-create
path drifted. Add the field to handleVPCCreate, log the cidr alongside
vpc_id, extend TestTopologyHandler_VPCCreate to assert the external_id
is set.
Refs: mulga-siv-106
* fix(vpcd): make handleDeleteNAT idempotent on 'already absent' (mulga-siv-107)
Instance termination fans out through multiple cleanup paths
(instanceCleanerAdapter.ReleasePublicIP AND
VPCServiceImpl.DeleteNetworkInterface), both of which publish
vpc.delete-nat for the same (vpc, logical_ip). The publish is
fire-and-forget (utils.PublishNATEvent uses PublishEvent for
vpc.delete-nat, not RequestEvent), so callers never branch on the
response. The second publisher then triggered a spurious
'ERROR vpcd: failed to delete dnat_and_snat rule ... not found' log
line ms after the first delete succeeded — confusing triage signal
with no actionable consequence.
Add ErrNATNotFound sentinel to the OVN client interface and wrap the
not-found return paths in both LiveOVNClient and MockOVNClient.
handleDeleteNAT matches errors.Is(err, ErrNATNotFound) and treats
that as success (logs INFO 'already absent (idempotent)').
Existing TestTopologyHandler_DeleteNAT_NotFound asserted the opposite
contract; the comment claimed EIP disassociation needed the error
signal, but DisassociateAddress also fire-and-forgets vpc.delete-nat,
so the contract never had a real consumer. Test renamed to
_Idempotent with assertSuccess.
Other failure modes (router missing, OVN unreachable, transact error)
still surface as ERROR + reply with the error string.
Refs: mulga-siv-107
* ci(e2e-go): cluster JUnit failures + name likely root cause (#230)
* ci(e2e-go): cluster JUnit failures + name likely root cause
New composite action `.github/actions/e2e-analyze` runs after
`go-junit-report` converts each suite log. For every `<failure>` it
strips ANSI, normalises dynamic noise (EC2 IDs, UUIDs, IPs, timestamps,
file:line) to a stable signature, picks the earliest non-cascade
failure per suite as the likely root cause, and buckets the rest by
signature. Output is appended to `$GITHUB_STEP_SUMMARY` and written to
`<log-dir>/analysis.md` so it travels with the artifact bundle.
Cascade markers (`must populate fix.`, `Should NOT be empty`,
`Expected value not to be nil`) collapse the typical "one upstream
phase fails, ten downstream `require.NotEmpty` assertions trip" pattern
into a single grouped entry under the real first failure.
The analyser is advisory — it always exits 0. The existing
"Fail job if any suite failed" gate keeps owning exit-code semantics.
Stage 1 of `docs/development/improvements/e2e-go-failure-analysis.md`
(mulga-siv-102). Stage 2 (per-failure log slicing) and Stage 3
(rule-based hints via `tests/e2e/.failure-rules.yaml`, placeholder
seeded here) are independent follow-ups.
CI wiring:
- e2e-go.yml: new "Analyze e2e failures" step between "Publish test
summary" and the artifact upload, always-runs.
- spinifex.yml: new `make test-actions` step so the golden-file tests
under `.github/actions/e2e-analyze/testdata/` run on every PR.
- Makefile: `test-actions` target; kept out of `test-cover` so CI-only
tooling doesn't dilute coverage %.
Golden fixtures: a synthetic cascade-vpcs case (one Phase 5
InvalidParameterValue root + 15 downstream cascades), one real
single-failure case captured from run 26037892507, and a clean
no-failures case.
* ci(e2e-go): per-failure log bundles with time-window slicing
Stage 2 of the e2e failure analyzer. For each failure surfaced by
Stage 1, materialise <log-dir>/analysis/NN-<TestName>/ containing:
failure.txt — summary (test, start, duration, file-site, error)
journal.log — spinifex-journal.log sliced to
[StartAt - 5s, StartAt + Duration + 5s]
test.log — test-<suite>.log block between `=== RUN <name>`
and the matching `--- FAIL` marker
<copies> — any files the harness already dropped at
<log-dir>/<TestName_with_/_replaced>/
Report links each entry to its bundle so a reader on the Actions
page goes straight to the relevant slice.
Workflow now invokes journalctl with --output=short-iso so the
analyzer can parse wall-clock timestamps for the journal window.
Lines without a parseable timestamp inherit the previous line's
classification (keeps multi-line records and stack traces
contiguous).
Render output is unchanged when BundlePath is empty, so the Stage 1
golden snapshots still match.
* ci(e2e-go): restore GOTOOLCHAIN=auto after setup-go v6 bump
setup-go@v6 flipped the GOTOOLCHAIN default from "auto" to "local",
which makes Go refuse to auto-download the toolchain version named
in go.mod. The self-hosted ci-single runner ships /usr/local/go
older than spinifex/go.mod's `go 1.26.3`, and build-install-artifacts.sh
prepends that path so the system Go wins the resolution race. Under
v5 ("auto"), Go silently fetched 1.26.3 on demand; under v6
("local"), the Build Install Artifacts step errors out:
go: spinifex/go.mod requires go >= 1.26.3 (running go 1.26.1; GOTOOLCHAIN=local)
Re-set the env to "auto" so behaviour matches pre-v6 runs.
* ci(e2e-go): re-export GOTOOLCHAIN=auto after setup-go
Workflow-level env doesn't help: setup-go@v6 writes GOTOOLCHAIN=local
to $GITHUB_ENV, which overrides workflow env in every later step.
Add a dedicated step right after setup-go that appends
GOTOOLCHAIN=auto to $GITHUB_ENV so the Build Install Artifacts step
(which uses /usr/local/go from a stale runner path) can self-heal by
fetching the toolchain named in go.mod, matching pre-v6 behaviour.
* fix(vpcd): add flows-ready barrier after dnat_and_snat add
handleAddNAT wrote the NAT rule to OVN NB and returned immediately,
mirroring the gap siv-105 closed for handleIGWAttach but on the NAT
add path. ec2d/RunInstances then returned success while ovn-northd
was still compiling NB->SB and ovn-controller hadn't installed the
DNAT/SNAT flows on the gateway chassis, so the freshly launched VM
was unreachable on its public IP for whatever flow-install latency
the chassis happened to incur.
Reproduced in CI run 26072432957 Phase8b_VPCSubnetE2E: VM at
192.168.0.212 timed out at 3min while diagnostics showed
Port_Binding up=true for both the VM eni and the chassis-redirect
gateway. SB-side convergence was complete; the missing piece was the
same `ovn-nbctl --wait=hv sync` barrier siv-105 added to IGW attach,
applied at the end of handleAddNAT as well.
Reuses the existing package-var `waitForFlowsHV` so tests can stub
it; no new infra. Parallel unit test asserts the call site fires
exactly once per add-nat.
* ci(e2e-analyze): real-artifact fixes for journal slicing
Three bugs surfaced after Stage 2 ran against real CI artifacts in
run 26072432957:
1. Regex `[+-]\d{4}` rejected the colon-offset form (e.g. `+10:00`)
that systemd's `journalctl --output=short-iso` actually emits on
most versions. Added a parser that tries RFC3339 first and falls
back to the no-colon `-0700` layout. Result: 0-line journal slices
become populated.
2. go-junit-report's `timestamp` attr on <testsuite> records the
moment the XML was produced (post-suite), not when the tests
started. Computing StartAt as `that + cumulative-priors` placed
every failure AFTER the journal collection ended, so windows
always missed. Workflow now writes `${ARTIFACT_DIR}/test-<suite>.start`
right before running each suite; analyzer prefers that file over
the junit timestamp.
3. Persisted `OffsetFromSuiteStart` on each Failure so the
suite-start override can recompute StartAt without re-parsing the
XML.
Unit tests added for the colon-offset journal format. End-to-end
test (TestWriteBundles_EndToEnd) still passes with the new layout.
* ci(e2e-analyze): skip parent testcases when summing suite offsets
go-junit-report emits one <testcase> for the parent Go test AND each
subtest, with the parent's time = sum of its children. ParseFile was
summing every testcase into the suite-cumulative-offset counter, so
every leaf's offset was inflated by ~the parent's total duration.
Visible on run 26073817490 as Phase8b's StartAt landing 13 minutes
after the suite finished collecting logs — beyond the journal window,
so the per-failure journal slice was empty.
Pre-compute the leaf set per suite (a name is a leaf iff no other
testcase name starts with `<name>/`) and only accumulate cumul on
leaves. Golden snapshots updated; the offset for the only failure in
each fixture now matches the actual leaf-sum start time.
* test(e2e-harness): capture IP-targeted datapath state on Phase8b SSH fail
Phase 1 diagnostics for mulga-siv-111. The Phase8b SSH timeout
reproduced on the e2e-failure-analysis branch with siv-105's IGW
barrier AND the new add-nat barrier BOTH firing cleanly, so the
residual gap is downstream of `--wait=hv sync` — most likely stale
OVS conntrack, stale upstream-host ARP, or an OF flow-install miss.
DumpVPCFlowDiagnostics now takes a VPCDiagnosticsOpts carrying the
external IP, logical IP, and per-test artifact dir. When set, the
helper greps `ovs-ofctl dump-flows br-int`, `conntrack -L`, and
`ip -4 neigh show` for the external IP / logical IP and snapshots
`ovn-nbctl find NAT external_ip=…`. Each capture is both printed
inline (existing affordance) and written to a separate file under
fix.Artifacts so the Stage 2 analyzer bundler picks it up alongside
the journal + test-log slices, no analyzer changes needed.
Phase8b's caller now passes pubIP / privIP / fix.Artifacts so the
captures fire on the next failure; updates the t.Fatalf message to
point at siv-111 instead of siv-105 (which is the now-closed parent).
* fix(e2e-single): authorize SSH ingress on fresh-VPC default SG in Phase8b
Phase 8b stands up a non-default VPC and runs an instance there. Its
auto-created default SG denies all ingress, so the SSH probe times out
even when every datapath barrier converges — OF flow captures show the
SYN reaches OVN, gets DNAT'd at table 13, then drops at the ACL between
tables 13 and 17. The bash driver relies on admin init having pre-baked
this rule on the *default* VPC's default SG; a fresh VPC has not.
Add a DescribeSecurityGroups + AuthorizeSSHIngress step between
route-table setup and run-instances, using the existing harness helper.
Fix surfaced via mulga-siv-111's Phase 1 diagnostics (OF flow + ACL
drop counts), invalidating the H1/H2/H3 stale-state hypotheses in the
bead plan doc.
* test(e2e): dual-detect ubuntu image version (24.04/26.04) during gold-image transition
After mulga gold image bumped to v6-fastboot (stages ubuntu-26.04 / resolute
instead of ubuntu-24.04 / noble), all consumers that hardcoded 24.04 break on
the new image. Make them detect at runtime so v3 and v6 envs both pass.
- tests/e2e/run-e2e.sh: DescribeImages filter accepts both
ami-ubuntu-{26,24}.04-<arch>, then derives IMAGE_NAME from the match.
- tests/e2e/run-pseudo-multinode-e2e.sh: prefer ~/images/ubuntu-26.04.img,
fall back to ubuntu-24.04.img; pass detected version to spx import.
- tests/e2e/single/image_test.go: walk candidate catalog names (26.04 first,
then 24.04), stop on the first one DescribeImages can resolve.
- .github/workflows/e2e-baremetal.yml: download resolute (26.04) into
BM_ISO_CACHE; update BM_UBUNTU_IMAGE env var path.
- docs/terraform-workbooks/{bastion-private-subnet,s3-webapp,nginx-webserver}:
AMI filter matches `*ubuntu-26.04*` OR `*ubuntu-24.04*` (AWS values=
array is ORed).
Companion changes for mulga (bootstrap scripts, baremetal, iso pipeline)
land in scripts/tofu-cluster/* and scripts/{tofu-iso,baremetal,iso-builder}/*
under bead mulga-siv-114. Drop the 24.04 fallbacks once all envs are on the
v6 gold image and 24.04 entries are removed from spinifex/utils/images.go.
* feat(e2e): add service-interfaces.yaml manifest + validator
Source of truth for which spinifex services produce/consume which NATS
subjects, and which e2e suites cover which services. Consumed downstream
by the targeted suite selector (Bead 4) to derive subset E2E runs from a
PR diff so single-service changes stop paying the full 25 min suite cost.
- docs/service-interfaces.yaml: 7 services, 4 active suites (cert, lb,
single, ddil). Subscribes / publishes / depends_on filled from grep
of NATS subjects in source. spinifexui marked covers_no_suite so
UI-only diffs skip e2e entirely. Future suites (multinode, reboot,
tofu, baremetal) commented in until Go ports land.
- tests/e2e/manifest-check: schema + cross-reference validator (path
existence, subject regex, duplicate detection, suite covers resolves
to a real service, fixtures references resolve). 8 unit tests cover
the failure modes plus a load-and-validate on the real manifest.
- Makefile: manifest-check target, hooked into preflight.
- docs/DESIGN.md: link to the manifest under "NATS Messaging".
mulga-siv-116. See docs/development/improvements/e2e-targeted-suite-selection.md.
* fix(e2e): drop catalog-download import path from phase4_Image
`spx admin images import --name ubuntu-26.04-x86_64` with wantErr=true
was meant to detect that bootstrap-install.sh already imported the AMI
and short-circuit on the duplicate exit code. After the v6 gold-image
bump preloaded `~/images/ubuntu-26.04.img` and bootstrap-install.sh
switched to `--file` for the import, two things broke at once:
- The catalog `--name` path no longer hits a duplicate guard for the
--file-imported AMI, so the import re-downloads ~857MB and either
succeeds (failing the wantErr=true assertion) or wedges the job
timeout while the catalog server crawls along at 16KB/s.
- `--file` from the test won't work either: the test binary runs as
`tf-user` over SSH, while bootstrap-install.sh placed the image at
`/home/spinifex/images/...` (mode 0700 home dir).
bootstrap-install.sh is already responsible for staging the AMI. The
test's job is verifying DescribeImages round-trips by-name and by-ID —
not re-exercising import. Trust the bootstrap-staged AMI and stop
calling import.
Keeps the 26.04→24.04 dual-detect for the v3 gold image transition;
drop the 24.04 candidate when no envs remain on v3.
Refs mulga-siv-115 (caught by the EC2 cold-boot fast-path E2E run on
feat/ec2-coldboot-fastpath).
* ci(e2e-go): default to e2e-single + harden status-report SHA fallback
Two changes to .github/workflows/e2e-go.yml:
1. Default `suites` input now includes e2e-single so a no-input
manual dispatch exercises the full single-node ladder. Single-node
Go harness is greenable now (image_test fix at
|