mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
07dce613eb5ac760bf5316b606eef5664e595eb3
42 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Julian Sommer
|
f919a257ef | fix: make the SG default-deny ACLs match every ethertype (#947) | ||
|
Julian Sommer
|
c84200e519 | feat: probe real qemu state (#839) | ||
|
Julian Sommer
|
17cea6cd4b |
fix: gate the SG datapath probe on the target's HTTP listener
TestSGPolicyDatapath/AllowedTraffic waited 60s for curl to reach the target's HTTP server but nothing ever waited for that server to start. Its cloud-init runs later than the client's sshd, which is what the preceding stage actually gates on, so the budget was measured from the wrong baseline and the gap between the two is unbounded on a loaded node. The failures reported curl exit 7, a refused connection: a missing SG allow is a drop ACL here, which would time out as exit 28. The target guest now reports its listener on the serial console, the only channel back to a guest whose SG has no SSH ingress, and a new TargetHTTPReady stage waits for that marker before AllowedTraffic runs at all. AllowedTraffic keeps its 60s budget, which now only has to cover chassis flow install, and dumps VPC flow diagnostics when it does fail so a genuine propagation failure is attributable from CI artifacts. |
||
|
Julian Sommer
|
2e09461080 | chore: cleanup code comments | ||
|
Josh Siviour
|
9a1fb44065 |
fix: CreateImage live drain + Alpine .raw sparsify (#647)
* fix(ec2): drain the attached volume before CreateImage reads a live checkpoint CreateImage against a running instance (--no-reboot) read the volume's checkpoint directly from predastore without draining in-flight writes first, unlike CreateSnapshot. A checkpoint captured before a guest- triggered write (e.g. growpart rewriting the GPT after a root volume is launched larger than the base image) could be stale, producing an AMI that boots to a UEFI shell instead of the OS. Extracts the existing snapshot-package drain logic into a shared, reusable DrainVolume so both live-snapshot call sites route through the same drain-before-read guarantee instead of being able to diverge. * test(e2e): boot-verify CreateImage AMIs instead of trusting State=available describe-images reporting State=available is not evidence an AMI boots — a well-formed but unbootable AMI (stuck at the UEFI shell) still registers cleanly. Extend the CreateImage e2e test to launch a probe instance from each produced AMI (running-path and stopped-path) and require a real SSH login, with console-dump-on-failure wired up for diagnosis. * fix(images): sparsify the Alpine guestfish .raw before conversion The Alpine build path grows the qcow2 in place via qemu-img resize + guestfish resize2fs, which can leave metadata scattered across the newly-grown region. virt-sparsify zeroes the guest's unused blocks before the raw conversion so they collapse into real holes, matching the Ubuntu virt-resize path's sparseness. |
||
|
Julian Sommer
|
7e356b5b9b | test: cover DescribeInstanceStatus and the SDK instance-status-ok waiter (#588) | ||
|
Julian Sommer
|
5b8f20b89f | test: assert ModifyVolume resize reaches the guest block device (#585) | ||
|
Josh Siviour
|
cae76ba1b6 |
test: complete the e2e live-vs-mock rebalance (epic mulga-8oyg3) (#566)
* test(volume): assert chunk GC stays wired on in CreateVolume GCEnabled was zero-valued at every viperblock construction site for a long time and no test noticed. It is not part of viperblock.VBState, so it never round-trips through a saved config.json and cannot be observed by reading volume state back. Extract the viperblock.VB assembly out of CreateVolume into buildVBConfig so the GC wiring is assertable directly against the returned config, without a live viperblock or S3 backend. The config is returned by pointer because viperblock.VB carries atomic.Uint64 fields that vet's copylocks check rightly refuses to see copied. * test(e2e): snapshot backend byte accounting before volume purge DeleteVolume purges a volume's entire backend prefix, including any chunks a leak left behind, before anything measures it. That is the same masking that hid the storage-growth leak in production: a passing E2E cannot fail on backend growth when the evidence is deleted first. Snapshot the volume's backend chunk footprint in EnsureVolume's cleanup closure, immediately before DeleteVolume, so every volume the shared teardown path purges is measured rather than silently discarded. The snapshot is always taken and logged; the hard ratio check additionally runs for callers that registered a guest-written-bytes baseline, since most callers never attach or write to the volume they create and so have no baseline to compare against. The tolerance is the RS(2,1) erasure floor widened for per-chunk header and tail-chunk overhead. It is sized to distinguish unbounded growth from a footprint settling at the floor, not to catch bounded residual garbage. The check is non-fatal inside the cleanup callback: t.Fatalf's runtime.Goexit would abort the DeleteVolume call queued behind it and turn a reported leak into a real one. * test(e2e): add memory-constrained concurrent-write durability regression The existing durability tests write one 4MiB sentinel, sequentially, on a comfortably sized guest, which is the exact inverse of the known-open data-loss bug: acked writes read back zero-filled after a cold reread when a guest is memory-starved during concurrent writeback. No happy-path test can reach it, because the fault lives in the intersection of memory pressure, write concurrency and cache-cold reread. Add a suite that encodes those three conditions together: a guest discovered at or below the working set, N parallel large writers launched as background jobs in a single command so they genuinely overlap, swap disabled, then sync and drop_caches before rereading. Assertions are per stream rather than aggregate so a failure names the losing writer instead of leaving whoever reads it to re-derive which of N writers lost data from one combined digest. GuestExec gains a timeout-parameterised variant. Its fixed two-minute default is sized for sub-second payloads, and a multi-gigabyte write to a network-backed volume would exceed it, so the suite would have failed by timeout while looking like it had caught the corruption. The budget scales with the working set, and a deadline is reported distinctly from a data mismatch so the two red states are never confused. The suite is dispatchable but deliberately absent from the default SUITES: the bug it reproduces is still open, so it is expected to fail and must not sit in a merge-gating path. It is registered in the workflow allow-list so it stays discoverable rather than being buried behind a skip. Also record on the live IAM instance-profile test why its assertions were not moved down to the integration tier, so the trace is not repeated. * test(e2e): drop backend byte accounting, assert a node-wide delta instead The accounting helper listed the internal predastore bucket to attribute bytes per volume. That listing is refused unconditionally: predastore authorises bucket access by same-account ownership, and the bucket belongs to the service account, so a test caller gets 403 no matter what it holds. Granting the test a service credential would invert the trust boundary to buy a measurement, which is the wrong trade. Replace it with what the S3 surface actually permits: sample node-wide allocated bytes either side of the write and bound the delta against the bytes the guest wrote. Coarser, but it measures something real rather than depending on a call that cannot succeed. The tolerance moves into the suite that uses it. It encodes RS(2,1)'s 1.5x expansion floor plus headroom; a cluster running the recommended RS(3,2) has a 1.667x floor and only ~12% of margin left, so the constant needs revisiting if the encoding changes. * revert: drop memdurability e2e suite The suite asserted the guest read back what it wrote after a memory-constrained concurrent write, but the assertion could not fail for the reason it claimed: it compared the guest's own read against the guest's own write with the page cache still warm, so it passed whether or not the data reached the backend. A test that is green on a build with a known-open durability bug is worse than no test — it advertises coverage that does not exist. Removes the suite and its Makefile/workflow wiring, plus DiscoverInstanceTypeAtLeastMemory, which had no other caller. GuestExecTimeout stays: GuestExec now delegates to it, and the old two-minute hard cap it replaced was a latent trap for any long guest command. * test(e2e): add partial-block concurrent-write durability suite Replaces the removed memdurability suite, which passed on a build that still carried the bug it claimed to test. The engine reaches its read-modify-write path only for a partial block; a fully covered block takes a full-overwrite branch that reads nothing and can lose nothing. The old workload wrote through ext4 with 1 MiB buffers, which the filesystem coalesced into aligned full-block requests: block-layer tracing recorded zero unaligned and zero sub-4096 requests, so the path under test never ran, and writers 4 MiB apart could not collide on a block even in principle. This suite writes the shape the defect needs. Two writers issue concurrent O_DIRECT writes into disjoint halves of the same 4096-byte blocks on a raw, unformatted device, stamping every half with the round number, then drop the page cache and reread with O_DIRECT asserting every half carries the final round. One half reading back at an earlier round is the signature of the lost update. The old suite also could not fail even had it reached the path: its write phase ended in 'wait; sync; sha256sum' and reported sha256sum's exit status, while bare wait returns 0 unconditionally, so writer errors were invisible and a backend ENOSPC presented as a pass. Here each writer's exit status and exact byte count are asserted before any durability claim is made. Working set is 8 MiB rather than 2048 MiB, and the volume is deleted on teardown; the predecessor left roughly 3 GiB of backend per run and twice drove a node under the nearfull watermark. Dispatchable but not in the default suite list until a green run and a demonstrated fail-first against a pre-fix engine exist. * test(e2e): let the fixture own the working-set volume delete The suite deleted the volume in its own cleanup and the fixture deleted it again on close, so every run ended with an InvalidVolume.NotFound from the second attempt. Detach stays — DeleteVolume against a still attached volume is what leaks volumes on teardown — and the delete is left to the fixture so the volume is released exactly once. * test(e2e): gate merges on the partial-block durability suite Verified on one node with the same test binary and workload, varying only the engine: 1758 of 8192 halves reverted a generation against the pre-fix build, 0 of 8192 against the fixed one. That is the fail-first evidence the suite needed before it could be trusted to gate anything. Adding it to the default suite list rather than leaving it dispatchable is the point. Its predecessor sat outside every path that runs, which is how it survived long enough to be discovered green on a build that still lost data. * test(integration): cover the ECR data plane in the front gate The live ecr/ suite drove docker, crane and skopeo against a real registry, yet none of it boots a guest: it needs a registry, an object store and an HTTP client. The in-process gateway already serves the OCI Distribution v2 surface on an ephemeral port with a memory-backed object store, so these run here in 0.18s instead of behind full provisioning. Covers the three claims the live tests made beyond authorization, which is already down here: a push/pull round trip asserting blobs and manifests read back byte-identical, refusal to auto-create a repository on push, and retagging an existing manifest without disturbing the original tag. Content is compared rather than status codes alone. A registry that stores a blob under the wrong digest, truncates it, or serves a different one answers 200 to every request in the sequence. The capacity/load leg deliberately stays live: it exists to put real parallel pressure on a real predastore, and a memory-backed object store cannot make that claim. * fix(e2e): make volume teardown release the volume before deleting it Fixture cleanups drain LIFO, so a suite that creates its instance before its volume tears the volume down first, while it is still attached. DeleteVolume answered VolumeInUse, nothing retried, and the volume was stranded on the node. Reordering the chain cannot fix this — registration order has no way to express the instance/volume dependency. Instead teardown is now self-sufficient for any creation order: it force-detaches, waits for the volume to reach available, and only then deletes. Instance teardown waits for terminated for the same reason, since TerminateInstances returns as soon as the request is accepted. Two things kept the leak invisible. A failing cleanup was logged and forgotten, so a run stayed green with resources left behind; failures are now reported as leaks, failing the bound test or surfacing through Close() for TestMain to fail the run. And a panic in one cleanup aborted the whole chain, stranding every resource behind it; each callback is now isolated. RegisterVolumeTeardown and the fixture had grown separate teardown paths, only one of which detached. Both now share it. |
||
|
Josh Siviour
|
48749576c3 |
test: rebalance E2E into in-process integration tier + shared-guest live scenarios (#556)
* test: add in-process integration test tier Roughly half of the behavioural tests under tests/e2e/ assert control-plane API shapes, authz and error codes that need no live environment, yet they can only run against a tofu-provisioned environment on a scarce self-hosted runner. This adds the tier they can move down to. The real gateway router runs in-process: SetupRoutes() is a pure function of GatewayConfig, so it serves from an httptest server over embedded NATS JetStream with real IAM/STS services and genuine SigV4 verification. Only the daemon side of the wire is faked -- the ec2.*/ebs.* NATS subjects a live node would answer are stubbed per test. Nothing is provisioned and no fixed ports are bound, so the tier runs concurrently on a shared build host and is safe as a CI front gate ahead of the live suites. It is build-tagged and kept out of preflight, which stays unit-only. * test(integration): add IAM/STS client accessors and port the zero-stub authz tests Adds IAMClient/STSClient plus ClientsWithCreds/ClientsWithSessionCreds so a test can act as a principal other than the seeded root user, and stubs the node-discovery subject in StartGateway so call paths that trigger discovery resolve immediately instead of riding out the fan-out timeout. Ports the four enforcement tests that guard on ec2:DescribeRegions, a gateway-local action, so they need no daemon-side stub at all. * test(integration): port remaining IAM/STS authz tests from e2e Ports TestIAMUserCRUD, TestIAMAccessKeyLifecycle, TestIAMUserAuthentication, TestIAMPolicyCRUD, TestIAMPolicyAttachmentEnforcement, TestIAMPolicyLifecycle, TestIAMGroupsLifecycle, TestIAMRolesAndProfiles and TestSTSAssumeRoleAndGetCallerIdentity from tests/e2e/iam/ into the in-process integration tier. Each test now self-bootstraps its own users/policies/roles instead of relying on shared suite-level state, since every integration test gets its own gateway and embedded NATS. TestIAMRolesAndProfiles stubs ec2.IamProfileAssociation.describe because DeleteInstanceProfile always counts live associations over NATS regardless of which guard actually fires. TestIAMCleanup is not ported: it only swept a shared long-lived E2E environment, and every ordering/dependency assertion it made is already covered by TestIAMPolicyLifecycle, TestIAMGroupsLifecycle and TestIAMRolesAndProfiles's own delete-conflict guards. * test(e2e): remove IAM/STS authz tests migrated to the integration tier All 13 signed-request authz tests plus TestIAMCleanup now have faithful, mutation-proven equivalents in tests/integration/ — the gateway dispatches IAM/STS actions directly with no daemon hop, so the in-process tier is a faithful and far cheaper substitute. The iam/ e2e suite now runs only TestIAMInstanceProfileAssociation and TestIMDS. Drop the now fully-dead iam_helpers_test.go alongside the six deleted test files, and remove harness.IAMGroupARN/IAMDeleteGroupBestEffort, which had no remaining caller anywhere in the repo once iam_groups_test.go was gone. * feat(integration): daemon-lite harness for key/tags/routetable/vpc Adds DaemonLite, an in-process stand-in for a live spinifex daemon that subscribes the real key, tags, route table, and VPC/subnet/SG/IGW service implementations directly to their NATS subjects, backed by memory and embedded-JetStream stores. Ports TestKeyPairs, TestTagManagement, TestRouteTableValidation, and TestReplaceRouteConvergence from tests/e2e/single so the integration tier drives genuine daemon-side business logic instead of StubSubject canned replies. vpcd (OVN) is out of scope, so its "vpc.create-sg"/"vpc.delete-sg" acknowledgements are stubbed with a fixed success reply — the SG record itself is still written by the real service impl before that event fires, so the stub only unblocks the synchronous round-trip rather than replacing tested behavior. * ci(e2e): gate live E2E jobs behind the in-process integration tier Runs the in-process integration suite first, on the ci-single runner host rather than inside a provisioned VM, so a broken control plane is caught before either live job pays the Spinifex install. Both e2e_single and e2e_multi now require it, and e2e_status treats a skipped integration job as a failure so the gating status cannot go green on a gate that never ran. Drops the shadow-mode suite selector: targeted per-diff suite selection is not the direction being pursued, and leaving a non-functional shadow job in the gating workflow only invited it to be mistaken for a live gate. * test(e2e): remove single/ tests superseded by daemon-lite ports TestKeyPairs, TestTagManagement, TestRouteTableValidation, and TestReplaceRouteConvergence now have proven-equivalent ports in tests/integration/ (mutation testing confirmed parity). Delete the e2e originals and their runX helpers, along with keypair_test.go's and tags_test.go's file-local helpers left with no caller. generateImportPubKey is shared by other single/ tests so it moves to helpers_test.go instead of being deleted. Slims the manifest-lint baseline for the removed create-call entries. * ci: run the lb e2e suite on multinode only Both e2e jobs filter SUITES by exclusion, and neither excluded lb, so the suite ran twice per e2e run: 354s on single and 654s on multi. Balancing across instances on one node is not a deployment mode we support, and ELBv2 proper requires two AZs, so the single-node run was redundant. Drops ~354s from the single-node job, which is the gate critical path. * test(integration): port 3 of 7 ECR tests to the in-process tier Wires the gateway's ECR control plane (repository/policy/lifecycle metadata over an in-process daemon-lite NATS subscriber, blob and manifest bytes via a memory object store, JWT issuer/verifier keyed off the existing IAM master key) into the integration harness, and ports TestECRControlPlane, TestECRGetLoginPassword, and TestECRIAMAuthorization_AssumedRolePullOnly from tests/e2e/ecr — each proven equivalent by mutation (evaluatePrincipalPolicy forced to allow, and the repo-describe path forced to not-found) before its E2E original was removed. The other 4 authorization tests (PullOnly, ReadOnly, DetachPolicyDeniesImmediately, DeactivatedKeyReturns401) stay on the E2E tier: AttachUserPolicy lacks the AWS-managed-ARN existence-check bypass that AttachRolePolicy/AttachGroupPolicy already have, so attaching AmazonEC2ContainerRegistryPullOnly/ReadOnly straight to a user fails NoSuchEntity in both tiers alike. Left untouched pending a fix to that shared IAM code. * fix(iam): accept AWS-managed policy ARNs on AttachUserPolicy AttachUserPolicy called getPolicyByARN unconditionally, so attaching an AWS-managed ARN to a user returned NoSuchEntity. AttachRolePolicy and AttachGroupPolicy already skip that lookup for AWS-managed ARNs and store them opaquely; the user path was missed when that guard was introduced. ListAttachedUserPolicies had the mirror of the same gap: it resolved every ARN through getPolicyByARN and silently dropped the ones it could not find, so a managed ARN would disappear from the listing rather than round-trip. Evaluation already handled these correctly via resolveAttachedPolicy, so only the write and list paths needed the guard. * test(integration): port remaining 4 ECR authorization tests Ports TestECRIAMAuthorization_PullOnly, _ReadOnly, _DetachPolicyDeniesImmediately, and _DeactivatedKeyReturns401 from tests/e2e/ecr now that AttachUserPolicy accepts AWS-managed policy ARNs. Each was proven equivalent by mutation before its E2E original was removed: PullOnly/ReadOnly/DetachPolicyDeniesImmediately fail when evaluatePrincipalPolicy is forced to always allow, and DeactivatedKeyReturns401 fails when the access-key-active check in resolveECRLongLivedPrincipal is bypassed. All mutants were reverted and a clean run confirmed before committing. tests/e2e/ecr/iam_authorization_test.go is now fully removed — every test it held has an in-process equivalent, and its helpers had no other callers in the package. * ci: select e2e suites by allow-list instead of deny-list Both jobs filtered SUITES by exclusion, so any suite named in neither deny-list ran in both jobs. That is how lb came to run twice per e2e run, and it makes adding a suite silently fan it out everywhere. Each job now names the suites it is eligible to run. Selection is still driven by SUITES, so this is behaviour-preserving for the current default: single/iam/cert on the single-node job, multinode/lb/cert on the multi-node job, exactly as before. eks, ecs and storagegrowth are listed as eligible on the single-node job but are not yet in the default SUITES. * test(e2e): delete seven tests with no coverage TestFinalClusterStats and TestInstanceLaunch assert nothing beyond what other callers of their shared helpers already imply. The multinode baseline, volume, guest-SSH, and preflight tests duplicate single-node coverage or exercise per-guest behaviour that isn't cluster-specific. TestMultinodeInstanceDistribution can't fail: three independent RunInstances calls with no placement group make a strict spread assertion flaky and a loose one meaningless. Preflight checks (/dev/kvm writability, peer SSH reachability) move into requireMultiNodeFixture so the package fixture singleton still fails fast with a clear message instead of disappearing along with the dedicated test. The remaining callers of needInstanceTrio (CrossNodeOps, CrossNodeGateway, NodeFailure, VolumeDurability) still trigger the shared trio launch on their own, so removing the distribution and volume-lifecycle tests doesn't leave it unbuilt. Also slims the manifest-lint baseline entry for the deleted multinode volume test. * fix(e2e): require full 3-node spread in TestMultinodeSpread SpreadPlacement passed at unique>=3, warned via t.Logf at unique==2, and only failed below that. A 33% spread failure (two instances colocated) passed with a log line. The 243.6s NAT-egress scenario that follows only proves what it claims to if the private trio actually landed on three distinct chassis, so that premise needs to be a hard assertion, not a warning. * refactor(e2e): drop the unused spread-assertion harness helpers AssertSpreadAcrossNodes and CountInstancesPerNode lost their only caller when the instance-distribution test was removed. Beyond being dead, the assertion helper documented minNodes=1 as an acceptable logging-only mode inherited from the bash driver, which is the pattern that let a placement test pass while every instance landed on one node. InstanceHostingNode and its ps helpers stay; five multinode tests use them. * fix(ec2): stop leaked goroutines on prepareEFIVolume error paths newViperblock starts the chunk uploader and WAL syncer unconditionally, but prepareEFIVolume only released them on the happy path via Close(). Six error returns between acquisition and Close left both goroutines running. Add a defer immediately after acquisition, matching the sibling prepareRootVolume; Close() remains the durability boundary. * test(predastore): add real predastore fixture for integration tests Extract predastore_integration_test.go's inline daemon startup into a reusable testutil/predastore package so tests/integration can point DaemonLite-style wiring at a real S3-compatible backend instead of an unreachable host. Kept as its own leaf package rather than folding into the shared testutil package: it transitively imports predastore's quicclient, whose package-level connection pool starts a permanent goroutine at init that would trip goleak checks in any of the ~80 other packages that import testutil for unrelated NATS helpers. * test(integration): migrate CreateVolume to assert real predastore persistence Add StartVolumeDaemonLite, an opt-in DaemonLite variant wired to the real predastore fixture, and migrate TestCreateVolume_PersistsToRealPredastore off memory-backed DaemonLite so CreateVolume/DescribeVolumes exercise viperblock's real S3 backend end to end instead of stopping at input validation. * test(integration): add real storage-state oracle for CreateVolume DescribeVolumes only proves the handler's own read path agrees with its own write path, both via the same deserialization; it cannot catch a bug in a field it never re-derives (e.g. a unit-conversion error in the bytes handed to viperblock, since VolumeMetadata.SizeGiB is copied from the request directly). vbscan reads a volume's persisted config.json and chunk objects straight over S3, independent of any spinifex handler code, so TestCreateVolume_PersistsToRealPredastore can assert on what viperblock actually wrote rather than only on what the handler claims. vbscan is a local reimplementation of scripts/vbrefscan's read-only scan rather than an import: spinifex cannot depend on mulga, the module that contains it. It only reimplements the config/chunk-listing half of vbrefscan — the checkpoint-based reference-integrity half needs viperblock.ParseBlockCheckpointBytes, which does not exist in the viperblock version spinifex currently depends on (v1.12.1) and is also moot for this volume: CreateVolume never calls SaveLiveCheckpoint, so a freshly created volume has no checkpoint to scan regardless of version. * refactor(test): move the predastore daemon fixture out of the coverage scope test-cover runs go test ./spinifex/..., so anything under spinifex/ is compiled and measured. The daemon fixture cannot be exercised by unit tests, so it reported a coverage gap no unit test could close. The e2e harness already avoids this by living under tests/; the fixture now does the same. vbscan stays put, being ordinary library code with its own unit tests. * test(integration): migrate RunInstances multi-count expansion off live guests RunInstances with MinCount/MaxCount=2 expands into a single per-node targeted launch (distributeInstances in gateway/ec2/instance/placement.go) before the daemon-facing NATS hop, so the real gateway router plus a stubbed daemon response is sufficient to prove it — no throwaway VM is needed to exercise the expansion logic itself. The live single-node E2E test's WaitForInstanceState/TerminateInstances sequence was teardown hygiene for the real guests it booted, not an assertion of the expansion behavior, so it is dropped rather than emulated. The migrated test additionally asserts the per-node launch request itself carries MinCount=MaxCount=2, proving the count actually propagates rather than the launch succeeding by coincidence or via the zero-daemon degenerate path in placement.go. * docs: drop a doc reference to the deleted live test * test(integration): migrate launch-template RunInstances expansion off live guests RunInstances --launch-template resolves ImageId/InstanceType from the stored template entirely gateway-side (expandLaunchTemplate, before validation/placement/the daemon-facing NATS hop), so proving it forwards the resolved fields never needed a real guest boot. Ports the assertion to tests/integration via a new opt-in StartLaunchTemplateDaemonLite (real LaunchTemplateServiceImpl over embedded JetStream) plus a direct subscription on the per-node launch subject to capture what the gateway actually dispatched, rather than only the client-visible response. The live test keeps its CRUD/versioning lifecycle (create/modify/delete version, tag-filtered describe) — none of it touches a guest — and drops only the RunInstances-from-template + wait-for-running + terminate tail, whose only job was waiting out a real boot to read back fields already proven resolved. * test(integration): migrate KV-only account-scoping checks off live tenants Ports the KeyPair/VPC-Subnet/IGW-EIGW/AccountSettings cross-account isolation assertions from tests/e2e/single/account_test.go to the in-process integration tier. Each of these resources resolves ownership from a plain account-scoped KV key (utils.AccountKey(accountID, id), or an account-keyed settings record) with no vm.Manager or real guest involved, so DaemonLite now also wires the real EgressOnlyIGWServiceImpl and AccountSettingsServiceImpl alongside the existing VPC/IGW/Key services. Two independently minted tenant accounts (via IAMService.CreateAccount/CreateUser/CreateAccessKey/CreatePolicy/ AttachUserPolicy, in-process — the same calls runAccountCreate makes without the CLI/stdout round-trip) replace the live tier's carousel so isolation is checked between real, separately owned accounts. Left live: instance scoping (Step2) and volume scoping (Step3) — checkInstanceOwnership, the actual cross-account gate for stop/start/ reboot/terminate/attach-volume, runs daemon-side only after a real vm.Manager lookup, so it cannot be exercised without a live guest. Snapshot scoping (Step5) needs a real backing volume for CreateSnapshot (viperblock, out of DaemonLite's scope), so it stays live entirely rather than split off only its two KV-only assertions. Global-resources (Step9) carries no account branching in the handlers at all, so asserting its equality across two accounts wouldn't exercise anything a fake node responder didn't already trivially satisfy — not migrated. * test(integration): migrate launch-template CRUD lifecycle off live guests TestLaunchTemplates booted zero guests: its RunInstances coverage had already moved to TestRunInstances_LaunchTemplateExpansion, leaving only template CRUD/versioning assertions occupying a full live-cluster slot. Port the remaining create/describe/version/modify/delete lifecycle, including the SourceVersion merge and a nested timestamp round-trip, into TestLaunchTemplates_CRUDLifecycle against the in-process integration gateway. * refactor(tests): move spot instance lifecycle coverage to integration tier RequestSpotInstances is a mock over on-demand RunInstances (no bidding, interruption, or reclamation), so booting real VMs in the e2e tier proved nothing spot-specific. Migrate the SIR CRUD lifecycle (request, describe, cancel, close-on-terminate, insufficient-capacity error) to a NATS-level integration test that exercises the real gateway orchestration and daemon service impl without a live guest. Real-VM-boot assertions are already covered by TestRunInstances_MultiCountExpansion and the e2e baseline tests. * fix(tests): share one embedded NATS server across the integration package ClientTokenStore in spinifex/gateway/ec2/instance is a process-wide sync.Once singleton bound to whichever *nats.Conn first initialises it. Every integration test booted its own embedded NATS server, so the first RunInstances-calling test captured the singleton, and any later test inherited a KV handle bound to that server's now-closed connection. TestMain now boots one embedded NATS+JetStream server for the whole package; every StartGateway call connects into its own dynamically provisioned NATS account (accountAuthenticator in shared_nats.go), giving each test a private JetStream namespace despite the shared server, and none of the per-test connections are closed until every test has finished - whichever one the singleton latches onto stays live for the run. Also wire the suite into the required PR gate: it was previously only invoked from e2e.yml's push-triggered (main/dev), self-hosted "Integration (front gate)" job, so a regression here never blocked a PR. The whole package runs in single-digit seconds, so it now runs inline in `make preflight` and as its own step in spinifex.yml's build_and_test job. * test(integration): migrate serial-console access checks off live guests Enable/Disable/GetSerialConsoleAccessEnabled is pure gateway+account-settings logic with no guest involved, so it moves to the in-process integration tier against the real AccountSettingsServiceImpl wired into DaemonLite. * test(integration): migrate DescribeImages round-trip off live guests runImage's AMI describe round-trip needs no real cluster: it registers an AMI against a hand-seeded snapshot via the real ImageServiceImpl, backed by a new StartImageDaemonLite helper wiring RegisterImage/DescribeImages onto an in-memory object store. * test(integration): migrate predastore object lifecycle off live guests runPredastoreObjectLifecycle only needs a real predastore daemon, not a tofu-provisioned cluster: it now runs against tests/fixtures/predastore's in-process fixture, covering the same create/put/get/delete/list contract. * test(integration): migrate ListFoundationModels catalog check off live cluster ListFoundationModels is pure gateway-side static catalog logic with no NATS/daemon hop, so it moves to the integration tier via a new BedrockClient harness accessor; this tier configures no credential resolver, so the Anthropic entry's absence is asserted explicitly rather than left optional. * test(integration): cover RunInstances client-token idempotency Proves replaying the same ClientToken through the real gateway returns the original reservation without a second daemon dispatch, and a different token launches independently. * test(integration): cover RunInstances launch-template expansion Proves version selection (omitted Version resolves to the template default, explicit Version selects that version) and field precedence (explicit RunInstances parameters override the template, unset fields inherit it) through the real gateway. * test(integration): cover RunInstances malformed-input rejection Proves a malformed ImageId and a malformed targeted-launch CapacityReservationId are rejected before any NATS dispatch, with no daemon stub registered so a regression that let either through would fail fast instead of hanging. * test(integration): cover RunInstances iam:PassRole enforcement Proves a caller granted ec2:RunInstances but not iam:PassRole on an instance profile's role is denied, and the identical session succeeds once PassRole is granted for that exact role ARN. * test(integration): cover RunInstances placement-group strategy routing Adds a placement-group daemon-lite wiring a real PlacementGroupServiceImpl over NATS, then proves spread strategy assigns exactly one instance per node while cluster strategy pins the whole launch to the single highest-capacity node. * test(integration): cover RunInstances capacity distribution across nodes Proves the default (non-placement-group) path packs a multi-instance launch across nodes proportionally to spare capacity rather than exhausting one node first. * test(e2e): merge SameSGComms and SGToSGDatapath around one client/target pair Both booted their own instance pair to check SG-to-SG datapath enforcement and same-default-SG east-west connectivity. Merging them around one shared client/target pair cuts four boots to two, and converting the inline harness.Step sequence into named t.Run stages (PortGroupMembership, AllowedTraffic, DeniedTraffic, RevokeAndVerifyBlocked, ReauthorizeAndVerifyRestored, SameSGComms) gives each its own pass/fail attribution instead of one opaque test result. SameSGComms now runs last and joins the default SG onto each ENI via ModifyNetworkInterfaceAttribute only after every SG-to-SG allow/deny assertion has already recorded its own result, since the default SG's self-reference rule admits all traffic between co-members and would otherwise defeat the denied-traffic check. * test(e2e): merge default-SG reachability, guest DNS, and egress enforcement DefaultSGReachabilityBaseline, GuestDNSResolution, and SecurityGroupEgress each booted their own instance to check one facet of "a guest is reachable only when policy says so": ingress deny/authorize, resolver + WAN egress, and egress ACL enforcement. Merging them around one shared guest cuts three boots to one, with named t.Run stages (Deny, Authorize, Resolver, WANEgress, RevokeEgress/RestoreEgress) giving each its own attribution. The egress revoke/restore round-trip now mutates this scenario's own dedicated SG instead of the shared default SG the rest of the suite depends on, isolating its egress-rule mutation from every other test. * fix(ci): single-quote e2e test_run regex across the ssh boundary * fix(e2e): anchor SG egress-enforcement probe to a WAN IP, not the gateway * test(e2e): merge single-node guest churn tests into one durability scenario Combines TestSSHProbe, TestENIHotplug, TestENIHotplugReconcile, TestVolumeDurability, TestModifyInstanceAttribute, and TestRebootInstance into TestGuestChurnDurability, a single shared-guest scenario that re-verifies a data-integrity sentinel sha256 after every perturbation stage (ENI hotplug, daemon restart, ENI detach, instance-type resize, stop/start, reboot). This proves the sentinel survives the full churn sequence rather than just one isolated stop/start. * fix(vpc): clear subnet drop gate when installing egress reroute A subnet with an attached IGW but no 0.0.0.0/0 route carries a priority-1100 drop gate on the VPC router. When a route later lands (NATGW reroute at 900, IGW reroute at 1000) the reroute is installed but the drop gate was never removed, so the higher-priority gate wins and black-holes the subnet's egress in lr_in_policy before it reaches routing or SNAT. This left NAT-gateway private-subnet egress with 100% packet loss: the SNAT stage saw zero packets and no conntrack was committed. Remove the subnet's drop gate for the same prefix once a reroute is installed. Idempotent and prefix-scoped, so a specific-prefix reroute is a no-op and leaves the default gate intact. * test(e2e): merge VPC subnet, NAT gateway, and EIP single-node tests Merges the former TestVPCSubnetE2E, TestNewVPCEgressBaseline, TestNATGateway, and TestInstanceEIP into one staged scenario, TestVPCEgressPaths, around one scenario-owned VPC and its two guests, cutting five guest boots to three. The public guest serves PublicSubnetEgress, then the NAT Gateway bastion hop, then (last, since association rewrites its public IP) the EIP flip. The private guest serves the NAT Gateway path, launched directly into the same VPC's already-existing private subnet instead of a dedicated one. RouteBeforeSubnet keeps its own second VPC and instance since it exercises the implicit main-route-table code path, a different one than the explicit route table this scenario's own VPC uses for its public subnet — collapsing the two would drop coverage of whichever path got merged away. The NAT Gateway up/down flip now repeats for an env-tunable round count (SPINIFEX_VPCEGRESS_NAT_ROUNDS, default 1), matching every other merged single-node scenario. * test: TEMPORARY mutation-check — revert |
||
|
Ben Duncan
|
28a876d17d | feat(dns): resolvable ALB/NLB + EKS DNS, forwarder hardening, cross-tenant reconcile (#532) | ||
|
Josh Siviour
|
183a08353e |
feat(network): harden routed-NAT v1 (VPC cap, e2e gate, docs) (#504)
Three related nat-mode fixes: - ura01: nat-transit pool now emits gw_lrp_range_start/end. admin init sets 100.127.0.16-.254 (239 VPC gateways) instead of the auto-derived top-16 (~15). Adds GwLrpRange* to PoolData + the spinifex.toml template so operators can widen any pool. - 724mm: detectPoolMode returns true only for external_mode=pool. nat and disabled clusters now skip Phase 8b/8d, which otherwise failed on AllocateAddress UnsupportedOperation. - pnz2i: document that system instances (ECS/EKS/LB agents) are unsupported in nat v1 (require pool mode) in the vpc-networking README and COMMANDS.md; document the gw_lrp_range_* fields. |
||
|
Julian Sommer
|
8fe8a9436c | feat(ec2): EC2 Launch Templates (#498) | ||
|
Josh Siviour
|
eac6f449fc |
fix(e2e): hard-cap runSSHQuiet ssh process to stop EIP probe hang (#481)
The EIP "verify unreachable after disassociate" step polls runSSHQuiet inside EventuallyErr(90s), but a wedged ssh made os/exec Wait block for 14 minutes — EventuallyErr cannot interrupt an in-flight CombinedOutput, so one probe consumed the whole 30-min test-binary budget and panicked. The panic skipped t.Cleanup, leaking the EIP and NAT-gateway VMs, which held single-node capacity and starved the lb suite's sys.micro launch (the daemon unsubscribes the full topic, so NATS returns no-responders). Bound the ssh child at the process layer with exec.CommandContext (30s deadline) plus WaitDelay, returning a descriptive timeout error. This is version- and phase-independent, unlike ssh's own ConnectTimeout. 30s is safe for all callers (worst legit call is ~15s) and well under the 90s poll budget, so a wedged probe now yields a clean bounded failure with cleanup intact. |
||
|
tomnewton-mulga
|
4f5fd88c86 |
feat: snapshot ami (#279)
* feat(gpu): add H200/RTX Pro 6000 SE models and fix MIG host resource allocation - Fix H100 SXM/PCIe device ID labels swapped in models.go (10de:2330=SXM, 10de:2331=PCIe) - Add H200 SXM (10de:2335) to knownModels, computeModels, and migCapableModels - Add NVIDIArtxPro6000BlackwellSE to definitions.go mapped to g7e family (matching AWS g7e instance type) - Add MIGHostResources() policy table: 4 vCPUs / 32 GiB host RAM per compute slice, covering all known NVIDIA MIG profiles - GenerateMIGTypes now emits non-zero vCPU/memory for EC2 instance type descriptors * fix(daemon): enforce CPU/memory limits for MIG instance types Whole-GPU passthrough remains bypass-only (gpuManager.Claim is the sole gate). MIG types now fall through to canAllocateCount so host CPU and RAM are checked before launch — prevents over-committing a host where one physical GPU backs multiple concurrent slices, each with non-trivial vCPU/memory allocation. * feat: Add snapshot image promotion flow * fix: Small changes to viperblock/spinifex so snapshot captures delta properly and instances can boot from snapshot AMIs * fix: test coverage and more consistent prefix check * fixed testing * more tests * fix: added new graceful process killer in utils, bumped NATS timeout for testing * fix: complete at-rest encryption launch + volume lifecycle (#361) * chore: bump viperblock to encrypted-snapshot VolumeUUID nonce fix Pull viperblock 1044c2a (feat/snapshot-uuid-nonce-fix) so E2E builds against the fix for the encrypted AMI snapshot metadata tag-verify failure (cipher: message authentication failed) on instance launch. Pairs with viperblock PR #22. * fix: forward viperblock master key to nbdkit mount path viperblockd loaded the master key but never passed it to the nbdkit plugin it spawns for volume mounts, so encrypted boot volumes opened in cleartext and failed LoadState with an encryption-state mismatch. Forward cfg.EncryptionKeyFile into NBDKitConfig and emit the plugin's encryption_key_file= argument when set. * chore: bump viperblock to fresh-volume VolumeUUID mint fix * chore: bump viperblock to snapshot-link LoadState ordering fix * fix: route encrypted-volume config writes through viperblock keyholder config.json is a sealed VBState whose AES-GCM tag and StateSeqNum-derived nonce can only be advanced by the master-key holder that owns the volume. The EC2 volume handler wrote it directly to S3, so mergeVolumeConfig had to refuse encrypted volumes (rewriting would strip the tag) — bricking UpdateVolumeState (attach in-use / detach available) and ModifyVolume. Route encrypted-volume config updates through viperblockd, mirroring the ebs.snapshot idiom: per-volume ebs.config.{volumeID} hits the owning node's live VB (the single StateSeqNum writer); on ErrNoResponders the ebs.config queue group opens the detached volume exclusively and reseals. Cleartext volumes keep the direct-S3 fast path. No crypto leaves viperblockd. mulga-siv-323 * fix: unwrap encryption envelope when snapshot handler reads config.json CreateSnapshot and snapshotInUseByVolumes decoded config.json straight into a VBState. For encrypted volumes config.json is an at-rest envelope {payload, authtag}; a raw decode yields a zero VBState (SizeGiB==0), so CreateSnapshot 500s on the size guard and the in-use scan silently drops encrypted volumes' snapshots. Unwrap via viperblock.StateBody first, mirroring the volume handler's mergeVolumeConfig path. mulga-siv-324 * test: cover viperblockd ebs.config handlers Unit/integration coverage for applyConfigUpdate, makeConfigUpdateHandler (ebs.config.{volumeID}), and the ebs.config queue-group fallback: grow-only size reconcile, live-VB apply, invalid JSON, apply errors, and the detached-volume open path. Clears the diff-coverage gate (68.8%). * chore: pin viperblock to dev tip (#22 merged) * fix(viperblockd): seal volume block-map to predastore on detach + durability e2es (#371) * feat: cloud-init EC2 datasource (#411) * feat: Add snapshot image promotion flow * fix: Small changes to viperblock/spinifex so snapshot captures delta properly and instances can boot from snapshot AMIs * fix: test coverage and more consistent prefix check * merge changes * chore: remove dead SnapshotSub field and SIGTERM snapshot remnants from viperblockd SnapshotSub was added during the NATS/SIGTERM snapshot attempts but is never set post-rebase. Removes the field from MountedVolume and the two nil-checked unsubscribe blocks. Re-adds the nbd import which is still needed for NBDKitConfig. * fix: post-rebase gofmt fixes and remove dead snapshot handler code Split gofmt-squashed lines in go.mod, vm_adapters.go, nbd_test.go, viperblockd.go, volumes.go, and volume_durability_test.go. Removes the dead unix-socket snapshot handler tests from viperblockd_handlers_test.go. * test: add viperblock snapshot S3 coverage tests Exercises snapshotVolume, snapshotRunningVolume, and snapshotStoppedVolume past the nil-config guard using an httptest server that returns 403. The AWS SDK credential check fails synchronously without network I/O, keeping the tests fast while covering the S3Config construction, master key load, and viperblock.New code paths. Brings diff coverage from 45% to 69.9%. * fix: cloneAMIToVolume reads config via format-agnostic path LoadStateRequest expects a viperblock binary envelope (encrypted), but CreateImageFromInstance writes the AMI config as plain JSON via putAMIConfig. Replace LoadStateRequest with s.amiLoader.GetAMIConfig which uses viperblock.StateBody to handle both formats, matching the path used by ListImages and all other AMI readers. * fix: pin viperblock to feat/snapshot-ami pseudo-version for LoadLiveCheckpoint go.mod referenced v1.10.0 (released from dev) which pre-dates LoadLiveCheckpoint. CI builds without go.work so it resolved to the published version and failed. Pin to the snapshot-ami HEAD pseudo-version until the branch is merged and tagged. * fix: replace indexOf with slices.Index, revert e2e runner and viperblock version - nbd_test.go: remove local indexOf helper, use slices.Index from stdlib - e2e-suites.yml: revert ubuntu-latest back to ubuntu-26.04 with correct formatting - go.mod: revert viperblock back to v1.10.0 (will resolve once viperblock PR merges) * fix: remove pseudo-version pin from go.sum * feat: drain viperblock via snapshot socket before CreateSnapshot Before LoadLiveCheckpoint(), dial {data_dir}/viperblock/{vol}/snapshot.sock and wait for DrainToBackend() to complete (30s timeout). This ensures WAL chunks and the live checkpoint are current in S3 before the snapshot VB reads them. If the socket is absent the instance is stopped and Close() already drained. fix: socket path used s.config.WalDir which is empty in the daemon config; changed to s.config.DataDir + "/viperblock" to match the path the NBD plugin uses when creating the socket. * feat: wire snapshot/AMI UI actions and add PromoteImage NATS endpoint - Register as AMI from snapshot detail (RegisterImage with BDM pointing at snapshot) - Deregister from image detail (DeregisterImage with navigate-away on success) - Promote to System from image detail (admin-only, irreversible via UI) - NATS subject spinifex.image.promote wired through gateway → daemon → existing admin.PromoteSystemImage - signedFetch extended with optional params to support action-keyed extra fields * refactor: address PR review on PromoteImage types and subscription ordering - Merge PromoteImageInput/PromoteImageOutput into PromoteImageOpts/PromoteImageResult (add JSON tags; drop the duplicate pair) - Inline the input variable in gateway/spx/promote_image.go - Move spinifex.image.promote subscription to the spinifex.* group in daemon.go * test: cover snapshotVolume/Running/Stopped error paths for diff coverage Add encryption-key-load-error and LoadState-failure tests for the three new viperblock snapshot paths (snapshotVolume in snapshot service, snapshotRunningVolume and snapshotStoppedVolume in image service). Diff coverage: 58.8% → 65.5%. --------- Co-authored-by: Josh Siviour <mrk@mulgadc.com> Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com> |
||
|
Josh Siviour
|
491f5fa7b9 |
fix: claim gateway cr-port on every reconcile miss + surface node launch failures (siv-499) (#463)
* test(e2e): capture SNAT datapath on NAT GW egress failure TestNATGateway captured nothing when the private-VM egress probe failed, so the baremetal nightly's intermittent 100% packet loss left no OVN/OVS state in the failure bundle to root-cause it. Register harness.OnFailure before the egress-verify poll to dump the SNAT datapath (OVN show, NAT rules, port bindings, br-int flows, conntrack, ARP) keyed on the NAT GW external IP, matching the eip/vpc datapath tests. Also add a br-ext localnet flow capture to dumpDatapathState: br-int flows miss the egress hop where SNAT'd traffic leaves for the physical uplink, which is exactly where an egress black-hole would show. * test(e2e): add OVN pipeline diagnostics to NAT GW egress capture Capture the gateway router's per-stage logical-flow packet counts (ovn-sbctl --stats lflow-list), MAC_Binding, static routes and the VM's logical switch port alongside the existing OVS/conntrack/ARP dumps. When the SNAT flow reads n_packets=0 while the guest transmits, these pinpoint which pipeline stage drops the packet upstream of lr_out_snat. * test(e2e): dump full lflow-list + datapath ct/flows for egress RCA The IP-filtered lflow dump hides the default-drop ACL flow (its match carries no IP), so a pre-SNAT egress drop at ls_out_acl only showed the allow-flows at n_packets=0, never the deny that ate the packets. Replace it with a full ovn-sbctl --stats lflow-list, echoed so it survives in the CI stdout log. Swap the conntrack capture to ovs-appctl dpctl/dump-conntrack (the standalone conntrack binary is absent on the baremetal runner) and add dpctl/dump-flows -m for kernel megaflow hit counts. * fix: claim gateway cr-port on every reconcile miss and stop masking launch failures ensureGatewayClaimed nudged ovn-controller recompute only once then polled until timeout, so a fresh VPC bring-up or chassis flap that fired the single nudge too early left the gateway chassisredirect port permanently unclaimed. Recompute on every miss instead, mirroring ensureGuestPortDatapath. aggregateResults and distributeInstancesSpread masked any node launch failure (including the resulting 5-minute launch timeout) as InsufficientInstance Capacity. Surface the real node error when one exists; keep the capacity code only for genuine capacity shortfalls. |
||
|
Julian Sommer
|
988a9b305c | feat: EC2 Spot Instances v1 (mock SIRs over RunInstances) (#448) | ||
|
joshsiv-mulga
|
3046240915 |
fix(e2e): ENI reconcile test SSHes as ubuntu to match stock AMI default
PR #411 (cloud-init EC2 datasource) dropped the cloud-init ISO that created the ec2-user account and baked the harness SSH key; guests now boot the stock Ubuntu gold AMI (default user ubuntu) bootstrapped via the Ec2/IMDS datasource. eni_reconcile_test.go was the lone single-suite test still SSHing as ec2-user, so TestENIHotplugReconcile failed publickey at the first SSH. Align it with every other single-suite test. |
||
|
Julian Sommer
|
8a94886b13 | feat: cloud-init EC2 datasource (#411) | ||
|
Josh Siviour
|
5e58645192 |
feat(eni): hot-plug reconciler + orphan sweep + e2e (Sprint 3d/3e) (#417)
* feat(eni): hot-plug reconciler + orphan sweep (Sprint 3d) Add eniReconciler, a node-local vm.Reaper that converges ENI hot-plug state across a daemon restart or crash: KV ENIRecord <-> in-memory PCIe slot map <-> guest query-pci device list. Resolves records stuck in attaching/detaching, re-adopts the slot map after restart, and reaps orphan QMP devices and slot entries. A staleness gate (new ENIRecord.AttachmentStateAt + 30s threshold) keeps the reconciler off records a live attach/detach handler is mid-pipeline on. Wired into the existing GarbageCollector backstop. * test(eni): unit-cover hot-plug reconciler vm helpers * test(eni): e2e reconciler convergence across daemon restart (Sprint 3e) |
||
|
Bryn Mailer
|
15dede64ad |
feat: predastore compaction E2E (#407)
* feat: add predastore compaction E2E
Inject a short compaction interval into test-cluster predastore config via a
new --predastore-compaction-interval flag on admin init/join, gated so
production output stays byte-identical when unset. Add a harness helper that
drives write-then-delete churn against predastore S3, brackets it with
du-over-SSH usage, and asserts end-of-churn usage settles below 70% of the
churn peak; wire it into the multinode suite as TestPredastoreCompaction.
* fix: compaction E2E gate writes churn to its own bucket
The reclaim gate signed PutObject against the config-defined system
'predastore' bucket, which belongs to account 000000000000 with only a
static [[auth]] grant. The harness admin identity is a different IAM
account, so predastore returned 403 AccessDenied cross-account.
Create a per-run bucket the admin identity owns (AdministratorAccess)
and churn into it. Objects still erasure-code into distributed/nodes/
node-* so the du-based reclaim measurement is unchanged. The gateway
(:9999) does not proxy S3, so the client stays predastore-direct.
* test: run predastore compaction gate before destructive node tests
The compaction reclaim gate was declared last in the multinode suite, so it
measured shard-store usage after NodeFailure/NodeRecovery had torn down and
restarted node2. NodeRecovery only gates on NATS/gateway/EC2 readiness, not
predastore, so node2's shard store was still settling and the du baseline came
back empty. Move the test before the destructive node tests so it runs against
a pristine, fully-converged cluster primed by the volume tests.
* fix: race-free SSH output capture + empty-tolerant du probe
The compaction reclaim gate crashed with ParseInt("") when node1's du
probe returned empty combined output. Harden the probe and the SSH layer:
- ssh.Run used one bytes.Buffer for both session.Stdout and session.Stderr,
which x/crypto/ssh copies in two concurrent goroutines — a data race on a
non-concurrent-safe type. Switch to session.CombinedOutput, which guards the
combined stream with a mutex-wrapped singleWriter.
- shardStoreUsageBytes now terminates the remote command with a sentinel echo.
A missing sentinel is reported as output lost in SSH retrieval; a present
sentinel with no number is treated as a 0 measurement (anomalous, logged)
instead of crashing the gate.
The sentinel lets the next run distinguish a transport-layer drop from a
genuinely empty remote measurement.
* test(e2e): replace predastore du-reclaim gate with object lifecycle contract
The multinode compaction gate measured shard-store disk usage over SSH (du)
and asserted reclaim after churn. It could never pass: segments only seal at
the 4 GiB roll threshold, so test-scale churn produces no compactable segment,
and the du probe read 0 bytes before and after. Reclaim correctness is already
covered deterministically by predastore's store-level unit tests.
Delete AssertPredastoreCompaction and its du/churn/poll/sentinel machinery,
the harness unit tests that exercised them, and the multinode TestPredastore
Compaction entry. Add a single-node TestPredastoreObjectLifecycle that checks
the user-visible contract: an object is readable after PutObject, gone after
DeleteObject (GET returns NoSuchKey and ListObjectsV2 is empty), and the store
keeps serving fresh writes. newPredastoreS3 is retained, refactored to take a
host so it works outside the multinode cluster type.
* test(e2e): drop orphaned multinode TestPredastoreCompaction
Complete the du-reclaim -> object-lifecycle refactor: remove the
multinode TestPredastoreCompaction caller (and its now-unused context
and harness imports) that still referenced the deleted
harness.AssertPredastoreCompaction, which broke e2e-build. Wire the
single-node TestPredastoreObjectLifecycle entry that replaces it.
|
||
|
Julian Sommer
|
eced3f733a | feat: split IAM e2e (#372) | ||
|
Julian Sommer
|
9288ecc7dc | chore: delete no op tests + speedup e2e | ||
|
Julian Sommer
|
058da3eff4 | fix(viperblockd): seal volume block-map to predastore on detach + durability e2es (#371) | ||
|
Julian Sommer
|
2e9501524b | feat(e2e): more e2e coverage (#365) | ||
|
Josh Siviour
|
9a474220ce |
fix: preserve terminated-instance describe-visibility window in GC reaper (#363)
* fix: preserve terminated-instance describe-visibility window in GC reaper The TerminatedTeardownReaper purged a terminated record the moment it drove the instance's pending NAT/OVN teardown to done. Normal terminations always arrive NAT/OVN-pending, so the first 2m GC sweep completed and purged them immediately, removing the instance from DescribeInstances before clients could observe state=terminated — an AWS-compat divergence that flaked e2e-single TestRunInstancesMultiCount. Stamp TerminatedAt on transition and purge a completed record only once it is older than a 15m visibility window; fresher records are left to the terminated bucket's 1h TTL, keeping the instance describable as terminated. The ADR-0003 §3 purge-on-completion invariants still hold (their seeded records predate the window). Closes mulga-siv-328. * test(e2e): poll intra-SG east-west ping until L2 converges TestSameSGComms fired a single un-retried 'ping -c 3 -W 2' immediately after the source instance's SSH became reachable, but the ICMP target is the destination's east-west data plane. Two instances boot in parallel and the destination guest NIC / OVN port flows come up several seconds after the EC2 running transition the launch waits on. When the probe wins that race the source kernel reports 'Destination Host Unreachable' (ARP unresolved), not an ACL drop, and the test flaked. Replace the one-shot ping with pingConverged, a bounded poll that retries until 0% packet loss or 45s. A genuine self-ingress/datapath break still stays at 100% loss for the full window and fails, so coverage is preserved. |
||
|
Julian Sommer
|
1f7adcc4dd | chore: more e2e tests for IAM + STS + IMDS (#364) | ||
|
Josh Siviour
|
1873f45d26 |
Resilient resource lifecycle (ADR-0001..0006): EKS multi-node teardown, idempotent deletes, GC backstops (#357)
* fix: idempotent Delete* across VPC family + volume (RLC1) Delete handlers now return empty success when the resource is already absent (nats.ErrKeyNotFound), instead of *NotFound. Transient read errors still map to ServerInternal. Covers DeleteVpc, DeleteSubnet, DeleteSecurityGroup, DeleteNetworkInterface, DeleteInternetGateway, DeleteNatGateway, DeleteRouteTable and DeleteVolume. Makes destroy retries converge instead of failing mid-graph, removing a source of orphaned/zombie resources between deploys. Adds a TestRLC1_*DeleteIdempotentOnAbsent invariant per package and updates the daemon delegation table; supersedes the old absent->NotFound tests. * feat: prune orphan ENI LSPs in reconcile applyPorts (ADR-0003 §4) applyPorts was create-or-update only, so a guest LSP whose spinifex:eni_id no longer maps to an intent ENI survived instance terminate and VM-host reinstall — the orphan-LSP capacity-drift root cause. It now takes pruneOrphans (mirroring applySGs): on a full reconcile it lists LSPs and tears down each managed ENI port absent from intent via the composed topology.DeletePort cascade (clears port-group memberships, then deletes the LSP). The apply-only startup path does not prune, preserving in-flight ports until subscribers converge. Adds ListLogicalSwitchPorts to the OVN client (live + mock). Invariant TestRLC5_OrphanLSPPrunedAfterReconcile guards ADR-0003 §4; a startup-race test guards the apply-only path. * feat: idempotent + tracked instance terminate (ADR-0003 §1/§2) Make terminate idempotent across all three layers and break the un-terminable-ENI deadlock, and record per-dependency teardown so the GC backstop can later tell a fully-reaped instance from a dangling one. - vm.Manager.Terminate: absent / already-terminated / shutting-down now return nil instead of NotFound / invalid-transition (ADR-0003 §2). - VM.Teardown map + teardown.go marks: sync steps stamped Done/Failed via cleaner error returns; fire-and-forget OVN LSP and NAT rule stamped Pending for the reconciler / GC reaper (ADR-0003 §1). - InstanceCleaner: DeleteVolumes / ReleasePublicIP / DetachAndDeleteENI / RemoveFromPlacementGroup / ReleaseGPU now return error. - vpc ForceDeleteInstanceENI bypasses the in-use guard for the owning instance, breaking the failed-detach deadlock; plain delete still rejects an in-use ENI held by another instance. - gateway TerminateInstances: KV-health-gated idempotent absence (a failed terminated-bucket query does not fabricate success, ADR-0003 §3). - Invariants: TestRLC1_TerminateIdempotentOnAbsent, TestRLC4 teardown-map stamping + failed-dependency, TestRLC_ENINonDeadlock. * feat: reconciler GC backstop framework + terminated-teardown reaper (ADR-0003 §3) Add the shared reality->desired garbage-collection framework (inverse of Restore) and its first reaper, closing the gap 295.3 deferred: nothing retried pending/failed teardown or purged the terminated record when its dependents reached done. - vm/gc.go: GarbageCollector + Reaper interface (Class/Scope/Sweep). KV-health gated — holds and alarms when the desired-state cannot be trusted, never reaps against it. ClusterWide reapers run only on the elected leader (optional elector); NodeLocal run on every node. Start sweeps immediately then every GCInterval. - vm/teardown_reaper.go: TerminatedTeardownReaper (node-local). Scans this node's terminated records, re-drives each not-done dependent through the idempotent cleaner, and purges the record once all reach done. Records already complete on arrival are left to the bucket's 1h TTL. - StateStore gains DeleteTerminatedInstance (JetStream impl already existed). - daemon: wires the GC after the pending watchdog, gated on a cheap JetStream AccountInfo health probe. - Cluster OVN-LSP arm already lands via the reconciler drift prune (295.2). - Invariants: TestRLC5_GCRetriesPendingTeardownToDoneThenPurges, TestRLC5_GCNeverPurgesIncompleteRecord, TestRLC5_GCHoldsWhenKVDegraded, TestRLC5_GCSkipsOtherNodesRecords. * feat: VPC-family dependency guards (ADR-0004 rule #3) DeleteSubnet blocks while a live ENI attachment (a resident instance) remains; available/orphan ENIs do not pin it. DeleteNetworkInterface guards on the structured attachment (eniIsLiveAttachment), not a bare in-use string. DeleteNatGateway blocks while a live route table forwards to it and disassociates its EIP keeping the allocation (AWS parity); CreateNatGateway now marks the EIP associated so the disassociate is real and a second NAT GW cannot reuse the allocation. Binding invariants: TestRLC3 dependency-guard completeness across the subnet/ENI/NAT-gateway handlers. * feat: EBS volume data-safety GC override (ADR-0005) The shared GC backstop reaps actual-desired for every other class; for volumes that default is forbidden because they carry customer data. Add VolumeLeakReaper (node-local, rides the 295.5 GC): it marks a volume left attached to a definitively-gone instance as orphaned (a non-destructive tag) and alarms, but NEVER issues a delete. The daemon feeds it only this node's terminated instances whose volumes-teardown failed, so the shared predastore scan can never false-mark another node's live-instance volume. DeleteVolume idempotency + the detach/snapshot guards + the DeleteOnTermination teardown retry already landed (295.1/295.3/295.5). Binding invariants: TestRLC5 (GC marks+alarms, never deletes data), TestRLC3 (detach-before-delete). * test(elbv2): lock ADR-0002 §3 live-only TG in-use guard (mulga-siv-295.4) * feat(eks): idempotent DeleteCluster on absent + ADR-0006 lifecycle invariants (mulga-siv-295.8) DeleteCluster returns empty success when the cluster meta is already swept (Common Resource Lifecycle Contract #1), so tofu destroy retries converge. Adds TestRLC1 (idempotent delete), TestRLC2 (billable-before-KV-sweep ordering) and TestRLC3 (NLB target-group no-orphan, riding ADR-0002/172). * fix(eks): GC reaper terminates orphan control-plane VM after cluster delete (mulga-siv-294) A normal DeleteCluster can leave a running control-plane VM behind (terminate completes async; a daemon restart re-launches it from saved running-state) after the cluster meta is swept — a billable orphan with no owner. Add EKSBillableReaper on the 0003 GC backstop: a running managed-by=eks CP VM whose cluster meta is definitively gone is terminated. The CP ENI now carries the customer-account tag so a system-account VM can be mapped back to its cluster meta; the reaper never reaps on uncertainty (ENI gone/untagged or meta present). * feat: reclaim orphan EKS NLB + NAT-GW EIP in billable reaper The billable GC reaper terminated an orphan control-plane VM but left the cluster's NAT-gateway EIP and NLB front-end NAT behind when the meta was gone. After a successful terminate, run the name/tag-driven DeleteClusterNLB + DeleteClusterCPVPC teardown (idempotent, meta-independent) so the billable NAT-GW EIP and NLB NAT are reclaimed too. ADR-0006 §5. * fix: persist NAT-GW & route-table tags and reorder CP-VPC teardown to reclaim EKS NAT-GW EIP CreateNatGateway and CreateRouteTable both dropped input.TagSpecifications, so the tag-driven DeleteClusterCPVPC teardown could not discover the cluster NAT gateway or its private route table. The explicit DeleteNatGateway + ReleaseAddress were skipped and the billable NAT-GW EIP leaked under the system account on every clean DeleteCluster. Three fixes: - CreateNatGateway persists ExtractTags(TagSpecifications, natgateway). - CreateRouteTable persists ExtractTags(TagSpecifications, route-table). - DeleteClusterCPVPC tears down route tables before the NAT gateway, so the rule #3 route-ref guard (DependencyViolation / InvalidAddress.Locked) is not tripped and the EIP releases. Adds tag-persistence regression tests for both create paths and a teardown ordering invariant (TestRLC5). Validated live: NAT-GW EIP released cleanly, no DependencyViolation/Locked. * fix: route EKS control-plane VM terminate cross-node + DELETING teardown backstop Multi-node EKS cluster delete wedged in DELETING and stranded the billable NAT-GW EIP. Two root causes, both invisible on a single-node box: Cross-node terminate was never routed. TerminateSystemInstance was node-local, so terminating a CP VM owned by another node returned NotFound (treated as idempotent) and the teardown deleted a still-attached ENI (InvalidNetworkInterface.InUse). That error short-circuited purgeClusterInfra before the managed CP-VPC teardown that releases the NAT-GW EIP. Split TerminateSystemInstance into a local body plus a routing wrapper that issues system.TerminateInstance.<id> to the owning node. Bind that subscription for every system-managed VM (elbv2 + eks) via the OnInstanceUp funnel, so CP VMs placed on the coordinator's own node (and recovered after a daemon restart) are reachable too, not only remotely-launched ones. No DELETING backstop. A failed synchronous DeleteCluster left the cluster DELETING with nothing to re-drive it. Add EKSDeletingReaper (vm.Reaper, class eks-deleting) that re-runs purgeClusterInfra for clusters wedged past a min-age, gated by the per-cluster reconciler leader lease. Stamp ClusterMeta.DeletingSince on the transition into DELETING. TerminateK3sServerVM additionally waits out any residual ENI-InUse with a bounded budget as defense-in-depth. Validated live on env19 3-node: first delete-cluster converges in 29s, both remote CP VMs terminated cross-node, NAT-GW EIP released, backstop never fires. * fix(network): gate gateway reconcile on real datapath reachability Post-reboot the external OVN datapath can stay dead while every control-plane signal is green: the chassisredirect SB binding is claimed (up=true), NAT rows are present and the host neigh is primed, yet ovn-controller leaves stale gateway/localnet flows so EIP north-south and the VM->gateway heartbeat forward nothing. ensureGatewayClaimed only nudges recompute when the binding is unclaimed, so a claimed-but-dead datapath is never detected or repaired. Add ensureGatewayDatapath: after the SB claim converges, probe the gateway LRP IP (OVN answers ICMP natively, no guest or security-group dependency), and on a miss force one ovn-controller recompute and re-probe until a short deadline. Reuses the existing recompute nudge and no-ops when no verifier is wired or no gateway IP resolves. * fix: self-heal external uplink after reboot boot-race Post-reboot, ovs-vswitchd/ovn-controller order after the passive network.target rather than network-online.target, so they start concurrently with systemd-networkd. Under a heavy boot the WAN-glue veth (veth-wan-ovs) can come up admin-down before networkd brings it up, leaving br-wan with no carrier and the external datapath dead while every control-plane signal reconciles green. Broaden the datapath health-gate remediation from recompute-only to RepairDatapath: re-assert the veth admin state (covers the admin-down link, which a flow recompute cannot revive) then recompute (covers a stale gateway ofport). Idempotent; the veth ops are skipped in physical-uplink mode where the device is absent. Also capture ip -d link (veth admin/carrier) and ovs-ofctl show br-ext (ofport) in the reboot diag bundle to disambiguate the two failure shapes from CI artifacts. * fix: restore AWS-faithful Delete* NotFound; move idempotency to teardown callers (mulga-siv-306) RLC1 (f9218982) made EC2/VPC/EBS Delete* handlers return success on an absent resource, breaking AWS compatibility and the cross-account isolation signal (a cross-account delete reads as absent -> success instead of *.NotFound). Restore the AWS contract at the API while keeping lifecycle convergence: - Handlers return canonical *.NotFound on absent: DeleteVpc, DeleteSubnet, DeleteSecurityGroup, DeleteInternetGateway, DeleteNatGateway, DeleteRouteTable, DeleteVolume, and the public DeleteNetworkInterface (ForceDeleteInstanceENI keeps its idempotent force path, ADR-0003 section 2). - Add awserrors.IsNotFound; destroy orchestration (instance terminate, EKS CP-VPC teardown, EKS IGW/SG cleanup, k3s rollback, ELBv2 ENI/SG teardown) tolerates NotFound so re-issued/double-targeted deletes converge. - Flip the RLC1 unit invariants to assert *.NotFound on absent; revert the daemon round-trip cases; add IsNotFound + CP-VPC convergence tests. ELBv2 deletes and TerminateInstances stay idempotent (AWS itself is). * fix(network): gate ingress EIP datapath on guest SB Port_Binding convergence TestMultinodeGuestSSH intermittently times out (Connection timed out, not #355's reset-by-peer) connecting to a fresh trio instance's public IP after the suite churns instances across all three chassis. The instance is running and AddEIP has installed the DNAT row, flushed/primed the host neigh, and barriered the gateway-chassis flow — but none of that proves the gatewayLRP->guest geneve hop: until the guest ENI's southbound Port_Binding is up on its hosting chassis the cross-node flow is missing and the DNAT-translated packet blackholes. Add ensureGuestPortDatapath, the guest-side analog of ensureGatewayDatapath: after AddEIP, probe the guest LSP's SB Port_Binding (ovn-sbctl up column); on a miss force one ovn-controller recompute (binds the port / installs the remote flow) and re-probe until a short deadline. Recompute, not RepairDatapath: this is flow/binding convergence, not a WAN-uplink fault. No-op when no verifier is wired or the EIP carries no guest port. spec.PortName is populated for every associated EIP regardless of NAT mode. Extends GatewayClaimVerifier with GuestPortUp and wires the gate into applyEIPs so it runs in the continuous reconcile loop on every node. * Revert "fix(network): gate ingress EIP datapath on guest SB Port_Binding convergence" This reverts commit 728b1a9cec6eaf5c7ef5e9eda4469f239b4d8fc0. * fix(network): key EIP DNAT delete on external IP, not logical IP A dnat_and_snat row's identity on the VPC gateway router is its external IP. The logical (private) IP is not a stable key: the EIP pool round-robins and instance private IPs are reused as instances recycle, so the same logical IP is held by different EIPs over time. DeleteEIP deleted the row by logical IP, so a stale or retried delete for a freed EIP clobbered whichever live EIP now held that recycled private IP -- the next owner's row vanished with no re-add and its public IP went dark (SSH SYN never left br-wan: Connection timed out). Lifecycle-resilience teardown/GC retries amplified this by firing vpc.delete-nat repeatedly. Key the delete on the external IP via DeleteNATByExternalIP and skip when no external IP is supplied (no EIP means no dnat_and_snat row). A stale delete now targets the freed EIP's own external IP (ErrNATNotFound no-op) and can no longer touch the live row. * refactor: relax DeleteNatGateway to AWS-faithful Remove the route-reference dependency guard (checkNatGatewayRouteRefs) so a NAT gateway deletes even while route tables still forward to it, matching AWS (the route blackholes). The reconciler's loadNATGWs drops the SNAT once the record leaves the active bucket, so a forwarding route is not a delete-blocking dependent. EIP is still disassociated on delete (no leak). * test: invert NAT-GW route-ref invariant to AWS-faithful TestRLC3_NatGatewayBlocksWhileRouted asserted DependencyViolation while a route forwarded to the NAT gateway, which is AWS-infaithful. Replace with TestRLC3_NatGatewayDeletesWhileRouted: delete must succeed whether or not a live route still targets it (the route blackholes; the reconciler drops the SNAT). Per ADR-0004 §2. * fix: tear down NAT-GW SNAT on delete (restore delete-time fan-out) The guard commit (53ed55a9) dropped publishDeleteEventsForNatGateway on the assumption a routed NAT GW could never be deleted. With the route-ref guard removed, deleting a NAT GW while routes still target it left the SNAT in place (applyNATGWs is create-only, so the reconciler never prunes it) and egress kept working — TestNATGateway's post-teardown 'lost internet' check failed. Restore the fan-out: on delete, scan route tables and publish vpc.delete-nat-gateway per associated subnet so egress stops, while the route entry survives as a blackhole (AWS-faithful). * fix: probe EIP datapath in reboot self-heal, not the gateway LRP IP The post-reboot gateway datapath self-heal probed the gateway LRP IP for liveness, which OVN answers natively at the router — so it stayed green even when the EIP NAT pipeline was stranded (ofproto/trace dropped EIP traffic at table 0). RepairDatapath never fired and TestRebootResilience timed out, with the drift loop re-probing the same false-green every cycle. Prefer probing an associated EIP: force a fresh ARP of the dnat_and_snat external IP (flush neigh, ping to trigger, read the resolved MAC). OVN answers ARP for the external IP from the gateway chassis independent of the guest, so a resolved MAC proves the WAN uplink forwards and the NAT flows are installed — the signal the LRP IP cannot give. Fall back to the LRP IP for VPCs with no EIP. The existing drift-loop cadence now re-heals a later flow strand. * fix: lb-agent heartbeats over mgmt port, not the WAN/EIP datapath The lb-agent heartbeat to awsgw is control-plane and must survive a host reboot that strands the OVN/EIP north-south data plane. Customer (internet-facing) lb-agents heartbeated over the WAN GatewayURL, riding the EIP datapath, so after a reboot the ALB never reported healthy until the datapath self-healed. System LBs already heartbeated over the on-link br-mgmt NIC; this extends that path to every lb-agent. buildLBAgentEnv now always uses mgmtGatewayURL (on-link MgmtBridgeIP, the mgmt NIC the lb-agent VM already carries), falling back to the WAN URL only when no mgmt bridge exists. Drops the account-id branch. * fix: re-heal guest ENI SB Port_Binding post-reboot so EIP ingress converges Post-reboot every EIP in a VPC goes dark north-south: the DNAT row, host neigh and gateway-chassis flow are all in place (ARP to the EIP resolves) but the gatewayLRP->guest flow is missing until ovn-controller binds the guest ENI's SB Port_Binding, so the DNAT-translated packet blackholes and the public IP is unreachable on an otherwise-running instance. Restore the guest-port convergence gate in applyEIPs: probe the guest LSP's SB Port_Binding up column; on a miss force an ovn-controller recompute and re-probe until a deadline. Unlike the earlier reverted form, recompute on every miss, not once: after a host reboot the guest tap is replumbed several seconds after the reconcile runs, so a single early nudge fires before the port exists and never binds it. The window is widened to span guest boot. This is the southbound analog of the gateway datapath self-heal; the EIP ARP probe cannot detect it because the gateway chassis answers ARP for the external IP even while the guest delivery flow is absent. * fix(network): force post-reboot EIP recompute on guest-port bind, startup-only The reboot suite passed only by accident: the pre-#351 gateway-claim verifier polled the always-chassis-less gateway LRP, so it fired an ovn-controller inc-engine/recompute on every reconcile. That spurious recompute was load-bearing - it reinstalled the centralised gateway DNAT flows that ovn-controller's incremental engine misses on a cold (post-reboot) start. #351 correctly switched the check to the chassisredirect port, eliminating the every-cycle recompute, and the EIP datapath went dark post-reboot with every control-plane signal green. Restore the kick, scoped to the post-reboot startup path: applyEIPs now gates a single recompute on the guest ENI's SB Port_Binding chassis claim - the reliable signal the tap is back, unlike the up column which stays stale-true across a reboot. Drift never recomputes, preserving the #351 fix for freshly attached EIPs. Probe renamed GuestPortUp -> GuestPortBound (chassis, not up). * Revert "fix(network): force post-reboot EIP recompute on guest-port bind, startup-only" This reverts commit 792e318211ef03efbe24febeb470749d88c0b03c. * test(e2e/reboot): capture dataplane diag on ALB serve failure The post-reboot ALB-serve assertion (8.6) failed with no flow/ARP evidence: the OVN/OVS dataplane dumps only ran on the 8.5 targets-unhealthy path, but 8.5 passes on the daemon's own health view while the north-south EIP datapath is dark, so the serve failure (8.6) left nothing to localise the drop. On a serve failure, capture dumpALBProbe/dumpOVNDataplane/dumpOVNClaimState/ dumpHeartbeatDiagnostics/dumpAppDiagnostics before failing. Add to the OVN dataplane dump: dump-flows br-ext and fdb/show br-ext (north-south bridge, previously only br-int), and a real ARP probe that flushes the neigh entry first so primeReachability's pre-seeded reachable entry can no longer mask an unanswered ARP. Pure diagnostics; no production change. * test(e2e/reboot): decisive datapath diag — untruncate br-int, real-ingress ofproto/trace, conntrack + tap capture * test(e2e/reboot): add south->north return-path ofproto/trace to localize egress drop * test(e2e/reboot): derive LRP MAC from realized flow + remediation experiment (recompute vs ovn-controller restart) * test(e2e/reboot): dump persisted SB MAC_Binding + external MACs; flush experiments (conntrack vs MAC_Binding) to isolate recompute-resistant return-path drop * test(e2e/reboot): untruncate Port_Binding + enumerate DGP/chassisredirect claim state (does cr-gw row exist + resident chassis) * test(e2e/reboot): dump NB source of return-path drop (lflow stage, NAT external_mac/options, LR policy, LSP port_security) * fix: exempt public-IP instances from subnet egress drop gate (mulga-siv-311) Post-reboot reconcile installs the priority-1100 subnet egress drop gate for an IGW-attached subnet that has no 0.0.0.0/0 route, but nothing punched the subnet's public-IP instances through it. lr_in_policy runs before lr_out un-DNAT/SNAT, so an inbound connection's reply still carries its private source when it hits the gate and is dropped. The ALB/EIP datapath goes dark south->north post-reboot while NAT, ARP, chassis and guest-port binding all read green; the pre-reboot path survives only because the event-driven gate never evaluates a subnet created after IGW attach. Install the existing priority-1200 /32 reroute (above the gate) for every public-IP instance during reconcile. Reroute only: the instance's dnat_and_snat already supplies SNAT, unlike EnsureSystemInstanceEgress which also adds a plain snat. Public IPs come from two disjoint sources -- auto-assigned/ELB addresses on the ENI record and user EIPs in the EIP bucket -- so drive the exemption from both intent.Ports (new PortSpec.PublicIP) and intent.EIPs. * refactor: scope public-IP drop-gate exemption to gated subnets Only install the priority-1200 reroute when the instance's subnet actually carries a drop gate. Routed subnets egress via their priority-1000 reroute and need no exemption, so bounding on gate presence keeps the change inert for IGW-routed and NATGW-routed subnets (no redundant or overriding policy). Adds a negative test. * refactor: drop unused RemoveEIPInstanceEgress No production caller; the exemption is install-only in reconcile. Removed the interface method and impl rather than leave dead API surface. * test: dump guest console + OVN/datapath diag on SG-to-SG client SSH timeout TestSGToSGDatapath waited for client-vm SSH via EventuallyErr and reported only 'exit status 255' on timeout — no guest-side or datapath state. When the fresh public-IP VM is unreachable for the full window the failure was undiagnosable from CI artifacts. Swap the wait to the non-fatal trySSHReady probe and, on timeout, call DumpVPCFlowDiagnostics (guest console tail, ovn-nbctl/sbctl show, NAT rows, br-int flows / conntrack / host ARP keyed on the public + private IP) before Fatal, mirroring vpc_test/baseline_test. Captures persist to the artifact dir for the Stage 2 analyzer bundle. * fix: ownership-scope public-IP teardown to survive IP recycling A live instance's dnat_and_snat rule, host ARP, and IPAM lease were torn down by the delayed garbage-collection teardown of a different, long-dead instance that previously held the same recycled public IP. The 2-min vm/gc instance-teardown reaper re-runs ReleasePublicIP on a stale terminated record; because the external IPAM round-robins a small pool, the IP had already been reassigned to a live instance, and both teardown primitives acted on the IP by value with no ownership check: - policy.DeleteEIP deleted the dnat_and_snat row by external IP alone, discarding the logical IP it receives. - StaticPoolAllocator.Release freed the pool entry by IP value. Scope both to the current owner so a stale or duplicated teardown for a recycled IP is a no-op: - DeleteEIP looks up the row by external IP and skips the delete (and the ARP flush) when its logical IP no longer matches, mirroring AddEIP's idempotency lookup. Defeats both external- and logical-IP reuse races. - Allocator.Release / ExternalIPAM.ReleaseIP take an ownerENIID; the static allocator no-ops a release whose ENI differs from the live allocation, and treats an already-freed owner-scoped release as idempotent. Teardown and rollback callers pass their ENI; user-driven EIP release passes none. Proven via E2E single TestSGToSGDatapath (run 27515293563): client-vm SSH black-holed after its NAT was deleted by the dead instance's GC sweep. * fix(eks): single-flight DeleteCluster teardown per cluster SDK retries of one DeleteCluster get fanned across nodes by the worker queue group, so purgeClusterInfra raced the same cluster on multiple nodes: duplicate ENI/NLB teardown, duplicate TargetGroupNotFound deregisters, and a real double-release risk on the billable NAT-GW EIP. Idempotency kept it correct but noisy. Add a dedicated per-cluster teardown lease (teardownLeaderKey = reconcilerLeaderKey + "/teardown"), distinct from the reconciler lease so a delete never contends with the still-live reconciler that holds the reconciler lease until it observes DELETING and exits. DeleteCluster acquires it after the NotFound-idempotency check: the loser returns the cluster as DELETING (AWS-async semantics) without running the teardown; the winner drives purgeClusterInfra and releases. The DELETING backstop reaper shares the same lease, so at most one purgeClusterInfra runs per cluster cluster-wide. A nil leaderKV (single-node) skips gating, so local behavior is unchanged. * fix: env-driven SSH-ready budget for slow baremetal guests Add SPINIFEX_SSH_READY_TIMEOUT override for the single-node SSH-handshake probes (default unchanged at 3m). Baremetal boots q35 guests slower (OVMF + NBD-backed disks) than CI loop devices and intermittently misses the 3m window; the harness can now widen it to 5m without affecting shared-runner e2e. Applied at all single-node probe sites; failure messages now report the actual budget. * fix: shared external switch to stop per-VPC localnet ARP collision Per-VPC external switches each attached an untagged localnet port to the same physical network (br-ext). OVN cannot keep multiple untagged localnets on one physical network L2-separate, so with two or more concurrent public-IP VPCs ARP for a public IP landed on the wrong VPC's switch and the owning gateway router's ARP responder never fired, leaving the public IP unreachable. Move to a single shared external switch (provider-network pattern): one switch with one localnet on network_name=external. Each VPC's gateway LRP attaches via its own router-type switch port; per-VPC NAT isolation is unchanged. AttachIGW ensures the shared switch idempotently and is migration-safe; DetachIGW removes only the per-VPC port + LRP and preserves the shared switch. DeleteVPC removes the per-VPC gateway switch port from the shared switch so it does not dangle. Reconcile detects attachment via the gateway LRP since the shared switch carries no vpc_id. * fix: owner-scope DeleteEIP by ENI port to survive identical pair recycling A stale or GC-re-emitted vpc.delete-nat for a terminated instance tore down a live instance's dnat_and_snat rule (and flushed the host ARP) when the public IP and private IP both recycled, together, to that live instance in the same VPC. The (external_ip, logical_ip) pair-key matches under identical recycling, so the existing logical-IP guard fell through. Stamp the owning ENI port (spinifex:logical_port) on the dnat_and_snat row at AddEIP, thread the port through handleDeleteNAT -> DetachEIP -> DeleteEIP, and skip the delete when the row's stamped port differs from the incoming one. The IPAM release was already owner-scoped; this mirrors that guard at the NAT layer so any stale delete for a recycled pair is a no-op. Empty port preserves the prior pair-key behaviour. |
||
|
Julian Sommer
|
50b5495d26 | chore: more comment cleanup | ||
|
Julian Sommer
|
1dda4cb37c | feat(gateway): enforce assumed-role session policies (#354) | ||
|
Julian Sommer
|
c4926d423f | feat: IMDS public-keys (#350) | ||
|
Julian Sommer
|
15cd1a1ad7 | feat: sts get session token (#321) | ||
|
Josh Siviour
|
527308c342 |
fix(baremetal): empty modules.dep + capacity-aware single-node e2e (#305)
* feat(elbv2): localize internal-LB heartbeat failures for baremetal debug Add diagnostics so a stuck-in-provisioning internal LB can be pinned to one of three points on the heartbeat path: - lb-agent failure log now carries the gateway target + region, so the microVM serial console shows whether the heartbeat hit a transport error (never reached AWSGW) or an HTTP-level rejection. - AWSGW LBAgentHeartbeat logs receipt (debug) and warns on the previously-silent not-found/account-mismatch reject path, so the journal distinguishes 'heartbeat arrived but cannot transition' from 'heartbeat never arrived'. * test(e2e/lb): fail fast on stuck internal LB + probe fw_cfg on microvm boot - lb suite: gate the remaining internal subtests on Internal_ALB. The internal LBs share one mgmt-return datapath and fixture, so if the first never reaches active the rest time out identically — skip them instead of burning ~5m per subtest (~20m saved on a known-bad run). - microvm init.sh: widen the fw_cfg netcfg poll window and, on the not-available path, dump DMI / loaded modules / qemu_fw_cfg by_name listing / dmesg fw_cfg to the serial console before init exits, so a boot panic is diagnosable from the captured console log. * fix(microvm): insmod fallback for fw_cfg + fail build on bad modules.dep Baremetal-built microVM images shipped an unusable modules.dep (the build host's depmod errors were swallowed), so busybox modprobe in the guest init silently no-oped, qemu_fw_cfg never loaded, /sys/firmware/qemu_fw_cfg was absent, init could not read opt/spinifex/netcfg, and the LB microVM panicked before lb-agent ran — leaving internal LBs stuck in provisioning. init.sh: load_mod() tries modprobe, then falls back to insmod by explicit path (.ko/.ko.gz) which needs no modules.dep, so the guest boots regardless of which host built the image. build-microvm-image.sh: require depmod, drop the error swallow, and assert modules.dep is non-empty after depmod so a boot-broken image fails the build loudly instead of silently. * fix(microvm): decompress kernel modules before depmod Debian trixie's kmod 34 depmod silently emits an empty modules.dep when fed Alpine's gzip-compressed (.ko.gz) modules — reproduced: 928 modules in, 0-byte modules.dep out. The guest's busybox modprobe then no-ops, qemu_fw_cfg never loads, and internal LB microVMs panic before lb-agent runs (stuck provisioning on bare metal; GitHub-hosted runners ship a kmod that reads .gz, so normal CI was unaffected). Decompress kept modules to plain .ko after the keep-list strip, before depmod. Host depmod then produces a usable map regardless of its gzip support, and the guest loads modules without a decompressor. * feat(e2e): capacity-aware scheduling for dense single-node runs Denominate daemon schedulable capacity in physical cores instead of logical threads, and auto-serialise e2e launches when the cluster is at capacity, so a single hyperthreaded node stops over-packing guests. Daemon: NewResourceManager sizes hostVCPU from physicalCoreCount() (distinct physical-id:core-id pairs in /proc/cpuinfo) rather than runtime.NumCPU(). Falls back to logical CPUs when topology is absent (non-Linux, container, ARM cpuinfo without core id). On a hyperthreaded box this halves the over-pack: with the unchanged 2 vCPU reserve the i5-12600H schedules 10 vCPU (12 cores - 2) instead of 14, keeping the boot-spike CPU within real cores. Harness: capacityRetryEC2 decorates the EC2 client and, on InsufficientInstanceCapacity, waits and retries RunInstances instead of failing — turning the daemon admission gate into back-pressure so concurrent subtests queue. AWSClient.EC2 is now the wrapped ec2iface.EC2API; EC2Conf keeps the raw *ec2.EC2 for .Config access. Adds unit tests for parsePhysicalCores (hyperthreaded/hybrid/dual-socket/ fallback) and the retry decorator (retry-then-succeed, non-capacity passthrough, deadline). * fix(daemon): SPINIFEX_HOST_VCPU override so small CI hosts can start Denominating capacity by physical cores starves tiny hosts: a 2-physical- core CI runner leaves 0 schedulable after the default 2-vCPU reserve, so NewResourceManager refuses to start and every daemon-constructing unit test fails. The previous runtime.NumCPU() count (4 logical on the same runner) incidentally cleared the reserve floor. Add a SPINIFEX_HOST_VCPU escape hatch (symmetric with the existing SPINIFEX_RESERVED_* overrides) that pins the detected core count for hosts whose /proc/cpuinfo topology misreports, and pin it to 4 in the daemon TestMain so unit tests are deterministic regardless of runner topology. Reserve stays at the default 2. |
||
|
joshsiv-mulga
|
eef337fb80 |
test(e2e): make egress baseline reproduce the real IGW gap (fresh VPC)
The previous new-subnet egress baseline used the default VPC, whose main route table is prebaked with 0.0.0.0/0 -> IGW at bootstrap and reconciled by the vpcd drift loop, so a subnet created in it is reachable regardless of the create-time gap — the test passed and never flagged the bug. Replace it with runNewVPCEgressBaseline: a brand-new VPC where the IGW route is added to the MAIN route table BEFORE the public subnet is created. The subnet then implicitly joins a main RT that already carries the route, but CreateSubnet does not recompute per-subnet egress, so the subnet's OVN egress LRP is never installed and the instance has no datapath to the IGW. With an open SG, routing is the only barrier, so the instance is unreachable — red until CreateSubnet triggers a per-subnet egress recompute. PoolMode-gated like runVPCSubnetE2E (fresh VPC needs OVN IPAM for a public IP). runVPCSubnetE2E doesn't catch this: it associates a custom RT then adds the route, so the route mutation enumerates the subnet and installs the LRP. |
||
|
joshsiv-mulga
|
c9a4e904a8 |
test(e2e): fresh-install reachability baselines + drop qemu hostfwd fallback
Add baseline tests that confirm two fresh-install regressions the existing
suite masks (it authorizes tcp/22 on the default SG before every SSH probe):
- default SG default-deny gates external reach of instances in the
default subnet (SG, not routing)
- a newly-created public subnet never gets its OVN IGW egress LRP, so it
has no route to the VPC IGW despite MapPublicIpOnLaunch
Plus a positive test: instances sharing the default SG can communicate via
ICMP (same-host single-node, cross-host multinode).
Single-node baselines run after TestClusterStatsCLI (0-VM gate); multinode
after TestMultinodePreflight (fixture init). Instances are self-cleaning so
they don't perturb the singleton-VM ordering.
Remove the qemu hostfwd SSH fallback from the harness (InstancePublicSSHHost,
GuestSSHEndpoint) - it bypasses the OVN datapath and masks these regressions.
Endpoints now t.Fatal when no public IP is assigned.
|
||
|
Julian Sommer
|
ff0dc42562 | feat: IMDS subnet level (#292) | ||
|
Julian Sommer
|
bcb49cd771 | feat: IMDS v1 (#266) | ||
|
Julian Sommer
|
94aa482931 | feat: sts v1 (#261) | ||
|
Josh Siviour
|
fd3dc6b680 |
feat(network): ADR-0006 layer redesign + IPsec/OVN hardening (#255)
* refactor(vpcd): remove macvlan bridge mode
Macvlan was a single-NIC SSH-safe hack that predated veth mode and was
never selected by any active config path. Deletion is Phase 0 item 1 of
the network layer redesign (mulga-siv-125.1).
Removed:
- BridgeModeMacvlan const + setMacvlanMAC + ifaceIsMacvlan probe
- Macvlan MAC alignment block in launchService (only fired on dead path)
- Macvlan branch in detectBridgeMode + matching tests
- Macvlan duplicates of IGW/AddNAT centralized-NAT tests (veth versions
already cover the same path)
- --macvlan flag + macvlan case branch in setup-ovn.sh
Refactored:
- useCentralizedNAT() now == BridgeModeVeth (was != BridgeModeDirect);
verifyBridgeMode already rejects empty/unknown at startup
- Three IGW-pool tests now explicitly WithBridgeMode(BridgeModeVeth)
(previously relied on empty-mode defaulting to centralized)
Kept intentionally:
- Deprecation notes in docs/compute/vpc-networking/README.md
- Legacy macvlan-interface cleanup in reset-dev-env.sh + ansible
teardown role (migration for operator systems with stale spx-ext-*
interfaces from pre-veth installs)
* feat(vpcd): resolve external CIDR at startup with bounded retry
Block service start until the WAN bridge has an IPv4 address, with a 30s
retry window. Phase 0 item 2 of the network layer redesign
(mulga-siv-125.1). Guards the boot race where vpcd starts before
systemd-networkd finishes DHCP on br-wan, or before netplan applies a
static config.
Skipped when ExternalMode is empty (overlay-only deployments).
Forward-compatible with ADR-0006: the new resolveExternalCIDR /
externalCIDRFromBridge helpers become the backing implementation of
HostWiring.ExternalCIDR() once the network/host package lands in Phase 1.
Until then the resolved value is only used to fail-start on missing
uplink.
* fix(vpcd): cover external CIDR startup paths
Extract launchService's CIDR-resolve block to ensureExternalCIDRReady
helper so it's unit-testable, and add direct tests for
externalCIDRFromBridge (loopback + unknown-interface) to exercise the
original literal body. Closes the diff-coverage gap on Phase 0 commit
15b564d which left these paths unreachable from unit tests.
* refactor(network/ovn): extract L1 OVN client to network/ovn package
Move OVNClient interface, LiveOVNClient, ACLSpec, ErrNATNotFound and
namedUUID helper from services/vpcd to a new spinifex/network/ovn
package. The mock relocates to spinifex/network/ovn/mock as a
sub-package. Storage fields on the mock become exported so vpcd tests
can continue to seed/inspect state across the new package boundary.
services/vpcd retains thin type-alias and var-alias shims so existing
callers compile unchanged; Phase 2 removes those shims as vpcd is torn
down.
Adds -coverpkg=./spinifex/... to test-cover so cross-package test
exercise of mock methods continues to count toward coverage now that
the mock lives outside vpcd.
First sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md
(bead mulga-siv-125.2). Pure refactor; no functional change. All
existing tests pass; preflight green.
* feat(network/topology): TopologyManager interface + vpcd adapter
Add spinifex/network/topology with Manager interface, VPC/Subnet/Port
specs, and OVN object naming helpers covering the L2 stable contract
(§7 of the redesign plan). Phase 1 keeps existing in-tree OVN names
(rtr-, ext-port-, sg_) and notes the §7.3 targets for a later
reconciliation migration.
TopologyHandler in services/vpcd grows EnsureVPC / DeleteVPC /
EnsureSubnet / DeleteSubnet / EnsurePort / DeletePort /
SetPortSecurityGroups methods that mirror the corresponding NATS
handlers' OVN ops with Spec inputs instead of nats.Msg envelopes,
satisfying topology.Manager via a compile-time check. Existing handlers
are unchanged; the duplication is short-lived (Phase 2 collapses them).
Smoke tests in topology_manager_test.go cover the idempotent create
path, delete cascade, and declarative SG membership move. Deliverable:
TopologyManager.EnsureVPC/Subnet/Port callable; all existing tests
green.
Second sub-step of Phase 1 of docs/development/feature/spinifex-network-redesign.md
(bead mulga-siv-125.2).
* feat(network/host): L0 HostWiring interface + physical/veth impls
Phase 1 deliverable for mulga-siv-125.2. Introduces network/host (L0 of
ADR-0006): UplinkMode enum, Wiring interface, and Physical/Veth
implementations with injectable Runner + InterfaceReader for testability.
Not yet wired into daemon/vpcd — Phase 2 replaces daemon.SetupComputeNode
and vpcd's resolveExternalCIDR with calls through this package.
* feat(network/policy): L3 SecurityGroup/NAT/Route managers
Introduce network/policy as the data-plane policy layer (ADR-0006 L3).
Three managers, all backed by network/ovn:
- SecurityGroupManager: clear+add ACL replacement on an existing port
group; port group lifecycle stays with L2 (topology.Manager).
- NATManager: bound to one NATMode at construction (derived from
host.UplinkMode). Distributed sets logical_port/external_mac on EIP
dnat_and_snat rules; centralised leaves them unset. Per-EIP, NAT GW,
and IGW-default SNAT entrypoints; all deletes idempotent.
- RouteManager: AddStaticRoute is read-then-decide so retries no longer
duplicate rows (closing the long-standing non-idempotency bug in
vpcd/topology.go's IGW default-route path).
ACL priority constants and infrastructure ACL builders move from
vpcd/{security,acl}.go into policy/acl.go with their semantics preserved
(default-deny 900/800 logged, DHCP-allow 1050, tenant 1000). No vpcd
wiring changes here — additive only; the reconciler (bead .3.2) and
network/external (bead .3.4) will consume these managers.
bead: mulga-siv-125.3.1
* feat(network/external): L5 IGW/EIP/NATGW managers
Pure additive — new package, no topology.Manager changes, no rewiring of
existing vpcd handlers. IGWManager builds the external switch + localnet +
gateway LRP + default route + chassis HA, with a flows barrier callers can
inject (production: ovn-nbctl --wait=hv sync). EIP/NATGW are thin L5
wrappers over policy.NATManager. GatewayIPAllocator interface lets the
DHCP-backed allocator stay in vpcd until the DHCP-mgr removal bead lands;
LinkLocalAllocator + StaticRangeAllocator ship here.
* feat(network/reconcile): L6 intent-actual reconciler replaces 5-pass startup
Build spinifex/spinifex/network/reconcile/ package per Phase 2.2 of the
network layer redesign. Reads desired state from NATS KV filtered by
vpc.AZ (with legacy empty-AZ matches-all rule), scans actual state from
OVN NB DB, diffs and applies with VPC → Subnet → Port → SG → IGW →
EIP/NATGW topological ordering.
Atomic port-group join via CreateLogicalSwitchPortInGroups closes the
racy gap the legacy scanENIPortMembership pass used to revive 30s later.
A 5-minute drift goroutine (leader-gated via JetStream KV CAS) replaces
the old 30s ReconcileSGsLoop.
Adds AZ field to VPCRecord stamped at create time. Empty AZ on pre-Phase
2.2 records matches every local AZ, so no migration pass is required.
Replaces and removes the 5-pass startup sequence in services/vpcd:
- Reconcile + ReconcileFromKV
- ReconcileSGsOnce + ReconcileSGsLoop
- RetrofitAllExternalLocalnetOptions
- RetrofitAllGatewayPortNetworks
IGW fresh-create continues to flow through the legacy NATS handler path
(handleIGWAttach) so the dhcpManager wiring stays intact until Phase
2.3; the reconciler only rebinds gateway-chassis on existing IGW
topology.
Closes mulga-siv-125.3.2.
* refactor(network/topology): promote live Manager impl to network/topology
Move the topology.Manager implementation out of services/vpcd/topology_manager.go
into network/topology/manager_live.go as a standalone liveManager constructed
via NewLiveManager(ovn.Client, opts...). Inputs that varied per-handler
(DNS server resolution from the external pool) are now passed as functional
options (WithDNSServer).
TopologyHandler still satisfies topology.Manager but its methods delegate
to a cached liveManager. The shim stays until the rest of mulga-siv-129
routes NATS subscribers through topology.Manager directly.
Also exports SubnetGatewayCIDR + FormatDNSServerList from network/topology
so the reconciler in network/reconcile can share helpers with the live
manager (audit pass to follow per plan §6).
Test cases ported one-for-one from services/vpcd/topology_manager_test.go
to network/topology/manager_live_test.go; the vpcd-side test file is
deleted. New WithDNSServer test asserts the option propagates into
DHCPOptions rows.
Refs mulga-siv-129 (Phase 2.6 vpcd subscriber migration, Slice 1 of N).
* refactor(vpcd): collapse SG subscribers onto network/{topology,policy} managers
Phase 2.6 / mulga-siv-129 slice 2. Rewrites the vpc.create-sg /
vpc.update-sg / vpc.delete-sg NATS subscribers in services/vpcd as
thin wrappers over topology.Manager (port group lifecycle) and
policy.SecurityGroupManager (ACL programming). Deletes the ~1000
lines of duplicated ACL builder and provisioning logic — its
authoritative copy lives in network/policy.
Surface additions:
- ovn.ErrPortGroupNotFound sentinel; mock + live GetPortGroup wrap
not-found via fmt.Errorf("%w: ...") so callers can use errors.Is.
- topology.Manager grows EnsureSGPortGroup / DeleteSGPortGroup
(resolves the apply.go TODO from §2.1 of the redesign plan).
liveManager implements them with GetPortGroup-then-Create /
GetPortGroup-then-Clear+Delete idempotency.
Behaviour delta: the old provisionSG rolled back the OVN port group
on AddACLs failure ("all-or-nothing"). The new path lets the port
group survive and relies on network/reconcile to retry EnsureSG on
its next sweep. The all-or-nothing fail-fast test cases are dropped;
ACL emission semantics are covered by network/policy/sg_test.go.
* refactor(vpcd): migrate VPC/Subnet/ENI subscribers to topology.Manager
Phase 2.6 / mulga-siv-129 slice 3. Rewrites vpc.create / vpc.delete /
vpc.create-subnet / vpc.delete-subnet / vpc.create-port /
vpc.delete-port / vpc.update-port-sgs NATS subscribers as thin
wrappers (≤30 lines each) over topology.Manager. Subscriber bodies
now: parse event → translate to spec → delegate to manager.
Deletes:
- vpcd.subnetGateway (and helpers_test.go + topology_test.go's
TestSubnetGateway block); parity coverage lives in
network/topology.TestSubnetGatewayCIDR.
- vpcd.reconcilePortSGs (the live manager already exposes the same
diff-then-transact via reconcilePortSGs on the L2 side).
- vpcd.TopologyHandler.dnsServer; lmDNSServer (passed via
topology.WithDNSServer to the live manager) is the live path.
Behaviour delta: handlers now strict-parse PrivateIpAddress /
MacAddress / CidrBlock at the NATS boundary and reject malformed
payloads instead of degrading to "no DHCP options". The reconciler
guarantees that field validity is upheld upstream of the publish.
* refactor(network/reconcile): drive SG port groups via topology.Manager
applySGs now calls topology.Manager.EnsureSGPortGroup /
DeleteSGPortGroupByName instead of reaching directly into the OVN client.
Reconcile.Config gains a Topology field (required); the production caller
in services/vpcd passes the TopologyHandler shim, the reconcile package
tests construct a fresh topology.NewLiveManager bound to the mock OVN
client.
DeleteSGPortGroupByName is added to the topology.Manager interface for the
reconciler's orphan-removal path, which iterates raw OVN port-group names
without a groupID round-trip. The existing DeleteSGPortGroup(groupID)
delegates to it.
Resolves the TODO at apply.go:137 carried over from slice 2 of
mulga-siv-129.
* feat(network/policy): add FlowsBarrier option to NATManager
NATManager.AddEIP needs to block on `ovn-nbctl --wait=hv sync` after
writing the dnat_and_snat row so the NATS reply only returns once every
chassis has the rewrite flow. The legacy vpc.add-nat handler does this at
the tail of handleAddNAT; the manager has lacked it since slice 2.1.
NewNATManager grows a functional-options tail (Option / WithFlowsBarrier).
Defaults to a no-op for tests; production wiring in services/vpcd passes
waitForFlowsHV. AddEIP now invokes the barrier after AddNAT, logging at
warn level on failure to match the legacy handler's _ = waitForFlowsHV()
swallowing semantics.
Pre-work for slice 4 of mulga-siv-129 — without the barrier the NAT EIP
subscriber migration would lose the post-write flow-install gate.
* refactor(vpcd): migrate NAT GW subscribers to policy.NATManager
handleAddNATGateway / handleDeleteNATGateway now decode the NATS event
and delegate to policy.NATManager.{AddNATGateway,DeleteNATGateway}. The
manager carries identical OVN logic — same snat type, same ExternalIDs
(spinifex:vpc_id + spinifex:nat_gateway_id) — so existing
TestTopologyHandler_AddNATGateway / DeleteNATGateway suites stay green.
Lazy-constructed NATManager added on TopologyHandler (natmOnce). Mode is
derived from the bridge mode (veth → centralized, direct → distributed)
and the FlowsBarrier is bound to waitForFlowsHV for parity with the
legacy EIP path's tail barrier (used by the next slice, vpc.add-nat).
Slice 4 of mulga-siv-129, sub-step b.
* refactor(vpcd): migrate EIP NAT subscribers to policy.NATManager
handleAddNAT / handleDeleteNAT now decode the NATS event and delegate to
policy.NATManager.{AddEIP,DeleteEIP}. The behavioural contract is
preserved end-to-end:
- Idempotency-skip on a matching dnat_and_snat row (same VPC + LogicalIP,
plus MAC + LogicalPort under distributed NAT) moves into AddEIP so
duplicate vpc.add-nat publishes never tear down the rule the previous
invocation just installed (mulga-siv-124).
- Stale-rule cleanup across every router stays in AddEIP.
- Post-write waitForFlowsHV barrier is now driven by the manager via the
FlowsBarrier option wired in services/vpcd (slice 4a).
- DeleteEIP returns nil on ErrNATNotFound, matching the handler's
fire-and-forget "already absent = success" semantic.
The vpcd "errors" import drops out with the inline error-comparison.
Slice 4 of mulga-siv-129, sub-step c.
* refactor(vpcd): migrate IGW subscribers to external.IGWManager (partial)
handleIGWAttach now branches at decode time:
- Centralised NAT + DHCP-sourced pool keeps using attachIGWLegacy because
the gateway-LRP IP comes from vpc.dhcp.acquire, and that DHCP client is
the vpcd-local manager that bead mulga-siv-125.3.3 will retire.
- All other shapes (distributed NAT, static-range pool, no pool)
delegate to external.IGWManager.AttachIGW. The manager mirrors every
step the legacy handler ran — external switch + localnet port +
gateway LRP + switch-side router peer + default route + chassis HA +
waitForFlowsHV tail barrier — so observable behaviour is unchanged.
handleIGWDetach drops its inline OVN walk entirely; both paths exit
through external.IGWManager.DetachIGW followed by releaseGatewayLRPLease.
The lease release stays in vpcd because the upstream client lives in
services/vpcd/dhcp; calling it after every detach is safe because the
upstream server expires unknown leases on its own.
Lazy-constructed IGWManager added on TopologyHandler (igwmOnce). Pool,
allocator (StaticRangeAllocator), chassis, NAT mode, NAT manager, route
manager, and FlowsBarrier are all derived from the existing
TopologyHandler state.
The DHCP-coupled legacy branch retains its full inline OVN logic; it
will disappear in slice-after-125.3.3 follow-on. Slice 4 of
mulga-siv-129, sub-step d.
* feat(ec2/instance): honor AssociatePublicIpAddress per-NIC override
RunInstances --network-interfaces "AssociatePublicIpAddress=true|false"
was silently ignored, only subnet.MapPublicIpOnLaunch gated public IP
assignment. AWS contract is per-NIC override wins.
Extend TestPrepareRunInstances_PublicIPAutoAssigned to table-driven
3 cases: subnet true no-override, subnet false override true, subnet
true override false.
* refactor(config): drop upstream-DHCP config surface (static WAN pool only)
Hard-error in LoadConfig on legacy keys: [network] external_dhcp,
[nodes.<n>.vpcd] dhcp_bind_bridge, and [[network.external_pools]]
source = "dhcp". Add pool range validation (must fit inside the
gateway/prefix_len CIDR; per-cluster non-overlap).
Strip ExternalDHCP from config.NetworkConfig + formation.NetworkConfig
+ admin.ConfigSettings, and DhcpBindBridge from config.VPCDConfig +
admin.ConfigSettings + vpcd.Config. vpcd.ExternalPoolConfig.IsDHCP()
short-circuits to false; the field and downstream DHCP branches in
external_ipam / topology.go remain for Slice C to delete together with
dhcp_manager.
cmd/spinifex/cmd/admin.go: drop useExternalDHCP path and the
detectedDhcpBindBridge helper; require --external-pool when
--external-mode=pool; auto-detect default no longer infers source=dhcp.
service.go's wan_bridge legacy-key rejection now advises removing the
key entirely (vpcd auto-detects br-wan), not renaming to
dhcp_bind_bridge.
Two topology_test cases that exercise the DHCP gateway-LRP path are
skipped — Slice C deletes attachIGWLegacy and these tests together.
* refactor(vpcd): delete dhcp_manager + DHCP gateway-LRP path (Slice C)
Removes upstream-DHCP-client gateway-LRP allocation in favor of static
WAN-pool allocation (StaticRangeAllocator). Part of bead mulga-siv-125.3.3.
- delete services/vpcd/dhcp_manager.go + dhcp/ subpackage (~865 lines)
- delete allocateGatewayLRPIPViaDHCP + releaseGatewayLRPLease paths
- drop ExternalPoolConfig.Source + DhcpBindBridge + IsDHCP() fields
- drop ExternalIPAllocation lease fields (Expiry/ServerID/HWAddr) and
ExternalIPAMRecord.Source
- drop NewExternalIPAM nc *nats.Conn argument; update all 4 call sites
- rename verifyBridgeMode local dhcpBindBridge -> wanBridge; update error
strings and tests
Slice D (delete attachIGWLegacy + dispatcher) intentionally deferred.
* refactor(vpcd): delete attachIGWLegacy + IsDHCP stub (Slice D)
Closes mulga-siv-129 slice 4d. Part of bead mulga-siv-125.3.3.
handleIGWAttach now unconditionally delegates to external.IGWManager.
attachIGWLegacy (~207 lines, ~5 OVN write steps + chassis scheduling)
deleted along with its dispatcher branch in handleIGWAttach. The
ExternalPoolConfig.IsDHCP() dead stub is gone.
No code paths or NATS subjects regress: the static-pool / distributed
branches were already going through external.IGWManager since Slice 4d
(6719dc3).
* feat(ipam): Purpose field on every IPAM record (Slice B)
Part of bead mulga-siv-125.3.3.
Adds a Purpose enum (handlers/ec2/vpc/purpose.go) tagging every IP
allocation with its semantic role (eni-primary, eni-public, eip,
igw-lrp, natgw-external, …) so multi-VPC clusters can reclaim/audit
by (owner, purpose).
External IPAM (spinifex-external-ipam, v1→v2):
- ExternalIPAllocation.Type → .Purpose; legacy values mapped via
LegacyExternalTypeToPurpose (gateway→igw-lrp, auto_assign→eni-public,
elastic_ip→eip). Unknown legacy values warn and become "unknown".
- AllocateIP / AllocateFromPool / allocateFromPool param rename
allocType→purpose. All call sites pass Purpose* constants.
- migrate/002_external_ipam_purpose.go: idempotent CAS-retry migration
with v1/v2-union record shape so partial migrations resume cleanly.
Internal IPAM (spinifex-vpc-ipam, v1→v2):
- IPAMRecord.Allocated []string → []IPEntry{IP, Purpose, OwnerID}.
- AllocateIP gains purpose+ownerID args; ReleaseIP unchanged (still
keyed on IP).
- AllocatedIPs returns []IPEntry.
- eni.go reorders eniId generation before AllocateIP so the owning
resource ID lands on the new entry.
- migrate/003_ipam_purpose.go: cross-bucket backfill that reads the
ENI bucket via the new RunKVWithJetStream path. Orphaned IPs
(no owning ENI) warn and become Purpose=unknown.
Migrate framework:
- KVContext gains an optional JetStream field; new
Registry.RunKVWithJetStream method threads it through. RunKV
signature unchanged — all other callers are unaffected.
Tests:
- ipam_test.go: AllocateIP signature updates; AllocatedIPs now asserts
IP / Purpose / OwnerID on the returned entries.
- external_ipam_test.go: replaces "auto_assign"/"elastic_ip" literals
with PurposeENIPublic/PurposeEIP and asserts alloc.Purpose.
- migrate/002 + 003 tests: schema-detect, idempotent rerun,
no-JetStream fallback, empty bucket.
* refactor(daemon): tombstone obsolete spinifex-dhcp-leases KV bucket
The dhcp_manager removed in Phase 2.3 (bead mulga-siv-125.3.3) owned the
spinifex-dhcp-leases KV bucket. The migrate framework only handles
per-bucket schema version bumps, not bucket deletion, so the obsolete
bucket would linger on upgraded clusters.
Add a one-shot tombstone after initJetStream that deletes the bucket if
present. DeleteKVBucketIfExists treats ErrBucketNotFound and
ErrStreamNotFound as success, so the call is idempotent across restarts.
* feat(config): add network.ipsec_enabled cluster flag (default true)
Add IPSecEnabled to NetworkConfig in both the cluster TOML schema and
the formation handshake payload. Default true so AWS-parity intra-AZ
encryption is on by default. Operators can flip false in
[network] for trusted single-rack lab topologies.
Phase 2.5 (bead mulga-siv-125.3.5) consumes the flag from admin init
(strongSwan provisioning), formation server (propagates to joiners),
and daemon startup (ovs-vsctl ipsec_encapsulation toggle).
* feat(admin): IPsec peer cert generator signed by cluster CA
Add GenerateIPSecPeerCert + IPSecCertPaths in spinifex/admin so admin
init (and the future admin add-node path) can issue per-node IKEv2 peer
credentials signed by the existing cluster CA. The cert carries
id-kp-ipsecIKE (RFC 4945) so strongSwan accepts it as an IKEv2 peer
credential when ovs-monitor-ipsec materialises tunnel configs.
Cluster CA is reused directly — no intermediate strongSwan PKI today;
when intermediate CAs land that becomes a separate workstream.
The cert/key pair lives under {configDir}/ipsec/peer.{pem,key}; the
daemon points OVS Open_vSwitch.other_config (certificate, private_key,
ca_cert) at these paths when network.ipsec_enabled is true.
* feat(admin): wire IPsec peer cert into admin init + join paths
Add --ipsec bool flag (default true) on adminInitCmd. Single-node init
and the multi-node leader path both emit an IPsec peer cert after the
existing CA+server-cert generation. The multi-node formation server
propagates the IPSecEnabled flag to joining nodes via NetworkConfig;
admin join honours it and emits its own peer cert during the same
cert-issuance step that already produces the server cert.
spinifex.toml template gains an unconditional [network] block carrying
ipsec_enabled so the value is visible to operators on every install
regardless of whether external networking is configured. The daemon
reads it via cluster config (NetworkConfig.IPSecEnabled).
Bead mulga-siv-125.3.5.
* build: install openvswitch-ipsec for OVN native IPsec
openvswitch-ipsec provides ovs-monitor-ipsec, which reads the
Open_vSwitch.other_config cert/key/CA pointers and materialises
strongSwan configs for every Geneve tunnel managed by ovn-controller.
Pulls in strongswan + strongswan-pki as transitive deps.
Required by bead mulga-siv-125.3.5.
* feat(daemon): enable OVN native IPsec on intra-AZ Geneve at startup
When network.ipsec_enabled is true (default), the daemon points the
local OVS DB other_config at the per-node IPsec peer cert that admin
init / admin join provisioned (configDir/ipsec/peer.{pem,key} + the
cluster CA at configDir/ca.pem), then sets
other_config:ipsec_encapsulation=true.
ovs-monitor-ipsec (shipped by the openvswitch-ipsec package) consumes
those pointers and materialises strongSwan configs for every Geneve
tunnel ovn-controller programs — including OVN-IC inter-AZ tunnels
when federation lands in Phase 3a.
Missing cert files log a warning and leave Geneve plaintext; operators
upgrading an older cluster reissue certs via spx admin to close the
gap.
Bead mulga-siv-125.3.5.
* test(e2e): multinode IPsec wiring + xfrm SAs + ESP capture
TestMultinodeIPSec asserts:
- OVS DB on every node carries certificate/private_key/ca_cert pointers
and other_config:ipsec_encapsulation=true (what daemon.enableOVNIPSec
writes).
- ip xfrm state on every node has at least one AEAD SA in AES-GCM mode
(rfc4106(gcm(aes)) or gcm(aes)) within 90s.
- tcpdump catches ESP (ip proto 50) on the underlay (best-effort —
logged only; geneve may be idle during the 5s capture window).
Skips cleanly when the cluster was bootstrapped with --ipsec=false
(no ipsec_encapsulation key on first node).
Read-only over SSH; runs in the parallel bucket alongside the other
non-mutating multinode tests.
* fix(igw): write attachment State="attached" for AWS parity
Reconciler intent loader (network/reconcile/intent.go::loadIGWs) filters
IGW records with `State == "attached"`, but AttachInternetGateway was
writing `State = "available"`. The reconciler therefore never picked up
any attached IGW — `intent_igws=0` on every pass — so the OVN external
switch + gateway LRP + WAN bridge wiring built by IGWManager.AttachIGW
never ran for default-VPC IGWs (whose live `vpc.igw-attach` publish at
daemon bootstrap is lost — vpcd subscriber not yet wired).
Net effect on single-node e2e: every SSH probe to 192.168.x public IPs
timed out (TestSSHProbe, TestModifyInstanceAttribute, TestRebootInstance,
TestNATGateway, TestSGToSGDatapath) because SNAT/DNAT rules on the VPC
router had no path to the external network.
Match AWS InternetGatewayAttachment.State semantics:
- Attach: State="attached"
- Detach: State="detached"
Detach value is internal — recordToEC2 gates Attachments on VpcId != "",
so the detached State is never surfaced to API consumers.
Bead: mulga-siv-131
* fix(reconcile): call AttachIGW when IGW topology missing
applyIGWs previously deferred to the NATS handler when an intent IGW
had no external switch in OVN. When the vpc.igw-attach event is lost
(bootstrap-time default-VPC IGW published before vpcd subscriber is
ready), deferral never resolves and VM public-IP traffic dies.
Call IGWManager.AttachIGW directly — it is idempotent (short-circuits
when the external switch already exists), so the reconciler converges
default-VPC IGWs on first pass without duplicating work on subsequent
passes.
Closes mulga-siv-132.
* fix(test): align NewExternalIPAM call with 2-arg signature
Rebase onto dev pulled in daemon_system_instance_test.go which used
the legacy 3-arg NewExternalIPAM(jsNC, js, pools). The feature branch
already narrowed the signature to (js, pools) via the upstream-DHCP
config drop; align the test with the current signature.
* fix(vpcd): EnsureSubnet must EnsureVPC first to avoid NATS race
vpc.create-vpc and vpc.create-subnet are independent NATS events with
no inter-event ordering guarantee. Tenant bootstrap on iam.account.created
publishes both within <1ms, so the subnet handler's
CreateLogicalRouterPort can race the parent VPC's CreateLogicalRouter and
fail with "logical router not found". The cascade fails CreateNetworkInterface
which fails RunInstances which surfaces as InsufficientInstanceCapacity to
the SDK — masking the underlying topology race.
handleSubnetCreate now calls EnsureVPC(VPCID-only) before EnsureSubnet.
EnsureVPC short-circuits on existing LR (idempotent), so the race-winning
vpc.create-vpc handler's full CIDR/VNI ExternalIDs are preserved; if subnet
wins the race the LR is created with bare ExternalIDs (cosmetic) and the
follow-on vpc.create-vpc short-circuits.
Adds TestIntegration_SubnetBeforeVPC regression covering both orderings.
Closes mulga-siv-133.
* feat(ansible): require --external-pool, drop NAT, forward EXTRA_VARS
spx admin init now rejects --external-mode=nat and demands --external-pool
when mode=pool. The ansible init role silently skipped pool flags when the
range was unset, so installs fell over inside the spx validator. Make
targets also swallowed `-e` (make's env-override flag), so operators could
not override pool start/end without editing defaults.yml.
- vars/defaults.yml: rename nat_gateway_ip → external_gateway_ip; add
spinifex_external_no_external opt-out; document the spx flag contract.
- roles/init/tasks/main.yml: drop NAT branch; assert pool range upfront
with a fail_msg pointing at EXTRA_VARS / defaults.yml; emit gateway,
prefix-len, gateway-ip only when explicitly set (let spx auto-detect);
honour --no-external.
- playbooks/dev-reset.yml: rename captured var; coerce legacy
external_mode=nat to pool with a warn; tighten gateway regex with \b.
- Makefile: ansible-dev-* targets now forward EXTRA_VARS="key=val ..."
through --extra-vars so `make` callers can override without editing
defaults.yml.
- scripts/ansible/README.md: document pool requirement, both invocation
styles (direct -e vs make EXTRA_VARS), NAT removal.
* fix(ipsec): guard ipsec_encapsulation on ovs-monitor-ipsec liveness
enableOVNIPSec set OVS cert pointers + ipsec_encapsulation=true without
ensuring ovs-monitor-ipsec.service was running. Without that daemon, OVS
holds the cert pointers unused, strongSwan never gets IKE config, and
tunnel traffic ovn-controller marks for encryption gets silently
dropped. TestMultinodeIPSec fails on `node1 xfrm has no AEAD SAs`.
- systemd: spinifex-daemon Wants=/After= ovs-monitor-ipsec.service
- daemon: ensureOVSMonitorIPSecActive — systemctl enable --now +
is-active poll; refuses to flip ipsec_encapsulation if daemon is dead
- daemon: single-node short-circuit — no peers means no tunnels means
the flag is at best inert and at worst load-bearing-silent-drop
- ci: capture-net-diag.sh + workflow steps to snapshot OVS/OVN/xfrm
state on failure, fan out across all 3 multinode nodes
Refs: mulga-siv-136
* test(e2e): sticky-skip downstream SSH tests when probe fails
When TestSSHProbe times out (3 min, typically because of mulga-siv-134
datapath bug), four downstream tests each re-probe SSH on the same
broken datapath and time out identically — single-node job runs ~24 min
instead of ~12 min, with the same error logged five times.
First waitForSSHReady timeout now sets a package-level atomic flag;
subsequent waitForSSHReady / trySSHReady / requireSSHHealthy calls skip
immediately. Preserves the original red signal on TestSSHProbe and on
any test that genuinely needed SSH (Skip ≠ Pass). Bool variant
(trySSHReady) consults but does not set the flag — its caller already
treats False as advisory warning, not fatal.
TestSGToSGDatapath has its own EventuallyErr SSH wait (not via
waitForSSHReady) and needs an explicit requireSSHHealthy(t) gate at
the top of runSGToSGDatapath. TestNATGateway already calls
waitForSSHReady first so the gate fires naturally.
Refs: mulga-siv-137
* fix(ipsec): use Debian unit name openvswitch-ipsec.service
CI artifact shows `Unit ovs-monitor-ipsec.service could not be found`
on every node. Debian's openvswitch-ipsec package ships the unit as
openvswitch-ipsec.service (the binary inside is ovs-monitor-ipsec, but
systemd's name follows the package). The previous commit guarded on
the wrong unit name, so:
- spinifex-daemon.service Wants=/After= silently no-op'd (missing unit)
- enableOVNIPSec's `systemctl enable --now ovs-monitor-ipsec.service`
failed, daemon logged a warning and continued without flipping
ipsec_encapsulation=true
- TestMultinodeIPSec skipped with `node1 reports ""` because OVS DB
never received the cert pointers
Rename across all 5 spots (daemon, systemd unit, test, CI diag script,
workflow journal capture) to match the Debian package layout.
Note: the package itself still must be present on the CI VM image —
that's tracked in the mulga repo (image-builder provision.sh +
bootstrap-install.sh shim).
Refs: mulga-siv-136
* fix(e2e): authorize default SG tcp/22 + icmp in single-node fixture
admin init leaves the default SG closed (AWS parity); the Go single suite
never ported the Phase-5 SG-ingress block that run-e2e.sh has had since day
one, so every test runner SSH/ping probe hits the OVN port-group ACL drop.
- ec2helpers: add AuthorizeICMPIngress + EnsureDefaultSGOpen helpers.
- single/main_test: open default SG once per process in pkgFixOnce.Do.
- witness: extend ensureDefaultSGSSHIngress to authorize ICMP too.
- run-e2e.sh: append matching ICMP authorize next to tcp/22 block.
mulga-siv-134
* fix(network/reconcile): key DHCP_Options lookup by subnet_id not CIDR
Multi-tenant default VPCs share the same subnet CIDR (172.31.0.0/20) per
account. The existing FindDHCPOptionsByCIDR gate at apply.go:107 found
the first account's DHCP_Options row and skipped creation for every
subsequent subnet with the same CIDR. Those subnets' ports then attached
nothing (FindDHCPOptionsByExternalID misses on subnet_id), VMs got no
DHCPv4 lease, and SSH/ICMP into them silently failed even though SG
ACLs and NAT were correct.
Switch the gate to FindDHCPOptionsByExternalID keyed on spinifex:subnet_id
so each subnet owns its own row, matching the attach lookup at line 203.
Verified: account 000000000001 default VPC RunInstances now produces an
LSP with dhcpv4_options populated, VM cloud-init reports a 172.31.0.5/20
lease + default route via 172.31.0.1, and ping + ssh TCP handshake to
the EIP succeed end-to-end. Stale DHCP_Options row for the prior dead
subnet was wiped manually; reconciler recreated the correct row for the
live subnet on its next pass.
* fix(ipsec): enable openvswitch-ipsec at provision time, drop daemon write
Provisioning now runs `systemctl enable --now openvswitch-ipsec.service`
alongside openvswitch-switch and ovn-controller. The spinifex-daemon
process drops its `enable --now` shell-out and only polls is-active.
Sudoers gains one read-only line:
spinifex-daemon ALL=(root) NOPASSWD: /usr/bin/systemctl is-active openvswitch-ipsec.service
This mirrors the existing vpcd ovn-controller is-active precedent and
removes the daemon's need for systemd-write capability. Aligns with
least-privilege service account pattern: provision enables services,
daemon only reads status.
If the unit is inactive at runtime, the daemon refuses to flip
ipsec_encapsulation=true (silent-drop guard from mulga-siv-136) and
surfaces a clear error pointing at scripts/setup-ovn.sh.
* chore: unblock preflight on feat/network-layer-redesign
Two pre-existing failures blocked preflight on the branch:
- govulncheck flagged golang.org/x/net@v0.54.0 (GO-2026-5026, ASCII Punycode
rejection bug in idna.ToASCII); reached via utils.DownloadFileWithProgress.
Bump to v0.55.0; go mod tidy carries x/sys to v0.45.0.
- TestTopologyHandler_SubnetCreate_RouterPortFails asserted subnet create
fails when the parent VPC router is missing. That contract changed in
|
||
|
Julian Sommer
|
f525006d87 | feat: IAM Roles part 2 (#256) | ||
|
joshsiv-mulga
|
38202206bc |
fix(e2e/single): TestClusterStatsCLI back to sequential pre-launch
TestClusterStatsCLI asserts baseline cluster state ("0/" CPU used, "No VMs found") which doesn't hold once the singleton VM is up. Moving it into bucket #1 parallel (siv-128) ran it AFTER all sequential VM launches, so the spx top nodes output reported 4/12 CPU + 1 VM and the assertion failed.
Move it back into the sequential block, ahead of TestInstanceLaunch, so the baseline assertion sees the cluster pre-launch.
|
||
|
Josh Siviour
|
40f8c4da94 |
refactor(e2e/single): drop phase numbers + parallel bucket #1 (#247)
- Rename 36 phaseN_X / phaseIAM_N funcs to runX. - Drop numeric suffix from 7 TestIAMN_X wrappers (TestIAM1_UserCRUD -> TestIAMUserCRUD). - Rewrite harness.Phase log strings "Phase N - X" -> "Single - X" to mirror multinode convention. - Add t.Parallel() to bucket #1: 10 read-only / own-everything top-level Tests (Environment, ClusterStatsCLI, Discovery, SerialConsoleAccess, KeyPairs, Image, TagManagement, AccountScoping, VPCSubnetE2E, RouteTableValidation). Sequential bucket retains source-order singleton lifecycle (InstanceLaunch through FinalClusterStats). |
||
|
Josh Siviour
|
050ac8f861 |
feat(e2e): Go E2E harness + single-node migration to Ensure*/Discover* + reboot suite (#241)
* feat(e2e): add Go test harness + cert scenario, port run-cert-e2e.sh
Lands step 1 of docs/development/improvements/e2e-go-harness.md:
shared harness primitives (env, wait, artifacts, cert, tls) plus the
TestCertIssuance scenario that supersedes run-cert-e2e.sh. All files
gated by //go:build e2e so default builds and `make preflight` are
unaffected. Bash script stays in place for one CI cycle before deletion.
* feat(e2e): port run-lb-e2e.sh to TestLoadBalancer Go scenario
Step 3 of docs/development/improvements/e2e-go-harness.md. Replaces 1246
lines of bash with a typed Go scenario exercising all 4 LB variants
(ALB/NLB × internet-facing/internal) against a shared VPC + app instance
pool. Internet-facing phase auto-skips when no peer node is configured;
internal phase always runs.
Adds shared harness primitives:
- harness/aws.go: AWS SDK client factory wired to Spinifex AWS gateway
- harness/lb.go: WaitForLBActive, WaitForTargetsHealthy, WaitForENICleanup,
WaitForInstanceRunning/Terminated, InstanceENI
- harness/traffic.go: HTTPRoundRobin, TCPRoundRobin, distribution asserts
- harness/ssh.go: minimal peer SSH wrapper shelling out to /usr/bin/ssh
(replaced by typed x/crypto/ssh client hoisted from DDIL in mulga-siv-74)
User-data scripts embedded via //go:embed under lb/testdata/.
Bash script stays in place for one CI cycle before deletion.
* fix(e2e): log client launch params + dump daemon log on InvalidInstanceType
Adds diagnostics for the LB internal client launch path. Logs the exact
InstanceType, AMI, subnet, key name, and user-data size right before
RunInstances, and registers a daemon-log dump on failure so we can
correlate the 400 InvalidInstanceType with what spinifex saw.
* refactor(e2e): serialise LB suites + share probe client across internal tests
Splits TestLoadBalancer into 4 sequential subtests (ALB/NLB ×
internet-facing/internal). Each suite creates its own TG + LB + listener,
runs its traffic test, and tears down before the next starts. Replaces
the per-suite throwaway client VM with a single long-lived probe server
launched at fixture setup; per-suite internal traffic tests GET
/probe?ip=...&proto=... against it instead of cold-booting a fresh client.
Drops peak instance count from 5 to 4 (2 app + 1 LB + 1 client), so the
internal phase passes on capacity-constrained dev nodes that previously
hit InvalidInstanceType (see mulga-siv-76, mulga-siv-77 for the underlying
spinifex bugs).
* feat(daemon): make hostReserve env-configurable for dev capacity squeeze
Adds SPINIFEX_RESERVED_VCPU and SPINIFEX_RESERVED_MEM_GB env-var overrides
read at daemon startup. Invalid values are logged and ignored so a typo
doesn't silently widen the reserve. Unblocks the LB E2E on capacity-tight
dev hosts (8 vCPU box: default 2-vCPU reserve + 3x t3.nano test load
leaves 0 free for the sys.micro LB VM). Lays groundwork for the planned
operator-tunable [capacity] config.
bead: mulga-siv-78
* fix(e2e): revert LB to per-suite client + fail-fast on terminal LB state
Reverts the shared probe-server VM that wouldn't come up reliably during
the InternalFacing suites — back to the proven bash pattern of launching
a fresh client VM per suite (one of ALB_IP / NLB_IP populated based on
the suite kind). Adds renderClientUserData + a conditional template so
each client probes only its target LB.
WaitForLBActive now bails immediately on terminal LB states (failed,
provisioning_failed) and emits a "still waiting" log line every 30s so
hangs are visible instead of silently burning the full 5-minute timeout.
* ci(e2e): add e2e-go.yml workflow + GOTESTSUM Makefile opt-in
Standalone workflow runs the new Go-based E2E harness on the existing
ci-single self-hosted runner — provisions a tofu VM, bootstraps spinifex,
installs gotestsum, runs each requested suite via the e2e Makefile, and
uploads per-test artifacts + JUnit XML + JSON streams as a build artifact.
Test summary parses the JUnit and emits ::error file=...:: annotations so
failures surface in the PR diff.
Gated to the feat/e2e-go-harness branch + workflow_dispatch so it doesn't
interfere with the existing e2e.yml while the Go conversion lands.
tests/e2e/Makefile gains a GOTESTSUM=1 opt-in: when set, the e2e target
runs gotestsum with --junitfile and --jsonfile rooted in ARTIFACT_DIR
under a suite-tagged filename derived from the first PKG component.
* feat(e2e): share probe client across both internal LB suites + SG fixes
- Replaces per-suite probe client with a single shared VM launched in the
fixture. Saves ~60s by avoiding a cold boot between the ALB and NLB
internal suites.
- Client cloud-init exposes a JSON trigger endpoint on :9090 in addition
to the http.server on :80 (results dir). Each suite POSTs
{proto, ip, count, outfile, http_port, tcp_port} to run a probe loop
against the LB private IP and writes the result file under /tmp/httpd
for the driver to fetch back.
- configureDefaultSG now uses the structured IpPermissions form (top-level
shortcut is silently ignored by vpcd — tracked separately) and opens
port 9090 in addition to 80 + 9000.
- deleteLB captures the underlying sys.micro VM id via the LB ENI before
delete, then polls DescribeLoadBalancers until LoadBalancerNotFound so
the next suite doesn't race the capacity reclaim.
- Internal_NLB sleeps 15s up front to let the ALB sys.micro deallocate
fully before its own createLB fires. DescribeLoadBalancers can report
the ALB gone before the vCPU/memory allocation is actually released on
capacity-tight dev hosts; without this settle, NLB's createLB trips the
sys.micro reserve and lands in terminal state=failed. Still cheaper
than cold-booting a per-suite client.
* ci(e2e): install Go on tofu VM before running Go E2E suites
The tofu-provisioned VM ships with the spx binary and bash test scripts
but no go toolchain, so the prior workflow's "Install gotestsum on VM"
step failed with `bash: line 2: go: command not found`. Add a curl-based
Go 1.26.3 install (matching go.mod) ahead of the gotestsum install and
update the test step's PATH so make e2e-* runs the freshly-installed
go binary.
* ci(e2e): bump gotestsum to @latest for Go 1.26 compatibility
gotestsum v1.12.0 pins golang.org/x/tools v0.19.0 which fails to compile
under Go 1.26 with:
invalid array length -delta * delta (constant -256 of type int64)
Switching to @latest pulls a newer x/tools that builds clean.
* ci(e2e): ship spinifex source to VM so make -C tests/e2e can run there
Single-node tofu VM only has the spx binary + bash test bundle at
~/e2e-tests; the Go e2e workflow was running 'make -C tests/e2e e2e-cert'
from $HOME/Development/mulga/spinifex which doesn't exist, producing:
make: *** No rule to make target 'e2e-cert'. Stop.
Add a checkout of the spinifex repo on the runner and rsync it (minus
.git, build/, bin/, node_modules) to the VM as
~/Development/mulga/spinifex before installing Go + gotestsum.
Interim solution; mulga-siv-80 tracks replacing the rsync with shipping
pre-built test binaries from the runner.
* ci(e2e-go): replace rsync with tar-over-ssh; VM lacks rsync
bootstrap-install.sh ships a minimal binary VM with no rsync; the
ship-source and collect-artifacts steps failed with
'rsync: command not found'. Stream a tarball over ssh instead — same
exclude list, no extra dependency on the VM.
* fix(e2e): discover single-node IPs + awsgw bind in LoadEnv
run-cert-e2e.sh single mode probes awsgw.host from spinifex.toml and
adds every non-loopback global IPv4 to the node list. The Go harness
defaulted to 127.0.0.1 only, so CI on a tofu VM (awsgw bound to the
LAN IP, not loopback) saw 'connection refused' on every TLS check.
Port the same discovery: ModeSingle now walks net.InterfaceAddrs for
NodeIPs and grep-parses spinifex.toml for the awsgw bind. Both stay
overridable via SPINIFEX_NODE_IPS / SPINIFEX_SERVICE_IPS.
* ci(e2e-go): pre-build test binaries on runner; drop go toolchain from VM
Replaces the source-tree + go + gotestsum payload that the e2e-go.yml
workflow previously shipped to the tofu CI VM with self-contained
test binaries built on the runner via `go test -c`. testdata for the
LB suite is already //go:embed'd into the binaries, so the bundle on
the VM is just `<suite>.test` files (cert: 14M, lb: 17M uncompressed).
Workflow changes
- actions/setup-go on runner.
- New step builds binaries via `make e2e-build SUITES_BUILD=...`.
- Install go-junit-report on runner; convert each VM-side test log to
junit-<suite>.xml after artifact pull so "Publish test summary" still
parses JUnit XML.
- Drop "Install Go + gotestsum on VM" entirely (saves ~60s + network).
- Drop source-shipping rsync/tar (saves ~30s + a lot of bandwidth).
- VM step runs `./<suite>.test -test.v -test.timeout=30m` directly.
Makefile changes
- New `e2e-build` target compiles each `SUITES_BUILD` entry into
`BIN_DIR/<suite>.test` via `go test -c -tags=e2e`. `clean` also
removes BIN_DIR.
Closes mulga-siv-80.
* ci(e2e-go): disable setup-go cache; upload dominates self-hosted run time
* refactor(e2e): hoist generic DDIL harness primitives to shared tests/e2e/harness
Step 1.5 of docs/development/improvements/e2e-go-harness.md. Promotes
Cluster/Node/SSH/DaemonClient/LocalStateRevision/ResetAllNodes/Witness
into the shared package so the LB, multinode, and BM scenarios in
steps 4-5 can build on the same primitives instead of duplicating SSH
+ cluster plumbing.
Moves
- ddil/harness/{cluster,ssh,daemon_client,state,reset,witness}.go
-> harness/ (plus the witness testdata embed dir).
- ddil/harness/{fault,netem,profiles,snapshot}.go -> ddil/fault/ as a
new package; fault.PartitionNode/HealNode/KillNATS/StartNATS/
RestartDaemonOnly/ApplyNetem/ClearNetem/LinkProfile*/TakeSnapshot
now take harness.Cluster / harness.Node / harness.SSH explicitly.
- ddil/harness/retry.go::Run retry-with-reset extracted as
harness.RetryWithReset in harness/wait.go; ddil/harness/quarantine.go
keeps the DDIL-only quarantine wrapper that calls into it.
- WaitForMode moved from fault.go to harness/daemon_client.go since
it polls daemon state and has no fault-injection concern.
- The pre-hoist harness/ssh.go (PeerSSH, subprocess ssh) is preserved
as harness/peer_ssh.go so the lb_test suite keeps working unchanged;
the typed x/crypto/ssh client from DDIL now owns harness/ssh.go.
- New harness/util.go exports ShellQuote (was unexported shellQuote in
netem.go) so both ddil/fault and reset.go/witness.go can use it.
Env var rename
- SPINIFEX_NODES / SPINIFEX_SSH_USER / SPINIFEX_SSH_KEY replace
DDIL_NODES / DDIL_SSH_USER / DDIL_SSH_KEY. Old names still honoured
with a stderr deprecation note (envWithLegacy) so the DDIL CI keeps
passing until .github/workflows/e2e-ddil.yml drops the old names.
DDIL_QUARANTINED stays DDIL-specific.
DDIL parity preserved — scenarios still call ddilh.Run for quarantine
semantics; the underlying retry-with-reset is the same code path,
just shared.
Closes mulga-siv-74.
* fix(e2e): open default SG :22 ingress via structured form before witness launch
The daemon currently drops the top-level shortcut AuthorizeSecurityGroupIngress
form silently (mulga-siv-79), so the default SG starts with no inbound :22 and
the witness EIP awaitBaseline tunnel times out under DDIL CI. Mirror the
workaround already present in lb_test by issuing the structured IpPermissions
form before LaunchWitnessVM. Idempotent on InvalidPermission.Duplicate so reruns
in the same cluster are safe.
* ci: render JUnit into job summary + collapsible groups in noisy steps
Add a reusable composite action (.github/actions/junit-report) that parses
a glob of JUnit XML files, writes a totals table + collapsible per-failure
details into $GITHUB_STEP_SUMMARY, and emits one ::error annotation per
failed/errored test case so failures surface in the run header instead of
being buried in step logs.
Apply the action to both e2e-go.yml and e2e-ddil.yml (e2e-ddil gotestsum
now also emits --junitfile so the action has input).
Same workflows also wrap noisy phases (tofu apply/destroy, scrub-env,
tar-over-ssh, per-suite test runs, cluster-log collection) in
::group::/::endgroup:: markers so the Actions UI collapses the chatter
by default and the human can expand only the failing phase. Tofu output
still tees to $TOFU_LOG so the artifact bundle is unchanged.
Future workflows (multinode, baremetal) can adopt the same pattern via
the README in .github/actions/junit-report.
* test(ddil): quarantine Scenario C until predastore §2 lands
CI run 26016840819 surfaced that ScenarioC_CleanPartition silently
depends on predastore-ddil-hardening §2 (write-quorum with deferred
parity repair). Root cause from node3 daemon log:
predastore: write udp ...->192.168.0.22:9991: sendmsg: operation not permitted
predastore: putObjectViaQUIC: put failed
spx QMP: BLOCK_IO_ERROR instanceId=<witness>
spx: StopOrTerminateInstance: Terminating
spx: Instance state transition running → shutting-down
The 'sendmsg: operation not permitted' is the EPERM Linux returns when
iptables drops an outbound packet — exactly the OUTPUT DROP rules
PartitionNode installs. The witness VM's block I/O is backed by
distributed predastore, so the same partition that severs node3↔peer
NATS traffic also severs the witness disk's cross-shard QUIC writes.
QEMU power-downs the guest after BLOCK_IO_ERROR, the daemon's
reconciler terminates the instance, and the next AssertProgressed dials
a sshd that's already gone (hence the 'handshake failed: EOF' the test
reports).
Quarantine by collapsing C to the same scenarioSkip stub D/E/F already
use, citing predastore-ddil-hardening §2 as the dependency. Flip C
back to ENABLED in the same PR that lands §2 — that's when the witness
disk can survive a peer partition without QEMU halting.
TEST_COVERAGE.md row C flipped ENABLED → SKIPPED; added a quarantine
note explaining the predastore dependency.
* ci: colored phase banners + quiet-on-success tofu output
Pretty up the live Actions log without flooding it. Three changes:
1. .github/scripts/ci-fmt.sh — sourceable bash helpers (banner, ok, bad,
warn, note, quiet_run) using ANSI colors that the Actions UI renders.
2. e2e-go.yml + e2e-ddil.yml — tofu phases switch back to quiet redirect
into $TOFU_LOG (the previous tee'd version regressed live noise vs
the original silent redirect). Each phase now prints one colored
ok/✗ line; on failure the tail of $TOFU_LOG dumps inside a single
collapsed group, so the live log stays scannable but the failing tail
is one click away.
3. junit-report render.py — compact summary: short status icons (✅/❌)
in a single column, no separate err column, single-line failure list
(truncated to 120 chars) instead of <details> blocks that wrap badly
on narrow screens.
Per-suite test runs still wrap in ::group:: with an inline cyan banner
so pass output collapses and the suite name is obvious when expanded.
* test(e2e): add harness.Phase / Step / Detail for human-readable test output
Test fixtures were emitting flat walls of t.Logf with every line prefixed
by the same `lb_test.go:61:` (file:line of the call site that the
t.Helper() chain bubbles up to) and forced 6-space indent. Hard to skim,
hard to find where one phase ends and the next begins.
Add three helpers in tests/e2e/harness/log.go:
Phase(t, fmt, ...) — flush-left colored banner via fmt.Println.
Bypasses t.Logf so the line lands without the
testing framework's indent/file:line prefix.
Pure visual structure; the diagnostic detail
stays on t.Logf.
Step(t, fmt, ...) — single-line sub-phase marker via t.Logf.
Keeps file:line attribution so a triager can
find the source of a progress message.
Detail(t, k, v, ...) — key=value resource IDs via t.Logf. Same
attribution rationale as Step — IDs are
diagnostic and want to land in JUnit XML
bucketed with the owning test.
Why the split: Phase is purely a banner; losing JUnit attribution costs
nothing because banners aren't diagnostic. Step/Detail carry triage
context (instance IDs, AMI, etc.) and a failing test's JUnit testcase
node should still contain them, so t.Logf attribution is kept.
ANSI colors are auto-disabled when stdout isn't a TTY and GITHUB_ACTIONS
isn't set, so piped local `go test -v` stays clean.
Apply to lb_test setupSharedFixture — five Phase banners replace the
implicit run of helper logs ("Discovering cluster capacity",
"Ensuring SSH key pair", "Creating shared VPC topology",
"Launching app instances", "Launching probe client") so the live CI log
reads as a story rather than a flat resource dump.
* ci(e2e-go): strip indent + file:line on display, propagate color env over SSH
Two display bugs spotted on the previous run:
1. Phase banners came out white instead of bold-cyan because the test
binary runs over SSH on the tofu VM, where neither GITHUB_ACTIONS
nor a TTY is present, so harness.log.go init() disables ANSI escapes.
Fix: export GITHUB_ACTIONS=true and FORCE_COLOR=1 in the remote env
so the helper keeps emitting color codes.
2. Every t.Logf line still rendered as
lb_test.go:61: instance type: t3.nano
(4-space indent + `<file>:<line>:` prefix) because t.Helper()
bubbles file:line to the call site. Fix: pipe the test binary's
stdout through `sed -E 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'`
so the display strips both. Tee to the artifact log happens BEFORE
the sed so raw output is preserved for triage; only the live UI
sees the cleaned form.
* feat(e2e-harness): extend harness for single-node Go suite port
Stage A of mulga-siv-89 — add the primitives the single-node run-e2e.sh
port consumes:
- AWSClient gains IAM + STS service handles, plus NewAWSClientWithCreds
for per-tenant account-scoping in Phase 8.
- awserror.go: ErrorCodeIs, AssertAWSError, ExpectError — Go equivalent
of the bash expect_error helper used throughout the negative phases.
- poll.go: WaitForInstance/Volume/Snapshot/Image state to any target,
with per-resource default timeouts tuned from the bash sleep loops.
- spxcli.go: thin shellouts for spx get nodes/vms, spx top nodes,
spx admin account create with output parsing for tenant-id/access-key.
- rawhttp.go: signed raw POST against the AWS gateway for the Phase 8
InvalidAction probe (gateway rejects on auth before action parsing).
- ovn.go: OvnNbctl/OvnSbctl wrappers + SkipIfNoOVN for Phase 8b-e
datapath checks; WaitForPortGroupMember handles propagation polls.
- ec2helpers.go: InstancePublicSSHHost (public-IP-or-qemu-hostfwd),
LsblkRootGiB, serial-console-access trio, DiscoverDefaultVPC,
idempotent AuthorizeSSHIngress.
- account.go: AccountCarousel maps tenant name -> scoped AWSClient,
built atop NewAWSClientWithCreds for the Phase 8 isolation tests.
Pure groundwork — no new tests, no production behaviour change. The
cert + lb scenarios still compile against the extended harness.
* ci(e2e-go): auto-cancel + reap orphans to prevent runner wedge
A cancelled run on the single-runner ci-single host has been leaving
orphan tofu/terraform procs (and stale tfstate locks) behind, which
caused the next push's tofu destroy to hang for 45m+ and saturate the
runner host until sshd refused new connections.
- Add `concurrency: e2e-go-${ref}, cancel-in-progress: true` so a new
push to feat/e2e-go-harness cancels the in-flight run instead of
stacking on the same self-hosted runner.
- New `Reap Orphan Procs` step at job start: SIGTERM then SIGKILL any
lingering `tofu`/`terraform` from the runner user, and unlink any
stale `.terraform.tfstate.lock.info`.
- Harden `Teardown`: drop `set -e` (scrub-env must always run as the
libvirt-level fallback), reap orphans before destroy, and wrap
`tofu destroy` in `timeout 600` so a wedged provider can't pin the
job to its 45m timeout.
* ci(e2e-go): drop push trigger, manual dispatch only
Pull the on.push branch trigger so the workflow only fires from
workflow_dispatch — matches the e2e-ddil.yml Phase-1 pattern. Push
trigger lifts again once the single-node Go port (mulga-siv-89) is
green end-to-end and run-e2e.sh is retired.
* test(e2e-single): port run-e2e.sh phases 1-4 to Go
Stage B of mulga-siv-89. Lands the TestSingleNode orchestrator + Fixture
that threads state between sequential phase subtests, plus phases:
- Phase 1: KVM check, AWS gateway TLS reachability, daemon NATS
readiness polling, region/AZ smoke.
- Phase 1b: spx get nodes / top nodes / get vms (single-node only).
- Phase 2: nano instance-type + arch discovery.
- Phase 2b: serial-console-access default/enable/disable round-trips.
- Phase 3: create-key-pair + import-key-pair lifecycle; test-key-1 PEM
kept for Phase 5 instance launch (cleanup deferred to teardown phase
in stage G).
- Phase 4: architecture-aware AMI discovery via name filter
ami-ubuntu-24.04-{x86_64,arm64}, falls back from spx import output.
Phase ordering is contractual (Phase 5+ depend on AMI, key pair, AZ).
detectPoolMode reads [network] external_mode from spinifex.toml so
stages F/G can skip pool-only phases on dev_networking fixtures.
No production code changed. cert + lb scenarios continue to compile.
* test(e2e-single): port run-e2e.sh phases 5-5e to Go
Stage C of mulga-siv-89 — the instance/volume/snapshot/image lifecycle
that everything else hangs off:
- Phase 5: run-instances (default VPC + default SG with tcp/22 opened
via idempotent harness.AuthorizeSSHIngress), wait running, resolve
root volume from BlockDeviceMappings.
- Phase 5a-pre: spx get vms now contains the running instance.
- Phase 5a: describe-instances metadata round-trip (type/key/AMI/BDMs).
- Phase 5a-ii: TCP wait, ssh id, lsblk cross-check vs API root volume
size (±1 GiB tolerance — lsblk rounds down), soft hostname check.
- Phase 5a-iii: get-console-output round-trip on InstanceId.
- Phase 5b: create-volume (10 GiB) → modify-volume (20 GiB) →
attach-volume → detach-volume → delete-volume, polling each state
transition through harness.WaitForVolumeState.
- Phase 5b-ii: describe-volume-status on the root volume.
- Phase 5c: create-snapshot → wait completed → copy-snapshot → both
visible → delete original → copy survives → delete copy.
- Phase 5d: AMI snapshot-backed launch verification via
DescribeImages.BlockDeviceMappings[].Ebs.SnapshotId. Predastore-side
config.json reads are TODO (no S3 client in harness yet).
- Phase 5e: create-image --no-reboot, wait available, capture backing
snapshot ID for Phase 9 cleanup.
Phase 7a (attach to stopped → error) stubbed in volume_test.go for
Stage D to fill once the lifecycle stop/start lands.
* test(e2e-single): port run-e2e.sh phases 5f-8 to Go
Stage D of the single-node Go port (mulga-siv-89). Adds four files
under tests/e2e/single/ covering Phase 5f (SG egress ACL enforcement),
Phase 6 (tag CRUD), Phases 7 / 7a / 7b / 7c-pre / 7c (stop, start,
attach-to-stopped negative, ModifyInstanceAttribute, reboot, multi-count
RunInstances), and Phase 8 negatives (18 sub-tests asserting AWS error
codes via harness.ExpectError / AssertAWSError / PostAWSAction).
Wired into TestSingleNode in main_test.go; phase ordering preserved
because Phase 7 chain mutates the shared instance state (stopped at end
of 7, .small after 7b, running after 7c-pre, untouched by 7c siblings)
and Phase 8 negatives need a running instance with the AMI/key created
earlier.
Removes the placeholder phase7a_AttachToStoppedError stub from
volume_test.go (real impl now in lifecycle_test.go).
run-e2e.sh stays in tree until step-7 cleanup per the parent plan.
* ci(e2e-go): wire e2e-single target + add single to build set
Lets the in-progress single-node Go port (mulga-siv-89, Stages A-D
landed) be exercised in CI via explicit `suites: e2e-single` dispatch
input ahead of Stage G flipping the workflow default.
Workflow default suites unchanged until Phase 9 teardown lands.
* test(e2e-single): port run-e2e.sh IAM 1-7 + Phase 8 account scoping to Go
Stage E of the single-node Go port (mulga-siv-89). Adds two files
under tests/e2e/single/:
- iam_test.go (903 lines): IAM Phases 1-7 — User CRUD, Access Key
Lifecycle, User Authentication (via harness.NewAWSClientWithCreds),
Policy CRUD (+ CreatePolicyVersion / SetDefaultPolicyVersion /
DeletePolicyVersion which bash lacks), Policy Attachment &
Enforcement (allow / deny / wildcard / root bypass), Policy Lifecycle
(DeleteConflict while attached), and idempotent Cleanup.
- account_test.go (1186 lines): Phase 8 EC2 Account Scoping — 11 Steps
(Account Setup, Instance / Volume / KeyPair / Snapshot / VPC-Subnet /
IGW-EIGW Scoping, Account Settings, Global Resources, Edge Cases,
Cleanup). Uses harness.AccountCarousel for the multi-profile
credential map and SpxAdminAccountCreate for admin account
bootstrap.
Fixture extended with IAM cross-phase state (IAMAdminAccount,
IAM{Alice,Bob,Charlie}{KeyID,Secret}) so Phase 7 cleanup is defensive
even when an intermediate IAM phase short-circuits. All 11 Phase 8
Steps share local state in the outer function; t.Cleanup mirrors
Step 11 LIFO so a mid-Step failure still tears down accounts.
Wired into TestSingleNode. run-e2e.sh stays in tree until step-7
cleanup per parent plan.
* fix(e2e-harness): use kebab-case is-default filter for DescribeVpcs
AWS EC2 DescribeVpcs filter names are kebab-case (is-default), and the
daemon's filter allowlist enforces that. The harness sent the camelCase
SDK field name (isDefault), so the daemon returned InvalidParameterValue
400 and DiscoverDefaultVPC fataled — leaving Phase 5 InstanceID empty
and cascading 16 downstream Single-Node E2E subtests to fail.
* test(e2e-single): port run-e2e.sh phases 8b-8e to Go
Stage F of the single-node Go port (mulga-siv-89). Adds three files
under tests/e2e/single/:
- vpc_test.go (553 lines): Phase 8b (pool-mode VPC public/private
subnet E2E — CreateVpc, dual subnets with MapPublicIpOnLaunch
assertions, IGW attach, custom route table + 0.0.0.0/0 route,
instance launch + SSH + external connectivity probe) and Phase 8c
(route table validation — main RT association, local + IGW route
presence, custom RT CRUD + association/disassociation, negative
delete-local + delete-main assertions).
- natgw_test.go (467 lines): Phase 8d (pool-mode NAT GW E2E —
CreateNatGateway with EIP, private subnet route via NAT, bastion-
hopped SSH into private VM, baseline egress drop / post-NAT egress
succeed / post-teardown egress lost). Inline waitForNATGatewayState
helper bounds the pending->available->deleted state machine via
harness.EventuallyErr.
- datapath_test.go (326 lines): Phase 8e (OVN-gated SG-to-SG datapath
— two SGs with UserIdGroupPair ingress rule, OVN port-group
membership + SB address-set wait loops, allowed traffic curl,
denied traffic nc, revoke->drop, re-add->restore).
Pool-mode gating via fix.PoolMode for 8b/8d; OVN gating via
harness.SkipIfNoOVN for 8e. Per-phase t.Cleanup LIFO ensures
teardown unwinds even on partial failure. Wired into TestSingleNode.
* test(e2e-single): port run-e2e.sh phases 9, 9a, 9b to Go
Stage G of the single-node Go port (mulga-siv-89) — final teardown +
post-teardown verification. Adds tests/e2e/single/teardown_test.go
(226 lines):
- phase9_Teardown: DeleteSnapshot(CustomAMISnapID) -> DeregisterImage
(CustomAMIID) -> TerminateInstances(InstanceID) + wait terminated ->
DeleteKeyPair(KeyName). Each call swallows *NotFound via
harness.ErrorCodeIs; unexpected errors logged as warnings so 9a/9b
always run.
- phase9a_VerifyTeardown: DescribeInstances must show terminated or
NotFound; DescribeImages must NOT include the deregistered AMI;
DescribeKeyPairs must NOT include the deleted key. Clears the
consumed Fixture fields.
- phase9b_FinalClusterStats: harness.SpxGetVMs must NOT list the
terminated InstanceID; orphan i- rows surface as soft Logf warnings
to avoid flakes when parallel suites share the node.
Wired into TestSingleNode. With this, mulga-siv-89's single-node Go
port (Stages A-G) is feature-complete; run-e2e.sh remains in tree
until a separate cleanup bead deletes it after CI burn-in.
* ci(e2e-go): unbuffer sed in test pipeline for live log streaming
GNU sed defaults to block-buffered stdout when stdout is a pipe
(GitHub Actions runner). Accumulates 4-64KB before flushing, so
test output appeared in sporadic batches unaligned to test
boundaries. -u flag forces line-buffered output so each go test -v
line surfaces in the Actions UI as it is produced.
Side effect: none. No process count change, no SSH session change,
no test execution change. Pure output flushing on the runner host.
* fix(e2e-single): persist key PEM across phases and skip unimplemented IAM ops
Three bugs blocked the Single-Node Go E2E suite after the DescribeVpcs
filter fix:
1. Phase 3 wrote the SSH PEM into `t.TempDir()`, which Go cleans up when
the Phase 3 subtest exits. Phases 5a-ii / 5a-iii / 7b / 7c-pre then
saw "Permission denied (publickey)" / "Identity file not accessible"
and the cascade left the instance in a state that broke 7c-pre's
`start-instances`. Move the PEM under a TestSingleNode-scoped
`fix.TmpDir` so it survives until the whole test exits.
2. IAM Phase 2's `require.NotEmpty(t, fix.IAMAdminAccount, "")` was
intended as a no-op (Phase 4 populates the field), but require.NotEmpty
fails on empty regardless of message. Removed the bogus check.
3. The daemon does not yet implement `UpdateUser` (IAM Phase 1) or the
policy-version family (IAM Phase 4): `CreatePolicyVersion`,
`ListPolicyVersions`, `SetDefaultPolicyVersion`, `DeletePolicyVersion`.
They return `InvalidAction` and abort the phase. Wrap each block in a
t.Run sub-test with t.Skip so the surrounding phase still runs and
the skips show up explicitly in the test report.
* test(e2e): parallelise hot phases + tune fast-path polls
Squash merge of feat/e2e-parallel-quickwins (mulga-siv-96). Four
quick wins from decomposing run-e2e.sh into Go phase subtests:
- Phase 8 negatives (single/negative_test.go): t.Parallel() inside
each of the 18 read-only error-path sub-tests. ~5-10x speedup.
- IAM Phase 4 policies (single/iam_test.go): EC2ReadOnly stays
seeded sequentially (populates fix.IAMAdminAccount); the other
four policies fan out as siblings under a wrapping
create_policies_parallel t.Run. ~5x on the Phase 4 sub-step.
- Volume + snapshot fast paths (single/{volume,snapshot}_test.go):
pass harness.WithPoll(500ms) on the five state transitions that
finish in <1s on local QEMU. Halves Phase 5b / 5c wall-clock.
- Cert per-IP loop (cert/cert_test.go): t.Parallel() inside the 4
inner t.Run(ip, ...) blocks. Cuts cert phase by len(nodes).
No harness changes, no Fixture changes, no test-execution-order
changes. mulga-siv-96.
* test(e2e-lb): resolve peer availability before shared fixture setup
setupSharedFixture builds a VPC + 2 app VMs + probe client (minutes
on capacity-tight dev hosts) and previously ran BEFORE the peer
pickup + ssh.Ping. On single-node, the InternetFacing_* subtests
would correctly t.Skip on peer == "" — but only after the fixture
work and once subtests started, so the wait appeared to be attached
to the first subtest name gotestsum printed.
Reorder: pickPeer + ssh.Ping first, log "internet-facing will skip"
immediately, then build the fixture for the internal subtests.
No behavior change for multi-node + peer-reachable; on single-node
the skip rationale surfaces before the fixture wait.
* test(e2e-single): extend SSH wait budget to 3m on slow boot paths
CI run 26034322018 timed out Phase5a_ii_SSH (60s budget) and
Phase8b_VPCSubnetE2E (2m budget) on the TCP-port-open poll while
sshd was still finishing pam/cloud-init. Bump all four wait sites
to 3m as a stop-gap:
- single/instance_test.go phase5aii_SSHProbe
- single/lifecycle_test.go waitForSSHReady (Phase 7b/7c/7c-pre)
- single/vpc_test.go phase8b_VPCSubnetE2E
- single/natgw_test.go phase8d bastion SSH wait
Proper fix tracked under mulga-siv-101 (hoist a harness.WaitForSSH
that probes the actual handshake, not just tcp/22 reachability).
* test(e2e-single): skip-gate IAM3/IAM5/IAM6 until daemon handlers land
CI run 26034322018 surfaced three IAM daemon gaps:
- IAM3_UserAuthentication: alice's freshly-created access key returns
403 from DescribeInstances — daemon doesn't honour scoped IAM key
signatures yet.
- IAM5_PolicyAttachmentEnforcement: cascades from IAM3 (alice can't
sign).
- IAM6_PolicyLifecycle: DetachUserPolicy returns 404 even on
attachments Phase 5 confirmed — daemon attachment ledger gap.
Skip-gate each phase with t.Skip + a pointer to mulga-siv-100 so the
remaining IAM coverage (IAM1/IAM2/IAM4/IAM7) keeps surfacing
regressions, and the suite as a whole doesn't fail on known gaps.
Flip to real assertions once the daemon handlers land.
* test(e2e-single): probe full SSH handshake (not just TCP port open)
CI run 26036025293 hit "Connection timed out during banner exchange"
on Phase5a-ii / 8b / 8d even with the 3m TCP-port-open budget: tcp/22
opens immediately, but sshd is still finishing pam/cloud-init, so the
first real runSSH call afterwards races the half-open daemon.
Switch waitForSSHReady to issue an actual ssh BatchMode + ConnectTimeout
handshake against the target, retrying until the daemon completes the
banner exchange. Drop the four TCP-only Eventually loops in favour of
the upgraded helper:
- instance_test.go phase5aii_SSHProbe
- lifecycle_test.go waitForSSHReady (Phase 7b / 7c-pre / 7c)
- vpc_test.go phase8b_VPCSubnetE2E
- natgw_test.go phase8d_NATGateway (bastion)
Signature: waitForSSHReady(t, host, port, keyPath). Backwards-compat
alias waitForSSHHandshake exposed for the instance_test caller. Proper
harness hoist tracked under mulga-siv-101.
* test(e2e-harness): dual-emit Step + Detail for live CI streaming
go test's testing framework buffers t.Logf output per-test and releases
it only on PASS/FAIL. Long subtests (Phase8Acct_AccountScoping ~115s,
Internal_NLB ~60s, Phase8b_VPCSubnetE2E 2-3m, Phase8d_NATGateway ~90s)
therefore went silent in CI for the entire subtest duration, even with
sed -u line-buffering in the workflow pipeline.
harness.Phase already wrote to os.Stdout directly for this reason.
Extend the pattern: have Step and Detail dual-emit — write the same
line to os.Stdout (streams live) AND t.Logf (keeps JUnit per-test
attribution). The duplicate on a failure is acceptable; both copies
carry identical content.
mulga-siv-103 (note: numbering skipped past siv-102 which was already
filed in the parent branch).
* test(e2e-harness): drop t.Logf duplicate from Step + Detail
The dual-emit (Fprintln + t.Logf) caused every Step/Detail line to
appear twice after long subtests: once live via stdout, once again
when the testing framework released its per-test buffer at
PASS/FAIL. Most visible after lb subtests (Internal_NLB ~60s) where
the bulk release dumped ~dozens of duplicated lines at once.
Drop the t.Logf copy. go-junit-report parses stdout between
`=== RUN` and `--- PASS/FAIL` markers, so per-test attribution in
JUnit XML is preserved without the duplicate.
Refs: mulga-siv-103
* test(e2e-harness): stream lb-waiter progress live in CI
WaitForLBActive, WaitForTargetsHealthy, WaitForENICleanup, and
WaitForInstanceRunning emitted progress via t.Logf, which the
testing framework buffers per-test. After the previous commit
dropped the dual-emit, that meant every poll loop went silent
until subtest end — visible in CI as 'nlb is active' hanging on
screen for the whole duration of the subsequent traffic check.
Route progress through harness.Step (direct stdout). Also emit a
'waiting for X' Step at the start of each helper so observers
know what the silent stretches actually represent.
Refs: mulga-siv-103
* test(e2e): dump VPC/IGW datapath diagnostics on Phase8b SSH timeout
Phase8b's 3min SSH timeout on a fresh-VPC VM (CI run 26037892507) is
the customer-visible surface of mulga-siv-105 — vpcd's handleIGWAttach
returns before ovn-northd compiles + the gateway chassis installs
flows. Two other cleanup-only bugs (siv-106 missing spinifex:cidr,
siv-107 dnat_and_snat double-delete) showed up in the same triage.
Keep the test hard-failing so the regression stays visible, but dump a
triage bundle on timeout so each occurrence is actionable:
- guest console output (last 4KB — boot + cloud-init state)
- ovn-nbctl show + ovn-sbctl show
- Logical_Router external_ids (confirms siv-106 reproduction)
- NAT rules (confirms siv-105 + siv-107 surface)
- Port_Binding chassis,up (confirms gateway port up state)
Diagnostics go straight to stdout (Step/Detail rationale) so they
stream live in CI rather than dumping at PASS/FAIL.
Refs: mulga-siv-105, mulga-siv-106, mulga-siv-107
* ci(junit-report): drop parent t.Run containers from tally + list skips with reasons
GitHub Actions test summary was counting Go's top-level Test* parent
plus its failing subtest as two distinct failures (the parent is just
a rollup of its children). The summary now de-dupes by dropping any
test case whose name is a strict prefix of another's, so the per-file
counts and the Failures block both match the actual leaf-test
outcome. CI run 26037892507 was reporting '2 failed' for single when
it should have been 1.
Also adds a collapsible Skipped section listing each skipped leaf test
with its t.Skipf call-site reason (the .go:<line>: line from the
CDATA). ANSI escapes from harness.Phase banners are stripped so the
reason is plain text. Reasons are useful triage signal — current run
has 9 skips spread across daemon gaps (mulga-siv-100) and pool-mode
gates (InternetFacing_*).
Refs: mulga-siv-89
* ci(e2e-go): propagate cancel to remote .test procs + nuke VM before destroy
Two related leaks made cancelled runs strand QEMU work on the libvirt host:
1. The long `ssh tf-user@VM "for SUITE in …; do ./$SUITE.test"` ran
without -tt, so when the runner cancelled the step the local ssh
died but sshd never sent SIGHUP through (no PTY). The remote
`.test` binary orphaned and kept running until its own 30m
`-test.timeout`, holding tmp dirs, NATS connections, and qcow2 file
handles.
2. Teardown went straight to `tofu destroy` without first killing any
stragglers on the VM, so destroy raced the .test's open file handles
on the root disk — sometimes successfully, sometimes leaving the
libvirt host with a half-released qcow2.
Fix:
- Allocate a PTY on the run-suites ssh (`-tt`) so sshd HUPs children
when the local ssh dies.
- Add a trap (EXIT INT TERM) in the run-suites step that ssh's back
to the VM and pkill's `/spinifex-e2e-bin/*.test` belt-and-braces.
- Mirror the same pkill at the top of Teardown so destroy runs against
a quiesced VM.
Both cleanups are scoped to the per-run wan_ip — safe to land while
multiple ci-single runner agents run concurrent jobs against different
tofu envs on the same host (we deliberately keep that parallelism).
Refs: runner host wedged 2026-05-19 (post-cancel orphans suspected;
no journal access yet to confirm).
* ci(e2e-go): scope orphan reap to GITHUB_WORKSPACE so it spares sibling runners
Reap Orphan Procs (job start) and the in-Teardown pkill block were using
`pkill -u $UID_SELF -x tofu/terraform` plus `find $HOME -maxdepth 6 -name
.terraform.tfstate.lock.info`. The single host runs 5+ self-hosted runners
all as `tf-user`, so every e2e-go run was SIGKILLing in-flight tofu from
sibling workflows (e2e.yml, e2e-ddil.yml, e2e-baremetal.yml, ...) and
deleting their state-lock files — exactly the kind of cross-runner
collateral the reap was meant to recover from.
Scope by `/proc/$pid/cwd` starting with $GITHUB_WORKSPACE; each runner's
workspace dir is unique on the host, so only this run's own tofu procs
match. Find for stale lockfiles is likewise scoped to $GITHUB_WORKSPACE.
* ci(e2e-go): bump setup-go v5->v6, upload-artifact v4->v7
GitHub now warns Node 20 actions are deprecated. e2e-go.yml lagged the
rest of the repo; bump to match versions already in use:
- actions/setup-go@v5 -> @v6 (matches spinifex.yml)
- actions/upload-artifact@v4 -> @v7 (matches release.yml, e2e-ddil.yml)
checkout@v6 and create-github-app-token@v3 are already current across
all workflows. The local junit-report action is `using: composite`
(shell-only), so the Node runtime change does not apply.
* fix(vpcd): add flows-ready barrier after IGW attach (mulga-siv-105)
handleIGWAttach wrote the external switch + gateway port + LRP + default
route to OVN NB and returned immediately, before ovn-northd had compiled
NB -> SB and before ovn-controller on the gateway chassis had installed
the resulting flows. A subsequent RunInstances -> vpc.add-nat could
complete while the datapath was still dark; the launched VM was then
unreachable on its public IP for whatever flow-install latency the
chassis happened to incur. Customer-visible as 'I just created a VPC,
launched a VM, and can't SSH in' for ~minutes.
Reproduced deterministically in CI run 26037892507 Phase8b: a VM in
a fresh VPC was unreachable at 192.168.0.202:22 for 3+ minutes, while
a VM at the same public IP in the warm default VPC (Phase8d) was
reachable in ~40 seconds.
Block at the end of handleIGWAttach on `ovn-nbctl --wait=hv sync`,
which bumps NB_Global.nb_cfg and waits for every connected chassis to
acknowledge the new sequence number — i.e. flows installed. Bounded by
--timeout=30; overruns log a WARN and continue (a 30s OVN stall is
already a deeper problem than failing VPC create would surface).
waitForFlowsHV is a package var so tests can stub it; in environments
without ovn-nbctl on PATH the shell-out fails fast and the WARN path
returns nil, so existing IGW tests stay green without a live OVN.
Refs: mulga-siv-105
* fix(vpcd): store spinifex:cidr on VPC create (mulga-siv-106)
handleVPCCreate built the OVN logical router with only spinifex:vpc_id
and spinifex:vni in ExternalIDs. Cleanup paths (handleIGWDetach + NAT
sweeps) read external_ids['spinifex:cidr'] to scope SNAT removal to
the VPC's own CIDR; when unset, they fell back to an overbroad
10.0.0.0/8 and emitted a WARN.
reconcileVPC already stored the same key at startup; the live-create
path drifted. Add the field to handleVPCCreate, log the cidr alongside
vpc_id, extend TestTopologyHandler_VPCCreate to assert the external_id
is set.
Refs: mulga-siv-106
* fix(vpcd): make handleDeleteNAT idempotent on 'already absent' (mulga-siv-107)
Instance termination fans out through multiple cleanup paths
(instanceCleanerAdapter.ReleasePublicIP AND
VPCServiceImpl.DeleteNetworkInterface), both of which publish
vpc.delete-nat for the same (vpc, logical_ip). The publish is
fire-and-forget (utils.PublishNATEvent uses PublishEvent for
vpc.delete-nat, not RequestEvent), so callers never branch on the
response. The second publisher then triggered a spurious
'ERROR vpcd: failed to delete dnat_and_snat rule ... not found' log
line ms after the first delete succeeded — confusing triage signal
with no actionable consequence.
Add ErrNATNotFound sentinel to the OVN client interface and wrap the
not-found return paths in both LiveOVNClient and MockOVNClient.
handleDeleteNAT matches errors.Is(err, ErrNATNotFound) and treats
that as success (logs INFO 'already absent (idempotent)').
Existing TestTopologyHandler_DeleteNAT_NotFound asserted the opposite
contract; the comment claimed EIP disassociation needed the error
signal, but DisassociateAddress also fire-and-forgets vpc.delete-nat,
so the contract never had a real consumer. Test renamed to
_Idempotent with assertSuccess.
Other failure modes (router missing, OVN unreachable, transact error)
still surface as ERROR + reply with the error string.
Refs: mulga-siv-107
* ci(e2e-go): cluster JUnit failures + name likely root cause (#230)
* ci(e2e-go): cluster JUnit failures + name likely root cause
New composite action `.github/actions/e2e-analyze` runs after
`go-junit-report` converts each suite log. For every `<failure>` it
strips ANSI, normalises dynamic noise (EC2 IDs, UUIDs, IPs, timestamps,
file:line) to a stable signature, picks the earliest non-cascade
failure per suite as the likely root cause, and buckets the rest by
signature. Output is appended to `$GITHUB_STEP_SUMMARY` and written to
`<log-dir>/analysis.md` so it travels with the artifact bundle.
Cascade markers (`must populate fix.`, `Should NOT be empty`,
`Expected value not to be nil`) collapse the typical "one upstream
phase fails, ten downstream `require.NotEmpty` assertions trip" pattern
into a single grouped entry under the real first failure.
The analyser is advisory — it always exits 0. The existing
"Fail job if any suite failed" gate keeps owning exit-code semantics.
Stage 1 of `docs/development/improvements/e2e-go-failure-analysis.md`
(mulga-siv-102). Stage 2 (per-failure log slicing) and Stage 3
(rule-based hints via `tests/e2e/.failure-rules.yaml`, placeholder
seeded here) are independent follow-ups.
CI wiring:
- e2e-go.yml: new "Analyze e2e failures" step between "Publish test
summary" and the artifact upload, always-runs.
- spinifex.yml: new `make test-actions` step so the golden-file tests
under `.github/actions/e2e-analyze/testdata/` run on every PR.
- Makefile: `test-actions` target; kept out of `test-cover` so CI-only
tooling doesn't dilute coverage %.
Golden fixtures: a synthetic cascade-vpcs case (one Phase 5
InvalidParameterValue root + 15 downstream cascades), one real
single-failure case captured from run 26037892507, and a clean
no-failures case.
* ci(e2e-go): per-failure log bundles with time-window slicing
Stage 2 of the e2e failure analyzer. For each failure surfaced by
Stage 1, materialise <log-dir>/analysis/NN-<TestName>/ containing:
failure.txt — summary (test, start, duration, file-site, error)
journal.log — spinifex-journal.log sliced to
[StartAt - 5s, StartAt + Duration + 5s]
test.log — test-<suite>.log block between `=== RUN <name>`
and the matching `--- FAIL` marker
<copies> — any files the harness already dropped at
<log-dir>/<TestName_with_/_replaced>/
Report links each entry to its bundle so a reader on the Actions
page goes straight to the relevant slice.
Workflow now invokes journalctl with --output=short-iso so the
analyzer can parse wall-clock timestamps for the journal window.
Lines without a parseable timestamp inherit the previous line's
classification (keeps multi-line records and stack traces
contiguous).
Render output is unchanged when BundlePath is empty, so the Stage 1
golden snapshots still match.
* ci(e2e-go): restore GOTOOLCHAIN=auto after setup-go v6 bump
setup-go@v6 flipped the GOTOOLCHAIN default from "auto" to "local",
which makes Go refuse to auto-download the toolchain version named
in go.mod. The self-hosted ci-single runner ships /usr/local/go
older than spinifex/go.mod's `go 1.26.3`, and build-install-artifacts.sh
prepends that path so the system Go wins the resolution race. Under
v5 ("auto"), Go silently fetched 1.26.3 on demand; under v6
("local"), the Build Install Artifacts step errors out:
go: spinifex/go.mod requires go >= 1.26.3 (running go 1.26.1; GOTOOLCHAIN=local)
Re-set the env to "auto" so behaviour matches pre-v6 runs.
* ci(e2e-go): re-export GOTOOLCHAIN=auto after setup-go
Workflow-level env doesn't help: setup-go@v6 writes GOTOOLCHAIN=local
to $GITHUB_ENV, which overrides workflow env in every later step.
Add a dedicated step right after setup-go that appends
GOTOOLCHAIN=auto to $GITHUB_ENV so the Build Install Artifacts step
(which uses /usr/local/go from a stale runner path) can self-heal by
fetching the toolchain named in go.mod, matching pre-v6 behaviour.
* fix(vpcd): add flows-ready barrier after dnat_and_snat add
handleAddNAT wrote the NAT rule to OVN NB and returned immediately,
mirroring the gap siv-105 closed for handleIGWAttach but on the NAT
add path. ec2d/RunInstances then returned success while ovn-northd
was still compiling NB->SB and ovn-controller hadn't installed the
DNAT/SNAT flows on the gateway chassis, so the freshly launched VM
was unreachable on its public IP for whatever flow-install latency
the chassis happened to incur.
Reproduced in CI run 26072432957 Phase8b_VPCSubnetE2E: VM at
192.168.0.212 timed out at 3min while diagnostics showed
Port_Binding up=true for both the VM eni and the chassis-redirect
gateway. SB-side convergence was complete; the missing piece was the
same `ovn-nbctl --wait=hv sync` barrier siv-105 added to IGW attach,
applied at the end of handleAddNAT as well.
Reuses the existing package-var `waitForFlowsHV` so tests can stub
it; no new infra. Parallel unit test asserts the call site fires
exactly once per add-nat.
* ci(e2e-analyze): real-artifact fixes for journal slicing
Three bugs surfaced after Stage 2 ran against real CI artifacts in
run 26072432957:
1. Regex `[+-]\d{4}` rejected the colon-offset form (e.g. `+10:00`)
that systemd's `journalctl --output=short-iso` actually emits on
most versions. Added a parser that tries RFC3339 first and falls
back to the no-colon `-0700` layout. Result: 0-line journal slices
become populated.
2. go-junit-report's `timestamp` attr on <testsuite> records the
moment the XML was produced (post-suite), not when the tests
started. Computing StartAt as `that + cumulative-priors` placed
every failure AFTER the journal collection ended, so windows
always missed. Workflow now writes `${ARTIFACT_DIR}/test-<suite>.start`
right before running each suite; analyzer prefers that file over
the junit timestamp.
3. Persisted `OffsetFromSuiteStart` on each Failure so the
suite-start override can recompute StartAt without re-parsing the
XML.
Unit tests added for the colon-offset journal format. End-to-end
test (TestWriteBundles_EndToEnd) still passes with the new layout.
* ci(e2e-analyze): skip parent testcases when summing suite offsets
go-junit-report emits one <testcase> for the parent Go test AND each
subtest, with the parent's time = sum of its children. ParseFile was
summing every testcase into the suite-cumulative-offset counter, so
every leaf's offset was inflated by ~the parent's total duration.
Visible on run 26073817490 as Phase8b's StartAt landing 13 minutes
after the suite finished collecting logs — beyond the journal window,
so the per-failure journal slice was empty.
Pre-compute the leaf set per suite (a name is a leaf iff no other
testcase name starts with `<name>/`) and only accumulate cumul on
leaves. Golden snapshots updated; the offset for the only failure in
each fixture now matches the actual leaf-sum start time.
* test(e2e-harness): capture IP-targeted datapath state on Phase8b SSH fail
Phase 1 diagnostics for mulga-siv-111. The Phase8b SSH timeout
reproduced on the e2e-failure-analysis branch with siv-105's IGW
barrier AND the new add-nat barrier BOTH firing cleanly, so the
residual gap is downstream of `--wait=hv sync` — most likely stale
OVS conntrack, stale upstream-host ARP, or an OF flow-install miss.
DumpVPCFlowDiagnostics now takes a VPCDiagnosticsOpts carrying the
external IP, logical IP, and per-test artifact dir. When set, the
helper greps `ovs-ofctl dump-flows br-int`, `conntrack -L`, and
`ip -4 neigh show` for the external IP / logical IP and snapshots
`ovn-nbctl find NAT external_ip=…`. Each capture is both printed
inline (existing affordance) and written to a separate file under
fix.Artifacts so the Stage 2 analyzer bundler picks it up alongside
the journal + test-log slices, no analyzer changes needed.
Phase8b's caller now passes pubIP / privIP / fix.Artifacts so the
captures fire on the next failure; updates the t.Fatalf message to
point at siv-111 instead of siv-105 (which is the now-closed parent).
* fix(e2e-single): authorize SSH ingress on fresh-VPC default SG in Phase8b
Phase 8b stands up a non-default VPC and runs an instance there. Its
auto-created default SG denies all ingress, so the SSH probe times out
even when every datapath barrier converges — OF flow captures show the
SYN reaches OVN, gets DNAT'd at table 13, then drops at the ACL between
tables 13 and 17. The bash driver relies on admin init having pre-baked
this rule on the *default* VPC's default SG; a fresh VPC has not.
Add a DescribeSecurityGroups + AuthorizeSSHIngress step between
route-table setup and run-instances, using the existing harness helper.
Fix surfaced via mulga-siv-111's Phase 1 diagnostics (OF flow + ACL
drop counts), invalidating the H1/H2/H3 stale-state hypotheses in the
bead plan doc.
* test(e2e): dual-detect ubuntu image version (24.04/26.04) during gold-image transition
After mulga gold image bumped to v6-fastboot (stages ubuntu-26.04 / resolute
instead of ubuntu-24.04 / noble), all consumers that hardcoded 24.04 break on
the new image. Make them detect at runtime so v3 and v6 envs both pass.
- tests/e2e/run-e2e.sh: DescribeImages filter accepts both
ami-ubuntu-{26,24}.04-<arch>, then derives IMAGE_NAME from the match.
- tests/e2e/run-pseudo-multinode-e2e.sh: prefer ~/images/ubuntu-26.04.img,
fall back to ubuntu-24.04.img; pass detected version to spx import.
- tests/e2e/single/image_test.go: walk candidate catalog names (26.04 first,
then 24.04), stop on the first one DescribeImages can resolve.
- .github/workflows/e2e-baremetal.yml: download resolute (26.04) into
BM_ISO_CACHE; update BM_UBUNTU_IMAGE env var path.
- docs/terraform-workbooks/{bastion-private-subnet,s3-webapp,nginx-webserver}:
AMI filter matches `*ubuntu-26.04*` OR `*ubuntu-24.04*` (AWS values=
array is ORed).
Companion changes for mulga (bootstrap scripts, baremetal, iso pipeline)
land in scripts/tofu-cluster/* and scripts/{tofu-iso,baremetal,iso-builder}/*
under bead mulga-siv-114. Drop the 24.04 fallbacks once all envs are on the
v6 gold image and 24.04 entries are removed from spinifex/utils/images.go.
* feat(e2e): add service-interfaces.yaml manifest + validator
Source of truth for which spinifex services produce/consume which NATS
subjects, and which e2e suites cover which services. Consumed downstream
by the targeted suite selector (Bead 4) to derive subset E2E runs from a
PR diff so single-service changes stop paying the full 25 min suite cost.
- docs/service-interfaces.yaml: 7 services, 4 active suites (cert, lb,
single, ddil). Subscribes / publishes / depends_on filled from grep
of NATS subjects in source. spinifexui marked covers_no_suite so
UI-only diffs skip e2e entirely. Future suites (multinode, reboot,
tofu, baremetal) commented in until Go ports land.
- tests/e2e/manifest-check: schema + cross-reference validator (path
existence, subject regex, duplicate detection, suite covers resolves
to a real service, fixtures references resolve). 8 unit tests cover
the failure modes plus a load-and-validate on the real manifest.
- Makefile: manifest-check target, hooked into preflight.
- docs/DESIGN.md: link to the manifest under "NATS Messaging".
mulga-siv-116. See docs/development/improvements/e2e-targeted-suite-selection.md.
* fix(e2e): drop catalog-download import path from phase4_Image
`spx admin images import --name ubuntu-26.04-x86_64` with wantErr=true
was meant to detect that bootstrap-install.sh already imported the AMI
and short-circuit on the duplicate exit code. After the v6 gold-image
bump preloaded `~/images/ubuntu-26.04.img` and bootstrap-install.sh
switched to `--file` for the import, two things broke at once:
- The catalog `--name` path no longer hits a duplicate guard for the
--file-imported AMI, so the import re-downloads ~857MB and either
succeeds (failing the wantErr=true assertion) or wedges the job
timeout while the catalog server crawls along at 16KB/s.
- `--file` from the test won't work either: the test binary runs as
`tf-user` over SSH, while bootstrap-install.sh placed the image at
`/home/spinifex/images/...` (mode 0700 home dir).
bootstrap-install.sh is already responsible for staging the AMI. The
test's job is verifying DescribeImages round-trips by-name and by-ID —
not re-exercising import. Trust the bootstrap-staged AMI and stop
calling import.
Keeps the 26.04→24.04 dual-detect for the v3 gold image transition;
drop the 24.04 candidate when no envs remain on v3.
Refs mulga-siv-115 (caught by the EC2 cold-boot fast-path E2E run on
feat/ec2-coldboot-fastpath).
* ci(e2e-go): default to e2e-single + harden status-report SHA fallback
Two changes to .github/workflows/e2e-go.yml:
1. Default `suites` input now includes e2e-single so a no-input
manual dispatch exercises the full single-node ladder. Single-node
Go harness is greenable now (image_test fix at
|