mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-19 10:52:04 +00:00
d26e441007e327f3ecb392fee09986af7300a89f
68 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Julian Sommer
|
65d2501e6b | feat: add MariaDB as the second RDS engine (#762) | ||
|
Julian Sommer
|
5e17169661 | feat(e2e): complete the RDS end-to-end suite (#608) | ||
|
Josh Siviour
|
5de598e85a |
ci: trim unused E2E builds, move IMDS to the SDK, derive predastore bind in spx (#656)
* ci: skip building unused E2E suite binaries in e2e.yml's single/multi jobs These two jobs immediately rebuild suite-aware binaries after Build Install Artifacts and never consume the tar's _bin contents, so the cross-compiled suite build was pure waste. build-install-artifacts.sh gains an optional --skip-e2e-build flag (positional contract unchanged); e2e-nightly.yml and the other callers keep building the full set. * refactor: replace hand-rolled IMDS client with AWS SDK v2 internal/imdscreds, cmd/ecs-agent's metadata fetch and its credentials provider used a hand-written IMDSv2 token/GET flow with manual RFC3339 expiry parsing. A malformed Expiration silently fell back to "never expires" instead of failing. Replace it with feature/ec2/imds, credentials/ec2rolecreds and aws.CredentialsCache, following the internal/gwsign/signer.go precedent. An unparseable Expiration is now a hard error from Fetch, covered by TestFetch_BadExpirationErrors. Output shapes to callers are unchanged, and IMDS base URLs ending in /latest keep working. * refactor: derive predastore bind config from spinifex.toml in spx itself predastore-start.sh awk-parsed spinifex.toml for the bind host/port and node_id, exported them as env vars, then exec'd spx. Move that derivation into predastoreStartCmd via derivePredastoreBind, which reads the raw (non-DIAL-normalized) nodes.<node>.predastore section, defaulting node_id to -1 (co-located) when the key is absent, same as the script. An explicit --host/--port/--node-id flag or SPINIFEX_PREDASTORE_* env var still overrides the derived value. Delete the now-unused script and point the systemd unit's ExecStart directly at spx service predastore start, matching the spinifex-viperblock.service precedent. |
||
|
Julian Sommer
|
a55544b7d4 | chore: remove integration from e2e | ||
|
Josh Siviour
|
68abc0bfb3 |
fix(ci): scope e2e telemetry per cell/job and define suite topology eligibility once (#573)
* feat(ci): extend per-cell telemetry scoping to every e2e workflow ci.cell only reached e2e-nightly and e2e-suites, so a run of e2e, e2e-ddil or gpu-e2e still landed in the sink as one undifferentiated bucket. Stamp it on the remaining workflows: single/multi on e2e's two provisioning jobs, ddil and gpu on their single-job workflows. The best-effort numeric job id lookup moves out of e2e-nightly into a shared composite action, so the four other provisioning jobs get ci.job_id without copying the gh/jq incantation. The lookup now matches on env.JOB_NAME rather than interpolating the name into the jq filter, which broke on names carrying quotes or non-ASCII separators, and no-ops when telemetry is disabled. gpu-e2e gets ci.cell but deliberately no job id lookup: nothing in that job bakes the attrs into a daemon, so the API call would be paid for an env var no exporter reads. * fix(ci): define e2e suite topology eligibility once Three workflows carried three different answers to which suites a topology can run. e2e-suites and e2e-nightly ran lb on their single-node leg, where every internet-facing subtest skips for want of a peer, so a fraction of the suite reported as a green lb run. e2e-suites' multi leg ran multinode alone, dropping the lb and cert coverage that only a multi-node env can complete — so full lb coverage never ran there at all. .github/scripts/suite-sets.sh is now the single definition, sourced by all three. e2e.yml's two allow-list regexes derive from it, e2e-suites derives both matrices from it (multi_provision gains the suite-matrix step single_provision already had), and e2e-nightly's deliberately narrower per-cell lists are declared there too, guarded by an assert that fails the cell if a subset ever names a suite its topology cannot fully exercise. The nightly stays narrow on purpose: its cells have a 35-minute budget and exist to prove each install/network/host-OS permutation boots and serves. Suite breadth is the PR gate's job and already runs on every push. That is now stated in one place rather than implied by a third divergent list. |
||
|
Josh Siviour
|
cae76ba1b6 |
test: complete the e2e live-vs-mock rebalance (epic mulga-8oyg3) (#566)
* test(volume): assert chunk GC stays wired on in CreateVolume GCEnabled was zero-valued at every viperblock construction site for a long time and no test noticed. It is not part of viperblock.VBState, so it never round-trips through a saved config.json and cannot be observed by reading volume state back. Extract the viperblock.VB assembly out of CreateVolume into buildVBConfig so the GC wiring is assertable directly against the returned config, without a live viperblock or S3 backend. The config is returned by pointer because viperblock.VB carries atomic.Uint64 fields that vet's copylocks check rightly refuses to see copied. * test(e2e): snapshot backend byte accounting before volume purge DeleteVolume purges a volume's entire backend prefix, including any chunks a leak left behind, before anything measures it. That is the same masking that hid the storage-growth leak in production: a passing E2E cannot fail on backend growth when the evidence is deleted first. Snapshot the volume's backend chunk footprint in EnsureVolume's cleanup closure, immediately before DeleteVolume, so every volume the shared teardown path purges is measured rather than silently discarded. The snapshot is always taken and logged; the hard ratio check additionally runs for callers that registered a guest-written-bytes baseline, since most callers never attach or write to the volume they create and so have no baseline to compare against. The tolerance is the RS(2,1) erasure floor widened for per-chunk header and tail-chunk overhead. It is sized to distinguish unbounded growth from a footprint settling at the floor, not to catch bounded residual garbage. The check is non-fatal inside the cleanup callback: t.Fatalf's runtime.Goexit would abort the DeleteVolume call queued behind it and turn a reported leak into a real one. * test(e2e): add memory-constrained concurrent-write durability regression The existing durability tests write one 4MiB sentinel, sequentially, on a comfortably sized guest, which is the exact inverse of the known-open data-loss bug: acked writes read back zero-filled after a cold reread when a guest is memory-starved during concurrent writeback. No happy-path test can reach it, because the fault lives in the intersection of memory pressure, write concurrency and cache-cold reread. Add a suite that encodes those three conditions together: a guest discovered at or below the working set, N parallel large writers launched as background jobs in a single command so they genuinely overlap, swap disabled, then sync and drop_caches before rereading. Assertions are per stream rather than aggregate so a failure names the losing writer instead of leaving whoever reads it to re-derive which of N writers lost data from one combined digest. GuestExec gains a timeout-parameterised variant. Its fixed two-minute default is sized for sub-second payloads, and a multi-gigabyte write to a network-backed volume would exceed it, so the suite would have failed by timeout while looking like it had caught the corruption. The budget scales with the working set, and a deadline is reported distinctly from a data mismatch so the two red states are never confused. The suite is dispatchable but deliberately absent from the default SUITES: the bug it reproduces is still open, so it is expected to fail and must not sit in a merge-gating path. It is registered in the workflow allow-list so it stays discoverable rather than being buried behind a skip. Also record on the live IAM instance-profile test why its assertions were not moved down to the integration tier, so the trace is not repeated. * test(e2e): drop backend byte accounting, assert a node-wide delta instead The accounting helper listed the internal predastore bucket to attribute bytes per volume. That listing is refused unconditionally: predastore authorises bucket access by same-account ownership, and the bucket belongs to the service account, so a test caller gets 403 no matter what it holds. Granting the test a service credential would invert the trust boundary to buy a measurement, which is the wrong trade. Replace it with what the S3 surface actually permits: sample node-wide allocated bytes either side of the write and bound the delta against the bytes the guest wrote. Coarser, but it measures something real rather than depending on a call that cannot succeed. The tolerance moves into the suite that uses it. It encodes RS(2,1)'s 1.5x expansion floor plus headroom; a cluster running the recommended RS(3,2) has a 1.667x floor and only ~12% of margin left, so the constant needs revisiting if the encoding changes. * revert: drop memdurability e2e suite The suite asserted the guest read back what it wrote after a memory-constrained concurrent write, but the assertion could not fail for the reason it claimed: it compared the guest's own read against the guest's own write with the page cache still warm, so it passed whether or not the data reached the backend. A test that is green on a build with a known-open durability bug is worse than no test — it advertises coverage that does not exist. Removes the suite and its Makefile/workflow wiring, plus DiscoverInstanceTypeAtLeastMemory, which had no other caller. GuestExecTimeout stays: GuestExec now delegates to it, and the old two-minute hard cap it replaced was a latent trap for any long guest command. * test(e2e): add partial-block concurrent-write durability suite Replaces the removed memdurability suite, which passed on a build that still carried the bug it claimed to test. The engine reaches its read-modify-write path only for a partial block; a fully covered block takes a full-overwrite branch that reads nothing and can lose nothing. The old workload wrote through ext4 with 1 MiB buffers, which the filesystem coalesced into aligned full-block requests: block-layer tracing recorded zero unaligned and zero sub-4096 requests, so the path under test never ran, and writers 4 MiB apart could not collide on a block even in principle. This suite writes the shape the defect needs. Two writers issue concurrent O_DIRECT writes into disjoint halves of the same 4096-byte blocks on a raw, unformatted device, stamping every half with the round number, then drop the page cache and reread with O_DIRECT asserting every half carries the final round. One half reading back at an earlier round is the signature of the lost update. The old suite also could not fail even had it reached the path: its write phase ended in 'wait; sync; sha256sum' and reported sha256sum's exit status, while bare wait returns 0 unconditionally, so writer errors were invisible and a backend ENOSPC presented as a pass. Here each writer's exit status and exact byte count are asserted before any durability claim is made. Working set is 8 MiB rather than 2048 MiB, and the volume is deleted on teardown; the predecessor left roughly 3 GiB of backend per run and twice drove a node under the nearfull watermark. Dispatchable but not in the default suite list until a green run and a demonstrated fail-first against a pre-fix engine exist. * test(e2e): let the fixture own the working-set volume delete The suite deleted the volume in its own cleanup and the fixture deleted it again on close, so every run ended with an InvalidVolume.NotFound from the second attempt. Detach stays — DeleteVolume against a still attached volume is what leaks volumes on teardown — and the delete is left to the fixture so the volume is released exactly once. * test(e2e): gate merges on the partial-block durability suite Verified on one node with the same test binary and workload, varying only the engine: 1758 of 8192 halves reverted a generation against the pre-fix build, 0 of 8192 against the fixed one. That is the fail-first evidence the suite needed before it could be trusted to gate anything. Adding it to the default suite list rather than leaving it dispatchable is the point. Its predecessor sat outside every path that runs, which is how it survived long enough to be discovered green on a build that still lost data. * test(integration): cover the ECR data plane in the front gate The live ecr/ suite drove docker, crane and skopeo against a real registry, yet none of it boots a guest: it needs a registry, an object store and an HTTP client. The in-process gateway already serves the OCI Distribution v2 surface on an ephemeral port with a memory-backed object store, so these run here in 0.18s instead of behind full provisioning. Covers the three claims the live tests made beyond authorization, which is already down here: a push/pull round trip asserting blobs and manifests read back byte-identical, refusal to auto-create a repository on push, and retagging an existing manifest without disturbing the original tag. Content is compared rather than status codes alone. A registry that stores a blob under the wrong digest, truncates it, or serves a different one answers 200 to every request in the sequence. The capacity/load leg deliberately stays live: it exists to put real parallel pressure on a real predastore, and a memory-backed object store cannot make that claim. * fix(e2e): make volume teardown release the volume before deleting it Fixture cleanups drain LIFO, so a suite that creates its instance before its volume tears the volume down first, while it is still attached. DeleteVolume answered VolumeInUse, nothing retried, and the volume was stranded on the node. Reordering the chain cannot fix this — registration order has no way to express the instance/volume dependency. Instead teardown is now self-sufficient for any creation order: it force-detaches, waits for the volume to reach available, and only then deletes. Instance teardown waits for terminated for the same reason, since TerminateInstances returns as soon as the request is accepted. Two things kept the leak invisible. A failing cleanup was logged and forgotten, so a run stayed green with resources left behind; failures are now reported as leaks, failing the bound test or surfacing through Close() for TestMain to fail the run. And a panic in one cleanup aborted the whole chain, stranding every resource behind it; each callback is now isolated. RegisterVolumeTeardown and the fixture had grown separate teardown paths, only one of which detached. Both now share it. |
||
|
Josh Siviour
|
48749576c3 |
test: rebalance E2E into in-process integration tier + shared-guest live scenarios (#556)
* test: add in-process integration test tier Roughly half of the behavioural tests under tests/e2e/ assert control-plane API shapes, authz and error codes that need no live environment, yet they can only run against a tofu-provisioned environment on a scarce self-hosted runner. This adds the tier they can move down to. The real gateway router runs in-process: SetupRoutes() is a pure function of GatewayConfig, so it serves from an httptest server over embedded NATS JetStream with real IAM/STS services and genuine SigV4 verification. Only the daemon side of the wire is faked -- the ec2.*/ebs.* NATS subjects a live node would answer are stubbed per test. Nothing is provisioned and no fixed ports are bound, so the tier runs concurrently on a shared build host and is safe as a CI front gate ahead of the live suites. It is build-tagged and kept out of preflight, which stays unit-only. * test(integration): add IAM/STS client accessors and port the zero-stub authz tests Adds IAMClient/STSClient plus ClientsWithCreds/ClientsWithSessionCreds so a test can act as a principal other than the seeded root user, and stubs the node-discovery subject in StartGateway so call paths that trigger discovery resolve immediately instead of riding out the fan-out timeout. Ports the four enforcement tests that guard on ec2:DescribeRegions, a gateway-local action, so they need no daemon-side stub at all. * test(integration): port remaining IAM/STS authz tests from e2e Ports TestIAMUserCRUD, TestIAMAccessKeyLifecycle, TestIAMUserAuthentication, TestIAMPolicyCRUD, TestIAMPolicyAttachmentEnforcement, TestIAMPolicyLifecycle, TestIAMGroupsLifecycle, TestIAMRolesAndProfiles and TestSTSAssumeRoleAndGetCallerIdentity from tests/e2e/iam/ into the in-process integration tier. Each test now self-bootstraps its own users/policies/roles instead of relying on shared suite-level state, since every integration test gets its own gateway and embedded NATS. TestIAMRolesAndProfiles stubs ec2.IamProfileAssociation.describe because DeleteInstanceProfile always counts live associations over NATS regardless of which guard actually fires. TestIAMCleanup is not ported: it only swept a shared long-lived E2E environment, and every ordering/dependency assertion it made is already covered by TestIAMPolicyLifecycle, TestIAMGroupsLifecycle and TestIAMRolesAndProfiles's own delete-conflict guards. * test(e2e): remove IAM/STS authz tests migrated to the integration tier All 13 signed-request authz tests plus TestIAMCleanup now have faithful, mutation-proven equivalents in tests/integration/ — the gateway dispatches IAM/STS actions directly with no daemon hop, so the in-process tier is a faithful and far cheaper substitute. The iam/ e2e suite now runs only TestIAMInstanceProfileAssociation and TestIMDS. Drop the now fully-dead iam_helpers_test.go alongside the six deleted test files, and remove harness.IAMGroupARN/IAMDeleteGroupBestEffort, which had no remaining caller anywhere in the repo once iam_groups_test.go was gone. * feat(integration): daemon-lite harness for key/tags/routetable/vpc Adds DaemonLite, an in-process stand-in for a live spinifex daemon that subscribes the real key, tags, route table, and VPC/subnet/SG/IGW service implementations directly to their NATS subjects, backed by memory and embedded-JetStream stores. Ports TestKeyPairs, TestTagManagement, TestRouteTableValidation, and TestReplaceRouteConvergence from tests/e2e/single so the integration tier drives genuine daemon-side business logic instead of StubSubject canned replies. vpcd (OVN) is out of scope, so its "vpc.create-sg"/"vpc.delete-sg" acknowledgements are stubbed with a fixed success reply — the SG record itself is still written by the real service impl before that event fires, so the stub only unblocks the synchronous round-trip rather than replacing tested behavior. * ci(e2e): gate live E2E jobs behind the in-process integration tier Runs the in-process integration suite first, on the ci-single runner host rather than inside a provisioned VM, so a broken control plane is caught before either live job pays the Spinifex install. Both e2e_single and e2e_multi now require it, and e2e_status treats a skipped integration job as a failure so the gating status cannot go green on a gate that never ran. Drops the shadow-mode suite selector: targeted per-diff suite selection is not the direction being pursued, and leaving a non-functional shadow job in the gating workflow only invited it to be mistaken for a live gate. * test(e2e): remove single/ tests superseded by daemon-lite ports TestKeyPairs, TestTagManagement, TestRouteTableValidation, and TestReplaceRouteConvergence now have proven-equivalent ports in tests/integration/ (mutation testing confirmed parity). Delete the e2e originals and their runX helpers, along with keypair_test.go's and tags_test.go's file-local helpers left with no caller. generateImportPubKey is shared by other single/ tests so it moves to helpers_test.go instead of being deleted. Slims the manifest-lint baseline for the removed create-call entries. * ci: run the lb e2e suite on multinode only Both e2e jobs filter SUITES by exclusion, and neither excluded lb, so the suite ran twice per e2e run: 354s on single and 654s on multi. Balancing across instances on one node is not a deployment mode we support, and ELBv2 proper requires two AZs, so the single-node run was redundant. Drops ~354s from the single-node job, which is the gate critical path. * test(integration): port 3 of 7 ECR tests to the in-process tier Wires the gateway's ECR control plane (repository/policy/lifecycle metadata over an in-process daemon-lite NATS subscriber, blob and manifest bytes via a memory object store, JWT issuer/verifier keyed off the existing IAM master key) into the integration harness, and ports TestECRControlPlane, TestECRGetLoginPassword, and TestECRIAMAuthorization_AssumedRolePullOnly from tests/e2e/ecr — each proven equivalent by mutation (evaluatePrincipalPolicy forced to allow, and the repo-describe path forced to not-found) before its E2E original was removed. The other 4 authorization tests (PullOnly, ReadOnly, DetachPolicyDeniesImmediately, DeactivatedKeyReturns401) stay on the E2E tier: AttachUserPolicy lacks the AWS-managed-ARN existence-check bypass that AttachRolePolicy/AttachGroupPolicy already have, so attaching AmazonEC2ContainerRegistryPullOnly/ReadOnly straight to a user fails NoSuchEntity in both tiers alike. Left untouched pending a fix to that shared IAM code. * fix(iam): accept AWS-managed policy ARNs on AttachUserPolicy AttachUserPolicy called getPolicyByARN unconditionally, so attaching an AWS-managed ARN to a user returned NoSuchEntity. AttachRolePolicy and AttachGroupPolicy already skip that lookup for AWS-managed ARNs and store them opaquely; the user path was missed when that guard was introduced. ListAttachedUserPolicies had the mirror of the same gap: it resolved every ARN through getPolicyByARN and silently dropped the ones it could not find, so a managed ARN would disappear from the listing rather than round-trip. Evaluation already handled these correctly via resolveAttachedPolicy, so only the write and list paths needed the guard. * test(integration): port remaining 4 ECR authorization tests Ports TestECRIAMAuthorization_PullOnly, _ReadOnly, _DetachPolicyDeniesImmediately, and _DeactivatedKeyReturns401 from tests/e2e/ecr now that AttachUserPolicy accepts AWS-managed policy ARNs. Each was proven equivalent by mutation before its E2E original was removed: PullOnly/ReadOnly/DetachPolicyDeniesImmediately fail when evaluatePrincipalPolicy is forced to always allow, and DeactivatedKeyReturns401 fails when the access-key-active check in resolveECRLongLivedPrincipal is bypassed. All mutants were reverted and a clean run confirmed before committing. tests/e2e/ecr/iam_authorization_test.go is now fully removed — every test it held has an in-process equivalent, and its helpers had no other callers in the package. * ci: select e2e suites by allow-list instead of deny-list Both jobs filtered SUITES by exclusion, so any suite named in neither deny-list ran in both jobs. That is how lb came to run twice per e2e run, and it makes adding a suite silently fan it out everywhere. Each job now names the suites it is eligible to run. Selection is still driven by SUITES, so this is behaviour-preserving for the current default: single/iam/cert on the single-node job, multinode/lb/cert on the multi-node job, exactly as before. eks, ecs and storagegrowth are listed as eligible on the single-node job but are not yet in the default SUITES. * test(e2e): delete seven tests with no coverage TestFinalClusterStats and TestInstanceLaunch assert nothing beyond what other callers of their shared helpers already imply. The multinode baseline, volume, guest-SSH, and preflight tests duplicate single-node coverage or exercise per-guest behaviour that isn't cluster-specific. TestMultinodeInstanceDistribution can't fail: three independent RunInstances calls with no placement group make a strict spread assertion flaky and a loose one meaningless. Preflight checks (/dev/kvm writability, peer SSH reachability) move into requireMultiNodeFixture so the package fixture singleton still fails fast with a clear message instead of disappearing along with the dedicated test. The remaining callers of needInstanceTrio (CrossNodeOps, CrossNodeGateway, NodeFailure, VolumeDurability) still trigger the shared trio launch on their own, so removing the distribution and volume-lifecycle tests doesn't leave it unbuilt. Also slims the manifest-lint baseline entry for the deleted multinode volume test. * fix(e2e): require full 3-node spread in TestMultinodeSpread SpreadPlacement passed at unique>=3, warned via t.Logf at unique==2, and only failed below that. A 33% spread failure (two instances colocated) passed with a log line. The 243.6s NAT-egress scenario that follows only proves what it claims to if the private trio actually landed on three distinct chassis, so that premise needs to be a hard assertion, not a warning. * refactor(e2e): drop the unused spread-assertion harness helpers AssertSpreadAcrossNodes and CountInstancesPerNode lost their only caller when the instance-distribution test was removed. Beyond being dead, the assertion helper documented minNodes=1 as an acceptable logging-only mode inherited from the bash driver, which is the pattern that let a placement test pass while every instance landed on one node. InstanceHostingNode and its ps helpers stay; five multinode tests use them. * fix(ec2): stop leaked goroutines on prepareEFIVolume error paths newViperblock starts the chunk uploader and WAL syncer unconditionally, but prepareEFIVolume only released them on the happy path via Close(). Six error returns between acquisition and Close left both goroutines running. Add a defer immediately after acquisition, matching the sibling prepareRootVolume; Close() remains the durability boundary. * test(predastore): add real predastore fixture for integration tests Extract predastore_integration_test.go's inline daemon startup into a reusable testutil/predastore package so tests/integration can point DaemonLite-style wiring at a real S3-compatible backend instead of an unreachable host. Kept as its own leaf package rather than folding into the shared testutil package: it transitively imports predastore's quicclient, whose package-level connection pool starts a permanent goroutine at init that would trip goleak checks in any of the ~80 other packages that import testutil for unrelated NATS helpers. * test(integration): migrate CreateVolume to assert real predastore persistence Add StartVolumeDaemonLite, an opt-in DaemonLite variant wired to the real predastore fixture, and migrate TestCreateVolume_PersistsToRealPredastore off memory-backed DaemonLite so CreateVolume/DescribeVolumes exercise viperblock's real S3 backend end to end instead of stopping at input validation. * test(integration): add real storage-state oracle for CreateVolume DescribeVolumes only proves the handler's own read path agrees with its own write path, both via the same deserialization; it cannot catch a bug in a field it never re-derives (e.g. a unit-conversion error in the bytes handed to viperblock, since VolumeMetadata.SizeGiB is copied from the request directly). vbscan reads a volume's persisted config.json and chunk objects straight over S3, independent of any spinifex handler code, so TestCreateVolume_PersistsToRealPredastore can assert on what viperblock actually wrote rather than only on what the handler claims. vbscan is a local reimplementation of scripts/vbrefscan's read-only scan rather than an import: spinifex cannot depend on mulga, the module that contains it. It only reimplements the config/chunk-listing half of vbrefscan — the checkpoint-based reference-integrity half needs viperblock.ParseBlockCheckpointBytes, which does not exist in the viperblock version spinifex currently depends on (v1.12.1) and is also moot for this volume: CreateVolume never calls SaveLiveCheckpoint, so a freshly created volume has no checkpoint to scan regardless of version. * refactor(test): move the predastore daemon fixture out of the coverage scope test-cover runs go test ./spinifex/..., so anything under spinifex/ is compiled and measured. The daemon fixture cannot be exercised by unit tests, so it reported a coverage gap no unit test could close. The e2e harness already avoids this by living under tests/; the fixture now does the same. vbscan stays put, being ordinary library code with its own unit tests. * test(integration): migrate RunInstances multi-count expansion off live guests RunInstances with MinCount/MaxCount=2 expands into a single per-node targeted launch (distributeInstances in gateway/ec2/instance/placement.go) before the daemon-facing NATS hop, so the real gateway router plus a stubbed daemon response is sufficient to prove it — no throwaway VM is needed to exercise the expansion logic itself. The live single-node E2E test's WaitForInstanceState/TerminateInstances sequence was teardown hygiene for the real guests it booted, not an assertion of the expansion behavior, so it is dropped rather than emulated. The migrated test additionally asserts the per-node launch request itself carries MinCount=MaxCount=2, proving the count actually propagates rather than the launch succeeding by coincidence or via the zero-daemon degenerate path in placement.go. * docs: drop a doc reference to the deleted live test * test(integration): migrate launch-template RunInstances expansion off live guests RunInstances --launch-template resolves ImageId/InstanceType from the stored template entirely gateway-side (expandLaunchTemplate, before validation/placement/the daemon-facing NATS hop), so proving it forwards the resolved fields never needed a real guest boot. Ports the assertion to tests/integration via a new opt-in StartLaunchTemplateDaemonLite (real LaunchTemplateServiceImpl over embedded JetStream) plus a direct subscription on the per-node launch subject to capture what the gateway actually dispatched, rather than only the client-visible response. The live test keeps its CRUD/versioning lifecycle (create/modify/delete version, tag-filtered describe) — none of it touches a guest — and drops only the RunInstances-from-template + wait-for-running + terminate tail, whose only job was waiting out a real boot to read back fields already proven resolved. * test(integration): migrate KV-only account-scoping checks off live tenants Ports the KeyPair/VPC-Subnet/IGW-EIGW/AccountSettings cross-account isolation assertions from tests/e2e/single/account_test.go to the in-process integration tier. Each of these resources resolves ownership from a plain account-scoped KV key (utils.AccountKey(accountID, id), or an account-keyed settings record) with no vm.Manager or real guest involved, so DaemonLite now also wires the real EgressOnlyIGWServiceImpl and AccountSettingsServiceImpl alongside the existing VPC/IGW/Key services. Two independently minted tenant accounts (via IAMService.CreateAccount/CreateUser/CreateAccessKey/CreatePolicy/ AttachUserPolicy, in-process — the same calls runAccountCreate makes without the CLI/stdout round-trip) replace the live tier's carousel so isolation is checked between real, separately owned accounts. Left live: instance scoping (Step2) and volume scoping (Step3) — checkInstanceOwnership, the actual cross-account gate for stop/start/ reboot/terminate/attach-volume, runs daemon-side only after a real vm.Manager lookup, so it cannot be exercised without a live guest. Snapshot scoping (Step5) needs a real backing volume for CreateSnapshot (viperblock, out of DaemonLite's scope), so it stays live entirely rather than split off only its two KV-only assertions. Global-resources (Step9) carries no account branching in the handlers at all, so asserting its equality across two accounts wouldn't exercise anything a fake node responder didn't already trivially satisfy — not migrated. * test(integration): migrate launch-template CRUD lifecycle off live guests TestLaunchTemplates booted zero guests: its RunInstances coverage had already moved to TestRunInstances_LaunchTemplateExpansion, leaving only template CRUD/versioning assertions occupying a full live-cluster slot. Port the remaining create/describe/version/modify/delete lifecycle, including the SourceVersion merge and a nested timestamp round-trip, into TestLaunchTemplates_CRUDLifecycle against the in-process integration gateway. * refactor(tests): move spot instance lifecycle coverage to integration tier RequestSpotInstances is a mock over on-demand RunInstances (no bidding, interruption, or reclamation), so booting real VMs in the e2e tier proved nothing spot-specific. Migrate the SIR CRUD lifecycle (request, describe, cancel, close-on-terminate, insufficient-capacity error) to a NATS-level integration test that exercises the real gateway orchestration and daemon service impl without a live guest. Real-VM-boot assertions are already covered by TestRunInstances_MultiCountExpansion and the e2e baseline tests. * fix(tests): share one embedded NATS server across the integration package ClientTokenStore in spinifex/gateway/ec2/instance is a process-wide sync.Once singleton bound to whichever *nats.Conn first initialises it. Every integration test booted its own embedded NATS server, so the first RunInstances-calling test captured the singleton, and any later test inherited a KV handle bound to that server's now-closed connection. TestMain now boots one embedded NATS+JetStream server for the whole package; every StartGateway call connects into its own dynamically provisioned NATS account (accountAuthenticator in shared_nats.go), giving each test a private JetStream namespace despite the shared server, and none of the per-test connections are closed until every test has finished - whichever one the singleton latches onto stays live for the run. Also wire the suite into the required PR gate: it was previously only invoked from e2e.yml's push-triggered (main/dev), self-hosted "Integration (front gate)" job, so a regression here never blocked a PR. The whole package runs in single-digit seconds, so it now runs inline in `make preflight` and as its own step in spinifex.yml's build_and_test job. * test(integration): migrate serial-console access checks off live guests Enable/Disable/GetSerialConsoleAccessEnabled is pure gateway+account-settings logic with no guest involved, so it moves to the in-process integration tier against the real AccountSettingsServiceImpl wired into DaemonLite. * test(integration): migrate DescribeImages round-trip off live guests runImage's AMI describe round-trip needs no real cluster: it registers an AMI against a hand-seeded snapshot via the real ImageServiceImpl, backed by a new StartImageDaemonLite helper wiring RegisterImage/DescribeImages onto an in-memory object store. * test(integration): migrate predastore object lifecycle off live guests runPredastoreObjectLifecycle only needs a real predastore daemon, not a tofu-provisioned cluster: it now runs against tests/fixtures/predastore's in-process fixture, covering the same create/put/get/delete/list contract. * test(integration): migrate ListFoundationModels catalog check off live cluster ListFoundationModels is pure gateway-side static catalog logic with no NATS/daemon hop, so it moves to the integration tier via a new BedrockClient harness accessor; this tier configures no credential resolver, so the Anthropic entry's absence is asserted explicitly rather than left optional. * test(integration): cover RunInstances client-token idempotency Proves replaying the same ClientToken through the real gateway returns the original reservation without a second daemon dispatch, and a different token launches independently. * test(integration): cover RunInstances launch-template expansion Proves version selection (omitted Version resolves to the template default, explicit Version selects that version) and field precedence (explicit RunInstances parameters override the template, unset fields inherit it) through the real gateway. * test(integration): cover RunInstances malformed-input rejection Proves a malformed ImageId and a malformed targeted-launch CapacityReservationId are rejected before any NATS dispatch, with no daemon stub registered so a regression that let either through would fail fast instead of hanging. * test(integration): cover RunInstances iam:PassRole enforcement Proves a caller granted ec2:RunInstances but not iam:PassRole on an instance profile's role is denied, and the identical session succeeds once PassRole is granted for that exact role ARN. * test(integration): cover RunInstances placement-group strategy routing Adds a placement-group daemon-lite wiring a real PlacementGroupServiceImpl over NATS, then proves spread strategy assigns exactly one instance per node while cluster strategy pins the whole launch to the single highest-capacity node. * test(integration): cover RunInstances capacity distribution across nodes Proves the default (non-placement-group) path packs a multi-instance launch across nodes proportionally to spare capacity rather than exhausting one node first. * test(e2e): merge SameSGComms and SGToSGDatapath around one client/target pair Both booted their own instance pair to check SG-to-SG datapath enforcement and same-default-SG east-west connectivity. Merging them around one shared client/target pair cuts four boots to two, and converting the inline harness.Step sequence into named t.Run stages (PortGroupMembership, AllowedTraffic, DeniedTraffic, RevokeAndVerifyBlocked, ReauthorizeAndVerifyRestored, SameSGComms) gives each its own pass/fail attribution instead of one opaque test result. SameSGComms now runs last and joins the default SG onto each ENI via ModifyNetworkInterfaceAttribute only after every SG-to-SG allow/deny assertion has already recorded its own result, since the default SG's self-reference rule admits all traffic between co-members and would otherwise defeat the denied-traffic check. * test(e2e): merge default-SG reachability, guest DNS, and egress enforcement DefaultSGReachabilityBaseline, GuestDNSResolution, and SecurityGroupEgress each booted their own instance to check one facet of "a guest is reachable only when policy says so": ingress deny/authorize, resolver + WAN egress, and egress ACL enforcement. Merging them around one shared guest cuts three boots to one, with named t.Run stages (Deny, Authorize, Resolver, WANEgress, RevokeEgress/RestoreEgress) giving each its own attribution. The egress revoke/restore round-trip now mutates this scenario's own dedicated SG instead of the shared default SG the rest of the suite depends on, isolating its egress-rule mutation from every other test. * fix(ci): single-quote e2e test_run regex across the ssh boundary * fix(e2e): anchor SG egress-enforcement probe to a WAN IP, not the gateway * test(e2e): merge single-node guest churn tests into one durability scenario Combines TestSSHProbe, TestENIHotplug, TestENIHotplugReconcile, TestVolumeDurability, TestModifyInstanceAttribute, and TestRebootInstance into TestGuestChurnDurability, a single shared-guest scenario that re-verifies a data-integrity sentinel sha256 after every perturbation stage (ENI hotplug, daemon restart, ENI detach, instance-type resize, stop/start, reboot). This proves the sentinel survives the full churn sequence rather than just one isolated stop/start. * fix(vpc): clear subnet drop gate when installing egress reroute A subnet with an attached IGW but no 0.0.0.0/0 route carries a priority-1100 drop gate on the VPC router. When a route later lands (NATGW reroute at 900, IGW reroute at 1000) the reroute is installed but the drop gate was never removed, so the higher-priority gate wins and black-holes the subnet's egress in lr_in_policy before it reaches routing or SNAT. This left NAT-gateway private-subnet egress with 100% packet loss: the SNAT stage saw zero packets and no conntrack was committed. Remove the subnet's drop gate for the same prefix once a reroute is installed. Idempotent and prefix-scoped, so a specific-prefix reroute is a no-op and leaves the default gate intact. * test(e2e): merge VPC subnet, NAT gateway, and EIP single-node tests Merges the former TestVPCSubnetE2E, TestNewVPCEgressBaseline, TestNATGateway, and TestInstanceEIP into one staged scenario, TestVPCEgressPaths, around one scenario-owned VPC and its two guests, cutting five guest boots to three. The public guest serves PublicSubnetEgress, then the NAT Gateway bastion hop, then (last, since association rewrites its public IP) the EIP flip. The private guest serves the NAT Gateway path, launched directly into the same VPC's already-existing private subnet instead of a dedicated one. RouteBeforeSubnet keeps its own second VPC and instance since it exercises the implicit main-route-table code path, a different one than the explicit route table this scenario's own VPC uses for its public subnet — collapsing the two would drop coverage of whichever path got merged away. The NAT Gateway up/down flip now repeats for an env-tunable round count (SPINIFEX_VPCEGRESS_NAT_ROUNDS, default 1), matching every other merged single-node scenario. * test: TEMPORARY mutation-check — revert |
||
|
dependabot[bot]
|
5dc945a6df |
chore: bump the github-actions group with 2 updates (#552)
Bumps the github-actions group with 2 updates: [actions/setup-go](https://github.com/actions/setup-go) and [actions/setup-node](https://github.com/actions/setup-node). Updates `actions/setup-go` from 6 to 7 - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](https://github.com/actions/setup-go/compare/v6...v7) Updates `actions/setup-node` from 6 to 7 - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](https://github.com/actions/setup-node/compare/v6...v7) --- updated-dependencies: - dependency-name: actions/setup-go dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: actions/setup-node dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Josh Siviour
|
ffc34b735b |
ci: emit per-run ES summary doc from e2e workflow (#551)
Add e2e_es_summary, a self-hosted job that aggregates suite-status.txt from both e2e legs and POSTs one doc per run to the Elastic sink, keyed by labels.ci_run_id, so Kibana can answer "did run X pass, and where are its artifacts" without hand-aggregating traces-apm*/logs-apm.error*. Runs on ci-single (LAN route to the sink) since e2e_status stays on GitHub-hosted ubuntu-26.04 for the PR status-report check. State is computed with the same single_ok/multi_ok logic as e2e_status so the two can never disagree. |
||
|
Josh Siviour
|
e78d0dbac9 |
fix(ci): upload full tofu apply/destroy log as an artifact after teardown (#489)
Every tofu-based e2e workflow set TOFU_LOG to a /tmp path outside ARTIFACT_DIR. ci-fmt.sh's quiet_run helper appends all tofu init/apply/ destroy output to that file, but the Teardown step (which runs the destroy) always ran AFTER the existing upload-artifact step, so the full log — including destroy-time failures — was silently discarded on every run. Add an if: always() upload-artifact step after Teardown in each tofu-based job (e2e.yml, e2e-suites.yml, e2e-nightly.yml, e2e-ddil.yml) that uploads $TOFU_LOG directly. Since quiet_run appends both apply and destroy output to the same file, one upload after Teardown now captures everything, recoverable even when destroy itself fails. Shipping this log to Elasticsearch/OTLP is deferred to mulga-5n7fe, which will consume this artifact. |
||
|
Josh Siviour
|
a7b689fec3 |
feat(otel): full-stack telemetry — tracing, metrics, logs + per-VM guest metrics (#479)
* feat(otelsetup): OTel init, resource attrs, trace-stamping slog handler Env-gated on OTEL_EXPORTER_OTLP_ENDPOINT: with no endpoint only the W3C propagator is installed and globals stay no-op, so instrumented binaries ship everywhere at zero cost. Resource carries service/host/mulga.env/ mulga.source plus OTEL_RESOURCE_ATTRIBUTES (ci.run_id in CI). * feat(otelsetup): wire telemetry init into all spx services Every spx service start command installs the trace-stamping JSON slog default and the OTel providers via initTelemetry, flushed on shutdown. Gateway's own slog setup now routes through otelsetup so its level handling keeps trace correlation. * feat(gateway): HTTP tracing middleware on awsgw and daemon cluster API Server span per request with W3C traceparent extraction; awsgw spans are renamed to the resolved SigV4 service.Action post-auth with account and region attributes. Request log lines carry trace_id via context. * feat(nats): propagate W3C trace context over command subjects NATSRequestCtx injects traceparent beside X-Account-ID and opens a client span per hop; ServeNATSRequest opens a consumer span joining the producer trace, with a Ctx variant for handlers that thread context. RunInstances is plumbed end-to-end (gateway dispatch through placement fan-out to the daemon handler) and viperblockd ebs.mount/unmount join inbound traces. * feat(systemd): source optional /etc/spinifex/telemetry.env in service units Lets operators enable OTLP export (OTEL_EXPORTER_OTLP_ENDPOINT, MULGA_ENV, MULGA_SOURCE) by dropping a single env file, with no unit edits. File is optional; absent means telemetry stays disabled. * feat(ci): export OTel telemetry from E2E envs to the central Elastic sink All E2E workflows set SPINIFEX_OTLP_ENDPOINT (repo variable override, default wattle) plus ci.run_id/ci.workflow resource attributes, so bootstrap-install.sh writes telemetry.env on every node. Teardown runs flush-telemetry.sh before the final tofu destroy so buffered spans and metrics reach the sink. * feat(otelsetup): request metrics and Go runtime metrics via OTLP mulga.requests counter + mulga.request.duration histogram recorded by the same middleware that opens spans: awsgw carries the resolved SigV4 action, the daemon cluster API uses the chi route pattern, and NATS handlers record per-topic at the subscribe chokepoints (node names stripped to keep cardinality flat). Go runtime metrics start inside the same env gate. * feat(daemon): deep spans for QMP commands, VM launch phases, EBS ops Single chokepoint span per QMP command in sendQMPCommand covering all callsites; vm.launch phase children (mount_volumes, start_qemu, qmp_connect, persist_state); ec2.cmd consumer span with traceparent extraction plus start/stop/reboot/terminate child ops; ebs.mount/unmount spans in viperblockd with volume attributes. * feat(telemetry): thread request context through NATS, object store and VM paths Migrate utils.NATSRequest call sites to NATSRequestCtx (241 sites; W3C traceparent now flows gateway -> daemon on every request), add GatherCtx and migrate all Gather call sites, inject traceparent on manual EKS publishes. ObjectStore interface is ctx-first with aws-sdk WithContext calls and an otelhttp transport, so predastore spans join the caller trace. NATS consumer spans record error status when the response envelope carries an error. vm.Manager and the ec2 handler service interfaces take ctx-first so vm.launch and QMP spans join the request trace; detached background work uses explicit context.Background(). Hot-path slog calls use *Context variants for trace_id binding. * feat(telemetry): suppress health-check, heartbeat and unrooted trace noise Health probes record metrics only (no span per poll); the QMP heartbeat poller marks its context so query-status ticks stop rooting a trace each. rootSampler honors MULGA_ROOT_TRACE_RATIO (ParentBased ratio, default 1.0) and the predastore unit sets 0.05: guest block I/O roots are sampled while request-linked spans are always kept. * feat(otel): export deployment.environment resource attribute Elastic APM maps it to service.environment, enabling the native environment selector across the APM UI (Services, Traces, Dependencies). * fix(telemetry): stop objectstore rooting method-only HTTP transactions Background S3 calls (state sync, cleanup loops) carried no caller span, so otelhttp rooted a trace per request and APM surfaced thousands of parentless 'HTTP GET/PUT/DELETE' transactions with expected-404 probes counted as failures. Emit the client span only when the request context already carries one, matching the viperblock backend. Bump viperblock/predastore to branch heads so CI builds also pick up the equivalent span filter and root sampler. * feat(telemetry): per-VM guest metrics via dedicated QMP socket + qmp-collector Goanna Phase 1 producer, operator-consumed first. Each VM gets a second -qmp chardev (qmp-telemetry-<id>.sock) plus a sidecar discovery file carrying instance/account/vcpus/taps/monitoring-tier; the new qmp-collector service polls query-cpus-fast (vCPU thread /proc jiffies), query-blockstats, query-balloon and host tap sysfs counters, and publishes CloudWatch-mappable goanna_ec2_* series to NATS metrics.ec2.<instance-id>. A queue-group bridge in the same process forwards the series to the node-local OTLP receiver so they reach the operator sink until Goanna consumes the subjects. The collector never touches the manager's control QMP socket; telemetry failures never block a launch. * fix(qmp-collector): dial NATS via DialTarget, drop ProtectProc, fix meta race The collector connected to the raw NATS host (0.0.0.0), failing TLS verification against a cert valid for 127.0.0.1; route through admin.DialTarget like the other daemons. ProtectProc=invisible hid the QEMU vCPU threads (dumpable=0) that the collector must read from /proc/<tid>/stat, so every tick failed and re-primed without ever publishing. Drop it to default. Surface collect failures at Warn so silent metric loss is visible, drop per-tick publish to Debug, and read InstanceID under the lock in run() to fix a race with updateMeta on ENI hot-plug. * chore(deps): re-pin predastore + viperblock to dev-landed commits |
||
|
Julian Sommer
|
485cca941f | feat: IAM inline role policies (Put/Get/DeleteRolePolicy + real ListRolePolicies) (#429) | ||
|
Julian Sommer
|
446f93ae8c | chore: rename ubuntu-latest to ubuntu-26.04 | ||
|
Julian Sommer
|
8a94886b13 | feat: cloud-init EC2 datasource (#411) | ||
|
dependabot[bot]
|
79e881f1c8 | chore: bump actions/checkout from 6 to 7 in the github-actions group (#405) | ||
|
Josh Siviour
|
41dc53b7f8 |
fix(ci): prevent stale suite-status from false-failing e2e re-runs (#387)
The per-run ARTIFACT_DIR lives in /tmp, outside GITHUB_WORKSPACE, so a re-run on the same self-hosted runner reused a prior attempt's append-only suite-status.txt. The 'Fail job if any suite failed' gate's -s check then failed the job even when the surviving attempt passed with rc=0. Wipe ARTIFACT_DIR in Clean Workspace and gate solely on the per-attempt steps.e2e.outputs.rc (fail-closed: an empty rc still fails). |
||
|
Julian Sommer
|
28c6e8ccff | chore: bump CI time | ||
|
Julian Sommer
|
eced3f733a | feat: split IAM e2e (#372) | ||
|
Julian Sommer
|
058da3eff4 | fix(viperblockd): seal volume block-map to predastore on detach + durability e2es (#371) | ||
|
Josh Siviour
|
17c83cb3ad |
fix: build microVM rootfs via buildx to stop overlay2 export leak (#358)
docker export of a container leaks its overlay mount on Docker's overlay2 driver — the mount survives docker rm, so the layer dirs are never reaped except by a daemon restart. The lb-agent microVM build ran docker export on every e2e job across ~9 runners, accumulating ~69G of orphaned overlay2 layers on the CI host (disk hit 94%). Build the rootfs with `docker buildx --output type=local` from a small generated Dockerfile instead, which writes the image filesystem to a directory and releases all mounts — nothing leaks. Keep the podman path (its store is unaffected). Tighten the e2e cleanup image prune to dangling-only so the alpine base stays cached instead of re-pulling. |
||
|
Julian Sommer
|
2df35d3822 | chore: remove reboot e2e | ||
|
Josh Siviour
|
d076a434d4 |
Feat/e2e targeted suites (#352)
* test(e2e): add targeted suite selector + shadow-mode CI step
select-suites maps a diff to the e2e suites it must run: changed files →
services (manifest path/additional_paths) → consumer closure over
depends_on vs publishes/subscribes → suites whose covers intersect.
Special cases force-all (infra/harness/workflow, ≥3 services, submodule
bumps) or a lone suite dir; empty diffs fall back to default suites. An
AST scan of each suite's harness.Ensure* call graph derives per-test
service coverage and emits an optional -test.run subset (RUN_PATTERN).
Wired into e2e.yml as a non-gating 'Select Suites (shadow)' job that
logs the computed selection to the run summary so predicted vs actual
can be compared before the selector drives SUITES.
* ci(e2e): add dispatch-only per-suite job blocks grouped by env
Restructure the looped e2e flow into env-grouped provision -> matrix
suites -> teardown blocks so each suite surfaces as its own node in the
Actions graph instead of a single green tick:
single_provision -> single_suite [single,cert,lb,eks] -> single_teardown
multi_provision -> multi_suite [multinode] -> multi_teardown
Provision resolves the VM/cluster and ships test binaries, emitting WAN
IP + ssh key path as job outputs; each matrix leg runs one suite over
SSH and uploads its own junit/analysis artifacts; teardown (if:always)
reuses the persisted tofu workspace on the same runner to destroy.
workflow_dispatch only -- runs alongside the gating e2e.yml for
validation and posts no commit status.
* test(e2e): add manifest drift lint (fixture-create + NATS subject)
Bead 5 of the targeted-suite-selection plan. Two ratcheted drift guards in
tests/e2e/manifest-lint, wired into make preflight and the spinifex.yml PR job:
- Fixture lint: AST-flags direct resource-create AWS calls
(c.EC2.RunInstances, c.ELBv2.CreateLoadBalancer, ...) in suite test files,
steering them through harness.Ensure* fixtures. ~104 pre-existing call sites
are captured in a checked-in baseline; the lint fails only on NEW creates.
Intentional create-path tests opt out with an inline // e2e:allow-create.
- Subject lint: extracts NATS subject literals from the natsSub{} registry and
literal Subscribe/Publish/Request args, cross-referenced wildcard-aware
against service-interfaces.yaml. Code subjects missing from the manifest fail
(ratcheted); manifest subjects with no static reference warn only (many are
produced dynamically). Fixed real drift surfaced on first run: declared the
five spinifex.cluster.shutdown.* phase subjects under spinifex_daemon.
manifest-check is now CI-enforced alongside manifest-lint (previously
preflight/local-only).
* feat: run-scoped e2e teardown sweep
Tag every Ensure* fixture resource with e2e:run=<run-id> when
SPINIFEX_E2E_RUN_ID is set, and add a teardown.test suite that sweeps
everything carrying that tag at end of run — reclaiming resources leaked
by a crashed suite, an interrupted run, or a persistent cluster.
- harness/sweep.go: tagRunResources (EC2 CreateTags) + tagRunELB (ELBv2
AddTags) + SweepRunResources in dependency order (LB, instances, NAT,
snapshots, volumes, images, SGs, subnets, key pairs). Blank run id
no-ops both tagging and sweeping, so the sweep can never touch an
untagged or foreign resource.
- harness/fixtures.go: stamp the run tag on each created resource.
- tests/e2e/teardown: thin suite that runs the sweep; skips without a
cluster or run id, logs the report, never fails the run.
- e2e-suites.yml: build + ship teardown.test, pass the run id into suite
legs, sweep on the VM in both teardown jobs before tofu destroy.
|
||
|
Josh Siviour
|
4ef19fc488 | feat(ui): EKS console (#310) | ||
|
Josh Siviour
|
5ac8dc3fb9 |
fix(network): round-robin external pool cursor — stop instant EIP reuse (recycled EIP unreachable, EKS CI) (#320)
* Revert "fix(network): prime host neigh with gateway port MAC on centralised EIP attach"
This reverts commit
|
||
|
Josh Siviour
|
a1f3fcc531 |
ci(e2e): exclude eks suite from multinode job (#312)
EKS create-cluster fails deterministically on the multinode job with ServiceUnavailableException: the EKS KV is a clustered (Raft) JetStream stream and create-cluster needs an elected leader, but multinode JetStream clustering isn't reliably forming yet (see mulga-omq/z2z/5yg). EKS is validated single-node only at this stage. Drop eks from the multinode build+run suite filters (alongside the existing reboot/single exclusions) so the multinode job reflects what's actually supported. Single-node continues to run the full eks suite. |
||
|
Josh Siviour
|
44d0548fd4 |
ci: make microVM depmod best-effort, drop apt kmod install (#308)
build-microvm-image required depmod (kmod) as a mandatory tool to generate modules.dep. Self-hosted CI runners don't ship kmod and can't reliably apt-install it, so the inline 'apt-get install kmod' steps added to e2e/e2e-nightly/e2e-ddil were unreliable. modules.dep is an optimisation, not a boot requirement: the guest init's load_mod() (build/microvm/init.sh) falls back to insmod-by-path when modprobe can't resolve a module. So: - Drop depmod from the mandatory-tool preflight. - Run depmod only when present; still assert a non-empty modules.dep in that case to keep catching the trixie-kmod empty-dep bug. - Warn and skip when depmod is absent (guest uses insmod fallback). - Remove the apt-get kmod install steps from all three workflows. Gold-image bake still installs kmod (provision.sh), so production images get a proper modules.dep; only kmod-less CI runners take the fallback. |
||
|
Josh Siviour
|
8e0edef7c2 |
ci: install kmod (depmod) before microVM image build (e2e + nightly + ddil) (#307)
* ci(e2e): install kmod (depmod) before microVM image build #305 made depmod a hard requirement in build-microvm-image.sh, but the self-hosted ci-single/ci-multi runners don't ship the kmod package, so the Build Install Artifacts step aborts with 'required tool not found: depmod' before any E2E suite runs. Add an idempotent kmod install ahead of each Build Install Artifacts step (single/multi/eks jobs); the gold-image builder bakes kmod in too, so this no-ops once runners pick up the new image. * ci: install kmod (depmod) in nightly + ddil too e2e-nightly and e2e-ddil call the same build-install-artifacts.sh (-> make build-microvm-image) on the same self-hosted ci-single/ci-multi runners as e2e.yml, so they hit the identical 'depmod not found' abort. Add the same idempotent kmod guard before Build Install Artifacts in both. release distro builds run on GitHub-hosted ubuntu (kmod present); e2e-baremetal runs on ci-baremetal which already has it (green today) — neither needs the guard. |
||
|
Josh Siviour
|
496424c41c |
test(eks): EKS control-plane E2E suite (Sprint 6g) (#304)
* test(eks): EKS control-plane e2e suite (Phase 1, API-level)
Add tests/e2e/eks/eks_test.go driving the EKS lifecycle against awsgw:
CreateCluster -> ACTIVE (endpoint + CA asserted), kubeconfig artifact
assembly, AccessEntry CRUD, get-token presign decode, DeleteCluster -> gone.
Harness: EKS SDK client on AWSClient + WaitForEKSCluster{Active,Deleted}
pollers. Wire e2e-eks into the Makefile and the single-node CI suite list.
* test(eks): Phase 2 kubectl get-nodes data-plane gate
Add the reachability shim (harness/eks_mgmt.go) that reads the control-plane
VM's br-mgmt IP from the cluster KV meta over NATS, plus a kubectl wrapper
(harness/kubectl.go) that skips when kubectl is absent. KubectlGetNodes points
kubectl at https://<mgmtIP>:6443 (insecure-skip-tls-verify; cert SANs the NLB
DNS, not the mgmt IP) with an aws eks get-token exec block and polls until the
control-plane node is Ready — the get-token webhook auth regression gate.
IRSA pod + the real NLB/Route53 passthrough are deferred to a follow-up.
* test(eks): tolerate doubled NotFound code + dial loopback NATS
DeleteCluster detection matched the SDK constant ResourceNotFoundException,
but awsgw emits ResourceNotFoundExceptionException on the wire — match the
NotFound code by substring so deletion is detected immediately instead of
timing out. The mgmt-IP shim dialed NATS.Host verbatim (0.0.0.0:4222), which
the serving cert does not SAN; rewrite the wildcard bind to loopback.
Validated live: all 6 subtests green, kubectl get nodes shows the
control-plane node Ready via the get-token webhook auth path.
* feat: build system images with libguestfs (no host block devices)
Replace the qemu-nbd + sudo mount + chroot + /dev-bind flow in
build-system-image.sh with libguestfs (virt-customize / virt-resize /
guestfish). Every operation runs in an isolated userspace appliance VM with
its own kernel, so the build touches no host block device, no host mount, and
needs no sudo — removing the failure class that wiped the host's /dev on shared
CI runners (a leaked /dev bind traversed by rm -rf).
Applies to all distros: alpine resizes via guestfish e2fsck+resize2fs, ubuntu
via virt-resize; customization (packages, binaries, files, setup script,
service enable, cache cleanup) is one ordered virt-customize invocation.
LIBGUESTFS_BACKEND=direct (no libvirt on runners).
Wire the eks-node AMI into the single-node e2e job: the runner builds the
device-free raw, ships it to the VM, and the VM imports it (predastore/NATS are
VM-only) so the eks suite's CreateCluster has an AMI to boot.
* feat(eks): bootstrap + state-report via AWS gateway broker
Converge the EKS control-plane → host channel onto the ELBv2 lb-agent model:
the K3s server VM publishes its bootstrap envelopes and periodic state reports
via SigV4-signed HTTPS POST to the AWS gateway, instead of dialing core NATS
directly. A new gateway action (PublishInternal, POST
/clusters/{name}/internal-publish) relays the payload verbatim onto the
existing eks.bus.* / eks.state.* subjects, so the daemon-side subscriber, KV
persistence/encryption, and reconciler are unchanged.
This removes the VM's need to reach NATS at all, so NATS can stay
cluster-internal behind its bind/firewall/mTLS controls — the prior direct
path broke when NATS was not VM-reachable (e2e CreateCluster timeout) and was
DDIL-fragile (fire-and-forget nats pub, no ack/retry).
- gateway/eks: PublishInternal broker + route; closed-set channel/kind validation
- cmd/eks-gateway-publish: one-shot SigV4 publisher (retry-until-ack) baked into
the eks-node AMI; replaces the nats-cli publisher in k3s-first-boot.sh and
mulga-eks-state-report.sh
- handlers/eks + daemon: hand the K3s server VM the mgmt-reachable gateway URL +
system (Predastore) SigV4 creds + gateway CA instead of NATS URL/token/CA;
new resolveSystemGatewayBaseURL (mgmt-bridge-first)
- eks-node image: build/install eks-gateway-publish, drop nats-cli download
* feat: migrate eks-token-webhook to the AWS gateway broker
The TokenReview webhook dialed core NATS directly (STS verify + AccessEntry
KV read). On a mgmt-only control-plane VM with NATS bound off-subnet that
fails, so the webhook exited at startup, the apiserver token-webhook
kubeconfig was never written, and the apiserver crash-looped — the cluster
never reached ACTIVE.
Converge it onto the same gateway-broker model as bootstrap/state-report and
ELBv2's lb-agent: the webhook SigV4-signs an HTTPS POST {accountId,token} to
/clusters/{name}/token-review, and the gateway runs the verify + lookup
host-side and returns the resolved identity. The VM now speaks no core NATS.
- internal/eksgw: shared on-VM SigV4 client (also adopted by
eks-gateway-publish).
- handlers/eks/token_review.go: Authenticate decision core (moved from the
webhook binary) + ResolveTokenReview host-side wiring.
- gateway/eks token-review route + WebhookTokenReview handler.
- webhook: drop NATS, relay via eksgw.Client; broker fault -> 503 (retryable)
not a silent deny.
- initd: export EKS_GATEWAY_* / EKS_ACCESS_* creds, not SPINIFEX_NATS_*.
* fix: build eks-node helpers with GOFIPS140; cover token-review broker
eks-gateway-publish blank-imports internal/fipsboot, whose init() panics
unless the binary is built with GOFIPS140=v1.0.0. The eks-node AMI build
omitted the flag, so the binary paniced at first boot right after k3s came
up — k3s-first-boot (set -eu) died before publishing the bootstrap token and
the cluster never reached ACTIVE. Build both eks-node helpers with
GOFIPS140=v1.0.0, matching cmd/lb-agent.
Add NATS-backed unit tests for ResolveTokenReview and the WebhookTokenReview
gateway handler (verify responder + account KV) to cover the new broker path.
* fix: export first-boot env so eks-gateway-publish child sees it
k3s-first-boot.sh and mulga-eks-state-report.sh sourced /etc/spinifex-eks/
first-boot.env but never exported the vars, so the eks-gateway-publish child
process inherited an empty environment and exited '--account-id is required'.
The control-plane VM came up healthy but never published its bootstrap
envelopes, so the cluster never reached ACTIVE. Wrap the source in set -a /
set +a so the KEY=value lines export to child processes.
* fix: make awsgw start --host override actually bind
The 'awsgw start --host' override printed 'Overwriting awsgw host' but the
assignment was commented out, so AWSGW always bound the spinifex.toml host
(a specific WAN IP, e.g. 192.168.0.10). A system instance (EKS K3s control
plane, internal LB agent) has a mgmt-only NIC on-link to the br-mgmt IP with
no route to the WAN IP, so its SigV4 POSTs to the gateway were refused and the
cluster never bootstrapped. Honour --host (e2e passes 0.0.0.0:9999) so AWSGW
listens on all interfaces incl. br-mgmt; the serving cert already SANs every
local interface IP, and WAN clients still reach it.
* test(eks): install kubectl on e2e VM so KubectlGetNodes runs
The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full
aws eks get-token -> kubectl -> apiserver -> token-webhook -> gateway broker
-> STS verify path — the only end-to-end exercise of the token-review broker.
The e2e binaries run on the VM, where kubectl was absent, so
harness.NewKubectl t.Skip'd it and the path went unvalidated. Install a
checksum-verified kubectl (pinned to the k3s server version) on the VM,
gated on e2e-eks.
|
||
|
joshsiv-mulga
|
da26b1db69 |
fix(ci): stop image-build /dev bind from wiping host /dev
A killed ubuntu system-image build can leave `mount --bind /dev` under its build dir. The e2e Clean Workspace step then ran `sudo rm -rf "$GITHUB_WORKSPACE"/*` as root, traversed the leaked bind, and deleted the host's /dev (incl. /dev/null), wedging the runner. - e2e.yml: umount anything under the workspace (deepest-first) before deleting, and pass `rm --one-file-system` so rm can never cross a mount boundary into host /dev. Hard backstop, applied to all 3 Clean Workspace steps. - build-system-image.sh: recursive lazy unmount in cleanup() detaches the chroot binds even when busy, and mark the build subtree rprivate so the /dev bind can't propagate into the host namespace. |
||
|
joshsiv-mulga
|
af690b3299 |
test(elbv2): collect LB microVM guest console in CI + surface NLB probe failures
NLB target health stalls at 0/N in CI while the data plane works locally. The lb-agent's nginx engine activation, any nginx -t/reload error, and its active-probe results are logged to the LB microVM serial console — which the host journal never captures and the GetConsoleOutput API path reports as NotFound, so the failure is undiagnosable from CI artifacts. - e2e.yml: pull /run/spinifex/console-*.log off each node in the single- and multi-node collect steps (files persist past VM teardown) into lb-console/. - lbagent healthcheck: log probe failures at Warn (was Debug) so unreachable targets surface at the agent's default level on the captured console. |
||
|
Julian Sommer
|
c666c8146d | fix: e2e.yml | ||
|
Julian Sommer
|
044d35737e | feat: remove bash e2e for e2e-go | ||
|
joshsiv-mulga
|
7cd054cbf1 | chore(e2e): dump Logical_Router_Policy + Logical_Router_Static_Route in W5 diag | ||
|
joshsiv-mulga
|
ba7253e2c6 |
ci(e2e): fix Start Packet Capture EACCES + PID-tracking on tcpdump (siv-167)
Two bugs in
|
||
|
joshsiv-mulga
|
afd63ed967 |
ci(e2e): Phase 1 diag — tcpdump br-wan + bridge fdb + hairpin modes (siv-167)
Mulga-siv-167 Phase 1. Adds evidence-capture for the LB single-host return-path drop class: e2e_single only: - New step 'Start Packet Capture' between Bootstrap and Run E2E Tests. SSHes to test VM and starts background tcpdump on br-wan capturing TCP/22/80/443/9999 + ARP into /tmp/spinifex-diag/br-wan.pcap. PID stored at /tmp/spinifex-diag/tcpdump.pid. All three jobs (single, pseudo-multi, real-multi): - Collect Diagnostics: kill tcpdump (single only — PID file guarded) -INT for graceful flush, falls through to -KILL after 5s. - Adds journal-spinifex-full.log (no -n truncation) alongside the existing -n 10000 capped one. - Adds bridge fdb show, bridge link, and hairpin_mode/learning/flood for every br-wan brif — needed to confirm/disconfirm the hairpin-loss hypothesis in the plan doc. - Adds 'chmod -R a+r /tmp/spinifex-diag' before tar to ensure pcap (created by sudo tcpdump as root) is readable by tf-user. bash -n verified on all four rendered scripts. YAML parses. Zero new gold-image deps (tcpdump, bridge present in base image). |
||
|
joshsiv-mulga
|
3f4613c28b |
ci(e2e): capture OVN/OVS/journal diagnostics on every run
Adds a Collect Diagnostics + Upload Diagnostics step pair to all three
E2E jobs (single, pseudo-multi, real-multi). Runs before Teardown,
if: always() so green runs produce baselines too. SSHes to test
VM(s), dumps:
- journalctl for spinifex-*, ovn-*, openvswitch-*
- ovs-vsctl/ofctl/appctl: bridge state, br-int + br-ex flows,
dpctl flow + conntrack tables
- ovn-nbctl/sbctl: show, Port_Binding, Chassis, NAT, Logical_Router
- ovn-appctl ct-zone-list (LR -> ct_zone mapping)
- ip addr/route/neigh, iptables-save, nft list, dmesg, virsh list
Tarred, scp'd back, uploaded as e2e-diag-{single,pseudo,multi}-<run>
artifact (14-day retention).
OVS-native ovs-appctl dpctl/dump-conntrack — no new gold-image deps.
Multi-node job writes script to RUNNER_TEMP then ssh-stdin's it
per-node (avoids YAML/heredoc indent traps inside for-loop). bash -n
verified on all three rendered scripts.
Unblocks siv-111 debug loop: bash LB driver does not emit an
analysis bundle on failure, so post-W2 recurrences have been blind.
Workflow-level capture is hypothesis-free.
|
||
|
Julian Sommer
|
97701737cb | feat: describe instance status (#221) | ||
|
Julian Sommer
|
dc3b48f17e | feat: security groups enforcement (#201) | ||
|
Julian Sommer
|
bb08335831 | fix: explicit workflow permission | ||
|
Julian Sommer
|
cbe335e903 | chore: bump single node e2e timeout to 35 mins | ||
|
Julian Sommer
|
84278b794f |
chore: reintroduce dev branch in CI + scripts
- workflows: add 'dev' to push/PR triggers (spinifex.yml, ui.yml, e2e.yml) - workflows: drop "Resolve Mulga Branch" fallback step; mulga is single-branch so checkout@v6 uses the default branch implicitly - dependabot: route gomod/github-actions/npm PRs to dev - diff-coverage.sh: pick origin/dev as base for feat branches, origin/main for dev - iac/proxmox-template: default git_branch to dev for cluster builds Refs docs/development/improvements/reintroduce-dev-branch.md |
||
|
Julian Sommer
|
c7d4040dd3 | feat: nightly e2e (#155) | ||
|
Julian Sommer
|
1ca8621e16 | fix: allow mulga feat branch | ||
|
Julian Sommer
|
ff9f9485d2 | feat: nats TLS (#144) | ||
|
Julian Sommer
|
5b39efb078 | fix: make e2e use env vars | ||
|
Julian Sommer
|
b32111102d | fix: README.md | ||
|
Josh Siviour
|
ab95566795 |
feat: initial ELBv2 implementation (#122)
* feat: add ALB target health reporting via HAProxy stats Agent-side: periodically query HAProxy stats socket and publish target health status to NATS (elbv2.alb.<id>.health topic). Control-plane: healthChecker subscribes to health reports from all ALB agents and updates target health state in the KV store. Wire into ELBv2ServiceImpl lifecycle and DeregisterTargets cleanup. E2E: remove health check skip — targets must now become healthy. * feat: allocate public IP for internet-facing ALB VMs in daemon Add Scheme field to SystemInstanceInput so the daemon knows whether an ALB is internet-facing or internal. For internet-facing, allocate a public IP via externalIPAM and publish vpc.add-nat for the OVN dnat_and_snat rule. Internal ALBs skip public IP assignment entirely. * fix: release auto-assigned public IPs on DeleteNetworkInterface DeleteNetworkInterface previously leaked public IPs and NAT rules. Now checks if the ENI has a public IP, verifies it's not EIP-owned, and releases via externalIPAM + publishes vpc.delete-nat. Daemon injects externalIPAM and EIP KV into VPC service at init time. Tests cover: auto-assigned IP release with NAT event, EIP-owned IP skipped, and no-op when externalIPAM is not configured. * feat: enforce multi-AZ for internet-facing ALBs in pool mode (mulga-770) In CreateLoadBalancer, when DevNetworking is false (pool mode), require 2+ subnets in different AZs for internet-facing scheme. Dev mode and internal scheme skip the validation. * feat: ALB scheme unit tests + DNS internal prefix + public IP in AZ response (mulga-771) Add 9 unit tests covering scheme-dependent ALB behavior: - DNS name uses internal- prefix for internal scheme (AWS-compatible) - Internet-facing ALBs include public IP in AvailabilityZones response - Internal ALBs have no public IP in AZ response - Launcher receives correct scheme, delete cleans up VM + ENIs Also implements the code changes these tests exercise: - AvailZoneInfo.PublicIP field populated from launcher output - lbRecordToSDK populates LoadBalancerAddresses for internet-facing * feat: add custom AMI for ALB to images list * feat: add internet-facing ALB data plane E2E test (mulga-772) New E2E script that validates internet-facing ALBs get public IPs and are reachable from a peer cluster node (external validation). Skips in dev mode. * fix: set ENDPOINT to runner node IP in ALB E2E workflow * fix: correct paths filter to reference alb-e2e.yml * fix: alb ami name and workflow step to import the image * fix: spx path * fix: remove ALB AMI import from workflow and added to bootstrap.sh * fix: use two subnets in different AZs for internet-facing ALB E2E Multi-AZ enforcement (mulga-770) requires internet-facing ALBs to have subnets in 2+ AZs in pool mode. Discover AZs and create a subnet in each. * refactor: remove multi-AZ enforcement for internet-facing ALBs Team decision: ALBs must work with single-AZ and single-node clusters. Removes the multi-AZ validation, pool mode helper, associated tests, and error constant. E2E script reverted to single subnet. * fix: use correct account ID for ALB ENI public IP assignment The ENI is created under the caller's account but LaunchSystemInstance was looking it up with GlobalAccountID (system account), causing the UpdateENIPublicIP call to fail with "ENI not found". Pass the caller's account ID via SystemInstanceInput.AccountID. * fix: add debug logging for ALB public IP assignment failure * fix: expand debug logging to include peer node and more keywords * fix: lazy system AMI discovery for ALB VMs The daemon was caching the system AMI at startup, but the ALB image is imported after services start (bootstrap step 9). Replace SetSystemAMI with SetSystemAMIFunc so the AMI is resolved at request time when CreateLoadBalancer actually needs it. * fix: resolve ALB VM instance type dynamically from resource manager The instance type was hardcoded to t3.nano which may not exist on all clusters. Now the daemon provides a resolver that picks the smallest available instance type from the resource manager. * push interim changes * refactor: rewrite ALB agent from NATS client to HTTP server Replace NATS subscriptions with an HTTP server on :8405 exposing POST /config, GET /ping, and GET /health. The daemon will push config and poll health over HTTP through the VPC network, removing the agent's direct NATS dependency. - agent.go: net/http server replaces nats.Conn/Subscriptions - health.go: remove push-based reportHealth; /health handler calls queryHAProxyStats synchronously on each request - cmd/alb-agent: --nats flag replaced with --listen - tests: rewritten with httptest instead of NATS test server - health_checker_test: inline topic strings (HealthTopic removed) * refactor: replace NATS with HTTP client for daemon-to-ALB agent comms Daemon now pushes config via POST and polls ping via GET to the ALB agent's HTTP server on the VPC IP, eliminating the agent's NATS dependency from the daemon side. - service_impl: pushConfig uses http.Post, waitForAgentReady uses http.Get, both target http://<vpcIP>:8405 - albVMUserData: drop natsURL param (agent no longer needs NATS) - types: add VPCIP field to LoadBalancerRecord - daemon: remove SetNATSURL call and 10.0.2.2 translation block - arn_test: update user data tests for new signature * refactor: rewrite ALB health checker from NATS subscription to HTTP polling Health checker now polls each active ALB's GET /health endpoint on a 10s ticker instead of subscribing to a NATS wildcard topic. Removes the NATS connection dependency from the health checker. - health_checker: replace nc/sub with httpClient + stopCh polling loop - newHealthChecker: drop nc param, add agentURLFn for testability - service_impl: update newHealthChecker call site - tests: rewrite with httptest, test pollAll/pollOne paths * refactor: remove NATS wiring from ALB image and make hostfwd dev-only - alb-setup.sh: remove --nats flag, use --listen=:8405; remove placeholder HAProxy frontend on 8405 (agent owns the port now); update description and comments - service_impl: HostfwdPorts (80/443) now only set when DevNetworking is enabled — production ALBs are reached via VPC IP directly * feat: system credentials file + daemon wiring for ALB agent SigV4 auth Write config/system-credentials.json during admin init/join with the system access key. Daemon reads it at startup, resolves the gateway URL from external IPAM (or 10.0.2.2 in dev mode), and wires both into the ELBv2 service. albVMUserData now injects ALB_GATEWAY_URL, ALB_ACCESS_KEY, and ALB_SECRET_KEY into cloud-init for ALB VMs. * feat: ALB agent gateway actions + daemon handlers (heartbeat, config) Add ALBAgentHeartbeat and GetALBConfig actions to the gateway so the ALB agent can communicate via the AWS gateway instead of direct HTTP. - Add agent_types.go with input/output types for both actions - Add ALBAgentHeartbeat: processes heartbeat, transitions provisioning→active on first contact, updates LastHeartbeat, processes health report - Add GetALBConfig: returns pre-computed HAProxy config text and hash - Replace pushConfig (HTTP POST to agent) with updateStoredConfig (stores config + SHA256 hash on LB record for agent pull) - Remove waitForAgentReady polling goroutine (heartbeat handles transition) - Remove HTTP client from ELBv2ServiceImpl (no longer needed) - Strip health checker HTTP polling (pollLoop/pollAll/pollOne) — health reports now arrive directly via ALBAgentHeartbeat - Add ConfigText, ConfigHash, LastHeartbeat fields to LoadBalancerRecord - Wire NATS subscriptions + daemon handlers for both new actions - Add gateway handler files and elbv2Actions map entries - Comprehensive tests for heartbeat, config, and updateStoredConfig * feat: rewrite ALB agent as AWS SDK HTTP client (Bead 3r) Replace the HTTP server agent with a poll-based client that communicates with the gateway using SigV4-signed requests. The agent now sends ALBAgentHeartbeat every 5s with health reports and fetches config via GetALBConfig when the config hash changes. * feat: add VPC service route for ALB agent gateway access ALB VMs need to reach the gateway for heartbeat/config polling, but VPCs without an IGW have no external routing. Instead of requiring an IGW, the daemon now publishes a vpc.ensure-service-route event before launching ALB VMs. vpcd handles this by creating minimal external connectivity (external switch, localnet port, gateway router port, chassis bindings) if not already present, then adds a /32 static route for the gateway IP. Refactors handleIGWAttach to share the external connectivity setup via ensureExternalConnectivity helper. * Revert "feat: add VPC service route for ALB agent gateway access" This reverts commit |
||
|
Julian Sommer
|
a56a2baf1f | feat: privilege separation (#114) | ||
|
Julian Sommer
|
dda4575888 | fix: use binary install for single node e2e (#111) | ||
|
Julian Sommer
|
f5ec8c3bae | feat: formation server HTTPS (#108) |