mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
d178ef5992eb0bbd4f96b00ffdc587c8956fe518
177 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Julian Sommer
|
e5f528f877 | chore: add rds image to publish | ||
|
Julian Sommer
|
d2d0b357a9 | fix: tofu e2e | ||
|
Julian Sommer
|
2e09461080 | chore: cleanup code comments | ||
|
Julian Sommer
|
5e17169661 | feat(e2e): complete the RDS end-to-end suite (#608) | ||
|
joshsiv-mulga
|
42c7b9b663 | Stop segscan workflow running on push | ||
|
Josh Siviour
|
a2c451128d |
fix: unblock the Ubuntu 26.04 installer (qemu-kvm, sudo-rs) (#665)
setup.sh aborts twice on resolute. qemu-kvm is a virtual package with two providers there, so apt refuses to resolve it; the token was redundant anyway since $QEMU_PACKAGES already installs qemu-system-x86/arm and spinifex execs the qemu-system-* binaries directly. sudo-rs, Ubuntu's default sudo since 25.10, then rejects the sudoers file: it implements no pam_session Defaults setting, and forbids a wildcard inside a command argument. pam_session is now emitted only for classic sudo, and the sysctl grant names a fixed-verb helper instead of a wildcard. The helper is narrower than the rule it replaces: it accepts one interface, a key restricted to rp_filter or accept_local, and a value restricted to 0 or 1, where the old grant permitted any net.ipv4.conf key at any value. Granting sysctl with an unconstrained argument was rejected as root-equivalent. Adds scripts/lint-apt-packages.sh and an Apt Packages workflow so the runtime package list is resolved against each supported suite's real indices in CI. |
||
|
Josh Siviour
|
3e1f9b1198 |
feat: segscan storage oracle for the real-predastore integration tier (#657)
* feat: add segscan storage oracle for the real-predastore integration tier Adds spinifex/testutil/segscanoracle, which copies a fixture node data dir and execs the mulga umbrella repo's scripts/segscan (a separate Go module that cannot be imported) to assert live/dead-tombstoned/dead-orphan byte accounting against real predastore writes. Kept off the default PR path: a new segscanoracle build tag excludes the test from make test-integration, and a separate segscan-oracle.yml workflow (push to main/dev + manual dispatch only, mirroring e2e.yml's non-gating trigger) checks out scripts/segscan rather than adding that checkout cost to spinifex.yml. * test: cover segscanoracle's pure logic to satisfy diff coverage segscanoracle has no build tag and always compiles into the default build, but nothing in that build exercised it, so every new line counted as uncovered against the 60% diff-coverage gate. Adds unit tests for JSON decoding (including the malformed-payload path), the Locate resolution order, and the directory-copy routine, plus a throwaway stand-in module (not the real scripts/segscan) to exercise build+exec+decode end to end. Pulls decodeReport out of Run and resolveSegscanDir out of Locate so the pure logic is testable without a *testing.T or a real binary. |
||
|
Josh Siviour
|
5de598e85a |
ci: trim unused E2E builds, move IMDS to the SDK, derive predastore bind in spx (#656)
* ci: skip building unused E2E suite binaries in e2e.yml's single/multi jobs These two jobs immediately rebuild suite-aware binaries after Build Install Artifacts and never consume the tar's _bin contents, so the cross-compiled suite build was pure waste. build-install-artifacts.sh gains an optional --skip-e2e-build flag (positional contract unchanged); e2e-nightly.yml and the other callers keep building the full set. * refactor: replace hand-rolled IMDS client with AWS SDK v2 internal/imdscreds, cmd/ecs-agent's metadata fetch and its credentials provider used a hand-written IMDSv2 token/GET flow with manual RFC3339 expiry parsing. A malformed Expiration silently fell back to "never expires" instead of failing. Replace it with feature/ec2/imds, credentials/ec2rolecreds and aws.CredentialsCache, following the internal/gwsign/signer.go precedent. An unparseable Expiration is now a hard error from Fetch, covered by TestFetch_BadExpirationErrors. Output shapes to callers are unchanged, and IMDS base URLs ending in /latest keep working. * refactor: derive predastore bind config from spinifex.toml in spx itself predastore-start.sh awk-parsed spinifex.toml for the bind host/port and node_id, exported them as env vars, then exec'd spx. Move that derivation into predastoreStartCmd via derivePredastoreBind, which reads the raw (non-DIAL-normalized) nodes.<node>.predastore section, defaulting node_id to -1 (co-located) when the key is absent, same as the script. An explicit --host/--port/--node-id flag or SPINIFEX_PREDASTORE_* env var still overrides the derived value. Delete the now-unused script and point the systemd unit's ExecStart directly at spx service predastore start, matching the spinifex-viperblock.service precedent. |
||
|
Josh Siviour
|
9c9bb9430c |
test: gate image shell scripts in CI and pin the spot vCPU quota ordering (#641)
* test: gate scripts/images/ shell suites + shellcheck in CI The eks-node/rds-postgres *_test.sh suites and shellcheck only ever ran by hand: spinifex.yml filters pushes on Go-only paths and no job invoked them, so a broken helper could land unnoticed. Add a `make test-images` target and a path-filtered image-scripts.yml workflow so both run automatically on scripts/images/** and images/** changes, kept separate from preflight so image-asset churn doesn't run on every Go contributor's commit. Also clears the shellcheck findings that would have started this gate red (SC1090 on a runtime-resolved source path, SC2154 x2 on eval-assigned variables shellcheck can't trace). * test: pin vCPU quota gate ordering in RequestSpotInstances Nothing asserted that RequestSpotInstances runs the vCPU quota gate before dispatching capacity or launching a VM, so a regression that dropped or reordered EnforceLaunch would pass silently. Add an ordering test using a real quota.Service (embedded JetStream KV) and real NATS responders standing in for the capacity and launch daemons, recording a global call sequence and asserting the gate's index precedes both. Verified by removing the gate: the first quota-KV read then comes from the post-launch charge instead, landing after capacity dispatch and launch, and the ordering assertion fails as expected. |
||
|
Josh Siviour
|
044d7af85e |
fix(ui): pin router-plugin version and commit deterministic route tree (#635)
TanStack router-plugin <1.168.21 sorted routes by coercing route nodes to strings, so output order followed filesystem-walk order and flipped depending on which generator version a contributor's node_modules resolved. Pin the version so it cannot drift, commit the sorted output from the fixed generator, and fail CI if a build produces a different route tree than the one committed. |
||
|
Josh Siviour
|
0cdf591c7c |
ci(nightly): move ubuntu cells to the 26.04 gold image (#611)
Cells 2, 3, 9 and 10 ran on spinifex-image-builder-ubuntu-24.04.qcow2, built 2026-04-20 — five weeks before openvswitch-ipsec and strongswan-charon were added to the image builder. Those cells therefore had no OVN IPsec support and ran intra-AZ Geneve in plaintext. Every other workflow is already on 26.04. The host_image_override guards now match ubuntu-* rather than a pinned version. An exact match is what let this drift go unnoticed: bumping the matrix release silently stops overriding the image and drops the cell onto the debian gold. |
||
|
Julian Sommer
|
a55544b7d4 | chore: remove integration from e2e | ||
|
joshsiv-mulga
|
d6b356b658 | Stop GPU E2E on push and merge | ||
|
joshsiv-mulga
|
e0259b7638 | update preflight | ||
|
Josh Siviour
|
88b57b1805 |
chore: give the integration tier its own CI job and drop it from preflight (#577)
The tier ran inside the Build and Test job, serialised ahead of test-cover, and again in preflight. Neither placement bought signal that a parallel job does not: it provisions nothing and finishes in seconds, so the only effect was wall time on the two paths developers wait for most. It stays PR-blocking as its own job, which is what distinguishes it from e2e.yml's front gate — that one is push-triggered on a self-hosted runner and so never gates a PR before merge. |
||
|
Josh Siviour
|
68abc0bfb3 |
fix(ci): scope e2e telemetry per cell/job and define suite topology eligibility once (#573)
* feat(ci): extend per-cell telemetry scoping to every e2e workflow ci.cell only reached e2e-nightly and e2e-suites, so a run of e2e, e2e-ddil or gpu-e2e still landed in the sink as one undifferentiated bucket. Stamp it on the remaining workflows: single/multi on e2e's two provisioning jobs, ddil and gpu on their single-job workflows. The best-effort numeric job id lookup moves out of e2e-nightly into a shared composite action, so the four other provisioning jobs get ci.job_id without copying the gh/jq incantation. The lookup now matches on env.JOB_NAME rather than interpolating the name into the jq filter, which broke on names carrying quotes or non-ASCII separators, and no-ops when telemetry is disabled. gpu-e2e gets ci.cell but deliberately no job id lookup: nothing in that job bakes the attrs into a daemon, so the API call would be paid for an env var no exporter reads. * fix(ci): define e2e suite topology eligibility once Three workflows carried three different answers to which suites a topology can run. e2e-suites and e2e-nightly ran lb on their single-node leg, where every internet-facing subtest skips for want of a peer, so a fraction of the suite reported as a green lb run. e2e-suites' multi leg ran multinode alone, dropping the lb and cert coverage that only a multi-node env can complete — so full lb coverage never ran there at all. .github/scripts/suite-sets.sh is now the single definition, sourced by all three. e2e.yml's two allow-list regexes derive from it, e2e-suites derives both matrices from it (multi_provision gains the suite-matrix step single_provision already had), and e2e-nightly's deliberately narrower per-cell lists are declared there too, guarded by an assert that fails the cell if a subset ever names a suite its topology cannot fully exercise. The nightly stays narrow on purpose: its cells have a 35-minute budget and exist to prove each install/network/host-OS permutation boots and serves. Suite breadth is the PR gate's job and already runs on every push. That is now stated in one place rather than implied by a third divergent list. |
||
|
Josh Siviour
|
cae76ba1b6 |
test: complete the e2e live-vs-mock rebalance (epic mulga-8oyg3) (#566)
* test(volume): assert chunk GC stays wired on in CreateVolume GCEnabled was zero-valued at every viperblock construction site for a long time and no test noticed. It is not part of viperblock.VBState, so it never round-trips through a saved config.json and cannot be observed by reading volume state back. Extract the viperblock.VB assembly out of CreateVolume into buildVBConfig so the GC wiring is assertable directly against the returned config, without a live viperblock or S3 backend. The config is returned by pointer because viperblock.VB carries atomic.Uint64 fields that vet's copylocks check rightly refuses to see copied. * test(e2e): snapshot backend byte accounting before volume purge DeleteVolume purges a volume's entire backend prefix, including any chunks a leak left behind, before anything measures it. That is the same masking that hid the storage-growth leak in production: a passing E2E cannot fail on backend growth when the evidence is deleted first. Snapshot the volume's backend chunk footprint in EnsureVolume's cleanup closure, immediately before DeleteVolume, so every volume the shared teardown path purges is measured rather than silently discarded. The snapshot is always taken and logged; the hard ratio check additionally runs for callers that registered a guest-written-bytes baseline, since most callers never attach or write to the volume they create and so have no baseline to compare against. The tolerance is the RS(2,1) erasure floor widened for per-chunk header and tail-chunk overhead. It is sized to distinguish unbounded growth from a footprint settling at the floor, not to catch bounded residual garbage. The check is non-fatal inside the cleanup callback: t.Fatalf's runtime.Goexit would abort the DeleteVolume call queued behind it and turn a reported leak into a real one. * test(e2e): add memory-constrained concurrent-write durability regression The existing durability tests write one 4MiB sentinel, sequentially, on a comfortably sized guest, which is the exact inverse of the known-open data-loss bug: acked writes read back zero-filled after a cold reread when a guest is memory-starved during concurrent writeback. No happy-path test can reach it, because the fault lives in the intersection of memory pressure, write concurrency and cache-cold reread. Add a suite that encodes those three conditions together: a guest discovered at or below the working set, N parallel large writers launched as background jobs in a single command so they genuinely overlap, swap disabled, then sync and drop_caches before rereading. Assertions are per stream rather than aggregate so a failure names the losing writer instead of leaving whoever reads it to re-derive which of N writers lost data from one combined digest. GuestExec gains a timeout-parameterised variant. Its fixed two-minute default is sized for sub-second payloads, and a multi-gigabyte write to a network-backed volume would exceed it, so the suite would have failed by timeout while looking like it had caught the corruption. The budget scales with the working set, and a deadline is reported distinctly from a data mismatch so the two red states are never confused. The suite is dispatchable but deliberately absent from the default SUITES: the bug it reproduces is still open, so it is expected to fail and must not sit in a merge-gating path. It is registered in the workflow allow-list so it stays discoverable rather than being buried behind a skip. Also record on the live IAM instance-profile test why its assertions were not moved down to the integration tier, so the trace is not repeated. * test(e2e): drop backend byte accounting, assert a node-wide delta instead The accounting helper listed the internal predastore bucket to attribute bytes per volume. That listing is refused unconditionally: predastore authorises bucket access by same-account ownership, and the bucket belongs to the service account, so a test caller gets 403 no matter what it holds. Granting the test a service credential would invert the trust boundary to buy a measurement, which is the wrong trade. Replace it with what the S3 surface actually permits: sample node-wide allocated bytes either side of the write and bound the delta against the bytes the guest wrote. Coarser, but it measures something real rather than depending on a call that cannot succeed. The tolerance moves into the suite that uses it. It encodes RS(2,1)'s 1.5x expansion floor plus headroom; a cluster running the recommended RS(3,2) has a 1.667x floor and only ~12% of margin left, so the constant needs revisiting if the encoding changes. * revert: drop memdurability e2e suite The suite asserted the guest read back what it wrote after a memory-constrained concurrent write, but the assertion could not fail for the reason it claimed: it compared the guest's own read against the guest's own write with the page cache still warm, so it passed whether or not the data reached the backend. A test that is green on a build with a known-open durability bug is worse than no test — it advertises coverage that does not exist. Removes the suite and its Makefile/workflow wiring, plus DiscoverInstanceTypeAtLeastMemory, which had no other caller. GuestExecTimeout stays: GuestExec now delegates to it, and the old two-minute hard cap it replaced was a latent trap for any long guest command. * test(e2e): add partial-block concurrent-write durability suite Replaces the removed memdurability suite, which passed on a build that still carried the bug it claimed to test. The engine reaches its read-modify-write path only for a partial block; a fully covered block takes a full-overwrite branch that reads nothing and can lose nothing. The old workload wrote through ext4 with 1 MiB buffers, which the filesystem coalesced into aligned full-block requests: block-layer tracing recorded zero unaligned and zero sub-4096 requests, so the path under test never ran, and writers 4 MiB apart could not collide on a block even in principle. This suite writes the shape the defect needs. Two writers issue concurrent O_DIRECT writes into disjoint halves of the same 4096-byte blocks on a raw, unformatted device, stamping every half with the round number, then drop the page cache and reread with O_DIRECT asserting every half carries the final round. One half reading back at an earlier round is the signature of the lost update. The old suite also could not fail even had it reached the path: its write phase ended in 'wait; sync; sha256sum' and reported sha256sum's exit status, while bare wait returns 0 unconditionally, so writer errors were invisible and a backend ENOSPC presented as a pass. Here each writer's exit status and exact byte count are asserted before any durability claim is made. Working set is 8 MiB rather than 2048 MiB, and the volume is deleted on teardown; the predecessor left roughly 3 GiB of backend per run and twice drove a node under the nearfull watermark. Dispatchable but not in the default suite list until a green run and a demonstrated fail-first against a pre-fix engine exist. * test(e2e): let the fixture own the working-set volume delete The suite deleted the volume in its own cleanup and the fixture deleted it again on close, so every run ended with an InvalidVolume.NotFound from the second attempt. Detach stays — DeleteVolume against a still attached volume is what leaks volumes on teardown — and the delete is left to the fixture so the volume is released exactly once. * test(e2e): gate merges on the partial-block durability suite Verified on one node with the same test binary and workload, varying only the engine: 1758 of 8192 halves reverted a generation against the pre-fix build, 0 of 8192 against the fixed one. That is the fail-first evidence the suite needed before it could be trusted to gate anything. Adding it to the default suite list rather than leaving it dispatchable is the point. Its predecessor sat outside every path that runs, which is how it survived long enough to be discovered green on a build that still lost data. * test(integration): cover the ECR data plane in the front gate The live ecr/ suite drove docker, crane and skopeo against a real registry, yet none of it boots a guest: it needs a registry, an object store and an HTTP client. The in-process gateway already serves the OCI Distribution v2 surface on an ephemeral port with a memory-backed object store, so these run here in 0.18s instead of behind full provisioning. Covers the three claims the live tests made beyond authorization, which is already down here: a push/pull round trip asserting blobs and manifests read back byte-identical, refusal to auto-create a repository on push, and retagging an existing manifest without disturbing the original tag. Content is compared rather than status codes alone. A registry that stores a blob under the wrong digest, truncates it, or serves a different one answers 200 to every request in the sequence. The capacity/load leg deliberately stays live: it exists to put real parallel pressure on a real predastore, and a memory-backed object store cannot make that claim. * fix(e2e): make volume teardown release the volume before deleting it Fixture cleanups drain LIFO, so a suite that creates its instance before its volume tears the volume down first, while it is still attached. DeleteVolume answered VolumeInUse, nothing retried, and the volume was stranded on the node. Reordering the chain cannot fix this — registration order has no way to express the instance/volume dependency. Instead teardown is now self-sufficient for any creation order: it force-detaches, waits for the volume to reach available, and only then deletes. Instance teardown waits for terminated for the same reason, since TerminateInstances returns as soon as the request is accepted. Two things kept the leak invisible. A failing cleanup was logged and forgotten, so a run stayed green with resources left behind; failures are now reported as leaks, failing the bound test or surfacing through Close() for TestMain to fail the run. And a panic in one cleanup aborted the whole chain, stranding every resource behind it; each callback is now isolated. RegisterVolumeTeardown and the fixture had grown separate teardown paths, only one of which detached. Both now share it. |
||
|
Josh Siviour
|
2e7136c893 |
feat(ci): stamp per-cell scoping labels on e2e telemetry (#564)
The nightly matrix runs every cell under a single ci.run_id, and each cell names its hosts node1/node2/node3, so telemetry from all cells collapses into one undifferentiated bucket in the ES sink. Root-causing a single failed cell means aggregating across every other cell in the run. Stamp ci.cell and ci.matrix alongside the existing ci.run_id. Workflow-level env cannot reference the matrix context, so the attrs are overridden in each matrixed job's own env block, leaving the workflow-level value as the non-matrix fallback. The numeric GitHub job id is not exposed in any context (github.job is the YAML key, not the id), so resolve it best-effort via the jobs API and append it if found. The lookup never fails the job. e2e-suites can only distinguish single from multi: daemon telemetry is baked into the VM at provision time, before the suite matrix exists. Baremetal shipped no telemetry at all — neither the endpoint nor the attrs crossed the workflow_call boundary. Pass them through as an explicit input so the cell reaches the sink for the first time. |
||
|
Josh Siviour
|
48749576c3 |
test: rebalance E2E into in-process integration tier + shared-guest live scenarios (#556)
* test: add in-process integration test tier Roughly half of the behavioural tests under tests/e2e/ assert control-plane API shapes, authz and error codes that need no live environment, yet they can only run against a tofu-provisioned environment on a scarce self-hosted runner. This adds the tier they can move down to. The real gateway router runs in-process: SetupRoutes() is a pure function of GatewayConfig, so it serves from an httptest server over embedded NATS JetStream with real IAM/STS services and genuine SigV4 verification. Only the daemon side of the wire is faked -- the ec2.*/ebs.* NATS subjects a live node would answer are stubbed per test. Nothing is provisioned and no fixed ports are bound, so the tier runs concurrently on a shared build host and is safe as a CI front gate ahead of the live suites. It is build-tagged and kept out of preflight, which stays unit-only. * test(integration): add IAM/STS client accessors and port the zero-stub authz tests Adds IAMClient/STSClient plus ClientsWithCreds/ClientsWithSessionCreds so a test can act as a principal other than the seeded root user, and stubs the node-discovery subject in StartGateway so call paths that trigger discovery resolve immediately instead of riding out the fan-out timeout. Ports the four enforcement tests that guard on ec2:DescribeRegions, a gateway-local action, so they need no daemon-side stub at all. * test(integration): port remaining IAM/STS authz tests from e2e Ports TestIAMUserCRUD, TestIAMAccessKeyLifecycle, TestIAMUserAuthentication, TestIAMPolicyCRUD, TestIAMPolicyAttachmentEnforcement, TestIAMPolicyLifecycle, TestIAMGroupsLifecycle, TestIAMRolesAndProfiles and TestSTSAssumeRoleAndGetCallerIdentity from tests/e2e/iam/ into the in-process integration tier. Each test now self-bootstraps its own users/policies/roles instead of relying on shared suite-level state, since every integration test gets its own gateway and embedded NATS. TestIAMRolesAndProfiles stubs ec2.IamProfileAssociation.describe because DeleteInstanceProfile always counts live associations over NATS regardless of which guard actually fires. TestIAMCleanup is not ported: it only swept a shared long-lived E2E environment, and every ordering/dependency assertion it made is already covered by TestIAMPolicyLifecycle, TestIAMGroupsLifecycle and TestIAMRolesAndProfiles's own delete-conflict guards. * test(e2e): remove IAM/STS authz tests migrated to the integration tier All 13 signed-request authz tests plus TestIAMCleanup now have faithful, mutation-proven equivalents in tests/integration/ — the gateway dispatches IAM/STS actions directly with no daemon hop, so the in-process tier is a faithful and far cheaper substitute. The iam/ e2e suite now runs only TestIAMInstanceProfileAssociation and TestIMDS. Drop the now fully-dead iam_helpers_test.go alongside the six deleted test files, and remove harness.IAMGroupARN/IAMDeleteGroupBestEffort, which had no remaining caller anywhere in the repo once iam_groups_test.go was gone. * feat(integration): daemon-lite harness for key/tags/routetable/vpc Adds DaemonLite, an in-process stand-in for a live spinifex daemon that subscribes the real key, tags, route table, and VPC/subnet/SG/IGW service implementations directly to their NATS subjects, backed by memory and embedded-JetStream stores. Ports TestKeyPairs, TestTagManagement, TestRouteTableValidation, and TestReplaceRouteConvergence from tests/e2e/single so the integration tier drives genuine daemon-side business logic instead of StubSubject canned replies. vpcd (OVN) is out of scope, so its "vpc.create-sg"/"vpc.delete-sg" acknowledgements are stubbed with a fixed success reply — the SG record itself is still written by the real service impl before that event fires, so the stub only unblocks the synchronous round-trip rather than replacing tested behavior. * ci(e2e): gate live E2E jobs behind the in-process integration tier Runs the in-process integration suite first, on the ci-single runner host rather than inside a provisioned VM, so a broken control plane is caught before either live job pays the Spinifex install. Both e2e_single and e2e_multi now require it, and e2e_status treats a skipped integration job as a failure so the gating status cannot go green on a gate that never ran. Drops the shadow-mode suite selector: targeted per-diff suite selection is not the direction being pursued, and leaving a non-functional shadow job in the gating workflow only invited it to be mistaken for a live gate. * test(e2e): remove single/ tests superseded by daemon-lite ports TestKeyPairs, TestTagManagement, TestRouteTableValidation, and TestReplaceRouteConvergence now have proven-equivalent ports in tests/integration/ (mutation testing confirmed parity). Delete the e2e originals and their runX helpers, along with keypair_test.go's and tags_test.go's file-local helpers left with no caller. generateImportPubKey is shared by other single/ tests so it moves to helpers_test.go instead of being deleted. Slims the manifest-lint baseline for the removed create-call entries. * ci: run the lb e2e suite on multinode only Both e2e jobs filter SUITES by exclusion, and neither excluded lb, so the suite ran twice per e2e run: 354s on single and 654s on multi. Balancing across instances on one node is not a deployment mode we support, and ELBv2 proper requires two AZs, so the single-node run was redundant. Drops ~354s from the single-node job, which is the gate critical path. * test(integration): port 3 of 7 ECR tests to the in-process tier Wires the gateway's ECR control plane (repository/policy/lifecycle metadata over an in-process daemon-lite NATS subscriber, blob and manifest bytes via a memory object store, JWT issuer/verifier keyed off the existing IAM master key) into the integration harness, and ports TestECRControlPlane, TestECRGetLoginPassword, and TestECRIAMAuthorization_AssumedRolePullOnly from tests/e2e/ecr — each proven equivalent by mutation (evaluatePrincipalPolicy forced to allow, and the repo-describe path forced to not-found) before its E2E original was removed. The other 4 authorization tests (PullOnly, ReadOnly, DetachPolicyDeniesImmediately, DeactivatedKeyReturns401) stay on the E2E tier: AttachUserPolicy lacks the AWS-managed-ARN existence-check bypass that AttachRolePolicy/AttachGroupPolicy already have, so attaching AmazonEC2ContainerRegistryPullOnly/ReadOnly straight to a user fails NoSuchEntity in both tiers alike. Left untouched pending a fix to that shared IAM code. * fix(iam): accept AWS-managed policy ARNs on AttachUserPolicy AttachUserPolicy called getPolicyByARN unconditionally, so attaching an AWS-managed ARN to a user returned NoSuchEntity. AttachRolePolicy and AttachGroupPolicy already skip that lookup for AWS-managed ARNs and store them opaquely; the user path was missed when that guard was introduced. ListAttachedUserPolicies had the mirror of the same gap: it resolved every ARN through getPolicyByARN and silently dropped the ones it could not find, so a managed ARN would disappear from the listing rather than round-trip. Evaluation already handled these correctly via resolveAttachedPolicy, so only the write and list paths needed the guard. * test(integration): port remaining 4 ECR authorization tests Ports TestECRIAMAuthorization_PullOnly, _ReadOnly, _DetachPolicyDeniesImmediately, and _DeactivatedKeyReturns401 from tests/e2e/ecr now that AttachUserPolicy accepts AWS-managed policy ARNs. Each was proven equivalent by mutation before its E2E original was removed: PullOnly/ReadOnly/DetachPolicyDeniesImmediately fail when evaluatePrincipalPolicy is forced to always allow, and DeactivatedKeyReturns401 fails when the access-key-active check in resolveECRLongLivedPrincipal is bypassed. All mutants were reverted and a clean run confirmed before committing. tests/e2e/ecr/iam_authorization_test.go is now fully removed — every test it held has an in-process equivalent, and its helpers had no other callers in the package. * ci: select e2e suites by allow-list instead of deny-list Both jobs filtered SUITES by exclusion, so any suite named in neither deny-list ran in both jobs. That is how lb came to run twice per e2e run, and it makes adding a suite silently fan it out everywhere. Each job now names the suites it is eligible to run. Selection is still driven by SUITES, so this is behaviour-preserving for the current default: single/iam/cert on the single-node job, multinode/lb/cert on the multi-node job, exactly as before. eks, ecs and storagegrowth are listed as eligible on the single-node job but are not yet in the default SUITES. * test(e2e): delete seven tests with no coverage TestFinalClusterStats and TestInstanceLaunch assert nothing beyond what other callers of their shared helpers already imply. The multinode baseline, volume, guest-SSH, and preflight tests duplicate single-node coverage or exercise per-guest behaviour that isn't cluster-specific. TestMultinodeInstanceDistribution can't fail: three independent RunInstances calls with no placement group make a strict spread assertion flaky and a loose one meaningless. Preflight checks (/dev/kvm writability, peer SSH reachability) move into requireMultiNodeFixture so the package fixture singleton still fails fast with a clear message instead of disappearing along with the dedicated test. The remaining callers of needInstanceTrio (CrossNodeOps, CrossNodeGateway, NodeFailure, VolumeDurability) still trigger the shared trio launch on their own, so removing the distribution and volume-lifecycle tests doesn't leave it unbuilt. Also slims the manifest-lint baseline entry for the deleted multinode volume test. * fix(e2e): require full 3-node spread in TestMultinodeSpread SpreadPlacement passed at unique>=3, warned via t.Logf at unique==2, and only failed below that. A 33% spread failure (two instances colocated) passed with a log line. The 243.6s NAT-egress scenario that follows only proves what it claims to if the private trio actually landed on three distinct chassis, so that premise needs to be a hard assertion, not a warning. * refactor(e2e): drop the unused spread-assertion harness helpers AssertSpreadAcrossNodes and CountInstancesPerNode lost their only caller when the instance-distribution test was removed. Beyond being dead, the assertion helper documented minNodes=1 as an acceptable logging-only mode inherited from the bash driver, which is the pattern that let a placement test pass while every instance landed on one node. InstanceHostingNode and its ps helpers stay; five multinode tests use them. * fix(ec2): stop leaked goroutines on prepareEFIVolume error paths newViperblock starts the chunk uploader and WAL syncer unconditionally, but prepareEFIVolume only released them on the happy path via Close(). Six error returns between acquisition and Close left both goroutines running. Add a defer immediately after acquisition, matching the sibling prepareRootVolume; Close() remains the durability boundary. * test(predastore): add real predastore fixture for integration tests Extract predastore_integration_test.go's inline daemon startup into a reusable testutil/predastore package so tests/integration can point DaemonLite-style wiring at a real S3-compatible backend instead of an unreachable host. Kept as its own leaf package rather than folding into the shared testutil package: it transitively imports predastore's quicclient, whose package-level connection pool starts a permanent goroutine at init that would trip goleak checks in any of the ~80 other packages that import testutil for unrelated NATS helpers. * test(integration): migrate CreateVolume to assert real predastore persistence Add StartVolumeDaemonLite, an opt-in DaemonLite variant wired to the real predastore fixture, and migrate TestCreateVolume_PersistsToRealPredastore off memory-backed DaemonLite so CreateVolume/DescribeVolumes exercise viperblock's real S3 backend end to end instead of stopping at input validation. * test(integration): add real storage-state oracle for CreateVolume DescribeVolumes only proves the handler's own read path agrees with its own write path, both via the same deserialization; it cannot catch a bug in a field it never re-derives (e.g. a unit-conversion error in the bytes handed to viperblock, since VolumeMetadata.SizeGiB is copied from the request directly). vbscan reads a volume's persisted config.json and chunk objects straight over S3, independent of any spinifex handler code, so TestCreateVolume_PersistsToRealPredastore can assert on what viperblock actually wrote rather than only on what the handler claims. vbscan is a local reimplementation of scripts/vbrefscan's read-only scan rather than an import: spinifex cannot depend on mulga, the module that contains it. It only reimplements the config/chunk-listing half of vbrefscan — the checkpoint-based reference-integrity half needs viperblock.ParseBlockCheckpointBytes, which does not exist in the viperblock version spinifex currently depends on (v1.12.1) and is also moot for this volume: CreateVolume never calls SaveLiveCheckpoint, so a freshly created volume has no checkpoint to scan regardless of version. * refactor(test): move the predastore daemon fixture out of the coverage scope test-cover runs go test ./spinifex/..., so anything under spinifex/ is compiled and measured. The daemon fixture cannot be exercised by unit tests, so it reported a coverage gap no unit test could close. The e2e harness already avoids this by living under tests/; the fixture now does the same. vbscan stays put, being ordinary library code with its own unit tests. * test(integration): migrate RunInstances multi-count expansion off live guests RunInstances with MinCount/MaxCount=2 expands into a single per-node targeted launch (distributeInstances in gateway/ec2/instance/placement.go) before the daemon-facing NATS hop, so the real gateway router plus a stubbed daemon response is sufficient to prove it — no throwaway VM is needed to exercise the expansion logic itself. The live single-node E2E test's WaitForInstanceState/TerminateInstances sequence was teardown hygiene for the real guests it booted, not an assertion of the expansion behavior, so it is dropped rather than emulated. The migrated test additionally asserts the per-node launch request itself carries MinCount=MaxCount=2, proving the count actually propagates rather than the launch succeeding by coincidence or via the zero-daemon degenerate path in placement.go. * docs: drop a doc reference to the deleted live test * test(integration): migrate launch-template RunInstances expansion off live guests RunInstances --launch-template resolves ImageId/InstanceType from the stored template entirely gateway-side (expandLaunchTemplate, before validation/placement/the daemon-facing NATS hop), so proving it forwards the resolved fields never needed a real guest boot. Ports the assertion to tests/integration via a new opt-in StartLaunchTemplateDaemonLite (real LaunchTemplateServiceImpl over embedded JetStream) plus a direct subscription on the per-node launch subject to capture what the gateway actually dispatched, rather than only the client-visible response. The live test keeps its CRUD/versioning lifecycle (create/modify/delete version, tag-filtered describe) — none of it touches a guest — and drops only the RunInstances-from-template + wait-for-running + terminate tail, whose only job was waiting out a real boot to read back fields already proven resolved. * test(integration): migrate KV-only account-scoping checks off live tenants Ports the KeyPair/VPC-Subnet/IGW-EIGW/AccountSettings cross-account isolation assertions from tests/e2e/single/account_test.go to the in-process integration tier. Each of these resources resolves ownership from a plain account-scoped KV key (utils.AccountKey(accountID, id), or an account-keyed settings record) with no vm.Manager or real guest involved, so DaemonLite now also wires the real EgressOnlyIGWServiceImpl and AccountSettingsServiceImpl alongside the existing VPC/IGW/Key services. Two independently minted tenant accounts (via IAMService.CreateAccount/CreateUser/CreateAccessKey/CreatePolicy/ AttachUserPolicy, in-process — the same calls runAccountCreate makes without the CLI/stdout round-trip) replace the live tier's carousel so isolation is checked between real, separately owned accounts. Left live: instance scoping (Step2) and volume scoping (Step3) — checkInstanceOwnership, the actual cross-account gate for stop/start/ reboot/terminate/attach-volume, runs daemon-side only after a real vm.Manager lookup, so it cannot be exercised without a live guest. Snapshot scoping (Step5) needs a real backing volume for CreateSnapshot (viperblock, out of DaemonLite's scope), so it stays live entirely rather than split off only its two KV-only assertions. Global-resources (Step9) carries no account branching in the handlers at all, so asserting its equality across two accounts wouldn't exercise anything a fake node responder didn't already trivially satisfy — not migrated. * test(integration): migrate launch-template CRUD lifecycle off live guests TestLaunchTemplates booted zero guests: its RunInstances coverage had already moved to TestRunInstances_LaunchTemplateExpansion, leaving only template CRUD/versioning assertions occupying a full live-cluster slot. Port the remaining create/describe/version/modify/delete lifecycle, including the SourceVersion merge and a nested timestamp round-trip, into TestLaunchTemplates_CRUDLifecycle against the in-process integration gateway. * refactor(tests): move spot instance lifecycle coverage to integration tier RequestSpotInstances is a mock over on-demand RunInstances (no bidding, interruption, or reclamation), so booting real VMs in the e2e tier proved nothing spot-specific. Migrate the SIR CRUD lifecycle (request, describe, cancel, close-on-terminate, insufficient-capacity error) to a NATS-level integration test that exercises the real gateway orchestration and daemon service impl without a live guest. Real-VM-boot assertions are already covered by TestRunInstances_MultiCountExpansion and the e2e baseline tests. * fix(tests): share one embedded NATS server across the integration package ClientTokenStore in spinifex/gateway/ec2/instance is a process-wide sync.Once singleton bound to whichever *nats.Conn first initialises it. Every integration test booted its own embedded NATS server, so the first RunInstances-calling test captured the singleton, and any later test inherited a KV handle bound to that server's now-closed connection. TestMain now boots one embedded NATS+JetStream server for the whole package; every StartGateway call connects into its own dynamically provisioned NATS account (accountAuthenticator in shared_nats.go), giving each test a private JetStream namespace despite the shared server, and none of the per-test connections are closed until every test has finished - whichever one the singleton latches onto stays live for the run. Also wire the suite into the required PR gate: it was previously only invoked from e2e.yml's push-triggered (main/dev), self-hosted "Integration (front gate)" job, so a regression here never blocked a PR. The whole package runs in single-digit seconds, so it now runs inline in `make preflight` and as its own step in spinifex.yml's build_and_test job. * test(integration): migrate serial-console access checks off live guests Enable/Disable/GetSerialConsoleAccessEnabled is pure gateway+account-settings logic with no guest involved, so it moves to the in-process integration tier against the real AccountSettingsServiceImpl wired into DaemonLite. * test(integration): migrate DescribeImages round-trip off live guests runImage's AMI describe round-trip needs no real cluster: it registers an AMI against a hand-seeded snapshot via the real ImageServiceImpl, backed by a new StartImageDaemonLite helper wiring RegisterImage/DescribeImages onto an in-memory object store. * test(integration): migrate predastore object lifecycle off live guests runPredastoreObjectLifecycle only needs a real predastore daemon, not a tofu-provisioned cluster: it now runs against tests/fixtures/predastore's in-process fixture, covering the same create/put/get/delete/list contract. * test(integration): migrate ListFoundationModels catalog check off live cluster ListFoundationModels is pure gateway-side static catalog logic with no NATS/daemon hop, so it moves to the integration tier via a new BedrockClient harness accessor; this tier configures no credential resolver, so the Anthropic entry's absence is asserted explicitly rather than left optional. * test(integration): cover RunInstances client-token idempotency Proves replaying the same ClientToken through the real gateway returns the original reservation without a second daemon dispatch, and a different token launches independently. * test(integration): cover RunInstances launch-template expansion Proves version selection (omitted Version resolves to the template default, explicit Version selects that version) and field precedence (explicit RunInstances parameters override the template, unset fields inherit it) through the real gateway. * test(integration): cover RunInstances malformed-input rejection Proves a malformed ImageId and a malformed targeted-launch CapacityReservationId are rejected before any NATS dispatch, with no daemon stub registered so a regression that let either through would fail fast instead of hanging. * test(integration): cover RunInstances iam:PassRole enforcement Proves a caller granted ec2:RunInstances but not iam:PassRole on an instance profile's role is denied, and the identical session succeeds once PassRole is granted for that exact role ARN. * test(integration): cover RunInstances placement-group strategy routing Adds a placement-group daemon-lite wiring a real PlacementGroupServiceImpl over NATS, then proves spread strategy assigns exactly one instance per node while cluster strategy pins the whole launch to the single highest-capacity node. * test(integration): cover RunInstances capacity distribution across nodes Proves the default (non-placement-group) path packs a multi-instance launch across nodes proportionally to spare capacity rather than exhausting one node first. * test(e2e): merge SameSGComms and SGToSGDatapath around one client/target pair Both booted their own instance pair to check SG-to-SG datapath enforcement and same-default-SG east-west connectivity. Merging them around one shared client/target pair cuts four boots to two, and converting the inline harness.Step sequence into named t.Run stages (PortGroupMembership, AllowedTraffic, DeniedTraffic, RevokeAndVerifyBlocked, ReauthorizeAndVerifyRestored, SameSGComms) gives each its own pass/fail attribution instead of one opaque test result. SameSGComms now runs last and joins the default SG onto each ENI via ModifyNetworkInterfaceAttribute only after every SG-to-SG allow/deny assertion has already recorded its own result, since the default SG's self-reference rule admits all traffic between co-members and would otherwise defeat the denied-traffic check. * test(e2e): merge default-SG reachability, guest DNS, and egress enforcement DefaultSGReachabilityBaseline, GuestDNSResolution, and SecurityGroupEgress each booted their own instance to check one facet of "a guest is reachable only when policy says so": ingress deny/authorize, resolver + WAN egress, and egress ACL enforcement. Merging them around one shared guest cuts three boots to one, with named t.Run stages (Deny, Authorize, Resolver, WANEgress, RevokeEgress/RestoreEgress) giving each its own attribution. The egress revoke/restore round-trip now mutates this scenario's own dedicated SG instead of the shared default SG the rest of the suite depends on, isolating its egress-rule mutation from every other test. * fix(ci): single-quote e2e test_run regex across the ssh boundary * fix(e2e): anchor SG egress-enforcement probe to a WAN IP, not the gateway * test(e2e): merge single-node guest churn tests into one durability scenario Combines TestSSHProbe, TestENIHotplug, TestENIHotplugReconcile, TestVolumeDurability, TestModifyInstanceAttribute, and TestRebootInstance into TestGuestChurnDurability, a single shared-guest scenario that re-verifies a data-integrity sentinel sha256 after every perturbation stage (ENI hotplug, daemon restart, ENI detach, instance-type resize, stop/start, reboot). This proves the sentinel survives the full churn sequence rather than just one isolated stop/start. * fix(vpc): clear subnet drop gate when installing egress reroute A subnet with an attached IGW but no 0.0.0.0/0 route carries a priority-1100 drop gate on the VPC router. When a route later lands (NATGW reroute at 900, IGW reroute at 1000) the reroute is installed but the drop gate was never removed, so the higher-priority gate wins and black-holes the subnet's egress in lr_in_policy before it reaches routing or SNAT. This left NAT-gateway private-subnet egress with 100% packet loss: the SNAT stage saw zero packets and no conntrack was committed. Remove the subnet's drop gate for the same prefix once a reroute is installed. Idempotent and prefix-scoped, so a specific-prefix reroute is a no-op and leaves the default gate intact. * test(e2e): merge VPC subnet, NAT gateway, and EIP single-node tests Merges the former TestVPCSubnetE2E, TestNewVPCEgressBaseline, TestNATGateway, and TestInstanceEIP into one staged scenario, TestVPCEgressPaths, around one scenario-owned VPC and its two guests, cutting five guest boots to three. The public guest serves PublicSubnetEgress, then the NAT Gateway bastion hop, then (last, since association rewrites its public IP) the EIP flip. The private guest serves the NAT Gateway path, launched directly into the same VPC's already-existing private subnet instead of a dedicated one. RouteBeforeSubnet keeps its own second VPC and instance since it exercises the implicit main-route-table code path, a different one than the explicit route table this scenario's own VPC uses for its public subnet — collapsing the two would drop coverage of whichever path got merged away. The NAT Gateway up/down flip now repeats for an env-tunable round count (SPINIFEX_VPCEGRESS_NAT_ROUNDS, default 1), matching every other merged single-node scenario. * test: TEMPORARY mutation-check — revert |
||
|
dependabot[bot]
|
5dc945a6df |
chore: bump the github-actions group with 2 updates (#552)
Bumps the github-actions group with 2 updates: [actions/setup-go](https://github.com/actions/setup-go) and [actions/setup-node](https://github.com/actions/setup-node). Updates `actions/setup-go` from 6 to 7 - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](https://github.com/actions/setup-go/compare/v6...v7) Updates `actions/setup-node` from 6 to 7 - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](https://github.com/actions/setup-node/compare/v6...v7) --- updated-dependencies: - dependency-name: actions/setup-go dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: actions/setup-node dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Josh Siviour
|
ffc34b735b |
ci: emit per-run ES summary doc from e2e workflow (#551)
Add e2e_es_summary, a self-hosted job that aggregates suite-status.txt from both e2e legs and POSTs one doc per run to the Elastic sink, keyed by labels.ci_run_id, so Kibana can answer "did run X pass, and where are its artifacts" without hand-aggregating traces-apm*/logs-apm.error*. Runs on ci-single (LAN route to the sink) since e2e_status stays on GitHub-hosted ubuntu-26.04 for the PR status-report check. State is computed with the same single_ok/multi_ok logic as e2e_status so the two can never disagree. |
||
|
Josh Siviour
|
8205f88b04 |
ci: build sibling repos from a co-named branch, falling back to dev (#550)
The Build/Lint/Race jobs checked out viperblock and predastore at a hardcoded ref: dev and overlaid them with a go workspace, so the go.mod pins were ignored and a spinifex change depending on an unmerged sibling change could never compile in CI. Resolve each sibling to a branch of the same name when it exists on that repo, else dev, mirroring the e2e path in scripts/tofu-cluster/build-install-artifacts.sh so unit CI and e2e CI agree. |
||
|
joshsiv-mulga
|
b147e2b869 |
ci: make Setup Go Workspace idempotent on ci-single runner
The publish-eks-ami matrix fans out four legs on the single persistent ci-single runner, all reusing one workspace root. go work init is create-only and checkout never scrubs the root go.work, so legs after the first fail with 'go.work already exists'. Remove any stale go.work before init so each leg re-inits cleanly. |
||
|
joshsiv-mulga
|
2b44de27bc |
ci: publish eks-node-gpu + ecs-agent-gpu AMIs
Expand the publish workflow to build the two Ubuntu NVIDIA GPU variants alongside the Alpine eks-node/ecs-agent AMIs. The image build+publish scripts are already manifest-driven and Ubuntu-aware; only the workflow lacked the GPU legs. Refactor the single-job body into a matrix over the four image manifests, resolved by a new 'variant' input (all|cpu|gpu) so the slow 16G GPU legs can be skipped when only the CPU AMIs need republishing. GPU images do not fit a RAM-backed /tmp, so set SYSTEM_IMAGE_BUILD_DIR onto the runner's on-disk _temp; legs run serially (max-parallel 1) on the single ci runner. |
||
|
Josh Siviour
|
1e1d2f3c6d |
ci: GPU e2e workflow on ci-gpu self-hosted runner (#518)
* ci: GPU e2e workflow on ci-gpu self-hosted runner Runs tests/e2e/gpu (EC2/ECS/EKS GPU passthrough) against the persistent, hand-bootstrapped wattle shared host, driven over SSH by a banksia runner labelled ci-gpu (matches the ci-single/ci-multi convention) — no runner installed on the shared box. - runs-on [self-hosted, ci-gpu]; concurrency ci-gpu, cancel-in-progress false (single A1000 = single-flight, queue not stomp) - workflow_dispatch + GPU-path filter (not every push) - manual-work mutex gate (~/.ci-gpu-paused / CI_GPU_PAUSED) so hands-on GPU testing blocks CI - if: always() cleanup terminates launched GPU instances; never resets the box (no setup-ovn.sh/dev-reset/cluster-reset) - drives the control plane with spx + the already-configured aws spinifex profile directly (no wrapper scripts, nothing to install on wattle) * ci(gpu-e2e): advisory AMI preflight, fail only if zero GPU AMIs The suite is self-guarding — each test skips when its AMI is absent. The hard-fail on the base ubuntu GPU AMI blocked the whole run even when the ECS GPU AMI (the only one imported) could execute. Inventory the AMIs and fail only when none are present. * ci(gpu-e2e): self-provision base GPU AMI, hard-require it in preflight The GPU E2E workflow was green-because-skipped: with no GPU AMIs imported the self-guarding suite skipped every test, passing as a false green. Make the raw-EC2 GPU passthrough test the always-on baseline. - Add an import-if-absent setup step that pulls the prebuilt base GPU image from the catalog by name (ubuntu-26.04-nvidia-gpu-x86_64), naming the AMI exactly what tests/e2e/gpu filters for. Idempotent; caches on wattle's persistent loop-fs. Runs spx under sg spinifex (local storage write). - Preflight now hard-requires the base AMI (fail, not skip) so a silent import failure can't pass as green; ECS/EKS node AMIs stay opt-in warn. - Reconcile the local build path: ubuntu-gpu-nvidia.conf AMI_NAME and launch-gpu.sh's nvidia default now use the same canonical name, so a locally built+imported image also satisfies the suite. * ci(gpu-e2e): pass --ami-name on base GPU import so the AMI name matches spx admin images import --name selects the catalog entry but names the resulting AMI generically (ami-<distro>-<version>-<arch>), so the import landed as ami-ubuntu-26.04-x86_64 and the tests/e2e/gpu name filter never matched. Pass --ami-name ubuntu-26.04-nvidia-gpu-x86_64 explicitly (same flag build-system-image.sh uses) and poll briefly for availability. * fix(network,vm): reconcile auto-assigned public-IP NAT + scale VFIO QMP timeout Auto-assigned public IPs were never carried through the network reconciler. applyEIPs ran AddEIP + ensureGuestPortDatapath for user EIPs only; auto-assigned IPs (intent.Ports[].PublicIP) got egress exemption but no NAT re-assert and no gatewayLRP->guest convergence. Their dnat_and_snat was created once at RunInstances and never touched, so a vpcd/OVN rebuild dropped it and the guest-port hop never converged: inbound DNATs at the gateway but never reaches the guest (TCP/SSH dark) while ICMP is answered by the gateway LR (ping OK). floatingIPSpecs now folds every auto-assigned/ELB public IP into the same AddEIP + guest-port convergence path as user EIPs (deduped by private IP; a user EIP wins), making the datapath self-healing across reconcile ticks and deploys. qmpGreetingTimeout for VFIO guests now scales with guest RAM (base 30s + 8s/GiB, floored at 180s, capped at 600s) instead of a flat 180s, so a large guest is not SIGKILLed mid-pin while the whole guest RAM is IOMMU-locked before QMP answers. * fix(vm): scale QMP handshake command deadline for VFIO launches The greeting timeout scaling alone did not fix the VFIO launch SIGKILL: a GPU guest sends its QMP greeting before the RAM pin completes, then cannot service qmp_capabilities until the pin finishes. sendQMPCommand hardcoded a 30s response deadline, so the handshake decode timed out (~43s for a 16 GiB g5.xlarge) and the launch cleanup SIGKILLed a QEMU that would have come up. Parameterize the command deadline (sendQMPCommandWithTimeout); the handshake now uses the RAM-scaled qmpGreetingTimeout (base 30s + 8s/GiB, floor 180s, cap 600s) for qmp_capabilities. Other callers keep the 30s default. * test(gpu-e2e): share one GPU instance across passthrough subtests to avoid mid-suite IP recycle |
||
|
Josh Siviour
|
e78d0dbac9 |
fix(ci): upload full tofu apply/destroy log as an artifact after teardown (#489)
Every tofu-based e2e workflow set TOFU_LOG to a /tmp path outside ARTIFACT_DIR. ci-fmt.sh's quiet_run helper appends all tofu init/apply/ destroy output to that file, but the Teardown step (which runs the destroy) always ran AFTER the existing upload-artifact step, so the full log — including destroy-time failures — was silently discarded on every run. Add an if: always() upload-artifact step after Teardown in each tofu-based job (e2e.yml, e2e-suites.yml, e2e-nightly.yml, e2e-ddil.yml) that uploads $TOFU_LOG directly. Since quiet_run appends both apply and destroy output to the same file, one upload after Teardown now captures everything, recoverable even when destroy itself fails. Shipping this log to Elasticsearch/OTLP is deferred to mulga-5n7fe, which will consume this artifact. |
||
|
Julian Sommer
|
47604f43dc | chore: go 1.26.5 | ||
|
Josh Siviour
|
a7b689fec3 |
feat(otel): full-stack telemetry — tracing, metrics, logs + per-VM guest metrics (#479)
* feat(otelsetup): OTel init, resource attrs, trace-stamping slog handler Env-gated on OTEL_EXPORTER_OTLP_ENDPOINT: with no endpoint only the W3C propagator is installed and globals stay no-op, so instrumented binaries ship everywhere at zero cost. Resource carries service/host/mulga.env/ mulga.source plus OTEL_RESOURCE_ATTRIBUTES (ci.run_id in CI). * feat(otelsetup): wire telemetry init into all spx services Every spx service start command installs the trace-stamping JSON slog default and the OTel providers via initTelemetry, flushed on shutdown. Gateway's own slog setup now routes through otelsetup so its level handling keeps trace correlation. * feat(gateway): HTTP tracing middleware on awsgw and daemon cluster API Server span per request with W3C traceparent extraction; awsgw spans are renamed to the resolved SigV4 service.Action post-auth with account and region attributes. Request log lines carry trace_id via context. * feat(nats): propagate W3C trace context over command subjects NATSRequestCtx injects traceparent beside X-Account-ID and opens a client span per hop; ServeNATSRequest opens a consumer span joining the producer trace, with a Ctx variant for handlers that thread context. RunInstances is plumbed end-to-end (gateway dispatch through placement fan-out to the daemon handler) and viperblockd ebs.mount/unmount join inbound traces. * feat(systemd): source optional /etc/spinifex/telemetry.env in service units Lets operators enable OTLP export (OTEL_EXPORTER_OTLP_ENDPOINT, MULGA_ENV, MULGA_SOURCE) by dropping a single env file, with no unit edits. File is optional; absent means telemetry stays disabled. * feat(ci): export OTel telemetry from E2E envs to the central Elastic sink All E2E workflows set SPINIFEX_OTLP_ENDPOINT (repo variable override, default wattle) plus ci.run_id/ci.workflow resource attributes, so bootstrap-install.sh writes telemetry.env on every node. Teardown runs flush-telemetry.sh before the final tofu destroy so buffered spans and metrics reach the sink. * feat(otelsetup): request metrics and Go runtime metrics via OTLP mulga.requests counter + mulga.request.duration histogram recorded by the same middleware that opens spans: awsgw carries the resolved SigV4 action, the daemon cluster API uses the chi route pattern, and NATS handlers record per-topic at the subscribe chokepoints (node names stripped to keep cardinality flat). Go runtime metrics start inside the same env gate. * feat(daemon): deep spans for QMP commands, VM launch phases, EBS ops Single chokepoint span per QMP command in sendQMPCommand covering all callsites; vm.launch phase children (mount_volumes, start_qemu, qmp_connect, persist_state); ec2.cmd consumer span with traceparent extraction plus start/stop/reboot/terminate child ops; ebs.mount/unmount spans in viperblockd with volume attributes. * feat(telemetry): thread request context through NATS, object store and VM paths Migrate utils.NATSRequest call sites to NATSRequestCtx (241 sites; W3C traceparent now flows gateway -> daemon on every request), add GatherCtx and migrate all Gather call sites, inject traceparent on manual EKS publishes. ObjectStore interface is ctx-first with aws-sdk WithContext calls and an otelhttp transport, so predastore spans join the caller trace. NATS consumer spans record error status when the response envelope carries an error. vm.Manager and the ec2 handler service interfaces take ctx-first so vm.launch and QMP spans join the request trace; detached background work uses explicit context.Background(). Hot-path slog calls use *Context variants for trace_id binding. * feat(telemetry): suppress health-check, heartbeat and unrooted trace noise Health probes record metrics only (no span per poll); the QMP heartbeat poller marks its context so query-status ticks stop rooting a trace each. rootSampler honors MULGA_ROOT_TRACE_RATIO (ParentBased ratio, default 1.0) and the predastore unit sets 0.05: guest block I/O roots are sampled while request-linked spans are always kept. * feat(otel): export deployment.environment resource attribute Elastic APM maps it to service.environment, enabling the native environment selector across the APM UI (Services, Traces, Dependencies). * fix(telemetry): stop objectstore rooting method-only HTTP transactions Background S3 calls (state sync, cleanup loops) carried no caller span, so otelhttp rooted a trace per request and APM surfaced thousands of parentless 'HTTP GET/PUT/DELETE' transactions with expected-404 probes counted as failures. Emit the client span only when the request context already carries one, matching the viperblock backend. Bump viperblock/predastore to branch heads so CI builds also pick up the equivalent span filter and root sampler. * feat(telemetry): per-VM guest metrics via dedicated QMP socket + qmp-collector Goanna Phase 1 producer, operator-consumed first. Each VM gets a second -qmp chardev (qmp-telemetry-<id>.sock) plus a sidecar discovery file carrying instance/account/vcpus/taps/monitoring-tier; the new qmp-collector service polls query-cpus-fast (vCPU thread /proc jiffies), query-blockstats, query-balloon and host tap sysfs counters, and publishes CloudWatch-mappable goanna_ec2_* series to NATS metrics.ec2.<instance-id>. A queue-group bridge in the same process forwards the series to the node-local OTLP receiver so they reach the operator sink until Goanna consumes the subjects. The collector never touches the manager's control QMP socket; telemetry failures never block a launch. * fix(qmp-collector): dial NATS via DialTarget, drop ProtectProc, fix meta race The collector connected to the raw NATS host (0.0.0.0), failing TLS verification against a cert valid for 127.0.0.1; route through admin.DialTarget like the other daemons. ProtectProc=invisible hid the QEMU vCPU threads (dumpable=0) that the collector must read from /proc/<tid>/stat, so every tick failed and re-primed without ever publishing. Drop it to default. Surface collect failures at Warn so silent metric loss is visible, drop per-tick publish to Debug, and read InstanceID under the lock in run() to fix a race with updateMeta on ENI hot-plug. * chore(deps): re-pin predastore + viperblock to dev-landed commits |
||
|
joshsiv-mulga
|
53a87fad8c |
ci(e2e): add nightly cell 19 for routed-NAT single-node suite
Cell 19 bootstraps with --external-mode=nat plus the CI static pool as the public wan pool and runs the natuplink suite, covering the Tier-1 transit datapath and Tier-2 EIP ingress. The run step exits 0 when natuplink.test is absent from the tests tar so scheduled runs against branches predating the suite stay green. |
||
|
joshsiv-mulga
|
ee5c467a40 |
ci: publish ecs-agent AMI alongside eks-node
Add a second build+publish step to the EKS AMI workflow so the same manual run also builds the ecs-agent system AMI and uploads its qcow2 + .sha256 sidecar to R2. Reuses the shared checkout, Go, and tooling setup; renames the workflow to cover both node images. |
||
|
Julian Sommer
|
485cca941f | feat: IAM inline role policies (Put/Get/DeleteRolePolicy + real ListRolePolicies) (#429) | ||
|
Julian Sommer
|
446f93ae8c | chore: rename ubuntu-latest to ubuntu-26.04 | ||
|
Julian Sommer
|
8a94886b13 | feat: cloud-init EC2 datasource (#411) | ||
|
dependabot[bot]
|
79e881f1c8 | chore: bump actions/checkout from 6 to 7 in the github-actions group (#405) | ||
|
Julian Sommer
|
51706938e8 | chore: use ubuntu-26.04 runners | ||
|
Josh Siviour
|
41dc53b7f8 |
fix(ci): prevent stale suite-status from false-failing e2e re-runs (#387)
The per-run ARTIFACT_DIR lives in /tmp, outside GITHUB_WORKSPACE, so a re-run on the same self-hosted runner reused a prior attempt's append-only suite-status.txt. The 'Fail job if any suite failed' gate's -s check then failed the job even when the surviving attempt passed with rc=0. Wipe ARTIFACT_DIR in Clean Workspace and gate solely on the per-attempt steps.e2e.outputs.rc (fail-closed: an empty rc still fails). |
||
|
Julian Sommer
|
28c6e8ccff | chore: bump CI time | ||
|
Julian Sommer
|
eced3f733a | feat: split IAM e2e (#372) | ||
|
Julian Sommer
|
058da3eff4 | fix(viperblockd): seal volume block-map to predastore on detach + durability e2es (#371) | ||
|
Josh Siviour
|
17c83cb3ad |
fix: build microVM rootfs via buildx to stop overlay2 export leak (#358)
docker export of a container leaks its overlay mount on Docker's overlay2 driver — the mount survives docker rm, so the layer dirs are never reaped except by a daemon restart. The lb-agent microVM build ran docker export on every e2e job across ~9 runners, accumulating ~69G of orphaned overlay2 layers on the CI host (disk hit 94%). Build the rootfs with `docker buildx --output type=local` from a small generated Dockerfile instead, which writes the image filesystem to a directory and releases all mounts — nothing leaks. Keep the podman path (its store is unaffected). Tighten the e2e cleanup image prune to dangling-only so the alpine base stays cached instead of re-pulling. |
||
|
Julian Sommer
|
c6c26c04e2 | chore: trim workflow comments | ||
|
Julian Sommer
|
2df35d3822 | chore: remove reboot e2e | ||
|
Josh Siviour
|
d076a434d4 |
Feat/e2e targeted suites (#352)
* test(e2e): add targeted suite selector + shadow-mode CI step
select-suites maps a diff to the e2e suites it must run: changed files →
services (manifest path/additional_paths) → consumer closure over
depends_on vs publishes/subscribes → suites whose covers intersect.
Special cases force-all (infra/harness/workflow, ≥3 services, submodule
bumps) or a lone suite dir; empty diffs fall back to default suites. An
AST scan of each suite's harness.Ensure* call graph derives per-test
service coverage and emits an optional -test.run subset (RUN_PATTERN).
Wired into e2e.yml as a non-gating 'Select Suites (shadow)' job that
logs the computed selection to the run summary so predicted vs actual
can be compared before the selector drives SUITES.
* ci(e2e): add dispatch-only per-suite job blocks grouped by env
Restructure the looped e2e flow into env-grouped provision -> matrix
suites -> teardown blocks so each suite surfaces as its own node in the
Actions graph instead of a single green tick:
single_provision -> single_suite [single,cert,lb,eks] -> single_teardown
multi_provision -> multi_suite [multinode] -> multi_teardown
Provision resolves the VM/cluster and ships test binaries, emitting WAN
IP + ssh key path as job outputs; each matrix leg runs one suite over
SSH and uploads its own junit/analysis artifacts; teardown (if:always)
reuses the persisted tofu workspace on the same runner to destroy.
workflow_dispatch only -- runs alongside the gating e2e.yml for
validation and posts no commit status.
* test(e2e): add manifest drift lint (fixture-create + NATS subject)
Bead 5 of the targeted-suite-selection plan. Two ratcheted drift guards in
tests/e2e/manifest-lint, wired into make preflight and the spinifex.yml PR job:
- Fixture lint: AST-flags direct resource-create AWS calls
(c.EC2.RunInstances, c.ELBv2.CreateLoadBalancer, ...) in suite test files,
steering them through harness.Ensure* fixtures. ~104 pre-existing call sites
are captured in a checked-in baseline; the lint fails only on NEW creates.
Intentional create-path tests opt out with an inline // e2e:allow-create.
- Subject lint: extracts NATS subject literals from the natsSub{} registry and
literal Subscribe/Publish/Request args, cross-referenced wildcard-aware
against service-interfaces.yaml. Code subjects missing from the manifest fail
(ratcheted); manifest subjects with no static reference warn only (many are
produced dynamically). Fixed real drift surfaced on first run: declared the
five spinifex.cluster.shutdown.* phase subjects under spinifex_daemon.
manifest-check is now CI-enforced alongside manifest-lint (previously
preflight/local-only).
* feat: run-scoped e2e teardown sweep
Tag every Ensure* fixture resource with e2e:run=<run-id> when
SPINIFEX_E2E_RUN_ID is set, and add a teardown.test suite that sweeps
everything carrying that tag at end of run — reclaiming resources leaked
by a crashed suite, an interrupted run, or a persistent cluster.
- harness/sweep.go: tagRunResources (EC2 CreateTags) + tagRunELB (ELBv2
AddTags) + SweepRunResources in dependency order (LB, instances, NAT,
snapshots, volumes, images, SGs, subnets, key pairs). Blank run id
no-ops both tagging and sweeping, so the sweep can never touch an
untagged or foreign resource.
- harness/fixtures.go: stamp the run tag on each created resource.
- tests/e2e/teardown: thin suite that runs the sweep; skips without a
cluster or run id, logs the report, never fails the run.
- e2e-suites.yml: build + ship teardown.test, pass the run id into suite
legs, sweep on the VM in both teardown jobs before tofu destroy.
|
||
|
Josh Siviour
|
90369371ee |
Feat/nightly e2e fix (#330)
* ci(e2e-nightly): run Go harness suites instead of bash run-*.sh Swap nightly cells 1-3 (single), 8-10 (real-multi) and 18 (reboot) to the compiled <suite>.test binaries, mirroring e2e.yml: SPINIFEX_E2E=1 with SPINIFEX_MODE single|multinode, per-suite logs to ARTIFACT_DIR, then go-junit-report + e2e-analyze + upload-artifact. Adds a spinifex checkout for the composite actions and pulls /run/spinifex/console-*.log so LB microVM (lb-agent) state is captured on failure. Cells 17 (tofu) and 20 (baremetal) stay on bash — no Go suite exists. * ci(e2e-nightly): cell-17 RCA analytics + cell-20 cosmetic alignment run-tofu-examples-e2e.sh now emits go test -v markers per workbook so the cell-17 step's tee'd test-tofu.log flows through the same go-junit-report -> e2e-analyze -> upload-artifact chain as the Go suites, producing junit-tofu.xml and an analysis bundle for RCA. Align e2e-baremetal.yml (cell 20) ARTIFACT_DIR, analyze title, and artifact name with the matrix cells so CI-RCA discovers its bundle uniformly. * ci(e2e-nightly): wire host journal + VM consoles into RCA bundles Collect now emits the canonical spinifex-journal.log (concat of all nodes) that the e2e-analyze bundler slices into each failure's journal.log, so elbv2-service/vpcd/daemon lines in the failure window travel with the bundle (previously the journal uploaded as spinifex-journal-node*.log, a name the bundler never reads). Rename the console capture to vm-console-node* and note it also carries guest cloud-init/sshd consoles, covering the cell-3 internal-LB, cell-18 reboot-datapath, and cell-9/10 guest-ssh blind spots. * ci(e2e): self-heal runner /dev + --one-file-system on workspace clean Clean Workspace now detaches any leaked iso-builder rootfs mount, recreates /dev/null if a prior build clobbered it, and uses rm --one-file-system so the clean can't descend into a still-mounted devtmpfs and delete host device nodes. Guards both the matrix cells and the bare-metal job against the host-wide /dev/null wipe that fails go build with 'open /dev/null: no such file or directory'. |
||
|
Josh Siviour
|
7bcd365549 |
ci(eks-ami): build on self-hosted ci-single runner (#326)
* ci(eks-ami): build on self-hosted ci-single-1 runner GitHub-hosted ubuntu-24.04 cannot run the libguestfs appliance network: passt fails (PID file denied under its AppArmor confine / privilege drop) and the SLIRP fallback has no outbound, so virt-customize --network dies. The self-hosted ci-single-1 runner has /dev/kvm and real networking, so the existing libguestfs build works unmodified. * fix(build-system-image): add --no-cache, force fresh build on publish The <10min cached-raw skip avoids duplicate work for concurrent e2e jobs, but on the persistent publish runner it could republish a stale image from a prior or concurrent run. Add --no-cache to bypass the skip and have publish-system-image.sh pass it on every build. * ci(eks-ami): target ci-single label (no ci-single-1 label exists) The self-hosted runners are named ci-single/-2/-3 but all share the single label 'ci-single'; runs-on matches labels, not names, so ci-single-1 never schedules. * ci(eks-ami): disable Go cache on self-hosted runner setup-go's cache-save (Post Setup Go) re-uploaded the Go cache to the Actions backend on every run, taking ~2.5min vs the 36s build. The self-hosted runner already keeps the cache on local disk, so cache:false. |
||
|
Josh Siviour
|
4ef19fc488 | feat(ui): EKS console (#310) | ||
|
Josh Siviour
|
5ac8dc3fb9 |
fix(network): round-robin external pool cursor — stop instant EIP reuse (recycled EIP unreachable, EKS CI) (#320)
* Revert "fix(network): prime host neigh with gateway port MAC on centralised EIP attach"
This reverts commit
|
||
|
Josh Siviour
|
467e4de715 |
feat(eks): publish eks-node AMI to R2 + spx catalogue entry (#313)
* feat(eks): publish eks-node AMI to R2 + spx catalogue entry Add a spinifex-eks-node catalogue entry so operators install the EKS node image with a single command: spx admin images import --name spinifex-eks-node The entry pulls the Alpine+K3s image from iso.mulgadc.com, verifies its sha256, and stamps spinifex:managed-by=eks — the tag the control-plane and node-group launchers resolve the boot AMI by, so no extra config is needed. Add scripts/publish-system-image.sh (+ make publish-eks-node-image) to convert the built raw to a compressed qcow2, emit a checksum sidecar, and upload both to the R2 bucket backing iso.mulgadc.com/system-ami via the S3 API. * ci(eks): add manual workflow to build + publish eks-node AMI to R2 Build the eks-node AMI on the runner (libguestfs) and publish via publish-system-image.sh using the existing CF_R2_* secrets. Manual workflow_dispatch with a dry-run input. Fix the publish script's default bucket to spinifex-iso (the bucket iso.mulgadc.com is served from). |
||
|
Josh Siviour
|
a1f3fcc531 |
ci(e2e): exclude eks suite from multinode job (#312)
EKS create-cluster fails deterministically on the multinode job with ServiceUnavailableException: the EKS KV is a clustered (Raft) JetStream stream and create-cluster needs an elected leader, but multinode JetStream clustering isn't reliably forming yet (see mulga-omq/z2z/5yg). EKS is validated single-node only at this stage. Drop eks from the multinode build+run suite filters (alongside the existing reboot/single exclusions) so the multinode job reflects what's actually supported. Single-node continues to run the full eks suite. |
||
|
Josh Siviour
|
a4817b1afc |
ci: run baremetal E2E nightly + hard-gate releases on it (#306)
Make e2e-baremetal.yml a reusable workflow (workflow_call + a ref input) so the nightly and release pipelines can invoke the real PXE-reimaged hardware suite, which previously only ran on manual dispatch. Nightly: add a baremetal job (conceptual cell 20) that runs every weeknight on the existing cron, or on dispatch when 20 is selected. It tests dev on schedule, matching GIT_BRANCH. Sits outside the tofu matrix (different runner, provisioning, 120m timeout) and shares the e2e-baremetal-cluster concurrency group, so all baremetal runs serialise on the single physical node. Release: a release builds and uploads an ISO to R2 but never booted that ISO/squashfs path on hardware. Add a baremetal job (needs: release) and hard-gate on it: promotion moves to a dedicated promote job needs:[verify, baremetal] (with a matching mark-failed job on failure), and the iso job's upload gate now waits for both verify and baremetal to conclude success before pushing to R2. Bump the iso job timeout to 180m since it now waits on the parallel ~2h baremetal run. |