19 Commits

Author	SHA1	Message	Date
Julian Sommer	5865fc418e	chore(ci): lock workflow action dependencies with actions.lock (#1207)	2026-10-15 13:55:16 +11:00
Julian Sommer	368f4efa40	chore: trim comments from workflows	2026-09-10 11:48:14 +10:00
Julian Sommer	60509e7921	fix: remove currency block on e2e.yml	2026-09-10 11:39:43 +10:00
Ben Duncan	18853cef31	fix: close the four defects keeping the e2e nightly red (#1007) ... * fix: stop the e2e analyser inventing failures and let a detach outlast its seal go-junit-report emits <error message="No test result found"> for a === RUN with no matching result line, and attaches the test's own captured log as the body. The analyser counted those as failures, so the baremetal cell reported 25 failures against a junit file recording 1, with log output reading as assertions. Separate the marker from a real error in both the parse and the roll-up, count the unresolved cases and say so in the summary. The gateway gave a detach 30s while the daemon budgets 180s for the ebs.unmount seal and 180s more for the QMP unplug chain, so any detach whose seal replayed a WAL was reported as ServerInternal while it went on to succeed. Also switch four nightly job gates from always() to !cancelled(): always() overrides cancellation, so a cancelled run kept starting jobs. * fix: serialise the e2e workflows on one group and retry a dropped libvirt e2e.yml had no concurrency block at all, so every push to dev could provision alongside every other push and alongside the nightly. All four workflows that build on these hypervisors now share e2e-hypervisors, queued rather than cancelled. A per-environment group is not expressible: GitHub picks the group before it picks a runner, and the environment is a property of the runner. tofu apply also failed outright on a transient libvirt connect, which fails a 45-minute job in 90 seconds and reports it as a test failure. retry_run retries only when that attempt's own output names a connection failure, so a real failure still fails once. Also collect the unfiltered journal from the matrix cells and the node journal from cell-21: the unit filter drops the kernel and OOM lines that explain why one node behaved unlike its siblings, and a passing run's journal is what makes a failing one readable. * fix: build the baremetal junit from the unfiltered suite log run-baremetal-e2e.sh streams the full suite output to suite-output.log and a grep-filtered summary to its stdout, which the workflow tees into test-baremetal.log. That filter anchors --- PASS at column 0 and Go indents every subtest result by four spaces, so the junit was generated from a log with no subtest results in it: go-junit-report saw each === RUN with no outcome and emitted <error message="No test result found">. 17 of cell-28's 39 testcases and 93 of cell-20's 120 were that. Regenerating cell-28 from suite-output.log gives 39 tests, 0 failures, 0 errors. * fix: converge released EIPs instead of trusting a fire-and-forget teardown A disassociate leaves the ENI running, so the orphan sweep's owning-port test never fires for it. The teardown then rests entirely on vpc.delete-nat, which is core NATS and is lost whenever vpcd is restarting — and the intent KV watch is UpdatesOnly, so the same window hides the write that would have driven a repair pass. The dnat_and_snat row and its host /32 route to a released public address survived until something else happened to touch them, which for a live ENI was nothing at all. Give the sweep the second test it was missing: AddEIP is the only writer of dnat_and_snat, so the floating IPs intent asks for are the complete set of addresses that may hold one, and a row for any other address is stale however alive its port. Ports and external IPs travel together in LiveEIPs, both unioned from the start-of-pass snapshot and the fresh re-read, so a mid-pass launch is still spared. The host unbind now runs even when the row had already gone, which is the state a crash between the OVN delete and the unbind leaves. Run that sweep on the startup pass too. Startup is the pass that follows the blind window, so skipping it deferred repair of exactly the changes that were missed. The port-group and ENI-port sweeps stay gated there for the in-flight create they were written for; the EIP sweep re-reads intent immediately before deciding and abandons the whole sweep if that read fails, which is the same protection by a stronger route. * fix: keep a release verify off the hypervisors an e2e run is using release.yml is the fifth workflow that provisions on the ci-single pool and had no concurrency block at all, so a tag push could land a tofu apply on the same libvirt host a nightly cell was mid-provision on. The group goes on the verify job rather than the workflow: the build jobs touch no hypervisor and should not queue behind a nightly. * fix: stop cell-27 handing a 50-minute suite a 50-minute job The rds suite is budgeted 50m in suite-sets.sh and cell-27 took the matrix default of 50 minutes for the whole job, so provisioning, install, the suite and teardown had to fit inside the suite's allowance alone. It never could: the last measured run spent 6 minutes reaching the suite and was killed 44 minutes into it, mid-TestAutomatedBackups. A job timeout reports "The operation was canceled" and nothing else, which is why this cell has no verdict in any nightly rather than a failure anyone could read. cell-27 gets 75 minutes, and the select job now refuses to start a run whose job budget does not clear its suite budget plus a setup allowance. The inner timeout firing first is the property that matters: go test says where it stopped and what failed, and the job timeout says nothing at all. cell-29 already had the same fix applied by hand; nothing stopped the next suite from repeating it. * fix: converge host EIP plumbing against intent, not against the NAT row The released-EIP sweep landed and did not fire. Its premise was that a stranded dnat_and_snat row is what a stranded host route can be found by, and on the path that actually strands one there is no row left. AddEIP scrubs the predecessor row sharing the private IP before installing the new one, so when reconcile re-added the instance's auto-assigned address it silently deleted the released EIP's row — and applyEIPs runs ahead of the sweep, which then walked a table with nothing stale in it while the /32 route and its proxy-ARP stayed up. Give the sweep the host's own inventory. ListEIPIngress reads the EIPs back from the FORWARD accepts EnsureEIPIngress stamps with its own comment, so a binding is found by what installed it rather than by a row some other path is free to delete. Anything plumbed that intent no longer asks for is torn down, which also covers a crash between the OVN delete and the unbind. A failed inventory fails the pass: an empty list and an unreadable one are indistinguishable, and reading the second as the first is a sweep that reports success having done nothing. Close the window inside the pass too. The predecessor scrub now reads the row it is about to drop and releases that address's host state, so the strand does not exist between applyEIPs and the sweep rather than merely being repaired a step later. * fix: retry the release verify's libvirt calls like every other job does The verify job drives the same libvirt provider on the same hypervisors as the e2e cells, where a dropped connection under concurrent load is the steady state and is retried. Here it was a hard failure that reads as a broken release. It was skipped earlier only because the job checks out mulga and not spinifex, so the shared helpers were not on disk — a sparse checkout of .github/scripts costs nothing and makes apply and destroy behave like their e2e counterparts. * fix: do not report a rebooting DB instance available on the pre-reboot agent's health RegisterDBInstance stamped a fresh LastSeen while leaving Agent.EngineHealth at the value the previous agent process reported. After a reboot the new agent registers before it has probed anything, so the record read as a healthy engine with a beat newer than the transition, and the reconciler moved the instance to available while PostgreSQL was still starting up. A client that connects the moment the status turns green then gets "the database system is starting up". A register now discards the previous process's verdict and records when this agent process started. The restart reconciler requires that registration to be newer than the transition, which also closes the window where a beat lands between the engine being stopped and the VM going down. * test: record the credential kubectl sent when an EKS call comes back Unauthorized The IRSA pod subtest's first kubectl is the gateway-CA configmap, which lands before the pod dumps are registered, so a 401 there left no evidence at all. A verbose read-only probe reports whether an Authorization header was sent, which separates a token the apiserver rejected from one the exec plugin never made. * fix: report a starting engine promptly, and budget CI jobs from the suite The agent waited a full heartbeat interval before its first beat, so a restart could not be observed as finished for up to 30 seconds after the engine was already serving. Beat at startup instead, and report every 5 seconds until the engine has answered once — the control plane is actively waiting on that beat, and the steady-state cadence it hands back is for a database already serving. The rds suite is the one most sensitive to what else the hypervisor is doing: its DB VMs serialise behind a 4 GiB semaphore, so its wall clock is set by how long it waits for the budget. The same pass ran 45m beside four nightly cells and overran 50m beside nine, panicking with two tests still running. Raise its budget to 60m. That budget was then larger than the jobs meant to contain it. A job cancelled on timeout reports only "The operation was canceled" and takes the artifacts with it, so the inner `go test` timeout must always fire first — and it could not for rds on e2e.yml, nor for storagefault or diskperf, which have asked for 90m and 180m inside a 60m job since they were added. Both workflows now resolve their job budget from suite-sets.sh rather than carrying a literal: e2e.yml from the largest requested suite, e2e-suites.yml per matrix leg, so a leg is bounded by its own suite and not by the longest one beside it. The default push set resolves to exactly the timeouts both workflows had before. * fix: check a permutation cell's job budget against the suites it runs The guard only saw cells that name a suite, so the nine that run a topology's nightly subset were never checked against it — the same class of mistake the guard exists to catch, in the majority of the cells. Bound each by the longest suite in its subset: max() and not sum(), because whichever suite hangs must panic on its own timeout first, and a sum would demand a budget the set has never come close to using. Every cell clears it today. * fix: report why a PostgreSQL engine would not start, and stop failing a stop that landed Three defects, all found by the first rds pass that ever ran deep enough to reach them. A wait that saw an instance reach `failed` called t.Fatalf from inside the polling condition, and testify runs that condition on its own goroutine. The Goexit killed the tick and left the collector empty, which reads as the condition being satisfied — so the wait returned successfully on an instance it had just declared broken, logged "reached status available" about one that was failed, and let the test carry on against it. Six conditions across four suites call Fatal or require the same way. EventuallyErr now runs its condition on the test's goroutine, which makes all six behave as written. A DB VM stop has two drivers: the API call and the reconciler resuming one whose caller it assumes has died. When they overlap the loser is told the VM is not in a state to be stopped — which is the state the stop was heading for — and the customer's StopDBInstance returned InvalidDBInstanceState for a stop that had just completed. Both paths now treat a refused command the way they already treat an unanswered one: it says nothing about the VM, so the fleet view settles it. PostgreSQL had no equivalent of the engine log tail MariaDB has carried since it was added, so a postmaster that refuses a setting produced "engine did not respond on /run/postgresql:5432" and nothing else — on a guest with no SSH, whose console records only that pg_ctl could not start the server. The probe now quotes the tail of the log the image already directs the postmaster to, and the reboot and modify paths carry the agent's reason into the failure the way the create path always has. The reason reaches the customer through StatusInfos. The rds suite's budget goes to 90m and cell-27's job to 100m. Its wall clock is what the DB-VM queue costs rather than what the tests do — single tests logged 11m, 22m and 26m waiting for budget — and the pass above still had three of seventeen tests running at 60m. * fix: keep a DB VM recoverable when its agent cannot start, and flush the volume before a snapshot An rds-agent that failed to construct exited, so a transient IMDS credential resolution took the guest down for good: rds-datadir and rds-init cascade on the missing handoff, the engine never starts, and the control plane sees only silence until it declares the instance failed ten minutes later. Construction is retried like every other boot step instead. OpenRC skips start_post when the start itself fails, which is the one case with no diagnostic at all — the console carried nothing but "died". The log mirror moves out of the handoff wait so a failed start uses it too. A snapshot quiesced the engine and nothing else. The engine repairs its own files from the WAL, but postgresql.conf sits beside them and it does not: a restore came up on a torn tail and the postmaster stopped on a syntax error at the unterminated include hook. The quiesce now flushes the whole data mount, and the hook is renamed into place rather than appended, so a boot that finds a half written one replaces it instead of appending after its open quote. * fix: flush the config cloud-init wrote before a reset can take it A DB instance rebooted seconds after it went available came back with no RDS_GATEWAY_URL, so rds-agent could not reach the control plane and the engine never started. cloud-init writes agent.env once, per-instance, and never again; a reboot is a QMP system_reset with no guest shutdown, so the file was still dirty page cache when the machine reset under it and the next boot read nothing. The handoff wait now syncs, which is the point at which the agent has proved it read that file. * fix: stop dropping an EIP teardown published while vpcd is restarting DisassociateAddress publishes vpc.delete-nat fire-and-forget. On run 34046316024 vpcd disconnected from NATS at 03:51:26.73 and its replacement subscribed at 03:51:27.09; the publish landed at 03:51:26.92, in the gap, and went nowhere. ReleaseAddress then returned the address to the pool two minutes later with its /32 host route still delivering to the guest that had held it. vpc.delete-nat now uses request-reply like vpc.add-nat, and retries only on no-responders — which returns immediately, so the cost is paid solely when nothing is listening. A teardown that still fails is logged as an error naming the address rather than lost silently. Neither is fatal to the caller: the reconciler's orphan sweep remains the backstop. vpc.add-nat on the EIP path stays fire-and-forget. The shared helper blocks it on the flows barrier for up to 45s, and associating an address should not inherit that. Two reconcile passes in that run had the address absent from intent and did not sweep its host route, and the journal cannot say why: the sweep logged only when it acted, so a route it kept was indistinguishable from one it never saw. It now logs the bindings it found, kept and wanted. * ci: render one run-level table for the e2e nightly Reading a nightly meant opening nineteen jobs, and the Actions UI does not separate a cell that failed from one cancelled out from under it — a distinction every investigation of this suite has needed, and one the run 34046316024 analysis had to reach for the jobs API to get. A final job, needs: every cell and if: always(), renders cell, topology, result, duration, runner and per-cell test counts into the step summary, with run-level totals, the three slowest cells and the failing test names. Result, duration and runner come from the jobs API so a cell that died before writing junit still gets a row. Counts come from a new junit-only artifact per cell rather than the full cell artifact, so the roll-up does not pull nineteen journals it has no use for. Verified against run 34046316024: 14 success, 2 cancelled, 1 failure, with the two cancellations shown as cancelled rather than folded into failure. * fix: assert EC2's actual EIP semantics, and probe EKS auth on an authenticated path natuplink asserted that an instance's auto-assigned public IP is still plumbed after an EIP associated to the same instance is torn down. EC2 does the opposite: associating an EIP releases the auto-assigned address back into the pool, and it cannot be reused. Our datapath already does that — run 34059879666 shows 192.168.0.72's ingress removed at 07:09:44.000, 36ms before .73's was installed — so the test was asserting behaviour neither AWS nor we have. It now asserts the release at the point it happens, and that no public delivery remains after the teardown. This surfaced only once the teardown itself started working: the run's other assertions passed, including .73's removal, which was the failure before. Our records still disagree with our datapath — the ENI keeps the released address, which is why termination tears the old one down twice and never the current one. That is the remaining half of mulga-7zc9w and is not fixed here; it needs an IPAM release on the associate path. Separately, AuthDiagnostics read /version, which is anonymously readable, so its 200 was consistent with both branches it claimed to distinguish. It now runs auth whoami, reads kube-system, and invokes the credential plugin directly, reporting exit code, stderr and whether a token was produced — never the token, since CI logs outlive it. * test: capture per-guest state when a spread guest never answers SSH TestMultinodeSpread's only failure dump is dumpPlacementNATDiag, which collects OVN, OVS, ARP and route state aimed at the NAT and EIP half of the test. The failure it actually hits is a guest that never accepted SSH — run 34040324968, the guest on node3 at 10.100.2.5 — and for that it recorded no console, no port binding and no tap state. There was no way to tell from the artifact whether the guest booted at all. Now dumps, per guest: the console, the NB logical switch port, the SB port binding, the OVS interface backing its tap, and br-int's port list. Probed on every node rather than the expected one, because a port bound on the wrong chassis is exactly what a hosting-node-only dump cannot show. Covers all three guests, not just the failing one, so a working guest is available as the control. * ci: sample hypervisor load either side of every cell Eight runner names map onto three hypervisors, so cells that never share a runner still share a host: when cell 10's SSH wait expired on run 34040324968, cells 22, 17, 18 and 28 were concurrently live, two of them multi-node. Nothing in the artifacts measured host load, which left co-tenancy as an explanation that could be neither confirmed nor ruled out. Records loadavg, MemAvailable, SwapFree, the cpu line carrying steal, and the QEMU process count before and after each cell, into the cell artifact. * ci: note that main's missing concurrency block is why the nightly runs ungrouped Three refs carry three different groupings: main has no concurrency block at all, dev uses e2e-nightly, this branch uses e2e-hypervisors. Scheduled runs load the workflow from the default branch, so every cron nightly runs ungrouped and cannot queue against anything — which is how a dispatch of 34050630627 ran alongside the scheduled 34046316024 and cost it cells 24 and 27. e2e-hypervisors is the right group and deliberately broader than dev's, since e2e.yml, e2e-suites.yml and e2e-ddil.yml draw on the same runner pools. Nothing on a feature branch can fix this; the block has to reach main.	2026-09-06 16:53:32 -05:00
Josh Siviour	0e08b9d256	feat: ship per-test JUnit results to the Elasticsearch sink (#829) ... * feat: ship per-test JUnit results to the Elasticsearch sink Every e2e workflow already converts its test log to JUnit XML and renders it into the job summary, then discards it with the artifact. mulga-ci-runs records which suite failed; nothing records which test failed, so root-causing a failure means reading raw Actions logs across historical runs. The junit-to-es action bulk-indexes one doc per leaf test case into mulga-ci-test-results: name, subtest, status, duration, failure text, run id, cell, suite, branch and sha. Parent t.Run wrappers are dropped so a failing subtest counts once, matching the job summary renderer. The doc id is deterministic, so a replayed post overwrites rather than double-counting. Called from every workflow that produces JUnit XML, and tolerated on failure: an unreachable sink must not turn a green test run red. * feat: classify unparsed JUnit cases as unknown, not error go-junit-report emits <error message="No test result found"> when a === RUN line has no matching result line, which means the log it was fed was truncated or filtered. Recording that as a failure would put 78 phantom failures per green bare-metal run into the pass rate.	2026-08-18 22:54:54 +10:00
Julian Sommer	2e09461080	chore: cleanup code comments	2026-08-04 14:20:19 +10:00
Josh Siviour	68abc0bfb3	fix(ci): scope e2e telemetry per cell/job and define suite topology eligibility once (#573) ... * feat(ci): extend per-cell telemetry scoping to every e2e workflow ci.cell only reached e2e-nightly and e2e-suites, so a run of e2e, e2e-ddil or gpu-e2e still landed in the sink as one undifferentiated bucket. Stamp it on the remaining workflows: single/multi on e2e's two provisioning jobs, ddil and gpu on their single-job workflows. The best-effort numeric job id lookup moves out of e2e-nightly into a shared composite action, so the four other provisioning jobs get ci.job_id without copying the gh/jq incantation. The lookup now matches on env.JOB_NAME rather than interpolating the name into the jq filter, which broke on names carrying quotes or non-ASCII separators, and no-ops when telemetry is disabled. gpu-e2e gets ci.cell but deliberately no job id lookup: nothing in that job bakes the attrs into a daemon, so the API call would be paid for an env var no exporter reads. * fix(ci): define e2e suite topology eligibility once Three workflows carried three different answers to which suites a topology can run. e2e-suites and e2e-nightly ran lb on their single-node leg, where every internet-facing subtest skips for want of a peer, so a fraction of the suite reported as a green lb run. e2e-suites' multi leg ran multinode alone, dropping the lb and cert coverage that only a multi-node env can complete — so full lb coverage never ran there at all. .github/scripts/suite-sets.sh is now the single definition, sourced by all three. e2e.yml's two allow-list regexes derive from it, e2e-suites derives both matrices from it (multi_provision gains the suite-matrix step single_provision already had), and e2e-nightly's deliberately narrower per-cell lists are declared there too, guarded by an assert that fails the cell if a subset ever names a suite its topology cannot fully exercise. The nightly stays narrow on purpose: its cells have a 35-minute budget and exist to prove each install/network/host-OS permutation boots and serves. Suite breadth is the PR gate's job and already runs on every push. That is now stated in one place rather than implied by a third divergent list.	2026-07-22 17:18:55 +10:00
Josh Siviour	e78d0dbac9	fix(ci): upload full tofu apply/destroy log as an artifact after teardown (#489) ... Every tofu-based e2e workflow set TOFU_LOG to a /tmp path outside ARTIFACT_DIR. ci-fmt.sh's quiet_run helper appends all tofu init/apply/ destroy output to that file, but the Teardown step (which runs the destroy) always ran AFTER the existing upload-artifact step, so the full log — including destroy-time failures — was silently discarded on every run. Add an if: always() upload-artifact step after Teardown in each tofu-based job (e2e.yml, e2e-suites.yml, e2e-nightly.yml, e2e-ddil.yml) that uploads $TOFU_LOG directly. Since quiet_run appends both apply and destroy output to the same file, one upload after Teardown now captures everything, recoverable even when destroy itself fails. Shipping this log to Elasticsearch/OTLP is deferred to mulga-5n7fe, which will consume this artifact.	2026-07-09 22:20:48 +10:00
Josh Siviour	a7b689fec3	feat(otel): full-stack telemetry — tracing, metrics, logs + per-VM guest metrics (#479) ... * feat(otelsetup): OTel init, resource attrs, trace-stamping slog handler Env-gated on OTEL_EXPORTER_OTLP_ENDPOINT: with no endpoint only the W3C propagator is installed and globals stay no-op, so instrumented binaries ship everywhere at zero cost. Resource carries service/host/mulga.env/ mulga.source plus OTEL_RESOURCE_ATTRIBUTES (ci.run_id in CI). * feat(otelsetup): wire telemetry init into all spx services Every spx service start command installs the trace-stamping JSON slog default and the OTel providers via initTelemetry, flushed on shutdown. Gateway's own slog setup now routes through otelsetup so its level handling keeps trace correlation. * feat(gateway): HTTP tracing middleware on awsgw and daemon cluster API Server span per request with W3C traceparent extraction; awsgw spans are renamed to the resolved SigV4 service.Action post-auth with account and region attributes. Request log lines carry trace_id via context. * feat(nats): propagate W3C trace context over command subjects NATSRequestCtx injects traceparent beside X-Account-ID and opens a client span per hop; ServeNATSRequest opens a consumer span joining the producer trace, with a Ctx variant for handlers that thread context. RunInstances is plumbed end-to-end (gateway dispatch through placement fan-out to the daemon handler) and viperblockd ebs.mount/unmount join inbound traces. * feat(systemd): source optional /etc/spinifex/telemetry.env in service units Lets operators enable OTLP export (OTEL_EXPORTER_OTLP_ENDPOINT, MULGA_ENV, MULGA_SOURCE) by dropping a single env file, with no unit edits. File is optional; absent means telemetry stays disabled. * feat(ci): export OTel telemetry from E2E envs to the central Elastic sink All E2E workflows set SPINIFEX_OTLP_ENDPOINT (repo variable override, default wattle) plus ci.run_id/ci.workflow resource attributes, so bootstrap-install.sh writes telemetry.env on every node. Teardown runs flush-telemetry.sh before the final tofu destroy so buffered spans and metrics reach the sink. * feat(otelsetup): request metrics and Go runtime metrics via OTLP mulga.requests counter + mulga.request.duration histogram recorded by the same middleware that opens spans: awsgw carries the resolved SigV4 action, the daemon cluster API uses the chi route pattern, and NATS handlers record per-topic at the subscribe chokepoints (node names stripped to keep cardinality flat). Go runtime metrics start inside the same env gate. * feat(daemon): deep spans for QMP commands, VM launch phases, EBS ops Single chokepoint span per QMP command in sendQMPCommand covering all callsites; vm.launch phase children (mount_volumes, start_qemu, qmp_connect, persist_state); ec2.cmd consumer span with traceparent extraction plus start/stop/reboot/terminate child ops; ebs.mount/unmount spans in viperblockd with volume attributes. * feat(telemetry): thread request context through NATS, object store and VM paths Migrate utils.NATSRequest call sites to NATSRequestCtx (241 sites; W3C traceparent now flows gateway -> daemon on every request), add GatherCtx and migrate all Gather call sites, inject traceparent on manual EKS publishes. ObjectStore interface is ctx-first with aws-sdk WithContext calls and an otelhttp transport, so predastore spans join the caller trace. NATS consumer spans record error status when the response envelope carries an error. vm.Manager and the ec2 handler service interfaces take ctx-first so vm.launch and QMP spans join the request trace; detached background work uses explicit context.Background(). Hot-path slog calls use *Context variants for trace_id binding. * feat(telemetry): suppress health-check, heartbeat and unrooted trace noise Health probes record metrics only (no span per poll); the QMP heartbeat poller marks its context so query-status ticks stop rooting a trace each. rootSampler honors MULGA_ROOT_TRACE_RATIO (ParentBased ratio, default 1.0) and the predastore unit sets 0.05: guest block I/O roots are sampled while request-linked spans are always kept. * feat(otel): export deployment.environment resource attribute Elastic APM maps it to service.environment, enabling the native environment selector across the APM UI (Services, Traces, Dependencies). * fix(telemetry): stop objectstore rooting method-only HTTP transactions Background S3 calls (state sync, cleanup loops) carried no caller span, so otelhttp rooted a trace per request and APM surfaced thousands of parentless 'HTTP GET/PUT/DELETE' transactions with expected-404 probes counted as failures. Emit the client span only when the request context already carries one, matching the viperblock backend. Bump viperblock/predastore to branch heads so CI builds also pick up the equivalent span filter and root sampler. * feat(telemetry): per-VM guest metrics via dedicated QMP socket + qmp-collector Goanna Phase 1 producer, operator-consumed first. Each VM gets a second -qmp chardev (qmp-telemetry-<id>.sock) plus a sidecar discovery file carrying instance/account/vcpus/taps/monitoring-tier; the new qmp-collector service polls query-cpus-fast (vCPU thread /proc jiffies), query-blockstats, query-balloon and host tap sysfs counters, and publishes CloudWatch-mappable goanna_ec2_* series to NATS metrics.ec2.<instance-id>. A queue-group bridge in the same process forwards the series to the node-local OTLP receiver so they reach the operator sink until Goanna consumes the subjects. The collector never touches the manager's control QMP socket; telemetry failures never block a launch. * fix(qmp-collector): dial NATS via DialTarget, drop ProtectProc, fix meta race The collector connected to the raw NATS host (0.0.0.0), failing TLS verification against a cert valid for 127.0.0.1; route through admin.DialTarget like the other daemons. ProtectProc=invisible hid the QEMU vCPU threads (dumpable=0) that the collector must read from /proc/<tid>/stat, so every tick failed and re-primed without ever publishing. Drop it to default. Surface collect failures at Warn so silent metric loss is visible, drop per-tick publish to Debug, and read InstanceID under the lock in run() to fix a race with updateMeta on ENI hot-plug. * chore(deps): re-pin predastore + viperblock to dev-landed commits	2026-07-09 09:43:32 +10:00
dependabot[bot]	79e881f1c8	chore: bump actions/checkout from 6 to 7 in the github-actions group (#405)	2026-06-23 09:11:02 +10:00
Julian Sommer	c6c26c04e2	chore: trim workflow comments	2026-06-12 11:04:46 +10:00
Josh Siviour	050ac8f861	feat(e2e): Go E2E harness + single-node migration to Ensure*/Discover* + reboot suite (#241) ... * feat(e2e): add Go test harness + cert scenario, port run-cert-e2e.sh Lands step 1 of docs/development/improvements/e2e-go-harness.md: shared harness primitives (env, wait, artifacts, cert, tls) plus the TestCertIssuance scenario that supersedes run-cert-e2e.sh. All files gated by //go:build e2e so default builds and `make preflight` are unaffected. Bash script stays in place for one CI cycle before deletion. * feat(e2e): port run-lb-e2e.sh to TestLoadBalancer Go scenario Step 3 of docs/development/improvements/e2e-go-harness.md. Replaces 1246 lines of bash with a typed Go scenario exercising all 4 LB variants (ALB/NLB × internet-facing/internal) against a shared VPC + app instance pool. Internet-facing phase auto-skips when no peer node is configured; internal phase always runs. Adds shared harness primitives: - harness/aws.go: AWS SDK client factory wired to Spinifex AWS gateway - harness/lb.go: WaitForLBActive, WaitForTargetsHealthy, WaitForENICleanup, WaitForInstanceRunning/Terminated, InstanceENI - harness/traffic.go: HTTPRoundRobin, TCPRoundRobin, distribution asserts - harness/ssh.go: minimal peer SSH wrapper shelling out to /usr/bin/ssh (replaced by typed x/crypto/ssh client hoisted from DDIL in mulga-siv-74) User-data scripts embedded via //go:embed under lb/testdata/. Bash script stays in place for one CI cycle before deletion. * fix(e2e): log client launch params + dump daemon log on InvalidInstanceType Adds diagnostics for the LB internal client launch path. Logs the exact InstanceType, AMI, subnet, key name, and user-data size right before RunInstances, and registers a daemon-log dump on failure so we can correlate the 400 InvalidInstanceType with what spinifex saw. * refactor(e2e): serialise LB suites + share probe client across internal tests Splits TestLoadBalancer into 4 sequential subtests (ALB/NLB × internet-facing/internal). Each suite creates its own TG + LB + listener, runs its traffic test, and tears down before the next starts. Replaces the per-suite throwaway client VM with a single long-lived probe server launched at fixture setup; per-suite internal traffic tests GET /probe?ip=...&proto=... against it instead of cold-booting a fresh client. Drops peak instance count from 5 to 4 (2 app + 1 LB + 1 client), so the internal phase passes on capacity-constrained dev nodes that previously hit InvalidInstanceType (see mulga-siv-76, mulga-siv-77 for the underlying spinifex bugs). * feat(daemon): make hostReserve env-configurable for dev capacity squeeze Adds SPINIFEX_RESERVED_VCPU and SPINIFEX_RESERVED_MEM_GB env-var overrides read at daemon startup. Invalid values are logged and ignored so a typo doesn't silently widen the reserve. Unblocks the LB E2E on capacity-tight dev hosts (8 vCPU box: default 2-vCPU reserve + 3x t3.nano test load leaves 0 free for the sys.micro LB VM). Lays groundwork for the planned operator-tunable [capacity] config. bead: mulga-siv-78 * fix(e2e): revert LB to per-suite client + fail-fast on terminal LB state Reverts the shared probe-server VM that wouldn't come up reliably during the InternalFacing suites — back to the proven bash pattern of launching a fresh client VM per suite (one of ALB_IP / NLB_IP populated based on the suite kind). Adds renderClientUserData + a conditional template so each client probes only its target LB. WaitForLBActive now bails immediately on terminal LB states (failed, provisioning_failed) and emits a "still waiting" log line every 30s so hangs are visible instead of silently burning the full 5-minute timeout. * ci(e2e): add e2e-go.yml workflow + GOTESTSUM Makefile opt-in Standalone workflow runs the new Go-based E2E harness on the existing ci-single self-hosted runner — provisions a tofu VM, bootstraps spinifex, installs gotestsum, runs each requested suite via the e2e Makefile, and uploads per-test artifacts + JUnit XML + JSON streams as a build artifact. Test summary parses the JUnit and emits ::error file=...:: annotations so failures surface in the PR diff. Gated to the feat/e2e-go-harness branch + workflow_dispatch so it doesn't interfere with the existing e2e.yml while the Go conversion lands. tests/e2e/Makefile gains a GOTESTSUM=1 opt-in: when set, the e2e target runs gotestsum with --junitfile and --jsonfile rooted in ARTIFACT_DIR under a suite-tagged filename derived from the first PKG component. * feat(e2e): share probe client across both internal LB suites + SG fixes - Replaces per-suite probe client with a single shared VM launched in the fixture. Saves ~60s by avoiding a cold boot between the ALB and NLB internal suites. - Client cloud-init exposes a JSON trigger endpoint on :9090 in addition to the http.server on :80 (results dir). Each suite POSTs {proto, ip, count, outfile, http_port, tcp_port} to run a probe loop against the LB private IP and writes the result file under /tmp/httpd for the driver to fetch back. - configureDefaultSG now uses the structured IpPermissions form (top-level shortcut is silently ignored by vpcd — tracked separately) and opens port 9090 in addition to 80 + 9000. - deleteLB captures the underlying sys.micro VM id via the LB ENI before delete, then polls DescribeLoadBalancers until LoadBalancerNotFound so the next suite doesn't race the capacity reclaim. - Internal_NLB sleeps 15s up front to let the ALB sys.micro deallocate fully before its own createLB fires. DescribeLoadBalancers can report the ALB gone before the vCPU/memory allocation is actually released on capacity-tight dev hosts; without this settle, NLB's createLB trips the sys.micro reserve and lands in terminal state=failed. Still cheaper than cold-booting a per-suite client. * ci(e2e): install Go on tofu VM before running Go E2E suites The tofu-provisioned VM ships with the spx binary and bash test scripts but no go toolchain, so the prior workflow's "Install gotestsum on VM" step failed with `bash: line 2: go: command not found`. Add a curl-based Go 1.26.3 install (matching go.mod) ahead of the gotestsum install and update the test step's PATH so make e2e-* runs the freshly-installed go binary. * ci(e2e): bump gotestsum to @latest for Go 1.26 compatibility gotestsum v1.12.0 pins golang.org/x/tools v0.19.0 which fails to compile under Go 1.26 with: invalid array length -delta * delta (constant -256 of type int64) Switching to @latest pulls a newer x/tools that builds clean. * ci(e2e): ship spinifex source to VM so make -C tests/e2e can run there Single-node tofu VM only has the spx binary + bash test bundle at ~/e2e-tests; the Go e2e workflow was running 'make -C tests/e2e e2e-cert' from $HOME/Development/mulga/spinifex which doesn't exist, producing: make: *** No rule to make target 'e2e-cert'. Stop. Add a checkout of the spinifex repo on the runner and rsync it (minus .git, build/, bin/, node_modules) to the VM as ~/Development/mulga/spinifex before installing Go + gotestsum. Interim solution; mulga-siv-80 tracks replacing the rsync with shipping pre-built test binaries from the runner. * ci(e2e-go): replace rsync with tar-over-ssh; VM lacks rsync bootstrap-install.sh ships a minimal binary VM with no rsync; the ship-source and collect-artifacts steps failed with 'rsync: command not found'. Stream a tarball over ssh instead — same exclude list, no extra dependency on the VM. * fix(e2e): discover single-node IPs + awsgw bind in LoadEnv run-cert-e2e.sh single mode probes awsgw.host from spinifex.toml and adds every non-loopback global IPv4 to the node list. The Go harness defaulted to 127.0.0.1 only, so CI on a tofu VM (awsgw bound to the LAN IP, not loopback) saw 'connection refused' on every TLS check. Port the same discovery: ModeSingle now walks net.InterfaceAddrs for NodeIPs and grep-parses spinifex.toml for the awsgw bind. Both stay overridable via SPINIFEX_NODE_IPS / SPINIFEX_SERVICE_IPS. * ci(e2e-go): pre-build test binaries on runner; drop go toolchain from VM Replaces the source-tree + go + gotestsum payload that the e2e-go.yml workflow previously shipped to the tofu CI VM with self-contained test binaries built on the runner via `go test -c`. testdata for the LB suite is already //go:embed'd into the binaries, so the bundle on the VM is just `<suite>.test` files (cert: 14M, lb: 17M uncompressed). Workflow changes - actions/setup-go on runner. - New step builds binaries via `make e2e-build SUITES_BUILD=...`. - Install go-junit-report on runner; convert each VM-side test log to junit-<suite>.xml after artifact pull so "Publish test summary" still parses JUnit XML. - Drop "Install Go + gotestsum on VM" entirely (saves ~60s + network). - Drop source-shipping rsync/tar (saves ~30s + a lot of bandwidth). - VM step runs `./<suite>.test -test.v -test.timeout=30m` directly. Makefile changes - New `e2e-build` target compiles each `SUITES_BUILD` entry into `BIN_DIR/<suite>.test` via `go test -c -tags=e2e`. `clean` also removes BIN_DIR. Closes mulga-siv-80. * ci(e2e-go): disable setup-go cache; upload dominates self-hosted run time * refactor(e2e): hoist generic DDIL harness primitives to shared tests/e2e/harness Step 1.5 of docs/development/improvements/e2e-go-harness.md. Promotes Cluster/Node/SSH/DaemonClient/LocalStateRevision/ResetAllNodes/Witness into the shared package so the LB, multinode, and BM scenarios in steps 4-5 can build on the same primitives instead of duplicating SSH + cluster plumbing. Moves - ddil/harness/{cluster,ssh,daemon_client,state,reset,witness}.go -> harness/ (plus the witness testdata embed dir). - ddil/harness/{fault,netem,profiles,snapshot}.go -> ddil/fault/ as a new package; fault.PartitionNode/HealNode/KillNATS/StartNATS/ RestartDaemonOnly/ApplyNetem/ClearNetem/LinkProfile*/TakeSnapshot now take harness.Cluster / harness.Node / harness.SSH explicitly. - ddil/harness/retry.go::Run retry-with-reset extracted as harness.RetryWithReset in harness/wait.go; ddil/harness/quarantine.go keeps the DDIL-only quarantine wrapper that calls into it. - WaitForMode moved from fault.go to harness/daemon_client.go since it polls daemon state and has no fault-injection concern. - The pre-hoist harness/ssh.go (PeerSSH, subprocess ssh) is preserved as harness/peer_ssh.go so the lb_test suite keeps working unchanged; the typed x/crypto/ssh client from DDIL now owns harness/ssh.go. - New harness/util.go exports ShellQuote (was unexported shellQuote in netem.go) so both ddil/fault and reset.go/witness.go can use it. Env var rename - SPINIFEX_NODES / SPINIFEX_SSH_USER / SPINIFEX_SSH_KEY replace DDIL_NODES / DDIL_SSH_USER / DDIL_SSH_KEY. Old names still honoured with a stderr deprecation note (envWithLegacy) so the DDIL CI keeps passing until .github/workflows/e2e-ddil.yml drops the old names. DDIL_QUARANTINED stays DDIL-specific. DDIL parity preserved — scenarios still call ddilh.Run for quarantine semantics; the underlying retry-with-reset is the same code path, just shared. Closes mulga-siv-74. * fix(e2e): open default SG :22 ingress via structured form before witness launch The daemon currently drops the top-level shortcut AuthorizeSecurityGroupIngress form silently (mulga-siv-79), so the default SG starts with no inbound :22 and the witness EIP awaitBaseline tunnel times out under DDIL CI. Mirror the workaround already present in lb_test by issuing the structured IpPermissions form before LaunchWitnessVM. Idempotent on InvalidPermission.Duplicate so reruns in the same cluster are safe. * ci: render JUnit into job summary + collapsible groups in noisy steps Add a reusable composite action (.github/actions/junit-report) that parses a glob of JUnit XML files, writes a totals table + collapsible per-failure details into $GITHUB_STEP_SUMMARY, and emits one ::error annotation per failed/errored test case so failures surface in the run header instead of being buried in step logs. Apply the action to both e2e-go.yml and e2e-ddil.yml (e2e-ddil gotestsum now also emits --junitfile so the action has input). Same workflows also wrap noisy phases (tofu apply/destroy, scrub-env, tar-over-ssh, per-suite test runs, cluster-log collection) in ::group::/::endgroup:: markers so the Actions UI collapses the chatter by default and the human can expand only the failing phase. Tofu output still tees to $TOFU_LOG so the artifact bundle is unchanged. Future workflows (multinode, baremetal) can adopt the same pattern via the README in .github/actions/junit-report. * test(ddil): quarantine Scenario C until predastore §2 lands CI run 26016840819 surfaced that ScenarioC_CleanPartition silently depends on predastore-ddil-hardening §2 (write-quorum with deferred parity repair). Root cause from node3 daemon log: predastore: write udp ...->192.168.0.22:9991: sendmsg: operation not permitted predastore: putObjectViaQUIC: put failed spx QMP: BLOCK_IO_ERROR instanceId=<witness> spx: StopOrTerminateInstance: Terminating spx: Instance state transition running → shutting-down The 'sendmsg: operation not permitted' is the EPERM Linux returns when iptables drops an outbound packet — exactly the OUTPUT DROP rules PartitionNode installs. The witness VM's block I/O is backed by distributed predastore, so the same partition that severs node3↔peer NATS traffic also severs the witness disk's cross-shard QUIC writes. QEMU power-downs the guest after BLOCK_IO_ERROR, the daemon's reconciler terminates the instance, and the next AssertProgressed dials a sshd that's already gone (hence the 'handshake failed: EOF' the test reports). Quarantine by collapsing C to the same scenarioSkip stub D/E/F already use, citing predastore-ddil-hardening §2 as the dependency. Flip C back to ENABLED in the same PR that lands §2 — that's when the witness disk can survive a peer partition without QEMU halting. TEST_COVERAGE.md row C flipped ENABLED → SKIPPED; added a quarantine note explaining the predastore dependency. * ci: colored phase banners + quiet-on-success tofu output Pretty up the live Actions log without flooding it. Three changes: 1. .github/scripts/ci-fmt.sh — sourceable bash helpers (banner, ok, bad, warn, note, quiet_run) using ANSI colors that the Actions UI renders. 2. e2e-go.yml + e2e-ddil.yml — tofu phases switch back to quiet redirect into $TOFU_LOG (the previous tee'd version regressed live noise vs the original silent redirect). Each phase now prints one colored ok/✗ line; on failure the tail of $TOFU_LOG dumps inside a single collapsed group, so the live log stays scannable but the failing tail is one click away. 3. junit-report render.py — compact summary: short status icons (✅/❌) in a single column, no separate err column, single-line failure list (truncated to 120 chars) instead of <details> blocks that wrap badly on narrow screens. Per-suite test runs still wrap in ::group:: with an inline cyan banner so pass output collapses and the suite name is obvious when expanded. * test(e2e): add harness.Phase / Step / Detail for human-readable test output Test fixtures were emitting flat walls of t.Logf with every line prefixed by the same `lb_test.go:61:` (file:line of the call site that the t.Helper() chain bubbles up to) and forced 6-space indent. Hard to skim, hard to find where one phase ends and the next begins. Add three helpers in tests/e2e/harness/log.go: Phase(t, fmt, ...) — flush-left colored banner via fmt.Println. Bypasses t.Logf so the line lands without the testing framework's indent/file:line prefix. Pure visual structure; the diagnostic detail stays on t.Logf. Step(t, fmt, ...) — single-line sub-phase marker via t.Logf. Keeps file:line attribution so a triager can find the source of a progress message. Detail(t, k, v, ...) — key=value resource IDs via t.Logf. Same attribution rationale as Step — IDs are diagnostic and want to land in JUnit XML bucketed with the owning test. Why the split: Phase is purely a banner; losing JUnit attribution costs nothing because banners aren't diagnostic. Step/Detail carry triage context (instance IDs, AMI, etc.) and a failing test's JUnit testcase node should still contain them, so t.Logf attribution is kept. ANSI colors are auto-disabled when stdout isn't a TTY and GITHUB_ACTIONS isn't set, so piped local `go test -v` stays clean. Apply to lb_test setupSharedFixture — five Phase banners replace the implicit run of helper logs ("Discovering cluster capacity", "Ensuring SSH key pair", "Creating shared VPC topology", "Launching app instances", "Launching probe client") so the live CI log reads as a story rather than a flat resource dump. * ci(e2e-go): strip indent + file:line on display, propagate color env over SSH Two display bugs spotted on the previous run: 1. Phase banners came out white instead of bold-cyan because the test binary runs over SSH on the tofu VM, where neither GITHUB_ACTIONS nor a TTY is present, so harness.log.go init() disables ANSI escapes. Fix: export GITHUB_ACTIONS=true and FORCE_COLOR=1 in the remote env so the helper keeps emitting color codes. 2. Every t.Logf line still rendered as lb_test.go:61: instance type: t3.nano (4-space indent + `<file>:<line>:` prefix) because t.Helper() bubbles file:line to the call site. Fix: pipe the test binary's stdout through `sed -E 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'` so the display strips both. Tee to the artifact log happens BEFORE the sed so raw output is preserved for triage; only the live UI sees the cleaned form. * feat(e2e-harness): extend harness for single-node Go suite port Stage A of mulga-siv-89 — add the primitives the single-node run-e2e.sh port consumes: - AWSClient gains IAM + STS service handles, plus NewAWSClientWithCreds for per-tenant account-scoping in Phase 8. - awserror.go: ErrorCodeIs, AssertAWSError, ExpectError — Go equivalent of the bash expect_error helper used throughout the negative phases. - poll.go: WaitForInstance/Volume/Snapshot/Image state to any target, with per-resource default timeouts tuned from the bash sleep loops. - spxcli.go: thin shellouts for spx get nodes/vms, spx top nodes, spx admin account create with output parsing for tenant-id/access-key. - rawhttp.go: signed raw POST against the AWS gateway for the Phase 8 InvalidAction probe (gateway rejects on auth before action parsing). - ovn.go: OvnNbctl/OvnSbctl wrappers + SkipIfNoOVN for Phase 8b-e datapath checks; WaitForPortGroupMember handles propagation polls. - ec2helpers.go: InstancePublicSSHHost (public-IP-or-qemu-hostfwd), LsblkRootGiB, serial-console-access trio, DiscoverDefaultVPC, idempotent AuthorizeSSHIngress. - account.go: AccountCarousel maps tenant name -> scoped AWSClient, built atop NewAWSClientWithCreds for the Phase 8 isolation tests. Pure groundwork — no new tests, no production behaviour change. The cert + lb scenarios still compile against the extended harness. * ci(e2e-go): auto-cancel + reap orphans to prevent runner wedge A cancelled run on the single-runner ci-single host has been leaving orphan tofu/terraform procs (and stale tfstate locks) behind, which caused the next push's tofu destroy to hang for 45m+ and saturate the runner host until sshd refused new connections. - Add `concurrency: e2e-go-${ref}, cancel-in-progress: true` so a new push to feat/e2e-go-harness cancels the in-flight run instead of stacking on the same self-hosted runner. - New `Reap Orphan Procs` step at job start: SIGTERM then SIGKILL any lingering `tofu`/`terraform` from the runner user, and unlink any stale `.terraform.tfstate.lock.info`. - Harden `Teardown`: drop `set -e` (scrub-env must always run as the libvirt-level fallback), reap orphans before destroy, and wrap `tofu destroy` in `timeout 600` so a wedged provider can't pin the job to its 45m timeout. * ci(e2e-go): drop push trigger, manual dispatch only Pull the on.push branch trigger so the workflow only fires from workflow_dispatch — matches the e2e-ddil.yml Phase-1 pattern. Push trigger lifts again once the single-node Go port (mulga-siv-89) is green end-to-end and run-e2e.sh is retired. * test(e2e-single): port run-e2e.sh phases 1-4 to Go Stage B of mulga-siv-89. Lands the TestSingleNode orchestrator + Fixture that threads state between sequential phase subtests, plus phases: - Phase 1: KVM check, AWS gateway TLS reachability, daemon NATS readiness polling, region/AZ smoke. - Phase 1b: spx get nodes / top nodes / get vms (single-node only). - Phase 2: nano instance-type + arch discovery. - Phase 2b: serial-console-access default/enable/disable round-trips. - Phase 3: create-key-pair + import-key-pair lifecycle; test-key-1 PEM kept for Phase 5 instance launch (cleanup deferred to teardown phase in stage G). - Phase 4: architecture-aware AMI discovery via name filter ami-ubuntu-24.04-{x86_64,arm64}, falls back from spx import output. Phase ordering is contractual (Phase 5+ depend on AMI, key pair, AZ). detectPoolMode reads [network] external_mode from spinifex.toml so stages F/G can skip pool-only phases on dev_networking fixtures. No production code changed. cert + lb scenarios continue to compile. * test(e2e-single): port run-e2e.sh phases 5-5e to Go Stage C of mulga-siv-89 — the instance/volume/snapshot/image lifecycle that everything else hangs off: - Phase 5: run-instances (default VPC + default SG with tcp/22 opened via idempotent harness.AuthorizeSSHIngress), wait running, resolve root volume from BlockDeviceMappings. - Phase 5a-pre: spx get vms now contains the running instance. - Phase 5a: describe-instances metadata round-trip (type/key/AMI/BDMs). - Phase 5a-ii: TCP wait, ssh id, lsblk cross-check vs API root volume size (±1 GiB tolerance — lsblk rounds down), soft hostname check. - Phase 5a-iii: get-console-output round-trip on InstanceId. - Phase 5b: create-volume (10 GiB) → modify-volume (20 GiB) → attach-volume → detach-volume → delete-volume, polling each state transition through harness.WaitForVolumeState. - Phase 5b-ii: describe-volume-status on the root volume. - Phase 5c: create-snapshot → wait completed → copy-snapshot → both visible → delete original → copy survives → delete copy. - Phase 5d: AMI snapshot-backed launch verification via DescribeImages.BlockDeviceMappings[].Ebs.SnapshotId. Predastore-side config.json reads are TODO (no S3 client in harness yet). - Phase 5e: create-image --no-reboot, wait available, capture backing snapshot ID for Phase 9 cleanup. Phase 7a (attach to stopped → error) stubbed in volume_test.go for Stage D to fill once the lifecycle stop/start lands. * test(e2e-single): port run-e2e.sh phases 5f-8 to Go Stage D of the single-node Go port (mulga-siv-89). Adds four files under tests/e2e/single/ covering Phase 5f (SG egress ACL enforcement), Phase 6 (tag CRUD), Phases 7 / 7a / 7b / 7c-pre / 7c (stop, start, attach-to-stopped negative, ModifyInstanceAttribute, reboot, multi-count RunInstances), and Phase 8 negatives (18 sub-tests asserting AWS error codes via harness.ExpectError / AssertAWSError / PostAWSAction). Wired into TestSingleNode in main_test.go; phase ordering preserved because Phase 7 chain mutates the shared instance state (stopped at end of 7, .small after 7b, running after 7c-pre, untouched by 7c siblings) and Phase 8 negatives need a running instance with the AMI/key created earlier. Removes the placeholder phase7a_AttachToStoppedError stub from volume_test.go (real impl now in lifecycle_test.go). run-e2e.sh stays in tree until step-7 cleanup per the parent plan. * ci(e2e-go): wire e2e-single target + add single to build set Lets the in-progress single-node Go port (mulga-siv-89, Stages A-D landed) be exercised in CI via explicit `suites: e2e-single` dispatch input ahead of Stage G flipping the workflow default. Workflow default suites unchanged until Phase 9 teardown lands. * test(e2e-single): port run-e2e.sh IAM 1-7 + Phase 8 account scoping to Go Stage E of the single-node Go port (mulga-siv-89). Adds two files under tests/e2e/single/: - iam_test.go (903 lines): IAM Phases 1-7 — User CRUD, Access Key Lifecycle, User Authentication (via harness.NewAWSClientWithCreds), Policy CRUD (+ CreatePolicyVersion / SetDefaultPolicyVersion / DeletePolicyVersion which bash lacks), Policy Attachment & Enforcement (allow / deny / wildcard / root bypass), Policy Lifecycle (DeleteConflict while attached), and idempotent Cleanup. - account_test.go (1186 lines): Phase 8 EC2 Account Scoping — 11 Steps (Account Setup, Instance / Volume / KeyPair / Snapshot / VPC-Subnet / IGW-EIGW Scoping, Account Settings, Global Resources, Edge Cases, Cleanup). Uses harness.AccountCarousel for the multi-profile credential map and SpxAdminAccountCreate for admin account bootstrap. Fixture extended with IAM cross-phase state (IAMAdminAccount, IAM{Alice,Bob,Charlie}{KeyID,Secret}) so Phase 7 cleanup is defensive even when an intermediate IAM phase short-circuits. All 11 Phase 8 Steps share local state in the outer function; t.Cleanup mirrors Step 11 LIFO so a mid-Step failure still tears down accounts. Wired into TestSingleNode. run-e2e.sh stays in tree until step-7 cleanup per parent plan. * fix(e2e-harness): use kebab-case is-default filter for DescribeVpcs AWS EC2 DescribeVpcs filter names are kebab-case (is-default), and the daemon's filter allowlist enforces that. The harness sent the camelCase SDK field name (isDefault), so the daemon returned InvalidParameterValue 400 and DiscoverDefaultVPC fataled — leaving Phase 5 InstanceID empty and cascading 16 downstream Single-Node E2E subtests to fail. * test(e2e-single): port run-e2e.sh phases 8b-8e to Go Stage F of the single-node Go port (mulga-siv-89). Adds three files under tests/e2e/single/: - vpc_test.go (553 lines): Phase 8b (pool-mode VPC public/private subnet E2E — CreateVpc, dual subnets with MapPublicIpOnLaunch assertions, IGW attach, custom route table + 0.0.0.0/0 route, instance launch + SSH + external connectivity probe) and Phase 8c (route table validation — main RT association, local + IGW route presence, custom RT CRUD + association/disassociation, negative delete-local + delete-main assertions). - natgw_test.go (467 lines): Phase 8d (pool-mode NAT GW E2E — CreateNatGateway with EIP, private subnet route via NAT, bastion- hopped SSH into private VM, baseline egress drop / post-NAT egress succeed / post-teardown egress lost). Inline waitForNATGatewayState helper bounds the pending->available->deleted state machine via harness.EventuallyErr. - datapath_test.go (326 lines): Phase 8e (OVN-gated SG-to-SG datapath — two SGs with UserIdGroupPair ingress rule, OVN port-group membership + SB address-set wait loops, allowed traffic curl, denied traffic nc, revoke->drop, re-add->restore). Pool-mode gating via fix.PoolMode for 8b/8d; OVN gating via harness.SkipIfNoOVN for 8e. Per-phase t.Cleanup LIFO ensures teardown unwinds even on partial failure. Wired into TestSingleNode. * test(e2e-single): port run-e2e.sh phases 9, 9a, 9b to Go Stage G of the single-node Go port (mulga-siv-89) — final teardown + post-teardown verification. Adds tests/e2e/single/teardown_test.go (226 lines): - phase9_Teardown: DeleteSnapshot(CustomAMISnapID) -> DeregisterImage (CustomAMIID) -> TerminateInstances(InstanceID) + wait terminated -> DeleteKeyPair(KeyName). Each call swallows *NotFound via harness.ErrorCodeIs; unexpected errors logged as warnings so 9a/9b always run. - phase9a_VerifyTeardown: DescribeInstances must show terminated or NotFound; DescribeImages must NOT include the deregistered AMI; DescribeKeyPairs must NOT include the deleted key. Clears the consumed Fixture fields. - phase9b_FinalClusterStats: harness.SpxGetVMs must NOT list the terminated InstanceID; orphan i- rows surface as soft Logf warnings to avoid flakes when parallel suites share the node. Wired into TestSingleNode. With this, mulga-siv-89's single-node Go port (Stages A-G) is feature-complete; run-e2e.sh remains in tree until a separate cleanup bead deletes it after CI burn-in. * ci(e2e-go): unbuffer sed in test pipeline for live log streaming GNU sed defaults to block-buffered stdout when stdout is a pipe (GitHub Actions runner). Accumulates 4-64KB before flushing, so test output appeared in sporadic batches unaligned to test boundaries. -u flag forces line-buffered output so each go test -v line surfaces in the Actions UI as it is produced. Side effect: none. No process count change, no SSH session change, no test execution change. Pure output flushing on the runner host. * fix(e2e-single): persist key PEM across phases and skip unimplemented IAM ops Three bugs blocked the Single-Node Go E2E suite after the DescribeVpcs filter fix: 1. Phase 3 wrote the SSH PEM into `t.TempDir()`, which Go cleans up when the Phase 3 subtest exits. Phases 5a-ii / 5a-iii / 7b / 7c-pre then saw "Permission denied (publickey)" / "Identity file not accessible" and the cascade left the instance in a state that broke 7c-pre's `start-instances`. Move the PEM under a TestSingleNode-scoped `fix.TmpDir` so it survives until the whole test exits. 2. IAM Phase 2's `require.NotEmpty(t, fix.IAMAdminAccount, "")` was intended as a no-op (Phase 4 populates the field), but require.NotEmpty fails on empty regardless of message. Removed the bogus check. 3. The daemon does not yet implement `UpdateUser` (IAM Phase 1) or the policy-version family (IAM Phase 4): `CreatePolicyVersion`, `ListPolicyVersions`, `SetDefaultPolicyVersion`, `DeletePolicyVersion`. They return `InvalidAction` and abort the phase. Wrap each block in a t.Run sub-test with t.Skip so the surrounding phase still runs and the skips show up explicitly in the test report. * test(e2e): parallelise hot phases + tune fast-path polls Squash merge of feat/e2e-parallel-quickwins (mulga-siv-96). Four quick wins from decomposing run-e2e.sh into Go phase subtests: - Phase 8 negatives (single/negative_test.go): t.Parallel() inside each of the 18 read-only error-path sub-tests. ~5-10x speedup. - IAM Phase 4 policies (single/iam_test.go): EC2ReadOnly stays seeded sequentially (populates fix.IAMAdminAccount); the other four policies fan out as siblings under a wrapping create_policies_parallel t.Run. ~5x on the Phase 4 sub-step. - Volume + snapshot fast paths (single/{volume,snapshot}_test.go): pass harness.WithPoll(500ms) on the five state transitions that finish in <1s on local QEMU. Halves Phase 5b / 5c wall-clock. - Cert per-IP loop (cert/cert_test.go): t.Parallel() inside the 4 inner t.Run(ip, ...) blocks. Cuts cert phase by len(nodes). No harness changes, no Fixture changes, no test-execution-order changes. mulga-siv-96. * test(e2e-lb): resolve peer availability before shared fixture setup setupSharedFixture builds a VPC + 2 app VMs + probe client (minutes on capacity-tight dev hosts) and previously ran BEFORE the peer pickup + ssh.Ping. On single-node, the InternetFacing_* subtests would correctly t.Skip on peer == "" — but only after the fixture work and once subtests started, so the wait appeared to be attached to the first subtest name gotestsum printed. Reorder: pickPeer + ssh.Ping first, log "internet-facing will skip" immediately, then build the fixture for the internal subtests. No behavior change for multi-node + peer-reachable; on single-node the skip rationale surfaces before the fixture wait. * test(e2e-single): extend SSH wait budget to 3m on slow boot paths CI run 26034322018 timed out Phase5a_ii_SSH (60s budget) and Phase8b_VPCSubnetE2E (2m budget) on the TCP-port-open poll while sshd was still finishing pam/cloud-init. Bump all four wait sites to 3m as a stop-gap: - single/instance_test.go phase5aii_SSHProbe - single/lifecycle_test.go waitForSSHReady (Phase 7b/7c/7c-pre) - single/vpc_test.go phase8b_VPCSubnetE2E - single/natgw_test.go phase8d bastion SSH wait Proper fix tracked under mulga-siv-101 (hoist a harness.WaitForSSH that probes the actual handshake, not just tcp/22 reachability). * test(e2e-single): skip-gate IAM3/IAM5/IAM6 until daemon handlers land CI run 26034322018 surfaced three IAM daemon gaps: - IAM3_UserAuthentication: alice's freshly-created access key returns 403 from DescribeInstances — daemon doesn't honour scoped IAM key signatures yet. - IAM5_PolicyAttachmentEnforcement: cascades from IAM3 (alice can't sign). - IAM6_PolicyLifecycle: DetachUserPolicy returns 404 even on attachments Phase 5 confirmed — daemon attachment ledger gap. Skip-gate each phase with t.Skip + a pointer to mulga-siv-100 so the remaining IAM coverage (IAM1/IAM2/IAM4/IAM7) keeps surfacing regressions, and the suite as a whole doesn't fail on known gaps. Flip to real assertions once the daemon handlers land. * test(e2e-single): probe full SSH handshake (not just TCP port open) CI run 26036025293 hit "Connection timed out during banner exchange" on Phase5a-ii / 8b / 8d even with the 3m TCP-port-open budget: tcp/22 opens immediately, but sshd is still finishing pam/cloud-init, so the first real runSSH call afterwards races the half-open daemon. Switch waitForSSHReady to issue an actual ssh BatchMode + ConnectTimeout handshake against the target, retrying until the daemon completes the banner exchange. Drop the four TCP-only Eventually loops in favour of the upgraded helper: - instance_test.go phase5aii_SSHProbe - lifecycle_test.go waitForSSHReady (Phase 7b / 7c-pre / 7c) - vpc_test.go phase8b_VPCSubnetE2E - natgw_test.go phase8d_NATGateway (bastion) Signature: waitForSSHReady(t, host, port, keyPath). Backwards-compat alias waitForSSHHandshake exposed for the instance_test caller. Proper harness hoist tracked under mulga-siv-101. * test(e2e-harness): dual-emit Step + Detail for live CI streaming go test's testing framework buffers t.Logf output per-test and releases it only on PASS/FAIL. Long subtests (Phase8Acct_AccountScoping ~115s, Internal_NLB ~60s, Phase8b_VPCSubnetE2E 2-3m, Phase8d_NATGateway ~90s) therefore went silent in CI for the entire subtest duration, even with sed -u line-buffering in the workflow pipeline. harness.Phase already wrote to os.Stdout directly for this reason. Extend the pattern: have Step and Detail dual-emit — write the same line to os.Stdout (streams live) AND t.Logf (keeps JUnit per-test attribution). The duplicate on a failure is acceptable; both copies carry identical content. mulga-siv-103 (note: numbering skipped past siv-102 which was already filed in the parent branch). * test(e2e-harness): drop t.Logf duplicate from Step + Detail The dual-emit (Fprintln + t.Logf) caused every Step/Detail line to appear twice after long subtests: once live via stdout, once again when the testing framework released its per-test buffer at PASS/FAIL. Most visible after lb subtests (Internal_NLB ~60s) where the bulk release dumped ~dozens of duplicated lines at once. Drop the t.Logf copy. go-junit-report parses stdout between `=== RUN` and `--- PASS/FAIL` markers, so per-test attribution in JUnit XML is preserved without the duplicate. Refs: mulga-siv-103 * test(e2e-harness): stream lb-waiter progress live in CI WaitForLBActive, WaitForTargetsHealthy, WaitForENICleanup, and WaitForInstanceRunning emitted progress via t.Logf, which the testing framework buffers per-test. After the previous commit dropped the dual-emit, that meant every poll loop went silent until subtest end — visible in CI as 'nlb is active' hanging on screen for the whole duration of the subsequent traffic check. Route progress through harness.Step (direct stdout). Also emit a 'waiting for X' Step at the start of each helper so observers know what the silent stretches actually represent. Refs: mulga-siv-103 * test(e2e): dump VPC/IGW datapath diagnostics on Phase8b SSH timeout Phase8b's 3min SSH timeout on a fresh-VPC VM (CI run 26037892507) is the customer-visible surface of mulga-siv-105 — vpcd's handleIGWAttach returns before ovn-northd compiles + the gateway chassis installs flows. Two other cleanup-only bugs (siv-106 missing spinifex:cidr, siv-107 dnat_and_snat double-delete) showed up in the same triage. Keep the test hard-failing so the regression stays visible, but dump a triage bundle on timeout so each occurrence is actionable: - guest console output (last 4KB — boot + cloud-init state) - ovn-nbctl show + ovn-sbctl show - Logical_Router external_ids (confirms siv-106 reproduction) - NAT rules (confirms siv-105 + siv-107 surface) - Port_Binding chassis,up (confirms gateway port up state) Diagnostics go straight to stdout (Step/Detail rationale) so they stream live in CI rather than dumping at PASS/FAIL. Refs: mulga-siv-105, mulga-siv-106, mulga-siv-107 * ci(junit-report): drop parent t.Run containers from tally + list skips with reasons GitHub Actions test summary was counting Go's top-level Test* parent plus its failing subtest as two distinct failures (the parent is just a rollup of its children). The summary now de-dupes by dropping any test case whose name is a strict prefix of another's, so the per-file counts and the Failures block both match the actual leaf-test outcome. CI run 26037892507 was reporting '2 failed' for single when it should have been 1. Also adds a collapsible Skipped section listing each skipped leaf test with its t.Skipf call-site reason (the .go:<line>: line from the CDATA). ANSI escapes from harness.Phase banners are stripped so the reason is plain text. Reasons are useful triage signal — current run has 9 skips spread across daemon gaps (mulga-siv-100) and pool-mode gates (InternetFacing_*). Refs: mulga-siv-89 * ci(e2e-go): propagate cancel to remote .test procs + nuke VM before destroy Two related leaks made cancelled runs strand QEMU work on the libvirt host: 1. The long `ssh tf-user@VM "for SUITE in …; do ./$SUITE.test"` ran without -tt, so when the runner cancelled the step the local ssh died but sshd never sent SIGHUP through (no PTY). The remote `.test` binary orphaned and kept running until its own 30m `-test.timeout`, holding tmp dirs, NATS connections, and qcow2 file handles. 2. Teardown went straight to `tofu destroy` without first killing any stragglers on the VM, so destroy raced the .test's open file handles on the root disk — sometimes successfully, sometimes leaving the libvirt host with a half-released qcow2. Fix: - Allocate a PTY on the run-suites ssh (`-tt`) so sshd HUPs children when the local ssh dies. - Add a trap (EXIT INT TERM) in the run-suites step that ssh's back to the VM and pkill's `/spinifex-e2e-bin/*.test` belt-and-braces. - Mirror the same pkill at the top of Teardown so destroy runs against a quiesced VM. Both cleanups are scoped to the per-run wan_ip — safe to land while multiple ci-single runner agents run concurrent jobs against different tofu envs on the same host (we deliberately keep that parallelism). Refs: runner host wedged 2026-05-19 (post-cancel orphans suspected; no journal access yet to confirm). * ci(e2e-go): scope orphan reap to GITHUB_WORKSPACE so it spares sibling runners Reap Orphan Procs (job start) and the in-Teardown pkill block were using `pkill -u $UID_SELF -x tofu/terraform` plus `find $HOME -maxdepth 6 -name .terraform.tfstate.lock.info`. The single host runs 5+ self-hosted runners all as `tf-user`, so every e2e-go run was SIGKILLing in-flight tofu from sibling workflows (e2e.yml, e2e-ddil.yml, e2e-baremetal.yml, ...) and deleting their state-lock files — exactly the kind of cross-runner collateral the reap was meant to recover from. Scope by `/proc/$pid/cwd` starting with $GITHUB_WORKSPACE; each runner's workspace dir is unique on the host, so only this run's own tofu procs match. Find for stale lockfiles is likewise scoped to $GITHUB_WORKSPACE. * ci(e2e-go): bump setup-go v5->v6, upload-artifact v4->v7 GitHub now warns Node 20 actions are deprecated. e2e-go.yml lagged the rest of the repo; bump to match versions already in use: - actions/setup-go@v5 -> @v6 (matches spinifex.yml) - actions/upload-artifact@v4 -> @v7 (matches release.yml, e2e-ddil.yml) checkout@v6 and create-github-app-token@v3 are already current across all workflows. The local junit-report action is `using: composite` (shell-only), so the Node runtime change does not apply. * fix(vpcd): add flows-ready barrier after IGW attach (mulga-siv-105) handleIGWAttach wrote the external switch + gateway port + LRP + default route to OVN NB and returned immediately, before ovn-northd had compiled NB -> SB and before ovn-controller on the gateway chassis had installed the resulting flows. A subsequent RunInstances -> vpc.add-nat could complete while the datapath was still dark; the launched VM was then unreachable on its public IP for whatever flow-install latency the chassis happened to incur. Customer-visible as 'I just created a VPC, launched a VM, and can't SSH in' for ~minutes. Reproduced deterministically in CI run 26037892507 Phase8b: a VM in a fresh VPC was unreachable at 192.168.0.202:22 for 3+ minutes, while a VM at the same public IP in the warm default VPC (Phase8d) was reachable in ~40 seconds. Block at the end of handleIGWAttach on `ovn-nbctl --wait=hv sync`, which bumps NB_Global.nb_cfg and waits for every connected chassis to acknowledge the new sequence number — i.e. flows installed. Bounded by --timeout=30; overruns log a WARN and continue (a 30s OVN stall is already a deeper problem than failing VPC create would surface). waitForFlowsHV is a package var so tests can stub it; in environments without ovn-nbctl on PATH the shell-out fails fast and the WARN path returns nil, so existing IGW tests stay green without a live OVN. Refs: mulga-siv-105 * fix(vpcd): store spinifex:cidr on VPC create (mulga-siv-106) handleVPCCreate built the OVN logical router with only spinifex:vpc_id and spinifex:vni in ExternalIDs. Cleanup paths (handleIGWDetach + NAT sweeps) read external_ids['spinifex:cidr'] to scope SNAT removal to the VPC's own CIDR; when unset, they fell back to an overbroad 10.0.0.0/8 and emitted a WARN. reconcileVPC already stored the same key at startup; the live-create path drifted. Add the field to handleVPCCreate, log the cidr alongside vpc_id, extend TestTopologyHandler_VPCCreate to assert the external_id is set. Refs: mulga-siv-106 * fix(vpcd): make handleDeleteNAT idempotent on 'already absent' (mulga-siv-107) Instance termination fans out through multiple cleanup paths (instanceCleanerAdapter.ReleasePublicIP AND VPCServiceImpl.DeleteNetworkInterface), both of which publish vpc.delete-nat for the same (vpc, logical_ip). The publish is fire-and-forget (utils.PublishNATEvent uses PublishEvent for vpc.delete-nat, not RequestEvent), so callers never branch on the response. The second publisher then triggered a spurious 'ERROR vpcd: failed to delete dnat_and_snat rule ... not found' log line ms after the first delete succeeded — confusing triage signal with no actionable consequence. Add ErrNATNotFound sentinel to the OVN client interface and wrap the not-found return paths in both LiveOVNClient and MockOVNClient. handleDeleteNAT matches errors.Is(err, ErrNATNotFound) and treats that as success (logs INFO 'already absent (idempotent)'). Existing TestTopologyHandler_DeleteNAT_NotFound asserted the opposite contract; the comment claimed EIP disassociation needed the error signal, but DisassociateAddress also fire-and-forgets vpc.delete-nat, so the contract never had a real consumer. Test renamed to _Idempotent with assertSuccess. Other failure modes (router missing, OVN unreachable, transact error) still surface as ERROR + reply with the error string. Refs: mulga-siv-107 * ci(e2e-go): cluster JUnit failures + name likely root cause (#230) * ci(e2e-go): cluster JUnit failures + name likely root cause New composite action `.github/actions/e2e-analyze` runs after `go-junit-report` converts each suite log. For every `<failure>` it strips ANSI, normalises dynamic noise (EC2 IDs, UUIDs, IPs, timestamps, file:line) to a stable signature, picks the earliest non-cascade failure per suite as the likely root cause, and buckets the rest by signature. Output is appended to `$GITHUB_STEP_SUMMARY` and written to `<log-dir>/analysis.md` so it travels with the artifact bundle. Cascade markers (`must populate fix.`, `Should NOT be empty`, `Expected value not to be nil`) collapse the typical "one upstream phase fails, ten downstream `require.NotEmpty` assertions trip" pattern into a single grouped entry under the real first failure. The analyser is advisory — it always exits 0. The existing "Fail job if any suite failed" gate keeps owning exit-code semantics. Stage 1 of `docs/development/improvements/e2e-go-failure-analysis.md` (mulga-siv-102). Stage 2 (per-failure log slicing) and Stage 3 (rule-based hints via `tests/e2e/.failure-rules.yaml`, placeholder seeded here) are independent follow-ups. CI wiring: - e2e-go.yml: new "Analyze e2e failures" step between "Publish test summary" and the artifact upload, always-runs. - spinifex.yml: new `make test-actions` step so the golden-file tests under `.github/actions/e2e-analyze/testdata/` run on every PR. - Makefile: `test-actions` target; kept out of `test-cover` so CI-only tooling doesn't dilute coverage %. Golden fixtures: a synthetic cascade-vpcs case (one Phase 5 InvalidParameterValue root + 15 downstream cascades), one real single-failure case captured from run 26037892507, and a clean no-failures case. * ci(e2e-go): per-failure log bundles with time-window slicing Stage 2 of the e2e failure analyzer. For each failure surfaced by Stage 1, materialise <log-dir>/analysis/NN-<TestName>/ containing: failure.txt — summary (test, start, duration, file-site, error) journal.log — spinifex-journal.log sliced to [StartAt - 5s, StartAt + Duration + 5s] test.log — test-<suite>.log block between `=== RUN <name>` and the matching `--- FAIL` marker <copies> — any files the harness already dropped at <log-dir>/<TestName_with_/_replaced>/ Report links each entry to its bundle so a reader on the Actions page goes straight to the relevant slice. Workflow now invokes journalctl with --output=short-iso so the analyzer can parse wall-clock timestamps for the journal window. Lines without a parseable timestamp inherit the previous line's classification (keeps multi-line records and stack traces contiguous). Render output is unchanged when BundlePath is empty, so the Stage 1 golden snapshots still match. * ci(e2e-go): restore GOTOOLCHAIN=auto after setup-go v6 bump setup-go@v6 flipped the GOTOOLCHAIN default from "auto" to "local", which makes Go refuse to auto-download the toolchain version named in go.mod. The self-hosted ci-single runner ships /usr/local/go older than spinifex/go.mod's `go 1.26.3`, and build-install-artifacts.sh prepends that path so the system Go wins the resolution race. Under v5 ("auto"), Go silently fetched 1.26.3 on demand; under v6 ("local"), the Build Install Artifacts step errors out: go: spinifex/go.mod requires go >= 1.26.3 (running go 1.26.1; GOTOOLCHAIN=local) Re-set the env to "auto" so behaviour matches pre-v6 runs. * ci(e2e-go): re-export GOTOOLCHAIN=auto after setup-go Workflow-level env doesn't help: setup-go@v6 writes GOTOOLCHAIN=local to $GITHUB_ENV, which overrides workflow env in every later step. Add a dedicated step right after setup-go that appends GOTOOLCHAIN=auto to $GITHUB_ENV so the Build Install Artifacts step (which uses /usr/local/go from a stale runner path) can self-heal by fetching the toolchain named in go.mod, matching pre-v6 behaviour. * fix(vpcd): add flows-ready barrier after dnat_and_snat add handleAddNAT wrote the NAT rule to OVN NB and returned immediately, mirroring the gap siv-105 closed for handleIGWAttach but on the NAT add path. ec2d/RunInstances then returned success while ovn-northd was still compiling NB->SB and ovn-controller hadn't installed the DNAT/SNAT flows on the gateway chassis, so the freshly launched VM was unreachable on its public IP for whatever flow-install latency the chassis happened to incur. Reproduced in CI run 26072432957 Phase8b_VPCSubnetE2E: VM at 192.168.0.212 timed out at 3min while diagnostics showed Port_Binding up=true for both the VM eni and the chassis-redirect gateway. SB-side convergence was complete; the missing piece was the same `ovn-nbctl --wait=hv sync` barrier siv-105 added to IGW attach, applied at the end of handleAddNAT as well. Reuses the existing package-var `waitForFlowsHV` so tests can stub it; no new infra. Parallel unit test asserts the call site fires exactly once per add-nat. * ci(e2e-analyze): real-artifact fixes for journal slicing Three bugs surfaced after Stage 2 ran against real CI artifacts in run 26072432957: 1. Regex `[+-]\d{4}` rejected the colon-offset form (e.g. `+10:00`) that systemd's `journalctl --output=short-iso` actually emits on most versions. Added a parser that tries RFC3339 first and falls back to the no-colon `-0700` layout. Result: 0-line journal slices become populated. 2. go-junit-report's `timestamp` attr on <testsuite> records the moment the XML was produced (post-suite), not when the tests started. Computing StartAt as `that + cumulative-priors` placed every failure AFTER the journal collection ended, so windows always missed. Workflow now writes `${ARTIFACT_DIR}/test-<suite>.start` right before running each suite; analyzer prefers that file over the junit timestamp. 3. Persisted `OffsetFromSuiteStart` on each Failure so the suite-start override can recompute StartAt without re-parsing the XML. Unit tests added for the colon-offset journal format. End-to-end test (TestWriteBundles_EndToEnd) still passes with the new layout. * ci(e2e-analyze): skip parent testcases when summing suite offsets go-junit-report emits one <testcase> for the parent Go test AND each subtest, with the parent's time = sum of its children. ParseFile was summing every testcase into the suite-cumulative-offset counter, so every leaf's offset was inflated by ~the parent's total duration. Visible on run 26073817490 as Phase8b's StartAt landing 13 minutes after the suite finished collecting logs — beyond the journal window, so the per-failure journal slice was empty. Pre-compute the leaf set per suite (a name is a leaf iff no other testcase name starts with `<name>/`) and only accumulate cumul on leaves. Golden snapshots updated; the offset for the only failure in each fixture now matches the actual leaf-sum start time. * test(e2e-harness): capture IP-targeted datapath state on Phase8b SSH fail Phase 1 diagnostics for mulga-siv-111. The Phase8b SSH timeout reproduced on the e2e-failure-analysis branch with siv-105's IGW barrier AND the new add-nat barrier BOTH firing cleanly, so the residual gap is downstream of `--wait=hv sync` — most likely stale OVS conntrack, stale upstream-host ARP, or an OF flow-install miss. DumpVPCFlowDiagnostics now takes a VPCDiagnosticsOpts carrying the external IP, logical IP, and per-test artifact dir. When set, the helper greps `ovs-ofctl dump-flows br-int`, `conntrack -L`, and `ip -4 neigh show` for the external IP / logical IP and snapshots `ovn-nbctl find NAT external_ip=…`. Each capture is both printed inline (existing affordance) and written to a separate file under fix.Artifacts so the Stage 2 analyzer bundler picks it up alongside the journal + test-log slices, no analyzer changes needed. Phase8b's caller now passes pubIP / privIP / fix.Artifacts so the captures fire on the next failure; updates the t.Fatalf message to point at siv-111 instead of siv-105 (which is the now-closed parent). * fix(e2e-single): authorize SSH ingress on fresh-VPC default SG in Phase8b Phase 8b stands up a non-default VPC and runs an instance there. Its auto-created default SG denies all ingress, so the SSH probe times out even when every datapath barrier converges — OF flow captures show the SYN reaches OVN, gets DNAT'd at table 13, then drops at the ACL between tables 13 and 17. The bash driver relies on admin init having pre-baked this rule on the *default* VPC's default SG; a fresh VPC has not. Add a DescribeSecurityGroups + AuthorizeSSHIngress step between route-table setup and run-instances, using the existing harness helper. Fix surfaced via mulga-siv-111's Phase 1 diagnostics (OF flow + ACL drop counts), invalidating the H1/H2/H3 stale-state hypotheses in the bead plan doc. * test(e2e): dual-detect ubuntu image version (24.04/26.04) during gold-image transition After mulga gold image bumped to v6-fastboot (stages ubuntu-26.04 / resolute instead of ubuntu-24.04 / noble), all consumers that hardcoded 24.04 break on the new image. Make them detect at runtime so v3 and v6 envs both pass. - tests/e2e/run-e2e.sh: DescribeImages filter accepts both ami-ubuntu-{26,24}.04-<arch>, then derives IMAGE_NAME from the match. - tests/e2e/run-pseudo-multinode-e2e.sh: prefer ~/images/ubuntu-26.04.img, fall back to ubuntu-24.04.img; pass detected version to spx import. - tests/e2e/single/image_test.go: walk candidate catalog names (26.04 first, then 24.04), stop on the first one DescribeImages can resolve. - .github/workflows/e2e-baremetal.yml: download resolute (26.04) into BM_ISO_CACHE; update BM_UBUNTU_IMAGE env var path. - docs/terraform-workbooks/{bastion-private-subnet,s3-webapp,nginx-webserver}: AMI filter matches `*ubuntu-26.04*` OR `*ubuntu-24.04*` (AWS values= array is ORed). Companion changes for mulga (bootstrap scripts, baremetal, iso pipeline) land in scripts/tofu-cluster/* and scripts/{tofu-iso,baremetal,iso-builder}/* under bead mulga-siv-114. Drop the 24.04 fallbacks once all envs are on the v6 gold image and 24.04 entries are removed from spinifex/utils/images.go. * feat(e2e): add service-interfaces.yaml manifest + validator Source of truth for which spinifex services produce/consume which NATS subjects, and which e2e suites cover which services. Consumed downstream by the targeted suite selector (Bead 4) to derive subset E2E runs from a PR diff so single-service changes stop paying the full 25 min suite cost. - docs/service-interfaces.yaml: 7 services, 4 active suites (cert, lb, single, ddil). Subscribes / publishes / depends_on filled from grep of NATS subjects in source. spinifexui marked covers_no_suite so UI-only diffs skip e2e entirely. Future suites (multinode, reboot, tofu, baremetal) commented in until Go ports land. - tests/e2e/manifest-check: schema + cross-reference validator (path existence, subject regex, duplicate detection, suite covers resolves to a real service, fixtures references resolve). 8 unit tests cover the failure modes plus a load-and-validate on the real manifest. - Makefile: manifest-check target, hooked into preflight. - docs/DESIGN.md: link to the manifest under "NATS Messaging". mulga-siv-116. See docs/development/improvements/e2e-targeted-suite-selection.md. * fix(e2e): drop catalog-download import path from phase4_Image `spx admin images import --name ubuntu-26.04-x86_64` with wantErr=true was meant to detect that bootstrap-install.sh already imported the AMI and short-circuit on the duplicate exit code. After the v6 gold-image bump preloaded `~/images/ubuntu-26.04.img` and bootstrap-install.sh switched to `--file` for the import, two things broke at once: - The catalog `--name` path no longer hits a duplicate guard for the --file-imported AMI, so the import re-downloads ~857MB and either succeeds (failing the wantErr=true assertion) or wedges the job timeout while the catalog server crawls along at 16KB/s. - `--file` from the test won't work either: the test binary runs as `tf-user` over SSH, while bootstrap-install.sh placed the image at `/home/spinifex/images/...` (mode 0700 home dir). bootstrap-install.sh is already responsible for staging the AMI. The test's job is verifying DescribeImages round-trips by-name and by-ID — not re-exercising import. Trust the bootstrap-staged AMI and stop calling import. Keeps the 26.04→24.04 dual-detect for the v3 gold image transition; drop the 24.04 candidate when no envs remain on v3. Refs mulga-siv-115 (caught by the EC2 cold-boot fast-path E2E run on feat/ec2-coldboot-fastpath). * ci(e2e-go): default to e2e-single + harden status-report SHA fallback Two changes to .github/workflows/e2e-go.yml: 1. Default `suites` input now includes e2e-single so a no-input manual dispatch exercises the full single-node ladder. Single-node Go harness is greenable now (image_test fix at 10eb38a). 2. Report E2E (Go) Status job no longer gated on `inputs.sha` and falls back to `github.sha` when the input is unset. Previously a manual dispatch without a PR head SHA either skipped the status post entirely or hit `gh api repos/.../statuses/` with an empty path component (the 'invalid hash' error) — tests passed, the report step failed silently. Now status posts on every dispatch against the dispatched ref's commit when no explicit SHA given. Closes mulga-siv-112. * feat(e2e): add Ensure* fixture API with memoization + cleanup Introduces tests/e2e/harness/fixtures.go: the memoized, cleanup-aware provisioning helpers that Bead 3 (single-fixture-migration) will rewrite phase tests around. Each Ensure* helper: - takes (t *testing.T, fx *Fixture, args...) - memoizes by a key derived from args (sync.Mutex + singleflight) - registers t.Cleanup on the Fixture's parent test so memoized resources survive child-subtest lifetimes - bakes readiness polling into the helper (no sleeps in callers) Exported helpers: EnsureKeyPair, EnsureAMI, EnsureDefaultVPC, EnsureSubnet, EnsureSG, EnsureInstance, EnsureVolume, EnsureSnapshot, EnsureNATGateway, EnsureLoadBalancer. Fixture stores ec2iface.EC2API + elbv2iface.ELBV2API so unit tests can inject fakes without building a real AWSClient (TLS + env). Compile-time checks guard that the SDK concrete types satisfy the stored interfaces. Manifest fixtures: block populated — Bead 4 selector will resolve test coverage from AST-scanned Ensure* call sites via this map. Makefile: new test-harness target runs the harness unit tests under -tags=e2e (mocked, no infra). Hooked into preflight so signature drift and memo/cleanup regressions surface locally. Bead 2 of docs/development/improvements/e2e-targeted-suite-selection.md. No existing tests modified — fixture API is net-new. * feat(e2e): port run-reboot-e2e.sh to Go Adds tests/e2e/reboot/reboot_test.go — Go port of the single-node host-reboot resilience driver (cell #18). Runs from the GitHub Actions runner (not shipped to the VM like cert/lb/single) because systemctl reboot kills any in-VM process mid-step. All AWS API traffic crosses the WAN to https://WAN_IP:9999 via the SDK; SSH-level ops (reboot, ovn-nbctl, journalctl, HTTP burst) hop via harness.PeerSSH. Trust path tries to SCP the cluster CA off the VM and falls back to InsecureSkipVerify with a log message if SCP fails — the cert suite already validates real trust, so the fallback is intentional. Harness changes: - env: WANHost field sourced from SPINIFEX_WAN_IP, distinct from ServiceIPs[0] (which is the awsgw bind IP, often loopback in-VM). - aws: SPINIFEX_AWS_INSECURE=1 escape hatch when the runner has no CA on disk. Used as the fallback when CA fetch fails. - cert: SPINIFEX_CA_CERT explicit path override so the reboot test can point ResolveCACert at the SCP'd tmp file. - peer_ssh: IsReachable() probe wrapper for cleanup paths that must skip rather than fatal when the host is mid-reboot. Workflow (.github/workflows/e2e-go.yml): - New e2e_go_reboot job, opt-in via `e2e-reboot` in the SUITES dispatch input. needs: e2e_go_single to serialise on the ci-single self-hosted runner. Fetches AWS creds from the VM's ~/.aws/credentials over SSH (mirrors the cred hop already proven by e2e-ddil.yml). - e2e_go_single filters 'reboot' out of its build + run loops so the in-VM driver doesn't try to ship reboot.test to the VM and reboot it from inside. - e2e_go_status treats skipped reboot as success. Refs mulga-siv-118. Plan: docs/development/improvements/e2e-go-reboot-port.md (in the mulga parent repo). * ci(e2e-go): run reboot job in parallel with single Drop `needs: e2e_go_single` from e2e_go_reboot. Both jobs target the same [self-hosted, ci-single] label, so GitHub Actions serialises on runner allocation rather than blocking via job ordering — parallel-by- spec, queued-by-runner. Lets a reboot-only dispatch start immediately instead of waiting for an unrelated single-job pre-condition. * fix(e2e/reboot): wait for both backends to serve before post-reboot burst Phase 8.6 previously probed until ANY HTTP response from the ALB, then fired a 20-req burst. Post-reboot, only one of the two backends may be serving (the second is still bringing its responder online) while the daemon's TG cache still shows pre-reboot 'healthy' for both — so WaitForTargetsHealthy returns in ms instead of seconds, the ANY probe matches the first-up backend, and the burst lands 20/0 instead of 10/10. Probe until BOTH backends' instance_ids each appear at least once before the round-robin burst — confirms recovery has reached both, not just one. Underlying daemon-side TG cache staleness tracked separately as mulga-siv-119. This wait is the test-side guard; the daemon fix will make WaitForTargetsHealthy meaningful again post-reboot. Refs mulga-siv-118. * ci(e2e-go): skip single job when dispatch requests only reboot Add if: gate to e2e_go_single that skips the job when the workflow_dispatch suites input contains none of {e2e-single, e2e-cert, e2e-lb}. Push events and the default suites list still trigger it, so plain PR runs are unaffected. Avoids ~15min of wasted tofu provision+bootstrap+teardown when only the runner-resident reboot suite is requested. * fix(e2e/reboot): bump post-reboot LB recovery timeout to 5min Pre-reboot the test creates the LB AFTER apps are already running, so HAProxy starts probing against an already-serving responder — ~75s to healthy. Post-reboot the daemon's recovery scan launches app VMs and the ALB sys.micro VM simultaneously, so health checks start probing within seconds of app boot beginning and both targets honestly report unhealthy until the responder unit finishes starting. Cloud-init's per-instance semaphore means no shortcut path on relaunch, so guest second-boot can be slower than first boot. Bash run-reboot-e2e.sh hid this behind stale 'healthy' state read from the daemon's pre-reboot TG cache — fixed via ResetTargetHealthOnStartup (mulga-siv-119), which made the test honestly wait on real lb-agent reports. The 180s bash inherited turned out to be wishful thinking backed by the cache bug, not measured recovery time. 5min matches the actual post-reboot eventually-consistent window observed on run 26134156355. Refs mulga-siv-118. * feat(e2e/reboot): ssh into guest VMs on failure for HTTP responder diag Post-reboot failures bottom out at "0/2 healthy" with no insight into whether the responder unit died, port 80 isn't bound, or the OVN path is broken. Capture the keypair material from CreateKeyPair so OnFailure can hop host -> qemu hostfwd:22 -> guest and dump systemctl status, port listeners, curl localhost, and the responder journal per instance. * fix(e2e/reboot): probe app instances via public IP (no QEMU hostfwd) Previous hostfwd-SSH approach didn't apply: app VMs use pure OVN tap interfaces (qemu argv has no hostfwd=tcp:*:*-:22), so the regex never matched. Replace with host-side probes against the IGW-NAT'd public IP: ping, /dev/tcp connect, and curl. Combined with the qemu ps check and ovn-nbctl list nat dump, this localises post-reboot failures to one of: VM not running, OVN DNAT missing, responder down, or sys.micro-only reachability (path asymmetry). * feat(e2e/reboot): probe ALB + dump host OVN/OVS state on failure Last run showed apps unreachable from host post-reboot (ping 100% loss, tcp refused) even though OVN dnat_snat rules looked correct. Need to distinguish gateway/ext-bridge breakage from app-side breakage: add a parallel ALB-public-IP probe (works pre-reboot, so any post-reboot failure isolates the regression) plus a full host net-state dump (ip route, ip neigh, ovn-nbctl show, ovs-vsctl show, br-int port list). * feat(e2e/reboot): dump qemu serial console per app on failure /run/spinifex/console-i-<id>.log captures cloud-init, kernel boot, systemd, anything the guest emitted to ttyS0. Tail to 256KiB (~4x a typical Linux boot) so the latest boot lands in full plus the tail of the prior. Surfaces guest-side boot failures (kernel panic, fs corruption, cloud-init OOM, etc.) that are invisible to host-side network probes. * feat(e2e/reboot): SSH into guest VMs via IGW for responder root-cause Last run nailed the symptom: ALB returns 503, app public IPs answer TCP with RST (port 80 not bound), so reboot-e2e-httpd.service isn't running on relaunched VMs. To find out why (unit never installed? installed but not enabled? failed to start?), open tcp/22 ingress on the shared SG and add a guest-SSH diagnostic via the IGW NAT path. The diagnostic dumps the unit state, port listeners, /var/lib/httpd, cloud-init result.json, cloud-init logs, and the instance-id semaphore directory. * test(e2e/reboot): always run guest SSH diag in phase 8 Last run passed after adding tcp/22 SG ingress. Two possibilities to distinguish: (a) responder was running all along but blocked at the OVN/SG layer, (b) responder wasn't running and HC was satisfied by some other path. Capture in-guest unit + cloud-init state on every run so we can confirm which without needing the test to fail. * refactor(e2e/single): migrate VPC group to EnsureDefaultVPC [skip ci] Bead 3a of mulga-siv-120 (e2e-single-fixture-migration). Replaces the "discover default VPC + cache on Fixture" pattern in vpc_test.go, natgw_test.go, securitygroup_test.go with harness.EnsureDefaultVPC. - Adds Harness *harness.Fixture field on single.Fixture; constructed in TestSingleNode alongside AWSClient. Parent test for memoized Ensure* resources is TestSingleNode itself. - Removes fix.DefaultVPCID / fix.DefaultSGID / fix.DefaultSubnetID reads in the three target files. Fields stay on the struct until 3e+3h migrate the instance / datapath / umbrella sites. - Custom VPC graphs created by phase8b (10.99.0.0/16) and the bastion graph in phase8d remain inline — those are the resource lifecycle assertions, not prereqs. CI gate skipped per dev request — full single-node suite verifies when upstream main is unbroken. * refactor(e2e/single): migrate Phase 3 key pair to EnsureKeyPair [skip ci] Bead 3c of mulga-siv-120 (e2e-single-fixture-migration). Phase 3 now calls harness.EnsureKeyPair for the primary key pair (Phase 5+ prereq). The Ensure* path owns the create assertion (PEM shape), memoizes across phases, and registers TestSingleNode-scoped cleanup — replacing the former TODO(stage-G) deferred-cleanup marker. - Drops hardcoded "test-key-1" name; EnsureKeyPair uses scratch-suffixed name so parallel CI runs don't collide on the namespace-flat AWS key pair name. - fix.KeyName / fix.KeyPath still assigned for downstream Phase 5+ readers. Per 3e migration plan, those reads move to in-test EnsureKeyPair calls and the Fixture fields drop with 3h. - Import / describe / delete dance for test-key-2 stays inline — it is the lifecycle assertion for the import path, not a prereq. * refactor(e2e/single): migrate Phase 4 + 5e AMI paths to EnsureAMI [skip ci] Bead 3d of mulga-siv-120 (e2e-single-fixture-migration). - Phase 4 (image_test.go): discover catalog AMI by canonical name, then route through harness.EnsureAMI({Existing}) so the state-available wait + memoization live with the fixture. - Phase 5e (createimage_test.go): replace inline CreateImage block with harness.EnsureAMI({CreateFrom}). The custom AMI build now memoizes, registers DeregisterImage cleanup at TestSingleNode exit, and reuses the harness state poll. Harness fix: extend AMICreateSpec with NoReboot bool — Phase 5e bakes the AMI while leaving the source instance running for Phase 6/7 lifecycle assertions; Bead 2 missed this option. fix.AMIID / fix.CustomAMIID / fix.CustomAMISnapID stay on Fixture for downstream Phase 5/6/7 + Phase 9 cleanup. Those reads migrate with 3e/3h. * refactor(e2e/single): migrate Phase 5 instance lifecycle to EnsureInstance [skip ci] Bead 3e of e2e-single-fixture-migration. - phase5_LaunchInstance: provision via EnsureInstance + re-describe for BDM/root-volume extraction; default VPC via EnsureDefaultVPC. - phase8e_SGToSGDatapath: replace inline DiscoverDefaultVPC with EnsureDefaultVPC. createSG / runSGEInstance take vpcID/subnetID arguments so phase 8e no longer depends on fix.Default* fields. - runSGEInstance stays direct RunInstances: phase 8e VMs are test subjects (need fresh ENI/private-IP per run), not memoized prereqs. - harness.EnsureInstance memo key now includes InstanceType, SGID, and UserData hash — a second call with a different SG would have silently returned the first instance ID (Bead 2 bug). * refactor(e2e/single): migrate Phase 5b/5c volume + snapshot to Ensure* [skip ci] Bead 3f of e2e-single-fixture-migration. - phase5b_VolumeLifecycle: create+wait via EnsureVolume; subsequent modify/attach/detach/delete stay inline (test subject lifecycle). - phase5c_SnapshotLifecycle: create+wait via EnsureSnapshot. Drop the create-response State/Progress shape assertions — after EnsureSnapshot the snapshot is already "completed", and the describe block below re-asserts VolumeId / VolumeSize / Description. - harness.EnsureSnapshot now takes SnapshotSpec{VolumeID, Description} so callers can pin a description (snapshot_test asserts on it). Memo key includes description hash so distinct-description calls do not silently collide on the volumeID prefix. * refactor(e2e/single): isolate IAM stage E via parent-scoped cleanup [skip ci] Bead 3g of e2e-single-fixture-migration. - harness.Fixture: add RegisterCleanup helper that wraps fx.parent.Cleanup so test code can register parent-scoped teardown for resources created outside ensureOnce (IAM users / policies threaded across phase subtests). Registering on the calling subtest would tear down before the next dependent phase ran. - phaseIAM1_UserCRUD: register parent-cleanup for alice/bob/charlie. - phaseIAM4_PolicyCRUD: register parent-cleanup for the 5 test policies. - IAM stage no longer relies solely on phaseIAM7 to clean up after a mid-stage failure; phase 7 still runs and now serves as an idempotent post-cleanup assertion. No new Ensure* helper: IAM resources have no memoization story (users and policies are test subjects, secrets only returned once on create). IAM phases also have no EC2 prereqs, so EnsureDefaultVPC / EnsureInstance are not applicable here. * refactor(e2e/single): drop dead Fixture fields after Ensure* migration [skip ci] First half of bead 3h. Removes per-phase Fixture fields with zero remaining readers after bead 3a-3g: - DefaultVPCID / DefaultSGID / DefaultSubnetID: superseded by harness.EnsureDefaultVPC (3a wired phase8b/8c/8d/5f; 3e wired phase5 + phase8e). - OVNAvailable: never assigned anywhere — leftover from the bash port. Phase fields still in use (AMIID, InstanceID, IAM*, etc.) stay until the umbrella TestSingleNode is replaced with top-level Test* + package- level fixture singleton in the second half of 3h. * refactor(e2e): add process-mode Fixture + TestMain singleton infra [skip ci] Second half of bead 3h. Lands the infrastructure required to dismantle the TestSingleNode umbrella in a follow-up bead, without changing the umbrella body today. harness.Fixture changes: - NewProcessFixture(aws) builds a Fixture with no *testing.T parent. Cleanup callbacks queue on an internal LIFO drained by Close() — intended to be invoked from TestMain after m.Run(). Lets a singleton fixture span every top-level Test* in a package. - Existing NewFixture(t, aws) unchanged; cleanups still route via t.Cleanup. registerCleanup() and logf() helpers fork on parent==nil so ensureOnce and RegisterCleanup work in either mode. - TestProcessFixture_CloseRunsCleanupsLIFO covers Close ordering, one-shot semantics, and the no-op second-Close path. single/main_test.go additions: - TestMain owns process-level lifecycle: drains pkgFix.Harness.Close() and removes pkgFix.TmpDir after m.Run(). - requireSingleNodeFixture(t) returns the package-scoped singleton, building it lazily on first call. Skips when SPINIFEX_E2E is unset or mode != single; fails on construction error. Currently unreferenced — TestSingleNode keeps its own per-test Fixture for now. The helper exists so the follow-up bead (per-phase top-level Test* migration) has a stable API to call from day one. Deferred to follow-up bead: extract Phase 4 AMI-name discovery + Phase 2 InstanceType/AZ selection into harness helpers so each phase function can self-bootstrap, then promote phase subtests to top-level Test* and delete the TestSingleNode umbrella. * fix(e2e/harness): EnsureDefaultVPC filter name is-default not isDefault Daemon rejects the camelCase `isDefault` form with 400 InvalidParameterValue. AWS CLI / DiscoverDefaultVPC use `is-default`. Surfaced as Phase 5_LaunchInstance fatal in CI run 26140908505 — cascade-failed every downstream phase that needed fix.InstanceID. * fix(e2e/single): phase8 8j/8k probe fix.KeyName not hardcoded test-key-1 Bead 3c migrated phase 3 to harness.EnsureKeyPair which produces a scratch-suffixed name (e2e-key-<random>), not the literal "test-key-1" the bash port used. 8j_CreateKeyPairDuplicate / 8k_ImportKeyPairDuplicate kept the old name and so created a fresh key pair instead of triggering InvalidKeyPair.Duplicate — surfaced as the only two FAILs in CI run 26143227369 (everything else green after the EnsureDefaultVPC fix). Read fix.KeyName (set by phase3_KeyPairs) so the probe hits the staged primary key. Header comments updated to match. make preflight diff-coverage against origin/dev fails at 57.9% due to unrelated commits inherited from the recent rebase onto feat/e2e-go-harness; against the fork base the diff is 0 instrumentable lines (test-only change). * refactor(e2e/single): dismantle TestSingleNode umbrella, per-phase self-bootstrap mulga-siv-121. Each phase from run-e2e.sh now runs as a top-level Test* in tests/e2e/single, runnable in isolation via `go test -run TestX`. Highlights: * TestSingleNode umbrella removed. Phase ordering dropped — each Test* self-bootstraps prerequisites through harness.Ensure*/harness.Discover* + the new package-local need*/iamEnsure* helpers. * Fixture shrunk to {Env, AWS, Harness, Artifacts, TmpDir, PoolMode}. Every per-phase resource ID now lives on harness.Fixture via EnsureKeyPair/EnsureAMI/EnsureDefaultVPC/EnsureInstance/etc. * TestMain owns the package singleton Fixture; harness.Fixture.Close drains every cleanup at process exit. phase9 + phase9a (in-line teardown) deleted; phase9b retained as TestFinalClusterStats. * IAM Test* share state via sync.Once-gated helpers in iam_helpers_test.go (alice/bob/charlie users + keys + admin account). * New harness/discover.go: DiscoverDefaultAZ, DiscoverNanoInstanceType, DiscoverUbuntuAMI — memoized on the Fixture and routed through EnsureAMI so any Test* can resolve them without phase coupling. * New tests_test.go: descriptive top-level Test* wrappers (TestVPCSubnetE2E, TestKeyPairs, TestSnapshotLifecycle, …) that delegate to phaseN_X(t, requireSingleNodeFixture(t)). * test(e2e/reboot): SSH as ec2-user, dump unit + network-online state spinifex's cloud-init template hardcodes Username="ec2-user" (handlers/ec2/instance/service_impl.go:1218) regardless of AMI, so even on Ubuntu the SSH-injectable account is ec2-user, not ubuntu. Switch the diag SSH-in to ec2-user so we actually get in (was "Permission denied (publickey)"). Add diagnostics to distinguish the two leading post-reboot hypotheses: (a) viperblock write loss = unit file gone vs (b) network-online.target never fires = unit stuck on After=network-online.target. New dumps: unit file content + enable symlink, network-online.target status, networkctl, ip addr / route, prior-boot journal grep for the responder. * fix(e2e/single): resolve admin account ID before TestIAM7 deletes alice iamEnsureAdminAccountID probes via GetUser(alice). TestIAM7_Cleanup deleted alice/bob/charlie first, then called the helper at line 808 — NoSuchEntity if IAM7 was the first IAM test to run (no memoized account ID). Move the call above the deletion loop.	2026-05-20 21:04:13 +10:00
Julian Sommer	97701737cb	feat: describe instance status (#221)	2026-05-13 15:55:20 +10:00
Josh Siviour	45555dde49	test(ddil): flip Scenarios A/B/C to real assertions (#205) ... * test(ddil): flip Scenarios A/B/C to real assertions Replaces the t.Skip stubs landed in #204 with the assertions described in docs/development/improvements/ddil-e2e-test-harness.md §3: - A NATSKill — KillNATS on node3, assert ModeStandalone within 30s, /local/instances still serves on the isolated node, majority /health on node1+node2 stays up, witness counters on all three nodes keep advancing, snapshot survives reconnect. - B DaemonRestartWithoutNATS — KillNATS + RestartDaemonOnly, assert ModeStandalone within 30s of restart, recovered instances present with live QEMU PIDs, cluster rejoin after StartNATS. Strict pre/post PID equality dropped — see file comment; recovery + a non-zero PID give the same regression coverage without coupling to the daemon unit's KillMode. - C CleanPartition — PartitionNode peer-IP DROP, majority daemons keep serving /health and /local/instances, isolated node3 reports ModeStandalone with /local/* responsive, witness counter on the partitioned side keeps advancing, heal converges and snapshot is preserved. Supporting changes: - harness/daemon_client.go — daemonPort 8443→4432; 8443 is predastore, the deployed daemon ClusterManager binds [nodes.<id>.daemon].host. The Phase-1 stub used a placeholder that would have blocked any /local/* assertion against the real cluster. - harness/witness.go — export WitnessVM.Terminate so scenarios can clean up their counter VMs from t.Cleanup. - scenarios/main_test.go — requireLiveCluster + launchWitnesses helpers to keep the three flipped scenarios tight. - TEST_COVERAGE.md — A/B/C status flipped SKIPPED → ENABLED; TestCoverageDrift continues to enforce alignment with the package. Validation: built with -tags=e2e, dry-run scenarios pass, coverage drift passes, preflight clean. workflow_dispatch e2e-ddil.yml against tofu cluster is the next step. * fix(ddil): countDropRules — single iptables -S call + filter Phase 1's countDropRules ran `iptables -S INPUT -S OUTPUT` to grep DROPs across both chains, but iptables -S takes a single optional chain — the second -S errors to stderr, the combined SSH buffer poisons the grep input, and Atoi fails ("ip" is what we saw on tofu-cluster). Hit by TestHarnessSmoke and every flipped scenario (A/B/C) via AssertCleanState. Switch to one `iptables -S` (lists every chain in iptables-save format), filter for INPUT/OUTPUT lines, count `-j DROP`. `2>/dev/null` silences sudo/iptables stderr so a noisy MOTD or warning cannot leak into stdin. Error path now also surfaces the raw SSH output for the next time this breaks somewhere unexpected. * ci(e2e-ddil): export spinifex AWS creds for witness SDK The witness EC2 client (harness/witness.go NewWitness) calls session.NewSession with no static credentials, so the AWS SDK falls through its default chain — env vars → shared profile → IAM role — and on the CI runner none of those exist. Result on the live cluster: "NoCredentialProviders: no valid providers in chain. Deprecated." launches every Scenario A/B/C attempt straight to t.Fatal at launchWitnesses. spx admin init writes [spinifex] aws_access_key_id / aws_secret_access_key to the operator's ~/.aws/credentials on node1 during bootstrap. Pull them over SSH and export as AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY so the SDK's env-var step resolves before it gets to the empty profile/IAM checks. Hard-fail the step if either field is missing — better than silently watching every scenario t.Fatal on the same line. * fix(ddil): CI green-up — instance-type discovery, gotestsum, EC2 envelope Three fixes to take the DDIL E2E suite from "scenarios flipped" to actually green on the tofu cluster. mulga-siv-56. 1. Witness auto-discovers the smallest registered instance type (DescribeInstanceTypes, sort by memory then vCPUs). The hard-coded t2.micro default returned InvalidInstanceType on every Scenario A/B/C launch because the cluster doesn't register that type. DDIL_WITNESS_- INSTANCE_TYPE stays as an operator override. 2. e2e-ddil.yml uses gotestsum so the Actions UI shows per-test PASS/FAIL/ SKIP lines instead of a JSON wall. results.json artifact preserved for future flake-rate dashboarding. 3. GenerateEC2ErrorResponse emits the real EC2 query-API envelope <Response><Errors><Error>... instead of the IAM-shaped <ErrorResponse>. aws-sdk-go v1's ec2query handler rejects the latter with SerializationError and drops the code, leaving Go callers without an awserr.Code() to inspect. New SDK round-trip test in gateway_test.go guards the regression. * ci(e2e-ddil): install gotestsum from prebuilt tarball `go install gotest.tools/gotestsum@v1.12.0` failed on the self-hosted runner because v1.12.0 pulls golang.org/x/tools v0.19.0, whose tokeninternal static-assert pegs a narrow Go window (~1.22). Both older and newer runner Go versions trip `invalid array length -delta * delta`. A prebuilt linux_amd64 binary sidesteps toolchain coupling entirely without changing the gotestsum version pin. * ci(e2e-ddil): collect cluster logs on failure Witness `findHost` is currently failing with "could not locate QEMU host for i-XXX across 3 nodes" — daemon reports running but no QEMU fork is visible on any node. Cluster gets destroyed during teardown, so logs have to be pulled before that runs. Adds a failure-only step that ssh's each node and bundles journalctl spinifex-*, process listings, and state dirs into the ddil-results artifact for offline triage. * fix(ddil): support tap-mode QEMU networking in witness harness findHost no longer extracts hostfwd port — production launches go through auto-allocated ENI on the OVS bridge and emit `-netdev tap,...`, never hostfwd. Witness was timing out across all 3 nodes. Discover guest IP via DescribeInstances.PrivateIpAddress (populated by PrepareRunInstances on ENI auto-create) and tunnel SSH through the host node to guestIP:22. Host routes to the tap bridge subnet, so the relay works without runner-side bridge routing. WitnessVM.SSHPort → WitnessVM.GuestIP. hostfwdPortRE removed. Closes mulga-siv-57. * fix(ddil): SSH witness via public IP, drop unreachable tunnel path tofu-cluster bootstraps with --external-mode=pool which forces dev_networking=false → no QEMU hostfwd dev NIC and no host route to the ENI's PrivateIpAddress (tap on OVS br-int, 172.31.0.0/16 invisible to host kernel). Previous tunnel-via-host fix could find the host but couldn't dial the guest. Switch to the path shell e2e already uses: default subnet has MapPublicIpOnLaunch=true, daemon auto-allocates an EIP from the external pool, and the runner reaches that IP via the same WAN subnet as the cluster nodes. No host relay required, survives Scenario C (partition only DROPs peer cluster IPs, not the runner's source IP). Workflow now scp's ~/.ssh/spinifex-key from the primary node (the counterpart to the public material bootstrap-install.sh imports) and exports DDIL_GUEST_SSH_KEY so the harness can auth as ubuntu against cloud-init's authorized_keys. WitnessVM.GuestIP → WitnessVM.PublicIP. hostSigner / dialHost removed. Continues mulga-siv-57. * fix(ddil): tunnel guest SSH through host node EIP from MapPublicIpOnLaunch sits on the cluster's external pool subnet, which is OVN-routed — reachable from any cluster node, not from the off-cluster runner sitting on the DDIL_NODES WAN subnet. Restore the host-jump pattern: SSH the hosting node (already known from findHost), dial publicIP:22 over that connection, run a guest SSH handshake over the tunnel. Symmetric with how shell e2e reaches guests from inside a node. Continues mulga-siv-57. * fix(ddil): wait for witness sshd before baseline read Previous run reached the guest EIP via the host tunnel but hit 'Connection refused' — sshd hadn't bound yet. waitForRunning only checks EC2 state, which flips on QEMU start, not cloud-init completion. The OVN dnat_and_snat rule for the auto-allocated EIP is also written asynchronously, so the tunnel can land before either layer is ready. awaitGuestSSH polls the host-tunnelled SSH handshake to publicIP:22 for up to 90s, retrying on transient transport errors (refused/reset/EOF/ timeout/no-route) and failing fast on auth errors so a bad key surfaces immediately. Called once after resolvePublicIP, before the baseline counter read. Continues mulga-siv-57. * fix(ddil): case-insensitive transient SSH error match awaitGuestSSH's retry matcher used lowercase substrings, but x/crypto/ssh emits "ssh: rejected: connect failed (Connection refused)" with capital C — strings.Contains returned false and the helper bailed on first attempt instead of polling. Last run failed in 49s, not the 90s budget. Lowercase the error before matching, and add "connect failed" as the canonical signal for "jump host dial through tunnel failed". Continues mulga-siv-57. * fix(ddil): default guest SSH user to ec2-user Shell E2E SSHes the same ami-ubuntu-* images as ec2-user (tests/e2e/run-multinode-e2e.sh:650). DDIL harness defaulted to ubuntu and hit 'unable to authenticate, attempted methods [none publickey]' — authorized_keys is provisioned under the AMI default account, which is ec2-user, not ubuntu. Continues mulga-siv-57. * fix(ddil): wait for witness-counter.service to seed /var/lib/counter awaitGuestSSH passed at sshd-up, but cloud-init's runcmd that enables witness-counter.service runs in the final cloud-init stage — after sshd binds. Baseline read landed in the window where sshd accepts but ExecStartPre hasn't created /var/lib/counter: cat: /var/lib/counter: No such file or directory Collapse awaitGuestSSH + readCounter into awaitBaseline that polls the full path (tunnel + SSH handshake + cat) and retries on the transport errors plus "no such file or directory". Single 2-minute budget covers cloud-init final stage on tofu-cluster hardware. Continues mulga-siv-57. * fix(systemd): drop Wants=spinifex-nats from daemon unit Wants= reactivates a stopped unit on `systemctl restart`, so a deliberately killed spinifex-nats was getting pulled back up the moment the daemon restarted — masking the DDIL Tier 1 standalone-startup path and breaking the Scenario B assertion (mulga-siv-58). Cold-boot coverage stays intact: spinifex.target's Wants= activates every spinifex unit on boot. After= remains for ordering and is non-blocking when NATS is absent. scripts/test-ddil-phase1.sh hit the same trap and worked around it with a runtime drop-in plus sentinel file; the fix here makes that workaround unnecessary for any caller that restarts the daemon while NATS is stopped. * feat(daemon): peer-health probe drives standalone mode on partition Mode() is now derived from two atomic signals — natsConnected (NATS client TCP link, flipped by the existing disconnect/reconnect callbacks) and peersReachable (any peer daemon's /health responded in the most recent probe cycle). Reports cluster iff both are up; standalone otherwise. The peer-health probe is the missing signal for DDIL Scenario C: an iptables DROP between peer IPs blocks NATS cluster-routing traffic but leaves each node's local NATS client TCP connection healthy, so nats.Conn.IsConnected() keeps returning true and onNATSDisconnect never fires. The probe dials each peer's /health over the cluster network with a 2s timeout on a 2s ticker; worst-case detection of total partition is ~6s on a 3-node cluster, well inside the 30s Scenario C budget. Single-node clusters keep peersReachable permanently true (nothing to probe) so the derived mode flips to cluster as soon as the local NATS connect succeeds. The probe runs from startLocal so the partition signal exists even before (or in the absence of) a successful NATS connect. It does not touch NATS or JetStream, so the §1e Tier 1 invariant asserted by assertNoClusterServicesInitialised is unaffected. Tests: - TestMode_RequiresBothSignals — every nats×peers combination. - TestProbePeersOnce_FlipsOnHealthChange — deterministic probe-only test. - TestProbePeerHealth_RejectsBadAddr — fails closed on missing addresses. - TestMonitorPeerReachability_EndToEnd — ticker-driven false→true. Closes mulga-siv-58. * fix(daemon): peer probe falls back to local daemon port `spx admin init` only renders `[nodes.<self>.daemon]` in spinifex.toml; remote-node blocks carry just `host = "<ip>"` with no daemon sub-table. So `peer.Daemon.Host` was always empty in a deployed cluster and the probe failed closed, leaving peersReachable permanently false. Tofu-cluster run 25665158964 surfaced this as a Scenario A regression: after StartNATS, natsConnected flipped true on reconnect but Mode() stayed standalone because the derived signal required peersReachable. `a_nats_kill_test.go:75: node3 did not return to cluster mode … (still reporting another mode)`. Add peerDaemonPort(p) that prefers an explicit peer Daemon.Host port when present and otherwise reuses the local daemon's port (clusters are symmetric — every node binds the same port the template renders). New TestPeerDaemonPort_FallbackToSelf covers both branches plus the no-port- available case; TestProbePeerHealth_RejectsBadAddr is updated to clear self config so it still asserts the fail-closed path. * fix(daemon): SIGTERM preserves VMs + KillMode=process on the unit DDIL Scenario B asserts that `systemctl restart spinifex-daemon` preserves every guest VM so the new daemon can reattach via the persisted local state file. The previous setupShutdown SIGTERM handler called vmMgr.StopAll(), gracefully QMP-powering-down every guest before the daemon exited — ~90s wall-clock and zero surviving QEMU on the other side. CI run 25666369604 surfaced this as `recovered instance i-... has no live QEMU PID`. Two changes restore the Tier 1 contract: - setupShutdown no longer calls StopAll. It still flips d.shuttingDown so crash/restart handlers bail out, persists state, unsubscribes from NATS, and exits. VMs are only stopped when coordinated cluster shutdown drains them — handleShutdownDrain still calls StopAll and flips the flag before the SIGTERM path runs. Operator-visible behaviour change: `systemctl stop spinifex-daemon` leaves guest VMs running. To stop VMs, use the cluster-shutdown topics or terminate via the AWS API. - The systemd unit gains `KillMode=process` so systemd only signals the daemon's main PID. With the default control-group mode, QEMU children were killed when systemd tore down the daemon's cgroup — defeating the SIGTERM change above. `TimeoutStopSec=15` bounds the new fast-exit grace window. Also nil-guards natsConn.Close in the shutdown path: when a daemon restarts while NATS is unreachable (Scenario B), natsConn is nil and the close would panic. Mulga-siv-58 (reopened). * fix(vm): QEMU stdout/stderr to shared log file (SIGPIPE survives daemon restart) QEMU stdout/stderr were piped to daemon-side scanner goroutines. When the daemon exits (e.g. `systemctl restart spinifex-daemon`), the pipe read ends close; QEMU's next write triggers SIGPIPE and the guest dies — defeating DDIL Scenario B's "daemon restart preserves running instances" guarantee. Open `<runtimeDir>/qemu.log` and assign it to cmd.Stdout/cmd.Stderr instead. The FD is dup'd into QEMU at fork, so closing it on the daemon side after Start() leaves QEMU writing into the file indefinitely. POSIX O_APPEND makes concurrent QEMU writes interleave atomically per line, so a single shared log is safe and avoids per-instance clutter in RuntimeDir. Trade-off: QEMU lines no longer land in journalctl -u spinifex-daemon. Operators inspect <runtimeDir>/qemu.log directly (path logged at instance start under the qemu_log field). mulga-siv-58 (DDIL Scenario B iteration 4) * fix(daemon): pin XDG_RUNTIME_DIR=/run/spinifex; revert SIGPIPE workaround Root cause of Scenario B's "no live QEMU PID" was not SIGPIPE — QEMU ignores SIGPIPE (SIG_IGN set early in QEMU init). It was systemd's `PrivateTmp=yes` on the daemon unit combined with `utils.pidPath()` falling through to `os.TempDir()` (=/tmp): - spinifex-daemon HOME=/var/lib/spinifex/spinifex, so `$HOME/spinifex` (=/var/lib/spinifex/spinifex/spinifex) does not exist and pidPath() returns /tmp. - Every daemon start gets a fresh private /tmp namespace. QEMU stays in the OLD namespace after the daemon dies (mount namespaces aren't changed by reparenting). The new daemon's /tmp is empty. - `/local/instances` reads `<pidPath>/<id>.pid`; in the new namespace the file doesn't exist, so PID=0. Fix: set `Environment=XDG_RUNTIME_DIR=/run/spinifex` on the daemon unit. /run/spinifex is host-wide tmpfs (declared via tmpfiles.d, already in ReadWritePaths) and survives daemon restart. pidPath() short-circuits on XDG_RUNTIME_DIR, so QEMU pidfiles, QMP sockets, and RuntimeDir() all resolve to the stable path. Both daemon incarnations and QEMU children see the same files. Revert iteration 4's pipe→file QEMU log change (commit 8a9a6f9f). SIGPIPE was never killing QEMU, and the file-redirect dropped QEMU lines from `journalctl -u spinifex-daemon` for no gain. mulga-siv-58 (DDIL Scenario B iteration 5) * feat(daemon): siv-60 heal reconcile — phase 1 (probe-heal trigger) (#211) * feat(daemon): wire reconcileOnHeal on peer-probe heal edge (siv-60 phase 1) Scenario C heal triggered zero resync: the local NATS client stayed connected to its loopback server throughout the partition, so onNATSReconnect never fired. The peer-probe false→true transition was the only heal signal but had no resync action wired to it. Adds reconcileOnHeal, called from both edges and coalesced via a reconciling atomic.Bool so the two callbacks landing on the same heal don't double-push. For phase 1 the body is the existing WriteState (re-asserts our per-node row in the instance-state KV). Phase 2 will extend it to pull KV → merge → push for the stopped/terminated buckets and re-fire per-node bootstrap. Plan: docs/development/feature/ddil-heal-reconcile.md * feat(daemon): siv-60 phase 2 — idempotent default-VPC bootstrap re-fire on heal Calls ensureDefaultVPCInfrastructure() from reconcileOnHeal after the WriteState push. Idempotent: each branch (default VPC lookup, IGW attach, default-SG create, default route insert) checks for existing infrastructure before creating. Covers the failure mode where startCluster ran during partition and the initial creates returned errors against an unreachable NATS — without a heal-time retry the cluster would be stuck without a default VPC until the next daemon restart. Scope note: the bead's "pull KV → merge → push" assumed the daemon cached peer-node running state, stopped instances, terminated instances, and VPC/IGW/SG state locally. The codebase audit (see plan doc) confirms it doesn't — every consumer reads its KV bucket fresh per request, and vmMgr only tracks VMs running on this node. The bootstrap re-fire is the one heal-time action that has a real in-memory target. Watchdog sweep is unnecessary because vm.Manager.CrashHandler is event-driven from cmd.Wait(). * fix(daemon): drop ensureDefaultVPCInfrastructure from reconcileOnHeal (siv-60 follow-up) (#212) * fix(daemon): drop ensureDefaultVPCInfrastructure from reconcileOnHeal DDIL Scenario C regressed at the witness-SSH step after siv-60 landed in #211. Root cause: the first peer-probe tick on multi-node startup flips peersReachable false→true (its zero value is false), so reconcileOnHeal fires concurrently with startCluster's own ensureDefaultVPCInfrastructure call. The Describe→Create→Attach sequence is not cluster-wide singleton, so multiple daemons + double calls per daemon raced and produced duplicate IGW attachments, corrupting default-VPC routing — which broke the witness VM's tunnel SSH path once Scenario C's partition was installed. Phase 1 (the heal trigger + coalescing) is the meaningful fix and stays. Phase 2's bootstrap re-fire needs leader election or a debounce against the startup-time path before being re-introduced; leaving an inline NOTE so the next attempt has the context. Failed run: actions/runs/25706585637 (Scenario C, attempts 1+2). Last green DDIL run: 25671044714 (pre-merge, SHA 40e3d847). * fix(daemon): init peersReachable=true to suppress startup heal edge DDIL Scenario C kept flaking at attempt-1 even after dropping the bootstrap re-fire. Remaining trigger: peersReachable starts as the zero value (false) on multi-node clusters, and the first probePeersOnce tick flips it false→true. That counts as a "heal" edge under siv-60 phase 1, so reconcileOnHeal fires at boot — concurrent with startCluster's bootstrap — perturbing timing in a way that the harness's Scenario C exposes (witness sshd handshake EOF mid-test). Initialise peersReachable optimistically. Mode() also requires natsConnected (starts false), so the optimistic peer init can never falsely report cluster mode — but it does eliminate the spurious startup edge. Real partition→heal transitions still fire the trigger exactly as designed: true→false on partition, false→true on heal. --------- Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>	2026-05-12 15:56:17 +10:00
Julian Sommer	bb08335831	fix: explicit workflow permission	2026-05-12 11:57:03 +10:00
Julian Sommer	84278b794f	chore: reintroduce dev branch in CI + scripts ... - workflows: add 'dev' to push/PR triggers (spinifex.yml, ui.yml, e2e.yml) - workflows: drop "Resolve Mulga Branch" fallback step; mulga is single-branch so checkout@v6 uses the default branch implicitly - dependabot: route gomod/github-actions/npm PRs to dev - diff-coverage.sh: pick origin/dev as base for feat branches, origin/main for dev - iac/proxmox-template: default git_branch to dev for cluster builds Refs docs/development/improvements/reintroduce-dev-branch.md	2026-04-30 11:50:13 +10:00
dependabot[bot]	f3cfee3189	chore: bump actions/upload-artifact in the github-actions group (#163)	2026-04-28 09:16:05 +10:00
Julian Sommer	c7d4040dd3	feat: nightly e2e (#155)	2026-04-24 12:25:03 +10:00
Josh Siviour	068aef5589	feat: DDIL E2E test harness (mulga-siv-5) (#160) ... * feat: scaffold DDIL E2E harness (cluster, SSH, daemon client) Phase 1 of the DDIL E2E test harness (mulga-siv-5.1). Adds the tests/e2e/ddil/ package tree with the three harness primitives required by every sibling scenario/fault bead: - harness/cluster.go — Cluster, Node, ClusterFromEnv, NodeFromEnv - harness/ssh.go — typed SSH transport backed by golang.org/x/crypto/ssh, interface so dry-run and tests can stub it - harness/daemon_client.go — thin HTTPS wrapper over /local/status, /local/instances (placeholder types until daemon-local-autonomy lands), plus /health for scaffolding validation Also adds doc.go and README.md (run instructions + env-var contract). All harness files carry //go:build e2e so default builds skip them. Plan: docs/development/improvements/ddil-e2e-test-harness.md §0, §1. * feat: add DDIL harness fault injection and link profiles Phase 2 of the DDIL E2E test harness (mulga-siv-5.2). Adds the fault injection and netem primitives consumed by Scenarios A–F: - harness/fault.go — PartitionNode (per-peer iptables DROP with orchestrator-SSH sanity check), HealNode, KillNATS, StartNATS, RestartDaemonOnly, WaitForMode (polls /local/status for typed DaemonMode enum) - harness/netem.go — ApplyNetem (tc qdisc replace, idempotent), ClearNetem (idempotent clear) - harness/profiles.go — LinkProfile struct plus LAN, WAN, LTEDegraded, SATCOM, HFData, Flapping constants (unvalidated approximations per plan §2 table; validation tracking lives in TEST_COVERAGE.md) All files carry //go:build e2e. Flapping profile is defined but ApplyNetem rejects it — flap orchestration lands with the scenarios that need it. Plan: docs/development/improvements/ddil-e2e-test-harness.md §1, §2. * feat: add DDIL harness witness VM helpers LaunchWitnessVM launches a counter VM via the cluster's AWS gateway, locates its QEMU host via ps, and captures a baseline counter reading. ReadCounter/AssertProgressed tunnel SSH through the host's QEMU hostfwd port to verify the workload progressed across a fault. README documents the new DDIL_WITNESS_* and DDIL_GUEST_SSH_* env vars. * feat: add DDIL harness snapshot and state assertions TakeSnapshot / AssertPreserved compare /local/instances pre and post fault, failing if IDs disappear or a running/pending instance regresses to a terminal state. LocalStateRevision / AssertMonotonic read the local state file revision from /local/status so scenarios can detect the daemon rewinding its persisted state. * feat: add DDIL harness reset, retry, and quarantine ResetAllNodes brings every node back to a clean baseline (iptables -F, tc netem clear on any interface carrying one, spinifex-nats and spinifex-daemon started) then waits for cluster health — daemon /health 200 plus NATS routez peer count matching len(nodes)-1, mirroring the shell suite's verify_nats_cluster predicate. AssertCleanState is a pre-scenario baseline check that recovers a dirty state via ResetAllNodes instead of failing outright, since a dirty baseline usually means a previous scenario's cleanup was skipped. Run wraps each scenario with quarantine handling (DDIL_QUARANTINED skip) and a single reset-and-retry on failure. First-attempt failures are not hidden; the retry is a diagnostic aid rather than a flake mask. bd: mulga-siv-5.5 * feat: DDIL scenarios skeleton (6x t.Skip + TestMain + coverage drift) Phase 1 of mulga-siv-5 (DDIL E2E harness): all six scenarios land as TestScenario<L>_... stubs that t.Skipf on their hardening-epic dependency. TestMain installs a SIGINT/SIGTERM trap that ResetAllNodes before exiting; DDIL_DRY_RUN=1 short-circuits cluster setup and logs the planned scenarios. TestCoverageDrift AST-parses sibling *_test.go files and reconciles them against the DDIL table in TEST_COVERAGE.md so Phase 2/3 flips cannot silently diverge doc from code. TestHarnessSmoke covers the AssertCleanState + ResetAllNodes round trip for the "full run on real cluster" acceptance bullet. * feat: add DDIL E2E CI workflow (workflow_dispatch only) Phase 1 CI wiring for mulga-siv-5. Mirrors e2e_real_multi's provision/bootstrap/teardown shape against the same ci-multi runner and tofu-cluster, but runs the Go harness on the runner (outside the cluster) so its SSH path survives the peer-traffic iptables rules that Scenario C installs on a cluster node. Triggers are workflow_dispatch only — nightly cron and on-main-merge come in mulga-siv-5.8 once ≥4 scenarios are asserting. results.json (go test -json) uploads as an artifact so flake-rate visibility can be built on gh run view + structured output, per the design doc.	2026-04-23 22:38:29 +10:00