23 Commits

Author	SHA1	Message	Date
Ben Duncan	8ab96b29a1	fix: replicate every JetStream KV bucket across the cluster (#1117) ... * fix: replicate every JetStream KV bucket across the cluster A KV bucket created without a replica count gets one replica, and one replica lives on one JetStream-chosen server recorded in no config. Losing, restarting or partitioning that server makes the bucket return "no responders available" from every node at once, so a single-node event becomes a cluster-wide outage. The reconciler leader-lease buckets were the worst case: acquiring a lease is a write, so a lease bucket without a quorum cannot be acquired by anyone, and every reconciler sharing it stops cluster-wide at exactly the moment a node has failed. The cluster's node count is now declared once, in config.LoadConfig, and read by the one helper every bucket creation goes through. Every process that creates a bucket loads the cluster config first, so no service can forget to pass it and no signature has to carry it. An undeclared size is an error at creation rather than a silent fall back to one replica. Existing buckets are raised rather than left behind: a service opening one raises it, the daemon sweeps them all after formation, and "spx admin kv replicas --repair" does it on demand. Raising reads the live stream config and changes only the replica count, and never lowers one. Removes the ~30 replicas/clusterSize parameters that only forwarded a number, and JetStreamManager.UpdateReplicas, which set len(nodes) with no clamp and would have failed outright on a cluster of six or more. Proved by a repo-wide invariant test pinning bucket creation to the one helper, a whole-cluster sweep in every multi-node nightly cell, and a new kvquorum suite (cell 33) that takes away the node leading the most buckets and requires the survivors to keep answering for all of them. * chore: name cell 33 in the nightly dispatch input * fix: a KV bucket that cannot be raised must still open The first version of the replica fix treated a raise it could not place as a hard error, and on a real three-node cluster that turned the fix into the outage it prevents. A node whose config named a fourth server wanted four replicas, NATS answered "no suitable peers for placement" for three, and the daemon retried for its full five-minute budget without finishing bootstrap. The bucket was present, quorate and serving on three nodes throughout. A config naming a node before that node serves is the ordinary middle of growing a cluster, so that is a state to tolerate rather than refuse. The line is now drawn between the two cases instead of around both. Creating a bucket below the cluster's replica count still fails, because that is the defect and a fallback there reintroduces it where it is hardest to notice. Failing to raise an existing bucket warns and hands the bucket back, and the raise is retried on every open besides the daemon sweep and --repair. This matters most in kvlease, which runs on every reconcile tick: refusing there would stop every reconciler sharing a lease bucket that was working. getOrCreateBucket now attaches before it creates. A create asking for more replicas than the cluster can place is refused before the server looks at the name, so with create-first "this bucket already exists" was an inference from which error came back, and the wrong one comes back on a single-node server. RaiseAllBucketReplicas attempts every bucket and joins the failures, and --repair keeps going too. Stopping at the first would leave the rest of a cluster short because one bucket could not be placed. Also adds nats.ServerName/NodeFromServerName beside the template that renders server_name, with a test tying them together. The kvquorum suite had mapped stream peers to nodes by comparing them to node names, which they are not: they are the node name behind a "spinifex-nats-" prefix, so nothing matched and the node-loss test could not choose a victim. * test: wait for the right NATS peer count after a node rejoins /routez reports a node's routes to its peers, not its own membership, so a fully formed three-node cluster is two peers per node. The kvquorum suite waited for three and spent its whole five-minute rejoin budget on a cluster that had already reformed. Every other caller passes len(Nodes)-1; the helper's doc comment now says so. The assertion the test exists for had already passed: node1 was leading 18 of 47 buckets when it was taken away, and node2 and node3 each still answered for all 47. * fix: replicate KV buckets across the largest odd count a cluster can place A bucket was replicated across min(nodes, 5) servers, which stores a fourth full copy on a four-node cluster and still survives exactly one loss — the same tolerance R3 gives for three quarters of the write amplification. A Raft majority needs an odd count, so the rule becomes the largest odd count the cluster can place, capped at five: 1, 3, 3, 5, 5. It is one expression with two entry points, so no caller can hold a stale copy of the policy and keep passing after the policy changes. Three retry loops could not tell a configuration fault from a cluster that had yet to form. An undeclared cluster size never becomes declared by waiting, so each of them spent its whole budget on one and then reported quorum as the cause. They now stop on a permanent error and say what is actually wrong. This is what the integration suite was hitting: it builds its gateway by hand and never loads a cluster config, so the size was never declared and the IAM retry sat in its sleep until the test timed out. Both ends are fixed. The audit reported configuration alone. A bucket at three replicas with one peer offline and one lagging is one failure from unavailable and read as ok. BucketReport now carries the leader and a count of the peers JetStream itself calls current and reachable, quorum is measured against that count, and spx admin kv replicas exits 0, 1 or 2 so a deployment gate can tell a problem --repair fixes from one needing a human. Declaration is write-once: daemon.Start refuses a conflicting redeclaration because it is about to create buckets, and it declares before JetStream init rather than after it. config.LoadConfig warns instead, because it also reads config files that are not this process's own cluster. spx admin init --nodes 2 is refused — no replica count on two nodes both survives a loss and accepts writes — and single-node replication is asserted in the single e2e suite, so it runs in every nightly host permutation. * fix: withhold the vLLM serving image until its artifact is published The catalog offers ubuntu-26.04-vllm-serving-x86_64, and the qcow2 it names 404s on iso.mulgadc.com, so spx admin images import would fail for every operator who tried it. Its GPU siblings are published; this one never was. The entry moves to WithheldImages rather than being deleted, because the entry is the definition — the URL, the checksum and the two tags resolveServingAMI filters on. Publishing the artifact is then a move back, not a retype. Bedrock already handles the image not being registered: resolveServingAMI returns ErrServingAMINotFound, which is the honest answer while there is nothing to launch. The bedrock tag contract is still pinned, now against the map the definition lives in, and it additionally asserts the entry is not offered for import. The e2e reachability walk covers AvailableImages alone, so a withheld entry is out of it by construction. * fix: drop the cluster size from the call sites dev added Two constructors on dev still pass a replica count that no longer exists as a parameter: the cluster size is a declared process fact, so EKSServiceDeps has no ClusterSize field and NewIAMServiceImpl takes three arguments. The merge is textually clean and breaks the build, which is the shape of collision a signature removal always has. * docs: explain what an operator sees when a cluster server fails The replication section said how many copies a bucket has and stopped there, which leaves the question an operator actually has during an incident: one server is gone, what still works. It now covers the majority side carrying on, an isolated server refusing changes on purpose and why that is the behaviour you want, guests continuing to run while anything that has to be recorded does not, the risk of reading state from a server that is cut off, and a returning server catching up by itself. The surrounding prose reads from the operator's side rather than the implementation's: what you see and what to do, not which consensus protocol is underneath.	2026-09-28 12:34:55 -05:00
Ben Duncan	021f439bab	feat: recover instances automatically from a node that stopped heartbeating (#1111) ... * feat: recover instances automatically from a node that stopped heartbeating A node that dies hard takes its guests with it, and nothing brings them back until an operator brings that node back. Every daemon now runs the same small reconciler: it reads the instance records, selects desired-running instances whose owner is stale, and relaunches them here. The claim is a CAS on Status.LastNode and that is the whole of the coordination. Several survivors race one instance, exactly one revision wins, and the losers find a record naming a live owner and stop considering it. A claimant that dies mid-launch leaves the record naming a node that is itself stale, which is the same candidate the next pass derives again, so there is no lease to expire and nothing to reconcile by hand. Ownership is not permission to write. The volume leases still decide whether the guest can move, and VolumeMounter.Mount runs before startQEMU, so an attempt that loses the volume race has touched no tap, no OVS port and no OVN binding. A storage failure is not a host failure and wants the opposite response. A guest paused by werror=stop publishes Health.IOErrorSince, and a record carrying it is refused: the volumes are the same objects in the same store from anywhere, so relaunching moves the pause rather than ending it. A node whose own object store is not answering stands down entirely, which separates a store broken here from a store broken everywhere without either having to be diagnosed. A launch that does not complete backs this node off that instance, doubling to a cap, because nothing on the mount path can yet say which of the two it hit. Recovery is off by default and inert below three nodes. It also waits until it has seen its peers live, or five minutes, so the first node back from a coordinated shutdown does not recover guests from nodes that are still booting. * test: prove instance recovery moves a host failure and refuses a storage failure The e2e half of automatic instance recovery, asserted from both sides because a reconciler that cannot tell the two failures apart passes one and fails the other. InstanceAutoRecovery takes a node down and requires the guest back on exactly one survivor, identified by the qemu process rather than by what the API reports. InstanceRecoveryRefusesStorageFault stops predastore cluster-wide first, so the guest is paused by werror=stop when its node goes away, and requires that nothing moves. Also fixes the settle window, which re-evaluated every pass and so held a daemon back for its first five minutes over exactly the failure the reconciler exists to handle. It now latches once every peer has been seen live. harness.InstanceHostingNodes replaces the single-node lookup so a caller can assert there is exactly one host rather than assume it, and skip a node it has deliberately taken down. * fix: make automatic instance recovery a cluster-wide setting Recovery was read from the per-node Config, so it had to be written three times on a three-node cluster, once inside each [nodes.<name>] table. A top-level [recovery] table — the obvious thing to write, and what the e2e suite writes — was parsed into nothing and the daemon reported recovery disabled on every node with the setting plainly in the file. Cluster-wide is also the honest shape. A node with recovery off is not opting out of anything: recovery is what survivors do to a dead node's instances, so the setting belongs where there is one of it. * test: drive real I/O into the storage-fault case, and freeze the store rather than stop it Measured on a three-node cluster: predastore stopped cluster-wide for twelve minutes against idle guests paused none of them. Viperblock serves what it has cached and an idle guest issues nothing that reaches the object store, so the test asserted that nothing moved in a run where nothing was ever faulted. The victim now launches with cloud-init user-data that writes to its root volume continuously. User-data rather than SSH because a routed-NAT cell gives guests no public address, so a test that had to log in could not run on half of them. The fault is SIGSTOP rather than a unit stop, for the reason the storagefault suite already gives: a frozen process holds its connections open and answers nothing. Under that fault and that load the first BLOCK_IO_ERROR arrived 4m47s later, so the pause budget is now ten minutes rather than three. * test: give instance recovery its own suite and its own nightly cell Both tests take a node away and one of them freezes predastore cluster-wide, so anything sharing the environment reports that outage as its own failure — the same argument storagefault already makes for being its own package. They are also slow by nature: a dead owner has to be seen stale twice, a guest has to boot, and the storage case waits for a pause measured at 4m47s and then holds four minutes to prove nothing moved. Roughly twenty minutes on top of a permutation cell budgeted at thirty-five. Cell 31 is three-node and static, matching storagefault's, with a 75-minute job budget over a 60-minute suite timeout. The suite is eligible on multi-node environments and deliberately not in the nightly permutation set. * fix: give a guest long enough to open a volume the object store is slow to serve QEMU opens every drive before its monitor answers a greeting, so the QMP greeting deadline is really a deadline on opening the guest's volumes. At 30s it was shorter than a cold viperblock open costs on a cluster that has lost a node, where every object read is served degraded: measured at 35s with one node of three away. The launch killed the guest five seconds short of ready and then did it again on every retry, which is a recovery that can never complete. A network-backed drive now gets its own deadline for the same reason VFIO already had one — synchronous work before the monitor answers — and the instancerecovery suite is what found it. Also make the suite's predastore thaw best effort per node. It runs after a node has been taken away, where there is no process to signal, and a node it cannot reach must not stop it restoring the rest. * docs: name the two newest cells in the dispatch input The cells input is the operator's index of what a number means, and it stopped at 29 while 30 and 31 were added. * test: prove the guest is writing before freezing the store in the recovery suite The storage-fault test froze the object store as soon as RunInstances reported running, which is when QEMU has launched rather than when the guest has booted. On the nightly cluster the freeze landed while the guest was still in UEFI, so the only drive it could reach was the EFI varstore — and pflash has no werror=stop to pause on, so the guest never paused and the test reported a product failure it had caused itself. The load driver now prints to the serial console after each pass it has written and synced, and the test waits for that before freezing. The console is read through GetConsoleOutput, so this still needs no login and still runs on cells whose guests have no public address. * feat: recover instances automatically with nothing to configure The switch is gone. An instance whose host has failed is down whether or not recovery is enabled, so the only outcome a disabled reconciler can produce is leaving it down — which is the state recovery exists to end, not a safer one to fall back to. Against that, the switch cost a cluster that can be configured into never recovering anything, with no symptom until a node dies. This feature already hit that once: a top-level table parsed into nothing and every node reported recovery off with the setting plainly in the file. What makes recovery inert is now only facts about the cluster — no cluster config, no JetStream, or fewer than three nodes — and each says in the log which one applied. The e2e no longer writes config or restarts daemons: it waits for every node to report the reconciler active on a cluster nothing configured, so reintroducing a switch fails there as well as in the unit tests. * feat: stop retrying a recovery that is never going to work Every failed attempt used to go to the same exponential backoff and stay there. An instance nothing could launch was retried every fifteen minutes forever, still observed-running on a node that is gone, with no account of it anywhere the customer or an operator would look. A failure is now classified and counted. Two classes are read from what the launch actually reports: the node it was taken from still holds a volume lease, which clears on that lease's clock and is retried on a faster one, and this node has no room, which another survivor may. There is no terminal class, because nothing on this path can prove one — the launch collapses everything else into ServerInternal, so a verdict of "never going to work" would be a guess, and the guess that abandons a recoverable instance is the expensive one. The budget bounds the unknown case instead. Every class counts against it, since a class that did not would be the old unbounded retry under a narrower condition. Exhausting it leaves the instance where AWS leaves one whose host failed and could not be brought back: stopped, carrying a StateReason that names the cause and the last error, one StartInstances from running again. Writing the desired state down is also what makes one node's decision cluster-wide, because selection reads desired state — the same property the claim already has. A record another node has since claimed is never touched, so a node exhausting its attempts cannot stop a guest that is now running elsewhere. * fix: give a guest the time for every cost it pays before its monitor answers A GPU instance booting from a network-backed root volume pays both costs: QEMU opens every drive across the object store and then pins the whole of guest RAM, one after the other, all before the monitor answers. The deadline took whichever was larger, so such a guest was given less time than it needs. That the two floors are both 180s is why nothing has been seen yet — the VFIO branch has always returned at least the network-drive deadline. It is the same defect either way, and the symptom when it lands is a GPU instance SIGKILLed on every attempt while its plain sibling launches, which reads as a GPU fault rather than as a deadline. The costs are now added, the plain QEMU startup is counted once rather than replaced, and the cap covers both floors together so it cannot clip a guest before either cost has been waited out. * test: assert the AWS-visible arc of a host failure end to end A customer cannot see which host an instance is on, and AWS never tells them. What they can see is DescribeInstanceStatus, so that is where a recovery has to be provable, and the suite never looked at it. Three frames are now required, in order: running with system-status ok before anything is broken, system-status impaired while the owning node is gone, and ok again once a survivor is running it. The middle frame is the one worth having — the window opens when the owner's heartbeat goes stale, which is strictly before any survivor may claim, and closes when the relaunch registers, so it is bounded rather than lucky. Reporting ok throughout would be the more comfortable lie. The availability zone is compared across the first and last frames as well. A recovery that crossed one would break every placement promise made to the customer, and nothing else in the suite would have noticed. * test: prove a half-healthy node fences itself and is never a second writer The two existing recovery cases take a node away by stopping its services, which is the failure this feature handles least interestingly: a node that is gone has no opinion about who owns its volumes. The one that matters is the node that is still up, still running QEMU, still answering its own local clients, and simply cannot reach consensus. That is the state in which two nodes can each believe they own a volume, and the only thing between that and a corrupted disk is that the partitioned node gives up first, on a clock it can evaluate without reaching anything. So this drops NATS on 4222 and 4248 between one node and its peers and nothing else, then asserts the ordering the safety argument rests on rather than the outcome it produces. A watch samples every node every three seconds for the whole run, including the heal, and fails on any single sample naming two hosts. Two guests writing one volume through one object store is data loss already under way, not a race that resolves, and no point-in-time assertion can see one that appeared between its own samples. The injector takes its own nft table rather than touching inet spinifex_filter, because a test that flushed the node's real firewall would leave it either unfirewalled or misfirewalled with no sign of which, and a separate table at priority -300 heals in one atomic delete. A transient systemd timer removes it independently, so a cancelled workflow does not leave a node cut off from its cluster until somebody notices. It proves SSH still works and proves a peer port is genuinely unreachable before anything is asserted, since a rule set that installed cleanly and dropped nothing would read as the system tolerating a partition. The suite budget goes to 90m and cell 31 to 105m to fit the third case. * fix: move an instance's boot state, volumes, logical port and address with it A record-driven launch assumes the node reading the record is the node that wrote it. On a daemon restart that holds. On a recovery onto a survivor it does not, and five things follow from it — one that stops a moved instance booting at all, and four that stop it running anywhere. A load balancer never comes back from a host failure. Its boot configuration is a set of files under the runtime directory and the record holds their paths, so the survivor restores paths that exist only on the node that died. The hook that rebuilds them was reachable from Restore alone, and the cross-node path goes straight to vmMgr.Run — so QEMU is handed a netcfg, an agent env and a CA cert that are not there. The gateway URL baked into that env names the creating node's management bridge as well, which the rebuild corrects because it rebuilds from whichever node is running it. Firing the hook on both paths fixes both, and a rebuild that fails now refuses the launch: a guest that boots without its network configuration is worse than one that does not boot, because the records would say it recovered. The export outlives the guest and holds the volume lease. A returning node is healthy, so it renews that lease indefinitely and the new owner never acquires: the guest is stopped here and refused there, recovery spends its attempt budget on a volume that was never going to be free, and gives up on an instance that was never unrecoverable. So the export is given up too, over a new node-addressed request that tears it down exactly as the fence does. Deliberately not an unmount — an unmount seals, this node's block map is the stale one, and sealing publishes it over the winner's. Unlike the fence it deletes the lease key rather than leaving it to expire, because the lease is still this node's to hand back. The tap outlives the guest, and the tap is what claims the instance's OVN logical port. The binding names one chassis, so two nodes offering the same iface-id do not split the traffic — they contend, and the address answers from whichever won last. It also blocks the layer above: the external IP's host ingress is pruned on the evidence that the owning guest sits on another chassis, and that stays false while this node is still claiming the port. The management address outlives the guest as a cluster-wide reservation, and the release was unconditional. br-mgmt is one flat L2 segment, so the node an instance left was freeing an address still in use and offering it to the next instance anywhere. Only the node an entry names may release it now, with the takeover recorded on arrival so the node actually running the instance can still give the address back. An entry written before the node field existed names nobody and stays releasable, because there is no other holder to collide with. The fence path already cleaned up the tap, so the release fix repairs that path too. * test: prove a load balancer keeps its name, its address and its traffic when its host dies Takes away the node running an internet-facing ALB and asserts the four things a customer of one actually holds: the name keeps resolving, the address behind it does not move, requests through the name are answered again without anybody being called, and the backends are healthy again from the recovered load balancer's point of view. The backends are placed around the load balancer rather than beside it. A load balancer lands wherever a node answers its queue group, so its node is discovered and the backends are relaunched until both are elsewhere; one on the node about to be taken away would be recovered too, and the run could not then tell an ALB that came back serving from one that came back with nothing to serve. DNS is asked of a node's own northstar over the wire, and the HTTP probes resolve through the same server, so the claim under test is what this cluster answers for a name rather than what the machine running the tests was configured to forward to. A watch samples every surviving node every five seconds for the whole run, because an answer that went missing or pointed elsewhere for one reconcile pass is the outage, and no single sample reaches it. The returning node then has to give up what it no longer owns: no second copy of the guest, and no tap still claiming the load balancer's logical port, since OVN binds a port to one chassis and a second claimant contends for it rather than splitting the traffic. The last assertion is that the name still serves with that node back in the cluster. * docs: name the new cell in the nightly's cell list * test: stop the disk-load guest from filling the console it is judged by The load generator announced every write, once a second, for as long as the test ran. The console is a ring buffer this suite reads back, so an hour of that pushes out everything explaining a failure, including the kernel's own account of the I/O error the test exists to produce. Announce twenty times, then keep writing quietly. * docs: say what the OCI guide has been proven on, and what a host failure does The guide read as a support statement for OCI rather than as an account of one tested path, promised seven steps and then gave eight, and said nothing about what happens to a guest or its public address when the node under it fails. State the evidence, state that the nightly guards the datapath and not the cloud, and give the failure behaviour as a table with the reason codes a customer will actually see. Also say plainly what the volume lease is: one running guest and a bounded, logged window, not a write barrier the object store enforces. Someone sizing a workload against this should read that before they read the table. * docs: say why the mount classifier refuses to retry a lease another node holds It reads as an omission — a lease held elsewhere is the most transient refusal there is. It is not. This classifier drives the restore path, where the holder is a node that is running the guest, so retrying would relaunch against a live owner. Recovery wants the opposite and reads the refusal itself. * feat: claim a recovered guest's public address in line, and verify it arrived A recovered instance was reported running before anything moved its public address off the node that died, so for up to fifteen seconds the records said recovered and the address answered nothing. The recovery path now runs the affinity pass itself, which is idempotent and is the same pass the ticker runs. Two things the pass could not do before. It abandoned every remaining address on the first API failure, though a throttle or a denial is per-object and the guests behind the others are just as dark. And it trusted the move's own response, which is OCI agreeing to the request rather than OCI having done it, so an accepted move that never landed read as success. It now re-reads what OCI delivers here and reports the difference. An address that will not arrive makes the guest a system-status failure with the time it went dark, the same shape AWS uses for a host-level fault. The guest passes every check it can be asked about itself, which is exactly why reporting it healthy was the worst available answer. * feat: give a recovery attempt one trace across all three of its phases The volumes opening, QEMU starting and its monitor answering already trace themselves, but nothing tied them to the attempt that asked for them: three orphan traces per recovery, no way to ask which phase a slow one spent its time in, and nothing naming the node the guest came from. A span over the attempt makes them its children, and the reconciler's own log lines carry its trace ID because the handler stamps it from the context. A failed attempt records the fault and the attempt count on the span, so the budget running out is readable as a sequence rather than as twelve unrelated warnings. * fix: finish a fence once its writer finally exits A kill that did not take effect was logged, counted and dropped. The export stayed registered and the lease stayed held, which is correct while the writer lives and wrong forever after it dies: a lease a healthy node keeps renewing never expires, so the volume could not start anywhere. The failure is now re-reported every 30 s until it resolves, and the teardown completes as soon as the writer is confirmed gone. Only a delivered kill is waited on — a refused one says nothing about what became of the writer. * test: stop judging a detached stop or terminate on a transient state Both tests read the instance state immediately after the ack, but the transition runs on its own goroutine, so the read races it in both directions: too early sees running, too late sees the final state. The race gate caught the first of those on a loaded runner. * fix: replicate the volume lease and dirty buckets across the cluster Both were created with the default single replica, so both lived on one node. That node holds the answer to "who may write this volume" and the marker every mount reads, and losing it takes volume leasing and mounting down cluster-wide. A partitioned node was fenced correctly and then no survivor could relaunch its guest: every attempt failed on "read dirty mark: no responders available", for ten minutes, until the budget ran out. The replica count is now the cluster's node count, as the gateway's own buckets already do. Existing buckets are raised in place rather than requiring a wipe. The e2e diagnostics that should have said this are fixed too. The fence and the surrender are logged by the storage service, not the daemon, so the assertions read the wrong journal; and Eventuallyf evaluates its message at the call site, so a ten-minute wait was explained by a log that stopped before it began.	2026-09-27 15:56:09 -05:00
Ben Duncan	828b5df4aa	fix: pause a guest on a backend I/O error instead of handing it EIO (#977) ... * fix: fail DRAIN when a volume seal failed shutdownAndUnmount logged the volume unmount error and returned nothing, so StopAll returned nil and the DRAIN phase ACKed success with vms_stopped set. The coordinator then advanced to STORAGE and stopped viperblock while the seal was still in flight, tearing the VBState envelope and leaving the guest unable to mount its root volume on the next boot. The deploy reported success. shutdownAndUnmount now returns ErrVolumeSealFailed when the unmount did not seal; QMP, PID-file, fw_cfg and telemetry cleanup stay best-effort. stopCleanup and stopOne carry it without aborting the rest of the teardown, since QEMU is already down by then, and StopAll aggregates the per-VM failures so handleShutdownDrain ACKs with Error set and never reports vms_stopped. volumeMounterAdapter.Unmount was the second swallow site: it logged each failed seal and returned nil unconditionally. It now attempts every volume and returns the aggregate, which is what the VolumeMounter contract already described. Terminate still tolerates a failed seal — the volumes are deleted next, so an unsealed block map loses nothing a caller can ask for. This does not fix the seal itself. It converts silent data loss into a refused upgrade that an operator can act on. * test: add the storagefault e2e suite for backend-outage guest survival Freezes predastore with SIGSTOP to hold connections open while answering nothing, which is the fault a timeout-free path cannot distinguish from a slow peer. On one node that is a total outage; on three it freezes every node but the guest's own, breaking the RS(2,1) shard floor. The suite writes and flushes a verifiable region before the fault and churns a separate unverified region during it, so an interrupted write is never mistaken for lost data. The root-filesystem load runs from the user's home rather than /tmp, which is tmpfs on this image and would move no bytes on the root device. * test: verify the root device is under write pressure, and size the outage The previous guard matched /^ *vda / against /proc/diskstats, where the device name is the third field, and read $7 which is ms_reading rather than sectors_written. It printed empty and asserted nothing, so a green run could not distinguish a surviving guest from an idle disk. It now samples sectors_written twice and fails unless it advances. The outage is sized from viperblock's absorption rather than guessed: a workload must fill DefaultMaxPendingBytes before backpressure engages at all, so a short freeze measures the retry budget rather than the fault. * test: add nbdkit fault injection to the storage-fault e2e suite The predastore freeze takes the whole storage cluster away from every guest on it, which is too broad to run beside anything else. Removing a single volume's NBD server is the same fault at one-volume scope: one host, one guest, nothing cluster-wide stopped. Both signals are here because they sit on opposite sides of the defect. SIGSTOP holds the socket open and answers nothing, so the guest blocks and no error reaches ext4. SIGKILL closes it, and QEMU's nbd driver at the default reconnect-delay of 0 fails requests at once, which under werror=report aborts the journal. Signalling is guarded: a pid is matched on comm, our plugin and the volume argument together, and re-confirmed immediately before the signal, because these hosts run nbdkit for other volumes and pids are reused. * fix: pause a guest on a backend I/O error instead of handing it EIO Every virtio-blk device was created with werror=report,rerror=report, so a backend outage was delivered to the guest as EIO. ext4 aborts its journal on that and remounts read-only, and the write that failed is simply gone. Under werror=stop QEMU pauses the VM and holds the failed request, so a backend that comes back resumes the guest with nothing lost. The boot drive, the cold-boot data volume and the hot-plugged data volume all move to stop. Alongside it the NBD transport gets reconnect-delay=30, which pauses requests rather than failing them while a storage process restarts on the same socket; on the legacy -drive form it has to be file.reconnect-delay, because the bare name reaches the raw format driver, which has no such option. The QMP heartbeat now recognises run state io-error and issues cont. After ten attempts the instance is reported impaired and the cadence drops to one attempt per ten heartbeats, but it never stops: QEMU holds the request for as long as the guest exists, so a backend that returns an hour later can still be resumed. The storagefault suite gains the fault this was written for — kill the volume's own nbdkit, hold, then put it back on the same socket — and it now verifies a sha256 region across the outage rather than trusting the guest to be reachable. Three harness defects went with it: the socket-liveness poll ran unprivileged against a 0770 directory and could never succeed, the relaunched socket was left 0755 where QEMU needs group write, and the two variants shared one guest so a fault in the first was diagnosed in the second. * test: cover the io-error resume policy * test: stop two parallel tests racing on the terminate backoff variable * fix: report a guest paused on a backend I/O error as impaired A guest held by werror=stop kept reporting ok on both status checks. QMP stays responsive while the guest is paused -- cont is being sent and answered -- so the QMP failure gate that DescribeInstanceStatus keys on never trips, and the one state a customer most needs to see looked healthy. Instance status now reports impaired/failed while the resume budget is non-zero, timestamped from IOErrorSince so impairedSince is when the guest paused rather than when it was escalated. System status reports impaired too: a backend I/O error is an infrastructure fault, not a guest one, and that distinction is what the two checks exist to draw. The counter is cleared by the first poll that sees the guest running, so both clear on recovery. * test: reap the relaunched nbdkit when the fault test ends relaunchNbdkit detaches the server with setsid so it outlives the SSH session that started it, and viperblockd never learned it exists. Nothing reaped it, so every run left an nbdkit holding a socket and a cache for a volume that was later detached or deleted -- four were found running on dev-prod against two detached and two deleted volumes. The PID is resolved from the socket path rather than the volume id, so a legitimate viperblockd server for the same volume is never the one killed. * test: prove the backend is serving before a fault is called cleared Thawing the process a fault stopped is not the same as restoring the service. The outage test treated SIGCONT as the end of the fault and went straight to judging the guest, so it passed while the cluster it ran on stayed unable to serve a single object for the next 35 minutes, and the next test would have inherited that. thawPredastore now puts and gets an object through every gate, with a budget, and fails the test if the backend does not come back. * fix: do not hand back a mount entry whose export is gone A volume whose unmount failed kept its entry in MountedVolumes while its nbdkit was gone. Every later ebs.mount then matched that entry and returned its socket path, so vmMgr waited for a socket nothing would ever create and the instance could not be started again — on dev-prod a guest stopped against a frozen backend was left permanently stopped, four start attempts failing identically on a socket the node had already removed. Drop the entry when there is positive evidence the export is dead: the socket is absent, or the recorded pid is not running. Neither can be true of a live export, so the double-writer hazard the guard exists to prevent is unaffected. Also make the durability test's pattern helpers report failures. The read dd sat in a pipeline, so its exit status was discarded and the harness hashed nothing and returned the empty-string sha256 as a checksum. * test: report the guest console when a restart never answers SSH A restart that reaches running but never comes back is indistinguishable from a dead root filesystem without the serial console, and the log is gone once the fixture terminates the instance. * fix: converge guest port datapaths concurrently, not one deadline after another Each guest port that misses its SB binding burns the full convergence deadline, and applyEIPs spent them in series. Three ports belonging to long-stopped instances therefore held a newly launched guest's EIP dark for the sum of them before its own probe started. AddEIP stays serial since it mutates NB. Only the waits fan out, bounded, and sb-cluster-state-reset is now serialised with a minimum interval so concurrent probes cannot each fire a cluster-wide resync. * fix: refuse a stale cross-node start after the owning node is gone The volume lease already refused a cross-node mount, but it is a liveness claim: 45s TTL, renewed every 15s. A node that dies holding un-uploaded writes stops renewing, and 45 seconds later nothing refuses a start elsewhere that loads a stale backend checkpoint. A failed seal now writes a durable marker naming the node holding the only current copy, cleared on a seal that reaches the backend and on delete. The lease covers a live holder, the marker a dead one. The refusal also reaches the caller instead of being flattened to ServerInternal, which told an operator to retry something that cannot succeed while the holder is down. * fix: keep the daemon's message when a start is refused Rebuilding the error from the code alone rendered the generic ErrorLookup text, so a refusal naming the node holding the volume reached the caller as boilerplate about attaching volumes. * fix(ec2): terminate detaches every DeleteOnTermination=false volume TerminateStoppedInstance skipped the detach for a non-Boot DoT=false volume on the assumption that Stop's Unmount had already cleared its attachment. Unmount only clears it when the seal succeeded, so a volume whose seal failed stayed in-use pointing at an instance that terminate then deleted, leaving it unreachable with no owner to release it. The running-instance path already detaches every DoT=false volume; the stopped path now matches it, and tolerates an already-gone volume the same way. DeleteVolume also keeps its provider error on the way out. The terminate reaper retries the delete every two minutes and logged a bare ServerInternal, which made a blob node being down indistinguishable from a transient fault. * fix(ec2): DetachVolume --force releases a volume whose instance is gone The ordinary detach routes to ec2.cmd.{instanceID} for the guest unplug, so a volume left attached by a part-way teardown could never be detached: the instance no longer runs anywhere and nothing answers. That also made it undeletable, since DeleteVolume refuses an attached volume. ForceDetachVolume already existed for this deadlock but was reachable only from account teardown. Force now falls back to it when the instance has no responder, which is the one case where clearing the attachment cannot strand a live guest. * fix(viperblock): unsealed writes steer placement instead of pinning a volume The dirty marker refused any mount from a node other than the one holding unsealed writes. Instance start already forwards to the node that last ran the instance and falls back after a 30s window, so that refusal turned a working fallback into a volume that could run nowhere once its owner was gone. It traded a rare data-loss risk for a permanent availability failure. The marker is now a record and a warning, not a gate. A takeover logs the previous holder and why it could not seal, then adopts the marker so the old node is no longer treated as the better source. Live exclusion is unchanged and still enforced by the lease. It is also written on open rather than on seal failure. A node killed mid-write never reaches its seal, so the old trigger left no trace in exactly the case the marker exists to describe. Adds 'spx admin volumes unsealed' to see which volumes would cost something if started elsewhere. * fix(viperblock): fence a volume whose lease moved instead of logging it The lease was checked once, at mount, and never again. A node that lost it kept its export up and kept writing into a volume another node owned, so both copies advanced and the backend settled it by arrival order. Losing a renewal now decides: another holder means tear the export down, no holder means the entry merely aged out, so reclaim it and leave the guest running. Fencing never seals — this node's copy is the stale one — and leaves the dirty marker for the winner to take over. This is not an epoch fence. It narrows the window from the 45s TTL to one renewal interval at no write-path cost; the backend still cannot refuse a stale writer. * fix(network): advertise EIPs so a recycled address is reachable The gateway switch port only asked ovn-northd to publish the router's NAT addresses in centralised and routed modes, so on a distributed cluster nothing ever announced an elastic IP. OVN still answers ARP for it, which is why a first-time allocation works and hid this — but no gratuitous ARP is sent when the address moves to a new ENI, so every host on the segment keeps resolving it to the previous holder's MAC until its cache ages out. Reusing an address from the pool then lands on a dead MAC and the instance is unreachable from outside. Distributed mode needs the option as much as the others: northd scopes each entry with is_chassis_resident(<vif port>), so the announcement leaves from the chassis running the instance rather than the gateway, which is what distributed NAT wants. Converge on the already-attached path too. AttachIGW returns early once the gateway port exists and nothing else rewrites it, so without that an existing VPC would stay silent for its whole life. Confirmed on dev-prod: setting the option moved three EIPs from unadvertised to carrying their correct external_mac, each scoped to its own ENI's port. * fix(viperblock): make the volume fence and its dirty marker ownership-safe Four corrections to the lease-loss fence, all found in review. Drop the reacquire path. Losing a lease and finding no successor was treated as a lapsed entry worth reclaiming, but leases.acquire returns the cached entry — already flagged lost, with a dead renewal goroutine — so the "reclaimed" lease renewed nothing and the node went on writing a volume it could not prove it owned. The two errors are not symmetric: a false fence costs one guest a restart, a false reclaim leaves two nodes writing one volume. Fence in both cases, and report which happened through the outcome attribute. Surrender a lease this node cannot confirm. Renewal only noticed a loss when the server rejected an update, so a node partitioned from NATS kept its export up while the TTL expired and another node took the volume. Track the last confirmed renewal and give the volume up once that ages past the point where the server may admit another owner. Bound the renewal call too, so a hung JetStream request cannot hold the goroutine past that point. Do not act on a kill that did not take. Both the fence and ebs.unmount removed the mount entry, released the lease and announced the result before knowing nbdkit had exited — the fence then reported a writer stopped that was still running, and the unmount sealed into a directory a live process was writing. Wait for the exit, and on failure leave the volume mounted and say so. An already-exited process now counts as success, which is what the caller asked. Give the dirty marker a generation. It recorded an owner and nothing to order two nodes' claims, so a node returning after a takeover could clear or overwrite the winner's marker and point placement back at the stale copy. Stamp the lease generation, refuse a write from an older one, and only clear a marker this node still owns. A marker that cannot be written now fails the open rather than leaving writes nothing records. * test(viperblock): cover the stall surrender and the unmount kill guard The two ownership cases the fence work left untested. A holder partitioned from NATS while the winner keeps it never gets an update rejected, so revision checking cannot see it — it has to give the volume up on its own clock. Pins that validity sits below the server TTL with room for a check in between, that a lapsed holder reports the loss as a stall rather than as a takeover, and that releasing a surrendered lease leaves the entry for whoever holds it now. The unmount side of the kill guard: sealing under a live nbdkit writes a state torn between two writers, so a kill that did not take has to fail the unmount and leave the entry mounted for a retry. * fix(e2e): keep the guest private key for as long as the key pair EnsureKeyPair computed the PEM path from the caller's directory but wrote the file inside the ensureOnce closure, which runs only on the first call. Callers passed a per-test artifact dir, and ArtifactDir removes that dir when the test passes — so the first test's success deleted the only copy of the private key while the key pair stayed registered, and every later call returned a path nothing had ever been written to. Guests after the first were then launched against a public key whose private half no longer existed, so the harness could never authenticate and gave up after the full five-minute budget. The guests were healthy and reachable on tcp/22 throughout, which is why this read as a storage or network fault for so long. The directory parameter was the defect, so it is gone: the PEM now lives in a directory the fixture owns and removes with the key pair, and no caller can supply one that is shorter-lived. Also reorder the killed-nbdkit teardown. The reaper registered its own cleanup after the detach registered one, so LIFO killed the block server before the platform detached the disk and DetachVolume then failed with "Node ... is in use". The detach cleanup now reaps once it is done. * fix: stop leaking an nbdkit export when recovery cannot adopt it Restart recovery adopts a guest's boot volume, then routinely fails the -efi volume with "claim lease: context deadline exceeded" -- JetStream is still catching up at daemon start and the KV claim runs on the client's 5s default. Recovery gave up on the first error and never retried, so the export stayed live with no registry entry. At terminate the unmount found nothing to seal, returned NotFound and killed nothing, and the process outlived its guest holding the volume and socket open. EFI is always the second volume adopted, which is why it was always the one that leaked. Two changes, because either alone leaves a hole: - rebuildMountedVolume retries the build (4 attempts, 3s apart) so a JetStream that is merely slow no longer orphans an export for the life of the process. - unmountVolume reaps a corroborated export for the volume it was asked to unmount when the registry has no entry for it, so an export orphaned by any means is still cleaned up. The reap goes through reapOrphanedNbdkit, which declines anything a live process references: at unmount the guest's blockdev may not be torn down yet, and killing the server underneath it strands QEMU. * test: hold the backend-serving probe and assert predastore readiness The backend-outage test declared the backend restored on one successful round trip through each gate. A gate can answer the request that arrives just after a thaw and then lose its blob peers under the load that follows, which is a backend that never recovered reported as one that did -- and that is what happened: the probe reported all three gates serving minutes before the guest was still taking BLOCK_IO_ERROR on every resume. The round trip has to hold across three rounds now, and the count restarts rather than decays when one fails, so a gate that serves and then stops is the failure the hold is looking for rather than noise it averages out. It also checks /readyz on each host and reports any failed check by name. The round trip only proves this workstation can write through a gate; it says nothing about which blob peers that gate can reach, and a gate one node short of the write floor serves a small object and fails a guest. A host that serves no admin listener is logged and skipped, since the port is optional configuration. * fix: scope predastore's admin port to the cluster plane in the host policy predastore's admin listener binds the host's cluster bind_addr and serves /healthz and /readyz unauthenticated. Readiness now names which blob peers a gate cannot reach, which is exactly the kind of topology detail 4432 is peer-scoped for, so 8660 joins it in the peer-scoped rule rather than the public one. * fix: keep resuming a paused guest at heartbeat rate after a long outage Past ten attempts the cont dropped to one per ten heartbeats, so a guest whose backend had come back could sit paused for another five minutes before anything retried it. Measured on dev-prod: the backend was proved serving through all three gates while the guest was still waiting on attempt 60, five minutes after attempt 50. The slowdown was there to avoid paying a failed I/O every thirty seconds on a backend still down. That is nearly free, and five minutes of downtime after recovery is not, so only the log line is throttled now. * test: fail the run when a key pair name resolves to more than one pair A create that lands twice leaves two key pairs with one name and different material. RunInstances resolves the name to one of them while the harness holds the private key of the other, and the only symptom is every guest in the run refusing the key at the SSH gate — five minutes of waiting per guest, four guests, and nothing anywhere naming the cause. Observed on dev-prod: two pairs created a millisecond apart, both named e2e-key-30de10ba. One DescribeKeyPairs after the create says so outright instead. * test: tolerate an already-released volume in the detach teardown Teardown runs DetachVolumeWait after a terminate that releases the volume itself, so the detach can arrive at a volume already on its way to available. That is the outcome it wanted, and treating the IncorrectState refusal as fatal failed a backend-outage run whose assertions had all passed — the guest came back, every acknowledged write verified, and the test still went red on the way out. VolumeInUse is still fatal: that one means the volume is attached somewhere it should not be. * test: carry the merged suites onto the reworked key-pair and account-scoped seams The diskperf suite arrived on dev calling EnsureKeyPair with the artifact dir this branch removed, and DeleteVolume's metadata lookup is account-scoped now, so the provider-failure test has to seed and delete under the same account. * refactor: make the fence outcome a type in one const block Four separate exported string constants with a paragraph of comment each, passed to RecordVolumeFence as a bare string. Group them, give them a FenceOutcome type, and take that type at both call sites so an arbitrary value can no longer reach the metric attribute. * refactor: move viperblockd's wire contract into a leaf vbwire package The daemon imported viperblockd for exactly two symbols -- the fenced volume subject and its event -- which put the storage service back on the daemon's dependency graph to name something it only listens for. vbwire holds the contract instead: the subject and event, the dirty bucket name and record shape, and the read-only listing an operator tool needs. It imports nothing from spinifex, so both the daemon and the admin CLI drop their edge to viperblockd entirely. cmd/service.go keeps its one, which is genuine: it constructs the service. * docs: drop a specific host name from the start-refusal comment Named a real dev-prod host in a comment and in test fixtures. Say what the name is instead. * ci: give storagefault a dedicated nightly cell The suite is eligible for both topologies and runs on the PR gate, but no nightly cell ran it: the permutation cells run a deliberately narrow subset and none of the dedicated suite cells named it. Cell 29, single-node, following eks/ecs/rds. It cannot join a permutation cell -- it SIGSTOPs a cluster-wide service, so every suite sharing that environment would report the outage as its own failure. * ci: run storagefault nightly on three nodes, not one The previous commit put its dedicated cell on a single-node runner. That was the wrong topology and the wrong justification: single-node is RS(1,0), where the shard floor cannot be broken and a cross-node takeover after a failed seal does not exist, so the two assertions worth having a dedicated cell for were the two it could not make. Cell 29 is now real-multi on ci-multi. It also corrects the claim that the PR gate already covers the multi-node case. The suite is eligible for both topologies, but e2e.yml's default suite list never names it, so it has never run there. A full pass is ~63 minutes on dev-prod bare metal, almost all of it deliberate waiting -- one test holds predastore SIGSTOPped for seven minutes -- which is too long to add to a gate that currently finishes well inside its budget. Nightly is where it belongs. Consequences: a per-cell timeout_minutes, defaulting to the previous 50, since 90 minutes of suite cannot run in a 50-minute job; the permutation steps gated on matrix.suite == '' so the dedicated cells do not also run them; and a Run Multi Suite E2E step that builds NODES_CSV from the tofu inventory and asserts the suite against E2E_SUITES_MULTI before running it. * fix: stop a startup KV race leaving vpcd serving keyless guests vpcd opens five KV buckets while NATS is still coming up, each attempt carrying a ~5s JetStream API deadline, and applied three different policies to the same failure: spinifex-vpc-enis was fatal, while instance-state, security-groups, subnets and vpcs each degraded silently for the lifetime of the process. A five-second blip during a cold start therefore decided how a node served metadata until someone restarted it. On dev-prod all three nodes hit this after a deploy. Two were saved by accident: their ENI bucket also timed out, so they exited 1 and systemd restarted them cleanly. The third's ENI bucket opened in time, so it never crashed and ran for hours with no instance record loader. With the local state file unreadable as well, every instance-backed IMDS leaf 404'd, cloud-init's crawl died on ami-id and fell back to DataSourceNone, and every guest launched on that node booted with no authorized_keys. kvstore.OpenWithRetry retries for a bounded window before any policy applies, logs the cause per attempt and the wait on a late success, wraps the last error with the bucket name, and returns at once on a cancelled context. All five opens use it. Failing to open the instance record space is now fatal, matching the ENI bucket: the dependency whose loss stops guests logging in should not be treated more leniently than the one whose loss stops ENI resolution. Also fixes the reason the fallback was load bearing at all. The local state file was written 0600 and vpcd reads it as a different user in the shared spinifex group, so the local-first path had never once worked and its mtime+size cache had never fired. It is 0640 now, set with an explicit Chmod so an inherited umask cannot undo it. Closes mulga-gtpzz, mulga-dhqxr. Still open: mulga-mm2oc, IMDS serving an unresolvable instance as "no key pair" with nothing logged. * fix: scope the ime-* firewall accept to IMDS and VPC DNS only The per-tap OVS demux delivers any guest packet addressed to 169.254.169.254 or .253 onto an ime-* host port regardless of L4 port, and iifname "ime-*" accept then admitted all of it. Every wildcard-bound host service — sshd, the AWS gateway, northstar, the console — answered guest packets on those addresses; only asymmetric reply routing kept the handshake from completing, and a guest could still drive northstar to recurse unthrottled. Scope the accept to IMDS (.254:80) and VPC DNS (.253:53), letting everything else fall through to policy drop. * docs: document the instance-to-host plane and how to protect it without the ISO The metadata/VPC-DNS link-local plane (169.254.169.254/.253) is served by host services and is invisible to a NIC-level or cloud firewall, since the traffic is intra-host on the OVS bridge. Explain that the ISO ships the scoping rule armed, that the binary installer ships the firewall off, and that a non-ISO install must enable it with setup.sh --firewall=on or firewall_enabled=true. Include verification from an instance and the host. * fix: withhold the IMDS meta-data listing when the instance does not resolve Every omission from the meta-data root is a positive claim about the instance — public-keys/ absent means "launched without a key pair", and cloud-init believes it, writes no authorized_keys and boots the guest keyless. So resolving the instance was best-effort for a listing whose whole meaning depends on it: a backend error or a miss shortened the listing instead of failing it, and nothing anywhere recorded that the answer was a guess. Withhold the listing with a 500 instead. cloud-init retries a 5xx and cannot retry a 200 that is merely wrong, so this converts a permanent silent failure into a transient loud one. servePublicKeys already had this policy for the leaf; the root is what contradicted it. A profile lookup that fails still serves the listing minus iam/, and now logs. The key pair is the field a guest cannot recover from losing, while a missing role surfaces through the credentials leaf, which reports the miss itself. Also stops the storagefault cross-node test failing on its own access path. It tolerates the pattern changing across a takeover, but readPattern needs sudo, and a volume opened from the backend checkpoint may predate cloud-init writing its sudoers drop-in — indistinguishable from an unreadable device at the dd call site. It now checks sudo first, and reads cloud-init's recorded datasource to tell a rolled-back root filesystem from IMDS never having answered, failing loudly on the latter rather than accepting it as storage loss. --------- Co-authored-by: brynmailer-mdc <bdm@mulgadc.com>	2026-09-03 19:45:33 -05:00
Bryn Mailer	18aa67b0e9	feat: add the e2e guest disk-performance gate (#964) ... * feat: add the e2e guest disk-performance gate Nothing measured guest disk performance in CI, which is why a quadratic index scan shipped in viperblock 35dd9f2 and survived three subsequent fixes to adjacent code. It is also why the two availability clauses at the end of the ebs-performance plan's Stage 0 have never been evaluated: a sustained random-write workload took the guest, and then the host, down, and nothing since has demonstrated that it no longer does. tests/e2e/diskperf/ launches an instance, attaches a dedicated blank volume per repetition, and drives a pinned fio profile over guest SSH. Two profiles: randwrite-4k at 4 jobs x 4 GiB, the shape the exit criterion names, and randrw-70-30-4k, the mixed workload the headline IOPS came from. The target is a raw device with direct=1 rather than a file on the boot disk, so guest page cache, filesystem allocation and journal traffic are out of the path. Two classes of assertion, kept apart because folding them together lets a run that left the host unreachable pass on a respectable IOPS figure. Availability is hard pass/fail and absolute, so it gates from the first run: the job completes inside its budget, a guest responsiveness probe answers throughout on a second SSH channel, and the host's own sshd answers throughout. The guest probe reads the root volume while the workload runs on a different one, so a pass is evidence of isolation between volumes rather than of the job merely being slow. The host probe is a TCP connect plus an SSH banner read, which needs no credential and still proves the kernel scheduled sshd. Throughput and latency are banded against a committed baseline: fail above 25% regression, warn above 10%, per-metric median across repetitions. baseline.json is committed and embedded, with every figure zeroed, and a zero reference records and warns rather than failing. Only the availability class gates until numbers are captured on the bare-metal cell; a throughput band nobody measured would be a fabricated gate. A fresh volume per repetition is the reset. Extent count grows monotonically, so reusing a volume makes the gate order-dependent and drifts its own numbers upward over time, slowly normalising the regression it exists to catch. Each job is its own guest invocation and fio's JSON is fetched by a separate command, so a wedged job loses only itself and reports as a budget breach rather than a parse failure. e2e-baremetal.yml gains suites, suite-env and cell inputs so one reusable workflow serves both the correctness cell and nightly cell 28, which queues behind cell 20 on the shared concurrency group. Cells 25-27 are the dedicated eks, ecs and rds cells. The per-suite Go test timeout is resolved from e2e_suite_timeout in suite-sets.sh, where diskperf declares its 180-minute budget beside rds's rather than in a second place. The gate's own arithmetic runs in preflight via test-harness: band edges, the unmeasured cases, per-stream metric selection, fio JSON parsing, and the probe accounting including a slow-but-successful answer counting as a breach. A gate whose comparison logic is untested is the failure mode the plan is otherwise full of. The extent-count ceiling named as a fourth availability assertion is not implemented: it needs viperblock.index.entries from S1.5, which does not exist. * feat: let a manual bare-metal run choose its e2e suites A dispatch run only sees inputs declared under workflow_dispatch, which had none, so suites and suite-env resolved empty and fell back to the correctness set. Capturing a diskperf baseline by hand was therefore unreachable without a workflow_call caller. * fix: build the bare-metal suites the run step will execute The build step packs pre-built binaries into the tarball and did not see the suite selection, so a selected suite was absent from the node unless it happened to be one of the three the build script hardcoded. * feat: commit the measured disk-performance baseline Medians of three repetitions on the bare-metal cell. The throughput and latency bands now gate instead of warning, so a 2x regression in either job fails the suite. randwrite-4k's write tail is two orders above the mixed job's. It is recorded as measured rather than zeroed: an ungated metric would miss the regression class the gate exists to catch.	2026-09-03 14:33:21 +10:00
Ben Duncan	f41f5530a0	test: add an e2e suite proving quotas gate every capped resource	2026-08-20 02:26:27 +10:00
Julian Sommer	1c4d41bffc	fix: reach the DB endpoint from any subnet of the VPC (#738)	2026-08-06 13:56:03 +10:00
Julian Sommer	2e09461080	chore: cleanup code comments	2026-08-04 14:20:19 +10:00
Julian Sommer	5e17169661	feat(e2e): complete the RDS end-to-end suite (#608)	2026-08-03 15:38:07 +10:00
Julian Sommer	2569601e3d	feat: implement RDS CreateDBInstance, reconciler and endpoint DNS (#594)	2026-07-29 11:32:59 +10:00
Josh Siviour	cae76ba1b6	test: complete the e2e live-vs-mock rebalance (epic mulga-8oyg3) (#566) ... * test(volume): assert chunk GC stays wired on in CreateVolume GCEnabled was zero-valued at every viperblock construction site for a long time and no test noticed. It is not part of viperblock.VBState, so it never round-trips through a saved config.json and cannot be observed by reading volume state back. Extract the viperblock.VB assembly out of CreateVolume into buildVBConfig so the GC wiring is assertable directly against the returned config, without a live viperblock or S3 backend. The config is returned by pointer because viperblock.VB carries atomic.Uint64 fields that vet's copylocks check rightly refuses to see copied. * test(e2e): snapshot backend byte accounting before volume purge DeleteVolume purges a volume's entire backend prefix, including any chunks a leak left behind, before anything measures it. That is the same masking that hid the storage-growth leak in production: a passing E2E cannot fail on backend growth when the evidence is deleted first. Snapshot the volume's backend chunk footprint in EnsureVolume's cleanup closure, immediately before DeleteVolume, so every volume the shared teardown path purges is measured rather than silently discarded. The snapshot is always taken and logged; the hard ratio check additionally runs for callers that registered a guest-written-bytes baseline, since most callers never attach or write to the volume they create and so have no baseline to compare against. The tolerance is the RS(2,1) erasure floor widened for per-chunk header and tail-chunk overhead. It is sized to distinguish unbounded growth from a footprint settling at the floor, not to catch bounded residual garbage. The check is non-fatal inside the cleanup callback: t.Fatalf's runtime.Goexit would abort the DeleteVolume call queued behind it and turn a reported leak into a real one. * test(e2e): add memory-constrained concurrent-write durability regression The existing durability tests write one 4MiB sentinel, sequentially, on a comfortably sized guest, which is the exact inverse of the known-open data-loss bug: acked writes read back zero-filled after a cold reread when a guest is memory-starved during concurrent writeback. No happy-path test can reach it, because the fault lives in the intersection of memory pressure, write concurrency and cache-cold reread. Add a suite that encodes those three conditions together: a guest discovered at or below the working set, N parallel large writers launched as background jobs in a single command so they genuinely overlap, swap disabled, then sync and drop_caches before rereading. Assertions are per stream rather than aggregate so a failure names the losing writer instead of leaving whoever reads it to re-derive which of N writers lost data from one combined digest. GuestExec gains a timeout-parameterised variant. Its fixed two-minute default is sized for sub-second payloads, and a multi-gigabyte write to a network-backed volume would exceed it, so the suite would have failed by timeout while looking like it had caught the corruption. The budget scales with the working set, and a deadline is reported distinctly from a data mismatch so the two red states are never confused. The suite is dispatchable but deliberately absent from the default SUITES: the bug it reproduces is still open, so it is expected to fail and must not sit in a merge-gating path. It is registered in the workflow allow-list so it stays discoverable rather than being buried behind a skip. Also record on the live IAM instance-profile test why its assertions were not moved down to the integration tier, so the trace is not repeated. * test(e2e): drop backend byte accounting, assert a node-wide delta instead The accounting helper listed the internal predastore bucket to attribute bytes per volume. That listing is refused unconditionally: predastore authorises bucket access by same-account ownership, and the bucket belongs to the service account, so a test caller gets 403 no matter what it holds. Granting the test a service credential would invert the trust boundary to buy a measurement, which is the wrong trade. Replace it with what the S3 surface actually permits: sample node-wide allocated bytes either side of the write and bound the delta against the bytes the guest wrote. Coarser, but it measures something real rather than depending on a call that cannot succeed. The tolerance moves into the suite that uses it. It encodes RS(2,1)'s 1.5x expansion floor plus headroom; a cluster running the recommended RS(3,2) has a 1.667x floor and only ~12% of margin left, so the constant needs revisiting if the encoding changes. * revert: drop memdurability e2e suite The suite asserted the guest read back what it wrote after a memory-constrained concurrent write, but the assertion could not fail for the reason it claimed: it compared the guest's own read against the guest's own write with the page cache still warm, so it passed whether or not the data reached the backend. A test that is green on a build with a known-open durability bug is worse than no test — it advertises coverage that does not exist. Removes the suite and its Makefile/workflow wiring, plus DiscoverInstanceTypeAtLeastMemory, which had no other caller. GuestExecTimeout stays: GuestExec now delegates to it, and the old two-minute hard cap it replaced was a latent trap for any long guest command. * test(e2e): add partial-block concurrent-write durability suite Replaces the removed memdurability suite, which passed on a build that still carried the bug it claimed to test. The engine reaches its read-modify-write path only for a partial block; a fully covered block takes a full-overwrite branch that reads nothing and can lose nothing. The old workload wrote through ext4 with 1 MiB buffers, which the filesystem coalesced into aligned full-block requests: block-layer tracing recorded zero unaligned and zero sub-4096 requests, so the path under test never ran, and writers 4 MiB apart could not collide on a block even in principle. This suite writes the shape the defect needs. Two writers issue concurrent O_DIRECT writes into disjoint halves of the same 4096-byte blocks on a raw, unformatted device, stamping every half with the round number, then drop the page cache and reread with O_DIRECT asserting every half carries the final round. One half reading back at an earlier round is the signature of the lost update. The old suite also could not fail even had it reached the path: its write phase ended in 'wait; sync; sha256sum' and reported sha256sum's exit status, while bare wait returns 0 unconditionally, so writer errors were invisible and a backend ENOSPC presented as a pass. Here each writer's exit status and exact byte count are asserted before any durability claim is made. Working set is 8 MiB rather than 2048 MiB, and the volume is deleted on teardown; the predecessor left roughly 3 GiB of backend per run and twice drove a node under the nearfull watermark. Dispatchable but not in the default suite list until a green run and a demonstrated fail-first against a pre-fix engine exist. * test(e2e): let the fixture own the working-set volume delete The suite deleted the volume in its own cleanup and the fixture deleted it again on close, so every run ended with an InvalidVolume.NotFound from the second attempt. Detach stays — DeleteVolume against a still attached volume is what leaks volumes on teardown — and the delete is left to the fixture so the volume is released exactly once. * test(e2e): gate merges on the partial-block durability suite Verified on one node with the same test binary and workload, varying only the engine: 1758 of 8192 halves reverted a generation against the pre-fix build, 0 of 8192 against the fixed one. That is the fail-first evidence the suite needed before it could be trusted to gate anything. Adding it to the default suite list rather than leaving it dispatchable is the point. Its predecessor sat outside every path that runs, which is how it survived long enough to be discovered green on a build that still lost data. * test(integration): cover the ECR data plane in the front gate The live ecr/ suite drove docker, crane and skopeo against a real registry, yet none of it boots a guest: it needs a registry, an object store and an HTTP client. The in-process gateway already serves the OCI Distribution v2 surface on an ephemeral port with a memory-backed object store, so these run here in 0.18s instead of behind full provisioning. Covers the three claims the live tests made beyond authorization, which is already down here: a push/pull round trip asserting blobs and manifests read back byte-identical, refusal to auto-create a repository on push, and retagging an existing manifest without disturbing the original tag. Content is compared rather than status codes alone. A registry that stores a blob under the wrong digest, truncates it, or serves a different one answers 200 to every request in the sequence. The capacity/load leg deliberately stays live: it exists to put real parallel pressure on a real predastore, and a memory-backed object store cannot make that claim. * fix(e2e): make volume teardown release the volume before deleting it Fixture cleanups drain LIFO, so a suite that creates its instance before its volume tears the volume down first, while it is still attached. DeleteVolume answered VolumeInUse, nothing retried, and the volume was stranded on the node. Reordering the chain cannot fix this — registration order has no way to express the instance/volume dependency. Instead teardown is now self-sufficient for any creation order: it force-detaches, waits for the volume to reach available, and only then deletes. Instance teardown waits for terminated for the same reason, since TerminateInstances returns as soon as the request is accepted. Two things kept the leak invisible. A failing cleanup was logged and forgotten, so a run stayed green with resources left behind; failures are now reported as leaks, failing the bound test or surfacing through Close() for TestMain to fail the run. And a panic in one cleanup aborted the whole chain, stranding every resource behind it; each callback is now isolated. RegisterVolumeTeardown and the fixture had grown separate teardown paths, only one of which detached. Both now share it.	2026-07-22 17:02:20 +10:00
joshsiv-mulga	9ccf147fc8	Merge remote-tracking branch 'origin/feat/storagegrowth-rootvol' into harness-merge-dev ... # Conflicts: # tests/e2e/Makefile	2026-07-17 18:27:05 +10:00
Ben Duncan	28a876d17d	feat(dns): resolvable ALB/NLB + EKS DNS, forwarder hardening, cross-tenant reconcile (#532)	2026-07-17 12:02:59 +10:00
joshsiv-mulga	4cbd2a5c88	test(e2e): add container image-pull storage-growth workload ... Pulls a container image inside a guest onto a viperblock volume and samples guest used bytes, host backing-store bytes, nbdkit RSS and the live checkpoint size as it goes, so backing-store cost can be fitted against the bytes the guest actually wrote. Complements the N-pass overwrite workload, which rewrites one extent and so pins the block map size — that shape cannot show whether cost grows in proportion to bytes written or faster than them. Redirects both dockerd's data-root and containerd's root onto the target volume. Redirecting data-root alone leaves containerd extracting layers into /var/lib/containerd on the AMI's small root disk, which fails with ENOSPC while the target volume sits empty.	2026-07-16 08:10:24 +10:00
joshsiv-mulga	6e043f6453	test(e2e): add storage-growth N-pass overwrite suite ... Rewrites one logical extent N times through full attach/format-write/detach cycles, so the guest-visible footprint stays constant while total bytes written scales with N. That shape isolates the leak where viperblock mints chunk objects under unique monotonic keys and never deletes the superseded ones: a sequential write-once supersedes nothing and so cannot exercise it. The suite drives the workload only and deliberately does not measure the backend. Measuring on-disk bytes from here would mean either a host du (which cannot separate live from dead bytes, and so cannot distinguish this leak from the unrelated one where an overwritten key's old extent is orphaned without a tombstone) or a dependency on tooling outside this repository, which would stop the suite building standalone. It instead reports what it did to a caller-named result file and asserts only what a guest can observe: a sha256 round-trip through a fresh mount per pass, and a footprint that stays constant across them. The volume is retained on request, because deleting it removes the evidence: DeleteVolume enumerates the bucket by prefix and drops every key under it, including the leaked chunks.	2026-07-15 15:03:21 +10:00
Josh Siviour	cee73671ff	feat(bedrock): Ochre runtime MVP — Converse over self-host + Anthropic-direct (#520) ... * feat(bedrock): Ochre runtime MVP — Converse over self-host + Anthropic-direct Add the AWS-Bedrock-compatible gateway services (bedrock, bedrock-runtime) following the EKS REST-JSON pattern: ListFoundationModels / GetFoundationModel over a static connectivity-tiered catalog, and non-streaming Converse routed per modelId to one of two provider backends — self-hosted vLLM (OpenAI wire) and Anthropic-direct (Anthropic Messages wire). Per-account provider API keys are a master-key-encrypted JetStream KV secret with an optional platform default; the advertised catalog gates external models on a resolvable credential. Self-host endpoints are pinned via config (OCHRE_VLLM_ENDPOINTS) behind an EndpointResolver seam. Adds gateway/bedrock/ (catalog, provider router, Anthropic + vLLM adapters, credential store), awserrors REST-JSON codes, and the bedrock E2E suite (list/get light; Converse gated on live backends). Daemon/NATS control plane and dynamic endpoint lifecycle are deferred to Phase 3. * test(bedrock): cover dispatch, router, credentials KV + endpoint config Add unit coverage for the Ochre Phase 1 diff: Bedrock_Request / BedrockRuntime_Request end-to-end dispatch, provider router, JSON response/error writers, upstream-status mapping, the credential-store KV round-trip, the Anthropic bound-provider path, and parseBedrockEndpoints. Lifts PR diff coverage above the 60% gate.	2026-07-13 21:40:34 +10:00
Josh Siviour	e1c4f85a36	feat: routed-NAT external mode with two-tier ingress (#475) ... * feat(network): add routed uplink mode, Routed wiring, and NAT egress rule helper * feat(external): route gateway LRP over transit pool and install VPC SNAT in routed NAT mode * feat(vpcd): support bridge_mode=nat (routed uplink, transit pool, egress rules) * feat(daemon): register EIP handlers unconditionally with disabled-mode stub; seed default subnet per external mode * feat(admin): support --external-mode=nat and suggest it for non-bridgeable uplinks * feat(scripts): add --nat-uplink routed mode to setup-ovn.sh * test(e2e): add routed-NAT single-node natuplink suite Covers the external_mode=nat lane end to end on a nat-initialized node: host transit wiring (veth, br-ext port, ip_forward, iptables egress rules), spinifex.toml contents, disabled EIP surface (empty describe-addresses, UnsupportedOperation on allocate), default subnet MapPublicIpOnLaunch=false, OVN gateway LRP + VPC SNAT on the transit /24, guest outbound egress proven via a serial-console probe (nat mode has no inbound path), and unique transit gateway IPs across VPCs. Suite self-gates on external_mode=nat; registered as e2e-natuplink and in SUITES_BUILD so it compiles in CI. Manifest entry stays commented until CI can provision a nat-mode environment. * feat(ovn): Address_Set CRUD and NAT exempted_ext_ips column Adds the exempted_ext_ips column (optional strong Address_Set ref) to the nbdb NAT model, Address_Set ensure/get with wait-op create semantics, and an in-place SetNATExemptedExtIPs update on the client interface, live client, and mock. Test NB schema gains the column and covers insert-resolves-UUID, converge, clear, and missing-NAT paths. * feat(policy): exempt address set on routed SNAT/EIP rows Adds WithSNATExemptSet so routed mode stamps exempted_ext_ips (the spinifex_nat_exempt Address_Set: transit /24 plus operator extras) on IGW snat and EIP dnat_and_snat rows, keeping VM replies to host-initiated flows un-NATted. AddSNAT gains an idempotency guard keyed on (router, VPC CIDR) with stale-IP replacement, and both AddSNAT and the AddEIP idempotent-skip path patch the ref in place on pre-existing rows. The set is a singleton strong ref and is never deleted. Non-routed modes are unchanged. * feat(host): VPC ingress route primitives for routed-NAT mode EnsureVPCIngressRoute / RemoveVPCIngressRoute manage the host route that sends VPC-CIDR traffic into OVN via the gateway LRP transit IP on the spx-nat veth. Replace/del semantics keep both idempotent; a missing route on removal is not an error. * feat(external): routed ingress leg on IGW attach/detach, reconcile-safe AttachIGW's routed leg now also installs the host VPC ingress route via RoutedIngressHooks, and re-runs on the already-attached path so the reconcile loop restores host routes (and patches legacy SNAT exempt refs) after a reboot — OVN state survives restarts but host routes do not. DetachIGW removes the route alongside the SNAT. vpcd wires the hooks and the spinifex_nat_exempt set (transit /24 + [network] nat_exempt_cidrs, validated nat-mode-only) exclusively in routed-NAT mode; pool/veth/direct pass nil hooks and are unchanged. * test(natuplink): Tier 1 host-ingress phase; docs for jumpbox pattern New Phase 7 asserts the automatic ingress tier: host route to the VPC CIDR via the gateway LRP transit IP, exempted_ext_ips ref on the SNAT row pointing at spinifex_nat_exempt (containing the transit /24), and — after opening the SG from the transit net (default-closed, AWS parity) — ping and an sshd TCP handshake against the instance private IP from the host, which proves the un-SNATted return path. Phase 8 additionally waits for the second VPC's ingress route to appear on IGW attach. vpc-networking README documents the jumpbox pattern, SG note, and nat_exempt_cidrs. * feat(config,admin): public wan pool alongside nat-transit Restructure admin config generation from single-pool scalars to a Pools slice so nat mode can emit both the transit pool and an optional public wan pool. nat mode now accepts --external-pool/--external-source (plus --external-bind-bridge/--gateway-ip when a public pool is requested), validated by the shared resolvePublicPoolFlags helper also used by pool mode. Transit pool always renders first so the IGW allocator picks it. Formation wire format unchanged (nat mode is single-node only). * feat(daemon): enable EIP service and auto-assign in nat mode with public pool External IPAM (and with it the real EIP service, RunInstances public IP auto-assign, and MapPublicIpOnLaunch on default subnets) now activates whenever public pools exist: pool mode as before, plus nat mode when a wan pool rides alongside the transit segment. The transit pool is filtered out of IPAM — its addresses are gateway-LRP plumbing, never allocatable EIPs. Transit-only nat configs keep the disabled EIP stub. * feat(host): per-EIP ingress primitives for routed-NAT mode DetectUplinkFor parses ip route get to find the interface and source IP facing the pool gateway (re-run per ensure: WiFi/cellular uplinks move). EnsureEIPIngress steers an EIP into OVN with a /32 route via the VPC gateway LRP transit IP (src-hinted so host-local traffic still DNATs), answers ARP for the EIP with a proxy neighbor entry (L3 only, same MAC, WiFi-safe), sends gratuitous ARP, and adds per-EIP FORWARD accepts for DROP-policy hosts. RemoveEIPIngress mirrors; missing state tolerated. * feat(policy,vpcd): host EIP binder option on NATManager WithHostEIPBinder injects routed-mode host plumbing hooks. AddEIP binds after priming on both the fresh-add and idempotent-skip paths (host routes and proxy-ARP are volatile even when the OVN row survives, so reconcile re-ensures them after reboot); DeleteEIP unbinds best-effort. The next hop comes from the new gatewayPortIP helper reading the gateway LRP's stamped spinifex:gateway_ip (Networks fallback), with the ext-ID key now an exported policy const shared with the external package. vpcd selects the IGW pool by name instead of index — with two pools in nat mode the gateway allocator must stay on the transit pool — and wires the binder over host.EnsureEIPIngress/RemoveEIPIngress when a public pool is present. EnsureEIPIngress gains an uplink hint so dhcp-sourced pools (gateway unknown until lease) proxy-ARP on their bind bridge. * feat(dhcp): interface-MAC lease variant with MAC-keyed-router detection dhcp_mac = "derived" (default) | "interface" on external pools. The interface variant leases with the bind interface's own MAC — WiFi/WWAN uplinks drop frames with foreign source MACs, so derived chaddr MACs never complete DORA there. Option 61 stays the lease discriminator. UseIfaceMAC threads AllocateRequest → NATS wire → nclient4 → lease and persists through KV so restart re-acquires repeat it. Routers that key leases on MAC and ignore client-id would hand every lease the same IP; the manager detects an ACK matching the interface's own address or another client-id's live lease, releases the colliding lease, and hard-errors advising a static range instead. admin init auto-writes dhcp_mac="interface" for nat-mode public dhcp pools bound to non-bridgeable uplinks (wlan/wwan/ppp). * test(natuplink)+docs: Tier 2 EIP ingress e2e phases and public-pool docs The natuplink suite now detects a public (non-transit) pool in spinifex.toml and switches lanes: Tier-1-only configs keep the EIP-disabled and no-public-IP assertions; public-pool configs assert the EIP surface is enabled, the default subnet has MapPublicIpOnLaunch=true, instances auto-assign public IPs, and a new Phase 9 exercises the full EIP ingress lifecycle — host /32 route via the gateway LRP, proxy-ARP on the uplink, per-EIP FORWARD accepts, exempt-stamped dnat_and_snat, TCP handshake to the EIP, reconcile replay across a vpcd restart, and teardown on disassociate. Docs: vpc-networking README gains a public-pool section for routed-NAT nodes (init flags, dhcp_mac=interface behaviour, host-to-EIP caveat) and an updated mode comparison; COMMANDS.md notes nat-mode EIP support behind a public pool. * fix(setup): whitelist iptables, neigh sysctl, and arping for spinifex-vpcd Routed-NAT mode shells out to iptables for the transit masquerade and per-EIP FORWARD accepts, tunes proxy_delay, and sends gratuitous ARP for host-delivered EIPs. Without these sudoers entries vpcd crash-loops with "command not allowed" on nat-mode nodes. * fix(ovn): scope DeleteStaticRoute prefix match to the target router Every VPC router carries a 0.0.0.0/0 static route, so an unscoped cache match could grab another router's row and fail with a referential integrity violation, wedging IGW attach on multi-VPC nodes. Match only rows referenced by the target router, mirroring FindStaticRoute. Mock updated to the same semantics, with a regression test covering drift replacement across two routers. * fix(reconcile): always replay AttachIGW so host ingress state is re-ensured applyIGWs skipped AttachIGW whenever the gateway switch port existed in actual state, so the already-attached re-ensure path never ran and routed-NAT host ingress routes lost on reboot were never reinstalled. AttachIGW is idempotent; run it unconditionally. * fix(vpcd): resolve duplicate-CIDR host ingress route collisions, bootstrap VPC wins Every account's default VPC shares 172.31.0.0/16, so Tier-1 host routes collide on the single host route table and the last IGW attach silently stole the route (route replace). Pass vpcID and the gateway LRP IP through RoutedIngressHooks; the bootstrap VPC always installs its route, other VPCs only install when the CIDR is free and only remove a route their own gateway IP holds. Adds host.VPCIngressRouteVia to read the current route holder. * ci(e2e): add nightly cell 19 for routed-NAT single-node suite Cell 19 bootstraps with --external-mode=nat plus the CI static pool as the public wan pool and runs the natuplink suite, covering the Tier-1 transit datapath and Tier-2 EIP ingress. The run step exits 0 when natuplink.test is absent from the tests tar so scheduled runs against branches predating the suite stay green. * fix(admin): nat-mode init summary reflects configured public pool The summary printed 'outbound-only — no public IPs/EIPs' even when --external-pool/--external-source configured a public wan pool. Print the pool range (or dhcp bind) and note EIPs are enabled. * fix(eip,vpc): thread ctx through disabled EIP stub and NAT call sites Reconcile the routed-NAT branch with dev's ctx-first service APIs after rebase: DisabledEIPService and its tests take ctx to satisfy the EIPService interface, the vm_adapters public-IP resolver asserts the ctx-signed AssociatedPublicIPForInstance, and the default-subnet mapping test threads ctx into DescribeSubnets.	2026-07-09 09:43:32 +10:00
Josh Siviour	47c9808a96	test(e2e): add ECS end-to-end suite (awsvpc, bridge, host, ELB service) (#439) ... Drives the ECS data plane against the local awsgw: a customer VPC plus a real container instance launched from the spinifex-ecs-node AMI, then standalone tasks in all three network modes and an awsvpc service fronted by an ALB target group. Adds an ECS client to the harness, a Makefile target, and a service-interfaces suite entry. bridge/host tasks share the host netns where the agent credential endpoint owns 169.254.170.2:80, so they run a no-port command and assert placement + mode; awsvpc and the ELB service prove real :80 serving. waitTaskRunning re-checks after a settle so a transient RUNNING cannot pass.	2026-06-29 14:16:23 +10:00
Josh Siviour	f9858d4e5d	test(e2e): ECR end-to-end suite (control plane + docker/crane/skopeo) (#398) ... * test(e2e): add ECR end-to-end suite (control plane + docker/crane/skopeo) New tests/e2e/ecr/ suite plus harness support: - harness: ECR SDK client, control-plane helpers (repo CRUD, get-login-password, describe-images), and docker/crane/skopeo CLI wrappers that skip when the binary or registry-host resolution is absent. - suite: control-plane CRUD + policy + lifecycle round-trip; docker login/push/ pull; crane append/pull and skopeo copy exercising the Bearer /v2/token flow. - registered e2e-ecr in the Makefile + service manifest. Predastore capacity/load leg is a skipped placeholder pending mulga-siv-359/356. * test(e2e): carry awsgw port in ECR registry host for dev clusters Dev clusters bind awsgw on a non-standard port (e.g. 9999); production uses :443. ECRRegistryHost now appends SPINIFEX_ECR_PORT when set so docker/crane/ skopeo address the right endpoint, and RequireRegistryResolves strips the port before DNS lookup.	2026-06-19 11:34:38 +10:00
Julian Sommer	eced3f733a	feat: split IAM e2e (#372)	2026-06-17 15:06:16 +10:00
Josh Siviour	496424c41c	test(eks): EKS control-plane E2E suite (Sprint 6g) (#304) ... * test(eks): EKS control-plane e2e suite (Phase 1, API-level) Add tests/e2e/eks/eks_test.go driving the EKS lifecycle against awsgw: CreateCluster -> ACTIVE (endpoint + CA asserted), kubeconfig artifact assembly, AccessEntry CRUD, get-token presign decode, DeleteCluster -> gone. Harness: EKS SDK client on AWSClient + WaitForEKSCluster{Active,Deleted} pollers. Wire e2e-eks into the Makefile and the single-node CI suite list. * test(eks): Phase 2 kubectl get-nodes data-plane gate Add the reachability shim (harness/eks_mgmt.go) that reads the control-plane VM's br-mgmt IP from the cluster KV meta over NATS, plus a kubectl wrapper (harness/kubectl.go) that skips when kubectl is absent. KubectlGetNodes points kubectl at https://<mgmtIP>:6443 (insecure-skip-tls-verify; cert SANs the NLB DNS, not the mgmt IP) with an aws eks get-token exec block and polls until the control-plane node is Ready — the get-token webhook auth regression gate. IRSA pod + the real NLB/Route53 passthrough are deferred to a follow-up. * test(eks): tolerate doubled NotFound code + dial loopback NATS DeleteCluster detection matched the SDK constant ResourceNotFoundException, but awsgw emits ResourceNotFoundExceptionException on the wire — match the NotFound code by substring so deletion is detected immediately instead of timing out. The mgmt-IP shim dialed NATS.Host verbatim (0.0.0.0:4222), which the serving cert does not SAN; rewrite the wildcard bind to loopback. Validated live: all 6 subtests green, kubectl get nodes shows the control-plane node Ready via the get-token webhook auth path. * feat: build system images with libguestfs (no host block devices) Replace the qemu-nbd + sudo mount + chroot + /dev-bind flow in build-system-image.sh with libguestfs (virt-customize / virt-resize / guestfish). Every operation runs in an isolated userspace appliance VM with its own kernel, so the build touches no host block device, no host mount, and needs no sudo — removing the failure class that wiped the host's /dev on shared CI runners (a leaked /dev bind traversed by rm -rf). Applies to all distros: alpine resizes via guestfish e2fsck+resize2fs, ubuntu via virt-resize; customization (packages, binaries, files, setup script, service enable, cache cleanup) is one ordered virt-customize invocation. LIBGUESTFS_BACKEND=direct (no libvirt on runners). Wire the eks-node AMI into the single-node e2e job: the runner builds the device-free raw, ships it to the VM, and the VM imports it (predastore/NATS are VM-only) so the eks suite's CreateCluster has an AMI to boot. * feat(eks): bootstrap + state-report via AWS gateway broker Converge the EKS control-plane → host channel onto the ELBv2 lb-agent model: the K3s server VM publishes its bootstrap envelopes and periodic state reports via SigV4-signed HTTPS POST to the AWS gateway, instead of dialing core NATS directly. A new gateway action (PublishInternal, POST /clusters/{name}/internal-publish) relays the payload verbatim onto the existing eks.bus.* / eks.state.* subjects, so the daemon-side subscriber, KV persistence/encryption, and reconciler are unchanged. This removes the VM's need to reach NATS at all, so NATS can stay cluster-internal behind its bind/firewall/mTLS controls — the prior direct path broke when NATS was not VM-reachable (e2e CreateCluster timeout) and was DDIL-fragile (fire-and-forget nats pub, no ack/retry). - gateway/eks: PublishInternal broker + route; closed-set channel/kind validation - cmd/eks-gateway-publish: one-shot SigV4 publisher (retry-until-ack) baked into the eks-node AMI; replaces the nats-cli publisher in k3s-first-boot.sh and mulga-eks-state-report.sh - handlers/eks + daemon: hand the K3s server VM the mgmt-reachable gateway URL + system (Predastore) SigV4 creds + gateway CA instead of NATS URL/token/CA; new resolveSystemGatewayBaseURL (mgmt-bridge-first) - eks-node image: build/install eks-gateway-publish, drop nats-cli download * feat: migrate eks-token-webhook to the AWS gateway broker The TokenReview webhook dialed core NATS directly (STS verify + AccessEntry KV read). On a mgmt-only control-plane VM with NATS bound off-subnet that fails, so the webhook exited at startup, the apiserver token-webhook kubeconfig was never written, and the apiserver crash-looped — the cluster never reached ACTIVE. Converge it onto the same gateway-broker model as bootstrap/state-report and ELBv2's lb-agent: the webhook SigV4-signs an HTTPS POST {accountId,token} to /clusters/{name}/token-review, and the gateway runs the verify + lookup host-side and returns the resolved identity. The VM now speaks no core NATS. - internal/eksgw: shared on-VM SigV4 client (also adopted by eks-gateway-publish). - handlers/eks/token_review.go: Authenticate decision core (moved from the webhook binary) + ResolveTokenReview host-side wiring. - gateway/eks token-review route + WebhookTokenReview handler. - webhook: drop NATS, relay via eksgw.Client; broker fault -> 503 (retryable) not a silent deny. - initd: export EKS_GATEWAY_* / EKS_ACCESS_* creds, not SPINIFEX_NATS_*. * fix: build eks-node helpers with GOFIPS140; cover token-review broker eks-gateway-publish blank-imports internal/fipsboot, whose init() panics unless the binary is built with GOFIPS140=v1.0.0. The eks-node AMI build omitted the flag, so the binary paniced at first boot right after k3s came up — k3s-first-boot (set -eu) died before publishing the bootstrap token and the cluster never reached ACTIVE. Build both eks-node helpers with GOFIPS140=v1.0.0, matching cmd/lb-agent. Add NATS-backed unit tests for ResolveTokenReview and the WebhookTokenReview gateway handler (verify responder + account KV) to cover the new broker path. * fix: export first-boot env so eks-gateway-publish child sees it k3s-first-boot.sh and mulga-eks-state-report.sh sourced /etc/spinifex-eks/ first-boot.env but never exported the vars, so the eks-gateway-publish child process inherited an empty environment and exited '--account-id is required'. The control-plane VM came up healthy but never published its bootstrap envelopes, so the cluster never reached ACTIVE. Wrap the source in set -a / set +a so the KEY=value lines export to child processes. * fix: make awsgw start --host override actually bind The 'awsgw start --host' override printed 'Overwriting awsgw host' but the assignment was commented out, so AWSGW always bound the spinifex.toml host (a specific WAN IP, e.g. 192.168.0.10). A system instance (EKS K3s control plane, internal LB agent) has a mgmt-only NIC on-link to the br-mgmt IP with no route to the WAN IP, so its SigV4 POSTs to the gateway were refused and the cluster never bootstrapped. Honour --host (e2e passes 0.0.0.0:9999) so AWSGW listens on all interfaces incl. br-mgmt; the serving cert already SANs every local interface IP, and WAN clients still reach it. * test(eks): install kubectl on e2e VM so KubectlGetNodes runs The eks data-plane subtest (TestEKS/KubectlGetNodes) drives the full aws eks get-token -> kubectl -> apiserver -> token-webhook -> gateway broker -> STS verify path — the only end-to-end exercise of the token-review broker. The e2e binaries run on the VM, where kubectl was absent, so harness.NewKubectl t.Skip'd it and the path went unvalidated. Install a checksum-verified kubectl (pinned to the k3s server version) on the VM, gated on e2e-eks.	2026-06-07 20:42:25 +10:00
tomnewton-mulga	412eb618b0	Adding GPU passthrough e2e tests - intended to be run on Wattle	2026-06-05 13:47:11 +10:00
Josh Siviour	bedc35344f	feat(e2e/multinode) (#242) ... * feat(e2e/harness): multinode cluster wait helpers + AWSClientForGateway (Stage A) Stage A of mulga-siv-90. Compile-only; no tests wired yet. multinode_cluster.go: - Cluster.WaitNATSPeers polls /routez per node, dedupes by remote_id. want=2 on healthy 3-node, =1 after a node-down event. - Cluster.WaitPredastoreHealthy / WaitGatewayHealthy probe HTTPS per node via insecureHTTPClient (multinode certs are self-signed across peers). - Cluster.WaitDaemonReady runs DescribeInstanceTypes through every node's gateway, not just env.ServiceIPs[0] — catches a daemon that's only registered on one node. multinode_aws_via.go: - AWSClientForGateway clones *Env, swaps ServiceIPs to a single node's Addr, returns a fresh AWSClient. Lets phases 6/7 issue describe/ stop/start across every node in turn (bash aws_via equivalent). Init/Join helpers deferred to the pseudo-multinode bead (mulga-siv-91) where they belong — real-multinode bootstrap is owned by tofu + bootstrap-install.sh before the Go suite starts. * feat(e2e/multinode): TestMultinodePreflight + TestMultinodeClusterHealth (Stage B) Stage B of mulga-siv-90. First runnable Test* against a 3-node cluster. multinode/main_test.go: - Package-scoped singleton fixture, mirrors single/main_test.go. - requireMultiNodeFixture skips when SPINIFEX_E2E unset or mode != multinode. - Fixture = {Env, AWS, Harness, Cluster, Artifacts, TmpDir} — adds Cluster to the single-node shape via harness.ClusterFromEnv (reads SPINIFEX_NODES + SPINIFEX_SSH_USER/KEY). - TestMain drains harness.Fixture.Close + removes TmpDir at process exit. multinode/helpers_test.go: - needAZ / needInstanceTypeArch / needAMI shims (memoized via harness). - readyNodeCount substring counter — matches bash `grep -c "Ready"`. multinode/tests_test.go: - Top-level Test* wrappers (TestMultinodePreflight, TestMultinodeClusterHealth). multinode/preflight_test.go (phase 1): - /dev/kvm writable check (no /dev/kvm = test environment is wrong, fail fast). - peer_ssh hostname on every remote node. Skips node[0] to match bash — the runner's loopback SSHd is not part of the contract. multinode/cluster_health_test.go (phase 2): - Cluster.WaitNATSPeers >= 2 per node. - Cluster.WaitPredastoreHealthy / WaitGatewayHealthy across nodes. - Cluster.WaitDaemonReady (DescribeInstanceTypes per gateway). - spx get nodes shows N Ready, where N = cluster size. - spx get vms returns (length only — content may be empty). * feat(e2e/multinode): TestMultinodeInstanceDistribution + GuestSSH (Stage C) Stage C of mulga-siv-90. Phase 3 (distribution) + phase 4 (guest SSH). harness/multinode_distribution.go: - InstanceHostingNode: SSH each peer, ps grep qemu-system | <id>. Reuses NewPeerSSH; returns nil on no match (caller decides fatality). - CountInstancesPerNode + AssertSpreadAcrossNodes: fan-out + minNodes gate. Bash phase 3 warns-on-stacking; Go matches by accepting minNodes=1 + caller-side log. harness/multinode_guest_ssh.go: - GuestSSHPort polls hosting node's qemu cmdline for hostfwd=tcp:... reusing the existing hostfwdRE from ec2helpers.go (single regex, two capture groups: host + port). - GuestSSHEndpoint prefers PublicIpAddress, falls back to hosting-node + hostfwd. Mirrors the phase 4 conditional. - GuestSSHReady polls `echo ready` via SSH with bash-equivalent -o flags. multinode/helpers_test.go: - needKeyPair (EnsureKeyPair shim). - needInstanceTrio: pkg-level sync.Once launches 3 nano instances on default VPC, registers terminate-on-process-exit. Phase 4+ reuse the same trio without re-launch. multinode/distribution_test.go (phase 3): - launch trio + assert >= 1 node hosting (log warn on stacking). - spx get vms contains every trio ID. multinode/guest_ssh_test.go (phase 4): - For each trio instance: resolve endpoint, poll SSH ready, assert id reports ec2-user, log lsblk line count. multinode/tests_test.go: TestMultinodeInstanceDistribution + TestMultinodeGuestSSH. * feat(e2e/multinode): TestMultinode{Volume,CrossNodeGateway,CrossNodeOps} (Stage D) Stage D of mulga-siv-90. Phases 5/6/7 — pure AWS API exercise (no node-state churn). volume_test.go (phase 5): - Create 10GiB volume in discovered AZ, attach to trio[0] at /dev/sdf, detach, delete. Inline lifecycle — DeleteVolume in-test would race a registered Ensure* cleanup, surface as InvalidVolume.NotFound. gateway_test.go (phase 6): - Baseline DescribeInstances via the local gateway, then fan out AWSClientForGateway per non-local node. Require.Equal across all — cross-node fan-out regression surfaces immediately. crossnode_ops_test.go (phase 7): - Find trio[0]'s hosting node, stop via a different gateway, start via a third (or same other on 2-node fallback). Returns to running so later Test*s (phase 4 SSH, etc.) don't cascade-fail. otherTwoGateways mirrors the bash fallback chain. * feat(e2e/multinode): TestMultinodeNode{Failure,Recovery} (Stage E) Stage E of mulga-siv-90. Phase 8 (failure) + phase 9 (recovery) + diagnostic primitives. harness/multinode_node_lifecycle.go: - StopNode SSHes `sudo systemctl stop spinifex.target`. Non-fatal on exit error (bash matches — cluster shutdown races the SSH conn). - StartNode mirrors with sudo systemctl start. - WaitNodeServiceReady polls a single node's HTTPS gateway. Distinct from cluster-wide WaitGatewayHealthy so a recovering node can be tracked without re-checking peers. harness/multinode_diag.go: - DumpAllNodeLogs fans out journalctl -u 'spinifex-*' per node into artifactsDir/journal-<node>.log. Tolerates per-node SSH failure. - SpxGetNodesAcrossCluster strips trailing whitespace off SpxGetNodes. multinode/node_failure_test.go (phase 8): - Stop node2, register t.Cleanup restart safety net. - WaitNATSPeers want=1 (degraded), DescribeInstanceTypes via node1+node3, DescribeInstances still returns the trio. multinode/node_recovery_test.go (phase 9): - StartNode node2 (idempotent — phase 9 in isolation still works). - WaitNATSPeers want=2, WaitNodeServiceReady, spx get nodes 3 Ready, DescribeInstanceTypes via the recovered gateway. * feat(e2e/multinode): TestMultinodeVPCNetworking (Stage F) Ports phase 10 from run-multinode-e2e.sh. Stands up a fresh non-default VPC + subnet (10.200.0.0/16 + 10.200.1.0/24), launches 3 instances into the subnet, then asserts: - PrivateIpAddress is non-empty and matches 10.200.1.0/24 - All 3 IPs are unique - SubnetId / VpcId on the instance match the freshly-created ids - IPs persist across stop -> start cycles Independent of the package-level trio (which uses the default VPC) — this phase needs full control over subnet allocation so resources are created and cleaned up inline via t.Cleanup (LIFO: instances -> subnet -> VPC). * feat(e2e/multinode): TestMultinodePlacementAndNAT (Stage G) Ports phase 11 from run-multinode-e2e.sh — the largest single phase (~340 bash lines covering spread placement + NAT Gateway egress). Eight sub-steps preserved with matching harness.Step markers so the Go run cross-references cleanly with the bash log: Step 1: VPC + pub/priv subnets + IGW + main RT route + 2 SGs (bastion: tcp/22 anywhere; private: tcp/22 from bastion-sg, icmp from VPC) Step 2: bastion launch, wait SSH, scp key Step 3: spread placement group + 3 private instances via the group Step 4: validate spread (>=2 unique hosting nodes; >=3 = full PASS) Step 5: pre-NAT baseline — confirm NO internet from private VMs Step 6: allocate EIP + create NAT GW + private RT + 0.0.0.0/0 -> NGW Step 7: post-NAT — every private VM reaches 8.8.8.8 via NAT GW Step 8: inline teardown of NAT GW + private RT + EIP Each resource is owned by the test (no Ensure* memoization) so a failure mid-flight tears the full VPC graph down via t.Cleanup in LIFO order, matching the bash teardown sequence. NAT GW state polling is duplicated locally (waitForNATGatewayStateFatal/Best); the equivalents in single/natgw_test.go are unexported. Promote to harness/poll.go once a third consumer lands. * ci(e2e-go): add multi-node Go E2E job + cert/lb on 3-node cluster (Stage H) Adds e2e_go_multinode as a parallel job alongside e2e_go_single on the e2e-go workflow. The single-node job is untouched apart from extending its SUITES filter to drop the multinode suite (which only runs on the ci-multi cluster). What the new job does: - Provisions a 3-node tofu cluster via the same build-install-artifacts + bootstrap-install path as e2e_real_multi (bash). - Builds multinode.test / cert.test / lb.test on the runner, ships them to node1. - Runs each suite with SPINIFEX_MODE=multinode and the cluster csv in SPINIFEX_NODES / SPINIFEX_NODE_IPS. cert / lb branch internally on env.Mode so the same binaries cover both single + multi. - Pulls per-node journals (spinifex-journal-node1..N.log) for the e2e-analyze action so a node2/3 failure isn't blind. - Same JUnit + analyze + artifact-upload + status pipeline as the single-node job. Other moves: - Default SUITES now includes e2e-multinode so push triggers fan out across single + multi automatically. e2e_go_reboot stays explicit-opt-in (unchanged). - tests/e2e/Makefile: e2e-multinode now sets MODE=multinode so `make e2e-multinode` doesn't fall back to single (which would skip every TestMultinode* on the SPINIFEX_MODE check). - tests/e2e/TEST_COVERAGE.md: cross-reference the Go port at the head of the Real Multi-Node section. The bash driver stays in place as executable spec until the Go run is soak-tested. - e2e_go_status now needs e2e_go_multinode; multi=skipped counts as success so a single-only dispatch still posts green. * fix(e2e/multinode): NATS routez via peer_ssh + StartNode timeout + spx best-effort CI run 26161792146 surfaced three regressions in the freshly-landed multinode suite: 1. WaitNATSPeers dialled node.Addr:8222 directly. NATS monitor binds 127.0.0.1:8222 only (cmd/spinifex/cmd/templates/nats.conf:65), so every routez poll returned "connection refused". Switched to peer_ssh + curl http://127.0.0.1:8222/routez per node — matches the bash run-multinode-e2e.sh phase 2 pattern. 2. StartNode / StopNode capped ssh at a 30s context. spinifex.target has 6+ dependent units (predastore, NATS, awsgw, vpcd, daemon, ui) and a clean systemctl start runs 60-90s; the 30s budget was killing ssh mid-command ("signal: killed"). Bump StartNode to 5min and StopNode to 3min. 3. SpxGetVMs is best-effort in bash (`spx get vms --timeout 5s 2>/dev/null` with no exit-code check) because the CLI's NATS dial can race the cluster join shortly after bootstrap. The Go port fatal'd on non-zero exit and broke TestMultinodeClusterHealth + TestMultinodeInstanceDistribution downstream. Added harness.SpxRunBestEffort and rewired both call sites. * fix(e2e/multinode): spx CLI lenience + trio capacity retry + diag Run 26163994815 cascaded from two issues that bash silently tolerated. The Go port hard-failed where bash counts the failure and continues, so 3+ tests cascaded off transient first-failures. - spx get nodes: cluster_health + node_recovery now downgrade <Ready count to WARN with SpxRunBestEffort, matching bash phase 2 fail_test counter and phase 9 `|| echo ""`. Gateway+daemon health checks above already prove end-to-end NATS routing. - needInstanceTrio: retry RunInstances up to 6× on InsufficientInstanceCapacity (10s backoff). Daemon resourceMgr can briefly report 0 capacity right after node bootstrap before its first inventory scan completes — AWS itself surfaces this transiently. Stagger 2s between launches to match bash phase 3. - CI workflow: new "Pre-flight diagnostics on node1" step dumps groups/perms/spx output/systemctl status/journal tail to preflight-diag.log so next run's artifacts surface what tf-user sees before the test binary starts. * fix(e2e): strip SPINIFEX_* from spx child env Root cause of `spx get nodes` returning "nats: no servers available for connection" in the multinode Go suite (mulga-siv-90 runs 26163994815, 26195522455): spx CLI calls viper.AutomaticEnv() with prefix "SPINIFEX" inside config.LoadConfig. The CI workflow exports SPINIFEX_NODES (csv of cluster WAN IPs) so harness.ClusterFromEnv can build the *Cluster. viper picks SPINIFEX_NODES up as the `nodes` config key and silently clobbers the loaded `[nodes.*]` map → nodeConfig.NATS.Host = "" → nats.Connect("") → ErrNoServers. Bash never set SPINIFEX_NODES so the bash port was unaffected. Direct ssh `spx get nodes` from the runner also worked because the env was clean. Only the test-binary-spawned spx hit the collision. Fix: SpxRun / SpxRunBestEffort build a sanitized child env that preserves the small SPINIFEX_* set spx's root command explicitly BindEnvs (config path, AWS keys, host, base-dir, NATS dial overrides) and drops everything else. * fix(vpcd): make handleAddNAT idempotent (mulga-siv-124) PR #232 (commit 5887927) added a second vpc.add-nat publish in daemon/vm_adapters.go:onInstanceUpHook to cover instance recovery. The publish fires on every instance up event including fresh launches, racing the original publish from handlers/ec2/instance/service_impl.go. Both publishes are queue-grouped to "vpcd-workers" so two cluster vpcd instances handle them ~13s apart. handleAddNAT's existing DeleteAllNATsByExternalIP -> AddNAT sequence is not idempotent under concurrent writers: the second invocation deletes the rule the first installed (after ovn-northd compiled SB flows and chassis installed them), creating a flow-install gap during which the EIP is unreachable. Symptom: TestMultinodePlacementAndNAT fails with ssh: Connection timed out to the bastion EIP for 150s (mulga-siv-90 CI runs 26195522455, 26197248656). Bash port last green on e2e-baremetal run 26008057481 before #232 merged. Fix: before delete+add, look up the existing rule via FindNATByExternalIP. If it exists and its LogicalIP + spinifex:vpc_id ExternalID (plus ExternalMAC + LogicalPort under distributed-NAT mode) already match the target, no-op. Falls through to the existing delete-then-add for the IP-reuse-across-VPCs case the original code already handled. Adds TestTopologyHandler_AddNAT_Idempotent covering the UUID-witness + flow-barrier counter (two adds with identical state must not recreate the row or re-arm the barrier). * test(e2e/multinode): bump bastion SSH wait + ICMP probe Run 26198221557 confirmed the vpcd dnat_and_snat race (mulga-siv-124) was fixed — no "cleaned up stale NAT" lines for the bastion EIP, only one writer + an idempotent skip. But TestMultinodePlacementAndNAT still timed out at exactly 150s on SSH to the bastion's public IP, suggesting a separate OVN flow-install lag on the freshly-created VPC's external switch. - Bump waitForBastionSSH budget 150s -> 5min. Default-VPC GuestSSH reaches sshd in ~70s; a brand-new VPC needs longer for ovn-northd to compile SB + every chassis to install the external ARP responder + dnat_and_snat flow. 5min covers cloud-init + OVN warm. - Add an ICMP probe per iteration. Pure diagnostic — ping failure does not fail the step, but the next iteration's SSH error carries ping_output so the log splits "datapath down" (ping fail) from "sshd not up" (ping ok, ssh refused/timeout). Matches the RCA agent's fix candidate 3. * test(e2e/multinode): dump cluster datapath state on phase 11 failure waitForBastionSSH currently fails opaquely when the EIP datapath is broken — the log shows "ping=fail" but doesn't pin which node owns the gw chassis or whether ARP / OVN NAT rules / chassis encap IPs are sane. Register a t.Cleanup hook (after bastionPubIP is known) that fans out arp / ip route / ovs-vsctl / ovn-sbctl / ovn-nbctl probes to every cluster node and writes results to <Artifacts>/diag-<node>-<probe>.log on failure only. Plus a local-side ping / traceroute / ip route block so a future post-mortem can distinguish runner-network breakage from cluster datapath breakage. Best-effort: per-node SSH errors log but never re-fail the test. * test(e2e/multinode): use IpPermissions form for SG authorize calls Two AuthorizeSecurityGroupIngress calls in phase 11 used the legacy flat form (top-level IpProtocol/FromPort/ToPort/CidrIp) instead of the IpPermissions array. The Spinifex VPC handler at spinifex/handlers/ec2/vpc/security_group.go:667 reads only input.IpPermissions and silently drops the flat fields — newRules=0, no error returned. Run 26200626265 journal: groupId=sg-922de57bc74b85c9c (natgw-bastion) newRules=0 groupId=sg-758b05b4a5e7161fd (natgw-private) newRules=1 Bastion SG had zero ingress rules; TCP SYN to bastion :22 was dropped by ACL, causing waitForBastionSSH to time out after 5min. Bash version passes because AWS CLI v2 normalizes --protocol/--port/--cidr shorthand into IpPermissions.N.* on the wire. Fix: re-shape both calls (bastion tcp/22, private icmp) to use IpPermissions, matching the third authorize call in the same function (line 163-175) that already works. Server-side fix to honor flat form follows in a separate bead.	2026-05-21 15:40:00 +10:00
Josh Siviour	050ac8f861	feat(e2e): Go E2E harness + single-node migration to Ensure*/Discover* + reboot suite (#241) ... * feat(e2e): add Go test harness + cert scenario, port run-cert-e2e.sh Lands step 1 of docs/development/improvements/e2e-go-harness.md: shared harness primitives (env, wait, artifacts, cert, tls) plus the TestCertIssuance scenario that supersedes run-cert-e2e.sh. All files gated by //go:build e2e so default builds and `make preflight` are unaffected. Bash script stays in place for one CI cycle before deletion. * feat(e2e): port run-lb-e2e.sh to TestLoadBalancer Go scenario Step 3 of docs/development/improvements/e2e-go-harness.md. Replaces 1246 lines of bash with a typed Go scenario exercising all 4 LB variants (ALB/NLB × internet-facing/internal) against a shared VPC + app instance pool. Internet-facing phase auto-skips when no peer node is configured; internal phase always runs. Adds shared harness primitives: - harness/aws.go: AWS SDK client factory wired to Spinifex AWS gateway - harness/lb.go: WaitForLBActive, WaitForTargetsHealthy, WaitForENICleanup, WaitForInstanceRunning/Terminated, InstanceENI - harness/traffic.go: HTTPRoundRobin, TCPRoundRobin, distribution asserts - harness/ssh.go: minimal peer SSH wrapper shelling out to /usr/bin/ssh (replaced by typed x/crypto/ssh client hoisted from DDIL in mulga-siv-74) User-data scripts embedded via //go:embed under lb/testdata/. Bash script stays in place for one CI cycle before deletion. * fix(e2e): log client launch params + dump daemon log on InvalidInstanceType Adds diagnostics for the LB internal client launch path. Logs the exact InstanceType, AMI, subnet, key name, and user-data size right before RunInstances, and registers a daemon-log dump on failure so we can correlate the 400 InvalidInstanceType with what spinifex saw. * refactor(e2e): serialise LB suites + share probe client across internal tests Splits TestLoadBalancer into 4 sequential subtests (ALB/NLB × internet-facing/internal). Each suite creates its own TG + LB + listener, runs its traffic test, and tears down before the next starts. Replaces the per-suite throwaway client VM with a single long-lived probe server launched at fixture setup; per-suite internal traffic tests GET /probe?ip=...&proto=... against it instead of cold-booting a fresh client. Drops peak instance count from 5 to 4 (2 app + 1 LB + 1 client), so the internal phase passes on capacity-constrained dev nodes that previously hit InvalidInstanceType (see mulga-siv-76, mulga-siv-77 for the underlying spinifex bugs). * feat(daemon): make hostReserve env-configurable for dev capacity squeeze Adds SPINIFEX_RESERVED_VCPU and SPINIFEX_RESERVED_MEM_GB env-var overrides read at daemon startup. Invalid values are logged and ignored so a typo doesn't silently widen the reserve. Unblocks the LB E2E on capacity-tight dev hosts (8 vCPU box: default 2-vCPU reserve + 3x t3.nano test load leaves 0 free for the sys.micro LB VM). Lays groundwork for the planned operator-tunable [capacity] config. bead: mulga-siv-78 * fix(e2e): revert LB to per-suite client + fail-fast on terminal LB state Reverts the shared probe-server VM that wouldn't come up reliably during the InternalFacing suites — back to the proven bash pattern of launching a fresh client VM per suite (one of ALB_IP / NLB_IP populated based on the suite kind). Adds renderClientUserData + a conditional template so each client probes only its target LB. WaitForLBActive now bails immediately on terminal LB states (failed, provisioning_failed) and emits a "still waiting" log line every 30s so hangs are visible instead of silently burning the full 5-minute timeout. * ci(e2e): add e2e-go.yml workflow + GOTESTSUM Makefile opt-in Standalone workflow runs the new Go-based E2E harness on the existing ci-single self-hosted runner — provisions a tofu VM, bootstraps spinifex, installs gotestsum, runs each requested suite via the e2e Makefile, and uploads per-test artifacts + JUnit XML + JSON streams as a build artifact. Test summary parses the JUnit and emits ::error file=...:: annotations so failures surface in the PR diff. Gated to the feat/e2e-go-harness branch + workflow_dispatch so it doesn't interfere with the existing e2e.yml while the Go conversion lands. tests/e2e/Makefile gains a GOTESTSUM=1 opt-in: when set, the e2e target runs gotestsum with --junitfile and --jsonfile rooted in ARTIFACT_DIR under a suite-tagged filename derived from the first PKG component. * feat(e2e): share probe client across both internal LB suites + SG fixes - Replaces per-suite probe client with a single shared VM launched in the fixture. Saves ~60s by avoiding a cold boot between the ALB and NLB internal suites. - Client cloud-init exposes a JSON trigger endpoint on :9090 in addition to the http.server on :80 (results dir). Each suite POSTs {proto, ip, count, outfile, http_port, tcp_port} to run a probe loop against the LB private IP and writes the result file under /tmp/httpd for the driver to fetch back. - configureDefaultSG now uses the structured IpPermissions form (top-level shortcut is silently ignored by vpcd — tracked separately) and opens port 9090 in addition to 80 + 9000. - deleteLB captures the underlying sys.micro VM id via the LB ENI before delete, then polls DescribeLoadBalancers until LoadBalancerNotFound so the next suite doesn't race the capacity reclaim. - Internal_NLB sleeps 15s up front to let the ALB sys.micro deallocate fully before its own createLB fires. DescribeLoadBalancers can report the ALB gone before the vCPU/memory allocation is actually released on capacity-tight dev hosts; without this settle, NLB's createLB trips the sys.micro reserve and lands in terminal state=failed. Still cheaper than cold-booting a per-suite client. * ci(e2e): install Go on tofu VM before running Go E2E suites The tofu-provisioned VM ships with the spx binary and bash test scripts but no go toolchain, so the prior workflow's "Install gotestsum on VM" step failed with `bash: line 2: go: command not found`. Add a curl-based Go 1.26.3 install (matching go.mod) ahead of the gotestsum install and update the test step's PATH so make e2e-* runs the freshly-installed go binary. * ci(e2e): bump gotestsum to @latest for Go 1.26 compatibility gotestsum v1.12.0 pins golang.org/x/tools v0.19.0 which fails to compile under Go 1.26 with: invalid array length -delta * delta (constant -256 of type int64) Switching to @latest pulls a newer x/tools that builds clean. * ci(e2e): ship spinifex source to VM so make -C tests/e2e can run there Single-node tofu VM only has the spx binary + bash test bundle at ~/e2e-tests; the Go e2e workflow was running 'make -C tests/e2e e2e-cert' from $HOME/Development/mulga/spinifex which doesn't exist, producing: make: *** No rule to make target 'e2e-cert'. Stop. Add a checkout of the spinifex repo on the runner and rsync it (minus .git, build/, bin/, node_modules) to the VM as ~/Development/mulga/spinifex before installing Go + gotestsum. Interim solution; mulga-siv-80 tracks replacing the rsync with shipping pre-built test binaries from the runner. * ci(e2e-go): replace rsync with tar-over-ssh; VM lacks rsync bootstrap-install.sh ships a minimal binary VM with no rsync; the ship-source and collect-artifacts steps failed with 'rsync: command not found'. Stream a tarball over ssh instead — same exclude list, no extra dependency on the VM. * fix(e2e): discover single-node IPs + awsgw bind in LoadEnv run-cert-e2e.sh single mode probes awsgw.host from spinifex.toml and adds every non-loopback global IPv4 to the node list. The Go harness defaulted to 127.0.0.1 only, so CI on a tofu VM (awsgw bound to the LAN IP, not loopback) saw 'connection refused' on every TLS check. Port the same discovery: ModeSingle now walks net.InterfaceAddrs for NodeIPs and grep-parses spinifex.toml for the awsgw bind. Both stay overridable via SPINIFEX_NODE_IPS / SPINIFEX_SERVICE_IPS. * ci(e2e-go): pre-build test binaries on runner; drop go toolchain from VM Replaces the source-tree + go + gotestsum payload that the e2e-go.yml workflow previously shipped to the tofu CI VM with self-contained test binaries built on the runner via `go test -c`. testdata for the LB suite is already //go:embed'd into the binaries, so the bundle on the VM is just `<suite>.test` files (cert: 14M, lb: 17M uncompressed). Workflow changes - actions/setup-go on runner. - New step builds binaries via `make e2e-build SUITES_BUILD=...`. - Install go-junit-report on runner; convert each VM-side test log to junit-<suite>.xml after artifact pull so "Publish test summary" still parses JUnit XML. - Drop "Install Go + gotestsum on VM" entirely (saves ~60s + network). - Drop source-shipping rsync/tar (saves ~30s + a lot of bandwidth). - VM step runs `./<suite>.test -test.v -test.timeout=30m` directly. Makefile changes - New `e2e-build` target compiles each `SUITES_BUILD` entry into `BIN_DIR/<suite>.test` via `go test -c -tags=e2e`. `clean` also removes BIN_DIR. Closes mulga-siv-80. * ci(e2e-go): disable setup-go cache; upload dominates self-hosted run time * refactor(e2e): hoist generic DDIL harness primitives to shared tests/e2e/harness Step 1.5 of docs/development/improvements/e2e-go-harness.md. Promotes Cluster/Node/SSH/DaemonClient/LocalStateRevision/ResetAllNodes/Witness into the shared package so the LB, multinode, and BM scenarios in steps 4-5 can build on the same primitives instead of duplicating SSH + cluster plumbing. Moves - ddil/harness/{cluster,ssh,daemon_client,state,reset,witness}.go -> harness/ (plus the witness testdata embed dir). - ddil/harness/{fault,netem,profiles,snapshot}.go -> ddil/fault/ as a new package; fault.PartitionNode/HealNode/KillNATS/StartNATS/ RestartDaemonOnly/ApplyNetem/ClearNetem/LinkProfile*/TakeSnapshot now take harness.Cluster / harness.Node / harness.SSH explicitly. - ddil/harness/retry.go::Run retry-with-reset extracted as harness.RetryWithReset in harness/wait.go; ddil/harness/quarantine.go keeps the DDIL-only quarantine wrapper that calls into it. - WaitForMode moved from fault.go to harness/daemon_client.go since it polls daemon state and has no fault-injection concern. - The pre-hoist harness/ssh.go (PeerSSH, subprocess ssh) is preserved as harness/peer_ssh.go so the lb_test suite keeps working unchanged; the typed x/crypto/ssh client from DDIL now owns harness/ssh.go. - New harness/util.go exports ShellQuote (was unexported shellQuote in netem.go) so both ddil/fault and reset.go/witness.go can use it. Env var rename - SPINIFEX_NODES / SPINIFEX_SSH_USER / SPINIFEX_SSH_KEY replace DDIL_NODES / DDIL_SSH_USER / DDIL_SSH_KEY. Old names still honoured with a stderr deprecation note (envWithLegacy) so the DDIL CI keeps passing until .github/workflows/e2e-ddil.yml drops the old names. DDIL_QUARANTINED stays DDIL-specific. DDIL parity preserved — scenarios still call ddilh.Run for quarantine semantics; the underlying retry-with-reset is the same code path, just shared. Closes mulga-siv-74. * fix(e2e): open default SG :22 ingress via structured form before witness launch The daemon currently drops the top-level shortcut AuthorizeSecurityGroupIngress form silently (mulga-siv-79), so the default SG starts with no inbound :22 and the witness EIP awaitBaseline tunnel times out under DDIL CI. Mirror the workaround already present in lb_test by issuing the structured IpPermissions form before LaunchWitnessVM. Idempotent on InvalidPermission.Duplicate so reruns in the same cluster are safe. * ci: render JUnit into job summary + collapsible groups in noisy steps Add a reusable composite action (.github/actions/junit-report) that parses a glob of JUnit XML files, writes a totals table + collapsible per-failure details into $GITHUB_STEP_SUMMARY, and emits one ::error annotation per failed/errored test case so failures surface in the run header instead of being buried in step logs. Apply the action to both e2e-go.yml and e2e-ddil.yml (e2e-ddil gotestsum now also emits --junitfile so the action has input). Same workflows also wrap noisy phases (tofu apply/destroy, scrub-env, tar-over-ssh, per-suite test runs, cluster-log collection) in ::group::/::endgroup:: markers so the Actions UI collapses the chatter by default and the human can expand only the failing phase. Tofu output still tees to $TOFU_LOG so the artifact bundle is unchanged. Future workflows (multinode, baremetal) can adopt the same pattern via the README in .github/actions/junit-report. * test(ddil): quarantine Scenario C until predastore §2 lands CI run 26016840819 surfaced that ScenarioC_CleanPartition silently depends on predastore-ddil-hardening §2 (write-quorum with deferred parity repair). Root cause from node3 daemon log: predastore: write udp ...->192.168.0.22:9991: sendmsg: operation not permitted predastore: putObjectViaQUIC: put failed spx QMP: BLOCK_IO_ERROR instanceId=<witness> spx: StopOrTerminateInstance: Terminating spx: Instance state transition running → shutting-down The 'sendmsg: operation not permitted' is the EPERM Linux returns when iptables drops an outbound packet — exactly the OUTPUT DROP rules PartitionNode installs. The witness VM's block I/O is backed by distributed predastore, so the same partition that severs node3↔peer NATS traffic also severs the witness disk's cross-shard QUIC writes. QEMU power-downs the guest after BLOCK_IO_ERROR, the daemon's reconciler terminates the instance, and the next AssertProgressed dials a sshd that's already gone (hence the 'handshake failed: EOF' the test reports). Quarantine by collapsing C to the same scenarioSkip stub D/E/F already use, citing predastore-ddil-hardening §2 as the dependency. Flip C back to ENABLED in the same PR that lands §2 — that's when the witness disk can survive a peer partition without QEMU halting. TEST_COVERAGE.md row C flipped ENABLED → SKIPPED; added a quarantine note explaining the predastore dependency. * ci: colored phase banners + quiet-on-success tofu output Pretty up the live Actions log without flooding it. Three changes: 1. .github/scripts/ci-fmt.sh — sourceable bash helpers (banner, ok, bad, warn, note, quiet_run) using ANSI colors that the Actions UI renders. 2. e2e-go.yml + e2e-ddil.yml — tofu phases switch back to quiet redirect into $TOFU_LOG (the previous tee'd version regressed live noise vs the original silent redirect). Each phase now prints one colored ok/✗ line; on failure the tail of $TOFU_LOG dumps inside a single collapsed group, so the live log stays scannable but the failing tail is one click away. 3. junit-report render.py — compact summary: short status icons (✅/❌) in a single column, no separate err column, single-line failure list (truncated to 120 chars) instead of <details> blocks that wrap badly on narrow screens. Per-suite test runs still wrap in ::group:: with an inline cyan banner so pass output collapses and the suite name is obvious when expanded. * test(e2e): add harness.Phase / Step / Detail for human-readable test output Test fixtures were emitting flat walls of t.Logf with every line prefixed by the same `lb_test.go:61:` (file:line of the call site that the t.Helper() chain bubbles up to) and forced 6-space indent. Hard to skim, hard to find where one phase ends and the next begins. Add three helpers in tests/e2e/harness/log.go: Phase(t, fmt, ...) — flush-left colored banner via fmt.Println. Bypasses t.Logf so the line lands without the testing framework's indent/file:line prefix. Pure visual structure; the diagnostic detail stays on t.Logf. Step(t, fmt, ...) — single-line sub-phase marker via t.Logf. Keeps file:line attribution so a triager can find the source of a progress message. Detail(t, k, v, ...) — key=value resource IDs via t.Logf. Same attribution rationale as Step — IDs are diagnostic and want to land in JUnit XML bucketed with the owning test. Why the split: Phase is purely a banner; losing JUnit attribution costs nothing because banners aren't diagnostic. Step/Detail carry triage context (instance IDs, AMI, etc.) and a failing test's JUnit testcase node should still contain them, so t.Logf attribution is kept. ANSI colors are auto-disabled when stdout isn't a TTY and GITHUB_ACTIONS isn't set, so piped local `go test -v` stays clean. Apply to lb_test setupSharedFixture — five Phase banners replace the implicit run of helper logs ("Discovering cluster capacity", "Ensuring SSH key pair", "Creating shared VPC topology", "Launching app instances", "Launching probe client") so the live CI log reads as a story rather than a flat resource dump. * ci(e2e-go): strip indent + file:line on display, propagate color env over SSH Two display bugs spotted on the previous run: 1. Phase banners came out white instead of bold-cyan because the test binary runs over SSH on the tofu VM, where neither GITHUB_ACTIONS nor a TTY is present, so harness.log.go init() disables ANSI escapes. Fix: export GITHUB_ACTIONS=true and FORCE_COLOR=1 in the remote env so the helper keeps emitting color codes. 2. Every t.Logf line still rendered as lb_test.go:61: instance type: t3.nano (4-space indent + `<file>:<line>:` prefix) because t.Helper() bubbles file:line to the call site. Fix: pipe the test binary's stdout through `sed -E 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'` so the display strips both. Tee to the artifact log happens BEFORE the sed so raw output is preserved for triage; only the live UI sees the cleaned form. * feat(e2e-harness): extend harness for single-node Go suite port Stage A of mulga-siv-89 — add the primitives the single-node run-e2e.sh port consumes: - AWSClient gains IAM + STS service handles, plus NewAWSClientWithCreds for per-tenant account-scoping in Phase 8. - awserror.go: ErrorCodeIs, AssertAWSError, ExpectError — Go equivalent of the bash expect_error helper used throughout the negative phases. - poll.go: WaitForInstance/Volume/Snapshot/Image state to any target, with per-resource default timeouts tuned from the bash sleep loops. - spxcli.go: thin shellouts for spx get nodes/vms, spx top nodes, spx admin account create with output parsing for tenant-id/access-key. - rawhttp.go: signed raw POST against the AWS gateway for the Phase 8 InvalidAction probe (gateway rejects on auth before action parsing). - ovn.go: OvnNbctl/OvnSbctl wrappers + SkipIfNoOVN for Phase 8b-e datapath checks; WaitForPortGroupMember handles propagation polls. - ec2helpers.go: InstancePublicSSHHost (public-IP-or-qemu-hostfwd), LsblkRootGiB, serial-console-access trio, DiscoverDefaultVPC, idempotent AuthorizeSSHIngress. - account.go: AccountCarousel maps tenant name -> scoped AWSClient, built atop NewAWSClientWithCreds for the Phase 8 isolation tests. Pure groundwork — no new tests, no production behaviour change. The cert + lb scenarios still compile against the extended harness. * ci(e2e-go): auto-cancel + reap orphans to prevent runner wedge A cancelled run on the single-runner ci-single host has been leaving orphan tofu/terraform procs (and stale tfstate locks) behind, which caused the next push's tofu destroy to hang for 45m+ and saturate the runner host until sshd refused new connections. - Add `concurrency: e2e-go-${ref}, cancel-in-progress: true` so a new push to feat/e2e-go-harness cancels the in-flight run instead of stacking on the same self-hosted runner. - New `Reap Orphan Procs` step at job start: SIGTERM then SIGKILL any lingering `tofu`/`terraform` from the runner user, and unlink any stale `.terraform.tfstate.lock.info`. - Harden `Teardown`: drop `set -e` (scrub-env must always run as the libvirt-level fallback), reap orphans before destroy, and wrap `tofu destroy` in `timeout 600` so a wedged provider can't pin the job to its 45m timeout. * ci(e2e-go): drop push trigger, manual dispatch only Pull the on.push branch trigger so the workflow only fires from workflow_dispatch — matches the e2e-ddil.yml Phase-1 pattern. Push trigger lifts again once the single-node Go port (mulga-siv-89) is green end-to-end and run-e2e.sh is retired. * test(e2e-single): port run-e2e.sh phases 1-4 to Go Stage B of mulga-siv-89. Lands the TestSingleNode orchestrator + Fixture that threads state between sequential phase subtests, plus phases: - Phase 1: KVM check, AWS gateway TLS reachability, daemon NATS readiness polling, region/AZ smoke. - Phase 1b: spx get nodes / top nodes / get vms (single-node only). - Phase 2: nano instance-type + arch discovery. - Phase 2b: serial-console-access default/enable/disable round-trips. - Phase 3: create-key-pair + import-key-pair lifecycle; test-key-1 PEM kept for Phase 5 instance launch (cleanup deferred to teardown phase in stage G). - Phase 4: architecture-aware AMI discovery via name filter ami-ubuntu-24.04-{x86_64,arm64}, falls back from spx import output. Phase ordering is contractual (Phase 5+ depend on AMI, key pair, AZ). detectPoolMode reads [network] external_mode from spinifex.toml so stages F/G can skip pool-only phases on dev_networking fixtures. No production code changed. cert + lb scenarios continue to compile. * test(e2e-single): port run-e2e.sh phases 5-5e to Go Stage C of mulga-siv-89 — the instance/volume/snapshot/image lifecycle that everything else hangs off: - Phase 5: run-instances (default VPC + default SG with tcp/22 opened via idempotent harness.AuthorizeSSHIngress), wait running, resolve root volume from BlockDeviceMappings. - Phase 5a-pre: spx get vms now contains the running instance. - Phase 5a: describe-instances metadata round-trip (type/key/AMI/BDMs). - Phase 5a-ii: TCP wait, ssh id, lsblk cross-check vs API root volume size (±1 GiB tolerance — lsblk rounds down), soft hostname check. - Phase 5a-iii: get-console-output round-trip on InstanceId. - Phase 5b: create-volume (10 GiB) → modify-volume (20 GiB) → attach-volume → detach-volume → delete-volume, polling each state transition through harness.WaitForVolumeState. - Phase 5b-ii: describe-volume-status on the root volume. - Phase 5c: create-snapshot → wait completed → copy-snapshot → both visible → delete original → copy survives → delete copy. - Phase 5d: AMI snapshot-backed launch verification via DescribeImages.BlockDeviceMappings[].Ebs.SnapshotId. Predastore-side config.json reads are TODO (no S3 client in harness yet). - Phase 5e: create-image --no-reboot, wait available, capture backing snapshot ID for Phase 9 cleanup. Phase 7a (attach to stopped → error) stubbed in volume_test.go for Stage D to fill once the lifecycle stop/start lands. * test(e2e-single): port run-e2e.sh phases 5f-8 to Go Stage D of the single-node Go port (mulga-siv-89). Adds four files under tests/e2e/single/ covering Phase 5f (SG egress ACL enforcement), Phase 6 (tag CRUD), Phases 7 / 7a / 7b / 7c-pre / 7c (stop, start, attach-to-stopped negative, ModifyInstanceAttribute, reboot, multi-count RunInstances), and Phase 8 negatives (18 sub-tests asserting AWS error codes via harness.ExpectError / AssertAWSError / PostAWSAction). Wired into TestSingleNode in main_test.go; phase ordering preserved because Phase 7 chain mutates the shared instance state (stopped at end of 7, .small after 7b, running after 7c-pre, untouched by 7c siblings) and Phase 8 negatives need a running instance with the AMI/key created earlier. Removes the placeholder phase7a_AttachToStoppedError stub from volume_test.go (real impl now in lifecycle_test.go). run-e2e.sh stays in tree until step-7 cleanup per the parent plan. * ci(e2e-go): wire e2e-single target + add single to build set Lets the in-progress single-node Go port (mulga-siv-89, Stages A-D landed) be exercised in CI via explicit `suites: e2e-single` dispatch input ahead of Stage G flipping the workflow default. Workflow default suites unchanged until Phase 9 teardown lands. * test(e2e-single): port run-e2e.sh IAM 1-7 + Phase 8 account scoping to Go Stage E of the single-node Go port (mulga-siv-89). Adds two files under tests/e2e/single/: - iam_test.go (903 lines): IAM Phases 1-7 — User CRUD, Access Key Lifecycle, User Authentication (via harness.NewAWSClientWithCreds), Policy CRUD (+ CreatePolicyVersion / SetDefaultPolicyVersion / DeletePolicyVersion which bash lacks), Policy Attachment & Enforcement (allow / deny / wildcard / root bypass), Policy Lifecycle (DeleteConflict while attached), and idempotent Cleanup. - account_test.go (1186 lines): Phase 8 EC2 Account Scoping — 11 Steps (Account Setup, Instance / Volume / KeyPair / Snapshot / VPC-Subnet / IGW-EIGW Scoping, Account Settings, Global Resources, Edge Cases, Cleanup). Uses harness.AccountCarousel for the multi-profile credential map and SpxAdminAccountCreate for admin account bootstrap. Fixture extended with IAM cross-phase state (IAMAdminAccount, IAM{Alice,Bob,Charlie}{KeyID,Secret}) so Phase 7 cleanup is defensive even when an intermediate IAM phase short-circuits. All 11 Phase 8 Steps share local state in the outer function; t.Cleanup mirrors Step 11 LIFO so a mid-Step failure still tears down accounts. Wired into TestSingleNode. run-e2e.sh stays in tree until step-7 cleanup per parent plan. * fix(e2e-harness): use kebab-case is-default filter for DescribeVpcs AWS EC2 DescribeVpcs filter names are kebab-case (is-default), and the daemon's filter allowlist enforces that. The harness sent the camelCase SDK field name (isDefault), so the daemon returned InvalidParameterValue 400 and DiscoverDefaultVPC fataled — leaving Phase 5 InstanceID empty and cascading 16 downstream Single-Node E2E subtests to fail. * test(e2e-single): port run-e2e.sh phases 8b-8e to Go Stage F of the single-node Go port (mulga-siv-89). Adds three files under tests/e2e/single/: - vpc_test.go (553 lines): Phase 8b (pool-mode VPC public/private subnet E2E — CreateVpc, dual subnets with MapPublicIpOnLaunch assertions, IGW attach, custom route table + 0.0.0.0/0 route, instance launch + SSH + external connectivity probe) and Phase 8c (route table validation — main RT association, local + IGW route presence, custom RT CRUD + association/disassociation, negative delete-local + delete-main assertions). - natgw_test.go (467 lines): Phase 8d (pool-mode NAT GW E2E — CreateNatGateway with EIP, private subnet route via NAT, bastion- hopped SSH into private VM, baseline egress drop / post-NAT egress succeed / post-teardown egress lost). Inline waitForNATGatewayState helper bounds the pending->available->deleted state machine via harness.EventuallyErr. - datapath_test.go (326 lines): Phase 8e (OVN-gated SG-to-SG datapath — two SGs with UserIdGroupPair ingress rule, OVN port-group membership + SB address-set wait loops, allowed traffic curl, denied traffic nc, revoke->drop, re-add->restore). Pool-mode gating via fix.PoolMode for 8b/8d; OVN gating via harness.SkipIfNoOVN for 8e. Per-phase t.Cleanup LIFO ensures teardown unwinds even on partial failure. Wired into TestSingleNode. * test(e2e-single): port run-e2e.sh phases 9, 9a, 9b to Go Stage G of the single-node Go port (mulga-siv-89) — final teardown + post-teardown verification. Adds tests/e2e/single/teardown_test.go (226 lines): - phase9_Teardown: DeleteSnapshot(CustomAMISnapID) -> DeregisterImage (CustomAMIID) -> TerminateInstances(InstanceID) + wait terminated -> DeleteKeyPair(KeyName). Each call swallows *NotFound via harness.ErrorCodeIs; unexpected errors logged as warnings so 9a/9b always run. - phase9a_VerifyTeardown: DescribeInstances must show terminated or NotFound; DescribeImages must NOT include the deregistered AMI; DescribeKeyPairs must NOT include the deleted key. Clears the consumed Fixture fields. - phase9b_FinalClusterStats: harness.SpxGetVMs must NOT list the terminated InstanceID; orphan i- rows surface as soft Logf warnings to avoid flakes when parallel suites share the node. Wired into TestSingleNode. With this, mulga-siv-89's single-node Go port (Stages A-G) is feature-complete; run-e2e.sh remains in tree until a separate cleanup bead deletes it after CI burn-in. * ci(e2e-go): unbuffer sed in test pipeline for live log streaming GNU sed defaults to block-buffered stdout when stdout is a pipe (GitHub Actions runner). Accumulates 4-64KB before flushing, so test output appeared in sporadic batches unaligned to test boundaries. -u flag forces line-buffered output so each go test -v line surfaces in the Actions UI as it is produced. Side effect: none. No process count change, no SSH session change, no test execution change. Pure output flushing on the runner host. * fix(e2e-single): persist key PEM across phases and skip unimplemented IAM ops Three bugs blocked the Single-Node Go E2E suite after the DescribeVpcs filter fix: 1. Phase 3 wrote the SSH PEM into `t.TempDir()`, which Go cleans up when the Phase 3 subtest exits. Phases 5a-ii / 5a-iii / 7b / 7c-pre then saw "Permission denied (publickey)" / "Identity file not accessible" and the cascade left the instance in a state that broke 7c-pre's `start-instances`. Move the PEM under a TestSingleNode-scoped `fix.TmpDir` so it survives until the whole test exits. 2. IAM Phase 2's `require.NotEmpty(t, fix.IAMAdminAccount, "")` was intended as a no-op (Phase 4 populates the field), but require.NotEmpty fails on empty regardless of message. Removed the bogus check. 3. The daemon does not yet implement `UpdateUser` (IAM Phase 1) or the policy-version family (IAM Phase 4): `CreatePolicyVersion`, `ListPolicyVersions`, `SetDefaultPolicyVersion`, `DeletePolicyVersion`. They return `InvalidAction` and abort the phase. Wrap each block in a t.Run sub-test with t.Skip so the surrounding phase still runs and the skips show up explicitly in the test report. * test(e2e): parallelise hot phases + tune fast-path polls Squash merge of feat/e2e-parallel-quickwins (mulga-siv-96). Four quick wins from decomposing run-e2e.sh into Go phase subtests: - Phase 8 negatives (single/negative_test.go): t.Parallel() inside each of the 18 read-only error-path sub-tests. ~5-10x speedup. - IAM Phase 4 policies (single/iam_test.go): EC2ReadOnly stays seeded sequentially (populates fix.IAMAdminAccount); the other four policies fan out as siblings under a wrapping create_policies_parallel t.Run. ~5x on the Phase 4 sub-step. - Volume + snapshot fast paths (single/{volume,snapshot}_test.go): pass harness.WithPoll(500ms) on the five state transitions that finish in <1s on local QEMU. Halves Phase 5b / 5c wall-clock. - Cert per-IP loop (cert/cert_test.go): t.Parallel() inside the 4 inner t.Run(ip, ...) blocks. Cuts cert phase by len(nodes). No harness changes, no Fixture changes, no test-execution-order changes. mulga-siv-96. * test(e2e-lb): resolve peer availability before shared fixture setup setupSharedFixture builds a VPC + 2 app VMs + probe client (minutes on capacity-tight dev hosts) and previously ran BEFORE the peer pickup + ssh.Ping. On single-node, the InternetFacing_* subtests would correctly t.Skip on peer == "" — but only after the fixture work and once subtests started, so the wait appeared to be attached to the first subtest name gotestsum printed. Reorder: pickPeer + ssh.Ping first, log "internet-facing will skip" immediately, then build the fixture for the internal subtests. No behavior change for multi-node + peer-reachable; on single-node the skip rationale surfaces before the fixture wait. * test(e2e-single): extend SSH wait budget to 3m on slow boot paths CI run 26034322018 timed out Phase5a_ii_SSH (60s budget) and Phase8b_VPCSubnetE2E (2m budget) on the TCP-port-open poll while sshd was still finishing pam/cloud-init. Bump all four wait sites to 3m as a stop-gap: - single/instance_test.go phase5aii_SSHProbe - single/lifecycle_test.go waitForSSHReady (Phase 7b/7c/7c-pre) - single/vpc_test.go phase8b_VPCSubnetE2E - single/natgw_test.go phase8d bastion SSH wait Proper fix tracked under mulga-siv-101 (hoist a harness.WaitForSSH that probes the actual handshake, not just tcp/22 reachability). * test(e2e-single): skip-gate IAM3/IAM5/IAM6 until daemon handlers land CI run 26034322018 surfaced three IAM daemon gaps: - IAM3_UserAuthentication: alice's freshly-created access key returns 403 from DescribeInstances — daemon doesn't honour scoped IAM key signatures yet. - IAM5_PolicyAttachmentEnforcement: cascades from IAM3 (alice can't sign). - IAM6_PolicyLifecycle: DetachUserPolicy returns 404 even on attachments Phase 5 confirmed — daemon attachment ledger gap. Skip-gate each phase with t.Skip + a pointer to mulga-siv-100 so the remaining IAM coverage (IAM1/IAM2/IAM4/IAM7) keeps surfacing regressions, and the suite as a whole doesn't fail on known gaps. Flip to real assertions once the daemon handlers land. * test(e2e-single): probe full SSH handshake (not just TCP port open) CI run 26036025293 hit "Connection timed out during banner exchange" on Phase5a-ii / 8b / 8d even with the 3m TCP-port-open budget: tcp/22 opens immediately, but sshd is still finishing pam/cloud-init, so the first real runSSH call afterwards races the half-open daemon. Switch waitForSSHReady to issue an actual ssh BatchMode + ConnectTimeout handshake against the target, retrying until the daemon completes the banner exchange. Drop the four TCP-only Eventually loops in favour of the upgraded helper: - instance_test.go phase5aii_SSHProbe - lifecycle_test.go waitForSSHReady (Phase 7b / 7c-pre / 7c) - vpc_test.go phase8b_VPCSubnetE2E - natgw_test.go phase8d_NATGateway (bastion) Signature: waitForSSHReady(t, host, port, keyPath). Backwards-compat alias waitForSSHHandshake exposed for the instance_test caller. Proper harness hoist tracked under mulga-siv-101. * test(e2e-harness): dual-emit Step + Detail for live CI streaming go test's testing framework buffers t.Logf output per-test and releases it only on PASS/FAIL. Long subtests (Phase8Acct_AccountScoping ~115s, Internal_NLB ~60s, Phase8b_VPCSubnetE2E 2-3m, Phase8d_NATGateway ~90s) therefore went silent in CI for the entire subtest duration, even with sed -u line-buffering in the workflow pipeline. harness.Phase already wrote to os.Stdout directly for this reason. Extend the pattern: have Step and Detail dual-emit — write the same line to os.Stdout (streams live) AND t.Logf (keeps JUnit per-test attribution). The duplicate on a failure is acceptable; both copies carry identical content. mulga-siv-103 (note: numbering skipped past siv-102 which was already filed in the parent branch). * test(e2e-harness): drop t.Logf duplicate from Step + Detail The dual-emit (Fprintln + t.Logf) caused every Step/Detail line to appear twice after long subtests: once live via stdout, once again when the testing framework released its per-test buffer at PASS/FAIL. Most visible after lb subtests (Internal_NLB ~60s) where the bulk release dumped ~dozens of duplicated lines at once. Drop the t.Logf copy. go-junit-report parses stdout between `=== RUN` and `--- PASS/FAIL` markers, so per-test attribution in JUnit XML is preserved without the duplicate. Refs: mulga-siv-103 * test(e2e-harness): stream lb-waiter progress live in CI WaitForLBActive, WaitForTargetsHealthy, WaitForENICleanup, and WaitForInstanceRunning emitted progress via t.Logf, which the testing framework buffers per-test. After the previous commit dropped the dual-emit, that meant every poll loop went silent until subtest end — visible in CI as 'nlb is active' hanging on screen for the whole duration of the subsequent traffic check. Route progress through harness.Step (direct stdout). Also emit a 'waiting for X' Step at the start of each helper so observers know what the silent stretches actually represent. Refs: mulga-siv-103 * test(e2e): dump VPC/IGW datapath diagnostics on Phase8b SSH timeout Phase8b's 3min SSH timeout on a fresh-VPC VM (CI run 26037892507) is the customer-visible surface of mulga-siv-105 — vpcd's handleIGWAttach returns before ovn-northd compiles + the gateway chassis installs flows. Two other cleanup-only bugs (siv-106 missing spinifex:cidr, siv-107 dnat_and_snat double-delete) showed up in the same triage. Keep the test hard-failing so the regression stays visible, but dump a triage bundle on timeout so each occurrence is actionable: - guest console output (last 4KB — boot + cloud-init state) - ovn-nbctl show + ovn-sbctl show - Logical_Router external_ids (confirms siv-106 reproduction) - NAT rules (confirms siv-105 + siv-107 surface) - Port_Binding chassis,up (confirms gateway port up state) Diagnostics go straight to stdout (Step/Detail rationale) so they stream live in CI rather than dumping at PASS/FAIL. Refs: mulga-siv-105, mulga-siv-106, mulga-siv-107 * ci(junit-report): drop parent t.Run containers from tally + list skips with reasons GitHub Actions test summary was counting Go's top-level Test* parent plus its failing subtest as two distinct failures (the parent is just a rollup of its children). The summary now de-dupes by dropping any test case whose name is a strict prefix of another's, so the per-file counts and the Failures block both match the actual leaf-test outcome. CI run 26037892507 was reporting '2 failed' for single when it should have been 1. Also adds a collapsible Skipped section listing each skipped leaf test with its t.Skipf call-site reason (the .go:<line>: line from the CDATA). ANSI escapes from harness.Phase banners are stripped so the reason is plain text. Reasons are useful triage signal — current run has 9 skips spread across daemon gaps (mulga-siv-100) and pool-mode gates (InternetFacing_*). Refs: mulga-siv-89 * ci(e2e-go): propagate cancel to remote .test procs + nuke VM before destroy Two related leaks made cancelled runs strand QEMU work on the libvirt host: 1. The long `ssh tf-user@VM "for SUITE in …; do ./$SUITE.test"` ran without -tt, so when the runner cancelled the step the local ssh died but sshd never sent SIGHUP through (no PTY). The remote `.test` binary orphaned and kept running until its own 30m `-test.timeout`, holding tmp dirs, NATS connections, and qcow2 file handles. 2. Teardown went straight to `tofu destroy` without first killing any stragglers on the VM, so destroy raced the .test's open file handles on the root disk — sometimes successfully, sometimes leaving the libvirt host with a half-released qcow2. Fix: - Allocate a PTY on the run-suites ssh (`-tt`) so sshd HUPs children when the local ssh dies. - Add a trap (EXIT INT TERM) in the run-suites step that ssh's back to the VM and pkill's `/spinifex-e2e-bin/*.test` belt-and-braces. - Mirror the same pkill at the top of Teardown so destroy runs against a quiesced VM. Both cleanups are scoped to the per-run wan_ip — safe to land while multiple ci-single runner agents run concurrent jobs against different tofu envs on the same host (we deliberately keep that parallelism). Refs: runner host wedged 2026-05-19 (post-cancel orphans suspected; no journal access yet to confirm). * ci(e2e-go): scope orphan reap to GITHUB_WORKSPACE so it spares sibling runners Reap Orphan Procs (job start) and the in-Teardown pkill block were using `pkill -u $UID_SELF -x tofu/terraform` plus `find $HOME -maxdepth 6 -name .terraform.tfstate.lock.info`. The single host runs 5+ self-hosted runners all as `tf-user`, so every e2e-go run was SIGKILLing in-flight tofu from sibling workflows (e2e.yml, e2e-ddil.yml, e2e-baremetal.yml, ...) and deleting their state-lock files — exactly the kind of cross-runner collateral the reap was meant to recover from. Scope by `/proc/$pid/cwd` starting with $GITHUB_WORKSPACE; each runner's workspace dir is unique on the host, so only this run's own tofu procs match. Find for stale lockfiles is likewise scoped to $GITHUB_WORKSPACE. * ci(e2e-go): bump setup-go v5->v6, upload-artifact v4->v7 GitHub now warns Node 20 actions are deprecated. e2e-go.yml lagged the rest of the repo; bump to match versions already in use: - actions/setup-go@v5 -> @v6 (matches spinifex.yml) - actions/upload-artifact@v4 -> @v7 (matches release.yml, e2e-ddil.yml) checkout@v6 and create-github-app-token@v3 are already current across all workflows. The local junit-report action is `using: composite` (shell-only), so the Node runtime change does not apply. * fix(vpcd): add flows-ready barrier after IGW attach (mulga-siv-105) handleIGWAttach wrote the external switch + gateway port + LRP + default route to OVN NB and returned immediately, before ovn-northd had compiled NB -> SB and before ovn-controller on the gateway chassis had installed the resulting flows. A subsequent RunInstances -> vpc.add-nat could complete while the datapath was still dark; the launched VM was then unreachable on its public IP for whatever flow-install latency the chassis happened to incur. Customer-visible as 'I just created a VPC, launched a VM, and can't SSH in' for ~minutes. Reproduced deterministically in CI run 26037892507 Phase8b: a VM in a fresh VPC was unreachable at 192.168.0.202:22 for 3+ minutes, while a VM at the same public IP in the warm default VPC (Phase8d) was reachable in ~40 seconds. Block at the end of handleIGWAttach on `ovn-nbctl --wait=hv sync`, which bumps NB_Global.nb_cfg and waits for every connected chassis to acknowledge the new sequence number — i.e. flows installed. Bounded by --timeout=30; overruns log a WARN and continue (a 30s OVN stall is already a deeper problem than failing VPC create would surface). waitForFlowsHV is a package var so tests can stub it; in environments without ovn-nbctl on PATH the shell-out fails fast and the WARN path returns nil, so existing IGW tests stay green without a live OVN. Refs: mulga-siv-105 * fix(vpcd): store spinifex:cidr on VPC create (mulga-siv-106) handleVPCCreate built the OVN logical router with only spinifex:vpc_id and spinifex:vni in ExternalIDs. Cleanup paths (handleIGWDetach + NAT sweeps) read external_ids['spinifex:cidr'] to scope SNAT removal to the VPC's own CIDR; when unset, they fell back to an overbroad 10.0.0.0/8 and emitted a WARN. reconcileVPC already stored the same key at startup; the live-create path drifted. Add the field to handleVPCCreate, log the cidr alongside vpc_id, extend TestTopologyHandler_VPCCreate to assert the external_id is set. Refs: mulga-siv-106 * fix(vpcd): make handleDeleteNAT idempotent on 'already absent' (mulga-siv-107) Instance termination fans out through multiple cleanup paths (instanceCleanerAdapter.ReleasePublicIP AND VPCServiceImpl.DeleteNetworkInterface), both of which publish vpc.delete-nat for the same (vpc, logical_ip). The publish is fire-and-forget (utils.PublishNATEvent uses PublishEvent for vpc.delete-nat, not RequestEvent), so callers never branch on the response. The second publisher then triggered a spurious 'ERROR vpcd: failed to delete dnat_and_snat rule ... not found' log line ms after the first delete succeeded — confusing triage signal with no actionable consequence. Add ErrNATNotFound sentinel to the OVN client interface and wrap the not-found return paths in both LiveOVNClient and MockOVNClient. handleDeleteNAT matches errors.Is(err, ErrNATNotFound) and treats that as success (logs INFO 'already absent (idempotent)'). Existing TestTopologyHandler_DeleteNAT_NotFound asserted the opposite contract; the comment claimed EIP disassociation needed the error signal, but DisassociateAddress also fire-and-forgets vpc.delete-nat, so the contract never had a real consumer. Test renamed to _Idempotent with assertSuccess. Other failure modes (router missing, OVN unreachable, transact error) still surface as ERROR + reply with the error string. Refs: mulga-siv-107 * ci(e2e-go): cluster JUnit failures + name likely root cause (#230) * ci(e2e-go): cluster JUnit failures + name likely root cause New composite action `.github/actions/e2e-analyze` runs after `go-junit-report` converts each suite log. For every `<failure>` it strips ANSI, normalises dynamic noise (EC2 IDs, UUIDs, IPs, timestamps, file:line) to a stable signature, picks the earliest non-cascade failure per suite as the likely root cause, and buckets the rest by signature. Output is appended to `$GITHUB_STEP_SUMMARY` and written to `<log-dir>/analysis.md` so it travels with the artifact bundle. Cascade markers (`must populate fix.`, `Should NOT be empty`, `Expected value not to be nil`) collapse the typical "one upstream phase fails, ten downstream `require.NotEmpty` assertions trip" pattern into a single grouped entry under the real first failure. The analyser is advisory — it always exits 0. The existing "Fail job if any suite failed" gate keeps owning exit-code semantics. Stage 1 of `docs/development/improvements/e2e-go-failure-analysis.md` (mulga-siv-102). Stage 2 (per-failure log slicing) and Stage 3 (rule-based hints via `tests/e2e/.failure-rules.yaml`, placeholder seeded here) are independent follow-ups. CI wiring: - e2e-go.yml: new "Analyze e2e failures" step between "Publish test summary" and the artifact upload, always-runs. - spinifex.yml: new `make test-actions` step so the golden-file tests under `.github/actions/e2e-analyze/testdata/` run on every PR. - Makefile: `test-actions` target; kept out of `test-cover` so CI-only tooling doesn't dilute coverage %. Golden fixtures: a synthetic cascade-vpcs case (one Phase 5 InvalidParameterValue root + 15 downstream cascades), one real single-failure case captured from run 26037892507, and a clean no-failures case. * ci(e2e-go): per-failure log bundles with time-window slicing Stage 2 of the e2e failure analyzer. For each failure surfaced by Stage 1, materialise <log-dir>/analysis/NN-<TestName>/ containing: failure.txt — summary (test, start, duration, file-site, error) journal.log — spinifex-journal.log sliced to [StartAt - 5s, StartAt + Duration + 5s] test.log — test-<suite>.log block between `=== RUN <name>` and the matching `--- FAIL` marker <copies> — any files the harness already dropped at <log-dir>/<TestName_with_/_replaced>/ Report links each entry to its bundle so a reader on the Actions page goes straight to the relevant slice. Workflow now invokes journalctl with --output=short-iso so the analyzer can parse wall-clock timestamps for the journal window. Lines without a parseable timestamp inherit the previous line's classification (keeps multi-line records and stack traces contiguous). Render output is unchanged when BundlePath is empty, so the Stage 1 golden snapshots still match. * ci(e2e-go): restore GOTOOLCHAIN=auto after setup-go v6 bump setup-go@v6 flipped the GOTOOLCHAIN default from "auto" to "local", which makes Go refuse to auto-download the toolchain version named in go.mod. The self-hosted ci-single runner ships /usr/local/go older than spinifex/go.mod's `go 1.26.3`, and build-install-artifacts.sh prepends that path so the system Go wins the resolution race. Under v5 ("auto"), Go silently fetched 1.26.3 on demand; under v6 ("local"), the Build Install Artifacts step errors out: go: spinifex/go.mod requires go >= 1.26.3 (running go 1.26.1; GOTOOLCHAIN=local) Re-set the env to "auto" so behaviour matches pre-v6 runs. * ci(e2e-go): re-export GOTOOLCHAIN=auto after setup-go Workflow-level env doesn't help: setup-go@v6 writes GOTOOLCHAIN=local to $GITHUB_ENV, which overrides workflow env in every later step. Add a dedicated step right after setup-go that appends GOTOOLCHAIN=auto to $GITHUB_ENV so the Build Install Artifacts step (which uses /usr/local/go from a stale runner path) can self-heal by fetching the toolchain named in go.mod, matching pre-v6 behaviour. * fix(vpcd): add flows-ready barrier after dnat_and_snat add handleAddNAT wrote the NAT rule to OVN NB and returned immediately, mirroring the gap siv-105 closed for handleIGWAttach but on the NAT add path. ec2d/RunInstances then returned success while ovn-northd was still compiling NB->SB and ovn-controller hadn't installed the DNAT/SNAT flows on the gateway chassis, so the freshly launched VM was unreachable on its public IP for whatever flow-install latency the chassis happened to incur. Reproduced in CI run 26072432957 Phase8b_VPCSubnetE2E: VM at 192.168.0.212 timed out at 3min while diagnostics showed Port_Binding up=true for both the VM eni and the chassis-redirect gateway. SB-side convergence was complete; the missing piece was the same `ovn-nbctl --wait=hv sync` barrier siv-105 added to IGW attach, applied at the end of handleAddNAT as well. Reuses the existing package-var `waitForFlowsHV` so tests can stub it; no new infra. Parallel unit test asserts the call site fires exactly once per add-nat. * ci(e2e-analyze): real-artifact fixes for journal slicing Three bugs surfaced after Stage 2 ran against real CI artifacts in run 26072432957: 1. Regex `[+-]\d{4}` rejected the colon-offset form (e.g. `+10:00`) that systemd's `journalctl --output=short-iso` actually emits on most versions. Added a parser that tries RFC3339 first and falls back to the no-colon `-0700` layout. Result: 0-line journal slices become populated. 2. go-junit-report's `timestamp` attr on <testsuite> records the moment the XML was produced (post-suite), not when the tests started. Computing StartAt as `that + cumulative-priors` placed every failure AFTER the journal collection ended, so windows always missed. Workflow now writes `${ARTIFACT_DIR}/test-<suite>.start` right before running each suite; analyzer prefers that file over the junit timestamp. 3. Persisted `OffsetFromSuiteStart` on each Failure so the suite-start override can recompute StartAt without re-parsing the XML. Unit tests added for the colon-offset journal format. End-to-end test (TestWriteBundles_EndToEnd) still passes with the new layout. * ci(e2e-analyze): skip parent testcases when summing suite offsets go-junit-report emits one <testcase> for the parent Go test AND each subtest, with the parent's time = sum of its children. ParseFile was summing every testcase into the suite-cumulative-offset counter, so every leaf's offset was inflated by ~the parent's total duration. Visible on run 26073817490 as Phase8b's StartAt landing 13 minutes after the suite finished collecting logs — beyond the journal window, so the per-failure journal slice was empty. Pre-compute the leaf set per suite (a name is a leaf iff no other testcase name starts with `<name>/`) and only accumulate cumul on leaves. Golden snapshots updated; the offset for the only failure in each fixture now matches the actual leaf-sum start time. * test(e2e-harness): capture IP-targeted datapath state on Phase8b SSH fail Phase 1 diagnostics for mulga-siv-111. The Phase8b SSH timeout reproduced on the e2e-failure-analysis branch with siv-105's IGW barrier AND the new add-nat barrier BOTH firing cleanly, so the residual gap is downstream of `--wait=hv sync` — most likely stale OVS conntrack, stale upstream-host ARP, or an OF flow-install miss. DumpVPCFlowDiagnostics now takes a VPCDiagnosticsOpts carrying the external IP, logical IP, and per-test artifact dir. When set, the helper greps `ovs-ofctl dump-flows br-int`, `conntrack -L`, and `ip -4 neigh show` for the external IP / logical IP and snapshots `ovn-nbctl find NAT external_ip=…`. Each capture is both printed inline (existing affordance) and written to a separate file under fix.Artifacts so the Stage 2 analyzer bundler picks it up alongside the journal + test-log slices, no analyzer changes needed. Phase8b's caller now passes pubIP / privIP / fix.Artifacts so the captures fire on the next failure; updates the t.Fatalf message to point at siv-111 instead of siv-105 (which is the now-closed parent). * fix(e2e-single): authorize SSH ingress on fresh-VPC default SG in Phase8b Phase 8b stands up a non-default VPC and runs an instance there. Its auto-created default SG denies all ingress, so the SSH probe times out even when every datapath barrier converges — OF flow captures show the SYN reaches OVN, gets DNAT'd at table 13, then drops at the ACL between tables 13 and 17. The bash driver relies on admin init having pre-baked this rule on the *default* VPC's default SG; a fresh VPC has not. Add a DescribeSecurityGroups + AuthorizeSSHIngress step between route-table setup and run-instances, using the existing harness helper. Fix surfaced via mulga-siv-111's Phase 1 diagnostics (OF flow + ACL drop counts), invalidating the H1/H2/H3 stale-state hypotheses in the bead plan doc. * test(e2e): dual-detect ubuntu image version (24.04/26.04) during gold-image transition After mulga gold image bumped to v6-fastboot (stages ubuntu-26.04 / resolute instead of ubuntu-24.04 / noble), all consumers that hardcoded 24.04 break on the new image. Make them detect at runtime so v3 and v6 envs both pass. - tests/e2e/run-e2e.sh: DescribeImages filter accepts both ami-ubuntu-{26,24}.04-<arch>, then derives IMAGE_NAME from the match. - tests/e2e/run-pseudo-multinode-e2e.sh: prefer ~/images/ubuntu-26.04.img, fall back to ubuntu-24.04.img; pass detected version to spx import. - tests/e2e/single/image_test.go: walk candidate catalog names (26.04 first, then 24.04), stop on the first one DescribeImages can resolve. - .github/workflows/e2e-baremetal.yml: download resolute (26.04) into BM_ISO_CACHE; update BM_UBUNTU_IMAGE env var path. - docs/terraform-workbooks/{bastion-private-subnet,s3-webapp,nginx-webserver}: AMI filter matches `*ubuntu-26.04*` OR `*ubuntu-24.04*` (AWS values= array is ORed). Companion changes for mulga (bootstrap scripts, baremetal, iso pipeline) land in scripts/tofu-cluster/* and scripts/{tofu-iso,baremetal,iso-builder}/* under bead mulga-siv-114. Drop the 24.04 fallbacks once all envs are on the v6 gold image and 24.04 entries are removed from spinifex/utils/images.go. * feat(e2e): add service-interfaces.yaml manifest + validator Source of truth for which spinifex services produce/consume which NATS subjects, and which e2e suites cover which services. Consumed downstream by the targeted suite selector (Bead 4) to derive subset E2E runs from a PR diff so single-service changes stop paying the full 25 min suite cost. - docs/service-interfaces.yaml: 7 services, 4 active suites (cert, lb, single, ddil). Subscribes / publishes / depends_on filled from grep of NATS subjects in source. spinifexui marked covers_no_suite so UI-only diffs skip e2e entirely. Future suites (multinode, reboot, tofu, baremetal) commented in until Go ports land. - tests/e2e/manifest-check: schema + cross-reference validator (path existence, subject regex, duplicate detection, suite covers resolves to a real service, fixtures references resolve). 8 unit tests cover the failure modes plus a load-and-validate on the real manifest. - Makefile: manifest-check target, hooked into preflight. - docs/DESIGN.md: link to the manifest under "NATS Messaging". mulga-siv-116. See docs/development/improvements/e2e-targeted-suite-selection.md. * fix(e2e): drop catalog-download import path from phase4_Image `spx admin images import --name ubuntu-26.04-x86_64` with wantErr=true was meant to detect that bootstrap-install.sh already imported the AMI and short-circuit on the duplicate exit code. After the v6 gold-image bump preloaded `~/images/ubuntu-26.04.img` and bootstrap-install.sh switched to `--file` for the import, two things broke at once: - The catalog `--name` path no longer hits a duplicate guard for the --file-imported AMI, so the import re-downloads ~857MB and either succeeds (failing the wantErr=true assertion) or wedges the job timeout while the catalog server crawls along at 16KB/s. - `--file` from the test won't work either: the test binary runs as `tf-user` over SSH, while bootstrap-install.sh placed the image at `/home/spinifex/images/...` (mode 0700 home dir). bootstrap-install.sh is already responsible for staging the AMI. The test's job is verifying DescribeImages round-trips by-name and by-ID — not re-exercising import. Trust the bootstrap-staged AMI and stop calling import. Keeps the 26.04→24.04 dual-detect for the v3 gold image transition; drop the 24.04 candidate when no envs remain on v3. Refs mulga-siv-115 (caught by the EC2 cold-boot fast-path E2E run on feat/ec2-coldboot-fastpath). * ci(e2e-go): default to e2e-single + harden status-report SHA fallback Two changes to .github/workflows/e2e-go.yml: 1. Default `suites` input now includes e2e-single so a no-input manual dispatch exercises the full single-node ladder. Single-node Go harness is greenable now (image_test fix at 10eb38a). 2. Report E2E (Go) Status job no longer gated on `inputs.sha` and falls back to `github.sha` when the input is unset. Previously a manual dispatch without a PR head SHA either skipped the status post entirely or hit `gh api repos/.../statuses/` with an empty path component (the 'invalid hash' error) — tests passed, the report step failed silently. Now status posts on every dispatch against the dispatched ref's commit when no explicit SHA given. Closes mulga-siv-112. * feat(e2e): add Ensure* fixture API with memoization + cleanup Introduces tests/e2e/harness/fixtures.go: the memoized, cleanup-aware provisioning helpers that Bead 3 (single-fixture-migration) will rewrite phase tests around. Each Ensure* helper: - takes (t *testing.T, fx *Fixture, args...) - memoizes by a key derived from args (sync.Mutex + singleflight) - registers t.Cleanup on the Fixture's parent test so memoized resources survive child-subtest lifetimes - bakes readiness polling into the helper (no sleeps in callers) Exported helpers: EnsureKeyPair, EnsureAMI, EnsureDefaultVPC, EnsureSubnet, EnsureSG, EnsureInstance, EnsureVolume, EnsureSnapshot, EnsureNATGateway, EnsureLoadBalancer. Fixture stores ec2iface.EC2API + elbv2iface.ELBV2API so unit tests can inject fakes without building a real AWSClient (TLS + env). Compile-time checks guard that the SDK concrete types satisfy the stored interfaces. Manifest fixtures: block populated — Bead 4 selector will resolve test coverage from AST-scanned Ensure* call sites via this map. Makefile: new test-harness target runs the harness unit tests under -tags=e2e (mocked, no infra). Hooked into preflight so signature drift and memo/cleanup regressions surface locally. Bead 2 of docs/development/improvements/e2e-targeted-suite-selection.md. No existing tests modified — fixture API is net-new. * feat(e2e): port run-reboot-e2e.sh to Go Adds tests/e2e/reboot/reboot_test.go — Go port of the single-node host-reboot resilience driver (cell #18). Runs from the GitHub Actions runner (not shipped to the VM like cert/lb/single) because systemctl reboot kills any in-VM process mid-step. All AWS API traffic crosses the WAN to https://WAN_IP:9999 via the SDK; SSH-level ops (reboot, ovn-nbctl, journalctl, HTTP burst) hop via harness.PeerSSH. Trust path tries to SCP the cluster CA off the VM and falls back to InsecureSkipVerify with a log message if SCP fails — the cert suite already validates real trust, so the fallback is intentional. Harness changes: - env: WANHost field sourced from SPINIFEX_WAN_IP, distinct from ServiceIPs[0] (which is the awsgw bind IP, often loopback in-VM). - aws: SPINIFEX_AWS_INSECURE=1 escape hatch when the runner has no CA on disk. Used as the fallback when CA fetch fails. - cert: SPINIFEX_CA_CERT explicit path override so the reboot test can point ResolveCACert at the SCP'd tmp file. - peer_ssh: IsReachable() probe wrapper for cleanup paths that must skip rather than fatal when the host is mid-reboot. Workflow (.github/workflows/e2e-go.yml): - New e2e_go_reboot job, opt-in via `e2e-reboot` in the SUITES dispatch input. needs: e2e_go_single to serialise on the ci-single self-hosted runner. Fetches AWS creds from the VM's ~/.aws/credentials over SSH (mirrors the cred hop already proven by e2e-ddil.yml). - e2e_go_single filters 'reboot' out of its build + run loops so the in-VM driver doesn't try to ship reboot.test to the VM and reboot it from inside. - e2e_go_status treats skipped reboot as success. Refs mulga-siv-118. Plan: docs/development/improvements/e2e-go-reboot-port.md (in the mulga parent repo). * ci(e2e-go): run reboot job in parallel with single Drop `needs: e2e_go_single` from e2e_go_reboot. Both jobs target the same [self-hosted, ci-single] label, so GitHub Actions serialises on runner allocation rather than blocking via job ordering — parallel-by- spec, queued-by-runner. Lets a reboot-only dispatch start immediately instead of waiting for an unrelated single-job pre-condition. * fix(e2e/reboot): wait for both backends to serve before post-reboot burst Phase 8.6 previously probed until ANY HTTP response from the ALB, then fired a 20-req burst. Post-reboot, only one of the two backends may be serving (the second is still bringing its responder online) while the daemon's TG cache still shows pre-reboot 'healthy' for both — so WaitForTargetsHealthy returns in ms instead of seconds, the ANY probe matches the first-up backend, and the burst lands 20/0 instead of 10/10. Probe until BOTH backends' instance_ids each appear at least once before the round-robin burst — confirms recovery has reached both, not just one. Underlying daemon-side TG cache staleness tracked separately as mulga-siv-119. This wait is the test-side guard; the daemon fix will make WaitForTargetsHealthy meaningful again post-reboot. Refs mulga-siv-118. * ci(e2e-go): skip single job when dispatch requests only reboot Add if: gate to e2e_go_single that skips the job when the workflow_dispatch suites input contains none of {e2e-single, e2e-cert, e2e-lb}. Push events and the default suites list still trigger it, so plain PR runs are unaffected. Avoids ~15min of wasted tofu provision+bootstrap+teardown when only the runner-resident reboot suite is requested. * fix(e2e/reboot): bump post-reboot LB recovery timeout to 5min Pre-reboot the test creates the LB AFTER apps are already running, so HAProxy starts probing against an already-serving responder — ~75s to healthy. Post-reboot the daemon's recovery scan launches app VMs and the ALB sys.micro VM simultaneously, so health checks start probing within seconds of app boot beginning and both targets honestly report unhealthy until the responder unit finishes starting. Cloud-init's per-instance semaphore means no shortcut path on relaunch, so guest second-boot can be slower than first boot. Bash run-reboot-e2e.sh hid this behind stale 'healthy' state read from the daemon's pre-reboot TG cache — fixed via ResetTargetHealthOnStartup (mulga-siv-119), which made the test honestly wait on real lb-agent reports. The 180s bash inherited turned out to be wishful thinking backed by the cache bug, not measured recovery time. 5min matches the actual post-reboot eventually-consistent window observed on run 26134156355. Refs mulga-siv-118. * feat(e2e/reboot): ssh into guest VMs on failure for HTTP responder diag Post-reboot failures bottom out at "0/2 healthy" with no insight into whether the responder unit died, port 80 isn't bound, or the OVN path is broken. Capture the keypair material from CreateKeyPair so OnFailure can hop host -> qemu hostfwd:22 -> guest and dump systemctl status, port listeners, curl localhost, and the responder journal per instance. * fix(e2e/reboot): probe app instances via public IP (no QEMU hostfwd) Previous hostfwd-SSH approach didn't apply: app VMs use pure OVN tap interfaces (qemu argv has no hostfwd=tcp:*:*-:22), so the regex never matched. Replace with host-side probes against the IGW-NAT'd public IP: ping, /dev/tcp connect, and curl. Combined with the qemu ps check and ovn-nbctl list nat dump, this localises post-reboot failures to one of: VM not running, OVN DNAT missing, responder down, or sys.micro-only reachability (path asymmetry). * feat(e2e/reboot): probe ALB + dump host OVN/OVS state on failure Last run showed apps unreachable from host post-reboot (ping 100% loss, tcp refused) even though OVN dnat_snat rules looked correct. Need to distinguish gateway/ext-bridge breakage from app-side breakage: add a parallel ALB-public-IP probe (works pre-reboot, so any post-reboot failure isolates the regression) plus a full host net-state dump (ip route, ip neigh, ovn-nbctl show, ovs-vsctl show, br-int port list). * feat(e2e/reboot): dump qemu serial console per app on failure /run/spinifex/console-i-<id>.log captures cloud-init, kernel boot, systemd, anything the guest emitted to ttyS0. Tail to 256KiB (~4x a typical Linux boot) so the latest boot lands in full plus the tail of the prior. Surfaces guest-side boot failures (kernel panic, fs corruption, cloud-init OOM, etc.) that are invisible to host-side network probes. * feat(e2e/reboot): SSH into guest VMs via IGW for responder root-cause Last run nailed the symptom: ALB returns 503, app public IPs answer TCP with RST (port 80 not bound), so reboot-e2e-httpd.service isn't running on relaunched VMs. To find out why (unit never installed? installed but not enabled? failed to start?), open tcp/22 ingress on the shared SG and add a guest-SSH diagnostic via the IGW NAT path. The diagnostic dumps the unit state, port listeners, /var/lib/httpd, cloud-init result.json, cloud-init logs, and the instance-id semaphore directory. * test(e2e/reboot): always run guest SSH diag in phase 8 Last run passed after adding tcp/22 SG ingress. Two possibilities to distinguish: (a) responder was running all along but blocked at the OVN/SG layer, (b) responder wasn't running and HC was satisfied by some other path. Capture in-guest unit + cloud-init state on every run so we can confirm which without needing the test to fail. * refactor(e2e/single): migrate VPC group to EnsureDefaultVPC [skip ci] Bead 3a of mulga-siv-120 (e2e-single-fixture-migration). Replaces the "discover default VPC + cache on Fixture" pattern in vpc_test.go, natgw_test.go, securitygroup_test.go with harness.EnsureDefaultVPC. - Adds Harness *harness.Fixture field on single.Fixture; constructed in TestSingleNode alongside AWSClient. Parent test for memoized Ensure* resources is TestSingleNode itself. - Removes fix.DefaultVPCID / fix.DefaultSGID / fix.DefaultSubnetID reads in the three target files. Fields stay on the struct until 3e+3h migrate the instance / datapath / umbrella sites. - Custom VPC graphs created by phase8b (10.99.0.0/16) and the bastion graph in phase8d remain inline — those are the resource lifecycle assertions, not prereqs. CI gate skipped per dev request — full single-node suite verifies when upstream main is unbroken. * refactor(e2e/single): migrate Phase 3 key pair to EnsureKeyPair [skip ci] Bead 3c of mulga-siv-120 (e2e-single-fixture-migration). Phase 3 now calls harness.EnsureKeyPair for the primary key pair (Phase 5+ prereq). The Ensure* path owns the create assertion (PEM shape), memoizes across phases, and registers TestSingleNode-scoped cleanup — replacing the former TODO(stage-G) deferred-cleanup marker. - Drops hardcoded "test-key-1" name; EnsureKeyPair uses scratch-suffixed name so parallel CI runs don't collide on the namespace-flat AWS key pair name. - fix.KeyName / fix.KeyPath still assigned for downstream Phase 5+ readers. Per 3e migration plan, those reads move to in-test EnsureKeyPair calls and the Fixture fields drop with 3h. - Import / describe / delete dance for test-key-2 stays inline — it is the lifecycle assertion for the import path, not a prereq. * refactor(e2e/single): migrate Phase 4 + 5e AMI paths to EnsureAMI [skip ci] Bead 3d of mulga-siv-120 (e2e-single-fixture-migration). - Phase 4 (image_test.go): discover catalog AMI by canonical name, then route through harness.EnsureAMI({Existing}) so the state-available wait + memoization live with the fixture. - Phase 5e (createimage_test.go): replace inline CreateImage block with harness.EnsureAMI({CreateFrom}). The custom AMI build now memoizes, registers DeregisterImage cleanup at TestSingleNode exit, and reuses the harness state poll. Harness fix: extend AMICreateSpec with NoReboot bool — Phase 5e bakes the AMI while leaving the source instance running for Phase 6/7 lifecycle assertions; Bead 2 missed this option. fix.AMIID / fix.CustomAMIID / fix.CustomAMISnapID stay on Fixture for downstream Phase 5/6/7 + Phase 9 cleanup. Those reads migrate with 3e/3h. * refactor(e2e/single): migrate Phase 5 instance lifecycle to EnsureInstance [skip ci] Bead 3e of e2e-single-fixture-migration. - phase5_LaunchInstance: provision via EnsureInstance + re-describe for BDM/root-volume extraction; default VPC via EnsureDefaultVPC. - phase8e_SGToSGDatapath: replace inline DiscoverDefaultVPC with EnsureDefaultVPC. createSG / runSGEInstance take vpcID/subnetID arguments so phase 8e no longer depends on fix.Default* fields. - runSGEInstance stays direct RunInstances: phase 8e VMs are test subjects (need fresh ENI/private-IP per run), not memoized prereqs. - harness.EnsureInstance memo key now includes InstanceType, SGID, and UserData hash — a second call with a different SG would have silently returned the first instance ID (Bead 2 bug). * refactor(e2e/single): migrate Phase 5b/5c volume + snapshot to Ensure* [skip ci] Bead 3f of e2e-single-fixture-migration. - phase5b_VolumeLifecycle: create+wait via EnsureVolume; subsequent modify/attach/detach/delete stay inline (test subject lifecycle). - phase5c_SnapshotLifecycle: create+wait via EnsureSnapshot. Drop the create-response State/Progress shape assertions — after EnsureSnapshot the snapshot is already "completed", and the describe block below re-asserts VolumeId / VolumeSize / Description. - harness.EnsureSnapshot now takes SnapshotSpec{VolumeID, Description} so callers can pin a description (snapshot_test asserts on it). Memo key includes description hash so distinct-description calls do not silently collide on the volumeID prefix. * refactor(e2e/single): isolate IAM stage E via parent-scoped cleanup [skip ci] Bead 3g of e2e-single-fixture-migration. - harness.Fixture: add RegisterCleanup helper that wraps fx.parent.Cleanup so test code can register parent-scoped teardown for resources created outside ensureOnce (IAM users / policies threaded across phase subtests). Registering on the calling subtest would tear down before the next dependent phase ran. - phaseIAM1_UserCRUD: register parent-cleanup for alice/bob/charlie. - phaseIAM4_PolicyCRUD: register parent-cleanup for the 5 test policies. - IAM stage no longer relies solely on phaseIAM7 to clean up after a mid-stage failure; phase 7 still runs and now serves as an idempotent post-cleanup assertion. No new Ensure* helper: IAM resources have no memoization story (users and policies are test subjects, secrets only returned once on create). IAM phases also have no EC2 prereqs, so EnsureDefaultVPC / EnsureInstance are not applicable here. * refactor(e2e/single): drop dead Fixture fields after Ensure* migration [skip ci] First half of bead 3h. Removes per-phase Fixture fields with zero remaining readers after bead 3a-3g: - DefaultVPCID / DefaultSGID / DefaultSubnetID: superseded by harness.EnsureDefaultVPC (3a wired phase8b/8c/8d/5f; 3e wired phase5 + phase8e). - OVNAvailable: never assigned anywhere — leftover from the bash port. Phase fields still in use (AMIID, InstanceID, IAM*, etc.) stay until the umbrella TestSingleNode is replaced with top-level Test* + package- level fixture singleton in the second half of 3h. * refactor(e2e): add process-mode Fixture + TestMain singleton infra [skip ci] Second half of bead 3h. Lands the infrastructure required to dismantle the TestSingleNode umbrella in a follow-up bead, without changing the umbrella body today. harness.Fixture changes: - NewProcessFixture(aws) builds a Fixture with no *testing.T parent. Cleanup callbacks queue on an internal LIFO drained by Close() — intended to be invoked from TestMain after m.Run(). Lets a singleton fixture span every top-level Test* in a package. - Existing NewFixture(t, aws) unchanged; cleanups still route via t.Cleanup. registerCleanup() and logf() helpers fork on parent==nil so ensureOnce and RegisterCleanup work in either mode. - TestProcessFixture_CloseRunsCleanupsLIFO covers Close ordering, one-shot semantics, and the no-op second-Close path. single/main_test.go additions: - TestMain owns process-level lifecycle: drains pkgFix.Harness.Close() and removes pkgFix.TmpDir after m.Run(). - requireSingleNodeFixture(t) returns the package-scoped singleton, building it lazily on first call. Skips when SPINIFEX_E2E is unset or mode != single; fails on construction error. Currently unreferenced — TestSingleNode keeps its own per-test Fixture for now. The helper exists so the follow-up bead (per-phase top-level Test* migration) has a stable API to call from day one. Deferred to follow-up bead: extract Phase 4 AMI-name discovery + Phase 2 InstanceType/AZ selection into harness helpers so each phase function can self-bootstrap, then promote phase subtests to top-level Test* and delete the TestSingleNode umbrella. * fix(e2e/harness): EnsureDefaultVPC filter name is-default not isDefault Daemon rejects the camelCase `isDefault` form with 400 InvalidParameterValue. AWS CLI / DiscoverDefaultVPC use `is-default`. Surfaced as Phase 5_LaunchInstance fatal in CI run 26140908505 — cascade-failed every downstream phase that needed fix.InstanceID. * fix(e2e/single): phase8 8j/8k probe fix.KeyName not hardcoded test-key-1 Bead 3c migrated phase 3 to harness.EnsureKeyPair which produces a scratch-suffixed name (e2e-key-<random>), not the literal "test-key-1" the bash port used. 8j_CreateKeyPairDuplicate / 8k_ImportKeyPairDuplicate kept the old name and so created a fresh key pair instead of triggering InvalidKeyPair.Duplicate — surfaced as the only two FAILs in CI run 26143227369 (everything else green after the EnsureDefaultVPC fix). Read fix.KeyName (set by phase3_KeyPairs) so the probe hits the staged primary key. Header comments updated to match. make preflight diff-coverage against origin/dev fails at 57.9% due to unrelated commits inherited from the recent rebase onto feat/e2e-go-harness; against the fork base the diff is 0 instrumentable lines (test-only change). * refactor(e2e/single): dismantle TestSingleNode umbrella, per-phase self-bootstrap mulga-siv-121. Each phase from run-e2e.sh now runs as a top-level Test* in tests/e2e/single, runnable in isolation via `go test -run TestX`. Highlights: * TestSingleNode umbrella removed. Phase ordering dropped — each Test* self-bootstraps prerequisites through harness.Ensure*/harness.Discover* + the new package-local need*/iamEnsure* helpers. * Fixture shrunk to {Env, AWS, Harness, Artifacts, TmpDir, PoolMode}. Every per-phase resource ID now lives on harness.Fixture via EnsureKeyPair/EnsureAMI/EnsureDefaultVPC/EnsureInstance/etc. * TestMain owns the package singleton Fixture; harness.Fixture.Close drains every cleanup at process exit. phase9 + phase9a (in-line teardown) deleted; phase9b retained as TestFinalClusterStats. * IAM Test* share state via sync.Once-gated helpers in iam_helpers_test.go (alice/bob/charlie users + keys + admin account). * New harness/discover.go: DiscoverDefaultAZ, DiscoverNanoInstanceType, DiscoverUbuntuAMI — memoized on the Fixture and routed through EnsureAMI so any Test* can resolve them without phase coupling. * New tests_test.go: descriptive top-level Test* wrappers (TestVPCSubnetE2E, TestKeyPairs, TestSnapshotLifecycle, …) that delegate to phaseN_X(t, requireSingleNodeFixture(t)). * test(e2e/reboot): SSH as ec2-user, dump unit + network-online state spinifex's cloud-init template hardcodes Username="ec2-user" (handlers/ec2/instance/service_impl.go:1218) regardless of AMI, so even on Ubuntu the SSH-injectable account is ec2-user, not ubuntu. Switch the diag SSH-in to ec2-user so we actually get in (was "Permission denied (publickey)"). Add diagnostics to distinguish the two leading post-reboot hypotheses: (a) viperblock write loss = unit file gone vs (b) network-online.target never fires = unit stuck on After=network-online.target. New dumps: unit file content + enable symlink, network-online.target status, networkctl, ip addr / route, prior-boot journal grep for the responder. * fix(e2e/single): resolve admin account ID before TestIAM7 deletes alice iamEnsureAdminAccountID probes via GetUser(alice). TestIAM7_Cleanup deleted alice/bob/charlie first, then called the helper at line 808 — NoSuchEntity if IAM7 was the first IAM test to run (no memoized account ID). Move the call above the deletion loop.	2026-05-20 21:04:13 +10:00