7 Commits

Author	SHA1	Message	Date
Julian Sommer	8a94886b13	feat: cloud-init EC2 datasource (#411)	2026-06-25 13:48:08 +10:00
Julian Sommer	2e9501524b	feat(e2e): more e2e coverage (#365)	2026-06-17 09:52:50 +10:00
Josh Siviour	1873f45d26	Resilient resource lifecycle (ADR-0001..0006): EKS multi-node teardown, idempotent deletes, GC backstops (#357) ... * fix: idempotent Delete* across VPC family + volume (RLC1) Delete handlers now return empty success when the resource is already absent (nats.ErrKeyNotFound), instead of *NotFound. Transient read errors still map to ServerInternal. Covers DeleteVpc, DeleteSubnet, DeleteSecurityGroup, DeleteNetworkInterface, DeleteInternetGateway, DeleteNatGateway, DeleteRouteTable and DeleteVolume. Makes destroy retries converge instead of failing mid-graph, removing a source of orphaned/zombie resources between deploys. Adds a TestRLC1_*DeleteIdempotentOnAbsent invariant per package and updates the daemon delegation table; supersedes the old absent->NotFound tests. * feat: prune orphan ENI LSPs in reconcile applyPorts (ADR-0003 §4) applyPorts was create-or-update only, so a guest LSP whose spinifex:eni_id no longer maps to an intent ENI survived instance terminate and VM-host reinstall — the orphan-LSP capacity-drift root cause. It now takes pruneOrphans (mirroring applySGs): on a full reconcile it lists LSPs and tears down each managed ENI port absent from intent via the composed topology.DeletePort cascade (clears port-group memberships, then deletes the LSP). The apply-only startup path does not prune, preserving in-flight ports until subscribers converge. Adds ListLogicalSwitchPorts to the OVN client (live + mock). Invariant TestRLC5_OrphanLSPPrunedAfterReconcile guards ADR-0003 §4; a startup-race test guards the apply-only path. * feat: idempotent + tracked instance terminate (ADR-0003 §1/§2) Make terminate idempotent across all three layers and break the un-terminable-ENI deadlock, and record per-dependency teardown so the GC backstop can later tell a fully-reaped instance from a dangling one. - vm.Manager.Terminate: absent / already-terminated / shutting-down now return nil instead of NotFound / invalid-transition (ADR-0003 §2). - VM.Teardown map + teardown.go marks: sync steps stamped Done/Failed via cleaner error returns; fire-and-forget OVN LSP and NAT rule stamped Pending for the reconciler / GC reaper (ADR-0003 §1). - InstanceCleaner: DeleteVolumes / ReleasePublicIP / DetachAndDeleteENI / RemoveFromPlacementGroup / ReleaseGPU now return error. - vpc ForceDeleteInstanceENI bypasses the in-use guard for the owning instance, breaking the failed-detach deadlock; plain delete still rejects an in-use ENI held by another instance. - gateway TerminateInstances: KV-health-gated idempotent absence (a failed terminated-bucket query does not fabricate success, ADR-0003 §3). - Invariants: TestRLC1_TerminateIdempotentOnAbsent, TestRLC4 teardown-map stamping + failed-dependency, TestRLC_ENINonDeadlock. * feat: reconciler GC backstop framework + terminated-teardown reaper (ADR-0003 §3) Add the shared reality->desired garbage-collection framework (inverse of Restore) and its first reaper, closing the gap 295.3 deferred: nothing retried pending/failed teardown or purged the terminated record when its dependents reached done. - vm/gc.go: GarbageCollector + Reaper interface (Class/Scope/Sweep). KV-health gated — holds and alarms when the desired-state cannot be trusted, never reaps against it. ClusterWide reapers run only on the elected leader (optional elector); NodeLocal run on every node. Start sweeps immediately then every GCInterval. - vm/teardown_reaper.go: TerminatedTeardownReaper (node-local). Scans this node's terminated records, re-drives each not-done dependent through the idempotent cleaner, and purges the record once all reach done. Records already complete on arrival are left to the bucket's 1h TTL. - StateStore gains DeleteTerminatedInstance (JetStream impl already existed). - daemon: wires the GC after the pending watchdog, gated on a cheap JetStream AccountInfo health probe. - Cluster OVN-LSP arm already lands via the reconciler drift prune (295.2). - Invariants: TestRLC5_GCRetriesPendingTeardownToDoneThenPurges, TestRLC5_GCNeverPurgesIncompleteRecord, TestRLC5_GCHoldsWhenKVDegraded, TestRLC5_GCSkipsOtherNodesRecords. * feat: VPC-family dependency guards (ADR-0004 rule #3) DeleteSubnet blocks while a live ENI attachment (a resident instance) remains; available/orphan ENIs do not pin it. DeleteNetworkInterface guards on the structured attachment (eniIsLiveAttachment), not a bare in-use string. DeleteNatGateway blocks while a live route table forwards to it and disassociates its EIP keeping the allocation (AWS parity); CreateNatGateway now marks the EIP associated so the disassociate is real and a second NAT GW cannot reuse the allocation. Binding invariants: TestRLC3 dependency-guard completeness across the subnet/ENI/NAT-gateway handlers. * feat: EBS volume data-safety GC override (ADR-0005) The shared GC backstop reaps actual-desired for every other class; for volumes that default is forbidden because they carry customer data. Add VolumeLeakReaper (node-local, rides the 295.5 GC): it marks a volume left attached to a definitively-gone instance as orphaned (a non-destructive tag) and alarms, but NEVER issues a delete. The daemon feeds it only this node's terminated instances whose volumes-teardown failed, so the shared predastore scan can never false-mark another node's live-instance volume. DeleteVolume idempotency + the detach/snapshot guards + the DeleteOnTermination teardown retry already landed (295.1/295.3/295.5). Binding invariants: TestRLC5 (GC marks+alarms, never deletes data), TestRLC3 (detach-before-delete). * test(elbv2): lock ADR-0002 §3 live-only TG in-use guard (mulga-siv-295.4) * feat(eks): idempotent DeleteCluster on absent + ADR-0006 lifecycle invariants (mulga-siv-295.8) DeleteCluster returns empty success when the cluster meta is already swept (Common Resource Lifecycle Contract #1), so tofu destroy retries converge. Adds TestRLC1 (idempotent delete), TestRLC2 (billable-before-KV-sweep ordering) and TestRLC3 (NLB target-group no-orphan, riding ADR-0002/172). * fix(eks): GC reaper terminates orphan control-plane VM after cluster delete (mulga-siv-294) A normal DeleteCluster can leave a running control-plane VM behind (terminate completes async; a daemon restart re-launches it from saved running-state) after the cluster meta is swept — a billable orphan with no owner. Add EKSBillableReaper on the 0003 GC backstop: a running managed-by=eks CP VM whose cluster meta is definitively gone is terminated. The CP ENI now carries the customer-account tag so a system-account VM can be mapped back to its cluster meta; the reaper never reaps on uncertainty (ENI gone/untagged or meta present). * feat: reclaim orphan EKS NLB + NAT-GW EIP in billable reaper The billable GC reaper terminated an orphan control-plane VM but left the cluster's NAT-gateway EIP and NLB front-end NAT behind when the meta was gone. After a successful terminate, run the name/tag-driven DeleteClusterNLB + DeleteClusterCPVPC teardown (idempotent, meta-independent) so the billable NAT-GW EIP and NLB NAT are reclaimed too. ADR-0006 §5. * fix: persist NAT-GW & route-table tags and reorder CP-VPC teardown to reclaim EKS NAT-GW EIP CreateNatGateway and CreateRouteTable both dropped input.TagSpecifications, so the tag-driven DeleteClusterCPVPC teardown could not discover the cluster NAT gateway or its private route table. The explicit DeleteNatGateway + ReleaseAddress were skipped and the billable NAT-GW EIP leaked under the system account on every clean DeleteCluster. Three fixes: - CreateNatGateway persists ExtractTags(TagSpecifications, natgateway). - CreateRouteTable persists ExtractTags(TagSpecifications, route-table). - DeleteClusterCPVPC tears down route tables before the NAT gateway, so the rule #3 route-ref guard (DependencyViolation / InvalidAddress.Locked) is not tripped and the EIP releases. Adds tag-persistence regression tests for both create paths and a teardown ordering invariant (TestRLC5). Validated live: NAT-GW EIP released cleanly, no DependencyViolation/Locked. * fix: route EKS control-plane VM terminate cross-node + DELETING teardown backstop Multi-node EKS cluster delete wedged in DELETING and stranded the billable NAT-GW EIP. Two root causes, both invisible on a single-node box: Cross-node terminate was never routed. TerminateSystemInstance was node-local, so terminating a CP VM owned by another node returned NotFound (treated as idempotent) and the teardown deleted a still-attached ENI (InvalidNetworkInterface.InUse). That error short-circuited purgeClusterInfra before the managed CP-VPC teardown that releases the NAT-GW EIP. Split TerminateSystemInstance into a local body plus a routing wrapper that issues system.TerminateInstance.<id> to the owning node. Bind that subscription for every system-managed VM (elbv2 + eks) via the OnInstanceUp funnel, so CP VMs placed on the coordinator's own node (and recovered after a daemon restart) are reachable too, not only remotely-launched ones. No DELETING backstop. A failed synchronous DeleteCluster left the cluster DELETING with nothing to re-drive it. Add EKSDeletingReaper (vm.Reaper, class eks-deleting) that re-runs purgeClusterInfra for clusters wedged past a min-age, gated by the per-cluster reconciler leader lease. Stamp ClusterMeta.DeletingSince on the transition into DELETING. TerminateK3sServerVM additionally waits out any residual ENI-InUse with a bounded budget as defense-in-depth. Validated live on env19 3-node: first delete-cluster converges in 29s, both remote CP VMs terminated cross-node, NAT-GW EIP released, backstop never fires. * fix(network): gate gateway reconcile on real datapath reachability Post-reboot the external OVN datapath can stay dead while every control-plane signal is green: the chassisredirect SB binding is claimed (up=true), NAT rows are present and the host neigh is primed, yet ovn-controller leaves stale gateway/localnet flows so EIP north-south and the VM->gateway heartbeat forward nothing. ensureGatewayClaimed only nudges recompute when the binding is unclaimed, so a claimed-but-dead datapath is never detected or repaired. Add ensureGatewayDatapath: after the SB claim converges, probe the gateway LRP IP (OVN answers ICMP natively, no guest or security-group dependency), and on a miss force one ovn-controller recompute and re-probe until a short deadline. Reuses the existing recompute nudge and no-ops when no verifier is wired or no gateway IP resolves. * fix: self-heal external uplink after reboot boot-race Post-reboot, ovs-vswitchd/ovn-controller order after the passive network.target rather than network-online.target, so they start concurrently with systemd-networkd. Under a heavy boot the WAN-glue veth (veth-wan-ovs) can come up admin-down before networkd brings it up, leaving br-wan with no carrier and the external datapath dead while every control-plane signal reconciles green. Broaden the datapath health-gate remediation from recompute-only to RepairDatapath: re-assert the veth admin state (covers the admin-down link, which a flow recompute cannot revive) then recompute (covers a stale gateway ofport). Idempotent; the veth ops are skipped in physical-uplink mode where the device is absent. Also capture ip -d link (veth admin/carrier) and ovs-ofctl show br-ext (ofport) in the reboot diag bundle to disambiguate the two failure shapes from CI artifacts. * fix: restore AWS-faithful Delete* NotFound; move idempotency to teardown callers (mulga-siv-306) RLC1 (f9218982) made EC2/VPC/EBS Delete* handlers return success on an absent resource, breaking AWS compatibility and the cross-account isolation signal (a cross-account delete reads as absent -> success instead of *.NotFound). Restore the AWS contract at the API while keeping lifecycle convergence: - Handlers return canonical *.NotFound on absent: DeleteVpc, DeleteSubnet, DeleteSecurityGroup, DeleteInternetGateway, DeleteNatGateway, DeleteRouteTable, DeleteVolume, and the public DeleteNetworkInterface (ForceDeleteInstanceENI keeps its idempotent force path, ADR-0003 section 2). - Add awserrors.IsNotFound; destroy orchestration (instance terminate, EKS CP-VPC teardown, EKS IGW/SG cleanup, k3s rollback, ELBv2 ENI/SG teardown) tolerates NotFound so re-issued/double-targeted deletes converge. - Flip the RLC1 unit invariants to assert *.NotFound on absent; revert the daemon round-trip cases; add IsNotFound + CP-VPC convergence tests. ELBv2 deletes and TerminateInstances stay idempotent (AWS itself is). * fix(network): gate ingress EIP datapath on guest SB Port_Binding convergence TestMultinodeGuestSSH intermittently times out (Connection timed out, not #355's reset-by-peer) connecting to a fresh trio instance's public IP after the suite churns instances across all three chassis. The instance is running and AddEIP has installed the DNAT row, flushed/primed the host neigh, and barriered the gateway-chassis flow — but none of that proves the gatewayLRP->guest geneve hop: until the guest ENI's southbound Port_Binding is up on its hosting chassis the cross-node flow is missing and the DNAT-translated packet blackholes. Add ensureGuestPortDatapath, the guest-side analog of ensureGatewayDatapath: after AddEIP, probe the guest LSP's SB Port_Binding (ovn-sbctl up column); on a miss force one ovn-controller recompute (binds the port / installs the remote flow) and re-probe until a short deadline. Recompute, not RepairDatapath: this is flow/binding convergence, not a WAN-uplink fault. No-op when no verifier is wired or the EIP carries no guest port. spec.PortName is populated for every associated EIP regardless of NAT mode. Extends GatewayClaimVerifier with GuestPortUp and wires the gate into applyEIPs so it runs in the continuous reconcile loop on every node. * Revert "fix(network): gate ingress EIP datapath on guest SB Port_Binding convergence" This reverts commit 728b1a9cec6eaf5c7ef5e9eda4469f239b4d8fc0. * fix(network): key EIP DNAT delete on external IP, not logical IP A dnat_and_snat row's identity on the VPC gateway router is its external IP. The logical (private) IP is not a stable key: the EIP pool round-robins and instance private IPs are reused as instances recycle, so the same logical IP is held by different EIPs over time. DeleteEIP deleted the row by logical IP, so a stale or retried delete for a freed EIP clobbered whichever live EIP now held that recycled private IP -- the next owner's row vanished with no re-add and its public IP went dark (SSH SYN never left br-wan: Connection timed out). Lifecycle-resilience teardown/GC retries amplified this by firing vpc.delete-nat repeatedly. Key the delete on the external IP via DeleteNATByExternalIP and skip when no external IP is supplied (no EIP means no dnat_and_snat row). A stale delete now targets the freed EIP's own external IP (ErrNATNotFound no-op) and can no longer touch the live row. * refactor: relax DeleteNatGateway to AWS-faithful Remove the route-reference dependency guard (checkNatGatewayRouteRefs) so a NAT gateway deletes even while route tables still forward to it, matching AWS (the route blackholes). The reconciler's loadNATGWs drops the SNAT once the record leaves the active bucket, so a forwarding route is not a delete-blocking dependent. EIP is still disassociated on delete (no leak). * test: invert NAT-GW route-ref invariant to AWS-faithful TestRLC3_NatGatewayBlocksWhileRouted asserted DependencyViolation while a route forwarded to the NAT gateway, which is AWS-infaithful. Replace with TestRLC3_NatGatewayDeletesWhileRouted: delete must succeed whether or not a live route still targets it (the route blackholes; the reconciler drops the SNAT). Per ADR-0004 §2. * fix: tear down NAT-GW SNAT on delete (restore delete-time fan-out) The guard commit (53ed55a9) dropped publishDeleteEventsForNatGateway on the assumption a routed NAT GW could never be deleted. With the route-ref guard removed, deleting a NAT GW while routes still target it left the SNAT in place (applyNATGWs is create-only, so the reconciler never prunes it) and egress kept working — TestNATGateway's post-teardown 'lost internet' check failed. Restore the fan-out: on delete, scan route tables and publish vpc.delete-nat-gateway per associated subnet so egress stops, while the route entry survives as a blackhole (AWS-faithful). * fix: probe EIP datapath in reboot self-heal, not the gateway LRP IP The post-reboot gateway datapath self-heal probed the gateway LRP IP for liveness, which OVN answers natively at the router — so it stayed green even when the EIP NAT pipeline was stranded (ofproto/trace dropped EIP traffic at table 0). RepairDatapath never fired and TestRebootResilience timed out, with the drift loop re-probing the same false-green every cycle. Prefer probing an associated EIP: force a fresh ARP of the dnat_and_snat external IP (flush neigh, ping to trigger, read the resolved MAC). OVN answers ARP for the external IP from the gateway chassis independent of the guest, so a resolved MAC proves the WAN uplink forwards and the NAT flows are installed — the signal the LRP IP cannot give. Fall back to the LRP IP for VPCs with no EIP. The existing drift-loop cadence now re-heals a later flow strand. * fix: lb-agent heartbeats over mgmt port, not the WAN/EIP datapath The lb-agent heartbeat to awsgw is control-plane and must survive a host reboot that strands the OVN/EIP north-south data plane. Customer (internet-facing) lb-agents heartbeated over the WAN GatewayURL, riding the EIP datapath, so after a reboot the ALB never reported healthy until the datapath self-healed. System LBs already heartbeated over the on-link br-mgmt NIC; this extends that path to every lb-agent. buildLBAgentEnv now always uses mgmtGatewayURL (on-link MgmtBridgeIP, the mgmt NIC the lb-agent VM already carries), falling back to the WAN URL only when no mgmt bridge exists. Drops the account-id branch. * fix: re-heal guest ENI SB Port_Binding post-reboot so EIP ingress converges Post-reboot every EIP in a VPC goes dark north-south: the DNAT row, host neigh and gateway-chassis flow are all in place (ARP to the EIP resolves) but the gatewayLRP->guest flow is missing until ovn-controller binds the guest ENI's SB Port_Binding, so the DNAT-translated packet blackholes and the public IP is unreachable on an otherwise-running instance. Restore the guest-port convergence gate in applyEIPs: probe the guest LSP's SB Port_Binding up column; on a miss force an ovn-controller recompute and re-probe until a deadline. Unlike the earlier reverted form, recompute on every miss, not once: after a host reboot the guest tap is replumbed several seconds after the reconcile runs, so a single early nudge fires before the port exists and never binds it. The window is widened to span guest boot. This is the southbound analog of the gateway datapath self-heal; the EIP ARP probe cannot detect it because the gateway chassis answers ARP for the external IP even while the guest delivery flow is absent. * fix(network): force post-reboot EIP recompute on guest-port bind, startup-only The reboot suite passed only by accident: the pre-#351 gateway-claim verifier polled the always-chassis-less gateway LRP, so it fired an ovn-controller inc-engine/recompute on every reconcile. That spurious recompute was load-bearing - it reinstalled the centralised gateway DNAT flows that ovn-controller's incremental engine misses on a cold (post-reboot) start. #351 correctly switched the check to the chassisredirect port, eliminating the every-cycle recompute, and the EIP datapath went dark post-reboot with every control-plane signal green. Restore the kick, scoped to the post-reboot startup path: applyEIPs now gates a single recompute on the guest ENI's SB Port_Binding chassis claim - the reliable signal the tap is back, unlike the up column which stays stale-true across a reboot. Drift never recomputes, preserving the #351 fix for freshly attached EIPs. Probe renamed GuestPortUp -> GuestPortBound (chassis, not up). * Revert "fix(network): force post-reboot EIP recompute on guest-port bind, startup-only" This reverts commit 792e318211ef03efbe24febeb470749d88c0b03c. * test(e2e/reboot): capture dataplane diag on ALB serve failure The post-reboot ALB-serve assertion (8.6) failed with no flow/ARP evidence: the OVN/OVS dataplane dumps only ran on the 8.5 targets-unhealthy path, but 8.5 passes on the daemon's own health view while the north-south EIP datapath is dark, so the serve failure (8.6) left nothing to localise the drop. On a serve failure, capture dumpALBProbe/dumpOVNDataplane/dumpOVNClaimState/ dumpHeartbeatDiagnostics/dumpAppDiagnostics before failing. Add to the OVN dataplane dump: dump-flows br-ext and fdb/show br-ext (north-south bridge, previously only br-int), and a real ARP probe that flushes the neigh entry first so primeReachability's pre-seeded reachable entry can no longer mask an unanswered ARP. Pure diagnostics; no production change. * test(e2e/reboot): decisive datapath diag — untruncate br-int, real-ingress ofproto/trace, conntrack + tap capture * test(e2e/reboot): add south->north return-path ofproto/trace to localize egress drop * test(e2e/reboot): derive LRP MAC from realized flow + remediation experiment (recompute vs ovn-controller restart) * test(e2e/reboot): dump persisted SB MAC_Binding + external MACs; flush experiments (conntrack vs MAC_Binding) to isolate recompute-resistant return-path drop * test(e2e/reboot): untruncate Port_Binding + enumerate DGP/chassisredirect claim state (does cr-gw row exist + resident chassis) * test(e2e/reboot): dump NB source of return-path drop (lflow stage, NAT external_mac/options, LR policy, LSP port_security) * fix: exempt public-IP instances from subnet egress drop gate (mulga-siv-311) Post-reboot reconcile installs the priority-1100 subnet egress drop gate for an IGW-attached subnet that has no 0.0.0.0/0 route, but nothing punched the subnet's public-IP instances through it. lr_in_policy runs before lr_out un-DNAT/SNAT, so an inbound connection's reply still carries its private source when it hits the gate and is dropped. The ALB/EIP datapath goes dark south->north post-reboot while NAT, ARP, chassis and guest-port binding all read green; the pre-reboot path survives only because the event-driven gate never evaluates a subnet created after IGW attach. Install the existing priority-1200 /32 reroute (above the gate) for every public-IP instance during reconcile. Reroute only: the instance's dnat_and_snat already supplies SNAT, unlike EnsureSystemInstanceEgress which also adds a plain snat. Public IPs come from two disjoint sources -- auto-assigned/ELB addresses on the ENI record and user EIPs in the EIP bucket -- so drive the exemption from both intent.Ports (new PortSpec.PublicIP) and intent.EIPs. * refactor: scope public-IP drop-gate exemption to gated subnets Only install the priority-1200 reroute when the instance's subnet actually carries a drop gate. Routed subnets egress via their priority-1000 reroute and need no exemption, so bounding on gate presence keeps the change inert for IGW-routed and NATGW-routed subnets (no redundant or overriding policy). Adds a negative test. * refactor: drop unused RemoveEIPInstanceEgress No production caller; the exemption is install-only in reconcile. Removed the interface method and impl rather than leave dead API surface. * test: dump guest console + OVN/datapath diag on SG-to-SG client SSH timeout TestSGToSGDatapath waited for client-vm SSH via EventuallyErr and reported only 'exit status 255' on timeout — no guest-side or datapath state. When the fresh public-IP VM is unreachable for the full window the failure was undiagnosable from CI artifacts. Swap the wait to the non-fatal trySSHReady probe and, on timeout, call DumpVPCFlowDiagnostics (guest console tail, ovn-nbctl/sbctl show, NAT rows, br-int flows / conntrack / host ARP keyed on the public + private IP) before Fatal, mirroring vpc_test/baseline_test. Captures persist to the artifact dir for the Stage 2 analyzer bundle. * fix: ownership-scope public-IP teardown to survive IP recycling A live instance's dnat_and_snat rule, host ARP, and IPAM lease were torn down by the delayed garbage-collection teardown of a different, long-dead instance that previously held the same recycled public IP. The 2-min vm/gc instance-teardown reaper re-runs ReleasePublicIP on a stale terminated record; because the external IPAM round-robins a small pool, the IP had already been reassigned to a live instance, and both teardown primitives acted on the IP by value with no ownership check: - policy.DeleteEIP deleted the dnat_and_snat row by external IP alone, discarding the logical IP it receives. - StaticPoolAllocator.Release freed the pool entry by IP value. Scope both to the current owner so a stale or duplicated teardown for a recycled IP is a no-op: - DeleteEIP looks up the row by external IP and skips the delete (and the ARP flush) when its logical IP no longer matches, mirroring AddEIP's idempotency lookup. Defeats both external- and logical-IP reuse races. - Allocator.Release / ExternalIPAM.ReleaseIP take an ownerENIID; the static allocator no-ops a release whose ENI differs from the live allocation, and treats an already-freed owner-scoped release as idempotent. Teardown and rollback callers pass their ENI; user-driven EIP release passes none. Proven via E2E single TestSGToSGDatapath (run 27515293563): client-vm SSH black-holed after its NAT was deleted by the dead instance's GC sweep. * fix(eks): single-flight DeleteCluster teardown per cluster SDK retries of one DeleteCluster get fanned across nodes by the worker queue group, so purgeClusterInfra raced the same cluster on multiple nodes: duplicate ENI/NLB teardown, duplicate TargetGroupNotFound deregisters, and a real double-release risk on the billable NAT-GW EIP. Idempotency kept it correct but noisy. Add a dedicated per-cluster teardown lease (teardownLeaderKey = reconcilerLeaderKey + "/teardown"), distinct from the reconciler lease so a delete never contends with the still-live reconciler that holds the reconciler lease until it observes DELETING and exits. DeleteCluster acquires it after the NotFound-idempotency check: the loser returns the cluster as DELETING (AWS-async semantics) without running the teardown; the winner drives purgeClusterInfra and releases. The DELETING backstop reaper shares the same lease, so at most one purgeClusterInfra runs per cluster cluster-wide. A nil leaderKV (single-node) skips gating, so local behavior is unchanged. * fix: env-driven SSH-ready budget for slow baremetal guests Add SPINIFEX_SSH_READY_TIMEOUT override for the single-node SSH-handshake probes (default unchanged at 3m). Baremetal boots q35 guests slower (OVMF + NBD-backed disks) than CI loop devices and intermittently misses the 3m window; the harness can now widen it to 5m without affecting shared-runner e2e. Applied at all single-node probe sites; failure messages now report the actual budget. * fix: shared external switch to stop per-VPC localnet ARP collision Per-VPC external switches each attached an untagged localnet port to the same physical network (br-ext). OVN cannot keep multiple untagged localnets on one physical network L2-separate, so with two or more concurrent public-IP VPCs ARP for a public IP landed on the wrong VPC's switch and the owning gateway router's ARP responder never fired, leaving the public IP unreachable. Move to a single shared external switch (provider-network pattern): one switch with one localnet on network_name=external. Each VPC's gateway LRP attaches via its own router-type switch port; per-VPC NAT isolation is unchanged. AttachIGW ensures the shared switch idempotently and is migration-safe; DetachIGW removes only the per-VPC port + LRP and preserves the shared switch. DeleteVPC removes the per-VPC gateway switch port from the shared switch so it does not dangle. Reconcile detects attachment via the gateway LRP since the shared switch carries no vpc_id. * fix: owner-scope DeleteEIP by ENI port to survive identical pair recycling A stale or GC-re-emitted vpc.delete-nat for a terminated instance tore down a live instance's dnat_and_snat rule (and flushed the host ARP) when the public IP and private IP both recycled, together, to that live instance in the same VPC. The (external_ip, logical_ip) pair-key matches under identical recycling, so the existing logical-IP guard fell through. Stamp the owning ENI port (spinifex:logical_port) on the dnat_and_snat row at AddEIP, thread the port through handleDeleteNAT -> DetachEIP -> DeleteEIP, and skip the delete when the row's stamped port differs from the incoming one. The IPAM release was already owner-scoped; this mirrors that guard at the NAT layer so any stale delete for a recycled pair is a no-op. Empty port preserves the prior pair-key behaviour.	2026-06-15 23:27:39 +10:00
Julian Sommer	50b5495d26	chore: more comment cleanup	2026-06-12 12:38:31 +10:00
Julian Sommer	bcb49cd771	feat: IMDS v1 (#266)	2026-06-04 13:32:44 +10:00
Josh Siviour	40f8c4da94	refactor(e2e/single): drop phase numbers + parallel bucket #1 (#247) ... - Rename 36 phaseN_X / phaseIAM_N funcs to runX. - Drop numeric suffix from 7 TestIAMN_X wrappers (TestIAM1_UserCRUD -> TestIAMUserCRUD). - Rewrite harness.Phase log strings "Phase N - X" -> "Single - X" to mirror multinode convention. - Add t.Parallel() to bucket #1: 10 read-only / own-everything top-level Tests (Environment, ClusterStatsCLI, Discovery, SerialConsoleAccess, KeyPairs, Image, TagManagement, AccountScoping, VPCSubnetE2E, RouteTableValidation). Sequential bucket retains source-order singleton lifecycle (InstanceLaunch through FinalClusterStats).	2026-05-21 22:47:26 +10:00
Josh Siviour	050ac8f861	feat(e2e): Go E2E harness + single-node migration to Ensure*/Discover* + reboot suite (#241) ... * feat(e2e): add Go test harness + cert scenario, port run-cert-e2e.sh Lands step 1 of docs/development/improvements/e2e-go-harness.md: shared harness primitives (env, wait, artifacts, cert, tls) plus the TestCertIssuance scenario that supersedes run-cert-e2e.sh. All files gated by //go:build e2e so default builds and `make preflight` are unaffected. Bash script stays in place for one CI cycle before deletion. * feat(e2e): port run-lb-e2e.sh to TestLoadBalancer Go scenario Step 3 of docs/development/improvements/e2e-go-harness.md. Replaces 1246 lines of bash with a typed Go scenario exercising all 4 LB variants (ALB/NLB × internet-facing/internal) against a shared VPC + app instance pool. Internet-facing phase auto-skips when no peer node is configured; internal phase always runs. Adds shared harness primitives: - harness/aws.go: AWS SDK client factory wired to Spinifex AWS gateway - harness/lb.go: WaitForLBActive, WaitForTargetsHealthy, WaitForENICleanup, WaitForInstanceRunning/Terminated, InstanceENI - harness/traffic.go: HTTPRoundRobin, TCPRoundRobin, distribution asserts - harness/ssh.go: minimal peer SSH wrapper shelling out to /usr/bin/ssh (replaced by typed x/crypto/ssh client hoisted from DDIL in mulga-siv-74) User-data scripts embedded via //go:embed under lb/testdata/. Bash script stays in place for one CI cycle before deletion. * fix(e2e): log client launch params + dump daemon log on InvalidInstanceType Adds diagnostics for the LB internal client launch path. Logs the exact InstanceType, AMI, subnet, key name, and user-data size right before RunInstances, and registers a daemon-log dump on failure so we can correlate the 400 InvalidInstanceType with what spinifex saw. * refactor(e2e): serialise LB suites + share probe client across internal tests Splits TestLoadBalancer into 4 sequential subtests (ALB/NLB × internet-facing/internal). Each suite creates its own TG + LB + listener, runs its traffic test, and tears down before the next starts. Replaces the per-suite throwaway client VM with a single long-lived probe server launched at fixture setup; per-suite internal traffic tests GET /probe?ip=...&proto=... against it instead of cold-booting a fresh client. Drops peak instance count from 5 to 4 (2 app + 1 LB + 1 client), so the internal phase passes on capacity-constrained dev nodes that previously hit InvalidInstanceType (see mulga-siv-76, mulga-siv-77 for the underlying spinifex bugs). * feat(daemon): make hostReserve env-configurable for dev capacity squeeze Adds SPINIFEX_RESERVED_VCPU and SPINIFEX_RESERVED_MEM_GB env-var overrides read at daemon startup. Invalid values are logged and ignored so a typo doesn't silently widen the reserve. Unblocks the LB E2E on capacity-tight dev hosts (8 vCPU box: default 2-vCPU reserve + 3x t3.nano test load leaves 0 free for the sys.micro LB VM). Lays groundwork for the planned operator-tunable [capacity] config. bead: mulga-siv-78 * fix(e2e): revert LB to per-suite client + fail-fast on terminal LB state Reverts the shared probe-server VM that wouldn't come up reliably during the InternalFacing suites — back to the proven bash pattern of launching a fresh client VM per suite (one of ALB_IP / NLB_IP populated based on the suite kind). Adds renderClientUserData + a conditional template so each client probes only its target LB. WaitForLBActive now bails immediately on terminal LB states (failed, provisioning_failed) and emits a "still waiting" log line every 30s so hangs are visible instead of silently burning the full 5-minute timeout. * ci(e2e): add e2e-go.yml workflow + GOTESTSUM Makefile opt-in Standalone workflow runs the new Go-based E2E harness on the existing ci-single self-hosted runner — provisions a tofu VM, bootstraps spinifex, installs gotestsum, runs each requested suite via the e2e Makefile, and uploads per-test artifacts + JUnit XML + JSON streams as a build artifact. Test summary parses the JUnit and emits ::error file=...:: annotations so failures surface in the PR diff. Gated to the feat/e2e-go-harness branch + workflow_dispatch so it doesn't interfere with the existing e2e.yml while the Go conversion lands. tests/e2e/Makefile gains a GOTESTSUM=1 opt-in: when set, the e2e target runs gotestsum with --junitfile and --jsonfile rooted in ARTIFACT_DIR under a suite-tagged filename derived from the first PKG component. * feat(e2e): share probe client across both internal LB suites + SG fixes - Replaces per-suite probe client with a single shared VM launched in the fixture. Saves ~60s by avoiding a cold boot between the ALB and NLB internal suites. - Client cloud-init exposes a JSON trigger endpoint on :9090 in addition to the http.server on :80 (results dir). Each suite POSTs {proto, ip, count, outfile, http_port, tcp_port} to run a probe loop against the LB private IP and writes the result file under /tmp/httpd for the driver to fetch back. - configureDefaultSG now uses the structured IpPermissions form (top-level shortcut is silently ignored by vpcd — tracked separately) and opens port 9090 in addition to 80 + 9000. - deleteLB captures the underlying sys.micro VM id via the LB ENI before delete, then polls DescribeLoadBalancers until LoadBalancerNotFound so the next suite doesn't race the capacity reclaim. - Internal_NLB sleeps 15s up front to let the ALB sys.micro deallocate fully before its own createLB fires. DescribeLoadBalancers can report the ALB gone before the vCPU/memory allocation is actually released on capacity-tight dev hosts; without this settle, NLB's createLB trips the sys.micro reserve and lands in terminal state=failed. Still cheaper than cold-booting a per-suite client. * ci(e2e): install Go on tofu VM before running Go E2E suites The tofu-provisioned VM ships with the spx binary and bash test scripts but no go toolchain, so the prior workflow's "Install gotestsum on VM" step failed with `bash: line 2: go: command not found`. Add a curl-based Go 1.26.3 install (matching go.mod) ahead of the gotestsum install and update the test step's PATH so make e2e-* runs the freshly-installed go binary. * ci(e2e): bump gotestsum to @latest for Go 1.26 compatibility gotestsum v1.12.0 pins golang.org/x/tools v0.19.0 which fails to compile under Go 1.26 with: invalid array length -delta * delta (constant -256 of type int64) Switching to @latest pulls a newer x/tools that builds clean. * ci(e2e): ship spinifex source to VM so make -C tests/e2e can run there Single-node tofu VM only has the spx binary + bash test bundle at ~/e2e-tests; the Go e2e workflow was running 'make -C tests/e2e e2e-cert' from $HOME/Development/mulga/spinifex which doesn't exist, producing: make: *** No rule to make target 'e2e-cert'. Stop. Add a checkout of the spinifex repo on the runner and rsync it (minus .git, build/, bin/, node_modules) to the VM as ~/Development/mulga/spinifex before installing Go + gotestsum. Interim solution; mulga-siv-80 tracks replacing the rsync with shipping pre-built test binaries from the runner. * ci(e2e-go): replace rsync with tar-over-ssh; VM lacks rsync bootstrap-install.sh ships a minimal binary VM with no rsync; the ship-source and collect-artifacts steps failed with 'rsync: command not found'. Stream a tarball over ssh instead — same exclude list, no extra dependency on the VM. * fix(e2e): discover single-node IPs + awsgw bind in LoadEnv run-cert-e2e.sh single mode probes awsgw.host from spinifex.toml and adds every non-loopback global IPv4 to the node list. The Go harness defaulted to 127.0.0.1 only, so CI on a tofu VM (awsgw bound to the LAN IP, not loopback) saw 'connection refused' on every TLS check. Port the same discovery: ModeSingle now walks net.InterfaceAddrs for NodeIPs and grep-parses spinifex.toml for the awsgw bind. Both stay overridable via SPINIFEX_NODE_IPS / SPINIFEX_SERVICE_IPS. * ci(e2e-go): pre-build test binaries on runner; drop go toolchain from VM Replaces the source-tree + go + gotestsum payload that the e2e-go.yml workflow previously shipped to the tofu CI VM with self-contained test binaries built on the runner via `go test -c`. testdata for the LB suite is already //go:embed'd into the binaries, so the bundle on the VM is just `<suite>.test` files (cert: 14M, lb: 17M uncompressed). Workflow changes - actions/setup-go on runner. - New step builds binaries via `make e2e-build SUITES_BUILD=...`. - Install go-junit-report on runner; convert each VM-side test log to junit-<suite>.xml after artifact pull so "Publish test summary" still parses JUnit XML. - Drop "Install Go + gotestsum on VM" entirely (saves ~60s + network). - Drop source-shipping rsync/tar (saves ~30s + a lot of bandwidth). - VM step runs `./<suite>.test -test.v -test.timeout=30m` directly. Makefile changes - New `e2e-build` target compiles each `SUITES_BUILD` entry into `BIN_DIR/<suite>.test` via `go test -c -tags=e2e`. `clean` also removes BIN_DIR. Closes mulga-siv-80. * ci(e2e-go): disable setup-go cache; upload dominates self-hosted run time * refactor(e2e): hoist generic DDIL harness primitives to shared tests/e2e/harness Step 1.5 of docs/development/improvements/e2e-go-harness.md. Promotes Cluster/Node/SSH/DaemonClient/LocalStateRevision/ResetAllNodes/Witness into the shared package so the LB, multinode, and BM scenarios in steps 4-5 can build on the same primitives instead of duplicating SSH + cluster plumbing. Moves - ddil/harness/{cluster,ssh,daemon_client,state,reset,witness}.go -> harness/ (plus the witness testdata embed dir). - ddil/harness/{fault,netem,profiles,snapshot}.go -> ddil/fault/ as a new package; fault.PartitionNode/HealNode/KillNATS/StartNATS/ RestartDaemonOnly/ApplyNetem/ClearNetem/LinkProfile*/TakeSnapshot now take harness.Cluster / harness.Node / harness.SSH explicitly. - ddil/harness/retry.go::Run retry-with-reset extracted as harness.RetryWithReset in harness/wait.go; ddil/harness/quarantine.go keeps the DDIL-only quarantine wrapper that calls into it. - WaitForMode moved from fault.go to harness/daemon_client.go since it polls daemon state and has no fault-injection concern. - The pre-hoist harness/ssh.go (PeerSSH, subprocess ssh) is preserved as harness/peer_ssh.go so the lb_test suite keeps working unchanged; the typed x/crypto/ssh client from DDIL now owns harness/ssh.go. - New harness/util.go exports ShellQuote (was unexported shellQuote in netem.go) so both ddil/fault and reset.go/witness.go can use it. Env var rename - SPINIFEX_NODES / SPINIFEX_SSH_USER / SPINIFEX_SSH_KEY replace DDIL_NODES / DDIL_SSH_USER / DDIL_SSH_KEY. Old names still honoured with a stderr deprecation note (envWithLegacy) so the DDIL CI keeps passing until .github/workflows/e2e-ddil.yml drops the old names. DDIL_QUARANTINED stays DDIL-specific. DDIL parity preserved — scenarios still call ddilh.Run for quarantine semantics; the underlying retry-with-reset is the same code path, just shared. Closes mulga-siv-74. * fix(e2e): open default SG :22 ingress via structured form before witness launch The daemon currently drops the top-level shortcut AuthorizeSecurityGroupIngress form silently (mulga-siv-79), so the default SG starts with no inbound :22 and the witness EIP awaitBaseline tunnel times out under DDIL CI. Mirror the workaround already present in lb_test by issuing the structured IpPermissions form before LaunchWitnessVM. Idempotent on InvalidPermission.Duplicate so reruns in the same cluster are safe. * ci: render JUnit into job summary + collapsible groups in noisy steps Add a reusable composite action (.github/actions/junit-report) that parses a glob of JUnit XML files, writes a totals table + collapsible per-failure details into $GITHUB_STEP_SUMMARY, and emits one ::error annotation per failed/errored test case so failures surface in the run header instead of being buried in step logs. Apply the action to both e2e-go.yml and e2e-ddil.yml (e2e-ddil gotestsum now also emits --junitfile so the action has input). Same workflows also wrap noisy phases (tofu apply/destroy, scrub-env, tar-over-ssh, per-suite test runs, cluster-log collection) in ::group::/::endgroup:: markers so the Actions UI collapses the chatter by default and the human can expand only the failing phase. Tofu output still tees to $TOFU_LOG so the artifact bundle is unchanged. Future workflows (multinode, baremetal) can adopt the same pattern via the README in .github/actions/junit-report. * test(ddil): quarantine Scenario C until predastore §2 lands CI run 26016840819 surfaced that ScenarioC_CleanPartition silently depends on predastore-ddil-hardening §2 (write-quorum with deferred parity repair). Root cause from node3 daemon log: predastore: write udp ...->192.168.0.22:9991: sendmsg: operation not permitted predastore: putObjectViaQUIC: put failed spx QMP: BLOCK_IO_ERROR instanceId=<witness> spx: StopOrTerminateInstance: Terminating spx: Instance state transition running → shutting-down The 'sendmsg: operation not permitted' is the EPERM Linux returns when iptables drops an outbound packet — exactly the OUTPUT DROP rules PartitionNode installs. The witness VM's block I/O is backed by distributed predastore, so the same partition that severs node3↔peer NATS traffic also severs the witness disk's cross-shard QUIC writes. QEMU power-downs the guest after BLOCK_IO_ERROR, the daemon's reconciler terminates the instance, and the next AssertProgressed dials a sshd that's already gone (hence the 'handshake failed: EOF' the test reports). Quarantine by collapsing C to the same scenarioSkip stub D/E/F already use, citing predastore-ddil-hardening §2 as the dependency. Flip C back to ENABLED in the same PR that lands §2 — that's when the witness disk can survive a peer partition without QEMU halting. TEST_COVERAGE.md row C flipped ENABLED → SKIPPED; added a quarantine note explaining the predastore dependency. * ci: colored phase banners + quiet-on-success tofu output Pretty up the live Actions log without flooding it. Three changes: 1. .github/scripts/ci-fmt.sh — sourceable bash helpers (banner, ok, bad, warn, note, quiet_run) using ANSI colors that the Actions UI renders. 2. e2e-go.yml + e2e-ddil.yml — tofu phases switch back to quiet redirect into $TOFU_LOG (the previous tee'd version regressed live noise vs the original silent redirect). Each phase now prints one colored ok/✗ line; on failure the tail of $TOFU_LOG dumps inside a single collapsed group, so the live log stays scannable but the failing tail is one click away. 3. junit-report render.py — compact summary: short status icons (✅/❌) in a single column, no separate err column, single-line failure list (truncated to 120 chars) instead of <details> blocks that wrap badly on narrow screens. Per-suite test runs still wrap in ::group:: with an inline cyan banner so pass output collapses and the suite name is obvious when expanded. * test(e2e): add harness.Phase / Step / Detail for human-readable test output Test fixtures were emitting flat walls of t.Logf with every line prefixed by the same `lb_test.go:61:` (file:line of the call site that the t.Helper() chain bubbles up to) and forced 6-space indent. Hard to skim, hard to find where one phase ends and the next begins. Add three helpers in tests/e2e/harness/log.go: Phase(t, fmt, ...) — flush-left colored banner via fmt.Println. Bypasses t.Logf so the line lands without the testing framework's indent/file:line prefix. Pure visual structure; the diagnostic detail stays on t.Logf. Step(t, fmt, ...) — single-line sub-phase marker via t.Logf. Keeps file:line attribution so a triager can find the source of a progress message. Detail(t, k, v, ...) — key=value resource IDs via t.Logf. Same attribution rationale as Step — IDs are diagnostic and want to land in JUnit XML bucketed with the owning test. Why the split: Phase is purely a banner; losing JUnit attribution costs nothing because banners aren't diagnostic. Step/Detail carry triage context (instance IDs, AMI, etc.) and a failing test's JUnit testcase node should still contain them, so t.Logf attribution is kept. ANSI colors are auto-disabled when stdout isn't a TTY and GITHUB_ACTIONS isn't set, so piped local `go test -v` stays clean. Apply to lb_test setupSharedFixture — five Phase banners replace the implicit run of helper logs ("Discovering cluster capacity", "Ensuring SSH key pair", "Creating shared VPC topology", "Launching app instances", "Launching probe client") so the live CI log reads as a story rather than a flat resource dump. * ci(e2e-go): strip indent + file:line on display, propagate color env over SSH Two display bugs spotted on the previous run: 1. Phase banners came out white instead of bold-cyan because the test binary runs over SSH on the tofu VM, where neither GITHUB_ACTIONS nor a TTY is present, so harness.log.go init() disables ANSI escapes. Fix: export GITHUB_ACTIONS=true and FORCE_COLOR=1 in the remote env so the helper keeps emitting color codes. 2. Every t.Logf line still rendered as lb_test.go:61: instance type: t3.nano (4-space indent + `<file>:<line>:` prefix) because t.Helper() bubbles file:line to the call site. Fix: pipe the test binary's stdout through `sed -E 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'` so the display strips both. Tee to the artifact log happens BEFORE the sed so raw output is preserved for triage; only the live UI sees the cleaned form. * feat(e2e-harness): extend harness for single-node Go suite port Stage A of mulga-siv-89 — add the primitives the single-node run-e2e.sh port consumes: - AWSClient gains IAM + STS service handles, plus NewAWSClientWithCreds for per-tenant account-scoping in Phase 8. - awserror.go: ErrorCodeIs, AssertAWSError, ExpectError — Go equivalent of the bash expect_error helper used throughout the negative phases. - poll.go: WaitForInstance/Volume/Snapshot/Image state to any target, with per-resource default timeouts tuned from the bash sleep loops. - spxcli.go: thin shellouts for spx get nodes/vms, spx top nodes, spx admin account create with output parsing for tenant-id/access-key. - rawhttp.go: signed raw POST against the AWS gateway for the Phase 8 InvalidAction probe (gateway rejects on auth before action parsing). - ovn.go: OvnNbctl/OvnSbctl wrappers + SkipIfNoOVN for Phase 8b-e datapath checks; WaitForPortGroupMember handles propagation polls. - ec2helpers.go: InstancePublicSSHHost (public-IP-or-qemu-hostfwd), LsblkRootGiB, serial-console-access trio, DiscoverDefaultVPC, idempotent AuthorizeSSHIngress. - account.go: AccountCarousel maps tenant name -> scoped AWSClient, built atop NewAWSClientWithCreds for the Phase 8 isolation tests. Pure groundwork — no new tests, no production behaviour change. The cert + lb scenarios still compile against the extended harness. * ci(e2e-go): auto-cancel + reap orphans to prevent runner wedge A cancelled run on the single-runner ci-single host has been leaving orphan tofu/terraform procs (and stale tfstate locks) behind, which caused the next push's tofu destroy to hang for 45m+ and saturate the runner host until sshd refused new connections. - Add `concurrency: e2e-go-${ref}, cancel-in-progress: true` so a new push to feat/e2e-go-harness cancels the in-flight run instead of stacking on the same self-hosted runner. - New `Reap Orphan Procs` step at job start: SIGTERM then SIGKILL any lingering `tofu`/`terraform` from the runner user, and unlink any stale `.terraform.tfstate.lock.info`. - Harden `Teardown`: drop `set -e` (scrub-env must always run as the libvirt-level fallback), reap orphans before destroy, and wrap `tofu destroy` in `timeout 600` so a wedged provider can't pin the job to its 45m timeout. * ci(e2e-go): drop push trigger, manual dispatch only Pull the on.push branch trigger so the workflow only fires from workflow_dispatch — matches the e2e-ddil.yml Phase-1 pattern. Push trigger lifts again once the single-node Go port (mulga-siv-89) is green end-to-end and run-e2e.sh is retired. * test(e2e-single): port run-e2e.sh phases 1-4 to Go Stage B of mulga-siv-89. Lands the TestSingleNode orchestrator + Fixture that threads state between sequential phase subtests, plus phases: - Phase 1: KVM check, AWS gateway TLS reachability, daemon NATS readiness polling, region/AZ smoke. - Phase 1b: spx get nodes / top nodes / get vms (single-node only). - Phase 2: nano instance-type + arch discovery. - Phase 2b: serial-console-access default/enable/disable round-trips. - Phase 3: create-key-pair + import-key-pair lifecycle; test-key-1 PEM kept for Phase 5 instance launch (cleanup deferred to teardown phase in stage G). - Phase 4: architecture-aware AMI discovery via name filter ami-ubuntu-24.04-{x86_64,arm64}, falls back from spx import output. Phase ordering is contractual (Phase 5+ depend on AMI, key pair, AZ). detectPoolMode reads [network] external_mode from spinifex.toml so stages F/G can skip pool-only phases on dev_networking fixtures. No production code changed. cert + lb scenarios continue to compile. * test(e2e-single): port run-e2e.sh phases 5-5e to Go Stage C of mulga-siv-89 — the instance/volume/snapshot/image lifecycle that everything else hangs off: - Phase 5: run-instances (default VPC + default SG with tcp/22 opened via idempotent harness.AuthorizeSSHIngress), wait running, resolve root volume from BlockDeviceMappings. - Phase 5a-pre: spx get vms now contains the running instance. - Phase 5a: describe-instances metadata round-trip (type/key/AMI/BDMs). - Phase 5a-ii: TCP wait, ssh id, lsblk cross-check vs API root volume size (±1 GiB tolerance — lsblk rounds down), soft hostname check. - Phase 5a-iii: get-console-output round-trip on InstanceId. - Phase 5b: create-volume (10 GiB) → modify-volume (20 GiB) → attach-volume → detach-volume → delete-volume, polling each state transition through harness.WaitForVolumeState. - Phase 5b-ii: describe-volume-status on the root volume. - Phase 5c: create-snapshot → wait completed → copy-snapshot → both visible → delete original → copy survives → delete copy. - Phase 5d: AMI snapshot-backed launch verification via DescribeImages.BlockDeviceMappings[].Ebs.SnapshotId. Predastore-side config.json reads are TODO (no S3 client in harness yet). - Phase 5e: create-image --no-reboot, wait available, capture backing snapshot ID for Phase 9 cleanup. Phase 7a (attach to stopped → error) stubbed in volume_test.go for Stage D to fill once the lifecycle stop/start lands. * test(e2e-single): port run-e2e.sh phases 5f-8 to Go Stage D of the single-node Go port (mulga-siv-89). Adds four files under tests/e2e/single/ covering Phase 5f (SG egress ACL enforcement), Phase 6 (tag CRUD), Phases 7 / 7a / 7b / 7c-pre / 7c (stop, start, attach-to-stopped negative, ModifyInstanceAttribute, reboot, multi-count RunInstances), and Phase 8 negatives (18 sub-tests asserting AWS error codes via harness.ExpectError / AssertAWSError / PostAWSAction). Wired into TestSingleNode in main_test.go; phase ordering preserved because Phase 7 chain mutates the shared instance state (stopped at end of 7, .small after 7b, running after 7c-pre, untouched by 7c siblings) and Phase 8 negatives need a running instance with the AMI/key created earlier. Removes the placeholder phase7a_AttachToStoppedError stub from volume_test.go (real impl now in lifecycle_test.go). run-e2e.sh stays in tree until step-7 cleanup per the parent plan. * ci(e2e-go): wire e2e-single target + add single to build set Lets the in-progress single-node Go port (mulga-siv-89, Stages A-D landed) be exercised in CI via explicit `suites: e2e-single` dispatch input ahead of Stage G flipping the workflow default. Workflow default suites unchanged until Phase 9 teardown lands. * test(e2e-single): port run-e2e.sh IAM 1-7 + Phase 8 account scoping to Go Stage E of the single-node Go port (mulga-siv-89). Adds two files under tests/e2e/single/: - iam_test.go (903 lines): IAM Phases 1-7 — User CRUD, Access Key Lifecycle, User Authentication (via harness.NewAWSClientWithCreds), Policy CRUD (+ CreatePolicyVersion / SetDefaultPolicyVersion / DeletePolicyVersion which bash lacks), Policy Attachment & Enforcement (allow / deny / wildcard / root bypass), Policy Lifecycle (DeleteConflict while attached), and idempotent Cleanup. - account_test.go (1186 lines): Phase 8 EC2 Account Scoping — 11 Steps (Account Setup, Instance / Volume / KeyPair / Snapshot / VPC-Subnet / IGW-EIGW Scoping, Account Settings, Global Resources, Edge Cases, Cleanup). Uses harness.AccountCarousel for the multi-profile credential map and SpxAdminAccountCreate for admin account bootstrap. Fixture extended with IAM cross-phase state (IAMAdminAccount, IAM{Alice,Bob,Charlie}{KeyID,Secret}) so Phase 7 cleanup is defensive even when an intermediate IAM phase short-circuits. All 11 Phase 8 Steps share local state in the outer function; t.Cleanup mirrors Step 11 LIFO so a mid-Step failure still tears down accounts. Wired into TestSingleNode. run-e2e.sh stays in tree until step-7 cleanup per parent plan. * fix(e2e-harness): use kebab-case is-default filter for DescribeVpcs AWS EC2 DescribeVpcs filter names are kebab-case (is-default), and the daemon's filter allowlist enforces that. The harness sent the camelCase SDK field name (isDefault), so the daemon returned InvalidParameterValue 400 and DiscoverDefaultVPC fataled — leaving Phase 5 InstanceID empty and cascading 16 downstream Single-Node E2E subtests to fail. * test(e2e-single): port run-e2e.sh phases 8b-8e to Go Stage F of the single-node Go port (mulga-siv-89). Adds three files under tests/e2e/single/: - vpc_test.go (553 lines): Phase 8b (pool-mode VPC public/private subnet E2E — CreateVpc, dual subnets with MapPublicIpOnLaunch assertions, IGW attach, custom route table + 0.0.0.0/0 route, instance launch + SSH + external connectivity probe) and Phase 8c (route table validation — main RT association, local + IGW route presence, custom RT CRUD + association/disassociation, negative delete-local + delete-main assertions). - natgw_test.go (467 lines): Phase 8d (pool-mode NAT GW E2E — CreateNatGateway with EIP, private subnet route via NAT, bastion- hopped SSH into private VM, baseline egress drop / post-NAT egress succeed / post-teardown egress lost). Inline waitForNATGatewayState helper bounds the pending->available->deleted state machine via harness.EventuallyErr. - datapath_test.go (326 lines): Phase 8e (OVN-gated SG-to-SG datapath — two SGs with UserIdGroupPair ingress rule, OVN port-group membership + SB address-set wait loops, allowed traffic curl, denied traffic nc, revoke->drop, re-add->restore). Pool-mode gating via fix.PoolMode for 8b/8d; OVN gating via harness.SkipIfNoOVN for 8e. Per-phase t.Cleanup LIFO ensures teardown unwinds even on partial failure. Wired into TestSingleNode. * test(e2e-single): port run-e2e.sh phases 9, 9a, 9b to Go Stage G of the single-node Go port (mulga-siv-89) — final teardown + post-teardown verification. Adds tests/e2e/single/teardown_test.go (226 lines): - phase9_Teardown: DeleteSnapshot(CustomAMISnapID) -> DeregisterImage (CustomAMIID) -> TerminateInstances(InstanceID) + wait terminated -> DeleteKeyPair(KeyName). Each call swallows *NotFound via harness.ErrorCodeIs; unexpected errors logged as warnings so 9a/9b always run. - phase9a_VerifyTeardown: DescribeInstances must show terminated or NotFound; DescribeImages must NOT include the deregistered AMI; DescribeKeyPairs must NOT include the deleted key. Clears the consumed Fixture fields. - phase9b_FinalClusterStats: harness.SpxGetVMs must NOT list the terminated InstanceID; orphan i- rows surface as soft Logf warnings to avoid flakes when parallel suites share the node. Wired into TestSingleNode. With this, mulga-siv-89's single-node Go port (Stages A-G) is feature-complete; run-e2e.sh remains in tree until a separate cleanup bead deletes it after CI burn-in. * ci(e2e-go): unbuffer sed in test pipeline for live log streaming GNU sed defaults to block-buffered stdout when stdout is a pipe (GitHub Actions runner). Accumulates 4-64KB before flushing, so test output appeared in sporadic batches unaligned to test boundaries. -u flag forces line-buffered output so each go test -v line surfaces in the Actions UI as it is produced. Side effect: none. No process count change, no SSH session change, no test execution change. Pure output flushing on the runner host. * fix(e2e-single): persist key PEM across phases and skip unimplemented IAM ops Three bugs blocked the Single-Node Go E2E suite after the DescribeVpcs filter fix: 1. Phase 3 wrote the SSH PEM into `t.TempDir()`, which Go cleans up when the Phase 3 subtest exits. Phases 5a-ii / 5a-iii / 7b / 7c-pre then saw "Permission denied (publickey)" / "Identity file not accessible" and the cascade left the instance in a state that broke 7c-pre's `start-instances`. Move the PEM under a TestSingleNode-scoped `fix.TmpDir` so it survives until the whole test exits. 2. IAM Phase 2's `require.NotEmpty(t, fix.IAMAdminAccount, "")` was intended as a no-op (Phase 4 populates the field), but require.NotEmpty fails on empty regardless of message. Removed the bogus check. 3. The daemon does not yet implement `UpdateUser` (IAM Phase 1) or the policy-version family (IAM Phase 4): `CreatePolicyVersion`, `ListPolicyVersions`, `SetDefaultPolicyVersion`, `DeletePolicyVersion`. They return `InvalidAction` and abort the phase. Wrap each block in a t.Run sub-test with t.Skip so the surrounding phase still runs and the skips show up explicitly in the test report. * test(e2e): parallelise hot phases + tune fast-path polls Squash merge of feat/e2e-parallel-quickwins (mulga-siv-96). Four quick wins from decomposing run-e2e.sh into Go phase subtests: - Phase 8 negatives (single/negative_test.go): t.Parallel() inside each of the 18 read-only error-path sub-tests. ~5-10x speedup. - IAM Phase 4 policies (single/iam_test.go): EC2ReadOnly stays seeded sequentially (populates fix.IAMAdminAccount); the other four policies fan out as siblings under a wrapping create_policies_parallel t.Run. ~5x on the Phase 4 sub-step. - Volume + snapshot fast paths (single/{volume,snapshot}_test.go): pass harness.WithPoll(500ms) on the five state transitions that finish in <1s on local QEMU. Halves Phase 5b / 5c wall-clock. - Cert per-IP loop (cert/cert_test.go): t.Parallel() inside the 4 inner t.Run(ip, ...) blocks. Cuts cert phase by len(nodes). No harness changes, no Fixture changes, no test-execution-order changes. mulga-siv-96. * test(e2e-lb): resolve peer availability before shared fixture setup setupSharedFixture builds a VPC + 2 app VMs + probe client (minutes on capacity-tight dev hosts) and previously ran BEFORE the peer pickup + ssh.Ping. On single-node, the InternetFacing_* subtests would correctly t.Skip on peer == "" — but only after the fixture work and once subtests started, so the wait appeared to be attached to the first subtest name gotestsum printed. Reorder: pickPeer + ssh.Ping first, log "internet-facing will skip" immediately, then build the fixture for the internal subtests. No behavior change for multi-node + peer-reachable; on single-node the skip rationale surfaces before the fixture wait. * test(e2e-single): extend SSH wait budget to 3m on slow boot paths CI run 26034322018 timed out Phase5a_ii_SSH (60s budget) and Phase8b_VPCSubnetE2E (2m budget) on the TCP-port-open poll while sshd was still finishing pam/cloud-init. Bump all four wait sites to 3m as a stop-gap: - single/instance_test.go phase5aii_SSHProbe - single/lifecycle_test.go waitForSSHReady (Phase 7b/7c/7c-pre) - single/vpc_test.go phase8b_VPCSubnetE2E - single/natgw_test.go phase8d bastion SSH wait Proper fix tracked under mulga-siv-101 (hoist a harness.WaitForSSH that probes the actual handshake, not just tcp/22 reachability). * test(e2e-single): skip-gate IAM3/IAM5/IAM6 until daemon handlers land CI run 26034322018 surfaced three IAM daemon gaps: - IAM3_UserAuthentication: alice's freshly-created access key returns 403 from DescribeInstances — daemon doesn't honour scoped IAM key signatures yet. - IAM5_PolicyAttachmentEnforcement: cascades from IAM3 (alice can't sign). - IAM6_PolicyLifecycle: DetachUserPolicy returns 404 even on attachments Phase 5 confirmed — daemon attachment ledger gap. Skip-gate each phase with t.Skip + a pointer to mulga-siv-100 so the remaining IAM coverage (IAM1/IAM2/IAM4/IAM7) keeps surfacing regressions, and the suite as a whole doesn't fail on known gaps. Flip to real assertions once the daemon handlers land. * test(e2e-single): probe full SSH handshake (not just TCP port open) CI run 26036025293 hit "Connection timed out during banner exchange" on Phase5a-ii / 8b / 8d even with the 3m TCP-port-open budget: tcp/22 opens immediately, but sshd is still finishing pam/cloud-init, so the first real runSSH call afterwards races the half-open daemon. Switch waitForSSHReady to issue an actual ssh BatchMode + ConnectTimeout handshake against the target, retrying until the daemon completes the banner exchange. Drop the four TCP-only Eventually loops in favour of the upgraded helper: - instance_test.go phase5aii_SSHProbe - lifecycle_test.go waitForSSHReady (Phase 7b / 7c-pre / 7c) - vpc_test.go phase8b_VPCSubnetE2E - natgw_test.go phase8d_NATGateway (bastion) Signature: waitForSSHReady(t, host, port, keyPath). Backwards-compat alias waitForSSHHandshake exposed for the instance_test caller. Proper harness hoist tracked under mulga-siv-101. * test(e2e-harness): dual-emit Step + Detail for live CI streaming go test's testing framework buffers t.Logf output per-test and releases it only on PASS/FAIL. Long subtests (Phase8Acct_AccountScoping ~115s, Internal_NLB ~60s, Phase8b_VPCSubnetE2E 2-3m, Phase8d_NATGateway ~90s) therefore went silent in CI for the entire subtest duration, even with sed -u line-buffering in the workflow pipeline. harness.Phase already wrote to os.Stdout directly for this reason. Extend the pattern: have Step and Detail dual-emit — write the same line to os.Stdout (streams live) AND t.Logf (keeps JUnit per-test attribution). The duplicate on a failure is acceptable; both copies carry identical content. mulga-siv-103 (note: numbering skipped past siv-102 which was already filed in the parent branch). * test(e2e-harness): drop t.Logf duplicate from Step + Detail The dual-emit (Fprintln + t.Logf) caused every Step/Detail line to appear twice after long subtests: once live via stdout, once again when the testing framework released its per-test buffer at PASS/FAIL. Most visible after lb subtests (Internal_NLB ~60s) where the bulk release dumped ~dozens of duplicated lines at once. Drop the t.Logf copy. go-junit-report parses stdout between `=== RUN` and `--- PASS/FAIL` markers, so per-test attribution in JUnit XML is preserved without the duplicate. Refs: mulga-siv-103 * test(e2e-harness): stream lb-waiter progress live in CI WaitForLBActive, WaitForTargetsHealthy, WaitForENICleanup, and WaitForInstanceRunning emitted progress via t.Logf, which the testing framework buffers per-test. After the previous commit dropped the dual-emit, that meant every poll loop went silent until subtest end — visible in CI as 'nlb is active' hanging on screen for the whole duration of the subsequent traffic check. Route progress through harness.Step (direct stdout). Also emit a 'waiting for X' Step at the start of each helper so observers know what the silent stretches actually represent. Refs: mulga-siv-103 * test(e2e): dump VPC/IGW datapath diagnostics on Phase8b SSH timeout Phase8b's 3min SSH timeout on a fresh-VPC VM (CI run 26037892507) is the customer-visible surface of mulga-siv-105 — vpcd's handleIGWAttach returns before ovn-northd compiles + the gateway chassis installs flows. Two other cleanup-only bugs (siv-106 missing spinifex:cidr, siv-107 dnat_and_snat double-delete) showed up in the same triage. Keep the test hard-failing so the regression stays visible, but dump a triage bundle on timeout so each occurrence is actionable: - guest console output (last 4KB — boot + cloud-init state) - ovn-nbctl show + ovn-sbctl show - Logical_Router external_ids (confirms siv-106 reproduction) - NAT rules (confirms siv-105 + siv-107 surface) - Port_Binding chassis,up (confirms gateway port up state) Diagnostics go straight to stdout (Step/Detail rationale) so they stream live in CI rather than dumping at PASS/FAIL. Refs: mulga-siv-105, mulga-siv-106, mulga-siv-107 * ci(junit-report): drop parent t.Run containers from tally + list skips with reasons GitHub Actions test summary was counting Go's top-level Test* parent plus its failing subtest as two distinct failures (the parent is just a rollup of its children). The summary now de-dupes by dropping any test case whose name is a strict prefix of another's, so the per-file counts and the Failures block both match the actual leaf-test outcome. CI run 26037892507 was reporting '2 failed' for single when it should have been 1. Also adds a collapsible Skipped section listing each skipped leaf test with its t.Skipf call-site reason (the .go:<line>: line from the CDATA). ANSI escapes from harness.Phase banners are stripped so the reason is plain text. Reasons are useful triage signal — current run has 9 skips spread across daemon gaps (mulga-siv-100) and pool-mode gates (InternetFacing_*). Refs: mulga-siv-89 * ci(e2e-go): propagate cancel to remote .test procs + nuke VM before destroy Two related leaks made cancelled runs strand QEMU work on the libvirt host: 1. The long `ssh tf-user@VM "for SUITE in …; do ./$SUITE.test"` ran without -tt, so when the runner cancelled the step the local ssh died but sshd never sent SIGHUP through (no PTY). The remote `.test` binary orphaned and kept running until its own 30m `-test.timeout`, holding tmp dirs, NATS connections, and qcow2 file handles. 2. Teardown went straight to `tofu destroy` without first killing any stragglers on the VM, so destroy raced the .test's open file handles on the root disk — sometimes successfully, sometimes leaving the libvirt host with a half-released qcow2. Fix: - Allocate a PTY on the run-suites ssh (`-tt`) so sshd HUPs children when the local ssh dies. - Add a trap (EXIT INT TERM) in the run-suites step that ssh's back to the VM and pkill's `/spinifex-e2e-bin/*.test` belt-and-braces. - Mirror the same pkill at the top of Teardown so destroy runs against a quiesced VM. Both cleanups are scoped to the per-run wan_ip — safe to land while multiple ci-single runner agents run concurrent jobs against different tofu envs on the same host (we deliberately keep that parallelism). Refs: runner host wedged 2026-05-19 (post-cancel orphans suspected; no journal access yet to confirm). * ci(e2e-go): scope orphan reap to GITHUB_WORKSPACE so it spares sibling runners Reap Orphan Procs (job start) and the in-Teardown pkill block were using `pkill -u $UID_SELF -x tofu/terraform` plus `find $HOME -maxdepth 6 -name .terraform.tfstate.lock.info`. The single host runs 5+ self-hosted runners all as `tf-user`, so every e2e-go run was SIGKILLing in-flight tofu from sibling workflows (e2e.yml, e2e-ddil.yml, e2e-baremetal.yml, ...) and deleting their state-lock files — exactly the kind of cross-runner collateral the reap was meant to recover from. Scope by `/proc/$pid/cwd` starting with $GITHUB_WORKSPACE; each runner's workspace dir is unique on the host, so only this run's own tofu procs match. Find for stale lockfiles is likewise scoped to $GITHUB_WORKSPACE. * ci(e2e-go): bump setup-go v5->v6, upload-artifact v4->v7 GitHub now warns Node 20 actions are deprecated. e2e-go.yml lagged the rest of the repo; bump to match versions already in use: - actions/setup-go@v5 -> @v6 (matches spinifex.yml) - actions/upload-artifact@v4 -> @v7 (matches release.yml, e2e-ddil.yml) checkout@v6 and create-github-app-token@v3 are already current across all workflows. The local junit-report action is `using: composite` (shell-only), so the Node runtime change does not apply. * fix(vpcd): add flows-ready barrier after IGW attach (mulga-siv-105) handleIGWAttach wrote the external switch + gateway port + LRP + default route to OVN NB and returned immediately, before ovn-northd had compiled NB -> SB and before ovn-controller on the gateway chassis had installed the resulting flows. A subsequent RunInstances -> vpc.add-nat could complete while the datapath was still dark; the launched VM was then unreachable on its public IP for whatever flow-install latency the chassis happened to incur. Customer-visible as 'I just created a VPC, launched a VM, and can't SSH in' for ~minutes. Reproduced deterministically in CI run 26037892507 Phase8b: a VM in a fresh VPC was unreachable at 192.168.0.202:22 for 3+ minutes, while a VM at the same public IP in the warm default VPC (Phase8d) was reachable in ~40 seconds. Block at the end of handleIGWAttach on `ovn-nbctl --wait=hv sync`, which bumps NB_Global.nb_cfg and waits for every connected chassis to acknowledge the new sequence number — i.e. flows installed. Bounded by --timeout=30; overruns log a WARN and continue (a 30s OVN stall is already a deeper problem than failing VPC create would surface). waitForFlowsHV is a package var so tests can stub it; in environments without ovn-nbctl on PATH the shell-out fails fast and the WARN path returns nil, so existing IGW tests stay green without a live OVN. Refs: mulga-siv-105 * fix(vpcd): store spinifex:cidr on VPC create (mulga-siv-106) handleVPCCreate built the OVN logical router with only spinifex:vpc_id and spinifex:vni in ExternalIDs. Cleanup paths (handleIGWDetach + NAT sweeps) read external_ids['spinifex:cidr'] to scope SNAT removal to the VPC's own CIDR; when unset, they fell back to an overbroad 10.0.0.0/8 and emitted a WARN. reconcileVPC already stored the same key at startup; the live-create path drifted. Add the field to handleVPCCreate, log the cidr alongside vpc_id, extend TestTopologyHandler_VPCCreate to assert the external_id is set. Refs: mulga-siv-106 * fix(vpcd): make handleDeleteNAT idempotent on 'already absent' (mulga-siv-107) Instance termination fans out through multiple cleanup paths (instanceCleanerAdapter.ReleasePublicIP AND VPCServiceImpl.DeleteNetworkInterface), both of which publish vpc.delete-nat for the same (vpc, logical_ip). The publish is fire-and-forget (utils.PublishNATEvent uses PublishEvent for vpc.delete-nat, not RequestEvent), so callers never branch on the response. The second publisher then triggered a spurious 'ERROR vpcd: failed to delete dnat_and_snat rule ... not found' log line ms after the first delete succeeded — confusing triage signal with no actionable consequence. Add ErrNATNotFound sentinel to the OVN client interface and wrap the not-found return paths in both LiveOVNClient and MockOVNClient. handleDeleteNAT matches errors.Is(err, ErrNATNotFound) and treats that as success (logs INFO 'already absent (idempotent)'). Existing TestTopologyHandler_DeleteNAT_NotFound asserted the opposite contract; the comment claimed EIP disassociation needed the error signal, but DisassociateAddress also fire-and-forgets vpc.delete-nat, so the contract never had a real consumer. Test renamed to _Idempotent with assertSuccess. Other failure modes (router missing, OVN unreachable, transact error) still surface as ERROR + reply with the error string. Refs: mulga-siv-107 * ci(e2e-go): cluster JUnit failures + name likely root cause (#230) * ci(e2e-go): cluster JUnit failures + name likely root cause New composite action `.github/actions/e2e-analyze` runs after `go-junit-report` converts each suite log. For every `<failure>` it strips ANSI, normalises dynamic noise (EC2 IDs, UUIDs, IPs, timestamps, file:line) to a stable signature, picks the earliest non-cascade failure per suite as the likely root cause, and buckets the rest by signature. Output is appended to `$GITHUB_STEP_SUMMARY` and written to `<log-dir>/analysis.md` so it travels with the artifact bundle. Cascade markers (`must populate fix.`, `Should NOT be empty`, `Expected value not to be nil`) collapse the typical "one upstream phase fails, ten downstream `require.NotEmpty` assertions trip" pattern into a single grouped entry under the real first failure. The analyser is advisory — it always exits 0. The existing "Fail job if any suite failed" gate keeps owning exit-code semantics. Stage 1 of `docs/development/improvements/e2e-go-failure-analysis.md` (mulga-siv-102). Stage 2 (per-failure log slicing) and Stage 3 (rule-based hints via `tests/e2e/.failure-rules.yaml`, placeholder seeded here) are independent follow-ups. CI wiring: - e2e-go.yml: new "Analyze e2e failures" step between "Publish test summary" and the artifact upload, always-runs. - spinifex.yml: new `make test-actions` step so the golden-file tests under `.github/actions/e2e-analyze/testdata/` run on every PR. - Makefile: `test-actions` target; kept out of `test-cover` so CI-only tooling doesn't dilute coverage %. Golden fixtures: a synthetic cascade-vpcs case (one Phase 5 InvalidParameterValue root + 15 downstream cascades), one real single-failure case captured from run 26037892507, and a clean no-failures case. * ci(e2e-go): per-failure log bundles with time-window slicing Stage 2 of the e2e failure analyzer. For each failure surfaced by Stage 1, materialise <log-dir>/analysis/NN-<TestName>/ containing: failure.txt — summary (test, start, duration, file-site, error) journal.log — spinifex-journal.log sliced to [StartAt - 5s, StartAt + Duration + 5s] test.log — test-<suite>.log block between `=== RUN <name>` and the matching `--- FAIL` marker <copies> — any files the harness already dropped at <log-dir>/<TestName_with_/_replaced>/ Report links each entry to its bundle so a reader on the Actions page goes straight to the relevant slice. Workflow now invokes journalctl with --output=short-iso so the analyzer can parse wall-clock timestamps for the journal window. Lines without a parseable timestamp inherit the previous line's classification (keeps multi-line records and stack traces contiguous). Render output is unchanged when BundlePath is empty, so the Stage 1 golden snapshots still match. * ci(e2e-go): restore GOTOOLCHAIN=auto after setup-go v6 bump setup-go@v6 flipped the GOTOOLCHAIN default from "auto" to "local", which makes Go refuse to auto-download the toolchain version named in go.mod. The self-hosted ci-single runner ships /usr/local/go older than spinifex/go.mod's `go 1.26.3`, and build-install-artifacts.sh prepends that path so the system Go wins the resolution race. Under v5 ("auto"), Go silently fetched 1.26.3 on demand; under v6 ("local"), the Build Install Artifacts step errors out: go: spinifex/go.mod requires go >= 1.26.3 (running go 1.26.1; GOTOOLCHAIN=local) Re-set the env to "auto" so behaviour matches pre-v6 runs. * ci(e2e-go): re-export GOTOOLCHAIN=auto after setup-go Workflow-level env doesn't help: setup-go@v6 writes GOTOOLCHAIN=local to $GITHUB_ENV, which overrides workflow env in every later step. Add a dedicated step right after setup-go that appends GOTOOLCHAIN=auto to $GITHUB_ENV so the Build Install Artifacts step (which uses /usr/local/go from a stale runner path) can self-heal by fetching the toolchain named in go.mod, matching pre-v6 behaviour. * fix(vpcd): add flows-ready barrier after dnat_and_snat add handleAddNAT wrote the NAT rule to OVN NB and returned immediately, mirroring the gap siv-105 closed for handleIGWAttach but on the NAT add path. ec2d/RunInstances then returned success while ovn-northd was still compiling NB->SB and ovn-controller hadn't installed the DNAT/SNAT flows on the gateway chassis, so the freshly launched VM was unreachable on its public IP for whatever flow-install latency the chassis happened to incur. Reproduced in CI run 26072432957 Phase8b_VPCSubnetE2E: VM at 192.168.0.212 timed out at 3min while diagnostics showed Port_Binding up=true for both the VM eni and the chassis-redirect gateway. SB-side convergence was complete; the missing piece was the same `ovn-nbctl --wait=hv sync` barrier siv-105 added to IGW attach, applied at the end of handleAddNAT as well. Reuses the existing package-var `waitForFlowsHV` so tests can stub it; no new infra. Parallel unit test asserts the call site fires exactly once per add-nat. * ci(e2e-analyze): real-artifact fixes for journal slicing Three bugs surfaced after Stage 2 ran against real CI artifacts in run 26072432957: 1. Regex `[+-]\d{4}` rejected the colon-offset form (e.g. `+10:00`) that systemd's `journalctl --output=short-iso` actually emits on most versions. Added a parser that tries RFC3339 first and falls back to the no-colon `-0700` layout. Result: 0-line journal slices become populated. 2. go-junit-report's `timestamp` attr on <testsuite> records the moment the XML was produced (post-suite), not when the tests started. Computing StartAt as `that + cumulative-priors` placed every failure AFTER the journal collection ended, so windows always missed. Workflow now writes `${ARTIFACT_DIR}/test-<suite>.start` right before running each suite; analyzer prefers that file over the junit timestamp. 3. Persisted `OffsetFromSuiteStart` on each Failure so the suite-start override can recompute StartAt without re-parsing the XML. Unit tests added for the colon-offset journal format. End-to-end test (TestWriteBundles_EndToEnd) still passes with the new layout. * ci(e2e-analyze): skip parent testcases when summing suite offsets go-junit-report emits one <testcase> for the parent Go test AND each subtest, with the parent's time = sum of its children. ParseFile was summing every testcase into the suite-cumulative-offset counter, so every leaf's offset was inflated by ~the parent's total duration. Visible on run 26073817490 as Phase8b's StartAt landing 13 minutes after the suite finished collecting logs — beyond the journal window, so the per-failure journal slice was empty. Pre-compute the leaf set per suite (a name is a leaf iff no other testcase name starts with `<name>/`) and only accumulate cumul on leaves. Golden snapshots updated; the offset for the only failure in each fixture now matches the actual leaf-sum start time. * test(e2e-harness): capture IP-targeted datapath state on Phase8b SSH fail Phase 1 diagnostics for mulga-siv-111. The Phase8b SSH timeout reproduced on the e2e-failure-analysis branch with siv-105's IGW barrier AND the new add-nat barrier BOTH firing cleanly, so the residual gap is downstream of `--wait=hv sync` — most likely stale OVS conntrack, stale upstream-host ARP, or an OF flow-install miss. DumpVPCFlowDiagnostics now takes a VPCDiagnosticsOpts carrying the external IP, logical IP, and per-test artifact dir. When set, the helper greps `ovs-ofctl dump-flows br-int`, `conntrack -L`, and `ip -4 neigh show` for the external IP / logical IP and snapshots `ovn-nbctl find NAT external_ip=…`. Each capture is both printed inline (existing affordance) and written to a separate file under fix.Artifacts so the Stage 2 analyzer bundler picks it up alongside the journal + test-log slices, no analyzer changes needed. Phase8b's caller now passes pubIP / privIP / fix.Artifacts so the captures fire on the next failure; updates the t.Fatalf message to point at siv-111 instead of siv-105 (which is the now-closed parent). * fix(e2e-single): authorize SSH ingress on fresh-VPC default SG in Phase8b Phase 8b stands up a non-default VPC and runs an instance there. Its auto-created default SG denies all ingress, so the SSH probe times out even when every datapath barrier converges — OF flow captures show the SYN reaches OVN, gets DNAT'd at table 13, then drops at the ACL between tables 13 and 17. The bash driver relies on admin init having pre-baked this rule on the *default* VPC's default SG; a fresh VPC has not. Add a DescribeSecurityGroups + AuthorizeSSHIngress step between route-table setup and run-instances, using the existing harness helper. Fix surfaced via mulga-siv-111's Phase 1 diagnostics (OF flow + ACL drop counts), invalidating the H1/H2/H3 stale-state hypotheses in the bead plan doc. * test(e2e): dual-detect ubuntu image version (24.04/26.04) during gold-image transition After mulga gold image bumped to v6-fastboot (stages ubuntu-26.04 / resolute instead of ubuntu-24.04 / noble), all consumers that hardcoded 24.04 break on the new image. Make them detect at runtime so v3 and v6 envs both pass. - tests/e2e/run-e2e.sh: DescribeImages filter accepts both ami-ubuntu-{26,24}.04-<arch>, then derives IMAGE_NAME from the match. - tests/e2e/run-pseudo-multinode-e2e.sh: prefer ~/images/ubuntu-26.04.img, fall back to ubuntu-24.04.img; pass detected version to spx import. - tests/e2e/single/image_test.go: walk candidate catalog names (26.04 first, then 24.04), stop on the first one DescribeImages can resolve. - .github/workflows/e2e-baremetal.yml: download resolute (26.04) into BM_ISO_CACHE; update BM_UBUNTU_IMAGE env var path. - docs/terraform-workbooks/{bastion-private-subnet,s3-webapp,nginx-webserver}: AMI filter matches `*ubuntu-26.04*` OR `*ubuntu-24.04*` (AWS values= array is ORed). Companion changes for mulga (bootstrap scripts, baremetal, iso pipeline) land in scripts/tofu-cluster/* and scripts/{tofu-iso,baremetal,iso-builder}/* under bead mulga-siv-114. Drop the 24.04 fallbacks once all envs are on the v6 gold image and 24.04 entries are removed from spinifex/utils/images.go. * feat(e2e): add service-interfaces.yaml manifest + validator Source of truth for which spinifex services produce/consume which NATS subjects, and which e2e suites cover which services. Consumed downstream by the targeted suite selector (Bead 4) to derive subset E2E runs from a PR diff so single-service changes stop paying the full 25 min suite cost. - docs/service-interfaces.yaml: 7 services, 4 active suites (cert, lb, single, ddil). Subscribes / publishes / depends_on filled from grep of NATS subjects in source. spinifexui marked covers_no_suite so UI-only diffs skip e2e entirely. Future suites (multinode, reboot, tofu, baremetal) commented in until Go ports land. - tests/e2e/manifest-check: schema + cross-reference validator (path existence, subject regex, duplicate detection, suite covers resolves to a real service, fixtures references resolve). 8 unit tests cover the failure modes plus a load-and-validate on the real manifest. - Makefile: manifest-check target, hooked into preflight. - docs/DESIGN.md: link to the manifest under "NATS Messaging". mulga-siv-116. See docs/development/improvements/e2e-targeted-suite-selection.md. * fix(e2e): drop catalog-download import path from phase4_Image `spx admin images import --name ubuntu-26.04-x86_64` with wantErr=true was meant to detect that bootstrap-install.sh already imported the AMI and short-circuit on the duplicate exit code. After the v6 gold-image bump preloaded `~/images/ubuntu-26.04.img` and bootstrap-install.sh switched to `--file` for the import, two things broke at once: - The catalog `--name` path no longer hits a duplicate guard for the --file-imported AMI, so the import re-downloads ~857MB and either succeeds (failing the wantErr=true assertion) or wedges the job timeout while the catalog server crawls along at 16KB/s. - `--file` from the test won't work either: the test binary runs as `tf-user` over SSH, while bootstrap-install.sh placed the image at `/home/spinifex/images/...` (mode 0700 home dir). bootstrap-install.sh is already responsible for staging the AMI. The test's job is verifying DescribeImages round-trips by-name and by-ID — not re-exercising import. Trust the bootstrap-staged AMI and stop calling import. Keeps the 26.04→24.04 dual-detect for the v3 gold image transition; drop the 24.04 candidate when no envs remain on v3. Refs mulga-siv-115 (caught by the EC2 cold-boot fast-path E2E run on feat/ec2-coldboot-fastpath). * ci(e2e-go): default to e2e-single + harden status-report SHA fallback Two changes to .github/workflows/e2e-go.yml: 1. Default `suites` input now includes e2e-single so a no-input manual dispatch exercises the full single-node ladder. Single-node Go harness is greenable now (image_test fix at 10eb38a). 2. Report E2E (Go) Status job no longer gated on `inputs.sha` and falls back to `github.sha` when the input is unset. Previously a manual dispatch without a PR head SHA either skipped the status post entirely or hit `gh api repos/.../statuses/` with an empty path component (the 'invalid hash' error) — tests passed, the report step failed silently. Now status posts on every dispatch against the dispatched ref's commit when no explicit SHA given. Closes mulga-siv-112. * feat(e2e): add Ensure* fixture API with memoization + cleanup Introduces tests/e2e/harness/fixtures.go: the memoized, cleanup-aware provisioning helpers that Bead 3 (single-fixture-migration) will rewrite phase tests around. Each Ensure* helper: - takes (t *testing.T, fx *Fixture, args...) - memoizes by a key derived from args (sync.Mutex + singleflight) - registers t.Cleanup on the Fixture's parent test so memoized resources survive child-subtest lifetimes - bakes readiness polling into the helper (no sleeps in callers) Exported helpers: EnsureKeyPair, EnsureAMI, EnsureDefaultVPC, EnsureSubnet, EnsureSG, EnsureInstance, EnsureVolume, EnsureSnapshot, EnsureNATGateway, EnsureLoadBalancer. Fixture stores ec2iface.EC2API + elbv2iface.ELBV2API so unit tests can inject fakes without building a real AWSClient (TLS + env). Compile-time checks guard that the SDK concrete types satisfy the stored interfaces. Manifest fixtures: block populated — Bead 4 selector will resolve test coverage from AST-scanned Ensure* call sites via this map. Makefile: new test-harness target runs the harness unit tests under -tags=e2e (mocked, no infra). Hooked into preflight so signature drift and memo/cleanup regressions surface locally. Bead 2 of docs/development/improvements/e2e-targeted-suite-selection.md. No existing tests modified — fixture API is net-new. * feat(e2e): port run-reboot-e2e.sh to Go Adds tests/e2e/reboot/reboot_test.go — Go port of the single-node host-reboot resilience driver (cell #18). Runs from the GitHub Actions runner (not shipped to the VM like cert/lb/single) because systemctl reboot kills any in-VM process mid-step. All AWS API traffic crosses the WAN to https://WAN_IP:9999 via the SDK; SSH-level ops (reboot, ovn-nbctl, journalctl, HTTP burst) hop via harness.PeerSSH. Trust path tries to SCP the cluster CA off the VM and falls back to InsecureSkipVerify with a log message if SCP fails — the cert suite already validates real trust, so the fallback is intentional. Harness changes: - env: WANHost field sourced from SPINIFEX_WAN_IP, distinct from ServiceIPs[0] (which is the awsgw bind IP, often loopback in-VM). - aws: SPINIFEX_AWS_INSECURE=1 escape hatch when the runner has no CA on disk. Used as the fallback when CA fetch fails. - cert: SPINIFEX_CA_CERT explicit path override so the reboot test can point ResolveCACert at the SCP'd tmp file. - peer_ssh: IsReachable() probe wrapper for cleanup paths that must skip rather than fatal when the host is mid-reboot. Workflow (.github/workflows/e2e-go.yml): - New e2e_go_reboot job, opt-in via `e2e-reboot` in the SUITES dispatch input. needs: e2e_go_single to serialise on the ci-single self-hosted runner. Fetches AWS creds from the VM's ~/.aws/credentials over SSH (mirrors the cred hop already proven by e2e-ddil.yml). - e2e_go_single filters 'reboot' out of its build + run loops so the in-VM driver doesn't try to ship reboot.test to the VM and reboot it from inside. - e2e_go_status treats skipped reboot as success. Refs mulga-siv-118. Plan: docs/development/improvements/e2e-go-reboot-port.md (in the mulga parent repo). * ci(e2e-go): run reboot job in parallel with single Drop `needs: e2e_go_single` from e2e_go_reboot. Both jobs target the same [self-hosted, ci-single] label, so GitHub Actions serialises on runner allocation rather than blocking via job ordering — parallel-by- spec, queued-by-runner. Lets a reboot-only dispatch start immediately instead of waiting for an unrelated single-job pre-condition. * fix(e2e/reboot): wait for both backends to serve before post-reboot burst Phase 8.6 previously probed until ANY HTTP response from the ALB, then fired a 20-req burst. Post-reboot, only one of the two backends may be serving (the second is still bringing its responder online) while the daemon's TG cache still shows pre-reboot 'healthy' for both — so WaitForTargetsHealthy returns in ms instead of seconds, the ANY probe matches the first-up backend, and the burst lands 20/0 instead of 10/10. Probe until BOTH backends' instance_ids each appear at least once before the round-robin burst — confirms recovery has reached both, not just one. Underlying daemon-side TG cache staleness tracked separately as mulga-siv-119. This wait is the test-side guard; the daemon fix will make WaitForTargetsHealthy meaningful again post-reboot. Refs mulga-siv-118. * ci(e2e-go): skip single job when dispatch requests only reboot Add if: gate to e2e_go_single that skips the job when the workflow_dispatch suites input contains none of {e2e-single, e2e-cert, e2e-lb}. Push events and the default suites list still trigger it, so plain PR runs are unaffected. Avoids ~15min of wasted tofu provision+bootstrap+teardown when only the runner-resident reboot suite is requested. * fix(e2e/reboot): bump post-reboot LB recovery timeout to 5min Pre-reboot the test creates the LB AFTER apps are already running, so HAProxy starts probing against an already-serving responder — ~75s to healthy. Post-reboot the daemon's recovery scan launches app VMs and the ALB sys.micro VM simultaneously, so health checks start probing within seconds of app boot beginning and both targets honestly report unhealthy until the responder unit finishes starting. Cloud-init's per-instance semaphore means no shortcut path on relaunch, so guest second-boot can be slower than first boot. Bash run-reboot-e2e.sh hid this behind stale 'healthy' state read from the daemon's pre-reboot TG cache — fixed via ResetTargetHealthOnStartup (mulga-siv-119), which made the test honestly wait on real lb-agent reports. The 180s bash inherited turned out to be wishful thinking backed by the cache bug, not measured recovery time. 5min matches the actual post-reboot eventually-consistent window observed on run 26134156355. Refs mulga-siv-118. * feat(e2e/reboot): ssh into guest VMs on failure for HTTP responder diag Post-reboot failures bottom out at "0/2 healthy" with no insight into whether the responder unit died, port 80 isn't bound, or the OVN path is broken. Capture the keypair material from CreateKeyPair so OnFailure can hop host -> qemu hostfwd:22 -> guest and dump systemctl status, port listeners, curl localhost, and the responder journal per instance. * fix(e2e/reboot): probe app instances via public IP (no QEMU hostfwd) Previous hostfwd-SSH approach didn't apply: app VMs use pure OVN tap interfaces (qemu argv has no hostfwd=tcp:*:*-:22), so the regex never matched. Replace with host-side probes against the IGW-NAT'd public IP: ping, /dev/tcp connect, and curl. Combined with the qemu ps check and ovn-nbctl list nat dump, this localises post-reboot failures to one of: VM not running, OVN DNAT missing, responder down, or sys.micro-only reachability (path asymmetry). * feat(e2e/reboot): probe ALB + dump host OVN/OVS state on failure Last run showed apps unreachable from host post-reboot (ping 100% loss, tcp refused) even though OVN dnat_snat rules looked correct. Need to distinguish gateway/ext-bridge breakage from app-side breakage: add a parallel ALB-public-IP probe (works pre-reboot, so any post-reboot failure isolates the regression) plus a full host net-state dump (ip route, ip neigh, ovn-nbctl show, ovs-vsctl show, br-int port list). * feat(e2e/reboot): dump qemu serial console per app on failure /run/spinifex/console-i-<id>.log captures cloud-init, kernel boot, systemd, anything the guest emitted to ttyS0. Tail to 256KiB (~4x a typical Linux boot) so the latest boot lands in full plus the tail of the prior. Surfaces guest-side boot failures (kernel panic, fs corruption, cloud-init OOM, etc.) that are invisible to host-side network probes. * feat(e2e/reboot): SSH into guest VMs via IGW for responder root-cause Last run nailed the symptom: ALB returns 503, app public IPs answer TCP with RST (port 80 not bound), so reboot-e2e-httpd.service isn't running on relaunched VMs. To find out why (unit never installed? installed but not enabled? failed to start?), open tcp/22 ingress on the shared SG and add a guest-SSH diagnostic via the IGW NAT path. The diagnostic dumps the unit state, port listeners, /var/lib/httpd, cloud-init result.json, cloud-init logs, and the instance-id semaphore directory. * test(e2e/reboot): always run guest SSH diag in phase 8 Last run passed after adding tcp/22 SG ingress. Two possibilities to distinguish: (a) responder was running all along but blocked at the OVN/SG layer, (b) responder wasn't running and HC was satisfied by some other path. Capture in-guest unit + cloud-init state on every run so we can confirm which without needing the test to fail. * refactor(e2e/single): migrate VPC group to EnsureDefaultVPC [skip ci] Bead 3a of mulga-siv-120 (e2e-single-fixture-migration). Replaces the "discover default VPC + cache on Fixture" pattern in vpc_test.go, natgw_test.go, securitygroup_test.go with harness.EnsureDefaultVPC. - Adds Harness *harness.Fixture field on single.Fixture; constructed in TestSingleNode alongside AWSClient. Parent test for memoized Ensure* resources is TestSingleNode itself. - Removes fix.DefaultVPCID / fix.DefaultSGID / fix.DefaultSubnetID reads in the three target files. Fields stay on the struct until 3e+3h migrate the instance / datapath / umbrella sites. - Custom VPC graphs created by phase8b (10.99.0.0/16) and the bastion graph in phase8d remain inline — those are the resource lifecycle assertions, not prereqs. CI gate skipped per dev request — full single-node suite verifies when upstream main is unbroken. * refactor(e2e/single): migrate Phase 3 key pair to EnsureKeyPair [skip ci] Bead 3c of mulga-siv-120 (e2e-single-fixture-migration). Phase 3 now calls harness.EnsureKeyPair for the primary key pair (Phase 5+ prereq). The Ensure* path owns the create assertion (PEM shape), memoizes across phases, and registers TestSingleNode-scoped cleanup — replacing the former TODO(stage-G) deferred-cleanup marker. - Drops hardcoded "test-key-1" name; EnsureKeyPair uses scratch-suffixed name so parallel CI runs don't collide on the namespace-flat AWS key pair name. - fix.KeyName / fix.KeyPath still assigned for downstream Phase 5+ readers. Per 3e migration plan, those reads move to in-test EnsureKeyPair calls and the Fixture fields drop with 3h. - Import / describe / delete dance for test-key-2 stays inline — it is the lifecycle assertion for the import path, not a prereq. * refactor(e2e/single): migrate Phase 4 + 5e AMI paths to EnsureAMI [skip ci] Bead 3d of mulga-siv-120 (e2e-single-fixture-migration). - Phase 4 (image_test.go): discover catalog AMI by canonical name, then route through harness.EnsureAMI({Existing}) so the state-available wait + memoization live with the fixture. - Phase 5e (createimage_test.go): replace inline CreateImage block with harness.EnsureAMI({CreateFrom}). The custom AMI build now memoizes, registers DeregisterImage cleanup at TestSingleNode exit, and reuses the harness state poll. Harness fix: extend AMICreateSpec with NoReboot bool — Phase 5e bakes the AMI while leaving the source instance running for Phase 6/7 lifecycle assertions; Bead 2 missed this option. fix.AMIID / fix.CustomAMIID / fix.CustomAMISnapID stay on Fixture for downstream Phase 5/6/7 + Phase 9 cleanup. Those reads migrate with 3e/3h. * refactor(e2e/single): migrate Phase 5 instance lifecycle to EnsureInstance [skip ci] Bead 3e of e2e-single-fixture-migration. - phase5_LaunchInstance: provision via EnsureInstance + re-describe for BDM/root-volume extraction; default VPC via EnsureDefaultVPC. - phase8e_SGToSGDatapath: replace inline DiscoverDefaultVPC with EnsureDefaultVPC. createSG / runSGEInstance take vpcID/subnetID arguments so phase 8e no longer depends on fix.Default* fields. - runSGEInstance stays direct RunInstances: phase 8e VMs are test subjects (need fresh ENI/private-IP per run), not memoized prereqs. - harness.EnsureInstance memo key now includes InstanceType, SGID, and UserData hash — a second call with a different SG would have silently returned the first instance ID (Bead 2 bug). * refactor(e2e/single): migrate Phase 5b/5c volume + snapshot to Ensure* [skip ci] Bead 3f of e2e-single-fixture-migration. - phase5b_VolumeLifecycle: create+wait via EnsureVolume; subsequent modify/attach/detach/delete stay inline (test subject lifecycle). - phase5c_SnapshotLifecycle: create+wait via EnsureSnapshot. Drop the create-response State/Progress shape assertions — after EnsureSnapshot the snapshot is already "completed", and the describe block below re-asserts VolumeId / VolumeSize / Description. - harness.EnsureSnapshot now takes SnapshotSpec{VolumeID, Description} so callers can pin a description (snapshot_test asserts on it). Memo key includes description hash so distinct-description calls do not silently collide on the volumeID prefix. * refactor(e2e/single): isolate IAM stage E via parent-scoped cleanup [skip ci] Bead 3g of e2e-single-fixture-migration. - harness.Fixture: add RegisterCleanup helper that wraps fx.parent.Cleanup so test code can register parent-scoped teardown for resources created outside ensureOnce (IAM users / policies threaded across phase subtests). Registering on the calling subtest would tear down before the next dependent phase ran. - phaseIAM1_UserCRUD: register parent-cleanup for alice/bob/charlie. - phaseIAM4_PolicyCRUD: register parent-cleanup for the 5 test policies. - IAM stage no longer relies solely on phaseIAM7 to clean up after a mid-stage failure; phase 7 still runs and now serves as an idempotent post-cleanup assertion. No new Ensure* helper: IAM resources have no memoization story (users and policies are test subjects, secrets only returned once on create). IAM phases also have no EC2 prereqs, so EnsureDefaultVPC / EnsureInstance are not applicable here. * refactor(e2e/single): drop dead Fixture fields after Ensure* migration [skip ci] First half of bead 3h. Removes per-phase Fixture fields with zero remaining readers after bead 3a-3g: - DefaultVPCID / DefaultSGID / DefaultSubnetID: superseded by harness.EnsureDefaultVPC (3a wired phase8b/8c/8d/5f; 3e wired phase5 + phase8e). - OVNAvailable: never assigned anywhere — leftover from the bash port. Phase fields still in use (AMIID, InstanceID, IAM*, etc.) stay until the umbrella TestSingleNode is replaced with top-level Test* + package- level fixture singleton in the second half of 3h. * refactor(e2e): add process-mode Fixture + TestMain singleton infra [skip ci] Second half of bead 3h. Lands the infrastructure required to dismantle the TestSingleNode umbrella in a follow-up bead, without changing the umbrella body today. harness.Fixture changes: - NewProcessFixture(aws) builds a Fixture with no *testing.T parent. Cleanup callbacks queue on an internal LIFO drained by Close() — intended to be invoked from TestMain after m.Run(). Lets a singleton fixture span every top-level Test* in a package. - Existing NewFixture(t, aws) unchanged; cleanups still route via t.Cleanup. registerCleanup() and logf() helpers fork on parent==nil so ensureOnce and RegisterCleanup work in either mode. - TestProcessFixture_CloseRunsCleanupsLIFO covers Close ordering, one-shot semantics, and the no-op second-Close path. single/main_test.go additions: - TestMain owns process-level lifecycle: drains pkgFix.Harness.Close() and removes pkgFix.TmpDir after m.Run(). - requireSingleNodeFixture(t) returns the package-scoped singleton, building it lazily on first call. Skips when SPINIFEX_E2E is unset or mode != single; fails on construction error. Currently unreferenced — TestSingleNode keeps its own per-test Fixture for now. The helper exists so the follow-up bead (per-phase top-level Test* migration) has a stable API to call from day one. Deferred to follow-up bead: extract Phase 4 AMI-name discovery + Phase 2 InstanceType/AZ selection into harness helpers so each phase function can self-bootstrap, then promote phase subtests to top-level Test* and delete the TestSingleNode umbrella. * fix(e2e/harness): EnsureDefaultVPC filter name is-default not isDefault Daemon rejects the camelCase `isDefault` form with 400 InvalidParameterValue. AWS CLI / DiscoverDefaultVPC use `is-default`. Surfaced as Phase 5_LaunchInstance fatal in CI run 26140908505 — cascade-failed every downstream phase that needed fix.InstanceID. * fix(e2e/single): phase8 8j/8k probe fix.KeyName not hardcoded test-key-1 Bead 3c migrated phase 3 to harness.EnsureKeyPair which produces a scratch-suffixed name (e2e-key-<random>), not the literal "test-key-1" the bash port used. 8j_CreateKeyPairDuplicate / 8k_ImportKeyPairDuplicate kept the old name and so created a fresh key pair instead of triggering InvalidKeyPair.Duplicate — surfaced as the only two FAILs in CI run 26143227369 (everything else green after the EnsureDefaultVPC fix). Read fix.KeyName (set by phase3_KeyPairs) so the probe hits the staged primary key. Header comments updated to match. make preflight diff-coverage against origin/dev fails at 57.9% due to unrelated commits inherited from the recent rebase onto feat/e2e-go-harness; against the fork base the diff is 0 instrumentable lines (test-only change). * refactor(e2e/single): dismantle TestSingleNode umbrella, per-phase self-bootstrap mulga-siv-121. Each phase from run-e2e.sh now runs as a top-level Test* in tests/e2e/single, runnable in isolation via `go test -run TestX`. Highlights: * TestSingleNode umbrella removed. Phase ordering dropped — each Test* self-bootstraps prerequisites through harness.Ensure*/harness.Discover* + the new package-local need*/iamEnsure* helpers. * Fixture shrunk to {Env, AWS, Harness, Artifacts, TmpDir, PoolMode}. Every per-phase resource ID now lives on harness.Fixture via EnsureKeyPair/EnsureAMI/EnsureDefaultVPC/EnsureInstance/etc. * TestMain owns the package singleton Fixture; harness.Fixture.Close drains every cleanup at process exit. phase9 + phase9a (in-line teardown) deleted; phase9b retained as TestFinalClusterStats. * IAM Test* share state via sync.Once-gated helpers in iam_helpers_test.go (alice/bob/charlie users + keys + admin account). * New harness/discover.go: DiscoverDefaultAZ, DiscoverNanoInstanceType, DiscoverUbuntuAMI — memoized on the Fixture and routed through EnsureAMI so any Test* can resolve them without phase coupling. * New tests_test.go: descriptive top-level Test* wrappers (TestVPCSubnetE2E, TestKeyPairs, TestSnapshotLifecycle, …) that delegate to phaseN_X(t, requireSingleNodeFixture(t)). * test(e2e/reboot): SSH as ec2-user, dump unit + network-online state spinifex's cloud-init template hardcodes Username="ec2-user" (handlers/ec2/instance/service_impl.go:1218) regardless of AMI, so even on Ubuntu the SSH-injectable account is ec2-user, not ubuntu. Switch the diag SSH-in to ec2-user so we actually get in (was "Permission denied (publickey)"). Add diagnostics to distinguish the two leading post-reboot hypotheses: (a) viperblock write loss = unit file gone vs (b) network-online.target never fires = unit stuck on After=network-online.target. New dumps: unit file content + enable symlink, network-online.target status, networkctl, ip addr / route, prior-boot journal grep for the responder. * fix(e2e/single): resolve admin account ID before TestIAM7 deletes alice iamEnsureAdminAccountID probes via GetUser(alice). TestIAM7_Cleanup deleted alice/bob/charlie first, then called the helper at line 808 — NoSuchEntity if IAM7 was the first IAM test to run (no memoized account ID). Move the call above the deletion loop.	2026-05-20 21:04:13 +10:00