mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-20 11:21:49 +00:00
Ben Duncan
42d87d7a5a
* feat(ebs): make nbdkit thread count and per-volume read cache configurable Both numbers that set a volume's export concurrency and memory were hard-coded: nbdkit ran on its own default of 16 threads per connection because no -t was ever passed, and the read cache was a 128 MiB literal in the mount path. Neither could be changed without a recompile, and neither was visible in the logs or the process argv. Add default_threads and cache_size_mb to [nodes.<node>.ebs]. Defaults are unchanged, so a config that does not set them behaves exactly as before. CacheSizeMB is a pointer so an explicit 0 (cache disabled) stays distinguishable from an unset key, matching how ViperblockConfig treats its own toggles. -efi volumes stay uncached regardless. Validation is a range check only. Both are operator tunables applied per nbdkit process, and there is one of those per volume, so the host cost is the value times the volume count -- left for the operator to weigh rather than second-guessed here. * feat(security): arm the firewall by install path and sync the clock at firstboot The host firewall armed itself from a code default, so a curl-to-bash install onto an existing Debian or Ubuntu machine applied a default-deny INPUT policy seconds after bootstrap. Anything the operator was already running that is not on the allow list stopped accepting connections, with nothing logged and no config key to set beforehand — no template writes firewall_enabled, so every install fell through to the default. Arming is now the install path's decision. setup.sh takes --firewall=on|off and records it in /etc/spinifex/firewall/mode, defaulting on under ISO_BUILD and off everywhere else: the ISO is an appliance on hardware we define, a package install is a guest on a machine that was already doing something. firewall_enabled becomes a three-state *bool so "unset" and "explicitly false" stay distinguishable. Flipping the default to false instead would have torn down a working policy on every existing node, because false is an instruction to disable rather than an absence. Resolution is config, then mode file, then on, so a node installed before any of this keeps what it has. The ISO had the opposite problem. setup.sh is the only place that enables spinifex-firewall.service and it returns before that line under ISO_BUILD, with no systemd in the chroot to talk to. Nothing else enabled it, so after a reboot the ruleset — runtime state — was gone, and ReconcileFirewall short-circuited on an unchanged peer file and never put it back. An ISO node was firewalled until its first reboot and open permanently after. firstboot now enables the unit, and the reconcile checks the table is loaded rather than treating an unchanged peer set as proof the policy is applied. Two policy changes: 443 is held for the nginx TLS edge so that rollout needs no per-node firewall change, and the SSH port is read from sshd_config and its Include drop-ins instead of being hardcoded to 22 — a host on Port 2222 was allowed exactly one port that was not the one it uses, and ct state established kept the operator's session alive until they disconnected. 53 stays open on every node, now with a comment saying so: northstar serves public authoritative DNS. Clock: firstboot steps and waits for sync before spx admin init mints credentials, since a certificate issued against a wrong RTC carries a NotBefore that outlives the correction. Bounded to 60s and non-fatal so an air-gapped install still completes, and once is enough because rtcsync writes the corrected time back. A chrony drop-in replaces Debian's makestep 1 3, which stops stepping after three updates and then slews at roughly 7 seconds a day — a node that burned those before the network came up never visibly converged. * fix(security): let a node join a cluster whose firewall is already armed Expanding a single-node ISO install into a cluster could not succeed. Two independent blockers, both from the same root: the peer set is derived from converged cluster state, but the ports needed to reach convergence are gated on that same set. The formation port is peer-scoped, and a node dialling in to join is not a peer yet, so the handshake was dropped rather than refused and the joiner retried in silence until it timed out. It cannot simply move to the public rule: outside formation the same port carries the daemon cluster manager, whose /health and /local/* routes report node topology, service inventory and running instances unauthenticated. So it is opened for the formation window only, by spx admin init, and closed again on every exit path. The reconcile then deadlocked. Holding out for the full chassis list is right on a node with no policy loaded, but on one armed for a smaller cluster the ports a joining node needs in order to register its chassis are exactly the ones the stale set blocks, so the list can never complete. Widen the peer set on the tunnel set already loaded, and report the partial state so the caller retries sooner than the steady-state interval. A converged cluster whose Southbound DB is briefly unreadable is still left untouched. * docs(install): explain the firewall around cluster formation An ISO-installed server is a cluster of one, and its firewall admits internal cluster traffic only from members it recognises. Forming a cluster out of such servers therefore needs the firewall off first: step 3 of the multi-node guide has each server dial node 1's OVN database on the raft ports, and those are peer-scoped like everything else on the cluster plane. Neither guide said so, and the failure is a silent drop that looks like a network fault. Both guides now walk the off / form / on sequence, and the multi-node guide gains a reference section covering which ports are public, how arming differs between the ISO and the binary installer, and how to verify membership landed. Re-enable the boot-time unit whenever the daemon arms the policy. Disabling the firewall to form a cluster also disables that unit, so re-arming used to look complete while leaving the node bare after its next reboot. * fix: make the ISO-to-cluster conversion path work end to end Forming a cluster from four ISO-imaged servers surfaced a run of bugs that share a starting state CI never exercises: the ISO delivers an initialized, armed, running node, while the conversion path was written against the binary installer's unconfigured one. - install-node.sh never disarmed the host firewall, so nodes 2 and 3 could not dial the first node on the OVN raft ports and formation could not proceed. It now owns the whole lifecycle: disarm, form, re-arm, then verify the loaded policy against the expected peer set. - spx admin init left the admin access key and secret empty on the preserve path. The pair cannot be read back, and the IAM store is rebuilt by formation, so the empty strings reached every joiner's bootstrap.json and crash-looped awsgw. It is reissued instead, and IAM seeding now skips an empty admin block rather than taking the gateway down on every restart. - node-reset.sh left /etc/default/ovn-central behind. Raft membership lives in that file, not in the databases the reset deletes, so ovn-central recreated the database as a cluster member dialling a peer that had itself been wiped. - node-reset.sh also destroyed the installed firewall policy, which nothing regenerates. set-peers then failed on every attempt and the node stayed open permanently and silently. The arming mode is read before the wipe and the stage regenerated after it. - spinifex-firewall-apply rejected every verb when the policy file was missing, including disable. Turning the firewall off is the recovery path, so that left a node that could be neither armed nor disarmed. - viperblock exited 0 on a fatal startup error, so Restart=on-failure never fired. During formation its JetStream bucket has no leader yet, so the EBS provider died for good and every volume request failed with no responders. predastore and qemunbd had the same bug. - spx admin images import created image directories at 0700, so the first sudo import locked group members out of a tree deliberately made group-writable. Three of these hid behind checks that cannot succeed: ovn-sbctl show against a clustered database, and the OVN northbound and southbound waits in setup-ovn.sh. All three failed to stderr and let the run continue, so a check that could never pass was indistinguishable from one that was merely slow. The USB install guide claimed the ISO leaves Spinifex unconfigured and told the operator to run spx admin init. firstboot has always done that itself, so the guide now describes what the ISO actually produces and how to start using it. * fix: let spread placement launch across every node A spread placement group reserved four distinct nodes correctly and then failed the launch with ServerInternal, having started only two or three of them. The scheduler was never at fault; two faults downstream stopped the instances following the reservation. vpcd subscribes to vpc.add-nat with QueueSubscribe, which delivers one message at a time, and the handler holds its reply across the OVN flows barrier. The 45s caller deadline was sized against a single 30s barrier, which is right for one instance and wrong for the N a spread launch issues at once — the third request waits behind two full barriers and times out. That one subscription now overlaps its handlers under a bounded semaphore. Each barrier still syncs after its own write, so the guarantee is unchanged while N barriers collapse into roughly one. With the requests overlapping, every barrier still ran to its alarm. install-node.sh computed OVN_REMOTE after building the database nodes and gave it only to compute nodes, so the database nodes were configured without --ovn-remote. setup-ovn.sh writes ovn-northd's NB/SB connections from that list and skips it on the localhost default, and it already documents what follows: a northd dialling only its local member stops advancing nb_cfg once raft leadership moves, wedging ovn-nbctl --wait=hv. The list is knowable before any node is touched, so it is computed first and passed to the database nodes too. Nothing reported an OVN fault while this was happening. Every chassis was registered, ovn-controller and IPsec were up, and the raft cluster was healthy. The only signal was the barrier logging that it had overrun and was continuing without confirmation — which reads like slowness and was a permanent wedge. * feat: sweep block sizes in the disk benchmark and add an iperf3 network benchmark * fix: use iperf 2 with CSV output and report why a remote start failed * fix: stop pkill matching its own shell and verify the iperf server is listening * ci: run the predastore throughput benchmark at the end of the nightly * ci: boot the ISO in the nightly, single-node and three-node * ci: use the existing CI node password secret in the ISO cells * ci: report free space before the benchmark instead of failing inside warp * ci: keep the benchmarks and node disks off tmpfs * ci: export the work directories from RUNNER_TEMP rather than job env * ci: run the ISO and benchmark workloads from the nvme scratch area * perf: run guest NICs on vhost-net with multiqueue Guest-to-guest VPC throughput measured 1.68 Gbit/s single-stream against 21.8 Gbit/s host-to-host on the same underlay, and did not improve with more streams. QEMU's -netdev tap defaults vhost to off, so the whole datapath ran in the main loop thread in userspace: that thread sat at 59-65% CPU while all four vCPU threads idled at ~4%. Every tap netdev now carries vhost=on, and VPC ENIs additionally run multiqueue at min(vCPUs, 8) queue pairs. Multiqueue needs all three of the tap, the netdev and the device to agree, so TapNetDev and NetDevice derive the netdev options and the 2N+2 MSI-X vector count from one queue figure rather than leaving callers to keep them in step. The QMP hot-plug path takes the same options as cold boot, with a test comparing it against the rendered cold-boot strings so the two cannot drift. MMIO machines stay single-queue: they have no MSI-X. * ci: check out the sibling repos the go workspace needs in the ISO cells * ci: skip cell-21 workload cleanup when the workload never ran * ci: size the cell-21 node at 8 GiB and keep its serial log * ci: keep the cell-21 node diagnostics alongside the serial log * fix: let the daemon open /dev/vhost-net and delete multi_queue taps Enabling vhost=on broke instance launch two ways. DevicePolicy=closed gates the cgroup device list before capabilities are consulted, so QEMU's open of /dev/vhost-net returned EPERM despite CAP_DAC_OVERRIDE; add the device to the allow list and bump the unit version. Separately, `ip tuntap del` re-opens the device with TUNSETIFF and fails with EINVAL on a multi_queue tap, leaking it on every teardown; route deletion through `ip link del`, which is flag-agnostic. * perf: default guest NICs to a single queue Multiqueue measured a net loss. Spreading one flow across queue pairs reorders packets and TCP reads that as loss: a 4-vCPU guest fell from 1.76 to 1.50 Gbit/s, with cwnd collapsing on spurious retransmits. The host overlay funnels a node pair onto one softirq core, so the extra queues have no headroom to win back what the reordering costs. Keep the plumbing and its tests behind MultiqueueNICs so re-enabling is one line once that funnel is fixed. vhost-net stays on unconditionally; it moves the datapath off the QEMU main loop, measured at 60% of a core under load against 3% with it. * fix: raise the UDP receive buffer for predastore's QUIC transport Predastore carries blob, meta and raft over QUIC and quic-go asks for a 7MiB receive buffer. Left at the kernel default of 208KiB it silently dropped datagrams under load: the four-node cluster had shed 80k-125k each, which failed shard writes, put raft into repeated elections, and ultimately handed guests an I/O error that cost two of them their root filesystems. Adds a sysctl stage writing /etc/sysctl.d/99-spinifex-net.conf, so the setting lands in the ISO image and on every node push rather than living in a runbook. * ci: give the cell-21 workload an identity the node accepts * feat: tie the guest MTU and NIC multiqueue to ipsec_enabled ESP costs 34 bytes on top of geneve's 58, so a cluster running the overlay in plaintext can advertise 1442 instead of 1408 — worth roughly 10% of guest throughput. Derive it from network.ipsec_enabled rather than hardcoding the conservative figure. Multiqueue moves with the same flag, because its sign depends on it. Behind IPsec it costs ~12%: ESP has no L4 ports, so RSS funnels a node pair onto one core and extra queues only reorder packets, which TCP reads as loss. Without IPsec the sending host's single vhost thread saturates first and queues are worth +54%. IPsec on with multiqueue on is the worst of the four combinations, so it stops being a standalone knob and becomes vm.Deps.MultiqueueNICs. The reconciler only ever created a subnet's DHCPOptions row and never revisited it, so the create-time MTU stuck for the life of the subnet and this change would have applied to new subnets only. Worse in the other direction: turning IPsec back on would have left subnets advertising 1442 over ESP, which drops large segments with no ICMP to recover from. applySubnets now diffs and updates in place. reconcile.Config takes IPSecDisabled rather than IPSecEnabled so the zero value is the conservative MTU — a caller that forgets it is merely slower. * docs: add an install-time performance tuning guide Measured on the four-node cluster rather than inferred: the UDP receive buffer sysctl, what IPsec actually costs and why, the multiqueue sign flip, the overlay MTU budget, and the jumbo-frame check that must precede enabling it. A starter doc — several sections end in an open question rather than a recommendation. * ci: keep the cell-21 node alive for the workload step * ci: collect the cell-21 qemu log and widen its timeout * feat: derive the guest MTU from the fabric, and deepen the virtio NIC config The overlay overhead is passed on to the guest rather than hidden from it. Hiding it means running the fabric far enough above 1500 that a standard guest MTU fits in the headroom, which is what the large public clouds do — but they own the switches. On a fabric we do not control, the honest figure is the one the path can carry, and a jumbo underlay should reach the guest: 9000 gives 8942, or 8908 with IPsec. network.underlay_mtu is the lever; SubnetMTU spends it. Configured rather than probed, because a node's own NIC MTU says nothing about the switch between it and its peers. host_mtu sets VIRTIO_NET_F_MTU so a guest that never takes a DHCP lease — statically addressed, or IPv6-only — still learns the ceiling. rx/tx_queue_size raise the virtio ring from QEMU's default 256. That default is a packet-rate ceiling rather than a memory saving: a vCPU preempted for a fraction of a millisecond fills all 256 slots and the overflow is dropped. ethtool is the only way to read NIC offload, ring and channel state, so a node without it cannot be diagnosed for throughput at all. * ci: give the cell-21 workload its own guest key * fix: set the guest MTU on the tap, not just in the guest The tap was created and brought up without an MTU, so it stayed at the kernel default of 1500 while DHCP and host_mtu told the guest something else. Harmless at 1442; it silently blackholed every large frame on a jumbo underlay, with no ICMP back because the drop is at the first hop. TapSpec.MTU comes from the same guest overlay MTU the DHCP option and host_mtu already use, and is applied in the bring-up call so a size the fabric cannot carry fails the launch instead of degrading quietly. * fix: stop setting tx_queue_size, which QEMU caps at 256 for tap QEMU only honours a tx_queue_size above 256 for vhost-user and vhost-vdpa backends. On a tap it rejects the value outright rather than clamping it, so every launch died with "must be a power of 2 between 256 and 256" and the instance went straight to terminated. rx_queue_size is unaffected and stays at 1024. A test pins the absence of tx_queue_size across machine types and queue counts. * docs: lead the tuning guide with the overlay segmentation finding The guide's headline was a 5.4x that came almost entirely from turning IPsec off, and it treated MTU and the fabric as the main levers. The host reaches 92.6 Gbit/s at MTU 1500 on the same NIC, so the fabric is not the constraint and the encrypted default is not the thing to tune away from. Adds the host-versus-guest comparison up front, documents network.underlay_mtu, and makes the missing segmentation offload the first open question. * ci: stop deselected matrix cells failing in teardown * docs: quantify what disabling overlay IPsec is worth The guide described the trade without the numbers an operator needs to make it. Adds the measured 4.9x, notes that the MTU and multiqueue changes follow ipsec_enabled rather than being separate knobs, and records two transition hazards: a stale wide MTU on running guests, and RunInstances failing while the OVN flows barrier catches up with the new SAs. * ci: collect and retain the e2e benchmark and cluster logs Cells 21 and 22 both copied two paths that only ever exist on the nodes, so the benchmark JSON was never uploaded and cell-22 collected nothing at all. Both now gather what they actually produce, and keep it for 90 days. * fix: quieten spx by installing an explicit CLI logger default spx never set a process logger, so it inherited whatever a linked package left behind — until now a JSON handler at Info on stdout, interleaving library log records with command output and breaking piped use. Execute() claims the default before any command runs: text on stderr, LevelError, so commands report failures through their own stderr writes and libraries stay quiet. --verbose/-v raises to info and SPX_LOG_LEVEL sets an arbitrary level, the flag winning over the environment. Service subcommands still call initTelemetry, which replaces the handler outright. No root --debug flag: viperblock, qemunbd, nats and awsgw each define their own --debug with a service-specific meaning, and a root persistent flag would shadow inconsistently depending on the command. The guard test samples the handler at package-init time rather than inside the test, because tests in this package start services that install their own logger legitimately. * feat: private /admin/CreateAccount for self-service tenant signup Add POST /admin/<Method> to the AWS gateway: SigV4-signed, JSON in and JSON out, callable only by an IAM user in the super-admin account holding spinifex:<Method>. CreateAccount is the one method, so a signup form can provision a tenant without an operator shell. No new service, port, unit or certificate — it mounts inside the gateway's existing authenticated group and inherits its auth, throttling and tracing. Provisioning is resumable rather than transactional, which fits stores that have no transaction. Two CAS creates carry the correctness: the client token admits one in-flight attempt and makes a lost response replayable for 24 hours, and the name reservation makes duplicate account names impossible. Every step after the account exists is create-if-absent, so a retry with the same token finishes the job instead of failing. The CLI and the endpoint share one provisioning helper, which closes a gap on both at once: CreateAccount never checked names, so running `spx admin account create` twice produced two accounts that differed only by ID. The response waits for the account's default VPC over a new ec2.EnsureDefaultVpc request/reply, rather than trusting the existing fire-and-forget iam.account.created publish — credentials for an account that cannot launch anything are worse than a slower call. [signup] max_accounts bounds how many accounts may exist; absent means uncapped, which is what every cluster does today. * fix: stop the awsgw template hardcoding one deployment's console URL console_url shipped as https://console.spx3.com, so every fresh install — including an on-premise cluster that will never run signup — was configured to point new accounts at our console. It is the only deployment-specific domain in any template. It now ships empty, and the gateway warns when it provisions an account without one, since a blank sign-in link is otherwise visible only to the customer who receives it. Extract the remote client from the command body while here. The signing, request shape and error decoding are the part a Cloudflare Worker mirrors, and as a function they are testable against a TLS test server rather than only by hand against a live cluster. * refactor: drop console_url from signup and default max_accounts to 128 The gateway never read console_url — it passed the value straight through to the response, and the idempotency record froze it for 24 hours so a corrected typo still replayed stale. The client that made the request already knows where it wants to send the new tenant. max_accounts is now a pointer in the parsed config so an absent [signup] section takes the 128 default while an explicit 0 still means uncapped. * feat: run the predastore fault-injection harness as cell-24 Cell-23 asks how fast predastore is when every node is healthy; cell-24 asks whether it survives one that is not. It runs after cell-23 rather than beside it: both want a quiet single runner, and a benchmark sharing a machine with a fault injection run measures the wrong thing. The port offset keeps the two clear of each other even so, and the job clears stale nodes on entry and exit, since a run killed before its cleanup trap leaves one holding ports that fail the next run somewhere unrelated. * ci: keep the throughput benchmark's output with its results The run log only existed in the job log, so the artifact it uploaded could not be read without going back to the run that produced it. * ci: keep the fault-injection run log with its results events.txt carries the timeline and the FAIL line, but not the build output or anything Warp wrote before the harness took over, so a failed cell could only be read from the job log it was printed into. * fix: serve the cluster region to the console instead of hardcoding it The console hardcoded ap-southeast-2 as its SigV4 region in seven files, so on any other cluster the credential scope was wrong and the gateway rejected every signed request. The SPA is embedded in the binary, so a build-time value would pin one binary to one region. The UI server now serves GET /api/config with the region, no-store so a proxy fronting several clusters cannot serve one the other's region, and the SPA loads it once before render. The region comes from the serving node's config because awsgw pins signature verification to that same field. Reading the cluster-wide [aws] region instead would let a cluster whose two keys disagree pass review and still fail to authenticate. Also relabels the login card as Spinifex credentials, since the fields are not AWS ones, and hides the CA install banner unless the console is reached directly: behind a proxy with a publicly trusted certificate there is nothing for the cluster CA to fix. * fix: say once when the host firewall will not be maintained Both quiet outcomes of the reconcile — no cluster config, and the policy switched off — return nil without logging, so a node that never arms is indistinguishable in the journal from one that armed on its first pass. * test: arm the host firewall before asserting on its peer set Only the ISO install arms the policy; every other path writes off to the mode file, so the daemon disables the firewall and never writes a peer set. The test read that file as its baseline, which it could never have. It now switches the mode on for its own duration and puts it back, waits for the peer set the way install-node.sh does, and on failure reports the mode, the node count and what the daemon said rather than just the missing file. * test: move the console region test to the external cmd package It was in package cmd, which test-package-check rejects, so preflight failed on the branch. The hook follows the pattern already in export_test.go. * test: make the reboot EIP diagnostic capable of naming the drop The two ofproto/trace probes injected packets with no TTL, so OVN's router killed them at lr_in_ip_input and both traces ended early for a reason that had nothing to do with the failure under investigation. A keyword filter then stripped the table-match lines, hiding where each trace stopped. Traces now carry a TTL and are kept whole, the real datapath megaflows for the guest reply are dumped alongside them, and any drop matching the guest source is resolved from its OpenFlow cookie back to the Southbound logical flow that installed it. * fix: exempt an EIP instance from the subnet drop gate with allow, not reroute A subnet with no route of its own carries a priority-1100 drop policy, and an instance with an external IP is lifted above it by a priority-1200 policy. That policy rerouted to the pool gateway, which in distributed NAT is outside every logical router port subnet, because the gateway LRP is the link-local /30. The static default route survives that by naming an output port; a routing policy has no such field, so ovn-northd discarded all three of them and the drop gate below swallowed everything the instances sent. The effect was that every external IP answered inbound traffic and none of it came back: the SYN reached the guest, the guest replied, and the reply died in the router's policy stage. Guest-initiated egress was lost the same way. The policy only ever needed to lift the gate. The router's default route already points at the gateway, so allow leaves the routing decision intact and gives northd nothing to reject. * test: match the log the firewall quorum gate actually emits The assertion waited for "chassis encap addresses; not writing a partial peer set", which no code has ever logged. The reconcile reports a short chassis list as "OVN reports N of M chassis encap addresses" and the maintain loop carries that into its retry warning, so the wait could only ever time out. Arming the node in the previous commit got the test as far as this assertion for the first time, which is why it surfaced now. The failure also prints the journal tail and the same inputs the earlier wait reports. * fix: stop one unreadable volume document failing every volume listing A single volume document whose shards no longer reconstruct made DescribeVolumes, DescribeVolumeStatus and DescribeVolumesModifications fail for every account, because the listing returned the first fetch error rather than skipping the document. ListVolumes now skips a document it cannot read or decode and logs the skip, mirroring ListAMIs. Callers whose answer would be wrong rather than merely partial take ListVolumesStrict: the orphan scan and the snapshot in-use check both gate destructive work on a complete list. The tolerance covers fetch failures as well as decode failures. An object whose shards no longer join is as unusable as one whose bytes will not parse, and it was the fetch path that took the cluster down.
1456 lines
56 KiB
Go
1456 lines
56 KiB
Go
//go:build e2e
|
|
|
|
// Package reboot is the single-node host-reboot resilience suite. It MUST run outside
|
|
// the cluster (on the CI runner) because systemctl reboot kills any in-VM process.
|
|
// AWS API traffic crosses the WAN via the SDK; SSH ops use harness.PeerSSH.
|
|
package reboot
|
|
|
|
import (
|
|
"context"
|
|
"encoding/base64"
|
|
"encoding/json"
|
|
"errors"
|
|
"fmt"
|
|
"net"
|
|
"os"
|
|
"path/filepath"
|
|
"strings"
|
|
"sync"
|
|
"testing"
|
|
"time"
|
|
|
|
"github.com/aws/aws-sdk-go/aws"
|
|
"github.com/aws/aws-sdk-go/aws/awserr"
|
|
"github.com/aws/aws-sdk-go/service/ec2"
|
|
"github.com/aws/aws-sdk-go/service/elbv2"
|
|
"github.com/mulgadc/spinifex/tests/e2e/harness"
|
|
"github.com/stretchr/testify/assert"
|
|
"github.com/stretchr/testify/require"
|
|
)
|
|
|
|
const (
|
|
keyName = "reboot-e2e-key"
|
|
tgName = "reboot-e2e-tg"
|
|
albName = "reboot-e2e-alb"
|
|
sgName = "reboot-e2e-sg"
|
|
vpcCIDR = "10.210.0.0/16"
|
|
subnetCIDR = "10.210.1.0/24"
|
|
httpPort = 80
|
|
probesPerRun = 20
|
|
appHTTPUnit = "reboot-e2e-httpd.service"
|
|
rebootDoctype = "reboot-e2e"
|
|
)
|
|
|
|
// appUserData provisions a systemd-managed HTTP responder. Using a unit (not nohup) ensures
|
|
// the responder survives the guest's own restart after host reboot.
|
|
const appUserData = `#!/bin/bash
|
|
set -e
|
|
INSTANCE_ID=$(hostname)
|
|
install -d -m 0755 /var/lib/httpd
|
|
echo "{\"instance_id\": \"${INSTANCE_ID}\"}" > /var/lib/httpd/index.html
|
|
cat > /etc/systemd/system/reboot-e2e-httpd.service <<'UNIT'
|
|
[Unit]
|
|
Description=Reboot E2E HTTP responder
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
|
|
[Service]
|
|
ExecStart=/usr/bin/python3 -m http.server 80 --bind 0.0.0.0 --directory /var/lib/httpd
|
|
Restart=always
|
|
RestartSec=1
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
UNIT
|
|
systemctl daemon-reload
|
|
systemctl enable --now reboot-e2e-httpd.service
|
|
|
|
# Workaround for viperblock WAL <4MB skip-upload bug:
|
|
# WriteWALToChunk(false) is a no-op when the WAL is under 4MB so small
|
|
# user-data writes (the unit file, index.html) sit in the on-host WAL
|
|
# and are lost when the host reboots before the WAL hits the threshold.
|
|
# Write 8MB of zeros + sync to push WAL over the threshold so the
|
|
# unit-file write definitely lands in predastore before the host reboot.
|
|
# Drop when upstream fix lands.
|
|
dd if=/dev/zero of=/var/lib/spinifex-e2e-pad bs=1M count=8 status=none
|
|
sync
|
|
rm -f /var/lib/spinifex-e2e-pad
|
|
sync
|
|
`
|
|
|
|
type fixture struct {
|
|
env *harness.Env
|
|
artifacts string
|
|
ssh *harness.PeerSSH
|
|
aws *harness.AWSClient
|
|
|
|
instanceType string
|
|
amiID string
|
|
|
|
vpcID string
|
|
subnetID string
|
|
igwID string
|
|
sgID string
|
|
|
|
appInstanceIDs []string
|
|
preIPs map[string]string
|
|
privateKeyPath string // local PEM written from CreateKeyPair KeyMaterial
|
|
|
|
tgArn string
|
|
albArn string
|
|
albID string
|
|
listenerArn string
|
|
albPublicIP string
|
|
albEniMAC string
|
|
|
|
preSwitches int
|
|
prePorts int
|
|
|
|
rebootStart time.Time
|
|
rebootDone bool
|
|
mu sync.Mutex // guards rebootDone for the on-failure dump
|
|
|
|
timeouts timeouts
|
|
}
|
|
|
|
type timeouts struct {
|
|
rebootWait time.Duration
|
|
daemonReady time.Duration
|
|
instanceRunning time.Duration
|
|
lbRecover time.Duration
|
|
}
|
|
|
|
// TestRebootResilience runs the 8-phase reboot resilience sequence.
|
|
// Phases are sequential: the reboot is a global event and each phase depends on the prior.
|
|
func TestRebootResilience(t *testing.T) {
|
|
env := harness.LoadEnv(t)
|
|
requireRunnerMode(t, env)
|
|
|
|
fix := &fixture{
|
|
env: env,
|
|
artifacts: harness.ArtifactDir(t, env),
|
|
ssh: harness.NewPeerSSH(),
|
|
preIPs: map[string]string{},
|
|
timeouts: timeouts{
|
|
rebootWait: durationEnv("REBOOT_WAIT_SECS", 300*time.Second),
|
|
daemonReady: durationEnv("DAEMON_READY_SECS", 180*time.Second),
|
|
instanceRunning: durationEnv("INSTANCE_RUNNING_SECS", 120*time.Second),
|
|
// 5min budget: post-reboot the daemon relaunches app VMs and the ALB sys.micro
|
|
// simultaneously; HC probing starts immediately but targets honestly report
|
|
// unhealthy until the responder unit finishes starting (no cloud-init shortcut).
|
|
lbRecover: durationEnv("LB_RECOVER_SECS", 300*time.Second),
|
|
},
|
|
}
|
|
|
|
resolveTrust(t, fix)
|
|
fix.aws = harness.NewAWSClient(t, env)
|
|
|
|
t.Cleanup(func() { cleanup(t, fix) })
|
|
|
|
harness.OnFailure(t, func() {
|
|
// Dump diagnostics windowed from reboot onwards (if reboot fired) so
|
|
// the analysis surfaces journal lines from the recovery path rather
|
|
// than the per-instance state-transition spam during setup.
|
|
fix.mu.Lock()
|
|
since := fix.rebootStart
|
|
done := fix.rebootDone
|
|
fix.mu.Unlock()
|
|
if !done {
|
|
since = time.Time{}
|
|
}
|
|
dumpDiagnostics(t, fix, since)
|
|
dumpAppDiagnostics(t, fix)
|
|
})
|
|
|
|
harness.Phase(t, "Phase 0 — Prerequisites")
|
|
phase0Prereqs(t, fix)
|
|
|
|
harness.Phase(t, "Phase 1 — VPC + Subnet + Security Group")
|
|
phase1Network(t, fix)
|
|
|
|
harness.Phase(t, "Phase 2 — App instances")
|
|
phase2Instances(t, fix)
|
|
|
|
harness.Phase(t, "Phase 3 — ALB + target group + listener")
|
|
phase3LB(t, fix)
|
|
|
|
harness.Phase(t, "Phase 4 — Pre-reboot traffic")
|
|
runHTTPBurst(t, fix, "ALB pre-reboot")
|
|
|
|
harness.Phase(t, "Phase 5 — Snapshot pre-reboot state")
|
|
phase5Snapshot(t, fix)
|
|
|
|
harness.Phase(t, "Phase 6 — systemctl reboot")
|
|
phase6Reboot(t, fix)
|
|
|
|
harness.Phase(t, "Phase 7 — Wait for spinifex readiness")
|
|
phase7Readiness(t, fix)
|
|
|
|
harness.Phase(t, "Phase 8 — Post-reboot assertions")
|
|
phase8Asserts(t, fix)
|
|
}
|
|
|
|
// --- Setup helpers -------------------------------------------------------
|
|
|
|
// requireRunnerMode bails early if the env isn't runner-resident (SPINIFEX_WAN_IP or endpoint missing).
|
|
func requireRunnerMode(t *testing.T, env *harness.Env) {
|
|
t.Helper()
|
|
if env.WANHost == "" {
|
|
t.Fatalf("reboot suite requires SPINIFEX_WAN_IP (no WAN host resolved)")
|
|
}
|
|
if env.Mode != harness.ModeSingle {
|
|
t.Skipf("reboot suite is single-node only (mode=%s)", env.Mode)
|
|
}
|
|
if endpoint := os.Getenv("SPINIFEX_AWS_ENDPOINT"); endpoint == "" {
|
|
t.Fatalf("reboot suite requires SPINIFEX_AWS_ENDPOINT=https://%s:%d", env.WANHost, env.AWSGWPort)
|
|
}
|
|
}
|
|
|
|
// resolveTrust SCPs the spinifex CA to a tmp file for real TLS verification.
|
|
// Falls back to SPINIFEX_AWS_INSECURE=1 on any error; CA trust is the cert suite's concern.
|
|
func resolveTrust(t *testing.T, fix *fixture) {
|
|
t.Helper()
|
|
if os.Getenv("SPINIFEX_AWS_INSECURE") == "1" {
|
|
t.Logf("trust: SPINIFEX_AWS_INSECURE=1 set; skipping CA fetch")
|
|
return
|
|
}
|
|
if os.Getenv("SPINIFEX_CA_CERT") != "" {
|
|
t.Logf("trust: SPINIFEX_CA_CERT already set; skipping CA fetch")
|
|
return
|
|
}
|
|
|
|
tmpCA := filepath.Join(fix.artifacts, "spinifex-ca.pem")
|
|
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
|
defer cancel()
|
|
|
|
candidates := []string{
|
|
"/etc/spinifex/ca.pem",
|
|
"$HOME/spinifex/config/ca.pem",
|
|
}
|
|
var lastErr error
|
|
for _, remote := range candidates {
|
|
out, err := fix.ssh.Run(ctx, fix.env.WANHost, "cat "+remote)
|
|
if err == nil && len(out) > 0 {
|
|
if writeErr := os.WriteFile(tmpCA, out, 0o600); writeErr == nil {
|
|
t.Logf("trust: fetched CA from %s -> %s", remote, tmpCA)
|
|
t.Setenv("SPINIFEX_CA_CERT", tmpCA)
|
|
return
|
|
} else {
|
|
lastErr = writeErr
|
|
}
|
|
} else {
|
|
lastErr = err
|
|
}
|
|
}
|
|
t.Logf("trust: CA fetch failed (%v); falling back to InsecureSkipVerify", lastErr)
|
|
t.Setenv("SPINIFEX_AWS_INSECURE", "1")
|
|
}
|
|
|
|
// --- Phases --------------------------------------------------------------
|
|
|
|
func phase0Prereqs(t *testing.T, fix *fixture) {
|
|
t.Helper()
|
|
harness.Step(t, "WAN_IP=%s", fix.env.WANHost)
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
|
|
defer cancel()
|
|
require.NoError(t, fix.ssh.Ping(ctx, fix.env.WANHost), "ssh to %s", fix.env.WANHost)
|
|
harness.Detail(t, "ssh", "ok")
|
|
|
|
fix.instanceType = discoverNanoInstanceType(t, fix.aws)
|
|
harness.Detail(t, "instance_type", fix.instanceType)
|
|
|
|
fix.amiID = discoverAMI(t, fix.aws)
|
|
harness.Detail(t, "ami", fix.amiID)
|
|
|
|
// Capture KeyMaterial so post-reboot diagnostics can SSH into guest VMs.
|
|
_, _ = fix.aws.EC2.DeleteKeyPair(&ec2.DeleteKeyPairInput{KeyName: aws.String(keyName)})
|
|
kpOut, err := fix.aws.EC2.CreateKeyPair(&ec2.CreateKeyPairInput{KeyName: aws.String(keyName)})
|
|
require.NoErrorf(t, err, "create-key-pair %s", keyName)
|
|
material := aws.StringValue(kpOut.KeyMaterial)
|
|
require.NotEmptyf(t, material, "create-key-pair %s returned no KeyMaterial", keyName)
|
|
fix.privateKeyPath = filepath.Join(fix.artifacts, keyName+".pem")
|
|
require.NoError(t, os.WriteFile(fix.privateKeyPath, []byte(material), 0o600), "write private key")
|
|
harness.Detail(t, "key_pair", keyName)
|
|
}
|
|
|
|
func phase1Network(t *testing.T, fix *fixture) {
|
|
t.Helper()
|
|
|
|
vpcOut, err := fix.aws.EC2.CreateVpc(&ec2.CreateVpcInput{CidrBlock: aws.String(vpcCIDR)})
|
|
require.NoError(t, err, "create-vpc")
|
|
fix.vpcID = aws.StringValue(vpcOut.Vpc.VpcId)
|
|
harness.Detail(t, "vpc", fix.vpcID)
|
|
|
|
igwOut, err := fix.aws.EC2.CreateInternetGateway(&ec2.CreateInternetGatewayInput{})
|
|
require.NoError(t, err, "create-internet-gateway")
|
|
fix.igwID = aws.StringValue(igwOut.InternetGateway.InternetGatewayId)
|
|
_, err = fix.aws.EC2.AttachInternetGateway(&ec2.AttachInternetGatewayInput{
|
|
InternetGatewayId: aws.String(fix.igwID),
|
|
VpcId: aws.String(fix.vpcID),
|
|
})
|
|
require.NoError(t, err, "attach-internet-gateway")
|
|
harness.Detail(t, "igw", fix.igwID)
|
|
|
|
subnetOut, err := fix.aws.EC2.CreateSubnet(&ec2.CreateSubnetInput{
|
|
VpcId: aws.String(fix.vpcID),
|
|
CidrBlock: aws.String(subnetCIDR),
|
|
})
|
|
require.NoError(t, err, "create-subnet")
|
|
fix.subnetID = aws.StringValue(subnetOut.Subnet.SubnetId)
|
|
_, _ = fix.aws.EC2.ModifySubnetAttribute(&ec2.ModifySubnetAttributeInput{
|
|
SubnetId: aws.String(fix.subnetID),
|
|
MapPublicIpOnLaunch: &ec2.AttributeBooleanValue{Value: aws.Bool(true)},
|
|
})
|
|
harness.Detail(t, "subnet", fix.subnetID)
|
|
|
|
sgOut, err := fix.aws.EC2.CreateSecurityGroup(&ec2.CreateSecurityGroupInput{
|
|
GroupName: aws.String(sgName),
|
|
Description: aws.String("Reboot E2E shared SG (ALB + app instances)"),
|
|
VpcId: aws.String(fix.vpcID),
|
|
})
|
|
require.NoError(t, err, "create-security-group")
|
|
fix.sgID = aws.StringValue(sgOut.GroupId)
|
|
|
|
// Use IpPermissions form — vpcd ignores the top-level shorthand field.
|
|
// tcp/22 is opened so on-failure diagnostics can SSH into app guests.
|
|
_, err = fix.aws.EC2.AuthorizeSecurityGroupIngress(&ec2.AuthorizeSecurityGroupIngressInput{
|
|
GroupId: aws.String(fix.sgID),
|
|
IpPermissions: []*ec2.IpPermission{{
|
|
IpProtocol: aws.String("tcp"),
|
|
FromPort: aws.Int64(httpPort),
|
|
ToPort: aws.Int64(httpPort),
|
|
IpRanges: []*ec2.IpRange{{CidrIp: aws.String("0.0.0.0/0")}},
|
|
}, {
|
|
IpProtocol: aws.String("tcp"),
|
|
FromPort: aws.Int64(22),
|
|
ToPort: aws.Int64(22),
|
|
IpRanges: []*ec2.IpRange{{CidrIp: aws.String("0.0.0.0/0")}},
|
|
}},
|
|
})
|
|
require.NoError(t, err, "authorize tcp/80 on %s", fix.sgID)
|
|
harness.Detail(t, "sg", fix.sgID)
|
|
}
|
|
|
|
func phase2Instances(t *testing.T, fix *fixture) {
|
|
t.Helper()
|
|
for i := 0; i < 2; i++ {
|
|
out, err := fix.aws.EC2.RunInstances(&ec2.RunInstancesInput{
|
|
ImageId: aws.String(fix.amiID),
|
|
InstanceType: aws.String(fix.instanceType),
|
|
KeyName: aws.String(keyName),
|
|
SubnetId: aws.String(fix.subnetID),
|
|
SecurityGroupIds: []*string{aws.String(fix.sgID)},
|
|
UserData: aws.String(base64.StdEncoding.EncodeToString([]byte(appUserData))),
|
|
MinCount: aws.Int64(1),
|
|
MaxCount: aws.Int64(1),
|
|
})
|
|
require.NoErrorf(t, err, "run-instances app%d", i+1)
|
|
require.NotEmpty(t, out.Instances)
|
|
id := aws.StringValue(out.Instances[0].InstanceId)
|
|
fix.appInstanceIDs = append(fix.appInstanceIDs, id)
|
|
harness.Detail(t, fmt.Sprintf("app%d", i+1), id)
|
|
}
|
|
|
|
for _, id := range fix.appInstanceIDs {
|
|
harness.WaitForInstanceRunning(t, fix.aws, id, fix.timeouts.instanceRunning)
|
|
}
|
|
|
|
for _, id := range fix.appInstanceIDs {
|
|
ip := describePrivateIP(t, fix.aws, id)
|
|
require.NotEmptyf(t, ip, "%s has no PrivateIpAddress", id)
|
|
fix.preIPs[id] = ip
|
|
harness.Detail(t, id, ip)
|
|
}
|
|
}
|
|
|
|
func phase3LB(t *testing.T, fix *fixture) {
|
|
t.Helper()
|
|
|
|
tgOut, err := fix.aws.ELBv2.CreateTargetGroup(&elbv2.CreateTargetGroupInput{
|
|
Name: aws.String(tgName),
|
|
Protocol: aws.String("HTTP"),
|
|
Port: aws.Int64(httpPort),
|
|
VpcId: aws.String(fix.vpcID),
|
|
HealthCheckPath: aws.String("/index.html"),
|
|
HealthCheckIntervalSeconds: aws.Int64(5),
|
|
HealthyThresholdCount: aws.Int64(2),
|
|
UnhealthyThresholdCount: aws.Int64(2),
|
|
})
|
|
require.NoError(t, err, "create-target-group")
|
|
fix.tgArn = aws.StringValue(tgOut.TargetGroups[0].TargetGroupArn)
|
|
harness.Detail(t, "tg", fix.tgArn)
|
|
|
|
targets := make([]*elbv2.TargetDescription, len(fix.appInstanceIDs))
|
|
for i, id := range fix.appInstanceIDs {
|
|
targets[i] = &elbv2.TargetDescription{Id: aws.String(id)}
|
|
}
|
|
_, err = fix.aws.ELBv2.RegisterTargets(&elbv2.RegisterTargetsInput{
|
|
TargetGroupArn: aws.String(fix.tgArn),
|
|
Targets: targets,
|
|
})
|
|
require.NoError(t, err, "register-targets")
|
|
|
|
lbOut, err := fix.aws.ELBv2.CreateLoadBalancer(&elbv2.CreateLoadBalancerInput{
|
|
Name: aws.String(albName),
|
|
Scheme: aws.String("internet-facing"),
|
|
Subnets: []*string{aws.String(fix.subnetID)},
|
|
SecurityGroups: []*string{aws.String(fix.sgID)},
|
|
})
|
|
require.NoError(t, err, "create-load-balancer")
|
|
require.NotEmpty(t, lbOut.LoadBalancers)
|
|
fix.albArn = aws.StringValue(lbOut.LoadBalancers[0].LoadBalancerArn)
|
|
parts := strings.Split(fix.albArn, "/")
|
|
fix.albID = parts[len(parts)-1]
|
|
harness.Detail(t, "alb", fix.albArn)
|
|
|
|
listenerOut, err := fix.aws.ELBv2.CreateListener(&elbv2.CreateListenerInput{
|
|
LoadBalancerArn: aws.String(fix.albArn),
|
|
Protocol: aws.String("HTTP"),
|
|
Port: aws.Int64(httpPort),
|
|
DefaultActions: []*elbv2.Action{{
|
|
Type: aws.String("forward"),
|
|
TargetGroupArn: aws.String(fix.tgArn),
|
|
}},
|
|
})
|
|
require.NoError(t, err, "create-listener")
|
|
fix.listenerArn = aws.StringValue(listenerOut.Listeners[0].ListenerArn)
|
|
|
|
harness.WaitForLBActive(t, fix.aws, fix.albArn, "ALB", 5*time.Minute)
|
|
|
|
eniDesc := fmt.Sprintf("ELB app/%s/%s", albName, fix.albID)
|
|
var eni *ec2.NetworkInterface
|
|
harness.EventuallyErr(t, func() error {
|
|
out, err := fix.aws.EC2.DescribeNetworkInterfaces(&ec2.DescribeNetworkInterfacesInput{
|
|
Filters: []*ec2.Filter{{
|
|
Name: aws.String("description"),
|
|
Values: []*string{aws.String(eniDesc)},
|
|
}},
|
|
})
|
|
if err != nil {
|
|
return err
|
|
}
|
|
if len(out.NetworkInterfaces) == 0 {
|
|
return fmt.Errorf("no ENI for %s", eniDesc)
|
|
}
|
|
eni = out.NetworkInterfaces[0]
|
|
return nil
|
|
}, 30*time.Second, 2*time.Second)
|
|
|
|
if eni.Association != nil {
|
|
fix.albPublicIP = aws.StringValue(eni.Association.PublicIp)
|
|
}
|
|
fix.albEniMAC = aws.StringValue(eni.MacAddress)
|
|
require.NotEmpty(t, fix.albPublicIP, "ALB has no public IP")
|
|
harness.Detail(t, "alb_public_ip", fix.albPublicIP, "alb_mac", fix.albEniMAC)
|
|
|
|
harness.WaitForTargetsHealthy(t, fix.aws, fix.tgArn, 2, "ALB", 3*time.Minute)
|
|
}
|
|
|
|
func phase5Snapshot(t *testing.T, fix *fixture) {
|
|
t.Helper()
|
|
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
|
defer cancel()
|
|
out, err := fix.ssh.Run(ctx, fix.env.WANHost, "sudo ovn-nbctl show 2>/dev/null || true")
|
|
if err != nil {
|
|
t.Logf("ovn-nbctl pre-snapshot: %v (continuing with zero counts)", err)
|
|
return
|
|
}
|
|
fix.preSwitches, fix.prePorts = countOVN(string(out))
|
|
harness.Detail(t, "pre_switches", fix.preSwitches, "pre_ports", fix.prePorts)
|
|
harness.DumpFile(t, fix.artifacts, "ovn-pre.txt", out)
|
|
|
|
// Capture OVN chassis identity pre-reboot so the post-reboot dump can be diffed;
|
|
// SB Chassis _uuid churn across reboot orphans every Port_Binding claim.
|
|
cstate, err := fix.ssh.Run(ctx, fix.env.WANHost, ovnChassisStateBundle)
|
|
if err != nil {
|
|
t.Logf("ovn chassis pre-snapshot: %v (non-fatal)", err)
|
|
}
|
|
harness.DumpFile(t, fix.artifacts, "ovn-pre-chassis.txt", cstate)
|
|
}
|
|
|
|
func phase6Reboot(t *testing.T, fix *fixture) {
|
|
t.Helper()
|
|
fix.mu.Lock()
|
|
fix.rebootStart = time.Now()
|
|
fix.mu.Unlock()
|
|
harness.Step(t, "issuing reboot via ssh (connection will drop)")
|
|
|
|
// SSH dies mid-command when the host shuts down; ignore the error.
|
|
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
|
defer cancel()
|
|
_, _ = fix.ssh.Run(ctx, fix.env.WANHost, "sudo systemctl reboot")
|
|
|
|
// Wait for shutdown to begin so the pre-reboot SSH window isn't mistaken for "back up".
|
|
time.Sleep(5 * time.Second)
|
|
|
|
harness.Step(t, "polling TCP/22 (timeout %s)", fix.timeouts.rebootWait)
|
|
deadline := time.Now().Add(fix.timeouts.rebootWait)
|
|
var lastErr error
|
|
for {
|
|
if time.Now().After(deadline) {
|
|
t.Fatalf("host SSH did not come back within %s (last err: %v)", fix.timeouts.rebootWait, lastErr)
|
|
}
|
|
conn, dialErr := net.DialTimeout("tcp", net.JoinHostPort(fix.env.WANHost, "22"), 5*time.Second)
|
|
if dialErr == nil {
|
|
_ = conn.Close()
|
|
pingCtx, pingCancel := context.WithTimeout(context.Background(), 15*time.Second)
|
|
err := fix.ssh.Ping(pingCtx, fix.env.WANHost)
|
|
pingCancel()
|
|
if err == nil {
|
|
break
|
|
}
|
|
lastErr = err
|
|
} else {
|
|
lastErr = dialErr
|
|
}
|
|
time.Sleep(5 * time.Second)
|
|
}
|
|
|
|
fix.mu.Lock()
|
|
fix.rebootDone = true
|
|
elapsed := time.Since(fix.rebootStart).Round(time.Second)
|
|
fix.mu.Unlock()
|
|
harness.Detail(t, "ssh_back_after", elapsed)
|
|
}
|
|
|
|
func phase7Readiness(t *testing.T, fix *fixture) {
|
|
t.Helper()
|
|
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
|
defer cancel()
|
|
for _, svc := range serviceList {
|
|
out, err := fix.ssh.Run(ctx, fix.env.WANHost, "systemctl is-active "+svc)
|
|
state := strings.TrimSpace(string(out))
|
|
if err != nil && state == "" {
|
|
state = "n/a"
|
|
}
|
|
harness.Detail(t, svc, state)
|
|
}
|
|
|
|
harness.Step(t, "polling describe-instance-types (timeout %s)", fix.timeouts.daemonReady)
|
|
harness.EventuallyErr(t, func() error {
|
|
_, err := fix.aws.EC2.DescribeInstanceTypes(&ec2.DescribeInstanceTypesInput{})
|
|
return err
|
|
}, fix.timeouts.daemonReady, 5*time.Second)
|
|
harness.Detail(t, "gateway", "responding")
|
|
}
|
|
|
|
func phase8Asserts(t *testing.T, fix *fixture) {
|
|
t.Helper()
|
|
|
|
// 8.1 instances back to running
|
|
harness.Step(t, "polling instance state (timeout %s)", fix.timeouts.instanceRunning)
|
|
harness.EventuallyErr(t, func() error {
|
|
for _, id := range fix.appInstanceIDs {
|
|
state := describeInstanceState(t, fix.aws, id)
|
|
if state != "running" {
|
|
return fmt.Errorf("%s state=%s", id, state)
|
|
}
|
|
}
|
|
return nil
|
|
}, fix.timeouts.instanceRunning, 5*time.Second)
|
|
harness.Detail(t, "instances", "all running")
|
|
|
|
// 8.2 private IPs preserved
|
|
for _, id := range fix.appInstanceIDs {
|
|
post := describePrivateIP(t, fix.aws, id)
|
|
assert.Equalf(t, fix.preIPs[id], post, "%s IP drift", id)
|
|
}
|
|
|
|
// 8.3 LaunchTime > REBOOT_START (relaunch signal)
|
|
for _, id := range fix.appInstanceIDs {
|
|
lt := describeLaunchTime(t, fix.aws, id)
|
|
require.Falsef(t, lt.IsZero(), "%s missing LaunchTime", id)
|
|
assert.Truef(t, !lt.Before(fix.rebootStart),
|
|
"%s LaunchTime=%s predates reboot=%s", id, lt, fix.rebootStart)
|
|
}
|
|
|
|
// 8.4 ALB active
|
|
harness.WaitForLBActive(t, fix.aws, fix.albArn, "ALB post-reboot", fix.timeouts.lbRecover)
|
|
|
|
// 8.5 targets healthy — capture diagnostics before failing so a timeout can be
|
|
// pinned to network-level drop vs. app/gateway stall.
|
|
if !pollTargetsHealthy(fix.aws, fix.tgArn, 2, fix.timeouts.lbRecover) {
|
|
harness.Step(t, "ALB post-reboot: targets NOT healthy — capturing diagnostics")
|
|
dctx, dcancel := context.WithTimeout(context.Background(), 120*time.Second)
|
|
dumpHeartbeatDiagnostics(dctx, t, fix)
|
|
dumpOVNDataplane(dctx, t, fix)
|
|
dumpOVNClaimState(dctx, t, fix)
|
|
dumpAppDiagnostics(t, fix)
|
|
dcancel()
|
|
t.Fatalf("ALB post-reboot: targets not healthy within %s (see diag-heartbeat.txt / diag-ovn-flows.txt / diag-ovn-claim.txt / diag-host-net.txt)", fix.timeouts.lbRecover)
|
|
}
|
|
harness.Step(t, "ALB post-reboot: 2 targets healthy")
|
|
|
|
// 8.6 Probe until both backends' instance_ids are seen before asserting round-robin:
|
|
// healthy-cache state may show stale "healthy" until both responders are actually up.
|
|
harness.Step(t, "waiting for ALB to serve HTTP from both backends")
|
|
probeCtx, probeCancel := context.WithTimeout(context.Background(), 180*time.Second)
|
|
defer probeCancel()
|
|
seen := map[string]bool{}
|
|
served := false
|
|
serveDeadline := time.Now().Add(180 * time.Second)
|
|
for time.Now().Before(serveDeadline) {
|
|
out, err := fix.ssh.Run(probeCtx, fix.env.WANHost,
|
|
fmt.Sprintf("curl -s --connect-timeout 2 --max-time 3 'http://%s:%d/' 2>/dev/null", fix.albPublicIP, httpPort))
|
|
if err == nil {
|
|
if id := instanceIDFromResponse(out); id != "" {
|
|
seen[id] = true
|
|
}
|
|
}
|
|
if len(seen) >= 2 {
|
|
served = true
|
|
break
|
|
}
|
|
time.Sleep(1 * time.Second)
|
|
}
|
|
// 8.5 reports targets healthy via the daemon's own view, which can be green
|
|
// while the north-south EIP datapath is dark. Capture dataplane diagnostics on
|
|
// a serve failure too — otherwise this path leaves no flow/ARP evidence.
|
|
if !served {
|
|
harness.Step(t, "ALB serve FAILED — capturing datapath diagnostics")
|
|
dctx, dcancel := context.WithTimeout(context.Background(), 120*time.Second)
|
|
dumpALBProbe(dctx, t, fix)
|
|
dumpOVNDataplane(dctx, t, fix)
|
|
dumpOVNClaimState(dctx, t, fix)
|
|
dumpHeartbeatDiagnostics(dctx, t, fix)
|
|
dumpAppDiagnostics(t, fix)
|
|
dcancel()
|
|
t.Fatalf("ALB did not serve from both backends within 180s post-reboot (saw=%v) "+
|
|
"— see diag-ovn-flows.txt / diag-alb-probe.txt / diag-heartbeat.txt", seen)
|
|
}
|
|
|
|
runHTTPBurst(t, fix, "ALB post-reboot")
|
|
|
|
// 8.6.5 Always capture in-guest responder state (pass or fail) to confirm genuine liveness.
|
|
dumpAppDiagnostics(t, fix)
|
|
|
|
// 8.7 ovn-nbctl drift check
|
|
dumpCtx, dumpCancel := context.WithTimeout(context.Background(), 30*time.Second)
|
|
defer dumpCancel()
|
|
out, err := fix.ssh.Run(dumpCtx, fix.env.WANHost, "sudo ovn-nbctl show 2>/dev/null || true")
|
|
if err != nil {
|
|
t.Logf("ovn-nbctl post-snapshot: %v (skipping drift assert)", err)
|
|
return
|
|
}
|
|
harness.DumpFile(t, fix.artifacts, "ovn-post.txt", out)
|
|
postSwitches, postPorts := countOVN(string(out))
|
|
harness.Detail(t, "post_switches", postSwitches, "post_ports", postPorts)
|
|
assert.Equalf(t, fix.preSwitches, postSwitches,
|
|
"ovn switch count drift: pre=%d post=%d", fix.preSwitches, postSwitches)
|
|
assert.Equalf(t, fix.prePorts, postPorts,
|
|
"ovn port count drift: pre=%d post=%d", fix.prePorts, postPorts)
|
|
}
|
|
|
|
// --- HTTP burst ----------------------------------------------------------
|
|
|
|
func runHTTPBurst(t *testing.T, fix *fixture, label string) {
|
|
t.Helper()
|
|
harness.Step(t, "sending %d HTTP requests (%s)", probesPerRun, label)
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
|
|
defer cancel()
|
|
cmd := fmt.Sprintf(
|
|
"for i in $(seq 1 %d); do curl -s --max-time 5 'http://%s:%d/' 2>/dev/null; echo; done",
|
|
probesPerRun, fix.albPublicIP, httpPort,
|
|
)
|
|
out, err := fix.ssh.Run(ctx, fix.env.WANHost, cmd)
|
|
if err != nil {
|
|
t.Logf("%s: ssh burst error: %v", label, err)
|
|
}
|
|
|
|
counts := map[string]int{}
|
|
totalOK := 0
|
|
for _, line := range strings.Split(string(out), "\n") {
|
|
line = strings.TrimSpace(line)
|
|
if line == "" {
|
|
continue
|
|
}
|
|
var payload struct {
|
|
InstanceID string `json:"instance_id"`
|
|
}
|
|
if jsonErr := json.Unmarshal([]byte(line), &payload); jsonErr != nil || payload.InstanceID == "" {
|
|
continue
|
|
}
|
|
counts[payload.InstanceID]++
|
|
totalOK++
|
|
}
|
|
|
|
for id, n := range counts {
|
|
harness.Detail(t, id, fmt.Sprintf("%d responses", n))
|
|
}
|
|
assert.GreaterOrEqualf(t, len(counts), 2,
|
|
"%s round-robin: expected >=2 unique responders, got %d (counts=%v)", label, len(counts), counts)
|
|
assert.GreaterOrEqualf(t, totalOK, probesPerRun/2,
|
|
"%s success rate: only %d/%d", label, totalOK, probesPerRun)
|
|
}
|
|
|
|
// --- Discovery -----------------------------------------------------------
|
|
|
|
func discoverNanoInstanceType(t *testing.T, c *harness.AWSClient) string {
|
|
t.Helper()
|
|
out, err := c.EC2.DescribeInstanceTypes(&ec2.DescribeInstanceTypesInput{})
|
|
require.NoError(t, err, "describe-instance-types")
|
|
for _, it := range out.InstanceTypes {
|
|
name := aws.StringValue(it.InstanceType)
|
|
if strings.Contains(name, "nano") {
|
|
return name
|
|
}
|
|
}
|
|
t.Fatal("no nano instance type available")
|
|
return ""
|
|
}
|
|
|
|
// discoverAMI prefers ubuntu, then non-alpine, then anything — matches
|
|
// run-reboot-e2e.sh's three-stage AMI fallback chain.
|
|
func discoverAMI(t *testing.T, c *harness.AWSClient) string {
|
|
t.Helper()
|
|
out, err := c.EC2.DescribeImages(&ec2.DescribeImagesInput{})
|
|
require.NoError(t, err, "describe-images")
|
|
var ubuntu, nonAlpine, anyAMI string
|
|
for _, img := range out.Images {
|
|
id := aws.StringValue(img.ImageId)
|
|
name := aws.StringValue(img.Name)
|
|
if anyAMI == "" {
|
|
anyAMI = id
|
|
}
|
|
if !strings.Contains(strings.ToLower(name), "alpine") && nonAlpine == "" {
|
|
nonAlpine = id
|
|
}
|
|
if strings.HasPrefix(name, "ami-ubuntu") {
|
|
ubuntu = id
|
|
break
|
|
}
|
|
}
|
|
for _, candidate := range []string{ubuntu, nonAlpine, anyAMI} {
|
|
if candidate != "" {
|
|
return candidate
|
|
}
|
|
}
|
|
t.Fatal("no AMIs available")
|
|
return ""
|
|
}
|
|
|
|
func describePrivateIP(t *testing.T, c *harness.AWSClient, id string) string {
|
|
t.Helper()
|
|
out, err := c.EC2.DescribeInstances(&ec2.DescribeInstancesInput{
|
|
InstanceIds: []*string{aws.String(id)},
|
|
})
|
|
if err != nil {
|
|
t.Logf("describe %s: %v", id, err)
|
|
return ""
|
|
}
|
|
if len(out.Reservations) == 0 || len(out.Reservations[0].Instances) == 0 {
|
|
return ""
|
|
}
|
|
return aws.StringValue(out.Reservations[0].Instances[0].PrivateIpAddress)
|
|
}
|
|
|
|
func describePublicIP(t *testing.T, c *harness.AWSClient, id string) string {
|
|
t.Helper()
|
|
out, err := c.EC2.DescribeInstances(&ec2.DescribeInstancesInput{
|
|
InstanceIds: []*string{aws.String(id)},
|
|
})
|
|
if err != nil {
|
|
t.Logf("describe %s: %v", id, err)
|
|
return ""
|
|
}
|
|
if len(out.Reservations) == 0 || len(out.Reservations[0].Instances) == 0 {
|
|
return ""
|
|
}
|
|
return aws.StringValue(out.Reservations[0].Instances[0].PublicIpAddress)
|
|
}
|
|
|
|
func describeInstanceState(t *testing.T, c *harness.AWSClient, id string) string {
|
|
t.Helper()
|
|
out, err := c.EC2.DescribeInstances(&ec2.DescribeInstancesInput{
|
|
InstanceIds: []*string{aws.String(id)},
|
|
})
|
|
if err != nil {
|
|
return "missing"
|
|
}
|
|
if len(out.Reservations) == 0 || len(out.Reservations[0].Instances) == 0 {
|
|
return "missing"
|
|
}
|
|
return aws.StringValue(out.Reservations[0].Instances[0].State.Name)
|
|
}
|
|
|
|
func describeLaunchTime(t *testing.T, c *harness.AWSClient, id string) time.Time {
|
|
t.Helper()
|
|
out, err := c.EC2.DescribeInstances(&ec2.DescribeInstancesInput{
|
|
InstanceIds: []*string{aws.String(id)},
|
|
})
|
|
if err != nil {
|
|
return time.Time{}
|
|
}
|
|
if len(out.Reservations) == 0 || len(out.Reservations[0].Instances) == 0 {
|
|
return time.Time{}
|
|
}
|
|
lt := out.Reservations[0].Instances[0].LaunchTime
|
|
if lt == nil {
|
|
return time.Time{}
|
|
}
|
|
return *lt
|
|
}
|
|
|
|
// --- OVN parsing ---------------------------------------------------------
|
|
|
|
// countOVN extracts logical-switch and port counts from `ovn-nbctl show`
|
|
// output by line-prefix matching (matches the bash `grep -c '^switch '` and
|
|
// `grep -c '^ port '` checks 1:1).
|
|
func countOVN(out string) (switches, ports int) {
|
|
for _, line := range strings.Split(out, "\n") {
|
|
switch {
|
|
case strings.HasPrefix(line, "switch "):
|
|
switches++
|
|
case strings.HasPrefix(line, " port "):
|
|
ports++
|
|
}
|
|
}
|
|
return
|
|
}
|
|
|
|
func instanceIDFromResponse(out []byte) string {
|
|
var payload struct {
|
|
InstanceID string `json:"instance_id"`
|
|
}
|
|
if err := json.Unmarshal(out, &payload); err != nil {
|
|
return ""
|
|
}
|
|
return payload.InstanceID
|
|
}
|
|
|
|
// --- Diagnostics ---------------------------------------------------------
|
|
|
|
var serviceList = []string{
|
|
"spinifex-nats", "spinifex-predastore", "spinifex-viperblock",
|
|
"spinifex-daemon", "spinifex-awsgw", "spinifex-vpcd",
|
|
"ovn-controller", "ovs-vswitchd", "ovn-northd",
|
|
}
|
|
|
|
func dumpDiagnostics(t *testing.T, fix *fixture, since time.Time) {
|
|
t.Helper()
|
|
var modeFlag string
|
|
if since.IsZero() {
|
|
modeFlag = "-n 200"
|
|
harness.Step(t, "diagnostics: last 200 lines per service")
|
|
} else {
|
|
modeFlag = fmt.Sprintf("--since=@%d", since.Unix())
|
|
harness.Step(t, "diagnostics: since reboot @%d", since.Unix())
|
|
}
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
|
|
defer cancel()
|
|
|
|
if !fix.ssh.IsReachable(ctx, fix.env.WANHost) {
|
|
harness.Step(t, "diagnostics: node unreachable; skipping journal dump")
|
|
return
|
|
}
|
|
|
|
for _, svc := range serviceList {
|
|
out, err := fix.ssh.Run(ctx, fix.env.WANHost,
|
|
fmt.Sprintf("sudo journalctl -u %s --no-pager %s 2>/dev/null || true", svc, modeFlag))
|
|
if err != nil {
|
|
harness.Detail(t, svc, fmt.Sprintf("journal err: %v", err))
|
|
continue
|
|
}
|
|
harness.DumpFile(t, fix.artifacts, fmt.Sprintf("journal-%s.log", svc), out)
|
|
}
|
|
}
|
|
|
|
// dumpAppDiagnostics probes each app instance's public IP from the host.
|
|
// App VMs have no QEMU hostfwd, so public-IP probes are the only available path.
|
|
func dumpAppDiagnostics(t *testing.T, fix *fixture) {
|
|
t.Helper()
|
|
if len(fix.appInstanceIDs) == 0 {
|
|
return
|
|
}
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 120*time.Second)
|
|
defer cancel()
|
|
|
|
if !fix.ssh.IsReachable(ctx, fix.env.WANHost) {
|
|
harness.Step(t, "app diagnostics: host unreachable; skipping")
|
|
return
|
|
}
|
|
|
|
harness.Step(t, "app diagnostics: probing %d app instance(s)", len(fix.appInstanceIDs))
|
|
|
|
dumpHostNetState(ctx, t, fix)
|
|
dumpALBProbe(ctx, t, fix)
|
|
|
|
for _, id := range fix.appInstanceIDs {
|
|
diagnoseAppInstance(ctx, t, fix, id)
|
|
}
|
|
}
|
|
|
|
// dumpHostNetState captures OVN/OVS/route state on the host in a single round-trip.
|
|
func dumpHostNetState(ctx context.Context, t *testing.T, fix *fixture) {
|
|
t.Helper()
|
|
bundle := `set +e
|
|
echo "=== ip route show ==="
|
|
ip route show 2>&1
|
|
echo "=== ip route show 192.168.0.0/24 ==="
|
|
ip route show 192.168.0.0/24 2>&1
|
|
echo "=== ip neigh show dev br-wan ==="
|
|
ip neigh show dev br-wan 2>&1 || ip neigh show 2>&1
|
|
echo "=== ovn-nbctl list nat ==="
|
|
sudo ovn-nbctl list nat 2>&1
|
|
echo "=== ovn-nbctl show ==="
|
|
sudo ovn-nbctl show 2>&1
|
|
echo "=== ovs-vsctl show ==="
|
|
sudo ovs-vsctl show 2>&1
|
|
echo "=== ovs-vsctl list-ports br-int ==="
|
|
sudo ovs-vsctl list-ports br-int 2>&1
|
|
echo "=== ip -d link show veth-wan-ovs (admin/carrier state) ==="
|
|
ip -d link show veth-wan-ovs 2>&1
|
|
echo "=== ip -d link show veth-wan-br (admin/carrier state) ==="
|
|
ip -d link show veth-wan-br 2>&1
|
|
echo "=== ovs-ofctl show br-ext (ofport numbering) ==="
|
|
sudo ovs-ofctl show br-ext 2>&1
|
|
`
|
|
out, err := fix.ssh.Run(ctx, fix.env.WANHost, bundle)
|
|
harness.DumpFile(t, fix.artifacts, "diag-host-net.txt", out)
|
|
if err != nil {
|
|
harness.Detail(t, "host-net", fmt.Sprintf("dump err: %v", err))
|
|
}
|
|
}
|
|
|
|
// pollTargetsHealthy is the non-fatal twin of harness.WaitForTargetsHealthy:
|
|
// returns true on healthy, false on timeout, so the caller can capture diagnostics before aborting.
|
|
func pollTargetsHealthy(c *harness.AWSClient, tgArn string, expected int, timeout time.Duration) bool {
|
|
deadline := time.Now().Add(timeout)
|
|
for {
|
|
out, err := c.ELBv2.DescribeTargetHealth(&elbv2.DescribeTargetHealthInput{
|
|
TargetGroupArn: aws.String(tgArn),
|
|
})
|
|
if err == nil {
|
|
healthy := 0
|
|
for _, th := range out.TargetHealthDescriptions {
|
|
if aws.StringValue(th.TargetHealth.State) == "healthy" {
|
|
healthy++
|
|
}
|
|
}
|
|
if healthy >= expected {
|
|
return true
|
|
}
|
|
}
|
|
if time.Now().After(deadline) {
|
|
return false
|
|
}
|
|
time.Sleep(5 * time.Second)
|
|
}
|
|
}
|
|
|
|
// dumpHeartbeatDiagnostics captures whether the lb-agent's heartbeat to the gateway is
|
|
// failing at the network layer or the app/gateway layer. Tcpdumps both the gateway port
|
|
// (any iface) and the br-wan north-south path; also pulls lb-agent console and awsgw journal.
|
|
func dumpHeartbeatDiagnostics(ctx context.Context, t *testing.T, fix *fixture) {
|
|
t.Helper()
|
|
if fix.albPublicIP == "" {
|
|
harness.Detail(t, "heartbeat", "no ALB public IP captured")
|
|
return
|
|
}
|
|
bundle := fmt.Sprintf(`set +e
|
|
echo "=== tcpdump ANY iface, gateway port :%[2]d (real heartbeat path), 14s ==="
|
|
sudo timeout 14 tcpdump -ni any -c 80 "tcp port %[2]d" 2>&1
|
|
echo "=== tcpdump br-wan ALB floating IP %[1]s + ARP (north-south), 8s ==="
|
|
sudo timeout 8 tcpdump -ni br-wan -c 40 "host %[1]s or (arp and host %[1]s)" 2>&1
|
|
echo "=== neigh for ALB floating IP %[1]s ==="
|
|
ip neigh show %[1]s 2>&1 || ip neigh show 2>&1 | grep -F %[1]s
|
|
echo "=== conntrack for gateway port %[2]d ==="
|
|
sudo conntrack -L 2>/dev/null | grep -E "dport=%[2]d|sport=%[2]d" | head || echo "(no conntrack entries / tool absent)"
|
|
echo "=== lb-agent console: Heartbeat / Config / Agent lines ==="
|
|
sudo grep -aE "Heartbeat|Config (hash|applied)|Agent started" /run/spinifex/console-*.log 2>/dev/null | tail -25
|
|
echo "=== awsgw journal: did the gateway RECEIVE heartbeats? ==="
|
|
sudo journalctl -u spinifex-awsgw --no-pager --since "3 min ago" 2>/dev/null | grep -iE "heartbeat|lb-agent|/lb/|target.?health" | tail -20
|
|
`, fix.albPublicIP, fix.env.AWSGWPort)
|
|
|
|
out, err := fix.ssh.Run(ctx, fix.env.WANHost, bundle)
|
|
harness.DumpFile(t, fix.artifacts, "diag-heartbeat.txt", out)
|
|
if err != nil {
|
|
harness.Detail(t, "heartbeat", fmt.Sprintf("dump err: %v", err))
|
|
return
|
|
}
|
|
body := string(out)
|
|
// Detect any L4 packet on the gateway port.
|
|
sawGatewayL4 := strings.Contains(body, fmt.Sprintf(".%d:", fix.env.AWSGWPort)) ||
|
|
strings.Contains(body, fmt.Sprintf(".%d >", fix.env.AWSGWPort)) ||
|
|
strings.Contains(body, fmt.Sprintf("> %s.%d", fix.env.WANHost, fix.env.AWSGWPort))
|
|
switch {
|
|
case sawGatewayL4:
|
|
harness.Detail(t, "heartbeat", "gateway-port PACKETS FLOW — network path healthy; failure is app/gateway-layer (TLS/HTTP/gateway state)")
|
|
default:
|
|
harness.Detail(t, "heartbeat", "NO packets on gateway port — network-level drop; check diag-ovn-flows.txt for br-int forwarding")
|
|
}
|
|
}
|
|
|
|
// dumpOVNDataplane captures OVN/OVS dataplane state at failure time: topology, br-int flows,
|
|
// Port_Bindings, NAT rows, neigh table, and ALB floating-IP reachability from the host.
|
|
func dumpOVNDataplane(ctx context.Context, t *testing.T, fix *fixture) {
|
|
t.Helper()
|
|
bundle := fmt.Sprintf(`set +e
|
|
EIP=%[1]s
|
|
echo "=== ovs-vsctl show (bridges + ports) ==="
|
|
sudo ovs-vsctl show 2>&1 | head -120
|
|
echo "=== ovs-ofctl show br-int (port->ofport map) ==="
|
|
sudo ovs-ofctl show br-int 2>&1 | grep -aE '\(' | head -50
|
|
echo "=== br-int flows: DNAT/conntrack (ct_/nat/ct) ==="
|
|
sudo ovs-ofctl dump-flows br-int 2>&1 | grep -aE 'ct_|nat\(|ct\(' | head -120
|
|
echo "=== br-int flows: EIP + guest subnet 10.210 (forwarding) ==="
|
|
sudo ovs-ofctl dump-flows br-int 2>&1 | grep -aE "nw_dst=$EIP|10\.210\.1\.|192\.168\.0\.21" | head -150
|
|
echo "=== br-int flows: terminal output: actions ==="
|
|
sudo ovs-ofctl dump-flows br-int 2>&1 | grep -aE 'output:|resubmit' | grep -avE 'table=(0|8|9|10|11),' | head -60
|
|
echo "=== ovs-ofctl dump-flows br-ext ==="
|
|
sudo ovs-ofctl dump-flows br-ext 2>&1 | head -120
|
|
echo "=== ovs-appctl fdb/show br-ext (WAN-side MAC learning) ==="
|
|
sudo ovs-appctl fdb/show br-ext 2>&1 | head -40
|
|
echo "=== ovn-sbctl Port_Binding (chassis assignment) ==="
|
|
sudo ovn-sbctl --no-leader-only list Port_Binding 2>&1 | grep -E "logical_port|chassis|mac|type " | head -400
|
|
echo "=== DGP / chassisredirect claim state (untruncated) ==="
|
|
echo "--- every chassisredirect Port_Binding -> resident chassis ---"
|
|
for cr in $(sudo ovn-sbctl --no-leader-only --bare --columns=logical_port find Port_Binding type=chassisredirect 2>/dev/null); do
|
|
ch=$(sudo ovn-sbctl --no-leader-only --bare --columns=chassis find Port_Binding logical_port="$cr" 2>/dev/null)
|
|
hg=$(sudo ovn-sbctl --no-leader-only --bare --columns=ha_chassis_group find Port_Binding logical_port="$cr" 2>/dev/null)
|
|
echo " CR=[$cr] chassis=[$ch] ha_chassis_group=[$hg]"
|
|
done
|
|
echo "--- every gw-vpc LRP -> its referenced chassis-redirect-port (does the CR row exist?) ---"
|
|
for gp in $(sudo ovn-sbctl --no-leader-only --bare --columns=logical_port find Port_Binding 2>/dev/null | tr ' ' '\n' | grep -aE '^gw-vpc'); do
|
|
crp=$(sudo ovn-sbctl --no-leader-only --bare --columns=options find Port_Binding logical_port="$gp" 2>/dev/null | tr ',' '\n' | grep -a chassis-redirect-port | cut -d= -f2)
|
|
exists=$(sudo ovn-sbctl --no-leader-only --bare --columns=_uuid find Port_Binding logical_port="$crp" 2>/dev/null)
|
|
echo " GW-LRP=[$gp] redirect-port=[$crp] CR_row_exists=[$exists]"
|
|
done
|
|
echo "--- NB Gateway_Chassis bindings (DGP HA) ---"
|
|
sudo ovn-nbctl --no-leader-only list Gateway_Chassis 2>&1 | grep -aE 'name|chassis_name|priority' | head -60
|
|
echo "--- NB Logical_Router_Port (name + gateway_chassis + networks) ---"
|
|
sudo ovn-nbctl --no-leader-only --columns=name,gateway_chassis,networks list Logical_Router_Port 2>&1 | head -80
|
|
echo "=== ovn-nbctl dnat_and_snat NAT rows ==="
|
|
sudo ovn-nbctl --no-leader-only list NAT 2>&1 | grep -E "external_ip|logical_ip|external_mac|type " | head -60
|
|
echo "=== host neigh (all floating IPs) ==="
|
|
ip neigh show 2>&1 | grep -E "br-wan|br-int" | head -40
|
|
echo "=== REAL ARP to EIP $EIP (flush primed neigh first) ==="
|
|
sudo ip neigh flush $EIP 2>&1 || true
|
|
ping -c 2 -W 2 $EIP 2>&1
|
|
ip neigh show $EIP 2>&1
|
|
GWMAC=$(ip neigh show $EIP | awk '{for(i=1;i<=NF;i++) if($i=="lladdr") print $(i+1)}')
|
|
echo "discovered cr-gw MAC for EIP: [$GWMAC]"
|
|
echo "=== ofproto/trace north-south DNAT (real ingress per br-int patch port, dl_dst=cr-gw) ==="
|
|
if [ -n "$GWMAC" ]; then
|
|
for p in $(sudo ovs-vsctl list-ports br-int 2>/dev/null | grep -aE 'patch-br-int-to-ext'); do
|
|
echo "--- in_port=$p ---"
|
|
sudo ovs-appctl ofproto/trace br-int "in_port=$p,tcp,dl_src=02:00:00:00:00:01,dl_dst=$GWMAC,nw_src=192.168.0.11,nw_dst=$EIP,nw_ttl=64,tp_src=40000,tp_dst=80" 2>&1 | head -200
|
|
done
|
|
else
|
|
echo "(no cr-gw MAC resolved — ARP itself failed)"
|
|
fi
|
|
echo "=== south->north RETURN trace, unfiltered (guest reply -> un-DNAT -> WAN) ==="
|
|
PRIV=$(sudo ovn-nbctl --no-leader-only --bare --columns=logical_ip find NAT external_ip="\"$EIP\"" 2>/dev/null | head -1)
|
|
CLIENT=$({ sudo ovs-appctl dpctl/dump-conntrack 2>/dev/null || sudo conntrack -L 2>/dev/null; } | grep -aE "dst=$EIP" | grep -aoE 'src=[0-9.]+' | head -1 | cut -d= -f2)
|
|
[ -z "$CLIENT" ] && CLIENT=192.168.1.1
|
|
GLSP=""; GMAC=""
|
|
for lp in $(sudo ovn-sbctl --no-leader-only --bare --columns=logical_port find Port_Binding type='""' 2>/dev/null); do
|
|
m=$(sudo ovn-sbctl --no-leader-only --bare --columns=mac find Port_Binding logical_port="$lp" 2>/dev/null)
|
|
case "$m" in *"$PRIV"*) GLSP="$lp"; GMAC=$(echo "$m" | awk '{print $1}'); break;; esac
|
|
done
|
|
TAP=$(sudo ovs-vsctl --bare --columns=name find Interface external_ids:iface-id="$GLSP" 2>/dev/null | head -1)
|
|
SUBNET_GW=$(echo "$PRIV" | awk -F. '{print $1"."$2"."$3".1"}')
|
|
LRPMAC=$(sudo ovs-ofctl dump-flows br-int 2>/dev/null | grep -a "arp_tpa=$SUBNET_GW," | grep -aoE 'mod_dl_src:[0-9a-f:]+' | head -1 | cut -d: -f2-)
|
|
echo "PRIV=[$PRIV] CLIENT=[$CLIENT] GLSP=[$GLSP] GMAC=[$GMAC] TAP=[$TAP] LRPMAC=[$LRPMAC]"
|
|
if [ -n "$TAP" ] && [ -n "$GMAC" ] && [ -n "$LRPMAC" ]; then
|
|
sudo ovs-appctl ofproto/trace br-int "in_port=$TAP,tcp,dl_src=$GMAC,dl_dst=$LRPMAC,nw_src=$PRIV,nw_dst=$CLIENT,nw_ttl=64,tp_src=80,tp_dst=40000,tcp_flags=ack" 2>&1 | head -300
|
|
else
|
|
echo "(return-trace discovery incomplete — skipping)"
|
|
fi
|
|
echo "=== conntrack DNAT entries (EIP/guest) ==="
|
|
{ sudo ovs-appctl dpctl/dump-conntrack 2>/dev/null || sudo conntrack -L 2>/dev/null; } | grep -aE "$EIP|10\.210\.1\." | head -30 || echo "(none)"
|
|
echo "=== tap capture: does the SYN reach the guest? (tcpdump -i any while curling EIP) ==="
|
|
sudo timeout 6 tcpdump -i any -nn -c 25 "host $EIP or net 10.210.0.0/16" 2>&1 &
|
|
TCPID=$!
|
|
sleep 1
|
|
curl -s --connect-timeout 2 --max-time 4 "http://$EIP/" >/dev/null 2>&1 &
|
|
wait $TCPID 2>/dev/null
|
|
echo "=== (end tap capture) ==="
|
|
echo "=== datapath megaflows carrying the guest reply (what the kernel actually did, not a trace) ==="
|
|
sudo ovs-appctl dpctl/dump-flows -m 2>/dev/null | grep -aF "src=$PRIV" | head -30 || echo "(none)"
|
|
echo "=== PERSISTED-STATE DUMP (survives ovn-controller restart) ==="
|
|
echo "--- external next-hop MACs (regenerated each boot) ---"
|
|
for IF in veth-wan-ovs veth-wan-br br-wan br-ext; do
|
|
M=$(ip link show "$IF" 2>/dev/null | grep -aoE 'link/ether [0-9a-f:]+' | awk '{print $2}')
|
|
echo " $IF mac=[$M]"
|
|
done
|
|
echo "--- SB MAC_Binding (logical_port ip mac) ---"
|
|
sudo ovn-sbctl --no-leader-only list MAC_Binding 2>&1 | grep -aE 'ip|mac|logical_port' | head -40 || echo "(none)"
|
|
echo "--- host neigh on external subnet ---"
|
|
ip neigh show 2>/dev/null | grep -aE '192\.168\.' | head -20 || echo "(none)"
|
|
echo "=== live failure marker (confirms drop is active during dump) ==="
|
|
curl -s -o /dev/null -w "eip-probe HTTP=%%{http_code} time=%%{time_total}\n" --connect-timeout 2 --max-time 5 "http://$EIP/" 2>&1
|
|
echo "=== NB-SOURCE OF THE RETURN-PATH DROP (names what installs table-79 drop) ==="
|
|
echo "--- SB lflow stages that DROP the guest reply (match nw_src=$PRIV) ---"
|
|
sudo ovn-sbctl --no-leader-only lflow-list 2>/dev/null | grep -aiE 'drop' | grep -aF "$PRIV" | head -20 || echo "(none — drop is not a northd logical flow; likely ovn-controller port-security)"
|
|
echo "--- SB lflow lines mentioning guest MAC $GMAC (stage + match context) ---"
|
|
sudo ovn-sbctl --no-leader-only lflow-list 2>/dev/null | grep -aiF "$GMAC" | head -20 || echo "(none)"
|
|
echo "--- NB NAT rows: type/external_mac/logical_port/options (centralised=[] vs distributed) ---"
|
|
sudo ovn-nbctl --no-leader-only --columns=external_ip,logical_ip,type,external_mac,logical_port,options list NAT 2>/dev/null | head -60 || echo "(none)"
|
|
echo "--- NB Logical_Router_Policy (drop-gate + per-instance reroute) ---"
|
|
sudo ovn-nbctl --no-leader-only --columns=priority,match,action,nexthop list Logical_Router_Policy 2>/dev/null | head -40 || echo "(none)"
|
|
echo "--- NB LSP addresses + port_security (source if drop is physical port-sec) ---"
|
|
sudo ovn-nbctl --no-leader-only --columns=name,addresses,port_security list Logical_Switch_Port 2>/dev/null | grep -aF -A0 "$PRIV" | head -20 || echo "(none)"
|
|
echo "=== (end NB-source dump) ==="
|
|
echo "=== ATTRIBUTION OF ANY DROP CARRYING THE GUEST SOURCE (OF cookie -> SB Logical_Flow) ==="
|
|
echo "--- note: OVN table 79 is MAC-binding accounting; its drop is a counted clone, not a loss ---"
|
|
DROPS=$(sudo ovs-ofctl dump-flows br-int 2>/dev/null | grep -a 'actions=drop' | grep -aF "nw_src=$PRIV")
|
|
echo "${DROPS:-(none)}" | head -20
|
|
LF=$(mktemp)
|
|
sudo ovn-sbctl --no-leader-only --columns=_uuid,pipeline,table_id,priority,match,actions \
|
|
list Logical_Flow >"$LF" 2>/dev/null
|
|
for C in $(echo "$DROPS" | grep -aoE 'cookie=0x[0-9a-f]+' | cut -d= -f2 | sort -u); do
|
|
P=$(printf '%%08x' "$C" 2>/dev/null)
|
|
echo "--- cookie $C is Logical_Flow $P* ---"
|
|
awk -v p="$P" 'BEGIN{RS=""} index($0,p)' "$LF" | head -12
|
|
grep -qaF "$P" "$LF" || echo " (no SB lflow — installed by ovn-controller itself, not northd)"
|
|
done
|
|
rm -f "$LF"
|
|
echo "--- is_chassis_resident inputs: bindings this ovn-controller claims locally ---"
|
|
sudo ovn-appctl -t ovn-controller debug/dump-local-bindings 2>&1 | head -40
|
|
echo "--- Port_Binding for guest LSP $GLSP (chassis must be this chassis) ---"
|
|
sudo ovn-sbctl --no-leader-only list Port_Binding "$GLSP" 2>&1 \
|
|
| grep -aE '_uuid|chassis|logical_port|^up|requested' | head -20
|
|
echo "=== (end drop attribution) ==="
|
|
`, fix.albPublicIP)
|
|
|
|
out, err := fix.ssh.Run(ctx, fix.env.WANHost, bundle)
|
|
harness.DumpFile(t, fix.artifacts, "diag-ovn-flows.txt", out)
|
|
if err != nil {
|
|
harness.Detail(t, "ovn", fmt.Sprintf("dump err: %v", err))
|
|
return
|
|
}
|
|
body := string(out)
|
|
reachable := strings.Contains(body, " 0% packet loss")
|
|
hasFlows := strings.Contains(body, "cookie=") && strings.Contains(body, "table=")
|
|
switch {
|
|
case reachable:
|
|
harness.Detail(t, "ovn", "ALB floating IP reachable from host — forwarding OK, failure is elsewhere")
|
|
case hasFlows:
|
|
harness.Detail(t, "ovn", "br-int has flows but floating IP unreachable — flow/forwarding mismatch (inspect dump-flows + ofproto/trace)")
|
|
default:
|
|
harness.Detail(t, "ovn", "br-int flows MISSING and floating IP unreachable — dataplane flow-restore gap post-reboot")
|
|
}
|
|
}
|
|
|
|
// ovnChassisStateBundle captures OVN chassis identity + SB records for pre/post diff.
|
|
// _uuid churn across reboot orphans every Port_Binding claim.
|
|
const ovnChassisStateBundle = `set +e
|
|
echo "=== OVS system-id (chassis identity; must be stable across reboot) ==="
|
|
sudo cat /etc/openvswitch/system-id.conf 2>&1
|
|
sudo ovs-vsctl get Open_vSwitch . external_ids:system-id 2>&1
|
|
echo "=== ovn-sbctl list Chassis (name/_uuid/hostname/encaps) ==="
|
|
sudo ovn-sbctl --no-leader-only list Chassis 2>&1 | grep -E "_uuid|name|hostname|ip " | head -40
|
|
echo "=== ovn-sbctl list Chassis_Private (claimed-by + nb_cfg) ==="
|
|
sudo ovn-sbctl --no-leader-only list Chassis_Private 2>&1 | grep -E "_uuid|name|chassis|nb_cfg" | head -40
|
|
echo "=== ovn get-connection (did set-connection persist?) ==="
|
|
sudo ovn-nbctl get-connection 2>&1
|
|
sudo ovn-sbctl get-connection 2>&1
|
|
`
|
|
|
|
// dumpOVNClaimState captures evidence for why ovn-controller is not claiming Port_Bindings.
|
|
// Pulls /var/log/ovn directly (not journald), SB connection state, and boot ordering
|
|
// to distinguish chassis-identity churn from an SB/NB startup-ordering race.
|
|
func dumpOVNClaimState(ctx context.Context, t *testing.T, fix *fixture) {
|
|
t.Helper()
|
|
bundle := ovnChassisStateBundle + `echo "=== ovn-controller SB connection-status ==="
|
|
sudo env OVS_RUNDIR=/var/run/ovn ovs-appctl -t ovn-controller connection-status 2>&1
|
|
echo "=== ovn-controller debug/status ==="
|
|
sudo env OVS_RUNDIR=/var/run/ovn ovs-appctl -t ovn-controller debug/status 2>&1 | head -20
|
|
echo "=== gw/eni Port_Binding claim detail (chassis/up/requested_chassis) ==="
|
|
sudo ovn-sbctl --no-leader-only --columns=logical_port,type,chassis,up,requested_chassis find Port_Binding 2>&1 | grep -A4 -E "gw-vpc|port-eni" | head -60
|
|
echo "=== ovn-central vs ovn-controller boot ordering ==="
|
|
systemctl show ovn-central ovn-controller openvswitch-switch -p Id -p ActiveEnterTimestamp -p ExecMainStartTimestamp 2>&1
|
|
echo "=== tail /var/log/ovn/ovn-controller.log ==="
|
|
sudo tail -n 60 /var/log/ovn/ovn-controller.log 2>&1
|
|
echo "=== tail /var/log/ovn/ovn-northd.log ==="
|
|
sudo tail -n 40 /var/log/ovn/ovn-northd.log 2>&1
|
|
`
|
|
out, err := fix.ssh.Run(ctx, fix.env.WANHost, bundle)
|
|
harness.DumpFile(t, fix.artifacts, "diag-ovn-claim.txt", out)
|
|
if err != nil {
|
|
harness.Detail(t, "ovn-claim", fmt.Sprintf("dump err: %v", err))
|
|
return
|
|
}
|
|
body := string(out)
|
|
switch {
|
|
case strings.Contains(body, "not connected"):
|
|
harness.Detail(t, "ovn-claim", "ovn-controller NOT connected to SB — claim failure is an SB-connection/ordering issue")
|
|
case strings.Contains(body, "chassis : []"), strings.Contains(body, "chassis : []"):
|
|
harness.Detail(t, "ovn-claim", "gw/eni Port_Binding unclaimed (chassis empty) — diff ovn-pre-chassis.txt vs this for identity churn")
|
|
default:
|
|
harness.Detail(t, "ovn-claim", "see diag-ovn-claim.txt — compare Chassis _uuid against ovn-pre-chassis.txt")
|
|
}
|
|
}
|
|
|
|
// dumpALBProbe probes the ALB public IP from host — pre-reboot this path
|
|
// worked (the burst hit it). If post-reboot ALB is also unreachable, the
|
|
// gateway/ext-bridge is broken; if ALB reaches but apps don't, only the
|
|
// app-VM side is broken. Complements per-app probes.
|
|
func dumpALBProbe(ctx context.Context, t *testing.T, fix *fixture) {
|
|
t.Helper()
|
|
if fix.albPublicIP == "" {
|
|
harness.Detail(t, "alb", "no public IP captured")
|
|
return
|
|
}
|
|
probe := fmt.Sprintf(`set +e
|
|
echo "=== ping ALB %[1]s ==="
|
|
ping -c 2 -W 2 %[1]s 2>&1
|
|
echo "=== tcp-connect ALB %[1]s:80 ==="
|
|
timeout 5 bash -c 'exec 3<>/dev/tcp/%[1]s/80 && echo "tcp OPEN" || echo "tcp CLOSED/UNREACH"'
|
|
echo "=== curl ALB http://%[1]s/ ==="
|
|
curl -sS -m 5 -w 'HTTP=%%{http_code} time=%%{time_total}s\n' http://%[1]s/ 2>&1
|
|
`, fix.albPublicIP)
|
|
out, err := fix.ssh.Run(ctx, fix.env.WANHost, probe)
|
|
harness.DumpFile(t, fix.artifacts, "diag-alb-probe.txt", out)
|
|
if err != nil {
|
|
harness.Detail(t, "alb", fmt.Sprintf("probe err: %v", err))
|
|
return
|
|
}
|
|
body := string(out)
|
|
switch {
|
|
case strings.Contains(body, `"instance_id"`):
|
|
harness.Detail(t, "alb", "ALB serving (responder reachable through HAProxy)")
|
|
case strings.Contains(body, "tcp OPEN"):
|
|
harness.Detail(t, "alb", "ALB tcp open, no body (HAProxy up but backends silent)")
|
|
case strings.Contains(body, "tcp CLOSED"):
|
|
harness.Detail(t, "alb", "ALB tcp closed/unreach (gateway or sys.micro broken)")
|
|
default:
|
|
harness.Detail(t, "alb", "probe inconclusive — see artifact")
|
|
}
|
|
}
|
|
|
|
// diagnoseAppInstance probes one instance from the host VM: qemu running,
|
|
// ping, TCP-connect on :80, HTTP GET. Single shell session per instance.
|
|
func diagnoseAppInstance(ctx context.Context, t *testing.T, fix *fixture, id string) {
|
|
t.Helper()
|
|
|
|
psOut, err := fix.ssh.Run(ctx, fix.env.WANHost,
|
|
fmt.Sprintf("ps -eo args= | grep -F %q | grep -v grep || true", id))
|
|
harness.DumpFile(t, fix.artifacts, fmt.Sprintf("diag-%s-qemu-ps.txt", id), psOut)
|
|
switch {
|
|
case err != nil:
|
|
harness.Detail(t, id, fmt.Sprintf("qemu ps err: %v", err))
|
|
case len(strings.TrimSpace(string(psOut))) == 0:
|
|
harness.Detail(t, id, "qemu NOT running on host")
|
|
default:
|
|
harness.Detail(t, id, "qemu running")
|
|
}
|
|
|
|
// Serial console (ttyS0) — cloud-init, kernel oops, systemd boot events.
|
|
consoleOut, _ := fix.ssh.Run(ctx, fix.env.WANHost,
|
|
fmt.Sprintf("sudo tail -c 262144 /run/spinifex/console-%s.log 2>/dev/null || true", id))
|
|
harness.DumpFile(t, fix.artifacts, fmt.Sprintf("diag-%s-console.log", id), consoleOut)
|
|
|
|
priv := fix.preIPs[id]
|
|
pub := describePublicIP(t, fix.aws, id)
|
|
harness.Detail(t, id, fmt.Sprintf("priv=%s pub=%s", priv, pub))
|
|
if pub == "" {
|
|
harness.Detail(t, id, "no public IP — cannot probe from host")
|
|
return
|
|
}
|
|
|
|
probe := fmt.Sprintf(`set +e
|
|
echo "=== ping %[1]s ==="
|
|
ping -c 2 -W 2 %[1]s 2>&1
|
|
echo "=== tcp-connect %[1]s:80 ==="
|
|
timeout 5 bash -c 'exec 3<>/dev/tcp/%[1]s/80 && echo "tcp OPEN" || echo "tcp CLOSED/UNREACH"'
|
|
echo "=== curl http://%[1]s/ ==="
|
|
curl -sS -m 5 -o /tmp/diag-body-%[2]s -w 'HTTP=%%{http_code} time=%%{time_total}s size=%%{size_download}\n' http://%[1]s/
|
|
echo "=== body ==="
|
|
cat /tmp/diag-body-%[2]s 2>/dev/null; echo
|
|
rm -f /tmp/diag-body-%[2]s
|
|
`, pub, id)
|
|
|
|
out, err := fix.ssh.Run(ctx, fix.env.WANHost, probe)
|
|
harness.DumpFile(t, fix.artifacts, fmt.Sprintf("diag-%s-net-probe.txt", id), out)
|
|
if err != nil {
|
|
harness.Detail(t, id, fmt.Sprintf("probe err: %v", err))
|
|
return
|
|
}
|
|
|
|
body := string(out)
|
|
switch {
|
|
case strings.Contains(body, `"instance_id"`):
|
|
harness.Detail(t, id, "responder OK from host")
|
|
case strings.Contains(body, "tcp OPEN"):
|
|
harness.Detail(t, id, "tcp open but responder silent")
|
|
case strings.Contains(body, "tcp CLOSED"):
|
|
harness.Detail(t, id, "tcp closed/unreach (responder down OR OVN path broken)")
|
|
default:
|
|
harness.Detail(t, id, "probe inconclusive — see artifact")
|
|
}
|
|
|
|
// SSH-in via IGW; dumps responder unit + cloud-init state inside the guest.
|
|
sshIntoApp(ctx, t, fix, id, pub)
|
|
}
|
|
|
|
// sshIntoApp uploads the key to the host then SSHes into the app guest to dump
|
|
// responder, cloud-init, and network state. Best-effort: no-ops if key unavailable or SSH fails.
|
|
func sshIntoApp(ctx context.Context, t *testing.T, fix *fixture, id, pub string) {
|
|
t.Helper()
|
|
if fix.privateKeyPath == "" {
|
|
return
|
|
}
|
|
keyData, err := os.ReadFile(fix.privateKeyPath)
|
|
if err != nil {
|
|
harness.Detail(t, id, fmt.Sprintf("ssh-in: read key err: %v", err))
|
|
return
|
|
}
|
|
enc := base64.StdEncoding.EncodeToString(keyData)
|
|
remoteKey := fmt.Sprintf("/tmp/reboot-e2e-%s.pem", id)
|
|
|
|
diag := `set +e
|
|
echo "=== uname / uptime ==="
|
|
uname -a; uptime
|
|
echo "=== systemctl is-active reboot-e2e-httpd.service ==="
|
|
systemctl is-active reboot-e2e-httpd.service
|
|
systemctl is-enabled reboot-e2e-httpd.service
|
|
echo "=== systemctl status (head 40) ==="
|
|
systemctl status --no-pager reboot-e2e-httpd.service 2>&1 | head -40
|
|
echo "=== unit file ==="
|
|
ls -l /etc/systemd/system/reboot-e2e-httpd.service 2>&1
|
|
echo "=== unit file content ==="
|
|
sudo cat /etc/systemd/system/reboot-e2e-httpd.service 2>&1
|
|
echo "=== enable symlink ==="
|
|
ls -l /etc/systemd/system/multi-user.target.wants/reboot-e2e-httpd.service 2>&1
|
|
echo "=== network-online.target ==="
|
|
systemctl is-active network-online.target
|
|
systemctl status --no-pager network-online.target 2>&1 | head -20
|
|
echo "=== networkctl ==="
|
|
networkctl status 2>&1 | head -40
|
|
echo "=== ip addr ==="
|
|
ip -br addr
|
|
echo "=== ip route ==="
|
|
ip route
|
|
echo "=== port :80 listeners ==="
|
|
sudo ss -tlnp 'sport = :80' 2>/dev/null
|
|
echo "=== curl localhost ==="
|
|
curl -s -m 3 -w '\nHTTP=%{http_code}\n' http://127.0.0.1/ || echo "curl localhost FAILED"
|
|
echo "=== /var/lib/httpd ==="
|
|
ls -l /var/lib/httpd 2>&1
|
|
cat /var/lib/httpd/index.html 2>&1
|
|
echo "=== responder journal ==="
|
|
sudo journalctl -u reboot-e2e-httpd.service --no-pager -n 60 2>&1
|
|
echo "=== boot list ==="
|
|
sudo journalctl --list-boots 2>&1 | tail -5
|
|
echo "=== last-boot journal grep responder ==="
|
|
sudo journalctl -b -1 --no-pager 2>&1 | grep -i "reboot-e2e\|network-online\|cloud-init" | tail -40
|
|
echo "=== this-boot journal grep responder ==="
|
|
sudo journalctl -b --no-pager 2>&1 | grep -i "reboot-e2e\|network-online\|cloud-init" | tail -40
|
|
echo "=== cloud-init result.json ==="
|
|
cat /run/cloud-init/result.json 2>&1
|
|
echo "=== cloud-init.log tail ==="
|
|
sudo tail -80 /var/log/cloud-init.log 2>&1
|
|
echo "=== cloud-init-output.log tail ==="
|
|
sudo tail -40 /var/log/cloud-init-output.log 2>&1
|
|
echo "=== instance-id sem ==="
|
|
ls -l /var/lib/cloud/instance 2>&1
|
|
ls -l /var/lib/cloud/instances/ 2>&1
|
|
echo "=== /var/lib/cloud/instances/*/sem ==="
|
|
ls -l /var/lib/cloud/instances/*/sem/ 2>&1
|
|
`
|
|
// Single outer SSH: write the key then hop into the guest. One round trip.
|
|
outerCmd := fmt.Sprintf(`set +e
|
|
umask 077 && printf '%%s' '%s' | base64 -d > %s && chmod 600 %s
|
|
ssh -i %s -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
|
|
-o ConnectTimeout=8 -o BatchMode=yes -o ServerAliveInterval=5 \
|
|
ubuntu@%s bash -s <<'EOFDIAG'
|
|
%s
|
|
EOFDIAG
|
|
rm -f %s
|
|
`, enc, remoteKey, remoteKey, remoteKey, pub, diag, remoteKey)
|
|
|
|
out, err := fix.ssh.Run(ctx, fix.env.WANHost, outerCmd)
|
|
harness.DumpFile(t, fix.artifacts, fmt.Sprintf("diag-%s-guest-ssh.txt", id), out)
|
|
if err != nil {
|
|
harness.Detail(t, id, fmt.Sprintf("ssh-in err: %v", err))
|
|
return
|
|
}
|
|
harness.Detail(t, id, "ssh-in OK (see diag-*-guest-ssh.txt)")
|
|
}
|
|
|
|
// --- Cleanup -------------------------------------------------------------
|
|
|
|
// cleanup tears down all test resources; SG is deleted last as it can't be removed
|
|
// while instance or ALB ENIs still reference it.
|
|
func cleanup(t *testing.T, fix *fixture) {
|
|
t.Helper()
|
|
harness.Phase(t, "Cleanup")
|
|
|
|
if !fix.ssh.IsReachable(context.Background(), fix.env.WANHost) {
|
|
harness.Step(t, "cleanup skipped: node unreachable")
|
|
return
|
|
}
|
|
|
|
if fix.listenerArn != "" {
|
|
_, _ = fix.aws.ELBv2.DeleteListener(&elbv2.DeleteListenerInput{ListenerArn: aws.String(fix.listenerArn)})
|
|
}
|
|
if fix.albArn != "" {
|
|
_, _ = fix.aws.ELBv2.DeleteLoadBalancer(&elbv2.DeleteLoadBalancerInput{LoadBalancerArn: aws.String(fix.albArn)})
|
|
}
|
|
if fix.tgArn != "" {
|
|
_, _ = fix.aws.ELBv2.DeleteTargetGroup(&elbv2.DeleteTargetGroupInput{TargetGroupArn: aws.String(fix.tgArn)})
|
|
}
|
|
|
|
if len(fix.appInstanceIDs) > 0 {
|
|
ids := make([]*string, len(fix.appInstanceIDs))
|
|
for i, id := range fix.appInstanceIDs {
|
|
ids[i] = aws.String(id)
|
|
}
|
|
_, _ = fix.aws.EC2.TerminateInstances(&ec2.TerminateInstancesInput{InstanceIds: ids})
|
|
harness.WaitForInstanceTerminated(t, fix.aws, fix.appInstanceIDs, 60*time.Second)
|
|
}
|
|
|
|
_, _ = fix.aws.EC2.DeleteKeyPair(&ec2.DeleteKeyPairInput{KeyName: aws.String(keyName)})
|
|
|
|
// SG must come after instances + ALB ENIs are gone.
|
|
if fix.sgID != "" {
|
|
if _, err := fix.aws.EC2.DeleteSecurityGroup(&ec2.DeleteSecurityGroupInput{
|
|
GroupId: aws.String(fix.sgID),
|
|
}); err != nil {
|
|
var aerr awserr.Error
|
|
if !errors.As(err, &aerr) || aerr.Code() != "InvalidGroup.NotFound" {
|
|
t.Logf("delete SG %s: %v", fix.sgID, err)
|
|
}
|
|
}
|
|
}
|
|
|
|
if fix.igwID != "" && fix.vpcID != "" {
|
|
_, _ = fix.aws.EC2.DetachInternetGateway(&ec2.DetachInternetGatewayInput{
|
|
InternetGatewayId: aws.String(fix.igwID),
|
|
VpcId: aws.String(fix.vpcID),
|
|
})
|
|
_, _ = fix.aws.EC2.DeleteInternetGateway(&ec2.DeleteInternetGatewayInput{
|
|
InternetGatewayId: aws.String(fix.igwID),
|
|
})
|
|
}
|
|
if fix.subnetID != "" {
|
|
_, _ = fix.aws.EC2.DeleteSubnet(&ec2.DeleteSubnetInput{SubnetId: aws.String(fix.subnetID)})
|
|
}
|
|
if fix.vpcID != "" {
|
|
_, _ = fix.aws.EC2.DeleteVpc(&ec2.DeleteVpcInput{VpcId: aws.String(fix.vpcID)})
|
|
}
|
|
}
|
|
|
|
// --- Misc ----------------------------------------------------------------
|
|
|
|
func durationEnv(key string, def time.Duration) time.Duration {
|
|
v := os.Getenv(key)
|
|
if v == "" {
|
|
return def
|
|
}
|
|
// Accept a bare integer (treated as seconds) or a time.ParseDuration string.
|
|
if d, err := time.ParseDuration(v); err == nil {
|
|
return d
|
|
}
|
|
if d, err := time.ParseDuration(v + "s"); err == nil {
|
|
return d
|
|
}
|
|
return def
|
|
}
|