mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
feat(e2e): Go E2E harness + single-node migration to Ensure*/Discover* + reboot suite (#241)
* feat(e2e): add Go test harness + cert scenario, port run-cert-e2e.sh
Lands step 1 of docs/development/improvements/e2e-go-harness.md:
shared harness primitives (env, wait, artifacts, cert, tls) plus the
TestCertIssuance scenario that supersedes run-cert-e2e.sh. All files
gated by //go:build e2e so default builds and `make preflight` are
unaffected. Bash script stays in place for one CI cycle before deletion.
* feat(e2e): port run-lb-e2e.sh to TestLoadBalancer Go scenario
Step 3 of docs/development/improvements/e2e-go-harness.md. Replaces 1246
lines of bash with a typed Go scenario exercising all 4 LB variants
(ALB/NLB × internet-facing/internal) against a shared VPC + app instance
pool. Internet-facing phase auto-skips when no peer node is configured;
internal phase always runs.
Adds shared harness primitives:
- harness/aws.go: AWS SDK client factory wired to Spinifex AWS gateway
- harness/lb.go: WaitForLBActive, WaitForTargetsHealthy, WaitForENICleanup,
WaitForInstanceRunning/Terminated, InstanceENI
- harness/traffic.go: HTTPRoundRobin, TCPRoundRobin, distribution asserts
- harness/ssh.go: minimal peer SSH wrapper shelling out to /usr/bin/ssh
(replaced by typed x/crypto/ssh client hoisted from DDIL in mulga-siv-74)
User-data scripts embedded via //go:embed under lb/testdata/.
Bash script stays in place for one CI cycle before deletion.
* fix(e2e): log client launch params + dump daemon log on InvalidInstanceType
Adds diagnostics for the LB internal client launch path. Logs the exact
InstanceType, AMI, subnet, key name, and user-data size right before
RunInstances, and registers a daemon-log dump on failure so we can
correlate the 400 InvalidInstanceType with what spinifex saw.
* refactor(e2e): serialise LB suites + share probe client across internal tests
Splits TestLoadBalancer into 4 sequential subtests (ALB/NLB ×
internet-facing/internal). Each suite creates its own TG + LB + listener,
runs its traffic test, and tears down before the next starts. Replaces
the per-suite throwaway client VM with a single long-lived probe server
launched at fixture setup; per-suite internal traffic tests GET
/probe?ip=...&proto=... against it instead of cold-booting a fresh client.
Drops peak instance count from 5 to 4 (2 app + 1 LB + 1 client), so the
internal phase passes on capacity-constrained dev nodes that previously
hit InvalidInstanceType (see mulga-siv-76, mulga-siv-77 for the underlying
spinifex bugs).
* feat(daemon): make hostReserve env-configurable for dev capacity squeeze
Adds SPINIFEX_RESERVED_VCPU and SPINIFEX_RESERVED_MEM_GB env-var overrides
read at daemon startup. Invalid values are logged and ignored so a typo
doesn't silently widen the reserve. Unblocks the LB E2E on capacity-tight
dev hosts (8 vCPU box: default 2-vCPU reserve + 3x t3.nano test load
leaves 0 free for the sys.micro LB VM). Lays groundwork for the planned
operator-tunable [capacity] config.
bead: mulga-siv-78
* fix(e2e): revert LB to per-suite client + fail-fast on terminal LB state
Reverts the shared probe-server VM that wouldn't come up reliably during
the InternalFacing suites — back to the proven bash pattern of launching
a fresh client VM per suite (one of ALB_IP / NLB_IP populated based on
the suite kind). Adds renderClientUserData + a conditional template so
each client probes only its target LB.
WaitForLBActive now bails immediately on terminal LB states (failed,
provisioning_failed) and emits a "still waiting" log line every 30s so
hangs are visible instead of silently burning the full 5-minute timeout.
* ci(e2e): add e2e-go.yml workflow + GOTESTSUM Makefile opt-in
Standalone workflow runs the new Go-based E2E harness on the existing
ci-single self-hosted runner — provisions a tofu VM, bootstraps spinifex,
installs gotestsum, runs each requested suite via the e2e Makefile, and
uploads per-test artifacts + JUnit XML + JSON streams as a build artifact.
Test summary parses the JUnit and emits ::error file=...:: annotations so
failures surface in the PR diff.
Gated to the feat/e2e-go-harness branch + workflow_dispatch so it doesn't
interfere with the existing e2e.yml while the Go conversion lands.
tests/e2e/Makefile gains a GOTESTSUM=1 opt-in: when set, the e2e target
runs gotestsum with --junitfile and --jsonfile rooted in ARTIFACT_DIR
under a suite-tagged filename derived from the first PKG component.
* feat(e2e): share probe client across both internal LB suites + SG fixes
- Replaces per-suite probe client with a single shared VM launched in the
fixture. Saves ~60s by avoiding a cold boot between the ALB and NLB
internal suites.
- Client cloud-init exposes a JSON trigger endpoint on :9090 in addition
to the http.server on :80 (results dir). Each suite POSTs
{proto, ip, count, outfile, http_port, tcp_port} to run a probe loop
against the LB private IP and writes the result file under /tmp/httpd
for the driver to fetch back.
- configureDefaultSG now uses the structured IpPermissions form (top-level
shortcut is silently ignored by vpcd — tracked separately) and opens
port 9090 in addition to 80 + 9000.
- deleteLB captures the underlying sys.micro VM id via the LB ENI before
delete, then polls DescribeLoadBalancers until LoadBalancerNotFound so
the next suite doesn't race the capacity reclaim.
- Internal_NLB sleeps 15s up front to let the ALB sys.micro deallocate
fully before its own createLB fires. DescribeLoadBalancers can report
the ALB gone before the vCPU/memory allocation is actually released on
capacity-tight dev hosts; without this settle, NLB's createLB trips the
sys.micro reserve and lands in terminal state=failed. Still cheaper
than cold-booting a per-suite client.
* ci(e2e): install Go on tofu VM before running Go E2E suites
The tofu-provisioned VM ships with the spx binary and bash test scripts
but no go toolchain, so the prior workflow's "Install gotestsum on VM"
step failed with `bash: line 2: go: command not found`. Add a curl-based
Go 1.26.3 install (matching go.mod) ahead of the gotestsum install and
update the test step's PATH so make e2e-* runs the freshly-installed
go binary.
* ci(e2e): bump gotestsum to @latest for Go 1.26 compatibility
gotestsum v1.12.0 pins golang.org/x/tools v0.19.0 which fails to compile
under Go 1.26 with:
invalid array length -delta * delta (constant -256 of type int64)
Switching to @latest pulls a newer x/tools that builds clean.
* ci(e2e): ship spinifex source to VM so make -C tests/e2e can run there
Single-node tofu VM only has the spx binary + bash test bundle at
~/e2e-tests; the Go e2e workflow was running 'make -C tests/e2e e2e-cert'
from $HOME/Development/mulga/spinifex which doesn't exist, producing:
make: *** No rule to make target 'e2e-cert'. Stop.
Add a checkout of the spinifex repo on the runner and rsync it (minus
.git, build/, bin/, node_modules) to the VM as
~/Development/mulga/spinifex before installing Go + gotestsum.
Interim solution; mulga-siv-80 tracks replacing the rsync with shipping
pre-built test binaries from the runner.
* ci(e2e-go): replace rsync with tar-over-ssh; VM lacks rsync
bootstrap-install.sh ships a minimal binary VM with no rsync; the
ship-source and collect-artifacts steps failed with
'rsync: command not found'. Stream a tarball over ssh instead — same
exclude list, no extra dependency on the VM.
* fix(e2e): discover single-node IPs + awsgw bind in LoadEnv
run-cert-e2e.sh single mode probes awsgw.host from spinifex.toml and
adds every non-loopback global IPv4 to the node list. The Go harness
defaulted to 127.0.0.1 only, so CI on a tofu VM (awsgw bound to the
LAN IP, not loopback) saw 'connection refused' on every TLS check.
Port the same discovery: ModeSingle now walks net.InterfaceAddrs for
NodeIPs and grep-parses spinifex.toml for the awsgw bind. Both stay
overridable via SPINIFEX_NODE_IPS / SPINIFEX_SERVICE_IPS.
* ci(e2e-go): pre-build test binaries on runner; drop go toolchain from VM
Replaces the source-tree + go + gotestsum payload that the e2e-go.yml
workflow previously shipped to the tofu CI VM with self-contained
test binaries built on the runner via `go test -c`. testdata for the
LB suite is already //go:embed'd into the binaries, so the bundle on
the VM is just `<suite>.test` files (cert: 14M, lb: 17M uncompressed).
Workflow changes
- actions/setup-go on runner.
- New step builds binaries via `make e2e-build SUITES_BUILD=...`.
- Install go-junit-report on runner; convert each VM-side test log to
junit-<suite>.xml after artifact pull so "Publish test summary" still
parses JUnit XML.
- Drop "Install Go + gotestsum on VM" entirely (saves ~60s + network).
- Drop source-shipping rsync/tar (saves ~30s + a lot of bandwidth).
- VM step runs `./<suite>.test -test.v -test.timeout=30m` directly.
Makefile changes
- New `e2e-build` target compiles each `SUITES_BUILD` entry into
`BIN_DIR/<suite>.test` via `go test -c -tags=e2e`. `clean` also
removes BIN_DIR.
Closes mulga-siv-80.
* ci(e2e-go): disable setup-go cache; upload dominates self-hosted run time
* refactor(e2e): hoist generic DDIL harness primitives to shared tests/e2e/harness
Step 1.5 of docs/development/improvements/e2e-go-harness.md. Promotes
Cluster/Node/SSH/DaemonClient/LocalStateRevision/ResetAllNodes/Witness
into the shared package so the LB, multinode, and BM scenarios in
steps 4-5 can build on the same primitives instead of duplicating SSH
+ cluster plumbing.
Moves
- ddil/harness/{cluster,ssh,daemon_client,state,reset,witness}.go
-> harness/ (plus the witness testdata embed dir).
- ddil/harness/{fault,netem,profiles,snapshot}.go -> ddil/fault/ as a
new package; fault.PartitionNode/HealNode/KillNATS/StartNATS/
RestartDaemonOnly/ApplyNetem/ClearNetem/LinkProfile*/TakeSnapshot
now take harness.Cluster / harness.Node / harness.SSH explicitly.
- ddil/harness/retry.go::Run retry-with-reset extracted as
harness.RetryWithReset in harness/wait.go; ddil/harness/quarantine.go
keeps the DDIL-only quarantine wrapper that calls into it.
- WaitForMode moved from fault.go to harness/daemon_client.go since
it polls daemon state and has no fault-injection concern.
- The pre-hoist harness/ssh.go (PeerSSH, subprocess ssh) is preserved
as harness/peer_ssh.go so the lb_test suite keeps working unchanged;
the typed x/crypto/ssh client from DDIL now owns harness/ssh.go.
- New harness/util.go exports ShellQuote (was unexported shellQuote in
netem.go) so both ddil/fault and reset.go/witness.go can use it.
Env var rename
- SPINIFEX_NODES / SPINIFEX_SSH_USER / SPINIFEX_SSH_KEY replace
DDIL_NODES / DDIL_SSH_USER / DDIL_SSH_KEY. Old names still honoured
with a stderr deprecation note (envWithLegacy) so the DDIL CI keeps
passing until .github/workflows/e2e-ddil.yml drops the old names.
DDIL_QUARANTINED stays DDIL-specific.
DDIL parity preserved — scenarios still call ddilh.Run for quarantine
semantics; the underlying retry-with-reset is the same code path,
just shared.
Closes mulga-siv-74.
* fix(e2e): open default SG :22 ingress via structured form before witness launch
The daemon currently drops the top-level shortcut AuthorizeSecurityGroupIngress
form silently (mulga-siv-79), so the default SG starts with no inbound :22 and
the witness EIP awaitBaseline tunnel times out under DDIL CI. Mirror the
workaround already present in lb_test by issuing the structured IpPermissions
form before LaunchWitnessVM. Idempotent on InvalidPermission.Duplicate so reruns
in the same cluster are safe.
* ci: render JUnit into job summary + collapsible groups in noisy steps
Add a reusable composite action (.github/actions/junit-report) that parses
a glob of JUnit XML files, writes a totals table + collapsible per-failure
details into $GITHUB_STEP_SUMMARY, and emits one ::error annotation per
failed/errored test case so failures surface in the run header instead of
being buried in step logs.
Apply the action to both e2e-go.yml and e2e-ddil.yml (e2e-ddil gotestsum
now also emits --junitfile so the action has input).
Same workflows also wrap noisy phases (tofu apply/destroy, scrub-env,
tar-over-ssh, per-suite test runs, cluster-log collection) in
::group::/::endgroup:: markers so the Actions UI collapses the chatter
by default and the human can expand only the failing phase. Tofu output
still tees to $TOFU_LOG so the artifact bundle is unchanged.
Future workflows (multinode, baremetal) can adopt the same pattern via
the README in .github/actions/junit-report.
* test(ddil): quarantine Scenario C until predastore §2 lands
CI run 26016840819 surfaced that ScenarioC_CleanPartition silently
depends on predastore-ddil-hardening §2 (write-quorum with deferred
parity repair). Root cause from node3 daemon log:
predastore: write udp ...->192.168.0.22:9991: sendmsg: operation not permitted
predastore: putObjectViaQUIC: put failed
spx QMP: BLOCK_IO_ERROR instanceId=<witness>
spx: StopOrTerminateInstance: Terminating
spx: Instance state transition running → shutting-down
The 'sendmsg: operation not permitted' is the EPERM Linux returns when
iptables drops an outbound packet — exactly the OUTPUT DROP rules
PartitionNode installs. The witness VM's block I/O is backed by
distributed predastore, so the same partition that severs node3↔peer
NATS traffic also severs the witness disk's cross-shard QUIC writes.
QEMU power-downs the guest after BLOCK_IO_ERROR, the daemon's
reconciler terminates the instance, and the next AssertProgressed dials
a sshd that's already gone (hence the 'handshake failed: EOF' the test
reports).
Quarantine by collapsing C to the same scenarioSkip stub D/E/F already
use, citing predastore-ddil-hardening §2 as the dependency. Flip C
back to ENABLED in the same PR that lands §2 — that's when the witness
disk can survive a peer partition without QEMU halting.
TEST_COVERAGE.md row C flipped ENABLED → SKIPPED; added a quarantine
note explaining the predastore dependency.
* ci: colored phase banners + quiet-on-success tofu output
Pretty up the live Actions log without flooding it. Three changes:
1. .github/scripts/ci-fmt.sh — sourceable bash helpers (banner, ok, bad,
warn, note, quiet_run) using ANSI colors that the Actions UI renders.
2. e2e-go.yml + e2e-ddil.yml — tofu phases switch back to quiet redirect
into $TOFU_LOG (the previous tee'd version regressed live noise vs
the original silent redirect). Each phase now prints one colored
ok/✗ line; on failure the tail of $TOFU_LOG dumps inside a single
collapsed group, so the live log stays scannable but the failing tail
is one click away.
3. junit-report render.py — compact summary: short status icons (✅/❌)
in a single column, no separate err column, single-line failure list
(truncated to 120 chars) instead of <details> blocks that wrap badly
on narrow screens.
Per-suite test runs still wrap in ::group:: with an inline cyan banner
so pass output collapses and the suite name is obvious when expanded.
* test(e2e): add harness.Phase / Step / Detail for human-readable test output
Test fixtures were emitting flat walls of t.Logf with every line prefixed
by the same `lb_test.go:61:` (file:line of the call site that the
t.Helper() chain bubbles up to) and forced 6-space indent. Hard to skim,
hard to find where one phase ends and the next begins.
Add three helpers in tests/e2e/harness/log.go:
Phase(t, fmt, ...) — flush-left colored banner via fmt.Println.
Bypasses t.Logf so the line lands without the
testing framework's indent/file:line prefix.
Pure visual structure; the diagnostic detail
stays on t.Logf.
Step(t, fmt, ...) — single-line sub-phase marker via t.Logf.
Keeps file:line attribution so a triager can
find the source of a progress message.
Detail(t, k, v, ...) — key=value resource IDs via t.Logf. Same
attribution rationale as Step — IDs are
diagnostic and want to land in JUnit XML
bucketed with the owning test.
Why the split: Phase is purely a banner; losing JUnit attribution costs
nothing because banners aren't diagnostic. Step/Detail carry triage
context (instance IDs, AMI, etc.) and a failing test's JUnit testcase
node should still contain them, so t.Logf attribution is kept.
ANSI colors are auto-disabled when stdout isn't a TTY and GITHUB_ACTIONS
isn't set, so piped local `go test -v` stays clean.
Apply to lb_test setupSharedFixture — five Phase banners replace the
implicit run of helper logs ("Discovering cluster capacity",
"Ensuring SSH key pair", "Creating shared VPC topology",
"Launching app instances", "Launching probe client") so the live CI log
reads as a story rather than a flat resource dump.
* ci(e2e-go): strip indent + file:line on display, propagate color env over SSH
Two display bugs spotted on the previous run:
1. Phase banners came out white instead of bold-cyan because the test
binary runs over SSH on the tofu VM, where neither GITHUB_ACTIONS
nor a TTY is present, so harness.log.go init() disables ANSI escapes.
Fix: export GITHUB_ACTIONS=true and FORCE_COLOR=1 in the remote env
so the helper keeps emitting color codes.
2. Every t.Logf line still rendered as
lb_test.go:61: instance type: t3.nano
(4-space indent + `<file>:<line>:` prefix) because t.Helper()
bubbles file:line to the call site. Fix: pipe the test binary's
stdout through `sed -E 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'`
so the display strips both. Tee to the artifact log happens BEFORE
the sed so raw output is preserved for triage; only the live UI
sees the cleaned form.
* feat(e2e-harness): extend harness for single-node Go suite port
Stage A of mulga-siv-89 — add the primitives the single-node run-e2e.sh
port consumes:
- AWSClient gains IAM + STS service handles, plus NewAWSClientWithCreds
for per-tenant account-scoping in Phase 8.
- awserror.go: ErrorCodeIs, AssertAWSError, ExpectError — Go equivalent
of the bash expect_error helper used throughout the negative phases.
- poll.go: WaitForInstance/Volume/Snapshot/Image state to any target,
with per-resource default timeouts tuned from the bash sleep loops.
- spxcli.go: thin shellouts for spx get nodes/vms, spx top nodes,
spx admin account create with output parsing for tenant-id/access-key.
- rawhttp.go: signed raw POST against the AWS gateway for the Phase 8
InvalidAction probe (gateway rejects on auth before action parsing).
- ovn.go: OvnNbctl/OvnSbctl wrappers + SkipIfNoOVN for Phase 8b-e
datapath checks; WaitForPortGroupMember handles propagation polls.
- ec2helpers.go: InstancePublicSSHHost (public-IP-or-qemu-hostfwd),
LsblkRootGiB, serial-console-access trio, DiscoverDefaultVPC,
idempotent AuthorizeSSHIngress.
- account.go: AccountCarousel maps tenant name -> scoped AWSClient,
built atop NewAWSClientWithCreds for the Phase 8 isolation tests.
Pure groundwork — no new tests, no production behaviour change. The
cert + lb scenarios still compile against the extended harness.
* ci(e2e-go): auto-cancel + reap orphans to prevent runner wedge
A cancelled run on the single-runner ci-single host has been leaving
orphan tofu/terraform procs (and stale tfstate locks) behind, which
caused the next push's tofu destroy to hang for 45m+ and saturate the
runner host until sshd refused new connections.
- Add `concurrency: e2e-go-${ref}, cancel-in-progress: true` so a new
push to feat/e2e-go-harness cancels the in-flight run instead of
stacking on the same self-hosted runner.
- New `Reap Orphan Procs` step at job start: SIGTERM then SIGKILL any
lingering `tofu`/`terraform` from the runner user, and unlink any
stale `.terraform.tfstate.lock.info`.
- Harden `Teardown`: drop `set -e` (scrub-env must always run as the
libvirt-level fallback), reap orphans before destroy, and wrap
`tofu destroy` in `timeout 600` so a wedged provider can't pin the
job to its 45m timeout.
* ci(e2e-go): drop push trigger, manual dispatch only
Pull the on.push branch trigger so the workflow only fires from
workflow_dispatch — matches the e2e-ddil.yml Phase-1 pattern. Push
trigger lifts again once the single-node Go port (mulga-siv-89) is
green end-to-end and run-e2e.sh is retired.
* test(e2e-single): port run-e2e.sh phases 1-4 to Go
Stage B of mulga-siv-89. Lands the TestSingleNode orchestrator + Fixture
that threads state between sequential phase subtests, plus phases:
- Phase 1: KVM check, AWS gateway TLS reachability, daemon NATS
readiness polling, region/AZ smoke.
- Phase 1b: spx get nodes / top nodes / get vms (single-node only).
- Phase 2: nano instance-type + arch discovery.
- Phase 2b: serial-console-access default/enable/disable round-trips.
- Phase 3: create-key-pair + import-key-pair lifecycle; test-key-1 PEM
kept for Phase 5 instance launch (cleanup deferred to teardown phase
in stage G).
- Phase 4: architecture-aware AMI discovery via name filter
ami-ubuntu-24.04-{x86_64,arm64}, falls back from spx import output.
Phase ordering is contractual (Phase 5+ depend on AMI, key pair, AZ).
detectPoolMode reads [network] external_mode from spinifex.toml so
stages F/G can skip pool-only phases on dev_networking fixtures.
No production code changed. cert + lb scenarios continue to compile.
* test(e2e-single): port run-e2e.sh phases 5-5e to Go
Stage C of mulga-siv-89 — the instance/volume/snapshot/image lifecycle
that everything else hangs off:
- Phase 5: run-instances (default VPC + default SG with tcp/22 opened
via idempotent harness.AuthorizeSSHIngress), wait running, resolve
root volume from BlockDeviceMappings.
- Phase 5a-pre: spx get vms now contains the running instance.
- Phase 5a: describe-instances metadata round-trip (type/key/AMI/BDMs).
- Phase 5a-ii: TCP wait, ssh id, lsblk cross-check vs API root volume
size (±1 GiB tolerance — lsblk rounds down), soft hostname check.
- Phase 5a-iii: get-console-output round-trip on InstanceId.
- Phase 5b: create-volume (10 GiB) → modify-volume (20 GiB) →
attach-volume → detach-volume → delete-volume, polling each state
transition through harness.WaitForVolumeState.
- Phase 5b-ii: describe-volume-status on the root volume.
- Phase 5c: create-snapshot → wait completed → copy-snapshot → both
visible → delete original → copy survives → delete copy.
- Phase 5d: AMI snapshot-backed launch verification via
DescribeImages.BlockDeviceMappings[].Ebs.SnapshotId. Predastore-side
config.json reads are TODO (no S3 client in harness yet).
- Phase 5e: create-image --no-reboot, wait available, capture backing
snapshot ID for Phase 9 cleanup.
Phase 7a (attach to stopped → error) stubbed in volume_test.go for
Stage D to fill once the lifecycle stop/start lands.
* test(e2e-single): port run-e2e.sh phases 5f-8 to Go
Stage D of the single-node Go port (mulga-siv-89). Adds four files
under tests/e2e/single/ covering Phase 5f (SG egress ACL enforcement),
Phase 6 (tag CRUD), Phases 7 / 7a / 7b / 7c-pre / 7c (stop, start,
attach-to-stopped negative, ModifyInstanceAttribute, reboot, multi-count
RunInstances), and Phase 8 negatives (18 sub-tests asserting AWS error
codes via harness.ExpectError / AssertAWSError / PostAWSAction).
Wired into TestSingleNode in main_test.go; phase ordering preserved
because Phase 7 chain mutates the shared instance state (stopped at end
of 7, .small after 7b, running after 7c-pre, untouched by 7c siblings)
and Phase 8 negatives need a running instance with the AMI/key created
earlier.
Removes the placeholder phase7a_AttachToStoppedError stub from
volume_test.go (real impl now in lifecycle_test.go).
run-e2e.sh stays in tree until step-7 cleanup per the parent plan.
* ci(e2e-go): wire e2e-single target + add single to build set
Lets the in-progress single-node Go port (mulga-siv-89, Stages A-D
landed) be exercised in CI via explicit `suites: e2e-single` dispatch
input ahead of Stage G flipping the workflow default.
Workflow default suites unchanged until Phase 9 teardown lands.
* test(e2e-single): port run-e2e.sh IAM 1-7 + Phase 8 account scoping to Go
Stage E of the single-node Go port (mulga-siv-89). Adds two files
under tests/e2e/single/:
- iam_test.go (903 lines): IAM Phases 1-7 — User CRUD, Access Key
Lifecycle, User Authentication (via harness.NewAWSClientWithCreds),
Policy CRUD (+ CreatePolicyVersion / SetDefaultPolicyVersion /
DeletePolicyVersion which bash lacks), Policy Attachment &
Enforcement (allow / deny / wildcard / root bypass), Policy Lifecycle
(DeleteConflict while attached), and idempotent Cleanup.
- account_test.go (1186 lines): Phase 8 EC2 Account Scoping — 11 Steps
(Account Setup, Instance / Volume / KeyPair / Snapshot / VPC-Subnet /
IGW-EIGW Scoping, Account Settings, Global Resources, Edge Cases,
Cleanup). Uses harness.AccountCarousel for the multi-profile
credential map and SpxAdminAccountCreate for admin account
bootstrap.
Fixture extended with IAM cross-phase state (IAMAdminAccount,
IAM{Alice,Bob,Charlie}{KeyID,Secret}) so Phase 7 cleanup is defensive
even when an intermediate IAM phase short-circuits. All 11 Phase 8
Steps share local state in the outer function; t.Cleanup mirrors
Step 11 LIFO so a mid-Step failure still tears down accounts.
Wired into TestSingleNode. run-e2e.sh stays in tree until step-7
cleanup per parent plan.
* fix(e2e-harness): use kebab-case is-default filter for DescribeVpcs
AWS EC2 DescribeVpcs filter names are kebab-case (is-default), and the
daemon's filter allowlist enforces that. The harness sent the camelCase
SDK field name (isDefault), so the daemon returned InvalidParameterValue
400 and DiscoverDefaultVPC fataled — leaving Phase 5 InstanceID empty
and cascading 16 downstream Single-Node E2E subtests to fail.
* test(e2e-single): port run-e2e.sh phases 8b-8e to Go
Stage F of the single-node Go port (mulga-siv-89). Adds three files
under tests/e2e/single/:
- vpc_test.go (553 lines): Phase 8b (pool-mode VPC public/private
subnet E2E — CreateVpc, dual subnets with MapPublicIpOnLaunch
assertions, IGW attach, custom route table + 0.0.0.0/0 route,
instance launch + SSH + external connectivity probe) and Phase 8c
(route table validation — main RT association, local + IGW route
presence, custom RT CRUD + association/disassociation, negative
delete-local + delete-main assertions).
- natgw_test.go (467 lines): Phase 8d (pool-mode NAT GW E2E —
CreateNatGateway with EIP, private subnet route via NAT, bastion-
hopped SSH into private VM, baseline egress drop / post-NAT egress
succeed / post-teardown egress lost). Inline waitForNATGatewayState
helper bounds the pending->available->deleted state machine via
harness.EventuallyErr.
- datapath_test.go (326 lines): Phase 8e (OVN-gated SG-to-SG datapath
— two SGs with UserIdGroupPair ingress rule, OVN port-group
membership + SB address-set wait loops, allowed traffic curl,
denied traffic nc, revoke->drop, re-add->restore).
Pool-mode gating via fix.PoolMode for 8b/8d; OVN gating via
harness.SkipIfNoOVN for 8e. Per-phase t.Cleanup LIFO ensures
teardown unwinds even on partial failure. Wired into TestSingleNode.
* test(e2e-single): port run-e2e.sh phases 9, 9a, 9b to Go
Stage G of the single-node Go port (mulga-siv-89) — final teardown +
post-teardown verification. Adds tests/e2e/single/teardown_test.go
(226 lines):
- phase9_Teardown: DeleteSnapshot(CustomAMISnapID) -> DeregisterImage
(CustomAMIID) -> TerminateInstances(InstanceID) + wait terminated ->
DeleteKeyPair(KeyName). Each call swallows *NotFound via
harness.ErrorCodeIs; unexpected errors logged as warnings so 9a/9b
always run.
- phase9a_VerifyTeardown: DescribeInstances must show terminated or
NotFound; DescribeImages must NOT include the deregistered AMI;
DescribeKeyPairs must NOT include the deleted key. Clears the
consumed Fixture fields.
- phase9b_FinalClusterStats: harness.SpxGetVMs must NOT list the
terminated InstanceID; orphan i- rows surface as soft Logf warnings
to avoid flakes when parallel suites share the node.
Wired into TestSingleNode. With this, mulga-siv-89's single-node Go
port (Stages A-G) is feature-complete; run-e2e.sh remains in tree
until a separate cleanup bead deletes it after CI burn-in.
* ci(e2e-go): unbuffer sed in test pipeline for live log streaming
GNU sed defaults to block-buffered stdout when stdout is a pipe
(GitHub Actions runner). Accumulates 4-64KB before flushing, so
test output appeared in sporadic batches unaligned to test
boundaries. -u flag forces line-buffered output so each go test -v
line surfaces in the Actions UI as it is produced.
Side effect: none. No process count change, no SSH session change,
no test execution change. Pure output flushing on the runner host.
* fix(e2e-single): persist key PEM across phases and skip unimplemented IAM ops
Three bugs blocked the Single-Node Go E2E suite after the DescribeVpcs
filter fix:
1. Phase 3 wrote the SSH PEM into `t.TempDir()`, which Go cleans up when
the Phase 3 subtest exits. Phases 5a-ii / 5a-iii / 7b / 7c-pre then
saw "Permission denied (publickey)" / "Identity file not accessible"
and the cascade left the instance in a state that broke 7c-pre's
`start-instances`. Move the PEM under a TestSingleNode-scoped
`fix.TmpDir` so it survives until the whole test exits.
2. IAM Phase 2's `require.NotEmpty(t, fix.IAMAdminAccount, "")` was
intended as a no-op (Phase 4 populates the field), but require.NotEmpty
fails on empty regardless of message. Removed the bogus check.
3. The daemon does not yet implement `UpdateUser` (IAM Phase 1) or the
policy-version family (IAM Phase 4): `CreatePolicyVersion`,
`ListPolicyVersions`, `SetDefaultPolicyVersion`, `DeletePolicyVersion`.
They return `InvalidAction` and abort the phase. Wrap each block in a
t.Run sub-test with t.Skip so the surrounding phase still runs and
the skips show up explicitly in the test report.
* test(e2e): parallelise hot phases + tune fast-path polls
Squash merge of feat/e2e-parallel-quickwins (mulga-siv-96). Four
quick wins from decomposing run-e2e.sh into Go phase subtests:
- Phase 8 negatives (single/negative_test.go): t.Parallel() inside
each of the 18 read-only error-path sub-tests. ~5-10x speedup.
- IAM Phase 4 policies (single/iam_test.go): EC2ReadOnly stays
seeded sequentially (populates fix.IAMAdminAccount); the other
four policies fan out as siblings under a wrapping
create_policies_parallel t.Run. ~5x on the Phase 4 sub-step.
- Volume + snapshot fast paths (single/{volume,snapshot}_test.go):
pass harness.WithPoll(500ms) on the five state transitions that
finish in <1s on local QEMU. Halves Phase 5b / 5c wall-clock.
- Cert per-IP loop (cert/cert_test.go): t.Parallel() inside the 4
inner t.Run(ip, ...) blocks. Cuts cert phase by len(nodes).
No harness changes, no Fixture changes, no test-execution-order
changes. mulga-siv-96.
* test(e2e-lb): resolve peer availability before shared fixture setup
setupSharedFixture builds a VPC + 2 app VMs + probe client (minutes
on capacity-tight dev hosts) and previously ran BEFORE the peer
pickup + ssh.Ping. On single-node, the InternetFacing_* subtests
would correctly t.Skip on peer == "" — but only after the fixture
work and once subtests started, so the wait appeared to be attached
to the first subtest name gotestsum printed.
Reorder: pickPeer + ssh.Ping first, log "internet-facing will skip"
immediately, then build the fixture for the internal subtests.
No behavior change for multi-node + peer-reachable; on single-node
the skip rationale surfaces before the fixture wait.
* test(e2e-single): extend SSH wait budget to 3m on slow boot paths
CI run 26034322018 timed out Phase5a_ii_SSH (60s budget) and
Phase8b_VPCSubnetE2E (2m budget) on the TCP-port-open poll while
sshd was still finishing pam/cloud-init. Bump all four wait sites
to 3m as a stop-gap:
- single/instance_test.go phase5aii_SSHProbe
- single/lifecycle_test.go waitForSSHReady (Phase 7b/7c/7c-pre)
- single/vpc_test.go phase8b_VPCSubnetE2E
- single/natgw_test.go phase8d bastion SSH wait
Proper fix tracked under mulga-siv-101 (hoist a harness.WaitForSSH
that probes the actual handshake, not just tcp/22 reachability).
* test(e2e-single): skip-gate IAM3/IAM5/IAM6 until daemon handlers land
CI run 26034322018 surfaced three IAM daemon gaps:
- IAM3_UserAuthentication: alice's freshly-created access key returns
403 from DescribeInstances — daemon doesn't honour scoped IAM key
signatures yet.
- IAM5_PolicyAttachmentEnforcement: cascades from IAM3 (alice can't
sign).
- IAM6_PolicyLifecycle: DetachUserPolicy returns 404 even on
attachments Phase 5 confirmed — daemon attachment ledger gap.
Skip-gate each phase with t.Skip + a pointer to mulga-siv-100 so the
remaining IAM coverage (IAM1/IAM2/IAM4/IAM7) keeps surfacing
regressions, and the suite as a whole doesn't fail on known gaps.
Flip to real assertions once the daemon handlers land.
* test(e2e-single): probe full SSH handshake (not just TCP port open)
CI run 26036025293 hit "Connection timed out during banner exchange"
on Phase5a-ii / 8b / 8d even with the 3m TCP-port-open budget: tcp/22
opens immediately, but sshd is still finishing pam/cloud-init, so the
first real runSSH call afterwards races the half-open daemon.
Switch waitForSSHReady to issue an actual ssh BatchMode + ConnectTimeout
handshake against the target, retrying until the daemon completes the
banner exchange. Drop the four TCP-only Eventually loops in favour of
the upgraded helper:
- instance_test.go phase5aii_SSHProbe
- lifecycle_test.go waitForSSHReady (Phase 7b / 7c-pre / 7c)
- vpc_test.go phase8b_VPCSubnetE2E
- natgw_test.go phase8d_NATGateway (bastion)
Signature: waitForSSHReady(t, host, port, keyPath). Backwards-compat
alias waitForSSHHandshake exposed for the instance_test caller. Proper
harness hoist tracked under mulga-siv-101.
* test(e2e-harness): dual-emit Step + Detail for live CI streaming
go test's testing framework buffers t.Logf output per-test and releases
it only on PASS/FAIL. Long subtests (Phase8Acct_AccountScoping ~115s,
Internal_NLB ~60s, Phase8b_VPCSubnetE2E 2-3m, Phase8d_NATGateway ~90s)
therefore went silent in CI for the entire subtest duration, even with
sed -u line-buffering in the workflow pipeline.
harness.Phase already wrote to os.Stdout directly for this reason.
Extend the pattern: have Step and Detail dual-emit — write the same
line to os.Stdout (streams live) AND t.Logf (keeps JUnit per-test
attribution). The duplicate on a failure is acceptable; both copies
carry identical content.
mulga-siv-103 (note: numbering skipped past siv-102 which was already
filed in the parent branch).
* test(e2e-harness): drop t.Logf duplicate from Step + Detail
The dual-emit (Fprintln + t.Logf) caused every Step/Detail line to
appear twice after long subtests: once live via stdout, once again
when the testing framework released its per-test buffer at
PASS/FAIL. Most visible after lb subtests (Internal_NLB ~60s) where
the bulk release dumped ~dozens of duplicated lines at once.
Drop the t.Logf copy. go-junit-report parses stdout between
`=== RUN` and `--- PASS/FAIL` markers, so per-test attribution in
JUnit XML is preserved without the duplicate.
Refs: mulga-siv-103
* test(e2e-harness): stream lb-waiter progress live in CI
WaitForLBActive, WaitForTargetsHealthy, WaitForENICleanup, and
WaitForInstanceRunning emitted progress via t.Logf, which the
testing framework buffers per-test. After the previous commit
dropped the dual-emit, that meant every poll loop went silent
until subtest end — visible in CI as 'nlb is active' hanging on
screen for the whole duration of the subsequent traffic check.
Route progress through harness.Step (direct stdout). Also emit a
'waiting for X' Step at the start of each helper so observers
know what the silent stretches actually represent.
Refs: mulga-siv-103
* test(e2e): dump VPC/IGW datapath diagnostics on Phase8b SSH timeout
Phase8b's 3min SSH timeout on a fresh-VPC VM (CI run 26037892507) is
the customer-visible surface of mulga-siv-105 — vpcd's handleIGWAttach
returns before ovn-northd compiles + the gateway chassis installs
flows. Two other cleanup-only bugs (siv-106 missing spinifex:cidr,
siv-107 dnat_and_snat double-delete) showed up in the same triage.
Keep the test hard-failing so the regression stays visible, but dump a
triage bundle on timeout so each occurrence is actionable:
- guest console output (last 4KB — boot + cloud-init state)
- ovn-nbctl show + ovn-sbctl show
- Logical_Router external_ids (confirms siv-106 reproduction)
- NAT rules (confirms siv-105 + siv-107 surface)
- Port_Binding chassis,up (confirms gateway port up state)
Diagnostics go straight to stdout (Step/Detail rationale) so they
stream live in CI rather than dumping at PASS/FAIL.
Refs: mulga-siv-105, mulga-siv-106, mulga-siv-107
* ci(junit-report): drop parent t.Run containers from tally + list skips with reasons
GitHub Actions test summary was counting Go's top-level Test* parent
plus its failing subtest as two distinct failures (the parent is just
a rollup of its children). The summary now de-dupes by dropping any
test case whose name is a strict prefix of another's, so the per-file
counts and the Failures block both match the actual leaf-test
outcome. CI run 26037892507 was reporting '2 failed' for single when
it should have been 1.
Also adds a collapsible Skipped section listing each skipped leaf test
with its t.Skipf call-site reason (the .go:<line>: line from the
CDATA). ANSI escapes from harness.Phase banners are stripped so the
reason is plain text. Reasons are useful triage signal — current run
has 9 skips spread across daemon gaps (mulga-siv-100) and pool-mode
gates (InternetFacing_*).
Refs: mulga-siv-89
* ci(e2e-go): propagate cancel to remote .test procs + nuke VM before destroy
Two related leaks made cancelled runs strand QEMU work on the libvirt host:
1. The long `ssh tf-user@VM "for SUITE in …; do ./$SUITE.test"` ran
without -tt, so when the runner cancelled the step the local ssh
died but sshd never sent SIGHUP through (no PTY). The remote
`.test` binary orphaned and kept running until its own 30m
`-test.timeout`, holding tmp dirs, NATS connections, and qcow2 file
handles.
2. Teardown went straight to `tofu destroy` without first killing any
stragglers on the VM, so destroy raced the .test's open file handles
on the root disk — sometimes successfully, sometimes leaving the
libvirt host with a half-released qcow2.
Fix:
- Allocate a PTY on the run-suites ssh (`-tt`) so sshd HUPs children
when the local ssh dies.
- Add a trap (EXIT INT TERM) in the run-suites step that ssh's back
to the VM and pkill's `/spinifex-e2e-bin/*.test` belt-and-braces.
- Mirror the same pkill at the top of Teardown so destroy runs against
a quiesced VM.
Both cleanups are scoped to the per-run wan_ip — safe to land while
multiple ci-single runner agents run concurrent jobs against different
tofu envs on the same host (we deliberately keep that parallelism).
Refs: runner host wedged 2026-05-19 (post-cancel orphans suspected;
no journal access yet to confirm).
* ci(e2e-go): scope orphan reap to GITHUB_WORKSPACE so it spares sibling runners
Reap Orphan Procs (job start) and the in-Teardown pkill block were using
`pkill -u $UID_SELF -x tofu/terraform` plus `find $HOME -maxdepth 6 -name
.terraform.tfstate.lock.info`. The single host runs 5+ self-hosted runners
all as `tf-user`, so every e2e-go run was SIGKILLing in-flight tofu from
sibling workflows (e2e.yml, e2e-ddil.yml, e2e-baremetal.yml, ...) and
deleting their state-lock files — exactly the kind of cross-runner
collateral the reap was meant to recover from.
Scope by `/proc/$pid/cwd` starting with $GITHUB_WORKSPACE; each runner's
workspace dir is unique on the host, so only this run's own tofu procs
match. Find for stale lockfiles is likewise scoped to $GITHUB_WORKSPACE.
* ci(e2e-go): bump setup-go v5->v6, upload-artifact v4->v7
GitHub now warns Node 20 actions are deprecated. e2e-go.yml lagged the
rest of the repo; bump to match versions already in use:
- actions/setup-go@v5 -> @v6 (matches spinifex.yml)
- actions/upload-artifact@v4 -> @v7 (matches release.yml, e2e-ddil.yml)
checkout@v6 and create-github-app-token@v3 are already current across
all workflows. The local junit-report action is `using: composite`
(shell-only), so the Node runtime change does not apply.
* fix(vpcd): add flows-ready barrier after IGW attach (mulga-siv-105)
handleIGWAttach wrote the external switch + gateway port + LRP + default
route to OVN NB and returned immediately, before ovn-northd had compiled
NB -> SB and before ovn-controller on the gateway chassis had installed
the resulting flows. A subsequent RunInstances -> vpc.add-nat could
complete while the datapath was still dark; the launched VM was then
unreachable on its public IP for whatever flow-install latency the
chassis happened to incur. Customer-visible as 'I just created a VPC,
launched a VM, and can't SSH in' for ~minutes.
Reproduced deterministically in CI run 26037892507 Phase8b: a VM in
a fresh VPC was unreachable at 192.168.0.202:22 for 3+ minutes, while
a VM at the same public IP in the warm default VPC (Phase8d) was
reachable in ~40 seconds.
Block at the end of handleIGWAttach on `ovn-nbctl --wait=hv sync`,
which bumps NB_Global.nb_cfg and waits for every connected chassis to
acknowledge the new sequence number — i.e. flows installed. Bounded by
--timeout=30; overruns log a WARN and continue (a 30s OVN stall is
already a deeper problem than failing VPC create would surface).
waitForFlowsHV is a package var so tests can stub it; in environments
without ovn-nbctl on PATH the shell-out fails fast and the WARN path
returns nil, so existing IGW tests stay green without a live OVN.
Refs: mulga-siv-105
* fix(vpcd): store spinifex:cidr on VPC create (mulga-siv-106)
handleVPCCreate built the OVN logical router with only spinifex:vpc_id
and spinifex:vni in ExternalIDs. Cleanup paths (handleIGWDetach + NAT
sweeps) read external_ids['spinifex:cidr'] to scope SNAT removal to
the VPC's own CIDR; when unset, they fell back to an overbroad
10.0.0.0/8 and emitted a WARN.
reconcileVPC already stored the same key at startup; the live-create
path drifted. Add the field to handleVPCCreate, log the cidr alongside
vpc_id, extend TestTopologyHandler_VPCCreate to assert the external_id
is set.
Refs: mulga-siv-106
* fix(vpcd): make handleDeleteNAT idempotent on 'already absent' (mulga-siv-107)
Instance termination fans out through multiple cleanup paths
(instanceCleanerAdapter.ReleasePublicIP AND
VPCServiceImpl.DeleteNetworkInterface), both of which publish
vpc.delete-nat for the same (vpc, logical_ip). The publish is
fire-and-forget (utils.PublishNATEvent uses PublishEvent for
vpc.delete-nat, not RequestEvent), so callers never branch on the
response. The second publisher then triggered a spurious
'ERROR vpcd: failed to delete dnat_and_snat rule ... not found' log
line ms after the first delete succeeded — confusing triage signal
with no actionable consequence.
Add ErrNATNotFound sentinel to the OVN client interface and wrap the
not-found return paths in both LiveOVNClient and MockOVNClient.
handleDeleteNAT matches errors.Is(err, ErrNATNotFound) and treats
that as success (logs INFO 'already absent (idempotent)').
Existing TestTopologyHandler_DeleteNAT_NotFound asserted the opposite
contract; the comment claimed EIP disassociation needed the error
signal, but DisassociateAddress also fire-and-forgets vpc.delete-nat,
so the contract never had a real consumer. Test renamed to
_Idempotent with assertSuccess.
Other failure modes (router missing, OVN unreachable, transact error)
still surface as ERROR + reply with the error string.
Refs: mulga-siv-107
* ci(e2e-go): cluster JUnit failures + name likely root cause (#230)
* ci(e2e-go): cluster JUnit failures + name likely root cause
New composite action `.github/actions/e2e-analyze` runs after
`go-junit-report` converts each suite log. For every `<failure>` it
strips ANSI, normalises dynamic noise (EC2 IDs, UUIDs, IPs, timestamps,
file:line) to a stable signature, picks the earliest non-cascade
failure per suite as the likely root cause, and buckets the rest by
signature. Output is appended to `$GITHUB_STEP_SUMMARY` and written to
`<log-dir>/analysis.md` so it travels with the artifact bundle.
Cascade markers (`must populate fix.`, `Should NOT be empty`,
`Expected value not to be nil`) collapse the typical "one upstream
phase fails, ten downstream `require.NotEmpty` assertions trip" pattern
into a single grouped entry under the real first failure.
The analyser is advisory — it always exits 0. The existing
"Fail job if any suite failed" gate keeps owning exit-code semantics.
Stage 1 of `docs/development/improvements/e2e-go-failure-analysis.md`
(mulga-siv-102). Stage 2 (per-failure log slicing) and Stage 3
(rule-based hints via `tests/e2e/.failure-rules.yaml`, placeholder
seeded here) are independent follow-ups.
CI wiring:
- e2e-go.yml: new "Analyze e2e failures" step between "Publish test
summary" and the artifact upload, always-runs.
- spinifex.yml: new `make test-actions` step so the golden-file tests
under `.github/actions/e2e-analyze/testdata/` run on every PR.
- Makefile: `test-actions` target; kept out of `test-cover` so CI-only
tooling doesn't dilute coverage %.
Golden fixtures: a synthetic cascade-vpcs case (one Phase 5
InvalidParameterValue root + 15 downstream cascades), one real
single-failure case captured from run 26037892507, and a clean
no-failures case.
* ci(e2e-go): per-failure log bundles with time-window slicing
Stage 2 of the e2e failure analyzer. For each failure surfaced by
Stage 1, materialise <log-dir>/analysis/NN-<TestName>/ containing:
failure.txt — summary (test, start, duration, file-site, error)
journal.log — spinifex-journal.log sliced to
[StartAt - 5s, StartAt + Duration + 5s]
test.log — test-<suite>.log block between `=== RUN <name>`
and the matching `--- FAIL` marker
<copies> — any files the harness already dropped at
<log-dir>/<TestName_with_/_replaced>/
Report links each entry to its bundle so a reader on the Actions
page goes straight to the relevant slice.
Workflow now invokes journalctl with --output=short-iso so the
analyzer can parse wall-clock timestamps for the journal window.
Lines without a parseable timestamp inherit the previous line's
classification (keeps multi-line records and stack traces
contiguous).
Render output is unchanged when BundlePath is empty, so the Stage 1
golden snapshots still match.
* ci(e2e-go): restore GOTOOLCHAIN=auto after setup-go v6 bump
setup-go@v6 flipped the GOTOOLCHAIN default from "auto" to "local",
which makes Go refuse to auto-download the toolchain version named
in go.mod. The self-hosted ci-single runner ships /usr/local/go
older than spinifex/go.mod's `go 1.26.3`, and build-install-artifacts.sh
prepends that path so the system Go wins the resolution race. Under
v5 ("auto"), Go silently fetched 1.26.3 on demand; under v6
("local"), the Build Install Artifacts step errors out:
go: spinifex/go.mod requires go >= 1.26.3 (running go 1.26.1; GOTOOLCHAIN=local)
Re-set the env to "auto" so behaviour matches pre-v6 runs.
* ci(e2e-go): re-export GOTOOLCHAIN=auto after setup-go
Workflow-level env doesn't help: setup-go@v6 writes GOTOOLCHAIN=local
to $GITHUB_ENV, which overrides workflow env in every later step.
Add a dedicated step right after setup-go that appends
GOTOOLCHAIN=auto to $GITHUB_ENV so the Build Install Artifacts step
(which uses /usr/local/go from a stale runner path) can self-heal by
fetching the toolchain named in go.mod, matching pre-v6 behaviour.
* fix(vpcd): add flows-ready barrier after dnat_and_snat add
handleAddNAT wrote the NAT rule to OVN NB and returned immediately,
mirroring the gap siv-105 closed for handleIGWAttach but on the NAT
add path. ec2d/RunInstances then returned success while ovn-northd
was still compiling NB->SB and ovn-controller hadn't installed the
DNAT/SNAT flows on the gateway chassis, so the freshly launched VM
was unreachable on its public IP for whatever flow-install latency
the chassis happened to incur.
Reproduced in CI run 26072432957 Phase8b_VPCSubnetE2E: VM at
192.168.0.212 timed out at 3min while diagnostics showed
Port_Binding up=true for both the VM eni and the chassis-redirect
gateway. SB-side convergence was complete; the missing piece was the
same `ovn-nbctl --wait=hv sync` barrier siv-105 added to IGW attach,
applied at the end of handleAddNAT as well.
Reuses the existing package-var `waitForFlowsHV` so tests can stub
it; no new infra. Parallel unit test asserts the call site fires
exactly once per add-nat.
* ci(e2e-analyze): real-artifact fixes for journal slicing
Three bugs surfaced after Stage 2 ran against real CI artifacts in
run 26072432957:
1. Regex `[+-]\d{4}` rejected the colon-offset form (e.g. `+10:00`)
that systemd's `journalctl --output=short-iso` actually emits on
most versions. Added a parser that tries RFC3339 first and falls
back to the no-colon `-0700` layout. Result: 0-line journal slices
become populated.
2. go-junit-report's `timestamp` attr on <testsuite> records the
moment the XML was produced (post-suite), not when the tests
started. Computing StartAt as `that + cumulative-priors` placed
every failure AFTER the journal collection ended, so windows
always missed. Workflow now writes `${ARTIFACT_DIR}/test-<suite>.start`
right before running each suite; analyzer prefers that file over
the junit timestamp.
3. Persisted `OffsetFromSuiteStart` on each Failure so the
suite-start override can recompute StartAt without re-parsing the
XML.
Unit tests added for the colon-offset journal format. End-to-end
test (TestWriteBundles_EndToEnd) still passes with the new layout.
* ci(e2e-analyze): skip parent testcases when summing suite offsets
go-junit-report emits one <testcase> for the parent Go test AND each
subtest, with the parent's time = sum of its children. ParseFile was
summing every testcase into the suite-cumulative-offset counter, so
every leaf's offset was inflated by ~the parent's total duration.
Visible on run 26073817490 as Phase8b's StartAt landing 13 minutes
after the suite finished collecting logs — beyond the journal window,
so the per-failure journal slice was empty.
Pre-compute the leaf set per suite (a name is a leaf iff no other
testcase name starts with `<name>/`) and only accumulate cumul on
leaves. Golden snapshots updated; the offset for the only failure in
each fixture now matches the actual leaf-sum start time.
* test(e2e-harness): capture IP-targeted datapath state on Phase8b SSH fail
Phase 1 diagnostics for mulga-siv-111. The Phase8b SSH timeout
reproduced on the e2e-failure-analysis branch with siv-105's IGW
barrier AND the new add-nat barrier BOTH firing cleanly, so the
residual gap is downstream of `--wait=hv sync` — most likely stale
OVS conntrack, stale upstream-host ARP, or an OF flow-install miss.
DumpVPCFlowDiagnostics now takes a VPCDiagnosticsOpts carrying the
external IP, logical IP, and per-test artifact dir. When set, the
helper greps `ovs-ofctl dump-flows br-int`, `conntrack -L`, and
`ip -4 neigh show` for the external IP / logical IP and snapshots
`ovn-nbctl find NAT external_ip=…`. Each capture is both printed
inline (existing affordance) and written to a separate file under
fix.Artifacts so the Stage 2 analyzer bundler picks it up alongside
the journal + test-log slices, no analyzer changes needed.
Phase8b's caller now passes pubIP / privIP / fix.Artifacts so the
captures fire on the next failure; updates the t.Fatalf message to
point at siv-111 instead of siv-105 (which is the now-closed parent).
* fix(e2e-single): authorize SSH ingress on fresh-VPC default SG in Phase8b
Phase 8b stands up a non-default VPC and runs an instance there. Its
auto-created default SG denies all ingress, so the SSH probe times out
even when every datapath barrier converges — OF flow captures show the
SYN reaches OVN, gets DNAT'd at table 13, then drops at the ACL between
tables 13 and 17. The bash driver relies on admin init having pre-baked
this rule on the *default* VPC's default SG; a fresh VPC has not.
Add a DescribeSecurityGroups + AuthorizeSSHIngress step between
route-table setup and run-instances, using the existing harness helper.
Fix surfaced via mulga-siv-111's Phase 1 diagnostics (OF flow + ACL
drop counts), invalidating the H1/H2/H3 stale-state hypotheses in the
bead plan doc.
* test(e2e): dual-detect ubuntu image version (24.04/26.04) during gold-image transition
After mulga gold image bumped to v6-fastboot (stages ubuntu-26.04 / resolute
instead of ubuntu-24.04 / noble), all consumers that hardcoded 24.04 break on
the new image. Make them detect at runtime so v3 and v6 envs both pass.
- tests/e2e/run-e2e.sh: DescribeImages filter accepts both
ami-ubuntu-{26,24}.04-<arch>, then derives IMAGE_NAME from the match.
- tests/e2e/run-pseudo-multinode-e2e.sh: prefer ~/images/ubuntu-26.04.img,
fall back to ubuntu-24.04.img; pass detected version to spx import.
- tests/e2e/single/image_test.go: walk candidate catalog names (26.04 first,
then 24.04), stop on the first one DescribeImages can resolve.
- .github/workflows/e2e-baremetal.yml: download resolute (26.04) into
BM_ISO_CACHE; update BM_UBUNTU_IMAGE env var path.
- docs/terraform-workbooks/{bastion-private-subnet,s3-webapp,nginx-webserver}:
AMI filter matches `*ubuntu-26.04*` OR `*ubuntu-24.04*` (AWS values=
array is ORed).
Companion changes for mulga (bootstrap scripts, baremetal, iso pipeline)
land in scripts/tofu-cluster/* and scripts/{tofu-iso,baremetal,iso-builder}/*
under bead mulga-siv-114. Drop the 24.04 fallbacks once all envs are on the
v6 gold image and 24.04 entries are removed from spinifex/utils/images.go.
* feat(e2e): add service-interfaces.yaml manifest + validator
Source of truth for which spinifex services produce/consume which NATS
subjects, and which e2e suites cover which services. Consumed downstream
by the targeted suite selector (Bead 4) to derive subset E2E runs from a
PR diff so single-service changes stop paying the full 25 min suite cost.
- docs/service-interfaces.yaml: 7 services, 4 active suites (cert, lb,
single, ddil). Subscribes / publishes / depends_on filled from grep
of NATS subjects in source. spinifexui marked covers_no_suite so
UI-only diffs skip e2e entirely. Future suites (multinode, reboot,
tofu, baremetal) commented in until Go ports land.
- tests/e2e/manifest-check: schema + cross-reference validator (path
existence, subject regex, duplicate detection, suite covers resolves
to a real service, fixtures references resolve). 8 unit tests cover
the failure modes plus a load-and-validate on the real manifest.
- Makefile: manifest-check target, hooked into preflight.
- docs/DESIGN.md: link to the manifest under "NATS Messaging".
mulga-siv-116. See docs/development/improvements/e2e-targeted-suite-selection.md.
* fix(e2e): drop catalog-download import path from phase4_Image
`spx admin images import --name ubuntu-26.04-x86_64` with wantErr=true
was meant to detect that bootstrap-install.sh already imported the AMI
and short-circuit on the duplicate exit code. After the v6 gold-image
bump preloaded `~/images/ubuntu-26.04.img` and bootstrap-install.sh
switched to `--file` for the import, two things broke at once:
- The catalog `--name` path no longer hits a duplicate guard for the
--file-imported AMI, so the import re-downloads ~857MB and either
succeeds (failing the wantErr=true assertion) or wedges the job
timeout while the catalog server crawls along at 16KB/s.
- `--file` from the test won't work either: the test binary runs as
`tf-user` over SSH, while bootstrap-install.sh placed the image at
`/home/spinifex/images/...` (mode 0700 home dir).
bootstrap-install.sh is already responsible for staging the AMI. The
test's job is verifying DescribeImages round-trips by-name and by-ID —
not re-exercising import. Trust the bootstrap-staged AMI and stop
calling import.
Keeps the 26.04→24.04 dual-detect for the v3 gold image transition;
drop the 24.04 candidate when no envs remain on v3.
Refs mulga-siv-115 (caught by the EC2 cold-boot fast-path E2E run on
feat/ec2-coldboot-fastpath).
* ci(e2e-go): default to e2e-single + harden status-report SHA fallback
Two changes to .github/workflows/e2e-go.yml:
1. Default `suites` input now includes e2e-single so a no-input
manual dispatch exercises the full single-node ladder. Single-node
Go harness is greenable now (image_test fix at 10eb38a).
2. Report E2E (Go) Status job no longer gated on `inputs.sha` and
falls back to `github.sha` when the input is unset. Previously a
manual dispatch without a PR head SHA either skipped the status
post entirely or hit `gh api repos/.../statuses/` with an empty
path component (the 'invalid hash' error) — tests passed, the
report step failed silently. Now status posts on every dispatch
against the dispatched ref's commit when no explicit SHA given.
Closes mulga-siv-112.
* feat(e2e): add Ensure* fixture API with memoization + cleanup
Introduces tests/e2e/harness/fixtures.go: the memoized, cleanup-aware
provisioning helpers that Bead 3 (single-fixture-migration) will rewrite
phase tests around.
Each Ensure* helper:
- takes (t *testing.T, fx *Fixture, args...)
- memoizes by a key derived from args (sync.Mutex + singleflight)
- registers t.Cleanup on the Fixture's parent test so memoized resources
survive child-subtest lifetimes
- bakes readiness polling into the helper (no sleeps in callers)
Exported helpers: EnsureKeyPair, EnsureAMI, EnsureDefaultVPC,
EnsureSubnet, EnsureSG, EnsureInstance, EnsureVolume, EnsureSnapshot,
EnsureNATGateway, EnsureLoadBalancer.
Fixture stores ec2iface.EC2API + elbv2iface.ELBV2API so unit tests can
inject fakes without building a real AWSClient (TLS + env). Compile-time
checks guard that the SDK concrete types satisfy the stored interfaces.
Manifest fixtures: block populated — Bead 4 selector will resolve test
coverage from AST-scanned Ensure* call sites via this map.
Makefile: new test-harness target runs the harness unit tests under
-tags=e2e (mocked, no infra). Hooked into preflight so signature drift
and memo/cleanup regressions surface locally.
Bead 2 of docs/development/improvements/e2e-targeted-suite-selection.md.
No existing tests modified — fixture API is net-new.
* feat(e2e): port run-reboot-e2e.sh to Go
Adds tests/e2e/reboot/reboot_test.go — Go port of the single-node host-reboot
resilience driver (cell #18). Runs from the GitHub Actions runner (not
shipped to the VM like cert/lb/single) because systemctl reboot kills any
in-VM process mid-step.
All AWS API traffic crosses the WAN to https://WAN_IP:9999 via the SDK;
SSH-level ops (reboot, ovn-nbctl, journalctl, HTTP burst) hop via
harness.PeerSSH. Trust path tries to SCP the cluster CA off the VM and
falls back to InsecureSkipVerify with a log message if SCP fails — the
cert suite already validates real trust, so the fallback is intentional.
Harness changes:
- env: WANHost field sourced from SPINIFEX_WAN_IP, distinct from
ServiceIPs[0] (which is the awsgw bind IP, often loopback in-VM).
- aws: SPINIFEX_AWS_INSECURE=1 escape hatch when the runner has no CA on
disk. Used as the fallback when CA fetch fails.
- cert: SPINIFEX_CA_CERT explicit path override so the reboot test can
point ResolveCACert at the SCP'd tmp file.
- peer_ssh: IsReachable() probe wrapper for cleanup paths that must skip
rather than fatal when the host is mid-reboot.
Workflow (.github/workflows/e2e-go.yml):
- New e2e_go_reboot job, opt-in via `e2e-reboot` in the SUITES dispatch
input. needs: e2e_go_single to serialise on the ci-single self-hosted
runner. Fetches AWS creds from the VM's ~/.aws/credentials over SSH
(mirrors the cred hop already proven by e2e-ddil.yml).
- e2e_go_single filters 'reboot' out of its build + run loops so the
in-VM driver doesn't try to ship reboot.test to the VM and reboot it
from inside.
- e2e_go_status treats skipped reboot as success.
Refs mulga-siv-118. Plan: docs/development/improvements/e2e-go-reboot-port.md
(in the mulga parent repo).
* ci(e2e-go): run reboot job in parallel with single
Drop `needs: e2e_go_single` from e2e_go_reboot. Both jobs target the
same [self-hosted, ci-single] label, so GitHub Actions serialises on
runner allocation rather than blocking via job ordering — parallel-by-
spec, queued-by-runner. Lets a reboot-only dispatch start immediately
instead of waiting for an unrelated single-job pre-condition.
* fix(e2e/reboot): wait for both backends to serve before post-reboot burst
Phase 8.6 previously probed until ANY HTTP response from the ALB, then
fired a 20-req burst. Post-reboot, only one of the two backends may be
serving (the second is still bringing its responder online) while the
daemon's TG cache still shows pre-reboot 'healthy' for both — so
WaitForTargetsHealthy returns in ms instead of seconds, the ANY probe
matches the first-up backend, and the burst lands 20/0 instead of 10/10.
Probe until BOTH backends' instance_ids each appear at least once before
the round-robin burst — confirms recovery has reached both, not just one.
Underlying daemon-side TG cache staleness tracked separately as
mulga-siv-119. This wait is the test-side guard; the daemon fix will
make WaitForTargetsHealthy meaningful again post-reboot.
Refs mulga-siv-118.
* ci(e2e-go): skip single job when dispatch requests only reboot
Add if: gate to e2e_go_single that skips the job when the workflow_dispatch
suites input contains none of {e2e-single, e2e-cert, e2e-lb}. Push events
and the default suites list still trigger it, so plain PR runs are
unaffected. Avoids ~15min of wasted tofu provision+bootstrap+teardown
when only the runner-resident reboot suite is requested.
* fix(e2e/reboot): bump post-reboot LB recovery timeout to 5min
Pre-reboot the test creates the LB AFTER apps are already running, so
HAProxy starts probing against an already-serving responder — ~75s to
healthy. Post-reboot the daemon's recovery scan launches app VMs and
the ALB sys.micro VM simultaneously, so health checks start probing
within seconds of app boot beginning and both targets honestly report
unhealthy until the responder unit finishes starting. Cloud-init's
per-instance semaphore means no shortcut path on relaunch, so guest
second-boot can be slower than first boot.
Bash run-reboot-e2e.sh hid this behind stale 'healthy' state read from
the daemon's pre-reboot TG cache — fixed via ResetTargetHealthOnStartup
(mulga-siv-119), which made the test honestly wait on real lb-agent
reports. The 180s bash inherited turned out to be wishful thinking
backed by the cache bug, not measured recovery time.
5min matches the actual post-reboot eventually-consistent window
observed on run 26134156355. Refs mulga-siv-118.
* feat(e2e/reboot): ssh into guest VMs on failure for HTTP responder diag
Post-reboot failures bottom out at "0/2 healthy" with no insight into
whether the responder unit died, port 80 isn't bound, or the OVN path is
broken. Capture the keypair material from CreateKeyPair so OnFailure can
hop host -> qemu hostfwd:22 -> guest and dump systemctl status, port
listeners, curl localhost, and the responder journal per instance.
* fix(e2e/reboot): probe app instances via public IP (no QEMU hostfwd)
Previous hostfwd-SSH approach didn't apply: app VMs use pure OVN tap
interfaces (qemu argv has no hostfwd=tcp:*:*-:22), so the regex never
matched. Replace with host-side probes against the IGW-NAT'd public IP:
ping, /dev/tcp connect, and curl. Combined with the qemu ps check and
ovn-nbctl list nat dump, this localises post-reboot failures to one of:
VM not running, OVN DNAT missing, responder down, or sys.micro-only
reachability (path asymmetry).
* feat(e2e/reboot): probe ALB + dump host OVN/OVS state on failure
Last run showed apps unreachable from host post-reboot (ping 100% loss,
tcp refused) even though OVN dnat_snat rules looked correct. Need to
distinguish gateway/ext-bridge breakage from app-side breakage: add a
parallel ALB-public-IP probe (works pre-reboot, so any post-reboot
failure isolates the regression) plus a full host net-state dump (ip
route, ip neigh, ovn-nbctl show, ovs-vsctl show, br-int port list).
* feat(e2e/reboot): dump qemu serial console per app on failure
/run/spinifex/console-i-<id>.log captures cloud-init, kernel boot,
systemd, anything the guest emitted to ttyS0. Tail to 256KiB (~4x a
typical Linux boot) so the latest boot lands in full plus the tail of
the prior. Surfaces guest-side boot failures (kernel panic, fs
corruption, cloud-init OOM, etc.) that are invisible to host-side
network probes.
* feat(e2e/reboot): SSH into guest VMs via IGW for responder root-cause
Last run nailed the symptom: ALB returns 503, app public IPs answer TCP
with RST (port 80 not bound), so reboot-e2e-httpd.service isn't running
on relaunched VMs. To find out why (unit never installed? installed but
not enabled? failed to start?), open tcp/22 ingress on the shared SG and
add a guest-SSH diagnostic via the IGW NAT path. The diagnostic dumps
the unit state, port listeners, /var/lib/httpd, cloud-init result.json,
cloud-init logs, and the instance-id semaphore directory.
* test(e2e/reboot): always run guest SSH diag in phase 8
Last run passed after adding tcp/22 SG ingress. Two possibilities to
distinguish: (a) responder was running all along but blocked at the
OVN/SG layer, (b) responder wasn't running and HC was satisfied by some
other path. Capture in-guest unit + cloud-init state on every run so we
can confirm which without needing the test to fail.
* refactor(e2e/single): migrate VPC group to EnsureDefaultVPC [skip ci]
Bead 3a of mulga-siv-120 (e2e-single-fixture-migration). Replaces the
"discover default VPC + cache on Fixture" pattern in vpc_test.go,
natgw_test.go, securitygroup_test.go with harness.EnsureDefaultVPC.
- Adds Harness *harness.Fixture field on single.Fixture; constructed in
TestSingleNode alongside AWSClient. Parent test for memoized Ensure*
resources is TestSingleNode itself.
- Removes fix.DefaultVPCID / fix.DefaultSGID / fix.DefaultSubnetID reads
in the three target files. Fields stay on the struct until 3e+3h
migrate the instance / datapath / umbrella sites.
- Custom VPC graphs created by phase8b (10.99.0.0/16) and the bastion
graph in phase8d remain inline — those are the resource lifecycle
assertions, not prereqs.
CI gate skipped per dev request — full single-node suite verifies when
upstream main is unbroken.
* refactor(e2e/single): migrate Phase 3 key pair to EnsureKeyPair [skip ci]
Bead 3c of mulga-siv-120 (e2e-single-fixture-migration). Phase 3 now
calls harness.EnsureKeyPair for the primary key pair (Phase 5+ prereq).
The Ensure* path owns the create assertion (PEM shape), memoizes across
phases, and registers TestSingleNode-scoped cleanup — replacing the
former TODO(stage-G) deferred-cleanup marker.
- Drops hardcoded "test-key-1" name; EnsureKeyPair uses scratch-suffixed
name so parallel CI runs don't collide on the namespace-flat AWS key
pair name.
- fix.KeyName / fix.KeyPath still assigned for downstream Phase 5+
readers. Per 3e migration plan, those reads move to in-test
EnsureKeyPair calls and the Fixture fields drop with 3h.
- Import / describe / delete dance for test-key-2 stays inline — it is
the lifecycle assertion for the import path, not a prereq.
* refactor(e2e/single): migrate Phase 4 + 5e AMI paths to EnsureAMI [skip ci]
Bead 3d of mulga-siv-120 (e2e-single-fixture-migration).
- Phase 4 (image_test.go): discover catalog AMI by canonical name, then
route through harness.EnsureAMI({Existing}) so the state-available
wait + memoization live with the fixture.
- Phase 5e (createimage_test.go): replace inline CreateImage block with
harness.EnsureAMI({CreateFrom}). The custom AMI build now memoizes,
registers DeregisterImage cleanup at TestSingleNode exit, and reuses
the harness state poll.
Harness fix: extend AMICreateSpec with NoReboot bool — Phase 5e bakes
the AMI while leaving the source instance running for Phase 6/7
lifecycle assertions; Bead 2 missed this option.
fix.AMIID / fix.CustomAMIID / fix.CustomAMISnapID stay on Fixture for
downstream Phase 5/6/7 + Phase 9 cleanup. Those reads migrate with
3e/3h.
* refactor(e2e/single): migrate Phase 5 instance lifecycle to EnsureInstance [skip ci]
Bead 3e of e2e-single-fixture-migration.
- phase5_LaunchInstance: provision via EnsureInstance + re-describe for
BDM/root-volume extraction; default VPC via EnsureDefaultVPC.
- phase8e_SGToSGDatapath: replace inline DiscoverDefaultVPC with
EnsureDefaultVPC. createSG / runSGEInstance take vpcID/subnetID
arguments so phase 8e no longer depends on fix.Default* fields.
- runSGEInstance stays direct RunInstances: phase 8e VMs are test
subjects (need fresh ENI/private-IP per run), not memoized prereqs.
- harness.EnsureInstance memo key now includes InstanceType, SGID, and
UserData hash — a second call with a different SG would have silently
returned the first instance ID (Bead 2 bug).
* refactor(e2e/single): migrate Phase 5b/5c volume + snapshot to Ensure* [skip ci]
Bead 3f of e2e-single-fixture-migration.
- phase5b_VolumeLifecycle: create+wait via EnsureVolume; subsequent
modify/attach/detach/delete stay inline (test subject lifecycle).
- phase5c_SnapshotLifecycle: create+wait via EnsureSnapshot. Drop the
create-response State/Progress shape assertions — after EnsureSnapshot
the snapshot is already "completed", and the describe block below
re-asserts VolumeId / VolumeSize / Description.
- harness.EnsureSnapshot now takes SnapshotSpec{VolumeID, Description}
so callers can pin a description (snapshot_test asserts on it). Memo
key includes description hash so distinct-description calls do not
silently collide on the volumeID prefix.
* refactor(e2e/single): isolate IAM stage E via parent-scoped cleanup [skip ci]
Bead 3g of e2e-single-fixture-migration.
- harness.Fixture: add RegisterCleanup helper that wraps fx.parent.Cleanup
so test code can register parent-scoped teardown for resources created
outside ensureOnce (IAM users / policies threaded across phase
subtests). Registering on the calling subtest would tear down before
the next dependent phase ran.
- phaseIAM1_UserCRUD: register parent-cleanup for alice/bob/charlie.
- phaseIAM4_PolicyCRUD: register parent-cleanup for the 5 test policies.
- IAM stage no longer relies solely on phaseIAM7 to clean up after a
mid-stage failure; phase 7 still runs and now serves as an idempotent
post-cleanup assertion.
No new Ensure* helper: IAM resources have no memoization story (users
and policies are test subjects, secrets only returned once on create).
IAM phases also have no EC2 prereqs, so EnsureDefaultVPC / EnsureInstance
are not applicable here.
* refactor(e2e/single): drop dead Fixture fields after Ensure* migration [skip ci]
First half of bead 3h. Removes per-phase Fixture fields with zero
remaining readers after bead 3a-3g:
- DefaultVPCID / DefaultSGID / DefaultSubnetID: superseded by
harness.EnsureDefaultVPC (3a wired phase8b/8c/8d/5f; 3e wired phase5
+ phase8e).
- OVNAvailable: never assigned anywhere — leftover from the bash port.
Phase fields still in use (AMIID, InstanceID, IAM*, etc.) stay until
the umbrella TestSingleNode is replaced with top-level Test* + package-
level fixture singleton in the second half of 3h.
* refactor(e2e): add process-mode Fixture + TestMain singleton infra [skip ci]
Second half of bead 3h. Lands the infrastructure required to dismantle
the TestSingleNode umbrella in a follow-up bead, without changing the
umbrella body today.
harness.Fixture changes:
- NewProcessFixture(aws) builds a Fixture with no *testing.T parent.
Cleanup callbacks queue on an internal LIFO drained by Close() —
intended to be invoked from TestMain after m.Run(). Lets a singleton
fixture span every top-level Test* in a package.
- Existing NewFixture(t, aws) unchanged; cleanups still route via
t.Cleanup. registerCleanup() and logf() helpers fork on parent==nil
so ensureOnce and RegisterCleanup work in either mode.
- TestProcessFixture_CloseRunsCleanupsLIFO covers Close ordering,
one-shot semantics, and the no-op second-Close path.
single/main_test.go additions:
- TestMain owns process-level lifecycle: drains pkgFix.Harness.Close()
and removes pkgFix.TmpDir after m.Run().
- requireSingleNodeFixture(t) returns the package-scoped singleton,
building it lazily on first call. Skips when SPINIFEX_E2E is unset
or mode != single; fails on construction error. Currently
unreferenced — TestSingleNode keeps its own per-test Fixture for
now. The helper exists so the follow-up bead (per-phase top-level
Test* migration) has a stable API to call from day one.
Deferred to follow-up bead: extract Phase 4 AMI-name discovery + Phase
2 InstanceType/AZ selection into harness helpers so each phase function
can self-bootstrap, then promote phase subtests to top-level Test* and
delete the TestSingleNode umbrella.
* fix(e2e/harness): EnsureDefaultVPC filter name is-default not isDefault
Daemon rejects the camelCase `isDefault` form with 400
InvalidParameterValue. AWS CLI / DiscoverDefaultVPC use `is-default`.
Surfaced as Phase 5_LaunchInstance fatal in CI run 26140908505 —
cascade-failed every downstream phase that needed fix.InstanceID.
* fix(e2e/single): phase8 8j/8k probe fix.KeyName not hardcoded test-key-1
Bead 3c migrated phase 3 to harness.EnsureKeyPair which produces a
scratch-suffixed name (e2e-key-<random>), not the literal "test-key-1"
the bash port used. 8j_CreateKeyPairDuplicate / 8k_ImportKeyPairDuplicate
kept the old name and so created a fresh key pair instead of triggering
InvalidKeyPair.Duplicate — surfaced as the only two FAILs in CI run
26143227369 (everything else green after the EnsureDefaultVPC fix).
Read fix.KeyName (set by phase3_KeyPairs) so the probe hits the staged
primary key. Header comments updated to match.
make preflight diff-coverage against origin/dev fails at 57.9% due to
unrelated commits inherited from the recent rebase onto
feat/e2e-go-harness; against the fork base the diff is 0 instrumentable
lines (test-only change).
* refactor(e2e/single): dismantle TestSingleNode umbrella, per-phase self-bootstrap
mulga-siv-121. Each phase from run-e2e.sh now runs as a top-level Test*
in tests/e2e/single, runnable in isolation via `go test -run TestX`.
Highlights:
* TestSingleNode umbrella removed. Phase ordering dropped — each Test*
self-bootstraps prerequisites through harness.Ensure*/harness.Discover*
+ the new package-local need*/iamEnsure* helpers.
* Fixture shrunk to {Env, AWS, Harness, Artifacts, TmpDir, PoolMode}.
Every per-phase resource ID now lives on harness.Fixture via
EnsureKeyPair/EnsureAMI/EnsureDefaultVPC/EnsureInstance/etc.
* TestMain owns the package singleton Fixture; harness.Fixture.Close
drains every cleanup at process exit. phase9 + phase9a (in-line
teardown) deleted; phase9b retained as TestFinalClusterStats.
* IAM Test* share state via sync.Once-gated helpers in
iam_helpers_test.go (alice/bob/charlie users + keys + admin account).
* New harness/discover.go: DiscoverDefaultAZ, DiscoverNanoInstanceType,
DiscoverUbuntuAMI — memoized on the Fixture and routed through
EnsureAMI so any Test* can resolve them without phase coupling.
* New tests_test.go: descriptive top-level Test* wrappers
(TestVPCSubnetE2E, TestKeyPairs, TestSnapshotLifecycle, …) that
delegate to phaseN_X(t, requireSingleNodeFixture(t)).
* test(e2e/reboot): SSH as ec2-user, dump unit + network-online state
spinifex's cloud-init template hardcodes Username="ec2-user"
(handlers/ec2/instance/service_impl.go:1218) regardless of AMI,
so even on Ubuntu the SSH-injectable account is ec2-user, not
ubuntu. Switch the diag SSH-in to ec2-user so we actually get
in (was "Permission denied (publickey)").
Add diagnostics to distinguish the two leading post-reboot
hypotheses: (a) viperblock write loss = unit file gone vs
(b) network-online.target never fires = unit stuck on
After=network-online.target. New dumps: unit file content +
enable symlink, network-online.target status, networkctl,
ip addr / route, prior-boot journal grep for the responder.
* fix(e2e/single): resolve admin account ID before TestIAM7 deletes alice
iamEnsureAdminAccountID probes via GetUser(alice). TestIAM7_Cleanup
deleted alice/bob/charlie first, then called the helper at line 808 —
NoSuchEntity if IAM7 was the first IAM test to run (no memoized
account ID). Move the call above the deletion loop.
This commit is contained in:
Josh Siviour
@@ -0,0 +1,104 @@
|
||||
# e2e-analyze
|
||||
|
||||
Post-test failure-clustering action for the Go E2E suites.
|
||||
|
||||
## What it does
|
||||
|
||||
Runs after `go-junit-report` has converted each suite's stdout into
|
||||
`junit-<suite>.xml`. For every JUnit file matched by `junit-glob`:
|
||||
|
||||
1. Parses each `<testcase>` with a `<failure>` / `<error>` child.
|
||||
2. Strips ANSI escapes, then normalises dynamic noise (UUIDs, EC2 IDs,
|
||||
IPs, timestamps, request-ids, source line numbers) to produce a stable
|
||||
_signature_ per failure.
|
||||
3. Picks the **earliest non-cascade** failure per suite as the likely
|
||||
root cause. "Cascade" = failure message matches a fixture-dependency
|
||||
marker (`must populate fix.`, `Should NOT be empty`,
|
||||
`Expected value not to be nil`). When every failure is a cascade
|
||||
(whole suite tripped on one upstream gap), the earliest cascade is
|
||||
the fallback root.
|
||||
4. Buckets the remaining failures by signature and renders a markdown
|
||||
report. The report is appended to `$GITHUB_STEP_SUMMARY` and also
|
||||
written to `<log-dir>/analysis.md` so it travels with the uploaded
|
||||
artifact bundle.
|
||||
|
||||
The action **never fails the job** — exit-code semantics stay with the
|
||||
existing "Fail job if any suite failed" gate.
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Required | Description |
|
||||
|--------------|----------|-------------|
|
||||
| `junit-glob` | yes | Shell glob for the JUnit XML files (e.g. `${ARTIFACT_DIR}/junit-*.xml`). |
|
||||
| `log-dir` | yes | Directory the rendered report is written into (`analysis.md`). |
|
||||
| `title` | no | Heading at the top of the summary section. Default `E2E failure analysis`. |
|
||||
|
||||
## Outputs
|
||||
|
||||
None. The summary is appended to `$GITHUB_STEP_SUMMARY`; the same
|
||||
content lands at `<log-dir>/analysis.md`.
|
||||
|
||||
## Local invocation
|
||||
|
||||
Outside Actions, the analyser prints to stdout instead of appending to a
|
||||
summary file:
|
||||
|
||||
```sh
|
||||
cd spinifex/.github/actions/e2e-analyze
|
||||
go run . -junit-glob "/tmp/artifacts/junit-*.xml" -log-dir /tmp/artifacts
|
||||
```
|
||||
|
||||
## Tests
|
||||
|
||||
Golden-file tests live in `testdata/`. Each subdir holds a scenario:
|
||||
|
||||
| Scenario | Source |
|
||||
|--------------------|-------------------------------------------|
|
||||
| `cascade-vpcs` | Synthetic: one Phase 5 root + 15 cascades |
|
||||
| `single-failure` | Real artifact from run `26037892507` |
|
||||
| `no-failures` | Synthetic: clean run |
|
||||
|
||||
Run normally:
|
||||
|
||||
```sh
|
||||
go test ./.github/actions/e2e-analyze/
|
||||
```
|
||||
|
||||
When the report format changes intentionally, regenerate the snapshots:
|
||||
|
||||
```sh
|
||||
go test ./.github/actions/e2e-analyze/ -update
|
||||
```
|
||||
|
||||
## Per-failure bundles (Stage 2)
|
||||
|
||||
After parsing, the action materialises a bundle directory per failure
|
||||
under `<log-dir>/analysis/NN-<TestName>/`. The report links to it from
|
||||
each entry so a reader on the Actions page can click straight through
|
||||
to the relevant log slice.
|
||||
|
||||
Each bundle contains:
|
||||
|
||||
| File | Source |
|
||||
|-------------------|-----------------------------------------------------------------------------------|
|
||||
| `failure.txt` | Synthesised summary: test, start, duration, file-site, error line. |
|
||||
| `journal.log` | `spinifex-journal.log` sliced to `[StartAt − 5s, StartAt + Duration + 5s]`. |
|
||||
| `test.log` | `test-<suite>.log` block between `=== RUN <name>` and the matching `--- FAIL`. |
|
||||
| `<other files>` | Anything the harness already dropped into `<log-dir>/<TestName>/` at run-time. |
|
||||
|
||||
The journal slice relies on the workflow running
|
||||
`journalctl --output=short-iso` so timestamps are parseable. Lines
|
||||
without a parseable timestamp inherit the previous line's
|
||||
in-window/out-of-window classification (so multi-line records and stack
|
||||
traces stay contiguous).
|
||||
|
||||
## Roadmap
|
||||
|
||||
- **Stage 1 (shipped):** first-failure surfacing + cascade clustering.
|
||||
- **Stage 2 (this update):** time-window correlation — per-failure bundle
|
||||
with sliced journal + test-log + any per-test artifacts.
|
||||
- **Stage 3:** rule-based root-cause classifier driven by
|
||||
`spinifex/tests/e2e/.failure-rules.yaml`.
|
||||
|
||||
See `docs/development/improvements/e2e-go-failure-analysis.md` in the
|
||||
parent mulga repo.
|
||||
@@ -0,0 +1,38 @@
|
||||
name: E2E Failure Analyzer
|
||||
description: >-
|
||||
Post-test analyzer for the Go E2E suites. Parses junit-*.xml emitted by
|
||||
go-junit-report, clusters failures by signature, picks the earliest
|
||||
non-cascade failure per suite as the likely root cause, and appends a
|
||||
human-readable report to $GITHUB_STEP_SUMMARY. Always exits 0 — the
|
||||
existing "Fail job if any suite failed" gate keeps owning exit-code
|
||||
semantics.
|
||||
|
||||
inputs:
|
||||
junit-glob:
|
||||
description: Shell glob matching JUnit XML files to analyze.
|
||||
required: true
|
||||
log-dir:
|
||||
description: >-
|
||||
Artifact directory. The rendered report is also written to
|
||||
$log-dir/analysis.md so it travels with the uploaded artifact.
|
||||
required: true
|
||||
title:
|
||||
description: Heading rendered at the top of the summary section.
|
||||
required: false
|
||||
default: "E2E failure analysis"
|
||||
|
||||
runs:
|
||||
using: composite
|
||||
steps:
|
||||
- id: analyze
|
||||
shell: bash
|
||||
working-directory: ${{ github.action_path }}
|
||||
env:
|
||||
ANALYZE_JUNIT_GLOB: ${{ inputs.junit-glob }}
|
||||
ANALYZE_LOG_DIR: ${{ inputs.log-dir }}
|
||||
ANALYZE_TITLE: ${{ inputs.title }}
|
||||
run: |
|
||||
go run . \
|
||||
-junit-glob "$ANALYZE_JUNIT_GLOB" \
|
||||
-log-dir "$ANALYZE_LOG_DIR" \
|
||||
-title "$ANALYZE_TITLE"
|
||||
@@ -0,0 +1,552 @@
|
||||
// Package main implements the e2e-analyze GitHub Action.
|
||||
//
|
||||
// Stage 1 of docs/development/improvements/e2e-go-failure-analysis.md:
|
||||
// cluster failures from go-junit-report's junit-*.xml by error signature,
|
||||
// pick the earliest non-cascade failure per suite as the likely root cause,
|
||||
// and render a human-readable report.
|
||||
//
|
||||
// This file contains the pure logic (parsing, signature extraction,
|
||||
// clustering, rendering). Side-effect handling lives in main.go.
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/xml"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// JUnit XML schema produced by go-junit-report v2. Stable across versions
|
||||
// we care about; only fields used downstream are decoded.
|
||||
type junitSuites struct {
|
||||
XMLName xml.Name `xml:"testsuites"`
|
||||
Suites []junitSuite `xml:"testsuite"`
|
||||
}
|
||||
|
||||
type junitSuite struct {
|
||||
Name string `xml:"name,attr"`
|
||||
Tests int `xml:"tests,attr"`
|
||||
Failures int `xml:"failures,attr"`
|
||||
Errors int `xml:"errors,attr"`
|
||||
Skipped int `xml:"skipped,attr"`
|
||||
Timestamp string `xml:"timestamp,attr"`
|
||||
Time float64 `xml:"time,attr"`
|
||||
Cases []junitTC `xml:"testcase"`
|
||||
}
|
||||
|
||||
type junitTC struct {
|
||||
Name string `xml:"name,attr"`
|
||||
Classname string `xml:"classname,attr"`
|
||||
Time float64 `xml:"time,attr"`
|
||||
Failure *junitFailure `xml:"failure"`
|
||||
Error *junitFailure `xml:"error"`
|
||||
Skipped *junitSkipped `xml:"skipped"`
|
||||
SystemOut string `xml:"system-out"`
|
||||
}
|
||||
|
||||
type junitFailure struct {
|
||||
Message string `xml:"message,attr"`
|
||||
Type string `xml:"type,attr"`
|
||||
Body string `xml:",chardata"`
|
||||
}
|
||||
|
||||
type junitSkipped struct {
|
||||
Message string `xml:"message,attr"`
|
||||
}
|
||||
|
||||
// Failure is the analyzer's normalized view of one failed testcase.
|
||||
type Failure struct {
|
||||
SuiteFile string // e.g. "single" (derived from junit-single.xml)
|
||||
Name string // e.g. "TestSingleNode/Phase5_LaunchInstance"
|
||||
Duration float64 // seconds
|
||||
Order int // position within the suite (0-based)
|
||||
StartAt time.Time // estimated wall-clock start: suite.timestamp + sum(prior durations)
|
||||
Error string // last meaningful error line, ANSI-stripped
|
||||
FileHint string // e.g. "vpc_test.go:227", parsed from failure body
|
||||
Cascade bool // signature matched a known downstream pattern
|
||||
Signature string // normalized error key for bucketing
|
||||
// OffsetFromSuiteStart is sum(prior testcase durations) at the time
|
||||
// this test started. Persisted separately from StartAt so a caller
|
||||
// that has a more authoritative suite wall-clock origin (e.g. a
|
||||
// `test-<suite>.start` file written by the workflow before the suite
|
||||
// runs) can recompute StartAt without re-parsing the XML.
|
||||
OffsetFromSuiteStart time.Duration
|
||||
// BundlePath is the relative path (from log-dir) to the per-failure
|
||||
// bundle written by Stage 2's WriteBundles. Empty when bundles were
|
||||
// not generated (e.g. unit tests that call Render directly).
|
||||
BundlePath string
|
||||
}
|
||||
|
||||
// SuiteReport summarises one junit-*.xml file.
|
||||
type SuiteReport struct {
|
||||
File string // basename, e.g. "junit-single.xml"
|
||||
Label string // short name, e.g. "single"
|
||||
Total int
|
||||
FailCount int
|
||||
Root *Failure // nil if no failures
|
||||
Cascades []Failure // everything else that failed in this suite
|
||||
Buckets [][]Failure // failures grouped by signature (root's bucket first)
|
||||
}
|
||||
|
||||
// Report is the full analysis across every junit file.
|
||||
type Report struct {
|
||||
Title string
|
||||
Suites []SuiteReport
|
||||
}
|
||||
|
||||
// ansiSeq matches ANSI CSI escapes the harness sprinkles through log
|
||||
// output (colours, bold). Three forms in the wild:
|
||||
// - real ESC: `\x1b[36m…`
|
||||
// - U+FFFD replacement (XML serialiser dropped the ESC byte): `�[36m…`
|
||||
// - bare CSI with the leading byte gone entirely: `[36m…`
|
||||
//
|
||||
// All three are stripped so colour state doesn't fork the signature bucket.
|
||||
var ansiSeq = regexp.MustCompile(
|
||||
"\x1b\\[[0-9;]*[a-zA-Z]" + // real ESC CSI
|
||||
"|�\\[[0-9;]*[a-zA-Z]" + // replacement-char CSI
|
||||
"|\\[[0-9]{1,2}(?:;[0-9]{1,2})*m", // bare colour code, "mode" form only
|
||||
)
|
||||
|
||||
// cascadeMarkers are message fragments that only appear in tests that
|
||||
// depend on a fixture field populated by an earlier phase. Their presence
|
||||
// means the failure is downstream of some other failure that happened
|
||||
// first; the report collapses them under that root cause.
|
||||
var cascadeMarkers = []string{
|
||||
"must populate fix.",
|
||||
"Should NOT be empty",
|
||||
"Expected value not to be nil",
|
||||
}
|
||||
|
||||
// noiseReplacers normalise dynamic identifiers so that "ten failures, same
|
||||
// bug" bucket together. Pattern ordering matters: longer/more-specific
|
||||
// patterns must precede the catch-alls.
|
||||
var noiseReplacers = []struct {
|
||||
re *regexp.Regexp
|
||||
repl string
|
||||
}{
|
||||
{regexp.MustCompile(`\b(vpc|subnet|sg|rtb|igw|i|ami|vol|snap|rtbassoc|key)-[0-9a-f]{8,}\b`), `${1}-<id>`},
|
||||
{regexp.MustCompile(`\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b`), `<uuid>`},
|
||||
{regexp.MustCompile(`\b[0-9a-f]{32,}\b`), `<hex>`},
|
||||
{regexp.MustCompile(`\b\d{1,3}(\.\d{1,3}){3}(:\d+)?\b`), `<ip>`},
|
||||
{regexp.MustCompile(`\b\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:?\d{2})?\b`), `<ts>`},
|
||||
{regexp.MustCompile(`request-id:\s*\S+`), `request-id: <id>`},
|
||||
{regexp.MustCompile(`(_test\.go):\d+`), `${1}:<line>`},
|
||||
{regexp.MustCompile(`\s+`), ` `},
|
||||
}
|
||||
|
||||
// fileHintRe pulls the last "foo_test.go:NNN" we see in a failure body so
|
||||
// the report can point a reader at the source line. Test files dominate the
|
||||
// hint set; harness/helper file matches go through goFileLineRe below.
|
||||
var fileHintRe = regexp.MustCompile(`([A-Za-z0-9_]+_test\.go:\d+)`)
|
||||
|
||||
// goFileLineRe matches any "foo.go:NNN:" prefix (test or non-test). Used to
|
||||
// surface harness/helper-level error text like
|
||||
// `ec2helpers.go:164: describe-vpcs: InvalidParameterValue …` which Go test
|
||||
// emits when a non-test file calls t.Errorf.
|
||||
var goFileLineRe = regexp.MustCompile(`^[A-Za-z0-9_]+\.go:\d+:\s*(.+)$`)
|
||||
|
||||
// testifyErrorPlaceholder is testify's "Error:" boilerplate when the
|
||||
// assertion failed because of a wrapped error. The next non-blank line in
|
||||
// the trace holds the actual error text.
|
||||
const testifyErrorPlaceholder = "Received unexpected error:"
|
||||
|
||||
// stripANSI removes CSI escapes; leaves everything else intact.
|
||||
func stripANSI(s string) string {
|
||||
return ansiSeq.ReplaceAllString(s, "")
|
||||
}
|
||||
|
||||
// extractErrorLine finds the most useful single-line summary in a failure
|
||||
// body. Priority order, highest first:
|
||||
// 1. Testify "Messages:" line — user-supplied context, the most signal-rich.
|
||||
// 2. Testify "Error:" content (with one-line lookahead when the placeholder
|
||||
// "Received unexpected error:" defers to the next line).
|
||||
// 3. Leading "foo.go:NNN: text" content line — t.Errorf from non-test code.
|
||||
// 4. Last `_test.go:NNN: text` line — bare t.Fatalf/Errorf from a test file.
|
||||
// 5. First non-empty line of the body.
|
||||
func extractErrorLine(body string) string {
|
||||
body = stripANSI(body)
|
||||
lines := strings.Split(body, "\n")
|
||||
|
||||
var (
|
||||
msgLine string
|
||||
errLine string
|
||||
goFileLine string
|
||||
lastTestLine string
|
||||
)
|
||||
for i, raw := range lines {
|
||||
l := strings.TrimSpace(raw)
|
||||
if l == "" {
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(l, "Messages:") {
|
||||
msgLine = strings.TrimSpace(strings.TrimPrefix(l, "Messages:"))
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(l, "Error:") && errLine == "" {
|
||||
content := strings.TrimSpace(strings.TrimPrefix(l, "Error:"))
|
||||
if content == testifyErrorPlaceholder {
|
||||
// Look ahead for the wrapped error text on the next non-blank line.
|
||||
for j := i + 1; j < len(lines); j++ {
|
||||
nxt := strings.TrimSpace(lines[j])
|
||||
if nxt == "" {
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(nxt, "Test:") || strings.HasPrefix(nxt, "Messages:") {
|
||||
break
|
||||
}
|
||||
content = content + " " + nxt
|
||||
break
|
||||
}
|
||||
}
|
||||
errLine = content
|
||||
continue
|
||||
}
|
||||
if m := goFileLineRe.FindStringSubmatch(l); m != nil {
|
||||
// Keep the LAST occurrence: t.Log / harness traces emit
|
||||
// progress lines from the same file:line pattern before the
|
||||
// failing assertion finally fires.
|
||||
goFileLine = strings.TrimSpace(m[1])
|
||||
continue
|
||||
}
|
||||
if fileHintRe.MatchString(l) {
|
||||
lastTestLine = l
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case msgLine != "":
|
||||
return msgLine
|
||||
case errLine != "":
|
||||
return errLine
|
||||
case goFileLine != "":
|
||||
return goFileLine
|
||||
case lastTestLine != "":
|
||||
// "vpc_test.go:227: Eventually: condition not met …" → trim the file prefix.
|
||||
if i := strings.Index(lastTestLine, ": "); i > 0 {
|
||||
return strings.TrimSpace(lastTestLine[i+2:])
|
||||
}
|
||||
return lastTestLine
|
||||
}
|
||||
// Last resort: first non-empty line of the body.
|
||||
for _, raw := range lines {
|
||||
l := strings.TrimSpace(raw)
|
||||
if l != "" {
|
||||
return l
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// errorTraceRe extracts the path from testify's "Error Trace:" line — e.g.
|
||||
//
|
||||
// Error Trace: tests/e2e/harness/ec2helpers.go:164
|
||||
//
|
||||
// — which points at the assertion site even when it lives in helper code
|
||||
// outside *_test.go.
|
||||
var errorTraceRe = regexp.MustCompile(`Error Trace:\s*(\S+\.go:\d+)`)
|
||||
|
||||
// extractFileHint returns the most precise pointer at the assertion site
|
||||
// we can pull from the failure body: testify Error Trace wins, otherwise
|
||||
// the last `*_test.go:NNN` line in the body.
|
||||
func extractFileHint(body string) string {
|
||||
body = stripANSI(body)
|
||||
if m := errorTraceRe.FindStringSubmatch(body); m != nil {
|
||||
return m[1]
|
||||
}
|
||||
all := fileHintRe.FindAllString(body, -1)
|
||||
if len(all) == 0 {
|
||||
return ""
|
||||
}
|
||||
return all[len(all)-1]
|
||||
}
|
||||
|
||||
// signature normalises an error line so different runs of the same root
|
||||
// cause bucket together. Empty signatures (e.g. parent-test placeholders
|
||||
// where the failure body is empty) bucket together under "".
|
||||
func signature(errLine string) string {
|
||||
s := strings.ToLower(errLine)
|
||||
for _, r := range noiseReplacers {
|
||||
s = r.re.ReplaceAllString(s, r.repl)
|
||||
}
|
||||
return strings.TrimSpace(s)
|
||||
}
|
||||
|
||||
// isCascade reports whether the error line contains any of the well-known
|
||||
// fixture-propagation markers — i.e. "this test failed because something
|
||||
// upstream of it failed first".
|
||||
func isCascade(errLine string) bool {
|
||||
for _, m := range cascadeMarkers {
|
||||
if strings.Contains(errLine, m) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// computeLeafSet returns the testcase names that are leaves — i.e. no
|
||||
// other testcase name in `cases` starts with `<name>/`. go-junit-report
|
||||
// emits one <testcase> for the parent Go test AND each subtest, with the
|
||||
// parent's time = sum of its children, so summing every entry into the
|
||||
// suite's cumulative offset double-counts. Restrict accumulation to
|
||||
// leaves only.
|
||||
func computeLeafSet(cases []junitTC) map[string]bool {
|
||||
leaf := make(map[string]bool, len(cases))
|
||||
for _, tc := range cases {
|
||||
leaf[tc.Name] = true
|
||||
}
|
||||
for _, tc := range cases {
|
||||
// Strip the deepest path segment; any ancestor of this test is
|
||||
// not a leaf.
|
||||
name := tc.Name
|
||||
for {
|
||||
i := strings.LastIndex(name, "/")
|
||||
if i <= 0 {
|
||||
break
|
||||
}
|
||||
name = name[:i]
|
||||
if _, ok := leaf[name]; ok {
|
||||
leaf[name] = false
|
||||
}
|
||||
}
|
||||
}
|
||||
return leaf
|
||||
}
|
||||
|
||||
// parseStartTime turns a junit timestamp attr (RFC3339) into a time.Time;
|
||||
// zero value if unparseable.
|
||||
func parseStartTime(s string) time.Time {
|
||||
if s == "" {
|
||||
return time.Time{}
|
||||
}
|
||||
t, err := time.Parse(time.RFC3339, s)
|
||||
if err != nil {
|
||||
return time.Time{}
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
// suiteLabel turns "junit-single.xml" into "single".
|
||||
func suiteLabel(path string) string {
|
||||
base := filepath.Base(path)
|
||||
base = strings.TrimSuffix(base, ".xml")
|
||||
base = strings.TrimPrefix(base, "junit-")
|
||||
if base == "" {
|
||||
return filepath.Base(path)
|
||||
}
|
||||
return base
|
||||
}
|
||||
|
||||
// ParseFile decodes one junit-*.xml file into our normalized failure list.
|
||||
// Returns the suite label, total testcase count, and the (ordered) failures.
|
||||
func ParseFile(path string, data []byte) (SuiteReport, error) {
|
||||
var doc junitSuites
|
||||
if err := xml.Unmarshal(data, &doc); err != nil {
|
||||
return SuiteReport{}, fmt.Errorf("parse %s: %w", path, err)
|
||||
}
|
||||
|
||||
rep := SuiteReport{
|
||||
File: filepath.Base(path),
|
||||
Label: suiteLabel(path),
|
||||
}
|
||||
|
||||
order := 0
|
||||
for _, suite := range doc.Suites {
|
||||
start := parseStartTime(suite.Timestamp)
|
||||
// go-junit-report emits a testcase for both the parent Go test
|
||||
// (e.g. `TestSingleNode`) AND each subtest (`TestSingleNode/PhaseX`).
|
||||
// The parent's `time` attribute is the aggregate of its children, so
|
||||
// summing every testcase double-counts. Pre-compute the leaf set so
|
||||
// only leaves contribute to the suite's running offset.
|
||||
isLeaf := computeLeafSet(suite.Cases)
|
||||
cumul := 0.0
|
||||
for _, tc := range suite.Cases {
|
||||
rep.Total++
|
||||
leaf := isLeaf[tc.Name]
|
||||
var body string
|
||||
switch {
|
||||
case tc.Failure != nil:
|
||||
body = tc.Failure.Body
|
||||
case tc.Error != nil:
|
||||
body = tc.Error.Body
|
||||
default:
|
||||
if leaf {
|
||||
cumul += tc.Time
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
// Skip the synthetic parent-test "Failed" stub that go-junit-report
|
||||
// emits for any Go test whose subtests failed. It has an empty body
|
||||
// and would otherwise outrank real failures by XML order.
|
||||
trimmed := strings.TrimSpace(stripANSI(body))
|
||||
if trimmed == "" {
|
||||
if leaf {
|
||||
cumul += tc.Time
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
errLine := extractErrorLine(body)
|
||||
offset := time.Duration(cumul * float64(time.Second))
|
||||
f := Failure{
|
||||
SuiteFile: rep.Label,
|
||||
Name: tc.Name,
|
||||
Duration: tc.Time,
|
||||
Order: order,
|
||||
Error: errLine,
|
||||
FileHint: extractFileHint(body),
|
||||
Cascade: isCascade(errLine),
|
||||
Signature: signature(errLine),
|
||||
OffsetFromSuiteStart: offset,
|
||||
}
|
||||
if !start.IsZero() {
|
||||
f.StartAt = start.Add(offset)
|
||||
}
|
||||
rep.FailCount++
|
||||
rep.Cascades = append(rep.Cascades, f) // temp staging; split below
|
||||
order++
|
||||
if leaf {
|
||||
cumul += tc.Time
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if rep.FailCount == 0 {
|
||||
rep.Cascades = nil
|
||||
return rep, nil
|
||||
}
|
||||
|
||||
all := rep.Cascades
|
||||
rep.Cascades = nil
|
||||
|
||||
// Root = earliest non-cascade. If every failure is a cascade (the whole
|
||||
// suite tripped on the same upstream gap), fall back to earliest overall.
|
||||
rootIdx := -1
|
||||
for i, f := range all {
|
||||
if !f.Cascade {
|
||||
rootIdx = i
|
||||
break
|
||||
}
|
||||
}
|
||||
if rootIdx == -1 {
|
||||
rootIdx = 0
|
||||
}
|
||||
root := all[rootIdx]
|
||||
rep.Root = &root
|
||||
|
||||
// Bucket the remainder by signature so the report can collapse the
|
||||
// "ten failures, same bug" pattern.
|
||||
bucketKeys := []string{}
|
||||
buckets := map[string][]Failure{}
|
||||
for i, f := range all {
|
||||
if i == rootIdx {
|
||||
continue
|
||||
}
|
||||
if _, seen := buckets[f.Signature]; !seen {
|
||||
bucketKeys = append(bucketKeys, f.Signature)
|
||||
}
|
||||
buckets[f.Signature] = append(buckets[f.Signature], f)
|
||||
rep.Cascades = append(rep.Cascades, f)
|
||||
}
|
||||
|
||||
// Largest bucket first; ties broken by first-seen order so output is
|
||||
// deterministic across runs.
|
||||
sort.SliceStable(bucketKeys, func(i, j int) bool {
|
||||
return len(buckets[bucketKeys[i]]) > len(buckets[bucketKeys[j]])
|
||||
})
|
||||
for _, k := range bucketKeys {
|
||||
rep.Buckets = append(rep.Buckets, buckets[k])
|
||||
}
|
||||
|
||||
return rep, nil
|
||||
}
|
||||
|
||||
// Render writes the markdown report. Format is documented in
|
||||
// docs/development/improvements/e2e-go-failure-analysis.md (Stage 1).
|
||||
func Render(r Report) string {
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "## %s\n\n", r.Title)
|
||||
|
||||
totalFail := 0
|
||||
for _, s := range r.Suites {
|
||||
totalFail += s.FailCount
|
||||
}
|
||||
if totalFail == 0 {
|
||||
b.WriteString("✅ No failures across any suite.\n")
|
||||
return b.String()
|
||||
}
|
||||
|
||||
for _, s := range r.Suites {
|
||||
if s.FailCount == 0 {
|
||||
fmt.Fprintf(&b, "### Suite `%s`: ✅ pass (%d tests)\n\n", s.Label, s.Total)
|
||||
continue
|
||||
}
|
||||
fmt.Fprintf(&b, "### Suite `%s`: %d failed, 1 root cause likely\n\n", s.Label, s.FailCount)
|
||||
|
||||
if s.Root != nil {
|
||||
b.WriteString("**Root cause (earliest non-cascade)**\n\n")
|
||||
fmt.Fprintf(&b, "- Test: `%s`\n", s.Root.Name)
|
||||
if !s.Root.StartAt.IsZero() {
|
||||
fmt.Fprintf(&b, "- Start: %s (duration %.1fs)\n",
|
||||
s.Root.StartAt.Format("15:04:05"), s.Root.Duration)
|
||||
} else {
|
||||
fmt.Fprintf(&b, "- Duration: %.1fs\n", s.Root.Duration)
|
||||
}
|
||||
if s.Root.Error != "" {
|
||||
fmt.Fprintf(&b, "- Error: %s\n", oneLine(s.Root.Error))
|
||||
}
|
||||
if s.Root.FileHint != "" {
|
||||
fmt.Fprintf(&b, "- File: `%s`\n", s.Root.FileHint)
|
||||
}
|
||||
if s.Root.BundlePath != "" {
|
||||
fmt.Fprintf(&b, "- Bundle: `%s/`\n", s.Root.BundlePath)
|
||||
}
|
||||
b.WriteString("\n")
|
||||
}
|
||||
|
||||
if len(s.Buckets) > 0 {
|
||||
fmt.Fprintf(&b, "**Cascaded failures (%d) — grouped by signature**\n\n",
|
||||
len(s.Cascades))
|
||||
for _, bucket := range s.Buckets {
|
||||
sample := bucket[0]
|
||||
label := oneLine(sample.Error)
|
||||
if label == "" {
|
||||
label = "(no error message)"
|
||||
}
|
||||
fmt.Fprintf(&b, "- %d× %s\n", len(bucket), label)
|
||||
// List up to 5 test names per bucket; collapse the rest.
|
||||
const cap = 5
|
||||
for i, f := range bucket {
|
||||
if i == cap && len(bucket) > cap {
|
||||
fmt.Fprintf(&b, " - … and %d more\n", len(bucket)-cap)
|
||||
break
|
||||
}
|
||||
if f.BundlePath != "" {
|
||||
fmt.Fprintf(&b, " - `%s` → `%s/`\n", f.Name, f.BundlePath)
|
||||
} else {
|
||||
fmt.Fprintf(&b, " - `%s`\n", f.Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
b.WriteString("\n")
|
||||
}
|
||||
}
|
||||
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// oneLine collapses whitespace and caps length so a runaway assertion
|
||||
// message doesn't blow out the summary card width.
|
||||
func oneLine(s string) string {
|
||||
s = strings.TrimSpace(s)
|
||||
s = strings.ReplaceAll(s, "\n", " ")
|
||||
s = strings.ReplaceAll(s, "\r", " ")
|
||||
s = regexp.MustCompile(`\s+`).ReplaceAllString(s, " ")
|
||||
const max = 160
|
||||
if len(s) > max {
|
||||
s = s[:max-1] + "…"
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,191 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// updateGolden lets `go test ./.github/actions/e2e-analyze/ -update` rewrite
|
||||
// the expected analysis.md files when the rendering format changes
|
||||
// intentionally. Day-to-day, leave it unset: tests diff against the on-disk
|
||||
// snapshot to catch unintended drift in signature extraction or layout.
|
||||
var updateGolden = flag.Bool("update", false, "rewrite testdata/*/analysis.md from current output")
|
||||
|
||||
// TestGoldenFixtures runs the full ParseFile + Render pipeline against each
|
||||
// fixture directory under testdata/ and asserts the output matches the
|
||||
// adjacent analysis.md snapshot. Each subdir is one named scenario; the
|
||||
// fixture XMLs match the same junit-*.xml glob the production action uses
|
||||
// so behaviour stays representative of CI.
|
||||
func TestGoldenFixtures(t *testing.T) {
|
||||
entries, err := os.ReadDir("testdata")
|
||||
if err != nil {
|
||||
t.Fatalf("read testdata: %v", err)
|
||||
}
|
||||
for _, e := range entries {
|
||||
if !e.IsDir() {
|
||||
continue
|
||||
}
|
||||
name := e.Name()
|
||||
t.Run(name, func(t *testing.T) {
|
||||
dir := filepath.Join("testdata", name)
|
||||
matches, err := filepath.Glob(filepath.Join(dir, "junit-*.xml"))
|
||||
if err != nil {
|
||||
t.Fatalf("glob: %v", err)
|
||||
}
|
||||
sort.Strings(matches)
|
||||
if len(matches) == 0 {
|
||||
t.Fatalf("no junit-*.xml fixtures in %s", dir)
|
||||
}
|
||||
|
||||
rep := Report{Title: "E2E failure analysis"}
|
||||
for _, p := range matches {
|
||||
data, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
t.Fatalf("read %s: %v", p, err)
|
||||
}
|
||||
sr, err := ParseFile(p, data)
|
||||
if err != nil {
|
||||
t.Fatalf("parse %s: %v", p, err)
|
||||
}
|
||||
rep.Suites = append(rep.Suites, sr)
|
||||
}
|
||||
|
||||
got := Render(rep)
|
||||
goldenPath := filepath.Join(dir, "analysis.md")
|
||||
if *updateGolden {
|
||||
if err := os.WriteFile(goldenPath, []byte(got), 0o644); err != nil {
|
||||
t.Fatalf("write golden: %v", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
want, err := os.ReadFile(goldenPath)
|
||||
if err != nil {
|
||||
t.Fatalf("read golden (run with -update to seed): %v", err)
|
||||
}
|
||||
if string(want) != got {
|
||||
t.Errorf("rendered output differs from %s\n--- want ---\n%s\n--- got ---\n%s",
|
||||
goldenPath, string(want), got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestSignatureCollapsesNoise(t *testing.T) {
|
||||
a := signature("describe-vpcs: InvalidParameterValue request-id: abc-123 vpc-deadbeefcafe1234")
|
||||
b := signature("describe-vpcs: InvalidParameterValue request-id: xyz-999 vpc-1111222233334444")
|
||||
if a != b {
|
||||
t.Errorf("signatures should collapse to same bucket:\n a=%q\n b=%q", a, b)
|
||||
}
|
||||
if !strings.Contains(a, "<id>") || !strings.Contains(a, "vpc-<id>") {
|
||||
t.Errorf("expected normalised placeholders in signature, got %q", a)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCascadeDetection(t *testing.T) {
|
||||
cases := []struct {
|
||||
in string
|
||||
want bool
|
||||
}{
|
||||
{"Phase 5 must populate fix.InstanceID", true},
|
||||
{"Should NOT be empty", true},
|
||||
{"Expected value not to be nil", true},
|
||||
{"describe-vpcs: InvalidParameterValue", false},
|
||||
{"Eventually: condition not met within 3m0s", false},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := isCascade(c.in); got != c.want {
|
||||
t.Errorf("isCascade(%q) = %v, want %v", c.in, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractErrorLine_MessagesWinsOverError(t *testing.T) {
|
||||
body := ` lifecycle_test.go:28:
|
||||
Error Trace: tests/e2e/single/lifecycle_test.go:28
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase5a_pre_ClusterStats`
|
||||
got := extractErrorLine(body)
|
||||
want := "Phase 5 must populate fix.InstanceID"
|
||||
if got != want {
|
||||
t.Errorf("extractErrorLine = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractErrorLine_FallsBackToLastTestLine(t *testing.T) {
|
||||
body := " vpc_test.go:227: Eventually: condition not met within 3m0s: [SSH handshake never completed]"
|
||||
got := extractErrorLine(body)
|
||||
if !strings.HasPrefix(got, "Eventually:") {
|
||||
t.Errorf("expected fallback to test-line content, got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractFileHint_PicksLastMatch(t *testing.T) {
|
||||
body := "ec2helpers.go:50: setup\n vpc_test.go:227: Eventually: condition not met"
|
||||
got := extractFileHint(body)
|
||||
want := "vpc_test.go:227"
|
||||
if got != want {
|
||||
t.Errorf("extractFileHint = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseFile_SkipsEmptyParentFailure(t *testing.T) {
|
||||
// The synthetic "TestSingleNode" parent gets a <failure message="Failed"/>
|
||||
// with no body whenever any subtest fails. The analyzer must skip it so
|
||||
// it doesn't outrank the real first failure on XML order.
|
||||
xml := []byte(`<?xml version="1.0"?>
|
||||
<testsuites><testsuite name="" tests="2" failures="2">
|
||||
<testcase name="TestX" time="1.0"><failure message="Failed"></failure></testcase>
|
||||
<testcase name="TestX/RealFailure" time="0.5"><failure message="Failed"><![CDATA[
|
||||
foo_test.go:10:
|
||||
Messages: real assertion message
|
||||
]]></failure></testcase>
|
||||
</testsuite></testsuites>`)
|
||||
sr, err := ParseFile("junit-x.xml", xml)
|
||||
if err != nil {
|
||||
t.Fatalf("ParseFile: %v", err)
|
||||
}
|
||||
if sr.FailCount != 1 {
|
||||
t.Fatalf("FailCount = %d, want 1 (empty parent should be skipped)", sr.FailCount)
|
||||
}
|
||||
if sr.Root == nil || sr.Root.Name != "TestX/RealFailure" {
|
||||
t.Fatalf("expected real failure as root, got %+v", sr.Root)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseFile_CascadeFallbackWhenAllCascades(t *testing.T) {
|
||||
// If every failure is a cascade marker, the earliest one becomes root
|
||||
// so the report isn't blank.
|
||||
xml := []byte(`<?xml version="1.0"?>
|
||||
<testsuites><testsuite name="" tests="2" failures="2">
|
||||
<testcase name="TestX/A" time="1.0"><failure message="Failed"><![CDATA[
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
]]></failure></testcase>
|
||||
<testcase name="TestX/B" time="1.0"><failure message="Failed"><![CDATA[
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
]]></failure></testcase>
|
||||
</testsuite></testsuites>`)
|
||||
sr, err := ParseFile("junit-x.xml", xml)
|
||||
if err != nil {
|
||||
t.Fatalf("ParseFile: %v", err)
|
||||
}
|
||||
if sr.Root == nil || sr.Root.Name != "TestX/A" {
|
||||
t.Fatalf("expected TestX/A as fallback root, got %+v", sr.Root)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRender_NoFailuresIsClean(t *testing.T) {
|
||||
out := Render(Report{
|
||||
Title: "E2E failure analysis",
|
||||
Suites: []SuiteReport{
|
||||
{File: "junit-single.xml", Label: "single", Total: 3, FailCount: 0},
|
||||
},
|
||||
})
|
||||
if !strings.Contains(out, "No failures") {
|
||||
t.Errorf("expected zero-failure banner, got:\n%s", out)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,208 @@
|
||||
package main
|
||||
|
||||
// Stage 2 of docs/development/improvements/e2e-go-failure-analysis.md:
|
||||
// for each failure surfaced by Stage 1, materialise a per-failure bundle
|
||||
// directory containing the failure summary, a time-window slice of the
|
||||
// daemon journal, a testcase-window slice of the suite's stdout log, and
|
||||
// any per-test artifacts the harness already dumped at run-time.
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// bundlePad is the per-failure window padding either side of the
|
||||
// [StartAt, StartAt+Duration] testcase span. Five seconds matches the
|
||||
// plan doc: long enough to catch the daemon's pre-test setup and
|
||||
// post-test cleanup chatter, short enough to keep the slice focused.
|
||||
const bundlePad = 5 * time.Second
|
||||
|
||||
// nameSanitiseRe replaces characters unsafe in directory names with `_`.
|
||||
// Tracks tests/e2e/harness/artifacts.go which uses the same convention
|
||||
// when the test's own dump dir is created, so we can pick its contents
|
||||
// up by path without a second translation table.
|
||||
var nameSanitiseRe = regexp.MustCompile(`[^A-Za-z0-9._-]+`)
|
||||
|
||||
func sanitiseTestName(s string) string {
|
||||
return nameSanitiseRe.ReplaceAllString(s, "_")
|
||||
}
|
||||
|
||||
// ApplySuiteStartFiles overrides each failure's StartAt by reading
|
||||
// <logDir>/test-<suiteLabel>.start (RFC3339 timestamp written by the
|
||||
// workflow before the suite runs). Required because go-junit-report's
|
||||
// `timestamp` attribute on <testsuite> records the time the XML was
|
||||
// produced — i.e. after the suite has finished — so it can't be used
|
||||
// as the suite wall-clock origin for journal slicing.
|
||||
//
|
||||
// Missing or unparseable files are silently ignored: the analyzer
|
||||
// falls back to the junit timestamp and the bundle still gets created,
|
||||
// it just may have an empty (or off-window) journal slice.
|
||||
func ApplySuiteStartFiles(rep *Report, logDir string) {
|
||||
if logDir == "" {
|
||||
return
|
||||
}
|
||||
for si := range rep.Suites {
|
||||
s := &rep.Suites[si]
|
||||
startPath := filepath.Join(logDir, fmt.Sprintf("test-%s.start", s.Label))
|
||||
data, err := os.ReadFile(startPath)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
ts, err := time.Parse(time.RFC3339, strings.TrimSpace(string(data)))
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: parse %s: %v\n", startPath, err)
|
||||
continue
|
||||
}
|
||||
if s.Root != nil {
|
||||
s.Root.StartAt = ts.Add(s.Root.OffsetFromSuiteStart)
|
||||
}
|
||||
for ci := range s.Cascades {
|
||||
s.Cascades[ci].StartAt = ts.Add(s.Cascades[ci].OffsetFromSuiteStart)
|
||||
}
|
||||
for bi := range s.Buckets {
|
||||
for fi := range s.Buckets[bi] {
|
||||
f := &s.Buckets[bi][fi]
|
||||
f.StartAt = ts.Add(f.OffsetFromSuiteStart)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// WriteBundles materialises per-failure bundles under <logDir>/analysis/.
|
||||
// Mutates rep in-place so each Failure.BundlePath holds the relative
|
||||
// path the renderer will link from the summary.
|
||||
//
|
||||
// Failures with no parseable wall-clock start still get a bundle dir
|
||||
// (and the testcase-window test.log slice), but no journal slice — the
|
||||
// time window is unknown.
|
||||
//
|
||||
// Errors are non-fatal: a bundle that partially fails still gets linked
|
||||
// from the report so a reader sees what's available, and the analyzer
|
||||
// keeps printing warnings so CI logs surface the gap.
|
||||
func WriteBundles(rep *Report, logDir string) {
|
||||
if logDir == "" {
|
||||
return
|
||||
}
|
||||
base := filepath.Join(logDir, "analysis")
|
||||
if err := os.MkdirAll(base, 0o755); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: mkdir %s: %v\n", base, err)
|
||||
return
|
||||
}
|
||||
|
||||
for si := range rep.Suites {
|
||||
s := &rep.Suites[si]
|
||||
if s.Root != nil {
|
||||
writeOneBundle(logDir, base, s.Label, s.Root)
|
||||
}
|
||||
for ci := range s.Cascades {
|
||||
writeOneBundle(logDir, base, s.Label, &s.Cascades[ci])
|
||||
}
|
||||
// Bucket entries share storage with Cascades but live in a
|
||||
// separate slice in SuiteReport; mirror the bundle path so the
|
||||
// renderer's bucket loop sees it too.
|
||||
for bi := range s.Buckets {
|
||||
for fi := range s.Buckets[bi] {
|
||||
f := &s.Buckets[bi][fi]
|
||||
if bp := findBundlePath(s, f.Name); bp != "" {
|
||||
f.BundlePath = bp
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// findBundlePath looks up a Failure's already-written BundlePath by test
|
||||
// name within a suite. Buckets reference the same logical failure as
|
||||
// Cascades but are independent slice values, so we copy the path across
|
||||
// rather than re-deriving it.
|
||||
func findBundlePath(s *SuiteReport, testName string) string {
|
||||
if s.Root != nil && s.Root.Name == testName {
|
||||
return s.Root.BundlePath
|
||||
}
|
||||
for _, c := range s.Cascades {
|
||||
if c.Name == testName {
|
||||
return c.BundlePath
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func writeOneBundle(logDir, base, suiteLabel string, f *Failure) {
|
||||
name := fmt.Sprintf("%02d-%s", f.Order+1, sanitiseTestName(f.Name))
|
||||
dir := filepath.Join(base, name)
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: mkdir %s: %v\n", dir, err)
|
||||
return
|
||||
}
|
||||
f.BundlePath = filepath.Join("analysis", name)
|
||||
|
||||
writeFailureSummary(dir, suiteLabel, f)
|
||||
|
||||
if !f.StartAt.IsZero() {
|
||||
start := f.StartAt.Add(-bundlePad)
|
||||
end := f.StartAt.Add(time.Duration(f.Duration * float64(time.Second))).Add(bundlePad)
|
||||
journalSrc := filepath.Join(logDir, "spinifex-journal.log")
|
||||
if _, err := SliceJournal(journalSrc, start, end, filepath.Join(dir, "journal.log")); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: slice journal for %s: %v\n", f.Name, err)
|
||||
}
|
||||
}
|
||||
|
||||
testLogSrc := filepath.Join(logDir, fmt.Sprintf("test-%s.log", suiteLabel))
|
||||
if _, err := SliceTestLog(testLogSrc, f.Name, filepath.Join(dir, "test.log")); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: slice test log for %s: %v\n", f.Name, err)
|
||||
}
|
||||
|
||||
// Per-test artifact subdir written by harness.ArtifactDir lives at
|
||||
// <logDir>/<TestName with / replaced by _>/. Copy any files into
|
||||
// the bundle so qemu/ovs/etc. dumps already collected by the test
|
||||
// travel alongside the analysis.
|
||||
perTestSrc := filepath.Join(logDir, strings.ReplaceAll(f.Name, "/", "_"))
|
||||
if entries, err := os.ReadDir(perTestSrc); err == nil {
|
||||
for _, e := range entries {
|
||||
if e.IsDir() {
|
||||
continue
|
||||
}
|
||||
_ = copyFile(filepath.Join(perTestSrc, e.Name()), filepath.Join(dir, e.Name()))
|
||||
}
|
||||
} else if !os.IsNotExist(err) && !errorsIsNotDir(err) {
|
||||
fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: read per-test dir %s: %v\n", perTestSrc, err)
|
||||
}
|
||||
}
|
||||
|
||||
// errorsIsNotDir handles the case where the harness wrote a file at the
|
||||
// per-test name instead of a directory — best-effort skip.
|
||||
func errorsIsNotDir(err error) bool {
|
||||
var pe *fs.PathError
|
||||
if !errors.As(err, &pe) {
|
||||
return false
|
||||
}
|
||||
return pe.Err != nil && strings.Contains(pe.Err.Error(), "not a directory")
|
||||
}
|
||||
|
||||
func writeFailureSummary(dir, suiteLabel string, f *Failure) {
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "Suite: %s\n", suiteLabel)
|
||||
fmt.Fprintf(&b, "Test: %s\n", f.Name)
|
||||
if !f.StartAt.IsZero() {
|
||||
fmt.Fprintf(&b, "Start: %s\n", f.StartAt.Format(time.RFC3339))
|
||||
}
|
||||
fmt.Fprintf(&b, "Duration: %.3fs\n", f.Duration)
|
||||
if f.FileHint != "" {
|
||||
fmt.Fprintf(&b, "Site: %s\n", f.FileHint)
|
||||
}
|
||||
if f.Cascade {
|
||||
b.WriteString("Cascade: yes (downstream of another failure)\n")
|
||||
}
|
||||
if f.Error != "" {
|
||||
b.WriteString("\nError:\n")
|
||||
b.WriteString(oneLine(f.Error))
|
||||
b.WriteString("\n")
|
||||
}
|
||||
_ = os.WriteFile(filepath.Join(dir, "failure.txt"), []byte(b.String()), 0o644)
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestSliceJournal_KeepsOnlyWindow(t *testing.T) {
|
||||
src := filepath.Join(t.TempDir(), "journal.log")
|
||||
dst := filepath.Join(t.TempDir(), "out.log")
|
||||
body := strings.Join([]string{
|
||||
"2026-05-19T12:32:30+1000 host spinifex-daemon[1]: pre",
|
||||
" continuation of pre",
|
||||
"2026-05-19T12:32:35+1000 host spinifex-daemon[1]: IN window",
|
||||
" multiline inside window",
|
||||
"2026-05-19T12:32:45+1000 host spinifex-daemon[1]: post",
|
||||
"",
|
||||
}, "\n")
|
||||
if err := os.WriteFile(src, []byte(body), 0o644); err != nil {
|
||||
t.Fatalf("write src: %v", err)
|
||||
}
|
||||
start, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:33+10:00")
|
||||
end, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:40+10:00")
|
||||
n, err := SliceJournal(src, start, end, dst)
|
||||
if err != nil {
|
||||
t.Fatalf("SliceJournal: %v", err)
|
||||
}
|
||||
if n != 2 {
|
||||
t.Fatalf("written = %d, want 2", n)
|
||||
}
|
||||
got, _ := os.ReadFile(dst)
|
||||
if !strings.Contains(string(got), "IN window") || !strings.Contains(string(got), "multiline inside window") {
|
||||
t.Errorf("missing in-window content:\n%s", got)
|
||||
}
|
||||
if strings.Contains(string(got), "pre") || strings.Contains(string(got), "post") {
|
||||
t.Errorf("kept out-of-window content:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSliceJournal_AcceptsRFC3339ColonOffset(t *testing.T) {
|
||||
// systemd's `journalctl --output=short-iso` ships RFC3339 timestamps
|
||||
// (offset includes a colon, e.g. "+10:00") on most versions. Earlier
|
||||
// regex only matched "+1000" — make sure both forms work.
|
||||
src := filepath.Join(t.TempDir(), "journal.log")
|
||||
dst := filepath.Join(t.TempDir(), "out.log")
|
||||
body := strings.Join([]string{
|
||||
"2026-05-19T12:32:30+10:00 host spinifex-daemon[1]: pre",
|
||||
"2026-05-19T12:32:35+10:00 host spinifex-daemon[1]: IN window",
|
||||
"2026-05-19T12:32:50+10:00 host spinifex-daemon[1]: post",
|
||||
"",
|
||||
}, "\n")
|
||||
if err := os.WriteFile(src, []byte(body), 0o644); err != nil {
|
||||
t.Fatalf("write src: %v", err)
|
||||
}
|
||||
start, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:33+10:00")
|
||||
end, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:40+10:00")
|
||||
n, err := SliceJournal(src, start, end, dst)
|
||||
if err != nil {
|
||||
t.Fatalf("SliceJournal: %v", err)
|
||||
}
|
||||
if n != 1 {
|
||||
t.Fatalf("written = %d, want 1", n)
|
||||
}
|
||||
got, _ := os.ReadFile(dst)
|
||||
if !strings.Contains(string(got), "IN window") {
|
||||
t.Errorf("missing in-window content:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSliceJournal_MissingSrcIsNotError(t *testing.T) {
|
||||
dst := filepath.Join(t.TempDir(), "out.log")
|
||||
n, err := SliceJournal("/no/such/file", time.Now(), time.Now(), dst)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected err: %v", err)
|
||||
}
|
||||
if n != 0 {
|
||||
t.Errorf("written = %d, want 0", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSliceTestLog_PicksOneTestcase(t *testing.T) {
|
||||
src := filepath.Join(t.TempDir(), "test.log")
|
||||
dst := filepath.Join(t.TempDir(), "out.log")
|
||||
body := strings.Join([]string{
|
||||
"=== RUN TestSingleNode",
|
||||
" before subtests",
|
||||
"=== RUN TestSingleNode/Phase5_LaunchInstance",
|
||||
" runinstances",
|
||||
" ec2helpers.go:164: describe-vpcs: InvalidParameterValue",
|
||||
"--- FAIL: TestSingleNode/Phase5_LaunchInstance (2.50s)",
|
||||
"=== RUN TestSingleNode/Phase6_Volumes",
|
||||
" other content",
|
||||
"--- FAIL: TestSingleNode/Phase6_Volumes (0.10s)",
|
||||
"--- FAIL: TestSingleNode (10.0s)",
|
||||
"",
|
||||
}, "\n")
|
||||
if err := os.WriteFile(src, []byte(body), 0o644); err != nil {
|
||||
t.Fatalf("write src: %v", err)
|
||||
}
|
||||
n, err := SliceTestLog(src, "TestSingleNode/Phase5_LaunchInstance", dst)
|
||||
if err != nil {
|
||||
t.Fatalf("SliceTestLog: %v", err)
|
||||
}
|
||||
if n != 4 {
|
||||
t.Fatalf("written = %d, want 4 (RUN + 2 inner + FAIL)", n)
|
||||
}
|
||||
got, _ := os.ReadFile(dst)
|
||||
s := string(got)
|
||||
if !strings.Contains(s, "describe-vpcs: InvalidParameterValue") {
|
||||
t.Errorf("missing failure line:\n%s", s)
|
||||
}
|
||||
if strings.Contains(s, "Phase6_Volumes") || strings.Contains(s, "before subtests") {
|
||||
t.Errorf("leaked sibling test content:\n%s", s)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSliceTestLog_UnknownTestcaseIsEmpty(t *testing.T) {
|
||||
src := filepath.Join(t.TempDir(), "test.log")
|
||||
dst := filepath.Join(t.TempDir(), "out.log")
|
||||
if err := os.WriteFile(src, []byte("=== RUN TestX\n--- PASS: TestX (0.0s)\n"), 0o644); err != nil {
|
||||
t.Fatalf("write src: %v", err)
|
||||
}
|
||||
n, err := SliceTestLog(src, "TestY", dst)
|
||||
if err != nil {
|
||||
t.Fatalf("SliceTestLog: %v", err)
|
||||
}
|
||||
if n != 0 {
|
||||
t.Errorf("written = %d, want 0", n)
|
||||
}
|
||||
if _, err := os.Stat(dst); err != nil {
|
||||
t.Errorf("dst not created: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestWriteBundles_EndToEnd builds a fake artifact dir with a junit XML,
|
||||
// a journal log, a test log, and a per-test artifact subdir, then runs
|
||||
// the full ParseFile → WriteBundles pipeline and asserts the bundle dir
|
||||
// contains everything the report links to.
|
||||
func TestWriteBundles_EndToEnd(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
|
||||
junit := `<?xml version="1.0"?>
|
||||
<testsuites><testsuite name="" tests="2" failures="1" timestamp="2026-05-19T12:32:30+10:00" time="3.0">
|
||||
<testcase name="TestSingleNode/Phase5_LaunchInstance" time="2.500">
|
||||
<failure message="Failed"><![CDATA[
|
||||
ec2helpers.go:164: describe-vpcs: InvalidParameterValue (status 400)
|
||||
Error Trace: tests/e2e/harness/ec2helpers.go:164
|
||||
Test: TestSingleNode/Phase5_LaunchInstance]]></failure>
|
||||
</testcase>
|
||||
</testsuite></testsuites>`
|
||||
if err := os.WriteFile(filepath.Join(dir, "junit-single.xml"), []byte(junit), 0o644); err != nil {
|
||||
t.Fatalf("write junit: %v", err)
|
||||
}
|
||||
|
||||
journal := strings.Join([]string{
|
||||
"2026-05-19T12:32:20+1000 host spinifex-daemon[1]: too-early",
|
||||
"2026-05-19T12:32:30+1000 host spinifex-daemon[1]: in-window-start",
|
||||
"2026-05-19T12:32:32+1000 host spinifex-daemon[1]: describe-vpcs filter=is_default",
|
||||
"2026-05-19T12:32:50+1000 host spinifex-daemon[1]: too-late",
|
||||
"",
|
||||
}, "\n")
|
||||
if err := os.WriteFile(filepath.Join(dir, "spinifex-journal.log"), []byte(journal), 0o644); err != nil {
|
||||
t.Fatalf("write journal: %v", err)
|
||||
}
|
||||
|
||||
testlog := strings.Join([]string{
|
||||
"=== RUN TestSingleNode/Phase5_LaunchInstance",
|
||||
" ec2helpers.go:164: describe-vpcs: InvalidParameterValue",
|
||||
"--- FAIL: TestSingleNode/Phase5_LaunchInstance (2.50s)",
|
||||
"",
|
||||
}, "\n")
|
||||
if err := os.WriteFile(filepath.Join(dir, "test-single.log"), []byte(testlog), 0o644); err != nil {
|
||||
t.Fatalf("write test log: %v", err)
|
||||
}
|
||||
|
||||
// Per-test artifact subdir (matches harness.ArtifactDir convention).
|
||||
perTest := filepath.Join(dir, "TestSingleNode_Phase5_LaunchInstance")
|
||||
if err := os.MkdirAll(perTest, 0o755); err != nil {
|
||||
t.Fatalf("mkdir per-test: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(perTest, "qemu-console.log"), []byte("[qemu] boot\n"), 0o644); err != nil {
|
||||
t.Fatalf("write per-test file: %v", err)
|
||||
}
|
||||
|
||||
data, _ := os.ReadFile(filepath.Join(dir, "junit-single.xml"))
|
||||
sr, err := ParseFile("junit-single.xml", data)
|
||||
if err != nil {
|
||||
t.Fatalf("ParseFile: %v", err)
|
||||
}
|
||||
rep := Report{Title: "E2E failure analysis", Suites: []SuiteReport{sr}}
|
||||
WriteBundles(&rep, dir)
|
||||
|
||||
if rep.Suites[0].Root == nil {
|
||||
t.Fatal("no root")
|
||||
}
|
||||
bp := rep.Suites[0].Root.BundlePath
|
||||
if bp == "" {
|
||||
t.Fatal("BundlePath not set")
|
||||
}
|
||||
bundle := filepath.Join(dir, bp)
|
||||
for _, want := range []string{"failure.txt", "journal.log", "test.log", "qemu-console.log"} {
|
||||
if _, err := os.Stat(filepath.Join(bundle, want)); err != nil {
|
||||
t.Errorf("missing %s in bundle %s: %v", want, bp, err)
|
||||
}
|
||||
}
|
||||
|
||||
jl, _ := os.ReadFile(filepath.Join(bundle, "journal.log"))
|
||||
if !strings.Contains(string(jl), "describe-vpcs filter=is_default") {
|
||||
t.Errorf("journal.log lacks in-window line:\n%s", jl)
|
||||
}
|
||||
if strings.Contains(string(jl), "too-early") || strings.Contains(string(jl), "too-late") {
|
||||
t.Errorf("journal.log includes out-of-window content:\n%s", jl)
|
||||
}
|
||||
|
||||
tl, _ := os.ReadFile(filepath.Join(bundle, "test.log"))
|
||||
if !strings.Contains(string(tl), "--- FAIL: TestSingleNode/Phase5_LaunchInstance") {
|
||||
t.Errorf("test.log missing FAIL marker:\n%s", tl)
|
||||
}
|
||||
|
||||
out := Render(rep)
|
||||
wantLink := fmt.Sprintf("- Bundle: `%s/`", bp)
|
||||
if !strings.Contains(out, wantLink) {
|
||||
t.Errorf("render missing bundle link %q:\n%s", wantLink, out)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
)
|
||||
|
||||
func main() {
|
||||
var junitGlob, logDir, title string
|
||||
flag.StringVar(&junitGlob, "junit-glob", os.Getenv("ANALYZE_JUNIT_GLOB"), "glob matching junit-*.xml files")
|
||||
flag.StringVar(&logDir, "log-dir", os.Getenv("ANALYZE_LOG_DIR"), "artifact dir to write analysis.md into")
|
||||
flag.StringVar(&title, "title", os.Getenv("ANALYZE_TITLE"), "report heading")
|
||||
flag.Parse()
|
||||
|
||||
if title == "" {
|
||||
title = "E2E failure analysis"
|
||||
}
|
||||
if junitGlob == "" {
|
||||
fmt.Fprintln(os.Stderr, "::error::e2e-analyze: -junit-glob is required")
|
||||
os.Exit(0) // analyzer is advisory only; existing gate owns exit code
|
||||
}
|
||||
|
||||
paths, err := filepath.Glob(junitGlob)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: bad glob %q: %v\n", junitGlob, err)
|
||||
os.Exit(0)
|
||||
}
|
||||
sort.Strings(paths)
|
||||
|
||||
report := Report{Title: title}
|
||||
for _, p := range paths {
|
||||
data, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: read %s: %v\n", p, err)
|
||||
continue
|
||||
}
|
||||
sr, err := ParseFile(p, data)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: %v\n", err)
|
||||
continue
|
||||
}
|
||||
report.Suites = append(report.Suites, sr)
|
||||
}
|
||||
|
||||
// Stage 2: prefer authoritative per-suite start times written by the
|
||||
// workflow over junit-report's conversion-time `timestamp` attribute,
|
||||
// then materialise per-failure bundles so Render can link them.
|
||||
if logDir != "" {
|
||||
ApplySuiteStartFiles(&report, logDir)
|
||||
WriteBundles(&report, logDir)
|
||||
}
|
||||
|
||||
out := Render(report)
|
||||
|
||||
if logDir != "" {
|
||||
if err := os.MkdirAll(logDir, 0o755); err == nil {
|
||||
if err := os.WriteFile(filepath.Join(logDir, "analysis.md"), []byte(out), 0o644); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: write analysis.md: %v\n", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if summaryPath := os.Getenv("GITHUB_STEP_SUMMARY"); summaryPath != "" {
|
||||
f, err := os.OpenFile(summaryPath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
|
||||
if err == nil {
|
||||
defer f.Close()
|
||||
if _, werr := f.WriteString(out); werr != nil {
|
||||
fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: append summary: %v\n", werr)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// Local invocation (developer running the tool by hand). Mirror to stdout.
|
||||
fmt.Print(out)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,176 @@
|
||||
package main
|
||||
|
||||
// Stage 2 of docs/development/improvements/e2e-go-failure-analysis.md:
|
||||
// per-failure log slicers used by bundle.go to materialise time-window /
|
||||
// testcase-window log excerpts next to each named failure.
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"io"
|
||||
"os"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// shortIsoTSRe matches the timestamp prefix produced by
|
||||
// `journalctl --output=short-iso`. The offset format varies across
|
||||
// systemd versions: most ship "+10:00" (RFC3339), some ship "+1000"
|
||||
// (no colon). Match either — the parser below picks the right layout.
|
||||
var shortIsoTSRe = regexp.MustCompile(`^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:[+-]\d{2}:?\d{2}|Z))\s`)
|
||||
|
||||
// shortIsoLayouts are tried in order; the first parse that succeeds wins.
|
||||
var shortIsoLayouts = []string{
|
||||
time.RFC3339, // 2026-05-19T12:32:30+10:00
|
||||
"2006-01-02T15:04:05-0700", // 2026-05-19T12:32:30+1000
|
||||
}
|
||||
|
||||
func parseShortIso(s string) (time.Time, error) {
|
||||
var lastErr error
|
||||
for _, layout := range shortIsoLayouts {
|
||||
t, err := time.Parse(layout, s)
|
||||
if err == nil {
|
||||
return t, nil
|
||||
}
|
||||
lastErr = err
|
||||
}
|
||||
return time.Time{}, lastErr
|
||||
}
|
||||
|
||||
// SliceJournal copies lines from src whose `short-iso` timestamp falls in
|
||||
// [start, end] to dst. Lines without a parseable timestamp inherit the
|
||||
// previous line's classification — that keeps multi-line journal records
|
||||
// (stack traces, panics) intact rather than orphaning their continuation
|
||||
// lines. Lines that precede the first parseable timestamp are dropped.
|
||||
//
|
||||
// Returns the number of lines written; missing src is not an error (the
|
||||
// journal may be empty if collection failed).
|
||||
func SliceJournal(src string, start, end time.Time, dst string) (int, error) {
|
||||
f, err := os.Open(src)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return 0, nil
|
||||
}
|
||||
return 0, err
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
out, err := os.Create(dst)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer out.Close()
|
||||
|
||||
bw := bufio.NewWriter(out)
|
||||
defer bw.Flush()
|
||||
|
||||
sc := bufio.NewScanner(f)
|
||||
// Journal lines can be long (panic dumps); bump scanner buffer.
|
||||
sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
|
||||
|
||||
keep := false
|
||||
written := 0
|
||||
for sc.Scan() {
|
||||
line := sc.Text()
|
||||
if m := shortIsoTSRe.FindStringSubmatch(line); m != nil {
|
||||
ts, err := parseShortIso(m[1])
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
keep = !ts.Before(start) && !ts.After(end)
|
||||
}
|
||||
if keep {
|
||||
if _, err := bw.WriteString(line); err != nil {
|
||||
return written, err
|
||||
}
|
||||
if err := bw.WriteByte('\n'); err != nil {
|
||||
return written, err
|
||||
}
|
||||
written++
|
||||
}
|
||||
}
|
||||
return written, sc.Err()
|
||||
}
|
||||
|
||||
// runRePrefix and finishRePrefix bracket a single testcase's output in
|
||||
// `go test -v` stdout. Subtests have their own `=== RUN` / `--- FAIL`
|
||||
// markers, so slicing on the exact testcase name yields just that test's
|
||||
// lines (including any nested phase logs).
|
||||
var (
|
||||
runRe = regexp.MustCompile(`^=== RUN\s+(\S+)$`)
|
||||
finishRe = regexp.MustCompile(`^\s*---\s+(PASS|FAIL|SKIP):\s+(\S+)\s+\(`)
|
||||
)
|
||||
|
||||
// SliceTestLog copies lines belonging to the named testcase from src to
|
||||
// dst. The block starts at `=== RUN <name>` and ends at the matching
|
||||
// `--- PASS|FAIL|SKIP: <name> (…)` line (inclusive). If the testcase is
|
||||
// never seen the destination file is created empty so the bundle has a
|
||||
// stable shape.
|
||||
//
|
||||
// Returns the number of lines written; missing src is not an error.
|
||||
func SliceTestLog(src, testName, dst string) (int, error) {
|
||||
f, err := os.Open(src)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return 0, nil
|
||||
}
|
||||
return 0, err
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
out, err := os.Create(dst)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer out.Close()
|
||||
|
||||
bw := bufio.NewWriter(out)
|
||||
defer bw.Flush()
|
||||
|
||||
sc := bufio.NewScanner(f)
|
||||
sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
|
||||
|
||||
in := false
|
||||
written := 0
|
||||
for sc.Scan() {
|
||||
line := sc.Text()
|
||||
if !in {
|
||||
if m := runRe.FindStringSubmatch(strings.TrimRight(line, "\r")); m != nil && m[1] == testName {
|
||||
in = true
|
||||
} else {
|
||||
continue
|
||||
}
|
||||
}
|
||||
if _, err := bw.WriteString(line); err != nil {
|
||||
return written, err
|
||||
}
|
||||
if err := bw.WriteByte('\n'); err != nil {
|
||||
return written, err
|
||||
}
|
||||
written++
|
||||
if m := finishRe.FindStringSubmatch(line); m != nil && m[2] == testName {
|
||||
return written, nil
|
||||
}
|
||||
}
|
||||
return written, sc.Err()
|
||||
}
|
||||
|
||||
// copyFile is a small helper used when bundling pre-collected per-test
|
||||
// artifacts. Best-effort: missing src is silently skipped.
|
||||
func copyFile(src, dst string) error {
|
||||
in, err := os.Open(src)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
defer in.Close()
|
||||
out, err := os.Create(dst)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer out.Close()
|
||||
_, err = io.Copy(out, in)
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
## E2E failure analysis
|
||||
|
||||
### Suite `single`: 16 failed, 1 root cause likely
|
||||
|
||||
**Root cause (earliest non-cascade)**
|
||||
|
||||
- Test: `TestSingleNode/Phase5_LaunchInstance`
|
||||
- Start: 12:30:00 (duration 2.5s)
|
||||
- Error: Received unexpected error: InvalidParameterValue (HTTP 400)
|
||||
- File: `tests/e2e/harness/ec2helpers.go:164`
|
||||
|
||||
**Cascaded failures (15) — grouped by signature**
|
||||
|
||||
- 14× Phase 5 must populate fix.InstanceID
|
||||
- `TestSingleNode/Phase5a_pre_ClusterStats`
|
||||
- `TestSingleNode/Phase5a_Metadata`
|
||||
- `TestSingleNode/Phase5b_DescribeInstances`
|
||||
- `TestSingleNode/Phase5c_StopStart`
|
||||
- `TestSingleNode/Phase6_Volumes`
|
||||
- … and 9 more
|
||||
- 1× Phase 2 must populate fix.AZName
|
||||
- `TestSingleNode/Phase8_NatGW`
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<testsuites tests="20" failures="17" skipped="0">
|
||||
<testsuite name="" tests="20" failures="17" errors="0" id="0" hostname="banksia" skipped="0" time="600.000" timestamp="2026-04-12T12:30:00+10:00">
|
||||
<testcase name="TestSingleNode" classname="" time="600.000">
|
||||
<failure message="Failed"></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase1_Environment" classname="" time="0.300">
|
||||
<system-out><![CDATA[
|
||||
━━ Phase 1 ━━]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase2_Discovery" classname="" time="0.010"/>
|
||||
<testcase name="TestSingleNode/Phase3_KeyPairs" classname="" time="0.540"/>
|
||||
<testcase name="TestSingleNode/Phase4_Image" classname="" time="0.060"/>
|
||||
<testcase name="TestSingleNode/Phase5_LaunchInstance" classname="" time="2.500">
|
||||
<failure message="Failed"><![CDATA[
|
||||
ec2helpers.go:164: describe-vpcs: InvalidParameterValue: Filter 'is_default' is invalid (status 400)
|
||||
Error Trace: tests/e2e/harness/ec2helpers.go:164
|
||||
Error: Received unexpected error:
|
||||
InvalidParameterValue (HTTP 400)
|
||||
Test: TestSingleNode/Phase5_LaunchInstance]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5a_pre_ClusterStats" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
lifecycle_test.go:28:
|
||||
Error Trace: tests/e2e/single/lifecycle_test.go:28
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase5a_pre_ClusterStats]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5a_Metadata" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
lifecycle_test.go:57:
|
||||
Error Trace: tests/e2e/single/lifecycle_test.go:57
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase5a_Metadata]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5b_DescribeInstances" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
lifecycle_test.go:109:
|
||||
Error Trace: tests/e2e/single/lifecycle_test.go:109
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase5b_DescribeInstances]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5c_StopStart" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
lifecycle_test.go:222:
|
||||
Error Trace: tests/e2e/single/lifecycle_test.go:222
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase5c_StopStart]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase6_Volumes" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
volume_test.go:23:
|
||||
Error Trace: tests/e2e/single/volume_test.go:23
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase6_Volumes]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase6b_AttachVolume" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
volume_test.go:24:
|
||||
Error Trace: tests/e2e/single/volume_test.go:24
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase6b_AttachVolume]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase7_Tags" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
tags_test.go:23:
|
||||
Error Trace: tests/e2e/single/tags_test.go:23
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase7_Tags]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase7a_Snapshots" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
snapshot_test.go:23:
|
||||
Error Trace: tests/e2e/single/snapshot_test.go:23
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase7a_Snapshots]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase7b_DescribeInstanceAttribute" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
instance_test.go:81:
|
||||
Error Trace: tests/e2e/single/instance_test.go:81
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase7b_DescribeInstanceAttribute]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase7c_ModifyInstanceAttribute" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
instance_test.go:92:
|
||||
Error Trace: tests/e2e/single/instance_test.go:92
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase7c_ModifyInstanceAttribute]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase7d_Console" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
instance_test.go:121:
|
||||
Error Trace: tests/e2e/single/instance_test.go:121
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase7d_Console]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase7e_Negative" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
negative_test.go:43:
|
||||
Error Trace: tests/e2e/single/negative_test.go:43
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase7e_Negative]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NatGW" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
natgw_test.go:39:
|
||||
Error Trace: tests/e2e/single/natgw_test.go:39
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 2 must populate fix.AZName
|
||||
Test: TestSingleNode/Phase8_NatGW]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8a_DataPath" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
datapath_test.go:48:
|
||||
Error Trace: tests/e2e/single/datapath_test.go:48
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase8a_DataPath]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase9_SecurityGroups" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
securitygroup_test.go:35:
|
||||
Error Trace: tests/e2e/single/securitygroup_test.go:35
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase9_SecurityGroups]]></failure>
|
||||
</testcase>
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -0,0 +1,3 @@
|
||||
## E2E failure analysis
|
||||
|
||||
✅ No failures across any suite.
|
||||
@@ -0,0 +1,8 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<testsuites tests="3" failures="0" skipped="0">
|
||||
<testsuite name="" tests="3" failures="0" errors="0" id="0" hostname="banksia" skipped="0" time="1.500" timestamp="2026-04-12T12:30:00+10:00">
|
||||
<testcase name="TestSingleNode" classname="" time="1.500"/>
|
||||
<testcase name="TestSingleNode/Phase1_Environment" classname="" time="0.300"/>
|
||||
<testcase name="TestSingleNode/Phase2_Discovery" classname="" time="1.200"/>
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -0,0 +1,11 @@
|
||||
## E2E failure analysis
|
||||
|
||||
### Suite `single`: 1 failed, 1 root cause likely
|
||||
|
||||
**Root cause (earliest non-cascade)**
|
||||
|
||||
- Test: `TestSingleNode/Phase8b_VPCSubnetE2E`
|
||||
- Start: 00:26:18 (duration 200.5s)
|
||||
- Error: Eventually: condition not met within 3m0s: [SSH handshake 192.168.0.202:22 never completed]
|
||||
- File: `vpc_test.go:227`
|
||||
|
||||
@@ -0,0 +1,993 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<testsuites tests="79" failures="2" skipped="6">
|
||||
<testsuite name="" tests="79" failures="2" errors="0" id="0" hostname="banksia" skipped="6" time="1708.140" timestamp="2026-05-19T00:19:45+10:00">
|
||||
<testcase name="TestSingleNode" classname="" time="800.320">
|
||||
<failure message="Failed"></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase1_Environment" classname="" time="0.300">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 1 — Environment ━━�[0m
|
||||
�[2m· checking /dev/kvm�[0m
|
||||
discovery_test.go:29: �[2m· checking /dev/kvm�[0m
|
||||
�[2mkvm�[0m=writable
|
||||
discovery_test.go:38: �[2mkvm�[0m=writable
|
||||
�[2m· waiting for AWS gateway�[0m
|
||||
discovery_test.go:45: �[2m· waiting for AWS gateway�[0m
|
||||
�[2mgateway�[0m=https://192.168.0.10:9999/
|
||||
discovery_test.go:55: �[2mgateway�[0m=https://192.168.0.10:9999/
|
||||
�[2m· waiting for daemon NATS subscriptions�[0m
|
||||
discovery_test.go:60: �[2m· waiting for daemon NATS subscriptions�[0m
|
||||
�[2m· describe-regions / describe-availability-zones�[0m
|
||||
discovery_test.go:73: �[2m· describe-regions / describe-availability-zones�[0m
|
||||
�[2maz�[0m=ap-southeast-2a �[2mregion�[0m=ap-southeast-2
|
||||
discovery_test.go:84: �[2maz�[0m=ap-southeast-2a �[2mregion�[0m=ap-southeast-2]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase1b_ClusterStats" classname="" time="9.200">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 1b — Cluster Stats CLI ━━�[0m
|
||||
�[2m· spx get nodes�[0m
|
||||
discovery_test.go:101: �[2m· spx get nodes�[0m
|
||||
�[2m· spx top nodes�[0m
|
||||
discovery_test.go:105: �[2m· spx top nodes�[0m
|
||||
�[2m· spx get vms (empty before any launches)�[0m
|
||||
discovery_test.go:112: �[2m· spx get vms (empty before any launches)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase2_Discovery" classname="" time="0.010">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 2 — Discovery & Metadata ━━�[0m
|
||||
�[2m· discovering nano instance type�[0m
|
||||
discovery_test.go:133: �[2m· discovering nano instance type�[0m
|
||||
�[2minstance_type�[0m=t3.nano �[2march�[0m=x86_64 �[2maz�[0m=ap-southeast-2a
|
||||
discovery_test.go:159: �[2minstance_type�[0m=t3.nano �[2march�[0m=x86_64 �[2maz�[0m=ap-southeast-2a]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase2b_SerialConsole" classname="" time="0.020">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 2b — Serial Console Access ━━�[0m
|
||||
�[2m· default state should be disabled�[0m
|
||||
discovery_test.go:168: �[2m· default state should be disabled�[0m
|
||||
�[2m· enable�[0m
|
||||
discovery_test.go:173: �[2m· enable�[0m
|
||||
�[2mstate�[0m=enabled
|
||||
discovery_test.go:180: �[2mstate�[0m=enabled
|
||||
�[2m· disable�[0m
|
||||
discovery_test.go:182: �[2m· disable�[0m
|
||||
�[2mstate�[0m=disabled
|
||||
discovery_test.go:189: �[2mstate�[0m=disabled]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase3_KeyPairs" classname="" time="0.540">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 3 — SSH Key Management ━━�[0m
|
||||
�[2m· create-key-pair "test-key-1"�[0m
|
||||
keypair_test.go:37: �[2m· create-key-pair "test-key-1"�[0m
|
||||
�[2mkey�[0m=test-key-1 �[2mpem�[0m=/tmp/TestSingleNode1289015487/001/test-key-1.pem
|
||||
keypair_test.go:51: �[2mkey�[0m=test-key-1 �[2mpem�[0m=/tmp/TestSingleNode1289015487/001/test-key-1.pem
|
||||
�[2m· import-key-pair "test-key-2" (local-generated RSA)�[0m
|
||||
keypair_test.go:58: �[2m· import-key-pair "test-key-2" (local-generated RSA)�[0m
|
||||
�[2mimported�[0m=test-key-2
|
||||
keypair_test.go:65: �[2mimported�[0m=test-key-2
|
||||
�[2m· describe-key-pairs (both present)�[0m
|
||||
keypair_test.go:67: �[2m· describe-key-pairs (both present)�[0m
|
||||
�[2m· delete "test-key-2"�[0m
|
||||
keypair_test.go:72: �[2m· delete "test-key-2"�[0m
|
||||
�[2m· describe-key-pairs (only test-key-1 remains)�[0m
|
||||
keypair_test.go:76: �[2m· describe-key-pairs (only test-key-1 remains)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase4_Image" classname="" time="0.060">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 4 — Image Management ━━�[0m
|
||||
�[2mimage�[0m=ubuntu-24.04-x86_64 �[2march�[0m=x86_64
|
||||
image_test.go:33: �[2mimage�[0m=ubuntu-24.04-x86_64 �[2march�[0m=x86_64
|
||||
�[2m· spx admin images import --name ubuntu-24.04-x86_64�[0m
|
||||
image_test.go:35: �[2m· spx admin images import --name ubuntu-24.04-x86_64�[0m
|
||||
�[2m· spx import did not yield AMI ID; falling back to describe-images�[0m
|
||||
image_test.go:48: �[2m· spx import did not yield AMI ID; falling back to describe-images�[0m
|
||||
�[2m· describe-images by AMI ID (verify exactly one match)�[0m
|
||||
image_test.go:78: �[2m· describe-images by AMI ID (verify exactly one match)�[0m
|
||||
�[2mami�[0m=ami-6f60f2b3bf04b5453 �[2mname�[0m=ami-ubuntu-24.04-x86_64
|
||||
image_test.go:86: �[2mami�[0m=ami-6f60f2b3bf04b5453 �[2mname�[0m=ami-ubuntu-24.04-x86_64]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5_LaunchInstance" classname="" time="10.830">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 5 — Instance Lifecycle ━━�[0m
|
||||
�[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msg�[0m=sg-764a7fa2a39577204 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63
|
||||
instance_test.go:31: �[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msg�[0m=sg-764a7fa2a39577204 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63
|
||||
�[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.nano key=test-key-1�[0m
|
||||
instance_test.go:35: �[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.nano key=test-key-1�[0m
|
||||
�[2minstance�[0m=i-0c615e5dd74050f53
|
||||
instance_test.go:48: �[2minstance�[0m=i-0c615e5dd74050f53
|
||||
instance_test.go:50: instance i-0c615e5dd74050f53 reached state running
|
||||
�[2mroot_volume�[0m=vol-2230138124fb51ee2 �[2mroot_device�[0m=
|
||||
instance_test.go:70: �[2mroot_volume�[0m=vol-2230138124fb51ee2 �[2mroot_device�[0m=]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5a_pre_ClusterStats" classname="" time="3.040">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 5a-pre — Cluster Stats CLI (with running VM) ━━�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5a_Metadata" classname="" time="0.510">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 5a — Instance Metadata Validation ━━�[0m
|
||||
�[2mtype�[0m=t3.nano �[2mkey�[0m=test-key-1 �[2mimage�[0m=ami-6f60f2b3bf04b5453 �[2mbdms�[0m=1
|
||||
instance_test.go:108: �[2mtype�[0m=t3.nano �[2mkey�[0m=test-key-1 �[2mimage�[0m=ami-6f60f2b3bf04b5453 �[2mbdms�[0m=1]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5a_ii_SSH" classname="" time="49.450">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 5a-ii — SSH Connectivity & Volume Verification ━━�[0m
|
||||
�[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
|
||||
instance_test.go:127: �[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
|
||||
�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
|
||||
instance_test.go:130: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
|
||||
�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
|
||||
lifecycle_test.go:384: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
|
||||
�[2m· ssh id�[0m
|
||||
instance_test.go:136: �[2m· ssh id�[0m
|
||||
�[2m· lsblk root-volume cross-check vs API�[0m
|
||||
instance_test.go:140: �[2m· lsblk root-volume cross-check vs API�[0m
|
||||
�[2mguest_gib�[0m=4 �[2mapi_gib�[0m=4
|
||||
instance_test.go:159: �[2mguest_gib�[0m=4 �[2mapi_gib�[0m=4
|
||||
�[2m· ssh hostname�[0m
|
||||
instance_test.go:161: �[2m· ssh hostname�[0m
|
||||
�[2mhostname�[0m=spinifex-vm-0c615e5d �[2mmatches_short_id�[0m=0c615e5d
|
||||
instance_test.go:172: �[2mhostname�[0m=spinifex-vm-0c615e5d �[2mmatches_short_id�[0m=0c615e5d]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5a_iii_Console" classname="" time="0.010">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 5a-iii — Console Output ━━�[0m
|
||||
�[2minstance�[0m=i-0c615e5dd74050f53 �[2mhas_output�[0m=true
|
||||
instance_test.go:191: �[2minstance�[0m=i-0c615e5dd74050f53 �[2mhas_output�[0m=true]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5b_Volume" classname="" time="4.540">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 5b — Volume Lifecycle (Attach/Detach) ━━�[0m
|
||||
�[2m· create-volume size=10 az=ap-southeast-2a�[0m
|
||||
volume_test.go:26: �[2m· create-volume size=10 az=ap-southeast-2a�[0m
|
||||
�[2mvolume�[0m=vol-a885f79173e0a8030
|
||||
volume_test.go:34: �[2mvolume�[0m=vol-a885f79173e0a8030
|
||||
volume_test.go:39: volume vol-a885f79173e0a8030 reached state available
|
||||
�[2m· modify-volume size=20�[0m
|
||||
volume_test.go:42: �[2m· modify-volume size=20�[0m
|
||||
�[2mresized_gib�[0m=20
|
||||
volume_test.go:66: �[2mresized_gib�[0m=20
|
||||
�[2m· attach-volume vol-a885f79173e0a8030 -> i-0c615e5dd74050f53 as /dev/sdf�[0m
|
||||
volume_test.go:68: �[2m· attach-volume vol-a885f79173e0a8030 -> i-0c615e5dd74050f53 as /dev/sdf�[0m
|
||||
volume_test.go:76: volume vol-a885f79173e0a8030 reached state in-use
|
||||
�[2m· detach-volume vol-a885f79173e0a8030�[0m
|
||||
volume_test.go:90: �[2m· detach-volume vol-a885f79173e0a8030�[0m
|
||||
volume_test.go:96: volume vol-a885f79173e0a8030 reached state available
|
||||
�[2m· delete-volume vol-a885f79173e0a8030�[0m
|
||||
volume_test.go:98: �[2m· delete-volume vol-a885f79173e0a8030�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5b_ii_VolumeStatus" classname="" time="0.000">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 5b-ii — DescribeVolumeStatus ━━�[0m
|
||||
�[2mvolume�[0m=vol-2230138124fb51ee2 �[2mstatus�[0m=ok
|
||||
volume_test.go:158: �[2mvolume�[0m=vol-2230138124fb51ee2 �[2mstatus�[0m=ok]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5c_Snapshot" classname="" time="0.190">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 5c — Snapshot Lifecycle ━━�[0m
|
||||
�[2m· create-snapshot volume=vol-2230138124fb51ee2�[0m
|
||||
snapshot_test.go:34: �[2m· create-snapshot volume=vol-2230138124fb51ee2�[0m
|
||||
�[2msnapshot�[0m=snap-1188f92980e6470aa �[2msize�[0m=4 �[2mstate�[0m=completed �[2mprogress�[0m=100%
|
||||
snapshot_test.go:50: �[2msnapshot�[0m=snap-1188f92980e6470aa �[2msize�[0m=4 �[2mstate�[0m=completed �[2mprogress�[0m=100%
|
||||
snapshot_test.go:59: snapshot snap-1188f92980e6470aa reached state completed
|
||||
�[2m· describe-snapshots snap-1188f92980e6470aa�[0m
|
||||
snapshot_test.go:61: �[2m· describe-snapshots snap-1188f92980e6470aa�[0m
|
||||
�[2m· copy-snapshot src=snap-1188f92980e6470aa region=ap-southeast-2�[0m
|
||||
snapshot_test.go:76: �[2m· copy-snapshot src=snap-1188f92980e6470aa region=ap-southeast-2�[0m
|
||||
�[2mcopy_snapshot�[0m=snap-cde6b5ddf467e2c0f
|
||||
snapshot_test.go:86: �[2mcopy_snapshot�[0m=snap-cde6b5ddf467e2c0f
|
||||
snapshot_test.go:88: snapshot snap-cde6b5ddf467e2c0f reached state completed
|
||||
�[2m· describe-snapshots snap-1188f92980e6470aa,snap-cde6b5ddf467e2c0f (both visible)�[0m
|
||||
snapshot_test.go:90: �[2m· describe-snapshots snap-1188f92980e6470aa,snap-cde6b5ddf467e2c0f (both visible)�[0m
|
||||
�[2m· delete-snapshot snap-1188f92980e6470aa (original)�[0m
|
||||
snapshot_test.go:107: �[2m· delete-snapshot snap-1188f92980e6470aa (original)�[0m
|
||||
�[2m· verify copy snap-cde6b5ddf467e2c0f still completed�[0m
|
||||
snapshot_test.go:135: �[2m· verify copy snap-cde6b5ddf467e2c0f still completed�[0m
|
||||
�[2m· delete-snapshot snap-cde6b5ddf467e2c0f (copy)�[0m
|
||||
snapshot_test.go:144: �[2m· delete-snapshot snap-cde6b5ddf467e2c0f (copy)�[0m
|
||||
�[2mdeleted_original�[0m=snap-1188f92980e6470aa
|
||||
snapshot_test.go:151: �[2mdeleted_original�[0m=snap-1188f92980e6470aa]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5d_SnapshotBackedLaunch" classname="" time="0.010">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 5d — Snapshot-Backed Instance Launch ━━�[0m
|
||||
�[2mami�[0m=ami-6f60f2b3bf04b5453 �[2msnapshot�[0m=snap-ami-6f60f2b3bf04b5453
|
||||
snapshot_test.go:182: �[2mami�[0m=ami-6f60f2b3bf04b5453 �[2msnapshot�[0m=snap-ami-6f60f2b3bf04b5453]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5e_CreateImage" classname="" time="0.660">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 5e — CreateImage Lifecycle ━━�[0m
|
||||
�[2m· create-image instance=i-0c615e5dd74050f53 name=e2e-custom-ami (no-reboot)�[0m
|
||||
createimage_test.go:26: �[2m· create-image instance=i-0c615e5dd74050f53 name=e2e-custom-ami (no-reboot)�[0m
|
||||
�[2mcustom_ami�[0m=ami-7dcf263a412dd14c6
|
||||
createimage_test.go:36: �[2mcustom_ami�[0m=ami-7dcf263a412dd14c6
|
||||
createimage_test.go:38: image ami-7dcf263a412dd14c6 reached state available
|
||||
�[2m· describe-images ami-7dcf263a412dd14c6�[0m
|
||||
createimage_test.go:40: �[2m· describe-images ami-7dcf263a412dd14c6�[0m
|
||||
�[2mcustom_ami_snapshot�[0m=snap-dde89b9ea2ef2e05a
|
||||
createimage_test.go:67: �[2mcustom_ami_snapshot�[0m=snap-dde89b9ea2ef2e05a]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5f_SecurityGroupEgress" classname="" time="4.880">
|
||||
<skipped message="Skipped"><![CDATA[
|
||||
�[1m�[36m━━ Phase 5f — Security Group Enforcement (egress ACL) ━━�[0m
|
||||
�[2m· discover default gateway inside VM�[0m
|
||||
securitygroup_test.go:53: �[2m· discover default gateway inside VM�[0m
|
||||
�[2mprobe_gateway�[0m=172.31.0.1
|
||||
securitygroup_test.go:66: �[2mprobe_gateway�[0m=172.31.0.1
|
||||
�[2m· 5f-1 baseline egress (allow-all)�[0m
|
||||
securitygroup_test.go:77: �[2m· 5f-1 baseline egress (allow-all)�[0m
|
||||
securitygroup_test.go:80: Phase 5f: baseline ICMP did not succeed; env may block ICMP regardless of SG
|
||||
Output:
|
||||
PING 172.31.0.1 (172.31.0.1) 56(84) bytes of data.
|
||||
|
||||
--- 172.31.0.1 ping statistics ---
|
||||
3 packets transmitted, 0 received, 100% packet loss, time 2044ms
|
||||
]]></skipped>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase6_TagManagement" classname="" time="0.380">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 6 — Tag Management ━━�[0m
|
||||
�[2m· 6a create-tags instance=i-0c615e5dd74050f53 (Name, Environment, DeleteMe)�[0m
|
||||
tags_test.go:41: �[2m· 6a create-tags instance=i-0c615e5dd74050f53 (Name, Environment, DeleteMe)�[0m
|
||||
�[2m· 6b describe-tags resource-id=i-0c615e5dd74050f53 want=3�[0m
|
||||
tags_test.go:53: �[2m· 6b describe-tags resource-id=i-0c615e5dd74050f53 want=3�[0m
|
||||
�[2m· 6c create-tags volume=vol-2230138124fb51ee2 (Name, Environment)�[0m
|
||||
tags_test.go:65: �[2m· 6c create-tags volume=vol-2230138124fb51ee2 (Name, Environment)�[0m
|
||||
�[2m· 6d describe-tags key=Environment want=2�[0m
|
||||
tags_test.go:76: �[2m· 6d describe-tags key=Environment want=2�[0m
|
||||
�[2m· 6e describe-tags resource-type=instance want=3�[0m
|
||||
tags_test.go:85: �[2m· 6e describe-tags resource-type=instance want=3�[0m
|
||||
�[2m· 6f overwrite instance Name -> e2e-test-updated�[0m
|
||||
tags_test.go:95: �[2m· 6f overwrite instance Name -> e2e-test-updated�[0m
|
||||
�[2m· 6g delete-tags Key=DeleteMe (no value)�[0m
|
||||
tags_test.go:114: �[2m· 6g delete-tags Key=DeleteMe (no value)�[0m
|
||||
�[2m· 6h delete-tags Key=Environment,Value=production (no-op)�[0m
|
||||
tags_test.go:130: �[2m· 6h delete-tags Key=Environment,Value=production (no-op)�[0m
|
||||
�[2m· 6i delete-tags Key=Environment,Value=testing (instance only)�[0m
|
||||
tags_test.go:148: �[2m· 6i delete-tags Key=Environment,Value=testing (instance only)�[0m
|
||||
�[2m· 6j final describe-tags resource-id=i-0c615e5dd74050f53 want=1�[0m
|
||||
tags_test.go:171: �[2m· 6j final describe-tags resource-id=i-0c615e5dd74050f53 want=1�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase7_StopStart" classname="" time="8.090">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 7 — Instance State Transitions (Stop) ━━�[0m
|
||||
�[2m· stop-instances i-0c615e5dd74050f53�[0m
|
||||
lifecycle_test.go:30: �[2m· stop-instances i-0c615e5dd74050f53�[0m
|
||||
lifecycle_test.go:36: instance i-0c615e5dd74050f53 reached state stopped
|
||||
�[2m· reboot-instances on stopped instance (should fail)�[0m
|
||||
lifecycle_test.go:40: �[2m· reboot-instances on stopped instance (should fail)�[0m
|
||||
�[2mstopped_reject�[0m=IncorrectInstanceState
|
||||
lifecycle_test.go:47: �[2mstopped_reject�[0m=IncorrectInstanceState]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase7a_AttachToStoppedError" classname="" time="0.300">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 7a — Attach Volume to Stopped Instance (Error Path) ━━�[0m
|
||||
�[2m· create-volume size=10 az=ap-southeast-2a�[0m
|
||||
lifecycle_test.go:60: �[2m· create-volume size=10 az=ap-southeast-2a�[0m
|
||||
�[2mtest_volume�[0m=vol-8d3e1335938a378cd
|
||||
lifecycle_test.go:68: �[2mtest_volume�[0m=vol-8d3e1335938a378cd
|
||||
lifecycle_test.go:82: volume vol-8d3e1335938a378cd reached state available
|
||||
�[2m· attach-volume vol-8d3e1335938a378cd -> i-0c615e5dd74050f53 (should fail)�[0m
|
||||
lifecycle_test.go:84: �[2m· attach-volume vol-8d3e1335938a378cd -> i-0c615e5dd74050f53 (should fail)�[0m
|
||||
�[2m· delete-volume vol-8d3e1335938a378cd�[0m
|
||||
lifecycle_test.go:94: �[2m· delete-volume vol-8d3e1335938a378cd�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase7b_ModifyInstanceAttribute" classname="" time="55.620">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 7b — ModifyInstanceAttribute ━━�[0m
|
||||
�[2mfrom_type�[0m=t3.nano �[2mto_type�[0m=t3.small
|
||||
lifecycle_test.go:120: �[2mfrom_type�[0m=t3.nano �[2mto_type�[0m=t3.small
|
||||
�[2mexpected_vcpus�[0m=2 �[2mexpected_mem_mib�[0m=2048
|
||||
lifecycle_test.go:142: �[2mexpected_vcpus�[0m=2 �[2mexpected_mem_mib�[0m=2048
|
||||
�[2m· modify-instance-attribute i-0c615e5dd74050f53 type=t3.small�[0m
|
||||
lifecycle_test.go:144: �[2m· modify-instance-attribute i-0c615e5dd74050f53 type=t3.small�[0m
|
||||
�[2m· start-instances i-0c615e5dd74050f53�[0m
|
||||
lifecycle_test.go:165: �[2m· start-instances i-0c615e5dd74050f53�[0m
|
||||
lifecycle_test.go:171: instance i-0c615e5dd74050f53 reached state running
|
||||
�[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
|
||||
lifecycle_test.go:177: �[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
|
||||
�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
|
||||
lifecycle_test.go:179: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
|
||||
�[2m· ssh nproc�[0m
|
||||
lifecycle_test.go:182: �[2m· ssh nproc�[0m
|
||||
�[2mvm_vcpus�[0m=2
|
||||
lifecycle_test.go:188: �[2mvm_vcpus�[0m=2
|
||||
�[2m· ssh MemTotal�[0m
|
||||
lifecycle_test.go:190: �[2m· ssh MemTotal�[0m
|
||||
�[2mvm_mem_mib�[0m=1967 �[2mthreshold_mib�[0m=1740
|
||||
lifecycle_test.go:200: �[2mvm_mem_mib�[0m=1967 �[2mthreshold_mib�[0m=1740
|
||||
�[2m· stop-instances i-0c615e5dd74050f53 (end of 7b)�[0m
|
||||
lifecycle_test.go:206: �[2m· stop-instances i-0c615e5dd74050f53 (end of 7b)�[0m
|
||||
lifecycle_test.go:211: instance i-0c615e5dd74050f53 reached state stopped]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase7c_pre_Reboot" classname="" time="83.480">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 7c-pre — Reboot Running Instance ━━�[0m
|
||||
�[2m· start-instances i-0c615e5dd74050f53�[0m
|
||||
lifecycle_test.go:226: �[2m· start-instances i-0c615e5dd74050f53�[0m
|
||||
lifecycle_test.go:231: instance i-0c615e5dd74050f53 reached state running
|
||||
�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
|
||||
lifecycle_test.go:236: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
|
||||
�[2mpre_reboot_private_ip�[0m=172.31.0.4
|
||||
lifecycle_test.go:246: �[2mpre_reboot_private_ip�[0m=172.31.0.4
|
||||
�[2m· reboot-instances i-0c615e5dd74050f53�[0m
|
||||
lifecycle_test.go:248: �[2m· reboot-instances i-0c615e5dd74050f53�[0m
|
||||
�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
|
||||
lifecycle_test.go:279: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
|
||||
�[2m· ssh uptime�[0m
|
||||
lifecycle_test.go:282: �[2m· ssh uptime�[0m
|
||||
�[2muptime_secs�[0m=25
|
||||
lifecycle_test.go:288: �[2muptime_secs�[0m=25]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase7c_RunInstancesMultiCount" classname="" time="47.560">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 7c — RunInstances with MinCount/MaxCount > 1 ━━�[0m
|
||||
�[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.small count=2�[0m
|
||||
lifecycle_test.go:309: �[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.small count=2�[0m
|
||||
�[2msibling_1�[0m=i-9bfadd8ebd378f846 �[2msibling_2�[0m=i-448a9e2bf45c58658
|
||||
lifecycle_test.go:327: �[2msibling_1�[0m=i-9bfadd8ebd378f846 �[2msibling_2�[0m=i-448a9e2bf45c58658
|
||||
lifecycle_test.go:342: instance i-9bfadd8ebd378f846 reached state running
|
||||
lifecycle_test.go:342: instance i-448a9e2bf45c58658 reached state running
|
||||
�[2m· terminate-instances i-9bfadd8ebd378f846 i-448a9e2bf45c58658�[0m
|
||||
lifecycle_test.go:345: �[2m· terminate-instances i-9bfadd8ebd378f846 i-448a9e2bf45c58658�[0m
|
||||
lifecycle_test.go:352: instance i-9bfadd8ebd378f846 reached state terminated
|
||||
lifecycle_test.go:352: instance i-448a9e2bf45c58658 reached state terminated]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths" classname="" time="0.000">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 8 — Negative / Error Path Tests ━━�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8a_InvalidAMIIDMalformed" classname="" time="0.010">
|
||||
<system-out><![CDATA[�[2m· run-instances image-id=notanami�[0m
|
||||
negative_test.go:55: �[2m· run-instances image-id=notanami�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8b_InvalidInstanceType" classname="" time="0.210">
|
||||
<system-out><![CDATA[�[2m· run-instances instance-type=x99.superlarge�[0m
|
||||
negative_test.go:70: �[2m· run-instances instance-type=x99.superlarge�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8c_VolumeInUse" classname="" time="0.010">
|
||||
<system-out><![CDATA[�[2m· attach-volume vol-2230138124fb51ee2 (root, in-use)�[0m
|
||||
negative_test.go:85: �[2m· attach-volume vol-2230138124fb51ee2 (root, in-use)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8d_DetachRootForbidden" classname="" time="0.010">
|
||||
<system-out><![CDATA[�[2m· detach-volume vol-2230138124fb51ee2 (root)�[0m
|
||||
negative_test.go:97: �[2m· detach-volume vol-2230138124fb51ee2 (root)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8e_DeleteSnapshotNotFound" classname="" time="0.030">
|
||||
<system-out><![CDATA[�[2m· delete-snapshot snap-nonexistent000000�[0m
|
||||
negative_test.go:108: �[2m· delete-snapshot snap-nonexistent000000�[0m
|
||||
�[2m· describe-volumes vol-0000000000000dead�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8f_InvalidActionRawHTTP" classname="" time="0.040">
|
||||
<system-out><![CDATA[�[2m· POST Action=DescribeFakeThings�[0m
|
||||
negative_test.go:122: �[2m· POST Action=DescribeFakeThings�[0m
|
||||
negative_test.go:125: �[2mstatus�[0m=400 �[2mcode�[0m=InvalidAction]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8g_InvalidAMIIDNotFound" classname="" time="0.240">
|
||||
<system-out><![CDATA[�[2m· reboot-instances i-nonexistent�[0m
|
||||
�[2m· run-instances image-id=ami-0000000000000dead�[0m
|
||||
negative_test.go:138: �[2m· run-instances image-id=ami-0000000000000dead�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8h_InvalidKeyPairNotFound" classname="" time="0.260">
|
||||
<system-out><![CDATA[�[2m· run-instances key-name=nonexistent-key-xyz�[0m
|
||||
negative_test.go:153: �[2m· run-instances key-name=nonexistent-key-xyz�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8i_DeleteVolumeNotFound" classname="" time="0.040">
|
||||
<system-out><![CDATA[�[2m· delete-volume vol-0000000000000dead�[0m
|
||||
negative_test.go:168: �[2m· delete-volume vol-0000000000000dead�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8j_CreateKeyPairDuplicate" classname="" time="0.040">
|
||||
<system-out><![CDATA[ negative_test.go:178: �[2m· create-key-pair test-key-1 (duplicate)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8k_ImportKeyPairDuplicate" classname="" time="0.040">
|
||||
<system-out><![CDATA[�[2m· import-key-pair test-key-1 (duplicate)�[0m
|
||||
negative_test.go:198: �[2m· import-key-pair test-key-1 (duplicate)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8l_InvalidKeyFormat" classname="" time="0.040">
|
||||
<system-out><![CDATA[ negative_test.go:210: �[2m· import-key-pair bad-format-key (bad material)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8m_DescribeVolumesNotFound" classname="" time="0.040">
|
||||
<system-out><![CDATA[ negative_test.go:230: �[2m· describe-volumes vol-0000000000000dead�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8n_DescribeImagesNotFound" classname="" time="0.040">
|
||||
<system-out><![CDATA[�[2m· describe-images ami-0000000000000dead�[0m
|
||||
negative_test.go:240: �[2m· describe-images ami-0000000000000dead�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8o_CreateImageDuplicateName" classname="" time="0.560">
|
||||
<system-out><![CDATA[�[2m· create-key-pair test-key-1 (duplicate)�[0m
|
||||
�[2m· create-image name=e2e-custom-ami (duplicate)�[0m
|
||||
�[2m· import-key-pair bad-format-key (bad material)�[0m
|
||||
negative_test.go:252: �[2m· create-image name=e2e-custom-ami (duplicate)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8p_DeleteKeyPairIdempotent" classname="" time="0.020">
|
||||
<system-out><![CDATA[�[2m· delete-key-pair nonexistent-key-99999 (idempotent)�[0m
|
||||
negative_test.go:276: �[2m· delete-key-pair nonexistent-key-99999 (idempotent)�[0m
|
||||
�[2mstatus�[0m=400 �[2mcode�[0m=InvalidAction]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8q_ModifyAttributeOnRunning" classname="" time="0.010">
|
||||
<system-out><![CDATA[�[2m· modify-instance-attribute i-0c615e5dd74050f53 (running)�[0m
|
||||
negative_test.go:289: �[2m· modify-instance-attribute i-0c615e5dd74050f53 (running)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8r_RebootInstanceNotFound" classname="" time="0.040">
|
||||
<system-out><![CDATA[ negative_test.go:302: �[2m· reboot-instances i-nonexistent�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM1_UserCRUD" classname="" time="0.070">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ IAM Phase 1 — User CRUD ━━�[0m
|
||||
�[2m· list-users (root auth sanity)�[0m
|
||||
iam_test.go:109: �[2m· list-users (root auth sanity)�[0m
|
||||
�[2m· create-user "alice"�[0m
|
||||
iam_test.go:114: �[2m· create-user "alice"�[0m
|
||||
�[2muser�[0m=alice �[2marn�[0m=arn:aws:iam::000000000001:user/alice
|
||||
iam_test.go:120: �[2muser�[0m=alice �[2marn�[0m=arn:aws:iam::000000000001:user/alice
|
||||
�[2m· create-user "bob" path="/engineering/"�[0m
|
||||
iam_test.go:122: �[2m· create-user "bob" path="/engineering/"�[0m
|
||||
�[2m· create-user "alice" again (expect EntityAlreadyExists)�[0m
|
||||
iam_test.go:131: �[2m· create-user "alice" again (expect EntityAlreadyExists)�[0m
|
||||
�[2m· get-user "alice"�[0m
|
||||
iam_test.go:137: �[2m· get-user "alice"�[0m
|
||||
�[2m· get-user nonexistent (expect NoSuchEntity)�[0m
|
||||
iam_test.go:142: �[2m· get-user nonexistent (expect NoSuchEntity)�[0m
|
||||
�[2m· list-users (>=3 root+alice+bob)�[0m
|
||||
iam_test.go:148: �[2m· list-users (>=3 root+alice+bob)�[0m
|
||||
�[2m· list-users --path-prefix "/engineering/" (expect 1)�[0m
|
||||
iam_test.go:154: �[2m· list-users --path-prefix "/engineering/" (expect 1)�[0m
|
||||
iam_test.go:193: �[2m· create-user "iam-phase1-ephemeral" (delete idempotency probe)�[0m
|
||||
�[2m· delete-user "iam-phase1-ephemeral"�[0m
|
||||
iam_test.go:197: �[2m· delete-user "iam-phase1-ephemeral"�[0m
|
||||
�[2m· delete-user "iam-phase1-ephemeral" again (expect NoSuchEntity)�[0m
|
||||
iam_test.go:201: �[2m· delete-user "iam-phase1-ephemeral" again (expect NoSuchEntity)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM1_UserCRUD/UpdateUser" classname="" time="0.000">
|
||||
<skipped message="Skipped"><![CDATA[ iam_test.go:165: daemon UpdateUser handler not implemented — InvalidAction
|
||||
�[2m· create-user "iam-phase1-ephemeral" (delete idempotency probe)�[0m]]></skipped>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM2_AccessKeyLifecycle" classname="" time="0.040">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ IAM Phase 2 — Access Key Lifecycle ━━�[0m
|
||||
�[2m· create-access-key user=alice (key 1)�[0m
|
||||
iam_test.go:218: �[2m· create-access-key user=alice (key 1)�[0m
|
||||
�[2mkey1�[0m=AKIA4984AE57CF8D721A879A
|
||||
iam_test.go:227: �[2mkey1�[0m=AKIA4984AE57CF8D721A879A
|
||||
�[2m· create-access-key user=alice (key 2)�[0m
|
||||
iam_test.go:229: �[2m· create-access-key user=alice (key 2)�[0m
|
||||
�[2mkey2�[0m=AKIAA1CD38D4F0E573216E11
|
||||
iam_test.go:236: �[2mkey2�[0m=AKIAA1CD38D4F0E573216E11
|
||||
�[2m· create-access-key user=alice key 3 (expect LimitExceeded)�[0m
|
||||
iam_test.go:238: �[2m· create-access-key user=alice key 3 (expect LimitExceeded)�[0m
|
||||
�[2m· create-access-key user=ghost (expect NoSuchEntity)�[0m
|
||||
iam_test.go:246: �[2m· create-access-key user=ghost (expect NoSuchEntity)�[0m
|
||||
�[2m· list-access-keys user=alice (expect 2)�[0m
|
||||
iam_test.go:254: �[2m· list-access-keys user=alice (expect 2)�[0m
|
||||
�[2m· list-access-keys user=bob (expect 0)�[0m
|
||||
iam_test.go:261: �[2m· list-access-keys user=bob (expect 0)�[0m
|
||||
�[2m· update-access-key AKIA4984AE57CF8D721A879A -> Inactive�[0m
|
||||
iam_test.go:268: �[2m· update-access-key AKIA4984AE57CF8D721A879A -> Inactive�[0m
|
||||
�[2m· update-access-key AKIA4984AE57CF8D721A879A -> Active�[0m
|
||||
iam_test.go:279: �[2m· update-access-key AKIA4984AE57CF8D721A879A -> Active�[0m
|
||||
�[2m· delete-access-key AKIAA1CD38D4F0E573216E11 (key 2)�[0m
|
||||
iam_test.go:287: �[2m· delete-access-key AKIAA1CD38D4F0E573216E11 (key 2)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM3_UserAuthentication" classname="" time="0.000">
|
||||
<skipped message="Skipped"><![CDATA[
|
||||
�[1m�[36m━━ IAM Phase 3 — User Authentication ━━�[0m
|
||||
iam_test.go:311: daemon IAM signature/principal lookup not implemented — mulga-siv-100]]></skipped>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM4_PolicyCRUD" classname="" time="0.040">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ IAM Phase 4 — Policy CRUD ━━�[0m
|
||||
�[2m· create-policy EC2ReadOnly�[0m
|
||||
iam_test.go:384: �[2m· create-policy EC2ReadOnly�[0m
|
||||
�[2mpolicy�[0m=EC2ReadOnly �[2marn�[0m=arn:aws:iam::000000000001:policy/EC2ReadOnly �[2maccount�[0m=000000000001
|
||||
iam_test.go:393: �[2mpolicy�[0m=EC2ReadOnly �[2marn�[0m=arn:aws:iam::000000000001:policy/EC2ReadOnly �[2maccount�[0m=000000000001
|
||||
iam_test.go:440: �[2m· create-policy duplicate (expect EntityAlreadyExists)�[0m
|
||||
�[2m· create-policy malformed (expect MalformedPolicyDocument)�[0m
|
||||
iam_test.go:449: �[2m· create-policy malformed (expect MalformedPolicyDocument)�[0m
|
||||
�[2m· get-policy EC2ReadOnly�[0m
|
||||
iam_test.go:458: �[2m· get-policy EC2ReadOnly�[0m
|
||||
�[2m· get-policy nonexistent (expect NoSuchEntity)�[0m
|
||||
iam_test.go:463: �[2m· get-policy nonexistent (expect NoSuchEntity)�[0m
|
||||
�[2m· get-policy-version EC2ReadOnly v1�[0m
|
||||
iam_test.go:470: �[2m· get-policy-version EC2ReadOnly v1�[0m
|
||||
iam_test.go:519: �[2m· list-policies (>=5)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel" classname="" time="0.000"></testcase>
|
||||
<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/FullAdmin" classname="" time="0.010">
|
||||
<system-out><![CDATA[�[2m· create-policy FullAdmin path=/admin/�[0m
|
||||
iam_test.go:402: �[2m· create-policy FullAdmin path=/admin/�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/DenyTerminate" classname="" time="0.010">
|
||||
<system-out><![CDATA[�[2m· create-policy DenyTerminate (mixed Allow+Deny)�[0m
|
||||
iam_test.go:413: �[2m· create-policy DenyTerminate (mixed Allow+Deny)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/IAMReadOnly" classname="" time="0.020">
|
||||
<system-out><![CDATA[�[2m· create-policy IAMReadOnly�[0m
|
||||
iam_test.go:422: �[2m· create-policy IAMReadOnly�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/EC2DescribeAll" classname="" time="0.020">
|
||||
<system-out><![CDATA[�[2m· create-policy EC2DescribeAll (wildcard)�[0m
|
||||
iam_test.go:431: �[2m· create-policy EC2DescribeAll (wildcard)�[0m
|
||||
�[2m· create-policy duplicate (expect EntityAlreadyExists)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM4_PolicyCRUD/PolicyVersionLifecycle" classname="" time="0.000">
|
||||
<skipped message="Skipped"><![CDATA[ iam_test.go:484: daemon CreatePolicyVersion handler not implemented — InvalidAction
|
||||
�[2m· list-policies (>=5)�[0m]]></skipped>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM5_PolicyAttachmentEnforcement" classname="" time="0.000">
|
||||
<skipped message="Skipped"><![CDATA[
|
||||
�[1m�[36m━━ IAM Phase 5 — Policy Attachment & Enforcement ━━�[0m
|
||||
iam_test.go:535: daemon IAM scoped-credential enforcement not implemented — mulga-siv-100]]></skipped>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM6_PolicyLifecycle" classname="" time="0.000">
|
||||
<skipped message="Skipped"><![CDATA[
|
||||
�[1m�[36m━━ IAM Phase 6 — Policy Lifecycle (Detach & Delete) ━━�[0m
|
||||
iam_test.go:718: daemon DetachUserPolicy not implemented — mulga-siv-100]]></skipped>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM7_Cleanup" classname="" time="0.050">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ IAM Phase 7 — Cleanup ━━�[0m
|
||||
�[2m· cleanup user alice�[0m
|
||||
iam_test.go:770: �[2m· cleanup user alice�[0m
|
||||
�[2m· cleanup user bob�[0m
|
||||
iam_test.go:770: �[2m· cleanup user bob�[0m
|
||||
�[2m· cleanup user charlie�[0m
|
||||
iam_test.go:770: �[2m· cleanup user charlie�[0m
|
||||
�[2mremaining_users�[0m=1 �[2mnames�[0m=[admin]
|
||||
iam_test.go:779: �[2mremaining_users�[0m=1 �[2mnames�[0m=[admin]
|
||||
�[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2ReadOnly�[0m
|
||||
iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2ReadOnly�[0m
|
||||
�[2m· cleanup policy arn:aws:iam::000000000001:policy/admin/FullAdmin�[0m
|
||||
iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/admin/FullAdmin�[0m
|
||||
�[2m· cleanup policy arn:aws:iam::000000000001:policy/IAMReadOnly�[0m
|
||||
iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/IAMReadOnly�[0m
|
||||
�[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2DescribeAll�[0m
|
||||
iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2DescribeAll�[0m
|
||||
�[2m· cleanup policy arn:aws:iam::000000000001:policy/DenyTerminate�[0m
|
||||
iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/DenyTerminate�[0m
|
||||
�[2mremaining_policies�[0m=1 �[2mnames�[0m=[AdministratorAccess]
|
||||
iam_test.go:812: �[2mremaining_policies�[0m=1 �[2mnames�[0m=[AdministratorAccess]]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping" classname="" time="114.890">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 8 (acct) — EC2 Account Scoping ━━�[0m
|
||||
account_test.go:75: �[2m· Phase 8 acct cleanup (deferred)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step1_AccountSetup" classname="" time="1.220">
|
||||
<system-out><![CDATA[�[2m· create account 'Team Alpha'�[0m
|
||||
account_test.go:206: �[2m· create account 'Team Alpha'�[0m
|
||||
�[2malpha_account�[0m=000000000002 �[2malpha_key�[0m=AKIA374CBBB2A15D3E2EE822
|
||||
account_test.go:211: �[2malpha_account�[0m=000000000002 �[2malpha_key�[0m=AKIA374CBBB2A15D3E2EE822
|
||||
�[2m· create account 'Team Beta'�[0m
|
||||
account_test.go:213: �[2m· create account 'Team Beta'�[0m
|
||||
�[2mbeta_account�[0m=000000000003 �[2mbeta_key�[0m=AKIA0AA0B2845E3BD5A26D01
|
||||
account_test.go:218: �[2mbeta_account�[0m=000000000003 �[2mbeta_key�[0m=AKIA0AA0B2845E3BD5A26D01]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step2_InstanceScoping" classname="" time="52.380">
|
||||
<system-out><![CDATA[�[2m· create per-account key pairs�[0m
|
||||
account_test.go:242: �[2m· create per-account key pairs�[0m
|
||||
�[2m· alpha run-instances�[0m
|
||||
account_test.go:250: �[2m· alpha run-instances�[0m
|
||||
�[2malpha_instance�[0m=i-a3072c2387630b8ee
|
||||
account_test.go:262: �[2malpha_instance�[0m=i-a3072c2387630b8ee
|
||||
�[2m· beta run-instances�[0m
|
||||
account_test.go:264: �[2m· beta run-instances�[0m
|
||||
�[2mbeta_instance�[0m=i-ad32cbb0afaee5eee
|
||||
account_test.go:276: �[2mbeta_instance�[0m=i-ad32cbb0afaee5eee
|
||||
account_test.go:278: instance i-a3072c2387630b8ee reached state running
|
||||
account_test.go:279: instance i-ad32cbb0afaee5eee reached state running
|
||||
�[2m· alpha sees only own instances�[0m
|
||||
account_test.go:281: �[2m· alpha sees only own instances�[0m
|
||||
�[2m· beta sees only own instances�[0m
|
||||
account_test.go:285: �[2m· beta sees only own instances�[0m
|
||||
�[2m· alpha OwnerId matches alpha account�[0m
|
||||
account_test.go:289: �[2m· alpha OwnerId matches alpha account�[0m
|
||||
�[2m· cross-account stop alpha->beta blocked�[0m
|
||||
account_test.go:296: �[2m· cross-account stop alpha->beta blocked�[0m
|
||||
�[2m· cross-account terminate beta->alpha blocked�[0m
|
||||
account_test.go:304: �[2m· cross-account terminate beta->alpha blocked�[0m
|
||||
�[2m· cross-account reboot alpha->beta blocked�[0m
|
||||
account_test.go:312: �[2m· cross-account reboot alpha->beta blocked�[0m
|
||||
�[2m· stop alpha instance for cross-account start/modify/console tests�[0m
|
||||
account_test.go:321: �[2m· stop alpha instance for cross-account start/modify/console tests�[0m
|
||||
account_test.go:326: instance i-a3072c2387630b8ee reached state stopped
|
||||
�[2m· cross-account start beta->alpha blocked�[0m
|
||||
account_test.go:328: �[2m· cross-account start beta->alpha blocked�[0m
|
||||
�[2m· cross-account modify-instance-attribute beta->alpha blocked�[0m
|
||||
account_test.go:336: �[2m· cross-account modify-instance-attribute beta->alpha blocked�[0m
|
||||
�[2m· cross-account get-console-output beta->alpha blocked�[0m
|
||||
account_test.go:345: �[2m· cross-account get-console-output beta->alpha blocked�[0m
|
||||
�[2m· restart alpha instance�[0m
|
||||
account_test.go:354: �[2m· restart alpha instance�[0m
|
||||
account_test.go:359: instance i-a3072c2387630b8ee reached state running]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step3_VolumeScoping" classname="" time="24.140">
|
||||
<system-out><![CDATA[�[2m· alpha create-volume�[0m
|
||||
account_test.go:366: �[2m· alpha create-volume�[0m
|
||||
�[2malpha_vol�[0m=vol-ceef82652369780af
|
||||
account_test.go:375: �[2malpha_vol�[0m=vol-ceef82652369780af
|
||||
�[2m· beta create-volume�[0m
|
||||
account_test.go:377: �[2m· beta create-volume�[0m
|
||||
�[2mbeta_vol�[0m=vol-08024f32d23d9e18c
|
||||
account_test.go:386: �[2mbeta_vol�[0m=vol-08024f32d23d9e18c
|
||||
account_test.go:388: volume vol-ceef82652369780af reached state available
|
||||
account_test.go:389: volume vol-08024f32d23d9e18c reached state available
|
||||
�[2m· alpha sees only own volumes�[0m
|
||||
account_test.go:391: �[2m· alpha sees only own volumes�[0m
|
||||
�[2m· cross-account describe-volume by id blocked�[0m
|
||||
account_test.go:395: �[2m· cross-account describe-volume by id blocked�[0m
|
||||
�[2m· cross-account delete-volume blocked�[0m
|
||||
account_test.go:403: �[2m· cross-account delete-volume blocked�[0m
|
||||
�[2m· cross-account attach-volume (beta attaches alpha's vol) blocked�[0m
|
||||
account_test.go:411: �[2m· cross-account attach-volume (beta attaches alpha's vol) blocked�[0m
|
||||
�[2m· alpha attach-volume to own instance�[0m
|
||||
account_test.go:422: �[2m· alpha attach-volume to own instance�[0m
|
||||
account_test.go:429: volume vol-ceef82652369780af reached state in-use
|
||||
�[2m· cross-account detach-volume blocked�[0m
|
||||
account_test.go:431: �[2m· cross-account detach-volume blocked�[0m
|
||||
�[2m· cross-account modify-volume blocked�[0m
|
||||
account_test.go:439: �[2m· cross-account modify-volume blocked�[0m
|
||||
�[2m· alpha detach own volume�[0m
|
||||
account_test.go:449: �[2m· alpha detach own volume�[0m
|
||||
account_test.go:454: volume vol-ceef82652369780af reached state available]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step4_KeyPairScoping" classname="" time="0.370">
|
||||
<system-out><![CDATA[�[2m· alpha create-key-pair alpha-key�[0m
|
||||
account_test.go:466: �[2m· alpha create-key-pair alpha-key�[0m
|
||||
�[2m· beta create-key-pair beta-key�[0m
|
||||
account_test.go:473: �[2m· beta create-key-pair beta-key�[0m
|
||||
�[2m· alpha sees only own keys�[0m
|
||||
account_test.go:478: �[2m· alpha sees only own keys�[0m
|
||||
�[2m· namespace isolation: shared-name in both accounts�[0m
|
||||
account_test.go:483: �[2m· namespace isolation: shared-name in both accounts�[0m
|
||||
�[2m· beta deletes alpha-key (idempotent, no cross-account effect)�[0m
|
||||
account_test.go:497: �[2m· beta deletes alpha-key (idempotent, no cross-account effect)�[0m
|
||||
�[2m· alpha import-key-pair imported-key�[0m
|
||||
account_test.go:503: �[2m· alpha import-key-pair imported-key�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step5_SnapshotScoping" classname="" time="0.060">
|
||||
<system-out><![CDATA[�[2m· alpha create-snapshot�[0m
|
||||
account_test.go:522: �[2m· alpha create-snapshot�[0m
|
||||
�[2malpha_snap�[0m=snap-2a47896f4304d805c
|
||||
account_test.go:530: �[2malpha_snap�[0m=snap-2a47896f4304d805c
|
||||
�[2m· beta create-snapshot�[0m
|
||||
account_test.go:532: �[2m· beta create-snapshot�[0m
|
||||
�[2mbeta_snap�[0m=snap-bf5744e78bd7c6908
|
||||
account_test.go:540: �[2mbeta_snap�[0m=snap-bf5744e78bd7c6908
|
||||
�[2m· alpha sees only own snapshots (owner-ids=self)�[0m
|
||||
account_test.go:542: �[2m· alpha sees only own snapshots (owner-ids=self)�[0m
|
||||
�[2m· cross-account delete-snapshot blocked (UnauthorizedOperation)�[0m
|
||||
account_test.go:560: �[2m· cross-account delete-snapshot blocked (UnauthorizedOperation)�[0m
|
||||
�[2m· cross-account create-snapshot of alpha's volume blocked�[0m
|
||||
account_test.go:568: �[2m· cross-account create-snapshot of alpha's volume blocked�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step6_VPCSubnetScoping" classname="" time="0.030">
|
||||
<system-out><![CDATA[�[2m· alpha create-vpc 10.0.0.0/16�[0m
|
||||
account_test.go:582: �[2m· alpha create-vpc 10.0.0.0/16�[0m
|
||||
�[2malpha_vpc�[0m=vpc-7f60df5e87183fb59
|
||||
account_test.go:589: �[2malpha_vpc�[0m=vpc-7f60df5e87183fb59
|
||||
�[2m· beta create-vpc 10.0.0.0/16 (same CIDR, no conflict)�[0m
|
||||
account_test.go:591: �[2m· beta create-vpc 10.0.0.0/16 (same CIDR, no conflict)�[0m
|
||||
�[2mbeta_vpc�[0m=vpc-7e8c859d3985c8218
|
||||
account_test.go:598: �[2mbeta_vpc�[0m=vpc-7e8c859d3985c8218
|
||||
�[2m· alpha describe-vpcs isolation�[0m
|
||||
account_test.go:600: �[2m· alpha describe-vpcs isolation�[0m
|
||||
�[2m· cross-account describe-vpc by id blocked�[0m
|
||||
account_test.go:605: �[2m· cross-account describe-vpc by id blocked�[0m
|
||||
�[2m· cross-account delete-vpc blocked�[0m
|
||||
account_test.go:613: �[2m· cross-account delete-vpc blocked�[0m
|
||||
�[2m· alpha create-subnet 10.0.1.0/24�[0m
|
||||
account_test.go:621: �[2m· alpha create-subnet 10.0.1.0/24�[0m
|
||||
�[2m· beta create-subnet 10.0.1.0/24�[0m
|
||||
account_test.go:630: �[2m· beta create-subnet 10.0.1.0/24�[0m
|
||||
�[2m· alpha describe-subnets isolation�[0m
|
||||
account_test.go:639: �[2m· alpha describe-subnets isolation�[0m
|
||||
�[2m· cross-account create-subnet in other VPC blocked�[0m
|
||||
account_test.go:648: �[2m· cross-account create-subnet in other VPC blocked�[0m
|
||||
�[2m· cross-account delete-subnet blocked�[0m
|
||||
account_test.go:657: �[2m· cross-account delete-subnet blocked�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step7_IGWEIGWScoping" classname="" time="0.030">
|
||||
<system-out><![CDATA[�[2m· alpha create-internet-gateway�[0m
|
||||
account_test.go:673: �[2m· alpha create-internet-gateway�[0m
|
||||
�[2malpha_igw�[0m=igw-ee966e3e923caf630
|
||||
account_test.go:678: �[2malpha_igw�[0m=igw-ee966e3e923caf630
|
||||
�[2m· beta create-internet-gateway�[0m
|
||||
account_test.go:680: �[2m· beta create-internet-gateway�[0m
|
||||
�[2mbeta_igw�[0m=igw-b46e66f2065889bb9
|
||||
account_test.go:685: �[2mbeta_igw�[0m=igw-b46e66f2065889bb9
|
||||
�[2m· alpha describe-internet-gateways isolation�[0m
|
||||
account_test.go:687: �[2m· alpha describe-internet-gateways isolation�[0m
|
||||
�[2m· cross-account describe IGW by id blocked�[0m
|
||||
account_test.go:696: �[2m· cross-account describe IGW by id blocked�[0m
|
||||
�[2m· cross-account delete IGW blocked�[0m
|
||||
account_test.go:704: �[2m· cross-account delete IGW blocked�[0m
|
||||
�[2m· cross-account attach IGW (alpha attaches beta's IGW) blocked�[0m
|
||||
account_test.go:712: �[2m· cross-account attach IGW (alpha attaches beta's IGW) blocked�[0m
|
||||
�[2m· alpha attach own IGW to own VPC�[0m
|
||||
account_test.go:722: �[2m· alpha attach own IGW to own VPC�[0m
|
||||
�[2m· cross-account detach IGW blocked�[0m
|
||||
account_test.go:729: �[2m· cross-account detach IGW blocked�[0m
|
||||
�[2m· alpha create-egress-only-internet-gateway�[0m
|
||||
account_test.go:739: �[2m· alpha create-egress-only-internet-gateway�[0m
|
||||
�[2malpha_eigw�[0m=eigw-51ba82da927e49393
|
||||
account_test.go:746: �[2malpha_eigw�[0m=eigw-51ba82da927e49393
|
||||
�[2m· beta create-egress-only-internet-gateway�[0m
|
||||
account_test.go:748: �[2m· beta create-egress-only-internet-gateway�[0m
|
||||
�[2mbeta_eigw�[0m=eigw-e88d642e78ab4fe5a
|
||||
account_test.go:755: �[2mbeta_eigw�[0m=eigw-e88d642e78ab4fe5a
|
||||
�[2m· alpha describe-eigws isolation�[0m
|
||||
account_test.go:757: �[2m· alpha describe-eigws isolation�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step8_AccountSettings" classname="" time="0.010">
|
||||
<system-out><![CDATA[�[2m· alpha enable-ebs-encryption-by-default�[0m
|
||||
account_test.go:794: �[2m· alpha enable-ebs-encryption-by-default�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step9_GlobalResources" classname="" time="0.010">
|
||||
<system-out><![CDATA[�[2m· describe-regions identical across accounts�[0m
|
||||
account_test.go:829: �[2m· describe-regions identical across accounts�[0m
|
||||
�[2m· describe-availability-zones identical across accounts�[0m
|
||||
account_test.go:834: �[2m· describe-availability-zones identical across accounts�[0m
|
||||
�[2m· describe-instance-types identical across accounts�[0m
|
||||
account_test.go:839: �[2m· describe-instance-types identical across accounts�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step10_EdgeCases" classname="" time="1.200">
|
||||
<system-out><![CDATA[�[2m· create empty Gamma account�[0m
|
||||
account_test.go:849: �[2m· create empty Gamma account�[0m
|
||||
�[2mgamma_account�[0m=000000000004
|
||||
account_test.go:852: �[2mgamma_account�[0m=000000000004
|
||||
�[2m· root create-key-pair root-scoping-key�[0m
|
||||
account_test.go:879: �[2m· root create-key-pair root-scoping-key�[0m
|
||||
�[2m· non-existent volume id -> InvalidVolume.NotFound�[0m
|
||||
account_test.go:895: �[2m· non-existent volume id -> InvalidVolume.NotFound�[0m
|
||||
�[2m· non-existent snapshot id -> InvalidSnapshot.NotFound�[0m
|
||||
account_test.go:903: �[2m· non-existent snapshot id -> InvalidSnapshot.NotFound�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step11_Cleanup" classname="" time="32.440">
|
||||
<system-out><![CDATA[�[2m· terminate alpha instance�[0m
|
||||
account_test.go:920: �[2m· terminate alpha instance�[0m
|
||||
�[2m· terminate beta instance�[0m
|
||||
account_test.go:926: �[2m· terminate beta instance�[0m
|
||||
�[2m· delete snapshots�[0m
|
||||
account_test.go:939: �[2m· delete snapshots�[0m
|
||||
�[2m· delete volumes�[0m
|
||||
account_test.go:955: �[2m· delete volumes�[0m
|
||||
�[2m· delete key pairs�[0m
|
||||
account_test.go:969: �[2m· delete key pairs�[0m
|
||||
�[2m· delete EIGWs�[0m
|
||||
account_test.go:979: �[2m· delete EIGWs�[0m
|
||||
�[2m· detach + delete IGWs�[0m
|
||||
account_test.go:993: �[2m· detach + delete IGWs�[0m
|
||||
�[2m· delete subnets�[0m
|
||||
account_test.go:1013: �[2m· delete subnets�[0m
|
||||
�[2m· delete VPCs�[0m
|
||||
account_test.go:1027: �[2m· delete VPCs�[0m
|
||||
�[2m· Phase 8 acct cleanup (deferred)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8b_VPCSubnetE2E" classname="" time="200.480">
|
||||
<failure message="Failed"><![CDATA[
|
||||
�[1m�[36m━━ Phase 8b — VPC Public/Private Subnet E2E ━━�[0m
|
||||
�[2m· create-vpc 10.99.0.0/16�[0m
|
||||
vpc_test.go:56: �[2m· create-vpc 10.99.0.0/16�[0m
|
||||
�[2mvpc�[0m=vpc-39eff985799c43069 �[2mcidr�[0m=10.99.0.0/16
|
||||
vpc_test.go:67: �[2mvpc�[0m=vpc-39eff985799c43069 �[2mcidr�[0m=10.99.0.0/16
|
||||
�[2m· create-subnet 10.99.1.0/24 (public)�[0m
|
||||
vpc_test.go:70: �[2m· create-subnet 10.99.1.0/24 (public)�[0m
|
||||
�[2m· modify-subnet-attribute MapPublicIpOnLaunch=true subnet-22aba054e10588254�[0m
|
||||
vpc_test.go:87: �[2m· modify-subnet-attribute MapPublicIpOnLaunch=true subnet-22aba054e10588254�[0m
|
||||
�[2mpub_subnet�[0m=subnet-22aba054e10588254 �[2mmap_public_ip�[0m=true
|
||||
vpc_test.go:93: �[2mpub_subnet�[0m=subnet-22aba054e10588254 �[2mmap_public_ip�[0m=true
|
||||
�[2m· create-subnet 10.99.2.0/24 (private)�[0m
|
||||
vpc_test.go:96: �[2m· create-subnet 10.99.2.0/24 (private)�[0m
|
||||
�[2mpriv_subnet�[0m=subnet-f53aeddb73c134696 �[2mmap_public_ip�[0m=false
|
||||
vpc_test.go:111: �[2mpriv_subnet�[0m=subnet-f53aeddb73c134696 �[2mmap_public_ip�[0m=false
|
||||
�[2m· create-internet-gateway�[0m
|
||||
vpc_test.go:114: �[2m· create-internet-gateway�[0m
|
||||
�[2m· attach-internet-gateway igw-1520b5a34480da982 -> vpc-39eff985799c43069�[0m
|
||||
vpc_test.go:133: �[2m· attach-internet-gateway igw-1520b5a34480da982 -> vpc-39eff985799c43069�[0m
|
||||
�[2migw�[0m=igw-1520b5a34480da982
|
||||
vpc_test.go:139: �[2migw�[0m=igw-1520b5a34480da982
|
||||
�[2m· create-route-table (vpc=vpc-39eff985799c43069)�[0m
|
||||
vpc_test.go:142: �[2m· create-route-table (vpc=vpc-39eff985799c43069)�[0m
|
||||
�[2m· associate-route-table rtb-662df5e424cbdb60c <- subnet-22aba054e10588254�[0m
|
||||
vpc_test.go:159: �[2m· associate-route-table rtb-662df5e424cbdb60c <- subnet-22aba054e10588254�[0m
|
||||
�[2m· create-route 0.0.0.0/0 -> igw-1520b5a34480da982�[0m
|
||||
vpc_test.go:176: �[2m· create-route 0.0.0.0/0 -> igw-1520b5a34480da982�[0m
|
||||
�[2mrtb�[0m=rtb-662df5e424cbdb60c �[2massoc�[0m=rtbassoc-5510bf1dc6924eb0b �[2mroute�[0m=0.0.0.0/0->igw-1520b5a34480da982
|
||||
vpc_test.go:192: �[2mrtb�[0m=rtb-662df5e424cbdb60c �[2massoc�[0m=rtbassoc-5510bf1dc6924eb0b �[2mroute�[0m=0.0.0.0/0->igw-1520b5a34480da982
|
||||
�[2m· run-instances ami=ami-6f60f2b3bf04b5453 subnet=subnet-22aba054e10588254�[0m
|
||||
vpc_test.go:195: �[2m· run-instances ami=ami-6f60f2b3bf04b5453 subnet=subnet-22aba054e10588254�[0m
|
||||
�[2minstance�[0m=i-3177888cba3796d42
|
||||
vpc_test.go:216: �[2minstance�[0m=i-3177888cba3796d42
|
||||
vpc_test.go:218: instance i-3177888cba3796d42 reached state running
|
||||
�[2mpublic_ip�[0m=192.168.0.202 �[2mprivate_ip�[0m=10.99.1.4
|
||||
vpc_test.go:224: �[2mpublic_ip�[0m=192.168.0.202 �[2mprivate_ip�[0m=10.99.1.4
|
||||
�[2m· waiting for SSH handshake 192.168.0.202:22�[0m
|
||||
vpc_test.go:227: �[2m· waiting for SSH handshake 192.168.0.202:22�[0m
|
||||
vpc_test.go:227: Eventually: condition not met within 3m0s: [SSH handshake 192.168.0.202:22 never completed]]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8c_RouteTableValidation" classname="" time="0.030">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 8c — Route Table Validation ━━�[0m
|
||||
�[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63
|
||||
vpc_test.go:292: �[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8c_RouteTableValidation/Step1_DefaultMainRouteTable" classname="" time="0.000">
|
||||
<system-out><![CDATA[�[2m· describe-route-tables filter=vpc-id,association.main�[0m
|
||||
vpc_test.go:297: �[2m· describe-route-tables filter=vpc-id,association.main�[0m
|
||||
�[2mmain_rtb�[0m=rtb-e42012569f92f7184
|
||||
vpc_test.go:309: �[2mmain_rtb�[0m=rtb-e42012569f92f7184
|
||||
�[2mlocal_route�[0m=172.31.0.0/16
|
||||
vpc_test.go:332: �[2mlocal_route�[0m=172.31.0.0/16
|
||||
�[2mdefault_route�[0m=0.0.0.0/0->igw-79c2e643660b4b7e3
|
||||
vpc_test.go:350: �[2mdefault_route�[0m=0.0.0.0/0->igw-79c2e643660b4b7e3]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8c_RouteTableValidation/Step2_CustomRouteTableLifecycle" classname="" time="0.020">
|
||||
<system-out><![CDATA[�[2m· create-route-table (vpc=vpc-531252db3b4e1fc08)�[0m
|
||||
vpc_test.go:363: �[2m· create-route-table (vpc=vpc-531252db3b4e1fc08)�[0m
|
||||
�[2mcustom_rtb�[0m=rtb-5e6c46936f3e44167
|
||||
vpc_test.go:379: �[2mcustom_rtb�[0m=rtb-5e6c46936f3e44167
|
||||
�[2mcustom_local_route�[0m=172.31.0.0/16
|
||||
vpc_test.go:396: �[2mcustom_local_route�[0m=172.31.0.0/16
|
||||
�[2mvpc_igw�[0m=igw-79c2e643660b4b7e3
|
||||
vpc_test.go:410: �[2mvpc_igw�[0m=igw-79c2e643660b4b7e3
|
||||
�[2m· create-route 0.0.0.0/0 -> igw-79c2e643660b4b7e3�[0m
|
||||
vpc_test.go:415: �[2m· create-route 0.0.0.0/0 -> igw-79c2e643660b4b7e3�[0m
|
||||
�[2m· associate-route-table rtb-5e6c46936f3e44167 <- subnet-cd62e4c1c3cb50c63�[0m
|
||||
vpc_test.go:434: �[2m· associate-route-table rtb-5e6c46936f3e44167 <- subnet-cd62e4c1c3cb50c63�[0m
|
||||
�[2mcustom_assoc�[0m=rtbassoc-da6377d37847d3d4a
|
||||
vpc_test.go:450: �[2mcustom_assoc�[0m=rtbassoc-da6377d37847d3d4a
|
||||
�[2m· disassociate-route-table rtbassoc-da6377d37847d3d4a�[0m
|
||||
vpc_test.go:452: �[2m· disassociate-route-table rtbassoc-da6377d37847d3d4a�[0m
|
||||
�[2m· delete-route 0.0.0.0/0�[0m
|
||||
vpc_test.go:460: �[2m· delete-route 0.0.0.0/0�[0m
|
||||
�[2m· delete-route-table rtb-5e6c46936f3e44167�[0m
|
||||
vpc_test.go:468: �[2m· delete-route-table rtb-5e6c46936f3e44167�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8c_RouteTableValidation/Step3_ErrorPaths" classname="" time="0.010">
|
||||
<system-out><![CDATA[�[2m· create-route-table (scratch for error paths)�[0m
|
||||
vpc_test.go:481: �[2m· create-route-table (scratch for error paths)�[0m
|
||||
�[2m· delete-route local=172.31.0.0/16 (must fail)�[0m
|
||||
vpc_test.go:517: �[2m· delete-route local=172.31.0.0/16 (must fail)�[0m
|
||||
�[2mdelete_local_err�[0m=InvalidParameterValue
|
||||
vpc_test.go:529: �[2mdelete_local_err�[0m=InvalidParameterValue
|
||||
�[2m· delete-route-table rtb-e42012569f92f7184 (main, must fail)�[0m
|
||||
vpc_test.go:533: �[2m· delete-route-table rtb-e42012569f92f7184 (main, must fail)�[0m
|
||||
�[2mdelete_main_err�[0m=DependencyViolation
|
||||
vpc_test.go:540: �[2mdelete_main_err�[0m=DependencyViolation]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8d_NATGateway" classname="" time="106.010">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 8d — NAT Gateway E2E ━━�[0m
|
||||
�[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2mpub_subnet�[0m=subnet-cd62e4c1c3cb50c63 �[2msg�[0m=sg-764a7fa2a39577204
|
||||
natgw_test.go:54: �[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2mpub_subnet�[0m=subnet-cd62e4c1c3cb50c63 �[2msg�[0m=sg-764a7fa2a39577204
|
||||
�[2m· create-subnet 172.31.16.0/20 (private)�[0m
|
||||
natgw_test.go:58: �[2m· create-subnet 172.31.16.0/20 (private)�[0m
|
||||
�[2mpriv_subnet�[0m=subnet-2360a40e28371e214
|
||||
natgw_test.go:71: �[2mpriv_subnet�[0m=subnet-2360a40e28371e214
|
||||
�[2m· run-instances bastion (default subnet)�[0m
|
||||
natgw_test.go:74: �[2m· run-instances bastion (default subnet)�[0m
|
||||
�[2m· run-instances private (private subnet)�[0m
|
||||
natgw_test.go:98: �[2m· run-instances private (private subnet)�[0m
|
||||
natgw_test.go:118: instance i-268402f0e28b5e477 reached state running
|
||||
natgw_test.go:119: instance i-db7f3ed6e4cdaa5aa reached state running
|
||||
�[2mbastion�[0m=i-268402f0e28b5e477 �[2mbastion_ip�[0m=192.168.0.202 �[2mprivate�[0m=i-db7f3ed6e4cdaa5aa �[2mprivate_ip�[0m=172.31.16.4
|
||||
natgw_test.go:128: �[2mbastion�[0m=i-268402f0e28b5e477 �[2mbastion_ip�[0m=192.168.0.202 �[2mprivate�[0m=i-db7f3ed6e4cdaa5aa �[2mprivate_ip�[0m=172.31.16.4
|
||||
�[2m· waiting for SSH handshake 192.168.0.202:22�[0m
|
||||
natgw_test.go:133: �[2m· waiting for SSH handshake 192.168.0.202:22�[0m
|
||||
�[2m· scp keypair -> bastion:/tmp/key.pem�[0m
|
||||
natgw_test.go:139: �[2m· scp keypair -> bastion:/tmp/key.pem�[0m
|
||||
�[2m· wait for private SSH via bastion�[0m
|
||||
natgw_test.go:151: �[2m· wait for private SSH via bastion�[0m
|
||||
�[2m· baseline: private VM has no internet�[0m
|
||||
natgw_test.go:172: �[2m· baseline: private VM has no internet�[0m
|
||||
�[2m· allocate-address (NAT EIP)�[0m
|
||||
natgw_test.go:178: �[2m· allocate-address (NAT EIP)�[0m
|
||||
�[2meip�[0m=192.168.0.203 �[2malloc�[0m=eipalloc-2d81753157cd62a50
|
||||
natgw_test.go:195: �[2meip�[0m=192.168.0.203 �[2malloc�[0m=eipalloc-2d81753157cd62a50
|
||||
�[2m· create-nat-gateway in subnet-cd62e4c1c3cb50c63�[0m
|
||||
natgw_test.go:197: �[2m· create-nat-gateway in subnet-cd62e4c1c3cb50c63�[0m
|
||||
�[2mnat_gw�[0m=nat-812216dd5105180c5
|
||||
natgw_test.go:216: �[2mnat_gw�[0m=nat-812216dd5105180c5
|
||||
natgw_test.go:218: nat gateway nat-812216dd5105180c5 reached state available
|
||||
�[2m· create-route-table�[0m
|
||||
natgw_test.go:223: �[2m· create-route-table�[0m
|
||||
�[2m· associate-route-table rtb-8d6a4abd5f666ec13 <- subnet-2360a40e28371e214�[0m
|
||||
natgw_test.go:241: �[2m· associate-route-table rtb-8d6a4abd5f666ec13 <- subnet-2360a40e28371e214�[0m
|
||||
�[2m· create-route 0.0.0.0/0 -> nat-812216dd5105180c5�[0m
|
||||
natgw_test.go:259: �[2m· create-route 0.0.0.0/0 -> nat-812216dd5105180c5�[0m
|
||||
�[2m· verify private VM reaches 8.8.8.8 via NAT GW�[0m
|
||||
natgw_test.go:279: �[2m· verify private VM reaches 8.8.8.8 via NAT GW�[0m
|
||||
�[2m· delete-nat-gateway nat-812216dd5105180c5�[0m
|
||||
natgw_test.go:293: �[2m· delete-nat-gateway nat-812216dd5105180c5�[0m
|
||||
natgw_test.go:318: nat gateway nat-812216dd5105180c5 reached state deleted
|
||||
�[2m· verify private VM lost internet after NAT GW teardown�[0m
|
||||
natgw_test.go:329: �[2m· verify private VM lost internet after NAT GW teardown�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8e_SGToSGDatapath" classname="" time="80.940">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 8e — Security Group SG-to-SG Datapath (OVN) ━━�[0m
|
||||
�[2m· 8e-1 create sge-client + sge-target security groups�[0m
|
||||
datapath_test.go:61: �[2m· 8e-1 create sge-client + sge-target security groups�[0m
|
||||
�[2mclient_sg�[0m=sg-56e73c764a649f7ff �[2mtarget_sg�[0m=sg-67ad5efaa8be83b79
|
||||
datapath_test.go:83: �[2mclient_sg�[0m=sg-56e73c764a649f7ff �[2mtarget_sg�[0m=sg-67ad5efaa8be83b79
|
||||
�[2m· 8e-1 authorize target-sg ingress tcp/8080 from sg-56e73c764a649f7ff�[0m
|
||||
datapath_test.go:91: �[2m· 8e-1 authorize target-sg ingress tcp/8080 from sg-56e73c764a649f7ff�[0m
|
||||
�[2m· 8e-2 launch client-vm + target-vm�[0m
|
||||
datapath_test.go:109: �[2m· 8e-2 launch client-vm + target-vm�[0m
|
||||
datapath_test.go:132: instance i-eae6263d9b9fda069 reached state running
|
||||
datapath_test.go:133: instance i-9d9411a83549d34e3 reached state running
|
||||
�[2mclient_priv�[0m=172.31.0.5 �[2mclient_eni�[0m=eni-0f878da0143028915 �[2mtarget_priv�[0m=172.31.0.6 �[2mtarget_eni�[0m=eni-3bd355e19c7fdd45c
|
||||
datapath_test.go:142: �[2mclient_priv�[0m=172.31.0.5 �[2mclient_eni�[0m=eni-0f878da0143028915 �[2mtarget_priv�[0m=172.31.0.6 �[2mtarget_eni�[0m=eni-3bd355e19c7fdd45c
|
||||
�[2m· 8e-3 ovn-nbctl port_group membership�[0m
|
||||
datapath_test.go:151: �[2m· 8e-3 ovn-nbctl port_group membership�[0m
|
||||
�[2mclient_pg�[0m=sg_56e73c764a649f7ff �[2mtarget_pg�[0m=sg_67ad5efaa8be83b79
|
||||
datapath_test.go:163: �[2mclient_pg�[0m=sg_56e73c764a649f7ff �[2mtarget_pg�[0m=sg_67ad5efaa8be83b79
|
||||
�[2m· 8e-3 ovn-sbctl address_set sg_56e73c764a649f7ff_ip4 contains 172.31.0.5�[0m
|
||||
datapath_test.go:170: �[2m· 8e-3 ovn-sbctl address_set sg_56e73c764a649f7ff_ip4 contains 172.31.0.5�[0m
|
||||
�[2m· 8e-3 wait for client-vm SSH at 192.168.0.202:22�[0m
|
||||
datapath_test.go:188: �[2m· 8e-3 wait for client-vm SSH at 192.168.0.202:22�[0m
|
||||
�[2m· 8e-4 allowed traffic client -> target:172.31.0.6:8080�[0m
|
||||
datapath_test.go:199: �[2m· 8e-4 allowed traffic client -> target:172.31.0.6:8080�[0m
|
||||
�[2mstep4�[0m=allowed_traffic_ok
|
||||
datapath_test.go:208: �[2mstep4�[0m=allowed_traffic_ok
|
||||
�[2m· 8e-5 denied traffic client -> target:22 (no SSH ingress)�[0m
|
||||
datapath_test.go:213: �[2m· 8e-5 denied traffic client -> target:22 (no SSH ingress)�[0m
|
||||
�[2mstep5�[0m=denied_traffic_ok
|
||||
datapath_test.go:217: �[2mstep5�[0m=denied_traffic_ok
|
||||
�[2m· 8e-6 revoke target-sg ingress, retest (sync RequestEvent contract)�[0m
|
||||
datapath_test.go:222: �[2m· 8e-6 revoke target-sg ingress, retest (sync RequestEvent contract)�[0m
|
||||
�[2m· 8e-6 verify client -> target:8080 now blocked�[0m
|
||||
datapath_test.go:240: �[2m· 8e-6 verify client -> target:8080 now blocked�[0m
|
||||
�[2mstep6�[0m=revoke_blocked_ok
|
||||
datapath_test.go:244: �[2mstep6�[0m=revoke_blocked_ok
|
||||
�[2m· 8e-7 re-authorize target-sg ingress, verify traffic restored�[0m
|
||||
datapath_test.go:250: �[2m· 8e-7 re-authorize target-sg ingress, verify traffic restored�[0m
|
||||
�[2mstep7�[0m=restore_ok
|
||||
datapath_test.go:271: �[2mstep7�[0m=restore_ok
|
||||
datapath_test.go:127: instance i-eae6263d9b9fda069 reached state terminated
|
||||
datapath_test.go:127: instance i-9d9411a83549d34e3 reached state terminated]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase9_Teardown" classname="" time="8.300">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 9 — Teardown ━━�[0m
|
||||
�[2m· delete-snapshot snap-dde89b9ea2ef2e05a (custom AMI backing)�[0m
|
||||
teardown_test.go:33: �[2m· delete-snapshot snap-dde89b9ea2ef2e05a (custom AMI backing)�[0m
|
||||
�[2mdeleted_snapshot�[0m=snap-dde89b9ea2ef2e05a
|
||||
teardown_test.go:39: �[2mdeleted_snapshot�[0m=snap-dde89b9ea2ef2e05a
|
||||
�[2m· deregister-image ami-7dcf263a412dd14c6�[0m
|
||||
teardown_test.go:51: �[2m· deregister-image ami-7dcf263a412dd14c6�[0m
|
||||
�[2mderegistered_ami�[0m=ami-7dcf263a412dd14c6
|
||||
teardown_test.go:57: �[2mderegistered_ami�[0m=ami-7dcf263a412dd14c6
|
||||
�[2m· terminate-instances i-0c615e5dd74050f53�[0m
|
||||
teardown_test.go:74: �[2m· terminate-instances i-0c615e5dd74050f53�[0m
|
||||
�[2mterminated_instance�[0m=i-0c615e5dd74050f53
|
||||
teardown_test.go:81: �[2mterminated_instance�[0m=i-0c615e5dd74050f53
|
||||
�[2m· delete-key-pair test-key-1�[0m
|
||||
teardown_test.go:94: �[2m· delete-key-pair test-key-1�[0m
|
||||
�[2mdeleted_keypair�[0m=test-key-1
|
||||
teardown_test.go:100: �[2mdeleted_keypair�[0m=test-key-1]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase9a_VerifyTeardown" classname="" time="0.560">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 9a — Verify Teardown ━━�[0m
|
||||
�[2m· describe-instances i-0c615e5dd74050f53�[0m
|
||||
teardown_test.go:123: �[2m· describe-instances i-0c615e5dd74050f53�[0m
|
||||
�[2minstance_state�[0m=terminated
|
||||
teardown_test.go:135: �[2minstance_state�[0m=terminated
|
||||
�[2m· describe-images ami-7dcf263a412dd14c6�[0m
|
||||
teardown_test.go:147: �[2m· describe-images ami-7dcf263a412dd14c6�[0m
|
||||
�[2mami_reaped�[0m=ami-7dcf263a412dd14c6
|
||||
teardown_test.go:159: �[2mami_reaped�[0m=ami-7dcf263a412dd14c6
|
||||
�[2m· describe-key-pairs (must not contain test-key-1)�[0m
|
||||
teardown_test.go:167: �[2m· describe-key-pairs (must not contain test-key-1)�[0m
|
||||
�[2mkeypairs_listed�[0m=1
|
||||
teardown_test.go:176: �[2mkeypairs_listed�[0m=1]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase9b_FinalClusterStats" classname="" time="3.070">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 9b — Final Cluster Stats ━━�[0m
|
||||
�[2mspx_get_vms_bytes�[0m=118
|
||||
teardown_test.go:208: �[2mspx_get_vms_bytes�[0m=118]]></system-out>
|
||||
</testcase>
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -0,0 +1 @@
|
||||
__pycache__/
|
||||
@@ -0,0 +1,46 @@
|
||||
# junit-report
|
||||
|
||||
Composite action that renders a glob of JUnit XML files into the GitHub
|
||||
Actions job summary and emits one `::error` annotation per failed test case.
|
||||
|
||||
## Usage
|
||||
|
||||
```yaml
|
||||
- name: Publish test summary
|
||||
if: always()
|
||||
uses: ./spinifex/.github/actions/junit-report # or your own checkout path
|
||||
with:
|
||||
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
|
||||
title: "Go E2E results"
|
||||
# fail-on-empty: "true" # opt-in: fail the step if the glob matches nothing
|
||||
```
|
||||
|
||||
Outputs:
|
||||
|
||||
- `failures` — total `<failure>` + `<error>` counts across all matched files.
|
||||
- `total` — total `<testcase>` count across all matched files.
|
||||
|
||||
The action expects standard JUnit XML (e.g. emitted by
|
||||
`gotestsum --junitfile`, `go-junit-report`, or pytest's `--junitxml`). Both
|
||||
single `<testsuite>` and wrapping `<testsuites>` roots are accepted.
|
||||
|
||||
## Companion convention: `::group::` for noisy steps
|
||||
|
||||
Long-running shell phases (tofu apply, package installs, tar-over-ssh) should
|
||||
wrap their output in `::group::name` / `::endgroup::` markers so the Actions
|
||||
UI collapses them by default. Pattern used in the e2e-go / e2e-ddil workflows:
|
||||
|
||||
```bash
|
||||
run() {
|
||||
local label="$1"; shift
|
||||
echo "::group::${label}"
|
||||
"$@" 2>&1 | tee -a "$TOFU_LOG"
|
||||
local rc=${PIPESTATUS[0]}
|
||||
echo "::endgroup::"
|
||||
return "$rc"
|
||||
}
|
||||
run "tofu apply" tofu apply -var-file=... -auto-approve -input=false
|
||||
```
|
||||
|
||||
Combined: groups make the live UI readable; this action makes the run header
|
||||
land on the failures without scrolling the log.
|
||||
@@ -0,0 +1,41 @@
|
||||
name: JUnit Report
|
||||
description: >-
|
||||
Render JUnit XML files into the GitHub Actions job summary and emit
|
||||
inline ::error annotations for each failed test case. Designed to be
|
||||
called once per job after the test step (with `if: always()`) so summary
|
||||
+ annotations appear on both passing and failing runs.
|
||||
|
||||
inputs:
|
||||
junit-glob:
|
||||
description: Shell glob matching JUnit XML files to render.
|
||||
required: true
|
||||
title:
|
||||
description: Heading rendered at the top of the job summary section.
|
||||
required: false
|
||||
default: "Test results"
|
||||
fail-on-empty:
|
||||
description: >-
|
||||
When "true", the action fails if the glob matches no files. Useful
|
||||
for catching silently-skipped test phases.
|
||||
required: false
|
||||
default: "false"
|
||||
|
||||
outputs:
|
||||
failures:
|
||||
description: Total failed + errored test cases across all matched files.
|
||||
value: ${{ steps.render.outputs.failures }}
|
||||
total:
|
||||
description: Total test cases across all matched files.
|
||||
value: ${{ steps.render.outputs.total }}
|
||||
|
||||
runs:
|
||||
using: composite
|
||||
steps:
|
||||
- id: render
|
||||
shell: bash
|
||||
env:
|
||||
JUNIT_GLOB: ${{ inputs.junit-glob }}
|
||||
JUNIT_TITLE: ${{ inputs.title }}
|
||||
JUNIT_FAIL_ON_EMPTY: ${{ inputs.fail-on-empty }}
|
||||
run: |
|
||||
python3 "${{ github.action_path }}/render.py"
|
||||
@@ -0,0 +1,281 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Render JUnit XML files into GitHub Actions job summary + annotations.
|
||||
|
||||
Inputs (env):
|
||||
JUNIT_GLOB Glob matching JUnit XML files.
|
||||
JUNIT_TITLE Heading text for the summary section.
|
||||
JUNIT_FAIL_ON_EMPTY "true" to exit non-zero when glob matches nothing.
|
||||
|
||||
GitHub-Actions outputs (via $GITHUB_OUTPUT):
|
||||
failures Sum of <failures> + <errors> across all suites.
|
||||
total Sum of <tests> across all suites.
|
||||
|
||||
Side effects:
|
||||
Appends a markdown section to $GITHUB_STEP_SUMMARY with a per-file
|
||||
totals table and a collapsible details block per failed test.
|
||||
Emits one `::error` workflow command per failed/errored test case so
|
||||
they surface as annotations in the run UI.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import glob
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import xml.etree.ElementTree as ET
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
# Strip ANSI SGR escapes from skipped/failure CDATA so the rendered reason is
|
||||
# plain text (CDATA preserves the colour codes Go emits via harness.Phase).
|
||||
_ANSI_RE = re.compile(r"\x1b\[[0-9;]*m")
|
||||
|
||||
|
||||
def _strip_ansi(s: str) -> str:
|
||||
return _ANSI_RE.sub("", s)
|
||||
|
||||
|
||||
# Last `<file>.go:<line>: <reason>` line in a CDATA block is the t.Skipf /
|
||||
# t.Fatal call site — that's the actionable signal. Phase banners and probe
|
||||
# logs above it are noise for the summary view.
|
||||
_SKIP_REASON_RE = re.compile(r"([A-Za-z0-9_.\-/]+\.go:\d+:\s*.+)$")
|
||||
|
||||
|
||||
def _extract_skip_reason(cdata: str) -> str:
|
||||
cdata = _strip_ansi(cdata).strip()
|
||||
if not cdata:
|
||||
return ""
|
||||
for line in reversed(cdata.splitlines()):
|
||||
line = line.strip()
|
||||
m = _SKIP_REASON_RE.search(line)
|
||||
if m:
|
||||
return m.group(1).strip()
|
||||
return cdata.splitlines()[-1].strip()
|
||||
|
||||
|
||||
# Workflow commands carry their payload on one line; literal newlines must
|
||||
# be percent-escaped per GitHub's workflow-command spec.
|
||||
def _esc(s: str) -> str:
|
||||
return (
|
||||
s.replace("%", "%25")
|
||||
.replace("\r", "%0D")
|
||||
.replace("\n", "%0A")
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Case:
|
||||
suite: str
|
||||
classname: str
|
||||
name: str
|
||||
time: float
|
||||
status: str # "pass" | "fail" | "error" | "skip"
|
||||
message: str
|
||||
detail: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class FileTotals:
|
||||
path: str
|
||||
tests: int
|
||||
failures: int
|
||||
errors: int
|
||||
skipped: int
|
||||
time: float
|
||||
cases: list[Case]
|
||||
|
||||
|
||||
def _parse_file(path: str) -> FileTotals:
|
||||
root = ET.parse(path).getroot()
|
||||
# JUnit XML has two shapes: <testsuite> as root, or <testsuites> wrapping
|
||||
# multiple <testsuite> children. Iterating with .iter() handles both.
|
||||
suites = list(root.iter("testsuite"))
|
||||
tests = sum(int(s.get("tests", 0) or 0) for s in suites)
|
||||
failures = sum(int(s.get("failures", 0) or 0) for s in suites)
|
||||
errors = sum(int(s.get("errors", 0) or 0) for s in suites)
|
||||
skipped = sum(int(s.get("skipped", 0) or 0) for s in suites)
|
||||
time = sum(float(s.get("time", 0) or 0) for s in suites)
|
||||
|
||||
cases: list[Case] = []
|
||||
for s in suites:
|
||||
suite_name = s.get("name", "")
|
||||
for tc in s.findall("testcase"):
|
||||
status = "pass"
|
||||
message = ""
|
||||
detail = ""
|
||||
for tag in ("failure", "error"):
|
||||
node = tc.find(tag)
|
||||
if node is not None:
|
||||
status = "fail" if tag == "failure" else "error"
|
||||
message = (node.get("message") or "").strip()
|
||||
detail = (node.text or "").strip()
|
||||
break
|
||||
skip_node = tc.find("skipped")
|
||||
if skip_node is not None and status == "pass":
|
||||
status = "skip"
|
||||
# Skipped reason: prefer the CDATA call-site line, fall back
|
||||
# to the @message attribute if CDATA is empty.
|
||||
cdata = (skip_node.text or "").strip()
|
||||
message = _extract_skip_reason(cdata) or (
|
||||
skip_node.get("message") or ""
|
||||
).strip()
|
||||
cases.append(
|
||||
Case(
|
||||
suite=suite_name,
|
||||
classname=tc.get("classname", ""),
|
||||
name=tc.get("name", ""),
|
||||
time=float(tc.get("time", 0) or 0),
|
||||
status=status,
|
||||
message=message,
|
||||
detail=detail,
|
||||
)
|
||||
)
|
||||
|
||||
# Drop container test cases: any case whose name is a strict prefix of
|
||||
# another case's name (with "/" separator) is a parent t.Run wrapper.
|
||||
# Go reports the parent's status as a rollup of its children, so counting
|
||||
# both double-tallies the same outcome (e.g. TestSingleNode failing
|
||||
# because Phase8b_VPCSubnetE2E failed would show as 2 failures, not 1).
|
||||
names = {c.name for c in cases}
|
||||
prefixes = {n.rsplit("/", 1)[0] for n in names if "/" in n}
|
||||
# A name is a container if some other name starts with "<name>/".
|
||||
# Use the prefixes set rather than O(n^2) scanning.
|
||||
def _is_container(name: str) -> bool:
|
||||
return name in prefixes
|
||||
|
||||
leaf_cases = [c for c in cases if not _is_container(c.name)]
|
||||
|
||||
# Recompute totals from leaf cases so the per-file counts in the summary
|
||||
# table match the de-duplicated failure list below it.
|
||||
leaf_tests = len(leaf_cases)
|
||||
leaf_failures = sum(1 for c in leaf_cases if c.status == "fail")
|
||||
leaf_errors = sum(1 for c in leaf_cases if c.status == "error")
|
||||
leaf_skipped = sum(1 for c in leaf_cases if c.status == "skip")
|
||||
|
||||
return FileTotals(
|
||||
path=path,
|
||||
tests=leaf_tests,
|
||||
failures=leaf_failures,
|
||||
errors=leaf_errors,
|
||||
skipped=leaf_skipped,
|
||||
time=time,
|
||||
cases=leaf_cases,
|
||||
)
|
||||
|
||||
|
||||
def _file_label(path: str) -> str:
|
||||
base = os.path.basename(path)
|
||||
if base.endswith(".xml"):
|
||||
base = base[:-4]
|
||||
if base.startswith("junit-"):
|
||||
base = base[len("junit-"):]
|
||||
return base or path
|
||||
|
||||
|
||||
def main() -> int:
|
||||
pattern = os.environ.get("JUNIT_GLOB", "").strip()
|
||||
title = os.environ.get("JUNIT_TITLE", "Test results").strip() or "Test results"
|
||||
fail_on_empty = os.environ.get("JUNIT_FAIL_ON_EMPTY", "").strip().lower() == "true"
|
||||
|
||||
if not pattern:
|
||||
print("::error::junit-report: junit-glob input is empty", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
paths = sorted(glob.glob(pattern))
|
||||
summary_path = os.environ.get("GITHUB_STEP_SUMMARY", "/dev/null")
|
||||
output_path = os.environ.get("GITHUB_OUTPUT", "/dev/null")
|
||||
|
||||
with open(summary_path, "a", encoding="utf-8") as summary, \
|
||||
open(output_path, "a", encoding="utf-8") as outputs:
|
||||
|
||||
summary.write(f"## {title}\n\n")
|
||||
|
||||
if not paths:
|
||||
summary.write(f"_No JUnit files matched `{pattern}`._\n\n")
|
||||
outputs.write("failures=0\n")
|
||||
outputs.write("total=0\n")
|
||||
if fail_on_empty:
|
||||
print(f"::error::junit-report: no files matched {pattern}", file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
totals_by_file = [_parse_file(p) for p in paths]
|
||||
grand_total = sum(f.tests for f in totals_by_file)
|
||||
grand_fail = sum(f.failures + f.errors for f in totals_by_file)
|
||||
grand_skip = sum(f.skipped for f in totals_by_file)
|
||||
grand_pass = grand_total - grand_fail - grand_skip
|
||||
grand_time = sum(f.time for f in totals_by_file)
|
||||
|
||||
if grand_fail == 0:
|
||||
verdict = f"✅ **pass** — {grand_pass} / {grand_total}"
|
||||
else:
|
||||
verdict = f"❌ **{grand_fail} failed** — {grand_pass} / {grand_total} pass"
|
||||
skip_part = f", {grand_skip} skipped" if grand_skip else ""
|
||||
summary.write(f"{verdict}{skip_part} · {grand_time:.1f}s\n\n")
|
||||
|
||||
# Compact table — single status column, shorter headers. Drops
|
||||
# separate err column (errors fold into fail count) so narrow
|
||||
# screens don't wrap.
|
||||
summary.write("| | Suite | Pass | Fail | Skip | Time |\n")
|
||||
summary.write("|:-:|---|---:|---:|---:|---:|\n")
|
||||
for f in totals_by_file:
|
||||
fails = f.failures + f.errors
|
||||
passed = f.tests - fails - f.skipped
|
||||
icon = "✅" if fails == 0 else "❌"
|
||||
summary.write(
|
||||
f"| {icon} | `{_file_label(f.path)}` | {passed} | {fails} | "
|
||||
f"{f.skipped} | {f.time:.1f}s |\n"
|
||||
)
|
||||
summary.write("\n")
|
||||
|
||||
failed_cases = [
|
||||
c for f in totals_by_file for c in f.cases if c.status in ("fail", "error")
|
||||
]
|
||||
if failed_cases:
|
||||
summary.write("---\n\n**Failures**\n\n")
|
||||
for c in failed_cases:
|
||||
header = f"{c.classname}.{c.name}" if c.classname else c.name
|
||||
# One line per failure: name + truncated message. Full
|
||||
# detail lives in the test artifact log; deep nesting and
|
||||
# long messages wrap badly on narrow screens.
|
||||
msg_short = (c.message or "").splitlines()[0] if c.message else ""
|
||||
if len(msg_short) > 120:
|
||||
msg_short = msg_short[:117] + "…"
|
||||
line = f"- ❌ `{header}`"
|
||||
if msg_short:
|
||||
line += f" — {msg_short}"
|
||||
summary.write(line + "\n")
|
||||
|
||||
# Workflow annotation — surfaces in run header + Files-changed view.
|
||||
# classname for Go tests is the package path; map to a file when we can.
|
||||
file_hint = c.classname.replace(".", "/")
|
||||
msg = c.message or f"{c.status}: {header}"
|
||||
print(f"::error file={file_hint},title={header}::{_esc(msg)}")
|
||||
summary.write("\n")
|
||||
|
||||
skipped_cases = [
|
||||
c for f in totals_by_file for c in f.cases if c.status == "skip"
|
||||
]
|
||||
if skipped_cases:
|
||||
summary.write("---\n\n<details><summary><b>Skipped</b> ")
|
||||
summary.write(f"({len(skipped_cases)})</summary>\n\n")
|
||||
for c in skipped_cases:
|
||||
header = f"{c.classname}.{c.name}" if c.classname else c.name
|
||||
reason = (c.message or "").splitlines()[0] if c.message else ""
|
||||
if len(reason) > 160:
|
||||
reason = reason[:157] + "…"
|
||||
line = f"- ⏭️ `{header}`"
|
||||
if reason:
|
||||
line += f" — {reason}"
|
||||
summary.write(line + "\n")
|
||||
summary.write("\n</details>\n\n")
|
||||
|
||||
outputs.write(f"failures={grand_fail}\n")
|
||||
outputs.write(f"total={grand_total}\n")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,63 @@
|
||||
# shellcheck shell=bash
|
||||
# Source from a workflow `run:` block:
|
||||
# source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
|
||||
#
|
||||
# Provides colored banners + ::group::/::endgroup:: wrappers. ANSI escapes
|
||||
# render in the GitHub Actions live log; the same lines also land in any
|
||||
# artifact log (tee'd output) without breaking parsers because they only
|
||||
# affect terminal display.
|
||||
#
|
||||
# Functions kept short and unindented on purpose — wide nested output wraps
|
||||
# badly on narrow terminals.
|
||||
|
||||
if [[ -n "${GITHUB_ACTIONS:-}" ]] || [[ -t 1 ]]; then
|
||||
_R=$'\033[0m'
|
||||
_B=$'\033[1m' # bold
|
||||
_D=$'\033[2m' # dim
|
||||
_RED=$'\033[31m'
|
||||
_GRN=$'\033[32m'
|
||||
_YEL=$'\033[33m'
|
||||
_BLU=$'\033[34m'
|
||||
_CYA=$'\033[36m'
|
||||
else
|
||||
_R="" _B="" _D="" _RED="" _GRN="" _YEL="" _BLU="" _CYA=""
|
||||
fi
|
||||
|
||||
# Single-line rule using box-drawing chars — narrow enough that GitHub's
|
||||
# timestamp prefix + this rule still fit on an 80-col terminal.
|
||||
# banner LABEL — bold cyan title with surrounding rule. Used at the start
|
||||
# of an inherently multi-line phase (test run, artifact pull) so the live
|
||||
# log has an obvious visual marker.
|
||||
banner() {
|
||||
printf '\n%s%s━━ %s ━━%s\n' "$_B" "$_CYA" "$*" "$_R"
|
||||
}
|
||||
|
||||
# Status indicators. Single char prefix only — keeps width on narrow logs.
|
||||
ok() { printf '%s✓%s %s\n' "$_GRN" "$_R" "$*"; }
|
||||
bad() { printf '%s✗%s %s\n' "$_RED" "$_R" "$*"; }
|
||||
warn() { printf '%s!%s %s\n' "$_YEL" "$_R" "$*"; }
|
||||
note() { printf '%s·%s %s\n' "$_D" "$_R" "$*"; }
|
||||
|
||||
# group_open LABEL / group_close — thin wrapper around GitHub's collapse
|
||||
# markers. No extra body printed; banner() is the visible inside-group
|
||||
# heading when callers want one.
|
||||
group_open() { echo "::group::$*"; }
|
||||
group_close() { echo "::endgroup::"; }
|
||||
|
||||
# quiet_run LABEL LOG_FILE -- CMD ARGS …
|
||||
# Run CMD redirected to LOG_FILE (append). Print one colored ok/✗ line.
|
||||
# On failure, dump the last 80 lines of LOG_FILE inside a collapsed group
|
||||
# so the failing tail lands in the UI without surfacing the whole apply.
|
||||
quiet_run() {
|
||||
local label="$1" log="$2"; shift 2
|
||||
if "$@" >> "$log" 2>&1; then
|
||||
ok "$label"
|
||||
else
|
||||
local rc=$?
|
||||
bad "$label (rc=$rc) — last 80 lines of $(basename "$log"):"
|
||||
group_open "tail $(basename "$log")"
|
||||
tail -n 80 "$log"
|
||||
group_close
|
||||
return "$rc"
|
||||
fi
|
||||
}
|
||||
@@ -87,12 +87,16 @@ jobs:
|
||||
- name: Provision VMs
|
||||
working-directory: mulga/scripts/tofu-cluster
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# shellcheck source=/dev/null
|
||||
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
|
||||
export TF_VAR_git_branch="${GIT_BRANCH}"
|
||||
tofu init -input=false >> "$TOFU_LOG" 2>&1
|
||||
tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
|
||||
tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
|
||||
./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
|
||||
tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
|
||||
quiet_run "tofu init" "$TOFU_LOG" tofu init -input=false
|
||||
quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
|
||||
quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
|
||||
quiet_run "scrub-env (pre-clean)" "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
|
||||
quiet_run "tofu apply" "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
|
||||
ok "VMs provisioned"
|
||||
|
||||
- name: Bootstrap via Binary Install
|
||||
working-directory: mulga/scripts/tofu-cluster
|
||||
@@ -147,8 +151,13 @@ jobs:
|
||||
|
||||
mkdir -p "${RUNNER_TEMP}/ddil-results"
|
||||
cd "${GITHUB_WORKSPACE}/spinifex"
|
||||
# gotestsum: human-readable PASS/FAIL/SKIP lines in Actions stdout
|
||||
# while still emitting the machine-readable jsonfile artifact.
|
||||
# gotestsum: human-readable PASS/FAIL/SKIP lines in Actions stdout,
|
||||
# plus machine-readable jsonfile + JUnit XML artifacts. The JUnit
|
||||
# file feeds the junit-report composite action below so failures
|
||||
# become summary table rows + workflow annotations.
|
||||
# shellcheck source=/dev/null
|
||||
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
|
||||
banner "DDIL scenarios"
|
||||
DDIL_NODES="${NODE_IPS}" \
|
||||
DDIL_SSH_USER="${SSH_USER}" \
|
||||
DDIL_SSH_KEY="${SSH_KEY}" \
|
||||
@@ -158,8 +167,19 @@ jobs:
|
||||
AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" \
|
||||
AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" \
|
||||
gotestsum --jsonfile "${RUNNER_TEMP}/ddil-results/results.json" \
|
||||
--junitfile "${RUNNER_TEMP}/ddil-results/junit-ddil.xml" \
|
||||
--format testname \
|
||||
-- -tags=e2e -timeout 45m ./tests/e2e/ddil/scenarios/...
|
||||
rc=$?
|
||||
if [[ $rc -eq 0 ]]; then ok "all scenarios passed"; else bad "scenarios failed (rc=$rc)"; fi
|
||||
exit $rc
|
||||
|
||||
- name: Publish test summary
|
||||
if: always()
|
||||
uses: ./spinifex/.github/actions/junit-report
|
||||
with:
|
||||
junit-glob: ${{ runner.temp }}/ddil-results/junit-*.xml
|
||||
title: "DDIL scenario results"
|
||||
|
||||
- name: Collect Cluster Logs
|
||||
if: failure()
|
||||
@@ -177,13 +197,15 @@ jobs:
|
||||
LOG_DIR="${RUNNER_TEMP}/ddil-results/cluster-logs"
|
||||
mkdir -p "${LOG_DIR}"
|
||||
|
||||
# shellcheck source=/dev/null
|
||||
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
|
||||
mapfile -t NODE_IPS < <(echo "$INVENTORY" | jq -r '.nodes[].wan_ip')
|
||||
for ip in "${NODE_IPS[@]}"; do
|
||||
echo "=== collecting ${ip} ==="
|
||||
SSH="ssh -i ${SSH_KEY} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${SSH_USER}@${ip}"
|
||||
${SSH} 'sudo journalctl -u "spinifex-*" --no-pager -n 5000' > "${LOG_DIR}/${ip}-spinifex.log" 2>&1
|
||||
${SSH} 'ps auxw | grep -E "qemu-system|spinifex" | grep -v grep' > "${LOG_DIR}/${ip}-procs.txt" 2>&1
|
||||
${SSH} 'ls -la /var/lib/spinifex 2>/dev/null; ls -la /var/log/spinifex 2>/dev/null' > "${LOG_DIR}/${ip}-state.txt" 2>&1
|
||||
ok "$ip: $(wc -l < "${LOG_DIR}/${ip}-spinifex.log") journal lines, $(wc -l < "${LOG_DIR}/${ip}-procs.txt") procs"
|
||||
done
|
||||
|
||||
- name: Upload DDIL Results
|
||||
@@ -198,6 +220,10 @@ jobs:
|
||||
if: always()
|
||||
working-directory: mulga/scripts/tofu-cluster
|
||||
run: |
|
||||
tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
|
||||
tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
|
||||
./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
|
||||
set -euo pipefail
|
||||
# shellcheck source=/dev/null
|
||||
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
|
||||
quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
|
||||
quiet_run "tofu destroy" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
|
||||
quiet_run "scrub-env" "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
|
||||
ok "teardown complete"
|
||||
|
||||
@@ -0,0 +1,676 @@
|
||||
name: E2E (Go)
|
||||
|
||||
# Standalone workflow for the Go E2E harness (tests/e2e/{cert,lb,...}).
|
||||
# Mirrors e2e.yml's single-node provisioning chain but runs `go test` via
|
||||
# gotestsum so we get JUnit XML + per-test artifact directories surfaced to
|
||||
# GitHub Actions. Manual-dispatch only while the conversion lands — push
|
||||
# trigger lifts once the single-node Go port (mulga-siv-89) is green and
|
||||
# replaces run-e2e.sh; matches e2e-ddil.yml's Phase 1 trigger model.
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
sha:
|
||||
description: "PR head SHA to report status on"
|
||||
required: false
|
||||
suites:
|
||||
description: "Space-separated Make targets to run (e.g. 'e2e-single e2e-cert e2e-lb')"
|
||||
required: false
|
||||
default: "e2e-single e2e-cert e2e-lb"
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
statuses: write
|
||||
|
||||
# Pushes to the same ref stack on the single `ci-single` self-hosted runner,
|
||||
# and a cancelled run can leave orphan `tofu`/`go test -c` procs behind that
|
||||
# wedge the runner host. Auto-cancel keeps at most one in-flight run per ref.
|
||||
concurrency:
|
||||
group: e2e-go-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
GIT_BRANCH: ${{ github.ref_name }}
|
||||
SUITES: ${{ inputs.suites || 'e2e-single e2e-cert e2e-lb' }}
|
||||
|
||||
jobs:
|
||||
e2e_go_single:
|
||||
name: Single-Node E2E (Go)
|
||||
# Skip when the dispatch input requests no in-VM suites (e.g.
|
||||
# suites='e2e-reboot' only). Push triggers and the default SUITES list
|
||||
# both include single/cert/lb so the inputs.suites null-check keeps the
|
||||
# job running on plain PR pushes.
|
||||
if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-single') || contains(github.event.inputs.suites, 'e2e-cert') || contains(github.event.inputs.suites, 'e2e-lb')
|
||||
runs-on: [ self-hosted, ci-single ]
|
||||
timeout-minutes: 45
|
||||
env:
|
||||
TOFU_LOG: /tmp/e2e-go-tofu-single-${{ github.run_id }}.log
|
||||
ARTIFACT_DIR: /tmp/spinifex-e2e-artifacts-${{ github.run_id }}
|
||||
steps:
|
||||
- name: Reap Orphan Procs
|
||||
# A cancelled prior run can leave `tofu`/`terraform` mid-apply or a
|
||||
# libvirt `qemu+ssh` connection orphaned. Kill only procs whose cwd is
|
||||
# under THIS runner's workspace — every self-hosted runner on this host
|
||||
# shares the `tf-user` UID, so a UID-wide pkill would SIGKILL sibling
|
||||
# workflows (e2e.yml/e2e-ddil.yml/...) mid-apply. SIGTERM first to let
|
||||
# tofu release its state lock, then SIGKILL.
|
||||
run: |
|
||||
set +e
|
||||
UID_SELF=$(id -u)
|
||||
WS="${GITHUB_WORKSPACE}"
|
||||
reap() {
|
||||
local sig="$1"
|
||||
for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
|
||||
cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
|
||||
case "$cwd" in
|
||||
"$WS"|"$WS"/*) echo "reap $sig pid=$pid cwd=$cwd"; kill "-$sig" "$pid" 2>/dev/null ;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
reap TERM
|
||||
sleep 3
|
||||
reap KILL
|
||||
# Drop only this workspace's stale tofu state locks from a SIGKILL'd apply.
|
||||
find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
|
||||
true
|
||||
|
||||
- name: Clean Workspace
|
||||
run: sudo rm -rf "$GITHUB_WORKSPACE"/*
|
||||
|
||||
- uses: actions/create-github-app-token@v3
|
||||
id: app-token
|
||||
with:
|
||||
client-id: ${{ secrets.APP_ID }}
|
||||
private-key: ${{ secrets.APP_PRIVATE_KEY }}
|
||||
repositories: mulga
|
||||
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
repository: mulgadc/mulga
|
||||
token: ${{ steps.app-token.outputs.token }}
|
||||
sparse-checkout: scripts/tofu-cluster
|
||||
path: mulga
|
||||
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
path: spinifex
|
||||
|
||||
- uses: actions/setup-go@v6
|
||||
with:
|
||||
go-version-file: spinifex/go.mod
|
||||
# Self-hosted runner; the cache upload to GitHub blob storage
|
||||
# dominates total job time and the local Go module cache already
|
||||
# warms across runs.
|
||||
cache: false
|
||||
|
||||
- name: Re-enable Go toolchain auto-download
|
||||
# setup-go@v6 exports GOTOOLCHAIN=local to $GITHUB_ENV (v5 left it
|
||||
# "auto"). build-install-artifacts.sh prepends /usr/local/go/bin to
|
||||
# PATH on the self-hosted runner, and that Go is older than
|
||||
# spinifex/go.mod's `go` directive; under "local" the build fails
|
||||
# with `requires go >= X.Y.Z`. Restoring "auto" lets the resolved
|
||||
# Go self-heal by fetching the named minor on demand.
|
||||
run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
|
||||
|
||||
- name: Build Install Artifacts
|
||||
working-directory: mulga/scripts/tofu-cluster
|
||||
run: |
|
||||
./build-install-artifacts.sh "${GIT_BRANCH}" \
|
||||
"${RUNNER_TEMP}/spinifex-distro.tar.gz" \
|
||||
"${RUNNER_TEMP}/setup.sh" \
|
||||
"${RUNNER_TEMP}/spinifex-tests.tar"
|
||||
|
||||
- name: Provision VM
|
||||
working-directory: mulga/scripts/tofu-cluster
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# shellcheck source=/dev/null
|
||||
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
|
||||
export TF_VAR_git_branch="${GIT_BRANCH}"
|
||||
quiet_run "tofu init" "$TOFU_LOG" tofu init -input=false
|
||||
quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
|
||||
quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
|
||||
quiet_run "scrub-env (pre-clean)" "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
|
||||
quiet_run "tofu apply" "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
|
||||
ok "VM provisioned"
|
||||
|
||||
- name: Bootstrap via Binary Install
|
||||
working-directory: mulga/scripts/tofu-cluster
|
||||
run: |
|
||||
SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
|
||||
./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
|
||||
"${RUNNER_TEMP}/setup.sh" \
|
||||
"${RUNNER_TEMP}/spinifex-distro.tar.gz" \
|
||||
--test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
|
||||
|
||||
- name: Resolve VM SSH details
|
||||
id: vm
|
||||
working-directory: mulga/scripts/tofu-cluster
|
||||
run: |
|
||||
INVENTORY=$(tofu output -json inventory)
|
||||
WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
|
||||
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
|
||||
SSH_KEY="${SSH_KEY/#\~/$HOME}"
|
||||
echo "wan_ip=${WAN_IP}" >> "$GITHUB_OUTPUT"
|
||||
echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Build E2E test binaries on runner
|
||||
working-directory: spinifex/tests/e2e
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# shellcheck source=/dev/null
|
||||
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
|
||||
# SUITES is in make-target form (e.g. "e2e-cert e2e-lb"); strip the
|
||||
# "e2e-" prefix to get package directory names for `go test -c`.
|
||||
# 'reboot' is filtered out here — that suite runs runner-resident
|
||||
# via the e2e_go_reboot job, not in-VM, so this build step doesn't
|
||||
# need its binary.
|
||||
SUITES_BUILD=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -v '^reboot$' | tr '\n' ' ')
|
||||
if [ -z "$(echo "${SUITES_BUILD}" | tr -d ' ')" ]; then
|
||||
echo "no in-VM suites requested (SUITES=${SUITES}); nothing to build"
|
||||
exit 0
|
||||
fi
|
||||
banner "build ${SUITES_BUILD// /, }"
|
||||
group_open "go test -c"
|
||||
make e2e-build SUITES_BUILD="${SUITES_BUILD}"
|
||||
group_close
|
||||
for f in _bin/*.test; do
|
||||
ok "$(basename "$f") $(stat -c '%s' "$f" | numfmt --to=iec)"
|
||||
done
|
||||
|
||||
- name: Install go-junit-report on runner
|
||||
run: |
|
||||
# Converts `go test -v` text output (from the pre-built binaries on
|
||||
# the VM) into the junit-*.xml that "Publish test summary" expects.
|
||||
go install github.com/jstemmer/go-junit-report/v2@latest
|
||||
echo "$HOME/go/bin" >> "$GITHUB_PATH"
|
||||
|
||||
- name: Ship test binaries to VM
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# shellcheck source=/dev/null
|
||||
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
|
||||
SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
|
||||
VM=tf-user@${{ steps.vm.outputs.wan_ip }}
|
||||
ssh "${SSH_OPTS[@]}" "$VM" "rm -rf \$HOME/spinifex-e2e-bin && mkdir -p \$HOME/spinifex-e2e-bin"
|
||||
tar -C spinifex/tests/e2e/_bin -cf - . | ssh "${SSH_OPTS[@]}" "$VM" "tar -xf - -C \$HOME/spinifex-e2e-bin"
|
||||
ok "shipped binaries to $VM"
|
||||
|
||||
- name: Run Go E2E suites
|
||||
id: e2e
|
||||
run: |
|
||||
set +e
|
||||
# SUITES are make-target names ("e2e-cert" → cert.test). Each
|
||||
# pre-built binary is self-contained (testdata //go:embed'd) so the
|
||||
# VM needs no go toolchain, no source tree, no gotestsum. Each
|
||||
# suite gets its own ::group:: + cyan banner inside so the Actions
|
||||
# UI collapses suite output by default; expand only what failed.
|
||||
# 'reboot' is filtered: runs via e2e_go_reboot on the runner, not here.
|
||||
SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g' | tr ' ' '\n' | grep -v '^reboot$' | tr '\n' ' ')
|
||||
if [ -z "$(echo "${SUITES_RUN}" | tr -d ' ')" ]; then
|
||||
echo "no in-VM suites requested (SUITES=${SUITES}); skipping"
|
||||
echo "rc=0" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
VM_HOST="tf-user@${{ steps.vm.outputs.wan_ip }}"
|
||||
SSH_KEY="${{ steps.vm.outputs.ssh_key }}"
|
||||
# On cancel / runner SIGTERM / step timeout, force-kill any test
|
||||
# binaries left running on the VM. Without this, the remote `.test`
|
||||
# process keeps holding the suite's NATS connections + tmp dirs and
|
||||
# the subsequent Teardown's tofu-destroy races with its file
|
||||
# handles. -o ConnectTimeout=5 so a wedged VM doesn't block cleanup.
|
||||
cleanup_remote() {
|
||||
ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" "$VM_HOST" \
|
||||
'pkill -u tf-user -TERM -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; true' \
|
||||
2>/dev/null || true
|
||||
}
|
||||
trap cleanup_remote EXIT INT TERM
|
||||
# `ssh -tt` forces PTY allocation. When the runner cancels this
|
||||
# step, the local ssh dies and the remote sshd sends SIGHUP through
|
||||
# the PTY to its children — without -tt the remote `.test` would
|
||||
# orphan and keep running until its own 30m timeout.
|
||||
ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" \
|
||||
"$VM_HOST" \
|
||||
"set -u; mkdir -p '${ARTIFACT_DIR}'; \
|
||||
cd \$HOME/spinifex-e2e-bin; \
|
||||
R=\$'\033[0m' B=\$'\033[1m' C=\$'\033[36m' G=\$'\033[32m' RED=\$'\033[31m'; \
|
||||
FAIL=0; \
|
||||
for SUITE in ${SUITES_RUN}; do \
|
||||
echo \"::group::suite \$SUITE\"; \
|
||||
printf '\n%s%s━━ suite %s ━━%s\n' \"\$B\" \"\$C\" \"\$SUITE\" \"\$R\"; \
|
||||
# Record suite wall-clock start so the analyzer can locate
|
||||
# the right journal window. junit-report's timestamp attr
|
||||
# marks conversion time, not test start, so we cannot use it.
|
||||
date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
|
||||
GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
|
||||
ARTIFACT_DIR='${ARTIFACT_DIR}' \
|
||||
./\$SUITE.test -test.v -test.timeout=30m \
|
||||
2>&1 | tee '${ARTIFACT_DIR}/test-'\$SUITE'.log' \
|
||||
| sed -uE 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'; \
|
||||
rc=\${PIPESTATUS[0]}; \
|
||||
echo \"::endgroup::\"; \
|
||||
if [ \$rc -ne 0 ]; then \
|
||||
printf '%s✗%s suite %s failed (exit %d)\n' \"\$RED\" \"\$R\" \"\$SUITE\" \"\$rc\"; \
|
||||
echo \"::error::suite \$SUITE failed (exit \$rc)\"; \
|
||||
echo \"\$SUITE: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; \
|
||||
FAIL=1; \
|
||||
else \
|
||||
printf '%s✓%s suite %s passed\n' \"\$G\" \"\$R\" \"\$SUITE\"; \
|
||||
fi; \
|
||||
done; \
|
||||
exit \$FAIL"
|
||||
E2E_RC=$?
|
||||
echo "rc=${E2E_RC}" >> "$GITHUB_OUTPUT"
|
||||
exit 0 # don't fail the step — artifact upload + final gate happen below
|
||||
|
||||
- name: Collect artifacts from VM
|
||||
if: always()
|
||||
run: |
|
||||
set +e
|
||||
# shellcheck source=/dev/null
|
||||
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
|
||||
mkdir -p "${ARTIFACT_DIR}"
|
||||
SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
|
||||
VM=tf-user@${{ steps.vm.outputs.wan_ip }}
|
||||
# `|| true` so a missing dir on an early failure doesn't mask the
|
||||
# original error.
|
||||
ssh "${SSH_OPTS[@]}" "$VM" "tar -C '${ARTIFACT_DIR}' -cf - . 2>/dev/null || true" \
|
||||
| tar -C "${ARTIFACT_DIR}" -xf - 2>/dev/null
|
||||
ok "artifacts pulled to ${ARTIFACT_DIR}"
|
||||
# `-o short-iso` writes RFC3339-ish timestamps (e.g.
|
||||
# `2026-05-19T12:32:34+1000 host …`) so the analyzer can window-slice
|
||||
# the journal around each failure's wall-clock span.
|
||||
ssh "${SSH_OPTS[@]}" "$VM" 'sudo journalctl -u "spinifex-*" --no-pager --output=short-iso --since "60 min ago"' \
|
||||
> "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null
|
||||
ok "journal: $(wc -l < "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null || echo 0) lines"
|
||||
|
||||
- name: Convert test logs to JUnit XML
|
||||
if: always()
|
||||
run: |
|
||||
# VM runs the pre-built test binaries with `-test.v` and tees plain
|
||||
# text logs (no Go toolchain on the VM to emit test2json). Convert
|
||||
# each log to the junit-<suite>.xml shape "Publish test summary"
|
||||
# already parses.
|
||||
shopt -s nullglob
|
||||
for LOG in "${ARTIFACT_DIR}"/test-*.log; do
|
||||
BASE=$(basename "$LOG" .log)
|
||||
SUITE=${BASE#test-}
|
||||
go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
|
||||
done
|
||||
|
||||
- name: Publish test summary
|
||||
if: always()
|
||||
uses: ./spinifex/.github/actions/junit-report
|
||||
with:
|
||||
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
|
||||
title: "Go E2E results"
|
||||
|
||||
- name: Analyze e2e failures
|
||||
if: always()
|
||||
uses: ./spinifex/.github/actions/e2e-analyze
|
||||
with:
|
||||
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
|
||||
log-dir: ${{ env.ARTIFACT_DIR }}
|
||||
title: "E2E failure analysis"
|
||||
|
||||
- uses: actions/upload-artifact@v7
|
||||
if: always()
|
||||
with:
|
||||
name: e2e-go-artifacts-${{ github.run_id }}
|
||||
path: ${{ env.ARTIFACT_DIR }}
|
||||
if-no-files-found: warn
|
||||
retention-days: 14
|
||||
|
||||
- name: Teardown
|
||||
if: always()
|
||||
working-directory: mulga/scripts/tofu-cluster
|
||||
# `set -e` off — teardown must run scrub-env even if tofu destroy
|
||||
# fails; scrub is the libvirt-level fallback when tofu state is
|
||||
# inconsistent with the remote Proxmox node.
|
||||
run: |
|
||||
set +e
|
||||
# shellcheck source=/dev/null
|
||||
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
|
||||
# Kill any test binaries still running on the VM (cancelled mid-suite
|
||||
# paths) BEFORE tofu destroy yanks the qcow2 from underneath them.
|
||||
# Scoped to this run's VM by wan_ip — safe with sibling agents on
|
||||
# the same host. ConnectTimeout=5 so an already-wedged VM doesn't
|
||||
# block teardown.
|
||||
if [ -n "${{ steps.vm.outputs.wan_ip }}" ] && [ -n "${{ steps.vm.outputs.ssh_key }}" ]; then
|
||||
ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
|
||||
-i "${{ steps.vm.outputs.ssh_key }}" \
|
||||
tf-user@${{ steps.vm.outputs.wan_ip }} \
|
||||
'pkill -u tf-user -TERM -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; true' \
|
||||
2>/dev/null || true
|
||||
fi
|
||||
# Cancelled jobs from this point on can orphan an in-flight
|
||||
# tofu/libvirt SSH; clear them before destroy so the next run's
|
||||
# pre-clean isn't doing it. Scoped to this runner's workspace via
|
||||
# /proc/$pid/cwd — sibling runners share the tf-user UID, so a
|
||||
# UID-wide pkill would nuke their tofu mid-apply.
|
||||
UID_SELF=$(id -u)
|
||||
WS="${GITHUB_WORKSPACE}"
|
||||
reap() {
|
||||
local sig="$1"
|
||||
for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
|
||||
cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
|
||||
case "$cwd" in
|
||||
"$WS"|"$WS"/*) kill "-$sig" "$pid" 2>/dev/null ;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
reap TERM
|
||||
sleep 2
|
||||
reap KILL
|
||||
find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
|
||||
quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
|
||||
quiet_run "tofu destroy" "$TOFU_LOG" timeout 600 tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
|
||||
quiet_run "scrub-env" "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
|
||||
ok "teardown complete"
|
||||
|
||||
- name: Fail job if any suite failed
|
||||
if: always()
|
||||
run: |
|
||||
if [ "${{ steps.e2e.outputs.rc }}" != "0" ] || [ -s "${ARTIFACT_DIR}/suite-status.txt" ]; then
|
||||
echo "::error::Go E2E suites failed (rc=${{ steps.e2e.outputs.rc }})"
|
||||
cat "${ARTIFACT_DIR}/suite-status.txt" 2>/dev/null || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
e2e_go_reboot:
|
||||
name: Reboot Resilience E2E (Go)
|
||||
# Opt-in: only fires when the SUITES dispatch input lists e2e-reboot.
|
||||
# The reboot test reboots the spinifex VM, so it must run on the runner
|
||||
# (not shipped to the VM like the cert/lb/single suites). Runs in
|
||||
# parallel with e2e_go_single — both target [self-hosted, ci-single]
|
||||
# and GitHub Actions serialises on the underlying runner allocation,
|
||||
# so no explicit needs: ordering is required.
|
||||
if: contains(github.event.inputs.suites, 'e2e-reboot')
|
||||
runs-on: [ self-hosted, ci-single ]
|
||||
timeout-minutes: 60
|
||||
env:
|
||||
TOFU_LOG: /tmp/e2e-go-tofu-reboot-${{ github.run_id }}.log
|
||||
ARTIFACT_DIR: /tmp/spinifex-e2e-reboot-artifacts-${{ github.run_id }}
|
||||
steps:
|
||||
- name: Reap Orphan Procs
|
||||
run: |
|
||||
set +e
|
||||
UID_SELF=$(id -u)
|
||||
WS="${GITHUB_WORKSPACE}"
|
||||
reap() {
|
||||
local sig="$1"
|
||||
for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
|
||||
cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
|
||||
case "$cwd" in
|
||||
"$WS"|"$WS"/*) echo "reap $sig pid=$pid cwd=$cwd"; kill "-$sig" "$pid" 2>/dev/null ;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
reap TERM
|
||||
sleep 3
|
||||
reap KILL
|
||||
find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
|
||||
true
|
||||
|
||||
- name: Clean Workspace
|
||||
run: sudo rm -rf "$GITHUB_WORKSPACE"/*
|
||||
|
||||
- uses: actions/create-github-app-token@v3
|
||||
id: app-token
|
||||
with:
|
||||
client-id: ${{ secrets.APP_ID }}
|
||||
private-key: ${{ secrets.APP_PRIVATE_KEY }}
|
||||
repositories: mulga
|
||||
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
repository: mulgadc/mulga
|
||||
token: ${{ steps.app-token.outputs.token }}
|
||||
sparse-checkout: scripts/tofu-cluster
|
||||
path: mulga
|
||||
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
path: spinifex
|
||||
|
||||
- uses: actions/setup-go@v6
|
||||
with:
|
||||
go-version-file: spinifex/go.mod
|
||||
cache: false
|
||||
|
||||
- name: Re-enable Go toolchain auto-download
|
||||
run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
|
||||
|
||||
- name: Build Install Artifacts
|
||||
working-directory: mulga/scripts/tofu-cluster
|
||||
run: |
|
||||
./build-install-artifacts.sh "${GIT_BRANCH}" \
|
||||
"${RUNNER_TEMP}/spinifex-distro.tar.gz" \
|
||||
"${RUNNER_TEMP}/setup.sh" \
|
||||
"${RUNNER_TEMP}/spinifex-tests.tar"
|
||||
|
||||
- name: Provision VM
|
||||
working-directory: mulga/scripts/tofu-cluster
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# shellcheck source=/dev/null
|
||||
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
|
||||
export TF_VAR_git_branch="${GIT_BRANCH}"
|
||||
quiet_run "tofu init" "$TOFU_LOG" tofu init -input=false
|
||||
quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
|
||||
quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
|
||||
quiet_run "scrub-env (pre-clean)" "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
|
||||
quiet_run "tofu apply" "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
|
||||
ok "VM provisioned"
|
||||
|
||||
- name: Bootstrap via Binary Install
|
||||
working-directory: mulga/scripts/tofu-cluster
|
||||
run: |
|
||||
SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
|
||||
./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
|
||||
"${RUNNER_TEMP}/setup.sh" \
|
||||
"${RUNNER_TEMP}/spinifex-distro.tar.gz" \
|
||||
--test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
|
||||
|
||||
- name: Resolve VM SSH details
|
||||
id: vm
|
||||
working-directory: mulga/scripts/tofu-cluster
|
||||
run: |
|
||||
INVENTORY=$(tofu output -json inventory)
|
||||
WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
|
||||
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
|
||||
SSH_KEY="${SSH_KEY/#\~/$HOME}"
|
||||
echo "wan_ip=${WAN_IP}" >> "$GITHUB_OUTPUT"
|
||||
echo "ssh_key=${SSH_KEY}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Build reboot.test on runner
|
||||
working-directory: spinifex/tests/e2e
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# shellcheck source=/dev/null
|
||||
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
|
||||
banner "build reboot"
|
||||
group_open "go test -c"
|
||||
make e2e-build SUITES_BUILD="reboot"
|
||||
group_close
|
||||
ok "$(basename _bin/reboot.test) $(stat -c '%s' _bin/reboot.test | numfmt --to=iec)"
|
||||
|
||||
- name: Fetch AWS credentials from VM
|
||||
id: creds
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# The spinifex CLI on the VM (via `spx admin init` during bootstrap)
|
||||
# writes the [spinifex] profile to tf-user's ~/.aws/credentials.
|
||||
# Pull it over SSH and expose as SPINIFEX_AWS_* env vars (the same
|
||||
# static-cred path used by harness.NewAWSClient). Mirrors the cred
|
||||
# hop already proven by e2e-ddil.yml.
|
||||
SSH_OPTS=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
|
||||
-i "${{ steps.vm.outputs.ssh_key }}")
|
||||
VM=tf-user@${{ steps.vm.outputs.wan_ip }}
|
||||
CREDS=$(ssh "${SSH_OPTS[@]}" "$VM" 'cat ~/.aws/credentials')
|
||||
AWS_AK=$(echo "$CREDS" | awk -F '= *' '/^aws_access_key_id/ {print $2; exit}')
|
||||
AWS_SK=$(echo "$CREDS" | awk -F '= *' '/^aws_secret_access_key/ {print $2; exit}')
|
||||
if [ -z "$AWS_AK" ] || [ -z "$AWS_SK" ]; then
|
||||
echo "::error::Failed to extract spinifex AWS credentials from ${{ steps.vm.outputs.wan_ip }}"
|
||||
exit 1
|
||||
fi
|
||||
echo "::add-mask::${AWS_AK}"
|
||||
echo "::add-mask::${AWS_SK}"
|
||||
echo "ak=${AWS_AK}" >> "$GITHUB_OUTPUT"
|
||||
echo "sk=${AWS_SK}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Run reboot suite on runner
|
||||
id: e2e
|
||||
run: |
|
||||
set +e
|
||||
mkdir -p "${ARTIFACT_DIR}"
|
||||
# Banner + capture wall-clock start so the e2e-analyze action can
|
||||
# window-slice the journal around the test span (matches the
|
||||
# convention in e2e_go_single).
|
||||
date -u +%Y-%m-%dT%H:%M:%SZ > "${ARTIFACT_DIR}/test-reboot.start"
|
||||
R=$'\033[0m'; B=$'\033[1m'; C=$'\033[36m'
|
||||
printf '\n%s%s━━ suite reboot ━━%s\n' "$B" "$C" "$R"
|
||||
|
||||
GITHUB_ACTIONS=true FORCE_COLOR=1 \
|
||||
SPINIFEX_E2E=1 SPINIFEX_MODE=single \
|
||||
SPINIFEX_WAN_IP="${{ steps.vm.outputs.wan_ip }}" \
|
||||
SPINIFEX_SSH_KEY="${{ steps.vm.outputs.ssh_key }}" \
|
||||
SPINIFEX_AWS_ENDPOINT="https://${{ steps.vm.outputs.wan_ip }}:9999" \
|
||||
SPINIFEX_AWS_ACCESS_KEY_ID="${{ steps.creds.outputs.ak }}" \
|
||||
SPINIFEX_AWS_SECRET_ACCESS_KEY="${{ steps.creds.outputs.sk }}" \
|
||||
ARTIFACT_DIR="${ARTIFACT_DIR}" \
|
||||
./spinifex/tests/e2e/_bin/reboot.test -test.v -test.timeout=30m \
|
||||
2>&1 | tee "${ARTIFACT_DIR}/test-reboot.log" \
|
||||
| sed -uE 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'
|
||||
rc=${PIPESTATUS[0]}
|
||||
echo "rc=${rc}" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
|
||||
- name: Collect VM journals
|
||||
if: always()
|
||||
run: |
|
||||
set +e
|
||||
# shellcheck source=/dev/null
|
||||
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
|
||||
mkdir -p "${ARTIFACT_DIR}"
|
||||
SSH_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=5 \
|
||||
-i "${{ steps.vm.outputs.ssh_key }}")
|
||||
VM=tf-user@${{ steps.vm.outputs.wan_ip }}
|
||||
ssh "${SSH_OPTS[@]}" "$VM" 'sudo journalctl -u "spinifex-*" --no-pager --output=short-iso --since "60 min ago"' \
|
||||
> "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null || true
|
||||
ok "journal: $(wc -l < "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null || echo 0) lines"
|
||||
|
||||
- name: Install go-junit-report
|
||||
if: always()
|
||||
run: |
|
||||
go install github.com/jstemmer/go-junit-report/v2@latest
|
||||
echo "$HOME/go/bin" >> "$GITHUB_PATH"
|
||||
|
||||
- name: Convert test log to JUnit XML
|
||||
if: always()
|
||||
run: |
|
||||
shopt -s nullglob
|
||||
for LOG in "${ARTIFACT_DIR}"/test-*.log; do
|
||||
BASE=$(basename "$LOG" .log)
|
||||
SUITE=${BASE#test-}
|
||||
go-junit-report -in "$LOG" -out "${ARTIFACT_DIR}/junit-${SUITE}.xml" || true
|
||||
done
|
||||
|
||||
- name: Publish test summary
|
||||
if: always()
|
||||
uses: ./spinifex/.github/actions/junit-report
|
||||
with:
|
||||
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
|
||||
title: "Go E2E reboot results"
|
||||
|
||||
- name: Analyze e2e failures
|
||||
if: always()
|
||||
uses: ./spinifex/.github/actions/e2e-analyze
|
||||
with:
|
||||
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
|
||||
log-dir: ${{ env.ARTIFACT_DIR }}
|
||||
title: "Reboot E2E failure analysis"
|
||||
|
||||
- uses: actions/upload-artifact@v7
|
||||
if: always()
|
||||
with:
|
||||
name: e2e-go-reboot-artifacts-${{ github.run_id }}
|
||||
path: ${{ env.ARTIFACT_DIR }}
|
||||
if-no-files-found: warn
|
||||
retention-days: 14
|
||||
|
||||
- name: Teardown
|
||||
if: always()
|
||||
working-directory: mulga/scripts/tofu-cluster
|
||||
run: |
|
||||
set +e
|
||||
# shellcheck source=/dev/null
|
||||
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
|
||||
UID_SELF=$(id -u)
|
||||
WS="${GITHUB_WORKSPACE}"
|
||||
reap() {
|
||||
local sig="$1"
|
||||
for pid in $(pgrep -u "$UID_SELF" -x tofu) $(pgrep -u "$UID_SELF" -x terraform); do
|
||||
cwd=$(readlink "/proc/$pid/cwd" 2>/dev/null) || continue
|
||||
case "$cwd" in
|
||||
"$WS"|"$WS"/*) kill "-$sig" "$pid" 2>/dev/null ;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
reap TERM
|
||||
sleep 2
|
||||
reap KILL
|
||||
find "$WS" -name '.terraform.tfstate.lock.info' -delete 2>/dev/null
|
||||
quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
|
||||
quiet_run "tofu destroy" "$TOFU_LOG" timeout 600 tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
|
||||
quiet_run "scrub-env" "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}"
|
||||
ok "teardown complete"
|
||||
|
||||
- name: Fail job if reboot suite failed
|
||||
if: always()
|
||||
run: |
|
||||
if [ "${{ steps.e2e.outputs.rc }}" != "0" ]; then
|
||||
echo "::error::Reboot suite failed (rc=${{ steps.e2e.outputs.rc }})"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
e2e_go_status:
|
||||
name: Report E2E (Go) Status
|
||||
# Always post status. When inputs.sha is unset (manual dispatch without
|
||||
# a PR head), fall back to github.sha — the resolved commit SHA of the
|
||||
# dispatched ref, always present. The previous `&& inputs.sha` gate
|
||||
# plus `statuses/${{ inputs.sha }}` produced an 'invalid hash' error
|
||||
# whenever the workflow was dispatched without the optional input
|
||||
# (tests passed, status step silently failed). Refs mulga-siv-112.
|
||||
if: always()
|
||||
needs: [ e2e_go_single, e2e_go_reboot ]
|
||||
runs-on: [ self-hosted, ci-single ]
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- name: Report Status to PR
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
REPORT_SHA: ${{ inputs.sha || github.sha }}
|
||||
run: |
|
||||
if [[ -z "${REPORT_SHA}" ]]; then
|
||||
echo "::error::no SHA available (inputs.sha and github.sha both empty)"
|
||||
exit 1
|
||||
fi
|
||||
# Treat skipped as success — e2e_go_reboot only runs when opted in
|
||||
# via the SUITES input, so a non-reboot dispatch should still post
|
||||
# a green status from the single job alone.
|
||||
single_ok=true; reboot_ok=true
|
||||
case "${{ needs.e2e_go_single.result }}" in success|skipped) ;; *) single_ok=false ;; esac
|
||||
case "${{ needs.e2e_go_reboot.result }}" in success|skipped) ;; *) reboot_ok=false ;; esac
|
||||
if $single_ok && $reboot_ok; then
|
||||
STATE="success"
|
||||
DESC="Go E2E suites passed"
|
||||
else
|
||||
STATE="failure"
|
||||
DESC="Go E2E failed: single=${{ needs.e2e_go_single.result }} reboot=${{ needs.e2e_go_reboot.result }}"
|
||||
fi
|
||||
gh api "repos/${{ github.repository }}/statuses/${REPORT_SHA}" \
|
||||
-f state="$STATE" \
|
||||
-f context=e2e-go \
|
||||
-f description="$DESC" \
|
||||
-f target_url="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
|
||||
@@ -61,6 +61,9 @@ jobs:
|
||||
- name: Run Tests with Coverage
|
||||
run: make -C spinifex test-cover
|
||||
|
||||
- name: Run Action Tests
|
||||
run: make -C spinifex test-actions
|
||||
|
||||
- name: Check Diff Coverage
|
||||
run: cd spinifex && scripts/diff-coverage.sh coverage.out --base HEAD~1
|
||||
|
||||
|
||||
Reference in New Issue
Block a user