Josh Siviour
545ca94a8e
Render summed FailCount across suites and printed "No failures across any suite" whenever the total was zero, without asking whether any test had run. A cell that dies during provisioning emits junit with no testcases at all, so it took the clean path. Nightly run 30467638631 cell 20 failed with "install callback never fired after 3 attempts; PXE never triggered install" and its analysis.md read "No failures across any suite" — absence of evidence rendered as positive evidence. Zero testcases now renders a warning naming the layer above the suites. A run with tests and no failures is unchanged.
e2e-analyze
Post-test failure-clustering action for the Go E2E suites.
What it does
Runs after go-junit-report has converted each suite's stdout into
junit-<suite>.xml. For every JUnit file matched by junit-glob:
- Parses each
<testcase>with a<failure>/<error>child. - Strips ANSI escapes, then normalises dynamic noise (UUIDs, EC2 IDs, IPs, timestamps, request-ids, source line numbers) to produce a stable signature per failure.
- Picks the earliest non-cascade failure per suite as the likely
root cause. "Cascade" = failure message matches a fixture-dependency
marker (
must populate fix.,Should NOT be empty,Expected value not to be nil). When every failure is a cascade (whole suite tripped on one upstream gap), the earliest cascade is the fallback root. - Buckets the remaining failures by signature and renders a markdown
report. The report is appended to
$GITHUB_STEP_SUMMARYand also written to<log-dir>/analysis.mdso it travels with the uploaded artifact bundle.
The action never fails the job — exit-code semantics stay with the existing "Fail job if any suite failed" gate.
Inputs
| Input | Required | Description |
|---|---|---|
junit-glob |
yes | Shell glob for the JUnit XML files (e.g. ${ARTIFACT_DIR}/junit-*.xml). |
log-dir |
yes | Directory the rendered report is written into (analysis.md). |
title |
no | Heading at the top of the summary section. Default E2E failure analysis. |
Outputs
None. The summary is appended to $GITHUB_STEP_SUMMARY; the same
content lands at <log-dir>/analysis.md.
Local invocation
Outside Actions, the analyser prints to stdout instead of appending to a summary file:
cd spinifex/.github/actions/e2e-analyze
go run . -junit-glob "/tmp/artifacts/junit-*.xml" -log-dir /tmp/artifacts
Tests
Golden-file tests live in testdata/. Each subdir holds a scenario:
| Scenario | Source |
|---|---|
cascade-vpcs |
Synthetic: one Phase 5 root + 15 cascades |
single-failure |
Real artifact from run 26037892507 |
no-failures |
Synthetic: clean run |
Run normally:
go test ./.github/actions/e2e-analyze/
When the report format changes intentionally, regenerate the snapshots:
go test ./.github/actions/e2e-analyze/ -update
Per-failure bundles (Stage 2)
After parsing, the action materialises a bundle directory per failure
under <log-dir>/analysis/NN-<TestName>/. The report links to it from
each entry so a reader on the Actions page can click straight through
to the relevant log slice.
Each bundle contains:
| File | Source |
|---|---|
failure.txt |
Synthesised summary: test, start, duration, file-site, error line. |
journal.log |
spinifex-journal.log sliced to [StartAt − 5s, StartAt + Duration + 5s]. |
test.log |
test-<suite>.log block between === RUN <name> and the matching --- FAIL. |
<other files> |
Anything the harness already dropped into <log-dir>/<TestName>/ at run-time. |
The journal slice relies on the workflow running
journalctl --output=short-iso so timestamps are parseable. Lines
without a parseable timestamp inherit the previous line's
in-window/out-of-window classification (so multi-line records and stack
traces stay contiguous).
Roadmap
- Stage 1 (shipped): first-failure surfacing + cascade clustering.
- Stage 2 (this update): time-window correlation — per-failure bundle with sliced journal + test-log + any per-test artifacts.
- Stage 3: rule-based root-cause classifier driven by
spinifex/tests/e2e/.failure-rules.yaml.
See docs/development/improvements/e2e-go-failure-analysis.md in the
parent mulga repo.