mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 09:01:31 +00:00
ci(e2e-go): cluster JUnit failures + name likely root cause (#230)
* ci(e2e-go): cluster JUnit failures + name likely root cause
New composite action `.github/actions/e2e-analyze` runs after
`go-junit-report` converts each suite log. For every `<failure>` it
strips ANSI, normalises dynamic noise (EC2 IDs, UUIDs, IPs, timestamps,
file:line) to a stable signature, picks the earliest non-cascade
failure per suite as the likely root cause, and buckets the rest by
signature. Output is appended to `$GITHUB_STEP_SUMMARY` and written to
`<log-dir>/analysis.md` so it travels with the artifact bundle.
Cascade markers (`must populate fix.`, `Should NOT be empty`,
`Expected value not to be nil`) collapse the typical "one upstream
phase fails, ten downstream `require.NotEmpty` assertions trip" pattern
into a single grouped entry under the real first failure.
The analyser is advisory — it always exits 0. The existing
"Fail job if any suite failed" gate keeps owning exit-code semantics.
Stage 1 of `docs/development/improvements/e2e-go-failure-analysis.md`
(mulga-siv-102). Stage 2 (per-failure log slicing) and Stage 3
(rule-based hints via `tests/e2e/.failure-rules.yaml`, placeholder
seeded here) are independent follow-ups.
CI wiring:
- e2e-go.yml: new "Analyze e2e failures" step between "Publish test
summary" and the artifact upload, always-runs.
- spinifex.yml: new `make test-actions` step so the golden-file tests
under `.github/actions/e2e-analyze/testdata/` run on every PR.
- Makefile: `test-actions` target; kept out of `test-cover` so CI-only
tooling doesn't dilute coverage %.
Golden fixtures: a synthetic cascade-vpcs case (one Phase 5
InvalidParameterValue root + 15 downstream cascades), one real
single-failure case captured from run 26037892507, and a clean
no-failures case.
* ci(e2e-go): per-failure log bundles with time-window slicing
Stage 2 of the e2e failure analyzer. For each failure surfaced by
Stage 1, materialise <log-dir>/analysis/NN-<TestName>/ containing:
failure.txt — summary (test, start, duration, file-site, error)
journal.log — spinifex-journal.log sliced to
[StartAt - 5s, StartAt + Duration + 5s]
test.log — test-<suite>.log block between `=== RUN <name>`
and the matching `--- FAIL` marker
<copies> — any files the harness already dropped at
<log-dir>/<TestName_with_/_replaced>/
Report links each entry to its bundle so a reader on the Actions
page goes straight to the relevant slice.
Workflow now invokes journalctl with --output=short-iso so the
analyzer can parse wall-clock timestamps for the journal window.
Lines without a parseable timestamp inherit the previous line's
classification (keeps multi-line records and stack traces
contiguous).
Render output is unchanged when BundlePath is empty, so the Stage 1
golden snapshots still match.
* ci(e2e-go): restore GOTOOLCHAIN=auto after setup-go v6 bump
setup-go@v6 flipped the GOTOOLCHAIN default from "auto" to "local",
which makes Go refuse to auto-download the toolchain version named
in go.mod. The self-hosted ci-single runner ships /usr/local/go
older than spinifex/go.mod's `go 1.26.3`, and build-install-artifacts.sh
prepends that path so the system Go wins the resolution race. Under
v5 ("auto"), Go silently fetched 1.26.3 on demand; under v6
("local"), the Build Install Artifacts step errors out:
go: spinifex/go.mod requires go >= 1.26.3 (running go 1.26.1; GOTOOLCHAIN=local)
Re-set the env to "auto" so behaviour matches pre-v6 runs.
* ci(e2e-go): re-export GOTOOLCHAIN=auto after setup-go
Workflow-level env doesn't help: setup-go@v6 writes GOTOOLCHAIN=local
to $GITHUB_ENV, which overrides workflow env in every later step.
Add a dedicated step right after setup-go that appends
GOTOOLCHAIN=auto to $GITHUB_ENV so the Build Install Artifacts step
(which uses /usr/local/go from a stale runner path) can self-heal by
fetching the toolchain named in go.mod, matching pre-v6 behaviour.
* fix(vpcd): add flows-ready barrier after dnat_and_snat add
handleAddNAT wrote the NAT rule to OVN NB and returned immediately,
mirroring the gap siv-105 closed for handleIGWAttach but on the NAT
add path. ec2d/RunInstances then returned success while ovn-northd
was still compiling NB->SB and ovn-controller hadn't installed the
DNAT/SNAT flows on the gateway chassis, so the freshly launched VM
was unreachable on its public IP for whatever flow-install latency
the chassis happened to incur.
Reproduced in CI run 26072432957 Phase8b_VPCSubnetE2E: VM at
192.168.0.212 timed out at 3min while diagnostics showed
Port_Binding up=true for both the VM eni and the chassis-redirect
gateway. SB-side convergence was complete; the missing piece was the
same `ovn-nbctl --wait=hv sync` barrier siv-105 added to IGW attach,
applied at the end of handleAddNAT as well.
Reuses the existing package-var `waitForFlowsHV` so tests can stub
it; no new infra. Parallel unit test asserts the call site fires
exactly once per add-nat.
* ci(e2e-analyze): real-artifact fixes for journal slicing
Three bugs surfaced after Stage 2 ran against real CI artifacts in
run 26072432957:
1. Regex `[+-]\d{4}` rejected the colon-offset form (e.g. `+10:00`)
that systemd's `journalctl --output=short-iso` actually emits on
most versions. Added a parser that tries RFC3339 first and falls
back to the no-colon `-0700` layout. Result: 0-line journal slices
become populated.
2. go-junit-report's `timestamp` attr on <testsuite> records the
moment the XML was produced (post-suite), not when the tests
started. Computing StartAt as `that + cumulative-priors` placed
every failure AFTER the journal collection ended, so windows
always missed. Workflow now writes `${ARTIFACT_DIR}/test-<suite>.start`
right before running each suite; analyzer prefers that file over
the junit timestamp.
3. Persisted `OffsetFromSuiteStart` on each Failure so the
suite-start override can recompute StartAt without re-parsing the
XML.
Unit tests added for the colon-offset journal format. End-to-end
test (TestWriteBundles_EndToEnd) still passes with the new layout.
* ci(e2e-analyze): skip parent testcases when summing suite offsets
go-junit-report emits one <testcase> for the parent Go test AND each
subtest, with the parent's time = sum of its children. ParseFile was
summing every testcase into the suite-cumulative-offset counter, so
every leaf's offset was inflated by ~the parent's total duration.
Visible on run 26073817490 as Phase8b's StartAt landing 13 minutes
after the suite finished collecting logs — beyond the journal window,
so the per-failure journal slice was empty.
Pre-compute the leaf set per suite (a name is a leaf iff no other
testcase name starts with `<name>/`) and only accumulate cumul on
leaves. Golden snapshots updated; the offset for the only failure in
each fixture now matches the actual leaf-sum start time.
* test(e2e-harness): capture IP-targeted datapath state on Phase8b SSH fail
Phase 1 diagnostics for mulga-siv-111. The Phase8b SSH timeout
reproduced on the e2e-failure-analysis branch with siv-105's IGW
barrier AND the new add-nat barrier BOTH firing cleanly, so the
residual gap is downstream of `--wait=hv sync` — most likely stale
OVS conntrack, stale upstream-host ARP, or an OF flow-install miss.
DumpVPCFlowDiagnostics now takes a VPCDiagnosticsOpts carrying the
external IP, logical IP, and per-test artifact dir. When set, the
helper greps `ovs-ofctl dump-flows br-int`, `conntrack -L`, and
`ip -4 neigh show` for the external IP / logical IP and snapshots
`ovn-nbctl find NAT external_ip=…`. Each capture is both printed
inline (existing affordance) and written to a separate file under
fix.Artifacts so the Stage 2 analyzer bundler picks it up alongside
the journal + test-log slices, no analyzer changes needed.
Phase8b's caller now passes pubIP / privIP / fix.Artifacts so the
captures fire on the next failure; updates the t.Fatalf message to
point at siv-111 instead of siv-105 (which is the now-closed parent).
* fix(e2e-single): authorize SSH ingress on fresh-VPC default SG in Phase8b
Phase 8b stands up a non-default VPC and runs an instance there. Its
auto-created default SG denies all ingress, so the SSH probe times out
even when every datapath barrier converges — OF flow captures show the
SYN reaches OVN, gets DNAT'd at table 13, then drops at the ACL between
tables 13 and 17. The bash driver relies on admin init having pre-baked
this rule on the *default* VPC's default SG; a fresh VPC has not.
Add a DescribeSecurityGroups + AuthorizeSSHIngress step between
route-table setup and run-instances, using the existing harness helper.
Fix surfaced via mulga-siv-111's Phase 1 diagnostics (OF flow + ACL
drop counts), invalidating the H1/H2/H3 stale-state hypotheses in the
bead plan doc.
This commit is contained in:
Josh Siviour
@@ -0,0 +1,104 @@
|
||||
# e2e-analyze
|
||||
|
||||
Post-test failure-clustering action for the Go E2E suites.
|
||||
|
||||
## What it does
|
||||
|
||||
Runs after `go-junit-report` has converted each suite's stdout into
|
||||
`junit-<suite>.xml`. For every JUnit file matched by `junit-glob`:
|
||||
|
||||
1. Parses each `<testcase>` with a `<failure>` / `<error>` child.
|
||||
2. Strips ANSI escapes, then normalises dynamic noise (UUIDs, EC2 IDs,
|
||||
IPs, timestamps, request-ids, source line numbers) to produce a stable
|
||||
_signature_ per failure.
|
||||
3. Picks the **earliest non-cascade** failure per suite as the likely
|
||||
root cause. "Cascade" = failure message matches a fixture-dependency
|
||||
marker (`must populate fix.`, `Should NOT be empty`,
|
||||
`Expected value not to be nil`). When every failure is a cascade
|
||||
(whole suite tripped on one upstream gap), the earliest cascade is
|
||||
the fallback root.
|
||||
4. Buckets the remaining failures by signature and renders a markdown
|
||||
report. The report is appended to `$GITHUB_STEP_SUMMARY` and also
|
||||
written to `<log-dir>/analysis.md` so it travels with the uploaded
|
||||
artifact bundle.
|
||||
|
||||
The action **never fails the job** — exit-code semantics stay with the
|
||||
existing "Fail job if any suite failed" gate.
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Required | Description |
|
||||
|--------------|----------|-------------|
|
||||
| `junit-glob` | yes | Shell glob for the JUnit XML files (e.g. `${ARTIFACT_DIR}/junit-*.xml`). |
|
||||
| `log-dir` | yes | Directory the rendered report is written into (`analysis.md`). |
|
||||
| `title` | no | Heading at the top of the summary section. Default `E2E failure analysis`. |
|
||||
|
||||
## Outputs
|
||||
|
||||
None. The summary is appended to `$GITHUB_STEP_SUMMARY`; the same
|
||||
content lands at `<log-dir>/analysis.md`.
|
||||
|
||||
## Local invocation
|
||||
|
||||
Outside Actions, the analyser prints to stdout instead of appending to a
|
||||
summary file:
|
||||
|
||||
```sh
|
||||
cd spinifex/.github/actions/e2e-analyze
|
||||
go run . -junit-glob "/tmp/artifacts/junit-*.xml" -log-dir /tmp/artifacts
|
||||
```
|
||||
|
||||
## Tests
|
||||
|
||||
Golden-file tests live in `testdata/`. Each subdir holds a scenario:
|
||||
|
||||
| Scenario | Source |
|
||||
|--------------------|-------------------------------------------|
|
||||
| `cascade-vpcs` | Synthetic: one Phase 5 root + 15 cascades |
|
||||
| `single-failure` | Real artifact from run `26037892507` |
|
||||
| `no-failures` | Synthetic: clean run |
|
||||
|
||||
Run normally:
|
||||
|
||||
```sh
|
||||
go test ./.github/actions/e2e-analyze/
|
||||
```
|
||||
|
||||
When the report format changes intentionally, regenerate the snapshots:
|
||||
|
||||
```sh
|
||||
go test ./.github/actions/e2e-analyze/ -update
|
||||
```
|
||||
|
||||
## Per-failure bundles (Stage 2)
|
||||
|
||||
After parsing, the action materialises a bundle directory per failure
|
||||
under `<log-dir>/analysis/NN-<TestName>/`. The report links to it from
|
||||
each entry so a reader on the Actions page can click straight through
|
||||
to the relevant log slice.
|
||||
|
||||
Each bundle contains:
|
||||
|
||||
| File | Source |
|
||||
|-------------------|-----------------------------------------------------------------------------------|
|
||||
| `failure.txt` | Synthesised summary: test, start, duration, file-site, error line. |
|
||||
| `journal.log` | `spinifex-journal.log` sliced to `[StartAt − 5s, StartAt + Duration + 5s]`. |
|
||||
| `test.log` | `test-<suite>.log` block between `=== RUN <name>` and the matching `--- FAIL`. |
|
||||
| `<other files>` | Anything the harness already dropped into `<log-dir>/<TestName>/` at run-time. |
|
||||
|
||||
The journal slice relies on the workflow running
|
||||
`journalctl --output=short-iso` so timestamps are parseable. Lines
|
||||
without a parseable timestamp inherit the previous line's
|
||||
in-window/out-of-window classification (so multi-line records and stack
|
||||
traces stay contiguous).
|
||||
|
||||
## Roadmap
|
||||
|
||||
- **Stage 1 (shipped):** first-failure surfacing + cascade clustering.
|
||||
- **Stage 2 (this update):** time-window correlation — per-failure bundle
|
||||
with sliced journal + test-log + any per-test artifacts.
|
||||
- **Stage 3:** rule-based root-cause classifier driven by
|
||||
`spinifex/tests/e2e/.failure-rules.yaml`.
|
||||
|
||||
See `docs/development/improvements/e2e-go-failure-analysis.md` in the
|
||||
parent mulga repo.
|
||||
@@ -0,0 +1,38 @@
|
||||
name: E2E Failure Analyzer
|
||||
description: >-
|
||||
Post-test analyzer for the Go E2E suites. Parses junit-*.xml emitted by
|
||||
go-junit-report, clusters failures by signature, picks the earliest
|
||||
non-cascade failure per suite as the likely root cause, and appends a
|
||||
human-readable report to $GITHUB_STEP_SUMMARY. Always exits 0 — the
|
||||
existing "Fail job if any suite failed" gate keeps owning exit-code
|
||||
semantics.
|
||||
|
||||
inputs:
|
||||
junit-glob:
|
||||
description: Shell glob matching JUnit XML files to analyze.
|
||||
required: true
|
||||
log-dir:
|
||||
description: >-
|
||||
Artifact directory. The rendered report is also written to
|
||||
$log-dir/analysis.md so it travels with the uploaded artifact.
|
||||
required: true
|
||||
title:
|
||||
description: Heading rendered at the top of the summary section.
|
||||
required: false
|
||||
default: "E2E failure analysis"
|
||||
|
||||
runs:
|
||||
using: composite
|
||||
steps:
|
||||
- id: analyze
|
||||
shell: bash
|
||||
working-directory: ${{ github.action_path }}
|
||||
env:
|
||||
ANALYZE_JUNIT_GLOB: ${{ inputs.junit-glob }}
|
||||
ANALYZE_LOG_DIR: ${{ inputs.log-dir }}
|
||||
ANALYZE_TITLE: ${{ inputs.title }}
|
||||
run: |
|
||||
go run . \
|
||||
-junit-glob "$ANALYZE_JUNIT_GLOB" \
|
||||
-log-dir "$ANALYZE_LOG_DIR" \
|
||||
-title "$ANALYZE_TITLE"
|
||||
@@ -0,0 +1,552 @@
|
||||
// Package main implements the e2e-analyze GitHub Action.
|
||||
//
|
||||
// Stage 1 of docs/development/improvements/e2e-go-failure-analysis.md:
|
||||
// cluster failures from go-junit-report's junit-*.xml by error signature,
|
||||
// pick the earliest non-cascade failure per suite as the likely root cause,
|
||||
// and render a human-readable report.
|
||||
//
|
||||
// This file contains the pure logic (parsing, signature extraction,
|
||||
// clustering, rendering). Side-effect handling lives in main.go.
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/xml"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// JUnit XML schema produced by go-junit-report v2. Stable across versions
|
||||
// we care about; only fields used downstream are decoded.
|
||||
type junitSuites struct {
|
||||
XMLName xml.Name `xml:"testsuites"`
|
||||
Suites []junitSuite `xml:"testsuite"`
|
||||
}
|
||||
|
||||
type junitSuite struct {
|
||||
Name string `xml:"name,attr"`
|
||||
Tests int `xml:"tests,attr"`
|
||||
Failures int `xml:"failures,attr"`
|
||||
Errors int `xml:"errors,attr"`
|
||||
Skipped int `xml:"skipped,attr"`
|
||||
Timestamp string `xml:"timestamp,attr"`
|
||||
Time float64 `xml:"time,attr"`
|
||||
Cases []junitTC `xml:"testcase"`
|
||||
}
|
||||
|
||||
type junitTC struct {
|
||||
Name string `xml:"name,attr"`
|
||||
Classname string `xml:"classname,attr"`
|
||||
Time float64 `xml:"time,attr"`
|
||||
Failure *junitFailure `xml:"failure"`
|
||||
Error *junitFailure `xml:"error"`
|
||||
Skipped *junitSkipped `xml:"skipped"`
|
||||
SystemOut string `xml:"system-out"`
|
||||
}
|
||||
|
||||
type junitFailure struct {
|
||||
Message string `xml:"message,attr"`
|
||||
Type string `xml:"type,attr"`
|
||||
Body string `xml:",chardata"`
|
||||
}
|
||||
|
||||
type junitSkipped struct {
|
||||
Message string `xml:"message,attr"`
|
||||
}
|
||||
|
||||
// Failure is the analyzer's normalized view of one failed testcase.
|
||||
type Failure struct {
|
||||
SuiteFile string // e.g. "single" (derived from junit-single.xml)
|
||||
Name string // e.g. "TestSingleNode/Phase5_LaunchInstance"
|
||||
Duration float64 // seconds
|
||||
Order int // position within the suite (0-based)
|
||||
StartAt time.Time // estimated wall-clock start: suite.timestamp + sum(prior durations)
|
||||
Error string // last meaningful error line, ANSI-stripped
|
||||
FileHint string // e.g. "vpc_test.go:227", parsed from failure body
|
||||
Cascade bool // signature matched a known downstream pattern
|
||||
Signature string // normalized error key for bucketing
|
||||
// OffsetFromSuiteStart is sum(prior testcase durations) at the time
|
||||
// this test started. Persisted separately from StartAt so a caller
|
||||
// that has a more authoritative suite wall-clock origin (e.g. a
|
||||
// `test-<suite>.start` file written by the workflow before the suite
|
||||
// runs) can recompute StartAt without re-parsing the XML.
|
||||
OffsetFromSuiteStart time.Duration
|
||||
// BundlePath is the relative path (from log-dir) to the per-failure
|
||||
// bundle written by Stage 2's WriteBundles. Empty when bundles were
|
||||
// not generated (e.g. unit tests that call Render directly).
|
||||
BundlePath string
|
||||
}
|
||||
|
||||
// SuiteReport summarises one junit-*.xml file.
|
||||
type SuiteReport struct {
|
||||
File string // basename, e.g. "junit-single.xml"
|
||||
Label string // short name, e.g. "single"
|
||||
Total int
|
||||
FailCount int
|
||||
Root *Failure // nil if no failures
|
||||
Cascades []Failure // everything else that failed in this suite
|
||||
Buckets [][]Failure // failures grouped by signature (root's bucket first)
|
||||
}
|
||||
|
||||
// Report is the full analysis across every junit file.
|
||||
type Report struct {
|
||||
Title string
|
||||
Suites []SuiteReport
|
||||
}
|
||||
|
||||
// ansiSeq matches ANSI CSI escapes the harness sprinkles through log
|
||||
// output (colours, bold). Three forms in the wild:
|
||||
// - real ESC: `\x1b[36m…`
|
||||
// - U+FFFD replacement (XML serialiser dropped the ESC byte): `�[36m…`
|
||||
// - bare CSI with the leading byte gone entirely: `[36m…`
|
||||
//
|
||||
// All three are stripped so colour state doesn't fork the signature bucket.
|
||||
var ansiSeq = regexp.MustCompile(
|
||||
"\x1b\\[[0-9;]*[a-zA-Z]" + // real ESC CSI
|
||||
"|�\\[[0-9;]*[a-zA-Z]" + // replacement-char CSI
|
||||
"|\\[[0-9]{1,2}(?:;[0-9]{1,2})*m", // bare colour code, "mode" form only
|
||||
)
|
||||
|
||||
// cascadeMarkers are message fragments that only appear in tests that
|
||||
// depend on a fixture field populated by an earlier phase. Their presence
|
||||
// means the failure is downstream of some other failure that happened
|
||||
// first; the report collapses them under that root cause.
|
||||
var cascadeMarkers = []string{
|
||||
"must populate fix.",
|
||||
"Should NOT be empty",
|
||||
"Expected value not to be nil",
|
||||
}
|
||||
|
||||
// noiseReplacers normalise dynamic identifiers so that "ten failures, same
|
||||
// bug" bucket together. Pattern ordering matters: longer/more-specific
|
||||
// patterns must precede the catch-alls.
|
||||
var noiseReplacers = []struct {
|
||||
re *regexp.Regexp
|
||||
repl string
|
||||
}{
|
||||
{regexp.MustCompile(`\b(vpc|subnet|sg|rtb|igw|i|ami|vol|snap|rtbassoc|key)-[0-9a-f]{8,}\b`), `${1}-<id>`},
|
||||
{regexp.MustCompile(`\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b`), `<uuid>`},
|
||||
{regexp.MustCompile(`\b[0-9a-f]{32,}\b`), `<hex>`},
|
||||
{regexp.MustCompile(`\b\d{1,3}(\.\d{1,3}){3}(:\d+)?\b`), `<ip>`},
|
||||
{regexp.MustCompile(`\b\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:?\d{2})?\b`), `<ts>`},
|
||||
{regexp.MustCompile(`request-id:\s*\S+`), `request-id: <id>`},
|
||||
{regexp.MustCompile(`(_test\.go):\d+`), `${1}:<line>`},
|
||||
{regexp.MustCompile(`\s+`), ` `},
|
||||
}
|
||||
|
||||
// fileHintRe pulls the last "foo_test.go:NNN" we see in a failure body so
|
||||
// the report can point a reader at the source line. Test files dominate the
|
||||
// hint set; harness/helper file matches go through goFileLineRe below.
|
||||
var fileHintRe = regexp.MustCompile(`([A-Za-z0-9_]+_test\.go:\d+)`)
|
||||
|
||||
// goFileLineRe matches any "foo.go:NNN:" prefix (test or non-test). Used to
|
||||
// surface harness/helper-level error text like
|
||||
// `ec2helpers.go:164: describe-vpcs: InvalidParameterValue …` which Go test
|
||||
// emits when a non-test file calls t.Errorf.
|
||||
var goFileLineRe = regexp.MustCompile(`^[A-Za-z0-9_]+\.go:\d+:\s*(.+)$`)
|
||||
|
||||
// testifyErrorPlaceholder is testify's "Error:" boilerplate when the
|
||||
// assertion failed because of a wrapped error. The next non-blank line in
|
||||
// the trace holds the actual error text.
|
||||
const testifyErrorPlaceholder = "Received unexpected error:"
|
||||
|
||||
// stripANSI removes CSI escapes; leaves everything else intact.
|
||||
func stripANSI(s string) string {
|
||||
return ansiSeq.ReplaceAllString(s, "")
|
||||
}
|
||||
|
||||
// extractErrorLine finds the most useful single-line summary in a failure
|
||||
// body. Priority order, highest first:
|
||||
// 1. Testify "Messages:" line — user-supplied context, the most signal-rich.
|
||||
// 2. Testify "Error:" content (with one-line lookahead when the placeholder
|
||||
// "Received unexpected error:" defers to the next line).
|
||||
// 3. Leading "foo.go:NNN: text" content line — t.Errorf from non-test code.
|
||||
// 4. Last `_test.go:NNN: text` line — bare t.Fatalf/Errorf from a test file.
|
||||
// 5. First non-empty line of the body.
|
||||
func extractErrorLine(body string) string {
|
||||
body = stripANSI(body)
|
||||
lines := strings.Split(body, "\n")
|
||||
|
||||
var (
|
||||
msgLine string
|
||||
errLine string
|
||||
goFileLine string
|
||||
lastTestLine string
|
||||
)
|
||||
for i, raw := range lines {
|
||||
l := strings.TrimSpace(raw)
|
||||
if l == "" {
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(l, "Messages:") {
|
||||
msgLine = strings.TrimSpace(strings.TrimPrefix(l, "Messages:"))
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(l, "Error:") && errLine == "" {
|
||||
content := strings.TrimSpace(strings.TrimPrefix(l, "Error:"))
|
||||
if content == testifyErrorPlaceholder {
|
||||
// Look ahead for the wrapped error text on the next non-blank line.
|
||||
for j := i + 1; j < len(lines); j++ {
|
||||
nxt := strings.TrimSpace(lines[j])
|
||||
if nxt == "" {
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(nxt, "Test:") || strings.HasPrefix(nxt, "Messages:") {
|
||||
break
|
||||
}
|
||||
content = content + " " + nxt
|
||||
break
|
||||
}
|
||||
}
|
||||
errLine = content
|
||||
continue
|
||||
}
|
||||
if m := goFileLineRe.FindStringSubmatch(l); m != nil {
|
||||
// Keep the LAST occurrence: t.Log / harness traces emit
|
||||
// progress lines from the same file:line pattern before the
|
||||
// failing assertion finally fires.
|
||||
goFileLine = strings.TrimSpace(m[1])
|
||||
continue
|
||||
}
|
||||
if fileHintRe.MatchString(l) {
|
||||
lastTestLine = l
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case msgLine != "":
|
||||
return msgLine
|
||||
case errLine != "":
|
||||
return errLine
|
||||
case goFileLine != "":
|
||||
return goFileLine
|
||||
case lastTestLine != "":
|
||||
// "vpc_test.go:227: Eventually: condition not met …" → trim the file prefix.
|
||||
if i := strings.Index(lastTestLine, ": "); i > 0 {
|
||||
return strings.TrimSpace(lastTestLine[i+2:])
|
||||
}
|
||||
return lastTestLine
|
||||
}
|
||||
// Last resort: first non-empty line of the body.
|
||||
for _, raw := range lines {
|
||||
l := strings.TrimSpace(raw)
|
||||
if l != "" {
|
||||
return l
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// errorTraceRe extracts the path from testify's "Error Trace:" line — e.g.
|
||||
//
|
||||
// Error Trace: tests/e2e/harness/ec2helpers.go:164
|
||||
//
|
||||
// — which points at the assertion site even when it lives in helper code
|
||||
// outside *_test.go.
|
||||
var errorTraceRe = regexp.MustCompile(`Error Trace:\s*(\S+\.go:\d+)`)
|
||||
|
||||
// extractFileHint returns the most precise pointer at the assertion site
|
||||
// we can pull from the failure body: testify Error Trace wins, otherwise
|
||||
// the last `*_test.go:NNN` line in the body.
|
||||
func extractFileHint(body string) string {
|
||||
body = stripANSI(body)
|
||||
if m := errorTraceRe.FindStringSubmatch(body); m != nil {
|
||||
return m[1]
|
||||
}
|
||||
all := fileHintRe.FindAllString(body, -1)
|
||||
if len(all) == 0 {
|
||||
return ""
|
||||
}
|
||||
return all[len(all)-1]
|
||||
}
|
||||
|
||||
// signature normalises an error line so different runs of the same root
|
||||
// cause bucket together. Empty signatures (e.g. parent-test placeholders
|
||||
// where the failure body is empty) bucket together under "".
|
||||
func signature(errLine string) string {
|
||||
s := strings.ToLower(errLine)
|
||||
for _, r := range noiseReplacers {
|
||||
s = r.re.ReplaceAllString(s, r.repl)
|
||||
}
|
||||
return strings.TrimSpace(s)
|
||||
}
|
||||
|
||||
// isCascade reports whether the error line contains any of the well-known
|
||||
// fixture-propagation markers — i.e. "this test failed because something
|
||||
// upstream of it failed first".
|
||||
func isCascade(errLine string) bool {
|
||||
for _, m := range cascadeMarkers {
|
||||
if strings.Contains(errLine, m) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// computeLeafSet returns the testcase names that are leaves — i.e. no
|
||||
// other testcase name in `cases` starts with `<name>/`. go-junit-report
|
||||
// emits one <testcase> for the parent Go test AND each subtest, with the
|
||||
// parent's time = sum of its children, so summing every entry into the
|
||||
// suite's cumulative offset double-counts. Restrict accumulation to
|
||||
// leaves only.
|
||||
func computeLeafSet(cases []junitTC) map[string]bool {
|
||||
leaf := make(map[string]bool, len(cases))
|
||||
for _, tc := range cases {
|
||||
leaf[tc.Name] = true
|
||||
}
|
||||
for _, tc := range cases {
|
||||
// Strip the deepest path segment; any ancestor of this test is
|
||||
// not a leaf.
|
||||
name := tc.Name
|
||||
for {
|
||||
i := strings.LastIndex(name, "/")
|
||||
if i <= 0 {
|
||||
break
|
||||
}
|
||||
name = name[:i]
|
||||
if _, ok := leaf[name]; ok {
|
||||
leaf[name] = false
|
||||
}
|
||||
}
|
||||
}
|
||||
return leaf
|
||||
}
|
||||
|
||||
// parseStartTime turns a junit timestamp attr (RFC3339) into a time.Time;
|
||||
// zero value if unparseable.
|
||||
func parseStartTime(s string) time.Time {
|
||||
if s == "" {
|
||||
return time.Time{}
|
||||
}
|
||||
t, err := time.Parse(time.RFC3339, s)
|
||||
if err != nil {
|
||||
return time.Time{}
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
// suiteLabel turns "junit-single.xml" into "single".
|
||||
func suiteLabel(path string) string {
|
||||
base := filepath.Base(path)
|
||||
base = strings.TrimSuffix(base, ".xml")
|
||||
base = strings.TrimPrefix(base, "junit-")
|
||||
if base == "" {
|
||||
return filepath.Base(path)
|
||||
}
|
||||
return base
|
||||
}
|
||||
|
||||
// ParseFile decodes one junit-*.xml file into our normalized failure list.
|
||||
// Returns the suite label, total testcase count, and the (ordered) failures.
|
||||
func ParseFile(path string, data []byte) (SuiteReport, error) {
|
||||
var doc junitSuites
|
||||
if err := xml.Unmarshal(data, &doc); err != nil {
|
||||
return SuiteReport{}, fmt.Errorf("parse %s: %w", path, err)
|
||||
}
|
||||
|
||||
rep := SuiteReport{
|
||||
File: filepath.Base(path),
|
||||
Label: suiteLabel(path),
|
||||
}
|
||||
|
||||
order := 0
|
||||
for _, suite := range doc.Suites {
|
||||
start := parseStartTime(suite.Timestamp)
|
||||
// go-junit-report emits a testcase for both the parent Go test
|
||||
// (e.g. `TestSingleNode`) AND each subtest (`TestSingleNode/PhaseX`).
|
||||
// The parent's `time` attribute is the aggregate of its children, so
|
||||
// summing every testcase double-counts. Pre-compute the leaf set so
|
||||
// only leaves contribute to the suite's running offset.
|
||||
isLeaf := computeLeafSet(suite.Cases)
|
||||
cumul := 0.0
|
||||
for _, tc := range suite.Cases {
|
||||
rep.Total++
|
||||
leaf := isLeaf[tc.Name]
|
||||
var body string
|
||||
switch {
|
||||
case tc.Failure != nil:
|
||||
body = tc.Failure.Body
|
||||
case tc.Error != nil:
|
||||
body = tc.Error.Body
|
||||
default:
|
||||
if leaf {
|
||||
cumul += tc.Time
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
// Skip the synthetic parent-test "Failed" stub that go-junit-report
|
||||
// emits for any Go test whose subtests failed. It has an empty body
|
||||
// and would otherwise outrank real failures by XML order.
|
||||
trimmed := strings.TrimSpace(stripANSI(body))
|
||||
if trimmed == "" {
|
||||
if leaf {
|
||||
cumul += tc.Time
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
errLine := extractErrorLine(body)
|
||||
offset := time.Duration(cumul * float64(time.Second))
|
||||
f := Failure{
|
||||
SuiteFile: rep.Label,
|
||||
Name: tc.Name,
|
||||
Duration: tc.Time,
|
||||
Order: order,
|
||||
Error: errLine,
|
||||
FileHint: extractFileHint(body),
|
||||
Cascade: isCascade(errLine),
|
||||
Signature: signature(errLine),
|
||||
OffsetFromSuiteStart: offset,
|
||||
}
|
||||
if !start.IsZero() {
|
||||
f.StartAt = start.Add(offset)
|
||||
}
|
||||
rep.FailCount++
|
||||
rep.Cascades = append(rep.Cascades, f) // temp staging; split below
|
||||
order++
|
||||
if leaf {
|
||||
cumul += tc.Time
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if rep.FailCount == 0 {
|
||||
rep.Cascades = nil
|
||||
return rep, nil
|
||||
}
|
||||
|
||||
all := rep.Cascades
|
||||
rep.Cascades = nil
|
||||
|
||||
// Root = earliest non-cascade. If every failure is a cascade (the whole
|
||||
// suite tripped on the same upstream gap), fall back to earliest overall.
|
||||
rootIdx := -1
|
||||
for i, f := range all {
|
||||
if !f.Cascade {
|
||||
rootIdx = i
|
||||
break
|
||||
}
|
||||
}
|
||||
if rootIdx == -1 {
|
||||
rootIdx = 0
|
||||
}
|
||||
root := all[rootIdx]
|
||||
rep.Root = &root
|
||||
|
||||
// Bucket the remainder by signature so the report can collapse the
|
||||
// "ten failures, same bug" pattern.
|
||||
bucketKeys := []string{}
|
||||
buckets := map[string][]Failure{}
|
||||
for i, f := range all {
|
||||
if i == rootIdx {
|
||||
continue
|
||||
}
|
||||
if _, seen := buckets[f.Signature]; !seen {
|
||||
bucketKeys = append(bucketKeys, f.Signature)
|
||||
}
|
||||
buckets[f.Signature] = append(buckets[f.Signature], f)
|
||||
rep.Cascades = append(rep.Cascades, f)
|
||||
}
|
||||
|
||||
// Largest bucket first; ties broken by first-seen order so output is
|
||||
// deterministic across runs.
|
||||
sort.SliceStable(bucketKeys, func(i, j int) bool {
|
||||
return len(buckets[bucketKeys[i]]) > len(buckets[bucketKeys[j]])
|
||||
})
|
||||
for _, k := range bucketKeys {
|
||||
rep.Buckets = append(rep.Buckets, buckets[k])
|
||||
}
|
||||
|
||||
return rep, nil
|
||||
}
|
||||
|
||||
// Render writes the markdown report. Format is documented in
|
||||
// docs/development/improvements/e2e-go-failure-analysis.md (Stage 1).
|
||||
func Render(r Report) string {
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "## %s\n\n", r.Title)
|
||||
|
||||
totalFail := 0
|
||||
for _, s := range r.Suites {
|
||||
totalFail += s.FailCount
|
||||
}
|
||||
if totalFail == 0 {
|
||||
b.WriteString("✅ No failures across any suite.\n")
|
||||
return b.String()
|
||||
}
|
||||
|
||||
for _, s := range r.Suites {
|
||||
if s.FailCount == 0 {
|
||||
fmt.Fprintf(&b, "### Suite `%s`: ✅ pass (%d tests)\n\n", s.Label, s.Total)
|
||||
continue
|
||||
}
|
||||
fmt.Fprintf(&b, "### Suite `%s`: %d failed, 1 root cause likely\n\n", s.Label, s.FailCount)
|
||||
|
||||
if s.Root != nil {
|
||||
b.WriteString("**Root cause (earliest non-cascade)**\n\n")
|
||||
fmt.Fprintf(&b, "- Test: `%s`\n", s.Root.Name)
|
||||
if !s.Root.StartAt.IsZero() {
|
||||
fmt.Fprintf(&b, "- Start: %s (duration %.1fs)\n",
|
||||
s.Root.StartAt.Format("15:04:05"), s.Root.Duration)
|
||||
} else {
|
||||
fmt.Fprintf(&b, "- Duration: %.1fs\n", s.Root.Duration)
|
||||
}
|
||||
if s.Root.Error != "" {
|
||||
fmt.Fprintf(&b, "- Error: %s\n", oneLine(s.Root.Error))
|
||||
}
|
||||
if s.Root.FileHint != "" {
|
||||
fmt.Fprintf(&b, "- File: `%s`\n", s.Root.FileHint)
|
||||
}
|
||||
if s.Root.BundlePath != "" {
|
||||
fmt.Fprintf(&b, "- Bundle: `%s/`\n", s.Root.BundlePath)
|
||||
}
|
||||
b.WriteString("\n")
|
||||
}
|
||||
|
||||
if len(s.Buckets) > 0 {
|
||||
fmt.Fprintf(&b, "**Cascaded failures (%d) — grouped by signature**\n\n",
|
||||
len(s.Cascades))
|
||||
for _, bucket := range s.Buckets {
|
||||
sample := bucket[0]
|
||||
label := oneLine(sample.Error)
|
||||
if label == "" {
|
||||
label = "(no error message)"
|
||||
}
|
||||
fmt.Fprintf(&b, "- %d× %s\n", len(bucket), label)
|
||||
// List up to 5 test names per bucket; collapse the rest.
|
||||
const cap = 5
|
||||
for i, f := range bucket {
|
||||
if i == cap && len(bucket) > cap {
|
||||
fmt.Fprintf(&b, " - … and %d more\n", len(bucket)-cap)
|
||||
break
|
||||
}
|
||||
if f.BundlePath != "" {
|
||||
fmt.Fprintf(&b, " - `%s` → `%s/`\n", f.Name, f.BundlePath)
|
||||
} else {
|
||||
fmt.Fprintf(&b, " - `%s`\n", f.Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
b.WriteString("\n")
|
||||
}
|
||||
}
|
||||
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// oneLine collapses whitespace and caps length so a runaway assertion
|
||||
// message doesn't blow out the summary card width.
|
||||
func oneLine(s string) string {
|
||||
s = strings.TrimSpace(s)
|
||||
s = strings.ReplaceAll(s, "\n", " ")
|
||||
s = strings.ReplaceAll(s, "\r", " ")
|
||||
s = regexp.MustCompile(`\s+`).ReplaceAllString(s, " ")
|
||||
const max = 160
|
||||
if len(s) > max {
|
||||
s = s[:max-1] + "…"
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,191 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// updateGolden lets `go test ./.github/actions/e2e-analyze/ -update` rewrite
|
||||
// the expected analysis.md files when the rendering format changes
|
||||
// intentionally. Day-to-day, leave it unset: tests diff against the on-disk
|
||||
// snapshot to catch unintended drift in signature extraction or layout.
|
||||
var updateGolden = flag.Bool("update", false, "rewrite testdata/*/analysis.md from current output")
|
||||
|
||||
// TestGoldenFixtures runs the full ParseFile + Render pipeline against each
|
||||
// fixture directory under testdata/ and asserts the output matches the
|
||||
// adjacent analysis.md snapshot. Each subdir is one named scenario; the
|
||||
// fixture XMLs match the same junit-*.xml glob the production action uses
|
||||
// so behaviour stays representative of CI.
|
||||
func TestGoldenFixtures(t *testing.T) {
|
||||
entries, err := os.ReadDir("testdata")
|
||||
if err != nil {
|
||||
t.Fatalf("read testdata: %v", err)
|
||||
}
|
||||
for _, e := range entries {
|
||||
if !e.IsDir() {
|
||||
continue
|
||||
}
|
||||
name := e.Name()
|
||||
t.Run(name, func(t *testing.T) {
|
||||
dir := filepath.Join("testdata", name)
|
||||
matches, err := filepath.Glob(filepath.Join(dir, "junit-*.xml"))
|
||||
if err != nil {
|
||||
t.Fatalf("glob: %v", err)
|
||||
}
|
||||
sort.Strings(matches)
|
||||
if len(matches) == 0 {
|
||||
t.Fatalf("no junit-*.xml fixtures in %s", dir)
|
||||
}
|
||||
|
||||
rep := Report{Title: "E2E failure analysis"}
|
||||
for _, p := range matches {
|
||||
data, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
t.Fatalf("read %s: %v", p, err)
|
||||
}
|
||||
sr, err := ParseFile(p, data)
|
||||
if err != nil {
|
||||
t.Fatalf("parse %s: %v", p, err)
|
||||
}
|
||||
rep.Suites = append(rep.Suites, sr)
|
||||
}
|
||||
|
||||
got := Render(rep)
|
||||
goldenPath := filepath.Join(dir, "analysis.md")
|
||||
if *updateGolden {
|
||||
if err := os.WriteFile(goldenPath, []byte(got), 0o644); err != nil {
|
||||
t.Fatalf("write golden: %v", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
want, err := os.ReadFile(goldenPath)
|
||||
if err != nil {
|
||||
t.Fatalf("read golden (run with -update to seed): %v", err)
|
||||
}
|
||||
if string(want) != got {
|
||||
t.Errorf("rendered output differs from %s\n--- want ---\n%s\n--- got ---\n%s",
|
||||
goldenPath, string(want), got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestSignatureCollapsesNoise(t *testing.T) {
|
||||
a := signature("describe-vpcs: InvalidParameterValue request-id: abc-123 vpc-deadbeefcafe1234")
|
||||
b := signature("describe-vpcs: InvalidParameterValue request-id: xyz-999 vpc-1111222233334444")
|
||||
if a != b {
|
||||
t.Errorf("signatures should collapse to same bucket:\n a=%q\n b=%q", a, b)
|
||||
}
|
||||
if !strings.Contains(a, "<id>") || !strings.Contains(a, "vpc-<id>") {
|
||||
t.Errorf("expected normalised placeholders in signature, got %q", a)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCascadeDetection(t *testing.T) {
|
||||
cases := []struct {
|
||||
in string
|
||||
want bool
|
||||
}{
|
||||
{"Phase 5 must populate fix.InstanceID", true},
|
||||
{"Should NOT be empty", true},
|
||||
{"Expected value not to be nil", true},
|
||||
{"describe-vpcs: InvalidParameterValue", false},
|
||||
{"Eventually: condition not met within 3m0s", false},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := isCascade(c.in); got != c.want {
|
||||
t.Errorf("isCascade(%q) = %v, want %v", c.in, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractErrorLine_MessagesWinsOverError(t *testing.T) {
|
||||
body := ` lifecycle_test.go:28:
|
||||
Error Trace: tests/e2e/single/lifecycle_test.go:28
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase5a_pre_ClusterStats`
|
||||
got := extractErrorLine(body)
|
||||
want := "Phase 5 must populate fix.InstanceID"
|
||||
if got != want {
|
||||
t.Errorf("extractErrorLine = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractErrorLine_FallsBackToLastTestLine(t *testing.T) {
|
||||
body := " vpc_test.go:227: Eventually: condition not met within 3m0s: [SSH handshake never completed]"
|
||||
got := extractErrorLine(body)
|
||||
if !strings.HasPrefix(got, "Eventually:") {
|
||||
t.Errorf("expected fallback to test-line content, got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractFileHint_PicksLastMatch(t *testing.T) {
|
||||
body := "ec2helpers.go:50: setup\n vpc_test.go:227: Eventually: condition not met"
|
||||
got := extractFileHint(body)
|
||||
want := "vpc_test.go:227"
|
||||
if got != want {
|
||||
t.Errorf("extractFileHint = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseFile_SkipsEmptyParentFailure(t *testing.T) {
|
||||
// The synthetic "TestSingleNode" parent gets a <failure message="Failed"/>
|
||||
// with no body whenever any subtest fails. The analyzer must skip it so
|
||||
// it doesn't outrank the real first failure on XML order.
|
||||
xml := []byte(`<?xml version="1.0"?>
|
||||
<testsuites><testsuite name="" tests="2" failures="2">
|
||||
<testcase name="TestX" time="1.0"><failure message="Failed"></failure></testcase>
|
||||
<testcase name="TestX/RealFailure" time="0.5"><failure message="Failed"><![CDATA[
|
||||
foo_test.go:10:
|
||||
Messages: real assertion message
|
||||
]]></failure></testcase>
|
||||
</testsuite></testsuites>`)
|
||||
sr, err := ParseFile("junit-x.xml", xml)
|
||||
if err != nil {
|
||||
t.Fatalf("ParseFile: %v", err)
|
||||
}
|
||||
if sr.FailCount != 1 {
|
||||
t.Fatalf("FailCount = %d, want 1 (empty parent should be skipped)", sr.FailCount)
|
||||
}
|
||||
if sr.Root == nil || sr.Root.Name != "TestX/RealFailure" {
|
||||
t.Fatalf("expected real failure as root, got %+v", sr.Root)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseFile_CascadeFallbackWhenAllCascades(t *testing.T) {
|
||||
// If every failure is a cascade marker, the earliest one becomes root
|
||||
// so the report isn't blank.
|
||||
xml := []byte(`<?xml version="1.0"?>
|
||||
<testsuites><testsuite name="" tests="2" failures="2">
|
||||
<testcase name="TestX/A" time="1.0"><failure message="Failed"><![CDATA[
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
]]></failure></testcase>
|
||||
<testcase name="TestX/B" time="1.0"><failure message="Failed"><![CDATA[
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
]]></failure></testcase>
|
||||
</testsuite></testsuites>`)
|
||||
sr, err := ParseFile("junit-x.xml", xml)
|
||||
if err != nil {
|
||||
t.Fatalf("ParseFile: %v", err)
|
||||
}
|
||||
if sr.Root == nil || sr.Root.Name != "TestX/A" {
|
||||
t.Fatalf("expected TestX/A as fallback root, got %+v", sr.Root)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRender_NoFailuresIsClean(t *testing.T) {
|
||||
out := Render(Report{
|
||||
Title: "E2E failure analysis",
|
||||
Suites: []SuiteReport{
|
||||
{File: "junit-single.xml", Label: "single", Total: 3, FailCount: 0},
|
||||
},
|
||||
})
|
||||
if !strings.Contains(out, "No failures") {
|
||||
t.Errorf("expected zero-failure banner, got:\n%s", out)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,208 @@
|
||||
package main
|
||||
|
||||
// Stage 2 of docs/development/improvements/e2e-go-failure-analysis.md:
|
||||
// for each failure surfaced by Stage 1, materialise a per-failure bundle
|
||||
// directory containing the failure summary, a time-window slice of the
|
||||
// daemon journal, a testcase-window slice of the suite's stdout log, and
|
||||
// any per-test artifacts the harness already dumped at run-time.
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// bundlePad is the per-failure window padding either side of the
|
||||
// [StartAt, StartAt+Duration] testcase span. Five seconds matches the
|
||||
// plan doc: long enough to catch the daemon's pre-test setup and
|
||||
// post-test cleanup chatter, short enough to keep the slice focused.
|
||||
const bundlePad = 5 * time.Second
|
||||
|
||||
// nameSanitiseRe replaces characters unsafe in directory names with `_`.
|
||||
// Tracks tests/e2e/harness/artifacts.go which uses the same convention
|
||||
// when the test's own dump dir is created, so we can pick its contents
|
||||
// up by path without a second translation table.
|
||||
var nameSanitiseRe = regexp.MustCompile(`[^A-Za-z0-9._-]+`)
|
||||
|
||||
func sanitiseTestName(s string) string {
|
||||
return nameSanitiseRe.ReplaceAllString(s, "_")
|
||||
}
|
||||
|
||||
// ApplySuiteStartFiles overrides each failure's StartAt by reading
|
||||
// <logDir>/test-<suiteLabel>.start (RFC3339 timestamp written by the
|
||||
// workflow before the suite runs). Required because go-junit-report's
|
||||
// `timestamp` attribute on <testsuite> records the time the XML was
|
||||
// produced — i.e. after the suite has finished — so it can't be used
|
||||
// as the suite wall-clock origin for journal slicing.
|
||||
//
|
||||
// Missing or unparseable files are silently ignored: the analyzer
|
||||
// falls back to the junit timestamp and the bundle still gets created,
|
||||
// it just may have an empty (or off-window) journal slice.
|
||||
func ApplySuiteStartFiles(rep *Report, logDir string) {
|
||||
if logDir == "" {
|
||||
return
|
||||
}
|
||||
for si := range rep.Suites {
|
||||
s := &rep.Suites[si]
|
||||
startPath := filepath.Join(logDir, fmt.Sprintf("test-%s.start", s.Label))
|
||||
data, err := os.ReadFile(startPath)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
ts, err := time.Parse(time.RFC3339, strings.TrimSpace(string(data)))
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: parse %s: %v\n", startPath, err)
|
||||
continue
|
||||
}
|
||||
if s.Root != nil {
|
||||
s.Root.StartAt = ts.Add(s.Root.OffsetFromSuiteStart)
|
||||
}
|
||||
for ci := range s.Cascades {
|
||||
s.Cascades[ci].StartAt = ts.Add(s.Cascades[ci].OffsetFromSuiteStart)
|
||||
}
|
||||
for bi := range s.Buckets {
|
||||
for fi := range s.Buckets[bi] {
|
||||
f := &s.Buckets[bi][fi]
|
||||
f.StartAt = ts.Add(f.OffsetFromSuiteStart)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// WriteBundles materialises per-failure bundles under <logDir>/analysis/.
|
||||
// Mutates rep in-place so each Failure.BundlePath holds the relative
|
||||
// path the renderer will link from the summary.
|
||||
//
|
||||
// Failures with no parseable wall-clock start still get a bundle dir
|
||||
// (and the testcase-window test.log slice), but no journal slice — the
|
||||
// time window is unknown.
|
||||
//
|
||||
// Errors are non-fatal: a bundle that partially fails still gets linked
|
||||
// from the report so a reader sees what's available, and the analyzer
|
||||
// keeps printing warnings so CI logs surface the gap.
|
||||
func WriteBundles(rep *Report, logDir string) {
|
||||
if logDir == "" {
|
||||
return
|
||||
}
|
||||
base := filepath.Join(logDir, "analysis")
|
||||
if err := os.MkdirAll(base, 0o755); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: mkdir %s: %v\n", base, err)
|
||||
return
|
||||
}
|
||||
|
||||
for si := range rep.Suites {
|
||||
s := &rep.Suites[si]
|
||||
if s.Root != nil {
|
||||
writeOneBundle(logDir, base, s.Label, s.Root)
|
||||
}
|
||||
for ci := range s.Cascades {
|
||||
writeOneBundle(logDir, base, s.Label, &s.Cascades[ci])
|
||||
}
|
||||
// Bucket entries share storage with Cascades but live in a
|
||||
// separate slice in SuiteReport; mirror the bundle path so the
|
||||
// renderer's bucket loop sees it too.
|
||||
for bi := range s.Buckets {
|
||||
for fi := range s.Buckets[bi] {
|
||||
f := &s.Buckets[bi][fi]
|
||||
if bp := findBundlePath(s, f.Name); bp != "" {
|
||||
f.BundlePath = bp
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// findBundlePath looks up a Failure's already-written BundlePath by test
|
||||
// name within a suite. Buckets reference the same logical failure as
|
||||
// Cascades but are independent slice values, so we copy the path across
|
||||
// rather than re-deriving it.
|
||||
func findBundlePath(s *SuiteReport, testName string) string {
|
||||
if s.Root != nil && s.Root.Name == testName {
|
||||
return s.Root.BundlePath
|
||||
}
|
||||
for _, c := range s.Cascades {
|
||||
if c.Name == testName {
|
||||
return c.BundlePath
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func writeOneBundle(logDir, base, suiteLabel string, f *Failure) {
|
||||
name := fmt.Sprintf("%02d-%s", f.Order+1, sanitiseTestName(f.Name))
|
||||
dir := filepath.Join(base, name)
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: mkdir %s: %v\n", dir, err)
|
||||
return
|
||||
}
|
||||
f.BundlePath = filepath.Join("analysis", name)
|
||||
|
||||
writeFailureSummary(dir, suiteLabel, f)
|
||||
|
||||
if !f.StartAt.IsZero() {
|
||||
start := f.StartAt.Add(-bundlePad)
|
||||
end := f.StartAt.Add(time.Duration(f.Duration * float64(time.Second))).Add(bundlePad)
|
||||
journalSrc := filepath.Join(logDir, "spinifex-journal.log")
|
||||
if _, err := SliceJournal(journalSrc, start, end, filepath.Join(dir, "journal.log")); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: slice journal for %s: %v\n", f.Name, err)
|
||||
}
|
||||
}
|
||||
|
||||
testLogSrc := filepath.Join(logDir, fmt.Sprintf("test-%s.log", suiteLabel))
|
||||
if _, err := SliceTestLog(testLogSrc, f.Name, filepath.Join(dir, "test.log")); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: slice test log for %s: %v\n", f.Name, err)
|
||||
}
|
||||
|
||||
// Per-test artifact subdir written by harness.ArtifactDir lives at
|
||||
// <logDir>/<TestName with / replaced by _>/. Copy any files into
|
||||
// the bundle so qemu/ovs/etc. dumps already collected by the test
|
||||
// travel alongside the analysis.
|
||||
perTestSrc := filepath.Join(logDir, strings.ReplaceAll(f.Name, "/", "_"))
|
||||
if entries, err := os.ReadDir(perTestSrc); err == nil {
|
||||
for _, e := range entries {
|
||||
if e.IsDir() {
|
||||
continue
|
||||
}
|
||||
_ = copyFile(filepath.Join(perTestSrc, e.Name()), filepath.Join(dir, e.Name()))
|
||||
}
|
||||
} else if !os.IsNotExist(err) && !errorsIsNotDir(err) {
|
||||
fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: read per-test dir %s: %v\n", perTestSrc, err)
|
||||
}
|
||||
}
|
||||
|
||||
// errorsIsNotDir handles the case where the harness wrote a file at the
|
||||
// per-test name instead of a directory — best-effort skip.
|
||||
func errorsIsNotDir(err error) bool {
|
||||
var pe *fs.PathError
|
||||
if !errors.As(err, &pe) {
|
||||
return false
|
||||
}
|
||||
return pe.Err != nil && strings.Contains(pe.Err.Error(), "not a directory")
|
||||
}
|
||||
|
||||
func writeFailureSummary(dir, suiteLabel string, f *Failure) {
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "Suite: %s\n", suiteLabel)
|
||||
fmt.Fprintf(&b, "Test: %s\n", f.Name)
|
||||
if !f.StartAt.IsZero() {
|
||||
fmt.Fprintf(&b, "Start: %s\n", f.StartAt.Format(time.RFC3339))
|
||||
}
|
||||
fmt.Fprintf(&b, "Duration: %.3fs\n", f.Duration)
|
||||
if f.FileHint != "" {
|
||||
fmt.Fprintf(&b, "Site: %s\n", f.FileHint)
|
||||
}
|
||||
if f.Cascade {
|
||||
b.WriteString("Cascade: yes (downstream of another failure)\n")
|
||||
}
|
||||
if f.Error != "" {
|
||||
b.WriteString("\nError:\n")
|
||||
b.WriteString(oneLine(f.Error))
|
||||
b.WriteString("\n")
|
||||
}
|
||||
_ = os.WriteFile(filepath.Join(dir, "failure.txt"), []byte(b.String()), 0o644)
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestSliceJournal_KeepsOnlyWindow(t *testing.T) {
|
||||
src := filepath.Join(t.TempDir(), "journal.log")
|
||||
dst := filepath.Join(t.TempDir(), "out.log")
|
||||
body := strings.Join([]string{
|
||||
"2026-05-19T12:32:30+1000 host spinifex-daemon[1]: pre",
|
||||
" continuation of pre",
|
||||
"2026-05-19T12:32:35+1000 host spinifex-daemon[1]: IN window",
|
||||
" multiline inside window",
|
||||
"2026-05-19T12:32:45+1000 host spinifex-daemon[1]: post",
|
||||
"",
|
||||
}, "\n")
|
||||
if err := os.WriteFile(src, []byte(body), 0o644); err != nil {
|
||||
t.Fatalf("write src: %v", err)
|
||||
}
|
||||
start, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:33+10:00")
|
||||
end, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:40+10:00")
|
||||
n, err := SliceJournal(src, start, end, dst)
|
||||
if err != nil {
|
||||
t.Fatalf("SliceJournal: %v", err)
|
||||
}
|
||||
if n != 2 {
|
||||
t.Fatalf("written = %d, want 2", n)
|
||||
}
|
||||
got, _ := os.ReadFile(dst)
|
||||
if !strings.Contains(string(got), "IN window") || !strings.Contains(string(got), "multiline inside window") {
|
||||
t.Errorf("missing in-window content:\n%s", got)
|
||||
}
|
||||
if strings.Contains(string(got), "pre") || strings.Contains(string(got), "post") {
|
||||
t.Errorf("kept out-of-window content:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSliceJournal_AcceptsRFC3339ColonOffset(t *testing.T) {
|
||||
// systemd's `journalctl --output=short-iso` ships RFC3339 timestamps
|
||||
// (offset includes a colon, e.g. "+10:00") on most versions. Earlier
|
||||
// regex only matched "+1000" — make sure both forms work.
|
||||
src := filepath.Join(t.TempDir(), "journal.log")
|
||||
dst := filepath.Join(t.TempDir(), "out.log")
|
||||
body := strings.Join([]string{
|
||||
"2026-05-19T12:32:30+10:00 host spinifex-daemon[1]: pre",
|
||||
"2026-05-19T12:32:35+10:00 host spinifex-daemon[1]: IN window",
|
||||
"2026-05-19T12:32:50+10:00 host spinifex-daemon[1]: post",
|
||||
"",
|
||||
}, "\n")
|
||||
if err := os.WriteFile(src, []byte(body), 0o644); err != nil {
|
||||
t.Fatalf("write src: %v", err)
|
||||
}
|
||||
start, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:33+10:00")
|
||||
end, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:40+10:00")
|
||||
n, err := SliceJournal(src, start, end, dst)
|
||||
if err != nil {
|
||||
t.Fatalf("SliceJournal: %v", err)
|
||||
}
|
||||
if n != 1 {
|
||||
t.Fatalf("written = %d, want 1", n)
|
||||
}
|
||||
got, _ := os.ReadFile(dst)
|
||||
if !strings.Contains(string(got), "IN window") {
|
||||
t.Errorf("missing in-window content:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSliceJournal_MissingSrcIsNotError(t *testing.T) {
|
||||
dst := filepath.Join(t.TempDir(), "out.log")
|
||||
n, err := SliceJournal("/no/such/file", time.Now(), time.Now(), dst)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected err: %v", err)
|
||||
}
|
||||
if n != 0 {
|
||||
t.Errorf("written = %d, want 0", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSliceTestLog_PicksOneTestcase(t *testing.T) {
|
||||
src := filepath.Join(t.TempDir(), "test.log")
|
||||
dst := filepath.Join(t.TempDir(), "out.log")
|
||||
body := strings.Join([]string{
|
||||
"=== RUN TestSingleNode",
|
||||
" before subtests",
|
||||
"=== RUN TestSingleNode/Phase5_LaunchInstance",
|
||||
" runinstances",
|
||||
" ec2helpers.go:164: describe-vpcs: InvalidParameterValue",
|
||||
"--- FAIL: TestSingleNode/Phase5_LaunchInstance (2.50s)",
|
||||
"=== RUN TestSingleNode/Phase6_Volumes",
|
||||
" other content",
|
||||
"--- FAIL: TestSingleNode/Phase6_Volumes (0.10s)",
|
||||
"--- FAIL: TestSingleNode (10.0s)",
|
||||
"",
|
||||
}, "\n")
|
||||
if err := os.WriteFile(src, []byte(body), 0o644); err != nil {
|
||||
t.Fatalf("write src: %v", err)
|
||||
}
|
||||
n, err := SliceTestLog(src, "TestSingleNode/Phase5_LaunchInstance", dst)
|
||||
if err != nil {
|
||||
t.Fatalf("SliceTestLog: %v", err)
|
||||
}
|
||||
if n != 4 {
|
||||
t.Fatalf("written = %d, want 4 (RUN + 2 inner + FAIL)", n)
|
||||
}
|
||||
got, _ := os.ReadFile(dst)
|
||||
s := string(got)
|
||||
if !strings.Contains(s, "describe-vpcs: InvalidParameterValue") {
|
||||
t.Errorf("missing failure line:\n%s", s)
|
||||
}
|
||||
if strings.Contains(s, "Phase6_Volumes") || strings.Contains(s, "before subtests") {
|
||||
t.Errorf("leaked sibling test content:\n%s", s)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSliceTestLog_UnknownTestcaseIsEmpty(t *testing.T) {
|
||||
src := filepath.Join(t.TempDir(), "test.log")
|
||||
dst := filepath.Join(t.TempDir(), "out.log")
|
||||
if err := os.WriteFile(src, []byte("=== RUN TestX\n--- PASS: TestX (0.0s)\n"), 0o644); err != nil {
|
||||
t.Fatalf("write src: %v", err)
|
||||
}
|
||||
n, err := SliceTestLog(src, "TestY", dst)
|
||||
if err != nil {
|
||||
t.Fatalf("SliceTestLog: %v", err)
|
||||
}
|
||||
if n != 0 {
|
||||
t.Errorf("written = %d, want 0", n)
|
||||
}
|
||||
if _, err := os.Stat(dst); err != nil {
|
||||
t.Errorf("dst not created: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestWriteBundles_EndToEnd builds a fake artifact dir with a junit XML,
|
||||
// a journal log, a test log, and a per-test artifact subdir, then runs
|
||||
// the full ParseFile → WriteBundles pipeline and asserts the bundle dir
|
||||
// contains everything the report links to.
|
||||
func TestWriteBundles_EndToEnd(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
|
||||
junit := `<?xml version="1.0"?>
|
||||
<testsuites><testsuite name="" tests="2" failures="1" timestamp="2026-05-19T12:32:30+10:00" time="3.0">
|
||||
<testcase name="TestSingleNode/Phase5_LaunchInstance" time="2.500">
|
||||
<failure message="Failed"><![CDATA[
|
||||
ec2helpers.go:164: describe-vpcs: InvalidParameterValue (status 400)
|
||||
Error Trace: tests/e2e/harness/ec2helpers.go:164
|
||||
Test: TestSingleNode/Phase5_LaunchInstance]]></failure>
|
||||
</testcase>
|
||||
</testsuite></testsuites>`
|
||||
if err := os.WriteFile(filepath.Join(dir, "junit-single.xml"), []byte(junit), 0o644); err != nil {
|
||||
t.Fatalf("write junit: %v", err)
|
||||
}
|
||||
|
||||
journal := strings.Join([]string{
|
||||
"2026-05-19T12:32:20+1000 host spinifex-daemon[1]: too-early",
|
||||
"2026-05-19T12:32:30+1000 host spinifex-daemon[1]: in-window-start",
|
||||
"2026-05-19T12:32:32+1000 host spinifex-daemon[1]: describe-vpcs filter=is_default",
|
||||
"2026-05-19T12:32:50+1000 host spinifex-daemon[1]: too-late",
|
||||
"",
|
||||
}, "\n")
|
||||
if err := os.WriteFile(filepath.Join(dir, "spinifex-journal.log"), []byte(journal), 0o644); err != nil {
|
||||
t.Fatalf("write journal: %v", err)
|
||||
}
|
||||
|
||||
testlog := strings.Join([]string{
|
||||
"=== RUN TestSingleNode/Phase5_LaunchInstance",
|
||||
" ec2helpers.go:164: describe-vpcs: InvalidParameterValue",
|
||||
"--- FAIL: TestSingleNode/Phase5_LaunchInstance (2.50s)",
|
||||
"",
|
||||
}, "\n")
|
||||
if err := os.WriteFile(filepath.Join(dir, "test-single.log"), []byte(testlog), 0o644); err != nil {
|
||||
t.Fatalf("write test log: %v", err)
|
||||
}
|
||||
|
||||
// Per-test artifact subdir (matches harness.ArtifactDir convention).
|
||||
perTest := filepath.Join(dir, "TestSingleNode_Phase5_LaunchInstance")
|
||||
if err := os.MkdirAll(perTest, 0o755); err != nil {
|
||||
t.Fatalf("mkdir per-test: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(perTest, "qemu-console.log"), []byte("[qemu] boot\n"), 0o644); err != nil {
|
||||
t.Fatalf("write per-test file: %v", err)
|
||||
}
|
||||
|
||||
data, _ := os.ReadFile(filepath.Join(dir, "junit-single.xml"))
|
||||
sr, err := ParseFile("junit-single.xml", data)
|
||||
if err != nil {
|
||||
t.Fatalf("ParseFile: %v", err)
|
||||
}
|
||||
rep := Report{Title: "E2E failure analysis", Suites: []SuiteReport{sr}}
|
||||
WriteBundles(&rep, dir)
|
||||
|
||||
if rep.Suites[0].Root == nil {
|
||||
t.Fatal("no root")
|
||||
}
|
||||
bp := rep.Suites[0].Root.BundlePath
|
||||
if bp == "" {
|
||||
t.Fatal("BundlePath not set")
|
||||
}
|
||||
bundle := filepath.Join(dir, bp)
|
||||
for _, want := range []string{"failure.txt", "journal.log", "test.log", "qemu-console.log"} {
|
||||
if _, err := os.Stat(filepath.Join(bundle, want)); err != nil {
|
||||
t.Errorf("missing %s in bundle %s: %v", want, bp, err)
|
||||
}
|
||||
}
|
||||
|
||||
jl, _ := os.ReadFile(filepath.Join(bundle, "journal.log"))
|
||||
if !strings.Contains(string(jl), "describe-vpcs filter=is_default") {
|
||||
t.Errorf("journal.log lacks in-window line:\n%s", jl)
|
||||
}
|
||||
if strings.Contains(string(jl), "too-early") || strings.Contains(string(jl), "too-late") {
|
||||
t.Errorf("journal.log includes out-of-window content:\n%s", jl)
|
||||
}
|
||||
|
||||
tl, _ := os.ReadFile(filepath.Join(bundle, "test.log"))
|
||||
if !strings.Contains(string(tl), "--- FAIL: TestSingleNode/Phase5_LaunchInstance") {
|
||||
t.Errorf("test.log missing FAIL marker:\n%s", tl)
|
||||
}
|
||||
|
||||
out := Render(rep)
|
||||
wantLink := fmt.Sprintf("- Bundle: `%s/`", bp)
|
||||
if !strings.Contains(out, wantLink) {
|
||||
t.Errorf("render missing bundle link %q:\n%s", wantLink, out)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
)
|
||||
|
||||
func main() {
|
||||
var junitGlob, logDir, title string
|
||||
flag.StringVar(&junitGlob, "junit-glob", os.Getenv("ANALYZE_JUNIT_GLOB"), "glob matching junit-*.xml files")
|
||||
flag.StringVar(&logDir, "log-dir", os.Getenv("ANALYZE_LOG_DIR"), "artifact dir to write analysis.md into")
|
||||
flag.StringVar(&title, "title", os.Getenv("ANALYZE_TITLE"), "report heading")
|
||||
flag.Parse()
|
||||
|
||||
if title == "" {
|
||||
title = "E2E failure analysis"
|
||||
}
|
||||
if junitGlob == "" {
|
||||
fmt.Fprintln(os.Stderr, "::error::e2e-analyze: -junit-glob is required")
|
||||
os.Exit(0) // analyzer is advisory only; existing gate owns exit code
|
||||
}
|
||||
|
||||
paths, err := filepath.Glob(junitGlob)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: bad glob %q: %v\n", junitGlob, err)
|
||||
os.Exit(0)
|
||||
}
|
||||
sort.Strings(paths)
|
||||
|
||||
report := Report{Title: title}
|
||||
for _, p := range paths {
|
||||
data, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: read %s: %v\n", p, err)
|
||||
continue
|
||||
}
|
||||
sr, err := ParseFile(p, data)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: %v\n", err)
|
||||
continue
|
||||
}
|
||||
report.Suites = append(report.Suites, sr)
|
||||
}
|
||||
|
||||
// Stage 2: prefer authoritative per-suite start times written by the
|
||||
// workflow over junit-report's conversion-time `timestamp` attribute,
|
||||
// then materialise per-failure bundles so Render can link them.
|
||||
if logDir != "" {
|
||||
ApplySuiteStartFiles(&report, logDir)
|
||||
WriteBundles(&report, logDir)
|
||||
}
|
||||
|
||||
out := Render(report)
|
||||
|
||||
if logDir != "" {
|
||||
if err := os.MkdirAll(logDir, 0o755); err == nil {
|
||||
if err := os.WriteFile(filepath.Join(logDir, "analysis.md"), []byte(out), 0o644); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: write analysis.md: %v\n", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if summaryPath := os.Getenv("GITHUB_STEP_SUMMARY"); summaryPath != "" {
|
||||
f, err := os.OpenFile(summaryPath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
|
||||
if err == nil {
|
||||
defer f.Close()
|
||||
if _, werr := f.WriteString(out); werr != nil {
|
||||
fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: append summary: %v\n", werr)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// Local invocation (developer running the tool by hand). Mirror to stdout.
|
||||
fmt.Print(out)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,176 @@
|
||||
package main
|
||||
|
||||
// Stage 2 of docs/development/improvements/e2e-go-failure-analysis.md:
|
||||
// per-failure log slicers used by bundle.go to materialise time-window /
|
||||
// testcase-window log excerpts next to each named failure.
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"io"
|
||||
"os"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// shortIsoTSRe matches the timestamp prefix produced by
|
||||
// `journalctl --output=short-iso`. The offset format varies across
|
||||
// systemd versions: most ship "+10:00" (RFC3339), some ship "+1000"
|
||||
// (no colon). Match either — the parser below picks the right layout.
|
||||
var shortIsoTSRe = regexp.MustCompile(`^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:[+-]\d{2}:?\d{2}|Z))\s`)
|
||||
|
||||
// shortIsoLayouts are tried in order; the first parse that succeeds wins.
|
||||
var shortIsoLayouts = []string{
|
||||
time.RFC3339, // 2026-05-19T12:32:30+10:00
|
||||
"2006-01-02T15:04:05-0700", // 2026-05-19T12:32:30+1000
|
||||
}
|
||||
|
||||
func parseShortIso(s string) (time.Time, error) {
|
||||
var lastErr error
|
||||
for _, layout := range shortIsoLayouts {
|
||||
t, err := time.Parse(layout, s)
|
||||
if err == nil {
|
||||
return t, nil
|
||||
}
|
||||
lastErr = err
|
||||
}
|
||||
return time.Time{}, lastErr
|
||||
}
|
||||
|
||||
// SliceJournal copies lines from src whose `short-iso` timestamp falls in
|
||||
// [start, end] to dst. Lines without a parseable timestamp inherit the
|
||||
// previous line's classification — that keeps multi-line journal records
|
||||
// (stack traces, panics) intact rather than orphaning their continuation
|
||||
// lines. Lines that precede the first parseable timestamp are dropped.
|
||||
//
|
||||
// Returns the number of lines written; missing src is not an error (the
|
||||
// journal may be empty if collection failed).
|
||||
func SliceJournal(src string, start, end time.Time, dst string) (int, error) {
|
||||
f, err := os.Open(src)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return 0, nil
|
||||
}
|
||||
return 0, err
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
out, err := os.Create(dst)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer out.Close()
|
||||
|
||||
bw := bufio.NewWriter(out)
|
||||
defer bw.Flush()
|
||||
|
||||
sc := bufio.NewScanner(f)
|
||||
// Journal lines can be long (panic dumps); bump scanner buffer.
|
||||
sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
|
||||
|
||||
keep := false
|
||||
written := 0
|
||||
for sc.Scan() {
|
||||
line := sc.Text()
|
||||
if m := shortIsoTSRe.FindStringSubmatch(line); m != nil {
|
||||
ts, err := parseShortIso(m[1])
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
keep = !ts.Before(start) && !ts.After(end)
|
||||
}
|
||||
if keep {
|
||||
if _, err := bw.WriteString(line); err != nil {
|
||||
return written, err
|
||||
}
|
||||
if err := bw.WriteByte('\n'); err != nil {
|
||||
return written, err
|
||||
}
|
||||
written++
|
||||
}
|
||||
}
|
||||
return written, sc.Err()
|
||||
}
|
||||
|
||||
// runRePrefix and finishRePrefix bracket a single testcase's output in
|
||||
// `go test -v` stdout. Subtests have their own `=== RUN` / `--- FAIL`
|
||||
// markers, so slicing on the exact testcase name yields just that test's
|
||||
// lines (including any nested phase logs).
|
||||
var (
|
||||
runRe = regexp.MustCompile(`^=== RUN\s+(\S+)$`)
|
||||
finishRe = regexp.MustCompile(`^\s*---\s+(PASS|FAIL|SKIP):\s+(\S+)\s+\(`)
|
||||
)
|
||||
|
||||
// SliceTestLog copies lines belonging to the named testcase from src to
|
||||
// dst. The block starts at `=== RUN <name>` and ends at the matching
|
||||
// `--- PASS|FAIL|SKIP: <name> (…)` line (inclusive). If the testcase is
|
||||
// never seen the destination file is created empty so the bundle has a
|
||||
// stable shape.
|
||||
//
|
||||
// Returns the number of lines written; missing src is not an error.
|
||||
func SliceTestLog(src, testName, dst string) (int, error) {
|
||||
f, err := os.Open(src)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return 0, nil
|
||||
}
|
||||
return 0, err
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
out, err := os.Create(dst)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer out.Close()
|
||||
|
||||
bw := bufio.NewWriter(out)
|
||||
defer bw.Flush()
|
||||
|
||||
sc := bufio.NewScanner(f)
|
||||
sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
|
||||
|
||||
in := false
|
||||
written := 0
|
||||
for sc.Scan() {
|
||||
line := sc.Text()
|
||||
if !in {
|
||||
if m := runRe.FindStringSubmatch(strings.TrimRight(line, "\r")); m != nil && m[1] == testName {
|
||||
in = true
|
||||
} else {
|
||||
continue
|
||||
}
|
||||
}
|
||||
if _, err := bw.WriteString(line); err != nil {
|
||||
return written, err
|
||||
}
|
||||
if err := bw.WriteByte('\n'); err != nil {
|
||||
return written, err
|
||||
}
|
||||
written++
|
||||
if m := finishRe.FindStringSubmatch(line); m != nil && m[2] == testName {
|
||||
return written, nil
|
||||
}
|
||||
}
|
||||
return written, sc.Err()
|
||||
}
|
||||
|
||||
// copyFile is a small helper used when bundling pre-collected per-test
|
||||
// artifacts. Best-effort: missing src is silently skipped.
|
||||
func copyFile(src, dst string) error {
|
||||
in, err := os.Open(src)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
defer in.Close()
|
||||
out, err := os.Create(dst)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer out.Close()
|
||||
_, err = io.Copy(out, in)
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
## E2E failure analysis
|
||||
|
||||
### Suite `single`: 16 failed, 1 root cause likely
|
||||
|
||||
**Root cause (earliest non-cascade)**
|
||||
|
||||
- Test: `TestSingleNode/Phase5_LaunchInstance`
|
||||
- Start: 12:30:00 (duration 2.5s)
|
||||
- Error: Received unexpected error: InvalidParameterValue (HTTP 400)
|
||||
- File: `tests/e2e/harness/ec2helpers.go:164`
|
||||
|
||||
**Cascaded failures (15) — grouped by signature**
|
||||
|
||||
- 14× Phase 5 must populate fix.InstanceID
|
||||
- `TestSingleNode/Phase5a_pre_ClusterStats`
|
||||
- `TestSingleNode/Phase5a_Metadata`
|
||||
- `TestSingleNode/Phase5b_DescribeInstances`
|
||||
- `TestSingleNode/Phase5c_StopStart`
|
||||
- `TestSingleNode/Phase6_Volumes`
|
||||
- … and 9 more
|
||||
- 1× Phase 2 must populate fix.AZName
|
||||
- `TestSingleNode/Phase8_NatGW`
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<testsuites tests="20" failures="17" skipped="0">
|
||||
<testsuite name="" tests="20" failures="17" errors="0" id="0" hostname="banksia" skipped="0" time="600.000" timestamp="2026-04-12T12:30:00+10:00">
|
||||
<testcase name="TestSingleNode" classname="" time="600.000">
|
||||
<failure message="Failed"></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase1_Environment" classname="" time="0.300">
|
||||
<system-out><![CDATA[
|
||||
━━ Phase 1 ━━]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase2_Discovery" classname="" time="0.010"/>
|
||||
<testcase name="TestSingleNode/Phase3_KeyPairs" classname="" time="0.540"/>
|
||||
<testcase name="TestSingleNode/Phase4_Image" classname="" time="0.060"/>
|
||||
<testcase name="TestSingleNode/Phase5_LaunchInstance" classname="" time="2.500">
|
||||
<failure message="Failed"><![CDATA[
|
||||
ec2helpers.go:164: describe-vpcs: InvalidParameterValue: Filter 'is_default' is invalid (status 400)
|
||||
Error Trace: tests/e2e/harness/ec2helpers.go:164
|
||||
Error: Received unexpected error:
|
||||
InvalidParameterValue (HTTP 400)
|
||||
Test: TestSingleNode/Phase5_LaunchInstance]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5a_pre_ClusterStats" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
lifecycle_test.go:28:
|
||||
Error Trace: tests/e2e/single/lifecycle_test.go:28
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase5a_pre_ClusterStats]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5a_Metadata" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
lifecycle_test.go:57:
|
||||
Error Trace: tests/e2e/single/lifecycle_test.go:57
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase5a_Metadata]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5b_DescribeInstances" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
lifecycle_test.go:109:
|
||||
Error Trace: tests/e2e/single/lifecycle_test.go:109
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase5b_DescribeInstances]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5c_StopStart" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
lifecycle_test.go:222:
|
||||
Error Trace: tests/e2e/single/lifecycle_test.go:222
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase5c_StopStart]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase6_Volumes" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
volume_test.go:23:
|
||||
Error Trace: tests/e2e/single/volume_test.go:23
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase6_Volumes]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase6b_AttachVolume" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
volume_test.go:24:
|
||||
Error Trace: tests/e2e/single/volume_test.go:24
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase6b_AttachVolume]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase7_Tags" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
tags_test.go:23:
|
||||
Error Trace: tests/e2e/single/tags_test.go:23
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase7_Tags]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase7a_Snapshots" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
snapshot_test.go:23:
|
||||
Error Trace: tests/e2e/single/snapshot_test.go:23
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase7a_Snapshots]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase7b_DescribeInstanceAttribute" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
instance_test.go:81:
|
||||
Error Trace: tests/e2e/single/instance_test.go:81
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase7b_DescribeInstanceAttribute]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase7c_ModifyInstanceAttribute" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
instance_test.go:92:
|
||||
Error Trace: tests/e2e/single/instance_test.go:92
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase7c_ModifyInstanceAttribute]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase7d_Console" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
instance_test.go:121:
|
||||
Error Trace: tests/e2e/single/instance_test.go:121
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase7d_Console]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase7e_Negative" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
negative_test.go:43:
|
||||
Error Trace: tests/e2e/single/negative_test.go:43
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase7e_Negative]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NatGW" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
natgw_test.go:39:
|
||||
Error Trace: tests/e2e/single/natgw_test.go:39
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 2 must populate fix.AZName
|
||||
Test: TestSingleNode/Phase8_NatGW]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8a_DataPath" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
datapath_test.go:48:
|
||||
Error Trace: tests/e2e/single/datapath_test.go:48
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase8a_DataPath]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase9_SecurityGroups" classname="" time="0.020">
|
||||
<failure message="Failed"><![CDATA[
|
||||
securitygroup_test.go:35:
|
||||
Error Trace: tests/e2e/single/securitygroup_test.go:35
|
||||
Error: Should NOT be empty
|
||||
Messages: Phase 5 must populate fix.InstanceID
|
||||
Test: TestSingleNode/Phase9_SecurityGroups]]></failure>
|
||||
</testcase>
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -0,0 +1,3 @@
|
||||
## E2E failure analysis
|
||||
|
||||
✅ No failures across any suite.
|
||||
@@ -0,0 +1,8 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<testsuites tests="3" failures="0" skipped="0">
|
||||
<testsuite name="" tests="3" failures="0" errors="0" id="0" hostname="banksia" skipped="0" time="1.500" timestamp="2026-04-12T12:30:00+10:00">
|
||||
<testcase name="TestSingleNode" classname="" time="1.500"/>
|
||||
<testcase name="TestSingleNode/Phase1_Environment" classname="" time="0.300"/>
|
||||
<testcase name="TestSingleNode/Phase2_Discovery" classname="" time="1.200"/>
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -0,0 +1,11 @@
|
||||
## E2E failure analysis
|
||||
|
||||
### Suite `single`: 1 failed, 1 root cause likely
|
||||
|
||||
**Root cause (earliest non-cascade)**
|
||||
|
||||
- Test: `TestSingleNode/Phase8b_VPCSubnetE2E`
|
||||
- Start: 00:26:18 (duration 200.5s)
|
||||
- Error: Eventually: condition not met within 3m0s: [SSH handshake 192.168.0.202:22 never completed]
|
||||
- File: `vpc_test.go:227`
|
||||
|
||||
@@ -0,0 +1,993 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<testsuites tests="79" failures="2" skipped="6">
|
||||
<testsuite name="" tests="79" failures="2" errors="0" id="0" hostname="banksia" skipped="6" time="1708.140" timestamp="2026-05-19T00:19:45+10:00">
|
||||
<testcase name="TestSingleNode" classname="" time="800.320">
|
||||
<failure message="Failed"></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase1_Environment" classname="" time="0.300">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 1 — Environment ━━�[0m
|
||||
�[2m· checking /dev/kvm�[0m
|
||||
discovery_test.go:29: �[2m· checking /dev/kvm�[0m
|
||||
�[2mkvm�[0m=writable
|
||||
discovery_test.go:38: �[2mkvm�[0m=writable
|
||||
�[2m· waiting for AWS gateway�[0m
|
||||
discovery_test.go:45: �[2m· waiting for AWS gateway�[0m
|
||||
�[2mgateway�[0m=https://192.168.0.10:9999/
|
||||
discovery_test.go:55: �[2mgateway�[0m=https://192.168.0.10:9999/
|
||||
�[2m· waiting for daemon NATS subscriptions�[0m
|
||||
discovery_test.go:60: �[2m· waiting for daemon NATS subscriptions�[0m
|
||||
�[2m· describe-regions / describe-availability-zones�[0m
|
||||
discovery_test.go:73: �[2m· describe-regions / describe-availability-zones�[0m
|
||||
�[2maz�[0m=ap-southeast-2a �[2mregion�[0m=ap-southeast-2
|
||||
discovery_test.go:84: �[2maz�[0m=ap-southeast-2a �[2mregion�[0m=ap-southeast-2]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase1b_ClusterStats" classname="" time="9.200">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 1b — Cluster Stats CLI ━━�[0m
|
||||
�[2m· spx get nodes�[0m
|
||||
discovery_test.go:101: �[2m· spx get nodes�[0m
|
||||
�[2m· spx top nodes�[0m
|
||||
discovery_test.go:105: �[2m· spx top nodes�[0m
|
||||
�[2m· spx get vms (empty before any launches)�[0m
|
||||
discovery_test.go:112: �[2m· spx get vms (empty before any launches)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase2_Discovery" classname="" time="0.010">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 2 — Discovery & Metadata ━━�[0m
|
||||
�[2m· discovering nano instance type�[0m
|
||||
discovery_test.go:133: �[2m· discovering nano instance type�[0m
|
||||
�[2minstance_type�[0m=t3.nano �[2march�[0m=x86_64 �[2maz�[0m=ap-southeast-2a
|
||||
discovery_test.go:159: �[2minstance_type�[0m=t3.nano �[2march�[0m=x86_64 �[2maz�[0m=ap-southeast-2a]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase2b_SerialConsole" classname="" time="0.020">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 2b — Serial Console Access ━━�[0m
|
||||
�[2m· default state should be disabled�[0m
|
||||
discovery_test.go:168: �[2m· default state should be disabled�[0m
|
||||
�[2m· enable�[0m
|
||||
discovery_test.go:173: �[2m· enable�[0m
|
||||
�[2mstate�[0m=enabled
|
||||
discovery_test.go:180: �[2mstate�[0m=enabled
|
||||
�[2m· disable�[0m
|
||||
discovery_test.go:182: �[2m· disable�[0m
|
||||
�[2mstate�[0m=disabled
|
||||
discovery_test.go:189: �[2mstate�[0m=disabled]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase3_KeyPairs" classname="" time="0.540">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 3 — SSH Key Management ━━�[0m
|
||||
�[2m· create-key-pair "test-key-1"�[0m
|
||||
keypair_test.go:37: �[2m· create-key-pair "test-key-1"�[0m
|
||||
�[2mkey�[0m=test-key-1 �[2mpem�[0m=/tmp/TestSingleNode1289015487/001/test-key-1.pem
|
||||
keypair_test.go:51: �[2mkey�[0m=test-key-1 �[2mpem�[0m=/tmp/TestSingleNode1289015487/001/test-key-1.pem
|
||||
�[2m· import-key-pair "test-key-2" (local-generated RSA)�[0m
|
||||
keypair_test.go:58: �[2m· import-key-pair "test-key-2" (local-generated RSA)�[0m
|
||||
�[2mimported�[0m=test-key-2
|
||||
keypair_test.go:65: �[2mimported�[0m=test-key-2
|
||||
�[2m· describe-key-pairs (both present)�[0m
|
||||
keypair_test.go:67: �[2m· describe-key-pairs (both present)�[0m
|
||||
�[2m· delete "test-key-2"�[0m
|
||||
keypair_test.go:72: �[2m· delete "test-key-2"�[0m
|
||||
�[2m· describe-key-pairs (only test-key-1 remains)�[0m
|
||||
keypair_test.go:76: �[2m· describe-key-pairs (only test-key-1 remains)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase4_Image" classname="" time="0.060">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 4 — Image Management ━━�[0m
|
||||
�[2mimage�[0m=ubuntu-24.04-x86_64 �[2march�[0m=x86_64
|
||||
image_test.go:33: �[2mimage�[0m=ubuntu-24.04-x86_64 �[2march�[0m=x86_64
|
||||
�[2m· spx admin images import --name ubuntu-24.04-x86_64�[0m
|
||||
image_test.go:35: �[2m· spx admin images import --name ubuntu-24.04-x86_64�[0m
|
||||
�[2m· spx import did not yield AMI ID; falling back to describe-images�[0m
|
||||
image_test.go:48: �[2m· spx import did not yield AMI ID; falling back to describe-images�[0m
|
||||
�[2m· describe-images by AMI ID (verify exactly one match)�[0m
|
||||
image_test.go:78: �[2m· describe-images by AMI ID (verify exactly one match)�[0m
|
||||
�[2mami�[0m=ami-6f60f2b3bf04b5453 �[2mname�[0m=ami-ubuntu-24.04-x86_64
|
||||
image_test.go:86: �[2mami�[0m=ami-6f60f2b3bf04b5453 �[2mname�[0m=ami-ubuntu-24.04-x86_64]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5_LaunchInstance" classname="" time="10.830">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 5 — Instance Lifecycle ━━�[0m
|
||||
�[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msg�[0m=sg-764a7fa2a39577204 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63
|
||||
instance_test.go:31: �[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msg�[0m=sg-764a7fa2a39577204 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63
|
||||
�[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.nano key=test-key-1�[0m
|
||||
instance_test.go:35: �[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.nano key=test-key-1�[0m
|
||||
�[2minstance�[0m=i-0c615e5dd74050f53
|
||||
instance_test.go:48: �[2minstance�[0m=i-0c615e5dd74050f53
|
||||
instance_test.go:50: instance i-0c615e5dd74050f53 reached state running
|
||||
�[2mroot_volume�[0m=vol-2230138124fb51ee2 �[2mroot_device�[0m=
|
||||
instance_test.go:70: �[2mroot_volume�[0m=vol-2230138124fb51ee2 �[2mroot_device�[0m=]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5a_pre_ClusterStats" classname="" time="3.040">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 5a-pre — Cluster Stats CLI (with running VM) ━━�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5a_Metadata" classname="" time="0.510">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 5a — Instance Metadata Validation ━━�[0m
|
||||
�[2mtype�[0m=t3.nano �[2mkey�[0m=test-key-1 �[2mimage�[0m=ami-6f60f2b3bf04b5453 �[2mbdms�[0m=1
|
||||
instance_test.go:108: �[2mtype�[0m=t3.nano �[2mkey�[0m=test-key-1 �[2mimage�[0m=ami-6f60f2b3bf04b5453 �[2mbdms�[0m=1]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5a_ii_SSH" classname="" time="49.450">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 5a-ii — SSH Connectivity & Volume Verification ━━�[0m
|
||||
�[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
|
||||
instance_test.go:127: �[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
|
||||
�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
|
||||
instance_test.go:130: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
|
||||
�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
|
||||
lifecycle_test.go:384: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
|
||||
�[2m· ssh id�[0m
|
||||
instance_test.go:136: �[2m· ssh id�[0m
|
||||
�[2m· lsblk root-volume cross-check vs API�[0m
|
||||
instance_test.go:140: �[2m· lsblk root-volume cross-check vs API�[0m
|
||||
�[2mguest_gib�[0m=4 �[2mapi_gib�[0m=4
|
||||
instance_test.go:159: �[2mguest_gib�[0m=4 �[2mapi_gib�[0m=4
|
||||
�[2m· ssh hostname�[0m
|
||||
instance_test.go:161: �[2m· ssh hostname�[0m
|
||||
�[2mhostname�[0m=spinifex-vm-0c615e5d �[2mmatches_short_id�[0m=0c615e5d
|
||||
instance_test.go:172: �[2mhostname�[0m=spinifex-vm-0c615e5d �[2mmatches_short_id�[0m=0c615e5d]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5a_iii_Console" classname="" time="0.010">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 5a-iii — Console Output ━━�[0m
|
||||
�[2minstance�[0m=i-0c615e5dd74050f53 �[2mhas_output�[0m=true
|
||||
instance_test.go:191: �[2minstance�[0m=i-0c615e5dd74050f53 �[2mhas_output�[0m=true]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5b_Volume" classname="" time="4.540">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 5b — Volume Lifecycle (Attach/Detach) ━━�[0m
|
||||
�[2m· create-volume size=10 az=ap-southeast-2a�[0m
|
||||
volume_test.go:26: �[2m· create-volume size=10 az=ap-southeast-2a�[0m
|
||||
�[2mvolume�[0m=vol-a885f79173e0a8030
|
||||
volume_test.go:34: �[2mvolume�[0m=vol-a885f79173e0a8030
|
||||
volume_test.go:39: volume vol-a885f79173e0a8030 reached state available
|
||||
�[2m· modify-volume size=20�[0m
|
||||
volume_test.go:42: �[2m· modify-volume size=20�[0m
|
||||
�[2mresized_gib�[0m=20
|
||||
volume_test.go:66: �[2mresized_gib�[0m=20
|
||||
�[2m· attach-volume vol-a885f79173e0a8030 -> i-0c615e5dd74050f53 as /dev/sdf�[0m
|
||||
volume_test.go:68: �[2m· attach-volume vol-a885f79173e0a8030 -> i-0c615e5dd74050f53 as /dev/sdf�[0m
|
||||
volume_test.go:76: volume vol-a885f79173e0a8030 reached state in-use
|
||||
�[2m· detach-volume vol-a885f79173e0a8030�[0m
|
||||
volume_test.go:90: �[2m· detach-volume vol-a885f79173e0a8030�[0m
|
||||
volume_test.go:96: volume vol-a885f79173e0a8030 reached state available
|
||||
�[2m· delete-volume vol-a885f79173e0a8030�[0m
|
||||
volume_test.go:98: �[2m· delete-volume vol-a885f79173e0a8030�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5b_ii_VolumeStatus" classname="" time="0.000">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 5b-ii — DescribeVolumeStatus ━━�[0m
|
||||
�[2mvolume�[0m=vol-2230138124fb51ee2 �[2mstatus�[0m=ok
|
||||
volume_test.go:158: �[2mvolume�[0m=vol-2230138124fb51ee2 �[2mstatus�[0m=ok]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5c_Snapshot" classname="" time="0.190">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 5c — Snapshot Lifecycle ━━�[0m
|
||||
�[2m· create-snapshot volume=vol-2230138124fb51ee2�[0m
|
||||
snapshot_test.go:34: �[2m· create-snapshot volume=vol-2230138124fb51ee2�[0m
|
||||
�[2msnapshot�[0m=snap-1188f92980e6470aa �[2msize�[0m=4 �[2mstate�[0m=completed �[2mprogress�[0m=100%
|
||||
snapshot_test.go:50: �[2msnapshot�[0m=snap-1188f92980e6470aa �[2msize�[0m=4 �[2mstate�[0m=completed �[2mprogress�[0m=100%
|
||||
snapshot_test.go:59: snapshot snap-1188f92980e6470aa reached state completed
|
||||
�[2m· describe-snapshots snap-1188f92980e6470aa�[0m
|
||||
snapshot_test.go:61: �[2m· describe-snapshots snap-1188f92980e6470aa�[0m
|
||||
�[2m· copy-snapshot src=snap-1188f92980e6470aa region=ap-southeast-2�[0m
|
||||
snapshot_test.go:76: �[2m· copy-snapshot src=snap-1188f92980e6470aa region=ap-southeast-2�[0m
|
||||
�[2mcopy_snapshot�[0m=snap-cde6b5ddf467e2c0f
|
||||
snapshot_test.go:86: �[2mcopy_snapshot�[0m=snap-cde6b5ddf467e2c0f
|
||||
snapshot_test.go:88: snapshot snap-cde6b5ddf467e2c0f reached state completed
|
||||
�[2m· describe-snapshots snap-1188f92980e6470aa,snap-cde6b5ddf467e2c0f (both visible)�[0m
|
||||
snapshot_test.go:90: �[2m· describe-snapshots snap-1188f92980e6470aa,snap-cde6b5ddf467e2c0f (both visible)�[0m
|
||||
�[2m· delete-snapshot snap-1188f92980e6470aa (original)�[0m
|
||||
snapshot_test.go:107: �[2m· delete-snapshot snap-1188f92980e6470aa (original)�[0m
|
||||
�[2m· verify copy snap-cde6b5ddf467e2c0f still completed�[0m
|
||||
snapshot_test.go:135: �[2m· verify copy snap-cde6b5ddf467e2c0f still completed�[0m
|
||||
�[2m· delete-snapshot snap-cde6b5ddf467e2c0f (copy)�[0m
|
||||
snapshot_test.go:144: �[2m· delete-snapshot snap-cde6b5ddf467e2c0f (copy)�[0m
|
||||
�[2mdeleted_original�[0m=snap-1188f92980e6470aa
|
||||
snapshot_test.go:151: �[2mdeleted_original�[0m=snap-1188f92980e6470aa]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5d_SnapshotBackedLaunch" classname="" time="0.010">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 5d — Snapshot-Backed Instance Launch ━━�[0m
|
||||
�[2mami�[0m=ami-6f60f2b3bf04b5453 �[2msnapshot�[0m=snap-ami-6f60f2b3bf04b5453
|
||||
snapshot_test.go:182: �[2mami�[0m=ami-6f60f2b3bf04b5453 �[2msnapshot�[0m=snap-ami-6f60f2b3bf04b5453]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5e_CreateImage" classname="" time="0.660">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 5e — CreateImage Lifecycle ━━�[0m
|
||||
�[2m· create-image instance=i-0c615e5dd74050f53 name=e2e-custom-ami (no-reboot)�[0m
|
||||
createimage_test.go:26: �[2m· create-image instance=i-0c615e5dd74050f53 name=e2e-custom-ami (no-reboot)�[0m
|
||||
�[2mcustom_ami�[0m=ami-7dcf263a412dd14c6
|
||||
createimage_test.go:36: �[2mcustom_ami�[0m=ami-7dcf263a412dd14c6
|
||||
createimage_test.go:38: image ami-7dcf263a412dd14c6 reached state available
|
||||
�[2m· describe-images ami-7dcf263a412dd14c6�[0m
|
||||
createimage_test.go:40: �[2m· describe-images ami-7dcf263a412dd14c6�[0m
|
||||
�[2mcustom_ami_snapshot�[0m=snap-dde89b9ea2ef2e05a
|
||||
createimage_test.go:67: �[2mcustom_ami_snapshot�[0m=snap-dde89b9ea2ef2e05a]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase5f_SecurityGroupEgress" classname="" time="4.880">
|
||||
<skipped message="Skipped"><![CDATA[
|
||||
�[1m�[36m━━ Phase 5f — Security Group Enforcement (egress ACL) ━━�[0m
|
||||
�[2m· discover default gateway inside VM�[0m
|
||||
securitygroup_test.go:53: �[2m· discover default gateway inside VM�[0m
|
||||
�[2mprobe_gateway�[0m=172.31.0.1
|
||||
securitygroup_test.go:66: �[2mprobe_gateway�[0m=172.31.0.1
|
||||
�[2m· 5f-1 baseline egress (allow-all)�[0m
|
||||
securitygroup_test.go:77: �[2m· 5f-1 baseline egress (allow-all)�[0m
|
||||
securitygroup_test.go:80: Phase 5f: baseline ICMP did not succeed; env may block ICMP regardless of SG
|
||||
Output:
|
||||
PING 172.31.0.1 (172.31.0.1) 56(84) bytes of data.
|
||||
|
||||
--- 172.31.0.1 ping statistics ---
|
||||
3 packets transmitted, 0 received, 100% packet loss, time 2044ms
|
||||
]]></skipped>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase6_TagManagement" classname="" time="0.380">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 6 — Tag Management ━━�[0m
|
||||
�[2m· 6a create-tags instance=i-0c615e5dd74050f53 (Name, Environment, DeleteMe)�[0m
|
||||
tags_test.go:41: �[2m· 6a create-tags instance=i-0c615e5dd74050f53 (Name, Environment, DeleteMe)�[0m
|
||||
�[2m· 6b describe-tags resource-id=i-0c615e5dd74050f53 want=3�[0m
|
||||
tags_test.go:53: �[2m· 6b describe-tags resource-id=i-0c615e5dd74050f53 want=3�[0m
|
||||
�[2m· 6c create-tags volume=vol-2230138124fb51ee2 (Name, Environment)�[0m
|
||||
tags_test.go:65: �[2m· 6c create-tags volume=vol-2230138124fb51ee2 (Name, Environment)�[0m
|
||||
�[2m· 6d describe-tags key=Environment want=2�[0m
|
||||
tags_test.go:76: �[2m· 6d describe-tags key=Environment want=2�[0m
|
||||
�[2m· 6e describe-tags resource-type=instance want=3�[0m
|
||||
tags_test.go:85: �[2m· 6e describe-tags resource-type=instance want=3�[0m
|
||||
�[2m· 6f overwrite instance Name -> e2e-test-updated�[0m
|
||||
tags_test.go:95: �[2m· 6f overwrite instance Name -> e2e-test-updated�[0m
|
||||
�[2m· 6g delete-tags Key=DeleteMe (no value)�[0m
|
||||
tags_test.go:114: �[2m· 6g delete-tags Key=DeleteMe (no value)�[0m
|
||||
�[2m· 6h delete-tags Key=Environment,Value=production (no-op)�[0m
|
||||
tags_test.go:130: �[2m· 6h delete-tags Key=Environment,Value=production (no-op)�[0m
|
||||
�[2m· 6i delete-tags Key=Environment,Value=testing (instance only)�[0m
|
||||
tags_test.go:148: �[2m· 6i delete-tags Key=Environment,Value=testing (instance only)�[0m
|
||||
�[2m· 6j final describe-tags resource-id=i-0c615e5dd74050f53 want=1�[0m
|
||||
tags_test.go:171: �[2m· 6j final describe-tags resource-id=i-0c615e5dd74050f53 want=1�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase7_StopStart" classname="" time="8.090">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 7 — Instance State Transitions (Stop) ━━�[0m
|
||||
�[2m· stop-instances i-0c615e5dd74050f53�[0m
|
||||
lifecycle_test.go:30: �[2m· stop-instances i-0c615e5dd74050f53�[0m
|
||||
lifecycle_test.go:36: instance i-0c615e5dd74050f53 reached state stopped
|
||||
�[2m· reboot-instances on stopped instance (should fail)�[0m
|
||||
lifecycle_test.go:40: �[2m· reboot-instances on stopped instance (should fail)�[0m
|
||||
�[2mstopped_reject�[0m=IncorrectInstanceState
|
||||
lifecycle_test.go:47: �[2mstopped_reject�[0m=IncorrectInstanceState]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase7a_AttachToStoppedError" classname="" time="0.300">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 7a — Attach Volume to Stopped Instance (Error Path) ━━�[0m
|
||||
�[2m· create-volume size=10 az=ap-southeast-2a�[0m
|
||||
lifecycle_test.go:60: �[2m· create-volume size=10 az=ap-southeast-2a�[0m
|
||||
�[2mtest_volume�[0m=vol-8d3e1335938a378cd
|
||||
lifecycle_test.go:68: �[2mtest_volume�[0m=vol-8d3e1335938a378cd
|
||||
lifecycle_test.go:82: volume vol-8d3e1335938a378cd reached state available
|
||||
�[2m· attach-volume vol-8d3e1335938a378cd -> i-0c615e5dd74050f53 (should fail)�[0m
|
||||
lifecycle_test.go:84: �[2m· attach-volume vol-8d3e1335938a378cd -> i-0c615e5dd74050f53 (should fail)�[0m
|
||||
�[2m· delete-volume vol-8d3e1335938a378cd�[0m
|
||||
lifecycle_test.go:94: �[2m· delete-volume vol-8d3e1335938a378cd�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase7b_ModifyInstanceAttribute" classname="" time="55.620">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 7b — ModifyInstanceAttribute ━━�[0m
|
||||
�[2mfrom_type�[0m=t3.nano �[2mto_type�[0m=t3.small
|
||||
lifecycle_test.go:120: �[2mfrom_type�[0m=t3.nano �[2mto_type�[0m=t3.small
|
||||
�[2mexpected_vcpus�[0m=2 �[2mexpected_mem_mib�[0m=2048
|
||||
lifecycle_test.go:142: �[2mexpected_vcpus�[0m=2 �[2mexpected_mem_mib�[0m=2048
|
||||
�[2m· modify-instance-attribute i-0c615e5dd74050f53 type=t3.small�[0m
|
||||
lifecycle_test.go:144: �[2m· modify-instance-attribute i-0c615e5dd74050f53 type=t3.small�[0m
|
||||
�[2m· start-instances i-0c615e5dd74050f53�[0m
|
||||
lifecycle_test.go:165: �[2m· start-instances i-0c615e5dd74050f53�[0m
|
||||
lifecycle_test.go:171: instance i-0c615e5dd74050f53 reached state running
|
||||
�[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
|
||||
lifecycle_test.go:177: �[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
|
||||
�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
|
||||
lifecycle_test.go:179: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
|
||||
�[2m· ssh nproc�[0m
|
||||
lifecycle_test.go:182: �[2m· ssh nproc�[0m
|
||||
�[2mvm_vcpus�[0m=2
|
||||
lifecycle_test.go:188: �[2mvm_vcpus�[0m=2
|
||||
�[2m· ssh MemTotal�[0m
|
||||
lifecycle_test.go:190: �[2m· ssh MemTotal�[0m
|
||||
�[2mvm_mem_mib�[0m=1967 �[2mthreshold_mib�[0m=1740
|
||||
lifecycle_test.go:200: �[2mvm_mem_mib�[0m=1967 �[2mthreshold_mib�[0m=1740
|
||||
�[2m· stop-instances i-0c615e5dd74050f53 (end of 7b)�[0m
|
||||
lifecycle_test.go:206: �[2m· stop-instances i-0c615e5dd74050f53 (end of 7b)�[0m
|
||||
lifecycle_test.go:211: instance i-0c615e5dd74050f53 reached state stopped]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase7c_pre_Reboot" classname="" time="83.480">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 7c-pre — Reboot Running Instance ━━�[0m
|
||||
�[2m· start-instances i-0c615e5dd74050f53�[0m
|
||||
lifecycle_test.go:226: �[2m· start-instances i-0c615e5dd74050f53�[0m
|
||||
lifecycle_test.go:231: instance i-0c615e5dd74050f53 reached state running
|
||||
�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
|
||||
lifecycle_test.go:236: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
|
||||
�[2mpre_reboot_private_ip�[0m=172.31.0.4
|
||||
lifecycle_test.go:246: �[2mpre_reboot_private_ip�[0m=172.31.0.4
|
||||
�[2m· reboot-instances i-0c615e5dd74050f53�[0m
|
||||
lifecycle_test.go:248: �[2m· reboot-instances i-0c615e5dd74050f53�[0m
|
||||
�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
|
||||
lifecycle_test.go:279: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
|
||||
�[2m· ssh uptime�[0m
|
||||
lifecycle_test.go:282: �[2m· ssh uptime�[0m
|
||||
�[2muptime_secs�[0m=25
|
||||
lifecycle_test.go:288: �[2muptime_secs�[0m=25]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase7c_RunInstancesMultiCount" classname="" time="47.560">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 7c — RunInstances with MinCount/MaxCount > 1 ━━�[0m
|
||||
�[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.small count=2�[0m
|
||||
lifecycle_test.go:309: �[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.small count=2�[0m
|
||||
�[2msibling_1�[0m=i-9bfadd8ebd378f846 �[2msibling_2�[0m=i-448a9e2bf45c58658
|
||||
lifecycle_test.go:327: �[2msibling_1�[0m=i-9bfadd8ebd378f846 �[2msibling_2�[0m=i-448a9e2bf45c58658
|
||||
lifecycle_test.go:342: instance i-9bfadd8ebd378f846 reached state running
|
||||
lifecycle_test.go:342: instance i-448a9e2bf45c58658 reached state running
|
||||
�[2m· terminate-instances i-9bfadd8ebd378f846 i-448a9e2bf45c58658�[0m
|
||||
lifecycle_test.go:345: �[2m· terminate-instances i-9bfadd8ebd378f846 i-448a9e2bf45c58658�[0m
|
||||
lifecycle_test.go:352: instance i-9bfadd8ebd378f846 reached state terminated
|
||||
lifecycle_test.go:352: instance i-448a9e2bf45c58658 reached state terminated]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths" classname="" time="0.000">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 8 — Negative / Error Path Tests ━━�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8a_InvalidAMIIDMalformed" classname="" time="0.010">
|
||||
<system-out><![CDATA[�[2m· run-instances image-id=notanami�[0m
|
||||
negative_test.go:55: �[2m· run-instances image-id=notanami�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8b_InvalidInstanceType" classname="" time="0.210">
|
||||
<system-out><![CDATA[�[2m· run-instances instance-type=x99.superlarge�[0m
|
||||
negative_test.go:70: �[2m· run-instances instance-type=x99.superlarge�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8c_VolumeInUse" classname="" time="0.010">
|
||||
<system-out><![CDATA[�[2m· attach-volume vol-2230138124fb51ee2 (root, in-use)�[0m
|
||||
negative_test.go:85: �[2m· attach-volume vol-2230138124fb51ee2 (root, in-use)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8d_DetachRootForbidden" classname="" time="0.010">
|
||||
<system-out><![CDATA[�[2m· detach-volume vol-2230138124fb51ee2 (root)�[0m
|
||||
negative_test.go:97: �[2m· detach-volume vol-2230138124fb51ee2 (root)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8e_DeleteSnapshotNotFound" classname="" time="0.030">
|
||||
<system-out><![CDATA[�[2m· delete-snapshot snap-nonexistent000000�[0m
|
||||
negative_test.go:108: �[2m· delete-snapshot snap-nonexistent000000�[0m
|
||||
�[2m· describe-volumes vol-0000000000000dead�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8f_InvalidActionRawHTTP" classname="" time="0.040">
|
||||
<system-out><![CDATA[�[2m· POST Action=DescribeFakeThings�[0m
|
||||
negative_test.go:122: �[2m· POST Action=DescribeFakeThings�[0m
|
||||
negative_test.go:125: �[2mstatus�[0m=400 �[2mcode�[0m=InvalidAction]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8g_InvalidAMIIDNotFound" classname="" time="0.240">
|
||||
<system-out><![CDATA[�[2m· reboot-instances i-nonexistent�[0m
|
||||
�[2m· run-instances image-id=ami-0000000000000dead�[0m
|
||||
negative_test.go:138: �[2m· run-instances image-id=ami-0000000000000dead�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8h_InvalidKeyPairNotFound" classname="" time="0.260">
|
||||
<system-out><![CDATA[�[2m· run-instances key-name=nonexistent-key-xyz�[0m
|
||||
negative_test.go:153: �[2m· run-instances key-name=nonexistent-key-xyz�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8i_DeleteVolumeNotFound" classname="" time="0.040">
|
||||
<system-out><![CDATA[�[2m· delete-volume vol-0000000000000dead�[0m
|
||||
negative_test.go:168: �[2m· delete-volume vol-0000000000000dead�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8j_CreateKeyPairDuplicate" classname="" time="0.040">
|
||||
<system-out><![CDATA[ negative_test.go:178: �[2m· create-key-pair test-key-1 (duplicate)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8k_ImportKeyPairDuplicate" classname="" time="0.040">
|
||||
<system-out><![CDATA[�[2m· import-key-pair test-key-1 (duplicate)�[0m
|
||||
negative_test.go:198: �[2m· import-key-pair test-key-1 (duplicate)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8l_InvalidKeyFormat" classname="" time="0.040">
|
||||
<system-out><![CDATA[ negative_test.go:210: �[2m· import-key-pair bad-format-key (bad material)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8m_DescribeVolumesNotFound" classname="" time="0.040">
|
||||
<system-out><![CDATA[ negative_test.go:230: �[2m· describe-volumes vol-0000000000000dead�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8n_DescribeImagesNotFound" classname="" time="0.040">
|
||||
<system-out><![CDATA[�[2m· describe-images ami-0000000000000dead�[0m
|
||||
negative_test.go:240: �[2m· describe-images ami-0000000000000dead�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8o_CreateImageDuplicateName" classname="" time="0.560">
|
||||
<system-out><![CDATA[�[2m· create-key-pair test-key-1 (duplicate)�[0m
|
||||
�[2m· create-image name=e2e-custom-ami (duplicate)�[0m
|
||||
�[2m· import-key-pair bad-format-key (bad material)�[0m
|
||||
negative_test.go:252: �[2m· create-image name=e2e-custom-ami (duplicate)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8p_DeleteKeyPairIdempotent" classname="" time="0.020">
|
||||
<system-out><![CDATA[�[2m· delete-key-pair nonexistent-key-99999 (idempotent)�[0m
|
||||
negative_test.go:276: �[2m· delete-key-pair nonexistent-key-99999 (idempotent)�[0m
|
||||
�[2mstatus�[0m=400 �[2mcode�[0m=InvalidAction]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8q_ModifyAttributeOnRunning" classname="" time="0.010">
|
||||
<system-out><![CDATA[�[2m· modify-instance-attribute i-0c615e5dd74050f53 (running)�[0m
|
||||
negative_test.go:289: �[2m· modify-instance-attribute i-0c615e5dd74050f53 (running)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8r_RebootInstanceNotFound" classname="" time="0.040">
|
||||
<system-out><![CDATA[ negative_test.go:302: �[2m· reboot-instances i-nonexistent�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM1_UserCRUD" classname="" time="0.070">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ IAM Phase 1 — User CRUD ━━�[0m
|
||||
�[2m· list-users (root auth sanity)�[0m
|
||||
iam_test.go:109: �[2m· list-users (root auth sanity)�[0m
|
||||
�[2m· create-user "alice"�[0m
|
||||
iam_test.go:114: �[2m· create-user "alice"�[0m
|
||||
�[2muser�[0m=alice �[2marn�[0m=arn:aws:iam::000000000001:user/alice
|
||||
iam_test.go:120: �[2muser�[0m=alice �[2marn�[0m=arn:aws:iam::000000000001:user/alice
|
||||
�[2m· create-user "bob" path="/engineering/"�[0m
|
||||
iam_test.go:122: �[2m· create-user "bob" path="/engineering/"�[0m
|
||||
�[2m· create-user "alice" again (expect EntityAlreadyExists)�[0m
|
||||
iam_test.go:131: �[2m· create-user "alice" again (expect EntityAlreadyExists)�[0m
|
||||
�[2m· get-user "alice"�[0m
|
||||
iam_test.go:137: �[2m· get-user "alice"�[0m
|
||||
�[2m· get-user nonexistent (expect NoSuchEntity)�[0m
|
||||
iam_test.go:142: �[2m· get-user nonexistent (expect NoSuchEntity)�[0m
|
||||
�[2m· list-users (>=3 root+alice+bob)�[0m
|
||||
iam_test.go:148: �[2m· list-users (>=3 root+alice+bob)�[0m
|
||||
�[2m· list-users --path-prefix "/engineering/" (expect 1)�[0m
|
||||
iam_test.go:154: �[2m· list-users --path-prefix "/engineering/" (expect 1)�[0m
|
||||
iam_test.go:193: �[2m· create-user "iam-phase1-ephemeral" (delete idempotency probe)�[0m
|
||||
�[2m· delete-user "iam-phase1-ephemeral"�[0m
|
||||
iam_test.go:197: �[2m· delete-user "iam-phase1-ephemeral"�[0m
|
||||
�[2m· delete-user "iam-phase1-ephemeral" again (expect NoSuchEntity)�[0m
|
||||
iam_test.go:201: �[2m· delete-user "iam-phase1-ephemeral" again (expect NoSuchEntity)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM1_UserCRUD/UpdateUser" classname="" time="0.000">
|
||||
<skipped message="Skipped"><![CDATA[ iam_test.go:165: daemon UpdateUser handler not implemented — InvalidAction
|
||||
�[2m· create-user "iam-phase1-ephemeral" (delete idempotency probe)�[0m]]></skipped>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM2_AccessKeyLifecycle" classname="" time="0.040">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ IAM Phase 2 — Access Key Lifecycle ━━�[0m
|
||||
�[2m· create-access-key user=alice (key 1)�[0m
|
||||
iam_test.go:218: �[2m· create-access-key user=alice (key 1)�[0m
|
||||
�[2mkey1�[0m=AKIA4984AE57CF8D721A879A
|
||||
iam_test.go:227: �[2mkey1�[0m=AKIA4984AE57CF8D721A879A
|
||||
�[2m· create-access-key user=alice (key 2)�[0m
|
||||
iam_test.go:229: �[2m· create-access-key user=alice (key 2)�[0m
|
||||
�[2mkey2�[0m=AKIAA1CD38D4F0E573216E11
|
||||
iam_test.go:236: �[2mkey2�[0m=AKIAA1CD38D4F0E573216E11
|
||||
�[2m· create-access-key user=alice key 3 (expect LimitExceeded)�[0m
|
||||
iam_test.go:238: �[2m· create-access-key user=alice key 3 (expect LimitExceeded)�[0m
|
||||
�[2m· create-access-key user=ghost (expect NoSuchEntity)�[0m
|
||||
iam_test.go:246: �[2m· create-access-key user=ghost (expect NoSuchEntity)�[0m
|
||||
�[2m· list-access-keys user=alice (expect 2)�[0m
|
||||
iam_test.go:254: �[2m· list-access-keys user=alice (expect 2)�[0m
|
||||
�[2m· list-access-keys user=bob (expect 0)�[0m
|
||||
iam_test.go:261: �[2m· list-access-keys user=bob (expect 0)�[0m
|
||||
�[2m· update-access-key AKIA4984AE57CF8D721A879A -> Inactive�[0m
|
||||
iam_test.go:268: �[2m· update-access-key AKIA4984AE57CF8D721A879A -> Inactive�[0m
|
||||
�[2m· update-access-key AKIA4984AE57CF8D721A879A -> Active�[0m
|
||||
iam_test.go:279: �[2m· update-access-key AKIA4984AE57CF8D721A879A -> Active�[0m
|
||||
�[2m· delete-access-key AKIAA1CD38D4F0E573216E11 (key 2)�[0m
|
||||
iam_test.go:287: �[2m· delete-access-key AKIAA1CD38D4F0E573216E11 (key 2)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM3_UserAuthentication" classname="" time="0.000">
|
||||
<skipped message="Skipped"><![CDATA[
|
||||
�[1m�[36m━━ IAM Phase 3 — User Authentication ━━�[0m
|
||||
iam_test.go:311: daemon IAM signature/principal lookup not implemented — mulga-siv-100]]></skipped>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM4_PolicyCRUD" classname="" time="0.040">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ IAM Phase 4 — Policy CRUD ━━�[0m
|
||||
�[2m· create-policy EC2ReadOnly�[0m
|
||||
iam_test.go:384: �[2m· create-policy EC2ReadOnly�[0m
|
||||
�[2mpolicy�[0m=EC2ReadOnly �[2marn�[0m=arn:aws:iam::000000000001:policy/EC2ReadOnly �[2maccount�[0m=000000000001
|
||||
iam_test.go:393: �[2mpolicy�[0m=EC2ReadOnly �[2marn�[0m=arn:aws:iam::000000000001:policy/EC2ReadOnly �[2maccount�[0m=000000000001
|
||||
iam_test.go:440: �[2m· create-policy duplicate (expect EntityAlreadyExists)�[0m
|
||||
�[2m· create-policy malformed (expect MalformedPolicyDocument)�[0m
|
||||
iam_test.go:449: �[2m· create-policy malformed (expect MalformedPolicyDocument)�[0m
|
||||
�[2m· get-policy EC2ReadOnly�[0m
|
||||
iam_test.go:458: �[2m· get-policy EC2ReadOnly�[0m
|
||||
�[2m· get-policy nonexistent (expect NoSuchEntity)�[0m
|
||||
iam_test.go:463: �[2m· get-policy nonexistent (expect NoSuchEntity)�[0m
|
||||
�[2m· get-policy-version EC2ReadOnly v1�[0m
|
||||
iam_test.go:470: �[2m· get-policy-version EC2ReadOnly v1�[0m
|
||||
iam_test.go:519: �[2m· list-policies (>=5)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel" classname="" time="0.000"></testcase>
|
||||
<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/FullAdmin" classname="" time="0.010">
|
||||
<system-out><![CDATA[�[2m· create-policy FullAdmin path=/admin/�[0m
|
||||
iam_test.go:402: �[2m· create-policy FullAdmin path=/admin/�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/DenyTerminate" classname="" time="0.010">
|
||||
<system-out><![CDATA[�[2m· create-policy DenyTerminate (mixed Allow+Deny)�[0m
|
||||
iam_test.go:413: �[2m· create-policy DenyTerminate (mixed Allow+Deny)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/IAMReadOnly" classname="" time="0.020">
|
||||
<system-out><![CDATA[�[2m· create-policy IAMReadOnly�[0m
|
||||
iam_test.go:422: �[2m· create-policy IAMReadOnly�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/EC2DescribeAll" classname="" time="0.020">
|
||||
<system-out><![CDATA[�[2m· create-policy EC2DescribeAll (wildcard)�[0m
|
||||
iam_test.go:431: �[2m· create-policy EC2DescribeAll (wildcard)�[0m
|
||||
�[2m· create-policy duplicate (expect EntityAlreadyExists)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM4_PolicyCRUD/PolicyVersionLifecycle" classname="" time="0.000">
|
||||
<skipped message="Skipped"><![CDATA[ iam_test.go:484: daemon CreatePolicyVersion handler not implemented — InvalidAction
|
||||
�[2m· list-policies (>=5)�[0m]]></skipped>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM5_PolicyAttachmentEnforcement" classname="" time="0.000">
|
||||
<skipped message="Skipped"><![CDATA[
|
||||
�[1m�[36m━━ IAM Phase 5 — Policy Attachment & Enforcement ━━�[0m
|
||||
iam_test.go:535: daemon IAM scoped-credential enforcement not implemented — mulga-siv-100]]></skipped>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM6_PolicyLifecycle" classname="" time="0.000">
|
||||
<skipped message="Skipped"><![CDATA[
|
||||
�[1m�[36m━━ IAM Phase 6 — Policy Lifecycle (Detach & Delete) ━━�[0m
|
||||
iam_test.go:718: daemon DetachUserPolicy not implemented — mulga-siv-100]]></skipped>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/IAM7_Cleanup" classname="" time="0.050">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ IAM Phase 7 — Cleanup ━━�[0m
|
||||
�[2m· cleanup user alice�[0m
|
||||
iam_test.go:770: �[2m· cleanup user alice�[0m
|
||||
�[2m· cleanup user bob�[0m
|
||||
iam_test.go:770: �[2m· cleanup user bob�[0m
|
||||
�[2m· cleanup user charlie�[0m
|
||||
iam_test.go:770: �[2m· cleanup user charlie�[0m
|
||||
�[2mremaining_users�[0m=1 �[2mnames�[0m=[admin]
|
||||
iam_test.go:779: �[2mremaining_users�[0m=1 �[2mnames�[0m=[admin]
|
||||
�[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2ReadOnly�[0m
|
||||
iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2ReadOnly�[0m
|
||||
�[2m· cleanup policy arn:aws:iam::000000000001:policy/admin/FullAdmin�[0m
|
||||
iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/admin/FullAdmin�[0m
|
||||
�[2m· cleanup policy arn:aws:iam::000000000001:policy/IAMReadOnly�[0m
|
||||
iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/IAMReadOnly�[0m
|
||||
�[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2DescribeAll�[0m
|
||||
iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2DescribeAll�[0m
|
||||
�[2m· cleanup policy arn:aws:iam::000000000001:policy/DenyTerminate�[0m
|
||||
iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/DenyTerminate�[0m
|
||||
�[2mremaining_policies�[0m=1 �[2mnames�[0m=[AdministratorAccess]
|
||||
iam_test.go:812: �[2mremaining_policies�[0m=1 �[2mnames�[0m=[AdministratorAccess]]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping" classname="" time="114.890">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 8 (acct) — EC2 Account Scoping ━━�[0m
|
||||
account_test.go:75: �[2m· Phase 8 acct cleanup (deferred)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step1_AccountSetup" classname="" time="1.220">
|
||||
<system-out><![CDATA[�[2m· create account 'Team Alpha'�[0m
|
||||
account_test.go:206: �[2m· create account 'Team Alpha'�[0m
|
||||
�[2malpha_account�[0m=000000000002 �[2malpha_key�[0m=AKIA374CBBB2A15D3E2EE822
|
||||
account_test.go:211: �[2malpha_account�[0m=000000000002 �[2malpha_key�[0m=AKIA374CBBB2A15D3E2EE822
|
||||
�[2m· create account 'Team Beta'�[0m
|
||||
account_test.go:213: �[2m· create account 'Team Beta'�[0m
|
||||
�[2mbeta_account�[0m=000000000003 �[2mbeta_key�[0m=AKIA0AA0B2845E3BD5A26D01
|
||||
account_test.go:218: �[2mbeta_account�[0m=000000000003 �[2mbeta_key�[0m=AKIA0AA0B2845E3BD5A26D01]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step2_InstanceScoping" classname="" time="52.380">
|
||||
<system-out><![CDATA[�[2m· create per-account key pairs�[0m
|
||||
account_test.go:242: �[2m· create per-account key pairs�[0m
|
||||
�[2m· alpha run-instances�[0m
|
||||
account_test.go:250: �[2m· alpha run-instances�[0m
|
||||
�[2malpha_instance�[0m=i-a3072c2387630b8ee
|
||||
account_test.go:262: �[2malpha_instance�[0m=i-a3072c2387630b8ee
|
||||
�[2m· beta run-instances�[0m
|
||||
account_test.go:264: �[2m· beta run-instances�[0m
|
||||
�[2mbeta_instance�[0m=i-ad32cbb0afaee5eee
|
||||
account_test.go:276: �[2mbeta_instance�[0m=i-ad32cbb0afaee5eee
|
||||
account_test.go:278: instance i-a3072c2387630b8ee reached state running
|
||||
account_test.go:279: instance i-ad32cbb0afaee5eee reached state running
|
||||
�[2m· alpha sees only own instances�[0m
|
||||
account_test.go:281: �[2m· alpha sees only own instances�[0m
|
||||
�[2m· beta sees only own instances�[0m
|
||||
account_test.go:285: �[2m· beta sees only own instances�[0m
|
||||
�[2m· alpha OwnerId matches alpha account�[0m
|
||||
account_test.go:289: �[2m· alpha OwnerId matches alpha account�[0m
|
||||
�[2m· cross-account stop alpha->beta blocked�[0m
|
||||
account_test.go:296: �[2m· cross-account stop alpha->beta blocked�[0m
|
||||
�[2m· cross-account terminate beta->alpha blocked�[0m
|
||||
account_test.go:304: �[2m· cross-account terminate beta->alpha blocked�[0m
|
||||
�[2m· cross-account reboot alpha->beta blocked�[0m
|
||||
account_test.go:312: �[2m· cross-account reboot alpha->beta blocked�[0m
|
||||
�[2m· stop alpha instance for cross-account start/modify/console tests�[0m
|
||||
account_test.go:321: �[2m· stop alpha instance for cross-account start/modify/console tests�[0m
|
||||
account_test.go:326: instance i-a3072c2387630b8ee reached state stopped
|
||||
�[2m· cross-account start beta->alpha blocked�[0m
|
||||
account_test.go:328: �[2m· cross-account start beta->alpha blocked�[0m
|
||||
�[2m· cross-account modify-instance-attribute beta->alpha blocked�[0m
|
||||
account_test.go:336: �[2m· cross-account modify-instance-attribute beta->alpha blocked�[0m
|
||||
�[2m· cross-account get-console-output beta->alpha blocked�[0m
|
||||
account_test.go:345: �[2m· cross-account get-console-output beta->alpha blocked�[0m
|
||||
�[2m· restart alpha instance�[0m
|
||||
account_test.go:354: �[2m· restart alpha instance�[0m
|
||||
account_test.go:359: instance i-a3072c2387630b8ee reached state running]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step3_VolumeScoping" classname="" time="24.140">
|
||||
<system-out><![CDATA[�[2m· alpha create-volume�[0m
|
||||
account_test.go:366: �[2m· alpha create-volume�[0m
|
||||
�[2malpha_vol�[0m=vol-ceef82652369780af
|
||||
account_test.go:375: �[2malpha_vol�[0m=vol-ceef82652369780af
|
||||
�[2m· beta create-volume�[0m
|
||||
account_test.go:377: �[2m· beta create-volume�[0m
|
||||
�[2mbeta_vol�[0m=vol-08024f32d23d9e18c
|
||||
account_test.go:386: �[2mbeta_vol�[0m=vol-08024f32d23d9e18c
|
||||
account_test.go:388: volume vol-ceef82652369780af reached state available
|
||||
account_test.go:389: volume vol-08024f32d23d9e18c reached state available
|
||||
�[2m· alpha sees only own volumes�[0m
|
||||
account_test.go:391: �[2m· alpha sees only own volumes�[0m
|
||||
�[2m· cross-account describe-volume by id blocked�[0m
|
||||
account_test.go:395: �[2m· cross-account describe-volume by id blocked�[0m
|
||||
�[2m· cross-account delete-volume blocked�[0m
|
||||
account_test.go:403: �[2m· cross-account delete-volume blocked�[0m
|
||||
�[2m· cross-account attach-volume (beta attaches alpha's vol) blocked�[0m
|
||||
account_test.go:411: �[2m· cross-account attach-volume (beta attaches alpha's vol) blocked�[0m
|
||||
�[2m· alpha attach-volume to own instance�[0m
|
||||
account_test.go:422: �[2m· alpha attach-volume to own instance�[0m
|
||||
account_test.go:429: volume vol-ceef82652369780af reached state in-use
|
||||
�[2m· cross-account detach-volume blocked�[0m
|
||||
account_test.go:431: �[2m· cross-account detach-volume blocked�[0m
|
||||
�[2m· cross-account modify-volume blocked�[0m
|
||||
account_test.go:439: �[2m· cross-account modify-volume blocked�[0m
|
||||
�[2m· alpha detach own volume�[0m
|
||||
account_test.go:449: �[2m· alpha detach own volume�[0m
|
||||
account_test.go:454: volume vol-ceef82652369780af reached state available]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step4_KeyPairScoping" classname="" time="0.370">
|
||||
<system-out><![CDATA[�[2m· alpha create-key-pair alpha-key�[0m
|
||||
account_test.go:466: �[2m· alpha create-key-pair alpha-key�[0m
|
||||
�[2m· beta create-key-pair beta-key�[0m
|
||||
account_test.go:473: �[2m· beta create-key-pair beta-key�[0m
|
||||
�[2m· alpha sees only own keys�[0m
|
||||
account_test.go:478: �[2m· alpha sees only own keys�[0m
|
||||
�[2m· namespace isolation: shared-name in both accounts�[0m
|
||||
account_test.go:483: �[2m· namespace isolation: shared-name in both accounts�[0m
|
||||
�[2m· beta deletes alpha-key (idempotent, no cross-account effect)�[0m
|
||||
account_test.go:497: �[2m· beta deletes alpha-key (idempotent, no cross-account effect)�[0m
|
||||
�[2m· alpha import-key-pair imported-key�[0m
|
||||
account_test.go:503: �[2m· alpha import-key-pair imported-key�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step5_SnapshotScoping" classname="" time="0.060">
|
||||
<system-out><![CDATA[�[2m· alpha create-snapshot�[0m
|
||||
account_test.go:522: �[2m· alpha create-snapshot�[0m
|
||||
�[2malpha_snap�[0m=snap-2a47896f4304d805c
|
||||
account_test.go:530: �[2malpha_snap�[0m=snap-2a47896f4304d805c
|
||||
�[2m· beta create-snapshot�[0m
|
||||
account_test.go:532: �[2m· beta create-snapshot�[0m
|
||||
�[2mbeta_snap�[0m=snap-bf5744e78bd7c6908
|
||||
account_test.go:540: �[2mbeta_snap�[0m=snap-bf5744e78bd7c6908
|
||||
�[2m· alpha sees only own snapshots (owner-ids=self)�[0m
|
||||
account_test.go:542: �[2m· alpha sees only own snapshots (owner-ids=self)�[0m
|
||||
�[2m· cross-account delete-snapshot blocked (UnauthorizedOperation)�[0m
|
||||
account_test.go:560: �[2m· cross-account delete-snapshot blocked (UnauthorizedOperation)�[0m
|
||||
�[2m· cross-account create-snapshot of alpha's volume blocked�[0m
|
||||
account_test.go:568: �[2m· cross-account create-snapshot of alpha's volume blocked�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step6_VPCSubnetScoping" classname="" time="0.030">
|
||||
<system-out><![CDATA[�[2m· alpha create-vpc 10.0.0.0/16�[0m
|
||||
account_test.go:582: �[2m· alpha create-vpc 10.0.0.0/16�[0m
|
||||
�[2malpha_vpc�[0m=vpc-7f60df5e87183fb59
|
||||
account_test.go:589: �[2malpha_vpc�[0m=vpc-7f60df5e87183fb59
|
||||
�[2m· beta create-vpc 10.0.0.0/16 (same CIDR, no conflict)�[0m
|
||||
account_test.go:591: �[2m· beta create-vpc 10.0.0.0/16 (same CIDR, no conflict)�[0m
|
||||
�[2mbeta_vpc�[0m=vpc-7e8c859d3985c8218
|
||||
account_test.go:598: �[2mbeta_vpc�[0m=vpc-7e8c859d3985c8218
|
||||
�[2m· alpha describe-vpcs isolation�[0m
|
||||
account_test.go:600: �[2m· alpha describe-vpcs isolation�[0m
|
||||
�[2m· cross-account describe-vpc by id blocked�[0m
|
||||
account_test.go:605: �[2m· cross-account describe-vpc by id blocked�[0m
|
||||
�[2m· cross-account delete-vpc blocked�[0m
|
||||
account_test.go:613: �[2m· cross-account delete-vpc blocked�[0m
|
||||
�[2m· alpha create-subnet 10.0.1.0/24�[0m
|
||||
account_test.go:621: �[2m· alpha create-subnet 10.0.1.0/24�[0m
|
||||
�[2m· beta create-subnet 10.0.1.0/24�[0m
|
||||
account_test.go:630: �[2m· beta create-subnet 10.0.1.0/24�[0m
|
||||
�[2m· alpha describe-subnets isolation�[0m
|
||||
account_test.go:639: �[2m· alpha describe-subnets isolation�[0m
|
||||
�[2m· cross-account create-subnet in other VPC blocked�[0m
|
||||
account_test.go:648: �[2m· cross-account create-subnet in other VPC blocked�[0m
|
||||
�[2m· cross-account delete-subnet blocked�[0m
|
||||
account_test.go:657: �[2m· cross-account delete-subnet blocked�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step7_IGWEIGWScoping" classname="" time="0.030">
|
||||
<system-out><![CDATA[�[2m· alpha create-internet-gateway�[0m
|
||||
account_test.go:673: �[2m· alpha create-internet-gateway�[0m
|
||||
�[2malpha_igw�[0m=igw-ee966e3e923caf630
|
||||
account_test.go:678: �[2malpha_igw�[0m=igw-ee966e3e923caf630
|
||||
�[2m· beta create-internet-gateway�[0m
|
||||
account_test.go:680: �[2m· beta create-internet-gateway�[0m
|
||||
�[2mbeta_igw�[0m=igw-b46e66f2065889bb9
|
||||
account_test.go:685: �[2mbeta_igw�[0m=igw-b46e66f2065889bb9
|
||||
�[2m· alpha describe-internet-gateways isolation�[0m
|
||||
account_test.go:687: �[2m· alpha describe-internet-gateways isolation�[0m
|
||||
�[2m· cross-account describe IGW by id blocked�[0m
|
||||
account_test.go:696: �[2m· cross-account describe IGW by id blocked�[0m
|
||||
�[2m· cross-account delete IGW blocked�[0m
|
||||
account_test.go:704: �[2m· cross-account delete IGW blocked�[0m
|
||||
�[2m· cross-account attach IGW (alpha attaches beta's IGW) blocked�[0m
|
||||
account_test.go:712: �[2m· cross-account attach IGW (alpha attaches beta's IGW) blocked�[0m
|
||||
�[2m· alpha attach own IGW to own VPC�[0m
|
||||
account_test.go:722: �[2m· alpha attach own IGW to own VPC�[0m
|
||||
�[2m· cross-account detach IGW blocked�[0m
|
||||
account_test.go:729: �[2m· cross-account detach IGW blocked�[0m
|
||||
�[2m· alpha create-egress-only-internet-gateway�[0m
|
||||
account_test.go:739: �[2m· alpha create-egress-only-internet-gateway�[0m
|
||||
�[2malpha_eigw�[0m=eigw-51ba82da927e49393
|
||||
account_test.go:746: �[2malpha_eigw�[0m=eigw-51ba82da927e49393
|
||||
�[2m· beta create-egress-only-internet-gateway�[0m
|
||||
account_test.go:748: �[2m· beta create-egress-only-internet-gateway�[0m
|
||||
�[2mbeta_eigw�[0m=eigw-e88d642e78ab4fe5a
|
||||
account_test.go:755: �[2mbeta_eigw�[0m=eigw-e88d642e78ab4fe5a
|
||||
�[2m· alpha describe-eigws isolation�[0m
|
||||
account_test.go:757: �[2m· alpha describe-eigws isolation�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step8_AccountSettings" classname="" time="0.010">
|
||||
<system-out><![CDATA[�[2m· alpha enable-ebs-encryption-by-default�[0m
|
||||
account_test.go:794: �[2m· alpha enable-ebs-encryption-by-default�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step9_GlobalResources" classname="" time="0.010">
|
||||
<system-out><![CDATA[�[2m· describe-regions identical across accounts�[0m
|
||||
account_test.go:829: �[2m· describe-regions identical across accounts�[0m
|
||||
�[2m· describe-availability-zones identical across accounts�[0m
|
||||
account_test.go:834: �[2m· describe-availability-zones identical across accounts�[0m
|
||||
�[2m· describe-instance-types identical across accounts�[0m
|
||||
account_test.go:839: �[2m· describe-instance-types identical across accounts�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step10_EdgeCases" classname="" time="1.200">
|
||||
<system-out><![CDATA[�[2m· create empty Gamma account�[0m
|
||||
account_test.go:849: �[2m· create empty Gamma account�[0m
|
||||
�[2mgamma_account�[0m=000000000004
|
||||
account_test.go:852: �[2mgamma_account�[0m=000000000004
|
||||
�[2m· root create-key-pair root-scoping-key�[0m
|
||||
account_test.go:879: �[2m· root create-key-pair root-scoping-key�[0m
|
||||
�[2m· non-existent volume id -> InvalidVolume.NotFound�[0m
|
||||
account_test.go:895: �[2m· non-existent volume id -> InvalidVolume.NotFound�[0m
|
||||
�[2m· non-existent snapshot id -> InvalidSnapshot.NotFound�[0m
|
||||
account_test.go:903: �[2m· non-existent snapshot id -> InvalidSnapshot.NotFound�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step11_Cleanup" classname="" time="32.440">
|
||||
<system-out><![CDATA[�[2m· terminate alpha instance�[0m
|
||||
account_test.go:920: �[2m· terminate alpha instance�[0m
|
||||
�[2m· terminate beta instance�[0m
|
||||
account_test.go:926: �[2m· terminate beta instance�[0m
|
||||
�[2m· delete snapshots�[0m
|
||||
account_test.go:939: �[2m· delete snapshots�[0m
|
||||
�[2m· delete volumes�[0m
|
||||
account_test.go:955: �[2m· delete volumes�[0m
|
||||
�[2m· delete key pairs�[0m
|
||||
account_test.go:969: �[2m· delete key pairs�[0m
|
||||
�[2m· delete EIGWs�[0m
|
||||
account_test.go:979: �[2m· delete EIGWs�[0m
|
||||
�[2m· detach + delete IGWs�[0m
|
||||
account_test.go:993: �[2m· detach + delete IGWs�[0m
|
||||
�[2m· delete subnets�[0m
|
||||
account_test.go:1013: �[2m· delete subnets�[0m
|
||||
�[2m· delete VPCs�[0m
|
||||
account_test.go:1027: �[2m· delete VPCs�[0m
|
||||
�[2m· Phase 8 acct cleanup (deferred)�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8b_VPCSubnetE2E" classname="" time="200.480">
|
||||
<failure message="Failed"><![CDATA[
|
||||
�[1m�[36m━━ Phase 8b — VPC Public/Private Subnet E2E ━━�[0m
|
||||
�[2m· create-vpc 10.99.0.0/16�[0m
|
||||
vpc_test.go:56: �[2m· create-vpc 10.99.0.0/16�[0m
|
||||
�[2mvpc�[0m=vpc-39eff985799c43069 �[2mcidr�[0m=10.99.0.0/16
|
||||
vpc_test.go:67: �[2mvpc�[0m=vpc-39eff985799c43069 �[2mcidr�[0m=10.99.0.0/16
|
||||
�[2m· create-subnet 10.99.1.0/24 (public)�[0m
|
||||
vpc_test.go:70: �[2m· create-subnet 10.99.1.0/24 (public)�[0m
|
||||
�[2m· modify-subnet-attribute MapPublicIpOnLaunch=true subnet-22aba054e10588254�[0m
|
||||
vpc_test.go:87: �[2m· modify-subnet-attribute MapPublicIpOnLaunch=true subnet-22aba054e10588254�[0m
|
||||
�[2mpub_subnet�[0m=subnet-22aba054e10588254 �[2mmap_public_ip�[0m=true
|
||||
vpc_test.go:93: �[2mpub_subnet�[0m=subnet-22aba054e10588254 �[2mmap_public_ip�[0m=true
|
||||
�[2m· create-subnet 10.99.2.0/24 (private)�[0m
|
||||
vpc_test.go:96: �[2m· create-subnet 10.99.2.0/24 (private)�[0m
|
||||
�[2mpriv_subnet�[0m=subnet-f53aeddb73c134696 �[2mmap_public_ip�[0m=false
|
||||
vpc_test.go:111: �[2mpriv_subnet�[0m=subnet-f53aeddb73c134696 �[2mmap_public_ip�[0m=false
|
||||
�[2m· create-internet-gateway�[0m
|
||||
vpc_test.go:114: �[2m· create-internet-gateway�[0m
|
||||
�[2m· attach-internet-gateway igw-1520b5a34480da982 -> vpc-39eff985799c43069�[0m
|
||||
vpc_test.go:133: �[2m· attach-internet-gateway igw-1520b5a34480da982 -> vpc-39eff985799c43069�[0m
|
||||
�[2migw�[0m=igw-1520b5a34480da982
|
||||
vpc_test.go:139: �[2migw�[0m=igw-1520b5a34480da982
|
||||
�[2m· create-route-table (vpc=vpc-39eff985799c43069)�[0m
|
||||
vpc_test.go:142: �[2m· create-route-table (vpc=vpc-39eff985799c43069)�[0m
|
||||
�[2m· associate-route-table rtb-662df5e424cbdb60c <- subnet-22aba054e10588254�[0m
|
||||
vpc_test.go:159: �[2m· associate-route-table rtb-662df5e424cbdb60c <- subnet-22aba054e10588254�[0m
|
||||
�[2m· create-route 0.0.0.0/0 -> igw-1520b5a34480da982�[0m
|
||||
vpc_test.go:176: �[2m· create-route 0.0.0.0/0 -> igw-1520b5a34480da982�[0m
|
||||
�[2mrtb�[0m=rtb-662df5e424cbdb60c �[2massoc�[0m=rtbassoc-5510bf1dc6924eb0b �[2mroute�[0m=0.0.0.0/0->igw-1520b5a34480da982
|
||||
vpc_test.go:192: �[2mrtb�[0m=rtb-662df5e424cbdb60c �[2massoc�[0m=rtbassoc-5510bf1dc6924eb0b �[2mroute�[0m=0.0.0.0/0->igw-1520b5a34480da982
|
||||
�[2m· run-instances ami=ami-6f60f2b3bf04b5453 subnet=subnet-22aba054e10588254�[0m
|
||||
vpc_test.go:195: �[2m· run-instances ami=ami-6f60f2b3bf04b5453 subnet=subnet-22aba054e10588254�[0m
|
||||
�[2minstance�[0m=i-3177888cba3796d42
|
||||
vpc_test.go:216: �[2minstance�[0m=i-3177888cba3796d42
|
||||
vpc_test.go:218: instance i-3177888cba3796d42 reached state running
|
||||
�[2mpublic_ip�[0m=192.168.0.202 �[2mprivate_ip�[0m=10.99.1.4
|
||||
vpc_test.go:224: �[2mpublic_ip�[0m=192.168.0.202 �[2mprivate_ip�[0m=10.99.1.4
|
||||
�[2m· waiting for SSH handshake 192.168.0.202:22�[0m
|
||||
vpc_test.go:227: �[2m· waiting for SSH handshake 192.168.0.202:22�[0m
|
||||
vpc_test.go:227: Eventually: condition not met within 3m0s: [SSH handshake 192.168.0.202:22 never completed]]]></failure>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8c_RouteTableValidation" classname="" time="0.030">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 8c — Route Table Validation ━━�[0m
|
||||
�[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63
|
||||
vpc_test.go:292: �[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8c_RouteTableValidation/Step1_DefaultMainRouteTable" classname="" time="0.000">
|
||||
<system-out><![CDATA[�[2m· describe-route-tables filter=vpc-id,association.main�[0m
|
||||
vpc_test.go:297: �[2m· describe-route-tables filter=vpc-id,association.main�[0m
|
||||
�[2mmain_rtb�[0m=rtb-e42012569f92f7184
|
||||
vpc_test.go:309: �[2mmain_rtb�[0m=rtb-e42012569f92f7184
|
||||
�[2mlocal_route�[0m=172.31.0.0/16
|
||||
vpc_test.go:332: �[2mlocal_route�[0m=172.31.0.0/16
|
||||
�[2mdefault_route�[0m=0.0.0.0/0->igw-79c2e643660b4b7e3
|
||||
vpc_test.go:350: �[2mdefault_route�[0m=0.0.0.0/0->igw-79c2e643660b4b7e3]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8c_RouteTableValidation/Step2_CustomRouteTableLifecycle" classname="" time="0.020">
|
||||
<system-out><![CDATA[�[2m· create-route-table (vpc=vpc-531252db3b4e1fc08)�[0m
|
||||
vpc_test.go:363: �[2m· create-route-table (vpc=vpc-531252db3b4e1fc08)�[0m
|
||||
�[2mcustom_rtb�[0m=rtb-5e6c46936f3e44167
|
||||
vpc_test.go:379: �[2mcustom_rtb�[0m=rtb-5e6c46936f3e44167
|
||||
�[2mcustom_local_route�[0m=172.31.0.0/16
|
||||
vpc_test.go:396: �[2mcustom_local_route�[0m=172.31.0.0/16
|
||||
�[2mvpc_igw�[0m=igw-79c2e643660b4b7e3
|
||||
vpc_test.go:410: �[2mvpc_igw�[0m=igw-79c2e643660b4b7e3
|
||||
�[2m· create-route 0.0.0.0/0 -> igw-79c2e643660b4b7e3�[0m
|
||||
vpc_test.go:415: �[2m· create-route 0.0.0.0/0 -> igw-79c2e643660b4b7e3�[0m
|
||||
�[2m· associate-route-table rtb-5e6c46936f3e44167 <- subnet-cd62e4c1c3cb50c63�[0m
|
||||
vpc_test.go:434: �[2m· associate-route-table rtb-5e6c46936f3e44167 <- subnet-cd62e4c1c3cb50c63�[0m
|
||||
�[2mcustom_assoc�[0m=rtbassoc-da6377d37847d3d4a
|
||||
vpc_test.go:450: �[2mcustom_assoc�[0m=rtbassoc-da6377d37847d3d4a
|
||||
�[2m· disassociate-route-table rtbassoc-da6377d37847d3d4a�[0m
|
||||
vpc_test.go:452: �[2m· disassociate-route-table rtbassoc-da6377d37847d3d4a�[0m
|
||||
�[2m· delete-route 0.0.0.0/0�[0m
|
||||
vpc_test.go:460: �[2m· delete-route 0.0.0.0/0�[0m
|
||||
�[2m· delete-route-table rtb-5e6c46936f3e44167�[0m
|
||||
vpc_test.go:468: �[2m· delete-route-table rtb-5e6c46936f3e44167�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8c_RouteTableValidation/Step3_ErrorPaths" classname="" time="0.010">
|
||||
<system-out><![CDATA[�[2m· create-route-table (scratch for error paths)�[0m
|
||||
vpc_test.go:481: �[2m· create-route-table (scratch for error paths)�[0m
|
||||
�[2m· delete-route local=172.31.0.0/16 (must fail)�[0m
|
||||
vpc_test.go:517: �[2m· delete-route local=172.31.0.0/16 (must fail)�[0m
|
||||
�[2mdelete_local_err�[0m=InvalidParameterValue
|
||||
vpc_test.go:529: �[2mdelete_local_err�[0m=InvalidParameterValue
|
||||
�[2m· delete-route-table rtb-e42012569f92f7184 (main, must fail)�[0m
|
||||
vpc_test.go:533: �[2m· delete-route-table rtb-e42012569f92f7184 (main, must fail)�[0m
|
||||
�[2mdelete_main_err�[0m=DependencyViolation
|
||||
vpc_test.go:540: �[2mdelete_main_err�[0m=DependencyViolation]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8d_NATGateway" classname="" time="106.010">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 8d — NAT Gateway E2E ━━�[0m
|
||||
�[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2mpub_subnet�[0m=subnet-cd62e4c1c3cb50c63 �[2msg�[0m=sg-764a7fa2a39577204
|
||||
natgw_test.go:54: �[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2mpub_subnet�[0m=subnet-cd62e4c1c3cb50c63 �[2msg�[0m=sg-764a7fa2a39577204
|
||||
�[2m· create-subnet 172.31.16.0/20 (private)�[0m
|
||||
natgw_test.go:58: �[2m· create-subnet 172.31.16.0/20 (private)�[0m
|
||||
�[2mpriv_subnet�[0m=subnet-2360a40e28371e214
|
||||
natgw_test.go:71: �[2mpriv_subnet�[0m=subnet-2360a40e28371e214
|
||||
�[2m· run-instances bastion (default subnet)�[0m
|
||||
natgw_test.go:74: �[2m· run-instances bastion (default subnet)�[0m
|
||||
�[2m· run-instances private (private subnet)�[0m
|
||||
natgw_test.go:98: �[2m· run-instances private (private subnet)�[0m
|
||||
natgw_test.go:118: instance i-268402f0e28b5e477 reached state running
|
||||
natgw_test.go:119: instance i-db7f3ed6e4cdaa5aa reached state running
|
||||
�[2mbastion�[0m=i-268402f0e28b5e477 �[2mbastion_ip�[0m=192.168.0.202 �[2mprivate�[0m=i-db7f3ed6e4cdaa5aa �[2mprivate_ip�[0m=172.31.16.4
|
||||
natgw_test.go:128: �[2mbastion�[0m=i-268402f0e28b5e477 �[2mbastion_ip�[0m=192.168.0.202 �[2mprivate�[0m=i-db7f3ed6e4cdaa5aa �[2mprivate_ip�[0m=172.31.16.4
|
||||
�[2m· waiting for SSH handshake 192.168.0.202:22�[0m
|
||||
natgw_test.go:133: �[2m· waiting for SSH handshake 192.168.0.202:22�[0m
|
||||
�[2m· scp keypair -> bastion:/tmp/key.pem�[0m
|
||||
natgw_test.go:139: �[2m· scp keypair -> bastion:/tmp/key.pem�[0m
|
||||
�[2m· wait for private SSH via bastion�[0m
|
||||
natgw_test.go:151: �[2m· wait for private SSH via bastion�[0m
|
||||
�[2m· baseline: private VM has no internet�[0m
|
||||
natgw_test.go:172: �[2m· baseline: private VM has no internet�[0m
|
||||
�[2m· allocate-address (NAT EIP)�[0m
|
||||
natgw_test.go:178: �[2m· allocate-address (NAT EIP)�[0m
|
||||
�[2meip�[0m=192.168.0.203 �[2malloc�[0m=eipalloc-2d81753157cd62a50
|
||||
natgw_test.go:195: �[2meip�[0m=192.168.0.203 �[2malloc�[0m=eipalloc-2d81753157cd62a50
|
||||
�[2m· create-nat-gateway in subnet-cd62e4c1c3cb50c63�[0m
|
||||
natgw_test.go:197: �[2m· create-nat-gateway in subnet-cd62e4c1c3cb50c63�[0m
|
||||
�[2mnat_gw�[0m=nat-812216dd5105180c5
|
||||
natgw_test.go:216: �[2mnat_gw�[0m=nat-812216dd5105180c5
|
||||
natgw_test.go:218: nat gateway nat-812216dd5105180c5 reached state available
|
||||
�[2m· create-route-table�[0m
|
||||
natgw_test.go:223: �[2m· create-route-table�[0m
|
||||
�[2m· associate-route-table rtb-8d6a4abd5f666ec13 <- subnet-2360a40e28371e214�[0m
|
||||
natgw_test.go:241: �[2m· associate-route-table rtb-8d6a4abd5f666ec13 <- subnet-2360a40e28371e214�[0m
|
||||
�[2m· create-route 0.0.0.0/0 -> nat-812216dd5105180c5�[0m
|
||||
natgw_test.go:259: �[2m· create-route 0.0.0.0/0 -> nat-812216dd5105180c5�[0m
|
||||
�[2m· verify private VM reaches 8.8.8.8 via NAT GW�[0m
|
||||
natgw_test.go:279: �[2m· verify private VM reaches 8.8.8.8 via NAT GW�[0m
|
||||
�[2m· delete-nat-gateway nat-812216dd5105180c5�[0m
|
||||
natgw_test.go:293: �[2m· delete-nat-gateway nat-812216dd5105180c5�[0m
|
||||
natgw_test.go:318: nat gateway nat-812216dd5105180c5 reached state deleted
|
||||
�[2m· verify private VM lost internet after NAT GW teardown�[0m
|
||||
natgw_test.go:329: �[2m· verify private VM lost internet after NAT GW teardown�[0m]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase8e_SGToSGDatapath" classname="" time="80.940">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 8e — Security Group SG-to-SG Datapath (OVN) ━━�[0m
|
||||
�[2m· 8e-1 create sge-client + sge-target security groups�[0m
|
||||
datapath_test.go:61: �[2m· 8e-1 create sge-client + sge-target security groups�[0m
|
||||
�[2mclient_sg�[0m=sg-56e73c764a649f7ff �[2mtarget_sg�[0m=sg-67ad5efaa8be83b79
|
||||
datapath_test.go:83: �[2mclient_sg�[0m=sg-56e73c764a649f7ff �[2mtarget_sg�[0m=sg-67ad5efaa8be83b79
|
||||
�[2m· 8e-1 authorize target-sg ingress tcp/8080 from sg-56e73c764a649f7ff�[0m
|
||||
datapath_test.go:91: �[2m· 8e-1 authorize target-sg ingress tcp/8080 from sg-56e73c764a649f7ff�[0m
|
||||
�[2m· 8e-2 launch client-vm + target-vm�[0m
|
||||
datapath_test.go:109: �[2m· 8e-2 launch client-vm + target-vm�[0m
|
||||
datapath_test.go:132: instance i-eae6263d9b9fda069 reached state running
|
||||
datapath_test.go:133: instance i-9d9411a83549d34e3 reached state running
|
||||
�[2mclient_priv�[0m=172.31.0.5 �[2mclient_eni�[0m=eni-0f878da0143028915 �[2mtarget_priv�[0m=172.31.0.6 �[2mtarget_eni�[0m=eni-3bd355e19c7fdd45c
|
||||
datapath_test.go:142: �[2mclient_priv�[0m=172.31.0.5 �[2mclient_eni�[0m=eni-0f878da0143028915 �[2mtarget_priv�[0m=172.31.0.6 �[2mtarget_eni�[0m=eni-3bd355e19c7fdd45c
|
||||
�[2m· 8e-3 ovn-nbctl port_group membership�[0m
|
||||
datapath_test.go:151: �[2m· 8e-3 ovn-nbctl port_group membership�[0m
|
||||
�[2mclient_pg�[0m=sg_56e73c764a649f7ff �[2mtarget_pg�[0m=sg_67ad5efaa8be83b79
|
||||
datapath_test.go:163: �[2mclient_pg�[0m=sg_56e73c764a649f7ff �[2mtarget_pg�[0m=sg_67ad5efaa8be83b79
|
||||
�[2m· 8e-3 ovn-sbctl address_set sg_56e73c764a649f7ff_ip4 contains 172.31.0.5�[0m
|
||||
datapath_test.go:170: �[2m· 8e-3 ovn-sbctl address_set sg_56e73c764a649f7ff_ip4 contains 172.31.0.5�[0m
|
||||
�[2m· 8e-3 wait for client-vm SSH at 192.168.0.202:22�[0m
|
||||
datapath_test.go:188: �[2m· 8e-3 wait for client-vm SSH at 192.168.0.202:22�[0m
|
||||
�[2m· 8e-4 allowed traffic client -> target:172.31.0.6:8080�[0m
|
||||
datapath_test.go:199: �[2m· 8e-4 allowed traffic client -> target:172.31.0.6:8080�[0m
|
||||
�[2mstep4�[0m=allowed_traffic_ok
|
||||
datapath_test.go:208: �[2mstep4�[0m=allowed_traffic_ok
|
||||
�[2m· 8e-5 denied traffic client -> target:22 (no SSH ingress)�[0m
|
||||
datapath_test.go:213: �[2m· 8e-5 denied traffic client -> target:22 (no SSH ingress)�[0m
|
||||
�[2mstep5�[0m=denied_traffic_ok
|
||||
datapath_test.go:217: �[2mstep5�[0m=denied_traffic_ok
|
||||
�[2m· 8e-6 revoke target-sg ingress, retest (sync RequestEvent contract)�[0m
|
||||
datapath_test.go:222: �[2m· 8e-6 revoke target-sg ingress, retest (sync RequestEvent contract)�[0m
|
||||
�[2m· 8e-6 verify client -> target:8080 now blocked�[0m
|
||||
datapath_test.go:240: �[2m· 8e-6 verify client -> target:8080 now blocked�[0m
|
||||
�[2mstep6�[0m=revoke_blocked_ok
|
||||
datapath_test.go:244: �[2mstep6�[0m=revoke_blocked_ok
|
||||
�[2m· 8e-7 re-authorize target-sg ingress, verify traffic restored�[0m
|
||||
datapath_test.go:250: �[2m· 8e-7 re-authorize target-sg ingress, verify traffic restored�[0m
|
||||
�[2mstep7�[0m=restore_ok
|
||||
datapath_test.go:271: �[2mstep7�[0m=restore_ok
|
||||
datapath_test.go:127: instance i-eae6263d9b9fda069 reached state terminated
|
||||
datapath_test.go:127: instance i-9d9411a83549d34e3 reached state terminated]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase9_Teardown" classname="" time="8.300">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 9 — Teardown ━━�[0m
|
||||
�[2m· delete-snapshot snap-dde89b9ea2ef2e05a (custom AMI backing)�[0m
|
||||
teardown_test.go:33: �[2m· delete-snapshot snap-dde89b9ea2ef2e05a (custom AMI backing)�[0m
|
||||
�[2mdeleted_snapshot�[0m=snap-dde89b9ea2ef2e05a
|
||||
teardown_test.go:39: �[2mdeleted_snapshot�[0m=snap-dde89b9ea2ef2e05a
|
||||
�[2m· deregister-image ami-7dcf263a412dd14c6�[0m
|
||||
teardown_test.go:51: �[2m· deregister-image ami-7dcf263a412dd14c6�[0m
|
||||
�[2mderegistered_ami�[0m=ami-7dcf263a412dd14c6
|
||||
teardown_test.go:57: �[2mderegistered_ami�[0m=ami-7dcf263a412dd14c6
|
||||
�[2m· terminate-instances i-0c615e5dd74050f53�[0m
|
||||
teardown_test.go:74: �[2m· terminate-instances i-0c615e5dd74050f53�[0m
|
||||
�[2mterminated_instance�[0m=i-0c615e5dd74050f53
|
||||
teardown_test.go:81: �[2mterminated_instance�[0m=i-0c615e5dd74050f53
|
||||
�[2m· delete-key-pair test-key-1�[0m
|
||||
teardown_test.go:94: �[2m· delete-key-pair test-key-1�[0m
|
||||
�[2mdeleted_keypair�[0m=test-key-1
|
||||
teardown_test.go:100: �[2mdeleted_keypair�[0m=test-key-1]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase9a_VerifyTeardown" classname="" time="0.560">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 9a — Verify Teardown ━━�[0m
|
||||
�[2m· describe-instances i-0c615e5dd74050f53�[0m
|
||||
teardown_test.go:123: �[2m· describe-instances i-0c615e5dd74050f53�[0m
|
||||
�[2minstance_state�[0m=terminated
|
||||
teardown_test.go:135: �[2minstance_state�[0m=terminated
|
||||
�[2m· describe-images ami-7dcf263a412dd14c6�[0m
|
||||
teardown_test.go:147: �[2m· describe-images ami-7dcf263a412dd14c6�[0m
|
||||
�[2mami_reaped�[0m=ami-7dcf263a412dd14c6
|
||||
teardown_test.go:159: �[2mami_reaped�[0m=ami-7dcf263a412dd14c6
|
||||
�[2m· describe-key-pairs (must not contain test-key-1)�[0m
|
||||
teardown_test.go:167: �[2m· describe-key-pairs (must not contain test-key-1)�[0m
|
||||
�[2mkeypairs_listed�[0m=1
|
||||
teardown_test.go:176: �[2mkeypairs_listed�[0m=1]]></system-out>
|
||||
</testcase>
|
||||
<testcase name="TestSingleNode/Phase9b_FinalClusterStats" classname="" time="3.070">
|
||||
<system-out><![CDATA[
|
||||
�[1m�[36m━━ Phase 9b — Final Cluster Stats ━━�[0m
|
||||
�[2mspx_get_vms_bytes�[0m=118
|
||||
teardown_test.go:208: �[2mspx_get_vms_bytes�[0m=118]]></system-out>
|
||||
</testcase>
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -98,6 +98,15 @@ jobs:
|
||||
# warms across runs.
|
||||
cache: false
|
||||
|
||||
- name: Re-enable Go toolchain auto-download
|
||||
# setup-go@v6 exports GOTOOLCHAIN=local to $GITHUB_ENV (v5 left it
|
||||
# "auto"). build-install-artifacts.sh prepends /usr/local/go/bin to
|
||||
# PATH on the self-hosted runner, and that Go is older than
|
||||
# spinifex/go.mod's `go` directive; under "local" the build fails
|
||||
# with `requires go >= X.Y.Z`. Restoring "auto" lets the resolved
|
||||
# Go self-heal by fetching the named minor on demand.
|
||||
run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"
|
||||
|
||||
- name: Build Install Artifacts
|
||||
working-directory: mulga/scripts/tofu-cluster
|
||||
run: |
|
||||
@@ -211,6 +220,10 @@ jobs:
|
||||
for SUITE in ${SUITES_RUN}; do \
|
||||
echo \"::group::suite \$SUITE\"; \
|
||||
printf '\n%s%s━━ suite %s ━━%s\n' \"\$B\" \"\$C\" \"\$SUITE\" \"\$R\"; \
|
||||
# Record suite wall-clock start so the analyzer can locate
|
||||
# the right journal window. junit-report's timestamp attr
|
||||
# marks conversion time, not test start, so we cannot use it.
|
||||
date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
|
||||
GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
|
||||
ARTIFACT_DIR='${ARTIFACT_DIR}' \
|
||||
./\$SUITE.test -test.v -test.timeout=30m \
|
||||
@@ -246,7 +259,10 @@ jobs:
|
||||
ssh "${SSH_OPTS[@]}" "$VM" "tar -C '${ARTIFACT_DIR}' -cf - . 2>/dev/null || true" \
|
||||
| tar -C "${ARTIFACT_DIR}" -xf - 2>/dev/null
|
||||
ok "artifacts pulled to ${ARTIFACT_DIR}"
|
||||
ssh "${SSH_OPTS[@]}" "$VM" 'sudo journalctl -u "spinifex-*" --no-pager --since "60 min ago"' \
|
||||
# `-o short-iso` writes RFC3339-ish timestamps (e.g.
|
||||
# `2026-05-19T12:32:34+1000 host …`) so the analyzer can window-slice
|
||||
# the journal around each failure's wall-clock span.
|
||||
ssh "${SSH_OPTS[@]}" "$VM" 'sudo journalctl -u "spinifex-*" --no-pager --output=short-iso --since "60 min ago"' \
|
||||
> "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null
|
||||
ok "journal: $(wc -l < "${ARTIFACT_DIR}/spinifex-journal.log" 2>/dev/null || echo 0) lines"
|
||||
|
||||
@@ -271,6 +287,14 @@ jobs:
|
||||
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
|
||||
title: "Go E2E results"
|
||||
|
||||
- name: Analyze e2e failures
|
||||
if: always()
|
||||
uses: ./spinifex/.github/actions/e2e-analyze
|
||||
with:
|
||||
junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
|
||||
log-dir: ${{ env.ARTIFACT_DIR }}
|
||||
title: "E2E failure analysis"
|
||||
|
||||
- uses: actions/upload-artifact@v7
|
||||
if: always()
|
||||
with:
|
||||
|
||||
@@ -61,6 +61,9 @@ jobs:
|
||||
- name: Run Tests with Coverage
|
||||
run: make -C spinifex test-cover
|
||||
|
||||
- name: Run Action Tests
|
||||
run: make -C spinifex test-actions
|
||||
|
||||
- name: Check Diff Coverage
|
||||
run: cd spinifex && scripts/diff-coverage.sh coverage.out --base HEAD~1
|
||||
|
||||
|
||||
@@ -126,6 +126,12 @@ test-race:
|
||||
@echo -e "\n....Running tests with race detector for $(GO_PROJECT_NAME)...."
|
||||
$(_Q)LOG_IGNORE=1 go test -race -timeout 300s ./spinifex/... $(_RACEQ)
|
||||
|
||||
# Unit tests for in-repo GitHub Actions (e.g. .github/actions/e2e-analyze).
|
||||
# Kept out of `test-cover` so coverage % isn't diluted by CI-only tooling.
|
||||
test-actions:
|
||||
@echo -e "\n....Running action tests...."
|
||||
LOG_IGNORE=1 go test -timeout 60s ./.github/actions/...
|
||||
|
||||
# Check that new/changed code meets coverage threshold (runs tests first)
|
||||
diff-coverage: test-cover
|
||||
@QUIET=$(QUIET) scripts/diff-coverage.sh $(COVERPROFILE)
|
||||
@@ -275,7 +281,7 @@ ansible-dev-reset:
|
||||
ansible-dev-deploy:
|
||||
cd scripts/ansible && ansible-playbook playbooks/dev-deploy.yml
|
||||
|
||||
.PHONY: build build-ui build-installer build-lb-agent build-system-image build-microvm-image install-microvm go_build go_run preflight test test-cover test-race diff-coverage bench run \
|
||||
.PHONY: build build-ui build-installer build-lb-agent build-system-image build-microvm-image install-microvm go_build go_run preflight test test-cover test-race test-actions diff-coverage bench run \
|
||||
deploy reinstall clean \
|
||||
install-system install-go install-aws quickinstall \
|
||||
lint fix govulncheck \
|
||||
|
||||
@@ -1361,6 +1361,17 @@ func (h *TopologyHandler) handleAddNAT(msg *nats.Msg) {
|
||||
"logical_ip", evt.LogicalIP,
|
||||
"port", evt.PortName,
|
||||
)
|
||||
|
||||
// Block until ovn-northd compiles the new NAT into SB and every chassis
|
||||
// has installed the resulting flows. handleIGWAttach has the same
|
||||
// barrier (mulga-siv-105); without one here, the publisher (typically
|
||||
// ec2d/RunInstances) returns success while the DNAT/SNAT flows for the
|
||||
// VM's public IP are still being installed on the gateway chassis, so
|
||||
// the freshly launched VM is unreachable on its public IP for whatever
|
||||
// flow-install latency the chassis happens to incur. Reproduced in CI
|
||||
// run 26072432957 Phase8b — Port_Binding showed up=true at the 3min
|
||||
// timeout, but SSH never completed handshake.
|
||||
_ = waitForFlowsHV()
|
||||
respond(msg, nil)
|
||||
}
|
||||
|
||||
|
||||
@@ -923,6 +923,55 @@ func TestTopologyHandler_IGWAttach_InvokesFlowBarrier(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestTopologyHandler_AddNAT_InvokesFlowBarrier asserts handleAddNAT
|
||||
// calls waitForFlowsHV after writing the dnat_and_snat row to NB.
|
||||
// Without this barrier, RunInstances returns success while OF flows for
|
||||
// the new public IP are still being installed on the gateway chassis,
|
||||
// and the freshly launched VM is unreachable on its public IP for the
|
||||
// flow-install latency. Reproduced in CI run 26072432957 Phase8b.
|
||||
func TestTopologyHandler_AddNAT_InvokesFlowBarrier(t *testing.T) {
|
||||
orig := waitForFlowsHV
|
||||
var called int
|
||||
waitForFlowsHV = func() error { called++; return nil }
|
||||
defer func() { waitForFlowsHV = orig }()
|
||||
|
||||
_, nc := startTestNATS(t)
|
||||
mock := NewMockOVNClient()
|
||||
_ = mock.Connect(context.Background())
|
||||
ctx := context.Background()
|
||||
|
||||
topo := NewTopologyHandler(mock)
|
||||
subs, err := topo.Subscribe(nc)
|
||||
require.NoError(t, err)
|
||||
defer func() {
|
||||
for _, s := range subs {
|
||||
_ = s.Unsubscribe()
|
||||
}
|
||||
}()
|
||||
|
||||
require.NoError(t, mock.CreateLogicalRouter(ctx, &nbdb.LogicalRouter{
|
||||
Name: "vpc-vpc-natbarrier",
|
||||
ExternalIDs: map[string]string{
|
||||
"spinifex:vpc_id": "vpc-natbarrier",
|
||||
"spinifex:cidr": "10.0.0.0/16",
|
||||
},
|
||||
}))
|
||||
|
||||
evt := NATEvent{
|
||||
VpcId: "vpc-natbarrier",
|
||||
ExternalIP: "192.168.0.250",
|
||||
LogicalIP: "10.0.1.4",
|
||||
}
|
||||
data, _ := json.Marshal(evt)
|
||||
resp, err := nc.Request(TopicAddNAT, data, 5_000_000_000)
|
||||
require.NoError(t, err)
|
||||
assertSuccess(t, resp, "add NAT")
|
||||
|
||||
if called != 1 {
|
||||
t.Errorf("expected waitForFlowsHV to run exactly once after add-nat, got %d", called)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTopologyHandler_IGWAttach_WithExternalPool(t *testing.T) {
|
||||
_, nc := startTestNATS(t)
|
||||
mock := NewMockOVNClient()
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
# Rule table consumed by .github/actions/e2e-analyze in Stage 3 of
|
||||
# docs/development/improvements/e2e-go-failure-analysis.md.
|
||||
#
|
||||
# Stage 1 (current) ignores this file. Stage 3 will match a failure's
|
||||
# normalised signature against `match` / `also_match` regexes and surface
|
||||
# `hint` text inline in the GITHUB_STEP_SUMMARY analysis report.
|
||||
#
|
||||
# Add rules in the same PR as the test or feature that surfaced the
|
||||
# failure. Rules are advisory only — they never gate the build.
|
||||
#
|
||||
# Schema (Stage 3 will validate):
|
||||
# - id: stable identifier (kebab-case)
|
||||
# match: Go-syntax regex against the normalised signature line
|
||||
# also_match: optional second regex (must also match)
|
||||
# hint: |
|
||||
# Free text shown in the analysis report when match fires.
|
||||
# Multi-line; markdown rendered.
|
||||
# artifacts: [list of relative paths under ARTIFACT_DIR to surface]
|
||||
#
|
||||
# Empty list keeps the analyser additive — no rules means no hints.
|
||||
|
||||
rules: []
|
||||
+143
-1
@@ -6,7 +6,9 @@ import (
|
||||
"bytes"
|
||||
"encoding/base64"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -15,6 +17,27 @@ import (
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
)
|
||||
|
||||
// VPCDiagnosticsOpts carries the optional context dumpVPCFlowDiagnostics
|
||||
// uses to target IP-specific captures (OF flows, conntrack, ARP, NAT
|
||||
// lookup). Empty fields mean "skip the captures that need this input"
|
||||
// — the unconditional captures (console-output, ovn-nbctl/sbctl show)
|
||||
// still run.
|
||||
type VPCDiagnosticsOpts struct {
|
||||
// ExternalIP is the public IP the test was probing (e.g. the
|
||||
// dnat_and_snat external address). When set, the helper greps
|
||||
// br-int flows / conntrack / host ARP for this IP.
|
||||
ExternalIP string
|
||||
// LogicalIP is the VM's private IP. When set, the OF-flow grep
|
||||
// also matches this address so we see both halves of the NAT
|
||||
// translation.
|
||||
LogicalIP string
|
||||
// ArtifactDir is the per-test bundle directory (typically
|
||||
// `fix.Artifacts`). When set, every capture is also persisted as
|
||||
// a separate file so the Stage 2 analyzer bundler picks it up
|
||||
// alongside the journal + test-log slices.
|
||||
ArtifactDir string
|
||||
}
|
||||
|
||||
// DumpVPCFlowDiagnostics emits a triage bundle for VPC/IGW datapath failures
|
||||
// (e.g. Phase8b SSH-to-public-IP timeout). Dumps the guest's console tail
|
||||
// (boot + cloud-init state) plus a slim OVN state snapshot to identify
|
||||
@@ -25,16 +48,19 @@ import (
|
||||
// - mulga-siv-105 (handleIGWAttach lacks flows-ready barrier)
|
||||
// - mulga-siv-106 (handleVPCCreate doesn't store spinifex:cidr)
|
||||
// - mulga-siv-107 (dnat_and_snat double-delete race)
|
||||
// - mulga-siv-111 (EIP-recycle stale OVS state; this is the Phase 1
|
||||
// diagnostics surface that closes the bead's first acceptance gate)
|
||||
//
|
||||
// Skips silently if OVN tooling / passwordless sudo aren't available, so the
|
||||
// helper is safe to call from developer laptops too.
|
||||
func DumpVPCFlowDiagnostics(t *testing.T, c *AWSClient, instanceID, label string) {
|
||||
func DumpVPCFlowDiagnostics(t *testing.T, c *AWSClient, instanceID, label string, opts VPCDiagnosticsOpts) {
|
||||
t.Helper()
|
||||
fmt.Printf("\n%s%s── DIAGNOSTICS: %s (instance=%s) ──%s\n",
|
||||
colorBold, colorCyan, label, instanceID, colorReset)
|
||||
|
||||
dumpConsoleOutput(t, c, instanceID)
|
||||
dumpOVNState(t, instanceID)
|
||||
dumpDatapathState(t, opts)
|
||||
|
||||
fmt.Printf("%s%s── END DIAGNOSTICS ──%s\n\n", colorBold, colorCyan, colorReset)
|
||||
}
|
||||
@@ -110,6 +136,122 @@ func dumpOVNState(t *testing.T, instanceID string) {
|
||||
}
|
||||
}
|
||||
|
||||
// dumpDatapathState captures OVS / kernel-side state keyed on the
|
||||
// external IP being probed. The signal disambiguates between three
|
||||
// hypotheses on the residual SSH-timeout-despite-barrier failure
|
||||
// (mulga-siv-111): (1) stale conntrack, (2) stale upstream ARP, (3)
|
||||
// missing OF flow install. Each capture also writes to a separate
|
||||
// file under opts.ArtifactDir when set, so the Stage 2 analyzer
|
||||
// bundler picks them up.
|
||||
//
|
||||
// Skipped silently when the IP-specific inputs aren't supplied or the
|
||||
// shell tools aren't available.
|
||||
func dumpDatapathState(t *testing.T, opts VPCDiagnosticsOpts) {
|
||||
t.Helper()
|
||||
if opts.ExternalIP == "" {
|
||||
return
|
||||
}
|
||||
if err := exec.Command("sudo", "-n", "true").Run(); err != nil {
|
||||
fmt.Printf(" datapath: passwordless sudo unavailable (%v); skipping captures\n", err)
|
||||
return
|
||||
}
|
||||
|
||||
type capture struct {
|
||||
filename string
|
||||
label string
|
||||
argv []string
|
||||
// grepFor, when non-empty, post-filters stdout to lines
|
||||
// containing this substring. Lets us keep ovs-ofctl / conntrack
|
||||
// output tight without an external pipe.
|
||||
grepFor []string
|
||||
}
|
||||
caps := []capture{
|
||||
{
|
||||
filename: "ovs-flows-extip.txt",
|
||||
label: "ovs-ofctl dump-flows br-int (filtered)",
|
||||
argv: []string{"ovs-ofctl", "dump-flows", "br-int"},
|
||||
grepFor: []string{opts.ExternalIP, opts.LogicalIP},
|
||||
},
|
||||
{
|
||||
filename: "conntrack-extip.txt",
|
||||
label: "conntrack -L (filtered)",
|
||||
argv: []string{"conntrack", "-L"},
|
||||
grepFor: []string{opts.ExternalIP},
|
||||
},
|
||||
{
|
||||
filename: "ip-neigh-extip.txt",
|
||||
label: "ip neigh show (host ARP, filtered)",
|
||||
argv: []string{"ip", "-4", "neigh", "show"},
|
||||
grepFor: []string{opts.ExternalIP},
|
||||
},
|
||||
{
|
||||
filename: "ovn-nat-extip.txt",
|
||||
label: "ovn-nbctl find NAT external_ip",
|
||||
argv: []string{"ovn-nbctl", "--bare", "--columns=_uuid,type,external_ip,logical_ip,external_mac,logical_port",
|
||||
"find", "NAT", "external_ip=" + opts.ExternalIP},
|
||||
},
|
||||
}
|
||||
|
||||
for _, cap := range caps {
|
||||
fmt.Printf(" --- %s ---\n", cap.label)
|
||||
var stdout, stderr bytes.Buffer
|
||||
cmd := exec.Command("sudo", append([]string{"-n"}, cap.argv...)...)
|
||||
cmd.Stdout = &stdout
|
||||
cmd.Stderr = &stderr
|
||||
cmd.Env = append(cmd.Env, "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin")
|
||||
err := runWithTimeout(cmd, 5*time.Second)
|
||||
body := stdout.String()
|
||||
if len(cap.grepFor) > 0 {
|
||||
body = filterLines(body, cap.grepFor)
|
||||
}
|
||||
if err != nil {
|
||||
body = fmt.Sprintf("(capture failed: %v; stderr=%q)\n%s", err, stderr.String(), body)
|
||||
} else if strings.TrimSpace(body) == "" {
|
||||
body = "(no matches)\n"
|
||||
}
|
||||
for _, line := range strings.Split(strings.TrimRight(body, "\n"), "\n") {
|
||||
fmt.Printf(" %s\n", line)
|
||||
}
|
||||
if opts.ArtifactDir != "" {
|
||||
path := filepath.Join(opts.ArtifactDir, cap.filename)
|
||||
header := fmt.Sprintf("$ sudo %s\n", strings.Join(cap.argv, " "))
|
||||
if len(cap.grepFor) > 0 {
|
||||
header += "# filtered to lines matching: " + strings.Join(cap.grepFor, ", ") + "\n"
|
||||
}
|
||||
if werr := os.WriteFile(path, []byte(header+body), 0o644); werr != nil {
|
||||
fmt.Printf(" (artifact write failed: %v)\n", werr)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// filterLines returns only stdout lines that contain any of needles.
|
||||
// Empty needles slice → input passes through unchanged. Empty needle
|
||||
// strings are skipped so a caller can pass `opts.LogicalIP` without
|
||||
// guarding for the empty case.
|
||||
func filterLines(body string, needles []string) string {
|
||||
var clean []string
|
||||
for _, n := range needles {
|
||||
if n != "" {
|
||||
clean = append(clean, n)
|
||||
}
|
||||
}
|
||||
if len(clean) == 0 {
|
||||
return body
|
||||
}
|
||||
var b strings.Builder
|
||||
for _, line := range strings.Split(body, "\n") {
|
||||
for _, n := range clean {
|
||||
if strings.Contains(line, n) {
|
||||
b.WriteString(line)
|
||||
b.WriteByte('\n')
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func runWithTimeout(cmd *exec.Cmd, timeout time.Duration) error {
|
||||
if err := cmd.Start(); err != nil {
|
||||
return err
|
||||
|
||||
@@ -191,6 +191,28 @@ func phase8b_VPCSubnetE2E(t *testing.T, fix *Fixture) {
|
||||
})
|
||||
harness.Detail(t, "rtb", rtbID, "assoc", assocID, "route", "0.0.0.0/0->"+igwID)
|
||||
|
||||
// --- Security group ----------------------------------------------------
|
||||
// Fresh VPC's auto-created default SG denies all ingress. Authorize
|
||||
// tcp/22 from anywhere so the SSH probe below can complete — the bash
|
||||
// driver relies on the *default* VPC's default SG having been pre-baked
|
||||
// with this rule by admin init, but a newly-created VPC has not.
|
||||
// Without this step Phase 8b SSH times out via an ACL drop between OVN
|
||||
// tables 13 (DNAT) and 17 (NAT commit) even when every datapath barrier
|
||||
// is satisfied (tracking mulga-siv-111).
|
||||
harness.Step(t, "describe-security-groups (default) vpc=%s", vpcID)
|
||||
sgOut, err := c.EC2.DescribeSecurityGroups(&ec2.DescribeSecurityGroupsInput{
|
||||
Filters: []*ec2.Filter{
|
||||
{Name: aws.String("vpc-id"), Values: []*string{aws.String(vpcID)}},
|
||||
{Name: aws.String("group-name"), Values: []*string{aws.String("default")}},
|
||||
},
|
||||
})
|
||||
require.NoError(t, err, "describe-security-groups default")
|
||||
require.NotEmpty(t, sgOut.SecurityGroups, "no default SG for vpc=%s", vpcID)
|
||||
defaultSGID := aws.StringValue(sgOut.SecurityGroups[0].GroupId)
|
||||
require.NotEmpty(t, defaultSGID, "default SG GroupId empty")
|
||||
harness.AuthorizeSSHIngress(t, c, defaultSGID)
|
||||
harness.Detail(t, "default_sg", defaultSGID, "ingress", "tcp/22<-0.0.0.0/0")
|
||||
|
||||
// --- Public instance ---------------------------------------------------
|
||||
harness.Step(t, "run-instances ami=%s subnet=%s", fix.AMIID, pubSubnetID)
|
||||
runOut, err := c.EC2.RunInstances(&ec2.RunInstancesInput{
|
||||
@@ -226,11 +248,16 @@ func phase8b_VPCSubnetE2E(t *testing.T, fix *Fixture) {
|
||||
// --- SSH + external connectivity ---------------------------------------
|
||||
// Use the non-fatal probe so we can dump VPC/IGW datapath diagnostics
|
||||
// before Fatal. Same 3min budget — fresh-VPC unreachability beyond that
|
||||
// is a real product bug (mulga-siv-105), not test flake.
|
||||
// is a real product bug (mulga-siv-105 + mulga-siv-111), not test flake.
|
||||
if !trySSHReady(pubIP, 22, fix.KeyPath, 3*time.Minute) {
|
||||
harness.DumpVPCFlowDiagnostics(t, c, instID,
|
||||
fmt.Sprintf("Phase 8b SSH timeout — vpc=%s igw=%s pub=%s", vpcID, igwID, pubIP))
|
||||
t.Fatalf("SSH handshake %s:22 never completed within 3min (see diagnostics above; tracking mulga-siv-105)", pubIP)
|
||||
fmt.Sprintf("Phase 8b SSH timeout — vpc=%s igw=%s pub=%s", vpcID, igwID, pubIP),
|
||||
harness.VPCDiagnosticsOpts{
|
||||
ExternalIP: pubIP,
|
||||
LogicalIP: privIP,
|
||||
ArtifactDir: fix.Artifacts,
|
||||
})
|
||||
t.Fatalf("SSH handshake %s:22 never completed within 3min (see diagnostics above; tracking mulga-siv-111)", pubIP)
|
||||
}
|
||||
|
||||
tgt := harness.SSHTarget{User: "ec2-user", Host: pubIP, Port: 22, KeyPath: fix.KeyPath}
|
||||
|
||||
Reference in New Issue
Block a user