release: update go.mod for v1.6.0

After PR #270 the VPC-LR-wide 0.0.0.0/0 -> IGW static route is no longer
installed implicitly at attach-internet-gateway. Subnets whose effective
route table lacks an internet-bound route get an OVN drop policy at
SubnetEgressPriorityDrop (1100), which kills LB target reply traffic
before un-DNAT can fire and stalls the LB internal-client probe until
job timeout.

Add the standard AWS pattern: describe-route-tables -> create-route
0.0.0.0/0 -> IGW. This fires vpc.add-igw-route + vpc.ungate-subnet-egress,
removing the drop policy and installing a reroute policy at
SubnetEgressPriorityIGW (1000).

.github/CODEOWNERS

@@ -0,0 +1,21 @@
+# CODEOWNERS — GitHub PR review enforcement
+#
+# Format: <path pattern>  <owner> [<owner> …]
+# Order matters — last matching pattern wins.
+#
+# Set the matching branch protection rule on `dev` and `main` to
+# "Require review from Code Owners" or this file is advisory only.
+
+# Fall-through: any change needs a team review.
+*                           @mulgadc/mulgadc-engineers
+
+# Network layer is governed by ADR-0006 invariant tests.
+# Changes here must respect those clauses — see
+# spinifex/network/invariants/ and
+# docs/development/proposals/az-local-architecture/0006-spinifex-network-layer-contract.md
+/spinifex/network/          @joshsiv-mulga
+/spinifex/vpcd/             @joshsiv-mulga
+
+# Invariants suite is the executable ADR-0006 fence — any widening or
+# narrowing of the scan roots needs an explicit owner review.
+/spinifex/network/invariants/ @joshsiv-mulga

.github/actions/e2e-analyze/README.md

@@ -0,0 +1,104 @@
+# e2e-analyze
+
+Post-test failure-clustering action for the Go E2E suites.
+
+## What it does
+
+Runs after `go-junit-report` has converted each suite's stdout into
+`junit-<suite>.xml`. For every JUnit file matched by `junit-glob`:
+
+1. Parses each `<testcase>` with a `<failure>` / `<error>` child.
+2. Strips ANSI escapes, then normalises dynamic noise (UUIDs, EC2 IDs,
+   IPs, timestamps, request-ids, source line numbers) to produce a stable
+   _signature_ per failure.
+3. Picks the **earliest non-cascade** failure per suite as the likely
+   root cause. "Cascade" = failure message matches a fixture-dependency
+   marker (`must populate fix.`, `Should NOT be empty`,
+   `Expected value not to be nil`). When every failure is a cascade
+   (whole suite tripped on one upstream gap), the earliest cascade is
+   the fallback root.
+4. Buckets the remaining failures by signature and renders a markdown
+   report. The report is appended to `$GITHUB_STEP_SUMMARY` and also
+   written to `<log-dir>/analysis.md` so it travels with the uploaded
+   artifact bundle.
+
+The action **never fails the job** — exit-code semantics stay with the
+existing "Fail job if any suite failed" gate.
+
+## Inputs
+
+| Input        | Required | Description |
+|--------------|----------|-------------|
+| `junit-glob` | yes      | Shell glob for the JUnit XML files (e.g. `${ARTIFACT_DIR}/junit-*.xml`). |
+| `log-dir`    | yes      | Directory the rendered report is written into (`analysis.md`). |
+| `title`      | no       | Heading at the top of the summary section. Default `E2E failure analysis`. |
+
+## Outputs
+
+None. The summary is appended to `$GITHUB_STEP_SUMMARY`; the same
+content lands at `<log-dir>/analysis.md`.
+
+## Local invocation
+
+Outside Actions, the analyser prints to stdout instead of appending to a
+summary file:
+
+```sh
+cd spinifex/.github/actions/e2e-analyze
+go run . -junit-glob "/tmp/artifacts/junit-*.xml" -log-dir /tmp/artifacts
+```
+
+## Tests
+
+Golden-file tests live in `testdata/`. Each subdir holds a scenario:
+
+| Scenario           | Source                                    |
+|--------------------|-------------------------------------------|
+| `cascade-vpcs`     | Synthetic: one Phase 5 root + 15 cascades |
+| `single-failure`   | Real artifact from run `26037892507`      |
+| `no-failures`      | Synthetic: clean run                      |
+
+Run normally:
+
+```sh
+go test ./.github/actions/e2e-analyze/
+```
+
+When the report format changes intentionally, regenerate the snapshots:
+
+```sh
+go test ./.github/actions/e2e-analyze/ -update
+```
+
+## Per-failure bundles (Stage 2)
+
+After parsing, the action materialises a bundle directory per failure
+under `<log-dir>/analysis/NN-<TestName>/`. The report links to it from
+each entry so a reader on the Actions page can click straight through
+to the relevant log slice.
+
+Each bundle contains:
+
+| File              | Source                                                                            |
+|-------------------|-----------------------------------------------------------------------------------|
+| `failure.txt`     | Synthesised summary: test, start, duration, file-site, error line.                |
+| `journal.log`     | `spinifex-journal.log` sliced to `[StartAt − 5s, StartAt + Duration + 5s]`.       |
+| `test.log`        | `test-<suite>.log` block between `=== RUN <name>` and the matching `--- FAIL`.    |
+| `<other files>`   | Anything the harness already dropped into `<log-dir>/<TestName>/` at run-time.    |
+
+The journal slice relies on the workflow running
+`journalctl --output=short-iso` so timestamps are parseable. Lines
+without a parseable timestamp inherit the previous line's
+in-window/out-of-window classification (so multi-line records and stack
+traces stay contiguous).
+
+## Roadmap
+
+- **Stage 1 (shipped):** first-failure surfacing + cascade clustering.
+- **Stage 2 (this update):** time-window correlation — per-failure bundle
+  with sliced journal + test-log + any per-test artifacts.
+- **Stage 3:** rule-based root-cause classifier driven by
+  `spinifex/tests/e2e/.failure-rules.yaml`.
+
+See `docs/development/improvements/e2e-go-failure-analysis.md` in the
+parent mulga repo.

.github/actions/e2e-analyze/action.yml

@@ -0,0 +1,38 @@
+name: E2E Failure Analyzer
+description: >-
+  Post-test analyzer for the Go E2E suites. Parses junit-*.xml emitted by
+  go-junit-report, clusters failures by signature, picks the earliest
+  non-cascade failure per suite as the likely root cause, and appends a
+  human-readable report to $GITHUB_STEP_SUMMARY. Always exits 0 — the
+  existing "Fail job if any suite failed" gate keeps owning exit-code
+  semantics.
+
+inputs:
+  junit-glob:
+    description: Shell glob matching JUnit XML files to analyze.
+    required: true
+  log-dir:
+    description: >-
+      Artifact directory. The rendered report is also written to
+      $log-dir/analysis.md so it travels with the uploaded artifact.
+    required: true
+  title:
+    description: Heading rendered at the top of the summary section.
+    required: false
+    default: "E2E failure analysis"
+
+runs:
+  using: composite
+  steps:
+    - id: analyze
+      shell: bash
+      working-directory: ${{ github.action_path }}
+      env:
+        ANALYZE_JUNIT_GLOB: ${{ inputs.junit-glob }}
+        ANALYZE_LOG_DIR: ${{ inputs.log-dir }}
+        ANALYZE_TITLE: ${{ inputs.title }}
+      run: |
+        go run . \
+          -junit-glob "$ANALYZE_JUNIT_GLOB" \
+          -log-dir "$ANALYZE_LOG_DIR" \
+          -title "$ANALYZE_TITLE"

.github/actions/e2e-analyze/analyze.go

@@ -0,0 +1,552 @@
+// Package main implements the e2e-analyze GitHub Action.
+//
+// Stage 1 of docs/development/improvements/e2e-go-failure-analysis.md:
+// cluster failures from go-junit-report's junit-*.xml by error signature,
+// pick the earliest non-cascade failure per suite as the likely root cause,
+// and render a human-readable report.
+//
+// This file contains the pure logic (parsing, signature extraction,
+// clustering, rendering). Side-effect handling lives in main.go.
+package main
+
+import (
+	"encoding/xml"
+	"fmt"
+	"path/filepath"
+	"regexp"
+	"sort"
+	"strings"
+	"time"
+)
+
+// JUnit XML schema produced by go-junit-report v2. Stable across versions
+// we care about; only fields used downstream are decoded.
+type junitSuites struct {
+	XMLName xml.Name     `xml:"testsuites"`
+	Suites  []junitSuite `xml:"testsuite"`
+}
+
+type junitSuite struct {
+	Name      string    `xml:"name,attr"`
+	Tests     int       `xml:"tests,attr"`
+	Failures  int       `xml:"failures,attr"`
+	Errors    int       `xml:"errors,attr"`
+	Skipped   int       `xml:"skipped,attr"`
+	Timestamp string    `xml:"timestamp,attr"`
+	Time      float64   `xml:"time,attr"`
+	Cases     []junitTC `xml:"testcase"`
+}
+
+type junitTC struct {
+	Name      string        `xml:"name,attr"`
+	Classname string        `xml:"classname,attr"`
+	Time      float64       `xml:"time,attr"`
+	Failure   *junitFailure `xml:"failure"`
+	Error     *junitFailure `xml:"error"`
+	Skipped   *junitSkipped `xml:"skipped"`
+	SystemOut string        `xml:"system-out"`
+}
+
+type junitFailure struct {
+	Message string `xml:"message,attr"`
+	Type    string `xml:"type,attr"`
+	Body    string `xml:",chardata"`
+}
+
+type junitSkipped struct {
+	Message string `xml:"message,attr"`
+}
+
+// Failure is the analyzer's normalized view of one failed testcase.
+type Failure struct {
+	SuiteFile string    // e.g. "single" (derived from junit-single.xml)
+	Name      string    // e.g. "TestSingleNode/Phase5_LaunchInstance"
+	Duration  float64   // seconds
+	Order     int       // position within the suite (0-based)
+	StartAt   time.Time // estimated wall-clock start: suite.timestamp + sum(prior durations)
+	Error     string    // last meaningful error line, ANSI-stripped
+	FileHint  string    // e.g. "vpc_test.go:227", parsed from failure body
+	Cascade   bool      // signature matched a known downstream pattern
+	Signature string    // normalized error key for bucketing
+	// OffsetFromSuiteStart is sum(prior testcase durations) at the time
+	// this test started. Persisted separately from StartAt so a caller
+	// that has a more authoritative suite wall-clock origin (e.g. a
+	// `test-<suite>.start` file written by the workflow before the suite
+	// runs) can recompute StartAt without re-parsing the XML.
+	OffsetFromSuiteStart time.Duration
+	// BundlePath is the relative path (from log-dir) to the per-failure
+	// bundle written by Stage 2's WriteBundles. Empty when bundles were
+	// not generated (e.g. unit tests that call Render directly).
+	BundlePath string
+}
+
+// SuiteReport summarises one junit-*.xml file.
+type SuiteReport struct {
+	File      string // basename, e.g. "junit-single.xml"
+	Label     string // short name, e.g. "single"
+	Total     int
+	FailCount int
+	Root      *Failure    // nil if no failures
+	Cascades  []Failure   // everything else that failed in this suite
+	Buckets   [][]Failure // failures grouped by signature (root's bucket first)
+}
+
+// Report is the full analysis across every junit file.
+type Report struct {
+	Title  string
+	Suites []SuiteReport
+}
+
+// ansiSeq matches ANSI CSI escapes the harness sprinkles through log
+// output (colours, bold). Three forms in the wild:
+//   - real ESC: `\x1b[36m…`
+//   - U+FFFD replacement (XML serialiser dropped the ESC byte): `�[36m…`
+//   - bare CSI with the leading byte gone entirely: `[36m…`
+//
+// All three are stripped so colour state doesn't fork the signature bucket.
+var ansiSeq = regexp.MustCompile(
+	"\x1b\\[[0-9;]*[a-zA-Z]" + // real ESC CSI
+		"|�\\[[0-9;]*[a-zA-Z]" + // replacement-char CSI
+		"|\\[[0-9]{1,2}(?:;[0-9]{1,2})*m", // bare colour code, "mode" form only
+)
+
+// cascadeMarkers are message fragments that only appear in tests that
+// depend on a fixture field populated by an earlier phase. Their presence
+// means the failure is downstream of some other failure that happened
+// first; the report collapses them under that root cause.
+var cascadeMarkers = []string{
+	"must populate fix.",
+	"Should NOT be empty",
+	"Expected value not to be nil",
+}
+
+// noiseReplacers normalise dynamic identifiers so that "ten failures, same
+// bug" bucket together. Pattern ordering matters: longer/more-specific
+// patterns must precede the catch-alls.
+var noiseReplacers = []struct {
+	re   *regexp.Regexp
+	repl string
+}{
+	{regexp.MustCompile(`\b(vpc|subnet|sg|rtb|igw|i|ami|vol|snap|rtbassoc|key)-[0-9a-f]{8,}\b`), `${1}-<id>`},
+	{regexp.MustCompile(`\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b`), `<uuid>`},
+	{regexp.MustCompile(`\b[0-9a-f]{32,}\b`), `<hex>`},
+	{regexp.MustCompile(`\b\d{1,3}(\.\d{1,3}){3}(:\d+)?\b`), `<ip>`},
+	{regexp.MustCompile(`\b\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:?\d{2})?\b`), `<ts>`},
+	{regexp.MustCompile(`request-id:\s*\S+`), `request-id: <id>`},
+	{regexp.MustCompile(`(_test\.go):\d+`), `${1}:<line>`},
+	{regexp.MustCompile(`\s+`), ` `},
+}
+
+// fileHintRe pulls the last "foo_test.go:NNN" we see in a failure body so
+// the report can point a reader at the source line. Test files dominate the
+// hint set; harness/helper file matches go through goFileLineRe below.
+var fileHintRe = regexp.MustCompile(`([A-Za-z0-9_]+_test\.go:\d+)`)
+
+// goFileLineRe matches any "foo.go:NNN:" prefix (test or non-test). Used to
+// surface harness/helper-level error text like
+// `ec2helpers.go:164: describe-vpcs: InvalidParameterValue …` which Go test
+// emits when a non-test file calls t.Errorf.
+var goFileLineRe = regexp.MustCompile(`^[A-Za-z0-9_]+\.go:\d+:\s*(.+)$`)
+
+// testifyErrorPlaceholder is testify's "Error:" boilerplate when the
+// assertion failed because of a wrapped error. The next non-blank line in
+// the trace holds the actual error text.
+const testifyErrorPlaceholder = "Received unexpected error:"
+
+// stripANSI removes CSI escapes; leaves everything else intact.
+func stripANSI(s string) string {
+	return ansiSeq.ReplaceAllString(s, "")
+}
+
+// extractErrorLine finds the most useful single-line summary in a failure
+// body. Priority order, highest first:
+//  1. Testify "Messages:" line — user-supplied context, the most signal-rich.
+//  2. Testify "Error:" content (with one-line lookahead when the placeholder
+//     "Received unexpected error:" defers to the next line).
+//  3. Leading "foo.go:NNN: text" content line — t.Errorf from non-test code.
+//  4. Last `_test.go:NNN: text` line — bare t.Fatalf/Errorf from a test file.
+//  5. First non-empty line of the body.
+func extractErrorLine(body string) string {
+	body = stripANSI(body)
+	lines := strings.Split(body, "\n")
+
+	var (
+		msgLine      string
+		errLine      string
+		goFileLine   string
+		lastTestLine string
+	)
+	for i, raw := range lines {
+		l := strings.TrimSpace(raw)
+		if l == "" {
+			continue
+		}
+		if strings.HasPrefix(l, "Messages:") {
+			msgLine = strings.TrimSpace(strings.TrimPrefix(l, "Messages:"))
+			continue
+		}
+		if strings.HasPrefix(l, "Error:") && errLine == "" {
+			content := strings.TrimSpace(strings.TrimPrefix(l, "Error:"))
+			if content == testifyErrorPlaceholder {
+				// Look ahead for the wrapped error text on the next non-blank line.
+				for j := i + 1; j < len(lines); j++ {
+					nxt := strings.TrimSpace(lines[j])
+					if nxt == "" {
+						continue
+					}
+					if strings.HasPrefix(nxt, "Test:") || strings.HasPrefix(nxt, "Messages:") {
+						break
+					}
+					content = content + " " + nxt
+					break
+				}
+			}
+			errLine = content
+			continue
+		}
+		if m := goFileLineRe.FindStringSubmatch(l); m != nil {
+			// Keep the LAST occurrence: t.Log / harness traces emit
+			// progress lines from the same file:line pattern before the
+			// failing assertion finally fires.
+			goFileLine = strings.TrimSpace(m[1])
+			continue
+		}
+		if fileHintRe.MatchString(l) {
+			lastTestLine = l
+		}
+	}
+	switch {
+	case msgLine != "":
+		return msgLine
+	case errLine != "":
+		return errLine
+	case goFileLine != "":
+		return goFileLine
+	case lastTestLine != "":
+		// "vpc_test.go:227: Eventually: condition not met …" → trim the file prefix.
+		if i := strings.Index(lastTestLine, ": "); i > 0 {
+			return strings.TrimSpace(lastTestLine[i+2:])
+		}
+		return lastTestLine
+	}
+	// Last resort: first non-empty line of the body.
+	for _, raw := range lines {
+		l := strings.TrimSpace(raw)
+		if l != "" {
+			return l
+		}
+	}
+	return ""
+}
+
+// errorTraceRe extracts the path from testify's "Error Trace:" line — e.g.
+//
+//	Error Trace:	tests/e2e/harness/ec2helpers.go:164
+//
+// — which points at the assertion site even when it lives in helper code
+// outside *_test.go.
+var errorTraceRe = regexp.MustCompile(`Error Trace:\s*(\S+\.go:\d+)`)
+
+// extractFileHint returns the most precise pointer at the assertion site
+// we can pull from the failure body: testify Error Trace wins, otherwise
+// the last `*_test.go:NNN` line in the body.
+func extractFileHint(body string) string {
+	body = stripANSI(body)
+	if m := errorTraceRe.FindStringSubmatch(body); m != nil {
+		return m[1]
+	}
+	all := fileHintRe.FindAllString(body, -1)
+	if len(all) == 0 {
+		return ""
+	}
+	return all[len(all)-1]
+}
+
+// signature normalises an error line so different runs of the same root
+// cause bucket together. Empty signatures (e.g. parent-test placeholders
+// where the failure body is empty) bucket together under "".
+func signature(errLine string) string {
+	s := strings.ToLower(errLine)
+	for _, r := range noiseReplacers {
+		s = r.re.ReplaceAllString(s, r.repl)
+	}
+	return strings.TrimSpace(s)
+}
+
+// isCascade reports whether the error line contains any of the well-known
+// fixture-propagation markers — i.e. "this test failed because something
+// upstream of it failed first".
+func isCascade(errLine string) bool {
+	for _, m := range cascadeMarkers {
+		if strings.Contains(errLine, m) {
+			return true
+		}
+	}
+	return false
+}
+
+// computeLeafSet returns the testcase names that are leaves — i.e. no
+// other testcase name in `cases` starts with `<name>/`. go-junit-report
+// emits one <testcase> for the parent Go test AND each subtest, with the
+// parent's time = sum of its children, so summing every entry into the
+// suite's cumulative offset double-counts. Restrict accumulation to
+// leaves only.
+func computeLeafSet(cases []junitTC) map[string]bool {
+	leaf := make(map[string]bool, len(cases))
+	for _, tc := range cases {
+		leaf[tc.Name] = true
+	}
+	for _, tc := range cases {
+		// Strip the deepest path segment; any ancestor of this test is
+		// not a leaf.
+		name := tc.Name
+		for {
+			i := strings.LastIndex(name, "/")
+			if i <= 0 {
+				break
+			}
+			name = name[:i]
+			if _, ok := leaf[name]; ok {
+				leaf[name] = false
+			}
+		}
+	}
+	return leaf
+}
+
+// parseStartTime turns a junit timestamp attr (RFC3339) into a time.Time;
+// zero value if unparseable.
+func parseStartTime(s string) time.Time {
+	if s == "" {
+		return time.Time{}
+	}
+	t, err := time.Parse(time.RFC3339, s)
+	if err != nil {
+		return time.Time{}
+	}
+	return t
+}
+
+// suiteLabel turns "junit-single.xml" into "single".
+func suiteLabel(path string) string {
+	base := filepath.Base(path)
+	base = strings.TrimSuffix(base, ".xml")
+	base = strings.TrimPrefix(base, "junit-")
+	if base == "" {
+		return filepath.Base(path)
+	}
+	return base
+}
+
+// ParseFile decodes one junit-*.xml file into our normalized failure list.
+// Returns the suite label, total testcase count, and the (ordered) failures.
+func ParseFile(path string, data []byte) (SuiteReport, error) {
+	var doc junitSuites
+	if err := xml.Unmarshal(data, &doc); err != nil {
+		return SuiteReport{}, fmt.Errorf("parse %s: %w", path, err)
+	}
+
+	rep := SuiteReport{
+		File:  filepath.Base(path),
+		Label: suiteLabel(path),
+	}
+
+	order := 0
+	for _, suite := range doc.Suites {
+		start := parseStartTime(suite.Timestamp)
+		// go-junit-report emits a testcase for both the parent Go test
+		// (e.g. `TestSingleNode`) AND each subtest (`TestSingleNode/PhaseX`).
+		// The parent's `time` attribute is the aggregate of its children, so
+		// summing every testcase double-counts. Pre-compute the leaf set so
+		// only leaves contribute to the suite's running offset.
+		isLeaf := computeLeafSet(suite.Cases)
+		cumul := 0.0
+		for _, tc := range suite.Cases {
+			rep.Total++
+			leaf := isLeaf[tc.Name]
+			var body string
+			switch {
+			case tc.Failure != nil:
+				body = tc.Failure.Body
+			case tc.Error != nil:
+				body = tc.Error.Body
+			default:
+				if leaf {
+					cumul += tc.Time
+				}
+				continue
+			}
+
+			// Skip the synthetic parent-test "Failed" stub that go-junit-report
+			// emits for any Go test whose subtests failed. It has an empty body
+			// and would otherwise outrank real failures by XML order.
+			trimmed := strings.TrimSpace(stripANSI(body))
+			if trimmed == "" {
+				if leaf {
+					cumul += tc.Time
+				}
+				continue
+			}
+
+			errLine := extractErrorLine(body)
+			offset := time.Duration(cumul * float64(time.Second))
+			f := Failure{
+				SuiteFile:            rep.Label,
+				Name:                 tc.Name,
+				Duration:             tc.Time,
+				Order:                order,
+				Error:                errLine,
+				FileHint:             extractFileHint(body),
+				Cascade:              isCascade(errLine),
+				Signature:            signature(errLine),
+				OffsetFromSuiteStart: offset,
+			}
+			if !start.IsZero() {
+				f.StartAt = start.Add(offset)
+			}
+			rep.FailCount++
+			rep.Cascades = append(rep.Cascades, f) // temp staging; split below
+			order++
+			if leaf {
+				cumul += tc.Time
+			}
+		}
+	}
+
+	if rep.FailCount == 0 {
+		rep.Cascades = nil
+		return rep, nil
+	}
+
+	all := rep.Cascades
+	rep.Cascades = nil
+
+	// Root = earliest non-cascade. If every failure is a cascade (the whole
+	// suite tripped on the same upstream gap), fall back to earliest overall.
+	rootIdx := -1
+	for i, f := range all {
+		if !f.Cascade {
+			rootIdx = i
+			break
+		}
+	}
+	if rootIdx == -1 {
+		rootIdx = 0
+	}
+	root := all[rootIdx]
+	rep.Root = &root
+
+	// Bucket the remainder by signature so the report can collapse the
+	// "ten failures, same bug" pattern.
+	bucketKeys := []string{}
+	buckets := map[string][]Failure{}
+	for i, f := range all {
+		if i == rootIdx {
+			continue
+		}
+		if _, seen := buckets[f.Signature]; !seen {
+			bucketKeys = append(bucketKeys, f.Signature)
+		}
+		buckets[f.Signature] = append(buckets[f.Signature], f)
+		rep.Cascades = append(rep.Cascades, f)
+	}
+
+	// Largest bucket first; ties broken by first-seen order so output is
+	// deterministic across runs.
+	sort.SliceStable(bucketKeys, func(i, j int) bool {
+		return len(buckets[bucketKeys[i]]) > len(buckets[bucketKeys[j]])
+	})
+	for _, k := range bucketKeys {
+		rep.Buckets = append(rep.Buckets, buckets[k])
+	}
+
+	return rep, nil
+}
+
+// Render writes the markdown report. Format is documented in
+// docs/development/improvements/e2e-go-failure-analysis.md (Stage 1).
+func Render(r Report) string {
+	var b strings.Builder
+	fmt.Fprintf(&b, "## %s\n\n", r.Title)
+
+	totalFail := 0
+	for _, s := range r.Suites {
+		totalFail += s.FailCount
+	}
+	if totalFail == 0 {
+		b.WriteString("✅ No failures across any suite.\n")
+		return b.String()
+	}
+
+	for _, s := range r.Suites {
+		if s.FailCount == 0 {
+			fmt.Fprintf(&b, "### Suite `%s`: ✅ pass (%d tests)\n\n", s.Label, s.Total)
+			continue
+		}
+		fmt.Fprintf(&b, "### Suite `%s`: %d failed, 1 root cause likely\n\n", s.Label, s.FailCount)
+
+		if s.Root != nil {
+			b.WriteString("**Root cause (earliest non-cascade)**\n\n")
+			fmt.Fprintf(&b, "- Test: `%s`\n", s.Root.Name)
+			if !s.Root.StartAt.IsZero() {
+				fmt.Fprintf(&b, "- Start: %s (duration %.1fs)\n",
+					s.Root.StartAt.Format("15:04:05"), s.Root.Duration)
+			} else {
+				fmt.Fprintf(&b, "- Duration: %.1fs\n", s.Root.Duration)
+			}
+			if s.Root.Error != "" {
+				fmt.Fprintf(&b, "- Error: %s\n", oneLine(s.Root.Error))
+			}
+			if s.Root.FileHint != "" {
+				fmt.Fprintf(&b, "- File:  `%s`\n", s.Root.FileHint)
+			}
+			if s.Root.BundlePath != "" {
+				fmt.Fprintf(&b, "- Bundle: `%s/`\n", s.Root.BundlePath)
+			}
+			b.WriteString("\n")
+		}
+
+		if len(s.Buckets) > 0 {
+			fmt.Fprintf(&b, "**Cascaded failures (%d) — grouped by signature**\n\n",
+				len(s.Cascades))
+			for _, bucket := range s.Buckets {
+				sample := bucket[0]
+				label := oneLine(sample.Error)
+				if label == "" {
+					label = "(no error message)"
+				}
+				fmt.Fprintf(&b, "- %d× %s\n", len(bucket), label)
+				// List up to 5 test names per bucket; collapse the rest.
+				const cap = 5
+				for i, f := range bucket {
+					if i == cap && len(bucket) > cap {
+						fmt.Fprintf(&b, "  - … and %d more\n", len(bucket)-cap)
+						break
+					}
+					if f.BundlePath != "" {
+						fmt.Fprintf(&b, "  - `%s` → `%s/`\n", f.Name, f.BundlePath)
+					} else {
+						fmt.Fprintf(&b, "  - `%s`\n", f.Name)
+					}
+				}
+			}
+			b.WriteString("\n")
+		}
+	}
+
+	return b.String()
+}
+
+// oneLine collapses whitespace and caps length so a runaway assertion
+// message doesn't blow out the summary card width.
+func oneLine(s string) string {
+	s = strings.TrimSpace(s)
+	s = strings.ReplaceAll(s, "\n", " ")
+	s = strings.ReplaceAll(s, "\r", " ")
+	s = regexp.MustCompile(`\s+`).ReplaceAllString(s, " ")
+	const max = 160
+	if len(s) > max {
+		s = s[:max-1] + "…"
+	}
+	return s
+}

.github/actions/e2e-analyze/analyze_test.go

@@ -0,0 +1,191 @@
+package main
+
+import (
+	"flag"
+	"os"
+	"path/filepath"
+	"sort"
+	"strings"
+	"testing"
+)
+
+// updateGolden lets `go test ./.github/actions/e2e-analyze/ -update` rewrite
+// the expected analysis.md files when the rendering format changes
+// intentionally. Day-to-day, leave it unset: tests diff against the on-disk
+// snapshot to catch unintended drift in signature extraction or layout.
+var updateGolden = flag.Bool("update", false, "rewrite testdata/*/analysis.md from current output")
+
+// TestGoldenFixtures runs the full ParseFile + Render pipeline against each
+// fixture directory under testdata/ and asserts the output matches the
+// adjacent analysis.md snapshot. Each subdir is one named scenario; the
+// fixture XMLs match the same junit-*.xml glob the production action uses
+// so behaviour stays representative of CI.
+func TestGoldenFixtures(t *testing.T) {
+	entries, err := os.ReadDir("testdata")
+	if err != nil {
+		t.Fatalf("read testdata: %v", err)
+	}
+	for _, e := range entries {
+		if !e.IsDir() {
+			continue
+		}
+		name := e.Name()
+		t.Run(name, func(t *testing.T) {
+			dir := filepath.Join("testdata", name)
+			matches, err := filepath.Glob(filepath.Join(dir, "junit-*.xml"))
+			if err != nil {
+				t.Fatalf("glob: %v", err)
+			}
+			sort.Strings(matches)
+			if len(matches) == 0 {
+				t.Fatalf("no junit-*.xml fixtures in %s", dir)
+			}
+
+			rep := Report{Title: "E2E failure analysis"}
+			for _, p := range matches {
+				data, err := os.ReadFile(p)
+				if err != nil {
+					t.Fatalf("read %s: %v", p, err)
+				}
+				sr, err := ParseFile(p, data)
+				if err != nil {
+					t.Fatalf("parse %s: %v", p, err)
+				}
+				rep.Suites = append(rep.Suites, sr)
+			}
+
+			got := Render(rep)
+			goldenPath := filepath.Join(dir, "analysis.md")
+			if *updateGolden {
+				if err := os.WriteFile(goldenPath, []byte(got), 0o644); err != nil {
+					t.Fatalf("write golden: %v", err)
+				}
+				return
+			}
+			want, err := os.ReadFile(goldenPath)
+			if err != nil {
+				t.Fatalf("read golden (run with -update to seed): %v", err)
+			}
+			if string(want) != got {
+				t.Errorf("rendered output differs from %s\n--- want ---\n%s\n--- got ---\n%s",
+					goldenPath, string(want), got)
+			}
+		})
+	}
+}
+
+func TestSignatureCollapsesNoise(t *testing.T) {
+	a := signature("describe-vpcs: InvalidParameterValue request-id: abc-123 vpc-deadbeefcafe1234")
+	b := signature("describe-vpcs: InvalidParameterValue request-id: xyz-999 vpc-1111222233334444")
+	if a != b {
+		t.Errorf("signatures should collapse to same bucket:\n  a=%q\n  b=%q", a, b)
+	}
+	if !strings.Contains(a, "<id>") || !strings.Contains(a, "vpc-<id>") {
+		t.Errorf("expected normalised placeholders in signature, got %q", a)
+	}
+}
+
+func TestCascadeDetection(t *testing.T) {
+	cases := []struct {
+		in   string
+		want bool
+	}{
+		{"Phase 5 must populate fix.InstanceID", true},
+		{"Should NOT be empty", true},
+		{"Expected value not to be nil", true},
+		{"describe-vpcs: InvalidParameterValue", false},
+		{"Eventually: condition not met within 3m0s", false},
+	}
+	for _, c := range cases {
+		if got := isCascade(c.in); got != c.want {
+			t.Errorf("isCascade(%q) = %v, want %v", c.in, got, c.want)
+		}
+	}
+}
+
+func TestExtractErrorLine_MessagesWinsOverError(t *testing.T) {
+	body := `    lifecycle_test.go:28:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:28
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5a_pre_ClusterStats`
+	got := extractErrorLine(body)
+	want := "Phase 5 must populate fix.InstanceID"
+	if got != want {
+		t.Errorf("extractErrorLine = %q, want %q", got, want)
+	}
+}
+
+func TestExtractErrorLine_FallsBackToLastTestLine(t *testing.T) {
+	body := "    vpc_test.go:227: Eventually: condition not met within 3m0s: [SSH handshake never completed]"
+	got := extractErrorLine(body)
+	if !strings.HasPrefix(got, "Eventually:") {
+		t.Errorf("expected fallback to test-line content, got %q", got)
+	}
+}
+
+func TestExtractFileHint_PicksLastMatch(t *testing.T) {
+	body := "ec2helpers.go:50: setup\n    vpc_test.go:227: Eventually: condition not met"
+	got := extractFileHint(body)
+	want := "vpc_test.go:227"
+	if got != want {
+		t.Errorf("extractFileHint = %q, want %q", got, want)
+	}
+}
+
+func TestParseFile_SkipsEmptyParentFailure(t *testing.T) {
+	// The synthetic "TestSingleNode" parent gets a <failure message="Failed"/>
+	// with no body whenever any subtest fails. The analyzer must skip it so
+	// it doesn't outrank the real first failure on XML order.
+	xml := []byte(`<?xml version="1.0"?>
+<testsuites><testsuite name="" tests="2" failures="2">
+  <testcase name="TestX" time="1.0"><failure message="Failed"></failure></testcase>
+  <testcase name="TestX/RealFailure" time="0.5"><failure message="Failed"><![CDATA[
+    foo_test.go:10:
+        Messages: real assertion message
+]]></failure></testcase>
+</testsuite></testsuites>`)
+	sr, err := ParseFile("junit-x.xml", xml)
+	if err != nil {
+		t.Fatalf("ParseFile: %v", err)
+	}
+	if sr.FailCount != 1 {
+		t.Fatalf("FailCount = %d, want 1 (empty parent should be skipped)", sr.FailCount)
+	}
+	if sr.Root == nil || sr.Root.Name != "TestX/RealFailure" {
+		t.Fatalf("expected real failure as root, got %+v", sr.Root)
+	}
+}
+
+func TestParseFile_CascadeFallbackWhenAllCascades(t *testing.T) {
+	// If every failure is a cascade marker, the earliest one becomes root
+	// so the report isn't blank.
+	xml := []byte(`<?xml version="1.0"?>
+<testsuites><testsuite name="" tests="2" failures="2">
+  <testcase name="TestX/A" time="1.0"><failure message="Failed"><![CDATA[
+    Messages: Phase 5 must populate fix.InstanceID
+]]></failure></testcase>
+  <testcase name="TestX/B" time="1.0"><failure message="Failed"><![CDATA[
+    Messages: Phase 5 must populate fix.InstanceID
+]]></failure></testcase>
+</testsuite></testsuites>`)
+	sr, err := ParseFile("junit-x.xml", xml)
+	if err != nil {
+		t.Fatalf("ParseFile: %v", err)
+	}
+	if sr.Root == nil || sr.Root.Name != "TestX/A" {
+		t.Fatalf("expected TestX/A as fallback root, got %+v", sr.Root)
+	}
+}
+
+func TestRender_NoFailuresIsClean(t *testing.T) {
+	out := Render(Report{
+		Title: "E2E failure analysis",
+		Suites: []SuiteReport{
+			{File: "junit-single.xml", Label: "single", Total: 3, FailCount: 0},
+		},
+	})
+	if !strings.Contains(out, "No failures") {
+		t.Errorf("expected zero-failure banner, got:\n%s", out)
+	}
+}

.github/actions/e2e-analyze/bundle.go

@@ -0,0 +1,208 @@
+package main
+
+// Stage 2 of docs/development/improvements/e2e-go-failure-analysis.md:
+// for each failure surfaced by Stage 1, materialise a per-failure bundle
+// directory containing the failure summary, a time-window slice of the
+// daemon journal, a testcase-window slice of the suite's stdout log, and
+// any per-test artifacts the harness already dumped at run-time.
+
+import (
+	"errors"
+	"fmt"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"regexp"
+	"strings"
+	"time"
+)
+
+// bundlePad is the per-failure window padding either side of the
+// [StartAt, StartAt+Duration] testcase span. Five seconds matches the
+// plan doc: long enough to catch the daemon's pre-test setup and
+// post-test cleanup chatter, short enough to keep the slice focused.
+const bundlePad = 5 * time.Second
+
+// nameSanitiseRe replaces characters unsafe in directory names with `_`.
+// Tracks tests/e2e/harness/artifacts.go which uses the same convention
+// when the test's own dump dir is created, so we can pick its contents
+// up by path without a second translation table.
+var nameSanitiseRe = regexp.MustCompile(`[^A-Za-z0-9._-]+`)
+
+func sanitiseTestName(s string) string {
+	return nameSanitiseRe.ReplaceAllString(s, "_")
+}
+
+// ApplySuiteStartFiles overrides each failure's StartAt by reading
+// <logDir>/test-<suiteLabel>.start (RFC3339 timestamp written by the
+// workflow before the suite runs). Required because go-junit-report's
+// `timestamp` attribute on <testsuite> records the time the XML was
+// produced — i.e. after the suite has finished — so it can't be used
+// as the suite wall-clock origin for journal slicing.
+//
+// Missing or unparseable files are silently ignored: the analyzer
+// falls back to the junit timestamp and the bundle still gets created,
+// it just may have an empty (or off-window) journal slice.
+func ApplySuiteStartFiles(rep *Report, logDir string) {
+	if logDir == "" {
+		return
+	}
+	for si := range rep.Suites {
+		s := &rep.Suites[si]
+		startPath := filepath.Join(logDir, fmt.Sprintf("test-%s.start", s.Label))
+		data, err := os.ReadFile(startPath)
+		if err != nil {
+			continue
+		}
+		ts, err := time.Parse(time.RFC3339, strings.TrimSpace(string(data)))
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: parse %s: %v\n", startPath, err)
+			continue
+		}
+		if s.Root != nil {
+			s.Root.StartAt = ts.Add(s.Root.OffsetFromSuiteStart)
+		}
+		for ci := range s.Cascades {
+			s.Cascades[ci].StartAt = ts.Add(s.Cascades[ci].OffsetFromSuiteStart)
+		}
+		for bi := range s.Buckets {
+			for fi := range s.Buckets[bi] {
+				f := &s.Buckets[bi][fi]
+				f.StartAt = ts.Add(f.OffsetFromSuiteStart)
+			}
+		}
+	}
+}
+
+// WriteBundles materialises per-failure bundles under <logDir>/analysis/.
+// Mutates rep in-place so each Failure.BundlePath holds the relative
+// path the renderer will link from the summary.
+//
+// Failures with no parseable wall-clock start still get a bundle dir
+// (and the testcase-window test.log slice), but no journal slice — the
+// time window is unknown.
+//
+// Errors are non-fatal: a bundle that partially fails still gets linked
+// from the report so a reader sees what's available, and the analyzer
+// keeps printing warnings so CI logs surface the gap.
+func WriteBundles(rep *Report, logDir string) {
+	if logDir == "" {
+		return
+	}
+	base := filepath.Join(logDir, "analysis")
+	if err := os.MkdirAll(base, 0o755); err != nil {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: mkdir %s: %v\n", base, err)
+		return
+	}
+
+	for si := range rep.Suites {
+		s := &rep.Suites[si]
+		if s.Root != nil {
+			writeOneBundle(logDir, base, s.Label, s.Root)
+		}
+		for ci := range s.Cascades {
+			writeOneBundle(logDir, base, s.Label, &s.Cascades[ci])
+		}
+		// Bucket entries share storage with Cascades but live in a
+		// separate slice in SuiteReport; mirror the bundle path so the
+		// renderer's bucket loop sees it too.
+		for bi := range s.Buckets {
+			for fi := range s.Buckets[bi] {
+				f := &s.Buckets[bi][fi]
+				if bp := findBundlePath(s, f.Name); bp != "" {
+					f.BundlePath = bp
+				}
+			}
+		}
+	}
+}
+
+// findBundlePath looks up a Failure's already-written BundlePath by test
+// name within a suite. Buckets reference the same logical failure as
+// Cascades but are independent slice values, so we copy the path across
+// rather than re-deriving it.
+func findBundlePath(s *SuiteReport, testName string) string {
+	if s.Root != nil && s.Root.Name == testName {
+		return s.Root.BundlePath
+	}
+	for _, c := range s.Cascades {
+		if c.Name == testName {
+			return c.BundlePath
+		}
+	}
+	return ""
+}
+
+func writeOneBundle(logDir, base, suiteLabel string, f *Failure) {
+	name := fmt.Sprintf("%02d-%s", f.Order+1, sanitiseTestName(f.Name))
+	dir := filepath.Join(base, name)
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: mkdir %s: %v\n", dir, err)
+		return
+	}
+	f.BundlePath = filepath.Join("analysis", name)
+
+	writeFailureSummary(dir, suiteLabel, f)
+
+	if !f.StartAt.IsZero() {
+		start := f.StartAt.Add(-bundlePad)
+		end := f.StartAt.Add(time.Duration(f.Duration * float64(time.Second))).Add(bundlePad)
+		journalSrc := filepath.Join(logDir, "spinifex-journal.log")
+		if _, err := SliceJournal(journalSrc, start, end, filepath.Join(dir, "journal.log")); err != nil {
+			fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: slice journal for %s: %v\n", f.Name, err)
+		}
+	}
+
+	testLogSrc := filepath.Join(logDir, fmt.Sprintf("test-%s.log", suiteLabel))
+	if _, err := SliceTestLog(testLogSrc, f.Name, filepath.Join(dir, "test.log")); err != nil {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: slice test log for %s: %v\n", f.Name, err)
+	}
+
+	// Per-test artifact subdir written by harness.ArtifactDir lives at
+	// <logDir>/<TestName with / replaced by _>/. Copy any files into
+	// the bundle so qemu/ovs/etc. dumps already collected by the test
+	// travel alongside the analysis.
+	perTestSrc := filepath.Join(logDir, strings.ReplaceAll(f.Name, "/", "_"))
+	if entries, err := os.ReadDir(perTestSrc); err == nil {
+		for _, e := range entries {
+			if e.IsDir() {
+				continue
+			}
+			_ = copyFile(filepath.Join(perTestSrc, e.Name()), filepath.Join(dir, e.Name()))
+		}
+	} else if !os.IsNotExist(err) && !errorsIsNotDir(err) {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: read per-test dir %s: %v\n", perTestSrc, err)
+	}
+}
+
+// errorsIsNotDir handles the case where the harness wrote a file at the
+// per-test name instead of a directory — best-effort skip.
+func errorsIsNotDir(err error) bool {
+	var pe *fs.PathError
+	if !errors.As(err, &pe) {
+		return false
+	}
+	return pe.Err != nil && strings.Contains(pe.Err.Error(), "not a directory")
+}
+
+func writeFailureSummary(dir, suiteLabel string, f *Failure) {
+	var b strings.Builder
+	fmt.Fprintf(&b, "Suite:    %s\n", suiteLabel)
+	fmt.Fprintf(&b, "Test:     %s\n", f.Name)
+	if !f.StartAt.IsZero() {
+		fmt.Fprintf(&b, "Start:    %s\n", f.StartAt.Format(time.RFC3339))
+	}
+	fmt.Fprintf(&b, "Duration: %.3fs\n", f.Duration)
+	if f.FileHint != "" {
+		fmt.Fprintf(&b, "Site:     %s\n", f.FileHint)
+	}
+	if f.Cascade {
+		b.WriteString("Cascade:  yes (downstream of another failure)\n")
+	}
+	if f.Error != "" {
+		b.WriteString("\nError:\n")
+		b.WriteString(oneLine(f.Error))
+		b.WriteString("\n")
+	}
+	_ = os.WriteFile(filepath.Join(dir, "failure.txt"), []byte(b.String()), 0o644)
+}

.github/actions/e2e-analyze/bundle_test.go

@@ -0,0 +1,229 @@
+package main
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+func TestSliceJournal_KeepsOnlyWindow(t *testing.T) {
+	src := filepath.Join(t.TempDir(), "journal.log")
+	dst := filepath.Join(t.TempDir(), "out.log")
+	body := strings.Join([]string{
+		"2026-05-19T12:32:30+1000 host spinifex-daemon[1]: pre",
+		"  continuation of pre",
+		"2026-05-19T12:32:35+1000 host spinifex-daemon[1]: IN window",
+		"  multiline inside window",
+		"2026-05-19T12:32:45+1000 host spinifex-daemon[1]: post",
+		"",
+	}, "\n")
+	if err := os.WriteFile(src, []byte(body), 0o644); err != nil {
+		t.Fatalf("write src: %v", err)
+	}
+	start, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:33+10:00")
+	end, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:40+10:00")
+	n, err := SliceJournal(src, start, end, dst)
+	if err != nil {
+		t.Fatalf("SliceJournal: %v", err)
+	}
+	if n != 2 {
+		t.Fatalf("written = %d, want 2", n)
+	}
+	got, _ := os.ReadFile(dst)
+	if !strings.Contains(string(got), "IN window") || !strings.Contains(string(got), "multiline inside window") {
+		t.Errorf("missing in-window content:\n%s", got)
+	}
+	if strings.Contains(string(got), "pre") || strings.Contains(string(got), "post") {
+		t.Errorf("kept out-of-window content:\n%s", got)
+	}
+}
+
+func TestSliceJournal_AcceptsRFC3339ColonOffset(t *testing.T) {
+	// systemd's `journalctl --output=short-iso` ships RFC3339 timestamps
+	// (offset includes a colon, e.g. "+10:00") on most versions. Earlier
+	// regex only matched "+1000" — make sure both forms work.
+	src := filepath.Join(t.TempDir(), "journal.log")
+	dst := filepath.Join(t.TempDir(), "out.log")
+	body := strings.Join([]string{
+		"2026-05-19T12:32:30+10:00 host spinifex-daemon[1]: pre",
+		"2026-05-19T12:32:35+10:00 host spinifex-daemon[1]: IN window",
+		"2026-05-19T12:32:50+10:00 host spinifex-daemon[1]: post",
+		"",
+	}, "\n")
+	if err := os.WriteFile(src, []byte(body), 0o644); err != nil {
+		t.Fatalf("write src: %v", err)
+	}
+	start, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:33+10:00")
+	end, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:40+10:00")
+	n, err := SliceJournal(src, start, end, dst)
+	if err != nil {
+		t.Fatalf("SliceJournal: %v", err)
+	}
+	if n != 1 {
+		t.Fatalf("written = %d, want 1", n)
+	}
+	got, _ := os.ReadFile(dst)
+	if !strings.Contains(string(got), "IN window") {
+		t.Errorf("missing in-window content:\n%s", got)
+	}
+}
+
+func TestSliceJournal_MissingSrcIsNotError(t *testing.T) {
+	dst := filepath.Join(t.TempDir(), "out.log")
+	n, err := SliceJournal("/no/such/file", time.Now(), time.Now(), dst)
+	if err != nil {
+		t.Fatalf("unexpected err: %v", err)
+	}
+	if n != 0 {
+		t.Errorf("written = %d, want 0", n)
+	}
+}
+
+func TestSliceTestLog_PicksOneTestcase(t *testing.T) {
+	src := filepath.Join(t.TempDir(), "test.log")
+	dst := filepath.Join(t.TempDir(), "out.log")
+	body := strings.Join([]string{
+		"=== RUN   TestSingleNode",
+		"  before subtests",
+		"=== RUN   TestSingleNode/Phase5_LaunchInstance",
+		"    runinstances",
+		"    ec2helpers.go:164: describe-vpcs: InvalidParameterValue",
+		"--- FAIL: TestSingleNode/Phase5_LaunchInstance (2.50s)",
+		"=== RUN   TestSingleNode/Phase6_Volumes",
+		"    other content",
+		"--- FAIL: TestSingleNode/Phase6_Volumes (0.10s)",
+		"--- FAIL: TestSingleNode (10.0s)",
+		"",
+	}, "\n")
+	if err := os.WriteFile(src, []byte(body), 0o644); err != nil {
+		t.Fatalf("write src: %v", err)
+	}
+	n, err := SliceTestLog(src, "TestSingleNode/Phase5_LaunchInstance", dst)
+	if err != nil {
+		t.Fatalf("SliceTestLog: %v", err)
+	}
+	if n != 4 {
+		t.Fatalf("written = %d, want 4 (RUN + 2 inner + FAIL)", n)
+	}
+	got, _ := os.ReadFile(dst)
+	s := string(got)
+	if !strings.Contains(s, "describe-vpcs: InvalidParameterValue") {
+		t.Errorf("missing failure line:\n%s", s)
+	}
+	if strings.Contains(s, "Phase6_Volumes") || strings.Contains(s, "before subtests") {
+		t.Errorf("leaked sibling test content:\n%s", s)
+	}
+}
+
+func TestSliceTestLog_UnknownTestcaseIsEmpty(t *testing.T) {
+	src := filepath.Join(t.TempDir(), "test.log")
+	dst := filepath.Join(t.TempDir(), "out.log")
+	if err := os.WriteFile(src, []byte("=== RUN   TestX\n--- PASS: TestX (0.0s)\n"), 0o644); err != nil {
+		t.Fatalf("write src: %v", err)
+	}
+	n, err := SliceTestLog(src, "TestY", dst)
+	if err != nil {
+		t.Fatalf("SliceTestLog: %v", err)
+	}
+	if n != 0 {
+		t.Errorf("written = %d, want 0", n)
+	}
+	if _, err := os.Stat(dst); err != nil {
+		t.Errorf("dst not created: %v", err)
+	}
+}
+
+// TestWriteBundles_EndToEnd builds a fake artifact dir with a junit XML,
+// a journal log, a test log, and a per-test artifact subdir, then runs
+// the full ParseFile → WriteBundles pipeline and asserts the bundle dir
+// contains everything the report links to.
+func TestWriteBundles_EndToEnd(t *testing.T) {
+	dir := t.TempDir()
+
+	junit := `<?xml version="1.0"?>
+<testsuites><testsuite name="" tests="2" failures="1" timestamp="2026-05-19T12:32:30+10:00" time="3.0">
+  <testcase name="TestSingleNode/Phase5_LaunchInstance" time="2.500">
+    <failure message="Failed"><![CDATA[
+    ec2helpers.go:164: describe-vpcs: InvalidParameterValue (status 400)
+        Error Trace: tests/e2e/harness/ec2helpers.go:164
+        Test: TestSingleNode/Phase5_LaunchInstance]]></failure>
+  </testcase>
+</testsuite></testsuites>`
+	if err := os.WriteFile(filepath.Join(dir, "junit-single.xml"), []byte(junit), 0o644); err != nil {
+		t.Fatalf("write junit: %v", err)
+	}
+
+	journal := strings.Join([]string{
+		"2026-05-19T12:32:20+1000 host spinifex-daemon[1]: too-early",
+		"2026-05-19T12:32:30+1000 host spinifex-daemon[1]: in-window-start",
+		"2026-05-19T12:32:32+1000 host spinifex-daemon[1]: describe-vpcs filter=is_default",
+		"2026-05-19T12:32:50+1000 host spinifex-daemon[1]: too-late",
+		"",
+	}, "\n")
+	if err := os.WriteFile(filepath.Join(dir, "spinifex-journal.log"), []byte(journal), 0o644); err != nil {
+		t.Fatalf("write journal: %v", err)
+	}
+
+	testlog := strings.Join([]string{
+		"=== RUN   TestSingleNode/Phase5_LaunchInstance",
+		"    ec2helpers.go:164: describe-vpcs: InvalidParameterValue",
+		"--- FAIL: TestSingleNode/Phase5_LaunchInstance (2.50s)",
+		"",
+	}, "\n")
+	if err := os.WriteFile(filepath.Join(dir, "test-single.log"), []byte(testlog), 0o644); err != nil {
+		t.Fatalf("write test log: %v", err)
+	}
+
+	// Per-test artifact subdir (matches harness.ArtifactDir convention).
+	perTest := filepath.Join(dir, "TestSingleNode_Phase5_LaunchInstance")
+	if err := os.MkdirAll(perTest, 0o755); err != nil {
+		t.Fatalf("mkdir per-test: %v", err)
+	}
+	if err := os.WriteFile(filepath.Join(perTest, "qemu-console.log"), []byte("[qemu] boot\n"), 0o644); err != nil {
+		t.Fatalf("write per-test file: %v", err)
+	}
+
+	data, _ := os.ReadFile(filepath.Join(dir, "junit-single.xml"))
+	sr, err := ParseFile("junit-single.xml", data)
+	if err != nil {
+		t.Fatalf("ParseFile: %v", err)
+	}
+	rep := Report{Title: "E2E failure analysis", Suites: []SuiteReport{sr}}
+	WriteBundles(&rep, dir)
+
+	if rep.Suites[0].Root == nil {
+		t.Fatal("no root")
+	}
+	bp := rep.Suites[0].Root.BundlePath
+	if bp == "" {
+		t.Fatal("BundlePath not set")
+	}
+	bundle := filepath.Join(dir, bp)
+	for _, want := range []string{"failure.txt", "journal.log", "test.log", "qemu-console.log"} {
+		if _, err := os.Stat(filepath.Join(bundle, want)); err != nil {
+			t.Errorf("missing %s in bundle %s: %v", want, bp, err)
+		}
+	}
+
+	jl, _ := os.ReadFile(filepath.Join(bundle, "journal.log"))
+	if !strings.Contains(string(jl), "describe-vpcs filter=is_default") {
+		t.Errorf("journal.log lacks in-window line:\n%s", jl)
+	}
+	if strings.Contains(string(jl), "too-early") || strings.Contains(string(jl), "too-late") {
+		t.Errorf("journal.log includes out-of-window content:\n%s", jl)
+	}
+
+	tl, _ := os.ReadFile(filepath.Join(bundle, "test.log"))
+	if !strings.Contains(string(tl), "--- FAIL: TestSingleNode/Phase5_LaunchInstance") {
+		t.Errorf("test.log missing FAIL marker:\n%s", tl)
+	}
+
+	out := Render(rep)
+	wantLink := fmt.Sprintf("- Bundle: `%s/`", bp)
+	if !strings.Contains(out, wantLink) {
+		t.Errorf("render missing bundle link %q:\n%s", wantLink, out)
+	}
+}

.github/actions/e2e-analyze/main.go

@@ -0,0 +1,78 @@
+package main
+
+import (
+	"flag"
+	"fmt"
+	"os"
+	"path/filepath"
+	"sort"
+)
+
+func main() {
+	var junitGlob, logDir, title string
+	flag.StringVar(&junitGlob, "junit-glob", os.Getenv("ANALYZE_JUNIT_GLOB"), "glob matching junit-*.xml files")
+	flag.StringVar(&logDir, "log-dir", os.Getenv("ANALYZE_LOG_DIR"), "artifact dir to write analysis.md into")
+	flag.StringVar(&title, "title", os.Getenv("ANALYZE_TITLE"), "report heading")
+	flag.Parse()
+
+	if title == "" {
+		title = "E2E failure analysis"
+	}
+	if junitGlob == "" {
+		fmt.Fprintln(os.Stderr, "::error::e2e-analyze: -junit-glob is required")
+		os.Exit(0) // analyzer is advisory only; existing gate owns exit code
+	}
+
+	paths, err := filepath.Glob(junitGlob)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: bad glob %q: %v\n", junitGlob, err)
+		os.Exit(0)
+	}
+	sort.Strings(paths)
+
+	report := Report{Title: title}
+	for _, p := range paths {
+		data, err := os.ReadFile(p)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: read %s: %v\n", p, err)
+			continue
+		}
+		sr, err := ParseFile(p, data)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: %v\n", err)
+			continue
+		}
+		report.Suites = append(report.Suites, sr)
+	}
+
+	// Stage 2: prefer authoritative per-suite start times written by the
+	// workflow over junit-report's conversion-time `timestamp` attribute,
+	// then materialise per-failure bundles so Render can link them.
+	if logDir != "" {
+		ApplySuiteStartFiles(&report, logDir)
+		WriteBundles(&report, logDir)
+	}
+
+	out := Render(report)
+
+	if logDir != "" {
+		if err := os.MkdirAll(logDir, 0o755); err == nil {
+			if err := os.WriteFile(filepath.Join(logDir, "analysis.md"), []byte(out), 0o644); err != nil {
+				fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: write analysis.md: %v\n", err)
+			}
+		}
+	}
+
+	if summaryPath := os.Getenv("GITHUB_STEP_SUMMARY"); summaryPath != "" {
+		f, err := os.OpenFile(summaryPath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
+		if err == nil {
+			defer f.Close()
+			if _, werr := f.WriteString(out); werr != nil {
+				fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: append summary: %v\n", werr)
+			}
+		}
+	} else {
+		// Local invocation (developer running the tool by hand). Mirror to stdout.
+		fmt.Print(out)
+	}
+}

.github/actions/e2e-analyze/slice.go

@@ -0,0 +1,176 @@
+package main
+
+// Stage 2 of docs/development/improvements/e2e-go-failure-analysis.md:
+// per-failure log slicers used by bundle.go to materialise time-window /
+// testcase-window log excerpts next to each named failure.
+
+import (
+	"bufio"
+	"io"
+	"os"
+	"regexp"
+	"strings"
+	"time"
+)
+
+// shortIsoTSRe matches the timestamp prefix produced by
+// `journalctl --output=short-iso`. The offset format varies across
+// systemd versions: most ship "+10:00" (RFC3339), some ship "+1000"
+// (no colon). Match either — the parser below picks the right layout.
+var shortIsoTSRe = regexp.MustCompile(`^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:[+-]\d{2}:?\d{2}|Z))\s`)
+
+// shortIsoLayouts are tried in order; the first parse that succeeds wins.
+var shortIsoLayouts = []string{
+	time.RFC3339,               // 2026-05-19T12:32:30+10:00
+	"2006-01-02T15:04:05-0700", // 2026-05-19T12:32:30+1000
+}
+
+func parseShortIso(s string) (time.Time, error) {
+	var lastErr error
+	for _, layout := range shortIsoLayouts {
+		t, err := time.Parse(layout, s)
+		if err == nil {
+			return t, nil
+		}
+		lastErr = err
+	}
+	return time.Time{}, lastErr
+}
+
+// SliceJournal copies lines from src whose `short-iso` timestamp falls in
+// [start, end] to dst. Lines without a parseable timestamp inherit the
+// previous line's classification — that keeps multi-line journal records
+// (stack traces, panics) intact rather than orphaning their continuation
+// lines. Lines that precede the first parseable timestamp are dropped.
+//
+// Returns the number of lines written; missing src is not an error (the
+// journal may be empty if collection failed).
+func SliceJournal(src string, start, end time.Time, dst string) (int, error) {
+	f, err := os.Open(src)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return 0, nil
+		}
+		return 0, err
+	}
+	defer f.Close()
+
+	out, err := os.Create(dst)
+	if err != nil {
+		return 0, err
+	}
+	defer out.Close()
+
+	bw := bufio.NewWriter(out)
+	defer bw.Flush()
+
+	sc := bufio.NewScanner(f)
+	// Journal lines can be long (panic dumps); bump scanner buffer.
+	sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
+
+	keep := false
+	written := 0
+	for sc.Scan() {
+		line := sc.Text()
+		if m := shortIsoTSRe.FindStringSubmatch(line); m != nil {
+			ts, err := parseShortIso(m[1])
+			if err != nil {
+				continue
+			}
+			keep = !ts.Before(start) && !ts.After(end)
+		}
+		if keep {
+			if _, err := bw.WriteString(line); err != nil {
+				return written, err
+			}
+			if err := bw.WriteByte('\n'); err != nil {
+				return written, err
+			}
+			written++
+		}
+	}
+	return written, sc.Err()
+}
+
+// runRePrefix and finishRePrefix bracket a single testcase's output in
+// `go test -v` stdout. Subtests have their own `=== RUN` / `--- FAIL`
+// markers, so slicing on the exact testcase name yields just that test's
+// lines (including any nested phase logs).
+var (
+	runRe    = regexp.MustCompile(`^=== RUN\s+(\S+)$`)
+	finishRe = regexp.MustCompile(`^\s*---\s+(PASS|FAIL|SKIP):\s+(\S+)\s+\(`)
+)
+
+// SliceTestLog copies lines belonging to the named testcase from src to
+// dst. The block starts at `=== RUN <name>` and ends at the matching
+// `--- PASS|FAIL|SKIP: <name> (…)` line (inclusive). If the testcase is
+// never seen the destination file is created empty so the bundle has a
+// stable shape.
+//
+// Returns the number of lines written; missing src is not an error.
+func SliceTestLog(src, testName, dst string) (int, error) {
+	f, err := os.Open(src)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return 0, nil
+		}
+		return 0, err
+	}
+	defer f.Close()
+
+	out, err := os.Create(dst)
+	if err != nil {
+		return 0, err
+	}
+	defer out.Close()
+
+	bw := bufio.NewWriter(out)
+	defer bw.Flush()
+
+	sc := bufio.NewScanner(f)
+	sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
+
+	in := false
+	written := 0
+	for sc.Scan() {
+		line := sc.Text()
+		if !in {
+			if m := runRe.FindStringSubmatch(strings.TrimRight(line, "\r")); m != nil && m[1] == testName {
+				in = true
+			} else {
+				continue
+			}
+		}
+		if _, err := bw.WriteString(line); err != nil {
+			return written, err
+		}
+		if err := bw.WriteByte('\n'); err != nil {
+			return written, err
+		}
+		written++
+		if m := finishRe.FindStringSubmatch(line); m != nil && m[2] == testName {
+			return written, nil
+		}
+	}
+	return written, sc.Err()
+}
+
+// copyFile is a small helper used when bundling pre-collected per-test
+// artifacts. Best-effort: missing src is silently skipped.
+func copyFile(src, dst string) error {
+	in, err := os.Open(src)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return nil
+		}
+		return err
+	}
+	defer in.Close()
+	out, err := os.Create(dst)
+	if err != nil {
+		return err
+	}
+	defer out.Close()
+	_, err = io.Copy(out, in)
+	return err
+}

.github/actions/e2e-analyze/testdata/cascade-vpcs/analysis.md

@@ -0,0 +1,23 @@
+## E2E failure analysis
+
+### Suite `single`: 16 failed, 1 root cause likely
+
+**Root cause (earliest non-cascade)**
+
+- Test: `TestSingleNode/Phase5_LaunchInstance`
+- Start: 12:30:00 (duration 2.5s)
+- Error: Received unexpected error: InvalidParameterValue (HTTP 400)
+- File:  `tests/e2e/harness/ec2helpers.go:164`
+
+**Cascaded failures (15) — grouped by signature**
+
+- 14× Phase 5 must populate fix.InstanceID
+  - `TestSingleNode/Phase5a_pre_ClusterStats`
+  - `TestSingleNode/Phase5a_Metadata`
+  - `TestSingleNode/Phase5b_DescribeInstances`
+  - `TestSingleNode/Phase5c_StopStart`
+  - `TestSingleNode/Phase6_Volumes`
+  - … and 9 more
+- 1× Phase 2 must populate fix.AZName
+  - `TestSingleNode/Phase8_NatGW`
+

.github/actions/e2e-analyze/testdata/cascade-vpcs/junit-single.xml

@@ -0,0 +1,143 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<testsuites tests="20" failures="17" skipped="0">
+	<testsuite name="" tests="20" failures="17" errors="0" id="0" hostname="banksia" skipped="0" time="600.000" timestamp="2026-04-12T12:30:00+10:00">
+		<testcase name="TestSingleNode" classname="" time="600.000">
+			<failure message="Failed"></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase1_Environment" classname="" time="0.300">
+			<system-out><![CDATA[
+&#x2501;&#x2501; Phase 1 &#x2501;&#x2501;]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase2_Discovery" classname="" time="0.010"/>
+		<testcase name="TestSingleNode/Phase3_KeyPairs" classname="" time="0.540"/>
+		<testcase name="TestSingleNode/Phase4_Image" classname="" time="0.060"/>
+		<testcase name="TestSingleNode/Phase5_LaunchInstance" classname="" time="2.500">
+			<failure message="Failed"><![CDATA[
+    ec2helpers.go:164: describe-vpcs: InvalidParameterValue: Filter 'is_default' is invalid (status 400)
+    	Error Trace:	tests/e2e/harness/ec2helpers.go:164
+    	Error:      	Received unexpected error:
+    	            	InvalidParameterValue (HTTP 400)
+    	Test:       	TestSingleNode/Phase5_LaunchInstance]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_pre_ClusterStats" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    lifecycle_test.go:28:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:28
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5a_pre_ClusterStats]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_Metadata" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    lifecycle_test.go:57:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:57
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5a_Metadata]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5b_DescribeInstances" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    lifecycle_test.go:109:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:109
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5b_DescribeInstances]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5c_StopStart" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    lifecycle_test.go:222:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:222
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5c_StopStart]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase6_Volumes" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    volume_test.go:23:
+        Error Trace:	tests/e2e/single/volume_test.go:23
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase6_Volumes]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase6b_AttachVolume" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    volume_test.go:24:
+        Error Trace:	tests/e2e/single/volume_test.go:24
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase6b_AttachVolume]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7_Tags" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    tags_test.go:23:
+        Error Trace:	tests/e2e/single/tags_test.go:23
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7_Tags]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7a_Snapshots" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    snapshot_test.go:23:
+        Error Trace:	tests/e2e/single/snapshot_test.go:23
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7a_Snapshots]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7b_DescribeInstanceAttribute" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    instance_test.go:81:
+        Error Trace:	tests/e2e/single/instance_test.go:81
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7b_DescribeInstanceAttribute]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7c_ModifyInstanceAttribute" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    instance_test.go:92:
+        Error Trace:	tests/e2e/single/instance_test.go:92
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7c_ModifyInstanceAttribute]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7d_Console" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    instance_test.go:121:
+        Error Trace:	tests/e2e/single/instance_test.go:121
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7d_Console]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7e_Negative" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    negative_test.go:43:
+        Error Trace:	tests/e2e/single/negative_test.go:43
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7e_Negative]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NatGW" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    natgw_test.go:39:
+        Error Trace:	tests/e2e/single/natgw_test.go:39
+        Error:      	Should NOT be empty
+        Messages:   	Phase 2 must populate fix.AZName
+        Test:       	TestSingleNode/Phase8_NatGW]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8a_DataPath" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    datapath_test.go:48:
+        Error Trace:	tests/e2e/single/datapath_test.go:48
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase8a_DataPath]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase9_SecurityGroups" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    securitygroup_test.go:35:
+        Error Trace:	tests/e2e/single/securitygroup_test.go:35
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase9_SecurityGroups]]></failure>
+		</testcase>
+	</testsuite>
+</testsuites>

.github/actions/e2e-analyze/testdata/no-failures/analysis.md

@@ -0,0 +1,3 @@
+## E2E failure analysis
+
+✅ No failures across any suite.

.github/actions/e2e-analyze/testdata/no-failures/junit-single.xml

@@ -0,0 +1,8 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<testsuites tests="3" failures="0" skipped="0">
+	<testsuite name="" tests="3" failures="0" errors="0" id="0" hostname="banksia" skipped="0" time="1.500" timestamp="2026-04-12T12:30:00+10:00">
+		<testcase name="TestSingleNode" classname="" time="1.500"/>
+		<testcase name="TestSingleNode/Phase1_Environment" classname="" time="0.300"/>
+		<testcase name="TestSingleNode/Phase2_Discovery" classname="" time="1.200"/>
+	</testsuite>
+</testsuites>

.github/actions/e2e-analyze/testdata/single-failure/analysis.md

@@ -0,0 +1,11 @@
+## E2E failure analysis
+
+### Suite `single`: 1 failed, 1 root cause likely
+
+**Root cause (earliest non-cascade)**
+
+- Test: `TestSingleNode/Phase8b_VPCSubnetE2E`
+- Start: 00:26:18 (duration 200.5s)
+- Error: Eventually: condition not met within 3m0s: [SSH handshake 192.168.0.202:22 never completed]
+- File:  `vpc_test.go:227`
+

.github/actions/e2e-analyze/testdata/single-failure/junit-single.xml

@@ -0,0 +1,993 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<testsuites tests="79" failures="2" skipped="6">
+	<testsuite name="" tests="79" failures="2" errors="0" id="0" hostname="banksia" skipped="6" time="1708.140" timestamp="2026-05-19T00:19:45+10:00">
+		<testcase name="TestSingleNode" classname="" time="800.320">
+			<failure message="Failed"></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase1_Environment" classname="" time="0.300">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 1 — Environment ━━�[0m
+�[2m· checking /dev/kvm�[0m
+    discovery_test.go:29: �[2m· checking /dev/kvm�[0m
+�[2mkvm�[0m=writable
+    discovery_test.go:38: �[2mkvm�[0m=writable
+�[2m· waiting for AWS gateway�[0m
+    discovery_test.go:45: �[2m· waiting for AWS gateway�[0m
+�[2mgateway�[0m=https://192.168.0.10:9999/
+    discovery_test.go:55: �[2mgateway�[0m=https://192.168.0.10:9999/
+�[2m· waiting for daemon NATS subscriptions�[0m
+    discovery_test.go:60: �[2m· waiting for daemon NATS subscriptions�[0m
+�[2m· describe-regions / describe-availability-zones�[0m
+    discovery_test.go:73: �[2m· describe-regions / describe-availability-zones�[0m
+�[2maz�[0m=ap-southeast-2a �[2mregion�[0m=ap-southeast-2
+    discovery_test.go:84: �[2maz�[0m=ap-southeast-2a �[2mregion�[0m=ap-southeast-2]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase1b_ClusterStats" classname="" time="9.200">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 1b — Cluster Stats CLI ━━�[0m
+�[2m· spx get nodes�[0m
+    discovery_test.go:101: �[2m· spx get nodes�[0m
+�[2m· spx top nodes�[0m
+    discovery_test.go:105: �[2m· spx top nodes�[0m
+�[2m· spx get vms (empty before any launches)�[0m
+    discovery_test.go:112: �[2m· spx get vms (empty before any launches)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase2_Discovery" classname="" time="0.010">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 2 — Discovery & Metadata ━━�[0m
+�[2m· discovering nano instance type�[0m
+    discovery_test.go:133: �[2m· discovering nano instance type�[0m
+�[2minstance_type�[0m=t3.nano �[2march�[0m=x86_64 �[2maz�[0m=ap-southeast-2a
+    discovery_test.go:159: �[2minstance_type�[0m=t3.nano �[2march�[0m=x86_64 �[2maz�[0m=ap-southeast-2a]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase2b_SerialConsole" classname="" time="0.020">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 2b — Serial Console Access ━━�[0m
+�[2m· default state should be disabled�[0m
+    discovery_test.go:168: �[2m· default state should be disabled�[0m
+�[2m· enable�[0m
+    discovery_test.go:173: �[2m· enable�[0m
+�[2mstate�[0m=enabled
+    discovery_test.go:180: �[2mstate�[0m=enabled
+�[2m· disable�[0m
+    discovery_test.go:182: �[2m· disable�[0m
+�[2mstate�[0m=disabled
+    discovery_test.go:189: �[2mstate�[0m=disabled]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase3_KeyPairs" classname="" time="0.540">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 3 — SSH Key Management ━━�[0m
+�[2m· create-key-pair "test-key-1"�[0m
+    keypair_test.go:37: �[2m· create-key-pair "test-key-1"�[0m
+�[2mkey�[0m=test-key-1 �[2mpem�[0m=/tmp/TestSingleNode1289015487/001/test-key-1.pem
+    keypair_test.go:51: �[2mkey�[0m=test-key-1 �[2mpem�[0m=/tmp/TestSingleNode1289015487/001/test-key-1.pem
+�[2m· import-key-pair "test-key-2" (local-generated RSA)�[0m
+    keypair_test.go:58: �[2m· import-key-pair "test-key-2" (local-generated RSA)�[0m
+�[2mimported�[0m=test-key-2
+    keypair_test.go:65: �[2mimported�[0m=test-key-2
+�[2m· describe-key-pairs (both present)�[0m
+    keypair_test.go:67: �[2m· describe-key-pairs (both present)�[0m
+�[2m· delete "test-key-2"�[0m
+    keypair_test.go:72: �[2m· delete "test-key-2"�[0m
+�[2m· describe-key-pairs (only test-key-1 remains)�[0m
+    keypair_test.go:76: �[2m· describe-key-pairs (only test-key-1 remains)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase4_Image" classname="" time="0.060">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 4 — Image Management ━━�[0m
+�[2mimage�[0m=ubuntu-24.04-x86_64 �[2march�[0m=x86_64
+    image_test.go:33: �[2mimage�[0m=ubuntu-24.04-x86_64 �[2march�[0m=x86_64
+�[2m· spx admin images import --name ubuntu-24.04-x86_64�[0m
+    image_test.go:35: �[2m· spx admin images import --name ubuntu-24.04-x86_64�[0m
+�[2m· spx import did not yield AMI ID; falling back to describe-images�[0m
+    image_test.go:48: �[2m· spx import did not yield AMI ID; falling back to describe-images�[0m
+�[2m· describe-images by AMI ID (verify exactly one match)�[0m
+    image_test.go:78: �[2m· describe-images by AMI ID (verify exactly one match)�[0m
+�[2mami�[0m=ami-6f60f2b3bf04b5453 �[2mname�[0m=ami-ubuntu-24.04-x86_64
+    image_test.go:86: �[2mami�[0m=ami-6f60f2b3bf04b5453 �[2mname�[0m=ami-ubuntu-24.04-x86_64]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5_LaunchInstance" classname="" time="10.830">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5 — Instance Lifecycle ━━�[0m
+�[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msg�[0m=sg-764a7fa2a39577204 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63
+    instance_test.go:31: �[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msg�[0m=sg-764a7fa2a39577204 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63
+�[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.nano key=test-key-1�[0m
+    instance_test.go:35: �[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.nano key=test-key-1�[0m
+�[2minstance�[0m=i-0c615e5dd74050f53
+    instance_test.go:48: �[2minstance�[0m=i-0c615e5dd74050f53
+    instance_test.go:50: instance i-0c615e5dd74050f53 reached state running
+�[2mroot_volume�[0m=vol-2230138124fb51ee2 �[2mroot_device�[0m=
+    instance_test.go:70: �[2mroot_volume�[0m=vol-2230138124fb51ee2 �[2mroot_device�[0m=]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_pre_ClusterStats" classname="" time="3.040">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5a-pre — Cluster Stats CLI (with running VM) ━━�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_Metadata" classname="" time="0.510">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5a — Instance Metadata Validation ━━�[0m
+�[2mtype�[0m=t3.nano �[2mkey�[0m=test-key-1 �[2mimage�[0m=ami-6f60f2b3bf04b5453 �[2mbdms�[0m=1
+    instance_test.go:108: �[2mtype�[0m=t3.nano �[2mkey�[0m=test-key-1 �[2mimage�[0m=ami-6f60f2b3bf04b5453 �[2mbdms�[0m=1]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_ii_SSH" classname="" time="49.450">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5a-ii — SSH Connectivity & Volume Verification ━━�[0m
+�[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
+    instance_test.go:127: �[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    instance_test.go:130: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    lifecycle_test.go:384: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2m· ssh id�[0m
+    instance_test.go:136: �[2m· ssh id�[0m
+�[2m· lsblk root-volume cross-check vs API�[0m
+    instance_test.go:140: �[2m· lsblk root-volume cross-check vs API�[0m
+�[2mguest_gib�[0m=4 �[2mapi_gib�[0m=4
+    instance_test.go:159: �[2mguest_gib�[0m=4 �[2mapi_gib�[0m=4
+�[2m· ssh hostname�[0m
+    instance_test.go:161: �[2m· ssh hostname�[0m
+�[2mhostname�[0m=spinifex-vm-0c615e5d �[2mmatches_short_id�[0m=0c615e5d
+    instance_test.go:172: �[2mhostname�[0m=spinifex-vm-0c615e5d �[2mmatches_short_id�[0m=0c615e5d]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_iii_Console" classname="" time="0.010">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5a-iii — Console Output ━━�[0m
+�[2minstance�[0m=i-0c615e5dd74050f53 �[2mhas_output�[0m=true
+    instance_test.go:191: �[2minstance�[0m=i-0c615e5dd74050f53 �[2mhas_output�[0m=true]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5b_Volume" classname="" time="4.540">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5b — Volume Lifecycle (Attach/Detach) ━━�[0m
+�[2m· create-volume size=10 az=ap-southeast-2a�[0m
+    volume_test.go:26: �[2m· create-volume size=10 az=ap-southeast-2a�[0m
+�[2mvolume�[0m=vol-a885f79173e0a8030
+    volume_test.go:34: �[2mvolume�[0m=vol-a885f79173e0a8030
+    volume_test.go:39: volume vol-a885f79173e0a8030 reached state available
+�[2m· modify-volume size=20�[0m
+    volume_test.go:42: �[2m· modify-volume size=20�[0m
+�[2mresized_gib�[0m=20
+    volume_test.go:66: �[2mresized_gib�[0m=20
+�[2m· attach-volume vol-a885f79173e0a8030 -> i-0c615e5dd74050f53 as /dev/sdf�[0m
+    volume_test.go:68: �[2m· attach-volume vol-a885f79173e0a8030 -> i-0c615e5dd74050f53 as /dev/sdf�[0m
+    volume_test.go:76: volume vol-a885f79173e0a8030 reached state in-use
+�[2m· detach-volume vol-a885f79173e0a8030�[0m
+    volume_test.go:90: �[2m· detach-volume vol-a885f79173e0a8030�[0m
+    volume_test.go:96: volume vol-a885f79173e0a8030 reached state available
+�[2m· delete-volume vol-a885f79173e0a8030�[0m
+    volume_test.go:98: �[2m· delete-volume vol-a885f79173e0a8030�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5b_ii_VolumeStatus" classname="" time="0.000">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5b-ii — DescribeVolumeStatus ━━�[0m
+�[2mvolume�[0m=vol-2230138124fb51ee2 �[2mstatus�[0m=ok
+    volume_test.go:158: �[2mvolume�[0m=vol-2230138124fb51ee2 �[2mstatus�[0m=ok]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5c_Snapshot" classname="" time="0.190">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5c — Snapshot Lifecycle ━━�[0m
+�[2m· create-snapshot volume=vol-2230138124fb51ee2�[0m
+    snapshot_test.go:34: �[2m· create-snapshot volume=vol-2230138124fb51ee2�[0m
+�[2msnapshot�[0m=snap-1188f92980e6470aa �[2msize�[0m=4 �[2mstate�[0m=completed �[2mprogress�[0m=100%
+    snapshot_test.go:50: �[2msnapshot�[0m=snap-1188f92980e6470aa �[2msize�[0m=4 �[2mstate�[0m=completed �[2mprogress�[0m=100%
+    snapshot_test.go:59: snapshot snap-1188f92980e6470aa reached state completed
+�[2m· describe-snapshots snap-1188f92980e6470aa�[0m
+    snapshot_test.go:61: �[2m· describe-snapshots snap-1188f92980e6470aa�[0m
+�[2m· copy-snapshot src=snap-1188f92980e6470aa region=ap-southeast-2�[0m
+    snapshot_test.go:76: �[2m· copy-snapshot src=snap-1188f92980e6470aa region=ap-southeast-2�[0m
+�[2mcopy_snapshot�[0m=snap-cde6b5ddf467e2c0f
+    snapshot_test.go:86: �[2mcopy_snapshot�[0m=snap-cde6b5ddf467e2c0f
+    snapshot_test.go:88: snapshot snap-cde6b5ddf467e2c0f reached state completed
+�[2m· describe-snapshots snap-1188f92980e6470aa,snap-cde6b5ddf467e2c0f (both visible)�[0m
+    snapshot_test.go:90: �[2m· describe-snapshots snap-1188f92980e6470aa,snap-cde6b5ddf467e2c0f (both visible)�[0m
+�[2m· delete-snapshot snap-1188f92980e6470aa (original)�[0m
+    snapshot_test.go:107: �[2m· delete-snapshot snap-1188f92980e6470aa (original)�[0m
+�[2m· verify copy snap-cde6b5ddf467e2c0f still completed�[0m
+    snapshot_test.go:135: �[2m· verify copy snap-cde6b5ddf467e2c0f still completed�[0m
+�[2m· delete-snapshot snap-cde6b5ddf467e2c0f (copy)�[0m
+    snapshot_test.go:144: �[2m· delete-snapshot snap-cde6b5ddf467e2c0f (copy)�[0m
+�[2mdeleted_original�[0m=snap-1188f92980e6470aa
+    snapshot_test.go:151: �[2mdeleted_original�[0m=snap-1188f92980e6470aa]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5d_SnapshotBackedLaunch" classname="" time="0.010">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5d — Snapshot-Backed Instance Launch ━━�[0m
+�[2mami�[0m=ami-6f60f2b3bf04b5453 �[2msnapshot�[0m=snap-ami-6f60f2b3bf04b5453
+    snapshot_test.go:182: �[2mami�[0m=ami-6f60f2b3bf04b5453 �[2msnapshot�[0m=snap-ami-6f60f2b3bf04b5453]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5e_CreateImage" classname="" time="0.660">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5e — CreateImage Lifecycle ━━�[0m
+�[2m· create-image instance=i-0c615e5dd74050f53 name=e2e-custom-ami (no-reboot)�[0m
+    createimage_test.go:26: �[2m· create-image instance=i-0c615e5dd74050f53 name=e2e-custom-ami (no-reboot)�[0m
+�[2mcustom_ami�[0m=ami-7dcf263a412dd14c6
+    createimage_test.go:36: �[2mcustom_ami�[0m=ami-7dcf263a412dd14c6
+    createimage_test.go:38: image ami-7dcf263a412dd14c6 reached state available
+�[2m· describe-images ami-7dcf263a412dd14c6�[0m
+    createimage_test.go:40: �[2m· describe-images ami-7dcf263a412dd14c6�[0m
+�[2mcustom_ami_snapshot�[0m=snap-dde89b9ea2ef2e05a
+    createimage_test.go:67: �[2mcustom_ami_snapshot�[0m=snap-dde89b9ea2ef2e05a]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5f_SecurityGroupEgress" classname="" time="4.880">
+			<skipped message="Skipped"><![CDATA[
+�[1m�[36m━━ Phase 5f — Security Group Enforcement (egress ACL) ━━�[0m
+�[2m· discover default gateway inside VM�[0m
+    securitygroup_test.go:53: �[2m· discover default gateway inside VM�[0m
+�[2mprobe_gateway�[0m=172.31.0.1
+    securitygroup_test.go:66: �[2mprobe_gateway�[0m=172.31.0.1
+�[2m· 5f-1 baseline egress (allow-all)�[0m
+    securitygroup_test.go:77: �[2m· 5f-1 baseline egress (allow-all)�[0m
+    securitygroup_test.go:80: Phase 5f: baseline ICMP did not succeed; env may block ICMP regardless of SG
+        Output:
+        PING 172.31.0.1 (172.31.0.1) 56(84) bytes of data.
+        
+        --- 172.31.0.1 ping statistics ---
+        3 packets transmitted, 0 received, 100% packet loss, time 2044ms
+        ]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/Phase6_TagManagement" classname="" time="0.380">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 6 — Tag Management ━━�[0m
+�[2m· 6a create-tags instance=i-0c615e5dd74050f53 (Name, Environment, DeleteMe)�[0m
+    tags_test.go:41: �[2m· 6a create-tags instance=i-0c615e5dd74050f53 (Name, Environment, DeleteMe)�[0m
+�[2m· 6b describe-tags resource-id=i-0c615e5dd74050f53 want=3�[0m
+    tags_test.go:53: �[2m· 6b describe-tags resource-id=i-0c615e5dd74050f53 want=3�[0m
+�[2m· 6c create-tags volume=vol-2230138124fb51ee2 (Name, Environment)�[0m
+    tags_test.go:65: �[2m· 6c create-tags volume=vol-2230138124fb51ee2 (Name, Environment)�[0m
+�[2m· 6d describe-tags key=Environment want=2�[0m
+    tags_test.go:76: �[2m· 6d describe-tags key=Environment want=2�[0m
+�[2m· 6e describe-tags resource-type=instance want=3�[0m
+    tags_test.go:85: �[2m· 6e describe-tags resource-type=instance want=3�[0m
+�[2m· 6f overwrite instance Name -> e2e-test-updated�[0m
+    tags_test.go:95: �[2m· 6f overwrite instance Name -> e2e-test-updated�[0m
+�[2m· 6g delete-tags Key=DeleteMe (no value)�[0m
+    tags_test.go:114: �[2m· 6g delete-tags Key=DeleteMe (no value)�[0m
+�[2m· 6h delete-tags Key=Environment,Value=production (no-op)�[0m
+    tags_test.go:130: �[2m· 6h delete-tags Key=Environment,Value=production (no-op)�[0m
+�[2m· 6i delete-tags Key=Environment,Value=testing (instance only)�[0m
+    tags_test.go:148: �[2m· 6i delete-tags Key=Environment,Value=testing (instance only)�[0m
+�[2m· 6j final describe-tags resource-id=i-0c615e5dd74050f53 want=1�[0m
+    tags_test.go:171: �[2m· 6j final describe-tags resource-id=i-0c615e5dd74050f53 want=1�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7_StopStart" classname="" time="8.090">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7 — Instance State Transitions (Stop) ━━�[0m
+�[2m· stop-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:30: �[2m· stop-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:36: instance i-0c615e5dd74050f53 reached state stopped
+�[2m· reboot-instances on stopped instance (should fail)�[0m
+    lifecycle_test.go:40: �[2m· reboot-instances on stopped instance (should fail)�[0m
+�[2mstopped_reject�[0m=IncorrectInstanceState
+    lifecycle_test.go:47: �[2mstopped_reject�[0m=IncorrectInstanceState]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7a_AttachToStoppedError" classname="" time="0.300">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7a — Attach Volume to Stopped Instance (Error Path) ━━�[0m
+�[2m· create-volume size=10 az=ap-southeast-2a�[0m
+    lifecycle_test.go:60: �[2m· create-volume size=10 az=ap-southeast-2a�[0m
+�[2mtest_volume�[0m=vol-8d3e1335938a378cd
+    lifecycle_test.go:68: �[2mtest_volume�[0m=vol-8d3e1335938a378cd
+    lifecycle_test.go:82: volume vol-8d3e1335938a378cd reached state available
+�[2m· attach-volume vol-8d3e1335938a378cd -> i-0c615e5dd74050f53 (should fail)�[0m
+    lifecycle_test.go:84: �[2m· attach-volume vol-8d3e1335938a378cd -> i-0c615e5dd74050f53 (should fail)�[0m
+�[2m· delete-volume vol-8d3e1335938a378cd�[0m
+    lifecycle_test.go:94: �[2m· delete-volume vol-8d3e1335938a378cd�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7b_ModifyInstanceAttribute" classname="" time="55.620">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7b — ModifyInstanceAttribute ━━�[0m
+�[2mfrom_type�[0m=t3.nano �[2mto_type�[0m=t3.small
+    lifecycle_test.go:120: �[2mfrom_type�[0m=t3.nano �[2mto_type�[0m=t3.small
+�[2mexpected_vcpus�[0m=2 �[2mexpected_mem_mib�[0m=2048
+    lifecycle_test.go:142: �[2mexpected_vcpus�[0m=2 �[2mexpected_mem_mib�[0m=2048
+�[2m· modify-instance-attribute i-0c615e5dd74050f53 type=t3.small�[0m
+    lifecycle_test.go:144: �[2m· modify-instance-attribute i-0c615e5dd74050f53 type=t3.small�[0m
+�[2m· start-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:165: �[2m· start-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:171: instance i-0c615e5dd74050f53 reached state running
+�[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
+    lifecycle_test.go:177: �[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    lifecycle_test.go:179: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2m· ssh nproc�[0m
+    lifecycle_test.go:182: �[2m· ssh nproc�[0m
+�[2mvm_vcpus�[0m=2
+    lifecycle_test.go:188: �[2mvm_vcpus�[0m=2
+�[2m· ssh MemTotal�[0m
+    lifecycle_test.go:190: �[2m· ssh MemTotal�[0m
+�[2mvm_mem_mib�[0m=1967 �[2mthreshold_mib�[0m=1740
+    lifecycle_test.go:200: �[2mvm_mem_mib�[0m=1967 �[2mthreshold_mib�[0m=1740
+�[2m· stop-instances i-0c615e5dd74050f53 (end of 7b)�[0m
+    lifecycle_test.go:206: �[2m· stop-instances i-0c615e5dd74050f53 (end of 7b)�[0m
+    lifecycle_test.go:211: instance i-0c615e5dd74050f53 reached state stopped]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7c_pre_Reboot" classname="" time="83.480">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7c-pre — Reboot Running Instance ━━�[0m
+�[2m· start-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:226: �[2m· start-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:231: instance i-0c615e5dd74050f53 reached state running
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    lifecycle_test.go:236: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2mpre_reboot_private_ip�[0m=172.31.0.4
+    lifecycle_test.go:246: �[2mpre_reboot_private_ip�[0m=172.31.0.4
+�[2m· reboot-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:248: �[2m· reboot-instances i-0c615e5dd74050f53�[0m
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    lifecycle_test.go:279: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2m· ssh uptime�[0m
+    lifecycle_test.go:282: �[2m· ssh uptime�[0m
+�[2muptime_secs�[0m=25
+    lifecycle_test.go:288: �[2muptime_secs�[0m=25]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7c_RunInstancesMultiCount" classname="" time="47.560">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7c — RunInstances with MinCount/MaxCount > 1 ━━�[0m
+�[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.small count=2�[0m
+    lifecycle_test.go:309: �[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.small count=2�[0m
+�[2msibling_1�[0m=i-9bfadd8ebd378f846 �[2msibling_2�[0m=i-448a9e2bf45c58658
+    lifecycle_test.go:327: �[2msibling_1�[0m=i-9bfadd8ebd378f846 �[2msibling_2�[0m=i-448a9e2bf45c58658
+    lifecycle_test.go:342: instance i-9bfadd8ebd378f846 reached state running
+    lifecycle_test.go:342: instance i-448a9e2bf45c58658 reached state running
+�[2m· terminate-instances i-9bfadd8ebd378f846 i-448a9e2bf45c58658�[0m
+    lifecycle_test.go:345: �[2m· terminate-instances i-9bfadd8ebd378f846 i-448a9e2bf45c58658�[0m
+    lifecycle_test.go:352: instance i-9bfadd8ebd378f846 reached state terminated
+    lifecycle_test.go:352: instance i-448a9e2bf45c58658 reached state terminated]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths" classname="" time="0.000">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8 — Negative / Error Path Tests ━━�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8a_InvalidAMIIDMalformed" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· run-instances image-id=notanami�[0m
+    negative_test.go:55: �[2m· run-instances image-id=notanami�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8b_InvalidInstanceType" classname="" time="0.210">
+			<system-out><![CDATA[�[2m· run-instances instance-type=x99.superlarge�[0m
+    negative_test.go:70: �[2m· run-instances instance-type=x99.superlarge�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8c_VolumeInUse" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· attach-volume vol-2230138124fb51ee2 (root, in-use)�[0m
+    negative_test.go:85: �[2m· attach-volume vol-2230138124fb51ee2 (root, in-use)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8d_DetachRootForbidden" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· detach-volume vol-2230138124fb51ee2 (root)�[0m
+    negative_test.go:97: �[2m· detach-volume vol-2230138124fb51ee2 (root)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8e_DeleteSnapshotNotFound" classname="" time="0.030">
+			<system-out><![CDATA[�[2m· delete-snapshot snap-nonexistent000000�[0m
+    negative_test.go:108: �[2m· delete-snapshot snap-nonexistent000000�[0m
+�[2m· describe-volumes vol-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8f_InvalidActionRawHTTP" classname="" time="0.040">
+			<system-out><![CDATA[�[2m· POST Action=DescribeFakeThings�[0m
+    negative_test.go:122: �[2m· POST Action=DescribeFakeThings�[0m
+    negative_test.go:125: �[2mstatus�[0m=400 �[2mcode�[0m=InvalidAction]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8g_InvalidAMIIDNotFound" classname="" time="0.240">
+			<system-out><![CDATA[�[2m· reboot-instances i-nonexistent�[0m
+�[2m· run-instances image-id=ami-0000000000000dead�[0m
+    negative_test.go:138: �[2m· run-instances image-id=ami-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8h_InvalidKeyPairNotFound" classname="" time="0.260">
+			<system-out><![CDATA[�[2m· run-instances key-name=nonexistent-key-xyz�[0m
+    negative_test.go:153: �[2m· run-instances key-name=nonexistent-key-xyz�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8i_DeleteVolumeNotFound" classname="" time="0.040">
+			<system-out><![CDATA[�[2m· delete-volume vol-0000000000000dead�[0m
+    negative_test.go:168: �[2m· delete-volume vol-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8j_CreateKeyPairDuplicate" classname="" time="0.040">
+			<system-out><![CDATA[    negative_test.go:178: �[2m· create-key-pair test-key-1 (duplicate)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8k_ImportKeyPairDuplicate" classname="" time="0.040">
+			<system-out><![CDATA[�[2m· import-key-pair test-key-1 (duplicate)�[0m
+    negative_test.go:198: �[2m· import-key-pair test-key-1 (duplicate)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8l_InvalidKeyFormat" classname="" time="0.040">
+			<system-out><![CDATA[    negative_test.go:210: �[2m· import-key-pair bad-format-key (bad material)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8m_DescribeVolumesNotFound" classname="" time="0.040">
+			<system-out><![CDATA[    negative_test.go:230: �[2m· describe-volumes vol-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8n_DescribeImagesNotFound" classname="" time="0.040">
+			<system-out><![CDATA[�[2m· describe-images ami-0000000000000dead�[0m
+    negative_test.go:240: �[2m· describe-images ami-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8o_CreateImageDuplicateName" classname="" time="0.560">
+			<system-out><![CDATA[�[2m· create-key-pair test-key-1 (duplicate)�[0m
+�[2m· create-image name=e2e-custom-ami (duplicate)�[0m
+�[2m· import-key-pair bad-format-key (bad material)�[0m
+    negative_test.go:252: �[2m· create-image name=e2e-custom-ami (duplicate)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8p_DeleteKeyPairIdempotent" classname="" time="0.020">
+			<system-out><![CDATA[�[2m· delete-key-pair nonexistent-key-99999 (idempotent)�[0m
+    negative_test.go:276: �[2m· delete-key-pair nonexistent-key-99999 (idempotent)�[0m
+�[2mstatus�[0m=400 �[2mcode�[0m=InvalidAction]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8q_ModifyAttributeOnRunning" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· modify-instance-attribute i-0c615e5dd74050f53 (running)�[0m
+    negative_test.go:289: �[2m· modify-instance-attribute i-0c615e5dd74050f53 (running)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8r_RebootInstanceNotFound" classname="" time="0.040">
+			<system-out><![CDATA[    negative_test.go:302: �[2m· reboot-instances i-nonexistent�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM1_UserCRUD" classname="" time="0.070">
+			<system-out><![CDATA[
+�[1m�[36m━━ IAM Phase 1 — User CRUD ━━�[0m
+�[2m· list-users (root auth sanity)�[0m
+    iam_test.go:109: �[2m· list-users (root auth sanity)�[0m
+�[2m· create-user "alice"�[0m
+    iam_test.go:114: �[2m· create-user "alice"�[0m
+�[2muser�[0m=alice �[2marn�[0m=arn:aws:iam::000000000001:user/alice
+    iam_test.go:120: �[2muser�[0m=alice �[2marn�[0m=arn:aws:iam::000000000001:user/alice
+�[2m· create-user "bob" path="/engineering/"�[0m
+    iam_test.go:122: �[2m· create-user "bob" path="/engineering/"�[0m
+�[2m· create-user "alice" again (expect EntityAlreadyExists)�[0m
+    iam_test.go:131: �[2m· create-user "alice" again (expect EntityAlreadyExists)�[0m
+�[2m· get-user "alice"�[0m
+    iam_test.go:137: �[2m· get-user "alice"�[0m
+�[2m· get-user nonexistent (expect NoSuchEntity)�[0m
+    iam_test.go:142: �[2m· get-user nonexistent (expect NoSuchEntity)�[0m
+�[2m· list-users (>=3 root+alice+bob)�[0m
+    iam_test.go:148: �[2m· list-users (>=3 root+alice+bob)�[0m
+�[2m· list-users --path-prefix "/engineering/" (expect 1)�[0m
+    iam_test.go:154: �[2m· list-users --path-prefix "/engineering/" (expect 1)�[0m
+    iam_test.go:193: �[2m· create-user "iam-phase1-ephemeral" (delete idempotency probe)�[0m
+�[2m· delete-user "iam-phase1-ephemeral"�[0m
+    iam_test.go:197: �[2m· delete-user "iam-phase1-ephemeral"�[0m
+�[2m· delete-user "iam-phase1-ephemeral" again (expect NoSuchEntity)�[0m
+    iam_test.go:201: �[2m· delete-user "iam-phase1-ephemeral" again (expect NoSuchEntity)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM1_UserCRUD/UpdateUser" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[    iam_test.go:165: daemon UpdateUser handler not implemented — InvalidAction
+�[2m· create-user "iam-phase1-ephemeral" (delete idempotency probe)�[0m]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM2_AccessKeyLifecycle" classname="" time="0.040">
+			<system-out><![CDATA[
+�[1m�[36m━━ IAM Phase 2 — Access Key Lifecycle ━━�[0m
+�[2m· create-access-key user=alice (key 1)�[0m
+    iam_test.go:218: �[2m· create-access-key user=alice (key 1)�[0m
+�[2mkey1�[0m=AKIA4984AE57CF8D721A879A
+    iam_test.go:227: �[2mkey1�[0m=AKIA4984AE57CF8D721A879A
+�[2m· create-access-key user=alice (key 2)�[0m
+    iam_test.go:229: �[2m· create-access-key user=alice (key 2)�[0m
+�[2mkey2�[0m=AKIAA1CD38D4F0E573216E11
+    iam_test.go:236: �[2mkey2�[0m=AKIAA1CD38D4F0E573216E11
+�[2m· create-access-key user=alice key 3 (expect LimitExceeded)�[0m
+    iam_test.go:238: �[2m· create-access-key user=alice key 3 (expect LimitExceeded)�[0m
+�[2m· create-access-key user=ghost (expect NoSuchEntity)�[0m
+    iam_test.go:246: �[2m· create-access-key user=ghost (expect NoSuchEntity)�[0m
+�[2m· list-access-keys user=alice (expect 2)�[0m
+    iam_test.go:254: �[2m· list-access-keys user=alice (expect 2)�[0m
+�[2m· list-access-keys user=bob (expect 0)�[0m
+    iam_test.go:261: �[2m· list-access-keys user=bob (expect 0)�[0m
+�[2m· update-access-key AKIA4984AE57CF8D721A879A -> Inactive�[0m
+    iam_test.go:268: �[2m· update-access-key AKIA4984AE57CF8D721A879A -> Inactive�[0m
+�[2m· update-access-key AKIA4984AE57CF8D721A879A -> Active�[0m
+    iam_test.go:279: �[2m· update-access-key AKIA4984AE57CF8D721A879A -> Active�[0m
+�[2m· delete-access-key AKIAA1CD38D4F0E573216E11 (key 2)�[0m
+    iam_test.go:287: �[2m· delete-access-key AKIAA1CD38D4F0E573216E11 (key 2)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM3_UserAuthentication" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[
+�[1m�[36m━━ IAM Phase 3 — User Authentication ━━�[0m
+    iam_test.go:311: daemon IAM signature/principal lookup not implemented — mulga-siv-100]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD" classname="" time="0.040">
+			<system-out><![CDATA[
+�[1m�[36m━━ IAM Phase 4 — Policy CRUD ━━�[0m
+�[2m· create-policy EC2ReadOnly�[0m
+    iam_test.go:384: �[2m· create-policy EC2ReadOnly�[0m
+�[2mpolicy�[0m=EC2ReadOnly �[2marn�[0m=arn:aws:iam::000000000001:policy/EC2ReadOnly �[2maccount�[0m=000000000001
+    iam_test.go:393: �[2mpolicy�[0m=EC2ReadOnly �[2marn�[0m=arn:aws:iam::000000000001:policy/EC2ReadOnly �[2maccount�[0m=000000000001
+    iam_test.go:440: �[2m· create-policy duplicate (expect EntityAlreadyExists)�[0m
+�[2m· create-policy malformed (expect MalformedPolicyDocument)�[0m
+    iam_test.go:449: �[2m· create-policy malformed (expect MalformedPolicyDocument)�[0m
+�[2m· get-policy EC2ReadOnly�[0m
+    iam_test.go:458: �[2m· get-policy EC2ReadOnly�[0m
+�[2m· get-policy nonexistent (expect NoSuchEntity)�[0m
+    iam_test.go:463: �[2m· get-policy nonexistent (expect NoSuchEntity)�[0m
+�[2m· get-policy-version EC2ReadOnly v1�[0m
+    iam_test.go:470: �[2m· get-policy-version EC2ReadOnly v1�[0m
+    iam_test.go:519: �[2m· list-policies (>=5)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel" classname="" time="0.000"></testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/FullAdmin" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· create-policy FullAdmin path=/admin/�[0m
+    iam_test.go:402: �[2m· create-policy FullAdmin path=/admin/�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/DenyTerminate" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· create-policy DenyTerminate (mixed Allow+Deny)�[0m
+    iam_test.go:413: �[2m· create-policy DenyTerminate (mixed Allow+Deny)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/IAMReadOnly" classname="" time="0.020">
+			<system-out><![CDATA[�[2m· create-policy IAMReadOnly�[0m
+    iam_test.go:422: �[2m· create-policy IAMReadOnly�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/EC2DescribeAll" classname="" time="0.020">
+			<system-out><![CDATA[�[2m· create-policy EC2DescribeAll (wildcard)�[0m
+    iam_test.go:431: �[2m· create-policy EC2DescribeAll (wildcard)�[0m
+�[2m· create-policy duplicate (expect EntityAlreadyExists)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/PolicyVersionLifecycle" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[    iam_test.go:484: daemon CreatePolicyVersion handler not implemented — InvalidAction
+�[2m· list-policies (>=5)�[0m]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM5_PolicyAttachmentEnforcement" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[
+�[1m�[36m━━ IAM Phase 5 — Policy Attachment & Enforcement ━━�[0m
+    iam_test.go:535: daemon IAM scoped-credential enforcement not implemented — mulga-siv-100]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM6_PolicyLifecycle" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[
+�[1m�[36m━━ IAM Phase 6 — Policy Lifecycle (Detach & Delete) ━━�[0m
+    iam_test.go:718: daemon DetachUserPolicy not implemented — mulga-siv-100]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM7_Cleanup" classname="" time="0.050">
+			<system-out><![CDATA[
+�[1m�[36m━━ IAM Phase 7 — Cleanup ━━�[0m
+�[2m· cleanup user alice�[0m
+    iam_test.go:770: �[2m· cleanup user alice�[0m
+�[2m· cleanup user bob�[0m
+    iam_test.go:770: �[2m· cleanup user bob�[0m
+�[2m· cleanup user charlie�[0m
+    iam_test.go:770: �[2m· cleanup user charlie�[0m
+�[2mremaining_users�[0m=1 �[2mnames�[0m=[admin]
+    iam_test.go:779: �[2mremaining_users�[0m=1 �[2mnames�[0m=[admin]
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2ReadOnly�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2ReadOnly�[0m
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/admin/FullAdmin�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/admin/FullAdmin�[0m
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/IAMReadOnly�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/IAMReadOnly�[0m
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2DescribeAll�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2DescribeAll�[0m
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/DenyTerminate�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/DenyTerminate�[0m
+�[2mremaining_policies�[0m=1 �[2mnames�[0m=[AdministratorAccess]
+    iam_test.go:812: �[2mremaining_policies�[0m=1 �[2mnames�[0m=[AdministratorAccess]]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping" classname="" time="114.890">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8 (acct) — EC2 Account Scoping ━━�[0m
+    account_test.go:75: �[2m· Phase 8 acct cleanup (deferred)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step1_AccountSetup" classname="" time="1.220">
+			<system-out><![CDATA[�[2m· create account 'Team Alpha'�[0m
+    account_test.go:206: �[2m· create account 'Team Alpha'�[0m
+�[2malpha_account�[0m=000000000002 �[2malpha_key�[0m=AKIA374CBBB2A15D3E2EE822
+    account_test.go:211: �[2malpha_account�[0m=000000000002 �[2malpha_key�[0m=AKIA374CBBB2A15D3E2EE822
+�[2m· create account 'Team Beta'�[0m
+    account_test.go:213: �[2m· create account 'Team Beta'�[0m
+�[2mbeta_account�[0m=000000000003 �[2mbeta_key�[0m=AKIA0AA0B2845E3BD5A26D01
+    account_test.go:218: �[2mbeta_account�[0m=000000000003 �[2mbeta_key�[0m=AKIA0AA0B2845E3BD5A26D01]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step2_InstanceScoping" classname="" time="52.380">
+			<system-out><![CDATA[�[2m· create per-account key pairs�[0m
+    account_test.go:242: �[2m· create per-account key pairs�[0m
+�[2m· alpha run-instances�[0m
+    account_test.go:250: �[2m· alpha run-instances�[0m
+�[2malpha_instance�[0m=i-a3072c2387630b8ee
+    account_test.go:262: �[2malpha_instance�[0m=i-a3072c2387630b8ee
+�[2m· beta run-instances�[0m
+    account_test.go:264: �[2m· beta run-instances�[0m
+�[2mbeta_instance�[0m=i-ad32cbb0afaee5eee
+    account_test.go:276: �[2mbeta_instance�[0m=i-ad32cbb0afaee5eee
+    account_test.go:278: instance i-a3072c2387630b8ee reached state running
+    account_test.go:279: instance i-ad32cbb0afaee5eee reached state running
+�[2m· alpha sees only own instances�[0m
+    account_test.go:281: �[2m· alpha sees only own instances�[0m
+�[2m· beta sees only own instances�[0m
+    account_test.go:285: �[2m· beta sees only own instances�[0m
+�[2m· alpha OwnerId matches alpha account�[0m
+    account_test.go:289: �[2m· alpha OwnerId matches alpha account�[0m
+�[2m· cross-account stop alpha->beta blocked�[0m
+    account_test.go:296: �[2m· cross-account stop alpha->beta blocked�[0m
+�[2m· cross-account terminate beta->alpha blocked�[0m
+    account_test.go:304: �[2m· cross-account terminate beta->alpha blocked�[0m
+�[2m· cross-account reboot alpha->beta blocked�[0m
+    account_test.go:312: �[2m· cross-account reboot alpha->beta blocked�[0m
+�[2m· stop alpha instance for cross-account start/modify/console tests�[0m
+    account_test.go:321: �[2m· stop alpha instance for cross-account start/modify/console tests�[0m
+    account_test.go:326: instance i-a3072c2387630b8ee reached state stopped
+�[2m· cross-account start beta->alpha blocked�[0m
+    account_test.go:328: �[2m· cross-account start beta->alpha blocked�[0m
+�[2m· cross-account modify-instance-attribute beta->alpha blocked�[0m
+    account_test.go:336: �[2m· cross-account modify-instance-attribute beta->alpha blocked�[0m
+�[2m· cross-account get-console-output beta->alpha blocked�[0m
+    account_test.go:345: �[2m· cross-account get-console-output beta->alpha blocked�[0m
+�[2m· restart alpha instance�[0m
+    account_test.go:354: �[2m· restart alpha instance�[0m
+    account_test.go:359: instance i-a3072c2387630b8ee reached state running]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step3_VolumeScoping" classname="" time="24.140">
+			<system-out><![CDATA[�[2m· alpha create-volume�[0m
+    account_test.go:366: �[2m· alpha create-volume�[0m
+�[2malpha_vol�[0m=vol-ceef82652369780af
+    account_test.go:375: �[2malpha_vol�[0m=vol-ceef82652369780af
+�[2m· beta create-volume�[0m
+    account_test.go:377: �[2m· beta create-volume�[0m
+�[2mbeta_vol�[0m=vol-08024f32d23d9e18c
+    account_test.go:386: �[2mbeta_vol�[0m=vol-08024f32d23d9e18c
+    account_test.go:388: volume vol-ceef82652369780af reached state available
+    account_test.go:389: volume vol-08024f32d23d9e18c reached state available
+�[2m· alpha sees only own volumes�[0m
+    account_test.go:391: �[2m· alpha sees only own volumes�[0m
+�[2m· cross-account describe-volume by id blocked�[0m
+    account_test.go:395: �[2m· cross-account describe-volume by id blocked�[0m
+�[2m· cross-account delete-volume blocked�[0m
+    account_test.go:403: �[2m· cross-account delete-volume blocked�[0m
+�[2m· cross-account attach-volume (beta attaches alpha's vol) blocked�[0m
+    account_test.go:411: �[2m· cross-account attach-volume (beta attaches alpha's vol) blocked�[0m
+�[2m· alpha attach-volume to own instance�[0m
+    account_test.go:422: �[2m· alpha attach-volume to own instance�[0m
+    account_test.go:429: volume vol-ceef82652369780af reached state in-use
+�[2m· cross-account detach-volume blocked�[0m
+    account_test.go:431: �[2m· cross-account detach-volume blocked�[0m
+�[2m· cross-account modify-volume blocked�[0m
+    account_test.go:439: �[2m· cross-account modify-volume blocked�[0m
+�[2m· alpha detach own volume�[0m
+    account_test.go:449: �[2m· alpha detach own volume�[0m
+    account_test.go:454: volume vol-ceef82652369780af reached state available]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step4_KeyPairScoping" classname="" time="0.370">
+			<system-out><![CDATA[�[2m· alpha create-key-pair alpha-key�[0m
+    account_test.go:466: �[2m· alpha create-key-pair alpha-key�[0m
+�[2m· beta create-key-pair beta-key�[0m
+    account_test.go:473: �[2m· beta create-key-pair beta-key�[0m
+�[2m· alpha sees only own keys�[0m
+    account_test.go:478: �[2m· alpha sees only own keys�[0m
+�[2m· namespace isolation: shared-name in both accounts�[0m
+    account_test.go:483: �[2m· namespace isolation: shared-name in both accounts�[0m
+�[2m· beta deletes alpha-key (idempotent, no cross-account effect)�[0m
+    account_test.go:497: �[2m· beta deletes alpha-key (idempotent, no cross-account effect)�[0m
+�[2m· alpha import-key-pair imported-key�[0m
+    account_test.go:503: �[2m· alpha import-key-pair imported-key�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step5_SnapshotScoping" classname="" time="0.060">
+			<system-out><![CDATA[�[2m· alpha create-snapshot�[0m
+    account_test.go:522: �[2m· alpha create-snapshot�[0m
+�[2malpha_snap�[0m=snap-2a47896f4304d805c
+    account_test.go:530: �[2malpha_snap�[0m=snap-2a47896f4304d805c
+�[2m· beta create-snapshot�[0m
+    account_test.go:532: �[2m· beta create-snapshot�[0m
+�[2mbeta_snap�[0m=snap-bf5744e78bd7c6908
+    account_test.go:540: �[2mbeta_snap�[0m=snap-bf5744e78bd7c6908
+�[2m· alpha sees only own snapshots (owner-ids=self)�[0m
+    account_test.go:542: �[2m· alpha sees only own snapshots (owner-ids=self)�[0m
+�[2m· cross-account delete-snapshot blocked (UnauthorizedOperation)�[0m
+    account_test.go:560: �[2m· cross-account delete-snapshot blocked (UnauthorizedOperation)�[0m
+�[2m· cross-account create-snapshot of alpha's volume blocked�[0m
+    account_test.go:568: �[2m· cross-account create-snapshot of alpha's volume blocked�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step6_VPCSubnetScoping" classname="" time="0.030">
+			<system-out><![CDATA[�[2m· alpha create-vpc 10.0.0.0/16�[0m
+    account_test.go:582: �[2m· alpha create-vpc 10.0.0.0/16�[0m
+�[2malpha_vpc�[0m=vpc-7f60df5e87183fb59
+    account_test.go:589: �[2malpha_vpc�[0m=vpc-7f60df5e87183fb59
+�[2m· beta create-vpc 10.0.0.0/16 (same CIDR, no conflict)�[0m
+    account_test.go:591: �[2m· beta create-vpc 10.0.0.0/16 (same CIDR, no conflict)�[0m
+�[2mbeta_vpc�[0m=vpc-7e8c859d3985c8218
+    account_test.go:598: �[2mbeta_vpc�[0m=vpc-7e8c859d3985c8218
+�[2m· alpha describe-vpcs isolation�[0m
+    account_test.go:600: �[2m· alpha describe-vpcs isolation�[0m
+�[2m· cross-account describe-vpc by id blocked�[0m
+    account_test.go:605: �[2m· cross-account describe-vpc by id blocked�[0m
+�[2m· cross-account delete-vpc blocked�[0m
+    account_test.go:613: �[2m· cross-account delete-vpc blocked�[0m
+�[2m· alpha create-subnet 10.0.1.0/24�[0m
+    account_test.go:621: �[2m· alpha create-subnet 10.0.1.0/24�[0m
+�[2m· beta create-subnet 10.0.1.0/24�[0m
+    account_test.go:630: �[2m· beta create-subnet 10.0.1.0/24�[0m
+�[2m· alpha describe-subnets isolation�[0m
+    account_test.go:639: �[2m· alpha describe-subnets isolation�[0m
+�[2m· cross-account create-subnet in other VPC blocked�[0m
+    account_test.go:648: �[2m· cross-account create-subnet in other VPC blocked�[0m
+�[2m· cross-account delete-subnet blocked�[0m
+    account_test.go:657: �[2m· cross-account delete-subnet blocked�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step7_IGWEIGWScoping" classname="" time="0.030">
+			<system-out><![CDATA[�[2m· alpha create-internet-gateway�[0m
+    account_test.go:673: �[2m· alpha create-internet-gateway�[0m
+�[2malpha_igw�[0m=igw-ee966e3e923caf630
+    account_test.go:678: �[2malpha_igw�[0m=igw-ee966e3e923caf630
+�[2m· beta create-internet-gateway�[0m
+    account_test.go:680: �[2m· beta create-internet-gateway�[0m
+�[2mbeta_igw�[0m=igw-b46e66f2065889bb9
+    account_test.go:685: �[2mbeta_igw�[0m=igw-b46e66f2065889bb9
+�[2m· alpha describe-internet-gateways isolation�[0m
+    account_test.go:687: �[2m· alpha describe-internet-gateways isolation�[0m
+�[2m· cross-account describe IGW by id blocked�[0m
+    account_test.go:696: �[2m· cross-account describe IGW by id blocked�[0m
+�[2m· cross-account delete IGW blocked�[0m
+    account_test.go:704: �[2m· cross-account delete IGW blocked�[0m
+�[2m· cross-account attach IGW (alpha attaches beta's IGW) blocked�[0m
+    account_test.go:712: �[2m· cross-account attach IGW (alpha attaches beta's IGW) blocked�[0m
+�[2m· alpha attach own IGW to own VPC�[0m
+    account_test.go:722: �[2m· alpha attach own IGW to own VPC�[0m
+�[2m· cross-account detach IGW blocked�[0m
+    account_test.go:729: �[2m· cross-account detach IGW blocked�[0m
+�[2m· alpha create-egress-only-internet-gateway�[0m
+    account_test.go:739: �[2m· alpha create-egress-only-internet-gateway�[0m
+�[2malpha_eigw�[0m=eigw-51ba82da927e49393
+    account_test.go:746: �[2malpha_eigw�[0m=eigw-51ba82da927e49393
+�[2m· beta create-egress-only-internet-gateway�[0m
+    account_test.go:748: �[2m· beta create-egress-only-internet-gateway�[0m
+�[2mbeta_eigw�[0m=eigw-e88d642e78ab4fe5a
+    account_test.go:755: �[2mbeta_eigw�[0m=eigw-e88d642e78ab4fe5a
+�[2m· alpha describe-eigws isolation�[0m
+    account_test.go:757: �[2m· alpha describe-eigws isolation�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step8_AccountSettings" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· alpha enable-ebs-encryption-by-default�[0m
+    account_test.go:794: �[2m· alpha enable-ebs-encryption-by-default�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step9_GlobalResources" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· describe-regions identical across accounts�[0m
+    account_test.go:829: �[2m· describe-regions identical across accounts�[0m
+�[2m· describe-availability-zones identical across accounts�[0m
+    account_test.go:834: �[2m· describe-availability-zones identical across accounts�[0m
+�[2m· describe-instance-types identical across accounts�[0m
+    account_test.go:839: �[2m· describe-instance-types identical across accounts�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step10_EdgeCases" classname="" time="1.200">
+			<system-out><![CDATA[�[2m· create empty Gamma account�[0m
+    account_test.go:849: �[2m· create empty Gamma account�[0m
+�[2mgamma_account�[0m=000000000004
+    account_test.go:852: �[2mgamma_account�[0m=000000000004
+�[2m· root create-key-pair root-scoping-key�[0m
+    account_test.go:879: �[2m· root create-key-pair root-scoping-key�[0m
+�[2m· non-existent volume id -> InvalidVolume.NotFound�[0m
+    account_test.go:895: �[2m· non-existent volume id -> InvalidVolume.NotFound�[0m
+�[2m· non-existent snapshot id -> InvalidSnapshot.NotFound�[0m
+    account_test.go:903: �[2m· non-existent snapshot id -> InvalidSnapshot.NotFound�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step11_Cleanup" classname="" time="32.440">
+			<system-out><![CDATA[�[2m· terminate alpha instance�[0m
+    account_test.go:920: �[2m· terminate alpha instance�[0m
+�[2m· terminate beta instance�[0m
+    account_test.go:926: �[2m· terminate beta instance�[0m
+�[2m· delete snapshots�[0m
+    account_test.go:939: �[2m· delete snapshots�[0m
+�[2m· delete volumes�[0m
+    account_test.go:955: �[2m· delete volumes�[0m
+�[2m· delete key pairs�[0m
+    account_test.go:969: �[2m· delete key pairs�[0m
+�[2m· delete EIGWs�[0m
+    account_test.go:979: �[2m· delete EIGWs�[0m
+�[2m· detach + delete IGWs�[0m
+    account_test.go:993: �[2m· detach + delete IGWs�[0m
+�[2m· delete subnets�[0m
+    account_test.go:1013: �[2m· delete subnets�[0m
+�[2m· delete VPCs�[0m
+    account_test.go:1027: �[2m· delete VPCs�[0m
+�[2m· Phase 8 acct cleanup (deferred)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8b_VPCSubnetE2E" classname="" time="200.480">
+			<failure message="Failed"><![CDATA[
+�[1m�[36m━━ Phase 8b — VPC Public/Private Subnet E2E ━━�[0m
+�[2m· create-vpc 10.99.0.0/16�[0m
+    vpc_test.go:56: �[2m· create-vpc 10.99.0.0/16�[0m
+�[2mvpc�[0m=vpc-39eff985799c43069 �[2mcidr�[0m=10.99.0.0/16
+    vpc_test.go:67: �[2mvpc�[0m=vpc-39eff985799c43069 �[2mcidr�[0m=10.99.0.0/16
+�[2m· create-subnet 10.99.1.0/24 (public)�[0m
+    vpc_test.go:70: �[2m· create-subnet 10.99.1.0/24 (public)�[0m
+�[2m· modify-subnet-attribute MapPublicIpOnLaunch=true subnet-22aba054e10588254�[0m
+    vpc_test.go:87: �[2m· modify-subnet-attribute MapPublicIpOnLaunch=true subnet-22aba054e10588254�[0m
+�[2mpub_subnet�[0m=subnet-22aba054e10588254 �[2mmap_public_ip�[0m=true
+    vpc_test.go:93: �[2mpub_subnet�[0m=subnet-22aba054e10588254 �[2mmap_public_ip�[0m=true
+�[2m· create-subnet 10.99.2.0/24 (private)�[0m
+    vpc_test.go:96: �[2m· create-subnet 10.99.2.0/24 (private)�[0m
+�[2mpriv_subnet�[0m=subnet-f53aeddb73c134696 �[2mmap_public_ip�[0m=false
+    vpc_test.go:111: �[2mpriv_subnet�[0m=subnet-f53aeddb73c134696 �[2mmap_public_ip�[0m=false
+�[2m· create-internet-gateway�[0m
+    vpc_test.go:114: �[2m· create-internet-gateway�[0m
+�[2m· attach-internet-gateway igw-1520b5a34480da982 -> vpc-39eff985799c43069�[0m
+    vpc_test.go:133: �[2m· attach-internet-gateway igw-1520b5a34480da982 -> vpc-39eff985799c43069�[0m
+�[2migw�[0m=igw-1520b5a34480da982
+    vpc_test.go:139: �[2migw�[0m=igw-1520b5a34480da982
+�[2m· create-route-table (vpc=vpc-39eff985799c43069)�[0m
+    vpc_test.go:142: �[2m· create-route-table (vpc=vpc-39eff985799c43069)�[0m
+�[2m· associate-route-table rtb-662df5e424cbdb60c <- subnet-22aba054e10588254�[0m
+    vpc_test.go:159: �[2m· associate-route-table rtb-662df5e424cbdb60c <- subnet-22aba054e10588254�[0m
+�[2m· create-route 0.0.0.0/0 -> igw-1520b5a34480da982�[0m
+    vpc_test.go:176: �[2m· create-route 0.0.0.0/0 -> igw-1520b5a34480da982�[0m
+�[2mrtb�[0m=rtb-662df5e424cbdb60c �[2massoc�[0m=rtbassoc-5510bf1dc6924eb0b �[2mroute�[0m=0.0.0.0/0->igw-1520b5a34480da982
+    vpc_test.go:192: �[2mrtb�[0m=rtb-662df5e424cbdb60c �[2massoc�[0m=rtbassoc-5510bf1dc6924eb0b �[2mroute�[0m=0.0.0.0/0->igw-1520b5a34480da982
+�[2m· run-instances ami=ami-6f60f2b3bf04b5453 subnet=subnet-22aba054e10588254�[0m
+    vpc_test.go:195: �[2m· run-instances ami=ami-6f60f2b3bf04b5453 subnet=subnet-22aba054e10588254�[0m
+�[2minstance�[0m=i-3177888cba3796d42
+    vpc_test.go:216: �[2minstance�[0m=i-3177888cba3796d42
+    vpc_test.go:218: instance i-3177888cba3796d42 reached state running
+�[2mpublic_ip�[0m=192.168.0.202 �[2mprivate_ip�[0m=10.99.1.4
+    vpc_test.go:224: �[2mpublic_ip�[0m=192.168.0.202 �[2mprivate_ip�[0m=10.99.1.4
+�[2m· waiting for SSH handshake 192.168.0.202:22�[0m
+    vpc_test.go:227: �[2m· waiting for SSH handshake 192.168.0.202:22�[0m
+    vpc_test.go:227: Eventually: condition not met within 3m0s: [SSH handshake 192.168.0.202:22 never completed]]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8c_RouteTableValidation" classname="" time="0.030">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8c — Route Table Validation ━━�[0m
+�[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63
+    vpc_test.go:292: �[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8c_RouteTableValidation/Step1_DefaultMainRouteTable" classname="" time="0.000">
+			<system-out><![CDATA[�[2m· describe-route-tables filter=vpc-id,association.main�[0m
+    vpc_test.go:297: �[2m· describe-route-tables filter=vpc-id,association.main�[0m
+�[2mmain_rtb�[0m=rtb-e42012569f92f7184
+    vpc_test.go:309: �[2mmain_rtb�[0m=rtb-e42012569f92f7184
+�[2mlocal_route�[0m=172.31.0.0/16
+    vpc_test.go:332: �[2mlocal_route�[0m=172.31.0.0/16
+�[2mdefault_route�[0m=0.0.0.0/0->igw-79c2e643660b4b7e3
+    vpc_test.go:350: �[2mdefault_route�[0m=0.0.0.0/0->igw-79c2e643660b4b7e3]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8c_RouteTableValidation/Step2_CustomRouteTableLifecycle" classname="" time="0.020">
+			<system-out><![CDATA[�[2m· create-route-table (vpc=vpc-531252db3b4e1fc08)�[0m
+    vpc_test.go:363: �[2m· create-route-table (vpc=vpc-531252db3b4e1fc08)�[0m
+�[2mcustom_rtb�[0m=rtb-5e6c46936f3e44167
+    vpc_test.go:379: �[2mcustom_rtb�[0m=rtb-5e6c46936f3e44167
+�[2mcustom_local_route�[0m=172.31.0.0/16
+    vpc_test.go:396: �[2mcustom_local_route�[0m=172.31.0.0/16
+�[2mvpc_igw�[0m=igw-79c2e643660b4b7e3
+    vpc_test.go:410: �[2mvpc_igw�[0m=igw-79c2e643660b4b7e3
+�[2m· create-route 0.0.0.0/0 -> igw-79c2e643660b4b7e3�[0m
+    vpc_test.go:415: �[2m· create-route 0.0.0.0/0 -> igw-79c2e643660b4b7e3�[0m
+�[2m· associate-route-table rtb-5e6c46936f3e44167 <- subnet-cd62e4c1c3cb50c63�[0m
+    vpc_test.go:434: �[2m· associate-route-table rtb-5e6c46936f3e44167 <- subnet-cd62e4c1c3cb50c63�[0m
+�[2mcustom_assoc�[0m=rtbassoc-da6377d37847d3d4a
+    vpc_test.go:450: �[2mcustom_assoc�[0m=rtbassoc-da6377d37847d3d4a
+�[2m· disassociate-route-table rtbassoc-da6377d37847d3d4a�[0m
+    vpc_test.go:452: �[2m· disassociate-route-table rtbassoc-da6377d37847d3d4a�[0m
+�[2m· delete-route 0.0.0.0/0�[0m
+    vpc_test.go:460: �[2m· delete-route 0.0.0.0/0�[0m
+�[2m· delete-route-table rtb-5e6c46936f3e44167�[0m
+    vpc_test.go:468: �[2m· delete-route-table rtb-5e6c46936f3e44167�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8c_RouteTableValidation/Step3_ErrorPaths" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· create-route-table (scratch for error paths)�[0m
+    vpc_test.go:481: �[2m· create-route-table (scratch for error paths)�[0m
+�[2m· delete-route local=172.31.0.0/16 (must fail)�[0m
+    vpc_test.go:517: �[2m· delete-route local=172.31.0.0/16 (must fail)�[0m
+�[2mdelete_local_err�[0m=InvalidParameterValue
+    vpc_test.go:529: �[2mdelete_local_err�[0m=InvalidParameterValue
+�[2m· delete-route-table rtb-e42012569f92f7184 (main, must fail)�[0m
+    vpc_test.go:533: �[2m· delete-route-table rtb-e42012569f92f7184 (main, must fail)�[0m
+�[2mdelete_main_err�[0m=DependencyViolation
+    vpc_test.go:540: �[2mdelete_main_err�[0m=DependencyViolation]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8d_NATGateway" classname="" time="106.010">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8d — NAT Gateway E2E ━━�[0m
+�[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2mpub_subnet�[0m=subnet-cd62e4c1c3cb50c63 �[2msg�[0m=sg-764a7fa2a39577204
+    natgw_test.go:54: �[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2mpub_subnet�[0m=subnet-cd62e4c1c3cb50c63 �[2msg�[0m=sg-764a7fa2a39577204
+�[2m· create-subnet 172.31.16.0/20 (private)�[0m
+    natgw_test.go:58: �[2m· create-subnet 172.31.16.0/20 (private)�[0m
+�[2mpriv_subnet�[0m=subnet-2360a40e28371e214
+    natgw_test.go:71: �[2mpriv_subnet�[0m=subnet-2360a40e28371e214
+�[2m· run-instances bastion (default subnet)�[0m
+    natgw_test.go:74: �[2m· run-instances bastion (default subnet)�[0m
+�[2m· run-instances private (private subnet)�[0m
+    natgw_test.go:98: �[2m· run-instances private (private subnet)�[0m
+    natgw_test.go:118: instance i-268402f0e28b5e477 reached state running
+    natgw_test.go:119: instance i-db7f3ed6e4cdaa5aa reached state running
+�[2mbastion�[0m=i-268402f0e28b5e477 �[2mbastion_ip�[0m=192.168.0.202 �[2mprivate�[0m=i-db7f3ed6e4cdaa5aa �[2mprivate_ip�[0m=172.31.16.4
+    natgw_test.go:128: �[2mbastion�[0m=i-268402f0e28b5e477 �[2mbastion_ip�[0m=192.168.0.202 �[2mprivate�[0m=i-db7f3ed6e4cdaa5aa �[2mprivate_ip�[0m=172.31.16.4
+�[2m· waiting for SSH handshake 192.168.0.202:22�[0m
+    natgw_test.go:133: �[2m· waiting for SSH handshake 192.168.0.202:22�[0m
+�[2m· scp keypair -> bastion:/tmp/key.pem�[0m
+    natgw_test.go:139: �[2m· scp keypair -> bastion:/tmp/key.pem�[0m
+�[2m· wait for private SSH via bastion�[0m
+    natgw_test.go:151: �[2m· wait for private SSH via bastion�[0m
+�[2m· baseline: private VM has no internet�[0m
+    natgw_test.go:172: �[2m· baseline: private VM has no internet�[0m
+�[2m· allocate-address (NAT EIP)�[0m
+    natgw_test.go:178: �[2m· allocate-address (NAT EIP)�[0m
+�[2meip�[0m=192.168.0.203 �[2malloc�[0m=eipalloc-2d81753157cd62a50
+    natgw_test.go:195: �[2meip�[0m=192.168.0.203 �[2malloc�[0m=eipalloc-2d81753157cd62a50
+�[2m· create-nat-gateway in subnet-cd62e4c1c3cb50c63�[0m
+    natgw_test.go:197: �[2m· create-nat-gateway in subnet-cd62e4c1c3cb50c63�[0m
+�[2mnat_gw�[0m=nat-812216dd5105180c5
+    natgw_test.go:216: �[2mnat_gw�[0m=nat-812216dd5105180c5
+    natgw_test.go:218: nat gateway nat-812216dd5105180c5 reached state available
+�[2m· create-route-table�[0m
+    natgw_test.go:223: �[2m· create-route-table�[0m
+�[2m· associate-route-table rtb-8d6a4abd5f666ec13 <- subnet-2360a40e28371e214�[0m
+    natgw_test.go:241: �[2m· associate-route-table rtb-8d6a4abd5f666ec13 <- subnet-2360a40e28371e214�[0m
+�[2m· create-route 0.0.0.0/0 -> nat-812216dd5105180c5�[0m
+    natgw_test.go:259: �[2m· create-route 0.0.0.0/0 -> nat-812216dd5105180c5�[0m
+�[2m· verify private VM reaches 8.8.8.8 via NAT GW�[0m
+    natgw_test.go:279: �[2m· verify private VM reaches 8.8.8.8 via NAT GW�[0m
+�[2m· delete-nat-gateway nat-812216dd5105180c5�[0m
+    natgw_test.go:293: �[2m· delete-nat-gateway nat-812216dd5105180c5�[0m
+    natgw_test.go:318: nat gateway nat-812216dd5105180c5 reached state deleted
+�[2m· verify private VM lost internet after NAT GW teardown�[0m
+    natgw_test.go:329: �[2m· verify private VM lost internet after NAT GW teardown�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8e_SGToSGDatapath" classname="" time="80.940">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8e — Security Group SG-to-SG Datapath (OVN) ━━�[0m
+�[2m· 8e-1 create sge-client + sge-target security groups�[0m
+    datapath_test.go:61: �[2m· 8e-1 create sge-client + sge-target security groups�[0m
+�[2mclient_sg�[0m=sg-56e73c764a649f7ff �[2mtarget_sg�[0m=sg-67ad5efaa8be83b79
+    datapath_test.go:83: �[2mclient_sg�[0m=sg-56e73c764a649f7ff �[2mtarget_sg�[0m=sg-67ad5efaa8be83b79
+�[2m· 8e-1 authorize target-sg ingress tcp/8080 from sg-56e73c764a649f7ff�[0m
+    datapath_test.go:91: �[2m· 8e-1 authorize target-sg ingress tcp/8080 from sg-56e73c764a649f7ff�[0m
+�[2m· 8e-2 launch client-vm + target-vm�[0m
+    datapath_test.go:109: �[2m· 8e-2 launch client-vm + target-vm�[0m
+    datapath_test.go:132: instance i-eae6263d9b9fda069 reached state running
+    datapath_test.go:133: instance i-9d9411a83549d34e3 reached state running
+�[2mclient_priv�[0m=172.31.0.5 �[2mclient_eni�[0m=eni-0f878da0143028915 �[2mtarget_priv�[0m=172.31.0.6 �[2mtarget_eni�[0m=eni-3bd355e19c7fdd45c
+    datapath_test.go:142: �[2mclient_priv�[0m=172.31.0.5 �[2mclient_eni�[0m=eni-0f878da0143028915 �[2mtarget_priv�[0m=172.31.0.6 �[2mtarget_eni�[0m=eni-3bd355e19c7fdd45c
+�[2m· 8e-3 ovn-nbctl port_group membership�[0m
+    datapath_test.go:151: �[2m· 8e-3 ovn-nbctl port_group membership�[0m
+�[2mclient_pg�[0m=sg_56e73c764a649f7ff �[2mtarget_pg�[0m=sg_67ad5efaa8be83b79
+    datapath_test.go:163: �[2mclient_pg�[0m=sg_56e73c764a649f7ff �[2mtarget_pg�[0m=sg_67ad5efaa8be83b79
+�[2m· 8e-3 ovn-sbctl address_set sg_56e73c764a649f7ff_ip4 contains 172.31.0.5�[0m
+    datapath_test.go:170: �[2m· 8e-3 ovn-sbctl address_set sg_56e73c764a649f7ff_ip4 contains 172.31.0.5�[0m
+�[2m· 8e-3 wait for client-vm SSH at 192.168.0.202:22�[0m
+    datapath_test.go:188: �[2m· 8e-3 wait for client-vm SSH at 192.168.0.202:22�[0m
+�[2m· 8e-4 allowed traffic client -> target:172.31.0.6:8080�[0m
+    datapath_test.go:199: �[2m· 8e-4 allowed traffic client -> target:172.31.0.6:8080�[0m
+�[2mstep4�[0m=allowed_traffic_ok
+    datapath_test.go:208: �[2mstep4�[0m=allowed_traffic_ok
+�[2m· 8e-5 denied traffic client -> target:22 (no SSH ingress)�[0m
+    datapath_test.go:213: �[2m· 8e-5 denied traffic client -> target:22 (no SSH ingress)�[0m
+�[2mstep5�[0m=denied_traffic_ok
+    datapath_test.go:217: �[2mstep5�[0m=denied_traffic_ok
+�[2m· 8e-6 revoke target-sg ingress, retest (sync RequestEvent contract)�[0m
+    datapath_test.go:222: �[2m· 8e-6 revoke target-sg ingress, retest (sync RequestEvent contract)�[0m
+�[2m· 8e-6 verify client -> target:8080 now blocked�[0m
+    datapath_test.go:240: �[2m· 8e-6 verify client -> target:8080 now blocked�[0m
+�[2mstep6�[0m=revoke_blocked_ok
+    datapath_test.go:244: �[2mstep6�[0m=revoke_blocked_ok
+�[2m· 8e-7 re-authorize target-sg ingress, verify traffic restored�[0m
+    datapath_test.go:250: �[2m· 8e-7 re-authorize target-sg ingress, verify traffic restored�[0m
+�[2mstep7�[0m=restore_ok
+    datapath_test.go:271: �[2mstep7�[0m=restore_ok
+    datapath_test.go:127: instance i-eae6263d9b9fda069 reached state terminated
+    datapath_test.go:127: instance i-9d9411a83549d34e3 reached state terminated]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase9_Teardown" classname="" time="8.300">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 9 — Teardown ━━�[0m
+�[2m· delete-snapshot snap-dde89b9ea2ef2e05a (custom AMI backing)�[0m
+    teardown_test.go:33: �[2m· delete-snapshot snap-dde89b9ea2ef2e05a (custom AMI backing)�[0m
+�[2mdeleted_snapshot�[0m=snap-dde89b9ea2ef2e05a
+    teardown_test.go:39: �[2mdeleted_snapshot�[0m=snap-dde89b9ea2ef2e05a
+�[2m· deregister-image ami-7dcf263a412dd14c6�[0m
+    teardown_test.go:51: �[2m· deregister-image ami-7dcf263a412dd14c6�[0m
+�[2mderegistered_ami�[0m=ami-7dcf263a412dd14c6
+    teardown_test.go:57: �[2mderegistered_ami�[0m=ami-7dcf263a412dd14c6
+�[2m· terminate-instances i-0c615e5dd74050f53�[0m
+    teardown_test.go:74: �[2m· terminate-instances i-0c615e5dd74050f53�[0m
+�[2mterminated_instance�[0m=i-0c615e5dd74050f53
+    teardown_test.go:81: �[2mterminated_instance�[0m=i-0c615e5dd74050f53
+�[2m· delete-key-pair test-key-1�[0m
+    teardown_test.go:94: �[2m· delete-key-pair test-key-1�[0m
+�[2mdeleted_keypair�[0m=test-key-1
+    teardown_test.go:100: �[2mdeleted_keypair�[0m=test-key-1]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase9a_VerifyTeardown" classname="" time="0.560">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 9a — Verify Teardown ━━�[0m
+�[2m· describe-instances i-0c615e5dd74050f53�[0m
+    teardown_test.go:123: �[2m· describe-instances i-0c615e5dd74050f53�[0m
+�[2minstance_state�[0m=terminated
+    teardown_test.go:135: �[2minstance_state�[0m=terminated
+�[2m· describe-images ami-7dcf263a412dd14c6�[0m
+    teardown_test.go:147: �[2m· describe-images ami-7dcf263a412dd14c6�[0m
+�[2mami_reaped�[0m=ami-7dcf263a412dd14c6
+    teardown_test.go:159: �[2mami_reaped�[0m=ami-7dcf263a412dd14c6
+�[2m· describe-key-pairs (must not contain test-key-1)�[0m
+    teardown_test.go:167: �[2m· describe-key-pairs (must not contain test-key-1)�[0m
+�[2mkeypairs_listed�[0m=1
+    teardown_test.go:176: �[2mkeypairs_listed�[0m=1]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase9b_FinalClusterStats" classname="" time="3.070">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 9b — Final Cluster Stats ━━�[0m
+�[2mspx_get_vms_bytes�[0m=118
+    teardown_test.go:208: �[2mspx_get_vms_bytes�[0m=118]]></system-out>
+		</testcase>
+	</testsuite>
+</testsuites>

.github/actions/junit-report/.gitignore

@@ -0,0 +1 @@
+__pycache__/

.github/actions/junit-report/README.md

@@ -0,0 +1,46 @@
+# junit-report
+
+Composite action that renders a glob of JUnit XML files into the GitHub
+Actions job summary and emits one `::error` annotation per failed test case.
+
+## Usage
+
+```yaml
+- name: Publish test summary
+  if: always()
+  uses: ./spinifex/.github/actions/junit-report   # or your own checkout path
+  with:
+    junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+    title: "Go E2E results"
+    # fail-on-empty: "true"   # opt-in: fail the step if the glob matches nothing
+```
+
+Outputs:
+
+- `failures` — total `<failure>` + `<error>` counts across all matched files.
+- `total`    — total `<testcase>` count across all matched files.
+
+The action expects standard JUnit XML (e.g. emitted by
+`gotestsum --junitfile`, `go-junit-report`, or pytest's `--junitxml`). Both
+single `<testsuite>` and wrapping `<testsuites>` roots are accepted.
+
+## Companion convention: `::group::` for noisy steps
+
+Long-running shell phases (tofu apply, package installs, tar-over-ssh) should
+wrap their output in `::group::name` / `::endgroup::` markers so the Actions
+UI collapses them by default. Pattern used in the e2e-go / e2e-ddil workflows:
+
+```bash
+run() {
+  local label="$1"; shift
+  echo "::group::${label}"
+  "$@" 2>&1 | tee -a "$TOFU_LOG"
+  local rc=${PIPESTATUS[0]}
+  echo "::endgroup::"
+  return "$rc"
+}
+run "tofu apply" tofu apply -var-file=... -auto-approve -input=false
+```
+
+Combined: groups make the live UI readable; this action makes the run header
+land on the failures without scrolling the log.

.github/actions/junit-report/action.yml

@@ -0,0 +1,41 @@
+name: JUnit Report
+description: >-
+  Render JUnit XML files into the GitHub Actions job summary and emit
+  inline ::error annotations for each failed test case. Designed to be
+  called once per job after the test step (with `if: always()`) so summary
+  + annotations appear on both passing and failing runs.
+
+inputs:
+  junit-glob:
+    description: Shell glob matching JUnit XML files to render.
+    required: true
+  title:
+    description: Heading rendered at the top of the job summary section.
+    required: false
+    default: "Test results"
+  fail-on-empty:
+    description: >-
+      When "true", the action fails if the glob matches no files. Useful
+      for catching silently-skipped test phases.
+    required: false
+    default: "false"
+
+outputs:
+  failures:
+    description: Total failed + errored test cases across all matched files.
+    value: ${{ steps.render.outputs.failures }}
+  total:
+    description: Total test cases across all matched files.
+    value: ${{ steps.render.outputs.total }}
+
+runs:
+  using: composite
+  steps:
+    - id: render
+      shell: bash
+      env:
+        JUNIT_GLOB: ${{ inputs.junit-glob }}
+        JUNIT_TITLE: ${{ inputs.title }}
+        JUNIT_FAIL_ON_EMPTY: ${{ inputs.fail-on-empty }}
+      run: |
+        python3 "${{ github.action_path }}/render.py"

.github/actions/junit-report/render.py

@@ -0,0 +1,281 @@
+#!/usr/bin/env python3
+"""Render JUnit XML files into GitHub Actions job summary + annotations.
+
+Inputs (env):
+    JUNIT_GLOB           Glob matching JUnit XML files.
+    JUNIT_TITLE          Heading text for the summary section.
+    JUNIT_FAIL_ON_EMPTY  "true" to exit non-zero when glob matches nothing.
+
+GitHub-Actions outputs (via $GITHUB_OUTPUT):
+    failures             Sum of <failures> + <errors> across all suites.
+    total                Sum of <tests> across all suites.
+
+Side effects:
+    Appends a markdown section to $GITHUB_STEP_SUMMARY with a per-file
+    totals table and a collapsible details block per failed test.
+    Emits one `::error` workflow command per failed/errored test case so
+    they surface as annotations in the run UI.
+"""
+
+from __future__ import annotations
+
+import glob
+import os
+import re
+import sys
+import xml.etree.ElementTree as ET
+from dataclasses import dataclass
+
+
+# Strip ANSI SGR escapes from skipped/failure CDATA so the rendered reason is
+# plain text (CDATA preserves the colour codes Go emits via harness.Phase).
+_ANSI_RE = re.compile(r"\x1b\[[0-9;]*m")
+
+
+def _strip_ansi(s: str) -> str:
+    return _ANSI_RE.sub("", s)
+
+
+# Last `<file>.go:<line>: <reason>` line in a CDATA block is the t.Skipf /
+# t.Fatal call site — that's the actionable signal. Phase banners and probe
+# logs above it are noise for the summary view.
+_SKIP_REASON_RE = re.compile(r"([A-Za-z0-9_.\-/]+\.go:\d+:\s*.+)$")
+
+
+def _extract_skip_reason(cdata: str) -> str:
+    cdata = _strip_ansi(cdata).strip()
+    if not cdata:
+        return ""
+    for line in reversed(cdata.splitlines()):
+        line = line.strip()
+        m = _SKIP_REASON_RE.search(line)
+        if m:
+            return m.group(1).strip()
+    return cdata.splitlines()[-1].strip()
+
+
+# Workflow commands carry their payload on one line; literal newlines must
+# be percent-escaped per GitHub's workflow-command spec.
+def _esc(s: str) -> str:
+    return (
+        s.replace("%", "%25")
+        .replace("\r", "%0D")
+        .replace("\n", "%0A")
+    )
+
+
+@dataclass
+class Case:
+    suite: str
+    classname: str
+    name: str
+    time: float
+    status: str  # "pass" | "fail" | "error" | "skip"
+    message: str
+    detail: str
+
+
+@dataclass
+class FileTotals:
+    path: str
+    tests: int
+    failures: int
+    errors: int
+    skipped: int
+    time: float
+    cases: list[Case]
+
+
+def _parse_file(path: str) -> FileTotals:
+    root = ET.parse(path).getroot()
+    # JUnit XML has two shapes: <testsuite> as root, or <testsuites> wrapping
+    # multiple <testsuite> children. Iterating with .iter() handles both.
+    suites = list(root.iter("testsuite"))
+    tests = sum(int(s.get("tests", 0) or 0) for s in suites)
+    failures = sum(int(s.get("failures", 0) or 0) for s in suites)
+    errors = sum(int(s.get("errors", 0) or 0) for s in suites)
+    skipped = sum(int(s.get("skipped", 0) or 0) for s in suites)
+    time = sum(float(s.get("time", 0) or 0) for s in suites)
+
+    cases: list[Case] = []
+    for s in suites:
+        suite_name = s.get("name", "")
+        for tc in s.findall("testcase"):
+            status = "pass"
+            message = ""
+            detail = ""
+            for tag in ("failure", "error"):
+                node = tc.find(tag)
+                if node is not None:
+                    status = "fail" if tag == "failure" else "error"
+                    message = (node.get("message") or "").strip()
+                    detail = (node.text or "").strip()
+                    break
+            skip_node = tc.find("skipped")
+            if skip_node is not None and status == "pass":
+                status = "skip"
+                # Skipped reason: prefer the CDATA call-site line, fall back
+                # to the @message attribute if CDATA is empty.
+                cdata = (skip_node.text or "").strip()
+                message = _extract_skip_reason(cdata) or (
+                    skip_node.get("message") or ""
+                ).strip()
+            cases.append(
+                Case(
+                    suite=suite_name,
+                    classname=tc.get("classname", ""),
+                    name=tc.get("name", ""),
+                    time=float(tc.get("time", 0) or 0),
+                    status=status,
+                    message=message,
+                    detail=detail,
+                )
+            )
+
+    # Drop container test cases: any case whose name is a strict prefix of
+    # another case's name (with "/" separator) is a parent t.Run wrapper.
+    # Go reports the parent's status as a rollup of its children, so counting
+    # both double-tallies the same outcome (e.g. TestSingleNode failing
+    # because Phase8b_VPCSubnetE2E failed would show as 2 failures, not 1).
+    names = {c.name for c in cases}
+    prefixes = {n.rsplit("/", 1)[0] for n in names if "/" in n}
+    # A name is a container if some other name starts with "<name>/".
+    # Use the prefixes set rather than O(n^2) scanning.
+    def _is_container(name: str) -> bool:
+        return name in prefixes
+
+    leaf_cases = [c for c in cases if not _is_container(c.name)]
+
+    # Recompute totals from leaf cases so the per-file counts in the summary
+    # table match the de-duplicated failure list below it.
+    leaf_tests = len(leaf_cases)
+    leaf_failures = sum(1 for c in leaf_cases if c.status == "fail")
+    leaf_errors = sum(1 for c in leaf_cases if c.status == "error")
+    leaf_skipped = sum(1 for c in leaf_cases if c.status == "skip")
+
+    return FileTotals(
+        path=path,
+        tests=leaf_tests,
+        failures=leaf_failures,
+        errors=leaf_errors,
+        skipped=leaf_skipped,
+        time=time,
+        cases=leaf_cases,
+    )
+
+
+def _file_label(path: str) -> str:
+    base = os.path.basename(path)
+    if base.endswith(".xml"):
+        base = base[:-4]
+    if base.startswith("junit-"):
+        base = base[len("junit-"):]
+    return base or path
+
+
+def main() -> int:
+    pattern = os.environ.get("JUNIT_GLOB", "").strip()
+    title = os.environ.get("JUNIT_TITLE", "Test results").strip() or "Test results"
+    fail_on_empty = os.environ.get("JUNIT_FAIL_ON_EMPTY", "").strip().lower() == "true"
+
+    if not pattern:
+        print("::error::junit-report: junit-glob input is empty", file=sys.stderr)
+        return 1
+
+    paths = sorted(glob.glob(pattern))
+    summary_path = os.environ.get("GITHUB_STEP_SUMMARY", "/dev/null")
+    output_path = os.environ.get("GITHUB_OUTPUT", "/dev/null")
+
+    with open(summary_path, "a", encoding="utf-8") as summary, \
+            open(output_path, "a", encoding="utf-8") as outputs:
+
+        summary.write(f"## {title}\n\n")
+
+        if not paths:
+            summary.write(f"_No JUnit files matched `{pattern}`._\n\n")
+            outputs.write("failures=0\n")
+            outputs.write("total=0\n")
+            if fail_on_empty:
+                print(f"::error::junit-report: no files matched {pattern}", file=sys.stderr)
+                return 1
+            return 0
+
+        totals_by_file = [_parse_file(p) for p in paths]
+        grand_total = sum(f.tests for f in totals_by_file)
+        grand_fail = sum(f.failures + f.errors for f in totals_by_file)
+        grand_skip = sum(f.skipped for f in totals_by_file)
+        grand_pass = grand_total - grand_fail - grand_skip
+        grand_time = sum(f.time for f in totals_by_file)
+
+        if grand_fail == 0:
+            verdict = f"✅ **pass** — {grand_pass} / {grand_total}"
+        else:
+            verdict = f"❌ **{grand_fail} failed** — {grand_pass} / {grand_total} pass"
+        skip_part = f", {grand_skip} skipped" if grand_skip else ""
+        summary.write(f"{verdict}{skip_part} &nbsp; · &nbsp; {grand_time:.1f}s\n\n")
+
+        # Compact table — single status column, shorter headers. Drops
+        # separate err column (errors fold into fail count) so narrow
+        # screens don't wrap.
+        summary.write("| | Suite | Pass | Fail | Skip | Time |\n")
+        summary.write("|:-:|---|---:|---:|---:|---:|\n")
+        for f in totals_by_file:
+            fails = f.failures + f.errors
+            passed = f.tests - fails - f.skipped
+            icon = "✅" if fails == 0 else "❌"
+            summary.write(
+                f"| {icon} | `{_file_label(f.path)}` | {passed} | {fails} | "
+                f"{f.skipped} | {f.time:.1f}s |\n"
+            )
+        summary.write("\n")
+
+        failed_cases = [
+            c for f in totals_by_file for c in f.cases if c.status in ("fail", "error")
+        ]
+        if failed_cases:
+            summary.write("---\n\n**Failures**\n\n")
+            for c in failed_cases:
+                header = f"{c.classname}.{c.name}" if c.classname else c.name
+                # One line per failure: name + truncated message. Full
+                # detail lives in the test artifact log; deep nesting and
+                # long messages wrap badly on narrow screens.
+                msg_short = (c.message or "").splitlines()[0] if c.message else ""
+                if len(msg_short) > 120:
+                    msg_short = msg_short[:117] + "…"
+                line = f"- ❌ `{header}`"
+                if msg_short:
+                    line += f" — {msg_short}"
+                summary.write(line + "\n")
+
+                # Workflow annotation — surfaces in run header + Files-changed view.
+                # classname for Go tests is the package path; map to a file when we can.
+                file_hint = c.classname.replace(".", "/")
+                msg = c.message or f"{c.status}: {header}"
+                print(f"::error file={file_hint},title={header}::{_esc(msg)}")
+            summary.write("\n")
+
+        skipped_cases = [
+            c for f in totals_by_file for c in f.cases if c.status == "skip"
+        ]
+        if skipped_cases:
+            summary.write("---\n\n<details><summary><b>Skipped</b> ")
+            summary.write(f"({len(skipped_cases)})</summary>\n\n")
+            for c in skipped_cases:
+                header = f"{c.classname}.{c.name}" if c.classname else c.name
+                reason = (c.message or "").splitlines()[0] if c.message else ""
+                if len(reason) > 160:
+                    reason = reason[:157] + "…"
+                line = f"- ⏭️ `{header}`"
+                if reason:
+                    line += f" — {reason}"
+                summary.write(line + "\n")
+            summary.write("\n</details>\n\n")
+
+        outputs.write(f"failures={grand_fail}\n")
+        outputs.write(f"total={grand_total}\n")
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

.github/assets/architecture.svg

@@ -0,0 +1,225 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="620" viewBox="0 0 900 620">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+    <linearGradient id="bar-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#161b22"/>
+      <stop offset="45%" style="stop-color:#1f2028"/>
+      <stop offset="100%" style="stop-color:#161b22"/>
+    </linearGradient>
+    <linearGradient id="client-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#3fb95022"/>
+      <stop offset="100%" style="stop-color:#3fb95008"/>
+    </linearGradient>
+    <linearGradient id="gw-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#DC262633"/>
+      <stop offset="100%" style="stop-color:#7F1D1D14"/>
+    </linearGradient>
+    <linearGradient id="nats-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#F87171"/>
+      <stop offset="50%" style="stop-color:#DC2626"/>
+      <stop offset="100%" style="stop-color:#7F1D1D"/>
+    </linearGradient>
+  </defs>
+
+  <rect width="900" height="620" fill="url(#bg)" rx="12"/>
+
+  <!-- Title bar -->
+  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">SPINIFEX ARCHITECTURE</text>
+  <text x="24" y="52" font-family="'Segoe UI', system-ui, sans-serif" font-size="16" fill="#f5f5f5" font-weight="500">AWS SDK → Gateway → NATS → Service daemons</text>
+  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">stateless · message-driven · horizontally scalable</text>
+
+  <!-- ═══ CLIENT LAYER ════════════════════════════════════ -->
+  <rect x="16" y="72" width="868" height="78" rx="8" fill="url(#client-grad)" stroke="#3fb950" stroke-width="1" stroke-opacity="0.5"/>
+  <text x="32" y="93" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" font-weight="700" letter-spacing="1">CLIENTS</text>
+  <text x="32" y="108" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">Standard AWS tooling — unmodified, no shims, no custom SDK</text>
+
+  <g transform="translate(32, 120)" font-family="'SF Mono', ui-monospace, monospace" font-size="10">
+    <rect x="0" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
+    <text x="43" y="15" fill="#7ee787" text-anchor="middle">aws-cli</text>
+
+    <rect x="94" y="0" width="96" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
+    <text x="142" y="15" fill="#7ee787" text-anchor="middle">Terraform</text>
+
+    <rect x="198" y="0" width="76" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
+    <text x="236" y="15" fill="#7ee787" text-anchor="middle">boto3</text>
+
+    <rect x="282" y="0" width="98" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
+    <text x="331" y="15" fill="#7ee787" text-anchor="middle">aws-sdk-go</text>
+
+    <rect x="388" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
+    <text x="431" y="15" fill="#7ee787" text-anchor="middle">kubectl</text>
+
+    <rect x="482" y="0" width="86" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
+    <text x="525" y="15" fill="#7ee787" text-anchor="middle">Packer</text>
+
+    <rect x="576" y="0" width="120" height="22" rx="4" fill="#3fb950" fill-opacity="0.12" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.8"/>
+    <text x="636" y="15" fill="#7ee787" text-anchor="middle">your application</text>
+  </g>
+
+  <!-- ═══ HTTPS + SigV4 connector ═══════════════════════════ -->
+  <line x1="450" y1="150" x2="450" y2="178" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
+  <rect x="390" y="154" width="120" height="18" rx="9" fill="#0a0a0a" stroke="#DC2626" stroke-width="0.8" stroke-opacity="0.5"/>
+  <text x="450" y="166" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#F87171" text-anchor="middle">HTTPS · SigV4 · :9999</text>
+
+  <!-- ═══ AWS GATEWAY ════════════════════════════════════ -->
+  <rect x="16" y="178" width="868" height="68" rx="8" fill="url(#gw-grad)" stroke="#DC2626" stroke-width="1.3" stroke-opacity="0.75"/>
+  <text x="32" y="200" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" font-weight="700" letter-spacing="1">AWS GATEWAY</text>
+  <text x="32" y="215" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">spinifex/services/awsgw · TLS termination · SigV4 verification · IAM policy eval · NATS routing</text>
+
+  <g transform="translate(32, 224)" font-family="'SF Mono', ui-monospace, monospace" font-size="9">
+    <rect x="0" y="0" width="78" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="39" y="11" fill="#FCA5A5" text-anchor="middle">TLS 1.3</text>
+
+    <rect x="86" y="0" width="78" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="125" y="11" fill="#FCA5A5" text-anchor="middle">SigV4 auth</text>
+
+    <rect x="172" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="221" y="11" fill="#FCA5A5" text-anchor="middle">policy eval</text>
+
+    <rect x="278" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="327" y="11" fill="#FCA5A5" text-anchor="middle">req routing</text>
+
+    <rect x="384" y="0" width="88" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="428" y="11" fill="#FCA5A5" text-anchor="middle">rate-limit</text>
+
+    <rect x="480" y="0" width="98" height="16" rx="3" fill="#7F1D1D" fill-opacity="0.3" stroke="#DC2626" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="529" y="11" fill="#FCA5A5" text-anchor="middle">audit log</text>
+  </g>
+
+  <!-- ═══ NATS BUS ═══════════════════════════════════════ -->
+  <line x1="450" y1="246" x2="450" y2="272" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
+  <rect x="16" y="272" width="868" height="34" rx="6" fill="url(#nats-grad)" opacity="0.92"/>
+  <text x="32" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#f5f5f5" font-weight="700" letter-spacing="1.5">NATS BUS</text>
+  <text x="140" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#fee2e2" opacity="0.95">ec2.*   ebs.*   s3.*   vpc.*   iam.*</text>
+  <text x="876" y="293" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#fee2e2" text-anchor="end" opacity="0.9">queue groups · fan-out · request/reply</text>
+
+  <!-- ═══ DAEMON FANOUT CONNECTORS ═══════════════════════ -->
+  <g stroke="#DC2626" stroke-width="0.9" stroke-dasharray="3,3" opacity="0.55" fill="none">
+    <line x1="100" y1="306" x2="100" y2="328"/>
+    <line x1="275" y1="306" x2="275" y2="328"/>
+    <line x1="450" y1="306" x2="450" y2="328"/>
+    <line x1="625" y1="306" x2="625" y2="328"/>
+    <line x1="800" y1="306" x2="800" y2="328"/>
+  </g>
+
+  <!-- ═══ SERVICE DAEMONS ROW ═══════════════════════════ -->
+  <!-- EC2 -->
+  <g transform="translate(28, 328)">
+    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
+    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">ec2.* daemon</text>
+    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
+    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">RunInstances</text>
+    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">TerminateInstances</text>
+    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">QMP · cloud-init</text>
+  </g>
+
+  <!-- EBS -->
+  <g transform="translate(204, 328)">
+    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
+    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">ebs.* daemon</text>
+    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
+    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">CreateVolume</text>
+    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">AttachVolume</text>
+    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">snapshots · WAL</text>
+  </g>
+
+  <!-- S3 -->
+  <g transform="translate(380, 328)">
+    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
+    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">s3.* daemon</text>
+    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
+    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">PutObject · GetObject</text>
+    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Multipart · presigned</text>
+    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">erasure-coded</text>
+  </g>
+
+  <!-- VPC -->
+  <g transform="translate(556, 328)">
+    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
+    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">vpc.* daemon</text>
+    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
+    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Subnets · SGs · RTs</text>
+    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">ALB · NAT · IGW</text>
+    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">OVN · OVS</text>
+  </g>
+
+  <!-- IAM -->
+  <g transform="translate(732, 328)">
+    <rect width="144" height="82" rx="7" fill="#7F1D1D" fill-opacity="0.14" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
+    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">iam.* daemon</text>
+    <line x1="16" y1="30" x2="128" y2="30" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
+    <text x="72" y="48" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Users · Roles · Keys</text>
+    <text x="72" y="62" font-family="'Segoe UI', sans-serif" font-size="10" fill="#f5f5f5" text-anchor="middle">Policy evaluation</text>
+    <text x="72" y="76" font-family="'Segoe UI', sans-serif" font-size="9" fill="#a1a1aa" text-anchor="middle">audit trail</text>
+  </g>
+
+  <!-- ═══ DAEMON → BACKEND CONNECTORS ═══════════════════ -->
+  <g stroke="#71717a" stroke-width="0.8" stroke-dasharray="2,3" opacity="0.55" fill="none">
+    <line x1="100" y1="410" x2="100" y2="440"/>
+    <line x1="276" y1="410" x2="276" y2="440"/>
+    <line x1="452" y1="410" x2="452" y2="440"/>
+    <line x1="628" y1="410" x2="628" y2="440"/>
+    <line x1="804" y1="410" x2="804" y2="440"/>
+  </g>
+
+  <!-- ═══ BACKEND RESOURCES BAR ═════════════════════════ -->
+  <rect x="16" y="440" width="868" height="26" rx="5" fill="url(#bar-grad)"/>
+  <text x="450" y="458" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle" letter-spacing="1.5">BACKEND RESOURCES · local hardware · no cloud dependency</text>
+
+  <!-- ═══ BACKEND ROW ═══════════════════════════════════ -->
+  <!-- QEMU/KVM -->
+  <g transform="translate(28, 474)">
+    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
+    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">QEMU / KVM</text>
+    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">hypervisor</text>
+    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">host kernel · virtio</text>
+  </g>
+
+  <!-- Viperblock -->
+  <g transform="translate(204, 474)">
+    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
+    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Viperblock</text>
+    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">block storage</text>
+    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">NVMe WAL · NBD</text>
+  </g>
+
+  <!-- Predastore -->
+  <g transform="translate(380, 474)">
+    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
+    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Predastore</text>
+    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">object storage</text>
+    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">Reed–Solomon</text>
+  </g>
+
+  <!-- OVN/OVS -->
+  <g transform="translate(556, 474)">
+    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
+    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">OVN / OVS</text>
+    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">SDN fabric</text>
+    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">overlays · flow rules</text>
+  </g>
+
+  <!-- IAM store -->
+  <g transform="translate(732, 474)">
+    <rect width="144" height="62" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
+    <text x="72" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" text-anchor="middle" font-weight="600">Identity store</text>
+    <text x="72" y="40" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">users · keys · rules</text>
+    <text x="72" y="54" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a" text-anchor="middle">local · audit-logged</text>
+  </g>
+
+  <!-- ═══ FOOTER NOTES ══════════════════════════════════ -->
+  <g transform="translate(16, 558)">
+    <rect width="868" height="48" rx="6" fill="#0a0a0a" stroke="#27272a" stroke-width="0.8"/>
+    <text x="24" y="20" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">
+      <tspan fill="#F87171" font-weight="600">Queue groups</tspan> load-balance ops across daemon replicas.
+      <tspan fill="#F87171" font-weight="600"> Fan-out</tspan> (no queue group) dispatches describe calls to every node.
+    </text>
+    <text x="24" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">
+      Daemons are <tspan fill="#F87171" font-weight="600">stateless</tspan> — scale horizontally by starting more.
+      No etcd, no Kubernetes, no external control plane. Systemd units, a NATS cluster, and your hardware.
+    </text>
+  </g>
+</svg>

.github/assets/banner.svg

@@ -0,0 +1,154 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="300" viewBox="0 0 900 300">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a;stop-opacity:1" />
+      <stop offset="50%" style="stop-color:#0d1117;stop-opacity:1" />
+      <stop offset="100%" style="stop-color:#0a0a0a;stop-opacity:1" />
+    </linearGradient>
+    <radialGradient id="vignette" cx="50%" cy="50%" r="70%">
+      <stop offset="0%" style="stop-color:#7F1D1D;stop-opacity:0.08"/>
+      <stop offset="100%" style="stop-color:#7F1D1D;stop-opacity:0"/>
+    </radialGradient>
+    <linearGradient id="mulga-grad" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#DC2626;stop-opacity:0.95" />
+      <stop offset="100%" style="stop-color:#7F1D1D;stop-opacity:0.95" />
+    </linearGradient>
+    <linearGradient id="rule" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#DC2626;stop-opacity:0" />
+      <stop offset="50%" style="stop-color:#F87171;stop-opacity:0.45" />
+      <stop offset="100%" style="stop-color:#7F1D1D;stop-opacity:0" />
+    </linearGradient>
+    <filter id="node-glow">
+      <feGaussianBlur in="SourceGraphic" stdDeviation="3" />
+    </filter>
+  </defs>
+
+  <!-- Background -->
+  <rect width="900" height="300" fill="url(#bg)" rx="12"/>
+  <rect width="900" height="300" fill="url(#vignette)" rx="12"/>
+
+  <!-- Mesh network background (mulgadc style) -->
+  <g opacity="0.35">
+    <!-- Connections (drawn first, under nodes) -->
+    <g stroke="#7F1D1D" stroke-width="0.6" fill="none">
+      <line x1="80" y1="60" x2="200" y2="40"/>
+      <line x1="80" y1="60" x2="150" y2="140"/>
+      <line x1="200" y1="40" x2="330" y2="90"/>
+      <line x1="150" y1="140" x2="260" y2="180"/>
+      <line x1="150" y1="140" x2="60" y2="220"/>
+      <line x1="330" y1="90" x2="260" y2="180"/>
+      <line x1="260" y1="180" x2="120" y2="260"/>
+      <line x1="60" y1="220" x2="120" y2="260"/>
+
+      <line x1="620" y1="50" x2="760" y2="90"/>
+      <line x1="620" y1="50" x2="700" y2="160"/>
+      <line x1="760" y1="90" x2="850" y2="160"/>
+      <line x1="700" y1="160" x2="820" y2="220"/>
+      <line x1="850" y1="160" x2="820" y2="220"/>
+      <line x1="700" y1="160" x2="600" y2="240"/>
+      <line x1="600" y1="240" x2="820" y2="220"/>
+      <line x1="760" y1="90" x2="700" y2="160"/>
+    </g>
+    <!-- Nodes -->
+    <g fill="#DC2626">
+      <circle cx="80" cy="60" r="2.5"/>
+      <circle cx="200" cy="40" r="2"/>
+      <circle cx="330" cy="90" r="2.5"/>
+      <circle cx="150" cy="140" r="3"/>
+      <circle cx="260" cy="180" r="2"/>
+      <circle cx="60" cy="220" r="2"/>
+      <circle cx="120" cy="260" r="2.5"/>
+
+      <circle cx="620" cy="50" r="2"/>
+      <circle cx="760" cy="90" r="3"/>
+      <circle cx="700" cy="160" r="2.5"/>
+      <circle cx="850" cy="160" r="2"/>
+      <circle cx="820" cy="220" r="2.5"/>
+      <circle cx="600" cy="240" r="2"/>
+    </g>
+    <!-- Pulse rings (static "about-to-pulse" look) -->
+    <g fill="none" stroke="#F87171" stroke-width="0.8" opacity="0.5">
+      <circle cx="150" cy="140" r="6"/>
+      <circle cx="760" cy="90" r="6"/>
+      <circle cx="700" cy="160" r="8"/>
+    </g>
+  </g>
+
+  <!-- Main emblem: cluster-node stack with NATS bus -->
+  <g transform="translate(200, 150)">
+    <!-- Outer glow ring -->
+    <circle cx="0" cy="0" r="64" fill="none" stroke="url(#mulga-grad)" stroke-width="1.5" opacity="0.35" filter="url(#node-glow)"/>
+
+    <!-- Top node -->
+    <rect x="-24" y="-52" width="48" height="22" rx="3" fill="none" stroke="#DC2626" stroke-width="1.8"/>
+    <circle cx="-16" cy="-41" r="1.6" fill="#F87171"/>
+    <circle cx="-8" cy="-41" r="1.6" fill="#F87171"/>
+    <line x1="0" y1="-41" x2="16" y2="-41" stroke="#F87171" stroke-width="0.9" opacity="0.6"/>
+
+    <!-- Middle node (active) -->
+    <rect x="-30" y="-14" width="60" height="28" rx="3" fill="url(#mulga-grad)" opacity="0.22" stroke="#DC2626" stroke-width="2"/>
+    <circle cx="-20" cy="0" r="2" fill="#F87171"/>
+    <circle cx="-10" cy="0" r="2" fill="#F87171"/>
+    <line x1="-2" y1="0" x2="22" y2="0" stroke="#F87171" stroke-width="1.2" opacity="0.8"/>
+
+    <!-- Bottom node -->
+    <rect x="-24" y="30" width="48" height="22" rx="3" fill="none" stroke="#7F1D1D" stroke-width="1.8"/>
+    <circle cx="-16" cy="41" r="1.6" fill="#DC2626"/>
+    <circle cx="-8" cy="41" r="1.6" fill="#DC2626"/>
+    <line x1="0" y1="41" x2="16" y2="41" stroke="#DC2626" stroke-width="0.9" opacity="0.6"/>
+
+    <!-- Radiating mesh ports -->
+    <g stroke-width="0.9" opacity="0.55">
+      <line x1="-30" y1="0" x2="-58" y2="-26" stroke="#DC2626"/>
+      <line x1="-30" y1="0" x2="-64" y2="0" stroke="#DC2626"/>
+      <line x1="-30" y1="0" x2="-58" y2="26" stroke="#DC2626"/>
+      <line x1="30" y1="0" x2="58" y2="-26" stroke="#7F1D1D"/>
+      <line x1="30" y1="0" x2="64" y2="0" stroke="#7F1D1D"/>
+      <line x1="30" y1="0" x2="58" y2="26" stroke="#7F1D1D"/>
+    </g>
+    <g fill="#F87171">
+      <circle cx="-58" cy="-26" r="2.5"/>
+      <circle cx="-64" cy="0" r="2.5"/>
+      <circle cx="-58" cy="26" r="2.5"/>
+    </g>
+    <g fill="#DC2626">
+      <circle cx="58" cy="-26" r="2.5"/>
+      <circle cx="64" cy="0" r="2.5"/>
+      <circle cx="58" cy="26" r="2.5"/>
+    </g>
+  </g>
+
+  <!-- Title -->
+  <text x="340" y="120" font-family="'Segoe UI', system-ui, -apple-system, sans-serif" font-size="56" font-weight="700" fill="#f5f5f5" letter-spacing="-1.5">Spinifex</text>
+
+  <!-- Subtitle -->
+  <text x="340" y="155" font-family="'Segoe UI', system-ui, -apple-system, sans-serif" font-size="17" fill="#a1a1aa" letter-spacing="0.3">Open-source AWS-compatible platform</text>
+  <text x="340" y="180" font-family="'Segoe UI', system-ui, -apple-system, sans-serif" font-size="17" fill="#a1a1aa" letter-spacing="0.3">for secure edge deployments</text>
+
+  <!-- Rule -->
+  <rect x="340" y="200" width="440" height="1" fill="url(#rule)" rx="1"/>
+
+  <!-- Service pills -->
+  <g font-family="'Segoe UI', system-ui, -apple-system, sans-serif" font-size="11" font-weight="500">
+    <rect x="340" y="215" width="50" height="24" rx="12" fill="#7F1D1D" fill-opacity="0.25" stroke="#DC2626" stroke-width="0.5" stroke-opacity="0.55"/>
+    <text x="365" y="231" fill="#F87171" text-anchor="middle">EC2</text>
+
+    <rect x="398" y="215" width="50" height="24" rx="12" fill="#7F1D1D" fill-opacity="0.25" stroke="#DC2626" stroke-width="0.5" stroke-opacity="0.55"/>
+    <text x="423" y="231" fill="#F87171" text-anchor="middle">EBS</text>
+
+    <rect x="456" y="215" width="45" height="24" rx="12" fill="#7F1D1D" fill-opacity="0.25" stroke="#DC2626" stroke-width="0.5" stroke-opacity="0.55"/>
+    <text x="478" y="231" fill="#F87171" text-anchor="middle">S3</text>
+
+    <rect x="509" y="215" width="55" height="24" rx="12" fill="#7F1D1D" fill-opacity="0.25" stroke="#DC2626" stroke-width="0.5" stroke-opacity="0.55"/>
+    <text x="536" y="231" fill="#F87171" text-anchor="middle">VPC</text>
+
+    <rect x="572" y="215" width="50" height="24" rx="12" fill="#7F1D1D" fill-opacity="0.25" stroke="#DC2626" stroke-width="0.5" stroke-opacity="0.55"/>
+    <text x="597" y="231" fill="#F87171" text-anchor="middle">IAM</text>
+
+    <rect x="630" y="215" width="78" height="24" rx="12" fill="#18181b" stroke="#52525b" stroke-width="0.5" stroke-opacity="0.8"/>
+    <text x="669" y="231" fill="#a1a1aa" text-anchor="middle">Air-Gapped</text>
+  </g>
+
+  <!-- Tagline -->
+  <text x="340" y="265" font-family="'Segoe UI', system-ui, -apple-system, sans-serif" font-size="11" fill="#71717a" letter-spacing="1.8">YOUR INFRASTRUCTURE · YOUR HARDWARE · YOUR CLOUD</text>
+</svg>

.github/assets/design-architecture.svg

@@ -0,0 +1,175 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="720" viewBox="0 0 900 720">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+    <linearGradient id="bar-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#161b22"/>
+      <stop offset="45%" style="stop-color:#1f2028"/>
+      <stop offset="100%" style="stop-color:#161b22"/>
+    </linearGradient>
+    <linearGradient id="client-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#3fb95022"/>
+      <stop offset="100%" style="stop-color:#3fb95008"/>
+    </linearGradient>
+    <linearGradient id="gw-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#DC262633"/>
+      <stop offset="100%" style="stop-color:#7F1D1D14"/>
+    </linearGradient>
+    <linearGradient id="nats-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#F87171"/>
+      <stop offset="50%" style="stop-color:#DC2626"/>
+      <stop offset="100%" style="stop-color:#7F1D1D"/>
+    </linearGradient>
+    <linearGradient id="daemon-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#DC262622"/>
+      <stop offset="100%" style="stop-color:#7F1D1D0a"/>
+    </linearGradient>
+  </defs>
+
+  <rect width="900" height="720" fill="url(#bg)" rx="12"/>
+
+  <!-- Title -->
+  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">HIGH-LEVEL ARCHITECTURE</text>
+  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">SDK → Gateway → NATS → Daemons → VMs</text>
+
+  <!-- ═══ AWS SDK / CLI ══════════════════════════════════ -->
+  <rect x="16" y="50" width="868" height="76" rx="8" fill="url(#client-grad)" stroke="#3fb950" stroke-width="1" stroke-opacity="0.5"/>
+  <text x="32" y="72" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700" letter-spacing="1">AWS SDK / CLI</text>
+  <text x="32" y="90" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">Users interact with Spinifex using standard AWS SDKs and CLIs against a custom endpoint URL</text>
+  <g font-family="'SF Mono', ui-monospace, monospace" font-size="10.5">
+    <rect x="32" y="98" width="480" height="20" rx="4" fill="#0a0a0a" stroke="#3fb950" stroke-width="0.6" stroke-opacity="0.4"/>
+    <text x="42" y="112" fill="#7ee787">$ aws ec2 run-instances --endpoint-url https://localhost:9999 ...</text>
+  </g>
+
+  <!-- Connector: HTTPS + SigV4 -->
+  <line x1="450" y1="126" x2="450" y2="154" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
+  <rect x="380" y="128" width="140" height="20" rx="10" fill="#0a0a0a" stroke="#DC2626" stroke-width="0.8" stroke-opacity="0.5"/>
+  <text x="450" y="142" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#F87171" text-anchor="middle">HTTPS · SigV4 Auth</text>
+
+  <!-- ═══ AWS GATEWAY ════════════════════════════════════ -->
+  <rect x="16" y="154" width="868" height="104" rx="8" fill="url(#gw-grad)" stroke="#DC2626" stroke-width="1.3" stroke-opacity="0.75"/>
+  <text x="32" y="176" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" font-weight="700" letter-spacing="1">AWS GATEWAY · port 9999</text>
+  <text x="876" y="176" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#71717a" text-anchor="end">spinifex/services/awsgw/awsgw.go</text>
+
+  <!-- Two-column feature bullets -->
+  <g font-family="'Segoe UI', sans-serif" font-size="10.5" fill="#f5f5f5">
+    <text x="40" y="200">● TLS termination</text>
+    <text x="40" y="220">● EC2 Query Protocol parsing</text>
+    <text x="40" y="240">● XML response generation</text>
+
+    <text x="460" y="200">● AWS SigV4 authentication</text>
+    <text x="460" y="220">● Routes to service handlers</text>
+    <text x="460" y="240">● NATS client connection</text>
+  </g>
+
+  <!-- Connector: JSON over NATS -->
+  <line x1="450" y1="258" x2="450" y2="286" stroke="#DC2626" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.7"/>
+  <rect x="380" y="260" width="140" height="20" rx="10" fill="#0a0a0a" stroke="#DC2626" stroke-width="0.8" stroke-opacity="0.5"/>
+  <text x="450" y="274" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#F87171" text-anchor="middle">JSON over NATS</text>
+
+  <!-- ═══ NATS BROKER ═══════════════════════════════════ -->
+  <rect x="16" y="286" width="868" height="124" rx="8" fill="#1a0a0a" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.65"/>
+  <rect x="16" y="286" width="868" height="28" rx="8" fill="url(#nats-grad)" opacity="0.9"/>
+  <text x="32" y="305" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#f5f5f5" font-weight="700" letter-spacing="1.5">NATS MESSAGE BROKER</text>
+  <text x="876" y="305" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#fee2e2" text-anchor="end" opacity="0.9">request/reply · queue groups · fan-out</text>
+
+  <!-- Two-column topics/queue groups -->
+  <g font-family="'SF Mono', ui-monospace, monospace" font-size="10">
+    <text x="40" y="332" fill="#F87171" font-weight="600">Topics</text>
+    <text x="40" y="350" fill="#f5f5f5">ec2.RunInstances</text>
+    <text x="40" y="368" fill="#f5f5f5">ec2.DescribeInstances</text>
+    <text x="40" y="386" fill="#f5f5f5">ec2.cmd.{instanceID}</text>
+    <text x="40" y="404" fill="#f5f5f5">ebs.mount  ·  ebs.unmount</text>
+
+    <text x="460" y="332" fill="#F87171" font-weight="600">Delivery mode</text>
+    <text x="460" y="350" fill="#a1a1aa">spinifex-workers <tspan fill="#71717a">(load-balanced)</tspan></text>
+    <text x="460" y="368" fill="#a1a1aa">no queue group <tspan fill="#71717a">(fan-out to all nodes)</tspan></text>
+    <text x="460" y="386" fill="#a1a1aa">per-instance <tspan fill="#71717a">(start / stop / terminate)</tspan></text>
+    <text x="460" y="404" fill="#a1a1aa">Viperblock request/reply</text>
+  </g>
+
+  <!-- Fan-out connectors to 3 daemons -->
+  <g stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6" fill="none">
+    <line x1="450" y1="410" x2="450" y2="430"/>
+    <line x1="450" y1="430" x2="155" y2="452"/>
+    <line x1="450" y1="430" x2="450" y2="452"/>
+    <line x1="450" y1="430" x2="745" y2="452"/>
+  </g>
+
+  <!-- ═══ DAEMON ROW ════════════════════════════════════ -->
+  <!-- Daemon 1 -->
+  <g transform="translate(32, 452)">
+    <rect width="246" height="124" rx="8" fill="url(#daemon-grad)" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
+    <text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" font-weight="700">Spinifex Daemon</text>
+    <text x="226" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="end">Node 1</text>
+    <line x1="16" y1="32" x2="230" y2="32" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
+    <g font-family="'Segoe UI', sans-serif" font-size="10.5" fill="#f5f5f5">
+      <text x="20" y="52">● NATS subscriber</text>
+      <text x="20" y="70">● Resource manager</text>
+      <text x="20" y="88">● QEMU/KVM launcher</text>
+      <text x="20" y="106">● QMP monitoring</text>
+    </g>
+    <text x="20" y="119" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#71717a">daemon.go</text>
+  </g>
+
+  <!-- Daemon 2 -->
+  <g transform="translate(327, 452)">
+    <rect width="246" height="124" rx="8" fill="url(#daemon-grad)" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
+    <text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" font-weight="700">Spinifex Daemon</text>
+    <text x="226" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="end">Node 2</text>
+    <line x1="16" y1="32" x2="230" y2="32" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
+    <g font-family="'Segoe UI', sans-serif" font-size="10.5" fill="#f5f5f5">
+      <text x="20" y="52">● NATS subscriber</text>
+      <text x="20" y="70">● Resource manager</text>
+      <text x="20" y="88">● QEMU/KVM launcher</text>
+      <text x="20" y="106">● QMP monitoring</text>
+    </g>
+    <text x="20" y="119" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#71717a">daemon.go</text>
+  </g>
+
+  <!-- Daemon N -->
+  <g transform="translate(622, 452)">
+    <rect width="246" height="124" rx="8" fill="url(#daemon-grad)" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.7"/>
+    <text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" font-weight="700">Spinifex Daemon</text>
+    <text x="226" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="end">Node N</text>
+    <line x1="16" y1="32" x2="230" y2="32" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
+    <g font-family="'Segoe UI', sans-serif" font-size="10.5" fill="#f5f5f5">
+      <text x="20" y="52">● NATS subscriber</text>
+      <text x="20" y="70">● Resource manager</text>
+      <text x="20" y="88">● QEMU/KVM launcher</text>
+      <text x="20" y="106">● QMP monitoring</text>
+    </g>
+    <text x="20" y="119" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#71717a">daemon.go</text>
+  </g>
+
+  <!-- Daemon → VM connectors -->
+  <g stroke="#71717a" stroke-width="0.8" stroke-dasharray="2,3" opacity="0.55" fill="none">
+    <line x1="155" y1="576" x2="155" y2="610"/>
+    <line x1="450" y1="576" x2="450" y2="610"/>
+    <line x1="745" y1="576" x2="745" y2="610"/>
+  </g>
+
+  <!-- ═══ VM BAR + ROW ═══════════════════════════════════ -->
+  <rect x="16" y="610" width="868" height="22" rx="5" fill="url(#bar-grad)"/>
+  <text x="450" y="625" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle" letter-spacing="1.5">WORKLOADS · QEMU/KVM virtual machines per node</text>
+
+  <!-- VM boxes -->
+  <g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
+    <rect x="32" y="642" width="246" height="60" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
+    <text x="155" y="665" fill="#e4e4e7" text-anchor="middle" font-weight="600">QEMU / KVM VMs</text>
+    <text x="155" y="683" fill="#a1a1aa" text-anchor="middle" font-size="9.5">virtio-blk · virtio-net · cloud-init</text>
+    <text x="155" y="695" fill="#71717a" text-anchor="middle" font-size="9">Node 1 workloads</text>
+
+    <rect x="327" y="642" width="246" height="60" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
+    <text x="450" y="665" fill="#e4e4e7" text-anchor="middle" font-weight="600">QEMU / KVM VMs</text>
+    <text x="450" y="683" fill="#a1a1aa" text-anchor="middle" font-size="9.5">virtio-blk · virtio-net · cloud-init</text>
+    <text x="450" y="695" fill="#71717a" text-anchor="middle" font-size="9">Node 2 workloads</text>
+
+    <rect x="622" y="642" width="246" height="60" rx="7" fill="#27272a" fill-opacity="0.7" stroke="#71717a" stroke-width="1" stroke-opacity="0.6"/>
+    <text x="745" y="665" fill="#e4e4e7" text-anchor="middle" font-weight="600">QEMU / KVM VMs</text>
+    <text x="745" y="683" fill="#a1a1aa" text-anchor="middle" font-size="9.5">virtio-blk · virtio-net · cloud-init</text>
+    <text x="745" y="695" fill="#71717a" text-anchor="middle" font-size="9">Node N workloads</text>
+  </g>
+</svg>

.github/assets/diagrams/tf-bastion.svg

@@ -0,0 +1,46 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="180" viewBox="0 0 900 180">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+  </defs>
+  <rect width="900" height="180" fill="url(#bg)" rx="10"/>
+
+  <text x="24" y="28" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">BASTION · only entry point into a private subnet</text>
+
+  <!-- WAN -->
+  <g transform="translate(40, 56)">
+    <rect width="200" height="60" rx="8" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
+    <text x="100" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">WAN</text>
+    <text x="100" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">operator / laptop</text>
+  </g>
+
+  <!-- Arrow: SSH -->
+  <line x1="240" y1="86" x2="320" y2="86" stroke="#58a6ff" stroke-width="1.4" stroke-dasharray="4,3" opacity="0.75"/>
+  <polygon points="316,82 328,86 316,90" fill="#58a6ff" opacity="0.8"/>
+  <rect x="244" y="62" width="80" height="18" rx="9" fill="#0a0a0a" stroke="#58a6ff" stroke-width="0.6" stroke-opacity="0.55"/>
+  <text x="284" y="75" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#8ab4f8" text-anchor="middle">SSH</text>
+
+  <!-- Bastion -->
+  <g transform="translate(338, 56)">
+    <rect width="220" height="60" rx="8" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.3" stroke-opacity="0.6"/>
+    <text x="110" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">Bastion</text>
+    <text x="110" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#7ee787" text-anchor="middle">public subnet</text>
+  </g>
+
+  <!-- Arrow: SSH (private IP) -->
+  <line x1="558" y1="86" x2="638" y2="86" stroke="#a855f7" stroke-width="1.4" stroke-dasharray="4,3" opacity="0.75"/>
+  <polygon points="634,82 646,86 634,90" fill="#a855f7" opacity="0.8"/>
+  <rect x="552" y="62" width="140" height="18" rx="9" fill="#0a0a0a" stroke="#a855f7" stroke-width="0.6" stroke-opacity="0.55"/>
+  <text x="622" y="75" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#c084fc" text-anchor="middle">SSH (private IP)</text>
+
+  <!-- App server -->
+  <g transform="translate(656, 56)">
+    <rect width="220" height="60" rx="8" fill="#a855f71a" stroke="#a855f7" stroke-width="1.3" stroke-opacity="0.6"/>
+    <text x="110" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#c084fc" text-anchor="middle" font-weight="700">App Server</text>
+    <text x="110" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#d8b4fe" text-anchor="middle">private subnet · no internet</text>
+  </g>
+
+  <text x="450" y="156" font-family="'Segoe UI', sans-serif" font-size="10" fill="#71717a" text-anchor="middle" font-style="italic">App server has no route to the internet — all access hops through the bastion.</text>
+</svg>

.github/assets/diagrams/tf-nginx-alb.svg

@@ -0,0 +1,98 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="480" viewBox="0 0 900 480">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+    <linearGradient id="inet-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#3fb95022"/>
+      <stop offset="100%" style="stop-color:#3fb95008"/>
+    </linearGradient>
+    <linearGradient id="pub-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#3fb9501f"/>
+      <stop offset="100%" style="stop-color:#3fb95008"/>
+    </linearGradient>
+    <linearGradient id="priv-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#a855f71f"/>
+      <stop offset="100%" style="stop-color:#a855f708"/>
+    </linearGradient>
+  </defs>
+  <rect width="900" height="480" fill="url(#bg)" rx="12"/>
+
+  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">NGINX + ALB · VPC WITH PUBLIC AND PRIVATE SUBNETS</text>
+  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">10.20.0.0/16 · multi-AZ</text>
+
+  <!-- Internet -->
+  <rect x="350" y="54" width="200" height="40" rx="8" fill="url(#inet-grad)" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.55"/>
+  <text x="450" y="78" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">Internet</text>
+
+  <!-- connector to IGW -->
+  <line x1="450" y1="94" x2="450" y2="116" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+  <polygon points="446,112 454,112 450,120" fill="#71717a" opacity="0.7"/>
+
+  <!-- IGW -->
+  <rect x="380" y="122" width="140" height="44" rx="7" fill="#DC26261a" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.65"/>
+  <text x="450" y="142" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">IGW</text>
+  <text x="450" y="158" font-family="'Segoe UI', sans-serif" font-size="10" fill="#FCA5A5" text-anchor="middle">nginx-alb-igw</text>
+
+  <!-- Fan-out to two public subnets -->
+  <line x1="450" y1="166" x2="450" y2="184" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
+  <line x1="230" y1="184" x2="670" y2="184" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
+  <line x1="230" y1="184" x2="230" y2="202" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
+  <line x1="670" y1="184" x2="670" y2="202" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
+
+  <!-- public_a -->
+  <g transform="translate(108, 202)">
+    <rect width="244" height="104" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
+    <text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">public_a</text>
+    <text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">az-a</text>
+    <line x1="16" y1="32" x2="228" y2="32" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.5"/>
+
+    <rect x="20" y="42" width="200" height="22" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="120" y="57" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">ALB ENI</text>
+
+    <rect x="20" y="70" width="200" height="22" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="120" y="85" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">NAT GW</text>
+  </g>
+
+  <!-- public_b -->
+  <g transform="translate(548, 202)">
+    <rect width="244" height="104" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
+    <text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">public_b</text>
+    <text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#7ee787" text-anchor="end">az-b</text>
+    <line x1="16" y1="32" x2="228" y2="32" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.5"/>
+
+    <rect x="20" y="56" width="200" height="22" rx="4" fill="#3fb9501a" stroke="#3fb950" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="120" y="71" font-family="'SF Mono', ui-monospace, monospace" font-size="10.5" fill="#7ee787" text-anchor="middle" font-weight="600">ALB ENI</text>
+  </g>
+
+  <!-- SNAT arrow from public_a NAT → private row -->
+  <line x1="230" y1="306" x2="230" y2="340" stroke="#F87171" stroke-width="1.2" stroke-dasharray="3,3" opacity="0.65"/>
+  <polygon points="226,336 234,336 230,344" fill="#F87171" opacity="0.8"/>
+  <rect x="256" y="314" width="240" height="20" rx="10" fill="#0a0a0a" stroke="#F87171" stroke-width="0.6" stroke-opacity="0.55"/>
+  <text x="376" y="328" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#FCA5A5" text-anchor="middle">SNAT for private subnets (egress)</text>
+
+  <!-- private_a -->
+  <g transform="translate(108, 350)">
+    <rect width="244" height="104" rx="8" fill="url(#priv-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.55"/>
+    <text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#c084fc" font-weight="700">private_a</text>
+    <text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#d8b4fe" text-anchor="end">az-a</text>
+    <line x1="16" y1="32" x2="228" y2="32" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.5"/>
+
+    <rect x="20" y="44" width="200" height="48" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="120" y="64" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#d8b4fe" text-anchor="middle" font-weight="700">nginx_1</text>
+    <text x="120" y="80" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">Debian 12 · cloud-init</text>
+  </g>
+
+  <!-- private_b -->
+  <g transform="translate(548, 350)">
+    <rect width="244" height="104" rx="8" fill="url(#priv-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.55"/>
+    <text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#c084fc" font-weight="700">private_b</text>
+    <text x="224" y="24" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#d8b4fe" text-anchor="end">az-b</text>
+    <line x1="16" y1="32" x2="228" y2="32" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.5"/>
+
+    <rect x="20" y="44" width="200" height="48" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.5" stroke-width="0.6"/>
+    <text x="120" y="64" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#d8b4fe" text-anchor="middle" font-weight="700">nginx_2</text>
+    <text x="120" y="80" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">Debian 12 · cloud-init</text>
+  </g>
+</svg>

.github/assets/diagrams/tf-s3-webapp.svg

@@ -0,0 +1,44 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="180" viewBox="0 0 900 180">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+  </defs>
+  <rect width="900" height="180" fill="url(#bg)" rx="10"/>
+
+  <text x="24" y="28" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">S3 WEBAPP · Flask on EC2 with Predastore backend</text>
+
+  <!-- Browser -->
+  <g transform="translate(40, 56)">
+    <rect width="200" height="60" rx="8" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
+    <text x="100" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">Browser</text>
+    <text x="100" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">end user</text>
+  </g>
+
+  <!-- Arrow: HTTP -->
+  <line x1="240" y1="86" x2="320" y2="86" stroke="#58a6ff" stroke-width="1.4" stroke-dasharray="4,3" opacity="0.75"/>
+  <polygon points="316,82 328,86 316,90" fill="#58a6ff" opacity="0.8"/>
+  <rect x="244" y="62" width="80" height="18" rx="9" fill="#0a0a0a" stroke="#58a6ff" stroke-width="0.6" stroke-opacity="0.55"/>
+  <text x="284" y="75" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#8ab4f8" text-anchor="middle">HTTP</text>
+
+  <!-- EC2 Instance -->
+  <g transform="translate(338, 56)">
+    <rect width="240" height="60" rx="8" fill="#DC26261a" stroke="#DC2626" stroke-width="1.3" stroke-opacity="0.65"/>
+    <text x="120" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" text-anchor="middle" font-weight="700">EC2 Instance</text>
+    <text x="120" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#FCA5A5" text-anchor="middle">Flask webapp · port 80</text>
+  </g>
+
+  <!-- Arrow: S3 API (boto3) -->
+  <line x1="578" y1="86" x2="658" y2="86" stroke="#a855f7" stroke-width="1.4" stroke-dasharray="4,3" opacity="0.75"/>
+  <polygon points="654,82 666,86 654,90" fill="#a855f7" opacity="0.8"/>
+  <rect x="570" y="62" width="144" height="18" rx="9" fill="#0a0a0a" stroke="#a855f7" stroke-width="0.6" stroke-opacity="0.55"/>
+  <text x="642" y="75" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#c084fc" text-anchor="middle">S3 API (boto3)</text>
+
+  <!-- Predastore -->
+  <g transform="translate(676, 56)">
+    <rect width="200" height="60" rx="8" fill="#a855f71a" stroke="#a855f7" stroke-width="1.3" stroke-opacity="0.6"/>
+    <text x="100" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#c084fc" text-anchor="middle" font-weight="700">Predastore</text>
+    <text x="100" y="46" font-family="'Segoe UI', sans-serif" font-size="10" fill="#d8b4fe" text-anchor="middle">S3 store · port 8443</text>
+  </g>
+</svg>

.github/assets/diagrams/vpc-config-layers.svg

@@ -0,0 +1,78 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="330" viewBox="0 0 900 330">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+    <linearGradient id="cluster-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#3fb95022"/>
+      <stop offset="100%" style="stop-color:#3fb95008"/>
+    </linearGradient>
+    <linearGradient id="pool-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#DC262622"/>
+      <stop offset="100%" style="stop-color:#7F1D1D08"/>
+    </linearGradient>
+    <linearGradient id="node-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#58a6ff22"/>
+      <stop offset="100%" style="stop-color:#58a6ff08"/>
+    </linearGradient>
+  </defs>
+  <rect width="900" height="330" fill="url(#bg)" rx="12"/>
+
+  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">CONFIGURATION LEVELS · spinifex.toml</text>
+  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">three scopes: cluster · pool · node</text>
+
+  <!-- File badge -->
+  <rect x="24" y="50" width="140" height="24" rx="6" fill="#161b22" stroke="#52525b" stroke-width="0.6"/>
+  <text x="94" y="67" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#e4e4e7" text-anchor="middle">spinifex.toml</text>
+
+  <!-- spine -->
+  <line x1="52" y1="74" x2="52" y2="298" stroke="#52525b" stroke-width="1" opacity="0.6"/>
+
+  <!-- ═══ Level 1: [network] cluster-wide ═══════════════════ -->
+  <line x1="52" y1="108" x2="92" y2="108" stroke="#52525b" stroke-width="1" opacity="0.6"/>
+  <rect x="92" y="86" width="764" height="56" rx="8" fill="url(#cluster-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
+  <text x="108" y="108" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">[network]</text>
+  <text x="220" y="108" font-family="'Segoe UI', sans-serif" font-size="11" fill="#f5f5f5">Cluster-wide · which mode?</text>
+  <g font-family="'SF Mono', ui-monospace, monospace" font-size="10">
+    <text x="108" y="128" fill="#a1a1aa">└─</text>
+    <text x="128" y="128" fill="#7ee787">external_mode</text>
+    <text x="224" y="128" fill="#a1a1aa">=</text>
+    <text x="238" y="128" fill="#f5f5f5">"pool"</text>
+  </g>
+
+  <!-- ═══ Level 2: [[network.external_pools]] ═══════════════ -->
+  <line x1="52" y1="176" x2="92" y2="176" stroke="#52525b" stroke-width="1" opacity="0.6"/>
+  <rect x="92" y="154" width="764" height="78" rx="8" fill="url(#pool-grad)" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.65"/>
+  <text x="108" y="176" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" font-weight="700">[[network.external_pools]]</text>
+  <text x="328" y="176" font-family="'Segoe UI', sans-serif" font-size="11" fill="#f5f5f5">IP ranges · what IPs to use?</text>
+  <g font-family="'SF Mono', ui-monospace, monospace" font-size="10">
+    <text x="108" y="196" fill="#a1a1aa">├─</text>
+    <text x="128" y="196" fill="#FCA5A5">name</text>
+    <text x="158" y="196" fill="#71717a">·</text>
+    <text x="166" y="196" fill="#FCA5A5">range_start</text>
+    <text x="246" y="196" fill="#71717a">·</text>
+    <text x="254" y="196" fill="#FCA5A5">range_end</text>
+
+    <text x="108" y="212" fill="#a1a1aa">├─</text>
+    <text x="128" y="212" fill="#FCA5A5">gateway</text>
+    <text x="182" y="212" fill="#71717a">·</text>
+    <text x="190" y="212" fill="#FCA5A5">prefix_len</text>
+
+    <text x="108" y="228" fill="#a1a1aa">└─</text>
+    <text x="128" y="228" fill="#FCA5A5">region</text>
+    <text x="174" y="228" fill="#71717a">·</text>
+    <text x="182" y="228" fill="#FCA5A5">az</text>
+    <text x="202" y="228" fill="#71717a">(optional)</text>
+  </g>
+
+  <!-- ═══ Level 3: [nodes.NAME.vpcd] ════════════════════════ -->
+  <line x1="52" y1="268" x2="92" y2="268" stroke="#52525b" stroke-width="1" opacity="0.6"/>
+  <rect x="92" y="246" width="764" height="56" rx="8" fill="url(#node-grad)" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.6"/>
+  <text x="108" y="268" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#58a6ff" font-weight="700">[nodes.NAME.vpcd]</text>
+  <text x="266" y="268" font-family="'Segoe UI', sans-serif" font-size="11" fill="#f5f5f5">Per-node · which NIC?</text>
+  <g font-family="'SF Mono', ui-monospace, monospace" font-size="10">
+    <text x="108" y="288" fill="#a1a1aa">└─</text>
+    <text x="128" y="288" fill="#8ab4f8">external_interface</text>
+  </g>
+</svg>

.github/assets/diagrams/vpc-datapath-localnet.svg

@@ -0,0 +1,63 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="110" viewBox="0 0 900 110">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+  </defs>
+  <rect width="900" height="110" fill="url(#bg)" rx="10"/>
+
+  <text x="24" y="22" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" letter-spacing="2" font-weight="600">DATA PATH · VM to WAN via OVN localnet</text>
+
+  <!-- VM TAP -->
+  <g transform="translate(24, 40)">
+    <rect width="118" height="48" rx="7" fill="#a855f71a" stroke="#a855f7" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="59" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#c084fc" text-anchor="middle" font-weight="700">VM TAP</text>
+    <text x="59" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">guest interface</text>
+  </g>
+  <polygon points="148,60 160,64 148,68" fill="#71717a" opacity="0.7"/>
+  <line x1="142" y1="64" x2="148" y2="64" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+
+  <!-- br-int -->
+  <g transform="translate(166, 40)">
+    <rect width="102" height="48" rx="7" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="51" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#58a6ff" text-anchor="middle" font-weight="700">br-int</text>
+    <text x="51" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">OVS bridge</text>
+  </g>
+  <polygon points="274,60 286,64 274,68" fill="#71717a" opacity="0.7"/>
+  <line x1="268" y1="64" x2="274" y2="64" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+
+  <!-- OVN logical pipeline -->
+  <g transform="translate(292, 40)">
+    <rect width="192" height="48" rx="7" fill="#DC26261a" stroke="#DC2626" stroke-width="1.1" stroke-opacity="0.65"/>
+    <text x="96" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" text-anchor="middle" font-weight="700">OVN logical pipeline</text>
+    <text x="96" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#FCA5A5" text-anchor="middle">switch → router → SNAT/DNAT</text>
+  </g>
+  <polygon points="490,60 502,64 490,68" fill="#71717a" opacity="0.7"/>
+  <line x1="484" y1="64" x2="490" y2="64" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+
+  <!-- localnet -->
+  <g transform="translate(508, 40)">
+    <rect width="120" height="48" rx="7" fill="#DC26260d" stroke="#DC2626" stroke-width="1" stroke-opacity="0.5" stroke-dasharray="4,3"/>
+    <text x="60" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" text-anchor="middle" font-weight="700">localnet</text>
+    <text x="60" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#FCA5A5" text-anchor="middle">OVN port type</text>
+  </g>
+  <polygon points="634,60 646,64 634,68" fill="#71717a" opacity="0.7"/>
+  <line x1="628" y1="64" x2="634" y2="64" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+
+  <!-- br-wan -->
+  <g transform="translate(652, 40)">
+    <rect width="110" height="48" rx="7" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="55" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#58a6ff" text-anchor="middle" font-weight="700">br-wan</text>
+    <text x="55" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">OVS bridge</text>
+  </g>
+  <polygon points="768,60 780,64 768,68" fill="#71717a" opacity="0.7"/>
+  <line x1="762" y1="64" x2="768" y2="64" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+
+  <!-- WAN -->
+  <g transform="translate(786, 40)">
+    <rect width="90" height="48" rx="7" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="45" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" text-anchor="middle" font-weight="700">WAN</text>
+    <text x="45" y="38" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">uplink</text>
+  </g>
+</svg>

.github/assets/diagrams/vpc-dhcp-conversations.svg

@@ -0,0 +1,74 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="360" viewBox="0 0 900 360">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+    <linearGradient id="host-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#58a6ff22"/>
+      <stop offset="100%" style="stop-color:#58a6ff08"/>
+    </linearGradient>
+    <linearGradient id="vm-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#3fb95022"/>
+      <stop offset="100%" style="stop-color:#3fb95008"/>
+    </linearGradient>
+  </defs>
+  <rect width="900" height="360" fill="url(#bg)" rx="12"/>
+
+  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">TWO DHCP CONVERSATIONS · completely independent</text>
+  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">pool mode · static or dhcp source</text>
+
+  <!-- ═══ HOST ↔ ROUTER DHCP ══════════════════════════════ -->
+  <rect x="24" y="54" width="852" height="138" rx="10" fill="url(#host-grad)" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.6"/>
+  <text x="44" y="78" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#58a6ff" font-weight="700" letter-spacing="0.8">HOST ↔ ROUTER DHCP</text>
+  <text x="876" y="78" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#8ab4f8" text-anchor="end">on br-ext · veth pair → br-wan</text>
+  <line x1="24" y1="88" x2="876" y2="88" stroke="#58a6ff" stroke-opacity="0.25" stroke-width="0.5"/>
+
+  <g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
+    <!-- step 1 -->
+    <circle cx="54" cy="108" r="9" fill="#58a6ff" fill-opacity="0.15" stroke="#58a6ff" stroke-width="1"/>
+    <text x="54" y="112" fill="#58a6ff" text-anchor="middle" font-weight="700" font-size="10">1</text>
+    <text x="74" y="112" fill="#f5f5f5">Spinifex asks router:</text>
+    <text x="228" y="112" fill="#a1a1aa">"give me a public IP for this VM"</text>
+
+    <!-- step 2 -->
+    <circle cx="54" cy="132" r="9" fill="#58a6ff" fill-opacity="0.15" stroke="#58a6ff" stroke-width="1"/>
+    <text x="54" y="136" fill="#58a6ff" text-anchor="middle" font-weight="700" font-size="10">2</text>
+    <text x="74" y="136" fill="#f5f5f5">Router responds:</text>
+    <text x="196" y="136" fill="#8ab4f8" font-weight="600">192.168.1.75</text>
+
+    <!-- step 3 -->
+    <circle cx="54" cy="156" r="9" fill="#58a6ff" fill-opacity="0.15" stroke="#58a6ff" stroke-width="1"/>
+    <text x="54" y="160" fill="#58a6ff" text-anchor="middle" font-weight="700" font-size="10">3</text>
+    <text x="74" y="160" fill="#f5f5f5">Spinifex stores IP, tells OVN:</text>
+    <text x="274" y="160" fill="#F87171" font-weight="600">dnat_and_snat(192.168.1.75 ↔ 172.31.0.4)</text>
+  </g>
+
+  <text x="44" y="181" font-family="'Segoe UI', sans-serif" font-size="10" fill="#71717a" font-style="italic">Only used in DHCP source mode. Static mode picks from range.</text>
+
+  <!-- separator -->
+  <line x1="24" y1="206" x2="876" y2="206" stroke="#27272a" stroke-width="0.8"/>
+  <text x="450" y="220" font-family="'SF Mono', ui-monospace, monospace" font-size="9" fill="#52525b" text-anchor="middle" letter-spacing="2">INDEPENDENT · no packets cross between</text>
+
+  <!-- ═══ VM ↔ OVN DHCP ═══════════════════════════════════ -->
+  <rect x="24" y="234" width="852" height="110" rx="10" fill="url(#vm-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
+  <text x="44" y="258" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700" letter-spacing="0.8">VM ↔ OVN DHCP</text>
+  <text x="876" y="258" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#7ee787" text-anchor="end">on tap → br-int → OVN logical switch</text>
+  <line x1="24" y1="268" x2="876" y2="268" stroke="#3fb950" stroke-opacity="0.25" stroke-width="0.5"/>
+
+  <g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
+    <circle cx="54" cy="288" r="9" fill="#3fb950" fill-opacity="0.15" stroke="#3fb950" stroke-width="1"/>
+    <text x="54" y="292" fill="#3fb950" text-anchor="middle" font-weight="700" font-size="10">1</text>
+    <text x="74" y="292" fill="#f5f5f5">cloud-init asks:</text>
+    <text x="190" y="292" fill="#a1a1aa">"what's my IP?"</text>
+
+    <circle cx="54" cy="312" r="9" fill="#3fb950" fill-opacity="0.15" stroke="#3fb950" stroke-width="1"/>
+    <text x="54" y="316" fill="#3fb950" text-anchor="middle" font-weight="700" font-size="10">2</text>
+    <text x="74" y="316" fill="#f5f5f5">OVN DHCP responds:</text>
+    <text x="226" y="316" fill="#a1a1aa">"you are</text>
+    <text x="284" y="316" fill="#7ee787" font-weight="600">172.31.0.4</text>
+    <text x="356" y="316" fill="#a1a1aa">"</text>
+  </g>
+
+  <text x="44" y="336" font-family="'Segoe UI', sans-serif" font-size="10" fill="#71717a" font-style="italic">VM never sees 192.168.1.x or the router.</text>
+</svg>

.github/assets/diagrams/vpc-direct-bridge.svg

@@ -0,0 +1,44 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="170" viewBox="0 0 900 170">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+  </defs>
+  <rect width="900" height="170" fill="url(#bg)" rx="10"/>
+
+  <text x="24" y="24" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" letter-spacing="2" font-weight="600">DIRECT BRIDGE MODE · dedicated WAN NIC</text>
+
+  <!-- Row 1: Management NIC (independent, no WAN bridge path) -->
+  <g transform="translate(24, 42)">
+    <rect width="852" height="44" rx="7" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.55"/>
+    <text x="20" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" font-weight="700">Management NIC</text>
+    <text x="20" y="36" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa">host IP · SSH · spinifex services · overlay (Geneve)</text>
+    <text x="840" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#7ee787" text-anchor="end" font-weight="600">independent path</text>
+  </g>
+
+  <!-- Row 2: WAN NIC → br-wan → localnet -->
+  <g transform="translate(24, 100)">
+    <rect width="160" height="44" rx="7" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="80" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#58a6ff" text-anchor="middle" font-weight="700">WAN NIC</text>
+    <text x="80" y="36" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">dedicated physical</text>
+  </g>
+  <polygon points="198,118 210,122 198,126" fill="#71717a" opacity="0.7"/>
+  <line x1="186" y1="122" x2="198" y2="122" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+
+  <g transform="translate(214, 100)">
+    <rect width="220" height="44" rx="7" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="110" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#58a6ff" text-anchor="middle" font-weight="700">br-wan</text>
+    <text x="110" y="36" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#a1a1aa" text-anchor="middle">OVS port (NIC added directly)</text>
+  </g>
+  <polygon points="448,118 460,122 448,126" fill="#71717a" opacity="0.7"/>
+  <line x1="434" y1="122" x2="448" y2="122" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+
+  <g transform="translate(464, 100)">
+    <rect width="180" height="44" rx="7" fill="#DC26261a" stroke="#DC2626" stroke-width="1.1" stroke-opacity="0.65"/>
+    <text x="90" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" text-anchor="middle" font-weight="700">OVN localnet</text>
+    <text x="90" y="36" font-family="'Segoe UI', sans-serif" font-size="9.5" fill="#FCA5A5" text-anchor="middle">external switch port</text>
+  </g>
+
+  <text x="670" y="130" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#F87171" font-weight="600">distributed NAT · any MAC · all frames</text>
+</svg>

.github/assets/diagrams/vpc-host-flow.svg

@@ -0,0 +1,91 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="440" viewBox="0 0 900 440">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+    <linearGradient id="host-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#27272a88"/>
+      <stop offset="100%" style="stop-color:#27272a22"/>
+    </linearGradient>
+    <linearGradient id="br-int-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#a855f71f"/>
+      <stop offset="100%" style="stop-color:#a855f708"/>
+    </linearGradient>
+    <linearGradient id="br-wan-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#58a6ff1f"/>
+      <stop offset="100%" style="stop-color:#58a6ff08"/>
+    </linearGradient>
+  </defs>
+  <rect width="900" height="440" fill="url(#bg)" rx="12"/>
+
+  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">NETWORK FLOW · bare-metal host</text>
+  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">br-int overlay · br-wan WAN uplink</text>
+
+  <!-- ═══ Bare-Metal Host container ═══════════════════════ -->
+  <rect x="40" y="60" width="820" height="284" rx="12" fill="url(#host-grad)" stroke="#71717a" stroke-width="1.2" stroke-opacity="0.7"/>
+  <text x="58" y="84" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#e4e4e7" font-weight="700" letter-spacing="1">BARE-METAL HOST</text>
+  <line x1="40" y1="96" x2="860" y2="96" stroke="#52525b" stroke-opacity="0.5" stroke-width="0.5"/>
+
+  <!-- br-int bridge (left) -->
+  <g transform="translate(80, 122)">
+    <rect width="260" height="176" rx="10" fill="url(#br-int-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.6"/>
+    <text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#c084fc" font-weight="700">br-int</text>
+    <text x="240" y="24" font-family="'Segoe UI', sans-serif" font-size="10" fill="#d8b4fe" text-anchor="end">(overlay)</text>
+    <line x1="20" y1="34" x2="240" y2="34" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.5"/>
+
+    <g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
+      <rect x="30" y="52" width="200" height="28" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.4" stroke-width="0.6"/>
+      <text x="130" y="70" fill="#d8b4fe" text-anchor="middle">tap-eni1</text>
+
+      <rect x="30" y="88" width="200" height="28" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.4" stroke-width="0.6"/>
+      <text x="130" y="106" fill="#d8b4fe" text-anchor="middle">tap-eni2</text>
+
+      <rect x="30" y="124" width="200" height="28" rx="5" fill="#a855f71a" stroke="#a855f7" stroke-opacity="0.4" stroke-width="0.6"/>
+      <text x="130" y="142" fill="#d8b4fe" text-anchor="middle">tap-eni3</text>
+    </g>
+  </g>
+
+  <!-- OVN NAT pipeline arrow in middle -->
+  <g transform="translate(352, 200)">
+    <line x1="0" y1="10" x2="148" y2="10" stroke="#DC2626" stroke-width="1.4" stroke-dasharray="4,3" opacity="0.75"/>
+    <polygon points="148,5 160,10 148,15" fill="#DC2626" opacity="0.8"/>
+    <text x="78" y="0" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" text-anchor="middle" font-weight="700">OVN</text>
+    <text x="78" y="28" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#FCA5A5" text-anchor="middle">NAT pipeline</text>
+  </g>
+
+  <!-- br-wan bridge (right) -->
+  <g transform="translate(520, 122)">
+    <rect width="260" height="176" rx="10" fill="url(#br-wan-grad)" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.6"/>
+    <text x="20" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#58a6ff" font-weight="700">br-wan</text>
+    <text x="240" y="24" font-family="'Segoe UI', sans-serif" font-size="10" fill="#8ab4f8" text-anchor="end">(WAN)</text>
+    <line x1="20" y1="34" x2="240" y2="34" stroke="#58a6ff" stroke-opacity="0.3" stroke-width="0.5"/>
+
+    <g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
+      <rect x="30" y="52" width="200" height="60" rx="5" fill="#58a6ff1a" stroke="#58a6ff" stroke-opacity="0.45" stroke-width="0.8"/>
+      <text x="130" y="74" fill="#8ab4f8" text-anchor="middle" font-weight="600">spx-ext-{nic}</text>
+      <text x="130" y="92" fill="#a1a1aa" text-anchor="middle" font-size="10">(macvlan)</text>
+    </g>
+  </g>
+
+  <!-- macvlan bridge mode label -->
+  <text x="650" y="314" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="middle" font-style="italic">macvlan bridge mode</text>
+
+  <!-- WAN NIC inside host, bottom -->
+  <line x1="650" y1="298" x2="650" y2="320" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+
+  <g transform="translate(500, 320)">
+    <rect width="300" height="20" rx="5" fill="#161b22" stroke="#52525b" stroke-width="0.6"/>
+    <text x="150" y="14" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle">WAN NIC · host IP unchanged</text>
+  </g>
+
+  <!-- connector out -->
+  <line x1="650" y1="344" x2="650" y2="380" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+  <polygon points="646,376 654,376 650,384" fill="#71717a" opacity="0.7"/>
+
+  <!-- Physical WAN -->
+  <g transform="translate(500, 388)">
+    <rect width="300" height="40" rx="7" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.55"/>
+    <text x="150" y="25" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" text-anchor="middle" font-weight="700">Physical WAN</text>
+  </g>
+</svg>

.github/assets/diagrams/vpc-macvlan.svg

@@ -0,0 +1,57 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="320" viewBox="0 0 900 320">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+  </defs>
+  <rect width="900" height="320" fill="url(#bg)" rx="12"/>
+
+  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">MACVLAN FALLBACK · single-NIC host</text>
+  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">WAN NIC == SSH NIC · host IP preserved</text>
+
+  <!-- WAN NIC (parent) -->
+  <g transform="translate(280, 58)">
+    <rect width="340" height="54" rx="8" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.3" stroke-opacity="0.65"/>
+    <text x="170" y="24" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#58a6ff" text-anchor="middle" font-weight="700">WAN NIC</text>
+    <text x="170" y="42" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">host IP unchanged · parent interface</text>
+  </g>
+
+  <!-- Split point -->
+  <line x1="450" y1="112" x2="450" y2="132" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+  <line x1="260" y1="132" x2="640" y2="132" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+  <line x1="260" y1="132" x2="260" y2="148" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+  <line x1="640" y1="132" x2="640" y2="148" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+
+  <!-- Child 1: host traffic -->
+  <g transform="translate(120, 148)">
+    <rect width="280" height="48" rx="7" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="140" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#3fb950" text-anchor="middle" font-weight="700">host traffic</text>
+    <text x="140" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">SSH · management · overlay</text>
+  </g>
+
+  <!-- Child 2: macvlan -->
+  <g transform="translate(500, 148)">
+    <rect width="280" height="48" rx="7" fill="#DC26261a" stroke="#DC2626" stroke-width="1.1" stroke-opacity="0.65"/>
+    <text x="140" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" text-anchor="middle" font-weight="700">spx-ext-{nic}</text>
+    <text x="140" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#FCA5A5" text-anchor="middle">macvlan · bridge mode · no IP</text>
+  </g>
+
+  <!-- br-wan -->
+  <line x1="640" y1="196" x2="640" y2="216" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+  <polygon points="636,212 644,212 640,220" fill="#71717a" opacity="0.7"/>
+
+  <g transform="translate(500, 220)">
+    <rect width="280" height="40" rx="7" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="140" y="25" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#58a6ff" text-anchor="middle" font-weight="700">br-wan</text>
+  </g>
+
+  <!-- localnet -->
+  <line x1="640" y1="260" x2="640" y2="276" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+  <polygon points="636,272 644,272 640,280" fill="#71717a" opacity="0.7"/>
+
+  <g transform="translate(500, 280)">
+    <rect width="280" height="30" rx="6" fill="#DC26260d" stroke="#DC2626" stroke-width="1" stroke-opacity="0.5" stroke-dasharray="4,3"/>
+    <text x="140" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" text-anchor="middle" font-weight="600">localnet → OVN ext switch</text>
+  </g>
+</svg>

.github/assets/diagrams/vpc-network-path.svg

@@ -0,0 +1,131 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="560" viewBox="0 0 900 560">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+    <linearGradient id="wan-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#3fb95022"/>
+      <stop offset="100%" style="stop-color:#3fb95008"/>
+    </linearGradient>
+    <linearGradient id="ovs-grad" x1="0%" y1="0%" x2="100%" y2="0%">
+      <stop offset="0%" style="stop-color:#58a6ff22"/>
+      <stop offset="100%" style="stop-color:#58a6ff0a"/>
+    </linearGradient>
+    <linearGradient id="router-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#DC262622"/>
+      <stop offset="100%" style="stop-color:#7F1D1D0a"/>
+    </linearGradient>
+    <linearGradient id="pub-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#3fb95022"/>
+      <stop offset="100%" style="stop-color:#3fb95008"/>
+    </linearGradient>
+    <linearGradient id="priv-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#a855f722"/>
+      <stop offset="100%" style="stop-color:#a855f708"/>
+    </linearGradient>
+  </defs>
+
+  <rect width="900" height="560" fill="url(#bg)" rx="12"/>
+
+  <!-- Title -->
+  <text x="24" y="30" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" letter-spacing="2.5" font-weight="600">NETWORK PATH</text>
+  <text x="876" y="30" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="end">OVN logical topology · bridge mapping · NAT</text>
+
+  <!-- WAN -->
+  <rect x="300" y="54" width="300" height="40" rx="8" fill="url(#wan-grad)" stroke="#3fb950" stroke-width="1" stroke-opacity="0.55"/>
+  <text x="450" y="78" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">Physical Network (WAN)</text>
+
+  <!-- connector -->
+  <line x1="450" y1="94" x2="450" y2="112" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+
+  <!-- br-wan bridge -->
+  <rect x="340" y="112" width="220" height="52" rx="8" fill="url(#ovs-grad)" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.65"/>
+  <text x="450" y="134" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#58a6ff" text-anchor="middle" font-weight="700">br-wan</text>
+  <text x="450" y="152" font-family="'Segoe UI', sans-serif" font-size="10" fill="#a1a1aa" text-anchor="middle">OVS bridge · physical uplink</text>
+
+  <!-- connector with bridge-mappings label -->
+  <line x1="450" y1="164" x2="450" y2="186" stroke="#71717a" stroke-width="1" opacity="0.5"/>
+  <rect x="316" y="170" width="268" height="18" rx="9" fill="#0a0a0a" stroke="#58a6ff" stroke-width="0.6" stroke-opacity="0.5"/>
+  <text x="450" y="183" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#8ab4f8" text-anchor="middle">ovn-bridge-mappings: external:br-wan</text>
+
+  <!-- VPC Logical Router (the big box) -->
+  <rect x="60" y="208" width="780" height="196" rx="10" fill="url(#router-grad)" stroke="#DC2626" stroke-width="1.4" stroke-opacity="0.75"/>
+  <text x="78" y="232" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#F87171" font-weight="700" letter-spacing="1">VPC LOGICAL ROUTER</text>
+  <line x1="60" y1="242" x2="840" y2="242" stroke="#DC2626" stroke-opacity="0.25" stroke-width="0.5"/>
+
+  <!-- Router ports row -->
+  <g font-family="'SF Mono', ui-monospace, monospace" font-size="11">
+    <rect x="82" y="254" width="120" height="42" rx="5" fill="#0d1117" stroke="#F87171" stroke-opacity="0.6" stroke-width="0.8"/>
+    <text x="142" y="270" fill="#F87171" text-anchor="middle" font-weight="600">Gateway port</text>
+    <text x="142" y="286" fill="#a1a1aa" text-anchor="middle" font-size="9.5">public IP</text>
+
+    <text x="215" y="282" fill="#71717a">─────</text>
+
+    <rect x="260" y="254" width="120" height="42" rx="5" fill="#0d1117" stroke="#F87171" stroke-opacity="0.6" stroke-width="0.8"/>
+    <text x="320" y="270" fill="#F87171" text-anchor="middle" font-weight="600">ext switch</text>
+    <text x="320" y="286" fill="#a1a1aa" text-anchor="middle" font-size="9.5">OVN logical</text>
+
+    <text x="393" y="282" fill="#71717a">─────</text>
+
+    <rect x="438" y="254" width="120" height="42" rx="5" fill="#0d1117" stroke="#F87171" stroke-opacity="0.6" stroke-width="0.8"/>
+    <text x="498" y="270" fill="#F87171" text-anchor="middle" font-weight="600">localnet port</text>
+    <text x="498" y="286" fill="#a1a1aa" text-anchor="middle" font-size="9.5">→ br-wan</text>
+  </g>
+
+  <!-- NAT rules + Routes, two columns on right -->
+  <g font-family="'SF Mono', ui-monospace, monospace" font-size="10.5">
+    <text x="598" y="260" fill="#F87171" font-weight="600">NAT rules</text>
+    <text x="598" y="276" fill="#f5f5f5">SNAT:</text>
+    <text x="644" y="276" fill="#a1a1aa">VPC CIDR → gw IP</text>
+    <text x="598" y="292" fill="#f5f5f5">dnat_and_snat:</text>
+    <text x="704" y="292" fill="#a1a1aa">pub ↔ priv</text>
+  </g>
+
+  <g font-family="'SF Mono', ui-monospace, monospace" font-size="10.5">
+    <text x="82" y="330" fill="#F87171" font-weight="600">Routes</text>
+    <text x="82" y="346" fill="#f5f5f5">0.0.0.0/0</text>
+    <text x="154" y="346" fill="#a1a1aa">→ WAN gateway</text>
+  </g>
+
+  <!-- Subnet branches inside router -->
+  <line x1="150" y1="362" x2="150" y2="386" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
+  <line x1="150" y1="386" x2="305" y2="386" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
+  <line x1="305" y1="386" x2="305" y2="404" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
+
+  <line x1="150" y1="386" x2="515" y2="386" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
+  <line x1="515" y1="386" x2="515" y2="404" stroke="#DC2626" stroke-width="1" stroke-dasharray="3,3" opacity="0.6"/>
+
+  <!-- Subnet A (public) -->
+  <g transform="translate(215, 404)">
+    <rect width="180" height="124" rx="8" fill="url(#pub-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.6"/>
+    <text x="90" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#3fb950" text-anchor="middle" font-weight="700">subnet-a</text>
+    <text x="90" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#7ee787" text-anchor="middle">(public)</text>
+    <line x1="16" y1="46" x2="164" y2="46" stroke="#3fb950" stroke-opacity="0.3" stroke-width="0.5"/>
+    <text x="90" y="64" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#f5f5f5" text-anchor="middle">10.0.1.0/24</text>
+    <g font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa">
+      <circle cx="40" cy="88" r="2.5" fill="#3fb950"/>
+      <text x="52" y="92">eni-1</text>
+      <circle cx="40" cy="108" r="2.5" fill="#3fb950"/>
+      <text x="52" y="112">eni-2</text>
+    </g>
+  </g>
+
+  <!-- Subnet B (private) -->
+  <g transform="translate(425, 404)">
+    <rect width="180" height="124" rx="8" fill="url(#priv-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.6"/>
+    <text x="90" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="13" fill="#c084fc" text-anchor="middle" font-weight="700">subnet-b</text>
+    <text x="90" y="38" font-family="'Segoe UI', sans-serif" font-size="10" fill="#d8b4fe" text-anchor="middle">(private)</text>
+    <line x1="16" y1="46" x2="164" y2="46" stroke="#a855f7" stroke-opacity="0.3" stroke-width="0.5"/>
+    <text x="90" y="64" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#f5f5f5" text-anchor="middle">10.0.2.0/24</text>
+    <g font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa">
+      <circle cx="40" cy="88" r="2.5" fill="#a855f7"/>
+      <text x="52" y="92">eni-3</text>
+      <circle cx="40" cy="108" r="2.5" fill="#a855f7"/>
+      <text x="52" y="112">eni-4</text>
+    </g>
+  </g>
+
+  <!-- Footnote -->
+  <text x="24" y="548" font-family="'Segoe UI', sans-serif" font-size="10" fill="#71717a">Cross-host traffic uses Geneve tunnels (UDP 6081) over the overlay NIC · ovn-controller programs OpenFlow rules on br-int</text>
+</svg>

.github/assets/diagrams/vpc-private-subnet-flow.svg

@@ -0,0 +1,56 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="110" viewBox="0 0 900 110">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+  </defs>
+  <rect width="900" height="110" fill="url(#bg)" rx="10"/>
+
+  <text x="24" y="24" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" letter-spacing="2" font-weight="600">PRIVATE SUBNET · no default route</text>
+
+  <!-- Instance -->
+  <g transform="translate(40, 42)">
+    <rect width="180" height="44" rx="7" fill="#a855f71a" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.6"/>
+    <text x="90" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#c084fc" text-anchor="middle" font-weight="700">Instance</text>
+    <text x="90" y="34" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#d8b4fe" text-anchor="middle">10.0.2.5</text>
+  </g>
+
+  <!-- arrow -->
+  <g stroke="#DC2626" stroke-width="1.2" opacity="0.6" fill="none">
+    <line x1="220" y1="64" x2="260" y2="64" stroke-dasharray="3,3"/>
+  </g>
+  <polygon points="260,60 268,64 260,68" fill="#DC2626" opacity="0.6"/>
+
+  <!-- Router -->
+  <g transform="translate(275, 42)">
+    <rect width="180" height="44" rx="7" fill="#DC26261a" stroke="#DC2626" stroke-width="1.2" stroke-opacity="0.65"/>
+    <text x="90" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" text-anchor="middle" font-weight="700">VPC Router</text>
+    <text x="90" y="34" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#FCA5A5" text-anchor="middle">lookup 0.0.0.0/0</text>
+  </g>
+
+  <!-- arrow -->
+  <g stroke="#DC2626" stroke-width="1.2" opacity="0.6" fill="none">
+    <line x1="455" y1="64" x2="495" y2="64" stroke-dasharray="3,3"/>
+  </g>
+  <polygon points="495,60 503,64 495,68" fill="#DC2626" opacity="0.6"/>
+
+  <!-- [no default route] -->
+  <g transform="translate(510, 42)">
+    <rect width="196" height="44" rx="7" fill="#0a0a0a" stroke="#71717a" stroke-width="1" stroke-opacity="0.7" stroke-dasharray="4,3"/>
+    <text x="98" y="20" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#a1a1aa" text-anchor="middle" font-weight="600">[no default route]</text>
+    <text x="98" y="34" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#71717a" text-anchor="middle">no IGW attached</text>
+  </g>
+
+  <!-- arrow to dropped -->
+  <g stroke="#71717a" stroke-width="1.2" opacity="0.6" fill="none">
+    <line x1="706" y1="64" x2="746" y2="64" stroke-dasharray="3,3"/>
+  </g>
+  <polygon points="746,60 754,64 746,68" fill="#71717a" opacity="0.6"/>
+
+  <!-- dropped -->
+  <g transform="translate(760, 42)">
+    <rect width="116" height="44" rx="7" fill="#27272a" stroke="#52525b" stroke-width="1" stroke-opacity="0.7"/>
+    <text x="58" y="27" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#a1a1aa" text-anchor="middle" font-weight="600">dropped</text>
+  </g>
+</svg>

.github/assets/diagrams/vpc-public-subnet-flow.svg

@@ -0,0 +1,74 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="170" viewBox="0 0 900 170">
+  <defs>
+    <linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+  </defs>
+  <rect width="900" height="170" fill="url(#bg)" rx="10"/>
+
+  <text x="24" y="24" font-family="'Segoe UI', system-ui, sans-serif" font-size="10" fill="#a1a1aa" letter-spacing="2" font-weight="600">PUBLIC SUBNET · 1:1 NAT managed by OVN</text>
+
+  <!-- OUTBOUND row -->
+  <text x="24" y="52" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" font-weight="700">Outbound</text>
+
+  <g transform="translate(112, 38)">
+    <rect width="180" height="40" rx="6" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="90" y="18" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#7ee787" text-anchor="middle" font-weight="700">Instance</text>
+    <text x="90" y="32" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle">10.0.1.5</text>
+  </g>
+
+  <g stroke="#3fb950" stroke-width="1.1" opacity="0.6" fill="none">
+    <line x1="292" y1="58" x2="352" y2="58" stroke-dasharray="3,3"/>
+  </g>
+  <polygon points="352,54 360,58 352,62" fill="#3fb950" opacity="0.7"/>
+
+  <g transform="translate(368, 38)">
+    <rect width="130" height="40" rx="6" fill="#DC26261a" stroke="#DC2626" stroke-width="1.1" stroke-opacity="0.65"/>
+    <text x="65" y="25" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" text-anchor="middle" font-weight="700">SNAT</text>
+  </g>
+
+  <g stroke="#3fb950" stroke-width="1.1" opacity="0.6" fill="none">
+    <line x1="498" y1="58" x2="558" y2="58" stroke-dasharray="3,3"/>
+  </g>
+  <polygon points="558,54 566,58 558,62" fill="#3fb950" opacity="0.7"/>
+
+  <g transform="translate(574, 38)">
+    <rect width="272" height="40" rx="6" fill="#3fb9501a" stroke="#3fb950" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="136" y="18" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#7ee787" text-anchor="middle" font-weight="700">exits as 203.0.113.10</text>
+    <text x="136" y="32" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle">source IP rewritten</text>
+  </g>
+
+  <!-- INBOUND row -->
+  <text x="24" y="110" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#58a6ff" font-weight="700">Inbound</text>
+
+  <g transform="translate(112, 96)">
+    <rect width="180" height="40" rx="6" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="90" y="18" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#8ab4f8" text-anchor="middle" font-weight="700">WAN client</text>
+    <text x="90" y="32" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle">→ 203.0.113.10</text>
+  </g>
+
+  <g stroke="#58a6ff" stroke-width="1.1" opacity="0.6" fill="none">
+    <line x1="292" y1="116" x2="352" y2="116" stroke-dasharray="3,3"/>
+  </g>
+  <polygon points="352,112 360,116 352,120" fill="#58a6ff" opacity="0.7"/>
+
+  <g transform="translate(368, 96)">
+    <rect width="130" height="40" rx="6" fill="#DC26261a" stroke="#DC2626" stroke-width="1.1" stroke-opacity="0.65"/>
+    <text x="65" y="25" font-family="'SF Mono', ui-monospace, monospace" font-size="12" fill="#F87171" text-anchor="middle" font-weight="700">DNAT</text>
+  </g>
+
+  <g stroke="#58a6ff" stroke-width="1.1" opacity="0.6" fill="none">
+    <line x1="498" y1="116" x2="558" y2="116" stroke-dasharray="3,3"/>
+  </g>
+  <polygon points="558,112 566,116 558,120" fill="#58a6ff" opacity="0.7"/>
+
+  <g transform="translate(574, 96)">
+    <rect width="272" height="40" rx="6" fill="#58a6ff1a" stroke="#58a6ff" stroke-width="1.1" stroke-opacity="0.6"/>
+    <text x="136" y="18" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#8ab4f8" text-anchor="middle" font-weight="700">arrives at 10.0.1.5</text>
+    <text x="136" y="32" font-family="'SF Mono', ui-monospace, monospace" font-size="10" fill="#a1a1aa" text-anchor="middle">dest IP rewritten</text>
+  </g>
+
+  <!-- Footnote -->
+  <text x="24" y="158" font-family="'Segoe UI', sans-serif" font-size="9" fill="#71717a">The instance OS only sees its private IP (10.0.1.5) — the public address is managed entirely by OVN's dnat_and_snat rule.</text>
+</svg>

.github/assets/platform.svg

@@ -0,0 +1,144 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="900" height="560" viewBox="0 0 900 560">
+  <defs>
+    <linearGradient id="pf-bg" x1="0%" y1="0%" x2="100%" y2="100%">
+      <stop offset="0%" style="stop-color:#0a0a0a"/>
+      <stop offset="100%" style="stop-color:#0d1117"/>
+    </linearGradient>
+    <linearGradient id="hw-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#52525b;stop-opacity:0.18"/>
+      <stop offset="100%" style="stop-color:#52525b;stop-opacity:0.06"/>
+    </linearGradient>
+    <linearGradient id="os-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#a855f7;stop-opacity:0.18"/>
+      <stop offset="100%" style="stop-color:#a855f7;stop-opacity:0.05"/>
+    </linearGradient>
+    <linearGradient id="spx-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#DC2626;stop-opacity:0.28"/>
+      <stop offset="100%" style="stop-color:#7F1D1D;stop-opacity:0.1"/>
+    </linearGradient>
+    <linearGradient id="app-grad" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#3fb950;stop-opacity:0.18"/>
+      <stop offset="100%" style="stop-color:#3fb950;stop-opacity:0.05"/>
+    </linearGradient>
+  </defs>
+
+  <rect width="900" height="560" fill="url(#pf-bg)" rx="12"/>
+
+  <!-- Subtle mesh dots background -->
+  <g opacity="0.12" fill="#7F1D1D">
+    <circle cx="40" cy="40" r="1"/><circle cx="120" cy="30" r="1"/><circle cx="220" cy="50" r="1"/>
+    <circle cx="320" cy="25" r="1"/><circle cx="440" cy="45" r="1"/><circle cx="560" cy="30" r="1"/>
+    <circle cx="680" cy="50" r="1"/><circle cx="780" cy="30" r="1"/><circle cx="860" cy="45" r="1"/>
+    <circle cx="60" cy="530" r="1"/><circle cx="160" cy="520" r="1"/><circle cx="260" cy="540" r="1"/>
+    <circle cx="380" cy="525" r="1"/><circle cx="500" cy="540" r="1"/><circle cx="620" cy="520" r="1"/>
+    <circle cx="740" cy="540" r="1"/><circle cx="840" cy="525" r="1"/>
+  </g>
+
+  <!-- Title -->
+  <text x="450" y="32" font-family="'Segoe UI', system-ui, sans-serif" font-size="11" fill="#a1a1aa" text-anchor="middle" letter-spacing="2.8">THE MULGA PLATFORM</text>
+  <text x="450" y="58" font-family="'Segoe UI', system-ui, sans-serif" font-size="20" font-weight="600" fill="#f5f5f5" text-anchor="middle">AWS-style workflows, on your own stack</text>
+
+  <!-- Layer 4: Applications (top) -->
+  <g transform="translate(60, 95)">
+    <rect width="780" height="88" rx="10" fill="url(#app-grad)" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.55"/>
+    <text x="20" y="26" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#3fb950" letter-spacing="1.5" font-weight="600">LAYER 4 · APPLICATIONS &amp; TOOLING</text>
+    <text x="20" y="48" font-family="'Segoe UI', sans-serif" font-size="15" fill="#f5f5f5" font-weight="500">Your workloads — unmodified AWS tooling</text>
+
+    <g transform="translate(20, 60)" font-family="'Segoe UI', sans-serif" font-size="10.5">
+      <rect x="0" y="0" width="70" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
+      <text x="35" y="14" fill="#7ee787" text-anchor="middle">aws-cli</text>
+
+      <rect x="78" y="0" width="78" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
+      <text x="117" y="14" fill="#7ee787" text-anchor="middle">Terraform</text>
+
+      <rect x="164" y="0" width="88" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
+      <text x="208" y="14" fill="#7ee787" text-anchor="middle">AWS SDKs</text>
+
+      <rect x="260" y="0" width="88" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
+      <text x="304" y="14" fill="#7ee787" text-anchor="middle">Kubernetes</text>
+
+      <rect x="356" y="0" width="70" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
+      <text x="391" y="14" fill="#7ee787" text-anchor="middle">Packer</text>
+
+      <rect x="434" y="0" width="98" height="20" rx="10" fill="#3fb950" fill-opacity="0.14" stroke="#3fb950" stroke-opacity="0.45" stroke-width="0.5"/>
+      <text x="483" y="14" fill="#7ee787" text-anchor="middle">Your Services</text>
+    </g>
+  </g>
+
+  <!-- Down arrow -->
+  <g transform="translate(450, 188)" opacity="0.45">
+    <line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
+    <polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
+  </g>
+
+  <!-- Layer 3: Spinifex (highlighted) -->
+  <g transform="translate(60, 215)">
+    <rect width="780" height="140" rx="10" fill="url(#spx-grad)" stroke="#DC2626" stroke-width="1.5" stroke-opacity="0.75"/>
+    <text x="20" y="26" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#F87171" letter-spacing="1.5" font-weight="600">LAYER 3 · SPINIFEX · AWS ORCHESTRATION</text>
+    <text x="20" y="50" font-family="'Segoe UI', sans-serif" font-size="15" fill="#f5f5f5" font-weight="500">AWS Gateway → NATS → Service Daemons · SigV4, IAM, VPC-aware</text>
+
+    <g transform="translate(20, 66)" font-family="'Segoe UI', sans-serif">
+      <!-- EC2 -->
+      <rect x="0" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
+      <text x="59" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">EC2</text>
+      <text x="59" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">QEMU/KVM VMs</text>
+      <text x="59" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">cloud-init · AMIs</text>
+
+      <!-- EBS -->
+      <rect x="126" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
+      <text x="185" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">EBS</text>
+      <text x="185" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">Viperblock</text>
+      <text x="185" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">WAL · NBD · snapshots</text>
+
+      <!-- S3 -->
+      <rect x="252" y="0" width="118" height="58" rx="6" fill="#DC2626" fill-opacity="0.1" stroke="#DC2626" stroke-opacity="0.6" stroke-width="1"/>
+      <text x="311" y="18" font-size="13" fill="#F87171" text-anchor="middle" font-weight="600">S3</text>
+      <text x="311" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">Predastore</text>
+      <text x="311" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">Reed–Solomon EC</text>
+
+      <!-- VPC -->
+      <rect x="378" y="0" width="118" height="58" rx="6" fill="#7F1D1D" fill-opacity="0.22" stroke="#F87171" stroke-opacity="0.55" stroke-width="1"/>
+      <text x="437" y="18" font-size="13" fill="#FCA5A5" text-anchor="middle" font-weight="600">VPC</text>
+      <text x="437" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">OVN / OVS</text>
+      <text x="437" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">subnets · SG · ALB</text>
+
+      <!-- IAM -->
+      <rect x="504" y="0" width="118" height="58" rx="6" fill="#7F1D1D" fill-opacity="0.22" stroke="#F87171" stroke-opacity="0.55" stroke-width="1"/>
+      <text x="563" y="18" font-size="13" fill="#FCA5A5" text-anchor="middle" font-weight="600">IAM</text>
+      <text x="563" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">SigV4 · policies</text>
+      <text x="563" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">users · roles · keys</text>
+
+      <!-- Control -->
+      <rect x="630" y="0" width="110" height="58" rx="6" fill="#27272a" fill-opacity="0.8" stroke="#71717a" stroke-opacity="0.6" stroke-width="1"/>
+      <text x="685" y="18" font-size="13" fill="#e4e4e7" text-anchor="middle" font-weight="600">Control</text>
+      <text x="685" y="34" font-size="9.5" fill="#f5f5f5" text-anchor="middle">NATS bus</text>
+      <text x="685" y="48" font-size="9.5" fill="#a1a1aa" text-anchor="middle">spx · daemons</text>
+    </g>
+  </g>
+
+  <!-- Down arrow -->
+  <g transform="translate(450, 360)" opacity="0.45">
+    <line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
+    <polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
+  </g>
+
+  <!-- Layer 2: OS -->
+  <g transform="translate(60, 387)">
+    <rect width="780" height="62" rx="10" fill="url(#os-grad)" stroke="#a855f7" stroke-width="1.2" stroke-opacity="0.5"/>
+    <text x="20" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#c084fc" letter-spacing="1.5" font-weight="600">LAYER 2 · OPERATING SYSTEM</text>
+    <text x="20" y="44" font-family="'Segoe UI', sans-serif" font-size="14" fill="#f5f5f5">Debian / Ubuntu LTS · systemd · KVM · Linux bridge · headless autoinstall</text>
+  </g>
+
+  <!-- Down arrow -->
+  <g transform="translate(450, 454)" opacity="0.45">
+    <line x1="0" y1="0" x2="0" y2="14" stroke="#a1a1aa" stroke-width="1"/>
+    <polygon points="-4,12 4,12 0,18" fill="#a1a1aa"/>
+  </g>
+
+  <!-- Layer 1: Hardware -->
+  <g transform="translate(60, 481)">
+    <rect width="780" height="62" rx="10" fill="url(#hw-grad)" stroke="#71717a" stroke-width="1.2" stroke-opacity="0.6"/>
+    <text x="20" y="22" font-family="'SF Mono', ui-monospace, monospace" font-size="11" fill="#a1a1aa" letter-spacing="1.5" font-weight="600">LAYER 1 · HARDWARE</text>
+    <text x="20" y="44" font-family="'Segoe UI', sans-serif" font-size="14" fill="#f5f5f5">Single node · multi-node cluster · edge appliance · air-gapped rack · commodity x86 / ARM</text>
+  </g>
+</svg>

.github/dependabot.yml

@@ -8,7 +8,7 @@ updates:
       day: monday
       timezone: Australia/Sydney
     cooldown:
-      default-days: 1
+      default-days: 3
     groups:
       go-dependencies:
         patterns:
@@ -22,7 +22,7 @@ updates:
       day: monday
       timezone: Australia/Sydney
     cooldown:
-      default-days: 1
+      default-days: 3
     ignore:
       - dependency-name: "*"
         update-types:
@@ -41,7 +41,7 @@ updates:
       day: monday
       timezone: Australia/Sydney
     cooldown:
-      default-days: 1
+      default-days: 3
     ignore:
       - dependency-name: "@types/node"
         update-types:

.github/scripts/capture-net-diag.sh

@@ -0,0 +1,88 @@
+#!/usr/bin/env bash
+# capture-net-diag.sh — write a snapshot of network / OVN / OVS / IPsec
+# state into $1 (a directory the caller already created on the local host).
+# Designed to run on a spinifex node after an E2E suite, before tofu destroy
+# pulls the rug out. Every command is independent — `|| true` so a missing
+# binary on one path doesn't abort the rest of the capture.
+#
+# Output layout (under $1/net-diag/):
+#   openvswitch-ipsec.status         systemctl status (output-only)
+#   openvswitch-ipsec.journal        last 200 journal lines
+#   ipsec.openvswitch_conf           ovs-vsctl get Open_vSwitch . other_config
+#   ipsec.xfrm_state                 ip xfrm state list
+#   ipsec.xfrm_policy                ip xfrm policy list
+#   ovn.sb_show                      ovn-sbctl show
+#   ovn.nb_show                      ovn-nbctl show
+#   ovs.show                         ovs-vsctl show
+#   ovs.flows_br_int                 ovs-ofctl dump-flows br-int
+#   ovs.tunnels                      ovs-appctl ofproto/list-tunnels br-int
+#   net.links                        ip -s link show
+#   net.routes                       ip route show table all
+#   net.addrs                        ip -br addr show
+#   systemd.failed_units             systemctl list-units --state=failed
+set +e
+OUT="${1:?usage: capture-net-diag.sh <output-dir>}"
+mkdir -p "$OUT"
+
+run() {
+    local label="$1"; shift
+    {
+        echo "# $*"
+        "$@" 2>&1
+    } > "$OUT/$label" || true
+}
+
+run openvswitch-ipsec.status            sudo systemctl status openvswitch-ipsec.service --no-pager --full --lines=50
+run openvswitch-ipsec.journal           sudo journalctl -u openvswitch-ipsec.service --no-pager --output=short-iso -n 200
+run ipsec.openvswitch_conf              sudo ovs-vsctl get Open_vSwitch . other_config
+run ipsec.xfrm_state                    sudo ip xfrm state list
+run ipsec.xfrm_policy                   sudo ip xfrm policy list
+
+run ovn.sb_show                         sudo ovn-sbctl show
+run ovn.nb_show                         sudo ovn-nbctl show
+run ovn.sb_chassis                      sudo ovn-sbctl list chassis
+
+run_per_lr() {
+    local label="$1"; shift
+    local subcmd="$1"; shift
+    {
+        echo "# ovn-nbctl $subcmd <each-logical-router>"
+        local lrs
+        lrs="$(sudo ovn-nbctl --bare --columns=name list Logical_Router 2>/dev/null)"
+        for lr in $lrs; do
+            echo
+            echo "=== $lr ==="
+            sudo ovn-nbctl "$subcmd" "$lr" 2>&1
+        done
+    } > "$OUT/$label" || true
+}
+
+run_per_lr ovn.lr_routes                lr-route-list
+run_per_lr ovn.lr_policies              lr-policy-list
+run_per_lr ovn.lr_nat                   lr-nat-list
+
+run ovs.show                            sudo ovs-vsctl show
+run ovs.flows_br_int                    sudo ovs-ofctl dump-flows br-int
+run ovs.tunnels                         sudo ovs-appctl ofproto/list-tunnels br-int
+
+run net.links                           ip -s link show
+run net.routes                          ip route show table all
+run net.addrs                           ip -br addr show
+run net.neigh                           ip neigh show
+run net.bridge_fdb                      bridge fdb show
+run net.conntrack                       sudo conntrack -L
+
+run systemd.failed_units                sudo systemctl list-units --state=failed --no-pager
+
+# Background pcap (started by start-net-capture.sh, if present).
+if [ -f /tmp/spinifex-e2e-tcpdump.pid ]; then
+    PID="$(cat /tmp/spinifex-e2e-tcpdump.pid 2>/dev/null)"
+    sudo kill -INT "$PID" 2>/dev/null || true
+    sleep 1
+    if [ -f /tmp/spinifex-e2e-wan.pcap ]; then
+        sudo cp /tmp/spinifex-e2e-wan.pcap "$OUT/wan.pcap" 2>/dev/null || true
+        sudo chmod 0644 "$OUT/wan.pcap" 2>/dev/null || true
+    fi
+fi
+
+exit 0

.github/scripts/ci-fmt.sh

@@ -0,0 +1,63 @@
+# shellcheck shell=bash
+# Source from a workflow `run:` block:
+#   source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+#
+# Provides colored banners + ::group::/::endgroup:: wrappers. ANSI escapes
+# render in the GitHub Actions live log; the same lines also land in any
+# artifact log (tee'd output) without breaking parsers because they only
+# affect terminal display.
+#
+# Functions kept short and unindented on purpose — wide nested output wraps
+# badly on narrow terminals.
+
+if [[ -n "${GITHUB_ACTIONS:-}" ]] || [[ -t 1 ]]; then
+  _R=$'\033[0m'
+  _B=$'\033[1m'    # bold
+  _D=$'\033[2m'    # dim
+  _RED=$'\033[31m'
+  _GRN=$'\033[32m'
+  _YEL=$'\033[33m'
+  _BLU=$'\033[34m'
+  _CYA=$'\033[36m'
+else
+  _R="" _B="" _D="" _RED="" _GRN="" _YEL="" _BLU="" _CYA=""
+fi
+
+# Single-line rule using box-drawing chars — narrow enough that GitHub's
+# timestamp prefix + this rule still fit on an 80-col terminal.
+# banner LABEL — bold cyan title with surrounding rule. Used at the start
+# of an inherently multi-line phase (test run, artifact pull) so the live
+# log has an obvious visual marker.
+banner() {
+  printf '\n%s%s━━ %s ━━%s\n' "$_B" "$_CYA" "$*" "$_R"
+}
+
+# Status indicators. Single char prefix only — keeps width on narrow logs.
+ok()   { printf '%s✓%s %s\n' "$_GRN" "$_R" "$*"; }
+bad()  { printf '%s✗%s %s\n' "$_RED" "$_R" "$*"; }
+warn() { printf '%s!%s %s\n' "$_YEL" "$_R" "$*"; }
+note() { printf '%s·%s %s\n' "$_D"   "$_R" "$*"; }
+
+# group_open LABEL / group_close — thin wrapper around GitHub's collapse
+# markers. No extra body printed; banner() is the visible inside-group
+# heading when callers want one.
+group_open()  { echo "::group::$*"; }
+group_close() { echo "::endgroup::"; }
+
+# quiet_run LABEL LOG_FILE -- CMD ARGS …
+# Run CMD redirected to LOG_FILE (append). Print one colored ok/✗ line.
+# On failure, dump the last 80 lines of LOG_FILE inside a collapsed group
+# so the failing tail lands in the UI without surfacing the whole apply.
+quiet_run() {
+  local label="$1" log="$2"; shift 2
+  if "$@" >> "$log" 2>&1; then
+    ok "$label"
+  else
+    local rc=$?
+    bad "$label (rc=$rc) — last 80 lines of $(basename "$log"):"
+    group_open "tail $(basename "$log")"
+    tail -n 80 "$log"
+    group_close
+    return "$rc"
+  fi
+}

.github/scripts/start-net-capture.sh

@@ -0,0 +1,47 @@
+#!/usr/bin/env bash
+# start-net-capture.sh — kick off background tcpdump on br-wan.
+# Pair: capture-net-diag.sh stops it and copies the pcap into net-diag/.
+#
+# Filter: anything on br-wan involving the EIP pool (192.168.0.0/23) plus
+# ARP for the same range. Single ring buffer (-W 1 -G 0) → grows until stop;
+# capped at 64 MiB via -C 64. Snaplen 200 bytes (headers + a bit) to keep
+# size sane while preserving L2/L3/L4 + first payload bytes.
+set +e
+
+PCAP=/tmp/spinifex-e2e-wan.pcap
+PIDFILE=/tmp/spinifex-e2e-tcpdump.pid
+
+if [ -f "$PIDFILE" ] && kill -0 "$(cat "$PIDFILE" 2>/dev/null)" 2>/dev/null; then
+    echo "[start-net-capture] tcpdump already running pid=$(cat "$PIDFILE")"
+    exit 0
+fi
+
+sudo rm -f "$PCAP" "$PIDFILE"
+
+if ! command -v tcpdump >/dev/null 2>&1; then
+    echo "[start-net-capture] tcpdump not installed — skipping" >&2
+    exit 0
+fi
+if ! ip link show br-wan >/dev/null 2>&1; then
+    echo "[start-net-capture] br-wan absent — skipping" >&2
+    exit 0
+fi
+
+sudo nohup tcpdump -i br-wan -s 200 -C 64 -W 1 -U \
+    -w "$PCAP" \
+    '(net 192.168.0.0/23) or (arp and (arp[24:4]&0xfffffe00=0xc0a80000))' \
+    >/tmp/spinifex-e2e-tcpdump.log 2>&1 &
+PID=$!
+echo "$PID" | sudo tee "$PIDFILE" >/dev/null
+disown
+
+sleep 1
+if ! kill -0 "$PID" 2>/dev/null; then
+    echo "[start-net-capture] tcpdump died — see /tmp/spinifex-e2e-tcpdump.log" >&2
+    cat /tmp/spinifex-e2e-tcpdump.log >&2 || true
+    sudo rm -f "$PIDFILE"
+    exit 1
+fi
+
+echo "[start-net-capture] tcpdump pid=$PID pcap=$PCAP"
+exit 0

.github/workflows/e2e-baremetal.yml

@@ -0,0 +1,107 @@
+name: E2E Bare-Metal
+
+on:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  statuses: write
+
+env:
+  GIT_BRANCH: ${{ github.ref_name }}
+
+jobs:
+  e2e_baremetal:
+    name: Bare-Metal Multi-Node E2E
+    runs-on: [ self-hosted, ci-baremetal ]
+    timeout-minutes: 120
+    concurrency:
+      group: e2e-baremetal-cluster
+      cancel-in-progress: false
+    env:
+      BM_ISO_CACHE: /var/cache/spinifex-iso-builder
+    steps:
+      - name: Clean Workspace
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/checkout@v6
+
+      - uses: actions/create-github-app-token@v3
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      - uses: actions/checkout@v6
+        with:
+          repository: mulgadc/mulga
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: |
+            scripts/iso-builder
+            scripts/baremetal
+          path: mulga
+
+      - name: Install SSH Key
+        run: |
+          echo "${{ secrets.BM_SSH_PRIVATE_KEY }}" > "${RUNNER_TEMP}/bm-ci"
+          chmod 0600 "${RUNNER_TEMP}/bm-ci"
+          ssh-keygen -y -f "${RUNNER_TEMP}/bm-ci" > "${RUNNER_TEMP}/bm-ci.pub"
+
+      - name: Build Artifacts
+        run: |
+          mulga/scripts/baremetal/bm-build-artifacts.sh \
+            "$GITHUB_WORKSPACE" \
+            "${RUNNER_TEMP}/spinifex-tests.tar" \
+            "${RUNNER_TEMP}/bm-bins"
+
+      - name: Build PXE Artifacts
+        run: |
+          sudo make -C mulga/scripts/iso-builder \
+            build-rootfs build-squashfs build-initramfs \
+            CACHE="${BM_ISO_CACHE}" \
+            SPINIFEX_BIN="${RUNNER_TEMP}/bm-bins/spx" \
+            INSTALLER_BIN="${RUNNER_TEMP}/bm-bins/spinifex-installer" \
+            SPINIFEX_SRC="${GITHUB_WORKSPACE}" \
+            VIPERBLOCK_SRC="${RUNNER_TEMP}/bm-bins/viperblock" \
+            VERSION="$(cd "$GITHUB_WORKSPACE" && git describe --tags --always 2>/dev/null || echo dev)"
+
+      - name: Write ePDU credentials
+        run: |
+          install -m 0600 /dev/null "${RUNNER_TEMP}/epdu.env"
+          cat >> "${RUNNER_TEMP}/epdu.env" <<EOF
+          EPDU_HOST=192.168.1.63
+          EPDU_SNMP_USER=ci-bmpower
+          EPDU_SNMP_SEC_LEVEL=authPriv
+          EPDU_SNMP_AUTH_PROTO=MD5
+          EPDU_SNMP_AUTH_PASS=${{ secrets.BM_EPDU_AUTH_PASS }}
+          EPDU_SNMP_PRIV_PROTO=DES
+          EPDU_SNMP_PRIV_PASS=${{ secrets.BM_EPDU_PRIV_PASS }}
+          EOF
+
+      - name: Cache VM images
+        run: |
+          UBUNTU_IMG="${BM_ISO_CACHE}/ubuntu-26.04-cloudimg-amd64.img"
+          if [ ! -f "$UBUNTU_IMG" ]; then
+            echo "Downloading Ubuntu 26.04 cloud image..."
+            sudo curl -fsSL -o "$UBUNTU_IMG" \
+              "https://cloud-images.ubuntu.com/resolute/current/resolute-server-cloudimg-amd64.img"
+            echo "  Downloaded $(du -h "$UBUNTU_IMG" | cut -f1)"
+          else
+            echo "Ubuntu 26.04 cloud image cached ($(du -h "$UBUNTU_IMG" | cut -f1))"
+          fi
+
+
+      - name: Run Bare-Metal E2E
+        env:
+          BM_CI_PASSWORD: ${{ secrets.BM_CI_PASSWORD }}
+          BM_SSH_KEY: ${{ runner.temp }}/bm-ci
+          BM_EPDU_ENV: ${{ runner.temp }}/epdu.env
+          BM_UBUNTU_IMAGE: ${{ env.BM_ISO_CACHE }}/ubuntu-26.04-cloudimg-amd64.img
+          SPX_EXTERNAL_POOL: ${{ env.SPINIFEX_CI_POOL }}
+        run: |
+          sudo -E mulga/scripts/baremetal/run-baremetal-e2e.sh \
+            --vmlinuz   "${BM_ISO_CACHE}/vmlinuz" \
+            --initrd    "${BM_ISO_CACHE}/initrd.img" \
+            --squashfs  "${BM_ISO_CACHE}/spinifex.squashfs" \
+            --tests-tar "${RUNNER_TEMP}/spinifex-tests.tar"

.github/workflows/e2e-ddil.yml

@@ -0,0 +1,229 @@
+name: E2E DDIL
+
+# Phase 1 trigger: workflow_dispatch only. Nightly cron + on-main-merge
+# land in mulga-siv-5.8 once ≥4 scenarios are asserting. See
+# docs/development/improvements/ddil-e2e-test-harness.md §4 / §Dependencies.
+on:
+  workflow_dispatch:
+    inputs:
+      quarantined:
+        description: "Comma-separated scenario letters to quarantine (A..F)"
+        required: false
+        default: ""
+
+permissions:
+  contents: read
+
+env:
+  GIT_BRANCH: ${{ github.ref_name }}
+
+jobs:
+  ddil:
+    name: DDIL Real Multi-Node
+    runs-on: [ self-hosted, ci-multi ]
+    # Budget: ~45m go test + tofu provision/bootstrap/teardown on ci-multi.
+    # In Phase 1 the scenarios all t.Skip, so the real-world runtime is
+    # dominated by provisioning; the 60m ceiling leaves headroom for
+    # Phase 2/3 when assertions come online.
+    timeout-minutes: 60
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-ddil-${{ github.run_id }}.log
+    steps:
+      - name: Clean Workspace
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/create-github-app-token@v3
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      # Full spinifex checkout: the Go harness must run on the runner
+      # (outside the cluster) so its SSH path survives scenario partitions
+      # that drop peer-to-peer cluster traffic. Running inside the cluster
+      # would self-sever when Scenario C isolates a node.
+      - uses: actions/checkout@v6
+        name: Checkout Spinifex
+        with:
+          path: spinifex
+
+      - uses: actions/checkout@v6
+        name: Checkout Mulga (tofu-cluster)
+        with:
+          repository: mulgadc/mulga
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: scripts/tofu-cluster
+          path: mulga
+
+      - name: Install gotestsum
+        env:
+          GOTESTSUM_VERSION: "1.12.0"
+        run: |
+          # Prebuilt tarball, not `go install`: gotestsum v1.12.0 pulls
+          # x/tools v0.19.0 whose tokeninternal static-assert pegs a
+          # narrow Go window (~1.22). The self-hosted runner's Go falls
+          # outside that window, so source-install fails with
+          # `invalid array length -delta * delta`. A prebuilt binary
+          # sidesteps toolchain coupling entirely.
+          BIN_DIR="${RUNNER_TEMP}/gotestsum-bin"
+          mkdir -p "${BIN_DIR}"
+          if [ ! -x "${BIN_DIR}/gotestsum" ]; then
+            curl -fsSL \
+              "https://github.com/gotestyourself/gotestsum/releases/download/v${GOTESTSUM_VERSION}/gotestsum_${GOTESTSUM_VERSION}_linux_amd64.tar.gz" \
+              | tar -xz -C "${BIN_DIR}" gotestsum
+          fi
+          echo "${BIN_DIR}" >> "${GITHUB_PATH}"
+          "${BIN_DIR}/gotestsum" --version
+
+      - name: Build Install Artifacts
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          ./build-install-artifacts.sh "${GIT_BRANCH}" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Provision VMs
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set -euo pipefail
+          # shellcheck source=/dev/null
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          export TF_VAR_git_branch="${GIT_BRANCH}"
+          quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
+          quiet_run "tofu workspace select"    "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
+          quiet_run "scrub-env (pre-clean)"    "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
+          quiet_run "tofu apply"               "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          ok "VMs provisioned"
+
+      - name: Bootstrap via Binary Install
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
+          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Run DDIL Scenarios
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set -o pipefail
+          INVENTORY=$(tofu output -json inventory)
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          SSH_USER=$(echo "$INVENTORY" | jq -r '.ssh_user')
+          REGION=$(echo "$INVENTORY" | jq -r '.region')
+          # DDIL_NODES must be the cluster-traffic IPs: PartitionNode
+          # installs iptables DROP rules for each peer, so the runner's
+          # SSH source IP must differ from these. On single-NIC
+          # tofu-cluster, wan_ip == cluster IP and the runner's outbound
+          # IP is distinct — see e2e-ddil-test-harness.md §3 Scenario C.
+          NODE_IPS=$(echo "$INVENTORY" | jq -r '[.nodes[].wan_ip] | join(",")')
+          PRIMARY_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          echo "DDIL_NODES=${NODE_IPS}"
+
+          # Witness VMs talk to the cluster's AWS gateway via aws-sdk-go,
+          # which resolves credentials from the standard chain. spx admin
+          # init writes [spinifex] to ${SSH_USER}'s ~/.aws/credentials on
+          # node1 — pull them over SSH and export as the SDK's env-var
+          # entries so the harness can launch instances without depending
+          # on a cluster-bootstrapped profile being present on the runner.
+          CREDS=$(ssh -i "${SSH_KEY}" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
+            "${SSH_USER}@${PRIMARY_IP}" 'cat ~/.aws/credentials')
+          AWS_ACCESS_KEY_ID=$(echo "$CREDS" | awk -F '= *' '/^aws_access_key_id/ {print $2; exit}')
+          AWS_SECRET_ACCESS_KEY=$(echo "$CREDS" | awk -F '= *' '/^aws_secret_access_key/ {print $2; exit}')
+          if [ -z "$AWS_ACCESS_KEY_ID" ] || [ -z "$AWS_SECRET_ACCESS_KEY" ]; then
+            echo "::error::Failed to extract spinifex AWS credentials from ${PRIMARY_IP}"
+            exit 1
+          fi
+
+          # Guest SSH key: bootstrap-install.sh imports spinifex-key into EC2;
+          # the matching private material was generated on the primary node at
+          # ~/.ssh/spinifex-key. Pull it to the runner so the harness can SSH
+          # the witness via its MapPublicIpOnLaunch-assigned EIP.
+          GUEST_KEY="${RUNNER_TEMP}/spinifex-key"
+          scp -i "${SSH_KEY}" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
+            "${SSH_USER}@${PRIMARY_IP}:~/.ssh/spinifex-key" "${GUEST_KEY}"
+          chmod 600 "${GUEST_KEY}"
+
+          mkdir -p "${RUNNER_TEMP}/ddil-results"
+          cd "${GITHUB_WORKSPACE}/spinifex"
+          # gotestsum: human-readable PASS/FAIL/SKIP lines in Actions stdout,
+          # plus machine-readable jsonfile + JUnit XML artifacts. The JUnit
+          # file feeds the junit-report composite action below so failures
+          # become summary table rows + workflow annotations.
+          # shellcheck source=/dev/null
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          banner "DDIL scenarios"
+          DDIL_NODES="${NODE_IPS}" \
+          DDIL_SSH_USER="${SSH_USER}" \
+          DDIL_SSH_KEY="${SSH_KEY}" \
+          DDIL_GUEST_SSH_KEY="${GUEST_KEY}" \
+          DDIL_QUARANTINED="${{ inputs.quarantined }}" \
+          AWS_REGION="${REGION}" \
+          AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" \
+          AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" \
+            gotestsum --jsonfile "${RUNNER_TEMP}/ddil-results/results.json" \
+                      --junitfile "${RUNNER_TEMP}/ddil-results/junit-ddil.xml" \
+                      --format testname \
+                      -- -tags=e2e -timeout 45m ./tests/e2e/ddil/scenarios/...
+          rc=$?
+          if [[ $rc -eq 0 ]]; then ok "all scenarios passed"; else bad "scenarios failed (rc=$rc)"; fi
+          exit $rc
+
+      - name: Publish test summary
+        if: always()
+        uses: ./spinifex/.github/actions/junit-report
+        with:
+          junit-glob: ${{ runner.temp }}/ddil-results/junit-*.xml
+          title: "DDIL scenario results"
+
+      - name: Collect Cluster Logs
+        if: failure()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          # Pull spinifex-* journals + QEMU process listings from every
+          # node into the artifact bundle so failures can be triaged
+          # without re-provisioning the cluster. Best-effort — a missing
+          # node should not mask the original test failure.
+          set +e
+          INVENTORY=$(tofu output -json inventory)
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          SSH_USER=$(echo "$INVENTORY" | jq -r '.ssh_user')
+          LOG_DIR="${RUNNER_TEMP}/ddil-results/cluster-logs"
+          mkdir -p "${LOG_DIR}"
+
+          # shellcheck source=/dev/null
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          mapfile -t NODE_IPS < <(echo "$INVENTORY" | jq -r '.nodes[].wan_ip')
+          for ip in "${NODE_IPS[@]}"; do
+            SSH="ssh -i ${SSH_KEY} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${SSH_USER}@${ip}"
+            ${SSH} 'sudo journalctl -u "spinifex-*" --no-pager -n 5000' > "${LOG_DIR}/${ip}-spinifex.log" 2>&1
+            ${SSH} 'ps auxw | grep -E "qemu-system|spinifex" | grep -v grep' > "${LOG_DIR}/${ip}-procs.txt" 2>&1
+            ${SSH} 'ls -la /var/lib/spinifex 2>/dev/null; ls -la /var/log/spinifex 2>/dev/null' > "${LOG_DIR}/${ip}-state.txt" 2>&1
+            ok "$ip: $(wc -l < "${LOG_DIR}/${ip}-spinifex.log") journal lines, $(wc -l < "${LOG_DIR}/${ip}-procs.txt") procs"
+          done
+
+      - name: Upload DDIL Results
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: ddil-results-${{ github.run_id }}
+          path: ${{ runner.temp }}/ddil-results
+          retention-days: 30
+
+      - name: Teardown
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set -euo pipefail
+          # shellcheck source=/dev/null
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy"          "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
+          ok "teardown complete"

.github/workflows/e2e-nightly.yml

@@ -0,0 +1,218 @@
+name: E2E Nightly
+
+on:
+  schedule:
+    - cron: "0 14 * * 0-4" # 00:00 Mon-Fri AEST = 14:00 Sun-Thu UTC
+  workflow_dispatch:
+    inputs:
+      cells:
+        description: "Comma-separated cell numbers to run (blank = all)"
+        required: false
+
+permissions:
+  contents: read
+
+# Scheduled runs are loaded from the default branch (main), so the schedule
+# trigger must stay in sync on main for cron to fire.
+env:
+  GIT_BRANCH: ${{ github.event_name == 'schedule' && 'dev' || github.ref_name }}
+  UBUNTU_GOLD_IMAGE: /mnt/disk1/gold-images/spinifex-image-builder-ubuntu-24.04.qcow2
+
+jobs:
+  matrix:
+    name: "cell-${{ matrix.cell }} (${{ matrix.topo }}/${{ matrix.net }}/${{
+      matrix.host }}${{ matrix.extra && format('/{0}', matrix.extra) || '' }})"
+    strategy:
+      fail-fast: false
+      max-parallel: 3
+      matrix:
+        include:
+          - { cell: 1,  install: binary, topo: single,     net: dhcp,   host: debian-13,    arch: x86_64, runner: ci-single, extra: "" }
+          - { cell: 2,  install: binary, topo: single,     net: static, host: ubuntu-24.04, arch: x86_64, runner: ci-single, extra: "" }
+          - { cell: 3,  install: binary, topo: single,     net: dhcp,   host: ubuntu-24.04, arch: x86_64, runner: ci-single, extra: "" }
+          - { cell: 8,  install: binary, topo: real-multi, net: dhcp,   host: debian-13,    arch: x86_64, runner: ci-multi,  extra: "" }
+          - { cell: 9,  install: binary, topo: real-multi, net: static, host: ubuntu-24.04, arch: x86_64, runner: ci-multi,  extra: "" }
+          - { cell: 10, install: binary, topo: real-multi, net: dhcp,   host: ubuntu-24.04, arch: x86_64, runner: ci-multi,  extra: "" }
+          - { cell: 17, install: binary, topo: single,     net: static, host: debian-13,    arch: x86_64, runner: ci-pseudo, extra: "tofu-examples" }
+          - { cell: 18, install: binary, topo: single,     net: static, host: debian-13,    arch: x86_64, runner: ci-single, extra: "reboot-single" }
+
+    runs-on: [ self-hosted, "${{ matrix.runner }}" ]
+    timeout-minutes: 35
+    env:
+      TOFU_LOG: /tmp/e2e-nightly-cell-${{ matrix.cell }}-${{ github.run_id }}.log
+
+    steps:
+      # blank or scheduled run = full matrix.
+      # Input must be comma-separated with no whitespace (e.g. '1,2,17', not '1, 2').
+      - name: Gate on Dispatch Input
+        id: gate
+        env:
+          CELL: ${{ matrix.cell }}
+          CELLS: ${{ github.event_name == 'workflow_dispatch' && inputs.cells || '' }}
+        run: |
+          if [ -z "$CELLS" ] || [[ ",${CELLS}," == *",${CELL},"* ]]; then
+            echo "selected=true" >> "$GITHUB_OUTPUT"
+            echo "cell ${CELL} selected"
+          else
+            echo "selected=false" >> "$GITHUB_OUTPUT"
+            echo "cell ${CELL} not in '${CELLS}' — subsequent steps will skip"
+          fi
+
+      - name: Clean Workspace
+        if: steps.gate.outputs.selected == 'true'
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/create-github-app-token@v3
+        if: steps.gate.outputs.selected == 'true'
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      - uses: actions/checkout@v6
+        if: steps.gate.outputs.selected == 'true'
+        with:
+          repository: mulgadc/mulga
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: scripts/tofu-cluster
+          path: mulga
+
+      - name: Build Install Artifacts
+        if: steps.gate.outputs.selected == 'true'
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          ./build-install-artifacts.sh "${GIT_BRANCH}" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Provision VMs
+        if: steps.gate.outputs.selected == 'true'
+        working-directory: mulga/scripts/tofu-cluster
+        env:
+          MATRIX_HOST: ${{ matrix.host }}
+        run: |
+          export TF_VAR_git_branch="${GIT_BRANCH}"
+
+          TOFU_VAR_ARGS=""
+          if [ "$MATRIX_HOST" = "ubuntu-24.04" ]; then
+            TOFU_VAR_ARGS="-var=host_image_override=${UBUNTU_GOLD_IMAGE}"
+          fi
+
+          tofu init -input=false >> "$TOFU_LOG" 2>&1
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars ${TOFU_VAR_ARGS} -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+          tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars ${TOFU_VAR_ARGS} -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+
+      - name: Bootstrap via Binary Install
+        if: steps.gate.outputs.selected == 'true'
+        working-directory: mulga/scripts/tofu-cluster
+        env:
+          MATRIX_NET: ${{ matrix.net }}
+        run: |
+          if [ "$MATRIX_NET" = "dhcp" ]; then
+            unset SPX_EXTERNAL_POOL
+            export SPX_EXTERNAL_SOURCE=dhcp
+          else
+            export SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}"
+          fi
+
+          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Run Single Node E2E
+        if: steps.gate.outputs.selected == 'true' && matrix.topo == 'single' && matrix.extra == ''
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
+            "cd ~/e2e-tests && ./run-e2e.sh && ./run-cert-e2e.sh && ENDPOINT=https://${WAN_IP}:9999 ./run-lb-e2e.sh"
+
+      - name: Run Multi Node E2E
+        if: steps.gate.outputs.selected == 'true' && matrix.topo == 'real-multi'
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+
+          NODE_IPS=""
+          NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
+          for i in $(seq 0 $((NODE_COUNT - 1))); do
+            NODE_IPS="${NODE_IPS} $(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")"
+          done
+
+          ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
+            "cd ~/e2e-tests && ./run-multinode-e2e.sh ${NODE_IPS} && ./run-cert-e2e.sh --multinode ${NODE_IPS} && ENDPOINT=https://${NODE1_IP}:9999 ./run-lb-e2e.sh --nodes ${NODE_IPS}"
+
+      - name: Run Tofu Workbooks
+        if: steps.gate.outputs.selected == 'true' && matrix.extra == 'tofu-examples'
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
+            "cd ~/e2e-tests && ./run-tofu-examples-e2e.sh"
+
+      - name: Run Reboot Resilience E2E
+        if: steps.gate.outputs.selected == 'true' && matrix.extra == 'reboot-single'
+        working-directory: mulga/scripts/tofu-cluster
+        # Driver runs from the runner (NOT via ssh-and-run) so it survives the
+        # systemctl-reboot-induced ssh disconnect mid-run.
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+
+          REBOOT_DIR="${RUNNER_TEMP}/reboot-e2e"
+          mkdir -p "$REBOOT_DIR"
+          tar xf "${RUNNER_TEMP}/spinifex-tests.tar" -C "$REBOOT_DIR"
+          chmod +x "$REBOOT_DIR/run-reboot-e2e.sh"
+
+          WAN_IP="$WAN_IP" SSH_KEY="$SSH_KEY" "$REBOOT_DIR/run-reboot-e2e.sh"
+
+      - name: Dump Cluster Journals on Failure
+        if: failure() && steps.gate.outputs.selected == 'true'
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory 2>/dev/null) || { echo "no inventory; skipping"; exit 0; }
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
+          for i in $(seq 0 $((NODE_COUNT - 1))); do
+            IP=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
+            echo "===== node$((i+1)) ($IP) ====="
+            for svc in spinifex-nats spinifex-predastore spinifex-viperblock spinifex-daemon spinifex-awsgw spinifex-vpcd dhcpcd; do
+              echo "--- $svc ---"
+              ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" tf-user@${IP} \
+                "sudo journalctl -u $svc --no-pager -n 200 2>/dev/null || echo '(not found)'" 2>&1 || echo "(node unreachable)"
+            done
+            echo "--- DHCP client state ---"
+            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" tf-user@${IP} \
+              "systemctl is-enabled dhcpcd.service 2>/dev/null; systemctl is-active dhcpcd.service 2>/dev/null; sudo ss -lupn 'sport = 68' 2>/dev/null; pgrep -laf dhcpcd 2>/dev/null" 2>&1 || echo "(node unreachable)"
+          done
+
+      - name: Teardown
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        env:
+          MATRIX_HOST: ${{ matrix.host }}
+        run: |
+          TOFU_VAR_ARGS=""
+          if [ "$MATRIX_HOST" = "ubuntu-24.04" ]; then
+            TOFU_VAR_ARGS="-var=host_image_override=${UBUNTU_GOLD_IMAGE}"
+          fi
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars ${TOFU_VAR_ARGS} -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true

.github/workflows/e2e.yml

@@ -1,4 +1,4 @@
-name: Spinifex E2E Tests
+name: E2E
 
 on:
   push:
@@ -13,6 +13,14 @@ on:
       - tests/e2e/**
       - .github/workflows/e2e.yml
   workflow_dispatch:
+    inputs:
+      sha:
+        description: "PR head SHA to report status on"
+        required: false
+
+permissions:
+  contents: read
+  statuses: write
 
 env:
   GIT_BRANCH: ${{ github.ref_name }}
@@ -20,18 +28,18 @@ env:
 jobs:
   e2e_single:
     name: Single-Node E2E
-    runs-on: [self-hosted, ci-single]
-    timeout-minutes: 25
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 35
     env:
       TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
     steps:
-      - name: Clean workspace
-        run: rm -rf "$GITHUB_WORKSPACE"/*
+      - name: Clean Workspace
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         id: app-token
         with:
-          app-id: ${{ secrets.APP_ID }}
+          client-id: ${{ secrets.APP_ID }}
           private-key: ${{ secrets.APP_PRIVATE_KEY }}
           repositories: mulga
 
@@ -42,27 +50,63 @@ jobs:
           sparse-checkout: scripts/tofu-cluster
           path: mulga
 
-      - name: Clone and build
+      - name: Build Install Artifacts
         working-directory: mulga/scripts/tofu-cluster
-        run: ./build-artifacts.sh "${GIT_BRANCH}" "${RUNNER_TEMP}/spinifex-artifacts.tar"
+        run: |
+          ./build-install-artifacts.sh "${GIT_BRANCH}" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-tests.tar"
 
       - name: Provision VM
         working-directory: mulga/scripts/tofu-cluster
         run: |
           export TF_VAR_git_branch="${GIT_BRANCH}"
           tofu init -input=false >> "$TOFU_LOG" 2>&1
-          tofu workspace select -or-create env3 >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/env3.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
-          ./scrub-env.sh env3 >> "$TOFU_LOG" 2>&1 || true
-          tofu apply -var-file=envs/env3.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+          tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
 
-      - name: Bootstrap Cluster
+      - name: Bootstrap via Binary Install
         working-directory: mulga/scripts/tofu-cluster
         run: |
-          SPX_PREBUILT=true \
-          SPX_ARTIFACT_TAR="${RUNNER_TEMP}/spinifex-artifacts.tar" \
-          SPX_EXTERNAL_POOL="192.168.1.230-192.168.1.240" \
-          ./bootstrap.sh env3 >> "$TOFU_LOG" 2>&1
+          SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
+          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Start Packet Capture (siv-167)
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} 'bash -s' <<'REMOTE'
+          set -x
+          sudo mkdir -p /tmp/spinifex-diag
+          sudo chmod 0777 /tmp/spinifex-diag
+          sudo rm -f /tmp/spinifex-diag/br-wan.pcap /tmp/spinifex-diag/tcpdump.pid /tmp/spinifex-diag/tcpdump.log
+          sudo bash -c '
+            nohup tcpdump -i br-wan -nn -s 0 -U \
+              -w /tmp/spinifex-diag/br-wan.pcap \
+              "(tcp port 22 or tcp port 80 or tcp port 443 or tcp port 9999) or arp" \
+              </dev/null >/tmp/spinifex-diag/tcpdump.log 2>&1 &
+            echo $! > /tmp/spinifex-diag/tcpdump.pid
+          '
+          sleep 2
+          PID=$(cat /tmp/spinifex-diag/tcpdump.pid 2>/dev/null || echo "")
+          if [ -n "$PID" ] && sudo kill -0 "$PID" 2>/dev/null; then
+            echo "tcpdump running PID=$PID"
+            ls -la /tmp/spinifex-diag/
+          else
+            echo "tcpdump failed to start"
+            cat /tmp/spinifex-diag/tcpdump.log 2>&1 || true
+            exit 1
+          fi
+          REMOTE
 
       - name: Run E2E Tests
         working-directory: mulga/scripts/tofu-cluster
@@ -72,30 +116,97 @@ jobs:
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
           SSH_KEY="${SSH_KEY/#\~/$HOME}"
           ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
-            'cd ~/Development/mulga/spinifex && ./tests/e2e/run-e2e.sh && ./tests/e2e/run-cert-e2e.sh'
+            "cd ~/e2e-tests && ./run-e2e.sh && ./run-cert-e2e.sh && ENDPOINT=https://${WAN_IP}:9999 ./run-lb-e2e.sh"
+
+      - name: Collect Diagnostics
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          DIAG_DIR="${RUNNER_TEMP}/diag-single"
+          mkdir -p "$DIAG_DIR"
+          ssh -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i "$SSH_KEY" tf-user@${WAN_IP} 'bash -s' > "$DIAG_DIR/collect.log" 2>&1 <<'REMOTE' || true
+          set -x
+          DIAG=/tmp/spinifex-diag
+          sudo mkdir -p "$DIAG" && cd "$DIAG"
+          if [ -f tcpdump.pid ]; then
+            PID=$(cat tcpdump.pid)
+            sudo kill -INT "$PID" 2>/dev/null || true
+            for _ in 1 2 3 4 5; do sudo kill -0 "$PID" 2>/dev/null || break; sleep 1; done
+            sudo kill -KILL "$PID" 2>/dev/null || true
+          fi
+          sudo journalctl -u 'spinifex-*' --no-pager > journal-spinifex-full.log 2>&1 || true
+          sudo journalctl -u 'spinifex-*' --no-pager -n 10000 > journal-spinifex.log 2>&1 || true
+          sudo journalctl -u 'ovn-*' --no-pager -n 5000 > journal-ovn.log 2>&1 || true
+          sudo journalctl -u 'openvswitch-*' --no-pager -n 2000 > journal-ovs.log 2>&1 || true
+          sudo ovs-vsctl show > ovs-vsctl-show.txt 2>&1 || true
+          sudo ovs-ofctl dump-flows br-int > br-int-flows.txt 2>&1 || true
+          sudo ovs-ofctl dump-flows br-ex > br-ex-flows.txt 2>&1 || true
+          sudo ovs-appctl dpctl/dump-flows > dpctl-flows.txt 2>&1 || true
+          sudo ovs-appctl dpctl/dump-conntrack -m > dpctl-conntrack.txt 2>&1 || true
+          sudo ovn-nbctl show > ovn-nbctl-show.txt 2>&1 || true
+          sudo ovn-sbctl show > ovn-sbctl-show.txt 2>&1 || true
+          sudo ovn-sbctl list Port_Binding > sbdb-port-binding.txt 2>&1 || true
+          sudo ovn-sbctl list Chassis > sbdb-chassis.txt 2>&1 || true
+          sudo ovn-nbctl list NAT > nbdb-nat.txt 2>&1 || true
+          sudo ovn-nbctl list Logical_Router > nbdb-lr.txt 2>&1 || true
+          sudo ovn-nbctl list Logical_Router_Policy > nbdb-lr-policy.txt 2>&1 || true
+          sudo ovn-nbctl list Logical_Router_Static_Route > nbdb-lr-static-route.txt 2>&1 || true
+          sudo ovn-appctl -t ovn-controller ct-zone-list > ovn-ctzones.txt 2>&1 || true
+          sudo ip -d link > ip-link.txt 2>&1 || true
+          sudo ip addr > ip-addr.txt 2>&1 || true
+          sudo ip route > ip-route.txt 2>&1 || true
+          sudo ip neigh > ip-neigh.txt 2>&1 || true
+          sudo bridge fdb show > bridge-fdb.txt 2>&1 || true
+          sudo bridge link > bridge-link.txt 2>&1 || true
+          for f in /sys/class/net/br-wan/brif/*/hairpin_mode /sys/class/net/br-wan/brif/*/learning /sys/class/net/br-wan/brif/*/flood; do
+            [ -e "$f" ] && echo "$f $(cat $f)"
+          done > hairpin-modes.txt 2>&1 || true
+          sudo iptables-save > iptables.txt 2>&1 || true
+          sudo nft list ruleset > nft.txt 2>&1 || true
+          sudo dmesg --ctime > dmesg.txt 2>&1 || true
+          sudo virsh list --all > virsh-list.txt 2>&1 || true
+          ls -la /var/lib/spinifex /var/log/spinifex 2>&1 > spinifex-dirs.txt || true
+          sudo chmod -R a+r /tmp/spinifex-diag
+          tar czf /tmp/spinifex-diag.tar.gz -C /tmp spinifex-diag
+          REMOTE
+          scp -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i "$SSH_KEY" \
+            tf-user@${WAN_IP}:/tmp/spinifex-diag.tar.gz "$DIAG_DIR/spinifex-diag.tar.gz" || true
+
+      - name: Upload Diagnostics
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: e2e-diag-single-${{ github.run_id }}
+          path: ${{ runner.temp }}/diag-single/
+          if-no-files-found: warn
+          retention-days: 14
 
       - name: Teardown
         if: always()
         working-directory: mulga/scripts/tofu-cluster
         run: |
-          tofu workspace select -or-create env3 >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/env3.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
-          ./scrub-env.sh env3 >> "$TOFU_LOG" 2>&1 || true
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
 
   e2e_pseudo_multi:
     name: Pseudo Multi-Node E2E
-    runs-on: [self-hosted, ci-pseudo]
-    timeout-minutes: 30
+    runs-on: [ self-hosted, ci-pseudo ]
+    timeout-minutes: 25
     env:
       TOFU_LOG: /tmp/e2e-tofu-pseudo-${{ github.run_id }}.log
     steps:
-      - name: Clean workspace
-        run: rm -rf "$GITHUB_WORKSPACE"/*
+      - name: Clean Workspace
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         id: app-token
         with:
-          app-id: ${{ secrets.APP_ID }}
+          client-id: ${{ secrets.APP_ID }}
           private-key: ${{ secrets.APP_PRIVATE_KEY }}
           repositories: mulga
 
@@ -106,26 +217,27 @@ jobs:
           sparse-checkout: scripts/tofu-cluster
           path: mulga
 
-      - name: Clone and build
+      - name: Clone and Build
         working-directory: mulga/scripts/tofu-cluster
-        run: ./build-artifacts.sh "${GIT_BRANCH}" "${RUNNER_TEMP}/spinifex-artifacts.tar"
+        run: ./build-artifacts.sh "${GIT_BRANCH}"
+          "${RUNNER_TEMP}/spinifex-artifacts.tar"
 
       - name: Provision VM
         working-directory: mulga/scripts/tofu-cluster
         run: |
           export TF_VAR_git_branch="${GIT_BRANCH}"
           tofu init -input=false >> "$TOFU_LOG" 2>&1
-          tofu workspace select -or-create env4 >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/env4.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
-          ./scrub-env.sh env4 >> "$TOFU_LOG" 2>&1 || true
-          tofu apply -var-file=envs/env4.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+          tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
 
       - name: Bootstrap Cluster
         working-directory: mulga/scripts/tofu-cluster
         run: |
           SPX_PREBUILT=true \
           SPX_ARTIFACT_TAR="${RUNNER_TEMP}/spinifex-artifacts.tar" \
-          ./bootstrap.sh env4 >> "$TOFU_LOG" 2>&1
+          ./bootstrap.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
 
       - name: Run E2E Tests
         working-directory: mulga/scripts/tofu-cluster
@@ -135,30 +247,90 @@ jobs:
           SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
           SSH_KEY="${SSH_KEY/#\~/$HOME}"
           ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
-            'cd ~/Development/mulga/spinifex && SPX_EXTERNAL_POOL="192.168.1.220-192.168.1.229" ./tests/e2e/run-pseudo-multinode-e2e.sh'
+            "cd ~/Development/mulga/spinifex && SPX_EXTERNAL_POOL=\"${SPINIFEX_CI_POOL}\" ./tests/e2e/run-pseudo-multinode-e2e.sh"
+
+      - name: Collect Diagnostics
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          DIAG_DIR="${RUNNER_TEMP}/diag-pseudo"
+          mkdir -p "$DIAG_DIR"
+          ssh -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i "$SSH_KEY" tf-user@${WAN_IP} 'bash -s' > "$DIAG_DIR/collect.log" 2>&1 <<'REMOTE' || true
+          set -x
+          DIAG=/tmp/spinifex-diag
+          sudo rm -rf "$DIAG" && mkdir -p "$DIAG" && cd "$DIAG"
+          sudo journalctl -u 'spinifex-*' --no-pager > journal-spinifex-full.log 2>&1 || true
+          sudo journalctl -u 'spinifex-*' --no-pager -n 10000 > journal-spinifex.log 2>&1 || true
+          sudo journalctl -u 'ovn-*' --no-pager -n 5000 > journal-ovn.log 2>&1 || true
+          sudo journalctl -u 'openvswitch-*' --no-pager -n 2000 > journal-ovs.log 2>&1 || true
+          sudo ovs-vsctl show > ovs-vsctl-show.txt 2>&1 || true
+          sudo ovs-ofctl dump-flows br-int > br-int-flows.txt 2>&1 || true
+          sudo ovs-ofctl dump-flows br-ex > br-ex-flows.txt 2>&1 || true
+          sudo ovs-appctl dpctl/dump-flows > dpctl-flows.txt 2>&1 || true
+          sudo ovs-appctl dpctl/dump-conntrack -m > dpctl-conntrack.txt 2>&1 || true
+          sudo ovn-nbctl show > ovn-nbctl-show.txt 2>&1 || true
+          sudo ovn-sbctl show > ovn-sbctl-show.txt 2>&1 || true
+          sudo ovn-sbctl list Port_Binding > sbdb-port-binding.txt 2>&1 || true
+          sudo ovn-sbctl list Chassis > sbdb-chassis.txt 2>&1 || true
+          sudo ovn-nbctl list NAT > nbdb-nat.txt 2>&1 || true
+          sudo ovn-nbctl list Logical_Router > nbdb-lr.txt 2>&1 || true
+          sudo ovn-nbctl list Logical_Router_Policy > nbdb-lr-policy.txt 2>&1 || true
+          sudo ovn-nbctl list Logical_Router_Static_Route > nbdb-lr-static-route.txt 2>&1 || true
+          sudo ovn-appctl -t ovn-controller ct-zone-list > ovn-ctzones.txt 2>&1 || true
+          sudo ip -d link > ip-link.txt 2>&1 || true
+          sudo ip addr > ip-addr.txt 2>&1 || true
+          sudo ip route > ip-route.txt 2>&1 || true
+          sudo ip neigh > ip-neigh.txt 2>&1 || true
+          sudo bridge fdb show > bridge-fdb.txt 2>&1 || true
+          sudo bridge link > bridge-link.txt 2>&1 || true
+          for f in /sys/class/net/br-wan/brif/*/hairpin_mode /sys/class/net/br-wan/brif/*/learning /sys/class/net/br-wan/brif/*/flood; do
+            [ -e "$f" ] && echo "$f $(cat $f)"
+          done > hairpin-modes.txt 2>&1 || true
+          sudo iptables-save > iptables.txt 2>&1 || true
+          sudo nft list ruleset > nft.txt 2>&1 || true
+          sudo dmesg --ctime > dmesg.txt 2>&1 || true
+          sudo virsh list --all > virsh-list.txt 2>&1 || true
+          ls -la /var/lib/spinifex /var/log/spinifex 2>&1 > spinifex-dirs.txt || true
+          tar czf /tmp/spinifex-diag.tar.gz -C /tmp spinifex-diag
+          REMOTE
+          scp -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i "$SSH_KEY" \
+            tf-user@${WAN_IP}:/tmp/spinifex-diag.tar.gz "$DIAG_DIR/spinifex-diag.tar.gz" || true
+
+      - name: Upload Diagnostics
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: e2e-diag-pseudo-${{ github.run_id }}
+          path: ${{ runner.temp }}/diag-pseudo/
+          if-no-files-found: warn
+          retention-days: 14
 
       - name: Teardown
         if: always()
         working-directory: mulga/scripts/tofu-cluster
         run: |
-          tofu workspace select -or-create env4 >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/env4.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
-          ./scrub-env.sh env4 >> "$TOFU_LOG" 2>&1 || true
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
 
   e2e_real_multi:
     name: Real Multi-Node E2E
-    runs-on: [self-hosted, ci-multi]
-    timeout-minutes: 25
+    runs-on: [ self-hosted, ci-multi ]
+    timeout-minutes: 30
     env:
       TOFU_LOG: /tmp/e2e-tofu-multi-${{ github.run_id }}.log
     steps:
-      - name: Clean workspace
-        run: rm -rf "$GITHUB_WORKSPACE"/*
+      - name: Clean Workspace
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         id: app-token
         with:
-          app-id: ${{ secrets.APP_ID }}
+          client-id: ${{ secrets.APP_ID }}
           private-key: ${{ secrets.APP_PRIVATE_KEY }}
           repositories: mulga
 
@@ -169,27 +341,32 @@ jobs:
           sparse-checkout: scripts/tofu-cluster
           path: mulga
 
-      - name: Clone and build
+      - name: Build Install Artifacts
         working-directory: mulga/scripts/tofu-cluster
-        run: ./build-artifacts.sh "${GIT_BRANCH}" "${RUNNER_TEMP}/spinifex-artifacts.tar"
+        run: |
+          ./build-install-artifacts.sh "${GIT_BRANCH}" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-tests.tar"
 
       - name: Provision VMs
         working-directory: mulga/scripts/tofu-cluster
         run: |
           export TF_VAR_git_branch="${GIT_BRANCH}"
           tofu init -input=false >> "$TOFU_LOG" 2>&1
-          tofu workspace select -or-create env2 >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/env2.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
-          ./scrub-env.sh env2 >> "$TOFU_LOG" 2>&1 || true
-          tofu apply -var-file=envs/env2.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+          tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
 
-      - name: Bootstrap Cluster
+      - name: Bootstrap via Binary Install
         working-directory: mulga/scripts/tofu-cluster
         run: |
-          SPX_PREBUILT=true \
-          SPX_ARTIFACT_TAR="${RUNNER_TEMP}/spinifex-artifacts.tar" \
-          SPX_EXTERNAL_POOL="192.168.1.240-192.168.1.250" \
-          ./bootstrap.sh env2 >> "$TOFU_LOG" 2>&1
+          SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
+          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
 
       - name: Run E2E Tests
         working-directory: mulga/scripts/tofu-cluster
@@ -206,13 +383,107 @@ jobs:
             NODE_IPS="${NODE_IPS} $(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")"
           done
 
+          NODE2_IP=$(echo "$INVENTORY" | jq -r '.nodes[1].wan_ip')
+
           ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
-            "cd ~/Development/mulga/spinifex && ./tests/e2e/run-multinode-e2e.sh ${NODE_IPS} && ./tests/e2e/run-cert-e2e.sh --multinode ${NODE_IPS}"
+            "cd ~/e2e-tests && ./run-multinode-e2e.sh ${NODE_IPS} && ./run-cert-e2e.sh --multinode ${NODE_IPS} && ENDPOINT=https://${NODE1_IP}:9999 ./run-lb-e2e.sh --nodes ${NODE_IPS}"
+
+      - name: Collect Diagnostics
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          DIAG_DIR="${RUNNER_TEMP}/diag-multi"
+          mkdir -p "$DIAG_DIR"
+          cat > "${RUNNER_TEMP}/collect-diag.sh" <<'REMOTE'
+          set -x
+          DIAG=/tmp/spinifex-diag
+          sudo rm -rf "$DIAG" && mkdir -p "$DIAG" && cd "$DIAG"
+          sudo journalctl -u 'spinifex-*' --no-pager > journal-spinifex-full.log 2>&1 || true
+          sudo journalctl -u 'spinifex-*' --no-pager -n 10000 > journal-spinifex.log 2>&1 || true
+          sudo journalctl -u 'ovn-*' --no-pager -n 5000 > journal-ovn.log 2>&1 || true
+          sudo journalctl -u 'openvswitch-*' --no-pager -n 2000 > journal-ovs.log 2>&1 || true
+          sudo ovs-vsctl show > ovs-vsctl-show.txt 2>&1 || true
+          sudo ovs-ofctl dump-flows br-int > br-int-flows.txt 2>&1 || true
+          sudo ovs-ofctl dump-flows br-ex > br-ex-flows.txt 2>&1 || true
+          sudo ovs-appctl dpctl/dump-flows > dpctl-flows.txt 2>&1 || true
+          sudo ovs-appctl dpctl/dump-conntrack -m > dpctl-conntrack.txt 2>&1 || true
+          sudo ovn-nbctl show > ovn-nbctl-show.txt 2>&1 || true
+          sudo ovn-sbctl show > ovn-sbctl-show.txt 2>&1 || true
+          sudo ovn-sbctl list Port_Binding > sbdb-port-binding.txt 2>&1 || true
+          sudo ovn-sbctl list Chassis > sbdb-chassis.txt 2>&1 || true
+          sudo ovn-nbctl list NAT > nbdb-nat.txt 2>&1 || true
+          sudo ovn-nbctl list Logical_Router > nbdb-lr.txt 2>&1 || true
+          sudo ovn-nbctl list Logical_Router_Policy > nbdb-lr-policy.txt 2>&1 || true
+          sudo ovn-nbctl list Logical_Router_Static_Route > nbdb-lr-static-route.txt 2>&1 || true
+          sudo ovn-appctl -t ovn-controller ct-zone-list > ovn-ctzones.txt 2>&1 || true
+          sudo ip -d link > ip-link.txt 2>&1 || true
+          sudo ip addr > ip-addr.txt 2>&1 || true
+          sudo ip route > ip-route.txt 2>&1 || true
+          sudo ip neigh > ip-neigh.txt 2>&1 || true
+          sudo bridge fdb show > bridge-fdb.txt 2>&1 || true
+          sudo bridge link > bridge-link.txt 2>&1 || true
+          for f in /sys/class/net/br-wan/brif/*/hairpin_mode /sys/class/net/br-wan/brif/*/learning /sys/class/net/br-wan/brif/*/flood; do
+            [ -e "$f" ] && echo "$f $(cat $f)"
+          done > hairpin-modes.txt 2>&1 || true
+          sudo iptables-save > iptables.txt 2>&1 || true
+          sudo nft list ruleset > nft.txt 2>&1 || true
+          sudo dmesg --ctime > dmesg.txt 2>&1 || true
+          sudo virsh list --all > virsh-list.txt 2>&1 || true
+          ls -la /var/lib/spinifex /var/log/spinifex 2>&1 > spinifex-dirs.txt || true
+          tar czf /tmp/spinifex-diag.tar.gz -C /tmp spinifex-diag
+          REMOTE
+          NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
+          for i in $(seq 0 $((NODE_COUNT - 1))); do
+            NODE_IP=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
+            NODE_DIR="$DIAG_DIR/node${i}-${NODE_IP}"
+            mkdir -p "$NODE_DIR"
+            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i "$SSH_KEY" tf-user@${NODE_IP} 'bash -s' < "${RUNNER_TEMP}/collect-diag.sh" > "$NODE_DIR/collect.log" 2>&1 || true
+            scp -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i "$SSH_KEY" \
+              tf-user@${NODE_IP}:/tmp/spinifex-diag.tar.gz "$NODE_DIR/spinifex-diag.tar.gz" || true
+          done
+
+      - name: Upload Diagnostics
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: e2e-diag-multi-${{ github.run_id }}
+          path: ${{ runner.temp }}/diag-multi/
+          if-no-files-found: warn
+          retention-days: 14
 
       - name: Teardown
         if: always()
         working-directory: mulga/scripts/tofu-cluster
         run: |
-          tofu workspace select -or-create env2 >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/env2.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
-          ./scrub-env.sh env2 >> "$TOFU_LOG" 2>&1 || true
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+
+  e2e_status:
+    name: Report E2E Status
+    if: always() && inputs.sha
+    needs: [ e2e_single, e2e_pseudo_multi, e2e_real_multi ]
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 5
+    steps:
+      - name: Report Status to PR
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          if [[ "${{ needs.e2e_single.result }}" == "success" && \
+                "${{ needs.e2e_pseudo_multi.result }}" == "success" && \
+                "${{ needs.e2e_real_multi.result }}" == "success" ]]; then
+            STATE="success"
+            DESC="All E2E suites passed"
+          else
+            STATE="failure"
+            DESC="E2E failed: single=${{ needs.e2e_single.result }} pseudo=${{ needs.e2e_pseudo_multi.result }} multi=${{ needs.e2e_real_multi.result }}"
+          fi
+          gh api "repos/${{ github.repository }}/statuses/${{ inputs.sha }}" \
+            -f state="$STATE" \
+            -f context=e2e \
+            -f description="$DESC" \
+            -f target_url="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"

.github/workflows/release.yml

@@ -2,7 +2,7 @@ name: Release
 
 on:
   push:
-    tags: ["v*"]
+    tags: [ "v*" ]
 
 permissions:
   contents: write
@@ -31,6 +31,14 @@ jobs:
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v4
 
+      - name: Set up Go
+        uses: actions/setup-go@v6
+        with:
+          go-version: '1.26.4'
+
+      - name: Build microVM artifacts (lb-agent + kernel + initramfs)
+        run: cd spinifex && make build-microvm-image
+
       - name: Build amd64 Tarball
         env:
           VERSION: ${{ github.ref_name }}
@@ -82,7 +90,7 @@ jobs:
             spinifex/dist/spinifex-${{ github.ref_name }}-linux-arm64.tar.gz.sha256
 
   release:
-    needs: [distro-amd64, distro-arm64]
+    needs: [ distro-amd64, distro-arm64 ]
     runs-on: ubuntu-24.04
     steps:
       - name: Download amd64 Artifacts
@@ -98,7 +106,7 @@ jobs:
           path: artifacts
 
       - name: Create Draft GitHub Release
-        uses: softprops/action-gh-release@v2
+        uses: softprops/action-gh-release@v3
         with:
           draft: true
           files: |
@@ -115,7 +123,7 @@ jobs:
 
   verify:
     needs: release
-    runs-on: [self-hosted, ci-single]
+    runs-on: [ self-hosted, ci-single ]
     timeout-minutes: 25
     env:
       TOFU_LOG: /tmp/release-verify-${{ github.run_id }}.log
@@ -128,7 +136,7 @@ jobs:
       - uses: actions/create-github-app-token@v3
         id: app-token
         with:
-          app-id: ${{ secrets.APP_ID }}
+          client-id: ${{ secrets.APP_ID }}
           private-key: ${{ secrets.APP_PRIVATE_KEY }}
           repositories: mulga
 
@@ -149,40 +157,30 @@ jobs:
         working-directory: mulga/scripts/tofu-cluster
         run: |
           tofu init -input=false >> "$TOFU_LOG" 2>&1
-          tofu workspace select -or-create env3 >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/env3.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
-          ./scrub-env.sh env3 >> "$TOFU_LOG" 2>&1 || true
-          tofu apply -var-file=envs/env3.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+          tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
 
       - name: Bootstrap via Binary Install
         working-directory: mulga/scripts/tofu-cluster
         run: |
           SETUP_SH=$(find "${GITHUB_WORKSPACE}/release-assets" -name "setup.sh" | head -1)
           TARBALL=$(find "${GITHUB_WORKSPACE}/release-assets" -name "*-linux-amd64.tar.gz" | head -1)
-          ./bootstrap-install.sh env3 "$SETUP_SH" "$TARBALL"
-
-      - name: Run Smoke Test
-        working-directory: mulga/scripts/tofu-cluster
-        run: |
-          INVENTORY=$(tofu output -json inventory)
-          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
-          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
-          SSH_KEY="${SSH_KEY/#\~/$HOME}"
-          scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR \
-            -i "$SSH_KEY" \
-            smoke-test-install.sh \
-            "tf-user@${WAN_IP}:/tmp/smoke-test.sh"
-          ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR \
-            -i "$SSH_KEY" "tf-user@${WAN_IP}" \
-            'bash /tmp/smoke-test.sh'
+          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" "$SETUP_SH" "$TARBALL"
 
       - name: Promote Release
         if: success()
         env:
           GH_TOKEN: ${{ github.token }}
         run: |
-          gh release edit "$TAG" --repo "$REPO" --draft=false
-          echo "Release ${TAG} promoted to latest"
+          if [[ "$TAG" =~ -(dev|rc|alpha|beta) ]]; then
+            gh release edit "$TAG" --repo "$REPO" --draft=false --prerelease
+            echo "Release ${TAG} promoted as prerelease"
+          else
+            gh release edit "$TAG" --repo "$REPO" --draft=false
+            echo "Release ${TAG} promoted to latest"
+          fi
 
       - name: Mark Release Failed
         if: failure()
@@ -199,6 +197,154 @@ jobs:
         if: always()
         working-directory: mulga/scripts/tofu-cluster
         run: |
-          tofu workspace select -or-create env3 >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/env3.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
-          ./scrub-env.sh env3 >> "$TOFU_LOG" 2>&1 || true
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+
+  iso:
+    needs: release
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 90
+    steps:
+      - name: Clean Workspace
+        run: |
+          ROOTFS="${GITHUB_WORKSPACE}/mulga/scripts/iso-builder/.cache/rootfs"
+          # Mirror make clean's umount sequence unconditionally — lazy unmount
+          # detaches immediately even if the mount is busy, so rm -rf is safe.
+          sudo umount -l -R "${ROOTFS}" 2>/dev/null || true
+          sudo umount -l "${ROOTFS}/dev/pts" 2>/dev/null || true
+          sudo umount -l "${ROOTFS}/dev" 2>/dev/null || true
+          sudo umount -l "${ROOTFS}/sys" 2>/dev/null || true
+          sudo umount -l "${ROOTFS}/proc" 2>/dev/null || true
+          sudo rm -rf "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/create-github-app-token@v3
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      - name: Checkout Spinifex
+        uses: actions/checkout@v6
+        with:
+          path: spinifex
+          ref: ${{ github.ref }}
+
+      - name: Checkout Predastore
+        uses: actions/checkout@v6
+        with:
+          repository: mulgadc/predastore
+          path: predastore
+
+      - name: Checkout Viperblock
+        uses: actions/checkout@v6
+        with:
+          repository: mulgadc/viperblock
+          path: viperblock
+
+      - name: Checkout Mulga iso-builder
+        uses: actions/checkout@v6
+        with:
+          repository: mulgadc/mulga
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: scripts/iso-builder
+          path: mulga
+
+      - name: Clean ISO builder cache
+        run: sudo make -C mulga/scripts/iso-builder clean
+
+      - name: Set up Go
+        uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Configure Go workspace
+        run: go work init ./spinifex ./predastore ./viperblock
+
+      - name: Install ISO builder dependencies
+        run: |
+          sudo apt-get update -qq
+          sudo apt-get install -y \
+            debootstrap squashfs-tools xorriso grub-pc-bin grub-efi-amd64-bin \
+            grub-common mtools busybox-static ovmf zstd \
+            nbdkit nbdkit-plugin-dev pkg-config gcc
+
+      - name: Build spinifex binaries
+        run: cd spinifex && make build
+
+      - name: Build viperblock
+        run: cd viperblock && make build
+
+      - name: Build release ISO
+        id: build-iso
+        env:
+          VERSION: ${{ github.ref_name }}
+        run: |
+          sudo make -C mulga/scripts/iso-builder release \
+            VERSION="${VERSION}" \
+            SPINIFEX_SRC="${GITHUB_WORKSPACE}/spinifex" \
+            VIPERBLOCK_SRC="${GITHUB_WORKSPACE}/viperblock" \
+            SPINIFEX_BIN="${GITHUB_WORKSPACE}/spinifex/bin/spx" \
+            INSTALLER_BIN="${GITHUB_WORKSPACE}/spinifex/bin/spinifex-installer"
+
+      - name: Wait for verify
+        id: wait-verify
+        if: steps.build-iso.outcome == 'success'
+        continue-on-error: true
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          echo "Waiting for verify job to complete..."
+          while true; do
+            CONCLUSION=$(gh api \
+              "repos/${{ github.repository }}/actions/runs/${{ github.run_id }}/jobs" \
+              --jq '.jobs[] | select(.name == "verify") | .conclusion // empty')
+            if [ -n "$CONCLUSION" ]; then
+              echo "Verify conclusion: $CONCLUSION"
+              [ "$CONCLUSION" = "success" ] && exit 0
+              exit 1
+            fi
+            echo "Verify still running, retrying in 30s..."
+            sleep 30
+          done
+
+      - name: Ensure AWS CLI is installed
+        if: steps.build-iso.outcome == 'success' && steps.wait-verify.outcome == 'success'
+        run: |
+          if ! command -v aws &>/dev/null; then
+            curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o /tmp/awscliv2.zip
+            unzip -q /tmp/awscliv2.zip -d /tmp/aws-install
+            sudo /tmp/aws-install/aws/install
+            rm -rf /tmp/awscliv2.zip /tmp/aws-install
+          fi
+
+      - name: Upload ISO to Cloudflare R2
+        id: upload
+        if: steps.build-iso.outcome == 'success' && steps.wait-verify.outcome == 'success'
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.CF_R2_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.CF_R2_SECRET_ACCESS_KEY }}
+          CF_R2_ENDPOINT_URL: ${{ secrets.CF_R2_ENDPOINT_URL }}
+          VERSION: ${{ github.ref_name }}
+        run: |
+          aws --endpoint-url="${CF_R2_ENDPOINT_URL}" \
+            s3 cp "mulga/scripts/iso-builder/dist/spinifex-${VERSION}.iso" \
+                  "s3://spinifex-iso/spinifex-${VERSION}.iso"
+          aws --endpoint-url="${CF_R2_ENDPOINT_URL}" \
+            s3 cp "mulga/scripts/iso-builder/dist/spinifex-${VERSION}.iso.sha256" \
+                  "s3://spinifex-iso/spinifex-${VERSION}.iso.sha256"
+
+      - name: Mark ISO failed
+        if: always() && (steps.build-iso.outcome == 'failure' || steps.upload.outcome == 'failure')
+        env:
+          GH_TOKEN: ${{ github.token }}
+          TAG: ${{ github.ref_name }}
+          REPO: ${{ github.repository }}
+        run: |
+          EXISTING_BODY=$(gh release view "$TAG" --repo "$REPO" --json body -q .body)
+          NOTES_FILE=$(mktemp)
+          printf 'ISO build/upload failed — see workflow run for details.\n\n%s' "$EXISTING_BODY" > "$NOTES_FILE"
+          gh release edit "$TAG" --repo "$REPO" --notes-file "$NOTES_FILE"
+          rm -f "$NOTES_FILE"

.github/workflows/spinifex.yml

@@ -1,4 +1,4 @@
-name: Spinifex Build and Test
+name: Build and Test
 
 on:
   push:
@@ -17,9 +17,12 @@ on:
       - dev
   workflow_dispatch:
 
+permissions:
+  contents: read
+
 jobs:
   build_and_test:
-    name: Spinifex Build and Test
+    name: Build and Test
     runs-on: ubuntu-24.04
 
     steps:
@@ -33,12 +36,14 @@ jobs:
         uses: actions/checkout@v6
         with:
           repository: mulgadc/viperblock
+          ref: dev
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v6
         with:
           repository: mulgadc/predastore
+          ref: dev
           path: predastore
 
       - name: Setup Go
@@ -56,11 +61,14 @@ jobs:
       - name: Run Tests with Coverage
         run: make -C spinifex test-cover
 
+      - name: Run Action Tests
+        run: make -C spinifex test-actions
+
       - name: Check Diff Coverage
         run: cd spinifex && scripts/diff-coverage.sh coverage.out --base HEAD~1
 
   race_detection:
-    name: Spinifex Race Detection
+    name: Race Detection
     runs-on: ubuntu-24.04
 
     steps:
@@ -73,12 +81,14 @@ jobs:
         uses: actions/checkout@v6
         with:
           repository: mulgadc/viperblock
+          ref: dev
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v6
         with:
           repository: mulgadc/predastore
+          ref: dev
           path: predastore
 
       - name: Setup Go
@@ -94,7 +104,7 @@ jobs:
         run: make -C spinifex test-race
 
   lint_and_security:
-    name: Spinifex Lint and Security
+    name: Lint and Security
     runs-on: ubuntu-24.04
 
     steps:
@@ -107,12 +117,14 @@ jobs:
         uses: actions/checkout@v6
         with:
           repository: mulgadc/viperblock
+          ref: dev
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v6
         with:
           repository: mulgadc/predastore
+          ref: dev
           path: predastore
 
       - name: Setup Go
@@ -127,7 +139,7 @@ jobs:
       - name: Lint
         uses: golangci/golangci-lint-action@v9
         with:
-          version: v2.11.3
+          version: v2.12.2
           working-directory: spinifex
 
       - name: Govulncheck

.github/workflows/ui.yml

@@ -17,6 +17,9 @@ on:
       - .github/workflows/ui.yml
   workflow_dispatch:
 
+permissions:
+  contents: read
+
 jobs:
   ui_build_and_test:
     name: UI Build and Test
@@ -32,7 +35,7 @@ jobs:
           sparse-checkout: spinifex/services/spinifexui/frontend
 
       - name: Setup pnpm
-        uses: pnpm/action-setup@v5
+        uses: pnpm/action-setup@v6
         with:
           package_json_file: spinifex/services/spinifexui/frontend/package.json
 
@@ -46,8 +49,8 @@ jobs:
       - name: Install Dependencies
         run: pnpm install --frozen-lockfile
 
-      - name: Run Check
-        run: pnpm check
+      - name: Run Lint
+        run: pnpm lint
 
       - name: Run Tests with Coverage
         run: pnpm test:coverage

.gitignore

@@ -7,6 +7,9 @@
 !example.tfvars
 scripts/iac/proxmox/clusters/
 
+# Generated SSH private keys from terraform examples (e.g. local_file.nginx_pem)
+*.pem
+
 # Go workspace
 go.work
 go.work.sum
@@ -14,6 +17,8 @@ go.work.sum
 # Build outputs
 bin/
 dist/
+build/microvm/vmlinuz
+build/microvm/initramfs.cpio.gz
 !spinifex/services/spinifexui/frontend/dist/
 tmp/
 *.test
@@ -28,6 +33,7 @@ Thumbs.db
 
 # Environment
 .env
+.venv/
 
 # Test reports
 *-report.txt

.golangci.yml

@@ -22,6 +22,7 @@ linters:
     - errname # checks that sentinel errors are prefixed with Err, error types suffixed with Error
     - exptostd # detects golang.org/x/exp/ functions replaceable by stdlib
     - fatcontext # detects nested contexts in loops
+    - forcetypeassert # flags unchecked type assertions that can panic
     - gocheckcompilerdirectives # validates go compiler directive comments (//go:)
     - goprintffuncname # checks that printf-like functions are named with f at the end
     - gosec # inspects source code for security problems
@@ -29,11 +30,14 @@ linters:
     - ineffassign # detects assignments to existing variables that are not used
     - makezero # finds slice declarations with non-zero initial length followed by append
     - mirror # catches wrong mirror patterns in bytes/strings usage
+    - musttag # checks structs passed to marshal/unmarshal have proper tags
     - modernize # modernizes code to use newer Go idioms (go fix analyzers)
     - nakedret # finds naked returns in functions
     - nilerr # catches code that returns nil even after checking err != nil
     - nilnesserr # catches returning a different nil error variable after err != nil check
     - nolintlint # reports ill-formed or insufficient nolint directives
+    - nosprintfhostport # flags fmt.Sprintf host:port — use net.JoinHostPort for IPv6 safety
+    - perfsprint # detects fmt.Sprintf calls replaceable with faster alternatives
     - predeclared # catches shadowing of Go builtins (len, cap, error, etc.)
     - reassign # checks that package variables are not reassigned
     - recvcheck # checks for receiver type consistency (pointer vs value)
@@ -91,6 +95,12 @@ linters:
         - newexpr # don't replace aws.String() with new() — AWS SDK idiom
         - stringsbuilder # don't replace small string concatenations with strings.Builder
 
+    perfsprint:
+      concat-loop: false
+      err-error: false # don't flag err.Error()
+      errorf: false # don't convert fmt.Errorf to errors.New
+      strconcat: false
+
     nakedret:
       max-func-lines: 0
 
@@ -124,5 +134,3 @@ linters:
         linters: [gosec] # subprocess execution — legitimate in VM orchestration
       - text: "G304"
         linters: [gosec] # file path from variable — necessary for config loading
-      - text: "G402"
-        linters: [gosec] # TLS config — we handle TLS ourselves

Makefile

@@ -34,6 +34,9 @@ else
 endif
 
 # Quiet-mode filters (active when QUIET=1, set by preflight via recursive make)
+# Note: grep pipelines use PIPESTATUS[0] so the exit status of `go test`
+# propagates through the filter — otherwise a test failure is swallowed by
+# grep's own (success) exit code and preflight prints "passed" on red.
 ifdef QUIET
   _Q     = @
   _COVQ  = 2>&1 | { grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' | grep -v 'coverage: 0\.0%' || true; }; exit $${PIPESTATUS[0]}
@@ -46,8 +49,7 @@ else
   _SECQ  = 2>&1 | tee
 endif
 
-build:
-	$(MAKE) go_build
+build: go_build build-installer build-lb-agent
 
 # Build spinifex-ui frontend (requires pnpm)
 build-ui:
@@ -61,20 +63,54 @@ LDFLAGS := -s -w -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=$(VERSI
 
 go_build:
 	@echo -e "\n....Building $(GO_PROJECT_NAME)"
-	go build $(GO_BUILD_MOD) -ldflags "$(LDFLAGS)" -o ./bin/$(GO_PROJECT_NAME) cmd/spinifex/main.go
+	GOFIPS140=v1.0.0 go build $(GO_BUILD_MOD) -ldflags "$(LDFLAGS)" -o ./bin/$(GO_PROJECT_NAME) cmd/spinifex/main.go
 
-build-alb-agent:
-	@echo -e "\n....Building alb-agent (static)"
-	CGO_ENABLED=0 go build -ldflags "-s -w" -o ./bin/alb-agent cmd/alb-agent/main.go
+build-installer:
+	@echo -e "\n....Building spinifex-installer"
+	GOFIPS140=v1.0.0 go build -ldflags "-s -w" -o ./bin/spinifex-installer cmd/installer/main.go
 
-build-system-image: ## Build a system image from manifest (use IMAGE=alb)
+build-lb-agent:
+	@echo -e "\n....Building lb-agent (static)"
+	CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -ldflags "-s -w" -o ./bin/lb-agent cmd/lb-agent/main.go
+
+build-system-image: ## Build a system image from manifest (use IMAGE=lb or IMAGE=eks-server)
 ifndef IMAGE
-	$(error IMAGE is required. Usage: make build-system-image IMAGE=alb)
+	$(error IMAGE is required. Usage: make build-system-image IMAGE=lb)
 endif
-	./scripts/build-system-image.sh scripts/images/$(IMAGE).conf
+	@if [ -f scripts/images/$(IMAGE).conf ]; then \
+		./scripts/build-system-image.sh scripts/images/$(IMAGE).conf; \
+	elif [ -f scripts/images/$(IMAGE)/manifest.conf ]; then \
+		./scripts/build-system-image.sh scripts/images/$(IMAGE)/manifest.conf; \
+	else \
+		echo "ERROR: no manifest at scripts/images/$(IMAGE).conf or scripts/images/$(IMAGE)/manifest.conf"; \
+		exit 1; \
+	fi
 
-build-alb-image: ## Build ALB Alpine image
-	$(MAKE) build-system-image IMAGE=alb
+build-eks-server-image: ## Build the eks-server AMI (K3s control-plane variant)
+	$(MAKE) build-system-image IMAGE=eks-server
+
+build-eks-agent-image: ## Build the eks-agent AMI (K3s worker variant)
+	$(MAKE) build-system-image IMAGE=eks-agent
+
+MICROVM_OUT_DIR := build/microvm
+MICROVM_ARTIFACTS := $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz
+MICROVM_INPUTS := scripts/build-microvm-image.sh $(MICROVM_OUT_DIR)/init.sh $(MICROVM_OUT_DIR)/inittab bin/lb-agent
+
+# Grouped target — script writes both files in one run.
+$(MICROVM_ARTIFACTS) &: $(MICROVM_INPUTS)
+	./scripts/build-microvm-image.sh
+
+# Only triggers when bin/lb-agent is missing; preserves the artifact's mtime so
+# build-microvm-image stays correctly stale-aware.
+bin/lb-agent:
+	$(MAKE) build-lb-agent
+
+build-microvm-image: $(MICROVM_ARTIFACTS) ## Build microVM kernel + initramfs (incremental — skips when up to date)
+
+install-microvm: $(MICROVM_ARTIFACTS) ## Install microVM artifacts to /usr/share/spinifex/microvm/
+	sudo install -d /usr/share/spinifex/microvm
+	sudo install -m 0644 $(MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/vmlinuz
+	sudo install -m 0644 $(MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/initramfs.cpio.gz
 
 go_run:
 	@echo -e "\n....Running $(GO_PROJECT_NAME)...."
@@ -83,9 +119,23 @@ go_run:
 # Preflight — runs the same checks as GitHub Actions (lint + vuln + tests).
 # Use this before committing to catch CI failures locally.
 preflight:
-	@$(MAKE) --no-print-directory QUIET=1 lint govulncheck test-cover diff-coverage test-race
+	@$(MAKE) --no-print-directory QUIET=1 manifest-check lint govulncheck test-cover diff-coverage test-race test-harness
 	@echo -e "\n ✅ Preflight passed — safe to commit."
 
+# E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
+# default `go test ./spinifex/...`. Runs with mocked AWS clients — no
+# infrastructure required, safe to run in CI without a cluster.
+test-harness:
+	@echo -e "\n....Running e2e harness unit tests...."
+	$(_Q)LOG_IGNORE=1 go test -tags=e2e -timeout 60s ./tests/e2e/harness/... $(_RACEQ)
+
+# Validate docs/service-interfaces.yaml. Schema check + cross-reference
+# of services/suites/fixtures + on-disk path existence. Subject content
+# vs source is enforced separately in Bead 5 drift lint.
+manifest-check:
+	@echo -e "\n....Checking service-interfaces.yaml...."
+	@go run ./tests/e2e/manifest-check/cmd/manifest-check -repo-root . -manifest docs/service-interfaces.yaml
+
 # Run unit tests
 test:
 	@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
@@ -103,6 +153,12 @@ test-race:
 	@echo -e "\n....Running tests with race detector for $(GO_PROJECT_NAME)...."
 	$(_Q)LOG_IGNORE=1 go test -race -timeout 300s ./spinifex/... $(_RACEQ)
 
+# Unit tests for in-repo GitHub Actions (e.g. .github/actions/e2e-analyze).
+# Kept out of `test-cover` so coverage % isn't diluted by CI-only tooling.
+test-actions:
+	@echo -e "\n....Running action tests...."
+	LOG_IGNORE=1 go test -timeout 60s ./.github/actions/...
+
 # Check that new/changed code meets coverage threshold (runs tests first)
 diff-coverage: test-cover
 	@QUIET=$(QUIET) scripts/diff-coverage.sh $(COVERPROFILE)
@@ -116,6 +172,26 @@ run:
 	$(MAKE) go_build
 	$(MAKE) go_run
 
+# Fast iteration: build + install binary + restart all services.
+# Microvm artifacts are reinstalled when they already exist on disk — the rule's
+# input timestamps drive a rebuild only if anything actually changed. On a fresh
+# checkout (no build/microvm/vmlinuz yet) the install step is skipped; run
+# `make install-microvm` explicitly the first time.
+deploy: build
+	sudo install -m 755 bin/spx /usr/local/bin/spx
+	@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ]; then \
+		$(MAKE) install-microvm; \
+	else \
+		echo "[deploy] microvm artifacts absent — run 'make install-microvm' for first-time setup"; \
+	fi
+	sudo systemctl daemon-reload
+	sudo systemctl restart spinifex.target
+
+# Re-run setup.sh after changing systemd units, helper scripts, or logrotate config.
+# Not needed for code-only changes — use deploy for those.
+reinstall:
+	scripts/dev-install.sh
+
 clean:
 	rm -f ./bin/$(GO_PROJECT_NAME)
 	rm -rf spinifex/services/spinifexui/frontend/dist
@@ -127,14 +203,15 @@ install-system:
 		-o Dpkg::Options::="--force-confdef" \
 		-o Dpkg::Options::="--force-confold" \
 		nbdkit nbdkit-plugin-dev pkg-config $(QEMU_PACKAGES) qemu-utils qemu-kvm \
+		ovmf qemu-efi-aarch64 \
 		libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
 		iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
 		ovn-central ovn-host openvswitch-switch dhcpcd-base
 
 install-go:
-	@echo -e "\n....Installing Go 1.26.1 for $(ARCH) ($(GO_ARCH))...."
+	@echo -e "\n....Installing Go 1.26.4 for $(ARCH) ($(GO_ARCH))...."
 	@if [ ! -d "/usr/local/go" ]; then \
-		curl -L https://go.dev/dl/go1.26.1.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
+		curl -L https://go.dev/dl/go1.26.4.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
 	else \
 		echo "Go already installed in /usr/local/go"; \
 	fi
@@ -189,6 +266,14 @@ distro-amd64:
 		-f build/Dockerfile.distro \
 		--output type=local,dest=dist/amd64/ \
 		../
+	@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ] && [ -f $(MICROVM_OUT_DIR)/initramfs.cpio.gz ]; then \
+		echo "[distro-amd64] staging microvm artifacts into tarball"; \
+		mkdir -p dist/amd64/microvm; \
+		cp $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz dist/amd64/microvm/; \
+	else \
+		echo "[distro-amd64] WARNING: microvm artifacts missing — tarball will not include them"; \
+		echo "[distro-amd64]          run 'make build-microvm-image' before 'make distro-amd64'"; \
+	fi
 	tar -czf dist/spinifex-$(VERSION)-linux-amd64.tar.gz -C dist/amd64 .
 	sha256sum dist/spinifex-$(VERSION)-linux-amd64.tar.gz > dist/spinifex-$(VERSION)-linux-amd64.tar.gz.sha256
 
@@ -207,7 +292,55 @@ distro-arm64:
 distro-clean:
 	rm -rf dist/
 
-.PHONY: build build-ui build-alb-agent build-system-image build-alb-image go_build go_run preflight test test-cover test-race diff-coverage bench run clean \
+# Ansible dev lifecycle (experimental, parallel to dev-*.sh / reset-dev-env.sh).
+# See scripts/ansible/README.md and docs/development/improvements/ansible-dev-lifecycle.md.
+#
+# Variable overrides: pass EXTRA_VARS="key=val key2=val2" to forward as
+# `--extra-vars` to ansible-playbook. Plain `-e` on the make command line is
+# make's own flag and does NOT reach ansible.
+#   make ansible-dev-install EXTRA_VARS="spinifex_external_pool_start=192.168.1.90 spinifex_external_pool_end=192.168.1.99"
+EXTRA_VARS ?=
+_ANSIBLE_EXTRA = $(if $(strip $(EXTRA_VARS)),--extra-vars "$(EXTRA_VARS)",)
+
+ansible-dev-preflight:
+	cd scripts/ansible && ansible-playbook playbooks/dev-preflight.yml $(_ANSIBLE_EXTRA)
+
+ansible-dev-teardown:
+	cd scripts/ansible && ansible-playbook playbooks/dev-teardown.yml $(_ANSIBLE_EXTRA)
+
+ansible-dev-install:
+	cd scripts/ansible && ansible-playbook playbooks/dev-install.yml $(_ANSIBLE_EXTRA)
+
+ansible-dev-reset:
+	cd scripts/ansible && ansible-playbook playbooks/dev-reset.yml $(_ANSIBLE_EXTRA)
+
+ansible-dev-deploy:
+	cd scripts/ansible && ansible-playbook playbooks/dev-deploy.yml $(_ANSIBLE_EXTRA)
+
+ansible-dev-status:
+	cd scripts/ansible && ansible-playbook playbooks/dev-status.yml
+
+ansible-dev-logs:
+	cd scripts/ansible && ansible-playbook playbooks/dev-logs.yml
+
+# Snapshot / restore require an explicit -e snapshot_name=<name>. The
+# wrapper passes ANSIBLE_EXTRA through so devs can run
+# `make ansible-dev-snapshot ANSIBLE_EXTRA='-e snapshot_name=before-merge'`.
+ansible-dev-snapshot:
+	cd scripts/ansible && ansible-playbook playbooks/dev-snapshot.yml $(ANSIBLE_EXTRA)
+
+ansible-dev-restore:
+	cd scripts/ansible && ansible-playbook playbooks/dev-restore.yml $(ANSIBLE_EXTRA)
+
+ansible-dev-version:
+	cd scripts/ansible && ansible-playbook playbooks/dev-version.yml
+
+ansible-dev-vpc:
+	cd scripts/ansible && ansible-playbook playbooks/dev-vpc.yml $(ANSIBLE_EXTRA)
+
+.PHONY: build build-ui build-installer build-lb-agent build-system-image build-eks-server-image build-eks-agent-image build-microvm-image install-microvm go_build go_run preflight test test-cover test-race diff-coverage bench run test-actions test-harness manifest-check diff-coverage bench run \
+	deploy reinstall clean \
 	install-system install-go install-aws quickinstall \
 	lint fix govulncheck \
-	distro distro-amd64 distro-arm64 distro-clean
+	distro distro-amd64 distro-arm64 distro-clean \
+	ansible-dev-preflight ansible-dev-teardown ansible-dev-install ansible-dev-reset ansible-dev-deploy ansible-dev-status ansible-dev-logs ansible-dev-snapshot ansible-dev-restore ansible-dev-version ansible-dev-vpc

README.md

@@ -1,25 +1,54 @@
+<p align="center">
+  <img src=".github/assets/banner.svg" alt="Spinifex — AWS-Compatible Infrastructure Stack" width="900">
+</p>
+
+<p align="center">
+  <a href="https://go.dev"><img src="https://img.shields.io/badge/Go-1.26+-00ADD8?style=flat-square&logo=go&logoColor=white" alt="Go"></a>
+  <a href="LICENSE"><img src="https://img.shields.io/badge/License-AGPL--3.0-3fb950?style=flat-square" alt="License"></a>
+  <a href="https://docs.mulgadc.com"><img src="https://img.shields.io/badge/Docs-docs.mulgadc.com-DC2626?style=flat-square" alt="Docs"></a>
+</p>
+
+<p align="center">
+  <a href="#what-is-spinifex">What is Spinifex?</a> ·
+  <a href="#the-platform">Platform</a> ·
+  <a href="#core-components">Components</a> ·
+  <a href="#architecture-at-a-glance">Architecture</a> ·
+  <a href="#installation">Installation</a> ·
+  <a href="https://docs.mulgadc.com">Docs</a>
+</p>
+
+---
+
 # Spinifex: An Open Source AWS-Compatible Stack for Bare-Metal, Edge, and On-Prem Deployments
 
-(Developer Preview) - Spinifex developed by [Mulga Defense Corporation](https://mulgadc.com/) is an open source infrastructure platform that brings the core services of AWS—like EC2, VPC, EBS, and S3—to environments where running in the cloud isn't an option. Whether you're deploying to edge sites, private data-centers, or need to operate in low-connectivity or highly contested environments, Spinifex gives you AWS-style workflows on your own hardware.
+Spinifex developed by [Mulga Defense Corporation](https://mulgadc.com) is an open source infrastructure platform that brings the core services of AWS—like EC2, VPC, EBS, and S3—to environments where running in the cloud isn't an option. Whether you're deploying to edge sites, private data-centers, or need to operate in low-connectivity or highly contested environments, Spinifex gives you AWS-style workflows on your own hardware.
 
 ## What is Spinifex?
 
 Spinifex replicates essential AWS primitives—virtual machines, block volumes, and object storage—using lightweight, self-contained components that are easy to deploy and integrate.
 
-It’s designed for developers and operators who need:
+It's designed for developers and operators who need:
 
 - Cloud-like infrastructure without the cloud
 - Minimal dependencies, full control
 - Drop-in compatibility with tools like the AWS CLI, SDKs, and Terraform
 - A secure cloud environment you control and own the entire hardware, network and software stack
 
-You can run Spinifex on a few servers in a rack, a field site, or anywhere centralized cloud services aren’t feasible.
+You can run Spinifex on a few servers in a rack, a field site, or anywhere centralized cloud services aren't feasible.
+
+## The Platform
+
+<p align="center">
+  <img src=".github/assets/platform.svg" alt="The Mulga Platform — Hardware to Applications" width="900">
+</p>
+
+From commodity hardware up to unmodified AWS tooling, every layer is replaceable and yours to own.
 
 ## Core Components
 
 ### Spinifex (Compute Service – EC2 Alternative)
 
-Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs similar to EC2. It manages lifecycle operations like start, stop, and terminate, using QEMU’s QMP interface. Designed to be straightforward and scriptable, Spinifex lets you launch VMs using the AWS CLI, SDKs, or Terraform—without needing Kubernetes or heavyweight orchestrators. Keep in mind, you can also setup a Kubernetes environment using Spinifex with underlying instances.
+Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs similar to EC2. It manages lifecycle operations like start, stop, and terminate, using QEMU's QMP interface. Designed to be straightforward and scriptable, Spinifex lets you launch VMs using the AWS CLI, SDKs, or Terraform—without needing Kubernetes or heavyweight orchestrators. Keep in mind, you can also setup a Kubernetes environment using Spinifex with underlying instances.
 
 - EC2-like VM management on bare metal
 - Launches with cloud-init metadata support
@@ -27,7 +56,7 @@ Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs
 
 ### Viperblock (Block Storage – EBS Alternative)
 
-[Viperblock](https://github.com/mulgadc/viperblock) is a high-performance, WAL-backed block storage service that replicates volumes across multiple nodes. It’s built for reliability and speed, with support for snapshots, recovery, and direct connection to QEMU instances using NBD or virtio-blk.
+[Viperblock](https://github.com/mulgadc/viperblock) is a high-performance, WAL-backed block storage service that replicates volumes across multiple nodes. It's built for reliability and speed, with support for snapshots, recovery, and direct connection to QEMU instances using NBD or virtio-blk.
 
 - Fast, durable virtual disks
 - Replication for resilience
@@ -43,29 +72,52 @@ Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs
 - Multipart uploads, streaming reads/writes
 - Data redundancy with Reed-Solomon encoding
 
+## Architecture at a Glance
+
+<p align="center">
+  <img src=".github/assets/architecture.svg" alt="Spinifex message-driven architecture — AWS Gateway, NATS bus, service daemons" width="900">
+</p>
+
+Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless — scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Deep dive: **[`docs/DESIGN.md`](docs/DESIGN.md)**.
+
 ## Key Features
 
 AWS-Compatible Interfaces – Provision infrastructure with awscli, Terraform, or SDKs you already use.
 
-- Designed for Control – Run on your own terms, whether that’s in a datacenter, remote site, or sensitive environment.
+- Designed for Control – Run on your own terms, whether that's in a datacenter, remote site, or sensitive environment.
 - Minimal Dependencies – Each service is standalone and avoids complex orchestration layers.
 - Works Offline – No reliance on centralized cloud services or external networks.
 - Open Source – Licensed under AGPL 3.0. Fork it, extend it, or deploy it as-is.
 
 ## Installation
 
-Installation requires an Ubuntu / Debian system. See the detailed documentation at [https://docs.mulgadc.com](https://docs.mulgadc.com) for maintaining and installing Spinifex.
+Installation requires an Ubuntu / Debian system. See the detailed documentation at [docs.mulgadc.com](https://docs.mulgadc.com) for maintaining and installing Spinifex.
+
+### Bare Metal ISO
+
+The recommended installation is a [bootable x86 installer](https://iso.mulgadc.com/spinifex.iso) for bare-metal hardware.
+
+```bash
+curl -fLO https://iso.mulgadc.com/spinifex.iso
+```
+
+Follow the [USB install guide](https://docs.mulgadc.com/docs/install-usb) to write the ISO to USB and install on your hardware. The install guide walks through the full process.
+
 
 ### Single Node Install
 
-The installation is straightforward to set up and running on a single node for testing purposes.
+The installation is straightforward to set up and running on a single node for testing purposes. Debian 13 is currently supported, additional Linux distributions are on the immediate roadmap.
+
+>*Prerequisite:* Linux bridge for networking.
+
+Spinifex requires a Linux bridge configured on the host for VM networking. See the [single-node install guide](https://docs.mulgadc.com/docs/install#prerequisites) prerequisites for setup details.
 
 ```bash
-curl https://install.mulgadc.com | bash
+curl -fsSL https://install.mulgadc.com | bash
 
 sudo /usr/local/share/spinifex/setup-ovn.sh --management
 
-sudo spx admin init
+sudo spx admin init --node node1 --nodes 1
 
 sudo systemctl start spinifex.target
 
@@ -76,7 +128,7 @@ aws ec2 describe-instance-types
 
 ### Development Setup
 
-For a complete development environment see the [Source Install](https://docs.mulgadc.com/docs/source-install/) documentation
+For a complete development environment see the [Source Install](https://docs.mulgadc.com/docs/install-source) documentation
 
 ### Component Repositories
 
@@ -87,6 +139,26 @@ Spinifex coordinates these independent components:
 
 Each component can be developed independently. See component-specific documentation for focused development guides.
 
+## Spinifex UI
+
+Spinifex ships with a built-in web console — an optional alternative to the AWS CLI, SDKs, and Terraform. If you're familiar with the AWS Management Console, the Spinifex UI fills the same role: a browser-based view of your instances, volumes, buckets, VPCs, and IAM resources, without leaving your own network.
+
+<p align="center">
+  <img src=".github/assets/spinifex-ui.jpg" alt="Spinifex web console — dashboard view" width="900">
+</p>
+
+The console is served by each node on port `3000` over TLS, and becomes available as soon as `spinifex.target` is up:
+
+```bash
+open https://YOUR_NODE_IP:3000
+```
+
+- **Same API, different surface.** Every action in the UI is the same AWS SigV4 call the CLI makes — so RBAC, audit trails, and IAM policies apply uniformly.
+- **Single sign-on against your AWS credentials.** Log in with the access keys from `~/.aws/credentials` on the node where Spinifex is installed — no separate user database.
+- **Self-hosted, works offline.** The UI is embedded in the Spinifex binary and served from the node itself. No external CDN, no analytics calls, no cloud dependency.
+
+For the full walkthrough — first-time TLS certificate trust, login, and feature tour — see [**Launching the Web UI**](https://docs.mulgadc.com/docs/setting-up-your-cluster#7-launching-the-web-ui) in the cluster setup guide.
+
 ## Development Philosophy
 
 ### Built by Engineers, For Engineers
@@ -122,4 +194,4 @@ This hybrid approach ensures Spinifex benefits from both proven engineering expe
 
 ## License
 
-Spinifex is open source under the [GNU Affero General Public License v3.0](LICENSE). You're free to use, modify, and deploy it—anywhere you need reliable infrastructure without depending on centralized cloud platforms.
+Spinifex is open source under the [GNU Affero General Public License v3.0](LICENSE). You're free to use, modify, and deploy it anywhere you need reliable infrastructure without depending on centralized cloud platforms.

build/Dockerfile.distro

@@ -30,18 +30,19 @@ COPY viperblock/ ./viperblock/
 
 # Build spinifex binary (spx)
 RUN COMMIT=$(cd spinifex && git rev-parse --short HEAD 2>/dev/null || echo "unknown") && \
-    cd spinifex && go build -ldflags "-s -w \
+    cd spinifex && GOFIPS140=v1.0.0 go build -ldflags "-s -w \
     -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=${VERSION} \
     -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Commit=${COMMIT}" \
     -o /out/spx cmd/spinifex/main.go
 
 # Build nbdkit viperblock plugin (.so)
-RUN cd viperblock && go build -o /out/nbdkit-viperblock-plugin.so -buildmode=c-shared nbd/viperblock.go
+RUN cd viperblock && GOFIPS140=v1.0.0 go build -o /out/nbdkit-viperblock-plugin.so -buildmode=c-shared nbd/viperblock.go
 
 # Copy setup scripts and systemd units
 RUN cp spinifex/scripts/setup.sh /out/setup.sh 2>/dev/null || true
 RUN cp spinifex/scripts/setup-ovn.sh /out/setup-ovn.sh
 RUN mkdir -p /out/systemd && cp spinifex/build/systemd/* /out/systemd/
+RUN mkdir -p /out/scripts && cp spinifex/build/scripts/* /out/scripts/
 RUN cp spinifex/build/logrotate/spinifex /out/logrotate-spinifex
 
 FROM scratch AS export

build/microvm/init.sh

@@ -0,0 +1,56 @@
+#!/bin/sh
+mount -t proc  proc  /proc
+mount -t sysfs sys   /sys
+mount -t devtmpfs dev /dev
+
+# virtio_mmio is built-in (CONFIG_VIRTIO_MMIO=y) so virtio-net devices are
+# already enumerated by the time PID 1 runs — no modprobe needed.
+# qemu_fw_cfg and virtio_net are =m in stock Alpine linux-virt, so load them.
+modprobe qemu_fw_cfg 2>/dev/null || modprobe fw_cfg_sysfs 2>/dev/null || true
+modprobe virtio_net 2>/dev/null || true
+
+for i in 1 2 3 4 5; do
+    [ -f /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw ] && break
+    sleep 0.02
+done
+
+if [ ! -f /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw ]; then
+    echo "[init] FATAL: fw_cfg netcfg not available" >&2
+    exit 1
+fi
+
+. /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw
+
+for n in 0 1 2 3 4 5; do
+    eval mac=\$NIC${n}_MAC
+    [ -z "$mac" ] && continue
+
+    iface=""
+    for attempt in 1 2 3 4 5; do
+        iface=$(ip -o link | awk -v m="$mac" 'tolower($0) ~ tolower(m) {print $2}' | tr -d :)
+        [ -n "$iface" ] && break
+        sleep 0.02
+    done
+
+    if [ -z "$iface" ]; then
+        echo "[init] WARNING: no interface found for MAC $mac (NIC$n), skipping" >&2
+        continue
+    fi
+
+    eval cidr=\$NIC${n}_CIDR
+    eval gw=\$NIC${n}_GW
+    eval default=\$NIC${n}_DEFAULT
+    eval rdst=\$NIC${n}_ROUTE_DST
+    eval rvia=\$NIC${n}_ROUTE_VIA
+
+    ip link set "$iface" up
+    [ -n "$cidr" ] && ip addr add "$cidr" dev "$iface"
+    [ "$default" = "1" ] && [ -n "$gw" ] && ip route add default via "$gw" dev "$iface"
+    [ -n "$rdst" ] && [ -n "$rvia" ] && ip route add "$rdst" via "$rvia" dev "$iface"
+done
+
+cp /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/lb-agent-env/raw /etc/conf.d/lb-agent
+mkdir -p /etc/ssl/certs
+cp /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/ca-cert/raw /etc/ssl/certs/ca-certificates.crt
+
+exec /sbin/init

build/microvm/inittab

@@ -0,0 +1,4 @@
+::sysinit:/etc/rc.local
+::respawn:/bin/sh -c 'set -a; . /etc/conf.d/lb-agent; set +a; exec /usr/local/bin/lb-agent'
+::ctrlaltdel:/sbin/reboot
+::shutdown:/bin/umount -a -r

build/scripts/predastore-start.sh

@@ -0,0 +1,30 @@
+#!/bin/sh
+# Helper script for spinifex-predastore.service
+# Detects host, port, backend, and node-id from spinifex.toml
+# then execs spx. Falls back to single-node defaults if no config found.
+
+CONF=/etc/spinifex/spinifex.toml
+BIND="0.0.0.0:8443"
+# -1 = co-located mode (all [[db]] peers in one process). Single-node spinifex
+# templates list 3 DB peers on 127.0.0.1; only -1 can serve that topology.
+# Multi-node spinifex.toml emits node_id = N (N >= 1) which overrides below.
+# predastore rejects NODE_ID=0 since d3a02f0 (strconv.Atoi-of-garbage guard).
+NODE_ID="-1"
+
+if [ -f "$CONF" ]; then
+    H=$(awk -F'"' '/\[nodes\..*\.predastore\]/{f=1} f&&/^host/{print $2;exit}' "$CONF")
+    if [ -n "$H" ]; then
+        BIND="$H"
+    fi
+
+    N=$(awk -F'= *' '/node_id/{gsub(/ /,"",$2);print $2;exit}' "$CONF")
+    if [ -n "$N" ]; then
+        NODE_ID="$N"
+    fi
+fi
+
+export SPINIFEX_PREDASTORE_HOST="${BIND%%:*}"
+export SPINIFEX_PREDASTORE_PORT="${BIND##*:}"
+export SPINIFEX_PREDASTORE_NODE_ID="$NODE_ID"
+
+exec /usr/local/bin/spx service predastore start

build/scripts/wait-for-nats.sh

@@ -0,0 +1,37 @@
+#!/bin/sh
+# Wait for NATS to be reachable before starting a dependent service.
+# Reads the NATS address from spinifex.toml [nodes.*.nats] section,
+# falls back to nats.conf, then to 127.0.0.1:4222.
+
+CONF=/etc/spinifex/spinifex.toml
+NATS_CONF=/etc/spinifex/nats/nats.conf
+NATS_HOST=""
+NATS_PORT="4222"
+TIMEOUT=30
+
+# Try spinifex.toml first
+if [ -f "$CONF" ]; then
+    NATS_BIND=$(awk -F'"' '/\[nodes\..*\.nats\]/{f=1} f&&/^host/{print $2;exit}' "$CONF")
+    if [ -n "$NATS_BIND" ]; then
+        NATS_HOST="${NATS_BIND%%:*}"
+        NATS_PORT="${NATS_BIND##*:}"
+    fi
+fi
+
+# Fall back to nats.conf
+if [ -z "$NATS_HOST" ] && [ -f "$NATS_CONF" ]; then
+    NATS_HOST=$(grep "^listen:" "$NATS_CONF" | sed 's/listen: *//;s/:.*//')
+fi
+
+# Final fallback
+NATS_HOST="${NATS_HOST:-127.0.0.1}"
+
+for i in $(seq 1 "$TIMEOUT"); do
+    if nc -z "$NATS_HOST" "$NATS_PORT" 2>/dev/null; then
+        exit 0
+    fi
+    sleep 1
+done
+
+echo "Timed out waiting for NATS at $NATS_HOST:$NATS_PORT" >&2
+exit 1

build/systemd/regenerate-ssh-host-keys.service

@@ -0,0 +1,12 @@
+[Unit]
+Description=Regenerate SSH host keys
+ConditionPathExists=!/etc/ssh/ssh_host_ed25519_key
+Before=ssh.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/ssh-keygen -A
+RemainAfterExit=yes
+
+[Install]
+WantedBy=multi-user.target

build/systemd/spinifex-awsgw.service

@@ -2,21 +2,53 @@
 Description=Spinifex AWS Gateway
 PartOf=spinifex.target
 After=spinifex-nats.service spinifex-daemon.service
-Requires=spinifex-nats.service
+Wants=spinifex-nats.service
 
 [Service]
 Type=simple
-User=spinifex
+User=spinifex-gw
 Group=spinifex
-ExecStartPre=/bin/sh -c 'for i in $(seq 1 30); do nc -z 127.0.0.1 4222 && exit 0; sleep 1; done; exit 1'
+ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
 ExecStart=/usr/local/bin/spx service awsgw start
 Restart=on-failure
 RestartSec=5
+
 OOMScoreAdjust=-500
 Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
-Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/
+Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/awsgw/
 Environment=SPINIFEX_AWSGW_TLS_CERT=/etc/spinifex/server.pem
 Environment=SPINIFEX_AWSGW_TLS_KEY=/etc/spinifex/server.key
 
+# Filesystem access
+NoNewPrivileges=yes
+ProtectSystem=strict
+PrivateDevices=yes
+ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/master.key /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
+ReadWritePaths=/var/lib/spinifex/awsgw /var/log/spinifex
+
+# Hardening
+ProtectHome=yes
+PrivateTmp=yes
+RestrictSUIDSGID=yes
+LockPersonality=yes
+RestrictRealtime=yes
+RestrictNamespaces=yes
+ProtectClock=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+ProtectControlGroups=yes
+ProtectProc=invisible
+SystemCallArchitectures=native
+MemoryDenyWriteExecute=yes
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
+CapabilityBoundingSet=
+ProtectHostname=yes
+KeyringMode=private
+RemoveIPC=yes
+LimitCORE=0
+LimitNOFILE=65536
+LimitNPROC=4096
+
 [Install]
 WantedBy=spinifex.target

build/systemd/spinifex-daemon.service

@@ -1,21 +1,105 @@
 [Unit]
 Description=Spinifex Daemon (VM Orchestration)
 PartOf=spinifex.target
-After=spinifex-nats.service spinifex-viperblock.service
-Requires=spinifex-nats.service
+After=spinifex-nats.service spinifex-viperblock.service openvswitch-switch.service ovn-controller.service openvswitch-ipsec.service
+# Wants=openvswitch-ipsec.service so systemd brings the IPsec userspace
+# daemon up before spinifex flips other_config:ipsec_encapsulation=true.
+# Without that daemon, OVS holds the cert pointers unused and strongSwan
+# never gets IKE config — tunnel traffic is encrypted with no SAs and
+# silently dropped. Wants= (not Requires=) keeps daemon bootable on
+# single-node / lab hosts where the package is intentionally absent.
+Wants=openvswitch-ipsec.service
+# No Wants=spinifex-nats.service: Wants= reactivates a stopped unit during
+# `systemctl restart spinifex-daemon`, which would mask the standalone-startup
+# path the Tier 1 DDIL Scenario B exercises. spinifex.target Wants= covers
+# cold boot.
 
 [Service]
 Type=simple
-User=spinifex
+User=spinifex-daemon
 Group=spinifex
-ExecStartPre=/bin/sh -c 'for i in $(seq 1 30); do nc -z 127.0.0.1 4222 && exit 0; sleep 1; done; exit 1'
+# DDIL Tier 1 (siv-17): no ExecStartPre wait-for-nats. The daemon starts in
+# standalone mode and reconnects to NATS in the background — process-exiting
+# on a missing NATS would kill the local VM management plane.
+#
+# Opt-in strict-startup (§1d-strict, siv-44): set SPINIFEX_REQUIRE_NATS=1 to
+# restore pre-DDIL fail-fast UX. First NATS connect uses a ~30s bounded retry
+# and the daemon aborts on timeout. Intended for dev/test/single-node deploys
+# where silent NATS misconfig is more confusing than helpful. Do NOT set this
+# in the shipped unit — it re-introduces the SPOF that Tier 1 removed.
 ExecStart=/usr/local/bin/spx service spinifex start
 Restart=on-failure
 RestartSec=5
+# DDIL Tier 1: KillMode=process so systemd only signals the
+# daemon's main PID, not the cgroup. QEMU children stay alive across
+# `systemctl restart spinifex-daemon` and the new daemon reattaches to them
+# via the persisted local state file + pidfile + QMP socket. With the
+# default control-group mode, restart would terminate every guest VM —
+# defeating Scenario B's "daemon restart preserves running instances".
+KillMode=process
+# 15s upper bound on the SIGTERM grace window. The new SIGTERM handler
+# skips VM teardown (it now just persists state and exits), so 15s is
+# plenty of slack for the cluster-manager + NATS unsubscribe path.
+TimeoutStopSec=15
 OOMScoreAdjust=-500
 Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
 Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex/
 Environment=SPINIFEX_WAL_DIR=/var/lib/spinifex/spinifex/
+# DDIL Tier 1: QEMU pidfiles, QMP sockets, and the
+# shared qemu.log all resolve via utils.pidPath() — which falls back
+# to os.TempDir() (=/tmp) when $HOME/spinifex doesn't exist. With
+# PrivateTmp=yes, each daemon start gets a fresh /tmp namespace, so
+# pidfiles written by QEMU (running in the OLD daemon's namespace)
+# are invisible to the new daemon. The /local/instances handler then
+# reports PID=0 and Scenario B fails its "live QEMU PID" assertion.
+# Pin XDG_RUNTIME_DIR to the host-wide tmpfs at /run/spinifex (created
+# by tmpfiles.d, already in ReadWritePaths) so pidPath() returns a
+# stable, namespace-independent path that both daemon incarnations and
+# QEMU children resolve identically.
+Environment=XDG_RUNTIME_DIR=/run/spinifex
+
+# Filesystem access — NoNewPrivileges omitted (daemon uses sudo for ip, ovs-vsctl, dhcpcd)
+# CAP_SYS_ADMIN: required for GPU passthrough sysfs operations (vfio group management).
+# CAP_DAC_OVERRIDE: sysfs driver bind/unbind paths (/sys/bus/pci/drivers/vfio-pci/bind,
+# unbind) are root:root 0200; non-root processes need CAP_DAC_OVERRIDE to open them.
+AmbientCapabilities=CAP_SYS_ADMIN CAP_DAC_OVERRIDE
+ProtectSystem=full
+PrivateDevices=no
+DevicePolicy=closed
+DeviceAllow=/dev/kvm rw
+DeviceAllow=/dev/net/tun rw
+DeviceAllow=char-vfio rw
+DeviceAllow=/dev/vfio/vfio rw
+ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
+ReadWritePaths=/var/lib/spinifex/spinifex /var/log/spinifex /run/spinifex /run/spinifex/nbd
+
+# Hardening
+ProtectHome=yes
+PrivateTmp=yes
+RestrictSUIDSGID=yes
+LockPersonality=yes
+RestrictRealtime=yes
+RestrictNamespaces=yes
+ProtectClock=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+ProtectControlGroups=yes
+ProtectProc=invisible
+SystemCallArchitectures=native
+MemoryDenyWriteExecute=yes
+# DHCP lease acquisition moved to spinifex-vpcd (see dhcp-allocate-error.md);
+# the daemon no longer needs AF_PACKET or the softened kernel protections.
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
+ProtectHostname=yes
+KeyringMode=private
+RemoveIPC=yes
+LimitCORE=0
+LimitNOFILE=65536
+LimitNPROC=4096
+# VFIO GPU passthrough requires locking all guest RAM into physical memory.
+# Without this, QEMU fails with RLIMIT_MEMLOCK exceeded when pinning guest pages.
+LimitMEMLOCK=infinity
 
 [Install]
 WantedBy=spinifex.target

build/systemd/spinifex-nats.service

@@ -5,7 +5,7 @@ Before=spinifex-predastore.service spinifex-daemon.service spinifex-awsgw.servic
 
 [Service]
 Type=simple
-User=spinifex
+User=spinifex-nats
 Group=spinifex
 ExecStart=/usr/local/bin/spx service nats start
 Restart=on-failure
@@ -13,5 +13,36 @@ RestartSec=5
 OOMScoreAdjust=-500
 Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/nats/nats.conf
 
+# Filesystem access
+NoNewPrivileges=yes
+ProtectSystem=strict
+PrivateDevices=yes
+ReadOnlyPaths=/etc/spinifex/nats /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
+ReadWritePaths=/var/lib/spinifex/nats /var/log/spinifex
+
+# Hardening
+ProtectHome=yes
+PrivateTmp=yes
+RestrictSUIDSGID=yes
+LockPersonality=yes
+RestrictRealtime=yes
+RestrictNamespaces=yes
+ProtectClock=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+ProtectControlGroups=yes
+ProtectProc=invisible
+SystemCallArchitectures=native
+MemoryDenyWriteExecute=yes
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
+CapabilityBoundingSet=
+ProtectHostname=yes
+KeyringMode=private
+RemoveIPC=yes
+LimitCORE=0
+LimitNOFILE=65536
+LimitNPROC=4096
+
 [Install]
 WantedBy=spinifex.target

build/systemd/spinifex-predastore.service

@@ -2,14 +2,14 @@
 Description=Spinifex Predastore (S3-compatible Object Storage)
 PartOf=spinifex.target
 After=spinifex-nats.service
-Requires=spinifex-nats.service
+Wants=spinifex-nats.service
 
 [Service]
 Type=simple
-User=spinifex
+User=spinifex-storage
 Group=spinifex
-ExecStartPre=/bin/sh -c 'for i in $(seq 1 30); do nc -z 127.0.0.1 4222 && exit 0; sleep 1; done; exit 1'
-ExecStart=/usr/local/bin/spx service predastore start
+ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
+ExecStart=/var/lib/spinifex/predastore-start.sh
 Restart=on-failure
 RestartSec=5
 OOMScoreAdjust=-500
@@ -17,6 +17,38 @@ Environment=SPINIFEX_PREDASTORE_CONFIG_PATH=/etc/spinifex/predastore/predastore.
 Environment=SPINIFEX_PREDASTORE_BASE_PATH=/var/lib/spinifex/predastore/
 Environment=SPINIFEX_PREDASTORE_TLS_CERT=/etc/spinifex/server.pem
 Environment=SPINIFEX_PREDASTORE_TLS_KEY=/etc/spinifex/server.key
+Environment=SPINIFEX_PREDASTORE_ENCRYPTION_KEY_FILE=/etc/spinifex/predastore/encryption.key
+
+# Filesystem access
+NoNewPrivileges=yes
+ProtectSystem=strict
+PrivateDevices=yes
+ReadOnlyPaths=/etc/spinifex/predastore /etc/spinifex/spinifex.toml /etc/spinifex/master.key /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
+ReadWritePaths=/var/lib/spinifex/predastore /var/log/spinifex
+
+# Hardening
+ProtectHome=yes
+PrivateTmp=yes
+RestrictSUIDSGID=yes
+LockPersonality=yes
+RestrictRealtime=yes
+RestrictNamespaces=yes
+ProtectClock=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+ProtectControlGroups=yes
+ProtectProc=invisible
+SystemCallArchitectures=native
+MemoryDenyWriteExecute=yes
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
+CapabilityBoundingSet=
+ProtectHostname=yes
+KeyringMode=private
+RemoveIPC=yes
+LimitCORE=0
+LimitNOFILE=65536
+LimitNPROC=4096
 
 [Install]
 WantedBy=spinifex.target

build/systemd/spinifex-ui.service

@@ -5,12 +5,45 @@ After=spinifex-awsgw.service
 
 [Service]
 Type=simple
-User=spinifex
+User=spinifex-ui
 Group=spinifex
 ExecStart=/usr/local/bin/spx service spinifex-ui start
 Restart=on-failure
 RestartSec=5
 OOMScoreAdjust=-500
+Environment=SPINIFEX_UI_TLS_CERT=/etc/spinifex/server.pem
+Environment=SPINIFEX_UI_TLS_KEY=/etc/spinifex/server.key
+
+# Filesystem access
+NoNewPrivileges=yes
+ProtectSystem=strict
+PrivateDevices=yes
+ReadOnlyPaths=/etc/spinifex/server.pem /etc/spinifex/server.key
+ReadWritePaths=/var/log/spinifex
+
+# Hardening
+ProtectHome=yes
+PrivateTmp=yes
+RestrictSUIDSGID=yes
+LockPersonality=yes
+RestrictRealtime=yes
+RestrictNamespaces=yes
+ProtectClock=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+ProtectControlGroups=yes
+ProtectProc=invisible
+SystemCallArchitectures=native
+MemoryDenyWriteExecute=yes
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
+CapabilityBoundingSet=
+ProtectHostname=yes
+KeyringMode=private
+RemoveIPC=yes
+LimitCORE=0
+LimitNOFILE=65536
+LimitNPROC=4096
 
 [Install]
 WantedBy=spinifex.target

build/systemd/spinifex-viperblock.service

@@ -2,13 +2,13 @@
 Description=Spinifex Viperblock (Block Storage)
 PartOf=spinifex.target
 After=spinifex-nats.service spinifex-predastore.service
-Requires=spinifex-nats.service
+Wants=spinifex-nats.service
 
 [Service]
 Type=simple
-User=spinifex
+User=spinifex-viperblock
 Group=spinifex
-ExecStartPre=/bin/sh -c 'for i in $(seq 1 30); do nc -z 127.0.0.1 4222 && exit 0; sleep 1; done; exit 1'
+ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
 ExecStart=/usr/local/bin/spx service viperblock start
 Restart=on-failure
 RestartSec=5
@@ -17,5 +17,36 @@ EnvironmentFile=/etc/spinifex/systemd.env
 Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
 Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/viperblock/
 
+# Filesystem access
+NoNewPrivileges=yes
+ProtectSystem=strict
+PrivateDevices=yes
+ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/systemd.env /etc/spinifex/ca.pem
+ReadWritePaths=/var/lib/spinifex/viperblock /var/log/spinifex /run/spinifex/nbd
+
+# Hardening
+ProtectHome=yes
+PrivateTmp=yes
+RestrictSUIDSGID=yes
+LockPersonality=yes
+RestrictRealtime=yes
+RestrictNamespaces=yes
+ProtectClock=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+ProtectControlGroups=yes
+ProtectProc=invisible
+SystemCallArchitectures=native
+MemoryDenyWriteExecute=yes
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
+CapabilityBoundingSet=
+ProtectHostname=yes
+KeyringMode=private
+RemoveIPC=yes
+LimitCORE=0
+LimitNOFILE=65536
+LimitNPROC=4096
+
 [Install]
 WantedBy=spinifex.target

build/systemd/spinifex-vpcd.service

@@ -1,19 +1,59 @@
 [Unit]
 Description=Spinifex VPC Daemon (OVN Networking)
 PartOf=spinifex.target
-After=spinifex-nats.service spinifex-daemon.service
-Requires=spinifex-nats.service
+After=spinifex-nats.service spinifex-daemon.service systemd-networkd.service openvswitch-switch.service ovn-controller.service
+Wants=spinifex-nats.service ovn-controller.service
 
 [Service]
 Type=simple
-User=spinifex
+User=spinifex-vpcd
 Group=spinifex
-ExecStartPre=/bin/sh -c 'for i in $(seq 1 30); do nc -z 127.0.0.1 4222 && exit 0; sleep 1; done; exit 1'
+ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
 ExecStart=/usr/local/bin/spx service vpcd start
 Restart=on-failure
 RestartSec=5
 OOMScoreAdjust=-500
 Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
+Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/vpcd/
+
+# Filesystem access — NoNewPrivileges omitted (vpcd uses sudo for ovs-vsctl, ovn-nbctl, systemctl)
+ProtectSystem=full
+PrivateDevices=no
+ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/ca.pem
+ReadWritePaths=/var/lib/spinifex/vpcd /var/log/spinifex
+
+# Hardening
+ProtectHome=yes
+PrivateTmp=yes
+RestrictSUIDSGID=no
+LockPersonality=yes
+RestrictRealtime=yes
+RestrictNamespaces=yes
+ProtectClock=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+ProtectControlGroups=yes
+ProtectProc=invisible
+SystemCallArchitectures=native
+MemoryDenyWriteExecute=no
+# AF_PACKET: required by the in-process DHCP client (github.com/insomniacslk/dhcp
+# / nclient4) for raw L2 BOOTP on br-wan before an IP is configured on the
+# interface. See docs/development/bugs/dhcp-allocate-error.md.
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK AF_PACKET
+# CAP_NET_RAW: open AF_PACKET sockets.
+# CAP_NET_BIND_SERVICE: covers unicast RENEW to UDP :68.
+# CAP_NET_ADMIN: already used for ovs-vsctl / ip link wiring on the bridge.
+# Note, disabled for now, sudo issue with ovs
+AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID
+CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID
+NoNewPrivileges=no
+ProtectHostname=yes
+KeyringMode=private
+RemoveIPC=yes
+LimitCORE=0
+LimitNOFILE=65536
+LimitNPROC=4096
 
 [Install]
 WantedBy=spinifex.target

build/systemd/spinifex.target

@@ -2,7 +2,7 @@
 Description=Spinifex Infrastructure Platform
 After=network-online.target
 Wants=network-online.target
-Wants=spinifex-nats.service spinifex-predastore.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service
+Wants=spinifex-nats.service spinifex-predastore.service spinifex-viperblock.service spinifex-daemon.service spinifex-awsgw.service spinifex-vpcd.service spinifex-ui.service openvswitch-switch.service ovn-controller.service
 
 [Install]
 WantedBy=multi-user.target

build/udev/99-spinifex-vfio.rules

@@ -0,0 +1,5 @@
+# Allow spinifex group to access VFIO devices for GPU passthrough.
+# QEMU (launched by spinifex-daemon, group spinifex) must open /dev/vfio/N
+# to use vfio-pci passthrough devices. The kernel creates these nodes as
+# root:root 0600 by default; this rule grants group-level read-write access.
+SUBSYSTEM=="vfio", GROUP="spinifex", MODE="0660"

cmd/alb-agent/main.go

@@ -1,62 +0,0 @@
-// alb-agent runs inside ALB VMs and manages HAProxy configuration.
-//
-// It connects to NATS, subscribes to config updates for its load balancer,
-// writes HAProxy configs to disk, and reloads HAProxy on changes. It also
-// responds to health pings so the daemon can verify the agent is alive.
-//
-// Usage:
-//
-//	alb-agent --lb-id=lb-xxxxx --nats=nats://10.0.0.1:4222
-//
-// The LB ID defaults to the hostname (set by cloud-init to the LB ID).
-// The NATS URL defaults to nats://127.0.0.1:4222.
-package main
-
-import (
-	"flag"
-	"log/slog"
-	"os"
-	"os/signal"
-	"syscall"
-
-	"github.com/mulgadc/spinifex/spinifex/albagent"
-)
-
-func main() {
-	var (
-		lbID    string
-		natsURL string
-	)
-
-	hostname, _ := os.Hostname()
-
-	flag.StringVar(&lbID, "lb-id", hostname, "Load balancer ID (defaults to hostname)")
-	flag.StringVar(&natsURL, "nats", "nats://127.0.0.1:4222", "NATS server URL")
-	flag.Parse()
-
-	if lbID == "" {
-		slog.Error("--lb-id is required (or set hostname to the LB ID)")
-		os.Exit(1)
-	}
-
-	slog.Info("Starting ALB agent", "lbId", lbID, "nats", natsURL)
-
-	agent, err := albagent.New(lbID, natsURL)
-	if err != nil {
-		slog.Error("Failed to create agent", "err", err)
-		os.Exit(1)
-	}
-
-	if err := agent.Start(); err != nil {
-		slog.Error("Failed to start agent", "err", err)
-		os.Exit(1)
-	}
-
-	// Wait for shutdown signal
-	sigCh := make(chan os.Signal, 1)
-	signal.Notify(sigCh, syscall.SIGTERM, syscall.SIGINT)
-	sig := <-sigCh
-
-	slog.Info("Received signal, shutting down", "signal", sig)
-	agent.Stop()
-}

cmd/dhcptest/main.go

@@ -0,0 +1,172 @@
+// dhcptest is a diagnostic tool that runs a DHCP DORA handshake using the
+// same nclient4 library and options that spinifex-vpcd uses for gateway pool
+// leases. Run it on a node to verify DHCP works at the library level before
+// debugging the full spinifex stack.
+//
+// Usage (must run as root — needs AF_PACKET):
+//
+//	sudo dhcptest --iface=br-wan
+//	sudo dhcptest --iface=br-wan --mac=02:00:00:7c:b7:b9 --promisc
+//	sudo dhcptest --iface=br-wan --spinifex-id=gateway-wan1
+package main
+
+import (
+	"context"
+	"flag"
+	"fmt"
+	"log"
+	"net"
+	"os"
+	"os/exec"
+	"strings"
+	"time"
+
+	"github.com/insomniacslk/dhcp/dhcpv4"
+	"github.com/insomniacslk/dhcp/dhcpv4/nclient4"
+)
+
+func main() {
+	iface := flag.String("iface", "br-wan", "Interface/bridge to bind the DHCP client to")
+	macStr := flag.String("mac", "", "Hardware address for DORA (chaddr + AF_PACKET filter). If empty, uses the real interface MAC.")
+	spinifexID := flag.String("spinifex-id", "", "Generate a spinifex virtual MAC from this resource ID (e.g. 'gateway-wan1'). Overrides --mac.")
+	clientID := flag.String("client-id", "", "DHCP option 61 client identifier. Defaults to --spinifex-id or --mac value.")
+	hostname := flag.String("hostname", "", "DHCP option 12 hostname")
+	vendorClass := flag.String("vendor-class", "mulga-spinifex-gw", "DHCP option 60 vendor class identifier")
+	promisc := flag.Bool("promisc", false, "Enable promiscuous mode on the interface before the test (mirrors vpcd's setBridgePromisc)")
+	timeout := flag.Duration("timeout", 10*time.Second, "Per-attempt DHCP timeout")
+	retries := flag.Int("retries", 3, "Number of DORA attempts before giving up")
+	verbose := flag.Bool("verbose", false, "Print full DHCP packet contents")
+	flag.Parse()
+
+	// Resolve the hardware address to use.
+	var hwAddr net.HardwareAddr
+	var err error
+
+	switch {
+	case *spinifexID != "":
+		hwAddr, err = net.ParseMAC(generateMAC(*spinifexID))
+		if err != nil {
+			log.Fatalf("generateMAC(%q): %v", *spinifexID, err)
+		}
+		fmt.Printf("spinifex virtual MAC for %q: %s\n", *spinifexID, hwAddr)
+	case *macStr != "":
+		hwAddr, err = net.ParseMAC(*macStr)
+		if err != nil {
+			log.Fatalf("parse --mac %q: %v", *macStr, err)
+		}
+	default:
+		hwAddr, err = ifaceMAC(*iface)
+		if err != nil {
+			log.Fatalf("get MAC for %s: %v", *iface, err)
+		}
+		fmt.Printf("using real interface MAC: %s\n", hwAddr)
+	}
+
+	// Resolve client-id: default to spinifex-id, then MAC string.
+	cid := *clientID
+	if cid == "" {
+		if *spinifexID != "" {
+			cid = *spinifexID
+		} else {
+			cid = hwAddr.String()
+		}
+	}
+
+	fmt.Printf("iface:        %s\n", *iface)
+	fmt.Printf("hw-addr:      %s\n", hwAddr)
+	fmt.Printf("client-id:    %s\n", cid)
+	fmt.Printf("hostname:     %s\n", *hostname)
+	fmt.Printf("vendor-class: %s\n", *vendorClass)
+	fmt.Printf("promisc:      %v\n", *promisc)
+	fmt.Printf("timeout:      %s  retries: %d\n", *timeout, *retries)
+	fmt.Println()
+
+	if *promisc {
+		if err := setPromisc(*iface, true); err != nil {
+			log.Printf("WARNING: failed to set promisc on %s: %v", *iface, err)
+		} else {
+			fmt.Printf("set %s promisc ON\n\n", *iface)
+		}
+	}
+
+	// Build nclient4 options — standalone DHCP probe for upstream-router debugging.
+	opts := []nclient4.ClientOpt{
+		nclient4.WithHWAddr(hwAddr),
+		nclient4.WithTimeout(*timeout),
+		nclient4.WithRetry(*retries),
+	}
+	if *verbose {
+		opts = append(opts, nclient4.WithLogger(nclient4.ShortSummaryLogger{Printfer: log.New(os.Stderr, "[nclient4] ", 0)}))
+	}
+
+	client, err := nclient4.New(*iface, opts...)
+	if err != nil {
+		log.Fatalf("nclient4.New(%s): %v", *iface, err)
+	}
+	defer client.Close()
+
+	// Identity modifiers — client-id / hostname / vendor-class for upstream lease tagging.
+	var mods []dhcpv4.Modifier
+	if cid != "" {
+		mods = append(mods, dhcpv4.WithOption(dhcpv4.OptClientIdentifier([]byte(cid))))
+	}
+	if *hostname != "" {
+		mods = append(mods, dhcpv4.WithOption(dhcpv4.OptHostName(*hostname)))
+	}
+	if *vendorClass != "" {
+		mods = append(mods, dhcpv4.WithOption(dhcpv4.OptClassIdentifier(*vendorClass)))
+	}
+
+	fmt.Println("Sending DHCP Discover...")
+	start := time.Now()
+	ctx, cancel := context.WithTimeout(context.Background(), *timeout*time.Duration(*retries+1))
+	defer cancel()
+
+	lease, err := client.Request(ctx, mods...)
+	if err != nil {
+		fmt.Printf("\nFAILED after %s: %v\n", time.Since(start).Round(time.Millisecond), err)
+		os.Exit(1)
+	}
+
+	fmt.Printf("SUCCESS in %s\n\n", time.Since(start).Round(time.Millisecond))
+	ack := lease.ACK
+	fmt.Printf("  Your IP:     %s\n", ack.YourIPAddr)
+	fmt.Printf("  Subnet:      %s\n", ack.SubnetMask())
+	fmt.Printf("  Gateway:     %v\n", ack.Router())
+	fmt.Printf("  DNS:         %v\n", ack.DNS())
+	fmt.Printf("  Server ID:   %s\n", ack.ServerIdentifier())
+	fmt.Printf("  Lease time:  %s\n", ack.IPAddressLeaseTime(24*time.Hour))
+	if *verbose {
+		fmt.Printf("\n--- Full ACK ---\n%s\n", ack.Summary())
+	}
+}
+
+// generateMAC is the same deterministic hash used by vpcd's topology.go.
+// It produces a locally-administered MAC (02:00:00:xx:xx:xx) from a resource ID.
+func generateMAC(resourceID string) string {
+	h := uint32(0)
+	for _, c := range resourceID {
+		h = h*31 + uint32(c)
+	}
+	return fmt.Sprintf("02:00:00:%02x:%02x:%02x", (h>>16)&0xff, (h>>8)&0xff, h&0xff)
+}
+
+func ifaceMAC(name string) (net.HardwareAddr, error) {
+	ifc, err := net.InterfaceByName(name)
+	if err != nil {
+		return nil, err
+	}
+	return ifc.HardwareAddr, nil
+}
+
+func setPromisc(iface string, on bool) error {
+	state := "on"
+	if !on {
+		state = "off"
+	}
+	out, err := exec.Command("ip", "link", "set", iface, "promisc", state).CombinedOutput()
+	if err != nil {
+		return fmt.Errorf("%w: %s", err, strings.TrimSpace(string(out)))
+	}
+	return nil
+}

cmd/eks-token-webhook/main.go

@@ -0,0 +1,38 @@
+// eks-token-webhook is the K8s TokenReview webhook authenticator baked into
+// the eks-server AMI. It decodes the SigV4-presigned GetCallerIdentity URL
+// produced by `aws eks get-token`, enforces the X-K8s-Aws-Id cross-cluster
+// pin, calls Mulga STS over NATS, and resolves the principal to a TokenReview
+// response via the cluster's AccessEntry KV.
+//
+// This is a placeholder build that returns 503 Service Unavailable on every
+// path. It exists so the eks-server AMI build succeeds (INSTALL_BINARIES is
+// satisfied) and the K3s server VM boots cleanly. Real implementation lands
+// with bead mulga-cs-eks-6c.
+package main
+
+import (
+	"flag"
+	"log/slog"
+	"net/http"
+	"os"
+)
+
+func main() {
+	addr := flag.String("addr", "127.0.0.1:8443", "listen address")
+	flag.Parse()
+
+	slog.SetDefault(slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelInfo})))
+
+	mux := http.NewServeMux()
+	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Retry-After", "30")
+		w.Header().Set("Content-Type", "application/json")
+		http.Error(w, `{"error":"eks-token-webhook stub: real implementation not yet deployed"}`, http.StatusServiceUnavailable)
+	})
+
+	slog.Info("eks-token-webhook stub starting", "addr", *addr)
+	if err := http.ListenAndServe(*addr, mux); err != nil {
+		slog.Error("listen failed", "err", err)
+		os.Exit(1)
+	}
+}

cmd/installer/autoinstall/autoinstall.go

@@ -0,0 +1,355 @@
+/*
+Copyright © 2026 Mulga Defense Corporation
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <https://www.gnu.org/licenses/>.
+*/
+
+// Package autoinstall reads headless install parameters from environment
+// variables exported by spinifex-init.sh from the kernel cmdline.
+// Parameters are set in the GRUB "Headless" menu entry in grub.cfg.
+package autoinstall
+
+import (
+	"fmt"
+	"log/slog"
+	"net"
+	"os"
+	"os/exec"
+	"sort"
+	"strconv"
+	"strings"
+	"syscall"
+	"time"
+
+	"github.com/mulgadc/spinifex/cmd/installer/install"
+)
+
+// Load returns an install.Config built from SPINIFEX_* environment variables,
+// or nil if SPINIFEX_AUTO is not set to "1" (interactive mode).
+func Load() (*install.Config, error) {
+	if os.Getenv("SPINIFEX_AUTO") != "1" {
+		return nil, nil
+	}
+
+	slog.Info("autoinstall: headless mode enabled via kernel cmdline")
+
+	cfg, err := buildConfig()
+	if err != nil {
+		return nil, fmt.Errorf("autoinstall config: %w", err)
+	}
+	return cfg, nil
+}
+
+// EjectAndReboot ejects the boot device (best-effort) then reboots.
+func EjectAndReboot() {
+	srcDev, _ := os.ReadFile("/run/iso-dev")
+	if dev := strings.TrimSpace(string(srcDev)); dev != "" {
+		slog.Info("autoinstall: ejecting boot device", "device", dev)
+		_ = exec.Command("eject", dev).Run()
+	}
+
+	fmt.Println()
+	fmt.Println("Installation complete.")
+	fmt.Println("Remove the USB drive now if it was not ejected automatically.")
+	fmt.Println("Rebooting in 10 seconds...")
+	time.Sleep(10 * time.Second)
+	// Use the kernel syscall directly — the live environment runs spinifex-init
+	// as PID 1 (not systemd), so exec("reboot") fails trying to reach D-Bus.
+	if err := syscall.Reboot(syscall.LINUX_REBOOT_CMD_RESTART); err != nil {
+		slog.Error("autoinstall: reboot syscall failed", "err", err)
+	}
+}
+
+func buildConfig() (*install.Config, error) {
+	password := os.Getenv("SPINIFEX_PASSWORD")
+	if password == "" {
+		return nil, fmt.Errorf("SPINIFEX_PASSWORD is required")
+	}
+
+	hostname := os.Getenv("SPINIFEX_HOSTNAME")
+	if hostname == "" {
+		hostname = "spinifex-node"
+	}
+
+	disk, err := resolveDisk(os.Getenv("SPINIFEX_DISK"))
+	if err != nil {
+		return nil, fmt.Errorf("disk: %w", err)
+	}
+
+	wanIface, err := resolveNIC(os.Getenv("SPINIFEX_WAN_IFACE"), "")
+	if err != nil {
+		return nil, fmt.Errorf("WAN NIC: %w", err)
+	}
+
+	cfg := &install.Config{
+		Disk:         disk,
+		Hostname:     hostname,
+		RootPassword: password,
+		// Optional on the headless path — empty means no --email passed to
+		// spx admin init, which omits the operator identity from telemetry.
+		Email:        strings.TrimSpace(os.Getenv("SPINIFEX_EMAIL")),
+		WANInterface: wanIface,
+		WANDHCPMode:  strings.ToLower(os.Getenv("SPINIFEX_WAN_MODE")) != "static",
+	}
+
+	if !cfg.WANDHCPMode {
+		ip := os.Getenv("SPINIFEX_WAN_IP")
+		mask := os.Getenv("SPINIFEX_WAN_MASK")
+		gw := os.Getenv("SPINIFEX_WAN_GW")
+		if ip == "" || mask == "" || gw == "" {
+			return nil, fmt.Errorf("SPINIFEX_WAN_IP, SPINIFEX_WAN_MASK, SPINIFEX_WAN_GW required for static mode")
+		}
+		cfg.WANAddress = ip
+		cfg.WANMask = mask
+		cfg.WANGateway = gw
+		if dns := os.Getenv("SPINIFEX_WAN_DNS"); dns != "" {
+			cfg.WANDNS = strings.Split(dns, ",")
+		}
+	}
+
+	if lanIface := os.Getenv("SPINIFEX_LAN_IFACE"); lanIface != "" {
+		lan, err := resolveNIC(lanIface, wanIface)
+		if err != nil {
+			return nil, fmt.Errorf("LAN NIC: %w", err)
+		}
+		cfg.LANInterface = lan
+		cfg.LANDHCPMode = strings.ToLower(os.Getenv("SPINIFEX_LAN_MODE")) != "static"
+		if !cfg.LANDHCPMode {
+			cfg.LANAddress = os.Getenv("SPINIFEX_LAN_IP")
+			cfg.LANMask = os.Getenv("SPINIFEX_LAN_MASK")
+			if dns := os.Getenv("SPINIFEX_LAN_DNS"); dns != "" {
+				cfg.LANDNS = strings.Split(dns, ",")
+			}
+		}
+	}
+
+	if os.Getenv("SPINIFEX_GPU_PASSTHROUGH") == "1" {
+		cfg.GPUPassthrough = true
+	}
+
+	role := strings.ToLower(os.Getenv("SPINIFEX_ROLE"))
+	if role == "" {
+		role = "init"
+	}
+	cfg.ClusterRole = role
+	skipFormation := os.Getenv("SPINIFEX_SKIP_FORMATION") == "1"
+	if skipFormation {
+		cfg.SkipFormation = true
+	}
+
+	if role == "join" && !skipFormation {
+		joinAddr := os.Getenv("SPINIFEX_JOIN_ADDR")
+		if joinAddr == "" {
+			return nil, fmt.Errorf("SPINIFEX_JOIN_ADDR required when SPINIFEX_ROLE=join")
+		}
+		cfg.JoinAddr = joinAddr
+	}
+
+	return cfg, nil
+}
+
+// resolveDisk maps the spx_disk value from grub.cfg to a block device path.
+//
+// Supported values:
+//
+//	auto            — use the only non-removable disk; fail if multiple found
+//	largest         — largest non-removable disk (explicit opt-in)
+//	smallest        — smallest non-removable disk (typical OS-on-SSD pattern)
+//	nvme            — the only NVMe disk; fail if multiple found
+//	/dev/sda (etc.) — exact device path; fail if not present
+func resolveDisk(target string) (string, error) {
+	switch strings.ToLower(target) {
+	case "", "auto":
+		return singleDisk()
+	case "largest":
+		return diskBySize(true)
+	case "smallest":
+		return diskBySize(false)
+	case "nvme":
+		return nvmeDisk()
+	default:
+		if _, err := os.Stat(target); err != nil {
+			return "", fmt.Errorf("%q not found", target)
+		}
+		return target, nil
+	}
+}
+
+type diskCandidate struct {
+	dev   string
+	bytes int64
+}
+
+// nonRemovableDisks returns all non-removable, non-virtual block devices.
+func nonRemovableDisks() ([]diskCandidate, error) {
+	entries, err := os.ReadDir("/sys/block")
+	if err != nil {
+		return nil, err
+	}
+	var out []diskCandidate
+	for _, e := range entries {
+		dev := e.Name()
+		switch {
+		case strings.HasPrefix(dev, "loop"),
+			strings.HasPrefix(dev, "ram"),
+			strings.HasPrefix(dev, "dm-"),
+			strings.HasPrefix(dev, "sr"):
+			continue
+		}
+		removable, _ := os.ReadFile("/sys/block/" + dev + "/removable")
+		if strings.TrimSpace(string(removable)) == "1" {
+			continue
+		}
+		sizeRaw, _ := os.ReadFile("/sys/block/" + dev + "/size")
+		sectors, parseErr := strconv.ParseInt(strings.TrimSpace(string(sizeRaw)), 10, 64)
+		if parseErr != nil || sectors == 0 {
+			continue
+		}
+		out = append(out, diskCandidate{dev: dev, bytes: sectors * 512})
+	}
+	return out, nil
+}
+
+// diskList returns a human-readable list of candidates for error messages.
+func diskList(disks []diskCandidate) string {
+	lines := make([]string, len(disks))
+	for i, d := range disks {
+		lines[i] = fmt.Sprintf("  /dev/%s (%dG)", d.dev, d.bytes>>30)
+	}
+	return strings.Join(lines, "\n")
+}
+
+// singleDisk selects the only non-removable disk, or fails listing all found.
+func singleDisk() (string, error) {
+	disks, err := nonRemovableDisks()
+	if err != nil {
+		return "", err
+	}
+	switch len(disks) {
+	case 0:
+		return "", fmt.Errorf("no non-removable disks found")
+	case 1:
+		slog.Info("autoinstall: single disk selected", "disk", disks[0].dev, "size_gb", disks[0].bytes>>30)
+		return "/dev/" + disks[0].dev, nil
+	default:
+		return "", fmt.Errorf(
+			"multiple disks found — set SPINIFEX_DISK to one of:\n%s\n"+
+				"  largest   (largest disk)\n"+
+				"  smallest  (smallest disk)\n"+
+				"  nvme      (NVMe only)\n"+
+				"  /dev/sdX  (exact path)",
+			diskList(disks),
+		)
+	}
+}
+
+// diskBySize selects the largest (largest=true) or smallest non-removable disk.
+func diskBySize(largest bool) (string, error) {
+	disks, err := nonRemovableDisks()
+	if err != nil {
+		return "", err
+	}
+	if len(disks) == 0 {
+		return "", fmt.Errorf("no non-removable disks found")
+	}
+	sort.Slice(disks, func(i, j int) bool {
+		if largest {
+			return disks[i].bytes > disks[j].bytes
+		}
+		return disks[i].bytes < disks[j].bytes
+	})
+	label := "smallest"
+	if largest {
+		label = "largest"
+	}
+	slog.Info("autoinstall: disk selected by size", "mode", label, "disk", disks[0].dev, "size_gb", disks[0].bytes>>30)
+	return "/dev/" + disks[0].dev, nil
+}
+
+// nvmeDisk selects the only NVMe disk, or fails listing all NVMe disks found.
+func nvmeDisk() (string, error) {
+	disks, err := nonRemovableDisks()
+	if err != nil {
+		return "", err
+	}
+	var nvmes []diskCandidate
+	for _, d := range disks {
+		if strings.HasPrefix(d.dev, "nvme") {
+			nvmes = append(nvmes, d)
+		}
+	}
+	switch len(nvmes) {
+	case 0:
+		return "", fmt.Errorf("no NVMe disks found")
+	case 1:
+		slog.Info("autoinstall: NVMe disk selected", "disk", nvmes[0].dev, "size_gb", nvmes[0].bytes>>30)
+		return "/dev/" + nvmes[0].dev, nil
+	default:
+		return "", fmt.Errorf(
+			"multiple NVMe disks found — set SPINIFEX_DISK to one of:\n%s",
+			diskList(nvmes),
+		)
+	}
+}
+
+// virtualNICPrefixes lists interface name prefixes that identify non-physical
+// interfaces (bridges, tunnels, container/OVS veth pairs, etc.). These are
+// skipped when auto-selecting a NIC so we don't configure docker0 or
+// ovs-system as the management interface on a machine that previously ran
+// Docker or OVN.
+var virtualNICPrefixes = []string{
+	"docker", "veth", "virbr", "br-", "ovs-", "vxlan", "genev", "tun", "tap",
+}
+
+// resolveNIC returns the interface name to use. "auto" or empty picks the
+// first physical (non-loopback, non-virtual) interface that is not exclude.
+func resolveNIC(name, exclude string) (string, error) {
+	if name != "" && name != "auto" {
+		return name, nil
+	}
+	ifaces, err := net.Interfaces()
+	if err != nil {
+		return "", err
+	}
+	for _, iface := range ifaces {
+		if iface.Flags&net.FlagLoopback != 0 {
+			continue
+		}
+		// Require broadcast capability — filters out point-to-point tunnels.
+		if iface.Flags&net.FlagBroadcast == 0 {
+			continue
+		}
+		// Require a MAC address — virtual/tunnel interfaces have none.
+		if len(iface.HardwareAddr) == 0 {
+			continue
+		}
+		if iface.Name == exclude {
+			continue
+		}
+		// Skip known virtual interface prefixes.
+		virtual := false
+		for _, pfx := range virtualNICPrefixes {
+			if strings.HasPrefix(iface.Name, pfx) {
+				virtual = true
+				break
+			}
+		}
+		if virtual {
+			continue
+		}
+		slog.Info("autoinstall: NIC resolved", "mode", "auto", "selected", iface.Name)
+		return iface.Name, nil
+	}
+	return "", fmt.Errorf("no suitable physical NIC found (non-loopback, broadcast-capable, with MAC)")
+}

cmd/installer/autoinstall/autoinstall_test.go

@@ -0,0 +1,17 @@
+package autoinstall
+
+import (
+	"os"
+	"testing"
+)
+
+func TestLoadReturnsNilWhenNotAuto(t *testing.T) {
+	os.Unsetenv("SPINIFEX_AUTO")
+	cfg, err := Load()
+	if err != nil {
+		t.Fatalf("unexpected error: %v", err)
+	}
+	if cfg != nil {
+		t.Fatal("expected nil config when SPINIFEX_AUTO is not set")
+	}
+}

cmd/installer/branding/branding.go

@@ -0,0 +1,56 @@
+/*
+Copyright © 2026 Mulga Defense Corporation
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <https://www.gnu.org/licenses/>.
+*/
+
+// Package branding centralises Mulga/Spinifex visual identity constants used
+// by the installer TUI.
+//
+// TODO: Replace placeholder values with official Mulga/Spinifex assets once
+// the brand guide and logo files are located.
+package branding
+
+import "github.com/charmbracelet/lipgloss"
+
+// Logo is the ASCII art rendered on the installer welcome screen.
+// TODO: Replace with official Spinifex/Mulga ASCII logo.
+const Logo = `
+  ███████╗██████╗ ██╗███╗   ██╗██╗███████╗███████╗██╗  ██╗
+  ██╔════╝██╔══██╗██║████╗  ██║██║██╔════╝██╔════╝╚██╗██╔╝
+  ███████╗██████╔╝██║██╔██╗ ██║██║█████╗  █████╗   ╚███╔╝
+  ╚════██║██╔═══╝ ██║██║╚██╗██║██║██╔══╝  ██╔══╝   ██╔██╗
+  ███████║██║     ██║██║ ╚████║██║██║     ███████╗██╔╝ ██╗
+  ╚══════╝╚═╝     ╚═╝╚═╝  ╚═══╝╚═╝╚═╝     ╚══════╝╚═╝  ╚═╝`
+
+// Subtitle is displayed beneath the logo on the welcome screen.
+const Subtitle = "Bare-Metal Node Provisioner"
+
+// Publisher is shown in the welcome screen footer.
+const Publisher = "Mulga Defense Corporation"
+
+// Mulga color palette.
+// TODO: Replace with official hex values from the Mulga brand guide.
+const (
+	ColorPrimary    = lipgloss.Color("#00A3E0") // Mulga blue (placeholder)
+	ColorAccent     = lipgloss.Color("#FF6B00") // Mulga orange (placeholder)
+	ColorBackground = lipgloss.Color("#0D1117") // Dark background
+	ColorSurface    = lipgloss.Color("#161B22") // Slightly lighter surface
+	ColorText       = lipgloss.Color("#E6EDF3") // Light text
+	ColorMuted      = lipgloss.Color("#8B949E") // Muted / secondary text
+	ColorSuccess    = lipgloss.Color("#3FB950") // Green
+	ColorWarning    = lipgloss.Color("#D29922") // Amber
+	ColorError      = lipgloss.Color("#F85149") // Red
+	ColorBorder     = lipgloss.Color("#30363D") // Border
+)

cmd/installer/firstboot/firstboot.go

@@ -0,0 +1,266 @@
+/*
+Copyright © 2026 Mulga Defense Corporation
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <https://www.gnu.org/licenses/>.
+*/
+
+// Package firstboot writes the oneshot systemd service and configuration that
+// completes Spinifex provisioning on the first real boot after installation.
+package firstboot
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"strings"
+
+	"github.com/mulgadc/spinifex/cmd/installer/systemd"
+)
+
+// Config holds the values the firstboot service needs to configure the node.
+type Config struct {
+	Hostname string
+	// EncapIP is the Geneve tunnel IP for OVN. Set to the LAN bridge IP when a
+	// dedicated LAN NIC is present, otherwise the WAN bridge IP. Empty when DHCP
+	// is used — setup-ovn.sh auto-detects the IP from the default route in that case.
+	EncapIP string
+	// ClusterRole is "init" or "join".
+	ClusterRole string
+	// JoinAddr is host:port of the primary node, only used when ClusterRole is "join".
+	JoinAddr string
+	// Email is the operator email collected by the TUI or SPINIFEX_EMAIL on
+	// the headless path. Passed to `spx admin init --email=<value>` when set;
+	// omitted entirely when empty.
+	Email string
+	// GPUPassthrough enables VFIO GPU passthrough by passing --gpu-passthrough
+	// to `spx admin init`, which writes gpu_passthrough = true in the daemon config.
+	GPUPassthrough bool
+	// InstallCallback, when non-empty, is curled once at the end of firstboot
+	// after the success marker is written. Generic phone-home hook for
+	// provisioning controllers (PXE/MAAS-style flows). Best-effort: non-2xx
+	// and network failures are swallowed so the install is not gated on
+	// controller reachability.
+	InstallCallback string
+	// SkipFormation skips spx admin init/join. Used when a provisioning
+	// controller (e.g. bm-bootstrap.sh) owns cluster formation and will call
+	// spx admin init/join itself with the correct multi-node parameters.
+	SkipFormation bool
+}
+
+// Write drops the firstboot script and systemd unit into root, which should be
+// the path of the installed system's root filesystem (e.g. /mnt/spinifex-install).
+func Write(root string, cfg Config) error {
+	if err := writeScript(root, cfg); err != nil {
+		return fmt.Errorf("firstboot script: %w", err)
+	}
+	if err := systemd.WriteFirstbootUnit(root); err != nil {
+		return fmt.Errorf("firstboot unit: %w", err)
+	}
+	if err := systemd.EnableUnit(root, "spinifex-firstboot.service"); err != nil {
+		return err
+	}
+	// Banner unit is written but NOT enabled here. firstboot enables it
+	// (along with spinifex.target) at the end of its run, so on the first
+	// boot the only spinifex unit running is firstboot itself — no implicit
+	// pull of spinifex.target via the banner's Wants= relationship, which
+	// was the root cause of the boot-time race against firstboot.
+	if err := systemd.WriteBannerUnit(root); err != nil {
+		return fmt.Errorf("banner unit: %w", err)
+	}
+	return systemd.WriteGettyDropIn(root)
+}
+
+func writeScript(root string, cfg Config) error {
+	clusterCmd := buildClusterCmd(cfg)
+
+	// --encap-ip is optional: when DHCP is used the IP is unknown at install time
+	// and setup-ovn.sh auto-detects it from the default route at boot.
+	setupOVN := "/usr/local/bin/setup-ovn.sh --management"
+	if cfg.EncapIP != "" {
+		setupOVN += fmt.Sprintf(" --encap-ip=%s", cfg.EncapIP)
+	}
+
+	callbackBlock := ""
+	if cfg.InstallCallback != "" {
+		callbackBlock = fmt.Sprintf(
+			"\ncurl -fsS --max-time 10 --retry 2 --retry-delay 2 -o /dev/null %s || true\n",
+			shellEscapeSingle(cfg.InstallCallback),
+		)
+	}
+
+	script := fmt.Sprintf(`#!/bin/bash
+# Spinifex firstboot — runs once after ISO installation. On success, writes
+# /var/lib/spinifex/.firstboot-done and the systemd unit's
+# ConditionPathExists=! prevents re-execution on subsequent boots. A partial
+# run leaves no marker, so the next reboot retries from the top — safe
+# because every step below is idempotent (hostnamectl, setup-ovn.sh, spx
+# admin init are all "set if not set"; systemctl enable on an already-enabled
+# unit is a no-op).
+set -euo pipefail
+
+DONE_MARKER=/var/lib/spinifex/.firstboot-done
+
+# Idempotency: bail early if a previous run completed successfully. The unit
+# also has ConditionPathExists=!$DONE_MARKER so we shouldn't get here on
+# subsequent boots, but defend in depth in case the unit is re-triggered
+# manually before the operator notices the marker exists.
+if [ -f "$DONE_MARKER" ]; then
+    echo "[firstboot] already complete — skipping"
+    exit 0
+fi
+
+# Set hostname
+hostnamectl set-hostname %s
+
+# Pre-start OVS and OVN central so their databases are initialised before
+# setup-ovn.sh runs. On physical hardware, first-boot DB initialisation takes
+# longer than setup-ovn.sh's internal 15-second timeout allows. Starting them
+# here and waiting until the NB DB is ready means setup-ovn.sh sees a live DB
+# the moment it starts — no races, no timeout failures.
+systemctl start openvswitch-switch
+systemctl start ovn-central
+echo "Waiting for OVN NB DB to initialise..."
+for _i in $(seq 1 120); do
+    if ovn-nbctl --timeout=2 get-connection >/dev/null 2>&1; then
+        echo "OVN NB DB ready (${_i}s)"
+        break
+    fi
+    sleep 1
+done
+
+# Create default nameservers if DHCP server returns blank
+printf "nameserver 1.1.1.1\nnameserver 8.8.8.8\n" > /etc/resolvconf/resolv.conf.d/base
+resolvconf -u
+
+# Configure OVN networking.
+# br-wan (and br-lan if present) are Linux bridges managed by systemd-networkd
+# (declared in /etc/systemd/network/ by the installer). setup-ovn.sh
+# auto-detects br-wan as the default route device and wires it to OVS via a
+# veth pair — non-destructive, SSH-safe.
+%s
+
+# Write the banner (first time, spinifex-banner service will do this on reboot)
+/usr/local/bin/spx admin banner --boot-check
+
+# Cluster formation — capture credentials to file for display on console.
+%s 2>&1
+
+# Fix ownership of files spx admin init wrote. spx runs as root under
+# systemd, so /etc/spinifex/{spinifex.toml,master.key,ca.key,*.pem} and any
+# per-service files written under /var/lib/spinifex/* land as root:root.
+# Delegate to setup.sh's fix_file_ownership (single source of truth) — it
+# knows the per-file modes (ca.key 0600, *.pem 0644, toml/key 0640) that a
+# blunt recursive chmod would clobber.
+SETUP_STAGES=fixown /usr/local/share/spinifex/setup.sh
+# The /var/lib/spinifex/awsgw/config symlink was created at build time by
+# setup.sh's create_directories — it resolves to /etc/spinifex, so
+# {BaseDir}/config/master.key automatically points at /etc/spinifex/master.key.
+
+# Bootstrap the spinifex user's SSH directory with a generated key pair so
+# ~/.ssh exists and the operator can immediately use ssh-keygen / ssh-copy-id
+# without hitting "No such file or directory" errors documented in the setup guide.
+if [ ! -f /home/spinifex/.ssh/id_ed25519 ]; then
+    mkdir -p /home/spinifex/.ssh
+    chmod 700 /home/spinifex/.ssh
+    ssh-keygen -t ed25519 -f /home/spinifex/.ssh/id_ed25519 -N "" -C "spinifex@$(hostname)"
+    chown -R spinifex:spinifex /home/spinifex/.ssh
+    chmod 600 /home/spinifex/.ssh/id_ed25519
+    chmod 644 /home/spinifex/.ssh/id_ed25519.pub
+fi
+
+# Copy AWS credentials to the spinifex user's home directory.
+# spx admin init runs with HOME=/root (set by the systemd unit), so credentials
+# land in /root/.aws/. Copy them to the spinifex user's home so the operator
+# can use the AWS CLI without sudo.
+if [ -f /root/.aws/credentials ]; then
+    mkdir -p /home/spinifex/.aws
+    cp /root/.aws/credentials /home/spinifex/.aws/credentials
+    cp /root/.aws/config /home/spinifex/.aws/config 2>/dev/null || true
+    chown -R spinifex:spinifex /home/spinifex/.aws
+    chmod 700 /home/spinifex/.aws
+    chmod 600 /home/spinifex/.aws/credentials
+    [ -f /home/spinifex/.aws/config ] && chmod 600 /home/spinifex/.aws/config
+fi
+
+# Enable + activate spinifex.target and the banner now that all configs are
+# in place. "enable --now" creates the multi-user.target.wants/ symlinks (so
+# they start directly on every subsequent boot — no longer dependent on
+# firstboot running each time, since firstboot is condition-skipped after
+# this run via the marker) and activates them in this boot.
+# NOTE: Moved to firstinstall to enable
+#systemctl enable --now spinifex.target spinifex-banner.service
+
+# Enable services to start, on reboot
+systemctl enable spinifex.target spinifex-banner.service
+
+# Mark complete only after every step above has succeeded. Until this point,
+# any failure (set -e) leaves the marker absent and the next reboot retries
+# firstboot from the top.
+mkdir -p "$(dirname "$DONE_MARKER")"
+touch "$DONE_MARKER"
+%s
+systemctl start spinifex.target
+
+# Wait for the daemon to bring up external networking. When external_mode=pool
+# is configured the daemon creates br-ext (the OVN external bridge) during
+# startup; launching instances before it is ready results in no public IP being
+# assigned. ip link requires no root and avoids parsing daemon logs.
+if grep -q 'external_mode.*pool' /etc/spinifex/spinifex.toml 2>/dev/null; then
+    echo "[firstboot] waiting for external networking (br-ext)..."
+    for _i in $(seq 1 30); do
+        if ip link show br-ext >/dev/null 2>&1; then
+            echo "[firstboot] br-ext ready (${_i}s)"
+            break
+        fi
+        sleep 1
+    done
+    if ! ip link show br-ext >/dev/null 2>&1; then
+        echo "[firstboot] warning: br-ext not up after 30s — external networking may be delayed"
+    fi
+fi
+`, cfg.Hostname, setupOVN, clusterCmd, callbackBlock)
+
+	path := filepath.Join(root, "usr/local/bin/spinifex-firstboot.sh")
+	return os.WriteFile(path, []byte(script), 0o755)
+}
+
+func buildClusterCmd(cfg Config) string {
+	if cfg.SkipFormation {
+		return `echo "[firstboot] cluster formation skipped — provisioning controller will handle it"`
+	}
+	emailFlag := ""
+	if cfg.Email != "" {
+		// shellEscapeSingle keeps the email safe if it ever contains a
+		// character shell treats specially — belt-and-braces; the regex
+		// validator already rejects whitespace and @-chains.
+		emailFlag = " --email=" + shellEscapeSingle(cfg.Email)
+	}
+	gpuFlag := ""
+	if cfg.GPUPassthrough {
+		gpuFlag = " --gpu-passthrough"
+	}
+	switch cfg.ClusterRole {
+	case "join":
+		return fmt.Sprintf("spx admin join --node %s --host %s%s", cfg.Hostname, cfg.JoinAddr, emailFlag)
+	default:
+		return fmt.Sprintf("spx admin init --node %s --nodes 1%s%s", cfg.Hostname, emailFlag, gpuFlag)
+	}
+}
+
+// shellEscapeSingle wraps s in single quotes with any embedded single
+// quotes escaped. Minimal — we only need this because the email value is
+// interpolated into a shell script written by Write().
+func shellEscapeSingle(s string) string {
+	return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'"
+}