release: update go.mod for v1.6.0

After PR #270 the VPC-LR-wide 0.0.0.0/0 -> IGW static route is no longer
installed implicitly at attach-internet-gateway. Subnets whose effective
route table lacks an internet-bound route get an OVN drop policy at
SubnetEgressPriorityDrop (1100), which kills LB target reply traffic
before un-DNAT can fire and stalls the LB internal-client probe until
job timeout.

Add the standard AWS pattern: describe-route-tables -> create-route
0.0.0.0/0 -> IGW. This fires vpc.add-igw-route + vpc.ungate-subnet-egress,
removing the drop policy and installing a reroute policy at
SubnetEgressPriorityIGW (1000).

.github/CODEOWNERS

@@ -0,0 +1,21 @@
+# CODEOWNERS — GitHub PR review enforcement
+#
+# Format: <path pattern>  <owner> [<owner> …]
+# Order matters — last matching pattern wins.
+#
+# Set the matching branch protection rule on `dev` and `main` to
+# "Require review from Code Owners" or this file is advisory only.
+
+# Fall-through: any change needs a team review.
+*                           @mulgadc/mulgadc-engineers
+
+# Network layer is governed by ADR-0006 invariant tests.
+# Changes here must respect those clauses — see
+# spinifex/network/invariants/ and
+# docs/development/proposals/az-local-architecture/0006-spinifex-network-layer-contract.md
+/spinifex/network/          @joshsiv-mulga
+/spinifex/vpcd/             @joshsiv-mulga
+
+# Invariants suite is the executable ADR-0006 fence — any widening or
+# narrowing of the scan roots needs an explicit owner review.
+/spinifex/network/invariants/ @joshsiv-mulga

.github/actions/e2e-analyze/README.md

@@ -0,0 +1,104 @@
+# e2e-analyze
+
+Post-test failure-clustering action for the Go E2E suites.
+
+## What it does
+
+Runs after `go-junit-report` has converted each suite's stdout into
+`junit-<suite>.xml`. For every JUnit file matched by `junit-glob`:
+
+1. Parses each `<testcase>` with a `<failure>` / `<error>` child.
+2. Strips ANSI escapes, then normalises dynamic noise (UUIDs, EC2 IDs,
+   IPs, timestamps, request-ids, source line numbers) to produce a stable
+   _signature_ per failure.
+3. Picks the **earliest non-cascade** failure per suite as the likely
+   root cause. "Cascade" = failure message matches a fixture-dependency
+   marker (`must populate fix.`, `Should NOT be empty`,
+   `Expected value not to be nil`). When every failure is a cascade
+   (whole suite tripped on one upstream gap), the earliest cascade is
+   the fallback root.
+4. Buckets the remaining failures by signature and renders a markdown
+   report. The report is appended to `$GITHUB_STEP_SUMMARY` and also
+   written to `<log-dir>/analysis.md` so it travels with the uploaded
+   artifact bundle.
+
+The action **never fails the job** — exit-code semantics stay with the
+existing "Fail job if any suite failed" gate.
+
+## Inputs
+
+| Input        | Required | Description |
+|--------------|----------|-------------|
+| `junit-glob` | yes      | Shell glob for the JUnit XML files (e.g. `${ARTIFACT_DIR}/junit-*.xml`). |
+| `log-dir`    | yes      | Directory the rendered report is written into (`analysis.md`). |
+| `title`      | no       | Heading at the top of the summary section. Default `E2E failure analysis`. |
+
+## Outputs
+
+None. The summary is appended to `$GITHUB_STEP_SUMMARY`; the same
+content lands at `<log-dir>/analysis.md`.
+
+## Local invocation
+
+Outside Actions, the analyser prints to stdout instead of appending to a
+summary file:
+
+```sh
+cd spinifex/.github/actions/e2e-analyze
+go run . -junit-glob "/tmp/artifacts/junit-*.xml" -log-dir /tmp/artifacts
+```
+
+## Tests
+
+Golden-file tests live in `testdata/`. Each subdir holds a scenario:
+
+| Scenario           | Source                                    |
+|--------------------|-------------------------------------------|
+| `cascade-vpcs`     | Synthetic: one Phase 5 root + 15 cascades |
+| `single-failure`   | Real artifact from run `26037892507`      |
+| `no-failures`      | Synthetic: clean run                      |
+
+Run normally:
+
+```sh
+go test ./.github/actions/e2e-analyze/
+```
+
+When the report format changes intentionally, regenerate the snapshots:
+
+```sh
+go test ./.github/actions/e2e-analyze/ -update
+```
+
+## Per-failure bundles (Stage 2)
+
+After parsing, the action materialises a bundle directory per failure
+under `<log-dir>/analysis/NN-<TestName>/`. The report links to it from
+each entry so a reader on the Actions page can click straight through
+to the relevant log slice.
+
+Each bundle contains:
+
+| File              | Source                                                                            |
+|-------------------|-----------------------------------------------------------------------------------|
+| `failure.txt`     | Synthesised summary: test, start, duration, file-site, error line.                |
+| `journal.log`     | `spinifex-journal.log` sliced to `[StartAt − 5s, StartAt + Duration + 5s]`.       |
+| `test.log`        | `test-<suite>.log` block between `=== RUN <name>` and the matching `--- FAIL`.    |
+| `<other files>`   | Anything the harness already dropped into `<log-dir>/<TestName>/` at run-time.    |
+
+The journal slice relies on the workflow running
+`journalctl --output=short-iso` so timestamps are parseable. Lines
+without a parseable timestamp inherit the previous line's
+in-window/out-of-window classification (so multi-line records and stack
+traces stay contiguous).
+
+## Roadmap
+
+- **Stage 1 (shipped):** first-failure surfacing + cascade clustering.
+- **Stage 2 (this update):** time-window correlation — per-failure bundle
+  with sliced journal + test-log + any per-test artifacts.
+- **Stage 3:** rule-based root-cause classifier driven by
+  `spinifex/tests/e2e/.failure-rules.yaml`.
+
+See `docs/development/improvements/e2e-go-failure-analysis.md` in the
+parent mulga repo.

.github/actions/e2e-analyze/action.yml

@@ -0,0 +1,38 @@
+name: E2E Failure Analyzer
+description: >-
+  Post-test analyzer for the Go E2E suites. Parses junit-*.xml emitted by
+  go-junit-report, clusters failures by signature, picks the earliest
+  non-cascade failure per suite as the likely root cause, and appends a
+  human-readable report to $GITHUB_STEP_SUMMARY. Always exits 0 — the
+  existing "Fail job if any suite failed" gate keeps owning exit-code
+  semantics.
+
+inputs:
+  junit-glob:
+    description: Shell glob matching JUnit XML files to analyze.
+    required: true
+  log-dir:
+    description: >-
+      Artifact directory. The rendered report is also written to
+      $log-dir/analysis.md so it travels with the uploaded artifact.
+    required: true
+  title:
+    description: Heading rendered at the top of the summary section.
+    required: false
+    default: "E2E failure analysis"
+
+runs:
+  using: composite
+  steps:
+    - id: analyze
+      shell: bash
+      working-directory: ${{ github.action_path }}
+      env:
+        ANALYZE_JUNIT_GLOB: ${{ inputs.junit-glob }}
+        ANALYZE_LOG_DIR: ${{ inputs.log-dir }}
+        ANALYZE_TITLE: ${{ inputs.title }}
+      run: |
+        go run . \
+          -junit-glob "$ANALYZE_JUNIT_GLOB" \
+          -log-dir "$ANALYZE_LOG_DIR" \
+          -title "$ANALYZE_TITLE"

.github/actions/e2e-analyze/analyze.go

@@ -0,0 +1,552 @@
+// Package main implements the e2e-analyze GitHub Action.
+//
+// Stage 1 of docs/development/improvements/e2e-go-failure-analysis.md:
+// cluster failures from go-junit-report's junit-*.xml by error signature,
+// pick the earliest non-cascade failure per suite as the likely root cause,
+// and render a human-readable report.
+//
+// This file contains the pure logic (parsing, signature extraction,
+// clustering, rendering). Side-effect handling lives in main.go.
+package main
+
+import (
+	"encoding/xml"
+	"fmt"
+	"path/filepath"
+	"regexp"
+	"sort"
+	"strings"
+	"time"
+)
+
+// JUnit XML schema produced by go-junit-report v2. Stable across versions
+// we care about; only fields used downstream are decoded.
+type junitSuites struct {
+	XMLName xml.Name     `xml:"testsuites"`
+	Suites  []junitSuite `xml:"testsuite"`
+}
+
+type junitSuite struct {
+	Name      string    `xml:"name,attr"`
+	Tests     int       `xml:"tests,attr"`
+	Failures  int       `xml:"failures,attr"`
+	Errors    int       `xml:"errors,attr"`
+	Skipped   int       `xml:"skipped,attr"`
+	Timestamp string    `xml:"timestamp,attr"`
+	Time      float64   `xml:"time,attr"`
+	Cases     []junitTC `xml:"testcase"`
+}
+
+type junitTC struct {
+	Name      string        `xml:"name,attr"`
+	Classname string        `xml:"classname,attr"`
+	Time      float64       `xml:"time,attr"`
+	Failure   *junitFailure `xml:"failure"`
+	Error     *junitFailure `xml:"error"`
+	Skipped   *junitSkipped `xml:"skipped"`
+	SystemOut string        `xml:"system-out"`
+}
+
+type junitFailure struct {
+	Message string `xml:"message,attr"`
+	Type    string `xml:"type,attr"`
+	Body    string `xml:",chardata"`
+}
+
+type junitSkipped struct {
+	Message string `xml:"message,attr"`
+}
+
+// Failure is the analyzer's normalized view of one failed testcase.
+type Failure struct {
+	SuiteFile string    // e.g. "single" (derived from junit-single.xml)
+	Name      string    // e.g. "TestSingleNode/Phase5_LaunchInstance"
+	Duration  float64   // seconds
+	Order     int       // position within the suite (0-based)
+	StartAt   time.Time // estimated wall-clock start: suite.timestamp + sum(prior durations)
+	Error     string    // last meaningful error line, ANSI-stripped
+	FileHint  string    // e.g. "vpc_test.go:227", parsed from failure body
+	Cascade   bool      // signature matched a known downstream pattern
+	Signature string    // normalized error key for bucketing
+	// OffsetFromSuiteStart is sum(prior testcase durations) at the time
+	// this test started. Persisted separately from StartAt so a caller
+	// that has a more authoritative suite wall-clock origin (e.g. a
+	// `test-<suite>.start` file written by the workflow before the suite
+	// runs) can recompute StartAt without re-parsing the XML.
+	OffsetFromSuiteStart time.Duration
+	// BundlePath is the relative path (from log-dir) to the per-failure
+	// bundle written by Stage 2's WriteBundles. Empty when bundles were
+	// not generated (e.g. unit tests that call Render directly).
+	BundlePath string
+}
+
+// SuiteReport summarises one junit-*.xml file.
+type SuiteReport struct {
+	File      string // basename, e.g. "junit-single.xml"
+	Label     string // short name, e.g. "single"
+	Total     int
+	FailCount int
+	Root      *Failure    // nil if no failures
+	Cascades  []Failure   // everything else that failed in this suite
+	Buckets   [][]Failure // failures grouped by signature (root's bucket first)
+}
+
+// Report is the full analysis across every junit file.
+type Report struct {
+	Title  string
+	Suites []SuiteReport
+}
+
+// ansiSeq matches ANSI CSI escapes the harness sprinkles through log
+// output (colours, bold). Three forms in the wild:
+//   - real ESC: `\x1b[36m…`
+//   - U+FFFD replacement (XML serialiser dropped the ESC byte): `�[36m…`
+//   - bare CSI with the leading byte gone entirely: `[36m…`
+//
+// All three are stripped so colour state doesn't fork the signature bucket.
+var ansiSeq = regexp.MustCompile(
+	"\x1b\\[[0-9;]*[a-zA-Z]" + // real ESC CSI
+		"|�\\[[0-9;]*[a-zA-Z]" + // replacement-char CSI
+		"|\\[[0-9]{1,2}(?:;[0-9]{1,2})*m", // bare colour code, "mode" form only
+)
+
+// cascadeMarkers are message fragments that only appear in tests that
+// depend on a fixture field populated by an earlier phase. Their presence
+// means the failure is downstream of some other failure that happened
+// first; the report collapses them under that root cause.
+var cascadeMarkers = []string{
+	"must populate fix.",
+	"Should NOT be empty",
+	"Expected value not to be nil",
+}
+
+// noiseReplacers normalise dynamic identifiers so that "ten failures, same
+// bug" bucket together. Pattern ordering matters: longer/more-specific
+// patterns must precede the catch-alls.
+var noiseReplacers = []struct {
+	re   *regexp.Regexp
+	repl string
+}{
+	{regexp.MustCompile(`\b(vpc|subnet|sg|rtb|igw|i|ami|vol|snap|rtbassoc|key)-[0-9a-f]{8,}\b`), `${1}-<id>`},
+	{regexp.MustCompile(`\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b`), `<uuid>`},
+	{regexp.MustCompile(`\b[0-9a-f]{32,}\b`), `<hex>`},
+	{regexp.MustCompile(`\b\d{1,3}(\.\d{1,3}){3}(:\d+)?\b`), `<ip>`},
+	{regexp.MustCompile(`\b\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:?\d{2})?\b`), `<ts>`},
+	{regexp.MustCompile(`request-id:\s*\S+`), `request-id: <id>`},
+	{regexp.MustCompile(`(_test\.go):\d+`), `${1}:<line>`},
+	{regexp.MustCompile(`\s+`), ` `},
+}
+
+// fileHintRe pulls the last "foo_test.go:NNN" we see in a failure body so
+// the report can point a reader at the source line. Test files dominate the
+// hint set; harness/helper file matches go through goFileLineRe below.
+var fileHintRe = regexp.MustCompile(`([A-Za-z0-9_]+_test\.go:\d+)`)
+
+// goFileLineRe matches any "foo.go:NNN:" prefix (test or non-test). Used to
+// surface harness/helper-level error text like
+// `ec2helpers.go:164: describe-vpcs: InvalidParameterValue …` which Go test
+// emits when a non-test file calls t.Errorf.
+var goFileLineRe = regexp.MustCompile(`^[A-Za-z0-9_]+\.go:\d+:\s*(.+)$`)
+
+// testifyErrorPlaceholder is testify's "Error:" boilerplate when the
+// assertion failed because of a wrapped error. The next non-blank line in
+// the trace holds the actual error text.
+const testifyErrorPlaceholder = "Received unexpected error:"
+
+// stripANSI removes CSI escapes; leaves everything else intact.
+func stripANSI(s string) string {
+	return ansiSeq.ReplaceAllString(s, "")
+}
+
+// extractErrorLine finds the most useful single-line summary in a failure
+// body. Priority order, highest first:
+//  1. Testify "Messages:" line — user-supplied context, the most signal-rich.
+//  2. Testify "Error:" content (with one-line lookahead when the placeholder
+//     "Received unexpected error:" defers to the next line).
+//  3. Leading "foo.go:NNN: text" content line — t.Errorf from non-test code.
+//  4. Last `_test.go:NNN: text` line — bare t.Fatalf/Errorf from a test file.
+//  5. First non-empty line of the body.
+func extractErrorLine(body string) string {
+	body = stripANSI(body)
+	lines := strings.Split(body, "\n")
+
+	var (
+		msgLine      string
+		errLine      string
+		goFileLine   string
+		lastTestLine string
+	)
+	for i, raw := range lines {
+		l := strings.TrimSpace(raw)
+		if l == "" {
+			continue
+		}
+		if strings.HasPrefix(l, "Messages:") {
+			msgLine = strings.TrimSpace(strings.TrimPrefix(l, "Messages:"))
+			continue
+		}
+		if strings.HasPrefix(l, "Error:") && errLine == "" {
+			content := strings.TrimSpace(strings.TrimPrefix(l, "Error:"))
+			if content == testifyErrorPlaceholder {
+				// Look ahead for the wrapped error text on the next non-blank line.
+				for j := i + 1; j < len(lines); j++ {
+					nxt := strings.TrimSpace(lines[j])
+					if nxt == "" {
+						continue
+					}
+					if strings.HasPrefix(nxt, "Test:") || strings.HasPrefix(nxt, "Messages:") {
+						break
+					}
+					content = content + " " + nxt
+					break
+				}
+			}
+			errLine = content
+			continue
+		}
+		if m := goFileLineRe.FindStringSubmatch(l); m != nil {
+			// Keep the LAST occurrence: t.Log / harness traces emit
+			// progress lines from the same file:line pattern before the
+			// failing assertion finally fires.
+			goFileLine = strings.TrimSpace(m[1])
+			continue
+		}
+		if fileHintRe.MatchString(l) {
+			lastTestLine = l
+		}
+	}
+	switch {
+	case msgLine != "":
+		return msgLine
+	case errLine != "":
+		return errLine
+	case goFileLine != "":
+		return goFileLine
+	case lastTestLine != "":
+		// "vpc_test.go:227: Eventually: condition not met …" → trim the file prefix.
+		if i := strings.Index(lastTestLine, ": "); i > 0 {
+			return strings.TrimSpace(lastTestLine[i+2:])
+		}
+		return lastTestLine
+	}
+	// Last resort: first non-empty line of the body.
+	for _, raw := range lines {
+		l := strings.TrimSpace(raw)
+		if l != "" {
+			return l
+		}
+	}
+	return ""
+}
+
+// errorTraceRe extracts the path from testify's "Error Trace:" line — e.g.
+//
+//	Error Trace:	tests/e2e/harness/ec2helpers.go:164
+//
+// — which points at the assertion site even when it lives in helper code
+// outside *_test.go.
+var errorTraceRe = regexp.MustCompile(`Error Trace:\s*(\S+\.go:\d+)`)
+
+// extractFileHint returns the most precise pointer at the assertion site
+// we can pull from the failure body: testify Error Trace wins, otherwise
+// the last `*_test.go:NNN` line in the body.
+func extractFileHint(body string) string {
+	body = stripANSI(body)
+	if m := errorTraceRe.FindStringSubmatch(body); m != nil {
+		return m[1]
+	}
+	all := fileHintRe.FindAllString(body, -1)
+	if len(all) == 0 {
+		return ""
+	}
+	return all[len(all)-1]
+}
+
+// signature normalises an error line so different runs of the same root
+// cause bucket together. Empty signatures (e.g. parent-test placeholders
+// where the failure body is empty) bucket together under "".
+func signature(errLine string) string {
+	s := strings.ToLower(errLine)
+	for _, r := range noiseReplacers {
+		s = r.re.ReplaceAllString(s, r.repl)
+	}
+	return strings.TrimSpace(s)
+}
+
+// isCascade reports whether the error line contains any of the well-known
+// fixture-propagation markers — i.e. "this test failed because something
+// upstream of it failed first".
+func isCascade(errLine string) bool {
+	for _, m := range cascadeMarkers {
+		if strings.Contains(errLine, m) {
+			return true
+		}
+	}
+	return false
+}
+
+// computeLeafSet returns the testcase names that are leaves — i.e. no
+// other testcase name in `cases` starts with `<name>/`. go-junit-report
+// emits one <testcase> for the parent Go test AND each subtest, with the
+// parent's time = sum of its children, so summing every entry into the
+// suite's cumulative offset double-counts. Restrict accumulation to
+// leaves only.
+func computeLeafSet(cases []junitTC) map[string]bool {
+	leaf := make(map[string]bool, len(cases))
+	for _, tc := range cases {
+		leaf[tc.Name] = true
+	}
+	for _, tc := range cases {
+		// Strip the deepest path segment; any ancestor of this test is
+		// not a leaf.
+		name := tc.Name
+		for {
+			i := strings.LastIndex(name, "/")
+			if i <= 0 {
+				break
+			}
+			name = name[:i]
+			if _, ok := leaf[name]; ok {
+				leaf[name] = false
+			}
+		}
+	}
+	return leaf
+}
+
+// parseStartTime turns a junit timestamp attr (RFC3339) into a time.Time;
+// zero value if unparseable.
+func parseStartTime(s string) time.Time {
+	if s == "" {
+		return time.Time{}
+	}
+	t, err := time.Parse(time.RFC3339, s)
+	if err != nil {
+		return time.Time{}
+	}
+	return t
+}
+
+// suiteLabel turns "junit-single.xml" into "single".
+func suiteLabel(path string) string {
+	base := filepath.Base(path)
+	base = strings.TrimSuffix(base, ".xml")
+	base = strings.TrimPrefix(base, "junit-")
+	if base == "" {
+		return filepath.Base(path)
+	}
+	return base
+}
+
+// ParseFile decodes one junit-*.xml file into our normalized failure list.
+// Returns the suite label, total testcase count, and the (ordered) failures.
+func ParseFile(path string, data []byte) (SuiteReport, error) {
+	var doc junitSuites
+	if err := xml.Unmarshal(data, &doc); err != nil {
+		return SuiteReport{}, fmt.Errorf("parse %s: %w", path, err)
+	}
+
+	rep := SuiteReport{
+		File:  filepath.Base(path),
+		Label: suiteLabel(path),
+	}
+
+	order := 0
+	for _, suite := range doc.Suites {
+		start := parseStartTime(suite.Timestamp)
+		// go-junit-report emits a testcase for both the parent Go test
+		// (e.g. `TestSingleNode`) AND each subtest (`TestSingleNode/PhaseX`).
+		// The parent's `time` attribute is the aggregate of its children, so
+		// summing every testcase double-counts. Pre-compute the leaf set so
+		// only leaves contribute to the suite's running offset.
+		isLeaf := computeLeafSet(suite.Cases)
+		cumul := 0.0
+		for _, tc := range suite.Cases {
+			rep.Total++
+			leaf := isLeaf[tc.Name]
+			var body string
+			switch {
+			case tc.Failure != nil:
+				body = tc.Failure.Body
+			case tc.Error != nil:
+				body = tc.Error.Body
+			default:
+				if leaf {
+					cumul += tc.Time
+				}
+				continue
+			}
+
+			// Skip the synthetic parent-test "Failed" stub that go-junit-report
+			// emits for any Go test whose subtests failed. It has an empty body
+			// and would otherwise outrank real failures by XML order.
+			trimmed := strings.TrimSpace(stripANSI(body))
+			if trimmed == "" {
+				if leaf {
+					cumul += tc.Time
+				}
+				continue
+			}
+
+			errLine := extractErrorLine(body)
+			offset := time.Duration(cumul * float64(time.Second))
+			f := Failure{
+				SuiteFile:            rep.Label,
+				Name:                 tc.Name,
+				Duration:             tc.Time,
+				Order:                order,
+				Error:                errLine,
+				FileHint:             extractFileHint(body),
+				Cascade:              isCascade(errLine),
+				Signature:            signature(errLine),
+				OffsetFromSuiteStart: offset,
+			}
+			if !start.IsZero() {
+				f.StartAt = start.Add(offset)
+			}
+			rep.FailCount++
+			rep.Cascades = append(rep.Cascades, f) // temp staging; split below
+			order++
+			if leaf {
+				cumul += tc.Time
+			}
+		}
+	}
+
+	if rep.FailCount == 0 {
+		rep.Cascades = nil
+		return rep, nil
+	}
+
+	all := rep.Cascades
+	rep.Cascades = nil
+
+	// Root = earliest non-cascade. If every failure is a cascade (the whole
+	// suite tripped on the same upstream gap), fall back to earliest overall.
+	rootIdx := -1
+	for i, f := range all {
+		if !f.Cascade {
+			rootIdx = i
+			break
+		}
+	}
+	if rootIdx == -1 {
+		rootIdx = 0
+	}
+	root := all[rootIdx]
+	rep.Root = &root
+
+	// Bucket the remainder by signature so the report can collapse the
+	// "ten failures, same bug" pattern.
+	bucketKeys := []string{}
+	buckets := map[string][]Failure{}
+	for i, f := range all {
+		if i == rootIdx {
+			continue
+		}
+		if _, seen := buckets[f.Signature]; !seen {
+			bucketKeys = append(bucketKeys, f.Signature)
+		}
+		buckets[f.Signature] = append(buckets[f.Signature], f)
+		rep.Cascades = append(rep.Cascades, f)
+	}
+
+	// Largest bucket first; ties broken by first-seen order so output is
+	// deterministic across runs.
+	sort.SliceStable(bucketKeys, func(i, j int) bool {
+		return len(buckets[bucketKeys[i]]) > len(buckets[bucketKeys[j]])
+	})
+	for _, k := range bucketKeys {
+		rep.Buckets = append(rep.Buckets, buckets[k])
+	}
+
+	return rep, nil
+}
+
+// Render writes the markdown report. Format is documented in
+// docs/development/improvements/e2e-go-failure-analysis.md (Stage 1).
+func Render(r Report) string {
+	var b strings.Builder
+	fmt.Fprintf(&b, "## %s\n\n", r.Title)
+
+	totalFail := 0
+	for _, s := range r.Suites {
+		totalFail += s.FailCount
+	}
+	if totalFail == 0 {
+		b.WriteString("✅ No failures across any suite.\n")
+		return b.String()
+	}
+
+	for _, s := range r.Suites {
+		if s.FailCount == 0 {
+			fmt.Fprintf(&b, "### Suite `%s`: ✅ pass (%d tests)\n\n", s.Label, s.Total)
+			continue
+		}
+		fmt.Fprintf(&b, "### Suite `%s`: %d failed, 1 root cause likely\n\n", s.Label, s.FailCount)
+
+		if s.Root != nil {
+			b.WriteString("**Root cause (earliest non-cascade)**\n\n")
+			fmt.Fprintf(&b, "- Test: `%s`\n", s.Root.Name)
+			if !s.Root.StartAt.IsZero() {
+				fmt.Fprintf(&b, "- Start: %s (duration %.1fs)\n",
+					s.Root.StartAt.Format("15:04:05"), s.Root.Duration)
+			} else {
+				fmt.Fprintf(&b, "- Duration: %.1fs\n", s.Root.Duration)
+			}
+			if s.Root.Error != "" {
+				fmt.Fprintf(&b, "- Error: %s\n", oneLine(s.Root.Error))
+			}
+			if s.Root.FileHint != "" {
+				fmt.Fprintf(&b, "- File:  `%s`\n", s.Root.FileHint)
+			}
+			if s.Root.BundlePath != "" {
+				fmt.Fprintf(&b, "- Bundle: `%s/`\n", s.Root.BundlePath)
+			}
+			b.WriteString("\n")
+		}
+
+		if len(s.Buckets) > 0 {
+			fmt.Fprintf(&b, "**Cascaded failures (%d) — grouped by signature**\n\n",
+				len(s.Cascades))
+			for _, bucket := range s.Buckets {
+				sample := bucket[0]
+				label := oneLine(sample.Error)
+				if label == "" {
+					label = "(no error message)"
+				}
+				fmt.Fprintf(&b, "- %d× %s\n", len(bucket), label)
+				// List up to 5 test names per bucket; collapse the rest.
+				const cap = 5
+				for i, f := range bucket {
+					if i == cap && len(bucket) > cap {
+						fmt.Fprintf(&b, "  - … and %d more\n", len(bucket)-cap)
+						break
+					}
+					if f.BundlePath != "" {
+						fmt.Fprintf(&b, "  - `%s` → `%s/`\n", f.Name, f.BundlePath)
+					} else {
+						fmt.Fprintf(&b, "  - `%s`\n", f.Name)
+					}
+				}
+			}
+			b.WriteString("\n")
+		}
+	}
+
+	return b.String()
+}
+
+// oneLine collapses whitespace and caps length so a runaway assertion
+// message doesn't blow out the summary card width.
+func oneLine(s string) string {
+	s = strings.TrimSpace(s)
+	s = strings.ReplaceAll(s, "\n", " ")
+	s = strings.ReplaceAll(s, "\r", " ")
+	s = regexp.MustCompile(`\s+`).ReplaceAllString(s, " ")
+	const max = 160
+	if len(s) > max {
+		s = s[:max-1] + "…"
+	}
+	return s
+}

.github/actions/e2e-analyze/analyze_test.go

@@ -0,0 +1,191 @@
+package main
+
+import (
+	"flag"
+	"os"
+	"path/filepath"
+	"sort"
+	"strings"
+	"testing"
+)
+
+// updateGolden lets `go test ./.github/actions/e2e-analyze/ -update` rewrite
+// the expected analysis.md files when the rendering format changes
+// intentionally. Day-to-day, leave it unset: tests diff against the on-disk
+// snapshot to catch unintended drift in signature extraction or layout.
+var updateGolden = flag.Bool("update", false, "rewrite testdata/*/analysis.md from current output")
+
+// TestGoldenFixtures runs the full ParseFile + Render pipeline against each
+// fixture directory under testdata/ and asserts the output matches the
+// adjacent analysis.md snapshot. Each subdir is one named scenario; the
+// fixture XMLs match the same junit-*.xml glob the production action uses
+// so behaviour stays representative of CI.
+func TestGoldenFixtures(t *testing.T) {
+	entries, err := os.ReadDir("testdata")
+	if err != nil {
+		t.Fatalf("read testdata: %v", err)
+	}
+	for _, e := range entries {
+		if !e.IsDir() {
+			continue
+		}
+		name := e.Name()
+		t.Run(name, func(t *testing.T) {
+			dir := filepath.Join("testdata", name)
+			matches, err := filepath.Glob(filepath.Join(dir, "junit-*.xml"))
+			if err != nil {
+				t.Fatalf("glob: %v", err)
+			}
+			sort.Strings(matches)
+			if len(matches) == 0 {
+				t.Fatalf("no junit-*.xml fixtures in %s", dir)
+			}
+
+			rep := Report{Title: "E2E failure analysis"}
+			for _, p := range matches {
+				data, err := os.ReadFile(p)
+				if err != nil {
+					t.Fatalf("read %s: %v", p, err)
+				}
+				sr, err := ParseFile(p, data)
+				if err != nil {
+					t.Fatalf("parse %s: %v", p, err)
+				}
+				rep.Suites = append(rep.Suites, sr)
+			}
+
+			got := Render(rep)
+			goldenPath := filepath.Join(dir, "analysis.md")
+			if *updateGolden {
+				if err := os.WriteFile(goldenPath, []byte(got), 0o644); err != nil {
+					t.Fatalf("write golden: %v", err)
+				}
+				return
+			}
+			want, err := os.ReadFile(goldenPath)
+			if err != nil {
+				t.Fatalf("read golden (run with -update to seed): %v", err)
+			}
+			if string(want) != got {
+				t.Errorf("rendered output differs from %s\n--- want ---\n%s\n--- got ---\n%s",
+					goldenPath, string(want), got)
+			}
+		})
+	}
+}
+
+func TestSignatureCollapsesNoise(t *testing.T) {
+	a := signature("describe-vpcs: InvalidParameterValue request-id: abc-123 vpc-deadbeefcafe1234")
+	b := signature("describe-vpcs: InvalidParameterValue request-id: xyz-999 vpc-1111222233334444")
+	if a != b {
+		t.Errorf("signatures should collapse to same bucket:\n  a=%q\n  b=%q", a, b)
+	}
+	if !strings.Contains(a, "<id>") || !strings.Contains(a, "vpc-<id>") {
+		t.Errorf("expected normalised placeholders in signature, got %q", a)
+	}
+}
+
+func TestCascadeDetection(t *testing.T) {
+	cases := []struct {
+		in   string
+		want bool
+	}{
+		{"Phase 5 must populate fix.InstanceID", true},
+		{"Should NOT be empty", true},
+		{"Expected value not to be nil", true},
+		{"describe-vpcs: InvalidParameterValue", false},
+		{"Eventually: condition not met within 3m0s", false},
+	}
+	for _, c := range cases {
+		if got := isCascade(c.in); got != c.want {
+			t.Errorf("isCascade(%q) = %v, want %v", c.in, got, c.want)
+		}
+	}
+}
+
+func TestExtractErrorLine_MessagesWinsOverError(t *testing.T) {
+	body := `    lifecycle_test.go:28:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:28
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5a_pre_ClusterStats`
+	got := extractErrorLine(body)
+	want := "Phase 5 must populate fix.InstanceID"
+	if got != want {
+		t.Errorf("extractErrorLine = %q, want %q", got, want)
+	}
+}
+
+func TestExtractErrorLine_FallsBackToLastTestLine(t *testing.T) {
+	body := "    vpc_test.go:227: Eventually: condition not met within 3m0s: [SSH handshake never completed]"
+	got := extractErrorLine(body)
+	if !strings.HasPrefix(got, "Eventually:") {
+		t.Errorf("expected fallback to test-line content, got %q", got)
+	}
+}
+
+func TestExtractFileHint_PicksLastMatch(t *testing.T) {
+	body := "ec2helpers.go:50: setup\n    vpc_test.go:227: Eventually: condition not met"
+	got := extractFileHint(body)
+	want := "vpc_test.go:227"
+	if got != want {
+		t.Errorf("extractFileHint = %q, want %q", got, want)
+	}
+}
+
+func TestParseFile_SkipsEmptyParentFailure(t *testing.T) {
+	// The synthetic "TestSingleNode" parent gets a <failure message="Failed"/>
+	// with no body whenever any subtest fails. The analyzer must skip it so
+	// it doesn't outrank the real first failure on XML order.
+	xml := []byte(`<?xml version="1.0"?>
+<testsuites><testsuite name="" tests="2" failures="2">
+  <testcase name="TestX" time="1.0"><failure message="Failed"></failure></testcase>
+  <testcase name="TestX/RealFailure" time="0.5"><failure message="Failed"><![CDATA[
+    foo_test.go:10:
+        Messages: real assertion message
+]]></failure></testcase>
+</testsuite></testsuites>`)
+	sr, err := ParseFile("junit-x.xml", xml)
+	if err != nil {
+		t.Fatalf("ParseFile: %v", err)
+	}
+	if sr.FailCount != 1 {
+		t.Fatalf("FailCount = %d, want 1 (empty parent should be skipped)", sr.FailCount)
+	}
+	if sr.Root == nil || sr.Root.Name != "TestX/RealFailure" {
+		t.Fatalf("expected real failure as root, got %+v", sr.Root)
+	}
+}
+
+func TestParseFile_CascadeFallbackWhenAllCascades(t *testing.T) {
+	// If every failure is a cascade marker, the earliest one becomes root
+	// so the report isn't blank.
+	xml := []byte(`<?xml version="1.0"?>
+<testsuites><testsuite name="" tests="2" failures="2">
+  <testcase name="TestX/A" time="1.0"><failure message="Failed"><![CDATA[
+    Messages: Phase 5 must populate fix.InstanceID
+]]></failure></testcase>
+  <testcase name="TestX/B" time="1.0"><failure message="Failed"><![CDATA[
+    Messages: Phase 5 must populate fix.InstanceID
+]]></failure></testcase>
+</testsuite></testsuites>`)
+	sr, err := ParseFile("junit-x.xml", xml)
+	if err != nil {
+		t.Fatalf("ParseFile: %v", err)
+	}
+	if sr.Root == nil || sr.Root.Name != "TestX/A" {
+		t.Fatalf("expected TestX/A as fallback root, got %+v", sr.Root)
+	}
+}
+
+func TestRender_NoFailuresIsClean(t *testing.T) {
+	out := Render(Report{
+		Title: "E2E failure analysis",
+		Suites: []SuiteReport{
+			{File: "junit-single.xml", Label: "single", Total: 3, FailCount: 0},
+		},
+	})
+	if !strings.Contains(out, "No failures") {
+		t.Errorf("expected zero-failure banner, got:\n%s", out)
+	}
+}

.github/actions/e2e-analyze/bundle.go

@@ -0,0 +1,208 @@
+package main
+
+// Stage 2 of docs/development/improvements/e2e-go-failure-analysis.md:
+// for each failure surfaced by Stage 1, materialise a per-failure bundle
+// directory containing the failure summary, a time-window slice of the
+// daemon journal, a testcase-window slice of the suite's stdout log, and
+// any per-test artifacts the harness already dumped at run-time.
+
+import (
+	"errors"
+	"fmt"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"regexp"
+	"strings"
+	"time"
+)
+
+// bundlePad is the per-failure window padding either side of the
+// [StartAt, StartAt+Duration] testcase span. Five seconds matches the
+// plan doc: long enough to catch the daemon's pre-test setup and
+// post-test cleanup chatter, short enough to keep the slice focused.
+const bundlePad = 5 * time.Second
+
+// nameSanitiseRe replaces characters unsafe in directory names with `_`.
+// Tracks tests/e2e/harness/artifacts.go which uses the same convention
+// when the test's own dump dir is created, so we can pick its contents
+// up by path without a second translation table.
+var nameSanitiseRe = regexp.MustCompile(`[^A-Za-z0-9._-]+`)
+
+func sanitiseTestName(s string) string {
+	return nameSanitiseRe.ReplaceAllString(s, "_")
+}
+
+// ApplySuiteStartFiles overrides each failure's StartAt by reading
+// <logDir>/test-<suiteLabel>.start (RFC3339 timestamp written by the
+// workflow before the suite runs). Required because go-junit-report's
+// `timestamp` attribute on <testsuite> records the time the XML was
+// produced — i.e. after the suite has finished — so it can't be used
+// as the suite wall-clock origin for journal slicing.
+//
+// Missing or unparseable files are silently ignored: the analyzer
+// falls back to the junit timestamp and the bundle still gets created,
+// it just may have an empty (or off-window) journal slice.
+func ApplySuiteStartFiles(rep *Report, logDir string) {
+	if logDir == "" {
+		return
+	}
+	for si := range rep.Suites {
+		s := &rep.Suites[si]
+		startPath := filepath.Join(logDir, fmt.Sprintf("test-%s.start", s.Label))
+		data, err := os.ReadFile(startPath)
+		if err != nil {
+			continue
+		}
+		ts, err := time.Parse(time.RFC3339, strings.TrimSpace(string(data)))
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: parse %s: %v\n", startPath, err)
+			continue
+		}
+		if s.Root != nil {
+			s.Root.StartAt = ts.Add(s.Root.OffsetFromSuiteStart)
+		}
+		for ci := range s.Cascades {
+			s.Cascades[ci].StartAt = ts.Add(s.Cascades[ci].OffsetFromSuiteStart)
+		}
+		for bi := range s.Buckets {
+			for fi := range s.Buckets[bi] {
+				f := &s.Buckets[bi][fi]
+				f.StartAt = ts.Add(f.OffsetFromSuiteStart)
+			}
+		}
+	}
+}
+
+// WriteBundles materialises per-failure bundles under <logDir>/analysis/.
+// Mutates rep in-place so each Failure.BundlePath holds the relative
+// path the renderer will link from the summary.
+//
+// Failures with no parseable wall-clock start still get a bundle dir
+// (and the testcase-window test.log slice), but no journal slice — the
+// time window is unknown.
+//
+// Errors are non-fatal: a bundle that partially fails still gets linked
+// from the report so a reader sees what's available, and the analyzer
+// keeps printing warnings so CI logs surface the gap.
+func WriteBundles(rep *Report, logDir string) {
+	if logDir == "" {
+		return
+	}
+	base := filepath.Join(logDir, "analysis")
+	if err := os.MkdirAll(base, 0o755); err != nil {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: mkdir %s: %v\n", base, err)
+		return
+	}
+
+	for si := range rep.Suites {
+		s := &rep.Suites[si]
+		if s.Root != nil {
+			writeOneBundle(logDir, base, s.Label, s.Root)
+		}
+		for ci := range s.Cascades {
+			writeOneBundle(logDir, base, s.Label, &s.Cascades[ci])
+		}
+		// Bucket entries share storage with Cascades but live in a
+		// separate slice in SuiteReport; mirror the bundle path so the
+		// renderer's bucket loop sees it too.
+		for bi := range s.Buckets {
+			for fi := range s.Buckets[bi] {
+				f := &s.Buckets[bi][fi]
+				if bp := findBundlePath(s, f.Name); bp != "" {
+					f.BundlePath = bp
+				}
+			}
+		}
+	}
+}
+
+// findBundlePath looks up a Failure's already-written BundlePath by test
+// name within a suite. Buckets reference the same logical failure as
+// Cascades but are independent slice values, so we copy the path across
+// rather than re-deriving it.
+func findBundlePath(s *SuiteReport, testName string) string {
+	if s.Root != nil && s.Root.Name == testName {
+		return s.Root.BundlePath
+	}
+	for _, c := range s.Cascades {
+		if c.Name == testName {
+			return c.BundlePath
+		}
+	}
+	return ""
+}
+
+func writeOneBundle(logDir, base, suiteLabel string, f *Failure) {
+	name := fmt.Sprintf("%02d-%s", f.Order+1, sanitiseTestName(f.Name))
+	dir := filepath.Join(base, name)
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: mkdir %s: %v\n", dir, err)
+		return
+	}
+	f.BundlePath = filepath.Join("analysis", name)
+
+	writeFailureSummary(dir, suiteLabel, f)
+
+	if !f.StartAt.IsZero() {
+		start := f.StartAt.Add(-bundlePad)
+		end := f.StartAt.Add(time.Duration(f.Duration * float64(time.Second))).Add(bundlePad)
+		journalSrc := filepath.Join(logDir, "spinifex-journal.log")
+		if _, err := SliceJournal(journalSrc, start, end, filepath.Join(dir, "journal.log")); err != nil {
+			fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: slice journal for %s: %v\n", f.Name, err)
+		}
+	}
+
+	testLogSrc := filepath.Join(logDir, fmt.Sprintf("test-%s.log", suiteLabel))
+	if _, err := SliceTestLog(testLogSrc, f.Name, filepath.Join(dir, "test.log")); err != nil {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: slice test log for %s: %v\n", f.Name, err)
+	}
+
+	// Per-test artifact subdir written by harness.ArtifactDir lives at
+	// <logDir>/<TestName with / replaced by _>/. Copy any files into
+	// the bundle so qemu/ovs/etc. dumps already collected by the test
+	// travel alongside the analysis.
+	perTestSrc := filepath.Join(logDir, strings.ReplaceAll(f.Name, "/", "_"))
+	if entries, err := os.ReadDir(perTestSrc); err == nil {
+		for _, e := range entries {
+			if e.IsDir() {
+				continue
+			}
+			_ = copyFile(filepath.Join(perTestSrc, e.Name()), filepath.Join(dir, e.Name()))
+		}
+	} else if !os.IsNotExist(err) && !errorsIsNotDir(err) {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: read per-test dir %s: %v\n", perTestSrc, err)
+	}
+}
+
+// errorsIsNotDir handles the case where the harness wrote a file at the
+// per-test name instead of a directory — best-effort skip.
+func errorsIsNotDir(err error) bool {
+	var pe *fs.PathError
+	if !errors.As(err, &pe) {
+		return false
+	}
+	return pe.Err != nil && strings.Contains(pe.Err.Error(), "not a directory")
+}
+
+func writeFailureSummary(dir, suiteLabel string, f *Failure) {
+	var b strings.Builder
+	fmt.Fprintf(&b, "Suite:    %s\n", suiteLabel)
+	fmt.Fprintf(&b, "Test:     %s\n", f.Name)
+	if !f.StartAt.IsZero() {
+		fmt.Fprintf(&b, "Start:    %s\n", f.StartAt.Format(time.RFC3339))
+	}
+	fmt.Fprintf(&b, "Duration: %.3fs\n", f.Duration)
+	if f.FileHint != "" {
+		fmt.Fprintf(&b, "Site:     %s\n", f.FileHint)
+	}
+	if f.Cascade {
+		b.WriteString("Cascade:  yes (downstream of another failure)\n")
+	}
+	if f.Error != "" {
+		b.WriteString("\nError:\n")
+		b.WriteString(oneLine(f.Error))
+		b.WriteString("\n")
+	}
+	_ = os.WriteFile(filepath.Join(dir, "failure.txt"), []byte(b.String()), 0o644)
+}

.github/actions/e2e-analyze/bundle_test.go

@@ -0,0 +1,229 @@
+package main
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+func TestSliceJournal_KeepsOnlyWindow(t *testing.T) {
+	src := filepath.Join(t.TempDir(), "journal.log")
+	dst := filepath.Join(t.TempDir(), "out.log")
+	body := strings.Join([]string{
+		"2026-05-19T12:32:30+1000 host spinifex-daemon[1]: pre",
+		"  continuation of pre",
+		"2026-05-19T12:32:35+1000 host spinifex-daemon[1]: IN window",
+		"  multiline inside window",
+		"2026-05-19T12:32:45+1000 host spinifex-daemon[1]: post",
+		"",
+	}, "\n")
+	if err := os.WriteFile(src, []byte(body), 0o644); err != nil {
+		t.Fatalf("write src: %v", err)
+	}
+	start, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:33+10:00")
+	end, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:40+10:00")
+	n, err := SliceJournal(src, start, end, dst)
+	if err != nil {
+		t.Fatalf("SliceJournal: %v", err)
+	}
+	if n != 2 {
+		t.Fatalf("written = %d, want 2", n)
+	}
+	got, _ := os.ReadFile(dst)
+	if !strings.Contains(string(got), "IN window") || !strings.Contains(string(got), "multiline inside window") {
+		t.Errorf("missing in-window content:\n%s", got)
+	}
+	if strings.Contains(string(got), "pre") || strings.Contains(string(got), "post") {
+		t.Errorf("kept out-of-window content:\n%s", got)
+	}
+}
+
+func TestSliceJournal_AcceptsRFC3339ColonOffset(t *testing.T) {
+	// systemd's `journalctl --output=short-iso` ships RFC3339 timestamps
+	// (offset includes a colon, e.g. "+10:00") on most versions. Earlier
+	// regex only matched "+1000" — make sure both forms work.
+	src := filepath.Join(t.TempDir(), "journal.log")
+	dst := filepath.Join(t.TempDir(), "out.log")
+	body := strings.Join([]string{
+		"2026-05-19T12:32:30+10:00 host spinifex-daemon[1]: pre",
+		"2026-05-19T12:32:35+10:00 host spinifex-daemon[1]: IN window",
+		"2026-05-19T12:32:50+10:00 host spinifex-daemon[1]: post",
+		"",
+	}, "\n")
+	if err := os.WriteFile(src, []byte(body), 0o644); err != nil {
+		t.Fatalf("write src: %v", err)
+	}
+	start, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:33+10:00")
+	end, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:40+10:00")
+	n, err := SliceJournal(src, start, end, dst)
+	if err != nil {
+		t.Fatalf("SliceJournal: %v", err)
+	}
+	if n != 1 {
+		t.Fatalf("written = %d, want 1", n)
+	}
+	got, _ := os.ReadFile(dst)
+	if !strings.Contains(string(got), "IN window") {
+		t.Errorf("missing in-window content:\n%s", got)
+	}
+}
+
+func TestSliceJournal_MissingSrcIsNotError(t *testing.T) {
+	dst := filepath.Join(t.TempDir(), "out.log")
+	n, err := SliceJournal("/no/such/file", time.Now(), time.Now(), dst)
+	if err != nil {
+		t.Fatalf("unexpected err: %v", err)
+	}
+	if n != 0 {
+		t.Errorf("written = %d, want 0", n)
+	}
+}
+
+func TestSliceTestLog_PicksOneTestcase(t *testing.T) {
+	src := filepath.Join(t.TempDir(), "test.log")
+	dst := filepath.Join(t.TempDir(), "out.log")
+	body := strings.Join([]string{
+		"=== RUN   TestSingleNode",
+		"  before subtests",
+		"=== RUN   TestSingleNode/Phase5_LaunchInstance",
+		"    runinstances",
+		"    ec2helpers.go:164: describe-vpcs: InvalidParameterValue",
+		"--- FAIL: TestSingleNode/Phase5_LaunchInstance (2.50s)",
+		"=== RUN   TestSingleNode/Phase6_Volumes",
+		"    other content",
+		"--- FAIL: TestSingleNode/Phase6_Volumes (0.10s)",
+		"--- FAIL: TestSingleNode (10.0s)",
+		"",
+	}, "\n")
+	if err := os.WriteFile(src, []byte(body), 0o644); err != nil {
+		t.Fatalf("write src: %v", err)
+	}
+	n, err := SliceTestLog(src, "TestSingleNode/Phase5_LaunchInstance", dst)
+	if err != nil {
+		t.Fatalf("SliceTestLog: %v", err)
+	}
+	if n != 4 {
+		t.Fatalf("written = %d, want 4 (RUN + 2 inner + FAIL)", n)
+	}
+	got, _ := os.ReadFile(dst)
+	s := string(got)
+	if !strings.Contains(s, "describe-vpcs: InvalidParameterValue") {
+		t.Errorf("missing failure line:\n%s", s)
+	}
+	if strings.Contains(s, "Phase6_Volumes") || strings.Contains(s, "before subtests") {
+		t.Errorf("leaked sibling test content:\n%s", s)
+	}
+}
+
+func TestSliceTestLog_UnknownTestcaseIsEmpty(t *testing.T) {
+	src := filepath.Join(t.TempDir(), "test.log")
+	dst := filepath.Join(t.TempDir(), "out.log")
+	if err := os.WriteFile(src, []byte("=== RUN   TestX\n--- PASS: TestX (0.0s)\n"), 0o644); err != nil {
+		t.Fatalf("write src: %v", err)
+	}
+	n, err := SliceTestLog(src, "TestY", dst)
+	if err != nil {
+		t.Fatalf("SliceTestLog: %v", err)
+	}
+	if n != 0 {
+		t.Errorf("written = %d, want 0", n)
+	}
+	if _, err := os.Stat(dst); err != nil {
+		t.Errorf("dst not created: %v", err)
+	}
+}
+
+// TestWriteBundles_EndToEnd builds a fake artifact dir with a junit XML,
+// a journal log, a test log, and a per-test artifact subdir, then runs
+// the full ParseFile → WriteBundles pipeline and asserts the bundle dir
+// contains everything the report links to.
+func TestWriteBundles_EndToEnd(t *testing.T) {
+	dir := t.TempDir()
+
+	junit := `<?xml version="1.0"?>
+<testsuites><testsuite name="" tests="2" failures="1" timestamp="2026-05-19T12:32:30+10:00" time="3.0">
+  <testcase name="TestSingleNode/Phase5_LaunchInstance" time="2.500">
+    <failure message="Failed"><![CDATA[
+    ec2helpers.go:164: describe-vpcs: InvalidParameterValue (status 400)
+        Error Trace: tests/e2e/harness/ec2helpers.go:164
+        Test: TestSingleNode/Phase5_LaunchInstance]]></failure>
+  </testcase>
+</testsuite></testsuites>`
+	if err := os.WriteFile(filepath.Join(dir, "junit-single.xml"), []byte(junit), 0o644); err != nil {
+		t.Fatalf("write junit: %v", err)
+	}
+
+	journal := strings.Join([]string{
+		"2026-05-19T12:32:20+1000 host spinifex-daemon[1]: too-early",
+		"2026-05-19T12:32:30+1000 host spinifex-daemon[1]: in-window-start",
+		"2026-05-19T12:32:32+1000 host spinifex-daemon[1]: describe-vpcs filter=is_default",
+		"2026-05-19T12:32:50+1000 host spinifex-daemon[1]: too-late",
+		"",
+	}, "\n")
+	if err := os.WriteFile(filepath.Join(dir, "spinifex-journal.log"), []byte(journal), 0o644); err != nil {
+		t.Fatalf("write journal: %v", err)
+	}
+
+	testlog := strings.Join([]string{
+		"=== RUN   TestSingleNode/Phase5_LaunchInstance",
+		"    ec2helpers.go:164: describe-vpcs: InvalidParameterValue",
+		"--- FAIL: TestSingleNode/Phase5_LaunchInstance (2.50s)",
+		"",
+	}, "\n")
+	if err := os.WriteFile(filepath.Join(dir, "test-single.log"), []byte(testlog), 0o644); err != nil {
+		t.Fatalf("write test log: %v", err)
+	}
+
+	// Per-test artifact subdir (matches harness.ArtifactDir convention).
+	perTest := filepath.Join(dir, "TestSingleNode_Phase5_LaunchInstance")
+	if err := os.MkdirAll(perTest, 0o755); err != nil {
+		t.Fatalf("mkdir per-test: %v", err)
+	}
+	if err := os.WriteFile(filepath.Join(perTest, "qemu-console.log"), []byte("[qemu] boot\n"), 0o644); err != nil {
+		t.Fatalf("write per-test file: %v", err)
+	}
+
+	data, _ := os.ReadFile(filepath.Join(dir, "junit-single.xml"))
+	sr, err := ParseFile("junit-single.xml", data)
+	if err != nil {
+		t.Fatalf("ParseFile: %v", err)
+	}
+	rep := Report{Title: "E2E failure analysis", Suites: []SuiteReport{sr}}
+	WriteBundles(&rep, dir)
+
+	if rep.Suites[0].Root == nil {
+		t.Fatal("no root")
+	}
+	bp := rep.Suites[0].Root.BundlePath
+	if bp == "" {
+		t.Fatal("BundlePath not set")
+	}
+	bundle := filepath.Join(dir, bp)
+	for _, want := range []string{"failure.txt", "journal.log", "test.log", "qemu-console.log"} {
+		if _, err := os.Stat(filepath.Join(bundle, want)); err != nil {
+			t.Errorf("missing %s in bundle %s: %v", want, bp, err)
+		}
+	}
+
+	jl, _ := os.ReadFile(filepath.Join(bundle, "journal.log"))
+	if !strings.Contains(string(jl), "describe-vpcs filter=is_default") {
+		t.Errorf("journal.log lacks in-window line:\n%s", jl)
+	}
+	if strings.Contains(string(jl), "too-early") || strings.Contains(string(jl), "too-late") {
+		t.Errorf("journal.log includes out-of-window content:\n%s", jl)
+	}
+
+	tl, _ := os.ReadFile(filepath.Join(bundle, "test.log"))
+	if !strings.Contains(string(tl), "--- FAIL: TestSingleNode/Phase5_LaunchInstance") {
+		t.Errorf("test.log missing FAIL marker:\n%s", tl)
+	}
+
+	out := Render(rep)
+	wantLink := fmt.Sprintf("- Bundle: `%s/`", bp)
+	if !strings.Contains(out, wantLink) {
+		t.Errorf("render missing bundle link %q:\n%s", wantLink, out)
+	}
+}

.github/actions/e2e-analyze/main.go

@@ -0,0 +1,78 @@
+package main
+
+import (
+	"flag"
+	"fmt"
+	"os"
+	"path/filepath"
+	"sort"
+)
+
+func main() {
+	var junitGlob, logDir, title string
+	flag.StringVar(&junitGlob, "junit-glob", os.Getenv("ANALYZE_JUNIT_GLOB"), "glob matching junit-*.xml files")
+	flag.StringVar(&logDir, "log-dir", os.Getenv("ANALYZE_LOG_DIR"), "artifact dir to write analysis.md into")
+	flag.StringVar(&title, "title", os.Getenv("ANALYZE_TITLE"), "report heading")
+	flag.Parse()
+
+	if title == "" {
+		title = "E2E failure analysis"
+	}
+	if junitGlob == "" {
+		fmt.Fprintln(os.Stderr, "::error::e2e-analyze: -junit-glob is required")
+		os.Exit(0) // analyzer is advisory only; existing gate owns exit code
+	}
+
+	paths, err := filepath.Glob(junitGlob)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: bad glob %q: %v\n", junitGlob, err)
+		os.Exit(0)
+	}
+	sort.Strings(paths)
+
+	report := Report{Title: title}
+	for _, p := range paths {
+		data, err := os.ReadFile(p)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: read %s: %v\n", p, err)
+			continue
+		}
+		sr, err := ParseFile(p, data)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: %v\n", err)
+			continue
+		}
+		report.Suites = append(report.Suites, sr)
+	}
+
+	// Stage 2: prefer authoritative per-suite start times written by the
+	// workflow over junit-report's conversion-time `timestamp` attribute,
+	// then materialise per-failure bundles so Render can link them.
+	if logDir != "" {
+		ApplySuiteStartFiles(&report, logDir)
+		WriteBundles(&report, logDir)
+	}
+
+	out := Render(report)
+
+	if logDir != "" {
+		if err := os.MkdirAll(logDir, 0o755); err == nil {
+			if err := os.WriteFile(filepath.Join(logDir, "analysis.md"), []byte(out), 0o644); err != nil {
+				fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: write analysis.md: %v\n", err)
+			}
+		}
+	}
+
+	if summaryPath := os.Getenv("GITHUB_STEP_SUMMARY"); summaryPath != "" {
+		f, err := os.OpenFile(summaryPath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
+		if err == nil {
+			defer f.Close()
+			if _, werr := f.WriteString(out); werr != nil {
+				fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: append summary: %v\n", werr)
+			}
+		}
+	} else {
+		// Local invocation (developer running the tool by hand). Mirror to stdout.
+		fmt.Print(out)
+	}
+}

.github/actions/e2e-analyze/slice.go

@@ -0,0 +1,176 @@
+package main
+
+// Stage 2 of docs/development/improvements/e2e-go-failure-analysis.md:
+// per-failure log slicers used by bundle.go to materialise time-window /
+// testcase-window log excerpts next to each named failure.
+
+import (
+	"bufio"
+	"io"
+	"os"
+	"regexp"
+	"strings"
+	"time"
+)
+
+// shortIsoTSRe matches the timestamp prefix produced by
+// `journalctl --output=short-iso`. The offset format varies across
+// systemd versions: most ship "+10:00" (RFC3339), some ship "+1000"
+// (no colon). Match either — the parser below picks the right layout.
+var shortIsoTSRe = regexp.MustCompile(`^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:[+-]\d{2}:?\d{2}|Z))\s`)
+
+// shortIsoLayouts are tried in order; the first parse that succeeds wins.
+var shortIsoLayouts = []string{
+	time.RFC3339,               // 2026-05-19T12:32:30+10:00
+	"2006-01-02T15:04:05-0700", // 2026-05-19T12:32:30+1000
+}
+
+func parseShortIso(s string) (time.Time, error) {
+	var lastErr error
+	for _, layout := range shortIsoLayouts {
+		t, err := time.Parse(layout, s)
+		if err == nil {
+			return t, nil
+		}
+		lastErr = err
+	}
+	return time.Time{}, lastErr
+}
+
+// SliceJournal copies lines from src whose `short-iso` timestamp falls in
+// [start, end] to dst. Lines without a parseable timestamp inherit the
+// previous line's classification — that keeps multi-line journal records
+// (stack traces, panics) intact rather than orphaning their continuation
+// lines. Lines that precede the first parseable timestamp are dropped.
+//
+// Returns the number of lines written; missing src is not an error (the
+// journal may be empty if collection failed).
+func SliceJournal(src string, start, end time.Time, dst string) (int, error) {
+	f, err := os.Open(src)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return 0, nil
+		}
+		return 0, err
+	}
+	defer f.Close()
+
+	out, err := os.Create(dst)
+	if err != nil {
+		return 0, err
+	}
+	defer out.Close()
+
+	bw := bufio.NewWriter(out)
+	defer bw.Flush()
+
+	sc := bufio.NewScanner(f)
+	// Journal lines can be long (panic dumps); bump scanner buffer.
+	sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
+
+	keep := false
+	written := 0
+	for sc.Scan() {
+		line := sc.Text()
+		if m := shortIsoTSRe.FindStringSubmatch(line); m != nil {
+			ts, err := parseShortIso(m[1])
+			if err != nil {
+				continue
+			}
+			keep = !ts.Before(start) && !ts.After(end)
+		}
+		if keep {
+			if _, err := bw.WriteString(line); err != nil {
+				return written, err
+			}
+			if err := bw.WriteByte('\n'); err != nil {
+				return written, err
+			}
+			written++
+		}
+	}
+	return written, sc.Err()
+}
+
+// runRePrefix and finishRePrefix bracket a single testcase's output in
+// `go test -v` stdout. Subtests have their own `=== RUN` / `--- FAIL`
+// markers, so slicing on the exact testcase name yields just that test's
+// lines (including any nested phase logs).
+var (
+	runRe    = regexp.MustCompile(`^=== RUN\s+(\S+)$`)
+	finishRe = regexp.MustCompile(`^\s*---\s+(PASS|FAIL|SKIP):\s+(\S+)\s+\(`)
+)
+
+// SliceTestLog copies lines belonging to the named testcase from src to
+// dst. The block starts at `=== RUN <name>` and ends at the matching
+// `--- PASS|FAIL|SKIP: <name> (…)` line (inclusive). If the testcase is
+// never seen the destination file is created empty so the bundle has a
+// stable shape.
+//
+// Returns the number of lines written; missing src is not an error.
+func SliceTestLog(src, testName, dst string) (int, error) {
+	f, err := os.Open(src)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return 0, nil
+		}
+		return 0, err
+	}
+	defer f.Close()
+
+	out, err := os.Create(dst)
+	if err != nil {
+		return 0, err
+	}
+	defer out.Close()
+
+	bw := bufio.NewWriter(out)
+	defer bw.Flush()
+
+	sc := bufio.NewScanner(f)
+	sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
+
+	in := false
+	written := 0
+	for sc.Scan() {
+		line := sc.Text()
+		if !in {
+			if m := runRe.FindStringSubmatch(strings.TrimRight(line, "\r")); m != nil && m[1] == testName {
+				in = true
+			} else {
+				continue
+			}
+		}
+		if _, err := bw.WriteString(line); err != nil {
+			return written, err
+		}
+		if err := bw.WriteByte('\n'); err != nil {
+			return written, err
+		}
+		written++
+		if m := finishRe.FindStringSubmatch(line); m != nil && m[2] == testName {
+			return written, nil
+		}
+	}
+	return written, sc.Err()
+}
+
+// copyFile is a small helper used when bundling pre-collected per-test
+// artifacts. Best-effort: missing src is silently skipped.
+func copyFile(src, dst string) error {
+	in, err := os.Open(src)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return nil
+		}
+		return err
+	}
+	defer in.Close()
+	out, err := os.Create(dst)
+	if err != nil {
+		return err
+	}
+	defer out.Close()
+	_, err = io.Copy(out, in)
+	return err
+}

.github/actions/e2e-analyze/testdata/cascade-vpcs/analysis.md

@@ -0,0 +1,23 @@
+## E2E failure analysis
+
+### Suite `single`: 16 failed, 1 root cause likely
+
+**Root cause (earliest non-cascade)**
+
+- Test: `TestSingleNode/Phase5_LaunchInstance`
+- Start: 12:30:00 (duration 2.5s)
+- Error: Received unexpected error: InvalidParameterValue (HTTP 400)
+- File:  `tests/e2e/harness/ec2helpers.go:164`
+
+**Cascaded failures (15) — grouped by signature**
+
+- 14× Phase 5 must populate fix.InstanceID
+  - `TestSingleNode/Phase5a_pre_ClusterStats`
+  - `TestSingleNode/Phase5a_Metadata`
+  - `TestSingleNode/Phase5b_DescribeInstances`
+  - `TestSingleNode/Phase5c_StopStart`
+  - `TestSingleNode/Phase6_Volumes`
+  - … and 9 more
+- 1× Phase 2 must populate fix.AZName
+  - `TestSingleNode/Phase8_NatGW`
+

.github/actions/e2e-analyze/testdata/cascade-vpcs/junit-single.xml

@@ -0,0 +1,143 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<testsuites tests="20" failures="17" skipped="0">
+	<testsuite name="" tests="20" failures="17" errors="0" id="0" hostname="banksia" skipped="0" time="600.000" timestamp="2026-04-12T12:30:00+10:00">
+		<testcase name="TestSingleNode" classname="" time="600.000">
+			<failure message="Failed"></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase1_Environment" classname="" time="0.300">
+			<system-out><![CDATA[
+&#x2501;&#x2501; Phase 1 &#x2501;&#x2501;]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase2_Discovery" classname="" time="0.010"/>
+		<testcase name="TestSingleNode/Phase3_KeyPairs" classname="" time="0.540"/>
+		<testcase name="TestSingleNode/Phase4_Image" classname="" time="0.060"/>
+		<testcase name="TestSingleNode/Phase5_LaunchInstance" classname="" time="2.500">
+			<failure message="Failed"><![CDATA[
+    ec2helpers.go:164: describe-vpcs: InvalidParameterValue: Filter 'is_default' is invalid (status 400)
+    	Error Trace:	tests/e2e/harness/ec2helpers.go:164
+    	Error:      	Received unexpected error:
+    	            	InvalidParameterValue (HTTP 400)
+    	Test:       	TestSingleNode/Phase5_LaunchInstance]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_pre_ClusterStats" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    lifecycle_test.go:28:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:28
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5a_pre_ClusterStats]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_Metadata" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    lifecycle_test.go:57:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:57
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5a_Metadata]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5b_DescribeInstances" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    lifecycle_test.go:109:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:109
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5b_DescribeInstances]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5c_StopStart" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    lifecycle_test.go:222:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:222
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5c_StopStart]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase6_Volumes" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    volume_test.go:23:
+        Error Trace:	tests/e2e/single/volume_test.go:23
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase6_Volumes]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase6b_AttachVolume" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    volume_test.go:24:
+        Error Trace:	tests/e2e/single/volume_test.go:24
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase6b_AttachVolume]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7_Tags" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    tags_test.go:23:
+        Error Trace:	tests/e2e/single/tags_test.go:23
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7_Tags]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7a_Snapshots" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    snapshot_test.go:23:
+        Error Trace:	tests/e2e/single/snapshot_test.go:23
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7a_Snapshots]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7b_DescribeInstanceAttribute" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    instance_test.go:81:
+        Error Trace:	tests/e2e/single/instance_test.go:81
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7b_DescribeInstanceAttribute]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7c_ModifyInstanceAttribute" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    instance_test.go:92:
+        Error Trace:	tests/e2e/single/instance_test.go:92
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7c_ModifyInstanceAttribute]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7d_Console" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    instance_test.go:121:
+        Error Trace:	tests/e2e/single/instance_test.go:121
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7d_Console]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7e_Negative" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    negative_test.go:43:
+        Error Trace:	tests/e2e/single/negative_test.go:43
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7e_Negative]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NatGW" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    natgw_test.go:39:
+        Error Trace:	tests/e2e/single/natgw_test.go:39
+        Error:      	Should NOT be empty
+        Messages:   	Phase 2 must populate fix.AZName
+        Test:       	TestSingleNode/Phase8_NatGW]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8a_DataPath" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    datapath_test.go:48:
+        Error Trace:	tests/e2e/single/datapath_test.go:48
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase8a_DataPath]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase9_SecurityGroups" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    securitygroup_test.go:35:
+        Error Trace:	tests/e2e/single/securitygroup_test.go:35
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase9_SecurityGroups]]></failure>
+		</testcase>
+	</testsuite>
+</testsuites>

.github/actions/e2e-analyze/testdata/no-failures/analysis.md

@@ -0,0 +1,3 @@
+## E2E failure analysis
+
+✅ No failures across any suite.

.github/actions/e2e-analyze/testdata/no-failures/junit-single.xml

@@ -0,0 +1,8 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<testsuites tests="3" failures="0" skipped="0">
+	<testsuite name="" tests="3" failures="0" errors="0" id="0" hostname="banksia" skipped="0" time="1.500" timestamp="2026-04-12T12:30:00+10:00">
+		<testcase name="TestSingleNode" classname="" time="1.500"/>
+		<testcase name="TestSingleNode/Phase1_Environment" classname="" time="0.300"/>
+		<testcase name="TestSingleNode/Phase2_Discovery" classname="" time="1.200"/>
+	</testsuite>
+</testsuites>

.github/actions/e2e-analyze/testdata/single-failure/analysis.md

@@ -0,0 +1,11 @@
+## E2E failure analysis
+
+### Suite `single`: 1 failed, 1 root cause likely
+
+**Root cause (earliest non-cascade)**
+
+- Test: `TestSingleNode/Phase8b_VPCSubnetE2E`
+- Start: 00:26:18 (duration 200.5s)
+- Error: Eventually: condition not met within 3m0s: [SSH handshake 192.168.0.202:22 never completed]
+- File:  `vpc_test.go:227`
+

.github/actions/e2e-analyze/testdata/single-failure/junit-single.xml

@@ -0,0 +1,993 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<testsuites tests="79" failures="2" skipped="6">
+	<testsuite name="" tests="79" failures="2" errors="0" id="0" hostname="banksia" skipped="6" time="1708.140" timestamp="2026-05-19T00:19:45+10:00">
+		<testcase name="TestSingleNode" classname="" time="800.320">
+			<failure message="Failed"></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase1_Environment" classname="" time="0.300">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 1 — Environment ━━�[0m
+�[2m· checking /dev/kvm�[0m
+    discovery_test.go:29: �[2m· checking /dev/kvm�[0m
+�[2mkvm�[0m=writable
+    discovery_test.go:38: �[2mkvm�[0m=writable
+�[2m· waiting for AWS gateway�[0m
+    discovery_test.go:45: �[2m· waiting for AWS gateway�[0m
+�[2mgateway�[0m=https://192.168.0.10:9999/
+    discovery_test.go:55: �[2mgateway�[0m=https://192.168.0.10:9999/
+�[2m· waiting for daemon NATS subscriptions�[0m
+    discovery_test.go:60: �[2m· waiting for daemon NATS subscriptions�[0m
+�[2m· describe-regions / describe-availability-zones�[0m
+    discovery_test.go:73: �[2m· describe-regions / describe-availability-zones�[0m
+�[2maz�[0m=ap-southeast-2a �[2mregion�[0m=ap-southeast-2
+    discovery_test.go:84: �[2maz�[0m=ap-southeast-2a �[2mregion�[0m=ap-southeast-2]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase1b_ClusterStats" classname="" time="9.200">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 1b — Cluster Stats CLI ━━�[0m
+�[2m· spx get nodes�[0m
+    discovery_test.go:101: �[2m· spx get nodes�[0m
+�[2m· spx top nodes�[0m
+    discovery_test.go:105: �[2m· spx top nodes�[0m
+�[2m· spx get vms (empty before any launches)�[0m
+    discovery_test.go:112: �[2m· spx get vms (empty before any launches)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase2_Discovery" classname="" time="0.010">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 2 — Discovery & Metadata ━━�[0m
+�[2m· discovering nano instance type�[0m
+    discovery_test.go:133: �[2m· discovering nano instance type�[0m
+�[2minstance_type�[0m=t3.nano �[2march�[0m=x86_64 �[2maz�[0m=ap-southeast-2a
+    discovery_test.go:159: �[2minstance_type�[0m=t3.nano �[2march�[0m=x86_64 �[2maz�[0m=ap-southeast-2a]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase2b_SerialConsole" classname="" time="0.020">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 2b — Serial Console Access ━━�[0m
+�[2m· default state should be disabled�[0m
+    discovery_test.go:168: �[2m· default state should be disabled�[0m
+�[2m· enable�[0m
+    discovery_test.go:173: �[2m· enable�[0m
+�[2mstate�[0m=enabled
+    discovery_test.go:180: �[2mstate�[0m=enabled
+�[2m· disable�[0m
+    discovery_test.go:182: �[2m· disable�[0m
+�[2mstate�[0m=disabled
+    discovery_test.go:189: �[2mstate�[0m=disabled]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase3_KeyPairs" classname="" time="0.540">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 3 — SSH Key Management ━━�[0m
+�[2m· create-key-pair "test-key-1"�[0m
+    keypair_test.go:37: �[2m· create-key-pair "test-key-1"�[0m
+�[2mkey�[0m=test-key-1 �[2mpem�[0m=/tmp/TestSingleNode1289015487/001/test-key-1.pem
+    keypair_test.go:51: �[2mkey�[0m=test-key-1 �[2mpem�[0m=/tmp/TestSingleNode1289015487/001/test-key-1.pem
+�[2m· import-key-pair "test-key-2" (local-generated RSA)�[0m
+    keypair_test.go:58: �[2m· import-key-pair "test-key-2" (local-generated RSA)�[0m
+�[2mimported�[0m=test-key-2
+    keypair_test.go:65: �[2mimported�[0m=test-key-2
+�[2m· describe-key-pairs (both present)�[0m
+    keypair_test.go:67: �[2m· describe-key-pairs (both present)�[0m
+�[2m· delete "test-key-2"�[0m
+    keypair_test.go:72: �[2m· delete "test-key-2"�[0m
+�[2m· describe-key-pairs (only test-key-1 remains)�[0m
+    keypair_test.go:76: �[2m· describe-key-pairs (only test-key-1 remains)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase4_Image" classname="" time="0.060">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 4 — Image Management ━━�[0m
+�[2mimage�[0m=ubuntu-24.04-x86_64 �[2march�[0m=x86_64
+    image_test.go:33: �[2mimage�[0m=ubuntu-24.04-x86_64 �[2march�[0m=x86_64
+�[2m· spx admin images import --name ubuntu-24.04-x86_64�[0m
+    image_test.go:35: �[2m· spx admin images import --name ubuntu-24.04-x86_64�[0m
+�[2m· spx import did not yield AMI ID; falling back to describe-images�[0m
+    image_test.go:48: �[2m· spx import did not yield AMI ID; falling back to describe-images�[0m
+�[2m· describe-images by AMI ID (verify exactly one match)�[0m
+    image_test.go:78: �[2m· describe-images by AMI ID (verify exactly one match)�[0m
+�[2mami�[0m=ami-6f60f2b3bf04b5453 �[2mname�[0m=ami-ubuntu-24.04-x86_64
+    image_test.go:86: �[2mami�[0m=ami-6f60f2b3bf04b5453 �[2mname�[0m=ami-ubuntu-24.04-x86_64]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5_LaunchInstance" classname="" time="10.830">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5 — Instance Lifecycle ━━�[0m
+�[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msg�[0m=sg-764a7fa2a39577204 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63
+    instance_test.go:31: �[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msg�[0m=sg-764a7fa2a39577204 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63
+�[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.nano key=test-key-1�[0m
+    instance_test.go:35: �[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.nano key=test-key-1�[0m
+�[2minstance�[0m=i-0c615e5dd74050f53
+    instance_test.go:48: �[2minstance�[0m=i-0c615e5dd74050f53
+    instance_test.go:50: instance i-0c615e5dd74050f53 reached state running
+�[2mroot_volume�[0m=vol-2230138124fb51ee2 �[2mroot_device�[0m=
+    instance_test.go:70: �[2mroot_volume�[0m=vol-2230138124fb51ee2 �[2mroot_device�[0m=]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_pre_ClusterStats" classname="" time="3.040">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5a-pre — Cluster Stats CLI (with running VM) ━━�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_Metadata" classname="" time="0.510">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5a — Instance Metadata Validation ━━�[0m
+�[2mtype�[0m=t3.nano �[2mkey�[0m=test-key-1 �[2mimage�[0m=ami-6f60f2b3bf04b5453 �[2mbdms�[0m=1
+    instance_test.go:108: �[2mtype�[0m=t3.nano �[2mkey�[0m=test-key-1 �[2mimage�[0m=ami-6f60f2b3bf04b5453 �[2mbdms�[0m=1]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_ii_SSH" classname="" time="49.450">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5a-ii — SSH Connectivity & Volume Verification ━━�[0m
+�[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
+    instance_test.go:127: �[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    instance_test.go:130: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    lifecycle_test.go:384: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2m· ssh id�[0m
+    instance_test.go:136: �[2m· ssh id�[0m
+�[2m· lsblk root-volume cross-check vs API�[0m
+    instance_test.go:140: �[2m· lsblk root-volume cross-check vs API�[0m
+�[2mguest_gib�[0m=4 �[2mapi_gib�[0m=4
+    instance_test.go:159: �[2mguest_gib�[0m=4 �[2mapi_gib�[0m=4
+�[2m· ssh hostname�[0m
+    instance_test.go:161: �[2m· ssh hostname�[0m
+�[2mhostname�[0m=spinifex-vm-0c615e5d �[2mmatches_short_id�[0m=0c615e5d
+    instance_test.go:172: �[2mhostname�[0m=spinifex-vm-0c615e5d �[2mmatches_short_id�[0m=0c615e5d]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_iii_Console" classname="" time="0.010">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5a-iii — Console Output ━━�[0m
+�[2minstance�[0m=i-0c615e5dd74050f53 �[2mhas_output�[0m=true
+    instance_test.go:191: �[2minstance�[0m=i-0c615e5dd74050f53 �[2mhas_output�[0m=true]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5b_Volume" classname="" time="4.540">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5b — Volume Lifecycle (Attach/Detach) ━━�[0m
+�[2m· create-volume size=10 az=ap-southeast-2a�[0m
+    volume_test.go:26: �[2m· create-volume size=10 az=ap-southeast-2a�[0m
+�[2mvolume�[0m=vol-a885f79173e0a8030
+    volume_test.go:34: �[2mvolume�[0m=vol-a885f79173e0a8030
+    volume_test.go:39: volume vol-a885f79173e0a8030 reached state available
+�[2m· modify-volume size=20�[0m
+    volume_test.go:42: �[2m· modify-volume size=20�[0m
+�[2mresized_gib�[0m=20
+    volume_test.go:66: �[2mresized_gib�[0m=20
+�[2m· attach-volume vol-a885f79173e0a8030 -> i-0c615e5dd74050f53 as /dev/sdf�[0m
+    volume_test.go:68: �[2m· attach-volume vol-a885f79173e0a8030 -> i-0c615e5dd74050f53 as /dev/sdf�[0m
+    volume_test.go:76: volume vol-a885f79173e0a8030 reached state in-use
+�[2m· detach-volume vol-a885f79173e0a8030�[0m
+    volume_test.go:90: �[2m· detach-volume vol-a885f79173e0a8030�[0m
+    volume_test.go:96: volume vol-a885f79173e0a8030 reached state available
+�[2m· delete-volume vol-a885f79173e0a8030�[0m
+    volume_test.go:98: �[2m· delete-volume vol-a885f79173e0a8030�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5b_ii_VolumeStatus" classname="" time="0.000">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5b-ii — DescribeVolumeStatus ━━�[0m
+�[2mvolume�[0m=vol-2230138124fb51ee2 �[2mstatus�[0m=ok
+    volume_test.go:158: �[2mvolume�[0m=vol-2230138124fb51ee2 �[2mstatus�[0m=ok]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5c_Snapshot" classname="" time="0.190">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5c — Snapshot Lifecycle ━━�[0m
+�[2m· create-snapshot volume=vol-2230138124fb51ee2�[0m
+    snapshot_test.go:34: �[2m· create-snapshot volume=vol-2230138124fb51ee2�[0m
+�[2msnapshot�[0m=snap-1188f92980e6470aa �[2msize�[0m=4 �[2mstate�[0m=completed �[2mprogress�[0m=100%
+    snapshot_test.go:50: �[2msnapshot�[0m=snap-1188f92980e6470aa �[2msize�[0m=4 �[2mstate�[0m=completed �[2mprogress�[0m=100%
+    snapshot_test.go:59: snapshot snap-1188f92980e6470aa reached state completed
+�[2m· describe-snapshots snap-1188f92980e6470aa�[0m
+    snapshot_test.go:61: �[2m· describe-snapshots snap-1188f92980e6470aa�[0m
+�[2m· copy-snapshot src=snap-1188f92980e6470aa region=ap-southeast-2�[0m
+    snapshot_test.go:76: �[2m· copy-snapshot src=snap-1188f92980e6470aa region=ap-southeast-2�[0m
+�[2mcopy_snapshot�[0m=snap-cde6b5ddf467e2c0f
+    snapshot_test.go:86: �[2mcopy_snapshot�[0m=snap-cde6b5ddf467e2c0f
+    snapshot_test.go:88: snapshot snap-cde6b5ddf467e2c0f reached state completed
+�[2m· describe-snapshots snap-1188f92980e6470aa,snap-cde6b5ddf467e2c0f (both visible)�[0m
+    snapshot_test.go:90: �[2m· describe-snapshots snap-1188f92980e6470aa,snap-cde6b5ddf467e2c0f (both visible)�[0m
+�[2m· delete-snapshot snap-1188f92980e6470aa (original)�[0m
+    snapshot_test.go:107: �[2m· delete-snapshot snap-1188f92980e6470aa (original)�[0m
+�[2m· verify copy snap-cde6b5ddf467e2c0f still completed�[0m
+    snapshot_test.go:135: �[2m· verify copy snap-cde6b5ddf467e2c0f still completed�[0m
+�[2m· delete-snapshot snap-cde6b5ddf467e2c0f (copy)�[0m
+    snapshot_test.go:144: �[2m· delete-snapshot snap-cde6b5ddf467e2c0f (copy)�[0m
+�[2mdeleted_original�[0m=snap-1188f92980e6470aa
+    snapshot_test.go:151: �[2mdeleted_original�[0m=snap-1188f92980e6470aa]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5d_SnapshotBackedLaunch" classname="" time="0.010">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5d — Snapshot-Backed Instance Launch ━━�[0m
+�[2mami�[0m=ami-6f60f2b3bf04b5453 �[2msnapshot�[0m=snap-ami-6f60f2b3bf04b5453
+    snapshot_test.go:182: �[2mami�[0m=ami-6f60f2b3bf04b5453 �[2msnapshot�[0m=snap-ami-6f60f2b3bf04b5453]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5e_CreateImage" classname="" time="0.660">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5e — CreateImage Lifecycle ━━�[0m
+�[2m· create-image instance=i-0c615e5dd74050f53 name=e2e-custom-ami (no-reboot)�[0m
+    createimage_test.go:26: �[2m· create-image instance=i-0c615e5dd74050f53 name=e2e-custom-ami (no-reboot)�[0m
+�[2mcustom_ami�[0m=ami-7dcf263a412dd14c6
+    createimage_test.go:36: �[2mcustom_ami�[0m=ami-7dcf263a412dd14c6
+    createimage_test.go:38: image ami-7dcf263a412dd14c6 reached state available
+�[2m· describe-images ami-7dcf263a412dd14c6�[0m
+    createimage_test.go:40: �[2m· describe-images ami-7dcf263a412dd14c6�[0m
+�[2mcustom_ami_snapshot�[0m=snap-dde89b9ea2ef2e05a
+    createimage_test.go:67: �[2mcustom_ami_snapshot�[0m=snap-dde89b9ea2ef2e05a]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5f_SecurityGroupEgress" classname="" time="4.880">
+			<skipped message="Skipped"><![CDATA[
+�[1m�[36m━━ Phase 5f — Security Group Enforcement (egress ACL) ━━�[0m
+�[2m· discover default gateway inside VM�[0m
+    securitygroup_test.go:53: �[2m· discover default gateway inside VM�[0m
+�[2mprobe_gateway�[0m=172.31.0.1
+    securitygroup_test.go:66: �[2mprobe_gateway�[0m=172.31.0.1
+�[2m· 5f-1 baseline egress (allow-all)�[0m
+    securitygroup_test.go:77: �[2m· 5f-1 baseline egress (allow-all)�[0m
+    securitygroup_test.go:80: Phase 5f: baseline ICMP did not succeed; env may block ICMP regardless of SG
+        Output:
+        PING 172.31.0.1 (172.31.0.1) 56(84) bytes of data.
+        
+        --- 172.31.0.1 ping statistics ---
+        3 packets transmitted, 0 received, 100% packet loss, time 2044ms
+        ]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/Phase6_TagManagement" classname="" time="0.380">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 6 — Tag Management ━━�[0m
+�[2m· 6a create-tags instance=i-0c615e5dd74050f53 (Name, Environment, DeleteMe)�[0m
+    tags_test.go:41: �[2m· 6a create-tags instance=i-0c615e5dd74050f53 (Name, Environment, DeleteMe)�[0m
+�[2m· 6b describe-tags resource-id=i-0c615e5dd74050f53 want=3�[0m
+    tags_test.go:53: �[2m· 6b describe-tags resource-id=i-0c615e5dd74050f53 want=3�[0m
+�[2m· 6c create-tags volume=vol-2230138124fb51ee2 (Name, Environment)�[0m
+    tags_test.go:65: �[2m· 6c create-tags volume=vol-2230138124fb51ee2 (Name, Environment)�[0m
+�[2m· 6d describe-tags key=Environment want=2�[0m
+    tags_test.go:76: �[2m· 6d describe-tags key=Environment want=2�[0m
+�[2m· 6e describe-tags resource-type=instance want=3�[0m
+    tags_test.go:85: �[2m· 6e describe-tags resource-type=instance want=3�[0m
+�[2m· 6f overwrite instance Name -> e2e-test-updated�[0m
+    tags_test.go:95: �[2m· 6f overwrite instance Name -> e2e-test-updated�[0m
+�[2m· 6g delete-tags Key=DeleteMe (no value)�[0m
+    tags_test.go:114: �[2m· 6g delete-tags Key=DeleteMe (no value)�[0m
+�[2m· 6h delete-tags Key=Environment,Value=production (no-op)�[0m
+    tags_test.go:130: �[2m· 6h delete-tags Key=Environment,Value=production (no-op)�[0m
+�[2m· 6i delete-tags Key=Environment,Value=testing (instance only)�[0m
+    tags_test.go:148: �[2m· 6i delete-tags Key=Environment,Value=testing (instance only)�[0m
+�[2m· 6j final describe-tags resource-id=i-0c615e5dd74050f53 want=1�[0m
+    tags_test.go:171: �[2m· 6j final describe-tags resource-id=i-0c615e5dd74050f53 want=1�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7_StopStart" classname="" time="8.090">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7 — Instance State Transitions (Stop) ━━�[0m
+�[2m· stop-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:30: �[2m· stop-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:36: instance i-0c615e5dd74050f53 reached state stopped
+�[2m· reboot-instances on stopped instance (should fail)�[0m
+    lifecycle_test.go:40: �[2m· reboot-instances on stopped instance (should fail)�[0m
+�[2mstopped_reject�[0m=IncorrectInstanceState
+    lifecycle_test.go:47: �[2mstopped_reject�[0m=IncorrectInstanceState]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7a_AttachToStoppedError" classname="" time="0.300">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7a — Attach Volume to Stopped Instance (Error Path) ━━�[0m
+�[2m· create-volume size=10 az=ap-southeast-2a�[0m
+    lifecycle_test.go:60: �[2m· create-volume size=10 az=ap-southeast-2a�[0m
+�[2mtest_volume�[0m=vol-8d3e1335938a378cd
+    lifecycle_test.go:68: �[2mtest_volume�[0m=vol-8d3e1335938a378cd
+    lifecycle_test.go:82: volume vol-8d3e1335938a378cd reached state available
+�[2m· attach-volume vol-8d3e1335938a378cd -> i-0c615e5dd74050f53 (should fail)�[0m
+    lifecycle_test.go:84: �[2m· attach-volume vol-8d3e1335938a378cd -> i-0c615e5dd74050f53 (should fail)�[0m
+�[2m· delete-volume vol-8d3e1335938a378cd�[0m
+    lifecycle_test.go:94: �[2m· delete-volume vol-8d3e1335938a378cd�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7b_ModifyInstanceAttribute" classname="" time="55.620">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7b — ModifyInstanceAttribute ━━�[0m
+�[2mfrom_type�[0m=t3.nano �[2mto_type�[0m=t3.small
+    lifecycle_test.go:120: �[2mfrom_type�[0m=t3.nano �[2mto_type�[0m=t3.small
+�[2mexpected_vcpus�[0m=2 �[2mexpected_mem_mib�[0m=2048
+    lifecycle_test.go:142: �[2mexpected_vcpus�[0m=2 �[2mexpected_mem_mib�[0m=2048
+�[2m· modify-instance-attribute i-0c615e5dd74050f53 type=t3.small�[0m
+    lifecycle_test.go:144: �[2m· modify-instance-attribute i-0c615e5dd74050f53 type=t3.small�[0m
+�[2m· start-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:165: �[2m· start-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:171: instance i-0c615e5dd74050f53 reached state running
+�[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
+    lifecycle_test.go:177: �[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    lifecycle_test.go:179: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2m· ssh nproc�[0m
+    lifecycle_test.go:182: �[2m· ssh nproc�[0m
+�[2mvm_vcpus�[0m=2
+    lifecycle_test.go:188: �[2mvm_vcpus�[0m=2
+�[2m· ssh MemTotal�[0m
+    lifecycle_test.go:190: �[2m· ssh MemTotal�[0m
+�[2mvm_mem_mib�[0m=1967 �[2mthreshold_mib�[0m=1740
+    lifecycle_test.go:200: �[2mvm_mem_mib�[0m=1967 �[2mthreshold_mib�[0m=1740
+�[2m· stop-instances i-0c615e5dd74050f53 (end of 7b)�[0m
+    lifecycle_test.go:206: �[2m· stop-instances i-0c615e5dd74050f53 (end of 7b)�[0m
+    lifecycle_test.go:211: instance i-0c615e5dd74050f53 reached state stopped]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7c_pre_Reboot" classname="" time="83.480">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7c-pre — Reboot Running Instance ━━�[0m
+�[2m· start-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:226: �[2m· start-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:231: instance i-0c615e5dd74050f53 reached state running
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    lifecycle_test.go:236: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2mpre_reboot_private_ip�[0m=172.31.0.4
+    lifecycle_test.go:246: �[2mpre_reboot_private_ip�[0m=172.31.0.4
+�[2m· reboot-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:248: �[2m· reboot-instances i-0c615e5dd74050f53�[0m
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    lifecycle_test.go:279: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2m· ssh uptime�[0m
+    lifecycle_test.go:282: �[2m· ssh uptime�[0m
+�[2muptime_secs�[0m=25
+    lifecycle_test.go:288: �[2muptime_secs�[0m=25]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7c_RunInstancesMultiCount" classname="" time="47.560">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7c — RunInstances with MinCount/MaxCount > 1 ━━�[0m
+�[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.small count=2�[0m
+    lifecycle_test.go:309: �[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.small count=2�[0m
+�[2msibling_1�[0m=i-9bfadd8ebd378f846 �[2msibling_2�[0m=i-448a9e2bf45c58658
+    lifecycle_test.go:327: �[2msibling_1�[0m=i-9bfadd8ebd378f846 �[2msibling_2�[0m=i-448a9e2bf45c58658
+    lifecycle_test.go:342: instance i-9bfadd8ebd378f846 reached state running
+    lifecycle_test.go:342: instance i-448a9e2bf45c58658 reached state running
+�[2m· terminate-instances i-9bfadd8ebd378f846 i-448a9e2bf45c58658�[0m
+    lifecycle_test.go:345: �[2m· terminate-instances i-9bfadd8ebd378f846 i-448a9e2bf45c58658�[0m
+    lifecycle_test.go:352: instance i-9bfadd8ebd378f846 reached state terminated
+    lifecycle_test.go:352: instance i-448a9e2bf45c58658 reached state terminated]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths" classname="" time="0.000">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8 — Negative / Error Path Tests ━━�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8a_InvalidAMIIDMalformed" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· run-instances image-id=notanami�[0m
+    negative_test.go:55: �[2m· run-instances image-id=notanami�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8b_InvalidInstanceType" classname="" time="0.210">
+			<system-out><![CDATA[�[2m· run-instances instance-type=x99.superlarge�[0m
+    negative_test.go:70: �[2m· run-instances instance-type=x99.superlarge�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8c_VolumeInUse" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· attach-volume vol-2230138124fb51ee2 (root, in-use)�[0m
+    negative_test.go:85: �[2m· attach-volume vol-2230138124fb51ee2 (root, in-use)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8d_DetachRootForbidden" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· detach-volume vol-2230138124fb51ee2 (root)�[0m
+    negative_test.go:97: �[2m· detach-volume vol-2230138124fb51ee2 (root)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8e_DeleteSnapshotNotFound" classname="" time="0.030">
+			<system-out><![CDATA[�[2m· delete-snapshot snap-nonexistent000000�[0m
+    negative_test.go:108: �[2m· delete-snapshot snap-nonexistent000000�[0m
+�[2m· describe-volumes vol-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8f_InvalidActionRawHTTP" classname="" time="0.040">
+			<system-out><![CDATA[�[2m· POST Action=DescribeFakeThings�[0m
+    negative_test.go:122: �[2m· POST Action=DescribeFakeThings�[0m
+    negative_test.go:125: �[2mstatus�[0m=400 �[2mcode�[0m=InvalidAction]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8g_InvalidAMIIDNotFound" classname="" time="0.240">
+			<system-out><![CDATA[�[2m· reboot-instances i-nonexistent�[0m
+�[2m· run-instances image-id=ami-0000000000000dead�[0m
+    negative_test.go:138: �[2m· run-instances image-id=ami-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8h_InvalidKeyPairNotFound" classname="" time="0.260">
+			<system-out><![CDATA[�[2m· run-instances key-name=nonexistent-key-xyz�[0m
+    negative_test.go:153: �[2m· run-instances key-name=nonexistent-key-xyz�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8i_DeleteVolumeNotFound" classname="" time="0.040">
+			<system-out><![CDATA[�[2m· delete-volume vol-0000000000000dead�[0m
+    negative_test.go:168: �[2m· delete-volume vol-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8j_CreateKeyPairDuplicate" classname="" time="0.040">
+			<system-out><![CDATA[    negative_test.go:178: �[2m· create-key-pair test-key-1 (duplicate)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8k_ImportKeyPairDuplicate" classname="" time="0.040">
+			<system-out><![CDATA[�[2m· import-key-pair test-key-1 (duplicate)�[0m
+    negative_test.go:198: �[2m· import-key-pair test-key-1 (duplicate)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8l_InvalidKeyFormat" classname="" time="0.040">
+			<system-out><![CDATA[    negative_test.go:210: �[2m· import-key-pair bad-format-key (bad material)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8m_DescribeVolumesNotFound" classname="" time="0.040">
+			<system-out><![CDATA[    negative_test.go:230: �[2m· describe-volumes vol-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8n_DescribeImagesNotFound" classname="" time="0.040">
+			<system-out><![CDATA[�[2m· describe-images ami-0000000000000dead�[0m
+    negative_test.go:240: �[2m· describe-images ami-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8o_CreateImageDuplicateName" classname="" time="0.560">
+			<system-out><![CDATA[�[2m· create-key-pair test-key-1 (duplicate)�[0m
+�[2m· create-image name=e2e-custom-ami (duplicate)�[0m
+�[2m· import-key-pair bad-format-key (bad material)�[0m
+    negative_test.go:252: �[2m· create-image name=e2e-custom-ami (duplicate)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8p_DeleteKeyPairIdempotent" classname="" time="0.020">
+			<system-out><![CDATA[�[2m· delete-key-pair nonexistent-key-99999 (idempotent)�[0m
+    negative_test.go:276: �[2m· delete-key-pair nonexistent-key-99999 (idempotent)�[0m
+�[2mstatus�[0m=400 �[2mcode�[0m=InvalidAction]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8q_ModifyAttributeOnRunning" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· modify-instance-attribute i-0c615e5dd74050f53 (running)�[0m
+    negative_test.go:289: �[2m· modify-instance-attribute i-0c615e5dd74050f53 (running)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8r_RebootInstanceNotFound" classname="" time="0.040">
+			<system-out><![CDATA[    negative_test.go:302: �[2m· reboot-instances i-nonexistent�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM1_UserCRUD" classname="" time="0.070">
+			<system-out><![CDATA[
+�[1m�[36m━━ IAM Phase 1 — User CRUD ━━�[0m
+�[2m· list-users (root auth sanity)�[0m
+    iam_test.go:109: �[2m· list-users (root auth sanity)�[0m
+�[2m· create-user "alice"�[0m
+    iam_test.go:114: �[2m· create-user "alice"�[0m
+�[2muser�[0m=alice �[2marn�[0m=arn:aws:iam::000000000001:user/alice
+    iam_test.go:120: �[2muser�[0m=alice �[2marn�[0m=arn:aws:iam::000000000001:user/alice
+�[2m· create-user "bob" path="/engineering/"�[0m
+    iam_test.go:122: �[2m· create-user "bob" path="/engineering/"�[0m
+�[2m· create-user "alice" again (expect EntityAlreadyExists)�[0m
+    iam_test.go:131: �[2m· create-user "alice" again (expect EntityAlreadyExists)�[0m
+�[2m· get-user "alice"�[0m
+    iam_test.go:137: �[2m· get-user "alice"�[0m
+�[2m· get-user nonexistent (expect NoSuchEntity)�[0m
+    iam_test.go:142: �[2m· get-user nonexistent (expect NoSuchEntity)�[0m
+�[2m· list-users (>=3 root+alice+bob)�[0m
+    iam_test.go:148: �[2m· list-users (>=3 root+alice+bob)�[0m
+�[2m· list-users --path-prefix "/engineering/" (expect 1)�[0m
+    iam_test.go:154: �[2m· list-users --path-prefix "/engineering/" (expect 1)�[0m
+    iam_test.go:193: �[2m· create-user "iam-phase1-ephemeral" (delete idempotency probe)�[0m
+�[2m· delete-user "iam-phase1-ephemeral"�[0m
+    iam_test.go:197: �[2m· delete-user "iam-phase1-ephemeral"�[0m
+�[2m· delete-user "iam-phase1-ephemeral" again (expect NoSuchEntity)�[0m
+    iam_test.go:201: �[2m· delete-user "iam-phase1-ephemeral" again (expect NoSuchEntity)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM1_UserCRUD/UpdateUser" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[    iam_test.go:165: daemon UpdateUser handler not implemented — InvalidAction
+�[2m· create-user "iam-phase1-ephemeral" (delete idempotency probe)�[0m]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM2_AccessKeyLifecycle" classname="" time="0.040">
+			<system-out><![CDATA[
+�[1m�[36m━━ IAM Phase 2 — Access Key Lifecycle ━━�[0m
+�[2m· create-access-key user=alice (key 1)�[0m
+    iam_test.go:218: �[2m· create-access-key user=alice (key 1)�[0m
+�[2mkey1�[0m=AKIA4984AE57CF8D721A879A
+    iam_test.go:227: �[2mkey1�[0m=AKIA4984AE57CF8D721A879A
+�[2m· create-access-key user=alice (key 2)�[0m
+    iam_test.go:229: �[2m· create-access-key user=alice (key 2)�[0m
+�[2mkey2�[0m=AKIAA1CD38D4F0E573216E11
+    iam_test.go:236: �[2mkey2�[0m=AKIAA1CD38D4F0E573216E11
+�[2m· create-access-key user=alice key 3 (expect LimitExceeded)�[0m
+    iam_test.go:238: �[2m· create-access-key user=alice key 3 (expect LimitExceeded)�[0m
+�[2m· create-access-key user=ghost (expect NoSuchEntity)�[0m
+    iam_test.go:246: �[2m· create-access-key user=ghost (expect NoSuchEntity)�[0m
+�[2m· list-access-keys user=alice (expect 2)�[0m
+    iam_test.go:254: �[2m· list-access-keys user=alice (expect 2)�[0m
+�[2m· list-access-keys user=bob (expect 0)�[0m
+    iam_test.go:261: �[2m· list-access-keys user=bob (expect 0)�[0m
+�[2m· update-access-key AKIA4984AE57CF8D721A879A -> Inactive�[0m
+    iam_test.go:268: �[2m· update-access-key AKIA4984AE57CF8D721A879A -> Inactive�[0m
+�[2m· update-access-key AKIA4984AE57CF8D721A879A -> Active�[0m
+    iam_test.go:279: �[2m· update-access-key AKIA4984AE57CF8D721A879A -> Active�[0m
+�[2m· delete-access-key AKIAA1CD38D4F0E573216E11 (key 2)�[0m
+    iam_test.go:287: �[2m· delete-access-key AKIAA1CD38D4F0E573216E11 (key 2)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM3_UserAuthentication" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[
+�[1m�[36m━━ IAM Phase 3 — User Authentication ━━�[0m
+    iam_test.go:311: daemon IAM signature/principal lookup not implemented — mulga-siv-100]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD" classname="" time="0.040">
+			<system-out><![CDATA[
+�[1m�[36m━━ IAM Phase 4 — Policy CRUD ━━�[0m
+�[2m· create-policy EC2ReadOnly�[0m
+    iam_test.go:384: �[2m· create-policy EC2ReadOnly�[0m
+�[2mpolicy�[0m=EC2ReadOnly �[2marn�[0m=arn:aws:iam::000000000001:policy/EC2ReadOnly �[2maccount�[0m=000000000001
+    iam_test.go:393: �[2mpolicy�[0m=EC2ReadOnly �[2marn�[0m=arn:aws:iam::000000000001:policy/EC2ReadOnly �[2maccount�[0m=000000000001
+    iam_test.go:440: �[2m· create-policy duplicate (expect EntityAlreadyExists)�[0m
+�[2m· create-policy malformed (expect MalformedPolicyDocument)�[0m
+    iam_test.go:449: �[2m· create-policy malformed (expect MalformedPolicyDocument)�[0m
+�[2m· get-policy EC2ReadOnly�[0m
+    iam_test.go:458: �[2m· get-policy EC2ReadOnly�[0m
+�[2m· get-policy nonexistent (expect NoSuchEntity)�[0m
+    iam_test.go:463: �[2m· get-policy nonexistent (expect NoSuchEntity)�[0m
+�[2m· get-policy-version EC2ReadOnly v1�[0m
+    iam_test.go:470: �[2m· get-policy-version EC2ReadOnly v1�[0m
+    iam_test.go:519: �[2m· list-policies (>=5)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel" classname="" time="0.000"></testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/FullAdmin" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· create-policy FullAdmin path=/admin/�[0m
+    iam_test.go:402: �[2m· create-policy FullAdmin path=/admin/�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/DenyTerminate" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· create-policy DenyTerminate (mixed Allow+Deny)�[0m
+    iam_test.go:413: �[2m· create-policy DenyTerminate (mixed Allow+Deny)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/IAMReadOnly" classname="" time="0.020">
+			<system-out><![CDATA[�[2m· create-policy IAMReadOnly�[0m
+    iam_test.go:422: �[2m· create-policy IAMReadOnly�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/EC2DescribeAll" classname="" time="0.020">
+			<system-out><![CDATA[�[2m· create-policy EC2DescribeAll (wildcard)�[0m
+    iam_test.go:431: �[2m· create-policy EC2DescribeAll (wildcard)�[0m
+�[2m· create-policy duplicate (expect EntityAlreadyExists)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/PolicyVersionLifecycle" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[    iam_test.go:484: daemon CreatePolicyVersion handler not implemented — InvalidAction
+�[2m· list-policies (>=5)�[0m]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM5_PolicyAttachmentEnforcement" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[
+�[1m�[36m━━ IAM Phase 5 — Policy Attachment & Enforcement ━━�[0m
+    iam_test.go:535: daemon IAM scoped-credential enforcement not implemented — mulga-siv-100]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM6_PolicyLifecycle" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[
+�[1m�[36m━━ IAM Phase 6 — Policy Lifecycle (Detach & Delete) ━━�[0m
+    iam_test.go:718: daemon DetachUserPolicy not implemented — mulga-siv-100]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM7_Cleanup" classname="" time="0.050">
+			<system-out><![CDATA[
+�[1m�[36m━━ IAM Phase 7 — Cleanup ━━�[0m
+�[2m· cleanup user alice�[0m
+    iam_test.go:770: �[2m· cleanup user alice�[0m
+�[2m· cleanup user bob�[0m
+    iam_test.go:770: �[2m· cleanup user bob�[0m
+�[2m· cleanup user charlie�[0m
+    iam_test.go:770: �[2m· cleanup user charlie�[0m
+�[2mremaining_users�[0m=1 �[2mnames�[0m=[admin]
+    iam_test.go:779: �[2mremaining_users�[0m=1 �[2mnames�[0m=[admin]
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2ReadOnly�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2ReadOnly�[0m
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/admin/FullAdmin�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/admin/FullAdmin�[0m
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/IAMReadOnly�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/IAMReadOnly�[0m
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2DescribeAll�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2DescribeAll�[0m
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/DenyTerminate�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/DenyTerminate�[0m
+�[2mremaining_policies�[0m=1 �[2mnames�[0m=[AdministratorAccess]
+    iam_test.go:812: �[2mremaining_policies�[0m=1 �[2mnames�[0m=[AdministratorAccess]]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping" classname="" time="114.890">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8 (acct) — EC2 Account Scoping ━━�[0m
+    account_test.go:75: �[2m· Phase 8 acct cleanup (deferred)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step1_AccountSetup" classname="" time="1.220">
+			<system-out><![CDATA[�[2m· create account 'Team Alpha'�[0m
+    account_test.go:206: �[2m· create account 'Team Alpha'�[0m
+�[2malpha_account�[0m=000000000002 �[2malpha_key�[0m=AKIA374CBBB2A15D3E2EE822
+    account_test.go:211: �[2malpha_account�[0m=000000000002 �[2malpha_key�[0m=AKIA374CBBB2A15D3E2EE822
+�[2m· create account 'Team Beta'�[0m
+    account_test.go:213: �[2m· create account 'Team Beta'�[0m
+�[2mbeta_account�[0m=000000000003 �[2mbeta_key�[0m=AKIA0AA0B2845E3BD5A26D01
+    account_test.go:218: �[2mbeta_account�[0m=000000000003 �[2mbeta_key�[0m=AKIA0AA0B2845E3BD5A26D01]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step2_InstanceScoping" classname="" time="52.380">
+			<system-out><![CDATA[�[2m· create per-account key pairs�[0m
+    account_test.go:242: �[2m· create per-account key pairs�[0m
+�[2m· alpha run-instances�[0m
+    account_test.go:250: �[2m· alpha run-instances�[0m
+�[2malpha_instance�[0m=i-a3072c2387630b8ee
+    account_test.go:262: �[2malpha_instance�[0m=i-a3072c2387630b8ee
+�[2m· beta run-instances�[0m
+    account_test.go:264: �[2m· beta run-instances�[0m
+�[2mbeta_instance�[0m=i-ad32cbb0afaee5eee
+    account_test.go:276: �[2mbeta_instance�[0m=i-ad32cbb0afaee5eee
+    account_test.go:278: instance i-a3072c2387630b8ee reached state running
+    account_test.go:279: instance i-ad32cbb0afaee5eee reached state running
+�[2m· alpha sees only own instances�[0m
+    account_test.go:281: �[2m· alpha sees only own instances�[0m
+�[2m· beta sees only own instances�[0m
+    account_test.go:285: �[2m· beta sees only own instances�[0m
+�[2m· alpha OwnerId matches alpha account�[0m
+    account_test.go:289: �[2m· alpha OwnerId matches alpha account�[0m
+�[2m· cross-account stop alpha->beta blocked�[0m
+    account_test.go:296: �[2m· cross-account stop alpha->beta blocked�[0m
+�[2m· cross-account terminate beta->alpha blocked�[0m
+    account_test.go:304: �[2m· cross-account terminate beta->alpha blocked�[0m
+�[2m· cross-account reboot alpha->beta blocked�[0m
+    account_test.go:312: �[2m· cross-account reboot alpha->beta blocked�[0m
+�[2m· stop alpha instance for cross-account start/modify/console tests�[0m
+    account_test.go:321: �[2m· stop alpha instance for cross-account start/modify/console tests�[0m
+    account_test.go:326: instance i-a3072c2387630b8ee reached state stopped
+�[2m· cross-account start beta->alpha blocked�[0m
+    account_test.go:328: �[2m· cross-account start beta->alpha blocked�[0m
+�[2m· cross-account modify-instance-attribute beta->alpha blocked�[0m
+    account_test.go:336: �[2m· cross-account modify-instance-attribute beta->alpha blocked�[0m
+�[2m· cross-account get-console-output beta->alpha blocked�[0m
+    account_test.go:345: �[2m· cross-account get-console-output beta->alpha blocked�[0m
+�[2m· restart alpha instance�[0m
+    account_test.go:354: �[2m· restart alpha instance�[0m
+    account_test.go:359: instance i-a3072c2387630b8ee reached state running]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step3_VolumeScoping" classname="" time="24.140">
+			<system-out><![CDATA[�[2m· alpha create-volume�[0m
+    account_test.go:366: �[2m· alpha create-volume�[0m
+�[2malpha_vol�[0m=vol-ceef82652369780af
+    account_test.go:375: �[2malpha_vol�[0m=vol-ceef82652369780af
+�[2m· beta create-volume�[0m
+    account_test.go:377: �[2m· beta create-volume�[0m
+�[2mbeta_vol�[0m=vol-08024f32d23d9e18c
+    account_test.go:386: �[2mbeta_vol�[0m=vol-08024f32d23d9e18c
+    account_test.go:388: volume vol-ceef82652369780af reached state available
+    account_test.go:389: volume vol-08024f32d23d9e18c reached state available
+�[2m· alpha sees only own volumes�[0m
+    account_test.go:391: �[2m· alpha sees only own volumes�[0m
+�[2m· cross-account describe-volume by id blocked�[0m
+    account_test.go:395: �[2m· cross-account describe-volume by id blocked�[0m
+�[2m· cross-account delete-volume blocked�[0m
+    account_test.go:403: �[2m· cross-account delete-volume blocked�[0m
+�[2m· cross-account attach-volume (beta attaches alpha's vol) blocked�[0m
+    account_test.go:411: �[2m· cross-account attach-volume (beta attaches alpha's vol) blocked�[0m
+�[2m· alpha attach-volume to own instance�[0m
+    account_test.go:422: �[2m· alpha attach-volume to own instance�[0m
+    account_test.go:429: volume vol-ceef82652369780af reached state in-use
+�[2m· cross-account detach-volume blocked�[0m
+    account_test.go:431: �[2m· cross-account detach-volume blocked�[0m
+�[2m· cross-account modify-volume blocked�[0m
+    account_test.go:439: �[2m· cross-account modify-volume blocked�[0m
+�[2m· alpha detach own volume�[0m
+    account_test.go:449: �[2m· alpha detach own volume�[0m
+    account_test.go:454: volume vol-ceef82652369780af reached state available]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step4_KeyPairScoping" classname="" time="0.370">
+			<system-out><![CDATA[�[2m· alpha create-key-pair alpha-key�[0m
+    account_test.go:466: �[2m· alpha create-key-pair alpha-key�[0m
+�[2m· beta create-key-pair beta-key�[0m
+    account_test.go:473: �[2m· beta create-key-pair beta-key�[0m
+�[2m· alpha sees only own keys�[0m
+    account_test.go:478: �[2m· alpha sees only own keys�[0m
+�[2m· namespace isolation: shared-name in both accounts�[0m
+    account_test.go:483: �[2m· namespace isolation: shared-name in both accounts�[0m
+�[2m· beta deletes alpha-key (idempotent, no cross-account effect)�[0m
+    account_test.go:497: �[2m· beta deletes alpha-key (idempotent, no cross-account effect)�[0m
+�[2m· alpha import-key-pair imported-key�[0m
+    account_test.go:503: �[2m· alpha import-key-pair imported-key�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step5_SnapshotScoping" classname="" time="0.060">
+			<system-out><![CDATA[�[2m· alpha create-snapshot�[0m
+    account_test.go:522: �[2m· alpha create-snapshot�[0m
+�[2malpha_snap�[0m=snap-2a47896f4304d805c
+    account_test.go:530: �[2malpha_snap�[0m=snap-2a47896f4304d805c
+�[2m· beta create-snapshot�[0m
+    account_test.go:532: �[2m· beta create-snapshot�[0m
+�[2mbeta_snap�[0m=snap-bf5744e78bd7c6908
+    account_test.go:540: �[2mbeta_snap�[0m=snap-bf5744e78bd7c6908
+�[2m· alpha sees only own snapshots (owner-ids=self)�[0m
+    account_test.go:542: �[2m· alpha sees only own snapshots (owner-ids=self)�[0m
+�[2m· cross-account delete-snapshot blocked (UnauthorizedOperation)�[0m
+    account_test.go:560: �[2m· cross-account delete-snapshot blocked (UnauthorizedOperation)�[0m
+�[2m· cross-account create-snapshot of alpha's volume blocked�[0m
+    account_test.go:568: �[2m· cross-account create-snapshot of alpha's volume blocked�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step6_VPCSubnetScoping" classname="" time="0.030">
+			<system-out><![CDATA[�[2m· alpha create-vpc 10.0.0.0/16�[0m
+    account_test.go:582: �[2m· alpha create-vpc 10.0.0.0/16�[0m
+�[2malpha_vpc�[0m=vpc-7f60df5e87183fb59
+    account_test.go:589: �[2malpha_vpc�[0m=vpc-7f60df5e87183fb59
+�[2m· beta create-vpc 10.0.0.0/16 (same CIDR, no conflict)�[0m
+    account_test.go:591: �[2m· beta create-vpc 10.0.0.0/16 (same CIDR, no conflict)�[0m
+�[2mbeta_vpc�[0m=vpc-7e8c859d3985c8218
+    account_test.go:598: �[2mbeta_vpc�[0m=vpc-7e8c859d3985c8218
+�[2m· alpha describe-vpcs isolation�[0m
+    account_test.go:600: �[2m· alpha describe-vpcs isolation�[0m
+�[2m· cross-account describe-vpc by id blocked�[0m
+    account_test.go:605: �[2m· cross-account describe-vpc by id blocked�[0m
+�[2m· cross-account delete-vpc blocked�[0m
+    account_test.go:613: �[2m· cross-account delete-vpc blocked�[0m
+�[2m· alpha create-subnet 10.0.1.0/24�[0m
+    account_test.go:621: �[2m· alpha create-subnet 10.0.1.0/24�[0m
+�[2m· beta create-subnet 10.0.1.0/24�[0m
+    account_test.go:630: �[2m· beta create-subnet 10.0.1.0/24�[0m
+�[2m· alpha describe-subnets isolation�[0m
+    account_test.go:639: �[2m· alpha describe-subnets isolation�[0m
+�[2m· cross-account create-subnet in other VPC blocked�[0m
+    account_test.go:648: �[2m· cross-account create-subnet in other VPC blocked�[0m
+�[2m· cross-account delete-subnet blocked�[0m
+    account_test.go:657: �[2m· cross-account delete-subnet blocked�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step7_IGWEIGWScoping" classname="" time="0.030">
+			<system-out><![CDATA[�[2m· alpha create-internet-gateway�[0m
+    account_test.go:673: �[2m· alpha create-internet-gateway�[0m
+�[2malpha_igw�[0m=igw-ee966e3e923caf630
+    account_test.go:678: �[2malpha_igw�[0m=igw-ee966e3e923caf630
+�[2m· beta create-internet-gateway�[0m
+    account_test.go:680: �[2m· beta create-internet-gateway�[0m
+�[2mbeta_igw�[0m=igw-b46e66f2065889bb9
+    account_test.go:685: �[2mbeta_igw�[0m=igw-b46e66f2065889bb9
+�[2m· alpha describe-internet-gateways isolation�[0m
+    account_test.go:687: �[2m· alpha describe-internet-gateways isolation�[0m
+�[2m· cross-account describe IGW by id blocked�[0m
+    account_test.go:696: �[2m· cross-account describe IGW by id blocked�[0m
+�[2m· cross-account delete IGW blocked�[0m
+    account_test.go:704: �[2m· cross-account delete IGW blocked�[0m
+�[2m· cross-account attach IGW (alpha attaches beta's IGW) blocked�[0m
+    account_test.go:712: �[2m· cross-account attach IGW (alpha attaches beta's IGW) blocked�[0m
+�[2m· alpha attach own IGW to own VPC�[0m
+    account_test.go:722: �[2m· alpha attach own IGW to own VPC�[0m
+�[2m· cross-account detach IGW blocked�[0m
+    account_test.go:729: �[2m· cross-account detach IGW blocked�[0m
+�[2m· alpha create-egress-only-internet-gateway�[0m
+    account_test.go:739: �[2m· alpha create-egress-only-internet-gateway�[0m
+�[2malpha_eigw�[0m=eigw-51ba82da927e49393
+    account_test.go:746: �[2malpha_eigw�[0m=eigw-51ba82da927e49393
+�[2m· beta create-egress-only-internet-gateway�[0m
+    account_test.go:748: �[2m· beta create-egress-only-internet-gateway�[0m
+�[2mbeta_eigw�[0m=eigw-e88d642e78ab4fe5a
+    account_test.go:755: �[2mbeta_eigw�[0m=eigw-e88d642e78ab4fe5a
+�[2m· alpha describe-eigws isolation�[0m
+    account_test.go:757: �[2m· alpha describe-eigws isolation�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step8_AccountSettings" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· alpha enable-ebs-encryption-by-default�[0m
+    account_test.go:794: �[2m· alpha enable-ebs-encryption-by-default�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step9_GlobalResources" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· describe-regions identical across accounts�[0m
+    account_test.go:829: �[2m· describe-regions identical across accounts�[0m
+�[2m· describe-availability-zones identical across accounts�[0m
+    account_test.go:834: �[2m· describe-availability-zones identical across accounts�[0m
+�[2m· describe-instance-types identical across accounts�[0m
+    account_test.go:839: �[2m· describe-instance-types identical across accounts�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step10_EdgeCases" classname="" time="1.200">
+			<system-out><![CDATA[�[2m· create empty Gamma account�[0m
+    account_test.go:849: �[2m· create empty Gamma account�[0m
+�[2mgamma_account�[0m=000000000004
+    account_test.go:852: �[2mgamma_account�[0m=000000000004
+�[2m· root create-key-pair root-scoping-key�[0m
+    account_test.go:879: �[2m· root create-key-pair root-scoping-key�[0m
+�[2m· non-existent volume id -> InvalidVolume.NotFound�[0m
+    account_test.go:895: �[2m· non-existent volume id -> InvalidVolume.NotFound�[0m
+�[2m· non-existent snapshot id -> InvalidSnapshot.NotFound�[0m
+    account_test.go:903: �[2m· non-existent snapshot id -> InvalidSnapshot.NotFound�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step11_Cleanup" classname="" time="32.440">
+			<system-out><![CDATA[�[2m· terminate alpha instance�[0m
+    account_test.go:920: �[2m· terminate alpha instance�[0m
+�[2m· terminate beta instance�[0m
+    account_test.go:926: �[2m· terminate beta instance�[0m
+�[2m· delete snapshots�[0m
+    account_test.go:939: �[2m· delete snapshots�[0m
+�[2m· delete volumes�[0m
+    account_test.go:955: �[2m· delete volumes�[0m
+�[2m· delete key pairs�[0m
+    account_test.go:969: �[2m· delete key pairs�[0m
+�[2m· delete EIGWs�[0m
+    account_test.go:979: �[2m· delete EIGWs�[0m
+�[2m· detach + delete IGWs�[0m
+    account_test.go:993: �[2m· detach + delete IGWs�[0m
+�[2m· delete subnets�[0m
+    account_test.go:1013: �[2m· delete subnets�[0m
+�[2m· delete VPCs�[0m
+    account_test.go:1027: �[2m· delete VPCs�[0m
+�[2m· Phase 8 acct cleanup (deferred)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8b_VPCSubnetE2E" classname="" time="200.480">
+			<failure message="Failed"><![CDATA[
+�[1m�[36m━━ Phase 8b — VPC Public/Private Subnet E2E ━━�[0m
+�[2m· create-vpc 10.99.0.0/16�[0m
+    vpc_test.go:56: �[2m· create-vpc 10.99.0.0/16�[0m
+�[2mvpc�[0m=vpc-39eff985799c43069 �[2mcidr�[0m=10.99.0.0/16
+    vpc_test.go:67: �[2mvpc�[0m=vpc-39eff985799c43069 �[2mcidr�[0m=10.99.0.0/16
+�[2m· create-subnet 10.99.1.0/24 (public)�[0m
+    vpc_test.go:70: �[2m· create-subnet 10.99.1.0/24 (public)�[0m
+�[2m· modify-subnet-attribute MapPublicIpOnLaunch=true subnet-22aba054e10588254�[0m
+    vpc_test.go:87: �[2m· modify-subnet-attribute MapPublicIpOnLaunch=true subnet-22aba054e10588254�[0m
+�[2mpub_subnet�[0m=subnet-22aba054e10588254 �[2mmap_public_ip�[0m=true
+    vpc_test.go:93: �[2mpub_subnet�[0m=subnet-22aba054e10588254 �[2mmap_public_ip�[0m=true
+�[2m· create-subnet 10.99.2.0/24 (private)�[0m
+    vpc_test.go:96: �[2m· create-subnet 10.99.2.0/24 (private)�[0m
+�[2mpriv_subnet�[0m=subnet-f53aeddb73c134696 �[2mmap_public_ip�[0m=false
+    vpc_test.go:111: �[2mpriv_subnet�[0m=subnet-f53aeddb73c134696 �[2mmap_public_ip�[0m=false
+�[2m· create-internet-gateway�[0m
+    vpc_test.go:114: �[2m· create-internet-gateway�[0m
+�[2m· attach-internet-gateway igw-1520b5a34480da982 -> vpc-39eff985799c43069�[0m
+    vpc_test.go:133: �[2m· attach-internet-gateway igw-1520b5a34480da982 -> vpc-39eff985799c43069�[0m
+�[2migw�[0m=igw-1520b5a34480da982
+    vpc_test.go:139: �[2migw�[0m=igw-1520b5a34480da982
+�[2m· create-route-table (vpc=vpc-39eff985799c43069)�[0m
+    vpc_test.go:142: �[2m· create-route-table (vpc=vpc-39eff985799c43069)�[0m
+�[2m· associate-route-table rtb-662df5e424cbdb60c <- subnet-22aba054e10588254�[0m
+    vpc_test.go:159: �[2m· associate-route-table rtb-662df5e424cbdb60c <- subnet-22aba054e10588254�[0m
+�[2m· create-route 0.0.0.0/0 -> igw-1520b5a34480da982�[0m
+    vpc_test.go:176: �[2m· create-route 0.0.0.0/0 -> igw-1520b5a34480da982�[0m
+�[2mrtb�[0m=rtb-662df5e424cbdb60c �[2massoc�[0m=rtbassoc-5510bf1dc6924eb0b �[2mroute�[0m=0.0.0.0/0->igw-1520b5a34480da982
+    vpc_test.go:192: �[2mrtb�[0m=rtb-662df5e424cbdb60c �[2massoc�[0m=rtbassoc-5510bf1dc6924eb0b �[2mroute�[0m=0.0.0.0/0->igw-1520b5a34480da982
+�[2m· run-instances ami=ami-6f60f2b3bf04b5453 subnet=subnet-22aba054e10588254�[0m
+    vpc_test.go:195: �[2m· run-instances ami=ami-6f60f2b3bf04b5453 subnet=subnet-22aba054e10588254�[0m
+�[2minstance�[0m=i-3177888cba3796d42
+    vpc_test.go:216: �[2minstance�[0m=i-3177888cba3796d42
+    vpc_test.go:218: instance i-3177888cba3796d42 reached state running
+�[2mpublic_ip�[0m=192.168.0.202 �[2mprivate_ip�[0m=10.99.1.4
+    vpc_test.go:224: �[2mpublic_ip�[0m=192.168.0.202 �[2mprivate_ip�[0m=10.99.1.4
+�[2m· waiting for SSH handshake 192.168.0.202:22�[0m
+    vpc_test.go:227: �[2m· waiting for SSH handshake 192.168.0.202:22�[0m
+    vpc_test.go:227: Eventually: condition not met within 3m0s: [SSH handshake 192.168.0.202:22 never completed]]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8c_RouteTableValidation" classname="" time="0.030">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8c — Route Table Validation ━━�[0m
+�[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63
+    vpc_test.go:292: �[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8c_RouteTableValidation/Step1_DefaultMainRouteTable" classname="" time="0.000">
+			<system-out><![CDATA[�[2m· describe-route-tables filter=vpc-id,association.main�[0m
+    vpc_test.go:297: �[2m· describe-route-tables filter=vpc-id,association.main�[0m
+�[2mmain_rtb�[0m=rtb-e42012569f92f7184
+    vpc_test.go:309: �[2mmain_rtb�[0m=rtb-e42012569f92f7184
+�[2mlocal_route�[0m=172.31.0.0/16
+    vpc_test.go:332: �[2mlocal_route�[0m=172.31.0.0/16
+�[2mdefault_route�[0m=0.0.0.0/0->igw-79c2e643660b4b7e3
+    vpc_test.go:350: �[2mdefault_route�[0m=0.0.0.0/0->igw-79c2e643660b4b7e3]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8c_RouteTableValidation/Step2_CustomRouteTableLifecycle" classname="" time="0.020">
+			<system-out><![CDATA[�[2m· create-route-table (vpc=vpc-531252db3b4e1fc08)�[0m
+    vpc_test.go:363: �[2m· create-route-table (vpc=vpc-531252db3b4e1fc08)�[0m
+�[2mcustom_rtb�[0m=rtb-5e6c46936f3e44167
+    vpc_test.go:379: �[2mcustom_rtb�[0m=rtb-5e6c46936f3e44167
+�[2mcustom_local_route�[0m=172.31.0.0/16
+    vpc_test.go:396: �[2mcustom_local_route�[0m=172.31.0.0/16
+�[2mvpc_igw�[0m=igw-79c2e643660b4b7e3
+    vpc_test.go:410: �[2mvpc_igw�[0m=igw-79c2e643660b4b7e3
+�[2m· create-route 0.0.0.0/0 -> igw-79c2e643660b4b7e3�[0m
+    vpc_test.go:415: �[2m· create-route 0.0.0.0/0 -> igw-79c2e643660b4b7e3�[0m
+�[2m· associate-route-table rtb-5e6c46936f3e44167 <- subnet-cd62e4c1c3cb50c63�[0m
+    vpc_test.go:434: �[2m· associate-route-table rtb-5e6c46936f3e44167 <- subnet-cd62e4c1c3cb50c63�[0m
+�[2mcustom_assoc�[0m=rtbassoc-da6377d37847d3d4a
+    vpc_test.go:450: �[2mcustom_assoc�[0m=rtbassoc-da6377d37847d3d4a
+�[2m· disassociate-route-table rtbassoc-da6377d37847d3d4a�[0m
+    vpc_test.go:452: �[2m· disassociate-route-table rtbassoc-da6377d37847d3d4a�[0m
+�[2m· delete-route 0.0.0.0/0�[0m
+    vpc_test.go:460: �[2m· delete-route 0.0.0.0/0�[0m
+�[2m· delete-route-table rtb-5e6c46936f3e44167�[0m
+    vpc_test.go:468: �[2m· delete-route-table rtb-5e6c46936f3e44167�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8c_RouteTableValidation/Step3_ErrorPaths" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· create-route-table (scratch for error paths)�[0m
+    vpc_test.go:481: �[2m· create-route-table (scratch for error paths)�[0m
+�[2m· delete-route local=172.31.0.0/16 (must fail)�[0m
+    vpc_test.go:517: �[2m· delete-route local=172.31.0.0/16 (must fail)�[0m
+�[2mdelete_local_err�[0m=InvalidParameterValue
+    vpc_test.go:529: �[2mdelete_local_err�[0m=InvalidParameterValue
+�[2m· delete-route-table rtb-e42012569f92f7184 (main, must fail)�[0m
+    vpc_test.go:533: �[2m· delete-route-table rtb-e42012569f92f7184 (main, must fail)�[0m
+�[2mdelete_main_err�[0m=DependencyViolation
+    vpc_test.go:540: �[2mdelete_main_err�[0m=DependencyViolation]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8d_NATGateway" classname="" time="106.010">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8d — NAT Gateway E2E ━━�[0m
+�[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2mpub_subnet�[0m=subnet-cd62e4c1c3cb50c63 �[2msg�[0m=sg-764a7fa2a39577204
+    natgw_test.go:54: �[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2mpub_subnet�[0m=subnet-cd62e4c1c3cb50c63 �[2msg�[0m=sg-764a7fa2a39577204
+�[2m· create-subnet 172.31.16.0/20 (private)�[0m
+    natgw_test.go:58: �[2m· create-subnet 172.31.16.0/20 (private)�[0m
+�[2mpriv_subnet�[0m=subnet-2360a40e28371e214
+    natgw_test.go:71: �[2mpriv_subnet�[0m=subnet-2360a40e28371e214
+�[2m· run-instances bastion (default subnet)�[0m
+    natgw_test.go:74: �[2m· run-instances bastion (default subnet)�[0m
+�[2m· run-instances private (private subnet)�[0m
+    natgw_test.go:98: �[2m· run-instances private (private subnet)�[0m
+    natgw_test.go:118: instance i-268402f0e28b5e477 reached state running
+    natgw_test.go:119: instance i-db7f3ed6e4cdaa5aa reached state running
+�[2mbastion�[0m=i-268402f0e28b5e477 �[2mbastion_ip�[0m=192.168.0.202 �[2mprivate�[0m=i-db7f3ed6e4cdaa5aa �[2mprivate_ip�[0m=172.31.16.4
+    natgw_test.go:128: �[2mbastion�[0m=i-268402f0e28b5e477 �[2mbastion_ip�[0m=192.168.0.202 �[2mprivate�[0m=i-db7f3ed6e4cdaa5aa �[2mprivate_ip�[0m=172.31.16.4
+�[2m· waiting for SSH handshake 192.168.0.202:22�[0m
+    natgw_test.go:133: �[2m· waiting for SSH handshake 192.168.0.202:22�[0m
+�[2m· scp keypair -> bastion:/tmp/key.pem�[0m
+    natgw_test.go:139: �[2m· scp keypair -> bastion:/tmp/key.pem�[0m
+�[2m· wait for private SSH via bastion�[0m
+    natgw_test.go:151: �[2m· wait for private SSH via bastion�[0m
+�[2m· baseline: private VM has no internet�[0m
+    natgw_test.go:172: �[2m· baseline: private VM has no internet�[0m
+�[2m· allocate-address (NAT EIP)�[0m
+    natgw_test.go:178: �[2m· allocate-address (NAT EIP)�[0m
+�[2meip�[0m=192.168.0.203 �[2malloc�[0m=eipalloc-2d81753157cd62a50
+    natgw_test.go:195: �[2meip�[0m=192.168.0.203 �[2malloc�[0m=eipalloc-2d81753157cd62a50
+�[2m· create-nat-gateway in subnet-cd62e4c1c3cb50c63�[0m
+    natgw_test.go:197: �[2m· create-nat-gateway in subnet-cd62e4c1c3cb50c63�[0m
+�[2mnat_gw�[0m=nat-812216dd5105180c5
+    natgw_test.go:216: �[2mnat_gw�[0m=nat-812216dd5105180c5
+    natgw_test.go:218: nat gateway nat-812216dd5105180c5 reached state available
+�[2m· create-route-table�[0m
+    natgw_test.go:223: �[2m· create-route-table�[0m
+�[2m· associate-route-table rtb-8d6a4abd5f666ec13 <- subnet-2360a40e28371e214�[0m
+    natgw_test.go:241: �[2m· associate-route-table rtb-8d6a4abd5f666ec13 <- subnet-2360a40e28371e214�[0m
+�[2m· create-route 0.0.0.0/0 -> nat-812216dd5105180c5�[0m
+    natgw_test.go:259: �[2m· create-route 0.0.0.0/0 -> nat-812216dd5105180c5�[0m
+�[2m· verify private VM reaches 8.8.8.8 via NAT GW�[0m
+    natgw_test.go:279: �[2m· verify private VM reaches 8.8.8.8 via NAT GW�[0m
+�[2m· delete-nat-gateway nat-812216dd5105180c5�[0m
+    natgw_test.go:293: �[2m· delete-nat-gateway nat-812216dd5105180c5�[0m
+    natgw_test.go:318: nat gateway nat-812216dd5105180c5 reached state deleted
+�[2m· verify private VM lost internet after NAT GW teardown�[0m
+    natgw_test.go:329: �[2m· verify private VM lost internet after NAT GW teardown�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8e_SGToSGDatapath" classname="" time="80.940">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8e — Security Group SG-to-SG Datapath (OVN) ━━�[0m
+�[2m· 8e-1 create sge-client + sge-target security groups�[0m
+    datapath_test.go:61: �[2m· 8e-1 create sge-client + sge-target security groups�[0m
+�[2mclient_sg�[0m=sg-56e73c764a649f7ff �[2mtarget_sg�[0m=sg-67ad5efaa8be83b79
+    datapath_test.go:83: �[2mclient_sg�[0m=sg-56e73c764a649f7ff �[2mtarget_sg�[0m=sg-67ad5efaa8be83b79
+�[2m· 8e-1 authorize target-sg ingress tcp/8080 from sg-56e73c764a649f7ff�[0m
+    datapath_test.go:91: �[2m· 8e-1 authorize target-sg ingress tcp/8080 from sg-56e73c764a649f7ff�[0m
+�[2m· 8e-2 launch client-vm + target-vm�[0m
+    datapath_test.go:109: �[2m· 8e-2 launch client-vm + target-vm�[0m
+    datapath_test.go:132: instance i-eae6263d9b9fda069 reached state running
+    datapath_test.go:133: instance i-9d9411a83549d34e3 reached state running
+�[2mclient_priv�[0m=172.31.0.5 �[2mclient_eni�[0m=eni-0f878da0143028915 �[2mtarget_priv�[0m=172.31.0.6 �[2mtarget_eni�[0m=eni-3bd355e19c7fdd45c
+    datapath_test.go:142: �[2mclient_priv�[0m=172.31.0.5 �[2mclient_eni�[0m=eni-0f878da0143028915 �[2mtarget_priv�[0m=172.31.0.6 �[2mtarget_eni�[0m=eni-3bd355e19c7fdd45c
+�[2m· 8e-3 ovn-nbctl port_group membership�[0m
+    datapath_test.go:151: �[2m· 8e-3 ovn-nbctl port_group membership�[0m
+�[2mclient_pg�[0m=sg_56e73c764a649f7ff �[2mtarget_pg�[0m=sg_67ad5efaa8be83b79
+    datapath_test.go:163: �[2mclient_pg�[0m=sg_56e73c764a649f7ff �[2mtarget_pg�[0m=sg_67ad5efaa8be83b79
+�[2m· 8e-3 ovn-sbctl address_set sg_56e73c764a649f7ff_ip4 contains 172.31.0.5�[0m
+    datapath_test.go:170: �[2m· 8e-3 ovn-sbctl address_set sg_56e73c764a649f7ff_ip4 contains 172.31.0.5�[0m
+�[2m· 8e-3 wait for client-vm SSH at 192.168.0.202:22�[0m
+    datapath_test.go:188: �[2m· 8e-3 wait for client-vm SSH at 192.168.0.202:22�[0m
+�[2m· 8e-4 allowed traffic client -> target:172.31.0.6:8080�[0m
+    datapath_test.go:199: �[2m· 8e-4 allowed traffic client -> target:172.31.0.6:8080�[0m
+�[2mstep4�[0m=allowed_traffic_ok
+    datapath_test.go:208: �[2mstep4�[0m=allowed_traffic_ok
+�[2m· 8e-5 denied traffic client -> target:22 (no SSH ingress)�[0m
+    datapath_test.go:213: �[2m· 8e-5 denied traffic client -> target:22 (no SSH ingress)�[0m
+�[2mstep5�[0m=denied_traffic_ok
+    datapath_test.go:217: �[2mstep5�[0m=denied_traffic_ok
+�[2m· 8e-6 revoke target-sg ingress, retest (sync RequestEvent contract)�[0m
+    datapath_test.go:222: �[2m· 8e-6 revoke target-sg ingress, retest (sync RequestEvent contract)�[0m
+�[2m· 8e-6 verify client -> target:8080 now blocked�[0m
+    datapath_test.go:240: �[2m· 8e-6 verify client -> target:8080 now blocked�[0m
+�[2mstep6�[0m=revoke_blocked_ok
+    datapath_test.go:244: �[2mstep6�[0m=revoke_blocked_ok
+�[2m· 8e-7 re-authorize target-sg ingress, verify traffic restored�[0m
+    datapath_test.go:250: �[2m· 8e-7 re-authorize target-sg ingress, verify traffic restored�[0m
+�[2mstep7�[0m=restore_ok
+    datapath_test.go:271: �[2mstep7�[0m=restore_ok
+    datapath_test.go:127: instance i-eae6263d9b9fda069 reached state terminated
+    datapath_test.go:127: instance i-9d9411a83549d34e3 reached state terminated]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase9_Teardown" classname="" time="8.300">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 9 — Teardown ━━�[0m
+�[2m· delete-snapshot snap-dde89b9ea2ef2e05a (custom AMI backing)�[0m
+    teardown_test.go:33: �[2m· delete-snapshot snap-dde89b9ea2ef2e05a (custom AMI backing)�[0m
+�[2mdeleted_snapshot�[0m=snap-dde89b9ea2ef2e05a
+    teardown_test.go:39: �[2mdeleted_snapshot�[0m=snap-dde89b9ea2ef2e05a
+�[2m· deregister-image ami-7dcf263a412dd14c6�[0m
+    teardown_test.go:51: �[2m· deregister-image ami-7dcf263a412dd14c6�[0m
+�[2mderegistered_ami�[0m=ami-7dcf263a412dd14c6
+    teardown_test.go:57: �[2mderegistered_ami�[0m=ami-7dcf263a412dd14c6
+�[2m· terminate-instances i-0c615e5dd74050f53�[0m
+    teardown_test.go:74: �[2m· terminate-instances i-0c615e5dd74050f53�[0m
+�[2mterminated_instance�[0m=i-0c615e5dd74050f53
+    teardown_test.go:81: �[2mterminated_instance�[0m=i-0c615e5dd74050f53
+�[2m· delete-key-pair test-key-1�[0m
+    teardown_test.go:94: �[2m· delete-key-pair test-key-1�[0m
+�[2mdeleted_keypair�[0m=test-key-1
+    teardown_test.go:100: �[2mdeleted_keypair�[0m=test-key-1]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase9a_VerifyTeardown" classname="" time="0.560">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 9a — Verify Teardown ━━�[0m
+�[2m· describe-instances i-0c615e5dd74050f53�[0m
+    teardown_test.go:123: �[2m· describe-instances i-0c615e5dd74050f53�[0m
+�[2minstance_state�[0m=terminated
+    teardown_test.go:135: �[2minstance_state�[0m=terminated
+�[2m· describe-images ami-7dcf263a412dd14c6�[0m
+    teardown_test.go:147: �[2m· describe-images ami-7dcf263a412dd14c6�[0m
+�[2mami_reaped�[0m=ami-7dcf263a412dd14c6
+    teardown_test.go:159: �[2mami_reaped�[0m=ami-7dcf263a412dd14c6
+�[2m· describe-key-pairs (must not contain test-key-1)�[0m
+    teardown_test.go:167: �[2m· describe-key-pairs (must not contain test-key-1)�[0m
+�[2mkeypairs_listed�[0m=1
+    teardown_test.go:176: �[2mkeypairs_listed�[0m=1]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase9b_FinalClusterStats" classname="" time="3.070">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 9b — Final Cluster Stats ━━�[0m
+�[2mspx_get_vms_bytes�[0m=118
+    teardown_test.go:208: �[2mspx_get_vms_bytes�[0m=118]]></system-out>
+		</testcase>
+	</testsuite>
+</testsuites>

.github/actions/junit-report/.gitignore

@@ -0,0 +1 @@
+__pycache__/

.github/actions/junit-report/README.md

@@ -0,0 +1,46 @@
+# junit-report
+
+Composite action that renders a glob of JUnit XML files into the GitHub
+Actions job summary and emits one `::error` annotation per failed test case.
+
+## Usage
+
+```yaml
+- name: Publish test summary
+  if: always()
+  uses: ./spinifex/.github/actions/junit-report   # or your own checkout path
+  with:
+    junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+    title: "Go E2E results"
+    # fail-on-empty: "true"   # opt-in: fail the step if the glob matches nothing
+```
+
+Outputs:
+
+- `failures` — total `<failure>` + `<error>` counts across all matched files.
+- `total`    — total `<testcase>` count across all matched files.
+
+The action expects standard JUnit XML (e.g. emitted by
+`gotestsum --junitfile`, `go-junit-report`, or pytest's `--junitxml`). Both
+single `<testsuite>` and wrapping `<testsuites>` roots are accepted.
+
+## Companion convention: `::group::` for noisy steps
+
+Long-running shell phases (tofu apply, package installs, tar-over-ssh) should
+wrap their output in `::group::name` / `::endgroup::` markers so the Actions
+UI collapses them by default. Pattern used in the e2e-go / e2e-ddil workflows:
+
+```bash
+run() {
+  local label="$1"; shift
+  echo "::group::${label}"
+  "$@" 2>&1 | tee -a "$TOFU_LOG"
+  local rc=${PIPESTATUS[0]}
+  echo "::endgroup::"
+  return "$rc"
+}
+run "tofu apply" tofu apply -var-file=... -auto-approve -input=false
+```
+
+Combined: groups make the live UI readable; this action makes the run header
+land on the failures without scrolling the log.

.github/actions/junit-report/action.yml

@@ -0,0 +1,41 @@
+name: JUnit Report
+description: >-
+  Render JUnit XML files into the GitHub Actions job summary and emit
+  inline ::error annotations for each failed test case. Designed to be
+  called once per job after the test step (with `if: always()`) so summary
+  + annotations appear on both passing and failing runs.
+
+inputs:
+  junit-glob:
+    description: Shell glob matching JUnit XML files to render.
+    required: true
+  title:
+    description: Heading rendered at the top of the job summary section.
+    required: false
+    default: "Test results"
+  fail-on-empty:
+    description: >-
+      When "true", the action fails if the glob matches no files. Useful
+      for catching silently-skipped test phases.
+    required: false
+    default: "false"
+
+outputs:
+  failures:
+    description: Total failed + errored test cases across all matched files.
+    value: ${{ steps.render.outputs.failures }}
+  total:
+    description: Total test cases across all matched files.
+    value: ${{ steps.render.outputs.total }}
+
+runs:
+  using: composite
+  steps:
+    - id: render
+      shell: bash
+      env:
+        JUNIT_GLOB: ${{ inputs.junit-glob }}
+        JUNIT_TITLE: ${{ inputs.title }}
+        JUNIT_FAIL_ON_EMPTY: ${{ inputs.fail-on-empty }}
+      run: |
+        python3 "${{ github.action_path }}/render.py"

.github/actions/junit-report/render.py

@@ -0,0 +1,281 @@
+#!/usr/bin/env python3
+"""Render JUnit XML files into GitHub Actions job summary + annotations.
+
+Inputs (env):
+    JUNIT_GLOB           Glob matching JUnit XML files.
+    JUNIT_TITLE          Heading text for the summary section.
+    JUNIT_FAIL_ON_EMPTY  "true" to exit non-zero when glob matches nothing.
+
+GitHub-Actions outputs (via $GITHUB_OUTPUT):
+    failures             Sum of <failures> + <errors> across all suites.
+    total                Sum of <tests> across all suites.
+
+Side effects:
+    Appends a markdown section to $GITHUB_STEP_SUMMARY with a per-file
+    totals table and a collapsible details block per failed test.
+    Emits one `::error` workflow command per failed/errored test case so
+    they surface as annotations in the run UI.
+"""
+
+from __future__ import annotations
+
+import glob
+import os
+import re
+import sys
+import xml.etree.ElementTree as ET
+from dataclasses import dataclass
+
+
+# Strip ANSI SGR escapes from skipped/failure CDATA so the rendered reason is
+# plain text (CDATA preserves the colour codes Go emits via harness.Phase).
+_ANSI_RE = re.compile(r"\x1b\[[0-9;]*m")
+
+
+def _strip_ansi(s: str) -> str:
+    return _ANSI_RE.sub("", s)
+
+
+# Last `<file>.go:<line>: <reason>` line in a CDATA block is the t.Skipf /
+# t.Fatal call site — that's the actionable signal. Phase banners and probe
+# logs above it are noise for the summary view.
+_SKIP_REASON_RE = re.compile(r"([A-Za-z0-9_.\-/]+\.go:\d+:\s*.+)$")
+
+
+def _extract_skip_reason(cdata: str) -> str:
+    cdata = _strip_ansi(cdata).strip()
+    if not cdata:
+        return ""
+    for line in reversed(cdata.splitlines()):
+        line = line.strip()
+        m = _SKIP_REASON_RE.search(line)
+        if m:
+            return m.group(1).strip()
+    return cdata.splitlines()[-1].strip()
+
+
+# Workflow commands carry their payload on one line; literal newlines must
+# be percent-escaped per GitHub's workflow-command spec.
+def _esc(s: str) -> str:
+    return (
+        s.replace("%", "%25")
+        .replace("\r", "%0D")
+        .replace("\n", "%0A")
+    )
+
+
+@dataclass
+class Case:
+    suite: str
+    classname: str
+    name: str
+    time: float
+    status: str  # "pass" | "fail" | "error" | "skip"
+    message: str
+    detail: str
+
+
+@dataclass
+class FileTotals:
+    path: str
+    tests: int
+    failures: int
+    errors: int
+    skipped: int
+    time: float
+    cases: list[Case]
+
+
+def _parse_file(path: str) -> FileTotals:
+    root = ET.parse(path).getroot()
+    # JUnit XML has two shapes: <testsuite> as root, or <testsuites> wrapping
+    # multiple <testsuite> children. Iterating with .iter() handles both.
+    suites = list(root.iter("testsuite"))
+    tests = sum(int(s.get("tests", 0) or 0) for s in suites)
+    failures = sum(int(s.get("failures", 0) or 0) for s in suites)
+    errors = sum(int(s.get("errors", 0) or 0) for s in suites)
+    skipped = sum(int(s.get("skipped", 0) or 0) for s in suites)
+    time = sum(float(s.get("time", 0) or 0) for s in suites)
+
+    cases: list[Case] = []
+    for s in suites:
+        suite_name = s.get("name", "")
+        for tc in s.findall("testcase"):
+            status = "pass"
+            message = ""
+            detail = ""
+            for tag in ("failure", "error"):
+                node = tc.find(tag)
+                if node is not None:
+                    status = "fail" if tag == "failure" else "error"
+                    message = (node.get("message") or "").strip()
+                    detail = (node.text or "").strip()
+                    break
+            skip_node = tc.find("skipped")
+            if skip_node is not None and status == "pass":
+                status = "skip"
+                # Skipped reason: prefer the CDATA call-site line, fall back
+                # to the @message attribute if CDATA is empty.
+                cdata = (skip_node.text or "").strip()
+                message = _extract_skip_reason(cdata) or (
+                    skip_node.get("message") or ""
+                ).strip()
+            cases.append(
+                Case(
+                    suite=suite_name,
+                    classname=tc.get("classname", ""),
+                    name=tc.get("name", ""),
+                    time=float(tc.get("time", 0) or 0),
+                    status=status,
+                    message=message,
+                    detail=detail,
+                )
+            )
+
+    # Drop container test cases: any case whose name is a strict prefix of
+    # another case's name (with "/" separator) is a parent t.Run wrapper.
+    # Go reports the parent's status as a rollup of its children, so counting
+    # both double-tallies the same outcome (e.g. TestSingleNode failing
+    # because Phase8b_VPCSubnetE2E failed would show as 2 failures, not 1).
+    names = {c.name for c in cases}
+    prefixes = {n.rsplit("/", 1)[0] for n in names if "/" in n}
+    # A name is a container if some other name starts with "<name>/".
+    # Use the prefixes set rather than O(n^2) scanning.
+    def _is_container(name: str) -> bool:
+        return name in prefixes
+
+    leaf_cases = [c for c in cases if not _is_container(c.name)]
+
+    # Recompute totals from leaf cases so the per-file counts in the summary
+    # table match the de-duplicated failure list below it.
+    leaf_tests = len(leaf_cases)
+    leaf_failures = sum(1 for c in leaf_cases if c.status == "fail")
+    leaf_errors = sum(1 for c in leaf_cases if c.status == "error")
+    leaf_skipped = sum(1 for c in leaf_cases if c.status == "skip")
+
+    return FileTotals(
+        path=path,
+        tests=leaf_tests,
+        failures=leaf_failures,
+        errors=leaf_errors,
+        skipped=leaf_skipped,
+        time=time,
+        cases=leaf_cases,
+    )
+
+
+def _file_label(path: str) -> str:
+    base = os.path.basename(path)
+    if base.endswith(".xml"):
+        base = base[:-4]
+    if base.startswith("junit-"):
+        base = base[len("junit-"):]
+    return base or path
+
+
+def main() -> int:
+    pattern = os.environ.get("JUNIT_GLOB", "").strip()
+    title = os.environ.get("JUNIT_TITLE", "Test results").strip() or "Test results"
+    fail_on_empty = os.environ.get("JUNIT_FAIL_ON_EMPTY", "").strip().lower() == "true"
+
+    if not pattern:
+        print("::error::junit-report: junit-glob input is empty", file=sys.stderr)
+        return 1
+
+    paths = sorted(glob.glob(pattern))
+    summary_path = os.environ.get("GITHUB_STEP_SUMMARY", "/dev/null")
+    output_path = os.environ.get("GITHUB_OUTPUT", "/dev/null")
+
+    with open(summary_path, "a", encoding="utf-8") as summary, \
+            open(output_path, "a", encoding="utf-8") as outputs:
+
+        summary.write(f"## {title}\n\n")
+
+        if not paths:
+            summary.write(f"_No JUnit files matched `{pattern}`._\n\n")
+            outputs.write("failures=0\n")
+            outputs.write("total=0\n")
+            if fail_on_empty:
+                print(f"::error::junit-report: no files matched {pattern}", file=sys.stderr)
+                return 1
+            return 0
+
+        totals_by_file = [_parse_file(p) for p in paths]
+        grand_total = sum(f.tests for f in totals_by_file)
+        grand_fail = sum(f.failures + f.errors for f in totals_by_file)
+        grand_skip = sum(f.skipped for f in totals_by_file)
+        grand_pass = grand_total - grand_fail - grand_skip
+        grand_time = sum(f.time for f in totals_by_file)
+
+        if grand_fail == 0:
+            verdict = f"✅ **pass** — {grand_pass} / {grand_total}"
+        else:
+            verdict = f"❌ **{grand_fail} failed** — {grand_pass} / {grand_total} pass"
+        skip_part = f", {grand_skip} skipped" if grand_skip else ""
+        summary.write(f"{verdict}{skip_part} &nbsp; · &nbsp; {grand_time:.1f}s\n\n")
+
+        # Compact table — single status column, shorter headers. Drops
+        # separate err column (errors fold into fail count) so narrow
+        # screens don't wrap.
+        summary.write("| | Suite | Pass | Fail | Skip | Time |\n")
+        summary.write("|:-:|---|---:|---:|---:|---:|\n")
+        for f in totals_by_file:
+            fails = f.failures + f.errors
+            passed = f.tests - fails - f.skipped
+            icon = "✅" if fails == 0 else "❌"
+            summary.write(
+                f"| {icon} | `{_file_label(f.path)}` | {passed} | {fails} | "
+                f"{f.skipped} | {f.time:.1f}s |\n"
+            )
+        summary.write("\n")
+
+        failed_cases = [
+            c for f in totals_by_file for c in f.cases if c.status in ("fail", "error")
+        ]
+        if failed_cases:
+            summary.write("---\n\n**Failures**\n\n")
+            for c in failed_cases:
+                header = f"{c.classname}.{c.name}" if c.classname else c.name
+                # One line per failure: name + truncated message. Full
+                # detail lives in the test artifact log; deep nesting and
+                # long messages wrap badly on narrow screens.
+                msg_short = (c.message or "").splitlines()[0] if c.message else ""
+                if len(msg_short) > 120:
+                    msg_short = msg_short[:117] + "…"
+                line = f"- ❌ `{header}`"
+                if msg_short:
+                    line += f" — {msg_short}"
+                summary.write(line + "\n")
+
+                # Workflow annotation — surfaces in run header + Files-changed view.
+                # classname for Go tests is the package path; map to a file when we can.
+                file_hint = c.classname.replace(".", "/")
+                msg = c.message or f"{c.status}: {header}"
+                print(f"::error file={file_hint},title={header}::{_esc(msg)}")
+            summary.write("\n")
+
+        skipped_cases = [
+            c for f in totals_by_file for c in f.cases if c.status == "skip"
+        ]
+        if skipped_cases:
+            summary.write("---\n\n<details><summary><b>Skipped</b> ")
+            summary.write(f"({len(skipped_cases)})</summary>\n\n")
+            for c in skipped_cases:
+                header = f"{c.classname}.{c.name}" if c.classname else c.name
+                reason = (c.message or "").splitlines()[0] if c.message else ""
+                if len(reason) > 160:
+                    reason = reason[:157] + "…"
+                line = f"- ⏭️ `{header}`"
+                if reason:
+                    line += f" — {reason}"
+                summary.write(line + "\n")
+            summary.write("\n</details>\n\n")
+
+        outputs.write(f"failures={grand_fail}\n")
+        outputs.write(f"total={grand_total}\n")
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

.github/scripts/capture-net-diag.sh

@@ -0,0 +1,88 @@
+#!/usr/bin/env bash
+# capture-net-diag.sh — write a snapshot of network / OVN / OVS / IPsec
+# state into $1 (a directory the caller already created on the local host).
+# Designed to run on a spinifex node after an E2E suite, before tofu destroy
+# pulls the rug out. Every command is independent — `|| true` so a missing
+# binary on one path doesn't abort the rest of the capture.
+#
+# Output layout (under $1/net-diag/):
+#   openvswitch-ipsec.status         systemctl status (output-only)
+#   openvswitch-ipsec.journal        last 200 journal lines
+#   ipsec.openvswitch_conf           ovs-vsctl get Open_vSwitch . other_config
+#   ipsec.xfrm_state                 ip xfrm state list
+#   ipsec.xfrm_policy                ip xfrm policy list
+#   ovn.sb_show                      ovn-sbctl show
+#   ovn.nb_show                      ovn-nbctl show
+#   ovs.show                         ovs-vsctl show
+#   ovs.flows_br_int                 ovs-ofctl dump-flows br-int
+#   ovs.tunnels                      ovs-appctl ofproto/list-tunnels br-int
+#   net.links                        ip -s link show
+#   net.routes                       ip route show table all
+#   net.addrs                        ip -br addr show
+#   systemd.failed_units             systemctl list-units --state=failed
+set +e
+OUT="${1:?usage: capture-net-diag.sh <output-dir>}"
+mkdir -p "$OUT"
+
+run() {
+    local label="$1"; shift
+    {
+        echo "# $*"
+        "$@" 2>&1
+    } > "$OUT/$label" || true
+}
+
+run openvswitch-ipsec.status            sudo systemctl status openvswitch-ipsec.service --no-pager --full --lines=50
+run openvswitch-ipsec.journal           sudo journalctl -u openvswitch-ipsec.service --no-pager --output=short-iso -n 200
+run ipsec.openvswitch_conf              sudo ovs-vsctl get Open_vSwitch . other_config
+run ipsec.xfrm_state                    sudo ip xfrm state list
+run ipsec.xfrm_policy                   sudo ip xfrm policy list
+
+run ovn.sb_show                         sudo ovn-sbctl show
+run ovn.nb_show                         sudo ovn-nbctl show
+run ovn.sb_chassis                      sudo ovn-sbctl list chassis
+
+run_per_lr() {
+    local label="$1"; shift
+    local subcmd="$1"; shift
+    {
+        echo "# ovn-nbctl $subcmd <each-logical-router>"
+        local lrs
+        lrs="$(sudo ovn-nbctl --bare --columns=name list Logical_Router 2>/dev/null)"
+        for lr in $lrs; do
+            echo
+            echo "=== $lr ==="
+            sudo ovn-nbctl "$subcmd" "$lr" 2>&1
+        done
+    } > "$OUT/$label" || true
+}
+
+run_per_lr ovn.lr_routes                lr-route-list
+run_per_lr ovn.lr_policies              lr-policy-list
+run_per_lr ovn.lr_nat                   lr-nat-list
+
+run ovs.show                            sudo ovs-vsctl show
+run ovs.flows_br_int                    sudo ovs-ofctl dump-flows br-int
+run ovs.tunnels                         sudo ovs-appctl ofproto/list-tunnels br-int
+
+run net.links                           ip -s link show
+run net.routes                          ip route show table all
+run net.addrs                           ip -br addr show
+run net.neigh                           ip neigh show
+run net.bridge_fdb                      bridge fdb show
+run net.conntrack                       sudo conntrack -L
+
+run systemd.failed_units                sudo systemctl list-units --state=failed --no-pager
+
+# Background pcap (started by start-net-capture.sh, if present).
+if [ -f /tmp/spinifex-e2e-tcpdump.pid ]; then
+    PID="$(cat /tmp/spinifex-e2e-tcpdump.pid 2>/dev/null)"
+    sudo kill -INT "$PID" 2>/dev/null || true
+    sleep 1
+    if [ -f /tmp/spinifex-e2e-wan.pcap ]; then
+        sudo cp /tmp/spinifex-e2e-wan.pcap "$OUT/wan.pcap" 2>/dev/null || true
+        sudo chmod 0644 "$OUT/wan.pcap" 2>/dev/null || true
+    fi
+fi
+
+exit 0

.github/scripts/ci-fmt.sh

@@ -0,0 +1,63 @@
+# shellcheck shell=bash
+# Source from a workflow `run:` block:
+#   source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+#
+# Provides colored banners + ::group::/::endgroup:: wrappers. ANSI escapes
+# render in the GitHub Actions live log; the same lines also land in any
+# artifact log (tee'd output) without breaking parsers because they only
+# affect terminal display.
+#
+# Functions kept short and unindented on purpose — wide nested output wraps
+# badly on narrow terminals.
+
+if [[ -n "${GITHUB_ACTIONS:-}" ]] || [[ -t 1 ]]; then
+  _R=$'\033[0m'
+  _B=$'\033[1m'    # bold
+  _D=$'\033[2m'    # dim
+  _RED=$'\033[31m'
+  _GRN=$'\033[32m'
+  _YEL=$'\033[33m'
+  _BLU=$'\033[34m'
+  _CYA=$'\033[36m'
+else
+  _R="" _B="" _D="" _RED="" _GRN="" _YEL="" _BLU="" _CYA=""
+fi
+
+# Single-line rule using box-drawing chars — narrow enough that GitHub's
+# timestamp prefix + this rule still fit on an 80-col terminal.
+# banner LABEL — bold cyan title with surrounding rule. Used at the start
+# of an inherently multi-line phase (test run, artifact pull) so the live
+# log has an obvious visual marker.
+banner() {
+  printf '\n%s%s━━ %s ━━%s\n' "$_B" "$_CYA" "$*" "$_R"
+}
+
+# Status indicators. Single char prefix only — keeps width on narrow logs.
+ok()   { printf '%s✓%s %s\n' "$_GRN" "$_R" "$*"; }
+bad()  { printf '%s✗%s %s\n' "$_RED" "$_R" "$*"; }
+warn() { printf '%s!%s %s\n' "$_YEL" "$_R" "$*"; }
+note() { printf '%s·%s %s\n' "$_D"   "$_R" "$*"; }
+
+# group_open LABEL / group_close — thin wrapper around GitHub's collapse
+# markers. No extra body printed; banner() is the visible inside-group
+# heading when callers want one.
+group_open()  { echo "::group::$*"; }
+group_close() { echo "::endgroup::"; }
+
+# quiet_run LABEL LOG_FILE -- CMD ARGS …
+# Run CMD redirected to LOG_FILE (append). Print one colored ok/✗ line.
+# On failure, dump the last 80 lines of LOG_FILE inside a collapsed group
+# so the failing tail lands in the UI without surfacing the whole apply.
+quiet_run() {
+  local label="$1" log="$2"; shift 2
+  if "$@" >> "$log" 2>&1; then
+    ok "$label"
+  else
+    local rc=$?
+    bad "$label (rc=$rc) — last 80 lines of $(basename "$log"):"
+    group_open "tail $(basename "$log")"
+    tail -n 80 "$log"
+    group_close
+    return "$rc"
+  fi
+}

.github/scripts/start-net-capture.sh

@@ -0,0 +1,47 @@
+#!/usr/bin/env bash
+# start-net-capture.sh — kick off background tcpdump on br-wan.
+# Pair: capture-net-diag.sh stops it and copies the pcap into net-diag/.
+#
+# Filter: anything on br-wan involving the EIP pool (192.168.0.0/23) plus
+# ARP for the same range. Single ring buffer (-W 1 -G 0) → grows until stop;
+# capped at 64 MiB via -C 64. Snaplen 200 bytes (headers + a bit) to keep
+# size sane while preserving L2/L3/L4 + first payload bytes.
+set +e
+
+PCAP=/tmp/spinifex-e2e-wan.pcap
+PIDFILE=/tmp/spinifex-e2e-tcpdump.pid
+
+if [ -f "$PIDFILE" ] && kill -0 "$(cat "$PIDFILE" 2>/dev/null)" 2>/dev/null; then
+    echo "[start-net-capture] tcpdump already running pid=$(cat "$PIDFILE")"
+    exit 0
+fi
+
+sudo rm -f "$PCAP" "$PIDFILE"
+
+if ! command -v tcpdump >/dev/null 2>&1; then
+    echo "[start-net-capture] tcpdump not installed — skipping" >&2
+    exit 0
+fi
+if ! ip link show br-wan >/dev/null 2>&1; then
+    echo "[start-net-capture] br-wan absent — skipping" >&2
+    exit 0
+fi
+
+sudo nohup tcpdump -i br-wan -s 200 -C 64 -W 1 -U \
+    -w "$PCAP" \
+    '(net 192.168.0.0/23) or (arp and (arp[24:4]&0xfffffe00=0xc0a80000))' \
+    >/tmp/spinifex-e2e-tcpdump.log 2>&1 &
+PID=$!
+echo "$PID" | sudo tee "$PIDFILE" >/dev/null
+disown
+
+sleep 1
+if ! kill -0 "$PID" 2>/dev/null; then
+    echo "[start-net-capture] tcpdump died — see /tmp/spinifex-e2e-tcpdump.log" >&2
+    cat /tmp/spinifex-e2e-tcpdump.log >&2 || true
+    sudo rm -f "$PIDFILE"
+    exit 1
+fi
+
+echo "[start-net-capture] tcpdump pid=$PID pcap=$PCAP"
+exit 0

.github/workflows/e2e-baremetal.yml

@@ -81,33 +81,23 @@ jobs:
 
       - name: Cache VM images
         run: |
-          UBUNTU_IMG="${BM_ISO_CACHE}/ubuntu-24.04-cloudimg-amd64.img"
+          UBUNTU_IMG="${BM_ISO_CACHE}/ubuntu-26.04-cloudimg-amd64.img"
           if [ ! -f "$UBUNTU_IMG" ]; then
-            echo "Downloading Ubuntu 24.04 cloud image..."
+            echo "Downloading Ubuntu 26.04 cloud image..."
             sudo curl -fsSL -o "$UBUNTU_IMG" \
-              "https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img"
+              "https://cloud-images.ubuntu.com/resolute/current/resolute-server-cloudimg-amd64.img"
             echo "  Downloaded $(du -h "$UBUNTU_IMG" | cut -f1)"
           else
-            echo "Ubuntu 24.04 cloud image cached ($(du -h "$UBUNTU_IMG" | cut -f1))"
+            echo "Ubuntu 26.04 cloud image cached ($(du -h "$UBUNTU_IMG" | cut -f1))"
           fi
 
-          LB_IMG="${BM_ISO_CACHE}/lb-alpine-3.21.6-x86_64.raw"
-          if [ ! -f "$LB_IMG" ]; then
-            echo "Downloading lb-alpine image..."
-            sudo curl -fsSL -o "$LB_IMG" \
-              "https://iso.mulgadc.com/system-ami/lb-alpine-3.21.6-x86_64.raw"
-            echo "  Downloaded $(du -h "$LB_IMG" | cut -f1)"
-          else
-            echo "lb-alpine image cached ($(du -h "$LB_IMG" | cut -f1))"
-          fi
 
       - name: Run Bare-Metal E2E
         env:
           BM_CI_PASSWORD: ${{ secrets.BM_CI_PASSWORD }}
           BM_SSH_KEY: ${{ runner.temp }}/bm-ci
           BM_EPDU_ENV: ${{ runner.temp }}/epdu.env
-          BM_UBUNTU_IMAGE: ${{ env.BM_ISO_CACHE }}/ubuntu-24.04-cloudimg-amd64.img
-          BM_LB_ALPINE_IMAGE: ${{ env.BM_ISO_CACHE }}/lb-alpine-3.21.6-x86_64.raw
+          BM_UBUNTU_IMAGE: ${{ env.BM_ISO_CACHE }}/ubuntu-26.04-cloudimg-amd64.img
           SPX_EXTERNAL_POOL: ${{ env.SPINIFEX_CI_POOL }}
         run: |
           sudo -E mulga/scripts/baremetal/run-baremetal-e2e.sh \

.github/workflows/e2e-ddil.yml

@@ -87,12 +87,16 @@ jobs:
       - name: Provision VMs
         working-directory: mulga/scripts/tofu-cluster
         run: |
+          set -euo pipefail
+          # shellcheck source=/dev/null
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           export TF_VAR_git_branch="${GIT_BRANCH}"
-          tofu init -input=false >> "$TOFU_LOG" 2>&1
-          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
-          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
-          tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
+          quiet_run "tofu workspace select"    "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
+          quiet_run "scrub-env (pre-clean)"    "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
+          quiet_run "tofu apply"               "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          ok "VMs provisioned"
 
       - name: Bootstrap via Binary Install
         working-directory: mulga/scripts/tofu-cluster
@@ -147,8 +151,13 @@ jobs:
 
           mkdir -p "${RUNNER_TEMP}/ddil-results"
           cd "${GITHUB_WORKSPACE}/spinifex"
-          # gotestsum: human-readable PASS/FAIL/SKIP lines in Actions stdout
-          # while still emitting the machine-readable jsonfile artifact.
+          # gotestsum: human-readable PASS/FAIL/SKIP lines in Actions stdout,
+          # plus machine-readable jsonfile + JUnit XML artifacts. The JUnit
+          # file feeds the junit-report composite action below so failures
+          # become summary table rows + workflow annotations.
+          # shellcheck source=/dev/null
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          banner "DDIL scenarios"
           DDIL_NODES="${NODE_IPS}" \
           DDIL_SSH_USER="${SSH_USER}" \
           DDIL_SSH_KEY="${SSH_KEY}" \
@@ -158,8 +167,19 @@ jobs:
           AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" \
           AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" \
             gotestsum --jsonfile "${RUNNER_TEMP}/ddil-results/results.json" \
+                      --junitfile "${RUNNER_TEMP}/ddil-results/junit-ddil.xml" \
                       --format testname \
                       -- -tags=e2e -timeout 45m ./tests/e2e/ddil/scenarios/...
+          rc=$?
+          if [[ $rc -eq 0 ]]; then ok "all scenarios passed"; else bad "scenarios failed (rc=$rc)"; fi
+          exit $rc
+
+      - name: Publish test summary
+        if: always()
+        uses: ./spinifex/.github/actions/junit-report
+        with:
+          junit-glob: ${{ runner.temp }}/ddil-results/junit-*.xml
+          title: "DDIL scenario results"
 
       - name: Collect Cluster Logs
         if: failure()
@@ -177,13 +197,15 @@ jobs:
           LOG_DIR="${RUNNER_TEMP}/ddil-results/cluster-logs"
           mkdir -p "${LOG_DIR}"
 
+          # shellcheck source=/dev/null
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           mapfile -t NODE_IPS < <(echo "$INVENTORY" | jq -r '.nodes[].wan_ip')
           for ip in "${NODE_IPS[@]}"; do
-            echo "=== collecting ${ip} ==="
             SSH="ssh -i ${SSH_KEY} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${SSH_USER}@${ip}"
             ${SSH} 'sudo journalctl -u "spinifex-*" --no-pager -n 5000' > "${LOG_DIR}/${ip}-spinifex.log" 2>&1
             ${SSH} 'ps auxw | grep -E "qemu-system|spinifex" | grep -v grep' > "${LOG_DIR}/${ip}-procs.txt" 2>&1
             ${SSH} 'ls -la /var/lib/spinifex 2>/dev/null; ls -la /var/log/spinifex 2>/dev/null' > "${LOG_DIR}/${ip}-state.txt" 2>&1
+            ok "$ip: $(wc -l < "${LOG_DIR}/${ip}-spinifex.log") journal lines, $(wc -l < "${LOG_DIR}/${ip}-procs.txt") procs"
           done
 
       - name: Upload DDIL Results
@@ -198,6 +220,10 @@ jobs:
         if: always()
         working-directory: mulga/scripts/tofu-cluster
         run: |
-          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
-          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+          set -euo pipefail
+          # shellcheck source=/dev/null
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy"          "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
+          ok "teardown complete"

.github/workflows/e2e.yml

@@ -77,6 +77,37 @@ jobs:
             "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
             --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
 
+      - name: Start Packet Capture (siv-167)
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} 'bash -s' <<'REMOTE'
+          set -x
+          sudo mkdir -p /tmp/spinifex-diag
+          sudo chmod 0777 /tmp/spinifex-diag
+          sudo rm -f /tmp/spinifex-diag/br-wan.pcap /tmp/spinifex-diag/tcpdump.pid /tmp/spinifex-diag/tcpdump.log
+          sudo bash -c '
+            nohup tcpdump -i br-wan -nn -s 0 -U \
+              -w /tmp/spinifex-diag/br-wan.pcap \
+              "(tcp port 22 or tcp port 80 or tcp port 443 or tcp port 9999) or arp" \
+              </dev/null >/tmp/spinifex-diag/tcpdump.log 2>&1 &
+            echo $! > /tmp/spinifex-diag/tcpdump.pid
+          '
+          sleep 2
+          PID=$(cat /tmp/spinifex-diag/tcpdump.pid 2>/dev/null || echo "")
+          if [ -n "$PID" ] && sudo kill -0 "$PID" 2>/dev/null; then
+            echo "tcpdump running PID=$PID"
+            ls -la /tmp/spinifex-diag/
+          else
+            echo "tcpdump failed to start"
+            cat /tmp/spinifex-diag/tcpdump.log 2>&1 || true
+            exit 1
+          fi
+          REMOTE
+
       - name: Run E2E Tests
         working-directory: mulga/scripts/tofu-cluster
         run: |
@@ -87,6 +118,73 @@ jobs:
           ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
             "cd ~/e2e-tests && ./run-e2e.sh && ./run-cert-e2e.sh && ENDPOINT=https://${WAN_IP}:9999 ./run-lb-e2e.sh"
 
+      - name: Collect Diagnostics
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          DIAG_DIR="${RUNNER_TEMP}/diag-single"
+          mkdir -p "$DIAG_DIR"
+          ssh -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i "$SSH_KEY" tf-user@${WAN_IP} 'bash -s' > "$DIAG_DIR/collect.log" 2>&1 <<'REMOTE' || true
+          set -x
+          DIAG=/tmp/spinifex-diag
+          sudo mkdir -p "$DIAG" && cd "$DIAG"
+          if [ -f tcpdump.pid ]; then
+            PID=$(cat tcpdump.pid)
+            sudo kill -INT "$PID" 2>/dev/null || true
+            for _ in 1 2 3 4 5; do sudo kill -0 "$PID" 2>/dev/null || break; sleep 1; done
+            sudo kill -KILL "$PID" 2>/dev/null || true
+          fi
+          sudo journalctl -u 'spinifex-*' --no-pager > journal-spinifex-full.log 2>&1 || true
+          sudo journalctl -u 'spinifex-*' --no-pager -n 10000 > journal-spinifex.log 2>&1 || true
+          sudo journalctl -u 'ovn-*' --no-pager -n 5000 > journal-ovn.log 2>&1 || true
+          sudo journalctl -u 'openvswitch-*' --no-pager -n 2000 > journal-ovs.log 2>&1 || true
+          sudo ovs-vsctl show > ovs-vsctl-show.txt 2>&1 || true
+          sudo ovs-ofctl dump-flows br-int > br-int-flows.txt 2>&1 || true
+          sudo ovs-ofctl dump-flows br-ex > br-ex-flows.txt 2>&1 || true
+          sudo ovs-appctl dpctl/dump-flows > dpctl-flows.txt 2>&1 || true
+          sudo ovs-appctl dpctl/dump-conntrack -m > dpctl-conntrack.txt 2>&1 || true
+          sudo ovn-nbctl show > ovn-nbctl-show.txt 2>&1 || true
+          sudo ovn-sbctl show > ovn-sbctl-show.txt 2>&1 || true
+          sudo ovn-sbctl list Port_Binding > sbdb-port-binding.txt 2>&1 || true
+          sudo ovn-sbctl list Chassis > sbdb-chassis.txt 2>&1 || true
+          sudo ovn-nbctl list NAT > nbdb-nat.txt 2>&1 || true
+          sudo ovn-nbctl list Logical_Router > nbdb-lr.txt 2>&1 || true
+          sudo ovn-nbctl list Logical_Router_Policy > nbdb-lr-policy.txt 2>&1 || true
+          sudo ovn-nbctl list Logical_Router_Static_Route > nbdb-lr-static-route.txt 2>&1 || true
+          sudo ovn-appctl -t ovn-controller ct-zone-list > ovn-ctzones.txt 2>&1 || true
+          sudo ip -d link > ip-link.txt 2>&1 || true
+          sudo ip addr > ip-addr.txt 2>&1 || true
+          sudo ip route > ip-route.txt 2>&1 || true
+          sudo ip neigh > ip-neigh.txt 2>&1 || true
+          sudo bridge fdb show > bridge-fdb.txt 2>&1 || true
+          sudo bridge link > bridge-link.txt 2>&1 || true
+          for f in /sys/class/net/br-wan/brif/*/hairpin_mode /sys/class/net/br-wan/brif/*/learning /sys/class/net/br-wan/brif/*/flood; do
+            [ -e "$f" ] && echo "$f $(cat $f)"
+          done > hairpin-modes.txt 2>&1 || true
+          sudo iptables-save > iptables.txt 2>&1 || true
+          sudo nft list ruleset > nft.txt 2>&1 || true
+          sudo dmesg --ctime > dmesg.txt 2>&1 || true
+          sudo virsh list --all > virsh-list.txt 2>&1 || true
+          ls -la /var/lib/spinifex /var/log/spinifex 2>&1 > spinifex-dirs.txt || true
+          sudo chmod -R a+r /tmp/spinifex-diag
+          tar czf /tmp/spinifex-diag.tar.gz -C /tmp spinifex-diag
+          REMOTE
+          scp -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i "$SSH_KEY" \
+            tf-user@${WAN_IP}:/tmp/spinifex-diag.tar.gz "$DIAG_DIR/spinifex-diag.tar.gz" || true
+
+      - name: Upload Diagnostics
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: e2e-diag-single-${{ github.run_id }}
+          path: ${{ runner.temp }}/diag-single/
+          if-no-files-found: warn
+          retention-days: 14
+
       - name: Teardown
         if: always()
         working-directory: mulga/scripts/tofu-cluster
@@ -151,6 +249,66 @@ jobs:
           ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
             "cd ~/Development/mulga/spinifex && SPX_EXTERNAL_POOL=\"${SPINIFEX_CI_POOL}\" ./tests/e2e/run-pseudo-multinode-e2e.sh"
 
+      - name: Collect Diagnostics
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          DIAG_DIR="${RUNNER_TEMP}/diag-pseudo"
+          mkdir -p "$DIAG_DIR"
+          ssh -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i "$SSH_KEY" tf-user@${WAN_IP} 'bash -s' > "$DIAG_DIR/collect.log" 2>&1 <<'REMOTE' || true
+          set -x
+          DIAG=/tmp/spinifex-diag
+          sudo rm -rf "$DIAG" && mkdir -p "$DIAG" && cd "$DIAG"
+          sudo journalctl -u 'spinifex-*' --no-pager > journal-spinifex-full.log 2>&1 || true
+          sudo journalctl -u 'spinifex-*' --no-pager -n 10000 > journal-spinifex.log 2>&1 || true
+          sudo journalctl -u 'ovn-*' --no-pager -n 5000 > journal-ovn.log 2>&1 || true
+          sudo journalctl -u 'openvswitch-*' --no-pager -n 2000 > journal-ovs.log 2>&1 || true
+          sudo ovs-vsctl show > ovs-vsctl-show.txt 2>&1 || true
+          sudo ovs-ofctl dump-flows br-int > br-int-flows.txt 2>&1 || true
+          sudo ovs-ofctl dump-flows br-ex > br-ex-flows.txt 2>&1 || true
+          sudo ovs-appctl dpctl/dump-flows > dpctl-flows.txt 2>&1 || true
+          sudo ovs-appctl dpctl/dump-conntrack -m > dpctl-conntrack.txt 2>&1 || true
+          sudo ovn-nbctl show > ovn-nbctl-show.txt 2>&1 || true
+          sudo ovn-sbctl show > ovn-sbctl-show.txt 2>&1 || true
+          sudo ovn-sbctl list Port_Binding > sbdb-port-binding.txt 2>&1 || true
+          sudo ovn-sbctl list Chassis > sbdb-chassis.txt 2>&1 || true
+          sudo ovn-nbctl list NAT > nbdb-nat.txt 2>&1 || true
+          sudo ovn-nbctl list Logical_Router > nbdb-lr.txt 2>&1 || true
+          sudo ovn-nbctl list Logical_Router_Policy > nbdb-lr-policy.txt 2>&1 || true
+          sudo ovn-nbctl list Logical_Router_Static_Route > nbdb-lr-static-route.txt 2>&1 || true
+          sudo ovn-appctl -t ovn-controller ct-zone-list > ovn-ctzones.txt 2>&1 || true
+          sudo ip -d link > ip-link.txt 2>&1 || true
+          sudo ip addr > ip-addr.txt 2>&1 || true
+          sudo ip route > ip-route.txt 2>&1 || true
+          sudo ip neigh > ip-neigh.txt 2>&1 || true
+          sudo bridge fdb show > bridge-fdb.txt 2>&1 || true
+          sudo bridge link > bridge-link.txt 2>&1 || true
+          for f in /sys/class/net/br-wan/brif/*/hairpin_mode /sys/class/net/br-wan/brif/*/learning /sys/class/net/br-wan/brif/*/flood; do
+            [ -e "$f" ] && echo "$f $(cat $f)"
+          done > hairpin-modes.txt 2>&1 || true
+          sudo iptables-save > iptables.txt 2>&1 || true
+          sudo nft list ruleset > nft.txt 2>&1 || true
+          sudo dmesg --ctime > dmesg.txt 2>&1 || true
+          sudo virsh list --all > virsh-list.txt 2>&1 || true
+          ls -la /var/lib/spinifex /var/log/spinifex 2>&1 > spinifex-dirs.txt || true
+          tar czf /tmp/spinifex-diag.tar.gz -C /tmp spinifex-diag
+          REMOTE
+          scp -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i "$SSH_KEY" \
+            tf-user@${WAN_IP}:/tmp/spinifex-diag.tar.gz "$DIAG_DIR/spinifex-diag.tar.gz" || true
+
+      - name: Upload Diagnostics
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: e2e-diag-pseudo-${{ github.run_id }}
+          path: ${{ runner.temp }}/diag-pseudo/
+          if-no-files-found: warn
+          retention-days: 14
+
       - name: Teardown
         if: always()
         working-directory: mulga/scripts/tofu-cluster
@@ -230,6 +388,72 @@ jobs:
           ssh -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
             "cd ~/e2e-tests && ./run-multinode-e2e.sh ${NODE_IPS} && ./run-cert-e2e.sh --multinode ${NODE_IPS} && ENDPOINT=https://${NODE1_IP}:9999 ./run-lb-e2e.sh --nodes ${NODE_IPS}"
 
+      - name: Collect Diagnostics
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          INVENTORY=$(tofu output -json inventory)
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          DIAG_DIR="${RUNNER_TEMP}/diag-multi"
+          mkdir -p "$DIAG_DIR"
+          cat > "${RUNNER_TEMP}/collect-diag.sh" <<'REMOTE'
+          set -x
+          DIAG=/tmp/spinifex-diag
+          sudo rm -rf "$DIAG" && mkdir -p "$DIAG" && cd "$DIAG"
+          sudo journalctl -u 'spinifex-*' --no-pager > journal-spinifex-full.log 2>&1 || true
+          sudo journalctl -u 'spinifex-*' --no-pager -n 10000 > journal-spinifex.log 2>&1 || true
+          sudo journalctl -u 'ovn-*' --no-pager -n 5000 > journal-ovn.log 2>&1 || true
+          sudo journalctl -u 'openvswitch-*' --no-pager -n 2000 > journal-ovs.log 2>&1 || true
+          sudo ovs-vsctl show > ovs-vsctl-show.txt 2>&1 || true
+          sudo ovs-ofctl dump-flows br-int > br-int-flows.txt 2>&1 || true
+          sudo ovs-ofctl dump-flows br-ex > br-ex-flows.txt 2>&1 || true
+          sudo ovs-appctl dpctl/dump-flows > dpctl-flows.txt 2>&1 || true
+          sudo ovs-appctl dpctl/dump-conntrack -m > dpctl-conntrack.txt 2>&1 || true
+          sudo ovn-nbctl show > ovn-nbctl-show.txt 2>&1 || true
+          sudo ovn-sbctl show > ovn-sbctl-show.txt 2>&1 || true
+          sudo ovn-sbctl list Port_Binding > sbdb-port-binding.txt 2>&1 || true
+          sudo ovn-sbctl list Chassis > sbdb-chassis.txt 2>&1 || true
+          sudo ovn-nbctl list NAT > nbdb-nat.txt 2>&1 || true
+          sudo ovn-nbctl list Logical_Router > nbdb-lr.txt 2>&1 || true
+          sudo ovn-nbctl list Logical_Router_Policy > nbdb-lr-policy.txt 2>&1 || true
+          sudo ovn-nbctl list Logical_Router_Static_Route > nbdb-lr-static-route.txt 2>&1 || true
+          sudo ovn-appctl -t ovn-controller ct-zone-list > ovn-ctzones.txt 2>&1 || true
+          sudo ip -d link > ip-link.txt 2>&1 || true
+          sudo ip addr > ip-addr.txt 2>&1 || true
+          sudo ip route > ip-route.txt 2>&1 || true
+          sudo ip neigh > ip-neigh.txt 2>&1 || true
+          sudo bridge fdb show > bridge-fdb.txt 2>&1 || true
+          sudo bridge link > bridge-link.txt 2>&1 || true
+          for f in /sys/class/net/br-wan/brif/*/hairpin_mode /sys/class/net/br-wan/brif/*/learning /sys/class/net/br-wan/brif/*/flood; do
+            [ -e "$f" ] && echo "$f $(cat $f)"
+          done > hairpin-modes.txt 2>&1 || true
+          sudo iptables-save > iptables.txt 2>&1 || true
+          sudo nft list ruleset > nft.txt 2>&1 || true
+          sudo dmesg --ctime > dmesg.txt 2>&1 || true
+          sudo virsh list --all > virsh-list.txt 2>&1 || true
+          ls -la /var/lib/spinifex /var/log/spinifex 2>&1 > spinifex-dirs.txt || true
+          tar czf /tmp/spinifex-diag.tar.gz -C /tmp spinifex-diag
+          REMOTE
+          NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
+          for i in $(seq 0 $((NODE_COUNT - 1))); do
+            NODE_IP=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
+            NODE_DIR="$DIAG_DIR/node${i}-${NODE_IP}"
+            mkdir -p "$NODE_DIR"
+            ssh -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i "$SSH_KEY" tf-user@${NODE_IP} 'bash -s' < "${RUNNER_TEMP}/collect-diag.sh" > "$NODE_DIR/collect.log" 2>&1 || true
+            scp -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i "$SSH_KEY" \
+              tf-user@${NODE_IP}:/tmp/spinifex-diag.tar.gz "$NODE_DIR/spinifex-diag.tar.gz" || true
+          done
+
+      - name: Upload Diagnostics
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: e2e-diag-multi-${{ github.run_id }}
+          path: ${{ runner.temp }}/diag-multi/
+          if-no-files-found: warn
+          retention-days: 14
+
       - name: Teardown
         if: always()
         working-directory: mulga/scripts/tofu-cluster

.github/workflows/release.yml

@@ -32,9 +32,9 @@ jobs:
         uses: docker/setup-buildx-action@v4
 
       - name: Set up Go
-        uses: actions/setup-go@v5
+        uses: actions/setup-go@v6
         with:
-          go-version: '1.26.3'
+          go-version: '1.26.4'
 
       - name: Build microVM artifacts (lb-agent + kernel + initramfs)
         run: cd spinifex && make build-microvm-image
@@ -326,12 +326,13 @@ jobs:
         env:
           AWS_ACCESS_KEY_ID: ${{ secrets.CF_R2_ACCESS_KEY_ID }}
           AWS_SECRET_ACCESS_KEY: ${{ secrets.CF_R2_SECRET_ACCESS_KEY }}
+          CF_R2_ENDPOINT_URL: ${{ secrets.CF_R2_ENDPOINT_URL }}
           VERSION: ${{ github.ref_name }}
         run: |
-          aws --endpoint-url=https://e80e329f7dbcf75b80e019c1dce640f2.r2.cloudflarestorage.com \
+          aws --endpoint-url="${CF_R2_ENDPOINT_URL}" \
             s3 cp "mulga/scripts/iso-builder/dist/spinifex-${VERSION}.iso" \
                   "s3://spinifex-iso/spinifex-${VERSION}.iso"
-          aws --endpoint-url=https://e80e329f7dbcf75b80e019c1dce640f2.r2.cloudflarestorage.com \
+          aws --endpoint-url="${CF_R2_ENDPOINT_URL}" \
             s3 cp "mulga/scripts/iso-builder/dist/spinifex-${VERSION}.iso.sha256" \
                   "s3://spinifex-iso/spinifex-${VERSION}.iso.sha256"
 

.github/workflows/spinifex.yml

@@ -61,6 +61,9 @@ jobs:
       - name: Run Tests with Coverage
         run: make -C spinifex test-cover
 
+      - name: Run Action Tests
+        run: make -C spinifex test-actions
+
       - name: Check Diff Coverage
         run: cd spinifex && scripts/diff-coverage.sh coverage.out --base HEAD~1
 
@@ -136,7 +139,7 @@ jobs:
       - name: Lint
         uses: golangci/golangci-lint-action@v9
         with:
-          version: v2.11.3
+          version: v2.12.2
           working-directory: spinifex
 
       - name: Govulncheck

Makefile

@@ -73,11 +73,24 @@ build-lb-agent:
 	@echo -e "\n....Building lb-agent (static)"
 	CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -ldflags "-s -w" -o ./bin/lb-agent cmd/lb-agent/main.go
 
-build-system-image: ## Build a system image from manifest (use IMAGE=lb)
+build-system-image: ## Build a system image from manifest (use IMAGE=lb or IMAGE=eks-server)
 ifndef IMAGE
 	$(error IMAGE is required. Usage: make build-system-image IMAGE=lb)
 endif
-	./scripts/build-system-image.sh scripts/images/$(IMAGE).conf
+	@if [ -f scripts/images/$(IMAGE).conf ]; then \
+		./scripts/build-system-image.sh scripts/images/$(IMAGE).conf; \
+	elif [ -f scripts/images/$(IMAGE)/manifest.conf ]; then \
+		./scripts/build-system-image.sh scripts/images/$(IMAGE)/manifest.conf; \
+	else \
+		echo "ERROR: no manifest at scripts/images/$(IMAGE).conf or scripts/images/$(IMAGE)/manifest.conf"; \
+		exit 1; \
+	fi
+
+build-eks-server-image: ## Build the eks-server AMI (K3s control-plane variant)
+	$(MAKE) build-system-image IMAGE=eks-server
+
+build-eks-agent-image: ## Build the eks-agent AMI (K3s worker variant)
+	$(MAKE) build-system-image IMAGE=eks-agent
 
 MICROVM_OUT_DIR := build/microvm
 MICROVM_ARTIFACTS := $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz
@@ -106,9 +119,23 @@ go_run:
 # Preflight — runs the same checks as GitHub Actions (lint + vuln + tests).
 # Use this before committing to catch CI failures locally.
 preflight:
-	@$(MAKE) --no-print-directory QUIET=1 lint govulncheck test-cover diff-coverage test-race
+	@$(MAKE) --no-print-directory QUIET=1 manifest-check lint govulncheck test-cover diff-coverage test-race test-harness
 	@echo -e "\n ✅ Preflight passed — safe to commit."
 
+# E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
+# default `go test ./spinifex/...`. Runs with mocked AWS clients — no
+# infrastructure required, safe to run in CI without a cluster.
+test-harness:
+	@echo -e "\n....Running e2e harness unit tests...."
+	$(_Q)LOG_IGNORE=1 go test -tags=e2e -timeout 60s ./tests/e2e/harness/... $(_RACEQ)
+
+# Validate docs/service-interfaces.yaml. Schema check + cross-reference
+# of services/suites/fixtures + on-disk path existence. Subject content
+# vs source is enforced separately in Bead 5 drift lint.
+manifest-check:
+	@echo -e "\n....Checking service-interfaces.yaml...."
+	@go run ./tests/e2e/manifest-check/cmd/manifest-check -repo-root . -manifest docs/service-interfaces.yaml
+
 # Run unit tests
 test:
 	@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
@@ -126,6 +153,12 @@ test-race:
 	@echo -e "\n....Running tests with race detector for $(GO_PROJECT_NAME)...."
 	$(_Q)LOG_IGNORE=1 go test -race -timeout 300s ./spinifex/... $(_RACEQ)
 
+# Unit tests for in-repo GitHub Actions (e.g. .github/actions/e2e-analyze).
+# Kept out of `test-cover` so coverage % isn't diluted by CI-only tooling.
+test-actions:
+	@echo -e "\n....Running action tests...."
+	LOG_IGNORE=1 go test -timeout 60s ./.github/actions/...
+
 # Check that new/changed code meets coverage threshold (runs tests first)
 diff-coverage: test-cover
 	@QUIET=$(QUIET) scripts/diff-coverage.sh $(COVERPROFILE)
@@ -170,14 +203,15 @@ install-system:
 		-o Dpkg::Options::="--force-confdef" \
 		-o Dpkg::Options::="--force-confold" \
 		nbdkit nbdkit-plugin-dev pkg-config $(QEMU_PACKAGES) qemu-utils qemu-kvm \
+		ovmf qemu-efi-aarch64 \
 		libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
 		iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
 		ovn-central ovn-host openvswitch-switch dhcpcd-base
 
 install-go:
-	@echo -e "\n....Installing Go 1.26.3 for $(ARCH) ($(GO_ARCH))...."
+	@echo -e "\n....Installing Go 1.26.4 for $(ARCH) ($(GO_ARCH))...."
 	@if [ ! -d "/usr/local/go" ]; then \
-		curl -L https://go.dev/dl/go1.26.3.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
+		curl -L https://go.dev/dl/go1.26.4.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
 	else \
 		echo "Go already installed in /usr/local/go"; \
 	fi
@@ -260,24 +294,53 @@ distro-clean:
 
 # Ansible dev lifecycle (experimental, parallel to dev-*.sh / reset-dev-env.sh).
 # See scripts/ansible/README.md and docs/development/improvements/ansible-dev-lifecycle.md.
+#
+# Variable overrides: pass EXTRA_VARS="key=val key2=val2" to forward as
+# `--extra-vars` to ansible-playbook. Plain `-e` on the make command line is
+# make's own flag and does NOT reach ansible.
+#   make ansible-dev-install EXTRA_VARS="spinifex_external_pool_start=192.168.1.90 spinifex_external_pool_end=192.168.1.99"
+EXTRA_VARS ?=
+_ANSIBLE_EXTRA = $(if $(strip $(EXTRA_VARS)),--extra-vars "$(EXTRA_VARS)",)
+
 ansible-dev-preflight:
-	cd scripts/ansible && ansible-playbook playbooks/dev-preflight.yml
+	cd scripts/ansible && ansible-playbook playbooks/dev-preflight.yml $(_ANSIBLE_EXTRA)
 
 ansible-dev-teardown:
-	cd scripts/ansible && ansible-playbook playbooks/dev-teardown.yml
+	cd scripts/ansible && ansible-playbook playbooks/dev-teardown.yml $(_ANSIBLE_EXTRA)
 
 ansible-dev-install:
-	cd scripts/ansible && ansible-playbook playbooks/dev-install.yml
+	cd scripts/ansible && ansible-playbook playbooks/dev-install.yml $(_ANSIBLE_EXTRA)
 
 ansible-dev-reset:
-	cd scripts/ansible && ansible-playbook playbooks/dev-reset.yml
+	cd scripts/ansible && ansible-playbook playbooks/dev-reset.yml $(_ANSIBLE_EXTRA)
 
 ansible-dev-deploy:
-	cd scripts/ansible && ansible-playbook playbooks/dev-deploy.yml
+	cd scripts/ansible && ansible-playbook playbooks/dev-deploy.yml $(_ANSIBLE_EXTRA)
 
-.PHONY: build build-ui build-installer build-lb-agent build-system-image build-microvm-image install-microvm go_build go_run preflight test test-cover test-race diff-coverage bench run \
+ansible-dev-status:
+	cd scripts/ansible && ansible-playbook playbooks/dev-status.yml
+
+ansible-dev-logs:
+	cd scripts/ansible && ansible-playbook playbooks/dev-logs.yml
+
+# Snapshot / restore require an explicit -e snapshot_name=<name>. The
+# wrapper passes ANSIBLE_EXTRA through so devs can run
+# `make ansible-dev-snapshot ANSIBLE_EXTRA='-e snapshot_name=before-merge'`.
+ansible-dev-snapshot:
+	cd scripts/ansible && ansible-playbook playbooks/dev-snapshot.yml $(ANSIBLE_EXTRA)
+
+ansible-dev-restore:
+	cd scripts/ansible && ansible-playbook playbooks/dev-restore.yml $(ANSIBLE_EXTRA)
+
+ansible-dev-version:
+	cd scripts/ansible && ansible-playbook playbooks/dev-version.yml
+
+ansible-dev-vpc:
+	cd scripts/ansible && ansible-playbook playbooks/dev-vpc.yml $(ANSIBLE_EXTRA)
+
+.PHONY: build build-ui build-installer build-lb-agent build-system-image build-eks-server-image build-eks-agent-image build-microvm-image install-microvm go_build go_run preflight test test-cover test-race diff-coverage bench run test-actions test-harness manifest-check diff-coverage bench run \
 	deploy reinstall clean \
 	install-system install-go install-aws quickinstall \
 	lint fix govulncheck \
 	distro distro-amd64 distro-arm64 distro-clean \
-	ansible-dev-preflight ansible-dev-teardown ansible-dev-install ansible-dev-reset ansible-dev-deploy
+	ansible-dev-preflight ansible-dev-teardown ansible-dev-install ansible-dev-reset ansible-dev-deploy ansible-dev-status ansible-dev-logs ansible-dev-snapshot ansible-dev-restore ansible-dev-version ansible-dev-vpc

build/systemd/spinifex-daemon.service

@@ -1,7 +1,14 @@
 [Unit]
 Description=Spinifex Daemon (VM Orchestration)
 PartOf=spinifex.target
-After=spinifex-nats.service spinifex-viperblock.service openvswitch-switch.service ovn-controller.service
+After=spinifex-nats.service spinifex-viperblock.service openvswitch-switch.service ovn-controller.service openvswitch-ipsec.service
+# Wants=openvswitch-ipsec.service so systemd brings the IPsec userspace
+# daemon up before spinifex flips other_config:ipsec_encapsulation=true.
+# Without that daemon, OVS holds the cert pointers unused and strongSwan
+# never gets IKE config — tunnel traffic is encrypted with no SAs and
+# silently dropped. Wants= (not Requires=) keeps daemon bootable on
+# single-node / lab hosts where the package is intentionally absent.
+Wants=openvswitch-ipsec.service
 # No Wants=spinifex-nats.service: Wants= reactivates a stopped unit during
 # `systemctl restart spinifex-daemon`, which would mask the standalone-startup
 # path the Tier 1 DDIL Scenario B exercises. spinifex.target Wants= covers
@@ -23,7 +30,7 @@ Group=spinifex
 ExecStart=/usr/local/bin/spx service spinifex start
 Restart=on-failure
 RestartSec=5
-# DDIL Tier 1 (mulga-siv-58): KillMode=process so systemd only signals the
+# DDIL Tier 1: KillMode=process so systemd only signals the
 # daemon's main PID, not the cgroup. QEMU children stay alive across
 # `systemctl restart spinifex-daemon` and the new daemon reattaches to them
 # via the persisted local state file + pidfile + QMP socket. With the
@@ -38,7 +45,7 @@ OOMScoreAdjust=-500
 Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
 Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex/
 Environment=SPINIFEX_WAL_DIR=/var/lib/spinifex/spinifex/
-# DDIL Tier 1 (mulga-siv-58): QEMU pidfiles, QMP sockets, and the
+# DDIL Tier 1: QEMU pidfiles, QMP sockets, and the
 # shared qemu.log all resolve via utils.pidPath() — which falls back
 # to os.TempDir() (=/tmp) when $HOME/spinifex doesn't exist. With
 # PrivateTmp=yes, each daemon start gets a fresh /tmp namespace, so

cmd/dhcptest/main.go

@@ -89,7 +89,7 @@ func main() {
 		}
 	}
 
-	// Build nclient4 options — identical to spinifex/services/vpcd/dhcp/nclient4.go.
+	// Build nclient4 options — standalone DHCP probe for upstream-router debugging.
 	opts := []nclient4.ClientOpt{
 		nclient4.WithHWAddr(hwAddr),
 		nclient4.WithTimeout(*timeout),
@@ -105,7 +105,7 @@ func main() {
 	}
 	defer client.Close()
 
-	// Identity modifiers — identical to identityModifiers() in vpcd dhcp package.
+	// Identity modifiers — client-id / hostname / vendor-class for upstream lease tagging.
 	var mods []dhcpv4.Modifier
 	if cid != "" {
 		mods = append(mods, dhcpv4.WithOption(dhcpv4.OptClientIdentifier([]byte(cid))))

cmd/eks-token-webhook/main.go

@@ -0,0 +1,38 @@
+// eks-token-webhook is the K8s TokenReview webhook authenticator baked into
+// the eks-server AMI. It decodes the SigV4-presigned GetCallerIdentity URL
+// produced by `aws eks get-token`, enforces the X-K8s-Aws-Id cross-cluster
+// pin, calls Mulga STS over NATS, and resolves the principal to a TokenReview
+// response via the cluster's AccessEntry KV.
+//
+// This is a placeholder build that returns 503 Service Unavailable on every
+// path. It exists so the eks-server AMI build succeeds (INSTALL_BINARIES is
+// satisfied) and the K3s server VM boots cleanly. Real implementation lands
+// with bead mulga-cs-eks-6c.
+package main
+
+import (
+	"flag"
+	"log/slog"
+	"net/http"
+	"os"
+)
+
+func main() {
+	addr := flag.String("addr", "127.0.0.1:8443", "listen address")
+	flag.Parse()
+
+	slog.SetDefault(slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelInfo})))
+
+	mux := http.NewServeMux()
+	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Retry-After", "30")
+		w.Header().Set("Content-Type", "application/json")
+		http.Error(w, `{"error":"eks-token-webhook stub: real implementation not yet deployed"}`, http.StatusServiceUnavailable)
+	})
+
+	slog.Info("eks-token-webhook stub starting", "addr", *addr)
+	if err := http.ListenAndServe(*addr, mux); err != nil {
+		slog.Error("listen failed", "err", err)
+		os.Exit(1)
+	}
+}

cmd/installer/install/install.go

@@ -27,6 +27,7 @@ import (
 	"os"
 	"os/exec"
 	"path/filepath"
+	"slices"
 	"strings"
 	"syscall"
 	"time"
@@ -303,9 +304,10 @@ func installSpinifex(cfg *Config) error {
 		return err
 	}
 
-	// dhcpcd-base is present on the installed system (used by setup-ovn.sh for
-	// macvlan mode). Mask the standalone dhcpcd.service so it never auto-starts
-	// and races with systemd-networkd's built-in DHCP client on br-wan.
+	// dhcpcd-base is present on the installed system (used by setup-ovn.sh
+	// for WAN DHCP acquisition). Mask the standalone dhcpcd.service so it
+	// never auto-starts and races with systemd-networkd's built-in DHCP
+	// client on br-wan.
 	if err := maskSystemdUnit(mountRoot, "dhcpcd.service"); err != nil {
 		slog.Warn("installSpinifex: failed to mask dhcpcd.service", "err", err)
 	}
@@ -850,8 +852,8 @@ func bindChrootMounts() error {
 // unbindChrootMounts unmounts the virtual filesystems in reverse order.
 // Errors are logged but not returned — this is best-effort cleanup.
 func unbindChrootMounts() {
-	for i := len(chrootMountPaths) - 1; i >= 0; i-- {
-		_ = run("umount", filepath.Join(mountRoot, chrootMountPaths[i]))
+	for _, v := range slices.Backward(chrootMountPaths) {
+		_ = run("umount", filepath.Join(mountRoot, v))
 	}
 }
 

cmd/spinifex/cmd/admin.go

@@ -17,6 +17,7 @@ along with this program.  If not, see <https://www.gnu.org/licenses/>.
 package cmd
 
 import (
+	"bufio"
 	"bytes"
 	"context"
 	"crypto/tls"
@@ -48,6 +49,7 @@ import (
 	"github.com/mulgadc/spinifex/spinifex/gpu"
 	handlers_ec2_vpc "github.com/mulgadc/spinifex/spinifex/handlers/ec2/vpc"
 	handlers_iam "github.com/mulgadc/spinifex/spinifex/handlers/iam"
+	"github.com/mulgadc/spinifex/spinifex/objectstore"
 	"github.com/mulgadc/spinifex/spinifex/utils"
 	"github.com/mulgadc/viperblock/viperblock"
 	"github.com/mulgadc/viperblock/viperblock/backends/s3"
@@ -84,6 +86,14 @@ var supportedPlatforms = map[string]bool{
 	"Windows":    true,
 }
 
+// Mirrors the gateway RegisterImage allowlist so admin imports can't write an
+// AMI with a boot mode that RegisterImage would reject.
+var supportedBootModes = map[string]bool{
+	"bios":           true,
+	"uefi":           true,
+	"uefi-preferred": true,
+}
+
 var adminCmd = &cobra.Command{
 	Use:   "admin",
 	Short: "Administrative commands for Spinifex platform management",
@@ -143,6 +153,19 @@ var imagesListCmd = &cobra.Command{
 	Run:   runimagesListCmd,
 }
 
+var imagesRemoveCmd = &cobra.Command{
+	Use:   "remove",
+	Short: "Remove an admin-imported system AMI",
+	Long: `Delete an AMI imported via 'spx admin images import', including its
+backing block storage and snapshot artefacts. Only operates on AMIs with a
+non-account owner (e.g. "system"); account-owned AMIs must be removed via
+'aws ec2 deregister-image' followed by 'aws ec2 delete-snapshot'.
+
+Refuses to delete an AMI that has dependent volumes or copied snapshots/AMIs
+unless --force is passed. Prompts for confirmation unless --yes is passed.`,
+	Run: runimagesRemoveCmd,
+}
+
 var accountCmd = &cobra.Command{
 	Use:   "account",
 	Short: "Manage Spinifex accounts",
@@ -225,6 +248,7 @@ func init() {
 	adminCmd.AddCommand(imagesCmd)
 	imagesCmd.AddCommand(imagesImportCmd)
 	imagesCmd.AddCommand(imagesListCmd)
+	imagesCmd.AddCommand(imagesRemoveCmd)
 
 	adminCmd.AddCommand(accountCmd)
 	accountCmd.AddCommand(accountCreateCmd)
@@ -270,12 +294,14 @@ func init() {
 	adminInitCmd.Flags().String("external-mode", "", "External network mode: 'pool' (default when WAN detected) or '' (disabled)")
 	adminInitCmd.Flags().String("external-iface", "", "WAN NIC for br-external (auto-detected from default route)")
 	adminInitCmd.Flags().String("external-source", "", "Pool IP source: 'dhcp' (default when no --external-pool) or 'static' (uses --external-pool range)")
+	adminInitCmd.Flags().String("external-bind-bridge", "", "Linux bridge for upstream DHCP DORA (default 'br-wan' when --external-source=dhcp)")
 	adminInitCmd.Flags().String("external-pool", "", "External IP pool range as start-end (e.g., 192.168.1.150-192.168.1.250)")
 	adminInitCmd.Flags().String("external-gateway", "", "WAN gateway IP (auto-detected from default route)")
 	adminInitCmd.Flags().String("gateway-ip", "", "OVN gateway router's external IP for SNAT (default: pool range_start for pool mode, required for nat mode without DHCP)")
 	adminInitCmd.Flags().Int("external-prefix-len", 24, "External pool subnet prefix length (auto-detected)")
 	adminInitCmd.Flags().Bool("no-external", false, "Disable external networking (overlay-only, no internet for VMs)")
 	adminInitCmd.Flags().Bool("gpu-passthrough", false, "Enable VFIO GPU passthrough (sets gpu_passthrough = true in daemon config)")
+	adminInitCmd.Flags().Bool("ipsec", true, "Encrypt intra-AZ Geneve via OVN native IPsec (cluster-wide); disable only for trusted single-rack lab")
 
 	// Flags for admin join
 	adminJoinCmd.Flags().String("region", "ap-southeast-2", "Region for this node")
@@ -304,9 +330,17 @@ func init() {
 	imagesImportCmd.Flags().String("version", "", "Specified distro version (e.g 12)")
 	imagesImportCmd.Flags().String("arch", "", "Specified distro arch (e.g aarch64, arm64, x86_64)")
 	imagesImportCmd.Flags().String("platform", "Linux/UNIX", "Specified platform (e.g Linux/UNIX, Windows)")
+	imagesImportCmd.Flags().String("boot-mode", "", "Boot mode for the imported AMI (bios|uefi|uefi-preferred). Required with --file. Overrides the catalog value when used with --name.")
 	imagesImportCmd.Flags().StringSlice("tag", nil, "Tag to apply to the imported AMI as key=value (repeatable; e.g. --tag spinifex:managed-by=elbv2)")
 	imagesImportCmd.Flags().Bool("force", false, "Force command execution (overwrites existing files)")
 	imagesImportCmd.Flags().Bool("skip-verify", false, "Skip catalog-image checksum verification (INSECURE; operator assumes integrity responsibility)")
+
+	imagesRemoveCmd.Flags().String("image-id", "", "AMI ID to remove (required)")
+	imagesRemoveCmd.Flags().Bool("force", false, "Bypass dependency, ownership and config-corrupt checks (salvage mode)")
+	imagesRemoveCmd.Flags().Bool("yes", false, "Skip interactive confirmation prompt")
+	if err := imagesRemoveCmd.MarkFlagRequired("image-id"); err != nil {
+		panic(err)
+	}
 }
 
 func runimagesImportCmd(cmd *cobra.Command, args []string) {
@@ -376,6 +410,22 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 		}
 	}
 
+	// --file imports have no catalog metadata to inherit from, so the operator
+	// must declare the boot mode explicitly — guessing would silently produce
+	// a BIOS AMI from a UEFI-only image (or vice versa) and fail at launch.
+	// --name imports inherit from the catalog; the flag overrides when set.
+	bootModeFlag, _ := cmd.Flags().GetString("boot-mode")
+	if bootModeFlag != "" {
+		if !supportedBootModes[bootModeFlag] {
+			fmt.Fprintf(os.Stderr, "Unsupported --boot-mode %q (expected bios|uefi|uefi-preferred)\n", bootModeFlag)
+			os.Exit(1)
+		}
+		image.BootMode = bootModeFlag
+	} else if imageName == "" {
+		fmt.Fprintf(os.Stderr, "--boot-mode is required when importing via --file (expected bios|uefi|uefi-preferred)\n")
+		os.Exit(1)
+	}
+
 	// --tag k=v (repeatable) merges user-supplied tags into the AMI. Overrides
 	// catalog tags on key collision so operators can re-tag a known image.
 	tagFlags, _ := cmd.Flags().GetStringSlice("tag")
@@ -496,6 +546,9 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 	manifest.AMIMetadata.Virtualization = "hvm"
 	manifest.AMIMetadata.ImageOwnerAlias = "system"
 	manifest.AMIMetadata.VolumeSizeGiB = utils.SafeInt64ToUint64(imageStat.Size() / 1024 / 1024 / 1024)
+	manifest.AMIMetadata.BootMode = image.BootMode
+	manifest.AMIMetadata.Distro = image.Distro
+	manifest.AMIMetadata.DistroFamily = utils.DistroFamily(image.Distro)
 
 	// Copy catalog-provided tags (e.g. spinifex:managed-by for system AMIs)
 	// onto the imported AMI so the UI can filter them out.
@@ -550,6 +603,12 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 		Host:       appConfig.Nodes[appConfig.Node].Predastore.Host,
 	}
 
+	mkey, err := utils.LoadViperblockMasterKey(appConfig.Nodes[appConfig.Node].Viperblock.EncryptionKeyFile)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Could not load viperblock encryption key: %v\n", err)
+		os.Exit(1)
+	}
+
 	vbConfig := viperblock.VB{
 		VolumeName: volumeId,
 		VolumeSize: utils.SafeInt64ToUint64(imageStat.Size()),
@@ -559,7 +618,9 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 				Size: 0,
 			},
 		},
-		VolumeConfig: manifest,
+		VolumeConfig:      manifest,
+		MasterKey:         mkey,
+		EncryptionEnabled: mkey != nil,
 	}
 
 	err = v_utils.ImportDiskImage(&s3Config, &vbConfig, extractedImagePath)
@@ -574,6 +635,136 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 	fmt.Printf("✅ Image import complete. Image-ID (AMI): %s\n", volumeId)
 }
 
+func runimagesRemoveCmd(cmd *cobra.Command, args []string) {
+	imageID, _ := cmd.Flags().GetString("image-id")
+	force, _ := cmd.Flags().GetBool("force")
+	yes, _ := cmd.Flags().GetBool("yes")
+
+	cfgFile, _ := cmd.Flags().GetString("config")
+	if cfgFile == "" {
+		cfgFile = DefaultConfigFile()
+	}
+
+	appConfig, err := config.LoadConfig(cfgFile)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "Error loading config file:", err)
+		os.Exit(1)
+	}
+
+	node := appConfig.Nodes[appConfig.Node]
+	store := objectstore.NewS3ObjectStoreFromConfig(
+		node.Predastore.Host,
+		node.Predastore.Region,
+		node.Predastore.AccessKey,
+		node.Predastore.SecretKey,
+	)
+	bucket := node.Predastore.Bucket
+
+	preview, err := admin.PreviewRemoveSystemImage(store, bucket, imageID)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "Failed to inspect AMI:", err)
+		os.Exit(1)
+	}
+
+	// Print metadata block.
+	fmt.Println("About to remove system AMI:")
+	fmt.Println()
+	fmt.Printf("  Image ID:        %s\n", preview.ImageID)
+	switch {
+	case !preview.ConfigPresent && !preview.ConfigCorrupt:
+		fmt.Println("  Name:            <unknown — config.json missing>")
+		fmt.Println("  Owner:           <unknown>")
+	case preview.ConfigCorrupt:
+		fmt.Println("  Name:            <unknown — config.json corrupt>")
+		fmt.Println("  Owner:           <unknown>")
+	default:
+		fmt.Printf("  Name:            %s\n", preview.Name)
+		fmt.Printf("  Owner:           %s\n", preview.Owner)
+		if !preview.Created.IsZero() {
+			fmt.Printf("  Created:         %s\n", preview.Created.UTC().Format("2006-01-02T15:04:05Z"))
+		}
+	}
+	fmt.Printf("  Backing storage: %s/      (%d objects, %s)\n",
+		preview.ImageID, preview.AMIObjectCount, utils.HumanBytes(utils.SafeInt64ToUint64(preview.AMIBytesTotal)))
+	fmt.Printf("                   %s/ (%d objects, %s)\n",
+		admin.SnapPrefix(preview.ImageID), preview.SnapObjectCount, utils.HumanBytes(utils.SafeInt64ToUint64(preview.SnapBytesTotal)))
+	fmt.Println()
+
+	// Account-owned guard before salvage / dependents — the AWS-flow hint is
+	// the most useful thing to surface for this kind of mistake.
+	if preview.ConfigPresent && !preview.IsSystemOwned && !force {
+		fmt.Fprintf(os.Stderr,
+			"Refusing to remove: %s is account-owned (%s).\n"+
+				"Use 'aws ec2 deregister-image --image-id %s' followed by 'aws ec2 delete-snapshot ...'.\n",
+			preview.ImageID, preview.Owner, preview.ImageID)
+		os.Exit(1)
+	}
+
+	if !preview.ConfigPresent && !force {
+		fmt.Fprintln(os.Stderr, "AMI config.json missing or corrupt; re-run with --force to salvage backing blocks.")
+		os.Exit(1)
+	}
+
+	if !preview.Dependents.Empty() && !force {
+		fmt.Fprintln(os.Stderr, "Refusing to remove: dependent resources reference this image.")
+		printDependents(os.Stderr, preview.Dependents)
+		fmt.Fprintln(os.Stderr, "Remove them first (e.g. 'aws ec2 terminate-instances', 'aws ec2 delete-snapshot', 'aws ec2 deregister-image') or re-run with --force.")
+		os.Exit(1)
+	}
+
+	if force && (!preview.ConfigPresent || !preview.Dependents.Empty()) {
+		fmt.Println("⚠️  --force: skipping dependency check and ownership check.")
+		if !preview.Dependents.Empty() {
+			printDependents(os.Stdout, preview.Dependents)
+		}
+		fmt.Println()
+	}
+
+	fmt.Println("This is permanent and cannot be undone.")
+	if !yes {
+		fmt.Print("Type 'yes' to proceed: ")
+		reader := bufio.NewReader(os.Stdin)
+		answer, _ := reader.ReadString('\n')
+		if strings.TrimSpace(answer) != "yes" {
+			fmt.Println("Aborted.")
+			return
+		}
+	}
+
+	res, err := admin.RemoveSystemImage(store, bucket, admin.RemoveImageOpts{
+		ImageID: imageID,
+		Force:   force,
+	})
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "Remove failed:", err)
+		os.Exit(1)
+	}
+
+	fmt.Printf("✅ Removed AMI %s (freed %s across %d objects).\n",
+		imageID, utils.HumanBytes(utils.SafeInt64ToUint64(res.BytesFreed)), res.ObjectsDeleted)
+}
+
+func printDependents(w io.Writer, d admin.Dependents) {
+	if len(d.Volumes) > 0 {
+		fmt.Fprintf(w, "  Volumes (%d):\n", len(d.Volumes))
+		for _, v := range d.Volumes {
+			fmt.Fprintf(w, "    - %s\n", v)
+		}
+	}
+	if len(d.Snapshots) > 0 {
+		fmt.Fprintf(w, "  Snapshots (%d):\n", len(d.Snapshots))
+		for _, s := range d.Snapshots {
+			fmt.Fprintf(w, "    - %s\n", s)
+		}
+	}
+	if len(d.AMIs) > 0 {
+		fmt.Fprintf(w, "  AMIs (%d):\n", len(d.AMIs))
+		for _, a := range d.AMIs {
+			fmt.Fprintf(w, "    - %s\n", a)
+		}
+	}
+}
+
 // List remote images available
 func runimagesListCmd(cmd *cobra.Command, args []string) {
 	//fmt.Println(availableImages)
@@ -657,12 +848,14 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	externalMode, _ := cmd.Flags().GetString("external-mode")
 	externalIface, _ := cmd.Flags().GetString("external-iface")
 	externalSource, _ := cmd.Flags().GetString("external-source")
+	externalBindBridge, _ := cmd.Flags().GetString("external-bind-bridge")
 	externalPool, _ := cmd.Flags().GetString("external-pool")
 	externalGateway, _ := cmd.Flags().GetString("external-gateway")
 	externalPrefixLen, _ := cmd.Flags().GetInt("external-prefix-len")
 	gatewayIP, _ := cmd.Flags().GetString("gateway-ip")
 	noExternal, _ := cmd.Flags().GetBool("no-external")
 	gpuPassthrough, _ := cmd.Flags().GetBool("gpu-passthrough")
+	ipsecEnabled, _ := cmd.Flags().GetBool("ipsec")
 
 	// Fire telemetry in background (completes during init work, waited at end)
 	noTelemetry, _ := cmd.Flags().GetBool("no-telemetry")
@@ -712,9 +905,9 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 				fmt.Printf("  %-14s %-18s %-20s %-16s %s\n", iface.Name, iface.IP, iface.Subnet, gw, strings.ToUpper(iface.Role))
 			}
 			if detected.LANCount == 0 {
-				fmt.Println("\n  Mode: single-NIC (macvlan for external bridge)")
+				fmt.Println("\n  Mode: single-NIC (veth-bridged external)")
 			} else {
-				fmt.Printf("\n  Mode: %d LAN + 1 WAN (macvlan for external bridge)\n", detected.LANCount)
+				fmt.Printf("\n  Mode: %d LAN + 1 WAN (veth-bridged external)\n", detected.LANCount)
 			}
 
 			// Apply auto-detected values when flags not explicitly set
@@ -729,13 +922,11 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 					externalPrefixLen = detected.WAN.PrefixLen
 				}
 
-				// Default mode: always "pool" — with static range if --external-pool
-				// is given, otherwise with source=dhcp (gateway IP from router DHCP)
+				// Default mode: always "pool". Source defaults to "static"; if
+				// --external-pool is omitted the validator below will error with
+				// a SuggestPoolRange hint.
 				if externalMode == "" && !cmd.Flags().Changed("external-mode") {
 					externalMode = "pool"
-					if externalPool == "" && externalSource == "" {
-						externalSource = "dhcp"
-					}
 				}
 			}
 		}
@@ -747,25 +938,30 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 		os.Exit(1)
 	}
 	if externalMode == "pool" {
-		// Resolve source: if not explicitly set, infer from whether a pool range was given
+		// Default source: dhcp when no --external-pool, else static.
 		if externalSource == "" {
-			if externalPool != "" {
-				externalSource = "static"
-			} else {
+			if externalPool == "" {
 				externalSource = "dhcp"
+			} else {
+				externalSource = "static"
 			}
 		}
-		if externalSource == "dhcp" {
-			// DHCP source: no static range needed, just gateway
-			if externalGateway == "" {
-				fmt.Fprintf(os.Stderr, "❌ Error: --external-gateway is required when --external-mode=pool\n")
+		switch externalSource {
+		case "dhcp":
+			if externalPool != "" {
+				fmt.Fprintf(os.Stderr, "❌ Error: --external-pool not allowed with --external-source=dhcp (addresses come from upstream DHCP server)\n")
+				os.Exit(1)
+			}
+			if externalBindBridge == "" {
+				externalBindBridge = "br-wan"
+			}
+		case "static":
+			if externalBindBridge != "" {
+				fmt.Fprintf(os.Stderr, "❌ Error: --external-bind-bridge only valid with --external-source=dhcp\n")
 				os.Exit(1)
 			}
-		} else {
-			// Static source: need pool range
 			if externalPool == "" {
-				fmt.Fprintf(os.Stderr, "❌ Error: --external-pool is required when --external-mode=pool (e.g., 192.168.1.150-192.168.1.250)\n")
-				fmt.Fprintf(os.Stderr, "   Or use --external-source=dhcp to get IPs from router DHCP\n")
+				fmt.Fprintf(os.Stderr, "❌ Error: --external-pool is required with --external-source=static (e.g., 192.168.1.150-192.168.1.250)\n")
 				if detectedNet != nil && detectedNet.WAN != nil {
 					sugStart, sugEnd := admin.SuggestPoolRange(detectedNet.WAN)
 					fmt.Fprintf(os.Stderr, "   Suggested: --external-pool=%s-%s\n", sugStart, sugEnd)
@@ -773,7 +969,7 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 				os.Exit(1)
 			}
 			if externalGateway == "" {
-				fmt.Fprintf(os.Stderr, "❌ Error: --external-gateway is required when --external-mode=pool\n")
+				fmt.Fprintf(os.Stderr, "❌ Error: --external-gateway is required with --external-source=static\n")
 				os.Exit(1)
 			}
 			parts := strings.SplitN(externalPool, "-", 2)
@@ -782,6 +978,9 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 				os.Exit(1)
 			}
 			poolStart, poolEnd = parts[0], parts[1]
+		default:
+			fmt.Fprintf(os.Stderr, "❌ Error: --external-source must be 'static' or 'dhcp', got: %s\n", externalSource)
+			os.Exit(1)
 		}
 	}
 	if externalGateway != "" && net.ParseIP(externalGateway) == nil {
@@ -802,10 +1001,6 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 		}
 	}
 
-	// For pool/dhcp source, the gateway IP is obtained via DHCP on the
-	// macvlan/bridge interface (no static pool range or explicit gateway-ip).
-	useExternalDHCP := externalMode == "pool" && externalSource == "dhcp" && gatewayIP == ""
-
 	// Validate IP address format
 	if net.ParseIP(bindIP) == nil {
 		fmt.Fprintf(os.Stderr, "❌ Error: Invalid IP address for --bind: %s\n", bindIP)
@@ -923,6 +1118,18 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	// Generate SSL certificates (with bind IP in SANs for multi-node support)
 	certPath := admin.GenerateCertificatesIfNeeded(configDir, force, bindIP)
 
+	// Generate per-node IPsec peer cert when cluster-wide IPsec is enabled
+	// (default true). Reuses the cluster CA — no intermediate strongSwan PKI.
+	if ipsecEnabled {
+		caCertPath := filepath.Join(configDir, "ca.pem")
+		caKeyPath := filepath.Join(configDir, "ca.key")
+		if err := admin.GenerateIPSecPeerCert(configDir, caCertPath, caKeyPath, node, bindIP); err != nil {
+			fmt.Fprintf(os.Stderr, "Error generating IPsec peer certificate: %v\n", err)
+			os.Exit(1)
+		}
+		fmt.Println("🔐 IPsec peer certificate generated (intra-AZ Geneve encryption ON)")
+	}
+
 	// Install CA certificate into system trust store
 	installCACertificate(filepath.Join(configDir, "ca.pem"))
 
@@ -944,30 +1151,32 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	isMultiNode := nodes >= 2
 
 	if isMultiNode {
-		// Build cluster-wide network config for propagation to joining nodes
-		var networkConfig *formation.NetworkConfig
+		// Build cluster-wide network config for propagation to joining nodes.
+		// Always emit so the IPSecEnabled flag reaches joiners even when
+		// external networking is disabled.
+		networkConfig := &formation.NetworkConfig{
+			IPSecEnabled: ipsecEnabled,
+		}
 		if externalMode != "" {
 			bootstrapVpcId := utils.GenerateResourceID("vpc")
 			bootstrapSubnetId := utils.GenerateResourceID("subnet")
 			bootstrapIgwId := utils.GenerateResourceID("igw")
-			networkConfig = &formation.NetworkConfig{
-				ExternalMode:        externalMode,
-				ExternalDHCP:        useExternalDHCP,
-				PoolName:            "wan",
-				PoolSource:          externalSource,
-				PoolStart:           poolStart,
-				PoolEnd:             poolEnd,
-				PoolGateway:         externalGateway,
-				PoolGatewayIP:       gatewayIP,
-				PoolPrefixLen:       externalPrefixLen,
-				PoolDNSServers:      dnsServers,
-				BootstrapAccountId:  admin.DefaultAccountID(),
-				BootstrapVpcId:      bootstrapVpcId,
-				BootstrapSubnetId:   bootstrapSubnetId,
-				BootstrapIgwId:      bootstrapIgwId,
-				BootstrapCidr:       handlers_ec2_vpc.DefaultVPCCidr,
-				BootstrapSubnetCidr: handlers_ec2_vpc.DefaultSubnetCidr,
-			}
+			networkConfig.ExternalMode = externalMode
+			networkConfig.PoolName = "wan"
+			networkConfig.PoolSource = externalSource
+			networkConfig.PoolBindBridge = externalBindBridge
+			networkConfig.PoolStart = poolStart
+			networkConfig.PoolEnd = poolEnd
+			networkConfig.PoolGateway = externalGateway
+			networkConfig.PoolGatewayIP = gatewayIP
+			networkConfig.PoolPrefixLen = externalPrefixLen
+			networkConfig.PoolDNSServers = dnsServers
+			networkConfig.BootstrapAccountId = admin.DefaultAccountID()
+			networkConfig.BootstrapVpcId = bootstrapVpcId
+			networkConfig.BootstrapSubnetId = bootstrapSubnetId
+			networkConfig.BootstrapIgwId = bootstrapIgwId
+			networkConfig.BootstrapCidr = handlers_ec2_vpc.DefaultVPCCidr
+			networkConfig.BootstrapSubnetCidr = handlers_ec2_vpc.DefaultSubnetCidr
 		}
 
 		runAdminInitMultiNode(cmd, accessKey, secretKey, accountID, natsToken, clusterName,
@@ -1067,10 +1276,9 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 
 		ExternalMode:   externalMode,
 		ExternalIface:  externalIface,
-		DhcpBindBridge: detectedDhcpBindBridge(detectedNet),
-		ExternalDHCP:   useExternalDHCP,
 		PoolName:       "wan",
 		PoolSource:     externalSource,
+		PoolBindBridge: externalBindBridge,
 		PoolStart:      poolStart,
 		PoolEnd:        poolEnd,
 		PoolGateway:    externalGateway,
@@ -1087,20 +1295,23 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 		BootstrapSubnetCidr: handlers_ec2_vpc.DefaultSubnetCidr,
 
 		GPUPassthrough: gpuPassthrough,
+		IPSecEnabled:   ipsecEnabled,
 	}
 
 	// Print external networking summary
 	if externalMode != "" {
 		fmt.Printf("\n📡 External networking: %s\n", externalMode)
 		fmt.Printf("  WAN interface: %s\n", externalIface)
-		if poolStart != "" {
+		switch externalSource {
+		case "static":
 			fmt.Printf("  Source:        static (IP range)\n")
 			fmt.Printf("  IP pool:       %s - %s\n", poolStart, poolEnd)
 			fmt.Printf("  ⚠️  Ensure %s-%s is excluded from your router's DHCP range.\n", poolStart, poolEnd)
-		} else if useExternalDHCP {
-			fmt.Printf("  Source:        dhcp (from router DHCP)\n")
-			fmt.Println("  Gateway IP:    DHCP (obtained during OVN setup)")
-		} else if gatewayIP != "" {
+		case "dhcp":
+			fmt.Printf("  Source:        dhcp (upstream DHCP server)\n")
+			fmt.Printf("  Bind bridge:   %s\n", externalBindBridge)
+		}
+		if gatewayIP != "" {
 			fmt.Printf("  Gateway IP:    %s (static)\n", gatewayIP)
 		}
 	}
@@ -1572,7 +1783,7 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 		}
 
 		fmt.Printf("   Waiting for cluster formation... (%d/%d nodes joined)\n", statusResp.Joined, statusResp.Expected)
-		time.Sleep(2 * time.Second)
+		time.Sleep(500 * time.Millisecond)
 	}
 
 	fmt.Printf("✅ Cluster formation complete! (%d nodes)\n\n", statusResp.Expected)
@@ -1680,6 +1891,24 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 	}
 	fmt.Printf("✅ Server certificate generated with bind IP: %s\n\n", bindIP)
 
+	// Match the leader's intra-AZ IPsec posture: NetworkConfig.IPSecEnabled is
+	// authoritative. When the formation response omits NetworkConfig entirely
+	// (no current code path; defensive guard against future regressions) fall
+	// back to the AWS-parity default of ON.
+	ipsecEnabled := true
+	if statusResp.NetworkConfig != nil {
+		ipsecEnabled = statusResp.NetworkConfig.IPSecEnabled
+	}
+	if ipsecEnabled {
+		caCertPath := filepath.Join(configDir, "ca.pem")
+		caKeyPath := filepath.Join(configDir, "ca.key")
+		if err := admin.GenerateIPSecPeerCert(configDir, caCertPath, caKeyPath, node, bindIP); err != nil {
+			fmt.Fprintf(os.Stderr, "Error generating IPsec peer certificate: %v\n", err)
+			os.Exit(1)
+		}
+		fmt.Println("🔐 IPsec peer certificate generated (intra-AZ Geneve encryption ON)")
+	}
+
 	// Build cluster topology from formation data
 	clusterRoutes := formation.BuildClusterRoutes(statusResp.Nodes)
 	predastoreNodes := formation.BuildPredastoreNodes(statusResp.Nodes)
@@ -2089,10 +2318,11 @@ func finalizeNodeSetup(dataDir, certPath, adminAccessKey, adminSecretKey, region
 // applyNetworkConfig copies cluster-wide network settings from a formation
 // NetworkConfig into ConfigSettings and auto-detects the local WAN interface.
 func applyNetworkConfig(settings *admin.ConfigSettings, nc *formation.NetworkConfig) {
+	settings.IPSecEnabled = nc.IPSecEnabled
 	settings.ExternalMode = nc.ExternalMode
-	settings.ExternalDHCP = nc.ExternalDHCP
 	settings.PoolName = nc.PoolName
 	settings.PoolSource = nc.PoolSource
+	settings.PoolBindBridge = nc.PoolBindBridge
 	settings.PoolStart = nc.PoolStart
 	settings.PoolEnd = nc.PoolEnd
 	settings.PoolGateway = nc.PoolGateway
@@ -2111,7 +2341,6 @@ func applyNetworkConfig(settings *admin.ConfigSettings, nc *formation.NetworkCon
 		detected, err := admin.DetectNetwork()
 		if err == nil && detected.WAN != nil {
 			settings.ExternalIface = detected.WAN.Name
-			settings.DhcpBindBridge = detectedDhcpBindBridge(detected)
 		}
 	}
 }
@@ -2220,29 +2449,6 @@ func writeBootstrapFilesWithAdmin(configDir, bootstrapDir string, masterKey []by
 	return handlers_iam.SaveBootstrapData(filepath.Join(bootstrapDir, "bootstrap.json"), bd)
 }
 
-// detectedDhcpBindBridge returns the bridge name where the vpcd DHCP client
-// should bind its AF_PACKET socket — i.e. the interface that physically sees
-// LAN DHCP traffic.
-//
-//   - Default route on a bridge (Linux or OVS, `br-*` prefix): return the
-//     bridge name verbatim. In veth mode this is the Linux bridge holding the
-//     WAN NIC; in direct mode this is the OVS bridge holding the WAN NIC.
-//   - Default route on a bare physical NIC: return "br-wan" as the name the
-//     installer will create.
-//
-// Never returns "br-ext". br-ext is the OVN-side bridge owned by
-// setup-ovn.sh's ovn-bridge-mappings and never sees LAN DHCP frames.
-func detectedDhcpBindBridge(detected *admin.DetectedNetwork) string {
-	if detected == nil || detected.WAN == nil {
-		return ""
-	}
-	name := detected.WAN.Name
-	if strings.HasPrefix(name, "br-") {
-		return name
-	}
-	return "br-wan"
-}
-
 // detectDNSServers auto-detects DNS servers from the host for the specified
 // interface. Uses resolvectl (systemd-resolved) first, then falls back to
 // /etc/resolv.conf. Returns up to 3 servers. Falls back to public DNS if none found.

cmd/spinifex/cmd/admin_gpu.go

@@ -52,16 +52,62 @@ Must be run as root.`,
 	Run: runAdminGpuSetup,
 }
 
+var adminGpuMigCmd = &cobra.Command{
+	Use:   "mig",
+	Short: "Manage NVIDIA MIG partitioning on this node",
+}
+
+var adminGpuMigStatusCmd = &cobra.Command{
+	Use:   "status",
+	Short: "Show MIG capability, mode, and slice state for all GPUs on this node",
+	Run:   runAdminGpuMigStatus,
+}
+
+var adminGpuMigEnableCmd = &cobra.Command{
+	Use:   "enable",
+	Short: "Enable MIG mode on all MIG-capable GPUs (or a specific one)",
+	Long: `Enable NVIDIA MIG mode on all MIG-capable GPUs (or a specific one).
+
+Without --profile the daemon handles slice creation dynamically: each
+RunInstances call for a mig.* instance type carves the right profile on
+demand and destroys the slices when the last instance stops.
+
+With --profile, slices are pre-carved immediately. The daemon recovers
+these slices on restart and serves them as a static pool for that profile.
+
+Must be run as root.`,
+	Run: runAdminGpuMigEnable,
+}
+
+var adminGpuMigDisableCmd = &cobra.Command{
+	Use:   "disable",
+	Short: "Destroy all MIG instances and disable MIG mode",
+	Long: `Destroy all GPU instances and disable NVIDIA MIG mode on all MIG-capable
+GPUs (or a specific one). Blocked while GPU instances are running.
+
+Must be run as root.`,
+	Run: runAdminGpuMigDisable,
+}
+
 func init() {
 	adminCmd.AddCommand(adminGpuCmd)
 	adminGpuCmd.AddCommand(adminGpuStatusCmd)
 	adminGpuCmd.AddCommand(adminGpuEnableCmd)
 	adminGpuCmd.AddCommand(adminGpuDisableCmd)
 	adminGpuCmd.AddCommand(adminGpuSetupCmd)
+	adminGpuCmd.AddCommand(adminGpuMigCmd)
+	adminGpuMigCmd.AddCommand(adminGpuMigStatusCmd)
+	adminGpuMigCmd.AddCommand(adminGpuMigEnableCmd)
+	adminGpuMigCmd.AddCommand(adminGpuMigDisableCmd)
 
 	adminGpuStatusCmd.Flags().String("node", "", "Target node name (default: local node)")
-	// enable, disable, and setup write to the local spinifex.toml and signal the
-	// local daemon — they must be run directly on the target host.
+
+	adminGpuMigEnableCmd.Flags().String("profile", "", "MIG profile to pre-carve (e.g. \"1g.10gb\"); omit for fully dynamic slice management")
+	adminGpuMigEnableCmd.Flags().String("gpu", "", "PCI address of GPU to configure (default: all MIG-capable GPUs)")
+
+	adminGpuMigDisableCmd.Flags().String("gpu", "", "PCI address of GPU to configure (default: all MIG-capable GPUs)")
+	// enable, disable, setup, and mig subcommands write to the local spinifex.toml
+	// and signal the local daemon — they must be run directly on the target host.
 }
 
 // gpuNodeStatus queries NATS and returns the NodeStatusResponse for the target node.
@@ -417,6 +463,277 @@ func runAdminGpuSetup(_ *cobra.Command, _ []string) {
 	gpuToggle(nil, true)
 }
 
+func runAdminGpuMigStatus(_ *cobra.Command, _ []string) {
+	devices, err := gpu.Discover()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "GPU discovery failed: %v\n", err)
+		os.Exit(1)
+	}
+	if len(devices) == 0 {
+		fmt.Println("No GPUs detected on this node.")
+		return
+	}
+
+	anyMIG := false
+	for _, d := range devices {
+		if d.MIGCapable {
+			anyMIG = true
+		}
+		migMode := "N/A"
+		if d.MIGCapable {
+			if d.MIGEnabled {
+				migMode = "enabled"
+			} else {
+				migMode = "disabled"
+			}
+		}
+		fmt.Printf("GPU:           %s\n", d.PCIAddress)
+		fmt.Printf("  Model:       %s\n", d.Model)
+		fmt.Printf("  MIG capable: %v\n", d.MIGCapable)
+		fmt.Printf("  MIG mode:    %s\n", migMode)
+
+		if d.MIGEnabled {
+			instances, listErr := gpu.ListInstances(d.PCIAddress)
+			if listErr != nil {
+				fmt.Printf("  Slices:      (error: %v)\n", listErr)
+			} else {
+				fmt.Printf("  Slices:      %d\n", len(instances))
+				for _, inst := range instances {
+					fmt.Printf("    GI %d: %-12s  mdev: %s\n",
+						inst.GIID, inst.Profile.Name, inst.MdevPath)
+				}
+			}
+		}
+		fmt.Println()
+	}
+
+	if !anyMIG {
+		fmt.Println("No MIG-capable GPUs found. MIG requires an NVIDIA A100, A30, or H100 GPU.")
+	}
+}
+
+func runAdminGpuMigEnable(cmd *cobra.Command, _ []string) {
+	if os.Getuid() != 0 {
+		fmt.Fprintln(os.Stderr, "Error: spx admin gpu mig enable must be run as root.")
+		os.Exit(1)
+	}
+
+	profileName, _ := cmd.Flags().GetString("profile")
+	targetPCI, _ := cmd.Flags().GetString("gpu")
+
+	cfgPath := viper.GetString("config")
+	if cfgPath == "" {
+		cfgPath = DefaultConfigFile()
+	}
+	cfg, err := config.LoadConfig(cfgPath)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error loading config: %v\n", err)
+		os.Exit(1)
+	}
+
+	resp, err := gpuNodeStatus(cfg.Node)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
+		os.Exit(1)
+	}
+	if resp.AllocGPUs > 0 {
+		fmt.Fprintf(os.Stderr, "Error: %d GPU instance(s) are running. Terminate them first.\n", resp.AllocGPUs)
+		os.Exit(1)
+	}
+
+	devices, err := gpu.Discover()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "GPU discovery failed: %v\n", err)
+		os.Exit(1)
+	}
+
+	targets := migFilterTargets(devices, targetPCI)
+	if len(targets) == 0 {
+		if targetPCI != "" {
+			fmt.Fprintf(os.Stderr, "Error: no MIG-capable GPU found at PCI address %s\n", targetPCI)
+		} else {
+			fmt.Fprintln(os.Stderr, "Error: no MIG-capable GPUs found. MIG requires an NVIDIA A100, A30, or H100 GPU.")
+		}
+		os.Exit(1)
+	}
+
+	for _, d := range targets {
+		fmt.Printf("==> Configuring MIG on %s (%s)\n", d.PCIAddress, d.Model)
+
+		if !d.MIGEnabled {
+			fmt.Println("    Enabling MIG mode...")
+			if err := gpu.EnableMIGMode(d.PCIAddress); err != nil {
+				fmt.Fprintf(os.Stderr, "    Error: %v\n", err)
+				os.Exit(1)
+			}
+		} else {
+			fmt.Println("    MIG mode already enabled")
+		}
+
+		if profileName == "" {
+			// Dynamic mode: the daemon carves slices on demand at RunInstances time.
+			// Destroy any pre-carved slices so the daemon starts with a free GPU.
+			existing, _ := gpu.ListInstances(d.PCIAddress)
+			if len(existing) > 0 {
+				fmt.Printf("    Destroying %d pre-carved instance(s) so daemon can manage slices dynamically...\n", len(existing))
+				if err := gpu.DestroyAllInstances(d.PCIAddress); err != nil {
+					fmt.Fprintf(os.Stderr, "    Error: %v\n", err)
+					os.Exit(1)
+				}
+			}
+			fmt.Println("    Dynamic slice management enabled — use 'mig.*' instance types at RunInstances time.")
+			continue
+		}
+
+		// Static pre-carve: create a fixed set of slices for the given profile.
+		profiles, err := gpu.ListProfiles(d.PCIAddress)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "    Error listing profiles: %v\n", err)
+			os.Exit(1)
+		}
+
+		var selected *gpu.MIGProfile
+		for i, p := range profiles {
+			if p.Name == profileName {
+				selected = &profiles[i]
+				break
+			}
+		}
+		if selected == nil {
+			names := make([]string, len(profiles))
+			for i, p := range profiles {
+				names[i] = p.Name
+			}
+			fmt.Fprintf(os.Stderr, "    Error: profile %q not found. Available: %s\n",
+				profileName, strings.Join(names, ", "))
+			os.Exit(1)
+		}
+
+		existing, _ := gpu.ListInstances(d.PCIAddress)
+		if len(existing) > 0 {
+			fmt.Printf("    Destroying %d existing instance(s)...\n", len(existing))
+			if err := gpu.DestroyAllInstances(d.PCIAddress); err != nil {
+				fmt.Fprintf(os.Stderr, "    Error: %v\n", err)
+				os.Exit(1)
+			}
+		}
+
+		fmt.Printf("    Creating MIG instances (profile: %s)...\n", selected.Name)
+		instances, err := gpu.CreateInstances(d.PCIAddress, *selected)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "    Error: %v\n", err)
+			os.Exit(1)
+		}
+		fmt.Printf("    Created %d instance(s)\n", len(instances))
+		for _, inst := range instances {
+			fmt.Printf("      GI %d: %s  mdev: %s\n", inst.GIID, inst.Profile.Name, inst.MdevPath)
+		}
+	}
+
+	migReloadDaemon()
+	fmt.Println("MIG enabled. Run 'spx admin gpu mig status' to verify.")
+}
+
+func runAdminGpuMigDisable(cmd *cobra.Command, _ []string) {
+	if os.Getuid() != 0 {
+		fmt.Fprintln(os.Stderr, "Error: spx admin gpu mig disable must be run as root.")
+		os.Exit(1)
+	}
+
+	targetPCI, _ := cmd.Flags().GetString("gpu")
+
+	cfgPath := viper.GetString("config")
+	if cfgPath == "" {
+		cfgPath = DefaultConfigFile()
+	}
+	cfg, err := config.LoadConfig(cfgPath)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error loading config: %v\n", err)
+		os.Exit(1)
+	}
+
+	resp, err := gpuNodeStatus(cfg.Node)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
+		os.Exit(1)
+	}
+	if resp.AllocGPUs > 0 {
+		fmt.Fprintf(os.Stderr, "Error: %d GPU instance(s) are running. Terminate them first.\n", resp.AllocGPUs)
+		os.Exit(1)
+	}
+
+	devices, err := gpu.Discover()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "GPU discovery failed: %v\n", err)
+		os.Exit(1)
+	}
+
+	targets := migFilterTargets(devices, targetPCI)
+	if len(targets) == 0 {
+		if targetPCI != "" {
+			fmt.Fprintf(os.Stderr, "Error: no MIG-capable GPU found at PCI address %s\n", targetPCI)
+		} else {
+			fmt.Fprintln(os.Stderr, "Error: no MIG-capable GPUs found on this node.")
+		}
+		os.Exit(1)
+	}
+
+	for _, d := range targets {
+		fmt.Printf("==> Disabling MIG on %s (%s)\n", d.PCIAddress, d.Model)
+		if !d.MIGEnabled {
+			fmt.Println("    MIG mode not active — skipping")
+			continue
+		}
+		fmt.Println("    Destroying all GPU instances...")
+		if err := gpu.DestroyAllInstances(d.PCIAddress); err != nil {
+			fmt.Fprintf(os.Stderr, "    Error: %v\n", err)
+			os.Exit(1)
+		}
+		fmt.Println("    Disabling MIG mode...")
+		if err := gpu.DisableMIGMode(d.PCIAddress); err != nil {
+			fmt.Fprintf(os.Stderr, "    Error: %v\n", err)
+			os.Exit(1)
+		}
+	}
+
+	fmt.Println("==> Clearing mig_profile from config...")
+	if err := admin.SetMIGProfile(cfgPath, cfg.Node, ""); err != nil {
+		fmt.Fprintf(os.Stderr, "Error writing config: %v\n", err)
+		os.Exit(1)
+	}
+
+	migReloadDaemon()
+	fmt.Println("MIG disabled. GPUs will operate in whole-GPU passthrough mode.")
+}
+
+// migFilterTargets returns all MIG-capable GPUs, optionally filtered to a
+// specific PCI address.
+func migFilterTargets(devices []gpu.GPUDevice, targetPCI string) []gpu.GPUDevice {
+	var out []gpu.GPUDevice
+	for _, d := range devices {
+		if !d.MIGCapable {
+			continue
+		}
+		if targetPCI != "" && d.PCIAddress != targetPCI {
+			continue
+		}
+		out = append(out, d)
+	}
+	return out
+}
+
+// migReloadDaemon sends SIGHUP to spinifex-daemon so it picks up the updated
+// MIG configuration without a full restart.
+func migReloadDaemon() {
+	out, err := exec.Command("systemctl", "kill", "-s", "HUP", "spinifex-daemon").CombinedOutput()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Warning: failed to signal daemon: %v\n%s\n", err, out)
+		fmt.Fprintln(os.Stderr, "Restart the daemon manually: sudo systemctl restart spinifex-daemon")
+		return
+	}
+	fmt.Println("==> Daemon signalled (SIGHUP) — MIG pool will be rebuilt on next request.")
+}
+
 // gpuSetupCollectVFIOIDs collects vendor:device PCI ID pairs for each GPU and
 // all sibling PCI functions on the same device slot (e.g. GPU + HDMI audio).
 // All siblings must bind to vfio-pci together for IOMMU group isolation.

cmd/spinifex/cmd/admin_test.go

@@ -142,49 +142,6 @@ func TestSpinifexTomlTemplate_AdvertiseOmittedWhenEmpty(t *testing.T) {
 	assert.NotContains(t, string(data), "advertise =")
 }
 
-// detectedDhcpBindBridge must return the default-route interface name verbatim
-// when it's a bridge (Linux or OVS, br-* prefix). The old detectedWanBridge()
-// returned hardcoded "br-ext" for Linux bridges — that value broke DHCP on
-// consumer-router LANs because br-ext never sees LAN DHCP traffic (mulga-998).
-func TestDetectedDhcpBindBridge(t *testing.T) {
-	tests := []struct {
-		name     string
-		detected *admin.DetectedNetwork
-		want     string
-	}{
-		{
-			name:     "Linux bridge default route returns bridge name (not 'br-ext')",
-			detected: &admin.DetectedNetwork{WAN: &admin.DetectedInterface{Name: "br-wan"}},
-			want:     "br-wan",
-		},
-		{
-			name:     "OVS bridge default route returned verbatim",
-			detected: &admin.DetectedNetwork{WAN: &admin.DetectedInterface{Name: "br-ext"}},
-			want:     "br-ext",
-		},
-		{
-			name:     "physical NIC defaults to 'br-wan' (bridge the installer creates)",
-			detected: &admin.DetectedNetwork{WAN: &admin.DetectedInterface{Name: "enp0s3"}},
-			want:     "br-wan",
-		},
-		{
-			name:     "nil DetectedNetwork",
-			detected: nil,
-			want:     "",
-		},
-		{
-			name:     "DetectedNetwork with nil WAN",
-			detected: &admin.DetectedNetwork{},
-			want:     "",
-		},
-	}
-	for _, tt := range tests {
-		t.Run(tt.name, func(t *testing.T) {
-			assert.Equal(t, tt.want, detectedDhcpBindBridge(tt.detected))
-		})
-	}
-}
-
 // Legacy `wan_bridge` TOML key must fail-start vpcd with guidance, not silently
 // alias (per mulga-998 D3). Prevents the footgun where operators inherited the
 // old key pointing at 'br-ext' and got broken DHCP on veth-mode hosts.
@@ -206,8 +163,8 @@ wan_bridge = "br-ext"
 
 	err := checkLegacyWanBridgeKey("node1", cfgPath)
 	require.Error(t, err)
-	assert.Contains(t, err.Error(), "dhcp_bind_bridge")
 	assert.Contains(t, err.Error(), "wan_bridge")
+	assert.Contains(t, err.Error(), "Remove")
 }
 
 func TestCheckLegacyWanBridgeKey_EnvVarRejected(t *testing.T) {
@@ -218,7 +175,7 @@ func TestCheckLegacyWanBridgeKey_EnvVarRejected(t *testing.T) {
 	err := checkLegacyWanBridgeKey("node1", "/tmp/unused.toml")
 	require.Error(t, err)
 	assert.Contains(t, err.Error(), "SPINIFEX_VPCD_WAN_BRIDGE")
-	assert.Contains(t, err.Error(), "dhcp_bind_bridge")
+	assert.Contains(t, err.Error(), "Remove")
 }
 
 func TestCheckLegacyWanBridgeKey_CleanConfigPasses(t *testing.T) {
@@ -228,7 +185,7 @@ func TestCheckLegacyWanBridgeKey_CleanConfigPasses(t *testing.T) {
 node = "node1"
 [nodes.node1.vpcd]
 ovn_nb_addr = "tcp:127.0.0.1:6641"
-dhcp_bind_bridge = "br-wan"
+external_interface = "enp0s3"
 `), 0o644))
 
 	viper.Reset()
@@ -310,3 +267,122 @@ func TestWritePredastoreEncryptionKey_CreatesMissingDir(t *testing.T) {
 	require.NoError(t, err)
 	assert.True(t, dirInfo.IsDir())
 }
+
+// TestImagesRemoveCmd_FlagSchema guards the public flag surface for
+// `spx admin images remove`. --image-id is required so cobra rejects a bare
+// invocation; --force/--yes default to false.
+func TestImagesRemoveCmd_FlagSchema(t *testing.T) {
+	imageIDFlag := imagesRemoveCmd.Flags().Lookup("image-id")
+	require.NotNil(t, imageIDFlag, "--image-id must be defined")
+	assert.Equal(t, []string{"true"}, imageIDFlag.Annotations[cobraRequiredAnnotation],
+		"--image-id must be marked required")
+
+	forceFlag := imagesRemoveCmd.Flags().Lookup("force")
+	require.NotNil(t, forceFlag)
+	assert.Equal(t, "false", forceFlag.DefValue)
+
+	yesFlag := imagesRemoveCmd.Flags().Lookup("yes")
+	require.NotNil(t, yesFlag)
+	assert.Equal(t, "false", yesFlag.DefValue)
+}
+
+// cobraRequiredAnnotation is the annotation key cobra uses to mark required flags.
+const cobraRequiredAnnotation = "cobra_annotation_bash_completion_one_required_flag"
+
+// `spx admin init` must expose --external-source and --external-bind-bridge so
+// operators can pick the DHCP path for [[network.external_pools]].
+func TestAdminInitCmd_ExternalDHCPFlagSchema(t *testing.T) {
+	sourceFlag := adminInitCmd.Flags().Lookup("external-source")
+	require.NotNil(t, sourceFlag, "--external-source must be defined")
+	assert.Equal(t, "", sourceFlag.DefValue)
+
+	bindFlag := adminInitCmd.Flags().Lookup("external-bind-bridge")
+	require.NotNil(t, bindFlag, "--external-bind-bridge must be defined")
+	assert.Equal(t, "", bindFlag.DefValue)
+
+	poolFlag := adminInitCmd.Flags().Lookup("external-pool")
+	require.NotNil(t, poolFlag)
+	assert.Equal(t, "", poolFlag.DefValue)
+}
+
+// DHCP-sourced pool must emit source + bind_bridge into spinifex.toml so the
+// daemon's validator and vpcd's DHCPManager find the bridge they DORA on.
+func TestSpinifexTomlTemplate_ExternalPoolDHCPSource(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "spinifex.toml")
+	settings := admin.ConfigSettings{
+		Node:           "node1",
+		Az:             "ap-southeast-2a",
+		Port:           "4432",
+		Region:         "ap-southeast-2",
+		BindIP:         "10.11.12.1",
+		ConfigDir:      dir,
+		OVNNBAddr:      "tcp:127.0.0.1:6641",
+		OVNSBAddr:      "tcp:127.0.0.1:6642",
+		ExternalMode:   "pool",
+		ExternalIface:  "eth0",
+		PoolName:       "wan",
+		PoolSource:     "dhcp",
+		PoolBindBridge: "br-wan",
+		PoolPrefixLen:  24,
+	}
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	content := string(data)
+	assert.Contains(t, content, `source      = "dhcp"`)
+	assert.Contains(t, content, `bind_bridge = "br-wan"`)
+	assert.NotContains(t, content, "range_start", "dhcp pool must not emit range_start")
+	assert.NotContains(t, content, "range_end", "dhcp pool must not emit range_end")
+	assert.NotContains(t, content, "gateway     =", "gateway omitted for dhcp pool (discovered from OFFER)")
+}
+
+// Static-sourced pool must not emit bind_bridge — the validator rejects it.
+func TestSpinifexTomlTemplate_ExternalPoolStaticSource(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "spinifex.toml")
+	settings := admin.ConfigSettings{
+		Node:          "node1",
+		Az:            "ap-southeast-2a",
+		Port:          "4432",
+		Region:        "ap-southeast-2",
+		BindIP:        "10.11.12.1",
+		ConfigDir:     dir,
+		OVNNBAddr:     "tcp:127.0.0.1:6641",
+		OVNSBAddr:     "tcp:127.0.0.1:6642",
+		ExternalMode:  "pool",
+		ExternalIface: "eth0",
+		PoolName:      "wan",
+		PoolSource:    "static",
+		PoolStart:     "192.168.1.150",
+		PoolEnd:       "192.168.1.250",
+		PoolGateway:   "192.168.1.1",
+		PoolPrefixLen: 24,
+	}
+	require.NoError(t, admin.GenerateConfigFile(path, spinifexTomlTemplate, settings))
+
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	content := string(data)
+	assert.Contains(t, content, `source      = "static"`)
+	assert.NotContains(t, content, "bind_bridge", "static pool must not emit bind_bridge")
+	assert.Contains(t, content, `range_start = "192.168.1.150"`)
+	assert.Contains(t, content, `gateway     = "192.168.1.1"`)
+}
+
+// Formation joiners must receive PoolBindBridge from the init node so the
+// cluster-wide config reaches every chassis intact.
+func TestApplyNetworkConfig_PropagatesPoolBindBridge(t *testing.T) {
+	settings := &admin.ConfigSettings{}
+	nc := &formation.NetworkConfig{
+		ExternalMode:   "pool",
+		PoolName:       "wan",
+		PoolSource:     "dhcp",
+		PoolBindBridge: "br-wan",
+		PoolPrefixLen:  24,
+	}
+	applyNetworkConfig(settings, nc)
+	assert.Equal(t, "dhcp", settings.PoolSource)
+	assert.Equal(t, "br-wan", settings.PoolBindBridge)
+}

cmd/spinifex/cmd/cluster.go

@@ -98,7 +98,7 @@ func runClusterShutdown(cmd *cobra.Command, args []string) {
 			}
 			nc.Flush()
 			// Wait briefly for messages to propagate
-			time.Sleep(2 * time.Second)
+			time.Sleep(200 * time.Millisecond)
 			fmt.Printf("[INFRA] Complete\n")
 			break
 		}

cmd/spinifex/cmd/service.go

@@ -29,7 +29,7 @@ import (
 	"github.com/mulgadc/spinifex/spinifex/services/predastore"
 	"github.com/mulgadc/spinifex/spinifex/services/spinifexui"
 	"github.com/mulgadc/spinifex/spinifex/services/viperblockd"
-	"github.com/mulgadc/spinifex/spinifex/services/vpcd"
+	"github.com/mulgadc/spinifex/spinifex/vpcd"
 	"github.com/spf13/cobra"
 	"github.com/spf13/viper"
 )
@@ -283,19 +283,25 @@ var viperblockStartCmd = &cobra.Command{
 			shardWAL = *nodeConfig.Viperblock.ShardWAL
 		}
 
+		encryptionKeyFile := nodeConfig.Viperblock.EncryptionKeyFile
+		if envKey := viper.GetString("viperblock-encryption-key-file"); envKey != "" {
+			encryptionKeyFile = envKey
+		}
+
 		service, err := service.New("viperblock", &viperblockd.Config{
-			NatsHost:   nodeConfig.NATS.Host,
-			NatsToken:  nodeConfig.NATS.ACL.Token,
-			NatsCACert: nodeConfig.NATS.CACert,
-			PluginPath: pluginPath,
-			S3Host:     nodeConfig.Predastore.Host,
-			Bucket:     nodeConfig.Predastore.Bucket,
-			Region:     nodeConfig.Predastore.Region,
-			AccessKey:  nodeConfig.Predastore.AccessKey,
-			SecretKey:  nodeConfig.Predastore.SecretKey,
-			BaseDir:    nodeConfig.Predastore.BaseDir,
-			NodeName:   clusterConfig.Node,
-			ShardWAL:   shardWAL,
+			NatsHost:          nodeConfig.NATS.Host,
+			NatsToken:         nodeConfig.NATS.ACL.Token,
+			NatsCACert:        nodeConfig.NATS.CACert,
+			PluginPath:        pluginPath,
+			S3Host:            nodeConfig.Predastore.Host,
+			Bucket:            nodeConfig.Predastore.Bucket,
+			Region:            nodeConfig.Predastore.Region,
+			AccessKey:         nodeConfig.Predastore.AccessKey,
+			SecretKey:         nodeConfig.Predastore.SecretKey,
+			BaseDir:           nodeConfig.Predastore.BaseDir,
+			NodeName:          clusterConfig.Node,
+			ShardWAL:          shardWAL,
+			EncryptionKeyFile: encryptionKeyFile,
 		})
 
 		if err != nil {
@@ -678,11 +684,8 @@ var spinifexUIStatusCmd = &cobra.Command{
 }
 
 // checkLegacyWanBridgeKey fails vpcd startup if the deprecated `wan_bridge`
-// TOML key or `SPINIFEX_VPCD_WAN_BRIDGE` env-var is present. The key was
-// renamed to `dhcp_bind_bridge` (see mulga-998.a) because the old name misled
-// operators into pointing at the OVN-side bridge ("br-ext"), which never sees
-// LAN DHCP traffic. Per D3: no silent alias, no auto-rewrite — operator must
-// rename the key before vpcd will start.
+// TOML key or `SPINIFEX_VPCD_WAN_BRIDGE` env-var is present. Per D3: no silent
+// alias, no auto-rewrite — operator must remove the key before vpcd will start.
 func checkLegacyWanBridgeKey(node, cfgFile string) error {
 	legacyInTOML := viper.IsSet("nodes." + node + ".vpcd.wan_bridge")
 	legacyInEnv := os.Getenv("SPINIFEX_VPCD_WAN_BRIDGE") != ""
@@ -695,9 +698,7 @@ func checkLegacyWanBridgeKey(node, cfgFile string) error {
 	}
 	return fmt.Errorf(
 		"vpcd: deprecated 'wan_bridge' key found in %s. "+
-			"Rename 'wan_bridge' to 'dhcp_bind_bridge'. The value may also be wrong — verify it is "+
-			"the Linux bridge holding your WAN NIC (veth mode) or the OVS bridge holding your WAN NIC (direct mode); "+
-			"never 'br-ext'. Typical value on a consumer-router LAN: 'br-wan'. "+
+			"Remove the key entirely; vpcd auto-detects the WAN bridge (br-wan). "+
 			"Then: sudo systemctl restart spinifex-vpcd",
 		source,
 	)
@@ -734,6 +735,7 @@ var vpcdStartCmd = &cobra.Command{
 			extPools = append(extPools, vpcd.ExternalPoolConfig{
 				Name:            p.Name,
 				Source:          p.Source,
+				BindBridge:      p.BindBridge,
 				RangeStart:      p.RangeStart,
 				RangeEnd:        p.RangeEnd,
 				Gateway:         p.Gateway,
@@ -742,7 +744,6 @@ var vpcdStartCmd = &cobra.Command{
 				DNSServers:      p.DNSServers,
 				Region:          p.Region,
 				AZ:              p.AZ,
-				DhcpBindBridge:  nodeConfig.VPCD.DhcpBindBridge,
 				GwLrpRangeStart: p.GwLrpRangeStart,
 				GwLrpRangeEnd:   p.GwLrpRangeEnd,
 			})
@@ -778,8 +779,8 @@ var vpcdStartCmd = &cobra.Command{
 			ExternalPools:     extPools,
 			Bootstrap:         bootstrap,
 			ExternalInterface: nodeConfig.VPCD.ExternalInterface,
-			DhcpBindBridge:    nodeConfig.VPCD.DhcpBindBridge,
 			BridgeMode:        nodeConfig.VPCD.BridgeMode,
+			AZ:                nodeConfig.AZ,
 		})
 		if err != nil {
 			fmt.Println("Error starting vpcd service:", err)
@@ -916,6 +917,14 @@ func init() {
 	viper.BindEnv("plugin-path", "SPINIFEX_VIPERBLOCK_PLUGIN_PATH")
 	viper.BindPFlag("plugin-path", predastoreCmd.PersistentFlags().Lookup("plugin-path"))
 
+	// Viperblock at-rest encryption master key (shared with other on-node
+	// services via group ownership; mode 0640 or stricter). Distinct viper
+	// key from predastore's encryption-key-file so the two BindPFlag calls
+	// don't collide globally.
+	viperblockCmd.PersistentFlags().String("encryption-key-file", "", "Path to the 32-byte AES-256 master key file for at-rest encryption (empty disables encryption)")
+	viper.BindEnv("viperblock-encryption-key-file", "SPINIFEX_VIPERBLOCK_ENCRYPTION_KEY_FILE")
+	viper.BindPFlag("viperblock-encryption-key-file", viperblockCmd.PersistentFlags().Lookup("encryption-key-file"))
+
 	viperblockCmd.AddCommand(viperblockStartCmd)
 	viperblockCmd.AddCommand(viperblockStopCmd)
 	viperblockCmd.AddCommand(viperblockStatusCmd)

cmd/spinifex/cmd/templates/predastore.toml

@@ -51,19 +51,19 @@ secret_access_key = "{{.SecretKey}}"
 # Using distributed hash ring for object > shard placement.
 [[nodes]]
 id = 1
-host = "0.0.0.0"
+host = "127.0.0.1"
 port = 9991
 path = "distributed/nodes/node-1/"
 
 [[nodes]]
 id = 2
-host = "0.0.0.0"
+host = "127.0.0.1"
 port = 9992
 path = "distributed/nodes/node-2/"
 
 [[nodes]]
 id = 3
-host = "0.0.0.0"
+host = "127.0.0.1"
 port = 9993
 path = "distributed/nodes/node-3/"
 

cmd/spinifex/cmd/templates/spinifex.toml

@@ -1,5 +1,5 @@
 # Domain configuration file in TOML format.
-version = "2"
+version = "3"
 epoch = 1
 node = "{{.Node}}"
 {{- if .OperatorEmail}}
@@ -7,24 +7,27 @@ node = "{{.Node}}"
 [operator]
 email = "{{.OperatorEmail}}"
 {{- end}}
-{{- if .ExternalMode}}
 
 [network]
+ipsec_enabled = {{.IPSecEnabled}}
+{{- if .ExternalMode}}
 external_mode = "{{.ExternalMode}}"
-{{- if .ExternalDHCP}}
-external_dhcp = true
-{{- end}}
 
 [[network.external_pools]]
 name        = "{{.PoolName}}"
 {{- if .PoolSource}}
 source      = "{{.PoolSource}}"
 {{- end}}
+{{- if .PoolBindBridge}}
+bind_bridge = "{{.PoolBindBridge}}"
+{{- end}}
 {{- if .PoolStart}}
 range_start = "{{.PoolStart}}"
 range_end   = "{{.PoolEnd}}"
 {{- end}}
+{{- if .PoolGateway}}
 gateway     = "{{.PoolGateway}}"
+{{- end}}
 {{- if .PoolGatewayIP}}
 gateway_ip  = "{{.PoolGatewayIP}}"
 {{- end}}
@@ -97,15 +100,6 @@ ovn_sb_addr = "{{.OVNSBAddr}}"
 {{- if .ExternalIface}}
 external_interface = "{{.ExternalIface}}"
 {{- end}}
-{{- if .DhcpBindBridge}}
-# dhcp_bind_bridge: the bridge where vpcd's DHCP client binds its AF_PACKET
-# socket. On Linux-bridge-WAN hosts (netplan default with the WAN NIC enslaved),
-# this is the Linux bridge name (e.g. "br-wan"). On direct-OVS hosts, this is
-# the OVS bridge holding the WAN NIC.
-# NEVER set this to "br-ext" — that is the OVN-side bridge and never sees LAN
-# DHCP traffic.
-dhcp_bind_bridge = "{{.DhcpBindBridge}}"
-{{- end}}
 
 [nodes.{{.Node}}.predastore]
 host = "{{.BindIP}}:8443"

docs/DESIGN.md

@@ -16,14 +16,14 @@ Users interact with Spinifex using standard AWS SDKs or the AWS CLI by using the
 
 ```bash
 AWS_PROFILE=spinifex aws ec2 run-instances \
-    --image-id ami-debian12 \
+    --image-id ami-debian13 \
     --instance-type t3.micro \
     --key-name my-keypair
 ```
 
 The AWS SDK formats this as an HTTPS POST with:
 - AWS SigV4 authentication headers
-- EC2 Query Protocol body (`Action=RunInstances&ImageId=ami-debian12&...`)
+- EC2 Query Protocol body (`Action=RunInstances&ImageId=ami-debian13&...`)
 
 ### 2. AWS Gateway
 
@@ -84,6 +84,13 @@ The EC2 handler (`spinifex/gateway/ec2.go`) parses the `Action` parameter and de
 
 ### 4. NATS Messaging
 
+The full service-to-subject map — which service subscribes to what, what
+each one publishes downstream, what each depends on — is captured in
+[`service-interfaces.yaml`](./service-interfaces.yaml). It is the source
+of truth for targeted e2e suite selection (see
+`docs/development/improvements/e2e-targeted-suite-selection.md`) and is
+validated by `make manifest-check`.
+
 The gateway communicates with daemons via NATS request/response. Most calls go through `utils.NATSRequest`, which marshals the input, attaches the account ID as a NATS header, and unmarshals the typed response:
 
 ```go
@@ -297,7 +304,7 @@ sudo systemctl start spinifex.target
 export AWS_PROFILE=spinifex
 
 # Run instance
-aws ec2 run-instances --image-id ami-debian12 --instance-type t3.micro
+aws ec2 run-instances --image-id ami-debian13 --instance-type t3.micro
 
 # List instances (queries all nodes)
 aws ec2 describe-instances

docs/compute/gpu-passthrough/README.md

@@ -7,6 +7,11 @@ tags:
   - passthrough
   - vfio
   - ec2
+sections:
+  - overview
+  - prerequisites
+  - instructions
+  - troubleshooting
 resources:
   - title: "Spinifex Repository"
     url: "https://github.com/mulgadc/spinifex"
@@ -14,26 +19,6 @@ resources:
     url: "https://docs.aws.amazon.com/ec2/latest/instancetypes/ac.html"
 ---
 
-# GPU Passthrough
-
-> Configure VFIO GPU passthrough on a Spinifex node to expose NVIDIA or AMD GPUs to EC2 instances.
-
-> **Note:** GPU passthrough is supported on x86_64 hosts only.
-
-## Table of Contents
-
-- [Overview](#overview)
-- [Prerequisites](#prerequisites)
-- [Host Setup](#host-setup)
-- [Enable Passthrough](#enable-passthrough)
-- [Status and Monitoring](#status-and-monitoring)
-- [GPU Instance Types](#gpu-instance-types)
-- [Launching GPU Instances](#launching-gpu-instances)
-- [Disable Passthrough](#disable-passthrough)
-- [Troubleshooting](#troubleshooting)
-
----
-
 ## Overview
 
 Spinifex supports VFIO-based GPU passthrough, binding NVIDIA and AMD GPUs directly to guest VMs via the `vfio-pci` kernel driver. Once configured, the node exposes GPU-enabled EC2 instance types and the GPU is allocated exclusively to individual instances.
@@ -42,9 +27,9 @@ At startup, Spinifex always probes for GPU hardware and surfaces the result in t
 
 The passthrough state is reflected in the login banner:
 
-![GPU not yet enabled](../../../.github/assets/images/gpu-banner1.png)
+<img src="../../../.github/assets/images/gpu-banner1.png" alt="GPU not yet enabled">
 
----
+> **Note:** GPU passthrough is supported on x86_64 hosts only.
 
 ## Prerequisites
 
@@ -52,9 +37,9 @@ The passthrough state is reflected in the login banner:
 - An NVIDIA or AMD GPU must be physically installed.
 - The Spinifex node must be running on bare metal (not inside a VM).
 
----
+## Instructions
 
-## Host Setup
+### Host Setup
 
 Run the one-time host configuration as root. This command is idempotent — safe to re-run after a reboot.
 
@@ -79,9 +64,7 @@ Setup complete — reboot required.
   Then run: sudo spx admin gpu enable
 ```
 
----
-
-## Enable Passthrough
+### Enable Passthrough
 
 After setup and reboot, enable GPU passthrough:
 
@@ -91,14 +74,11 @@ sudo spx admin gpu enable
 
 The command writes the configuration, signals the daemon (`SIGHUP`), and waits up to 30 seconds for the daemon to confirm the new state. On success it prints the current GPU status.
 
+<img src="../../../.github/assets/images/gpu-enabled.png" alt="GPU enabled">
 
-![GPU enabled](../../../.github/assets/images/gpu-enabled.png)
+### Status and Monitoring
 
----
-
-## Status and Monitoring
-
-### Per-node status
+**Per-node status:**
 
 ```bash
 spx admin gpu status
@@ -108,7 +88,7 @@ spx admin gpu status --node <node-name>
 
 Output includes hardware detected, IOMMU state, vfio-pci binding, passthrough enabled/disabled, GPU pool allocation (`allocated/total`), and the GPU-capable EC2 instance types available on that node.
 
-### Cluster view
+**Cluster view:**
 
 `spx top nodes` includes a `GPU (used/total)` column:
 
@@ -122,30 +102,30 @@ Output includes hardware detected, IOMMU state, vfio-pci binding, passthrough en
 spx top nodes
 ```
 
----
+### GPU Instance Types
 
-## GPU Instance Types
-
-GPU instance types are derived from detected hardware.
-
-To list available GPU instance types on the current node:
+GPU instance types are derived from detected hardware. To list available GPU instance types on the current node:
 
 ```bash
 aws ec2 describe-instance-types \
   --filters Name=instance-type,Values=g*
 ```
 
----
+### Launching GPU Instances
 
-## Launching GPU Instances
-
-Import the GPU-enabled AMI before launching:
+Spinifex ships two pre-built GPU guest images. Import the one that matches your host hardware before launching:
 
 ```bash
-spx admin images import --name ubuntu-gpu-nvidia   # NVIDIA hosts
-spx admin images import --name ubuntu-gpu-amd      # AMD hosts
+spx admin images import --name ubuntu-26.04-nvidia-gpu-x86_64   # NVIDIA hosts
+spx admin images import --name ubuntu-26.04-amd-gpu-x86_64      # AMD hosts
 ```
 
+These images are built on Ubuntu 26.04 and serve as a base for GPU workloads. Both include Docker CE, Python 3, and common utilities (`git`, `curl`, `htop`, `tmux`, `ffmpeg`).
+
+**NVIDIA** (`ubuntu-26.04-nvidia-gpu-x86_64`): NVIDIA server driver with DKMS pre-built against the image kernel, `nvidia-smi`, nvidia-container-toolkit wired into Docker (`--gpus all` works on first boot). CUDA toolkit and cuDNN are not included — use an NGC container or install inside the instance.
+
+**AMD** (`ubuntu-26.04-amd-gpu-x86_64`): `linux-firmware` (amdgpu firmware blobs), ROCm 7.2 CLI tools (`rocm-smi`, `rocminfo`, `amd-smi`). Full ROCm compute libraries (`rocblas`, etc.) are not included — install inside the instance as needed.
+
 Launch a GPU instance:
 
 ```bash
@@ -165,9 +145,7 @@ nvidia-smi
 rocm-smi
 ```
 
----
-
-## Disable Passthrough
+### Disable Passthrough
 
 Passthrough cannot be disabled while GPU instances are running. Terminate all GPU instances first, then:
 
@@ -177,8 +155,6 @@ sudo spx admin gpu disable
 
 The command signals the daemon and waits for confirmation. The GPU is released back to the host kernel on the next reboot (the vfio-pci binding persists until `setup` is re-run without the blacklists).
 
----
-
 ## Troubleshooting
 
 ### IOMMU not active after reboot

docs/getting-started/install-airgapped/README.md

@@ -57,6 +57,7 @@ sudo apt update
 sudo apt install --download-only -y \
   nbdkit nbdkit-plugin-dev pkg-config \
   qemu-system-x86 qemu-utils qemu-kvm \
+  ovmf qemu-efi-aarch64 \
   libvirt-daemon-system libvirt-clients libvirt-dev \
   ovn-central ovn-host openvswitch-switch \
   dhcpcd-base make gcc jq curl iproute2 netcat-openbsd \
@@ -74,7 +75,7 @@ Mirror the cloud image you intend to run as guest VMs:
 
 ```bash
 mkdir -p images
-curl -fsSL "https://cloud.debian.org/images/cloud/trixie/20260501-2465/debian-13-genericcloud-amd64-20260501-2465.qcow2" \
+curl -fsSL "https://cloud.debian.org/images/cloud/trixie/20260518-2482/debian-13-genericcloud-amd64-20260518-2482.qcow2" \
   -o images/debian-13-amd64.qcow2
 ```
 
@@ -141,7 +142,7 @@ Register the pre-staged image with Spinifex:
 
 ```bash
 sudo spx admin images import --file /mnt/usb/images/debian-13-amd64.qcow2 \
-  --distro debian --version 13 --arch x86_64
+  --distro debian --version 13 --arch x86_64 --boot-mode uefi
 ```
 
 ## Step 9. Verify
@@ -183,4 +184,4 @@ sudo INSTALL_SPINIFEX_TARBALL=/mnt/usb/tarball/spinifex-...tar.gz \
 
 ### Image Import Fails
 
-`spx admin images import --file` requires the distro/version/arch flags so the image registers in the catalogue. If the import succeeds but `aws ec2 describe-images` returns empty, check `journalctl -u spinifex-daemon -f` for predastore upload errors.
+`spx admin images import --file` requires the distro/version/arch/boot-mode flags so the image registers in the catalogue. If the import succeeds but `aws ec2 describe-images` returns empty, check `journalctl -u spinifex-daemon -f` for predastore upload errors.

docs/getting-started/install-source/README.md

@@ -32,7 +32,7 @@ This guide builds Spinifex from source. For production deployments, the [binary
 **Requirements:**
 
 - Ubuntu 26.04 / Debian 13
-- Go 1.26.3+
+- Go 1.26.4+
 - GCC, make, pkg-config
 - QEMU/KVM, OVN/Open vSwitch, AWS CLI v2, dhcpcd-base
 

docs/getting-started/setting-up-your-cluster/README.md

@@ -58,10 +58,10 @@ spx admin images list
 
 ```
 NAME                    | DISTRO | VERSION | ARCH   | BOOT
-debian-13-arm64         | debian | 13      | arm64  | bios
-debian-13-x86_64        | debian | 13      | x86_64 | bios
-ubuntu-26.04-arm64      | ubuntu | 26.04   | arm64  | bios
-ubuntu-26.04-x86_64     | ubuntu | 26.04   | x86_64 | bios
+debian-13-arm64         | debian | 13      | arm64  | uefi
+debian-13-x86_64        | debian | 13      | x86_64 | uefi
+ubuntu-26.04-arm64      | ubuntu | 26.04   | arm64  | uefi
+ubuntu-26.04-x86_64     | ubuntu | 26.04   | x86_64 | uefi
 ```
 
 Import an image:
@@ -73,7 +73,7 @@ spx admin images import --name ubuntu-26.04-x86_64
 Or import a local image file:
 
 ```bash
-spx admin images import --file ~/images/ubuntu-26.04.img --distro ubuntu --version 26.04 --arch x86_64
+spx admin images import --file ~/images/ubuntu-26.04.img --distro ubuntu --version 26.04 --arch x86_64 --boot-mode uefi
 ```
 
 Verify the import and note the AMI ID:

docs/reference-architectures/SMCI-H14-orchestration/README.md

@@ -0,0 +1,266 @@
+---
+title: "Multi-Tenant AI on Supermicro H14 with AMD MI350X"
+description: "Provision isolated GPU VMs for simultaneous AI workloads on a Supermicro H14 bare-metal node using Spinifex's EC2-compatible API."
+category: "Reference Architectures"
+tags:
+  - supermicro
+  - amd
+  - mi350x
+  - gpu-passthrough
+  - multi-tenant
+  - vllm
+resources:
+  - title: "GPU Passthrough"
+    url: "/docs/gpu-passthrough"
+  - title: "Launching Instances"
+    url: "/docs/launching-instances"
+  - title: "AMD Instinct MI350X"
+    url: "https://www.amd.com/en/products/accelerators/instinct/mi300/mi350x.html"
+  - title: "vLLM Documentation"
+    url: "https://docs.vllm.ai"
+sections:
+  - overview
+  - prerequisites
+  - instructions
+---
+
+## Overview
+
+This document serves as a tutorial for setting up multi-tenant AI workloads with Spinifex. The reference architecture for this tutorial is the Supermicro H14 platform - specifications listed below.
+
+The H14 chassis comes equipped with **8× AMD Instinct MI350X** (288 GB HBM3e each, 2.3 TB total). We used Spinifex to provision three isolated VMs, each assigned 2× MI350X via direct PCIe passthrough, and ran three computationally intensive workloads simultaneously: a YOLO11x vision model, a Qwen3-VL 235B FP8 vision language model, and a Llama 3.3 70B large language model.
+
+<p><video src="https://iso.mulgadc.com/clipped-demo.mp4" controls width="100%" style="border-radius:6px"></video></p>
+
+### Platform
+
+| Component | Specification |
+|---|---|
+| **Bare-metal host** | Supermicro H14 |
+| **Host OS** | Ubuntu 24.04 LTS or Debian 13 (minimum) |
+| **Orchestration** | Spinifex — EC2-compatible bare-metal API |
+| **Guest OS** | Ubuntu 26.04 LTS |
+| **GPUs** | 8× AMD Instinct MI350X (288 GB HBM3e each, 2.3 TB total) |
+| **GPU passthrough** | PCIe passthrough via vfio-pci |
+| **Instance type** | `g7e.12xlarge` — 2× MI350X per VM |
+| **Container runtime** | Docker (with ROCm device access) |
+| **Inference runtime** | vLLM (`rocm/vllm` image) |
+| **VM network** | Internal VPC, 192.168.10.0/24 |
+| **Block storage** | Viperblock — EBS-compatible, local NVMe-backed |
+
+Spinifex runs on the bare-metal host and exposes an EC2-compatible API to any standard `aws ec2` tooling. Guest VMs receive their GPU allocation at the hardware level — each VM's OS sees the MI350Xs as native PCIe devices with no virtualisation layer in the data path. Workloads across VMs share no GPU memory, no address space, and no network segment except through the VPC.
+
+GPU passthrough requires a host kernel and OS that supports vfio-pci. Ubuntu 24.04 LTS (kernel 6.8+) and Debian 13 (kernel 6.12+) are the tested minimum baselines for the host. Guest VMs run Ubuntu 26.04 LTS.
+
+## Prerequisites
+
+- Supermicro H14 with 8× AMD Instinct MI350X installed
+- Host OS: **Ubuntu 24.04 LTS** (kernel 6.8+) or **Debian 13** (kernel 6.12+) — minimum for vfio-pci support
+- Spinifex installed and all services running (`systemctl status spinifex.target`)
+- AMD GPU AMI registered (`ami-ubuntu-amd-gpu`) — Ubuntu 26.04 LTS with ROCm-compatible kernel
+- AWS CLI configured with `AWS_PROFILE=spinifex` pointing at the Spinifex endpoint
+- SSH key pair imported into Spinifex, VPC and security group created (see [Launching Instances](/docs/launching-instances))
+
+GPU passthrough must be configured before launching GPU instances. This is a one-time step per host.
+
+```bash
+# Detect GPUs and bind to vfio-pci (requires reboot)
+sudo spx admin gpu setup
+
+# After rebooting — confirm passthrough is active and signal the daemon
+sudo spx admin gpu enable
+```
+
+`spx admin gpu setup` blacklists the AMD driver on the host and binds each GPU to `vfio-pci`. After reboot, `spx admin gpu enable` verifies the binding and makes the GPU pool available to `RunInstances`.
+
+<img src="../../../.github/assets/images/gpu-enabled.png" alt="Spinifex console confirming PCIe passthrough is active for each GPU">
+
+To check available GPU instance types:
+
+```bash
+export AWS_PROFILE=spinifex
+aws ec2 describe-instance-types \
+    --query 'InstanceTypes[?GpuInfo].[InstanceType,GpuInfo.Gpus[0].Count,GpuInfo.Gpus[0].Name]' \
+    --output table
+```
+
+## Instructions
+
+### 1. Provision the VMs
+
+Each VM was provisioned using standard AWS EC2 CLI commands — the only change from a normal AWS workflow is `AWS_PROFILE=spinifex`, which redirects the CLI to Spinifex's local EC2-compatible endpoint instead of AWS:
+
+| VM | Instance type | GPUs | Role | Model |
+|---|---|---|---|---|
+| `vm-yolo` | g7e.12xlarge | 2× MI350X | Real-time object detection | YOLO11x |
+| `vm-vlm`  | g7e.12xlarge | 2× MI350X | Multi-modal scene analysis | Qwen3-VL 235B FP8 |
+| `vm-chat` | g7e.12xlarge | 2× MI350X | Conversational LLM | Llama 3.3 70B |
+
+Six of the eight available MI350Xs are claimed here; the remaining two stay idle on the host, available for a fourth tenant without touching the existing three.
+
+```bash
+export AWS_PROFILE=spinifex
+
+# vm-yolo — YOLO11x object detection, 200 GB disk
+YOLO_ID=$(aws ec2 run-instances \
+    --image-id ami-ubuntu-amd-gpu \
+    --instance-type g7e.12xlarge \
+    --key-name spinifex-key \
+    --subnet-id <subnet-id> \
+    --security-group-ids <sg-id> \
+    --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=200,DeleteOnTermination=true}' \
+    --count 1 \
+    --query 'Instances[0].InstanceId' --output text)
+echo "vm-yolo launched: $YOLO_ID"
+
+# vm-vlm — Qwen3-VL 235B FP8, 600 GB disk for weights
+VLM_ID=$(aws ec2 run-instances \
+    --image-id ami-ubuntu-amd-gpu \
+    --instance-type g7e.12xlarge \
+    --key-name spinifex-key \
+    --subnet-id <subnet-id> \
+    --security-group-ids <sg-id> \
+    --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=600,DeleteOnTermination=true}' \
+    --count 1 \
+    --query 'Instances[0].InstanceId' --output text)
+echo "vm-vlm launched: $VLM_ID"
+
+# vm-chat — Llama 3.3 70B, 300 GB disk
+CHAT_ID=$(aws ec2 run-instances \
+    --image-id ami-ubuntu-amd-gpu \
+    --instance-type g7e.12xlarge \
+    --key-name spinifex-key \
+    --subnet-id <subnet-id> \
+    --security-group-ids <sg-id> \
+    --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=300,DeleteOnTermination=true}' \
+    --count 1 \
+    --query 'Instances[0].InstanceId' --output text)
+echo "vm-chat launched: $CHAT_ID"
+
+# Wait for all three to reach running state
+for ID in "$YOLO_ID" "$VLM_ID" "$CHAT_ID"; do
+    aws ec2 wait instance-running --instance-ids "$ID"
+    echo "$ID is running"
+done
+
+# Retrieve IPs (Spinifex assigns addresses from the internal VPC)
+YOLO_IP=$(aws ec2 describe-instances --instance-ids "$YOLO_ID" \
+    --query 'Reservations[0].Instances[0].PublicIpAddress' --output text)
+VLM_IP=$(aws ec2 describe-instances --instance-ids "$VLM_ID" \
+    --query 'Reservations[0].Instances[0].PublicIpAddress' --output text)
+CHAT_IP=$(aws ec2 describe-instances --instance-ids "$CHAT_ID" \
+    --query 'Reservations[0].Instances[0].PublicIpAddress' --output text)
+
+echo "vm-yolo: $YOLO_IP"
+echo "vm-vlm:  $VLM_IP"
+echo "vm-chat: $CHAT_IP"
+```
+
+### 2. Verify GPU access
+
+Once SSH is available, confirm both GPUs are visible inside each VM:
+
+```bash
+ssh -i ~/.ssh/spinifex-key ubuntu@$CHAT_IP 'lspci | grep -i amd'
+```
+
+Or install and run `amd-smi`:
+
+```bash
+ssh -i ~/.ssh/spinifex-key ubuntu@$CHAT_IP \
+    'curl -fsSL https://repo.radeon.com/rocm/rocm.gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/rocm.gpg >/dev/null && \
+     echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/6.3 noble main" | sudo tee /etc/apt/sources.list.d/rocm.list && \
+     sudo apt-get update -qq && sudo apt-get install -y -q amd-smi-lib && \
+     amd-smi list'
+```
+
+<img src="../../../.github/assets/images/2gpus.png" alt="amd-smi inside vm-chat confirming two MI350Xs are directly attached, each with a unique UUID">
+
+Two MI350X entries with distinct UUIDs confirm direct PCIe passthrough is working.
+
+### 3. The orchestration layer
+
+Spinifex provides an EC2-compatible API on bare metal. The relevant capabilities for this demo:
+
+- **Multi-GPU claim per VM.** Each `g7e.12xlarge` requires two PCIe addresses claimed as a unit. Spinifex queries the available GPU pool on the host and assigns them to the guest VM at creation time, rolling back if the full allocation cannot be satisfied.
+- **PCIe passthrough via vfio-pci.** All 8 MI350Xs are bound to `vfio-pci` on the bare-metal host. Each VM gets its GPUs at the hardware level.
+- **Internal VPC (192.168.10.0/24).** VMs communicate directly without traversing the host's public network.
+
+The result: three completely independent EC2 instances, each running their own separate workloads using only the GPU resources they were allocated.
+
+### 4. Test methodology
+
+Two capture runs were made against the same workload script: **30 s baseline** (YOLO and VLM models running, no chat load) → **25 scripted Llama chat prompts** at 15 s intervals → **30 s settle tail**.
+
+The only difference between the two runs was tensor parallelism:
+
+- **Demo 1 (TP=1):** each VM uses one of its two allocated GPUs. The second sits idle.
+- **Demo 2 (TP=2):** each VM shards its model symmetrically across both GPUs using AMD Infinity Fabric (XGMI).
+
+| Metric | TP=1 baseline (Demo 1) | TP=2 (Demo 2) |
+|---|---|---|
+| Wall clock | 613 s | 600 s |
+| YOLO avg fps | 16.91 (min 15.3 / max 18.0) | 17.09 (min 15.6 / max 18.1) |
+| Qwen peak tok/s | 61.7 | 53.0 |
+| Llama peak tok/s | 36.4 | **53.1** |
+| Both GPUs active (chat VM) | no — GPU 1 idle | **yes — symmetric** |
+| VRAM per GPU (Qwen) | 259 GB on one, 0.3 GB on other | 241 GB on **both** |
+| VRAM per GPU (Llama) | 261 GB on one, 0.3 GB on other | 248 GB on **both** |
+
+### 5. Results
+
+### Inter-tenant isolation
+
+The charts below show the three independent workloads spiking activity on their associated GPUs, measured in GFX utilisation and power draw. YOLO average FPS stays at ~17 despite utilisation in the YOLO VM never exceeding 20%, indicating a bottleneck in networking or encoding rather than inference. Each chart highlights workload independence — pulses of GPU activity share no correlation across VMs.
+
+<img src="../../../.github/assets/images/smci/smci-demo1-gpu-utilisation.png" alt="GPU utilisation across all three VMs — Demo 1 TP=1">
+
+GPU utilisation per VM across the full Demo 1 capture.
+
+<img src="../../../.github/assets/images/smci/smci-demo2-gpu-power.png" alt="GPU power draw per workload — Demo 2 TP=2">
+
+Power draw per VM during Demo 2.
+
+### Single vs double GPU
+
+When TP=2 is enabled, each VM shards its model symmetrically across both GPUs. Inside the chat VM, both GPUs pulse together on every request, each holding exactly **248 GB of VRAM** (weights split + KV cache shards). With TP=1, GPU 1 sat at 0% for the entire run.
+
+<img src="../../../.github/assets/images/smci/smci-demo2-per-gpu-chat.png" alt="Llama 3.3 70B per-GPU activity — TP=2">
+
+Per-GPU utilisation inside `vm-chat` with TP=2.
+
+<img src="../../../.github/assets/images/smci/smci-demo1-per-gpu-chat.png" alt="Llama 3.3 70B per-GPU activity — TP=1">
+
+Same VM with TP=1.
+
+### vLLM generation throughput
+
+With TP=1, Qwen peaks at ~60 tok/s and Llama caps around 35 tok/s — both compute-bound on a single MI350X. Enabling TP=2 lifts Llama's peak to ~50 tok/s (+47%) as generation is distributed symmetrically across both GPUs. Qwen's throughput is slightly lower under TP=2 due to collective overhead on the shared-memory transport.
+
+<img src="../../../.github/assets/images/smci/smci-demo1-vllm-throughput.png" alt="vLLM generation throughput — TP=1">
+
+Generation throughput during Demo 1 (TP=1).
+
+<img src="../../../.github/assets/images/smci/smci-demo2-vllm-throughput.png" alt="vLLM generation throughput — TP=2">
+
+Generation throughput during Demo 2 (TP=2).
+
+### 6. Teardown
+
+```bash
+# Stop workloads on each VM
+for IP in "$YOLO_IP" "$VLM_IP" "$CHAT_IP"; do
+    ssh -i ~/.ssh/spinifex-key ubuntu@$IP 'docker stop $(docker ps -q)' || true
+done
+
+# Terminate all three instances — releases 6× MI350X back to the Spinifex pool
+aws ec2 terminate-instances --instance-ids "$YOLO_ID" "$VLM_ID" "$CHAT_ID"
+```
+
+The six MI350Xs are immediately returned to the pool once the instances terminate, ready for a new allocation without touching the host.
+
+### 7. Conclusion
+
+This tutorial demonstrates how Spinifex can turn a single bare-metal chassis into a multi-tenant AI serving platform. Spinifex utilises the flexibility of PCIe passthrough to allocate GPU resources in whichever configuration is required by the workload/s.
+
+Importantly, it does so with standard `aws ec2` CLI calls — `run-instances`, `describe-instances`, `terminate-instances` — against Spinifex's EC2-compatible endpoint. Teams already operating AWS infrastructure can point their existing tooling at a Spinifex node with a single profile change, against GPUs that sit in their own rack.

docs/reference-architectures/arcee-trinity-mi350x/README.md

@@ -0,0 +1,319 @@
+---
+title: "Arcee Trinity 400B FP8 on AMD MI350X with Spinifex"
+description: "Provision, load, and benchmark arcee-ai/Trinity-Large-Preview-FP8 across two AMD MI350X GPUs on a Supermicro H14 using Spinifex's EC2-compatible API."
+category: "Reference Architectures"
+tags:
+  - supermicro
+  - amd
+  - mi350x
+  - gpu-passthrough
+  - vllm
+  - llm
+  - arcee
+resources:
+  - title: "GPU Passthrough"
+    url: "/docs/gpu-passthrough"
+  - title: "Launching Instances"
+    url: "/docs/launching-instances"
+  - title: "Arcee AI Trinity"
+    url: "https://www.arcee.ai/trinity"
+  - title: "Trinity on HuggingFace"
+    url: "https://huggingface.co/arcee-ai/Trinity-Large-Preview-FP8"
+  - title: "AMD Instinct MI350X"
+    url: "https://www.amd.com/en/products/accelerators/instinct/mi300/mi350x.html"
+  - title: "vLLM Documentation"
+    url: "https://docs.vllm.ai"
+sections:
+  - overview
+  - prerequisites
+  - instructions
+---
+
+## Overview
+
+### About Trinity
+
+[Trinity](https://www.arcee.ai/trinity) is a family of open-weight language models from [Arcee AI](https://www.arcee.ai), built specifically for production agentic workloads: reliable tool calling, structured JSON output, coherent multi-turn conversations, and long-context reasoning. The family spans four sizes — Nano (6B), Mini (26B), Large Preview (400B), and Large Thinking (400B reasoning-optimised), with consistent capabilities and API surface across all of them. A workflow validated on Nano can be promoted to Large with no prompt changes.
+
+Trinity Large Preview is a **sparse Mixture-of-Experts (MoE)** model. The headline number is 400B total parameters, but only **13B parameters activate per token** — the MoE routing selects a subset of experts for each forward pass, so inference cost is closer to a much smaller dense model than the raw parameter count suggests. The result is a 400B-class model with a **512K token context window** that fits on two high-memory GPUs and generates at competitive throughput.
+
+Arcee has trained Trinity with a heavy focus on agent reliability — function selection accuracy, valid parameter generation, graceful failure recovery, and schema adherence for structured outputs. It is available as open weights on HuggingFace and is natively compatible with vLLM, SGLang, and llama.cpp.
+
+### This guide
+
+This writeup covers running Trinity Large Preview FP8 on a Supermicro H14 bare-metal node via Spinifex — our EC2-compatible orchestration layer. Two AMD Instinct MI350X GPUs (288 GB HBM3e each, 576 GB combined) provide just enough headroom for the FP8 weights plus active KV cache, provisioned and managed with standard `aws ec2` CLI commands.
+
+At that scale the FP8 weights require around 400 GB of GPU memory — more than a single MI350X can hold. This guide covers splitting the model symmetrically across two MI350Xs using vLLM tensor parallelism (TP=2), running on a `g7e.12xlarge` instance provisioned via Spinifex on the H14.
+
+The complete stack:
+
+| Layer | Detail |
+|---|---|
+| Bare-metal host | Supermicro H14, 8× AMD Instinct MI350X |
+| Orchestration | Spinifex (EC2-compatible API) |
+| Instance type | `g7e.12xlarge` — 2× MI350X via PCIe passthrough |
+| Disk | 800 GB (model weights alone are ~400 GB) |
+| Inference runtime | vLLM (`rocm/vllm` Docker image) |
+| Tensor parallelism | TP=2 — model sharded symmetrically across GPU 0 and GPU 1 |
+| Model | `arcee-ai/Trinity-Large-Preview-FP8` |
+
+### Platform
+
+| Component | Specification |
+|---|---|
+| **Bare-metal host** | Supermicro H14 |
+| **Host OS** | Ubuntu 24.04 LTS or Debian 13 (minimum) |
+| **Orchestration** | Spinifex — EC2-compatible bare-metal API |
+| **Guest OS** | Ubuntu 26.04 LTS |
+| **GPUs** | 2× AMD Instinct MI350X (288 GB HBM3e each, 576 GB combined) |
+| **GPU passthrough** | PCIe passthrough via vfio-pci |
+| **Instance type** | `g7e.12xlarge` |
+| **Container runtime** | Docker (with ROCm device access) |
+| **Inference runtime** | vLLM (`rocm/vllm` image), tensor parallelism TP=2 |
+| **Block storage** | Viperblock — EBS-compatible, 800 GB |
+| **Model** | `arcee-ai/Trinity-Large-Preview-FP8` (400B MoE, ~13B active/token) |
+
+Spinifex runs on the bare-metal host and presents an EC2-compatible API endpoint. Launching a `g7e.12xlarge` atomically claims two MI350Xs from the host's GPU pool and binds them to the guest VM via PCIe passthrough — the guest OS communicates with the hardware directly, with no software virtualisation layer in the data path. When the instance terminates, those two GPUs are immediately returned to the pool.
+
+GPU passthrough requires a host kernel and OS that supports vfio-pci. Ubuntu 24.04 LTS (kernel 6.8+) and Debian 13 (kernel 6.12+) are the tested minimum baselines for the host. Guest VMs run Ubuntu 26.04 LTS, which ships with the ROCm-compatible kernel and userspace expected by the AMD GPU AMI.
+
+## Prerequisites
+
+- Supermicro H14 with at least 2× AMD Instinct MI350X installed
+- Host OS: **Ubuntu 24.04 LTS** (kernel 6.8+) or **Debian 13** (kernel 6.12+) — minimum for vfio-pci support
+- Spinifex installed and all services running (`systemctl status spinifex.target`)
+- GPU passthrough configured — `spx admin gpu setup` (reboot required) then `spx admin gpu enable`
+- AMD GPU AMI registered (`ami-ubuntu-amd-gpu`) — Ubuntu 26.04 LTS with ROCm-compatible kernel
+- AWS CLI configured with `AWS_PROFILE=spinifex` pointing at the Spinifex endpoint
+- SSH key pair imported, VPC and security group created (see [Launching Instances](/docs/launching-instances))
+- Docker installed in the guest VM (included in the AMD GPU AMI)
+- HuggingFace account with access to `arcee-ai/Trinity-Large-Preview-FP8`
+
+GPU passthrough must be configured before launching GPU instances:
+
+```bash
+sudo spx admin gpu setup   # binds GPUs to vfio-pci — requires reboot
+# ... reboot ...
+sudo spx admin gpu enable  # confirms passthrough and makes GPU pool available
+```
+
+## Instructions
+
+### 1. Provision the instance
+
+Spinifex exposes a standard EC2 API on the bare-metal host. The only difference from a real AWS workflow is `AWS_PROFILE=spinifex`.
+
+```bash
+export AWS_PROFILE=spinifex
+
+# Check available GPU instance types
+aws ec2 describe-instance-types \
+    --query 'InstanceTypes[?GpuInfo].[InstanceType,GpuInfo.Gpus[0].Count,GpuInfo.Gpus[0].Name]' \
+    --output table
+
+# Launch a g7e.12xlarge with 800 GB disk
+# Spinifex atomically claims 2× MI350X via PCIe passthrough for this instance type
+INSTANCE_ID=$(aws ec2 run-instances \
+    --image-id ami-ubuntu-amd-gpu \
+    --instance-type g7e.12xlarge \
+    --key-name spinifex-key \
+    --subnet-id <subnet-id> \
+    --security-group-ids <sg-id> \
+    --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=800,DeleteOnTermination=true}' \
+    --count 1 \
+    --query 'Instances[0].InstanceId' --output text)
+
+echo "Launched: $INSTANCE_ID"
+
+# Wait for running state
+aws ec2 wait instance-running --instance-ids "$INSTANCE_ID"
+
+# Get the IP
+INSTANCE_IP=$(aws ec2 describe-instances \
+    --instance-ids "$INSTANCE_ID" \
+    --query 'Reservations[0].Instances[0].PublicIpAddress' --output text)
+
+echo "IP: $INSTANCE_IP"
+```
+
+Once SSH is available, confirm both GPUs are visible:
+
+```bash
+ssh -i ~/.ssh/spinifex-key ubuntu@$INSTANCE_IP 'lspci | grep -i amd'
+```
+
+Or install and run `amd-smi` on the instance:
+
+<img src="../../../.github/assets/images/2gpus.png" alt="amd-smi inside the guest VM confirming two MI350Xs are directly attached, each with a unique UUID">
+
+Two MI350X entries with distinct UUIDs confirm direct PCIe passthrough.
+
+### 2. Pull the vLLM Docker image
+
+```bash
+ssh -i ~/.ssh/spinifex-key ubuntu@$INSTANCE_IP
+
+# On the VM:
+docker pull rocm/vllm:latest
+```
+
+> The `rocm/vllm` image is large (~20 GB). Pull it while the model is downloading in parallel if bandwidth allows.
+
+### 3. Pull the model
+
+Trinity Large Preview FP8 weighs roughly 400 GB, so expect the model download to take some time.
+
+```bash
+# On the VM:
+pip install 'huggingface-hub[hf_xet]'
+hf download 'arcee-ai/Trinity-Large-Preview-FP8' --repo-type model
+
+# Monitor progress — looking for ~400 GB total
+du -sh ~/.cache/huggingface/hub/
+```
+
+The model lands in `~/.cache/huggingface/hub/`. vLLM picks it up automatically from there on serve.
+
+### 4. Run vLLM
+
+Trinity needs TP=2 to fit across both MI350Xs:
+
+```bash
+# On the VM:
+docker run --rm -d \
+    --name trinity \
+    --device /dev/kfd \
+    --device /dev/dri \
+    --group-add video \
+    --ipc host \
+    --network host \
+    -v ~/.cache/huggingface:/root/.cache/huggingface \
+    rocm/vllm:latest \
+    vllm serve arcee-ai/Trinity-Large-Preview-FP8 \
+        --tensor-parallel-size 2 \
+        --dtype auto \
+        --max-model-len 8192 \
+        --gpu-memory-utilization 0.90 \
+        --host 0.0.0.0 \
+        --port 8003
+```
+
+Watch the startup log:
+
+```bash
+docker logs -f trinity
+```
+
+Loading a 400B FP8 model across two GPUs takes **2–4 minutes**. Wait for:
+
+```
+INFO:     Application startup complete.
+```
+
+Then verify:
+
+```bash
+curl http://localhost:8003/health
+curl http://localhost:8003/v1/models | python3 -m json.tool
+```
+
+### 5. Test it
+
+A quick sanity check via the OpenAI-compatible API before running benchmarks:
+
+```bash
+curl http://localhost:8003/v1/chat/completions \
+    -H "Content-Type: application/json" \
+    -d '{
+        "model": "arcee-ai/Trinity-Large-Preview-FP8",
+        "messages": [{"role": "user", "content": "Explain tensor parallelism in three sentences."}],
+        "max_tokens": 256
+    }'
+```
+
+The first request will be slowest — KV cache is cold. Subsequent requests warm up noticeably.
+
+### 6. Dashboard
+
+We created a quick dashboard for a live view of the model's telemetry — throughput, latency, GPU utilisation, VRAM, and KV cache — streamed via SSE from the vLLM metrics endpoint and the GPU stats sidecar.
+
+<p><video src="https://iso.mulgadc.com/trinity-workload.mp4" controls width="100%" style="border-radius:6px"></video></p>
+
+### 7. Benchmark results
+
+Two captures were run against the live model:
+
+- **Chat session:** A short interactive session — low concurrency, variable cadence, 56 s total. Reflects real conversational use.
+- **Scripted benchmark:** 5-minute sustained load driver — rotating batch of 15 long-form technical prompts at 1.5 s cadence. Reflects peak throughput.
+
+| Metric | Chat session | Benchmark (5 min) |
+|---|---|---|
+| Peak tok/s | 67.1 | 67.0 |
+| Avg tok/s (under load) | 18.7 | **52.9** |
+| Avg TTFT | 1668 ms | 758 ms |
+| Min TTFT | 1502 ms | 538 ms |
+| Max TTFT | 1871 ms | 1165 ms |
+| GPU 0 util (avg) | 30.2% | 88.8% |
+| GPU 1 util (avg) | 30.1% | 88.7% |
+| VRAM per GPU | 259.6 GB | 259.6 GB |
+| Avg power (both GPUs) | 664 W | 788 W |
+| Peak power | 828 W | 839 W |
+| GPU temp | 65 °C | 65 °C |
+
+The lower avg utilisation in the chat session reflects idle gaps between conversational turns; peak throughput is identical at ~67 tok/s in both runs, indicating the hardware ceiling rather than a software one.
+
+### Generation throughput
+
+<img src="../../../.github/assets/images/trinity/trinity-throughput.png" alt="Trinity generation throughput — scripted benchmark">
+
+Generation and prompt-ingest tok/s over the 5-minute benchmark. The model sustains ~53 tok/s average at continuous load, peaking at 67 tok/s.
+
+### Per-GPU compute utilisation
+
+<img src="../../../.github/assets/images/trinity/trinity-per-gpu-util.png" alt="Trinity per-GPU utilisation — scripted benchmark">
+
+GPU 0 and GPU 1 track each other closely throughout the benchmark — TP=2 distributes attention and FFN layers symmetrically across both MI350Xs.
+
+### Per-GPU VRAM
+
+<img src="../../../.github/assets/images/trinity/trinity-per-gpu-vram.png" alt="Trinity per-GPU VRAM — scripted benchmark">
+
+Both GPUs hold 259.6 GB — model weights sharded evenly, plus KV cache. 90.2% of HBM3e is occupied at rest, leaving ~26 GB headroom per GPU for active KV cache during generation.
+
+### Per-GPU power
+
+<img src="../../../.github/assets/images/trinity/trinity-per-gpu-power.png" alt="Trinity per-GPU power — scripted benchmark">
+
+Power draw during active generation peaks at ~420 W per GPU, for a combined chassis draw well under the MI350X's 750 W TDP per card. Idle between prompts returns to ~300 W.
+
+### Latency
+
+<img src="../../../.github/assets/images/trinity/trinity-latency.png" alt="Trinity latency — scripted benchmark">
+
+Cumulative average TTFT and E2E latency over the benchmark. TTFT settles to ~758 ms avg once the KV cache is warm; E2E tracks request length as expected.
+
+### KV cache utilisation
+
+<img src="../../../.github/assets/images/trinity/trinity-kv-cache.png" alt="Trinity KV cache utilisation — scripted benchmark">
+
+KV cache occupancy rises as requests pile up during the sustained load phase. The queue depth stays near zero — the model keeps pace with the 1.5 s prompt cadence.
+
+### 8. Teardown
+
+```bash
+# Stop the container
+ssh -i ~/.ssh/spinifex-key ubuntu@$INSTANCE_IP 'docker stop trinity'
+
+# Terminate the instance — releases the 2× MI350X back to the Spinifex pool
+aws ec2 terminate-instances --instance-ids "$INSTANCE_ID"
+```
+
+The two MI350Xs are immediately available to Spinifex for a new workload once the instance terminates.
+
+### 9. Conclusion
+
+Trinity Large Preview demonstrates that frontier-class open-weight models are now within reach of a two-GPU bare-metal setup. The MoE architecture is crucial: 400B total parameters with only 13B active per token means the model generates at ~53 tok/s sustained and peaks at 67 tok/s on a pair of MI350Xs — throughput that would be difficult to match with a dense model of comparable quality at that memory footprint.
+
+Combining this with Spinifex as the infrastructure layer allows teams to own the entire workflow. Provisioning a `g7e.12xlarge` instance, pulling a 400 GB model, and serving it through vLLM took a handful of standard `aws ec2` commands and a single `docker run`. The same workflow runs identically on any Spinifex-managed bare-metal node, with the GPUs returned to the pool when instances terminate.
+
+For teams evaluating Trinity for production agentic workloads — tool calling, structured outputs, long-context reasoning — this represents a credible self-hosted deployment path: open weights, Apache 2.0 license, owned hardware and familiar tooling.

docs/security/malware-protection/README.md

@@ -73,7 +73,7 @@ All Spinifex nodes must include the following in their malware protection scope.
 |------|----------|
 | `/usr/local/bin/spx` | Spinifex CLI and service entry point. All `spinifex-*.service` units invoke `spx service <name> start`, so every service binary is reached through this one path. |
 | `/usr/bin/ovn-*`, `/usr/bin/ovs-*` | OVN / OVS control-plane and datapath binaries (installed via `apt` from the `ovn-central`, `ovn-host`, `openvswitch-switch` packages). |
-| `/usr/lib/<arch>-linux-gnu/nbdkit/plugins/nbdkit-viperblock-plugin.so` | Viperblock NBD plugin loaded by `nbdkit` to expose EBS volumes as block devices. The exact path comes from `nbdkit --dump-config` (`plugindir=`); on Debian 12 amd64 it is `/usr/lib/x86_64-linux-gnu/nbdkit/plugins/`. |
+| `/usr/lib/<arch>-linux-gnu/nbdkit/plugins/nbdkit-viperblock-plugin.so` | Viperblock NBD plugin loaded by `nbdkit` to expose EBS volumes as block devices. The exact path comes from `nbdkit --dump-config` (`plugindir=`); on Debian 13 amd64 it is `/usr/lib/x86_64-linux-gnu/nbdkit/plugins/`. |
 | `/usr/local/share/ca-certificates/spinifex-ca.crt` | Cluster CA cert installed into the system trust store |
 
 ### 1.2 Configuration and Secrets
@@ -157,7 +157,7 @@ All scan runs — scheduled and real-time — must produce a log entry that incl
 
 ## 4. Reference Configurations
 
-The configurations below are reference baselines. They are known to meet the CMMC L1 objectives on a standard Debian 12 Spinifex node; they are not the only valid implementation. Operators running a commercial EDR should treat [§1](#1-designated-locations-sil1-3142-a) and [§3](#3-scan-schedule-sil1-3145) as the contract and configure their chosen product to match.
+The configurations below are reference baselines. They are known to meet the CMMC L1 objectives on a standard Debian 13 Spinifex node; they are not the only valid implementation. Operators running a commercial EDR should treat [§1](#1-designated-locations-sil1-3142-a) and [§3](#3-scan-schedule-sil1-3145) as the contract and configure their chosen product to match.
 
 ### 4.1 ClamAV (signature-based AV)
 

docs/security/media-sanitization/README.md

@@ -0,0 +1,258 @@
+---
+title: "Media Sanitization and Disposal"
+description: "Operator guide for sanitizing and disposing of storage media used by Spinifex nodes"
+category: "Security"
+sections:
+  - overview
+tags:
+  - security
+  - compliance
+  - cmmc
+  - media
+  - sanitization
+  - disposal
+  - decommissioning
+resources:
+  - title: "NIST SP 800-88 Rev 1 — Guidelines for Media Sanitization"
+    url: "https://csrc.nist.gov/pubs/sp/800/88/r1/final"
+  - title: "NIST SP 800-171 Rev 3"
+    url: "https://csrc.nist.gov/pubs/sp/800/171/r3/final"
+  - title: "CMMC Level 1 Self-Assessment Guide v2.0"
+    url: "https://dodcio.defense.gov/CMMC/Documentation"
+  - title: "ATA/ATAPI Command Set — Secure Erase"
+    url: "https://www.t13.org"
+  - title: "NVM Express — Sanitize Command"
+    url: "https://nvmexpress.org/specifications"
+---
+
+# Media Sanitization and Disposal
+
+> Operator guide for sanitizing and disposing of storage media used by Spinifex nodes
+
+## Table of Contents
+
+- [Overview](#overview)
+- [CMMC Practices Covered](#cmmc-practices-covered)
+- [Approach](#approach)
+- [1. Media in Scope](#1-media-in-scope)
+- [2. Sanitization Method Selection](#2-sanitization-method-selection)
+- [3. Volume-Level Sanitization — Before Reuse (MP.L1-3.8.3 [b])](#3-volume-level-sanitization--before-reuse-mpl1-383-b)
+- [4. Whole-Drive and Node Decommissioning — Before Disposal (MP.L1-3.8.3 [a])](#4-whole-drive-and-node-decommissioning--before-disposal-mpl1-383-a)
+- [5. Key Destruction](#5-key-destruction)
+- [6. Removable and Backup Media](#6-removable-and-backup-media)
+- [7. Evidence and Record Keeping](#7-evidence-and-record-keeping)
+- [8. Operator Checklist](#8-operator-checklist)
+
+---
+
+## Overview
+
+**Audience:** Operators decommissioning, reassigning, returning under warranty, or disposing of hardware that has been used to run Spinifex nodes in environments subject to CMMC Level 1, or any site that requires documented media sanitization.
+
+**Scope:** Every storage medium that has, or may have, held Federal Contract Information (FCI) as part of a Spinifex deployment — node system disks, Viperblock WAL/chunk disks, Predastore object stores, NATS JetStream disks, removable master-key tokens, backup media, and any loose drives that previously occupied a Spinifex role. Guest VM internal media (disks as seen from inside a tenant VM) is the workload owner's responsibility and out of scope.
+
+## CMMC Practices Covered
+
+| Practice | Title | Objective |
+|----------|-------|-----------|
+| MP.L1-3.8.3 | Media Disposal | [a] System media containing FCI is sanitized or destroyed before disposal. [b] System media containing FCI is sanitized before it is released for reuse. |
+
+## Approach
+
+NIST SP 800-88 Rev 1 defines three sanitization categories — **Clear**, **Purge**, and **Destroy** — chosen by the confidentiality of the data and whether the medium will leave the operator's control. Spinifex discharges MP.L1-3.8.3 along two tracks:
+
+1. **Volume-level sanitization** for EBS volumes and VM disks returning to the free pool. Handled in-platform by **cryptographic erase** — the per-volume data encryption key (DEK) is deleted, rendering the ciphertext in Predastore S3 irrecoverable. This is the NIST SP 800-88 **Purge**-level method for encrypted media.
+2. **Whole-drive sanitization** for physical media leaving a Spinifex node — drive replacement, node retirement, warranty return, resale, or destruction. The operator runs the sanitization; Spinifex does not and cannot reach the firmware commands required. This guide prescribes the method per media type and the records to keep.
+
+The key principle: every piece of media that ever held plaintext FCI (or keys that wrapped FCI ciphertext) must be sanitized to at least **Purge** before release for reuse, and to **Purge** or **Destroy** before leaving operator control. When in doubt, Destroy.
+
+## 1. Media in Scope
+
+Any medium in any of these roles on a Spinifex node has held, or may have held, FCI:
+
+| Medium | Typical hardware | Contents |
+|--------|------------------|----------|
+| Node system disk | NVMe/SATA SSD | `/etc/spinifex/master.key`, cluster CA key, per-node TLS keys, service configs, logs, journal, swap. |
+| Viperblock WAL device | NVMe/SATA SSD | Plaintext of in-flight block writes before chunking. High-churn, short residency, but FCI lands here. |
+| Viperblock chunk cache / local backing | NVMe/SATA SSD or HDD | Encrypted volume chunks. |
+| Predastore object store | NVMe/SATA SSD or HDD | All S3 object data — AMIs, snapshots, user-uploaded artifacts, IAM state files, tenant data. |
+| NATS JetStream disk | NVMe/SATA SSD | IAM NATS token, cluster metadata, pending job state, NATS KV entries (including wrapped DEKs). |
+| Removable master-key media | USB flash token, HSM, smart card | Master encryption key (Phase 4 USB-mount path). Destroying this sanitizes every DEK-encrypted volume cluster-wide. |
+| Backup media | LTO tape, removable HDD/SSD, off-site cloud backup | Any of the above, point-in-time. |
+| BMC / iDRAC / iLO storage | Embedded flash | Console recordings, SEL logs, BMC credentials, cached operator certificates. Sanitize via the BMC "Reset to defaults" / "Erase user data" command before disposal. |
+| Switch / router config storage | Embedded flash | Cluster VLAN, management IPs, ACLs. Operator network kit, out of Spinifex scope but noted for completeness. |
+| Optical / write-once media | DVD/BD-R | If used to transport keys or images. Always Destroy. |
+
+Any drive whose history cannot be traced — pulled from a spares bin, recovered from a failed host, found unlabelled — must be treated as though it held FCI.
+
+## 2. Sanitization Method Selection
+
+The method depends on (a) media type and (b) whether the media remains inside the protected boundary after sanitization.
+
+| Scenario | Minimum method | Reference |
+|----------|----------------|-----------|
+| Encrypted volume returning to free pool, DEK deletable | **Cryptographic Erase** (Purge) via DEK deletion | [§3](#3-volume-level-sanitization--before-reuse-mpl1-383-b) |
+| SSD / NVMe leaving the node | **Cryptographic Erase** + **Purge** (SANITIZE block-erase or ATA Secure Erase) | [§4.1](#41-nvme-ssd) |
+| Magnetic HDD leaving the node, operational | **Purge** (single-pass overwrite with verify, or ATA Secure Erase on drives that support it) | [§4.3](#43-hdd) |
+| Magnetic HDD leaving the node, faulty (not writable) | **Destroy** (degauss followed by shred/incinerate) | [§4.3](#43-hdd) |
+| Any media leaving operator control (resale, warranty, disposal) | **Destroy** if residual confidentiality concern remains after Purge; otherwise Purge + third-party attestation | [§4.4](#44-destruction) |
+| Tape | **Destroy** (degauss + shred) — overwrite is not reliable on LTO with compression | [§6](#6-removable-and-backup-media) |
+| Optical write-once | **Destroy** (shred/incinerate) | [§6](#6-removable-and-backup-media) |
+| BMC / switch flash | Vendor "factory reset + erase user data"; Destroy chip if procedure is not available | [§1](#1-media-in-scope) |
+
+**Cryptographic erase is only valid when** the key was generated and handled under modern cryptographic hygiene (AES-256 or equivalent, key never exfiltrated, key store itself sanitized) and the encrypted data was not also written anywhere in plaintext. For Spinifex volumes these conditions hold.
+
+**Self-encrypting drives (SEDs)** support a single-command cryptographic erase via the TCG Opal "RevertSP" / `sedutil-cli --revertNoErase` or the ATA `SECURITY ERASE UNIT ENHANCED` that triggers the on-drive MEK rotation. When using an SED as a node system or data disk, record the SED model and method in the decommissioning procedure.
+
+## 3. Volume-Level Sanitization — Before Reuse (MP.L1-3.8.3 [b])
+
+Applies to EBS volumes, AMIs, and snapshots whose underlying storage will be reused for a different tenant or workload.
+
+### 3.1 Cryptographic Erase via DEK Deletion
+
+Deletion is sanitization:
+
+- `DeleteVolume` removes the wrapped DEK from NATS KV. The chunks in Predastore S3 remain but are unrecoverable ciphertext. No further operator action is required.
+- `TerminateInstances` with the default `DeleteOnTermination=true` on root volumes performs the same cryptographic erase for the terminated instance's root.
+- `DeleteSnapshot` removes the snapshot's DEK wrapper; the underlying chunks become irrecoverable ciphertext.
+
+This is the sanitization-before-reuse path for every in-platform object. Operators do not need to take separate action — the API call is the sanitization.
+
+### 3.2 What Happens to the Physical Storage
+
+Cryptographic erase leaves ciphertext chunks in Predastore. These chunks are eventually overwritten as new volumes reuse the space. For CMMC purposes, the ciphertext without the DEK satisfies Purge. The underlying drive still requires whole-drive sanitization when it eventually leaves the cluster — see [§4](#4-whole-drive-and-node-decommissioning--before-disposal-mpl1-383-a).
+
+## 4. Whole-Drive and Node Decommissioning — Before Disposal (MP.L1-3.8.3 [a])
+
+Applies when a physical drive — or a whole node — leaves the Spinifex cluster: drive swap, node retirement, warranty return, resale, recycling, destruction.
+
+### 4.1 NVMe SSD
+
+Preferred: the NVMe **SANITIZE** command with the **Block Erase** action. Supported on most enterprise NVMe drives.
+
+```bash
+# Confirm support
+nvme id-ctrl /dev/nvme0 | grep -i sanicap
+
+# Purge — block erase
+nvme sanitize /dev/nvme0 --sanact=2
+
+# Poll until complete
+nvme sanitize-log /dev/nvme0
+```
+
+Fallback for drives without SANITIZE support: NVMe **Format** with Secure Erase set to 1 (`--ses=1`), or vendor tool. Cryptographic Erase (`--sanact=4`) is acceptable for SEDs whose encryption posture is documented; otherwise Block Erase is the safer default.
+
+Boot media: NVMe targets must be the non-boot drive when running from the OS. For the node system disk, boot a sanitization-purpose live USB (e.g. [PartedMagic](https://partedmagic.com/), the vendor's diagnostic ISO) or pull the drive and sanitize in a dedicated sanitization workstation.
+
+### 4.2 SATA/SAS SSD
+
+Preferred: **ATA Secure Erase (Enhanced)** via `hdparm`:
+
+```bash
+# Confirm support and not frozen
+hdparm -I /dev/sdX | grep -A1 "Security"
+
+# If frozen, power-cycle (suspend/resume or hot-swap) without unplugging the OS disk
+
+# Set password and issue enhanced erase
+hdparm --user-master u --security-set-pass p /dev/sdX
+hdparm --user-master u --security-erase-enhanced p /dev/sdX
+```
+
+Fallback: vendor Secure Erase utility, or `blkdiscard --secure /dev/sdX` on drives that report `TRIM deterministic + RZAT`. A single-pass overwrite with `shred -v -n 1 /dev/sdX` is **not** sufficient for SSDs because of wear-levelling reserve blocks — use Secure Erase or Destroy.
+
+### 4.3 HDD
+
+Preferred: **ATA Secure Erase** (same `hdparm` sequence as [§4.2](#42-satasas-ssd)). Most modern HDDs support it.
+
+Fallback: single-pass overwrite with verify:
+
+```bash
+shred -v -n 1 -z /dev/sdX
+# or
+dd if=/dev/zero of=/dev/sdX bs=1M status=progress && \
+  cmp /dev/zero /dev/sdX          # expect "EOF on /dev/zero" only
+```
+
+For HDDs that fail or refuse sanitization, or that held high-sensitivity data, **degauss then Destroy** — degaussing alone renders a modern HDD non-functional, so there is no reuse path after degauss. Degaussing requires an NSA/CSS-listed degausser appropriate to the drive's coercivity.
+
+### 4.4 Destruction
+
+When a drive cannot be sanitized (failed, SED with no working password, sanitize aborts) or when leaving operator control with any residual confidentiality concern, Destroy:
+
+- **SSD/NVMe:** shred to ≤2 mm particle size (NSA/CSS EPL-listed shredder) or incinerate. Crushing alone is not sufficient for modern flash — chips can survive.
+- **HDD:** degauss (on drives with magnetic platters) followed by shredding, drilling multiple holes through the platters and the head assembly, or incineration.
+- **Tape, optical:** shredding.
+- **Chain of custody:** transport destroyed media inside tamper-evident containers. The destruction record ([§7](#7-evidence-and-record-keeping)) must name the destruction method, destruction vendor (if third-party), and operator witness.
+
+Third-party destruction vendors must provide a certificate of destruction listing every drive serial. Reconcile against the asset register before the destruction record is closed.
+
+### 4.5 Node Decommissioning Runbook
+
+When retiring a whole node:
+
+1. **Drain** — migrate or terminate all instances hosted on the node; confirm Predastore and Viperblock roles are reassigned if the node held one. Volume DEKs of terminated instances are already cryptographically erased per [§3.1](#31-cryptographic-erase-via-dek-deletion).
+2. **Deregister** — `bd`-linked change ticket; the operator removes the node from the cluster (`spx admin node remove`) so NATS routes, OVN chassis, and predastore distributed membership drop it.
+3. **Stop services and unmount** — `systemctl stop 'spinifex-*'`, `nbdkit` sessions, OVN agent. Confirm `/run/spinifex/nbd/` is empty.
+4. **Destroy on-disk keys** — see [§5](#5-key-destruction). This is the single most important step: sanitization of volume data reduces to sanitization of keys once data is encrypted.
+5. **Sanitize each drive** by media type per [§4.1](#41-nvme-ssd) – [§4.3](#43-hdd). Every drive bay is sanitized, including any unpopulated cache or WAL devices.
+6. **Verify** — capture the output of the sanitize/erase command (exit code + sanitize-log for NVMe, `hdparm -I` "not enabled, not locked" for ATA, overwrite verify for HDD overwrite) into the decommissioning record.
+7. **Label** drives "Sanitized — <method> — <date> — <operator>" before they leave the rack or enter a transit container. Unlabelled drives are treated as unsanitized if reintroduced.
+8. **Record** per [§7](#7-evidence-and-record-keeping).
+
+## 5. Key Destruction
+
+Cryptographic erase relies on the key being unrecoverable. A drive that previously held `/etc/spinifex/master.key` must have that key destroyed — otherwise an attacker with both the drive image and the key recovers the cleartext even after "erase".
+
+| Key | Location | Destruction |
+|-----|----------|-------------|
+| `/etc/spinifex/master.key` | Node system disk (every node) | Overwrite the file (`shred -u /etc/spinifex/master.key`) before the drive is sanitized, in addition to drive sanitization. Once the master key is gone cluster-wide, every DEK it wrapped is effectively destroyed. |
+| Cluster CA private key (`/etc/spinifex/ca.key`) | Leader node system disk | Same treatment. Loss of the CA key does not sanitize data, but prevents impersonation of the cluster identity post-disposal. |
+| Per-node TLS private key (`/etc/spinifex/server.key`) | Each node system disk | `shred -u` before drive sanitization. |
+| Wrapped DEKs in NATS KV | `/var/lib/spinifex/nats/` on NATS-carrying nodes | Deleted via the `DeleteVolume` API; the KV record is compacted. Sanitization of the NATS JetStream disk per [§4](#4-whole-drive-and-node-decommissioning--before-disposal-mpl1-383-a) ensures no recoverable copies. |
+| Removable master-key media (Phase 4 USB) | USB flash token | Physically destroy the token when the cluster is decommissioned; alternatively retain inside a secured location as evidence of cluster-wide cryptographic erase. Do not reuse for another cluster without full sanitization (see [§6](#6-removable-and-backup-media)). |
+| Backup-copy keys | Any off-site or escrow copy | Destroy simultaneously with the primary. A surviving backup defeats the cryptographic erase. |
+
+If even one copy of the master key survives and the encrypted drives are recoverable, cryptographic erase has **not** been achieved. Track every key copy in the device register ([Physical Security Guide §5](/docs/security/physical-security-guide#5-manage-physical-access-devices-pel1-3105)).
+
+## 6. Removable and Backup Media
+
+| Media | Sanitization |
+|-------|--------------|
+| USB flash (master-key token, key-transport USB, image-import USB) | Destroy. The price of a replacement does not justify the risk of partial wear-levelled sanitization. Record the serial in the device register and the destruction in the decommissioning log. |
+| External HDD/SSD (backup) | Treat as [§4.2](#42-satasas-ssd) / [§4.3](#43-hdd). |
+| LTO tape | Degauss with a tape-rated degausser, then shred. Overwrite is not reliable under LTO hardware compression and is not accepted by 800-88 for tape Purge. |
+| Optical (CD/DVD/BD-R) | Shred or incinerate. |
+| Off-site cloud backup | Issue the cloud provider's delete-and-purge API; retain the provider's deletion-confirmation receipt. For cryptographically wrapped backups, destroy the wrapping key cluster-side and retain that deletion as evidence. |
+| Printed material (console photos, recovery codes, admin handover sheets) | Cross-cut shred. |
+
+Any removable medium that has entered the protected boundary and been written to must be tracked in the device register (see [Physical Security Guide §5](/docs/security/physical-security-guide#5-manage-physical-access-devices-pel1-3105)) and accounted for at decommissioning.
+
+## 7. Evidence and Record Keeping
+
+For CMMC assessment, retain the following for at least three years (or longer where contract policy requires):
+
+- **Asset register** listing every drive and removable medium by serial number, media type, role in the cluster (system / WAL / chunk / predastore / NATS / backup), and in-service / decommissioned status.
+- **Decommissioning record** per drive or node with: date, operator, sanitization method, command output or tool report, verification result, final disposition (reuse / return / destroy), destination (e.g. "warranty RMA #12345", "shredder vendor X, CoD #6789").
+- **Certificates of destruction** from third-party vendors, reconciled against the asset register.
+- **Key-destruction record** confirming `master.key`, CA key, and per-node keys were shredded before drive sanitization.
+- **Exceptions log** — faulty drives that could not be sanitized and were destroyed instead; any sanitize failures and their resolution.
+- **Annual attestation** from the operator confirming the procedures in this guide operated for the prior 12 months, signed by the named security owner.
+
+Cross-reference each decommissioning record to the change ticket that drove the retirement, so drive disposal can be audited against cluster state change.
+
+## 8. Operator Checklist
+
+Use this list to confirm a Spinifex deployment meets MP.L1-3.8.3:
+
+- Asset register enumerates every drive and removable medium by serial, media type, and role.
+- Every volume-producing path uses encryption so that `DeleteVolume` is sanitization.
+- Node-decommissioning runbook exists and matches [§4.5](#45-node-decommissioning-runbook), including the drain → deregister → key-destroy → drive-sanitize sequence.
+- Sanitization method is pre-selected per media type per [§2](#2-sanitization-method-selection); tool availability (hdparm, nvme-cli, vendor ISOs, degausser, shredder or destruction vendor) is verified before a decommissioning begins.
+- `master.key`, CA key, and per-node TLS keys are shredded before the drive holding them is sanitized.
+- Every backup copy of any key is destroyed at the same time as the primary; surviving backup copies are tracked and accounted for.
+- Removable media (USB, tape, optical) is destroyed rather than erased unless the medium is an SED with documented cryptographic erase.
+- Decommissioning records capture method, tool output, verification, final disposition, and operator; certificates of destruction are reconciled against the asset register.
+- Drives awaiting sanitization or awaiting pickup by a destruction vendor are held inside the physical protection boundary ([Physical Security Guide §1](/docs/security/physical-security-guide#1-protected-assets)) with chain of custody logged.
+- System security plan references this guide and names the sanitization tooling, destruction vendor (if any), retention period for records, and the security owner attesting annually.

docs/security/network-connections/README.md

@@ -105,8 +105,8 @@ Spinifex nodes initiate a small, fixed set of outbound connections.
 
 | Destination | Purpose | Protocol | Verification |
 |-------------|---------|----------|--------------|
-| `https://cloud.debian.org/images/cloud/trixie/20260501-2465` | Debian 13 cloud image download | HTTPS | TLS + checksum verification |
-| `https://cloud-images.ubuntu.com/resolute/current` | Ubuntu 26.04 LTS cloud image download | HTTPS | TLS + checksum verification |
+| `https://cloud.debian.org/images/cloud/trixie/` | Debian 13 cloud image download | HTTPS | TLS + checksum verification |
+| `https://cloud-images.ubuntu.com/resolute/` | Ubuntu 26.04 LTS cloud image download | HTTPS | TLS + checksum verification |
 | `https://d2yp8ipz5jfqcw.cloudfront.net` | Alpine image for managed HAProxy load-balancer | HTTPS | TLS + checksum verification |
 | `https://install.mulgadc.com/install` | One-shot install telemetry POST on `spx admin init` / `join`. | HTTPS | TLS |
 

docs/service-interfaces.yaml

@@ -0,0 +1,305 @@
+# Service interface manifest.
+#
+# Source of truth for "which services produce/consume which NATS subjects"
+# and "which e2e suites cover which services". Consumed by
+# tests/e2e/select-suites (Bead 4) to derive targeted suite selection from
+# a PR diff.
+#
+# See docs/development/improvements/e2e-targeted-suite-selection.md.
+#
+# Editing rules
+# -------------
+# - subscribes / publishes: subjects this service receives / sends. Both
+#   lists use the literal subject string, or a "<prefix>.*" wildcard for
+#   pattern subscriptions (e.g. "ebs.snapshot.*").
+# - depends_on: subjects this service requires from peers. Used by the
+#   selector to compute "if X changes, which suites might break".
+# - Subjects must match what is actually referenced in source. The lint
+#   in Bead 5 greps `"(ec2|vpc|ebs|s3|iam|elbv2|spinifex)\.[a-zA-Z._-]+"`
+#   and fails if any subject is absent from this manifest.
+
+version: 1
+
+services:
+
+  awsgw:
+    path: spinifex/services/awsgw
+    # TLS frontend, SigV4 auth. Pure router: HTTP → NATS request/response.
+    # No subscribes; only publishes (via NATSRequest).
+    subscribes: []
+    publishes:
+      - "ec2.*"
+      - "elbv2.*"
+      - "iam.*"
+    depends_on:
+      - "ec2.*"
+      - "elbv2.*"
+      - "iam.*"
+
+  spinifex_daemon:
+    path: spinifex/services/spinifex
+    # Orchestrator. Owns most ec2.* + all elbv2.* + iam event publication.
+    # Translates VPC / EBS / S3 operations into vpc.* / ebs.* / s3.* and
+    # forwards to vpcd / viperblockd / predastore.
+    #
+    # Source layout: spinifex/services/spinifex (entry), spinifex/daemon
+    # (subscribe registration), spinifex/handlers/{ec2,elbv2,iam} (per-
+    # subject handler bodies). Selector treats all four paths as belonging
+    # to this service via the `additional_paths` field.
+    additional_paths:
+      - spinifex/daemon
+      - spinifex/handlers/ec2
+      - spinifex/handlers/elbv2
+      - spinifex/handlers/iam
+    subscribes:
+      # EC2 instance lifecycle.
+      - "ec2.RunInstances"
+      - "ec2.DescribeInstances"
+      - "ec2.DescribeStoppedInstances"
+      - "ec2.DescribeTerminatedInstances"
+      - "ec2.DescribeInstanceAttribute"
+      - "ec2.DescribeInstanceStatus"
+      - "ec2.DescribeInstanceTypes"
+      - "ec2.ModifyInstanceAttribute"
+      - "ec2.ModifyVolume"
+      - "ec2.start"
+      - "ec2.terminate"
+      - "ec2.launch"
+      - "ec2.cmd.*"
+      # EC2 VPC / network ops handled by daemon before delegation to vpcd.
+      - "ec2.CreateVpc"
+      - "ec2.DescribeVpcs"
+      - "ec2.CreateSubnet"
+      - "ec2.CreateInternetGateway"
+      - "ec2.CreateKeyPair"
+      - "ec2.CreateImage"
+      - "ec2.DescribePlacementGroups"
+      # Cluster coordination subjects.
+      - "ec2.ReserveClusterNode"
+      - "ec2.FinalizeClusterInstances"
+      - "spinifex.node.status"
+      - "spinifex.node.vms"
+      - "spinifex.nodes.discover"
+      - "spinifex.cluster.shutdown.progress"
+      - "spinifex.storage.config"
+      # ELBv2 control plane (handlers/elbv2/service_nats.go topic map).
+      - "elbv2.CreateLoadBalancer"
+      - "elbv2.DeleteLoadBalancer"
+      - "elbv2.DescribeLoadBalancers"
+      - "elbv2.CreateTargetGroup"
+      - "elbv2.DeleteTargetGroup"
+      - "elbv2.DescribeTargetGroups"
+      - "elbv2.RegisterTargets"
+      - "elbv2.DeregisterTargets"
+      - "elbv2.DescribeTargetHealth"
+      - "elbv2.CreateListener"
+      - "elbv2.DeleteListener"
+      - "elbv2.DescribeListeners"
+    publishes:
+      # → vpcd
+      - "vpc.create"
+      - "vpc.delete"
+      - "vpc.create-subnet"
+      - "vpc.delete-subnet"
+      - "vpc.create-sg"
+      - "vpc.delete-sg"
+      - "vpc.update-sg"
+      - "vpc.create-port"
+      - "vpc.delete-port"
+      - "vpc.update-port-sgs"
+      - "vpc.igw-attach"
+      - "vpc.igw-detach"
+      - "vpc.add-nat-gateway"
+      - "vpc.delete-nat-gateway"
+      - "vpc.add-nat"
+      - "vpc.delete-nat"
+      - "vpc.dhcp.acquire"
+      - "vpc.dhcp.release"
+      # → viperblockd
+      - "ebs.mount"
+      - "ebs.unmount"
+      - "ebs.delete"
+      - "ebs.sync"
+      - "ebs.snapshot.*"
+      # → predastore (S3 / AMI / keypair store)
+      - "s3.*"
+      # IAM lifecycle event
+      - "iam.account.created"
+    depends_on:
+      - "vpc.*"
+      - "ebs.*"
+      - "s3.*"
+
+  vpcd:
+    path: spinifex/vpcd
+    # OVN-backed VPC / subnet / SG / NAT / IGW data plane.
+    subscribes:
+      - "vpc.create"
+      - "vpc.delete"
+      - "vpc.create-subnet"
+      - "vpc.delete-subnet"
+      - "vpc.create-sg"
+      - "vpc.delete-sg"
+      - "vpc.update-sg"
+      - "vpc.create-port"
+      - "vpc.delete-port"
+      - "vpc.update-port-sgs"
+      - "vpc.igw-attach"
+      - "vpc.igw-detach"
+      - "vpc.add-nat-gateway"
+      - "vpc.delete-nat-gateway"
+      - "vpc.add-nat"
+      - "vpc.delete-nat"
+      - "vpc.dhcp.acquire"
+      - "vpc.dhcp.release"
+    publishes:
+      - "vpc.port-status"
+      - "vpc.dhcp.lease-expired"
+    depends_on: []
+
+  viperblockd:
+    path: spinifex/services/viperblockd
+    # EBS data plane. Backed by the viperblock storage engine in the
+    # mulga/viperblock submodule. Submodule pointer bump → selector forces
+    # "run all" (contract changes can be invisible at the spinifex source
+    # level).
+    subscribes:
+      - "ebs.mount"
+      - "ebs.unmount"
+      - "ebs.delete"
+      - "ebs.sync"
+      - "ebs.snapshot.*"
+    publishes:
+      - "ebs.mount.response"
+      - "ebs.unmount.response"
+    depends_on: []
+
+  predastore:
+    path: spinifex/services/predastore
+    # S3 / AMI / keypair storage. External submodule under mulga/predastore;
+    # this dir contains only the spinifex-side adapter. Submodule pointer
+    # bump → selector forces "run all".
+    subscribes:
+      - "s3.*"
+    publishes: []
+    depends_on: []
+
+  nats:
+    path: spinifex/services/nats
+    # NATS broker bootstrap / cluster config. Not a NATS pub/sub
+    # participant. Any change here is infra-path; selector forces
+    # "run all".
+    subscribes: []
+    publishes: []
+    depends_on: []
+
+  spinifexui:
+    path: spinifex/services/spinifexui
+    # Web UI. Talks to awsgw over HTTP, never NATS. Diffs limited to this
+    # dir do not need any e2e suite to run.
+    subscribes: []
+    publishes: []
+    depends_on: []
+    covers_no_suite: true
+
+
+# Fixture → service mapping. Each Ensure* helper in
+# tests/e2e/harness/fixtures.go exercises this set of services; the
+# Bead 4 selector AST-scans test files for harness.Ensure* call sites
+# and resolves coverage via this block.
+fixtures:
+
+  EnsureKeyPair:
+    services: [spinifex_daemon, predastore]
+
+  EnsureAMI:
+    services: [spinifex_daemon, predastore, viperblockd]
+
+  EnsureDefaultVPC:
+    services: [vpcd, spinifex_daemon]
+
+  EnsureSubnet:
+    services: [vpcd, spinifex_daemon]
+
+  EnsureSG:
+    services: [vpcd, spinifex_daemon]
+
+  EnsureInstance:
+    services: [spinifex_daemon, viperblockd, vpcd]
+
+  EnsureVolume:
+    services: [spinifex_daemon, viperblockd]
+
+  EnsureSnapshot:
+    services: [spinifex_daemon, viperblockd, predastore]
+
+  EnsureNATGateway:
+    services: [vpcd, spinifex_daemon]
+
+  EnsureLoadBalancer:
+    services: [spinifex_daemon, awsgw]
+
+
+suites:
+
+  e2e-cert:
+    path: tests/e2e/cert
+    covers: [awsgw, spinifex_daemon, vpcd, predastore]
+
+  e2e-lb:
+    path: tests/e2e/lb
+    covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+    # Exercises full ELBv2 control plane (handlers/elbv2) + data plane
+    # (LB agent on the system VM). ELBv2 is part of spinifex_daemon in
+    # this manifest; revisit if it splits into a standalone service.
+
+  e2e-single:
+    path: tests/e2e/single
+    covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+    # Full single-node lifecycle: VPC, AMI, keypair, instance, volume,
+    # snapshot, IAM. Currently runs all phases sequentially under one
+    # `TestSingleNode`. Bead 3 (single-fixture-migration) splits this
+    # into per-test fixture-driven coverage so the selector can pick
+    # subtests, not just whole suites.
+
+  e2e-ddil:
+    path: tests/e2e/ddil
+    covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+    # DDIL = degraded / disconnected / intermittent / low-bandwidth.
+    # Exercises the full daemon stack under network failure injection;
+    # any service touching the wire path is in scope.
+
+  # Suites below are bash-only today. Entries activate as the Go ports
+  # land (Bead 2 fixtures unblock new-shape ports for multinode etc.).
+  #
+  # e2e-multinode:
+  #   path: spinifex/tests/e2e/multinode
+  #   covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+  # e2e-reboot:
+  #   path: spinifex/tests/e2e/reboot
+  #   covers: [spinifex_daemon, viperblockd]
+  # e2e-tofu:
+  #   path: spinifex/tests/e2e/tofu
+  #   covers: [awsgw, spinifex_daemon, vpcd]
+  # e2e-bm:
+  #   path: spinifex/tests/e2e/baremetal
+  #   covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+
+
+# Special-case rules consumed by the selector (Bead 4). Hard-coded in
+# the tool, listed here for visibility.
+#
+# infra_paths_force_all:
+#   - spinifex/services/nats/**
+#   - spinifex/tests/e2e/harness/**
+#   - spinifex/tests/e2e/lib/**
+#   - spinifex/.github/workflows/e2e-go.yml
+#   - spinifex/.github/scripts/**
+#   - spinifex/Makefile
+#   - mulga/scripts/tofu-cluster/**
+#
+# submodule_force_all:
+#   - mulga/predastore (submodule pointer)
+#   - mulga/viperblock (submodule pointer)
+#
+# many_services_force_all_threshold: 3   # ≥ N services touched → run all

docs/terraform-workbooks/bastion-private-subnet/main.tf

@@ -96,7 +96,7 @@ data "aws_ami" "ubuntu" {
 
   filter {
     name   = "name"
-    values = ["*ubuntu-24.04*"]
+    values = ["*ubuntu-26.04*", "*ubuntu-24.04*"]
   }
 
   filter {

docs/terraform-workbooks/nginx-alb/main.tf

@@ -102,7 +102,7 @@ data "aws_ami" "ubuntu" {
 
   filter {
     name   = "name"
-    values = ["*ubuntu-24.04*"]
+    values = ["*ubuntu-26.04*", "*ubuntu-24.04*"]
   }
 }
 

docs/terraform-workbooks/nginx-webserver/main.tf

@@ -85,7 +85,7 @@ data "aws_ami" "ubuntu" {
 
   filter {
     name   = "name"
-    values = ["*ubuntu-24.04*"]
+    values = ["*ubuntu-26.04*", "*ubuntu-24.04*"]
   }
 
   filter {

docs/terraform-workbooks/s3-webapp/main.tf

@@ -132,7 +132,7 @@ data "aws_ami" "ubuntu" {
 
   filter {
     name   = "name"
-    values = ["*ubuntu-24.04*"]
+    values = ["*ubuntu-26.04*", "*ubuntu-24.04*"]
   }
 
   filter {

go.mod

@@ -1,20 +1,21 @@
 module github.com/mulgadc/spinifex
 
-go 1.26.3
+go 1.26.4
 
 require (
 	github.com/aws/aws-sdk-go v1.55.8
 	github.com/charmbracelet/bubbles v1.0.0
 	github.com/charmbracelet/bubbletea v1.3.10
 	github.com/charmbracelet/lipgloss v1.1.0
-	github.com/go-chi/chi/v5 v5.2.5
+	github.com/go-chi/chi/v5 v5.3.0
+	github.com/golang-jwt/jwt/v5 v5.3.1
 	github.com/google/uuid v1.6.0
 	github.com/insomniacslk/dhcp v0.0.0-20260407060928-11b94ed970f2
 	github.com/kdomanski/iso9660 v0.4.0
 	github.com/klauspost/cpuid/v2 v2.3.0
-	github.com/mulgadc/predastore v1.4.0
-	github.com/mulgadc/viperblock v1.4.0
-	github.com/nats-io/nats-server/v2 v2.14.0
+	github.com/mulgadc/predastore v1.6.0
+	github.com/mulgadc/viperblock v1.6.0
+	github.com/nats-io/nats-server/v2 v2.14.1
 	github.com/nats-io/nats.go v1.52.0
 	github.com/ovn-kubernetes/libovsdb v0.8.1
 	github.com/pelletier/go-toml/v2 v2.3.1
@@ -23,8 +24,11 @@ require (
 	github.com/spf13/viper v1.21.0
 	github.com/stretchr/testify v1.11.1
 	go.uber.org/goleak v1.3.0
-	golang.org/x/crypto v0.51.0
+	golang.org/x/crypto v0.52.0
+	golang.org/x/sync v0.20.0
+	golang.org/x/sys v0.45.0
 	gopkg.in/ini.v1 v1.67.2
+	gopkg.in/yaml.v3 v3.0.1
 )
 
 require (
@@ -34,6 +38,8 @@ require (
 	github.com/antithesishq/antithesis-sdk-go v0.7.0 // indirect
 	github.com/armon/go-metrics v0.4.1 // indirect
 	github.com/atotto/clipboard v0.1.4 // indirect
+	github.com/aws/aws-sdk-go-v2 v1.41.9 // indirect
+	github.com/aws/smithy-go v1.26.0 // indirect
 	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/boltdb/bolt v1.3.1 // indirect
@@ -78,7 +84,7 @@ require (
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
 	github.com/jmespath/go-jmespath v0.4.0 // indirect
 	github.com/josharian/native v1.1.0 // indirect
-	github.com/klauspost/compress v1.18.5 // indirect
+	github.com/klauspost/compress v1.18.6 // indirect
 	github.com/klauspost/reedsolomon v1.14.0 // indirect
 	github.com/leodido/go-urn v1.4.0 // indirect
 	github.com/lithammer/fuzzysearch v1.1.8 // indirect
@@ -121,9 +127,7 @@ require (
 	go.yaml.in/yaml/v2 v2.4.4 // indirect
 	go.yaml.in/yaml/v3 v3.0.4 // indirect
 	golang.org/x/mod v0.35.0 // indirect
-	golang.org/x/net v0.54.0 // indirect
-	golang.org/x/sync v0.20.0 // indirect
-	golang.org/x/sys v0.44.0 // indirect
+	golang.org/x/net v0.55.0 // indirect
 	golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa // indirect
 	golang.org/x/term v0.43.0 // indirect
 	golang.org/x/text v0.37.0 // indirect
@@ -131,7 +135,6 @@ require (
 	golang.org/x/tools v0.44.0 // indirect
 	golang.org/x/vuln v1.1.4 // indirect
 	google.golang.org/protobuf v1.36.11 // indirect
-	gopkg.in/yaml.v3 v3.0.1 // indirect
 )
 
 tool golang.org/x/vuln/cmd/govulncheck

go.sum

@@ -31,30 +31,30 @@ github.com/atotto/clipboard v0.1.4 h1:EH0zSVneZPSuFR11BlR9YppQTVDbh5+16AmcJi4g1z
 github.com/atotto/clipboard v0.1.4/go.mod h1:ZY9tmq7sm5xIbd9bOK4onWV4S6X0u6GY7Vn0Yu86PYI=
 github.com/aws/aws-sdk-go v1.55.8 h1:JRmEUbU52aJQZ2AjX4q4Wu7t4uZjOu71uyNmaWlUkJQ=
 github.com/aws/aws-sdk-go v1.55.8/go.mod h1:ZkViS9AqA6otK+JBBNH2++sx1sgxrPKcSzPPvQkUtXk=
-github.com/aws/aws-sdk-go-v2 v1.41.7 h1:DWpAJt66FmnnaRIOT/8ASTucrvuDPZASqhhLey6tLY8=
-github.com/aws/aws-sdk-go-v2 v1.41.7/go.mod h1:4LAfZOPHNVNQEckOACQx60Y8pSRjIkNZQz1w92xpMJc=
-github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10 h1:gx1AwW1Iyk9Z9dD9F4akX5gnN3QZwUB20GGKH/I+Rho=
-github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10/go.mod h1:qqY157uZoqm5OXq/amuaBJyC9hgBCBQnsaWnPe905GY=
-github.com/aws/aws-sdk-go-v2/credentials v1.19.16 h1:r3RJBuU7X9ibt8RHbMjWE6y60QbKBiII6wSrXnapxSU=
-github.com/aws/aws-sdk-go-v2/credentials v1.19.16/go.mod h1:6cx7zqDENJDbBIIWX6P8s0h6hqHC8Avbjh9Dseo27ug=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23 h1:GpT/TrnBYuE5gan2cZbTtvP+JlHsutdmlV2YfEyNde0=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23/go.mod h1:xYWD6BS9ywC5bS3sz9Xh04whO/hzK2plt2Zkyrp4JuA=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23 h1:bpd8vxhlQi2r1hiueOw02f/duEPTMK59Q4QMAoTTtTo=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23/go.mod h1:15DfR2nw+CRHIk0tqNyifu3G1YdAOy68RftkhMDDwYk=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.24 h1:OQqn11BtaYv1WLUowvcA30MpzIu8Ti4pcLPIIyoKZrA=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.24/go.mod h1:X5ZJyfwVrWA96GzPmUCWFQaEARPR7gCrpq2E92PJwAE=
-github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9 h1:FLudkZLt5ci0ozzgkVo8BJGwvqNaZbTWb3UcucAateA=
-github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9/go.mod h1:w7wZ/s9qK7c8g4al+UyoF1Sp/Z45UwMGcqIzLWVQHWk=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.15 h1:ieLCO1JxUWuxTZ1cRd0GAaeX7O6cIxnwk7tc1LsQhC4=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.15/go.mod h1:e3IzZvQ3kAWNykvE0Tr0RDZCMFInMvhku3qNpcIQXhM=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.23 h1:pbrxO/kuIwgEsOPLkaHu0O+m4fNgLU8B3vxQ+72jTPw=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.23/go.mod h1:/CMNUqoj46HpS3MNRDEDIwcgEnrtZlKRaHNaHxIFpNA=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.23 h1:03xatSQO4+AM1lTAbnRg5OK528EUg744nW7F73U8DKw=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.23/go.mod h1:M8l3mwgx5ToK7wot2sBBce/ojzgnPzZXUV445gTSyE8=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0 h1:etqBTKY581iwLL/H/S2sVgk3C9lAsTJFeXWFDsDcWOU=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0/go.mod h1:L2dcoOgS2VSgbPLvpak2NyUPsO1TBN7M45Z4H7DlRc4=
-github.com/aws/smithy-go v1.25.1 h1:J8ERsGSU7d+aCmdQur5Txg6bVoYelvQJgtZehD12GkI=
-github.com/aws/smithy-go v1.25.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
+github.com/aws/aws-sdk-go-v2 v1.41.9 h1:/rYeyO2+HrMztAmxAq9++XJtFMqSIpSsNA0yDGALYq4=
+github.com/aws/aws-sdk-go-v2 v1.41.9/go.mod h1:+HsoOEX80qAVUitj1A2DhCNTjmb3edVyuDypb6LNEeo=
+github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.11 h1:h5+3VT69KUBK24grGuuA5saDJTj2IIjLb9au668Fo5I=
+github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.11/go.mod h1:dnakxebH6UwFvcvujL0LVggYQ8nEvBGjU4G/V79Nv94=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.19 h1:yuFzSV1U0aRNYCQGVaTY2zW2M/L93pYHnXnrJUphYhU=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.19/go.mod h1:7y63L1kGzeoDlJaQ3Z578KrnmfBut96JjvJUzGwR+YE=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.25 h1:Uii3frf9ztec/ABM2/FSH9/z7PLzxfpG8h4RpkUFflQ=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.25/go.mod h1:G6kntsA2GorAxDPbap6xgB2F+amSLUF8GJTi7PUoX44=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.25 h1:r1+/l6m+WaUJF9HISEsNOLHSNj5EXYQxK8VX6Cz9NlA=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.25/go.mod h1:cKf+D+NMDK1LndD7BowHbBZPgR9V0/5HubH0PFWvA+c=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.26 h1:A1PmWU2zfkIm9EyFlJncFXL4W4phML+h8KjltUsCvNQ=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.26/go.mod h1:dY4MRzXEizrD4hqtpKvWVGPX7QleSGGVY+EBolo1RmM=
+github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.10 h1:d5/908OJ4bXg8lyjeMPvXetEKqoDoLi5Owy1zNue3yg=
+github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.10/go.mod h1:a57l7Hwh+FWI+we50g5NPJHYUKeJKfXbc4w8SyXu8Ig=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.18 h1:W/EyPFl9A5rXrtoilfwHYEvzHER+K4SpBPtMXi24Mos=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.18/go.mod h1:UG50K+pvd/uy6xExbobg0rjqFBFZe6I3l75EPDZw4tg=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.25 h1:dD3dhHNglpd98gs72my22Ndqi1hqQGllFFg1F+twfxg=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.25/go.mod h1:0yAbjPfd64gG7mj85RW+fMEYdfBgCRZw8g/oWcL1pjc=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.25 h1:2pQEbwf+/6EDbiit/GcBE2K4IUpMZymaA0kOz3xK978=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.25/go.mod h1:KvT6NCcQ0EZ+ZkVRrlBMt04Po3ok23YELEp7WimhLhM=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.102.2 h1:ie4ElCmUKS26pzrZcIk/lmt4yWjAqLLcawstyQCh298=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.102.2/go.mod h1:zjsomFeX5duj+4PlMB+o4JoWTIx+G0XMyzjYrUbQkN0=
+github.com/aws/smithy-go v1.26.0 h1:9ouqbi+NyKP7fV3Te7UElCwdAb6Y8uk7LGwPE5tVe/s=
+github.com/aws/smithy-go v1.26.0/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
 github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
@@ -122,8 +122,8 @@ github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S
 github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
 github.com/gabriel-vasile/mimetype v1.4.13 h1:46nXokslUBsAJE/wMsp5gtO500a4F3Nkz9Ufpk2AcUM=
 github.com/gabriel-vasile/mimetype v1.4.13/go.mod h1:d+9Oxyo1wTzWdyVUPMmXFvp4F9tea18J8ufA774AB3s=
-github.com/go-chi/chi/v5 v5.2.5 h1:Eg4myHZBjyvJmAFjFvWgrqDTXFyOzjj7YIm3L3mu6Ug=
-github.com/go-chi/chi/v5 v5.2.5/go.mod h1:X7Gx4mteadT3eDOMTsXzmI4/rwUpOwBHLpAfupzFJP0=
+github.com/go-chi/chi/v5 v5.3.0 h1:halUjDxhshgXHMrao5bB8eNBXo/rnzwr8m5m36glehM=
+github.com/go-chi/chi/v5 v5.3.0/go.mod h1:R+tYY2hNuVUUjxoPtqUdgBqevM9s9njzkTLutVsOCto=
 github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
 github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
 github.com/go-kit/log v0.1.0/go.mod h1:zbhenjAZHb184qTLMA9ZjW7ThYL0H2mk7Q6pNt4vbaY=
@@ -147,6 +147,8 @@ github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/me
 github.com/go-viper/mapstructure/v2 v2.5.0 h1:vM5IJoUAy3d7zRSVtIwQgBj7BiWtMPfmPEgAXnvj1Ro=
 github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
 github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
+github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY=
+github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=
 github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
 github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
 github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
@@ -229,8 +231,8 @@ github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7V
 github.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM=
 github.com/kdomanski/iso9660 v0.4.0 h1:BPKKdcINz3m0MdjIMwS0wx1nofsOjxOq8TOr45WGHFg=
 github.com/kdomanski/iso9660 v0.4.0/go.mod h1:OxUSupHsO9ceI8lBLPJKWBTphLemjrCQY8LPXM7qSzU=
-github.com/klauspost/compress v1.18.5 h1:/h1gH5Ce+VWNLSWqPzOVn6XBO+vJbCNGvjoaGBFW2IE=
-github.com/klauspost/compress v1.18.5/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
+github.com/klauspost/compress v1.18.6 h1:2jupLlAwFm95+YDR+NwD2MEfFO9d4z4Prjl1XXDjuao=
+github.com/klauspost/compress v1.18.6/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
 github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
 github.com/klauspost/cpuid/v2 v2.0.10/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuObKfj5c0PQa7c=
 github.com/klauspost/cpuid/v2 v2.0.12/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuObKfj5c0PQa7c=
@@ -286,18 +288,18 @@ github.com/muesli/cancelreader v0.2.2 h1:3I4Kt4BQjOR54NavqnDogx/MIoWBFa0StPA8ELU
 github.com/muesli/cancelreader v0.2.2/go.mod h1:3XuTXfFS2VjM+HTLZY9Ak0l6eUKfijIfMUZ4EgX0QYo=
 github.com/muesli/termenv v0.16.0 h1:S5AlUN9dENB57rsbnkPyfdGuWIlkmzJjbFf0Tf5FWUc=
 github.com/muesli/termenv v0.16.0/go.mod h1:ZRfOIKPFDYQoDFF4Olj7/QJbW60Ol/kL1pU3VfY/Cnk=
-github.com/mulgadc/predastore v1.4.0 h1:txPZCaVT1cDpBotx8zHPXHZD3Mz62+n207zXISkK4uU=
-github.com/mulgadc/predastore v1.4.0/go.mod h1:fXIGz7yCKcHh68ybcw7aigdOdzIfyt19h/0diPF+JCU=
-github.com/mulgadc/viperblock v1.4.0 h1:3Yr6lRDnFgxkdMOWu+Tq1b97qWHQDNwkPWmCp38CIy8=
-github.com/mulgadc/viperblock v1.4.0/go.mod h1:hHdfo02iJgCdNktiPAq+x+VQE0ZNYOLJXw0abJ81LeE=
+github.com/mulgadc/predastore v1.6.0 h1:a0xJtrmAvmLlc19B5xUR9upbkl7VlDntpAKnxbkM07E=
+github.com/mulgadc/predastore v1.6.0/go.mod h1:RJh0V0TG27s0q6n0o4CIQFu7CjjPYv4yGnKpIiP9tLs=
+github.com/mulgadc/viperblock v1.6.0 h1:mFx0eVOJoT3WL+X/GPrZOI0ySqerpiCtUW4vPpUxbtI=
+github.com/mulgadc/viperblock v1.6.0/go.mod h1:cwE2FVrvxUD+QaMWnAZeKVnxF8VtTG9zxiUaP1m5oL0=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
 github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
 github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
 github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
-github.com/nats-io/nats-server/v2 v2.14.0 h1:+8q0HrDFotwLLcGH/legOEOnowunhK+aZ4GYBIWpQlM=
-github.com/nats-io/nats-server/v2 v2.14.0/go.mod h1:ImVUUDvfClJbb6cuJQRc1VmgDCXKM5ds0OoiG9MVOKo=
+github.com/nats-io/nats-server/v2 v2.14.1 h1:wXs/a5fw9Hzm3CvuzLxGeIwpjPulSa7gMT3eSuhGkcg=
+github.com/nats-io/nats-server/v2 v2.14.1/go.mod h1:4N17zLpuS7WMbG8T9gsE2B7z9hC9PraPyulVBfpK6nU=
 github.com/nats-io/nats.go v1.52.0 h1:n3avV4VBsCgsdwh71TppsTwtv+QdPs7ntSKM8qJLGsc=
 github.com/nats-io/nats.go v1.52.0/go.mod h1:26HypzazeOkyO3/mqd1zZd53STJN0EjCYF9Uy2ZOBno=
 github.com/nats-io/nkeys v0.4.15 h1:JACV5jRVO9V856KOapQ7x+EY8Jo3qw1vJt/9Jpwzkk4=
@@ -424,8 +426,8 @@ golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnf
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
-golang.org/x/crypto v0.51.0 h1:IBPXwPfKxY7cWQZ38ZCIRPI50YLeevDLlLnyC5wRGTI=
-golang.org/x/crypto v0.51.0/go.mod h1:8AdwkbraGNABw2kOX6YFPs3WM22XqI4EXEd8g+x7Oc8=
+golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988=
+golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc=
 golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa h1:FRnLl4eNAQl8hwxVVC17teOw8kdjVDVAiFMtgUdTSRQ=
 golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa/go.mod h1:zk2irFbV9DP96SEBUUAy67IdHUaZuSnrz1n472HUCLE=
 golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
@@ -442,8 +444,8 @@ golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81R
 golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
 golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
 golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
-golang.org/x/net v0.54.0 h1:2zJIZAxAHV/OHCDTCOHAYehQzLfSXuf/5SoL/Dv6w/w=
-golang.org/x/net v0.54.0/go.mod h1:Sj4oj8jK6XmHpBZU/zWHw3BV3abl4Kvi+Ut7cQcY+cQ=
+golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8=
+golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -484,8 +486,8 @@ golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
-golang.org/x/sys v0.44.0 h1:ildZl3J4uzeKP07r2F++Op7E9B29JRUy+a27EibtBTQ=
-golang.org/x/sys v0.44.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
+golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
+golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
 golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa h1:efT73AJZfAAUV7SOip6pWGkwJDzIGiKBZGVzHYa+ve4=
 golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa/go.mod h1:kHjTxDEnAu6/Nl9lDkzjWpR+bmKfxeiRuSDlsMb70gE=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=

internal/fipsboot/fipsboot.go

@@ -11,7 +11,6 @@ package fipsboot
 
 import (
 	"crypto/fips140"
-	"log/slog"
 )
 
 func init() {
@@ -19,8 +18,4 @@ func init() {
 		panic("fipsboot: FIPS 140-3 mode is not enabled at runtime — refusing to start. " +
 			"Build with GOFIPS140=v1.0.0 and do not set GODEBUG=fips140=off.")
 	}
-	slog.Info("FIPS 140-3 mode enabled",
-		"module", "Go Cryptographic Module v1.0.0",
-		"cmvp_cert", "5247",
-	)
 }

internal/tlsconfig/curves.go

@@ -0,0 +1,37 @@
+// Package tlsconfig pins the cluster-wide TLS 1.3 curve preference list.
+//
+// The list is hardcoded because mixed classification on a single cluster is
+// not supported and the certified FIPS build is itself the per-deployment
+// choice. A typo in a config string would silently downgrade where a missing
+// constant cannot.
+package tlsconfig
+
+import "crypto/tls"
+
+// Curves is the fixed TLS 1.3 curve allowlist:
+//
+//   - X25519MLKEM768     — Cat 3 ML-KEM hybrid, wins under Go 1.26 stdlib filter
+//   - SecP384r1MLKEM1024 — Cat 5 ML-KEM hybrid, fallback for peers that prefer it
+//   - X25519             — classical TLS 1.3 fallback for SDKs without ML-KEM (aws CLI v2 today)
+//   - CurveP384          — classical TLS 1.3 fallback (Java/.NET enterprise SDKs)
+//
+// PQ-capable peers (browsers since 2024, Go aws-sdk v2 with Go 1.24+) always
+// negotiate the hybrid first; classical fallback only triggers for peers that
+// cannot do ML-KEM. Those connections still get TLS 1.3 + AES-256-GCM but
+// remain HNDL-exposed — accepted trade-off until AWS CLI v2 / bundled
+// OpenSSL ships ML-KEM (currently lagging upstream OpenSSL 3.5).
+//
+// Weak/niche entries from the Go stdlib default (SecP256r1MLKEM768, P-256,
+// P-521) are intentionally excluded so they cannot be advertised.
+//
+// Go 1.26 stdlib treats tls.Config.CurvePreferences as an allowlist filter
+// applied to the stdlib default order, not as a user-supplied order.
+//
+// Callers must not mutate this slice. Assign it directly to
+// tls.Config.CurvePreferences; do not append to it.
+var Curves = []tls.CurveID{
+	tls.X25519MLKEM768,
+	tls.SecP384r1MLKEM1024,
+	tls.X25519,
+	tls.CurveP384,
+}

internal/tlsconfig/curves_test.go

@@ -0,0 +1,169 @@
+package tlsconfig_test
+
+import (
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/tls"
+	"crypto/x509"
+	"crypto/x509/pkix"
+	"encoding/pem"
+	"io"
+	"math/big"
+	"net"
+	"net/http"
+	"net/http/httptest"
+	"slices"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/mulgadc/spinifex/internal/tlsconfig"
+)
+
+func TestCurves_OnlyApprovedEntries(t *testing.T) {
+	want := []tls.CurveID{
+		tls.X25519MLKEM768,
+		tls.SecP384r1MLKEM1024,
+		tls.X25519,
+		tls.CurveP384,
+	}
+	if !slices.Equal(tlsconfig.Curves, want) {
+		t.Errorf("Curves = %v, want %v", tlsconfig.Curves, want)
+	}
+}
+
+func TestCurves_ExcludesWeakPrimitives(t *testing.T) {
+	forbidden := []tls.CurveID{
+		tls.SecP256r1MLKEM768, // weakest ML-KEM hybrid (P-256 component)
+		tls.CurveP256,         // not CNSA 2.0
+		tls.CurveP521,         // niche, no current peers
+	}
+	for _, c := range forbidden {
+		if slices.Contains(tlsconfig.Curves, c) {
+			t.Errorf("Curves includes forbidden curve %v", c)
+		}
+	}
+}
+
+// TestIntegration_NegotiatesPQHybrid spins up an httptest TLS server
+// configured with Curves + TLS 1.3 minimum, dials it, and asserts a PQ
+// hybrid is negotiated at TLS 1.3. Per Go 1.26 stdlib filter behavior the
+// winner is X25519MLKEM768; the assertion checks for any approved hybrid
+// to remain stable across stdlib reorderings of our 4 entries.
+func TestIntegration_NegotiatesPQHybrid(t *testing.T) {
+	cert := selfSignedCert(t)
+	srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = io.WriteString(w, "ok")
+	}))
+	srv.TLS = &tls.Config{
+		Certificates:     []tls.Certificate{cert},
+		MinVersion:       tls.VersionTLS13,
+		CurvePreferences: tlsconfig.Curves,
+	}
+	srv.StartTLS()
+	defer srv.Close()
+
+	pool := x509.NewCertPool()
+	pool.AddCert(srv.Certificate())
+
+	client := &http.Client{
+		Timeout: 5 * time.Second,
+		Transport: &http.Transport{
+			TLSClientConfig: &tls.Config{
+				RootCAs:          pool,
+				CurvePreferences: tlsconfig.Curves,
+			},
+		},
+	}
+	resp, err := client.Get(srv.URL)
+	if err != nil {
+		t.Fatalf("https get: %v", err)
+	}
+	defer resp.Body.Close()
+
+	state := resp.TLS
+	if state == nil {
+		t.Fatal("response has no TLS state")
+	}
+	if state.Version != tls.VersionTLS13 {
+		t.Errorf("negotiated version = 0x%x, want TLS 1.3 (0x%x)", state.Version, tls.VersionTLS13)
+	}
+	pqHybrids := []tls.CurveID{tls.X25519MLKEM768, tls.SecP384r1MLKEM1024}
+	if !slices.Contains(pqHybrids, state.CurveID) {
+		t.Errorf("negotiated curve = %v, want one of %v (PQ hybrid)", state.CurveID, pqHybrids)
+	}
+}
+
+// TestIntegration_TLS12ClientRejected confirms MinVersion: tls.VersionTLS13
+// is enforced — a TLS 1.2-pinned client gets a handshake error rather than
+// silently downgrading. This is the negative test that closes the HNDL gap
+// on external surfaces.
+func TestIntegration_TLS12ClientRejected(t *testing.T) {
+	cert := selfSignedCert(t)
+	srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = io.WriteString(w, "ok")
+	}))
+	srv.TLS = &tls.Config{
+		Certificates:     []tls.Certificate{cert},
+		MinVersion:       tls.VersionTLS13,
+		CurvePreferences: tlsconfig.Curves,
+	}
+	srv.StartTLS()
+	defer srv.Close()
+
+	pool := x509.NewCertPool()
+	pool.AddCert(srv.Certificate())
+
+	client := &http.Client{
+		Timeout: 5 * time.Second,
+		Transport: &http.Transport{
+			TLSClientConfig: &tls.Config{
+				RootCAs:    pool,
+				MinVersion: tls.VersionTLS12,
+				MaxVersion: tls.VersionTLS12,
+			},
+		},
+	}
+	_, err := client.Get(srv.URL)
+	if err == nil {
+		t.Fatal("TLS 1.2-pinned client succeeded; MinVersion not enforced")
+	}
+	if !strings.Contains(err.Error(), "protocol version") &&
+		!strings.Contains(err.Error(), "tls:") {
+		t.Errorf("expected protocol-version handshake error, got: %v", err)
+	}
+}
+
+func selfSignedCert(t *testing.T) tls.Certificate {
+	t.Helper()
+	key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+	if err != nil {
+		t.Fatalf("generate key: %v", err)
+	}
+	tmpl := &x509.Certificate{
+		SerialNumber: big.NewInt(1),
+		Subject:      pkix.Name{CommonName: "tlsconfig-test"},
+		NotBefore:    time.Now().Add(-time.Hour),
+		NotAfter:     time.Now().Add(time.Hour),
+		KeyUsage:     x509.KeyUsageDigitalSignature,
+		ExtKeyUsage:  []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
+		DNSNames:     []string{"localhost"},
+		IPAddresses:  []net.IP{net.ParseIP("127.0.0.1"), net.ParseIP("::1")},
+	}
+	der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key)
+	if err != nil {
+		t.Fatalf("create cert: %v", err)
+	}
+	keyDER, err := x509.MarshalECPrivateKey(key)
+	if err != nil {
+		t.Fatalf("marshal key: %v", err)
+	}
+	certPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
+	keyPEM := pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER})
+	pair, err := tls.X509KeyPair(certPEM, keyPEM)
+	if err != nil {
+		t.Fatalf("X509KeyPair: %v", err)
+	}
+	return pair
+}

scripts/ansible/.gitignore

@@ -0,0 +1,2 @@
+.creds/
+*.retry

scripts/ansible/README.md

@@ -5,7 +5,6 @@ Lives alongside the existing `scripts/dev-*.sh` and `scripts/reset-dev-env.sh`;
 neither set is removed. Decide which to keep after parity testing.
 
 Plan: `docs/development/improvements/ansible-dev-lifecycle.md` (mulga top-level).
-Bead: `mulga-siv-9`.
 
 ## Prerequisites
 
@@ -32,6 +31,12 @@ inventory and roles relative to CWD.
 | `playbooks/dev-install.yml` | Build + install + init + smoketest on clean box | `dev-install.sh` |
 | `playbooks/dev-reset.yml` | Capture settings → teardown → build → install → init → smoketest | `reset-dev-env.sh` (full) |
 | `playbooks/dev-deploy.yml` | Rebuild + swap binaries/microvm artifacts + restart (no setup.sh, no smoketest) | `make deploy` |
+| `playbooks/dev-status.yml` | Read-only health snapshot (units, ports, OVN/OVS, gateway, counts, drift) | none |
+| `playbooks/dev-logs.yml` | Dump journald + `/var/log/spinifex/*` + status/networking/versions into a tarball | none |
+| `playbooks/dev-snapshot.yml` | Quiesce spinifex+OVN+OVS, tar state paths to a named bundle, restart | none |
+| `playbooks/dev-restore.yml` | Stop services, wipe + untar a named bundle, refresh CA, restart | none |
+| `playbooks/dev-version.yml` | Build-vs-install drift report (spx, plugin, microvm) + sub-repo HEADs | none |
+| `playbooks/dev-vpc.yml` | Idempotent dev VPC fixture (VPC + public subnet + IGW + default route); `-e dev_vpc_state=absent` tears down | partial overlap with `scripts/iac/examples/*` |
 
 Upcoming (not yet implemented):
 
@@ -46,6 +51,12 @@ ansible-playbook playbooks/dev-teardown.yml
 ansible-playbook playbooks/dev-install.yml
 ansible-playbook playbooks/dev-reset.yml
 ansible-playbook playbooks/dev-deploy.yml
+ansible-playbook playbooks/dev-status.yml
+ansible-playbook playbooks/dev-logs.yml
+ansible-playbook playbooks/dev-snapshot.yml -e snapshot_name=before-merge
+ansible-playbook playbooks/dev-restore.yml  -e snapshot_name=before-merge
+ansible-playbook playbooks/dev-version.yml
+ansible-playbook playbooks/dev-vpc.yml
 ```
 
 Or via `make` (from `spinifex/`):
@@ -56,6 +67,13 @@ make ansible-dev-teardown
 make ansible-dev-install
 make ansible-dev-reset
 make ansible-dev-deploy
+make ansible-dev-status
+make ansible-dev-logs
+make ansible-dev-snapshot ANSIBLE_EXTRA='-e snapshot_name=before-merge'
+make ansible-dev-restore  ANSIBLE_EXTRA='-e snapshot_name=before-merge'
+make ansible-dev-version
+make ansible-dev-vpc
+make ansible-dev-vpc ANSIBLE_EXTRA='-e dev_vpc_state=absent'
 ```
 
 ### When to use which
@@ -64,6 +82,11 @@ make ansible-dev-deploy
 - Iterate on Go code, microvm initramfs, lb-agent → `ansible-dev-deploy` (fast)
 - Changed systemd units, helper scripts, logrotate, setup.sh → `ansible-dev-reset` (slow, full rebuild)
 - Need a clean slate without reinstall → `ansible-dev-teardown`
+- "Is my dev box healthy?" → `ansible-dev-status` (read-only, never mutates)
+- Filing a bug / capturing the state of a broken box → `ansible-dev-logs` (writes `/tmp/spinifex-logs-<host>-<ts>.tar.gz`; override `-e logs_since=10min` or `-e logs_include_dmesg=false`)
+- Checkpoint before a risky op (branch switch, viperblock surgery, ovn restart cycle) → `ansible-dev-snapshot ANSIBLE_EXTRA='-e snapshot_name=<name>'`; roll back via `ansible-dev-restore` instead of a full `dev-reset`. Bundles live in `/var/lib/spinifex-snapshots/<name>.tar.gz` (root-owned, survives `dev-teardown`).
+- "Do I need to `ansible-dev-deploy`?" → `ansible-dev-version` (drift-only check; faster than `dev-status`)
+- Want a known-good VPC/subnet for manual EC2 testing without writing tofu → `ansible-dev-vpc` (smoketest will then find and use it automatically). Tear down with `ANSIBLE_EXTRA='-e dev_vpc_state=absent'`.
 
 ## Variable overrides
 
@@ -83,14 +106,117 @@ Useful overrides:
 | `teardown_process_wait_seconds` | `30` | Grace period for processes to exit |
 | `spinifex_region` | `ap-southeast-2` | Region passed to `spx admin init` |
 | `spinifex_az` | `{{ region }}a` | AZ passed to `spx admin init` |
-| `spinifex_external_mode` | `pool` | `pool` \| `nat` external networking |
+| `spinifex_external_mode` | `pool` | External networking mode — `pool` or empty (NAT removed) |
+| `spinifex_external_no_external` | `false` | `true` → emit `--no-external` and skip external networking |
+| `spinifex_external_pool_start` | `""` | **Required** for `pool` mode — first IP of the static WAN pool |
+| `spinifex_external_pool_end` | `""` | **Required** for `pool` mode — last IP of the static WAN pool |
+| `spinifex_external_gateway` | `""` | WAN gateway (auto-detected when blank) |
+| `spinifex_external_prefix_len` | `""` | WAN prefix length (auto-detected when blank) |
+| `spinifex_external_gateway_ip` | `""` | OVN router external-IP override (`--gateway-ip`) |
+
+`pool` mode is mandatory unless you set `spinifex_external_no_external=true`.
+`spx admin init` rejects the run without a `--external-pool` range, so the
+init role asserts `spinifex_external_pool_start/_end` upfront. Either persist
+them in `vars/defaults.yml`, or pass them at invocation:
+
+Direct `ansible-playbook` (use `-e` here — this is ansible's flag):
+
+```
+ansible-playbook playbooks/dev-install.yml \
+  -e spinifex_external_pool_start=192.168.1.150 \
+  -e spinifex_external_pool_end=192.168.1.250
+```
+
+Via `make` (use `EXTRA_VARS=` — make eats its own `-e`):
+
+```
+make ansible-dev-install \
+  EXTRA_VARS="spinifex_external_pool_start=192.168.1.150 spinifex_external_pool_end=192.168.1.250"
+```
 
 `dev-reset.yml` automatically captures `region`, `az`, `external_mode`,
-pool range, gateway, prefix-len, nat gateway_ip from the existing
+pool range, gateway, prefix-len and `gateway_ip` from the existing
 `/etc/spinifex/spinifex.toml` before teardown and replays them into
-`init`. Use `-e` to override.
+`init`. Legacy `external_mode = "nat"` is coerced to `"pool"` with a warn.
+Use `-e` to override.
+
+## Helper scripts
+
+`scripts/aws-as-spinifex.sh` and `scripts/spx-as-spinifex.sh` wrap `aws`
+and `spx` invocations in `sg spinifex -c '...'` so they run with the
+`spinifex` supplementary group active. Both helpers preset
+`AWS_PROFILE=spinifex`. All AWS-calling tasks in the roles use these
+via `spinifex_aws_cli` / `spinifex_spx_cli` from `vars/defaults.yml`.
+
+Background: `/etc/spinifex/ca.pem` lives under `drwxr-x--- root:spinifex`,
+so the AWS CLI needs `spinifex` group membership to read the CA bundle.
+Ansible's `become_user` is a no-op when the target user equals the
+calling user (no `sudo -u` invoked → no `initgroups(3)` → supplementary
+groups inherited from the operator's shell). On a fresh login the
+operator's shell would have `spinifex` (added by `setup.sh`); on shells
+that predate the group add it does not. The helpers sidestep this
+entirely.
 
 ## Leak catalog
 
 `roles/teardown/README.md` tracks every known state source. When teardown
 misses something, add it there and in `roles/teardown/tasks/main.yml`.
+
+## Multi-node cluster bootstrap
+
+`playbooks/cluster-bootstrap.yml` provisions a 2+ node Spinifex cluster
+across a long-running tofu env (`env1`, `env2`, `bryn`, etc). Inventory
+is generated dynamically from `scripts/tofu-cluster/envs/<env>.tfvars`
+via `inventory/tofu.py` — first node in `nodes = [...]` is the init host,
+the rest are joins.
+
+```bash
+make ansible-cluster-bootstrap ENV=env1 POOL=192.168.1.150-192.168.1.159
+```
+
+`POOL` is optional; omit for DHCP fallback. Equivalent without make:
+
+```bash
+cd scripts/ansible
+CLUSTER_ENV=env1 ansible-playbook -i inventory/tofu.py \
+    playbooks/cluster-bootstrap.yml \
+    -e cluster_env=env1 \
+    -e cluster_external_pool=192.168.1.150-192.168.1.159
+```
+
+Plays sequence:
+
+1. **runner (localhost)** — `build` role produces `/tmp/spinifex-local.tar.gz`.
+2. **all env hosts (parallel)** — copies tarball, stops any dev-layout
+   services (`~/spinifex/`), runs `preflight` + `install` roles. `setup.sh`
+   creates `/etc/spinifex`, the `spinifex` user/group, systemd units, and
+   installs `/usr/local/bin/spx`.
+3. **init host only** — `cluster-init` role runs `spx admin init` in the
+   background, waits for the formation server, captures the join token.
+   CA is then copied into `/usr/local/share/ca-certificates/spinifex-ca.crt`
+   **before** `spinifex.target` starts (required — predastore enforces
+   strict TLS verify against the system trust store).
+4. **join hosts (parallel)** — `cluster-join` role runs `spx admin join`
+   against the formation server, then installs CA, then starts
+   `spinifex.target`.
+5. **all hosts** — polls `https://<ip>:4432/health` until `awsgw=ok`.
+6. **init host only** — `cluster-post` role imports the SSH key + Ubuntu
+   AMI (both idempotent: skip if already present).
+
+Re-running the playbook against an already-bootstrapped env is safe —
+`setup.sh`, `spx admin init --force`, CA copy, key/AMI import are all
+idempotent.
+
+### Migration from `bootstrap.sh`
+
+`scripts/tofu-cluster/bootstrap.sh` ran services under `~/spinifex/` as
+`tf-user` via `start-dev.sh` with no systemd supervision, and installed
+the cluster CA *after* services started — predastore would crash with
+`x509: certificate signed by unknown authority` on every first boot.
+`cluster-bootstrap.yml` replaces that flow with production-layout
+(`/etc/spinifex/` + systemd) and correct CA-before-services ordering.
+The first run on an env previously bootstrapped by `bootstrap.sh` will
+stop the dev-layout services, install the prod layout, and re-init the
+cluster from scratch via `spx admin init --force`.
+
+`bootstrap.sh` stays in-tree for CI and pseudo-multinode envs.

scripts/ansible/ansible.cfg

@@ -5,7 +5,13 @@ stdout_callback = ansible.builtin.default
 host_key_checking = False
 retry_files_enabled = False
 interpreter_python = auto_silent
-forks = 1
+forks = 10
+
+[inventory]
+# inventory/tofu.py is an executable script inventory used by
+# cluster-bootstrap.yml — opted into via `-i inventory/tofu.py` on the
+# command line; not default to keep dev-* playbooks pointing at localhost.
+enable_plugins = host_list, script, auto, yaml, ini, toml
 
 [callback_default]
 result_format = yaml

scripts/ansible/inventory/tofu.py

@@ -0,0 +1,170 @@
+#!/usr/bin/env python3
+"""Dynamic ansible inventory generator from tofu-cluster tfvars.
+
+Reads scripts/tofu-cluster/envs/<env>.tfvars (HCL subset) and emits ansible
+JSON inventory. First node in the tfvars `nodes = [...]` list is the init
+host (forms the cluster); the rest are join hosts.
+
+Usage:
+    inventory/tofu.py --list --env env1
+    inventory/tofu.py --host <ip> --env env1
+
+Selecting the env:
+    CLUSTER_ENV=env1 inventory/tofu.py --list
+    inventory/tofu.py --list --env env1
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import sys
+from pathlib import Path
+
+
+def _repo_root() -> Path:
+    here = Path(__file__).resolve()
+    for parent in here.parents:
+        if (parent / "go.work").exists() or (parent / ".gitmodules").exists():
+            return parent
+    raise SystemExit(f"could not locate mulga repo root from {here}")
+
+
+def _tfvars_path(env: str) -> Path:
+    root = _repo_root()
+    candidate = root / "scripts" / "tofu-cluster" / "envs" / f"{env}.tfvars"
+    if not candidate.exists():
+        raise SystemExit(f"tfvars file not found: {candidate}")
+    return candidate
+
+
+_STRING_RE = re.compile(r'^\s*([a-zA-Z_][a-zA-Z0-9_]*)\s*=\s*"([^"]*)"\s*$')
+_NUMBER_RE = re.compile(r'^\s*([a-zA-Z_][a-zA-Z0-9_]*)\s*=\s*([0-9]+)\s*$')
+_NODE_FIELD_RE = re.compile(r'(\w+)\s*=\s*"([^"]*)"')
+
+
+def _parse_tfvars(path: Path) -> dict:
+    """Minimal tfvars parser — handles the subset used by envs/*.tfvars.
+
+    Supports:
+      key = "string"
+      key = 12345
+      nodes = [
+        { name = "x", wan_ip = "1.2.3.4", ... },
+        ...
+      ]
+    """
+    out: dict = {"nodes": []}
+    text = path.read_text()
+
+    # Strip line comments
+    text = re.sub(r"#.*", "", text)
+
+    # Pull the nodes block out first, then process remaining lines.
+    nodes_match = re.search(r"nodes\s*=\s*\[(.+?)\]\s*$", text, re.DOTALL | re.MULTILINE)
+    if nodes_match:
+        nodes_body = nodes_match.group(1)
+        for entry in re.finditer(r"\{([^}]+)\}", nodes_body):
+            fields = dict(_NODE_FIELD_RE.findall(entry.group(1)))
+            if "name" in fields and "wan_ip" in fields:
+                out["nodes"].append(fields)
+        text = text.replace(nodes_match.group(0), "")
+
+    for line in text.splitlines():
+        if m := _STRING_RE.match(line):
+            out[m.group(1)] = m.group(2)
+        elif m := _NUMBER_RE.match(line):
+            out[m.group(1)] = int(m.group(2))
+
+    return out
+
+
+def _resolve_ssh_key(value: str | None) -> str:
+    if not value:
+        return os.path.expanduser("~/.ssh/tf-user-ap-southeast-2")
+    return os.path.expanduser(value)
+
+
+def _build_inventory(env: str) -> dict:
+    data = _parse_tfvars(_tfvars_path(env))
+    nodes = data.get("nodes") or []
+    if not nodes:
+        raise SystemExit(f"{env}.tfvars has no nodes")
+
+    ssh_user = data.get("ssh_user", "tf-user")
+    ssh_key = _resolve_ssh_key(data.get("ssh_public_key_path", "").replace(".pub", ""))
+
+    hostvars: dict[str, dict] = {}
+    all_hosts: list[str] = []
+    init_hosts: list[str] = []
+    join_hosts: list[str] = []
+
+    for idx, node in enumerate(nodes):
+        ip = node["wan_ip"]
+        role = "init" if idx == 0 else "join"
+        hostvars[ip] = {
+            "ansible_host": ip,
+            "ansible_user": ssh_user,
+            "ansible_ssh_private_key_file": ssh_key,
+            "ansible_ssh_common_args": "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR",
+            "spinifex_node_name": node["name"],
+            "spinifex_node_role": role,
+            "spinifex_node_index": idx,
+            "spinifex_wan_ip": ip,
+        }
+        all_hosts.append(ip)
+        (init_hosts if role == "init" else join_hosts).append(ip)
+
+    # Group-level vars derived from tfvars (operator can override with -e).
+    cluster_init_host = init_hosts[0]
+    group_vars = {
+        "cluster_env": env,
+        "cluster_init_host": cluster_init_host,
+        "cluster_init_ip": cluster_init_host,
+        "spinifex_node_count": len(nodes),
+        "spinifex_region": data.get("region", "ap-southeast-2"),
+        "spinifex_az": data.get("az", "ap-southeast-2a"),
+    }
+    if cidr := data.get("env_cidr"):
+        group_vars["cluster_env_cidr"] = cidr
+
+    inventory = {
+        "_meta": {"hostvars": hostvars},
+        env: {"hosts": all_hosts, "vars": group_vars},
+        "cluster": {"children": [env]},
+        "init": {"hosts": init_hosts},
+        "join": {"hosts": join_hosts},
+    }
+    return inventory
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--list", action="store_true", help="emit full inventory")
+    parser.add_argument("--host", help="emit hostvars for a single host (ignored — _meta carries them)")
+    parser.add_argument(
+        "--env",
+        default=os.environ.get("CLUSTER_ENV"),
+        help="tofu env name (matches scripts/tofu-cluster/envs/<env>.tfvars); also via CLUSTER_ENV env var",
+    )
+    args = parser.parse_args()
+
+    if not args.env:
+        sys.stderr.write("error: --env required (or CLUSTER_ENV)\n")
+        return 64
+
+    if args.host:
+        json.dump({}, sys.stdout)
+        sys.stdout.write("\n")
+        return 0
+
+    inventory = _build_inventory(args.env)
+    json.dump(inventory, sys.stdout, indent=2)
+    sys.stdout.write("\n")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

scripts/ansible/playbooks/cluster-bootstrap.yml

@@ -0,0 +1,330 @@
+---
+# cluster-bootstrap.yml — multi-node Spinifex cluster bootstrap against a
+# tofu-provisioned env. Inventory comes from inventory/tofu.py which parses
+# scripts/tofu-cluster/envs/<env>.tfvars.
+#
+# Usage:
+#   cd spinifex/scripts/ansible
+#   ansible-playbook -i inventory/tofu.py playbooks/cluster-bootstrap.yml \
+#     -e cluster_env=env1 -e cluster_external_pool=192.168.1.150-192.168.1.159
+#
+# Or via the make wrapper:
+#   make ansible-cluster-bootstrap ENV=env1 POOL=192.168.1.150-192.168.1.159
+#
+# Behaviour:
+#   - all hosts: preflight + install (setup.sh) — production layout
+#   - init host: spx admin init (backgrounded) + CA install + spinifex.target
+#   - join hosts: spx admin join + CA install + spinifex.target (parallel)
+#   - init host: import ssh key + ubuntu AMI (idempotent)
+
+- name: Cluster bootstrap — build artifact once on runner
+  hosts: localhost
+  gather_facts: true
+  vars_files:
+    - ../vars/defaults.yml
+  tasks:
+    - name: assert cluster_env supplied
+      ansible.builtin.assert:
+        that: cluster_env | default('') | length > 0
+        fail_msg: "Pass -e cluster_env=<env> (matches scripts/tofu-cluster/envs/<env>.tfvars)"
+  roles:
+    - build
+
+- name: Cluster bootstrap — install (all nodes, parallel)
+  hosts: "{{ cluster_env }}"
+  gather_facts: true
+  serial: 100%
+  vars:
+    install_run_ovn: false   # we drive setup-ovn ourselves with role-aware args
+    cluster_bootstrap_purge: true   # wipe stale single-node predastore.toml + raft data
+  vars_files:
+    - ../vars/defaults.yml
+  pre_tasks:
+    - name: copy build tarball to each node
+      ansible.builtin.copy:
+        src: "{{ build_tarball_path }}"
+        dest: "{{ build_tarball_path }}"
+        mode: "0644"
+
+    - name: stop dev-layout services if running (migration from ~/spinifex/)
+      become: true
+      # Bracket trick on the first letter prevents the pattern literal in the
+      # shell command line from matching the running shell itself (pkill -f
+      # against literal patterns is otherwise self-killing — rc=-9).
+      # `spx [a]dmin` catches lingering formation servers from prior bootstrap
+      # attempts that own port 4432 and steal joins from the new init.
+      ansible.builtin.shell:
+        cmd: |
+          pkill -9 -f "[b]in/spx service" 2>/dev/null || true
+          pkill -9 -f "spx [a]dmin" 2>/dev/null || true
+          pkill -9 -f "spinifex-[u]i" 2>/dev/null || true
+          pkill -9 -f "[n]ats-server" 2>/dev/null || true
+          pkill -9 -f "[p]redastore" 2>/dev/null || true
+          pkill -9 -f "[v]iperblock" 2>/dev/null || true
+          sleep 2
+          rm -rf "{{ ansible_env.HOME }}/spinifex" 2>/dev/null || true
+          exit 0
+      changed_when: false
+  roles:
+    - install
+
+# Wipe OVN/OVS state before setup-ovn runs. Without this, re-bootstrapping a
+# previously-built cluster leaves stale chassis rows in the SB DB, stale
+# external_ids on Open_vSwitch, stale br-int flows + ports, stale
+# systemd-networkd unit files, and stale ovs-monitor-ipsec strongSwan certs.
+# Symptom: ovn-controller flows reference dead encap endpoints; IPsec SAs
+# fail AUTHENTICATION_FAILED; new joins shadow ghosts under the same
+# node-name.
+- name: Cluster bootstrap — OVN/OVS state reset (all nodes, parallel)
+  hosts: "{{ cluster_env }}"
+  gather_facts: false
+  serial: 100%
+  vars_files:
+    - ../vars/defaults.yml
+  tasks:
+    - name: reset — stop OVN/OVS services
+      become: true
+      ansible.builtin.systemd:
+        name: "{{ item }}"
+        state: stopped
+      loop:
+        - ovn-controller
+        - ovn-central
+        - openvswitch-ipsec
+        - openvswitch-switch
+      failed_when: false
+
+    - name: reset — delete OVN NB/SB databases
+      become: true
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: absent
+      loop:
+        - "{{ ovn_nb_db }}"
+        - "{{ ovn_sb_db }}"
+
+    - name: reset — delete OVS conf.db + system-id (force fresh ovsdb)
+      become: true
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: absent
+      loop:
+        - /etc/openvswitch/conf.db
+        - /etc/openvswitch/.conf.db.~lock~
+        - /etc/openvswitch/system-id.conf
+
+    - name: reset — drop spinifex-managed systemd-networkd unit files
+      become: true
+      ansible.builtin.shell: |
+        rm -f /etc/systemd/network/14-spinifex-*.netdev \
+              /etc/systemd/network/15-spinifex-*.netdev \
+              /etc/systemd/network/15-spinifex-*.network \
+              /etc/systemd/network/16-spinifex-*.network
+        networkctl reload 2>/dev/null || true
+      changed_when: true
+
+    - name: reset — enumerate stale spinifex veth/tap netdevs
+      ansible.builtin.shell: |
+        ip -o link show | awk -F': ' '{print $2}' | cut -d'@' -f1 | \
+          grep -E '^(veth-wan-br|veth-wan-ovs|spx-ext-|spx-tap-|spx-)' || true
+      register: reset_links
+      changed_when: false
+
+    - name: reset — delete stale netdevs
+      become: true
+      ansible.builtin.command: "ip link delete {{ item }}"
+      loop: "{{ reset_links.stdout_lines }}"
+      changed_when: true
+      failed_when: false
+
+    - name: reset — clear ovs-monitor-ipsec strongSwan state
+      become: true
+      ansible.builtin.shell: |
+        rm -rf /var/lib/openvswitch/pki 2>/dev/null || true
+        rm -f /etc/ipsec.d/certs/ovs-* /etc/ipsec.d/private/ovs-* 2>/dev/null || true
+        rm -f /etc/ipsec.conf.d/ovs-* 2>/dev/null || true
+      changed_when: true
+
+    - name: reset — start openvswitch-switch with fresh conf.db
+      become: true
+      ansible.builtin.systemd:
+        name: openvswitch-switch
+        state: started
+        daemon_reload: true
+
+- name: Cluster bootstrap — OVN central (init host only)
+  hosts: init
+  gather_facts: false
+  vars_files:
+    - ../vars/defaults.yml
+  tasks:
+    - name: setup-ovn --management on init host
+      become: true
+      ansible.builtin.command:
+        cmd: >-
+          /usr/local/share/spinifex/setup-ovn.sh
+          --management
+          --encap-ip={{ spinifex_wan_ip }}
+          --node-name={{ hostvars[inventory_hostname]['spinifex_node_name'] }}
+      changed_when: true
+
+- name: Cluster bootstrap — OVN chassis (join hosts, parallel)
+  hosts: join
+  gather_facts: false
+  vars_files:
+    - ../vars/defaults.yml
+  tasks:
+    - name: setup-ovn --ovn-remote on join hosts
+      become: true
+      ansible.builtin.command:
+        cmd: >-
+          /usr/local/share/spinifex/setup-ovn.sh
+          --ovn-remote=tcp:{{ cluster_init_host }}:6642
+          --encap-ip={{ spinifex_wan_ip }}
+          --node-name={{ hostvars[inventory_hostname]['spinifex_node_name'] }}
+      changed_when: true
+
+- name: Cluster bootstrap — init host (start formation server)
+  hosts: init
+  gather_facts: false
+  vars_files:
+    - ../vars/defaults.yml
+  roles:
+    - cluster-init
+
+- name: Cluster bootstrap — join hosts run spx admin join (parallel)
+  hosts: join
+  gather_facts: false
+  serial: 100%
+  vars_files:
+    - ../vars/defaults.yml
+  roles:
+    - cluster-join
+
+- name: Cluster bootstrap — wait for init to finish writing multi-node configs
+  hosts: init
+  gather_facts: false
+  vars_files:
+    - ../vars/defaults.yml
+  tasks:
+    - name: wait for spx admin init async job to complete
+      become: true
+      ansible.builtin.async_status:
+        jid: "{{ cluster_init_async.ansible_job_id }}"
+      register: cluster_init_done
+      until: cluster_init_done.finished
+      retries: 60
+      delay: 5
+
+    - name: dump init log on failure
+      ansible.builtin.command: tail -100 /tmp/spx-admin-init.log
+      register: cluster_init_log
+      changed_when: false
+      when: cluster_init_done.rc | default(1) != 0
+
+    - name: assert init exited cleanly
+      ansible.builtin.assert:
+        that: cluster_init_done.rc == 0
+        fail_msg: |
+          spx admin init failed (rc={{ cluster_init_done.rc }})
+          {{ cluster_init_log.stdout | default('(log not captured)') }}
+
+    - name: confirm multi-node predastore.toml written
+      become: true
+      ansible.builtin.shell: |
+        if grep -q '127\.0\.0\.1' {{ spinifex_etc_dir }}/predastore/predastore.toml 2>/dev/null; then
+          echo "ERROR: predastore.toml still references loopback — multi-node config not written"
+          exit 1
+        fi
+        grep -c '^\[\[nodes\]\]' {{ spinifex_etc_dir }}/predastore/predastore.toml 2>/dev/null || echo 0
+      register: predastore_node_count
+      changed_when: false
+
+- name: Cluster bootstrap — install CA on all nodes (before service start)
+  hosts: "{{ cluster_env }}"
+  gather_facts: false
+  vars_files:
+    - ../vars/defaults.yml
+  tasks:
+    - name: install CA into system trust
+      become: true
+      ansible.builtin.copy:
+        src: "{{ spinifex_ca_src }}"
+        dest: "{{ spinifex_ca_trust_path }}"
+        mode: "0644"
+        remote_src: true
+
+    - name: rebuild CA bundle
+      become: true
+      ansible.builtin.command: update-ca-certificates
+      changed_when: true
+
+    # spinifex-ui reverse-proxies localhost:9999 (awsgw) and localhost:8443
+    # (predastore) — hardcoded in spinifex/services/spinifexui/spinifexui.go.
+    # spx admin init binds those services to BindIP only, so the loopback
+    # connection fails (BadGateway in UI). Patch to wildcard bind; cert
+    # already includes 127.0.0.1 in IP SANs.
+    - name: patch awsgw bind to 0.0.0.0 (UI proxies via localhost)
+      become: true
+      ansible.builtin.replace:
+        path: "{{ spinifex_config_file }}"
+        regexp: '^(\s*host\s*=\s*")[^"]+(:9999")'
+        replace: '\g<1>0.0.0.0\g<2>'
+
+    - name: patch predastore bind to 0.0.0.0 (UI proxies via localhost)
+      become: true
+      ansible.builtin.replace:
+        path: "{{ spinifex_config_file }}"
+        regexp: '^(\s*host\s*=\s*")[^"]+(:8443")'
+        replace: '\g<1>0.0.0.0\g<2>'
+
+- name: Cluster bootstrap — start services (init first, then joins)
+  hosts: init
+  gather_facts: false
+  vars_files:
+    - ../vars/defaults.yml
+  tasks:
+    - name: start spinifex.target on init
+      become: true
+      ansible.builtin.systemd:
+        name: spinifex.target
+        state: started
+
+- name: Cluster bootstrap — start services on join hosts
+  hosts: join
+  gather_facts: false
+  serial: 100%
+  vars_files:
+    - ../vars/defaults.yml
+  tasks:
+    - name: start spinifex.target on joiners
+      become: true
+      ansible.builtin.systemd:
+        name: spinifex.target
+        state: started
+
+- name: Cluster bootstrap — readiness check (all nodes)
+  hosts: "{{ cluster_env }}"
+  gather_facts: false
+  vars_files:
+    - ../vars/defaults.yml
+  tasks:
+    - name: wait for awsgw responsiveness
+      ansible.builtin.uri:
+        url: "https://{{ spinifex_wan_ip }}:{{ cluster_formation_port }}/health"
+        validate_certs: false
+        return_content: true
+      register: cluster_health
+      until:
+        - cluster_health.status == 200
+        - cluster_health.json.service_health.awsgw == "ok"
+      retries: 30
+      delay: 5
+
+- name: Cluster bootstrap — post-bootstrap fixtures (init host only)
+  hosts: init
+  gather_facts: true
+  vars_files:
+    - ../vars/defaults.yml
+  roles:
+    - cluster-post

scripts/ansible/playbooks/dev-logs.yml

@@ -0,0 +1,17 @@
+---
+# Capture spinifex + ovn/ovs + qemu/nbdkit logs from journald and
+# /var/log/spinifex/ into a single tarball for bug reports.
+# Tail mode is intentionally not provided — use `journalctl -u 'spinifex-*' -f`
+# directly; ansible is the wrong tool for follow.
+- name: Dev logs — dump journald + varlog + status into a bundle
+  hosts: localhost
+  gather_facts: true
+  vars_files:
+    - ../vars/defaults.yml
+  vars:
+    logs_since: "1h"
+    logs_output_dir: "/tmp"
+    logs_include_dmesg: true
+
+  roles:
+    - logs

scripts/ansible/playbooks/dev-reset.yml

@@ -23,6 +23,7 @@
       when: reset_config_stat.stat.exists
 
     # List-concat-and-take-first pattern: missing regex match falls back to current default.
+    # `gateway\s*=` deliberately does not match `gateway_ip` / `gateway_lrp_*`.
     - name: capture — extract scalars
       ansible.builtin.set_fact:
         spinifex_region: "{{ (reset_config_txt | regex_findall('region\\s*=\\s*\"([^\"]+)\"') + [spinifex_region])[0] }}"
@@ -30,11 +31,25 @@
         spinifex_external_mode: "{{ (reset_config_txt | regex_findall('external_mode\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_mode])[0] }}"
         spinifex_external_pool_start: "{{ (reset_config_txt | regex_findall('range_start\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_pool_start])[0] }}"
         spinifex_external_pool_end: "{{ (reset_config_txt | regex_findall('range_end\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_pool_end])[0] }}"
-        spinifex_external_gateway: "{{ (reset_config_txt | regex_findall('gateway\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_gateway])[0] }}"
+        spinifex_external_gateway: "{{ (reset_config_txt | regex_findall('\\bgateway\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_gateway])[0] }}"
         spinifex_external_prefix_len: "{{ (reset_config_txt | regex_findall('prefix_len\\s*=\\s*([0-9]+)') + [spinifex_external_prefix_len | string])[0] }}"
-        spinifex_nat_gateway_ip: "{{ (reset_config_txt | regex_findall('gateway_ip\\s*=\\s*\"([^\"]+)\"') + [spinifex_nat_gateway_ip])[0] }}"
+        spinifex_external_gateway_ip: "{{ (reset_config_txt | regex_findall('gateway_ip\\s*=\\s*\"([^\"]+)\"') + [spinifex_external_gateway_ip])[0] }}"
       when: reset_config_stat.stat.exists
 
+    # Legacy: external_mode = "nat" was removed from spx. Coerce to "pool"
+    # so old configs still reset cleanly; warn so the operator notices.
+    - name: capture — coerce legacy external_mode=nat to pool
+      ansible.builtin.set_fact:
+        spinifex_external_mode: "pool"
+      when: spinifex_external_mode == 'nat'
+
+    - name: capture — warn on legacy nat coercion
+      ansible.builtin.debug:
+        msg: "⚠️  Captured external_mode=nat from legacy config; coerced to 'pool' (NAT mode removed from spx admin init)."
+      when:
+        - reset_config_stat.stat.exists
+        - (reset_config_txt | regex_findall('external_mode\\s*=\\s*\"nat\"')) | length > 0
+
     - name: capture — summary
       ansible.builtin.debug:
         msg: |
@@ -44,7 +59,7 @@
             external_mode={{ spinifex_external_mode }}
             pool={{ spinifex_external_pool_start }}-{{ spinifex_external_pool_end }}
             gateway={{ spinifex_external_gateway }}, prefix={{ spinifex_external_prefix_len }}
-            nat_gateway={{ spinifex_nat_gateway_ip }}
+            gateway_ip={{ spinifex_external_gateway_ip }}
           {% else %}
           No existing config; using defaults from vars/defaults.yml (or -e overrides).
           {% endif %}

scripts/ansible/playbooks/dev-restore.yml

@@ -0,0 +1,14 @@
+---
+# Restore a dev-snapshot bundle: stop services, wipe the snapshot path
+# set, untar the bundle, refresh CA trust, restart.
+#
+# Required: -e snapshot_name=<name>
+# Optional: -e snapshot_dir=/some/path
+- name: Dev restore — wipe + untar snapshot + restart
+  hosts: localhost
+  gather_facts: true
+  vars_files:
+    - ../vars/defaults.yml
+
+  roles:
+    - restore

scripts/ansible/playbooks/dev-snapshot.yml

@@ -0,0 +1,17 @@
+---
+# Quiesce spinifex + ovn + ovs, tar /etc/spinifex + /var/lib/spinifex +
+# /var/lib/ovn + /etc/openvswitch/conf.db into a named bundle, restart.
+# Use to checkpoint a working state before a risky operation (branch
+# switch, viperblock surgery, ovn-controller restart cycle). Roll back
+# via dev-restore without a full dev-reset.
+#
+# Required: -e snapshot_name=<name>
+# Optional: -e snapshot_dir=/some/path
+- name: Dev snapshot — quiesce + tar state + restart
+  hosts: localhost
+  gather_facts: true
+  vars_files:
+    - ../vars/defaults.yml
+
+  roles:
+    - snapshot

scripts/ansible/playbooks/dev-status.yml

@@ -0,0 +1,13 @@
+---
+# Read-only health snapshot for a Spinifex dev node. Never mutates state.
+# Run any time the operator wants to know "is my box healthy?" — pairs with
+# dev-preflight (run before install) as the post-install / running-system
+# diagnostic.
+- name: Dev status — health snapshot (read-only)
+  hosts: localhost
+  gather_facts: true
+  vars_files:
+    - ../vars/defaults.yml
+
+  roles:
+    - status

scripts/ansible/playbooks/dev-version.yml

@@ -0,0 +1,11 @@
+---
+# Detailed build-vs-install drift report. Faster than dev-status (no AWS,
+# OVN, or port probing). Answers "do I need to ansible-dev-deploy?".
+- name: Dev version — build vs install drift (read-only)
+  hosts: localhost
+  gather_facts: false
+  vars_files:
+    - ../vars/defaults.yml
+
+  roles:
+    - version

scripts/ansible/playbooks/dev-vpc.yml

@@ -0,0 +1,14 @@
+---
+# Idempotent dev-mode VPC fixture: spinifex-dev-vpc + public subnet +
+# IGW + default route. Hooks into smoketest's launch path (which finds
+# the public subnet automatically). Re-runs are no-ops once converged.
+#
+# Default state: present. Tear down with -e dev_vpc_state=absent.
+- name: Dev VPC — fixture VPC/subnet/IGW/route (idempotent)
+  hosts: localhost
+  gather_facts: false
+  vars_files:
+    - ../vars/defaults.yml
+
+  roles:
+    - vpc

scripts/ansible/roles/cluster-init/meta/main.yml

@@ -0,0 +1,9 @@
+---
+galaxy_info:
+  role_name: cluster-init
+  description: |
+    Run `spx admin init` on the cluster's init host, backgrounded so the
+    formation server accepts joins from the cluster-join role. Installs
+    cluster CA into the system trust store BEFORE starting spinifex.target
+    so predastore's strict TLS verify is satisfied at first start.
+dependencies: []

scripts/ansible/roles/cluster-init/tasks/main.yml

@@ -0,0 +1,77 @@
+---
+- name: cluster-init — assert this is the init host
+  ansible.builtin.assert:
+    that: spinifex_node_role == "init"
+    fail_msg: "cluster-init role must only run against hosts in the [init] group"
+
+# vars/defaults.yml sets spinifex_node_{name,count} for single-node dev use
+# (precedence 11). Inventory hostvars (precedence 4) lose that fight, so
+# we promote the inventory values via set_fact (precedence 15) before any
+# task templates `{{ spinifex_node_count }}`.
+- name: cluster-init — promote inventory node identity over defaults.yml
+  ansible.builtin.set_fact:
+    spinifex_node_name: "{{ hostvars[inventory_hostname]['spinifex_node_name'] }}"
+    spinifex_node_count: "{{ groups[cluster_env] | length }}"
+
+- name: cluster-init — assemble external network flags
+  ansible.builtin.set_fact:
+    cluster_external_flags: >-
+      {% if cluster_external_pool | length > 0 %}--external-mode=pool --external-pool={{ cluster_external_pool }}{% endif %}
+
+- name: cluster-init — run spx admin init (backgrounded, formation server waits for joins)
+  become: true
+  ansible.builtin.shell: |
+    set -o pipefail
+    spx admin init --force \
+      --node {{ spinifex_node_name }} \
+      --nodes {{ spinifex_node_count }} \
+      --bind {{ spinifex_wan_ip }} \
+      --cluster-bind {{ spinifex_wan_ip }} \
+      --port {{ cluster_formation_port }} \
+      --region {{ spinifex_region }} \
+      --az {{ spinifex_az }} \
+      {{ cluster_external_flags }} \
+      > /tmp/spx-admin-init.log 2>&1
+  async: 600
+  poll: 0
+  register: cluster_init_async
+
+- name: cluster-init — brief pause so init can fail fast if misconfigured
+  ansible.builtin.pause:
+    seconds: 5
+
+- name: cluster-init — check if init exited prematurely (sanity)
+  ansible.builtin.shell: |
+    if pgrep -f 'spx admin init' >/dev/null; then
+      echo "running"
+    else
+      tail -50 /tmp/spx-admin-init.log 2>/dev/null
+      exit 1
+    fi
+  register: cluster_init_alive
+  changed_when: false
+  failed_when: cluster_init_alive.rc != 0
+
+- name: cluster-init — wait for formation server health
+  ansible.builtin.uri:
+    url: "https://{{ spinifex_wan_ip }}:{{ cluster_formation_port }}/formation/health"
+    validate_certs: false
+    status_code: 200
+  register: cluster_init_health
+  until: cluster_init_health.status == 200
+  retries: "{{ cluster_formation_timeout // 2 }}"
+  delay: 2
+
+- name: cluster-init — fetch join token (delegated read for joiners)
+  become: true
+  ansible.builtin.slurp:
+    src: "{{ cluster_join_token_path }}"
+  register: cluster_join_token_raw
+
+- name: cluster-init — register join token as host fact
+  ansible.builtin.set_fact:
+    cluster_join_token: "{{ cluster_join_token_raw.content | b64decode | trim }}"
+
+- name: cluster-init — join token captured; waiting for joins handled by later play
+  ansible.builtin.debug:
+    msg: "Formation server up on {{ spinifex_wan_ip }}:{{ cluster_formation_port }}, {{ spinifex_node_count }} nodes expected"

scripts/ansible/roles/cluster-join/meta/main.yml

@@ -0,0 +1,8 @@
+---
+galaxy_info:
+  role_name: cluster-join
+  description: |
+    Run `spx admin join` against the cluster init host's formation server.
+    Installs cluster CA + starts spinifex.target after join completes
+    (CA-before-services ordering avoids predastore TLS-verify crashes).
+dependencies: []

scripts/ansible/roles/cluster-join/tasks/main.yml

@@ -0,0 +1,39 @@
+---
+- name: cluster-join — assert this is a join host
+  ansible.builtin.assert:
+    that: spinifex_node_role == "join"
+    fail_msg: "cluster-join role must only run against hosts in the [join] group"
+
+# Promote inventory hostvars over vars/defaults.yml (see cluster-init).
+- name: cluster-join — promote inventory node identity over defaults.yml
+  ansible.builtin.set_fact:
+    spinifex_node_name: "{{ hostvars[inventory_hostname]['spinifex_node_name'] }}"
+
+- name: cluster-join — pull join token from init host
+  ansible.builtin.set_fact:
+    cluster_join_token: "{{ hostvars[cluster_init_host]['cluster_join_token'] }}"
+
+- name: cluster-join — assert token present
+  ansible.builtin.assert:
+    that: cluster_join_token | length > 0
+    fail_msg: "join token missing — init host did not export cluster_join_token"
+
+- name: cluster-join — run spx admin join (retry up to 5 times)
+  become: true
+  ansible.builtin.command:
+    cmd: >-
+      spx admin join
+      --node {{ spinifex_node_name }}
+      --bind {{ spinifex_wan_ip }}
+      --cluster-bind {{ spinifex_wan_ip }}
+      --host {{ cluster_init_host }}:{{ cluster_formation_port }}
+      --token {{ cluster_join_token }}
+      --region {{ spinifex_region }}
+      --az {{ spinifex_az }}
+  register: cluster_join_result
+  retries: 5
+  delay: 3
+  until: cluster_join_result.rc == 0
+
+# CA install + service start handled by cluster-bootstrap.yml plays that
+# run after all joins complete and init has written multi-node configs.

scripts/ansible/roles/cluster-post/meta/main.yml

@@ -0,0 +1,8 @@
+---
+galaxy_info:
+  role_name: cluster-post
+  description: |
+    Post-bootstrap idempotent fixtures on the init host: import SSH key,
+    import Ubuntu AMI (dual-version detect 24.04/26.04). Skip when
+    artifacts already exist.
+dependencies: []

scripts/ansible/roles/cluster-post/tasks/main.yml

@@ -0,0 +1,205 @@
+---
+- name: cluster-post — locate ssh public key for import
+  ansible.builtin.find:
+    paths: "{{ ansible_env.HOME }}/.ssh"
+    patterns:
+      - spinifex-key.pub
+      - tf-user-*.pub
+    file_type: file
+  register: cluster_post_ssh_pubkey
+
+- name: cluster-post — fail when no public key found
+  ansible.builtin.assert:
+    that: cluster_post_ssh_pubkey.files | length > 0
+    fail_msg: "no public key in ~/.ssh — needed for spinifex-key import"
+
+- name: cluster-post — set pubkey path fact
+  ansible.builtin.set_fact:
+    cluster_post_pubkey_path: "{{ cluster_post_ssh_pubkey.files[0].path }}"
+
+- name: cluster-post — check whether spinifex-key already imported
+  ansible.builtin.command:
+    cmd: "{{ spinifex_aws_cli }} ec2 describe-key-pairs --key-names spinifex-key"
+  register: cluster_post_key_check
+  changed_when: false
+  failed_when: false
+
+- name: cluster-post — import spinifex-key (if absent)
+  ansible.builtin.command:
+    cmd: >-
+      {{ spinifex_aws_cli }} ec2 import-key-pair
+      --key-name spinifex-key
+      --public-key-material fileb://{{ cluster_post_pubkey_path }}
+  register: cluster_post_key_import
+  retries: 10
+  delay: 6
+  until: cluster_post_key_import.rc == 0
+  when: cluster_post_key_check.rc != 0
+
+- name: cluster-post — locate ubuntu cloud image (26.04 preferred, 24.04 fallback)
+  ansible.builtin.stat:
+    path: "{{ item }}"
+  register: cluster_post_ubuntu_stat
+  loop:
+    - "{{ ansible_env.HOME }}/images/ubuntu-26.04.img"
+    - "{{ ansible_env.HOME }}/images/ubuntu-24.04.img"
+
+- name: cluster-post — pick first available image
+  ansible.builtin.set_fact:
+    cluster_post_ubuntu_image: "{{ (cluster_post_ubuntu_stat.results | selectattr('stat.exists') | map(attribute='item') | list | first) | default('') }}"
+
+- name: cluster-post — fail when no ubuntu image present
+  ansible.builtin.assert:
+    that: cluster_post_ubuntu_image | length > 0
+    fail_msg: "no ubuntu-{24,26}.04.img in ~/images/ — gold image must stage it"
+
+- name: cluster-post — derive ubuntu version from image filename
+  ansible.builtin.set_fact:
+    cluster_post_ubuntu_version: "{{ cluster_post_ubuntu_image | regex_search('ubuntu-([0-9]+\\.[0-9]+)\\.img', '\\1') | first }}"
+
+- name: cluster-post — check whether ubuntu AMI already imported
+  ansible.builtin.command:
+    cmd: >-
+      {{ spinifex_aws_cli }} ec2 describe-images
+      --owners self
+      --filters Name=name,Values=ubuntu-{{ cluster_post_ubuntu_version }}
+  register: cluster_post_ami_check
+  changed_when: false
+  failed_when: false
+
+- name: cluster-post — import ubuntu AMI (if absent)
+  ansible.builtin.command:
+    cmd: >-
+      {{ spinifex_spx_cli }} admin images import
+      --file {{ cluster_post_ubuntu_image }}
+      --arch x86_64
+      --distro ubuntu
+      --version {{ cluster_post_ubuntu_version }}
+      --boot-mode uefi
+  when:
+    - cluster_post_ami_check.rc != 0
+       or "ubuntu-" + cluster_post_ubuntu_version not in cluster_post_ami_check.stdout
+
+- name: cluster-post — slurp final admin creds from ~/.aws/credentials
+  # Source of truth: SetupAWSCredentials writes the FINAL admin keys here.
+  # Init log shows obsolete keys (multi-node path silently regenerates them
+  # after the initial print — see cmd/spinifex/cmd/admin.go writeBootstrapFiles
+  # called twice in the multi-node flow).
+  ansible.builtin.slurp:
+    src: "{{ ansible_env.HOME }}/.aws/credentials"
+  register: cluster_post_aws_creds_raw
+
+- name: cluster-post — extract admin access/secret keys from spinifex profile
+  ansible.builtin.set_fact:
+    cluster_post_admin_access_key: >-
+      {{ (cluster_post_aws_creds_raw.content | b64decode
+          | regex_search('\[spinifex\][^\[]*aws_access_key_id\s*=\s*(\S+)', '\1')
+          | first) | default('') }}
+    cluster_post_admin_secret_key: >-
+      {{ (cluster_post_aws_creds_raw.content | b64decode
+          | regex_search('\[spinifex\][^\[]*aws_secret_access_key\s*=\s*(\S+)', '\1')
+          | first) | default('') }}
+
+- name: cluster-post — assert creds parsed
+  ansible.builtin.assert:
+    that:
+      - cluster_post_admin_access_key | length > 0
+      - cluster_post_admin_secret_key | length > 0
+    fail_msg: "could not parse admin keys from {{ ansible_env.HOME }}/.aws/credentials [spinifex] profile"
+
+- name: cluster-post — ensure local .creds dir exists
+  delegate_to: localhost
+  become: false
+  ansible.builtin.file:
+    path: "{{ playbook_dir }}/../.creds"
+    state: directory
+    mode: "0700"
+
+- name: cluster-post — write creds to operator workstation
+  delegate_to: localhost
+  become: false
+  ansible.builtin.copy:
+    dest: "{{ playbook_dir }}/../.creds/{{ cluster_env }}.txt"
+    mode: "0600"
+    content: |
+      # Spinifex cluster admin credentials — {{ cluster_env }}
+      # Generated: {{ ansible_facts['date_time']['iso8601'] }}
+      # Init host: {{ cluster_init_host }}
+      #
+      # UI:        https://{{ cluster_init_host }}:3000
+      # AWS API:   https://{{ cluster_init_host }}:4432
+      # Region:    {{ spinifex_region }}
+      #
+      # Login (admin user / AWS profile spinifex):
+      AWS_ACCESS_KEY_ID={{ cluster_post_admin_access_key }}
+      AWS_SECRET_ACCESS_KEY={{ cluster_post_admin_secret_key }}
+
+# Per-env AWS profile in operator's ~/.aws/credentials so tofu / aws CLI can
+# target this cluster via AWS_PROFILE=spinifex-{{ cluster_env }}. blockinfile
+# marker keeps re-runs idempotent — replaces the block in place, never
+# duplicates. Does not touch the user's existing [spinifex] profile (single-
+# node dev). Tofu side: provider "aws" { profile = "spinifex-env1" ... }.
+- name: cluster-post — ensure local ~/.aws dir exists
+  delegate_to: localhost
+  become: false
+  ansible.builtin.file:
+    path: "{{ lookup('env', 'HOME') }}/.aws"
+    state: directory
+    mode: "0700"
+
+- name: cluster-post — ensure local ~/.aws/credentials exists
+  delegate_to: localhost
+  become: false
+  ansible.builtin.copy:
+    dest: "{{ lookup('env', 'HOME') }}/.aws/credentials"
+    content: ""
+    force: false
+    mode: "0600"
+
+- name: cluster-post — upsert spinifex-{{ cluster_env }} profile in local ~/.aws/credentials
+  delegate_to: localhost
+  become: false
+  ansible.builtin.blockinfile:
+    path: "{{ lookup('env', 'HOME') }}/.aws/credentials"
+    marker: "# {mark} SPINIFEX-{{ cluster_env | upper }} MANAGED BLOCK"
+    block: |
+      [spinifex-{{ cluster_env }}]
+      aws_access_key_id = {{ cluster_post_admin_access_key }}
+      aws_secret_access_key = {{ cluster_post_admin_secret_key }}
+      region = {{ spinifex_region }}
+      ca_bundle = {{ playbook_dir }}/../.creds/{{ cluster_env }}-ca.pem
+      endpoint_url = https://{{ cluster_init_host }}:4432
+
+- name: cluster-post — slurp cluster CA from init host
+  ansible.builtin.slurp:
+    src: "{{ spinifex_ca_src }}"
+  register: cluster_post_ca_raw
+  become: true
+
+- name: cluster-post — write CA pem to operator workstation
+  delegate_to: localhost
+  become: false
+  ansible.builtin.copy:
+    dest: "{{ playbook_dir }}/../.creds/{{ cluster_env }}-ca.pem"
+    mode: "0644"
+    content: "{{ cluster_post_ca_raw.content | b64decode }}"
+
+- name: cluster-post — resolve absolute paths for summary
+  delegate_to: localhost
+  become: false
+  ansible.builtin.command: "readlink -f {{ playbook_dir }}/../.creds"
+  register: cluster_post_creds_abs
+  changed_when: false
+
+- name: cluster-post — summary
+  ansible.builtin.debug:
+    msg: |
+      Post-bootstrap done:
+        spinifex-key:    {{ 'imported now' if (cluster_post_key_import.changed | default(false)) else 'already present' }}
+        ubuntu AMI:      ubuntu-{{ cluster_post_ubuntu_version }}
+        admin creds:     {{ cluster_post_creds_abs.stdout }}/{{ cluster_env }}.txt
+        CA cert:         {{ cluster_post_creds_abs.stdout }}/{{ cluster_env }}-ca.pem
+        AWS profile:     spinifex-{{ cluster_env }} (in ~/.aws/credentials)
+        UI URL:          https://{{ cluster_init_host }}:3000
+        Access Key:      {{ cluster_post_admin_access_key }}
+        Tofu usage:      AWS_PROFILE=spinifex-{{ cluster_env }} tofu plan

scripts/ansible/roles/deploy/meta/main.yml

@@ -0,0 +1,7 @@
+---
+galaxy_info:
+  role_name: deploy
+  description: In-place binary + microvm artifact swap for a running Spinifex node (parity with `make deploy`).
+  license: BSD-3-Clause
+  min_ansible_version: "2.14"
+dependencies: []

scripts/ansible/roles/deploy/tasks/main.yml

@@ -127,11 +127,7 @@
     state: started
 
 - name: deploy — wait for awsgw to respond
-  become: true
-  become_user: "{{ spinifex_invoking_user }}"
-  environment:
-    AWS_PROFILE: spinifex
-  ansible.builtin.command: aws ec2 describe-regions
+  ansible.builtin.command: "{{ spinifex_aws_cli }} ec2 describe-regions"
   register: deploy_awsgw
   retries: 15
   delay: 2

scripts/ansible/roles/init/tasks/main.yml

@@ -8,28 +8,38 @@
       --node {{ spinifex_node_name }}
       --nodes {{ spinifex_node_count }}
 
+- name: init — no-external opt-out
+  ansible.builtin.set_fact:
+    init_external_flags: "--no-external"
+  when: spinifex_external_no_external | bool
+
+- name: init — assert pool range supplied when mode=pool
+  ansible.builtin.assert:
+    that:
+      - spinifex_external_pool_start | length > 0
+      - spinifex_external_pool_end | length > 0
+    fail_msg: |
+      spx admin init requires --external-pool when --external-mode=pool.
+      Set spinifex_external_pool_start / spinifex_external_pool_end in
+      scripts/ansible/vars/defaults.yml, or pass:
+        -e spinifex_external_pool_start=192.168.1.150
+        -e spinifex_external_pool_end=192.168.1.250
+      Pass -e spinifex_external_no_external=true to skip external networking.
+  when:
+    - not (spinifex_external_no_external | bool)
+    - spinifex_external_mode == 'pool'
+
 - name: init — pool-mode external flags
   ansible.builtin.set_fact:
     init_external_flags: >-
       --external-mode=pool
       --external-pool={{ spinifex_external_pool_start }}-{{ spinifex_external_pool_end }}
-      --external-gateway={{ spinifex_external_gateway }}
-      --external-prefix-len={{ spinifex_external_prefix_len }}
+      {% if spinifex_external_gateway | length > 0 %}--external-gateway={{ spinifex_external_gateway }}{% endif %}
+      {% if spinifex_external_prefix_len | string | length > 0 %}--external-prefix-len={{ spinifex_external_prefix_len }}{% endif %}
+      {% if spinifex_external_gateway_ip | length > 0 %}--gateway-ip={{ spinifex_external_gateway_ip }}{% endif %}
   when:
+    - not (spinifex_external_no_external | bool)
     - spinifex_external_mode == 'pool'
-    - spinifex_external_pool_start | length > 0
-    - spinifex_external_pool_end | length > 0
-
-- name: init — NAT-mode external flags
-  ansible.builtin.set_fact:
-    init_external_flags: >-
-      --external-mode=nat
-      --gateway-ip={{ spinifex_nat_gateway_ip }}
-      --external-gateway={{ spinifex_external_gateway }}
-      --external-prefix-len={{ spinifex_external_prefix_len }}
-  when:
-    - spinifex_external_mode == 'nat'
-    - spinifex_nat_gateway_ip | length > 0
 
 - name: init — assemble full flag string
   ansible.builtin.set_fact:
@@ -63,11 +73,7 @@
 # to NATS + daemons after that. Bash start-dev.sh has internal wait loops;
 # systemd path does not. Poll a benign call until gateway is fully ready.
 - name: init — wait for AWS gateway readiness
-  become: true
-  become_user: "{{ spinifex_invoking_user }}"
-  environment:
-    AWS_PROFILE: spinifex
-  ansible.builtin.command: aws ec2 describe-regions
+  ansible.builtin.command: "{{ spinifex_aws_cli }} ec2 describe-regions"
   register: init_gw_ready
   until: init_gw_ready.rc == 0
   retries: 30
@@ -79,6 +85,6 @@
     msg: |
       Init complete:
         region={{ spinifex_region }}, az={{ spinifex_az }}, node={{ spinifex_node_name }}
-        external_mode={{ spinifex_external_mode }}
+        external_mode={{ 'disabled' if (spinifex_external_no_external | bool) else spinifex_external_mode }}
         spinifex.target: started
         CA: {{ spinifex_ca_trust_path }}

scripts/ansible/roles/install/tasks/main.yml

@@ -40,6 +40,19 @@
     path: "{{ spinifex_data_dir }}/config"
     state: absent
 
+- name: install — purge stale cluster state (configs + raft + NATS KV) on bootstrap
+  become: true
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: absent
+  loop:
+    - "{{ spinifex_etc_dir }}/predastore/predastore.toml"
+    - "{{ spinifex_etc_dir }}/spinifex.toml"
+    - "{{ spinifex_data_dir }}/predastore"
+    - "{{ spinifex_data_dir }}/nats"
+    - "{{ spinifex_data_dir }}/awsgw"
+  when: cluster_bootstrap_purge | default(false) | bool
+
 - name: install — run setup.sh (creates users, dirs, systemd units)
   become: true
   ansible.builtin.command:
@@ -57,8 +70,9 @@
 - name: install — run setup-ovn.sh --management
   become: true
   ansible.builtin.command:
-    cmd: /usr/local/share/spinifex/setup-ovn.sh --management
+    cmd: "/usr/local/share/spinifex/setup-ovn.sh --management --node-name={{ spinifex_node_name }}"
   changed_when: true
+  when: install_run_ovn | default(true) | bool
 
 - name: install — summary
   ansible.builtin.debug:

scripts/ansible/roles/logs/meta/main.yml

@@ -0,0 +1,7 @@
+---
+galaxy_info:
+  role_name: logs
+  description: Dump spinifex + ovn + system logs into a tarball for bug reports.
+  license: BSD-3-Clause
+  min_ansible_version: "2.14"
+dependencies: []

scripts/ansible/roles/logs/tasks/main.yml

@@ -0,0 +1,153 @@
+---
+# Collect a self-contained log bundle for bug reports. NEVER mutates running
+# state — only reads journalctl + /var/log/spinifex/* + status output, then
+# tars into logs_output_dir. Errors on missing inputs are non-fatal so a
+# half-broken node still produces a bundle.
+
+- name: logs — resolve staging directory
+  ansible.builtin.set_fact:
+    logs_stage_dir: "/tmp/spinifex-logs-{{ ansible_facts['hostname'] }}-{{ ansible_facts['date_time']['iso8601_basic_short'] }}"
+    logs_bundle_path: "{{ logs_output_dir }}/spinifex-logs-{{ ansible_facts['hostname'] }}-{{ ansible_facts['date_time']['iso8601_basic_short'] }}.tar.gz"
+
+- name: logs — create staging tree
+  ansible.builtin.file:
+    path: "{{ logs_stage_dir }}/{{ item }}"
+    state: directory
+    mode: "0755"
+  loop:
+    - journal
+    - varlog
+    - status
+    - kernel
+
+- name: logs — capture journalctl for spinifex-* units
+  become: true
+  ansible.builtin.shell: |
+    journalctl --no-pager --output=short-iso \
+      -u 'spinifex-*' --since {{ logs_since | quote }} \
+      > {{ logs_stage_dir }}/journal/spinifex.log 2>&1 || true
+  changed_when: true
+
+- name: logs — capture journalctl for ovn-* + openvswitch-switch
+  become: true
+  ansible.builtin.shell: |
+    journalctl --no-pager --output=short-iso \
+      -u 'ovn-*' -u openvswitch-switch \
+      --since {{ logs_since | quote }} \
+      > {{ logs_stage_dir }}/journal/ovn-ovs.log 2>&1 || true
+  changed_when: true
+
+- name: logs — capture journalctl for nbdkit + qemu (best-effort match)
+  become: true
+  ansible.builtin.shell: |
+    journalctl --no-pager --output=short-iso \
+      --since {{ logs_since | quote }} \
+      _COMM=qemu-system-x86_64 + _COMM=qemu-system-aarch64 + _COMM=nbdkit \
+      > {{ logs_stage_dir }}/journal/qemu-nbdkit.log 2>&1 || true
+  changed_when: true
+
+- name: logs — copy /var/log/spinifex/*
+  become: true
+  ansible.builtin.shell: |
+    if [ -d {{ spinifex_log_dir | quote }} ]; then
+      cp -a {{ spinifex_log_dir | quote }}/. {{ logs_stage_dir | quote }}/varlog/ 2>/dev/null || true
+    fi
+  changed_when: true
+
+- name: logs — systemctl status snapshot (spinifex-*)
+  become: true
+  ansible.builtin.shell: |
+    systemctl status 'spinifex-*' --no-pager --all \
+      > {{ logs_stage_dir }}/status/spinifex-units.txt 2>&1 || true
+    systemctl status ovn-controller ovn-central openvswitch-switch \
+      --no-pager > {{ logs_stage_dir }}/status/ovn-ovs-units.txt 2>&1 || true
+    systemctl list-units 'spinifex-*' --all --no-legend --plain \
+      > {{ logs_stage_dir }}/status/spinifex-list.txt 2>&1 || true
+  changed_when: true
+
+- name: logs — networking snapshot (ip, ss, ovs)
+  become: true
+  ansible.builtin.shell: |
+    ip -d -s link show > {{ logs_stage_dir }}/status/ip-link.txt 2>&1 || true
+    ip addr show       > {{ logs_stage_dir }}/status/ip-addr.txt 2>&1 || true
+    ip route show table all > {{ logs_stage_dir }}/status/ip-route.txt 2>&1 || true
+    ss -tlnp           > {{ logs_stage_dir }}/status/ss-tlnp.txt 2>&1 || true
+    if command -v ovs-vsctl >/dev/null 2>&1; then
+      ovs-vsctl show     > {{ logs_stage_dir }}/status/ovs-show.txt 2>&1 || true
+      ovs-vsctl list-br  > {{ logs_stage_dir }}/status/ovs-br.txt 2>&1 || true
+    fi
+    if command -v ovn-nbctl >/dev/null 2>&1; then
+      ovn-nbctl show   > {{ logs_stage_dir }}/status/ovn-nb.txt 2>&1 || true
+      ovn-sbctl show   > {{ logs_stage_dir }}/status/ovn-sb.txt 2>&1 || true
+    fi
+  changed_when: true
+
+- name: logs — kernel ring buffer
+  become: true
+  ansible.builtin.shell: |
+    dmesg --time-format=iso > {{ logs_stage_dir }}/kernel/dmesg.txt 2>&1 || true
+  changed_when: true
+  when: logs_include_dmesg | bool
+
+- name: logs — capture spx + plugin versions
+  become: true
+  ansible.builtin.shell: |
+    {
+      echo '## spx version'
+      /usr/local/bin/spx version 2>&1 | grep -vE '^[0-9]{4}/' || true
+      echo
+      echo '## nbdkit plugin sha256'
+      pdir=$(nbdkit --dump-config 2>/dev/null | awk -F= '/^plugindir=/{print $2; exit}')
+      [ -n "$pdir" ] && sha256sum "$pdir/nbdkit-viperblock-plugin.so" 2>/dev/null || true
+      echo
+      echo '## microvm artifacts'
+      sha256sum /usr/share/spinifex/microvm/* 2>/dev/null || true
+      echo
+      echo '## kernel + os'
+      uname -a
+      cat /etc/os-release 2>/dev/null || true
+    } > {{ logs_stage_dir }}/status/versions.txt 2>&1 || true
+  changed_when: true
+
+- name: logs — create bundle tarball
+  become: true
+  ansible.builtin.shell: |
+    tar -C {{ logs_stage_dir | dirname | quote }} \
+      -czf {{ logs_bundle_path | quote }} \
+      {{ logs_stage_dir | basename | quote }}
+  changed_when: true
+
+- name: logs — make bundle owned by invoking user
+  become: true
+  ansible.builtin.file:
+    path: "{{ logs_bundle_path }}"
+    owner: "{{ spinifex_invoking_user }}"
+    group: "{{ spinifex_invoking_user }}"
+    mode: "0640"
+
+- name: logs — remove staging tree
+  become: true
+  ansible.builtin.file:
+    path: "{{ logs_stage_dir }}"
+    state: absent
+
+- name: logs — bundle size
+  ansible.builtin.stat:
+    path: "{{ logs_bundle_path }}"
+  register: logs_bundle_stat
+
+- name: logs — summary
+  ansible.builtin.debug:
+    msg: |
+      Log bundle written:
+        path  : {{ logs_bundle_path }}
+        size  : {{ (logs_bundle_stat.stat.size | int / 1024) | round(1) }} KiB
+        since : {{ logs_since }}
+      Contents:
+        journal/    spinifex-*, ovn-*+openvswitch-switch, qemu/nbdkit
+        varlog/     copy of {{ spinifex_log_dir }}
+        status/     systemctl status, ip/ss/ovs/ovn show, versions
+        kernel/     dmesg{{ '' if logs_include_dmesg else ' (skipped via -e logs_include_dmesg=false)' }}
+
+      Attach to bug report. For live tail use:
+        journalctl -u 'spinifex-*' -f

scripts/ansible/roles/restore/meta/main.yml

@@ -0,0 +1,7 @@
+---
+galaxy_info:
+  role_name: restore
+  description: Restore a spinifex+ovn+ovs snapshot bundle, refresh CA trust, restart.
+  license: BSD-3-Clause
+  min_ansible_version: "2.14"
+dependencies: []