release: update go.mod for v1.5.0

Self-hosted runner accumulated 18GB in overlay2 from leaked container
layers. Two causes: build-microvm-image.sh only removed the alpine
container on the success path, and no workflow step pruned docker
between runs.

- scripts/build-microvm-image.sh: container id now removed via the
  existing EXIT trap, so failures, signals, and runner kills no longer
  leak the writable layer.
- .github/workflows/e2e-go.yml: prune containers / dangling images /
  buildx cache before each Build Install Artifacts step, in all three
  e2e jobs (single, multinode, reboot).

.github/actions/e2e-analyze/README.md

@@ -0,0 +1,104 @@
+# e2e-analyze
+
+Post-test failure-clustering action for the Go E2E suites.
+
+## What it does
+
+Runs after `go-junit-report` has converted each suite's stdout into
+`junit-<suite>.xml`. For every JUnit file matched by `junit-glob`:
+
+1. Parses each `<testcase>` with a `<failure>` / `<error>` child.
+2. Strips ANSI escapes, then normalises dynamic noise (UUIDs, EC2 IDs,
+   IPs, timestamps, request-ids, source line numbers) to produce a stable
+   _signature_ per failure.
+3. Picks the **earliest non-cascade** failure per suite as the likely
+   root cause. "Cascade" = failure message matches a fixture-dependency
+   marker (`must populate fix.`, `Should NOT be empty`,
+   `Expected value not to be nil`). When every failure is a cascade
+   (whole suite tripped on one upstream gap), the earliest cascade is
+   the fallback root.
+4. Buckets the remaining failures by signature and renders a markdown
+   report. The report is appended to `$GITHUB_STEP_SUMMARY` and also
+   written to `<log-dir>/analysis.md` so it travels with the uploaded
+   artifact bundle.
+
+The action **never fails the job** — exit-code semantics stay with the
+existing "Fail job if any suite failed" gate.
+
+## Inputs
+
+| Input        | Required | Description |
+|--------------|----------|-------------|
+| `junit-glob` | yes      | Shell glob for the JUnit XML files (e.g. `${ARTIFACT_DIR}/junit-*.xml`). |
+| `log-dir`    | yes      | Directory the rendered report is written into (`analysis.md`). |
+| `title`      | no       | Heading at the top of the summary section. Default `E2E failure analysis`. |
+
+## Outputs
+
+None. The summary is appended to `$GITHUB_STEP_SUMMARY`; the same
+content lands at `<log-dir>/analysis.md`.
+
+## Local invocation
+
+Outside Actions, the analyser prints to stdout instead of appending to a
+summary file:
+
+```sh
+cd spinifex/.github/actions/e2e-analyze
+go run . -junit-glob "/tmp/artifacts/junit-*.xml" -log-dir /tmp/artifacts
+```
+
+## Tests
+
+Golden-file tests live in `testdata/`. Each subdir holds a scenario:
+
+| Scenario           | Source                                    |
+|--------------------|-------------------------------------------|
+| `cascade-vpcs`     | Synthetic: one Phase 5 root + 15 cascades |
+| `single-failure`   | Real artifact from run `26037892507`      |
+| `no-failures`      | Synthetic: clean run                      |
+
+Run normally:
+
+```sh
+go test ./.github/actions/e2e-analyze/
+```
+
+When the report format changes intentionally, regenerate the snapshots:
+
+```sh
+go test ./.github/actions/e2e-analyze/ -update
+```
+
+## Per-failure bundles (Stage 2)
+
+After parsing, the action materialises a bundle directory per failure
+under `<log-dir>/analysis/NN-<TestName>/`. The report links to it from
+each entry so a reader on the Actions page can click straight through
+to the relevant log slice.
+
+Each bundle contains:
+
+| File              | Source                                                                            |
+|-------------------|-----------------------------------------------------------------------------------|
+| `failure.txt`     | Synthesised summary: test, start, duration, file-site, error line.                |
+| `journal.log`     | `spinifex-journal.log` sliced to `[StartAt − 5s, StartAt + Duration + 5s]`.       |
+| `test.log`        | `test-<suite>.log` block between `=== RUN <name>` and the matching `--- FAIL`.    |
+| `<other files>`   | Anything the harness already dropped into `<log-dir>/<TestName>/` at run-time.    |
+
+The journal slice relies on the workflow running
+`journalctl --output=short-iso` so timestamps are parseable. Lines
+without a parseable timestamp inherit the previous line's
+in-window/out-of-window classification (so multi-line records and stack
+traces stay contiguous).
+
+## Roadmap
+
+- **Stage 1 (shipped):** first-failure surfacing + cascade clustering.
+- **Stage 2 (this update):** time-window correlation — per-failure bundle
+  with sliced journal + test-log + any per-test artifacts.
+- **Stage 3:** rule-based root-cause classifier driven by
+  `spinifex/tests/e2e/.failure-rules.yaml`.
+
+See `docs/development/improvements/e2e-go-failure-analysis.md` in the
+parent mulga repo.

.github/actions/e2e-analyze/action.yml

@@ -0,0 +1,38 @@
+name: E2E Failure Analyzer
+description: >-
+  Post-test analyzer for the Go E2E suites. Parses junit-*.xml emitted by
+  go-junit-report, clusters failures by signature, picks the earliest
+  non-cascade failure per suite as the likely root cause, and appends a
+  human-readable report to $GITHUB_STEP_SUMMARY. Always exits 0 — the
+  existing "Fail job if any suite failed" gate keeps owning exit-code
+  semantics.
+
+inputs:
+  junit-glob:
+    description: Shell glob matching JUnit XML files to analyze.
+    required: true
+  log-dir:
+    description: >-
+      Artifact directory. The rendered report is also written to
+      $log-dir/analysis.md so it travels with the uploaded artifact.
+    required: true
+  title:
+    description: Heading rendered at the top of the summary section.
+    required: false
+    default: "E2E failure analysis"
+
+runs:
+  using: composite
+  steps:
+    - id: analyze
+      shell: bash
+      working-directory: ${{ github.action_path }}
+      env:
+        ANALYZE_JUNIT_GLOB: ${{ inputs.junit-glob }}
+        ANALYZE_LOG_DIR: ${{ inputs.log-dir }}
+        ANALYZE_TITLE: ${{ inputs.title }}
+      run: |
+        go run . \
+          -junit-glob "$ANALYZE_JUNIT_GLOB" \
+          -log-dir "$ANALYZE_LOG_DIR" \
+          -title "$ANALYZE_TITLE"

.github/actions/e2e-analyze/analyze.go

@@ -0,0 +1,552 @@
+// Package main implements the e2e-analyze GitHub Action.
+//
+// Stage 1 of docs/development/improvements/e2e-go-failure-analysis.md:
+// cluster failures from go-junit-report's junit-*.xml by error signature,
+// pick the earliest non-cascade failure per suite as the likely root cause,
+// and render a human-readable report.
+//
+// This file contains the pure logic (parsing, signature extraction,
+// clustering, rendering). Side-effect handling lives in main.go.
+package main
+
+import (
+	"encoding/xml"
+	"fmt"
+	"path/filepath"
+	"regexp"
+	"sort"
+	"strings"
+	"time"
+)
+
+// JUnit XML schema produced by go-junit-report v2. Stable across versions
+// we care about; only fields used downstream are decoded.
+type junitSuites struct {
+	XMLName xml.Name     `xml:"testsuites"`
+	Suites  []junitSuite `xml:"testsuite"`
+}
+
+type junitSuite struct {
+	Name      string    `xml:"name,attr"`
+	Tests     int       `xml:"tests,attr"`
+	Failures  int       `xml:"failures,attr"`
+	Errors    int       `xml:"errors,attr"`
+	Skipped   int       `xml:"skipped,attr"`
+	Timestamp string    `xml:"timestamp,attr"`
+	Time      float64   `xml:"time,attr"`
+	Cases     []junitTC `xml:"testcase"`
+}
+
+type junitTC struct {
+	Name      string        `xml:"name,attr"`
+	Classname string        `xml:"classname,attr"`
+	Time      float64       `xml:"time,attr"`
+	Failure   *junitFailure `xml:"failure"`
+	Error     *junitFailure `xml:"error"`
+	Skipped   *junitSkipped `xml:"skipped"`
+	SystemOut string        `xml:"system-out"`
+}
+
+type junitFailure struct {
+	Message string `xml:"message,attr"`
+	Type    string `xml:"type,attr"`
+	Body    string `xml:",chardata"`
+}
+
+type junitSkipped struct {
+	Message string `xml:"message,attr"`
+}
+
+// Failure is the analyzer's normalized view of one failed testcase.
+type Failure struct {
+	SuiteFile string    // e.g. "single" (derived from junit-single.xml)
+	Name      string    // e.g. "TestSingleNode/Phase5_LaunchInstance"
+	Duration  float64   // seconds
+	Order     int       // position within the suite (0-based)
+	StartAt   time.Time // estimated wall-clock start: suite.timestamp + sum(prior durations)
+	Error     string    // last meaningful error line, ANSI-stripped
+	FileHint  string    // e.g. "vpc_test.go:227", parsed from failure body
+	Cascade   bool      // signature matched a known downstream pattern
+	Signature string    // normalized error key for bucketing
+	// OffsetFromSuiteStart is sum(prior testcase durations) at the time
+	// this test started. Persisted separately from StartAt so a caller
+	// that has a more authoritative suite wall-clock origin (e.g. a
+	// `test-<suite>.start` file written by the workflow before the suite
+	// runs) can recompute StartAt without re-parsing the XML.
+	OffsetFromSuiteStart time.Duration
+	// BundlePath is the relative path (from log-dir) to the per-failure
+	// bundle written by Stage 2's WriteBundles. Empty when bundles were
+	// not generated (e.g. unit tests that call Render directly).
+	BundlePath string
+}
+
+// SuiteReport summarises one junit-*.xml file.
+type SuiteReport struct {
+	File      string // basename, e.g. "junit-single.xml"
+	Label     string // short name, e.g. "single"
+	Total     int
+	FailCount int
+	Root      *Failure    // nil if no failures
+	Cascades  []Failure   // everything else that failed in this suite
+	Buckets   [][]Failure // failures grouped by signature (root's bucket first)
+}
+
+// Report is the full analysis across every junit file.
+type Report struct {
+	Title  string
+	Suites []SuiteReport
+}
+
+// ansiSeq matches ANSI CSI escapes the harness sprinkles through log
+// output (colours, bold). Three forms in the wild:
+//   - real ESC: `\x1b[36m…`
+//   - U+FFFD replacement (XML serialiser dropped the ESC byte): `�[36m…`
+//   - bare CSI with the leading byte gone entirely: `[36m…`
+//
+// All three are stripped so colour state doesn't fork the signature bucket.
+var ansiSeq = regexp.MustCompile(
+	"\x1b\\[[0-9;]*[a-zA-Z]" + // real ESC CSI
+		"|�\\[[0-9;]*[a-zA-Z]" + // replacement-char CSI
+		"|\\[[0-9]{1,2}(?:;[0-9]{1,2})*m", // bare colour code, "mode" form only
+)
+
+// cascadeMarkers are message fragments that only appear in tests that
+// depend on a fixture field populated by an earlier phase. Their presence
+// means the failure is downstream of some other failure that happened
+// first; the report collapses them under that root cause.
+var cascadeMarkers = []string{
+	"must populate fix.",
+	"Should NOT be empty",
+	"Expected value not to be nil",
+}
+
+// noiseReplacers normalise dynamic identifiers so that "ten failures, same
+// bug" bucket together. Pattern ordering matters: longer/more-specific
+// patterns must precede the catch-alls.
+var noiseReplacers = []struct {
+	re   *regexp.Regexp
+	repl string
+}{
+	{regexp.MustCompile(`\b(vpc|subnet|sg|rtb|igw|i|ami|vol|snap|rtbassoc|key)-[0-9a-f]{8,}\b`), `${1}-<id>`},
+	{regexp.MustCompile(`\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b`), `<uuid>`},
+	{regexp.MustCompile(`\b[0-9a-f]{32,}\b`), `<hex>`},
+	{regexp.MustCompile(`\b\d{1,3}(\.\d{1,3}){3}(:\d+)?\b`), `<ip>`},
+	{regexp.MustCompile(`\b\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:?\d{2})?\b`), `<ts>`},
+	{regexp.MustCompile(`request-id:\s*\S+`), `request-id: <id>`},
+	{regexp.MustCompile(`(_test\.go):\d+`), `${1}:<line>`},
+	{regexp.MustCompile(`\s+`), ` `},
+}
+
+// fileHintRe pulls the last "foo_test.go:NNN" we see in a failure body so
+// the report can point a reader at the source line. Test files dominate the
+// hint set; harness/helper file matches go through goFileLineRe below.
+var fileHintRe = regexp.MustCompile(`([A-Za-z0-9_]+_test\.go:\d+)`)
+
+// goFileLineRe matches any "foo.go:NNN:" prefix (test or non-test). Used to
+// surface harness/helper-level error text like
+// `ec2helpers.go:164: describe-vpcs: InvalidParameterValue …` which Go test
+// emits when a non-test file calls t.Errorf.
+var goFileLineRe = regexp.MustCompile(`^[A-Za-z0-9_]+\.go:\d+:\s*(.+)$`)
+
+// testifyErrorPlaceholder is testify's "Error:" boilerplate when the
+// assertion failed because of a wrapped error. The next non-blank line in
+// the trace holds the actual error text.
+const testifyErrorPlaceholder = "Received unexpected error:"
+
+// stripANSI removes CSI escapes; leaves everything else intact.
+func stripANSI(s string) string {
+	return ansiSeq.ReplaceAllString(s, "")
+}
+
+// extractErrorLine finds the most useful single-line summary in a failure
+// body. Priority order, highest first:
+//  1. Testify "Messages:" line — user-supplied context, the most signal-rich.
+//  2. Testify "Error:" content (with one-line lookahead when the placeholder
+//     "Received unexpected error:" defers to the next line).
+//  3. Leading "foo.go:NNN: text" content line — t.Errorf from non-test code.
+//  4. Last `_test.go:NNN: text` line — bare t.Fatalf/Errorf from a test file.
+//  5. First non-empty line of the body.
+func extractErrorLine(body string) string {
+	body = stripANSI(body)
+	lines := strings.Split(body, "\n")
+
+	var (
+		msgLine      string
+		errLine      string
+		goFileLine   string
+		lastTestLine string
+	)
+	for i, raw := range lines {
+		l := strings.TrimSpace(raw)
+		if l == "" {
+			continue
+		}
+		if strings.HasPrefix(l, "Messages:") {
+			msgLine = strings.TrimSpace(strings.TrimPrefix(l, "Messages:"))
+			continue
+		}
+		if strings.HasPrefix(l, "Error:") && errLine == "" {
+			content := strings.TrimSpace(strings.TrimPrefix(l, "Error:"))
+			if content == testifyErrorPlaceholder {
+				// Look ahead for the wrapped error text on the next non-blank line.
+				for j := i + 1; j < len(lines); j++ {
+					nxt := strings.TrimSpace(lines[j])
+					if nxt == "" {
+						continue
+					}
+					if strings.HasPrefix(nxt, "Test:") || strings.HasPrefix(nxt, "Messages:") {
+						break
+					}
+					content = content + " " + nxt
+					break
+				}
+			}
+			errLine = content
+			continue
+		}
+		if m := goFileLineRe.FindStringSubmatch(l); m != nil {
+			// Keep the LAST occurrence: t.Log / harness traces emit
+			// progress lines from the same file:line pattern before the
+			// failing assertion finally fires.
+			goFileLine = strings.TrimSpace(m[1])
+			continue
+		}
+		if fileHintRe.MatchString(l) {
+			lastTestLine = l
+		}
+	}
+	switch {
+	case msgLine != "":
+		return msgLine
+	case errLine != "":
+		return errLine
+	case goFileLine != "":
+		return goFileLine
+	case lastTestLine != "":
+		// "vpc_test.go:227: Eventually: condition not met …" → trim the file prefix.
+		if i := strings.Index(lastTestLine, ": "); i > 0 {
+			return strings.TrimSpace(lastTestLine[i+2:])
+		}
+		return lastTestLine
+	}
+	// Last resort: first non-empty line of the body.
+	for _, raw := range lines {
+		l := strings.TrimSpace(raw)
+		if l != "" {
+			return l
+		}
+	}
+	return ""
+}
+
+// errorTraceRe extracts the path from testify's "Error Trace:" line — e.g.
+//
+//	Error Trace:	tests/e2e/harness/ec2helpers.go:164
+//
+// — which points at the assertion site even when it lives in helper code
+// outside *_test.go.
+var errorTraceRe = regexp.MustCompile(`Error Trace:\s*(\S+\.go:\d+)`)
+
+// extractFileHint returns the most precise pointer at the assertion site
+// we can pull from the failure body: testify Error Trace wins, otherwise
+// the last `*_test.go:NNN` line in the body.
+func extractFileHint(body string) string {
+	body = stripANSI(body)
+	if m := errorTraceRe.FindStringSubmatch(body); m != nil {
+		return m[1]
+	}
+	all := fileHintRe.FindAllString(body, -1)
+	if len(all) == 0 {
+		return ""
+	}
+	return all[len(all)-1]
+}
+
+// signature normalises an error line so different runs of the same root
+// cause bucket together. Empty signatures (e.g. parent-test placeholders
+// where the failure body is empty) bucket together under "".
+func signature(errLine string) string {
+	s := strings.ToLower(errLine)
+	for _, r := range noiseReplacers {
+		s = r.re.ReplaceAllString(s, r.repl)
+	}
+	return strings.TrimSpace(s)
+}
+
+// isCascade reports whether the error line contains any of the well-known
+// fixture-propagation markers — i.e. "this test failed because something
+// upstream of it failed first".
+func isCascade(errLine string) bool {
+	for _, m := range cascadeMarkers {
+		if strings.Contains(errLine, m) {
+			return true
+		}
+	}
+	return false
+}
+
+// computeLeafSet returns the testcase names that are leaves — i.e. no
+// other testcase name in `cases` starts with `<name>/`. go-junit-report
+// emits one <testcase> for the parent Go test AND each subtest, with the
+// parent's time = sum of its children, so summing every entry into the
+// suite's cumulative offset double-counts. Restrict accumulation to
+// leaves only.
+func computeLeafSet(cases []junitTC) map[string]bool {
+	leaf := make(map[string]bool, len(cases))
+	for _, tc := range cases {
+		leaf[tc.Name] = true
+	}
+	for _, tc := range cases {
+		// Strip the deepest path segment; any ancestor of this test is
+		// not a leaf.
+		name := tc.Name
+		for {
+			i := strings.LastIndex(name, "/")
+			if i <= 0 {
+				break
+			}
+			name = name[:i]
+			if _, ok := leaf[name]; ok {
+				leaf[name] = false
+			}
+		}
+	}
+	return leaf
+}
+
+// parseStartTime turns a junit timestamp attr (RFC3339) into a time.Time;
+// zero value if unparseable.
+func parseStartTime(s string) time.Time {
+	if s == "" {
+		return time.Time{}
+	}
+	t, err := time.Parse(time.RFC3339, s)
+	if err != nil {
+		return time.Time{}
+	}
+	return t
+}
+
+// suiteLabel turns "junit-single.xml" into "single".
+func suiteLabel(path string) string {
+	base := filepath.Base(path)
+	base = strings.TrimSuffix(base, ".xml")
+	base = strings.TrimPrefix(base, "junit-")
+	if base == "" {
+		return filepath.Base(path)
+	}
+	return base
+}
+
+// ParseFile decodes one junit-*.xml file into our normalized failure list.
+// Returns the suite label, total testcase count, and the (ordered) failures.
+func ParseFile(path string, data []byte) (SuiteReport, error) {
+	var doc junitSuites
+	if err := xml.Unmarshal(data, &doc); err != nil {
+		return SuiteReport{}, fmt.Errorf("parse %s: %w", path, err)
+	}
+
+	rep := SuiteReport{
+		File:  filepath.Base(path),
+		Label: suiteLabel(path),
+	}
+
+	order := 0
+	for _, suite := range doc.Suites {
+		start := parseStartTime(suite.Timestamp)
+		// go-junit-report emits a testcase for both the parent Go test
+		// (e.g. `TestSingleNode`) AND each subtest (`TestSingleNode/PhaseX`).
+		// The parent's `time` attribute is the aggregate of its children, so
+		// summing every testcase double-counts. Pre-compute the leaf set so
+		// only leaves contribute to the suite's running offset.
+		isLeaf := computeLeafSet(suite.Cases)
+		cumul := 0.0
+		for _, tc := range suite.Cases {
+			rep.Total++
+			leaf := isLeaf[tc.Name]
+			var body string
+			switch {
+			case tc.Failure != nil:
+				body = tc.Failure.Body
+			case tc.Error != nil:
+				body = tc.Error.Body
+			default:
+				if leaf {
+					cumul += tc.Time
+				}
+				continue
+			}
+
+			// Skip the synthetic parent-test "Failed" stub that go-junit-report
+			// emits for any Go test whose subtests failed. It has an empty body
+			// and would otherwise outrank real failures by XML order.
+			trimmed := strings.TrimSpace(stripANSI(body))
+			if trimmed == "" {
+				if leaf {
+					cumul += tc.Time
+				}
+				continue
+			}
+
+			errLine := extractErrorLine(body)
+			offset := time.Duration(cumul * float64(time.Second))
+			f := Failure{
+				SuiteFile:            rep.Label,
+				Name:                 tc.Name,
+				Duration:             tc.Time,
+				Order:                order,
+				Error:                errLine,
+				FileHint:             extractFileHint(body),
+				Cascade:              isCascade(errLine),
+				Signature:            signature(errLine),
+				OffsetFromSuiteStart: offset,
+			}
+			if !start.IsZero() {
+				f.StartAt = start.Add(offset)
+			}
+			rep.FailCount++
+			rep.Cascades = append(rep.Cascades, f) // temp staging; split below
+			order++
+			if leaf {
+				cumul += tc.Time
+			}
+		}
+	}
+
+	if rep.FailCount == 0 {
+		rep.Cascades = nil
+		return rep, nil
+	}
+
+	all := rep.Cascades
+	rep.Cascades = nil
+
+	// Root = earliest non-cascade. If every failure is a cascade (the whole
+	// suite tripped on the same upstream gap), fall back to earliest overall.
+	rootIdx := -1
+	for i, f := range all {
+		if !f.Cascade {
+			rootIdx = i
+			break
+		}
+	}
+	if rootIdx == -1 {
+		rootIdx = 0
+	}
+	root := all[rootIdx]
+	rep.Root = &root
+
+	// Bucket the remainder by signature so the report can collapse the
+	// "ten failures, same bug" pattern.
+	bucketKeys := []string{}
+	buckets := map[string][]Failure{}
+	for i, f := range all {
+		if i == rootIdx {
+			continue
+		}
+		if _, seen := buckets[f.Signature]; !seen {
+			bucketKeys = append(bucketKeys, f.Signature)
+		}
+		buckets[f.Signature] = append(buckets[f.Signature], f)
+		rep.Cascades = append(rep.Cascades, f)
+	}
+
+	// Largest bucket first; ties broken by first-seen order so output is
+	// deterministic across runs.
+	sort.SliceStable(bucketKeys, func(i, j int) bool {
+		return len(buckets[bucketKeys[i]]) > len(buckets[bucketKeys[j]])
+	})
+	for _, k := range bucketKeys {
+		rep.Buckets = append(rep.Buckets, buckets[k])
+	}
+
+	return rep, nil
+}
+
+// Render writes the markdown report. Format is documented in
+// docs/development/improvements/e2e-go-failure-analysis.md (Stage 1).
+func Render(r Report) string {
+	var b strings.Builder
+	fmt.Fprintf(&b, "## %s\n\n", r.Title)
+
+	totalFail := 0
+	for _, s := range r.Suites {
+		totalFail += s.FailCount
+	}
+	if totalFail == 0 {
+		b.WriteString("✅ No failures across any suite.\n")
+		return b.String()
+	}
+
+	for _, s := range r.Suites {
+		if s.FailCount == 0 {
+			fmt.Fprintf(&b, "### Suite `%s`: ✅ pass (%d tests)\n\n", s.Label, s.Total)
+			continue
+		}
+		fmt.Fprintf(&b, "### Suite `%s`: %d failed, 1 root cause likely\n\n", s.Label, s.FailCount)
+
+		if s.Root != nil {
+			b.WriteString("**Root cause (earliest non-cascade)**\n\n")
+			fmt.Fprintf(&b, "- Test: `%s`\n", s.Root.Name)
+			if !s.Root.StartAt.IsZero() {
+				fmt.Fprintf(&b, "- Start: %s (duration %.1fs)\n",
+					s.Root.StartAt.Format("15:04:05"), s.Root.Duration)
+			} else {
+				fmt.Fprintf(&b, "- Duration: %.1fs\n", s.Root.Duration)
+			}
+			if s.Root.Error != "" {
+				fmt.Fprintf(&b, "- Error: %s\n", oneLine(s.Root.Error))
+			}
+			if s.Root.FileHint != "" {
+				fmt.Fprintf(&b, "- File:  `%s`\n", s.Root.FileHint)
+			}
+			if s.Root.BundlePath != "" {
+				fmt.Fprintf(&b, "- Bundle: `%s/`\n", s.Root.BundlePath)
+			}
+			b.WriteString("\n")
+		}
+
+		if len(s.Buckets) > 0 {
+			fmt.Fprintf(&b, "**Cascaded failures (%d) — grouped by signature**\n\n",
+				len(s.Cascades))
+			for _, bucket := range s.Buckets {
+				sample := bucket[0]
+				label := oneLine(sample.Error)
+				if label == "" {
+					label = "(no error message)"
+				}
+				fmt.Fprintf(&b, "- %d× %s\n", len(bucket), label)
+				// List up to 5 test names per bucket; collapse the rest.
+				const cap = 5
+				for i, f := range bucket {
+					if i == cap && len(bucket) > cap {
+						fmt.Fprintf(&b, "  - … and %d more\n", len(bucket)-cap)
+						break
+					}
+					if f.BundlePath != "" {
+						fmt.Fprintf(&b, "  - `%s` → `%s/`\n", f.Name, f.BundlePath)
+					} else {
+						fmt.Fprintf(&b, "  - `%s`\n", f.Name)
+					}
+				}
+			}
+			b.WriteString("\n")
+		}
+	}
+
+	return b.String()
+}
+
+// oneLine collapses whitespace and caps length so a runaway assertion
+// message doesn't blow out the summary card width.
+func oneLine(s string) string {
+	s = strings.TrimSpace(s)
+	s = strings.ReplaceAll(s, "\n", " ")
+	s = strings.ReplaceAll(s, "\r", " ")
+	s = regexp.MustCompile(`\s+`).ReplaceAllString(s, " ")
+	const max = 160
+	if len(s) > max {
+		s = s[:max-1] + "…"
+	}
+	return s
+}

.github/actions/e2e-analyze/analyze_test.go

@@ -0,0 +1,191 @@
+package main
+
+import (
+	"flag"
+	"os"
+	"path/filepath"
+	"sort"
+	"strings"
+	"testing"
+)
+
+// updateGolden lets `go test ./.github/actions/e2e-analyze/ -update` rewrite
+// the expected analysis.md files when the rendering format changes
+// intentionally. Day-to-day, leave it unset: tests diff against the on-disk
+// snapshot to catch unintended drift in signature extraction or layout.
+var updateGolden = flag.Bool("update", false, "rewrite testdata/*/analysis.md from current output")
+
+// TestGoldenFixtures runs the full ParseFile + Render pipeline against each
+// fixture directory under testdata/ and asserts the output matches the
+// adjacent analysis.md snapshot. Each subdir is one named scenario; the
+// fixture XMLs match the same junit-*.xml glob the production action uses
+// so behaviour stays representative of CI.
+func TestGoldenFixtures(t *testing.T) {
+	entries, err := os.ReadDir("testdata")
+	if err != nil {
+		t.Fatalf("read testdata: %v", err)
+	}
+	for _, e := range entries {
+		if !e.IsDir() {
+			continue
+		}
+		name := e.Name()
+		t.Run(name, func(t *testing.T) {
+			dir := filepath.Join("testdata", name)
+			matches, err := filepath.Glob(filepath.Join(dir, "junit-*.xml"))
+			if err != nil {
+				t.Fatalf("glob: %v", err)
+			}
+			sort.Strings(matches)
+			if len(matches) == 0 {
+				t.Fatalf("no junit-*.xml fixtures in %s", dir)
+			}
+
+			rep := Report{Title: "E2E failure analysis"}
+			for _, p := range matches {
+				data, err := os.ReadFile(p)
+				if err != nil {
+					t.Fatalf("read %s: %v", p, err)
+				}
+				sr, err := ParseFile(p, data)
+				if err != nil {
+					t.Fatalf("parse %s: %v", p, err)
+				}
+				rep.Suites = append(rep.Suites, sr)
+			}
+
+			got := Render(rep)
+			goldenPath := filepath.Join(dir, "analysis.md")
+			if *updateGolden {
+				if err := os.WriteFile(goldenPath, []byte(got), 0o644); err != nil {
+					t.Fatalf("write golden: %v", err)
+				}
+				return
+			}
+			want, err := os.ReadFile(goldenPath)
+			if err != nil {
+				t.Fatalf("read golden (run with -update to seed): %v", err)
+			}
+			if string(want) != got {
+				t.Errorf("rendered output differs from %s\n--- want ---\n%s\n--- got ---\n%s",
+					goldenPath, string(want), got)
+			}
+		})
+	}
+}
+
+func TestSignatureCollapsesNoise(t *testing.T) {
+	a := signature("describe-vpcs: InvalidParameterValue request-id: abc-123 vpc-deadbeefcafe1234")
+	b := signature("describe-vpcs: InvalidParameterValue request-id: xyz-999 vpc-1111222233334444")
+	if a != b {
+		t.Errorf("signatures should collapse to same bucket:\n  a=%q\n  b=%q", a, b)
+	}
+	if !strings.Contains(a, "<id>") || !strings.Contains(a, "vpc-<id>") {
+		t.Errorf("expected normalised placeholders in signature, got %q", a)
+	}
+}
+
+func TestCascadeDetection(t *testing.T) {
+	cases := []struct {
+		in   string
+		want bool
+	}{
+		{"Phase 5 must populate fix.InstanceID", true},
+		{"Should NOT be empty", true},
+		{"Expected value not to be nil", true},
+		{"describe-vpcs: InvalidParameterValue", false},
+		{"Eventually: condition not met within 3m0s", false},
+	}
+	for _, c := range cases {
+		if got := isCascade(c.in); got != c.want {
+			t.Errorf("isCascade(%q) = %v, want %v", c.in, got, c.want)
+		}
+	}
+}
+
+func TestExtractErrorLine_MessagesWinsOverError(t *testing.T) {
+	body := `    lifecycle_test.go:28:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:28
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5a_pre_ClusterStats`
+	got := extractErrorLine(body)
+	want := "Phase 5 must populate fix.InstanceID"
+	if got != want {
+		t.Errorf("extractErrorLine = %q, want %q", got, want)
+	}
+}
+
+func TestExtractErrorLine_FallsBackToLastTestLine(t *testing.T) {
+	body := "    vpc_test.go:227: Eventually: condition not met within 3m0s: [SSH handshake never completed]"
+	got := extractErrorLine(body)
+	if !strings.HasPrefix(got, "Eventually:") {
+		t.Errorf("expected fallback to test-line content, got %q", got)
+	}
+}
+
+func TestExtractFileHint_PicksLastMatch(t *testing.T) {
+	body := "ec2helpers.go:50: setup\n    vpc_test.go:227: Eventually: condition not met"
+	got := extractFileHint(body)
+	want := "vpc_test.go:227"
+	if got != want {
+		t.Errorf("extractFileHint = %q, want %q", got, want)
+	}
+}
+
+func TestParseFile_SkipsEmptyParentFailure(t *testing.T) {
+	// The synthetic "TestSingleNode" parent gets a <failure message="Failed"/>
+	// with no body whenever any subtest fails. The analyzer must skip it so
+	// it doesn't outrank the real first failure on XML order.
+	xml := []byte(`<?xml version="1.0"?>
+<testsuites><testsuite name="" tests="2" failures="2">
+  <testcase name="TestX" time="1.0"><failure message="Failed"></failure></testcase>
+  <testcase name="TestX/RealFailure" time="0.5"><failure message="Failed"><![CDATA[
+    foo_test.go:10:
+        Messages: real assertion message
+]]></failure></testcase>
+</testsuite></testsuites>`)
+	sr, err := ParseFile("junit-x.xml", xml)
+	if err != nil {
+		t.Fatalf("ParseFile: %v", err)
+	}
+	if sr.FailCount != 1 {
+		t.Fatalf("FailCount = %d, want 1 (empty parent should be skipped)", sr.FailCount)
+	}
+	if sr.Root == nil || sr.Root.Name != "TestX/RealFailure" {
+		t.Fatalf("expected real failure as root, got %+v", sr.Root)
+	}
+}
+
+func TestParseFile_CascadeFallbackWhenAllCascades(t *testing.T) {
+	// If every failure is a cascade marker, the earliest one becomes root
+	// so the report isn't blank.
+	xml := []byte(`<?xml version="1.0"?>
+<testsuites><testsuite name="" tests="2" failures="2">
+  <testcase name="TestX/A" time="1.0"><failure message="Failed"><![CDATA[
+    Messages: Phase 5 must populate fix.InstanceID
+]]></failure></testcase>
+  <testcase name="TestX/B" time="1.0"><failure message="Failed"><![CDATA[
+    Messages: Phase 5 must populate fix.InstanceID
+]]></failure></testcase>
+</testsuite></testsuites>`)
+	sr, err := ParseFile("junit-x.xml", xml)
+	if err != nil {
+		t.Fatalf("ParseFile: %v", err)
+	}
+	if sr.Root == nil || sr.Root.Name != "TestX/A" {
+		t.Fatalf("expected TestX/A as fallback root, got %+v", sr.Root)
+	}
+}
+
+func TestRender_NoFailuresIsClean(t *testing.T) {
+	out := Render(Report{
+		Title: "E2E failure analysis",
+		Suites: []SuiteReport{
+			{File: "junit-single.xml", Label: "single", Total: 3, FailCount: 0},
+		},
+	})
+	if !strings.Contains(out, "No failures") {
+		t.Errorf("expected zero-failure banner, got:\n%s", out)
+	}
+}

.github/actions/e2e-analyze/bundle.go

@@ -0,0 +1,208 @@
+package main
+
+// Stage 2 of docs/development/improvements/e2e-go-failure-analysis.md:
+// for each failure surfaced by Stage 1, materialise a per-failure bundle
+// directory containing the failure summary, a time-window slice of the
+// daemon journal, a testcase-window slice of the suite's stdout log, and
+// any per-test artifacts the harness already dumped at run-time.
+
+import (
+	"errors"
+	"fmt"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"regexp"
+	"strings"
+	"time"
+)
+
+// bundlePad is the per-failure window padding either side of the
+// [StartAt, StartAt+Duration] testcase span. Five seconds matches the
+// plan doc: long enough to catch the daemon's pre-test setup and
+// post-test cleanup chatter, short enough to keep the slice focused.
+const bundlePad = 5 * time.Second
+
+// nameSanitiseRe replaces characters unsafe in directory names with `_`.
+// Tracks tests/e2e/harness/artifacts.go which uses the same convention
+// when the test's own dump dir is created, so we can pick its contents
+// up by path without a second translation table.
+var nameSanitiseRe = regexp.MustCompile(`[^A-Za-z0-9._-]+`)
+
+func sanitiseTestName(s string) string {
+	return nameSanitiseRe.ReplaceAllString(s, "_")
+}
+
+// ApplySuiteStartFiles overrides each failure's StartAt by reading
+// <logDir>/test-<suiteLabel>.start (RFC3339 timestamp written by the
+// workflow before the suite runs). Required because go-junit-report's
+// `timestamp` attribute on <testsuite> records the time the XML was
+// produced — i.e. after the suite has finished — so it can't be used
+// as the suite wall-clock origin for journal slicing.
+//
+// Missing or unparseable files are silently ignored: the analyzer
+// falls back to the junit timestamp and the bundle still gets created,
+// it just may have an empty (or off-window) journal slice.
+func ApplySuiteStartFiles(rep *Report, logDir string) {
+	if logDir == "" {
+		return
+	}
+	for si := range rep.Suites {
+		s := &rep.Suites[si]
+		startPath := filepath.Join(logDir, fmt.Sprintf("test-%s.start", s.Label))
+		data, err := os.ReadFile(startPath)
+		if err != nil {
+			continue
+		}
+		ts, err := time.Parse(time.RFC3339, strings.TrimSpace(string(data)))
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: parse %s: %v\n", startPath, err)
+			continue
+		}
+		if s.Root != nil {
+			s.Root.StartAt = ts.Add(s.Root.OffsetFromSuiteStart)
+		}
+		for ci := range s.Cascades {
+			s.Cascades[ci].StartAt = ts.Add(s.Cascades[ci].OffsetFromSuiteStart)
+		}
+		for bi := range s.Buckets {
+			for fi := range s.Buckets[bi] {
+				f := &s.Buckets[bi][fi]
+				f.StartAt = ts.Add(f.OffsetFromSuiteStart)
+			}
+		}
+	}
+}
+
+// WriteBundles materialises per-failure bundles under <logDir>/analysis/.
+// Mutates rep in-place so each Failure.BundlePath holds the relative
+// path the renderer will link from the summary.
+//
+// Failures with no parseable wall-clock start still get a bundle dir
+// (and the testcase-window test.log slice), but no journal slice — the
+// time window is unknown.
+//
+// Errors are non-fatal: a bundle that partially fails still gets linked
+// from the report so a reader sees what's available, and the analyzer
+// keeps printing warnings so CI logs surface the gap.
+func WriteBundles(rep *Report, logDir string) {
+	if logDir == "" {
+		return
+	}
+	base := filepath.Join(logDir, "analysis")
+	if err := os.MkdirAll(base, 0o755); err != nil {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: mkdir %s: %v\n", base, err)
+		return
+	}
+
+	for si := range rep.Suites {
+		s := &rep.Suites[si]
+		if s.Root != nil {
+			writeOneBundle(logDir, base, s.Label, s.Root)
+		}
+		for ci := range s.Cascades {
+			writeOneBundle(logDir, base, s.Label, &s.Cascades[ci])
+		}
+		// Bucket entries share storage with Cascades but live in a
+		// separate slice in SuiteReport; mirror the bundle path so the
+		// renderer's bucket loop sees it too.
+		for bi := range s.Buckets {
+			for fi := range s.Buckets[bi] {
+				f := &s.Buckets[bi][fi]
+				if bp := findBundlePath(s, f.Name); bp != "" {
+					f.BundlePath = bp
+				}
+			}
+		}
+	}
+}
+
+// findBundlePath looks up a Failure's already-written BundlePath by test
+// name within a suite. Buckets reference the same logical failure as
+// Cascades but are independent slice values, so we copy the path across
+// rather than re-deriving it.
+func findBundlePath(s *SuiteReport, testName string) string {
+	if s.Root != nil && s.Root.Name == testName {
+		return s.Root.BundlePath
+	}
+	for _, c := range s.Cascades {
+		if c.Name == testName {
+			return c.BundlePath
+		}
+	}
+	return ""
+}
+
+func writeOneBundle(logDir, base, suiteLabel string, f *Failure) {
+	name := fmt.Sprintf("%02d-%s", f.Order+1, sanitiseTestName(f.Name))
+	dir := filepath.Join(base, name)
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: mkdir %s: %v\n", dir, err)
+		return
+	}
+	f.BundlePath = filepath.Join("analysis", name)
+
+	writeFailureSummary(dir, suiteLabel, f)
+
+	if !f.StartAt.IsZero() {
+		start := f.StartAt.Add(-bundlePad)
+		end := f.StartAt.Add(time.Duration(f.Duration * float64(time.Second))).Add(bundlePad)
+		journalSrc := filepath.Join(logDir, "spinifex-journal.log")
+		if _, err := SliceJournal(journalSrc, start, end, filepath.Join(dir, "journal.log")); err != nil {
+			fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: slice journal for %s: %v\n", f.Name, err)
+		}
+	}
+
+	testLogSrc := filepath.Join(logDir, fmt.Sprintf("test-%s.log", suiteLabel))
+	if _, err := SliceTestLog(testLogSrc, f.Name, filepath.Join(dir, "test.log")); err != nil {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: slice test log for %s: %v\n", f.Name, err)
+	}
+
+	// Per-test artifact subdir written by harness.ArtifactDir lives at
+	// <logDir>/<TestName with / replaced by _>/. Copy any files into
+	// the bundle so qemu/ovs/etc. dumps already collected by the test
+	// travel alongside the analysis.
+	perTestSrc := filepath.Join(logDir, strings.ReplaceAll(f.Name, "/", "_"))
+	if entries, err := os.ReadDir(perTestSrc); err == nil {
+		for _, e := range entries {
+			if e.IsDir() {
+				continue
+			}
+			_ = copyFile(filepath.Join(perTestSrc, e.Name()), filepath.Join(dir, e.Name()))
+		}
+	} else if !os.IsNotExist(err) && !errorsIsNotDir(err) {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: read per-test dir %s: %v\n", perTestSrc, err)
+	}
+}
+
+// errorsIsNotDir handles the case where the harness wrote a file at the
+// per-test name instead of a directory — best-effort skip.
+func errorsIsNotDir(err error) bool {
+	var pe *fs.PathError
+	if !errors.As(err, &pe) {
+		return false
+	}
+	return pe.Err != nil && strings.Contains(pe.Err.Error(), "not a directory")
+}
+
+func writeFailureSummary(dir, suiteLabel string, f *Failure) {
+	var b strings.Builder
+	fmt.Fprintf(&b, "Suite:    %s\n", suiteLabel)
+	fmt.Fprintf(&b, "Test:     %s\n", f.Name)
+	if !f.StartAt.IsZero() {
+		fmt.Fprintf(&b, "Start:    %s\n", f.StartAt.Format(time.RFC3339))
+	}
+	fmt.Fprintf(&b, "Duration: %.3fs\n", f.Duration)
+	if f.FileHint != "" {
+		fmt.Fprintf(&b, "Site:     %s\n", f.FileHint)
+	}
+	if f.Cascade {
+		b.WriteString("Cascade:  yes (downstream of another failure)\n")
+	}
+	if f.Error != "" {
+		b.WriteString("\nError:\n")
+		b.WriteString(oneLine(f.Error))
+		b.WriteString("\n")
+	}
+	_ = os.WriteFile(filepath.Join(dir, "failure.txt"), []byte(b.String()), 0o644)
+}

.github/actions/e2e-analyze/bundle_test.go

@@ -0,0 +1,229 @@
+package main
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+func TestSliceJournal_KeepsOnlyWindow(t *testing.T) {
+	src := filepath.Join(t.TempDir(), "journal.log")
+	dst := filepath.Join(t.TempDir(), "out.log")
+	body := strings.Join([]string{
+		"2026-05-19T12:32:30+1000 host spinifex-daemon[1]: pre",
+		"  continuation of pre",
+		"2026-05-19T12:32:35+1000 host spinifex-daemon[1]: IN window",
+		"  multiline inside window",
+		"2026-05-19T12:32:45+1000 host spinifex-daemon[1]: post",
+		"",
+	}, "\n")
+	if err := os.WriteFile(src, []byte(body), 0o644); err != nil {
+		t.Fatalf("write src: %v", err)
+	}
+	start, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:33+10:00")
+	end, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:40+10:00")
+	n, err := SliceJournal(src, start, end, dst)
+	if err != nil {
+		t.Fatalf("SliceJournal: %v", err)
+	}
+	if n != 2 {
+		t.Fatalf("written = %d, want 2", n)
+	}
+	got, _ := os.ReadFile(dst)
+	if !strings.Contains(string(got), "IN window") || !strings.Contains(string(got), "multiline inside window") {
+		t.Errorf("missing in-window content:\n%s", got)
+	}
+	if strings.Contains(string(got), "pre") || strings.Contains(string(got), "post") {
+		t.Errorf("kept out-of-window content:\n%s", got)
+	}
+}
+
+func TestSliceJournal_AcceptsRFC3339ColonOffset(t *testing.T) {
+	// systemd's `journalctl --output=short-iso` ships RFC3339 timestamps
+	// (offset includes a colon, e.g. "+10:00") on most versions. Earlier
+	// regex only matched "+1000" — make sure both forms work.
+	src := filepath.Join(t.TempDir(), "journal.log")
+	dst := filepath.Join(t.TempDir(), "out.log")
+	body := strings.Join([]string{
+		"2026-05-19T12:32:30+10:00 host spinifex-daemon[1]: pre",
+		"2026-05-19T12:32:35+10:00 host spinifex-daemon[1]: IN window",
+		"2026-05-19T12:32:50+10:00 host spinifex-daemon[1]: post",
+		"",
+	}, "\n")
+	if err := os.WriteFile(src, []byte(body), 0o644); err != nil {
+		t.Fatalf("write src: %v", err)
+	}
+	start, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:33+10:00")
+	end, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:40+10:00")
+	n, err := SliceJournal(src, start, end, dst)
+	if err != nil {
+		t.Fatalf("SliceJournal: %v", err)
+	}
+	if n != 1 {
+		t.Fatalf("written = %d, want 1", n)
+	}
+	got, _ := os.ReadFile(dst)
+	if !strings.Contains(string(got), "IN window") {
+		t.Errorf("missing in-window content:\n%s", got)
+	}
+}
+
+func TestSliceJournal_MissingSrcIsNotError(t *testing.T) {
+	dst := filepath.Join(t.TempDir(), "out.log")
+	n, err := SliceJournal("/no/such/file", time.Now(), time.Now(), dst)
+	if err != nil {
+		t.Fatalf("unexpected err: %v", err)
+	}
+	if n != 0 {
+		t.Errorf("written = %d, want 0", n)
+	}
+}
+
+func TestSliceTestLog_PicksOneTestcase(t *testing.T) {
+	src := filepath.Join(t.TempDir(), "test.log")
+	dst := filepath.Join(t.TempDir(), "out.log")
+	body := strings.Join([]string{
+		"=== RUN   TestSingleNode",
+		"  before subtests",
+		"=== RUN   TestSingleNode/Phase5_LaunchInstance",
+		"    runinstances",
+		"    ec2helpers.go:164: describe-vpcs: InvalidParameterValue",
+		"--- FAIL: TestSingleNode/Phase5_LaunchInstance (2.50s)",
+		"=== RUN   TestSingleNode/Phase6_Volumes",
+		"    other content",
+		"--- FAIL: TestSingleNode/Phase6_Volumes (0.10s)",
+		"--- FAIL: TestSingleNode (10.0s)",
+		"",
+	}, "\n")
+	if err := os.WriteFile(src, []byte(body), 0o644); err != nil {
+		t.Fatalf("write src: %v", err)
+	}
+	n, err := SliceTestLog(src, "TestSingleNode/Phase5_LaunchInstance", dst)
+	if err != nil {
+		t.Fatalf("SliceTestLog: %v", err)
+	}
+	if n != 4 {
+		t.Fatalf("written = %d, want 4 (RUN + 2 inner + FAIL)", n)
+	}
+	got, _ := os.ReadFile(dst)
+	s := string(got)
+	if !strings.Contains(s, "describe-vpcs: InvalidParameterValue") {
+		t.Errorf("missing failure line:\n%s", s)
+	}
+	if strings.Contains(s, "Phase6_Volumes") || strings.Contains(s, "before subtests") {
+		t.Errorf("leaked sibling test content:\n%s", s)
+	}
+}
+
+func TestSliceTestLog_UnknownTestcaseIsEmpty(t *testing.T) {
+	src := filepath.Join(t.TempDir(), "test.log")
+	dst := filepath.Join(t.TempDir(), "out.log")
+	if err := os.WriteFile(src, []byte("=== RUN   TestX\n--- PASS: TestX (0.0s)\n"), 0o644); err != nil {
+		t.Fatalf("write src: %v", err)
+	}
+	n, err := SliceTestLog(src, "TestY", dst)
+	if err != nil {
+		t.Fatalf("SliceTestLog: %v", err)
+	}
+	if n != 0 {
+		t.Errorf("written = %d, want 0", n)
+	}
+	if _, err := os.Stat(dst); err != nil {
+		t.Errorf("dst not created: %v", err)
+	}
+}
+
+// TestWriteBundles_EndToEnd builds a fake artifact dir with a junit XML,
+// a journal log, a test log, and a per-test artifact subdir, then runs
+// the full ParseFile → WriteBundles pipeline and asserts the bundle dir
+// contains everything the report links to.
+func TestWriteBundles_EndToEnd(t *testing.T) {
+	dir := t.TempDir()
+
+	junit := `<?xml version="1.0"?>
+<testsuites><testsuite name="" tests="2" failures="1" timestamp="2026-05-19T12:32:30+10:00" time="3.0">
+  <testcase name="TestSingleNode/Phase5_LaunchInstance" time="2.500">
+    <failure message="Failed"><![CDATA[
+    ec2helpers.go:164: describe-vpcs: InvalidParameterValue (status 400)
+        Error Trace: tests/e2e/harness/ec2helpers.go:164
+        Test: TestSingleNode/Phase5_LaunchInstance]]></failure>
+  </testcase>
+</testsuite></testsuites>`
+	if err := os.WriteFile(filepath.Join(dir, "junit-single.xml"), []byte(junit), 0o644); err != nil {
+		t.Fatalf("write junit: %v", err)
+	}
+
+	journal := strings.Join([]string{
+		"2026-05-19T12:32:20+1000 host spinifex-daemon[1]: too-early",
+		"2026-05-19T12:32:30+1000 host spinifex-daemon[1]: in-window-start",
+		"2026-05-19T12:32:32+1000 host spinifex-daemon[1]: describe-vpcs filter=is_default",
+		"2026-05-19T12:32:50+1000 host spinifex-daemon[1]: too-late",
+		"",
+	}, "\n")
+	if err := os.WriteFile(filepath.Join(dir, "spinifex-journal.log"), []byte(journal), 0o644); err != nil {
+		t.Fatalf("write journal: %v", err)
+	}
+
+	testlog := strings.Join([]string{
+		"=== RUN   TestSingleNode/Phase5_LaunchInstance",
+		"    ec2helpers.go:164: describe-vpcs: InvalidParameterValue",
+		"--- FAIL: TestSingleNode/Phase5_LaunchInstance (2.50s)",
+		"",
+	}, "\n")
+	if err := os.WriteFile(filepath.Join(dir, "test-single.log"), []byte(testlog), 0o644); err != nil {
+		t.Fatalf("write test log: %v", err)
+	}
+
+	// Per-test artifact subdir (matches harness.ArtifactDir convention).
+	perTest := filepath.Join(dir, "TestSingleNode_Phase5_LaunchInstance")
+	if err := os.MkdirAll(perTest, 0o755); err != nil {
+		t.Fatalf("mkdir per-test: %v", err)
+	}
+	if err := os.WriteFile(filepath.Join(perTest, "qemu-console.log"), []byte("[qemu] boot\n"), 0o644); err != nil {
+		t.Fatalf("write per-test file: %v", err)
+	}
+
+	data, _ := os.ReadFile(filepath.Join(dir, "junit-single.xml"))
+	sr, err := ParseFile("junit-single.xml", data)
+	if err != nil {
+		t.Fatalf("ParseFile: %v", err)
+	}
+	rep := Report{Title: "E2E failure analysis", Suites: []SuiteReport{sr}}
+	WriteBundles(&rep, dir)
+
+	if rep.Suites[0].Root == nil {
+		t.Fatal("no root")
+	}
+	bp := rep.Suites[0].Root.BundlePath
+	if bp == "" {
+		t.Fatal("BundlePath not set")
+	}
+	bundle := filepath.Join(dir, bp)
+	for _, want := range []string{"failure.txt", "journal.log", "test.log", "qemu-console.log"} {
+		if _, err := os.Stat(filepath.Join(bundle, want)); err != nil {
+			t.Errorf("missing %s in bundle %s: %v", want, bp, err)
+		}
+	}
+
+	jl, _ := os.ReadFile(filepath.Join(bundle, "journal.log"))
+	if !strings.Contains(string(jl), "describe-vpcs filter=is_default") {
+		t.Errorf("journal.log lacks in-window line:\n%s", jl)
+	}
+	if strings.Contains(string(jl), "too-early") || strings.Contains(string(jl), "too-late") {
+		t.Errorf("journal.log includes out-of-window content:\n%s", jl)
+	}
+
+	tl, _ := os.ReadFile(filepath.Join(bundle, "test.log"))
+	if !strings.Contains(string(tl), "--- FAIL: TestSingleNode/Phase5_LaunchInstance") {
+		t.Errorf("test.log missing FAIL marker:\n%s", tl)
+	}
+
+	out := Render(rep)
+	wantLink := fmt.Sprintf("- Bundle: `%s/`", bp)
+	if !strings.Contains(out, wantLink) {
+		t.Errorf("render missing bundle link %q:\n%s", wantLink, out)
+	}
+}

.github/actions/e2e-analyze/main.go

@@ -0,0 +1,78 @@
+package main
+
+import (
+	"flag"
+	"fmt"
+	"os"
+	"path/filepath"
+	"sort"
+)
+
+func main() {
+	var junitGlob, logDir, title string
+	flag.StringVar(&junitGlob, "junit-glob", os.Getenv("ANALYZE_JUNIT_GLOB"), "glob matching junit-*.xml files")
+	flag.StringVar(&logDir, "log-dir", os.Getenv("ANALYZE_LOG_DIR"), "artifact dir to write analysis.md into")
+	flag.StringVar(&title, "title", os.Getenv("ANALYZE_TITLE"), "report heading")
+	flag.Parse()
+
+	if title == "" {
+		title = "E2E failure analysis"
+	}
+	if junitGlob == "" {
+		fmt.Fprintln(os.Stderr, "::error::e2e-analyze: -junit-glob is required")
+		os.Exit(0) // analyzer is advisory only; existing gate owns exit code
+	}
+
+	paths, err := filepath.Glob(junitGlob)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: bad glob %q: %v\n", junitGlob, err)
+		os.Exit(0)
+	}
+	sort.Strings(paths)
+
+	report := Report{Title: title}
+	for _, p := range paths {
+		data, err := os.ReadFile(p)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: read %s: %v\n", p, err)
+			continue
+		}
+		sr, err := ParseFile(p, data)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: %v\n", err)
+			continue
+		}
+		report.Suites = append(report.Suites, sr)
+	}
+
+	// Stage 2: prefer authoritative per-suite start times written by the
+	// workflow over junit-report's conversion-time `timestamp` attribute,
+	// then materialise per-failure bundles so Render can link them.
+	if logDir != "" {
+		ApplySuiteStartFiles(&report, logDir)
+		WriteBundles(&report, logDir)
+	}
+
+	out := Render(report)
+
+	if logDir != "" {
+		if err := os.MkdirAll(logDir, 0o755); err == nil {
+			if err := os.WriteFile(filepath.Join(logDir, "analysis.md"), []byte(out), 0o644); err != nil {
+				fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: write analysis.md: %v\n", err)
+			}
+		}
+	}
+
+	if summaryPath := os.Getenv("GITHUB_STEP_SUMMARY"); summaryPath != "" {
+		f, err := os.OpenFile(summaryPath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
+		if err == nil {
+			defer f.Close()
+			if _, werr := f.WriteString(out); werr != nil {
+				fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: append summary: %v\n", werr)
+			}
+		}
+	} else {
+		// Local invocation (developer running the tool by hand). Mirror to stdout.
+		fmt.Print(out)
+	}
+}

.github/actions/e2e-analyze/slice.go

@@ -0,0 +1,176 @@
+package main
+
+// Stage 2 of docs/development/improvements/e2e-go-failure-analysis.md:
+// per-failure log slicers used by bundle.go to materialise time-window /
+// testcase-window log excerpts next to each named failure.
+
+import (
+	"bufio"
+	"io"
+	"os"
+	"regexp"
+	"strings"
+	"time"
+)
+
+// shortIsoTSRe matches the timestamp prefix produced by
+// `journalctl --output=short-iso`. The offset format varies across
+// systemd versions: most ship "+10:00" (RFC3339), some ship "+1000"
+// (no colon). Match either — the parser below picks the right layout.
+var shortIsoTSRe = regexp.MustCompile(`^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:[+-]\d{2}:?\d{2}|Z))\s`)
+
+// shortIsoLayouts are tried in order; the first parse that succeeds wins.
+var shortIsoLayouts = []string{
+	time.RFC3339,               // 2026-05-19T12:32:30+10:00
+	"2006-01-02T15:04:05-0700", // 2026-05-19T12:32:30+1000
+}
+
+func parseShortIso(s string) (time.Time, error) {
+	var lastErr error
+	for _, layout := range shortIsoLayouts {
+		t, err := time.Parse(layout, s)
+		if err == nil {
+			return t, nil
+		}
+		lastErr = err
+	}
+	return time.Time{}, lastErr
+}
+
+// SliceJournal copies lines from src whose `short-iso` timestamp falls in
+// [start, end] to dst. Lines without a parseable timestamp inherit the
+// previous line's classification — that keeps multi-line journal records
+// (stack traces, panics) intact rather than orphaning their continuation
+// lines. Lines that precede the first parseable timestamp are dropped.
+//
+// Returns the number of lines written; missing src is not an error (the
+// journal may be empty if collection failed).
+func SliceJournal(src string, start, end time.Time, dst string) (int, error) {
+	f, err := os.Open(src)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return 0, nil
+		}
+		return 0, err
+	}
+	defer f.Close()
+
+	out, err := os.Create(dst)
+	if err != nil {
+		return 0, err
+	}
+	defer out.Close()
+
+	bw := bufio.NewWriter(out)
+	defer bw.Flush()
+
+	sc := bufio.NewScanner(f)
+	// Journal lines can be long (panic dumps); bump scanner buffer.
+	sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
+
+	keep := false
+	written := 0
+	for sc.Scan() {
+		line := sc.Text()
+		if m := shortIsoTSRe.FindStringSubmatch(line); m != nil {
+			ts, err := parseShortIso(m[1])
+			if err != nil {
+				continue
+			}
+			keep = !ts.Before(start) && !ts.After(end)
+		}
+		if keep {
+			if _, err := bw.WriteString(line); err != nil {
+				return written, err
+			}
+			if err := bw.WriteByte('\n'); err != nil {
+				return written, err
+			}
+			written++
+		}
+	}
+	return written, sc.Err()
+}
+
+// runRePrefix and finishRePrefix bracket a single testcase's output in
+// `go test -v` stdout. Subtests have their own `=== RUN` / `--- FAIL`
+// markers, so slicing on the exact testcase name yields just that test's
+// lines (including any nested phase logs).
+var (
+	runRe    = regexp.MustCompile(`^=== RUN\s+(\S+)$`)
+	finishRe = regexp.MustCompile(`^\s*---\s+(PASS|FAIL|SKIP):\s+(\S+)\s+\(`)
+)
+
+// SliceTestLog copies lines belonging to the named testcase from src to
+// dst. The block starts at `=== RUN <name>` and ends at the matching
+// `--- PASS|FAIL|SKIP: <name> (…)` line (inclusive). If the testcase is
+// never seen the destination file is created empty so the bundle has a
+// stable shape.
+//
+// Returns the number of lines written; missing src is not an error.
+func SliceTestLog(src, testName, dst string) (int, error) {
+	f, err := os.Open(src)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return 0, nil
+		}
+		return 0, err
+	}
+	defer f.Close()
+
+	out, err := os.Create(dst)
+	if err != nil {
+		return 0, err
+	}
+	defer out.Close()
+
+	bw := bufio.NewWriter(out)
+	defer bw.Flush()
+
+	sc := bufio.NewScanner(f)
+	sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
+
+	in := false
+	written := 0
+	for sc.Scan() {
+		line := sc.Text()
+		if !in {
+			if m := runRe.FindStringSubmatch(strings.TrimRight(line, "\r")); m != nil && m[1] == testName {
+				in = true
+			} else {
+				continue
+			}
+		}
+		if _, err := bw.WriteString(line); err != nil {
+			return written, err
+		}
+		if err := bw.WriteByte('\n'); err != nil {
+			return written, err
+		}
+		written++
+		if m := finishRe.FindStringSubmatch(line); m != nil && m[2] == testName {
+			return written, nil
+		}
+	}
+	return written, sc.Err()
+}
+
+// copyFile is a small helper used when bundling pre-collected per-test
+// artifacts. Best-effort: missing src is silently skipped.
+func copyFile(src, dst string) error {
+	in, err := os.Open(src)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return nil
+		}
+		return err
+	}
+	defer in.Close()
+	out, err := os.Create(dst)
+	if err != nil {
+		return err
+	}
+	defer out.Close()
+	_, err = io.Copy(out, in)
+	return err
+}

.github/actions/e2e-analyze/testdata/cascade-vpcs/analysis.md

@@ -0,0 +1,23 @@
+## E2E failure analysis
+
+### Suite `single`: 16 failed, 1 root cause likely
+
+**Root cause (earliest non-cascade)**
+
+- Test: `TestSingleNode/Phase5_LaunchInstance`
+- Start: 12:30:00 (duration 2.5s)
+- Error: Received unexpected error: InvalidParameterValue (HTTP 400)
+- File:  `tests/e2e/harness/ec2helpers.go:164`
+
+**Cascaded failures (15) — grouped by signature**
+
+- 14× Phase 5 must populate fix.InstanceID
+  - `TestSingleNode/Phase5a_pre_ClusterStats`
+  - `TestSingleNode/Phase5a_Metadata`
+  - `TestSingleNode/Phase5b_DescribeInstances`
+  - `TestSingleNode/Phase5c_StopStart`
+  - `TestSingleNode/Phase6_Volumes`
+  - … and 9 more
+- 1× Phase 2 must populate fix.AZName
+  - `TestSingleNode/Phase8_NatGW`
+

.github/actions/e2e-analyze/testdata/cascade-vpcs/junit-single.xml

@@ -0,0 +1,143 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<testsuites tests="20" failures="17" skipped="0">
+	<testsuite name="" tests="20" failures="17" errors="0" id="0" hostname="banksia" skipped="0" time="600.000" timestamp="2026-04-12T12:30:00+10:00">
+		<testcase name="TestSingleNode" classname="" time="600.000">
+			<failure message="Failed"></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase1_Environment" classname="" time="0.300">
+			<system-out><![CDATA[
+&#x2501;&#x2501; Phase 1 &#x2501;&#x2501;]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase2_Discovery" classname="" time="0.010"/>
+		<testcase name="TestSingleNode/Phase3_KeyPairs" classname="" time="0.540"/>
+		<testcase name="TestSingleNode/Phase4_Image" classname="" time="0.060"/>
+		<testcase name="TestSingleNode/Phase5_LaunchInstance" classname="" time="2.500">
+			<failure message="Failed"><![CDATA[
+    ec2helpers.go:164: describe-vpcs: InvalidParameterValue: Filter 'is_default' is invalid (status 400)
+    	Error Trace:	tests/e2e/harness/ec2helpers.go:164
+    	Error:      	Received unexpected error:
+    	            	InvalidParameterValue (HTTP 400)
+    	Test:       	TestSingleNode/Phase5_LaunchInstance]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_pre_ClusterStats" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    lifecycle_test.go:28:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:28
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5a_pre_ClusterStats]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_Metadata" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    lifecycle_test.go:57:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:57
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5a_Metadata]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5b_DescribeInstances" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    lifecycle_test.go:109:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:109
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5b_DescribeInstances]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5c_StopStart" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    lifecycle_test.go:222:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:222
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5c_StopStart]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase6_Volumes" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    volume_test.go:23:
+        Error Trace:	tests/e2e/single/volume_test.go:23
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase6_Volumes]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase6b_AttachVolume" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    volume_test.go:24:
+        Error Trace:	tests/e2e/single/volume_test.go:24
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase6b_AttachVolume]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7_Tags" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    tags_test.go:23:
+        Error Trace:	tests/e2e/single/tags_test.go:23
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7_Tags]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7a_Snapshots" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    snapshot_test.go:23:
+        Error Trace:	tests/e2e/single/snapshot_test.go:23
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7a_Snapshots]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7b_DescribeInstanceAttribute" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    instance_test.go:81:
+        Error Trace:	tests/e2e/single/instance_test.go:81
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7b_DescribeInstanceAttribute]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7c_ModifyInstanceAttribute" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    instance_test.go:92:
+        Error Trace:	tests/e2e/single/instance_test.go:92
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7c_ModifyInstanceAttribute]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7d_Console" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    instance_test.go:121:
+        Error Trace:	tests/e2e/single/instance_test.go:121
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7d_Console]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7e_Negative" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    negative_test.go:43:
+        Error Trace:	tests/e2e/single/negative_test.go:43
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7e_Negative]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NatGW" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    natgw_test.go:39:
+        Error Trace:	tests/e2e/single/natgw_test.go:39
+        Error:      	Should NOT be empty
+        Messages:   	Phase 2 must populate fix.AZName
+        Test:       	TestSingleNode/Phase8_NatGW]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8a_DataPath" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    datapath_test.go:48:
+        Error Trace:	tests/e2e/single/datapath_test.go:48
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase8a_DataPath]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase9_SecurityGroups" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    securitygroup_test.go:35:
+        Error Trace:	tests/e2e/single/securitygroup_test.go:35
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase9_SecurityGroups]]></failure>
+		</testcase>
+	</testsuite>
+</testsuites>

.github/actions/e2e-analyze/testdata/no-failures/analysis.md

@@ -0,0 +1,3 @@
+## E2E failure analysis
+
+✅ No failures across any suite.

.github/actions/e2e-analyze/testdata/no-failures/junit-single.xml

@@ -0,0 +1,8 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<testsuites tests="3" failures="0" skipped="0">
+	<testsuite name="" tests="3" failures="0" errors="0" id="0" hostname="banksia" skipped="0" time="1.500" timestamp="2026-04-12T12:30:00+10:00">
+		<testcase name="TestSingleNode" classname="" time="1.500"/>
+		<testcase name="TestSingleNode/Phase1_Environment" classname="" time="0.300"/>
+		<testcase name="TestSingleNode/Phase2_Discovery" classname="" time="1.200"/>
+	</testsuite>
+</testsuites>

.github/actions/e2e-analyze/testdata/single-failure/analysis.md

@@ -0,0 +1,11 @@
+## E2E failure analysis
+
+### Suite `single`: 1 failed, 1 root cause likely
+
+**Root cause (earliest non-cascade)**
+
+- Test: `TestSingleNode/Phase8b_VPCSubnetE2E`
+- Start: 00:26:18 (duration 200.5s)
+- Error: Eventually: condition not met within 3m0s: [SSH handshake 192.168.0.202:22 never completed]
+- File:  `vpc_test.go:227`
+

.github/actions/e2e-analyze/testdata/single-failure/junit-single.xml

@@ -0,0 +1,993 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<testsuites tests="79" failures="2" skipped="6">
+	<testsuite name="" tests="79" failures="2" errors="0" id="0" hostname="banksia" skipped="6" time="1708.140" timestamp="2026-05-19T00:19:45+10:00">
+		<testcase name="TestSingleNode" classname="" time="800.320">
+			<failure message="Failed"></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase1_Environment" classname="" time="0.300">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 1 — Environment ━━�[0m
+�[2m· checking /dev/kvm�[0m
+    discovery_test.go:29: �[2m· checking /dev/kvm�[0m
+�[2mkvm�[0m=writable
+    discovery_test.go:38: �[2mkvm�[0m=writable
+�[2m· waiting for AWS gateway�[0m
+    discovery_test.go:45: �[2m· waiting for AWS gateway�[0m
+�[2mgateway�[0m=https://192.168.0.10:9999/
+    discovery_test.go:55: �[2mgateway�[0m=https://192.168.0.10:9999/
+�[2m· waiting for daemon NATS subscriptions�[0m
+    discovery_test.go:60: �[2m· waiting for daemon NATS subscriptions�[0m
+�[2m· describe-regions / describe-availability-zones�[0m
+    discovery_test.go:73: �[2m· describe-regions / describe-availability-zones�[0m
+�[2maz�[0m=ap-southeast-2a �[2mregion�[0m=ap-southeast-2
+    discovery_test.go:84: �[2maz�[0m=ap-southeast-2a �[2mregion�[0m=ap-southeast-2]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase1b_ClusterStats" classname="" time="9.200">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 1b — Cluster Stats CLI ━━�[0m
+�[2m· spx get nodes�[0m
+    discovery_test.go:101: �[2m· spx get nodes�[0m
+�[2m· spx top nodes�[0m
+    discovery_test.go:105: �[2m· spx top nodes�[0m
+�[2m· spx get vms (empty before any launches)�[0m
+    discovery_test.go:112: �[2m· spx get vms (empty before any launches)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase2_Discovery" classname="" time="0.010">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 2 — Discovery & Metadata ━━�[0m
+�[2m· discovering nano instance type�[0m
+    discovery_test.go:133: �[2m· discovering nano instance type�[0m
+�[2minstance_type�[0m=t3.nano �[2march�[0m=x86_64 �[2maz�[0m=ap-southeast-2a
+    discovery_test.go:159: �[2minstance_type�[0m=t3.nano �[2march�[0m=x86_64 �[2maz�[0m=ap-southeast-2a]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase2b_SerialConsole" classname="" time="0.020">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 2b — Serial Console Access ━━�[0m
+�[2m· default state should be disabled�[0m
+    discovery_test.go:168: �[2m· default state should be disabled�[0m
+�[2m· enable�[0m
+    discovery_test.go:173: �[2m· enable�[0m
+�[2mstate�[0m=enabled
+    discovery_test.go:180: �[2mstate�[0m=enabled
+�[2m· disable�[0m
+    discovery_test.go:182: �[2m· disable�[0m
+�[2mstate�[0m=disabled
+    discovery_test.go:189: �[2mstate�[0m=disabled]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase3_KeyPairs" classname="" time="0.540">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 3 — SSH Key Management ━━�[0m
+�[2m· create-key-pair "test-key-1"�[0m
+    keypair_test.go:37: �[2m· create-key-pair "test-key-1"�[0m
+�[2mkey�[0m=test-key-1 �[2mpem�[0m=/tmp/TestSingleNode1289015487/001/test-key-1.pem
+    keypair_test.go:51: �[2mkey�[0m=test-key-1 �[2mpem�[0m=/tmp/TestSingleNode1289015487/001/test-key-1.pem
+�[2m· import-key-pair "test-key-2" (local-generated RSA)�[0m
+    keypair_test.go:58: �[2m· import-key-pair "test-key-2" (local-generated RSA)�[0m
+�[2mimported�[0m=test-key-2
+    keypair_test.go:65: �[2mimported�[0m=test-key-2
+�[2m· describe-key-pairs (both present)�[0m
+    keypair_test.go:67: �[2m· describe-key-pairs (both present)�[0m
+�[2m· delete "test-key-2"�[0m
+    keypair_test.go:72: �[2m· delete "test-key-2"�[0m
+�[2m· describe-key-pairs (only test-key-1 remains)�[0m
+    keypair_test.go:76: �[2m· describe-key-pairs (only test-key-1 remains)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase4_Image" classname="" time="0.060">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 4 — Image Management ━━�[0m
+�[2mimage�[0m=ubuntu-24.04-x86_64 �[2march�[0m=x86_64
+    image_test.go:33: �[2mimage�[0m=ubuntu-24.04-x86_64 �[2march�[0m=x86_64
+�[2m· spx admin images import --name ubuntu-24.04-x86_64�[0m
+    image_test.go:35: �[2m· spx admin images import --name ubuntu-24.04-x86_64�[0m
+�[2m· spx import did not yield AMI ID; falling back to describe-images�[0m
+    image_test.go:48: �[2m· spx import did not yield AMI ID; falling back to describe-images�[0m
+�[2m· describe-images by AMI ID (verify exactly one match)�[0m
+    image_test.go:78: �[2m· describe-images by AMI ID (verify exactly one match)�[0m
+�[2mami�[0m=ami-6f60f2b3bf04b5453 �[2mname�[0m=ami-ubuntu-24.04-x86_64
+    image_test.go:86: �[2mami�[0m=ami-6f60f2b3bf04b5453 �[2mname�[0m=ami-ubuntu-24.04-x86_64]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5_LaunchInstance" classname="" time="10.830">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5 — Instance Lifecycle ━━�[0m
+�[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msg�[0m=sg-764a7fa2a39577204 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63
+    instance_test.go:31: �[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msg�[0m=sg-764a7fa2a39577204 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63
+�[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.nano key=test-key-1�[0m
+    instance_test.go:35: �[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.nano key=test-key-1�[0m
+�[2minstance�[0m=i-0c615e5dd74050f53
+    instance_test.go:48: �[2minstance�[0m=i-0c615e5dd74050f53
+    instance_test.go:50: instance i-0c615e5dd74050f53 reached state running
+�[2mroot_volume�[0m=vol-2230138124fb51ee2 �[2mroot_device�[0m=
+    instance_test.go:70: �[2mroot_volume�[0m=vol-2230138124fb51ee2 �[2mroot_device�[0m=]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_pre_ClusterStats" classname="" time="3.040">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5a-pre — Cluster Stats CLI (with running VM) ━━�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_Metadata" classname="" time="0.510">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5a — Instance Metadata Validation ━━�[0m
+�[2mtype�[0m=t3.nano �[2mkey�[0m=test-key-1 �[2mimage�[0m=ami-6f60f2b3bf04b5453 �[2mbdms�[0m=1
+    instance_test.go:108: �[2mtype�[0m=t3.nano �[2mkey�[0m=test-key-1 �[2mimage�[0m=ami-6f60f2b3bf04b5453 �[2mbdms�[0m=1]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_ii_SSH" classname="" time="49.450">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5a-ii — SSH Connectivity & Volume Verification ━━�[0m
+�[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
+    instance_test.go:127: �[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    instance_test.go:130: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    lifecycle_test.go:384: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2m· ssh id�[0m
+    instance_test.go:136: �[2m· ssh id�[0m
+�[2m· lsblk root-volume cross-check vs API�[0m
+    instance_test.go:140: �[2m· lsblk root-volume cross-check vs API�[0m
+�[2mguest_gib�[0m=4 �[2mapi_gib�[0m=4
+    instance_test.go:159: �[2mguest_gib�[0m=4 �[2mapi_gib�[0m=4
+�[2m· ssh hostname�[0m
+    instance_test.go:161: �[2m· ssh hostname�[0m
+�[2mhostname�[0m=spinifex-vm-0c615e5d �[2mmatches_short_id�[0m=0c615e5d
+    instance_test.go:172: �[2mhostname�[0m=spinifex-vm-0c615e5d �[2mmatches_short_id�[0m=0c615e5d]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_iii_Console" classname="" time="0.010">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5a-iii — Console Output ━━�[0m
+�[2minstance�[0m=i-0c615e5dd74050f53 �[2mhas_output�[0m=true
+    instance_test.go:191: �[2minstance�[0m=i-0c615e5dd74050f53 �[2mhas_output�[0m=true]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5b_Volume" classname="" time="4.540">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5b — Volume Lifecycle (Attach/Detach) ━━�[0m
+�[2m· create-volume size=10 az=ap-southeast-2a�[0m
+    volume_test.go:26: �[2m· create-volume size=10 az=ap-southeast-2a�[0m
+�[2mvolume�[0m=vol-a885f79173e0a8030
+    volume_test.go:34: �[2mvolume�[0m=vol-a885f79173e0a8030
+    volume_test.go:39: volume vol-a885f79173e0a8030 reached state available
+�[2m· modify-volume size=20�[0m
+    volume_test.go:42: �[2m· modify-volume size=20�[0m
+�[2mresized_gib�[0m=20
+    volume_test.go:66: �[2mresized_gib�[0m=20
+�[2m· attach-volume vol-a885f79173e0a8030 -> i-0c615e5dd74050f53 as /dev/sdf�[0m
+    volume_test.go:68: �[2m· attach-volume vol-a885f79173e0a8030 -> i-0c615e5dd74050f53 as /dev/sdf�[0m
+    volume_test.go:76: volume vol-a885f79173e0a8030 reached state in-use
+�[2m· detach-volume vol-a885f79173e0a8030�[0m
+    volume_test.go:90: �[2m· detach-volume vol-a885f79173e0a8030�[0m
+    volume_test.go:96: volume vol-a885f79173e0a8030 reached state available
+�[2m· delete-volume vol-a885f79173e0a8030�[0m
+    volume_test.go:98: �[2m· delete-volume vol-a885f79173e0a8030�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5b_ii_VolumeStatus" classname="" time="0.000">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5b-ii — DescribeVolumeStatus ━━�[0m
+�[2mvolume�[0m=vol-2230138124fb51ee2 �[2mstatus�[0m=ok
+    volume_test.go:158: �[2mvolume�[0m=vol-2230138124fb51ee2 �[2mstatus�[0m=ok]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5c_Snapshot" classname="" time="0.190">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5c — Snapshot Lifecycle ━━�[0m
+�[2m· create-snapshot volume=vol-2230138124fb51ee2�[0m
+    snapshot_test.go:34: �[2m· create-snapshot volume=vol-2230138124fb51ee2�[0m
+�[2msnapshot�[0m=snap-1188f92980e6470aa �[2msize�[0m=4 �[2mstate�[0m=completed �[2mprogress�[0m=100%
+    snapshot_test.go:50: �[2msnapshot�[0m=snap-1188f92980e6470aa �[2msize�[0m=4 �[2mstate�[0m=completed �[2mprogress�[0m=100%
+    snapshot_test.go:59: snapshot snap-1188f92980e6470aa reached state completed
+�[2m· describe-snapshots snap-1188f92980e6470aa�[0m
+    snapshot_test.go:61: �[2m· describe-snapshots snap-1188f92980e6470aa�[0m
+�[2m· copy-snapshot src=snap-1188f92980e6470aa region=ap-southeast-2�[0m
+    snapshot_test.go:76: �[2m· copy-snapshot src=snap-1188f92980e6470aa region=ap-southeast-2�[0m
+�[2mcopy_snapshot�[0m=snap-cde6b5ddf467e2c0f
+    snapshot_test.go:86: �[2mcopy_snapshot�[0m=snap-cde6b5ddf467e2c0f
+    snapshot_test.go:88: snapshot snap-cde6b5ddf467e2c0f reached state completed
+�[2m· describe-snapshots snap-1188f92980e6470aa,snap-cde6b5ddf467e2c0f (both visible)�[0m
+    snapshot_test.go:90: �[2m· describe-snapshots snap-1188f92980e6470aa,snap-cde6b5ddf467e2c0f (both visible)�[0m
+�[2m· delete-snapshot snap-1188f92980e6470aa (original)�[0m
+    snapshot_test.go:107: �[2m· delete-snapshot snap-1188f92980e6470aa (original)�[0m
+�[2m· verify copy snap-cde6b5ddf467e2c0f still completed�[0m
+    snapshot_test.go:135: �[2m· verify copy snap-cde6b5ddf467e2c0f still completed�[0m
+�[2m· delete-snapshot snap-cde6b5ddf467e2c0f (copy)�[0m
+    snapshot_test.go:144: �[2m· delete-snapshot snap-cde6b5ddf467e2c0f (copy)�[0m
+�[2mdeleted_original�[0m=snap-1188f92980e6470aa
+    snapshot_test.go:151: �[2mdeleted_original�[0m=snap-1188f92980e6470aa]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5d_SnapshotBackedLaunch" classname="" time="0.010">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5d — Snapshot-Backed Instance Launch ━━�[0m
+�[2mami�[0m=ami-6f60f2b3bf04b5453 �[2msnapshot�[0m=snap-ami-6f60f2b3bf04b5453
+    snapshot_test.go:182: �[2mami�[0m=ami-6f60f2b3bf04b5453 �[2msnapshot�[0m=snap-ami-6f60f2b3bf04b5453]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5e_CreateImage" classname="" time="0.660">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5e — CreateImage Lifecycle ━━�[0m
+�[2m· create-image instance=i-0c615e5dd74050f53 name=e2e-custom-ami (no-reboot)�[0m
+    createimage_test.go:26: �[2m· create-image instance=i-0c615e5dd74050f53 name=e2e-custom-ami (no-reboot)�[0m
+�[2mcustom_ami�[0m=ami-7dcf263a412dd14c6
+    createimage_test.go:36: �[2mcustom_ami�[0m=ami-7dcf263a412dd14c6
+    createimage_test.go:38: image ami-7dcf263a412dd14c6 reached state available
+�[2m· describe-images ami-7dcf263a412dd14c6�[0m
+    createimage_test.go:40: �[2m· describe-images ami-7dcf263a412dd14c6�[0m
+�[2mcustom_ami_snapshot�[0m=snap-dde89b9ea2ef2e05a
+    createimage_test.go:67: �[2mcustom_ami_snapshot�[0m=snap-dde89b9ea2ef2e05a]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5f_SecurityGroupEgress" classname="" time="4.880">
+			<skipped message="Skipped"><![CDATA[
+�[1m�[36m━━ Phase 5f — Security Group Enforcement (egress ACL) ━━�[0m
+�[2m· discover default gateway inside VM�[0m
+    securitygroup_test.go:53: �[2m· discover default gateway inside VM�[0m
+�[2mprobe_gateway�[0m=172.31.0.1
+    securitygroup_test.go:66: �[2mprobe_gateway�[0m=172.31.0.1
+�[2m· 5f-1 baseline egress (allow-all)�[0m
+    securitygroup_test.go:77: �[2m· 5f-1 baseline egress (allow-all)�[0m
+    securitygroup_test.go:80: Phase 5f: baseline ICMP did not succeed; env may block ICMP regardless of SG
+        Output:
+        PING 172.31.0.1 (172.31.0.1) 56(84) bytes of data.
+        
+        --- 172.31.0.1 ping statistics ---
+        3 packets transmitted, 0 received, 100% packet loss, time 2044ms
+        ]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/Phase6_TagManagement" classname="" time="0.380">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 6 — Tag Management ━━�[0m
+�[2m· 6a create-tags instance=i-0c615e5dd74050f53 (Name, Environment, DeleteMe)�[0m
+    tags_test.go:41: �[2m· 6a create-tags instance=i-0c615e5dd74050f53 (Name, Environment, DeleteMe)�[0m
+�[2m· 6b describe-tags resource-id=i-0c615e5dd74050f53 want=3�[0m
+    tags_test.go:53: �[2m· 6b describe-tags resource-id=i-0c615e5dd74050f53 want=3�[0m
+�[2m· 6c create-tags volume=vol-2230138124fb51ee2 (Name, Environment)�[0m
+    tags_test.go:65: �[2m· 6c create-tags volume=vol-2230138124fb51ee2 (Name, Environment)�[0m
+�[2m· 6d describe-tags key=Environment want=2�[0m
+    tags_test.go:76: �[2m· 6d describe-tags key=Environment want=2�[0m
+�[2m· 6e describe-tags resource-type=instance want=3�[0m
+    tags_test.go:85: �[2m· 6e describe-tags resource-type=instance want=3�[0m
+�[2m· 6f overwrite instance Name -> e2e-test-updated�[0m
+    tags_test.go:95: �[2m· 6f overwrite instance Name -> e2e-test-updated�[0m
+�[2m· 6g delete-tags Key=DeleteMe (no value)�[0m
+    tags_test.go:114: �[2m· 6g delete-tags Key=DeleteMe (no value)�[0m
+�[2m· 6h delete-tags Key=Environment,Value=production (no-op)�[0m
+    tags_test.go:130: �[2m· 6h delete-tags Key=Environment,Value=production (no-op)�[0m
+�[2m· 6i delete-tags Key=Environment,Value=testing (instance only)�[0m
+    tags_test.go:148: �[2m· 6i delete-tags Key=Environment,Value=testing (instance only)�[0m
+�[2m· 6j final describe-tags resource-id=i-0c615e5dd74050f53 want=1�[0m
+    tags_test.go:171: �[2m· 6j final describe-tags resource-id=i-0c615e5dd74050f53 want=1�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7_StopStart" classname="" time="8.090">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7 — Instance State Transitions (Stop) ━━�[0m
+�[2m· stop-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:30: �[2m· stop-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:36: instance i-0c615e5dd74050f53 reached state stopped
+�[2m· reboot-instances on stopped instance (should fail)�[0m
+    lifecycle_test.go:40: �[2m· reboot-instances on stopped instance (should fail)�[0m
+�[2mstopped_reject�[0m=IncorrectInstanceState
+    lifecycle_test.go:47: �[2mstopped_reject�[0m=IncorrectInstanceState]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7a_AttachToStoppedError" classname="" time="0.300">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7a — Attach Volume to Stopped Instance (Error Path) ━━�[0m
+�[2m· create-volume size=10 az=ap-southeast-2a�[0m
+    lifecycle_test.go:60: �[2m· create-volume size=10 az=ap-southeast-2a�[0m
+�[2mtest_volume�[0m=vol-8d3e1335938a378cd
+    lifecycle_test.go:68: �[2mtest_volume�[0m=vol-8d3e1335938a378cd
+    lifecycle_test.go:82: volume vol-8d3e1335938a378cd reached state available
+�[2m· attach-volume vol-8d3e1335938a378cd -> i-0c615e5dd74050f53 (should fail)�[0m
+    lifecycle_test.go:84: �[2m· attach-volume vol-8d3e1335938a378cd -> i-0c615e5dd74050f53 (should fail)�[0m
+�[2m· delete-volume vol-8d3e1335938a378cd�[0m
+    lifecycle_test.go:94: �[2m· delete-volume vol-8d3e1335938a378cd�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7b_ModifyInstanceAttribute" classname="" time="55.620">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7b — ModifyInstanceAttribute ━━�[0m
+�[2mfrom_type�[0m=t3.nano �[2mto_type�[0m=t3.small
+    lifecycle_test.go:120: �[2mfrom_type�[0m=t3.nano �[2mto_type�[0m=t3.small
+�[2mexpected_vcpus�[0m=2 �[2mexpected_mem_mib�[0m=2048
+    lifecycle_test.go:142: �[2mexpected_vcpus�[0m=2 �[2mexpected_mem_mib�[0m=2048
+�[2m· modify-instance-attribute i-0c615e5dd74050f53 type=t3.small�[0m
+    lifecycle_test.go:144: �[2m· modify-instance-attribute i-0c615e5dd74050f53 type=t3.small�[0m
+�[2m· start-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:165: �[2m· start-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:171: instance i-0c615e5dd74050f53 reached state running
+�[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
+    lifecycle_test.go:177: �[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    lifecycle_test.go:179: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2m· ssh nproc�[0m
+    lifecycle_test.go:182: �[2m· ssh nproc�[0m
+�[2mvm_vcpus�[0m=2
+    lifecycle_test.go:188: �[2mvm_vcpus�[0m=2
+�[2m· ssh MemTotal�[0m
+    lifecycle_test.go:190: �[2m· ssh MemTotal�[0m
+�[2mvm_mem_mib�[0m=1967 �[2mthreshold_mib�[0m=1740
+    lifecycle_test.go:200: �[2mvm_mem_mib�[0m=1967 �[2mthreshold_mib�[0m=1740
+�[2m· stop-instances i-0c615e5dd74050f53 (end of 7b)�[0m
+    lifecycle_test.go:206: �[2m· stop-instances i-0c615e5dd74050f53 (end of 7b)�[0m
+    lifecycle_test.go:211: instance i-0c615e5dd74050f53 reached state stopped]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7c_pre_Reboot" classname="" time="83.480">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7c-pre — Reboot Running Instance ━━�[0m
+�[2m· start-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:226: �[2m· start-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:231: instance i-0c615e5dd74050f53 reached state running
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    lifecycle_test.go:236: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2mpre_reboot_private_ip�[0m=172.31.0.4
+    lifecycle_test.go:246: �[2mpre_reboot_private_ip�[0m=172.31.0.4
+�[2m· reboot-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:248: �[2m· reboot-instances i-0c615e5dd74050f53�[0m
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    lifecycle_test.go:279: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2m· ssh uptime�[0m
+    lifecycle_test.go:282: �[2m· ssh uptime�[0m
+�[2muptime_secs�[0m=25
+    lifecycle_test.go:288: �[2muptime_secs�[0m=25]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7c_RunInstancesMultiCount" classname="" time="47.560">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7c — RunInstances with MinCount/MaxCount > 1 ━━�[0m
+�[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.small count=2�[0m
+    lifecycle_test.go:309: �[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.small count=2�[0m
+�[2msibling_1�[0m=i-9bfadd8ebd378f846 �[2msibling_2�[0m=i-448a9e2bf45c58658
+    lifecycle_test.go:327: �[2msibling_1�[0m=i-9bfadd8ebd378f846 �[2msibling_2�[0m=i-448a9e2bf45c58658
+    lifecycle_test.go:342: instance i-9bfadd8ebd378f846 reached state running
+    lifecycle_test.go:342: instance i-448a9e2bf45c58658 reached state running
+�[2m· terminate-instances i-9bfadd8ebd378f846 i-448a9e2bf45c58658�[0m
+    lifecycle_test.go:345: �[2m· terminate-instances i-9bfadd8ebd378f846 i-448a9e2bf45c58658�[0m
+    lifecycle_test.go:352: instance i-9bfadd8ebd378f846 reached state terminated
+    lifecycle_test.go:352: instance i-448a9e2bf45c58658 reached state terminated]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths" classname="" time="0.000">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8 — Negative / Error Path Tests ━━�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8a_InvalidAMIIDMalformed" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· run-instances image-id=notanami�[0m
+    negative_test.go:55: �[2m· run-instances image-id=notanami�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8b_InvalidInstanceType" classname="" time="0.210">
+			<system-out><![CDATA[�[2m· run-instances instance-type=x99.superlarge�[0m
+    negative_test.go:70: �[2m· run-instances instance-type=x99.superlarge�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8c_VolumeInUse" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· attach-volume vol-2230138124fb51ee2 (root, in-use)�[0m
+    negative_test.go:85: �[2m· attach-volume vol-2230138124fb51ee2 (root, in-use)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8d_DetachRootForbidden" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· detach-volume vol-2230138124fb51ee2 (root)�[0m
+    negative_test.go:97: �[2m· detach-volume vol-2230138124fb51ee2 (root)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8e_DeleteSnapshotNotFound" classname="" time="0.030">
+			<system-out><![CDATA[�[2m· delete-snapshot snap-nonexistent000000�[0m
+    negative_test.go:108: �[2m· delete-snapshot snap-nonexistent000000�[0m
+�[2m· describe-volumes vol-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8f_InvalidActionRawHTTP" classname="" time="0.040">
+			<system-out><![CDATA[�[2m· POST Action=DescribeFakeThings�[0m
+    negative_test.go:122: �[2m· POST Action=DescribeFakeThings�[0m
+    negative_test.go:125: �[2mstatus�[0m=400 �[2mcode�[0m=InvalidAction]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8g_InvalidAMIIDNotFound" classname="" time="0.240">
+			<system-out><![CDATA[�[2m· reboot-instances i-nonexistent�[0m
+�[2m· run-instances image-id=ami-0000000000000dead�[0m
+    negative_test.go:138: �[2m· run-instances image-id=ami-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8h_InvalidKeyPairNotFound" classname="" time="0.260">
+			<system-out><![CDATA[�[2m· run-instances key-name=nonexistent-key-xyz�[0m
+    negative_test.go:153: �[2m· run-instances key-name=nonexistent-key-xyz�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8i_DeleteVolumeNotFound" classname="" time="0.040">
+			<system-out><![CDATA[�[2m· delete-volume vol-0000000000000dead�[0m
+    negative_test.go:168: �[2m· delete-volume vol-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8j_CreateKeyPairDuplicate" classname="" time="0.040">
+			<system-out><![CDATA[    negative_test.go:178: �[2m· create-key-pair test-key-1 (duplicate)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8k_ImportKeyPairDuplicate" classname="" time="0.040">
+			<system-out><![CDATA[�[2m· import-key-pair test-key-1 (duplicate)�[0m
+    negative_test.go:198: �[2m· import-key-pair test-key-1 (duplicate)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8l_InvalidKeyFormat" classname="" time="0.040">
+			<system-out><![CDATA[    negative_test.go:210: �[2m· import-key-pair bad-format-key (bad material)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8m_DescribeVolumesNotFound" classname="" time="0.040">
+			<system-out><![CDATA[    negative_test.go:230: �[2m· describe-volumes vol-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8n_DescribeImagesNotFound" classname="" time="0.040">
+			<system-out><![CDATA[�[2m· describe-images ami-0000000000000dead�[0m
+    negative_test.go:240: �[2m· describe-images ami-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8o_CreateImageDuplicateName" classname="" time="0.560">
+			<system-out><![CDATA[�[2m· create-key-pair test-key-1 (duplicate)�[0m
+�[2m· create-image name=e2e-custom-ami (duplicate)�[0m
+�[2m· import-key-pair bad-format-key (bad material)�[0m
+    negative_test.go:252: �[2m· create-image name=e2e-custom-ami (duplicate)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8p_DeleteKeyPairIdempotent" classname="" time="0.020">
+			<system-out><![CDATA[�[2m· delete-key-pair nonexistent-key-99999 (idempotent)�[0m
+    negative_test.go:276: �[2m· delete-key-pair nonexistent-key-99999 (idempotent)�[0m
+�[2mstatus�[0m=400 �[2mcode�[0m=InvalidAction]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8q_ModifyAttributeOnRunning" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· modify-instance-attribute i-0c615e5dd74050f53 (running)�[0m
+    negative_test.go:289: �[2m· modify-instance-attribute i-0c615e5dd74050f53 (running)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8r_RebootInstanceNotFound" classname="" time="0.040">
+			<system-out><![CDATA[    negative_test.go:302: �[2m· reboot-instances i-nonexistent�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM1_UserCRUD" classname="" time="0.070">
+			<system-out><![CDATA[
+�[1m�[36m━━ IAM Phase 1 — User CRUD ━━�[0m
+�[2m· list-users (root auth sanity)�[0m
+    iam_test.go:109: �[2m· list-users (root auth sanity)�[0m
+�[2m· create-user "alice"�[0m
+    iam_test.go:114: �[2m· create-user "alice"�[0m
+�[2muser�[0m=alice �[2marn�[0m=arn:aws:iam::000000000001:user/alice
+    iam_test.go:120: �[2muser�[0m=alice �[2marn�[0m=arn:aws:iam::000000000001:user/alice
+�[2m· create-user "bob" path="/engineering/"�[0m
+    iam_test.go:122: �[2m· create-user "bob" path="/engineering/"�[0m
+�[2m· create-user "alice" again (expect EntityAlreadyExists)�[0m
+    iam_test.go:131: �[2m· create-user "alice" again (expect EntityAlreadyExists)�[0m
+�[2m· get-user "alice"�[0m
+    iam_test.go:137: �[2m· get-user "alice"�[0m
+�[2m· get-user nonexistent (expect NoSuchEntity)�[0m
+    iam_test.go:142: �[2m· get-user nonexistent (expect NoSuchEntity)�[0m
+�[2m· list-users (>=3 root+alice+bob)�[0m
+    iam_test.go:148: �[2m· list-users (>=3 root+alice+bob)�[0m
+�[2m· list-users --path-prefix "/engineering/" (expect 1)�[0m
+    iam_test.go:154: �[2m· list-users --path-prefix "/engineering/" (expect 1)�[0m
+    iam_test.go:193: �[2m· create-user "iam-phase1-ephemeral" (delete idempotency probe)�[0m
+�[2m· delete-user "iam-phase1-ephemeral"�[0m
+    iam_test.go:197: �[2m· delete-user "iam-phase1-ephemeral"�[0m
+�[2m· delete-user "iam-phase1-ephemeral" again (expect NoSuchEntity)�[0m
+    iam_test.go:201: �[2m· delete-user "iam-phase1-ephemeral" again (expect NoSuchEntity)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM1_UserCRUD/UpdateUser" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[    iam_test.go:165: daemon UpdateUser handler not implemented — InvalidAction
+�[2m· create-user "iam-phase1-ephemeral" (delete idempotency probe)�[0m]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM2_AccessKeyLifecycle" classname="" time="0.040">
+			<system-out><![CDATA[
+�[1m�[36m━━ IAM Phase 2 — Access Key Lifecycle ━━�[0m
+�[2m· create-access-key user=alice (key 1)�[0m
+    iam_test.go:218: �[2m· create-access-key user=alice (key 1)�[0m
+�[2mkey1�[0m=AKIA4984AE57CF8D721A879A
+    iam_test.go:227: �[2mkey1�[0m=AKIA4984AE57CF8D721A879A
+�[2m· create-access-key user=alice (key 2)�[0m
+    iam_test.go:229: �[2m· create-access-key user=alice (key 2)�[0m
+�[2mkey2�[0m=AKIAA1CD38D4F0E573216E11
+    iam_test.go:236: �[2mkey2�[0m=AKIAA1CD38D4F0E573216E11
+�[2m· create-access-key user=alice key 3 (expect LimitExceeded)�[0m
+    iam_test.go:238: �[2m· create-access-key user=alice key 3 (expect LimitExceeded)�[0m
+�[2m· create-access-key user=ghost (expect NoSuchEntity)�[0m
+    iam_test.go:246: �[2m· create-access-key user=ghost (expect NoSuchEntity)�[0m
+�[2m· list-access-keys user=alice (expect 2)�[0m
+    iam_test.go:254: �[2m· list-access-keys user=alice (expect 2)�[0m
+�[2m· list-access-keys user=bob (expect 0)�[0m
+    iam_test.go:261: �[2m· list-access-keys user=bob (expect 0)�[0m
+�[2m· update-access-key AKIA4984AE57CF8D721A879A -> Inactive�[0m
+    iam_test.go:268: �[2m· update-access-key AKIA4984AE57CF8D721A879A -> Inactive�[0m
+�[2m· update-access-key AKIA4984AE57CF8D721A879A -> Active�[0m
+    iam_test.go:279: �[2m· update-access-key AKIA4984AE57CF8D721A879A -> Active�[0m
+�[2m· delete-access-key AKIAA1CD38D4F0E573216E11 (key 2)�[0m
+    iam_test.go:287: �[2m· delete-access-key AKIAA1CD38D4F0E573216E11 (key 2)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM3_UserAuthentication" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[
+�[1m�[36m━━ IAM Phase 3 — User Authentication ━━�[0m
+    iam_test.go:311: daemon IAM signature/principal lookup not implemented — mulga-siv-100]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD" classname="" time="0.040">
+			<system-out><![CDATA[
+�[1m�[36m━━ IAM Phase 4 — Policy CRUD ━━�[0m
+�[2m· create-policy EC2ReadOnly�[0m
+    iam_test.go:384: �[2m· create-policy EC2ReadOnly�[0m
+�[2mpolicy�[0m=EC2ReadOnly �[2marn�[0m=arn:aws:iam::000000000001:policy/EC2ReadOnly �[2maccount�[0m=000000000001
+    iam_test.go:393: �[2mpolicy�[0m=EC2ReadOnly �[2marn�[0m=arn:aws:iam::000000000001:policy/EC2ReadOnly �[2maccount�[0m=000000000001
+    iam_test.go:440: �[2m· create-policy duplicate (expect EntityAlreadyExists)�[0m
+�[2m· create-policy malformed (expect MalformedPolicyDocument)�[0m
+    iam_test.go:449: �[2m· create-policy malformed (expect MalformedPolicyDocument)�[0m
+�[2m· get-policy EC2ReadOnly�[0m
+    iam_test.go:458: �[2m· get-policy EC2ReadOnly�[0m
+�[2m· get-policy nonexistent (expect NoSuchEntity)�[0m
+    iam_test.go:463: �[2m· get-policy nonexistent (expect NoSuchEntity)�[0m
+�[2m· get-policy-version EC2ReadOnly v1�[0m
+    iam_test.go:470: �[2m· get-policy-version EC2ReadOnly v1�[0m
+    iam_test.go:519: �[2m· list-policies (>=5)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel" classname="" time="0.000"></testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/FullAdmin" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· create-policy FullAdmin path=/admin/�[0m
+    iam_test.go:402: �[2m· create-policy FullAdmin path=/admin/�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/DenyTerminate" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· create-policy DenyTerminate (mixed Allow+Deny)�[0m
+    iam_test.go:413: �[2m· create-policy DenyTerminate (mixed Allow+Deny)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/IAMReadOnly" classname="" time="0.020">
+			<system-out><![CDATA[�[2m· create-policy IAMReadOnly�[0m
+    iam_test.go:422: �[2m· create-policy IAMReadOnly�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/EC2DescribeAll" classname="" time="0.020">
+			<system-out><![CDATA[�[2m· create-policy EC2DescribeAll (wildcard)�[0m
+    iam_test.go:431: �[2m· create-policy EC2DescribeAll (wildcard)�[0m
+�[2m· create-policy duplicate (expect EntityAlreadyExists)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/PolicyVersionLifecycle" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[    iam_test.go:484: daemon CreatePolicyVersion handler not implemented — InvalidAction
+�[2m· list-policies (>=5)�[0m]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM5_PolicyAttachmentEnforcement" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[
+�[1m�[36m━━ IAM Phase 5 — Policy Attachment & Enforcement ━━�[0m
+    iam_test.go:535: daemon IAM scoped-credential enforcement not implemented — mulga-siv-100]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM6_PolicyLifecycle" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[
+�[1m�[36m━━ IAM Phase 6 — Policy Lifecycle (Detach & Delete) ━━�[0m
+    iam_test.go:718: daemon DetachUserPolicy not implemented — mulga-siv-100]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM7_Cleanup" classname="" time="0.050">
+			<system-out><![CDATA[
+�[1m�[36m━━ IAM Phase 7 — Cleanup ━━�[0m
+�[2m· cleanup user alice�[0m
+    iam_test.go:770: �[2m· cleanup user alice�[0m
+�[2m· cleanup user bob�[0m
+    iam_test.go:770: �[2m· cleanup user bob�[0m
+�[2m· cleanup user charlie�[0m
+    iam_test.go:770: �[2m· cleanup user charlie�[0m
+�[2mremaining_users�[0m=1 �[2mnames�[0m=[admin]
+    iam_test.go:779: �[2mremaining_users�[0m=1 �[2mnames�[0m=[admin]
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2ReadOnly�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2ReadOnly�[0m
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/admin/FullAdmin�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/admin/FullAdmin�[0m
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/IAMReadOnly�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/IAMReadOnly�[0m
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2DescribeAll�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2DescribeAll�[0m
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/DenyTerminate�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/DenyTerminate�[0m
+�[2mremaining_policies�[0m=1 �[2mnames�[0m=[AdministratorAccess]
+    iam_test.go:812: �[2mremaining_policies�[0m=1 �[2mnames�[0m=[AdministratorAccess]]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping" classname="" time="114.890">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8 (acct) — EC2 Account Scoping ━━�[0m
+    account_test.go:75: �[2m· Phase 8 acct cleanup (deferred)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step1_AccountSetup" classname="" time="1.220">
+			<system-out><![CDATA[�[2m· create account 'Team Alpha'�[0m
+    account_test.go:206: �[2m· create account 'Team Alpha'�[0m
+�[2malpha_account�[0m=000000000002 �[2malpha_key�[0m=AKIA374CBBB2A15D3E2EE822
+    account_test.go:211: �[2malpha_account�[0m=000000000002 �[2malpha_key�[0m=AKIA374CBBB2A15D3E2EE822
+�[2m· create account 'Team Beta'�[0m
+    account_test.go:213: �[2m· create account 'Team Beta'�[0m
+�[2mbeta_account�[0m=000000000003 �[2mbeta_key�[0m=AKIA0AA0B2845E3BD5A26D01
+    account_test.go:218: �[2mbeta_account�[0m=000000000003 �[2mbeta_key�[0m=AKIA0AA0B2845E3BD5A26D01]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step2_InstanceScoping" classname="" time="52.380">
+			<system-out><![CDATA[�[2m· create per-account key pairs�[0m
+    account_test.go:242: �[2m· create per-account key pairs�[0m
+�[2m· alpha run-instances�[0m
+    account_test.go:250: �[2m· alpha run-instances�[0m
+�[2malpha_instance�[0m=i-a3072c2387630b8ee
+    account_test.go:262: �[2malpha_instance�[0m=i-a3072c2387630b8ee
+�[2m· beta run-instances�[0m
+    account_test.go:264: �[2m· beta run-instances�[0m
+�[2mbeta_instance�[0m=i-ad32cbb0afaee5eee
+    account_test.go:276: �[2mbeta_instance�[0m=i-ad32cbb0afaee5eee
+    account_test.go:278: instance i-a3072c2387630b8ee reached state running
+    account_test.go:279: instance i-ad32cbb0afaee5eee reached state running
+�[2m· alpha sees only own instances�[0m
+    account_test.go:281: �[2m· alpha sees only own instances�[0m
+�[2m· beta sees only own instances�[0m
+    account_test.go:285: �[2m· beta sees only own instances�[0m
+�[2m· alpha OwnerId matches alpha account�[0m
+    account_test.go:289: �[2m· alpha OwnerId matches alpha account�[0m
+�[2m· cross-account stop alpha->beta blocked�[0m
+    account_test.go:296: �[2m· cross-account stop alpha->beta blocked�[0m
+�[2m· cross-account terminate beta->alpha blocked�[0m
+    account_test.go:304: �[2m· cross-account terminate beta->alpha blocked�[0m
+�[2m· cross-account reboot alpha->beta blocked�[0m
+    account_test.go:312: �[2m· cross-account reboot alpha->beta blocked�[0m
+�[2m· stop alpha instance for cross-account start/modify/console tests�[0m
+    account_test.go:321: �[2m· stop alpha instance for cross-account start/modify/console tests�[0m
+    account_test.go:326: instance i-a3072c2387630b8ee reached state stopped
+�[2m· cross-account start beta->alpha blocked�[0m
+    account_test.go:328: �[2m· cross-account start beta->alpha blocked�[0m
+�[2m· cross-account modify-instance-attribute beta->alpha blocked�[0m
+    account_test.go:336: �[2m· cross-account modify-instance-attribute beta->alpha blocked�[0m
+�[2m· cross-account get-console-output beta->alpha blocked�[0m
+    account_test.go:345: �[2m· cross-account get-console-output beta->alpha blocked�[0m
+�[2m· restart alpha instance�[0m
+    account_test.go:354: �[2m· restart alpha instance�[0m
+    account_test.go:359: instance i-a3072c2387630b8ee reached state running]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step3_VolumeScoping" classname="" time="24.140">
+			<system-out><![CDATA[�[2m· alpha create-volume�[0m
+    account_test.go:366: �[2m· alpha create-volume�[0m
+�[2malpha_vol�[0m=vol-ceef82652369780af
+    account_test.go:375: �[2malpha_vol�[0m=vol-ceef82652369780af
+�[2m· beta create-volume�[0m
+    account_test.go:377: �[2m· beta create-volume�[0m
+�[2mbeta_vol�[0m=vol-08024f32d23d9e18c
+    account_test.go:386: �[2mbeta_vol�[0m=vol-08024f32d23d9e18c
+    account_test.go:388: volume vol-ceef82652369780af reached state available
+    account_test.go:389: volume vol-08024f32d23d9e18c reached state available
+�[2m· alpha sees only own volumes�[0m
+    account_test.go:391: �[2m· alpha sees only own volumes�[0m
+�[2m· cross-account describe-volume by id blocked�[0m
+    account_test.go:395: �[2m· cross-account describe-volume by id blocked�[0m
+�[2m· cross-account delete-volume blocked�[0m
+    account_test.go:403: �[2m· cross-account delete-volume blocked�[0m
+�[2m· cross-account attach-volume (beta attaches alpha's vol) blocked�[0m
+    account_test.go:411: �[2m· cross-account attach-volume (beta attaches alpha's vol) blocked�[0m
+�[2m· alpha attach-volume to own instance�[0m
+    account_test.go:422: �[2m· alpha attach-volume to own instance�[0m
+    account_test.go:429: volume vol-ceef82652369780af reached state in-use
+�[2m· cross-account detach-volume blocked�[0m
+    account_test.go:431: �[2m· cross-account detach-volume blocked�[0m
+�[2m· cross-account modify-volume blocked�[0m
+    account_test.go:439: �[2m· cross-account modify-volume blocked�[0m
+�[2m· alpha detach own volume�[0m
+    account_test.go:449: �[2m· alpha detach own volume�[0m
+    account_test.go:454: volume vol-ceef82652369780af reached state available]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step4_KeyPairScoping" classname="" time="0.370">
+			<system-out><![CDATA[�[2m· alpha create-key-pair alpha-key�[0m
+    account_test.go:466: �[2m· alpha create-key-pair alpha-key�[0m
+�[2m· beta create-key-pair beta-key�[0m
+    account_test.go:473: �[2m· beta create-key-pair beta-key�[0m
+�[2m· alpha sees only own keys�[0m
+    account_test.go:478: �[2m· alpha sees only own keys�[0m
+�[2m· namespace isolation: shared-name in both accounts�[0m
+    account_test.go:483: �[2m· namespace isolation: shared-name in both accounts�[0m
+�[2m· beta deletes alpha-key (idempotent, no cross-account effect)�[0m
+    account_test.go:497: �[2m· beta deletes alpha-key (idempotent, no cross-account effect)�[0m
+�[2m· alpha import-key-pair imported-key�[0m
+    account_test.go:503: �[2m· alpha import-key-pair imported-key�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step5_SnapshotScoping" classname="" time="0.060">
+			<system-out><![CDATA[�[2m· alpha create-snapshot�[0m
+    account_test.go:522: �[2m· alpha create-snapshot�[0m
+�[2malpha_snap�[0m=snap-2a47896f4304d805c
+    account_test.go:530: �[2malpha_snap�[0m=snap-2a47896f4304d805c
+�[2m· beta create-snapshot�[0m
+    account_test.go:532: �[2m· beta create-snapshot�[0m
+�[2mbeta_snap�[0m=snap-bf5744e78bd7c6908
+    account_test.go:540: �[2mbeta_snap�[0m=snap-bf5744e78bd7c6908
+�[2m· alpha sees only own snapshots (owner-ids=self)�[0m
+    account_test.go:542: �[2m· alpha sees only own snapshots (owner-ids=self)�[0m
+�[2m· cross-account delete-snapshot blocked (UnauthorizedOperation)�[0m
+    account_test.go:560: �[2m· cross-account delete-snapshot blocked (UnauthorizedOperation)�[0m
+�[2m· cross-account create-snapshot of alpha's volume blocked�[0m
+    account_test.go:568: �[2m· cross-account create-snapshot of alpha's volume blocked�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step6_VPCSubnetScoping" classname="" time="0.030">
+			<system-out><![CDATA[�[2m· alpha create-vpc 10.0.0.0/16�[0m
+    account_test.go:582: �[2m· alpha create-vpc 10.0.0.0/16�[0m
+�[2malpha_vpc�[0m=vpc-7f60df5e87183fb59
+    account_test.go:589: �[2malpha_vpc�[0m=vpc-7f60df5e87183fb59
+�[2m· beta create-vpc 10.0.0.0/16 (same CIDR, no conflict)�[0m
+    account_test.go:591: �[2m· beta create-vpc 10.0.0.0/16 (same CIDR, no conflict)�[0m
+�[2mbeta_vpc�[0m=vpc-7e8c859d3985c8218
+    account_test.go:598: �[2mbeta_vpc�[0m=vpc-7e8c859d3985c8218
+�[2m· alpha describe-vpcs isolation�[0m
+    account_test.go:600: �[2m· alpha describe-vpcs isolation�[0m
+�[2m· cross-account describe-vpc by id blocked�[0m
+    account_test.go:605: �[2m· cross-account describe-vpc by id blocked�[0m
+�[2m· cross-account delete-vpc blocked�[0m
+    account_test.go:613: �[2m· cross-account delete-vpc blocked�[0m
+�[2m· alpha create-subnet 10.0.1.0/24�[0m
+    account_test.go:621: �[2m· alpha create-subnet 10.0.1.0/24�[0m
+�[2m· beta create-subnet 10.0.1.0/24�[0m
+    account_test.go:630: �[2m· beta create-subnet 10.0.1.0/24�[0m
+�[2m· alpha describe-subnets isolation�[0m
+    account_test.go:639: �[2m· alpha describe-subnets isolation�[0m
+�[2m· cross-account create-subnet in other VPC blocked�[0m
+    account_test.go:648: �[2m· cross-account create-subnet in other VPC blocked�[0m
+�[2m· cross-account delete-subnet blocked�[0m
+    account_test.go:657: �[2m· cross-account delete-subnet blocked�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step7_IGWEIGWScoping" classname="" time="0.030">
+			<system-out><![CDATA[�[2m· alpha create-internet-gateway�[0m
+    account_test.go:673: �[2m· alpha create-internet-gateway�[0m
+�[2malpha_igw�[0m=igw-ee966e3e923caf630
+    account_test.go:678: �[2malpha_igw�[0m=igw-ee966e3e923caf630
+�[2m· beta create-internet-gateway�[0m
+    account_test.go:680: �[2m· beta create-internet-gateway�[0m
+�[2mbeta_igw�[0m=igw-b46e66f2065889bb9
+    account_test.go:685: �[2mbeta_igw�[0m=igw-b46e66f2065889bb9
+�[2m· alpha describe-internet-gateways isolation�[0m
+    account_test.go:687: �[2m· alpha describe-internet-gateways isolation�[0m
+�[2m· cross-account describe IGW by id blocked�[0m
+    account_test.go:696: �[2m· cross-account describe IGW by id blocked�[0m
+�[2m· cross-account delete IGW blocked�[0m
+    account_test.go:704: �[2m· cross-account delete IGW blocked�[0m
+�[2m· cross-account attach IGW (alpha attaches beta's IGW) blocked�[0m
+    account_test.go:712: �[2m· cross-account attach IGW (alpha attaches beta's IGW) blocked�[0m
+�[2m· alpha attach own IGW to own VPC�[0m
+    account_test.go:722: �[2m· alpha attach own IGW to own VPC�[0m
+�[2m· cross-account detach IGW blocked�[0m
+    account_test.go:729: �[2m· cross-account detach IGW blocked�[0m
+�[2m· alpha create-egress-only-internet-gateway�[0m
+    account_test.go:739: �[2m· alpha create-egress-only-internet-gateway�[0m
+�[2malpha_eigw�[0m=eigw-51ba82da927e49393
+    account_test.go:746: �[2malpha_eigw�[0m=eigw-51ba82da927e49393
+�[2m· beta create-egress-only-internet-gateway�[0m
+    account_test.go:748: �[2m· beta create-egress-only-internet-gateway�[0m
+�[2mbeta_eigw�[0m=eigw-e88d642e78ab4fe5a
+    account_test.go:755: �[2mbeta_eigw�[0m=eigw-e88d642e78ab4fe5a
+�[2m· alpha describe-eigws isolation�[0m
+    account_test.go:757: �[2m· alpha describe-eigws isolation�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step8_AccountSettings" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· alpha enable-ebs-encryption-by-default�[0m
+    account_test.go:794: �[2m· alpha enable-ebs-encryption-by-default�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step9_GlobalResources" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· describe-regions identical across accounts�[0m
+    account_test.go:829: �[2m· describe-regions identical across accounts�[0m
+�[2m· describe-availability-zones identical across accounts�[0m
+    account_test.go:834: �[2m· describe-availability-zones identical across accounts�[0m
+�[2m· describe-instance-types identical across accounts�[0m
+    account_test.go:839: �[2m· describe-instance-types identical across accounts�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step10_EdgeCases" classname="" time="1.200">
+			<system-out><![CDATA[�[2m· create empty Gamma account�[0m
+    account_test.go:849: �[2m· create empty Gamma account�[0m
+�[2mgamma_account�[0m=000000000004
+    account_test.go:852: �[2mgamma_account�[0m=000000000004
+�[2m· root create-key-pair root-scoping-key�[0m
+    account_test.go:879: �[2m· root create-key-pair root-scoping-key�[0m
+�[2m· non-existent volume id -> InvalidVolume.NotFound�[0m
+    account_test.go:895: �[2m· non-existent volume id -> InvalidVolume.NotFound�[0m
+�[2m· non-existent snapshot id -> InvalidSnapshot.NotFound�[0m
+    account_test.go:903: �[2m· non-existent snapshot id -> InvalidSnapshot.NotFound�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step11_Cleanup" classname="" time="32.440">
+			<system-out><![CDATA[�[2m· terminate alpha instance�[0m
+    account_test.go:920: �[2m· terminate alpha instance�[0m
+�[2m· terminate beta instance�[0m
+    account_test.go:926: �[2m· terminate beta instance�[0m
+�[2m· delete snapshots�[0m
+    account_test.go:939: �[2m· delete snapshots�[0m
+�[2m· delete volumes�[0m
+    account_test.go:955: �[2m· delete volumes�[0m
+�[2m· delete key pairs�[0m
+    account_test.go:969: �[2m· delete key pairs�[0m
+�[2m· delete EIGWs�[0m
+    account_test.go:979: �[2m· delete EIGWs�[0m
+�[2m· detach + delete IGWs�[0m
+    account_test.go:993: �[2m· detach + delete IGWs�[0m
+�[2m· delete subnets�[0m
+    account_test.go:1013: �[2m· delete subnets�[0m
+�[2m· delete VPCs�[0m
+    account_test.go:1027: �[2m· delete VPCs�[0m
+�[2m· Phase 8 acct cleanup (deferred)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8b_VPCSubnetE2E" classname="" time="200.480">
+			<failure message="Failed"><![CDATA[
+�[1m�[36m━━ Phase 8b — VPC Public/Private Subnet E2E ━━�[0m
+�[2m· create-vpc 10.99.0.0/16�[0m
+    vpc_test.go:56: �[2m· create-vpc 10.99.0.0/16�[0m
+�[2mvpc�[0m=vpc-39eff985799c43069 �[2mcidr�[0m=10.99.0.0/16
+    vpc_test.go:67: �[2mvpc�[0m=vpc-39eff985799c43069 �[2mcidr�[0m=10.99.0.0/16
+�[2m· create-subnet 10.99.1.0/24 (public)�[0m
+    vpc_test.go:70: �[2m· create-subnet 10.99.1.0/24 (public)�[0m
+�[2m· modify-subnet-attribute MapPublicIpOnLaunch=true subnet-22aba054e10588254�[0m
+    vpc_test.go:87: �[2m· modify-subnet-attribute MapPublicIpOnLaunch=true subnet-22aba054e10588254�[0m
+�[2mpub_subnet�[0m=subnet-22aba054e10588254 �[2mmap_public_ip�[0m=true
+    vpc_test.go:93: �[2mpub_subnet�[0m=subnet-22aba054e10588254 �[2mmap_public_ip�[0m=true
+�[2m· create-subnet 10.99.2.0/24 (private)�[0m
+    vpc_test.go:96: �[2m· create-subnet 10.99.2.0/24 (private)�[0m
+�[2mpriv_subnet�[0m=subnet-f53aeddb73c134696 �[2mmap_public_ip�[0m=false
+    vpc_test.go:111: �[2mpriv_subnet�[0m=subnet-f53aeddb73c134696 �[2mmap_public_ip�[0m=false
+�[2m· create-internet-gateway�[0m
+    vpc_test.go:114: �[2m· create-internet-gateway�[0m
+�[2m· attach-internet-gateway igw-1520b5a34480da982 -> vpc-39eff985799c43069�[0m
+    vpc_test.go:133: �[2m· attach-internet-gateway igw-1520b5a34480da982 -> vpc-39eff985799c43069�[0m
+�[2migw�[0m=igw-1520b5a34480da982
+    vpc_test.go:139: �[2migw�[0m=igw-1520b5a34480da982
+�[2m· create-route-table (vpc=vpc-39eff985799c43069)�[0m
+    vpc_test.go:142: �[2m· create-route-table (vpc=vpc-39eff985799c43069)�[0m
+�[2m· associate-route-table rtb-662df5e424cbdb60c <- subnet-22aba054e10588254�[0m
+    vpc_test.go:159: �[2m· associate-route-table rtb-662df5e424cbdb60c <- subnet-22aba054e10588254�[0m
+�[2m· create-route 0.0.0.0/0 -> igw-1520b5a34480da982�[0m
+    vpc_test.go:176: �[2m· create-route 0.0.0.0/0 -> igw-1520b5a34480da982�[0m
+�[2mrtb�[0m=rtb-662df5e424cbdb60c �[2massoc�[0m=rtbassoc-5510bf1dc6924eb0b �[2mroute�[0m=0.0.0.0/0->igw-1520b5a34480da982
+    vpc_test.go:192: �[2mrtb�[0m=rtb-662df5e424cbdb60c �[2massoc�[0m=rtbassoc-5510bf1dc6924eb0b �[2mroute�[0m=0.0.0.0/0->igw-1520b5a34480da982
+�[2m· run-instances ami=ami-6f60f2b3bf04b5453 subnet=subnet-22aba054e10588254�[0m
+    vpc_test.go:195: �[2m· run-instances ami=ami-6f60f2b3bf04b5453 subnet=subnet-22aba054e10588254�[0m
+�[2minstance�[0m=i-3177888cba3796d42
+    vpc_test.go:216: �[2minstance�[0m=i-3177888cba3796d42
+    vpc_test.go:218: instance i-3177888cba3796d42 reached state running
+�[2mpublic_ip�[0m=192.168.0.202 �[2mprivate_ip�[0m=10.99.1.4
+    vpc_test.go:224: �[2mpublic_ip�[0m=192.168.0.202 �[2mprivate_ip�[0m=10.99.1.4
+�[2m· waiting for SSH handshake 192.168.0.202:22�[0m
+    vpc_test.go:227: �[2m· waiting for SSH handshake 192.168.0.202:22�[0m
+    vpc_test.go:227: Eventually: condition not met within 3m0s: [SSH handshake 192.168.0.202:22 never completed]]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8c_RouteTableValidation" classname="" time="0.030">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8c — Route Table Validation ━━�[0m
+�[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63
+    vpc_test.go:292: �[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8c_RouteTableValidation/Step1_DefaultMainRouteTable" classname="" time="0.000">
+			<system-out><![CDATA[�[2m· describe-route-tables filter=vpc-id,association.main�[0m
+    vpc_test.go:297: �[2m· describe-route-tables filter=vpc-id,association.main�[0m
+�[2mmain_rtb�[0m=rtb-e42012569f92f7184
+    vpc_test.go:309: �[2mmain_rtb�[0m=rtb-e42012569f92f7184
+�[2mlocal_route�[0m=172.31.0.0/16
+    vpc_test.go:332: �[2mlocal_route�[0m=172.31.0.0/16
+�[2mdefault_route�[0m=0.0.0.0/0->igw-79c2e643660b4b7e3
+    vpc_test.go:350: �[2mdefault_route�[0m=0.0.0.0/0->igw-79c2e643660b4b7e3]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8c_RouteTableValidation/Step2_CustomRouteTableLifecycle" classname="" time="0.020">
+			<system-out><![CDATA[�[2m· create-route-table (vpc=vpc-531252db3b4e1fc08)�[0m
+    vpc_test.go:363: �[2m· create-route-table (vpc=vpc-531252db3b4e1fc08)�[0m
+�[2mcustom_rtb�[0m=rtb-5e6c46936f3e44167
+    vpc_test.go:379: �[2mcustom_rtb�[0m=rtb-5e6c46936f3e44167
+�[2mcustom_local_route�[0m=172.31.0.0/16
+    vpc_test.go:396: �[2mcustom_local_route�[0m=172.31.0.0/16
+�[2mvpc_igw�[0m=igw-79c2e643660b4b7e3
+    vpc_test.go:410: �[2mvpc_igw�[0m=igw-79c2e643660b4b7e3
+�[2m· create-route 0.0.0.0/0 -> igw-79c2e643660b4b7e3�[0m
+    vpc_test.go:415: �[2m· create-route 0.0.0.0/0 -> igw-79c2e643660b4b7e3�[0m
+�[2m· associate-route-table rtb-5e6c46936f3e44167 <- subnet-cd62e4c1c3cb50c63�[0m
+    vpc_test.go:434: �[2m· associate-route-table rtb-5e6c46936f3e44167 <- subnet-cd62e4c1c3cb50c63�[0m
+�[2mcustom_assoc�[0m=rtbassoc-da6377d37847d3d4a
+    vpc_test.go:450: �[2mcustom_assoc�[0m=rtbassoc-da6377d37847d3d4a
+�[2m· disassociate-route-table rtbassoc-da6377d37847d3d4a�[0m
+    vpc_test.go:452: �[2m· disassociate-route-table rtbassoc-da6377d37847d3d4a�[0m
+�[2m· delete-route 0.0.0.0/0�[0m
+    vpc_test.go:460: �[2m· delete-route 0.0.0.0/0�[0m
+�[2m· delete-route-table rtb-5e6c46936f3e44167�[0m
+    vpc_test.go:468: �[2m· delete-route-table rtb-5e6c46936f3e44167�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8c_RouteTableValidation/Step3_ErrorPaths" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· create-route-table (scratch for error paths)�[0m
+    vpc_test.go:481: �[2m· create-route-table (scratch for error paths)�[0m
+�[2m· delete-route local=172.31.0.0/16 (must fail)�[0m
+    vpc_test.go:517: �[2m· delete-route local=172.31.0.0/16 (must fail)�[0m
+�[2mdelete_local_err�[0m=InvalidParameterValue
+    vpc_test.go:529: �[2mdelete_local_err�[0m=InvalidParameterValue
+�[2m· delete-route-table rtb-e42012569f92f7184 (main, must fail)�[0m
+    vpc_test.go:533: �[2m· delete-route-table rtb-e42012569f92f7184 (main, must fail)�[0m
+�[2mdelete_main_err�[0m=DependencyViolation
+    vpc_test.go:540: �[2mdelete_main_err�[0m=DependencyViolation]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8d_NATGateway" classname="" time="106.010">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8d — NAT Gateway E2E ━━�[0m
+�[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2mpub_subnet�[0m=subnet-cd62e4c1c3cb50c63 �[2msg�[0m=sg-764a7fa2a39577204
+    natgw_test.go:54: �[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2mpub_subnet�[0m=subnet-cd62e4c1c3cb50c63 �[2msg�[0m=sg-764a7fa2a39577204
+�[2m· create-subnet 172.31.16.0/20 (private)�[0m
+    natgw_test.go:58: �[2m· create-subnet 172.31.16.0/20 (private)�[0m
+�[2mpriv_subnet�[0m=subnet-2360a40e28371e214
+    natgw_test.go:71: �[2mpriv_subnet�[0m=subnet-2360a40e28371e214
+�[2m· run-instances bastion (default subnet)�[0m
+    natgw_test.go:74: �[2m· run-instances bastion (default subnet)�[0m
+�[2m· run-instances private (private subnet)�[0m
+    natgw_test.go:98: �[2m· run-instances private (private subnet)�[0m
+    natgw_test.go:118: instance i-268402f0e28b5e477 reached state running
+    natgw_test.go:119: instance i-db7f3ed6e4cdaa5aa reached state running
+�[2mbastion�[0m=i-268402f0e28b5e477 �[2mbastion_ip�[0m=192.168.0.202 �[2mprivate�[0m=i-db7f3ed6e4cdaa5aa �[2mprivate_ip�[0m=172.31.16.4
+    natgw_test.go:128: �[2mbastion�[0m=i-268402f0e28b5e477 �[2mbastion_ip�[0m=192.168.0.202 �[2mprivate�[0m=i-db7f3ed6e4cdaa5aa �[2mprivate_ip�[0m=172.31.16.4
+�[2m· waiting for SSH handshake 192.168.0.202:22�[0m
+    natgw_test.go:133: �[2m· waiting for SSH handshake 192.168.0.202:22�[0m
+�[2m· scp keypair -> bastion:/tmp/key.pem�[0m
+    natgw_test.go:139: �[2m· scp keypair -> bastion:/tmp/key.pem�[0m
+�[2m· wait for private SSH via bastion�[0m
+    natgw_test.go:151: �[2m· wait for private SSH via bastion�[0m
+�[2m· baseline: private VM has no internet�[0m
+    natgw_test.go:172: �[2m· baseline: private VM has no internet�[0m
+�[2m· allocate-address (NAT EIP)�[0m
+    natgw_test.go:178: �[2m· allocate-address (NAT EIP)�[0m
+�[2meip�[0m=192.168.0.203 �[2malloc�[0m=eipalloc-2d81753157cd62a50
+    natgw_test.go:195: �[2meip�[0m=192.168.0.203 �[2malloc�[0m=eipalloc-2d81753157cd62a50
+�[2m· create-nat-gateway in subnet-cd62e4c1c3cb50c63�[0m
+    natgw_test.go:197: �[2m· create-nat-gateway in subnet-cd62e4c1c3cb50c63�[0m
+�[2mnat_gw�[0m=nat-812216dd5105180c5
+    natgw_test.go:216: �[2mnat_gw�[0m=nat-812216dd5105180c5
+    natgw_test.go:218: nat gateway nat-812216dd5105180c5 reached state available
+�[2m· create-route-table�[0m
+    natgw_test.go:223: �[2m· create-route-table�[0m
+�[2m· associate-route-table rtb-8d6a4abd5f666ec13 <- subnet-2360a40e28371e214�[0m
+    natgw_test.go:241: �[2m· associate-route-table rtb-8d6a4abd5f666ec13 <- subnet-2360a40e28371e214�[0m
+�[2m· create-route 0.0.0.0/0 -> nat-812216dd5105180c5�[0m
+    natgw_test.go:259: �[2m· create-route 0.0.0.0/0 -> nat-812216dd5105180c5�[0m
+�[2m· verify private VM reaches 8.8.8.8 via NAT GW�[0m
+    natgw_test.go:279: �[2m· verify private VM reaches 8.8.8.8 via NAT GW�[0m
+�[2m· delete-nat-gateway nat-812216dd5105180c5�[0m
+    natgw_test.go:293: �[2m· delete-nat-gateway nat-812216dd5105180c5�[0m
+    natgw_test.go:318: nat gateway nat-812216dd5105180c5 reached state deleted
+�[2m· verify private VM lost internet after NAT GW teardown�[0m
+    natgw_test.go:329: �[2m· verify private VM lost internet after NAT GW teardown�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8e_SGToSGDatapath" classname="" time="80.940">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8e — Security Group SG-to-SG Datapath (OVN) ━━�[0m
+�[2m· 8e-1 create sge-client + sge-target security groups�[0m
+    datapath_test.go:61: �[2m· 8e-1 create sge-client + sge-target security groups�[0m
+�[2mclient_sg�[0m=sg-56e73c764a649f7ff �[2mtarget_sg�[0m=sg-67ad5efaa8be83b79
+    datapath_test.go:83: �[2mclient_sg�[0m=sg-56e73c764a649f7ff �[2mtarget_sg�[0m=sg-67ad5efaa8be83b79
+�[2m· 8e-1 authorize target-sg ingress tcp/8080 from sg-56e73c764a649f7ff�[0m
+    datapath_test.go:91: �[2m· 8e-1 authorize target-sg ingress tcp/8080 from sg-56e73c764a649f7ff�[0m
+�[2m· 8e-2 launch client-vm + target-vm�[0m
+    datapath_test.go:109: �[2m· 8e-2 launch client-vm + target-vm�[0m
+    datapath_test.go:132: instance i-eae6263d9b9fda069 reached state running
+    datapath_test.go:133: instance i-9d9411a83549d34e3 reached state running
+�[2mclient_priv�[0m=172.31.0.5 �[2mclient_eni�[0m=eni-0f878da0143028915 �[2mtarget_priv�[0m=172.31.0.6 �[2mtarget_eni�[0m=eni-3bd355e19c7fdd45c
+    datapath_test.go:142: �[2mclient_priv�[0m=172.31.0.5 �[2mclient_eni�[0m=eni-0f878da0143028915 �[2mtarget_priv�[0m=172.31.0.6 �[2mtarget_eni�[0m=eni-3bd355e19c7fdd45c
+�[2m· 8e-3 ovn-nbctl port_group membership�[0m
+    datapath_test.go:151: �[2m· 8e-3 ovn-nbctl port_group membership�[0m
+�[2mclient_pg�[0m=sg_56e73c764a649f7ff �[2mtarget_pg�[0m=sg_67ad5efaa8be83b79
+    datapath_test.go:163: �[2mclient_pg�[0m=sg_56e73c764a649f7ff �[2mtarget_pg�[0m=sg_67ad5efaa8be83b79
+�[2m· 8e-3 ovn-sbctl address_set sg_56e73c764a649f7ff_ip4 contains 172.31.0.5�[0m
+    datapath_test.go:170: �[2m· 8e-3 ovn-sbctl address_set sg_56e73c764a649f7ff_ip4 contains 172.31.0.5�[0m
+�[2m· 8e-3 wait for client-vm SSH at 192.168.0.202:22�[0m
+    datapath_test.go:188: �[2m· 8e-3 wait for client-vm SSH at 192.168.0.202:22�[0m
+�[2m· 8e-4 allowed traffic client -> target:172.31.0.6:8080�[0m
+    datapath_test.go:199: �[2m· 8e-4 allowed traffic client -> target:172.31.0.6:8080�[0m
+�[2mstep4�[0m=allowed_traffic_ok
+    datapath_test.go:208: �[2mstep4�[0m=allowed_traffic_ok
+�[2m· 8e-5 denied traffic client -> target:22 (no SSH ingress)�[0m
+    datapath_test.go:213: �[2m· 8e-5 denied traffic client -> target:22 (no SSH ingress)�[0m
+�[2mstep5�[0m=denied_traffic_ok
+    datapath_test.go:217: �[2mstep5�[0m=denied_traffic_ok
+�[2m· 8e-6 revoke target-sg ingress, retest (sync RequestEvent contract)�[0m
+    datapath_test.go:222: �[2m· 8e-6 revoke target-sg ingress, retest (sync RequestEvent contract)�[0m
+�[2m· 8e-6 verify client -> target:8080 now blocked�[0m
+    datapath_test.go:240: �[2m· 8e-6 verify client -> target:8080 now blocked�[0m
+�[2mstep6�[0m=revoke_blocked_ok
+    datapath_test.go:244: �[2mstep6�[0m=revoke_blocked_ok
+�[2m· 8e-7 re-authorize target-sg ingress, verify traffic restored�[0m
+    datapath_test.go:250: �[2m· 8e-7 re-authorize target-sg ingress, verify traffic restored�[0m
+�[2mstep7�[0m=restore_ok
+    datapath_test.go:271: �[2mstep7�[0m=restore_ok
+    datapath_test.go:127: instance i-eae6263d9b9fda069 reached state terminated
+    datapath_test.go:127: instance i-9d9411a83549d34e3 reached state terminated]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase9_Teardown" classname="" time="8.300">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 9 — Teardown ━━�[0m
+�[2m· delete-snapshot snap-dde89b9ea2ef2e05a (custom AMI backing)�[0m
+    teardown_test.go:33: �[2m· delete-snapshot snap-dde89b9ea2ef2e05a (custom AMI backing)�[0m
+�[2mdeleted_snapshot�[0m=snap-dde89b9ea2ef2e05a
+    teardown_test.go:39: �[2mdeleted_snapshot�[0m=snap-dde89b9ea2ef2e05a
+�[2m· deregister-image ami-7dcf263a412dd14c6�[0m
+    teardown_test.go:51: �[2m· deregister-image ami-7dcf263a412dd14c6�[0m
+�[2mderegistered_ami�[0m=ami-7dcf263a412dd14c6
+    teardown_test.go:57: �[2mderegistered_ami�[0m=ami-7dcf263a412dd14c6
+�[2m· terminate-instances i-0c615e5dd74050f53�[0m
+    teardown_test.go:74: �[2m· terminate-instances i-0c615e5dd74050f53�[0m
+�[2mterminated_instance�[0m=i-0c615e5dd74050f53
+    teardown_test.go:81: �[2mterminated_instance�[0m=i-0c615e5dd74050f53
+�[2m· delete-key-pair test-key-1�[0m
+    teardown_test.go:94: �[2m· delete-key-pair test-key-1�[0m
+�[2mdeleted_keypair�[0m=test-key-1
+    teardown_test.go:100: �[2mdeleted_keypair�[0m=test-key-1]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase9a_VerifyTeardown" classname="" time="0.560">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 9a — Verify Teardown ━━�[0m
+�[2m· describe-instances i-0c615e5dd74050f53�[0m
+    teardown_test.go:123: �[2m· describe-instances i-0c615e5dd74050f53�[0m
+�[2minstance_state�[0m=terminated
+    teardown_test.go:135: �[2minstance_state�[0m=terminated
+�[2m· describe-images ami-7dcf263a412dd14c6�[0m
+    teardown_test.go:147: �[2m· describe-images ami-7dcf263a412dd14c6�[0m
+�[2mami_reaped�[0m=ami-7dcf263a412dd14c6
+    teardown_test.go:159: �[2mami_reaped�[0m=ami-7dcf263a412dd14c6
+�[2m· describe-key-pairs (must not contain test-key-1)�[0m
+    teardown_test.go:167: �[2m· describe-key-pairs (must not contain test-key-1)�[0m
+�[2mkeypairs_listed�[0m=1
+    teardown_test.go:176: �[2mkeypairs_listed�[0m=1]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase9b_FinalClusterStats" classname="" time="3.070">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 9b — Final Cluster Stats ━━�[0m
+�[2mspx_get_vms_bytes�[0m=118
+    teardown_test.go:208: �[2mspx_get_vms_bytes�[0m=118]]></system-out>
+		</testcase>
+	</testsuite>
+</testsuites>

.github/actions/junit-report/.gitignore

@@ -0,0 +1 @@
+__pycache__/

.github/actions/junit-report/README.md

@@ -0,0 +1,46 @@
+# junit-report
+
+Composite action that renders a glob of JUnit XML files into the GitHub
+Actions job summary and emits one `::error` annotation per failed test case.
+
+## Usage
+
+```yaml
+- name: Publish test summary
+  if: always()
+  uses: ./spinifex/.github/actions/junit-report   # or your own checkout path
+  with:
+    junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+    title: "Go E2E results"
+    # fail-on-empty: "true"   # opt-in: fail the step if the glob matches nothing
+```
+
+Outputs:
+
+- `failures` — total `<failure>` + `<error>` counts across all matched files.
+- `total`    — total `<testcase>` count across all matched files.
+
+The action expects standard JUnit XML (e.g. emitted by
+`gotestsum --junitfile`, `go-junit-report`, or pytest's `--junitxml`). Both
+single `<testsuite>` and wrapping `<testsuites>` roots are accepted.
+
+## Companion convention: `::group::` for noisy steps
+
+Long-running shell phases (tofu apply, package installs, tar-over-ssh) should
+wrap their output in `::group::name` / `::endgroup::` markers so the Actions
+UI collapses them by default. Pattern used in the e2e-go / e2e-ddil workflows:
+
+```bash
+run() {
+  local label="$1"; shift
+  echo "::group::${label}"
+  "$@" 2>&1 | tee -a "$TOFU_LOG"
+  local rc=${PIPESTATUS[0]}
+  echo "::endgroup::"
+  return "$rc"
+}
+run "tofu apply" tofu apply -var-file=... -auto-approve -input=false
+```
+
+Combined: groups make the live UI readable; this action makes the run header
+land on the failures without scrolling the log.

.github/actions/junit-report/action.yml

@@ -0,0 +1,41 @@
+name: JUnit Report
+description: >-
+  Render JUnit XML files into the GitHub Actions job summary and emit
+  inline ::error annotations for each failed test case. Designed to be
+  called once per job after the test step (with `if: always()`) so summary
+  + annotations appear on both passing and failing runs.
+
+inputs:
+  junit-glob:
+    description: Shell glob matching JUnit XML files to render.
+    required: true
+  title:
+    description: Heading rendered at the top of the job summary section.
+    required: false
+    default: "Test results"
+  fail-on-empty:
+    description: >-
+      When "true", the action fails if the glob matches no files. Useful
+      for catching silently-skipped test phases.
+    required: false
+    default: "false"
+
+outputs:
+  failures:
+    description: Total failed + errored test cases across all matched files.
+    value: ${{ steps.render.outputs.failures }}
+  total:
+    description: Total test cases across all matched files.
+    value: ${{ steps.render.outputs.total }}
+
+runs:
+  using: composite
+  steps:
+    - id: render
+      shell: bash
+      env:
+        JUNIT_GLOB: ${{ inputs.junit-glob }}
+        JUNIT_TITLE: ${{ inputs.title }}
+        JUNIT_FAIL_ON_EMPTY: ${{ inputs.fail-on-empty }}
+      run: |
+        python3 "${{ github.action_path }}/render.py"

.github/actions/junit-report/render.py

@@ -0,0 +1,281 @@
+#!/usr/bin/env python3
+"""Render JUnit XML files into GitHub Actions job summary + annotations.
+
+Inputs (env):
+    JUNIT_GLOB           Glob matching JUnit XML files.
+    JUNIT_TITLE          Heading text for the summary section.
+    JUNIT_FAIL_ON_EMPTY  "true" to exit non-zero when glob matches nothing.
+
+GitHub-Actions outputs (via $GITHUB_OUTPUT):
+    failures             Sum of <failures> + <errors> across all suites.
+    total                Sum of <tests> across all suites.
+
+Side effects:
+    Appends a markdown section to $GITHUB_STEP_SUMMARY with a per-file
+    totals table and a collapsible details block per failed test.
+    Emits one `::error` workflow command per failed/errored test case so
+    they surface as annotations in the run UI.
+"""
+
+from __future__ import annotations
+
+import glob
+import os
+import re
+import sys
+import xml.etree.ElementTree as ET
+from dataclasses import dataclass
+
+
+# Strip ANSI SGR escapes from skipped/failure CDATA so the rendered reason is
+# plain text (CDATA preserves the colour codes Go emits via harness.Phase).
+_ANSI_RE = re.compile(r"\x1b\[[0-9;]*m")
+
+
+def _strip_ansi(s: str) -> str:
+    return _ANSI_RE.sub("", s)
+
+
+# Last `<file>.go:<line>: <reason>` line in a CDATA block is the t.Skipf /
+# t.Fatal call site — that's the actionable signal. Phase banners and probe
+# logs above it are noise for the summary view.
+_SKIP_REASON_RE = re.compile(r"([A-Za-z0-9_.\-/]+\.go:\d+:\s*.+)$")
+
+
+def _extract_skip_reason(cdata: str) -> str:
+    cdata = _strip_ansi(cdata).strip()
+    if not cdata:
+        return ""
+    for line in reversed(cdata.splitlines()):
+        line = line.strip()
+        m = _SKIP_REASON_RE.search(line)
+        if m:
+            return m.group(1).strip()
+    return cdata.splitlines()[-1].strip()
+
+
+# Workflow commands carry their payload on one line; literal newlines must
+# be percent-escaped per GitHub's workflow-command spec.
+def _esc(s: str) -> str:
+    return (
+        s.replace("%", "%25")
+        .replace("\r", "%0D")
+        .replace("\n", "%0A")
+    )
+
+
+@dataclass
+class Case:
+    suite: str
+    classname: str
+    name: str
+    time: float
+    status: str  # "pass" | "fail" | "error" | "skip"
+    message: str
+    detail: str
+
+
+@dataclass
+class FileTotals:
+    path: str
+    tests: int
+    failures: int
+    errors: int
+    skipped: int
+    time: float
+    cases: list[Case]
+
+
+def _parse_file(path: str) -> FileTotals:
+    root = ET.parse(path).getroot()
+    # JUnit XML has two shapes: <testsuite> as root, or <testsuites> wrapping
+    # multiple <testsuite> children. Iterating with .iter() handles both.
+    suites = list(root.iter("testsuite"))
+    tests = sum(int(s.get("tests", 0) or 0) for s in suites)
+    failures = sum(int(s.get("failures", 0) or 0) for s in suites)
+    errors = sum(int(s.get("errors", 0) or 0) for s in suites)
+    skipped = sum(int(s.get("skipped", 0) or 0) for s in suites)
+    time = sum(float(s.get("time", 0) or 0) for s in suites)
+
+    cases: list[Case] = []
+    for s in suites:
+        suite_name = s.get("name", "")
+        for tc in s.findall("testcase"):
+            status = "pass"
+            message = ""
+            detail = ""
+            for tag in ("failure", "error"):
+                node = tc.find(tag)
+                if node is not None:
+                    status = "fail" if tag == "failure" else "error"
+                    message = (node.get("message") or "").strip()
+                    detail = (node.text or "").strip()
+                    break
+            skip_node = tc.find("skipped")
+            if skip_node is not None and status == "pass":
+                status = "skip"
+                # Skipped reason: prefer the CDATA call-site line, fall back
+                # to the @message attribute if CDATA is empty.
+                cdata = (skip_node.text or "").strip()
+                message = _extract_skip_reason(cdata) or (
+                    skip_node.get("message") or ""
+                ).strip()
+            cases.append(
+                Case(
+                    suite=suite_name,
+                    classname=tc.get("classname", ""),
+                    name=tc.get("name", ""),
+                    time=float(tc.get("time", 0) or 0),
+                    status=status,
+                    message=message,
+                    detail=detail,
+                )
+            )
+
+    # Drop container test cases: any case whose name is a strict prefix of
+    # another case's name (with "/" separator) is a parent t.Run wrapper.
+    # Go reports the parent's status as a rollup of its children, so counting
+    # both double-tallies the same outcome (e.g. TestSingleNode failing
+    # because Phase8b_VPCSubnetE2E failed would show as 2 failures, not 1).
+    names = {c.name for c in cases}
+    prefixes = {n.rsplit("/", 1)[0] for n in names if "/" in n}
+    # A name is a container if some other name starts with "<name>/".
+    # Use the prefixes set rather than O(n^2) scanning.
+    def _is_container(name: str) -> bool:
+        return name in prefixes
+
+    leaf_cases = [c for c in cases if not _is_container(c.name)]
+
+    # Recompute totals from leaf cases so the per-file counts in the summary
+    # table match the de-duplicated failure list below it.
+    leaf_tests = len(leaf_cases)
+    leaf_failures = sum(1 for c in leaf_cases if c.status == "fail")
+    leaf_errors = sum(1 for c in leaf_cases if c.status == "error")
+    leaf_skipped = sum(1 for c in leaf_cases if c.status == "skip")
+
+    return FileTotals(
+        path=path,
+        tests=leaf_tests,
+        failures=leaf_failures,
+        errors=leaf_errors,
+        skipped=leaf_skipped,
+        time=time,
+        cases=leaf_cases,
+    )
+
+
+def _file_label(path: str) -> str:
+    base = os.path.basename(path)
+    if base.endswith(".xml"):
+        base = base[:-4]
+    if base.startswith("junit-"):
+        base = base[len("junit-"):]
+    return base or path
+
+
+def main() -> int:
+    pattern = os.environ.get("JUNIT_GLOB", "").strip()
+    title = os.environ.get("JUNIT_TITLE", "Test results").strip() or "Test results"
+    fail_on_empty = os.environ.get("JUNIT_FAIL_ON_EMPTY", "").strip().lower() == "true"
+
+    if not pattern:
+        print("::error::junit-report: junit-glob input is empty", file=sys.stderr)
+        return 1
+
+    paths = sorted(glob.glob(pattern))
+    summary_path = os.environ.get("GITHUB_STEP_SUMMARY", "/dev/null")
+    output_path = os.environ.get("GITHUB_OUTPUT", "/dev/null")
+
+    with open(summary_path, "a", encoding="utf-8") as summary, \
+            open(output_path, "a", encoding="utf-8") as outputs:
+
+        summary.write(f"## {title}\n\n")
+
+        if not paths:
+            summary.write(f"_No JUnit files matched `{pattern}`._\n\n")
+            outputs.write("failures=0\n")
+            outputs.write("total=0\n")
+            if fail_on_empty:
+                print(f"::error::junit-report: no files matched {pattern}", file=sys.stderr)
+                return 1
+            return 0
+
+        totals_by_file = [_parse_file(p) for p in paths]
+        grand_total = sum(f.tests for f in totals_by_file)
+        grand_fail = sum(f.failures + f.errors for f in totals_by_file)
+        grand_skip = sum(f.skipped for f in totals_by_file)
+        grand_pass = grand_total - grand_fail - grand_skip
+        grand_time = sum(f.time for f in totals_by_file)
+
+        if grand_fail == 0:
+            verdict = f"✅ **pass** — {grand_pass} / {grand_total}"
+        else:
+            verdict = f"❌ **{grand_fail} failed** — {grand_pass} / {grand_total} pass"
+        skip_part = f", {grand_skip} skipped" if grand_skip else ""
+        summary.write(f"{verdict}{skip_part} &nbsp; · &nbsp; {grand_time:.1f}s\n\n")
+
+        # Compact table — single status column, shorter headers. Drops
+        # separate err column (errors fold into fail count) so narrow
+        # screens don't wrap.
+        summary.write("| | Suite | Pass | Fail | Skip | Time |\n")
+        summary.write("|:-:|---|---:|---:|---:|---:|\n")
+        for f in totals_by_file:
+            fails = f.failures + f.errors
+            passed = f.tests - fails - f.skipped
+            icon = "✅" if fails == 0 else "❌"
+            summary.write(
+                f"| {icon} | `{_file_label(f.path)}` | {passed} | {fails} | "
+                f"{f.skipped} | {f.time:.1f}s |\n"
+            )
+        summary.write("\n")
+
+        failed_cases = [
+            c for f in totals_by_file for c in f.cases if c.status in ("fail", "error")
+        ]
+        if failed_cases:
+            summary.write("---\n\n**Failures**\n\n")
+            for c in failed_cases:
+                header = f"{c.classname}.{c.name}" if c.classname else c.name
+                # One line per failure: name + truncated message. Full
+                # detail lives in the test artifact log; deep nesting and
+                # long messages wrap badly on narrow screens.
+                msg_short = (c.message or "").splitlines()[0] if c.message else ""
+                if len(msg_short) > 120:
+                    msg_short = msg_short[:117] + "…"
+                line = f"- ❌ `{header}`"
+                if msg_short:
+                    line += f" — {msg_short}"
+                summary.write(line + "\n")
+
+                # Workflow annotation — surfaces in run header + Files-changed view.
+                # classname for Go tests is the package path; map to a file when we can.
+                file_hint = c.classname.replace(".", "/")
+                msg = c.message or f"{c.status}: {header}"
+                print(f"::error file={file_hint},title={header}::{_esc(msg)}")
+            summary.write("\n")
+
+        skipped_cases = [
+            c for f in totals_by_file for c in f.cases if c.status == "skip"
+        ]
+        if skipped_cases:
+            summary.write("---\n\n<details><summary><b>Skipped</b> ")
+            summary.write(f"({len(skipped_cases)})</summary>\n\n")
+            for c in skipped_cases:
+                header = f"{c.classname}.{c.name}" if c.classname else c.name
+                reason = (c.message or "").splitlines()[0] if c.message else ""
+                if len(reason) > 160:
+                    reason = reason[:157] + "…"
+                line = f"- ⏭️ `{header}`"
+                if reason:
+                    line += f" — {reason}"
+                summary.write(line + "\n")
+            summary.write("\n</details>\n\n")
+
+        outputs.write(f"failures={grand_fail}\n")
+        outputs.write(f"total={grand_total}\n")
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

.github/scripts/ci-fmt.sh

@@ -0,0 +1,63 @@
+# shellcheck shell=bash
+# Source from a workflow `run:` block:
+#   source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+#
+# Provides colored banners + ::group::/::endgroup:: wrappers. ANSI escapes
+# render in the GitHub Actions live log; the same lines also land in any
+# artifact log (tee'd output) without breaking parsers because they only
+# affect terminal display.
+#
+# Functions kept short and unindented on purpose — wide nested output wraps
+# badly on narrow terminals.
+
+if [[ -n "${GITHUB_ACTIONS:-}" ]] || [[ -t 1 ]]; then
+  _R=$'\033[0m'
+  _B=$'\033[1m'    # bold
+  _D=$'\033[2m'    # dim
+  _RED=$'\033[31m'
+  _GRN=$'\033[32m'
+  _YEL=$'\033[33m'
+  _BLU=$'\033[34m'
+  _CYA=$'\033[36m'
+else
+  _R="" _B="" _D="" _RED="" _GRN="" _YEL="" _BLU="" _CYA=""
+fi
+
+# Single-line rule using box-drawing chars — narrow enough that GitHub's
+# timestamp prefix + this rule still fit on an 80-col terminal.
+# banner LABEL — bold cyan title with surrounding rule. Used at the start
+# of an inherently multi-line phase (test run, artifact pull) so the live
+# log has an obvious visual marker.
+banner() {
+  printf '\n%s%s━━ %s ━━%s\n' "$_B" "$_CYA" "$*" "$_R"
+}
+
+# Status indicators. Single char prefix only — keeps width on narrow logs.
+ok()   { printf '%s✓%s %s\n' "$_GRN" "$_R" "$*"; }
+bad()  { printf '%s✗%s %s\n' "$_RED" "$_R" "$*"; }
+warn() { printf '%s!%s %s\n' "$_YEL" "$_R" "$*"; }
+note() { printf '%s·%s %s\n' "$_D"   "$_R" "$*"; }
+
+# group_open LABEL / group_close — thin wrapper around GitHub's collapse
+# markers. No extra body printed; banner() is the visible inside-group
+# heading when callers want one.
+group_open()  { echo "::group::$*"; }
+group_close() { echo "::endgroup::"; }
+
+# quiet_run LABEL LOG_FILE -- CMD ARGS …
+# Run CMD redirected to LOG_FILE (append). Print one colored ok/✗ line.
+# On failure, dump the last 80 lines of LOG_FILE inside a collapsed group
+# so the failing tail lands in the UI without surfacing the whole apply.
+quiet_run() {
+  local label="$1" log="$2"; shift 2
+  if "$@" >> "$log" 2>&1; then
+    ok "$label"
+  else
+    local rc=$?
+    bad "$label (rc=$rc) — last 80 lines of $(basename "$log"):"
+    group_open "tail $(basename "$log")"
+    tail -n 80 "$log"
+    group_close
+    return "$rc"
+  fi
+}

.github/workflows/e2e-baremetal.yml

@@ -12,9 +12,9 @@ env:
 
 jobs:
   e2e_baremetal:
-    name: Bare-Metal Single-Node E2E
+    name: Bare-Metal Multi-Node E2E
     runs-on: [ self-hosted, ci-baremetal ]
-    timeout-minutes: 60
+    timeout-minutes: 120
     concurrency:
       group: e2e-baremetal-cluster
       cancel-in-progress: false
@@ -22,7 +22,7 @@ jobs:
       BM_ISO_CACHE: /var/cache/spinifex-iso-builder
     steps:
       - name: Clean Workspace
-        run: rm -rf "$GITHUB_WORKSPACE"/*
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
 
       - uses: actions/checkout@v6
 
@@ -44,9 +44,9 @@ jobs:
 
       - name: Install SSH Key
         run: |
-          install -m 0700 -d ~/.ssh
-          echo "${{ secrets.BM_SSH_PRIVATE_KEY }}" > ~/.ssh/bm-ci
-          chmod 0600 ~/.ssh/bm-ci
+          echo "${{ secrets.BM_SSH_PRIVATE_KEY }}" > "${RUNNER_TEMP}/bm-ci"
+          chmod 0600 "${RUNNER_TEMP}/bm-ci"
+          ssh-keygen -y -f "${RUNNER_TEMP}/bm-ci" > "${RUNNER_TEMP}/bm-ci.pub"
 
       - name: Build Artifacts
         run: |
@@ -66,11 +66,38 @@ jobs:
             VIPERBLOCK_SRC="${RUNNER_TEMP}/bm-bins/viperblock" \
             VERSION="$(cd "$GITHUB_WORKSPACE" && git describe --tags --always 2>/dev/null || echo dev)"
 
+      - name: Write ePDU credentials
+        run: |
+          install -m 0600 /dev/null "${RUNNER_TEMP}/epdu.env"
+          cat >> "${RUNNER_TEMP}/epdu.env" <<EOF
+          EPDU_HOST=192.168.1.63
+          EPDU_SNMP_USER=ci-bmpower
+          EPDU_SNMP_SEC_LEVEL=authPriv
+          EPDU_SNMP_AUTH_PROTO=MD5
+          EPDU_SNMP_AUTH_PASS=${{ secrets.BM_EPDU_AUTH_PASS }}
+          EPDU_SNMP_PRIV_PROTO=DES
+          EPDU_SNMP_PRIV_PASS=${{ secrets.BM_EPDU_PRIV_PASS }}
+          EOF
+
+      - name: Cache VM images
+        run: |
+          UBUNTU_IMG="${BM_ISO_CACHE}/ubuntu-26.04-cloudimg-amd64.img"
+          if [ ! -f "$UBUNTU_IMG" ]; then
+            echo "Downloading Ubuntu 26.04 cloud image..."
+            sudo curl -fsSL -o "$UBUNTU_IMG" \
+              "https://cloud-images.ubuntu.com/resolute/current/resolute-server-cloudimg-amd64.img"
+            echo "  Downloaded $(du -h "$UBUNTU_IMG" | cut -f1)"
+          else
+            echo "Ubuntu 26.04 cloud image cached ($(du -h "$UBUNTU_IMG" | cut -f1))"
+          fi
+
+
       - name: Run Bare-Metal E2E
         env:
           BM_CI_PASSWORD: ${{ secrets.BM_CI_PASSWORD }}
-          BM_SSH_KEY: /home/tf-user/.ssh/bm-ci
-          BM_SKIP_EPDU: "1"
+          BM_SSH_KEY: ${{ runner.temp }}/bm-ci
+          BM_EPDU_ENV: ${{ runner.temp }}/epdu.env
+          BM_UBUNTU_IMAGE: ${{ env.BM_ISO_CACHE }}/ubuntu-26.04-cloudimg-amd64.img
           SPX_EXTERNAL_POOL: ${{ env.SPINIFEX_CI_POOL }}
         run: |
           sudo -E mulga/scripts/baremetal/run-baremetal-e2e.sh \

.github/workflows/e2e-ddil.yml

@@ -30,7 +30,7 @@ jobs:
       TOFU_LOG: /tmp/e2e-tofu-ddil-${{ github.run_id }}.log
     steps:
       - name: Clean Workspace
-        run: rm -rf "$GITHUB_WORKSPACE"/*
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         id: app-token
@@ -87,12 +87,16 @@ jobs:
       - name: Provision VMs
         working-directory: mulga/scripts/tofu-cluster
         run: |
+          set -euo pipefail
+          # shellcheck source=/dev/null
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           export TF_VAR_git_branch="${GIT_BRANCH}"
-          tofu init -input=false >> "$TOFU_LOG" 2>&1
-          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
-          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
-          tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
+          quiet_run "tofu workspace select"    "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
+          quiet_run "scrub-env (pre-clean)"    "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
+          quiet_run "tofu apply"               "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          ok "VMs provisioned"
 
       - name: Bootstrap via Binary Install
         working-directory: mulga/scripts/tofu-cluster
@@ -147,8 +151,13 @@ jobs:
 
           mkdir -p "${RUNNER_TEMP}/ddil-results"
           cd "${GITHUB_WORKSPACE}/spinifex"
-          # gotestsum: human-readable PASS/FAIL/SKIP lines in Actions stdout
-          # while still emitting the machine-readable jsonfile artifact.
+          # gotestsum: human-readable PASS/FAIL/SKIP lines in Actions stdout,
+          # plus machine-readable jsonfile + JUnit XML artifacts. The JUnit
+          # file feeds the junit-report composite action below so failures
+          # become summary table rows + workflow annotations.
+          # shellcheck source=/dev/null
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          banner "DDIL scenarios"
           DDIL_NODES="${NODE_IPS}" \
           DDIL_SSH_USER="${SSH_USER}" \
           DDIL_SSH_KEY="${SSH_KEY}" \
@@ -158,8 +167,19 @@ jobs:
           AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" \
           AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" \
             gotestsum --jsonfile "${RUNNER_TEMP}/ddil-results/results.json" \
+                      --junitfile "${RUNNER_TEMP}/ddil-results/junit-ddil.xml" \
                       --format testname \
                       -- -tags=e2e -timeout 45m ./tests/e2e/ddil/scenarios/...
+          rc=$?
+          if [[ $rc -eq 0 ]]; then ok "all scenarios passed"; else bad "scenarios failed (rc=$rc)"; fi
+          exit $rc
+
+      - name: Publish test summary
+        if: always()
+        uses: ./spinifex/.github/actions/junit-report
+        with:
+          junit-glob: ${{ runner.temp }}/ddil-results/junit-*.xml
+          title: "DDIL scenario results"
 
       - name: Collect Cluster Logs
         if: failure()
@@ -177,13 +197,15 @@ jobs:
           LOG_DIR="${RUNNER_TEMP}/ddil-results/cluster-logs"
           mkdir -p "${LOG_DIR}"
 
+          # shellcheck source=/dev/null
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           mapfile -t NODE_IPS < <(echo "$INVENTORY" | jq -r '.nodes[].wan_ip')
           for ip in "${NODE_IPS[@]}"; do
-            echo "=== collecting ${ip} ==="
             SSH="ssh -i ${SSH_KEY} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${SSH_USER}@${ip}"
             ${SSH} 'sudo journalctl -u "spinifex-*" --no-pager -n 5000' > "${LOG_DIR}/${ip}-spinifex.log" 2>&1
             ${SSH} 'ps auxw | grep -E "qemu-system|spinifex" | grep -v grep' > "${LOG_DIR}/${ip}-procs.txt" 2>&1
             ${SSH} 'ls -la /var/lib/spinifex 2>/dev/null; ls -la /var/log/spinifex 2>/dev/null' > "${LOG_DIR}/${ip}-state.txt" 2>&1
+            ok "$ip: $(wc -l < "${LOG_DIR}/${ip}-spinifex.log") journal lines, $(wc -l < "${LOG_DIR}/${ip}-procs.txt") procs"
           done
 
       - name: Upload DDIL Results
@@ -198,6 +220,10 @@ jobs:
         if: always()
         working-directory: mulga/scripts/tofu-cluster
         run: |
-          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
-          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+          set -euo pipefail
+          # shellcheck source=/dev/null
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy"          "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
+          ok "teardown complete"

.github/workflows/e2e-nightly.yml

@@ -12,11 +12,10 @@ on:
 permissions:
   contents: read
 
-# Scheduled runs target dev (active development trunk). GitHub loads scheduled
-# workflow files from the repo's default branch, so this file must be kept in
-# sync on main for the schedule to use it.
+# Scheduled runs are loaded from the default branch (main), so the schedule
+# trigger must stay in sync on main for cron to fire.
 env:
-  GIT_BRANCH: dev
+  GIT_BRANCH: ${{ github.event_name == 'schedule' && 'dev' || github.ref_name }}
   UBUNTU_GOLD_IMAGE: /mnt/disk1/gold-images/spinifex-image-builder-ubuntu-24.04.qcow2
 
 jobs:
@@ -61,7 +60,7 @@ jobs:
 
       - name: Clean Workspace
         if: steps.gate.outputs.selected == 'true'
-        run: rm -rf "$GITHUB_WORKSPACE"/*
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         if: steps.gate.outputs.selected == 'true'

.github/workflows/e2e.yml

@@ -34,7 +34,7 @@ jobs:
       TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
     steps:
       - name: Clean Workspace
-        run: rm -rf "$GITHUB_WORKSPACE"/*
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         id: app-token
@@ -103,7 +103,7 @@ jobs:
       TOFU_LOG: /tmp/e2e-tofu-pseudo-${{ github.run_id }}.log
     steps:
       - name: Clean Workspace
-        run: rm -rf "$GITHUB_WORKSPACE"/*
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         id: app-token
@@ -167,7 +167,7 @@ jobs:
       TOFU_LOG: /tmp/e2e-tofu-multi-${{ github.run_id }}.log
     steps:
       - name: Clean Workspace
-        run: rm -rf "$GITHUB_WORKSPACE"/*
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         id: app-token

.github/workflows/release.yml

@@ -31,6 +31,14 @@ jobs:
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v4
 
+      - name: Set up Go
+        uses: actions/setup-go@v6
+        with:
+          go-version: '1.26.3'
+
+      - name: Build microVM artifacts (lb-agent + kernel + initramfs)
+        run: cd spinifex && make build-microvm-image
+
       - name: Build amd64 Tarball
         env:
           VERSION: ${{ github.ref_name }}
@@ -192,3 +200,151 @@ jobs:
           tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
           tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
           ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+
+  iso:
+    needs: release
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 90
+    steps:
+      - name: Clean Workspace
+        run: |
+          ROOTFS="${GITHUB_WORKSPACE}/mulga/scripts/iso-builder/.cache/rootfs"
+          # Mirror make clean's umount sequence unconditionally — lazy unmount
+          # detaches immediately even if the mount is busy, so rm -rf is safe.
+          sudo umount -l -R "${ROOTFS}" 2>/dev/null || true
+          sudo umount -l "${ROOTFS}/dev/pts" 2>/dev/null || true
+          sudo umount -l "${ROOTFS}/dev" 2>/dev/null || true
+          sudo umount -l "${ROOTFS}/sys" 2>/dev/null || true
+          sudo umount -l "${ROOTFS}/proc" 2>/dev/null || true
+          sudo rm -rf "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/create-github-app-token@v3
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      - name: Checkout Spinifex
+        uses: actions/checkout@v6
+        with:
+          path: spinifex
+          ref: ${{ github.ref }}
+
+      - name: Checkout Predastore
+        uses: actions/checkout@v6
+        with:
+          repository: mulgadc/predastore
+          path: predastore
+
+      - name: Checkout Viperblock
+        uses: actions/checkout@v6
+        with:
+          repository: mulgadc/viperblock
+          path: viperblock
+
+      - name: Checkout Mulga iso-builder
+        uses: actions/checkout@v6
+        with:
+          repository: mulgadc/mulga
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: scripts/iso-builder
+          path: mulga
+
+      - name: Clean ISO builder cache
+        run: sudo make -C mulga/scripts/iso-builder clean
+
+      - name: Set up Go
+        uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Configure Go workspace
+        run: go work init ./spinifex ./predastore ./viperblock
+
+      - name: Install ISO builder dependencies
+        run: |
+          sudo apt-get update -qq
+          sudo apt-get install -y \
+            debootstrap squashfs-tools xorriso grub-pc-bin grub-efi-amd64-bin \
+            grub-common mtools busybox-static ovmf zstd \
+            nbdkit nbdkit-plugin-dev pkg-config gcc
+
+      - name: Build spinifex binaries
+        run: cd spinifex && make build
+
+      - name: Build viperblock
+        run: cd viperblock && make build
+
+      - name: Build release ISO
+        id: build-iso
+        env:
+          VERSION: ${{ github.ref_name }}
+        run: |
+          sudo make -C mulga/scripts/iso-builder release \
+            VERSION="${VERSION}" \
+            SPINIFEX_SRC="${GITHUB_WORKSPACE}/spinifex" \
+            VIPERBLOCK_SRC="${GITHUB_WORKSPACE}/viperblock" \
+            SPINIFEX_BIN="${GITHUB_WORKSPACE}/spinifex/bin/spx" \
+            INSTALLER_BIN="${GITHUB_WORKSPACE}/spinifex/bin/spinifex-installer"
+
+      - name: Wait for verify
+        id: wait-verify
+        if: steps.build-iso.outcome == 'success'
+        continue-on-error: true
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          echo "Waiting for verify job to complete..."
+          while true; do
+            CONCLUSION=$(gh api \
+              "repos/${{ github.repository }}/actions/runs/${{ github.run_id }}/jobs" \
+              --jq '.jobs[] | select(.name == "verify") | .conclusion // empty')
+            if [ -n "$CONCLUSION" ]; then
+              echo "Verify conclusion: $CONCLUSION"
+              [ "$CONCLUSION" = "success" ] && exit 0
+              exit 1
+            fi
+            echo "Verify still running, retrying in 30s..."
+            sleep 30
+          done
+
+      - name: Ensure AWS CLI is installed
+        if: steps.build-iso.outcome == 'success' && steps.wait-verify.outcome == 'success'
+        run: |
+          if ! command -v aws &>/dev/null; then
+            curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o /tmp/awscliv2.zip
+            unzip -q /tmp/awscliv2.zip -d /tmp/aws-install
+            sudo /tmp/aws-install/aws/install
+            rm -rf /tmp/awscliv2.zip /tmp/aws-install
+          fi
+
+      - name: Upload ISO to Cloudflare R2
+        id: upload
+        if: steps.build-iso.outcome == 'success' && steps.wait-verify.outcome == 'success'
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.CF_R2_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.CF_R2_SECRET_ACCESS_KEY }}
+          CF_R2_ENDPOINT_URL: ${{ secrets.CF_R2_ENDPOINT_URL }}
+          VERSION: ${{ github.ref_name }}
+        run: |
+          aws --endpoint-url="${CF_R2_ENDPOINT_URL}" \
+            s3 cp "mulga/scripts/iso-builder/dist/spinifex-${VERSION}.iso" \
+                  "s3://spinifex-iso/spinifex-${VERSION}.iso"
+          aws --endpoint-url="${CF_R2_ENDPOINT_URL}" \
+            s3 cp "mulga/scripts/iso-builder/dist/spinifex-${VERSION}.iso.sha256" \
+                  "s3://spinifex-iso/spinifex-${VERSION}.iso.sha256"
+
+      - name: Mark ISO failed
+        if: always() && (steps.build-iso.outcome == 'failure' || steps.upload.outcome == 'failure')
+        env:
+          GH_TOKEN: ${{ github.token }}
+          TAG: ${{ github.ref_name }}
+          REPO: ${{ github.repository }}
+        run: |
+          EXISTING_BODY=$(gh release view "$TAG" --repo "$REPO" --json body -q .body)
+          NOTES_FILE=$(mktemp)
+          printf 'ISO build/upload failed — see workflow run for details.\n\n%s' "$EXISTING_BODY" > "$NOTES_FILE"
+          gh release edit "$TAG" --repo "$REPO" --notes-file "$NOTES_FILE"
+          rm -f "$NOTES_FILE"

.github/workflows/spinifex.yml

@@ -36,12 +36,14 @@ jobs:
         uses: actions/checkout@v6
         with:
           repository: mulgadc/viperblock
+          ref: dev
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v6
         with:
           repository: mulgadc/predastore
+          ref: dev
           path: predastore
 
       - name: Setup Go
@@ -59,6 +61,9 @@ jobs:
       - name: Run Tests with Coverage
         run: make -C spinifex test-cover
 
+      - name: Run Action Tests
+        run: make -C spinifex test-actions
+
       - name: Check Diff Coverage
         run: cd spinifex && scripts/diff-coverage.sh coverage.out --base HEAD~1
 
@@ -76,12 +81,14 @@ jobs:
         uses: actions/checkout@v6
         with:
           repository: mulgadc/viperblock
+          ref: dev
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v6
         with:
           repository: mulgadc/predastore
+          ref: dev
           path: predastore
 
       - name: Setup Go
@@ -110,12 +117,14 @@ jobs:
         uses: actions/checkout@v6
         with:
           repository: mulgadc/viperblock
+          ref: dev
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v6
         with:
           repository: mulgadc/predastore
+          ref: dev
           path: predastore
 
       - name: Setup Go
@@ -130,7 +139,7 @@ jobs:
       - name: Lint
         uses: golangci/golangci-lint-action@v9
         with:
-          version: v2.11.3
+          version: v2.12.2
           working-directory: spinifex
 
       - name: Govulncheck

.gitignore

@@ -17,6 +17,8 @@ go.work.sum
 # Build outputs
 bin/
 dist/
+build/microvm/vmlinuz
+build/microvm/initramfs.cpio.gz
 !spinifex/services/spinifexui/frontend/dist/
 tmp/
 *.test

Makefile

@@ -63,15 +63,15 @@ LDFLAGS := -s -w -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=$(VERSI
 
 go_build:
 	@echo -e "\n....Building $(GO_PROJECT_NAME)"
-	go build $(GO_BUILD_MOD) -ldflags "$(LDFLAGS)" -o ./bin/$(GO_PROJECT_NAME) cmd/spinifex/main.go
+	GOFIPS140=v1.0.0 go build $(GO_BUILD_MOD) -ldflags "$(LDFLAGS)" -o ./bin/$(GO_PROJECT_NAME) cmd/spinifex/main.go
 
 build-installer:
 	@echo -e "\n....Building spinifex-installer"
-	go build -ldflags "-s -w" -o ./bin/spinifex-installer cmd/installer/main.go
+	GOFIPS140=v1.0.0 go build -ldflags "-s -w" -o ./bin/spinifex-installer cmd/installer/main.go
 
 build-lb-agent:
 	@echo -e "\n....Building lb-agent (static)"
-	CGO_ENABLED=0 go build -ldflags "-s -w" -o ./bin/lb-agent cmd/lb-agent/main.go
+	CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -ldflags "-s -w" -o ./bin/lb-agent cmd/lb-agent/main.go
 
 build-system-image: ## Build a system image from manifest (use IMAGE=lb)
 ifndef IMAGE
@@ -79,8 +79,25 @@ ifndef IMAGE
 endif
 	./scripts/build-system-image.sh scripts/images/$(IMAGE).conf
 
-build-lb-image: ## Build LB Alpine image
-	$(MAKE) build-system-image IMAGE=lb
+MICROVM_OUT_DIR := build/microvm
+MICROVM_ARTIFACTS := $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz
+MICROVM_INPUTS := scripts/build-microvm-image.sh $(MICROVM_OUT_DIR)/init.sh $(MICROVM_OUT_DIR)/inittab bin/lb-agent
+
+# Grouped target — script writes both files in one run.
+$(MICROVM_ARTIFACTS) &: $(MICROVM_INPUTS)
+	./scripts/build-microvm-image.sh
+
+# Only triggers when bin/lb-agent is missing; preserves the artifact's mtime so
+# build-microvm-image stays correctly stale-aware.
+bin/lb-agent:
+	$(MAKE) build-lb-agent
+
+build-microvm-image: $(MICROVM_ARTIFACTS) ## Build microVM kernel + initramfs (incremental — skips when up to date)
+
+install-microvm: $(MICROVM_ARTIFACTS) ## Install microVM artifacts to /usr/share/spinifex/microvm/
+	sudo install -d /usr/share/spinifex/microvm
+	sudo install -m 0644 $(MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/vmlinuz
+	sudo install -m 0644 $(MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/initramfs.cpio.gz
 
 go_run:
 	@echo -e "\n....Running $(GO_PROJECT_NAME)...."
@@ -89,9 +106,23 @@ go_run:
 # Preflight — runs the same checks as GitHub Actions (lint + vuln + tests).
 # Use this before committing to catch CI failures locally.
 preflight:
-	@$(MAKE) --no-print-directory QUIET=1 lint govulncheck test-cover diff-coverage test-race
+	@$(MAKE) --no-print-directory QUIET=1 manifest-check lint govulncheck test-cover diff-coverage test-race test-harness
 	@echo -e "\n ✅ Preflight passed — safe to commit."
 
+# E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
+# default `go test ./spinifex/...`. Runs with mocked AWS clients — no
+# infrastructure required, safe to run in CI without a cluster.
+test-harness:
+	@echo -e "\n....Running e2e harness unit tests...."
+	$(_Q)LOG_IGNORE=1 go test -tags=e2e -timeout 60s ./tests/e2e/harness/... $(_RACEQ)
+
+# Validate docs/service-interfaces.yaml. Schema check + cross-reference
+# of services/suites/fixtures + on-disk path existence. Subject content
+# vs source is enforced separately in Bead 5 drift lint.
+manifest-check:
+	@echo -e "\n....Checking service-interfaces.yaml...."
+	@go run ./tests/e2e/manifest-check/cmd/manifest-check -repo-root . -manifest docs/service-interfaces.yaml
+
 # Run unit tests
 test:
 	@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
@@ -109,6 +140,12 @@ test-race:
 	@echo -e "\n....Running tests with race detector for $(GO_PROJECT_NAME)...."
 	$(_Q)LOG_IGNORE=1 go test -race -timeout 300s ./spinifex/... $(_RACEQ)
 
+# Unit tests for in-repo GitHub Actions (e.g. .github/actions/e2e-analyze).
+# Kept out of `test-cover` so coverage % isn't diluted by CI-only tooling.
+test-actions:
+	@echo -e "\n....Running action tests...."
+	LOG_IGNORE=1 go test -timeout 60s ./.github/actions/...
+
 # Check that new/changed code meets coverage threshold (runs tests first)
 diff-coverage: test-cover
 	@QUIET=$(QUIET) scripts/diff-coverage.sh $(COVERPROFILE)
@@ -122,9 +159,18 @@ run:
 	$(MAKE) go_build
 	$(MAKE) go_run
 
-# Fast iteration: build + install binary + restart all services
+# Fast iteration: build + install binary + restart all services.
+# Microvm artifacts are reinstalled when they already exist on disk — the rule's
+# input timestamps drive a rebuild only if anything actually changed. On a fresh
+# checkout (no build/microvm/vmlinuz yet) the install step is skipped; run
+# `make install-microvm` explicitly the first time.
 deploy: build
 	sudo install -m 755 bin/spx /usr/local/bin/spx
+	@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ]; then \
+		$(MAKE) install-microvm; \
+	else \
+		echo "[deploy] microvm artifacts absent — run 'make install-microvm' for first-time setup"; \
+	fi
 	sudo systemctl daemon-reload
 	sudo systemctl restart spinifex.target
 
@@ -144,6 +190,7 @@ install-system:
 		-o Dpkg::Options::="--force-confdef" \
 		-o Dpkg::Options::="--force-confold" \
 		nbdkit nbdkit-plugin-dev pkg-config $(QEMU_PACKAGES) qemu-utils qemu-kvm \
+		ovmf qemu-efi-aarch64 \
 		libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
 		iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
 		ovn-central ovn-host openvswitch-switch dhcpcd-base
@@ -206,6 +253,14 @@ distro-amd64:
 		-f build/Dockerfile.distro \
 		--output type=local,dest=dist/amd64/ \
 		../
+	@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ] && [ -f $(MICROVM_OUT_DIR)/initramfs.cpio.gz ]; then \
+		echo "[distro-amd64] staging microvm artifacts into tarball"; \
+		mkdir -p dist/amd64/microvm; \
+		cp $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz dist/amd64/microvm/; \
+	else \
+		echo "[distro-amd64] WARNING: microvm artifacts missing — tarball will not include them"; \
+		echo "[distro-amd64]          run 'make build-microvm-image' before 'make distro-amd64'"; \
+	fi
 	tar -czf dist/spinifex-$(VERSION)-linux-amd64.tar.gz -C dist/amd64 .
 	sha256sum dist/spinifex-$(VERSION)-linux-amd64.tar.gz > dist/spinifex-$(VERSION)-linux-amd64.tar.gz.sha256
 
@@ -238,9 +293,12 @@ ansible-dev-install:
 ansible-dev-reset:
 	cd scripts/ansible && ansible-playbook playbooks/dev-reset.yml
 
-.PHONY: build build-ui build-installer build-lb-agent build-system-image build-lb-image go_build go_run preflight test test-cover test-race diff-coverage bench run \
+ansible-dev-deploy:
+	cd scripts/ansible && ansible-playbook playbooks/dev-deploy.yml
+
+.PHONY: build build-ui build-installer build-lb-agent build-system-image build-microvm-image install-microvm go_build go_run preflight test test-cover test-race test-actions test-harness manifest-check diff-coverage bench run \
 	deploy reinstall clean \
 	install-system install-go install-aws quickinstall \
 	lint fix govulncheck \
 	distro distro-amd64 distro-arm64 distro-clean \
-	ansible-dev-preflight ansible-dev-teardown ansible-dev-install ansible-dev-reset
+	ansible-dev-preflight ansible-dev-teardown ansible-dev-install ansible-dev-reset ansible-dev-deploy

build/Dockerfile.distro

@@ -30,13 +30,13 @@ COPY viperblock/ ./viperblock/
 
 # Build spinifex binary (spx)
 RUN COMMIT=$(cd spinifex && git rev-parse --short HEAD 2>/dev/null || echo "unknown") && \
-    cd spinifex && go build -ldflags "-s -w \
+    cd spinifex && GOFIPS140=v1.0.0 go build -ldflags "-s -w \
     -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=${VERSION} \
     -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Commit=${COMMIT}" \
     -o /out/spx cmd/spinifex/main.go
 
 # Build nbdkit viperblock plugin (.so)
-RUN cd viperblock && go build -o /out/nbdkit-viperblock-plugin.so -buildmode=c-shared nbd/viperblock.go
+RUN cd viperblock && GOFIPS140=v1.0.0 go build -o /out/nbdkit-viperblock-plugin.so -buildmode=c-shared nbd/viperblock.go
 
 # Copy setup scripts and systemd units
 RUN cp spinifex/scripts/setup.sh /out/setup.sh 2>/dev/null || true

build/microvm/init.sh

@@ -0,0 +1,56 @@
+#!/bin/sh
+mount -t proc  proc  /proc
+mount -t sysfs sys   /sys
+mount -t devtmpfs dev /dev
+
+# virtio_mmio is built-in (CONFIG_VIRTIO_MMIO=y) so virtio-net devices are
+# already enumerated by the time PID 1 runs — no modprobe needed.
+# qemu_fw_cfg and virtio_net are =m in stock Alpine linux-virt, so load them.
+modprobe qemu_fw_cfg 2>/dev/null || modprobe fw_cfg_sysfs 2>/dev/null || true
+modprobe virtio_net 2>/dev/null || true
+
+for i in 1 2 3 4 5; do
+    [ -f /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw ] && break
+    sleep 0.02
+done
+
+if [ ! -f /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw ]; then
+    echo "[init] FATAL: fw_cfg netcfg not available" >&2
+    exit 1
+fi
+
+. /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw
+
+for n in 0 1 2 3 4 5; do
+    eval mac=\$NIC${n}_MAC
+    [ -z "$mac" ] && continue
+
+    iface=""
+    for attempt in 1 2 3 4 5; do
+        iface=$(ip -o link | awk -v m="$mac" 'tolower($0) ~ tolower(m) {print $2}' | tr -d :)
+        [ -n "$iface" ] && break
+        sleep 0.02
+    done
+
+    if [ -z "$iface" ]; then
+        echo "[init] WARNING: no interface found for MAC $mac (NIC$n), skipping" >&2
+        continue
+    fi
+
+    eval cidr=\$NIC${n}_CIDR
+    eval gw=\$NIC${n}_GW
+    eval default=\$NIC${n}_DEFAULT
+    eval rdst=\$NIC${n}_ROUTE_DST
+    eval rvia=\$NIC${n}_ROUTE_VIA
+
+    ip link set "$iface" up
+    [ -n "$cidr" ] && ip addr add "$cidr" dev "$iface"
+    [ "$default" = "1" ] && [ -n "$gw" ] && ip route add default via "$gw" dev "$iface"
+    [ -n "$rdst" ] && [ -n "$rvia" ] && ip route add "$rdst" via "$rvia" dev "$iface"
+done
+
+cp /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/lb-agent-env/raw /etc/conf.d/lb-agent
+mkdir -p /etc/ssl/certs
+cp /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/ca-cert/raw /etc/ssl/certs/ca-certificates.crt
+
+exec /sbin/init

build/microvm/inittab

@@ -0,0 +1,4 @@
+::sysinit:/etc/rc.local
+::respawn:/bin/sh -c 'set -a; . /etc/conf.d/lb-agent; set +a; exec /usr/local/bin/lb-agent'
+::ctrlaltdel:/sbin/reboot
+::shutdown:/bin/umount -a -r

cmd/installer/autoinstall/autoinstall.go

@@ -143,7 +143,12 @@ func buildConfig() (*install.Config, error) {
 		role = "init"
 	}
 	cfg.ClusterRole = role
-	if role == "join" {
+	skipFormation := os.Getenv("SPINIFEX_SKIP_FORMATION") == "1"
+	if skipFormation {
+		cfg.SkipFormation = true
+	}
+
+	if role == "join" && !skipFormation {
 		joinAddr := os.Getenv("SPINIFEX_JOIN_ADDR")
 		if joinAddr == "" {
 			return nil, fmt.Errorf("SPINIFEX_JOIN_ADDR required when SPINIFEX_ROLE=join")

cmd/installer/firstboot/firstboot.go

@@ -52,6 +52,10 @@ type Config struct {
 	// and network failures are swallowed so the install is not gated on
 	// controller reachability.
 	InstallCallback string
+	// SkipFormation skips spx admin init/join. Used when a provisioning
+	// controller (e.g. bm-bootstrap.sh) owns cluster formation and will call
+	// spx admin init/join itself with the correct multi-node parameters.
+	SkipFormation bool
 }
 
 // Write drops the firstboot script and systemd unit into root, which should be
@@ -232,6 +236,9 @@ fi
 }
 
 func buildClusterCmd(cfg Config) string {
+	if cfg.SkipFormation {
+		return `echo "[firstboot] cluster formation skipped — provisioning controller will handle it"`
+	}
 	emailFlag := ""
 	if cfg.Email != "" {
 		// shellEscapeSingle keeps the email safe if it ever contains a

cmd/installer/install/config.go

@@ -55,6 +55,10 @@ type Config struct {
 	// JoinAddr is the primary node address (host:port) when ClusterRole is "join".
 	JoinAddr string
 
+	// SkipFormation skips spx admin init/join in firstboot; the provisioning
+	// controller (e.g. bm-bootstrap.sh) owns cluster formation instead.
+	SkipFormation bool
+
 	// CA certificate (PEM), optional.
 	HasCACert bool
 	CACert    string

cmd/installer/install/install.go

@@ -23,9 +23,11 @@ import (
 	"io"
 	"log/slog"
 	"net"
+	"net/http"
 	"os"
 	"os/exec"
 	"path/filepath"
+	"slices"
 	"strings"
 	"syscall"
 	"time"
@@ -77,6 +79,7 @@ func Run(cfg *Config) error {
 	}
 
 	slog.Info("installation complete")
+	fireInstallCallback()
 	promptRemoveUSB()
 	return reboot()
 }
@@ -172,6 +175,7 @@ func copyRootfs() error {
 		"--exclude=/etc/openvswitch/",
 		"--exclude=/var/lib/openvswitch/",
 		"--exclude=/var/lib/dhcpcd/",
+		"--exclude=/etc/ssh/ssh_host_*",
 		"--exclude=/lost+found",
 		"--exclude=/boot/efi",
 		"/", mountRoot+"/",
@@ -555,6 +559,11 @@ func installBootloader(disk string) error {
 		}
 		return biosErr
 	}
+	// Copy splash image and unicode font from the ISO (mounted at /cdrom) so the
+	// installed GRUB shows the same branded background as the installer GRUB.
+	// The font must be at /boot/grub/fonts/unicode.pf2 so update-grub finds it
+	// there and emits the same loadfont path as the ISO's grub.cfg — GRUB 2.12
+	// (trixie) needs the font in the boot partition, not just /usr/share/grub/.
 	copySplashImage(mountRoot)
 	copyGrubFont(mountRoot)
 
@@ -644,6 +653,26 @@ func promptRemoveUSB() {
 	}
 }
 
+// fireInstallCallback notifies the boot controller that installation is done
+// so it clears the PXE install flag before the node reboots. Without this,
+// a PXE-first boot order causes the node to reinstall on every reboot until
+// firstboot fires the callback — which it can never do if it never runs.
+// No-op when SPINIFEX_INSTALL_CALLBACK is not set (ISO/USB installs).
+func fireInstallCallback() {
+	url := strings.TrimSpace(os.Getenv("SPINIFEX_INSTALL_CALLBACK"))
+	if url == "" {
+		return
+	}
+	client := &http.Client{Timeout: 10 * time.Second}
+	resp, err := client.Get(url) //nolint:noctx // installer has no context; best-effort fire-and-forget
+	if err != nil {
+		slog.Warn("fireInstallCallback: request failed", "url", url, "err", err)
+		return
+	}
+	resp.Body.Close()
+	slog.Info("fireInstallCallback: notified boot controller", "url", url, "status", resp.StatusCode)
+}
+
 func reboot() error {
 	// sync filesystems before reboot so nothing is lost.
 	_ = run("sync")
@@ -668,6 +697,7 @@ func (c *Config) toFirstbootConfig() firstboot.Config {
 		JoinAddr:        c.JoinAddr,
 		Email:           c.Email,
 		InstallCallback: strings.TrimSpace(os.Getenv("SPINIFEX_INSTALL_CALLBACK")),
+		SkipFormation:   c.SkipFormation,
 	}
 }
 
@@ -713,9 +743,46 @@ func partitionPaths(disk string) (efi, root string) {
 	return disk + "2", disk + "3"
 }
 
-// copySplashImage copies the GRUB splash (embedded in the squashfs at build time by
-// inject-bins.sh) into the installed system so the post-install GRUB shows the same
-// branded background as the installer GRUB. Non-fatal — missing source is logged and skipped.
+// copyGrubFont copies the unicode.pf2 GRUB font into the installed system's
+// /boot/grub/fonts/ directory. This ensures update-grub finds the font at
+// /boot/grub/fonts/unicode.pf2 — the same path the ISO's grub.cfg uses —
+// so the generated grub.cfg enables gfxterm and the background image.
+// Without this, grub-mkconfig falls back to /usr/share/grub/unicode.pf2,
+// a path that GRUB 2.12 (trixie) may fail to resolve at boot time.
+// Non-fatal — a missing source is logged and skipped.
+func copyGrubFont(root string) {
+	candidates := []string{
+		"/cdrom/boot/grub/fonts/unicode.pf2", // ISO tree (preferred)
+		"/usr/share/grub/unicode.pf2",        // live system grub-common fallback
+	}
+	for _, src := range candidates {
+		in, err := os.Open(src)
+		if err != nil {
+			continue
+		}
+		defer in.Close()
+		dstDir := filepath.Join(root, "boot/grub/fonts")
+		if err := os.MkdirAll(dstDir, 0o755); err != nil {
+			slog.Warn("copyGrubFont: cannot create fonts dir", "err", err)
+			return
+		}
+		out, err := os.OpenFile(filepath.Join(dstDir, "unicode.pf2"), os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644)
+		if err != nil {
+			slog.Warn("copyGrubFont: cannot open destination", "err", err)
+			return
+		}
+		defer out.Close()
+		if _, err := io.Copy(out, in); err != nil {
+			slog.Warn("copyGrubFont: copy failed", "err", err)
+		}
+		return
+	}
+	slog.Warn("copyGrubFont: no unicode.pf2 found, splash may not display")
+}
+
+// copySplashImage copies the GRUB splash from the live ISO (/cdrom/boot/grub/splash.png)
+// into the installed system so the post-install GRUB shows the same branded background
+// as the installer. Non-fatal — a missing or unreadable source is logged and skipped.
 func copySplashImage(root string) {
 	const src = "/usr/share/spinifex/grub-splash.png"
 	in, err := os.Open(src)
@@ -741,34 +808,6 @@ func copySplashImage(root string) {
 	}
 }
 
-// copyGrubFont copies the unicode font into the installed system's
-// /boot/grub/fonts/ so the loadfont path in 05_spinifex resolves at boot.
-// grub-install does not copy fonts; we mirror what build-iso.sh does.
-func copyGrubFont(root string) {
-	const src = "/usr/share/grub/unicode.pf2"
-	in, err := os.Open(src)
-	if err != nil {
-		slog.Warn("copyGrubFont: font not found, graphical GRUB may not work", "path", src)
-		return
-	}
-	defer in.Close()
-
-	dstDir := filepath.Join(root, "boot/grub/fonts")
-	if err := os.MkdirAll(dstDir, 0o755); err != nil {
-		slog.Warn("copyGrubFont: cannot create fonts dir", "err", err)
-		return
-	}
-	out, err := os.OpenFile(filepath.Join(dstDir, "unicode.pf2"), os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644)
-	if err != nil {
-		slog.Warn("copyGrubFont: cannot open destination", "err", err)
-		return
-	}
-	defer out.Close()
-	if _, err := io.Copy(out, in); err != nil {
-		slog.Warn("copyGrubFont: copy failed", "err", err)
-	}
-}
-
 // maskSystemdUnit creates a symlink to /dev/null for the given unit, which is
 // the standard way to permanently disable a unit so systemd will never start it.
 func maskSystemdUnit(root, unit string) error {
@@ -812,8 +851,8 @@ func bindChrootMounts() error {
 // unbindChrootMounts unmounts the virtual filesystems in reverse order.
 // Errors are logged but not returned — this is best-effort cleanup.
 func unbindChrootMounts() {
-	for i := len(chrootMountPaths) - 1; i >= 0; i-- {
-		_ = run("umount", filepath.Join(mountRoot, chrootMountPaths[i]))
+	for _, v := range slices.Backward(chrootMountPaths) {
+		_ = run("umount", filepath.Join(mountRoot, v))
 	}
 }
 

cmd/installer/main.go

@@ -24,6 +24,7 @@ import (
 	"github.com/mulgadc/spinifex/cmd/installer/autoinstall"
 	"github.com/mulgadc/spinifex/cmd/installer/install"
 	"github.com/mulgadc/spinifex/cmd/installer/ui"
+	_ "github.com/mulgadc/spinifex/internal/fipsboot"
 )
 
 func main() {

cmd/lb-agent/main.go

@@ -19,6 +19,8 @@ import (
 	"syscall"
 
 	"github.com/mulgadc/spinifex/spinifex/lbagent"
+
+	_ "github.com/mulgadc/spinifex/internal/fipsboot"
 )
 
 func main() {

cmd/spinifex/cmd/admin.go

@@ -17,6 +17,7 @@ along with this program.  If not, see <https://www.gnu.org/licenses/>.
 package cmd
 
 import (
+	"bufio"
 	"bytes"
 	"context"
 	"crypto/tls"
@@ -48,6 +49,7 @@ import (
 	"github.com/mulgadc/spinifex/spinifex/gpu"
 	handlers_ec2_vpc "github.com/mulgadc/spinifex/spinifex/handlers/ec2/vpc"
 	handlers_iam "github.com/mulgadc/spinifex/spinifex/handlers/iam"
+	"github.com/mulgadc/spinifex/spinifex/objectstore"
 	"github.com/mulgadc/spinifex/spinifex/utils"
 	"github.com/mulgadc/viperblock/viperblock"
 	"github.com/mulgadc/viperblock/viperblock/backends/s3"
@@ -84,6 +86,14 @@ var supportedPlatforms = map[string]bool{
 	"Windows":    true,
 }
 
+// Mirrors the gateway RegisterImage allowlist so admin imports can't write an
+// AMI with a boot mode that RegisterImage would reject.
+var supportedBootModes = map[string]bool{
+	"bios":           true,
+	"uefi":           true,
+	"uefi-preferred": true,
+}
+
 var adminCmd = &cobra.Command{
 	Use:   "admin",
 	Short: "Administrative commands for Spinifex platform management",
@@ -143,6 +153,19 @@ var imagesListCmd = &cobra.Command{
 	Run:   runimagesListCmd,
 }
 
+var imagesRemoveCmd = &cobra.Command{
+	Use:   "remove",
+	Short: "Remove an admin-imported system AMI",
+	Long: `Delete an AMI imported via 'spx admin images import', including its
+backing block storage and snapshot artefacts. Only operates on AMIs with a
+non-account owner (e.g. "system"); account-owned AMIs must be removed via
+'aws ec2 deregister-image' followed by 'aws ec2 delete-snapshot'.
+
+Refuses to delete an AMI that has dependent volumes or copied snapshots/AMIs
+unless --force is passed. Prompts for confirmation unless --yes is passed.`,
+	Run: runimagesRemoveCmd,
+}
+
 var accountCmd = &cobra.Command{
 	Use:   "account",
 	Short: "Manage Spinifex accounts",
@@ -225,6 +248,7 @@ func init() {
 	adminCmd.AddCommand(imagesCmd)
 	imagesCmd.AddCommand(imagesImportCmd)
 	imagesCmd.AddCommand(imagesListCmd)
+	imagesCmd.AddCommand(imagesRemoveCmd)
 
 	adminCmd.AddCommand(accountCmd)
 	accountCmd.AddCommand(accountCreateCmd)
@@ -304,9 +328,17 @@ func init() {
 	imagesImportCmd.Flags().String("version", "", "Specified distro version (e.g 12)")
 	imagesImportCmd.Flags().String("arch", "", "Specified distro arch (e.g aarch64, arm64, x86_64)")
 	imagesImportCmd.Flags().String("platform", "Linux/UNIX", "Specified platform (e.g Linux/UNIX, Windows)")
+	imagesImportCmd.Flags().String("boot-mode", "", "Boot mode for the imported AMI (bios|uefi|uefi-preferred). Required with --file. Overrides the catalog value when used with --name.")
 	imagesImportCmd.Flags().StringSlice("tag", nil, "Tag to apply to the imported AMI as key=value (repeatable; e.g. --tag spinifex:managed-by=elbv2)")
 	imagesImportCmd.Flags().Bool("force", false, "Force command execution (overwrites existing files)")
 	imagesImportCmd.Flags().Bool("skip-verify", false, "Skip catalog-image checksum verification (INSECURE; operator assumes integrity responsibility)")
+
+	imagesRemoveCmd.Flags().String("image-id", "", "AMI ID to remove (required)")
+	imagesRemoveCmd.Flags().Bool("force", false, "Bypass dependency, ownership and config-corrupt checks (salvage mode)")
+	imagesRemoveCmd.Flags().Bool("yes", false, "Skip interactive confirmation prompt")
+	if err := imagesRemoveCmd.MarkFlagRequired("image-id"); err != nil {
+		panic(err)
+	}
 }
 
 func runimagesImportCmd(cmd *cobra.Command, args []string) {
@@ -376,6 +408,22 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 		}
 	}
 
+	// --file imports have no catalog metadata to inherit from, so the operator
+	// must declare the boot mode explicitly — guessing would silently produce
+	// a BIOS AMI from a UEFI-only image (or vice versa) and fail at launch.
+	// --name imports inherit from the catalog; the flag overrides when set.
+	bootModeFlag, _ := cmd.Flags().GetString("boot-mode")
+	if bootModeFlag != "" {
+		if !supportedBootModes[bootModeFlag] {
+			fmt.Fprintf(os.Stderr, "Unsupported --boot-mode %q (expected bios|uefi|uefi-preferred)\n", bootModeFlag)
+			os.Exit(1)
+		}
+		image.BootMode = bootModeFlag
+	} else if imageName == "" {
+		fmt.Fprintf(os.Stderr, "--boot-mode is required when importing via --file (expected bios|uefi|uefi-preferred)\n")
+		os.Exit(1)
+	}
+
 	// --tag k=v (repeatable) merges user-supplied tags into the AMI. Overrides
 	// catalog tags on key collision so operators can re-tag a known image.
 	tagFlags, _ := cmd.Flags().GetStringSlice("tag")
@@ -496,6 +544,9 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 	manifest.AMIMetadata.Virtualization = "hvm"
 	manifest.AMIMetadata.ImageOwnerAlias = "system"
 	manifest.AMIMetadata.VolumeSizeGiB = utils.SafeInt64ToUint64(imageStat.Size() / 1024 / 1024 / 1024)
+	manifest.AMIMetadata.BootMode = image.BootMode
+	manifest.AMIMetadata.Distro = image.Distro
+	manifest.AMIMetadata.DistroFamily = utils.DistroFamily(image.Distro)
 
 	// Copy catalog-provided tags (e.g. spinifex:managed-by for system AMIs)
 	// onto the imported AMI so the UI can filter them out.
@@ -574,6 +625,136 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 	fmt.Printf("✅ Image import complete. Image-ID (AMI): %s\n", volumeId)
 }
 
+func runimagesRemoveCmd(cmd *cobra.Command, args []string) {
+	imageID, _ := cmd.Flags().GetString("image-id")
+	force, _ := cmd.Flags().GetBool("force")
+	yes, _ := cmd.Flags().GetBool("yes")
+
+	cfgFile, _ := cmd.Flags().GetString("config")
+	if cfgFile == "" {
+		cfgFile = DefaultConfigFile()
+	}
+
+	appConfig, err := config.LoadConfig(cfgFile)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "Error loading config file:", err)
+		os.Exit(1)
+	}
+
+	node := appConfig.Nodes[appConfig.Node]
+	store := objectstore.NewS3ObjectStoreFromConfig(
+		node.Predastore.Host,
+		node.Predastore.Region,
+		node.Predastore.AccessKey,
+		node.Predastore.SecretKey,
+	)
+	bucket := node.Predastore.Bucket
+
+	preview, err := admin.PreviewRemoveSystemImage(store, bucket, imageID)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "Failed to inspect AMI:", err)
+		os.Exit(1)
+	}
+
+	// Print metadata block.
+	fmt.Println("About to remove system AMI:")
+	fmt.Println()
+	fmt.Printf("  Image ID:        %s\n", preview.ImageID)
+	switch {
+	case !preview.ConfigPresent && !preview.ConfigCorrupt:
+		fmt.Println("  Name:            <unknown — config.json missing>")
+		fmt.Println("  Owner:           <unknown>")
+	case preview.ConfigCorrupt:
+		fmt.Println("  Name:            <unknown — config.json corrupt>")
+		fmt.Println("  Owner:           <unknown>")
+	default:
+		fmt.Printf("  Name:            %s\n", preview.Name)
+		fmt.Printf("  Owner:           %s\n", preview.Owner)
+		if !preview.Created.IsZero() {
+			fmt.Printf("  Created:         %s\n", preview.Created.UTC().Format("2006-01-02T15:04:05Z"))
+		}
+	}
+	fmt.Printf("  Backing storage: %s/      (%d objects, %s)\n",
+		preview.ImageID, preview.AMIObjectCount, utils.HumanBytes(utils.SafeInt64ToUint64(preview.AMIBytesTotal)))
+	fmt.Printf("                   %s/ (%d objects, %s)\n",
+		admin.SnapPrefix(preview.ImageID), preview.SnapObjectCount, utils.HumanBytes(utils.SafeInt64ToUint64(preview.SnapBytesTotal)))
+	fmt.Println()
+
+	// Account-owned guard before salvage / dependents — the AWS-flow hint is
+	// the most useful thing to surface for this kind of mistake.
+	if preview.ConfigPresent && !preview.IsSystemOwned && !force {
+		fmt.Fprintf(os.Stderr,
+			"Refusing to remove: %s is account-owned (%s).\n"+
+				"Use 'aws ec2 deregister-image --image-id %s' followed by 'aws ec2 delete-snapshot ...'.\n",
+			preview.ImageID, preview.Owner, preview.ImageID)
+		os.Exit(1)
+	}
+
+	if !preview.ConfigPresent && !force {
+		fmt.Fprintln(os.Stderr, "AMI config.json missing or corrupt; re-run with --force to salvage backing blocks.")
+		os.Exit(1)
+	}
+
+	if !preview.Dependents.Empty() && !force {
+		fmt.Fprintln(os.Stderr, "Refusing to remove: dependent resources reference this image.")
+		printDependents(os.Stderr, preview.Dependents)
+		fmt.Fprintln(os.Stderr, "Remove them first (e.g. 'aws ec2 terminate-instances', 'aws ec2 delete-snapshot', 'aws ec2 deregister-image') or re-run with --force.")
+		os.Exit(1)
+	}
+
+	if force && (!preview.ConfigPresent || !preview.Dependents.Empty()) {
+		fmt.Println("⚠️  --force: skipping dependency check and ownership check.")
+		if !preview.Dependents.Empty() {
+			printDependents(os.Stdout, preview.Dependents)
+		}
+		fmt.Println()
+	}
+
+	fmt.Println("This is permanent and cannot be undone.")
+	if !yes {
+		fmt.Print("Type 'yes' to proceed: ")
+		reader := bufio.NewReader(os.Stdin)
+		answer, _ := reader.ReadString('\n')
+		if strings.TrimSpace(answer) != "yes" {
+			fmt.Println("Aborted.")
+			return
+		}
+	}
+
+	res, err := admin.RemoveSystemImage(store, bucket, admin.RemoveImageOpts{
+		ImageID: imageID,
+		Force:   force,
+	})
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "Remove failed:", err)
+		os.Exit(1)
+	}
+
+	fmt.Printf("✅ Removed AMI %s (freed %s across %d objects).\n",
+		imageID, utils.HumanBytes(utils.SafeInt64ToUint64(res.BytesFreed)), res.ObjectsDeleted)
+}
+
+func printDependents(w io.Writer, d admin.Dependents) {
+	if len(d.Volumes) > 0 {
+		fmt.Fprintf(w, "  Volumes (%d):\n", len(d.Volumes))
+		for _, v := range d.Volumes {
+			fmt.Fprintf(w, "    - %s\n", v)
+		}
+	}
+	if len(d.Snapshots) > 0 {
+		fmt.Fprintf(w, "  Snapshots (%d):\n", len(d.Snapshots))
+		for _, s := range d.Snapshots {
+			fmt.Fprintf(w, "    - %s\n", s)
+		}
+	}
+	if len(d.AMIs) > 0 {
+		fmt.Fprintf(w, "  AMIs (%d):\n", len(d.AMIs))
+		for _, a := range d.AMIs {
+			fmt.Fprintf(w, "    - %s\n", a)
+		}
+	}
+}
+
 // List remote images available
 func runimagesListCmd(cmd *cobra.Command, args []string) {
 	//fmt.Println(availableImages)
@@ -993,8 +1174,8 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	var predastoreNodeID int
 	if predastoreNodesStr != "" {
 		ips := strings.Split(predastoreNodesStr, ",")
-		if len(ips) < 3 {
-			fmt.Fprintf(os.Stderr, "❌ Error: --predastore-nodes requires at least 3 IPs, got %d\n", len(ips))
+		if len(ips) < 2 {
+			fmt.Fprintf(os.Stderr, "❌ Error: --predastore-nodes requires at least 2 IPs, got %d\n", len(ips))
 			os.Exit(1)
 		}
 
@@ -1113,6 +1294,20 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 
 	finalizeNodeSetup(spxRoot, certPath, bootstrapResult.AdminAccessKey, bootstrapResult.AdminSecretKey, region, bindIP, advertiseIP)
 
+	// Write node.conf so spx admin banner works on source installs (not just ISO).
+	nodeHostname, _ := os.Hostname()
+	if nodeHostname == "" {
+		nodeHostname = node
+	}
+	nodeConfPath := filepath.Join(configDir, "node.conf")
+	if err := writeNodeConf(nodeConfPath, map[string]string{
+		"MANAGEMENT_IP":    advertiseIP,
+		"MANAGEMENT_IFACE": "br-wan",
+		"NODE_HOSTNAME":    nodeHostname,
+	}); err != nil {
+		fmt.Fprintf(os.Stderr, "⚠️  Warning: could not write %s: %v\n", nodeConfPath, err)
+	}
+
 	// Print success message
 	fmt.Println("\n🎉 Spinifex initialization complete!")
 	fmt.Println()
@@ -1269,7 +1464,7 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 
 	// Generate multi-node predastore config
 	var predastoreNodeID int
-	hasPredastoreConfig := len(predastoreNodes) >= 3
+	hasPredastoreConfig := len(predastoreNodes) >= 2
 	if hasPredastoreConfig {
 		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, accessKey, secretKey, region, natsToken, configDir, bindIP)
 		if err != nil {
@@ -1558,7 +1753,7 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 		}
 
 		fmt.Printf("   Waiting for cluster formation... (%d/%d nodes joined)\n", statusResp.Joined, statusResp.Expected)
-		time.Sleep(2 * time.Second)
+		time.Sleep(500 * time.Millisecond)
 	}
 
 	fmt.Printf("✅ Cluster formation complete! (%d nodes)\n\n", statusResp.Expected)
@@ -1682,7 +1877,7 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 
 	// Generate multi-node predastore config
 	var predastoreNodeID int
-	hasPredastoreConfig := len(predastoreNodes) >= 3
+	hasPredastoreConfig := len(predastoreNodes) >= 2
 
 	if hasPredastoreConfig {
 		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, creds.AccessKey, creds.SecretKey, creds.Region, creds.NatsToken, configDir, bindIP)
@@ -2361,6 +2556,12 @@ func runAdminBanner(cmd *cobra.Command, _ []string) {
 	}
 
 	// Resolve current IP from the management interface at runtime.
+	// If node.conf is absent or has no MANAGEMENT_IFACE (source installs before
+	// the first spx admin init writes node.conf), fall back to br-wan which
+	// setup-ovn.sh always creates as the management bridge.
+	if iface == "" {
+		iface = "br-wan"
+	}
 	currentIP := resolveIfaceIP(iface)
 	if currentIP == "" {
 		currentIP = recordedIP // fall back to value recorded at install time

cmd/spinifex/cmd/admin_gpu.go

@@ -120,18 +120,16 @@ func runAdminGpuStatus(cmd *cobra.Command, _ []string) {
 		fmt.Printf("Passthrough:     enabled\n")
 		fmt.Printf("GPU pool:        %d/%d allocated\n", resp.AllocGPUs, resp.TotalGPUs)
 
-		var g5Types []string
+		var gpuTypes []string
 		for _, cap := range resp.InstanceTypes {
-			if strings.HasPrefix(cap.Name, "g5.") {
-				g5Types = append(g5Types, cap.Name)
-			}
+			gpuTypes = append(gpuTypes, cap.Name)
 		}
-		if len(g5Types) > 0 {
-			fmt.Printf("Instance types:  %s\n", strings.Join(g5Types, " "))
+		if len(gpuTypes) > 0 {
+			fmt.Printf("Instance types:  %s\n", strings.Join(gpuTypes, " "))
 		}
 	} else if resp.GPUCapable {
 		fmt.Printf("Passthrough:     disabled\n")
-		fmt.Printf("Instance types:  (none — run 'spx admin gpu enable' to activate g5.*)\n")
+		fmt.Printf("Instance types:  (none — run 'spx admin gpu enable' to activate)\n")
 	} else {
 		fmt.Printf("Passthrough:     disabled\n")
 		fmt.Printf("Instance types:  (prerequisites not met)\n")
@@ -209,6 +207,9 @@ func gpuToggle(_ *cobra.Command, enable bool) {
 		}
 		if r.GPUPassthrough == enable {
 			fmt.Println(" done.")
+			// Refresh the MOTD banner immediately so the login prompt
+			// reflects the new GPU passthrough state without requiring a reboot.
+			_ = exec.Command("spx", "admin", "banner", "--boot-check").Run()
 			runAdminGpuStatus(adminGpuStatusCmd, nil)
 			return
 		}
@@ -306,6 +307,30 @@ func runAdminGpuSetup(_ *cobra.Command, _ []string) {
 		fmt.Println("    Already blacklisted")
 	}
 
+	fmt.Println("==> Configuring amdgpu blacklist")
+	const amdgpuConf = "/etc/modprobe.d/blacklist-amdgpu.conf"
+	hasAMD := false
+	for _, d := range devices {
+		if d.Vendor == gpu.VendorAMD {
+			hasAMD = true
+			break
+		}
+	}
+	if hasAMD {
+		if _, err := os.Stat(amdgpuConf); os.IsNotExist(err) {
+			if err := os.WriteFile(amdgpuConf, []byte("blacklist amdgpu\n"), 0o644); err != nil {
+				fmt.Fprintf(os.Stderr, "Failed to write amdgpu blacklist: %v\n", err)
+				os.Exit(1)
+			}
+			fmt.Println("    amdgpu blacklisted")
+			rebootNeeded = true
+		} else {
+			fmt.Println("    Already blacklisted")
+		}
+	} else {
+		fmt.Println("    No AMD GPUs — skipping")
+	}
+
 	fmt.Println("==> Configuring vfio-pci early binding")
 	const vfioPCIConf = "/etc/modprobe.d/vfio-pci.conf"
 	existing, _ := os.ReadFile(vfioPCIConf)
@@ -350,7 +375,7 @@ func runAdminGpuSetup(_ *cobra.Command, _ []string) {
 		}
 		fmt.Println("\nSetup complete — reboot required.")
 		fmt.Println("  sudo reboot")
-		fmt.Println("  Then re-run: sudo spx admin gpu setup")
+		fmt.Println("  Then run: sudo spx admin gpu enable")
 		return
 	}
 
@@ -363,15 +388,29 @@ func runAdminGpuSetup(_ *cobra.Command, _ []string) {
 
 	fmt.Println("==> Verifying GPU driver binding")
 	for _, d := range devices {
-		driverLink := "/sys/bus/pci/devices/" + d.PCIAddress + "/driver"
-		target, err := os.Readlink(driverLink)
-		driver := filepath.Base(target)
-		if err != nil || driver != "vfio-pci" {
-			fmt.Fprintf(os.Stderr, "%s is bound to %q, not vfio-pci\n", d.PCIAddress, driver)
-			fmt.Fprintf(os.Stderr, "  Check: ls -la %s\n", driverLink)
+		current, err := gpu.CurrentDriver(d.PCIAddress)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "Cannot read driver for %s: %v\n", d.PCIAddress, err)
 			os.Exit(1)
 		}
-		fmt.Printf("    %s - vfio-pci\n", d.PCIAddress)
+		if current == "vfio-pci" {
+			fmt.Printf("    %s → vfio-pci\n", d.PCIAddress)
+			continue
+		}
+		if current != "" {
+			// Bound to native driver — blacklist/initramfs changes may not have taken effect.
+			fmt.Fprintf(os.Stderr, "%s is bound to %q, not vfio-pci — reboot may be required\n", d.PCIAddress, current)
+			fmt.Fprintf(os.Stderr, "  Check: ls -la /sys/bus/pci/devices/%s/driver\n", d.PCIAddress)
+			os.Exit(1)
+		}
+		// Device is unbound (ids= did not auto-claim it) — bind explicitly via driver_override.
+		fmt.Printf("    %s unbound — binding to vfio-pci\n", d.PCIAddress)
+		if _, err := gpu.BindVFIO(d.PCIAddress); err != nil {
+			fmt.Fprintf(os.Stderr, "Failed to bind %s to vfio-pci: %v\n", d.PCIAddress, err)
+			fmt.Fprintf(os.Stderr, "  Check: ls -la /sys/bus/pci/devices/%s/driver\n", d.PCIAddress)
+			os.Exit(1)
+		}
+		fmt.Printf("    %s → vfio-pci\n", d.PCIAddress)
 	}
 
 	fmt.Println("==> Enabling GPU passthrough")

cmd/spinifex/cmd/admin_test.go

@@ -310,3 +310,24 @@ func TestWritePredastoreEncryptionKey_CreatesMissingDir(t *testing.T) {
 	require.NoError(t, err)
 	assert.True(t, dirInfo.IsDir())
 }
+
+// TestImagesRemoveCmd_FlagSchema guards the public flag surface for
+// `spx admin images remove`. --image-id is required so cobra rejects a bare
+// invocation; --force/--yes default to false.
+func TestImagesRemoveCmd_FlagSchema(t *testing.T) {
+	imageIDFlag := imagesRemoveCmd.Flags().Lookup("image-id")
+	require.NotNil(t, imageIDFlag, "--image-id must be defined")
+	assert.Equal(t, []string{"true"}, imageIDFlag.Annotations[cobraRequiredAnnotation],
+		"--image-id must be marked required")
+
+	forceFlag := imagesRemoveCmd.Flags().Lookup("force")
+	require.NotNil(t, forceFlag)
+	assert.Equal(t, "false", forceFlag.DefValue)
+
+	yesFlag := imagesRemoveCmd.Flags().Lookup("yes")
+	require.NotNil(t, yesFlag)
+	assert.Equal(t, "false", yesFlag.DefValue)
+}
+
+// cobraRequiredAnnotation is the annotation key cobra uses to mark required flags.
+const cobraRequiredAnnotation = "cobra_annotation_bash_completion_one_required_flag"

cmd/spinifex/cmd/cluster.go

@@ -98,7 +98,7 @@ func runClusterShutdown(cmd *cobra.Command, args []string) {
 			}
 			nc.Flush()
 			// Wait briefly for messages to propagate
-			time.Sleep(2 * time.Second)
+			time.Sleep(200 * time.Millisecond)
 			fmt.Printf("[INFRA] Complete\n")
 			break
 		}

cmd/spinifex/cmd/templates/predastore-multinode.toml

@@ -9,8 +9,13 @@ port = 8443
 debug = false
 
 [rs]
+{{- if eq (len .Nodes) 2}}
+data = 1
+parity = 1
+{{- else}}
 data = 2
 parity = 1
+{{- end}}
 {{- range .Nodes}}
 
 [[db]]

cmd/spinifex/cmd/templates/predastore.toml

@@ -51,19 +51,19 @@ secret_access_key = "{{.SecretKey}}"
 # Using distributed hash ring for object > shard placement.
 [[nodes]]
 id = 1
-host = "0.0.0.0"
+host = "127.0.0.1"
 port = 9991
 path = "distributed/nodes/node-1/"
 
 [[nodes]]
 id = 2
-host = "0.0.0.0"
+host = "127.0.0.1"
 port = 9992
 path = "distributed/nodes/node-2/"
 
 [[nodes]]
 id = 3
-host = "0.0.0.0"
+host = "127.0.0.1"
 port = 9993
 path = "distributed/nodes/node-3/"
 

cmd/spinifex/main.go

@@ -16,7 +16,10 @@ along with this program.  If not, see <https://www.gnu.org/licenses/>.
 */
 package main
 
-import "github.com/mulgadc/spinifex/cmd/spinifex/cmd"
+import (
+	"github.com/mulgadc/spinifex/cmd/spinifex/cmd"
+	_ "github.com/mulgadc/spinifex/internal/fipsboot"
+)
 
 func main() {
 	cmd.Execute()

docs/COMMANDS.md

@@ -45,7 +45,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 
 | Command | Flags | Prerequisites | Basic Logic | Output | Status |
 |---------|-------|---------------|-------------|--------|--------|
-| `spx top nodes` | `--timeout` (default: 3s) | Cluster must be running (NATS) | Publishes to `spinifex.node.status` fan-out topic → collects CPU/memory usage per node → aggregates instance type capacity across all nodes → prints two tables: per-node resource usage and cluster-wide instance type availability | Table 1: NAME, CPU (used/total), MEM (used/total), VMs. Table 2: INSTANCE TYPE, AVAILABLE, VCPU, MEMORY | **DONE** |
+| `spx top nodes` | `--timeout` (default: 3s) | Cluster must be running (NATS) | Publishes to `spinifex.node.status` fan-out topic → collects CPU/memory usage per node → aggregates instance type capacity across all nodes → prints two tables: per-node resource usage and cluster-wide instance type availability | Table 1: NAME, CPU (used/total), MEM (used/total), GPU (used/total), VMs — GPU column shows `allocated/total` when passthrough is enabled, `0/count*` when the node is GPU-capable but passthrough is disabled, or `-` when no GPU is present. Table 2: INSTANCE TYPE, AVAILABLE, VCPU, MEMORY | **DONE** |
 
 ### Cluster Initialization
 
@@ -89,8 +89,9 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 
 | Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
 |---------|-------|---------------|-------------|------------|--------|
-| `spx admin images import` | `--name`, `--file`, `--force`, `--skip-verify` | Cluster must be running; either `--name` (catalog download) or `--file` (operator-supplied media) | Catalog imports (`--name`) download the image, fetch the catalog `Checksum` URL, and verify the SHA-256/SHA-512 digest before extraction. Mismatch fails closed; the cached file is left on disk and `--force` re-downloads. `--file` imports skip verification — operator-supplied media is outside Spinifex's trust boundary, and the skip is logged at INFO for audit. `--skip-verify` bypasses verification for catalog imports and emits a WARN slog + stderr notice; use only for debugging or when upstream mirrors are confirmed-broken. | 1. Import valid catalog image (verifies checksum)<br>2. Tampered cache hit fails with `ErrChecksumMismatch`<br>3. `--force` recovers after a mismatch<br>4. `--file` import skips verification<br>5. `--skip-verify` bypasses checksum with WARN | **DONE** |
+| `spx admin images import` | `--name`, `--file`, `--distro`, `--version`, `--arch`, `--platform`, `--boot-mode` (bios/uefi/uefi-preferred), `--tag`, `--force`, `--skip-verify` | Cluster must be running; either `--name` (catalog download) or `--file` (operator-supplied media). `--file` mode additionally requires `--distro`, `--version`, `--arch`, `--boot-mode`. | Catalog imports (`--name`) download the image, fetch the catalog `Checksum` URL, verify the SHA-256/SHA-512 digest, and inherit `BootMode` from the catalog entry. `--boot-mode` overrides the catalog value when set. Mismatch fails closed; the cached file is left on disk and `--force` re-downloads. `--file` imports skip checksum verification (operator-supplied media is outside Spinifex's trust boundary, the skip is logged at INFO for audit) and require an explicit `--boot-mode` because there is no catalog metadata to inherit from. `--skip-verify` bypasses verification for catalog imports and emits a WARN slog + stderr notice; use only for debugging or when upstream mirrors are confirmed-broken. | 1. Import valid catalog image (verifies checksum)<br>2. Tampered cache hit fails with `ErrChecksumMismatch`<br>3. `--force` recovers after a mismatch<br>4. `--file` import skips verification<br>5. `--skip-verify` bypasses checksum with WARN<br>6. `--file` without `--boot-mode` rejected<br>7. `--boot-mode=invalid` rejected | **DONE** |
 | `spx admin images list` | — | None | Lists available OS images that can be imported or downloaded | 1. List available images | **DONE** |
+| `spx admin images remove` | `--image-id` (required), `--force`, `--yes` | Cluster must be running; predastore reachable. Targets AMIs imported via `spx admin images import` (non-account `ImageOwnerAlias`, e.g. `"system"`). | Loads `ami-<id>/config.json`, walks transitive dependents — copied snapshots whose `VolumeID == imageID`, volumes whose `SnapshotID` references the internal `snap-ami-<id>` or any derived snap, and account AMIs created via `CopyImage` whose `SnapshotID` is a derived snap — then prompts (skipped with `--yes`) before deleting `ami-<id>/config.json` (the DescribeImages barrier) followed by the rest of `ami-<id>/` and `snap-ami-<id>/`. Account-owned AMIs are refused with a hint pointing at `aws ec2 deregister-image` + `aws ec2 delete-snapshot`. `--force` bypasses the dependency, ownership and config-corrupt checks for salvage of orphaned blocks. | 1. Happy path: system AMI, no deps → deleted<br>2. Account-owned → refused with AWS-flow hint<br>3. Missing/corrupt config → `InvalidAMIID.NotFound` (salvageable with `--force`)<br>4. Dependent volume (direct or via copied snap) → refused<br>5. Dependent account AMI → refused<br>6. `--force` overrides dependents<br>7. Idempotent re-run after partial delete | **DONE** |
 
 #### Image integrity verification (CMMC SI.L1-3.14.2)
 
@@ -123,6 +124,44 @@ served. A mirror compromise that swaps both image and sums file is not
 detected; closing that gap requires GPG signature verification of the sums
 file, deferred to a later phase.
 
+#### `spx admin images remove` caveats
+
+Admin-imported AMIs (`ImageOwnerAlias = "system"`) live
+under the `ami-<id>/` S3 prefix and use a viperblock-internal snap checkpoint
+at `snap-ami-<id>/` — there is no `snap-<id>/metadata.json`. The AWS handlers
+(`DeregisterImage`, `DeleteSnapshot`) reject system owners with
+`UnauthorizedOperation`, which is the right behaviour for tenant API callers
+but leaves no AWS-flow path to reclaim space. `spx admin images remove` is
+the admin-trust-boundary counterpart that performs the dependency walk and
+hard-deletes the blocks directly against predastore.
+
+`CopyImage` of a system AMI is metadata-only: it writes a fresh
+`snap-<acct>/metadata.json` whose `VolumeID` points at `ami-<sys>` and a new
+`ami-<acct>/config.json` referencing that snap. Volumes launched from the
+copied AMI read transitively from `ami-<sys>/chunks/...`. The remove command
+walks this transitive set and refuses if anything references the target.
+
+**TOCTOU window:** between the safety scan and the `config.json` delete a
+concurrent `RunInstances` against the AMI could create a new dependent
+volume. The window is sub-second on a healthy cluster. The admin running
+this command is expected to know the fleet's operational state; if the race
+fires the result is a `vol-<id>` with deleted backing blocks, recovered by
+terminating the orphaned instance.
+
+**`--force` bypasses every safety check** (dependents, ownership, missing /
+corrupt `config.json`). Use only for salvage of orphaned blocks. Running it
+against a live system AMI corrupts every dependent volume on the next disk
+read.
+
+### GPU Management
+
+| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
+|---------|-------|---------------|-------------|------------|--------|
+| `spx admin gpu status` | `--node` (default: local node) | Cluster must be running (NATS) | Queries `spinifex.node.status` fan-out → finds the target node response → prints Node, GPU hardware (model list or "none detected"), IOMMU state, vfio-pci state, passthrough enabled/disabled, and GPU pool allocation (`allocated/total`). Also lists GPU-capable instance types when passthrough is active. | 1. Local node with passthrough enabled (shows pool and instance types)<br>2. Local node capable but passthrough disabled (shows "run enable")<br>3. Node without GPU hardware (shows "none detected")<br>4. `--node` targets a remote node<br>5. Node not found or not responding (error) | **DONE** |
+| `spx admin gpu enable` | — | Cluster must be running (NATS); node must be GPU-capable (IOMMU + vfio-pci configured) | Checks current passthrough state via NATS → errors if already enabled or prerequisites not met (directs to `setup`) → writes `gpu_passthrough = true` to `spinifex.toml` via `admin.SetGPUPassthrough` → sends SIGHUP to `spinifex-daemon` via `systemctl kill -s HUP` → polls node status for up to 30 s until daemon confirms new state → prints final `gpu status` output. Must be run directly on the target host. | 1. Enable on capable node<br>2. Already enabled (no-op)<br>3. Prerequisites not met (error, suggests `setup`)<br>4. Daemon does not confirm within 30 s (timeout error) | **DONE** |
+| `spx admin gpu disable` | — | Cluster must be running (NATS); no GPU instances may be running | Checks current passthrough state via NATS → errors if already disabled or if `AllocGPUs > 0` (must terminate GPU instances first) → writes `gpu_passthrough = false` → sends SIGHUP to `spinifex-daemon` → polls for up to 30 s → prints final `gpu status` output. Must be run directly on the target host. | 1. Disable with no GPU instances running<br>2. Already disabled (no-op)<br>3. GPU instances running (error with count)<br>4. Daemon does not confirm within 30 s (timeout error) | **DONE** |
+| `spx admin gpu setup` | — | Must be run as root; NVIDIA or AMD GPU must be present | Idempotent host configuration for GPU passthrough. Steps (skipped if already applied): detect GPUs via `gpu.Discover` → collect PCI IDs for all IOMMU-group siblings → check/enable IOMMU in GRUB (`intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt`) → write vfio udev rule (`/etc/udev/rules.d/99-spinifex-vfio.rules`) → blacklist nouveau (`/etc/modprobe.d/blacklist-nouveau.conf`) → blacklist amdgpu if AMD GPU present (`/etc/modprobe.d/blacklist-amdgpu.conf`) → write vfio-pci early binding config (`/etc/modprobe.d/vfio-pci.conf`) → add vfio modules to initramfs. If any change requires a reboot: runs `update-initramfs -u` and exits with reboot instructions. After reboot: verifies `vfio_pci` module is loaded → verifies each GPU is bound to `vfio-pci` (binds explicitly via `driver_override` if unbound) → calls `gpu enable` to activate passthrough. | 1. First run on unconfigured host (GRUB updated, reboot required)<br>2. Re-run after reboot (verifies bindings, enables passthrough)<br>3. Already fully configured (all steps skipped, enables passthrough)<br>4. AMD GPU present (blacklists amdgpu in addition to nouveau)<br>5. Not run as root (error)<br>6. No GPU hardware detected (error) | **DONE** |
+
 ## AWS Commands
 
 ### EC2 - Instance Management
@@ -133,13 +172,14 @@ file, deferred to a later phase.
 | `describe-instances` | `--instance-ids`, `--filters` (instance-state-name, instance-id, instance-type, vpc-id, subnet-id, tag:\*, tag-key, tag-value) | `--max-results`, `--next-token`, `--dry-run` | None | Gateway fans out NATS `ec2.DescribeInstances` to all nodes (no queue group) → each daemon returns local instances → gateway aggregates and returns combined list. Filters applied per-node before aggregation (reduces payload). Also applies to stopped/terminated instances via `describeInstancesFromKV()`. | 1. Describe all instances (no filter)<br>2. Describe by instance ID<br>3. Describe with filters (e.g. instance-state-name)<br>4. Instance not found returns empty set<br>5. Multi-node aggregation returns instances from all nodes<br>6. Filter by tag<br>7. Unknown filter returns InvalidParameterValue | **DONE** |
 | `start-instances` | `--instance-ids` | `--dry-run`, `--force` | `run-instances` (instance must exist in stopped state) | Gateway sends NATS `ec2.cmd.{instance-id}` → daemon restarts stopped QEMU process with same config → state transitions stopped→pending→running | 1. Start a stopped instance<br>2. Start already-running instance (error: IncorrectInstanceState)<br>3. Start with invalid instance ID<br>4. Verify volumes re-mount on start | **DONE** |
 | `stop-instances` | `--instance-ids` | `--force`, `--hibernate`, `--dry-run` | `run-instances` (instance must be running) | Gateway sends NATS to target node → daemon issues QMP `system_powerdown` for graceful shutdown → monitors heartbeat until QEMU exits → state transitions running→stopping→stopped | 1. Graceful stop of running instance<br>2. Force stop (kills QEMU process)<br>3. Stop already-stopped instance (error)<br>4. Verify ~30s heartbeat detection | **DONE** |
-| `terminate-instances` | `--instance-ids`, `DeleteOnTermination` (per-volume flag, default true) | `--dry-run` | `run-instances` (instance must exist) | Gateway sends NATS to target node → daemon kills QEMU process → cleans up NBD mounts → deletes volumes with `DeleteOnTermination=true` via `volumeService.DeleteVolume()` (S3 cleanup of vol/, vol-efi/, vol-cloudinit/) → internal volumes (EFI, cloud-init) always cleaned up via `ebs.delete` NATS → volumes with `DeleteOnTermination=false` left in available state → state→terminated | 1. Terminate running instance<br>2. Terminate stopped instance<br>3. Terminate with DeleteOnTermination=true deletes volumes<br>4. Terminate with DeleteOnTermination=false preserves volumes<br>5. Terminate already-terminated (idempotent)<br>6. Internal volumes (EFI, cloud-init) always cleaned up<br>7. Invalid instance ID | **DONE** |
+| `terminate-instances` | `--instance-ids`, `DeleteOnTermination` (per-volume flag, default true) | `--dry-run` | `run-instances` (instance must exist) | Gateway sends NATS to target node → daemon checks `DisableApiTermination` first and rejects with `OperationNotPermitted` if set (running or stopped, no side effects) → kills QEMU process → cleans up NBD mounts → deletes volumes with `DeleteOnTermination=true` via `volumeService.DeleteVolume()` (S3 cleanup of vol/, vol-efi/, vol-cloudinit/) → internal volumes (EFI, cloud-init) always cleaned up via `ebs.delete` NATS → volumes with `DeleteOnTermination=false` left in available state → state→terminated | 1. Terminate running instance<br>2. Terminate stopped instance<br>3. Terminate with DeleteOnTermination=true deletes volumes<br>4. Terminate with DeleteOnTermination=false preserves volumes<br>5. Terminate already-terminated (idempotent)<br>6. Internal volumes (EFI, cloud-init) always cleaned up<br>7. Invalid instance ID<br>8. Running protected instance rejected (OperationNotPermitted, state stays running)<br>9. Stopped protected instance rejected (resources intact) | **DONE** |
 | `reboot-instances` | `--instance-ids` | `--dry-run` | `run-instances` (instance must be running) | Gateway validates instance IDs → sends EC2InstanceCommand with `RebootInstance=true` via NATS `ec2.cmd.{instanceId}` → daemon validates instance is in StateRunning (returns IncorrectInstanceState if stopped) → issues QMP `system_reset` → instance reboots without stopping, stays in running state | 1. Reboot running instance<br>2. Reboot multiple instances<br>3. Reboot stopped instance (error: IncorrectInstanceState)<br>4. Instance not found (error: InvalidInstanceID.NotFound)<br>5. Verify instance stays in running state after reboot | **DONE** |
 | `describe-instance-types` | `--filters` (capacity filter only) | `--instance-types`, `--max-results`, `--next-token`, `--dry-run`, all other filters | None | Gateway fans out NATS `ec2.DescribeInstanceTypes` to all nodes → each daemon reports supported types (t3.micro/small/medium/large) with vCPU/memory specs → gateway deduplicates and returns | 1. List all instance types<br>2. Filter by specific type<br>3. Filter with `capacity=true` shows available slots<br>4. Verify vCPU/memory specs match hardware | **DONE** |
-| `modify-instance-attribute` | `--instance-id`, `--instance-type`, `--user-data` | `--disable-api-termination`, `--ebs-optimized`, `--source-dest-check`, `--instance-initiated-shutdown-behavior`, `--block-device-mappings`, `--groups`, `--ena-support`, `--sriov-net-support` | Instance must be stopped (in NATS KV) | Gateway validates input (exactly one attribute per call, instance ID format) → NATS `ec2.ModifyInstanceAttribute` with `spinifex-workers` queue group → daemon loads stopped instance from JetStream KV → applies attribute change → writes back to KV → returns `{}` on success. **InstanceType**: updates vm.InstanceType, Config, and Instance fields; clears StateReason (enables recovery from instance-type-missing bug). **UserData**: stores decoded content in vm.UserData and re-encodes to base64 for RunInstancesInput (cloud-init on next start). No instance type pre-validation (matches AWS — invalid types accepted, fail at StartInstances time). | 1. Change instance type while stopped<br>2. Change user data while stopped<br>3. Modify running instance (error: NotFound — running instances not in KV)<br>4. Instance not found (error: InvalidInstanceID.NotFound)<br>5. Instance not stopped (error: IncorrectInstanceState)<br>6. Invalid instance type accepted (fails on start with InsufficientInstanceCapacity)<br>7. StateReason cleared on type change (recovery from capacity-unavailable)<br>8. Missing/malformed instance ID (error: InvalidInstanceID.Malformed)<br>9. No attribute set (error: InvalidParameterValue)<br>10. Multiple attributes in one call (error: InvalidParameterValue) | **DONE** |
+| `modify-instance-attribute` | `--instance-id`, `--instance-type`, `--user-data`, `--disable-api-termination` | `--ebs-optimized`, `--source-dest-check`, `--instance-initiated-shutdown-behavior`, `--block-device-mappings`, `--groups`, `--ena-support`, `--sriov-net-support` | Instance must exist; `instance-type` and `user-data` require stopped state, `disable-api-termination` works on running or stopped | Gateway validates input (exactly one attribute per call, instance ID format) → NATS `ec2.ModifyInstanceAttribute` with `spinifex-workers` queue group → daemon dispatches by state: stopped instances loaded from JetStream KV, running instances updated via `vm.Manager.UpdateAndPersist` (atomic flag check + persist under one lock to prevent Modify/Terminate race) → returns `{}` on success. **InstanceType**: updates vm.InstanceType, Config, and Instance fields; clears StateReason (enables recovery from instance-type-missing bug). **UserData**: stores decoded content in vm.UserData and re-encodes to base64 for RunInstancesInput (cloud-init on next start). **DisableApiTermination**: stamps `RunInstancesInput.DisableApiTermination`; flag survives stop/start and daemon restarts. No instance type pre-validation (matches AWS — invalid types accepted, fail at StartInstances time). | 1. Change instance type while stopped<br>2. Change user data while stopped<br>3. Toggle disable-api-termination on running instance<br>4. Toggle disable-api-termination on stopped instance<br>5. Modify instance-type on running instance (error: NotFound — running instances not in KV)<br>6. Instance not found (error: InvalidInstanceID.NotFound)<br>7. Instance not stopped for type/user-data change (error: IncorrectInstanceState)<br>8. Invalid instance type accepted (fails on start with InsufficientInstanceCapacity)<br>9. StateReason cleared on type change (recovery from capacity-unavailable)<br>10. Missing/malformed instance ID (error: InvalidInstanceID.Malformed)<br>11. No attribute set (error: InvalidParameterValue)<br>12. Multiple attributes in one call (error: InvalidParameterValue) | **DONE** |
 | `get-console-output` | `--instance-id` | `--latest` (always returns latest), `--dry-run` | Instance must be running on a node | Gateway sends NATS `ec2.{instanceId}.GetConsoleOutput` (per-instance topic, routed to owning node) → daemon reads console log file from disk → returns last 64KB base64-encoded with timestamp. Always available regardless of serial console access setting (matches AWS behavior). | 1. Get output from running instance<br>2. Empty log file returns empty output<br>3. Instance not found (error: InvalidInstanceID.NotFound) | **DONE** |
-| `describe-instance-attribute` | `--instance-id`, `--attribute` (instanceType, userData, instanceInitiatedShutdownBehavior, disableApiTermination, disableApiStop, ebsOptimized, enaSupport, sourceDestCheck, rootDeviceName, kernel, ramdisk) | `--dry-run` | Instance must exist (running or stopped) | Gateway validates input → NATS `ec2.DescribeInstanceAttribute` with `spinifex-workers` queue group → daemon checks running instances first (`d.Instances.VMS`), then stopped instances in JetStream KV → returns single attribute per call (matches AWS behavior). Stored attributes (`instanceType`, `userData`) return real values; unstored attributes return AWS defaults (`instanceInitiatedShutdownBehavior`=stop, `disableApiTermination`=false, etc.) | 1. Get instanceType from running instance<br>2. Get userData from stopped instance<br>3. Get default disableApiTermination<br>4. Invalid attribute name (error)<br>5. Instance not found (error: InvalidInstanceID.NotFound) | **DONE** |
+| `describe-instance-attribute` | `--instance-id`, `--attribute` (instanceType, userData, instanceInitiatedShutdownBehavior, disableApiTermination, disableApiStop, ebsOptimized, enaSupport, sourceDestCheck, rootDeviceName, kernel, ramdisk) | `--dry-run` | Instance must exist (running or stopped) | Gateway validates input → NATS `ec2.DescribeInstanceAttribute` with `spinifex-workers` queue group → daemon checks running instances first (`d.Instances.VMS`), then stopped instances in JetStream KV → returns single attribute per call (matches AWS behavior). Stored attributes (`instanceType`, `userData`, `disableApiTermination`) return real values from `RunInstancesInput`; remaining attributes return AWS defaults (`instanceInitiatedShutdownBehavior`=stop, etc.) | 1. Get instanceType from running instance<br>2. Get userData from stopped instance<br>3. Get disableApiTermination (reflects RunInstances flag + Modify updates)<br>4. Invalid attribute name (error)<br>5. Instance not found (error: InvalidInstanceID.NotFound) | **DONE** |
 | `describe-instance-credit-specifications` | `--instance-ids` | `--filters`, `--max-results`, `--dry-run` | None | Gateway-only stub — returns `CpuCredits: "standard"` for each requested instance ID. No daemon round-trip. T-series credit mode is not persisted. | 1. Get credit spec for T-series instance<br>2. Multiple instance IDs | **DONE** |
+| `describe-instance-status` | `--instance-ids`, `--include-all-instances`, `--filters` (availability-zone, instance-state-code, instance-state-name, tag:\*) | `--max-results`, `--next-token`, `--dry-run` (silently ignored), `event.*` / `instance-status.*` / `system-status.*` filters (rejected with InvalidParameterValue) | None | Gateway fans out NATS `ec2.DescribeInstanceStatus` to all nodes (no queue group). Each daemon validates input (i- prefix, filter names), iterates local VMs and renders InstanceStatus: `Running`→ok/passed, everything-else-included→not-applicable. When `IncludeAllInstances=true`, gateway also queries `ec2.DescribeStoppedInstances` in parallel and transforms reservation entries into InstanceStatus with `not-applicable`. Aggregated results dedup'd by InstanceId (running > stopped). Terminated and error-state instances never returned. No real health probing — static values reflect Mulga's lifecycle state only. | 1. Status of running instance (ok/passed)<br>2. Filter by instance ID<br>3. Default excludes stopped instances<br>4. IncludeAllInstances=true includes stopped (not-applicable)<br>5. Cross-account instance excluded<br>6. Unknown filter returns InvalidParameterValue<br>7. Malformed instance ID rejected<br>8. Tag filter narrows correctly | **DONE** |
 | `monitor-instances` | — | `--instance-ids` | Instance must exist | Enable basic monitoring (CPU, disk, network) → store metrics in NATS KV → return monitoring state | 1. Enable monitoring<br>2. Verify monitoring state in describe-instances | **NOT STARTED** |
 | `unmonitor-instances` | — | `--instance-ids` | Instance must exist | Disable detailed monitoring → revert to basic monitoring → return monitoring state | 1. Disable monitoring<br>2. Verify monitoring state reverts in describe-instances | **NOT STARTED** |
 
@@ -158,7 +198,7 @@ file, deferred to a later phase.
 |---------|-------------------|---------------|---------------|-------------|------------|--------|
 | `describe-images` | `--image-ids` (format validation only), `--owners` (self, account ID, alias), `--filters` (name, state, architecture, image-id, is-public, owner-id, description, image-type, tag:\*) | `--executable-users`, `--include-deprecated`, `--include-disabled`, `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeImages` → daemon reads AMI metadata from Predastore S3 buckets (ami-*) → filters by ImageIds, Owners, and Filters → returns image list with state, architecture, block device mappings | 1. List all images<br>2. Filter by image ID<br>3. Filter by owner (self/amazon)<br>4. Non-existent AMI returns empty<br>5. Verify metadata fields (architecture, state, rootDeviceName)<br>6. Filter by name wildcard<br>7. Unknown filter returns InvalidParameterValue | **DONE** |
 | `create-image` | `--instance-id`, `--name`, `--description`, `--tag-specifications` | `--no-reboot`, `--block-device-mappings`, `--dry-run` | Instance must exist (running or stopped) | Gateway validates input → NATS `ec2.{instanceId}.CreateImage` (per-instance topic) → daemon extracts root volume → snapshots via `ebs.snapshot` NATS (running) or offline S3 copy (stopped) → creates AMI metadata in S3 (`{amiId}/config.json`) → stores tags → returns ami-ID. Duplicate AMI name validation enforced. | 1. Create image from running instance<br>2. Create image from stopped instance<br>3. Invalid instance ID (error)<br>4. Duplicate AMI name (error)<br>5. Verify new AMI appears in describe-images<br>6. Launch new instance from created AMI | **DONE** |
-| `register-image` | `--name`, `--description`, `--architecture` (x86_64/arm64/i386), `--root-device-name`, `--virtualization-type` (hvm only), `--block-device-mappings` (root with `Ebs.SnapshotId`+optional `VolumeSize`), `--tag-specifications` | `--billing-products`, `--uefi-data` | Backing snapshot must exist in Predastore (`{snapshotId}/metadata.json`); caller must own snapshot or it must be system-owned | Gateway validates name length (3–128), `snap-` prefix, architecture/virtualization values → NATS `ec2.RegisterImage` → daemon checks AMI name uniqueness, reads snapshot metadata, verifies snapshot ownership, builds `viperblock.AMIMetadata` (defaults: `Architecture=x86_64`, `Virtualization=hvm`, `PlatformDetails=Linux/UNIX`, `RootDeviceType=ebs`), writes `{amiId}/config.json`. Pointer-only — never touches block data. `BootMode`, `KernelId`, `RamdiskId`, `TpmSupport`, `ImdsSupport`, `EnaSupport`, `SriovNetSupport`, `ImageLocation`, `paravirtual` virtualization rejected with `InvalidParameterValue`. `VolumeSize` smaller than snapshot rejected. | 1. Register with valid snapshot<br>2. Missing name/snapshot (error)<br>3. Duplicate AMI name (`InvalidAMIName.Duplicate`)<br>4. Snapshot not found (`InvalidSnapshot.NotFound`)<br>5. Cross-account snapshot (`UnauthorizedOperation`)<br>6. Tags from `TagSpecifications` persisted<br>7. Verify registered image in describe-images | **DONE** |
+| `register-image` | `--name`, `--description`, `--architecture` (x86_64/arm64/i386), `--root-device-name`, `--virtualization-type` (hvm only), `--boot-mode` (bios/uefi/uefi-preferred), `--block-device-mappings` (root with `Ebs.SnapshotId`+optional `VolumeSize`), `--tag-specifications` | `--billing-products`, `--uefi-data` | Backing snapshot must exist in Predastore (`{snapshotId}/metadata.json`); caller must own snapshot or it must be system-owned | Gateway validates name length (3–128), `snap-` prefix, architecture/virtualization/boot-mode values → NATS `ec2.RegisterImage` → daemon checks AMI name uniqueness, reads snapshot metadata, verifies snapshot ownership, builds `viperblock.AMIMetadata` (defaults: `Architecture=x86_64`, `Virtualization=hvm`, `PlatformDetails=Linux/UNIX`, `RootDeviceType=ebs`), writes `{amiId}/config.json`. Pointer-only — never touches block data. `BootMode` outside `bios|uefi|uefi-preferred`, `KernelId`, `RamdiskId`, `TpmSupport`, `ImdsSupport`, `EnaSupport`, `SriovNetSupport`, `ImageLocation`, `paravirtual` virtualization rejected with `InvalidParameterValue`. `uefi-preferred` is treated as `uefi` at launch. `VolumeSize` smaller than snapshot rejected. | 1. Register with valid snapshot<br>2. Missing name/snapshot (error)<br>3. Duplicate AMI name (`InvalidAMIName.Duplicate`)<br>4. Snapshot not found (`InvalidSnapshot.NotFound`)<br>5. Cross-account snapshot (`UnauthorizedOperation`)<br>6. Tags from `TagSpecifications` persisted<br>7. Verify registered image in describe-images<br>8. `--boot-mode=uefi` accepted, returned by `describe-images`<br>9. `--boot-mode=invalid` rejected (`InvalidParameterValue`) | **DONE** |
 | `deregister-image` | `--image-id` | `--dry-run` | AMI must exist; caller must own it (system AMIs immutable via this API) | Gateway validates `ami-` prefix → NATS `ec2.DeregisterImage` → daemon hard-deletes `{amiId}/config.json` from Predastore. Backing snapshot is left intact (matches AWS — operators run `delete-snapshot` separately to reclaim block storage). Cross-account/system-AMI mutations rejected with `UnauthorizedOperation`. Re-deregister returns `InvalidAMIID.NotFound` (no tombstone). | 1. Deregister existing AMI<br>2. Deregister non-existent AMI (`InvalidAMIID.NotFound`)<br>3. Re-deregister already-deleted AMI (`InvalidAMIID.NotFound`)<br>4. Cross-account AMI (`UnauthorizedOperation`)<br>5. System AMI (`UnauthorizedOperation`)<br>6. Verify deregistered AMI not in describe-images<br>7. Backing snapshot untouched | **DONE** |
 | `copy-image` | `--source-image-id`, `--source-region` (must equal gateway region), `--name`, `--description`, `--client-token` (accepted, not honoured), `--copy-image-tags`, `--tag-specifications` (image only) | `--encrypted`, `--kms-key-id`, `--destination-outpost-arn`, `--dry-run` | Source AMI must exist and be owned by caller OR be a system AMI; backing snapshot must still exist | Gateway validates `ami-` prefix, name length (3–128), same-region — cross-region / `Encrypted` / `KmsKeyId` / `DestinationOutpostArn` rejected with `InvalidParameterValue`. NATS `ec2.CopyImage` → daemon checks name uniqueness, reads source `ami-xxx/config.json`, visibility-filters cross-account sources as `InvalidAMIID.NotFound`, reads source `{snap-xxx}/metadata.json` (missing or empty SnapshotID → `InvalidAMIID.NotFound`), writes new `snap-yyy/metadata.json` that inherits source `VolumeID` (no block copy), writes new `ami-yyy/config.json` owned by caller. `Description` inherits from source when unset. Tag merge: `CopyImageTags=true` seeds tags from source, explicit image-resource `TagSpecifications` override colliding keys and append new ones; non-image tag specs ignored. | 1. Same-region copy produces new `ami-`/`snap-` pair with distinct IDs<br>2. New snap shares source `VolumeID` (metadata-only)<br>3. New AMI owned by caller, source untouched<br>4. System AMI copied into caller's account<br>5. Cross-account source (`InvalidAMIID.NotFound`)<br>6. Missing source AMI (`InvalidAMIID.NotFound`)<br>7. Orphaned source (missing/empty `SnapshotID`) → `InvalidAMIID.NotFound`<br>8. Duplicate name (`InvalidAMIName.Duplicate`)<br>9. Cross-region / `Encrypted` / `KmsKeyId` rejected (`InvalidParameterValue`)<br>10. `CopyImageTags` true/false tag inheritance semantics | **DONE** |
 | `import-image` | — | `--disk-containers` (Format, Url/S3Bucket+S3Key), `--description`, `--architecture`, `--platform` | S3 bucket with disk image | Download disk image from S3 → convert format (VMDK/VHD/RAW→QCOW2) → create viperblock volume → register as AMI | 1. Import QCOW2 image<br>2. Import RAW image<br>3. Invalid format (error)<br>4. Verify imported image launchable | **NOT STARTED** |
@@ -336,6 +376,7 @@ Requires OVN/OVS (`apt install openvswitch-switch ovn-host`). VPC/Subnet/ENI/SG
 | `authorize-security-group-egress` | `--group-id`, `--ip-permissions` | `--dry-run` | SG must exist | Gateway validates GroupId → NATS `ec2.AuthorizeSecurityGroupEgress` → daemon retrieves SG → appends rules to EgressRules → updates KV with optimistic locking → publishes `vpc.update-sg` event → returns true | 1. Allow HTTPS outbound<br>2. Restrict to specific CIDR<br>3. Missing GroupId (MissingParameter)<br>4. Non-existent SG (InvalidGroup.NotFound) | **DONE** |
 | `revoke-security-group-ingress` | `--group-id`, `--ip-permissions` (matching rules to remove) | `--dry-run` | SG must exist | Gateway validates GroupId → NATS `ec2.RevokeSecurityGroupIngress` → daemon retrieves SG → removes matching rules from IngressRules (set difference) → updates KV with optimistic locking → publishes `vpc.update-sg` event → returns true | 1. Revoke existing rule<br>2. Missing GroupId (MissingParameter)<br>3. Non-existent SG (InvalidGroup.NotFound) | **DONE** |
 | `revoke-security-group-egress` | `--group-id`, `--ip-permissions` (matching rules to remove) | `--dry-run` | SG must exist | Gateway validates GroupId → NATS `ec2.RevokeSecurityGroupEgress` → daemon retrieves SG → removes matching rules from EgressRules (set difference) → updates KV with optimistic locking → publishes `vpc.update-sg` event → returns true | 1. Revoke existing rule<br>2. Missing GroupId (MissingParameter)<br>3. Non-existent SG (InvalidGroup.NotFound) | **DONE** |
+| `describe-security-group-rules` | `--filters` (group-id, security-group-rule-id, tag:\*, tag-key), `--security-group-rule-ids` | `--max-results`, `--next-token`, `--dry-run` | None | Gateway validates `sgr-` format on SecurityGroupRuleIds (nil input = all rules in account) → NATS `ec2.DescribeSecurityGroupRules` → daemon iterates SG records in `spinifex-vpc-security-groups` KV under account → applies filters → emits one `SecurityGroupRule` per ingress (IsEgress=false) and egress (IsEgress=true) rule, each with stable `sgr-` ID, GroupOwnerId=accountID, ReferencedGroupInfo for SG-source rules → returns `InvalidSecurityGroupRuleId.NotFound` if any requested ID missing → v1→v2 KV migration backfills `sgr-` IDs on existing rules at daemon startup; MaxResults/NextToken accepted but ignored | 1. List all rules in account<br>2. Filter by group-id<br>3. Filter by security-group-rule-id<br>4. Non-existent rule ID (InvalidSecurityGroupRuleId.NotFound)<br>5. Malformed sgr- ID at gateway (InvalidParameterValue)<br>6. Tag filters match zero (Tags not yet populated on rules)<br>7. Unknown filter (InvalidParameterValue)<br>8. Migration backfills sgr- IDs on pre-v2 records | **DONE** |
 | `create-internet-gateway` | `--tag-specifications` | `--dry-run` | None | NATS `ec2.CreateInternetGateway` → daemon generates igw-ID → creates IGWRecord in `spinifex-igw` KV bucket with state=available (not attached) → returns InternetGateway with empty Attachments | 1. Create IGW<br>2. Verify in describe-internet-gateways<br>3. Tags applied at creation | **DONE** |
 | `attach-internet-gateway` | `--internet-gateway-id`, `--vpc-id` | `--dry-run` | IGW and VPC must exist | NATS `ec2.AttachInternetGateway` → daemon validates IGW exists and is not already attached (ResourceAlreadyAssociated) → sets VpcId on record → state=attached → publishes `vpc.igw-attach` event to vpcd (creates OVN external switch, gateway port, SNAT rules) | 1. Attach IGW to VPC<br>2. Attach already-attached IGW (ResourceAlreadyAssociated)<br>3. Non-existent IGW (error)<br>4. Missing params (MissingParameter) | **DONE** |
 | `detach-internet-gateway` | `--internet-gateway-id`, `--vpc-id` | `--dry-run` | IGW must be attached to specified VPC | NATS `ec2.DetachInternetGateway` → daemon validates IGW exists and is attached to specified VPC (GatewayNotAttached if wrong VPC) → clears VpcId → state=available → publishes `vpc.igw-detach` event to vpcd (removes OVN resources) | 1. Detach attached IGW<br>2. Detach from wrong VPC (GatewayNotAttached)<br>3. Detach unattached IGW (error) | **DONE** |

docs/DESIGN.md

@@ -16,14 +16,14 @@ Users interact with Spinifex using standard AWS SDKs or the AWS CLI by using the
 
 ```bash
 AWS_PROFILE=spinifex aws ec2 run-instances \
-    --image-id ami-debian12 \
+    --image-id ami-debian13 \
     --instance-type t3.micro \
     --key-name my-keypair
 ```
 
 The AWS SDK formats this as an HTTPS POST with:
 - AWS SigV4 authentication headers
-- EC2 Query Protocol body (`Action=RunInstances&ImageId=ami-debian12&...`)
+- EC2 Query Protocol body (`Action=RunInstances&ImageId=ami-debian13&...`)
 
 ### 2. AWS Gateway
 
@@ -84,6 +84,13 @@ The EC2 handler (`spinifex/gateway/ec2.go`) parses the `Action` parameter and de
 
 ### 4. NATS Messaging
 
+The full service-to-subject map — which service subscribes to what, what
+each one publishes downstream, what each depends on — is captured in
+[`service-interfaces.yaml`](./service-interfaces.yaml). It is the source
+of truth for targeted e2e suite selection (see
+`docs/development/improvements/e2e-targeted-suite-selection.md`) and is
+validated by `make manifest-check`.
+
 The gateway communicates with daemons via NATS request/response. Most calls go through `utils.NATSRequest`, which marshals the input, attaches the account ID as a NATS header, and unmarshals the typed response:
 
 ```go
@@ -297,7 +304,7 @@ sudo systemctl start spinifex.target
 export AWS_PROFILE=spinifex
 
 # Run instance
-aws ec2 run-instances --image-id ami-debian12 --instance-type t3.micro
+aws ec2 run-instances --image-id ami-debian13 --instance-type t3.micro
 
 # List instances (queries all nodes)
 aws ec2 describe-instances

docs/compute/gpu-passthrough/README.md

@@ -0,0 +1,223 @@
+---
+title: "GPU Passthrough"
+description: "Configure VFIO GPU passthrough on a Spinifex node to expose NVIDIA or AMD GPUs to EC2 instances."
+category: "Compute"
+tags:
+  - gpu
+  - passthrough
+  - vfio
+  - ec2
+resources:
+  - title: "Spinifex Repository"
+    url: "https://github.com/mulgadc/spinifex"
+  - title: "AWS EC2 Accelerated Computing Instance Types"
+    url: "https://docs.aws.amazon.com/ec2/latest/instancetypes/ac.html"
+---
+
+# GPU Passthrough
+
+> Configure VFIO GPU passthrough on a Spinifex node to expose NVIDIA or AMD GPUs to EC2 instances.
+
+> **Note:** GPU passthrough is supported on x86_64 hosts only.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Prerequisites](#prerequisites)
+- [Host Setup](#host-setup)
+- [Enable Passthrough](#enable-passthrough)
+- [Status and Monitoring](#status-and-monitoring)
+- [GPU Instance Types](#gpu-instance-types)
+- [Launching GPU Instances](#launching-gpu-instances)
+- [Disable Passthrough](#disable-passthrough)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Overview
+
+Spinifex supports VFIO-based GPU passthrough, binding NVIDIA and AMD GPUs directly to guest VMs via the `vfio-pci` kernel driver. Once configured, the node exposes GPU-enabled EC2 instance types and the GPU is allocated exclusively to individual instances.
+
+At startup, Spinifex always probes for GPU hardware and surfaces the result in the node banner and in `spx top nodes` — regardless of whether passthrough is enabled. This lets operators verify hardware before activating the feature.
+
+The passthrough state is reflected in the login banner:
+
+![GPU not yet enabled](../../../.github/assets/images/gpu-banner1.png)
+
+---
+
+## Prerequisites
+
+- IOMMU must be supported and enabled in the host BIOS/UEFI (`Intel VT-d` or `AMD-Vi`).
+- An NVIDIA or AMD GPU must be physically installed.
+- The Spinifex node must be running on bare metal (not inside a VM).
+
+---
+
+## Host Setup
+
+Run the one-time host configuration as root. This command is idempotent — safe to re-run after a reboot.
+
+```bash
+sudo spx admin gpu setup
+```
+
+What it does:
+
+1. Detects installed GPUs and their PCI IDs.
+2. Enables IOMMU in GRUB (`intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt`).
+3. Writes the vfio udev rule (`/etc/udev/rules.d/99-spinifex-vfio.rules`).
+4. Blacklists `nouveau` (NVIDIA) and `amdgpu` (AMD) so the host kernel does not claim the device.
+5. Configures `vfio-pci` early binding in `/etc/modprobe.d/vfio-pci.conf`.
+6. Adds vfio modules to initramfs.
+
+If any change requires a reboot, the command prints instructions and exits:
+
+```
+Setup complete — reboot required.
+  sudo reboot
+  Then run: sudo spx admin gpu enable
+```
+
+---
+
+## Enable Passthrough
+
+After setup and reboot, enable GPU passthrough:
+
+```bash
+sudo spx admin gpu enable
+```
+
+The command writes the configuration, signals the daemon (`SIGHUP`), and waits up to 30 seconds for the daemon to confirm the new state. On success it prints the current GPU status.
+
+
+![GPU enabled](../../../.github/assets/images/gpu-enabled.png)
+
+---
+
+## Status and Monitoring
+
+### Per-node status
+
+```bash
+spx admin gpu status
+# or for a specific node in the cluster:
+spx admin gpu status --node <node-name>
+```
+
+Output includes hardware detected, IOMMU state, vfio-pci binding, passthrough enabled/disabled, GPU pool allocation (`allocated/total`), and the GPU-capable EC2 instance types available on that node.
+
+### Cluster view
+
+`spx top nodes` includes a `GPU (used/total)` column:
+
+| Value | Meaning |
+|-------|---------|
+| `1/2` | Passthrough active — 1 of 2 GPUs allocated |
+| `0/1*` | Node has a GPU but passthrough is not enabled |
+| `-` | No GPU hardware detected |
+
+```bash
+spx top nodes
+```
+
+---
+
+## GPU Instance Types
+
+GPU instance types are derived from detected hardware.
+
+To list available GPU instance types on the current node:
+
+```bash
+aws ec2 describe-instance-types \
+  --filters Name=instance-type,Values=g*
+```
+
+---
+
+## Launching GPU Instances
+
+Import the GPU-enabled AMI before launching:
+
+```bash
+spx admin images import --name ubuntu-gpu-nvidia   # NVIDIA hosts
+spx admin images import --name ubuntu-gpu-amd      # AMD hosts
+```
+
+Launch a GPU instance:
+
+```bash
+aws ec2 run-instances \
+  --image-id $GPU_AMI \
+  --instance-type $INSTANCE_TYPE \
+  --key-name spinifex-key
+```
+
+To verify the GPU is visible from inside the instance, SSH in and run:
+
+```bash
+# NVIDIA
+nvidia-smi
+
+# AMD
+rocm-smi
+```
+
+---
+
+## Disable Passthrough
+
+Passthrough cannot be disabled while GPU instances are running. Terminate all GPU instances first, then:
+
+```bash
+sudo spx admin gpu disable
+```
+
+The command signals the daemon and waits for confirmation. The GPU is released back to the host kernel on the next reboot (the vfio-pci binding persists until `setup` is re-run without the blacklists).
+
+---
+
+## Troubleshooting
+
+### IOMMU not active after reboot
+
+Verify IOMMU is enabled in BIOS/UEFI (`Intel VT-d` or `AMD-Vi`). Then check GRUB picked up the parameters:
+
+```bash
+cat /proc/cmdline | grep iommu
+ls /sys/kernel/iommu_groups/
+```
+
+If `/sys/kernel/iommu_groups/` is empty, IOMMU is not active. Re-run `sudo spx admin gpu setup` after enabling it in firmware.
+
+### GPU bound to native driver instead of vfio-pci
+
+```bash
+lspci -k | grep -A3 -i "vga\|3d\|display"
+```
+
+If the driver is `amdgpu` or `nvidia` rather than `vfio-pci`, the blacklist or early binding did not take effect. Re-run setup and reboot:
+
+```bash
+sudo spx admin gpu setup
+sudo reboot
+```
+
+### `spx admin gpu enable` fails with "prerequisites not met"
+
+Run `setup` first to configure the host, then retry `enable`:
+
+```bash
+sudo spx admin gpu setup
+sudo spx admin gpu enable
+```
+
+### Daemon does not confirm within 30 seconds
+
+Check the daemon log for errors:
+
+```bash
+journalctl -u spinifex-daemon -n 50
+```

docs/compute/vpc-networking/README.md

@@ -573,6 +573,54 @@ external_interface = "br-wan"
 DHCP server when VMs launch and releases them on terminate. Requires `dhclient`
 on the host (`apt install isc-dhcp-client`).
 
+## Host-Local Subnet (No Upstream Router)
+
+```
+Network: 192.168.10.0/24 — host-local, reachable from the host only
+Host WAN: 198.51.100.10/24 on br-wan (existing address — unchanged)
+Gateway: 192.168.10.1 — second address added to br-wan
+```
+
+Add the VM pool gateway as a second address on `br-wan` alongside the existing WAN
+IP. The host acts as the gateway for the pool — no upstream router or DHCP server
+needed for this range.
+
+```yaml
+# /etc/netplan/…
+bridges:
+  br-wan:
+    addresses:
+      - 192.168.10.1/24      # VM pool gateway — host-local
+      - 198.51.100.10/24     # existing WAN IP — unchanged
+    routes:
+      - to: default
+        via: 198.51.100.1
+```
+
+```toml
+[network]
+external_mode = "pool"
+
+[[network.external_pools]]
+name        = "wan"
+source      = "static"         # required — no upstream DHCP for this range
+range_start = "192.168.10.2"
+range_end   = "192.168.10.100"
+gateway     = "192.168.10.1"   # second address on br-wan
+prefix_len  = 24
+dns_servers = ["8.8.8.8"]
+```
+
+**Setup:** Apply with `sudo netplan apply`. VMs are reachable from the host at
+`192.168.10.x`. For internet access through the host's WAN interface:
+
+```bash
+sysctl -w net.ipv4.ip_forward=1
+iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -o br-wan -j MASQUERADE
+```
+
+Persist via `/etc/sysctl.d/99-ip-forward.conf` and `netfilter-persistent save`.
+
 ## Datacenter / Colo (ISP Block)
 
 ```

docs/getting-started/install-airgapped/README.md

@@ -57,6 +57,7 @@ sudo apt update
 sudo apt install --download-only -y \
   nbdkit nbdkit-plugin-dev pkg-config \
   qemu-system-x86 qemu-utils qemu-kvm \
+  ovmf qemu-efi-aarch64 \
   libvirt-daemon-system libvirt-clients libvirt-dev \
   ovn-central ovn-host openvswitch-switch \
   dhcpcd-base make gcc jq curl iproute2 netcat-openbsd \
@@ -74,7 +75,7 @@ Mirror the cloud image you intend to run as guest VMs:
 
 ```bash
 mkdir -p images
-curl -fsSL "https://cloud.debian.org/images/cloud/trixie/20260501-2465/debian-13-genericcloud-amd64-20260501-2465.qcow2" \
+curl -fsSL "https://cloud.debian.org/images/cloud/trixie/20260518-2482/debian-13-genericcloud-amd64-20260518-2482.qcow2" \
   -o images/debian-13-amd64.qcow2
 ```
 
@@ -141,7 +142,7 @@ Register the pre-staged image with Spinifex:
 
 ```bash
 sudo spx admin images import --file /mnt/usb/images/debian-13-amd64.qcow2 \
-  --distro debian --version 13 --arch x86_64
+  --distro debian --version 13 --arch x86_64 --boot-mode uefi
 ```
 
 ## Step 9. Verify
@@ -183,4 +184,4 @@ sudo INSTALL_SPINIFEX_TARBALL=/mnt/usb/tarball/spinifex-...tar.gz \
 
 ### Image Import Fails
 
-`spx admin images import --file` requires the distro/version/arch flags so the image registers in the catalogue. If the import succeeds but `aws ec2 describe-images` returns empty, check `journalctl -u spinifex-daemon -f` for predastore upload errors.
+`spx admin images import --file` requires the distro/version/arch/boot-mode flags so the image registers in the catalogue. If the import succeeds but `aws ec2 describe-images` returns empty, check `journalctl -u spinifex-daemon -f` for predastore upload errors.

docs/getting-started/setting-up-your-cluster/README.md

@@ -58,11 +58,10 @@ spx admin images list
 
 ```
 NAME                    | DISTRO | VERSION | ARCH   | BOOT
-debian-13-arm64         | debian | 13      | arm64  | bios
-debian-13-x86_64        | debian | 13      | x86_64 | bios
-lb-alpine-3.21.6-x86_64 | alpine | 3.21.6  | x86_64 | bios
-ubuntu-26.04-arm64      | ubuntu | 26.04   | arm64  | bios
-ubuntu-26.04-x86_64     | ubuntu | 26.04   | x86_64 | bios
+debian-13-arm64         | debian | 13      | arm64  | uefi
+debian-13-x86_64        | debian | 13      | x86_64 | uefi
+ubuntu-26.04-arm64      | ubuntu | 26.04   | arm64  | uefi
+ubuntu-26.04-x86_64     | ubuntu | 26.04   | x86_64 | uefi
 ```
 
 Import an image:
@@ -74,7 +73,7 @@ spx admin images import --name ubuntu-26.04-x86_64
 Or import a local image file:
 
 ```bash
-spx admin images import --file ~/images/ubuntu-26.04.img --distro ubuntu --version 26.04 --arch x86_64
+spx admin images import --file ~/images/ubuntu-26.04.img --distro ubuntu --version 26.04 --arch x86_64 --boot-mode uefi
 ```
 
 Verify the import and note the AMI ID:

docs/gpu-passthrough/GPU-PASSTHROUGH.md

@@ -1,47 +0,0 @@
-# Configuring GPU Passthrough for Spinifex Nodes
-
-
-Spinifex is designed for bare-metal hardware, edge nodes and data-centre use. Follow this guide to configure your Spinifex node to enable GPU Passthrough to its VM instances if hardware is available.
-
->**Note:** this tutorial is for x86 architecture.
-
-
-
-### Getting Started - GPU Setup
-By default, Spinifex will probe for the presence of a GPU on its machine. After booting, the presence and availability of GPU hardware is printed in the info banner:
-![Installer complete](../../.github/assets/images/gpu-banner1.png)
-
-To enable GPU passthrough, you may first need to run:
-```bash
-sudo spx admin gpu setup
-```
-Note that this command will blacklist the GPU's drivers from the host machine such that they can be used by VM instances Therefore the GPU will **NOT** be available to the host machine until after the VM instance  It will also require rebooting the host machine before the changes take effect.
-
-### Enable Passthrough
-After reboot, the banner message should reflect that GPU passthrough is **ready** but not **enabled**.
-
-![Installer complete](../../.github/assets/images/gpu-banner2.png)
-
-This will require you to run:
-```bash
-sudo spx admin gpu enable
-```
-After running this command, you should see:
-
-![Installer complete](../../.github/assets/images/gpu-enabled.png)
-
-The host machine is now configured to provide GPU passthrough to EC2 instances on the host machine.
-
-Lastly, import the Spinifex GPU enabled AMI: [link]
-
-As a final check, spin up an EC2 instance, SSH into it, and check that the host machine's GPU is visible from the EC2 instance.
-
-### Multiple GPUs
-Spinifex is able to detect multiple GPUs. After running the setup commands, the total number of allocatable GPUs is printed.
-
-### Mapping GPUs to EC2 instances
-By defualt, any consumer grade GPUs are mapped to the g5 EC2 instance class, which is the most basic "Accelerated Computing" Ec2 instance provided by AWS. If using a consumer GPU, running:
-```bash
-aws ec2 describe-instance-types
-```
-will return the various sizes of g5 class. All other AWS "Accelerated Computing" EC2 instance classes map directly to their associated GPU, so the Spinifex host machine will need to have the specified GPU if a specific EC2 instance class is desired.

docs/security/malware-protection/README.md

@@ -73,7 +73,7 @@ All Spinifex nodes must include the following in their malware protection scope.
 |------|----------|
 | `/usr/local/bin/spx` | Spinifex CLI and service entry point. All `spinifex-*.service` units invoke `spx service <name> start`, so every service binary is reached through this one path. |
 | `/usr/bin/ovn-*`, `/usr/bin/ovs-*` | OVN / OVS control-plane and datapath binaries (installed via `apt` from the `ovn-central`, `ovn-host`, `openvswitch-switch` packages). |
-| `/usr/lib/<arch>-linux-gnu/nbdkit/plugins/nbdkit-viperblock-plugin.so` | Viperblock NBD plugin loaded by `nbdkit` to expose EBS volumes as block devices. The exact path comes from `nbdkit --dump-config` (`plugindir=`); on Debian 12 amd64 it is `/usr/lib/x86_64-linux-gnu/nbdkit/plugins/`. |
+| `/usr/lib/<arch>-linux-gnu/nbdkit/plugins/nbdkit-viperblock-plugin.so` | Viperblock NBD plugin loaded by `nbdkit` to expose EBS volumes as block devices. The exact path comes from `nbdkit --dump-config` (`plugindir=`); on Debian 13 amd64 it is `/usr/lib/x86_64-linux-gnu/nbdkit/plugins/`. |
 | `/usr/local/share/ca-certificates/spinifex-ca.crt` | Cluster CA cert installed into the system trust store |
 
 ### 1.2 Configuration and Secrets
@@ -157,7 +157,7 @@ All scan runs — scheduled and real-time — must produce a log entry that incl
 
 ## 4. Reference Configurations
 
-The configurations below are reference baselines. They are known to meet the CMMC L1 objectives on a standard Debian 12 Spinifex node; they are not the only valid implementation. Operators running a commercial EDR should treat [§1](#1-designated-locations-sil1-3142-a) and [§3](#3-scan-schedule-sil1-3145) as the contract and configure their chosen product to match.
+The configurations below are reference baselines. They are known to meet the CMMC L1 objectives on a standard Debian 13 Spinifex node; they are not the only valid implementation. Operators running a commercial EDR should treat [§1](#1-designated-locations-sil1-3142-a) and [§3](#3-scan-schedule-sil1-3145) as the contract and configure their chosen product to match.
 
 ### 4.1 ClamAV (signature-based AV)
 

docs/security/media-sanitization/README.md

@@ -0,0 +1,258 @@
+---
+title: "Media Sanitization and Disposal"
+description: "Operator guide for sanitizing and disposing of storage media used by Spinifex nodes"
+category: "Security"
+sections:
+  - overview
+tags:
+  - security
+  - compliance
+  - cmmc
+  - media
+  - sanitization
+  - disposal
+  - decommissioning
+resources:
+  - title: "NIST SP 800-88 Rev 1 — Guidelines for Media Sanitization"
+    url: "https://csrc.nist.gov/pubs/sp/800/88/r1/final"
+  - title: "NIST SP 800-171 Rev 3"
+    url: "https://csrc.nist.gov/pubs/sp/800/171/r3/final"
+  - title: "CMMC Level 1 Self-Assessment Guide v2.0"
+    url: "https://dodcio.defense.gov/CMMC/Documentation"
+  - title: "ATA/ATAPI Command Set — Secure Erase"
+    url: "https://www.t13.org"
+  - title: "NVM Express — Sanitize Command"
+    url: "https://nvmexpress.org/specifications"
+---
+
+# Media Sanitization and Disposal
+
+> Operator guide for sanitizing and disposing of storage media used by Spinifex nodes
+
+## Table of Contents
+
+- [Overview](#overview)
+- [CMMC Practices Covered](#cmmc-practices-covered)
+- [Approach](#approach)
+- [1. Media in Scope](#1-media-in-scope)
+- [2. Sanitization Method Selection](#2-sanitization-method-selection)
+- [3. Volume-Level Sanitization — Before Reuse (MP.L1-3.8.3 [b])](#3-volume-level-sanitization--before-reuse-mpl1-383-b)
+- [4. Whole-Drive and Node Decommissioning — Before Disposal (MP.L1-3.8.3 [a])](#4-whole-drive-and-node-decommissioning--before-disposal-mpl1-383-a)
+- [5. Key Destruction](#5-key-destruction)
+- [6. Removable and Backup Media](#6-removable-and-backup-media)
+- [7. Evidence and Record Keeping](#7-evidence-and-record-keeping)
+- [8. Operator Checklist](#8-operator-checklist)
+
+---
+
+## Overview
+
+**Audience:** Operators decommissioning, reassigning, returning under warranty, or disposing of hardware that has been used to run Spinifex nodes in environments subject to CMMC Level 1, or any site that requires documented media sanitization.
+
+**Scope:** Every storage medium that has, or may have, held Federal Contract Information (FCI) as part of a Spinifex deployment — node system disks, Viperblock WAL/chunk disks, Predastore object stores, NATS JetStream disks, removable master-key tokens, backup media, and any loose drives that previously occupied a Spinifex role. Guest VM internal media (disks as seen from inside a tenant VM) is the workload owner's responsibility and out of scope.
+
+## CMMC Practices Covered
+
+| Practice | Title | Objective |
+|----------|-------|-----------|
+| MP.L1-3.8.3 | Media Disposal | [a] System media containing FCI is sanitized or destroyed before disposal. [b] System media containing FCI is sanitized before it is released for reuse. |
+
+## Approach
+
+NIST SP 800-88 Rev 1 defines three sanitization categories — **Clear**, **Purge**, and **Destroy** — chosen by the confidentiality of the data and whether the medium will leave the operator's control. Spinifex discharges MP.L1-3.8.3 along two tracks:
+
+1. **Volume-level sanitization** for EBS volumes and VM disks returning to the free pool. Handled in-platform by **cryptographic erase** — the per-volume data encryption key (DEK) is deleted, rendering the ciphertext in Predastore S3 irrecoverable. This is the NIST SP 800-88 **Purge**-level method for encrypted media.
+2. **Whole-drive sanitization** for physical media leaving a Spinifex node — drive replacement, node retirement, warranty return, resale, or destruction. The operator runs the sanitization; Spinifex does not and cannot reach the firmware commands required. This guide prescribes the method per media type and the records to keep.
+
+The key principle: every piece of media that ever held plaintext FCI (or keys that wrapped FCI ciphertext) must be sanitized to at least **Purge** before release for reuse, and to **Purge** or **Destroy** before leaving operator control. When in doubt, Destroy.
+
+## 1. Media in Scope
+
+Any medium in any of these roles on a Spinifex node has held, or may have held, FCI:
+
+| Medium | Typical hardware | Contents |
+|--------|------------------|----------|
+| Node system disk | NVMe/SATA SSD | `/etc/spinifex/master.key`, cluster CA key, per-node TLS keys, service configs, logs, journal, swap. |
+| Viperblock WAL device | NVMe/SATA SSD | Plaintext of in-flight block writes before chunking. High-churn, short residency, but FCI lands here. |
+| Viperblock chunk cache / local backing | NVMe/SATA SSD or HDD | Encrypted volume chunks. |
+| Predastore object store | NVMe/SATA SSD or HDD | All S3 object data — AMIs, snapshots, user-uploaded artifacts, IAM state files, tenant data. |
+| NATS JetStream disk | NVMe/SATA SSD | IAM NATS token, cluster metadata, pending job state, NATS KV entries (including wrapped DEKs). |
+| Removable master-key media | USB flash token, HSM, smart card | Master encryption key (Phase 4 USB-mount path). Destroying this sanitizes every DEK-encrypted volume cluster-wide. |
+| Backup media | LTO tape, removable HDD/SSD, off-site cloud backup | Any of the above, point-in-time. |
+| BMC / iDRAC / iLO storage | Embedded flash | Console recordings, SEL logs, BMC credentials, cached operator certificates. Sanitize via the BMC "Reset to defaults" / "Erase user data" command before disposal. |
+| Switch / router config storage | Embedded flash | Cluster VLAN, management IPs, ACLs. Operator network kit, out of Spinifex scope but noted for completeness. |
+| Optical / write-once media | DVD/BD-R | If used to transport keys or images. Always Destroy. |
+
+Any drive whose history cannot be traced — pulled from a spares bin, recovered from a failed host, found unlabelled — must be treated as though it held FCI.
+
+## 2. Sanitization Method Selection
+
+The method depends on (a) media type and (b) whether the media remains inside the protected boundary after sanitization.
+
+| Scenario | Minimum method | Reference |
+|----------|----------------|-----------|
+| Encrypted volume returning to free pool, DEK deletable | **Cryptographic Erase** (Purge) via DEK deletion | [§3](#3-volume-level-sanitization--before-reuse-mpl1-383-b) |
+| SSD / NVMe leaving the node | **Cryptographic Erase** + **Purge** (SANITIZE block-erase or ATA Secure Erase) | [§4.1](#41-nvme-ssd) |
+| Magnetic HDD leaving the node, operational | **Purge** (single-pass overwrite with verify, or ATA Secure Erase on drives that support it) | [§4.3](#43-hdd) |
+| Magnetic HDD leaving the node, faulty (not writable) | **Destroy** (degauss followed by shred/incinerate) | [§4.3](#43-hdd) |
+| Any media leaving operator control (resale, warranty, disposal) | **Destroy** if residual confidentiality concern remains after Purge; otherwise Purge + third-party attestation | [§4.4](#44-destruction) |
+| Tape | **Destroy** (degauss + shred) — overwrite is not reliable on LTO with compression | [§6](#6-removable-and-backup-media) |
+| Optical write-once | **Destroy** (shred/incinerate) | [§6](#6-removable-and-backup-media) |
+| BMC / switch flash | Vendor "factory reset + erase user data"; Destroy chip if procedure is not available | [§1](#1-media-in-scope) |
+
+**Cryptographic erase is only valid when** the key was generated and handled under modern cryptographic hygiene (AES-256 or equivalent, key never exfiltrated, key store itself sanitized) and the encrypted data was not also written anywhere in plaintext. For Spinifex volumes these conditions hold.
+
+**Self-encrypting drives (SEDs)** support a single-command cryptographic erase via the TCG Opal "RevertSP" / `sedutil-cli --revertNoErase` or the ATA `SECURITY ERASE UNIT ENHANCED` that triggers the on-drive MEK rotation. When using an SED as a node system or data disk, record the SED model and method in the decommissioning procedure.
+
+## 3. Volume-Level Sanitization — Before Reuse (MP.L1-3.8.3 [b])
+
+Applies to EBS volumes, AMIs, and snapshots whose underlying storage will be reused for a different tenant or workload.
+
+### 3.1 Cryptographic Erase via DEK Deletion
+
+Deletion is sanitization:
+
+- `DeleteVolume` removes the wrapped DEK from NATS KV. The chunks in Predastore S3 remain but are unrecoverable ciphertext. No further operator action is required.
+- `TerminateInstances` with the default `DeleteOnTermination=true` on root volumes performs the same cryptographic erase for the terminated instance's root.
+- `DeleteSnapshot` removes the snapshot's DEK wrapper; the underlying chunks become irrecoverable ciphertext.
+
+This is the sanitization-before-reuse path for every in-platform object. Operators do not need to take separate action — the API call is the sanitization.
+
+### 3.2 What Happens to the Physical Storage
+
+Cryptographic erase leaves ciphertext chunks in Predastore. These chunks are eventually overwritten as new volumes reuse the space. For CMMC purposes, the ciphertext without the DEK satisfies Purge. The underlying drive still requires whole-drive sanitization when it eventually leaves the cluster — see [§4](#4-whole-drive-and-node-decommissioning--before-disposal-mpl1-383-a).
+
+## 4. Whole-Drive and Node Decommissioning — Before Disposal (MP.L1-3.8.3 [a])
+
+Applies when a physical drive — or a whole node — leaves the Spinifex cluster: drive swap, node retirement, warranty return, resale, recycling, destruction.
+
+### 4.1 NVMe SSD
+
+Preferred: the NVMe **SANITIZE** command with the **Block Erase** action. Supported on most enterprise NVMe drives.
+
+```bash
+# Confirm support
+nvme id-ctrl /dev/nvme0 | grep -i sanicap
+
+# Purge — block erase
+nvme sanitize /dev/nvme0 --sanact=2
+
+# Poll until complete
+nvme sanitize-log /dev/nvme0
+```
+
+Fallback for drives without SANITIZE support: NVMe **Format** with Secure Erase set to 1 (`--ses=1`), or vendor tool. Cryptographic Erase (`--sanact=4`) is acceptable for SEDs whose encryption posture is documented; otherwise Block Erase is the safer default.
+
+Boot media: NVMe targets must be the non-boot drive when running from the OS. For the node system disk, boot a sanitization-purpose live USB (e.g. [PartedMagic](https://partedmagic.com/), the vendor's diagnostic ISO) or pull the drive and sanitize in a dedicated sanitization workstation.
+
+### 4.2 SATA/SAS SSD
+
+Preferred: **ATA Secure Erase (Enhanced)** via `hdparm`:
+
+```bash
+# Confirm support and not frozen
+hdparm -I /dev/sdX | grep -A1 "Security"
+
+# If frozen, power-cycle (suspend/resume or hot-swap) without unplugging the OS disk
+
+# Set password and issue enhanced erase
+hdparm --user-master u --security-set-pass p /dev/sdX
+hdparm --user-master u --security-erase-enhanced p /dev/sdX
+```
+
+Fallback: vendor Secure Erase utility, or `blkdiscard --secure /dev/sdX` on drives that report `TRIM deterministic + RZAT`. A single-pass overwrite with `shred -v -n 1 /dev/sdX` is **not** sufficient for SSDs because of wear-levelling reserve blocks — use Secure Erase or Destroy.
+
+### 4.3 HDD
+
+Preferred: **ATA Secure Erase** (same `hdparm` sequence as [§4.2](#42-satasas-ssd)). Most modern HDDs support it.
+
+Fallback: single-pass overwrite with verify:
+
+```bash
+shred -v -n 1 -z /dev/sdX
+# or
+dd if=/dev/zero of=/dev/sdX bs=1M status=progress && \
+  cmp /dev/zero /dev/sdX          # expect "EOF on /dev/zero" only
+```
+
+For HDDs that fail or refuse sanitization, or that held high-sensitivity data, **degauss then Destroy** — degaussing alone renders a modern HDD non-functional, so there is no reuse path after degauss. Degaussing requires an NSA/CSS-listed degausser appropriate to the drive's coercivity.
+
+### 4.4 Destruction
+
+When a drive cannot be sanitized (failed, SED with no working password, sanitize aborts) or when leaving operator control with any residual confidentiality concern, Destroy:
+
+- **SSD/NVMe:** shred to ≤2 mm particle size (NSA/CSS EPL-listed shredder) or incinerate. Crushing alone is not sufficient for modern flash — chips can survive.
+- **HDD:** degauss (on drives with magnetic platters) followed by shredding, drilling multiple holes through the platters and the head assembly, or incineration.
+- **Tape, optical:** shredding.
+- **Chain of custody:** transport destroyed media inside tamper-evident containers. The destruction record ([§7](#7-evidence-and-record-keeping)) must name the destruction method, destruction vendor (if third-party), and operator witness.
+
+Third-party destruction vendors must provide a certificate of destruction listing every drive serial. Reconcile against the asset register before the destruction record is closed.
+
+### 4.5 Node Decommissioning Runbook
+
+When retiring a whole node:
+
+1. **Drain** — migrate or terminate all instances hosted on the node; confirm Predastore and Viperblock roles are reassigned if the node held one. Volume DEKs of terminated instances are already cryptographically erased per [§3.1](#31-cryptographic-erase-via-dek-deletion).
+2. **Deregister** — `bd`-linked change ticket; the operator removes the node from the cluster (`spx admin node remove`) so NATS routes, OVN chassis, and predastore distributed membership drop it.
+3. **Stop services and unmount** — `systemctl stop 'spinifex-*'`, `nbdkit` sessions, OVN agent. Confirm `/run/spinifex/nbd/` is empty.
+4. **Destroy on-disk keys** — see [§5](#5-key-destruction). This is the single most important step: sanitization of volume data reduces to sanitization of keys once data is encrypted.
+5. **Sanitize each drive** by media type per [§4.1](#41-nvme-ssd) – [§4.3](#43-hdd). Every drive bay is sanitized, including any unpopulated cache or WAL devices.
+6. **Verify** — capture the output of the sanitize/erase command (exit code + sanitize-log for NVMe, `hdparm -I` "not enabled, not locked" for ATA, overwrite verify for HDD overwrite) into the decommissioning record.
+7. **Label** drives "Sanitized — <method> — <date> — <operator>" before they leave the rack or enter a transit container. Unlabelled drives are treated as unsanitized if reintroduced.
+8. **Record** per [§7](#7-evidence-and-record-keeping).
+
+## 5. Key Destruction
+
+Cryptographic erase relies on the key being unrecoverable. A drive that previously held `/etc/spinifex/master.key` must have that key destroyed — otherwise an attacker with both the drive image and the key recovers the cleartext even after "erase".
+
+| Key | Location | Destruction |
+|-----|----------|-------------|
+| `/etc/spinifex/master.key` | Node system disk (every node) | Overwrite the file (`shred -u /etc/spinifex/master.key`) before the drive is sanitized, in addition to drive sanitization. Once the master key is gone cluster-wide, every DEK it wrapped is effectively destroyed. |
+| Cluster CA private key (`/etc/spinifex/ca.key`) | Leader node system disk | Same treatment. Loss of the CA key does not sanitize data, but prevents impersonation of the cluster identity post-disposal. |
+| Per-node TLS private key (`/etc/spinifex/server.key`) | Each node system disk | `shred -u` before drive sanitization. |
+| Wrapped DEKs in NATS KV | `/var/lib/spinifex/nats/` on NATS-carrying nodes | Deleted via the `DeleteVolume` API; the KV record is compacted. Sanitization of the NATS JetStream disk per [§4](#4-whole-drive-and-node-decommissioning--before-disposal-mpl1-383-a) ensures no recoverable copies. |
+| Removable master-key media (Phase 4 USB) | USB flash token | Physically destroy the token when the cluster is decommissioned; alternatively retain inside a secured location as evidence of cluster-wide cryptographic erase. Do not reuse for another cluster without full sanitization (see [§6](#6-removable-and-backup-media)). |
+| Backup-copy keys | Any off-site or escrow copy | Destroy simultaneously with the primary. A surviving backup defeats the cryptographic erase. |
+
+If even one copy of the master key survives and the encrypted drives are recoverable, cryptographic erase has **not** been achieved. Track every key copy in the device register ([Physical Security Guide §5](/docs/security/physical-security-guide#5-manage-physical-access-devices-pel1-3105)).
+
+## 6. Removable and Backup Media
+
+| Media | Sanitization |
+|-------|--------------|
+| USB flash (master-key token, key-transport USB, image-import USB) | Destroy. The price of a replacement does not justify the risk of partial wear-levelled sanitization. Record the serial in the device register and the destruction in the decommissioning log. |
+| External HDD/SSD (backup) | Treat as [§4.2](#42-satasas-ssd) / [§4.3](#43-hdd). |
+| LTO tape | Degauss with a tape-rated degausser, then shred. Overwrite is not reliable under LTO hardware compression and is not accepted by 800-88 for tape Purge. |
+| Optical (CD/DVD/BD-R) | Shred or incinerate. |
+| Off-site cloud backup | Issue the cloud provider's delete-and-purge API; retain the provider's deletion-confirmation receipt. For cryptographically wrapped backups, destroy the wrapping key cluster-side and retain that deletion as evidence. |
+| Printed material (console photos, recovery codes, admin handover sheets) | Cross-cut shred. |
+
+Any removable medium that has entered the protected boundary and been written to must be tracked in the device register (see [Physical Security Guide §5](/docs/security/physical-security-guide#5-manage-physical-access-devices-pel1-3105)) and accounted for at decommissioning.
+
+## 7. Evidence and Record Keeping
+
+For CMMC assessment, retain the following for at least three years (or longer where contract policy requires):
+
+- **Asset register** listing every drive and removable medium by serial number, media type, role in the cluster (system / WAL / chunk / predastore / NATS / backup), and in-service / decommissioned status.
+- **Decommissioning record** per drive or node with: date, operator, sanitization method, command output or tool report, verification result, final disposition (reuse / return / destroy), destination (e.g. "warranty RMA #12345", "shredder vendor X, CoD #6789").
+- **Certificates of destruction** from third-party vendors, reconciled against the asset register.
+- **Key-destruction record** confirming `master.key`, CA key, and per-node keys were shredded before drive sanitization.
+- **Exceptions log** — faulty drives that could not be sanitized and were destroyed instead; any sanitize failures and their resolution.
+- **Annual attestation** from the operator confirming the procedures in this guide operated for the prior 12 months, signed by the named security owner.
+
+Cross-reference each decommissioning record to the change ticket that drove the retirement, so drive disposal can be audited against cluster state change.
+
+## 8. Operator Checklist
+
+Use this list to confirm a Spinifex deployment meets MP.L1-3.8.3:
+
+- Asset register enumerates every drive and removable medium by serial, media type, and role.
+- Every volume-producing path uses encryption so that `DeleteVolume` is sanitization.
+- Node-decommissioning runbook exists and matches [§4.5](#45-node-decommissioning-runbook), including the drain → deregister → key-destroy → drive-sanitize sequence.
+- Sanitization method is pre-selected per media type per [§2](#2-sanitization-method-selection); tool availability (hdparm, nvme-cli, vendor ISOs, degausser, shredder or destruction vendor) is verified before a decommissioning begins.
+- `master.key`, CA key, and per-node TLS keys are shredded before the drive holding them is sanitized.
+- Every backup copy of any key is destroyed at the same time as the primary; surviving backup copies are tracked and accounted for.
+- Removable media (USB, tape, optical) is destroyed rather than erased unless the medium is an SED with documented cryptographic erase.
+- Decommissioning records capture method, tool output, verification, final disposition, and operator; certificates of destruction are reconciled against the asset register.
+- Drives awaiting sanitization or awaiting pickup by a destruction vendor are held inside the physical protection boundary ([Physical Security Guide §1](/docs/security/physical-security-guide#1-protected-assets)) with chain of custody logged.
+- System security plan references this guide and names the sanitization tooling, destruction vendor (if any), retention period for records, and the security owner attesting annually.

docs/security/network-connections/README.md

@@ -105,8 +105,8 @@ Spinifex nodes initiate a small, fixed set of outbound connections.
 
 | Destination | Purpose | Protocol | Verification |
 |-------------|---------|----------|--------------|
-| `https://cloud.debian.org/images/cloud/trixie/20260501-2465` | Debian 13 cloud image download | HTTPS | TLS + checksum verification |
-| `https://cloud-images.ubuntu.com/resolute/current` | Ubuntu 26.04 LTS cloud image download | HTTPS | TLS + checksum verification |
+| `https://cloud.debian.org/images/cloud/trixie/` | Debian 13 cloud image download | HTTPS | TLS + checksum verification |
+| `https://cloud-images.ubuntu.com/resolute/` | Ubuntu 26.04 LTS cloud image download | HTTPS | TLS + checksum verification |
 | `https://d2yp8ipz5jfqcw.cloudfront.net` | Alpine image for managed HAProxy load-balancer | HTTPS | TLS + checksum verification |
 | `https://install.mulgadc.com/install` | One-shot install telemetry POST on `spx admin init` / `join`. | HTTPS | TLS |
 

docs/service-interfaces.yaml

@@ -0,0 +1,305 @@
+# Service interface manifest.
+#
+# Source of truth for "which services produce/consume which NATS subjects"
+# and "which e2e suites cover which services". Consumed by
+# tests/e2e/select-suites (Bead 4) to derive targeted suite selection from
+# a PR diff.
+#
+# See docs/development/improvements/e2e-targeted-suite-selection.md.
+#
+# Editing rules
+# -------------
+# - subscribes / publishes: subjects this service receives / sends. Both
+#   lists use the literal subject string, or a "<prefix>.*" wildcard for
+#   pattern subscriptions (e.g. "ebs.snapshot.*").
+# - depends_on: subjects this service requires from peers. Used by the
+#   selector to compute "if X changes, which suites might break".
+# - Subjects must match what is actually referenced in source. The lint
+#   in Bead 5 greps `"(ec2|vpc|ebs|s3|iam|elbv2|spinifex)\.[a-zA-Z._-]+"`
+#   and fails if any subject is absent from this manifest.
+
+version: 1
+
+services:
+
+  awsgw:
+    path: spinifex/services/awsgw
+    # TLS frontend, SigV4 auth. Pure router: HTTP → NATS request/response.
+    # No subscribes; only publishes (via NATSRequest).
+    subscribes: []
+    publishes:
+      - "ec2.*"
+      - "elbv2.*"
+      - "iam.*"
+    depends_on:
+      - "ec2.*"
+      - "elbv2.*"
+      - "iam.*"
+
+  spinifex_daemon:
+    path: spinifex/services/spinifex
+    # Orchestrator. Owns most ec2.* + all elbv2.* + iam event publication.
+    # Translates VPC / EBS / S3 operations into vpc.* / ebs.* / s3.* and
+    # forwards to vpcd / viperblockd / predastore.
+    #
+    # Source layout: spinifex/services/spinifex (entry), spinifex/daemon
+    # (subscribe registration), spinifex/handlers/{ec2,elbv2,iam} (per-
+    # subject handler bodies). Selector treats all four paths as belonging
+    # to this service via the `additional_paths` field.
+    additional_paths:
+      - spinifex/daemon
+      - spinifex/handlers/ec2
+      - spinifex/handlers/elbv2
+      - spinifex/handlers/iam
+    subscribes:
+      # EC2 instance lifecycle.
+      - "ec2.RunInstances"
+      - "ec2.DescribeInstances"
+      - "ec2.DescribeStoppedInstances"
+      - "ec2.DescribeTerminatedInstances"
+      - "ec2.DescribeInstanceAttribute"
+      - "ec2.DescribeInstanceStatus"
+      - "ec2.DescribeInstanceTypes"
+      - "ec2.ModifyInstanceAttribute"
+      - "ec2.ModifyVolume"
+      - "ec2.start"
+      - "ec2.terminate"
+      - "ec2.launch"
+      - "ec2.cmd.*"
+      # EC2 VPC / network ops handled by daemon before delegation to vpcd.
+      - "ec2.CreateVpc"
+      - "ec2.DescribeVpcs"
+      - "ec2.CreateSubnet"
+      - "ec2.CreateInternetGateway"
+      - "ec2.CreateKeyPair"
+      - "ec2.CreateImage"
+      - "ec2.DescribePlacementGroups"
+      # Cluster coordination subjects.
+      - "ec2.ReserveClusterNode"
+      - "ec2.FinalizeClusterInstances"
+      - "spinifex.node.status"
+      - "spinifex.node.vms"
+      - "spinifex.nodes.discover"
+      - "spinifex.cluster.shutdown.progress"
+      - "spinifex.storage.config"
+      # ELBv2 control plane (handlers/elbv2/service_nats.go topic map).
+      - "elbv2.CreateLoadBalancer"
+      - "elbv2.DeleteLoadBalancer"
+      - "elbv2.DescribeLoadBalancers"
+      - "elbv2.CreateTargetGroup"
+      - "elbv2.DeleteTargetGroup"
+      - "elbv2.DescribeTargetGroups"
+      - "elbv2.RegisterTargets"
+      - "elbv2.DeregisterTargets"
+      - "elbv2.DescribeTargetHealth"
+      - "elbv2.CreateListener"
+      - "elbv2.DeleteListener"
+      - "elbv2.DescribeListeners"
+    publishes:
+      # → vpcd
+      - "vpc.create"
+      - "vpc.delete"
+      - "vpc.create-subnet"
+      - "vpc.delete-subnet"
+      - "vpc.create-sg"
+      - "vpc.delete-sg"
+      - "vpc.update-sg"
+      - "vpc.create-port"
+      - "vpc.delete-port"
+      - "vpc.update-port-sgs"
+      - "vpc.igw-attach"
+      - "vpc.igw-detach"
+      - "vpc.add-nat-gateway"
+      - "vpc.delete-nat-gateway"
+      - "vpc.add-nat"
+      - "vpc.delete-nat"
+      - "vpc.dhcp.acquire"
+      - "vpc.dhcp.release"
+      # → viperblockd
+      - "ebs.mount"
+      - "ebs.unmount"
+      - "ebs.delete"
+      - "ebs.sync"
+      - "ebs.snapshot.*"
+      # → predastore (S3 / AMI / keypair store)
+      - "s3.*"
+      # IAM lifecycle event
+      - "iam.account.created"
+    depends_on:
+      - "vpc.*"
+      - "ebs.*"
+      - "s3.*"
+
+  vpcd:
+    path: spinifex/services/vpcd
+    # OVN-backed VPC / subnet / SG / NAT / IGW data plane.
+    subscribes:
+      - "vpc.create"
+      - "vpc.delete"
+      - "vpc.create-subnet"
+      - "vpc.delete-subnet"
+      - "vpc.create-sg"
+      - "vpc.delete-sg"
+      - "vpc.update-sg"
+      - "vpc.create-port"
+      - "vpc.delete-port"
+      - "vpc.update-port-sgs"
+      - "vpc.igw-attach"
+      - "vpc.igw-detach"
+      - "vpc.add-nat-gateway"
+      - "vpc.delete-nat-gateway"
+      - "vpc.add-nat"
+      - "vpc.delete-nat"
+      - "vpc.dhcp.acquire"
+      - "vpc.dhcp.release"
+    publishes:
+      - "vpc.port-status"
+      - "vpc.dhcp.lease-expired"
+    depends_on: []
+
+  viperblockd:
+    path: spinifex/services/viperblockd
+    # EBS data plane. Backed by the viperblock storage engine in the
+    # mulga/viperblock submodule. Submodule pointer bump → selector forces
+    # "run all" (contract changes can be invisible at the spinifex source
+    # level).
+    subscribes:
+      - "ebs.mount"
+      - "ebs.unmount"
+      - "ebs.delete"
+      - "ebs.sync"
+      - "ebs.snapshot.*"
+    publishes:
+      - "ebs.mount.response"
+      - "ebs.unmount.response"
+    depends_on: []
+
+  predastore:
+    path: spinifex/services/predastore
+    # S3 / AMI / keypair storage. External submodule under mulga/predastore;
+    # this dir contains only the spinifex-side adapter. Submodule pointer
+    # bump → selector forces "run all".
+    subscribes:
+      - "s3.*"
+    publishes: []
+    depends_on: []
+
+  nats:
+    path: spinifex/services/nats
+    # NATS broker bootstrap / cluster config. Not a NATS pub/sub
+    # participant. Any change here is infra-path; selector forces
+    # "run all".
+    subscribes: []
+    publishes: []
+    depends_on: []
+
+  spinifexui:
+    path: spinifex/services/spinifexui
+    # Web UI. Talks to awsgw over HTTP, never NATS. Diffs limited to this
+    # dir do not need any e2e suite to run.
+    subscribes: []
+    publishes: []
+    depends_on: []
+    covers_no_suite: true
+
+
+# Fixture → service mapping. Each Ensure* helper in
+# tests/e2e/harness/fixtures.go exercises this set of services; the
+# Bead 4 selector AST-scans test files for harness.Ensure* call sites
+# and resolves coverage via this block.
+fixtures:
+
+  EnsureKeyPair:
+    services: [spinifex_daemon, predastore]
+
+  EnsureAMI:
+    services: [spinifex_daemon, predastore, viperblockd]
+
+  EnsureDefaultVPC:
+    services: [vpcd, spinifex_daemon]
+
+  EnsureSubnet:
+    services: [vpcd, spinifex_daemon]
+
+  EnsureSG:
+    services: [vpcd, spinifex_daemon]
+
+  EnsureInstance:
+    services: [spinifex_daemon, viperblockd, vpcd]
+
+  EnsureVolume:
+    services: [spinifex_daemon, viperblockd]
+
+  EnsureSnapshot:
+    services: [spinifex_daemon, viperblockd, predastore]
+
+  EnsureNATGateway:
+    services: [vpcd, spinifex_daemon]
+
+  EnsureLoadBalancer:
+    services: [spinifex_daemon, awsgw]
+
+
+suites:
+
+  e2e-cert:
+    path: tests/e2e/cert
+    covers: [awsgw, spinifex_daemon, vpcd, predastore]
+
+  e2e-lb:
+    path: tests/e2e/lb
+    covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+    # Exercises full ELBv2 control plane (handlers/elbv2) + data plane
+    # (LB agent on the system VM). ELBv2 is part of spinifex_daemon in
+    # this manifest; revisit if it splits into a standalone service.
+
+  e2e-single:
+    path: tests/e2e/single
+    covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+    # Full single-node lifecycle: VPC, AMI, keypair, instance, volume,
+    # snapshot, IAM. Currently runs all phases sequentially under one
+    # `TestSingleNode`. Bead 3 (single-fixture-migration) splits this
+    # into per-test fixture-driven coverage so the selector can pick
+    # subtests, not just whole suites.
+
+  e2e-ddil:
+    path: tests/e2e/ddil
+    covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+    # DDIL = degraded / disconnected / intermittent / low-bandwidth.
+    # Exercises the full daemon stack under network failure injection;
+    # any service touching the wire path is in scope.
+
+  # Suites below are bash-only today. Entries activate as the Go ports
+  # land (Bead 2 fixtures unblock new-shape ports for multinode etc.).
+  #
+  # e2e-multinode:
+  #   path: spinifex/tests/e2e/multinode
+  #   covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+  # e2e-reboot:
+  #   path: spinifex/tests/e2e/reboot
+  #   covers: [spinifex_daemon, viperblockd]
+  # e2e-tofu:
+  #   path: spinifex/tests/e2e/tofu
+  #   covers: [awsgw, spinifex_daemon, vpcd]
+  # e2e-bm:
+  #   path: spinifex/tests/e2e/baremetal
+  #   covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+
+
+# Special-case rules consumed by the selector (Bead 4). Hard-coded in
+# the tool, listed here for visibility.
+#
+# infra_paths_force_all:
+#   - spinifex/services/nats/**
+#   - spinifex/tests/e2e/harness/**
+#   - spinifex/tests/e2e/lib/**
+#   - spinifex/.github/workflows/e2e-go.yml
+#   - spinifex/.github/scripts/**
+#   - spinifex/Makefile
+#   - mulga/scripts/tofu-cluster/**
+#
+# submodule_force_all:
+#   - mulga/predastore (submodule pointer)
+#   - mulga/viperblock (submodule pointer)
+#
+# many_services_force_all_threshold: 3   # ≥ N services touched → run all

docs/terraform-workbooks/bastion-private-subnet/main.tf

@@ -96,7 +96,7 @@ data "aws_ami" "ubuntu" {
 
   filter {
     name   = "name"
-    values = ["*ubuntu-24.04*"]
+    values = ["*ubuntu-26.04*", "*ubuntu-24.04*"]
   }
 
   filter {

docs/terraform-workbooks/nginx-alb/README.md

@@ -86,22 +86,16 @@ Or create a `main.tf` file and paste the full configuration below.
 
 <!-- INCLUDE: main.tf lang:hcl -->
 
-### Step 2. Install Load Balancer AMI
+### Step 2. Install Debian AMI
 
-Next install the load balancer AMI images, which is used as the AMI disk image for launching the load-balancer and a requirement.
-
-```bash
-spx admin images import --name lb-alpine-3.21.6-x86_64
-```
-
-### Step 2.1 Install Debian AMI
-
-Next, install the Debian 13 AMI which is used in the example to host the `nginx` webservers as an EC2 instance.
+Install the Debian 13 AMI which is used in the example to host the `nginx` webservers as an EC2 instance.
 
 ```bash
 spx admin images import --name debian-13-x86_64
 ```
 
+> **Note:** The load balancer itself runs as a direct-boot QEMU microvm using the kernel + initramfs bundled with the Spinifex distribution (`/usr/share/spinifex/microvm/`), so no separate LB AMI import is required.
+
 ### Step 3. Deploy
 
 ```bash

docs/terraform-workbooks/nginx-alb/main.tf

@@ -102,7 +102,7 @@ data "aws_ami" "ubuntu" {
 
   filter {
     name   = "name"
-    values = ["*ubuntu-24.04*"]
+    values = ["*ubuntu-26.04*", "*ubuntu-24.04*"]
   }
 }
 

docs/terraform-workbooks/nginx-webserver/main.tf

@@ -85,7 +85,7 @@ data "aws_ami" "ubuntu" {
 
   filter {
     name   = "name"
-    values = ["*ubuntu-24.04*"]
+    values = ["*ubuntu-26.04*", "*ubuntu-24.04*"]
   }
 
   filter {

docs/terraform-workbooks/s3-webapp/main.tf

@@ -132,7 +132,7 @@ data "aws_ami" "ubuntu" {
 
   filter {
     name   = "name"
-    values = ["*ubuntu-24.04*"]
+    values = ["*ubuntu-26.04*", "*ubuntu-24.04*"]
   }
 
   filter {

go.mod

@@ -7,14 +7,14 @@ require (
 	github.com/charmbracelet/bubbles v1.0.0
 	github.com/charmbracelet/bubbletea v1.3.10
 	github.com/charmbracelet/lipgloss v1.1.0
-	github.com/go-chi/chi/v5 v5.2.5
+	github.com/go-chi/chi/v5 v5.3.0
 	github.com/google/uuid v1.6.0
 	github.com/insomniacslk/dhcp v0.0.0-20260407060928-11b94ed970f2
 	github.com/kdomanski/iso9660 v0.4.0
 	github.com/klauspost/cpuid/v2 v2.3.0
-	github.com/mulgadc/predastore v1.3.0
-	github.com/mulgadc/viperblock v1.3.0
-	github.com/nats-io/nats-server/v2 v2.14.0
+	github.com/mulgadc/predastore v1.5.0
+	github.com/mulgadc/viperblock v1.5.0
+	github.com/nats-io/nats-server/v2 v2.14.1
 	github.com/nats-io/nats.go v1.52.0
 	github.com/ovn-kubernetes/libovsdb v0.8.1
 	github.com/pelletier/go-toml/v2 v2.3.1
@@ -23,8 +23,10 @@ require (
 	github.com/spf13/viper v1.21.0
 	github.com/stretchr/testify v1.11.1
 	go.uber.org/goleak v1.3.0
-	golang.org/x/crypto v0.51.0
+	golang.org/x/crypto v0.52.0
+	golang.org/x/sync v0.20.0
 	gopkg.in/ini.v1 v1.67.2
+	gopkg.in/yaml.v3 v3.0.1
 )
 
 require (
@@ -34,6 +36,8 @@ require (
 	github.com/antithesishq/antithesis-sdk-go v0.7.0 // indirect
 	github.com/armon/go-metrics v0.4.1 // indirect
 	github.com/atotto/clipboard v0.1.4 // indirect
+	github.com/aws/aws-sdk-go-v2 v1.41.7 // indirect
+	github.com/aws/smithy-go v1.25.1 // indirect
 	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/boltdb/bolt v1.3.1 // indirect
@@ -78,7 +82,7 @@ require (
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
 	github.com/jmespath/go-jmespath v0.4.0 // indirect
 	github.com/josharian/native v1.1.0 // indirect
-	github.com/klauspost/compress v1.18.5 // indirect
+	github.com/klauspost/compress v1.18.6 // indirect
 	github.com/klauspost/reedsolomon v1.14.0 // indirect
 	github.com/leodido/go-urn v1.4.0 // indirect
 	github.com/lithammer/fuzzysearch v1.1.8 // indirect
@@ -104,7 +108,7 @@ require (
 	github.com/prometheus/client_model v0.6.2 // indirect
 	github.com/prometheus/common v0.67.5 // indirect
 	github.com/prometheus/procfs v0.20.1 // indirect
-	github.com/quic-go/quic-go v0.59.0 // indirect
+	github.com/quic-go/quic-go v0.59.1 // indirect
 	github.com/rivo/uniseg v0.4.7 // indirect
 	github.com/sagikazarmark/locafero v0.12.0 // indirect
 	github.com/spf13/afero v1.15.0 // indirect
@@ -121,9 +125,8 @@ require (
 	go.yaml.in/yaml/v2 v2.4.4 // indirect
 	go.yaml.in/yaml/v3 v3.0.4 // indirect
 	golang.org/x/mod v0.35.0 // indirect
-	golang.org/x/net v0.53.0 // indirect
-	golang.org/x/sync v0.20.0 // indirect
-	golang.org/x/sys v0.44.0 // indirect
+	golang.org/x/net v0.55.0 // indirect
+	golang.org/x/sys v0.45.0 // indirect
 	golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa // indirect
 	golang.org/x/term v0.43.0 // indirect
 	golang.org/x/text v0.37.0 // indirect
@@ -131,7 +134,6 @@ require (
 	golang.org/x/tools v0.44.0 // indirect
 	golang.org/x/vuln v1.1.4 // indirect
 	google.golang.org/protobuf v1.36.11 // indirect
-	gopkg.in/yaml.v3 v3.0.1 // indirect
 )
 
 tool golang.org/x/vuln/cmd/govulncheck

go.sum

@@ -122,8 +122,8 @@ github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S
 github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
 github.com/gabriel-vasile/mimetype v1.4.13 h1:46nXokslUBsAJE/wMsp5gtO500a4F3Nkz9Ufpk2AcUM=
 github.com/gabriel-vasile/mimetype v1.4.13/go.mod h1:d+9Oxyo1wTzWdyVUPMmXFvp4F9tea18J8ufA774AB3s=
-github.com/go-chi/chi/v5 v5.2.5 h1:Eg4myHZBjyvJmAFjFvWgrqDTXFyOzjj7YIm3L3mu6Ug=
-github.com/go-chi/chi/v5 v5.2.5/go.mod h1:X7Gx4mteadT3eDOMTsXzmI4/rwUpOwBHLpAfupzFJP0=
+github.com/go-chi/chi/v5 v5.3.0 h1:halUjDxhshgXHMrao5bB8eNBXo/rnzwr8m5m36glehM=
+github.com/go-chi/chi/v5 v5.3.0/go.mod h1:R+tYY2hNuVUUjxoPtqUdgBqevM9s9njzkTLutVsOCto=
 github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
 github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
 github.com/go-kit/log v0.1.0/go.mod h1:zbhenjAZHb184qTLMA9ZjW7ThYL0H2mk7Q6pNt4vbaY=
@@ -229,8 +229,8 @@ github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7V
 github.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM=
 github.com/kdomanski/iso9660 v0.4.0 h1:BPKKdcINz3m0MdjIMwS0wx1nofsOjxOq8TOr45WGHFg=
 github.com/kdomanski/iso9660 v0.4.0/go.mod h1:OxUSupHsO9ceI8lBLPJKWBTphLemjrCQY8LPXM7qSzU=
-github.com/klauspost/compress v1.18.5 h1:/h1gH5Ce+VWNLSWqPzOVn6XBO+vJbCNGvjoaGBFW2IE=
-github.com/klauspost/compress v1.18.5/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
+github.com/klauspost/compress v1.18.6 h1:2jupLlAwFm95+YDR+NwD2MEfFO9d4z4Prjl1XXDjuao=
+github.com/klauspost/compress v1.18.6/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
 github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
 github.com/klauspost/cpuid/v2 v2.0.10/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuObKfj5c0PQa7c=
 github.com/klauspost/cpuid/v2 v2.0.12/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuObKfj5c0PQa7c=
@@ -286,18 +286,18 @@ github.com/muesli/cancelreader v0.2.2 h1:3I4Kt4BQjOR54NavqnDogx/MIoWBFa0StPA8ELU
 github.com/muesli/cancelreader v0.2.2/go.mod h1:3XuTXfFS2VjM+HTLZY9Ak0l6eUKfijIfMUZ4EgX0QYo=
 github.com/muesli/termenv v0.16.0 h1:S5AlUN9dENB57rsbnkPyfdGuWIlkmzJjbFf0Tf5FWUc=
 github.com/muesli/termenv v0.16.0/go.mod h1:ZRfOIKPFDYQoDFF4Olj7/QJbW60Ol/kL1pU3VfY/Cnk=
-github.com/mulgadc/predastore v1.3.0 h1:Zf0rIuyQ1mi9Gpd4ihLqik1gqn8L/VKLf6P6Tk9Qs0Q=
-github.com/mulgadc/predastore v1.3.0/go.mod h1:nH5lBc8iXIw4fXXXNAnrJiRO47yZ+ZBCjaqjiCRoGOQ=
-github.com/mulgadc/viperblock v1.3.0 h1:vrDCNqNpLmolbTbgEHMSz8CBjoc8xkmQdtwKhZfrqew=
-github.com/mulgadc/viperblock v1.3.0/go.mod h1:WCP9h+u2DvN8/+ccai5oMOa48hOJPUzSbh7Lia0Q8nY=
+github.com/mulgadc/predastore v1.5.0 h1:FtL9EquoGKl3tcIRIoAXGhQStkN4uPDx0TOQZ89Mh+c=
+github.com/mulgadc/predastore v1.5.0/go.mod h1:fXIGz7yCKcHh68ybcw7aigdOdzIfyt19h/0diPF+JCU=
+github.com/mulgadc/viperblock v1.5.0 h1:2OprAI1oAtitUosS5A9LbenU/RCaw0oxUJvUhv8+tgU=
+github.com/mulgadc/viperblock v1.5.0/go.mod h1:lIuOaBO3gC0o/EwWhJPpCuhUEPQ3/y74xckGlcBKaos=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
 github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
 github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
 github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
-github.com/nats-io/nats-server/v2 v2.14.0 h1:+8q0HrDFotwLLcGH/legOEOnowunhK+aZ4GYBIWpQlM=
-github.com/nats-io/nats-server/v2 v2.14.0/go.mod h1:ImVUUDvfClJbb6cuJQRc1VmgDCXKM5ds0OoiG9MVOKo=
+github.com/nats-io/nats-server/v2 v2.14.1 h1:wXs/a5fw9Hzm3CvuzLxGeIwpjPulSa7gMT3eSuhGkcg=
+github.com/nats-io/nats-server/v2 v2.14.1/go.mod h1:4N17zLpuS7WMbG8T9gsE2B7z9hC9PraPyulVBfpK6nU=
 github.com/nats-io/nats.go v1.52.0 h1:n3avV4VBsCgsdwh71TppsTwtv+QdPs7ntSKM8qJLGsc=
 github.com/nats-io/nats.go v1.52.0/go.mod h1:26HypzazeOkyO3/mqd1zZd53STJN0EjCYF9Uy2ZOBno=
 github.com/nats-io/nkeys v0.4.15 h1:JACV5jRVO9V856KOapQ7x+EY8Jo3qw1vJt/9Jpwzkk4=
@@ -351,8 +351,8 @@ github.com/pterm/pterm v0.12.36/go.mod h1:NjiL09hFhT/vWjQHSj1athJpx6H8cjpHXNAK5b
 github.com/pterm/pterm v0.12.40/go.mod h1:ffwPLwlbXxP+rxT0GsgDTzS3y3rmpAO1NMjUkGTYf8s=
 github.com/pterm/pterm v0.12.83 h1:ie+YmGmA727VuhxBlyGr74Ks+7McV6kT99IB8EU80aA=
 github.com/pterm/pterm v0.12.83/go.mod h1:xlgc6bFWyJIMtmLJvGim+L7jhSReilOlOnodeIYe4Tk=
-github.com/quic-go/quic-go v0.59.0 h1:OLJkp1Mlm/aS7dpKgTc6cnpynnD2Xg7C1pwL6vy/SAw=
-github.com/quic-go/quic-go v0.59.0/go.mod h1:upnsH4Ju1YkqpLXC305eW3yDZ4NfnNbmQRCMWS58IKU=
+github.com/quic-go/quic-go v0.59.1 h1:0Gmua0HW1Tv7ANR7hUYwRyD0MG5OJfgvYSZasGZzBic=
+github.com/quic-go/quic-go v0.59.1/go.mod h1:upnsH4Ju1YkqpLXC305eW3yDZ4NfnNbmQRCMWS58IKU=
 github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
 github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
 github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
@@ -424,8 +424,8 @@ golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnf
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
-golang.org/x/crypto v0.51.0 h1:IBPXwPfKxY7cWQZ38ZCIRPI50YLeevDLlLnyC5wRGTI=
-golang.org/x/crypto v0.51.0/go.mod h1:8AdwkbraGNABw2kOX6YFPs3WM22XqI4EXEd8g+x7Oc8=
+golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988=
+golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc=
 golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa h1:FRnLl4eNAQl8hwxVVC17teOw8kdjVDVAiFMtgUdTSRQ=
 golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa/go.mod h1:zk2irFbV9DP96SEBUUAy67IdHUaZuSnrz1n472HUCLE=
 golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
@@ -442,8 +442,8 @@ golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81R
 golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
 golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
 golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
-golang.org/x/net v0.53.0 h1:d+qAbo5L0orcWAr0a9JweQpjXF19LMXJE8Ey7hwOdUA=
-golang.org/x/net v0.53.0/go.mod h1:JvMuJH7rrdiCfbeHoo3fCQU24Lf5JJwT9W3sJFulfgs=
+golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8=
+golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -484,8 +484,8 @@ golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
-golang.org/x/sys v0.44.0 h1:ildZl3J4uzeKP07r2F++Op7E9B29JRUy+a27EibtBTQ=
-golang.org/x/sys v0.44.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
+golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
+golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
 golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa h1:efT73AJZfAAUV7SOip6pWGkwJDzIGiKBZGVzHYa+ve4=
 golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa/go.mod h1:kHjTxDEnAu6/Nl9lDkzjWpR+bmKfxeiRuSDlsMb70gE=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=

internal/fipsboot/fipsboot.go

@@ -0,0 +1,21 @@
+// Package fipsboot enforces FIPS 140-3 mode at process startup.
+//
+// Blank-import this package from every binary's main:
+//
+//	import _ "github.com/mulgadc/spinifex/internal/fipsboot"
+//
+// Builds with GOFIPS140=v1.0.0 link the Go Cryptographic Module v1.0.0
+// (NIST CMVP cert #5247). At runtime FIPS mode can still be turned off
+// via GODEBUG=fips140=off; init() panics in that case to close the hole.
+package fipsboot
+
+import (
+	"crypto/fips140"
+)
+
+func init() {
+	if !fips140.Enabled() {
+		panic("fipsboot: FIPS 140-3 mode is not enabled at runtime — refusing to start. " +
+			"Build with GOFIPS140=v1.0.0 and do not set GODEBUG=fips140=off.")
+	}
+}

internal/tlsconfig/curves.go

@@ -0,0 +1,37 @@
+// Package tlsconfig pins the cluster-wide TLS 1.3 curve preference list.
+//
+// The list is hardcoded because mixed classification on a single cluster is
+// not supported and the certified FIPS build is itself the per-deployment
+// choice. A typo in a config string would silently downgrade where a missing
+// constant cannot.
+package tlsconfig
+
+import "crypto/tls"
+
+// Curves is the fixed TLS 1.3 curve allowlist:
+//
+//   - X25519MLKEM768     — Cat 3 ML-KEM hybrid, wins under Go 1.26 stdlib filter
+//   - SecP384r1MLKEM1024 — Cat 5 ML-KEM hybrid, fallback for peers that prefer it
+//   - X25519             — classical TLS 1.3 fallback for SDKs without ML-KEM (aws CLI v2 today)
+//   - CurveP384          — classical TLS 1.3 fallback (Java/.NET enterprise SDKs)
+//
+// PQ-capable peers (browsers since 2024, Go aws-sdk v2 with Go 1.24+) always
+// negotiate the hybrid first; classical fallback only triggers for peers that
+// cannot do ML-KEM. Those connections still get TLS 1.3 + AES-256-GCM but
+// remain HNDL-exposed — accepted trade-off until AWS CLI v2 / bundled
+// OpenSSL ships ML-KEM (currently lagging upstream OpenSSL 3.5).
+//
+// Weak/niche entries from the Go stdlib default (SecP256r1MLKEM768, P-256,
+// P-521) are intentionally excluded so they cannot be advertised.
+//
+// Go 1.26 stdlib treats tls.Config.CurvePreferences as an allowlist filter
+// applied to the stdlib default order, not as a user-supplied order.
+//
+// Callers must not mutate this slice. Assign it directly to
+// tls.Config.CurvePreferences; do not append to it.
+var Curves = []tls.CurveID{
+	tls.X25519MLKEM768,
+	tls.SecP384r1MLKEM1024,
+	tls.X25519,
+	tls.CurveP384,
+}

internal/tlsconfig/curves_test.go

@@ -0,0 +1,169 @@
+package tlsconfig_test
+
+import (
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/tls"
+	"crypto/x509"
+	"crypto/x509/pkix"
+	"encoding/pem"
+	"io"
+	"math/big"
+	"net"
+	"net/http"
+	"net/http/httptest"
+	"slices"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/mulgadc/spinifex/internal/tlsconfig"
+)
+
+func TestCurves_OnlyApprovedEntries(t *testing.T) {
+	want := []tls.CurveID{
+		tls.X25519MLKEM768,
+		tls.SecP384r1MLKEM1024,
+		tls.X25519,
+		tls.CurveP384,
+	}
+	if !slices.Equal(tlsconfig.Curves, want) {
+		t.Errorf("Curves = %v, want %v", tlsconfig.Curves, want)
+	}
+}
+
+func TestCurves_ExcludesWeakPrimitives(t *testing.T) {
+	forbidden := []tls.CurveID{
+		tls.SecP256r1MLKEM768, // weakest ML-KEM hybrid (P-256 component)
+		tls.CurveP256,         // not CNSA 2.0
+		tls.CurveP521,         // niche, no current peers
+	}
+	for _, c := range forbidden {
+		if slices.Contains(tlsconfig.Curves, c) {
+			t.Errorf("Curves includes forbidden curve %v", c)
+		}
+	}
+}
+
+// TestIntegration_NegotiatesPQHybrid spins up an httptest TLS server
+// configured with Curves + TLS 1.3 minimum, dials it, and asserts a PQ
+// hybrid is negotiated at TLS 1.3. Per Go 1.26 stdlib filter behavior the
+// winner is X25519MLKEM768; the assertion checks for any approved hybrid
+// to remain stable across stdlib reorderings of our 4 entries.
+func TestIntegration_NegotiatesPQHybrid(t *testing.T) {
+	cert := selfSignedCert(t)
+	srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = io.WriteString(w, "ok")
+	}))
+	srv.TLS = &tls.Config{
+		Certificates:     []tls.Certificate{cert},
+		MinVersion:       tls.VersionTLS13,
+		CurvePreferences: tlsconfig.Curves,
+	}
+	srv.StartTLS()
+	defer srv.Close()
+
+	pool := x509.NewCertPool()
+	pool.AddCert(srv.Certificate())
+
+	client := &http.Client{
+		Timeout: 5 * time.Second,
+		Transport: &http.Transport{
+			TLSClientConfig: &tls.Config{
+				RootCAs:          pool,
+				CurvePreferences: tlsconfig.Curves,
+			},
+		},
+	}
+	resp, err := client.Get(srv.URL)
+	if err != nil {
+		t.Fatalf("https get: %v", err)
+	}
+	defer resp.Body.Close()
+
+	state := resp.TLS
+	if state == nil {
+		t.Fatal("response has no TLS state")
+	}
+	if state.Version != tls.VersionTLS13 {
+		t.Errorf("negotiated version = 0x%x, want TLS 1.3 (0x%x)", state.Version, tls.VersionTLS13)
+	}
+	pqHybrids := []tls.CurveID{tls.X25519MLKEM768, tls.SecP384r1MLKEM1024}
+	if !slices.Contains(pqHybrids, state.CurveID) {
+		t.Errorf("negotiated curve = %v, want one of %v (PQ hybrid)", state.CurveID, pqHybrids)
+	}
+}
+
+// TestIntegration_TLS12ClientRejected confirms MinVersion: tls.VersionTLS13
+// is enforced — a TLS 1.2-pinned client gets a handshake error rather than
+// silently downgrading. This is the negative test that closes the HNDL gap
+// on external surfaces.
+func TestIntegration_TLS12ClientRejected(t *testing.T) {
+	cert := selfSignedCert(t)
+	srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = io.WriteString(w, "ok")
+	}))
+	srv.TLS = &tls.Config{
+		Certificates:     []tls.Certificate{cert},
+		MinVersion:       tls.VersionTLS13,
+		CurvePreferences: tlsconfig.Curves,
+	}
+	srv.StartTLS()
+	defer srv.Close()
+
+	pool := x509.NewCertPool()
+	pool.AddCert(srv.Certificate())
+
+	client := &http.Client{
+		Timeout: 5 * time.Second,
+		Transport: &http.Transport{
+			TLSClientConfig: &tls.Config{
+				RootCAs:    pool,
+				MinVersion: tls.VersionTLS12,
+				MaxVersion: tls.VersionTLS12,
+			},
+		},
+	}
+	_, err := client.Get(srv.URL)
+	if err == nil {
+		t.Fatal("TLS 1.2-pinned client succeeded; MinVersion not enforced")
+	}
+	if !strings.Contains(err.Error(), "protocol version") &&
+		!strings.Contains(err.Error(), "tls:") {
+		t.Errorf("expected protocol-version handshake error, got: %v", err)
+	}
+}
+
+func selfSignedCert(t *testing.T) tls.Certificate {
+	t.Helper()
+	key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+	if err != nil {
+		t.Fatalf("generate key: %v", err)
+	}
+	tmpl := &x509.Certificate{
+		SerialNumber: big.NewInt(1),
+		Subject:      pkix.Name{CommonName: "tlsconfig-test"},
+		NotBefore:    time.Now().Add(-time.Hour),
+		NotAfter:     time.Now().Add(time.Hour),
+		KeyUsage:     x509.KeyUsageDigitalSignature,
+		ExtKeyUsage:  []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
+		DNSNames:     []string{"localhost"},
+		IPAddresses:  []net.IP{net.ParseIP("127.0.0.1"), net.ParseIP("::1")},
+	}
+	der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key)
+	if err != nil {
+		t.Fatalf("create cert: %v", err)
+	}
+	keyDER, err := x509.MarshalECPrivateKey(key)
+	if err != nil {
+		t.Fatalf("marshal key: %v", err)
+	}
+	certPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
+	keyPEM := pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER})
+	pair, err := tls.X509KeyPair(certPEM, keyPEM)
+	if err != nil {
+		t.Fatalf("X509KeyPair: %v", err)
+	}
+	return pair
+}

scripts/ansible/README.md

@@ -31,6 +31,7 @@ inventory and roles relative to CWD.
 | `playbooks/dev-teardown.yml` | Wipe state only (no reinstall) | `reset-dev-env.sh` §teardown |
 | `playbooks/dev-install.yml` | Build + install + init + smoketest on clean box | `dev-install.sh` |
 | `playbooks/dev-reset.yml` | Capture settings → teardown → build → install → init → smoketest | `reset-dev-env.sh` (full) |
+| `playbooks/dev-deploy.yml` | Rebuild + swap binaries/microvm artifacts + restart (no setup.sh, no smoketest) | `make deploy` |
 
 Upcoming (not yet implemented):
 
@@ -44,6 +45,7 @@ ansible-playbook playbooks/dev-preflight.yml
 ansible-playbook playbooks/dev-teardown.yml
 ansible-playbook playbooks/dev-install.yml
 ansible-playbook playbooks/dev-reset.yml
+ansible-playbook playbooks/dev-deploy.yml
 ```
 
 Or via `make` (from `spinifex/`):
@@ -53,8 +55,16 @@ make ansible-dev-preflight
 make ansible-dev-teardown
 make ansible-dev-install
 make ansible-dev-reset
+make ansible-dev-deploy
 ```
 
+### When to use which
+
+- First-time / clean box → `ansible-dev-install`
+- Iterate on Go code, microvm initramfs, lb-agent → `ansible-dev-deploy` (fast)
+- Changed systemd units, helper scripts, logrotate, setup.sh → `ansible-dev-reset` (slow, full rebuild)
+- Need a clean slate without reinstall → `ansible-dev-teardown`
+
 ## Variable overrides
 
 Defaults in `vars/defaults.yml`. Override per run with `-e`:
@@ -74,7 +84,6 @@ Useful overrides:
 | `spinifex_region` | `ap-southeast-2` | Region passed to `spx admin init` |
 | `spinifex_az` | `{{ region }}a` | AZ passed to `spx admin init` |
 | `spinifex_external_mode` | `pool` | `pool` \| `nat` external networking |
-| `spinifex_build_lb_image` | `true` | Build + import LB system image after init |
 
 `dev-reset.yml` automatically captures `region`, `az`, `external_mode`,
 pool range, gateway, prefix-len, nat gateway_ip from the existing

scripts/ansible/playbooks/dev-deploy.yml

@@ -0,0 +1,14 @@
+---
+# In-place artifact swap: rebuilds + replaces binaries/microvm artifacts on
+# an already-installed node, then restarts spinifex.target. Use when iterating
+# on code without altering systemd units, users, dirs, or running the full
+# smoketest. For first-time or unit/script changes use dev-install / dev-reset.
+- name: Dev deploy — build + swap binaries + restart (no setup.sh, no smoketest)
+  hosts: localhost
+  gather_facts: true
+  vars_files:
+    - ../vars/defaults.yml
+
+  roles:
+    - build
+    - deploy

scripts/ansible/roles/build/tasks/main.yml

@@ -11,6 +11,18 @@
     chdir: "{{ mulga_root }}/viperblock"
   changed_when: true
 
+- name: build — detect microVM build script (present only on direct-boot branches)
+  ansible.builtin.stat:
+    path: "{{ spinifex_root }}/scripts/build-microvm-image.sh"
+  register: build_microvm_script
+
+- name: build — microVM kernel + initramfs (direct-boot LBs)
+  ansible.builtin.command:
+    cmd: make build-microvm-image
+    chdir: "{{ spinifex_root }}"
+  changed_when: true
+  when: build_microvm_script.stat.exists
+
 - name: build — verify build artifacts present
   ansible.builtin.stat:
     path: "{{ item }}"
@@ -40,9 +52,15 @@
     mkdir -p "$STAGING/scripts"
     cp "{{ spinifex_root }}/build/scripts/"* "$STAGING/scripts/"
     cp "{{ spinifex_root }}/build/logrotate/spinifex" "$STAGING/logrotate-spinifex"
+    # microVM artifacts ship only when the branch built them.
+    if [ -f "{{ spinifex_root }}/build/microvm/vmlinuz" ] && [ -f "{{ spinifex_root }}/build/microvm/initramfs.cpio.gz" ]; then
+        mkdir -p "$STAGING/microvm"
+        cp "{{ spinifex_root }}/build/microvm/vmlinuz" "$STAGING/microvm/"
+        cp "{{ spinifex_root }}/build/microvm/initramfs.cpio.gz" "$STAGING/microvm/"
+    fi
     tar czf "{{ build_tarball_path }}" -C "$STAGING" .
   changed_when: true
 
 - name: build — summary
   ansible.builtin.debug:
-    msg: "Built spx + nbdkit plugin, tarball at {{ build_tarball_path }}"
+    msg: "Built spx + nbdkit plugin{{ ' + microvm artifacts' if build_microvm_script.stat.exists else '' }}, tarball at {{ build_tarball_path }}"

scripts/ansible/roles/deploy/tasks/main.yml

@@ -0,0 +1,148 @@
+---
+# In-place artifact swap for an already-installed spinifex node. Mirrors
+# `make deploy`: replace binaries + microvm artifacts, restart services.
+# Does NOT run setup.sh (no user/dir/systemd-unit changes) or smoketest.
+
+- name: deploy — fail fast if spinifex isn't already installed
+  ansible.builtin.stat:
+    path: "{{ spinifex_config_file }}"
+  register: deploy_installed_stat
+  failed_when: not deploy_installed_stat.stat.exists
+
+- name: deploy — assert config present
+  ansible.builtin.assert:
+    that: deploy_installed_stat.stat.exists
+    fail_msg: |
+      {{ spinifex_config_file }} missing — this playbook only updates an
+      existing install. Run ansible-dev-install or ansible-dev-reset first.
+
+- name: deploy — verify build artifacts present
+  ansible.builtin.stat:
+    path: "{{ item }}"
+  register: deploy_artifacts
+  loop:
+    - "{{ spinifex_root }}/bin/spx"
+    - "{{ mulga_root }}/viperblock/lib/nbdkit-viperblock-plugin.so"
+
+- name: deploy — assert artifacts exist after build role
+  ansible.builtin.assert:
+    that: item.stat.exists
+    fail_msg: "Build artifact missing: {{ item.item }} — build role should have produced it"
+  loop: "{{ deploy_artifacts.results }}"
+  loop_control:
+    label: "{{ item.item }}"
+
+- name: deploy — detect microvm artifacts (direct-boot branches only)
+  ansible.builtin.stat:
+    path: "{{ spinifex_root }}/build/microvm/{{ item }}"
+  register: deploy_microvm_stat
+  loop:
+    - vmlinuz
+    - initramfs.cpio.gz
+
+- name: deploy — set microvm presence fact
+  ansible.builtin.set_fact:
+    deploy_microvm_present: "{{ deploy_microvm_stat.results | map(attribute='stat.exists') | min }}"
+
+- name: deploy — detect nbdkit plugin directory
+  ansible.builtin.shell: |
+    set -e
+    nbdkit --dump-config | awk -F= '/^plugindir=/{print $2; exit}'
+  register: deploy_plugindir_cmd
+  changed_when: false
+
+- name: deploy — set plugin path fact
+  ansible.builtin.set_fact:
+    deploy_plugindir: "{{ deploy_plugindir_cmd.stdout | trim }}"
+
+- name: deploy — assert plugindir resolved
+  ansible.builtin.assert:
+    that: deploy_plugindir | length > 0
+    fail_msg: "Could not detect nbdkit plugin directory via `nbdkit --dump-config`"
+
+- name: deploy — stop spinifex.target before binary swap
+  become: true
+  ansible.builtin.systemd:
+    name: spinifex.target
+    state: stopped
+  failed_when: false
+
+- name: deploy — install spx binary
+  become: true
+  ansible.builtin.copy:
+    src: "{{ spinifex_root }}/bin/spx"
+    dest: /usr/local/bin/spx
+    mode: "0755"
+    owner: root
+    group: root
+
+- name: deploy — install viperblock nbdkit plugin
+  become: true
+  ansible.builtin.copy:
+    src: "{{ mulga_root }}/viperblock/lib/nbdkit-viperblock-plugin.so"
+    dest: "{{ deploy_plugindir }}/nbdkit-viperblock-plugin.so"
+    mode: "0755"
+    owner: root
+    group: root
+
+- name: deploy — ensure microvm dir
+  become: true
+  ansible.builtin.file:
+    path: /usr/share/spinifex/microvm
+    state: directory
+    mode: "0755"
+    owner: root
+    group: root
+  when: deploy_microvm_present
+
+- name: deploy — install microvm kernel
+  become: true
+  ansible.builtin.copy:
+    src: "{{ spinifex_root }}/build/microvm/vmlinuz"
+    dest: /usr/share/spinifex/microvm/vmlinuz
+    mode: "0644"
+    owner: root
+    group: root
+  when: deploy_microvm_present
+
+- name: deploy — install microvm initramfs
+  become: true
+  ansible.builtin.copy:
+    src: "{{ spinifex_root }}/build/microvm/initramfs.cpio.gz"
+    dest: /usr/share/spinifex/microvm/initramfs.cpio.gz
+    mode: "0644"
+    owner: root
+    group: root
+  when: deploy_microvm_present
+
+- name: deploy — daemon-reload (covers any unit changes shipped via setup.sh later)
+  become: true
+  ansible.builtin.systemd:
+    daemon_reload: true
+
+- name: deploy — start spinifex.target
+  become: true
+  ansible.builtin.systemd:
+    name: spinifex.target
+    state: started
+
+- name: deploy — wait for awsgw to respond
+  become: true
+  become_user: "{{ spinifex_invoking_user }}"
+  environment:
+    AWS_PROFILE: spinifex
+  ansible.builtin.command: aws ec2 describe-regions
+  register: deploy_awsgw
+  retries: 15
+  delay: 2
+  until: deploy_awsgw.rc == 0
+  changed_when: false
+
+- name: deploy — summary
+  ansible.builtin.debug:
+    msg: |
+      Deployed:
+        spx           → /usr/local/bin/spx
+        nbdkit plugin → {{ deploy_plugindir }}/nbdkit-viperblock-plugin.so
+        microvm       → {{ '/usr/share/spinifex/microvm/{vmlinuz,initramfs.cpio.gz}' if deploy_microvm_present else 'skipped (not built on this branch)' }}
+      spinifex.target restarted; awsgw responding.

scripts/ansible/roles/init/tasks/main.yml

@@ -74,19 +74,6 @@
   delay: 2
   changed_when: false
 
-- name: init — build LB agent
-  ansible.builtin.command:
-    cmd: make build-lb-agent
-    chdir: "{{ spinifex_root }}"
-  changed_when: true
-  when: spinifex_build_lb_image | bool
-
-- name: init — import LB system image
-  ansible.builtin.command:
-    cmd: "{{ spinifex_root }}/scripts/build-system-image.sh {{ spinifex_lb_image_conf }} --import --quiet"
-  changed_when: true
-  when: spinifex_build_lb_image | bool
-
 - name: init — summary
   ansible.builtin.debug:
     msg: |
@@ -95,4 +82,3 @@
         external_mode={{ spinifex_external_mode }}
         spinifex.target: started
         CA: {{ spinifex_ca_trust_path }}
-        LB image: {{ 'imported' if spinifex_build_lb_image else 'skipped' }}

scripts/ansible/roles/smoketest/tasks/main.yml

@@ -61,6 +61,7 @@
     --distro {{ spinifex_smoketest_distro }}
     --version {{ spinifex_smoketest_version }}
     --arch {{ smoketest_arch }}
+    --boot-mode uefi
   register: smoketest_import_local
   until: >-
     smoketest_import_local.rc == 0 or

scripts/ansible/vars/defaults.yml

@@ -79,10 +79,6 @@ mulga_root: "{{ lookup('pipe', 'readlink -f ' ~ playbook_dir ~ '/../../../..') }
 # Build output.
 build_tarball_path: /tmp/spinifex-local.tar.gz
 
-# LB system image (built + imported at end of init).
-spinifex_build_lb_image: true
-spinifex_lb_image_conf: "{{ spinifex_root }}/scripts/images/lb.conf"
-
 # Smoketest image — parameterised so tofu examples can share the same base.
 spinifex_smoketest_distro: debian
 spinifex_smoketest_version: "13"

scripts/build-microvm-image.sh

@@ -0,0 +1,264 @@
+#!/bin/bash
+# build-microvm-image.sh — build a minimal Alpine-based microVM kernel + initramfs
+# for use with QEMU microvm machine type and Spinifex lb-agent.
+#
+# Outputs:
+#   $REPO_ROOT/build/microvm/vmlinuz
+#   $REPO_ROOT/build/microvm/initramfs.cpio.gz
+#
+# Rootfs strategy (first match wins):
+#   1. apk available on host → install directly into a temp chroot (Alpine CI/CD)
+#   2. docker available      → run alpine container, export filesystem
+#   3. podman available      → same via podman
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+BUILD_DIR="$REPO_ROOT/build/microvm"
+INIT_SH="$BUILD_DIR/init.sh"
+INITTAB="$BUILD_DIR/inittab"
+
+ALPINE_VERSION="${ALPINE_VERSION:-3.20}"
+ALPINE_MIRROR="${ALPINE_MIRROR:-https://dl-cdn.alpinelinux.org/alpine}"
+ALPINE_PACKAGES="busybox linux-virt haproxy iproute2 ca-certificates"
+
+echo "[build-microvm-image] repo root: $REPO_ROOT"
+echo "[build-microvm-image] build dir: $BUILD_DIR"
+mkdir -p "$BUILD_DIR"
+
+# --- Verify non-negotiable tools ---
+for tool in cpio gzip find; do
+    if ! command -v "$tool" >/dev/null 2>&1; then
+        echo "ERROR: required tool not found: $tool" >&2
+        exit 1
+    fi
+done
+
+# --- Create chroot ---
+CHROOT_DIR=$(mktemp -d)
+CONTAINER_TOOL=""
+CONTAINER_CID=""
+cleanup() {
+    echo "[build-microvm-image] cleaning up chroot: $CHROOT_DIR"
+    rm -rf "$CHROOT_DIR"
+    if [ -n "$CONTAINER_CID" ] && [ -n "$CONTAINER_TOOL" ]; then
+        echo "[build-microvm-image] cleaning up container: $CONTAINER_CID"
+        $CONTAINER_TOOL rm -f "$CONTAINER_CID" >/dev/null 2>&1 || true
+    fi
+}
+trap cleanup EXIT
+
+# --- Populate rootfs ---
+# Strategy 1: native apk (Alpine hosts, CI)
+if command -v apk >/dev/null 2>&1; then
+    echo "[build-microvm-image] rootfs: native apk"
+    mkdir -p \
+        "$CHROOT_DIR/etc/apk" \
+        "$CHROOT_DIR/lib/apk/db" \
+        "$CHROOT_DIR/var/cache/apk"
+    cat > "$CHROOT_DIR/etc/apk/repositories" <<EOF
+${ALPINE_MIRROR}/v${ALPINE_VERSION}/main
+${ALPINE_MIRROR}/v${ALPINE_VERSION}/community
+EOF
+    apk add \
+        --root "$CHROOT_DIR" \
+        --arch x86_64 \
+        --no-cache \
+        --allow-untrusted \
+        --initdb \
+        $ALPINE_PACKAGES
+
+# Strategy 2/3: container runtime (dev machines)
+else
+    if command -v docker >/dev/null 2>&1 && docker info >/dev/null 2>&1; then
+        CONTAINER_TOOL=docker
+    elif command -v podman >/dev/null 2>&1; then
+        CONTAINER_TOOL=podman
+    else
+        echo "ERROR: no rootfs build tool available." >&2
+        echo "       Install one of: apk (Alpine), docker, podman" >&2
+        exit 1
+    fi
+
+    echo "[build-microvm-image] rootfs: $CONTAINER_TOOL export (alpine:${ALPINE_VERSION})"
+    CONTAINER_CID=$($CONTAINER_TOOL run -d \
+        "alpine:${ALPINE_VERSION}" \
+        sh -c "apk add --no-cache ${ALPINE_PACKAGES}")
+
+    exit_code=$($CONTAINER_TOOL wait "$CONTAINER_CID")
+    if [ "$exit_code" != "0" ]; then
+        echo "ERROR: package installation failed in container (exit $exit_code)" >&2
+        exit 1
+    fi
+
+    $CONTAINER_TOOL export "$CONTAINER_CID" | tar -x -C "$CHROOT_DIR"
+fi
+
+# --- Fix /dev device nodes ---
+# tar extraction without root silently creates 0-byte regular files for device
+# nodes (mknod requires CAP_MKNOD). The kernel opens /dev/console to wire
+# init's stdio to the serial console; a regular file there causes all init
+# output to be silently discarded. Recreate the two nodes needed before
+# devtmpfs mounts (which provides the rest at runtime).
+rm -rf "$CHROOT_DIR/dev"
+mkdir "$CHROOT_DIR/dev"
+sudo mknod -m 600 "$CHROOT_DIR/dev/console" c 5 1
+sudo mknod -m 666 "$CHROOT_DIR/dev/null"    c 1 3
+
+# --- Place init script ---
+echo "[build-microvm-image] installing init.sh as /init..."
+if [ ! -f "$INIT_SH" ]; then
+    echo "ERROR: $INIT_SH not found — cannot build initramfs" >&2
+    exit 1
+fi
+install -m 0755 "$INIT_SH" "$CHROOT_DIR/init"
+
+# --- Place inittab ---
+if [ -f "$INITTAB" ]; then
+    echo "[build-microvm-image] installing inittab..."
+    install -m 0644 "$INITTAB" "$CHROOT_DIR/etc/inittab"
+fi
+
+# --- Copy lb-agent binary ---
+LB_AGENT_BIN="$REPO_ROOT/bin/lb-agent"
+if [ -f "$LB_AGENT_BIN" ]; then
+    echo "[build-microvm-image] copying lb-agent binary..."
+    mkdir -p "$CHROOT_DIR/usr/local/bin"
+    install -m 0755 "$LB_AGENT_BIN" "$CHROOT_DIR/usr/local/bin/lb-agent"
+else
+    echo "[build-microvm-image] WARNING: $LB_AGENT_BIN not found — lb-agent will be absent from initramfs" >&2
+fi
+
+# --- Create required directories for init ---
+mkdir -p \
+    "$CHROOT_DIR/proc" \
+    "$CHROOT_DIR/sys" \
+    "$CHROOT_DIR/dev" \
+    "$CHROOT_DIR/etc/ssl/certs" \
+    "$CHROOT_DIR/etc/conf.d"
+
+# --- Locate kernel ---
+KERNEL_IMG=$(find "$CHROOT_DIR/boot" -name "vmlinuz*" | sort | tail -1)
+if [ -z "$KERNEL_IMG" ]; then
+    echo "ERROR: no vmlinuz found in $CHROOT_DIR/boot" >&2
+    exit 1
+fi
+echo "[build-microvm-image] kernel image: $KERNEL_IMG"
+
+KERNEL_VER=$(basename "$KERNEL_IMG" | sed 's/vmlinuz-//')
+KERNEL_CONFIG="$CHROOT_DIR/boot/config-${KERNEL_VER}"
+
+# --- Assert fw_cfg module ---
+if ! find "$CHROOT_DIR/lib/modules" \
+        -name "qemu_fw_cfg.ko*" \
+        -o -name "fw_cfg_sysfs.ko*" \
+    | grep -q .; then
+    if ! grep -q "CONFIG_FW_CFG_SYSFS=y" "$KERNEL_CONFIG" 2>/dev/null; then
+        echo "ERROR: qemu_fw_cfg module missing from initramfs — fw_cfg reads will fail at boot" >&2
+        exit 1
+    fi
+    echo "[build-microvm-image] fw_cfg: built-in (CONFIG_FW_CFG_SYSFS=y)"
+else
+    echo "[build-microvm-image] fw_cfg: module found in lib/modules"
+fi
+
+# --- Strip kernel modules (Phase 2 keep-list approach) ---
+# Only modules needed by a virtio-mmio microvm guest are kept:
+#   virtio*.ko*    — virtio bus, net, blk, rng, console, and helpers
+#   *fw_cfg*.ko*   — QEMU fw_cfg sysfs driver (delivers boot blobs)
+#   *failover*.ko* — net_failover + failover (virtio_net deps for live migration)
+# All other loadable modules are removed. Drivers compiled into the kernel
+# (CONFIG_VIRTIO_MMIO=y etc.) are unaffected — they live in vmlinuz, not here.
+echo "[build-microvm-image] stripping kernel modules to virtio+fw_cfg only..."
+for kver_dir in "$CHROOT_DIR/lib/modules"/*/; do
+    kver="$(basename "$kver_dir")"
+    kernel_dir="$kver_dir/kernel"
+    [ -d "$kernel_dir" ] || continue
+
+    # Collect keeper modules and their relative paths from the kernel/ tree.
+    tmp_keep=$(mktemp -d)
+    find "$kernel_dir" -name "virtio*.ko.gz" -o -name "virtio*.ko" \
+                       -o -name "*fw_cfg*.ko.gz" -o -name "*fw_cfg*.ko" \
+                       -o -name "*failover*.ko.gz" -o -name "*failover*.ko" \
+        2>/dev/null | while read -r mod; do
+        rel="${mod#"$kernel_dir/"}"
+        dest_dir="$tmp_keep/$(dirname "$rel")"
+        mkdir -p "$dest_dir"
+        cp "$mod" "$dest_dir/"
+    done
+
+    # Replace kernel module tree with the keeper set.
+    rm -rf "$kernel_dir"
+    mkdir -p "$kernel_dir"
+    # Restore each kept file into the same relative path.
+    (cd "$tmp_keep" && find . -name "*.ko*" | while read -r f; do
+        dest_dir="$kernel_dir/$(dirname "${f#./}")"
+        mkdir -p "$dest_dir"
+        cp "$f" "$dest_dir/"
+    done)
+    rm -rf "$tmp_keep"
+
+    # Regenerate module dependency map from the surviving modules.
+    depmod -b "$CHROOT_DIR" "$kver" 2>/dev/null || true
+done
+
+# --- Strip package-manager artifacts (not needed at runtime) ---
+echo "[build-microvm-image] stripping package manager artifacts..."
+rm -rf \
+    "$CHROOT_DIR/lib/apk" \
+    "$CHROOT_DIR/var/cache/apk" \
+    "$CHROOT_DIR/etc/apk" \
+    "$CHROOT_DIR/usr/share/man" \
+    "$CHROOT_DIR/usr/share/doc" \
+    "$CHROOT_DIR/usr/share/apk" \
+    "$CHROOT_DIR/usr/include" \
+    "$CHROOT_DIR/usr/lib/pkgconfig"
+
+# --- Strip debug symbols from binaries ---
+echo "[build-microvm-image] stripping debug symbols from binaries..."
+find "$CHROOT_DIR/usr/sbin" "$CHROOT_DIR/usr/bin" \
+     "$CHROOT_DIR/sbin" "$CHROOT_DIR/bin" \
+     "$CHROOT_DIR/usr/local/bin" \
+    -type f 2>/dev/null | while read -r bin; do
+    strip --strip-all "$bin" 2>/dev/null || true
+done
+
+# --- Copy vmlinuz out before stripping /boot from chroot ---
+VMLINUZ_OUT="$BUILD_DIR/vmlinuz"
+echo "[build-microvm-image] copying kernel: $VMLINUZ_OUT"
+cp "$KERNEL_IMG" "$VMLINUZ_OUT"
+
+# Drop /boot from the chroot — vmlinuz, Alpine's stock initramfs-virt, System.map
+# and kernel config all live here (~25 MiB). None are needed at runtime: the
+# guest already has the kernel loaded by QEMU's -kernel, and Alpine's initramfs
+# is unused because we replace it with our own /init. Leaving /boot in the cpio
+# nearly doubles uncompressed initramfs size and forces a 512 MiB guest just to
+# survive decompression.
+rm -rf "$CHROOT_DIR/boot"
+
+# --- Build initramfs ---
+INITRAMFS_OUT="$BUILD_DIR/initramfs.cpio.gz"
+echo "[build-microvm-image] building initramfs: $INITRAMFS_OUT"
+(
+    cd "$CHROOT_DIR"
+    find . | cpio --quiet -o -H newc | gzip -9 > "$INITRAMFS_OUT"
+)
+
+# --- Log artifact sizes ---
+echo ""
+echo "[build-microvm-image] artifacts:"
+ls -lh "$VMLINUZ_OUT" "$INITRAMFS_OUT"
+echo ""
+
+# --- Phase 2 artifact size gate (50 MiB total) ---
+ARTIFACT_MAX_MiB="${MICROVM_ARTIFACT_MAX_MiB:-50}"
+total_bytes=$(( $(stat -c %s "$VMLINUZ_OUT") + $(stat -c %s "$INITRAMFS_OUT") ))
+max_bytes=$(( ARTIFACT_MAX_MiB * 1024 * 1024 ))
+total_mib=$(( (total_bytes + 1048575) / 1048576 ))
+if [ "$total_bytes" -gt "$max_bytes" ]; then
+    echo "ERROR: artifact size ${total_mib} MiB exceeds ${ARTIFACT_MAX_MiB} MiB gate" >&2
+    echo "       Set MICROVM_ARTIFACT_MAX_MiB=N to override during development." >&2
+    exit 1
+fi
+echo "[build-microvm-image] artifact size gate: PASS (${total_mib} MiB / ${ARTIFACT_MAX_MiB} MiB)"
+echo ""
+echo "[build-microvm-image] done."

scripts/build-system-image.sh

@@ -75,6 +75,7 @@ case "$DISTRO" in
         OUTPUT_RAW="${BUILD_DIR}/${IMAGE_NAME}-alpine.raw"
         DISTRO_VERSION="${ALPINE_VERSION}"
         ROOT_PART="${NBD_DEV}"
+        BOOT_MODE="bios"
         ;;
     ubuntu)
         if [[ -z "${UBUNTU_VERSION:-}" ]]; then
@@ -82,11 +83,12 @@ case "$DISTRO" in
             exit 1
         fi
         SOURCE_IMAGE="ubuntu-${UBUNTU_VERSION}-minimal-cloudimg-amd64.img"
-        SOURCE_URL="https://cloud-images.ubuntu.com/minimal/releases/noble/release/${SOURCE_IMAGE}"
+        SOURCE_URL="https://cloud-images.ubuntu.com/minimal/releases/${UBUNTU_CODENAME:-noble}/release/${SOURCE_IMAGE}"
         OUTPUT_IMAGE="${BUILD_DIR}/${IMAGE_NAME}-ubuntu.qcow2"
         OUTPUT_RAW="${BUILD_DIR}/${IMAGE_NAME}-ubuntu.raw"
         DISTRO_VERSION="${UBUNTU_VERSION}"
         ROOT_PART="${NBD_DEV}p1"
+        BOOT_MODE="uefi"
         ;;
     *)
         echo "ERROR: Unknown DISTRO: $DISTRO (supported: alpine, ubuntu)"
@@ -188,11 +190,12 @@ if [[ -f "$OUTPUT_RAW" ]] && [[ $(( $(date +%s) - $(stat -c %Y "$OUTPUT_RAW") ))
 
     if [[ "$DO_IMPORT" == true ]]; then
         echo "Importing as AMI..."
-        IMPORT_ARGS=(--file "$OUTPUT_RAW" --distro "${DISTRO}" --version "${DISTRO_VERSION}" --arch x86_64)
+        rm -f "$OUTPUT_IMAGE"
+        IMPORT_ARGS=(--file "$OUTPUT_RAW" --distro "${DISTRO}" --version "${DISTRO_VERSION}" --arch x86_64 --boot-mode "${BOOT_MODE}")
         if [[ -n "${SYSTEM_TAG:-}" ]]; then
             IMPORT_ARGS+=(--tag "$SYSTEM_TAG")
         fi
-        (cd "$PROJECT_DIR" && ./bin/spx admin images import "${IMPORT_ARGS[@]}")
+        spx admin images import "${IMPORT_ARGS[@]}"
     fi
     exit 0
 fi
@@ -253,9 +256,13 @@ if [[ "$DISTRO" == "ubuntu" ]]; then
     done
     echo "Resizing partition..."
     # After qemu-img resize the GPT backup header sits at the old end of disk.
-    # Move it to the new end before parted tries to resize, otherwise parted
-    # refuses with "Unable to satisfy all constraints on the partition."
-    sudo sgdisk --move-second-header "${NBD_DEV}" 2>/dev/null || true
+    # Relocate it to the new end (sfdisk --relocate is util-linux built-in,
+    # no extra packages needed), then re-read the partition table before parted
+    # runs — without this, parted sees stale GPT geometry and fails with
+    # "Unable to satisfy all constraints on the partition."
+    sudo sgdisk --move-second-header "${NBD_DEV}"
+    sudo partprobe "${NBD_DEV}" 2>/dev/null || true
+    sleep 1
     sudo parted --script "${NBD_DEV}" resizepart 1 100%
 fi
 
@@ -417,19 +424,20 @@ echo ""
 
 if [[ "$DO_IMPORT" == true ]]; then
     echo "Importing as AMI..."
-    IMPORT_ARGS=(--file "$OUTPUT_RAW" --distro "${DISTRO}" --version "${DISTRO_VERSION}" --arch x86_64)
+    rm -f "$OUTPUT_IMAGE"
+    IMPORT_ARGS=(--file "$OUTPUT_RAW" --distro "${DISTRO}" --version "${DISTRO_VERSION}" --arch x86_64 --boot-mode "${BOOT_MODE}")
     if [[ -n "${SYSTEM_TAG:-}" ]]; then
         IMPORT_ARGS+=(--tag "$SYSTEM_TAG")
     fi
-    (cd "$PROJECT_DIR" && sudo -u spinifex-storage ./bin/spx admin images import "${IMPORT_ARGS[@]}")
+    spx admin images import "${IMPORT_ARGS[@]}"
 else
     echo "To import as AMI, run:"
-    echo "  cd $PROJECT_DIR && ./bin/spx admin images import \\"
+    echo "  spx admin images import \\"
     echo "    --file $OUTPUT_RAW \\"
     if [[ -n "${SYSTEM_TAG:-}" ]]; then
-        echo "    --distro ${DISTRO} --version ${DISTRO_VERSION} --arch x86_64 \\"
+        echo "    --distro ${DISTRO} --version ${DISTRO_VERSION} --arch x86_64 --boot-mode ${BOOT_MODE} \\"
         echo "    --tag ${SYSTEM_TAG}"
     else
-        echo "    --distro ${DISTRO} --version ${DISTRO_VERSION} --arch x86_64"
+        echo "    --distro ${DISTRO} --version ${DISTRO_VERSION} --arch x86_64 --boot-mode ${BOOT_MODE}"
     fi
 fi

scripts/check-coverage.sh

@@ -5,7 +5,7 @@ set -euo pipefail
 # Usage: scripts/check-coverage.sh <coverprofile> [quiet]
 
 PROFILE="${1:?Usage: check-coverage.sh <coverprofile> [quiet]}"
-MIN=65.0
+MIN=70.0
 QUIET="${2:-}"
 
 if [[ ! -s "$PROFILE" ]]; then

scripts/demo/dashboard_server.py

@@ -0,0 +1,121 @@
+#!/usr/bin/env python3
+"""Dashboard server for the Spinifex GPU demo.
+
+Proxies streams from the GPU instances and serves the web UI.
+
+Environment variables:
+  YOLO_HOST     Base URL of the YOLO instance,   e.g. http://10.0.0.3:8080
+  OLLAMA_HOST   Base URL of the Ollama instance, e.g. http://10.0.0.2:11434
+  OLLAMA_MODEL  Model name to use for chat       (default: llama3.1:70b)
+  PORT          Server port                       (default: 8000)
+"""
+
+import asyncio
+import json
+import os
+
+import httpx
+import uvicorn
+from fastapi import FastAPI, Request
+from fastapi.responses import HTMLResponse, StreamingResponse
+from fastapi.staticfiles import StaticFiles
+
+YOLO_HOST = os.environ.get("YOLO_HOST", "http://localhost:8080")
+OLLAMA_HOST = os.environ.get("OLLAMA_HOST", "http://localhost:11434")
+OLLAMA_MODEL = os.environ.get("OLLAMA_MODEL", "llama3.1:70b")
+PORT = int(os.environ.get("PORT", "8000"))
+
+app = FastAPI()
+app.mount("/static", StaticFiles(directory="static"), name="static")
+
+
+@app.get("/", response_class=HTMLResponse)
+async def index():
+    with open("static/index.html") as f:
+        return f.read()
+
+
+@app.get("/proxy/video")
+async def proxy_video():
+    async def stream():
+        while True:
+            try:
+                async with httpx.AsyncClient(timeout=None) as client:
+                    async with client.stream("GET", f"{YOLO_HOST}/video") as resp:
+                        async for chunk in resp.aiter_bytes(8192):
+                            yield chunk
+            except Exception:
+                await asyncio.sleep(5)
+
+    return StreamingResponse(stream(), media_type="multipart/x-mixed-replace; boundary=frame")
+
+
+@app.get("/proxy/descriptions")
+async def proxy_descriptions():
+    async def stream():
+        while True:
+            try:
+                async with httpx.AsyncClient(timeout=httpx.Timeout(connect=10.0, read=None, write=None, pool=None)) as client:
+                    async with client.stream("GET", f"{YOLO_HOST}/descriptions") as resp:
+                        async for line in resp.aiter_lines():
+                            yield line + "\n"
+            except Exception:
+                yield 'data: {"type":"error","text":"YOLO stream offline, retrying..."}\n\n'
+                await asyncio.sleep(5)
+
+    return StreamingResponse(stream(), media_type="text/event-stream")
+
+
+@app.post("/chat")
+async def chat(request: Request):
+    body = await request.json()
+    prompt = body.get("prompt", "").strip()
+    if not prompt:
+        return {"error": "empty prompt"}
+
+    async def stream():
+        async with httpx.AsyncClient(timeout=120) as client:
+            async with client.stream(
+                "POST",
+                f"{OLLAMA_HOST}/v1/chat/completions",
+                json={
+                    "model": OLLAMA_MODEL,
+                    "messages": [{"role": "user", "content": prompt}],
+                    "stream": True,
+                },
+            ) as resp:
+                async for line in resp.aiter_lines():
+                    if not line.startswith("data: "):
+                        continue
+                    data_str = line[6:]
+                    if data_str.strip() == "[DONE]":
+                        yield f"data: {json.dumps({'token': '', 'done': True})}\n\n"
+                        break
+                    try:
+                        data = json.loads(data_str)
+                        token = data["choices"][0]["delta"].get("content", "")
+                        yield f"data: {json.dumps({'token': token, 'done': False})}\n\n"
+                    except (json.JSONDecodeError, KeyError, IndexError):
+                        pass
+
+    return StreamingResponse(stream(), media_type="text/event-stream")
+
+
+@app.get("/status")
+async def status():
+    results: dict[str, str] = {}
+    async with httpx.AsyncClient(timeout=3) as client:
+        for name, url in [
+            ("yolo", f"{YOLO_HOST}/health"),
+            ("ollama", f"{OLLAMA_HOST}/v1/models"),
+        ]:
+            try:
+                r = await client.get(url)
+                results[name] = "ok" if r.status_code < 500 else "error"
+            except Exception:
+                results[name] = "offline"
+    return results
+
+
+if __name__ == "__main__":
+    uvicorn.run(app, host="0.0.0.0", port=PORT)

scripts/demo/deploy.sh

@@ -0,0 +1,116 @@
+#!/bin/bash
+# deploy.sh — Deploy and start the Spinifex GPU demo across four instances.
+#
+# Usage:
+#   DASHBOARD_IP=<ip> YOLO_IP=<ip> OLLAMA_IP=<ip> QWEN_IP=<ip> \
+#     VIDEO_PATH=/path/to/video.mp4 \
+#     scripts/demo/deploy.sh
+#
+# Required env vars:
+#   DASHBOARD_IP   4x-1: serves the web dashboard
+#   YOLO_IP        4x-3: runs YOLO inference
+#   OLLAMA_IP      4x-2: runs Ollama + llama3.1:70b
+#   QWEN_IP        12x:  runs Ollama + qwen3-vl:235b
+#   VIDEO_PATH     Local path to the .mp4 file to run YOLO on
+#
+# Optional:
+#   SSH_KEY        (default: ~/.ssh/spinifex-key)
+#   SSH_USER       (default: ec2-user)
+
+set -euo pipefail
+
+DASHBOARD_IP="${DASHBOARD_IP:?set DASHBOARD_IP}"
+YOLO_IP="${YOLO_IP:?set YOLO_IP}"
+OLLAMA_IP="${OLLAMA_IP:?set OLLAMA_IP}"
+QWEN_IP="${QWEN_IP:?set QWEN_IP}"
+VIDEO_PATH="${VIDEO_PATH:?set VIDEO_PATH}"
+
+SSH_KEY="${SSH_KEY:-$HOME/.ssh/spinifex-key}"
+SSH_USER="${SSH_USER:-ec2-user}"
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+_ssh() {
+    local ip="$1"; shift
+    ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
+        -o ConnectTimeout=10 -o BatchMode=yes \
+        -i "$SSH_KEY" "${SSH_USER}@${ip}" "$@"
+}
+
+_scp() {
+    scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
+        -i "$SSH_KEY" -r "$@"
+}
+
+# ── YOLO instance (4x-3) ────────────────────────────────────────────────────
+echo "==> [YOLO] Deploying to $YOLO_IP..."
+_scp "$SCRIPT_DIR/yolo_stream.py" "$SCRIPT_DIR/requirements-yolo.txt" \
+    "${SSH_USER}@${YOLO_IP}:~/"
+_scp "$VIDEO_PATH" "${SSH_USER}@${YOLO_IP}:~/demo_video.mp4"
+
+echo "   Installing Python deps..."
+_ssh "$YOLO_IP" '
+    . ~/yolo-rocm/bin/activate
+    pip install --quiet -r ~/requirements-yolo.txt
+'
+
+echo "   Starting yolo_stream.py..."
+_ssh "$YOLO_IP" "
+    pkill -f yolo_stream.py || true
+    sleep 1
+    . ~/yolo-rocm/bin/activate
+    nohup env \
+        VIDEO_PATH=~/demo_video.mp4 \
+        QWEN_HOST=http://${QWEN_IP}:11434 \
+        YOLO_DEVICE=0 \
+        PORT=8080 \
+        python ~/yolo_stream.py > ~/yolo_stream.log 2>&1 &
+    echo \"   PID \$!\"
+"
+echo "   YOLO stream: http://${YOLO_IP}:8080/video"
+
+# ── Dashboard instance (4x-1) ───────────────────────────────────────────────
+echo ""
+echo "==> [Dashboard] Deploying to $DASHBOARD_IP..."
+_ssh "$DASHBOARD_IP" 'mkdir -p ~/demo/static'
+_scp "$SCRIPT_DIR/dashboard_server.py" "$SCRIPT_DIR/requirements-dashboard.txt" \
+    "${SSH_USER}@${DASHBOARD_IP}:~/demo/"
+_scp "$SCRIPT_DIR/static/index.html" \
+    "${SSH_USER}@${DASHBOARD_IP}:~/demo/static/"
+
+echo "   Installing Python deps..."
+_ssh "$DASHBOARD_IP" '
+    python3 -m venv ~/demo-venv 2>/dev/null || true
+    . ~/demo-venv/bin/activate
+    pip install --quiet -r ~/demo/requirements-dashboard.txt
+'
+
+echo "   Starting dashboard_server.py..."
+_ssh "$DASHBOARD_IP" "
+    pkill -f dashboard_server.py || true
+    sleep 1
+    . ~/demo-venv/bin/activate
+    cd ~/demo
+    nohup env \
+        YOLO_HOST=http://${YOLO_IP}:8080 \
+        OLLAMA_HOST=http://${OLLAMA_IP}:11434 \
+        OLLAMA_MODEL=llama3.1:70b \
+        PORT=8000 \
+        python dashboard_server.py > ~/dashboard.log 2>&1 &
+    echo \"   PID \$!\"
+"
+
+echo ""
+echo "✅ Demo deployed"
+echo ""
+echo "   Dashboard:  http://${DASHBOARD_IP}:8000"
+echo "   YOLO feed:  http://${YOLO_IP}:8080/video"
+echo "   Ollama:     http://${OLLAMA_IP}:11434"
+echo "   Qwen3-VL:   http://${QWEN_IP}:11434"
+echo ""
+echo "   Logs:"
+echo "     ssh -i ${SSH_KEY} ${SSH_USER}@${YOLO_IP}      tail -f ~/yolo_stream.log"
+echo "     ssh -i ${SSH_KEY} ${SSH_USER}@${DASHBOARD_IP} tail -f ~/dashboard.log"
+echo ""
+echo "   NOTE: Ollama and qwen3-vl must already be running on their instances."
+echo "   If not: _ssh \$IP 'sudo systemctl start ollama && ollama run <model>'"

scripts/demo/requirements-dashboard.txt

@@ -0,0 +1,4 @@
+fastapi
+uvicorn[standard]
+httpx
+python-multipart

scripts/demo/requirements-yolo.txt

@@ -0,0 +1,5 @@
+fastapi
+uvicorn[standard]
+httpx
+opencv-python-headless
+ultralytics

scripts/demo/static/index.html

@@ -0,0 +1,498 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="UTF-8">
+<meta name="viewport" content="width=device-width, initial-scale=1.0">
+<title>Spinifex — GPU Compute Demo</title>
+<style>
+* { margin: 0; padding: 0; box-sizing: border-box; }
+
+:root {
+  --bg:       #07070d;
+  --surface:  #0e0e18;
+  --border:   #1a1a2e;
+  --accent:   #6366f1;
+  --accent2:  #22d3ee;
+  --text:     #d4d4d8;
+  --muted:    #52525b;
+  --green:    #22c55e;
+  --red:      #ef4444;
+}
+
+body {
+  background: var(--bg);
+  color: var(--text);
+  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
+  height: 100vh;
+  display: flex;
+  flex-direction: column;
+  overflow: hidden;
+}
+
+/* ── Header ── */
+header {
+  background: var(--surface);
+  border-bottom: 1px solid var(--border);
+  padding: 0 24px;
+  height: 52px;
+  display: flex;
+  align-items: center;
+  gap: 20px;
+  flex-shrink: 0;
+  z-index: 10;
+}
+
+.logo {
+  font-size: 17px;
+  font-weight: 700;
+  letter-spacing: -0.3px;
+  color: #fff;
+  white-space: nowrap;
+}
+.logo em { color: var(--accent); font-style: normal; }
+
+.divider { width: 1px; height: 24px; background: var(--border); }
+
+.subtitle {
+  font-size: 12px;
+  color: var(--muted);
+  letter-spacing: 0.3px;
+}
+
+.instances {
+  margin-left: auto;
+  display: flex;
+  gap: 20px;
+}
+
+.inst {
+  display: flex;
+  align-items: center;
+  gap: 6px;
+  font-size: 11px;
+  color: var(--muted);
+}
+
+.inst-dot {
+  width: 7px; height: 7px;
+  border-radius: 50%;
+  background: var(--green);
+}
+.inst-dot.offline { background: var(--red); }
+.inst-dot.live {
+  background: var(--green);
+  box-shadow: 0 0 0 0 var(--green);
+  animation: ring 2s infinite;
+}
+
+@keyframes ring {
+  0%   { box-shadow: 0 0 0 0 rgba(34,197,94,.6); }
+  70%  { box-shadow: 0 0 0 6px rgba(34,197,94,0); }
+  100% { box-shadow: 0 0 0 0 rgba(34,197,94,0); }
+}
+
+/* ── Layout ── */
+main {
+  flex: 1;
+  display: grid;
+  grid-template-columns: 1fr 400px;
+  grid-template-rows: 1fr 1fr;
+  gap: 10px;
+  padding: 10px;
+  overflow: hidden;
+}
+
+/* ── Panel ── */
+.panel {
+  background: var(--surface);
+  border: 1px solid var(--border);
+  border-radius: 10px;
+  display: flex;
+  flex-direction: column;
+  overflow: hidden;
+  min-height: 0;
+}
+
+.panel-head {
+  padding: 9px 14px;
+  background: rgba(0,0,0,.3);
+  border-bottom: 1px solid var(--border);
+  display: flex;
+  align-items: center;
+  gap: 8px;
+  flex-shrink: 0;
+}
+
+.panel-head h2 {
+  font-size: 11px;
+  font-weight: 600;
+  text-transform: uppercase;
+  letter-spacing: 1px;
+  color: var(--muted);
+}
+
+.chip {
+  margin-left: auto;
+  font-size: 10px;
+  font-weight: 500;
+  padding: 2px 8px;
+  border-radius: 99px;
+  border: 1px solid var(--border);
+  color: var(--muted);
+  white-space: nowrap;
+}
+.chip.accent { border-color: rgba(99,102,241,.4); color: #818cf8; }
+.chip.cyan   { border-color: rgba(34,211,238,.3); color: var(--accent2); }
+
+/* ── Video panel ── */
+.video-panel { grid-row: 1 / 3; position: relative; }
+
+.video-wrap {
+  flex: 1;
+  background: #000;
+  display: flex;
+  align-items: center;
+  justify-content: center;
+  overflow: hidden;
+  min-height: 0;
+}
+
+.video-wrap img {
+  width: 100%;
+  height: 100%;
+  object-fit: contain;
+  display: block;
+}
+
+.video-footer {
+  padding: 8px 14px;
+  font-size: 10px;
+  color: var(--muted);
+  border-top: 1px solid var(--border);
+  flex-shrink: 0;
+  display: flex;
+  gap: 16px;
+}
+
+/* ── Description panel ── */
+.desc-content {
+  flex: 1;
+  overflow-y: auto;
+  padding: 14px;
+  display: flex;
+  flex-direction: column;
+  gap: 14px;
+  scroll-behavior: smooth;
+  min-height: 0;
+}
+
+.desc-entry {
+  padding-bottom: 14px;
+  border-bottom: 1px solid var(--border);
+}
+.desc-entry:last-child { border-bottom: none; padding-bottom: 0; }
+
+.desc-time {
+  font-size: 10px;
+  color: var(--muted);
+  margin-bottom: 5px;
+  font-variant-numeric: tabular-nums;
+}
+
+.desc-text {
+  font-size: 13px;
+  line-height: 1.65;
+  color: var(--text);
+}
+
+.cursor {
+  display: inline-block;
+  width: 2px; height: 13px;
+  background: var(--accent);
+  vertical-align: text-bottom;
+  margin-left: 1px;
+  animation: blink 1s step-end infinite;
+}
+@keyframes blink { 0%,100%{opacity:1} 50%{opacity:0} }
+
+.desc-placeholder {
+  font-size: 12px;
+  color: var(--muted);
+  font-style: italic;
+}
+
+/* ── Chat panel ── */
+.chat-messages {
+  flex: 1;
+  overflow-y: auto;
+  padding: 14px;
+  display: flex;
+  flex-direction: column;
+  gap: 10px;
+  min-height: 0;
+}
+
+.msg {
+  max-width: 88%;
+  padding: 9px 13px;
+  border-radius: 10px;
+  font-size: 13px;
+  line-height: 1.55;
+  word-break: break-word;
+}
+.msg.user {
+  background: #1e1e42;
+  align-self: flex-end;
+  color: #c7d2fe;
+}
+.msg.assistant {
+  background: #111120;
+  align-self: flex-start;
+  color: var(--text);
+  border: 1px solid var(--border);
+}
+
+.chat-foot {
+  border-top: 1px solid var(--border);
+  padding: 10px;
+  display: flex;
+  gap: 8px;
+  flex-shrink: 0;
+}
+
+.chat-foot input {
+  flex: 1;
+  background: var(--bg);
+  border: 1px solid var(--border);
+  border-radius: 7px;
+  padding: 8px 12px;
+  color: var(--text);
+  font-size: 13px;
+  outline: none;
+  transition: border-color .15s;
+}
+.chat-foot input:focus { border-color: var(--accent); }
+.chat-foot input::placeholder { color: var(--muted); }
+
+.send-btn {
+  background: var(--accent);
+  color: #fff;
+  border: none;
+  border-radius: 7px;
+  padding: 8px 16px;
+  font-size: 13px;
+  font-weight: 600;
+  cursor: pointer;
+  transition: background .15s, opacity .15s;
+  white-space: nowrap;
+}
+.send-btn:hover { background: #4f52d1; }
+.send-btn:disabled { opacity: .4; cursor: not-allowed; }
+
+/* scrollbar */
+::-webkit-scrollbar { width: 4px; }
+::-webkit-scrollbar-track { background: transparent; }
+::-webkit-scrollbar-thumb { background: var(--border); border-radius: 2px; }
+</style>
+</head>
+<body>
+
+<header>
+  <div class="logo">Spini<em>fex</em></div>
+  <div class="divider"></div>
+  <div class="subtitle">GPU Compute Platform &mdash; Live Demo</div>
+  <div class="instances">
+    <div class="inst">
+      <div class="inst-dot live" id="dot-yolo"></div>
+      <span>YOLO&nbsp;&middot;&nbsp;g7e.4xlarge</span>
+    </div>
+    <div class="inst">
+      <div class="inst-dot live" id="dot-qwen"></div>
+      <span>Qwen3-VL 235B&nbsp;&middot;&nbsp;g7e.12xlarge</span>
+    </div>
+    <div class="inst">
+      <div class="inst-dot live" id="dot-llama"></div>
+      <span>Llama&nbsp;3.1 70B&nbsp;&middot;&nbsp;g7e.4xlarge</span>
+    </div>
+  </div>
+</header>
+
+<main>
+
+  <!-- Left: YOLO video, full height -->
+  <div class="panel video-panel">
+    <div class="panel-head">
+      <h2>Live Object Detection</h2>
+      <span class="chip accent">YOLO11 &middot; AMD MI350X</span>
+    </div>
+    <div class="video-wrap">
+      <img id="video-feed" src="/proxy/video" alt="YOLO stream loading...">
+    </div>
+    <div class="video-footer">
+      <span>Real-time inference &mdash; AMD MI350X GPU</span>
+      <span id="fps-counter"></span>
+    </div>
+  </div>
+
+  <!-- Top right: qwen3-vl scene analysis -->
+  <div class="panel">
+    <div class="panel-head">
+      <h2>Scene Analysis</h2>
+      <span class="chip cyan">Qwen3-VL 235B &middot; MI350X</span>
+    </div>
+    <div class="desc-content" id="desc-content">
+      <p class="desc-placeholder">Waiting for first frame analysis&hellip;</p>
+    </div>
+  </div>
+
+  <!-- Bottom right: Llama3.1 chat -->
+  <div class="panel">
+    <div class="panel-head">
+      <h2>Chat</h2>
+      <span class="chip accent">Llama 3.1 70B &middot; MI350X</span>
+    </div>
+    <div class="chat-messages" id="chat-messages">
+      <div class="msg assistant">
+        Hello! I&rsquo;m Llama&nbsp;3.1&nbsp;70B running on an AMD MI350X GPU, orchestrated by Spinifex. Ask me anything.
+      </div>
+    </div>
+    <div class="chat-foot">
+      <input type="text" id="chat-input" placeholder="Ask Llama 3.1…" autocomplete="off">
+      <button class="send-btn" id="send-btn" onclick="sendChat()">Send</button>
+    </div>
+  </div>
+
+</main>
+
+<script>
+// ── Scene description stream ─────────────────────────────────────────────────
+const descContent = document.getElementById('desc-content');
+let activeEntry = null;
+let activeText  = '';
+
+function initDescriptions() {
+  const es = new EventSource('/proxy/descriptions');
+
+  es.onmessage = (e) => {
+    let data;
+    try { data = JSON.parse(e.data); } catch { return; }
+
+    if (data.type === 'token' && data.text) {
+      if (!activeEntry) {
+        // Remove placeholder if present
+        descContent.querySelector('.desc-placeholder')?.remove();
+
+        activeEntry = document.createElement('div');
+        activeEntry.className = 'desc-entry';
+        activeEntry.innerHTML = `<div class="desc-time">${new Date().toLocaleTimeString()}</div><div class="desc-text"></div>`;
+        descContent.appendChild(activeEntry);
+        activeText = '';
+      }
+      activeText += data.text;
+      const textEl = activeEntry.querySelector('.desc-text');
+      textEl.innerHTML = escHtml(activeText) + '<span class="cursor"></span>';
+      descContent.scrollTop = descContent.scrollHeight;
+    }
+
+    if (data.type === 'end' && activeEntry) {
+      activeEntry.querySelector('.desc-text').textContent = activeText;
+      activeEntry = null;
+      activeText  = '';
+      // Keep only the last 4 entries
+      const entries = descContent.querySelectorAll('.desc-entry');
+      entries.forEach((el, i) => { if (i < entries.length - 4) el.remove(); });
+    }
+  };
+
+  es.onerror = () => { setTimeout(initDescriptions, 3000); };
+}
+
+// ── Chat ─────────────────────────────────────────────────────────────────────
+const chatMessages = document.getElementById('chat-messages');
+const chatInput    = document.getElementById('chat-input');
+const sendBtn      = document.getElementById('send-btn');
+
+async function sendChat() {
+  const prompt = chatInput.value.trim();
+  if (!prompt || sendBtn.disabled) return;
+
+  chatInput.value = '';
+  sendBtn.disabled = true;
+
+  appendMsg('user', prompt);
+
+  const assistantEl = appendMsg('assistant', '');
+  let text = '';
+
+  try {
+    const resp = await fetch('/chat', {
+      method: 'POST',
+      headers: { 'Content-Type': 'application/json' },
+      body: JSON.stringify({ prompt }),
+    });
+
+    const reader  = resp.body.getReader();
+    const decoder = new TextDecoder();
+    let buf = '';
+
+    while (true) {
+      const { done, value } = await reader.read();
+      if (done) break;
+      buf += decoder.decode(value, { stream: true });
+      const lines = buf.split('\n');
+      buf = lines.pop();
+      for (const line of lines) {
+        if (!line.startsWith('data: ')) continue;
+        let d;
+        try { d = JSON.parse(line.slice(6)); } catch { continue; }
+        text += d.token;
+        assistantEl.innerHTML = escHtml(text) + (d.done ? '' : '<span class="cursor"></span>');
+        chatMessages.scrollTop = chatMessages.scrollHeight;
+      }
+    }
+  } catch (err) {
+    assistantEl.textContent = '(connection error)';
+  }
+
+  sendBtn.disabled = false;
+  chatInput.focus();
+}
+
+function appendMsg(role, text) {
+  const el = document.createElement('div');
+  el.className = `msg ${role}`;
+  el.textContent = text;
+  chatMessages.appendChild(el);
+  chatMessages.scrollTop = chatMessages.scrollHeight;
+  return el;
+}
+
+chatInput.addEventListener('keydown', (e) => { if (e.key === 'Enter') sendChat(); });
+
+// ── Status polling ────────────────────────────────────────────────────────────
+async function pollStatus() {
+  try {
+    const data = await fetch('/status').then(r => r.json());
+    setDot('dot-yolo',  data.yolo   === 'ok');
+    setDot('dot-llama', data.ollama === 'ok');
+  } catch {}
+  setTimeout(pollStatus, 5000);
+}
+
+function setDot(id, ok) {
+  const el = document.getElementById(id);
+  if (!el) return;
+  el.className = 'inst-dot' + (ok ? ' live' : ' offline');
+}
+
+// ── Helpers ───────────────────────────────────────────────────────────────────
+function escHtml(str) {
+  return str.replace(/&/g,'&amp;').replace(/</g,'&lt;').replace(/>/g,'&gt;');
+}
+
+// ── Boot ──────────────────────────────────────────────────────────────────────
+initDescriptions();
+pollStatus();
+</script>
+</body>
+</html>

scripts/demo/yolo_stream.py

@@ -0,0 +1,159 @@
+#!/usr/bin/env python3
+"""YOLO inference server for the Spinifex demo.
+
+Reads a video file, runs YOLO detection, and serves:
+  GET /video        — MJPEG stream of annotated frames
+  GET /descriptions — SSE stream of qwen3-vl scene descriptions
+  GET /health       — {"ok": true}
+
+Environment variables:
+  VIDEO_PATH        Path to input .mp4 (required)
+  QWEN_HOST         Base URL of the qwen3-vl Ollama instance, e.g. http://10.0.0.5:11434
+  YOLO_MODEL        Model name (default: yolo11n.pt)
+  YOLO_DEVICE       GPU device index (default: 0)
+  DESCRIBE_INTERVAL Seconds between qwen3-vl calls (default: 5.0)
+  PORT              Server port (default: 8080)
+"""
+
+import asyncio
+import base64
+import json
+import os
+import queue
+import threading
+import time
+
+import cv2
+import httpx
+import uvicorn
+from fastapi import FastAPI
+from fastapi.responses import StreamingResponse
+from ultralytics import YOLO
+
+VIDEO_PATH = os.environ["VIDEO_PATH"]
+QWEN_HOST = os.environ.get("QWEN_HOST", "")
+YOLO_MODEL = os.environ.get("YOLO_MODEL", "yolo11x.pt")
+YOLO_DEVICE = os.environ.get("YOLO_DEVICE", "0")
+DESCRIBE_INTERVAL = float(os.environ.get("DESCRIBE_INTERVAL", "5.0"))
+PORT = int(os.environ.get("PORT", "8080"))
+
+app = FastAPI()
+
+_frame_lock = threading.Lock()
+_latest_frame: bytes | None = None
+_desc_queue: queue.Queue = queue.Queue(maxsize=100)
+
+
+def _encode(frame, quality: int = 80) -> bytes:
+    _, buf = cv2.imencode(".jpg", frame, [cv2.IMWRITE_JPEG_QUALITY, quality])
+    return buf.tobytes()
+
+
+def _describe(img_b64: str) -> None:
+    if not QWEN_HOST:
+        return
+    try:
+        with httpx.Client(timeout=180) as client:
+            with client.stream(
+                "POST",
+                f"{QWEN_HOST}/api/generate",
+                json={
+                    "model": "qwen3-vl:235b",
+                    "prompt": "Briefly describe what is happening in this scene. Focus on the objects detected and any notable activity.",
+                    "images": [img_b64],
+                    "stream": True,
+                },
+            ) as resp:
+                for line in resp.iter_lines():
+                    if not line:
+                        continue
+                    try:
+                        data = json.loads(line)
+                        token = data.get("response", "")
+                        if token and not _desc_queue.full():
+                            _desc_queue.put(("token", token))
+                        if data.get("done"):
+                            _desc_queue.put(("end", ""))
+                    except json.JSONDecodeError:
+                        pass
+    except Exception as exc:
+        _desc_queue.put(("error", str(exc)))
+
+
+def inference_loop() -> None:
+    model = YOLO(YOLO_MODEL)
+    cap = cv2.VideoCapture(VIDEO_PATH)
+    if not cap.isOpened():
+        raise RuntimeError(f"Cannot open video: {VIDEO_PATH}")
+
+    fps = cap.get(cv2.CAP_PROP_FPS) or 25.0
+    frame_delay = 1.0 / fps
+    last_describe = 0.0
+
+    while True:
+        ret, frame = cap.read()
+        if not ret:
+            cap.set(cv2.CAP_PROP_POS_FRAMES, 0)
+            continue
+
+        results = model(frame, device=YOLO_DEVICE, verbose=False)
+        annotated = results[0].plot()
+
+        with _frame_lock:
+            global _latest_frame
+            _latest_frame = _encode(annotated)
+
+        now = time.monotonic()
+        if QWEN_HOST and now - last_describe >= DESCRIBE_INTERVAL:
+            last_describe = now
+            img_b64 = base64.b64encode(_encode(annotated, quality=60)).decode()
+            threading.Thread(target=_describe, args=(img_b64,), daemon=True).start()
+
+        time.sleep(frame_delay)
+
+
+@app.on_event("startup")
+async def startup() -> None:
+    threading.Thread(target=inference_loop, daemon=True).start()
+
+
+@app.get("/health")
+async def health():
+    return {"ok": True}
+
+
+@app.get("/video")
+async def video():
+    async def generate():
+        while True:
+            with _frame_lock:
+                frame = _latest_frame
+            if frame:
+                yield b"--frame\r\nContent-Type: image/jpeg\r\n\r\n" + frame + b"\r\n"
+            await asyncio.sleep(0.04)
+
+    return StreamingResponse(
+        generate(),
+        media_type="multipart/x-mixed-replace; boundary=frame",
+    )
+
+
+@app.get("/descriptions")
+async def descriptions():
+    async def generate():
+        yield 'data: {"type":"ready"}\n\n'
+        loop = asyncio.get_event_loop()
+        while True:
+            try:
+                kind, text = await loop.run_in_executor(
+                    None, lambda: _desc_queue.get(timeout=30)
+                )
+                yield f"data: {json.dumps({'type': kind, 'text': text})}\n\n"
+            except queue.Empty:
+                yield 'data: {"type":"ping"}\n\n'
+
+    return StreamingResponse(generate(), media_type="text/event-stream")
+
+
+if __name__ == "__main__":
+    uvicorn.run(app, host="0.0.0.0", port=PORT)

scripts/dev-install.sh

@@ -79,10 +79,6 @@ SPINIFEX_AZ="${SPINIFEX_AZ:-${SPINIFEX_REGION}a}"
 echo "=== Initializing (region=$SPINIFEX_REGION az=$SPINIFEX_AZ) ==="
 sudo spx admin init --force --region "$SPINIFEX_REGION" --az "$SPINIFEX_AZ" --node node1 --nodes 1
 
-echo "=== Installing CA certificate ==="
-sudo cp /etc/spinifex/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
-sudo update-ca-certificates
-
 echo "=== Starting services ==="
 sudo systemctl start spinifex.target
 
@@ -91,9 +87,8 @@ if ! id -Gn 2>/dev/null | grep -qw spinifex; then
     newgrp spinifex
 fi
 
-echo "=== Building and importing LB image ==="
-cd "$PROJECT_ROOT" && make build-lb-agent
-"$PROJECT_ROOT/scripts/build-system-image.sh" "$PROJECT_ROOT/scripts/images/lb.conf" --import --quiet
+echo "=== Building and installing microVM artifacts ==="
+cd "$PROJECT_ROOT" && make build-lb-agent install-microvm
 
 echo "=== Done ==="
 echo "Services: sudo systemctl status spinifex.target"

scripts/diff-coverage.sh

@@ -13,7 +13,7 @@ set -euo pipefail
 
 PROFILE=""
 BASE_REF=""
-THRESHOLD=65
+THRESHOLD=60
 QUIET="${QUIET:-}"
 
 while [[ $# -gt 0 ]]; do

scripts/iac/proxmox/README.md

@@ -8,9 +8,9 @@ OpenTofu/Terraform templates for provisioning Spinifex development clusters on P
 - 1 or more Proxmox VE nodes with:
   - A Terraform API token (`Datacenter > Permissions > API Tokens`)
   - SSH access for a `terraform` user on each node
-  - Debian 12 cloud image uploaded to `local:iso/` on each node
+  - Debian 13 cloud image uploaded to `local:iso/` on each node
 - SSH keypair for cloud-init VM access
-- A hardcoded Debian ISO is used, which needs to be pre-installed on each Proxmox host ("debian-12-genericcloud-amd64-20240211-1654.img") https://cdimage.debian.org/images/cloud/bookworm/20240211-1654/debian-12-genericcloud-amd64-20240211-1654.qcow2
+- A hardcoded Debian ISO is used, which needs to be pre-installed on each Proxmox host ("debian-13-genericcloud-amd64-20260518-2482.img") https://cloud.debian.org/images/cloud/trixie/20260518-2482/debian-13-genericcloud-amd64-20260518-2482.qcow2
 
 ## Quick Start (spinifex-test.sh)
 

scripts/images/lb-setup.sh

@@ -1,55 +0,0 @@
-#!/bin/sh
-# lb-setup.sh — LB-specific chroot setup
-#
-# Runs inside the Alpine image chroot after packages are installed.
-# Sets up HAProxy placeholder config and lb-agent OpenRC init script.
-# The lb-agent polls the AWS gateway for config updates and reports health
-# via SigV4-signed HTTP requests. Credentials come from cloud-init env vars.
-
-set -e
-
-# Create haproxy config directory and placeholder config.
-mkdir -p /etc/haproxy
-cat > /etc/haproxy/haproxy.cfg <<'EOF'
-# Placeholder config — replaced by lb-agent on first config fetch
-global
-    daemon
-    maxconn 256
-
-defaults
-    mode http
-    timeout connect 5s
-    timeout client 30s
-    timeout server 30s
-EOF
-
-# Create lb-agent OpenRC init script
-mkdir -p /etc/init.d
-cat > /etc/init.d/lb-agent <<'INITSCRIPT'
-#!/sbin/openrc-run
-
-description="LB Gateway Config Agent"
-command="/usr/local/bin/lb-agent"
-command_args="--lb-id=${LB_LB_ID:-unknown} --gateway=${LB_GATEWAY_URL} --access-key=${LB_ACCESS_KEY} --secret-key=${LB_SECRET_KEY}"
-command_background=true
-pidfile="/run/lb-agent.pid"
-output_log="/var/log/lb-agent.log"
-error_log="/var/log/lb-agent.log"
-
-depend() {
-    need net
-    after firewall
-}
-INITSCRIPT
-chmod 755 /etc/init.d/lb-agent
-
-# Move networking from boot to default runlevel so cloud-init can write
-# the network config before the networking service starts. Without this,
-# Alpine's networking service races cloud-init and starts with no config.
-rc-update del networking boot 2>/dev/null || true
-rc-update add networking default
-
-# Do NOT enable lb-agent at boot — cloud-init must write
-# /etc/conf.d/lb-agent (with LB_LB_ID, LB_GATEWAY_URL, LB_ACCESS_KEY,
-# LB_SECRET_KEY) before the service starts.
-# Cloud-init runcmd starts the service after write_files.

scripts/images/lb.conf

@@ -1,26 +0,0 @@
-# LB system image manifest
-# Used by build-system-image.sh to create an Alpine AMI with HAProxy + lb-agent
-
-IMAGE_NAME="lb"
-IMAGE_DESCRIPTION="LB VM with HAProxy and lb-agent"
-ALPINE_VERSION="3.21.6"
-IMAGE_SIZE="256M"
-
-# Tag applied to the imported AMI so the daemon can discover this as the
-# system image for ELBv2 load balancers. Must match tags.ManagedByELBv2.
-SYSTEM_TAG="spinifex:managed-by=elbv2"
-
-# Packages to install via apk
-APK_PACKAGES="haproxy curl ca-certificates"
-
-# Binaries to copy into the image (source:destination)
-INSTALL_BINARIES="bin/lb-agent:/usr/local/bin/lb-agent"
-
-# Build command for binaries (run before image build)
-BUILD_COMMANDS="make build-lb-agent"
-
-# OpenRC services to enable at boot
-ENABLE_SERVICES="haproxy"
-
-# Custom setup script (runs inside chroot after packages are installed)
-SETUP_SCRIPT="scripts/images/lb-setup.sh"

scripts/images/ubuntu-gpu-amd-setup.sh

@@ -0,0 +1,62 @@
+#!/bin/bash
+set -euo pipefail
+
+# ubuntu-gpu-amd-setup.sh — Chroot setup for an AMD GPU-capable guest image.
+#
+# Installs linux-firmware (AMD firmware blobs required by the amdgpu kernel
+# module), ROCm CLI tools, Python toolchain, and common utilities. Rebuilds
+# initramfs so AMD firmware is available at early boot inside the guest.
+#
+# Host-side VFIO setup (driver blacklisting, IOMMU, initramfs) is handled
+# by `spx admin gpu setup` — this script is for the guest VM image only.
+#
+# Must run inside an Ubuntu 26.04 chroot with /proc /sys /dev bind-mounted.
+
+export DEBIAN_FRONTEND=noninteractive
+
+apt-get update -qq
+apt-get install -y --no-install-recommends curl gnupg ca-certificates
+
+# AMD ROCm apt repository — rocminfo and rocm-smi-lib are not in the Ubuntu
+# default repos, so we add AMD's signed repo before the main package install.
+UBUNTU_CODENAME=$(. /etc/os-release && echo "${UBUNTU_CODENAME:-${VERSION_CODENAME}}")
+mkdir -p /etc/apt/keyrings
+curl -fsSL https://repo.radeon.com/rocm/rocm.gpg.key \
+    | gpg --dearmor -o /etc/apt/keyrings/rocm.gpg
+echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/6.3 ${UBUNTU_CODENAME} main" \
+    > /etc/apt/sources.list.d/rocm.list
+apt-get update -qq
+
+# linux-firmware carries the AMD GPU firmware blobs loaded by the amdgpu
+# kernel module at boot. Without them the GPU is invisible to guest userland.
+apt-get install -y -o Acquire::Retries=3 --no-install-recommends \
+    linux-image-generic \
+    linux-headers-generic \
+    linux-firmware \
+    initramfs-tools \
+    pciutils \
+    rocminfo rocm-smi-lib \
+    python3 python3-venv python3-pip \
+    git curl wget htop tmux \
+    ffmpeg libgl1 libglib2.0-0
+
+KVER=$(ls /boot/vmlinuz-* 2>/dev/null | sort -V | tail -1 | sed 's|/boot/vmlinuz-||')
+if [ -z "${KVER}" ]; then
+    echo "ERROR: No kernel found under /boot"
+    exit 1
+fi
+echo "Rebuilding initramfs for kernel: ${KVER}"
+update-initramfs -u -k "${KVER}"
+
+# Prevent unattended-upgrades from replacing linux-firmware mid-lifecycle;
+# a firmware change without a matching initramfs rebuild can make the GPU invisible.
+mkdir -p /etc/apt/apt.conf.d
+cat > /etc/apt/apt.conf.d/99-gpu-ami <<'EOF'
+Unattended-Upgrade::Package-Blacklist {
+    "linux-firmware";
+    "linux-image-";
+    "linux-headers-";
+};
+EOF
+
+echo "AMD GPU image setup complete: kernel=${KVER}, linux-firmware and ROCm userland installed"

scripts/images/ubuntu-gpu-amd.conf

@@ -0,0 +1,12 @@
+IMAGE_NAME="ubuntu-amd-gpu"
+IMAGE_DESCRIPTION="Ubuntu 26.04 AMD GPU guest image — linux-firmware, ROCm CLI, Python toolchain"
+DISTRO="ubuntu"
+UBUNTU_VERSION="26.04"
+UBUNTU_CODENAME="resolute"
+
+# 12G provides headroom for kernel headers, linux-firmware, and ROCm packages
+IMAGE_SIZE="12G"
+
+SETUP_SCRIPT="scripts/images/ubuntu-gpu-amd-setup.sh"
+
+SYSTEM_TAG="gpu-vendor=amd"

scripts/images/ubuntu-gpu-nvidia-setup.sh

@@ -0,0 +1,107 @@
+#!/bin/bash
+set -euo pipefail
+
+# ubuntu-gpu-nvidia-setup.sh — Chroot setup for an NVIDIA GPU-capable guest image.
+#
+# Pre-builds the NVIDIA headless server driver via DKMS against a pinned kernel
+# so guests can use the GPU immediately after launch. Also installs Python
+# toolchain and common utilities.
+#
+# nouveau is blacklisted inside the guest image so the NVIDIA kernel module can
+# bind — this is distinct from the host-side blacklist that `spx admin gpu setup`
+# configures for VFIO passthrough.
+#
+# Host-side VFIO setup (IOMMU, vfio-pci binding, host initramfs) is handled
+# by `spx admin gpu setup` — this script is for the guest VM image only.
+#
+# Must run inside an Ubuntu 26.04 chroot with /proc /sys /dev bind-mounted
+# (required by DKMS during the nvidia package post-install hook).
+
+export DEBIAN_FRONTEND=noninteractive
+
+apt-get update -qq
+
+# Detect the cloud image's existing kernel BEFORE installing anything.
+# Installing linux-image-generic would upgrade to a newer kernel version;
+# DKMS would then build for that version but the VM boots the original
+# cloud image kernel, causing a version mismatch at modprobe time.
+KVER=$(ls /boot/vmlinuz-* 2>/dev/null | sort -V | tail -1 | sed 's|/boot/vmlinuz-||')
+if [[ -z "${KVER}" ]]; then
+    echo "ERROR: No kernel found under /boot"
+    exit 1
+fi
+echo "Target kernel: ${KVER}"
+
+# Install headers for the exact kernel already present, plus initramfs-tools.
+apt-get install -y --no-install-recommends \
+    "linux-headers-${KVER}" \
+    initramfs-tools
+
+# Install the NVIDIA server DKMS driver. Use the distro default version (no
+# hardcoded suffix) so Ubuntu 26.04's 580 driver is used instead of 550.
+# nvidia-dkms-server is a meta package that resolves to the current recommended
+# version; nvidia-utils-server brings nvidia-smi and related tools.
+apt-get install -y --no-install-recommends \
+    nvidia-dkms-server \
+    nvidia-utils-server
+
+# Detect the installed NVIDIA DKMS module name + version for explicit build.
+NVIDIA_DKMS_NAME=$(dkms status 2>/dev/null | awk -F'[,/ ]+' '/nvidia/{print $1; exit}')
+NVIDIA_DKMS_VER=$(dkms status 2>/dev/null  | awk -F'[,/ ]+' '/nvidia/{print $2; exit}')
+
+if [[ -n "${NVIDIA_DKMS_NAME}" && -n "${NVIDIA_DKMS_VER}" ]]; then
+    echo "Building NVIDIA DKMS module ${NVIDIA_DKMS_NAME}/${NVIDIA_DKMS_VER} for kernel ${KVER}..."
+    dkms install -m "${NVIDIA_DKMS_NAME}" -v "${NVIDIA_DKMS_VER}" --kernelver "${KVER}" --force 2>&1
+else
+    echo "WARNING: nvidia DKMS module not found in dkms status — driver install may have failed"
+fi
+
+# Verify the module was actually built.
+NVIDIA_KO=$(find "/lib/modules/${KVER}/updates" -name "nvidia.ko*" 2>/dev/null | head -1)
+if [[ -z "${NVIDIA_KO}" ]]; then
+    echo "ERROR: nvidia.ko not found under /lib/modules/${KVER}/updates after DKMS build"
+    exit 1
+fi
+echo "NVIDIA kernel module: ${NVIDIA_KO}"
+
+# Pin the exact kernel + headers so unattended-upgrades cannot replace the
+# kernel the DKMS module was built for.
+apt-mark hold \
+    "linux-image-${KVER}" \
+    "linux-headers-${KVER}"
+
+NVIDIA_PKGS=$(dpkg --get-selections 'nvidia-*' 2>/dev/null | awk '/install$/{print $1}' || true)
+LIBNVIDIA_PKGS=$(dpkg --get-selections 'libnvidia-*' 2>/dev/null | awk '/install$/{print $1}' || true)
+# shellcheck disable=SC2086
+[[ -n "${NVIDIA_PKGS}" ]]    && apt-mark hold ${NVIDIA_PKGS}
+# shellcheck disable=SC2086
+[[ -n "${LIBNVIDIA_PKGS}" ]] && apt-mark hold ${LIBNVIDIA_PKGS}
+
+# Blacklist nouveau inside the guest — nvidia.ko and nouveau conflict, and
+# without this blacklist the NVIDIA driver will not bind after boot.
+cat > /etc/modprobe.d/blacklist-nouveau.conf <<'EOF'
+blacklist nouveau
+options nouveau modeset=0
+EOF
+
+# Common tooling matching the AMD image.
+apt-get install -y -o Acquire::Retries=3 \
+    pciutils \
+    python3 python3-venv python3-pip \
+    git curl wget htop tmux \
+    ffmpeg libgl1 libglib2.0-0
+
+mkdir -p /etc/apt/apt.conf.d
+cat > /etc/apt/apt.conf.d/99-gpu-ami <<'EOF'
+Unattended-Upgrade::Package-Blacklist {
+    "linux-";
+    "nvidia-";
+    "libnvidia-";
+};
+EOF
+
+# Rebuild initramfs with nouveau blacklisted and NVIDIA module included.
+update-initramfs -u -k "${KVER}"
+
+
+echo "NVIDIA GPU image setup complete: kernel=${KVER}, NVIDIA driver + DKMS pre-built"

scripts/images/ubuntu-gpu-nvidia.conf

@@ -0,0 +1,12 @@
+IMAGE_NAME="ubuntu-nvidia-gpu"
+IMAGE_DESCRIPTION="Ubuntu 26.04 NVIDIA GPU guest image — NVIDIA server driver (DKMS pre-built), Python toolchain"
+DISTRO="ubuntu"
+UBUNTU_VERSION="26.04"
+UBUNTU_CODENAME="resolute"
+
+# 12G provides headroom for kernel headers, NVIDIA driver, and DKMS build artefacts
+IMAGE_SIZE="12G"
+
+SETUP_SCRIPT="scripts/images/ubuntu-gpu-nvidia-setup.sh"
+
+SYSTEM_TAG="gpu-vendor=nvidia"

scripts/images/ubuntu-gpu-setup.sh

@@ -1,82 +0,0 @@
-#!/bin/bash
-set -euo pipefail
-
-# ubuntu-gpu-setup.sh — Chroot setup for Ubuntu 24.04 GPU AMI
-#
-# Installs NVIDIA headless server driver with DKMS module pre-built against
-# a pinned kernel. Blacklists nouveau. Prevents unattended-upgrades from
-# invalidating the pre-built module.
-#
-# Must run inside an Ubuntu 24.04 chroot with /proc /sys /dev bind-mounted
-# (required by DKMS during the nvidia package post-install hook).
-
-export DEBIAN_FRONTEND=noninteractive
-
-apt-get update
-
-# Install a pinned kernel and its headers so DKMS has a stable build target.
-# linux-image-generic and linux-headers-generic pull in the current HWE stack.
-apt-get install -y --no-install-recommends \
-    linux-image-generic \
-    linux-headers-generic
-
-# Capture the installed kernel version before adding the driver.
-KVER=$(ls /boot/vmlinuz-* 2>/dev/null | sort -V | tail -1 | sed 's|/boot/vmlinuz-||')
-if [[ -z "${KVER}" ]]; then
-    echo "ERROR: No kernel found under /boot after linux-image-generic install"
-    exit 1
-fi
-echo "Target kernel: ${KVER}"
-
-# Install the NVIDIA headless server driver.
-# The nvidia-dkms-550-server post-install hook runs `dkms autoinstall`, which
-# builds the module against every kernel found under /lib/modules/ — including
-# ${KVER} whose headers we just installed above.
-apt-get install -y --no-install-recommends \
-    nvidia-headless-550-server \
-    nvidia-utils-550-server
-
-# Verify the DKMS module was registered and built.
-if dkms status 2>/dev/null | grep -q "nvidia"; then
-    echo "NVIDIA DKMS module status: $(dkms status 2>/dev/null | grep nvidia | head -1)"
-else
-    echo "WARNING: nvidia DKMS module not found in dkms status — manual verification required"
-fi
-
-# Pin the kernel so unattended-upgrades cannot replace it and invalidate
-# the pre-built DKMS module.
-apt-mark hold \
-    linux-image-generic \
-    linux-headers-generic \
-    "linux-image-${KVER}" \
-    "linux-headers-${KVER}"
-
-# Pin all installed NVIDIA packages for the same reason.
-NVIDIA_PKGS=$(dpkg --get-selections 'nvidia-*' 2>/dev/null | awk '/install$/{print $1}' || true)
-LIBNVIDIA_PKGS=$(dpkg --get-selections 'libnvidia-*' 2>/dev/null | awk '/install$/{print $1}' || true)
-# shellcheck disable=SC2086
-[[ -n "${NVIDIA_PKGS}" ]]    && apt-mark hold ${NVIDIA_PKGS}
-# shellcheck disable=SC2086
-[[ -n "${LIBNVIDIA_PKGS}" ]] && apt-mark hold ${LIBNVIDIA_PKGS}
-
-# Blacklist nouveau so it does not conflict with the NVIDIA driver on boot.
-cat > /etc/modprobe.d/blacklist-nouveau.conf << 'EOF'
-blacklist nouveau
-options nouveau modeset=0
-EOF
-
-# Prevent unattended-upgrades from updating kernel or NVIDIA packages,
-# which would break the pre-built DKMS module.
-mkdir -p /etc/apt/apt.conf.d
-cat > /etc/apt/apt.conf.d/99-gpu-ami << 'EOF'
-Unattended-Upgrade::Package-Blacklist {
-    "linux-";
-    "nvidia-";
-    "libnvidia-";
-};
-EOF
-
-# Rebuild initramfs with nouveau blacklisted and NVIDIA module included.
-update-initramfs -u -k "${KVER}"
-
-echo "GPU AMI setup complete: kernel=${KVER}, NVIDIA 550 driver installed, DKMS module pre-built"

scripts/images/ubuntu-gpu.conf

@@ -1,12 +0,0 @@
-IMAGE_NAME="ubuntu-gpu"
-IMAGE_DESCRIPTION="Ubuntu 24.04 GPU-ready AMI (NVIDIA A10G / G5 family)"
-DISTRO="ubuntu"
-UBUNTU_VERSION="24.04"
-
-# 12G provides headroom for kernel headers, NVIDIA driver, and DKMS build artefacts
-IMAGE_SIZE="12G"
-
-# NVIDIA driver install and DKMS pre-build are handled in the setup script
-SETUP_SCRIPT="scripts/images/ubuntu-gpu-setup.sh"
-
-SYSTEM_TAG="gpu"

scripts/launch-gpu.sh

@@ -0,0 +1,248 @@
+#!/bin/bash
+set -euo pipefail
+
+export AWS_PROFILE=spinifex
+
+# Resolve the actual user's home — $HOME is /root when run under sudo.
+_USER_HOME=$(getent passwd "${SUDO_USER:-$(id -un)}" | cut -d: -f6)
+SSH_KEY="${SSH_KEY:-${_USER_HOME}/.ssh/spinifex-key}"
+SSH_USER="${SSH_USER:-ec2-user}"
+SSH_TIMEOUT="${SSH_TIMEOUT:-300}"
+
+usage() {
+    cat <<'EOF'
+launch-gpu.sh — Launch a GPU instance from the Spinifex AMD or NVIDIA GPU AMI.
+
+Detects the GPU brand from the specified instance type and selects the matching
+pre-built AMI automatically. Use --amd or --nvidia to override.
+
+Usage:
+  scripts/launch-gpu.sh --type <instance-type> [--disk GB] [--amd|--nvidia]
+
+Options:
+  --type TYPE    EC2 instance type to launch (required)
+  --disk GB      Root disk size in GB (default: 50)
+  --ami  NAME    Override the AMI name to use (default: ubuntu-amd-gpu or ubuntu-nvidia-gpu)
+  --amd          Force AMD GPU AMI  (ubuntu-amd-gpu)
+  --nvidia       Force NVIDIA GPU AMI (ubuntu-nvidia-gpu)
+
+Env overrides:
+  SSH_KEY      Path to SSH private key (default: ~/.ssh/spinifex-key)
+  SSH_USER     SSH user inside guest  (default: ec2-user)
+  SSH_TIMEOUT  Seconds to wait for initial SSH (default: 300)
+EOF
+}
+
+INSTANCE_TYPE=""
+DISK_GB=50
+GPU_BRAND=""
+AMI_OVERRIDE=""
+
+while [[ $# -gt 0 ]]; do
+    case "$1" in
+        --type)   INSTANCE_TYPE="$2"; shift 2 ;;
+        --disk)   DISK_GB="$2";        shift 2 ;;
+        --ami)    AMI_OVERRIDE="$2";   shift 2 ;;
+        --amd)    GPU_BRAND="amd";     shift ;;
+        --nvidia) GPU_BRAND="nvidia";  shift ;;
+        -h|--help) usage; exit 0 ;;
+        *) echo "Unknown option: $1"; usage; exit 1 ;;
+    esac
+done
+
+list_gpu_types() {
+    aws ec2 describe-instance-types \
+        --query 'InstanceTypes[?GpuInfo].[InstanceType,GpuInfo.Gpus[0].Count,GpuInfo.Gpus[0].Manufacturer,GpuInfo.Gpus[0].Name]' \
+        --output table 2>/dev/null || true
+}
+
+if [ -z "$INSTANCE_TYPE" ]; then
+    echo "Error: --type is required."
+    echo ""
+    echo "Available GPU instance types on this node:"
+    list_gpu_types
+    exit 1
+fi
+
+# --- Validate instance type and fetch GPU info in one call ---
+echo "==> Checking instance type: ${INSTANCE_TYPE}"
+INSTANCE_INFO=$(aws ec2 describe-instance-types \
+    --query "InstanceTypes[?InstanceType=='${INSTANCE_TYPE}' && GpuInfo].[GpuInfo.Gpus[0].Count,GpuInfo.Gpus[0].Manufacturer,GpuInfo.Gpus[0].Name]" \
+    --output text 2>/dev/null || true)
+
+if [ -z "$INSTANCE_INFO" ] || [ "$INSTANCE_INFO" = "None" ]; then
+    echo "'${INSTANCE_TYPE}' is not available or is not a GPU type on this node."
+    echo ""
+    echo "Available GPU instance types:"
+    list_gpu_types
+    exit 1
+fi
+
+IFS=$'\t' read -r GPU_COUNT GPU_MANUFACTURER GPU_NAME <<< "$INSTANCE_INFO"
+echo "   ${GPU_COUNT}× ${GPU_NAME} (${GPU_MANUFACTURER})"
+
+# --- Resolve GPU brand ---
+if [ -z "$GPU_BRAND" ]; then
+    case "$GPU_MANUFACTURER" in
+        AMD|amd)       GPU_BRAND="amd" ;;
+        NVIDIA|nvidia) GPU_BRAND="nvidia" ;;
+        *)
+            echo "Cannot auto-detect GPU brand from manufacturer '${GPU_MANUFACTURER}'."
+            echo "Use --amd or --nvidia to specify explicitly."
+            exit 1
+            ;;
+    esac
+fi
+
+case "$GPU_BRAND" in
+    amd)    AMI_NAME="ubuntu-amd-gpu" ;;
+    nvidia) AMI_NAME="ubuntu-nvidia-gpu" ;;
+esac
+
+# --- Resolve AMI ---
+AMI_LOOKUP="${AMI_OVERRIDE:-${AMI_NAME}}"
+echo "==> Resolving AMI: ${AMI_LOOKUP}"
+AMI_ID=$(aws ec2 describe-images \
+    --query "Images[?Name=='${AMI_LOOKUP}'].ImageId | [0]" --output text 2>/dev/null || true)
+if [ -z "$AMI_ID" ] || [ "$AMI_ID" = "None" ]; then
+    echo "No AMI found with name '${AMI_LOOKUP}'."
+    echo "Build and import it first, or specify the AMI name with --ami <name>"
+    echo "  scripts/build-system-image.sh scripts/images/ubuntu-gpu-${GPU_BRAND}.conf --import"
+    exit 1
+fi
+echo "   $AMI_ID"
+
+# --- Resolve subnet ---
+echo "==> Resolving subnet"
+SUBNET_ID=$(aws ec2 describe-subnets \
+    --query 'Subnets[?MapPublicIpOnLaunch==`true`].SubnetId | [0]' --output text 2>/dev/null || true)
+if [ -z "$SUBNET_ID" ] || [ "$SUBNET_ID" = "None" ]; then
+    echo "No public subnet found"
+    exit 1
+fi
+echo "   $SUBNET_ID"
+
+VPC_ID=$(aws ec2 describe-subnets --subnet-ids "$SUBNET_ID" \
+    --query 'Subnets[0].VpcId' --output text 2>/dev/null || true)
+if [ -z "$VPC_ID" ] || [ "$VPC_ID" = "None" ]; then
+    echo "Could not resolve VPC for subnet $SUBNET_ID"
+    exit 1
+fi
+
+# --- Ensure SSH security group ---
+# The default VPC SG only allows intra-SG traffic; we need a group that
+# permits port 22 from anywhere so the SSH wait below can connect.
+SSH_SG_NAME="spinifex-gpu-ssh"
+echo "==> Ensuring SSH security group (${SSH_SG_NAME})"
+SG_ID=$(aws ec2 describe-security-groups \
+    --filters "Name=group-name,Values=${SSH_SG_NAME}" "Name=vpc-id,Values=${VPC_ID}" \
+    --query 'SecurityGroups[0].GroupId' --output text 2>/dev/null || true)
+if [ -z "$SG_ID" ] || [ "$SG_ID" = "None" ]; then
+    SG_ID=$(aws ec2 create-security-group \
+        --group-name "$SSH_SG_NAME" \
+        --description "Allow SSH access to GPU instances" \
+        --vpc-id "$VPC_ID" \
+        --query 'GroupId' --output text)
+    aws ec2 authorize-security-group-ingress \
+        --group-id "$SG_ID" \
+        --protocol tcp \
+        --port 22 \
+        --cidr 0.0.0.0/0 >/dev/null
+    echo "   Created $SG_ID"
+else
+    echo "   Reusing $SG_ID"
+fi
+
+# --- Ensure SSH key ---
+if [ ! -f "$SSH_KEY" ]; then
+    echo "==> Generating SSH key"
+    sudo -u "${SUDO_USER:-$(id -un)}" mkdir -p "$(dirname "$SSH_KEY")"
+    sudo -u "${SUDO_USER:-$(id -un)}" ssh-keygen -t ed25519 -f "$SSH_KEY" -N ""
+fi
+# Delete before import — stale key pair material causes import to fail
+# silently, leaving the instance unreachable via SSH.
+echo "==> Importing SSH key"
+aws ec2 delete-key-pair --key-name spinifex-key >/dev/null 2>&1 || true
+aws ec2 import-key-pair --key-name spinifex-key \
+    --public-key-material "fileb://${SSH_KEY}.pub" >/dev/null 2>&1 || true
+
+# --- Launch ---
+echo "==> Launching ${INSTANCE_TYPE} (${DISK_GB} GB)..."
+ID=$(aws ec2 run-instances \
+    --image-id "$AMI_ID" \
+    --instance-type "$INSTANCE_TYPE" \
+    --key-name spinifex-key \
+    --subnet-id "$SUBNET_ID" \
+    --security-group-ids "$SG_ID" \
+    --count 1 \
+    --block-device-mappings "[{\"DeviceName\":\"/dev/sda1\",\"Ebs\":{\"VolumeSize\":${DISK_GB},\"VolumeType\":\"gp3\",\"DeleteOnTermination\":true}}]" \
+    --query 'Instances[0].InstanceId' --output text)
+if [ -z "$ID" ] || [ "$ID" = "None" ] || [ "$ID" = "null" ]; then
+    echo "run-instances returned no InstanceId"
+    exit 1
+fi
+echo "   $ID"
+
+# --- Wait for running ---
+echo "==> Waiting for running state..."
+COUNT=0; STATE="unknown"
+while [ $COUNT -lt 120 ]; do
+    STATE=$(aws ec2 describe-instances --instance-ids "$ID" \
+        --query "Reservations[0].Instances[0].State.Name" --output text 2>/dev/null || echo "unknown")
+    [ "$STATE" = "running" ] && break
+    [ "$STATE" = "terminated" ] && { echo "Instance terminated unexpectedly"; exit 1; }
+    sleep 2; COUNT=$((COUNT + 2))
+done
+[ "$STATE" = "running" ] || { echo "Instance did not reach running state (last: $STATE)"; exit 1; }
+
+# --- Wait for public IP ---
+echo "==> Waiting for public IP..."
+IP=""
+for _i in $(seq 1 120); do
+    IP=$(aws ec2 describe-instances --instance-ids "$ID" \
+        --query 'Reservations[0].Instances[0].PublicIpAddress' --output text 2>/dev/null || true)
+    [ -n "$IP" ] && [ "$IP" != "None" ] && [ "$IP" != "null" ] && break
+    sleep 1
+done
+[ -n "$IP" ] && [ "$IP" != "None" ] || { echo "No public IP assigned after 120s"; exit 1; }
+echo "   $IP"
+
+# Clear stale known_hosts entry — instances reuse IPs across runs.
+KNOWN_HOSTS="$HOME/.ssh/known_hosts"
+[ -f "$KNOWN_HOSTS" ] && ssh-keygen -f "$KNOWN_HOSTS" -R "$IP" >/dev/null 2>&1 || true
+
+# --- Wait for SSH ---
+echo "==> Waiting for SSH..."
+_ssh() {
+    ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
+        -o ConnectTimeout=5 -o BatchMode=yes \
+        -i "$SSH_KEY" "${SSH_USER}@${IP}" "$@"
+}
+
+elapsed=0
+while [ "$elapsed" -lt "$SSH_TIMEOUT" ]; do
+    _ssh 'echo ready' >/dev/null 2>&1 && break
+    sleep 5; elapsed=$((elapsed + 5))
+done
+[ "$elapsed" -lt "$SSH_TIMEOUT" ] || { echo "SSH timeout after ${SSH_TIMEOUT}s"; exit 1; }
+
+# --- GPU visibility check ---
+echo "==> Checking GPU visibility..."
+lspci_out=$(_ssh 'lspci -nn 2>/dev/null' || true)
+if printf '%s\n' "$lspci_out" | grep -qiE "1002:|10de:|Instinct|Aqua|Processing accelerator|NVIDIA"; then
+    printf '%s\n' "$lspci_out" \
+        | grep -iE "1002:|10de:|Instinct|Aqua|Processing accelerator|NVIDIA" \
+        | sed 's/^/   ✓ /'
+else
+    echo "   WARNING: GPU not visible in lspci — check host-side passthrough assignment"
+    printf '%s\n' "$lspci_out" | head -20 | sed 's/^/   /'
+fi
+
+# --- Done ---
+echo ""
+echo "✅ ${INSTANCE_TYPE} ready — ${ID} (${IP})"
+echo ""
+echo "   ssh -i ${SSH_KEY} -o StrictHostKeyChecking=no ${SSH_USER}@${IP}"
+echo ""
+echo "   Terminate when done:"
+echo "   AWS_PROFILE=spinifex aws ec2 terminate-instances --instance-ids ${ID}"

scripts/reset-dev-env.sh

@@ -268,10 +268,9 @@ sudo systemctl start spinifex.target
 # Wait for services to start
 sleep 5
 
-# --- Build + import LB image (needs services running) ---
-echo "==> Building and importing LB image"
-cd "$PROJECT_ROOT" && make build-lb-agent
-"$PROJECT_ROOT/scripts/build-system-image.sh" "$PROJECT_ROOT/scripts/images/lb.conf" --import --quiet
+# --- Build + install microVM artifacts (kernel + initramfs for direct-boot LBs) ---
+echo "==> Building and installing microVM artifacts"
+cd "$PROJECT_ROOT" && make build-lb-agent install-microvm
 
 # --- Smoke test ---
 sudo -u "$INVOKING_USER" "$SCRIPT_DIR/smoke-test.sh"

scripts/setup-ovn.sh

@@ -326,7 +326,7 @@ if [ -n "$WAN_BRIDGE" ]; then
                    /etc/systemd/network/15-spinifex-veth-wan.network \
                    /etc/systemd/network/16-spinifex-veth-wan-ovs.network
         sudo networkctl reload 2>/dev/null || true
-        sudo ovs-vsctl --if-exists del-port "$WAN_BRIDGE" veth-wan-ovs
+        sudo ovs-vsctl --if-exists del-port veth-wan-ovs 2>/dev/null || true
         sudo ip link del veth-wan-br 2>/dev/null || true
     fi
 

scripts/setup.sh

@@ -214,7 +214,7 @@ install_apt_deps() {
 
         DEBIAN_FRONTEND=noninteractive $SUDO apt-get install -y -qq \
             nbdkit \
-            $QEMU_PACKAGES qemu-utils qemu-kvm ovmf less \
+            $QEMU_PACKAGES qemu-utils qemu-kvm ovmf qemu-efi-aarch64 less \
             libvirt-daemon-system libvirt-clients \
             pciutils \
             jq curl iproute2 netcat-openbsd wget unzip xz-utils file \
@@ -346,6 +346,13 @@ install_files() {
         $SUDO install -m 0755 "$EXTRACT_DIR/setup.sh" /usr/local/share/spinifex/setup.sh
         info "  /usr/local/share/spinifex/setup.sh"
     fi
+
+    # microVM kernel + initramfs
+    $SUDO install -d /usr/share/spinifex/microvm
+    if [ -d "$EXTRACT_DIR/microvm" ]; then
+        $SUDO cp "$EXTRACT_DIR/microvm/"* /usr/share/spinifex/microvm/
+        info "  /usr/share/spinifex/microvm/*"
+    fi
 }
 
 # --- Create directories ---
@@ -380,11 +387,14 @@ create_directories() {
     # /run is tmpfs — direct mkdir doesn't survive reboot, and units have
     # ReadWritePaths= on these paths which fails namespace setup with ENOENT
     # if the dirs are absent at service start.
-    $SUDO install -m 0644 /dev/stdin /etc/tmpfiles.d/spinifex.conf <<'TMPEOF'
+    _tmpf=$(mktemp)
+    cat > "$_tmpf" <<'TMPEOF'
 # Type  Path               Mode  User                 Group                Age
 d       /run/spinifex      0770  root                 spinifex             -
 d       /run/spinifex/nbd  0770  spinifex-viperblock  spinifex-viperblock  -
 TMPEOF
+    $SUDO install -m 0644 "$_tmpf" /etc/tmpfiles.d/spinifex.conf
+    rm -f "$_tmpf"
     if [ "${ISO_BUILD:-0}" != "1" ]; then
         # Live systems: materialise the runtime dirs immediately so a re-run of
         # setup.sh on an existing host has correct /run state without a reboot.

scripts/smoke-test.sh

@@ -25,6 +25,12 @@ GPU_FAMILY="${GPU_FAMILY:-g5}"
 
 export AWS_PROFILE=spinifex
 
+# AWS commands run as the invoking user so their ~/.aws/config (spinifex profile
+# + CA bundle) is used when the script is called via sudo.
+INVOKING_USER="${SUDO_USER:-$(id -un)}"
+INVOKING_HOME=$(getent passwd "$INVOKING_USER" | cut -d: -f6)
+aws_as_user() { sudo -u "$INVOKING_USER" env HOME="$INVOKING_HOME" AWS_PROFILE=spinifex aws "$@"; }
+
 # --- Wait for EC2 daemon to subscribe to NATS ---
 # Port 3000 (UI) becomes ready before the daemon finishes initialising.
 # Poll describe-key-pairs until it doesn't return InternalError.
@@ -32,7 +38,7 @@ echo "==> Waiting for EC2 daemon to be ready"
 DAEMON_TIMEOUT=60
 DAEMON_ELAPSED=0
 while [ $DAEMON_ELAPSED -lt $DAEMON_TIMEOUT ]; do
-    if aws ec2 describe-key-pairs --output text >/dev/null 2>&1; then
+    if aws_as_user ec2 describe-key-pairs --output text >/dev/null 2>&1; then
         break
     fi
     sleep 2
@@ -53,9 +59,9 @@ if [ ! -f "$SSH_KEY.pub" ]; then
 fi
 
 echo "==> Importing SSH key"
-! aws ec2 import-key-pair --key-name spinifex-key \
-    --public-key-material "fileb://$SSH_KEY.pub"
-aws ec2 describe-key-pairs
+aws_as_user ec2 import-key-pair --key-name spinifex-key \
+    --public-key-material "fileb://$SSH_KEY.pub" || true
+aws_as_user ec2 describe-key-pairs
 
 # --- GPU pre-flight (TEST_GPU=1 only) ---
 if [[ "$TEST_GPU" == "1" ]]; then
@@ -82,7 +88,7 @@ if [[ "$TEST_GPU" == "1" ]]; then
     echo "   GPU PCI ID: ${GPU_VENDOR_ID}:${GPU_DEVICE_ID}"
 
     echo "==> [GPU] Verifying ${GPU_FAMILY} instance types advertised"
-    G5_TYPES=$(aws ec2 describe-instance-types \
+    G5_TYPES=$(aws_as_user ec2 describe-instance-types \
         --query "InstanceTypes[?starts_with(InstanceType, \`${GPU_FAMILY}\`)].InstanceType" \
         --output text 2>/dev/null || true)
     if [[ -z "$G5_TYPES" ]]; then
@@ -110,7 +116,7 @@ esac
 
 AMI_NAME="ami-${IMAGE_NAME}"
 
-EXISTING_AMI_ID=$(aws ec2 describe-images \
+EXISTING_AMI_ID=$(aws_as_user ec2 describe-images \
     --query "Images[?Name=='${AMI_NAME}'] | [0].ImageId" \
     --output text)
 
@@ -120,7 +126,7 @@ else
     if [ -f "$LOCAL_IMAGE" ]; then
         echo "  Using local image: $LOCAL_IMAGE"
         sudo /usr/local/bin/spx admin images import \
-            --file "$LOCAL_IMAGE" --distro ubuntu --version 26.04 --arch "$IMG_ARCH"
+            --file "$LOCAL_IMAGE" --distro ubuntu --version 26.04 --arch "$IMG_ARCH" --boot-mode uefi
     else
         echo "  Downloading image: $IMAGE_NAME"
         sudo /usr/local/bin/spx admin images import --name "$IMAGE_NAME"
@@ -137,7 +143,7 @@ else
     INSTANCE_TYPE="t3.small"
 fi
 
-AMI_ID=$(aws ec2 describe-images \
+AMI_ID=$(aws_as_user ec2 describe-images \
     --query "Images[?Name=='${AMI_NAME}'] | [0].ImageId" \
     --output text)
 
@@ -146,7 +152,7 @@ if [ -z "$AMI_ID" ] || [ "$AMI_ID" = "None" ]; then
     exit 1
 fi
 
-SUBNET_ID=$(aws ec2 describe-subnets --query 'Subnets[?MapPublicIpOnLaunch==`true`].SubnetId | [0]' --output text)
+SUBNET_ID=$(aws_as_user ec2 describe-subnets --query 'Subnets[?MapPublicIpOnLaunch==`true`].SubnetId | [0]' --output text)
 if [ -z "$SUBNET_ID" ] || [ "$SUBNET_ID" = "None" ]; then
     echo "❌ No subnet found"
     exit 1
@@ -154,7 +160,7 @@ fi
 
 echo "  AMI: $AMI_ID  type: $INSTANCE_TYPE  subnet: $SUBNET_ID"
 
-INSTANCE_ID=$(aws ec2 run-instances \
+INSTANCE_ID=$(aws_as_user ec2 run-instances \
     --image-id "$AMI_ID" \
     --instance-type "$INSTANCE_TYPE" \
     --key-name spinifex-key \
@@ -173,7 +179,7 @@ echo "==> Waiting for instance to reach running state"
 COUNT=0
 STATE="unknown"
 while [ $COUNT -lt 60 ]; do
-    DESCRIBE=$(aws ec2 describe-instances --instance-ids "$INSTANCE_ID" 2>/dev/null) || {
+    DESCRIBE=$(aws_as_user ec2 describe-instances --instance-ids "$INSTANCE_ID" 2>/dev/null) || {
         sleep 2; COUNT=$((COUNT + 1)); continue
     }
     STATE=$(echo "$DESCRIBE" | jq -r '.Reservations[0].Instances[0].State.Name // "not-found"')
@@ -196,7 +202,7 @@ echo "==> Waiting for public IP assignment (up to 300s)"
 SSH_INST_PORT=22
 SSH_INST_HOST=""
 for _i in $(seq 1 300); do
-    _ip=$(aws ec2 describe-instances --instance-ids "$INSTANCE_ID" \
+    _ip=$(aws_as_user ec2 describe-instances --instance-ids "$INSTANCE_ID" \
         --query 'Reservations[0].Instances[0].PublicIpAddress' --output text 2>/dev/null || true)
     if [ -n "$_ip" ] && [ "$_ip" != "None" ] && [ "$_ip" != "null" ]; then
         SSH_INST_HOST="$_ip"
@@ -256,14 +262,14 @@ if [[ "$TEST_GPU" == "1" ]]; then
                -o ConnectTimeout=5 -o BatchMode=yes \
                -p "$SSH_INST_PORT" -i "$SSH_KEY" \
                ec2-user@"$SSH_INST_HOST" \
-               'lspci 2>/dev/null | grep -i nvidia' 2>/dev/null; then
+               'lspci 2>/dev/null | grep -iE "nvidia|amdgpu|instinct"' 2>/dev/null; then
             GPU_VISIBLE=1
             break
         fi
         sleep 5
     done
     if [[ $GPU_VISIBLE -eq 0 ]]; then
-        echo "❌ NVIDIA GPU not visible in guest via lspci"
+        echo "❌ GPU not visible in guest via lspci"
         exit 1
     fi
     echo "   GPU visible in guest"

spinifex/admin/admin.go

@@ -886,8 +886,8 @@ func SetupAWSCredentials(accessKey, secretKey, region, certPath, bindIP, wanIP s
 // multi-node Predastore cluster. Each node gets its own DB entry (port 6660)
 // and shard entry (port 9991) on a distinct IP. Node ID 1 is the bootstrap leader.
 func GenerateMultiNodePredastoreConfig(templateStr string, nodes []PredastoreNodeConfig, accessKey, secretKey, region, natsToken, configDir, bindIP string) (string, error) {
-	if len(nodes) < 3 {
-		return "", fmt.Errorf("multi-node predastore requires at least 3 nodes, got %d", len(nodes))
+	if len(nodes) < 2 {
+		return "", fmt.Errorf("multi-node predastore requires at least 2 nodes, got %d", len(nodes))
 	}
 
 	data := struct {

spinifex/admin/admin_test.go

@@ -738,10 +738,9 @@ func TestGenerateMultiNodePredastoreConfig_MinimumNodes(t *testing.T) {
 
 	_, err := GenerateMultiNodePredastoreConfig(tmpl, []PredastoreNodeConfig{
 		{ID: 1, Host: "10.0.0.1"},
-		{ID: 2, Host: "10.0.0.2"},
 	}, "AK", "SK", "us-east-1", "nats-token", "/config", "10.0.0.1")
 	assert.Error(t, err)
-	assert.Contains(t, err.Error(), "at least 3 nodes")
+	assert.Contains(t, err.Error(), "at least 2 nodes")
 }
 
 func TestGenerateMultiNodePredastoreConfig_InvalidTemplate(t *testing.T) {

spinifex/admin/images_remove.go

@@ -0,0 +1,479 @@
+package admin
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"log/slog"
+	"strings"
+	"time"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/service/s3"
+	"github.com/mulgadc/spinifex/spinifex/awserrors"
+	handlers_ec2_image "github.com/mulgadc/spinifex/spinifex/handlers/ec2/image"
+	handlers_ec2_snapshot "github.com/mulgadc/spinifex/spinifex/handlers/ec2/snapshot"
+	"github.com/mulgadc/spinifex/spinifex/objectstore"
+	"github.com/mulgadc/spinifex/spinifex/utils"
+	"github.com/mulgadc/viperblock/viperblock"
+)
+
+// RemoveImageOpts configures RemoveSystemImage.
+type RemoveImageOpts struct {
+	ImageID string
+	// Force bypasses the dependency check, the ownership-shape check, and
+	// the "config.json missing/corrupt" check. Salvage-mode lever.
+	Force bool
+}
+
+// RemoveImageResult summarises what was deleted from object storage.
+type RemoveImageResult struct {
+	ObjectsDeleted int
+	BytesFreed     int64
+}
+
+// Dependents lists every resource that transitively backs an admin-imported
+// AMI. Removing the AMI while any of these exist would corrupt the dependent.
+type Dependents struct {
+	Snapshots []string // EC2 snapshots whose VolumeID == the AMI ID (i.e. CopyImage-derived snaps)
+	Volumes   []string // Volumes whose SnapshotID is snap-ami-<id> or a derived snap
+	AMIs      []string // AMIs whose SnapshotID is a derived snap (CopyImage of a system AMI then re-copy)
+}
+
+// Empty is true when nothing depends on the AMI.
+func (d Dependents) Empty() bool {
+	return len(d.Snapshots) == 0 && len(d.Volumes) == 0 && len(d.AMIs) == 0
+}
+
+// RemovePreview captures everything the CLI needs to render the confirmation
+// prompt and decide whether to proceed: AMI metadata (or zero values when the
+// config is missing/corrupt), backing-storage byte counts, and the dependent
+// resource set. PreviewRemoveSystemImage performs no deletions.
+type RemovePreview struct {
+	ImageID       string
+	Name          string
+	Owner         string
+	Created       time.Time
+	ConfigPresent bool // false when ami-<id>/config.json is missing
+	ConfigCorrupt bool // true when config.json exists but is undecodable
+	IsSystemOwned bool // ImageOwnerAlias is set and not an account ID
+
+	AMIObjectCount  int
+	AMIBytesTotal   int64
+	SnapObjectCount int
+	SnapBytesTotal  int64
+
+	Dependents Dependents
+}
+
+// SnapPrefix returns the viperblock-internal snapshot prefix backing an
+// admin-imported AMI. v_utils.ImportDiskImage writes the snapshot as
+// "snap-<volumeName>" where the volume name IS the AMI ID.
+func SnapPrefix(imageID string) string {
+	return "snap-" + imageID
+}
+
+// PreviewRemoveSystemImage walks the object store and gathers everything the
+// caller needs to decide whether to remove an admin-imported AMI. It performs
+// no mutations. Returned errors indicate unexpected I/O failures only —
+// AMI-not-found, corrupt config, and account-owned AMIs are reflected in the
+// preview fields, not as errors.
+func PreviewRemoveSystemImage(store objectstore.ObjectStore, bucket, imageID string) (*RemovePreview, error) {
+	if !strings.HasPrefix(imageID, "ami-") {
+		return nil, errors.New(awserrors.ErrorInvalidAMIIDMalformed)
+	}
+
+	preview := &RemovePreview{ImageID: imageID}
+
+	meta, configErr := readAMIConfig(store, bucket, imageID)
+	switch {
+	case configErr == nil:
+		preview.ConfigPresent = true
+		preview.Name = meta.Name
+		preview.Owner = meta.ImageOwnerAlias
+		preview.Created = meta.CreationDate
+		preview.IsSystemOwned = meta.ImageOwnerAlias != "" && !utils.IsAccountID(meta.ImageOwnerAlias)
+	case objectstore.IsNoSuchKeyError(configErr):
+		// Config absent — salvage candidate. Leave ConfigPresent=false.
+	case errors.Is(configErr, handlers_ec2_image.ErrCorruptAMIConfig):
+		preview.ConfigCorrupt = true
+	default:
+		return nil, fmt.Errorf("preview: read AMI config: %w", configErr)
+	}
+
+	amiCount, amiBytes, err := sumPrefix(store, bucket, imageID+"/")
+	if err != nil {
+		return nil, fmt.Errorf("preview: sum ami prefix: %w", err)
+	}
+	preview.AMIObjectCount = amiCount
+	preview.AMIBytesTotal = amiBytes
+
+	snapCount, snapBytes, err := sumPrefix(store, bucket, SnapPrefix(imageID)+"/")
+	if err != nil {
+		return nil, fmt.Errorf("preview: sum snap prefix: %w", err)
+	}
+	preview.SnapObjectCount = snapCount
+	preview.SnapBytesTotal = snapBytes
+
+	deps, err := FindAMIDependents(store, bucket, imageID)
+	if err != nil {
+		return nil, fmt.Errorf("preview: find dependents: %w", err)
+	}
+	preview.Dependents = deps
+
+	return preview, nil
+}
+
+// FindAMIDependents walks the bucket to find every resource that transitively
+// depends on the given admin-imported AMI:
+//
+//  1. snap-*/metadata.json whose VolumeID == imageID  → derivedSnaps
+//     (these are CopyImage-created snapshots that point back at the system AMI)
+//  2. vol-*/config.json whose VolumeMetadata.SnapshotID ∈ {snap-ami-<id>} ∪ derivedSnaps
+//  3. ami-*/config.json whose AMIMetadata.SnapshotID ∈ derivedSnaps
+//
+// The walk terminates one hop deep. CopyImage of an account AMI uses its own
+// VolumeID, not the system AMI's, so further recursion is not required today.
+func FindAMIDependents(store objectstore.ObjectStore, bucket, imageID string) (Dependents, error) {
+	var deps Dependents
+
+	prefixes, err := listCommonPrefixes(store, bucket)
+	if err != nil {
+		return Dependents{}, fmt.Errorf("list bucket prefixes: %w", err)
+	}
+
+	// Pass 1: collect derived snaps (CopyImage of this AMI writes a snap whose
+	// VolumeID points back at the source AMI ID).
+	derived := map[string]bool{}
+	for _, p := range prefixes {
+		if !strings.HasPrefix(p, "snap-") {
+			continue
+		}
+		snapID := strings.TrimSuffix(p, "/")
+		// Skip the viperblock-internal snap prefix for this AMI — it has no
+		// metadata.json (it was written by viperblock.CreateSnapshot, not by
+		// the EC2 snapshot service).
+		if snapID == SnapPrefix(imageID) {
+			continue
+		}
+		cfg, err := handlers_ec2_snapshot.ReadSnapshotConfig(store, bucket, snapID)
+		if err != nil {
+			if objectstore.IsNoSuchKeyError(err) || errors.Is(err, handlers_ec2_snapshot.ErrCorruptSnapshotMetadata) {
+				continue
+			}
+			return Dependents{}, fmt.Errorf("read snapshot %s: %w", snapID, err)
+		}
+		if cfg.VolumeID == imageID {
+			derived[snapID] = true
+			deps.Snapshots = append(deps.Snapshots, snapID)
+		}
+	}
+
+	// The set of snap IDs that a dependent volume might reference: the
+	// admin-import's internal snap plus every CopyImage-derived snap.
+	volSnapRefs := map[string]bool{SnapPrefix(imageID): true}
+	for s := range derived {
+		volSnapRefs[s] = true
+	}
+
+	// Pass 2: volumes (vol-*/config.json) whose SnapshotID matches.
+	for _, p := range prefixes {
+		if !strings.HasPrefix(p, "vol-") {
+			continue
+		}
+		volID := strings.TrimSuffix(p, "/")
+		cfg, err := readVolumeConfig(store, bucket, volID)
+		if err != nil {
+			if objectstore.IsNoSuchKeyError(err) || errors.Is(err, errCorruptVolumeConfig) {
+				continue
+			}
+			return Dependents{}, fmt.Errorf("read volume %s: %w", volID, err)
+		}
+		if volSnapRefs[cfg.VolumeMetadata.SnapshotID] {
+			deps.Volumes = append(deps.Volumes, volID)
+		}
+	}
+
+	// Pass 3: AMIs (ami-*/config.json) whose SnapshotID is a derived snap.
+	// Skip the target AMI itself.
+	for _, p := range prefixes {
+		if !strings.HasPrefix(p, "ami-") {
+			continue
+		}
+		otherAMI := strings.TrimSuffix(p, "/")
+		if otherAMI == imageID {
+			continue
+		}
+		meta, err := readAMIConfig(store, bucket, otherAMI)
+		if err != nil {
+			if objectstore.IsNoSuchKeyError(err) || errors.Is(err, handlers_ec2_image.ErrCorruptAMIConfig) {
+				continue
+			}
+			return Dependents{}, fmt.Errorf("read AMI %s: %w", otherAMI, err)
+		}
+		if derived[meta.SnapshotID] {
+			deps.AMIs = append(deps.AMIs, otherAMI)
+		}
+	}
+
+	return deps, nil
+}
+
+// RemoveSystemImage deletes an admin-imported AMI and its backing storage
+// after re-validating the preconditions:
+//
+//  1. ami-<id>/config.json exists and parses (skipped under --force = salvage)
+//  2. AMI is system-owned, not account-owned (--force bypasses)
+//  3. No volumes / snapshots / AMIs depend on it (--force bypasses)
+//
+// Deletion order is intentional: the config.json is deleted FIRST so the AMI
+// disappears from DescribeImages immediately, preventing new RunInstances
+// from racing the block cleanup.
+func RemoveSystemImage(store objectstore.ObjectStore, bucket string, opts RemoveImageOpts) (*RemoveImageResult, error) {
+	if !strings.HasPrefix(opts.ImageID, "ami-") {
+		return nil, errors.New(awserrors.ErrorInvalidAMIIDMalformed)
+	}
+
+	meta, configErr := readAMIConfig(store, bucket, opts.ImageID)
+	configMissing := objectstore.IsNoSuchKeyError(configErr)
+	configCorrupt := errors.Is(configErr, handlers_ec2_image.ErrCorruptAMIConfig)
+	switch {
+	case configErr == nil:
+		// fine
+	case configMissing, configCorrupt:
+		if !opts.Force {
+			return nil, errors.New(awserrors.ErrorInvalidAMIIDNotFound)
+		}
+	default:
+		slog.Error("RemoveSystemImage: read AMI config", "imageId", opts.ImageID, "err", configErr)
+		return nil, errors.New(awserrors.ErrorServerInternal)
+	}
+
+	if configErr == nil && !opts.Force {
+		if meta.ImageOwnerAlias != "" && utils.IsAccountID(meta.ImageOwnerAlias) {
+			return nil, fmt.Errorf("%s is account-owned (%s); use `aws ec2 deregister-image` followed by `aws ec2 delete-snapshot`",
+				opts.ImageID, meta.ImageOwnerAlias)
+		}
+	}
+
+	if !opts.Force {
+		deps, err := FindAMIDependents(store, bucket, opts.ImageID)
+		if err != nil {
+			slog.Error("RemoveSystemImage: dependency walk", "imageId", opts.ImageID, "err", err)
+			return nil, errors.New(awserrors.ErrorServerInternal)
+		}
+		if !deps.Empty() {
+			return nil, &DependentError{ImageID: opts.ImageID, Dependents: deps}
+		}
+	}
+
+	result := &RemoveImageResult{}
+
+	// Step 1: drop config.json first — the barrier that hides the AMI from
+	// DescribeImages so no new launches can land on the blocks we're deleting.
+	if configErr == nil || (opts.Force && !configMissing) {
+		n, b, err := deletePrefix(store, bucket, opts.ImageID+"/config.json")
+		if err != nil {
+			return nil, fmt.Errorf("delete config: %w", err)
+		}
+		result.ObjectsDeleted += n
+		result.BytesFreed += b
+	}
+
+	// Step 2: drop the rest of ami-<id>/ (chunks, WAL, checkpoints).
+	n, b, err := deletePrefix(store, bucket, opts.ImageID+"/")
+	if err != nil {
+		return nil, fmt.Errorf("delete ami prefix: %w", err)
+	}
+	result.ObjectsDeleted += n
+	result.BytesFreed += b
+
+	// Step 3: drop snap-<amiID>/ (the viperblock-internal snap checkpoint).
+	n, b, err = deletePrefix(store, bucket, SnapPrefix(opts.ImageID)+"/")
+	if err != nil {
+		return nil, fmt.Errorf("delete snap prefix: %w", err)
+	}
+	result.ObjectsDeleted += n
+	result.BytesFreed += b
+
+	slog.Info("RemoveSystemImage completed",
+		"imageId", opts.ImageID,
+		"objectsDeleted", result.ObjectsDeleted,
+		"bytesFreed", result.BytesFreed,
+		"force", opts.Force,
+	)
+	return result, nil
+}
+
+// DependentError is returned by RemoveSystemImage when dependent resources
+// block deletion. The CLI prints the dependents list and exits non-zero.
+type DependentError struct {
+	ImageID    string
+	Dependents Dependents
+}
+
+func (e *DependentError) Error() string {
+	return fmt.Sprintf("refusing to remove %s: %d volumes, %d snapshots, %d AMIs depend on this image",
+		e.ImageID, len(e.Dependents.Volumes), len(e.Dependents.Snapshots), len(e.Dependents.AMIs))
+}
+
+// readAMIConfig reads ami-<id>/config.json and returns the AMIMetadata.
+// Mirrors ImageServiceImpl.GetAMIConfig but operates package-locally so the
+// admin tooling doesn't require an ImageServiceImpl (which carries NATS).
+func readAMIConfig(store objectstore.ObjectStore, bucket, imageID string) (viperblock.AMIMetadata, error) {
+	key := imageID + "/config.json"
+	res, err := store.GetObject(&s3.GetObjectInput{
+		Bucket: aws.String(bucket),
+		Key:    aws.String(key),
+	})
+	if err != nil {
+		return viperblock.AMIMetadata{}, err
+	}
+	defer res.Body.Close()
+
+	body, err := io.ReadAll(res.Body)
+	if err != nil {
+		return viperblock.AMIMetadata{}, err
+	}
+
+	var state viperblock.VBState
+	if err := json.Unmarshal(body, &state); err != nil {
+		return viperblock.AMIMetadata{}, fmt.Errorf("%w: %s: %v", handlers_ec2_image.ErrCorruptAMIConfig, key, err)
+	}
+	return state.VolumeConfig.AMIMetadata, nil
+}
+
+// errCorruptVolumeConfig lets the dependency walk distinguish a vol-*/config.json
+// that can't be parsed (recovery is operator-owned, walk continues) from a
+// transient transport error (walk must fail closed — silently dropping a
+// dependent could lead to deleting blocks underneath a live volume).
+var errCorruptVolumeConfig = errors.New("corrupt volume config")
+
+// readVolumeConfig reads vol-<id>/config.json into VolumeConfig.
+type volumeConfigWrapper struct {
+	VolumeConfig viperblock.VolumeConfig `json:"VolumeConfig"`
+}
+
+func readVolumeConfig(store objectstore.ObjectStore, bucket, volumeID string) (*viperblock.VolumeConfig, error) {
+	key := volumeID + "/config.json"
+	res, err := store.GetObject(&s3.GetObjectInput{
+		Bucket: aws.String(bucket),
+		Key:    aws.String(key),
+	})
+	if err != nil {
+		return nil, err
+	}
+	defer res.Body.Close()
+
+	body, err := io.ReadAll(res.Body)
+	if err != nil {
+		return nil, err
+	}
+
+	var w volumeConfigWrapper
+	if err := json.Unmarshal(body, &w); err != nil {
+		return nil, fmt.Errorf("%w: %s: %v", errCorruptVolumeConfig, key, err)
+	}
+	return &w.VolumeConfig, nil
+}
+
+// listCommonPrefixes returns the top-level "directory" prefixes in the bucket
+// (e.g. "ami-xxx/", "vol-yyy/", "snap-zzz/"), exhausting any pagination.
+func listCommonPrefixes(store objectstore.ObjectStore, bucket string) ([]string, error) {
+	seen := map[string]bool{}
+	var token *string
+	for {
+		out, err := store.ListObjectsV2(&s3.ListObjectsV2Input{
+			Bucket:            aws.String(bucket),
+			Delimiter:         aws.String("/"),
+			ContinuationToken: token,
+		})
+		if err != nil {
+			return nil, err
+		}
+		for _, cp := range out.CommonPrefixes {
+			if cp.Prefix == nil {
+				continue
+			}
+			seen[*cp.Prefix] = true
+		}
+		if !aws.BoolValue(out.IsTruncated) {
+			break
+		}
+		token = out.NextContinuationToken
+	}
+	prefixes := make([]string, 0, len(seen))
+	for p := range seen {
+		prefixes = append(prefixes, p)
+	}
+	return prefixes, nil
+}
+
+// sumPrefix returns (object count, total bytes) for every object under prefix.
+// Used by the preview to surface what an operator is about to delete.
+func sumPrefix(store objectstore.ObjectStore, bucket, prefix string) (int, int64, error) {
+	var count int
+	var bytes int64
+	var token *string
+	for {
+		out, err := store.ListObjectsV2(&s3.ListObjectsV2Input{
+			Bucket:            aws.String(bucket),
+			Prefix:            aws.String(prefix),
+			ContinuationToken: token,
+		})
+		if err != nil {
+			return 0, 0, err
+		}
+		for _, obj := range out.Contents {
+			count++
+			if obj.Size != nil {
+				bytes += *obj.Size
+			}
+		}
+		if !aws.BoolValue(out.IsTruncated) {
+			break
+		}
+		token = out.NextContinuationToken
+	}
+	return count, bytes, nil
+}
+
+// deletePrefix removes every object under prefix and returns count+bytes
+// deleted. Used by RemoveSystemImage for both the single-key config.json
+// barrier and the bulk ami-<id>/ and snap-<id>/ teardown.
+func deletePrefix(store objectstore.ObjectStore, bucket, prefix string) (int, int64, error) {
+	var count int
+	var bytes int64
+	var token *string
+	for {
+		out, err := store.ListObjectsV2(&s3.ListObjectsV2Input{
+			Bucket:            aws.String(bucket),
+			Prefix:            aws.String(prefix),
+			ContinuationToken: token,
+		})
+		if err != nil {
+			return count, bytes, err
+		}
+		for _, obj := range out.Contents {
+			if obj.Key == nil {
+				continue
+			}
+			if _, err := store.DeleteObject(&s3.DeleteObjectInput{
+				Bucket: aws.String(bucket),
+				Key:    obj.Key,
+			}); err != nil {
+				return count, bytes, fmt.Errorf("delete %s: %w", *obj.Key, err)
+			}
+			count++
+			if obj.Size != nil {
+				bytes += *obj.Size
+			}
+		}
+		if !aws.BoolValue(out.IsTruncated) {
+			break
+		}
+		token = out.NextContinuationToken
+	}
+	return count, bytes, nil
+}