release: update go.mod for v1.5.0

Self-hosted runner accumulated 18GB in overlay2 from leaked container
layers. Two causes: build-microvm-image.sh only removed the alpine
container on the success path, and no workflow step pruned docker
between runs.

- scripts/build-microvm-image.sh: container id now removed via the
  existing EXIT trap, so failures, signals, and runner kills no longer
  leak the writable layer.
- .github/workflows/e2e-go.yml: prune containers / dangling images /
  buildx cache before each Build Install Artifacts step, in all three
  e2e jobs (single, multinode, reboot).

.github/actions/e2e-analyze/README.md

@@ -0,0 +1,104 @@
+# e2e-analyze
+
+Post-test failure-clustering action for the Go E2E suites.
+
+## What it does
+
+Runs after `go-junit-report` has converted each suite's stdout into
+`junit-<suite>.xml`. For every JUnit file matched by `junit-glob`:
+
+1. Parses each `<testcase>` with a `<failure>` / `<error>` child.
+2. Strips ANSI escapes, then normalises dynamic noise (UUIDs, EC2 IDs,
+   IPs, timestamps, request-ids, source line numbers) to produce a stable
+   _signature_ per failure.
+3. Picks the **earliest non-cascade** failure per suite as the likely
+   root cause. "Cascade" = failure message matches a fixture-dependency
+   marker (`must populate fix.`, `Should NOT be empty`,
+   `Expected value not to be nil`). When every failure is a cascade
+   (whole suite tripped on one upstream gap), the earliest cascade is
+   the fallback root.
+4. Buckets the remaining failures by signature and renders a markdown
+   report. The report is appended to `$GITHUB_STEP_SUMMARY` and also
+   written to `<log-dir>/analysis.md` so it travels with the uploaded
+   artifact bundle.
+
+The action **never fails the job** — exit-code semantics stay with the
+existing "Fail job if any suite failed" gate.
+
+## Inputs
+
+| Input        | Required | Description |
+|--------------|----------|-------------|
+| `junit-glob` | yes      | Shell glob for the JUnit XML files (e.g. `${ARTIFACT_DIR}/junit-*.xml`). |
+| `log-dir`    | yes      | Directory the rendered report is written into (`analysis.md`). |
+| `title`      | no       | Heading at the top of the summary section. Default `E2E failure analysis`. |
+
+## Outputs
+
+None. The summary is appended to `$GITHUB_STEP_SUMMARY`; the same
+content lands at `<log-dir>/analysis.md`.
+
+## Local invocation
+
+Outside Actions, the analyser prints to stdout instead of appending to a
+summary file:
+
+```sh
+cd spinifex/.github/actions/e2e-analyze
+go run . -junit-glob "/tmp/artifacts/junit-*.xml" -log-dir /tmp/artifacts
+```
+
+## Tests
+
+Golden-file tests live in `testdata/`. Each subdir holds a scenario:
+
+| Scenario           | Source                                    |
+|--------------------|-------------------------------------------|
+| `cascade-vpcs`     | Synthetic: one Phase 5 root + 15 cascades |
+| `single-failure`   | Real artifact from run `26037892507`      |
+| `no-failures`      | Synthetic: clean run                      |
+
+Run normally:
+
+```sh
+go test ./.github/actions/e2e-analyze/
+```
+
+When the report format changes intentionally, regenerate the snapshots:
+
+```sh
+go test ./.github/actions/e2e-analyze/ -update
+```
+
+## Per-failure bundles (Stage 2)
+
+After parsing, the action materialises a bundle directory per failure
+under `<log-dir>/analysis/NN-<TestName>/`. The report links to it from
+each entry so a reader on the Actions page can click straight through
+to the relevant log slice.
+
+Each bundle contains:
+
+| File              | Source                                                                            |
+|-------------------|-----------------------------------------------------------------------------------|
+| `failure.txt`     | Synthesised summary: test, start, duration, file-site, error line.                |
+| `journal.log`     | `spinifex-journal.log` sliced to `[StartAt − 5s, StartAt + Duration + 5s]`.       |
+| `test.log`        | `test-<suite>.log` block between `=== RUN <name>` and the matching `--- FAIL`.    |
+| `<other files>`   | Anything the harness already dropped into `<log-dir>/<TestName>/` at run-time.    |
+
+The journal slice relies on the workflow running
+`journalctl --output=short-iso` so timestamps are parseable. Lines
+without a parseable timestamp inherit the previous line's
+in-window/out-of-window classification (so multi-line records and stack
+traces stay contiguous).
+
+## Roadmap
+
+- **Stage 1 (shipped):** first-failure surfacing + cascade clustering.
+- **Stage 2 (this update):** time-window correlation — per-failure bundle
+  with sliced journal + test-log + any per-test artifacts.
+- **Stage 3:** rule-based root-cause classifier driven by
+  `spinifex/tests/e2e/.failure-rules.yaml`.
+
+See `docs/development/improvements/e2e-go-failure-analysis.md` in the
+parent mulga repo.

.github/actions/e2e-analyze/action.yml

@@ -0,0 +1,38 @@
+name: E2E Failure Analyzer
+description: >-
+  Post-test analyzer for the Go E2E suites. Parses junit-*.xml emitted by
+  go-junit-report, clusters failures by signature, picks the earliest
+  non-cascade failure per suite as the likely root cause, and appends a
+  human-readable report to $GITHUB_STEP_SUMMARY. Always exits 0 — the
+  existing "Fail job if any suite failed" gate keeps owning exit-code
+  semantics.
+
+inputs:
+  junit-glob:
+    description: Shell glob matching JUnit XML files to analyze.
+    required: true
+  log-dir:
+    description: >-
+      Artifact directory. The rendered report is also written to
+      $log-dir/analysis.md so it travels with the uploaded artifact.
+    required: true
+  title:
+    description: Heading rendered at the top of the summary section.
+    required: false
+    default: "E2E failure analysis"
+
+runs:
+  using: composite
+  steps:
+    - id: analyze
+      shell: bash
+      working-directory: ${{ github.action_path }}
+      env:
+        ANALYZE_JUNIT_GLOB: ${{ inputs.junit-glob }}
+        ANALYZE_LOG_DIR: ${{ inputs.log-dir }}
+        ANALYZE_TITLE: ${{ inputs.title }}
+      run: |
+        go run . \
+          -junit-glob "$ANALYZE_JUNIT_GLOB" \
+          -log-dir "$ANALYZE_LOG_DIR" \
+          -title "$ANALYZE_TITLE"

.github/actions/e2e-analyze/analyze.go

@@ -0,0 +1,552 @@
+// Package main implements the e2e-analyze GitHub Action.
+//
+// Stage 1 of docs/development/improvements/e2e-go-failure-analysis.md:
+// cluster failures from go-junit-report's junit-*.xml by error signature,
+// pick the earliest non-cascade failure per suite as the likely root cause,
+// and render a human-readable report.
+//
+// This file contains the pure logic (parsing, signature extraction,
+// clustering, rendering). Side-effect handling lives in main.go.
+package main
+
+import (
+	"encoding/xml"
+	"fmt"
+	"path/filepath"
+	"regexp"
+	"sort"
+	"strings"
+	"time"
+)
+
+// JUnit XML schema produced by go-junit-report v2. Stable across versions
+// we care about; only fields used downstream are decoded.
+type junitSuites struct {
+	XMLName xml.Name     `xml:"testsuites"`
+	Suites  []junitSuite `xml:"testsuite"`
+}
+
+type junitSuite struct {
+	Name      string    `xml:"name,attr"`
+	Tests     int       `xml:"tests,attr"`
+	Failures  int       `xml:"failures,attr"`
+	Errors    int       `xml:"errors,attr"`
+	Skipped   int       `xml:"skipped,attr"`
+	Timestamp string    `xml:"timestamp,attr"`
+	Time      float64   `xml:"time,attr"`
+	Cases     []junitTC `xml:"testcase"`
+}
+
+type junitTC struct {
+	Name      string        `xml:"name,attr"`
+	Classname string        `xml:"classname,attr"`
+	Time      float64       `xml:"time,attr"`
+	Failure   *junitFailure `xml:"failure"`
+	Error     *junitFailure `xml:"error"`
+	Skipped   *junitSkipped `xml:"skipped"`
+	SystemOut string        `xml:"system-out"`
+}
+
+type junitFailure struct {
+	Message string `xml:"message,attr"`
+	Type    string `xml:"type,attr"`
+	Body    string `xml:",chardata"`
+}
+
+type junitSkipped struct {
+	Message string `xml:"message,attr"`
+}
+
+// Failure is the analyzer's normalized view of one failed testcase.
+type Failure struct {
+	SuiteFile string    // e.g. "single" (derived from junit-single.xml)
+	Name      string    // e.g. "TestSingleNode/Phase5_LaunchInstance"
+	Duration  float64   // seconds
+	Order     int       // position within the suite (0-based)
+	StartAt   time.Time // estimated wall-clock start: suite.timestamp + sum(prior durations)
+	Error     string    // last meaningful error line, ANSI-stripped
+	FileHint  string    // e.g. "vpc_test.go:227", parsed from failure body
+	Cascade   bool      // signature matched a known downstream pattern
+	Signature string    // normalized error key for bucketing
+	// OffsetFromSuiteStart is sum(prior testcase durations) at the time
+	// this test started. Persisted separately from StartAt so a caller
+	// that has a more authoritative suite wall-clock origin (e.g. a
+	// `test-<suite>.start` file written by the workflow before the suite
+	// runs) can recompute StartAt without re-parsing the XML.
+	OffsetFromSuiteStart time.Duration
+	// BundlePath is the relative path (from log-dir) to the per-failure
+	// bundle written by Stage 2's WriteBundles. Empty when bundles were
+	// not generated (e.g. unit tests that call Render directly).
+	BundlePath string
+}
+
+// SuiteReport summarises one junit-*.xml file.
+type SuiteReport struct {
+	File      string // basename, e.g. "junit-single.xml"
+	Label     string // short name, e.g. "single"
+	Total     int
+	FailCount int
+	Root      *Failure    // nil if no failures
+	Cascades  []Failure   // everything else that failed in this suite
+	Buckets   [][]Failure // failures grouped by signature (root's bucket first)
+}
+
+// Report is the full analysis across every junit file.
+type Report struct {
+	Title  string
+	Suites []SuiteReport
+}
+
+// ansiSeq matches ANSI CSI escapes the harness sprinkles through log
+// output (colours, bold). Three forms in the wild:
+//   - real ESC: `\x1b[36m…`
+//   - U+FFFD replacement (XML serialiser dropped the ESC byte): `�[36m…`
+//   - bare CSI with the leading byte gone entirely: `[36m…`
+//
+// All three are stripped so colour state doesn't fork the signature bucket.
+var ansiSeq = regexp.MustCompile(
+	"\x1b\\[[0-9;]*[a-zA-Z]" + // real ESC CSI
+		"|�\\[[0-9;]*[a-zA-Z]" + // replacement-char CSI
+		"|\\[[0-9]{1,2}(?:;[0-9]{1,2})*m", // bare colour code, "mode" form only
+)
+
+// cascadeMarkers are message fragments that only appear in tests that
+// depend on a fixture field populated by an earlier phase. Their presence
+// means the failure is downstream of some other failure that happened
+// first; the report collapses them under that root cause.
+var cascadeMarkers = []string{
+	"must populate fix.",
+	"Should NOT be empty",
+	"Expected value not to be nil",
+}
+
+// noiseReplacers normalise dynamic identifiers so that "ten failures, same
+// bug" bucket together. Pattern ordering matters: longer/more-specific
+// patterns must precede the catch-alls.
+var noiseReplacers = []struct {
+	re   *regexp.Regexp
+	repl string
+}{
+	{regexp.MustCompile(`\b(vpc|subnet|sg|rtb|igw|i|ami|vol|snap|rtbassoc|key)-[0-9a-f]{8,}\b`), `${1}-<id>`},
+	{regexp.MustCompile(`\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b`), `<uuid>`},
+	{regexp.MustCompile(`\b[0-9a-f]{32,}\b`), `<hex>`},
+	{regexp.MustCompile(`\b\d{1,3}(\.\d{1,3}){3}(:\d+)?\b`), `<ip>`},
+	{regexp.MustCompile(`\b\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:?\d{2})?\b`), `<ts>`},
+	{regexp.MustCompile(`request-id:\s*\S+`), `request-id: <id>`},
+	{regexp.MustCompile(`(_test\.go):\d+`), `${1}:<line>`},
+	{regexp.MustCompile(`\s+`), ` `},
+}
+
+// fileHintRe pulls the last "foo_test.go:NNN" we see in a failure body so
+// the report can point a reader at the source line. Test files dominate the
+// hint set; harness/helper file matches go through goFileLineRe below.
+var fileHintRe = regexp.MustCompile(`([A-Za-z0-9_]+_test\.go:\d+)`)
+
+// goFileLineRe matches any "foo.go:NNN:" prefix (test or non-test). Used to
+// surface harness/helper-level error text like
+// `ec2helpers.go:164: describe-vpcs: InvalidParameterValue …` which Go test
+// emits when a non-test file calls t.Errorf.
+var goFileLineRe = regexp.MustCompile(`^[A-Za-z0-9_]+\.go:\d+:\s*(.+)$`)
+
+// testifyErrorPlaceholder is testify's "Error:" boilerplate when the
+// assertion failed because of a wrapped error. The next non-blank line in
+// the trace holds the actual error text.
+const testifyErrorPlaceholder = "Received unexpected error:"
+
+// stripANSI removes CSI escapes; leaves everything else intact.
+func stripANSI(s string) string {
+	return ansiSeq.ReplaceAllString(s, "")
+}
+
+// extractErrorLine finds the most useful single-line summary in a failure
+// body. Priority order, highest first:
+//  1. Testify "Messages:" line — user-supplied context, the most signal-rich.
+//  2. Testify "Error:" content (with one-line lookahead when the placeholder
+//     "Received unexpected error:" defers to the next line).
+//  3. Leading "foo.go:NNN: text" content line — t.Errorf from non-test code.
+//  4. Last `_test.go:NNN: text` line — bare t.Fatalf/Errorf from a test file.
+//  5. First non-empty line of the body.
+func extractErrorLine(body string) string {
+	body = stripANSI(body)
+	lines := strings.Split(body, "\n")
+
+	var (
+		msgLine      string
+		errLine      string
+		goFileLine   string
+		lastTestLine string
+	)
+	for i, raw := range lines {
+		l := strings.TrimSpace(raw)
+		if l == "" {
+			continue
+		}
+		if strings.HasPrefix(l, "Messages:") {
+			msgLine = strings.TrimSpace(strings.TrimPrefix(l, "Messages:"))
+			continue
+		}
+		if strings.HasPrefix(l, "Error:") && errLine == "" {
+			content := strings.TrimSpace(strings.TrimPrefix(l, "Error:"))
+			if content == testifyErrorPlaceholder {
+				// Look ahead for the wrapped error text on the next non-blank line.
+				for j := i + 1; j < len(lines); j++ {
+					nxt := strings.TrimSpace(lines[j])
+					if nxt == "" {
+						continue
+					}
+					if strings.HasPrefix(nxt, "Test:") || strings.HasPrefix(nxt, "Messages:") {
+						break
+					}
+					content = content + " " + nxt
+					break
+				}
+			}
+			errLine = content
+			continue
+		}
+		if m := goFileLineRe.FindStringSubmatch(l); m != nil {
+			// Keep the LAST occurrence: t.Log / harness traces emit
+			// progress lines from the same file:line pattern before the
+			// failing assertion finally fires.
+			goFileLine = strings.TrimSpace(m[1])
+			continue
+		}
+		if fileHintRe.MatchString(l) {
+			lastTestLine = l
+		}
+	}
+	switch {
+	case msgLine != "":
+		return msgLine
+	case errLine != "":
+		return errLine
+	case goFileLine != "":
+		return goFileLine
+	case lastTestLine != "":
+		// "vpc_test.go:227: Eventually: condition not met …" → trim the file prefix.
+		if i := strings.Index(lastTestLine, ": "); i > 0 {
+			return strings.TrimSpace(lastTestLine[i+2:])
+		}
+		return lastTestLine
+	}
+	// Last resort: first non-empty line of the body.
+	for _, raw := range lines {
+		l := strings.TrimSpace(raw)
+		if l != "" {
+			return l
+		}
+	}
+	return ""
+}
+
+// errorTraceRe extracts the path from testify's "Error Trace:" line — e.g.
+//
+//	Error Trace:	tests/e2e/harness/ec2helpers.go:164
+//
+// — which points at the assertion site even when it lives in helper code
+// outside *_test.go.
+var errorTraceRe = regexp.MustCompile(`Error Trace:\s*(\S+\.go:\d+)`)
+
+// extractFileHint returns the most precise pointer at the assertion site
+// we can pull from the failure body: testify Error Trace wins, otherwise
+// the last `*_test.go:NNN` line in the body.
+func extractFileHint(body string) string {
+	body = stripANSI(body)
+	if m := errorTraceRe.FindStringSubmatch(body); m != nil {
+		return m[1]
+	}
+	all := fileHintRe.FindAllString(body, -1)
+	if len(all) == 0 {
+		return ""
+	}
+	return all[len(all)-1]
+}
+
+// signature normalises an error line so different runs of the same root
+// cause bucket together. Empty signatures (e.g. parent-test placeholders
+// where the failure body is empty) bucket together under "".
+func signature(errLine string) string {
+	s := strings.ToLower(errLine)
+	for _, r := range noiseReplacers {
+		s = r.re.ReplaceAllString(s, r.repl)
+	}
+	return strings.TrimSpace(s)
+}
+
+// isCascade reports whether the error line contains any of the well-known
+// fixture-propagation markers — i.e. "this test failed because something
+// upstream of it failed first".
+func isCascade(errLine string) bool {
+	for _, m := range cascadeMarkers {
+		if strings.Contains(errLine, m) {
+			return true
+		}
+	}
+	return false
+}
+
+// computeLeafSet returns the testcase names that are leaves — i.e. no
+// other testcase name in `cases` starts with `<name>/`. go-junit-report
+// emits one <testcase> for the parent Go test AND each subtest, with the
+// parent's time = sum of its children, so summing every entry into the
+// suite's cumulative offset double-counts. Restrict accumulation to
+// leaves only.
+func computeLeafSet(cases []junitTC) map[string]bool {
+	leaf := make(map[string]bool, len(cases))
+	for _, tc := range cases {
+		leaf[tc.Name] = true
+	}
+	for _, tc := range cases {
+		// Strip the deepest path segment; any ancestor of this test is
+		// not a leaf.
+		name := tc.Name
+		for {
+			i := strings.LastIndex(name, "/")
+			if i <= 0 {
+				break
+			}
+			name = name[:i]
+			if _, ok := leaf[name]; ok {
+				leaf[name] = false
+			}
+		}
+	}
+	return leaf
+}
+
+// parseStartTime turns a junit timestamp attr (RFC3339) into a time.Time;
+// zero value if unparseable.
+func parseStartTime(s string) time.Time {
+	if s == "" {
+		return time.Time{}
+	}
+	t, err := time.Parse(time.RFC3339, s)
+	if err != nil {
+		return time.Time{}
+	}
+	return t
+}
+
+// suiteLabel turns "junit-single.xml" into "single".
+func suiteLabel(path string) string {
+	base := filepath.Base(path)
+	base = strings.TrimSuffix(base, ".xml")
+	base = strings.TrimPrefix(base, "junit-")
+	if base == "" {
+		return filepath.Base(path)
+	}
+	return base
+}
+
+// ParseFile decodes one junit-*.xml file into our normalized failure list.
+// Returns the suite label, total testcase count, and the (ordered) failures.
+func ParseFile(path string, data []byte) (SuiteReport, error) {
+	var doc junitSuites
+	if err := xml.Unmarshal(data, &doc); err != nil {
+		return SuiteReport{}, fmt.Errorf("parse %s: %w", path, err)
+	}
+
+	rep := SuiteReport{
+		File:  filepath.Base(path),
+		Label: suiteLabel(path),
+	}
+
+	order := 0
+	for _, suite := range doc.Suites {
+		start := parseStartTime(suite.Timestamp)
+		// go-junit-report emits a testcase for both the parent Go test
+		// (e.g. `TestSingleNode`) AND each subtest (`TestSingleNode/PhaseX`).
+		// The parent's `time` attribute is the aggregate of its children, so
+		// summing every testcase double-counts. Pre-compute the leaf set so
+		// only leaves contribute to the suite's running offset.
+		isLeaf := computeLeafSet(suite.Cases)
+		cumul := 0.0
+		for _, tc := range suite.Cases {
+			rep.Total++
+			leaf := isLeaf[tc.Name]
+			var body string
+			switch {
+			case tc.Failure != nil:
+				body = tc.Failure.Body
+			case tc.Error != nil:
+				body = tc.Error.Body
+			default:
+				if leaf {
+					cumul += tc.Time
+				}
+				continue
+			}
+
+			// Skip the synthetic parent-test "Failed" stub that go-junit-report
+			// emits for any Go test whose subtests failed. It has an empty body
+			// and would otherwise outrank real failures by XML order.
+			trimmed := strings.TrimSpace(stripANSI(body))
+			if trimmed == "" {
+				if leaf {
+					cumul += tc.Time
+				}
+				continue
+			}
+
+			errLine := extractErrorLine(body)
+			offset := time.Duration(cumul * float64(time.Second))
+			f := Failure{
+				SuiteFile:            rep.Label,
+				Name:                 tc.Name,
+				Duration:             tc.Time,
+				Order:                order,
+				Error:                errLine,
+				FileHint:             extractFileHint(body),
+				Cascade:              isCascade(errLine),
+				Signature:            signature(errLine),
+				OffsetFromSuiteStart: offset,
+			}
+			if !start.IsZero() {
+				f.StartAt = start.Add(offset)
+			}
+			rep.FailCount++
+			rep.Cascades = append(rep.Cascades, f) // temp staging; split below
+			order++
+			if leaf {
+				cumul += tc.Time
+			}
+		}
+	}
+
+	if rep.FailCount == 0 {
+		rep.Cascades = nil
+		return rep, nil
+	}
+
+	all := rep.Cascades
+	rep.Cascades = nil
+
+	// Root = earliest non-cascade. If every failure is a cascade (the whole
+	// suite tripped on the same upstream gap), fall back to earliest overall.
+	rootIdx := -1
+	for i, f := range all {
+		if !f.Cascade {
+			rootIdx = i
+			break
+		}
+	}
+	if rootIdx == -1 {
+		rootIdx = 0
+	}
+	root := all[rootIdx]
+	rep.Root = &root
+
+	// Bucket the remainder by signature so the report can collapse the
+	// "ten failures, same bug" pattern.
+	bucketKeys := []string{}
+	buckets := map[string][]Failure{}
+	for i, f := range all {
+		if i == rootIdx {
+			continue
+		}
+		if _, seen := buckets[f.Signature]; !seen {
+			bucketKeys = append(bucketKeys, f.Signature)
+		}
+		buckets[f.Signature] = append(buckets[f.Signature], f)
+		rep.Cascades = append(rep.Cascades, f)
+	}
+
+	// Largest bucket first; ties broken by first-seen order so output is
+	// deterministic across runs.
+	sort.SliceStable(bucketKeys, func(i, j int) bool {
+		return len(buckets[bucketKeys[i]]) > len(buckets[bucketKeys[j]])
+	})
+	for _, k := range bucketKeys {
+		rep.Buckets = append(rep.Buckets, buckets[k])
+	}
+
+	return rep, nil
+}
+
+// Render writes the markdown report. Format is documented in
+// docs/development/improvements/e2e-go-failure-analysis.md (Stage 1).
+func Render(r Report) string {
+	var b strings.Builder
+	fmt.Fprintf(&b, "## %s\n\n", r.Title)
+
+	totalFail := 0
+	for _, s := range r.Suites {
+		totalFail += s.FailCount
+	}
+	if totalFail == 0 {
+		b.WriteString("✅ No failures across any suite.\n")
+		return b.String()
+	}
+
+	for _, s := range r.Suites {
+		if s.FailCount == 0 {
+			fmt.Fprintf(&b, "### Suite `%s`: ✅ pass (%d tests)\n\n", s.Label, s.Total)
+			continue
+		}
+		fmt.Fprintf(&b, "### Suite `%s`: %d failed, 1 root cause likely\n\n", s.Label, s.FailCount)
+
+		if s.Root != nil {
+			b.WriteString("**Root cause (earliest non-cascade)**\n\n")
+			fmt.Fprintf(&b, "- Test: `%s`\n", s.Root.Name)
+			if !s.Root.StartAt.IsZero() {
+				fmt.Fprintf(&b, "- Start: %s (duration %.1fs)\n",
+					s.Root.StartAt.Format("15:04:05"), s.Root.Duration)
+			} else {
+				fmt.Fprintf(&b, "- Duration: %.1fs\n", s.Root.Duration)
+			}
+			if s.Root.Error != "" {
+				fmt.Fprintf(&b, "- Error: %s\n", oneLine(s.Root.Error))
+			}
+			if s.Root.FileHint != "" {
+				fmt.Fprintf(&b, "- File:  `%s`\n", s.Root.FileHint)
+			}
+			if s.Root.BundlePath != "" {
+				fmt.Fprintf(&b, "- Bundle: `%s/`\n", s.Root.BundlePath)
+			}
+			b.WriteString("\n")
+		}
+
+		if len(s.Buckets) > 0 {
+			fmt.Fprintf(&b, "**Cascaded failures (%d) — grouped by signature**\n\n",
+				len(s.Cascades))
+			for _, bucket := range s.Buckets {
+				sample := bucket[0]
+				label := oneLine(sample.Error)
+				if label == "" {
+					label = "(no error message)"
+				}
+				fmt.Fprintf(&b, "- %d× %s\n", len(bucket), label)
+				// List up to 5 test names per bucket; collapse the rest.
+				const cap = 5
+				for i, f := range bucket {
+					if i == cap && len(bucket) > cap {
+						fmt.Fprintf(&b, "  - … and %d more\n", len(bucket)-cap)
+						break
+					}
+					if f.BundlePath != "" {
+						fmt.Fprintf(&b, "  - `%s` → `%s/`\n", f.Name, f.BundlePath)
+					} else {
+						fmt.Fprintf(&b, "  - `%s`\n", f.Name)
+					}
+				}
+			}
+			b.WriteString("\n")
+		}
+	}
+
+	return b.String()
+}
+
+// oneLine collapses whitespace and caps length so a runaway assertion
+// message doesn't blow out the summary card width.
+func oneLine(s string) string {
+	s = strings.TrimSpace(s)
+	s = strings.ReplaceAll(s, "\n", " ")
+	s = strings.ReplaceAll(s, "\r", " ")
+	s = regexp.MustCompile(`\s+`).ReplaceAllString(s, " ")
+	const max = 160
+	if len(s) > max {
+		s = s[:max-1] + "…"
+	}
+	return s
+}

.github/actions/e2e-analyze/analyze_test.go

@@ -0,0 +1,191 @@
+package main
+
+import (
+	"flag"
+	"os"
+	"path/filepath"
+	"sort"
+	"strings"
+	"testing"
+)
+
+// updateGolden lets `go test ./.github/actions/e2e-analyze/ -update` rewrite
+// the expected analysis.md files when the rendering format changes
+// intentionally. Day-to-day, leave it unset: tests diff against the on-disk
+// snapshot to catch unintended drift in signature extraction or layout.
+var updateGolden = flag.Bool("update", false, "rewrite testdata/*/analysis.md from current output")
+
+// TestGoldenFixtures runs the full ParseFile + Render pipeline against each
+// fixture directory under testdata/ and asserts the output matches the
+// adjacent analysis.md snapshot. Each subdir is one named scenario; the
+// fixture XMLs match the same junit-*.xml glob the production action uses
+// so behaviour stays representative of CI.
+func TestGoldenFixtures(t *testing.T) {
+	entries, err := os.ReadDir("testdata")
+	if err != nil {
+		t.Fatalf("read testdata: %v", err)
+	}
+	for _, e := range entries {
+		if !e.IsDir() {
+			continue
+		}
+		name := e.Name()
+		t.Run(name, func(t *testing.T) {
+			dir := filepath.Join("testdata", name)
+			matches, err := filepath.Glob(filepath.Join(dir, "junit-*.xml"))
+			if err != nil {
+				t.Fatalf("glob: %v", err)
+			}
+			sort.Strings(matches)
+			if len(matches) == 0 {
+				t.Fatalf("no junit-*.xml fixtures in %s", dir)
+			}
+
+			rep := Report{Title: "E2E failure analysis"}
+			for _, p := range matches {
+				data, err := os.ReadFile(p)
+				if err != nil {
+					t.Fatalf("read %s: %v", p, err)
+				}
+				sr, err := ParseFile(p, data)
+				if err != nil {
+					t.Fatalf("parse %s: %v", p, err)
+				}
+				rep.Suites = append(rep.Suites, sr)
+			}
+
+			got := Render(rep)
+			goldenPath := filepath.Join(dir, "analysis.md")
+			if *updateGolden {
+				if err := os.WriteFile(goldenPath, []byte(got), 0o644); err != nil {
+					t.Fatalf("write golden: %v", err)
+				}
+				return
+			}
+			want, err := os.ReadFile(goldenPath)
+			if err != nil {
+				t.Fatalf("read golden (run with -update to seed): %v", err)
+			}
+			if string(want) != got {
+				t.Errorf("rendered output differs from %s\n--- want ---\n%s\n--- got ---\n%s",
+					goldenPath, string(want), got)
+			}
+		})
+	}
+}
+
+func TestSignatureCollapsesNoise(t *testing.T) {
+	a := signature("describe-vpcs: InvalidParameterValue request-id: abc-123 vpc-deadbeefcafe1234")
+	b := signature("describe-vpcs: InvalidParameterValue request-id: xyz-999 vpc-1111222233334444")
+	if a != b {
+		t.Errorf("signatures should collapse to same bucket:\n  a=%q\n  b=%q", a, b)
+	}
+	if !strings.Contains(a, "<id>") || !strings.Contains(a, "vpc-<id>") {
+		t.Errorf("expected normalised placeholders in signature, got %q", a)
+	}
+}
+
+func TestCascadeDetection(t *testing.T) {
+	cases := []struct {
+		in   string
+		want bool
+	}{
+		{"Phase 5 must populate fix.InstanceID", true},
+		{"Should NOT be empty", true},
+		{"Expected value not to be nil", true},
+		{"describe-vpcs: InvalidParameterValue", false},
+		{"Eventually: condition not met within 3m0s", false},
+	}
+	for _, c := range cases {
+		if got := isCascade(c.in); got != c.want {
+			t.Errorf("isCascade(%q) = %v, want %v", c.in, got, c.want)
+		}
+	}
+}
+
+func TestExtractErrorLine_MessagesWinsOverError(t *testing.T) {
+	body := `    lifecycle_test.go:28:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:28
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5a_pre_ClusterStats`
+	got := extractErrorLine(body)
+	want := "Phase 5 must populate fix.InstanceID"
+	if got != want {
+		t.Errorf("extractErrorLine = %q, want %q", got, want)
+	}
+}
+
+func TestExtractErrorLine_FallsBackToLastTestLine(t *testing.T) {
+	body := "    vpc_test.go:227: Eventually: condition not met within 3m0s: [SSH handshake never completed]"
+	got := extractErrorLine(body)
+	if !strings.HasPrefix(got, "Eventually:") {
+		t.Errorf("expected fallback to test-line content, got %q", got)
+	}
+}
+
+func TestExtractFileHint_PicksLastMatch(t *testing.T) {
+	body := "ec2helpers.go:50: setup\n    vpc_test.go:227: Eventually: condition not met"
+	got := extractFileHint(body)
+	want := "vpc_test.go:227"
+	if got != want {
+		t.Errorf("extractFileHint = %q, want %q", got, want)
+	}
+}
+
+func TestParseFile_SkipsEmptyParentFailure(t *testing.T) {
+	// The synthetic "TestSingleNode" parent gets a <failure message="Failed"/>
+	// with no body whenever any subtest fails. The analyzer must skip it so
+	// it doesn't outrank the real first failure on XML order.
+	xml := []byte(`<?xml version="1.0"?>
+<testsuites><testsuite name="" tests="2" failures="2">
+  <testcase name="TestX" time="1.0"><failure message="Failed"></failure></testcase>
+  <testcase name="TestX/RealFailure" time="0.5"><failure message="Failed"><![CDATA[
+    foo_test.go:10:
+        Messages: real assertion message
+]]></failure></testcase>
+</testsuite></testsuites>`)
+	sr, err := ParseFile("junit-x.xml", xml)
+	if err != nil {
+		t.Fatalf("ParseFile: %v", err)
+	}
+	if sr.FailCount != 1 {
+		t.Fatalf("FailCount = %d, want 1 (empty parent should be skipped)", sr.FailCount)
+	}
+	if sr.Root == nil || sr.Root.Name != "TestX/RealFailure" {
+		t.Fatalf("expected real failure as root, got %+v", sr.Root)
+	}
+}
+
+func TestParseFile_CascadeFallbackWhenAllCascades(t *testing.T) {
+	// If every failure is a cascade marker, the earliest one becomes root
+	// so the report isn't blank.
+	xml := []byte(`<?xml version="1.0"?>
+<testsuites><testsuite name="" tests="2" failures="2">
+  <testcase name="TestX/A" time="1.0"><failure message="Failed"><![CDATA[
+    Messages: Phase 5 must populate fix.InstanceID
+]]></failure></testcase>
+  <testcase name="TestX/B" time="1.0"><failure message="Failed"><![CDATA[
+    Messages: Phase 5 must populate fix.InstanceID
+]]></failure></testcase>
+</testsuite></testsuites>`)
+	sr, err := ParseFile("junit-x.xml", xml)
+	if err != nil {
+		t.Fatalf("ParseFile: %v", err)
+	}
+	if sr.Root == nil || sr.Root.Name != "TestX/A" {
+		t.Fatalf("expected TestX/A as fallback root, got %+v", sr.Root)
+	}
+}
+
+func TestRender_NoFailuresIsClean(t *testing.T) {
+	out := Render(Report{
+		Title: "E2E failure analysis",
+		Suites: []SuiteReport{
+			{File: "junit-single.xml", Label: "single", Total: 3, FailCount: 0},
+		},
+	})
+	if !strings.Contains(out, "No failures") {
+		t.Errorf("expected zero-failure banner, got:\n%s", out)
+	}
+}

.github/actions/e2e-analyze/bundle.go

@@ -0,0 +1,208 @@
+package main
+
+// Stage 2 of docs/development/improvements/e2e-go-failure-analysis.md:
+// for each failure surfaced by Stage 1, materialise a per-failure bundle
+// directory containing the failure summary, a time-window slice of the
+// daemon journal, a testcase-window slice of the suite's stdout log, and
+// any per-test artifacts the harness already dumped at run-time.
+
+import (
+	"errors"
+	"fmt"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"regexp"
+	"strings"
+	"time"
+)
+
+// bundlePad is the per-failure window padding either side of the
+// [StartAt, StartAt+Duration] testcase span. Five seconds matches the
+// plan doc: long enough to catch the daemon's pre-test setup and
+// post-test cleanup chatter, short enough to keep the slice focused.
+const bundlePad = 5 * time.Second
+
+// nameSanitiseRe replaces characters unsafe in directory names with `_`.
+// Tracks tests/e2e/harness/artifacts.go which uses the same convention
+// when the test's own dump dir is created, so we can pick its contents
+// up by path without a second translation table.
+var nameSanitiseRe = regexp.MustCompile(`[^A-Za-z0-9._-]+`)
+
+func sanitiseTestName(s string) string {
+	return nameSanitiseRe.ReplaceAllString(s, "_")
+}
+
+// ApplySuiteStartFiles overrides each failure's StartAt by reading
+// <logDir>/test-<suiteLabel>.start (RFC3339 timestamp written by the
+// workflow before the suite runs). Required because go-junit-report's
+// `timestamp` attribute on <testsuite> records the time the XML was
+// produced — i.e. after the suite has finished — so it can't be used
+// as the suite wall-clock origin for journal slicing.
+//
+// Missing or unparseable files are silently ignored: the analyzer
+// falls back to the junit timestamp and the bundle still gets created,
+// it just may have an empty (or off-window) journal slice.
+func ApplySuiteStartFiles(rep *Report, logDir string) {
+	if logDir == "" {
+		return
+	}
+	for si := range rep.Suites {
+		s := &rep.Suites[si]
+		startPath := filepath.Join(logDir, fmt.Sprintf("test-%s.start", s.Label))
+		data, err := os.ReadFile(startPath)
+		if err != nil {
+			continue
+		}
+		ts, err := time.Parse(time.RFC3339, strings.TrimSpace(string(data)))
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: parse %s: %v\n", startPath, err)
+			continue
+		}
+		if s.Root != nil {
+			s.Root.StartAt = ts.Add(s.Root.OffsetFromSuiteStart)
+		}
+		for ci := range s.Cascades {
+			s.Cascades[ci].StartAt = ts.Add(s.Cascades[ci].OffsetFromSuiteStart)
+		}
+		for bi := range s.Buckets {
+			for fi := range s.Buckets[bi] {
+				f := &s.Buckets[bi][fi]
+				f.StartAt = ts.Add(f.OffsetFromSuiteStart)
+			}
+		}
+	}
+}
+
+// WriteBundles materialises per-failure bundles under <logDir>/analysis/.
+// Mutates rep in-place so each Failure.BundlePath holds the relative
+// path the renderer will link from the summary.
+//
+// Failures with no parseable wall-clock start still get a bundle dir
+// (and the testcase-window test.log slice), but no journal slice — the
+// time window is unknown.
+//
+// Errors are non-fatal: a bundle that partially fails still gets linked
+// from the report so a reader sees what's available, and the analyzer
+// keeps printing warnings so CI logs surface the gap.
+func WriteBundles(rep *Report, logDir string) {
+	if logDir == "" {
+		return
+	}
+	base := filepath.Join(logDir, "analysis")
+	if err := os.MkdirAll(base, 0o755); err != nil {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: mkdir %s: %v\n", base, err)
+		return
+	}
+
+	for si := range rep.Suites {
+		s := &rep.Suites[si]
+		if s.Root != nil {
+			writeOneBundle(logDir, base, s.Label, s.Root)
+		}
+		for ci := range s.Cascades {
+			writeOneBundle(logDir, base, s.Label, &s.Cascades[ci])
+		}
+		// Bucket entries share storage with Cascades but live in a
+		// separate slice in SuiteReport; mirror the bundle path so the
+		// renderer's bucket loop sees it too.
+		for bi := range s.Buckets {
+			for fi := range s.Buckets[bi] {
+				f := &s.Buckets[bi][fi]
+				if bp := findBundlePath(s, f.Name); bp != "" {
+					f.BundlePath = bp
+				}
+			}
+		}
+	}
+}
+
+// findBundlePath looks up a Failure's already-written BundlePath by test
+// name within a suite. Buckets reference the same logical failure as
+// Cascades but are independent slice values, so we copy the path across
+// rather than re-deriving it.
+func findBundlePath(s *SuiteReport, testName string) string {
+	if s.Root != nil && s.Root.Name == testName {
+		return s.Root.BundlePath
+	}
+	for _, c := range s.Cascades {
+		if c.Name == testName {
+			return c.BundlePath
+		}
+	}
+	return ""
+}
+
+func writeOneBundle(logDir, base, suiteLabel string, f *Failure) {
+	name := fmt.Sprintf("%02d-%s", f.Order+1, sanitiseTestName(f.Name))
+	dir := filepath.Join(base, name)
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: mkdir %s: %v\n", dir, err)
+		return
+	}
+	f.BundlePath = filepath.Join("analysis", name)
+
+	writeFailureSummary(dir, suiteLabel, f)
+
+	if !f.StartAt.IsZero() {
+		start := f.StartAt.Add(-bundlePad)
+		end := f.StartAt.Add(time.Duration(f.Duration * float64(time.Second))).Add(bundlePad)
+		journalSrc := filepath.Join(logDir, "spinifex-journal.log")
+		if _, err := SliceJournal(journalSrc, start, end, filepath.Join(dir, "journal.log")); err != nil {
+			fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: slice journal for %s: %v\n", f.Name, err)
+		}
+	}
+
+	testLogSrc := filepath.Join(logDir, fmt.Sprintf("test-%s.log", suiteLabel))
+	if _, err := SliceTestLog(testLogSrc, f.Name, filepath.Join(dir, "test.log")); err != nil {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: slice test log for %s: %v\n", f.Name, err)
+	}
+
+	// Per-test artifact subdir written by harness.ArtifactDir lives at
+	// <logDir>/<TestName with / replaced by _>/. Copy any files into
+	// the bundle so qemu/ovs/etc. dumps already collected by the test
+	// travel alongside the analysis.
+	perTestSrc := filepath.Join(logDir, strings.ReplaceAll(f.Name, "/", "_"))
+	if entries, err := os.ReadDir(perTestSrc); err == nil {
+		for _, e := range entries {
+			if e.IsDir() {
+				continue
+			}
+			_ = copyFile(filepath.Join(perTestSrc, e.Name()), filepath.Join(dir, e.Name()))
+		}
+	} else if !os.IsNotExist(err) && !errorsIsNotDir(err) {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: read per-test dir %s: %v\n", perTestSrc, err)
+	}
+}
+
+// errorsIsNotDir handles the case where the harness wrote a file at the
+// per-test name instead of a directory — best-effort skip.
+func errorsIsNotDir(err error) bool {
+	var pe *fs.PathError
+	if !errors.As(err, &pe) {
+		return false
+	}
+	return pe.Err != nil && strings.Contains(pe.Err.Error(), "not a directory")
+}
+
+func writeFailureSummary(dir, suiteLabel string, f *Failure) {
+	var b strings.Builder
+	fmt.Fprintf(&b, "Suite:    %s\n", suiteLabel)
+	fmt.Fprintf(&b, "Test:     %s\n", f.Name)
+	if !f.StartAt.IsZero() {
+		fmt.Fprintf(&b, "Start:    %s\n", f.StartAt.Format(time.RFC3339))
+	}
+	fmt.Fprintf(&b, "Duration: %.3fs\n", f.Duration)
+	if f.FileHint != "" {
+		fmt.Fprintf(&b, "Site:     %s\n", f.FileHint)
+	}
+	if f.Cascade {
+		b.WriteString("Cascade:  yes (downstream of another failure)\n")
+	}
+	if f.Error != "" {
+		b.WriteString("\nError:\n")
+		b.WriteString(oneLine(f.Error))
+		b.WriteString("\n")
+	}
+	_ = os.WriteFile(filepath.Join(dir, "failure.txt"), []byte(b.String()), 0o644)
+}

.github/actions/e2e-analyze/bundle_test.go

@@ -0,0 +1,229 @@
+package main
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+func TestSliceJournal_KeepsOnlyWindow(t *testing.T) {
+	src := filepath.Join(t.TempDir(), "journal.log")
+	dst := filepath.Join(t.TempDir(), "out.log")
+	body := strings.Join([]string{
+		"2026-05-19T12:32:30+1000 host spinifex-daemon[1]: pre",
+		"  continuation of pre",
+		"2026-05-19T12:32:35+1000 host spinifex-daemon[1]: IN window",
+		"  multiline inside window",
+		"2026-05-19T12:32:45+1000 host spinifex-daemon[1]: post",
+		"",
+	}, "\n")
+	if err := os.WriteFile(src, []byte(body), 0o644); err != nil {
+		t.Fatalf("write src: %v", err)
+	}
+	start, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:33+10:00")
+	end, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:40+10:00")
+	n, err := SliceJournal(src, start, end, dst)
+	if err != nil {
+		t.Fatalf("SliceJournal: %v", err)
+	}
+	if n != 2 {
+		t.Fatalf("written = %d, want 2", n)
+	}
+	got, _ := os.ReadFile(dst)
+	if !strings.Contains(string(got), "IN window") || !strings.Contains(string(got), "multiline inside window") {
+		t.Errorf("missing in-window content:\n%s", got)
+	}
+	if strings.Contains(string(got), "pre") || strings.Contains(string(got), "post") {
+		t.Errorf("kept out-of-window content:\n%s", got)
+	}
+}
+
+func TestSliceJournal_AcceptsRFC3339ColonOffset(t *testing.T) {
+	// systemd's `journalctl --output=short-iso` ships RFC3339 timestamps
+	// (offset includes a colon, e.g. "+10:00") on most versions. Earlier
+	// regex only matched "+1000" — make sure both forms work.
+	src := filepath.Join(t.TempDir(), "journal.log")
+	dst := filepath.Join(t.TempDir(), "out.log")
+	body := strings.Join([]string{
+		"2026-05-19T12:32:30+10:00 host spinifex-daemon[1]: pre",
+		"2026-05-19T12:32:35+10:00 host spinifex-daemon[1]: IN window",
+		"2026-05-19T12:32:50+10:00 host spinifex-daemon[1]: post",
+		"",
+	}, "\n")
+	if err := os.WriteFile(src, []byte(body), 0o644); err != nil {
+		t.Fatalf("write src: %v", err)
+	}
+	start, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:33+10:00")
+	end, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:40+10:00")
+	n, err := SliceJournal(src, start, end, dst)
+	if err != nil {
+		t.Fatalf("SliceJournal: %v", err)
+	}
+	if n != 1 {
+		t.Fatalf("written = %d, want 1", n)
+	}
+	got, _ := os.ReadFile(dst)
+	if !strings.Contains(string(got), "IN window") {
+		t.Errorf("missing in-window content:\n%s", got)
+	}
+}
+
+func TestSliceJournal_MissingSrcIsNotError(t *testing.T) {
+	dst := filepath.Join(t.TempDir(), "out.log")
+	n, err := SliceJournal("/no/such/file", time.Now(), time.Now(), dst)
+	if err != nil {
+		t.Fatalf("unexpected err: %v", err)
+	}
+	if n != 0 {
+		t.Errorf("written = %d, want 0", n)
+	}
+}
+
+func TestSliceTestLog_PicksOneTestcase(t *testing.T) {
+	src := filepath.Join(t.TempDir(), "test.log")
+	dst := filepath.Join(t.TempDir(), "out.log")
+	body := strings.Join([]string{
+		"=== RUN   TestSingleNode",
+		"  before subtests",
+		"=== RUN   TestSingleNode/Phase5_LaunchInstance",
+		"    runinstances",
+		"    ec2helpers.go:164: describe-vpcs: InvalidParameterValue",
+		"--- FAIL: TestSingleNode/Phase5_LaunchInstance (2.50s)",
+		"=== RUN   TestSingleNode/Phase6_Volumes",
+		"    other content",
+		"--- FAIL: TestSingleNode/Phase6_Volumes (0.10s)",
+		"--- FAIL: TestSingleNode (10.0s)",
+		"",
+	}, "\n")
+	if err := os.WriteFile(src, []byte(body), 0o644); err != nil {
+		t.Fatalf("write src: %v", err)
+	}
+	n, err := SliceTestLog(src, "TestSingleNode/Phase5_LaunchInstance", dst)
+	if err != nil {
+		t.Fatalf("SliceTestLog: %v", err)
+	}
+	if n != 4 {
+		t.Fatalf("written = %d, want 4 (RUN + 2 inner + FAIL)", n)
+	}
+	got, _ := os.ReadFile(dst)
+	s := string(got)
+	if !strings.Contains(s, "describe-vpcs: InvalidParameterValue") {
+		t.Errorf("missing failure line:\n%s", s)
+	}
+	if strings.Contains(s, "Phase6_Volumes") || strings.Contains(s, "before subtests") {
+		t.Errorf("leaked sibling test content:\n%s", s)
+	}
+}
+
+func TestSliceTestLog_UnknownTestcaseIsEmpty(t *testing.T) {
+	src := filepath.Join(t.TempDir(), "test.log")
+	dst := filepath.Join(t.TempDir(), "out.log")
+	if err := os.WriteFile(src, []byte("=== RUN   TestX\n--- PASS: TestX (0.0s)\n"), 0o644); err != nil {
+		t.Fatalf("write src: %v", err)
+	}
+	n, err := SliceTestLog(src, "TestY", dst)
+	if err != nil {
+		t.Fatalf("SliceTestLog: %v", err)
+	}
+	if n != 0 {
+		t.Errorf("written = %d, want 0", n)
+	}
+	if _, err := os.Stat(dst); err != nil {
+		t.Errorf("dst not created: %v", err)
+	}
+}
+
+// TestWriteBundles_EndToEnd builds a fake artifact dir with a junit XML,
+// a journal log, a test log, and a per-test artifact subdir, then runs
+// the full ParseFile → WriteBundles pipeline and asserts the bundle dir
+// contains everything the report links to.
+func TestWriteBundles_EndToEnd(t *testing.T) {
+	dir := t.TempDir()
+
+	junit := `<?xml version="1.0"?>
+<testsuites><testsuite name="" tests="2" failures="1" timestamp="2026-05-19T12:32:30+10:00" time="3.0">
+  <testcase name="TestSingleNode/Phase5_LaunchInstance" time="2.500">
+    <failure message="Failed"><![CDATA[
+    ec2helpers.go:164: describe-vpcs: InvalidParameterValue (status 400)
+        Error Trace: tests/e2e/harness/ec2helpers.go:164
+        Test: TestSingleNode/Phase5_LaunchInstance]]></failure>
+  </testcase>
+</testsuite></testsuites>`
+	if err := os.WriteFile(filepath.Join(dir, "junit-single.xml"), []byte(junit), 0o644); err != nil {
+		t.Fatalf("write junit: %v", err)
+	}
+
+	journal := strings.Join([]string{
+		"2026-05-19T12:32:20+1000 host spinifex-daemon[1]: too-early",
+		"2026-05-19T12:32:30+1000 host spinifex-daemon[1]: in-window-start",
+		"2026-05-19T12:32:32+1000 host spinifex-daemon[1]: describe-vpcs filter=is_default",
+		"2026-05-19T12:32:50+1000 host spinifex-daemon[1]: too-late",
+		"",
+	}, "\n")
+	if err := os.WriteFile(filepath.Join(dir, "spinifex-journal.log"), []byte(journal), 0o644); err != nil {
+		t.Fatalf("write journal: %v", err)
+	}
+
+	testlog := strings.Join([]string{
+		"=== RUN   TestSingleNode/Phase5_LaunchInstance",
+		"    ec2helpers.go:164: describe-vpcs: InvalidParameterValue",
+		"--- FAIL: TestSingleNode/Phase5_LaunchInstance (2.50s)",
+		"",
+	}, "\n")
+	if err := os.WriteFile(filepath.Join(dir, "test-single.log"), []byte(testlog), 0o644); err != nil {
+		t.Fatalf("write test log: %v", err)
+	}
+
+	// Per-test artifact subdir (matches harness.ArtifactDir convention).
+	perTest := filepath.Join(dir, "TestSingleNode_Phase5_LaunchInstance")
+	if err := os.MkdirAll(perTest, 0o755); err != nil {
+		t.Fatalf("mkdir per-test: %v", err)
+	}
+	if err := os.WriteFile(filepath.Join(perTest, "qemu-console.log"), []byte("[qemu] boot\n"), 0o644); err != nil {
+		t.Fatalf("write per-test file: %v", err)
+	}
+
+	data, _ := os.ReadFile(filepath.Join(dir, "junit-single.xml"))
+	sr, err := ParseFile("junit-single.xml", data)
+	if err != nil {
+		t.Fatalf("ParseFile: %v", err)
+	}
+	rep := Report{Title: "E2E failure analysis", Suites: []SuiteReport{sr}}
+	WriteBundles(&rep, dir)
+
+	if rep.Suites[0].Root == nil {
+		t.Fatal("no root")
+	}
+	bp := rep.Suites[0].Root.BundlePath
+	if bp == "" {
+		t.Fatal("BundlePath not set")
+	}
+	bundle := filepath.Join(dir, bp)
+	for _, want := range []string{"failure.txt", "journal.log", "test.log", "qemu-console.log"} {
+		if _, err := os.Stat(filepath.Join(bundle, want)); err != nil {
+			t.Errorf("missing %s in bundle %s: %v", want, bp, err)
+		}
+	}
+
+	jl, _ := os.ReadFile(filepath.Join(bundle, "journal.log"))
+	if !strings.Contains(string(jl), "describe-vpcs filter=is_default") {
+		t.Errorf("journal.log lacks in-window line:\n%s", jl)
+	}
+	if strings.Contains(string(jl), "too-early") || strings.Contains(string(jl), "too-late") {
+		t.Errorf("journal.log includes out-of-window content:\n%s", jl)
+	}
+
+	tl, _ := os.ReadFile(filepath.Join(bundle, "test.log"))
+	if !strings.Contains(string(tl), "--- FAIL: TestSingleNode/Phase5_LaunchInstance") {
+		t.Errorf("test.log missing FAIL marker:\n%s", tl)
+	}
+
+	out := Render(rep)
+	wantLink := fmt.Sprintf("- Bundle: `%s/`", bp)
+	if !strings.Contains(out, wantLink) {
+		t.Errorf("render missing bundle link %q:\n%s", wantLink, out)
+	}
+}

.github/actions/e2e-analyze/main.go

@@ -0,0 +1,78 @@
+package main
+
+import (
+	"flag"
+	"fmt"
+	"os"
+	"path/filepath"
+	"sort"
+)
+
+func main() {
+	var junitGlob, logDir, title string
+	flag.StringVar(&junitGlob, "junit-glob", os.Getenv("ANALYZE_JUNIT_GLOB"), "glob matching junit-*.xml files")
+	flag.StringVar(&logDir, "log-dir", os.Getenv("ANALYZE_LOG_DIR"), "artifact dir to write analysis.md into")
+	flag.StringVar(&title, "title", os.Getenv("ANALYZE_TITLE"), "report heading")
+	flag.Parse()
+
+	if title == "" {
+		title = "E2E failure analysis"
+	}
+	if junitGlob == "" {
+		fmt.Fprintln(os.Stderr, "::error::e2e-analyze: -junit-glob is required")
+		os.Exit(0) // analyzer is advisory only; existing gate owns exit code
+	}
+
+	paths, err := filepath.Glob(junitGlob)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: bad glob %q: %v\n", junitGlob, err)
+		os.Exit(0)
+	}
+	sort.Strings(paths)
+
+	report := Report{Title: title}
+	for _, p := range paths {
+		data, err := os.ReadFile(p)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: read %s: %v\n", p, err)
+			continue
+		}
+		sr, err := ParseFile(p, data)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: %v\n", err)
+			continue
+		}
+		report.Suites = append(report.Suites, sr)
+	}
+
+	// Stage 2: prefer authoritative per-suite start times written by the
+	// workflow over junit-report's conversion-time `timestamp` attribute,
+	// then materialise per-failure bundles so Render can link them.
+	if logDir != "" {
+		ApplySuiteStartFiles(&report, logDir)
+		WriteBundles(&report, logDir)
+	}
+
+	out := Render(report)
+
+	if logDir != "" {
+		if err := os.MkdirAll(logDir, 0o755); err == nil {
+			if err := os.WriteFile(filepath.Join(logDir, "analysis.md"), []byte(out), 0o644); err != nil {
+				fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: write analysis.md: %v\n", err)
+			}
+		}
+	}
+
+	if summaryPath := os.Getenv("GITHUB_STEP_SUMMARY"); summaryPath != "" {
+		f, err := os.OpenFile(summaryPath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
+		if err == nil {
+			defer f.Close()
+			if _, werr := f.WriteString(out); werr != nil {
+				fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: append summary: %v\n", werr)
+			}
+		}
+	} else {
+		// Local invocation (developer running the tool by hand). Mirror to stdout.
+		fmt.Print(out)
+	}
+}

.github/actions/e2e-analyze/slice.go

@@ -0,0 +1,176 @@
+package main
+
+// Stage 2 of docs/development/improvements/e2e-go-failure-analysis.md:
+// per-failure log slicers used by bundle.go to materialise time-window /
+// testcase-window log excerpts next to each named failure.
+
+import (
+	"bufio"
+	"io"
+	"os"
+	"regexp"
+	"strings"
+	"time"
+)
+
+// shortIsoTSRe matches the timestamp prefix produced by
+// `journalctl --output=short-iso`. The offset format varies across
+// systemd versions: most ship "+10:00" (RFC3339), some ship "+1000"
+// (no colon). Match either — the parser below picks the right layout.
+var shortIsoTSRe = regexp.MustCompile(`^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:[+-]\d{2}:?\d{2}|Z))\s`)
+
+// shortIsoLayouts are tried in order; the first parse that succeeds wins.
+var shortIsoLayouts = []string{
+	time.RFC3339,               // 2026-05-19T12:32:30+10:00
+	"2006-01-02T15:04:05-0700", // 2026-05-19T12:32:30+1000
+}
+
+func parseShortIso(s string) (time.Time, error) {
+	var lastErr error
+	for _, layout := range shortIsoLayouts {
+		t, err := time.Parse(layout, s)
+		if err == nil {
+			return t, nil
+		}
+		lastErr = err
+	}
+	return time.Time{}, lastErr
+}
+
+// SliceJournal copies lines from src whose `short-iso` timestamp falls in
+// [start, end] to dst. Lines without a parseable timestamp inherit the
+// previous line's classification — that keeps multi-line journal records
+// (stack traces, panics) intact rather than orphaning their continuation
+// lines. Lines that precede the first parseable timestamp are dropped.
+//
+// Returns the number of lines written; missing src is not an error (the
+// journal may be empty if collection failed).
+func SliceJournal(src string, start, end time.Time, dst string) (int, error) {
+	f, err := os.Open(src)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return 0, nil
+		}
+		return 0, err
+	}
+	defer f.Close()
+
+	out, err := os.Create(dst)
+	if err != nil {
+		return 0, err
+	}
+	defer out.Close()
+
+	bw := bufio.NewWriter(out)
+	defer bw.Flush()
+
+	sc := bufio.NewScanner(f)
+	// Journal lines can be long (panic dumps); bump scanner buffer.
+	sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
+
+	keep := false
+	written := 0
+	for sc.Scan() {
+		line := sc.Text()
+		if m := shortIsoTSRe.FindStringSubmatch(line); m != nil {
+			ts, err := parseShortIso(m[1])
+			if err != nil {
+				continue
+			}
+			keep = !ts.Before(start) && !ts.After(end)
+		}
+		if keep {
+			if _, err := bw.WriteString(line); err != nil {
+				return written, err
+			}
+			if err := bw.WriteByte('\n'); err != nil {
+				return written, err
+			}
+			written++
+		}
+	}
+	return written, sc.Err()
+}
+
+// runRePrefix and finishRePrefix bracket a single testcase's output in
+// `go test -v` stdout. Subtests have their own `=== RUN` / `--- FAIL`
+// markers, so slicing on the exact testcase name yields just that test's
+// lines (including any nested phase logs).
+var (
+	runRe    = regexp.MustCompile(`^=== RUN\s+(\S+)$`)
+	finishRe = regexp.MustCompile(`^\s*---\s+(PASS|FAIL|SKIP):\s+(\S+)\s+\(`)
+)
+
+// SliceTestLog copies lines belonging to the named testcase from src to
+// dst. The block starts at `=== RUN <name>` and ends at the matching
+// `--- PASS|FAIL|SKIP: <name> (…)` line (inclusive). If the testcase is
+// never seen the destination file is created empty so the bundle has a
+// stable shape.
+//
+// Returns the number of lines written; missing src is not an error.
+func SliceTestLog(src, testName, dst string) (int, error) {
+	f, err := os.Open(src)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return 0, nil
+		}
+		return 0, err
+	}
+	defer f.Close()
+
+	out, err := os.Create(dst)
+	if err != nil {
+		return 0, err
+	}
+	defer out.Close()
+
+	bw := bufio.NewWriter(out)
+	defer bw.Flush()
+
+	sc := bufio.NewScanner(f)
+	sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
+
+	in := false
+	written := 0
+	for sc.Scan() {
+		line := sc.Text()
+		if !in {
+			if m := runRe.FindStringSubmatch(strings.TrimRight(line, "\r")); m != nil && m[1] == testName {
+				in = true
+			} else {
+				continue
+			}
+		}
+		if _, err := bw.WriteString(line); err != nil {
+			return written, err
+		}
+		if err := bw.WriteByte('\n'); err != nil {
+			return written, err
+		}
+		written++
+		if m := finishRe.FindStringSubmatch(line); m != nil && m[2] == testName {
+			return written, nil
+		}
+	}
+	return written, sc.Err()
+}
+
+// copyFile is a small helper used when bundling pre-collected per-test
+// artifacts. Best-effort: missing src is silently skipped.
+func copyFile(src, dst string) error {
+	in, err := os.Open(src)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return nil
+		}
+		return err
+	}
+	defer in.Close()
+	out, err := os.Create(dst)
+	if err != nil {
+		return err
+	}
+	defer out.Close()
+	_, err = io.Copy(out, in)
+	return err
+}

.github/actions/e2e-analyze/testdata/cascade-vpcs/analysis.md

@@ -0,0 +1,23 @@
+## E2E failure analysis
+
+### Suite `single`: 16 failed, 1 root cause likely
+
+**Root cause (earliest non-cascade)**
+
+- Test: `TestSingleNode/Phase5_LaunchInstance`
+- Start: 12:30:00 (duration 2.5s)
+- Error: Received unexpected error: InvalidParameterValue (HTTP 400)
+- File:  `tests/e2e/harness/ec2helpers.go:164`
+
+**Cascaded failures (15) — grouped by signature**
+
+- 14× Phase 5 must populate fix.InstanceID
+  - `TestSingleNode/Phase5a_pre_ClusterStats`
+  - `TestSingleNode/Phase5a_Metadata`
+  - `TestSingleNode/Phase5b_DescribeInstances`
+  - `TestSingleNode/Phase5c_StopStart`
+  - `TestSingleNode/Phase6_Volumes`
+  - … and 9 more
+- 1× Phase 2 must populate fix.AZName
+  - `TestSingleNode/Phase8_NatGW`
+

.github/actions/e2e-analyze/testdata/cascade-vpcs/junit-single.xml

@@ -0,0 +1,143 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<testsuites tests="20" failures="17" skipped="0">
+	<testsuite name="" tests="20" failures="17" errors="0" id="0" hostname="banksia" skipped="0" time="600.000" timestamp="2026-04-12T12:30:00+10:00">
+		<testcase name="TestSingleNode" classname="" time="600.000">
+			<failure message="Failed"></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase1_Environment" classname="" time="0.300">
+			<system-out><![CDATA[
+&#x2501;&#x2501; Phase 1 &#x2501;&#x2501;]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase2_Discovery" classname="" time="0.010"/>
+		<testcase name="TestSingleNode/Phase3_KeyPairs" classname="" time="0.540"/>
+		<testcase name="TestSingleNode/Phase4_Image" classname="" time="0.060"/>
+		<testcase name="TestSingleNode/Phase5_LaunchInstance" classname="" time="2.500">
+			<failure message="Failed"><![CDATA[
+    ec2helpers.go:164: describe-vpcs: InvalidParameterValue: Filter 'is_default' is invalid (status 400)
+    	Error Trace:	tests/e2e/harness/ec2helpers.go:164
+    	Error:      	Received unexpected error:
+    	            	InvalidParameterValue (HTTP 400)
+    	Test:       	TestSingleNode/Phase5_LaunchInstance]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_pre_ClusterStats" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    lifecycle_test.go:28:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:28
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5a_pre_ClusterStats]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_Metadata" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    lifecycle_test.go:57:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:57
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5a_Metadata]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5b_DescribeInstances" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    lifecycle_test.go:109:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:109
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5b_DescribeInstances]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5c_StopStart" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    lifecycle_test.go:222:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:222
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5c_StopStart]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase6_Volumes" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    volume_test.go:23:
+        Error Trace:	tests/e2e/single/volume_test.go:23
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase6_Volumes]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase6b_AttachVolume" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    volume_test.go:24:
+        Error Trace:	tests/e2e/single/volume_test.go:24
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase6b_AttachVolume]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7_Tags" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    tags_test.go:23:
+        Error Trace:	tests/e2e/single/tags_test.go:23
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7_Tags]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7a_Snapshots" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    snapshot_test.go:23:
+        Error Trace:	tests/e2e/single/snapshot_test.go:23
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7a_Snapshots]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7b_DescribeInstanceAttribute" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    instance_test.go:81:
+        Error Trace:	tests/e2e/single/instance_test.go:81
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7b_DescribeInstanceAttribute]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7c_ModifyInstanceAttribute" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    instance_test.go:92:
+        Error Trace:	tests/e2e/single/instance_test.go:92
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7c_ModifyInstanceAttribute]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7d_Console" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    instance_test.go:121:
+        Error Trace:	tests/e2e/single/instance_test.go:121
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7d_Console]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7e_Negative" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    negative_test.go:43:
+        Error Trace:	tests/e2e/single/negative_test.go:43
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7e_Negative]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NatGW" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    natgw_test.go:39:
+        Error Trace:	tests/e2e/single/natgw_test.go:39
+        Error:      	Should NOT be empty
+        Messages:   	Phase 2 must populate fix.AZName
+        Test:       	TestSingleNode/Phase8_NatGW]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8a_DataPath" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    datapath_test.go:48:
+        Error Trace:	tests/e2e/single/datapath_test.go:48
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase8a_DataPath]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase9_SecurityGroups" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    securitygroup_test.go:35:
+        Error Trace:	tests/e2e/single/securitygroup_test.go:35
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase9_SecurityGroups]]></failure>
+		</testcase>
+	</testsuite>
+</testsuites>

.github/actions/e2e-analyze/testdata/no-failures/analysis.md

@@ -0,0 +1,3 @@
+## E2E failure analysis
+
+✅ No failures across any suite.

.github/actions/e2e-analyze/testdata/no-failures/junit-single.xml

@@ -0,0 +1,8 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<testsuites tests="3" failures="0" skipped="0">
+	<testsuite name="" tests="3" failures="0" errors="0" id="0" hostname="banksia" skipped="0" time="1.500" timestamp="2026-04-12T12:30:00+10:00">
+		<testcase name="TestSingleNode" classname="" time="1.500"/>
+		<testcase name="TestSingleNode/Phase1_Environment" classname="" time="0.300"/>
+		<testcase name="TestSingleNode/Phase2_Discovery" classname="" time="1.200"/>
+	</testsuite>
+</testsuites>

.github/actions/e2e-analyze/testdata/single-failure/analysis.md

@@ -0,0 +1,11 @@
+## E2E failure analysis
+
+### Suite `single`: 1 failed, 1 root cause likely
+
+**Root cause (earliest non-cascade)**
+
+- Test: `TestSingleNode/Phase8b_VPCSubnetE2E`
+- Start: 00:26:18 (duration 200.5s)
+- Error: Eventually: condition not met within 3m0s: [SSH handshake 192.168.0.202:22 never completed]
+- File:  `vpc_test.go:227`
+

.github/actions/e2e-analyze/testdata/single-failure/junit-single.xml

@@ -0,0 +1,993 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<testsuites tests="79" failures="2" skipped="6">
+	<testsuite name="" tests="79" failures="2" errors="0" id="0" hostname="banksia" skipped="6" time="1708.140" timestamp="2026-05-19T00:19:45+10:00">
+		<testcase name="TestSingleNode" classname="" time="800.320">
+			<failure message="Failed"></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase1_Environment" classname="" time="0.300">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 1 — Environment ━━�[0m
+�[2m· checking /dev/kvm�[0m
+    discovery_test.go:29: �[2m· checking /dev/kvm�[0m
+�[2mkvm�[0m=writable
+    discovery_test.go:38: �[2mkvm�[0m=writable
+�[2m· waiting for AWS gateway�[0m
+    discovery_test.go:45: �[2m· waiting for AWS gateway�[0m
+�[2mgateway�[0m=https://192.168.0.10:9999/
+    discovery_test.go:55: �[2mgateway�[0m=https://192.168.0.10:9999/
+�[2m· waiting for daemon NATS subscriptions�[0m
+    discovery_test.go:60: �[2m· waiting for daemon NATS subscriptions�[0m
+�[2m· describe-regions / describe-availability-zones�[0m
+    discovery_test.go:73: �[2m· describe-regions / describe-availability-zones�[0m
+�[2maz�[0m=ap-southeast-2a �[2mregion�[0m=ap-southeast-2
+    discovery_test.go:84: �[2maz�[0m=ap-southeast-2a �[2mregion�[0m=ap-southeast-2]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase1b_ClusterStats" classname="" time="9.200">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 1b — Cluster Stats CLI ━━�[0m
+�[2m· spx get nodes�[0m
+    discovery_test.go:101: �[2m· spx get nodes�[0m
+�[2m· spx top nodes�[0m
+    discovery_test.go:105: �[2m· spx top nodes�[0m
+�[2m· spx get vms (empty before any launches)�[0m
+    discovery_test.go:112: �[2m· spx get vms (empty before any launches)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase2_Discovery" classname="" time="0.010">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 2 — Discovery & Metadata ━━�[0m
+�[2m· discovering nano instance type�[0m
+    discovery_test.go:133: �[2m· discovering nano instance type�[0m
+�[2minstance_type�[0m=t3.nano �[2march�[0m=x86_64 �[2maz�[0m=ap-southeast-2a
+    discovery_test.go:159: �[2minstance_type�[0m=t3.nano �[2march�[0m=x86_64 �[2maz�[0m=ap-southeast-2a]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase2b_SerialConsole" classname="" time="0.020">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 2b — Serial Console Access ━━�[0m
+�[2m· default state should be disabled�[0m
+    discovery_test.go:168: �[2m· default state should be disabled�[0m
+�[2m· enable�[0m
+    discovery_test.go:173: �[2m· enable�[0m
+�[2mstate�[0m=enabled
+    discovery_test.go:180: �[2mstate�[0m=enabled
+�[2m· disable�[0m
+    discovery_test.go:182: �[2m· disable�[0m
+�[2mstate�[0m=disabled
+    discovery_test.go:189: �[2mstate�[0m=disabled]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase3_KeyPairs" classname="" time="0.540">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 3 — SSH Key Management ━━�[0m
+�[2m· create-key-pair "test-key-1"�[0m
+    keypair_test.go:37: �[2m· create-key-pair "test-key-1"�[0m
+�[2mkey�[0m=test-key-1 �[2mpem�[0m=/tmp/TestSingleNode1289015487/001/test-key-1.pem
+    keypair_test.go:51: �[2mkey�[0m=test-key-1 �[2mpem�[0m=/tmp/TestSingleNode1289015487/001/test-key-1.pem
+�[2m· import-key-pair "test-key-2" (local-generated RSA)�[0m
+    keypair_test.go:58: �[2m· import-key-pair "test-key-2" (local-generated RSA)�[0m
+�[2mimported�[0m=test-key-2
+    keypair_test.go:65: �[2mimported�[0m=test-key-2
+�[2m· describe-key-pairs (both present)�[0m
+    keypair_test.go:67: �[2m· describe-key-pairs (both present)�[0m
+�[2m· delete "test-key-2"�[0m
+    keypair_test.go:72: �[2m· delete "test-key-2"�[0m
+�[2m· describe-key-pairs (only test-key-1 remains)�[0m
+    keypair_test.go:76: �[2m· describe-key-pairs (only test-key-1 remains)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase4_Image" classname="" time="0.060">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 4 — Image Management ━━�[0m
+�[2mimage�[0m=ubuntu-24.04-x86_64 �[2march�[0m=x86_64
+    image_test.go:33: �[2mimage�[0m=ubuntu-24.04-x86_64 �[2march�[0m=x86_64
+�[2m· spx admin images import --name ubuntu-24.04-x86_64�[0m
+    image_test.go:35: �[2m· spx admin images import --name ubuntu-24.04-x86_64�[0m
+�[2m· spx import did not yield AMI ID; falling back to describe-images�[0m
+    image_test.go:48: �[2m· spx import did not yield AMI ID; falling back to describe-images�[0m
+�[2m· describe-images by AMI ID (verify exactly one match)�[0m
+    image_test.go:78: �[2m· describe-images by AMI ID (verify exactly one match)�[0m
+�[2mami�[0m=ami-6f60f2b3bf04b5453 �[2mname�[0m=ami-ubuntu-24.04-x86_64
+    image_test.go:86: �[2mami�[0m=ami-6f60f2b3bf04b5453 �[2mname�[0m=ami-ubuntu-24.04-x86_64]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5_LaunchInstance" classname="" time="10.830">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5 — Instance Lifecycle ━━�[0m
+�[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msg�[0m=sg-764a7fa2a39577204 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63
+    instance_test.go:31: �[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msg�[0m=sg-764a7fa2a39577204 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63
+�[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.nano key=test-key-1�[0m
+    instance_test.go:35: �[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.nano key=test-key-1�[0m
+�[2minstance�[0m=i-0c615e5dd74050f53
+    instance_test.go:48: �[2minstance�[0m=i-0c615e5dd74050f53
+    instance_test.go:50: instance i-0c615e5dd74050f53 reached state running
+�[2mroot_volume�[0m=vol-2230138124fb51ee2 �[2mroot_device�[0m=
+    instance_test.go:70: �[2mroot_volume�[0m=vol-2230138124fb51ee2 �[2mroot_device�[0m=]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_pre_ClusterStats" classname="" time="3.040">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5a-pre — Cluster Stats CLI (with running VM) ━━�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_Metadata" classname="" time="0.510">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5a — Instance Metadata Validation ━━�[0m
+�[2mtype�[0m=t3.nano �[2mkey�[0m=test-key-1 �[2mimage�[0m=ami-6f60f2b3bf04b5453 �[2mbdms�[0m=1
+    instance_test.go:108: �[2mtype�[0m=t3.nano �[2mkey�[0m=test-key-1 �[2mimage�[0m=ami-6f60f2b3bf04b5453 �[2mbdms�[0m=1]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_ii_SSH" classname="" time="49.450">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5a-ii — SSH Connectivity & Volume Verification ━━�[0m
+�[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
+    instance_test.go:127: �[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    instance_test.go:130: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    lifecycle_test.go:384: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2m· ssh id�[0m
+    instance_test.go:136: �[2m· ssh id�[0m
+�[2m· lsblk root-volume cross-check vs API�[0m
+    instance_test.go:140: �[2m· lsblk root-volume cross-check vs API�[0m
+�[2mguest_gib�[0m=4 �[2mapi_gib�[0m=4
+    instance_test.go:159: �[2mguest_gib�[0m=4 �[2mapi_gib�[0m=4
+�[2m· ssh hostname�[0m
+    instance_test.go:161: �[2m· ssh hostname�[0m
+�[2mhostname�[0m=spinifex-vm-0c615e5d �[2mmatches_short_id�[0m=0c615e5d
+    instance_test.go:172: �[2mhostname�[0m=spinifex-vm-0c615e5d �[2mmatches_short_id�[0m=0c615e5d]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_iii_Console" classname="" time="0.010">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5a-iii — Console Output ━━�[0m
+�[2minstance�[0m=i-0c615e5dd74050f53 �[2mhas_output�[0m=true
+    instance_test.go:191: �[2minstance�[0m=i-0c615e5dd74050f53 �[2mhas_output�[0m=true]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5b_Volume" classname="" time="4.540">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5b — Volume Lifecycle (Attach/Detach) ━━�[0m
+�[2m· create-volume size=10 az=ap-southeast-2a�[0m
+    volume_test.go:26: �[2m· create-volume size=10 az=ap-southeast-2a�[0m
+�[2mvolume�[0m=vol-a885f79173e0a8030
+    volume_test.go:34: �[2mvolume�[0m=vol-a885f79173e0a8030
+    volume_test.go:39: volume vol-a885f79173e0a8030 reached state available
+�[2m· modify-volume size=20�[0m
+    volume_test.go:42: �[2m· modify-volume size=20�[0m
+�[2mresized_gib�[0m=20
+    volume_test.go:66: �[2mresized_gib�[0m=20
+�[2m· attach-volume vol-a885f79173e0a8030 -> i-0c615e5dd74050f53 as /dev/sdf�[0m
+    volume_test.go:68: �[2m· attach-volume vol-a885f79173e0a8030 -> i-0c615e5dd74050f53 as /dev/sdf�[0m
+    volume_test.go:76: volume vol-a885f79173e0a8030 reached state in-use
+�[2m· detach-volume vol-a885f79173e0a8030�[0m
+    volume_test.go:90: �[2m· detach-volume vol-a885f79173e0a8030�[0m
+    volume_test.go:96: volume vol-a885f79173e0a8030 reached state available
+�[2m· delete-volume vol-a885f79173e0a8030�[0m
+    volume_test.go:98: �[2m· delete-volume vol-a885f79173e0a8030�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5b_ii_VolumeStatus" classname="" time="0.000">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5b-ii — DescribeVolumeStatus ━━�[0m
+�[2mvolume�[0m=vol-2230138124fb51ee2 �[2mstatus�[0m=ok
+    volume_test.go:158: �[2mvolume�[0m=vol-2230138124fb51ee2 �[2mstatus�[0m=ok]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5c_Snapshot" classname="" time="0.190">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5c — Snapshot Lifecycle ━━�[0m
+�[2m· create-snapshot volume=vol-2230138124fb51ee2�[0m
+    snapshot_test.go:34: �[2m· create-snapshot volume=vol-2230138124fb51ee2�[0m
+�[2msnapshot�[0m=snap-1188f92980e6470aa �[2msize�[0m=4 �[2mstate�[0m=completed �[2mprogress�[0m=100%
+    snapshot_test.go:50: �[2msnapshot�[0m=snap-1188f92980e6470aa �[2msize�[0m=4 �[2mstate�[0m=completed �[2mprogress�[0m=100%
+    snapshot_test.go:59: snapshot snap-1188f92980e6470aa reached state completed
+�[2m· describe-snapshots snap-1188f92980e6470aa�[0m
+    snapshot_test.go:61: �[2m· describe-snapshots snap-1188f92980e6470aa�[0m
+�[2m· copy-snapshot src=snap-1188f92980e6470aa region=ap-southeast-2�[0m
+    snapshot_test.go:76: �[2m· copy-snapshot src=snap-1188f92980e6470aa region=ap-southeast-2�[0m
+�[2mcopy_snapshot�[0m=snap-cde6b5ddf467e2c0f
+    snapshot_test.go:86: �[2mcopy_snapshot�[0m=snap-cde6b5ddf467e2c0f
+    snapshot_test.go:88: snapshot snap-cde6b5ddf467e2c0f reached state completed
+�[2m· describe-snapshots snap-1188f92980e6470aa,snap-cde6b5ddf467e2c0f (both visible)�[0m
+    snapshot_test.go:90: �[2m· describe-snapshots snap-1188f92980e6470aa,snap-cde6b5ddf467e2c0f (both visible)�[0m
+�[2m· delete-snapshot snap-1188f92980e6470aa (original)�[0m
+    snapshot_test.go:107: �[2m· delete-snapshot snap-1188f92980e6470aa (original)�[0m
+�[2m· verify copy snap-cde6b5ddf467e2c0f still completed�[0m
+    snapshot_test.go:135: �[2m· verify copy snap-cde6b5ddf467e2c0f still completed�[0m
+�[2m· delete-snapshot snap-cde6b5ddf467e2c0f (copy)�[0m
+    snapshot_test.go:144: �[2m· delete-snapshot snap-cde6b5ddf467e2c0f (copy)�[0m
+�[2mdeleted_original�[0m=snap-1188f92980e6470aa
+    snapshot_test.go:151: �[2mdeleted_original�[0m=snap-1188f92980e6470aa]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5d_SnapshotBackedLaunch" classname="" time="0.010">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5d — Snapshot-Backed Instance Launch ━━�[0m
+�[2mami�[0m=ami-6f60f2b3bf04b5453 �[2msnapshot�[0m=snap-ami-6f60f2b3bf04b5453
+    snapshot_test.go:182: �[2mami�[0m=ami-6f60f2b3bf04b5453 �[2msnapshot�[0m=snap-ami-6f60f2b3bf04b5453]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5e_CreateImage" classname="" time="0.660">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5e — CreateImage Lifecycle ━━�[0m
+�[2m· create-image instance=i-0c615e5dd74050f53 name=e2e-custom-ami (no-reboot)�[0m
+    createimage_test.go:26: �[2m· create-image instance=i-0c615e5dd74050f53 name=e2e-custom-ami (no-reboot)�[0m
+�[2mcustom_ami�[0m=ami-7dcf263a412dd14c6
+    createimage_test.go:36: �[2mcustom_ami�[0m=ami-7dcf263a412dd14c6
+    createimage_test.go:38: image ami-7dcf263a412dd14c6 reached state available
+�[2m· describe-images ami-7dcf263a412dd14c6�[0m
+    createimage_test.go:40: �[2m· describe-images ami-7dcf263a412dd14c6�[0m
+�[2mcustom_ami_snapshot�[0m=snap-dde89b9ea2ef2e05a
+    createimage_test.go:67: �[2mcustom_ami_snapshot�[0m=snap-dde89b9ea2ef2e05a]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5f_SecurityGroupEgress" classname="" time="4.880">
+			<skipped message="Skipped"><![CDATA[
+�[1m�[36m━━ Phase 5f — Security Group Enforcement (egress ACL) ━━�[0m
+�[2m· discover default gateway inside VM�[0m
+    securitygroup_test.go:53: �[2m· discover default gateway inside VM�[0m
+�[2mprobe_gateway�[0m=172.31.0.1
+    securitygroup_test.go:66: �[2mprobe_gateway�[0m=172.31.0.1
+�[2m· 5f-1 baseline egress (allow-all)�[0m
+    securitygroup_test.go:77: �[2m· 5f-1 baseline egress (allow-all)�[0m
+    securitygroup_test.go:80: Phase 5f: baseline ICMP did not succeed; env may block ICMP regardless of SG
+        Output:
+        PING 172.31.0.1 (172.31.0.1) 56(84) bytes of data.
+        
+        --- 172.31.0.1 ping statistics ---
+        3 packets transmitted, 0 received, 100% packet loss, time 2044ms
+        ]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/Phase6_TagManagement" classname="" time="0.380">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 6 — Tag Management ━━�[0m
+�[2m· 6a create-tags instance=i-0c615e5dd74050f53 (Name, Environment, DeleteMe)�[0m
+    tags_test.go:41: �[2m· 6a create-tags instance=i-0c615e5dd74050f53 (Name, Environment, DeleteMe)�[0m
+�[2m· 6b describe-tags resource-id=i-0c615e5dd74050f53 want=3�[0m
+    tags_test.go:53: �[2m· 6b describe-tags resource-id=i-0c615e5dd74050f53 want=3�[0m
+�[2m· 6c create-tags volume=vol-2230138124fb51ee2 (Name, Environment)�[0m
+    tags_test.go:65: �[2m· 6c create-tags volume=vol-2230138124fb51ee2 (Name, Environment)�[0m
+�[2m· 6d describe-tags key=Environment want=2�[0m
+    tags_test.go:76: �[2m· 6d describe-tags key=Environment want=2�[0m
+�[2m· 6e describe-tags resource-type=instance want=3�[0m
+    tags_test.go:85: �[2m· 6e describe-tags resource-type=instance want=3�[0m
+�[2m· 6f overwrite instance Name -> e2e-test-updated�[0m
+    tags_test.go:95: �[2m· 6f overwrite instance Name -> e2e-test-updated�[0m
+�[2m· 6g delete-tags Key=DeleteMe (no value)�[0m
+    tags_test.go:114: �[2m· 6g delete-tags Key=DeleteMe (no value)�[0m
+�[2m· 6h delete-tags Key=Environment,Value=production (no-op)�[0m
+    tags_test.go:130: �[2m· 6h delete-tags Key=Environment,Value=production (no-op)�[0m
+�[2m· 6i delete-tags Key=Environment,Value=testing (instance only)�[0m
+    tags_test.go:148: �[2m· 6i delete-tags Key=Environment,Value=testing (instance only)�[0m
+�[2m· 6j final describe-tags resource-id=i-0c615e5dd74050f53 want=1�[0m
+    tags_test.go:171: �[2m· 6j final describe-tags resource-id=i-0c615e5dd74050f53 want=1�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7_StopStart" classname="" time="8.090">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7 — Instance State Transitions (Stop) ━━�[0m
+�[2m· stop-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:30: �[2m· stop-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:36: instance i-0c615e5dd74050f53 reached state stopped
+�[2m· reboot-instances on stopped instance (should fail)�[0m
+    lifecycle_test.go:40: �[2m· reboot-instances on stopped instance (should fail)�[0m
+�[2mstopped_reject�[0m=IncorrectInstanceState
+    lifecycle_test.go:47: �[2mstopped_reject�[0m=IncorrectInstanceState]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7a_AttachToStoppedError" classname="" time="0.300">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7a — Attach Volume to Stopped Instance (Error Path) ━━�[0m
+�[2m· create-volume size=10 az=ap-southeast-2a�[0m
+    lifecycle_test.go:60: �[2m· create-volume size=10 az=ap-southeast-2a�[0m
+�[2mtest_volume�[0m=vol-8d3e1335938a378cd
+    lifecycle_test.go:68: �[2mtest_volume�[0m=vol-8d3e1335938a378cd
+    lifecycle_test.go:82: volume vol-8d3e1335938a378cd reached state available
+�[2m· attach-volume vol-8d3e1335938a378cd -> i-0c615e5dd74050f53 (should fail)�[0m
+    lifecycle_test.go:84: �[2m· attach-volume vol-8d3e1335938a378cd -> i-0c615e5dd74050f53 (should fail)�[0m
+�[2m· delete-volume vol-8d3e1335938a378cd�[0m
+    lifecycle_test.go:94: �[2m· delete-volume vol-8d3e1335938a378cd�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7b_ModifyInstanceAttribute" classname="" time="55.620">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7b — ModifyInstanceAttribute ━━�[0m
+�[2mfrom_type�[0m=t3.nano �[2mto_type�[0m=t3.small
+    lifecycle_test.go:120: �[2mfrom_type�[0m=t3.nano �[2mto_type�[0m=t3.small
+�[2mexpected_vcpus�[0m=2 �[2mexpected_mem_mib�[0m=2048
+    lifecycle_test.go:142: �[2mexpected_vcpus�[0m=2 �[2mexpected_mem_mib�[0m=2048
+�[2m· modify-instance-attribute i-0c615e5dd74050f53 type=t3.small�[0m
+    lifecycle_test.go:144: �[2m· modify-instance-attribute i-0c615e5dd74050f53 type=t3.small�[0m
+�[2m· start-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:165: �[2m· start-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:171: instance i-0c615e5dd74050f53 reached state running
+�[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
+    lifecycle_test.go:177: �[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    lifecycle_test.go:179: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2m· ssh nproc�[0m
+    lifecycle_test.go:182: �[2m· ssh nproc�[0m
+�[2mvm_vcpus�[0m=2
+    lifecycle_test.go:188: �[2mvm_vcpus�[0m=2
+�[2m· ssh MemTotal�[0m
+    lifecycle_test.go:190: �[2m· ssh MemTotal�[0m
+�[2mvm_mem_mib�[0m=1967 �[2mthreshold_mib�[0m=1740
+    lifecycle_test.go:200: �[2mvm_mem_mib�[0m=1967 �[2mthreshold_mib�[0m=1740
+�[2m· stop-instances i-0c615e5dd74050f53 (end of 7b)�[0m
+    lifecycle_test.go:206: �[2m· stop-instances i-0c615e5dd74050f53 (end of 7b)�[0m
+    lifecycle_test.go:211: instance i-0c615e5dd74050f53 reached state stopped]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7c_pre_Reboot" classname="" time="83.480">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7c-pre — Reboot Running Instance ━━�[0m
+�[2m· start-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:226: �[2m· start-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:231: instance i-0c615e5dd74050f53 reached state running
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    lifecycle_test.go:236: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2mpre_reboot_private_ip�[0m=172.31.0.4
+    lifecycle_test.go:246: �[2mpre_reboot_private_ip�[0m=172.31.0.4
+�[2m· reboot-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:248: �[2m· reboot-instances i-0c615e5dd74050f53�[0m
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    lifecycle_test.go:279: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2m· ssh uptime�[0m
+    lifecycle_test.go:282: �[2m· ssh uptime�[0m
+�[2muptime_secs�[0m=25
+    lifecycle_test.go:288: �[2muptime_secs�[0m=25]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7c_RunInstancesMultiCount" classname="" time="47.560">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7c — RunInstances with MinCount/MaxCount > 1 ━━�[0m
+�[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.small count=2�[0m
+    lifecycle_test.go:309: �[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.small count=2�[0m
+�[2msibling_1�[0m=i-9bfadd8ebd378f846 �[2msibling_2�[0m=i-448a9e2bf45c58658
+    lifecycle_test.go:327: �[2msibling_1�[0m=i-9bfadd8ebd378f846 �[2msibling_2�[0m=i-448a9e2bf45c58658
+    lifecycle_test.go:342: instance i-9bfadd8ebd378f846 reached state running
+    lifecycle_test.go:342: instance i-448a9e2bf45c58658 reached state running
+�[2m· terminate-instances i-9bfadd8ebd378f846 i-448a9e2bf45c58658�[0m
+    lifecycle_test.go:345: �[2m· terminate-instances i-9bfadd8ebd378f846 i-448a9e2bf45c58658�[0m
+    lifecycle_test.go:352: instance i-9bfadd8ebd378f846 reached state terminated
+    lifecycle_test.go:352: instance i-448a9e2bf45c58658 reached state terminated]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths" classname="" time="0.000">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8 — Negative / Error Path Tests ━━�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8a_InvalidAMIIDMalformed" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· run-instances image-id=notanami�[0m
+    negative_test.go:55: �[2m· run-instances image-id=notanami�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8b_InvalidInstanceType" classname="" time="0.210">
+			<system-out><![CDATA[�[2m· run-instances instance-type=x99.superlarge�[0m
+    negative_test.go:70: �[2m· run-instances instance-type=x99.superlarge�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8c_VolumeInUse" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· attach-volume vol-2230138124fb51ee2 (root, in-use)�[0m
+    negative_test.go:85: �[2m· attach-volume vol-2230138124fb51ee2 (root, in-use)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8d_DetachRootForbidden" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· detach-volume vol-2230138124fb51ee2 (root)�[0m
+    negative_test.go:97: �[2m· detach-volume vol-2230138124fb51ee2 (root)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8e_DeleteSnapshotNotFound" classname="" time="0.030">
+			<system-out><![CDATA[�[2m· delete-snapshot snap-nonexistent000000�[0m
+    negative_test.go:108: �[2m· delete-snapshot snap-nonexistent000000�[0m
+�[2m· describe-volumes vol-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8f_InvalidActionRawHTTP" classname="" time="0.040">
+			<system-out><![CDATA[�[2m· POST Action=DescribeFakeThings�[0m
+    negative_test.go:122: �[2m· POST Action=DescribeFakeThings�[0m
+    negative_test.go:125: �[2mstatus�[0m=400 �[2mcode�[0m=InvalidAction]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8g_InvalidAMIIDNotFound" classname="" time="0.240">
+			<system-out><![CDATA[�[2m· reboot-instances i-nonexistent�[0m
+�[2m· run-instances image-id=ami-0000000000000dead�[0m
+    negative_test.go:138: �[2m· run-instances image-id=ami-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8h_InvalidKeyPairNotFound" classname="" time="0.260">
+			<system-out><![CDATA[�[2m· run-instances key-name=nonexistent-key-xyz�[0m
+    negative_test.go:153: �[2m· run-instances key-name=nonexistent-key-xyz�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8i_DeleteVolumeNotFound" classname="" time="0.040">
+			<system-out><![CDATA[�[2m· delete-volume vol-0000000000000dead�[0m
+    negative_test.go:168: �[2m· delete-volume vol-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8j_CreateKeyPairDuplicate" classname="" time="0.040">
+			<system-out><![CDATA[    negative_test.go:178: �[2m· create-key-pair test-key-1 (duplicate)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8k_ImportKeyPairDuplicate" classname="" time="0.040">
+			<system-out><![CDATA[�[2m· import-key-pair test-key-1 (duplicate)�[0m
+    negative_test.go:198: �[2m· import-key-pair test-key-1 (duplicate)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8l_InvalidKeyFormat" classname="" time="0.040">
+			<system-out><![CDATA[    negative_test.go:210: �[2m· import-key-pair bad-format-key (bad material)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8m_DescribeVolumesNotFound" classname="" time="0.040">
+			<system-out><![CDATA[    negative_test.go:230: �[2m· describe-volumes vol-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8n_DescribeImagesNotFound" classname="" time="0.040">
+			<system-out><![CDATA[�[2m· describe-images ami-0000000000000dead�[0m
+    negative_test.go:240: �[2m· describe-images ami-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8o_CreateImageDuplicateName" classname="" time="0.560">
+			<system-out><![CDATA[�[2m· create-key-pair test-key-1 (duplicate)�[0m
+�[2m· create-image name=e2e-custom-ami (duplicate)�[0m
+�[2m· import-key-pair bad-format-key (bad material)�[0m
+    negative_test.go:252: �[2m· create-image name=e2e-custom-ami (duplicate)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8p_DeleteKeyPairIdempotent" classname="" time="0.020">
+			<system-out><![CDATA[�[2m· delete-key-pair nonexistent-key-99999 (idempotent)�[0m
+    negative_test.go:276: �[2m· delete-key-pair nonexistent-key-99999 (idempotent)�[0m
+�[2mstatus�[0m=400 �[2mcode�[0m=InvalidAction]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8q_ModifyAttributeOnRunning" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· modify-instance-attribute i-0c615e5dd74050f53 (running)�[0m
+    negative_test.go:289: �[2m· modify-instance-attribute i-0c615e5dd74050f53 (running)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8r_RebootInstanceNotFound" classname="" time="0.040">
+			<system-out><![CDATA[    negative_test.go:302: �[2m· reboot-instances i-nonexistent�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM1_UserCRUD" classname="" time="0.070">
+			<system-out><![CDATA[
+�[1m�[36m━━ IAM Phase 1 — User CRUD ━━�[0m
+�[2m· list-users (root auth sanity)�[0m
+    iam_test.go:109: �[2m· list-users (root auth sanity)�[0m
+�[2m· create-user "alice"�[0m
+    iam_test.go:114: �[2m· create-user "alice"�[0m
+�[2muser�[0m=alice �[2marn�[0m=arn:aws:iam::000000000001:user/alice
+    iam_test.go:120: �[2muser�[0m=alice �[2marn�[0m=arn:aws:iam::000000000001:user/alice
+�[2m· create-user "bob" path="/engineering/"�[0m
+    iam_test.go:122: �[2m· create-user "bob" path="/engineering/"�[0m
+�[2m· create-user "alice" again (expect EntityAlreadyExists)�[0m
+    iam_test.go:131: �[2m· create-user "alice" again (expect EntityAlreadyExists)�[0m
+�[2m· get-user "alice"�[0m
+    iam_test.go:137: �[2m· get-user "alice"�[0m
+�[2m· get-user nonexistent (expect NoSuchEntity)�[0m
+    iam_test.go:142: �[2m· get-user nonexistent (expect NoSuchEntity)�[0m
+�[2m· list-users (>=3 root+alice+bob)�[0m
+    iam_test.go:148: �[2m· list-users (>=3 root+alice+bob)�[0m
+�[2m· list-users --path-prefix "/engineering/" (expect 1)�[0m
+    iam_test.go:154: �[2m· list-users --path-prefix "/engineering/" (expect 1)�[0m
+    iam_test.go:193: �[2m· create-user "iam-phase1-ephemeral" (delete idempotency probe)�[0m
+�[2m· delete-user "iam-phase1-ephemeral"�[0m
+    iam_test.go:197: �[2m· delete-user "iam-phase1-ephemeral"�[0m
+�[2m· delete-user "iam-phase1-ephemeral" again (expect NoSuchEntity)�[0m
+    iam_test.go:201: �[2m· delete-user "iam-phase1-ephemeral" again (expect NoSuchEntity)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM1_UserCRUD/UpdateUser" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[    iam_test.go:165: daemon UpdateUser handler not implemented — InvalidAction
+�[2m· create-user "iam-phase1-ephemeral" (delete idempotency probe)�[0m]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM2_AccessKeyLifecycle" classname="" time="0.040">
+			<system-out><![CDATA[
+�[1m�[36m━━ IAM Phase 2 — Access Key Lifecycle ━━�[0m
+�[2m· create-access-key user=alice (key 1)�[0m
+    iam_test.go:218: �[2m· create-access-key user=alice (key 1)�[0m
+�[2mkey1�[0m=AKIA4984AE57CF8D721A879A
+    iam_test.go:227: �[2mkey1�[0m=AKIA4984AE57CF8D721A879A
+�[2m· create-access-key user=alice (key 2)�[0m
+    iam_test.go:229: �[2m· create-access-key user=alice (key 2)�[0m
+�[2mkey2�[0m=AKIAA1CD38D4F0E573216E11
+    iam_test.go:236: �[2mkey2�[0m=AKIAA1CD38D4F0E573216E11
+�[2m· create-access-key user=alice key 3 (expect LimitExceeded)�[0m
+    iam_test.go:238: �[2m· create-access-key user=alice key 3 (expect LimitExceeded)�[0m
+�[2m· create-access-key user=ghost (expect NoSuchEntity)�[0m
+    iam_test.go:246: �[2m· create-access-key user=ghost (expect NoSuchEntity)�[0m
+�[2m· list-access-keys user=alice (expect 2)�[0m
+    iam_test.go:254: �[2m· list-access-keys user=alice (expect 2)�[0m
+�[2m· list-access-keys user=bob (expect 0)�[0m
+    iam_test.go:261: �[2m· list-access-keys user=bob (expect 0)�[0m
+�[2m· update-access-key AKIA4984AE57CF8D721A879A -> Inactive�[0m
+    iam_test.go:268: �[2m· update-access-key AKIA4984AE57CF8D721A879A -> Inactive�[0m
+�[2m· update-access-key AKIA4984AE57CF8D721A879A -> Active�[0m
+    iam_test.go:279: �[2m· update-access-key AKIA4984AE57CF8D721A879A -> Active�[0m
+�[2m· delete-access-key AKIAA1CD38D4F0E573216E11 (key 2)�[0m
+    iam_test.go:287: �[2m· delete-access-key AKIAA1CD38D4F0E573216E11 (key 2)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM3_UserAuthentication" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[
+�[1m�[36m━━ IAM Phase 3 — User Authentication ━━�[0m
+    iam_test.go:311: daemon IAM signature/principal lookup not implemented — mulga-siv-100]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD" classname="" time="0.040">
+			<system-out><![CDATA[
+�[1m�[36m━━ IAM Phase 4 — Policy CRUD ━━�[0m
+�[2m· create-policy EC2ReadOnly�[0m
+    iam_test.go:384: �[2m· create-policy EC2ReadOnly�[0m
+�[2mpolicy�[0m=EC2ReadOnly �[2marn�[0m=arn:aws:iam::000000000001:policy/EC2ReadOnly �[2maccount�[0m=000000000001
+    iam_test.go:393: �[2mpolicy�[0m=EC2ReadOnly �[2marn�[0m=arn:aws:iam::000000000001:policy/EC2ReadOnly �[2maccount�[0m=000000000001
+    iam_test.go:440: �[2m· create-policy duplicate (expect EntityAlreadyExists)�[0m
+�[2m· create-policy malformed (expect MalformedPolicyDocument)�[0m
+    iam_test.go:449: �[2m· create-policy malformed (expect MalformedPolicyDocument)�[0m
+�[2m· get-policy EC2ReadOnly�[0m
+    iam_test.go:458: �[2m· get-policy EC2ReadOnly�[0m
+�[2m· get-policy nonexistent (expect NoSuchEntity)�[0m
+    iam_test.go:463: �[2m· get-policy nonexistent (expect NoSuchEntity)�[0m
+�[2m· get-policy-version EC2ReadOnly v1�[0m
+    iam_test.go:470: �[2m· get-policy-version EC2ReadOnly v1�[0m
+    iam_test.go:519: �[2m· list-policies (>=5)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel" classname="" time="0.000"></testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/FullAdmin" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· create-policy FullAdmin path=/admin/�[0m
+    iam_test.go:402: �[2m· create-policy FullAdmin path=/admin/�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/DenyTerminate" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· create-policy DenyTerminate (mixed Allow+Deny)�[0m
+    iam_test.go:413: �[2m· create-policy DenyTerminate (mixed Allow+Deny)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/IAMReadOnly" classname="" time="0.020">
+			<system-out><![CDATA[�[2m· create-policy IAMReadOnly�[0m
+    iam_test.go:422: �[2m· create-policy IAMReadOnly�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/EC2DescribeAll" classname="" time="0.020">
+			<system-out><![CDATA[�[2m· create-policy EC2DescribeAll (wildcard)�[0m
+    iam_test.go:431: �[2m· create-policy EC2DescribeAll (wildcard)�[0m
+�[2m· create-policy duplicate (expect EntityAlreadyExists)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/PolicyVersionLifecycle" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[    iam_test.go:484: daemon CreatePolicyVersion handler not implemented — InvalidAction
+�[2m· list-policies (>=5)�[0m]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM5_PolicyAttachmentEnforcement" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[
+�[1m�[36m━━ IAM Phase 5 — Policy Attachment & Enforcement ━━�[0m
+    iam_test.go:535: daemon IAM scoped-credential enforcement not implemented — mulga-siv-100]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM6_PolicyLifecycle" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[
+�[1m�[36m━━ IAM Phase 6 — Policy Lifecycle (Detach & Delete) ━━�[0m
+    iam_test.go:718: daemon DetachUserPolicy not implemented — mulga-siv-100]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM7_Cleanup" classname="" time="0.050">
+			<system-out><![CDATA[
+�[1m�[36m━━ IAM Phase 7 — Cleanup ━━�[0m
+�[2m· cleanup user alice�[0m
+    iam_test.go:770: �[2m· cleanup user alice�[0m
+�[2m· cleanup user bob�[0m
+    iam_test.go:770: �[2m· cleanup user bob�[0m
+�[2m· cleanup user charlie�[0m
+    iam_test.go:770: �[2m· cleanup user charlie�[0m
+�[2mremaining_users�[0m=1 �[2mnames�[0m=[admin]
+    iam_test.go:779: �[2mremaining_users�[0m=1 �[2mnames�[0m=[admin]
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2ReadOnly�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2ReadOnly�[0m
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/admin/FullAdmin�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/admin/FullAdmin�[0m
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/IAMReadOnly�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/IAMReadOnly�[0m
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2DescribeAll�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2DescribeAll�[0m
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/DenyTerminate�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/DenyTerminate�[0m
+�[2mremaining_policies�[0m=1 �[2mnames�[0m=[AdministratorAccess]
+    iam_test.go:812: �[2mremaining_policies�[0m=1 �[2mnames�[0m=[AdministratorAccess]]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping" classname="" time="114.890">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8 (acct) — EC2 Account Scoping ━━�[0m
+    account_test.go:75: �[2m· Phase 8 acct cleanup (deferred)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step1_AccountSetup" classname="" time="1.220">
+			<system-out><![CDATA[�[2m· create account 'Team Alpha'�[0m
+    account_test.go:206: �[2m· create account 'Team Alpha'�[0m
+�[2malpha_account�[0m=000000000002 �[2malpha_key�[0m=AKIA374CBBB2A15D3E2EE822
+    account_test.go:211: �[2malpha_account�[0m=000000000002 �[2malpha_key�[0m=AKIA374CBBB2A15D3E2EE822
+�[2m· create account 'Team Beta'�[0m
+    account_test.go:213: �[2m· create account 'Team Beta'�[0m
+�[2mbeta_account�[0m=000000000003 �[2mbeta_key�[0m=AKIA0AA0B2845E3BD5A26D01
+    account_test.go:218: �[2mbeta_account�[0m=000000000003 �[2mbeta_key�[0m=AKIA0AA0B2845E3BD5A26D01]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step2_InstanceScoping" classname="" time="52.380">
+			<system-out><![CDATA[�[2m· create per-account key pairs�[0m
+    account_test.go:242: �[2m· create per-account key pairs�[0m
+�[2m· alpha run-instances�[0m
+    account_test.go:250: �[2m· alpha run-instances�[0m
+�[2malpha_instance�[0m=i-a3072c2387630b8ee
+    account_test.go:262: �[2malpha_instance�[0m=i-a3072c2387630b8ee
+�[2m· beta run-instances�[0m
+    account_test.go:264: �[2m· beta run-instances�[0m
+�[2mbeta_instance�[0m=i-ad32cbb0afaee5eee
+    account_test.go:276: �[2mbeta_instance�[0m=i-ad32cbb0afaee5eee
+    account_test.go:278: instance i-a3072c2387630b8ee reached state running
+    account_test.go:279: instance i-ad32cbb0afaee5eee reached state running
+�[2m· alpha sees only own instances�[0m
+    account_test.go:281: �[2m· alpha sees only own instances�[0m
+�[2m· beta sees only own instances�[0m
+    account_test.go:285: �[2m· beta sees only own instances�[0m
+�[2m· alpha OwnerId matches alpha account�[0m
+    account_test.go:289: �[2m· alpha OwnerId matches alpha account�[0m
+�[2m· cross-account stop alpha->beta blocked�[0m
+    account_test.go:296: �[2m· cross-account stop alpha->beta blocked�[0m
+�[2m· cross-account terminate beta->alpha blocked�[0m
+    account_test.go:304: �[2m· cross-account terminate beta->alpha blocked�[0m
+�[2m· cross-account reboot alpha->beta blocked�[0m
+    account_test.go:312: �[2m· cross-account reboot alpha->beta blocked�[0m
+�[2m· stop alpha instance for cross-account start/modify/console tests�[0m
+    account_test.go:321: �[2m· stop alpha instance for cross-account start/modify/console tests�[0m
+    account_test.go:326: instance i-a3072c2387630b8ee reached state stopped
+�[2m· cross-account start beta->alpha blocked�[0m
+    account_test.go:328: �[2m· cross-account start beta->alpha blocked�[0m
+�[2m· cross-account modify-instance-attribute beta->alpha blocked�[0m
+    account_test.go:336: �[2m· cross-account modify-instance-attribute beta->alpha blocked�[0m
+�[2m· cross-account get-console-output beta->alpha blocked�[0m
+    account_test.go:345: �[2m· cross-account get-console-output beta->alpha blocked�[0m
+�[2m· restart alpha instance�[0m
+    account_test.go:354: �[2m· restart alpha instance�[0m
+    account_test.go:359: instance i-a3072c2387630b8ee reached state running]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step3_VolumeScoping" classname="" time="24.140">
+			<system-out><![CDATA[�[2m· alpha create-volume�[0m
+    account_test.go:366: �[2m· alpha create-volume�[0m
+�[2malpha_vol�[0m=vol-ceef82652369780af
+    account_test.go:375: �[2malpha_vol�[0m=vol-ceef82652369780af
+�[2m· beta create-volume�[0m
+    account_test.go:377: �[2m· beta create-volume�[0m
+�[2mbeta_vol�[0m=vol-08024f32d23d9e18c
+    account_test.go:386: �[2mbeta_vol�[0m=vol-08024f32d23d9e18c
+    account_test.go:388: volume vol-ceef82652369780af reached state available
+    account_test.go:389: volume vol-08024f32d23d9e18c reached state available
+�[2m· alpha sees only own volumes�[0m
+    account_test.go:391: �[2m· alpha sees only own volumes�[0m
+�[2m· cross-account describe-volume by id blocked�[0m
+    account_test.go:395: �[2m· cross-account describe-volume by id blocked�[0m
+�[2m· cross-account delete-volume blocked�[0m
+    account_test.go:403: �[2m· cross-account delete-volume blocked�[0m
+�[2m· cross-account attach-volume (beta attaches alpha's vol) blocked�[0m
+    account_test.go:411: �[2m· cross-account attach-volume (beta attaches alpha's vol) blocked�[0m
+�[2m· alpha attach-volume to own instance�[0m
+    account_test.go:422: �[2m· alpha attach-volume to own instance�[0m
+    account_test.go:429: volume vol-ceef82652369780af reached state in-use
+�[2m· cross-account detach-volume blocked�[0m
+    account_test.go:431: �[2m· cross-account detach-volume blocked�[0m
+�[2m· cross-account modify-volume blocked�[0m
+    account_test.go:439: �[2m· cross-account modify-volume blocked�[0m
+�[2m· alpha detach own volume�[0m
+    account_test.go:449: �[2m· alpha detach own volume�[0m
+    account_test.go:454: volume vol-ceef82652369780af reached state available]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step4_KeyPairScoping" classname="" time="0.370">
+			<system-out><![CDATA[�[2m· alpha create-key-pair alpha-key�[0m
+    account_test.go:466: �[2m· alpha create-key-pair alpha-key�[0m
+�[2m· beta create-key-pair beta-key�[0m
+    account_test.go:473: �[2m· beta create-key-pair beta-key�[0m
+�[2m· alpha sees only own keys�[0m
+    account_test.go:478: �[2m· alpha sees only own keys�[0m
+�[2m· namespace isolation: shared-name in both accounts�[0m
+    account_test.go:483: �[2m· namespace isolation: shared-name in both accounts�[0m
+�[2m· beta deletes alpha-key (idempotent, no cross-account effect)�[0m
+    account_test.go:497: �[2m· beta deletes alpha-key (idempotent, no cross-account effect)�[0m
+�[2m· alpha import-key-pair imported-key�[0m
+    account_test.go:503: �[2m· alpha import-key-pair imported-key�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step5_SnapshotScoping" classname="" time="0.060">
+			<system-out><![CDATA[�[2m· alpha create-snapshot�[0m
+    account_test.go:522: �[2m· alpha create-snapshot�[0m
+�[2malpha_snap�[0m=snap-2a47896f4304d805c
+    account_test.go:530: �[2malpha_snap�[0m=snap-2a47896f4304d805c
+�[2m· beta create-snapshot�[0m
+    account_test.go:532: �[2m· beta create-snapshot�[0m
+�[2mbeta_snap�[0m=snap-bf5744e78bd7c6908
+    account_test.go:540: �[2mbeta_snap�[0m=snap-bf5744e78bd7c6908
+�[2m· alpha sees only own snapshots (owner-ids=self)�[0m
+    account_test.go:542: �[2m· alpha sees only own snapshots (owner-ids=self)�[0m
+�[2m· cross-account delete-snapshot blocked (UnauthorizedOperation)�[0m
+    account_test.go:560: �[2m· cross-account delete-snapshot blocked (UnauthorizedOperation)�[0m
+�[2m· cross-account create-snapshot of alpha's volume blocked�[0m
+    account_test.go:568: �[2m· cross-account create-snapshot of alpha's volume blocked�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step6_VPCSubnetScoping" classname="" time="0.030">
+			<system-out><![CDATA[�[2m· alpha create-vpc 10.0.0.0/16�[0m
+    account_test.go:582: �[2m· alpha create-vpc 10.0.0.0/16�[0m
+�[2malpha_vpc�[0m=vpc-7f60df5e87183fb59
+    account_test.go:589: �[2malpha_vpc�[0m=vpc-7f60df5e87183fb59
+�[2m· beta create-vpc 10.0.0.0/16 (same CIDR, no conflict)�[0m
+    account_test.go:591: �[2m· beta create-vpc 10.0.0.0/16 (same CIDR, no conflict)�[0m
+�[2mbeta_vpc�[0m=vpc-7e8c859d3985c8218
+    account_test.go:598: �[2mbeta_vpc�[0m=vpc-7e8c859d3985c8218
+�[2m· alpha describe-vpcs isolation�[0m
+    account_test.go:600: �[2m· alpha describe-vpcs isolation�[0m
+�[2m· cross-account describe-vpc by id blocked�[0m
+    account_test.go:605: �[2m· cross-account describe-vpc by id blocked�[0m
+�[2m· cross-account delete-vpc blocked�[0m
+    account_test.go:613: �[2m· cross-account delete-vpc blocked�[0m
+�[2m· alpha create-subnet 10.0.1.0/24�[0m
+    account_test.go:621: �[2m· alpha create-subnet 10.0.1.0/24�[0m
+�[2m· beta create-subnet 10.0.1.0/24�[0m
+    account_test.go:630: �[2m· beta create-subnet 10.0.1.0/24�[0m
+�[2m· alpha describe-subnets isolation�[0m
+    account_test.go:639: �[2m· alpha describe-subnets isolation�[0m
+�[2m· cross-account create-subnet in other VPC blocked�[0m
+    account_test.go:648: �[2m· cross-account create-subnet in other VPC blocked�[0m
+�[2m· cross-account delete-subnet blocked�[0m
+    account_test.go:657: �[2m· cross-account delete-subnet blocked�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step7_IGWEIGWScoping" classname="" time="0.030">
+			<system-out><![CDATA[�[2m· alpha create-internet-gateway�[0m
+    account_test.go:673: �[2m· alpha create-internet-gateway�[0m
+�[2malpha_igw�[0m=igw-ee966e3e923caf630
+    account_test.go:678: �[2malpha_igw�[0m=igw-ee966e3e923caf630
+�[2m· beta create-internet-gateway�[0m
+    account_test.go:680: �[2m· beta create-internet-gateway�[0m
+�[2mbeta_igw�[0m=igw-b46e66f2065889bb9
+    account_test.go:685: �[2mbeta_igw�[0m=igw-b46e66f2065889bb9
+�[2m· alpha describe-internet-gateways isolation�[0m
+    account_test.go:687: �[2m· alpha describe-internet-gateways isolation�[0m
+�[2m· cross-account describe IGW by id blocked�[0m
+    account_test.go:696: �[2m· cross-account describe IGW by id blocked�[0m
+�[2m· cross-account delete IGW blocked�[0m
+    account_test.go:704: �[2m· cross-account delete IGW blocked�[0m
+�[2m· cross-account attach IGW (alpha attaches beta's IGW) blocked�[0m
+    account_test.go:712: �[2m· cross-account attach IGW (alpha attaches beta's IGW) blocked�[0m
+�[2m· alpha attach own IGW to own VPC�[0m
+    account_test.go:722: �[2m· alpha attach own IGW to own VPC�[0m
+�[2m· cross-account detach IGW blocked�[0m
+    account_test.go:729: �[2m· cross-account detach IGW blocked�[0m
+�[2m· alpha create-egress-only-internet-gateway�[0m
+    account_test.go:739: �[2m· alpha create-egress-only-internet-gateway�[0m
+�[2malpha_eigw�[0m=eigw-51ba82da927e49393
+    account_test.go:746: �[2malpha_eigw�[0m=eigw-51ba82da927e49393
+�[2m· beta create-egress-only-internet-gateway�[0m
+    account_test.go:748: �[2m· beta create-egress-only-internet-gateway�[0m
+�[2mbeta_eigw�[0m=eigw-e88d642e78ab4fe5a
+    account_test.go:755: �[2mbeta_eigw�[0m=eigw-e88d642e78ab4fe5a
+�[2m· alpha describe-eigws isolation�[0m
+    account_test.go:757: �[2m· alpha describe-eigws isolation�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step8_AccountSettings" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· alpha enable-ebs-encryption-by-default�[0m
+    account_test.go:794: �[2m· alpha enable-ebs-encryption-by-default�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step9_GlobalResources" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· describe-regions identical across accounts�[0m
+    account_test.go:829: �[2m· describe-regions identical across accounts�[0m
+�[2m· describe-availability-zones identical across accounts�[0m
+    account_test.go:834: �[2m· describe-availability-zones identical across accounts�[0m
+�[2m· describe-instance-types identical across accounts�[0m
+    account_test.go:839: �[2m· describe-instance-types identical across accounts�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step10_EdgeCases" classname="" time="1.200">
+			<system-out><![CDATA[�[2m· create empty Gamma account�[0m
+    account_test.go:849: �[2m· create empty Gamma account�[0m
+�[2mgamma_account�[0m=000000000004
+    account_test.go:852: �[2mgamma_account�[0m=000000000004
+�[2m· root create-key-pair root-scoping-key�[0m
+    account_test.go:879: �[2m· root create-key-pair root-scoping-key�[0m
+�[2m· non-existent volume id -> InvalidVolume.NotFound�[0m
+    account_test.go:895: �[2m· non-existent volume id -> InvalidVolume.NotFound�[0m
+�[2m· non-existent snapshot id -> InvalidSnapshot.NotFound�[0m
+    account_test.go:903: �[2m· non-existent snapshot id -> InvalidSnapshot.NotFound�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step11_Cleanup" classname="" time="32.440">
+			<system-out><![CDATA[�[2m· terminate alpha instance�[0m
+    account_test.go:920: �[2m· terminate alpha instance�[0m
+�[2m· terminate beta instance�[0m
+    account_test.go:926: �[2m· terminate beta instance�[0m
+�[2m· delete snapshots�[0m
+    account_test.go:939: �[2m· delete snapshots�[0m
+�[2m· delete volumes�[0m
+    account_test.go:955: �[2m· delete volumes�[0m
+�[2m· delete key pairs�[0m
+    account_test.go:969: �[2m· delete key pairs�[0m
+�[2m· delete EIGWs�[0m
+    account_test.go:979: �[2m· delete EIGWs�[0m
+�[2m· detach + delete IGWs�[0m
+    account_test.go:993: �[2m· detach + delete IGWs�[0m
+�[2m· delete subnets�[0m
+    account_test.go:1013: �[2m· delete subnets�[0m
+�[2m· delete VPCs�[0m
+    account_test.go:1027: �[2m· delete VPCs�[0m
+�[2m· Phase 8 acct cleanup (deferred)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8b_VPCSubnetE2E" classname="" time="200.480">
+			<failure message="Failed"><![CDATA[
+�[1m�[36m━━ Phase 8b — VPC Public/Private Subnet E2E ━━�[0m
+�[2m· create-vpc 10.99.0.0/16�[0m
+    vpc_test.go:56: �[2m· create-vpc 10.99.0.0/16�[0m
+�[2mvpc�[0m=vpc-39eff985799c43069 �[2mcidr�[0m=10.99.0.0/16
+    vpc_test.go:67: �[2mvpc�[0m=vpc-39eff985799c43069 �[2mcidr�[0m=10.99.0.0/16
+�[2m· create-subnet 10.99.1.0/24 (public)�[0m
+    vpc_test.go:70: �[2m· create-subnet 10.99.1.0/24 (public)�[0m
+�[2m· modify-subnet-attribute MapPublicIpOnLaunch=true subnet-22aba054e10588254�[0m
+    vpc_test.go:87: �[2m· modify-subnet-attribute MapPublicIpOnLaunch=true subnet-22aba054e10588254�[0m
+�[2mpub_subnet�[0m=subnet-22aba054e10588254 �[2mmap_public_ip�[0m=true
+    vpc_test.go:93: �[2mpub_subnet�[0m=subnet-22aba054e10588254 �[2mmap_public_ip�[0m=true
+�[2m· create-subnet 10.99.2.0/24 (private)�[0m
+    vpc_test.go:96: �[2m· create-subnet 10.99.2.0/24 (private)�[0m
+�[2mpriv_subnet�[0m=subnet-f53aeddb73c134696 �[2mmap_public_ip�[0m=false
+    vpc_test.go:111: �[2mpriv_subnet�[0m=subnet-f53aeddb73c134696 �[2mmap_public_ip�[0m=false
+�[2m· create-internet-gateway�[0m
+    vpc_test.go:114: �[2m· create-internet-gateway�[0m
+�[2m· attach-internet-gateway igw-1520b5a34480da982 -> vpc-39eff985799c43069�[0m
+    vpc_test.go:133: �[2m· attach-internet-gateway igw-1520b5a34480da982 -> vpc-39eff985799c43069�[0m
+�[2migw�[0m=igw-1520b5a34480da982
+    vpc_test.go:139: �[2migw�[0m=igw-1520b5a34480da982
+�[2m· create-route-table (vpc=vpc-39eff985799c43069)�[0m
+    vpc_test.go:142: �[2m· create-route-table (vpc=vpc-39eff985799c43069)�[0m
+�[2m· associate-route-table rtb-662df5e424cbdb60c <- subnet-22aba054e10588254�[0m
+    vpc_test.go:159: �[2m· associate-route-table rtb-662df5e424cbdb60c <- subnet-22aba054e10588254�[0m
+�[2m· create-route 0.0.0.0/0 -> igw-1520b5a34480da982�[0m
+    vpc_test.go:176: �[2m· create-route 0.0.0.0/0 -> igw-1520b5a34480da982�[0m
+�[2mrtb�[0m=rtb-662df5e424cbdb60c �[2massoc�[0m=rtbassoc-5510bf1dc6924eb0b �[2mroute�[0m=0.0.0.0/0->igw-1520b5a34480da982
+    vpc_test.go:192: �[2mrtb�[0m=rtb-662df5e424cbdb60c �[2massoc�[0m=rtbassoc-5510bf1dc6924eb0b �[2mroute�[0m=0.0.0.0/0->igw-1520b5a34480da982
+�[2m· run-instances ami=ami-6f60f2b3bf04b5453 subnet=subnet-22aba054e10588254�[0m
+    vpc_test.go:195: �[2m· run-instances ami=ami-6f60f2b3bf04b5453 subnet=subnet-22aba054e10588254�[0m
+�[2minstance�[0m=i-3177888cba3796d42
+    vpc_test.go:216: �[2minstance�[0m=i-3177888cba3796d42
+    vpc_test.go:218: instance i-3177888cba3796d42 reached state running
+�[2mpublic_ip�[0m=192.168.0.202 �[2mprivate_ip�[0m=10.99.1.4
+    vpc_test.go:224: �[2mpublic_ip�[0m=192.168.0.202 �[2mprivate_ip�[0m=10.99.1.4
+�[2m· waiting for SSH handshake 192.168.0.202:22�[0m
+    vpc_test.go:227: �[2m· waiting for SSH handshake 192.168.0.202:22�[0m
+    vpc_test.go:227: Eventually: condition not met within 3m0s: [SSH handshake 192.168.0.202:22 never completed]]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8c_RouteTableValidation" classname="" time="0.030">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8c — Route Table Validation ━━�[0m
+�[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63
+    vpc_test.go:292: �[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8c_RouteTableValidation/Step1_DefaultMainRouteTable" classname="" time="0.000">
+			<system-out><![CDATA[�[2m· describe-route-tables filter=vpc-id,association.main�[0m
+    vpc_test.go:297: �[2m· describe-route-tables filter=vpc-id,association.main�[0m
+�[2mmain_rtb�[0m=rtb-e42012569f92f7184
+    vpc_test.go:309: �[2mmain_rtb�[0m=rtb-e42012569f92f7184
+�[2mlocal_route�[0m=172.31.0.0/16
+    vpc_test.go:332: �[2mlocal_route�[0m=172.31.0.0/16
+�[2mdefault_route�[0m=0.0.0.0/0->igw-79c2e643660b4b7e3
+    vpc_test.go:350: �[2mdefault_route�[0m=0.0.0.0/0->igw-79c2e643660b4b7e3]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8c_RouteTableValidation/Step2_CustomRouteTableLifecycle" classname="" time="0.020">
+			<system-out><![CDATA[�[2m· create-route-table (vpc=vpc-531252db3b4e1fc08)�[0m
+    vpc_test.go:363: �[2m· create-route-table (vpc=vpc-531252db3b4e1fc08)�[0m
+�[2mcustom_rtb�[0m=rtb-5e6c46936f3e44167
+    vpc_test.go:379: �[2mcustom_rtb�[0m=rtb-5e6c46936f3e44167
+�[2mcustom_local_route�[0m=172.31.0.0/16
+    vpc_test.go:396: �[2mcustom_local_route�[0m=172.31.0.0/16
+�[2mvpc_igw�[0m=igw-79c2e643660b4b7e3
+    vpc_test.go:410: �[2mvpc_igw�[0m=igw-79c2e643660b4b7e3
+�[2m· create-route 0.0.0.0/0 -> igw-79c2e643660b4b7e3�[0m
+    vpc_test.go:415: �[2m· create-route 0.0.0.0/0 -> igw-79c2e643660b4b7e3�[0m
+�[2m· associate-route-table rtb-5e6c46936f3e44167 <- subnet-cd62e4c1c3cb50c63�[0m
+    vpc_test.go:434: �[2m· associate-route-table rtb-5e6c46936f3e44167 <- subnet-cd62e4c1c3cb50c63�[0m
+�[2mcustom_assoc�[0m=rtbassoc-da6377d37847d3d4a
+    vpc_test.go:450: �[2mcustom_assoc�[0m=rtbassoc-da6377d37847d3d4a
+�[2m· disassociate-route-table rtbassoc-da6377d37847d3d4a�[0m
+    vpc_test.go:452: �[2m· disassociate-route-table rtbassoc-da6377d37847d3d4a�[0m
+�[2m· delete-route 0.0.0.0/0�[0m
+    vpc_test.go:460: �[2m· delete-route 0.0.0.0/0�[0m
+�[2m· delete-route-table rtb-5e6c46936f3e44167�[0m
+    vpc_test.go:468: �[2m· delete-route-table rtb-5e6c46936f3e44167�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8c_RouteTableValidation/Step3_ErrorPaths" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· create-route-table (scratch for error paths)�[0m
+    vpc_test.go:481: �[2m· create-route-table (scratch for error paths)�[0m
+�[2m· delete-route local=172.31.0.0/16 (must fail)�[0m
+    vpc_test.go:517: �[2m· delete-route local=172.31.0.0/16 (must fail)�[0m
+�[2mdelete_local_err�[0m=InvalidParameterValue
+    vpc_test.go:529: �[2mdelete_local_err�[0m=InvalidParameterValue
+�[2m· delete-route-table rtb-e42012569f92f7184 (main, must fail)�[0m
+    vpc_test.go:533: �[2m· delete-route-table rtb-e42012569f92f7184 (main, must fail)�[0m
+�[2mdelete_main_err�[0m=DependencyViolation
+    vpc_test.go:540: �[2mdelete_main_err�[0m=DependencyViolation]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8d_NATGateway" classname="" time="106.010">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8d — NAT Gateway E2E ━━�[0m
+�[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2mpub_subnet�[0m=subnet-cd62e4c1c3cb50c63 �[2msg�[0m=sg-764a7fa2a39577204
+    natgw_test.go:54: �[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2mpub_subnet�[0m=subnet-cd62e4c1c3cb50c63 �[2msg�[0m=sg-764a7fa2a39577204
+�[2m· create-subnet 172.31.16.0/20 (private)�[0m
+    natgw_test.go:58: �[2m· create-subnet 172.31.16.0/20 (private)�[0m
+�[2mpriv_subnet�[0m=subnet-2360a40e28371e214
+    natgw_test.go:71: �[2mpriv_subnet�[0m=subnet-2360a40e28371e214
+�[2m· run-instances bastion (default subnet)�[0m
+    natgw_test.go:74: �[2m· run-instances bastion (default subnet)�[0m
+�[2m· run-instances private (private subnet)�[0m
+    natgw_test.go:98: �[2m· run-instances private (private subnet)�[0m
+    natgw_test.go:118: instance i-268402f0e28b5e477 reached state running
+    natgw_test.go:119: instance i-db7f3ed6e4cdaa5aa reached state running
+�[2mbastion�[0m=i-268402f0e28b5e477 �[2mbastion_ip�[0m=192.168.0.202 �[2mprivate�[0m=i-db7f3ed6e4cdaa5aa �[2mprivate_ip�[0m=172.31.16.4
+    natgw_test.go:128: �[2mbastion�[0m=i-268402f0e28b5e477 �[2mbastion_ip�[0m=192.168.0.202 �[2mprivate�[0m=i-db7f3ed6e4cdaa5aa �[2mprivate_ip�[0m=172.31.16.4
+�[2m· waiting for SSH handshake 192.168.0.202:22�[0m
+    natgw_test.go:133: �[2m· waiting for SSH handshake 192.168.0.202:22�[0m
+�[2m· scp keypair -> bastion:/tmp/key.pem�[0m
+    natgw_test.go:139: �[2m· scp keypair -> bastion:/tmp/key.pem�[0m
+�[2m· wait for private SSH via bastion�[0m
+    natgw_test.go:151: �[2m· wait for private SSH via bastion�[0m
+�[2m· baseline: private VM has no internet�[0m
+    natgw_test.go:172: �[2m· baseline: private VM has no internet�[0m
+�[2m· allocate-address (NAT EIP)�[0m
+    natgw_test.go:178: �[2m· allocate-address (NAT EIP)�[0m
+�[2meip�[0m=192.168.0.203 �[2malloc�[0m=eipalloc-2d81753157cd62a50
+    natgw_test.go:195: �[2meip�[0m=192.168.0.203 �[2malloc�[0m=eipalloc-2d81753157cd62a50
+�[2m· create-nat-gateway in subnet-cd62e4c1c3cb50c63�[0m
+    natgw_test.go:197: �[2m· create-nat-gateway in subnet-cd62e4c1c3cb50c63�[0m
+�[2mnat_gw�[0m=nat-812216dd5105180c5
+    natgw_test.go:216: �[2mnat_gw�[0m=nat-812216dd5105180c5
+    natgw_test.go:218: nat gateway nat-812216dd5105180c5 reached state available
+�[2m· create-route-table�[0m
+    natgw_test.go:223: �[2m· create-route-table�[0m
+�[2m· associate-route-table rtb-8d6a4abd5f666ec13 <- subnet-2360a40e28371e214�[0m
+    natgw_test.go:241: �[2m· associate-route-table rtb-8d6a4abd5f666ec13 <- subnet-2360a40e28371e214�[0m
+�[2m· create-route 0.0.0.0/0 -> nat-812216dd5105180c5�[0m
+    natgw_test.go:259: �[2m· create-route 0.0.0.0/0 -> nat-812216dd5105180c5�[0m
+�[2m· verify private VM reaches 8.8.8.8 via NAT GW�[0m
+    natgw_test.go:279: �[2m· verify private VM reaches 8.8.8.8 via NAT GW�[0m
+�[2m· delete-nat-gateway nat-812216dd5105180c5�[0m
+    natgw_test.go:293: �[2m· delete-nat-gateway nat-812216dd5105180c5�[0m
+    natgw_test.go:318: nat gateway nat-812216dd5105180c5 reached state deleted
+�[2m· verify private VM lost internet after NAT GW teardown�[0m
+    natgw_test.go:329: �[2m· verify private VM lost internet after NAT GW teardown�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8e_SGToSGDatapath" classname="" time="80.940">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8e — Security Group SG-to-SG Datapath (OVN) ━━�[0m
+�[2m· 8e-1 create sge-client + sge-target security groups�[0m
+    datapath_test.go:61: �[2m· 8e-1 create sge-client + sge-target security groups�[0m
+�[2mclient_sg�[0m=sg-56e73c764a649f7ff �[2mtarget_sg�[0m=sg-67ad5efaa8be83b79
+    datapath_test.go:83: �[2mclient_sg�[0m=sg-56e73c764a649f7ff �[2mtarget_sg�[0m=sg-67ad5efaa8be83b79
+�[2m· 8e-1 authorize target-sg ingress tcp/8080 from sg-56e73c764a649f7ff�[0m
+    datapath_test.go:91: �[2m· 8e-1 authorize target-sg ingress tcp/8080 from sg-56e73c764a649f7ff�[0m
+�[2m· 8e-2 launch client-vm + target-vm�[0m
+    datapath_test.go:109: �[2m· 8e-2 launch client-vm + target-vm�[0m
+    datapath_test.go:132: instance i-eae6263d9b9fda069 reached state running
+    datapath_test.go:133: instance i-9d9411a83549d34e3 reached state running
+�[2mclient_priv�[0m=172.31.0.5 �[2mclient_eni�[0m=eni-0f878da0143028915 �[2mtarget_priv�[0m=172.31.0.6 �[2mtarget_eni�[0m=eni-3bd355e19c7fdd45c
+    datapath_test.go:142: �[2mclient_priv�[0m=172.31.0.5 �[2mclient_eni�[0m=eni-0f878da0143028915 �[2mtarget_priv�[0m=172.31.0.6 �[2mtarget_eni�[0m=eni-3bd355e19c7fdd45c
+�[2m· 8e-3 ovn-nbctl port_group membership�[0m
+    datapath_test.go:151: �[2m· 8e-3 ovn-nbctl port_group membership�[0m
+�[2mclient_pg�[0m=sg_56e73c764a649f7ff �[2mtarget_pg�[0m=sg_67ad5efaa8be83b79
+    datapath_test.go:163: �[2mclient_pg�[0m=sg_56e73c764a649f7ff �[2mtarget_pg�[0m=sg_67ad5efaa8be83b79
+�[2m· 8e-3 ovn-sbctl address_set sg_56e73c764a649f7ff_ip4 contains 172.31.0.5�[0m
+    datapath_test.go:170: �[2m· 8e-3 ovn-sbctl address_set sg_56e73c764a649f7ff_ip4 contains 172.31.0.5�[0m
+�[2m· 8e-3 wait for client-vm SSH at 192.168.0.202:22�[0m
+    datapath_test.go:188: �[2m· 8e-3 wait for client-vm SSH at 192.168.0.202:22�[0m
+�[2m· 8e-4 allowed traffic client -> target:172.31.0.6:8080�[0m
+    datapath_test.go:199: �[2m· 8e-4 allowed traffic client -> target:172.31.0.6:8080�[0m
+�[2mstep4�[0m=allowed_traffic_ok
+    datapath_test.go:208: �[2mstep4�[0m=allowed_traffic_ok
+�[2m· 8e-5 denied traffic client -> target:22 (no SSH ingress)�[0m
+    datapath_test.go:213: �[2m· 8e-5 denied traffic client -> target:22 (no SSH ingress)�[0m
+�[2mstep5�[0m=denied_traffic_ok
+    datapath_test.go:217: �[2mstep5�[0m=denied_traffic_ok
+�[2m· 8e-6 revoke target-sg ingress, retest (sync RequestEvent contract)�[0m
+    datapath_test.go:222: �[2m· 8e-6 revoke target-sg ingress, retest (sync RequestEvent contract)�[0m
+�[2m· 8e-6 verify client -> target:8080 now blocked�[0m
+    datapath_test.go:240: �[2m· 8e-6 verify client -> target:8080 now blocked�[0m
+�[2mstep6�[0m=revoke_blocked_ok
+    datapath_test.go:244: �[2mstep6�[0m=revoke_blocked_ok
+�[2m· 8e-7 re-authorize target-sg ingress, verify traffic restored�[0m
+    datapath_test.go:250: �[2m· 8e-7 re-authorize target-sg ingress, verify traffic restored�[0m
+�[2mstep7�[0m=restore_ok
+    datapath_test.go:271: �[2mstep7�[0m=restore_ok
+    datapath_test.go:127: instance i-eae6263d9b9fda069 reached state terminated
+    datapath_test.go:127: instance i-9d9411a83549d34e3 reached state terminated]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase9_Teardown" classname="" time="8.300">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 9 — Teardown ━━�[0m
+�[2m· delete-snapshot snap-dde89b9ea2ef2e05a (custom AMI backing)�[0m
+    teardown_test.go:33: �[2m· delete-snapshot snap-dde89b9ea2ef2e05a (custom AMI backing)�[0m
+�[2mdeleted_snapshot�[0m=snap-dde89b9ea2ef2e05a
+    teardown_test.go:39: �[2mdeleted_snapshot�[0m=snap-dde89b9ea2ef2e05a
+�[2m· deregister-image ami-7dcf263a412dd14c6�[0m
+    teardown_test.go:51: �[2m· deregister-image ami-7dcf263a412dd14c6�[0m
+�[2mderegistered_ami�[0m=ami-7dcf263a412dd14c6
+    teardown_test.go:57: �[2mderegistered_ami�[0m=ami-7dcf263a412dd14c6
+�[2m· terminate-instances i-0c615e5dd74050f53�[0m
+    teardown_test.go:74: �[2m· terminate-instances i-0c615e5dd74050f53�[0m
+�[2mterminated_instance�[0m=i-0c615e5dd74050f53
+    teardown_test.go:81: �[2mterminated_instance�[0m=i-0c615e5dd74050f53
+�[2m· delete-key-pair test-key-1�[0m
+    teardown_test.go:94: �[2m· delete-key-pair test-key-1�[0m
+�[2mdeleted_keypair�[0m=test-key-1
+    teardown_test.go:100: �[2mdeleted_keypair�[0m=test-key-1]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase9a_VerifyTeardown" classname="" time="0.560">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 9a — Verify Teardown ━━�[0m
+�[2m· describe-instances i-0c615e5dd74050f53�[0m
+    teardown_test.go:123: �[2m· describe-instances i-0c615e5dd74050f53�[0m
+�[2minstance_state�[0m=terminated
+    teardown_test.go:135: �[2minstance_state�[0m=terminated
+�[2m· describe-images ami-7dcf263a412dd14c6�[0m
+    teardown_test.go:147: �[2m· describe-images ami-7dcf263a412dd14c6�[0m
+�[2mami_reaped�[0m=ami-7dcf263a412dd14c6
+    teardown_test.go:159: �[2mami_reaped�[0m=ami-7dcf263a412dd14c6
+�[2m· describe-key-pairs (must not contain test-key-1)�[0m
+    teardown_test.go:167: �[2m· describe-key-pairs (must not contain test-key-1)�[0m
+�[2mkeypairs_listed�[0m=1
+    teardown_test.go:176: �[2mkeypairs_listed�[0m=1]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase9b_FinalClusterStats" classname="" time="3.070">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 9b — Final Cluster Stats ━━�[0m
+�[2mspx_get_vms_bytes�[0m=118
+    teardown_test.go:208: �[2mspx_get_vms_bytes�[0m=118]]></system-out>
+		</testcase>
+	</testsuite>
+</testsuites>

.github/actions/junit-report/.gitignore

@@ -0,0 +1 @@
+__pycache__/

.github/actions/junit-report/README.md

@@ -0,0 +1,46 @@
+# junit-report
+
+Composite action that renders a glob of JUnit XML files into the GitHub
+Actions job summary and emits one `::error` annotation per failed test case.
+
+## Usage
+
+```yaml
+- name: Publish test summary
+  if: always()
+  uses: ./spinifex/.github/actions/junit-report   # or your own checkout path
+  with:
+    junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+    title: "Go E2E results"
+    # fail-on-empty: "true"   # opt-in: fail the step if the glob matches nothing
+```
+
+Outputs:
+
+- `failures` — total `<failure>` + `<error>` counts across all matched files.
+- `total`    — total `<testcase>` count across all matched files.
+
+The action expects standard JUnit XML (e.g. emitted by
+`gotestsum --junitfile`, `go-junit-report`, or pytest's `--junitxml`). Both
+single `<testsuite>` and wrapping `<testsuites>` roots are accepted.
+
+## Companion convention: `::group::` for noisy steps
+
+Long-running shell phases (tofu apply, package installs, tar-over-ssh) should
+wrap their output in `::group::name` / `::endgroup::` markers so the Actions
+UI collapses them by default. Pattern used in the e2e-go / e2e-ddil workflows:
+
+```bash
+run() {
+  local label="$1"; shift
+  echo "::group::${label}"
+  "$@" 2>&1 | tee -a "$TOFU_LOG"
+  local rc=${PIPESTATUS[0]}
+  echo "::endgroup::"
+  return "$rc"
+}
+run "tofu apply" tofu apply -var-file=... -auto-approve -input=false
+```
+
+Combined: groups make the live UI readable; this action makes the run header
+land on the failures without scrolling the log.

.github/actions/junit-report/action.yml

@@ -0,0 +1,41 @@
+name: JUnit Report
+description: >-
+  Render JUnit XML files into the GitHub Actions job summary and emit
+  inline ::error annotations for each failed test case. Designed to be
+  called once per job after the test step (with `if: always()`) so summary
+  + annotations appear on both passing and failing runs.
+
+inputs:
+  junit-glob:
+    description: Shell glob matching JUnit XML files to render.
+    required: true
+  title:
+    description: Heading rendered at the top of the job summary section.
+    required: false
+    default: "Test results"
+  fail-on-empty:
+    description: >-
+      When "true", the action fails if the glob matches no files. Useful
+      for catching silently-skipped test phases.
+    required: false
+    default: "false"
+
+outputs:
+  failures:
+    description: Total failed + errored test cases across all matched files.
+    value: ${{ steps.render.outputs.failures }}
+  total:
+    description: Total test cases across all matched files.
+    value: ${{ steps.render.outputs.total }}
+
+runs:
+  using: composite
+  steps:
+    - id: render
+      shell: bash
+      env:
+        JUNIT_GLOB: ${{ inputs.junit-glob }}
+        JUNIT_TITLE: ${{ inputs.title }}
+        JUNIT_FAIL_ON_EMPTY: ${{ inputs.fail-on-empty }}
+      run: |
+        python3 "${{ github.action_path }}/render.py"

.github/actions/junit-report/render.py

@@ -0,0 +1,281 @@
+#!/usr/bin/env python3
+"""Render JUnit XML files into GitHub Actions job summary + annotations.
+
+Inputs (env):
+    JUNIT_GLOB           Glob matching JUnit XML files.
+    JUNIT_TITLE          Heading text for the summary section.
+    JUNIT_FAIL_ON_EMPTY  "true" to exit non-zero when glob matches nothing.
+
+GitHub-Actions outputs (via $GITHUB_OUTPUT):
+    failures             Sum of <failures> + <errors> across all suites.
+    total                Sum of <tests> across all suites.
+
+Side effects:
+    Appends a markdown section to $GITHUB_STEP_SUMMARY with a per-file
+    totals table and a collapsible details block per failed test.
+    Emits one `::error` workflow command per failed/errored test case so
+    they surface as annotations in the run UI.
+"""
+
+from __future__ import annotations
+
+import glob
+import os
+import re
+import sys
+import xml.etree.ElementTree as ET
+from dataclasses import dataclass
+
+
+# Strip ANSI SGR escapes from skipped/failure CDATA so the rendered reason is
+# plain text (CDATA preserves the colour codes Go emits via harness.Phase).
+_ANSI_RE = re.compile(r"\x1b\[[0-9;]*m")
+
+
+def _strip_ansi(s: str) -> str:
+    return _ANSI_RE.sub("", s)
+
+
+# Last `<file>.go:<line>: <reason>` line in a CDATA block is the t.Skipf /
+# t.Fatal call site — that's the actionable signal. Phase banners and probe
+# logs above it are noise for the summary view.
+_SKIP_REASON_RE = re.compile(r"([A-Za-z0-9_.\-/]+\.go:\d+:\s*.+)$")
+
+
+def _extract_skip_reason(cdata: str) -> str:
+    cdata = _strip_ansi(cdata).strip()
+    if not cdata:
+        return ""
+    for line in reversed(cdata.splitlines()):
+        line = line.strip()
+        m = _SKIP_REASON_RE.search(line)
+        if m:
+            return m.group(1).strip()
+    return cdata.splitlines()[-1].strip()
+
+
+# Workflow commands carry their payload on one line; literal newlines must
+# be percent-escaped per GitHub's workflow-command spec.
+def _esc(s: str) -> str:
+    return (
+        s.replace("%", "%25")
+        .replace("\r", "%0D")
+        .replace("\n", "%0A")
+    )
+
+
+@dataclass
+class Case:
+    suite: str
+    classname: str
+    name: str
+    time: float
+    status: str  # "pass" | "fail" | "error" | "skip"
+    message: str
+    detail: str
+
+
+@dataclass
+class FileTotals:
+    path: str
+    tests: int
+    failures: int
+    errors: int
+    skipped: int
+    time: float
+    cases: list[Case]
+
+
+def _parse_file(path: str) -> FileTotals:
+    root = ET.parse(path).getroot()
+    # JUnit XML has two shapes: <testsuite> as root, or <testsuites> wrapping
+    # multiple <testsuite> children. Iterating with .iter() handles both.
+    suites = list(root.iter("testsuite"))
+    tests = sum(int(s.get("tests", 0) or 0) for s in suites)
+    failures = sum(int(s.get("failures", 0) or 0) for s in suites)
+    errors = sum(int(s.get("errors", 0) or 0) for s in suites)
+    skipped = sum(int(s.get("skipped", 0) or 0) for s in suites)
+    time = sum(float(s.get("time", 0) or 0) for s in suites)
+
+    cases: list[Case] = []
+    for s in suites:
+        suite_name = s.get("name", "")
+        for tc in s.findall("testcase"):
+            status = "pass"
+            message = ""
+            detail = ""
+            for tag in ("failure", "error"):
+                node = tc.find(tag)
+                if node is not None:
+                    status = "fail" if tag == "failure" else "error"
+                    message = (node.get("message") or "").strip()
+                    detail = (node.text or "").strip()
+                    break
+            skip_node = tc.find("skipped")
+            if skip_node is not None and status == "pass":
+                status = "skip"
+                # Skipped reason: prefer the CDATA call-site line, fall back
+                # to the @message attribute if CDATA is empty.
+                cdata = (skip_node.text or "").strip()
+                message = _extract_skip_reason(cdata) or (
+                    skip_node.get("message") or ""
+                ).strip()
+            cases.append(
+                Case(
+                    suite=suite_name,
+                    classname=tc.get("classname", ""),
+                    name=tc.get("name", ""),
+                    time=float(tc.get("time", 0) or 0),
+                    status=status,
+                    message=message,
+                    detail=detail,
+                )
+            )
+
+    # Drop container test cases: any case whose name is a strict prefix of
+    # another case's name (with "/" separator) is a parent t.Run wrapper.
+    # Go reports the parent's status as a rollup of its children, so counting
+    # both double-tallies the same outcome (e.g. TestSingleNode failing
+    # because Phase8b_VPCSubnetE2E failed would show as 2 failures, not 1).
+    names = {c.name for c in cases}
+    prefixes = {n.rsplit("/", 1)[0] for n in names if "/" in n}
+    # A name is a container if some other name starts with "<name>/".
+    # Use the prefixes set rather than O(n^2) scanning.
+    def _is_container(name: str) -> bool:
+        return name in prefixes
+
+    leaf_cases = [c for c in cases if not _is_container(c.name)]
+
+    # Recompute totals from leaf cases so the per-file counts in the summary
+    # table match the de-duplicated failure list below it.
+    leaf_tests = len(leaf_cases)
+    leaf_failures = sum(1 for c in leaf_cases if c.status == "fail")
+    leaf_errors = sum(1 for c in leaf_cases if c.status == "error")
+    leaf_skipped = sum(1 for c in leaf_cases if c.status == "skip")
+
+    return FileTotals(
+        path=path,
+        tests=leaf_tests,
+        failures=leaf_failures,
+        errors=leaf_errors,
+        skipped=leaf_skipped,
+        time=time,
+        cases=leaf_cases,
+    )
+
+
+def _file_label(path: str) -> str:
+    base = os.path.basename(path)
+    if base.endswith(".xml"):
+        base = base[:-4]
+    if base.startswith("junit-"):
+        base = base[len("junit-"):]
+    return base or path
+
+
+def main() -> int:
+    pattern = os.environ.get("JUNIT_GLOB", "").strip()
+    title = os.environ.get("JUNIT_TITLE", "Test results").strip() or "Test results"
+    fail_on_empty = os.environ.get("JUNIT_FAIL_ON_EMPTY", "").strip().lower() == "true"
+
+    if not pattern:
+        print("::error::junit-report: junit-glob input is empty", file=sys.stderr)
+        return 1
+
+    paths = sorted(glob.glob(pattern))
+    summary_path = os.environ.get("GITHUB_STEP_SUMMARY", "/dev/null")
+    output_path = os.environ.get("GITHUB_OUTPUT", "/dev/null")
+
+    with open(summary_path, "a", encoding="utf-8") as summary, \
+            open(output_path, "a", encoding="utf-8") as outputs:
+
+        summary.write(f"## {title}\n\n")
+
+        if not paths:
+            summary.write(f"_No JUnit files matched `{pattern}`._\n\n")
+            outputs.write("failures=0\n")
+            outputs.write("total=0\n")
+            if fail_on_empty:
+                print(f"::error::junit-report: no files matched {pattern}", file=sys.stderr)
+                return 1
+            return 0
+
+        totals_by_file = [_parse_file(p) for p in paths]
+        grand_total = sum(f.tests for f in totals_by_file)
+        grand_fail = sum(f.failures + f.errors for f in totals_by_file)
+        grand_skip = sum(f.skipped for f in totals_by_file)
+        grand_pass = grand_total - grand_fail - grand_skip
+        grand_time = sum(f.time for f in totals_by_file)
+
+        if grand_fail == 0:
+            verdict = f"✅ **pass** — {grand_pass} / {grand_total}"
+        else:
+            verdict = f"❌ **{grand_fail} failed** — {grand_pass} / {grand_total} pass"
+        skip_part = f", {grand_skip} skipped" if grand_skip else ""
+        summary.write(f"{verdict}{skip_part} &nbsp; · &nbsp; {grand_time:.1f}s\n\n")
+
+        # Compact table — single status column, shorter headers. Drops
+        # separate err column (errors fold into fail count) so narrow
+        # screens don't wrap.
+        summary.write("| | Suite | Pass | Fail | Skip | Time |\n")
+        summary.write("|:-:|---|---:|---:|---:|---:|\n")
+        for f in totals_by_file:
+            fails = f.failures + f.errors
+            passed = f.tests - fails - f.skipped
+            icon = "✅" if fails == 0 else "❌"
+            summary.write(
+                f"| {icon} | `{_file_label(f.path)}` | {passed} | {fails} | "
+                f"{f.skipped} | {f.time:.1f}s |\n"
+            )
+        summary.write("\n")
+
+        failed_cases = [
+            c for f in totals_by_file for c in f.cases if c.status in ("fail", "error")
+        ]
+        if failed_cases:
+            summary.write("---\n\n**Failures**\n\n")
+            for c in failed_cases:
+                header = f"{c.classname}.{c.name}" if c.classname else c.name
+                # One line per failure: name + truncated message. Full
+                # detail lives in the test artifact log; deep nesting and
+                # long messages wrap badly on narrow screens.
+                msg_short = (c.message or "").splitlines()[0] if c.message else ""
+                if len(msg_short) > 120:
+                    msg_short = msg_short[:117] + "…"
+                line = f"- ❌ `{header}`"
+                if msg_short:
+                    line += f" — {msg_short}"
+                summary.write(line + "\n")
+
+                # Workflow annotation — surfaces in run header + Files-changed view.
+                # classname for Go tests is the package path; map to a file when we can.
+                file_hint = c.classname.replace(".", "/")
+                msg = c.message or f"{c.status}: {header}"
+                print(f"::error file={file_hint},title={header}::{_esc(msg)}")
+            summary.write("\n")
+
+        skipped_cases = [
+            c for f in totals_by_file for c in f.cases if c.status == "skip"
+        ]
+        if skipped_cases:
+            summary.write("---\n\n<details><summary><b>Skipped</b> ")
+            summary.write(f"({len(skipped_cases)})</summary>\n\n")
+            for c in skipped_cases:
+                header = f"{c.classname}.{c.name}" if c.classname else c.name
+                reason = (c.message or "").splitlines()[0] if c.message else ""
+                if len(reason) > 160:
+                    reason = reason[:157] + "…"
+                line = f"- ⏭️ `{header}`"
+                if reason:
+                    line += f" — {reason}"
+                summary.write(line + "\n")
+            summary.write("\n</details>\n\n")
+
+        outputs.write(f"failures={grand_fail}\n")
+        outputs.write(f"total={grand_total}\n")
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

.github/dependabot.yml

@@ -8,7 +8,7 @@ updates:
       day: monday
       timezone: Australia/Sydney
     cooldown:
-      default-days: 1
+      default-days: 3
     groups:
       go-dependencies:
         patterns:
@@ -22,7 +22,7 @@ updates:
       day: monday
       timezone: Australia/Sydney
     cooldown:
-      default-days: 1
+      default-days: 3
     ignore:
       - dependency-name: "*"
         update-types:
@@ -41,7 +41,7 @@ updates:
       day: monday
       timezone: Australia/Sydney
     cooldown:
-      default-days: 1
+      default-days: 3
     ignore:
       - dependency-name: "@types/node"
         update-types:

.github/scripts/ci-fmt.sh

@@ -0,0 +1,63 @@
+# shellcheck shell=bash
+# Source from a workflow `run:` block:
+#   source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+#
+# Provides colored banners + ::group::/::endgroup:: wrappers. ANSI escapes
+# render in the GitHub Actions live log; the same lines also land in any
+# artifact log (tee'd output) without breaking parsers because they only
+# affect terminal display.
+#
+# Functions kept short and unindented on purpose — wide nested output wraps
+# badly on narrow terminals.
+
+if [[ -n "${GITHUB_ACTIONS:-}" ]] || [[ -t 1 ]]; then
+  _R=$'\033[0m'
+  _B=$'\033[1m'    # bold
+  _D=$'\033[2m'    # dim
+  _RED=$'\033[31m'
+  _GRN=$'\033[32m'
+  _YEL=$'\033[33m'
+  _BLU=$'\033[34m'
+  _CYA=$'\033[36m'
+else
+  _R="" _B="" _D="" _RED="" _GRN="" _YEL="" _BLU="" _CYA=""
+fi
+
+# Single-line rule using box-drawing chars — narrow enough that GitHub's
+# timestamp prefix + this rule still fit on an 80-col terminal.
+# banner LABEL — bold cyan title with surrounding rule. Used at the start
+# of an inherently multi-line phase (test run, artifact pull) so the live
+# log has an obvious visual marker.
+banner() {
+  printf '\n%s%s━━ %s ━━%s\n' "$_B" "$_CYA" "$*" "$_R"
+}
+
+# Status indicators. Single char prefix only — keeps width on narrow logs.
+ok()   { printf '%s✓%s %s\n' "$_GRN" "$_R" "$*"; }
+bad()  { printf '%s✗%s %s\n' "$_RED" "$_R" "$*"; }
+warn() { printf '%s!%s %s\n' "$_YEL" "$_R" "$*"; }
+note() { printf '%s·%s %s\n' "$_D"   "$_R" "$*"; }
+
+# group_open LABEL / group_close — thin wrapper around GitHub's collapse
+# markers. No extra body printed; banner() is the visible inside-group
+# heading when callers want one.
+group_open()  { echo "::group::$*"; }
+group_close() { echo "::endgroup::"; }
+
+# quiet_run LABEL LOG_FILE -- CMD ARGS …
+# Run CMD redirected to LOG_FILE (append). Print one colored ok/✗ line.
+# On failure, dump the last 80 lines of LOG_FILE inside a collapsed group
+# so the failing tail lands in the UI without surfacing the whole apply.
+quiet_run() {
+  local label="$1" log="$2"; shift 2
+  if "$@" >> "$log" 2>&1; then
+    ok "$label"
+  else
+    local rc=$?
+    bad "$label (rc=$rc) — last 80 lines of $(basename "$log"):"
+    group_open "tail $(basename "$log")"
+    tail -n 80 "$log"
+    group_close
+    return "$rc"
+  fi
+}

.github/workflows/e2e-baremetal.yml

@@ -0,0 +1,107 @@
+name: E2E Bare-Metal
+
+on:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  statuses: write
+
+env:
+  GIT_BRANCH: ${{ github.ref_name }}
+
+jobs:
+  e2e_baremetal:
+    name: Bare-Metal Multi-Node E2E
+    runs-on: [ self-hosted, ci-baremetal ]
+    timeout-minutes: 120
+    concurrency:
+      group: e2e-baremetal-cluster
+      cancel-in-progress: false
+    env:
+      BM_ISO_CACHE: /var/cache/spinifex-iso-builder
+    steps:
+      - name: Clean Workspace
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/checkout@v6
+
+      - uses: actions/create-github-app-token@v3
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      - uses: actions/checkout@v6
+        with:
+          repository: mulgadc/mulga
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: |
+            scripts/iso-builder
+            scripts/baremetal
+          path: mulga
+
+      - name: Install SSH Key
+        run: |
+          echo "${{ secrets.BM_SSH_PRIVATE_KEY }}" > "${RUNNER_TEMP}/bm-ci"
+          chmod 0600 "${RUNNER_TEMP}/bm-ci"
+          ssh-keygen -y -f "${RUNNER_TEMP}/bm-ci" > "${RUNNER_TEMP}/bm-ci.pub"
+
+      - name: Build Artifacts
+        run: |
+          mulga/scripts/baremetal/bm-build-artifacts.sh \
+            "$GITHUB_WORKSPACE" \
+            "${RUNNER_TEMP}/spinifex-tests.tar" \
+            "${RUNNER_TEMP}/bm-bins"
+
+      - name: Build PXE Artifacts
+        run: |
+          sudo make -C mulga/scripts/iso-builder \
+            build-rootfs build-squashfs build-initramfs \
+            CACHE="${BM_ISO_CACHE}" \
+            SPINIFEX_BIN="${RUNNER_TEMP}/bm-bins/spx" \
+            INSTALLER_BIN="${RUNNER_TEMP}/bm-bins/spinifex-installer" \
+            SPINIFEX_SRC="${GITHUB_WORKSPACE}" \
+            VIPERBLOCK_SRC="${RUNNER_TEMP}/bm-bins/viperblock" \
+            VERSION="$(cd "$GITHUB_WORKSPACE" && git describe --tags --always 2>/dev/null || echo dev)"
+
+      - name: Write ePDU credentials
+        run: |
+          install -m 0600 /dev/null "${RUNNER_TEMP}/epdu.env"
+          cat >> "${RUNNER_TEMP}/epdu.env" <<EOF
+          EPDU_HOST=192.168.1.63
+          EPDU_SNMP_USER=ci-bmpower
+          EPDU_SNMP_SEC_LEVEL=authPriv
+          EPDU_SNMP_AUTH_PROTO=MD5
+          EPDU_SNMP_AUTH_PASS=${{ secrets.BM_EPDU_AUTH_PASS }}
+          EPDU_SNMP_PRIV_PROTO=DES
+          EPDU_SNMP_PRIV_PASS=${{ secrets.BM_EPDU_PRIV_PASS }}
+          EOF
+
+      - name: Cache VM images
+        run: |
+          UBUNTU_IMG="${BM_ISO_CACHE}/ubuntu-26.04-cloudimg-amd64.img"
+          if [ ! -f "$UBUNTU_IMG" ]; then
+            echo "Downloading Ubuntu 26.04 cloud image..."
+            sudo curl -fsSL -o "$UBUNTU_IMG" \
+              "https://cloud-images.ubuntu.com/resolute/current/resolute-server-cloudimg-amd64.img"
+            echo "  Downloaded $(du -h "$UBUNTU_IMG" | cut -f1)"
+          else
+            echo "Ubuntu 26.04 cloud image cached ($(du -h "$UBUNTU_IMG" | cut -f1))"
+          fi
+
+
+      - name: Run Bare-Metal E2E
+        env:
+          BM_CI_PASSWORD: ${{ secrets.BM_CI_PASSWORD }}
+          BM_SSH_KEY: ${{ runner.temp }}/bm-ci
+          BM_EPDU_ENV: ${{ runner.temp }}/epdu.env
+          BM_UBUNTU_IMAGE: ${{ env.BM_ISO_CACHE }}/ubuntu-26.04-cloudimg-amd64.img
+          SPX_EXTERNAL_POOL: ${{ env.SPINIFEX_CI_POOL }}
+        run: |
+          sudo -E mulga/scripts/baremetal/run-baremetal-e2e.sh \
+            --vmlinuz   "${BM_ISO_CACHE}/vmlinuz" \
+            --initrd    "${BM_ISO_CACHE}/initrd.img" \
+            --squashfs  "${BM_ISO_CACHE}/spinifex.squashfs" \
+            --tests-tar "${RUNNER_TEMP}/spinifex-tests.tar"

.github/workflows/e2e-ddil.yml

@@ -11,6 +11,9 @@ on:
         required: false
         default: ""
 
+permissions:
+  contents: read
+
 env:
   GIT_BRANCH: ${{ github.ref_name }}
 
@@ -27,7 +30,7 @@ jobs:
       TOFU_LOG: /tmp/e2e-tofu-ddil-${{ github.run_id }}.log
     steps:
       - name: Clean Workspace
-        run: rm -rf "$GITHUB_WORKSPACE"/*
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         id: app-token
@@ -53,6 +56,26 @@ jobs:
           sparse-checkout: scripts/tofu-cluster
           path: mulga
 
+      - name: Install gotestsum
+        env:
+          GOTESTSUM_VERSION: "1.12.0"
+        run: |
+          # Prebuilt tarball, not `go install`: gotestsum v1.12.0 pulls
+          # x/tools v0.19.0 whose tokeninternal static-assert pegs a
+          # narrow Go window (~1.22). The self-hosted runner's Go falls
+          # outside that window, so source-install fails with
+          # `invalid array length -delta * delta`. A prebuilt binary
+          # sidesteps toolchain coupling entirely.
+          BIN_DIR="${RUNNER_TEMP}/gotestsum-bin"
+          mkdir -p "${BIN_DIR}"
+          if [ ! -x "${BIN_DIR}/gotestsum" ]; then
+            curl -fsSL \
+              "https://github.com/gotestyourself/gotestsum/releases/download/v${GOTESTSUM_VERSION}/gotestsum_${GOTESTSUM_VERSION}_linux_amd64.tar.gz" \
+              | tar -xz -C "${BIN_DIR}" gotestsum
+          fi
+          echo "${BIN_DIR}" >> "${GITHUB_PATH}"
+          "${BIN_DIR}/gotestsum" --version
+
       - name: Build Install Artifacts
         working-directory: mulga/scripts/tofu-cluster
         run: |
@@ -64,12 +87,16 @@ jobs:
       - name: Provision VMs
         working-directory: mulga/scripts/tofu-cluster
         run: |
+          set -euo pipefail
+          # shellcheck source=/dev/null
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           export TF_VAR_git_branch="${GIT_BRANCH}"
-          tofu init -input=false >> "$TOFU_LOG" 2>&1
-          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
-          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
-          tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
+          quiet_run "tofu workspace select"    "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
+          quiet_run "scrub-env (pre-clean)"    "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
+          quiet_run "tofu apply"               "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          ok "VMs provisioned"
 
       - name: Bootstrap via Binary Install
         working-directory: mulga/scripts/tofu-cluster
@@ -95,18 +122,91 @@ jobs:
           # tofu-cluster, wan_ip == cluster IP and the runner's outbound
           # IP is distinct — see e2e-ddil-test-harness.md §3 Scenario C.
           NODE_IPS=$(echo "$INVENTORY" | jq -r '[.nodes[].wan_ip] | join(",")')
+          PRIMARY_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
           echo "DDIL_NODES=${NODE_IPS}"
 
+          # Witness VMs talk to the cluster's AWS gateway via aws-sdk-go,
+          # which resolves credentials from the standard chain. spx admin
+          # init writes [spinifex] to ${SSH_USER}'s ~/.aws/credentials on
+          # node1 — pull them over SSH and export as the SDK's env-var
+          # entries so the harness can launch instances without depending
+          # on a cluster-bootstrapped profile being present on the runner.
+          CREDS=$(ssh -i "${SSH_KEY}" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
+            "${SSH_USER}@${PRIMARY_IP}" 'cat ~/.aws/credentials')
+          AWS_ACCESS_KEY_ID=$(echo "$CREDS" | awk -F '= *' '/^aws_access_key_id/ {print $2; exit}')
+          AWS_SECRET_ACCESS_KEY=$(echo "$CREDS" | awk -F '= *' '/^aws_secret_access_key/ {print $2; exit}')
+          if [ -z "$AWS_ACCESS_KEY_ID" ] || [ -z "$AWS_SECRET_ACCESS_KEY" ]; then
+            echo "::error::Failed to extract spinifex AWS credentials from ${PRIMARY_IP}"
+            exit 1
+          fi
+
+          # Guest SSH key: bootstrap-install.sh imports spinifex-key into EC2;
+          # the matching private material was generated on the primary node at
+          # ~/.ssh/spinifex-key. Pull it to the runner so the harness can SSH
+          # the witness via its MapPublicIpOnLaunch-assigned EIP.
+          GUEST_KEY="${RUNNER_TEMP}/spinifex-key"
+          scp -i "${SSH_KEY}" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
+            "${SSH_USER}@${PRIMARY_IP}:~/.ssh/spinifex-key" "${GUEST_KEY}"
+          chmod 600 "${GUEST_KEY}"
+
           mkdir -p "${RUNNER_TEMP}/ddil-results"
           cd "${GITHUB_WORKSPACE}/spinifex"
+          # gotestsum: human-readable PASS/FAIL/SKIP lines in Actions stdout,
+          # plus machine-readable jsonfile + JUnit XML artifacts. The JUnit
+          # file feeds the junit-report composite action below so failures
+          # become summary table rows + workflow annotations.
+          # shellcheck source=/dev/null
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          banner "DDIL scenarios"
           DDIL_NODES="${NODE_IPS}" \
           DDIL_SSH_USER="${SSH_USER}" \
           DDIL_SSH_KEY="${SSH_KEY}" \
+          DDIL_GUEST_SSH_KEY="${GUEST_KEY}" \
           DDIL_QUARANTINED="${{ inputs.quarantined }}" \
           AWS_REGION="${REGION}" \
-            go test -tags=e2e -timeout 45m -json \
-              ./tests/e2e/ddil/scenarios/... \
-              | tee "${RUNNER_TEMP}/ddil-results/results.json"
+          AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" \
+          AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" \
+            gotestsum --jsonfile "${RUNNER_TEMP}/ddil-results/results.json" \
+                      --junitfile "${RUNNER_TEMP}/ddil-results/junit-ddil.xml" \
+                      --format testname \
+                      -- -tags=e2e -timeout 45m ./tests/e2e/ddil/scenarios/...
+          rc=$?
+          if [[ $rc -eq 0 ]]; then ok "all scenarios passed"; else bad "scenarios failed (rc=$rc)"; fi
+          exit $rc
+
+      - name: Publish test summary
+        if: always()
+        uses: ./spinifex/.github/actions/junit-report
+        with:
+          junit-glob: ${{ runner.temp }}/ddil-results/junit-*.xml
+          title: "DDIL scenario results"
+
+      - name: Collect Cluster Logs
+        if: failure()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          # Pull spinifex-* journals + QEMU process listings from every
+          # node into the artifact bundle so failures can be triaged
+          # without re-provisioning the cluster. Best-effort — a missing
+          # node should not mask the original test failure.
+          set +e
+          INVENTORY=$(tofu output -json inventory)
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          SSH_USER=$(echo "$INVENTORY" | jq -r '.ssh_user')
+          LOG_DIR="${RUNNER_TEMP}/ddil-results/cluster-logs"
+          mkdir -p "${LOG_DIR}"
+
+          # shellcheck source=/dev/null
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          mapfile -t NODE_IPS < <(echo "$INVENTORY" | jq -r '.nodes[].wan_ip')
+          for ip in "${NODE_IPS[@]}"; do
+            SSH="ssh -i ${SSH_KEY} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${SSH_USER}@${ip}"
+            ${SSH} 'sudo journalctl -u "spinifex-*" --no-pager -n 5000' > "${LOG_DIR}/${ip}-spinifex.log" 2>&1
+            ${SSH} 'ps auxw | grep -E "qemu-system|spinifex" | grep -v grep' > "${LOG_DIR}/${ip}-procs.txt" 2>&1
+            ${SSH} 'ls -la /var/lib/spinifex 2>/dev/null; ls -la /var/log/spinifex 2>/dev/null' > "${LOG_DIR}/${ip}-state.txt" 2>&1
+            ok "$ip: $(wc -l < "${LOG_DIR}/${ip}-spinifex.log") journal lines, $(wc -l < "${LOG_DIR}/${ip}-procs.txt") procs"
+          done
 
       - name: Upload DDIL Results
         if: always()
@@ -120,6 +220,10 @@ jobs:
         if: always()
         working-directory: mulga/scripts/tofu-cluster
         run: |
-          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
-          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+          set -euo pipefail
+          # shellcheck source=/dev/null
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy"          "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
+          ok "teardown complete"

.github/workflows/e2e-nightly.yml

@@ -12,8 +12,10 @@ on:
 permissions:
   contents: read
 
+# Scheduled runs are loaded from the default branch (main), so the schedule
+# trigger must stay in sync on main for cron to fire.
 env:
-  GIT_BRANCH: ${{ github.ref_name }}
+  GIT_BRANCH: ${{ github.event_name == 'schedule' && 'dev' || github.ref_name }}
   UBUNTU_GOLD_IMAGE: /mnt/disk1/gold-images/spinifex-image-builder-ubuntu-24.04.qcow2
 
 jobs:
@@ -58,7 +60,7 @@ jobs:
 
       - name: Clean Workspace
         if: steps.gate.outputs.selected == 'true'
-        run: rm -rf "$GITHUB_WORKSPACE"/*
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         if: steps.gate.outputs.selected == 'true'

.github/workflows/e2e.yml

@@ -19,6 +19,7 @@ on:
         required: false
 
 permissions:
+  contents: read
   statuses: write
 
 env:
@@ -28,12 +29,12 @@ jobs:
   e2e_single:
     name: Single-Node E2E
     runs-on: [ self-hosted, ci-single ]
-    timeout-minutes: 30
+    timeout-minutes: 35
     env:
       TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
     steps:
       - name: Clean Workspace
-        run: rm -rf "$GITHUB_WORKSPACE"/*
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         id: app-token
@@ -97,12 +98,12 @@ jobs:
   e2e_pseudo_multi:
     name: Pseudo Multi-Node E2E
     runs-on: [ self-hosted, ci-pseudo ]
-    timeout-minutes: 30
+    timeout-minutes: 25
     env:
       TOFU_LOG: /tmp/e2e-tofu-pseudo-${{ github.run_id }}.log
     steps:
       - name: Clean Workspace
-        run: rm -rf "$GITHUB_WORKSPACE"/*
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         id: app-token
@@ -161,12 +162,12 @@ jobs:
   e2e_real_multi:
     name: Real Multi-Node E2E
     runs-on: [ self-hosted, ci-multi ]
-    timeout-minutes: 35
+    timeout-minutes: 30
     env:
       TOFU_LOG: /tmp/e2e-tofu-multi-${{ github.run_id }}.log
     steps:
       - name: Clean Workspace
-        run: rm -rf "$GITHUB_WORKSPACE"/*
+        run: sudo rm -rf "$GITHUB_WORKSPACE"/*
 
       - uses: actions/create-github-app-token@v3
         id: app-token

.github/workflows/release.yml

@@ -31,6 +31,14 @@ jobs:
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v4
 
+      - name: Set up Go
+        uses: actions/setup-go@v6
+        with:
+          go-version: '1.26.3'
+
+      - name: Build microVM artifacts (lb-agent + kernel + initramfs)
+        run: cd spinifex && make build-microvm-image
+
       - name: Build amd64 Tarball
         env:
           VERSION: ${{ github.ref_name }}
@@ -192,3 +200,151 @@ jobs:
           tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
           tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
           ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+
+  iso:
+    needs: release
+    runs-on: [ self-hosted, ci-single ]
+    timeout-minutes: 90
+    steps:
+      - name: Clean Workspace
+        run: |
+          ROOTFS="${GITHUB_WORKSPACE}/mulga/scripts/iso-builder/.cache/rootfs"
+          # Mirror make clean's umount sequence unconditionally — lazy unmount
+          # detaches immediately even if the mount is busy, so rm -rf is safe.
+          sudo umount -l -R "${ROOTFS}" 2>/dev/null || true
+          sudo umount -l "${ROOTFS}/dev/pts" 2>/dev/null || true
+          sudo umount -l "${ROOTFS}/dev" 2>/dev/null || true
+          sudo umount -l "${ROOTFS}/sys" 2>/dev/null || true
+          sudo umount -l "${ROOTFS}/proc" 2>/dev/null || true
+          sudo rm -rf "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/create-github-app-token@v3
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      - name: Checkout Spinifex
+        uses: actions/checkout@v6
+        with:
+          path: spinifex
+          ref: ${{ github.ref }}
+
+      - name: Checkout Predastore
+        uses: actions/checkout@v6
+        with:
+          repository: mulgadc/predastore
+          path: predastore
+
+      - name: Checkout Viperblock
+        uses: actions/checkout@v6
+        with:
+          repository: mulgadc/viperblock
+          path: viperblock
+
+      - name: Checkout Mulga iso-builder
+        uses: actions/checkout@v6
+        with:
+          repository: mulgadc/mulga
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: scripts/iso-builder
+          path: mulga
+
+      - name: Clean ISO builder cache
+        run: sudo make -C mulga/scripts/iso-builder clean
+
+      - name: Set up Go
+        uses: actions/setup-go@v6
+        with:
+          go-version-file: spinifex/go.mod
+          cache: false
+
+      - name: Configure Go workspace
+        run: go work init ./spinifex ./predastore ./viperblock
+
+      - name: Install ISO builder dependencies
+        run: |
+          sudo apt-get update -qq
+          sudo apt-get install -y \
+            debootstrap squashfs-tools xorriso grub-pc-bin grub-efi-amd64-bin \
+            grub-common mtools busybox-static ovmf zstd \
+            nbdkit nbdkit-plugin-dev pkg-config gcc
+
+      - name: Build spinifex binaries
+        run: cd spinifex && make build
+
+      - name: Build viperblock
+        run: cd viperblock && make build
+
+      - name: Build release ISO
+        id: build-iso
+        env:
+          VERSION: ${{ github.ref_name }}
+        run: |
+          sudo make -C mulga/scripts/iso-builder release \
+            VERSION="${VERSION}" \
+            SPINIFEX_SRC="${GITHUB_WORKSPACE}/spinifex" \
+            VIPERBLOCK_SRC="${GITHUB_WORKSPACE}/viperblock" \
+            SPINIFEX_BIN="${GITHUB_WORKSPACE}/spinifex/bin/spx" \
+            INSTALLER_BIN="${GITHUB_WORKSPACE}/spinifex/bin/spinifex-installer"
+
+      - name: Wait for verify
+        id: wait-verify
+        if: steps.build-iso.outcome == 'success'
+        continue-on-error: true
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          echo "Waiting for verify job to complete..."
+          while true; do
+            CONCLUSION=$(gh api \
+              "repos/${{ github.repository }}/actions/runs/${{ github.run_id }}/jobs" \
+              --jq '.jobs[] | select(.name == "verify") | .conclusion // empty')
+            if [ -n "$CONCLUSION" ]; then
+              echo "Verify conclusion: $CONCLUSION"
+              [ "$CONCLUSION" = "success" ] && exit 0
+              exit 1
+            fi
+            echo "Verify still running, retrying in 30s..."
+            sleep 30
+          done
+
+      - name: Ensure AWS CLI is installed
+        if: steps.build-iso.outcome == 'success' && steps.wait-verify.outcome == 'success'
+        run: |
+          if ! command -v aws &>/dev/null; then
+            curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o /tmp/awscliv2.zip
+            unzip -q /tmp/awscliv2.zip -d /tmp/aws-install
+            sudo /tmp/aws-install/aws/install
+            rm -rf /tmp/awscliv2.zip /tmp/aws-install
+          fi
+
+      - name: Upload ISO to Cloudflare R2
+        id: upload
+        if: steps.build-iso.outcome == 'success' && steps.wait-verify.outcome == 'success'
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.CF_R2_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.CF_R2_SECRET_ACCESS_KEY }}
+          CF_R2_ENDPOINT_URL: ${{ secrets.CF_R2_ENDPOINT_URL }}
+          VERSION: ${{ github.ref_name }}
+        run: |
+          aws --endpoint-url="${CF_R2_ENDPOINT_URL}" \
+            s3 cp "mulga/scripts/iso-builder/dist/spinifex-${VERSION}.iso" \
+                  "s3://spinifex-iso/spinifex-${VERSION}.iso"
+          aws --endpoint-url="${CF_R2_ENDPOINT_URL}" \
+            s3 cp "mulga/scripts/iso-builder/dist/spinifex-${VERSION}.iso.sha256" \
+                  "s3://spinifex-iso/spinifex-${VERSION}.iso.sha256"
+
+      - name: Mark ISO failed
+        if: always() && (steps.build-iso.outcome == 'failure' || steps.upload.outcome == 'failure')
+        env:
+          GH_TOKEN: ${{ github.token }}
+          TAG: ${{ github.ref_name }}
+          REPO: ${{ github.repository }}
+        run: |
+          EXISTING_BODY=$(gh release view "$TAG" --repo "$REPO" --json body -q .body)
+          NOTES_FILE=$(mktemp)
+          printf 'ISO build/upload failed — see workflow run for details.\n\n%s' "$EXISTING_BODY" > "$NOTES_FILE"
+          gh release edit "$TAG" --repo "$REPO" --notes-file "$NOTES_FILE"
+          rm -f "$NOTES_FILE"

.github/workflows/spinifex.yml

@@ -17,6 +17,9 @@ on:
       - dev
   workflow_dispatch:
 
+permissions:
+  contents: read
+
 jobs:
   build_and_test:
     name: Build and Test
@@ -33,12 +36,14 @@ jobs:
         uses: actions/checkout@v6
         with:
           repository: mulgadc/viperblock
+          ref: dev
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v6
         with:
           repository: mulgadc/predastore
+          ref: dev
           path: predastore
 
       - name: Setup Go
@@ -56,6 +61,9 @@ jobs:
       - name: Run Tests with Coverage
         run: make -C spinifex test-cover
 
+      - name: Run Action Tests
+        run: make -C spinifex test-actions
+
       - name: Check Diff Coverage
         run: cd spinifex && scripts/diff-coverage.sh coverage.out --base HEAD~1
 
@@ -73,12 +81,14 @@ jobs:
         uses: actions/checkout@v6
         with:
           repository: mulgadc/viperblock
+          ref: dev
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v6
         with:
           repository: mulgadc/predastore
+          ref: dev
           path: predastore
 
       - name: Setup Go
@@ -107,12 +117,14 @@ jobs:
         uses: actions/checkout@v6
         with:
           repository: mulgadc/viperblock
+          ref: dev
           path: viperblock
 
       - name: Checkout Predastore
         uses: actions/checkout@v6
         with:
           repository: mulgadc/predastore
+          ref: dev
           path: predastore
 
       - name: Setup Go
@@ -127,7 +139,7 @@ jobs:
       - name: Lint
         uses: golangci/golangci-lint-action@v9
         with:
-          version: v2.11.3
+          version: v2.12.2
           working-directory: spinifex
 
       - name: Govulncheck

.github/workflows/ui.yml

@@ -17,6 +17,9 @@ on:
       - .github/workflows/ui.yml
   workflow_dispatch:
 
+permissions:
+  contents: read
+
 jobs:
   ui_build_and_test:
     name: UI Build and Test
@@ -46,8 +49,8 @@ jobs:
       - name: Install Dependencies
         run: pnpm install --frozen-lockfile
 
-      - name: Run Check
-        run: pnpm check
+      - name: Run Lint
+        run: pnpm lint
 
       - name: Run Tests with Coverage
         run: pnpm test:coverage

.gitignore

@@ -17,6 +17,8 @@ go.work.sum
 # Build outputs
 bin/
 dist/
+build/microvm/vmlinuz
+build/microvm/initramfs.cpio.gz
 !spinifex/services/spinifexui/frontend/dist/
 tmp/
 *.test

Makefile

@@ -63,15 +63,15 @@ LDFLAGS := -s -w -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=$(VERSI
 
 go_build:
 	@echo -e "\n....Building $(GO_PROJECT_NAME)"
-	go build $(GO_BUILD_MOD) -ldflags "$(LDFLAGS)" -o ./bin/$(GO_PROJECT_NAME) cmd/spinifex/main.go
+	GOFIPS140=v1.0.0 go build $(GO_BUILD_MOD) -ldflags "$(LDFLAGS)" -o ./bin/$(GO_PROJECT_NAME) cmd/spinifex/main.go
 
 build-installer:
 	@echo -e "\n....Building spinifex-installer"
-	go build -ldflags "-s -w" -o ./bin/spinifex-installer cmd/installer/main.go
+	GOFIPS140=v1.0.0 go build -ldflags "-s -w" -o ./bin/spinifex-installer cmd/installer/main.go
 
 build-lb-agent:
 	@echo -e "\n....Building lb-agent (static)"
-	CGO_ENABLED=0 go build -ldflags "-s -w" -o ./bin/lb-agent cmd/lb-agent/main.go
+	CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -ldflags "-s -w" -o ./bin/lb-agent cmd/lb-agent/main.go
 
 build-system-image: ## Build a system image from manifest (use IMAGE=lb)
 ifndef IMAGE
@@ -79,8 +79,25 @@ ifndef IMAGE
 endif
 	./scripts/build-system-image.sh scripts/images/$(IMAGE).conf
 
-build-lb-image: ## Build LB Alpine image
-	$(MAKE) build-system-image IMAGE=lb
+MICROVM_OUT_DIR := build/microvm
+MICROVM_ARTIFACTS := $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz
+MICROVM_INPUTS := scripts/build-microvm-image.sh $(MICROVM_OUT_DIR)/init.sh $(MICROVM_OUT_DIR)/inittab bin/lb-agent
+
+# Grouped target — script writes both files in one run.
+$(MICROVM_ARTIFACTS) &: $(MICROVM_INPUTS)
+	./scripts/build-microvm-image.sh
+
+# Only triggers when bin/lb-agent is missing; preserves the artifact's mtime so
+# build-microvm-image stays correctly stale-aware.
+bin/lb-agent:
+	$(MAKE) build-lb-agent
+
+build-microvm-image: $(MICROVM_ARTIFACTS) ## Build microVM kernel + initramfs (incremental — skips when up to date)
+
+install-microvm: $(MICROVM_ARTIFACTS) ## Install microVM artifacts to /usr/share/spinifex/microvm/
+	sudo install -d /usr/share/spinifex/microvm
+	sudo install -m 0644 $(MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/vmlinuz
+	sudo install -m 0644 $(MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/initramfs.cpio.gz
 
 go_run:
 	@echo -e "\n....Running $(GO_PROJECT_NAME)...."
@@ -89,9 +106,23 @@ go_run:
 # Preflight — runs the same checks as GitHub Actions (lint + vuln + tests).
 # Use this before committing to catch CI failures locally.
 preflight:
-	@$(MAKE) --no-print-directory QUIET=1 lint govulncheck test-cover diff-coverage test-race
+	@$(MAKE) --no-print-directory QUIET=1 manifest-check lint govulncheck test-cover diff-coverage test-race test-harness
 	@echo -e "\n ✅ Preflight passed — safe to commit."
 
+# E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
+# default `go test ./spinifex/...`. Runs with mocked AWS clients — no
+# infrastructure required, safe to run in CI without a cluster.
+test-harness:
+	@echo -e "\n....Running e2e harness unit tests...."
+	$(_Q)LOG_IGNORE=1 go test -tags=e2e -timeout 60s ./tests/e2e/harness/... $(_RACEQ)
+
+# Validate docs/service-interfaces.yaml. Schema check + cross-reference
+# of services/suites/fixtures + on-disk path existence. Subject content
+# vs source is enforced separately in Bead 5 drift lint.
+manifest-check:
+	@echo -e "\n....Checking service-interfaces.yaml...."
+	@go run ./tests/e2e/manifest-check/cmd/manifest-check -repo-root . -manifest docs/service-interfaces.yaml
+
 # Run unit tests
 test:
 	@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
@@ -109,6 +140,12 @@ test-race:
 	@echo -e "\n....Running tests with race detector for $(GO_PROJECT_NAME)...."
 	$(_Q)LOG_IGNORE=1 go test -race -timeout 300s ./spinifex/... $(_RACEQ)
 
+# Unit tests for in-repo GitHub Actions (e.g. .github/actions/e2e-analyze).
+# Kept out of `test-cover` so coverage % isn't diluted by CI-only tooling.
+test-actions:
+	@echo -e "\n....Running action tests...."
+	LOG_IGNORE=1 go test -timeout 60s ./.github/actions/...
+
 # Check that new/changed code meets coverage threshold (runs tests first)
 diff-coverage: test-cover
 	@QUIET=$(QUIET) scripts/diff-coverage.sh $(COVERPROFILE)
@@ -122,9 +159,18 @@ run:
 	$(MAKE) go_build
 	$(MAKE) go_run
 
-# Fast iteration: build + install binary + restart all services
+# Fast iteration: build + install binary + restart all services.
+# Microvm artifacts are reinstalled when they already exist on disk — the rule's
+# input timestamps drive a rebuild only if anything actually changed. On a fresh
+# checkout (no build/microvm/vmlinuz yet) the install step is skipped; run
+# `make install-microvm` explicitly the first time.
 deploy: build
 	sudo install -m 755 bin/spx /usr/local/bin/spx
+	@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ]; then \
+		$(MAKE) install-microvm; \
+	else \
+		echo "[deploy] microvm artifacts absent — run 'make install-microvm' for first-time setup"; \
+	fi
 	sudo systemctl daemon-reload
 	sudo systemctl restart spinifex.target
 
@@ -144,14 +190,15 @@ install-system:
 		-o Dpkg::Options::="--force-confdef" \
 		-o Dpkg::Options::="--force-confold" \
 		nbdkit nbdkit-plugin-dev pkg-config $(QEMU_PACKAGES) qemu-utils qemu-kvm \
+		ovmf qemu-efi-aarch64 \
 		libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
 		iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
 		ovn-central ovn-host openvswitch-switch dhcpcd-base
 
 install-go:
-	@echo -e "\n....Installing Go 1.26.2 for $(ARCH) ($(GO_ARCH))...."
+	@echo -e "\n....Installing Go 1.26.3 for $(ARCH) ($(GO_ARCH))...."
 	@if [ ! -d "/usr/local/go" ]; then \
-		curl -L https://go.dev/dl/go1.26.2.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
+		curl -L https://go.dev/dl/go1.26.3.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
 	else \
 		echo "Go already installed in /usr/local/go"; \
 	fi
@@ -206,6 +253,14 @@ distro-amd64:
 		-f build/Dockerfile.distro \
 		--output type=local,dest=dist/amd64/ \
 		../
+	@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ] && [ -f $(MICROVM_OUT_DIR)/initramfs.cpio.gz ]; then \
+		echo "[distro-amd64] staging microvm artifacts into tarball"; \
+		mkdir -p dist/amd64/microvm; \
+		cp $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz dist/amd64/microvm/; \
+	else \
+		echo "[distro-amd64] WARNING: microvm artifacts missing — tarball will not include them"; \
+		echo "[distro-amd64]          run 'make build-microvm-image' before 'make distro-amd64'"; \
+	fi
 	tar -czf dist/spinifex-$(VERSION)-linux-amd64.tar.gz -C dist/amd64 .
 	sha256sum dist/spinifex-$(VERSION)-linux-amd64.tar.gz > dist/spinifex-$(VERSION)-linux-amd64.tar.gz.sha256
 
@@ -238,9 +293,12 @@ ansible-dev-install:
 ansible-dev-reset:
 	cd scripts/ansible && ansible-playbook playbooks/dev-reset.yml
 
-.PHONY: build build-ui build-installer build-lb-agent build-system-image build-lb-image go_build go_run preflight test test-cover test-race diff-coverage bench run \
+ansible-dev-deploy:
+	cd scripts/ansible && ansible-playbook playbooks/dev-deploy.yml
+
+.PHONY: build build-ui build-installer build-lb-agent build-system-image build-microvm-image install-microvm go_build go_run preflight test test-cover test-race test-actions test-harness manifest-check diff-coverage bench run \
 	deploy reinstall clean \
 	install-system install-go install-aws quickinstall \
 	lint fix govulncheck \
 	distro distro-amd64 distro-arm64 distro-clean \
-	ansible-dev-preflight ansible-dev-teardown ansible-dev-install ansible-dev-reset
+	ansible-dev-preflight ansible-dev-teardown ansible-dev-install ansible-dev-reset ansible-dev-deploy

README.md

@@ -93,9 +93,24 @@ AWS-Compatible Interfaces – Provision infrastructure with awscli, Terraform, o
 
 Installation requires an Ubuntu / Debian system. See the detailed documentation at [docs.mulgadc.com](https://docs.mulgadc.com) for maintaining and installing Spinifex.
 
+### Bare Metal ISO
+
+The recommended installation is a [bootable x86 installer](https://iso.mulgadc.com/spinifex.iso) for bare-metal hardware.
+
+```bash
+curl -fLO https://iso.mulgadc.com/spinifex.iso
+```
+
+Follow the [USB install guide](https://docs.mulgadc.com/docs/install-usb) to write the ISO to USB and install on your hardware. The install guide walks through the full process.
+
+
 ### Single Node Install
 
-The installation is straightforward to set up and running on a single node for testing purposes.
+The installation is straightforward to set up and running on a single node for testing purposes. Debian 13 is currently supported, additional Linux distributions are on the immediate roadmap.
+
+>*Prerequisite:* Linux bridge for networking.
+
+Spinifex requires a Linux bridge configured on the host for VM networking. See the [single-node install guide](https://docs.mulgadc.com/docs/install#prerequisites) prerequisites for setup details.
 
 ```bash
 curl -fsSL https://install.mulgadc.com | bash

build/Dockerfile.distro

@@ -30,13 +30,13 @@ COPY viperblock/ ./viperblock/
 
 # Build spinifex binary (spx)
 RUN COMMIT=$(cd spinifex && git rev-parse --short HEAD 2>/dev/null || echo "unknown") && \
-    cd spinifex && go build -ldflags "-s -w \
+    cd spinifex && GOFIPS140=v1.0.0 go build -ldflags "-s -w \
     -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=${VERSION} \
     -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Commit=${COMMIT}" \
     -o /out/spx cmd/spinifex/main.go
 
 # Build nbdkit viperblock plugin (.so)
-RUN cd viperblock && go build -o /out/nbdkit-viperblock-plugin.so -buildmode=c-shared nbd/viperblock.go
+RUN cd viperblock && GOFIPS140=v1.0.0 go build -o /out/nbdkit-viperblock-plugin.so -buildmode=c-shared nbd/viperblock.go
 
 # Copy setup scripts and systemd units
 RUN cp spinifex/scripts/setup.sh /out/setup.sh 2>/dev/null || true

build/microvm/init.sh

@@ -0,0 +1,56 @@
+#!/bin/sh
+mount -t proc  proc  /proc
+mount -t sysfs sys   /sys
+mount -t devtmpfs dev /dev
+
+# virtio_mmio is built-in (CONFIG_VIRTIO_MMIO=y) so virtio-net devices are
+# already enumerated by the time PID 1 runs — no modprobe needed.
+# qemu_fw_cfg and virtio_net are =m in stock Alpine linux-virt, so load them.
+modprobe qemu_fw_cfg 2>/dev/null || modprobe fw_cfg_sysfs 2>/dev/null || true
+modprobe virtio_net 2>/dev/null || true
+
+for i in 1 2 3 4 5; do
+    [ -f /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw ] && break
+    sleep 0.02
+done
+
+if [ ! -f /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw ]; then
+    echo "[init] FATAL: fw_cfg netcfg not available" >&2
+    exit 1
+fi
+
+. /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/netcfg/raw
+
+for n in 0 1 2 3 4 5; do
+    eval mac=\$NIC${n}_MAC
+    [ -z "$mac" ] && continue
+
+    iface=""
+    for attempt in 1 2 3 4 5; do
+        iface=$(ip -o link | awk -v m="$mac" 'tolower($0) ~ tolower(m) {print $2}' | tr -d :)
+        [ -n "$iface" ] && break
+        sleep 0.02
+    done
+
+    if [ -z "$iface" ]; then
+        echo "[init] WARNING: no interface found for MAC $mac (NIC$n), skipping" >&2
+        continue
+    fi
+
+    eval cidr=\$NIC${n}_CIDR
+    eval gw=\$NIC${n}_GW
+    eval default=\$NIC${n}_DEFAULT
+    eval rdst=\$NIC${n}_ROUTE_DST
+    eval rvia=\$NIC${n}_ROUTE_VIA
+
+    ip link set "$iface" up
+    [ -n "$cidr" ] && ip addr add "$cidr" dev "$iface"
+    [ "$default" = "1" ] && [ -n "$gw" ] && ip route add default via "$gw" dev "$iface"
+    [ -n "$rdst" ] && [ -n "$rvia" ] && ip route add "$rdst" via "$rvia" dev "$iface"
+done
+
+cp /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/lb-agent-env/raw /etc/conf.d/lb-agent
+mkdir -p /etc/ssl/certs
+cp /sys/firmware/qemu_fw_cfg/by_name/opt/spinifex/ca-cert/raw /etc/ssl/certs/ca-certificates.crt
+
+exec /sbin/init

build/microvm/inittab

@@ -0,0 +1,4 @@
+::sysinit:/etc/rc.local
+::respawn:/bin/sh -c 'set -a; . /etc/conf.d/lb-agent; set +a; exec /usr/local/bin/lb-agent'
+::ctrlaltdel:/sbin/reboot
+::shutdown:/bin/umount -a -r

build/scripts/predastore-start.sh

@@ -5,17 +5,20 @@
 
 CONF=/etc/spinifex/spinifex.toml
 BIND="0.0.0.0:8443"
-NODE_ID="0"
+# -1 = co-located mode (all [[db]] peers in one process). Single-node spinifex
+# templates list 3 DB peers on 127.0.0.1; only -1 can serve that topology.
+# Multi-node spinifex.toml emits node_id = N (N >= 1) which overrides below.
+# predastore rejects NODE_ID=0 since d3a02f0 (strconv.Atoi-of-garbage guard).
+NODE_ID="-1"
 
 if [ -f "$CONF" ]; then
     H=$(awk -F'"' '/\[nodes\..*\.predastore\]/{f=1} f&&/^host/{print $2;exit}' "$CONF")
     if [ -n "$H" ]; then
         BIND="$H"
-        BACKEND="distributed"
     fi
 
     N=$(awk -F'= *' '/node_id/{gsub(/ /,"",$2);print $2;exit}' "$CONF")
-    if [ -n "$N" ] && [ "$N" != "0" ]; then
+    if [ -n "$N" ]; then
         NODE_ID="$N"
     fi
 fi

build/systemd/regenerate-ssh-host-keys.service

@@ -0,0 +1,12 @@
+[Unit]
+Description=Regenerate SSH host keys
+ConditionPathExists=!/etc/ssh/ssh_host_ed25519_key
+Before=ssh.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/ssh-keygen -A
+RemainAfterExit=yes
+
+[Install]
+WantedBy=multi-user.target

build/systemd/spinifex-daemon.service

@@ -2,27 +2,67 @@
 Description=Spinifex Daemon (VM Orchestration)
 PartOf=spinifex.target
 After=spinifex-nats.service spinifex-viperblock.service openvswitch-switch.service ovn-controller.service
-Wants=spinifex-nats.service
+# No Wants=spinifex-nats.service: Wants= reactivates a stopped unit during
+# `systemctl restart spinifex-daemon`, which would mask the standalone-startup
+# path the Tier 1 DDIL Scenario B exercises. spinifex.target Wants= covers
+# cold boot.
 
 [Service]
 Type=simple
 User=spinifex-daemon
 Group=spinifex
-ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
+# DDIL Tier 1 (siv-17): no ExecStartPre wait-for-nats. The daemon starts in
+# standalone mode and reconnects to NATS in the background — process-exiting
+# on a missing NATS would kill the local VM management plane.
+#
+# Opt-in strict-startup (§1d-strict, siv-44): set SPINIFEX_REQUIRE_NATS=1 to
+# restore pre-DDIL fail-fast UX. First NATS connect uses a ~30s bounded retry
+# and the daemon aborts on timeout. Intended for dev/test/single-node deploys
+# where silent NATS misconfig is more confusing than helpful. Do NOT set this
+# in the shipped unit — it re-introduces the SPOF that Tier 1 removed.
 ExecStart=/usr/local/bin/spx service spinifex start
 Restart=on-failure
 RestartSec=5
+# DDIL Tier 1 (mulga-siv-58): KillMode=process so systemd only signals the
+# daemon's main PID, not the cgroup. QEMU children stay alive across
+# `systemctl restart spinifex-daemon` and the new daemon reattaches to them
+# via the persisted local state file + pidfile + QMP socket. With the
+# default control-group mode, restart would terminate every guest VM —
+# defeating Scenario B's "daemon restart preserves running instances".
+KillMode=process
+# 15s upper bound on the SIGTERM grace window. The new SIGTERM handler
+# skips VM teardown (it now just persists state and exits), so 15s is
+# plenty of slack for the cluster-manager + NATS unsubscribe path.
+TimeoutStopSec=15
 OOMScoreAdjust=-500
 Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
 Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex/
 Environment=SPINIFEX_WAL_DIR=/var/lib/spinifex/spinifex/
+# DDIL Tier 1 (mulga-siv-58): QEMU pidfiles, QMP sockets, and the
+# shared qemu.log all resolve via utils.pidPath() — which falls back
+# to os.TempDir() (=/tmp) when $HOME/spinifex doesn't exist. With
+# PrivateTmp=yes, each daemon start gets a fresh /tmp namespace, so
+# pidfiles written by QEMU (running in the OLD daemon's namespace)
+# are invisible to the new daemon. The /local/instances handler then
+# reports PID=0 and Scenario B fails its "live QEMU PID" assertion.
+# Pin XDG_RUNTIME_DIR to the host-wide tmpfs at /run/spinifex (created
+# by tmpfiles.d, already in ReadWritePaths) so pidPath() returns a
+# stable, namespace-independent path that both daemon incarnations and
+# QEMU children resolve identically.
+Environment=XDG_RUNTIME_DIR=/run/spinifex
 
-# Filesystem access — NoNewPrivileges omitted (daemon uses sudo for ip, ovs-vsctl)
+# Filesystem access — NoNewPrivileges omitted (daemon uses sudo for ip, ovs-vsctl, dhcpcd)
+# CAP_SYS_ADMIN: required for GPU passthrough sysfs operations (vfio group management).
+# CAP_DAC_OVERRIDE: sysfs driver bind/unbind paths (/sys/bus/pci/drivers/vfio-pci/bind,
+# unbind) are root:root 0200; non-root processes need CAP_DAC_OVERRIDE to open them.
+AmbientCapabilities=CAP_SYS_ADMIN CAP_DAC_OVERRIDE
 ProtectSystem=full
 PrivateDevices=no
 DevicePolicy=closed
 DeviceAllow=/dev/kvm rw
 DeviceAllow=/dev/net/tun rw
+DeviceAllow=char-vfio rw
+DeviceAllow=/dev/vfio/vfio rw
 ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
 ReadWritePaths=/var/lib/spinifex/spinifex /var/log/spinifex /run/spinifex /run/spinifex/nbd
 
@@ -50,6 +90,9 @@ RemoveIPC=yes
 LimitCORE=0
 LimitNOFILE=65536
 LimitNPROC=4096
+# VFIO GPU passthrough requires locking all guest RAM into physical memory.
+# Without this, QEMU fails with RLIMIT_MEMLOCK exceeded when pinning guest pages.
+LimitMEMLOCK=infinity
 
 [Install]
 WantedBy=spinifex.target

build/systemd/spinifex-predastore.service

@@ -17,6 +17,7 @@ Environment=SPINIFEX_PREDASTORE_CONFIG_PATH=/etc/spinifex/predastore/predastore.
 Environment=SPINIFEX_PREDASTORE_BASE_PATH=/var/lib/spinifex/predastore/
 Environment=SPINIFEX_PREDASTORE_TLS_CERT=/etc/spinifex/server.pem
 Environment=SPINIFEX_PREDASTORE_TLS_KEY=/etc/spinifex/server.key
+Environment=SPINIFEX_PREDASTORE_ENCRYPTION_KEY_FILE=/etc/spinifex/predastore/encryption.key
 
 # Filesystem access
 NoNewPrivileges=yes

build/udev/99-spinifex-vfio.rules

@@ -0,0 +1,5 @@
+# Allow spinifex group to access VFIO devices for GPU passthrough.
+# QEMU (launched by spinifex-daemon, group spinifex) must open /dev/vfio/N
+# to use vfio-pci passthrough devices. The kernel creates these nodes as
+# root:root 0600 by default; this rule grants group-level read-write access.
+SUBSYSTEM=="vfio", GROUP="spinifex", MODE="0660"

cmd/dhcptest/main.go

@@ -0,0 +1,172 @@
+// dhcptest is a diagnostic tool that runs a DHCP DORA handshake using the
+// same nclient4 library and options that spinifex-vpcd uses for gateway pool
+// leases. Run it on a node to verify DHCP works at the library level before
+// debugging the full spinifex stack.
+//
+// Usage (must run as root — needs AF_PACKET):
+//
+//	sudo dhcptest --iface=br-wan
+//	sudo dhcptest --iface=br-wan --mac=02:00:00:7c:b7:b9 --promisc
+//	sudo dhcptest --iface=br-wan --spinifex-id=gateway-wan1
+package main
+
+import (
+	"context"
+	"flag"
+	"fmt"
+	"log"
+	"net"
+	"os"
+	"os/exec"
+	"strings"
+	"time"
+
+	"github.com/insomniacslk/dhcp/dhcpv4"
+	"github.com/insomniacslk/dhcp/dhcpv4/nclient4"
+)
+
+func main() {
+	iface := flag.String("iface", "br-wan", "Interface/bridge to bind the DHCP client to")
+	macStr := flag.String("mac", "", "Hardware address for DORA (chaddr + AF_PACKET filter). If empty, uses the real interface MAC.")
+	spinifexID := flag.String("spinifex-id", "", "Generate a spinifex virtual MAC from this resource ID (e.g. 'gateway-wan1'). Overrides --mac.")
+	clientID := flag.String("client-id", "", "DHCP option 61 client identifier. Defaults to --spinifex-id or --mac value.")
+	hostname := flag.String("hostname", "", "DHCP option 12 hostname")
+	vendorClass := flag.String("vendor-class", "mulga-spinifex-gw", "DHCP option 60 vendor class identifier")
+	promisc := flag.Bool("promisc", false, "Enable promiscuous mode on the interface before the test (mirrors vpcd's setBridgePromisc)")
+	timeout := flag.Duration("timeout", 10*time.Second, "Per-attempt DHCP timeout")
+	retries := flag.Int("retries", 3, "Number of DORA attempts before giving up")
+	verbose := flag.Bool("verbose", false, "Print full DHCP packet contents")
+	flag.Parse()
+
+	// Resolve the hardware address to use.
+	var hwAddr net.HardwareAddr
+	var err error
+
+	switch {
+	case *spinifexID != "":
+		hwAddr, err = net.ParseMAC(generateMAC(*spinifexID))
+		if err != nil {
+			log.Fatalf("generateMAC(%q): %v", *spinifexID, err)
+		}
+		fmt.Printf("spinifex virtual MAC for %q: %s\n", *spinifexID, hwAddr)
+	case *macStr != "":
+		hwAddr, err = net.ParseMAC(*macStr)
+		if err != nil {
+			log.Fatalf("parse --mac %q: %v", *macStr, err)
+		}
+	default:
+		hwAddr, err = ifaceMAC(*iface)
+		if err != nil {
+			log.Fatalf("get MAC for %s: %v", *iface, err)
+		}
+		fmt.Printf("using real interface MAC: %s\n", hwAddr)
+	}
+
+	// Resolve client-id: default to spinifex-id, then MAC string.
+	cid := *clientID
+	if cid == "" {
+		if *spinifexID != "" {
+			cid = *spinifexID
+		} else {
+			cid = hwAddr.String()
+		}
+	}
+
+	fmt.Printf("iface:        %s\n", *iface)
+	fmt.Printf("hw-addr:      %s\n", hwAddr)
+	fmt.Printf("client-id:    %s\n", cid)
+	fmt.Printf("hostname:     %s\n", *hostname)
+	fmt.Printf("vendor-class: %s\n", *vendorClass)
+	fmt.Printf("promisc:      %v\n", *promisc)
+	fmt.Printf("timeout:      %s  retries: %d\n", *timeout, *retries)
+	fmt.Println()
+
+	if *promisc {
+		if err := setPromisc(*iface, true); err != nil {
+			log.Printf("WARNING: failed to set promisc on %s: %v", *iface, err)
+		} else {
+			fmt.Printf("set %s promisc ON\n\n", *iface)
+		}
+	}
+
+	// Build nclient4 options — identical to spinifex/services/vpcd/dhcp/nclient4.go.
+	opts := []nclient4.ClientOpt{
+		nclient4.WithHWAddr(hwAddr),
+		nclient4.WithTimeout(*timeout),
+		nclient4.WithRetry(*retries),
+	}
+	if *verbose {
+		opts = append(opts, nclient4.WithLogger(nclient4.ShortSummaryLogger{Printfer: log.New(os.Stderr, "[nclient4] ", 0)}))
+	}
+
+	client, err := nclient4.New(*iface, opts...)
+	if err != nil {
+		log.Fatalf("nclient4.New(%s): %v", *iface, err)
+	}
+	defer client.Close()
+
+	// Identity modifiers — identical to identityModifiers() in vpcd dhcp package.
+	var mods []dhcpv4.Modifier
+	if cid != "" {
+		mods = append(mods, dhcpv4.WithOption(dhcpv4.OptClientIdentifier([]byte(cid))))
+	}
+	if *hostname != "" {
+		mods = append(mods, dhcpv4.WithOption(dhcpv4.OptHostName(*hostname)))
+	}
+	if *vendorClass != "" {
+		mods = append(mods, dhcpv4.WithOption(dhcpv4.OptClassIdentifier(*vendorClass)))
+	}
+
+	fmt.Println("Sending DHCP Discover...")
+	start := time.Now()
+	ctx, cancel := context.WithTimeout(context.Background(), *timeout*time.Duration(*retries+1))
+	defer cancel()
+
+	lease, err := client.Request(ctx, mods...)
+	if err != nil {
+		fmt.Printf("\nFAILED after %s: %v\n", time.Since(start).Round(time.Millisecond), err)
+		os.Exit(1)
+	}
+
+	fmt.Printf("SUCCESS in %s\n\n", time.Since(start).Round(time.Millisecond))
+	ack := lease.ACK
+	fmt.Printf("  Your IP:     %s\n", ack.YourIPAddr)
+	fmt.Printf("  Subnet:      %s\n", ack.SubnetMask())
+	fmt.Printf("  Gateway:     %v\n", ack.Router())
+	fmt.Printf("  DNS:         %v\n", ack.DNS())
+	fmt.Printf("  Server ID:   %s\n", ack.ServerIdentifier())
+	fmt.Printf("  Lease time:  %s\n", ack.IPAddressLeaseTime(24*time.Hour))
+	if *verbose {
+		fmt.Printf("\n--- Full ACK ---\n%s\n", ack.Summary())
+	}
+}
+
+// generateMAC is the same deterministic hash used by vpcd's topology.go.
+// It produces a locally-administered MAC (02:00:00:xx:xx:xx) from a resource ID.
+func generateMAC(resourceID string) string {
+	h := uint32(0)
+	for _, c := range resourceID {
+		h = h*31 + uint32(c)
+	}
+	return fmt.Sprintf("02:00:00:%02x:%02x:%02x", (h>>16)&0xff, (h>>8)&0xff, h&0xff)
+}
+
+func ifaceMAC(name string) (net.HardwareAddr, error) {
+	ifc, err := net.InterfaceByName(name)
+	if err != nil {
+		return nil, err
+	}
+	return ifc.HardwareAddr, nil
+}
+
+func setPromisc(iface string, on bool) error {
+	state := "on"
+	if !on {
+		state = "off"
+	}
+	out, err := exec.Command("ip", "link", "set", iface, "promisc", state).CombinedOutput()
+	if err != nil {
+		return fmt.Errorf("%w: %s", err, strings.TrimSpace(string(out)))
+	}
+	return nil
+}

cmd/installer/autoinstall/autoinstall.go

@@ -134,12 +134,21 @@ func buildConfig() (*install.Config, error) {
 		}
 	}
 
+	if os.Getenv("SPINIFEX_GPU_PASSTHROUGH") == "1" {
+		cfg.GPUPassthrough = true
+	}
+
 	role := strings.ToLower(os.Getenv("SPINIFEX_ROLE"))
 	if role == "" {
 		role = "init"
 	}
 	cfg.ClusterRole = role
-	if role == "join" {
+	skipFormation := os.Getenv("SPINIFEX_SKIP_FORMATION") == "1"
+	if skipFormation {
+		cfg.SkipFormation = true
+	}
+
+	if role == "join" && !skipFormation {
 		joinAddr := os.Getenv("SPINIFEX_JOIN_ADDR")
 		if joinAddr == "" {
 			return nil, fmt.Errorf("SPINIFEX_JOIN_ADDR required when SPINIFEX_ROLE=join")

cmd/installer/autoinstall/autoinstall_test.go

@@ -0,0 +1,17 @@
+package autoinstall
+
+import (
+	"os"
+	"testing"
+)
+
+func TestLoadReturnsNilWhenNotAuto(t *testing.T) {
+	os.Unsetenv("SPINIFEX_AUTO")
+	cfg, err := Load()
+	if err != nil {
+		t.Fatalf("unexpected error: %v", err)
+	}
+	if cfg != nil {
+		t.Fatal("expected nil config when SPINIFEX_AUTO is not set")
+	}
+}

cmd/installer/firstboot/firstboot.go

@@ -43,6 +43,19 @@ type Config struct {
 	// the headless path. Passed to `spx admin init --email=<value>` when set;
 	// omitted entirely when empty.
 	Email string
+	// GPUPassthrough enables VFIO GPU passthrough by passing --gpu-passthrough
+	// to `spx admin init`, which writes gpu_passthrough = true in the daemon config.
+	GPUPassthrough bool
+	// InstallCallback, when non-empty, is curled once at the end of firstboot
+	// after the success marker is written. Generic phone-home hook for
+	// provisioning controllers (PXE/MAAS-style flows). Best-effort: non-2xx
+	// and network failures are swallowed so the install is not gated on
+	// controller reachability.
+	InstallCallback string
+	// SkipFormation skips spx admin init/join. Used when a provisioning
+	// controller (e.g. bm-bootstrap.sh) owns cluster formation and will call
+	// spx admin init/join itself with the correct multi-node parameters.
+	SkipFormation bool
 }
 
 // Write drops the firstboot script and systemd unit into root, which should be
@@ -78,6 +91,14 @@ func writeScript(root string, cfg Config) error {
 		setupOVN += fmt.Sprintf(" --encap-ip=%s", cfg.EncapIP)
 	}
 
+	callbackBlock := ""
+	if cfg.InstallCallback != "" {
+		callbackBlock = fmt.Sprintf(
+			"\ncurl -fsS --max-time 10 --retry 2 --retry-delay 2 -o /dev/null %s || true\n",
+			shellEscapeSingle(cfg.InstallCallback),
+		)
+	}
+
 	script := fmt.Sprintf(`#!/bin/bash
 # Spinifex firstboot — runs once after ISO installation. On success, writes
 # /var/lib/spinifex/.firstboot-done and the systemd unit's
@@ -146,6 +167,18 @@ SETUP_STAGES=fixown /usr/local/share/spinifex/setup.sh
 # setup.sh's create_directories — it resolves to /etc/spinifex, so
 # {BaseDir}/config/master.key automatically points at /etc/spinifex/master.key.
 
+# Bootstrap the spinifex user's SSH directory with a generated key pair so
+# ~/.ssh exists and the operator can immediately use ssh-keygen / ssh-copy-id
+# without hitting "No such file or directory" errors documented in the setup guide.
+if [ ! -f /home/spinifex/.ssh/id_ed25519 ]; then
+    mkdir -p /home/spinifex/.ssh
+    chmod 700 /home/spinifex/.ssh
+    ssh-keygen -t ed25519 -f /home/spinifex/.ssh/id_ed25519 -N "" -C "spinifex@$(hostname)"
+    chown -R spinifex:spinifex /home/spinifex/.ssh
+    chmod 600 /home/spinifex/.ssh/id_ed25519
+    chmod 644 /home/spinifex/.ssh/id_ed25519.pub
+fi
+
 # Copy AWS credentials to the spinifex user's home directory.
 # spx admin init runs with HOME=/root (set by the systemd unit), so credentials
 # land in /root/.aws/. Copy them to the spinifex user's home so the operator
@@ -176,7 +209,7 @@ systemctl enable spinifex.target spinifex-banner.service
 # firstboot from the top.
 mkdir -p "$(dirname "$DONE_MARKER")"
 touch "$DONE_MARKER"
-
+%s
 systemctl start spinifex.target
 
 # Wait for the daemon to bring up external networking. When external_mode=pool
@@ -196,13 +229,16 @@ if grep -q 'external_mode.*pool' /etc/spinifex/spinifex.toml 2>/dev/null; then
         echo "[firstboot] warning: br-ext not up after 30s — external networking may be delayed"
     fi
 fi
-`, cfg.Hostname, setupOVN, clusterCmd)
+`, cfg.Hostname, setupOVN, clusterCmd, callbackBlock)
 
 	path := filepath.Join(root, "usr/local/bin/spinifex-firstboot.sh")
 	return os.WriteFile(path, []byte(script), 0o755)
 }
 
 func buildClusterCmd(cfg Config) string {
+	if cfg.SkipFormation {
+		return `echo "[firstboot] cluster formation skipped — provisioning controller will handle it"`
+	}
 	emailFlag := ""
 	if cfg.Email != "" {
 		// shellEscapeSingle keeps the email safe if it ever contains a
@@ -210,11 +246,15 @@ func buildClusterCmd(cfg Config) string {
 		// validator already rejects whitespace and @-chains.
 		emailFlag = " --email=" + shellEscapeSingle(cfg.Email)
 	}
+	gpuFlag := ""
+	if cfg.GPUPassthrough {
+		gpuFlag = " --gpu-passthrough"
+	}
 	switch cfg.ClusterRole {
 	case "join":
 		return fmt.Sprintf("spx admin join --node %s --host %s%s", cfg.Hostname, cfg.JoinAddr, emailFlag)
 	default:
-		return fmt.Sprintf("spx admin init --node %s --nodes 1%s", cfg.Hostname, emailFlag)
+		return fmt.Sprintf("spx admin init --node %s --nodes 1%s%s", cfg.Hostname, emailFlag, gpuFlag)
 	}
 }
 

cmd/installer/firstboot/firstboot_test.go

@@ -0,0 +1,100 @@
+package firstboot
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+func makeRootDirs(t *testing.T, root string) {
+	t.Helper()
+	for _, d := range []string{
+		"usr/local/bin",
+		"etc/systemd/system",
+		"etc/systemd/system/multi-user.target.wants",
+	} {
+		if err := os.MkdirAll(filepath.Join(root, d), 0o755); err != nil {
+			t.Fatalf("mkdir %s: %v", d, err)
+		}
+	}
+}
+
+func TestWriteScriptNoCallbackWhenEmpty(t *testing.T) {
+	root := t.TempDir()
+	makeRootDirs(t, root)
+
+	cfg := Config{Hostname: "test-node", ClusterRole: "init"}
+	if err := Write(root, cfg); err != nil {
+		t.Fatalf("Write: %v", err)
+	}
+
+	script, err := os.ReadFile(filepath.Join(root, "usr/local/bin/spinifex-firstboot.sh"))
+	if err != nil {
+		t.Fatalf("read script: %v", err)
+	}
+	if strings.Contains(string(script), "curl") {
+		t.Error("script should not contain curl when InstallCallback is empty")
+	}
+}
+
+func TestWriteScriptEmbedsCurlWhenCallbackSet(t *testing.T) {
+	root := t.TempDir()
+	makeRootDirs(t, root)
+
+	const callbackURL = "http://192.168.1.12/boot/done?mac=aa:bb:cc:dd:ee:ff"
+	cfg := Config{
+		Hostname:        "test-node",
+		ClusterRole:     "init",
+		InstallCallback: callbackURL,
+	}
+	if err := Write(root, cfg); err != nil {
+		t.Fatalf("Write: %v", err)
+	}
+
+	script, err := os.ReadFile(filepath.Join(root, "usr/local/bin/spinifex-firstboot.sh"))
+	if err != nil {
+		t.Fatalf("read script: %v", err)
+	}
+	content := string(script)
+
+	if !strings.Contains(content, "curl") {
+		t.Error("script missing curl command")
+	}
+	if !strings.Contains(content, callbackURL) {
+		t.Errorf("script missing callback URL %q", callbackURL)
+	}
+}
+
+func TestWriteScriptCallbackAfterDoneMarker(t *testing.T) {
+	root := t.TempDir()
+	makeRootDirs(t, root)
+
+	const callbackURL = "http://192.168.1.12/boot/done?mac=aa:bb:cc:dd:ee:ff"
+	cfg := Config{
+		Hostname:        "node1",
+		ClusterRole:     "init",
+		InstallCallback: callbackURL,
+	}
+	if err := Write(root, cfg); err != nil {
+		t.Fatalf("Write: %v", err)
+	}
+
+	script, err := os.ReadFile(filepath.Join(root, "usr/local/bin/spinifex-firstboot.sh"))
+	if err != nil {
+		t.Fatalf("read script: %v", err)
+	}
+	content := string(script)
+
+	doneIdx := strings.Index(content, "touch \"$DONE_MARKER\"")
+	curlIdx := strings.Index(content, "curl")
+	if doneIdx < 0 {
+		t.Fatal("done marker not found in script")
+	}
+	if curlIdx < 0 {
+		t.Fatal("curl not found in script")
+	}
+	if curlIdx < doneIdx {
+		t.Error("curl must appear after done marker write")
+	}
+}

cmd/installer/install/config.go

@@ -55,6 +55,10 @@ type Config struct {
 	// JoinAddr is the primary node address (host:port) when ClusterRole is "join".
 	JoinAddr string
 
+	// SkipFormation skips spx admin init/join in firstboot; the provisioning
+	// controller (e.g. bm-bootstrap.sh) owns cluster formation instead.
+	SkipFormation bool
+
 	// CA certificate (PEM), optional.
 	HasCACert bool
 	CACert    string
@@ -67,4 +71,9 @@ type Config struct {
 	// Required on interactive installs; may be empty on headless/CI installs
 	// when SPINIFEX_EMAIL is not supplied on the kernel cmdline.
 	Email string
+
+	// GPUPassthrough enables VFIO GPU passthrough on this node.
+	// Set via SPINIFEX_GPU_PASSTHROUGH=1 on headless installs.
+	// Passes --gpu-passthrough to `spx admin init` during firstboot.
+	GPUPassthrough bool
 }

cmd/installer/install/install.go

@@ -23,9 +23,11 @@ import (
 	"io"
 	"log/slog"
 	"net"
+	"net/http"
 	"os"
 	"os/exec"
 	"path/filepath"
+	"slices"
 	"strings"
 	"syscall"
 	"time"
@@ -77,6 +79,7 @@ func Run(cfg *Config) error {
 	}
 
 	slog.Info("installation complete")
+	fireInstallCallback()
 	promptRemoveUSB()
 	return reboot()
 }
@@ -172,6 +175,7 @@ func copyRootfs() error {
 		"--exclude=/etc/openvswitch/",
 		"--exclude=/var/lib/openvswitch/",
 		"--exclude=/var/lib/dhcpcd/",
+		"--exclude=/etc/ssh/ssh_host_*",
 		"--exclude=/lost+found",
 		"--exclude=/boot/efi",
 		"/", mountRoot+"/",
@@ -555,6 +559,11 @@ func installBootloader(disk string) error {
 		}
 		return biosErr
 	}
+	// Copy splash image and unicode font from the ISO (mounted at /cdrom) so the
+	// installed GRUB shows the same branded background as the installer GRUB.
+	// The font must be at /boot/grub/fonts/unicode.pf2 so update-grub finds it
+	// there and emits the same loadfont path as the ISO's grub.cfg — GRUB 2.12
+	// (trixie) needs the font in the boot partition, not just /usr/share/grub/.
 	copySplashImage(mountRoot)
 	copyGrubFont(mountRoot)
 
@@ -644,6 +653,26 @@ func promptRemoveUSB() {
 	}
 }
 
+// fireInstallCallback notifies the boot controller that installation is done
+// so it clears the PXE install flag before the node reboots. Without this,
+// a PXE-first boot order causes the node to reinstall on every reboot until
+// firstboot fires the callback — which it can never do if it never runs.
+// No-op when SPINIFEX_INSTALL_CALLBACK is not set (ISO/USB installs).
+func fireInstallCallback() {
+	url := strings.TrimSpace(os.Getenv("SPINIFEX_INSTALL_CALLBACK"))
+	if url == "" {
+		return
+	}
+	client := &http.Client{Timeout: 10 * time.Second}
+	resp, err := client.Get(url) //nolint:noctx // installer has no context; best-effort fire-and-forget
+	if err != nil {
+		slog.Warn("fireInstallCallback: request failed", "url", url, "err", err)
+		return
+	}
+	resp.Body.Close()
+	slog.Info("fireInstallCallback: notified boot controller", "url", url, "status", resp.StatusCode)
+}
+
 func reboot() error {
 	// sync filesystems before reboot so nothing is lost.
 	_ = run("sync")
@@ -662,11 +691,13 @@ func (c *Config) toFirstbootConfig() firstboot.Config {
 		encapIP = c.LANAddress
 	}
 	return firstboot.Config{
-		Hostname:    c.Hostname,
-		EncapIP:     encapIP,
-		ClusterRole: c.ClusterRole,
-		JoinAddr:    c.JoinAddr,
-		Email:       c.Email,
+		Hostname:        c.Hostname,
+		EncapIP:         encapIP,
+		ClusterRole:     c.ClusterRole,
+		JoinAddr:        c.JoinAddr,
+		Email:           c.Email,
+		InstallCallback: strings.TrimSpace(os.Getenv("SPINIFEX_INSTALL_CALLBACK")),
+		SkipFormation:   c.SkipFormation,
 	}
 }
 
@@ -712,9 +743,46 @@ func partitionPaths(disk string) (efi, root string) {
 	return disk + "2", disk + "3"
 }
 
-// copySplashImage copies the GRUB splash (embedded in the squashfs at build time by
-// inject-bins.sh) into the installed system so the post-install GRUB shows the same
-// branded background as the installer GRUB. Non-fatal — missing source is logged and skipped.
+// copyGrubFont copies the unicode.pf2 GRUB font into the installed system's
+// /boot/grub/fonts/ directory. This ensures update-grub finds the font at
+// /boot/grub/fonts/unicode.pf2 — the same path the ISO's grub.cfg uses —
+// so the generated grub.cfg enables gfxterm and the background image.
+// Without this, grub-mkconfig falls back to /usr/share/grub/unicode.pf2,
+// a path that GRUB 2.12 (trixie) may fail to resolve at boot time.
+// Non-fatal — a missing source is logged and skipped.
+func copyGrubFont(root string) {
+	candidates := []string{
+		"/cdrom/boot/grub/fonts/unicode.pf2", // ISO tree (preferred)
+		"/usr/share/grub/unicode.pf2",        // live system grub-common fallback
+	}
+	for _, src := range candidates {
+		in, err := os.Open(src)
+		if err != nil {
+			continue
+		}
+		defer in.Close()
+		dstDir := filepath.Join(root, "boot/grub/fonts")
+		if err := os.MkdirAll(dstDir, 0o755); err != nil {
+			slog.Warn("copyGrubFont: cannot create fonts dir", "err", err)
+			return
+		}
+		out, err := os.OpenFile(filepath.Join(dstDir, "unicode.pf2"), os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644)
+		if err != nil {
+			slog.Warn("copyGrubFont: cannot open destination", "err", err)
+			return
+		}
+		defer out.Close()
+		if _, err := io.Copy(out, in); err != nil {
+			slog.Warn("copyGrubFont: copy failed", "err", err)
+		}
+		return
+	}
+	slog.Warn("copyGrubFont: no unicode.pf2 found, splash may not display")
+}
+
+// copySplashImage copies the GRUB splash from the live ISO (/cdrom/boot/grub/splash.png)
+// into the installed system so the post-install GRUB shows the same branded background
+// as the installer. Non-fatal — a missing or unreadable source is logged and skipped.
 func copySplashImage(root string) {
 	const src = "/usr/share/spinifex/grub-splash.png"
 	in, err := os.Open(src)
@@ -740,34 +808,6 @@ func copySplashImage(root string) {
 	}
 }
 
-// copyGrubFont copies the unicode font into the installed system's
-// /boot/grub/fonts/ so the loadfont path in 05_spinifex resolves at boot.
-// grub-install does not copy fonts; we mirror what build-iso.sh does.
-func copyGrubFont(root string) {
-	const src = "/usr/share/grub/unicode.pf2"
-	in, err := os.Open(src)
-	if err != nil {
-		slog.Warn("copyGrubFont: font not found, graphical GRUB may not work", "path", src)
-		return
-	}
-	defer in.Close()
-
-	dstDir := filepath.Join(root, "boot/grub/fonts")
-	if err := os.MkdirAll(dstDir, 0o755); err != nil {
-		slog.Warn("copyGrubFont: cannot create fonts dir", "err", err)
-		return
-	}
-	out, err := os.OpenFile(filepath.Join(dstDir, "unicode.pf2"), os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644)
-	if err != nil {
-		slog.Warn("copyGrubFont: cannot open destination", "err", err)
-		return
-	}
-	defer out.Close()
-	if _, err := io.Copy(out, in); err != nil {
-		slog.Warn("copyGrubFont: copy failed", "err", err)
-	}
-}
-
 // maskSystemdUnit creates a symlink to /dev/null for the given unit, which is
 // the standard way to permanently disable a unit so systemd will never start it.
 func maskSystemdUnit(root, unit string) error {
@@ -811,8 +851,8 @@ func bindChrootMounts() error {
 // unbindChrootMounts unmounts the virtual filesystems in reverse order.
 // Errors are logged but not returned — this is best-effort cleanup.
 func unbindChrootMounts() {
-	for i := len(chrootMountPaths) - 1; i >= 0; i-- {
-		_ = run("umount", filepath.Join(mountRoot, chrootMountPaths[i]))
+	for _, v := range slices.Backward(chrootMountPaths) {
+		_ = run("umount", filepath.Join(mountRoot, v))
 	}
 }
 

cmd/installer/install/install_test.go

@@ -0,0 +1,27 @@
+package install
+
+import (
+	"os"
+	"testing"
+)
+
+func TestToFirstbootConfigIncludesInstallCallback(t *testing.T) {
+	const url = "http://192.168.1.12/boot/done?mac=aa:bb:cc:dd:ee:ff"
+	t.Setenv("SPINIFEX_INSTALL_CALLBACK", url)
+
+	cfg := &Config{Hostname: "node1", ClusterRole: "init"}
+	fb := cfg.toFirstbootConfig()
+	if fb.InstallCallback != url {
+		t.Errorf("InstallCallback = %q, want %q", fb.InstallCallback, url)
+	}
+}
+
+func TestToFirstbootConfigEmptyCallbackWhenEnvUnset(t *testing.T) {
+	os.Unsetenv("SPINIFEX_INSTALL_CALLBACK")
+
+	cfg := &Config{Hostname: "node1", ClusterRole: "init"}
+	fb := cfg.toFirstbootConfig()
+	if fb.InstallCallback != "" {
+		t.Errorf("InstallCallback = %q, want empty", fb.InstallCallback)
+	}
+}

cmd/installer/main.go

@@ -24,6 +24,7 @@ import (
 	"github.com/mulgadc/spinifex/cmd/installer/autoinstall"
 	"github.com/mulgadc/spinifex/cmd/installer/install"
 	"github.com/mulgadc/spinifex/cmd/installer/ui"
+	_ "github.com/mulgadc/spinifex/internal/fipsboot"
 )
 
 func main() {

cmd/lb-agent/main.go

@@ -19,6 +19,8 @@ import (
 	"syscall"
 
 	"github.com/mulgadc/spinifex/spinifex/lbagent"
+
+	_ "github.com/mulgadc/spinifex/internal/fipsboot"
 )
 
 func main() {

cmd/spinifex/cmd/admin.go

@@ -17,6 +17,7 @@ along with this program.  If not, see <https://www.gnu.org/licenses/>.
 package cmd
 
 import (
+	"bufio"
 	"bytes"
 	"context"
 	"crypto/tls"
@@ -45,8 +46,10 @@ import (
 	"github.com/mulgadc/spinifex/spinifex/admin"
 	"github.com/mulgadc/spinifex/spinifex/config"
 	"github.com/mulgadc/spinifex/spinifex/formation"
+	"github.com/mulgadc/spinifex/spinifex/gpu"
 	handlers_ec2_vpc "github.com/mulgadc/spinifex/spinifex/handlers/ec2/vpc"
 	handlers_iam "github.com/mulgadc/spinifex/spinifex/handlers/iam"
+	"github.com/mulgadc/spinifex/spinifex/objectstore"
 	"github.com/mulgadc/spinifex/spinifex/utils"
 	"github.com/mulgadc/viperblock/viperblock"
 	"github.com/mulgadc/viperblock/viperblock/backends/s3"
@@ -83,6 +86,14 @@ var supportedPlatforms = map[string]bool{
 	"Windows":    true,
 }
 
+// Mirrors the gateway RegisterImage allowlist so admin imports can't write an
+// AMI with a boot mode that RegisterImage would reject.
+var supportedBootModes = map[string]bool{
+	"bios":           true,
+	"uefi":           true,
+	"uefi-preferred": true,
+}
+
 var adminCmd = &cobra.Command{
 	Use:   "admin",
 	Short: "Administrative commands for Spinifex platform management",
@@ -142,6 +153,19 @@ var imagesListCmd = &cobra.Command{
 	Run:   runimagesListCmd,
 }
 
+var imagesRemoveCmd = &cobra.Command{
+	Use:   "remove",
+	Short: "Remove an admin-imported system AMI",
+	Long: `Delete an AMI imported via 'spx admin images import', including its
+backing block storage and snapshot artefacts. Only operates on AMIs with a
+non-account owner (e.g. "system"); account-owned AMIs must be removed via
+'aws ec2 deregister-image' followed by 'aws ec2 delete-snapshot'.
+
+Refuses to delete an AMI that has dependent volumes or copied snapshots/AMIs
+unless --force is passed. Prompts for confirmation unless --yes is passed.`,
+	Run: runimagesRemoveCmd,
+}
+
 var accountCmd = &cobra.Command{
 	Use:   "account",
 	Short: "Manage Spinifex accounts",
@@ -199,13 +223,13 @@ spx admin images list
 - fetches from remote endpoint for common/trusted images to bootstrap environment, or baked in from compile.
 
 // If --name specified, download
-spx admin images import --name debian-12-x86_64
+spx admin images import --name debian-13-x86_64
 
 // List available images
 spx admin images list
 
 // Manually import a path
-spx admin images import --file /path/to/image --distro debian --version 12 --arch x86_64
+spx admin images import --file /path/to/image --distro debian --version 13 --arch x86_64
 
 -> x <-
 */
@@ -224,6 +248,7 @@ func init() {
 	adminCmd.AddCommand(imagesCmd)
 	imagesCmd.AddCommand(imagesImportCmd)
 	imagesCmd.AddCommand(imagesListCmd)
+	imagesCmd.AddCommand(imagesRemoveCmd)
 
 	adminCmd.AddCommand(accountCmd)
 	accountCmd.AddCommand(accountCreateCmd)
@@ -274,6 +299,7 @@ func init() {
 	adminInitCmd.Flags().String("gateway-ip", "", "OVN gateway router's external IP for SNAT (default: pool range_start for pool mode, required for nat mode without DHCP)")
 	adminInitCmd.Flags().Int("external-prefix-len", 24, "External pool subnet prefix length (auto-detected)")
 	adminInitCmd.Flags().Bool("no-external", false, "Disable external networking (overlay-only, no internet for VMs)")
+	adminInitCmd.Flags().Bool("gpu-passthrough", false, "Enable VFIO GPU passthrough (sets gpu_passthrough = true in daemon config)")
 
 	// Flags for admin join
 	adminJoinCmd.Flags().String("region", "ap-southeast-2", "Region for this node")
@@ -302,9 +328,17 @@ func init() {
 	imagesImportCmd.Flags().String("version", "", "Specified distro version (e.g 12)")
 	imagesImportCmd.Flags().String("arch", "", "Specified distro arch (e.g aarch64, arm64, x86_64)")
 	imagesImportCmd.Flags().String("platform", "Linux/UNIX", "Specified platform (e.g Linux/UNIX, Windows)")
+	imagesImportCmd.Flags().String("boot-mode", "", "Boot mode for the imported AMI (bios|uefi|uefi-preferred). Required with --file. Overrides the catalog value when used with --name.")
 	imagesImportCmd.Flags().StringSlice("tag", nil, "Tag to apply to the imported AMI as key=value (repeatable; e.g. --tag spinifex:managed-by=elbv2)")
 	imagesImportCmd.Flags().Bool("force", false, "Force command execution (overwrites existing files)")
 	imagesImportCmd.Flags().Bool("skip-verify", false, "Skip catalog-image checksum verification (INSECURE; operator assumes integrity responsibility)")
+
+	imagesRemoveCmd.Flags().String("image-id", "", "AMI ID to remove (required)")
+	imagesRemoveCmd.Flags().Bool("force", false, "Bypass dependency, ownership and config-corrupt checks (salvage mode)")
+	imagesRemoveCmd.Flags().Bool("yes", false, "Skip interactive confirmation prompt")
+	if err := imagesRemoveCmd.MarkFlagRequired("image-id"); err != nil {
+		panic(err)
+	}
 }
 
 func runimagesImportCmd(cmd *cobra.Command, args []string) {
@@ -374,6 +408,22 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 		}
 	}
 
+	// --file imports have no catalog metadata to inherit from, so the operator
+	// must declare the boot mode explicitly — guessing would silently produce
+	// a BIOS AMI from a UEFI-only image (or vice versa) and fail at launch.
+	// --name imports inherit from the catalog; the flag overrides when set.
+	bootModeFlag, _ := cmd.Flags().GetString("boot-mode")
+	if bootModeFlag != "" {
+		if !supportedBootModes[bootModeFlag] {
+			fmt.Fprintf(os.Stderr, "Unsupported --boot-mode %q (expected bios|uefi|uefi-preferred)\n", bootModeFlag)
+			os.Exit(1)
+		}
+		image.BootMode = bootModeFlag
+	} else if imageName == "" {
+		fmt.Fprintf(os.Stderr, "--boot-mode is required when importing via --file (expected bios|uefi|uefi-preferred)\n")
+		os.Exit(1)
+	}
+
 	// --tag k=v (repeatable) merges user-supplied tags into the AMI. Overrides
 	// catalog tags on key collision so operators can re-tag a known image.
 	tagFlags, _ := cmd.Flags().GetStringSlice("tag")
@@ -494,6 +544,9 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 	manifest.AMIMetadata.Virtualization = "hvm"
 	manifest.AMIMetadata.ImageOwnerAlias = "system"
 	manifest.AMIMetadata.VolumeSizeGiB = utils.SafeInt64ToUint64(imageStat.Size() / 1024 / 1024 / 1024)
+	manifest.AMIMetadata.BootMode = image.BootMode
+	manifest.AMIMetadata.Distro = image.Distro
+	manifest.AMIMetadata.DistroFamily = utils.DistroFamily(image.Distro)
 
 	// Copy catalog-provided tags (e.g. spinifex:managed-by for system AMIs)
 	// onto the imported AMI so the UI can filter them out.
@@ -572,6 +625,136 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 	fmt.Printf("✅ Image import complete. Image-ID (AMI): %s\n", volumeId)
 }
 
+func runimagesRemoveCmd(cmd *cobra.Command, args []string) {
+	imageID, _ := cmd.Flags().GetString("image-id")
+	force, _ := cmd.Flags().GetBool("force")
+	yes, _ := cmd.Flags().GetBool("yes")
+
+	cfgFile, _ := cmd.Flags().GetString("config")
+	if cfgFile == "" {
+		cfgFile = DefaultConfigFile()
+	}
+
+	appConfig, err := config.LoadConfig(cfgFile)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "Error loading config file:", err)
+		os.Exit(1)
+	}
+
+	node := appConfig.Nodes[appConfig.Node]
+	store := objectstore.NewS3ObjectStoreFromConfig(
+		node.Predastore.Host,
+		node.Predastore.Region,
+		node.Predastore.AccessKey,
+		node.Predastore.SecretKey,
+	)
+	bucket := node.Predastore.Bucket
+
+	preview, err := admin.PreviewRemoveSystemImage(store, bucket, imageID)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "Failed to inspect AMI:", err)
+		os.Exit(1)
+	}
+
+	// Print metadata block.
+	fmt.Println("About to remove system AMI:")
+	fmt.Println()
+	fmt.Printf("  Image ID:        %s\n", preview.ImageID)
+	switch {
+	case !preview.ConfigPresent && !preview.ConfigCorrupt:
+		fmt.Println("  Name:            <unknown — config.json missing>")
+		fmt.Println("  Owner:           <unknown>")
+	case preview.ConfigCorrupt:
+		fmt.Println("  Name:            <unknown — config.json corrupt>")
+		fmt.Println("  Owner:           <unknown>")
+	default:
+		fmt.Printf("  Name:            %s\n", preview.Name)
+		fmt.Printf("  Owner:           %s\n", preview.Owner)
+		if !preview.Created.IsZero() {
+			fmt.Printf("  Created:         %s\n", preview.Created.UTC().Format("2006-01-02T15:04:05Z"))
+		}
+	}
+	fmt.Printf("  Backing storage: %s/      (%d objects, %s)\n",
+		preview.ImageID, preview.AMIObjectCount, utils.HumanBytes(utils.SafeInt64ToUint64(preview.AMIBytesTotal)))
+	fmt.Printf("                   %s/ (%d objects, %s)\n",
+		admin.SnapPrefix(preview.ImageID), preview.SnapObjectCount, utils.HumanBytes(utils.SafeInt64ToUint64(preview.SnapBytesTotal)))
+	fmt.Println()
+
+	// Account-owned guard before salvage / dependents — the AWS-flow hint is
+	// the most useful thing to surface for this kind of mistake.
+	if preview.ConfigPresent && !preview.IsSystemOwned && !force {
+		fmt.Fprintf(os.Stderr,
+			"Refusing to remove: %s is account-owned (%s).\n"+
+				"Use 'aws ec2 deregister-image --image-id %s' followed by 'aws ec2 delete-snapshot ...'.\n",
+			preview.ImageID, preview.Owner, preview.ImageID)
+		os.Exit(1)
+	}
+
+	if !preview.ConfigPresent && !force {
+		fmt.Fprintln(os.Stderr, "AMI config.json missing or corrupt; re-run with --force to salvage backing blocks.")
+		os.Exit(1)
+	}
+
+	if !preview.Dependents.Empty() && !force {
+		fmt.Fprintln(os.Stderr, "Refusing to remove: dependent resources reference this image.")
+		printDependents(os.Stderr, preview.Dependents)
+		fmt.Fprintln(os.Stderr, "Remove them first (e.g. 'aws ec2 terminate-instances', 'aws ec2 delete-snapshot', 'aws ec2 deregister-image') or re-run with --force.")
+		os.Exit(1)
+	}
+
+	if force && (!preview.ConfigPresent || !preview.Dependents.Empty()) {
+		fmt.Println("⚠️  --force: skipping dependency check and ownership check.")
+		if !preview.Dependents.Empty() {
+			printDependents(os.Stdout, preview.Dependents)
+		}
+		fmt.Println()
+	}
+
+	fmt.Println("This is permanent and cannot be undone.")
+	if !yes {
+		fmt.Print("Type 'yes' to proceed: ")
+		reader := bufio.NewReader(os.Stdin)
+		answer, _ := reader.ReadString('\n')
+		if strings.TrimSpace(answer) != "yes" {
+			fmt.Println("Aborted.")
+			return
+		}
+	}
+
+	res, err := admin.RemoveSystemImage(store, bucket, admin.RemoveImageOpts{
+		ImageID: imageID,
+		Force:   force,
+	})
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "Remove failed:", err)
+		os.Exit(1)
+	}
+
+	fmt.Printf("✅ Removed AMI %s (freed %s across %d objects).\n",
+		imageID, utils.HumanBytes(utils.SafeInt64ToUint64(res.BytesFreed)), res.ObjectsDeleted)
+}
+
+func printDependents(w io.Writer, d admin.Dependents) {
+	if len(d.Volumes) > 0 {
+		fmt.Fprintf(w, "  Volumes (%d):\n", len(d.Volumes))
+		for _, v := range d.Volumes {
+			fmt.Fprintf(w, "    - %s\n", v)
+		}
+	}
+	if len(d.Snapshots) > 0 {
+		fmt.Fprintf(w, "  Snapshots (%d):\n", len(d.Snapshots))
+		for _, s := range d.Snapshots {
+			fmt.Fprintf(w, "    - %s\n", s)
+		}
+	}
+	if len(d.AMIs) > 0 {
+		fmt.Fprintf(w, "  AMIs (%d):\n", len(d.AMIs))
+		for _, a := range d.AMIs {
+			fmt.Fprintf(w, "    - %s\n", a)
+		}
+	}
+}
+
 // List remote images available
 func runimagesListCmd(cmd *cobra.Command, args []string) {
 	//fmt.Println(availableImages)
@@ -660,6 +843,7 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	externalPrefixLen, _ := cmd.Flags().GetInt("external-prefix-len")
 	gatewayIP, _ := cmd.Flags().GetString("gateway-ip")
 	noExternal, _ := cmd.Flags().GetBool("no-external")
+	gpuPassthrough, _ := cmd.Flags().GetBool("gpu-passthrough")
 
 	// Fire telemetry in background (completes during init work, waited at end)
 	noTelemetry, _ := cmd.Flags().GetBool("no-telemetry")
@@ -901,6 +1085,16 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	fmt.Printf("   Bootstrap: %s\n", filepath.Join(bootstrapDir, "bootstrap.json"))
 	fmt.Printf("   System creds: %s\n", filepath.Join(configDir, "system-credentials.json"))
 
+	// Predastore encryption key is per-node and never transmitted; generate
+	// it locally now so the service has it on first start.
+	predastoreKeyPath, err := writePredastoreEncryptionKey(configDir)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error generating predastore encryption key: %v\n", err)
+		os.Exit(1)
+	}
+	fmt.Println("\n🔐 Generated predastore encryption key (per-node, never transmitted)")
+	fmt.Printf("   Key: %s\n", predastoreKeyPath)
+
 	fmt.Printf("\n🔑 Generated admin credentials (save these — they won't be shown again):\n")
 	fmt.Printf("   Access Key:  %s\n", bootstrapResult.AdminAccessKey)
 	fmt.Printf("   Secret Key:  %s\n", bootstrapResult.AdminSecretKey)
@@ -980,8 +1174,8 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	var predastoreNodeID int
 	if predastoreNodesStr != "" {
 		ips := strings.Split(predastoreNodesStr, ",")
-		if len(ips) < 3 {
-			fmt.Fprintf(os.Stderr, "❌ Error: --predastore-nodes requires at least 3 IPs, got %d\n", len(ips))
+		if len(ips) < 2 {
+			fmt.Fprintf(os.Stderr, "❌ Error: --predastore-nodes requires at least 2 IPs, got %d\n", len(ips))
 			os.Exit(1)
 		}
 
@@ -1072,6 +1266,8 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 		BootstrapIgwId:      bootstrapIgwId,
 		BootstrapCidr:       handlers_ec2_vpc.DefaultVPCCidr,
 		BootstrapSubnetCidr: handlers_ec2_vpc.DefaultSubnetCidr,
+
+		GPUPassthrough: gpuPassthrough,
 	}
 
 	// Print external networking summary
@@ -1098,6 +1294,20 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 
 	finalizeNodeSetup(spxRoot, certPath, bootstrapResult.AdminAccessKey, bootstrapResult.AdminSecretKey, region, bindIP, advertiseIP)
 
+	// Write node.conf so spx admin banner works on source installs (not just ISO).
+	nodeHostname, _ := os.Hostname()
+	if nodeHostname == "" {
+		nodeHostname = node
+	}
+	nodeConfPath := filepath.Join(configDir, "node.conf")
+	if err := writeNodeConf(nodeConfPath, map[string]string{
+		"MANAGEMENT_IP":    advertiseIP,
+		"MANAGEMENT_IFACE": "br-wan",
+		"NODE_HOSTNAME":    nodeHostname,
+	}); err != nil {
+		fmt.Fprintf(os.Stderr, "⚠️  Warning: could not write %s: %v\n", nodeConfPath, err)
+	}
+
 	// Print success message
 	fmt.Println("\n🎉 Spinifex initialization complete!")
 	fmt.Println()
@@ -1153,6 +1363,17 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 	fmt.Println("\n🔐 Generated IAM master key")
 	fmt.Printf("   Bootstrap: %s\n", filepath.Join(bootstrapDir, "bootstrap.json"))
 
+	// Predastore encryption key is per-node and never distributed via the
+	// formation server. Generate only the leader's own key here; each
+	// joiner generates its own during `spx admin join`.
+	predastoreKeyPath, err := writePredastoreEncryptionKey(configDir)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "❌ Error generating predastore encryption key: %v\n", err)
+		os.Exit(1)
+	}
+	fmt.Println("\n🔐 Generated predastore encryption key (per-node, never transmitted)")
+	fmt.Printf("   Key: %s\n", predastoreKeyPath)
+
 	// Read CA cert/key for distribution to joining nodes
 	caCertData, err := os.ReadFile(filepath.Join(configDir, "ca.pem"))
 	if err != nil {
@@ -1243,7 +1464,7 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 
 	// Generate multi-node predastore config
 	var predastoreNodeID int
-	hasPredastoreConfig := len(predastoreNodes) >= 3
+	hasPredastoreConfig := len(predastoreNodes) >= 2
 	if hasPredastoreConfig {
 		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, accessKey, secretKey, region, natsToken, configDir, bindIP)
 		if err != nil {
@@ -1532,7 +1753,7 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 		}
 
 		fmt.Printf("   Waiting for cluster formation... (%d/%d nodes joined)\n", statusResp.Joined, statusResp.Expected)
-		time.Sleep(2 * time.Second)
+		time.Sleep(500 * time.Millisecond)
 	}
 
 	fmt.Printf("✅ Cluster formation complete! (%d nodes)\n\n", statusResp.Expected)
@@ -1623,6 +1844,16 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 	fmt.Println("✅ IAM master key received from leader")
 	fmt.Printf("✅ Bootstrap file written: %s\n", filepath.Join(bootstrapDir, "bootstrap.json"))
 
+	// Predastore encryption key is per-node: generate locally rather than
+	// receiving from the leader. Each node only opens fragments it sealed
+	// itself, so there is no cluster-wide predastore key to share.
+	predastoreKeyPath, err := writePredastoreEncryptionKey(configDir)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "❌ Error generating predastore encryption key: %v\n", err)
+		os.Exit(1)
+	}
+	fmt.Printf("✅ Predastore encryption key generated: %s\n", predastoreKeyPath)
+
 	// Generate server cert signed by CA with this node's bind IP
 	if err := admin.GenerateServerCertOnly(configDir, bindIP); err != nil {
 		fmt.Fprintf(os.Stderr, "Error generating server certificate: %v\n", err)
@@ -1646,7 +1877,7 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 
 	// Generate multi-node predastore config
 	var predastoreNodeID int
-	hasPredastoreConfig := len(predastoreNodes) >= 3
+	hasPredastoreConfig := len(predastoreNodes) >= 2
 
 	if hasPredastoreConfig {
 		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, creds.AccessKey, creds.SecretKey, creds.Region, creds.NatsToken, configDir, bindIP)
@@ -2094,6 +2325,27 @@ func writeBootstrapFiles(configDir, bootstrapDir string, masterKey []byte, acces
 	}, nil
 }
 
+// writePredastoreEncryptionKey generates a fresh 32-byte AES-256 master key
+// for this node's predastore data dir and writes it to
+// <configDir>/predastore/encryption.key with mode 0600. The key is per-node:
+// every node generates its own at init/join time and never transmits it,
+// because predastore only opens fragments on the node that sealed them.
+func writePredastoreEncryptionKey(configDir string) (string, error) {
+	predastoreDir := filepath.Join(configDir, "predastore")
+	if err := os.MkdirAll(predastoreDir, 0750); err != nil {
+		return "", fmt.Errorf("create predastore config dir: %w", err)
+	}
+	key, err := handlers_iam.GenerateMasterKey()
+	if err != nil {
+		return "", fmt.Errorf("generate predastore encryption key: %w", err)
+	}
+	keyPath := filepath.Join(predastoreDir, "encryption.key")
+	if err := handlers_iam.SaveMasterKey(keyPath, key); err != nil {
+		return "", fmt.Errorf("save predastore encryption key: %w", err)
+	}
+	return keyPath, nil
+}
+
 // writeSystemCredentials writes the system access key to a plaintext JSON file.
 // The daemon reads this at startup to inject credentials into ALB VM cloud-init
 // for SigV4-authenticated communication with the AWS gateway.
@@ -2304,6 +2556,12 @@ func runAdminBanner(cmd *cobra.Command, _ []string) {
 	}
 
 	// Resolve current IP from the management interface at runtime.
+	// If node.conf is absent or has no MANAGEMENT_IFACE (source installs before
+	// the first spx admin init writes node.conf), fall back to br-wan which
+	// setup-ovn.sh always creates as the management bridge.
+	if iface == "" {
+		iface = "br-wan"
+	}
 	currentIP := resolveIfaceIP(iface)
 	if currentIP == "" {
 		currentIP = recordedIP // fall back to value recorded at install time
@@ -2331,17 +2589,17 @@ func runAdminBanner(cmd *cobra.Command, _ []string) {
 	}
 
 	banner := fmt.Sprintf(`
-  +----------------------------------------------------+
-  |         Spinifex  —  Mulga Defense Corporation     |
-  +----------------------------------------------------+
-  |  Node:      %-39s|
-  |  Login:     %-39s|
-  |  Dashboard: %-39s|
-  |  API:       %-39s|
-  |  SSH:       %-39s|
-  +----------------------------------------------------+
-  |  AWS credentials:  cat ~/.aws/credentials          |
-  +----------------------------------------------------+
+  +--------------------------------------------------------------+
+  |          Spinifex  —  Mulga Defense Corporation              |
+  +--------------------------------------------------------------+
+  |  Node:      %-49s|
+  |  Login:     %-49s|
+  |  Dashboard: %-49s|
+  |  API:       %-49s|
+  |  SSH:       %-49s|
+  +--------------------------------------------------------------+
+  |  AWS credentials:  cat ~/.aws/credentials                    |
+  +--------------------------------------------------------------+
 
 `,
 		hostname,
@@ -2351,6 +2609,8 @@ func runAdminBanner(cmd *cobra.Command, _ []string) {
 		"spinifex@"+currentIP,
 	)
 
+	banner += gpuBannerSection()
+
 	// Write to /etc/issue — displayed on the console before the login prompt.
 	// Overwrite entirely; this is a purpose-built appliance so we own this file.
 	if err := os.WriteFile("/etc/issue", []byte(banner), 0o644); err != nil {
@@ -2390,6 +2650,81 @@ func appendBannerToMotd(banner string) error {
 	return os.WriteFile(motdPath, []byte(base+sentinel+banner), 0o644)
 }
 
+// gpuBannerSection returns an optional banner box section describing GPU state.
+// Returns "" when no GPU hardware is detected. Safe to call at boot before the
+// daemon starts — all checks are sysfs/file reads, no NATS required.
+func gpuBannerSection() string {
+	devices, err := gpu.Discover()
+	if err != nil || len(devices) == 0 {
+		return ""
+	}
+
+	iommuEntries, _ := os.ReadDir("/sys/kernel/iommu_groups/")
+	iommuActive := len(iommuEntries) > 0
+
+	_, vfioErr := os.Stat("/sys/module/vfio_pci")
+	vfioPresent := vfioErr == nil
+
+	passthroughEnabled := false
+	cfgPath := DefaultConfigFile()
+	if cfg, err := config.LoadConfig(cfgPath); err == nil {
+		if nodeCfg, ok := cfg.Nodes[cfg.Node]; ok {
+			passthroughEnabled = nodeCfg.Daemon.GPUPassthrough
+		}
+	}
+
+	models := gpuModelSummary(devices)
+
+	var statusLine, hintLine string
+	switch {
+	case passthroughEnabled:
+		statusLine = "Passthrough enabled"
+	case iommuActive && vfioPresent:
+		statusLine = "Ready to enable"
+		hintLine = "sudo spx admin gpu enable"
+	default:
+		statusLine = "Setup required"
+		hintLine = "sudo spx admin gpu setup"
+	}
+
+	const (
+		sep    = "  +--------------------------------------------------------------+\n"
+		maxVal = 55
+	)
+	if len([]rune(models)) > maxVal {
+		models = string([]rune(models)[:maxVal-3]) + "..."
+	}
+
+	section := sep +
+		fmt.Sprintf("  |  GPU: %-55s|\n", models) +
+		fmt.Sprintf("  |       %-55s|\n", statusLine)
+	if hintLine != "" {
+		section += fmt.Sprintf("  |       %-55s|\n", hintLine)
+	}
+	section += sep + "\n"
+	return section
+}
+
+func gpuModelSummary(devices []gpu.GPUDevice) string {
+	counts := make(map[string]int)
+	var order []string
+	for _, d := range devices {
+		if counts[d.Model] == 0 {
+			order = append(order, d.Model)
+		}
+		counts[d.Model]++
+	}
+	var parts []string
+	for _, m := range order {
+		if n := counts[m]; n > 1 {
+			parts = append(parts, fmt.Sprintf("%dx %s", n, m))
+		} else {
+			parts = append(parts, m)
+		}
+	}
+	return strings.Join(parts, ", ")
+}
+
 // parseNodeConf reads a KEY=VALUE shell-format file and returns a map.
 // Lines starting with # and blank lines are ignored.
 func parseNodeConf(path string) map[string]string {

cmd/spinifex/cmd/admin_gpu.go

@@ -0,0 +1,486 @@
+package cmd
+
+import (
+	"encoding/json"
+	"fmt"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"regexp"
+	"strings"
+	"time"
+
+	"github.com/mulgadc/spinifex/spinifex/admin"
+	"github.com/mulgadc/spinifex/spinifex/config"
+	"github.com/mulgadc/spinifex/spinifex/gpu"
+	"github.com/mulgadc/spinifex/spinifex/types"
+	"github.com/spf13/cobra"
+	"github.com/spf13/viper"
+)
+
+var adminGpuCmd = &cobra.Command{
+	Use:   "gpu",
+	Short: "Manage GPU passthrough for a node",
+}
+
+var adminGpuStatusCmd = &cobra.Command{
+	Use:   "status",
+	Short: "Show GPU hardware and passthrough state",
+	Run:   runAdminGpuStatus,
+}
+
+var adminGpuEnableCmd = &cobra.Command{
+	Use:   "enable",
+	Short: "Enable GPU passthrough on a node",
+	Run:   runAdminGpuEnable,
+}
+
+var adminGpuDisableCmd = &cobra.Command{
+	Use:   "disable",
+	Short: "Disable GPU passthrough on a node (blocked while GPU instances are running)",
+	Run:   runAdminGpuDisable,
+}
+
+var adminGpuSetupCmd = &cobra.Command{
+	Use:   "setup",
+	Short: "Configure this host for GPU passthrough (IOMMU, vfio-pci, nouveau blacklist)",
+	Long: `Idempotent host setup for GPU passthrough. Run once before a reboot to configure
+GRUB, vfio-pci early binding, and the nouveau blacklist; then run again after the
+reboot to verify bindings and enable GPU passthrough in the daemon.
+
+Must be run as root.`,
+	Run: runAdminGpuSetup,
+}
+
+func init() {
+	adminCmd.AddCommand(adminGpuCmd)
+	adminGpuCmd.AddCommand(adminGpuStatusCmd)
+	adminGpuCmd.AddCommand(adminGpuEnableCmd)
+	adminGpuCmd.AddCommand(adminGpuDisableCmd)
+	adminGpuCmd.AddCommand(adminGpuSetupCmd)
+
+	adminGpuStatusCmd.Flags().String("node", "", "Target node name (default: local node)")
+	// enable, disable, and setup write to the local spinifex.toml and signal the
+	// local daemon — they must be run directly on the target host.
+}
+
+// gpuNodeStatus queries NATS and returns the NodeStatusResponse for the target node.
+func gpuNodeStatus(targetNode string) (*types.NodeStatusResponse, error) {
+	cfg, nc, err := loadConfigAndConnect()
+	if err != nil {
+		return nil, err
+	}
+	defer nc.Close()
+
+	if targetNode == "" {
+		targetNode = cfg.Node
+	}
+
+	responses, err := collectResponses(nc, "spinifex.node.status", 3*time.Second)
+	if err != nil {
+		return nil, fmt.Errorf("collect node status: %w", err)
+	}
+	for _, raw := range responses {
+		var resp types.NodeStatusResponse
+		if err := json.Unmarshal(raw, &resp); err != nil {
+			continue
+		}
+		if resp.Node == targetNode {
+			return &resp, nil
+		}
+	}
+	return nil, fmt.Errorf("node %q not found or not responding", targetNode)
+}
+
+func runAdminGpuStatus(cmd *cobra.Command, _ []string) {
+	targetNode, _ := cmd.Flags().GetString("node")
+	resp, err := gpuNodeStatus(targetNode)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
+		os.Exit(1)
+	}
+
+	iommu := "unknown"
+	vfio := "unknown"
+	if resp.GPUCapable || len(resp.GPUModels) > 0 {
+		iommu = "active"
+		vfio = "loaded"
+	}
+
+	fmt.Printf("Node:            %s\n", resp.Node)
+	if len(resp.GPUModels) > 0 {
+		fmt.Printf("GPU hardware:    %s\n", strings.Join(resp.GPUModels, ", "))
+	} else {
+		fmt.Printf("GPU hardware:    none detected\n")
+	}
+	fmt.Printf("IOMMU:           %s\n", iommu)
+	fmt.Printf("vfio-pci:        %s\n", vfio)
+
+	if resp.GPUPassthrough {
+		fmt.Printf("Passthrough:     enabled\n")
+		fmt.Printf("GPU pool:        %d/%d allocated\n", resp.AllocGPUs, resp.TotalGPUs)
+
+		var gpuTypes []string
+		for _, cap := range resp.InstanceTypes {
+			gpuTypes = append(gpuTypes, cap.Name)
+		}
+		if len(gpuTypes) > 0 {
+			fmt.Printf("Instance types:  %s\n", strings.Join(gpuTypes, " "))
+		}
+	} else if resp.GPUCapable {
+		fmt.Printf("Passthrough:     disabled\n")
+		fmt.Printf("Instance types:  (none — run 'spx admin gpu enable' to activate)\n")
+	} else {
+		fmt.Printf("Passthrough:     disabled\n")
+		fmt.Printf("Instance types:  (prerequisites not met)\n")
+	}
+}
+
+// gpuToggle writes the TOML setting, sends SIGHUP, and polls until the daemon confirms the new state.
+// enable/disable always operate on the local node; run the command directly on the target host.
+func gpuToggle(_ *cobra.Command, enable bool) {
+	cfgPath := viper.GetString("config")
+	if cfgPath == "" {
+		cfgPath = DefaultConfigFile()
+	}
+	cfg, err := config.LoadConfig(cfgPath)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error loading config: %v\n", err)
+		os.Exit(1)
+	}
+	localNode := cfg.Node
+
+	// Check current state first.
+	resp, err := gpuNodeStatus(localNode)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
+		os.Exit(1)
+	}
+
+	if enable {
+		if resp.GPUPassthrough {
+			fmt.Println("GPU passthrough is already enabled.")
+			return
+		}
+		if !resp.GPUCapable {
+			fmt.Fprintln(os.Stderr, "Error: prerequisites not met on this node.")
+			fmt.Fprintln(os.Stderr, "  Run 'sudo spx admin gpu setup' to configure the host.")
+			os.Exit(1)
+		}
+	} else {
+		if !resp.GPUPassthrough {
+			fmt.Println("GPU passthrough is already disabled.")
+			return
+		}
+		if resp.AllocGPUs > 0 {
+			fmt.Fprintf(os.Stderr, "Error: %d GPU instance(s) are running. Terminate them first.\n", resp.AllocGPUs)
+			os.Exit(1)
+		}
+	}
+
+	// Write the TOML setting.
+	if err := admin.SetGPUPassthrough(cfgPath, localNode, enable); err != nil {
+		fmt.Fprintf(os.Stderr, "Error writing config: %v\n", err)
+		os.Exit(1)
+	}
+
+	// Signal the daemon to reload.
+	out, err := exec.Command("systemctl", "kill", "-s", "HUP", "spinifex-daemon").CombinedOutput()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error sending SIGHUP: %v\n%s\n", err, out)
+		os.Exit(1)
+	}
+
+	// Poll until the daemon confirms the new state.
+	action := "enable"
+	if !enable {
+		action = "disable"
+	}
+	fmt.Printf("Waiting for daemon to %s GPU passthrough", action)
+	deadline := time.Now().Add(30 * time.Second)
+	for time.Now().Before(deadline) {
+		time.Sleep(1 * time.Second)
+		fmt.Print(".")
+		r, err := gpuNodeStatus(localNode)
+		if err != nil {
+			continue
+		}
+		if r.GPUPassthrough == enable {
+			fmt.Println(" done.")
+			// Refresh the MOTD banner immediately so the login prompt
+			// reflects the new GPU passthrough state without requiring a reboot.
+			_ = exec.Command("spx", "admin", "banner", "--boot-check").Run()
+			runAdminGpuStatus(adminGpuStatusCmd, nil)
+			return
+		}
+	}
+	fmt.Println()
+	fmt.Fprintln(os.Stderr, "Timed out waiting for daemon. Check: journalctl -u spinifex-daemon -n 50")
+	os.Exit(1)
+}
+
+func runAdminGpuEnable(cmd *cobra.Command, _ []string) {
+	gpuToggle(cmd, true)
+}
+
+func runAdminGpuDisable(cmd *cobra.Command, _ []string) {
+	gpuToggle(cmd, false)
+}
+
+func runAdminGpuSetup(_ *cobra.Command, _ []string) {
+	if os.Getuid() != 0 {
+		fmt.Fprintln(os.Stderr, "Error: spx admin gpu setup must be run as root.")
+		os.Exit(1)
+	}
+
+	fmt.Println("==> Detecting GPU")
+	devices, err := gpu.Discover()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "GPU discovery failed: %v\n", err)
+		os.Exit(1)
+	}
+	if len(devices) == 0 {
+		fmt.Fprintln(os.Stderr, "No NVIDIA or AMD GPU found — is the card seated?")
+		os.Exit(1)
+	}
+	for _, d := range devices {
+		fmt.Printf("    %s - %s\n", d.PCIAddress, d.Model)
+	}
+
+	fmt.Println("==> Collecting PCI IDs for vfio-pci")
+	ids := gpuSetupCollectVFIOIDs(devices)
+	idsCSV := strings.Join(ids, ",")
+	fmt.Printf("    IDs: %s\n", idsCSV)
+
+	rebootNeeded := false
+
+	fmt.Println("==> Checking IOMMU")
+	iommuEntries, _ := os.ReadDir("/sys/kernel/iommu_groups/")
+	if len(iommuEntries) > 0 {
+		fmt.Println("    Active")
+	} else {
+		params := gpuSetupIOMMUParams()
+		changed, err := gpuSetupAddGRUBParams(params)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "Failed to update GRUB: %v\n", err)
+			os.Exit(1)
+		}
+		if changed {
+			fmt.Printf("    GRUB updated with: %s\n", params)
+		} else {
+			fmt.Println("    GRUB already has IOMMU params — refreshing")
+		}
+		ug := exec.Command("update-grub")
+		ug.Stdout, ug.Stderr = os.Stdout, os.Stderr
+		if err := ug.Run(); err != nil {
+			fmt.Fprintf(os.Stderr, "update-grub failed: %v\n", err)
+			os.Exit(1)
+		}
+		rebootNeeded = true
+	}
+
+	fmt.Println("==> Configuring vfio udev rule")
+	const vfioUdevRule = "/etc/udev/rules.d/99-spinifex-vfio.rules"
+	if _, err := os.Stat(vfioUdevRule); os.IsNotExist(err) {
+		rule := "SUBSYSTEM==\"vfio\", GROUP=\"spinifex\", MODE=\"0660\"\n"
+		if err := os.WriteFile(vfioUdevRule, []byte(rule), 0o644); err != nil {
+			fmt.Fprintf(os.Stderr, "Failed to write vfio udev rule: %v\n", err)
+			os.Exit(1)
+		}
+		fmt.Println("    Rule installed")
+		_ = exec.Command("udevadm", "control", "--reload-rules").Run()
+		_ = exec.Command("udevadm", "trigger", "--subsystem-match=vfio").Run()
+	} else {
+		fmt.Println("    Rule already present")
+	}
+
+	fmt.Println("==> Configuring nouveau blacklist")
+	const nouveauConf = "/etc/modprobe.d/blacklist-nouveau.conf"
+	if _, err := os.Stat(nouveauConf); os.IsNotExist(err) {
+		if err := os.WriteFile(nouveauConf, []byte("blacklist nouveau\noptions nouveau modeset=0\n"), 0o644); err != nil {
+			fmt.Fprintf(os.Stderr, "Failed to write nouveau blacklist: %v\n", err)
+			os.Exit(1)
+		}
+		fmt.Println("    nouveau blacklisted")
+		rebootNeeded = true
+	} else {
+		fmt.Println("    Already blacklisted")
+	}
+
+	fmt.Println("==> Configuring amdgpu blacklist")
+	const amdgpuConf = "/etc/modprobe.d/blacklist-amdgpu.conf"
+	hasAMD := false
+	for _, d := range devices {
+		if d.Vendor == gpu.VendorAMD {
+			hasAMD = true
+			break
+		}
+	}
+	if hasAMD {
+		if _, err := os.Stat(amdgpuConf); os.IsNotExist(err) {
+			if err := os.WriteFile(amdgpuConf, []byte("blacklist amdgpu\n"), 0o644); err != nil {
+				fmt.Fprintf(os.Stderr, "Failed to write amdgpu blacklist: %v\n", err)
+				os.Exit(1)
+			}
+			fmt.Println("    amdgpu blacklisted")
+			rebootNeeded = true
+		} else {
+			fmt.Println("    Already blacklisted")
+		}
+	} else {
+		fmt.Println("    No AMD GPUs — skipping")
+	}
+
+	fmt.Println("==> Configuring vfio-pci early binding")
+	const vfioPCIConf = "/etc/modprobe.d/vfio-pci.conf"
+	existing, _ := os.ReadFile(vfioPCIConf)
+	if !strings.Contains(string(existing), "ids="+idsCSV) {
+		if err := os.WriteFile(vfioPCIConf, []byte("options vfio-pci ids="+idsCSV+"\n"), 0o644); err != nil {
+			fmt.Fprintf(os.Stderr, "Failed to write vfio-pci.conf: %v\n", err)
+			os.Exit(1)
+		}
+		fmt.Printf("    vfio-pci.conf written (ids=%s)\n", idsCSV)
+		rebootNeeded = true
+	} else {
+		fmt.Printf("    Already configured for %s\n", idsCSV)
+	}
+
+	const initramfsModules = "/etc/initramfs-tools/modules"
+	modData, _ := os.ReadFile(initramfsModules)
+	if !strings.Contains(string(modData), "vfio_pci") {
+		f, err := os.OpenFile(initramfsModules, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0o644)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "Failed to open initramfs modules: %v\n", err)
+			os.Exit(1)
+		}
+		_, werr := fmt.Fprint(f, "\n# vfio early binding for GPU passthrough\nvfio\nvfio_iommu_type1\nvfio_pci\n")
+		f.Close()
+		if werr != nil {
+			fmt.Fprintf(os.Stderr, "Failed to write initramfs modules: %v\n", werr)
+			os.Exit(1)
+		}
+		fmt.Println("    vfio modules added to initramfs")
+		rebootNeeded = true
+	} else {
+		fmt.Println("    vfio modules already in initramfs")
+	}
+
+	if rebootNeeded {
+		fmt.Println("==> Updating initramfs")
+		ui := exec.Command("update-initramfs", "-u")
+		ui.Stdout, ui.Stderr = os.Stdout, os.Stderr
+		if err := ui.Run(); err != nil {
+			fmt.Fprintf(os.Stderr, "update-initramfs failed: %v\n", err)
+			os.Exit(1)
+		}
+		fmt.Println("\nSetup complete — reboot required.")
+		fmt.Println("  sudo reboot")
+		fmt.Println("  Then run: sudo spx admin gpu enable")
+		return
+	}
+
+	fmt.Println("==> Verifying vfio-pci module")
+	if _, err := os.Stat("/sys/module/vfio_pci"); os.IsNotExist(err) {
+		fmt.Fprintln(os.Stderr, "vfio_pci not loaded — check: journalctl -b | grep vfio")
+		os.Exit(1)
+	}
+	fmt.Println("    Loaded")
+
+	fmt.Println("==> Verifying GPU driver binding")
+	for _, d := range devices {
+		current, err := gpu.CurrentDriver(d.PCIAddress)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "Cannot read driver for %s: %v\n", d.PCIAddress, err)
+			os.Exit(1)
+		}
+		if current == "vfio-pci" {
+			fmt.Printf("    %s → vfio-pci\n", d.PCIAddress)
+			continue
+		}
+		if current != "" {
+			// Bound to native driver — blacklist/initramfs changes may not have taken effect.
+			fmt.Fprintf(os.Stderr, "%s is bound to %q, not vfio-pci — reboot may be required\n", d.PCIAddress, current)
+			fmt.Fprintf(os.Stderr, "  Check: ls -la /sys/bus/pci/devices/%s/driver\n", d.PCIAddress)
+			os.Exit(1)
+		}
+		// Device is unbound (ids= did not auto-claim it) — bind explicitly via driver_override.
+		fmt.Printf("    %s unbound — binding to vfio-pci\n", d.PCIAddress)
+		if _, err := gpu.BindVFIO(d.PCIAddress); err != nil {
+			fmt.Fprintf(os.Stderr, "Failed to bind %s to vfio-pci: %v\n", d.PCIAddress, err)
+			fmt.Fprintf(os.Stderr, "  Check: ls -la /sys/bus/pci/devices/%s/driver\n", d.PCIAddress)
+			os.Exit(1)
+		}
+		fmt.Printf("    %s → vfio-pci\n", d.PCIAddress)
+	}
+
+	fmt.Println("==> Enabling GPU passthrough")
+	gpuToggle(nil, true)
+}
+
+// gpuSetupCollectVFIOIDs collects vendor:device PCI ID pairs for each GPU and
+// all sibling PCI functions on the same device slot (e.g. GPU + HDMI audio).
+// All siblings must bind to vfio-pci together for IOMMU group isolation.
+func gpuSetupCollectVFIOIDs(devices []gpu.GPUDevice) []string {
+	seen := make(map[string]bool)
+	var ids []string
+	for _, d := range devices {
+		dot := strings.LastIndex(d.PCIAddress, ".")
+		if dot < 0 {
+			continue
+		}
+		prefix := d.PCIAddress[:dot]
+		matches, _ := filepath.Glob("/sys/bus/pci/devices/" + prefix + ".*")
+		for _, m := range matches {
+			vb, err1 := os.ReadFile(m + "/vendor")
+			db, err2 := os.ReadFile(m + "/device")
+			if err1 != nil || err2 != nil {
+				continue
+			}
+			v := strings.TrimPrefix(strings.ToLower(strings.TrimSpace(string(vb))), "0x")
+			dv := strings.TrimPrefix(strings.ToLower(strings.TrimSpace(string(db))), "0x")
+			id := v + ":" + dv
+			if !seen[id] {
+				seen[id] = true
+				ids = append(ids, id)
+			}
+		}
+	}
+	return ids
+}
+
+func gpuSetupIOMMUParams() string {
+	data, _ := os.ReadFile("/proc/cpuinfo")
+	if strings.Contains(string(data), "GenuineIntel") {
+		return "intel_iommu=on iommu=pt"
+	}
+	return "amd_iommu=on iommu=pt"
+}
+
+// gpuSetupAddGRUBParams appends iommu kernel params to GRUB_CMDLINE_LINUX_DEFAULT.
+// Returns (true, nil) if the file was modified, (false, nil) if params were already present.
+func gpuSetupAddGRUBParams(params string) (bool, error) {
+	const grubFile = "/etc/default/grub"
+	data, err := os.ReadFile(grubFile)
+	if err != nil {
+		return false, fmt.Errorf("read %s: %w", grubFile, err)
+	}
+	content := string(data)
+	if strings.Contains(content, "intel_iommu=on") || strings.Contains(content, "amd_iommu=on") {
+		return false, nil
+	}
+	re := regexp.MustCompile(`(GRUB_CMDLINE_LINUX_DEFAULT="[^"]*)"`)
+	updated := re.ReplaceAllStringFunc(content, func(match string) string {
+		groups := re.FindStringSubmatch(match)
+		if len(groups) < 2 {
+			return match
+		}
+		return groups[1] + " " + params + `"`
+	})
+	if updated == content {
+		return false, fmt.Errorf("GRUB_CMDLINE_LINUX_DEFAULT not found in %s", grubFile)
+	}
+	if err := os.WriteFile(grubFile, []byte(updated), 0o644); err != nil {
+		return false, fmt.Errorf("write %s: %w", grubFile, err)
+	}
+	return true, nil
+}

cmd/spinifex/cmd/admin_test.go

@@ -22,7 +22,7 @@ import (
 func TestPrintChecksumError(t *testing.T) {
 	image := utils.Images{Checksum: "https://example.com/SUMS", ChecksumType: "sha512"}
 	const imageFile = "/var/lib/img.tar.xz"
-	const imageName = "debian-12-x86_64"
+	const imageName = "debian-13-x86_64"
 
 	errs := []error{
 		fmt.Errorf("%w: expected abc got def", utils.ErrChecksumMismatch),
@@ -262,3 +262,72 @@ func TestPredastoreMultinodeTemplate_NatsURLLoopbackShim(t *testing.T) {
 	require.NoError(t, err)
 	assert.Contains(t, content2, `nats_url = "nats://10.11.12.1:4222"`)
 }
+
+// writePredastoreEncryptionKey is called from both `spx admin init` and `spx
+// admin join`; both depend on it producing a 32-byte file at 0600 in the
+// expected layout. Predastore's keyfile loader rejects anything that isn't
+// exactly that, so a regression here would break service startup on every
+// node.
+func TestWritePredastoreEncryptionKey(t *testing.T) {
+	configDir := t.TempDir()
+
+	keyPath, err := writePredastoreEncryptionKey(configDir)
+	require.NoError(t, err)
+	assert.Equal(t, filepath.Join(configDir, "predastore", "encryption.key"), keyPath)
+
+	info, err := os.Stat(keyPath)
+	require.NoError(t, err)
+	assert.Equal(t, int64(32), info.Size(), "predastore master key must be exactly 32 bytes")
+	assert.Equal(t, os.FileMode(0600), info.Mode().Perm(), "predastore master key must be mode 0600")
+
+	// Per-node invariant: two successive calls (on what would be two
+	// different nodes) must produce different key material. If they ever
+	// returned the same bytes we'd have silently lost the per-node
+	// blast-radius property.
+	configDir2 := t.TempDir()
+	keyPath2, err := writePredastoreEncryptionKey(configDir2)
+	require.NoError(t, err)
+
+	key1, err := os.ReadFile(keyPath)
+	require.NoError(t, err)
+	key2, err := os.ReadFile(keyPath2)
+	require.NoError(t, err)
+	assert.NotEqual(t, key1, key2, "per-node keys must differ")
+}
+
+// Joiners run `spx admin join` against a configDir that doesn't yet contain
+// a predastore/ subdir. The helper must create it (predastore subdir is
+// owned by spinifex-storage in prod, but the helper just needs the dir to
+// exist).
+func TestWritePredastoreEncryptionKey_CreatesMissingDir(t *testing.T) {
+	configDir := t.TempDir()
+	// Don't pre-create the predastore subdir — simulate a fresh joiner.
+
+	keyPath, err := writePredastoreEncryptionKey(configDir)
+	require.NoError(t, err)
+
+	dirInfo, err := os.Stat(filepath.Dir(keyPath))
+	require.NoError(t, err)
+	assert.True(t, dirInfo.IsDir())
+}
+
+// TestImagesRemoveCmd_FlagSchema guards the public flag surface for
+// `spx admin images remove`. --image-id is required so cobra rejects a bare
+// invocation; --force/--yes default to false.
+func TestImagesRemoveCmd_FlagSchema(t *testing.T) {
+	imageIDFlag := imagesRemoveCmd.Flags().Lookup("image-id")
+	require.NotNil(t, imageIDFlag, "--image-id must be defined")
+	assert.Equal(t, []string{"true"}, imageIDFlag.Annotations[cobraRequiredAnnotation],
+		"--image-id must be marked required")
+
+	forceFlag := imagesRemoveCmd.Flags().Lookup("force")
+	require.NotNil(t, forceFlag)
+	assert.Equal(t, "false", forceFlag.DefValue)
+
+	yesFlag := imagesRemoveCmd.Flags().Lookup("yes")
+	require.NotNil(t, yesFlag)
+	assert.Equal(t, "false", yesFlag.DefValue)
+}
+
+// cobraRequiredAnnotation is the annotation key cobra uses to mark required flags.
+const cobraRequiredAnnotation = "cobra_annotation_bash_completion_one_required_flag"

cmd/spinifex/cmd/cluster.go

@@ -98,7 +98,7 @@ func runClusterShutdown(cmd *cobra.Command, args []string) {
 			}
 			nc.Flush()
 			// Wait briefly for messages to propagate
-			time.Sleep(2 * time.Second)
+			time.Sleep(200 * time.Millisecond)
 			fmt.Printf("[INFRA] Complete\n")
 			break
 		}

cmd/spinifex/cmd/service.go

@@ -115,6 +115,13 @@ var predastoreStartCmd = &cobra.Command{
 			return
 		}
 
+		encryptionKeyFile := viper.GetString("encryption-key-file")
+
+		if encryptionKeyFile == "" {
+			fmt.Println("Encryption key file is not set")
+			return
+		}
+
 		nodeID := viper.GetInt("node-id")
 		pprofEnabled := viper.GetBool("pprof")
 		pprofOutput := viper.GetString("pprof-output")
@@ -128,6 +135,8 @@ var predastoreStartCmd = &cobra.Command{
 			TlsCert:    tlsCert,
 			TlsKey:     tlsKey,
 
+			EncryptionKeyFile: encryptionKeyFile,
+
 			NodeID: nodeID,
 
 			PprofEnabled:    pprofEnabled,
@@ -858,13 +867,20 @@ func init() {
 	viper.BindEnv("tls-key", "SPINIFEX_PREDASTORE_TLS_KEY")
 	viper.BindPFlag("tls-key", predastoreCmd.PersistentFlags().Lookup("tls-key"))
 
+	// Predastore at-rest encryption master key (per node; mode 0600)
+	predastoreCmd.PersistentFlags().String("encryption-key-file", "", "Path to this node's 32-byte AES-256 master key file (required)")
+	viper.BindEnv("encryption-key-file", "SPINIFEX_PREDASTORE_ENCRYPTION_KEY_FILE")
+	viper.BindPFlag("encryption-key-file", predastoreCmd.PersistentFlags().Lookup("encryption-key-file"))
+
 	// Predastore Backend
 	predastoreCmd.PersistentFlags().String("backend", "distributed", "Predastore (S3) backend")
 	viper.BindEnv("backend", "SPINIFEX_PREDASTORE_BACKEND")
 	viper.BindPFlag("backend", predastoreCmd.PersistentFlags().Lookup("backend"))
 
-	// Predastore Node ID
-	predastoreCmd.PersistentFlags().Int("node-id", 0, "Predastore (S3) node ID")
+	// Predastore Node ID. Default -1 is dev mode (launch every configured
+	// QUIC node in-process). Production deployments set this to the node's
+	// real ID (>= 1) via SPINIFEX_PREDASTORE_NODE_ID or --node-id.
+	predastoreCmd.PersistentFlags().Int("node-id", -1, "Predastore (S3) node ID (-1 = dev mode, >= 1 = production)")
 	viper.BindEnv("node-id", "SPINIFEX_PREDASTORE_NODE_ID")
 	viper.BindPFlag("node-id", predastoreCmd.PersistentFlags().Lookup("node-id"))
 

cmd/spinifex/cmd/templates/predastore-multinode.toml

@@ -1,6 +1,6 @@
 # Predastore S3-compatible storage configuration file in TOML format.
 # Generated for multi-node cluster
-version = "3"
+version = "4"
 region = "{{.Region}}"
 
 host = "0.0.0.0"
@@ -9,8 +9,13 @@ port = 8443
 debug = false
 
 [rs]
+{{- if eq (len .Nodes) 2}}
+data = 1
+parity = 1
+{{- else}}
 data = 2
 parity = 1
+{{- end}}
 {{- range .Nodes}}
 
 [[db]]
@@ -39,10 +44,12 @@ region = "{{.Region}}"
 type = "distributed"
 public = false
 encryption = ""
+account_id = "000000000000"
 
 [[auth]]
 access_key_id = "{{.AccessKey}}"
 secret_access_key = "{{.SecretKey}}"
+account_id = "000000000000"
 policy = [
   { bucket = "predastore", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
 ]

cmd/spinifex/cmd/templates/predastore.toml

@@ -1,6 +1,6 @@
 # Predastore S3-compatible storage configuration file in TOML format.
 # MULTI NODE used for testing
-version = "3"
+version = "4"
 region = "{{.Region}}"
 
 # Optionally configure port and host, or specify via daemon CLI options
@@ -51,19 +51,19 @@ secret_access_key = "{{.SecretKey}}"
 # Using distributed hash ring for object > shard placement.
 [[nodes]]
 id = 1
-host = "0.0.0.0"
+host = "127.0.0.1"
 port = 9991
 path = "distributed/nodes/node-1/"
 
 [[nodes]]
 id = 2
-host = "0.0.0.0"
+host = "127.0.0.1"
 port = 9992
 path = "distributed/nodes/node-2/"
 
 [[nodes]]
 id = 3
-host = "0.0.0.0"
+host = "127.0.0.1"
 port = 9993
 path = "distributed/nodes/node-3/"
 
@@ -73,10 +73,12 @@ region = "ap-southeast-2"
 type = "distributed"
 public = false
 encryption = ""
+account_id = "000000000000"
 
 [[auth]]
 access_key_id = "{{.AccessKey}}"
 secret_access_key = "{{.SecretKey}}"
+account_id = "000000000000"
 policy = [
   { bucket = "predastore", actions = ["s3:ListBucket",  "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
 ]

cmd/spinifex/cmd/templates/spinifex.toml

@@ -52,6 +52,7 @@ advertise = "{{.AdvertiseIP}}"
 {{- end}}
 region = "{{.Region}}"
 az = "{{.Az}}"
+data_dir = "{{.DataDir}}"
 {{- if .Services}}
 services = [{{range $i, $s := .Services}}{{if $i}}, {{end}}"{{$s}}"{{end}}]
 {{- end}}
@@ -67,6 +68,9 @@ dev_networking = false
 {{- else}}
 dev_networking = true
 {{- end}}
+{{- if .GPUPassthrough}}
+gpu_passthrough = true
+{{- end}}
 
 [nodes.{{.Node}}.awsgw]
 host = "{{.BindIP}}:9999"

cmd/spinifex/cmd/top.go

@@ -88,7 +88,7 @@ func runTopNodes(cmd *cobra.Command, args []string) {
 	sort.Strings(nodeNames)
 
 	nodeTable := pterm.TableData{
-		{"NAME", "CPU (used/total)", "MEM (used/total)", "VMs"},
+		{"NAME", "CPU (used/total)", "MEM (used/total)", "GPU (used/total)", "VMs"},
 	}
 
 	// Aggregate instance type capacity across all nodes
@@ -96,10 +96,17 @@ func runTopNodes(cmd *cobra.Command, args []string) {
 
 	for _, name := range nodeNames {
 		if resp, ok := respondedNodes[name]; ok {
+			gpuCol := "-"
+			if resp.GPUPassthrough {
+				gpuCol = fmt.Sprintf("%d/%d", resp.AllocGPUs, resp.TotalGPUs)
+			} else if resp.GPUCapable {
+				gpuCol = fmt.Sprintf("0/%d*", len(resp.GPUModels))
+			}
 			nodeTable = append(nodeTable, []string{
 				resp.Node,
 				fmt.Sprintf("%d/%d", resp.AllocVCPU, resp.TotalVCPU),
 				fmt.Sprintf("%s/%s", formatMemGB(resp.AllocMemGB), formatMemGB(resp.TotalMemGB)),
+				gpuCol,
 				strconv.Itoa(resp.VMCount),
 			})
 
@@ -120,6 +127,7 @@ func runTopNodes(cmd *cobra.Command, args []string) {
 				"-",
 				"-",
 				"-",
+				"-",
 			})
 		}
 	}

cmd/spinifex/main.go

@@ -16,7 +16,10 @@ along with this program.  If not, see <https://www.gnu.org/licenses/>.
 */
 package main
 
-import "github.com/mulgadc/spinifex/cmd/spinifex/cmd"
+import (
+	"github.com/mulgadc/spinifex/cmd/spinifex/cmd"
+	_ "github.com/mulgadc/spinifex/internal/fipsboot"
+)
 
 func main() {
 	cmd.Execute()

docs/COMMANDS.md

@@ -45,7 +45,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 
 | Command | Flags | Prerequisites | Basic Logic | Output | Status |
 |---------|-------|---------------|-------------|--------|--------|
-| `spx top nodes` | `--timeout` (default: 3s) | Cluster must be running (NATS) | Publishes to `spinifex.node.status` fan-out topic → collects CPU/memory usage per node → aggregates instance type capacity across all nodes → prints two tables: per-node resource usage and cluster-wide instance type availability | Table 1: NAME, CPU (used/total), MEM (used/total), VMs. Table 2: INSTANCE TYPE, AVAILABLE, VCPU, MEMORY | **DONE** |
+| `spx top nodes` | `--timeout` (default: 3s) | Cluster must be running (NATS) | Publishes to `spinifex.node.status` fan-out topic → collects CPU/memory usage per node → aggregates instance type capacity across all nodes → prints two tables: per-node resource usage and cluster-wide instance type availability | Table 1: NAME, CPU (used/total), MEM (used/total), GPU (used/total), VMs — GPU column shows `allocated/total` when passthrough is enabled, `0/count*` when the node is GPU-capable but passthrough is disabled, or `-` when no GPU is present. Table 2: INSTANCE TYPE, AVAILABLE, VCPU, MEMORY | **DONE** |
 
 ### Cluster Initialization
 
@@ -89,8 +89,9 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 
 | Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
 |---------|-------|---------------|-------------|------------|--------|
-| `spx admin images import` | `--name`, `--file`, `--force`, `--skip-verify` | Cluster must be running; either `--name` (catalog download) or `--file` (operator-supplied media) | Catalog imports (`--name`) download the image, fetch the catalog `Checksum` URL, and verify the SHA-256/SHA-512 digest before extraction. Mismatch fails closed; the cached file is left on disk and `--force` re-downloads. `--file` imports skip verification — operator-supplied media is outside Spinifex's trust boundary, and the skip is logged at INFO for audit. `--skip-verify` bypasses verification for catalog imports and emits a WARN slog + stderr notice; use only for debugging or when upstream mirrors are confirmed-broken. | 1. Import valid catalog image (verifies checksum)<br>2. Tampered cache hit fails with `ErrChecksumMismatch`<br>3. `--force` recovers after a mismatch<br>4. `--file` import skips verification<br>5. `--skip-verify` bypasses checksum with WARN | **DONE** |
+| `spx admin images import` | `--name`, `--file`, `--distro`, `--version`, `--arch`, `--platform`, `--boot-mode` (bios/uefi/uefi-preferred), `--tag`, `--force`, `--skip-verify` | Cluster must be running; either `--name` (catalog download) or `--file` (operator-supplied media). `--file` mode additionally requires `--distro`, `--version`, `--arch`, `--boot-mode`. | Catalog imports (`--name`) download the image, fetch the catalog `Checksum` URL, verify the SHA-256/SHA-512 digest, and inherit `BootMode` from the catalog entry. `--boot-mode` overrides the catalog value when set. Mismatch fails closed; the cached file is left on disk and `--force` re-downloads. `--file` imports skip checksum verification (operator-supplied media is outside Spinifex's trust boundary, the skip is logged at INFO for audit) and require an explicit `--boot-mode` because there is no catalog metadata to inherit from. `--skip-verify` bypasses verification for catalog imports and emits a WARN slog + stderr notice; use only for debugging or when upstream mirrors are confirmed-broken. | 1. Import valid catalog image (verifies checksum)<br>2. Tampered cache hit fails with `ErrChecksumMismatch`<br>3. `--force` recovers after a mismatch<br>4. `--file` import skips verification<br>5. `--skip-verify` bypasses checksum with WARN<br>6. `--file` without `--boot-mode` rejected<br>7. `--boot-mode=invalid` rejected | **DONE** |
 | `spx admin images list` | — | None | Lists available OS images that can be imported or downloaded | 1. List available images | **DONE** |
+| `spx admin images remove` | `--image-id` (required), `--force`, `--yes` | Cluster must be running; predastore reachable. Targets AMIs imported via `spx admin images import` (non-account `ImageOwnerAlias`, e.g. `"system"`). | Loads `ami-<id>/config.json`, walks transitive dependents — copied snapshots whose `VolumeID == imageID`, volumes whose `SnapshotID` references the internal `snap-ami-<id>` or any derived snap, and account AMIs created via `CopyImage` whose `SnapshotID` is a derived snap — then prompts (skipped with `--yes`) before deleting `ami-<id>/config.json` (the DescribeImages barrier) followed by the rest of `ami-<id>/` and `snap-ami-<id>/`. Account-owned AMIs are refused with a hint pointing at `aws ec2 deregister-image` + `aws ec2 delete-snapshot`. `--force` bypasses the dependency, ownership and config-corrupt checks for salvage of orphaned blocks. | 1. Happy path: system AMI, no deps → deleted<br>2. Account-owned → refused with AWS-flow hint<br>3. Missing/corrupt config → `InvalidAMIID.NotFound` (salvageable with `--force`)<br>4. Dependent volume (direct or via copied snap) → refused<br>5. Dependent account AMI → refused<br>6. `--force` overrides dependents<br>7. Idempotent re-run after partial delete | **DONE** |
 
 #### Image integrity verification (CMMC SI.L1-3.14.2)
 
@@ -123,6 +124,44 @@ served. A mirror compromise that swaps both image and sums file is not
 detected; closing that gap requires GPG signature verification of the sums
 file, deferred to a later phase.
 
+#### `spx admin images remove` caveats
+
+Admin-imported AMIs (`ImageOwnerAlias = "system"`) live
+under the `ami-<id>/` S3 prefix and use a viperblock-internal snap checkpoint
+at `snap-ami-<id>/` — there is no `snap-<id>/metadata.json`. The AWS handlers
+(`DeregisterImage`, `DeleteSnapshot`) reject system owners with
+`UnauthorizedOperation`, which is the right behaviour for tenant API callers
+but leaves no AWS-flow path to reclaim space. `spx admin images remove` is
+the admin-trust-boundary counterpart that performs the dependency walk and
+hard-deletes the blocks directly against predastore.
+
+`CopyImage` of a system AMI is metadata-only: it writes a fresh
+`snap-<acct>/metadata.json` whose `VolumeID` points at `ami-<sys>` and a new
+`ami-<acct>/config.json` referencing that snap. Volumes launched from the
+copied AMI read transitively from `ami-<sys>/chunks/...`. The remove command
+walks this transitive set and refuses if anything references the target.
+
+**TOCTOU window:** between the safety scan and the `config.json` delete a
+concurrent `RunInstances` against the AMI could create a new dependent
+volume. The window is sub-second on a healthy cluster. The admin running
+this command is expected to know the fleet's operational state; if the race
+fires the result is a `vol-<id>` with deleted backing blocks, recovered by
+terminating the orphaned instance.
+
+**`--force` bypasses every safety check** (dependents, ownership, missing /
+corrupt `config.json`). Use only for salvage of orphaned blocks. Running it
+against a live system AMI corrupts every dependent volume on the next disk
+read.
+
+### GPU Management
+
+| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
+|---------|-------|---------------|-------------|------------|--------|
+| `spx admin gpu status` | `--node` (default: local node) | Cluster must be running (NATS) | Queries `spinifex.node.status` fan-out → finds the target node response → prints Node, GPU hardware (model list or "none detected"), IOMMU state, vfio-pci state, passthrough enabled/disabled, and GPU pool allocation (`allocated/total`). Also lists GPU-capable instance types when passthrough is active. | 1. Local node with passthrough enabled (shows pool and instance types)<br>2. Local node capable but passthrough disabled (shows "run enable")<br>3. Node without GPU hardware (shows "none detected")<br>4. `--node` targets a remote node<br>5. Node not found or not responding (error) | **DONE** |
+| `spx admin gpu enable` | — | Cluster must be running (NATS); node must be GPU-capable (IOMMU + vfio-pci configured) | Checks current passthrough state via NATS → errors if already enabled or prerequisites not met (directs to `setup`) → writes `gpu_passthrough = true` to `spinifex.toml` via `admin.SetGPUPassthrough` → sends SIGHUP to `spinifex-daemon` via `systemctl kill -s HUP` → polls node status for up to 30 s until daemon confirms new state → prints final `gpu status` output. Must be run directly on the target host. | 1. Enable on capable node<br>2. Already enabled (no-op)<br>3. Prerequisites not met (error, suggests `setup`)<br>4. Daemon does not confirm within 30 s (timeout error) | **DONE** |
+| `spx admin gpu disable` | — | Cluster must be running (NATS); no GPU instances may be running | Checks current passthrough state via NATS → errors if already disabled or if `AllocGPUs > 0` (must terminate GPU instances first) → writes `gpu_passthrough = false` → sends SIGHUP to `spinifex-daemon` → polls for up to 30 s → prints final `gpu status` output. Must be run directly on the target host. | 1. Disable with no GPU instances running<br>2. Already disabled (no-op)<br>3. GPU instances running (error with count)<br>4. Daemon does not confirm within 30 s (timeout error) | **DONE** |
+| `spx admin gpu setup` | — | Must be run as root; NVIDIA or AMD GPU must be present | Idempotent host configuration for GPU passthrough. Steps (skipped if already applied): detect GPUs via `gpu.Discover` → collect PCI IDs for all IOMMU-group siblings → check/enable IOMMU in GRUB (`intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt`) → write vfio udev rule (`/etc/udev/rules.d/99-spinifex-vfio.rules`) → blacklist nouveau (`/etc/modprobe.d/blacklist-nouveau.conf`) → blacklist amdgpu if AMD GPU present (`/etc/modprobe.d/blacklist-amdgpu.conf`) → write vfio-pci early binding config (`/etc/modprobe.d/vfio-pci.conf`) → add vfio modules to initramfs. If any change requires a reboot: runs `update-initramfs -u` and exits with reboot instructions. After reboot: verifies `vfio_pci` module is loaded → verifies each GPU is bound to `vfio-pci` (binds explicitly via `driver_override` if unbound) → calls `gpu enable` to activate passthrough. | 1. First run on unconfigured host (GRUB updated, reboot required)<br>2. Re-run after reboot (verifies bindings, enables passthrough)<br>3. Already fully configured (all steps skipped, enables passthrough)<br>4. AMD GPU present (blacklists amdgpu in addition to nouveau)<br>5. Not run as root (error)<br>6. No GPU hardware detected (error) | **DONE** |
+
 ## AWS Commands
 
 ### EC2 - Instance Management
@@ -133,13 +172,14 @@ file, deferred to a later phase.
 | `describe-instances` | `--instance-ids`, `--filters` (instance-state-name, instance-id, instance-type, vpc-id, subnet-id, tag:\*, tag-key, tag-value) | `--max-results`, `--next-token`, `--dry-run` | None | Gateway fans out NATS `ec2.DescribeInstances` to all nodes (no queue group) → each daemon returns local instances → gateway aggregates and returns combined list. Filters applied per-node before aggregation (reduces payload). Also applies to stopped/terminated instances via `describeInstancesFromKV()`. | 1. Describe all instances (no filter)<br>2. Describe by instance ID<br>3. Describe with filters (e.g. instance-state-name)<br>4. Instance not found returns empty set<br>5. Multi-node aggregation returns instances from all nodes<br>6. Filter by tag<br>7. Unknown filter returns InvalidParameterValue | **DONE** |
 | `start-instances` | `--instance-ids` | `--dry-run`, `--force` | `run-instances` (instance must exist in stopped state) | Gateway sends NATS `ec2.cmd.{instance-id}` → daemon restarts stopped QEMU process with same config → state transitions stopped→pending→running | 1. Start a stopped instance<br>2. Start already-running instance (error: IncorrectInstanceState)<br>3. Start with invalid instance ID<br>4. Verify volumes re-mount on start | **DONE** |
 | `stop-instances` | `--instance-ids` | `--force`, `--hibernate`, `--dry-run` | `run-instances` (instance must be running) | Gateway sends NATS to target node → daemon issues QMP `system_powerdown` for graceful shutdown → monitors heartbeat until QEMU exits → state transitions running→stopping→stopped | 1. Graceful stop of running instance<br>2. Force stop (kills QEMU process)<br>3. Stop already-stopped instance (error)<br>4. Verify ~30s heartbeat detection | **DONE** |
-| `terminate-instances` | `--instance-ids`, `DeleteOnTermination` (per-volume flag, default true) | `--dry-run` | `run-instances` (instance must exist) | Gateway sends NATS to target node → daemon kills QEMU process → cleans up NBD mounts → deletes volumes with `DeleteOnTermination=true` via `volumeService.DeleteVolume()` (S3 cleanup of vol/, vol-efi/, vol-cloudinit/) → internal volumes (EFI, cloud-init) always cleaned up via `ebs.delete` NATS → volumes with `DeleteOnTermination=false` left in available state → state→terminated | 1. Terminate running instance<br>2. Terminate stopped instance<br>3. Terminate with DeleteOnTermination=true deletes volumes<br>4. Terminate with DeleteOnTermination=false preserves volumes<br>5. Terminate already-terminated (idempotent)<br>6. Internal volumes (EFI, cloud-init) always cleaned up<br>7. Invalid instance ID | **DONE** |
+| `terminate-instances` | `--instance-ids`, `DeleteOnTermination` (per-volume flag, default true) | `--dry-run` | `run-instances` (instance must exist) | Gateway sends NATS to target node → daemon checks `DisableApiTermination` first and rejects with `OperationNotPermitted` if set (running or stopped, no side effects) → kills QEMU process → cleans up NBD mounts → deletes volumes with `DeleteOnTermination=true` via `volumeService.DeleteVolume()` (S3 cleanup of vol/, vol-efi/, vol-cloudinit/) → internal volumes (EFI, cloud-init) always cleaned up via `ebs.delete` NATS → volumes with `DeleteOnTermination=false` left in available state → state→terminated | 1. Terminate running instance<br>2. Terminate stopped instance<br>3. Terminate with DeleteOnTermination=true deletes volumes<br>4. Terminate with DeleteOnTermination=false preserves volumes<br>5. Terminate already-terminated (idempotent)<br>6. Internal volumes (EFI, cloud-init) always cleaned up<br>7. Invalid instance ID<br>8. Running protected instance rejected (OperationNotPermitted, state stays running)<br>9. Stopped protected instance rejected (resources intact) | **DONE** |
 | `reboot-instances` | `--instance-ids` | `--dry-run` | `run-instances` (instance must be running) | Gateway validates instance IDs → sends EC2InstanceCommand with `RebootInstance=true` via NATS `ec2.cmd.{instanceId}` → daemon validates instance is in StateRunning (returns IncorrectInstanceState if stopped) → issues QMP `system_reset` → instance reboots without stopping, stays in running state | 1. Reboot running instance<br>2. Reboot multiple instances<br>3. Reboot stopped instance (error: IncorrectInstanceState)<br>4. Instance not found (error: InvalidInstanceID.NotFound)<br>5. Verify instance stays in running state after reboot | **DONE** |
 | `describe-instance-types` | `--filters` (capacity filter only) | `--instance-types`, `--max-results`, `--next-token`, `--dry-run`, all other filters | None | Gateway fans out NATS `ec2.DescribeInstanceTypes` to all nodes → each daemon reports supported types (t3.micro/small/medium/large) with vCPU/memory specs → gateway deduplicates and returns | 1. List all instance types<br>2. Filter by specific type<br>3. Filter with `capacity=true` shows available slots<br>4. Verify vCPU/memory specs match hardware | **DONE** |
-| `modify-instance-attribute` | `--instance-id`, `--instance-type`, `--user-data` | `--disable-api-termination`, `--ebs-optimized`, `--source-dest-check`, `--instance-initiated-shutdown-behavior`, `--block-device-mappings`, `--groups`, `--ena-support`, `--sriov-net-support` | Instance must be stopped (in NATS KV) | Gateway validates input (exactly one attribute per call, instance ID format) → NATS `ec2.ModifyInstanceAttribute` with `spinifex-workers` queue group → daemon loads stopped instance from JetStream KV → applies attribute change → writes back to KV → returns `{}` on success. **InstanceType**: updates vm.InstanceType, Config, and Instance fields; clears StateReason (enables recovery from instance-type-missing bug). **UserData**: stores decoded content in vm.UserData and re-encodes to base64 for RunInstancesInput (cloud-init on next start). No instance type pre-validation (matches AWS — invalid types accepted, fail at StartInstances time). | 1. Change instance type while stopped<br>2. Change user data while stopped<br>3. Modify running instance (error: NotFound — running instances not in KV)<br>4. Instance not found (error: InvalidInstanceID.NotFound)<br>5. Instance not stopped (error: IncorrectInstanceState)<br>6. Invalid instance type accepted (fails on start with InsufficientInstanceCapacity)<br>7. StateReason cleared on type change (recovery from capacity-unavailable)<br>8. Missing/malformed instance ID (error: InvalidInstanceID.Malformed)<br>9. No attribute set (error: InvalidParameterValue)<br>10. Multiple attributes in one call (error: InvalidParameterValue) | **DONE** |
+| `modify-instance-attribute` | `--instance-id`, `--instance-type`, `--user-data`, `--disable-api-termination` | `--ebs-optimized`, `--source-dest-check`, `--instance-initiated-shutdown-behavior`, `--block-device-mappings`, `--groups`, `--ena-support`, `--sriov-net-support` | Instance must exist; `instance-type` and `user-data` require stopped state, `disable-api-termination` works on running or stopped | Gateway validates input (exactly one attribute per call, instance ID format) → NATS `ec2.ModifyInstanceAttribute` with `spinifex-workers` queue group → daemon dispatches by state: stopped instances loaded from JetStream KV, running instances updated via `vm.Manager.UpdateAndPersist` (atomic flag check + persist under one lock to prevent Modify/Terminate race) → returns `{}` on success. **InstanceType**: updates vm.InstanceType, Config, and Instance fields; clears StateReason (enables recovery from instance-type-missing bug). **UserData**: stores decoded content in vm.UserData and re-encodes to base64 for RunInstancesInput (cloud-init on next start). **DisableApiTermination**: stamps `RunInstancesInput.DisableApiTermination`; flag survives stop/start and daemon restarts. No instance type pre-validation (matches AWS — invalid types accepted, fail at StartInstances time). | 1. Change instance type while stopped<br>2. Change user data while stopped<br>3. Toggle disable-api-termination on running instance<br>4. Toggle disable-api-termination on stopped instance<br>5. Modify instance-type on running instance (error: NotFound — running instances not in KV)<br>6. Instance not found (error: InvalidInstanceID.NotFound)<br>7. Instance not stopped for type/user-data change (error: IncorrectInstanceState)<br>8. Invalid instance type accepted (fails on start with InsufficientInstanceCapacity)<br>9. StateReason cleared on type change (recovery from capacity-unavailable)<br>10. Missing/malformed instance ID (error: InvalidInstanceID.Malformed)<br>11. No attribute set (error: InvalidParameterValue)<br>12. Multiple attributes in one call (error: InvalidParameterValue) | **DONE** |
 | `get-console-output` | `--instance-id` | `--latest` (always returns latest), `--dry-run` | Instance must be running on a node | Gateway sends NATS `ec2.{instanceId}.GetConsoleOutput` (per-instance topic, routed to owning node) → daemon reads console log file from disk → returns last 64KB base64-encoded with timestamp. Always available regardless of serial console access setting (matches AWS behavior). | 1. Get output from running instance<br>2. Empty log file returns empty output<br>3. Instance not found (error: InvalidInstanceID.NotFound) | **DONE** |
-| `describe-instance-attribute` | `--instance-id`, `--attribute` (instanceType, userData, instanceInitiatedShutdownBehavior, disableApiTermination, disableApiStop, ebsOptimized, enaSupport, sourceDestCheck, rootDeviceName, kernel, ramdisk) | `--dry-run` | Instance must exist (running or stopped) | Gateway validates input → NATS `ec2.DescribeInstanceAttribute` with `spinifex-workers` queue group → daemon checks running instances first (`d.Instances.VMS`), then stopped instances in JetStream KV → returns single attribute per call (matches AWS behavior). Stored attributes (`instanceType`, `userData`) return real values; unstored attributes return AWS defaults (`instanceInitiatedShutdownBehavior`=stop, `disableApiTermination`=false, etc.) | 1. Get instanceType from running instance<br>2. Get userData from stopped instance<br>3. Get default disableApiTermination<br>4. Invalid attribute name (error)<br>5. Instance not found (error: InvalidInstanceID.NotFound) | **DONE** |
+| `describe-instance-attribute` | `--instance-id`, `--attribute` (instanceType, userData, instanceInitiatedShutdownBehavior, disableApiTermination, disableApiStop, ebsOptimized, enaSupport, sourceDestCheck, rootDeviceName, kernel, ramdisk) | `--dry-run` | Instance must exist (running or stopped) | Gateway validates input → NATS `ec2.DescribeInstanceAttribute` with `spinifex-workers` queue group → daemon checks running instances first (`d.Instances.VMS`), then stopped instances in JetStream KV → returns single attribute per call (matches AWS behavior). Stored attributes (`instanceType`, `userData`, `disableApiTermination`) return real values from `RunInstancesInput`; remaining attributes return AWS defaults (`instanceInitiatedShutdownBehavior`=stop, etc.) | 1. Get instanceType from running instance<br>2. Get userData from stopped instance<br>3. Get disableApiTermination (reflects RunInstances flag + Modify updates)<br>4. Invalid attribute name (error)<br>5. Instance not found (error: InvalidInstanceID.NotFound) | **DONE** |
 | `describe-instance-credit-specifications` | `--instance-ids` | `--filters`, `--max-results`, `--dry-run` | None | Gateway-only stub — returns `CpuCredits: "standard"` for each requested instance ID. No daemon round-trip. T-series credit mode is not persisted. | 1. Get credit spec for T-series instance<br>2. Multiple instance IDs | **DONE** |
+| `describe-instance-status` | `--instance-ids`, `--include-all-instances`, `--filters` (availability-zone, instance-state-code, instance-state-name, tag:\*) | `--max-results`, `--next-token`, `--dry-run` (silently ignored), `event.*` / `instance-status.*` / `system-status.*` filters (rejected with InvalidParameterValue) | None | Gateway fans out NATS `ec2.DescribeInstanceStatus` to all nodes (no queue group). Each daemon validates input (i- prefix, filter names), iterates local VMs and renders InstanceStatus: `Running`→ok/passed, everything-else-included→not-applicable. When `IncludeAllInstances=true`, gateway also queries `ec2.DescribeStoppedInstances` in parallel and transforms reservation entries into InstanceStatus with `not-applicable`. Aggregated results dedup'd by InstanceId (running > stopped). Terminated and error-state instances never returned. No real health probing — static values reflect Mulga's lifecycle state only. | 1. Status of running instance (ok/passed)<br>2. Filter by instance ID<br>3. Default excludes stopped instances<br>4. IncludeAllInstances=true includes stopped (not-applicable)<br>5. Cross-account instance excluded<br>6. Unknown filter returns InvalidParameterValue<br>7. Malformed instance ID rejected<br>8. Tag filter narrows correctly | **DONE** |
 | `monitor-instances` | — | `--instance-ids` | Instance must exist | Enable basic monitoring (CPU, disk, network) → store metrics in NATS KV → return monitoring state | 1. Enable monitoring<br>2. Verify monitoring state in describe-instances | **NOT STARTED** |
 | `unmonitor-instances` | — | `--instance-ids` | Instance must exist | Disable detailed monitoring → revert to basic monitoring → return monitoring state | 1. Disable monitoring<br>2. Verify monitoring state reverts in describe-instances | **NOT STARTED** |
 
@@ -158,7 +198,7 @@ file, deferred to a later phase.
 |---------|-------------------|---------------|---------------|-------------|------------|--------|
 | `describe-images` | `--image-ids` (format validation only), `--owners` (self, account ID, alias), `--filters` (name, state, architecture, image-id, is-public, owner-id, description, image-type, tag:\*) | `--executable-users`, `--include-deprecated`, `--include-disabled`, `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeImages` → daemon reads AMI metadata from Predastore S3 buckets (ami-*) → filters by ImageIds, Owners, and Filters → returns image list with state, architecture, block device mappings | 1. List all images<br>2. Filter by image ID<br>3. Filter by owner (self/amazon)<br>4. Non-existent AMI returns empty<br>5. Verify metadata fields (architecture, state, rootDeviceName)<br>6. Filter by name wildcard<br>7. Unknown filter returns InvalidParameterValue | **DONE** |
 | `create-image` | `--instance-id`, `--name`, `--description`, `--tag-specifications` | `--no-reboot`, `--block-device-mappings`, `--dry-run` | Instance must exist (running or stopped) | Gateway validates input → NATS `ec2.{instanceId}.CreateImage` (per-instance topic) → daemon extracts root volume → snapshots via `ebs.snapshot` NATS (running) or offline S3 copy (stopped) → creates AMI metadata in S3 (`{amiId}/config.json`) → stores tags → returns ami-ID. Duplicate AMI name validation enforced. | 1. Create image from running instance<br>2. Create image from stopped instance<br>3. Invalid instance ID (error)<br>4. Duplicate AMI name (error)<br>5. Verify new AMI appears in describe-images<br>6. Launch new instance from created AMI | **DONE** |
-| `register-image` | `--name`, `--description`, `--architecture` (x86_64/arm64/i386), `--root-device-name`, `--virtualization-type` (hvm only), `--block-device-mappings` (root with `Ebs.SnapshotId`+optional `VolumeSize`), `--tag-specifications` | `--billing-products`, `--uefi-data` | Backing snapshot must exist in Predastore (`{snapshotId}/metadata.json`); caller must own snapshot or it must be system-owned | Gateway validates name length (3–128), `snap-` prefix, architecture/virtualization values → NATS `ec2.RegisterImage` → daemon checks AMI name uniqueness, reads snapshot metadata, verifies snapshot ownership, builds `viperblock.AMIMetadata` (defaults: `Architecture=x86_64`, `Virtualization=hvm`, `PlatformDetails=Linux/UNIX`, `RootDeviceType=ebs`), writes `{amiId}/config.json`. Pointer-only — never touches block data. `BootMode`, `KernelId`, `RamdiskId`, `TpmSupport`, `ImdsSupport`, `EnaSupport`, `SriovNetSupport`, `ImageLocation`, `paravirtual` virtualization rejected with `InvalidParameterValue`. `VolumeSize` smaller than snapshot rejected. | 1. Register with valid snapshot<br>2. Missing name/snapshot (error)<br>3. Duplicate AMI name (`InvalidAMIName.Duplicate`)<br>4. Snapshot not found (`InvalidSnapshot.NotFound`)<br>5. Cross-account snapshot (`UnauthorizedOperation`)<br>6. Tags from `TagSpecifications` persisted<br>7. Verify registered image in describe-images | **DONE** |
+| `register-image` | `--name`, `--description`, `--architecture` (x86_64/arm64/i386), `--root-device-name`, `--virtualization-type` (hvm only), `--boot-mode` (bios/uefi/uefi-preferred), `--block-device-mappings` (root with `Ebs.SnapshotId`+optional `VolumeSize`), `--tag-specifications` | `--billing-products`, `--uefi-data` | Backing snapshot must exist in Predastore (`{snapshotId}/metadata.json`); caller must own snapshot or it must be system-owned | Gateway validates name length (3–128), `snap-` prefix, architecture/virtualization/boot-mode values → NATS `ec2.RegisterImage` → daemon checks AMI name uniqueness, reads snapshot metadata, verifies snapshot ownership, builds `viperblock.AMIMetadata` (defaults: `Architecture=x86_64`, `Virtualization=hvm`, `PlatformDetails=Linux/UNIX`, `RootDeviceType=ebs`), writes `{amiId}/config.json`. Pointer-only — never touches block data. `BootMode` outside `bios|uefi|uefi-preferred`, `KernelId`, `RamdiskId`, `TpmSupport`, `ImdsSupport`, `EnaSupport`, `SriovNetSupport`, `ImageLocation`, `paravirtual` virtualization rejected with `InvalidParameterValue`. `uefi-preferred` is treated as `uefi` at launch. `VolumeSize` smaller than snapshot rejected. | 1. Register with valid snapshot<br>2. Missing name/snapshot (error)<br>3. Duplicate AMI name (`InvalidAMIName.Duplicate`)<br>4. Snapshot not found (`InvalidSnapshot.NotFound`)<br>5. Cross-account snapshot (`UnauthorizedOperation`)<br>6. Tags from `TagSpecifications` persisted<br>7. Verify registered image in describe-images<br>8. `--boot-mode=uefi` accepted, returned by `describe-images`<br>9. `--boot-mode=invalid` rejected (`InvalidParameterValue`) | **DONE** |
 | `deregister-image` | `--image-id` | `--dry-run` | AMI must exist; caller must own it (system AMIs immutable via this API) | Gateway validates `ami-` prefix → NATS `ec2.DeregisterImage` → daemon hard-deletes `{amiId}/config.json` from Predastore. Backing snapshot is left intact (matches AWS — operators run `delete-snapshot` separately to reclaim block storage). Cross-account/system-AMI mutations rejected with `UnauthorizedOperation`. Re-deregister returns `InvalidAMIID.NotFound` (no tombstone). | 1. Deregister existing AMI<br>2. Deregister non-existent AMI (`InvalidAMIID.NotFound`)<br>3. Re-deregister already-deleted AMI (`InvalidAMIID.NotFound`)<br>4. Cross-account AMI (`UnauthorizedOperation`)<br>5. System AMI (`UnauthorizedOperation`)<br>6. Verify deregistered AMI not in describe-images<br>7. Backing snapshot untouched | **DONE** |
 | `copy-image` | `--source-image-id`, `--source-region` (must equal gateway region), `--name`, `--description`, `--client-token` (accepted, not honoured), `--copy-image-tags`, `--tag-specifications` (image only) | `--encrypted`, `--kms-key-id`, `--destination-outpost-arn`, `--dry-run` | Source AMI must exist and be owned by caller OR be a system AMI; backing snapshot must still exist | Gateway validates `ami-` prefix, name length (3–128), same-region — cross-region / `Encrypted` / `KmsKeyId` / `DestinationOutpostArn` rejected with `InvalidParameterValue`. NATS `ec2.CopyImage` → daemon checks name uniqueness, reads source `ami-xxx/config.json`, visibility-filters cross-account sources as `InvalidAMIID.NotFound`, reads source `{snap-xxx}/metadata.json` (missing or empty SnapshotID → `InvalidAMIID.NotFound`), writes new `snap-yyy/metadata.json` that inherits source `VolumeID` (no block copy), writes new `ami-yyy/config.json` owned by caller. `Description` inherits from source when unset. Tag merge: `CopyImageTags=true` seeds tags from source, explicit image-resource `TagSpecifications` override colliding keys and append new ones; non-image tag specs ignored. | 1. Same-region copy produces new `ami-`/`snap-` pair with distinct IDs<br>2. New snap shares source `VolumeID` (metadata-only)<br>3. New AMI owned by caller, source untouched<br>4. System AMI copied into caller's account<br>5. Cross-account source (`InvalidAMIID.NotFound`)<br>6. Missing source AMI (`InvalidAMIID.NotFound`)<br>7. Orphaned source (missing/empty `SnapshotID`) → `InvalidAMIID.NotFound`<br>8. Duplicate name (`InvalidAMIName.Duplicate`)<br>9. Cross-region / `Encrypted` / `KmsKeyId` rejected (`InvalidParameterValue`)<br>10. `CopyImageTags` true/false tag inheritance semantics | **DONE** |
 | `import-image` | — | `--disk-containers` (Format, Url/S3Bucket+S3Key), `--description`, `--architecture`, `--platform` | S3 bucket with disk image | Download disk image from S3 → convert format (VMDK/VHD/RAW→QCOW2) → create viperblock volume → register as AMI | 1. Import QCOW2 image<br>2. Import RAW image<br>3. Invalid format (error)<br>4. Verify imported image launchable | **NOT STARTED** |
@@ -336,6 +376,7 @@ Requires OVN/OVS (`apt install openvswitch-switch ovn-host`). VPC/Subnet/ENI/SG
 | `authorize-security-group-egress` | `--group-id`, `--ip-permissions` | `--dry-run` | SG must exist | Gateway validates GroupId → NATS `ec2.AuthorizeSecurityGroupEgress` → daemon retrieves SG → appends rules to EgressRules → updates KV with optimistic locking → publishes `vpc.update-sg` event → returns true | 1. Allow HTTPS outbound<br>2. Restrict to specific CIDR<br>3. Missing GroupId (MissingParameter)<br>4. Non-existent SG (InvalidGroup.NotFound) | **DONE** |
 | `revoke-security-group-ingress` | `--group-id`, `--ip-permissions` (matching rules to remove) | `--dry-run` | SG must exist | Gateway validates GroupId → NATS `ec2.RevokeSecurityGroupIngress` → daemon retrieves SG → removes matching rules from IngressRules (set difference) → updates KV with optimistic locking → publishes `vpc.update-sg` event → returns true | 1. Revoke existing rule<br>2. Missing GroupId (MissingParameter)<br>3. Non-existent SG (InvalidGroup.NotFound) | **DONE** |
 | `revoke-security-group-egress` | `--group-id`, `--ip-permissions` (matching rules to remove) | `--dry-run` | SG must exist | Gateway validates GroupId → NATS `ec2.RevokeSecurityGroupEgress` → daemon retrieves SG → removes matching rules from EgressRules (set difference) → updates KV with optimistic locking → publishes `vpc.update-sg` event → returns true | 1. Revoke existing rule<br>2. Missing GroupId (MissingParameter)<br>3. Non-existent SG (InvalidGroup.NotFound) | **DONE** |
+| `describe-security-group-rules` | `--filters` (group-id, security-group-rule-id, tag:\*, tag-key), `--security-group-rule-ids` | `--max-results`, `--next-token`, `--dry-run` | None | Gateway validates `sgr-` format on SecurityGroupRuleIds (nil input = all rules in account) → NATS `ec2.DescribeSecurityGroupRules` → daemon iterates SG records in `spinifex-vpc-security-groups` KV under account → applies filters → emits one `SecurityGroupRule` per ingress (IsEgress=false) and egress (IsEgress=true) rule, each with stable `sgr-` ID, GroupOwnerId=accountID, ReferencedGroupInfo for SG-source rules → returns `InvalidSecurityGroupRuleId.NotFound` if any requested ID missing → v1→v2 KV migration backfills `sgr-` IDs on existing rules at daemon startup; MaxResults/NextToken accepted but ignored | 1. List all rules in account<br>2. Filter by group-id<br>3. Filter by security-group-rule-id<br>4. Non-existent rule ID (InvalidSecurityGroupRuleId.NotFound)<br>5. Malformed sgr- ID at gateway (InvalidParameterValue)<br>6. Tag filters match zero (Tags not yet populated on rules)<br>7. Unknown filter (InvalidParameterValue)<br>8. Migration backfills sgr- IDs on pre-v2 records | **DONE** |
 | `create-internet-gateway` | `--tag-specifications` | `--dry-run` | None | NATS `ec2.CreateInternetGateway` → daemon generates igw-ID → creates IGWRecord in `spinifex-igw` KV bucket with state=available (not attached) → returns InternetGateway with empty Attachments | 1. Create IGW<br>2. Verify in describe-internet-gateways<br>3. Tags applied at creation | **DONE** |
 | `attach-internet-gateway` | `--internet-gateway-id`, `--vpc-id` | `--dry-run` | IGW and VPC must exist | NATS `ec2.AttachInternetGateway` → daemon validates IGW exists and is not already attached (ResourceAlreadyAssociated) → sets VpcId on record → state=attached → publishes `vpc.igw-attach` event to vpcd (creates OVN external switch, gateway port, SNAT rules) | 1. Attach IGW to VPC<br>2. Attach already-attached IGW (ResourceAlreadyAssociated)<br>3. Non-existent IGW (error)<br>4. Missing params (MissingParameter) | **DONE** |
 | `detach-internet-gateway` | `--internet-gateway-id`, `--vpc-id` | `--dry-run` | IGW must be attached to specified VPC | NATS `ec2.DetachInternetGateway` → daemon validates IGW exists and is attached to specified VPC (GatewayNotAttached if wrong VPC) → clears VpcId → state=available → publishes `vpc.igw-detach` event to vpcd (removes OVN resources) | 1. Detach attached IGW<br>2. Detach from wrong VPC (GatewayNotAttached)<br>3. Detach unattached IGW (error) | **DONE** |

docs/DESIGN.md

@@ -16,14 +16,14 @@ Users interact with Spinifex using standard AWS SDKs or the AWS CLI by using the
 
 ```bash
 AWS_PROFILE=spinifex aws ec2 run-instances \
-    --image-id ami-debian12 \
+    --image-id ami-debian13 \
     --instance-type t3.micro \
     --key-name my-keypair
 ```
 
 The AWS SDK formats this as an HTTPS POST with:
 - AWS SigV4 authentication headers
-- EC2 Query Protocol body (`Action=RunInstances&ImageId=ami-debian12&...`)
+- EC2 Query Protocol body (`Action=RunInstances&ImageId=ami-debian13&...`)
 
 ### 2. AWS Gateway
 
@@ -84,6 +84,13 @@ The EC2 handler (`spinifex/gateway/ec2.go`) parses the `Action` parameter and de
 
 ### 4. NATS Messaging
 
+The full service-to-subject map — which service subscribes to what, what
+each one publishes downstream, what each depends on — is captured in
+[`service-interfaces.yaml`](./service-interfaces.yaml). It is the source
+of truth for targeted e2e suite selection (see
+`docs/development/improvements/e2e-targeted-suite-selection.md`) and is
+validated by `make manifest-check`.
+
 The gateway communicates with daemons via NATS request/response. Most calls go through `utils.NATSRequest`, which marshals the input, attaches the account ID as a NATS header, and unmarshals the typed response:
 
 ```go
@@ -297,7 +304,7 @@ sudo systemctl start spinifex.target
 export AWS_PROFILE=spinifex
 
 # Run instance
-aws ec2 run-instances --image-id ami-debian12 --instance-type t3.micro
+aws ec2 run-instances --image-id ami-debian13 --instance-type t3.micro
 
 # List instances (queries all nodes)
 aws ec2 describe-instances

docs/administration/spinifex-admin-cli/README.md

@@ -77,7 +77,7 @@ Prints per-node CPU/memory usage and cluster-wide instance type availability.
 
 ```bash
 spx admin images list
-spx admin images import --name debian-12-arm64
+spx admin images import --name debian-13-arm64
 ```
 
 Catalog imports verify the image against the catalog-declared SHA-256/SHA-512 digest before extraction. Use `--file` to import operator-supplied media (verification skipped — operator is responsible for integrity), or `--force` to re-download after a checksum mismatch.

docs/compute/gpu-passthrough/README.md

@@ -0,0 +1,223 @@
+---
+title: "GPU Passthrough"
+description: "Configure VFIO GPU passthrough on a Spinifex node to expose NVIDIA or AMD GPUs to EC2 instances."
+category: "Compute"
+tags:
+  - gpu
+  - passthrough
+  - vfio
+  - ec2
+resources:
+  - title: "Spinifex Repository"
+    url: "https://github.com/mulgadc/spinifex"
+  - title: "AWS EC2 Accelerated Computing Instance Types"
+    url: "https://docs.aws.amazon.com/ec2/latest/instancetypes/ac.html"
+---
+
+# GPU Passthrough
+
+> Configure VFIO GPU passthrough on a Spinifex node to expose NVIDIA or AMD GPUs to EC2 instances.
+
+> **Note:** GPU passthrough is supported on x86_64 hosts only.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Prerequisites](#prerequisites)
+- [Host Setup](#host-setup)
+- [Enable Passthrough](#enable-passthrough)
+- [Status and Monitoring](#status-and-monitoring)
+- [GPU Instance Types](#gpu-instance-types)
+- [Launching GPU Instances](#launching-gpu-instances)
+- [Disable Passthrough](#disable-passthrough)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Overview
+
+Spinifex supports VFIO-based GPU passthrough, binding NVIDIA and AMD GPUs directly to guest VMs via the `vfio-pci` kernel driver. Once configured, the node exposes GPU-enabled EC2 instance types and the GPU is allocated exclusively to individual instances.
+
+At startup, Spinifex always probes for GPU hardware and surfaces the result in the node banner and in `spx top nodes` — regardless of whether passthrough is enabled. This lets operators verify hardware before activating the feature.
+
+The passthrough state is reflected in the login banner:
+
+![GPU not yet enabled](../../../.github/assets/images/gpu-banner1.png)
+
+---
+
+## Prerequisites
+
+- IOMMU must be supported and enabled in the host BIOS/UEFI (`Intel VT-d` or `AMD-Vi`).
+- An NVIDIA or AMD GPU must be physically installed.
+- The Spinifex node must be running on bare metal (not inside a VM).
+
+---
+
+## Host Setup
+
+Run the one-time host configuration as root. This command is idempotent — safe to re-run after a reboot.
+
+```bash
+sudo spx admin gpu setup
+```
+
+What it does:
+
+1. Detects installed GPUs and their PCI IDs.
+2. Enables IOMMU in GRUB (`intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt`).
+3. Writes the vfio udev rule (`/etc/udev/rules.d/99-spinifex-vfio.rules`).
+4. Blacklists `nouveau` (NVIDIA) and `amdgpu` (AMD) so the host kernel does not claim the device.
+5. Configures `vfio-pci` early binding in `/etc/modprobe.d/vfio-pci.conf`.
+6. Adds vfio modules to initramfs.
+
+If any change requires a reboot, the command prints instructions and exits:
+
+```
+Setup complete — reboot required.
+  sudo reboot
+  Then run: sudo spx admin gpu enable
+```
+
+---
+
+## Enable Passthrough
+
+After setup and reboot, enable GPU passthrough:
+
+```bash
+sudo spx admin gpu enable
+```
+
+The command writes the configuration, signals the daemon (`SIGHUP`), and waits up to 30 seconds for the daemon to confirm the new state. On success it prints the current GPU status.
+
+
+![GPU enabled](../../../.github/assets/images/gpu-enabled.png)
+
+---
+
+## Status and Monitoring
+
+### Per-node status
+
+```bash
+spx admin gpu status
+# or for a specific node in the cluster:
+spx admin gpu status --node <node-name>
+```
+
+Output includes hardware detected, IOMMU state, vfio-pci binding, passthrough enabled/disabled, GPU pool allocation (`allocated/total`), and the GPU-capable EC2 instance types available on that node.
+
+### Cluster view
+
+`spx top nodes` includes a `GPU (used/total)` column:
+
+| Value | Meaning |
+|-------|---------|
+| `1/2` | Passthrough active — 1 of 2 GPUs allocated |
+| `0/1*` | Node has a GPU but passthrough is not enabled |
+| `-` | No GPU hardware detected |
+
+```bash
+spx top nodes
+```
+
+---
+
+## GPU Instance Types
+
+GPU instance types are derived from detected hardware.
+
+To list available GPU instance types on the current node:
+
+```bash
+aws ec2 describe-instance-types \
+  --filters Name=instance-type,Values=g*
+```
+
+---
+
+## Launching GPU Instances
+
+Import the GPU-enabled AMI before launching:
+
+```bash
+spx admin images import --name ubuntu-gpu-nvidia   # NVIDIA hosts
+spx admin images import --name ubuntu-gpu-amd      # AMD hosts
+```
+
+Launch a GPU instance:
+
+```bash
+aws ec2 run-instances \
+  --image-id $GPU_AMI \
+  --instance-type $INSTANCE_TYPE \
+  --key-name spinifex-key
+```
+
+To verify the GPU is visible from inside the instance, SSH in and run:
+
+```bash
+# NVIDIA
+nvidia-smi
+
+# AMD
+rocm-smi
+```
+
+---
+
+## Disable Passthrough
+
+Passthrough cannot be disabled while GPU instances are running. Terminate all GPU instances first, then:
+
+```bash
+sudo spx admin gpu disable
+```
+
+The command signals the daemon and waits for confirmation. The GPU is released back to the host kernel on the next reboot (the vfio-pci binding persists until `setup` is re-run without the blacklists).
+
+---
+
+## Troubleshooting
+
+### IOMMU not active after reboot
+
+Verify IOMMU is enabled in BIOS/UEFI (`Intel VT-d` or `AMD-Vi`). Then check GRUB picked up the parameters:
+
+```bash
+cat /proc/cmdline | grep iommu
+ls /sys/kernel/iommu_groups/
+```
+
+If `/sys/kernel/iommu_groups/` is empty, IOMMU is not active. Re-run `sudo spx admin gpu setup` after enabling it in firmware.
+
+### GPU bound to native driver instead of vfio-pci
+
+```bash
+lspci -k | grep -A3 -i "vga\|3d\|display"
+```
+
+If the driver is `amdgpu` or `nvidia` rather than `vfio-pci`, the blacklist or early binding did not take effect. Re-run setup and reboot:
+
+```bash
+sudo spx admin gpu setup
+sudo reboot
+```
+
+### `spx admin gpu enable` fails with "prerequisites not met"
+
+Run `setup` first to configure the host, then retry `enable`:
+
+```bash
+sudo spx admin gpu setup
+sudo spx admin gpu enable
+```
+
+### Daemon does not confirm within 30 seconds
+
+Check the daemon log for errors:
+
+```bash
+journalctl -u spinifex-daemon -n 50
+```

docs/compute/launching-instances/README.md

@@ -157,7 +157,7 @@ If the AMI is for a different architecture (e.g. arm64 on an x86_64 host), impor
 
 ```bash
 spx admin images list
-spx admin images import --name debian-12-x86_64
+spx admin images import --name debian-13-x86_64
 ```
 
 ### Cannot SSH Into Instance

docs/compute/vpc-networking/README.md

@@ -573,6 +573,54 @@ external_interface = "br-wan"
 DHCP server when VMs launch and releases them on terminate. Requires `dhclient`
 on the host (`apt install isc-dhcp-client`).
 
+## Host-Local Subnet (No Upstream Router)
+
+```
+Network: 192.168.10.0/24 — host-local, reachable from the host only
+Host WAN: 198.51.100.10/24 on br-wan (existing address — unchanged)
+Gateway: 192.168.10.1 — second address added to br-wan
+```
+
+Add the VM pool gateway as a second address on `br-wan` alongside the existing WAN
+IP. The host acts as the gateway for the pool — no upstream router or DHCP server
+needed for this range.
+
+```yaml
+# /etc/netplan/…
+bridges:
+  br-wan:
+    addresses:
+      - 192.168.10.1/24      # VM pool gateway — host-local
+      - 198.51.100.10/24     # existing WAN IP — unchanged
+    routes:
+      - to: default
+        via: 198.51.100.1
+```
+
+```toml
+[network]
+external_mode = "pool"
+
+[[network.external_pools]]
+name        = "wan"
+source      = "static"         # required — no upstream DHCP for this range
+range_start = "192.168.10.2"
+range_end   = "192.168.10.100"
+gateway     = "192.168.10.1"   # second address on br-wan
+prefix_len  = 24
+dns_servers = ["8.8.8.8"]
+```
+
+**Setup:** Apply with `sudo netplan apply`. VMs are reachable from the host at
+`192.168.10.x`. For internet access through the host's WAN interface:
+
+```bash
+sysctl -w net.ipv4.ip_forward=1
+iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -o br-wan -j MASQUERADE
+```
+
+Persist via `/etc/sysctl.d/99-ip-forward.conf` and `netfilter-persistent save`.
+
 ## Datacenter / Colo (ISP Block)
 
 ```

docs/getting-started/install-airgapped/README.md

@@ -57,6 +57,7 @@ sudo apt update
 sudo apt install --download-only -y \
   nbdkit nbdkit-plugin-dev pkg-config \
   qemu-system-x86 qemu-utils qemu-kvm \
+  ovmf qemu-efi-aarch64 \
   libvirt-daemon-system libvirt-clients libvirt-dev \
   ovn-central ovn-host openvswitch-switch \
   dhcpcd-base make gcc jq curl iproute2 netcat-openbsd \
@@ -74,8 +75,8 @@ Mirror the cloud image you intend to run as guest VMs:
 
 ```bash
 mkdir -p images
-curl -fsSL "https://cdimage.debian.org/cdimage/cloud/bookworm/latest/debian-12-genericcloud-amd64.qcow2" \
-  -o images/debian-12-amd64.qcow2
+curl -fsSL "https://cloud.debian.org/images/cloud/trixie/20260518-2482/debian-13-genericcloud-amd64-20260518-2482.qcow2" \
+  -o images/debian-13-amd64.qcow2
 ```
 
 ## Step 3. Assemble Transfer Media
@@ -140,8 +141,8 @@ sudo systemctl start spinifex.target
 Register the pre-staged image with Spinifex:
 
 ```bash
-sudo spx admin images import --file /mnt/usb/images/debian-12-amd64.qcow2 \
-  --distro debian --version 12 --arch x86_64
+sudo spx admin images import --file /mnt/usb/images/debian-13-amd64.qcow2 \
+  --distro debian --version 13 --arch x86_64 --boot-mode uefi
 ```
 
 ## Step 9. Verify
@@ -183,4 +184,4 @@ sudo INSTALL_SPINIFEX_TARBALL=/mnt/usb/tarball/spinifex-...tar.gz \
 
 ### Image Import Fails
 
-`spx admin images import --file` requires the distro/version/arch flags so the image registers in the catalogue. If the import succeeds but `aws ec2 describe-images` returns empty, check `journalctl -u spinifex-daemon -f` for predastore upload errors.
+`spx admin images import --file` requires the distro/version/arch/boot-mode flags so the image registers in the catalogue. If the import succeeds but `aws ec2 describe-images` returns empty, check `journalctl -u spinifex-daemon -f` for predastore upload errors.

docs/getting-started/install-multi-node/README.md

@@ -22,6 +22,7 @@ resources:
 ## Table of Contents
 
 - [Overview](#overview)
+- [Prerequisites](#prerequisites)
 - [Instructions](#instructions)
 - [Troubleshooting](#troubleshooting)
 
@@ -37,6 +38,22 @@ A Spinifex cluster distributes services across multiple servers for high availab
 - UDP port 6081 open between hosts (Geneve tunnels)
 - TCP ports 4222, 4248, 6641, 6642 open between hosts (NATS, OVN)
 
+## Prerequisites
+
+> [!IMPORTANT]
+> **Prerequisite — WAN bridge required on every node.**
+>
+> Before running the installer on any server, that server's WAN interface **must** already be enslaved to a Linux bridge named `br-wan`. The host IP, default route, and DHCP must all live on the bridge — not on the bare NIC.
+>
+> The bootstrap installer does **not** create this bridge for you yet. Running it on a host whose default route is still on a bare NIC will leave the install in a non-working state. Auto-provisioning of `br-wan` will land in the next release.
+>
+> **Verify on every node before continuing:**
+>
+> - `ip -br link show br-wan` — bridge exists and is `UP`
+> - `ip route` — default route's `dev` is `br-wan`
+>
+> **Setup references:** [VPC Networking → Bridge Setup](/docs/vpc-networking#bridge-setup-physical-network-wiring) for the topology.
+
 ## Instructions
 
 ## Step 1. Install Spinifex on Each Server

docs/getting-started/install-source/README.md

@@ -31,8 +31,8 @@ This guide builds Spinifex from source. For production deployments, the [binary
 
 **Requirements:**
 
-- Ubuntu 22.04+ / Debian 12+
-- Go 1.26.2+
+- Ubuntu 26.04 / Debian 13
+- Go 1.26.3+
 - GCC, make, pkg-config
 - QEMU/KVM, OVN/Open vSwitch, AWS CLI v2, dhcpcd-base
 

docs/getting-started/install/README.md

@@ -22,6 +22,7 @@ resources:
 ## Table of Contents
 
 - [Overview](#overview)
+- [Prerequisites](#prerequisites)
 - [Instructions](#instructions)
 - [Troubleshooting](#troubleshooting)
 
@@ -35,8 +36,8 @@ This guide installs Spinifex on a single server using the binary installer. For
 
 **Supported Operating Systems:**
 
-- Ubuntu 22.04 / 24.04 / 25.10
-- Debian 12 / 13
+- Ubuntu 26.04
+- Debian 13
 
 **What Gets Installed:**
 
@@ -47,6 +48,22 @@ This guide installs Spinifex on a single server using the binary installer. For
 - Viperblock (EBS-compatible block storage)
 - AWS CLI v2
 
+## Prerequisites
+
+> [!IMPORTANT]
+> **Prerequisite — WAN bridge required.**
+>
+> Before running the installer, the host's WAN interface **must** already be enslaved to a Linux bridge named `br-wan`. The host IP, default route, and DHCP must all live on the bridge — not on the bare NIC.
+>
+> The bootstrap installer does **not** create this bridge for you yet. Running it on a host whose default route is still on a bare NIC will leave the install in a non-working state. Auto-provisioning of `br-wan` will land in the next release.
+>
+> **Verify before continuing:**
+>
+> - `ip -br link show br-wan` — bridge exists and is `UP`
+> - `ip route` — default route's `dev` is `br-wan`
+>
+> **Setup references:** [VPC Networking → Bridge Setup](/docs/vpc-networking#bridge-setup-physical-network-wiring) for the topology.
+
 ## Instructions
 
 ## Step 1. Install Spinifex

docs/getting-started/setting-up-your-cluster/README.md

@@ -58,23 +58,22 @@ spx admin images list
 
 ```
 NAME                    | DISTRO | VERSION | ARCH   | BOOT
-debian-12-arm64         | debian | 12      | arm64  | bios
-debian-12-x86_64        | debian | 12      | x86_64 | bios
-lb-alpine-3.21.6-x86_64 | alpine | 3.21.6  | x86_64 | bios
-ubuntu-24.04-arm64      | ubuntu | 24.04   | arm64  | bios
-ubuntu-24.04-x86_64     | ubuntu | 24.04   | x86_64 | bios
+debian-13-arm64         | debian | 13      | arm64  | uefi
+debian-13-x86_64        | debian | 13      | x86_64 | uefi
+ubuntu-26.04-arm64      | ubuntu | 26.04   | arm64  | uefi
+ubuntu-26.04-x86_64     | ubuntu | 26.04   | x86_64 | uefi
 ```
 
 Import an image:
 
 ```bash
-spx admin images import --name ubuntu-24.04-x86_64
+spx admin images import --name ubuntu-26.04-x86_64
 ```
 
 Or import a local image file:
 
 ```bash
-spx admin images import --file ~/images/ubuntu-24.04.img --distro ubuntu --version 24.04 --arch x86_64
+spx admin images import --file ~/images/ubuntu-26.04.img --distro ubuntu --version 26.04 --arch x86_64 --boot-mode uefi
 ```
 
 Verify the import and note the AMI ID:
@@ -173,7 +172,7 @@ Launch an instance in the public subnet, note this will select an instance type
 > **Note:** Replace `AMI_NAME` with the previously imported image above with the `ami-` prefix.
 
 ```bash
-AMI_NAME="ami-ubuntu-24.04-x86_64"
+AMI_NAME="ami-ubuntu-26.04-x86_64"
 
 AMI_ID=$(aws ec2 describe-images \
   --filters "Name=name,Values=$AMI_NAME" \

docs/security/malware-protection/README.md

@@ -73,7 +73,7 @@ All Spinifex nodes must include the following in their malware protection scope.
 |------|----------|
 | `/usr/local/bin/spx` | Spinifex CLI and service entry point. All `spinifex-*.service` units invoke `spx service <name> start`, so every service binary is reached through this one path. |
 | `/usr/bin/ovn-*`, `/usr/bin/ovs-*` | OVN / OVS control-plane and datapath binaries (installed via `apt` from the `ovn-central`, `ovn-host`, `openvswitch-switch` packages). |
-| `/usr/lib/<arch>-linux-gnu/nbdkit/plugins/nbdkit-viperblock-plugin.so` | Viperblock NBD plugin loaded by `nbdkit` to expose EBS volumes as block devices. The exact path comes from `nbdkit --dump-config` (`plugindir=`); on Debian 12 amd64 it is `/usr/lib/x86_64-linux-gnu/nbdkit/plugins/`. |
+| `/usr/lib/<arch>-linux-gnu/nbdkit/plugins/nbdkit-viperblock-plugin.so` | Viperblock NBD plugin loaded by `nbdkit` to expose EBS volumes as block devices. The exact path comes from `nbdkit --dump-config` (`plugindir=`); on Debian 13 amd64 it is `/usr/lib/x86_64-linux-gnu/nbdkit/plugins/`. |
 | `/usr/local/share/ca-certificates/spinifex-ca.crt` | Cluster CA cert installed into the system trust store |
 
 ### 1.2 Configuration and Secrets
@@ -157,7 +157,7 @@ All scan runs — scheduled and real-time — must produce a log entry that incl
 
 ## 4. Reference Configurations
 
-The configurations below are reference baselines. They are known to meet the CMMC L1 objectives on a standard Debian 12 Spinifex node; they are not the only valid implementation. Operators running a commercial EDR should treat [§1](#1-designated-locations-sil1-3142-a) and [§3](#3-scan-schedule-sil1-3145) as the contract and configure their chosen product to match.
+The configurations below are reference baselines. They are known to meet the CMMC L1 objectives on a standard Debian 13 Spinifex node; they are not the only valid implementation. Operators running a commercial EDR should treat [§1](#1-designated-locations-sil1-3142-a) and [§3](#3-scan-schedule-sil1-3145) as the contract and configure their chosen product to match.
 
 ### 4.1 ClamAV (signature-based AV)
 

docs/security/media-sanitization/README.md

@@ -0,0 +1,258 @@
+---
+title: "Media Sanitization and Disposal"
+description: "Operator guide for sanitizing and disposing of storage media used by Spinifex nodes"
+category: "Security"
+sections:
+  - overview
+tags:
+  - security
+  - compliance
+  - cmmc
+  - media
+  - sanitization
+  - disposal
+  - decommissioning
+resources:
+  - title: "NIST SP 800-88 Rev 1 — Guidelines for Media Sanitization"
+    url: "https://csrc.nist.gov/pubs/sp/800/88/r1/final"
+  - title: "NIST SP 800-171 Rev 3"
+    url: "https://csrc.nist.gov/pubs/sp/800/171/r3/final"
+  - title: "CMMC Level 1 Self-Assessment Guide v2.0"
+    url: "https://dodcio.defense.gov/CMMC/Documentation"
+  - title: "ATA/ATAPI Command Set — Secure Erase"
+    url: "https://www.t13.org"
+  - title: "NVM Express — Sanitize Command"
+    url: "https://nvmexpress.org/specifications"
+---
+
+# Media Sanitization and Disposal
+
+> Operator guide for sanitizing and disposing of storage media used by Spinifex nodes
+
+## Table of Contents
+
+- [Overview](#overview)
+- [CMMC Practices Covered](#cmmc-practices-covered)
+- [Approach](#approach)
+- [1. Media in Scope](#1-media-in-scope)
+- [2. Sanitization Method Selection](#2-sanitization-method-selection)
+- [3. Volume-Level Sanitization — Before Reuse (MP.L1-3.8.3 [b])](#3-volume-level-sanitization--before-reuse-mpl1-383-b)
+- [4. Whole-Drive and Node Decommissioning — Before Disposal (MP.L1-3.8.3 [a])](#4-whole-drive-and-node-decommissioning--before-disposal-mpl1-383-a)
+- [5. Key Destruction](#5-key-destruction)
+- [6. Removable and Backup Media](#6-removable-and-backup-media)
+- [7. Evidence and Record Keeping](#7-evidence-and-record-keeping)
+- [8. Operator Checklist](#8-operator-checklist)
+
+---
+
+## Overview
+
+**Audience:** Operators decommissioning, reassigning, returning under warranty, or disposing of hardware that has been used to run Spinifex nodes in environments subject to CMMC Level 1, or any site that requires documented media sanitization.
+
+**Scope:** Every storage medium that has, or may have, held Federal Contract Information (FCI) as part of a Spinifex deployment — node system disks, Viperblock WAL/chunk disks, Predastore object stores, NATS JetStream disks, removable master-key tokens, backup media, and any loose drives that previously occupied a Spinifex role. Guest VM internal media (disks as seen from inside a tenant VM) is the workload owner's responsibility and out of scope.
+
+## CMMC Practices Covered
+
+| Practice | Title | Objective |
+|----------|-------|-----------|
+| MP.L1-3.8.3 | Media Disposal | [a] System media containing FCI is sanitized or destroyed before disposal. [b] System media containing FCI is sanitized before it is released for reuse. |
+
+## Approach
+
+NIST SP 800-88 Rev 1 defines three sanitization categories — **Clear**, **Purge**, and **Destroy** — chosen by the confidentiality of the data and whether the medium will leave the operator's control. Spinifex discharges MP.L1-3.8.3 along two tracks:
+
+1. **Volume-level sanitization** for EBS volumes and VM disks returning to the free pool. Handled in-platform by **cryptographic erase** — the per-volume data encryption key (DEK) is deleted, rendering the ciphertext in Predastore S3 irrecoverable. This is the NIST SP 800-88 **Purge**-level method for encrypted media.
+2. **Whole-drive sanitization** for physical media leaving a Spinifex node — drive replacement, node retirement, warranty return, resale, or destruction. The operator runs the sanitization; Spinifex does not and cannot reach the firmware commands required. This guide prescribes the method per media type and the records to keep.
+
+The key principle: every piece of media that ever held plaintext FCI (or keys that wrapped FCI ciphertext) must be sanitized to at least **Purge** before release for reuse, and to **Purge** or **Destroy** before leaving operator control. When in doubt, Destroy.
+
+## 1. Media in Scope
+
+Any medium in any of these roles on a Spinifex node has held, or may have held, FCI:
+
+| Medium | Typical hardware | Contents |
+|--------|------------------|----------|
+| Node system disk | NVMe/SATA SSD | `/etc/spinifex/master.key`, cluster CA key, per-node TLS keys, service configs, logs, journal, swap. |
+| Viperblock WAL device | NVMe/SATA SSD | Plaintext of in-flight block writes before chunking. High-churn, short residency, but FCI lands here. |
+| Viperblock chunk cache / local backing | NVMe/SATA SSD or HDD | Encrypted volume chunks. |
+| Predastore object store | NVMe/SATA SSD or HDD | All S3 object data — AMIs, snapshots, user-uploaded artifacts, IAM state files, tenant data. |
+| NATS JetStream disk | NVMe/SATA SSD | IAM NATS token, cluster metadata, pending job state, NATS KV entries (including wrapped DEKs). |
+| Removable master-key media | USB flash token, HSM, smart card | Master encryption key (Phase 4 USB-mount path). Destroying this sanitizes every DEK-encrypted volume cluster-wide. |
+| Backup media | LTO tape, removable HDD/SSD, off-site cloud backup | Any of the above, point-in-time. |
+| BMC / iDRAC / iLO storage | Embedded flash | Console recordings, SEL logs, BMC credentials, cached operator certificates. Sanitize via the BMC "Reset to defaults" / "Erase user data" command before disposal. |
+| Switch / router config storage | Embedded flash | Cluster VLAN, management IPs, ACLs. Operator network kit, out of Spinifex scope but noted for completeness. |
+| Optical / write-once media | DVD/BD-R | If used to transport keys or images. Always Destroy. |
+
+Any drive whose history cannot be traced — pulled from a spares bin, recovered from a failed host, found unlabelled — must be treated as though it held FCI.
+
+## 2. Sanitization Method Selection
+
+The method depends on (a) media type and (b) whether the media remains inside the protected boundary after sanitization.
+
+| Scenario | Minimum method | Reference |
+|----------|----------------|-----------|
+| Encrypted volume returning to free pool, DEK deletable | **Cryptographic Erase** (Purge) via DEK deletion | [§3](#3-volume-level-sanitization--before-reuse-mpl1-383-b) |
+| SSD / NVMe leaving the node | **Cryptographic Erase** + **Purge** (SANITIZE block-erase or ATA Secure Erase) | [§4.1](#41-nvme-ssd) |
+| Magnetic HDD leaving the node, operational | **Purge** (single-pass overwrite with verify, or ATA Secure Erase on drives that support it) | [§4.3](#43-hdd) |
+| Magnetic HDD leaving the node, faulty (not writable) | **Destroy** (degauss followed by shred/incinerate) | [§4.3](#43-hdd) |
+| Any media leaving operator control (resale, warranty, disposal) | **Destroy** if residual confidentiality concern remains after Purge; otherwise Purge + third-party attestation | [§4.4](#44-destruction) |
+| Tape | **Destroy** (degauss + shred) — overwrite is not reliable on LTO with compression | [§6](#6-removable-and-backup-media) |
+| Optical write-once | **Destroy** (shred/incinerate) | [§6](#6-removable-and-backup-media) |
+| BMC / switch flash | Vendor "factory reset + erase user data"; Destroy chip if procedure is not available | [§1](#1-media-in-scope) |
+
+**Cryptographic erase is only valid when** the key was generated and handled under modern cryptographic hygiene (AES-256 or equivalent, key never exfiltrated, key store itself sanitized) and the encrypted data was not also written anywhere in plaintext. For Spinifex volumes these conditions hold.
+
+**Self-encrypting drives (SEDs)** support a single-command cryptographic erase via the TCG Opal "RevertSP" / `sedutil-cli --revertNoErase` or the ATA `SECURITY ERASE UNIT ENHANCED` that triggers the on-drive MEK rotation. When using an SED as a node system or data disk, record the SED model and method in the decommissioning procedure.
+
+## 3. Volume-Level Sanitization — Before Reuse (MP.L1-3.8.3 [b])
+
+Applies to EBS volumes, AMIs, and snapshots whose underlying storage will be reused for a different tenant or workload.
+
+### 3.1 Cryptographic Erase via DEK Deletion
+
+Deletion is sanitization:
+
+- `DeleteVolume` removes the wrapped DEK from NATS KV. The chunks in Predastore S3 remain but are unrecoverable ciphertext. No further operator action is required.
+- `TerminateInstances` with the default `DeleteOnTermination=true` on root volumes performs the same cryptographic erase for the terminated instance's root.
+- `DeleteSnapshot` removes the snapshot's DEK wrapper; the underlying chunks become irrecoverable ciphertext.
+
+This is the sanitization-before-reuse path for every in-platform object. Operators do not need to take separate action — the API call is the sanitization.
+
+### 3.2 What Happens to the Physical Storage
+
+Cryptographic erase leaves ciphertext chunks in Predastore. These chunks are eventually overwritten as new volumes reuse the space. For CMMC purposes, the ciphertext without the DEK satisfies Purge. The underlying drive still requires whole-drive sanitization when it eventually leaves the cluster — see [§4](#4-whole-drive-and-node-decommissioning--before-disposal-mpl1-383-a).
+
+## 4. Whole-Drive and Node Decommissioning — Before Disposal (MP.L1-3.8.3 [a])
+
+Applies when a physical drive — or a whole node — leaves the Spinifex cluster: drive swap, node retirement, warranty return, resale, recycling, destruction.
+
+### 4.1 NVMe SSD
+
+Preferred: the NVMe **SANITIZE** command with the **Block Erase** action. Supported on most enterprise NVMe drives.
+
+```bash
+# Confirm support
+nvme id-ctrl /dev/nvme0 | grep -i sanicap
+
+# Purge — block erase
+nvme sanitize /dev/nvme0 --sanact=2
+
+# Poll until complete
+nvme sanitize-log /dev/nvme0
+```
+
+Fallback for drives without SANITIZE support: NVMe **Format** with Secure Erase set to 1 (`--ses=1`), or vendor tool. Cryptographic Erase (`--sanact=4`) is acceptable for SEDs whose encryption posture is documented; otherwise Block Erase is the safer default.
+
+Boot media: NVMe targets must be the non-boot drive when running from the OS. For the node system disk, boot a sanitization-purpose live USB (e.g. [PartedMagic](https://partedmagic.com/), the vendor's diagnostic ISO) or pull the drive and sanitize in a dedicated sanitization workstation.
+
+### 4.2 SATA/SAS SSD
+
+Preferred: **ATA Secure Erase (Enhanced)** via `hdparm`:
+
+```bash
+# Confirm support and not frozen
+hdparm -I /dev/sdX | grep -A1 "Security"
+
+# If frozen, power-cycle (suspend/resume or hot-swap) without unplugging the OS disk
+
+# Set password and issue enhanced erase
+hdparm --user-master u --security-set-pass p /dev/sdX
+hdparm --user-master u --security-erase-enhanced p /dev/sdX
+```
+
+Fallback: vendor Secure Erase utility, or `blkdiscard --secure /dev/sdX` on drives that report `TRIM deterministic + RZAT`. A single-pass overwrite with `shred -v -n 1 /dev/sdX` is **not** sufficient for SSDs because of wear-levelling reserve blocks — use Secure Erase or Destroy.
+
+### 4.3 HDD
+
+Preferred: **ATA Secure Erase** (same `hdparm` sequence as [§4.2](#42-satasas-ssd)). Most modern HDDs support it.
+
+Fallback: single-pass overwrite with verify:
+
+```bash
+shred -v -n 1 -z /dev/sdX
+# or
+dd if=/dev/zero of=/dev/sdX bs=1M status=progress && \
+  cmp /dev/zero /dev/sdX          # expect "EOF on /dev/zero" only
+```
+
+For HDDs that fail or refuse sanitization, or that held high-sensitivity data, **degauss then Destroy** — degaussing alone renders a modern HDD non-functional, so there is no reuse path after degauss. Degaussing requires an NSA/CSS-listed degausser appropriate to the drive's coercivity.
+
+### 4.4 Destruction
+
+When a drive cannot be sanitized (failed, SED with no working password, sanitize aborts) or when leaving operator control with any residual confidentiality concern, Destroy:
+
+- **SSD/NVMe:** shred to ≤2 mm particle size (NSA/CSS EPL-listed shredder) or incinerate. Crushing alone is not sufficient for modern flash — chips can survive.
+- **HDD:** degauss (on drives with magnetic platters) followed by shredding, drilling multiple holes through the platters and the head assembly, or incineration.
+- **Tape, optical:** shredding.
+- **Chain of custody:** transport destroyed media inside tamper-evident containers. The destruction record ([§7](#7-evidence-and-record-keeping)) must name the destruction method, destruction vendor (if third-party), and operator witness.
+
+Third-party destruction vendors must provide a certificate of destruction listing every drive serial. Reconcile against the asset register before the destruction record is closed.
+
+### 4.5 Node Decommissioning Runbook
+
+When retiring a whole node:
+
+1. **Drain** — migrate or terminate all instances hosted on the node; confirm Predastore and Viperblock roles are reassigned if the node held one. Volume DEKs of terminated instances are already cryptographically erased per [§3.1](#31-cryptographic-erase-via-dek-deletion).
+2. **Deregister** — `bd`-linked change ticket; the operator removes the node from the cluster (`spx admin node remove`) so NATS routes, OVN chassis, and predastore distributed membership drop it.
+3. **Stop services and unmount** — `systemctl stop 'spinifex-*'`, `nbdkit` sessions, OVN agent. Confirm `/run/spinifex/nbd/` is empty.
+4. **Destroy on-disk keys** — see [§5](#5-key-destruction). This is the single most important step: sanitization of volume data reduces to sanitization of keys once data is encrypted.
+5. **Sanitize each drive** by media type per [§4.1](#41-nvme-ssd) – [§4.3](#43-hdd). Every drive bay is sanitized, including any unpopulated cache or WAL devices.
+6. **Verify** — capture the output of the sanitize/erase command (exit code + sanitize-log for NVMe, `hdparm -I` "not enabled, not locked" for ATA, overwrite verify for HDD overwrite) into the decommissioning record.
+7. **Label** drives "Sanitized — <method> — <date> — <operator>" before they leave the rack or enter a transit container. Unlabelled drives are treated as unsanitized if reintroduced.
+8. **Record** per [§7](#7-evidence-and-record-keeping).
+
+## 5. Key Destruction
+
+Cryptographic erase relies on the key being unrecoverable. A drive that previously held `/etc/spinifex/master.key` must have that key destroyed — otherwise an attacker with both the drive image and the key recovers the cleartext even after "erase".
+
+| Key | Location | Destruction |
+|-----|----------|-------------|
+| `/etc/spinifex/master.key` | Node system disk (every node) | Overwrite the file (`shred -u /etc/spinifex/master.key`) before the drive is sanitized, in addition to drive sanitization. Once the master key is gone cluster-wide, every DEK it wrapped is effectively destroyed. |
+| Cluster CA private key (`/etc/spinifex/ca.key`) | Leader node system disk | Same treatment. Loss of the CA key does not sanitize data, but prevents impersonation of the cluster identity post-disposal. |
+| Per-node TLS private key (`/etc/spinifex/server.key`) | Each node system disk | `shred -u` before drive sanitization. |
+| Wrapped DEKs in NATS KV | `/var/lib/spinifex/nats/` on NATS-carrying nodes | Deleted via the `DeleteVolume` API; the KV record is compacted. Sanitization of the NATS JetStream disk per [§4](#4-whole-drive-and-node-decommissioning--before-disposal-mpl1-383-a) ensures no recoverable copies. |
+| Removable master-key media (Phase 4 USB) | USB flash token | Physically destroy the token when the cluster is decommissioned; alternatively retain inside a secured location as evidence of cluster-wide cryptographic erase. Do not reuse for another cluster without full sanitization (see [§6](#6-removable-and-backup-media)). |
+| Backup-copy keys | Any off-site or escrow copy | Destroy simultaneously with the primary. A surviving backup defeats the cryptographic erase. |
+
+If even one copy of the master key survives and the encrypted drives are recoverable, cryptographic erase has **not** been achieved. Track every key copy in the device register ([Physical Security Guide §5](/docs/security/physical-security-guide#5-manage-physical-access-devices-pel1-3105)).
+
+## 6. Removable and Backup Media
+
+| Media | Sanitization |
+|-------|--------------|
+| USB flash (master-key token, key-transport USB, image-import USB) | Destroy. The price of a replacement does not justify the risk of partial wear-levelled sanitization. Record the serial in the device register and the destruction in the decommissioning log. |
+| External HDD/SSD (backup) | Treat as [§4.2](#42-satasas-ssd) / [§4.3](#43-hdd). |
+| LTO tape | Degauss with a tape-rated degausser, then shred. Overwrite is not reliable under LTO hardware compression and is not accepted by 800-88 for tape Purge. |
+| Optical (CD/DVD/BD-R) | Shred or incinerate. |
+| Off-site cloud backup | Issue the cloud provider's delete-and-purge API; retain the provider's deletion-confirmation receipt. For cryptographically wrapped backups, destroy the wrapping key cluster-side and retain that deletion as evidence. |
+| Printed material (console photos, recovery codes, admin handover sheets) | Cross-cut shred. |
+
+Any removable medium that has entered the protected boundary and been written to must be tracked in the device register (see [Physical Security Guide §5](/docs/security/physical-security-guide#5-manage-physical-access-devices-pel1-3105)) and accounted for at decommissioning.
+
+## 7. Evidence and Record Keeping
+
+For CMMC assessment, retain the following for at least three years (or longer where contract policy requires):
+
+- **Asset register** listing every drive and removable medium by serial number, media type, role in the cluster (system / WAL / chunk / predastore / NATS / backup), and in-service / decommissioned status.
+- **Decommissioning record** per drive or node with: date, operator, sanitization method, command output or tool report, verification result, final disposition (reuse / return / destroy), destination (e.g. "warranty RMA #12345", "shredder vendor X, CoD #6789").
+- **Certificates of destruction** from third-party vendors, reconciled against the asset register.
+- **Key-destruction record** confirming `master.key`, CA key, and per-node keys were shredded before drive sanitization.
+- **Exceptions log** — faulty drives that could not be sanitized and were destroyed instead; any sanitize failures and their resolution.
+- **Annual attestation** from the operator confirming the procedures in this guide operated for the prior 12 months, signed by the named security owner.
+
+Cross-reference each decommissioning record to the change ticket that drove the retirement, so drive disposal can be audited against cluster state change.
+
+## 8. Operator Checklist
+
+Use this list to confirm a Spinifex deployment meets MP.L1-3.8.3:
+
+- Asset register enumerates every drive and removable medium by serial, media type, and role.
+- Every volume-producing path uses encryption so that `DeleteVolume` is sanitization.
+- Node-decommissioning runbook exists and matches [§4.5](#45-node-decommissioning-runbook), including the drain → deregister → key-destroy → drive-sanitize sequence.
+- Sanitization method is pre-selected per media type per [§2](#2-sanitization-method-selection); tool availability (hdparm, nvme-cli, vendor ISOs, degausser, shredder or destruction vendor) is verified before a decommissioning begins.
+- `master.key`, CA key, and per-node TLS keys are shredded before the drive holding them is sanitized.
+- Every backup copy of any key is destroyed at the same time as the primary; surviving backup copies are tracked and accounted for.
+- Removable media (USB, tape, optical) is destroyed rather than erased unless the medium is an SED with documented cryptographic erase.
+- Decommissioning records capture method, tool output, verification, final disposition, and operator; certificates of destruction are reconciled against the asset register.
+- Drives awaiting sanitization or awaiting pickup by a destruction vendor are held inside the physical protection boundary ([Physical Security Guide §1](/docs/security/physical-security-guide#1-protected-assets)) with chain of custody logged.
+- System security plan references this guide and names the sanitization tooling, destruction vendor (if any), retention period for records, and the security owner attesting annually.

docs/security/network-connections/README.md

@@ -105,8 +105,8 @@ Spinifex nodes initiate a small, fixed set of outbound connections.
 
 | Destination | Purpose | Protocol | Verification |
 |-------------|---------|----------|--------------|
-| `https://cdimage.debian.org/cdimage/cloud/bookworm/latest` | Debian 12 cloud image download | HTTPS | TLS + checksum verification |
-| `https://cloud-images.ubuntu.com/noble/current` | Ubuntu 24.04 LTS cloud image download | HTTPS | TLS + checksum verification |
+| `https://cloud.debian.org/images/cloud/trixie/` | Debian 13 cloud image download | HTTPS | TLS + checksum verification |
+| `https://cloud-images.ubuntu.com/resolute/` | Ubuntu 26.04 LTS cloud image download | HTTPS | TLS + checksum verification |
 | `https://d2yp8ipz5jfqcw.cloudfront.net` | Alpine image for managed HAProxy load-balancer | HTTPS | TLS + checksum verification |
 | `https://install.mulgadc.com/install` | One-shot install telemetry POST on `spx admin init` / `join`. | HTTPS | TLS |
 

docs/service-interfaces.yaml

@@ -0,0 +1,305 @@
+# Service interface manifest.
+#
+# Source of truth for "which services produce/consume which NATS subjects"
+# and "which e2e suites cover which services". Consumed by
+# tests/e2e/select-suites (Bead 4) to derive targeted suite selection from
+# a PR diff.
+#
+# See docs/development/improvements/e2e-targeted-suite-selection.md.
+#
+# Editing rules
+# -------------
+# - subscribes / publishes: subjects this service receives / sends. Both
+#   lists use the literal subject string, or a "<prefix>.*" wildcard for
+#   pattern subscriptions (e.g. "ebs.snapshot.*").
+# - depends_on: subjects this service requires from peers. Used by the
+#   selector to compute "if X changes, which suites might break".
+# - Subjects must match what is actually referenced in source. The lint
+#   in Bead 5 greps `"(ec2|vpc|ebs|s3|iam|elbv2|spinifex)\.[a-zA-Z._-]+"`
+#   and fails if any subject is absent from this manifest.
+
+version: 1
+
+services:
+
+  awsgw:
+    path: spinifex/services/awsgw
+    # TLS frontend, SigV4 auth. Pure router: HTTP → NATS request/response.
+    # No subscribes; only publishes (via NATSRequest).
+    subscribes: []
+    publishes:
+      - "ec2.*"
+      - "elbv2.*"
+      - "iam.*"
+    depends_on:
+      - "ec2.*"
+      - "elbv2.*"
+      - "iam.*"
+
+  spinifex_daemon:
+    path: spinifex/services/spinifex
+    # Orchestrator. Owns most ec2.* + all elbv2.* + iam event publication.
+    # Translates VPC / EBS / S3 operations into vpc.* / ebs.* / s3.* and
+    # forwards to vpcd / viperblockd / predastore.
+    #
+    # Source layout: spinifex/services/spinifex (entry), spinifex/daemon
+    # (subscribe registration), spinifex/handlers/{ec2,elbv2,iam} (per-
+    # subject handler bodies). Selector treats all four paths as belonging
+    # to this service via the `additional_paths` field.
+    additional_paths:
+      - spinifex/daemon
+      - spinifex/handlers/ec2
+      - spinifex/handlers/elbv2
+      - spinifex/handlers/iam
+    subscribes:
+      # EC2 instance lifecycle.
+      - "ec2.RunInstances"
+      - "ec2.DescribeInstances"
+      - "ec2.DescribeStoppedInstances"
+      - "ec2.DescribeTerminatedInstances"
+      - "ec2.DescribeInstanceAttribute"
+      - "ec2.DescribeInstanceStatus"
+      - "ec2.DescribeInstanceTypes"
+      - "ec2.ModifyInstanceAttribute"
+      - "ec2.ModifyVolume"
+      - "ec2.start"
+      - "ec2.terminate"
+      - "ec2.launch"
+      - "ec2.cmd.*"
+      # EC2 VPC / network ops handled by daemon before delegation to vpcd.
+      - "ec2.CreateVpc"
+      - "ec2.DescribeVpcs"
+      - "ec2.CreateSubnet"
+      - "ec2.CreateInternetGateway"
+      - "ec2.CreateKeyPair"
+      - "ec2.CreateImage"
+      - "ec2.DescribePlacementGroups"
+      # Cluster coordination subjects.
+      - "ec2.ReserveClusterNode"
+      - "ec2.FinalizeClusterInstances"
+      - "spinifex.node.status"
+      - "spinifex.node.vms"
+      - "spinifex.nodes.discover"
+      - "spinifex.cluster.shutdown.progress"
+      - "spinifex.storage.config"
+      # ELBv2 control plane (handlers/elbv2/service_nats.go topic map).
+      - "elbv2.CreateLoadBalancer"
+      - "elbv2.DeleteLoadBalancer"
+      - "elbv2.DescribeLoadBalancers"
+      - "elbv2.CreateTargetGroup"
+      - "elbv2.DeleteTargetGroup"
+      - "elbv2.DescribeTargetGroups"
+      - "elbv2.RegisterTargets"
+      - "elbv2.DeregisterTargets"
+      - "elbv2.DescribeTargetHealth"
+      - "elbv2.CreateListener"
+      - "elbv2.DeleteListener"
+      - "elbv2.DescribeListeners"
+    publishes:
+      # → vpcd
+      - "vpc.create"
+      - "vpc.delete"
+      - "vpc.create-subnet"
+      - "vpc.delete-subnet"
+      - "vpc.create-sg"
+      - "vpc.delete-sg"
+      - "vpc.update-sg"
+      - "vpc.create-port"
+      - "vpc.delete-port"
+      - "vpc.update-port-sgs"
+      - "vpc.igw-attach"
+      - "vpc.igw-detach"
+      - "vpc.add-nat-gateway"
+      - "vpc.delete-nat-gateway"
+      - "vpc.add-nat"
+      - "vpc.delete-nat"
+      - "vpc.dhcp.acquire"
+      - "vpc.dhcp.release"
+      # → viperblockd
+      - "ebs.mount"
+      - "ebs.unmount"
+      - "ebs.delete"
+      - "ebs.sync"
+      - "ebs.snapshot.*"
+      # → predastore (S3 / AMI / keypair store)
+      - "s3.*"
+      # IAM lifecycle event
+      - "iam.account.created"
+    depends_on:
+      - "vpc.*"
+      - "ebs.*"
+      - "s3.*"
+
+  vpcd:
+    path: spinifex/services/vpcd
+    # OVN-backed VPC / subnet / SG / NAT / IGW data plane.
+    subscribes:
+      - "vpc.create"
+      - "vpc.delete"
+      - "vpc.create-subnet"
+      - "vpc.delete-subnet"
+      - "vpc.create-sg"
+      - "vpc.delete-sg"
+      - "vpc.update-sg"
+      - "vpc.create-port"
+      - "vpc.delete-port"
+      - "vpc.update-port-sgs"
+      - "vpc.igw-attach"
+      - "vpc.igw-detach"
+      - "vpc.add-nat-gateway"
+      - "vpc.delete-nat-gateway"
+      - "vpc.add-nat"
+      - "vpc.delete-nat"
+      - "vpc.dhcp.acquire"
+      - "vpc.dhcp.release"
+    publishes:
+      - "vpc.port-status"
+      - "vpc.dhcp.lease-expired"
+    depends_on: []
+
+  viperblockd:
+    path: spinifex/services/viperblockd
+    # EBS data plane. Backed by the viperblock storage engine in the
+    # mulga/viperblock submodule. Submodule pointer bump → selector forces
+    # "run all" (contract changes can be invisible at the spinifex source
+    # level).
+    subscribes:
+      - "ebs.mount"
+      - "ebs.unmount"
+      - "ebs.delete"
+      - "ebs.sync"
+      - "ebs.snapshot.*"
+    publishes:
+      - "ebs.mount.response"
+      - "ebs.unmount.response"
+    depends_on: []
+
+  predastore:
+    path: spinifex/services/predastore
+    # S3 / AMI / keypair storage. External submodule under mulga/predastore;
+    # this dir contains only the spinifex-side adapter. Submodule pointer
+    # bump → selector forces "run all".
+    subscribes:
+      - "s3.*"
+    publishes: []
+    depends_on: []
+
+  nats:
+    path: spinifex/services/nats
+    # NATS broker bootstrap / cluster config. Not a NATS pub/sub
+    # participant. Any change here is infra-path; selector forces
+    # "run all".
+    subscribes: []
+    publishes: []
+    depends_on: []
+
+  spinifexui:
+    path: spinifex/services/spinifexui
+    # Web UI. Talks to awsgw over HTTP, never NATS. Diffs limited to this
+    # dir do not need any e2e suite to run.
+    subscribes: []
+    publishes: []
+    depends_on: []
+    covers_no_suite: true
+
+
+# Fixture → service mapping. Each Ensure* helper in
+# tests/e2e/harness/fixtures.go exercises this set of services; the
+# Bead 4 selector AST-scans test files for harness.Ensure* call sites
+# and resolves coverage via this block.
+fixtures:
+
+  EnsureKeyPair:
+    services: [spinifex_daemon, predastore]
+
+  EnsureAMI:
+    services: [spinifex_daemon, predastore, viperblockd]
+
+  EnsureDefaultVPC:
+    services: [vpcd, spinifex_daemon]
+
+  EnsureSubnet:
+    services: [vpcd, spinifex_daemon]
+
+  EnsureSG:
+    services: [vpcd, spinifex_daemon]
+
+  EnsureInstance:
+    services: [spinifex_daemon, viperblockd, vpcd]
+
+  EnsureVolume:
+    services: [spinifex_daemon, viperblockd]
+
+  EnsureSnapshot:
+    services: [spinifex_daemon, viperblockd, predastore]
+
+  EnsureNATGateway:
+    services: [vpcd, spinifex_daemon]
+
+  EnsureLoadBalancer:
+    services: [spinifex_daemon, awsgw]
+
+
+suites:
+
+  e2e-cert:
+    path: tests/e2e/cert
+    covers: [awsgw, spinifex_daemon, vpcd, predastore]
+
+  e2e-lb:
+    path: tests/e2e/lb
+    covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+    # Exercises full ELBv2 control plane (handlers/elbv2) + data plane
+    # (LB agent on the system VM). ELBv2 is part of spinifex_daemon in
+    # this manifest; revisit if it splits into a standalone service.
+
+  e2e-single:
+    path: tests/e2e/single
+    covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+    # Full single-node lifecycle: VPC, AMI, keypair, instance, volume,
+    # snapshot, IAM. Currently runs all phases sequentially under one
+    # `TestSingleNode`. Bead 3 (single-fixture-migration) splits this
+    # into per-test fixture-driven coverage so the selector can pick
+    # subtests, not just whole suites.
+
+  e2e-ddil:
+    path: tests/e2e/ddil
+    covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+    # DDIL = degraded / disconnected / intermittent / low-bandwidth.
+    # Exercises the full daemon stack under network failure injection;
+    # any service touching the wire path is in scope.
+
+  # Suites below are bash-only today. Entries activate as the Go ports
+  # land (Bead 2 fixtures unblock new-shape ports for multinode etc.).
+  #
+  # e2e-multinode:
+  #   path: spinifex/tests/e2e/multinode
+  #   covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+  # e2e-reboot:
+  #   path: spinifex/tests/e2e/reboot
+  #   covers: [spinifex_daemon, viperblockd]
+  # e2e-tofu:
+  #   path: spinifex/tests/e2e/tofu
+  #   covers: [awsgw, spinifex_daemon, vpcd]
+  # e2e-bm:
+  #   path: spinifex/tests/e2e/baremetal
+  #   covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+
+
+# Special-case rules consumed by the selector (Bead 4). Hard-coded in
+# the tool, listed here for visibility.
+#
+# infra_paths_force_all:
+#   - spinifex/services/nats/**
+#   - spinifex/tests/e2e/harness/**
+#   - spinifex/tests/e2e/lib/**
+#   - spinifex/.github/workflows/e2e-go.yml
+#   - spinifex/.github/scripts/**
+#   - spinifex/Makefile
+#   - mulga/scripts/tofu-cluster/**
+#
+# submodule_force_all:
+#   - mulga/predastore (submodule pointer)
+#   - mulga/viperblock (submodule pointer)
+#
+# many_services_force_all_threshold: 3   # ≥ N services touched → run all

docs/terraform-workbooks/bastion-private-subnet/README.md

@@ -56,7 +56,7 @@ Deploy a VPC with public and private subnets where the private subnet has no int
 **Prerequisites:**
 
 - Spinifex installed and running (see [Installing Spinifex](/docs/install))
-- An Ubuntu 24.04 AMI imported (see [Setting Up Your Cluster](/docs/setting-up-your-cluster))
+- An Ubuntu 26.04 AMI imported (see [Setting Up Your Cluster](/docs/setting-up-your-cluster))
 - OpenTofu or Terraform installed
 
 ## Instructions
@@ -149,7 +149,7 @@ The private route table should have no `0.0.0.0/0` route. Also verify the privat
 
 ### AMI Not Found
 
-Ensure you have imported an Ubuntu 24.04 image:
+Ensure you have imported an Ubuntu 26.04 image:
 
 ```bash
 aws ec2 describe-images --owners 000000000000 --profile spinifex

docs/terraform-workbooks/bastion-private-subnet/main.tf

@@ -96,7 +96,7 @@ data "aws_ami" "ubuntu" {
 
   filter {
     name   = "name"
-    values = ["*ubuntu-24.04*"]
+    values = ["*ubuntu-26.04*", "*ubuntu-24.04*"]
   }
 
   filter {

docs/terraform-workbooks/nginx-alb/README.md

@@ -58,7 +58,7 @@ Deploy two Nginx web servers behind an internet-facing Application Load Balancer
 | Elastic IP | `nginx-alb-nat-eip` | Public address for the NAT Gateway |
 | NAT Gateway | `nginx-alb-nat` | Outbound internet for the private subnets (cloud-init apt bootstrap) |
 | Security Group | `nginx-alb-sg` | Allows SSH (22) and HTTP (80) inbound |
-| EC2 Instances | `nginx-alb-1`, `nginx-alb-2` | Ubuntu 24.04 with Nginx via cloud-init (private subnets) |
+| EC2 Instances | `nginx-alb-1`, `nginx-alb-2` | Ubuntu 26.04 with Nginx via cloud-init (private subnets) |
 | ALB | `nginx-alb` | Internet-facing Application Load Balancer on port 80 |
 | Target Group | `nginx-alb-tg` | HTTP health-checked group for both instances |
 | Listener | HTTP :80 | Forwards traffic to the target group |
@@ -66,7 +66,7 @@ Deploy two Nginx web servers behind an internet-facing Application Load Balancer
 **Prerequisites:**
 
 - Spinifex installed and running (see [Installing Spinifex](/docs/install))
-- An Ubuntu 24.04 AMI imported (see [Setting Up Your Cluster](/docs/setting-up-your-cluster))
+- An Ubuntu 26.04 AMI imported (see [Setting Up Your Cluster](/docs/setting-up-your-cluster))
 - OpenTofu or Terraform installed
 
 ## Instructions
@@ -86,21 +86,15 @@ Or create a `main.tf` file and paste the full configuration below.
 
 <!-- INCLUDE: main.tf lang:hcl -->
 
-### Step 2. Install Load Balancer AMI
+### Step 2. Install Debian AMI
 
-Next install the load balancer AMI images, which is used as the AMI disk image for launching the load-balancer and a requirement.
+Install the Debian 13 AMI which is used in the example to host the `nginx` webservers as an EC2 instance.
 
 ```bash
-spx admin images import --name lb-alpine-3.21.6-x86_64
+spx admin images import --name debian-13-x86_64
 ```
 
-### Step 2.1 Install Debian AMI
-
-Next, install the Debian 12 AMI which is used in the example to host the `nginx` webservers as an EC2 instance.
-
-```bash
-spx admin images import --name debian-12-x86_64
-```
+> **Note:** The load balancer itself runs as a direct-boot QEMU microvm using the kernel + initramfs bundled with the Spinifex distribution (`/usr/share/spinifex/microvm/`), so no separate LB AMI import is required.
 
 ### Step 3. Deploy
 
@@ -184,7 +178,7 @@ tofu destroy
 
 ### AMI Not Found
 
-Ensure you have imported an Ubuntu 24.04 image. Check available AMIs:
+Ensure you have imported an Ubuntu 26.04 image. Check available AMIs:
 
 ```bash
 aws ec2 describe-images --owners 000000000000 --profile spinifex
@@ -193,7 +187,7 @@ aws ec2 describe-images --owners 000000000000 --profile spinifex
 If missing import:
 
 ```bash
-spx admin images import --name ubuntu-24.04-x86_64
+spx admin images import --name ubuntu-26.04-x86_64
 ```
 
 ### Provider Connection Refused

docs/terraform-workbooks/nginx-alb/main.tf

@@ -102,7 +102,7 @@ data "aws_ami" "ubuntu" {
 
   filter {
     name   = "name"
-    values = ["*ubuntu-24.04*"]
+    values = ["*ubuntu-26.04*", "*ubuntu-24.04*"]
   }
 }
 

docs/terraform-workbooks/nginx-webserver/README.md

@@ -43,7 +43,7 @@ Deploy a complete Nginx web server on Spinifex using Terraform/OpenTofu. This wo
 **Prerequisites:**
 
 - Spinifex installed and running (see [Installing Spinifex](/docs/install))
-- An Ubuntu 24.04 AMI imported (see [Setting Up Your Cluster](/docs/setting-up-your-cluster))
+- An Ubuntu 26.04 AMI imported (see [Setting Up Your Cluster](/docs/setting-up-your-cluster))
 - OpenTofu or Terraform installed
 
 ## Instructions
@@ -96,7 +96,7 @@ tofu destroy
 
 ### AMI Not Found
 
-Ensure you have imported an Ubuntu 24.04 image. Check available AMIs:
+Ensure you have imported an Ubuntu 26.04 image. Check available AMIs:
 
 ```bash
 aws ec2 describe-images --owners 000000000000 --profile spinifex

docs/terraform-workbooks/nginx-webserver/main.tf

@@ -85,7 +85,7 @@ data "aws_ami" "ubuntu" {
 
   filter {
     name   = "name"
-    values = ["*ubuntu-24.04*"]
+    values = ["*ubuntu-26.04*", "*ubuntu-24.04*"]
   }
 
   filter {
@@ -220,7 +220,7 @@ resource "aws_security_group" "web" {
 }
 
 # ---------------------------------------------------------------------------
-# EC2 Instance — Ubuntu 24.04 with Nginx installed via cloud-init
+# EC2 Instance — Ubuntu 26.04 with Nginx installed via cloud-init
 # ---------------------------------------------------------------------------
 
 resource "aws_instance" "nginx" {

docs/terraform-workbooks/s3-webapp/README.md

@@ -53,7 +53,7 @@ Deploy an EC2 instance running a Flask file-sharing web application backed by S3
 
 - Spinifex installed and running (see [Installing Spinifex](/docs/install))
 - Predastore running (S3 API on port 8443)
-- An Ubuntu 24.04 AMI imported (see [Setting Up Your Cluster](/docs/setting-up-your-cluster))
+- An Ubuntu 26.04 AMI imported (see [Setting Up Your Cluster](/docs/setting-up-your-cluster))
 - OpenTofu or Terraform installed
 - The EC2 instance must be able to reach Predastore — use the host's br-wan IP, not localhost
 
@@ -153,7 +153,7 @@ cat /opt/webapp/.env
 
 ### AMI Not Found
 
-Ensure you have imported an Ubuntu 24.04 image:
+Ensure you have imported an Ubuntu 26.04 image:
 
 ```bash
 aws ec2 describe-images --owners 000000000000 --profile spinifex

docs/terraform-workbooks/s3-webapp/main.tf

@@ -132,7 +132,7 @@ data "aws_ami" "ubuntu" {
 
   filter {
     name   = "name"
-    values = ["*ubuntu-24.04*"]
+    values = ["*ubuntu-26.04*", "*ubuntu-24.04*"]
   }
 
   filter {

go.mod

@@ -1,31 +1,32 @@
 module github.com/mulgadc/spinifex
 
-go 1.26.2
+go 1.26.3
 
 require (
 	github.com/aws/aws-sdk-go v1.55.8
 	github.com/charmbracelet/bubbles v1.0.0
 	github.com/charmbracelet/bubbletea v1.3.10
 	github.com/charmbracelet/lipgloss v1.1.0
-	github.com/go-chi/chi/v5 v5.2.5
+	github.com/go-chi/chi/v5 v5.3.0
 	github.com/google/uuid v1.6.0
 	github.com/insomniacslk/dhcp v0.0.0-20260407060928-11b94ed970f2
 	github.com/kdomanski/iso9660 v0.4.0
 	github.com/klauspost/cpuid/v2 v2.3.0
-	github.com/mdlayher/packet v1.1.2
-	github.com/mulgadc/predastore v1.2.0
-	github.com/mulgadc/viperblock v1.2.1
-	github.com/nats-io/nats-server/v2 v2.12.7
-	github.com/nats-io/nats.go v1.51.0
+	github.com/mulgadc/predastore v1.5.0
+	github.com/mulgadc/viperblock v1.5.0
+	github.com/nats-io/nats-server/v2 v2.14.1
+	github.com/nats-io/nats.go v1.52.0
 	github.com/ovn-kubernetes/libovsdb v0.8.1
-	github.com/pelletier/go-toml/v2 v2.3.0
+	github.com/pelletier/go-toml/v2 v2.3.1
 	github.com/pterm/pterm v0.12.83
 	github.com/spf13/cobra v1.10.2
 	github.com/spf13/viper v1.21.0
 	github.com/stretchr/testify v1.11.1
-	golang.org/x/crypto v0.50.0
-	golang.org/x/sys v0.43.0
-	gopkg.in/ini.v1 v1.67.1
+	go.uber.org/goleak v1.3.0
+	golang.org/x/crypto v0.52.0
+	golang.org/x/sync v0.20.0
+	gopkg.in/ini.v1 v1.67.2
+	gopkg.in/yaml.v3 v3.0.1
 )
 
 require (
@@ -35,6 +36,8 @@ require (
 	github.com/antithesishq/antithesis-sdk-go v0.7.0 // indirect
 	github.com/armon/go-metrics v0.4.1 // indirect
 	github.com/atotto/clipboard v0.1.4 // indirect
+	github.com/aws/aws-sdk-go-v2 v1.41.7 // indirect
+	github.com/aws/smithy-go v1.25.1 // indirect
 	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/boltdb/bolt v1.3.1 // indirect
@@ -79,8 +82,8 @@ require (
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
 	github.com/jmespath/go-jmespath v0.4.0 // indirect
 	github.com/josharian/native v1.1.0 // indirect
-	github.com/klauspost/compress v1.18.5 // indirect
-	github.com/klauspost/reedsolomon v1.13.3 // indirect
+	github.com/klauspost/compress v1.18.6 // indirect
+	github.com/klauspost/reedsolomon v1.14.0 // indirect
 	github.com/leodido/go-urn v1.4.0 // indirect
 	github.com/lithammer/fuzzysearch v1.1.8 // indirect
 	github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
@@ -88,6 +91,7 @@ require (
 	github.com/mattn/go-isatty v0.0.20 // indirect
 	github.com/mattn/go-localereader v0.0.1 // indirect
 	github.com/mattn/go-runewidth v0.0.21 // indirect
+	github.com/mdlayher/packet v1.1.2 // indirect
 	github.com/mdlayher/socket v0.4.1 // indirect
 	github.com/minio/crc64nvme v1.1.1 // indirect
 	github.com/minio/highwayhash v1.0.4 // indirect
@@ -104,7 +108,7 @@ require (
 	github.com/prometheus/client_model v0.6.2 // indirect
 	github.com/prometheus/common v0.67.5 // indirect
 	github.com/prometheus/procfs v0.20.1 // indirect
-	github.com/quic-go/quic-go v0.59.0 // indirect
+	github.com/quic-go/quic-go v0.59.1 // indirect
 	github.com/rivo/uniseg v0.4.7 // indirect
 	github.com/sagikazarmark/locafero v0.12.0 // indirect
 	github.com/spf13/afero v1.15.0 // indirect
@@ -120,18 +124,16 @@ require (
 	go.opentelemetry.io/otel/trace v1.42.0 // indirect
 	go.yaml.in/yaml/v2 v2.4.4 // indirect
 	go.yaml.in/yaml/v3 v3.0.4 // indirect
-	golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa // indirect
-	golang.org/x/mod v0.34.0 // indirect
-	golang.org/x/net v0.53.0 // indirect
-	golang.org/x/sync v0.20.0 // indirect
-	golang.org/x/telemetry v0.0.0-20260311193753-579e4da9a98c // indirect
-	golang.org/x/term v0.42.0 // indirect
-	golang.org/x/text v0.36.0 // indirect
+	golang.org/x/mod v0.35.0 // indirect
+	golang.org/x/net v0.55.0 // indirect
+	golang.org/x/sys v0.45.0 // indirect
+	golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa // indirect
+	golang.org/x/term v0.43.0 // indirect
+	golang.org/x/text v0.37.0 // indirect
 	golang.org/x/time v0.15.0 // indirect
-	golang.org/x/tools v0.43.0 // indirect
+	golang.org/x/tools v0.44.0 // indirect
 	golang.org/x/vuln v1.1.4 // indirect
 	google.golang.org/protobuf v1.36.11 // indirect
-	gopkg.in/yaml.v3 v3.0.1 // indirect
 )
 
 tool golang.org/x/vuln/cmd/govulncheck

go.sum

@@ -31,30 +31,30 @@ github.com/atotto/clipboard v0.1.4 h1:EH0zSVneZPSuFR11BlR9YppQTVDbh5+16AmcJi4g1z
 github.com/atotto/clipboard v0.1.4/go.mod h1:ZY9tmq7sm5xIbd9bOK4onWV4S6X0u6GY7Vn0Yu86PYI=
 github.com/aws/aws-sdk-go v1.55.8 h1:JRmEUbU52aJQZ2AjX4q4Wu7t4uZjOu71uyNmaWlUkJQ=
 github.com/aws/aws-sdk-go v1.55.8/go.mod h1:ZkViS9AqA6otK+JBBNH2++sx1sgxrPKcSzPPvQkUtXk=
-github.com/aws/aws-sdk-go-v2 v1.41.6 h1:1AX0AthnBQzMx1vbmir3Y4WsnJgiydmnJjiLu+LvXOg=
-github.com/aws/aws-sdk-go-v2 v1.41.6/go.mod h1:dy0UzBIfwSeot4grGvY1AqFWN5zgziMmWGzysDnHFcQ=
-github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.9 h1:adBsCIIpLbLmYnkQU+nAChU5yhVTvu5PerROm+/Kq2A=
-github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.9/go.mod h1:uOYhgfgThm/ZyAuJGNQ5YgNyOlYfqnGpTHXvk3cpykg=
-github.com/aws/aws-sdk-go-v2/credentials v1.19.15 h1:fyvgWTszojq8hEnMi8PPBTvZdTtEVmAVyo+NFLHBhH4=
-github.com/aws/aws-sdk-go-v2/credentials v1.19.15/go.mod h1:gJiYyMOjNg8OEdRWOf3CrFQxM2a98qmrtjx1zuiQfB8=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.22 h1:GmLa5Kw1ESqtFpXsx5MmC84QWa/ZrLZvlJGa2y+4kcQ=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.22/go.mod h1:6sW9iWm9DK9YRpRGga/qzrzNLgKpT2cIxb7Vo2eNOp0=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.22 h1:dY4kWZiSaXIzxnKlj17nHnBcXXBfac6UlsAx2qL6XrU=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.22/go.mod h1:KIpEUx0JuRZLO7U6cbV204cWAEco2iC3l061IxlwLtI=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.23 h1:FPXsW9+gMuIeKmz7j6ENWcWtBGTe1kH8r9thNt5Uxx4=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.23/go.mod h1:7J8iGMdRKk6lw2C+cMIphgAnT8uTwBwNOsGkyOCm80U=
-github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.8 h1:HtOTYcbVcGABLOVuPYaIihj6IlkqubBwFj10K5fxRek=
-github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.8/go.mod h1:VsK9abqQeGlzPgUr+isNWzPlK2vKe9INMLWnY65f5Xs=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.14 h1:xnvDEnw+pnj5mctWiYuFbigrEzSm35x7k4KS/ZkCANg=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.14/go.mod h1:yS5rNogD8e0Wu9+l3MUwr6eENBzEeGejvINpN5PAYfY=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.22 h1:PUmZeJU6Y1Lbvt9WFuJ0ugUK2xn6hIWUBBbKuOWF30s=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.22/go.mod h1:nO6egFBoAaoXze24a2C0NjQCvdpk8OueRoYimvEB9jo=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.22 h1:SE+aQ4DEqG53RRCAIHlCf//B2ycxGH7jFkpnAh/kKPM=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.22/go.mod h1:ES3ynECd7fYeJIL6+oax+uIEljmfps0S70BaQzbMd/o=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.100.0 h1:7G26Sae6PMKn4kMcU5JzNfrm1YrKwyOhowXPYR2WiWY=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.100.0/go.mod h1:Fw9aqhJicIVee1VytBBjH+l+5ov6/PhbtIK/u3rt/ls=
-github.com/aws/smithy-go v1.25.0 h1:Sz/XJ64rwuiKtB6j98nDIPyYrV1nVNJ4YU74gttcl5U=
-github.com/aws/smithy-go v1.25.0/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
+github.com/aws/aws-sdk-go-v2 v1.41.7 h1:DWpAJt66FmnnaRIOT/8ASTucrvuDPZASqhhLey6tLY8=
+github.com/aws/aws-sdk-go-v2 v1.41.7/go.mod h1:4LAfZOPHNVNQEckOACQx60Y8pSRjIkNZQz1w92xpMJc=
+github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10 h1:gx1AwW1Iyk9Z9dD9F4akX5gnN3QZwUB20GGKH/I+Rho=
+github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10/go.mod h1:qqY157uZoqm5OXq/amuaBJyC9hgBCBQnsaWnPe905GY=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.16 h1:r3RJBuU7X9ibt8RHbMjWE6y60QbKBiII6wSrXnapxSU=
+github.com/aws/aws-sdk-go-v2/credentials v1.19.16/go.mod h1:6cx7zqDENJDbBIIWX6P8s0h6hqHC8Avbjh9Dseo27ug=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23 h1:GpT/TrnBYuE5gan2cZbTtvP+JlHsutdmlV2YfEyNde0=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23/go.mod h1:xYWD6BS9ywC5bS3sz9Xh04whO/hzK2plt2Zkyrp4JuA=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23 h1:bpd8vxhlQi2r1hiueOw02f/duEPTMK59Q4QMAoTTtTo=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23/go.mod h1:15DfR2nw+CRHIk0tqNyifu3G1YdAOy68RftkhMDDwYk=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.24 h1:OQqn11BtaYv1WLUowvcA30MpzIu8Ti4pcLPIIyoKZrA=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.24/go.mod h1:X5ZJyfwVrWA96GzPmUCWFQaEARPR7gCrpq2E92PJwAE=
+github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9 h1:FLudkZLt5ci0ozzgkVo8BJGwvqNaZbTWb3UcucAateA=
+github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9/go.mod h1:w7wZ/s9qK7c8g4al+UyoF1Sp/Z45UwMGcqIzLWVQHWk=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.15 h1:ieLCO1JxUWuxTZ1cRd0GAaeX7O6cIxnwk7tc1LsQhC4=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.15/go.mod h1:e3IzZvQ3kAWNykvE0Tr0RDZCMFInMvhku3qNpcIQXhM=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.23 h1:pbrxO/kuIwgEsOPLkaHu0O+m4fNgLU8B3vxQ+72jTPw=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.23/go.mod h1:/CMNUqoj46HpS3MNRDEDIwcgEnrtZlKRaHNaHxIFpNA=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.23 h1:03xatSQO4+AM1lTAbnRg5OK528EUg744nW7F73U8DKw=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.23/go.mod h1:M8l3mwgx5ToK7wot2sBBce/ojzgnPzZXUV445gTSyE8=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0 h1:etqBTKY581iwLL/H/S2sVgk3C9lAsTJFeXWFDsDcWOU=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0/go.mod h1:L2dcoOgS2VSgbPLvpak2NyUPsO1TBN7M45Z4H7DlRc4=
+github.com/aws/smithy-go v1.25.1 h1:J8ERsGSU7d+aCmdQur5Txg6bVoYelvQJgtZehD12GkI=
+github.com/aws/smithy-go v1.25.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
 github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
 github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
@@ -122,8 +122,8 @@ github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S
 github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
 github.com/gabriel-vasile/mimetype v1.4.13 h1:46nXokslUBsAJE/wMsp5gtO500a4F3Nkz9Ufpk2AcUM=
 github.com/gabriel-vasile/mimetype v1.4.13/go.mod h1:d+9Oxyo1wTzWdyVUPMmXFvp4F9tea18J8ufA774AB3s=
-github.com/go-chi/chi/v5 v5.2.5 h1:Eg4myHZBjyvJmAFjFvWgrqDTXFyOzjj7YIm3L3mu6Ug=
-github.com/go-chi/chi/v5 v5.2.5/go.mod h1:X7Gx4mteadT3eDOMTsXzmI4/rwUpOwBHLpAfupzFJP0=
+github.com/go-chi/chi/v5 v5.3.0 h1:halUjDxhshgXHMrao5bB8eNBXo/rnzwr8m5m36glehM=
+github.com/go-chi/chi/v5 v5.3.0/go.mod h1:R+tYY2hNuVUUjxoPtqUdgBqevM9s9njzkTLutVsOCto=
 github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
 github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
 github.com/go-kit/log v0.1.0/go.mod h1:zbhenjAZHb184qTLMA9ZjW7ThYL0H2mk7Q6pNt4vbaY=
@@ -229,15 +229,15 @@ github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7V
 github.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM=
 github.com/kdomanski/iso9660 v0.4.0 h1:BPKKdcINz3m0MdjIMwS0wx1nofsOjxOq8TOr45WGHFg=
 github.com/kdomanski/iso9660 v0.4.0/go.mod h1:OxUSupHsO9ceI8lBLPJKWBTphLemjrCQY8LPXM7qSzU=
-github.com/klauspost/compress v1.18.5 h1:/h1gH5Ce+VWNLSWqPzOVn6XBO+vJbCNGvjoaGBFW2IE=
-github.com/klauspost/compress v1.18.5/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
+github.com/klauspost/compress v1.18.6 h1:2jupLlAwFm95+YDR+NwD2MEfFO9d4z4Prjl1XXDjuao=
+github.com/klauspost/compress v1.18.6/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
 github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
 github.com/klauspost/cpuid/v2 v2.0.10/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuObKfj5c0PQa7c=
 github.com/klauspost/cpuid/v2 v2.0.12/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuObKfj5c0PQa7c=
 github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y=
 github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
-github.com/klauspost/reedsolomon v1.13.3 h1:01GwnO2xoCSaM0ShP4qwl+FsHg3csFShC6Tu/RS1ji0=
-github.com/klauspost/reedsolomon v1.13.3/go.mod h1:yjqqjgMTQkBUHSG97/rm4zipffCNbCiZcB3kTqr++sQ=
+github.com/klauspost/reedsolomon v1.14.0 h1:5YSZeclzSYg5nl349+GDG/agDtQ6MZiwUYXvVKN1Jx0=
+github.com/klauspost/reedsolomon v1.14.0/go.mod h1:yjqqjgMTQkBUHSG97/rm4zipffCNbCiZcB3kTqr++sQ=
 github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
 github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
 github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
@@ -286,20 +286,20 @@ github.com/muesli/cancelreader v0.2.2 h1:3I4Kt4BQjOR54NavqnDogx/MIoWBFa0StPA8ELU
 github.com/muesli/cancelreader v0.2.2/go.mod h1:3XuTXfFS2VjM+HTLZY9Ak0l6eUKfijIfMUZ4EgX0QYo=
 github.com/muesli/termenv v0.16.0 h1:S5AlUN9dENB57rsbnkPyfdGuWIlkmzJjbFf0Tf5FWUc=
 github.com/muesli/termenv v0.16.0/go.mod h1:ZRfOIKPFDYQoDFF4Olj7/QJbW60Ol/kL1pU3VfY/Cnk=
-github.com/mulgadc/predastore v1.2.0 h1:E4+ehAm2YEi2n5t4DmqguURMUXwTSDrK7792zVzw7JM=
-github.com/mulgadc/predastore v1.2.0/go.mod h1:+NWROaNCtpPE9f36fUxCHwskW8CuHHEIvjsDQrwOB00=
-github.com/mulgadc/viperblock v1.2.1 h1:G3cSgh/yspZd7Wdg3C9QGqkqzYbEfYpusd6uMddSDmg=
-github.com/mulgadc/viperblock v1.2.1/go.mod h1:CisHy13Uam0gUg5dxV0rKryJF2yGbgfDv09FYApRaNk=
+github.com/mulgadc/predastore v1.5.0 h1:FtL9EquoGKl3tcIRIoAXGhQStkN4uPDx0TOQZ89Mh+c=
+github.com/mulgadc/predastore v1.5.0/go.mod h1:fXIGz7yCKcHh68ybcw7aigdOdzIfyt19h/0diPF+JCU=
+github.com/mulgadc/viperblock v1.5.0 h1:2OprAI1oAtitUosS5A9LbenU/RCaw0oxUJvUhv8+tgU=
+github.com/mulgadc/viperblock v1.5.0/go.mod h1:lIuOaBO3gC0o/EwWhJPpCuhUEPQ3/y74xckGlcBKaos=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
 github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
 github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
 github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
-github.com/nats-io/nats-server/v2 v2.12.7 h1:prQ9cPiWHcnwfT81Wi5lU9LL8TLY+7pxDru6fQYLCQQ=
-github.com/nats-io/nats-server/v2 v2.12.7/go.mod h1:dOnmkprKMluTmTF7/QHZioxlau3sKHUM/LBPy9AiBPw=
-github.com/nats-io/nats.go v1.51.0 h1:ByW84XTz6W03GSSsygsZcA+xgKK8vPGaa/FCAAEHnAI=
-github.com/nats-io/nats.go v1.51.0/go.mod h1:26HypzazeOkyO3/mqd1zZd53STJN0EjCYF9Uy2ZOBno=
+github.com/nats-io/nats-server/v2 v2.14.1 h1:wXs/a5fw9Hzm3CvuzLxGeIwpjPulSa7gMT3eSuhGkcg=
+github.com/nats-io/nats-server/v2 v2.14.1/go.mod h1:4N17zLpuS7WMbG8T9gsE2B7z9hC9PraPyulVBfpK6nU=
+github.com/nats-io/nats.go v1.52.0 h1:n3avV4VBsCgsdwh71TppsTwtv+QdPs7ntSKM8qJLGsc=
+github.com/nats-io/nats.go v1.52.0/go.mod h1:26HypzazeOkyO3/mqd1zZd53STJN0EjCYF9Uy2ZOBno=
 github.com/nats-io/nkeys v0.4.15 h1:JACV5jRVO9V856KOapQ7x+EY8Jo3qw1vJt/9Jpwzkk4=
 github.com/nats-io/nkeys v0.4.15/go.mod h1:CpMchTXC9fxA5zrMo4KpySxNjiDVvr8ANOSZdiNfUrs=
 github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
@@ -308,8 +308,8 @@ github.com/ovn-kubernetes/libovsdb v0.8.1 h1:M2J8bcJt5mXCom0HqzfEtuHkT80CTSQRcYG
 github.com/ovn-kubernetes/libovsdb v0.8.1/go.mod h1:ZlnHLzagmLOSvyd9qfxBIZp6wOSOw0IsRsc+6lNUGbU=
 github.com/pascaldekloe/goe v0.1.0 h1:cBOtyMzM9HTpWjXfbbunk26uA6nG3a8n06Wieeh0MwY=
 github.com/pascaldekloe/goe v0.1.0/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
-github.com/pelletier/go-toml/v2 v2.3.0 h1:k59bC/lIZREW0/iVaQR8nDHxVq8OVlIzYCOJf421CaM=
-github.com/pelletier/go-toml/v2 v2.3.0/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
+github.com/pelletier/go-toml/v2 v2.3.1 h1:MYEvvGnQjeNkRF1qUuGolNtNExTDwct51yp7olPtrEc=
+github.com/pelletier/go-toml/v2 v2.3.1/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
 github.com/pierrec/lz4/v4 v4.1.14 h1:+fL8AQEZtz/ijeNnpduH0bROTu0O3NZAlPjQxGn8LwE=
 github.com/pierrec/lz4/v4 v4.1.14/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
 github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
@@ -351,8 +351,8 @@ github.com/pterm/pterm v0.12.36/go.mod h1:NjiL09hFhT/vWjQHSj1athJpx6H8cjpHXNAK5b
 github.com/pterm/pterm v0.12.40/go.mod h1:ffwPLwlbXxP+rxT0GsgDTzS3y3rmpAO1NMjUkGTYf8s=
 github.com/pterm/pterm v0.12.83 h1:ie+YmGmA727VuhxBlyGr74Ks+7McV6kT99IB8EU80aA=
 github.com/pterm/pterm v0.12.83/go.mod h1:xlgc6bFWyJIMtmLJvGim+L7jhSReilOlOnodeIYe4Tk=
-github.com/quic-go/quic-go v0.59.0 h1:OLJkp1Mlm/aS7dpKgTc6cnpynnD2Xg7C1pwL6vy/SAw=
-github.com/quic-go/quic-go v0.59.0/go.mod h1:upnsH4Ju1YkqpLXC305eW3yDZ4NfnNbmQRCMWS58IKU=
+github.com/quic-go/quic-go v0.59.1 h1:0Gmua0HW1Tv7ANR7hUYwRyD0MG5OJfgvYSZasGZzBic=
+github.com/quic-go/quic-go v0.59.1/go.mod h1:upnsH4Ju1YkqpLXC305eW3yDZ4NfnNbmQRCMWS58IKU=
 github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
 github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
 github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
@@ -424,14 +424,14 @@ golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnf
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
-golang.org/x/crypto v0.50.0 h1:zO47/JPrL6vsNkINmLoo/PH1gcxpls50DNogFvB5ZGI=
-golang.org/x/crypto v0.50.0/go.mod h1:3muZ7vA7PBCE6xgPX7nkzzjiUq87kRItoJQM1Yo8S+Q=
+golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988=
+golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc=
 golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa h1:FRnLl4eNAQl8hwxVVC17teOw8kdjVDVAiFMtgUdTSRQ=
 golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa/go.mod h1:zk2irFbV9DP96SEBUUAy67IdHUaZuSnrz1n472HUCLE=
 golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
 golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
-golang.org/x/mod v0.34.0 h1:xIHgNUUnW6sYkcM5Jleh05DvLOtwc6RitGHbDk4akRI=
-golang.org/x/mod v0.34.0/go.mod h1:ykgH52iCZe79kzLLMhyCUzhMci+nQj+0XkbXpNYtVjY=
+golang.org/x/mod v0.35.0 h1:Ww1D637e6Pg+Zb2KrWfHQUnH2dQRLBQyAtpr/haaJeM=
+golang.org/x/mod v0.35.0/go.mod h1:+GwiRhIInF8wPm+4AoT6L0FA1QWAad3OMdTRx4tFYlU=
 golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -442,8 +442,8 @@ golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81R
 golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
 golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
 golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
-golang.org/x/net v0.53.0 h1:d+qAbo5L0orcWAr0a9JweQpjXF19LMXJE8Ey7hwOdUA=
-golang.org/x/net v0.53.0/go.mod h1:JvMuJH7rrdiCfbeHoo3fCQU24Lf5JJwT9W3sJFulfgs=
+golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8=
+golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -484,33 +484,33 @@ golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
-golang.org/x/sys v0.43.0 h1:Rlag2XtaFTxp19wS8MXlJwTvoh8ArU6ezoyFsMyCTNI=
-golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
-golang.org/x/telemetry v0.0.0-20260311193753-579e4da9a98c h1:6a8FdnNk6bTXBjR4AGKFgUKuo+7GnR3FX5L7CbveeZc=
-golang.org/x/telemetry v0.0.0-20260311193753-579e4da9a98c/go.mod h1:TpUTTEp9frx7rTdLpC9gFG9kdI7zVLFTFFlqaH2Cncw=
+golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
+golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
+golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa h1:efT73AJZfAAUV7SOip6pWGkwJDzIGiKBZGVzHYa+ve4=
+golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa/go.mod h1:kHjTxDEnAu6/Nl9lDkzjWpR+bmKfxeiRuSDlsMb70gE=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
-golang.org/x/term v0.42.0 h1:UiKe+zDFmJobeJ5ggPwOshJIVt6/Ft0rcfrXZDLWAWY=
-golang.org/x/term v0.42.0/go.mod h1:Dq/D+snpsbazcBG5+F9Q1n2rXV8Ma+71xEjTRufARgY=
+golang.org/x/term v0.43.0 h1:S4RLU2sB31O/NCl+zFN9Aru9A/Cq2aqKpTZJ6B+DwT4=
+golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
 golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
 golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
 golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
-golang.org/x/text v0.36.0 h1:JfKh3XmcRPqZPKevfXVpI1wXPTqbkE5f7JA92a55Yxg=
-golang.org/x/text v0.36.0/go.mod h1:NIdBknypM8iqVmPiuco0Dh6P5Jcdk8lJL0CUebqK164=
+golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc=
+golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38=
 golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
 golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
 golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
-golang.org/x/tools v0.43.0 h1:12BdW9CeB3Z+J/I/wj34VMl8X+fEXBxVR90JeMX5E7s=
-golang.org/x/tools v0.43.0/go.mod h1:uHkMso649BX2cZK6+RpuIPXS3ho2hZo4FVwfoy1vIk0=
+golang.org/x/tools v0.44.0 h1:UP4ajHPIcuMjT1GqzDWRlalUEoY+uzoZKnhOjbIPD2c=
+golang.org/x/tools v0.44.0/go.mod h1:KA0AfVErSdxRZIsOVipbv3rQhVXTnlU6UhKxHd1seDI=
 golang.org/x/tools/go/expect v0.1.1-deprecated h1:jpBZDwmgPhXsKZC6WhL20P4b/wmnpsEAGHaNy0n/rJM=
 golang.org/x/tools/go/expect v0.1.1-deprecated/go.mod h1:eihoPOH+FgIqa3FpoTwguz/bVUSGBlGQU67vpBeOrBY=
 golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated h1:1h2MnaIAIXISqTFKdENegdpAgUXz6NrPEsbIeWaBRvM=
@@ -534,8 +534,8 @@ gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8
 gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
-gopkg.in/ini.v1 v1.67.1 h1:tVBILHy0R6e4wkYOn3XmiITt/hEVH4TFMYvAX2Ytz6k=
-gopkg.in/ini.v1 v1.67.1/go.mod h1:x/cyOwCgZqOkJoDIJ3c1KNHMo10+nLGAhh+kn3Zizss=
+gopkg.in/ini.v1 v1.67.2 h1:JtOSMb9OuaCZKr7h5D/h6iii14sK0hLbplTc6frx4Ss=
+gopkg.in/ini.v1 v1.67.2/go.mod h1:x/cyOwCgZqOkJoDIJ3c1KNHMo10+nLGAhh+kn3Zizss=
 gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
@@ -548,5 +548,5 @@ gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C
 gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
 gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
-pgregory.net/rapid v1.2.0 h1:keKAYRcjm+e1F0oAuU5F5+YPAWcyxNNRK2wud503Gnk=
-pgregory.net/rapid v1.2.0/go.mod h1:PY5XlDGj0+V1FCq0o192FdRhpKHGTRIWBgqjDBTrq04=
+pgregory.net/rapid v1.3.0 h1:vBvO0VSqti75J1jjYqpgPNBLKMd1+gxa9fYo7vk/Exc=
+pgregory.net/rapid v1.3.0/go.mod h1:dPlE4OBBxgXPqkP79flB6sJL1dx5azpI7HQ9MY9Z7uk=

internal/fipsboot/fipsboot.go

@@ -0,0 +1,21 @@
+// Package fipsboot enforces FIPS 140-3 mode at process startup.
+//
+// Blank-import this package from every binary's main:
+//
+//	import _ "github.com/mulgadc/spinifex/internal/fipsboot"
+//
+// Builds with GOFIPS140=v1.0.0 link the Go Cryptographic Module v1.0.0
+// (NIST CMVP cert #5247). At runtime FIPS mode can still be turned off
+// via GODEBUG=fips140=off; init() panics in that case to close the hole.
+package fipsboot
+
+import (
+	"crypto/fips140"
+)
+
+func init() {
+	if !fips140.Enabled() {
+		panic("fipsboot: FIPS 140-3 mode is not enabled at runtime — refusing to start. " +
+			"Build with GOFIPS140=v1.0.0 and do not set GODEBUG=fips140=off.")
+	}
+}

internal/tlsconfig/curves.go

@@ -0,0 +1,37 @@
+// Package tlsconfig pins the cluster-wide TLS 1.3 curve preference list.
+//
+// The list is hardcoded because mixed classification on a single cluster is
+// not supported and the certified FIPS build is itself the per-deployment
+// choice. A typo in a config string would silently downgrade where a missing
+// constant cannot.
+package tlsconfig
+
+import "crypto/tls"
+
+// Curves is the fixed TLS 1.3 curve allowlist:
+//
+//   - X25519MLKEM768     — Cat 3 ML-KEM hybrid, wins under Go 1.26 stdlib filter
+//   - SecP384r1MLKEM1024 — Cat 5 ML-KEM hybrid, fallback for peers that prefer it
+//   - X25519             — classical TLS 1.3 fallback for SDKs without ML-KEM (aws CLI v2 today)
+//   - CurveP384          — classical TLS 1.3 fallback (Java/.NET enterprise SDKs)
+//
+// PQ-capable peers (browsers since 2024, Go aws-sdk v2 with Go 1.24+) always
+// negotiate the hybrid first; classical fallback only triggers for peers that
+// cannot do ML-KEM. Those connections still get TLS 1.3 + AES-256-GCM but
+// remain HNDL-exposed — accepted trade-off until AWS CLI v2 / bundled
+// OpenSSL ships ML-KEM (currently lagging upstream OpenSSL 3.5).
+//
+// Weak/niche entries from the Go stdlib default (SecP256r1MLKEM768, P-256,
+// P-521) are intentionally excluded so they cannot be advertised.
+//
+// Go 1.26 stdlib treats tls.Config.CurvePreferences as an allowlist filter
+// applied to the stdlib default order, not as a user-supplied order.
+//
+// Callers must not mutate this slice. Assign it directly to
+// tls.Config.CurvePreferences; do not append to it.
+var Curves = []tls.CurveID{
+	tls.X25519MLKEM768,
+	tls.SecP384r1MLKEM1024,
+	tls.X25519,
+	tls.CurveP384,
+}

internal/tlsconfig/curves_test.go

@@ -0,0 +1,169 @@
+package tlsconfig_test
+
+import (
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/tls"
+	"crypto/x509"
+	"crypto/x509/pkix"
+	"encoding/pem"
+	"io"
+	"math/big"
+	"net"
+	"net/http"
+	"net/http/httptest"
+	"slices"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/mulgadc/spinifex/internal/tlsconfig"
+)
+
+func TestCurves_OnlyApprovedEntries(t *testing.T) {
+	want := []tls.CurveID{
+		tls.X25519MLKEM768,
+		tls.SecP384r1MLKEM1024,
+		tls.X25519,
+		tls.CurveP384,
+	}
+	if !slices.Equal(tlsconfig.Curves, want) {
+		t.Errorf("Curves = %v, want %v", tlsconfig.Curves, want)
+	}
+}
+
+func TestCurves_ExcludesWeakPrimitives(t *testing.T) {
+	forbidden := []tls.CurveID{
+		tls.SecP256r1MLKEM768, // weakest ML-KEM hybrid (P-256 component)
+		tls.CurveP256,         // not CNSA 2.0
+		tls.CurveP521,         // niche, no current peers
+	}
+	for _, c := range forbidden {
+		if slices.Contains(tlsconfig.Curves, c) {
+			t.Errorf("Curves includes forbidden curve %v", c)
+		}
+	}
+}
+
+// TestIntegration_NegotiatesPQHybrid spins up an httptest TLS server
+// configured with Curves + TLS 1.3 minimum, dials it, and asserts a PQ
+// hybrid is negotiated at TLS 1.3. Per Go 1.26 stdlib filter behavior the
+// winner is X25519MLKEM768; the assertion checks for any approved hybrid
+// to remain stable across stdlib reorderings of our 4 entries.
+func TestIntegration_NegotiatesPQHybrid(t *testing.T) {
+	cert := selfSignedCert(t)
+	srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = io.WriteString(w, "ok")
+	}))
+	srv.TLS = &tls.Config{
+		Certificates:     []tls.Certificate{cert},
+		MinVersion:       tls.VersionTLS13,
+		CurvePreferences: tlsconfig.Curves,
+	}
+	srv.StartTLS()
+	defer srv.Close()
+
+	pool := x509.NewCertPool()
+	pool.AddCert(srv.Certificate())
+
+	client := &http.Client{
+		Timeout: 5 * time.Second,
+		Transport: &http.Transport{
+			TLSClientConfig: &tls.Config{
+				RootCAs:          pool,
+				CurvePreferences: tlsconfig.Curves,
+			},
+		},
+	}
+	resp, err := client.Get(srv.URL)
+	if err != nil {
+		t.Fatalf("https get: %v", err)
+	}
+	defer resp.Body.Close()
+
+	state := resp.TLS
+	if state == nil {
+		t.Fatal("response has no TLS state")
+	}
+	if state.Version != tls.VersionTLS13 {
+		t.Errorf("negotiated version = 0x%x, want TLS 1.3 (0x%x)", state.Version, tls.VersionTLS13)
+	}
+	pqHybrids := []tls.CurveID{tls.X25519MLKEM768, tls.SecP384r1MLKEM1024}
+	if !slices.Contains(pqHybrids, state.CurveID) {
+		t.Errorf("negotiated curve = %v, want one of %v (PQ hybrid)", state.CurveID, pqHybrids)
+	}
+}
+
+// TestIntegration_TLS12ClientRejected confirms MinVersion: tls.VersionTLS13
+// is enforced — a TLS 1.2-pinned client gets a handshake error rather than
+// silently downgrading. This is the negative test that closes the HNDL gap
+// on external surfaces.
+func TestIntegration_TLS12ClientRejected(t *testing.T) {
+	cert := selfSignedCert(t)
+	srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = io.WriteString(w, "ok")
+	}))
+	srv.TLS = &tls.Config{
+		Certificates:     []tls.Certificate{cert},
+		MinVersion:       tls.VersionTLS13,
+		CurvePreferences: tlsconfig.Curves,
+	}
+	srv.StartTLS()
+	defer srv.Close()
+
+	pool := x509.NewCertPool()
+	pool.AddCert(srv.Certificate())
+
+	client := &http.Client{
+		Timeout: 5 * time.Second,
+		Transport: &http.Transport{
+			TLSClientConfig: &tls.Config{
+				RootCAs:    pool,
+				MinVersion: tls.VersionTLS12,
+				MaxVersion: tls.VersionTLS12,
+			},
+		},
+	}
+	_, err := client.Get(srv.URL)
+	if err == nil {
+		t.Fatal("TLS 1.2-pinned client succeeded; MinVersion not enforced")
+	}
+	if !strings.Contains(err.Error(), "protocol version") &&
+		!strings.Contains(err.Error(), "tls:") {
+		t.Errorf("expected protocol-version handshake error, got: %v", err)
+	}
+}
+
+func selfSignedCert(t *testing.T) tls.Certificate {
+	t.Helper()
+	key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+	if err != nil {
+		t.Fatalf("generate key: %v", err)
+	}
+	tmpl := &x509.Certificate{
+		SerialNumber: big.NewInt(1),
+		Subject:      pkix.Name{CommonName: "tlsconfig-test"},
+		NotBefore:    time.Now().Add(-time.Hour),
+		NotAfter:     time.Now().Add(time.Hour),
+		KeyUsage:     x509.KeyUsageDigitalSignature,
+		ExtKeyUsage:  []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
+		DNSNames:     []string{"localhost"},
+		IPAddresses:  []net.IP{net.ParseIP("127.0.0.1"), net.ParseIP("::1")},
+	}
+	der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key)
+	if err != nil {
+		t.Fatalf("create cert: %v", err)
+	}
+	keyDER, err := x509.MarshalECPrivateKey(key)
+	if err != nil {
+		t.Fatalf("marshal key: %v", err)
+	}
+	certPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
+	keyPEM := pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER})
+	pair, err := tls.X509KeyPair(certPEM, keyPEM)
+	if err != nil {
+		t.Fatalf("X509KeyPair: %v", err)
+	}
+	return pair
+}

scripts/ansible/README.md

@@ -31,6 +31,7 @@ inventory and roles relative to CWD.
 | `playbooks/dev-teardown.yml` | Wipe state only (no reinstall) | `reset-dev-env.sh` §teardown |
 | `playbooks/dev-install.yml` | Build + install + init + smoketest on clean box | `dev-install.sh` |
 | `playbooks/dev-reset.yml` | Capture settings → teardown → build → install → init → smoketest | `reset-dev-env.sh` (full) |
+| `playbooks/dev-deploy.yml` | Rebuild + swap binaries/microvm artifacts + restart (no setup.sh, no smoketest) | `make deploy` |
 
 Upcoming (not yet implemented):
 
@@ -44,6 +45,7 @@ ansible-playbook playbooks/dev-preflight.yml
 ansible-playbook playbooks/dev-teardown.yml
 ansible-playbook playbooks/dev-install.yml
 ansible-playbook playbooks/dev-reset.yml
+ansible-playbook playbooks/dev-deploy.yml
 ```
 
 Or via `make` (from `spinifex/`):
@@ -53,8 +55,16 @@ make ansible-dev-preflight
 make ansible-dev-teardown
 make ansible-dev-install
 make ansible-dev-reset
+make ansible-dev-deploy
 ```
 
+### When to use which
+
+- First-time / clean box → `ansible-dev-install`
+- Iterate on Go code, microvm initramfs, lb-agent → `ansible-dev-deploy` (fast)
+- Changed systemd units, helper scripts, logrotate, setup.sh → `ansible-dev-reset` (slow, full rebuild)
+- Need a clean slate without reinstall → `ansible-dev-teardown`
+
 ## Variable overrides
 
 Defaults in `vars/defaults.yml`. Override per run with `-e`:
@@ -74,7 +84,6 @@ Useful overrides:
 | `spinifex_region` | `ap-southeast-2` | Region passed to `spx admin init` |
 | `spinifex_az` | `{{ region }}a` | AZ passed to `spx admin init` |
 | `spinifex_external_mode` | `pool` | `pool` \| `nat` external networking |
-| `spinifex_build_lb_image` | `true` | Build + import LB system image after init |
 
 `dev-reset.yml` automatically captures `region`, `az`, `external_mode`,
 pool range, gateway, prefix-len, nat gateway_ip from the existing

scripts/ansible/playbooks/dev-deploy.yml

@@ -0,0 +1,14 @@
+---
+# In-place artifact swap: rebuilds + replaces binaries/microvm artifacts on
+# an already-installed node, then restarts spinifex.target. Use when iterating
+# on code without altering systemd units, users, dirs, or running the full
+# smoketest. For first-time or unit/script changes use dev-install / dev-reset.
+- name: Dev deploy — build + swap binaries + restart (no setup.sh, no smoketest)
+  hosts: localhost
+  gather_facts: true
+  vars_files:
+    - ../vars/defaults.yml
+
+  roles:
+    - build
+    - deploy

scripts/ansible/roles/build/tasks/main.yml

@@ -11,6 +11,18 @@
     chdir: "{{ mulga_root }}/viperblock"
   changed_when: true
 
+- name: build — detect microVM build script (present only on direct-boot branches)
+  ansible.builtin.stat:
+    path: "{{ spinifex_root }}/scripts/build-microvm-image.sh"
+  register: build_microvm_script
+
+- name: build — microVM kernel + initramfs (direct-boot LBs)
+  ansible.builtin.command:
+    cmd: make build-microvm-image
+    chdir: "{{ spinifex_root }}"
+  changed_when: true
+  when: build_microvm_script.stat.exists
+
 - name: build — verify build artifacts present
   ansible.builtin.stat:
     path: "{{ item }}"
@@ -40,9 +52,15 @@
     mkdir -p "$STAGING/scripts"
     cp "{{ spinifex_root }}/build/scripts/"* "$STAGING/scripts/"
     cp "{{ spinifex_root }}/build/logrotate/spinifex" "$STAGING/logrotate-spinifex"
+    # microVM artifacts ship only when the branch built them.
+    if [ -f "{{ spinifex_root }}/build/microvm/vmlinuz" ] && [ -f "{{ spinifex_root }}/build/microvm/initramfs.cpio.gz" ]; then
+        mkdir -p "$STAGING/microvm"
+        cp "{{ spinifex_root }}/build/microvm/vmlinuz" "$STAGING/microvm/"
+        cp "{{ spinifex_root }}/build/microvm/initramfs.cpio.gz" "$STAGING/microvm/"
+    fi
     tar czf "{{ build_tarball_path }}" -C "$STAGING" .
   changed_when: true
 
 - name: build — summary
   ansible.builtin.debug:
-    msg: "Built spx + nbdkit plugin, tarball at {{ build_tarball_path }}"
+    msg: "Built spx + nbdkit plugin{{ ' + microvm artifacts' if build_microvm_script.stat.exists else '' }}, tarball at {{ build_tarball_path }}"

scripts/ansible/roles/deploy/tasks/main.yml

@@ -0,0 +1,148 @@
+---
+# In-place artifact swap for an already-installed spinifex node. Mirrors
+# `make deploy`: replace binaries + microvm artifacts, restart services.
+# Does NOT run setup.sh (no user/dir/systemd-unit changes) or smoketest.
+
+- name: deploy — fail fast if spinifex isn't already installed
+  ansible.builtin.stat:
+    path: "{{ spinifex_config_file }}"
+  register: deploy_installed_stat
+  failed_when: not deploy_installed_stat.stat.exists
+
+- name: deploy — assert config present
+  ansible.builtin.assert:
+    that: deploy_installed_stat.stat.exists
+    fail_msg: |
+      {{ spinifex_config_file }} missing — this playbook only updates an
+      existing install. Run ansible-dev-install or ansible-dev-reset first.
+
+- name: deploy — verify build artifacts present
+  ansible.builtin.stat:
+    path: "{{ item }}"
+  register: deploy_artifacts
+  loop:
+    - "{{ spinifex_root }}/bin/spx"
+    - "{{ mulga_root }}/viperblock/lib/nbdkit-viperblock-plugin.so"
+
+- name: deploy — assert artifacts exist after build role
+  ansible.builtin.assert:
+    that: item.stat.exists
+    fail_msg: "Build artifact missing: {{ item.item }} — build role should have produced it"
+  loop: "{{ deploy_artifacts.results }}"
+  loop_control:
+    label: "{{ item.item }}"
+
+- name: deploy — detect microvm artifacts (direct-boot branches only)
+  ansible.builtin.stat:
+    path: "{{ spinifex_root }}/build/microvm/{{ item }}"
+  register: deploy_microvm_stat
+  loop:
+    - vmlinuz
+    - initramfs.cpio.gz
+
+- name: deploy — set microvm presence fact
+  ansible.builtin.set_fact:
+    deploy_microvm_present: "{{ deploy_microvm_stat.results | map(attribute='stat.exists') | min }}"
+
+- name: deploy — detect nbdkit plugin directory
+  ansible.builtin.shell: |
+    set -e
+    nbdkit --dump-config | awk -F= '/^plugindir=/{print $2; exit}'
+  register: deploy_plugindir_cmd
+  changed_when: false
+
+- name: deploy — set plugin path fact
+  ansible.builtin.set_fact:
+    deploy_plugindir: "{{ deploy_plugindir_cmd.stdout | trim }}"
+
+- name: deploy — assert plugindir resolved
+  ansible.builtin.assert:
+    that: deploy_plugindir | length > 0
+    fail_msg: "Could not detect nbdkit plugin directory via `nbdkit --dump-config`"
+
+- name: deploy — stop spinifex.target before binary swap
+  become: true
+  ansible.builtin.systemd:
+    name: spinifex.target
+    state: stopped
+  failed_when: false
+
+- name: deploy — install spx binary
+  become: true
+  ansible.builtin.copy:
+    src: "{{ spinifex_root }}/bin/spx"
+    dest: /usr/local/bin/spx
+    mode: "0755"
+    owner: root
+    group: root
+
+- name: deploy — install viperblock nbdkit plugin
+  become: true
+  ansible.builtin.copy:
+    src: "{{ mulga_root }}/viperblock/lib/nbdkit-viperblock-plugin.so"
+    dest: "{{ deploy_plugindir }}/nbdkit-viperblock-plugin.so"
+    mode: "0755"
+    owner: root
+    group: root
+
+- name: deploy — ensure microvm dir
+  become: true
+  ansible.builtin.file:
+    path: /usr/share/spinifex/microvm
+    state: directory
+    mode: "0755"
+    owner: root
+    group: root
+  when: deploy_microvm_present
+
+- name: deploy — install microvm kernel
+  become: true
+  ansible.builtin.copy:
+    src: "{{ spinifex_root }}/build/microvm/vmlinuz"
+    dest: /usr/share/spinifex/microvm/vmlinuz
+    mode: "0644"
+    owner: root
+    group: root
+  when: deploy_microvm_present
+
+- name: deploy — install microvm initramfs
+  become: true
+  ansible.builtin.copy:
+    src: "{{ spinifex_root }}/build/microvm/initramfs.cpio.gz"
+    dest: /usr/share/spinifex/microvm/initramfs.cpio.gz
+    mode: "0644"
+    owner: root
+    group: root
+  when: deploy_microvm_present
+
+- name: deploy — daemon-reload (covers any unit changes shipped via setup.sh later)
+  become: true
+  ansible.builtin.systemd:
+    daemon_reload: true
+
+- name: deploy — start spinifex.target
+  become: true
+  ansible.builtin.systemd:
+    name: spinifex.target
+    state: started
+
+- name: deploy — wait for awsgw to respond
+  become: true
+  become_user: "{{ spinifex_invoking_user }}"
+  environment:
+    AWS_PROFILE: spinifex
+  ansible.builtin.command: aws ec2 describe-regions
+  register: deploy_awsgw
+  retries: 15
+  delay: 2
+  until: deploy_awsgw.rc == 0
+  changed_when: false
+
+- name: deploy — summary
+  ansible.builtin.debug:
+    msg: |
+      Deployed:
+        spx           → /usr/local/bin/spx
+        nbdkit plugin → {{ deploy_plugindir }}/nbdkit-viperblock-plugin.so
+        microvm       → {{ '/usr/share/spinifex/microvm/{vmlinuz,initramfs.cpio.gz}' if deploy_microvm_present else 'skipped (not built on this branch)' }}
+      spinifex.target restarted; awsgw responding.

scripts/ansible/roles/init/tasks/main.yml

@@ -74,19 +74,6 @@
   delay: 2
   changed_when: false
 
-- name: init — build LB agent
-  ansible.builtin.command:
-    cmd: make build-lb-agent
-    chdir: "{{ spinifex_root }}"
-  changed_when: true
-  when: spinifex_build_lb_image | bool
-
-- name: init — import LB system image
-  ansible.builtin.command:
-    cmd: "{{ spinifex_root }}/scripts/build-system-image.sh {{ spinifex_lb_image_conf }} --import --quiet"
-  changed_when: true
-  when: spinifex_build_lb_image | bool
-
 - name: init — summary
   ansible.builtin.debug:
     msg: |
@@ -95,4 +82,3 @@
         external_mode={{ spinifex_external_mode }}
         spinifex.target: started
         CA: {{ spinifex_ca_trust_path }}
-        LB image: {{ 'imported' if spinifex_build_lb_image else 'skipped' }}

scripts/ansible/roles/smoketest/tasks/main.yml

@@ -61,6 +61,7 @@
     --distro {{ spinifex_smoketest_distro }}
     --version {{ spinifex_smoketest_version }}
     --arch {{ smoketest_arch }}
+    --boot-mode uefi
   register: smoketest_import_local
   until: >-
     smoketest_import_local.rc == 0 or

scripts/ansible/vars/defaults.yml

@@ -32,34 +32,34 @@ spinifex_nat_gateway_ip: ""
 
 # Preflight checks.
 spinifex_ports:
-  - { port: 4222, service: NATS }
-  - { port: 8443, service: Predastore }
-  - { port: 9999, service: "Spinifex Gateway" }
+    - { port: 4222, service: NATS }
+    - { port: 8443, service: Predastore }
+    - { port: 9999, service: "Spinifex Gateway" }
 
 spinifex_required_commands:
-  - go
-  - make
-  - nbdkit
+    - go
+    - make
+    - nbdkit
 spinifex_optional_commands:
-  - air
+    - air
 spinifex_ovn_commands:
-  - ovs-vsctl
-  - ovn-controller
-  - ovn-nbctl
+    - ovs-vsctl
+    - ovn-controller
+    - ovn-nbctl
 
 spinifex_components:
-  - viperblock
-  - predastore
+    - viperblock
+    - predastore
 
 # Process patterns for teardown.
 # Bracket trick ([b]in/spx etc.) prevents the pattern literal from matching
 # our own pgrep/shell cmdline. Exclude filter drops benign log followers and
 # ansible's own wrappers that also mention these service names.
-spinifex_process_pattern: '([b]in/spx|spinifex-[u]i|nats-[s]erver|[p]redastore|[v]iperblock|[v]pcd)'
-spinifex_process_exclude: '(journalctl|ansible|/bin/sh|pgrep|pkill|tail -[fF])'
+spinifex_process_pattern: "([b]in/spx|spinifex-[u]i|nats-[s]erver|[p]redastore|[v]iperblock|[v]pcd)"
+spinifex_process_exclude: "(journalctl|ansible|/bin/sh|pgrep|pkill|tail -[fF])"
 spinifex_qemu_processes:
-  - qemu-system-x86_64
-  - qemu-system-aarch64
+    - qemu-system-x86_64
+    - qemu-system-aarch64
 
 # Invoking user (when run under sudo).
 spinifex_invoking_user: "{{ lookup('env','SUDO_USER') | default(ansible_facts['user_id'], true) }}"
@@ -79,12 +79,8 @@ mulga_root: "{{ lookup('pipe', 'readlink -f ' ~ playbook_dir ~ '/../../../..') }
 # Build output.
 build_tarball_path: /tmp/spinifex-local.tar.gz
 
-# LB system image (built + imported at end of init).
-spinifex_build_lb_image: true
-spinifex_lb_image_conf: "{{ spinifex_root }}/scripts/images/lb.conf"
-
 # Smoketest image — parameterised so tofu examples can share the same base.
 spinifex_smoketest_distro: debian
-spinifex_smoketest_version: "12"
+spinifex_smoketest_version: "13"
 # Local image file path (if present, imported directly; otherwise fall back to --name).
 spinifex_smoketest_local_image: "~/images/{{ spinifex_smoketest_distro }}-{{ spinifex_smoketest_version }}.img"

scripts/build-microvm-image.sh

@@ -0,0 +1,264 @@
+#!/bin/bash
+# build-microvm-image.sh — build a minimal Alpine-based microVM kernel + initramfs
+# for use with QEMU microvm machine type and Spinifex lb-agent.
+#
+# Outputs:
+#   $REPO_ROOT/build/microvm/vmlinuz
+#   $REPO_ROOT/build/microvm/initramfs.cpio.gz
+#
+# Rootfs strategy (first match wins):
+#   1. apk available on host → install directly into a temp chroot (Alpine CI/CD)
+#   2. docker available      → run alpine container, export filesystem
+#   3. podman available      → same via podman
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+BUILD_DIR="$REPO_ROOT/build/microvm"
+INIT_SH="$BUILD_DIR/init.sh"
+INITTAB="$BUILD_DIR/inittab"
+
+ALPINE_VERSION="${ALPINE_VERSION:-3.20}"
+ALPINE_MIRROR="${ALPINE_MIRROR:-https://dl-cdn.alpinelinux.org/alpine}"
+ALPINE_PACKAGES="busybox linux-virt haproxy iproute2 ca-certificates"
+
+echo "[build-microvm-image] repo root: $REPO_ROOT"
+echo "[build-microvm-image] build dir: $BUILD_DIR"
+mkdir -p "$BUILD_DIR"
+
+# --- Verify non-negotiable tools ---
+for tool in cpio gzip find; do
+    if ! command -v "$tool" >/dev/null 2>&1; then
+        echo "ERROR: required tool not found: $tool" >&2
+        exit 1
+    fi
+done
+
+# --- Create chroot ---
+CHROOT_DIR=$(mktemp -d)
+CONTAINER_TOOL=""
+CONTAINER_CID=""
+cleanup() {
+    echo "[build-microvm-image] cleaning up chroot: $CHROOT_DIR"
+    rm -rf "$CHROOT_DIR"
+    if [ -n "$CONTAINER_CID" ] && [ -n "$CONTAINER_TOOL" ]; then
+        echo "[build-microvm-image] cleaning up container: $CONTAINER_CID"
+        $CONTAINER_TOOL rm -f "$CONTAINER_CID" >/dev/null 2>&1 || true
+    fi
+}
+trap cleanup EXIT
+
+# --- Populate rootfs ---
+# Strategy 1: native apk (Alpine hosts, CI)
+if command -v apk >/dev/null 2>&1; then
+    echo "[build-microvm-image] rootfs: native apk"
+    mkdir -p \
+        "$CHROOT_DIR/etc/apk" \
+        "$CHROOT_DIR/lib/apk/db" \
+        "$CHROOT_DIR/var/cache/apk"
+    cat > "$CHROOT_DIR/etc/apk/repositories" <<EOF
+${ALPINE_MIRROR}/v${ALPINE_VERSION}/main
+${ALPINE_MIRROR}/v${ALPINE_VERSION}/community
+EOF
+    apk add \
+        --root "$CHROOT_DIR" \
+        --arch x86_64 \
+        --no-cache \
+        --allow-untrusted \
+        --initdb \
+        $ALPINE_PACKAGES
+
+# Strategy 2/3: container runtime (dev machines)
+else
+    if command -v docker >/dev/null 2>&1 && docker info >/dev/null 2>&1; then
+        CONTAINER_TOOL=docker
+    elif command -v podman >/dev/null 2>&1; then
+        CONTAINER_TOOL=podman
+    else
+        echo "ERROR: no rootfs build tool available." >&2
+        echo "       Install one of: apk (Alpine), docker, podman" >&2
+        exit 1
+    fi
+
+    echo "[build-microvm-image] rootfs: $CONTAINER_TOOL export (alpine:${ALPINE_VERSION})"
+    CONTAINER_CID=$($CONTAINER_TOOL run -d \
+        "alpine:${ALPINE_VERSION}" \
+        sh -c "apk add --no-cache ${ALPINE_PACKAGES}")
+
+    exit_code=$($CONTAINER_TOOL wait "$CONTAINER_CID")
+    if [ "$exit_code" != "0" ]; then
+        echo "ERROR: package installation failed in container (exit $exit_code)" >&2
+        exit 1
+    fi
+
+    $CONTAINER_TOOL export "$CONTAINER_CID" | tar -x -C "$CHROOT_DIR"
+fi
+
+# --- Fix /dev device nodes ---
+# tar extraction without root silently creates 0-byte regular files for device
+# nodes (mknod requires CAP_MKNOD). The kernel opens /dev/console to wire
+# init's stdio to the serial console; a regular file there causes all init
+# output to be silently discarded. Recreate the two nodes needed before
+# devtmpfs mounts (which provides the rest at runtime).
+rm -rf "$CHROOT_DIR/dev"
+mkdir "$CHROOT_DIR/dev"
+sudo mknod -m 600 "$CHROOT_DIR/dev/console" c 5 1
+sudo mknod -m 666 "$CHROOT_DIR/dev/null"    c 1 3
+
+# --- Place init script ---
+echo "[build-microvm-image] installing init.sh as /init..."
+if [ ! -f "$INIT_SH" ]; then
+    echo "ERROR: $INIT_SH not found — cannot build initramfs" >&2
+    exit 1
+fi
+install -m 0755 "$INIT_SH" "$CHROOT_DIR/init"
+
+# --- Place inittab ---
+if [ -f "$INITTAB" ]; then
+    echo "[build-microvm-image] installing inittab..."
+    install -m 0644 "$INITTAB" "$CHROOT_DIR/etc/inittab"
+fi
+
+# --- Copy lb-agent binary ---
+LB_AGENT_BIN="$REPO_ROOT/bin/lb-agent"
+if [ -f "$LB_AGENT_BIN" ]; then
+    echo "[build-microvm-image] copying lb-agent binary..."
+    mkdir -p "$CHROOT_DIR/usr/local/bin"
+    install -m 0755 "$LB_AGENT_BIN" "$CHROOT_DIR/usr/local/bin/lb-agent"
+else
+    echo "[build-microvm-image] WARNING: $LB_AGENT_BIN not found — lb-agent will be absent from initramfs" >&2
+fi
+
+# --- Create required directories for init ---
+mkdir -p \
+    "$CHROOT_DIR/proc" \
+    "$CHROOT_DIR/sys" \
+    "$CHROOT_DIR/dev" \
+    "$CHROOT_DIR/etc/ssl/certs" \
+    "$CHROOT_DIR/etc/conf.d"
+
+# --- Locate kernel ---
+KERNEL_IMG=$(find "$CHROOT_DIR/boot" -name "vmlinuz*" | sort | tail -1)
+if [ -z "$KERNEL_IMG" ]; then
+    echo "ERROR: no vmlinuz found in $CHROOT_DIR/boot" >&2
+    exit 1
+fi
+echo "[build-microvm-image] kernel image: $KERNEL_IMG"
+
+KERNEL_VER=$(basename "$KERNEL_IMG" | sed 's/vmlinuz-//')
+KERNEL_CONFIG="$CHROOT_DIR/boot/config-${KERNEL_VER}"
+
+# --- Assert fw_cfg module ---
+if ! find "$CHROOT_DIR/lib/modules" \
+        -name "qemu_fw_cfg.ko*" \
+        -o -name "fw_cfg_sysfs.ko*" \
+    | grep -q .; then
+    if ! grep -q "CONFIG_FW_CFG_SYSFS=y" "$KERNEL_CONFIG" 2>/dev/null; then
+        echo "ERROR: qemu_fw_cfg module missing from initramfs — fw_cfg reads will fail at boot" >&2
+        exit 1
+    fi
+    echo "[build-microvm-image] fw_cfg: built-in (CONFIG_FW_CFG_SYSFS=y)"
+else
+    echo "[build-microvm-image] fw_cfg: module found in lib/modules"
+fi
+
+# --- Strip kernel modules (Phase 2 keep-list approach) ---
+# Only modules needed by a virtio-mmio microvm guest are kept:
+#   virtio*.ko*    — virtio bus, net, blk, rng, console, and helpers
+#   *fw_cfg*.ko*   — QEMU fw_cfg sysfs driver (delivers boot blobs)
+#   *failover*.ko* — net_failover + failover (virtio_net deps for live migration)
+# All other loadable modules are removed. Drivers compiled into the kernel
+# (CONFIG_VIRTIO_MMIO=y etc.) are unaffected — they live in vmlinuz, not here.
+echo "[build-microvm-image] stripping kernel modules to virtio+fw_cfg only..."
+for kver_dir in "$CHROOT_DIR/lib/modules"/*/; do
+    kver="$(basename "$kver_dir")"
+    kernel_dir="$kver_dir/kernel"
+    [ -d "$kernel_dir" ] || continue
+
+    # Collect keeper modules and their relative paths from the kernel/ tree.
+    tmp_keep=$(mktemp -d)
+    find "$kernel_dir" -name "virtio*.ko.gz" -o -name "virtio*.ko" \
+                       -o -name "*fw_cfg*.ko.gz" -o -name "*fw_cfg*.ko" \
+                       -o -name "*failover*.ko.gz" -o -name "*failover*.ko" \
+        2>/dev/null | while read -r mod; do
+        rel="${mod#"$kernel_dir/"}"
+        dest_dir="$tmp_keep/$(dirname "$rel")"
+        mkdir -p "$dest_dir"
+        cp "$mod" "$dest_dir/"
+    done
+
+    # Replace kernel module tree with the keeper set.
+    rm -rf "$kernel_dir"
+    mkdir -p "$kernel_dir"
+    # Restore each kept file into the same relative path.
+    (cd "$tmp_keep" && find . -name "*.ko*" | while read -r f; do
+        dest_dir="$kernel_dir/$(dirname "${f#./}")"
+        mkdir -p "$dest_dir"
+        cp "$f" "$dest_dir/"
+    done)
+    rm -rf "$tmp_keep"
+
+    # Regenerate module dependency map from the surviving modules.
+    depmod -b "$CHROOT_DIR" "$kver" 2>/dev/null || true
+done
+
+# --- Strip package-manager artifacts (not needed at runtime) ---
+echo "[build-microvm-image] stripping package manager artifacts..."
+rm -rf \
+    "$CHROOT_DIR/lib/apk" \
+    "$CHROOT_DIR/var/cache/apk" \
+    "$CHROOT_DIR/etc/apk" \
+    "$CHROOT_DIR/usr/share/man" \
+    "$CHROOT_DIR/usr/share/doc" \
+    "$CHROOT_DIR/usr/share/apk" \
+    "$CHROOT_DIR/usr/include" \
+    "$CHROOT_DIR/usr/lib/pkgconfig"
+
+# --- Strip debug symbols from binaries ---
+echo "[build-microvm-image] stripping debug symbols from binaries..."
+find "$CHROOT_DIR/usr/sbin" "$CHROOT_DIR/usr/bin" \
+     "$CHROOT_DIR/sbin" "$CHROOT_DIR/bin" \
+     "$CHROOT_DIR/usr/local/bin" \
+    -type f 2>/dev/null | while read -r bin; do
+    strip --strip-all "$bin" 2>/dev/null || true
+done
+
+# --- Copy vmlinuz out before stripping /boot from chroot ---
+VMLINUZ_OUT="$BUILD_DIR/vmlinuz"
+echo "[build-microvm-image] copying kernel: $VMLINUZ_OUT"
+cp "$KERNEL_IMG" "$VMLINUZ_OUT"
+
+# Drop /boot from the chroot — vmlinuz, Alpine's stock initramfs-virt, System.map
+# and kernel config all live here (~25 MiB). None are needed at runtime: the
+# guest already has the kernel loaded by QEMU's -kernel, and Alpine's initramfs
+# is unused because we replace it with our own /init. Leaving /boot in the cpio
+# nearly doubles uncompressed initramfs size and forces a 512 MiB guest just to
+# survive decompression.
+rm -rf "$CHROOT_DIR/boot"
+
+# --- Build initramfs ---
+INITRAMFS_OUT="$BUILD_DIR/initramfs.cpio.gz"
+echo "[build-microvm-image] building initramfs: $INITRAMFS_OUT"
+(
+    cd "$CHROOT_DIR"
+    find . | cpio --quiet -o -H newc | gzip -9 > "$INITRAMFS_OUT"
+)
+
+# --- Log artifact sizes ---
+echo ""
+echo "[build-microvm-image] artifacts:"
+ls -lh "$VMLINUZ_OUT" "$INITRAMFS_OUT"
+echo ""
+
+# --- Phase 2 artifact size gate (50 MiB total) ---
+ARTIFACT_MAX_MiB="${MICROVM_ARTIFACT_MAX_MiB:-50}"
+total_bytes=$(( $(stat -c %s "$VMLINUZ_OUT") + $(stat -c %s "$INITRAMFS_OUT") ))
+max_bytes=$(( ARTIFACT_MAX_MiB * 1024 * 1024 ))
+total_mib=$(( (total_bytes + 1048575) / 1048576 ))
+if [ "$total_bytes" -gt "$max_bytes" ]; then
+    echo "ERROR: artifact size ${total_mib} MiB exceeds ${ARTIFACT_MAX_MiB} MiB gate" >&2
+    echo "       Set MICROVM_ARTIFACT_MAX_MiB=N to override during development." >&2
+    exit 1
+fi
+echo "[build-microvm-image] artifact size gate: PASS (${total_mib} MiB / ${ARTIFACT_MAX_MiB} MiB)"
+echo ""
+echo "[build-microvm-image] done."

scripts/build-system-image.sh

@@ -1,10 +1,11 @@
 #!/bin/bash
 set -euo pipefail
 
-# build-system-image.sh — Build a minimal Alpine AMI from a manifest
+# build-system-image.sh — Build a minimal system AMI from a manifest
 #
-# Creates a pre-baked Alpine Linux image with custom packages, binaries,
-# and setup scripts installed, ready for import as a Spinifex AMI.
+# Supports Alpine Linux and Ubuntu 24.04 as base distros.
+# Creates a pre-baked image with custom packages, binaries, and setup scripts
+# installed, ready for import as a Spinifex AMI.
 #
 # Requirements: qemu-nbd, qemu-img, sudo (for mount/chroot), curl
 # Usage: ./scripts/build-system-image.sh <manifest.conf> [--import]
@@ -46,8 +47,11 @@ done
 # shellcheck source=/dev/null
 source "$MANIFEST"
 
+# Default distro is alpine for backward compatibility
+DISTRO="${DISTRO:-alpine}"
+
 # Validate required manifest fields
-for field in IMAGE_NAME ALPINE_VERSION IMAGE_SIZE; do
+for field in IMAGE_NAME IMAGE_SIZE; do
     if [[ -z "${!field:-}" ]]; then
         echo "ERROR: Manifest missing required field: $field"
         exit 1
@@ -56,16 +60,50 @@ done
 
 # Derived paths
 BUILD_DIR="/tmp/${IMAGE_NAME}-image-build"
-ALPINE_IMAGE="generic_alpine-${ALPINE_VERSION}-x86_64-bios-cloudinit-r0.qcow2"
-ALPINE_URL="https://dl-cdn.alpinelinux.org/alpine/v${ALPINE_VERSION%.*}/releases/cloud/${ALPINE_IMAGE}"
-OUTPUT_IMAGE="${BUILD_DIR}/${IMAGE_NAME}-alpine.qcow2"
-OUTPUT_RAW="${BUILD_DIR}/${IMAGE_NAME}-alpine.raw"
 NBD_DEV="/dev/nbd0"
 MOUNT_DIR="${BUILD_DIR}/mnt"
 
+case "$DISTRO" in
+    alpine)
+        if [[ -z "${ALPINE_VERSION:-}" ]]; then
+            echo "ERROR: Manifest missing required field: ALPINE_VERSION"
+            exit 1
+        fi
+        SOURCE_IMAGE="generic_alpine-${ALPINE_VERSION}-x86_64-bios-cloudinit-r0.qcow2"
+        SOURCE_URL="https://dl-cdn.alpinelinux.org/alpine/v${ALPINE_VERSION%.*}/releases/cloud/${SOURCE_IMAGE}"
+        OUTPUT_IMAGE="${BUILD_DIR}/${IMAGE_NAME}-alpine.qcow2"
+        OUTPUT_RAW="${BUILD_DIR}/${IMAGE_NAME}-alpine.raw"
+        DISTRO_VERSION="${ALPINE_VERSION}"
+        ROOT_PART="${NBD_DEV}"
+        BOOT_MODE="bios"
+        ;;
+    ubuntu)
+        if [[ -z "${UBUNTU_VERSION:-}" ]]; then
+            echo "ERROR: Manifest missing required field: UBUNTU_VERSION"
+            exit 1
+        fi
+        SOURCE_IMAGE="ubuntu-${UBUNTU_VERSION}-minimal-cloudimg-amd64.img"
+        SOURCE_URL="https://cloud-images.ubuntu.com/minimal/releases/${UBUNTU_CODENAME:-noble}/release/${SOURCE_IMAGE}"
+        OUTPUT_IMAGE="${BUILD_DIR}/${IMAGE_NAME}-ubuntu.qcow2"
+        OUTPUT_RAW="${BUILD_DIR}/${IMAGE_NAME}-ubuntu.raw"
+        DISTRO_VERSION="${UBUNTU_VERSION}"
+        ROOT_PART="${NBD_DEV}p1"
+        BOOT_MODE="uefi"
+        ;;
+    *)
+        echo "ERROR: Unknown DISTRO: $DISTRO (supported: alpine, ubuntu)"
+        exit 1
+        ;;
+esac
+
 cleanup() {
     echo "Cleaning up..."
-    sudo umount "${MOUNT_DIR}" 2>/dev/null || true
+    # Ubuntu chroot has bind mounts that must be released first
+    sudo umount "${MOUNT_DIR}/dev/pts" 2>/dev/null || true
+    sudo umount "${MOUNT_DIR}/dev"     2>/dev/null || true
+    sudo umount "${MOUNT_DIR}/sys"     2>/dev/null || true
+    sudo umount "${MOUNT_DIR}/proc"    2>/dev/null || true
+    sudo umount "${MOUNT_DIR}"         2>/dev/null || true
     sudo qemu-nbd --disconnect "${NBD_DEV}" 2>/dev/null || true
     exec 9>&- 2>/dev/null || true  # release nbd lock if held
     echo "Done."
@@ -80,7 +118,7 @@ fi
 
 echo "=== System Image Builder ==="
 echo "Image:   ${IMAGE_NAME} — ${IMAGE_DESCRIPTION:-}"
-echo "Alpine:  ${ALPINE_VERSION}"
+echo "Distro:  ${DISTRO} ${DISTRO_VERSION}"
 echo "Size:    ${IMAGE_SIZE}"
 echo "Build:   ${BUILD_DIR}"
 echo ""
@@ -96,6 +134,11 @@ if ! command -v qemu-img &>/dev/null; then
     exit 1
 fi
 
+if [[ "$DISTRO" == "ubuntu" ]] && ! command -v parted &>/dev/null; then
+    echo "ERROR: parted not found. Install parted."
+    exit 1
+fi
+
 # Build binaries if BUILD_COMMANDS is set
 if [[ -n "${BUILD_COMMANDS:-}" ]]; then
     echo "Building binaries: ${BUILD_COMMANDS}"
@@ -105,7 +148,7 @@ if [[ -n "${BUILD_COMMANDS:-}" ]]; then
     fi
 fi
 
-# Verify binaries exist and are statically linked
+# Verify binaries exist; Alpine also requires static linking (uses musl)
 if [[ -n "${INSTALL_BINARIES:-}" ]]; then
     IFS=' ' read -ra BINARY_PAIRS <<< "$INSTALL_BINARIES"
     for pair in "${BINARY_PAIRS[@]}"; do
@@ -115,7 +158,7 @@ if [[ -n "${INSTALL_BINARIES:-}" ]]; then
             echo "ERROR: Binary not found: $src_path"
             exit 1
         fi
-        if ! file "$src_path" | grep -q "statically linked"; then
+        if [[ "$DISTRO" == "alpine" ]] && ! file "$src_path" | grep -q "statically linked"; then
             echo "ERROR: $src_path is not statically linked (Alpine uses musl — dynamic glibc binaries will fail)"
             echo "  Rebuild with: CGO_ENABLED=0 go build ..."
             exit 1
@@ -147,31 +190,32 @@ if [[ -f "$OUTPUT_RAW" ]] && [[ $(( $(date +%s) - $(stat -c %Y "$OUTPUT_RAW") ))
 
     if [[ "$DO_IMPORT" == true ]]; then
         echo "Importing as AMI..."
-        IMPORT_ARGS=(--file "$OUTPUT_RAW" --distro alpine --version "${ALPINE_VERSION}" --arch x86_64)
+        rm -f "$OUTPUT_IMAGE"
+        IMPORT_ARGS=(--file "$OUTPUT_RAW" --distro "${DISTRO}" --version "${DISTRO_VERSION}" --arch x86_64 --boot-mode "${BOOT_MODE}")
         if [[ -n "${SYSTEM_TAG:-}" ]]; then
             IMPORT_ARGS+=(--tag "$SYSTEM_TAG")
         fi
-        (cd "$PROJECT_DIR" && ./bin/spx admin images import "${IMPORT_ARGS[@]}")
+        spx admin images import "${IMPORT_ARGS[@]}"
     fi
     exit 0
 fi
 
 mkdir -p "$BUILD_DIR" "$MOUNT_DIR"
 
-# Step 1: Download Alpine cloud image
-if [[ -f "${BUILD_DIR}/${ALPINE_IMAGE}" ]]; then
-    echo "Alpine image already downloaded."
+# Step 1: Download base cloud image
+if [[ -f "${BUILD_DIR}/${SOURCE_IMAGE}" ]]; then
+    echo "Base image already downloaded."
 else
-    echo "Downloading Alpine ${ALPINE_VERSION} cloud image..."
-    if ! curl --fail -L -o "${BUILD_DIR}/${ALPINE_IMAGE}" "$ALPINE_URL"; then
-        rm -f "${BUILD_DIR}/${ALPINE_IMAGE}"
-        echo "ERROR: Failed to download Alpine image from $ALPINE_URL"
+    echo "Downloading ${DISTRO} ${DISTRO_VERSION} cloud image..."
+    if ! curl --fail -L -o "${BUILD_DIR}/${SOURCE_IMAGE}" "$SOURCE_URL"; then
+        rm -f "${BUILD_DIR}/${SOURCE_IMAGE}"
+        echo "ERROR: Failed to download image from $SOURCE_URL"
         exit 1
     fi
-    # Verify the download is a valid qcow2 image
-    if ! qemu-img info "${BUILD_DIR}/${ALPINE_IMAGE}" &>/dev/null; then
-        rm -f "${BUILD_DIR}/${ALPINE_IMAGE}"
-        echo "ERROR: Downloaded file is not a valid qcow2 image"
+    # Verify the download is a valid qcow2/img
+    if ! qemu-img info "${BUILD_DIR}/${SOURCE_IMAGE}" &>/dev/null; then
+        rm -f "${BUILD_DIR}/${SOURCE_IMAGE}"
+        echo "ERROR: Downloaded file is not a valid disk image"
         exit 1
     fi
 fi
@@ -179,9 +223,9 @@ fi
 # Step 2: Copy image for customization
 rm -f "$OUTPUT_IMAGE"
 echo "Copying image for customization..."
-cp "${BUILD_DIR}/${ALPINE_IMAGE}" "$OUTPUT_IMAGE"
+cp "${BUILD_DIR}/${SOURCE_IMAGE}" "$OUTPUT_IMAGE"
 
-# Resize the image (Alpine cloud images are ~200MB, need room for packages)
+# Resize the image to provide room for packages
 qemu-img resize "$OUTPUT_IMAGE" "$IMAGE_SIZE"
 
 # Step 3: Connect via qemu-nbd
@@ -194,7 +238,7 @@ fi
 sudo qemu-nbd --disconnect "${NBD_DEV}" 2>/dev/null || true
 sudo qemu-nbd --connect="${NBD_DEV}" "$OUTPUT_IMAGE"
 
-# Wait for the nbd device to be ready (kernel needs time to scan the block device)
+# Wait for the nbd device to be ready
 for i in $(seq 1 10); do
     if sudo blockdev --getsize64 "${NBD_DEV}" &>/dev/null; then
         break
@@ -203,32 +247,61 @@ for i in $(seq 1 10); do
 done
 sleep 1
 
+# Ubuntu images have a partition table; wait for partition devices then resize
+# the partition to fill the expanded image before resizing the filesystem.
+if [[ "$DISTRO" == "ubuntu" ]]; then
+    for i in $(seq 1 10); do
+        if [[ -e "${ROOT_PART}" ]]; then break; fi
+        sleep 1
+    done
+    echo "Resizing partition..."
+    # After qemu-img resize the GPT backup header sits at the old end of disk.
+    # Relocate it to the new end (sfdisk --relocate is util-linux built-in,
+    # no extra packages needed), then re-read the partition table before parted
+    # runs — without this, parted sees stale GPT geometry and fails with
+    # "Unable to satisfy all constraints on the partition."
+    sudo sgdisk --move-second-header "${NBD_DEV}"
+    sudo partprobe "${NBD_DEV}" 2>/dev/null || true
+    sleep 1
+    sudo parted --script "${NBD_DEV}" resizepart 1 100%
+fi
+
 # Alpine cloud images have ext4 directly on the block device (no partition table).
-# Resize the filesystem to fill the resized image.
 echo "Checking filesystem..."
-sudo e2fsck -f -y "${NBD_DEV}" || {
+sudo e2fsck -f -y "${ROOT_PART}" || {
     ec=$?
     if [[ $ec -gt 1 ]]; then
-        echo "ERROR: e2fsck failed with exit code $ec on ${NBD_DEV}"
+        echo "ERROR: e2fsck failed with exit code $ec on ${ROOT_PART}"
         exit 1
     fi
 }
 
 echo "Resizing filesystem..."
-if ! sudo resize2fs "${NBD_DEV}"; then
-    echo "ERROR: resize2fs failed on ${NBD_DEV}"
+if ! sudo resize2fs "${ROOT_PART}"; then
+    echo "ERROR: resize2fs failed on ${ROOT_PART}"
     exit 1
 fi
 
 # Step 4: Mount and customize
 echo "Mounting root filesystem..."
-sudo mount "${NBD_DEV}" "$MOUNT_DIR"
+sudo mount "${ROOT_PART}" "$MOUNT_DIR"
 
-# Set up resolv.conf for DNS inside chroot
+# Set up resolv.conf for DNS inside chroot.
+# Ubuntu cloud images symlink /etc/resolv.conf → /run/systemd/resolve/stub-resolv.conf
+# which doesn't exist yet. Remove the symlink and write a real file.
+sudo rm -f "${MOUNT_DIR}/etc/resolv.conf"
 sudo cp /etc/resolv.conf "${MOUNT_DIR}/etc/resolv.conf"
 
+# Ubuntu chroot requires /proc /sys /dev bind mounts for systemd and DKMS
+if [[ "$DISTRO" == "ubuntu" ]]; then
+    sudo mount --bind /proc       "${MOUNT_DIR}/proc"
+    sudo mount --bind /sys        "${MOUNT_DIR}/sys"
+    sudo mount --bind /dev        "${MOUNT_DIR}/dev"
+    sudo mount --bind /dev/pts    "${MOUNT_DIR}/dev/pts"
+fi
+
 # Install packages
-if [[ -n "${APK_PACKAGES:-}" ]]; then
+if [[ "$DISTRO" == "alpine" ]] && [[ -n "${APK_PACKAGES:-}" ]]; then
     echo "Installing packages: ${APK_PACKAGES}..."
     sudo chroot "$MOUNT_DIR" /bin/sh -c "
 set -e
@@ -244,16 +317,30 @@ fi
 apk update
 apk add ${APK_PACKAGES}
 "
+elif [[ "$DISTRO" == "ubuntu" ]] && [[ -n "${APT_PACKAGES:-}" ]]; then
+    echo "Installing packages: ${APT_PACKAGES}..."
+    sudo chroot "$MOUNT_DIR" /bin/bash -c "
+export DEBIAN_FRONTEND=noninteractive
+apt-get update
+apt-get install -y --no-install-recommends ${APT_PACKAGES}
+"
 fi
 
-# Enable OpenRC services
+# Enable services
 if [[ -n "${ENABLE_SERVICES:-}" ]]; then
     echo "Enabling services: ${ENABLE_SERVICES}..."
     IFS=' ' read -ra SERVICES <<< "$ENABLE_SERVICES"
     for svc in "${SERVICES[@]}"; do
-        if ! sudo chroot "$MOUNT_DIR" /bin/sh -c "rc-update add ${svc} default"; then
-            echo "ERROR: Failed to enable service '${svc}' — does it exist in the image?"
-            exit 1
+        if [[ "$DISTRO" == "alpine" ]]; then
+            if ! sudo chroot "$MOUNT_DIR" /bin/sh -c "rc-update add ${svc} default"; then
+                echo "ERROR: Failed to enable service '${svc}' — does it exist in the image?"
+                exit 1
+            fi
+        else
+            if ! sudo chroot "$MOUNT_DIR" /bin/bash -c "systemctl enable ${svc}"; then
+                echo "ERROR: Failed to enable service '${svc}'"
+                exit 1
+            fi
         fi
     done
 fi
@@ -282,21 +369,40 @@ if [[ -n "${SETUP_SCRIPT:-}" ]]; then
     echo "Running setup script: ${SETUP_SCRIPT}..."
     sudo cp "$setup_path" "${MOUNT_DIR}/tmp/setup.sh"
     sudo chmod 755 "${MOUNT_DIR}/tmp/setup.sh"
-    sudo chroot "$MOUNT_DIR" /tmp/setup.sh
+    if [[ "$DISTRO" == "ubuntu" ]]; then
+        sudo chroot "$MOUNT_DIR" /bin/bash /tmp/setup.sh
+    else
+        sudo chroot "$MOUNT_DIR" /tmp/setup.sh
+    fi
     sudo rm -f "${MOUNT_DIR}/tmp/setup.sh"
 fi
 
 # Step 6: Clean up and unmount
 echo "Cleaning up image..."
-sudo chroot "$MOUNT_DIR" /bin/sh -c '
+if [[ "$DISTRO" == "alpine" ]]; then
+    sudo chroot "$MOUNT_DIR" /bin/sh -c '
 apk cache clean 2>/dev/null || true
 rm -rf /var/cache/apk/* /tmp/*
 '
+else
+    sudo chroot "$MOUNT_DIR" /bin/bash -c '
+export DEBIAN_FRONTEND=noninteractive
+apt-get clean
+rm -rf /var/lib/apt/lists/* /tmp/*
+'
+fi
 
-# Restore original resolv.conf (cloud-init will set it on boot)
+# Restore the systemd-resolved symlink (Ubuntu default); cloud-init sets DNS on boot.
 sudo rm -f "${MOUNT_DIR}/etc/resolv.conf"
+sudo ln -sf /run/systemd/resolve/stub-resolv.conf "${MOUNT_DIR}/etc/resolv.conf"
 
 echo "Unmounting..."
+if [[ "$DISTRO" == "ubuntu" ]]; then
+    sudo umount "${MOUNT_DIR}/dev/pts"
+    sudo umount "${MOUNT_DIR}/dev"
+    sudo umount "${MOUNT_DIR}/sys"
+    sudo umount "${MOUNT_DIR}/proc"
+fi
 sudo umount "$MOUNT_DIR"
 sudo qemu-nbd --disconnect "${NBD_DEV}"
 
@@ -318,19 +424,20 @@ echo ""
 
 if [[ "$DO_IMPORT" == true ]]; then
     echo "Importing as AMI..."
-    IMPORT_ARGS=(--file "$OUTPUT_RAW" --distro alpine --version "${ALPINE_VERSION}" --arch x86_64)
+    rm -f "$OUTPUT_IMAGE"
+    IMPORT_ARGS=(--file "$OUTPUT_RAW" --distro "${DISTRO}" --version "${DISTRO_VERSION}" --arch x86_64 --boot-mode "${BOOT_MODE}")
     if [[ -n "${SYSTEM_TAG:-}" ]]; then
         IMPORT_ARGS+=(--tag "$SYSTEM_TAG")
     fi
-    (cd "$PROJECT_DIR" && sudo -u spinifex-storage ./bin/spx admin images import "${IMPORT_ARGS[@]}")
+    spx admin images import "${IMPORT_ARGS[@]}"
 else
     echo "To import as AMI, run:"
-    echo "  cd $PROJECT_DIR && ./bin/spx admin images import \\"
+    echo "  spx admin images import \\"
     echo "    --file $OUTPUT_RAW \\"
     if [[ -n "${SYSTEM_TAG:-}" ]]; then
-        echo "    --distro alpine --version ${ALPINE_VERSION} --arch x86_64 \\"
+        echo "    --distro ${DISTRO} --version ${DISTRO_VERSION} --arch x86_64 --boot-mode ${BOOT_MODE} \\"
         echo "    --tag ${SYSTEM_TAG}"
     else
-        echo "    --distro alpine --version ${ALPINE_VERSION} --arch x86_64"
+        echo "    --distro ${DISTRO} --version ${DISTRO_VERSION} --arch x86_64 --boot-mode ${BOOT_MODE}"
     fi
 fi

scripts/check-coverage.sh

@@ -5,7 +5,7 @@ set -euo pipefail
 # Usage: scripts/check-coverage.sh <coverprofile> [quiet]
 
 PROFILE="${1:?Usage: check-coverage.sh <coverprofile> [quiet]}"
-MIN=65.0
+MIN=70.0
 QUIET="${2:-}"
 
 if [[ ! -s "$PROFILE" ]]; then