release: update go.mod for v1.5.0

Self-hosted runner accumulated 18GB in overlay2 from leaked container
layers. Two causes: build-microvm-image.sh only removed the alpine
container on the success path, and no workflow step pruned docker
between runs.

- scripts/build-microvm-image.sh: container id now removed via the
  existing EXIT trap, so failures, signals, and runner kills no longer
  leak the writable layer.
- .github/workflows/e2e-go.yml: prune containers / dangling images /
  buildx cache before each Build Install Artifacts step, in all three
  e2e jobs (single, multinode, reboot).

.github/actions/e2e-analyze/README.md

@@ -0,0 +1,104 @@
+# e2e-analyze
+
+Post-test failure-clustering action for the Go E2E suites.
+
+## What it does
+
+Runs after `go-junit-report` has converted each suite's stdout into
+`junit-<suite>.xml`. For every JUnit file matched by `junit-glob`:
+
+1. Parses each `<testcase>` with a `<failure>` / `<error>` child.
+2. Strips ANSI escapes, then normalises dynamic noise (UUIDs, EC2 IDs,
+   IPs, timestamps, request-ids, source line numbers) to produce a stable
+   _signature_ per failure.
+3. Picks the **earliest non-cascade** failure per suite as the likely
+   root cause. "Cascade" = failure message matches a fixture-dependency
+   marker (`must populate fix.`, `Should NOT be empty`,
+   `Expected value not to be nil`). When every failure is a cascade
+   (whole suite tripped on one upstream gap), the earliest cascade is
+   the fallback root.
+4. Buckets the remaining failures by signature and renders a markdown
+   report. The report is appended to `$GITHUB_STEP_SUMMARY` and also
+   written to `<log-dir>/analysis.md` so it travels with the uploaded
+   artifact bundle.
+
+The action **never fails the job** — exit-code semantics stay with the
+existing "Fail job if any suite failed" gate.
+
+## Inputs
+
+| Input        | Required | Description |
+|--------------|----------|-------------|
+| `junit-glob` | yes      | Shell glob for the JUnit XML files (e.g. `${ARTIFACT_DIR}/junit-*.xml`). |
+| `log-dir`    | yes      | Directory the rendered report is written into (`analysis.md`). |
+| `title`      | no       | Heading at the top of the summary section. Default `E2E failure analysis`. |
+
+## Outputs
+
+None. The summary is appended to `$GITHUB_STEP_SUMMARY`; the same
+content lands at `<log-dir>/analysis.md`.
+
+## Local invocation
+
+Outside Actions, the analyser prints to stdout instead of appending to a
+summary file:
+
+```sh
+cd spinifex/.github/actions/e2e-analyze
+go run . -junit-glob "/tmp/artifacts/junit-*.xml" -log-dir /tmp/artifacts
+```
+
+## Tests
+
+Golden-file tests live in `testdata/`. Each subdir holds a scenario:
+
+| Scenario           | Source                                    |
+|--------------------|-------------------------------------------|
+| `cascade-vpcs`     | Synthetic: one Phase 5 root + 15 cascades |
+| `single-failure`   | Real artifact from run `26037892507`      |
+| `no-failures`      | Synthetic: clean run                      |
+
+Run normally:
+
+```sh
+go test ./.github/actions/e2e-analyze/
+```
+
+When the report format changes intentionally, regenerate the snapshots:
+
+```sh
+go test ./.github/actions/e2e-analyze/ -update
+```
+
+## Per-failure bundles (Stage 2)
+
+After parsing, the action materialises a bundle directory per failure
+under `<log-dir>/analysis/NN-<TestName>/`. The report links to it from
+each entry so a reader on the Actions page can click straight through
+to the relevant log slice.
+
+Each bundle contains:
+
+| File              | Source                                                                            |
+|-------------------|-----------------------------------------------------------------------------------|
+| `failure.txt`     | Synthesised summary: test, start, duration, file-site, error line.                |
+| `journal.log`     | `spinifex-journal.log` sliced to `[StartAt − 5s, StartAt + Duration + 5s]`.       |
+| `test.log`        | `test-<suite>.log` block between `=== RUN <name>` and the matching `--- FAIL`.    |
+| `<other files>`   | Anything the harness already dropped into `<log-dir>/<TestName>/` at run-time.    |
+
+The journal slice relies on the workflow running
+`journalctl --output=short-iso` so timestamps are parseable. Lines
+without a parseable timestamp inherit the previous line's
+in-window/out-of-window classification (so multi-line records and stack
+traces stay contiguous).
+
+## Roadmap
+
+- **Stage 1 (shipped):** first-failure surfacing + cascade clustering.
+- **Stage 2 (this update):** time-window correlation — per-failure bundle
+  with sliced journal + test-log + any per-test artifacts.
+- **Stage 3:** rule-based root-cause classifier driven by
+  `spinifex/tests/e2e/.failure-rules.yaml`.
+
+See `docs/development/improvements/e2e-go-failure-analysis.md` in the
+parent mulga repo.

.github/actions/e2e-analyze/action.yml

@@ -0,0 +1,38 @@
+name: E2E Failure Analyzer
+description: >-
+  Post-test analyzer for the Go E2E suites. Parses junit-*.xml emitted by
+  go-junit-report, clusters failures by signature, picks the earliest
+  non-cascade failure per suite as the likely root cause, and appends a
+  human-readable report to $GITHUB_STEP_SUMMARY. Always exits 0 — the
+  existing "Fail job if any suite failed" gate keeps owning exit-code
+  semantics.
+
+inputs:
+  junit-glob:
+    description: Shell glob matching JUnit XML files to analyze.
+    required: true
+  log-dir:
+    description: >-
+      Artifact directory. The rendered report is also written to
+      $log-dir/analysis.md so it travels with the uploaded artifact.
+    required: true
+  title:
+    description: Heading rendered at the top of the summary section.
+    required: false
+    default: "E2E failure analysis"
+
+runs:
+  using: composite
+  steps:
+    - id: analyze
+      shell: bash
+      working-directory: ${{ github.action_path }}
+      env:
+        ANALYZE_JUNIT_GLOB: ${{ inputs.junit-glob }}
+        ANALYZE_LOG_DIR: ${{ inputs.log-dir }}
+        ANALYZE_TITLE: ${{ inputs.title }}
+      run: |
+        go run . \
+          -junit-glob "$ANALYZE_JUNIT_GLOB" \
+          -log-dir "$ANALYZE_LOG_DIR" \
+          -title "$ANALYZE_TITLE"

.github/actions/e2e-analyze/analyze.go

@@ -0,0 +1,552 @@
+// Package main implements the e2e-analyze GitHub Action.
+//
+// Stage 1 of docs/development/improvements/e2e-go-failure-analysis.md:
+// cluster failures from go-junit-report's junit-*.xml by error signature,
+// pick the earliest non-cascade failure per suite as the likely root cause,
+// and render a human-readable report.
+//
+// This file contains the pure logic (parsing, signature extraction,
+// clustering, rendering). Side-effect handling lives in main.go.
+package main
+
+import (
+	"encoding/xml"
+	"fmt"
+	"path/filepath"
+	"regexp"
+	"sort"
+	"strings"
+	"time"
+)
+
+// JUnit XML schema produced by go-junit-report v2. Stable across versions
+// we care about; only fields used downstream are decoded.
+type junitSuites struct {
+	XMLName xml.Name     `xml:"testsuites"`
+	Suites  []junitSuite `xml:"testsuite"`
+}
+
+type junitSuite struct {
+	Name      string    `xml:"name,attr"`
+	Tests     int       `xml:"tests,attr"`
+	Failures  int       `xml:"failures,attr"`
+	Errors    int       `xml:"errors,attr"`
+	Skipped   int       `xml:"skipped,attr"`
+	Timestamp string    `xml:"timestamp,attr"`
+	Time      float64   `xml:"time,attr"`
+	Cases     []junitTC `xml:"testcase"`
+}
+
+type junitTC struct {
+	Name      string        `xml:"name,attr"`
+	Classname string        `xml:"classname,attr"`
+	Time      float64       `xml:"time,attr"`
+	Failure   *junitFailure `xml:"failure"`
+	Error     *junitFailure `xml:"error"`
+	Skipped   *junitSkipped `xml:"skipped"`
+	SystemOut string        `xml:"system-out"`
+}
+
+type junitFailure struct {
+	Message string `xml:"message,attr"`
+	Type    string `xml:"type,attr"`
+	Body    string `xml:",chardata"`
+}
+
+type junitSkipped struct {
+	Message string `xml:"message,attr"`
+}
+
+// Failure is the analyzer's normalized view of one failed testcase.
+type Failure struct {
+	SuiteFile string    // e.g. "single" (derived from junit-single.xml)
+	Name      string    // e.g. "TestSingleNode/Phase5_LaunchInstance"
+	Duration  float64   // seconds
+	Order     int       // position within the suite (0-based)
+	StartAt   time.Time // estimated wall-clock start: suite.timestamp + sum(prior durations)
+	Error     string    // last meaningful error line, ANSI-stripped
+	FileHint  string    // e.g. "vpc_test.go:227", parsed from failure body
+	Cascade   bool      // signature matched a known downstream pattern
+	Signature string    // normalized error key for bucketing
+	// OffsetFromSuiteStart is sum(prior testcase durations) at the time
+	// this test started. Persisted separately from StartAt so a caller
+	// that has a more authoritative suite wall-clock origin (e.g. a
+	// `test-<suite>.start` file written by the workflow before the suite
+	// runs) can recompute StartAt without re-parsing the XML.
+	OffsetFromSuiteStart time.Duration
+	// BundlePath is the relative path (from log-dir) to the per-failure
+	// bundle written by Stage 2's WriteBundles. Empty when bundles were
+	// not generated (e.g. unit tests that call Render directly).
+	BundlePath string
+}
+
+// SuiteReport summarises one junit-*.xml file.
+type SuiteReport struct {
+	File      string // basename, e.g. "junit-single.xml"
+	Label     string // short name, e.g. "single"
+	Total     int
+	FailCount int
+	Root      *Failure    // nil if no failures
+	Cascades  []Failure   // everything else that failed in this suite
+	Buckets   [][]Failure // failures grouped by signature (root's bucket first)
+}
+
+// Report is the full analysis across every junit file.
+type Report struct {
+	Title  string
+	Suites []SuiteReport
+}
+
+// ansiSeq matches ANSI CSI escapes the harness sprinkles through log
+// output (colours, bold). Three forms in the wild:
+//   - real ESC: `\x1b[36m…`
+//   - U+FFFD replacement (XML serialiser dropped the ESC byte): `�[36m…`
+//   - bare CSI with the leading byte gone entirely: `[36m…`
+//
+// All three are stripped so colour state doesn't fork the signature bucket.
+var ansiSeq = regexp.MustCompile(
+	"\x1b\\[[0-9;]*[a-zA-Z]" + // real ESC CSI
+		"|�\\[[0-9;]*[a-zA-Z]" + // replacement-char CSI
+		"|\\[[0-9]{1,2}(?:;[0-9]{1,2})*m", // bare colour code, "mode" form only
+)
+
+// cascadeMarkers are message fragments that only appear in tests that
+// depend on a fixture field populated by an earlier phase. Their presence
+// means the failure is downstream of some other failure that happened
+// first; the report collapses them under that root cause.
+var cascadeMarkers = []string{
+	"must populate fix.",
+	"Should NOT be empty",
+	"Expected value not to be nil",
+}
+
+// noiseReplacers normalise dynamic identifiers so that "ten failures, same
+// bug" bucket together. Pattern ordering matters: longer/more-specific
+// patterns must precede the catch-alls.
+var noiseReplacers = []struct {
+	re   *regexp.Regexp
+	repl string
+}{
+	{regexp.MustCompile(`\b(vpc|subnet|sg|rtb|igw|i|ami|vol|snap|rtbassoc|key)-[0-9a-f]{8,}\b`), `${1}-<id>`},
+	{regexp.MustCompile(`\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b`), `<uuid>`},
+	{regexp.MustCompile(`\b[0-9a-f]{32,}\b`), `<hex>`},
+	{regexp.MustCompile(`\b\d{1,3}(\.\d{1,3}){3}(:\d+)?\b`), `<ip>`},
+	{regexp.MustCompile(`\b\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:?\d{2})?\b`), `<ts>`},
+	{regexp.MustCompile(`request-id:\s*\S+`), `request-id: <id>`},
+	{regexp.MustCompile(`(_test\.go):\d+`), `${1}:<line>`},
+	{regexp.MustCompile(`\s+`), ` `},
+}
+
+// fileHintRe pulls the last "foo_test.go:NNN" we see in a failure body so
+// the report can point a reader at the source line. Test files dominate the
+// hint set; harness/helper file matches go through goFileLineRe below.
+var fileHintRe = regexp.MustCompile(`([A-Za-z0-9_]+_test\.go:\d+)`)
+
+// goFileLineRe matches any "foo.go:NNN:" prefix (test or non-test). Used to
+// surface harness/helper-level error text like
+// `ec2helpers.go:164: describe-vpcs: InvalidParameterValue …` which Go test
+// emits when a non-test file calls t.Errorf.
+var goFileLineRe = regexp.MustCompile(`^[A-Za-z0-9_]+\.go:\d+:\s*(.+)$`)
+
+// testifyErrorPlaceholder is testify's "Error:" boilerplate when the
+// assertion failed because of a wrapped error. The next non-blank line in
+// the trace holds the actual error text.
+const testifyErrorPlaceholder = "Received unexpected error:"
+
+// stripANSI removes CSI escapes; leaves everything else intact.
+func stripANSI(s string) string {
+	return ansiSeq.ReplaceAllString(s, "")
+}
+
+// extractErrorLine finds the most useful single-line summary in a failure
+// body. Priority order, highest first:
+//  1. Testify "Messages:" line — user-supplied context, the most signal-rich.
+//  2. Testify "Error:" content (with one-line lookahead when the placeholder
+//     "Received unexpected error:" defers to the next line).
+//  3. Leading "foo.go:NNN: text" content line — t.Errorf from non-test code.
+//  4. Last `_test.go:NNN: text` line — bare t.Fatalf/Errorf from a test file.
+//  5. First non-empty line of the body.
+func extractErrorLine(body string) string {
+	body = stripANSI(body)
+	lines := strings.Split(body, "\n")
+
+	var (
+		msgLine      string
+		errLine      string
+		goFileLine   string
+		lastTestLine string
+	)
+	for i, raw := range lines {
+		l := strings.TrimSpace(raw)
+		if l == "" {
+			continue
+		}
+		if strings.HasPrefix(l, "Messages:") {
+			msgLine = strings.TrimSpace(strings.TrimPrefix(l, "Messages:"))
+			continue
+		}
+		if strings.HasPrefix(l, "Error:") && errLine == "" {
+			content := strings.TrimSpace(strings.TrimPrefix(l, "Error:"))
+			if content == testifyErrorPlaceholder {
+				// Look ahead for the wrapped error text on the next non-blank line.
+				for j := i + 1; j < len(lines); j++ {
+					nxt := strings.TrimSpace(lines[j])
+					if nxt == "" {
+						continue
+					}
+					if strings.HasPrefix(nxt, "Test:") || strings.HasPrefix(nxt, "Messages:") {
+						break
+					}
+					content = content + " " + nxt
+					break
+				}
+			}
+			errLine = content
+			continue
+		}
+		if m := goFileLineRe.FindStringSubmatch(l); m != nil {
+			// Keep the LAST occurrence: t.Log / harness traces emit
+			// progress lines from the same file:line pattern before the
+			// failing assertion finally fires.
+			goFileLine = strings.TrimSpace(m[1])
+			continue
+		}
+		if fileHintRe.MatchString(l) {
+			lastTestLine = l
+		}
+	}
+	switch {
+	case msgLine != "":
+		return msgLine
+	case errLine != "":
+		return errLine
+	case goFileLine != "":
+		return goFileLine
+	case lastTestLine != "":
+		// "vpc_test.go:227: Eventually: condition not met …" → trim the file prefix.
+		if i := strings.Index(lastTestLine, ": "); i > 0 {
+			return strings.TrimSpace(lastTestLine[i+2:])
+		}
+		return lastTestLine
+	}
+	// Last resort: first non-empty line of the body.
+	for _, raw := range lines {
+		l := strings.TrimSpace(raw)
+		if l != "" {
+			return l
+		}
+	}
+	return ""
+}
+
+// errorTraceRe extracts the path from testify's "Error Trace:" line — e.g.
+//
+//	Error Trace:	tests/e2e/harness/ec2helpers.go:164
+//
+// — which points at the assertion site even when it lives in helper code
+// outside *_test.go.
+var errorTraceRe = regexp.MustCompile(`Error Trace:\s*(\S+\.go:\d+)`)
+
+// extractFileHint returns the most precise pointer at the assertion site
+// we can pull from the failure body: testify Error Trace wins, otherwise
+// the last `*_test.go:NNN` line in the body.
+func extractFileHint(body string) string {
+	body = stripANSI(body)
+	if m := errorTraceRe.FindStringSubmatch(body); m != nil {
+		return m[1]
+	}
+	all := fileHintRe.FindAllString(body, -1)
+	if len(all) == 0 {
+		return ""
+	}
+	return all[len(all)-1]
+}
+
+// signature normalises an error line so different runs of the same root
+// cause bucket together. Empty signatures (e.g. parent-test placeholders
+// where the failure body is empty) bucket together under "".
+func signature(errLine string) string {
+	s := strings.ToLower(errLine)
+	for _, r := range noiseReplacers {
+		s = r.re.ReplaceAllString(s, r.repl)
+	}
+	return strings.TrimSpace(s)
+}
+
+// isCascade reports whether the error line contains any of the well-known
+// fixture-propagation markers — i.e. "this test failed because something
+// upstream of it failed first".
+func isCascade(errLine string) bool {
+	for _, m := range cascadeMarkers {
+		if strings.Contains(errLine, m) {
+			return true
+		}
+	}
+	return false
+}
+
+// computeLeafSet returns the testcase names that are leaves — i.e. no
+// other testcase name in `cases` starts with `<name>/`. go-junit-report
+// emits one <testcase> for the parent Go test AND each subtest, with the
+// parent's time = sum of its children, so summing every entry into the
+// suite's cumulative offset double-counts. Restrict accumulation to
+// leaves only.
+func computeLeafSet(cases []junitTC) map[string]bool {
+	leaf := make(map[string]bool, len(cases))
+	for _, tc := range cases {
+		leaf[tc.Name] = true
+	}
+	for _, tc := range cases {
+		// Strip the deepest path segment; any ancestor of this test is
+		// not a leaf.
+		name := tc.Name
+		for {
+			i := strings.LastIndex(name, "/")
+			if i <= 0 {
+				break
+			}
+			name = name[:i]
+			if _, ok := leaf[name]; ok {
+				leaf[name] = false
+			}
+		}
+	}
+	return leaf
+}
+
+// parseStartTime turns a junit timestamp attr (RFC3339) into a time.Time;
+// zero value if unparseable.
+func parseStartTime(s string) time.Time {
+	if s == "" {
+		return time.Time{}
+	}
+	t, err := time.Parse(time.RFC3339, s)
+	if err != nil {
+		return time.Time{}
+	}
+	return t
+}
+
+// suiteLabel turns "junit-single.xml" into "single".
+func suiteLabel(path string) string {
+	base := filepath.Base(path)
+	base = strings.TrimSuffix(base, ".xml")
+	base = strings.TrimPrefix(base, "junit-")
+	if base == "" {
+		return filepath.Base(path)
+	}
+	return base
+}
+
+// ParseFile decodes one junit-*.xml file into our normalized failure list.
+// Returns the suite label, total testcase count, and the (ordered) failures.
+func ParseFile(path string, data []byte) (SuiteReport, error) {
+	var doc junitSuites
+	if err := xml.Unmarshal(data, &doc); err != nil {
+		return SuiteReport{}, fmt.Errorf("parse %s: %w", path, err)
+	}
+
+	rep := SuiteReport{
+		File:  filepath.Base(path),
+		Label: suiteLabel(path),
+	}
+
+	order := 0
+	for _, suite := range doc.Suites {
+		start := parseStartTime(suite.Timestamp)
+		// go-junit-report emits a testcase for both the parent Go test
+		// (e.g. `TestSingleNode`) AND each subtest (`TestSingleNode/PhaseX`).
+		// The parent's `time` attribute is the aggregate of its children, so
+		// summing every testcase double-counts. Pre-compute the leaf set so
+		// only leaves contribute to the suite's running offset.
+		isLeaf := computeLeafSet(suite.Cases)
+		cumul := 0.0
+		for _, tc := range suite.Cases {
+			rep.Total++
+			leaf := isLeaf[tc.Name]
+			var body string
+			switch {
+			case tc.Failure != nil:
+				body = tc.Failure.Body
+			case tc.Error != nil:
+				body = tc.Error.Body
+			default:
+				if leaf {
+					cumul += tc.Time
+				}
+				continue
+			}
+
+			// Skip the synthetic parent-test "Failed" stub that go-junit-report
+			// emits for any Go test whose subtests failed. It has an empty body
+			// and would otherwise outrank real failures by XML order.
+			trimmed := strings.TrimSpace(stripANSI(body))
+			if trimmed == "" {
+				if leaf {
+					cumul += tc.Time
+				}
+				continue
+			}
+
+			errLine := extractErrorLine(body)
+			offset := time.Duration(cumul * float64(time.Second))
+			f := Failure{
+				SuiteFile:            rep.Label,
+				Name:                 tc.Name,
+				Duration:             tc.Time,
+				Order:                order,
+				Error:                errLine,
+				FileHint:             extractFileHint(body),
+				Cascade:              isCascade(errLine),
+				Signature:            signature(errLine),
+				OffsetFromSuiteStart: offset,
+			}
+			if !start.IsZero() {
+				f.StartAt = start.Add(offset)
+			}
+			rep.FailCount++
+			rep.Cascades = append(rep.Cascades, f) // temp staging; split below
+			order++
+			if leaf {
+				cumul += tc.Time
+			}
+		}
+	}
+
+	if rep.FailCount == 0 {
+		rep.Cascades = nil
+		return rep, nil
+	}
+
+	all := rep.Cascades
+	rep.Cascades = nil
+
+	// Root = earliest non-cascade. If every failure is a cascade (the whole
+	// suite tripped on the same upstream gap), fall back to earliest overall.
+	rootIdx := -1
+	for i, f := range all {
+		if !f.Cascade {
+			rootIdx = i
+			break
+		}
+	}
+	if rootIdx == -1 {
+		rootIdx = 0
+	}
+	root := all[rootIdx]
+	rep.Root = &root
+
+	// Bucket the remainder by signature so the report can collapse the
+	// "ten failures, same bug" pattern.
+	bucketKeys := []string{}
+	buckets := map[string][]Failure{}
+	for i, f := range all {
+		if i == rootIdx {
+			continue
+		}
+		if _, seen := buckets[f.Signature]; !seen {
+			bucketKeys = append(bucketKeys, f.Signature)
+		}
+		buckets[f.Signature] = append(buckets[f.Signature], f)
+		rep.Cascades = append(rep.Cascades, f)
+	}
+
+	// Largest bucket first; ties broken by first-seen order so output is
+	// deterministic across runs.
+	sort.SliceStable(bucketKeys, func(i, j int) bool {
+		return len(buckets[bucketKeys[i]]) > len(buckets[bucketKeys[j]])
+	})
+	for _, k := range bucketKeys {
+		rep.Buckets = append(rep.Buckets, buckets[k])
+	}
+
+	return rep, nil
+}
+
+// Render writes the markdown report. Format is documented in
+// docs/development/improvements/e2e-go-failure-analysis.md (Stage 1).
+func Render(r Report) string {
+	var b strings.Builder
+	fmt.Fprintf(&b, "## %s\n\n", r.Title)
+
+	totalFail := 0
+	for _, s := range r.Suites {
+		totalFail += s.FailCount
+	}
+	if totalFail == 0 {
+		b.WriteString("✅ No failures across any suite.\n")
+		return b.String()
+	}
+
+	for _, s := range r.Suites {
+		if s.FailCount == 0 {
+			fmt.Fprintf(&b, "### Suite `%s`: ✅ pass (%d tests)\n\n", s.Label, s.Total)
+			continue
+		}
+		fmt.Fprintf(&b, "### Suite `%s`: %d failed, 1 root cause likely\n\n", s.Label, s.FailCount)
+
+		if s.Root != nil {
+			b.WriteString("**Root cause (earliest non-cascade)**\n\n")
+			fmt.Fprintf(&b, "- Test: `%s`\n", s.Root.Name)
+			if !s.Root.StartAt.IsZero() {
+				fmt.Fprintf(&b, "- Start: %s (duration %.1fs)\n",
+					s.Root.StartAt.Format("15:04:05"), s.Root.Duration)
+			} else {
+				fmt.Fprintf(&b, "- Duration: %.1fs\n", s.Root.Duration)
+			}
+			if s.Root.Error != "" {
+				fmt.Fprintf(&b, "- Error: %s\n", oneLine(s.Root.Error))
+			}
+			if s.Root.FileHint != "" {
+				fmt.Fprintf(&b, "- File:  `%s`\n", s.Root.FileHint)
+			}
+			if s.Root.BundlePath != "" {
+				fmt.Fprintf(&b, "- Bundle: `%s/`\n", s.Root.BundlePath)
+			}
+			b.WriteString("\n")
+		}
+
+		if len(s.Buckets) > 0 {
+			fmt.Fprintf(&b, "**Cascaded failures (%d) — grouped by signature**\n\n",
+				len(s.Cascades))
+			for _, bucket := range s.Buckets {
+				sample := bucket[0]
+				label := oneLine(sample.Error)
+				if label == "" {
+					label = "(no error message)"
+				}
+				fmt.Fprintf(&b, "- %d× %s\n", len(bucket), label)
+				// List up to 5 test names per bucket; collapse the rest.
+				const cap = 5
+				for i, f := range bucket {
+					if i == cap && len(bucket) > cap {
+						fmt.Fprintf(&b, "  - … and %d more\n", len(bucket)-cap)
+						break
+					}
+					if f.BundlePath != "" {
+						fmt.Fprintf(&b, "  - `%s` → `%s/`\n", f.Name, f.BundlePath)
+					} else {
+						fmt.Fprintf(&b, "  - `%s`\n", f.Name)
+					}
+				}
+			}
+			b.WriteString("\n")
+		}
+	}
+
+	return b.String()
+}
+
+// oneLine collapses whitespace and caps length so a runaway assertion
+// message doesn't blow out the summary card width.
+func oneLine(s string) string {
+	s = strings.TrimSpace(s)
+	s = strings.ReplaceAll(s, "\n", " ")
+	s = strings.ReplaceAll(s, "\r", " ")
+	s = regexp.MustCompile(`\s+`).ReplaceAllString(s, " ")
+	const max = 160
+	if len(s) > max {
+		s = s[:max-1] + "…"
+	}
+	return s
+}

.github/actions/e2e-analyze/analyze_test.go

@@ -0,0 +1,191 @@
+package main
+
+import (
+	"flag"
+	"os"
+	"path/filepath"
+	"sort"
+	"strings"
+	"testing"
+)
+
+// updateGolden lets `go test ./.github/actions/e2e-analyze/ -update` rewrite
+// the expected analysis.md files when the rendering format changes
+// intentionally. Day-to-day, leave it unset: tests diff against the on-disk
+// snapshot to catch unintended drift in signature extraction or layout.
+var updateGolden = flag.Bool("update", false, "rewrite testdata/*/analysis.md from current output")
+
+// TestGoldenFixtures runs the full ParseFile + Render pipeline against each
+// fixture directory under testdata/ and asserts the output matches the
+// adjacent analysis.md snapshot. Each subdir is one named scenario; the
+// fixture XMLs match the same junit-*.xml glob the production action uses
+// so behaviour stays representative of CI.
+func TestGoldenFixtures(t *testing.T) {
+	entries, err := os.ReadDir("testdata")
+	if err != nil {
+		t.Fatalf("read testdata: %v", err)
+	}
+	for _, e := range entries {
+		if !e.IsDir() {
+			continue
+		}
+		name := e.Name()
+		t.Run(name, func(t *testing.T) {
+			dir := filepath.Join("testdata", name)
+			matches, err := filepath.Glob(filepath.Join(dir, "junit-*.xml"))
+			if err != nil {
+				t.Fatalf("glob: %v", err)
+			}
+			sort.Strings(matches)
+			if len(matches) == 0 {
+				t.Fatalf("no junit-*.xml fixtures in %s", dir)
+			}
+
+			rep := Report{Title: "E2E failure analysis"}
+			for _, p := range matches {
+				data, err := os.ReadFile(p)
+				if err != nil {
+					t.Fatalf("read %s: %v", p, err)
+				}
+				sr, err := ParseFile(p, data)
+				if err != nil {
+					t.Fatalf("parse %s: %v", p, err)
+				}
+				rep.Suites = append(rep.Suites, sr)
+			}
+
+			got := Render(rep)
+			goldenPath := filepath.Join(dir, "analysis.md")
+			if *updateGolden {
+				if err := os.WriteFile(goldenPath, []byte(got), 0o644); err != nil {
+					t.Fatalf("write golden: %v", err)
+				}
+				return
+			}
+			want, err := os.ReadFile(goldenPath)
+			if err != nil {
+				t.Fatalf("read golden (run with -update to seed): %v", err)
+			}
+			if string(want) != got {
+				t.Errorf("rendered output differs from %s\n--- want ---\n%s\n--- got ---\n%s",
+					goldenPath, string(want), got)
+			}
+		})
+	}
+}
+
+func TestSignatureCollapsesNoise(t *testing.T) {
+	a := signature("describe-vpcs: InvalidParameterValue request-id: abc-123 vpc-deadbeefcafe1234")
+	b := signature("describe-vpcs: InvalidParameterValue request-id: xyz-999 vpc-1111222233334444")
+	if a != b {
+		t.Errorf("signatures should collapse to same bucket:\n  a=%q\n  b=%q", a, b)
+	}
+	if !strings.Contains(a, "<id>") || !strings.Contains(a, "vpc-<id>") {
+		t.Errorf("expected normalised placeholders in signature, got %q", a)
+	}
+}
+
+func TestCascadeDetection(t *testing.T) {
+	cases := []struct {
+		in   string
+		want bool
+	}{
+		{"Phase 5 must populate fix.InstanceID", true},
+		{"Should NOT be empty", true},
+		{"Expected value not to be nil", true},
+		{"describe-vpcs: InvalidParameterValue", false},
+		{"Eventually: condition not met within 3m0s", false},
+	}
+	for _, c := range cases {
+		if got := isCascade(c.in); got != c.want {
+			t.Errorf("isCascade(%q) = %v, want %v", c.in, got, c.want)
+		}
+	}
+}
+
+func TestExtractErrorLine_MessagesWinsOverError(t *testing.T) {
+	body := `    lifecycle_test.go:28:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:28
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5a_pre_ClusterStats`
+	got := extractErrorLine(body)
+	want := "Phase 5 must populate fix.InstanceID"
+	if got != want {
+		t.Errorf("extractErrorLine = %q, want %q", got, want)
+	}
+}
+
+func TestExtractErrorLine_FallsBackToLastTestLine(t *testing.T) {
+	body := "    vpc_test.go:227: Eventually: condition not met within 3m0s: [SSH handshake never completed]"
+	got := extractErrorLine(body)
+	if !strings.HasPrefix(got, "Eventually:") {
+		t.Errorf("expected fallback to test-line content, got %q", got)
+	}
+}
+
+func TestExtractFileHint_PicksLastMatch(t *testing.T) {
+	body := "ec2helpers.go:50: setup\n    vpc_test.go:227: Eventually: condition not met"
+	got := extractFileHint(body)
+	want := "vpc_test.go:227"
+	if got != want {
+		t.Errorf("extractFileHint = %q, want %q", got, want)
+	}
+}
+
+func TestParseFile_SkipsEmptyParentFailure(t *testing.T) {
+	// The synthetic "TestSingleNode" parent gets a <failure message="Failed"/>
+	// with no body whenever any subtest fails. The analyzer must skip it so
+	// it doesn't outrank the real first failure on XML order.
+	xml := []byte(`<?xml version="1.0"?>
+<testsuites><testsuite name="" tests="2" failures="2">
+  <testcase name="TestX" time="1.0"><failure message="Failed"></failure></testcase>
+  <testcase name="TestX/RealFailure" time="0.5"><failure message="Failed"><![CDATA[
+    foo_test.go:10:
+        Messages: real assertion message
+]]></failure></testcase>
+</testsuite></testsuites>`)
+	sr, err := ParseFile("junit-x.xml", xml)
+	if err != nil {
+		t.Fatalf("ParseFile: %v", err)
+	}
+	if sr.FailCount != 1 {
+		t.Fatalf("FailCount = %d, want 1 (empty parent should be skipped)", sr.FailCount)
+	}
+	if sr.Root == nil || sr.Root.Name != "TestX/RealFailure" {
+		t.Fatalf("expected real failure as root, got %+v", sr.Root)
+	}
+}
+
+func TestParseFile_CascadeFallbackWhenAllCascades(t *testing.T) {
+	// If every failure is a cascade marker, the earliest one becomes root
+	// so the report isn't blank.
+	xml := []byte(`<?xml version="1.0"?>
+<testsuites><testsuite name="" tests="2" failures="2">
+  <testcase name="TestX/A" time="1.0"><failure message="Failed"><![CDATA[
+    Messages: Phase 5 must populate fix.InstanceID
+]]></failure></testcase>
+  <testcase name="TestX/B" time="1.0"><failure message="Failed"><![CDATA[
+    Messages: Phase 5 must populate fix.InstanceID
+]]></failure></testcase>
+</testsuite></testsuites>`)
+	sr, err := ParseFile("junit-x.xml", xml)
+	if err != nil {
+		t.Fatalf("ParseFile: %v", err)
+	}
+	if sr.Root == nil || sr.Root.Name != "TestX/A" {
+		t.Fatalf("expected TestX/A as fallback root, got %+v", sr.Root)
+	}
+}
+
+func TestRender_NoFailuresIsClean(t *testing.T) {
+	out := Render(Report{
+		Title: "E2E failure analysis",
+		Suites: []SuiteReport{
+			{File: "junit-single.xml", Label: "single", Total: 3, FailCount: 0},
+		},
+	})
+	if !strings.Contains(out, "No failures") {
+		t.Errorf("expected zero-failure banner, got:\n%s", out)
+	}
+}

.github/actions/e2e-analyze/bundle.go

@@ -0,0 +1,208 @@
+package main
+
+// Stage 2 of docs/development/improvements/e2e-go-failure-analysis.md:
+// for each failure surfaced by Stage 1, materialise a per-failure bundle
+// directory containing the failure summary, a time-window slice of the
+// daemon journal, a testcase-window slice of the suite's stdout log, and
+// any per-test artifacts the harness already dumped at run-time.
+
+import (
+	"errors"
+	"fmt"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"regexp"
+	"strings"
+	"time"
+)
+
+// bundlePad is the per-failure window padding either side of the
+// [StartAt, StartAt+Duration] testcase span. Five seconds matches the
+// plan doc: long enough to catch the daemon's pre-test setup and
+// post-test cleanup chatter, short enough to keep the slice focused.
+const bundlePad = 5 * time.Second
+
+// nameSanitiseRe replaces characters unsafe in directory names with `_`.
+// Tracks tests/e2e/harness/artifacts.go which uses the same convention
+// when the test's own dump dir is created, so we can pick its contents
+// up by path without a second translation table.
+var nameSanitiseRe = regexp.MustCompile(`[^A-Za-z0-9._-]+`)
+
+func sanitiseTestName(s string) string {
+	return nameSanitiseRe.ReplaceAllString(s, "_")
+}
+
+// ApplySuiteStartFiles overrides each failure's StartAt by reading
+// <logDir>/test-<suiteLabel>.start (RFC3339 timestamp written by the
+// workflow before the suite runs). Required because go-junit-report's
+// `timestamp` attribute on <testsuite> records the time the XML was
+// produced — i.e. after the suite has finished — so it can't be used
+// as the suite wall-clock origin for journal slicing.
+//
+// Missing or unparseable files are silently ignored: the analyzer
+// falls back to the junit timestamp and the bundle still gets created,
+// it just may have an empty (or off-window) journal slice.
+func ApplySuiteStartFiles(rep *Report, logDir string) {
+	if logDir == "" {
+		return
+	}
+	for si := range rep.Suites {
+		s := &rep.Suites[si]
+		startPath := filepath.Join(logDir, fmt.Sprintf("test-%s.start", s.Label))
+		data, err := os.ReadFile(startPath)
+		if err != nil {
+			continue
+		}
+		ts, err := time.Parse(time.RFC3339, strings.TrimSpace(string(data)))
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: parse %s: %v\n", startPath, err)
+			continue
+		}
+		if s.Root != nil {
+			s.Root.StartAt = ts.Add(s.Root.OffsetFromSuiteStart)
+		}
+		for ci := range s.Cascades {
+			s.Cascades[ci].StartAt = ts.Add(s.Cascades[ci].OffsetFromSuiteStart)
+		}
+		for bi := range s.Buckets {
+			for fi := range s.Buckets[bi] {
+				f := &s.Buckets[bi][fi]
+				f.StartAt = ts.Add(f.OffsetFromSuiteStart)
+			}
+		}
+	}
+}
+
+// WriteBundles materialises per-failure bundles under <logDir>/analysis/.
+// Mutates rep in-place so each Failure.BundlePath holds the relative
+// path the renderer will link from the summary.
+//
+// Failures with no parseable wall-clock start still get a bundle dir
+// (and the testcase-window test.log slice), but no journal slice — the
+// time window is unknown.
+//
+// Errors are non-fatal: a bundle that partially fails still gets linked
+// from the report so a reader sees what's available, and the analyzer
+// keeps printing warnings so CI logs surface the gap.
+func WriteBundles(rep *Report, logDir string) {
+	if logDir == "" {
+		return
+	}
+	base := filepath.Join(logDir, "analysis")
+	if err := os.MkdirAll(base, 0o755); err != nil {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: mkdir %s: %v\n", base, err)
+		return
+	}
+
+	for si := range rep.Suites {
+		s := &rep.Suites[si]
+		if s.Root != nil {
+			writeOneBundle(logDir, base, s.Label, s.Root)
+		}
+		for ci := range s.Cascades {
+			writeOneBundle(logDir, base, s.Label, &s.Cascades[ci])
+		}
+		// Bucket entries share storage with Cascades but live in a
+		// separate slice in SuiteReport; mirror the bundle path so the
+		// renderer's bucket loop sees it too.
+		for bi := range s.Buckets {
+			for fi := range s.Buckets[bi] {
+				f := &s.Buckets[bi][fi]
+				if bp := findBundlePath(s, f.Name); bp != "" {
+					f.BundlePath = bp
+				}
+			}
+		}
+	}
+}
+
+// findBundlePath looks up a Failure's already-written BundlePath by test
+// name within a suite. Buckets reference the same logical failure as
+// Cascades but are independent slice values, so we copy the path across
+// rather than re-deriving it.
+func findBundlePath(s *SuiteReport, testName string) string {
+	if s.Root != nil && s.Root.Name == testName {
+		return s.Root.BundlePath
+	}
+	for _, c := range s.Cascades {
+		if c.Name == testName {
+			return c.BundlePath
+		}
+	}
+	return ""
+}
+
+func writeOneBundle(logDir, base, suiteLabel string, f *Failure) {
+	name := fmt.Sprintf("%02d-%s", f.Order+1, sanitiseTestName(f.Name))
+	dir := filepath.Join(base, name)
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: mkdir %s: %v\n", dir, err)
+		return
+	}
+	f.BundlePath = filepath.Join("analysis", name)
+
+	writeFailureSummary(dir, suiteLabel, f)
+
+	if !f.StartAt.IsZero() {
+		start := f.StartAt.Add(-bundlePad)
+		end := f.StartAt.Add(time.Duration(f.Duration * float64(time.Second))).Add(bundlePad)
+		journalSrc := filepath.Join(logDir, "spinifex-journal.log")
+		if _, err := SliceJournal(journalSrc, start, end, filepath.Join(dir, "journal.log")); err != nil {
+			fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: slice journal for %s: %v\n", f.Name, err)
+		}
+	}
+
+	testLogSrc := filepath.Join(logDir, fmt.Sprintf("test-%s.log", suiteLabel))
+	if _, err := SliceTestLog(testLogSrc, f.Name, filepath.Join(dir, "test.log")); err != nil {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: slice test log for %s: %v\n", f.Name, err)
+	}
+
+	// Per-test artifact subdir written by harness.ArtifactDir lives at
+	// <logDir>/<TestName with / replaced by _>/. Copy any files into
+	// the bundle so qemu/ovs/etc. dumps already collected by the test
+	// travel alongside the analysis.
+	perTestSrc := filepath.Join(logDir, strings.ReplaceAll(f.Name, "/", "_"))
+	if entries, err := os.ReadDir(perTestSrc); err == nil {
+		for _, e := range entries {
+			if e.IsDir() {
+				continue
+			}
+			_ = copyFile(filepath.Join(perTestSrc, e.Name()), filepath.Join(dir, e.Name()))
+		}
+	} else if !os.IsNotExist(err) && !errorsIsNotDir(err) {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: read per-test dir %s: %v\n", perTestSrc, err)
+	}
+}
+
+// errorsIsNotDir handles the case where the harness wrote a file at the
+// per-test name instead of a directory — best-effort skip.
+func errorsIsNotDir(err error) bool {
+	var pe *fs.PathError
+	if !errors.As(err, &pe) {
+		return false
+	}
+	return pe.Err != nil && strings.Contains(pe.Err.Error(), "not a directory")
+}
+
+func writeFailureSummary(dir, suiteLabel string, f *Failure) {
+	var b strings.Builder
+	fmt.Fprintf(&b, "Suite:    %s\n", suiteLabel)
+	fmt.Fprintf(&b, "Test:     %s\n", f.Name)
+	if !f.StartAt.IsZero() {
+		fmt.Fprintf(&b, "Start:    %s\n", f.StartAt.Format(time.RFC3339))
+	}
+	fmt.Fprintf(&b, "Duration: %.3fs\n", f.Duration)
+	if f.FileHint != "" {
+		fmt.Fprintf(&b, "Site:     %s\n", f.FileHint)
+	}
+	if f.Cascade {
+		b.WriteString("Cascade:  yes (downstream of another failure)\n")
+	}
+	if f.Error != "" {
+		b.WriteString("\nError:\n")
+		b.WriteString(oneLine(f.Error))
+		b.WriteString("\n")
+	}
+	_ = os.WriteFile(filepath.Join(dir, "failure.txt"), []byte(b.String()), 0o644)
+}

.github/actions/e2e-analyze/bundle_test.go

@@ -0,0 +1,229 @@
+package main
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+func TestSliceJournal_KeepsOnlyWindow(t *testing.T) {
+	src := filepath.Join(t.TempDir(), "journal.log")
+	dst := filepath.Join(t.TempDir(), "out.log")
+	body := strings.Join([]string{
+		"2026-05-19T12:32:30+1000 host spinifex-daemon[1]: pre",
+		"  continuation of pre",
+		"2026-05-19T12:32:35+1000 host spinifex-daemon[1]: IN window",
+		"  multiline inside window",
+		"2026-05-19T12:32:45+1000 host spinifex-daemon[1]: post",
+		"",
+	}, "\n")
+	if err := os.WriteFile(src, []byte(body), 0o644); err != nil {
+		t.Fatalf("write src: %v", err)
+	}
+	start, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:33+10:00")
+	end, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:40+10:00")
+	n, err := SliceJournal(src, start, end, dst)
+	if err != nil {
+		t.Fatalf("SliceJournal: %v", err)
+	}
+	if n != 2 {
+		t.Fatalf("written = %d, want 2", n)
+	}
+	got, _ := os.ReadFile(dst)
+	if !strings.Contains(string(got), "IN window") || !strings.Contains(string(got), "multiline inside window") {
+		t.Errorf("missing in-window content:\n%s", got)
+	}
+	if strings.Contains(string(got), "pre") || strings.Contains(string(got), "post") {
+		t.Errorf("kept out-of-window content:\n%s", got)
+	}
+}
+
+func TestSliceJournal_AcceptsRFC3339ColonOffset(t *testing.T) {
+	// systemd's `journalctl --output=short-iso` ships RFC3339 timestamps
+	// (offset includes a colon, e.g. "+10:00") on most versions. Earlier
+	// regex only matched "+1000" — make sure both forms work.
+	src := filepath.Join(t.TempDir(), "journal.log")
+	dst := filepath.Join(t.TempDir(), "out.log")
+	body := strings.Join([]string{
+		"2026-05-19T12:32:30+10:00 host spinifex-daemon[1]: pre",
+		"2026-05-19T12:32:35+10:00 host spinifex-daemon[1]: IN window",
+		"2026-05-19T12:32:50+10:00 host spinifex-daemon[1]: post",
+		"",
+	}, "\n")
+	if err := os.WriteFile(src, []byte(body), 0o644); err != nil {
+		t.Fatalf("write src: %v", err)
+	}
+	start, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:33+10:00")
+	end, _ := time.Parse(time.RFC3339, "2026-05-19T12:32:40+10:00")
+	n, err := SliceJournal(src, start, end, dst)
+	if err != nil {
+		t.Fatalf("SliceJournal: %v", err)
+	}
+	if n != 1 {
+		t.Fatalf("written = %d, want 1", n)
+	}
+	got, _ := os.ReadFile(dst)
+	if !strings.Contains(string(got), "IN window") {
+		t.Errorf("missing in-window content:\n%s", got)
+	}
+}
+
+func TestSliceJournal_MissingSrcIsNotError(t *testing.T) {
+	dst := filepath.Join(t.TempDir(), "out.log")
+	n, err := SliceJournal("/no/such/file", time.Now(), time.Now(), dst)
+	if err != nil {
+		t.Fatalf("unexpected err: %v", err)
+	}
+	if n != 0 {
+		t.Errorf("written = %d, want 0", n)
+	}
+}
+
+func TestSliceTestLog_PicksOneTestcase(t *testing.T) {
+	src := filepath.Join(t.TempDir(), "test.log")
+	dst := filepath.Join(t.TempDir(), "out.log")
+	body := strings.Join([]string{
+		"=== RUN   TestSingleNode",
+		"  before subtests",
+		"=== RUN   TestSingleNode/Phase5_LaunchInstance",
+		"    runinstances",
+		"    ec2helpers.go:164: describe-vpcs: InvalidParameterValue",
+		"--- FAIL: TestSingleNode/Phase5_LaunchInstance (2.50s)",
+		"=== RUN   TestSingleNode/Phase6_Volumes",
+		"    other content",
+		"--- FAIL: TestSingleNode/Phase6_Volumes (0.10s)",
+		"--- FAIL: TestSingleNode (10.0s)",
+		"",
+	}, "\n")
+	if err := os.WriteFile(src, []byte(body), 0o644); err != nil {
+		t.Fatalf("write src: %v", err)
+	}
+	n, err := SliceTestLog(src, "TestSingleNode/Phase5_LaunchInstance", dst)
+	if err != nil {
+		t.Fatalf("SliceTestLog: %v", err)
+	}
+	if n != 4 {
+		t.Fatalf("written = %d, want 4 (RUN + 2 inner + FAIL)", n)
+	}
+	got, _ := os.ReadFile(dst)
+	s := string(got)
+	if !strings.Contains(s, "describe-vpcs: InvalidParameterValue") {
+		t.Errorf("missing failure line:\n%s", s)
+	}
+	if strings.Contains(s, "Phase6_Volumes") || strings.Contains(s, "before subtests") {
+		t.Errorf("leaked sibling test content:\n%s", s)
+	}
+}
+
+func TestSliceTestLog_UnknownTestcaseIsEmpty(t *testing.T) {
+	src := filepath.Join(t.TempDir(), "test.log")
+	dst := filepath.Join(t.TempDir(), "out.log")
+	if err := os.WriteFile(src, []byte("=== RUN   TestX\n--- PASS: TestX (0.0s)\n"), 0o644); err != nil {
+		t.Fatalf("write src: %v", err)
+	}
+	n, err := SliceTestLog(src, "TestY", dst)
+	if err != nil {
+		t.Fatalf("SliceTestLog: %v", err)
+	}
+	if n != 0 {
+		t.Errorf("written = %d, want 0", n)
+	}
+	if _, err := os.Stat(dst); err != nil {
+		t.Errorf("dst not created: %v", err)
+	}
+}
+
+// TestWriteBundles_EndToEnd builds a fake artifact dir with a junit XML,
+// a journal log, a test log, and a per-test artifact subdir, then runs
+// the full ParseFile → WriteBundles pipeline and asserts the bundle dir
+// contains everything the report links to.
+func TestWriteBundles_EndToEnd(t *testing.T) {
+	dir := t.TempDir()
+
+	junit := `<?xml version="1.0"?>
+<testsuites><testsuite name="" tests="2" failures="1" timestamp="2026-05-19T12:32:30+10:00" time="3.0">
+  <testcase name="TestSingleNode/Phase5_LaunchInstance" time="2.500">
+    <failure message="Failed"><![CDATA[
+    ec2helpers.go:164: describe-vpcs: InvalidParameterValue (status 400)
+        Error Trace: tests/e2e/harness/ec2helpers.go:164
+        Test: TestSingleNode/Phase5_LaunchInstance]]></failure>
+  </testcase>
+</testsuite></testsuites>`
+	if err := os.WriteFile(filepath.Join(dir, "junit-single.xml"), []byte(junit), 0o644); err != nil {
+		t.Fatalf("write junit: %v", err)
+	}
+
+	journal := strings.Join([]string{
+		"2026-05-19T12:32:20+1000 host spinifex-daemon[1]: too-early",
+		"2026-05-19T12:32:30+1000 host spinifex-daemon[1]: in-window-start",
+		"2026-05-19T12:32:32+1000 host spinifex-daemon[1]: describe-vpcs filter=is_default",
+		"2026-05-19T12:32:50+1000 host spinifex-daemon[1]: too-late",
+		"",
+	}, "\n")
+	if err := os.WriteFile(filepath.Join(dir, "spinifex-journal.log"), []byte(journal), 0o644); err != nil {
+		t.Fatalf("write journal: %v", err)
+	}
+
+	testlog := strings.Join([]string{
+		"=== RUN   TestSingleNode/Phase5_LaunchInstance",
+		"    ec2helpers.go:164: describe-vpcs: InvalidParameterValue",
+		"--- FAIL: TestSingleNode/Phase5_LaunchInstance (2.50s)",
+		"",
+	}, "\n")
+	if err := os.WriteFile(filepath.Join(dir, "test-single.log"), []byte(testlog), 0o644); err != nil {
+		t.Fatalf("write test log: %v", err)
+	}
+
+	// Per-test artifact subdir (matches harness.ArtifactDir convention).
+	perTest := filepath.Join(dir, "TestSingleNode_Phase5_LaunchInstance")
+	if err := os.MkdirAll(perTest, 0o755); err != nil {
+		t.Fatalf("mkdir per-test: %v", err)
+	}
+	if err := os.WriteFile(filepath.Join(perTest, "qemu-console.log"), []byte("[qemu] boot\n"), 0o644); err != nil {
+		t.Fatalf("write per-test file: %v", err)
+	}
+
+	data, _ := os.ReadFile(filepath.Join(dir, "junit-single.xml"))
+	sr, err := ParseFile("junit-single.xml", data)
+	if err != nil {
+		t.Fatalf("ParseFile: %v", err)
+	}
+	rep := Report{Title: "E2E failure analysis", Suites: []SuiteReport{sr}}
+	WriteBundles(&rep, dir)
+
+	if rep.Suites[0].Root == nil {
+		t.Fatal("no root")
+	}
+	bp := rep.Suites[0].Root.BundlePath
+	if bp == "" {
+		t.Fatal("BundlePath not set")
+	}
+	bundle := filepath.Join(dir, bp)
+	for _, want := range []string{"failure.txt", "journal.log", "test.log", "qemu-console.log"} {
+		if _, err := os.Stat(filepath.Join(bundle, want)); err != nil {
+			t.Errorf("missing %s in bundle %s: %v", want, bp, err)
+		}
+	}
+
+	jl, _ := os.ReadFile(filepath.Join(bundle, "journal.log"))
+	if !strings.Contains(string(jl), "describe-vpcs filter=is_default") {
+		t.Errorf("journal.log lacks in-window line:\n%s", jl)
+	}
+	if strings.Contains(string(jl), "too-early") || strings.Contains(string(jl), "too-late") {
+		t.Errorf("journal.log includes out-of-window content:\n%s", jl)
+	}
+
+	tl, _ := os.ReadFile(filepath.Join(bundle, "test.log"))
+	if !strings.Contains(string(tl), "--- FAIL: TestSingleNode/Phase5_LaunchInstance") {
+		t.Errorf("test.log missing FAIL marker:\n%s", tl)
+	}
+
+	out := Render(rep)
+	wantLink := fmt.Sprintf("- Bundle: `%s/`", bp)
+	if !strings.Contains(out, wantLink) {
+		t.Errorf("render missing bundle link %q:\n%s", wantLink, out)
+	}
+}

.github/actions/e2e-analyze/main.go

@@ -0,0 +1,78 @@
+package main
+
+import (
+	"flag"
+	"fmt"
+	"os"
+	"path/filepath"
+	"sort"
+)
+
+func main() {
+	var junitGlob, logDir, title string
+	flag.StringVar(&junitGlob, "junit-glob", os.Getenv("ANALYZE_JUNIT_GLOB"), "glob matching junit-*.xml files")
+	flag.StringVar(&logDir, "log-dir", os.Getenv("ANALYZE_LOG_DIR"), "artifact dir to write analysis.md into")
+	flag.StringVar(&title, "title", os.Getenv("ANALYZE_TITLE"), "report heading")
+	flag.Parse()
+
+	if title == "" {
+		title = "E2E failure analysis"
+	}
+	if junitGlob == "" {
+		fmt.Fprintln(os.Stderr, "::error::e2e-analyze: -junit-glob is required")
+		os.Exit(0) // analyzer is advisory only; existing gate owns exit code
+	}
+
+	paths, err := filepath.Glob(junitGlob)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: bad glob %q: %v\n", junitGlob, err)
+		os.Exit(0)
+	}
+	sort.Strings(paths)
+
+	report := Report{Title: title}
+	for _, p := range paths {
+		data, err := os.ReadFile(p)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: read %s: %v\n", p, err)
+			continue
+		}
+		sr, err := ParseFile(p, data)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: %v\n", err)
+			continue
+		}
+		report.Suites = append(report.Suites, sr)
+	}
+
+	// Stage 2: prefer authoritative per-suite start times written by the
+	// workflow over junit-report's conversion-time `timestamp` attribute,
+	// then materialise per-failure bundles so Render can link them.
+	if logDir != "" {
+		ApplySuiteStartFiles(&report, logDir)
+		WriteBundles(&report, logDir)
+	}
+
+	out := Render(report)
+
+	if logDir != "" {
+		if err := os.MkdirAll(logDir, 0o755); err == nil {
+			if err := os.WriteFile(filepath.Join(logDir, "analysis.md"), []byte(out), 0o644); err != nil {
+				fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: write analysis.md: %v\n", err)
+			}
+		}
+	}
+
+	if summaryPath := os.Getenv("GITHUB_STEP_SUMMARY"); summaryPath != "" {
+		f, err := os.OpenFile(summaryPath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
+		if err == nil {
+			defer f.Close()
+			if _, werr := f.WriteString(out); werr != nil {
+				fmt.Fprintf(os.Stderr, "::warning::e2e-analyze: append summary: %v\n", werr)
+			}
+		}
+	} else {
+		// Local invocation (developer running the tool by hand). Mirror to stdout.
+		fmt.Print(out)
+	}
+}

.github/actions/e2e-analyze/slice.go

@@ -0,0 +1,176 @@
+package main
+
+// Stage 2 of docs/development/improvements/e2e-go-failure-analysis.md:
+// per-failure log slicers used by bundle.go to materialise time-window /
+// testcase-window log excerpts next to each named failure.
+
+import (
+	"bufio"
+	"io"
+	"os"
+	"regexp"
+	"strings"
+	"time"
+)
+
+// shortIsoTSRe matches the timestamp prefix produced by
+// `journalctl --output=short-iso`. The offset format varies across
+// systemd versions: most ship "+10:00" (RFC3339), some ship "+1000"
+// (no colon). Match either — the parser below picks the right layout.
+var shortIsoTSRe = regexp.MustCompile(`^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:[+-]\d{2}:?\d{2}|Z))\s`)
+
+// shortIsoLayouts are tried in order; the first parse that succeeds wins.
+var shortIsoLayouts = []string{
+	time.RFC3339,               // 2026-05-19T12:32:30+10:00
+	"2006-01-02T15:04:05-0700", // 2026-05-19T12:32:30+1000
+}
+
+func parseShortIso(s string) (time.Time, error) {
+	var lastErr error
+	for _, layout := range shortIsoLayouts {
+		t, err := time.Parse(layout, s)
+		if err == nil {
+			return t, nil
+		}
+		lastErr = err
+	}
+	return time.Time{}, lastErr
+}
+
+// SliceJournal copies lines from src whose `short-iso` timestamp falls in
+// [start, end] to dst. Lines without a parseable timestamp inherit the
+// previous line's classification — that keeps multi-line journal records
+// (stack traces, panics) intact rather than orphaning their continuation
+// lines. Lines that precede the first parseable timestamp are dropped.
+//
+// Returns the number of lines written; missing src is not an error (the
+// journal may be empty if collection failed).
+func SliceJournal(src string, start, end time.Time, dst string) (int, error) {
+	f, err := os.Open(src)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return 0, nil
+		}
+		return 0, err
+	}
+	defer f.Close()
+
+	out, err := os.Create(dst)
+	if err != nil {
+		return 0, err
+	}
+	defer out.Close()
+
+	bw := bufio.NewWriter(out)
+	defer bw.Flush()
+
+	sc := bufio.NewScanner(f)
+	// Journal lines can be long (panic dumps); bump scanner buffer.
+	sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
+
+	keep := false
+	written := 0
+	for sc.Scan() {
+		line := sc.Text()
+		if m := shortIsoTSRe.FindStringSubmatch(line); m != nil {
+			ts, err := parseShortIso(m[1])
+			if err != nil {
+				continue
+			}
+			keep = !ts.Before(start) && !ts.After(end)
+		}
+		if keep {
+			if _, err := bw.WriteString(line); err != nil {
+				return written, err
+			}
+			if err := bw.WriteByte('\n'); err != nil {
+				return written, err
+			}
+			written++
+		}
+	}
+	return written, sc.Err()
+}
+
+// runRePrefix and finishRePrefix bracket a single testcase's output in
+// `go test -v` stdout. Subtests have their own `=== RUN` / `--- FAIL`
+// markers, so slicing on the exact testcase name yields just that test's
+// lines (including any nested phase logs).
+var (
+	runRe    = regexp.MustCompile(`^=== RUN\s+(\S+)$`)
+	finishRe = regexp.MustCompile(`^\s*---\s+(PASS|FAIL|SKIP):\s+(\S+)\s+\(`)
+)
+
+// SliceTestLog copies lines belonging to the named testcase from src to
+// dst. The block starts at `=== RUN <name>` and ends at the matching
+// `--- PASS|FAIL|SKIP: <name> (…)` line (inclusive). If the testcase is
+// never seen the destination file is created empty so the bundle has a
+// stable shape.
+//
+// Returns the number of lines written; missing src is not an error.
+func SliceTestLog(src, testName, dst string) (int, error) {
+	f, err := os.Open(src)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return 0, nil
+		}
+		return 0, err
+	}
+	defer f.Close()
+
+	out, err := os.Create(dst)
+	if err != nil {
+		return 0, err
+	}
+	defer out.Close()
+
+	bw := bufio.NewWriter(out)
+	defer bw.Flush()
+
+	sc := bufio.NewScanner(f)
+	sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
+
+	in := false
+	written := 0
+	for sc.Scan() {
+		line := sc.Text()
+		if !in {
+			if m := runRe.FindStringSubmatch(strings.TrimRight(line, "\r")); m != nil && m[1] == testName {
+				in = true
+			} else {
+				continue
+			}
+		}
+		if _, err := bw.WriteString(line); err != nil {
+			return written, err
+		}
+		if err := bw.WriteByte('\n'); err != nil {
+			return written, err
+		}
+		written++
+		if m := finishRe.FindStringSubmatch(line); m != nil && m[2] == testName {
+			return written, nil
+		}
+	}
+	return written, sc.Err()
+}
+
+// copyFile is a small helper used when bundling pre-collected per-test
+// artifacts. Best-effort: missing src is silently skipped.
+func copyFile(src, dst string) error {
+	in, err := os.Open(src)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return nil
+		}
+		return err
+	}
+	defer in.Close()
+	out, err := os.Create(dst)
+	if err != nil {
+		return err
+	}
+	defer out.Close()
+	_, err = io.Copy(out, in)
+	return err
+}

.github/actions/e2e-analyze/testdata/cascade-vpcs/analysis.md

@@ -0,0 +1,23 @@
+## E2E failure analysis
+
+### Suite `single`: 16 failed, 1 root cause likely
+
+**Root cause (earliest non-cascade)**
+
+- Test: `TestSingleNode/Phase5_LaunchInstance`
+- Start: 12:30:00 (duration 2.5s)
+- Error: Received unexpected error: InvalidParameterValue (HTTP 400)
+- File:  `tests/e2e/harness/ec2helpers.go:164`
+
+**Cascaded failures (15) — grouped by signature**
+
+- 14× Phase 5 must populate fix.InstanceID
+  - `TestSingleNode/Phase5a_pre_ClusterStats`
+  - `TestSingleNode/Phase5a_Metadata`
+  - `TestSingleNode/Phase5b_DescribeInstances`
+  - `TestSingleNode/Phase5c_StopStart`
+  - `TestSingleNode/Phase6_Volumes`
+  - … and 9 more
+- 1× Phase 2 must populate fix.AZName
+  - `TestSingleNode/Phase8_NatGW`
+

.github/actions/e2e-analyze/testdata/cascade-vpcs/junit-single.xml

@@ -0,0 +1,143 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<testsuites tests="20" failures="17" skipped="0">
+	<testsuite name="" tests="20" failures="17" errors="0" id="0" hostname="banksia" skipped="0" time="600.000" timestamp="2026-04-12T12:30:00+10:00">
+		<testcase name="TestSingleNode" classname="" time="600.000">
+			<failure message="Failed"></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase1_Environment" classname="" time="0.300">
+			<system-out><![CDATA[
+&#x2501;&#x2501; Phase 1 &#x2501;&#x2501;]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase2_Discovery" classname="" time="0.010"/>
+		<testcase name="TestSingleNode/Phase3_KeyPairs" classname="" time="0.540"/>
+		<testcase name="TestSingleNode/Phase4_Image" classname="" time="0.060"/>
+		<testcase name="TestSingleNode/Phase5_LaunchInstance" classname="" time="2.500">
+			<failure message="Failed"><![CDATA[
+    ec2helpers.go:164: describe-vpcs: InvalidParameterValue: Filter 'is_default' is invalid (status 400)
+    	Error Trace:	tests/e2e/harness/ec2helpers.go:164
+    	Error:      	Received unexpected error:
+    	            	InvalidParameterValue (HTTP 400)
+    	Test:       	TestSingleNode/Phase5_LaunchInstance]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_pre_ClusterStats" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    lifecycle_test.go:28:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:28
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5a_pre_ClusterStats]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_Metadata" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    lifecycle_test.go:57:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:57
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5a_Metadata]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5b_DescribeInstances" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    lifecycle_test.go:109:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:109
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5b_DescribeInstances]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5c_StopStart" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    lifecycle_test.go:222:
+        Error Trace:	tests/e2e/single/lifecycle_test.go:222
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase5c_StopStart]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase6_Volumes" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    volume_test.go:23:
+        Error Trace:	tests/e2e/single/volume_test.go:23
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase6_Volumes]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase6b_AttachVolume" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    volume_test.go:24:
+        Error Trace:	tests/e2e/single/volume_test.go:24
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase6b_AttachVolume]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7_Tags" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    tags_test.go:23:
+        Error Trace:	tests/e2e/single/tags_test.go:23
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7_Tags]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7a_Snapshots" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    snapshot_test.go:23:
+        Error Trace:	tests/e2e/single/snapshot_test.go:23
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7a_Snapshots]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7b_DescribeInstanceAttribute" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    instance_test.go:81:
+        Error Trace:	tests/e2e/single/instance_test.go:81
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7b_DescribeInstanceAttribute]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7c_ModifyInstanceAttribute" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    instance_test.go:92:
+        Error Trace:	tests/e2e/single/instance_test.go:92
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7c_ModifyInstanceAttribute]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7d_Console" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    instance_test.go:121:
+        Error Trace:	tests/e2e/single/instance_test.go:121
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7d_Console]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7e_Negative" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    negative_test.go:43:
+        Error Trace:	tests/e2e/single/negative_test.go:43
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase7e_Negative]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NatGW" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    natgw_test.go:39:
+        Error Trace:	tests/e2e/single/natgw_test.go:39
+        Error:      	Should NOT be empty
+        Messages:   	Phase 2 must populate fix.AZName
+        Test:       	TestSingleNode/Phase8_NatGW]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8a_DataPath" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    datapath_test.go:48:
+        Error Trace:	tests/e2e/single/datapath_test.go:48
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase8a_DataPath]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase9_SecurityGroups" classname="" time="0.020">
+			<failure message="Failed"><![CDATA[
+    securitygroup_test.go:35:
+        Error Trace:	tests/e2e/single/securitygroup_test.go:35
+        Error:      	Should NOT be empty
+        Messages:   	Phase 5 must populate fix.InstanceID
+        Test:       	TestSingleNode/Phase9_SecurityGroups]]></failure>
+		</testcase>
+	</testsuite>
+</testsuites>

.github/actions/e2e-analyze/testdata/no-failures/analysis.md

@@ -0,0 +1,3 @@
+## E2E failure analysis
+
+✅ No failures across any suite.

.github/actions/e2e-analyze/testdata/no-failures/junit-single.xml

@@ -0,0 +1,8 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<testsuites tests="3" failures="0" skipped="0">
+	<testsuite name="" tests="3" failures="0" errors="0" id="0" hostname="banksia" skipped="0" time="1.500" timestamp="2026-04-12T12:30:00+10:00">
+		<testcase name="TestSingleNode" classname="" time="1.500"/>
+		<testcase name="TestSingleNode/Phase1_Environment" classname="" time="0.300"/>
+		<testcase name="TestSingleNode/Phase2_Discovery" classname="" time="1.200"/>
+	</testsuite>
+</testsuites>

.github/actions/e2e-analyze/testdata/single-failure/analysis.md

@@ -0,0 +1,11 @@
+## E2E failure analysis
+
+### Suite `single`: 1 failed, 1 root cause likely
+
+**Root cause (earliest non-cascade)**
+
+- Test: `TestSingleNode/Phase8b_VPCSubnetE2E`
+- Start: 00:26:18 (duration 200.5s)
+- Error: Eventually: condition not met within 3m0s: [SSH handshake 192.168.0.202:22 never completed]
+- File:  `vpc_test.go:227`
+

.github/actions/e2e-analyze/testdata/single-failure/junit-single.xml

@@ -0,0 +1,993 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<testsuites tests="79" failures="2" skipped="6">
+	<testsuite name="" tests="79" failures="2" errors="0" id="0" hostname="banksia" skipped="6" time="1708.140" timestamp="2026-05-19T00:19:45+10:00">
+		<testcase name="TestSingleNode" classname="" time="800.320">
+			<failure message="Failed"></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase1_Environment" classname="" time="0.300">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 1 — Environment ━━�[0m
+�[2m· checking /dev/kvm�[0m
+    discovery_test.go:29: �[2m· checking /dev/kvm�[0m
+�[2mkvm�[0m=writable
+    discovery_test.go:38: �[2mkvm�[0m=writable
+�[2m· waiting for AWS gateway�[0m
+    discovery_test.go:45: �[2m· waiting for AWS gateway�[0m
+�[2mgateway�[0m=https://192.168.0.10:9999/
+    discovery_test.go:55: �[2mgateway�[0m=https://192.168.0.10:9999/
+�[2m· waiting for daemon NATS subscriptions�[0m
+    discovery_test.go:60: �[2m· waiting for daemon NATS subscriptions�[0m
+�[2m· describe-regions / describe-availability-zones�[0m
+    discovery_test.go:73: �[2m· describe-regions / describe-availability-zones�[0m
+�[2maz�[0m=ap-southeast-2a �[2mregion�[0m=ap-southeast-2
+    discovery_test.go:84: �[2maz�[0m=ap-southeast-2a �[2mregion�[0m=ap-southeast-2]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase1b_ClusterStats" classname="" time="9.200">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 1b — Cluster Stats CLI ━━�[0m
+�[2m· spx get nodes�[0m
+    discovery_test.go:101: �[2m· spx get nodes�[0m
+�[2m· spx top nodes�[0m
+    discovery_test.go:105: �[2m· spx top nodes�[0m
+�[2m· spx get vms (empty before any launches)�[0m
+    discovery_test.go:112: �[2m· spx get vms (empty before any launches)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase2_Discovery" classname="" time="0.010">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 2 — Discovery & Metadata ━━�[0m
+�[2m· discovering nano instance type�[0m
+    discovery_test.go:133: �[2m· discovering nano instance type�[0m
+�[2minstance_type�[0m=t3.nano �[2march�[0m=x86_64 �[2maz�[0m=ap-southeast-2a
+    discovery_test.go:159: �[2minstance_type�[0m=t3.nano �[2march�[0m=x86_64 �[2maz�[0m=ap-southeast-2a]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase2b_SerialConsole" classname="" time="0.020">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 2b — Serial Console Access ━━�[0m
+�[2m· default state should be disabled�[0m
+    discovery_test.go:168: �[2m· default state should be disabled�[0m
+�[2m· enable�[0m
+    discovery_test.go:173: �[2m· enable�[0m
+�[2mstate�[0m=enabled
+    discovery_test.go:180: �[2mstate�[0m=enabled
+�[2m· disable�[0m
+    discovery_test.go:182: �[2m· disable�[0m
+�[2mstate�[0m=disabled
+    discovery_test.go:189: �[2mstate�[0m=disabled]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase3_KeyPairs" classname="" time="0.540">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 3 — SSH Key Management ━━�[0m
+�[2m· create-key-pair "test-key-1"�[0m
+    keypair_test.go:37: �[2m· create-key-pair "test-key-1"�[0m
+�[2mkey�[0m=test-key-1 �[2mpem�[0m=/tmp/TestSingleNode1289015487/001/test-key-1.pem
+    keypair_test.go:51: �[2mkey�[0m=test-key-1 �[2mpem�[0m=/tmp/TestSingleNode1289015487/001/test-key-1.pem
+�[2m· import-key-pair "test-key-2" (local-generated RSA)�[0m
+    keypair_test.go:58: �[2m· import-key-pair "test-key-2" (local-generated RSA)�[0m
+�[2mimported�[0m=test-key-2
+    keypair_test.go:65: �[2mimported�[0m=test-key-2
+�[2m· describe-key-pairs (both present)�[0m
+    keypair_test.go:67: �[2m· describe-key-pairs (both present)�[0m
+�[2m· delete "test-key-2"�[0m
+    keypair_test.go:72: �[2m· delete "test-key-2"�[0m
+�[2m· describe-key-pairs (only test-key-1 remains)�[0m
+    keypair_test.go:76: �[2m· describe-key-pairs (only test-key-1 remains)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase4_Image" classname="" time="0.060">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 4 — Image Management ━━�[0m
+�[2mimage�[0m=ubuntu-24.04-x86_64 �[2march�[0m=x86_64
+    image_test.go:33: �[2mimage�[0m=ubuntu-24.04-x86_64 �[2march�[0m=x86_64
+�[2m· spx admin images import --name ubuntu-24.04-x86_64�[0m
+    image_test.go:35: �[2m· spx admin images import --name ubuntu-24.04-x86_64�[0m
+�[2m· spx import did not yield AMI ID; falling back to describe-images�[0m
+    image_test.go:48: �[2m· spx import did not yield AMI ID; falling back to describe-images�[0m
+�[2m· describe-images by AMI ID (verify exactly one match)�[0m
+    image_test.go:78: �[2m· describe-images by AMI ID (verify exactly one match)�[0m
+�[2mami�[0m=ami-6f60f2b3bf04b5453 �[2mname�[0m=ami-ubuntu-24.04-x86_64
+    image_test.go:86: �[2mami�[0m=ami-6f60f2b3bf04b5453 �[2mname�[0m=ami-ubuntu-24.04-x86_64]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5_LaunchInstance" classname="" time="10.830">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5 — Instance Lifecycle ━━�[0m
+�[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msg�[0m=sg-764a7fa2a39577204 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63
+    instance_test.go:31: �[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msg�[0m=sg-764a7fa2a39577204 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63
+�[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.nano key=test-key-1�[0m
+    instance_test.go:35: �[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.nano key=test-key-1�[0m
+�[2minstance�[0m=i-0c615e5dd74050f53
+    instance_test.go:48: �[2minstance�[0m=i-0c615e5dd74050f53
+    instance_test.go:50: instance i-0c615e5dd74050f53 reached state running
+�[2mroot_volume�[0m=vol-2230138124fb51ee2 �[2mroot_device�[0m=
+    instance_test.go:70: �[2mroot_volume�[0m=vol-2230138124fb51ee2 �[2mroot_device�[0m=]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_pre_ClusterStats" classname="" time="3.040">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5a-pre — Cluster Stats CLI (with running VM) ━━�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_Metadata" classname="" time="0.510">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5a — Instance Metadata Validation ━━�[0m
+�[2mtype�[0m=t3.nano �[2mkey�[0m=test-key-1 �[2mimage�[0m=ami-6f60f2b3bf04b5453 �[2mbdms�[0m=1
+    instance_test.go:108: �[2mtype�[0m=t3.nano �[2mkey�[0m=test-key-1 �[2mimage�[0m=ami-6f60f2b3bf04b5453 �[2mbdms�[0m=1]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_ii_SSH" classname="" time="49.450">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5a-ii — SSH Connectivity & Volume Verification ━━�[0m
+�[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
+    instance_test.go:127: �[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    instance_test.go:130: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    lifecycle_test.go:384: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2m· ssh id�[0m
+    instance_test.go:136: �[2m· ssh id�[0m
+�[2m· lsblk root-volume cross-check vs API�[0m
+    instance_test.go:140: �[2m· lsblk root-volume cross-check vs API�[0m
+�[2mguest_gib�[0m=4 �[2mapi_gib�[0m=4
+    instance_test.go:159: �[2mguest_gib�[0m=4 �[2mapi_gib�[0m=4
+�[2m· ssh hostname�[0m
+    instance_test.go:161: �[2m· ssh hostname�[0m
+�[2mhostname�[0m=spinifex-vm-0c615e5d �[2mmatches_short_id�[0m=0c615e5d
+    instance_test.go:172: �[2mhostname�[0m=spinifex-vm-0c615e5d �[2mmatches_short_id�[0m=0c615e5d]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5a_iii_Console" classname="" time="0.010">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5a-iii — Console Output ━━�[0m
+�[2minstance�[0m=i-0c615e5dd74050f53 �[2mhas_output�[0m=true
+    instance_test.go:191: �[2minstance�[0m=i-0c615e5dd74050f53 �[2mhas_output�[0m=true]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5b_Volume" classname="" time="4.540">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5b — Volume Lifecycle (Attach/Detach) ━━�[0m
+�[2m· create-volume size=10 az=ap-southeast-2a�[0m
+    volume_test.go:26: �[2m· create-volume size=10 az=ap-southeast-2a�[0m
+�[2mvolume�[0m=vol-a885f79173e0a8030
+    volume_test.go:34: �[2mvolume�[0m=vol-a885f79173e0a8030
+    volume_test.go:39: volume vol-a885f79173e0a8030 reached state available
+�[2m· modify-volume size=20�[0m
+    volume_test.go:42: �[2m· modify-volume size=20�[0m
+�[2mresized_gib�[0m=20
+    volume_test.go:66: �[2mresized_gib�[0m=20
+�[2m· attach-volume vol-a885f79173e0a8030 -> i-0c615e5dd74050f53 as /dev/sdf�[0m
+    volume_test.go:68: �[2m· attach-volume vol-a885f79173e0a8030 -> i-0c615e5dd74050f53 as /dev/sdf�[0m
+    volume_test.go:76: volume vol-a885f79173e0a8030 reached state in-use
+�[2m· detach-volume vol-a885f79173e0a8030�[0m
+    volume_test.go:90: �[2m· detach-volume vol-a885f79173e0a8030�[0m
+    volume_test.go:96: volume vol-a885f79173e0a8030 reached state available
+�[2m· delete-volume vol-a885f79173e0a8030�[0m
+    volume_test.go:98: �[2m· delete-volume vol-a885f79173e0a8030�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5b_ii_VolumeStatus" classname="" time="0.000">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5b-ii — DescribeVolumeStatus ━━�[0m
+�[2mvolume�[0m=vol-2230138124fb51ee2 �[2mstatus�[0m=ok
+    volume_test.go:158: �[2mvolume�[0m=vol-2230138124fb51ee2 �[2mstatus�[0m=ok]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5c_Snapshot" classname="" time="0.190">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5c — Snapshot Lifecycle ━━�[0m
+�[2m· create-snapshot volume=vol-2230138124fb51ee2�[0m
+    snapshot_test.go:34: �[2m· create-snapshot volume=vol-2230138124fb51ee2�[0m
+�[2msnapshot�[0m=snap-1188f92980e6470aa �[2msize�[0m=4 �[2mstate�[0m=completed �[2mprogress�[0m=100%
+    snapshot_test.go:50: �[2msnapshot�[0m=snap-1188f92980e6470aa �[2msize�[0m=4 �[2mstate�[0m=completed �[2mprogress�[0m=100%
+    snapshot_test.go:59: snapshot snap-1188f92980e6470aa reached state completed
+�[2m· describe-snapshots snap-1188f92980e6470aa�[0m
+    snapshot_test.go:61: �[2m· describe-snapshots snap-1188f92980e6470aa�[0m
+�[2m· copy-snapshot src=snap-1188f92980e6470aa region=ap-southeast-2�[0m
+    snapshot_test.go:76: �[2m· copy-snapshot src=snap-1188f92980e6470aa region=ap-southeast-2�[0m
+�[2mcopy_snapshot�[0m=snap-cde6b5ddf467e2c0f
+    snapshot_test.go:86: �[2mcopy_snapshot�[0m=snap-cde6b5ddf467e2c0f
+    snapshot_test.go:88: snapshot snap-cde6b5ddf467e2c0f reached state completed
+�[2m· describe-snapshots snap-1188f92980e6470aa,snap-cde6b5ddf467e2c0f (both visible)�[0m
+    snapshot_test.go:90: �[2m· describe-snapshots snap-1188f92980e6470aa,snap-cde6b5ddf467e2c0f (both visible)�[0m
+�[2m· delete-snapshot snap-1188f92980e6470aa (original)�[0m
+    snapshot_test.go:107: �[2m· delete-snapshot snap-1188f92980e6470aa (original)�[0m
+�[2m· verify copy snap-cde6b5ddf467e2c0f still completed�[0m
+    snapshot_test.go:135: �[2m· verify copy snap-cde6b5ddf467e2c0f still completed�[0m
+�[2m· delete-snapshot snap-cde6b5ddf467e2c0f (copy)�[0m
+    snapshot_test.go:144: �[2m· delete-snapshot snap-cde6b5ddf467e2c0f (copy)�[0m
+�[2mdeleted_original�[0m=snap-1188f92980e6470aa
+    snapshot_test.go:151: �[2mdeleted_original�[0m=snap-1188f92980e6470aa]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5d_SnapshotBackedLaunch" classname="" time="0.010">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5d — Snapshot-Backed Instance Launch ━━�[0m
+�[2mami�[0m=ami-6f60f2b3bf04b5453 �[2msnapshot�[0m=snap-ami-6f60f2b3bf04b5453
+    snapshot_test.go:182: �[2mami�[0m=ami-6f60f2b3bf04b5453 �[2msnapshot�[0m=snap-ami-6f60f2b3bf04b5453]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5e_CreateImage" classname="" time="0.660">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 5e — CreateImage Lifecycle ━━�[0m
+�[2m· create-image instance=i-0c615e5dd74050f53 name=e2e-custom-ami (no-reboot)�[0m
+    createimage_test.go:26: �[2m· create-image instance=i-0c615e5dd74050f53 name=e2e-custom-ami (no-reboot)�[0m
+�[2mcustom_ami�[0m=ami-7dcf263a412dd14c6
+    createimage_test.go:36: �[2mcustom_ami�[0m=ami-7dcf263a412dd14c6
+    createimage_test.go:38: image ami-7dcf263a412dd14c6 reached state available
+�[2m· describe-images ami-7dcf263a412dd14c6�[0m
+    createimage_test.go:40: �[2m· describe-images ami-7dcf263a412dd14c6�[0m
+�[2mcustom_ami_snapshot�[0m=snap-dde89b9ea2ef2e05a
+    createimage_test.go:67: �[2mcustom_ami_snapshot�[0m=snap-dde89b9ea2ef2e05a]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase5f_SecurityGroupEgress" classname="" time="4.880">
+			<skipped message="Skipped"><![CDATA[
+�[1m�[36m━━ Phase 5f — Security Group Enforcement (egress ACL) ━━�[0m
+�[2m· discover default gateway inside VM�[0m
+    securitygroup_test.go:53: �[2m· discover default gateway inside VM�[0m
+�[2mprobe_gateway�[0m=172.31.0.1
+    securitygroup_test.go:66: �[2mprobe_gateway�[0m=172.31.0.1
+�[2m· 5f-1 baseline egress (allow-all)�[0m
+    securitygroup_test.go:77: �[2m· 5f-1 baseline egress (allow-all)�[0m
+    securitygroup_test.go:80: Phase 5f: baseline ICMP did not succeed; env may block ICMP regardless of SG
+        Output:
+        PING 172.31.0.1 (172.31.0.1) 56(84) bytes of data.
+        
+        --- 172.31.0.1 ping statistics ---
+        3 packets transmitted, 0 received, 100% packet loss, time 2044ms
+        ]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/Phase6_TagManagement" classname="" time="0.380">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 6 — Tag Management ━━�[0m
+�[2m· 6a create-tags instance=i-0c615e5dd74050f53 (Name, Environment, DeleteMe)�[0m
+    tags_test.go:41: �[2m· 6a create-tags instance=i-0c615e5dd74050f53 (Name, Environment, DeleteMe)�[0m
+�[2m· 6b describe-tags resource-id=i-0c615e5dd74050f53 want=3�[0m
+    tags_test.go:53: �[2m· 6b describe-tags resource-id=i-0c615e5dd74050f53 want=3�[0m
+�[2m· 6c create-tags volume=vol-2230138124fb51ee2 (Name, Environment)�[0m
+    tags_test.go:65: �[2m· 6c create-tags volume=vol-2230138124fb51ee2 (Name, Environment)�[0m
+�[2m· 6d describe-tags key=Environment want=2�[0m
+    tags_test.go:76: �[2m· 6d describe-tags key=Environment want=2�[0m
+�[2m· 6e describe-tags resource-type=instance want=3�[0m
+    tags_test.go:85: �[2m· 6e describe-tags resource-type=instance want=3�[0m
+�[2m· 6f overwrite instance Name -> e2e-test-updated�[0m
+    tags_test.go:95: �[2m· 6f overwrite instance Name -> e2e-test-updated�[0m
+�[2m· 6g delete-tags Key=DeleteMe (no value)�[0m
+    tags_test.go:114: �[2m· 6g delete-tags Key=DeleteMe (no value)�[0m
+�[2m· 6h delete-tags Key=Environment,Value=production (no-op)�[0m
+    tags_test.go:130: �[2m· 6h delete-tags Key=Environment,Value=production (no-op)�[0m
+�[2m· 6i delete-tags Key=Environment,Value=testing (instance only)�[0m
+    tags_test.go:148: �[2m· 6i delete-tags Key=Environment,Value=testing (instance only)�[0m
+�[2m· 6j final describe-tags resource-id=i-0c615e5dd74050f53 want=1�[0m
+    tags_test.go:171: �[2m· 6j final describe-tags resource-id=i-0c615e5dd74050f53 want=1�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7_StopStart" classname="" time="8.090">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7 — Instance State Transitions (Stop) ━━�[0m
+�[2m· stop-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:30: �[2m· stop-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:36: instance i-0c615e5dd74050f53 reached state stopped
+�[2m· reboot-instances on stopped instance (should fail)�[0m
+    lifecycle_test.go:40: �[2m· reboot-instances on stopped instance (should fail)�[0m
+�[2mstopped_reject�[0m=IncorrectInstanceState
+    lifecycle_test.go:47: �[2mstopped_reject�[0m=IncorrectInstanceState]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7a_AttachToStoppedError" classname="" time="0.300">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7a — Attach Volume to Stopped Instance (Error Path) ━━�[0m
+�[2m· create-volume size=10 az=ap-southeast-2a�[0m
+    lifecycle_test.go:60: �[2m· create-volume size=10 az=ap-southeast-2a�[0m
+�[2mtest_volume�[0m=vol-8d3e1335938a378cd
+    lifecycle_test.go:68: �[2mtest_volume�[0m=vol-8d3e1335938a378cd
+    lifecycle_test.go:82: volume vol-8d3e1335938a378cd reached state available
+�[2m· attach-volume vol-8d3e1335938a378cd -> i-0c615e5dd74050f53 (should fail)�[0m
+    lifecycle_test.go:84: �[2m· attach-volume vol-8d3e1335938a378cd -> i-0c615e5dd74050f53 (should fail)�[0m
+�[2m· delete-volume vol-8d3e1335938a378cd�[0m
+    lifecycle_test.go:94: �[2m· delete-volume vol-8d3e1335938a378cd�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7b_ModifyInstanceAttribute" classname="" time="55.620">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7b — ModifyInstanceAttribute ━━�[0m
+�[2mfrom_type�[0m=t3.nano �[2mto_type�[0m=t3.small
+    lifecycle_test.go:120: �[2mfrom_type�[0m=t3.nano �[2mto_type�[0m=t3.small
+�[2mexpected_vcpus�[0m=2 �[2mexpected_mem_mib�[0m=2048
+    lifecycle_test.go:142: �[2mexpected_vcpus�[0m=2 �[2mexpected_mem_mib�[0m=2048
+�[2m· modify-instance-attribute i-0c615e5dd74050f53 type=t3.small�[0m
+    lifecycle_test.go:144: �[2m· modify-instance-attribute i-0c615e5dd74050f53 type=t3.small�[0m
+�[2m· start-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:165: �[2m· start-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:171: instance i-0c615e5dd74050f53 reached state running
+�[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
+    lifecycle_test.go:177: �[2mssh_host�[0m=192.168.0.201 �[2mssh_port�[0m=22
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    lifecycle_test.go:179: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2m· ssh nproc�[0m
+    lifecycle_test.go:182: �[2m· ssh nproc�[0m
+�[2mvm_vcpus�[0m=2
+    lifecycle_test.go:188: �[2mvm_vcpus�[0m=2
+�[2m· ssh MemTotal�[0m
+    lifecycle_test.go:190: �[2m· ssh MemTotal�[0m
+�[2mvm_mem_mib�[0m=1967 �[2mthreshold_mib�[0m=1740
+    lifecycle_test.go:200: �[2mvm_mem_mib�[0m=1967 �[2mthreshold_mib�[0m=1740
+�[2m· stop-instances i-0c615e5dd74050f53 (end of 7b)�[0m
+    lifecycle_test.go:206: �[2m· stop-instances i-0c615e5dd74050f53 (end of 7b)�[0m
+    lifecycle_test.go:211: instance i-0c615e5dd74050f53 reached state stopped]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7c_pre_Reboot" classname="" time="83.480">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7c-pre — Reboot Running Instance ━━�[0m
+�[2m· start-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:226: �[2m· start-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:231: instance i-0c615e5dd74050f53 reached state running
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    lifecycle_test.go:236: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2mpre_reboot_private_ip�[0m=172.31.0.4
+    lifecycle_test.go:246: �[2mpre_reboot_private_ip�[0m=172.31.0.4
+�[2m· reboot-instances i-0c615e5dd74050f53�[0m
+    lifecycle_test.go:248: �[2m· reboot-instances i-0c615e5dd74050f53�[0m
+�[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+    lifecycle_test.go:279: �[2m· waiting for SSH handshake 192.168.0.201:22�[0m
+�[2m· ssh uptime�[0m
+    lifecycle_test.go:282: �[2m· ssh uptime�[0m
+�[2muptime_secs�[0m=25
+    lifecycle_test.go:288: �[2muptime_secs�[0m=25]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase7c_RunInstancesMultiCount" classname="" time="47.560">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 7c — RunInstances with MinCount/MaxCount > 1 ━━�[0m
+�[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.small count=2�[0m
+    lifecycle_test.go:309: �[2m· run-instances ami=ami-6f60f2b3bf04b5453 type=t3.small count=2�[0m
+�[2msibling_1�[0m=i-9bfadd8ebd378f846 �[2msibling_2�[0m=i-448a9e2bf45c58658
+    lifecycle_test.go:327: �[2msibling_1�[0m=i-9bfadd8ebd378f846 �[2msibling_2�[0m=i-448a9e2bf45c58658
+    lifecycle_test.go:342: instance i-9bfadd8ebd378f846 reached state running
+    lifecycle_test.go:342: instance i-448a9e2bf45c58658 reached state running
+�[2m· terminate-instances i-9bfadd8ebd378f846 i-448a9e2bf45c58658�[0m
+    lifecycle_test.go:345: �[2m· terminate-instances i-9bfadd8ebd378f846 i-448a9e2bf45c58658�[0m
+    lifecycle_test.go:352: instance i-9bfadd8ebd378f846 reached state terminated
+    lifecycle_test.go:352: instance i-448a9e2bf45c58658 reached state terminated]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths" classname="" time="0.000">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8 — Negative / Error Path Tests ━━�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8a_InvalidAMIIDMalformed" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· run-instances image-id=notanami�[0m
+    negative_test.go:55: �[2m· run-instances image-id=notanami�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8b_InvalidInstanceType" classname="" time="0.210">
+			<system-out><![CDATA[�[2m· run-instances instance-type=x99.superlarge�[0m
+    negative_test.go:70: �[2m· run-instances instance-type=x99.superlarge�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8c_VolumeInUse" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· attach-volume vol-2230138124fb51ee2 (root, in-use)�[0m
+    negative_test.go:85: �[2m· attach-volume vol-2230138124fb51ee2 (root, in-use)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8d_DetachRootForbidden" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· detach-volume vol-2230138124fb51ee2 (root)�[0m
+    negative_test.go:97: �[2m· detach-volume vol-2230138124fb51ee2 (root)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8e_DeleteSnapshotNotFound" classname="" time="0.030">
+			<system-out><![CDATA[�[2m· delete-snapshot snap-nonexistent000000�[0m
+    negative_test.go:108: �[2m· delete-snapshot snap-nonexistent000000�[0m
+�[2m· describe-volumes vol-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8f_InvalidActionRawHTTP" classname="" time="0.040">
+			<system-out><![CDATA[�[2m· POST Action=DescribeFakeThings�[0m
+    negative_test.go:122: �[2m· POST Action=DescribeFakeThings�[0m
+    negative_test.go:125: �[2mstatus�[0m=400 �[2mcode�[0m=InvalidAction]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8g_InvalidAMIIDNotFound" classname="" time="0.240">
+			<system-out><![CDATA[�[2m· reboot-instances i-nonexistent�[0m
+�[2m· run-instances image-id=ami-0000000000000dead�[0m
+    negative_test.go:138: �[2m· run-instances image-id=ami-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8h_InvalidKeyPairNotFound" classname="" time="0.260">
+			<system-out><![CDATA[�[2m· run-instances key-name=nonexistent-key-xyz�[0m
+    negative_test.go:153: �[2m· run-instances key-name=nonexistent-key-xyz�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8i_DeleteVolumeNotFound" classname="" time="0.040">
+			<system-out><![CDATA[�[2m· delete-volume vol-0000000000000dead�[0m
+    negative_test.go:168: �[2m· delete-volume vol-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8j_CreateKeyPairDuplicate" classname="" time="0.040">
+			<system-out><![CDATA[    negative_test.go:178: �[2m· create-key-pair test-key-1 (duplicate)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8k_ImportKeyPairDuplicate" classname="" time="0.040">
+			<system-out><![CDATA[�[2m· import-key-pair test-key-1 (duplicate)�[0m
+    negative_test.go:198: �[2m· import-key-pair test-key-1 (duplicate)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8l_InvalidKeyFormat" classname="" time="0.040">
+			<system-out><![CDATA[    negative_test.go:210: �[2m· import-key-pair bad-format-key (bad material)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8m_DescribeVolumesNotFound" classname="" time="0.040">
+			<system-out><![CDATA[    negative_test.go:230: �[2m· describe-volumes vol-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8n_DescribeImagesNotFound" classname="" time="0.040">
+			<system-out><![CDATA[�[2m· describe-images ami-0000000000000dead�[0m
+    negative_test.go:240: �[2m· describe-images ami-0000000000000dead�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8o_CreateImageDuplicateName" classname="" time="0.560">
+			<system-out><![CDATA[�[2m· create-key-pair test-key-1 (duplicate)�[0m
+�[2m· create-image name=e2e-custom-ami (duplicate)�[0m
+�[2m· import-key-pair bad-format-key (bad material)�[0m
+    negative_test.go:252: �[2m· create-image name=e2e-custom-ami (duplicate)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8p_DeleteKeyPairIdempotent" classname="" time="0.020">
+			<system-out><![CDATA[�[2m· delete-key-pair nonexistent-key-99999 (idempotent)�[0m
+    negative_test.go:276: �[2m· delete-key-pair nonexistent-key-99999 (idempotent)�[0m
+�[2mstatus�[0m=400 �[2mcode�[0m=InvalidAction]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8q_ModifyAttributeOnRunning" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· modify-instance-attribute i-0c615e5dd74050f53 (running)�[0m
+    negative_test.go:289: �[2m· modify-instance-attribute i-0c615e5dd74050f53 (running)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8_NegativeErrorPaths/8r_RebootInstanceNotFound" classname="" time="0.040">
+			<system-out><![CDATA[    negative_test.go:302: �[2m· reboot-instances i-nonexistent�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM1_UserCRUD" classname="" time="0.070">
+			<system-out><![CDATA[
+�[1m�[36m━━ IAM Phase 1 — User CRUD ━━�[0m
+�[2m· list-users (root auth sanity)�[0m
+    iam_test.go:109: �[2m· list-users (root auth sanity)�[0m
+�[2m· create-user "alice"�[0m
+    iam_test.go:114: �[2m· create-user "alice"�[0m
+�[2muser�[0m=alice �[2marn�[0m=arn:aws:iam::000000000001:user/alice
+    iam_test.go:120: �[2muser�[0m=alice �[2marn�[0m=arn:aws:iam::000000000001:user/alice
+�[2m· create-user "bob" path="/engineering/"�[0m
+    iam_test.go:122: �[2m· create-user "bob" path="/engineering/"�[0m
+�[2m· create-user "alice" again (expect EntityAlreadyExists)�[0m
+    iam_test.go:131: �[2m· create-user "alice" again (expect EntityAlreadyExists)�[0m
+�[2m· get-user "alice"�[0m
+    iam_test.go:137: �[2m· get-user "alice"�[0m
+�[2m· get-user nonexistent (expect NoSuchEntity)�[0m
+    iam_test.go:142: �[2m· get-user nonexistent (expect NoSuchEntity)�[0m
+�[2m· list-users (>=3 root+alice+bob)�[0m
+    iam_test.go:148: �[2m· list-users (>=3 root+alice+bob)�[0m
+�[2m· list-users --path-prefix "/engineering/" (expect 1)�[0m
+    iam_test.go:154: �[2m· list-users --path-prefix "/engineering/" (expect 1)�[0m
+    iam_test.go:193: �[2m· create-user "iam-phase1-ephemeral" (delete idempotency probe)�[0m
+�[2m· delete-user "iam-phase1-ephemeral"�[0m
+    iam_test.go:197: �[2m· delete-user "iam-phase1-ephemeral"�[0m
+�[2m· delete-user "iam-phase1-ephemeral" again (expect NoSuchEntity)�[0m
+    iam_test.go:201: �[2m· delete-user "iam-phase1-ephemeral" again (expect NoSuchEntity)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM1_UserCRUD/UpdateUser" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[    iam_test.go:165: daemon UpdateUser handler not implemented — InvalidAction
+�[2m· create-user "iam-phase1-ephemeral" (delete idempotency probe)�[0m]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM2_AccessKeyLifecycle" classname="" time="0.040">
+			<system-out><![CDATA[
+�[1m�[36m━━ IAM Phase 2 — Access Key Lifecycle ━━�[0m
+�[2m· create-access-key user=alice (key 1)�[0m
+    iam_test.go:218: �[2m· create-access-key user=alice (key 1)�[0m
+�[2mkey1�[0m=AKIA4984AE57CF8D721A879A
+    iam_test.go:227: �[2mkey1�[0m=AKIA4984AE57CF8D721A879A
+�[2m· create-access-key user=alice (key 2)�[0m
+    iam_test.go:229: �[2m· create-access-key user=alice (key 2)�[0m
+�[2mkey2�[0m=AKIAA1CD38D4F0E573216E11
+    iam_test.go:236: �[2mkey2�[0m=AKIAA1CD38D4F0E573216E11
+�[2m· create-access-key user=alice key 3 (expect LimitExceeded)�[0m
+    iam_test.go:238: �[2m· create-access-key user=alice key 3 (expect LimitExceeded)�[0m
+�[2m· create-access-key user=ghost (expect NoSuchEntity)�[0m
+    iam_test.go:246: �[2m· create-access-key user=ghost (expect NoSuchEntity)�[0m
+�[2m· list-access-keys user=alice (expect 2)�[0m
+    iam_test.go:254: �[2m· list-access-keys user=alice (expect 2)�[0m
+�[2m· list-access-keys user=bob (expect 0)�[0m
+    iam_test.go:261: �[2m· list-access-keys user=bob (expect 0)�[0m
+�[2m· update-access-key AKIA4984AE57CF8D721A879A -> Inactive�[0m
+    iam_test.go:268: �[2m· update-access-key AKIA4984AE57CF8D721A879A -> Inactive�[0m
+�[2m· update-access-key AKIA4984AE57CF8D721A879A -> Active�[0m
+    iam_test.go:279: �[2m· update-access-key AKIA4984AE57CF8D721A879A -> Active�[0m
+�[2m· delete-access-key AKIAA1CD38D4F0E573216E11 (key 2)�[0m
+    iam_test.go:287: �[2m· delete-access-key AKIAA1CD38D4F0E573216E11 (key 2)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM3_UserAuthentication" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[
+�[1m�[36m━━ IAM Phase 3 — User Authentication ━━�[0m
+    iam_test.go:311: daemon IAM signature/principal lookup not implemented — mulga-siv-100]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD" classname="" time="0.040">
+			<system-out><![CDATA[
+�[1m�[36m━━ IAM Phase 4 — Policy CRUD ━━�[0m
+�[2m· create-policy EC2ReadOnly�[0m
+    iam_test.go:384: �[2m· create-policy EC2ReadOnly�[0m
+�[2mpolicy�[0m=EC2ReadOnly �[2marn�[0m=arn:aws:iam::000000000001:policy/EC2ReadOnly �[2maccount�[0m=000000000001
+    iam_test.go:393: �[2mpolicy�[0m=EC2ReadOnly �[2marn�[0m=arn:aws:iam::000000000001:policy/EC2ReadOnly �[2maccount�[0m=000000000001
+    iam_test.go:440: �[2m· create-policy duplicate (expect EntityAlreadyExists)�[0m
+�[2m· create-policy malformed (expect MalformedPolicyDocument)�[0m
+    iam_test.go:449: �[2m· create-policy malformed (expect MalformedPolicyDocument)�[0m
+�[2m· get-policy EC2ReadOnly�[0m
+    iam_test.go:458: �[2m· get-policy EC2ReadOnly�[0m
+�[2m· get-policy nonexistent (expect NoSuchEntity)�[0m
+    iam_test.go:463: �[2m· get-policy nonexistent (expect NoSuchEntity)�[0m
+�[2m· get-policy-version EC2ReadOnly v1�[0m
+    iam_test.go:470: �[2m· get-policy-version EC2ReadOnly v1�[0m
+    iam_test.go:519: �[2m· list-policies (>=5)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel" classname="" time="0.000"></testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/FullAdmin" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· create-policy FullAdmin path=/admin/�[0m
+    iam_test.go:402: �[2m· create-policy FullAdmin path=/admin/�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/DenyTerminate" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· create-policy DenyTerminate (mixed Allow+Deny)�[0m
+    iam_test.go:413: �[2m· create-policy DenyTerminate (mixed Allow+Deny)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/IAMReadOnly" classname="" time="0.020">
+			<system-out><![CDATA[�[2m· create-policy IAMReadOnly�[0m
+    iam_test.go:422: �[2m· create-policy IAMReadOnly�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/create_policies_parallel/EC2DescribeAll" classname="" time="0.020">
+			<system-out><![CDATA[�[2m· create-policy EC2DescribeAll (wildcard)�[0m
+    iam_test.go:431: �[2m· create-policy EC2DescribeAll (wildcard)�[0m
+�[2m· create-policy duplicate (expect EntityAlreadyExists)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/IAM4_PolicyCRUD/PolicyVersionLifecycle" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[    iam_test.go:484: daemon CreatePolicyVersion handler not implemented — InvalidAction
+�[2m· list-policies (>=5)�[0m]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM5_PolicyAttachmentEnforcement" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[
+�[1m�[36m━━ IAM Phase 5 — Policy Attachment & Enforcement ━━�[0m
+    iam_test.go:535: daemon IAM scoped-credential enforcement not implemented — mulga-siv-100]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM6_PolicyLifecycle" classname="" time="0.000">
+			<skipped message="Skipped"><![CDATA[
+�[1m�[36m━━ IAM Phase 6 — Policy Lifecycle (Detach & Delete) ━━�[0m
+    iam_test.go:718: daemon DetachUserPolicy not implemented — mulga-siv-100]]></skipped>
+		</testcase>
+		<testcase name="TestSingleNode/IAM7_Cleanup" classname="" time="0.050">
+			<system-out><![CDATA[
+�[1m�[36m━━ IAM Phase 7 — Cleanup ━━�[0m
+�[2m· cleanup user alice�[0m
+    iam_test.go:770: �[2m· cleanup user alice�[0m
+�[2m· cleanup user bob�[0m
+    iam_test.go:770: �[2m· cleanup user bob�[0m
+�[2m· cleanup user charlie�[0m
+    iam_test.go:770: �[2m· cleanup user charlie�[0m
+�[2mremaining_users�[0m=1 �[2mnames�[0m=[admin]
+    iam_test.go:779: �[2mremaining_users�[0m=1 �[2mnames�[0m=[admin]
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2ReadOnly�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2ReadOnly�[0m
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/admin/FullAdmin�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/admin/FullAdmin�[0m
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/IAMReadOnly�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/IAMReadOnly�[0m
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2DescribeAll�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/EC2DescribeAll�[0m
+�[2m· cleanup policy arn:aws:iam::000000000001:policy/DenyTerminate�[0m
+    iam_test.go:806: �[2m· cleanup policy arn:aws:iam::000000000001:policy/DenyTerminate�[0m
+�[2mremaining_policies�[0m=1 �[2mnames�[0m=[AdministratorAccess]
+    iam_test.go:812: �[2mremaining_policies�[0m=1 �[2mnames�[0m=[AdministratorAccess]]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping" classname="" time="114.890">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8 (acct) — EC2 Account Scoping ━━�[0m
+    account_test.go:75: �[2m· Phase 8 acct cleanup (deferred)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step1_AccountSetup" classname="" time="1.220">
+			<system-out><![CDATA[�[2m· create account 'Team Alpha'�[0m
+    account_test.go:206: �[2m· create account 'Team Alpha'�[0m
+�[2malpha_account�[0m=000000000002 �[2malpha_key�[0m=AKIA374CBBB2A15D3E2EE822
+    account_test.go:211: �[2malpha_account�[0m=000000000002 �[2malpha_key�[0m=AKIA374CBBB2A15D3E2EE822
+�[2m· create account 'Team Beta'�[0m
+    account_test.go:213: �[2m· create account 'Team Beta'�[0m
+�[2mbeta_account�[0m=000000000003 �[2mbeta_key�[0m=AKIA0AA0B2845E3BD5A26D01
+    account_test.go:218: �[2mbeta_account�[0m=000000000003 �[2mbeta_key�[0m=AKIA0AA0B2845E3BD5A26D01]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step2_InstanceScoping" classname="" time="52.380">
+			<system-out><![CDATA[�[2m· create per-account key pairs�[0m
+    account_test.go:242: �[2m· create per-account key pairs�[0m
+�[2m· alpha run-instances�[0m
+    account_test.go:250: �[2m· alpha run-instances�[0m
+�[2malpha_instance�[0m=i-a3072c2387630b8ee
+    account_test.go:262: �[2malpha_instance�[0m=i-a3072c2387630b8ee
+�[2m· beta run-instances�[0m
+    account_test.go:264: �[2m· beta run-instances�[0m
+�[2mbeta_instance�[0m=i-ad32cbb0afaee5eee
+    account_test.go:276: �[2mbeta_instance�[0m=i-ad32cbb0afaee5eee
+    account_test.go:278: instance i-a3072c2387630b8ee reached state running
+    account_test.go:279: instance i-ad32cbb0afaee5eee reached state running
+�[2m· alpha sees only own instances�[0m
+    account_test.go:281: �[2m· alpha sees only own instances�[0m
+�[2m· beta sees only own instances�[0m
+    account_test.go:285: �[2m· beta sees only own instances�[0m
+�[2m· alpha OwnerId matches alpha account�[0m
+    account_test.go:289: �[2m· alpha OwnerId matches alpha account�[0m
+�[2m· cross-account stop alpha->beta blocked�[0m
+    account_test.go:296: �[2m· cross-account stop alpha->beta blocked�[0m
+�[2m· cross-account terminate beta->alpha blocked�[0m
+    account_test.go:304: �[2m· cross-account terminate beta->alpha blocked�[0m
+�[2m· cross-account reboot alpha->beta blocked�[0m
+    account_test.go:312: �[2m· cross-account reboot alpha->beta blocked�[0m
+�[2m· stop alpha instance for cross-account start/modify/console tests�[0m
+    account_test.go:321: �[2m· stop alpha instance for cross-account start/modify/console tests�[0m
+    account_test.go:326: instance i-a3072c2387630b8ee reached state stopped
+�[2m· cross-account start beta->alpha blocked�[0m
+    account_test.go:328: �[2m· cross-account start beta->alpha blocked�[0m
+�[2m· cross-account modify-instance-attribute beta->alpha blocked�[0m
+    account_test.go:336: �[2m· cross-account modify-instance-attribute beta->alpha blocked�[0m
+�[2m· cross-account get-console-output beta->alpha blocked�[0m
+    account_test.go:345: �[2m· cross-account get-console-output beta->alpha blocked�[0m
+�[2m· restart alpha instance�[0m
+    account_test.go:354: �[2m· restart alpha instance�[0m
+    account_test.go:359: instance i-a3072c2387630b8ee reached state running]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step3_VolumeScoping" classname="" time="24.140">
+			<system-out><![CDATA[�[2m· alpha create-volume�[0m
+    account_test.go:366: �[2m· alpha create-volume�[0m
+�[2malpha_vol�[0m=vol-ceef82652369780af
+    account_test.go:375: �[2malpha_vol�[0m=vol-ceef82652369780af
+�[2m· beta create-volume�[0m
+    account_test.go:377: �[2m· beta create-volume�[0m
+�[2mbeta_vol�[0m=vol-08024f32d23d9e18c
+    account_test.go:386: �[2mbeta_vol�[0m=vol-08024f32d23d9e18c
+    account_test.go:388: volume vol-ceef82652369780af reached state available
+    account_test.go:389: volume vol-08024f32d23d9e18c reached state available
+�[2m· alpha sees only own volumes�[0m
+    account_test.go:391: �[2m· alpha sees only own volumes�[0m
+�[2m· cross-account describe-volume by id blocked�[0m
+    account_test.go:395: �[2m· cross-account describe-volume by id blocked�[0m
+�[2m· cross-account delete-volume blocked�[0m
+    account_test.go:403: �[2m· cross-account delete-volume blocked�[0m
+�[2m· cross-account attach-volume (beta attaches alpha's vol) blocked�[0m
+    account_test.go:411: �[2m· cross-account attach-volume (beta attaches alpha's vol) blocked�[0m
+�[2m· alpha attach-volume to own instance�[0m
+    account_test.go:422: �[2m· alpha attach-volume to own instance�[0m
+    account_test.go:429: volume vol-ceef82652369780af reached state in-use
+�[2m· cross-account detach-volume blocked�[0m
+    account_test.go:431: �[2m· cross-account detach-volume blocked�[0m
+�[2m· cross-account modify-volume blocked�[0m
+    account_test.go:439: �[2m· cross-account modify-volume blocked�[0m
+�[2m· alpha detach own volume�[0m
+    account_test.go:449: �[2m· alpha detach own volume�[0m
+    account_test.go:454: volume vol-ceef82652369780af reached state available]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step4_KeyPairScoping" classname="" time="0.370">
+			<system-out><![CDATA[�[2m· alpha create-key-pair alpha-key�[0m
+    account_test.go:466: �[2m· alpha create-key-pair alpha-key�[0m
+�[2m· beta create-key-pair beta-key�[0m
+    account_test.go:473: �[2m· beta create-key-pair beta-key�[0m
+�[2m· alpha sees only own keys�[0m
+    account_test.go:478: �[2m· alpha sees only own keys�[0m
+�[2m· namespace isolation: shared-name in both accounts�[0m
+    account_test.go:483: �[2m· namespace isolation: shared-name in both accounts�[0m
+�[2m· beta deletes alpha-key (idempotent, no cross-account effect)�[0m
+    account_test.go:497: �[2m· beta deletes alpha-key (idempotent, no cross-account effect)�[0m
+�[2m· alpha import-key-pair imported-key�[0m
+    account_test.go:503: �[2m· alpha import-key-pair imported-key�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step5_SnapshotScoping" classname="" time="0.060">
+			<system-out><![CDATA[�[2m· alpha create-snapshot�[0m
+    account_test.go:522: �[2m· alpha create-snapshot�[0m
+�[2malpha_snap�[0m=snap-2a47896f4304d805c
+    account_test.go:530: �[2malpha_snap�[0m=snap-2a47896f4304d805c
+�[2m· beta create-snapshot�[0m
+    account_test.go:532: �[2m· beta create-snapshot�[0m
+�[2mbeta_snap�[0m=snap-bf5744e78bd7c6908
+    account_test.go:540: �[2mbeta_snap�[0m=snap-bf5744e78bd7c6908
+�[2m· alpha sees only own snapshots (owner-ids=self)�[0m
+    account_test.go:542: �[2m· alpha sees only own snapshots (owner-ids=self)�[0m
+�[2m· cross-account delete-snapshot blocked (UnauthorizedOperation)�[0m
+    account_test.go:560: �[2m· cross-account delete-snapshot blocked (UnauthorizedOperation)�[0m
+�[2m· cross-account create-snapshot of alpha's volume blocked�[0m
+    account_test.go:568: �[2m· cross-account create-snapshot of alpha's volume blocked�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step6_VPCSubnetScoping" classname="" time="0.030">
+			<system-out><![CDATA[�[2m· alpha create-vpc 10.0.0.0/16�[0m
+    account_test.go:582: �[2m· alpha create-vpc 10.0.0.0/16�[0m
+�[2malpha_vpc�[0m=vpc-7f60df5e87183fb59
+    account_test.go:589: �[2malpha_vpc�[0m=vpc-7f60df5e87183fb59
+�[2m· beta create-vpc 10.0.0.0/16 (same CIDR, no conflict)�[0m
+    account_test.go:591: �[2m· beta create-vpc 10.0.0.0/16 (same CIDR, no conflict)�[0m
+�[2mbeta_vpc�[0m=vpc-7e8c859d3985c8218
+    account_test.go:598: �[2mbeta_vpc�[0m=vpc-7e8c859d3985c8218
+�[2m· alpha describe-vpcs isolation�[0m
+    account_test.go:600: �[2m· alpha describe-vpcs isolation�[0m
+�[2m· cross-account describe-vpc by id blocked�[0m
+    account_test.go:605: �[2m· cross-account describe-vpc by id blocked�[0m
+�[2m· cross-account delete-vpc blocked�[0m
+    account_test.go:613: �[2m· cross-account delete-vpc blocked�[0m
+�[2m· alpha create-subnet 10.0.1.0/24�[0m
+    account_test.go:621: �[2m· alpha create-subnet 10.0.1.0/24�[0m
+�[2m· beta create-subnet 10.0.1.0/24�[0m
+    account_test.go:630: �[2m· beta create-subnet 10.0.1.0/24�[0m
+�[2m· alpha describe-subnets isolation�[0m
+    account_test.go:639: �[2m· alpha describe-subnets isolation�[0m
+�[2m· cross-account create-subnet in other VPC blocked�[0m
+    account_test.go:648: �[2m· cross-account create-subnet in other VPC blocked�[0m
+�[2m· cross-account delete-subnet blocked�[0m
+    account_test.go:657: �[2m· cross-account delete-subnet blocked�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step7_IGWEIGWScoping" classname="" time="0.030">
+			<system-out><![CDATA[�[2m· alpha create-internet-gateway�[0m
+    account_test.go:673: �[2m· alpha create-internet-gateway�[0m
+�[2malpha_igw�[0m=igw-ee966e3e923caf630
+    account_test.go:678: �[2malpha_igw�[0m=igw-ee966e3e923caf630
+�[2m· beta create-internet-gateway�[0m
+    account_test.go:680: �[2m· beta create-internet-gateway�[0m
+�[2mbeta_igw�[0m=igw-b46e66f2065889bb9
+    account_test.go:685: �[2mbeta_igw�[0m=igw-b46e66f2065889bb9
+�[2m· alpha describe-internet-gateways isolation�[0m
+    account_test.go:687: �[2m· alpha describe-internet-gateways isolation�[0m
+�[2m· cross-account describe IGW by id blocked�[0m
+    account_test.go:696: �[2m· cross-account describe IGW by id blocked�[0m
+�[2m· cross-account delete IGW blocked�[0m
+    account_test.go:704: �[2m· cross-account delete IGW blocked�[0m
+�[2m· cross-account attach IGW (alpha attaches beta's IGW) blocked�[0m
+    account_test.go:712: �[2m· cross-account attach IGW (alpha attaches beta's IGW) blocked�[0m
+�[2m· alpha attach own IGW to own VPC�[0m
+    account_test.go:722: �[2m· alpha attach own IGW to own VPC�[0m
+�[2m· cross-account detach IGW blocked�[0m
+    account_test.go:729: �[2m· cross-account detach IGW blocked�[0m
+�[2m· alpha create-egress-only-internet-gateway�[0m
+    account_test.go:739: �[2m· alpha create-egress-only-internet-gateway�[0m
+�[2malpha_eigw�[0m=eigw-51ba82da927e49393
+    account_test.go:746: �[2malpha_eigw�[0m=eigw-51ba82da927e49393
+�[2m· beta create-egress-only-internet-gateway�[0m
+    account_test.go:748: �[2m· beta create-egress-only-internet-gateway�[0m
+�[2mbeta_eigw�[0m=eigw-e88d642e78ab4fe5a
+    account_test.go:755: �[2mbeta_eigw�[0m=eigw-e88d642e78ab4fe5a
+�[2m· alpha describe-eigws isolation�[0m
+    account_test.go:757: �[2m· alpha describe-eigws isolation�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step8_AccountSettings" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· alpha enable-ebs-encryption-by-default�[0m
+    account_test.go:794: �[2m· alpha enable-ebs-encryption-by-default�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step9_GlobalResources" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· describe-regions identical across accounts�[0m
+    account_test.go:829: �[2m· describe-regions identical across accounts�[0m
+�[2m· describe-availability-zones identical across accounts�[0m
+    account_test.go:834: �[2m· describe-availability-zones identical across accounts�[0m
+�[2m· describe-instance-types identical across accounts�[0m
+    account_test.go:839: �[2m· describe-instance-types identical across accounts�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step10_EdgeCases" classname="" time="1.200">
+			<system-out><![CDATA[�[2m· create empty Gamma account�[0m
+    account_test.go:849: �[2m· create empty Gamma account�[0m
+�[2mgamma_account�[0m=000000000004
+    account_test.go:852: �[2mgamma_account�[0m=000000000004
+�[2m· root create-key-pair root-scoping-key�[0m
+    account_test.go:879: �[2m· root create-key-pair root-scoping-key�[0m
+�[2m· non-existent volume id -> InvalidVolume.NotFound�[0m
+    account_test.go:895: �[2m· non-existent volume id -> InvalidVolume.NotFound�[0m
+�[2m· non-existent snapshot id -> InvalidSnapshot.NotFound�[0m
+    account_test.go:903: �[2m· non-existent snapshot id -> InvalidSnapshot.NotFound�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8Acct_AccountScoping/Step11_Cleanup" classname="" time="32.440">
+			<system-out><![CDATA[�[2m· terminate alpha instance�[0m
+    account_test.go:920: �[2m· terminate alpha instance�[0m
+�[2m· terminate beta instance�[0m
+    account_test.go:926: �[2m· terminate beta instance�[0m
+�[2m· delete snapshots�[0m
+    account_test.go:939: �[2m· delete snapshots�[0m
+�[2m· delete volumes�[0m
+    account_test.go:955: �[2m· delete volumes�[0m
+�[2m· delete key pairs�[0m
+    account_test.go:969: �[2m· delete key pairs�[0m
+�[2m· delete EIGWs�[0m
+    account_test.go:979: �[2m· delete EIGWs�[0m
+�[2m· detach + delete IGWs�[0m
+    account_test.go:993: �[2m· detach + delete IGWs�[0m
+�[2m· delete subnets�[0m
+    account_test.go:1013: �[2m· delete subnets�[0m
+�[2m· delete VPCs�[0m
+    account_test.go:1027: �[2m· delete VPCs�[0m
+�[2m· Phase 8 acct cleanup (deferred)�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8b_VPCSubnetE2E" classname="" time="200.480">
+			<failure message="Failed"><![CDATA[
+�[1m�[36m━━ Phase 8b — VPC Public/Private Subnet E2E ━━�[0m
+�[2m· create-vpc 10.99.0.0/16�[0m
+    vpc_test.go:56: �[2m· create-vpc 10.99.0.0/16�[0m
+�[2mvpc�[0m=vpc-39eff985799c43069 �[2mcidr�[0m=10.99.0.0/16
+    vpc_test.go:67: �[2mvpc�[0m=vpc-39eff985799c43069 �[2mcidr�[0m=10.99.0.0/16
+�[2m· create-subnet 10.99.1.0/24 (public)�[0m
+    vpc_test.go:70: �[2m· create-subnet 10.99.1.0/24 (public)�[0m
+�[2m· modify-subnet-attribute MapPublicIpOnLaunch=true subnet-22aba054e10588254�[0m
+    vpc_test.go:87: �[2m· modify-subnet-attribute MapPublicIpOnLaunch=true subnet-22aba054e10588254�[0m
+�[2mpub_subnet�[0m=subnet-22aba054e10588254 �[2mmap_public_ip�[0m=true
+    vpc_test.go:93: �[2mpub_subnet�[0m=subnet-22aba054e10588254 �[2mmap_public_ip�[0m=true
+�[2m· create-subnet 10.99.2.0/24 (private)�[0m
+    vpc_test.go:96: �[2m· create-subnet 10.99.2.0/24 (private)�[0m
+�[2mpriv_subnet�[0m=subnet-f53aeddb73c134696 �[2mmap_public_ip�[0m=false
+    vpc_test.go:111: �[2mpriv_subnet�[0m=subnet-f53aeddb73c134696 �[2mmap_public_ip�[0m=false
+�[2m· create-internet-gateway�[0m
+    vpc_test.go:114: �[2m· create-internet-gateway�[0m
+�[2m· attach-internet-gateway igw-1520b5a34480da982 -> vpc-39eff985799c43069�[0m
+    vpc_test.go:133: �[2m· attach-internet-gateway igw-1520b5a34480da982 -> vpc-39eff985799c43069�[0m
+�[2migw�[0m=igw-1520b5a34480da982
+    vpc_test.go:139: �[2migw�[0m=igw-1520b5a34480da982
+�[2m· create-route-table (vpc=vpc-39eff985799c43069)�[0m
+    vpc_test.go:142: �[2m· create-route-table (vpc=vpc-39eff985799c43069)�[0m
+�[2m· associate-route-table rtb-662df5e424cbdb60c <- subnet-22aba054e10588254�[0m
+    vpc_test.go:159: �[2m· associate-route-table rtb-662df5e424cbdb60c <- subnet-22aba054e10588254�[0m
+�[2m· create-route 0.0.0.0/0 -> igw-1520b5a34480da982�[0m
+    vpc_test.go:176: �[2m· create-route 0.0.0.0/0 -> igw-1520b5a34480da982�[0m
+�[2mrtb�[0m=rtb-662df5e424cbdb60c �[2massoc�[0m=rtbassoc-5510bf1dc6924eb0b �[2mroute�[0m=0.0.0.0/0->igw-1520b5a34480da982
+    vpc_test.go:192: �[2mrtb�[0m=rtb-662df5e424cbdb60c �[2massoc�[0m=rtbassoc-5510bf1dc6924eb0b �[2mroute�[0m=0.0.0.0/0->igw-1520b5a34480da982
+�[2m· run-instances ami=ami-6f60f2b3bf04b5453 subnet=subnet-22aba054e10588254�[0m
+    vpc_test.go:195: �[2m· run-instances ami=ami-6f60f2b3bf04b5453 subnet=subnet-22aba054e10588254�[0m
+�[2minstance�[0m=i-3177888cba3796d42
+    vpc_test.go:216: �[2minstance�[0m=i-3177888cba3796d42
+    vpc_test.go:218: instance i-3177888cba3796d42 reached state running
+�[2mpublic_ip�[0m=192.168.0.202 �[2mprivate_ip�[0m=10.99.1.4
+    vpc_test.go:224: �[2mpublic_ip�[0m=192.168.0.202 �[2mprivate_ip�[0m=10.99.1.4
+�[2m· waiting for SSH handshake 192.168.0.202:22�[0m
+    vpc_test.go:227: �[2m· waiting for SSH handshake 192.168.0.202:22�[0m
+    vpc_test.go:227: Eventually: condition not met within 3m0s: [SSH handshake 192.168.0.202:22 never completed]]]></failure>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8c_RouteTableValidation" classname="" time="0.030">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8c — Route Table Validation ━━�[0m
+�[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63
+    vpc_test.go:292: �[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2msubnet�[0m=subnet-cd62e4c1c3cb50c63]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8c_RouteTableValidation/Step1_DefaultMainRouteTable" classname="" time="0.000">
+			<system-out><![CDATA[�[2m· describe-route-tables filter=vpc-id,association.main�[0m
+    vpc_test.go:297: �[2m· describe-route-tables filter=vpc-id,association.main�[0m
+�[2mmain_rtb�[0m=rtb-e42012569f92f7184
+    vpc_test.go:309: �[2mmain_rtb�[0m=rtb-e42012569f92f7184
+�[2mlocal_route�[0m=172.31.0.0/16
+    vpc_test.go:332: �[2mlocal_route�[0m=172.31.0.0/16
+�[2mdefault_route�[0m=0.0.0.0/0->igw-79c2e643660b4b7e3
+    vpc_test.go:350: �[2mdefault_route�[0m=0.0.0.0/0->igw-79c2e643660b4b7e3]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8c_RouteTableValidation/Step2_CustomRouteTableLifecycle" classname="" time="0.020">
+			<system-out><![CDATA[�[2m· create-route-table (vpc=vpc-531252db3b4e1fc08)�[0m
+    vpc_test.go:363: �[2m· create-route-table (vpc=vpc-531252db3b4e1fc08)�[0m
+�[2mcustom_rtb�[0m=rtb-5e6c46936f3e44167
+    vpc_test.go:379: �[2mcustom_rtb�[0m=rtb-5e6c46936f3e44167
+�[2mcustom_local_route�[0m=172.31.0.0/16
+    vpc_test.go:396: �[2mcustom_local_route�[0m=172.31.0.0/16
+�[2mvpc_igw�[0m=igw-79c2e643660b4b7e3
+    vpc_test.go:410: �[2mvpc_igw�[0m=igw-79c2e643660b4b7e3
+�[2m· create-route 0.0.0.0/0 -> igw-79c2e643660b4b7e3�[0m
+    vpc_test.go:415: �[2m· create-route 0.0.0.0/0 -> igw-79c2e643660b4b7e3�[0m
+�[2m· associate-route-table rtb-5e6c46936f3e44167 <- subnet-cd62e4c1c3cb50c63�[0m
+    vpc_test.go:434: �[2m· associate-route-table rtb-5e6c46936f3e44167 <- subnet-cd62e4c1c3cb50c63�[0m
+�[2mcustom_assoc�[0m=rtbassoc-da6377d37847d3d4a
+    vpc_test.go:450: �[2mcustom_assoc�[0m=rtbassoc-da6377d37847d3d4a
+�[2m· disassociate-route-table rtbassoc-da6377d37847d3d4a�[0m
+    vpc_test.go:452: �[2m· disassociate-route-table rtbassoc-da6377d37847d3d4a�[0m
+�[2m· delete-route 0.0.0.0/0�[0m
+    vpc_test.go:460: �[2m· delete-route 0.0.0.0/0�[0m
+�[2m· delete-route-table rtb-5e6c46936f3e44167�[0m
+    vpc_test.go:468: �[2m· delete-route-table rtb-5e6c46936f3e44167�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8c_RouteTableValidation/Step3_ErrorPaths" classname="" time="0.010">
+			<system-out><![CDATA[�[2m· create-route-table (scratch for error paths)�[0m
+    vpc_test.go:481: �[2m· create-route-table (scratch for error paths)�[0m
+�[2m· delete-route local=172.31.0.0/16 (must fail)�[0m
+    vpc_test.go:517: �[2m· delete-route local=172.31.0.0/16 (must fail)�[0m
+�[2mdelete_local_err�[0m=InvalidParameterValue
+    vpc_test.go:529: �[2mdelete_local_err�[0m=InvalidParameterValue
+�[2m· delete-route-table rtb-e42012569f92f7184 (main, must fail)�[0m
+    vpc_test.go:533: �[2m· delete-route-table rtb-e42012569f92f7184 (main, must fail)�[0m
+�[2mdelete_main_err�[0m=DependencyViolation
+    vpc_test.go:540: �[2mdelete_main_err�[0m=DependencyViolation]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8d_NATGateway" classname="" time="106.010">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8d — NAT Gateway E2E ━━�[0m
+�[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2mpub_subnet�[0m=subnet-cd62e4c1c3cb50c63 �[2msg�[0m=sg-764a7fa2a39577204
+    natgw_test.go:54: �[2mvpc�[0m=vpc-531252db3b4e1fc08 �[2mpub_subnet�[0m=subnet-cd62e4c1c3cb50c63 �[2msg�[0m=sg-764a7fa2a39577204
+�[2m· create-subnet 172.31.16.0/20 (private)�[0m
+    natgw_test.go:58: �[2m· create-subnet 172.31.16.0/20 (private)�[0m
+�[2mpriv_subnet�[0m=subnet-2360a40e28371e214
+    natgw_test.go:71: �[2mpriv_subnet�[0m=subnet-2360a40e28371e214
+�[2m· run-instances bastion (default subnet)�[0m
+    natgw_test.go:74: �[2m· run-instances bastion (default subnet)�[0m
+�[2m· run-instances private (private subnet)�[0m
+    natgw_test.go:98: �[2m· run-instances private (private subnet)�[0m
+    natgw_test.go:118: instance i-268402f0e28b5e477 reached state running
+    natgw_test.go:119: instance i-db7f3ed6e4cdaa5aa reached state running
+�[2mbastion�[0m=i-268402f0e28b5e477 �[2mbastion_ip�[0m=192.168.0.202 �[2mprivate�[0m=i-db7f3ed6e4cdaa5aa �[2mprivate_ip�[0m=172.31.16.4
+    natgw_test.go:128: �[2mbastion�[0m=i-268402f0e28b5e477 �[2mbastion_ip�[0m=192.168.0.202 �[2mprivate�[0m=i-db7f3ed6e4cdaa5aa �[2mprivate_ip�[0m=172.31.16.4
+�[2m· waiting for SSH handshake 192.168.0.202:22�[0m
+    natgw_test.go:133: �[2m· waiting for SSH handshake 192.168.0.202:22�[0m
+�[2m· scp keypair -> bastion:/tmp/key.pem�[0m
+    natgw_test.go:139: �[2m· scp keypair -> bastion:/tmp/key.pem�[0m
+�[2m· wait for private SSH via bastion�[0m
+    natgw_test.go:151: �[2m· wait for private SSH via bastion�[0m
+�[2m· baseline: private VM has no internet�[0m
+    natgw_test.go:172: �[2m· baseline: private VM has no internet�[0m
+�[2m· allocate-address (NAT EIP)�[0m
+    natgw_test.go:178: �[2m· allocate-address (NAT EIP)�[0m
+�[2meip�[0m=192.168.0.203 �[2malloc�[0m=eipalloc-2d81753157cd62a50
+    natgw_test.go:195: �[2meip�[0m=192.168.0.203 �[2malloc�[0m=eipalloc-2d81753157cd62a50
+�[2m· create-nat-gateway in subnet-cd62e4c1c3cb50c63�[0m
+    natgw_test.go:197: �[2m· create-nat-gateway in subnet-cd62e4c1c3cb50c63�[0m
+�[2mnat_gw�[0m=nat-812216dd5105180c5
+    natgw_test.go:216: �[2mnat_gw�[0m=nat-812216dd5105180c5
+    natgw_test.go:218: nat gateway nat-812216dd5105180c5 reached state available
+�[2m· create-route-table�[0m
+    natgw_test.go:223: �[2m· create-route-table�[0m
+�[2m· associate-route-table rtb-8d6a4abd5f666ec13 <- subnet-2360a40e28371e214�[0m
+    natgw_test.go:241: �[2m· associate-route-table rtb-8d6a4abd5f666ec13 <- subnet-2360a40e28371e214�[0m
+�[2m· create-route 0.0.0.0/0 -> nat-812216dd5105180c5�[0m
+    natgw_test.go:259: �[2m· create-route 0.0.0.0/0 -> nat-812216dd5105180c5�[0m
+�[2m· verify private VM reaches 8.8.8.8 via NAT GW�[0m
+    natgw_test.go:279: �[2m· verify private VM reaches 8.8.8.8 via NAT GW�[0m
+�[2m· delete-nat-gateway nat-812216dd5105180c5�[0m
+    natgw_test.go:293: �[2m· delete-nat-gateway nat-812216dd5105180c5�[0m
+    natgw_test.go:318: nat gateway nat-812216dd5105180c5 reached state deleted
+�[2m· verify private VM lost internet after NAT GW teardown�[0m
+    natgw_test.go:329: �[2m· verify private VM lost internet after NAT GW teardown�[0m]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase8e_SGToSGDatapath" classname="" time="80.940">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 8e — Security Group SG-to-SG Datapath (OVN) ━━�[0m
+�[2m· 8e-1 create sge-client + sge-target security groups�[0m
+    datapath_test.go:61: �[2m· 8e-1 create sge-client + sge-target security groups�[0m
+�[2mclient_sg�[0m=sg-56e73c764a649f7ff �[2mtarget_sg�[0m=sg-67ad5efaa8be83b79
+    datapath_test.go:83: �[2mclient_sg�[0m=sg-56e73c764a649f7ff �[2mtarget_sg�[0m=sg-67ad5efaa8be83b79
+�[2m· 8e-1 authorize target-sg ingress tcp/8080 from sg-56e73c764a649f7ff�[0m
+    datapath_test.go:91: �[2m· 8e-1 authorize target-sg ingress tcp/8080 from sg-56e73c764a649f7ff�[0m
+�[2m· 8e-2 launch client-vm + target-vm�[0m
+    datapath_test.go:109: �[2m· 8e-2 launch client-vm + target-vm�[0m
+    datapath_test.go:132: instance i-eae6263d9b9fda069 reached state running
+    datapath_test.go:133: instance i-9d9411a83549d34e3 reached state running
+�[2mclient_priv�[0m=172.31.0.5 �[2mclient_eni�[0m=eni-0f878da0143028915 �[2mtarget_priv�[0m=172.31.0.6 �[2mtarget_eni�[0m=eni-3bd355e19c7fdd45c
+    datapath_test.go:142: �[2mclient_priv�[0m=172.31.0.5 �[2mclient_eni�[0m=eni-0f878da0143028915 �[2mtarget_priv�[0m=172.31.0.6 �[2mtarget_eni�[0m=eni-3bd355e19c7fdd45c
+�[2m· 8e-3 ovn-nbctl port_group membership�[0m
+    datapath_test.go:151: �[2m· 8e-3 ovn-nbctl port_group membership�[0m
+�[2mclient_pg�[0m=sg_56e73c764a649f7ff �[2mtarget_pg�[0m=sg_67ad5efaa8be83b79
+    datapath_test.go:163: �[2mclient_pg�[0m=sg_56e73c764a649f7ff �[2mtarget_pg�[0m=sg_67ad5efaa8be83b79
+�[2m· 8e-3 ovn-sbctl address_set sg_56e73c764a649f7ff_ip4 contains 172.31.0.5�[0m
+    datapath_test.go:170: �[2m· 8e-3 ovn-sbctl address_set sg_56e73c764a649f7ff_ip4 contains 172.31.0.5�[0m
+�[2m· 8e-3 wait for client-vm SSH at 192.168.0.202:22�[0m
+    datapath_test.go:188: �[2m· 8e-3 wait for client-vm SSH at 192.168.0.202:22�[0m
+�[2m· 8e-4 allowed traffic client -> target:172.31.0.6:8080�[0m
+    datapath_test.go:199: �[2m· 8e-4 allowed traffic client -> target:172.31.0.6:8080�[0m
+�[2mstep4�[0m=allowed_traffic_ok
+    datapath_test.go:208: �[2mstep4�[0m=allowed_traffic_ok
+�[2m· 8e-5 denied traffic client -> target:22 (no SSH ingress)�[0m
+    datapath_test.go:213: �[2m· 8e-5 denied traffic client -> target:22 (no SSH ingress)�[0m
+�[2mstep5�[0m=denied_traffic_ok
+    datapath_test.go:217: �[2mstep5�[0m=denied_traffic_ok
+�[2m· 8e-6 revoke target-sg ingress, retest (sync RequestEvent contract)�[0m
+    datapath_test.go:222: �[2m· 8e-6 revoke target-sg ingress, retest (sync RequestEvent contract)�[0m
+�[2m· 8e-6 verify client -> target:8080 now blocked�[0m
+    datapath_test.go:240: �[2m· 8e-6 verify client -> target:8080 now blocked�[0m
+�[2mstep6�[0m=revoke_blocked_ok
+    datapath_test.go:244: �[2mstep6�[0m=revoke_blocked_ok
+�[2m· 8e-7 re-authorize target-sg ingress, verify traffic restored�[0m
+    datapath_test.go:250: �[2m· 8e-7 re-authorize target-sg ingress, verify traffic restored�[0m
+�[2mstep7�[0m=restore_ok
+    datapath_test.go:271: �[2mstep7�[0m=restore_ok
+    datapath_test.go:127: instance i-eae6263d9b9fda069 reached state terminated
+    datapath_test.go:127: instance i-9d9411a83549d34e3 reached state terminated]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase9_Teardown" classname="" time="8.300">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 9 — Teardown ━━�[0m
+�[2m· delete-snapshot snap-dde89b9ea2ef2e05a (custom AMI backing)�[0m
+    teardown_test.go:33: �[2m· delete-snapshot snap-dde89b9ea2ef2e05a (custom AMI backing)�[0m
+�[2mdeleted_snapshot�[0m=snap-dde89b9ea2ef2e05a
+    teardown_test.go:39: �[2mdeleted_snapshot�[0m=snap-dde89b9ea2ef2e05a
+�[2m· deregister-image ami-7dcf263a412dd14c6�[0m
+    teardown_test.go:51: �[2m· deregister-image ami-7dcf263a412dd14c6�[0m
+�[2mderegistered_ami�[0m=ami-7dcf263a412dd14c6
+    teardown_test.go:57: �[2mderegistered_ami�[0m=ami-7dcf263a412dd14c6
+�[2m· terminate-instances i-0c615e5dd74050f53�[0m
+    teardown_test.go:74: �[2m· terminate-instances i-0c615e5dd74050f53�[0m
+�[2mterminated_instance�[0m=i-0c615e5dd74050f53
+    teardown_test.go:81: �[2mterminated_instance�[0m=i-0c615e5dd74050f53
+�[2m· delete-key-pair test-key-1�[0m
+    teardown_test.go:94: �[2m· delete-key-pair test-key-1�[0m
+�[2mdeleted_keypair�[0m=test-key-1
+    teardown_test.go:100: �[2mdeleted_keypair�[0m=test-key-1]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase9a_VerifyTeardown" classname="" time="0.560">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 9a — Verify Teardown ━━�[0m
+�[2m· describe-instances i-0c615e5dd74050f53�[0m
+    teardown_test.go:123: �[2m· describe-instances i-0c615e5dd74050f53�[0m
+�[2minstance_state�[0m=terminated
+    teardown_test.go:135: �[2minstance_state�[0m=terminated
+�[2m· describe-images ami-7dcf263a412dd14c6�[0m
+    teardown_test.go:147: �[2m· describe-images ami-7dcf263a412dd14c6�[0m
+�[2mami_reaped�[0m=ami-7dcf263a412dd14c6
+    teardown_test.go:159: �[2mami_reaped�[0m=ami-7dcf263a412dd14c6
+�[2m· describe-key-pairs (must not contain test-key-1)�[0m
+    teardown_test.go:167: �[2m· describe-key-pairs (must not contain test-key-1)�[0m
+�[2mkeypairs_listed�[0m=1
+    teardown_test.go:176: �[2mkeypairs_listed�[0m=1]]></system-out>
+		</testcase>
+		<testcase name="TestSingleNode/Phase9b_FinalClusterStats" classname="" time="3.070">
+			<system-out><![CDATA[
+�[1m�[36m━━ Phase 9b — Final Cluster Stats ━━�[0m
+�[2mspx_get_vms_bytes�[0m=118
+    teardown_test.go:208: �[2mspx_get_vms_bytes�[0m=118]]></system-out>
+		</testcase>
+	</testsuite>
+</testsuites>

.github/actions/junit-report/.gitignore

@@ -0,0 +1 @@
+__pycache__/

.github/actions/junit-report/README.md

@@ -0,0 +1,46 @@
+# junit-report
+
+Composite action that renders a glob of JUnit XML files into the GitHub
+Actions job summary and emits one `::error` annotation per failed test case.
+
+## Usage
+
+```yaml
+- name: Publish test summary
+  if: always()
+  uses: ./spinifex/.github/actions/junit-report   # or your own checkout path
+  with:
+    junit-glob: ${{ env.ARTIFACT_DIR }}/junit-*.xml
+    title: "Go E2E results"
+    # fail-on-empty: "true"   # opt-in: fail the step if the glob matches nothing
+```
+
+Outputs:
+
+- `failures` — total `<failure>` + `<error>` counts across all matched files.
+- `total`    — total `<testcase>` count across all matched files.
+
+The action expects standard JUnit XML (e.g. emitted by
+`gotestsum --junitfile`, `go-junit-report`, or pytest's `--junitxml`). Both
+single `<testsuite>` and wrapping `<testsuites>` roots are accepted.
+
+## Companion convention: `::group::` for noisy steps
+
+Long-running shell phases (tofu apply, package installs, tar-over-ssh) should
+wrap their output in `::group::name` / `::endgroup::` markers so the Actions
+UI collapses them by default. Pattern used in the e2e-go / e2e-ddil workflows:
+
+```bash
+run() {
+  local label="$1"; shift
+  echo "::group::${label}"
+  "$@" 2>&1 | tee -a "$TOFU_LOG"
+  local rc=${PIPESTATUS[0]}
+  echo "::endgroup::"
+  return "$rc"
+}
+run "tofu apply" tofu apply -var-file=... -auto-approve -input=false
+```
+
+Combined: groups make the live UI readable; this action makes the run header
+land on the failures without scrolling the log.

.github/actions/junit-report/action.yml

@@ -0,0 +1,41 @@
+name: JUnit Report
+description: >-
+  Render JUnit XML files into the GitHub Actions job summary and emit
+  inline ::error annotations for each failed test case. Designed to be
+  called once per job after the test step (with `if: always()`) so summary
+  + annotations appear on both passing and failing runs.
+
+inputs:
+  junit-glob:
+    description: Shell glob matching JUnit XML files to render.
+    required: true
+  title:
+    description: Heading rendered at the top of the job summary section.
+    required: false
+    default: "Test results"
+  fail-on-empty:
+    description: >-
+      When "true", the action fails if the glob matches no files. Useful
+      for catching silently-skipped test phases.
+    required: false
+    default: "false"
+
+outputs:
+  failures:
+    description: Total failed + errored test cases across all matched files.
+    value: ${{ steps.render.outputs.failures }}
+  total:
+    description: Total test cases across all matched files.
+    value: ${{ steps.render.outputs.total }}
+
+runs:
+  using: composite
+  steps:
+    - id: render
+      shell: bash
+      env:
+        JUNIT_GLOB: ${{ inputs.junit-glob }}
+        JUNIT_TITLE: ${{ inputs.title }}
+        JUNIT_FAIL_ON_EMPTY: ${{ inputs.fail-on-empty }}
+      run: |
+        python3 "${{ github.action_path }}/render.py"

.github/actions/junit-report/render.py

@@ -0,0 +1,281 @@
+#!/usr/bin/env python3
+"""Render JUnit XML files into GitHub Actions job summary + annotations.
+
+Inputs (env):
+    JUNIT_GLOB           Glob matching JUnit XML files.
+    JUNIT_TITLE          Heading text for the summary section.
+    JUNIT_FAIL_ON_EMPTY  "true" to exit non-zero when glob matches nothing.
+
+GitHub-Actions outputs (via $GITHUB_OUTPUT):
+    failures             Sum of <failures> + <errors> across all suites.
+    total                Sum of <tests> across all suites.
+
+Side effects:
+    Appends a markdown section to $GITHUB_STEP_SUMMARY with a per-file
+    totals table and a collapsible details block per failed test.
+    Emits one `::error` workflow command per failed/errored test case so
+    they surface as annotations in the run UI.
+"""
+
+from __future__ import annotations
+
+import glob
+import os
+import re
+import sys
+import xml.etree.ElementTree as ET
+from dataclasses import dataclass
+
+
+# Strip ANSI SGR escapes from skipped/failure CDATA so the rendered reason is
+# plain text (CDATA preserves the colour codes Go emits via harness.Phase).
+_ANSI_RE = re.compile(r"\x1b\[[0-9;]*m")
+
+
+def _strip_ansi(s: str) -> str:
+    return _ANSI_RE.sub("", s)
+
+
+# Last `<file>.go:<line>: <reason>` line in a CDATA block is the t.Skipf /
+# t.Fatal call site — that's the actionable signal. Phase banners and probe
+# logs above it are noise for the summary view.
+_SKIP_REASON_RE = re.compile(r"([A-Za-z0-9_.\-/]+\.go:\d+:\s*.+)$")
+
+
+def _extract_skip_reason(cdata: str) -> str:
+    cdata = _strip_ansi(cdata).strip()
+    if not cdata:
+        return ""
+    for line in reversed(cdata.splitlines()):
+        line = line.strip()
+        m = _SKIP_REASON_RE.search(line)
+        if m:
+            return m.group(1).strip()
+    return cdata.splitlines()[-1].strip()
+
+
+# Workflow commands carry their payload on one line; literal newlines must
+# be percent-escaped per GitHub's workflow-command spec.
+def _esc(s: str) -> str:
+    return (
+        s.replace("%", "%25")
+        .replace("\r", "%0D")
+        .replace("\n", "%0A")
+    )
+
+
+@dataclass
+class Case:
+    suite: str
+    classname: str
+    name: str
+    time: float
+    status: str  # "pass" | "fail" | "error" | "skip"
+    message: str
+    detail: str
+
+
+@dataclass
+class FileTotals:
+    path: str
+    tests: int
+    failures: int
+    errors: int
+    skipped: int
+    time: float
+    cases: list[Case]
+
+
+def _parse_file(path: str) -> FileTotals:
+    root = ET.parse(path).getroot()
+    # JUnit XML has two shapes: <testsuite> as root, or <testsuites> wrapping
+    # multiple <testsuite> children. Iterating with .iter() handles both.
+    suites = list(root.iter("testsuite"))
+    tests = sum(int(s.get("tests", 0) or 0) for s in suites)
+    failures = sum(int(s.get("failures", 0) or 0) for s in suites)
+    errors = sum(int(s.get("errors", 0) or 0) for s in suites)
+    skipped = sum(int(s.get("skipped", 0) or 0) for s in suites)
+    time = sum(float(s.get("time", 0) or 0) for s in suites)
+
+    cases: list[Case] = []
+    for s in suites:
+        suite_name = s.get("name", "")
+        for tc in s.findall("testcase"):
+            status = "pass"
+            message = ""
+            detail = ""
+            for tag in ("failure", "error"):
+                node = tc.find(tag)
+                if node is not None:
+                    status = "fail" if tag == "failure" else "error"
+                    message = (node.get("message") or "").strip()
+                    detail = (node.text or "").strip()
+                    break
+            skip_node = tc.find("skipped")
+            if skip_node is not None and status == "pass":
+                status = "skip"
+                # Skipped reason: prefer the CDATA call-site line, fall back
+                # to the @message attribute if CDATA is empty.
+                cdata = (skip_node.text or "").strip()
+                message = _extract_skip_reason(cdata) or (
+                    skip_node.get("message") or ""
+                ).strip()
+            cases.append(
+                Case(
+                    suite=suite_name,
+                    classname=tc.get("classname", ""),
+                    name=tc.get("name", ""),
+                    time=float(tc.get("time", 0) or 0),
+                    status=status,
+                    message=message,
+                    detail=detail,
+                )
+            )
+
+    # Drop container test cases: any case whose name is a strict prefix of
+    # another case's name (with "/" separator) is a parent t.Run wrapper.
+    # Go reports the parent's status as a rollup of its children, so counting
+    # both double-tallies the same outcome (e.g. TestSingleNode failing
+    # because Phase8b_VPCSubnetE2E failed would show as 2 failures, not 1).
+    names = {c.name for c in cases}
+    prefixes = {n.rsplit("/", 1)[0] for n in names if "/" in n}
+    # A name is a container if some other name starts with "<name>/".
+    # Use the prefixes set rather than O(n^2) scanning.
+    def _is_container(name: str) -> bool:
+        return name in prefixes
+
+    leaf_cases = [c for c in cases if not _is_container(c.name)]
+
+    # Recompute totals from leaf cases so the per-file counts in the summary
+    # table match the de-duplicated failure list below it.
+    leaf_tests = len(leaf_cases)
+    leaf_failures = sum(1 for c in leaf_cases if c.status == "fail")
+    leaf_errors = sum(1 for c in leaf_cases if c.status == "error")
+    leaf_skipped = sum(1 for c in leaf_cases if c.status == "skip")
+
+    return FileTotals(
+        path=path,
+        tests=leaf_tests,
+        failures=leaf_failures,
+        errors=leaf_errors,
+        skipped=leaf_skipped,
+        time=time,
+        cases=leaf_cases,
+    )
+
+
+def _file_label(path: str) -> str:
+    base = os.path.basename(path)
+    if base.endswith(".xml"):
+        base = base[:-4]
+    if base.startswith("junit-"):
+        base = base[len("junit-"):]
+    return base or path
+
+
+def main() -> int:
+    pattern = os.environ.get("JUNIT_GLOB", "").strip()
+    title = os.environ.get("JUNIT_TITLE", "Test results").strip() or "Test results"
+    fail_on_empty = os.environ.get("JUNIT_FAIL_ON_EMPTY", "").strip().lower() == "true"
+
+    if not pattern:
+        print("::error::junit-report: junit-glob input is empty", file=sys.stderr)
+        return 1
+
+    paths = sorted(glob.glob(pattern))
+    summary_path = os.environ.get("GITHUB_STEP_SUMMARY", "/dev/null")
+    output_path = os.environ.get("GITHUB_OUTPUT", "/dev/null")
+
+    with open(summary_path, "a", encoding="utf-8") as summary, \
+            open(output_path, "a", encoding="utf-8") as outputs:
+
+        summary.write(f"## {title}\n\n")
+
+        if not paths:
+            summary.write(f"_No JUnit files matched `{pattern}`._\n\n")
+            outputs.write("failures=0\n")
+            outputs.write("total=0\n")
+            if fail_on_empty:
+                print(f"::error::junit-report: no files matched {pattern}", file=sys.stderr)
+                return 1
+            return 0
+
+        totals_by_file = [_parse_file(p) for p in paths]
+        grand_total = sum(f.tests for f in totals_by_file)
+        grand_fail = sum(f.failures + f.errors for f in totals_by_file)
+        grand_skip = sum(f.skipped for f in totals_by_file)
+        grand_pass = grand_total - grand_fail - grand_skip
+        grand_time = sum(f.time for f in totals_by_file)
+
+        if grand_fail == 0:
+            verdict = f"✅ **pass** — {grand_pass} / {grand_total}"
+        else:
+            verdict = f"❌ **{grand_fail} failed** — {grand_pass} / {grand_total} pass"
+        skip_part = f", {grand_skip} skipped" if grand_skip else ""
+        summary.write(f"{verdict}{skip_part} &nbsp; · &nbsp; {grand_time:.1f}s\n\n")
+
+        # Compact table — single status column, shorter headers. Drops
+        # separate err column (errors fold into fail count) so narrow
+        # screens don't wrap.
+        summary.write("| | Suite | Pass | Fail | Skip | Time |\n")
+        summary.write("|:-:|---|---:|---:|---:|---:|\n")
+        for f in totals_by_file:
+            fails = f.failures + f.errors
+            passed = f.tests - fails - f.skipped
+            icon = "✅" if fails == 0 else "❌"
+            summary.write(
+                f"| {icon} | `{_file_label(f.path)}` | {passed} | {fails} | "
+                f"{f.skipped} | {f.time:.1f}s |\n"
+            )
+        summary.write("\n")
+
+        failed_cases = [
+            c for f in totals_by_file for c in f.cases if c.status in ("fail", "error")
+        ]
+        if failed_cases:
+            summary.write("---\n\n**Failures**\n\n")
+            for c in failed_cases:
+                header = f"{c.classname}.{c.name}" if c.classname else c.name
+                # One line per failure: name + truncated message. Full
+                # detail lives in the test artifact log; deep nesting and
+                # long messages wrap badly on narrow screens.
+                msg_short = (c.message or "").splitlines()[0] if c.message else ""
+                if len(msg_short) > 120:
+                    msg_short = msg_short[:117] + "…"
+                line = f"- ❌ `{header}`"
+                if msg_short:
+                    line += f" — {msg_short}"
+                summary.write(line + "\n")
+
+                # Workflow annotation — surfaces in run header + Files-changed view.
+                # classname for Go tests is the package path; map to a file when we can.
+                file_hint = c.classname.replace(".", "/")
+                msg = c.message or f"{c.status}: {header}"
+                print(f"::error file={file_hint},title={header}::{_esc(msg)}")
+            summary.write("\n")
+
+        skipped_cases = [
+            c for f in totals_by_file for c in f.cases if c.status == "skip"
+        ]
+        if skipped_cases:
+            summary.write("---\n\n<details><summary><b>Skipped</b> ")
+            summary.write(f"({len(skipped_cases)})</summary>\n\n")
+            for c in skipped_cases:
+                header = f"{c.classname}.{c.name}" if c.classname else c.name
+                reason = (c.message or "").splitlines()[0] if c.message else ""
+                if len(reason) > 160:
+                    reason = reason[:157] + "…"
+                line = f"- ⏭️ `{header}`"
+                if reason:
+                    line += f" — {reason}"
+                summary.write(line + "\n")
+            summary.write("\n</details>\n\n")
+
+        outputs.write(f"failures={grand_fail}\n")
+        outputs.write(f"total={grand_total}\n")
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

.github/scripts/ci-fmt.sh

@@ -0,0 +1,63 @@
+# shellcheck shell=bash
+# Source from a workflow `run:` block:
+#   source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+#
+# Provides colored banners + ::group::/::endgroup:: wrappers. ANSI escapes
+# render in the GitHub Actions live log; the same lines also land in any
+# artifact log (tee'd output) without breaking parsers because they only
+# affect terminal display.
+#
+# Functions kept short and unindented on purpose — wide nested output wraps
+# badly on narrow terminals.
+
+if [[ -n "${GITHUB_ACTIONS:-}" ]] || [[ -t 1 ]]; then
+  _R=$'\033[0m'
+  _B=$'\033[1m'    # bold
+  _D=$'\033[2m'    # dim
+  _RED=$'\033[31m'
+  _GRN=$'\033[32m'
+  _YEL=$'\033[33m'
+  _BLU=$'\033[34m'
+  _CYA=$'\033[36m'
+else
+  _R="" _B="" _D="" _RED="" _GRN="" _YEL="" _BLU="" _CYA=""
+fi
+
+# Single-line rule using box-drawing chars — narrow enough that GitHub's
+# timestamp prefix + this rule still fit on an 80-col terminal.
+# banner LABEL — bold cyan title with surrounding rule. Used at the start
+# of an inherently multi-line phase (test run, artifact pull) so the live
+# log has an obvious visual marker.
+banner() {
+  printf '\n%s%s━━ %s ━━%s\n' "$_B" "$_CYA" "$*" "$_R"
+}
+
+# Status indicators. Single char prefix only — keeps width on narrow logs.
+ok()   { printf '%s✓%s %s\n' "$_GRN" "$_R" "$*"; }
+bad()  { printf '%s✗%s %s\n' "$_RED" "$_R" "$*"; }
+warn() { printf '%s!%s %s\n' "$_YEL" "$_R" "$*"; }
+note() { printf '%s·%s %s\n' "$_D"   "$_R" "$*"; }
+
+# group_open LABEL / group_close — thin wrapper around GitHub's collapse
+# markers. No extra body printed; banner() is the visible inside-group
+# heading when callers want one.
+group_open()  { echo "::group::$*"; }
+group_close() { echo "::endgroup::"; }
+
+# quiet_run LABEL LOG_FILE -- CMD ARGS …
+# Run CMD redirected to LOG_FILE (append). Print one colored ok/✗ line.
+# On failure, dump the last 80 lines of LOG_FILE inside a collapsed group
+# so the failing tail lands in the UI without surfacing the whole apply.
+quiet_run() {
+  local label="$1" log="$2"; shift 2
+  if "$@" >> "$log" 2>&1; then
+    ok "$label"
+  else
+    local rc=$?
+    bad "$label (rc=$rc) — last 80 lines of $(basename "$log"):"
+    group_open "tail $(basename "$log")"
+    tail -n 80 "$log"
+    group_close
+    return "$rc"
+  fi
+}

.github/workflows/e2e-baremetal.yml

@@ -81,33 +81,23 @@ jobs:
 
       - name: Cache VM images
         run: |
-          UBUNTU_IMG="${BM_ISO_CACHE}/ubuntu-24.04-cloudimg-amd64.img"
+          UBUNTU_IMG="${BM_ISO_CACHE}/ubuntu-26.04-cloudimg-amd64.img"
           if [ ! -f "$UBUNTU_IMG" ]; then
-            echo "Downloading Ubuntu 24.04 cloud image..."
+            echo "Downloading Ubuntu 26.04 cloud image..."
             sudo curl -fsSL -o "$UBUNTU_IMG" \
-              "https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img"
+              "https://cloud-images.ubuntu.com/resolute/current/resolute-server-cloudimg-amd64.img"
             echo "  Downloaded $(du -h "$UBUNTU_IMG" | cut -f1)"
           else
-            echo "Ubuntu 24.04 cloud image cached ($(du -h "$UBUNTU_IMG" | cut -f1))"
+            echo "Ubuntu 26.04 cloud image cached ($(du -h "$UBUNTU_IMG" | cut -f1))"
           fi
 
-          LB_IMG="${BM_ISO_CACHE}/lb-alpine-3.21.6-x86_64.raw"
-          if [ ! -f "$LB_IMG" ]; then
-            echo "Downloading lb-alpine image..."
-            sudo curl -fsSL -o "$LB_IMG" \
-              "https://iso.mulgadc.com/system-ami/lb-alpine-3.21.6-x86_64.raw"
-            echo "  Downloaded $(du -h "$LB_IMG" | cut -f1)"
-          else
-            echo "lb-alpine image cached ($(du -h "$LB_IMG" | cut -f1))"
-          fi
 
       - name: Run Bare-Metal E2E
         env:
           BM_CI_PASSWORD: ${{ secrets.BM_CI_PASSWORD }}
           BM_SSH_KEY: ${{ runner.temp }}/bm-ci
           BM_EPDU_ENV: ${{ runner.temp }}/epdu.env
-          BM_UBUNTU_IMAGE: ${{ env.BM_ISO_CACHE }}/ubuntu-24.04-cloudimg-amd64.img
-          BM_LB_ALPINE_IMAGE: ${{ env.BM_ISO_CACHE }}/lb-alpine-3.21.6-x86_64.raw
+          BM_UBUNTU_IMAGE: ${{ env.BM_ISO_CACHE }}/ubuntu-26.04-cloudimg-amd64.img
           SPX_EXTERNAL_POOL: ${{ env.SPINIFEX_CI_POOL }}
         run: |
           sudo -E mulga/scripts/baremetal/run-baremetal-e2e.sh \

.github/workflows/e2e-ddil.yml

@@ -87,12 +87,16 @@ jobs:
       - name: Provision VMs
         working-directory: mulga/scripts/tofu-cluster
         run: |
+          set -euo pipefail
+          # shellcheck source=/dev/null
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           export TF_VAR_git_branch="${GIT_BRANCH}"
-          tofu init -input=false >> "$TOFU_LOG" 2>&1
-          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
-          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
-          tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          quiet_run "tofu init"                "$TOFU_LOG" tofu init -input=false
+          quiet_run "tofu workspace select"    "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy (pre-clean)" "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false || true
+          quiet_run "scrub-env (pre-clean)"    "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
+          quiet_run "tofu apply"               "$TOFU_LOG" tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          ok "VMs provisioned"
 
       - name: Bootstrap via Binary Install
         working-directory: mulga/scripts/tofu-cluster
@@ -147,8 +151,13 @@ jobs:
 
           mkdir -p "${RUNNER_TEMP}/ddil-results"
           cd "${GITHUB_WORKSPACE}/spinifex"
-          # gotestsum: human-readable PASS/FAIL/SKIP lines in Actions stdout
-          # while still emitting the machine-readable jsonfile artifact.
+          # gotestsum: human-readable PASS/FAIL/SKIP lines in Actions stdout,
+          # plus machine-readable jsonfile + JUnit XML artifacts. The JUnit
+          # file feeds the junit-report composite action below so failures
+          # become summary table rows + workflow annotations.
+          # shellcheck source=/dev/null
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          banner "DDIL scenarios"
           DDIL_NODES="${NODE_IPS}" \
           DDIL_SSH_USER="${SSH_USER}" \
           DDIL_SSH_KEY="${SSH_KEY}" \
@@ -158,8 +167,19 @@ jobs:
           AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" \
           AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" \
             gotestsum --jsonfile "${RUNNER_TEMP}/ddil-results/results.json" \
+                      --junitfile "${RUNNER_TEMP}/ddil-results/junit-ddil.xml" \
                       --format testname \
                       -- -tags=e2e -timeout 45m ./tests/e2e/ddil/scenarios/...
+          rc=$?
+          if [[ $rc -eq 0 ]]; then ok "all scenarios passed"; else bad "scenarios failed (rc=$rc)"; fi
+          exit $rc
+
+      - name: Publish test summary
+        if: always()
+        uses: ./spinifex/.github/actions/junit-report
+        with:
+          junit-glob: ${{ runner.temp }}/ddil-results/junit-*.xml
+          title: "DDIL scenario results"
 
       - name: Collect Cluster Logs
         if: failure()
@@ -177,13 +197,15 @@ jobs:
           LOG_DIR="${RUNNER_TEMP}/ddil-results/cluster-logs"
           mkdir -p "${LOG_DIR}"
 
+          # shellcheck source=/dev/null
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
           mapfile -t NODE_IPS < <(echo "$INVENTORY" | jq -r '.nodes[].wan_ip')
           for ip in "${NODE_IPS[@]}"; do
-            echo "=== collecting ${ip} ==="
             SSH="ssh -i ${SSH_KEY} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${SSH_USER}@${ip}"
             ${SSH} 'sudo journalctl -u "spinifex-*" --no-pager -n 5000' > "${LOG_DIR}/${ip}-spinifex.log" 2>&1
             ${SSH} 'ps auxw | grep -E "qemu-system|spinifex" | grep -v grep' > "${LOG_DIR}/${ip}-procs.txt" 2>&1
             ${SSH} 'ls -la /var/lib/spinifex 2>/dev/null; ls -la /var/log/spinifex 2>/dev/null' > "${LOG_DIR}/${ip}-state.txt" 2>&1
+            ok "$ip: $(wc -l < "${LOG_DIR}/${ip}-spinifex.log") journal lines, $(wc -l < "${LOG_DIR}/${ip}-procs.txt") procs"
           done
 
       - name: Upload DDIL Results
@@ -198,6 +220,10 @@ jobs:
         if: always()
         working-directory: mulga/scripts/tofu-cluster
         run: |
-          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
-          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
-          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+          set -euo pipefail
+          # shellcheck source=/dev/null
+          source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
+          quiet_run "tofu workspace select" "$TOFU_LOG" tofu workspace select -or-create "${SPINIFEX_CI_ENV}"
+          quiet_run "tofu destroy"          "$TOFU_LOG" tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false
+          quiet_run "scrub-env"             "$TOFU_LOG" ./scrub-env.sh "${SPINIFEX_CI_ENV}" || true
+          ok "teardown complete"

.github/workflows/release.yml

@@ -32,7 +32,7 @@ jobs:
         uses: docker/setup-buildx-action@v4
 
       - name: Set up Go
-        uses: actions/setup-go@v5
+        uses: actions/setup-go@v6
         with:
           go-version: '1.26.3'
 
@@ -326,12 +326,13 @@ jobs:
         env:
           AWS_ACCESS_KEY_ID: ${{ secrets.CF_R2_ACCESS_KEY_ID }}
           AWS_SECRET_ACCESS_KEY: ${{ secrets.CF_R2_SECRET_ACCESS_KEY }}
+          CF_R2_ENDPOINT_URL: ${{ secrets.CF_R2_ENDPOINT_URL }}
           VERSION: ${{ github.ref_name }}
         run: |
-          aws --endpoint-url=https://e80e329f7dbcf75b80e019c1dce640f2.r2.cloudflarestorage.com \
+          aws --endpoint-url="${CF_R2_ENDPOINT_URL}" \
             s3 cp "mulga/scripts/iso-builder/dist/spinifex-${VERSION}.iso" \
                   "s3://spinifex-iso/spinifex-${VERSION}.iso"
-          aws --endpoint-url=https://e80e329f7dbcf75b80e019c1dce640f2.r2.cloudflarestorage.com \
+          aws --endpoint-url="${CF_R2_ENDPOINT_URL}" \
             s3 cp "mulga/scripts/iso-builder/dist/spinifex-${VERSION}.iso.sha256" \
                   "s3://spinifex-iso/spinifex-${VERSION}.iso.sha256"
 

.github/workflows/spinifex.yml

@@ -61,6 +61,9 @@ jobs:
       - name: Run Tests with Coverage
         run: make -C spinifex test-cover
 
+      - name: Run Action Tests
+        run: make -C spinifex test-actions
+
       - name: Check Diff Coverage
         run: cd spinifex && scripts/diff-coverage.sh coverage.out --base HEAD~1
 
@@ -136,7 +139,7 @@ jobs:
       - name: Lint
         uses: golangci/golangci-lint-action@v9
         with:
-          version: v2.11.3
+          version: v2.12.2
           working-directory: spinifex
 
       - name: Govulncheck

Makefile

@@ -106,9 +106,23 @@ go_run:
 # Preflight — runs the same checks as GitHub Actions (lint + vuln + tests).
 # Use this before committing to catch CI failures locally.
 preflight:
-	@$(MAKE) --no-print-directory QUIET=1 lint govulncheck test-cover diff-coverage test-race
+	@$(MAKE) --no-print-directory QUIET=1 manifest-check lint govulncheck test-cover diff-coverage test-race test-harness
 	@echo -e "\n ✅ Preflight passed — safe to commit."
 
+# E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
+# default `go test ./spinifex/...`. Runs with mocked AWS clients — no
+# infrastructure required, safe to run in CI without a cluster.
+test-harness:
+	@echo -e "\n....Running e2e harness unit tests...."
+	$(_Q)LOG_IGNORE=1 go test -tags=e2e -timeout 60s ./tests/e2e/harness/... $(_RACEQ)
+
+# Validate docs/service-interfaces.yaml. Schema check + cross-reference
+# of services/suites/fixtures + on-disk path existence. Subject content
+# vs source is enforced separately in Bead 5 drift lint.
+manifest-check:
+	@echo -e "\n....Checking service-interfaces.yaml...."
+	@go run ./tests/e2e/manifest-check/cmd/manifest-check -repo-root . -manifest docs/service-interfaces.yaml
+
 # Run unit tests
 test:
 	@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
@@ -126,6 +140,12 @@ test-race:
 	@echo -e "\n....Running tests with race detector for $(GO_PROJECT_NAME)...."
 	$(_Q)LOG_IGNORE=1 go test -race -timeout 300s ./spinifex/... $(_RACEQ)
 
+# Unit tests for in-repo GitHub Actions (e.g. .github/actions/e2e-analyze).
+# Kept out of `test-cover` so coverage % isn't diluted by CI-only tooling.
+test-actions:
+	@echo -e "\n....Running action tests...."
+	LOG_IGNORE=1 go test -timeout 60s ./.github/actions/...
+
 # Check that new/changed code meets coverage threshold (runs tests first)
 diff-coverage: test-cover
 	@QUIET=$(QUIET) scripts/diff-coverage.sh $(COVERPROFILE)
@@ -170,6 +190,7 @@ install-system:
 		-o Dpkg::Options::="--force-confdef" \
 		-o Dpkg::Options::="--force-confold" \
 		nbdkit nbdkit-plugin-dev pkg-config $(QEMU_PACKAGES) qemu-utils qemu-kvm \
+		ovmf qemu-efi-aarch64 \
 		libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
 		iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
 		ovn-central ovn-host openvswitch-switch dhcpcd-base
@@ -275,7 +296,7 @@ ansible-dev-reset:
 ansible-dev-deploy:
 	cd scripts/ansible && ansible-playbook playbooks/dev-deploy.yml
 
-.PHONY: build build-ui build-installer build-lb-agent build-system-image build-microvm-image install-microvm go_build go_run preflight test test-cover test-race diff-coverage bench run \
+.PHONY: build build-ui build-installer build-lb-agent build-system-image build-microvm-image install-microvm go_build go_run preflight test test-cover test-race test-actions test-harness manifest-check diff-coverage bench run \
 	deploy reinstall clean \
 	install-system install-go install-aws quickinstall \
 	lint fix govulncheck \

cmd/installer/install/install.go

@@ -27,6 +27,7 @@ import (
 	"os"
 	"os/exec"
 	"path/filepath"
+	"slices"
 	"strings"
 	"syscall"
 	"time"
@@ -850,8 +851,8 @@ func bindChrootMounts() error {
 // unbindChrootMounts unmounts the virtual filesystems in reverse order.
 // Errors are logged but not returned — this is best-effort cleanup.
 func unbindChrootMounts() {
-	for i := len(chrootMountPaths) - 1; i >= 0; i-- {
-		_ = run("umount", filepath.Join(mountRoot, chrootMountPaths[i]))
+	for _, v := range slices.Backward(chrootMountPaths) {
+		_ = run("umount", filepath.Join(mountRoot, v))
 	}
 }
 

cmd/spinifex/cmd/admin.go

@@ -17,6 +17,7 @@ along with this program.  If not, see <https://www.gnu.org/licenses/>.
 package cmd
 
 import (
+	"bufio"
 	"bytes"
 	"context"
 	"crypto/tls"
@@ -48,6 +49,7 @@ import (
 	"github.com/mulgadc/spinifex/spinifex/gpu"
 	handlers_ec2_vpc "github.com/mulgadc/spinifex/spinifex/handlers/ec2/vpc"
 	handlers_iam "github.com/mulgadc/spinifex/spinifex/handlers/iam"
+	"github.com/mulgadc/spinifex/spinifex/objectstore"
 	"github.com/mulgadc/spinifex/spinifex/utils"
 	"github.com/mulgadc/viperblock/viperblock"
 	"github.com/mulgadc/viperblock/viperblock/backends/s3"
@@ -84,6 +86,14 @@ var supportedPlatforms = map[string]bool{
 	"Windows":    true,
 }
 
+// Mirrors the gateway RegisterImage allowlist so admin imports can't write an
+// AMI with a boot mode that RegisterImage would reject.
+var supportedBootModes = map[string]bool{
+	"bios":           true,
+	"uefi":           true,
+	"uefi-preferred": true,
+}
+
 var adminCmd = &cobra.Command{
 	Use:   "admin",
 	Short: "Administrative commands for Spinifex platform management",
@@ -143,6 +153,19 @@ var imagesListCmd = &cobra.Command{
 	Run:   runimagesListCmd,
 }
 
+var imagesRemoveCmd = &cobra.Command{
+	Use:   "remove",
+	Short: "Remove an admin-imported system AMI",
+	Long: `Delete an AMI imported via 'spx admin images import', including its
+backing block storage and snapshot artefacts. Only operates on AMIs with a
+non-account owner (e.g. "system"); account-owned AMIs must be removed via
+'aws ec2 deregister-image' followed by 'aws ec2 delete-snapshot'.
+
+Refuses to delete an AMI that has dependent volumes or copied snapshots/AMIs
+unless --force is passed. Prompts for confirmation unless --yes is passed.`,
+	Run: runimagesRemoveCmd,
+}
+
 var accountCmd = &cobra.Command{
 	Use:   "account",
 	Short: "Manage Spinifex accounts",
@@ -225,6 +248,7 @@ func init() {
 	adminCmd.AddCommand(imagesCmd)
 	imagesCmd.AddCommand(imagesImportCmd)
 	imagesCmd.AddCommand(imagesListCmd)
+	imagesCmd.AddCommand(imagesRemoveCmd)
 
 	adminCmd.AddCommand(accountCmd)
 	accountCmd.AddCommand(accountCreateCmd)
@@ -304,9 +328,17 @@ func init() {
 	imagesImportCmd.Flags().String("version", "", "Specified distro version (e.g 12)")
 	imagesImportCmd.Flags().String("arch", "", "Specified distro arch (e.g aarch64, arm64, x86_64)")
 	imagesImportCmd.Flags().String("platform", "Linux/UNIX", "Specified platform (e.g Linux/UNIX, Windows)")
+	imagesImportCmd.Flags().String("boot-mode", "", "Boot mode for the imported AMI (bios|uefi|uefi-preferred). Required with --file. Overrides the catalog value when used with --name.")
 	imagesImportCmd.Flags().StringSlice("tag", nil, "Tag to apply to the imported AMI as key=value (repeatable; e.g. --tag spinifex:managed-by=elbv2)")
 	imagesImportCmd.Flags().Bool("force", false, "Force command execution (overwrites existing files)")
 	imagesImportCmd.Flags().Bool("skip-verify", false, "Skip catalog-image checksum verification (INSECURE; operator assumes integrity responsibility)")
+
+	imagesRemoveCmd.Flags().String("image-id", "", "AMI ID to remove (required)")
+	imagesRemoveCmd.Flags().Bool("force", false, "Bypass dependency, ownership and config-corrupt checks (salvage mode)")
+	imagesRemoveCmd.Flags().Bool("yes", false, "Skip interactive confirmation prompt")
+	if err := imagesRemoveCmd.MarkFlagRequired("image-id"); err != nil {
+		panic(err)
+	}
 }
 
 func runimagesImportCmd(cmd *cobra.Command, args []string) {
@@ -376,6 +408,22 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 		}
 	}
 
+	// --file imports have no catalog metadata to inherit from, so the operator
+	// must declare the boot mode explicitly — guessing would silently produce
+	// a BIOS AMI from a UEFI-only image (or vice versa) and fail at launch.
+	// --name imports inherit from the catalog; the flag overrides when set.
+	bootModeFlag, _ := cmd.Flags().GetString("boot-mode")
+	if bootModeFlag != "" {
+		if !supportedBootModes[bootModeFlag] {
+			fmt.Fprintf(os.Stderr, "Unsupported --boot-mode %q (expected bios|uefi|uefi-preferred)\n", bootModeFlag)
+			os.Exit(1)
+		}
+		image.BootMode = bootModeFlag
+	} else if imageName == "" {
+		fmt.Fprintf(os.Stderr, "--boot-mode is required when importing via --file (expected bios|uefi|uefi-preferred)\n")
+		os.Exit(1)
+	}
+
 	// --tag k=v (repeatable) merges user-supplied tags into the AMI. Overrides
 	// catalog tags on key collision so operators can re-tag a known image.
 	tagFlags, _ := cmd.Flags().GetStringSlice("tag")
@@ -496,6 +544,9 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 	manifest.AMIMetadata.Virtualization = "hvm"
 	manifest.AMIMetadata.ImageOwnerAlias = "system"
 	manifest.AMIMetadata.VolumeSizeGiB = utils.SafeInt64ToUint64(imageStat.Size() / 1024 / 1024 / 1024)
+	manifest.AMIMetadata.BootMode = image.BootMode
+	manifest.AMIMetadata.Distro = image.Distro
+	manifest.AMIMetadata.DistroFamily = utils.DistroFamily(image.Distro)
 
 	// Copy catalog-provided tags (e.g. spinifex:managed-by for system AMIs)
 	// onto the imported AMI so the UI can filter them out.
@@ -574,6 +625,136 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
 	fmt.Printf("✅ Image import complete. Image-ID (AMI): %s\n", volumeId)
 }
 
+func runimagesRemoveCmd(cmd *cobra.Command, args []string) {
+	imageID, _ := cmd.Flags().GetString("image-id")
+	force, _ := cmd.Flags().GetBool("force")
+	yes, _ := cmd.Flags().GetBool("yes")
+
+	cfgFile, _ := cmd.Flags().GetString("config")
+	if cfgFile == "" {
+		cfgFile = DefaultConfigFile()
+	}
+
+	appConfig, err := config.LoadConfig(cfgFile)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "Error loading config file:", err)
+		os.Exit(1)
+	}
+
+	node := appConfig.Nodes[appConfig.Node]
+	store := objectstore.NewS3ObjectStoreFromConfig(
+		node.Predastore.Host,
+		node.Predastore.Region,
+		node.Predastore.AccessKey,
+		node.Predastore.SecretKey,
+	)
+	bucket := node.Predastore.Bucket
+
+	preview, err := admin.PreviewRemoveSystemImage(store, bucket, imageID)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "Failed to inspect AMI:", err)
+		os.Exit(1)
+	}
+
+	// Print metadata block.
+	fmt.Println("About to remove system AMI:")
+	fmt.Println()
+	fmt.Printf("  Image ID:        %s\n", preview.ImageID)
+	switch {
+	case !preview.ConfigPresent && !preview.ConfigCorrupt:
+		fmt.Println("  Name:            <unknown — config.json missing>")
+		fmt.Println("  Owner:           <unknown>")
+	case preview.ConfigCorrupt:
+		fmt.Println("  Name:            <unknown — config.json corrupt>")
+		fmt.Println("  Owner:           <unknown>")
+	default:
+		fmt.Printf("  Name:            %s\n", preview.Name)
+		fmt.Printf("  Owner:           %s\n", preview.Owner)
+		if !preview.Created.IsZero() {
+			fmt.Printf("  Created:         %s\n", preview.Created.UTC().Format("2006-01-02T15:04:05Z"))
+		}
+	}
+	fmt.Printf("  Backing storage: %s/      (%d objects, %s)\n",
+		preview.ImageID, preview.AMIObjectCount, utils.HumanBytes(utils.SafeInt64ToUint64(preview.AMIBytesTotal)))
+	fmt.Printf("                   %s/ (%d objects, %s)\n",
+		admin.SnapPrefix(preview.ImageID), preview.SnapObjectCount, utils.HumanBytes(utils.SafeInt64ToUint64(preview.SnapBytesTotal)))
+	fmt.Println()
+
+	// Account-owned guard before salvage / dependents — the AWS-flow hint is
+	// the most useful thing to surface for this kind of mistake.
+	if preview.ConfigPresent && !preview.IsSystemOwned && !force {
+		fmt.Fprintf(os.Stderr,
+			"Refusing to remove: %s is account-owned (%s).\n"+
+				"Use 'aws ec2 deregister-image --image-id %s' followed by 'aws ec2 delete-snapshot ...'.\n",
+			preview.ImageID, preview.Owner, preview.ImageID)
+		os.Exit(1)
+	}
+
+	if !preview.ConfigPresent && !force {
+		fmt.Fprintln(os.Stderr, "AMI config.json missing or corrupt; re-run with --force to salvage backing blocks.")
+		os.Exit(1)
+	}
+
+	if !preview.Dependents.Empty() && !force {
+		fmt.Fprintln(os.Stderr, "Refusing to remove: dependent resources reference this image.")
+		printDependents(os.Stderr, preview.Dependents)
+		fmt.Fprintln(os.Stderr, "Remove them first (e.g. 'aws ec2 terminate-instances', 'aws ec2 delete-snapshot', 'aws ec2 deregister-image') or re-run with --force.")
+		os.Exit(1)
+	}
+
+	if force && (!preview.ConfigPresent || !preview.Dependents.Empty()) {
+		fmt.Println("⚠️  --force: skipping dependency check and ownership check.")
+		if !preview.Dependents.Empty() {
+			printDependents(os.Stdout, preview.Dependents)
+		}
+		fmt.Println()
+	}
+
+	fmt.Println("This is permanent and cannot be undone.")
+	if !yes {
+		fmt.Print("Type 'yes' to proceed: ")
+		reader := bufio.NewReader(os.Stdin)
+		answer, _ := reader.ReadString('\n')
+		if strings.TrimSpace(answer) != "yes" {
+			fmt.Println("Aborted.")
+			return
+		}
+	}
+
+	res, err := admin.RemoveSystemImage(store, bucket, admin.RemoveImageOpts{
+		ImageID: imageID,
+		Force:   force,
+	})
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "Remove failed:", err)
+		os.Exit(1)
+	}
+
+	fmt.Printf("✅ Removed AMI %s (freed %s across %d objects).\n",
+		imageID, utils.HumanBytes(utils.SafeInt64ToUint64(res.BytesFreed)), res.ObjectsDeleted)
+}
+
+func printDependents(w io.Writer, d admin.Dependents) {
+	if len(d.Volumes) > 0 {
+		fmt.Fprintf(w, "  Volumes (%d):\n", len(d.Volumes))
+		for _, v := range d.Volumes {
+			fmt.Fprintf(w, "    - %s\n", v)
+		}
+	}
+	if len(d.Snapshots) > 0 {
+		fmt.Fprintf(w, "  Snapshots (%d):\n", len(d.Snapshots))
+		for _, s := range d.Snapshots {
+			fmt.Fprintf(w, "    - %s\n", s)
+		}
+	}
+	if len(d.AMIs) > 0 {
+		fmt.Fprintf(w, "  AMIs (%d):\n", len(d.AMIs))
+		for _, a := range d.AMIs {
+			fmt.Fprintf(w, "    - %s\n", a)
+		}
+	}
+}
+
 // List remote images available
 func runimagesListCmd(cmd *cobra.Command, args []string) {
 	//fmt.Println(availableImages)
@@ -1572,7 +1753,7 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 		}
 
 		fmt.Printf("   Waiting for cluster formation... (%d/%d nodes joined)\n", statusResp.Joined, statusResp.Expected)
-		time.Sleep(2 * time.Second)
+		time.Sleep(500 * time.Millisecond)
 	}
 
 	fmt.Printf("✅ Cluster formation complete! (%d nodes)\n\n", statusResp.Expected)

cmd/spinifex/cmd/admin_test.go

@@ -310,3 +310,24 @@ func TestWritePredastoreEncryptionKey_CreatesMissingDir(t *testing.T) {
 	require.NoError(t, err)
 	assert.True(t, dirInfo.IsDir())
 }
+
+// TestImagesRemoveCmd_FlagSchema guards the public flag surface for
+// `spx admin images remove`. --image-id is required so cobra rejects a bare
+// invocation; --force/--yes default to false.
+func TestImagesRemoveCmd_FlagSchema(t *testing.T) {
+	imageIDFlag := imagesRemoveCmd.Flags().Lookup("image-id")
+	require.NotNil(t, imageIDFlag, "--image-id must be defined")
+	assert.Equal(t, []string{"true"}, imageIDFlag.Annotations[cobraRequiredAnnotation],
+		"--image-id must be marked required")
+
+	forceFlag := imagesRemoveCmd.Flags().Lookup("force")
+	require.NotNil(t, forceFlag)
+	assert.Equal(t, "false", forceFlag.DefValue)
+
+	yesFlag := imagesRemoveCmd.Flags().Lookup("yes")
+	require.NotNil(t, yesFlag)
+	assert.Equal(t, "false", yesFlag.DefValue)
+}
+
+// cobraRequiredAnnotation is the annotation key cobra uses to mark required flags.
+const cobraRequiredAnnotation = "cobra_annotation_bash_completion_one_required_flag"

cmd/spinifex/cmd/cluster.go

@@ -98,7 +98,7 @@ func runClusterShutdown(cmd *cobra.Command, args []string) {
 			}
 			nc.Flush()
 			// Wait briefly for messages to propagate
-			time.Sleep(2 * time.Second)
+			time.Sleep(200 * time.Millisecond)
 			fmt.Printf("[INFRA] Complete\n")
 			break
 		}

cmd/spinifex/cmd/templates/predastore.toml

@@ -51,19 +51,19 @@ secret_access_key = "{{.SecretKey}}"
 # Using distributed hash ring for object > shard placement.
 [[nodes]]
 id = 1
-host = "0.0.0.0"
+host = "127.0.0.1"
 port = 9991
 path = "distributed/nodes/node-1/"
 
 [[nodes]]
 id = 2
-host = "0.0.0.0"
+host = "127.0.0.1"
 port = 9992
 path = "distributed/nodes/node-2/"
 
 [[nodes]]
 id = 3
-host = "0.0.0.0"
+host = "127.0.0.1"
 port = 9993
 path = "distributed/nodes/node-3/"
 

docs/COMMANDS.md

@@ -89,8 +89,9 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
 
 | Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
 |---------|-------|---------------|-------------|------------|--------|
-| `spx admin images import` | `--name`, `--file`, `--force`, `--skip-verify` | Cluster must be running; either `--name` (catalog download) or `--file` (operator-supplied media) | Catalog imports (`--name`) download the image, fetch the catalog `Checksum` URL, and verify the SHA-256/SHA-512 digest before extraction. Mismatch fails closed; the cached file is left on disk and `--force` re-downloads. `--file` imports skip verification — operator-supplied media is outside Spinifex's trust boundary, and the skip is logged at INFO for audit. `--skip-verify` bypasses verification for catalog imports and emits a WARN slog + stderr notice; use only for debugging or when upstream mirrors are confirmed-broken. | 1. Import valid catalog image (verifies checksum)<br>2. Tampered cache hit fails with `ErrChecksumMismatch`<br>3. `--force` recovers after a mismatch<br>4. `--file` import skips verification<br>5. `--skip-verify` bypasses checksum with WARN | **DONE** |
+| `spx admin images import` | `--name`, `--file`, `--distro`, `--version`, `--arch`, `--platform`, `--boot-mode` (bios/uefi/uefi-preferred), `--tag`, `--force`, `--skip-verify` | Cluster must be running; either `--name` (catalog download) or `--file` (operator-supplied media). `--file` mode additionally requires `--distro`, `--version`, `--arch`, `--boot-mode`. | Catalog imports (`--name`) download the image, fetch the catalog `Checksum` URL, verify the SHA-256/SHA-512 digest, and inherit `BootMode` from the catalog entry. `--boot-mode` overrides the catalog value when set. Mismatch fails closed; the cached file is left on disk and `--force` re-downloads. `--file` imports skip checksum verification (operator-supplied media is outside Spinifex's trust boundary, the skip is logged at INFO for audit) and require an explicit `--boot-mode` because there is no catalog metadata to inherit from. `--skip-verify` bypasses verification for catalog imports and emits a WARN slog + stderr notice; use only for debugging or when upstream mirrors are confirmed-broken. | 1. Import valid catalog image (verifies checksum)<br>2. Tampered cache hit fails with `ErrChecksumMismatch`<br>3. `--force` recovers after a mismatch<br>4. `--file` import skips verification<br>5. `--skip-verify` bypasses checksum with WARN<br>6. `--file` without `--boot-mode` rejected<br>7. `--boot-mode=invalid` rejected | **DONE** |
 | `spx admin images list` | — | None | Lists available OS images that can be imported or downloaded | 1. List available images | **DONE** |
+| `spx admin images remove` | `--image-id` (required), `--force`, `--yes` | Cluster must be running; predastore reachable. Targets AMIs imported via `spx admin images import` (non-account `ImageOwnerAlias`, e.g. `"system"`). | Loads `ami-<id>/config.json`, walks transitive dependents — copied snapshots whose `VolumeID == imageID`, volumes whose `SnapshotID` references the internal `snap-ami-<id>` or any derived snap, and account AMIs created via `CopyImage` whose `SnapshotID` is a derived snap — then prompts (skipped with `--yes`) before deleting `ami-<id>/config.json` (the DescribeImages barrier) followed by the rest of `ami-<id>/` and `snap-ami-<id>/`. Account-owned AMIs are refused with a hint pointing at `aws ec2 deregister-image` + `aws ec2 delete-snapshot`. `--force` bypasses the dependency, ownership and config-corrupt checks for salvage of orphaned blocks. | 1. Happy path: system AMI, no deps → deleted<br>2. Account-owned → refused with AWS-flow hint<br>3. Missing/corrupt config → `InvalidAMIID.NotFound` (salvageable with `--force`)<br>4. Dependent volume (direct or via copied snap) → refused<br>5. Dependent account AMI → refused<br>6. `--force` overrides dependents<br>7. Idempotent re-run after partial delete | **DONE** |
 
 #### Image integrity verification (CMMC SI.L1-3.14.2)
 
@@ -123,6 +124,35 @@ served. A mirror compromise that swaps both image and sums file is not
 detected; closing that gap requires GPG signature verification of the sums
 file, deferred to a later phase.
 
+#### `spx admin images remove` caveats
+
+Admin-imported AMIs (`ImageOwnerAlias = "system"`) live
+under the `ami-<id>/` S3 prefix and use a viperblock-internal snap checkpoint
+at `snap-ami-<id>/` — there is no `snap-<id>/metadata.json`. The AWS handlers
+(`DeregisterImage`, `DeleteSnapshot`) reject system owners with
+`UnauthorizedOperation`, which is the right behaviour for tenant API callers
+but leaves no AWS-flow path to reclaim space. `spx admin images remove` is
+the admin-trust-boundary counterpart that performs the dependency walk and
+hard-deletes the blocks directly against predastore.
+
+`CopyImage` of a system AMI is metadata-only: it writes a fresh
+`snap-<acct>/metadata.json` whose `VolumeID` points at `ami-<sys>` and a new
+`ami-<acct>/config.json` referencing that snap. Volumes launched from the
+copied AMI read transitively from `ami-<sys>/chunks/...`. The remove command
+walks this transitive set and refuses if anything references the target.
+
+**TOCTOU window:** between the safety scan and the `config.json` delete a
+concurrent `RunInstances` against the AMI could create a new dependent
+volume. The window is sub-second on a healthy cluster. The admin running
+this command is expected to know the fleet's operational state; if the race
+fires the result is a `vol-<id>` with deleted backing blocks, recovered by
+terminating the orphaned instance.
+
+**`--force` bypasses every safety check** (dependents, ownership, missing /
+corrupt `config.json`). Use only for salvage of orphaned blocks. Running it
+against a live system AMI corrupts every dependent volume on the next disk
+read.
+
 ### GPU Management
 
 | Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
@@ -168,7 +198,7 @@ file, deferred to a later phase.
 |---------|-------------------|---------------|---------------|-------------|------------|--------|
 | `describe-images` | `--image-ids` (format validation only), `--owners` (self, account ID, alias), `--filters` (name, state, architecture, image-id, is-public, owner-id, description, image-type, tag:\*) | `--executable-users`, `--include-deprecated`, `--include-disabled`, `--max-results`, `--next-token`, `--dry-run` | None | NATS `ec2.DescribeImages` → daemon reads AMI metadata from Predastore S3 buckets (ami-*) → filters by ImageIds, Owners, and Filters → returns image list with state, architecture, block device mappings | 1. List all images<br>2. Filter by image ID<br>3. Filter by owner (self/amazon)<br>4. Non-existent AMI returns empty<br>5. Verify metadata fields (architecture, state, rootDeviceName)<br>6. Filter by name wildcard<br>7. Unknown filter returns InvalidParameterValue | **DONE** |
 | `create-image` | `--instance-id`, `--name`, `--description`, `--tag-specifications` | `--no-reboot`, `--block-device-mappings`, `--dry-run` | Instance must exist (running or stopped) | Gateway validates input → NATS `ec2.{instanceId}.CreateImage` (per-instance topic) → daemon extracts root volume → snapshots via `ebs.snapshot` NATS (running) or offline S3 copy (stopped) → creates AMI metadata in S3 (`{amiId}/config.json`) → stores tags → returns ami-ID. Duplicate AMI name validation enforced. | 1. Create image from running instance<br>2. Create image from stopped instance<br>3. Invalid instance ID (error)<br>4. Duplicate AMI name (error)<br>5. Verify new AMI appears in describe-images<br>6. Launch new instance from created AMI | **DONE** |
-| `register-image` | `--name`, `--description`, `--architecture` (x86_64/arm64/i386), `--root-device-name`, `--virtualization-type` (hvm only), `--block-device-mappings` (root with `Ebs.SnapshotId`+optional `VolumeSize`), `--tag-specifications` | `--billing-products`, `--uefi-data` | Backing snapshot must exist in Predastore (`{snapshotId}/metadata.json`); caller must own snapshot or it must be system-owned | Gateway validates name length (3–128), `snap-` prefix, architecture/virtualization values → NATS `ec2.RegisterImage` → daemon checks AMI name uniqueness, reads snapshot metadata, verifies snapshot ownership, builds `viperblock.AMIMetadata` (defaults: `Architecture=x86_64`, `Virtualization=hvm`, `PlatformDetails=Linux/UNIX`, `RootDeviceType=ebs`), writes `{amiId}/config.json`. Pointer-only — never touches block data. `BootMode`, `KernelId`, `RamdiskId`, `TpmSupport`, `ImdsSupport`, `EnaSupport`, `SriovNetSupport`, `ImageLocation`, `paravirtual` virtualization rejected with `InvalidParameterValue`. `VolumeSize` smaller than snapshot rejected. | 1. Register with valid snapshot<br>2. Missing name/snapshot (error)<br>3. Duplicate AMI name (`InvalidAMIName.Duplicate`)<br>4. Snapshot not found (`InvalidSnapshot.NotFound`)<br>5. Cross-account snapshot (`UnauthorizedOperation`)<br>6. Tags from `TagSpecifications` persisted<br>7. Verify registered image in describe-images | **DONE** |
+| `register-image` | `--name`, `--description`, `--architecture` (x86_64/arm64/i386), `--root-device-name`, `--virtualization-type` (hvm only), `--boot-mode` (bios/uefi/uefi-preferred), `--block-device-mappings` (root with `Ebs.SnapshotId`+optional `VolumeSize`), `--tag-specifications` | `--billing-products`, `--uefi-data` | Backing snapshot must exist in Predastore (`{snapshotId}/metadata.json`); caller must own snapshot or it must be system-owned | Gateway validates name length (3–128), `snap-` prefix, architecture/virtualization/boot-mode values → NATS `ec2.RegisterImage` → daemon checks AMI name uniqueness, reads snapshot metadata, verifies snapshot ownership, builds `viperblock.AMIMetadata` (defaults: `Architecture=x86_64`, `Virtualization=hvm`, `PlatformDetails=Linux/UNIX`, `RootDeviceType=ebs`), writes `{amiId}/config.json`. Pointer-only — never touches block data. `BootMode` outside `bios|uefi|uefi-preferred`, `KernelId`, `RamdiskId`, `TpmSupport`, `ImdsSupport`, `EnaSupport`, `SriovNetSupport`, `ImageLocation`, `paravirtual` virtualization rejected with `InvalidParameterValue`. `uefi-preferred` is treated as `uefi` at launch. `VolumeSize` smaller than snapshot rejected. | 1. Register with valid snapshot<br>2. Missing name/snapshot (error)<br>3. Duplicate AMI name (`InvalidAMIName.Duplicate`)<br>4. Snapshot not found (`InvalidSnapshot.NotFound`)<br>5. Cross-account snapshot (`UnauthorizedOperation`)<br>6. Tags from `TagSpecifications` persisted<br>7. Verify registered image in describe-images<br>8. `--boot-mode=uefi` accepted, returned by `describe-images`<br>9. `--boot-mode=invalid` rejected (`InvalidParameterValue`) | **DONE** |
 | `deregister-image` | `--image-id` | `--dry-run` | AMI must exist; caller must own it (system AMIs immutable via this API) | Gateway validates `ami-` prefix → NATS `ec2.DeregisterImage` → daemon hard-deletes `{amiId}/config.json` from Predastore. Backing snapshot is left intact (matches AWS — operators run `delete-snapshot` separately to reclaim block storage). Cross-account/system-AMI mutations rejected with `UnauthorizedOperation`. Re-deregister returns `InvalidAMIID.NotFound` (no tombstone). | 1. Deregister existing AMI<br>2. Deregister non-existent AMI (`InvalidAMIID.NotFound`)<br>3. Re-deregister already-deleted AMI (`InvalidAMIID.NotFound`)<br>4. Cross-account AMI (`UnauthorizedOperation`)<br>5. System AMI (`UnauthorizedOperation`)<br>6. Verify deregistered AMI not in describe-images<br>7. Backing snapshot untouched | **DONE** |
 | `copy-image` | `--source-image-id`, `--source-region` (must equal gateway region), `--name`, `--description`, `--client-token` (accepted, not honoured), `--copy-image-tags`, `--tag-specifications` (image only) | `--encrypted`, `--kms-key-id`, `--destination-outpost-arn`, `--dry-run` | Source AMI must exist and be owned by caller OR be a system AMI; backing snapshot must still exist | Gateway validates `ami-` prefix, name length (3–128), same-region — cross-region / `Encrypted` / `KmsKeyId` / `DestinationOutpostArn` rejected with `InvalidParameterValue`. NATS `ec2.CopyImage` → daemon checks name uniqueness, reads source `ami-xxx/config.json`, visibility-filters cross-account sources as `InvalidAMIID.NotFound`, reads source `{snap-xxx}/metadata.json` (missing or empty SnapshotID → `InvalidAMIID.NotFound`), writes new `snap-yyy/metadata.json` that inherits source `VolumeID` (no block copy), writes new `ami-yyy/config.json` owned by caller. `Description` inherits from source when unset. Tag merge: `CopyImageTags=true` seeds tags from source, explicit image-resource `TagSpecifications` override colliding keys and append new ones; non-image tag specs ignored. | 1. Same-region copy produces new `ami-`/`snap-` pair with distinct IDs<br>2. New snap shares source `VolumeID` (metadata-only)<br>3. New AMI owned by caller, source untouched<br>4. System AMI copied into caller's account<br>5. Cross-account source (`InvalidAMIID.NotFound`)<br>6. Missing source AMI (`InvalidAMIID.NotFound`)<br>7. Orphaned source (missing/empty `SnapshotID`) → `InvalidAMIID.NotFound`<br>8. Duplicate name (`InvalidAMIName.Duplicate`)<br>9. Cross-region / `Encrypted` / `KmsKeyId` rejected (`InvalidParameterValue`)<br>10. `CopyImageTags` true/false tag inheritance semantics | **DONE** |
 | `import-image` | — | `--disk-containers` (Format, Url/S3Bucket+S3Key), `--description`, `--architecture`, `--platform` | S3 bucket with disk image | Download disk image from S3 → convert format (VMDK/VHD/RAW→QCOW2) → create viperblock volume → register as AMI | 1. Import QCOW2 image<br>2. Import RAW image<br>3. Invalid format (error)<br>4. Verify imported image launchable | **NOT STARTED** |

docs/DESIGN.md

@@ -16,14 +16,14 @@ Users interact with Spinifex using standard AWS SDKs or the AWS CLI by using the
 
 ```bash
 AWS_PROFILE=spinifex aws ec2 run-instances \
-    --image-id ami-debian12 \
+    --image-id ami-debian13 \
     --instance-type t3.micro \
     --key-name my-keypair
 ```
 
 The AWS SDK formats this as an HTTPS POST with:
 - AWS SigV4 authentication headers
-- EC2 Query Protocol body (`Action=RunInstances&ImageId=ami-debian12&...`)
+- EC2 Query Protocol body (`Action=RunInstances&ImageId=ami-debian13&...`)
 
 ### 2. AWS Gateway
 
@@ -84,6 +84,13 @@ The EC2 handler (`spinifex/gateway/ec2.go`) parses the `Action` parameter and de
 
 ### 4. NATS Messaging
 
+The full service-to-subject map — which service subscribes to what, what
+each one publishes downstream, what each depends on — is captured in
+[`service-interfaces.yaml`](./service-interfaces.yaml). It is the source
+of truth for targeted e2e suite selection (see
+`docs/development/improvements/e2e-targeted-suite-selection.md`) and is
+validated by `make manifest-check`.
+
 The gateway communicates with daemons via NATS request/response. Most calls go through `utils.NATSRequest`, which marshals the input, attaches the account ID as a NATS header, and unmarshals the typed response:
 
 ```go
@@ -297,7 +304,7 @@ sudo systemctl start spinifex.target
 export AWS_PROFILE=spinifex
 
 # Run instance
-aws ec2 run-instances --image-id ami-debian12 --instance-type t3.micro
+aws ec2 run-instances --image-id ami-debian13 --instance-type t3.micro
 
 # List instances (queries all nodes)
 aws ec2 describe-instances

docs/getting-started/install-airgapped/README.md

@@ -57,6 +57,7 @@ sudo apt update
 sudo apt install --download-only -y \
   nbdkit nbdkit-plugin-dev pkg-config \
   qemu-system-x86 qemu-utils qemu-kvm \
+  ovmf qemu-efi-aarch64 \
   libvirt-daemon-system libvirt-clients libvirt-dev \
   ovn-central ovn-host openvswitch-switch \
   dhcpcd-base make gcc jq curl iproute2 netcat-openbsd \
@@ -74,7 +75,7 @@ Mirror the cloud image you intend to run as guest VMs:
 
 ```bash
 mkdir -p images
-curl -fsSL "https://cloud.debian.org/images/cloud/trixie/20260501-2465/debian-13-genericcloud-amd64-20260501-2465.qcow2" \
+curl -fsSL "https://cloud.debian.org/images/cloud/trixie/20260518-2482/debian-13-genericcloud-amd64-20260518-2482.qcow2" \
   -o images/debian-13-amd64.qcow2
 ```
 
@@ -141,7 +142,7 @@ Register the pre-staged image with Spinifex:
 
 ```bash
 sudo spx admin images import --file /mnt/usb/images/debian-13-amd64.qcow2 \
-  --distro debian --version 13 --arch x86_64
+  --distro debian --version 13 --arch x86_64 --boot-mode uefi
 ```
 
 ## Step 9. Verify
@@ -183,4 +184,4 @@ sudo INSTALL_SPINIFEX_TARBALL=/mnt/usb/tarball/spinifex-...tar.gz \
 
 ### Image Import Fails
 
-`spx admin images import --file` requires the distro/version/arch flags so the image registers in the catalogue. If the import succeeds but `aws ec2 describe-images` returns empty, check `journalctl -u spinifex-daemon -f` for predastore upload errors.
+`spx admin images import --file` requires the distro/version/arch/boot-mode flags so the image registers in the catalogue. If the import succeeds but `aws ec2 describe-images` returns empty, check `journalctl -u spinifex-daemon -f` for predastore upload errors.

docs/getting-started/setting-up-your-cluster/README.md

@@ -58,10 +58,10 @@ spx admin images list
 
 ```
 NAME                    | DISTRO | VERSION | ARCH   | BOOT
-debian-13-arm64         | debian | 13      | arm64  | bios
-debian-13-x86_64        | debian | 13      | x86_64 | bios
-ubuntu-26.04-arm64      | ubuntu | 26.04   | arm64  | bios
-ubuntu-26.04-x86_64     | ubuntu | 26.04   | x86_64 | bios
+debian-13-arm64         | debian | 13      | arm64  | uefi
+debian-13-x86_64        | debian | 13      | x86_64 | uefi
+ubuntu-26.04-arm64      | ubuntu | 26.04   | arm64  | uefi
+ubuntu-26.04-x86_64     | ubuntu | 26.04   | x86_64 | uefi
 ```
 
 Import an image:
@@ -73,7 +73,7 @@ spx admin images import --name ubuntu-26.04-x86_64
 Or import a local image file:
 
 ```bash
-spx admin images import --file ~/images/ubuntu-26.04.img --distro ubuntu --version 26.04 --arch x86_64
+spx admin images import --file ~/images/ubuntu-26.04.img --distro ubuntu --version 26.04 --arch x86_64 --boot-mode uefi
 ```
 
 Verify the import and note the AMI ID:

docs/security/malware-protection/README.md

@@ -73,7 +73,7 @@ All Spinifex nodes must include the following in their malware protection scope.
 |------|----------|
 | `/usr/local/bin/spx` | Spinifex CLI and service entry point. All `spinifex-*.service` units invoke `spx service <name> start`, so every service binary is reached through this one path. |
 | `/usr/bin/ovn-*`, `/usr/bin/ovs-*` | OVN / OVS control-plane and datapath binaries (installed via `apt` from the `ovn-central`, `ovn-host`, `openvswitch-switch` packages). |
-| `/usr/lib/<arch>-linux-gnu/nbdkit/plugins/nbdkit-viperblock-plugin.so` | Viperblock NBD plugin loaded by `nbdkit` to expose EBS volumes as block devices. The exact path comes from `nbdkit --dump-config` (`plugindir=`); on Debian 12 amd64 it is `/usr/lib/x86_64-linux-gnu/nbdkit/plugins/`. |
+| `/usr/lib/<arch>-linux-gnu/nbdkit/plugins/nbdkit-viperblock-plugin.so` | Viperblock NBD plugin loaded by `nbdkit` to expose EBS volumes as block devices. The exact path comes from `nbdkit --dump-config` (`plugindir=`); on Debian 13 amd64 it is `/usr/lib/x86_64-linux-gnu/nbdkit/plugins/`. |
 | `/usr/local/share/ca-certificates/spinifex-ca.crt` | Cluster CA cert installed into the system trust store |
 
 ### 1.2 Configuration and Secrets
@@ -157,7 +157,7 @@ All scan runs — scheduled and real-time — must produce a log entry that incl
 
 ## 4. Reference Configurations
 
-The configurations below are reference baselines. They are known to meet the CMMC L1 objectives on a standard Debian 12 Spinifex node; they are not the only valid implementation. Operators running a commercial EDR should treat [§1](#1-designated-locations-sil1-3142-a) and [§3](#3-scan-schedule-sil1-3145) as the contract and configure their chosen product to match.
+The configurations below are reference baselines. They are known to meet the CMMC L1 objectives on a standard Debian 13 Spinifex node; they are not the only valid implementation. Operators running a commercial EDR should treat [§1](#1-designated-locations-sil1-3142-a) and [§3](#3-scan-schedule-sil1-3145) as the contract and configure their chosen product to match.
 
 ### 4.1 ClamAV (signature-based AV)
 

docs/security/media-sanitization/README.md

@@ -0,0 +1,258 @@
+---
+title: "Media Sanitization and Disposal"
+description: "Operator guide for sanitizing and disposing of storage media used by Spinifex nodes"
+category: "Security"
+sections:
+  - overview
+tags:
+  - security
+  - compliance
+  - cmmc
+  - media
+  - sanitization
+  - disposal
+  - decommissioning
+resources:
+  - title: "NIST SP 800-88 Rev 1 — Guidelines for Media Sanitization"
+    url: "https://csrc.nist.gov/pubs/sp/800/88/r1/final"
+  - title: "NIST SP 800-171 Rev 3"
+    url: "https://csrc.nist.gov/pubs/sp/800/171/r3/final"
+  - title: "CMMC Level 1 Self-Assessment Guide v2.0"
+    url: "https://dodcio.defense.gov/CMMC/Documentation"
+  - title: "ATA/ATAPI Command Set — Secure Erase"
+    url: "https://www.t13.org"
+  - title: "NVM Express — Sanitize Command"
+    url: "https://nvmexpress.org/specifications"
+---
+
+# Media Sanitization and Disposal
+
+> Operator guide for sanitizing and disposing of storage media used by Spinifex nodes
+
+## Table of Contents
+
+- [Overview](#overview)
+- [CMMC Practices Covered](#cmmc-practices-covered)
+- [Approach](#approach)
+- [1. Media in Scope](#1-media-in-scope)
+- [2. Sanitization Method Selection](#2-sanitization-method-selection)
+- [3. Volume-Level Sanitization — Before Reuse (MP.L1-3.8.3 [b])](#3-volume-level-sanitization--before-reuse-mpl1-383-b)
+- [4. Whole-Drive and Node Decommissioning — Before Disposal (MP.L1-3.8.3 [a])](#4-whole-drive-and-node-decommissioning--before-disposal-mpl1-383-a)
+- [5. Key Destruction](#5-key-destruction)
+- [6. Removable and Backup Media](#6-removable-and-backup-media)
+- [7. Evidence and Record Keeping](#7-evidence-and-record-keeping)
+- [8. Operator Checklist](#8-operator-checklist)
+
+---
+
+## Overview
+
+**Audience:** Operators decommissioning, reassigning, returning under warranty, or disposing of hardware that has been used to run Spinifex nodes in environments subject to CMMC Level 1, or any site that requires documented media sanitization.
+
+**Scope:** Every storage medium that has, or may have, held Federal Contract Information (FCI) as part of a Spinifex deployment — node system disks, Viperblock WAL/chunk disks, Predastore object stores, NATS JetStream disks, removable master-key tokens, backup media, and any loose drives that previously occupied a Spinifex role. Guest VM internal media (disks as seen from inside a tenant VM) is the workload owner's responsibility and out of scope.
+
+## CMMC Practices Covered
+
+| Practice | Title | Objective |
+|----------|-------|-----------|
+| MP.L1-3.8.3 | Media Disposal | [a] System media containing FCI is sanitized or destroyed before disposal. [b] System media containing FCI is sanitized before it is released for reuse. |
+
+## Approach
+
+NIST SP 800-88 Rev 1 defines three sanitization categories — **Clear**, **Purge**, and **Destroy** — chosen by the confidentiality of the data and whether the medium will leave the operator's control. Spinifex discharges MP.L1-3.8.3 along two tracks:
+
+1. **Volume-level sanitization** for EBS volumes and VM disks returning to the free pool. Handled in-platform by **cryptographic erase** — the per-volume data encryption key (DEK) is deleted, rendering the ciphertext in Predastore S3 irrecoverable. This is the NIST SP 800-88 **Purge**-level method for encrypted media.
+2. **Whole-drive sanitization** for physical media leaving a Spinifex node — drive replacement, node retirement, warranty return, resale, or destruction. The operator runs the sanitization; Spinifex does not and cannot reach the firmware commands required. This guide prescribes the method per media type and the records to keep.
+
+The key principle: every piece of media that ever held plaintext FCI (or keys that wrapped FCI ciphertext) must be sanitized to at least **Purge** before release for reuse, and to **Purge** or **Destroy** before leaving operator control. When in doubt, Destroy.
+
+## 1. Media in Scope
+
+Any medium in any of these roles on a Spinifex node has held, or may have held, FCI:
+
+| Medium | Typical hardware | Contents |
+|--------|------------------|----------|
+| Node system disk | NVMe/SATA SSD | `/etc/spinifex/master.key`, cluster CA key, per-node TLS keys, service configs, logs, journal, swap. |
+| Viperblock WAL device | NVMe/SATA SSD | Plaintext of in-flight block writes before chunking. High-churn, short residency, but FCI lands here. |
+| Viperblock chunk cache / local backing | NVMe/SATA SSD or HDD | Encrypted volume chunks. |
+| Predastore object store | NVMe/SATA SSD or HDD | All S3 object data — AMIs, snapshots, user-uploaded artifacts, IAM state files, tenant data. |
+| NATS JetStream disk | NVMe/SATA SSD | IAM NATS token, cluster metadata, pending job state, NATS KV entries (including wrapped DEKs). |
+| Removable master-key media | USB flash token, HSM, smart card | Master encryption key (Phase 4 USB-mount path). Destroying this sanitizes every DEK-encrypted volume cluster-wide. |
+| Backup media | LTO tape, removable HDD/SSD, off-site cloud backup | Any of the above, point-in-time. |
+| BMC / iDRAC / iLO storage | Embedded flash | Console recordings, SEL logs, BMC credentials, cached operator certificates. Sanitize via the BMC "Reset to defaults" / "Erase user data" command before disposal. |
+| Switch / router config storage | Embedded flash | Cluster VLAN, management IPs, ACLs. Operator network kit, out of Spinifex scope but noted for completeness. |
+| Optical / write-once media | DVD/BD-R | If used to transport keys or images. Always Destroy. |
+
+Any drive whose history cannot be traced — pulled from a spares bin, recovered from a failed host, found unlabelled — must be treated as though it held FCI.
+
+## 2. Sanitization Method Selection
+
+The method depends on (a) media type and (b) whether the media remains inside the protected boundary after sanitization.
+
+| Scenario | Minimum method | Reference |
+|----------|----------------|-----------|
+| Encrypted volume returning to free pool, DEK deletable | **Cryptographic Erase** (Purge) via DEK deletion | [§3](#3-volume-level-sanitization--before-reuse-mpl1-383-b) |
+| SSD / NVMe leaving the node | **Cryptographic Erase** + **Purge** (SANITIZE block-erase or ATA Secure Erase) | [§4.1](#41-nvme-ssd) |
+| Magnetic HDD leaving the node, operational | **Purge** (single-pass overwrite with verify, or ATA Secure Erase on drives that support it) | [§4.3](#43-hdd) |
+| Magnetic HDD leaving the node, faulty (not writable) | **Destroy** (degauss followed by shred/incinerate) | [§4.3](#43-hdd) |
+| Any media leaving operator control (resale, warranty, disposal) | **Destroy** if residual confidentiality concern remains after Purge; otherwise Purge + third-party attestation | [§4.4](#44-destruction) |
+| Tape | **Destroy** (degauss + shred) — overwrite is not reliable on LTO with compression | [§6](#6-removable-and-backup-media) |
+| Optical write-once | **Destroy** (shred/incinerate) | [§6](#6-removable-and-backup-media) |
+| BMC / switch flash | Vendor "factory reset + erase user data"; Destroy chip if procedure is not available | [§1](#1-media-in-scope) |
+
+**Cryptographic erase is only valid when** the key was generated and handled under modern cryptographic hygiene (AES-256 or equivalent, key never exfiltrated, key store itself sanitized) and the encrypted data was not also written anywhere in plaintext. For Spinifex volumes these conditions hold.
+
+**Self-encrypting drives (SEDs)** support a single-command cryptographic erase via the TCG Opal "RevertSP" / `sedutil-cli --revertNoErase` or the ATA `SECURITY ERASE UNIT ENHANCED` that triggers the on-drive MEK rotation. When using an SED as a node system or data disk, record the SED model and method in the decommissioning procedure.
+
+## 3. Volume-Level Sanitization — Before Reuse (MP.L1-3.8.3 [b])
+
+Applies to EBS volumes, AMIs, and snapshots whose underlying storage will be reused for a different tenant or workload.
+
+### 3.1 Cryptographic Erase via DEK Deletion
+
+Deletion is sanitization:
+
+- `DeleteVolume` removes the wrapped DEK from NATS KV. The chunks in Predastore S3 remain but are unrecoverable ciphertext. No further operator action is required.
+- `TerminateInstances` with the default `DeleteOnTermination=true` on root volumes performs the same cryptographic erase for the terminated instance's root.
+- `DeleteSnapshot` removes the snapshot's DEK wrapper; the underlying chunks become irrecoverable ciphertext.
+
+This is the sanitization-before-reuse path for every in-platform object. Operators do not need to take separate action — the API call is the sanitization.
+
+### 3.2 What Happens to the Physical Storage
+
+Cryptographic erase leaves ciphertext chunks in Predastore. These chunks are eventually overwritten as new volumes reuse the space. For CMMC purposes, the ciphertext without the DEK satisfies Purge. The underlying drive still requires whole-drive sanitization when it eventually leaves the cluster — see [§4](#4-whole-drive-and-node-decommissioning--before-disposal-mpl1-383-a).
+
+## 4. Whole-Drive and Node Decommissioning — Before Disposal (MP.L1-3.8.3 [a])
+
+Applies when a physical drive — or a whole node — leaves the Spinifex cluster: drive swap, node retirement, warranty return, resale, recycling, destruction.
+
+### 4.1 NVMe SSD
+
+Preferred: the NVMe **SANITIZE** command with the **Block Erase** action. Supported on most enterprise NVMe drives.
+
+```bash
+# Confirm support
+nvme id-ctrl /dev/nvme0 | grep -i sanicap
+
+# Purge — block erase
+nvme sanitize /dev/nvme0 --sanact=2
+
+# Poll until complete
+nvme sanitize-log /dev/nvme0
+```
+
+Fallback for drives without SANITIZE support: NVMe **Format** with Secure Erase set to 1 (`--ses=1`), or vendor tool. Cryptographic Erase (`--sanact=4`) is acceptable for SEDs whose encryption posture is documented; otherwise Block Erase is the safer default.
+
+Boot media: NVMe targets must be the non-boot drive when running from the OS. For the node system disk, boot a sanitization-purpose live USB (e.g. [PartedMagic](https://partedmagic.com/), the vendor's diagnostic ISO) or pull the drive and sanitize in a dedicated sanitization workstation.
+
+### 4.2 SATA/SAS SSD
+
+Preferred: **ATA Secure Erase (Enhanced)** via `hdparm`:
+
+```bash
+# Confirm support and not frozen
+hdparm -I /dev/sdX | grep -A1 "Security"
+
+# If frozen, power-cycle (suspend/resume or hot-swap) without unplugging the OS disk
+
+# Set password and issue enhanced erase
+hdparm --user-master u --security-set-pass p /dev/sdX
+hdparm --user-master u --security-erase-enhanced p /dev/sdX
+```
+
+Fallback: vendor Secure Erase utility, or `blkdiscard --secure /dev/sdX` on drives that report `TRIM deterministic + RZAT`. A single-pass overwrite with `shred -v -n 1 /dev/sdX` is **not** sufficient for SSDs because of wear-levelling reserve blocks — use Secure Erase or Destroy.
+
+### 4.3 HDD
+
+Preferred: **ATA Secure Erase** (same `hdparm` sequence as [§4.2](#42-satasas-ssd)). Most modern HDDs support it.
+
+Fallback: single-pass overwrite with verify:
+
+```bash
+shred -v -n 1 -z /dev/sdX
+# or
+dd if=/dev/zero of=/dev/sdX bs=1M status=progress && \
+  cmp /dev/zero /dev/sdX          # expect "EOF on /dev/zero" only
+```
+
+For HDDs that fail or refuse sanitization, or that held high-sensitivity data, **degauss then Destroy** — degaussing alone renders a modern HDD non-functional, so there is no reuse path after degauss. Degaussing requires an NSA/CSS-listed degausser appropriate to the drive's coercivity.
+
+### 4.4 Destruction
+
+When a drive cannot be sanitized (failed, SED with no working password, sanitize aborts) or when leaving operator control with any residual confidentiality concern, Destroy:
+
+- **SSD/NVMe:** shred to ≤2 mm particle size (NSA/CSS EPL-listed shredder) or incinerate. Crushing alone is not sufficient for modern flash — chips can survive.
+- **HDD:** degauss (on drives with magnetic platters) followed by shredding, drilling multiple holes through the platters and the head assembly, or incineration.
+- **Tape, optical:** shredding.
+- **Chain of custody:** transport destroyed media inside tamper-evident containers. The destruction record ([§7](#7-evidence-and-record-keeping)) must name the destruction method, destruction vendor (if third-party), and operator witness.
+
+Third-party destruction vendors must provide a certificate of destruction listing every drive serial. Reconcile against the asset register before the destruction record is closed.
+
+### 4.5 Node Decommissioning Runbook
+
+When retiring a whole node:
+
+1. **Drain** — migrate or terminate all instances hosted on the node; confirm Predastore and Viperblock roles are reassigned if the node held one. Volume DEKs of terminated instances are already cryptographically erased per [§3.1](#31-cryptographic-erase-via-dek-deletion).
+2. **Deregister** — `bd`-linked change ticket; the operator removes the node from the cluster (`spx admin node remove`) so NATS routes, OVN chassis, and predastore distributed membership drop it.
+3. **Stop services and unmount** — `systemctl stop 'spinifex-*'`, `nbdkit` sessions, OVN agent. Confirm `/run/spinifex/nbd/` is empty.
+4. **Destroy on-disk keys** — see [§5](#5-key-destruction). This is the single most important step: sanitization of volume data reduces to sanitization of keys once data is encrypted.
+5. **Sanitize each drive** by media type per [§4.1](#41-nvme-ssd) – [§4.3](#43-hdd). Every drive bay is sanitized, including any unpopulated cache or WAL devices.
+6. **Verify** — capture the output of the sanitize/erase command (exit code + sanitize-log for NVMe, `hdparm -I` "not enabled, not locked" for ATA, overwrite verify for HDD overwrite) into the decommissioning record.
+7. **Label** drives "Sanitized — <method> — <date> — <operator>" before they leave the rack or enter a transit container. Unlabelled drives are treated as unsanitized if reintroduced.
+8. **Record** per [§7](#7-evidence-and-record-keeping).
+
+## 5. Key Destruction
+
+Cryptographic erase relies on the key being unrecoverable. A drive that previously held `/etc/spinifex/master.key` must have that key destroyed — otherwise an attacker with both the drive image and the key recovers the cleartext even after "erase".
+
+| Key | Location | Destruction |
+|-----|----------|-------------|
+| `/etc/spinifex/master.key` | Node system disk (every node) | Overwrite the file (`shred -u /etc/spinifex/master.key`) before the drive is sanitized, in addition to drive sanitization. Once the master key is gone cluster-wide, every DEK it wrapped is effectively destroyed. |
+| Cluster CA private key (`/etc/spinifex/ca.key`) | Leader node system disk | Same treatment. Loss of the CA key does not sanitize data, but prevents impersonation of the cluster identity post-disposal. |
+| Per-node TLS private key (`/etc/spinifex/server.key`) | Each node system disk | `shred -u` before drive sanitization. |
+| Wrapped DEKs in NATS KV | `/var/lib/spinifex/nats/` on NATS-carrying nodes | Deleted via the `DeleteVolume` API; the KV record is compacted. Sanitization of the NATS JetStream disk per [§4](#4-whole-drive-and-node-decommissioning--before-disposal-mpl1-383-a) ensures no recoverable copies. |
+| Removable master-key media (Phase 4 USB) | USB flash token | Physically destroy the token when the cluster is decommissioned; alternatively retain inside a secured location as evidence of cluster-wide cryptographic erase. Do not reuse for another cluster without full sanitization (see [§6](#6-removable-and-backup-media)). |
+| Backup-copy keys | Any off-site or escrow copy | Destroy simultaneously with the primary. A surviving backup defeats the cryptographic erase. |
+
+If even one copy of the master key survives and the encrypted drives are recoverable, cryptographic erase has **not** been achieved. Track every key copy in the device register ([Physical Security Guide §5](/docs/security/physical-security-guide#5-manage-physical-access-devices-pel1-3105)).
+
+## 6. Removable and Backup Media
+
+| Media | Sanitization |
+|-------|--------------|
+| USB flash (master-key token, key-transport USB, image-import USB) | Destroy. The price of a replacement does not justify the risk of partial wear-levelled sanitization. Record the serial in the device register and the destruction in the decommissioning log. |
+| External HDD/SSD (backup) | Treat as [§4.2](#42-satasas-ssd) / [§4.3](#43-hdd). |
+| LTO tape | Degauss with a tape-rated degausser, then shred. Overwrite is not reliable under LTO hardware compression and is not accepted by 800-88 for tape Purge. |
+| Optical (CD/DVD/BD-R) | Shred or incinerate. |
+| Off-site cloud backup | Issue the cloud provider's delete-and-purge API; retain the provider's deletion-confirmation receipt. For cryptographically wrapped backups, destroy the wrapping key cluster-side and retain that deletion as evidence. |
+| Printed material (console photos, recovery codes, admin handover sheets) | Cross-cut shred. |
+
+Any removable medium that has entered the protected boundary and been written to must be tracked in the device register (see [Physical Security Guide §5](/docs/security/physical-security-guide#5-manage-physical-access-devices-pel1-3105)) and accounted for at decommissioning.
+
+## 7. Evidence and Record Keeping
+
+For CMMC assessment, retain the following for at least three years (or longer where contract policy requires):
+
+- **Asset register** listing every drive and removable medium by serial number, media type, role in the cluster (system / WAL / chunk / predastore / NATS / backup), and in-service / decommissioned status.
+- **Decommissioning record** per drive or node with: date, operator, sanitization method, command output or tool report, verification result, final disposition (reuse / return / destroy), destination (e.g. "warranty RMA #12345", "shredder vendor X, CoD #6789").
+- **Certificates of destruction** from third-party vendors, reconciled against the asset register.
+- **Key-destruction record** confirming `master.key`, CA key, and per-node keys were shredded before drive sanitization.
+- **Exceptions log** — faulty drives that could not be sanitized and were destroyed instead; any sanitize failures and their resolution.
+- **Annual attestation** from the operator confirming the procedures in this guide operated for the prior 12 months, signed by the named security owner.
+
+Cross-reference each decommissioning record to the change ticket that drove the retirement, so drive disposal can be audited against cluster state change.
+
+## 8. Operator Checklist
+
+Use this list to confirm a Spinifex deployment meets MP.L1-3.8.3:
+
+- Asset register enumerates every drive and removable medium by serial, media type, and role.
+- Every volume-producing path uses encryption so that `DeleteVolume` is sanitization.
+- Node-decommissioning runbook exists and matches [§4.5](#45-node-decommissioning-runbook), including the drain → deregister → key-destroy → drive-sanitize sequence.
+- Sanitization method is pre-selected per media type per [§2](#2-sanitization-method-selection); tool availability (hdparm, nvme-cli, vendor ISOs, degausser, shredder or destruction vendor) is verified before a decommissioning begins.
+- `master.key`, CA key, and per-node TLS keys are shredded before the drive holding them is sanitized.
+- Every backup copy of any key is destroyed at the same time as the primary; surviving backup copies are tracked and accounted for.
+- Removable media (USB, tape, optical) is destroyed rather than erased unless the medium is an SED with documented cryptographic erase.
+- Decommissioning records capture method, tool output, verification, final disposition, and operator; certificates of destruction are reconciled against the asset register.
+- Drives awaiting sanitization or awaiting pickup by a destruction vendor are held inside the physical protection boundary ([Physical Security Guide §1](/docs/security/physical-security-guide#1-protected-assets)) with chain of custody logged.
+- System security plan references this guide and names the sanitization tooling, destruction vendor (if any), retention period for records, and the security owner attesting annually.

docs/security/network-connections/README.md

@@ -105,8 +105,8 @@ Spinifex nodes initiate a small, fixed set of outbound connections.
 
 | Destination | Purpose | Protocol | Verification |
 |-------------|---------|----------|--------------|
-| `https://cloud.debian.org/images/cloud/trixie/20260501-2465` | Debian 13 cloud image download | HTTPS | TLS + checksum verification |
-| `https://cloud-images.ubuntu.com/resolute/current` | Ubuntu 26.04 LTS cloud image download | HTTPS | TLS + checksum verification |
+| `https://cloud.debian.org/images/cloud/trixie/` | Debian 13 cloud image download | HTTPS | TLS + checksum verification |
+| `https://cloud-images.ubuntu.com/resolute/` | Ubuntu 26.04 LTS cloud image download | HTTPS | TLS + checksum verification |
 | `https://d2yp8ipz5jfqcw.cloudfront.net` | Alpine image for managed HAProxy load-balancer | HTTPS | TLS + checksum verification |
 | `https://install.mulgadc.com/install` | One-shot install telemetry POST on `spx admin init` / `join`. | HTTPS | TLS |
 

docs/service-interfaces.yaml

@@ -0,0 +1,305 @@
+# Service interface manifest.
+#
+# Source of truth for "which services produce/consume which NATS subjects"
+# and "which e2e suites cover which services". Consumed by
+# tests/e2e/select-suites (Bead 4) to derive targeted suite selection from
+# a PR diff.
+#
+# See docs/development/improvements/e2e-targeted-suite-selection.md.
+#
+# Editing rules
+# -------------
+# - subscribes / publishes: subjects this service receives / sends. Both
+#   lists use the literal subject string, or a "<prefix>.*" wildcard for
+#   pattern subscriptions (e.g. "ebs.snapshot.*").
+# - depends_on: subjects this service requires from peers. Used by the
+#   selector to compute "if X changes, which suites might break".
+# - Subjects must match what is actually referenced in source. The lint
+#   in Bead 5 greps `"(ec2|vpc|ebs|s3|iam|elbv2|spinifex)\.[a-zA-Z._-]+"`
+#   and fails if any subject is absent from this manifest.
+
+version: 1
+
+services:
+
+  awsgw:
+    path: spinifex/services/awsgw
+    # TLS frontend, SigV4 auth. Pure router: HTTP → NATS request/response.
+    # No subscribes; only publishes (via NATSRequest).
+    subscribes: []
+    publishes:
+      - "ec2.*"
+      - "elbv2.*"
+      - "iam.*"
+    depends_on:
+      - "ec2.*"
+      - "elbv2.*"
+      - "iam.*"
+
+  spinifex_daemon:
+    path: spinifex/services/spinifex
+    # Orchestrator. Owns most ec2.* + all elbv2.* + iam event publication.
+    # Translates VPC / EBS / S3 operations into vpc.* / ebs.* / s3.* and
+    # forwards to vpcd / viperblockd / predastore.
+    #
+    # Source layout: spinifex/services/spinifex (entry), spinifex/daemon
+    # (subscribe registration), spinifex/handlers/{ec2,elbv2,iam} (per-
+    # subject handler bodies). Selector treats all four paths as belonging
+    # to this service via the `additional_paths` field.
+    additional_paths:
+      - spinifex/daemon
+      - spinifex/handlers/ec2
+      - spinifex/handlers/elbv2
+      - spinifex/handlers/iam
+    subscribes:
+      # EC2 instance lifecycle.
+      - "ec2.RunInstances"
+      - "ec2.DescribeInstances"
+      - "ec2.DescribeStoppedInstances"
+      - "ec2.DescribeTerminatedInstances"
+      - "ec2.DescribeInstanceAttribute"
+      - "ec2.DescribeInstanceStatus"
+      - "ec2.DescribeInstanceTypes"
+      - "ec2.ModifyInstanceAttribute"
+      - "ec2.ModifyVolume"
+      - "ec2.start"
+      - "ec2.terminate"
+      - "ec2.launch"
+      - "ec2.cmd.*"
+      # EC2 VPC / network ops handled by daemon before delegation to vpcd.
+      - "ec2.CreateVpc"
+      - "ec2.DescribeVpcs"
+      - "ec2.CreateSubnet"
+      - "ec2.CreateInternetGateway"
+      - "ec2.CreateKeyPair"
+      - "ec2.CreateImage"
+      - "ec2.DescribePlacementGroups"
+      # Cluster coordination subjects.
+      - "ec2.ReserveClusterNode"
+      - "ec2.FinalizeClusterInstances"
+      - "spinifex.node.status"
+      - "spinifex.node.vms"
+      - "spinifex.nodes.discover"
+      - "spinifex.cluster.shutdown.progress"
+      - "spinifex.storage.config"
+      # ELBv2 control plane (handlers/elbv2/service_nats.go topic map).
+      - "elbv2.CreateLoadBalancer"
+      - "elbv2.DeleteLoadBalancer"
+      - "elbv2.DescribeLoadBalancers"
+      - "elbv2.CreateTargetGroup"
+      - "elbv2.DeleteTargetGroup"
+      - "elbv2.DescribeTargetGroups"
+      - "elbv2.RegisterTargets"
+      - "elbv2.DeregisterTargets"
+      - "elbv2.DescribeTargetHealth"
+      - "elbv2.CreateListener"
+      - "elbv2.DeleteListener"
+      - "elbv2.DescribeListeners"
+    publishes:
+      # → vpcd
+      - "vpc.create"
+      - "vpc.delete"
+      - "vpc.create-subnet"
+      - "vpc.delete-subnet"
+      - "vpc.create-sg"
+      - "vpc.delete-sg"
+      - "vpc.update-sg"
+      - "vpc.create-port"
+      - "vpc.delete-port"
+      - "vpc.update-port-sgs"
+      - "vpc.igw-attach"
+      - "vpc.igw-detach"
+      - "vpc.add-nat-gateway"
+      - "vpc.delete-nat-gateway"
+      - "vpc.add-nat"
+      - "vpc.delete-nat"
+      - "vpc.dhcp.acquire"
+      - "vpc.dhcp.release"
+      # → viperblockd
+      - "ebs.mount"
+      - "ebs.unmount"
+      - "ebs.delete"
+      - "ebs.sync"
+      - "ebs.snapshot.*"
+      # → predastore (S3 / AMI / keypair store)
+      - "s3.*"
+      # IAM lifecycle event
+      - "iam.account.created"
+    depends_on:
+      - "vpc.*"
+      - "ebs.*"
+      - "s3.*"
+
+  vpcd:
+    path: spinifex/services/vpcd
+    # OVN-backed VPC / subnet / SG / NAT / IGW data plane.
+    subscribes:
+      - "vpc.create"
+      - "vpc.delete"
+      - "vpc.create-subnet"
+      - "vpc.delete-subnet"
+      - "vpc.create-sg"
+      - "vpc.delete-sg"
+      - "vpc.update-sg"
+      - "vpc.create-port"
+      - "vpc.delete-port"
+      - "vpc.update-port-sgs"
+      - "vpc.igw-attach"
+      - "vpc.igw-detach"
+      - "vpc.add-nat-gateway"
+      - "vpc.delete-nat-gateway"
+      - "vpc.add-nat"
+      - "vpc.delete-nat"
+      - "vpc.dhcp.acquire"
+      - "vpc.dhcp.release"
+    publishes:
+      - "vpc.port-status"
+      - "vpc.dhcp.lease-expired"
+    depends_on: []
+
+  viperblockd:
+    path: spinifex/services/viperblockd
+    # EBS data plane. Backed by the viperblock storage engine in the
+    # mulga/viperblock submodule. Submodule pointer bump → selector forces
+    # "run all" (contract changes can be invisible at the spinifex source
+    # level).
+    subscribes:
+      - "ebs.mount"
+      - "ebs.unmount"
+      - "ebs.delete"
+      - "ebs.sync"
+      - "ebs.snapshot.*"
+    publishes:
+      - "ebs.mount.response"
+      - "ebs.unmount.response"
+    depends_on: []
+
+  predastore:
+    path: spinifex/services/predastore
+    # S3 / AMI / keypair storage. External submodule under mulga/predastore;
+    # this dir contains only the spinifex-side adapter. Submodule pointer
+    # bump → selector forces "run all".
+    subscribes:
+      - "s3.*"
+    publishes: []
+    depends_on: []
+
+  nats:
+    path: spinifex/services/nats
+    # NATS broker bootstrap / cluster config. Not a NATS pub/sub
+    # participant. Any change here is infra-path; selector forces
+    # "run all".
+    subscribes: []
+    publishes: []
+    depends_on: []
+
+  spinifexui:
+    path: spinifex/services/spinifexui
+    # Web UI. Talks to awsgw over HTTP, never NATS. Diffs limited to this
+    # dir do not need any e2e suite to run.
+    subscribes: []
+    publishes: []
+    depends_on: []
+    covers_no_suite: true
+
+
+# Fixture → service mapping. Each Ensure* helper in
+# tests/e2e/harness/fixtures.go exercises this set of services; the
+# Bead 4 selector AST-scans test files for harness.Ensure* call sites
+# and resolves coverage via this block.
+fixtures:
+
+  EnsureKeyPair:
+    services: [spinifex_daemon, predastore]
+
+  EnsureAMI:
+    services: [spinifex_daemon, predastore, viperblockd]
+
+  EnsureDefaultVPC:
+    services: [vpcd, spinifex_daemon]
+
+  EnsureSubnet:
+    services: [vpcd, spinifex_daemon]
+
+  EnsureSG:
+    services: [vpcd, spinifex_daemon]
+
+  EnsureInstance:
+    services: [spinifex_daemon, viperblockd, vpcd]
+
+  EnsureVolume:
+    services: [spinifex_daemon, viperblockd]
+
+  EnsureSnapshot:
+    services: [spinifex_daemon, viperblockd, predastore]
+
+  EnsureNATGateway:
+    services: [vpcd, spinifex_daemon]
+
+  EnsureLoadBalancer:
+    services: [spinifex_daemon, awsgw]
+
+
+suites:
+
+  e2e-cert:
+    path: tests/e2e/cert
+    covers: [awsgw, spinifex_daemon, vpcd, predastore]
+
+  e2e-lb:
+    path: tests/e2e/lb
+    covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+    # Exercises full ELBv2 control plane (handlers/elbv2) + data plane
+    # (LB agent on the system VM). ELBv2 is part of spinifex_daemon in
+    # this manifest; revisit if it splits into a standalone service.
+
+  e2e-single:
+    path: tests/e2e/single
+    covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+    # Full single-node lifecycle: VPC, AMI, keypair, instance, volume,
+    # snapshot, IAM. Currently runs all phases sequentially under one
+    # `TestSingleNode`. Bead 3 (single-fixture-migration) splits this
+    # into per-test fixture-driven coverage so the selector can pick
+    # subtests, not just whole suites.
+
+  e2e-ddil:
+    path: tests/e2e/ddil
+    covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+    # DDIL = degraded / disconnected / intermittent / low-bandwidth.
+    # Exercises the full daemon stack under network failure injection;
+    # any service touching the wire path is in scope.
+
+  # Suites below are bash-only today. Entries activate as the Go ports
+  # land (Bead 2 fixtures unblock new-shape ports for multinode etc.).
+  #
+  # e2e-multinode:
+  #   path: spinifex/tests/e2e/multinode
+  #   covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+  # e2e-reboot:
+  #   path: spinifex/tests/e2e/reboot
+  #   covers: [spinifex_daemon, viperblockd]
+  # e2e-tofu:
+  #   path: spinifex/tests/e2e/tofu
+  #   covers: [awsgw, spinifex_daemon, vpcd]
+  # e2e-bm:
+  #   path: spinifex/tests/e2e/baremetal
+  #   covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
+
+
+# Special-case rules consumed by the selector (Bead 4). Hard-coded in
+# the tool, listed here for visibility.
+#
+# infra_paths_force_all:
+#   - spinifex/services/nats/**
+#   - spinifex/tests/e2e/harness/**
+#   - spinifex/tests/e2e/lib/**
+#   - spinifex/.github/workflows/e2e-go.yml
+#   - spinifex/.github/scripts/**
+#   - spinifex/Makefile
+#   - mulga/scripts/tofu-cluster/**
+#
+# submodule_force_all:
+#   - mulga/predastore (submodule pointer)
+#   - mulga/viperblock (submodule pointer)
+#
+# many_services_force_all_threshold: 3   # ≥ N services touched → run all

docs/terraform-workbooks/bastion-private-subnet/main.tf

@@ -96,7 +96,7 @@ data "aws_ami" "ubuntu" {
 
   filter {
     name   = "name"
-    values = ["*ubuntu-24.04*"]
+    values = ["*ubuntu-26.04*", "*ubuntu-24.04*"]
   }
 
   filter {

docs/terraform-workbooks/nginx-alb/main.tf

@@ -102,7 +102,7 @@ data "aws_ami" "ubuntu" {
 
   filter {
     name   = "name"
-    values = ["*ubuntu-24.04*"]
+    values = ["*ubuntu-26.04*", "*ubuntu-24.04*"]
   }
 }
 

docs/terraform-workbooks/nginx-webserver/main.tf

@@ -85,7 +85,7 @@ data "aws_ami" "ubuntu" {
 
   filter {
     name   = "name"
-    values = ["*ubuntu-24.04*"]
+    values = ["*ubuntu-26.04*", "*ubuntu-24.04*"]
   }
 
   filter {

docs/terraform-workbooks/s3-webapp/main.tf

@@ -132,7 +132,7 @@ data "aws_ami" "ubuntu" {
 
   filter {
     name   = "name"
-    values = ["*ubuntu-24.04*"]
+    values = ["*ubuntu-26.04*", "*ubuntu-24.04*"]
   }
 
   filter {

go.mod

@@ -7,14 +7,14 @@ require (
 	github.com/charmbracelet/bubbles v1.0.0
 	github.com/charmbracelet/bubbletea v1.3.10
 	github.com/charmbracelet/lipgloss v1.1.0
-	github.com/go-chi/chi/v5 v5.2.5
+	github.com/go-chi/chi/v5 v5.3.0
 	github.com/google/uuid v1.6.0
 	github.com/insomniacslk/dhcp v0.0.0-20260407060928-11b94ed970f2
 	github.com/kdomanski/iso9660 v0.4.0
 	github.com/klauspost/cpuid/v2 v2.3.0
-	github.com/mulgadc/predastore v1.4.0
-	github.com/mulgadc/viperblock v1.4.0
-	github.com/nats-io/nats-server/v2 v2.14.0
+	github.com/mulgadc/predastore v1.5.0
+	github.com/mulgadc/viperblock v1.5.0
+	github.com/nats-io/nats-server/v2 v2.14.1
 	github.com/nats-io/nats.go v1.52.0
 	github.com/ovn-kubernetes/libovsdb v0.8.1
 	github.com/pelletier/go-toml/v2 v2.3.1
@@ -23,8 +23,10 @@ require (
 	github.com/spf13/viper v1.21.0
 	github.com/stretchr/testify v1.11.1
 	go.uber.org/goleak v1.3.0
-	golang.org/x/crypto v0.51.0
+	golang.org/x/crypto v0.52.0
+	golang.org/x/sync v0.20.0
 	gopkg.in/ini.v1 v1.67.2
+	gopkg.in/yaml.v3 v3.0.1
 )
 
 require (
@@ -34,6 +36,8 @@ require (
 	github.com/antithesishq/antithesis-sdk-go v0.7.0 // indirect
 	github.com/armon/go-metrics v0.4.1 // indirect
 	github.com/atotto/clipboard v0.1.4 // indirect
+	github.com/aws/aws-sdk-go-v2 v1.41.7 // indirect
+	github.com/aws/smithy-go v1.25.1 // indirect
 	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/boltdb/bolt v1.3.1 // indirect
@@ -78,7 +82,7 @@ require (
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
 	github.com/jmespath/go-jmespath v0.4.0 // indirect
 	github.com/josharian/native v1.1.0 // indirect
-	github.com/klauspost/compress v1.18.5 // indirect
+	github.com/klauspost/compress v1.18.6 // indirect
 	github.com/klauspost/reedsolomon v1.14.0 // indirect
 	github.com/leodido/go-urn v1.4.0 // indirect
 	github.com/lithammer/fuzzysearch v1.1.8 // indirect
@@ -121,9 +125,8 @@ require (
 	go.yaml.in/yaml/v2 v2.4.4 // indirect
 	go.yaml.in/yaml/v3 v3.0.4 // indirect
 	golang.org/x/mod v0.35.0 // indirect
-	golang.org/x/net v0.54.0 // indirect
-	golang.org/x/sync v0.20.0 // indirect
-	golang.org/x/sys v0.44.0 // indirect
+	golang.org/x/net v0.55.0 // indirect
+	golang.org/x/sys v0.45.0 // indirect
 	golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa // indirect
 	golang.org/x/term v0.43.0 // indirect
 	golang.org/x/text v0.37.0 // indirect
@@ -131,7 +134,6 @@ require (
 	golang.org/x/tools v0.44.0 // indirect
 	golang.org/x/vuln v1.1.4 // indirect
 	google.golang.org/protobuf v1.36.11 // indirect
-	gopkg.in/yaml.v3 v3.0.1 // indirect
 )
 
 tool golang.org/x/vuln/cmd/govulncheck

go.sum

@@ -122,8 +122,8 @@ github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S
 github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
 github.com/gabriel-vasile/mimetype v1.4.13 h1:46nXokslUBsAJE/wMsp5gtO500a4F3Nkz9Ufpk2AcUM=
 github.com/gabriel-vasile/mimetype v1.4.13/go.mod h1:d+9Oxyo1wTzWdyVUPMmXFvp4F9tea18J8ufA774AB3s=
-github.com/go-chi/chi/v5 v5.2.5 h1:Eg4myHZBjyvJmAFjFvWgrqDTXFyOzjj7YIm3L3mu6Ug=
-github.com/go-chi/chi/v5 v5.2.5/go.mod h1:X7Gx4mteadT3eDOMTsXzmI4/rwUpOwBHLpAfupzFJP0=
+github.com/go-chi/chi/v5 v5.3.0 h1:halUjDxhshgXHMrao5bB8eNBXo/rnzwr8m5m36glehM=
+github.com/go-chi/chi/v5 v5.3.0/go.mod h1:R+tYY2hNuVUUjxoPtqUdgBqevM9s9njzkTLutVsOCto=
 github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
 github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
 github.com/go-kit/log v0.1.0/go.mod h1:zbhenjAZHb184qTLMA9ZjW7ThYL0H2mk7Q6pNt4vbaY=
@@ -229,8 +229,8 @@ github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7V
 github.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM=
 github.com/kdomanski/iso9660 v0.4.0 h1:BPKKdcINz3m0MdjIMwS0wx1nofsOjxOq8TOr45WGHFg=
 github.com/kdomanski/iso9660 v0.4.0/go.mod h1:OxUSupHsO9ceI8lBLPJKWBTphLemjrCQY8LPXM7qSzU=
-github.com/klauspost/compress v1.18.5 h1:/h1gH5Ce+VWNLSWqPzOVn6XBO+vJbCNGvjoaGBFW2IE=
-github.com/klauspost/compress v1.18.5/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
+github.com/klauspost/compress v1.18.6 h1:2jupLlAwFm95+YDR+NwD2MEfFO9d4z4Prjl1XXDjuao=
+github.com/klauspost/compress v1.18.6/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
 github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
 github.com/klauspost/cpuid/v2 v2.0.10/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuObKfj5c0PQa7c=
 github.com/klauspost/cpuid/v2 v2.0.12/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuObKfj5c0PQa7c=
@@ -286,18 +286,18 @@ github.com/muesli/cancelreader v0.2.2 h1:3I4Kt4BQjOR54NavqnDogx/MIoWBFa0StPA8ELU
 github.com/muesli/cancelreader v0.2.2/go.mod h1:3XuTXfFS2VjM+HTLZY9Ak0l6eUKfijIfMUZ4EgX0QYo=
 github.com/muesli/termenv v0.16.0 h1:S5AlUN9dENB57rsbnkPyfdGuWIlkmzJjbFf0Tf5FWUc=
 github.com/muesli/termenv v0.16.0/go.mod h1:ZRfOIKPFDYQoDFF4Olj7/QJbW60Ol/kL1pU3VfY/Cnk=
-github.com/mulgadc/predastore v1.4.0 h1:txPZCaVT1cDpBotx8zHPXHZD3Mz62+n207zXISkK4uU=
-github.com/mulgadc/predastore v1.4.0/go.mod h1:fXIGz7yCKcHh68ybcw7aigdOdzIfyt19h/0diPF+JCU=
-github.com/mulgadc/viperblock v1.4.0 h1:3Yr6lRDnFgxkdMOWu+Tq1b97qWHQDNwkPWmCp38CIy8=
-github.com/mulgadc/viperblock v1.4.0/go.mod h1:hHdfo02iJgCdNktiPAq+x+VQE0ZNYOLJXw0abJ81LeE=
+github.com/mulgadc/predastore v1.5.0 h1:FtL9EquoGKl3tcIRIoAXGhQStkN4uPDx0TOQZ89Mh+c=
+github.com/mulgadc/predastore v1.5.0/go.mod h1:fXIGz7yCKcHh68ybcw7aigdOdzIfyt19h/0diPF+JCU=
+github.com/mulgadc/viperblock v1.5.0 h1:2OprAI1oAtitUosS5A9LbenU/RCaw0oxUJvUhv8+tgU=
+github.com/mulgadc/viperblock v1.5.0/go.mod h1:lIuOaBO3gC0o/EwWhJPpCuhUEPQ3/y74xckGlcBKaos=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
 github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
 github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
 github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
-github.com/nats-io/nats-server/v2 v2.14.0 h1:+8q0HrDFotwLLcGH/legOEOnowunhK+aZ4GYBIWpQlM=
-github.com/nats-io/nats-server/v2 v2.14.0/go.mod h1:ImVUUDvfClJbb6cuJQRc1VmgDCXKM5ds0OoiG9MVOKo=
+github.com/nats-io/nats-server/v2 v2.14.1 h1:wXs/a5fw9Hzm3CvuzLxGeIwpjPulSa7gMT3eSuhGkcg=
+github.com/nats-io/nats-server/v2 v2.14.1/go.mod h1:4N17zLpuS7WMbG8T9gsE2B7z9hC9PraPyulVBfpK6nU=
 github.com/nats-io/nats.go v1.52.0 h1:n3avV4VBsCgsdwh71TppsTwtv+QdPs7ntSKM8qJLGsc=
 github.com/nats-io/nats.go v1.52.0/go.mod h1:26HypzazeOkyO3/mqd1zZd53STJN0EjCYF9Uy2ZOBno=
 github.com/nats-io/nkeys v0.4.15 h1:JACV5jRVO9V856KOapQ7x+EY8Jo3qw1vJt/9Jpwzkk4=
@@ -424,8 +424,8 @@ golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnf
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
-golang.org/x/crypto v0.51.0 h1:IBPXwPfKxY7cWQZ38ZCIRPI50YLeevDLlLnyC5wRGTI=
-golang.org/x/crypto v0.51.0/go.mod h1:8AdwkbraGNABw2kOX6YFPs3WM22XqI4EXEd8g+x7Oc8=
+golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988=
+golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc=
 golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa h1:FRnLl4eNAQl8hwxVVC17teOw8kdjVDVAiFMtgUdTSRQ=
 golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa/go.mod h1:zk2irFbV9DP96SEBUUAy67IdHUaZuSnrz1n472HUCLE=
 golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
@@ -442,8 +442,8 @@ golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81R
 golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
 golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
 golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
-golang.org/x/net v0.54.0 h1:2zJIZAxAHV/OHCDTCOHAYehQzLfSXuf/5SoL/Dv6w/w=
-golang.org/x/net v0.54.0/go.mod h1:Sj4oj8jK6XmHpBZU/zWHw3BV3abl4Kvi+Ut7cQcY+cQ=
+golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8=
+golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -484,8 +484,8 @@ golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
-golang.org/x/sys v0.44.0 h1:ildZl3J4uzeKP07r2F++Op7E9B29JRUy+a27EibtBTQ=
-golang.org/x/sys v0.44.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
+golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
+golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
 golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa h1:efT73AJZfAAUV7SOip6pWGkwJDzIGiKBZGVzHYa+ve4=
 golang.org/x/telemetry v0.0.0-20260409153401-be6f6cb8b1fa/go.mod h1:kHjTxDEnAu6/Nl9lDkzjWpR+bmKfxeiRuSDlsMb70gE=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=

internal/fipsboot/fipsboot.go

@@ -11,7 +11,6 @@ package fipsboot
 
 import (
 	"crypto/fips140"
-	"log/slog"
 )
 
 func init() {
@@ -19,8 +18,4 @@ func init() {
 		panic("fipsboot: FIPS 140-3 mode is not enabled at runtime — refusing to start. " +
 			"Build with GOFIPS140=v1.0.0 and do not set GODEBUG=fips140=off.")
 	}
-	slog.Info("FIPS 140-3 mode enabled",
-		"module", "Go Cryptographic Module v1.0.0",
-		"cmvp_cert", "5247",
-	)
 }

internal/tlsconfig/curves.go

@@ -0,0 +1,37 @@
+// Package tlsconfig pins the cluster-wide TLS 1.3 curve preference list.
+//
+// The list is hardcoded because mixed classification on a single cluster is
+// not supported and the certified FIPS build is itself the per-deployment
+// choice. A typo in a config string would silently downgrade where a missing
+// constant cannot.
+package tlsconfig
+
+import "crypto/tls"
+
+// Curves is the fixed TLS 1.3 curve allowlist:
+//
+//   - X25519MLKEM768     — Cat 3 ML-KEM hybrid, wins under Go 1.26 stdlib filter
+//   - SecP384r1MLKEM1024 — Cat 5 ML-KEM hybrid, fallback for peers that prefer it
+//   - X25519             — classical TLS 1.3 fallback for SDKs without ML-KEM (aws CLI v2 today)
+//   - CurveP384          — classical TLS 1.3 fallback (Java/.NET enterprise SDKs)
+//
+// PQ-capable peers (browsers since 2024, Go aws-sdk v2 with Go 1.24+) always
+// negotiate the hybrid first; classical fallback only triggers for peers that
+// cannot do ML-KEM. Those connections still get TLS 1.3 + AES-256-GCM but
+// remain HNDL-exposed — accepted trade-off until AWS CLI v2 / bundled
+// OpenSSL ships ML-KEM (currently lagging upstream OpenSSL 3.5).
+//
+// Weak/niche entries from the Go stdlib default (SecP256r1MLKEM768, P-256,
+// P-521) are intentionally excluded so they cannot be advertised.
+//
+// Go 1.26 stdlib treats tls.Config.CurvePreferences as an allowlist filter
+// applied to the stdlib default order, not as a user-supplied order.
+//
+// Callers must not mutate this slice. Assign it directly to
+// tls.Config.CurvePreferences; do not append to it.
+var Curves = []tls.CurveID{
+	tls.X25519MLKEM768,
+	tls.SecP384r1MLKEM1024,
+	tls.X25519,
+	tls.CurveP384,
+}

internal/tlsconfig/curves_test.go

@@ -0,0 +1,169 @@
+package tlsconfig_test
+
+import (
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/tls"
+	"crypto/x509"
+	"crypto/x509/pkix"
+	"encoding/pem"
+	"io"
+	"math/big"
+	"net"
+	"net/http"
+	"net/http/httptest"
+	"slices"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/mulgadc/spinifex/internal/tlsconfig"
+)
+
+func TestCurves_OnlyApprovedEntries(t *testing.T) {
+	want := []tls.CurveID{
+		tls.X25519MLKEM768,
+		tls.SecP384r1MLKEM1024,
+		tls.X25519,
+		tls.CurveP384,
+	}
+	if !slices.Equal(tlsconfig.Curves, want) {
+		t.Errorf("Curves = %v, want %v", tlsconfig.Curves, want)
+	}
+}
+
+func TestCurves_ExcludesWeakPrimitives(t *testing.T) {
+	forbidden := []tls.CurveID{
+		tls.SecP256r1MLKEM768, // weakest ML-KEM hybrid (P-256 component)
+		tls.CurveP256,         // not CNSA 2.0
+		tls.CurveP521,         // niche, no current peers
+	}
+	for _, c := range forbidden {
+		if slices.Contains(tlsconfig.Curves, c) {
+			t.Errorf("Curves includes forbidden curve %v", c)
+		}
+	}
+}
+
+// TestIntegration_NegotiatesPQHybrid spins up an httptest TLS server
+// configured with Curves + TLS 1.3 minimum, dials it, and asserts a PQ
+// hybrid is negotiated at TLS 1.3. Per Go 1.26 stdlib filter behavior the
+// winner is X25519MLKEM768; the assertion checks for any approved hybrid
+// to remain stable across stdlib reorderings of our 4 entries.
+func TestIntegration_NegotiatesPQHybrid(t *testing.T) {
+	cert := selfSignedCert(t)
+	srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = io.WriteString(w, "ok")
+	}))
+	srv.TLS = &tls.Config{
+		Certificates:     []tls.Certificate{cert},
+		MinVersion:       tls.VersionTLS13,
+		CurvePreferences: tlsconfig.Curves,
+	}
+	srv.StartTLS()
+	defer srv.Close()
+
+	pool := x509.NewCertPool()
+	pool.AddCert(srv.Certificate())
+
+	client := &http.Client{
+		Timeout: 5 * time.Second,
+		Transport: &http.Transport{
+			TLSClientConfig: &tls.Config{
+				RootCAs:          pool,
+				CurvePreferences: tlsconfig.Curves,
+			},
+		},
+	}
+	resp, err := client.Get(srv.URL)
+	if err != nil {
+		t.Fatalf("https get: %v", err)
+	}
+	defer resp.Body.Close()
+
+	state := resp.TLS
+	if state == nil {
+		t.Fatal("response has no TLS state")
+	}
+	if state.Version != tls.VersionTLS13 {
+		t.Errorf("negotiated version = 0x%x, want TLS 1.3 (0x%x)", state.Version, tls.VersionTLS13)
+	}
+	pqHybrids := []tls.CurveID{tls.X25519MLKEM768, tls.SecP384r1MLKEM1024}
+	if !slices.Contains(pqHybrids, state.CurveID) {
+		t.Errorf("negotiated curve = %v, want one of %v (PQ hybrid)", state.CurveID, pqHybrids)
+	}
+}
+
+// TestIntegration_TLS12ClientRejected confirms MinVersion: tls.VersionTLS13
+// is enforced — a TLS 1.2-pinned client gets a handshake error rather than
+// silently downgrading. This is the negative test that closes the HNDL gap
+// on external surfaces.
+func TestIntegration_TLS12ClientRejected(t *testing.T) {
+	cert := selfSignedCert(t)
+	srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = io.WriteString(w, "ok")
+	}))
+	srv.TLS = &tls.Config{
+		Certificates:     []tls.Certificate{cert},
+		MinVersion:       tls.VersionTLS13,
+		CurvePreferences: tlsconfig.Curves,
+	}
+	srv.StartTLS()
+	defer srv.Close()
+
+	pool := x509.NewCertPool()
+	pool.AddCert(srv.Certificate())
+
+	client := &http.Client{
+		Timeout: 5 * time.Second,
+		Transport: &http.Transport{
+			TLSClientConfig: &tls.Config{
+				RootCAs:    pool,
+				MinVersion: tls.VersionTLS12,
+				MaxVersion: tls.VersionTLS12,
+			},
+		},
+	}
+	_, err := client.Get(srv.URL)
+	if err == nil {
+		t.Fatal("TLS 1.2-pinned client succeeded; MinVersion not enforced")
+	}
+	if !strings.Contains(err.Error(), "protocol version") &&
+		!strings.Contains(err.Error(), "tls:") {
+		t.Errorf("expected protocol-version handshake error, got: %v", err)
+	}
+}
+
+func selfSignedCert(t *testing.T) tls.Certificate {
+	t.Helper()
+	key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+	if err != nil {
+		t.Fatalf("generate key: %v", err)
+	}
+	tmpl := &x509.Certificate{
+		SerialNumber: big.NewInt(1),
+		Subject:      pkix.Name{CommonName: "tlsconfig-test"},
+		NotBefore:    time.Now().Add(-time.Hour),
+		NotAfter:     time.Now().Add(time.Hour),
+		KeyUsage:     x509.KeyUsageDigitalSignature,
+		ExtKeyUsage:  []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
+		DNSNames:     []string{"localhost"},
+		IPAddresses:  []net.IP{net.ParseIP("127.0.0.1"), net.ParseIP("::1")},
+	}
+	der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key)
+	if err != nil {
+		t.Fatalf("create cert: %v", err)
+	}
+	keyDER, err := x509.MarshalECPrivateKey(key)
+	if err != nil {
+		t.Fatalf("marshal key: %v", err)
+	}
+	certPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
+	keyPEM := pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER})
+	pair, err := tls.X509KeyPair(certPEM, keyPEM)
+	if err != nil {
+		t.Fatalf("X509KeyPair: %v", err)
+	}
+	return pair
+}

scripts/ansible/roles/smoketest/tasks/main.yml

@@ -61,6 +61,7 @@
     --distro {{ spinifex_smoketest_distro }}
     --version {{ spinifex_smoketest_version }}
     --arch {{ smoketest_arch }}
+    --boot-mode uefi
   register: smoketest_import_local
   until: >-
     smoketest_import_local.rc == 0 or

scripts/build-microvm-image.sh

@@ -35,9 +35,15 @@ done
 
 # --- Create chroot ---
 CHROOT_DIR=$(mktemp -d)
+CONTAINER_TOOL=""
+CONTAINER_CID=""
 cleanup() {
     echo "[build-microvm-image] cleaning up chroot: $CHROOT_DIR"
     rm -rf "$CHROOT_DIR"
+    if [ -n "$CONTAINER_CID" ] && [ -n "$CONTAINER_TOOL" ]; then
+        echo "[build-microvm-image] cleaning up container: $CONTAINER_CID"
+        $CONTAINER_TOOL rm -f "$CONTAINER_CID" >/dev/null 2>&1 || true
+    fi
 }
 trap cleanup EXIT
 
@@ -63,7 +69,6 @@ EOF
 
 # Strategy 2/3: container runtime (dev machines)
 else
-    CONTAINER_TOOL=""
     if command -v docker >/dev/null 2>&1 && docker info >/dev/null 2>&1; then
         CONTAINER_TOOL=docker
     elif command -v podman >/dev/null 2>&1; then
@@ -75,19 +80,17 @@ else
     fi
 
     echo "[build-microvm-image] rootfs: $CONTAINER_TOOL export (alpine:${ALPINE_VERSION})"
-    cid=$($CONTAINER_TOOL run -d \
+    CONTAINER_CID=$($CONTAINER_TOOL run -d \
         "alpine:${ALPINE_VERSION}" \
         sh -c "apk add --no-cache ${ALPINE_PACKAGES}")
 
-    exit_code=$($CONTAINER_TOOL wait "$cid")
+    exit_code=$($CONTAINER_TOOL wait "$CONTAINER_CID")
     if [ "$exit_code" != "0" ]; then
         echo "ERROR: package installation failed in container (exit $exit_code)" >&2
-        $CONTAINER_TOOL rm "$cid" >/dev/null 2>&1 || true
         exit 1
     fi
 
-    $CONTAINER_TOOL export "$cid" | tar -x -C "$CHROOT_DIR"
-    $CONTAINER_TOOL rm "$cid" >/dev/null 2>&1
+    $CONTAINER_TOOL export "$CONTAINER_CID" | tar -x -C "$CHROOT_DIR"
 fi
 
 # --- Fix /dev device nodes ---

scripts/build-system-image.sh

@@ -75,6 +75,7 @@ case "$DISTRO" in
         OUTPUT_RAW="${BUILD_DIR}/${IMAGE_NAME}-alpine.raw"
         DISTRO_VERSION="${ALPINE_VERSION}"
         ROOT_PART="${NBD_DEV}"
+        BOOT_MODE="bios"
         ;;
     ubuntu)
         if [[ -z "${UBUNTU_VERSION:-}" ]]; then
@@ -87,6 +88,7 @@ case "$DISTRO" in
         OUTPUT_RAW="${BUILD_DIR}/${IMAGE_NAME}-ubuntu.raw"
         DISTRO_VERSION="${UBUNTU_VERSION}"
         ROOT_PART="${NBD_DEV}p1"
+        BOOT_MODE="uefi"
         ;;
     *)
         echo "ERROR: Unknown DISTRO: $DISTRO (supported: alpine, ubuntu)"
@@ -189,7 +191,7 @@ if [[ -f "$OUTPUT_RAW" ]] && [[ $(( $(date +%s) - $(stat -c %Y "$OUTPUT_RAW") ))
     if [[ "$DO_IMPORT" == true ]]; then
         echo "Importing as AMI..."
         rm -f "$OUTPUT_IMAGE"
-        IMPORT_ARGS=(--file "$OUTPUT_RAW" --distro "${DISTRO}" --version "${DISTRO_VERSION}" --arch x86_64)
+        IMPORT_ARGS=(--file "$OUTPUT_RAW" --distro "${DISTRO}" --version "${DISTRO_VERSION}" --arch x86_64 --boot-mode "${BOOT_MODE}")
         if [[ -n "${SYSTEM_TAG:-}" ]]; then
             IMPORT_ARGS+=(--tag "$SYSTEM_TAG")
         fi
@@ -423,7 +425,7 @@ echo ""
 if [[ "$DO_IMPORT" == true ]]; then
     echo "Importing as AMI..."
     rm -f "$OUTPUT_IMAGE"
-    IMPORT_ARGS=(--file "$OUTPUT_RAW" --distro "${DISTRO}" --version "${DISTRO_VERSION}" --arch x86_64)
+    IMPORT_ARGS=(--file "$OUTPUT_RAW" --distro "${DISTRO}" --version "${DISTRO_VERSION}" --arch x86_64 --boot-mode "${BOOT_MODE}")
     if [[ -n "${SYSTEM_TAG:-}" ]]; then
         IMPORT_ARGS+=(--tag "$SYSTEM_TAG")
     fi
@@ -433,9 +435,9 @@ else
     echo "  spx admin images import \\"
     echo "    --file $OUTPUT_RAW \\"
     if [[ -n "${SYSTEM_TAG:-}" ]]; then
-        echo "    --distro ${DISTRO} --version ${DISTRO_VERSION} --arch x86_64 \\"
+        echo "    --distro ${DISTRO} --version ${DISTRO_VERSION} --arch x86_64 --boot-mode ${BOOT_MODE} \\"
         echo "    --tag ${SYSTEM_TAG}"
     else
-        echo "    --distro ${DISTRO} --version ${DISTRO_VERSION} --arch x86_64"
+        echo "    --distro ${DISTRO} --version ${DISTRO_VERSION} --arch x86_64 --boot-mode ${BOOT_MODE}"
     fi
 fi

scripts/dev-install.sh

@@ -79,10 +79,6 @@ SPINIFEX_AZ="${SPINIFEX_AZ:-${SPINIFEX_REGION}a}"
 echo "=== Initializing (region=$SPINIFEX_REGION az=$SPINIFEX_AZ) ==="
 sudo spx admin init --force --region "$SPINIFEX_REGION" --az "$SPINIFEX_AZ" --node node1 --nodes 1
 
-echo "=== Installing CA certificate ==="
-sudo cp /etc/spinifex/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
-sudo update-ca-certificates
-
 echo "=== Starting services ==="
 sudo systemctl start spinifex.target
 

scripts/iac/proxmox/README.md

@@ -8,9 +8,9 @@ OpenTofu/Terraform templates for provisioning Spinifex development clusters on P
 - 1 or more Proxmox VE nodes with:
   - A Terraform API token (`Datacenter > Permissions > API Tokens`)
   - SSH access for a `terraform` user on each node
-  - Debian 12 cloud image uploaded to `local:iso/` on each node
+  - Debian 13 cloud image uploaded to `local:iso/` on each node
 - SSH keypair for cloud-init VM access
-- A hardcoded Debian ISO is used, which needs to be pre-installed on each Proxmox host ("debian-12-genericcloud-amd64-20240211-1654.img") https://cdimage.debian.org/images/cloud/bookworm/20240211-1654/debian-12-genericcloud-amd64-20240211-1654.qcow2
+- A hardcoded Debian ISO is used, which needs to be pre-installed on each Proxmox host ("debian-13-genericcloud-amd64-20260518-2482.img") https://cloud.debian.org/images/cloud/trixie/20260518-2482/debian-13-genericcloud-amd64-20260518-2482.qcow2
 
 ## Quick Start (spinifex-test.sh)
 

scripts/images/lb-setup.sh

@@ -1,55 +0,0 @@
-#!/bin/sh
-# lb-setup.sh — LB-specific chroot setup
-#
-# Runs inside the Alpine image chroot after packages are installed.
-# Sets up HAProxy placeholder config and lb-agent OpenRC init script.
-# The lb-agent polls the AWS gateway for config updates and reports health
-# via SigV4-signed HTTP requests. Credentials come from cloud-init env vars.
-
-set -e
-
-# Create haproxy config directory and placeholder config.
-mkdir -p /etc/haproxy
-cat > /etc/haproxy/haproxy.cfg <<'EOF'
-# Placeholder config — replaced by lb-agent on first config fetch
-global
-    daemon
-    maxconn 256
-
-defaults
-    mode http
-    timeout connect 5s
-    timeout client 30s
-    timeout server 30s
-EOF
-
-# Create lb-agent OpenRC init script
-mkdir -p /etc/init.d
-cat > /etc/init.d/lb-agent <<'INITSCRIPT'
-#!/sbin/openrc-run
-
-description="LB Gateway Config Agent"
-command="/usr/local/bin/lb-agent"
-command_args="--lb-id=${LB_LB_ID:-unknown} --gateway=${LB_GATEWAY_URL} --access-key=${LB_ACCESS_KEY} --secret-key=${LB_SECRET_KEY}"
-command_background=true
-pidfile="/run/lb-agent.pid"
-output_log="/var/log/lb-agent.log"
-error_log="/var/log/lb-agent.log"
-
-depend() {
-    need net
-    after firewall
-}
-INITSCRIPT
-chmod 755 /etc/init.d/lb-agent
-
-# Move networking from boot to default runlevel so cloud-init can write
-# the network config before the networking service starts. Without this,
-# Alpine's networking service races cloud-init and starts with no config.
-rc-update del networking boot 2>/dev/null || true
-rc-update add networking default
-
-# Do NOT enable lb-agent at boot — cloud-init must write
-# /etc/conf.d/lb-agent (with LB_LB_ID, LB_GATEWAY_URL, LB_ACCESS_KEY,
-# LB_SECRET_KEY) before the service starts.
-# Cloud-init runcmd starts the service after write_files.

scripts/images/lb.conf

@@ -1,26 +0,0 @@
-# LB system image manifest
-# Used by build-system-image.sh to create an Alpine AMI with HAProxy + lb-agent
-
-IMAGE_NAME="lb"
-IMAGE_DESCRIPTION="LB VM with HAProxy and lb-agent"
-ALPINE_VERSION="3.21.6"
-IMAGE_SIZE="256M"
-
-# Tag applied to the imported AMI so the daemon can discover this as the
-# system image for ELBv2 load balancers. Must match tags.ManagedByELBv2.
-SYSTEM_TAG="spinifex:managed-by=elbv2"
-
-# Packages to install via apk
-APK_PACKAGES="haproxy curl ca-certificates"
-
-# Binaries to copy into the image (source:destination)
-INSTALL_BINARIES="bin/lb-agent:/usr/local/bin/lb-agent"
-
-# Build command for binaries (run before image build)
-BUILD_COMMANDS="make build-lb-agent"
-
-# OpenRC services to enable at boot
-ENABLE_SERVICES="haproxy"
-
-# Custom setup script (runs inside chroot after packages are installed)
-SETUP_SCRIPT="scripts/images/lb-setup.sh"

scripts/setup.sh

@@ -214,7 +214,7 @@ install_apt_deps() {
 
         DEBIAN_FRONTEND=noninteractive $SUDO apt-get install -y -qq \
             nbdkit \
-            $QEMU_PACKAGES qemu-utils qemu-kvm ovmf less \
+            $QEMU_PACKAGES qemu-utils qemu-kvm ovmf qemu-efi-aarch64 less \
             libvirt-daemon-system libvirt-clients \
             pciutils \
             jq curl iproute2 netcat-openbsd wget unzip xz-utils file \

scripts/smoke-test.sh

@@ -126,7 +126,7 @@ else
     if [ -f "$LOCAL_IMAGE" ]; then
         echo "  Using local image: $LOCAL_IMAGE"
         sudo /usr/local/bin/spx admin images import \
-            --file "$LOCAL_IMAGE" --distro ubuntu --version 26.04 --arch "$IMG_ARCH"
+            --file "$LOCAL_IMAGE" --distro ubuntu --version 26.04 --arch "$IMG_ARCH" --boot-mode uefi
     else
         echo "  Downloading image: $IMAGE_NAME"
         sudo /usr/local/bin/spx admin images import --name "$IMAGE_NAME"

spinifex/admin/images_remove.go

@@ -0,0 +1,479 @@
+package admin
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"log/slog"
+	"strings"
+	"time"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/service/s3"
+	"github.com/mulgadc/spinifex/spinifex/awserrors"
+	handlers_ec2_image "github.com/mulgadc/spinifex/spinifex/handlers/ec2/image"
+	handlers_ec2_snapshot "github.com/mulgadc/spinifex/spinifex/handlers/ec2/snapshot"
+	"github.com/mulgadc/spinifex/spinifex/objectstore"
+	"github.com/mulgadc/spinifex/spinifex/utils"
+	"github.com/mulgadc/viperblock/viperblock"
+)
+
+// RemoveImageOpts configures RemoveSystemImage.
+type RemoveImageOpts struct {
+	ImageID string
+	// Force bypasses the dependency check, the ownership-shape check, and
+	// the "config.json missing/corrupt" check. Salvage-mode lever.
+	Force bool
+}
+
+// RemoveImageResult summarises what was deleted from object storage.
+type RemoveImageResult struct {
+	ObjectsDeleted int
+	BytesFreed     int64
+}
+
+// Dependents lists every resource that transitively backs an admin-imported
+// AMI. Removing the AMI while any of these exist would corrupt the dependent.
+type Dependents struct {
+	Snapshots []string // EC2 snapshots whose VolumeID == the AMI ID (i.e. CopyImage-derived snaps)
+	Volumes   []string // Volumes whose SnapshotID is snap-ami-<id> or a derived snap
+	AMIs      []string // AMIs whose SnapshotID is a derived snap (CopyImage of a system AMI then re-copy)
+}
+
+// Empty is true when nothing depends on the AMI.
+func (d Dependents) Empty() bool {
+	return len(d.Snapshots) == 0 && len(d.Volumes) == 0 && len(d.AMIs) == 0
+}
+
+// RemovePreview captures everything the CLI needs to render the confirmation
+// prompt and decide whether to proceed: AMI metadata (or zero values when the
+// config is missing/corrupt), backing-storage byte counts, and the dependent
+// resource set. PreviewRemoveSystemImage performs no deletions.
+type RemovePreview struct {
+	ImageID       string
+	Name          string
+	Owner         string
+	Created       time.Time
+	ConfigPresent bool // false when ami-<id>/config.json is missing
+	ConfigCorrupt bool // true when config.json exists but is undecodable
+	IsSystemOwned bool // ImageOwnerAlias is set and not an account ID
+
+	AMIObjectCount  int
+	AMIBytesTotal   int64
+	SnapObjectCount int
+	SnapBytesTotal  int64
+
+	Dependents Dependents
+}
+
+// SnapPrefix returns the viperblock-internal snapshot prefix backing an
+// admin-imported AMI. v_utils.ImportDiskImage writes the snapshot as
+// "snap-<volumeName>" where the volume name IS the AMI ID.
+func SnapPrefix(imageID string) string {
+	return "snap-" + imageID
+}
+
+// PreviewRemoveSystemImage walks the object store and gathers everything the
+// caller needs to decide whether to remove an admin-imported AMI. It performs
+// no mutations. Returned errors indicate unexpected I/O failures only —
+// AMI-not-found, corrupt config, and account-owned AMIs are reflected in the
+// preview fields, not as errors.
+func PreviewRemoveSystemImage(store objectstore.ObjectStore, bucket, imageID string) (*RemovePreview, error) {
+	if !strings.HasPrefix(imageID, "ami-") {
+		return nil, errors.New(awserrors.ErrorInvalidAMIIDMalformed)
+	}
+
+	preview := &RemovePreview{ImageID: imageID}
+
+	meta, configErr := readAMIConfig(store, bucket, imageID)
+	switch {
+	case configErr == nil:
+		preview.ConfigPresent = true
+		preview.Name = meta.Name
+		preview.Owner = meta.ImageOwnerAlias
+		preview.Created = meta.CreationDate
+		preview.IsSystemOwned = meta.ImageOwnerAlias != "" && !utils.IsAccountID(meta.ImageOwnerAlias)
+	case objectstore.IsNoSuchKeyError(configErr):
+		// Config absent — salvage candidate. Leave ConfigPresent=false.
+	case errors.Is(configErr, handlers_ec2_image.ErrCorruptAMIConfig):
+		preview.ConfigCorrupt = true
+	default:
+		return nil, fmt.Errorf("preview: read AMI config: %w", configErr)
+	}
+
+	amiCount, amiBytes, err := sumPrefix(store, bucket, imageID+"/")
+	if err != nil {
+		return nil, fmt.Errorf("preview: sum ami prefix: %w", err)
+	}
+	preview.AMIObjectCount = amiCount
+	preview.AMIBytesTotal = amiBytes
+
+	snapCount, snapBytes, err := sumPrefix(store, bucket, SnapPrefix(imageID)+"/")
+	if err != nil {
+		return nil, fmt.Errorf("preview: sum snap prefix: %w", err)
+	}
+	preview.SnapObjectCount = snapCount
+	preview.SnapBytesTotal = snapBytes
+
+	deps, err := FindAMIDependents(store, bucket, imageID)
+	if err != nil {
+		return nil, fmt.Errorf("preview: find dependents: %w", err)
+	}
+	preview.Dependents = deps
+
+	return preview, nil
+}
+
+// FindAMIDependents walks the bucket to find every resource that transitively
+// depends on the given admin-imported AMI:
+//
+//  1. snap-*/metadata.json whose VolumeID == imageID  → derivedSnaps
+//     (these are CopyImage-created snapshots that point back at the system AMI)
+//  2. vol-*/config.json whose VolumeMetadata.SnapshotID ∈ {snap-ami-<id>} ∪ derivedSnaps
+//  3. ami-*/config.json whose AMIMetadata.SnapshotID ∈ derivedSnaps
+//
+// The walk terminates one hop deep. CopyImage of an account AMI uses its own
+// VolumeID, not the system AMI's, so further recursion is not required today.
+func FindAMIDependents(store objectstore.ObjectStore, bucket, imageID string) (Dependents, error) {
+	var deps Dependents
+
+	prefixes, err := listCommonPrefixes(store, bucket)
+	if err != nil {
+		return Dependents{}, fmt.Errorf("list bucket prefixes: %w", err)
+	}
+
+	// Pass 1: collect derived snaps (CopyImage of this AMI writes a snap whose
+	// VolumeID points back at the source AMI ID).
+	derived := map[string]bool{}
+	for _, p := range prefixes {
+		if !strings.HasPrefix(p, "snap-") {
+			continue
+		}
+		snapID := strings.TrimSuffix(p, "/")
+		// Skip the viperblock-internal snap prefix for this AMI — it has no
+		// metadata.json (it was written by viperblock.CreateSnapshot, not by
+		// the EC2 snapshot service).
+		if snapID == SnapPrefix(imageID) {
+			continue
+		}
+		cfg, err := handlers_ec2_snapshot.ReadSnapshotConfig(store, bucket, snapID)
+		if err != nil {
+			if objectstore.IsNoSuchKeyError(err) || errors.Is(err, handlers_ec2_snapshot.ErrCorruptSnapshotMetadata) {
+				continue
+			}
+			return Dependents{}, fmt.Errorf("read snapshot %s: %w", snapID, err)
+		}
+		if cfg.VolumeID == imageID {
+			derived[snapID] = true
+			deps.Snapshots = append(deps.Snapshots, snapID)
+		}
+	}
+
+	// The set of snap IDs that a dependent volume might reference: the
+	// admin-import's internal snap plus every CopyImage-derived snap.
+	volSnapRefs := map[string]bool{SnapPrefix(imageID): true}
+	for s := range derived {
+		volSnapRefs[s] = true
+	}
+
+	// Pass 2: volumes (vol-*/config.json) whose SnapshotID matches.
+	for _, p := range prefixes {
+		if !strings.HasPrefix(p, "vol-") {
+			continue
+		}
+		volID := strings.TrimSuffix(p, "/")
+		cfg, err := readVolumeConfig(store, bucket, volID)
+		if err != nil {
+			if objectstore.IsNoSuchKeyError(err) || errors.Is(err, errCorruptVolumeConfig) {
+				continue
+			}
+			return Dependents{}, fmt.Errorf("read volume %s: %w", volID, err)
+		}
+		if volSnapRefs[cfg.VolumeMetadata.SnapshotID] {
+			deps.Volumes = append(deps.Volumes, volID)
+		}
+	}
+
+	// Pass 3: AMIs (ami-*/config.json) whose SnapshotID is a derived snap.
+	// Skip the target AMI itself.
+	for _, p := range prefixes {
+		if !strings.HasPrefix(p, "ami-") {
+			continue
+		}
+		otherAMI := strings.TrimSuffix(p, "/")
+		if otherAMI == imageID {
+			continue
+		}
+		meta, err := readAMIConfig(store, bucket, otherAMI)
+		if err != nil {
+			if objectstore.IsNoSuchKeyError(err) || errors.Is(err, handlers_ec2_image.ErrCorruptAMIConfig) {
+				continue
+			}
+			return Dependents{}, fmt.Errorf("read AMI %s: %w", otherAMI, err)
+		}
+		if derived[meta.SnapshotID] {
+			deps.AMIs = append(deps.AMIs, otherAMI)
+		}
+	}
+
+	return deps, nil
+}
+
+// RemoveSystemImage deletes an admin-imported AMI and its backing storage
+// after re-validating the preconditions:
+//
+//  1. ami-<id>/config.json exists and parses (skipped under --force = salvage)
+//  2. AMI is system-owned, not account-owned (--force bypasses)
+//  3. No volumes / snapshots / AMIs depend on it (--force bypasses)
+//
+// Deletion order is intentional: the config.json is deleted FIRST so the AMI
+// disappears from DescribeImages immediately, preventing new RunInstances
+// from racing the block cleanup.
+func RemoveSystemImage(store objectstore.ObjectStore, bucket string, opts RemoveImageOpts) (*RemoveImageResult, error) {
+	if !strings.HasPrefix(opts.ImageID, "ami-") {
+		return nil, errors.New(awserrors.ErrorInvalidAMIIDMalformed)
+	}
+
+	meta, configErr := readAMIConfig(store, bucket, opts.ImageID)
+	configMissing := objectstore.IsNoSuchKeyError(configErr)
+	configCorrupt := errors.Is(configErr, handlers_ec2_image.ErrCorruptAMIConfig)
+	switch {
+	case configErr == nil:
+		// fine
+	case configMissing, configCorrupt:
+		if !opts.Force {
+			return nil, errors.New(awserrors.ErrorInvalidAMIIDNotFound)
+		}
+	default:
+		slog.Error("RemoveSystemImage: read AMI config", "imageId", opts.ImageID, "err", configErr)
+		return nil, errors.New(awserrors.ErrorServerInternal)
+	}
+
+	if configErr == nil && !opts.Force {
+		if meta.ImageOwnerAlias != "" && utils.IsAccountID(meta.ImageOwnerAlias) {
+			return nil, fmt.Errorf("%s is account-owned (%s); use `aws ec2 deregister-image` followed by `aws ec2 delete-snapshot`",
+				opts.ImageID, meta.ImageOwnerAlias)
+		}
+	}
+
+	if !opts.Force {
+		deps, err := FindAMIDependents(store, bucket, opts.ImageID)
+		if err != nil {
+			slog.Error("RemoveSystemImage: dependency walk", "imageId", opts.ImageID, "err", err)
+			return nil, errors.New(awserrors.ErrorServerInternal)
+		}
+		if !deps.Empty() {
+			return nil, &DependentError{ImageID: opts.ImageID, Dependents: deps}
+		}
+	}
+
+	result := &RemoveImageResult{}
+
+	// Step 1: drop config.json first — the barrier that hides the AMI from
+	// DescribeImages so no new launches can land on the blocks we're deleting.
+	if configErr == nil || (opts.Force && !configMissing) {
+		n, b, err := deletePrefix(store, bucket, opts.ImageID+"/config.json")
+		if err != nil {
+			return nil, fmt.Errorf("delete config: %w", err)
+		}
+		result.ObjectsDeleted += n
+		result.BytesFreed += b
+	}
+
+	// Step 2: drop the rest of ami-<id>/ (chunks, WAL, checkpoints).
+	n, b, err := deletePrefix(store, bucket, opts.ImageID+"/")
+	if err != nil {
+		return nil, fmt.Errorf("delete ami prefix: %w", err)
+	}
+	result.ObjectsDeleted += n
+	result.BytesFreed += b
+
+	// Step 3: drop snap-<amiID>/ (the viperblock-internal snap checkpoint).
+	n, b, err = deletePrefix(store, bucket, SnapPrefix(opts.ImageID)+"/")
+	if err != nil {
+		return nil, fmt.Errorf("delete snap prefix: %w", err)
+	}
+	result.ObjectsDeleted += n
+	result.BytesFreed += b
+
+	slog.Info("RemoveSystemImage completed",
+		"imageId", opts.ImageID,
+		"objectsDeleted", result.ObjectsDeleted,
+		"bytesFreed", result.BytesFreed,
+		"force", opts.Force,
+	)
+	return result, nil
+}
+
+// DependentError is returned by RemoveSystemImage when dependent resources
+// block deletion. The CLI prints the dependents list and exits non-zero.
+type DependentError struct {
+	ImageID    string
+	Dependents Dependents
+}
+
+func (e *DependentError) Error() string {
+	return fmt.Sprintf("refusing to remove %s: %d volumes, %d snapshots, %d AMIs depend on this image",
+		e.ImageID, len(e.Dependents.Volumes), len(e.Dependents.Snapshots), len(e.Dependents.AMIs))
+}
+
+// readAMIConfig reads ami-<id>/config.json and returns the AMIMetadata.
+// Mirrors ImageServiceImpl.GetAMIConfig but operates package-locally so the
+// admin tooling doesn't require an ImageServiceImpl (which carries NATS).
+func readAMIConfig(store objectstore.ObjectStore, bucket, imageID string) (viperblock.AMIMetadata, error) {
+	key := imageID + "/config.json"
+	res, err := store.GetObject(&s3.GetObjectInput{
+		Bucket: aws.String(bucket),
+		Key:    aws.String(key),
+	})
+	if err != nil {
+		return viperblock.AMIMetadata{}, err
+	}
+	defer res.Body.Close()
+
+	body, err := io.ReadAll(res.Body)
+	if err != nil {
+		return viperblock.AMIMetadata{}, err
+	}
+
+	var state viperblock.VBState
+	if err := json.Unmarshal(body, &state); err != nil {
+		return viperblock.AMIMetadata{}, fmt.Errorf("%w: %s: %v", handlers_ec2_image.ErrCorruptAMIConfig, key, err)
+	}
+	return state.VolumeConfig.AMIMetadata, nil
+}
+
+// errCorruptVolumeConfig lets the dependency walk distinguish a vol-*/config.json
+// that can't be parsed (recovery is operator-owned, walk continues) from a
+// transient transport error (walk must fail closed — silently dropping a
+// dependent could lead to deleting blocks underneath a live volume).
+var errCorruptVolumeConfig = errors.New("corrupt volume config")
+
+// readVolumeConfig reads vol-<id>/config.json into VolumeConfig.
+type volumeConfigWrapper struct {
+	VolumeConfig viperblock.VolumeConfig `json:"VolumeConfig"`
+}
+
+func readVolumeConfig(store objectstore.ObjectStore, bucket, volumeID string) (*viperblock.VolumeConfig, error) {
+	key := volumeID + "/config.json"
+	res, err := store.GetObject(&s3.GetObjectInput{
+		Bucket: aws.String(bucket),
+		Key:    aws.String(key),
+	})
+	if err != nil {
+		return nil, err
+	}
+	defer res.Body.Close()
+
+	body, err := io.ReadAll(res.Body)
+	if err != nil {
+		return nil, err
+	}
+
+	var w volumeConfigWrapper
+	if err := json.Unmarshal(body, &w); err != nil {
+		return nil, fmt.Errorf("%w: %s: %v", errCorruptVolumeConfig, key, err)
+	}
+	return &w.VolumeConfig, nil
+}
+
+// listCommonPrefixes returns the top-level "directory" prefixes in the bucket
+// (e.g. "ami-xxx/", "vol-yyy/", "snap-zzz/"), exhausting any pagination.
+func listCommonPrefixes(store objectstore.ObjectStore, bucket string) ([]string, error) {
+	seen := map[string]bool{}
+	var token *string
+	for {
+		out, err := store.ListObjectsV2(&s3.ListObjectsV2Input{
+			Bucket:            aws.String(bucket),
+			Delimiter:         aws.String("/"),
+			ContinuationToken: token,
+		})
+		if err != nil {
+			return nil, err
+		}
+		for _, cp := range out.CommonPrefixes {
+			if cp.Prefix == nil {
+				continue
+			}
+			seen[*cp.Prefix] = true
+		}
+		if !aws.BoolValue(out.IsTruncated) {
+			break
+		}
+		token = out.NextContinuationToken
+	}
+	prefixes := make([]string, 0, len(seen))
+	for p := range seen {
+		prefixes = append(prefixes, p)
+	}
+	return prefixes, nil
+}
+
+// sumPrefix returns (object count, total bytes) for every object under prefix.
+// Used by the preview to surface what an operator is about to delete.
+func sumPrefix(store objectstore.ObjectStore, bucket, prefix string) (int, int64, error) {
+	var count int
+	var bytes int64
+	var token *string
+	for {
+		out, err := store.ListObjectsV2(&s3.ListObjectsV2Input{
+			Bucket:            aws.String(bucket),
+			Prefix:            aws.String(prefix),
+			ContinuationToken: token,
+		})
+		if err != nil {
+			return 0, 0, err
+		}
+		for _, obj := range out.Contents {
+			count++
+			if obj.Size != nil {
+				bytes += *obj.Size
+			}
+		}
+		if !aws.BoolValue(out.IsTruncated) {
+			break
+		}
+		token = out.NextContinuationToken
+	}
+	return count, bytes, nil
+}
+
+// deletePrefix removes every object under prefix and returns count+bytes
+// deleted. Used by RemoveSystemImage for both the single-key config.json
+// barrier and the bulk ami-<id>/ and snap-<id>/ teardown.
+func deletePrefix(store objectstore.ObjectStore, bucket, prefix string) (int, int64, error) {
+	var count int
+	var bytes int64
+	var token *string
+	for {
+		out, err := store.ListObjectsV2(&s3.ListObjectsV2Input{
+			Bucket:            aws.String(bucket),
+			Prefix:            aws.String(prefix),
+			ContinuationToken: token,
+		})
+		if err != nil {
+			return count, bytes, err
+		}
+		for _, obj := range out.Contents {
+			if obj.Key == nil {
+				continue
+			}
+			if _, err := store.DeleteObject(&s3.DeleteObjectInput{
+				Bucket: aws.String(bucket),
+				Key:    obj.Key,
+			}); err != nil {
+				return count, bytes, fmt.Errorf("delete %s: %w", *obj.Key, err)
+			}
+			count++
+			if obj.Size != nil {
+				bytes += *obj.Size
+			}
+		}
+		if !aws.BoolValue(out.IsTruncated) {
+			break
+		}
+		token = out.NextContinuationToken
+	}
+	return count, bytes, nil
+}

spinifex/admin/images_remove_test.go

@@ -0,0 +1,369 @@
+package admin
+
+import (
+	"bytes"
+	"encoding/json"
+	"errors"
+	"testing"
+
+	"github.com/aws/aws-sdk-go/aws"
+	awss3 "github.com/aws/aws-sdk-go/service/s3"
+	"github.com/mulgadc/spinifex/spinifex/awserrors"
+	handlers_ec2_snapshot "github.com/mulgadc/spinifex/spinifex/handlers/ec2/snapshot"
+	"github.com/mulgadc/spinifex/spinifex/objectstore"
+	"github.com/mulgadc/viperblock/viperblock"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+const (
+	testRemoveBucket    = "test-bucket"
+	testRemoveAccountID = "000000000001"
+)
+
+// putAMI writes an ami-<id>/config.json with the given owner. Owner "" produces
+// a corrupt config (empty ImageOwnerAlias is treated as corrupt by readers).
+func putAMI(t *testing.T, store *objectstore.MemoryObjectStore, imageID, name, owner, snapshotID string) {
+	t.Helper()
+	state := viperblock.VBState{
+		VolumeConfig: viperblock.VolumeConfig{
+			AMIMetadata: viperblock.AMIMetadata{
+				ImageID:         imageID,
+				Name:            name,
+				ImageOwnerAlias: owner,
+				SnapshotID:      snapshotID,
+				VolumeSizeGiB:   8,
+			},
+		},
+	}
+	data, err := json.Marshal(state)
+	require.NoError(t, err)
+	_, err = store.PutObject(&awss3.PutObjectInput{
+		Bucket: aws.String(testRemoveBucket),
+		Key:    aws.String(imageID + "/config.json"),
+		Body:   bytes.NewReader(data),
+	})
+	require.NoError(t, err)
+}
+
+// putAMIBlocks writes a few dummy chunk objects under ami-<id>/ so deletion
+// has bulk work and byte counts to verify.
+func putAMIBlocks(t *testing.T, store *objectstore.MemoryObjectStore, imageID string, n int, size int) {
+	t.Helper()
+	body := bytes.Repeat([]byte{0xab}, size)
+	for i := range n {
+		_, err := store.PutObject(&awss3.PutObjectInput{
+			Bucket: aws.String(testRemoveBucket),
+			Key:    aws.String(imageID + "/chunks/" + string(rune('a'+i)) + ".dat"),
+			Body:   bytes.NewReader(body),
+		})
+		require.NoError(t, err)
+	}
+}
+
+func putSnapBlocks(t *testing.T, store *objectstore.MemoryObjectStore, snapID string, n int, size int) {
+	t.Helper()
+	body := bytes.Repeat([]byte{0xcd}, size)
+	for i := range n {
+		_, err := store.PutObject(&awss3.PutObjectInput{
+			Bucket: aws.String(testRemoveBucket),
+			Key:    aws.String(snapID + "/cp/" + string(rune('a'+i)) + ".bin"),
+			Body:   bytes.NewReader(body),
+		})
+		require.NoError(t, err)
+	}
+}
+
+// putSnapMetadata writes a CopyImage-derived EC2 snapshot metadata that points
+// VolumeID back at an admin-imported AMI's ID.
+func putSnapMetadata(t *testing.T, store *objectstore.MemoryObjectStore, snapID, volumeID string) {
+	t.Helper()
+	cfg := &handlers_ec2_snapshot.SnapshotConfig{
+		SnapshotID: snapID,
+		VolumeID:   volumeID,
+		VolumeSize: 8,
+		State:      "completed",
+	}
+	require.NoError(t, handlers_ec2_snapshot.WriteSnapshotConfig(store, testRemoveBucket, snapID, cfg))
+}
+
+// putVolume writes vol-<id>/config.json with the given SnapshotID reference.
+func putVolume(t *testing.T, store *objectstore.MemoryObjectStore, volID, snapshotID string) {
+	t.Helper()
+	wrapper := volumeConfigWrapper{
+		VolumeConfig: viperblock.VolumeConfig{
+			VolumeMetadata: viperblock.VolumeMetadata{
+				VolumeID:   volID,
+				SnapshotID: snapshotID,
+				SizeGiB:    8,
+			},
+		},
+	}
+	data, err := json.Marshal(wrapper)
+	require.NoError(t, err)
+	_, err = store.PutObject(&awss3.PutObjectInput{
+		Bucket: aws.String(testRemoveBucket),
+		Key:    aws.String(volID + "/config.json"),
+		Body:   bytes.NewReader(data),
+	})
+	require.NoError(t, err)
+}
+
+func TestRemoveSystemImage_HappyPath(t *testing.T) {
+	store := objectstore.NewMemoryObjectStore()
+	const id = "ami-sys-deb13"
+	putAMI(t, store, id, "debian-13", "system", SnapPrefix(id))
+	putAMIBlocks(t, store, id, 3, 128)
+	putSnapBlocks(t, store, SnapPrefix(id), 2, 16)
+
+	preview, err := PreviewRemoveSystemImage(store, testRemoveBucket, id)
+	require.NoError(t, err)
+	assert.True(t, preview.ConfigPresent)
+	assert.True(t, preview.IsSystemOwned)
+	assert.Equal(t, "system", preview.Owner)
+	// config.json + 3 chunks = 4 ami-prefix objects.
+	assert.Equal(t, 4, preview.AMIObjectCount)
+	assert.Equal(t, 2, preview.SnapObjectCount)
+	assert.True(t, preview.Dependents.Empty())
+
+	res, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id})
+	require.NoError(t, err)
+	assert.Equal(t, preview.AMIObjectCount+preview.SnapObjectCount, res.ObjectsDeleted)
+	assert.Equal(t, preview.AMIBytesTotal+preview.SnapBytesTotal, res.BytesFreed)
+	assert.Equal(t, 0, store.Count())
+}
+
+func TestRemoveSystemImage_AccountOwned_Refused(t *testing.T) {
+	store := objectstore.NewMemoryObjectStore()
+	const id = "ami-acct-001"
+	putAMI(t, store, id, "user-ami", testRemoveAccountID, "snap-acct-001")
+	putAMIBlocks(t, store, id, 1, 16)
+
+	_, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id})
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), id)
+	assert.Contains(t, err.Error(), testRemoveAccountID)
+	assert.Contains(t, err.Error(), "account-owned")
+	assert.Contains(t, err.Error(), "aws ec2 deregister-image")
+	// Objects untouched.
+	assert.Equal(t, 2, store.Count())
+}
+
+func TestRemoveSystemImage_AccountOwned_ForceBypasses(t *testing.T) {
+	store := objectstore.NewMemoryObjectStore()
+	const id = "ami-acct-002"
+	putAMI(t, store, id, "user-ami", testRemoveAccountID, "snap-acct-002")
+	putAMIBlocks(t, store, id, 2, 16)
+
+	res, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id, Force: true})
+	require.NoError(t, err)
+	assert.Greater(t, res.ObjectsDeleted, 0)
+	assert.Equal(t, 0, store.Count())
+}
+
+func TestRemoveSystemImage_MissingConfig_NotFound(t *testing.T) {
+	store := objectstore.NewMemoryObjectStore()
+	const id = "ami-missing"
+	putAMIBlocks(t, store, id, 1, 16)
+
+	_, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id})
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorInvalidAMIIDNotFound, err.Error())
+}
+
+func TestRemoveSystemImage_CorruptConfig_NotFound(t *testing.T) {
+	store := objectstore.NewMemoryObjectStore()
+	const id = "ami-corrupt"
+	_, err := store.PutObject(&awss3.PutObjectInput{
+		Bucket: aws.String(testRemoveBucket),
+		Key:    aws.String(id + "/config.json"),
+		Body:   bytes.NewReader([]byte("{not valid json")),
+	})
+	require.NoError(t, err)
+
+	_, err = RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id})
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorInvalidAMIIDNotFound, err.Error())
+}
+
+func TestRemoveSystemImage_DependentVolume_Direct_Refused(t *testing.T) {
+	store := objectstore.NewMemoryObjectStore()
+	const id = "ami-sys-deb13"
+	putAMI(t, store, id, "debian-13", "system", SnapPrefix(id))
+	// Direct dependent: a volume launched from the admin import.
+	putVolume(t, store, "vol-aaa", SnapPrefix(id))
+
+	_, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id})
+	require.Error(t, err)
+	var depErr *DependentError
+	require.True(t, errors.As(err, &depErr))
+	assert.Contains(t, depErr.Dependents.Volumes, "vol-aaa")
+}
+
+func TestRemoveSystemImage_DependentVolume_Transitive_Refused(t *testing.T) {
+	store := objectstore.NewMemoryObjectStore()
+	const id = "ami-sys-deb13"
+	putAMI(t, store, id, "debian-13", "system", SnapPrefix(id))
+
+	// CopyImage of the system AMI wrote a snap whose VolumeID points back at the AMI ID.
+	const derivedSnap = "snap-derived-001"
+	putSnapMetadata(t, store, derivedSnap, id)
+	// A volume launched from the copied AMI references the derived snap.
+	putVolume(t, store, "vol-bbb", derivedSnap)
+
+	_, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id})
+	require.Error(t, err)
+	var depErr *DependentError
+	require.True(t, errors.As(err, &depErr))
+	assert.Contains(t, depErr.Dependents.Volumes, "vol-bbb")
+	assert.Contains(t, depErr.Dependents.Snapshots, derivedSnap)
+}
+
+func TestRemoveSystemImage_DependentAMI_Refused(t *testing.T) {
+	store := objectstore.NewMemoryObjectStore()
+	const id = "ami-sys-deb13"
+	putAMI(t, store, id, "debian-13", "system", SnapPrefix(id))
+
+	const derivedSnap = "snap-derived-002"
+	putSnapMetadata(t, store, derivedSnap, id)
+	// Account AMI created via CopyImage; its SnapshotID is the derived snap.
+	putAMI(t, store, "ami-acct-002", "copied", testRemoveAccountID, derivedSnap)
+
+	_, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id})
+	require.Error(t, err)
+	var depErr *DependentError
+	require.True(t, errors.As(err, &depErr))
+	assert.Contains(t, depErr.Dependents.AMIs, "ami-acct-002")
+}
+
+func TestRemoveSystemImage_Force_OverridesDependents(t *testing.T) {
+	store := objectstore.NewMemoryObjectStore()
+	const id = "ami-sys-deb13"
+	putAMI(t, store, id, "debian-13", "system", SnapPrefix(id))
+	putAMIBlocks(t, store, id, 2, 64)
+	putVolume(t, store, "vol-orphan", SnapPrefix(id))
+
+	res, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id, Force: true})
+	require.NoError(t, err)
+	assert.Greater(t, res.ObjectsDeleted, 0)
+	// vol-orphan/config.json remains; the AMI is gone.
+	_, err = store.GetObject(&awss3.GetObjectInput{
+		Bucket: aws.String(testRemoveBucket),
+		Key:    aws.String(id + "/config.json"),
+	})
+	require.Error(t, err)
+	_, err = store.GetObject(&awss3.GetObjectInput{
+		Bucket: aws.String(testRemoveBucket),
+		Key:    aws.String("vol-orphan/config.json"),
+	})
+	require.NoError(t, err)
+}
+
+func TestRemoveSystemImage_Salvage_MissingConfig_ForceCleans(t *testing.T) {
+	store := objectstore.NewMemoryObjectStore()
+	const id = "ami-salvage-1"
+	// No config.json — just orphaned blocks.
+	putAMIBlocks(t, store, id, 3, 32)
+	putSnapBlocks(t, store, SnapPrefix(id), 1, 8)
+
+	// Without --force: NotFound.
+	_, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id})
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorInvalidAMIIDNotFound, err.Error())
+
+	// With --force: salvage proceeds.
+	res, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id, Force: true})
+	require.NoError(t, err)
+	assert.Equal(t, 4, res.ObjectsDeleted)
+	assert.Equal(t, 0, store.Count())
+}
+
+func TestRemoveSystemImage_Salvage_CorruptConfig_ForceCleans(t *testing.T) {
+	store := objectstore.NewMemoryObjectStore()
+	const id = "ami-salvage-2"
+	_, err := store.PutObject(&awss3.PutObjectInput{
+		Bucket: aws.String(testRemoveBucket),
+		Key:    aws.String(id + "/config.json"),
+		Body:   bytes.NewReader([]byte("garbage")),
+	})
+	require.NoError(t, err)
+	putAMIBlocks(t, store, id, 1, 8)
+
+	// Without --force.
+	_, err = RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id})
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorInvalidAMIIDNotFound, err.Error())
+
+	// With --force: corrupt config + blocks all go.
+	res, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id, Force: true})
+	require.NoError(t, err)
+	assert.Equal(t, 2, res.ObjectsDeleted)
+	assert.Equal(t, 0, store.Count())
+}
+
+func TestRemoveSystemImage_IdempotentRerun(t *testing.T) {
+	store := objectstore.NewMemoryObjectStore()
+	const id = "ami-rerun"
+	putAMI(t, store, id, "debian-13", "system", SnapPrefix(id))
+	putAMIBlocks(t, store, id, 2, 16)
+
+	_, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id})
+	require.NoError(t, err)
+	require.Equal(t, 0, store.Count())
+
+	// Second call without --force should report NotFound.
+	_, err = RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id})
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorInvalidAMIIDNotFound, err.Error())
+
+	// Salvage re-run is a no-op.
+	res, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id, Force: true})
+	require.NoError(t, err)
+	assert.Equal(t, 0, res.ObjectsDeleted)
+}
+
+func TestRemoveSystemImage_MalformedID(t *testing.T) {
+	store := objectstore.NewMemoryObjectStore()
+	_, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: "vol-wrong"})
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorInvalidAMIIDMalformed, err.Error())
+}
+
+func TestPreviewRemoveSystemImage_Salvage(t *testing.T) {
+	store := objectstore.NewMemoryObjectStore()
+	const id = "ami-salvage-preview"
+	putAMIBlocks(t, store, id, 2, 4)
+
+	preview, err := PreviewRemoveSystemImage(store, testRemoveBucket, id)
+	require.NoError(t, err)
+	assert.False(t, preview.ConfigPresent)
+	assert.False(t, preview.ConfigCorrupt)
+	assert.Equal(t, 2, preview.AMIObjectCount)
+	assert.Equal(t, int64(8), preview.AMIBytesTotal)
+}
+
+func TestPreviewRemoveSystemImage_CorruptConfig(t *testing.T) {
+	store := objectstore.NewMemoryObjectStore()
+	const id = "ami-corrupt-preview"
+	_, err := store.PutObject(&awss3.PutObjectInput{
+		Bucket: aws.String(testRemoveBucket),
+		Key:    aws.String(id + "/config.json"),
+		Body:   bytes.NewReader([]byte("{nope")),
+	})
+	require.NoError(t, err)
+
+	preview, err := PreviewRemoveSystemImage(store, testRemoveBucket, id)
+	require.NoError(t, err)
+	assert.False(t, preview.ConfigPresent)
+	assert.True(t, preview.ConfigCorrupt)
+}
+
+func TestFindAMIDependents_SkipsTargetAMI(t *testing.T) {
+	store := objectstore.NewMemoryObjectStore()
+	const id = "ami-sys-only"
+	putAMI(t, store, id, "debian-13", "system", SnapPrefix(id))
+
+	deps, err := FindAMIDependents(store, testRemoveBucket, id)
+	require.NoError(t, err)
+	assert.True(t, deps.Empty())
+}

spinifex/daemon/daemon.go

@@ -28,6 +28,7 @@ import (
 	"github.com/aws/aws-sdk-go/aws"
 	"github.com/aws/aws-sdk-go/service/ec2"
 	"github.com/go-chi/chi/v5"
+	"github.com/mulgadc/spinifex/internal/tlsconfig"
 	"github.com/mulgadc/spinifex/spinifex/admin"
 	"github.com/mulgadc/spinifex/spinifex/awserrors"
 	"github.com/mulgadc/spinifex/spinifex/config"
@@ -408,11 +409,12 @@ func NewResourceManager(gpuModels []instancetypes.GPUModel, gpuMgr *gpu.Manager)
 		return nil, fmt.Errorf("detect system memory: %w", err)
 	}
 
-	reservedVCPU, reservedMem, err := applyHostReserve(defaultHostReserve, numCPU, totalMemGB)
+	reserve := resolveHostReserve(os.Getenv)
+	reservedVCPU, reservedMem, err := applyHostReserve(reserve, numCPU, totalMemGB)
 	if err != nil {
 		slog.Error("host below minimum reserve — daemon refuses to start",
 			"err", err, "hostVCPU", numCPU, "hostMemGB", totalMemGB,
-			"reserveVCPU", defaultHostReserve.vCPU, "reserveMemGB", defaultHostReserve.memGB)
+			"reserveVCPU", reserve.vCPU, "reserveMemGB", reserve.memGB)
 		return nil, fmt.Errorf("validate host reserve: %w", err)
 	}
 
@@ -757,26 +759,6 @@ func (d *Daemon) subscribeAll() error {
 		{"ec2.GetSerialConsoleAccessStatus", d.handleEC2GetSerialConsoleAccessStatus, "spinifex-workers"},
 		{"ec2.EnableSerialConsoleAccess", d.handleEC2EnableSerialConsoleAccess, "spinifex-workers"},
 		{"ec2.DisableSerialConsoleAccess", d.handleEC2DisableSerialConsoleAccess, "spinifex-workers"},
-		// ELBv2 operations
-		{"elbv2.CreateLoadBalancer", d.handleELBv2CreateLoadBalancer, "spinifex-workers"},
-		{"elbv2.DeleteLoadBalancer", d.handleELBv2DeleteLoadBalancer, "spinifex-workers"},
-		{"elbv2.DescribeLoadBalancers", d.handleELBv2DescribeLoadBalancers, "spinifex-workers"},
-		{"elbv2.CreateTargetGroup", d.handleELBv2CreateTargetGroup, "spinifex-workers"},
-		{"elbv2.DeleteTargetGroup", d.handleELBv2DeleteTargetGroup, "spinifex-workers"},
-		{"elbv2.DescribeTargetGroups", d.handleELBv2DescribeTargetGroups, "spinifex-workers"},
-		{"elbv2.RegisterTargets", d.handleELBv2RegisterTargets, "spinifex-workers"},
-		{"elbv2.DeregisterTargets", d.handleELBv2DeregisterTargets, "spinifex-workers"},
-		{"elbv2.DescribeTargetHealth", d.handleELBv2DescribeTargetHealth, "spinifex-workers"},
-		{"elbv2.CreateListener", d.handleELBv2CreateListener, "spinifex-workers"},
-		{"elbv2.DeleteListener", d.handleELBv2DeleteListener, "spinifex-workers"},
-		{"elbv2.DescribeListeners", d.handleELBv2DescribeListeners, "spinifex-workers"},
-		{"elbv2.DescribeTags", d.handleELBv2DescribeTags, "spinifex-workers"},
-		{"elbv2.LBAgentHeartbeat", d.handleELBv2LBAgentHeartbeat, "spinifex-workers"},
-		{"elbv2.GetLBConfig", d.handleELBv2GetLBConfig, "spinifex-workers"},
-		{"elbv2.ModifyTargetGroupAttributes", d.handleELBv2ModifyTargetGroupAttributes, "spinifex-workers"},
-		{"elbv2.DescribeTargetGroupAttributes", d.handleELBv2DescribeTargetGroupAttributes, "spinifex-workers"},
-		{"elbv2.ModifyLoadBalancerAttributes", d.handleELBv2ModifyLoadBalancerAttributes, "spinifex-workers"},
-		{"elbv2.DescribeLoadBalancerAttributes", d.handleELBv2DescribeLoadBalancerAttributes, "spinifex-workers"},
 		{fmt.Sprintf("spinifex.admin.%s.health", d.node), d.handleHealthCheck, ""},
 		{"spinifex.nodes.discover", d.handleNodeDiscover, ""},
 		{"spinifex.node.status", d.handleNodeStatus, ""},
@@ -792,6 +774,32 @@ func (d *Daemon) subscribeAll() error {
 		{"spinifex.cluster.shutdown.infra", d.handleShutdownInfra, ""},
 	}
 
+	// ELBv2 operations require a resolved gateway URL.
+	// Without a subscriber the gateway returns nats.ErrNoResponders → ServiceUnavailable.
+	if d.elbv2Service.GatewayURL != "" {
+		subs = append(subs,
+			natsSub{"elbv2.CreateLoadBalancer", d.handleELBv2CreateLoadBalancer, "spinifex-workers"},
+			natsSub{"elbv2.DeleteLoadBalancer", d.handleELBv2DeleteLoadBalancer, "spinifex-workers"},
+			natsSub{"elbv2.DescribeLoadBalancers", d.handleELBv2DescribeLoadBalancers, "spinifex-workers"},
+			natsSub{"elbv2.CreateTargetGroup", d.handleELBv2CreateTargetGroup, "spinifex-workers"},
+			natsSub{"elbv2.DeleteTargetGroup", d.handleELBv2DeleteTargetGroup, "spinifex-workers"},
+			natsSub{"elbv2.DescribeTargetGroups", d.handleELBv2DescribeTargetGroups, "spinifex-workers"},
+			natsSub{"elbv2.RegisterTargets", d.handleELBv2RegisterTargets, "spinifex-workers"},
+			natsSub{"elbv2.DeregisterTargets", d.handleELBv2DeregisterTargets, "spinifex-workers"},
+			natsSub{"elbv2.DescribeTargetHealth", d.handleELBv2DescribeTargetHealth, "spinifex-workers"},
+			natsSub{"elbv2.CreateListener", d.handleELBv2CreateListener, "spinifex-workers"},
+			natsSub{"elbv2.DeleteListener", d.handleELBv2DeleteListener, "spinifex-workers"},
+			natsSub{"elbv2.DescribeListeners", d.handleELBv2DescribeListeners, "spinifex-workers"},
+			natsSub{"elbv2.DescribeTags", d.handleELBv2DescribeTags, "spinifex-workers"},
+			natsSub{"elbv2.LBAgentHeartbeat", d.handleELBv2LBAgentHeartbeat, "spinifex-workers"},
+			natsSub{"elbv2.GetLBConfig", d.handleELBv2GetLBConfig, "spinifex-workers"},
+			natsSub{"elbv2.ModifyTargetGroupAttributes", d.handleELBv2ModifyTargetGroupAttributes, "spinifex-workers"},
+			natsSub{"elbv2.DescribeTargetGroupAttributes", d.handleELBv2DescribeTargetGroupAttributes, "spinifex-workers"},
+			natsSub{"elbv2.ModifyLoadBalancerAttributes", d.handleELBv2ModifyLoadBalancerAttributes, "spinifex-workers"},
+			natsSub{"elbv2.DescribeLoadBalancerAttributes", d.handleELBv2DescribeLoadBalancerAttributes, "spinifex-workers"},
+		)
+	}
+
 	// EIP operations require external IPAM (pool mode). Only subscribe when available;
 	// without a subscriber the gateway returns a NATS timeout → clean error to the client.
 	if d.eipService != nil {
@@ -1190,6 +1198,16 @@ func (d *Daemon) startCluster() error {
 		return "sys.micro"
 	})
 
+	// Invalidate persisted target HealthState before subscriptions go live.
+	// Stale "healthy" entries from a pre-restart cluster otherwise satisfy
+	// DescribeTargetHealth waiters before any actual post-restart health
+	// observation has been recorded. Best-effort: a failure here just leaves
+	// the old behavior in place rather than blocking daemon startup.
+	if err := d.elbv2Service.ResetTargetHealthOnStartup(context.Background()); err != nil {
+		slog.Warn("ELBv2: target-health reset failed; continuing with stale state",
+			"err", err)
+	}
+
 	// Ensure default VPC exists for system and admin accounts
 	// (matches AWS: every account has a default VPC with IGW + default SG)
 	if d.vpcService != nil {
@@ -1598,8 +1616,9 @@ func (d *Daemon) ClusterManager() error {
 	}
 
 	tlsConfig := &tls.Config{
-		Certificates: []tls.Certificate{cert},
-		MinVersion:   tls.VersionTLS12,
+		Certificates:     []tls.Certificate{cert},
+		MinVersion:       tls.VersionTLS13,
+		CurvePreferences: tlsconfig.Curves,
 	}
 
 	d.clusterServer = &http.Server{
@@ -1856,7 +1875,13 @@ func (d *Daemon) respondWithVolumeAttachment(msg *nats.Msg, volumeID, instanceID
 func (rm *ResourceManager) canAllocate(instanceType *ec2.InstanceTypeInfo, count int) int {
 	rm.mu.RLock()
 	defer rm.mu.RUnlock()
+	return rm.canAllocateLocked(instanceType, count)
+}
 
+// canAllocateLocked is the lock-free body of canAllocate. Caller must already
+// hold rm.mu for read or write. Extracted so allocate can re-check capacity
+// while holding the write lock without dropping it.
+func (rm *ResourceManager) canAllocateLocked(instanceType *ec2.InstanceTypeInfo, count int) int {
 	// GPU capacity is managed exclusively by gpuManager.Claim; don't double-gate
 	// on host CPU/memory which is always abundant on GPU-class hardware.
 	if instancetypes.IsGPUType(instanceType) {
@@ -1873,17 +1898,21 @@ func (rm *ResourceManager) canAllocate(instanceType *ec2.InstanceTypeInfo, count
 	)
 }
 
-// allocate reserves resources for an instance and updates NATS subscriptions
+// allocate reserves resources for one instance and updates NATS subscriptions.
+// Check and commit run under a single write-lock acquisition; without this,
+// two concurrent callers could both observe free capacity through the read
+// lock and then both commit, overcommitting the host. Multi-instance launch
+// paths loop on allocate per VM, relying on this per-call atomicity.
 func (rm *ResourceManager) allocate(instanceType *ec2.InstanceTypeInfo) error {
-	if rm.canAllocate(instanceType, 1) < 1 {
+	rm.mu.Lock()
+	if rm.canAllocateLocked(instanceType, 1) < 1 {
+		rm.mu.Unlock()
 		instanceTypeName := ""
 		if instanceType.InstanceType != nil {
 			instanceTypeName = *instanceType.InstanceType
 		}
 		return fmt.Errorf("insufficient resources for instance type %s", instanceTypeName)
 	}
-
-	rm.mu.Lock()
 	vCPUs := instanceTypeVCPUs(instanceType)
 	memoryGB := float64(instanceTypeMemoryMiB(instanceType)) / 1024.0
 	rm.allocatedVCPU += int(vCPUs)

spinifex/daemon/daemon_handlers_vpc.go

@@ -3,6 +3,7 @@ package daemon
 import (
 	"encoding/json"
 	"log/slog"
+	"time"
 
 	"github.com/aws/aws-sdk-go/service/ec2"
 	"github.com/mulgadc/spinifex/spinifex/admin"
@@ -82,82 +83,103 @@ func (d *Daemon) handleAccountCreated(msg *nats.Msg) {
 		// startup or handleAccountCreated event will retry.
 		return
 	}
-	d.ensureDefaultVPCInfrastructure(nil)
+	d.ensureDefaultVPCInfrastructureFor(evt.AccountID)
 }
 
 // ensureDefaultVPCInfrastructure attaches an IGW and a default 0.0.0.0/0 route
-// to each account's default VPC. The default SG is provisioned by
+// to each well-known account's default VPC. The default SG is provisioned by
 // EnsureDefaultVPC / CreateVpc itself, so this routine is IGW-only. Accounts
 // in skipAccounts are not touched — used to avoid attaching infrastructure to
 // a half-built VPC when EnsureDefaultVPC failed earlier in startup.
 func (d *Daemon) ensureDefaultVPCInfrastructure(skipAccounts map[string]struct{}) {
-	if d.igwService == nil || d.vpcService == nil {
-		return
-	}
-
 	for _, accountID := range []string{utils.GlobalAccountID, admin.DefaultAccountID()} {
 		if _, skip := skipAccounts[accountID]; skip {
 			continue
 		}
-		// Find the default VPC for this account
-		descOut, err := d.vpcService.DescribeVpcs(&ec2.DescribeVpcsInput{}, accountID)
+		d.ensureDefaultVPCInfrastructureFor(accountID)
+	}
+}
+
+// ensureDefaultVPCInfrastructureFor attaches an IGW and 0.0.0.0/0 route to the
+// given account's default VPC if not already present.
+func (d *Daemon) ensureDefaultVPCInfrastructureFor(accountID string) {
+	if d.igwService == nil || d.vpcService == nil {
+		return
+	}
+
+	// Find the default VPC for this account
+	descOut, err := d.vpcService.DescribeVpcs(&ec2.DescribeVpcsInput{}, accountID)
+	if err != nil {
+		slog.Warn("DescribeVpcs failed for default VPC infrastructure, retrying", "accountID", accountID, "err", err)
+		time.Sleep(500 * time.Millisecond)
+		descOut, err = d.vpcService.DescribeVpcs(&ec2.DescribeVpcsInput{}, accountID)
 		if err != nil {
-			continue
+			slog.Error("DescribeVpcs failed for default VPC infrastructure after retry",
+				"accountID", accountID, "err", err)
+			return
 		}
-		var defaultVpcId string
-		for _, vpc := range descOut.Vpcs {
-			if vpc.IsDefault != nil && *vpc.IsDefault {
-				defaultVpcId = *vpc.VpcId
+	}
+	var defaultVpcId string
+	for _, vpc := range descOut.Vpcs {
+		if vpc.IsDefault != nil && *vpc.IsDefault {
+			defaultVpcId = *vpc.VpcId
+			break
+		}
+	}
+	if defaultVpcId == "" {
+		return
+	}
+
+	// Check if IGW already attached
+	igwOut, err := d.igwService.DescribeInternetGateways(&ec2.DescribeInternetGatewaysInput{}, accountID)
+	if err != nil {
+		slog.Warn("DescribeInternetGateways failed for default VPC infrastructure, retrying",
+			"accountID", accountID, "err", err)
+		time.Sleep(500 * time.Millisecond)
+		igwOut, err = d.igwService.DescribeInternetGateways(&ec2.DescribeInternetGatewaysInput{}, accountID)
+		if err != nil {
+			slog.Error("DescribeInternetGateways failed for default VPC infrastructure after retry",
+				"accountID", accountID, "err", err)
+			return
+		}
+	}
+	hasIGW := false
+	for _, igw := range igwOut.InternetGateways {
+		for _, att := range igw.Attachments {
+			if att.VpcId != nil && *att.VpcId == defaultVpcId {
+				hasIGW = true
 				break
 			}
 		}
-		if defaultVpcId == "" {
-			continue
+	}
+	if !hasIGW {
+		// Create and attach an IGW — use bootstrap ID if available
+		var createOut *ec2.CreateInternetGatewayOutput
+		var err error
+		if accountID == admin.DefaultAccountID() && d.clusterConfig != nil && d.clusterConfig.Bootstrap.IgwId != "" {
+			createOut, err = d.igwService.CreateInternetGatewayWithID(&ec2.CreateInternetGatewayInput{}, accountID, d.clusterConfig.Bootstrap.IgwId)
+		} else {
+			createOut, err = d.igwService.CreateInternetGateway(&ec2.CreateInternetGatewayInput{}, accountID)
 		}
-
-		// Check if IGW already attached
-		igwOut, err := d.igwService.DescribeInternetGateways(&ec2.DescribeInternetGatewaysInput{}, accountID)
 		if err != nil {
-			continue
+			slog.Error("Failed to create default IGW", "accountID", accountID, "err", err)
+			return
 		}
-		hasIGW := false
-		for _, igw := range igwOut.InternetGateways {
-			for _, att := range igw.Attachments {
-				if att.VpcId != nil && *att.VpcId == defaultVpcId {
-					hasIGW = true
-					break
-				}
-			}
-		}
-		if !hasIGW {
-			// Create and attach an IGW — use bootstrap ID if available
-			var createOut *ec2.CreateInternetGatewayOutput
-			var err error
-			if accountID == admin.DefaultAccountID() && d.clusterConfig != nil && d.clusterConfig.Bootstrap.IgwId != "" {
-				createOut, err = d.igwService.CreateInternetGatewayWithID(&ec2.CreateInternetGatewayInput{}, accountID, d.clusterConfig.Bootstrap.IgwId)
-			} else {
-				createOut, err = d.igwService.CreateInternetGateway(&ec2.CreateInternetGatewayInput{}, accountID)
-			}
-			if err != nil {
-				slog.Error("Failed to create default IGW", "accountID", accountID, "err", err)
-				continue
-			}
-			igwId := *createOut.InternetGateway.InternetGatewayId
-			_, err = d.igwService.AttachInternetGateway(&ec2.AttachInternetGatewayInput{
-				InternetGatewayId: &igwId,
-				VpcId:             &defaultVpcId,
-			}, accountID)
-			if err != nil {
-				slog.Error("Failed to attach default IGW", "igwId", igwId, "vpcId", defaultVpcId, "err", err)
-			} else {
-				slog.Info("Attached default IGW to default VPC", "igwId", igwId, "vpcId", defaultVpcId, "accountID", accountID)
-			}
+		igwId := *createOut.InternetGateway.InternetGatewayId
+		_, err = d.igwService.AttachInternetGateway(&ec2.AttachInternetGatewayInput{
+			InternetGatewayId: &igwId,
+			VpcId:             &defaultVpcId,
+		}, accountID)
+		if err != nil {
+			slog.Error("Failed to attach default IGW", "igwId", igwId, "vpcId", defaultVpcId, "err", err)
+		} else {
+			slog.Info("Attached default IGW to default VPC", "igwId", igwId, "vpcId", defaultVpcId, "accountID", accountID)
 		}
+	}
 
-		// Add 0.0.0.0/0 → IGW route to the main route table (if not already present)
-		if d.routeTableService != nil {
-			d.ensureDefaultIGWRoute(accountID, defaultVpcId)
-		}
+	// Add 0.0.0.0/0 → IGW route to the main route table (if not already present)
+	if d.routeTableService != nil {
+		d.ensureDefaultIGWRoute(accountID, defaultVpcId)
 	}
 }
 

spinifex/daemon/daemon_system_instance.go

@@ -80,6 +80,10 @@ func (d *Daemon) LaunchSystemInstance(input *handlers_elbv2.SystemInstanceInput)
 	instance.Reservation.SetReservationId(utils.GenerateResourceID("r"))
 	instance.Reservation.SetOwnerId(accountID)
 	instance.Reservation.Instances = []*ec2.Instance{ec2Instance}
+	// Mirror the customer-instance path (handlers/ec2/instance/service_impl.go:334)
+	// so consumers reading instance.Instance (e.g. onInstanceUpHook's NAT
+	// republish, device_map, volumes) see the same metadata for system VMs.
+	instance.Instance = ec2Instance
 
 	// Attach ENI — either use pre-created one or auto-create
 	privateIP := ""
@@ -92,6 +96,17 @@ func (d *Daemon) LaunchSystemInstance(input *handlers_elbv2.SystemInstanceInput)
 		if input.SubnetID != "" {
 			ec2Instance.SetSubnetId(input.SubnetID)
 		}
+		// The auto-create-ENI branch below populates VpcId from the freshly
+		// created ENI; do the same for pre-created ENIs so consumers reading
+		// instance.Instance.VpcId (notably onInstanceUpHook's NAT republish)
+		// work uniformly across both paths.
+		if d.vpcService != nil {
+			if eniOut, descErr := d.vpcService.DescribeNetworkInterfaces(&ec2.DescribeNetworkInterfacesInput{
+				NetworkInterfaceIds: []*string{aws.String(input.ENIID)},
+			}, eniAccountID); descErr == nil && len(eniOut.NetworkInterfaces) > 0 && eniOut.NetworkInterfaces[0].VpcId != nil {
+				ec2Instance.SetVpcId(*eniOut.NetworkInterfaces[0].VpcId)
+			}
+		}
 		// Mark ENI as attached to this instance
 		if d.vpcService != nil {
 			if _, attachErr := d.vpcService.AttachENI(eniAccountID, instance.ENIId, instance.ID, 0); attachErr != nil {
@@ -212,7 +227,22 @@ func (d *Daemon) LaunchSystemInstance(input *handlers_elbv2.SystemInstanceInput)
 				vpcID = *result.NetworkInterfaces[0].VpcId
 			}
 			portName := "port-" + instance.ENIId
-			utils.PublishNATEvent(d.natsConn, "vpc.add-nat", vpcID, publicIP, privateIP, portName, instance.ENIMac)
+			if natErr := utils.AddNAT(d.natsConn, vpcID, publicIP, privateIP, portName, instance.ENIMac); natErr != nil {
+				slog.Error("LaunchSystemInstance: vpc.add-nat failed for ALB public IP — rolling back to avoid surfacing an unreachable address",
+					"instanceId", instance.ID, "publicIp", publicIP, "pool", poolName, "err", natErr)
+				// Timeout may have committed the rule after our window; neutralise before releasing the IP.
+				utils.PublishNATEvent(d.natsConn, "vpc.delete-nat", vpcID, publicIP, privateIP, portName, instance.ENIMac)
+				if clearErr := d.vpcService.UpdateENIPublicIP(eniAccountID, instance.ENIId, "", ""); clearErr != nil {
+					slog.Warn("LaunchSystemInstance: failed to clear ENI public IP during NAT-failure rollback",
+						"eniId", instance.ENIId, "publicIp", publicIP, "err", clearErr)
+				}
+				if relErr := d.externalIPAM.ReleaseIP(poolName, publicIP); relErr != nil {
+					slog.Warn("LaunchSystemInstance: failed to release public IP during NAT-failure rollback",
+						"publicIp", publicIP, "pool", poolName, "err", relErr)
+				}
+				d.cleanupFailedSystemInstance(instance, instanceType)
+				return nil, fmt.Errorf("commit NAT rule for internet-facing ALB public IP %s: %w", publicIP, natErr)
+			}
 			instance.PublicIP = publicIP
 			instance.PublicIPPool = poolName
 			slog.Info("LaunchSystemInstance: allocated public IP via direct IPAM",
@@ -377,6 +407,37 @@ func (d *Daemon) TerminateSystemInstance(instanceID string) error {
 	return nil
 }
 
+// refreshSystemInstanceState regenerates the tmpfs-backed fw_cfg blobs that
+// QEMU loads at boot. The blobs live under utils.RuntimeDir() (tmpfs on
+// production hosts) and are wiped on host reboot while the persisted
+// vm.Config still references the same paths. Customer VMs use only paths
+// under /var/lib/spinifex/ and are a no-op.
+func (d *Daemon) refreshSystemInstanceState(inst *vm.VM) error {
+	if inst.ManagedBy != tags.ManagedByELBv2 {
+		return nil
+	}
+	if d.elbv2Service == nil {
+		return fmt.Errorf("elbv2 service unavailable: cannot refresh fw_cfg blobs for system VM %s", inst.ID)
+	}
+	ctx := handlers_elbv2.RecoveryContext{
+		InstanceID:   inst.ID,
+		InstanceType: inst.InstanceType,
+		ENIMac:       inst.ENIMac,
+		MgmtMAC:      inst.MgmtMAC,
+		MgmtIP:       inst.MgmtIP,
+	}
+	input, err := d.elbv2Service.RebuildSystemInstanceInput(ctx)
+	if err != nil {
+		return fmt.Errorf("rebuild system instance input: %w", err)
+	}
+	if _, err := d.writeFwCfgBlobs(inst.ID, input); err != nil {
+		return fmt.Errorf("rewrite fw_cfg blobs: %w", err)
+	}
+	slog.Info("Refreshed system instance state",
+		"instanceId", inst.ID, "managedBy", inst.ManagedBy)
+	return nil
+}
+
 // cleanupFailedSystemInstance handles cleanup when a system instance launch
 // fails after partial setup (state added, volumes created, etc). Delegates
 // to vm.Manager.MarkFailed which runs the synchronous teardown chain

spinifex/daemon/daemon_system_instance_test.go

@@ -1,14 +1,23 @@
 package daemon
 
 import (
+	"encoding/json"
 	"fmt"
 	"os"
+	"path/filepath"
 	"strings"
 	"testing"
 	"time"
 
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/service/ec2"
+	handlers_ec2_vpc "github.com/mulgadc/spinifex/spinifex/handlers/ec2/vpc"
 	"github.com/mulgadc/spinifex/spinifex/handlers/elbv2"
+	"github.com/mulgadc/spinifex/spinifex/tags"
+	"github.com/mulgadc/spinifex/spinifex/testutil"
 	"github.com/mulgadc/spinifex/spinifex/vm"
+	"github.com/nats-io/nats-server/v2/server"
+	"github.com/nats-io/nats.go"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 )
@@ -309,3 +318,264 @@ func TestWriteFwCfgBlobs_InvalidNICs_NoDefault(t *testing.T) {
 			"unexpected tmpfile left behind: %s", e.Name())
 	}
 }
+
+// ---------------------------------------------------------------------------
+// refreshSystemInstanceState
+// ---------------------------------------------------------------------------
+
+// newRefreshTestDaemon returns a Daemon wired with vmMgr + a fresh ELBv2
+// service, with XDG_RUNTIME_DIR pointed at a per-test tmpdir so
+// writeFwCfgBlobs is sandboxed. The returned nats.Conn lets the caller seed
+// LB records via a separate Store handle sharing the same KV bucket.
+func newRefreshTestDaemon(t *testing.T) (*Daemon, *nats.Conn, string) {
+	t.Helper()
+	tmpDir := t.TempDir()
+	t.Setenv("XDG_RUNTIME_DIR", tmpDir)
+	_, nc, _ := testutil.StartTestJetStream(t)
+	svc, err := handlers_elbv2.NewELBv2ServiceImplWithNATS(nil, nc)
+	require.NoError(t, err)
+	t.Cleanup(svc.Close)
+	return &Daemon{vmMgr: vm.NewManager(), elbv2Service: svc}, nc, tmpDir
+}
+
+func TestRefreshSystemInstanceState_NonELBv2IsNoop(t *testing.T) {
+	d := &Daemon{vmMgr: vm.NewManager()}
+	require.NoError(t, d.refreshSystemInstanceState(&vm.VM{ID: "i-customer", ManagedBy: ""}))
+}
+
+func TestRefreshSystemInstanceState_NilELBv2ServiceErrors(t *testing.T) {
+	d := &Daemon{vmMgr: vm.NewManager(), elbv2Service: nil}
+	err := d.refreshSystemInstanceState(&vm.VM{ID: "i-no-svc", ManagedBy: tags.ManagedByELBv2})
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "elbv2 service unavailable")
+}
+
+func TestRefreshSystemInstanceState_NoLBRecord(t *testing.T) {
+	d, _, tmpDir := newRefreshTestDaemon(t)
+	inst := &vm.VM{ID: "i-orphan-alb", ManagedBy: tags.ManagedByELBv2, InstanceType: "sys.micro"}
+
+	err := d.refreshSystemInstanceState(inst)
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "rebuild system instance input")
+	assert.Contains(t, err.Error(), "no LB record references instance i-orphan-alb")
+
+	entries, _ := os.ReadDir(tmpDir)
+	for _, e := range entries {
+		assert.False(t, strings.HasPrefix(e.Name(), "fwcfg-i-orphan-alb"),
+			"stale tmpfile left behind on rebuild failure: %s", e.Name())
+	}
+}
+
+func TestRefreshSystemInstanceState_RewritesBlobs(t *testing.T) {
+	d, nc, tmpDir := newRefreshTestDaemon(t)
+	d.elbv2Service.SystemAccessKey = "AKID-recover"
+	d.elbv2Service.SystemSecretKey = "SECRET-recover"
+	d.elbv2Service.GatewayURL = "https://10.0.0.1:9999"
+	d.elbv2Service.CACert = "-----BEGIN CERTIFICATE-----\nfake-ca\n-----END CERTIFICATE-----\n"
+
+	seedStore, err := handlers_elbv2.NewStore(nc)
+	require.NoError(t, err)
+	require.NoError(t, seedStore.PutLoadBalancer(&handlers_elbv2.LoadBalancerRecord{
+		LoadBalancerID: "lb-recover",
+		Name:           "recover-alb",
+		Scheme:         handlers_elbv2.SchemeInternal,
+		Type:           handlers_elbv2.LoadBalancerTypeApplication,
+		State:          handlers_elbv2.StateActive,
+		Subnets:        []string{"subnet-aaa"},
+		ENIs:           []string{"eni-primary"},
+		InstanceID:     "i-recover-blobs",
+		VPCIP:          "10.0.1.5",
+		AccountID:      "123456789012",
+	}))
+
+	inst := &vm.VM{
+		ID:           "i-recover-blobs",
+		ManagedBy:    tags.ManagedByELBv2,
+		InstanceType: "sys.micro",
+		ENIMac:       "02:aa:bb:cc:dd:01",
+		MgmtMAC:      "02:a0:00:11:22:33",
+		MgmtIP:       "172.31.0.7",
+	}
+	require.NoError(t, d.refreshSystemInstanceState(inst))
+
+	netcfg, err := os.ReadFile(filepath.Join(tmpDir, "fwcfg-i-recover-blobs-netcfg.tmp"))
+	require.NoError(t, err)
+	assert.Contains(t, string(netcfg), "NIC0_MAC=02:aa:bb:cc:dd:01")
+	assert.Contains(t, string(netcfg), "NIC1_MAC=02:a0:00:11:22:33")
+
+	lbenv, err := os.ReadFile(filepath.Join(tmpDir, "fwcfg-i-recover-blobs-lbenv.tmp"))
+	require.NoError(t, err)
+	assert.Contains(t, string(lbenv), "LB_LB_ID=lb-recover")
+	assert.Contains(t, string(lbenv), "LB_ACCESS_KEY=AKID-recover")
+
+	cacert, err := os.ReadFile(filepath.Join(tmpDir, "fwcfg-i-recover-blobs-cacert.tmp"))
+	require.NoError(t, err)
+	assert.Contains(t, string(cacert), "fake-ca")
+}
+
+// TestRefreshSystemInstanceState_WriteErrorPropagates exercises the
+// writeFwCfgBlobs error path (plan §3): a write failure under the tmpfs
+// path must surface as a wrapped error so MarkRecoveryFailed gets called.
+func TestRefreshSystemInstanceState_WriteErrorPropagates(t *testing.T) {
+	d, nc, tmpDir := newRefreshTestDaemon(t)
+
+	seedStore, err := handlers_elbv2.NewStore(nc)
+	require.NoError(t, err)
+	require.NoError(t, seedStore.PutLoadBalancer(&handlers_elbv2.LoadBalancerRecord{
+		LoadBalancerID: "lb-werr",
+		Name:           "werr-alb",
+		Scheme:         handlers_elbv2.SchemeInternal,
+		Type:           handlers_elbv2.LoadBalancerTypeApplication,
+		State:          handlers_elbv2.StateActive,
+		Subnets:        []string{"subnet-aaa"},
+		ENIs:           []string{"eni-primary"},
+		InstanceID:     "i-werr",
+		VPCIP:          "10.0.1.5",
+		AccountID:      "123456789012",
+	}))
+
+	// Replace XDG_RUNTIME_DIR with a regular file so os.WriteFile in
+	// writeFwCfgBlobs fails with ENOTDIR. tmpDir from the helper is no
+	// longer the runtime dir for this test.
+	_ = tmpDir
+	notADir := filepath.Join(t.TempDir(), "blocking-file")
+	require.NoError(t, os.WriteFile(notADir, nil, 0600))
+	t.Setenv("XDG_RUNTIME_DIR", notADir)
+
+	inst := &vm.VM{
+		ID:           "i-werr",
+		ManagedBy:    tags.ManagedByELBv2,
+		InstanceType: "sys.micro",
+		ENIMac:       "02:aa:bb:cc:dd:01",
+		MgmtMAC:      "02:a0:00:11:22:33",
+		MgmtIP:       "172.31.0.7",
+	}
+	err = d.refreshSystemInstanceState(inst)
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "rewrite fw_cfg blobs")
+}
+
+// TestLaunchSystemInstance_NATFailureRollsBackPublicIP regresses the silent
+// corruption where an internet-facing ALB launch committed the IPAM
+// allocation and surfaced the public IP on the ENI record even when
+// vpc.add-nat failed — the OVN dnat_and_snat rule was never created and the
+// public IP black-holed with no reconciler. With the fix, NAT NACK must roll
+// back the ENI public IP and release the IPAM lease before returning.
+func TestLaunchSystemInstance_NATFailureRollsBackPublicIP(t *testing.T) {
+	d := createVPCTestDaemon(t)
+
+	// Wire an ExternalIPAM against a fresh JS server. The pool gives us a
+	// known IP that the rollback must release back.
+	jsNS, err := server.NewServer(&server.Options{
+		Host:      "127.0.0.1",
+		Port:      -1,
+		JetStream: true,
+		StoreDir:  t.TempDir(),
+		NoLog:     true,
+		NoSigs:    true,
+	})
+	require.NoError(t, err)
+	go jsNS.Start()
+	require.True(t, jsNS.ReadyForConnections(5*time.Second))
+	t.Cleanup(func() { jsNS.Shutdown() })
+
+	jsNC, err := nats.Connect(jsNS.ClientURL())
+	require.NoError(t, err)
+	t.Cleanup(func() { jsNC.Close() })
+
+	js, err := jsNC.JetStream()
+	require.NoError(t, err)
+	ipam, err := handlers_ec2_vpc.NewExternalIPAM(jsNC, js, []handlers_ec2_vpc.ExternalPoolConfig{
+		{Name: "wan-test", RangeStart: "203.0.113.10", RangeEnd: "203.0.113.20", Gateway: "203.0.113.1", PrefixLen: 24},
+	})
+	require.NoError(t, err)
+	d.externalIPAM = ipam
+
+	// Create a VPC + subnet + ENI so LaunchSystemInstance has a real ENI to
+	// attach to and to update with the public IP.
+	vpcOut, err := d.vpcService.CreateVpc(&ec2.CreateVpcInput{
+		CidrBlock: aws.String("10.50.0.0/16"),
+	}, testAccountID)
+	require.NoError(t, err)
+	subnetOut, err := d.vpcService.CreateSubnet(&ec2.CreateSubnetInput{
+		VpcId:     vpcOut.Vpc.VpcId,
+		CidrBlock: aws.String("10.50.1.0/24"),
+	}, testAccountID)
+	require.NoError(t, err)
+	eniOut, err := d.vpcService.CreateNetworkInterface(&ec2.CreateNetworkInterfaceInput{
+		SubnetId:    subnetOut.Subnet.SubnetId,
+		Description: aws.String("nat-fail-test"),
+	}, testAccountID)
+	require.NoError(t, err)
+	eniID := aws.StringValue(eniOut.NetworkInterface.NetworkInterfaceId)
+	eniMac := aws.StringValue(eniOut.NetworkInterface.MacAddress)
+	eniIP := aws.StringValue(eniOut.NetworkInterface.PrivateIpAddress)
+
+	// Stand up a vpcd-shaped NACK responder on the daemon's NATS conn —
+	// utils.AddNAT publishes on d.natsConn, so the responder must live there.
+	sub, err := d.natsConn.Subscribe("vpc.add-nat", func(msg *nats.Msg) {
+		_ = msg.Respond([]byte(`{"success":false,"error":"northd unavailable"}`))
+	})
+	require.NoError(t, err)
+	t.Cleanup(func() { _ = sub.Unsubscribe() })
+
+	// Capture vpc.delete-nat so we can assert the rollback neutralises any
+	// rule vpcd may have committed after the AddNAT timeout window.
+	deleteNATCh := make(chan map[string]string, 1)
+	delSub, err := d.natsConn.Subscribe("vpc.delete-nat", func(msg *nats.Msg) {
+		var p map[string]string
+		_ = json.Unmarshal(msg.Data, &p)
+		select {
+		case deleteNATCh <- p:
+		default:
+		}
+	})
+	require.NoError(t, err)
+	t.Cleanup(func() { _ = delSub.Unsubscribe() })
+
+	input := &handlers_elbv2.SystemInstanceInput{
+		InstanceType: getTestInstanceType(t),
+		SubnetID:     aws.StringValue(subnetOut.Subnet.SubnetId),
+		Scheme:       handlers_elbv2.SchemeInternetFacing,
+		AccountID:    testAccountID,
+		ENIID:        eniID,
+		ENIMac:       eniMac,
+		ENIIP:        eniIP,
+	}
+
+	out, err := d.LaunchSystemInstance(input)
+	require.Error(t, err, "NAT NACK must fail the launch")
+	assert.Nil(t, out, "no SystemInstanceOutput should be returned on NAT failure")
+	assert.Contains(t, err.Error(), "northd unavailable", "error must propagate the vpcd reason")
+
+	// ENI public IP record must be cleared so subsequent DescribeNetworkInterfaces
+	// does not surface the unreachable address.
+	descOut, err := d.vpcService.DescribeNetworkInterfaces(&ec2.DescribeNetworkInterfacesInput{
+		NetworkInterfaceIds: []*string{aws.String(eniID)},
+	}, testAccountID)
+	require.NoError(t, err)
+	require.Len(t, descOut.NetworkInterfaces, 1)
+	// Association is built only when the underlying record has a public IP;
+	// rollback clears that field, so the helper omits Association entirely.
+	assert.Nil(t, descOut.NetworkInterfaces[0].Association,
+		"ENI must not carry the unreachable public IP after rollback")
+
+	// IPAM lease must be back in the pool so the next allocation can reuse
+	// it. .10 is the gateway (reserved at init); the first allocable IP is
+	// .11, which is what the launch grabbed and the rollback released.
+	rec, err := ipam.GetPoolRecord("wan-test")
+	require.NoError(t, err)
+	_, stillAllocated := rec.Allocated["203.0.113.11"]
+	assert.False(t, stillAllocated, "rollback must release the allocated IP back to the pool")
+
+	// vpc.delete-nat must be published with the same external IP that AddNAT
+	// targeted so vpcd reaps any rule committed after the request timeout.
+	select {
+	case got := <-deleteNATCh:
+		assert.Equal(t, "203.0.113.11", got["external_ip"])
+		assert.Equal(t, eniIP, got["logical_ip"])
+		assert.Equal(t, "port-"+eniID, got["port_name"])
+	case <-time.After(time.Second):
+		t.Fatal("rollback must publish vpc.delete-nat to neutralise a half-committed rule")
+	}
+}

spinifex/daemon/daemon_test.go

@@ -21,6 +21,7 @@ import (
 	"runtime"
 	"strings"
 	"sync"
+	"sync/atomic"
 	"testing"
 	"time"
 
@@ -937,6 +938,84 @@ func TestCanAllocate_CountEdgeCases(t *testing.T) {
 	})
 }
 
+// TestAllocate_NoOvercommitUnderContention exercises the TOCTOU window that
+// the pre-fix code had between canAllocate's RUnlock and the subsequent
+// mu.Lock(): N concurrent allocate() callers on a pool with capacity for
+// exactly N-1 must produce exactly N-1 successes and 1 insufficient-capacity
+// failure. Run under -race to also catch any regression that re-introduces
+// a check-then-commit gap. Pre-fix this test would intermittently overcommit;
+// post-fix the per-call write-lock acquisition makes it deterministic.
+func TestAllocate_NoOvercommitUnderContention(t *testing.T) {
+	rm, err := NewResourceManager(nil, nil)
+	require.NoError(t, err)
+
+	var microType *ec2.InstanceTypeInfo
+	for key, it := range rm.instanceTypes {
+		if strings.HasSuffix(key, ".micro") {
+			microType = it
+			break
+		}
+	}
+	require.NotNil(t, microType, "test requires a .micro instance type")
+
+	vCPUs := int(instanceTypeVCPUs(microType))
+	memGB := float64(instanceTypeMemoryMiB(microType)) / 1024.0
+	require.Greater(t, vCPUs, 0, "micro type must report vCPU count")
+	require.Greater(t, memGB, float64(0), "micro type must report memory")
+
+	const capacityFor = 8 // slots
+	const goroutines = capacityFor + 1
+	rm.mu.Lock()
+	rm.hostVCPU = vCPUs * capacityFor
+	rm.hostMemGB = memGB * float64(capacityFor)
+	rm.reservedVCPU = 0
+	rm.reservedMem = 0
+	rm.allocatedVCPU = 0
+	rm.allocatedMem = 0
+	rm.mu.Unlock()
+
+	require.Equal(t, capacityFor, rm.canAllocate(microType, goroutines),
+		"setup sanity: pool should have exactly capacityFor slots")
+
+	var (
+		wg           sync.WaitGroup
+		startGate    = make(chan struct{})
+		successes    int64
+		insufficient int64
+		otherErrs    int64
+		errsMu       sync.Mutex
+		seenErrs     []string
+	)
+	for range goroutines {
+		wg.Go(func() {
+			<-startGate
+			if err := rm.allocate(microType); err == nil {
+				atomic.AddInt64(&successes, 1)
+			} else if strings.Contains(err.Error(), "insufficient resources") {
+				atomic.AddInt64(&insufficient, 1)
+			} else {
+				atomic.AddInt64(&otherErrs, 1)
+				errsMu.Lock()
+				seenErrs = append(seenErrs, err.Error())
+				errsMu.Unlock()
+			}
+		})
+	}
+	close(startGate)
+	wg.Wait()
+
+	assert.Equal(t, int64(0), otherErrs, "unexpected error classes: %v", seenErrs)
+	assert.Equal(t, int64(capacityFor), successes, "must commit exactly capacityFor allocations")
+	assert.Equal(t, int64(1), insufficient, "exactly one caller must observe insufficient capacity")
+
+	rm.mu.RLock()
+	finalVCPU := rm.allocatedVCPU
+	finalMem := rm.allocatedMem
+	rm.mu.RUnlock()
+	assert.Equal(t, vCPUs*capacityFor, finalVCPU, "allocated vCPU must not exceed schedulable pool")
+	assert.InDelta(t, memGB*float64(capacityFor), finalMem, 0.001, "allocated memory must not exceed schedulable pool")
+}
+
 // TestDescribeInstances_ReservationGrouping tests that instances are grouped by reservation ID
 func TestDescribeInstances_ReservationGrouping(t *testing.T) {
 	natsURL := sharedNATSURL

spinifex/daemon/daemon_wire_lb_test.go

@@ -1,11 +1,13 @@
 package daemon
 
 import (
+	"strings"
 	"testing"
 
 	"github.com/mulgadc/spinifex/spinifex/config"
 	handlers_elbv2 "github.com/mulgadc/spinifex/spinifex/handlers/elbv2"
 	"github.com/mulgadc/spinifex/spinifex/testutil"
+	"github.com/nats-io/nats.go"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 )
@@ -243,3 +245,52 @@ func TestWireLBAgentConfig_GatewayURL_SingleNodeBindMatchesAdvertise(t *testing.
 	assert.Empty(t, d.elbv2Service.MgmtRouteGateway)
 	assert.Empty(t, d.elbv2Service.MgmtRouteTarget)
 }
+
+// newSubscribeTestDaemon creates a minimal Daemon wired enough for subscribeAll.
+func newSubscribeTestDaemon(t *testing.T, nc *nats.Conn, gatewayURL, accessKey string) *Daemon {
+	t.Helper()
+	svc, err := handlers_elbv2.NewELBv2ServiceImplWithNATS(nil, nc)
+	require.NoError(t, err)
+	t.Cleanup(func() { svc.Close() })
+	svc.GatewayURL = gatewayURL
+
+	return &Daemon{
+		node:              "test-node",
+		config:            &config.Config{},
+		natsConn:          nc,
+		natsSubscriptions: make(map[string]*nats.Subscription),
+		elbv2Service:      svc,
+		systemAccessKey:   accessKey,
+	}
+}
+
+func hasELBv2Subs(subs map[string]*nats.Subscription) bool {
+	for topic := range subs {
+		if strings.HasPrefix(topic, "elbv2.") {
+			return true
+		}
+	}
+	return false
+}
+
+func TestSubscribeAll_SkipsELBv2WhenGatewayURLEmpty(t *testing.T) {
+	_, nc, _ := testutil.StartTestJetStream(t)
+	d := newSubscribeTestDaemon(t, nc, "", "AKID-test")
+
+	err := d.subscribeAll()
+	require.NoError(t, err)
+
+	assert.False(t, hasELBv2Subs(d.natsSubscriptions),
+		"elbv2 subscriptions must be skipped when GatewayURL is empty")
+}
+
+func TestSubscribeAll_RegistersELBv2WhenConfigured(t *testing.T) {
+	_, nc, _ := testutil.StartTestJetStream(t)
+	d := newSubscribeTestDaemon(t, nc, "https://10.0.0.1:9999", "AKID-test")
+
+	err := d.subscribeAll()
+	require.NoError(t, err)
+
+	assert.True(t, hasELBv2Subs(d.natsSubscriptions),
+		"elbv2 subscriptions must be registered when GatewayURL and systemAccessKey are set")
+}

spinifex/daemon/network.go

@@ -6,8 +6,8 @@ import (
 	"net"
 	"os"
 	"os/exec"
-	"os/user"
 	"sort"
+	"strconv"
 	"strings"
 
 	"github.com/mulgadc/spinifex/spinifex/utils"
@@ -55,12 +55,14 @@ func (p *OVSNetworkPlumber) SetupTap(spec vm.TapSpec) error {
 	// (drivers/net/tun.c: tun_chr_open + TUNSETIFF perm check). Without the
 	// `user` flag the tap defaults to root:root and qemu's TUNSETIFF fails
 	// with EPERM ("could not configure /dev/net/tun: Operation not permitted").
-	// Root daemons skip this — they have CAP_NET_ADMIN ambient.
+	// Pass numeric uid/gid directly — user.Current() can fail silently under
+	// hardened systemd units / static builds, dropping the flags and leaving
+	// the tap root-owned (LB microvm in cell-17/tofu-examples). Numeric values
+	// always work and skip the NSS lookup. Root daemons skip this — they have
+	// CAP_NET_ADMIN ambient.
 	addArgs := []string{"tuntap", "add", "dev", spec.Name, "mode", "tap"}
-	if os.Geteuid() != 0 {
-		if u, err := user.Current(); err == nil {
-			addArgs = append(addArgs, "user", u.Username, "group", u.Gid)
-		}
+	if uid := os.Geteuid(); uid != 0 {
+		addArgs = append(addArgs, "user", strconv.Itoa(uid), "group", strconv.Itoa(os.Getegid()))
 	}
 	if out, err := sudoCommand("ip", addArgs...).CombinedOutput(); err != nil {
 		return fmt.Errorf("create tap %s: %s: %w", spec.Name, strings.TrimSpace(string(out)), err)

spinifex/daemon/resource.go

@@ -4,6 +4,7 @@ import (
 	"errors"
 	"fmt"
 	"log/slog"
+	"strconv"
 
 	"github.com/aws/aws-sdk-go/service/ec2"
 	"github.com/mulgadc/spinifex/spinifex/types"
@@ -27,6 +28,30 @@ type hostReserve struct {
 // the daemon itself needs little.
 var defaultHostReserve = hostReserve{vCPU: 2, memGB: 2.0}
 
+// resolveHostReserve returns defaultHostReserve, with vCPU/memGB overridden
+// when SPINIFEX_RESERVED_VCPU / SPINIFEX_RESERVED_MEM_GB are set to valid
+// non-negative values. Invalid values are logged and ignored — keeps a
+// typo'd env from silently widening the reserve. Intended as a stop-gap
+// until the planned operator-tunable [capacity] config lands.
+func resolveHostReserve(getenv func(string) string) hostReserve {
+	r := defaultHostReserve
+	if v := getenv("SPINIFEX_RESERVED_VCPU"); v != "" {
+		if n, err := strconv.Atoi(v); err == nil && n >= 0 {
+			r.vCPU = n
+		} else {
+			slog.Warn("ignoring SPINIFEX_RESERVED_VCPU", "value", v, "err", err)
+		}
+	}
+	if v := getenv("SPINIFEX_RESERVED_MEM_GB"); v != "" {
+		if f, err := strconv.ParseFloat(v, 64); err == nil && f >= 0 {
+			r.memGB = f
+		} else {
+			slog.Warn("ignoring SPINIFEX_RESERVED_MEM_GB", "value", v, "err", err)
+		}
+	}
+	return r
+}
+
 // minHostMemHeadroomGB is the minimum schedulable memory we require above
 // the reserve, so a host that just meets the reserve still has a small
 // amount left to launch the smallest guest type.

spinifex/daemon/resource_test.go

@@ -373,3 +373,72 @@ func TestAllocateForLaunch(t *testing.T) {
 		})
 	}
 }
+
+func TestResolveHostReserve(t *testing.T) {
+	tests := []struct {
+		name     string
+		env      map[string]string
+		wantVCPU int
+		wantMem  float64
+	}{
+		{
+			name:     "no env returns default",
+			env:      map[string]string{},
+			wantVCPU: defaultHostReserve.vCPU,
+			wantMem:  defaultHostReserve.memGB,
+		},
+		{
+			name:     "vCPU override",
+			env:      map[string]string{"SPINIFEX_RESERVED_VCPU": "1"},
+			wantVCPU: 1,
+			wantMem:  defaultHostReserve.memGB,
+		},
+		{
+			name:     "memory override",
+			env:      map[string]string{"SPINIFEX_RESERVED_MEM_GB": "0.5"},
+			wantVCPU: defaultHostReserve.vCPU,
+			wantMem:  0.5,
+		},
+		{
+			name: "both overrides",
+			env: map[string]string{
+				"SPINIFEX_RESERVED_VCPU":   "0",
+				"SPINIFEX_RESERVED_MEM_GB": "1.5",
+			},
+			wantVCPU: 0,
+			wantMem:  1.5,
+		},
+		{
+			name:     "invalid vCPU falls back to default",
+			env:      map[string]string{"SPINIFEX_RESERVED_VCPU": "garbage"},
+			wantVCPU: defaultHostReserve.vCPU,
+			wantMem:  defaultHostReserve.memGB,
+		},
+		{
+			name:     "negative vCPU falls back to default",
+			env:      map[string]string{"SPINIFEX_RESERVED_VCPU": "-1"},
+			wantVCPU: defaultHostReserve.vCPU,
+			wantMem:  defaultHostReserve.memGB,
+		},
+		{
+			name:     "invalid memory falls back to default",
+			env:      map[string]string{"SPINIFEX_RESERVED_MEM_GB": "not-a-number"},
+			wantVCPU: defaultHostReserve.vCPU,
+			wantMem:  defaultHostReserve.memGB,
+		},
+		{
+			name:     "negative memory falls back to default",
+			env:      map[string]string{"SPINIFEX_RESERVED_MEM_GB": "-0.5"},
+			wantVCPU: defaultHostReserve.vCPU,
+			wantMem:  defaultHostReserve.memGB,
+		},
+	}
+
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			got := resolveHostReserve(func(k string) string { return tc.env[k] })
+			assert.Equal(t, tc.wantVCPU, got.vCPU)
+			assert.InDelta(t, tc.wantMem, got.memGB, 0.001)
+		})
+	}
+}

spinifex/daemon/shutdown.go

@@ -7,7 +7,6 @@ import (
 	"os"
 	"os/exec"
 	"path/filepath"
-	"time"
 
 	"github.com/mulgadc/spinifex/spinifex/utils"
 	"github.com/nats-io/nats.go"
@@ -263,8 +262,6 @@ func (d *Daemon) handleShutdownInfra(msg *nats.Msg) {
 
 	slog.Info("Shutdown INFRA phase complete, exiting", "node", d.node)
 
-	// Give a moment for log flush
-	time.Sleep(500 * time.Millisecond)
 	os.Exit(0)
 }
 

spinifex/daemon/vm_adapters.go

@@ -412,6 +412,29 @@ func (d *Daemon) onInstanceUpHook() func(*vm.VM) error {
 				}
 			}
 		}
+
+		// Republish vpc.add-nat so vpcd re-establishes the dnat_and_snat
+		// rule on host-reboot recovery. The OVN NB entry doesn't always
+		// survive the reboot, and vpcd's reconcile loop only re-creates
+		// EIP-allocated NATs (reconcile.go:391) — direct-add NATs from
+		// LaunchInstance / LaunchSystemInstance are otherwise unrecoverable.
+		// Idempotent: vpcd's handler deletes-then-adds (topology.go:1297),
+		// so this is a no-op on fresh launches where the initial publish
+		// already fired.
+		if d.natsConn != nil && instance.PublicIP != "" && instance.ENIId != "" && instance.Instance != nil {
+			vpcID := ""
+			privateIP := ""
+			if instance.Instance.VpcId != nil {
+				vpcID = *instance.Instance.VpcId
+			}
+			if instance.Instance.PrivateIpAddress != nil {
+				privateIP = *instance.Instance.PrivateIpAddress
+			}
+			if vpcID != "" && privateIP != "" {
+				portName := "port-" + instance.ENIId
+				utils.PublishNATEvent(d.natsConn, "vpc.add-nat", vpcID, instance.PublicIP, privateIP, portName, instance.ENIMac)
+			}
+		}
 		return nil
 	}
 }
@@ -456,9 +479,10 @@ func (d *Daemon) buildVMManagerDeps() vm.Deps {
 		VolumeStateUpdater: d.volumeService,
 		InstanceCleaner:    newInstanceCleanerAdapter(d),
 		Hooks: vm.ManagerHooks{
-			OnInstanceUp:         d.onInstanceUpHook(),
-			OnInstanceDown:       d.onInstanceDownHook(),
-			OnInstanceRecovering: d.onInstanceRecoveringHook(),
+			OnInstanceUp:           d.onInstanceUpHook(),
+			OnInstanceDown:         d.onInstanceDownHook(),
+			OnInstanceRecovering:   d.onInstanceRecoveringHook(),
+			BeforeInstanceRelaunch: d.refreshSystemInstanceState,
 		},
 		ShutdownSignal:             d.shuttingDown.Load,
 		CrashHandler:               d.vmMgr.HandleCrash,

spinifex/daemon/vm_adapters_test.go

@@ -2,9 +2,12 @@ package daemon
 
 import (
 	"encoding/json"
+	"reflect"
 	"testing"
 
+	"github.com/mulgadc/spinifex/spinifex/config"
 	"github.com/mulgadc/spinifex/spinifex/gpu"
+	"github.com/mulgadc/spinifex/spinifex/tags"
 	"github.com/mulgadc/spinifex/spinifex/types"
 	"github.com/mulgadc/spinifex/spinifex/vm"
 	"github.com/nats-io/nats.go"
@@ -295,3 +298,21 @@ func TestReleaseGPU_ManagerError_LogsWarning(t *testing.T) {
 	// Must not panic; the error is logged as a warning.
 	a.ReleaseGPU(instance)
 }
+
+// TestBuildVMManagerDeps_WiresBeforeInstanceRelaunch guards the single line
+// in buildVMManagerDeps that routes the recovery hook to
+// refreshSystemInstanceState. Dropping it would surface only in cell-18.
+func TestBuildVMManagerDeps_WiresBeforeInstanceRelaunch(t *testing.T) {
+	d := &Daemon{config: &config.Config{}, vmMgr: vm.NewManager()}
+	deps := d.buildVMManagerDeps()
+	require.NotNil(t, deps.Hooks.BeforeInstanceRelaunch)
+
+	wantPC := reflect.ValueOf(d.refreshSystemInstanceState).Pointer()
+	gotPC := reflect.ValueOf(deps.Hooks.BeforeInstanceRelaunch).Pointer()
+	assert.Equal(t, wantPC, gotPC, "hook must point at refreshSystemInstanceState")
+
+	// Sanity: the wired hook is callable and returns nil for non-ELBv2 VMs.
+	require.NoError(t, deps.Hooks.BeforeInstanceRelaunch(&vm.VM{ID: "i-noop", ManagedBy: ""}))
+	require.Error(t, deps.Hooks.BeforeInstanceRelaunch(&vm.VM{ID: "i-svc", ManagedBy: tags.ManagedByELBv2}),
+		"ELBv2 VM with nil elbv2Service must error rather than silently no-op")
+}

spinifex/daemon/vpc_adapter.go

@@ -47,6 +47,10 @@ func (a *daemonENICreator) AttachENI(accountID, eniID, instanceID string, device
 	return a.d.vpcService.AttachENI(accountID, eniID, instanceID, deviceIndex)
 }
 
+func (a *daemonENICreator) DetachENI(accountID, eniID string) error {
+	return a.d.vpcService.DetachENI(accountID, eniID)
+}
+
 func (a *daemonENICreator) UpdateENIPublicIP(accountID, eniID, publicIP, poolName string) error {
 	return a.d.vpcService.UpdateENIPublicIP(accountID, eniID, publicIP, poolName)
 }

spinifex/formation/formation.go

@@ -13,6 +13,8 @@ import (
 	"net/http"
 	"sync"
 	"time"
+
+	"github.com/mulgadc/spinifex/internal/tlsconfig"
 )
 
 // NodeInfo describes a node participating in cluster formation.
@@ -233,7 +235,9 @@ func (fs *FormationServer) Start(bindAddr string) error {
 		IdleTimeout:       30 * time.Second,
 		ReadHeaderTimeout: 5 * time.Second,
 		TLSConfig: &tls.Config{
-			Certificates: []tls.Certificate{tlsCert},
+			Certificates:     []tls.Certificate{tlsCert},
+			MinVersion:       tls.VersionTLS13,
+			CurvePreferences: tlsconfig.Curves,
 		},
 	}
 

spinifex/gateway/auth.go

@@ -3,15 +3,13 @@ package gateway
 import (
 	"bytes"
 	"context"
-	"crypto/subtle"
+	"crypto/sha256"
+	"encoding/hex"
 	"errors"
-	"fmt"
 	"io"
 	"log/slog"
 	"net/http"
-	"sort"
 	"strings"
-	"time"
 
 	"github.com/google/uuid"
 	"github.com/mulgadc/predastore/auth"
@@ -19,13 +17,8 @@ import (
 	handlers_iam "github.com/mulgadc/spinifex/spinifex/handlers/iam"
 )
 
-const (
-	// Maximum allowed clock skew for signature validation (5 minutes)
-	maxClockSkew = 5 * time.Minute
-
-	// Maximum request body size for signature validation (10 MB)
-	maxBodySize = 10 * 1024 * 1024
-)
+// Maximum request body size for signature validation (10 MB).
+const maxBodySize = 10 * 1024 * 1024
 
 // SigV4AuthMiddleware returns stdlib middleware that validates AWS Signature V4 authentication.
 func (gw *GatewayConfig) SigV4AuthMiddleware() func(http.Handler) http.Handler {
@@ -34,147 +27,66 @@ func (gw *GatewayConfig) SigV4AuthMiddleware() func(http.Handler) http.Handler {
 	}
 	return func(next http.Handler) http.Handler {
 		return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-			// Rate-limit check: reject locked-out IPs before any crypto work.
 			clientIP := extractClientIP(r)
 			if errCode := gw.RateLimiter.CheckIP(clientIP); errCode != "" {
 				gw.writeSigV4Error(w, r, errCode)
 				return
 			}
 
-			authHeader := r.Header.Get("Authorization")
-			if authHeader == "" {
-				gw.writeSigV4Error(w, r, awserrors.ErrorMissingAuthenticationToken)
+			sig, err := auth.ParseReq(r)
+			if err != nil {
+				gw.writeSigV4Error(w, r, parseSigV4ErrorCode(err))
 				return
 			}
 
-			// Parse the Authorization header
-			// Format: AWS4-HMAC-SHA256 Credential=ACCESS_KEY/DATE/REGION/SERVICE/aws4_request, SignedHeaders=..., Signature=...
-			if !strings.HasPrefix(authHeader, "AWS4-HMAC-SHA256 ") {
-				gw.writeSigV4Error(w, r, awserrors.ErrorIncompleteSignature)
+			// Reject services the gateway doesn't serve before any crypto:
+			// otherwise Verify re-signs with the client-claimed service and
+			// rubber-stamps the scope.
+			if !supportedServices[sig.Service] {
+				gw.writeSigV4Error(w, r, awserrors.ErrorSignatureDoesNotMatch)
 				return
 			}
 
-			// Extract components from the Authorization header
-			parts := strings.Split(authHeader[len("AWS4-HMAC-SHA256 "):], ", ")
-			if len(parts) != 3 {
-				gw.writeSigV4Error(w, r, awserrors.ErrorIncompleteSignature)
-				return
-			}
-
-			// Parse Credential
-			var accessKey, date, region, service string
-			if after, ok := strings.CutPrefix(parts[0], "Credential="); ok {
-				credParts := strings.Split(after, "/")
-				if len(credParts) != 5 || credParts[4] != "aws4_request" {
-					gw.writeSigV4Error(w, r, awserrors.ErrorIncompleteSignature)
-					return
-				}
-				accessKey = credParts[0]
-				date = credParts[1]
-				region = credParts[2]
-				service = credParts[3]
-			} else {
-				gw.writeSigV4Error(w, r, awserrors.ErrorIncompleteSignature)
-				return
-			}
-
-			// Parse SignedHeaders
-			var signedHeaders string
-			if after, ok := strings.CutPrefix(parts[1], "SignedHeaders="); ok {
-				signedHeaders = after
-			} else {
-				gw.writeSigV4Error(w, r, awserrors.ErrorIncompleteSignature)
-				return
-			}
-
-			// Enforce that host and x-amz-date are signed. AWS SDKs always sign
-			// both; omitting either lets a captured Authorization header replay
-			// against a different vhost or outside the X-Amz-Date skew window.
-			// Mirrors predastore/auth.RequireSignedHeaders.
-			if !signedHeadersIncludeHostAndDate(signedHeaders) {
-				gw.writeSigV4Error(w, r, awserrors.ErrorIncompleteSignature)
-				return
-			}
-
-			// Parse Signature
-			var providedSignature string
-			if after, ok := strings.CutPrefix(parts[2], "Signature="); ok {
-				providedSignature = after
-			} else {
-				gw.writeSigV4Error(w, r, awserrors.ErrorIncompleteSignature)
-				return
-			}
-
-			// Fail fast when NATS is down. IAMService.LookupAccessKey runs against
-			// JetStream KV and waits 5s per attempt before context.DeadlineExceeded;
-			// the catch-all IsConnected check in gateway.Request() is unreachable
-			// because this middleware runs first. Mirror it here so authenticated
-			// paths return the cluster-unavailable 503 promised by 1c. Only fires
-			// when NATSConn is set but disconnected — production always assigns
-			// the conn in awsgw.launchService; test helpers leaving it nil get
-			// the existing pre-1c behaviour.
+			// Fail fast when NATS is down: LookupAccessKey waits 5s per attempt
+			// before context.DeadlineExceeded, and gateway.Request()'s catch-all
+			// IsConnected check is unreachable from here. Nil NATSConn (test
+			// helpers) skips the short-circuit.
 			if gw.NATSConn != nil && !gw.NATSConn.IsConnected() {
-				gw.writeClusterUnavailable(w, r, service)
+				gw.writeClusterUnavailable(w, r, sig.Service)
 				return
 			}
 
-			// Lookup access key in IAM KV store
 			if gw.IAMService == nil {
 				slog.Error("SigV4 auth: IAM service not initialized")
 				gw.writeSigV4Error(w, r, awserrors.ErrorInternalError)
 				return
 			}
-
-			ak, err := gw.IAMService.LookupAccessKey(accessKey)
+			ak, err := gw.IAMService.LookupAccessKey(sig.AccessKeyID)
 			if err != nil {
 				if strings.Contains(err.Error(), awserrors.ErrorIAMNoSuchEntity) {
-					slog.Warn("Auth failure: access key not found", "accessKeyID", accessKey, "sourceIP", clientIP)
+					slog.Warn("Auth failure: access key not found", "accessKeyID", sig.AccessKeyID, "sourceIP", clientIP)
 					gw.RateLimiter.RecordFailure(clientIP)
 					gw.writeSigV4Error(w, r, awserrors.ErrorInvalidClientTokenId)
 					return
 				}
-				slog.Error("IAM lookup failed", "accessKeyID", accessKey, "err", err)
+				slog.Error("IAM lookup failed", "accessKeyID", sig.AccessKeyID, "err", err)
 				gw.writeSigV4Error(w, r, awserrors.ErrorInternalError)
 				return
 			}
 			if ak.Status != handlers_iam.AccessKeyStatusActive {
-				slog.Warn("Auth failure: access key inactive", "accessKeyID", accessKey, "sourceIP", clientIP)
+				slog.Warn("Auth failure: access key inactive", "accessKeyID", sig.AccessKeyID, "sourceIP", clientIP)
 				gw.RateLimiter.RecordFailure(clientIP)
 				gw.writeSigV4Error(w, r, awserrors.ErrorInvalidClientTokenId)
 				return
 			}
-
 			secret, err := gw.IAMService.DecryptSecret(ak.SecretAccessKey)
 			if err != nil {
-				slog.Error("Failed to decrypt IAM secret", "accessKeyID", accessKey, "err", err)
+				slog.Error("Failed to decrypt IAM secret", "accessKeyID", sig.AccessKeyID, "err", err)
 				gw.writeSigV4Error(w, r, awserrors.ErrorInternalError)
 				return
 			}
 
-			// Get timestamp from X-Amz-Date header
-			timestamp := r.Header.Get("X-Amz-Date")
-			if timestamp == "" {
-				gw.writeSigV4Error(w, r, awserrors.ErrorIncompleteSignature)
-				return
-			}
-
-			// Validate timestamp is within acceptable bounds to prevent replay attacks
-			parsedTime, err := time.Parse("20060102T150405Z", timestamp)
-			if err != nil {
-				slog.Debug("Invalid timestamp format", "timestamp", timestamp)
-				gw.writeSigV4Error(w, r, awserrors.ErrorIncompleteSignature)
-				return
-			}
-			if time.Since(parsedTime).Abs() > maxClockSkew {
-				slog.Debug("Signature expired", "timestamp", timestamp, "skew", time.Since(parsedTime))
-				gw.writeSigV4Error(w, r, awserrors.ErrorSignatureDoesNotMatch)
-				return
-			}
-
-			// Limit request body size to prevent OOM from unauthenticated requests
 			r.Body = http.MaxBytesReader(w, r.Body, maxBodySize)
-
-			// Read body once and re-buffer for downstream handlers
 			body, err := io.ReadAll(r.Body)
 			if err != nil {
 				var maxBytesErr *http.MaxBytesError
@@ -189,27 +101,32 @@ func (gw *GatewayConfig) SigV4AuthMiddleware() func(http.Handler) http.Handler {
 			}
 			r.Body = io.NopCloser(bytes.NewReader(body))
 
-			// Compute expected signature using decrypted secret
-			expectedSignature := computeSignatureWithSecret(r, body, secret, date, timestamp, region, service, signedHeaders)
+			sum := sha256.Sum256(body)
+			bodyHash := hex.EncodeToString(sum[:])
 
-			// Compare signatures using constant-time comparison to prevent timing attacks
-			if subtle.ConstantTimeCompare([]byte(expectedSignature), []byte(providedSignature)) != 1 {
-				slog.Warn("Auth failure: signature mismatch",
-					"accessKeyID", accessKey,
+			if err := sig.Verify(secret, sig.Service, gw.Region,
+				auth.WithBodyHash(bodyHash)); err != nil {
+				attrs := []any{
+					"accessKeyID", sig.AccessKeyID,
 					"sourceIP", clientIP,
-				)
+					"err", err,
+				}
+				var sme *auth.SigMismatchError
+				if errors.As(err, &sme) {
+					attrs = append(attrs, "expectedAuthHdr", sme.ExpectedAuthHdr, "providedAuthHdr", sme.ProvidedAuthHdr)
+				}
+				slog.Warn("Auth failure: verification failed", attrs...)
 				gw.RateLimiter.RecordFailure(clientIP)
-				gw.writeSigV4Error(w, r, awserrors.ErrorSignatureDoesNotMatch)
+				gw.writeSigV4Error(w, r, verifySigV4ErrorCode(err))
 				return
 			}
 
-			// Store parsed auth data in context for downstream handlers
 			ctx := r.Context()
 			ctx = context.WithValue(ctx, ctxIdentity, ak.UserName)
 			ctx = context.WithValue(ctx, ctxAccountID, ak.AccountID)
-			ctx = context.WithValue(ctx, ctxService, service)
-			ctx = context.WithValue(ctx, ctxRegion, region)
-			ctx = context.WithValue(ctx, ctxAccessKey, accessKey)
+			ctx = context.WithValue(ctx, ctxService, sig.Service)
+			ctx = context.WithValue(ctx, ctxRegion, sig.Region)
+			ctx = context.WithValue(ctx, ctxAccessKey, sig.AccessKeyID)
 
 			// Parse once; dispatchers reuse via ctxQueryArgs. On error, the
 			// dispatcher re-parses and returns MalformedQueryString.
@@ -220,151 +137,29 @@ func (gw *GatewayConfig) SigV4AuthMiddleware() func(http.Handler) http.Handler {
 				}
 			}
 
-			slog.Debug("SigV4 authentication successful", "accessKey", accessKey, "identity", ak.UserName)
+			slog.Debug("SigV4 authentication successful", "accessKey", sig.AccessKeyID, "identity", ak.UserName)
 			gw.RateLimiter.RecordSuccess(clientIP)
 			next.ServeHTTP(w, r.WithContext(ctx))
 		})
 	}
 }
 
-// computeSignatureWithSecret builds the canonical request and computes the AWS Signature V4 signature
-// using the provided secret key. The body is passed explicitly since it has already been read from r.Body.
-func computeSignatureWithSecret(r *http.Request, body []byte, secretKey, date, timestamp, region, service, signedHeaders string) string {
-	// Build canonical URI (URI-encoded path)
-	path := r.URL.Path
-	if path == "" {
-		path = "/"
+func parseSigV4ErrorCode(err error) string {
+	switch {
+	case errors.Is(err, auth.ErrMissingAuth):
+		return awserrors.ErrorMissingAuthenticationToken
+	default:
+		return awserrors.ErrorIncompleteSignature
 	}
-	canonicalURI := auth.UriEncode(path, false)
-	if canonicalURI == "" {
-		canonicalURI = "/"
-	}
-
-	// Build canonical query string (sorted, encoded)
-	canonicalQueryString := buildCanonicalQueryString(r.URL.RawQuery)
-
-	// Build canonical headers from SignedHeaders list
-	headersList := strings.Split(signedHeaders, ";")
-	sort.Strings(headersList)
-
-	var canonicalHeaders strings.Builder
-	for _, header := range headersList {
-		header = strings.ToLower(strings.TrimSpace(header))
-		var value string
-		if header == "host" {
-			value = r.Host
-		} else {
-			value = r.Header.Get(canonicalHeaderName(header))
-		}
-		// Trim leading/trailing whitespace and collapse multiple spaces
-		value = strings.TrimSpace(value)
-		canonicalHeaders.WriteString(header)
-		canonicalHeaders.WriteString(":")
-		canonicalHeaders.WriteString(value)
-		canonicalHeaders.WriteString("\n")
-	}
-
-	// Hash payload body with SHA256
-	payloadHash := auth.HashSHA256(string(body))
-
-	// Build canonical request
-	canonicalRequest := fmt.Sprintf(
-		"%s\n%s\n%s\n%s\n%s\n%s",
-		r.Method,
-		canonicalURI,
-		canonicalQueryString,
-		canonicalHeaders.String(),
-		signedHeaders,
-		payloadHash,
-	)
-
-	hashedCanonicalRequest := auth.HashSHA256(canonicalRequest)
-
-	// Build string-to-sign
-	scope := fmt.Sprintf("%s/%s/%s/aws4_request", date, region, service)
-	stringToSign := fmt.Sprintf(
-		"AWS4-HMAC-SHA256\n%s\n%s\n%s",
-		timestamp,
-		scope,
-		hashedCanonicalRequest,
-	)
-
-	// Derive signing key and compute signature
-	signingKey := auth.GetSigningKey(secretKey, date, region, service)
-	signature := auth.HmacSHA256Hex(signingKey, stringToSign)
-
-	return signature
 }
 
-// buildCanonicalQueryString creates the canonical query string according to AWS specs.
-func buildCanonicalQueryString(queryString string) string {
-	if queryString == "" {
-		return ""
+func verifySigV4ErrorCode(err error) string {
+	switch {
+	case errors.Is(err, auth.ErrMissingContentSHA):
+		return awserrors.ErrorIncompleteSignature
+	default:
+		return awserrors.ErrorSignatureDoesNotMatch
 	}
-
-	// Parse query parameters
-	params := make(map[string][]string)
-	pairs := strings.SplitSeq(queryString, "&")
-	for pair := range pairs {
-		if pair == "" {
-			continue
-		}
-		kv := strings.SplitN(pair, "=", 2)
-		key := kv[0]
-		value := ""
-		if len(kv) == 2 {
-			value = kv[1]
-		}
-		params[key] = append(params[key], value)
-	}
-
-	// Sort keys
-	keys := make([]string, 0, len(params))
-	for k := range params {
-		keys = append(keys, k)
-	}
-	sort.Strings(keys)
-
-	// Build canonical query string
-	var result []string
-	for _, key := range keys {
-		values := params[key]
-		sort.Strings(values)
-		encodedKey := auth.UriEncode(key, true)
-		for _, v := range values {
-			encodedValue := auth.UriEncode(v, true)
-			result = append(result, fmt.Sprintf("%s=%s", encodedKey, encodedValue))
-		}
-	}
-
-	return strings.Join(result, "&")
-}
-
-// signedHeadersIncludeHostAndDate reports whether the SigV4 SignedHeaders
-// list (";"-separated) includes both "host" and "x-amz-date".
-func signedHeadersIncludeHostAndDate(signedHeaders string) bool {
-	var hasHost, hasDate bool
-	for h := range strings.SplitSeq(signedHeaders, ";") {
-		switch strings.ToLower(strings.TrimSpace(h)) {
-		case "host":
-			hasHost = true
-		case "x-amz-date":
-			hasDate = true
-		}
-	}
-	return hasHost && hasDate
-}
-
-// canonicalHeaderName converts a lowercase header name to the canonical form for lookup.
-func canonicalHeaderName(header string) string {
-	// Convert header names like "x-amz-date" to "X-Amz-Date" for http.Header.Get
-	parts := strings.Split(header, "-")
-	for i, part := range parts {
-		if len(part) > 0 {
-			parts[i] = strings.ToUpper(part[:1]) + part[1:]
-		}
-	}
-	return strings.Join(parts, "-")
 }
 
 // writeSigV4Error writes an EC2-compatible XML error response for authentication failures.

spinifex/gateway/auth_test.go

@@ -1,6 +1,9 @@
 package gateway
 
 import (
+	"bytes"
+	"crypto/sha256"
+	"encoding/hex"
 	"errors"
 	"fmt"
 	"io"
@@ -18,6 +21,7 @@ import (
 	"github.com/mulgadc/spinifex/spinifex/testutil"
 	"github.com/mulgadc/spinifex/spinifex/utils"
 	"github.com/nats-io/nats.go"
+	"github.com/stretchr/testify/require"
 )
 
 const (
@@ -114,67 +118,13 @@ func init() {
 	}
 }
 
-// generateTestAuthHeader creates a valid AWS SigV4 Authorization header for testing.
-func generateTestAuthHeader(method, path, queryString, body, accessKey, secretKey, region, service string) (authHeader, timestamp string) {
-	now := time.Now().UTC()
-	timestamp = now.Format(auth.TimeFormat)
-	date := now.Format(auth.ShortTimeFormat)
-
-	// Build canonical URI
-	canonicalURI := auth.UriEncode(path, false)
-	if canonicalURI == "" {
-		canonicalURI = "/"
-	}
-
-	// Build canonical query string
-	canonicalQueryString := buildCanonicalQueryString(queryString)
-
-	// Build canonical headers
-	host := "localhost:9999"
-	canonicalHeaders := fmt.Sprintf("host:%s\nx-amz-date:%s\n", host, timestamp)
-	signedHeaders := "host;x-amz-date"
-
-	// Hash payload
-	payloadHash := auth.HashSHA256(body)
-
-	// Build canonical request
-	canonicalRequest := fmt.Sprintf(
-		"%s\n%s\n%s\n%s\n%s\n%s",
-		method,
-		canonicalURI,
-		canonicalQueryString,
-		canonicalHeaders,
-		signedHeaders,
-		payloadHash,
-	)
-
-	hashedCanonicalRequest := auth.HashSHA256(canonicalRequest)
-
-	// Build string-to-sign
-	scope := fmt.Sprintf("%s/%s/%s/aws4_request", date, region, service)
-	stringToSign := fmt.Sprintf(
-		"AWS4-HMAC-SHA256\n%s\n%s\n%s",
-		timestamp,
-		scope,
-		hashedCanonicalRequest,
-	)
-
-	// Derive signing key and compute signature
-	signingKey := auth.GetSigningKey(secretKey, date, region, service)
-	signature := auth.HmacSHA256Hex(signingKey, stringToSign)
-
-	// Build Authorization header
-	authHeader = fmt.Sprintf(
-		"AWS4-HMAC-SHA256 Credential=%s/%s/%s/%s/aws4_request, SignedHeaders=%s, Signature=%s",
-		accessKey,
-		date,
-		region,
-		service,
-		signedHeaders,
-		signature,
-	)
-
-	return authHeader, timestamp
+// signTestRequest signs req in place. body must match what the middleware will read from r.Body —
+// its sha256 populates X-Amz-Content-Sha256.
+func signTestRequest(t *testing.T, req *http.Request, body []byte, accessKey, secret string, optFns ...func(*auth.Options)) {
+	t.Helper()
+	sum := sha256.Sum256(body)
+	require.NoError(t, auth.SignReq(req, accessKey, secret,
+		hex.EncodeToString(sum[:]), testService, testRegion, optFns...))
 }
 
 func setupTestApp(accessKey, secretKey string) http.Handler {
@@ -247,7 +197,7 @@ func TestSigV4Auth_MalformedHeader(t *testing.T) {
 			req := httptest.NewRequest(http.MethodGet, "/", nil)
 			req.Host = "localhost:9999"
 			req.Header.Set("Authorization", tc.authHeader)
-			req.Header.Set("X-Amz-Date", time.Now().UTC().Format(auth.TimeFormat))
+			req.Header.Set("X-Amz-Date", time.Now().UTC().Format("20060102T150405Z"))
 
 			resp := doRequest(handler, req)
 
@@ -266,15 +216,9 @@ func TestSigV4Auth_MalformedHeader(t *testing.T) {
 func TestSigV4Auth_InvalidAccessKey(t *testing.T) {
 	handler := setupTestApp(testAccessKey, testSecretKey)
 
-	authHeader, timestamp := generateTestAuthHeader(
-		"GET", "/", "", "",
-		"INVALID_ACCESS_KEY", testSecretKey, testRegion, testService,
-	)
-
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
+	signTestRequest(t, req, nil, "INVALID_ACCESS_KEY", testSecretKey)
 
 	resp := doRequest(handler, req)
 
@@ -291,16 +235,9 @@ func TestSigV4Auth_InvalidAccessKey(t *testing.T) {
 func TestSigV4Auth_InvalidSignature(t *testing.T) {
 	handler := setupTestApp(testAccessKey, testSecretKey)
 
-	// Generate auth header with wrong secret key
-	authHeader, timestamp := generateTestAuthHeader(
-		"GET", "/", "", "",
-		testAccessKey, "WRONG_SECRET_KEY", testRegion, testService,
-	)
-
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
+	signTestRequest(t, req, nil, testAccessKey, "WRONG_SECRET_KEY")
 
 	resp := doRequest(handler, req)
 
@@ -317,15 +254,9 @@ func TestSigV4Auth_InvalidSignature(t *testing.T) {
 func TestSigV4Auth_ValidSignature(t *testing.T) {
 	handler := setupTestApp(testAccessKey, testSecretKey)
 
-	authHeader, timestamp := generateTestAuthHeader(
-		"GET", "/", "", "",
-		testAccessKey, testSecretKey, testRegion, testService,
-	)
-
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
+	signTestRequest(t, req, nil, testAccessKey, testSecretKey)
 
 	resp := doRequest(handler, req)
 
@@ -338,17 +269,11 @@ func TestSigV4Auth_ValidSignature(t *testing.T) {
 func TestSigV4Auth_ValidSignatureWithBody(t *testing.T) {
 	handler := setupTestApp(testAccessKey, testSecretKey)
 
-	body := "Action=DescribeInstances&Version=2016-11-15"
-	authHeader, timestamp := generateTestAuthHeader(
-		"POST", "/", "", body,
-		testAccessKey, testSecretKey, testRegion, testService,
-	)
-
-	req := httptest.NewRequest(http.MethodPost, "/", strings.NewReader(body))
+	body := []byte("Action=DescribeInstances&Version=2016-11-15")
+	req := httptest.NewRequest(http.MethodPost, "/", bytes.NewReader(body))
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
 	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	signTestRequest(t, req, body, testAccessKey, testSecretKey)
 
 	resp := doRequest(handler, req)
 
@@ -362,15 +287,9 @@ func TestSigV4Auth_ValidSignatureWithQueryString(t *testing.T) {
 	handler := setupTestApp(testAccessKey, testSecretKey)
 
 	queryString := "Action=DescribeInstances&Version=2016-11-15"
-	authHeader, timestamp := generateTestAuthHeader(
-		"GET", "/", queryString, "",
-		testAccessKey, testSecretKey, testRegion, testService,
-	)
-
 	req := httptest.NewRequest(http.MethodGet, "/?"+queryString, nil)
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
+	signTestRequest(t, req, nil, testAccessKey, testSecretKey)
 
 	resp := doRequest(handler, req)
 
@@ -410,15 +329,9 @@ func TestSigV4Auth_InactiveAccessKey(t *testing.T) {
 		w.Write([]byte("OK"))
 	})
 
-	authHeader, timestamp := generateTestAuthHeader(
-		"GET", "/", "", "",
-		testAccessKey, testSecretKey, testRegion, testService,
-	)
-
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
+	signTestRequest(t, req, nil, testAccessKey, testSecretKey)
 
 	resp := doRequest(r, req)
 
@@ -445,15 +358,9 @@ func TestSigV4Auth_NilIAMService(t *testing.T) {
 		w.Write([]byte("OK"))
 	})
 
-	authHeader, timestamp := generateTestAuthHeader(
-		"GET", "/", "", "",
-		testAccessKey, testSecretKey, testRegion, testService,
-	)
-
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
+	signTestRequest(t, req, nil, testAccessKey, testSecretKey)
 
 	resp := doRequest(r, req)
 
@@ -499,54 +406,6 @@ func TestParseAWSQueryArgs_URLDecoding(t *testing.T) {
 	}
 }
 
-func TestBuildCanonicalQueryString(t *testing.T) {
-	testCases := []struct {
-		name     string
-		input    string
-		expected string
-	}{
-		{"empty", "", ""},
-		{"single param", "Action=Test", "Action=Test"},
-		{"multiple params sorted", "Version=1&Action=Test", "Action=Test&Version=1"},
-		{"encoded values", "Name=Hello World", "Name=Hello%20World"},
-	}
-
-	for _, tc := range testCases {
-		t.Run(tc.name, func(t *testing.T) {
-			result := buildCanonicalQueryString(tc.input)
-			if result != tc.expected {
-				t.Errorf("Expected %q, got %q", tc.expected, result)
-			}
-		})
-	}
-}
-
-func TestBuildCanonicalQueryString_EdgeCases(t *testing.T) {
-	testCases := []struct {
-		name     string
-		input    string
-		expected string
-	}{
-		{"empty value", "key=", "key="},
-		{"param with multiple equals", "key=a=b", "key=a%3Db"},
-		{"duplicate keys sorted by value", "x=b&x=a", "x=a&x=b"},
-		{"unicode in value", "name=café", "name=caf%C3%A9"},
-		{"unicode in key", "clé=val", "cl%C3%A9=val"},
-		{"long query string", "a=" + strings.Repeat("x", 500), "a=" + strings.Repeat("x", 500)},
-		{"no value (key only)", "key", "key="},
-		{"empty pair skipped", "a=1&&b=2", "a=1&b=2"},
-	}
-
-	for _, tc := range testCases {
-		t.Run(tc.name, func(t *testing.T) {
-			result := buildCanonicalQueryString(tc.input)
-			if result != tc.expected {
-				t.Errorf("Expected %q, got %q", tc.expected, result)
-			}
-		})
-	}
-}
-
 func TestSigV4Auth_DecryptFailure(t *testing.T) {
 	// Encrypt secret with a DIFFERENT master key so decryption fails
 	otherKey, err := handlers_iam.GenerateMasterKey()
@@ -582,15 +441,9 @@ func TestSigV4Auth_DecryptFailure(t *testing.T) {
 		w.Write([]byte("OK"))
 	})
 
-	authHeader, timestamp := generateTestAuthHeader(
-		"GET", "/", "", "",
-		testAccessKey, testSecretKey, testRegion, testService,
-	)
-
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
+	signTestRequest(t, req, nil, testAccessKey, testSecretKey)
 
 	resp := doRequest(r, req)
 
@@ -608,17 +461,11 @@ func TestSigV4Auth_RequestBodyTooLarge(t *testing.T) {
 	handler := setupTestApp(testAccessKey, testSecretKey)
 
 	// Create a body that exceeds maxBodySize (10 MB + 1 byte)
-	oversizedBody := strings.Repeat("x", maxBodySize+1)
-	authHeader, timestamp := generateTestAuthHeader(
-		"POST", "/", "", oversizedBody,
-		testAccessKey, testSecretKey, testRegion, testService,
-	)
-
-	req := httptest.NewRequest(http.MethodPost, "/", strings.NewReader(oversizedBody))
+	oversizedBody := []byte(strings.Repeat("x", maxBodySize+1))
+	req := httptest.NewRequest(http.MethodPost, "/", bytes.NewReader(oversizedBody))
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
 	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	signTestRequest(t, req, oversizedBody, testAccessKey, testSecretKey)
 
 	resp := doRequest(handler, req)
 
@@ -632,113 +479,6 @@ func TestSigV4Auth_RequestBodyTooLarge(t *testing.T) {
 	}
 }
 
-func TestCanonicalHeaderName(t *testing.T) {
-	testCases := []struct {
-		input    string
-		expected string
-	}{
-		{"host", "Host"},
-		{"x-amz-date", "X-Amz-Date"},
-		{"content-type", "Content-Type"},
-		{"x-amz-security-token", "X-Amz-Security-Token"},
-		{"authorization", "Authorization"},
-		{"", ""},
-		{"Host", "Host"},             // already canonical (no-op for uppercase first char)
-		{"X-Amz-Date", "X-Amz-Date"}, // already canonical
-	}
-
-	for _, tc := range testCases {
-		t.Run(tc.input, func(t *testing.T) {
-			result := canonicalHeaderName(tc.input)
-			if result != tc.expected {
-				t.Errorf("Expected %q, got %q", tc.expected, result)
-			}
-		})
-	}
-}
-
-// generateTestAuthHeaderAt creates a valid AWS SigV4 Authorization header at a specific time.
-func generateTestAuthHeaderAt(method, path, queryString, body, accessKey, secretKey, region, service string, at time.Time) (authHeader, timestamp string) {
-	timestamp = at.Format(auth.TimeFormat)
-	date := at.Format(auth.ShortTimeFormat)
-
-	canonicalURI := auth.UriEncode(path, false)
-	if canonicalURI == "" {
-		canonicalURI = "/"
-	}
-
-	canonicalQueryString := buildCanonicalQueryString(queryString)
-
-	host := "localhost:9999"
-	canonicalHeaders := fmt.Sprintf("host:%s\nx-amz-date:%s\n", host, timestamp)
-	signedHeaders := "host;x-amz-date"
-	payloadHash := auth.HashSHA256(body)
-
-	canonicalRequest := fmt.Sprintf(
-		"%s\n%s\n%s\n%s\n%s\n%s",
-		method, canonicalURI, canonicalQueryString, canonicalHeaders, signedHeaders, payloadHash,
-	)
-
-	hashedCanonicalRequest := auth.HashSHA256(canonicalRequest)
-	scope := fmt.Sprintf("%s/%s/%s/aws4_request", date, region, service)
-	stringToSign := fmt.Sprintf("AWS4-HMAC-SHA256\n%s\n%s\n%s", timestamp, scope, hashedCanonicalRequest)
-
-	signingKey := auth.GetSigningKey(secretKey, date, region, service)
-	signature := auth.HmacSHA256Hex(signingKey, stringToSign)
-
-	authHeader = fmt.Sprintf(
-		"AWS4-HMAC-SHA256 Credential=%s/%s/%s/%s/aws4_request, SignedHeaders=%s, Signature=%s",
-		accessKey, date, region, service, signedHeaders, signature,
-	)
-	return authHeader, timestamp
-}
-
-// generateTestAuthHeaderWithSignedHeaders creates a SigV4 header with custom signed headers.
-func generateTestAuthHeaderWithSignedHeaders(method, path, queryString, body, accessKey, secretKey, region, service string, headers map[string]string, signedHeadersList string) (authHeader, timestamp string) {
-	now := time.Now().UTC()
-	timestamp = now.Format(auth.TimeFormat)
-	date := now.Format(auth.ShortTimeFormat)
-
-	canonicalURI := auth.UriEncode(path, false)
-	if canonicalURI == "" {
-		canonicalURI = "/"
-	}
-
-	canonicalQueryString := buildCanonicalQueryString(queryString)
-
-	// Build canonical headers from the signed headers list
-	headerNames := strings.Split(signedHeadersList, ";")
-	var canonicalHeaders strings.Builder
-	for _, h := range headerNames {
-		h = strings.TrimSpace(h)
-		value := headers[h]
-		canonicalHeaders.WriteString(h)
-		canonicalHeaders.WriteString(":")
-		canonicalHeaders.WriteString(value)
-		canonicalHeaders.WriteString("\n")
-	}
-
-	payloadHash := auth.HashSHA256(body)
-
-	canonicalRequest := fmt.Sprintf(
-		"%s\n%s\n%s\n%s\n%s\n%s",
-		method, canonicalURI, canonicalQueryString, canonicalHeaders.String(), signedHeadersList, payloadHash,
-	)
-
-	hashedCanonicalRequest := auth.HashSHA256(canonicalRequest)
-	scope := fmt.Sprintf("%s/%s/%s/aws4_request", date, region, service)
-	stringToSign := fmt.Sprintf("AWS4-HMAC-SHA256\n%s\n%s\n%s", timestamp, scope, hashedCanonicalRequest)
-
-	signingKey := auth.GetSigningKey(secretKey, date, region, service)
-	signature := auth.HmacSHA256Hex(signingKey, stringToSign)
-
-	authHeader = fmt.Sprintf(
-		"AWS4-HMAC-SHA256 Credential=%s/%s/%s/%s/aws4_request, SignedHeaders=%s, Signature=%s",
-		accessKey, date, region, service, signedHeadersList, signature,
-	)
-	return authHeader, timestamp
-}
-
 // --- Clock Skew / Replay Protection Tests ---
 
 func TestSigV4Auth_ExpiredTimestamp(t *testing.T) {
@@ -746,15 +486,9 @@ func TestSigV4Auth_ExpiredTimestamp(t *testing.T) {
 
 	// 6 minutes in the past — exceeds the 5-minute maxClockSkew
 	past := time.Now().UTC().Add(-6 * time.Minute)
-	authHeader, timestamp := generateTestAuthHeaderAt(
-		"GET", "/", "", "",
-		testAccessKey, testSecretKey, testRegion, testService, past,
-	)
-
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
+	signTestRequest(t, req, nil, testAccessKey, testSecretKey, auth.WithTime(past))
 
 	resp := doRequest(handler, req)
 
@@ -773,15 +507,9 @@ func TestSigV4Auth_FutureTimestamp(t *testing.T) {
 
 	// 6 minutes in the future — exceeds the 5-minute maxClockSkew
 	future := time.Now().UTC().Add(6 * time.Minute)
-	authHeader, timestamp := generateTestAuthHeaderAt(
-		"GET", "/", "", "",
-		testAccessKey, testSecretKey, testRegion, testService, future,
-	)
-
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
+	signTestRequest(t, req, nil, testAccessKey, testSecretKey, auth.WithTime(future))
 
 	resp := doRequest(handler, req)
 
@@ -800,15 +528,9 @@ func TestSigV4Auth_TimestampWithinSkew(t *testing.T) {
 
 	// 4 minutes ago — within the 5-minute window
 	recent := time.Now().UTC().Add(-4 * time.Minute)
-	authHeader, timestamp := generateTestAuthHeaderAt(
-		"GET", "/", "", "",
-		testAccessKey, testSecretKey, testRegion, testService, recent,
-	)
-
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
+	signTestRequest(t, req, nil, testAccessKey, testSecretKey, auth.WithTime(recent))
 
 	resp := doRequest(handler, req)
 
@@ -838,15 +560,9 @@ func TestSigV4Auth_ClockSkewBoundary(t *testing.T) {
 	for _, tc := range testCases {
 		t.Run(tc.name, func(t *testing.T) {
 			at := time.Now().UTC().Add(tc.offset)
-			authHeader, timestamp := generateTestAuthHeaderAt(
-				"GET", "/", "", "",
-				testAccessKey, testSecretKey, testRegion, testService, at,
-			)
-
 			req := httptest.NewRequest(http.MethodGet, "/", nil)
 			req.Host = "localhost:9999"
-			req.Header.Set("Authorization", authHeader)
-			req.Header.Set("X-Amz-Date", timestamp)
+			signTestRequest(t, req, nil, testAccessKey, testSecretKey, auth.WithTime(at))
 
 			resp := doRequest(handler, req)
 
@@ -864,15 +580,10 @@ func TestSigV4Auth_ClockSkewBoundary(t *testing.T) {
 func TestSigV4Auth_MissingXAmzDate(t *testing.T) {
 	handler := setupTestApp(testAccessKey, testSecretKey)
 
-	authHeader, _ := generateTestAuthHeader(
-		"GET", "/", "", "",
-		testAccessKey, testSecretKey, testRegion, testService,
-	)
-
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	// Deliberately omit X-Amz-Date
+	signTestRequest(t, req, nil, testAccessKey, testSecretKey)
+	req.Header.Del("X-Amz-Date")
 
 	resp := doRequest(handler, req)
 
@@ -889,14 +600,9 @@ func TestSigV4Auth_MissingXAmzDate(t *testing.T) {
 func TestSigV4Auth_MalformedTimestamp(t *testing.T) {
 	handler := setupTestApp(testAccessKey, testSecretKey)
 
-	authHeader, _ := generateTestAuthHeader(
-		"GET", "/", "", "",
-		testAccessKey, testSecretKey, testRegion, testService,
-	)
-
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
+	signTestRequest(t, req, nil, testAccessKey, testSecretKey)
 	req.Header.Set("X-Amz-Date", "not-a-valid-date")
 
 	resp := doRequest(handler, req)
@@ -1022,15 +728,9 @@ func TestSigV4Auth_ContextPropagation(t *testing.T) {
 		)
 	})
 
-	authHeader, timestamp := generateTestAuthHeader(
-		"GET", "/", "", "",
-		testAccessKey, testSecretKey, testRegion, testService,
-	)
-
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
+	signTestRequest(t, req, nil, testAccessKey, testSecretKey)
 
 	resp := doRequest(r, req)
 
@@ -1106,14 +806,9 @@ func TestSigV4Auth_ContextDoesNotLeakBetweenRequests(t *testing.T) {
 	})
 
 	// First request as alice
-	authHeader1, timestamp1 := generateTestAuthHeader(
-		"GET", "/", "", "",
-		testAccessKey, testSecretKey, testRegion, testService,
-	)
 	req1 := httptest.NewRequest(http.MethodGet, "/", nil)
 	req1.Host = "localhost:9999"
-	req1.Header.Set("Authorization", authHeader1)
-	req1.Header.Set("X-Amz-Date", timestamp1)
+	signTestRequest(t, req1, nil, testAccessKey, testSecretKey)
 
 	resp1 := doRequest(r, req1)
 	body1, _ := io.ReadAll(resp1.Body)
@@ -1122,14 +817,9 @@ func TestSigV4Auth_ContextDoesNotLeakBetweenRequests(t *testing.T) {
 	}
 
 	// Second request as bob
-	authHeader2, timestamp2 := generateTestAuthHeader(
-		"GET", "/", "", "",
-		secondKey, testSecretKey, testRegion, testService,
-	)
 	req2 := httptest.NewRequest(http.MethodGet, "/", nil)
 	req2.Host = "localhost:9999"
-	req2.Header.Set("Authorization", authHeader2)
-	req2.Header.Set("X-Amz-Date", timestamp2)
+	signTestRequest(t, req2, nil, secondKey, testSecretKey)
 
 	resp2 := doRequest(r, req2)
 	body2, _ := io.ReadAll(resp2.Body)
@@ -1154,15 +844,9 @@ func TestSigV4Auth_LookupAccessKeyUnexpectedError(t *testing.T) {
 		w.Write([]byte("OK"))
 	})
 
-	authHeader, timestamp := generateTestAuthHeader(
-		"GET", "/", "", "",
-		testAccessKey, testSecretKey, testRegion, testService,
-	)
-
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
+	signTestRequest(t, req, nil, testAccessKey, testSecretKey)
 
 	resp := doRequest(r, req)
 
@@ -1182,26 +866,19 @@ func TestSigV4Auth_PathWithSpecialCharacters(t *testing.T) {
 	handler := setupTestApp(testAccessKey, testSecretKey)
 
 	testCases := []struct {
-		name     string
-		signPath string // decoded path used for signing
-		reqPath  string // percent-encoded path for the HTTP request line
+		name    string
+		reqPath string
 	}{
-		{"encoded space", "/my resource", "/my%20resource"},
-		{"nested slashes", "/a/b/c/d", "/a/b/c/d"},
-		{"trailing slash", "/path/", "/path/"},
+		{"encoded space", "/my%20resource"},
+		{"nested slashes", "/a/b/c/d"},
+		{"trailing slash", "/path/"},
 	}
 
 	for _, tc := range testCases {
 		t.Run(tc.name, func(t *testing.T) {
-			authHeader, timestamp := generateTestAuthHeader(
-				"GET", tc.signPath, "", "",
-				testAccessKey, testSecretKey, testRegion, testService,
-			)
-
 			req := httptest.NewRequest(http.MethodGet, tc.reqPath, nil)
 			req.Host = "localhost:9999"
-			req.Header.Set("Authorization", authHeader)
-			req.Header.Set("X-Amz-Date", timestamp)
+			signTestRequest(t, req, nil, testAccessKey, testSecretKey)
 
 			resp := doRequest(handler, req)
 
@@ -1244,26 +921,11 @@ func TestSigV4Auth_SignedContentType(t *testing.T) {
 		w.Write([]byte("OK"))
 	})
 
-	body := "Action=DescribeInstances"
-	headers := map[string]string{
-		"content-type": "application/x-www-form-urlencoded",
-		"host":         "localhost:9999",
-	}
-
-	// Include content-type in signed headers
-	now := time.Now().UTC()
-	headers["x-amz-date"] = now.Format(auth.TimeFormat)
-	authHeader, timestamp := generateTestAuthHeaderWithSignedHeaders(
-		"POST", "/", "", body,
-		testAccessKey, testSecretKey, testRegion, testService,
-		headers, "content-type;host;x-amz-date",
-	)
-
-	req := httptest.NewRequest(http.MethodPost, "/", strings.NewReader(body))
+	body := []byte("Action=DescribeInstances")
+	req := httptest.NewRequest(http.MethodPost, "/", bytes.NewReader(body))
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
 	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	signTestRequest(t, req, body, testAccessKey, testSecretKey)
 
 	resp := doRequest(r, req)
 
@@ -1277,15 +939,9 @@ func TestSigV4Auth_EmptyBodyPOST(t *testing.T) {
 	handler := setupTestApp(testAccessKey, testSecretKey)
 
 	// POST with empty body — payload hash is hash of ""
-	authHeader, timestamp := generateTestAuthHeader(
-		"POST", "/", "", "",
-		testAccessKey, testSecretKey, testRegion, testService,
-	)
-
 	req := httptest.NewRequest(http.MethodPost, "/", strings.NewReader(""))
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
+	signTestRequest(t, req, nil, testAccessKey, testSecretKey)
 
 	resp := doRequest(handler, req)
 
@@ -1295,87 +951,14 @@ func TestSigV4Auth_EmptyBodyPOST(t *testing.T) {
 	}
 }
 
-// --- computeSignatureWithSecret Direct Unit Tests ---
-
-// TestComputeSignatureWithSecret_MethodSensitivity pins the contract that
-// the HTTP method participates in the canonical request: changing the method
-// (with all other inputs fixed) must change the signature. Catches a
-// regression class where method is dropped from the canonical-string
-// composition — signing would silently collapse across methods.
-func TestComputeSignatureWithSecret_MethodSensitivity(t *testing.T) {
-	secret := "test-secret-key"
-	date := "20260305"
-	timestamp := "20260305T120000Z"
-	region := "us-east-1"
-	service := "ec2"
-	signedHeaders := "host;x-amz-date"
-
-	methods := []string{"GET", "POST", "PUT", "PATCH", "DELETE"}
-	signatures := make(map[string]string, len(methods))
-
-	for _, method := range methods {
-		req := httptest.NewRequest(method, "/", nil)
-		req.Host = "localhost:9999"
-		req.Header.Set("X-Amz-Date", timestamp)
-
-		signatures[method] = computeSignatureWithSecret(req, nil, secret, date, timestamp, region, service, signedHeaders)
-	}
-
-	for i, m1 := range methods {
-		for _, m2 := range methods[i+1:] {
-			if signatures[m1] == signatures[m2] {
-				t.Errorf("methods %s and %s produced the same signature (method missing from canonical request?)", m1, m2)
-			}
-		}
-	}
-}
-
-func TestComputeSignatureWithSecret_Determinism(t *testing.T) {
-	secret := "test-secret-key"
-	date := "20260305"
-	timestamp := "20260305T120000Z"
-	region := "us-east-1"
-	service := "ec2"
-	signedHeaders := "host;x-amz-date"
-	body := []byte("Action=DescribeInstances")
-
-	var first string
-	for i := range 10 {
-		req := httptest.NewRequest(http.MethodPost, "/", nil)
-		req.Host = "localhost:9999"
-		req.Header.Set("X-Amz-Date", timestamp)
-
-		sig := computeSignatureWithSecret(req, body, secret, date, timestamp, region, service, signedHeaders)
-		if i == 0 {
-			first = sig
-		} else if sig != first {
-			t.Fatalf("Signature not deterministic: iteration %d got %q, expected %q", i, sig, first)
-		}
-	}
-}
-
-func TestComputeSignatureWithSecret_MultipartContentType(t *testing.T) {
+func TestSigV4Auth_MultipartContentType(t *testing.T) {
 	handler := setupTestApp(testAccessKey, testSecretKey)
 
-	body := "Action=DescribeInstances"
-	headers := map[string]string{
-		"content-type": "multipart/form-data; boundary=something",
-		"host":         "localhost:9999",
-	}
-
-	now := time.Now().UTC()
-	headers["x-amz-date"] = now.Format(auth.TimeFormat)
-	authHeader, timestamp := generateTestAuthHeaderWithSignedHeaders(
-		"POST", "/", "", body,
-		testAccessKey, testSecretKey, testRegion, testService,
-		headers, "content-type;host;x-amz-date",
-	)
-
-	req := httptest.NewRequest(http.MethodPost, "/", strings.NewReader(body))
+	body := []byte("Action=DescribeInstances")
+	req := httptest.NewRequest(http.MethodPost, "/", bytes.NewReader(body))
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
 	req.Header.Set("Content-Type", "multipart/form-data; boundary=something")
+	signTestRequest(t, req, body, testAccessKey, testSecretKey)
 
 	resp := doRequest(handler, req)
 
@@ -1399,15 +982,9 @@ func TestSigV4Auth_QueryStringEdgeCases(t *testing.T) {
 
 	for _, tc := range testCases {
 		t.Run(tc.name, func(t *testing.T) {
-			authHeader, timestamp := generateTestAuthHeader(
-				"GET", "/", tc.queryString, "",
-				testAccessKey, testSecretKey, testRegion, testService,
-			)
-
 			req := httptest.NewRequest(http.MethodGet, "/?"+tc.queryString, nil)
 			req.Host = "localhost:9999"
-			req.Header.Set("Authorization", authHeader)
-			req.Header.Set("X-Amz-Date", timestamp)
+			signTestRequest(t, req, nil, testAccessKey, testSecretKey)
 
 			resp := doRequest(handler, req)
 
@@ -1492,15 +1069,9 @@ func TestCheckPolicy_NonRootNoPolicies_Denied(t *testing.T) {
 
 	handler := setupPolicyTestHandler(gw)
 
-	authHeader, timestamp := generateTestAuthHeader(
-		"GET", "/", "", "",
-		testAccessKey, testSecretKey, testRegion, testService,
-	)
-
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
+	signTestRequest(t, req, nil, testAccessKey, testSecretKey)
 
 	resp := doRequest(handler, req)
 
@@ -1558,15 +1129,9 @@ func TestCheckPolicy_NonRootWithAllow_Passes(t *testing.T) {
 
 	handler := setupPolicyTestHandler(gw)
 
-	authHeader, timestamp := generateTestAuthHeader(
-		"GET", "/", "", "",
-		testAccessKey, testSecretKey, testRegion, testService,
-	)
-
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
+	signTestRequest(t, req, nil, testAccessKey, testSecretKey)
 
 	resp := doRequest(handler, req)
 
@@ -1624,15 +1189,9 @@ func TestCheckPolicy_NonRootWithExplicitDeny_Denied(t *testing.T) {
 
 	handler := setupPolicyTestHandler(gw)
 
-	authHeader, timestamp := generateTestAuthHeader(
-		"GET", "/", "", "",
-		testAccessKey, testSecretKey, testRegion, testService,
-	)
-
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
+	signTestRequest(t, req, nil, testAccessKey, testSecretKey)
 
 	resp := doRequest(handler, req)
 
@@ -1691,15 +1250,9 @@ func TestCheckPolicy_RootGlobalAccount_Bypasses(t *testing.T) {
 
 	handler := setupPolicyTestHandler(gw)
 
-	authHeader, timestamp := generateTestAuthHeader(
-		"GET", "/", "", "",
-		testAccessKey, testSecretKey, testRegion, testService,
-	)
-
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
+	signTestRequest(t, req, nil, testAccessKey, testSecretKey)
 
 	resp := doRequest(handler, req)
 
@@ -1739,15 +1292,9 @@ func TestCheckPolicy_MissingAccountID_InternalError(t *testing.T) {
 
 	handler := setupPolicyTestHandler(gw)
 
-	authHeader, timestamp := generateTestAuthHeader(
-		"GET", "/", "", "",
-		testAccessKey, testSecretKey, testRegion, testService,
-	)
-
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
+	signTestRequest(t, req, nil, testAccessKey, testSecretKey)
 
 	resp := doRequest(handler, req)
 
@@ -1789,15 +1336,9 @@ func TestCheckPolicy_GetUserPoliciesError_InternalError(t *testing.T) {
 
 	handler := setupPolicyTestHandler(gw)
 
-	authHeader, timestamp := generateTestAuthHeader(
-		"GET", "/", "", "",
-		testAccessKey, testSecretKey, testRegion, testService,
-	)
-
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
+	signTestRequest(t, req, nil, testAccessKey, testSecretKey)
 
 	resp := doRequest(handler, req)
 
@@ -1841,15 +1382,9 @@ func TestCheckPolicy_RootNonGlobalAccount_StillEvaluated(t *testing.T) {
 
 	handler := setupPolicyTestHandler(gw)
 
-	authHeader, timestamp := generateTestAuthHeader(
-		"GET", "/", "", "",
-		testAccessKey, testSecretKey, testRegion, testService,
-	)
-
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
+	signTestRequest(t, req, nil, testAccessKey, testSecretKey)
 
 	resp := doRequest(handler, req)
 
@@ -1889,16 +1424,11 @@ func TestSigV4Auth_RequireSignedHeaders(t *testing.T) {
 
 	for _, tc := range cases {
 		t.Run(tc.name, func(t *testing.T) {
-			authHeader, timestamp := generateTestAuthHeader(
-				"GET", "/", "", "",
-				testAccessKey, testSecretKey, testRegion, testService,
-			)
-			authHeader = rewriteSignedHeaders(authHeader, tc.signedList)
-
 			req := httptest.NewRequest(http.MethodGet, "/", nil)
 			req.Host = "localhost:9999"
-			req.Header.Set("Authorization", authHeader)
-			req.Header.Set("X-Amz-Date", timestamp)
+			signTestRequest(t, req, nil, testAccessKey, testSecretKey)
+			req.Header.Set("Authorization",
+				rewriteSignedHeaders(req.Header.Get("Authorization"), tc.signedList))
 
 			resp := doRequest(handler, req)
 
@@ -1914,35 +1444,8 @@ func TestSigV4Auth_RequireSignedHeaders(t *testing.T) {
 	}
 }
 
-func TestSignedHeadersIncludeHostAndDate(t *testing.T) {
-	cases := []struct {
-		in   string
-		want bool
-	}{
-		{"host;x-amz-date", true},
-		{"content-type;host;x-amz-date", true},
-		{"Host;X-Amz-Date", true},
-		{" host ; x-amz-date ", true},
-		{"x-amz-date", false},
-		{"host", false},
-		{"content-type", false},
-		{"", false},
-	}
-	for _, tc := range cases {
-		t.Run(tc.in, func(t *testing.T) {
-			got := signedHeadersIncludeHostAndDate(tc.in)
-			if got != tc.want {
-				t.Errorf("signedHeadersIncludeHostAndDate(%q) = %v, want %v", tc.in, got, tc.want)
-			}
-		})
-	}
-}
-
-// TestSigV4Auth_NATSDisconnectedShortCircuit verifies that the SigV4 middleware
-// returns the cluster-unavailable 503 promised by 1c without ever reaching the
-// IAM lookup when the NATS connection is disconnected. With IAMService nil the
-// post-lookup path would return 500 InternalError, so a 503 here proves the
-// short-circuit fired before the lookup.
+// With IAMService nil the post-lookup path would return 500 InternalError, so a 503
+// here proves the disconnected-NATS short-circuit fired before the lookup.
 func TestSigV4Auth_NATSDisconnectedShortCircuit(t *testing.T) {
 	ns, _ := testutil.StartTestNATS(t)
 	nc, err := nats.Connect(ns.ClientURL())
@@ -1964,14 +1467,9 @@ func TestSigV4Auth_NATSDisconnectedShortCircuit(t *testing.T) {
 		_, _ = w.Write([]byte("OK"))
 	})
 
-	authHeader, timestamp := generateTestAuthHeader(
-		"GET", "/", "", "",
-		testAccessKey, testSecretKey, testRegion, testService,
-	)
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
+	signTestRequest(t, req, nil, testAccessKey, testSecretKey)
 
 	resp := doRequest(r, req)
 

spinifex/gateway/ec2/image/RegisterImage.go

@@ -17,6 +17,14 @@ var validRegisterImageArchitectures = map[string]bool{
 	"i386":   true,
 }
 
+// uefi-preferred is accepted and treated as uefi at launch (single host, single
+// firmware path — the BIOS-fallback ambiguity AWS resolves doesn't apply here).
+var validRegisterImageBootModes = map[string]bool{
+	"bios":           true,
+	"uefi":           true,
+	"uefi-preferred": true,
+}
+
 // ValidateRegisterImageInput rejects inputs that ask for behaviour spinifex
 // doesn't have (PV, S3 bundles, kernel/ramdisk, IMDS/TPM/ENA hints) rather
 // than silently discarding them. Registration is pointer-only: caller supplies
@@ -48,7 +56,9 @@ func ValidateRegisterImageInput(input *ec2.RegisterImageInput) error {
 	}
 
 	if input.BootMode != nil && *input.BootMode != "" {
-		return errors.New(awserrors.ErrorInvalidParameterValue)
+		if !validRegisterImageBootModes[*input.BootMode] {
+			return errors.New(awserrors.ErrorInvalidParameterValue)
+		}
 	}
 	if input.KernelId != nil && *input.KernelId != "" {
 		return errors.New(awserrors.ErrorInvalidParameterValue)

spinifex/gateway/ec2/image/RegisterImage_test.go

@@ -124,8 +124,23 @@ func TestValidateRegisterImageInput(t *testing.T) {
 			errMsg:  awserrors.ErrorInvalidParameterValue,
 		},
 		{
-			name:    "BootModeRejected",
+			name:    "BootModeAcceptedBios",
+			mutate:  func(i *ec2.RegisterImageInput) { i.BootMode = aws.String("bios") },
+			wantErr: false,
+		},
+		{
+			name:    "BootModeAcceptedUEFI",
 			mutate:  func(i *ec2.RegisterImageInput) { i.BootMode = aws.String("uefi") },
+			wantErr: false,
+		},
+		{
+			name:    "BootModeAcceptedUEFIPreferred",
+			mutate:  func(i *ec2.RegisterImageInput) { i.BootMode = aws.String("uefi-preferred") },
+			wantErr: false,
+		},
+		{
+			name:    "BootModeRejectedInvalidValue",
+			mutate:  func(i *ec2.RegisterImageInput) { i.BootMode = aws.String("legacy-bios") },
 			wantErr: true,
 			errMsg:  awserrors.ErrorInvalidParameterValue,
 		},

spinifex/gateway/ratelimit_test.go

@@ -391,7 +391,7 @@ func TestRateLimitIntegration_LockedIPGets503(t *testing.T) {
 		req := httptest.NewRequest(http.MethodGet, "/", nil)
 		req.Host = "localhost:9999"
 		req.RemoteAddr = "10.99.0.1:54321"
-		req.Header.Set("Authorization", "AWS4-HMAC-SHA256 Credential=AKIAINVALIDKEY000000/20240101/us-east-1/ec2/aws4_request, SignedHeaders=host;x-amz-date, Signature=badsig")
+		req.Header.Set("Authorization", "AWS4-HMAC-SHA256 Credential=AKIAINVALIDKEY000000/20240101/us-east-1/ec2/aws4_request, SignedHeaders=host;x-amz-date, Signature=0000000000000000000000000000000000000000000000000000000000000000")
 		req.Header.Set("X-Amz-Date", time.Now().UTC().Format("20060102T150405Z"))
 		doRequest(handler, req)
 	}
@@ -400,7 +400,7 @@ func TestRateLimitIntegration_LockedIPGets503(t *testing.T) {
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
 	req.RemoteAddr = "10.99.0.1:54321"
-	req.Header.Set("Authorization", "AWS4-HMAC-SHA256 Credential=AKIAINVALIDKEY000000/20240101/us-east-1/ec2/aws4_request, SignedHeaders=host;x-amz-date, Signature=badsig")
+	req.Header.Set("Authorization", "AWS4-HMAC-SHA256 Credential=AKIAINVALIDKEY000000/20240101/us-east-1/ec2/aws4_request, SignedHeaders=host;x-amz-date, Signature=0000000000000000000000000000000000000000000000000000000000000000")
 	req.Header.Set("X-Amz-Date", time.Now().UTC().Format("20060102T150405Z"))
 
 	resp := doRequest(handler, req)
@@ -425,18 +425,16 @@ func TestRateLimitIntegration_SuccessResetsLockout(t *testing.T) {
 		req := httptest.NewRequest(http.MethodGet, "/", nil)
 		req.Host = "localhost:9999"
 		req.RemoteAddr = "10.99.0.2:54321"
-		req.Header.Set("Authorization", "AWS4-HMAC-SHA256 Credential=AKIAINVALIDKEY000000/20240101/us-east-1/ec2/aws4_request, SignedHeaders=host;x-amz-date, Signature=badsig")
+		req.Header.Set("Authorization", "AWS4-HMAC-SHA256 Credential=AKIAINVALIDKEY000000/20240101/us-east-1/ec2/aws4_request, SignedHeaders=host;x-amz-date, Signature=0000000000000000000000000000000000000000000000000000000000000000")
 		req.Header.Set("X-Amz-Date", time.Now().UTC().Format("20060102T150405Z"))
 		doRequest(handler, req)
 	}
 
 	// Now send a valid request — should succeed and clear failure state.
-	authHeader, timestamp := generateTestAuthHeader("GET", "/", "", "", testAccessKey, testSecretKey, testRegion, testService)
 	req := httptest.NewRequest(http.MethodGet, "/", nil)
 	req.Host = "localhost:9999"
 	req.RemoteAddr = "10.99.0.2:54321"
-	req.Header.Set("Authorization", authHeader)
-	req.Header.Set("X-Amz-Date", timestamp)
+	signTestRequest(t, req, nil, testAccessKey, testSecretKey)
 
 	resp := doRequest(handler, req)
 	if resp.StatusCode != http.StatusOK {

spinifex/handlers/ec2/image/service_impl.go

@@ -246,7 +246,9 @@ func (s *ImageServiceImpl) DescribeImages(input *ec2.DescribeImagesInput, accoun
 			ownerID = utils.GlobalAccountID
 		}
 
-		// Build EC2 Image from AMIMetadata
+		// Build EC2 Image from AMIMetadata. Empty BootMode passes through as
+		// empty so legacy AMIs registered before this field existed don't get
+		// a synthesized value.
 		image := &ec2.Image{
 			ImageId:            aws.String(amiMeta.ImageID),
 			Name:               aws.String(amiMeta.Name),
@@ -262,6 +264,7 @@ func (s *ImageServiceImpl) DescribeImages(input *ec2.DescribeImagesInput, accoun
 			State:              aws.String("available"),
 			ImageType:          aws.String("machine"),
 			Hypervisor:         aws.String("xen"), // Default hypervisor
+			BootMode:           aws.String(amiMeta.BootMode),
 		}
 
 		if bdms := synthesizeRootBlockDeviceMapping(amiMeta); bdms != nil {
@@ -445,6 +448,7 @@ func (s *ImageServiceImpl) CreateImageFromInstance(params CreateImageParams, acc
 		CreationDate:    time.Now(),
 		RootDeviceType:  ec2.DeviceTypeEbs,
 		ImageOwnerAlias: accountID,
+		BootMode:        sourceAMI.BootMode,
 	}
 
 	if err := s.putAMIConfig(amiID, meta); err != nil {
@@ -831,6 +835,7 @@ func (s *ImageServiceImpl) CopyImage(input *ec2.CopyImageInput, accountID string
 		RootDeviceType:  rootDeviceType,
 		ImageOwnerAlias: accountID,
 		CreationDate:    time.Now(),
+		BootMode:        srcMeta.BootMode,
 		Tags:            tags,
 	}
 

spinifex/handlers/ec2/image/service_impl_test.go

@@ -200,6 +200,52 @@ func TestDescribeImages_AfterCreate(t *testing.T) {
 	assert.Equal(t, testAccountID, *result.Images[0].OwnerId)
 }
 
+// TestDescribeImages_BootModeProjection asserts that AMIMetadata.BootMode
+// round-trips through DescribeImages. Empty BootMode (legacy AMIs registered
+// before the field existed) must pass through as an empty string, not get
+// synthesized to a default.
+func TestDescribeImages_BootModeProjection(t *testing.T) {
+	svc, store := setupTestImageService(t)
+
+	createTestAMIConfigFull(t, store, viperblock.AMIMetadata{
+		ImageID:         "ami-uefi001",
+		Name:            "uefi-image",
+		Architecture:    "x86_64",
+		PlatformDetails: "Linux/UNIX",
+		Virtualization:  "hvm",
+		RootDeviceType:  "ebs",
+		VolumeSizeGiB:   8,
+		ImageOwnerAlias: testAccountID,
+		BootMode:        "uefi",
+	})
+	createTestAMIConfigFull(t, store, viperblock.AMIMetadata{
+		ImageID:         "ami-legacy001",
+		Name:            "legacy-image",
+		Architecture:    "x86_64",
+		PlatformDetails: "Linux/UNIX",
+		Virtualization:  "hvm",
+		RootDeviceType:  "ebs",
+		VolumeSizeGiB:   8,
+		ImageOwnerAlias: testAccountID,
+	})
+
+	result, err := svc.DescribeImages(&ec2.DescribeImagesInput{
+		ImageIds: []*string{aws.String("ami-uefi001"), aws.String("ami-legacy001")},
+	}, testAccountID)
+	require.NoError(t, err)
+	require.Len(t, result.Images, 2)
+
+	byID := map[string]*ec2.Image{}
+	for _, img := range result.Images {
+		byID[aws.StringValue(img.ImageId)] = img
+	}
+	require.Contains(t, byID, "ami-uefi001")
+	require.Contains(t, byID, "ami-legacy001")
+	assert.Equal(t, "uefi", aws.StringValue(byID["ami-uefi001"].BootMode))
+	assert.Equal(t, "", aws.StringValue(byID["ami-legacy001"].BootMode),
+		"legacy AMIs (empty BootMode) must pass through as empty, not be backfilled")
+}
+
 func TestGetVolumeConfig(t *testing.T) {
 	svc, store := setupTestImageService(t)
 
@@ -1595,6 +1641,7 @@ func TestCopyImage_InheritsSourceFields(t *testing.T) {
 		VolumeSizeGiB:   32,
 		RootDeviceType:  "ebs",
 		Description:     "arm source",
+		BootMode:        "uefi",
 	})
 
 	before := time.Now()
@@ -1608,6 +1655,7 @@ func TestCopyImage_InheritsSourceFields(t *testing.T) {
 	assert.Equal(t, "hvm", newMeta.Virtualization)
 	assert.Equal(t, uint64(32), newMeta.VolumeSizeGiB)
 	assert.Equal(t, "ebs", newMeta.RootDeviceType)
+	assert.Equal(t, "uefi", newMeta.BootMode, "CopyImage must propagate BootMode from source")
 	assert.False(t, newMeta.CreationDate.Before(before), "CreationDate must be refreshed on copy, not inherited")
 }
 

spinifex/handlers/ec2/instance/service.go

@@ -118,12 +118,16 @@ type SubnetInfo struct {
 
 // ENICreator covers the VPC/ENI operations RunInstances performs while
 // auto-attaching a primary interface. Implemented via an adapter over
-// handlers/ec2/vpc.VPCServiceImpl on the daemon.
+// handlers/ec2/vpc.VPCServiceImpl on the daemon. DetachENI lives here (not on
+// ENIDeleter) because the launch-time NAT-rollback path needs to flip the ENI
+// to "available" before ENIDeleter.DeleteNetworkInterface, which rejects
+// in-use ENIs.
 type ENICreator interface {
 	GetDefaultSubnet(accountID string) (*SubnetInfo, error)
 	GetSubnet(accountID, subnetID string) (*SubnetInfo, error)
 	CreateNetworkInterface(input *ec2.CreateNetworkInterfaceInput, accountID string) (*ec2.CreateNetworkInterfaceOutput, error)
 	AttachENI(accountID, eniID, instanceID string, deviceIndex int64) (string, error)
+	DetachENI(accountID, eniID string) error
 	UpdateENIPublicIP(accountID, eniID, publicIP, poolName string) error
 }
 

spinifex/handlers/ec2/instance/service_impl.go

@@ -39,7 +39,7 @@ users:
   - name: {{.Username}}
     shell: /bin/sh
     groups:
-      - sudo
+      - {{.SudoGroup}}
     sudo: "ALL=(ALL) NOPASSWD:ALL"
     ssh_authorized_keys:
       - {{.SSHKey}}
@@ -60,16 +60,22 @@ ca_certs:
 {{.UserDataCloudConfig}}
 {{end}}
 
-{{if .UserDataScript}}
+{{if or .RHELWriteFiles .UserDataScript}}
 write_files:
+{{- if .RHELWriteFiles}}
+{{.RHELWriteFiles}}{{end}}{{if .UserDataScript}}
   - path: /tmp/cloud-init-startup.sh
     permissions: '0755'
     content: |
-{{.UserDataScript}}
-
+{{.UserDataScript}}{{end}}
+{{end}}
+{{if or .RHELRunCmd .UserDataScript}}
 runcmd:
+{{- if .RHELRunCmd}}
+{{.RHELRunCmd}}{{end}}{{if .UserDataScript}}
   - [ "/bin/bash", "/tmp/cloud-init-startup.sh" ]
 {{end}}
+{{end}}
 `
 
 const cloudInitMetaTemplate = `# meta-data
@@ -91,6 +97,26 @@ const cloudInitNetworkConfigWildcard = `network:
       dhcp-identifier: mac
 `
 
+// cloudInitNetworkConfigDisabled tells cloud-init to do no network rendering
+// at all. Used on RHEL-family guests where Spinifex writes NM keyfiles via
+// user-data write_files; without this stub cloud-init's RHEL network module
+// falls back to autodetect-DHCP and drops a second keyfile next to ours.
+const cloudInitNetworkConfigDisabled = `network:
+  config: disabled
+`
+
+// selectNetworkConfigForFamily returns the cloud-init network-config payload
+// for the given distro family. RHEL-family guests get an explicit
+// "config: disabled" stub because their NM keyfiles are written via user-data
+// write_files; shipping a netplan config alongside would let cloud-init's RHEL
+// network module drop a competing cloud-init-enpXsY.nmconnection.
+func selectNetworkConfigForFamily(distroFamily, eniMAC, devMAC, mgmtMAC, mgmtIP string, extraENIMACs []string) string {
+	if distroFamily == "rhel" {
+		return cloudInitNetworkConfigDisabled
+	}
+	return generateNetworkConfig(eniMAC, devMAC, mgmtMAC, mgmtIP, extraENIMACs)
+}
+
 // generateNetworkConfig produces the cloud-init network-config for the instance.
 //
 // Per-interface config is generated when eniMAC is present (VPC NIC). This allows
@@ -163,6 +189,116 @@ type CloudInitData struct {
 	UserDataCloudConfig string
 	UserDataScript      string
 	CACertPEM           string
+	// DistroFamily selects per-distro branches in the cloud-init template.
+	// "debian" | "rhel" | "alpine". Empty falls through to debian rendering
+	// (legacy AMIs registered before the field existed).
+	DistroFamily string
+	// SudoGroup is the OS group cloud-init adds the user to for passwordless
+	// sudo. "sudo" on debian/ubuntu/alpine, "wheel" on RHEL-family.
+	SudoGroup string
+	// RHELWriteFiles is the pre-indented YAML block of NM keyfile entries
+	// (one per NIC) appended under the merged write_files: key. Empty on
+	// non-RHEL guests; the network-config ISO file carries the YAML netplan
+	// equivalent instead.
+	RHELWriteFiles string
+	// RHELRunCmd is the pre-indented YAML block of runcmd entries (restorecon
+	// + nmcli reload/up) needed to load and bring up the keyfiles. Empty on
+	// non-RHEL guests.
+	RHELRunCmd string
+}
+
+// buildRHELCloudInit produces the cloud-init write_files (NM keyfiles) and
+// runcmd (restorecon + nmcli) blocks needed to bring up networking on a
+// RHEL-family guest. Returns empty strings when eniMAC is empty (wildcard /
+// non-VPC path) so NM's default autoconnect handles the interface.
+//
+// Mirrors generateNetworkConfig's per-interface structure: vpc0 + optional
+// vpc1..N for extra ENIs, dev0 (DHCP with route/DNS suppression), mgmt0
+// (static). Owning the keyfile bytes directly avoids relying on cloud-init's
+// v2-to-NM renderer round-tripping dhcp-identifier: mac → dhcp-client-id=mac,
+// which OVN's MAC-keyed DHCP requires.
+//
+// File mode 0600 root:root is load-bearing — NM ignores keyfiles otherwise.
+// restorecon defers to host SELinux policy (no-op on non-SELinux systems);
+// nmcli reload forces NM to rescan system-connections after write_files
+// completes in cloud-init's config stage, and nmcli up brings each interface
+// online without waiting on autoconnect heuristics.
+func buildRHELCloudInit(eniMAC, devMAC, mgmtMAC, mgmtIP string, extraENIMACs []string) (writeFiles, runCmd string) {
+	if eniMAC == "" {
+		return "", ""
+	}
+
+	var wf, rc strings.Builder
+
+	rc.WriteString("  - [ restorecon, -R, /etc/NetworkManager/system-connections/ ]\n")
+	rc.WriteString("  - [ nmcli, connection, reload ]\n")
+
+	appendRHELDHCPKeyfile(&wf, "vpc0", eniMAC, false)
+	rc.WriteString("  - [ nmcli, connection, up, vpc0 ]\n")
+
+	for i, mac := range extraENIMACs {
+		if mac == "" {
+			continue
+		}
+		name := fmt.Sprintf("vpc%d", i+1)
+		appendRHELDHCPKeyfile(&wf, name, mac, false)
+		fmt.Fprintf(&rc, "  - [ nmcli, connection, up, %s ]\n", name)
+	}
+
+	if devMAC != "" {
+		appendRHELDHCPKeyfile(&wf, "dev0", devMAC, true)
+		rc.WriteString("  - [ nmcli, connection, up, dev0 ]\n")
+	}
+
+	if mgmtMAC != "" && mgmtIP != "" {
+		appendRHELStaticKeyfile(&wf, "mgmt0", mgmtMAC, mgmtIP)
+		rc.WriteString("  - [ nmcli, connection, up, mgmt0 ]\n")
+	}
+
+	// Trim trailing newline so the template's {{.RHELWriteFiles}} doesn't
+	// produce a blank line before the user-script write_files entry on
+	// instances that have both.
+	return strings.TrimRight(wf.String(), "\n"), strings.TrimRight(rc.String(), "\n")
+}
+
+func appendRHELDHCPKeyfile(b *strings.Builder, name, mac string, suppressDefaults bool) {
+	fmt.Fprintf(b, "  - path: /etc/NetworkManager/system-connections/%s.nmconnection\n", name)
+	b.WriteString("    owner: root:root\n")
+	b.WriteString("    permissions: '0600'\n")
+	b.WriteString("    content: |\n")
+	b.WriteString("      [connection]\n")
+	fmt.Fprintf(b, "      id=%s\n", name)
+	b.WriteString("      type=ethernet\n")
+	b.WriteString("      [ethernet]\n")
+	fmt.Fprintf(b, "      mac-address=%s\n", mac)
+	b.WriteString("      [ipv4]\n")
+	b.WriteString("      method=auto\n")
+	b.WriteString("      dhcp-client-id=mac\n")
+	if suppressDefaults {
+		// Equivalent to netplan dhcp4-overrides {use-routes: false, use-dns: false}:
+		// dev NIC gets an IP for hostfwd but must not install a default route or DNS.
+		b.WriteString("      never-default=true\n")
+		b.WriteString("      ignore-auto-dns=true\n")
+	}
+	b.WriteString("      [ipv6]\n")
+	b.WriteString("      method=disabled\n")
+}
+
+func appendRHELStaticKeyfile(b *strings.Builder, name, mac, ip string) {
+	fmt.Fprintf(b, "  - path: /etc/NetworkManager/system-connections/%s.nmconnection\n", name)
+	b.WriteString("    owner: root:root\n")
+	b.WriteString("    permissions: '0600'\n")
+	b.WriteString("    content: |\n")
+	b.WriteString("      [connection]\n")
+	fmt.Fprintf(b, "      id=%s\n", name)
+	b.WriteString("      type=ethernet\n")
+	b.WriteString("      [ethernet]\n")
+	fmt.Fprintf(b, "      mac-address=%s\n", mac)
+	b.WriteString("      [ipv4]\n")
+	b.WriteString("      method=manual\n")
+	fmt.Fprintf(b, "      addresses=%s/24\n", ip)
+	b.WriteString("      [ipv6]\n")
+	b.WriteString("      method=disabled\n")
 }
 
 type CloudInitMetaData struct {
@@ -190,6 +326,33 @@ type volumeParams struct {
 	deleteOnTermination bool
 }
 
+// floorVolumeSizeToAMI ensures requested is at least the AMI's snapshot size.
+// Cloud images embed a fixed-size partition table; truncating below it (e.g.
+// a 10 GiB Rocky image requested at the 4 GiB default) hangs the guest in
+// dracut waiting on a root UUID that lives past the cut-off. AWS rejects
+// under-sized requests with InvalidParameterValue — spinifex silently rounds
+// up since we don't surface that error code today and the user-visible
+// failure (dracut hang) is worse than a slightly bigger volume.
+func floorVolumeSizeToAMI(loader AMIMetaLoader, imageID string, requested int) int {
+	if loader == nil || !strings.HasPrefix(imageID, "ami-") {
+		return requested
+	}
+	amiMeta, err := loader.GetAMIConfig(imageID)
+	if err != nil {
+		slog.Warn("floorVolumeSizeToAMI: AMI metadata fetch failed; using requested size — guest may dracut-hang if requested is smaller than the AMI snapshot", "imageId", imageID, "err", err)
+		return requested
+	}
+	if amiMeta.VolumeSizeGiB == 0 {
+		return requested
+	}
+	amiSize := utils.SafeUint64ToInt64(amiMeta.VolumeSizeGiB) * 1024 * 1024 * 1024
+	if amiSize <= int64(requested) {
+		return requested
+	}
+	slog.Info("floorVolumeSizeToAMI: rounding up to AMI snapshot size", "imageId", imageID, "requestedBytes", requested, "amiBytes", amiSize)
+	return int(amiSize)
+}
+
 // parseVolumeParams extracts volume parameters from RunInstancesInput,
 // applying defaults and resolving AMI-based image IDs.
 func parseVolumeParams(input *ec2.RunInstancesInput) volumeParams {
@@ -402,6 +565,9 @@ func (s *InstanceServiceImpl) PrepareRunInstances(input *ec2.RunInstancesInput,
 	launchCount := allocatableCount
 	slog.Info("PrepareRunInstances: count determined", "min", minCount, "max", maxCount, "launching", launchCount)
 
+	// Allocate is atomic per call (check + commit under a single write lock),
+	// so this loop can never overcommit. Under contention it may allocate
+	// fewer than allocatableCount; the < minCount branch below rolls back.
 	var allocatedCount int
 	for i := 0; i < launchCount; i++ {
 		if err := s.resourceMgr.Allocate(instanceType); err != nil {
@@ -431,6 +597,7 @@ func (s *InstanceServiceImpl) PrepareRunInstances(input *ec2.RunInstancesInput,
 			s.resourceMgr.Deallocate(instanceType)
 			continue
 		}
+		instance.BootMode = amiMeta.BootMode
 
 		// Terraform with associate_public_ip_address sends subnet/SG via
 		// NetworkInterfaces[0]; lift to top-level so the rest works uniformly.
@@ -471,7 +638,20 @@ func (s *InstanceServiceImpl) PrepareRunInstances(input *ec2.RunInstancesInput,
 			instance.ENIMac = *eni.MacAddress
 
 			if _, attachErr := s.eniCreator.AttachENI(accountID, instance.ENIId, instance.ID, 0); attachErr != nil {
-				slog.Error("PrepareRunInstances: failed to attach ENI to instance record — ELBv2 target IP resolution will fail", "eniId", instance.ENIId, "instanceId", instance.ID, "err", attachErr)
+				// Without the attachment, RegisterTargets (TargetType=instance)
+				// resolves ENIs via attachment.instance-id, finds none, and
+				// silently drops the target; terminate leaks the auto-EIP.
+				slog.Error("PrepareRunInstances: AttachENI failed — aborting launch",
+					"eniId", instance.ENIId, "instanceId", instance.ID, "err", attachErr)
+				if _, delErr := s.eniDeleter.DeleteNetworkInterface(&ec2.DeleteNetworkInterfaceInput{
+					NetworkInterfaceId: &instance.ENIId,
+				}, accountID); delErr != nil {
+					slog.Warn("PrepareRunInstances: failed to delete auto-created ENI after attach failure",
+						"eniId", instance.ENIId, "err", delErr)
+				}
+				lastRunErr = attachErr
+				s.resourceMgr.Deallocate(instanceType)
+				continue
 			}
 			ec2Instance.SetPrivateIpAddress(*eni.PrivateIpAddress)
 			ec2Instance.SetSubnetId(*input.SubnetId)
@@ -513,7 +693,16 @@ func (s *InstanceServiceImpl) PrepareRunInstances(input *ec2.RunInstancesInput,
 							slog.Warn("PrepareRunInstances: failed to update ENI with public IP", "eniId", *eni.NetworkInterfaceId, "err", updateErr)
 						}
 						portName := "port-" + *eni.NetworkInterfaceId
-						utils.PublishNATEvent(s.natsConn, "vpc.add-nat", *eni.VpcId, publicIP, *eni.PrivateIpAddress, portName, *eni.MacAddress)
+						if natErr := utils.AddNAT(s.natsConn, *eni.VpcId, publicIP, *eni.PrivateIpAddress, portName, *eni.MacAddress); natErr != nil {
+							slog.Error("PrepareRunInstances: vpc.add-nat failed — rolling back public IP to avoid surfacing an unreachable address",
+								"instanceId", instance.ID, "publicIp", publicIP, "pool", poolName, "err", natErr)
+							// Timeout may have committed the rule after our window; neutralise before releasing the IP.
+							utils.PublishNATEvent(s.natsConn, "vpc.delete-nat", *eni.VpcId, publicIP, *eni.PrivateIpAddress, portName, *eni.MacAddress)
+							s.rollbackAutoAssignedPublicIP(accountID, instance.ID, *eni.NetworkInterfaceId, publicIP, poolName)
+							lastRunErr = natErr
+							s.resourceMgr.Deallocate(instanceType)
+							continue
+						}
 						ec2Instance.PublicIpAddress = aws.String(publicIP)
 						instance.PublicIP = publicIP
 						instance.PublicIPPool = poolName
@@ -703,11 +892,18 @@ func (s *InstanceServiceImpl) StartInstance(instance *vm.VM, command spxtypes.EC
 	s.vmMgr.UpdateState(instance.ID, func(v *vm.VM) { v.Attributes = command.Attributes })
 
 	if err := s.vmMgr.Start(instance.ID); err != nil {
-		slog.Error("StartInstance: vmMgr.Start failed", "err", err)
 		if ok {
 			s.resourceMgr.Deallocate(instanceType)
 		}
-		return errors.New(awserrors.ErrorServerInternal)
+		switch {
+		case errors.Is(err, vm.ErrInstanceNotFound):
+			slog.Warn("StartInstance: instance not found in manager",
+				"instanceId", command.ID, "err", err)
+			return errors.New(awserrors.ErrorInvalidInstanceIDNotFound)
+		default:
+			slog.Error("StartInstance: vmMgr.Start failed", "err", err)
+			return errors.New(awserrors.ErrorServerInternal)
+		}
 	}
 
 	s.vmMgr.UpdateGuestDeviceNames(instance)
@@ -801,6 +997,9 @@ func (s *InstanceServiceImpl) StopOrTerminateInstance(instance *vm.VM, command s
 
 func (s *InstanceServiceImpl) GenerateVolumes(input *ec2.RunInstancesInput, instance *vm.VM) ([]VolumeInfo, error) {
 	p := parseVolumeParams(input)
+	if input.ImageId != nil {
+		p.size = floorVolumeSizeToAMI(s.amiLoader, *input.ImageId, p.size)
+	}
 
 	// Capture attach time for the root volume
 	attachTime := time.Now()
@@ -830,10 +1029,14 @@ func (s *InstanceServiceImpl) GenerateVolumes(input *ec2.RunInstancesInput, inst
 		return nil, err
 	}
 
-	// Step 2: Create EFI partition
-	err = s.prepareEFIVolume(imageId, volumeConfig, instance)
-	if err != nil {
-		return nil, err
+	// Step 2: Create EFI variable store (only when the AMI is UEFI; BIOS
+	// guests must not allocate an orphan VARS volume).
+	if instance.BootMode == "uefi" || instance.BootMode == "uefi-preferred" {
+		arch := instanceArchitecture(s.instanceTypes[*input.InstanceType])
+		err = s.prepareEFIVolume(imageId, volumeConfig, instance, arch)
+		if err != nil {
+			return nil, err
+		}
 	}
 
 	// Step 3: Create cloud-init volume if needed
@@ -992,73 +1195,96 @@ func (s *InstanceServiceImpl) cloneAMIToVolume(input *ec2.RunInstancesInput, siz
 	return nil
 }
 
-// prepareEFIVolume creates the EFI boot partition
-func (s *InstanceServiceImpl) prepareEFIVolume(imageId string, volumeConfig viperblock.VolumeConfig, instance *vm.VM) error {
-	efiVolumeName := fmt.Sprintf("%s-efi", imageId)
-	efiSize := 64 * 1024 * 1024 // 64MB
+// prepareEFIVolume creates the per-VM EFI variable store. The volume is
+// sized to match the firmware's VARS template (QEMU pflash refuses any other
+// size) and seeded with the template bytes so EFI variables — BootOrder,
+// BootNext, secure-boot state — survive across reboots. arch is the AMI
+// architecture string ("x86_64" | "arm64").
+func (s *InstanceServiceImpl) prepareEFIVolume(imageId string, volumeConfig viperblock.VolumeConfig, instance *vm.VM, arch string) error {
+	codePath, varsTemplate, varsSize, err := vm.FirmwarePaths(arch)
+	if err != nil {
+		slog.Error("UEFI firmware not installed on this host", "arch", arch, "err", err)
+		return errors.New(awserrors.ErrorServerInternal)
+	}
+	template, err := os.ReadFile(varsTemplate)
+	if err != nil {
+		slog.Error("Failed to read VARS template", "path", varsTemplate, "err", err)
+		return errors.New(awserrors.ErrorServerInternal)
+	}
+	if int64(len(template)) != varsSize {
+		slog.Error("VARS template size mismatch between stat and read", "path", varsTemplate, "statSize", varsSize, "readSize", len(template))
+		return errors.New(awserrors.ErrorServerInternal)
+	}
+	slog.Info("Preparing EFI variable store", "arch", arch, "code", codePath, "varsTemplate", varsTemplate, "size", varsSize)
 
-	// Update VolumeID to match the EFI volume name
+	efiVolumeName := fmt.Sprintf("%s-efi", imageId)
 	efiVolumeConfig := volumeConfig
 	efiVolumeConfig.VolumeMetadata.VolumeID = efiVolumeName
+	// Zero SizeGiB so viperblock's LoadState doesn't reconcile the EFI
+	// volume's persisted byte-exact size up to the parent root volume's
+	// GiB-rounded size — QEMU pflash rejects any VARS volume larger than
+	// the firmware's expected variable region.
+	efiVolumeConfig.VolumeMetadata.SizeGiB = 0
 
-	efiVb, err := s.newViperblock(efiVolumeName, efiSize, efiVolumeConfig)
+	efiVb, err := s.newViperblock(efiVolumeName, int(varsSize), efiVolumeConfig)
 	if err != nil {
 		slog.Error("Could not create EFI viperblock", "err", err)
 		return errors.New(awserrors.ErrorServerInternal)
 	}
 
-	// Initialize the backend
 	slog.Debug("Initializing EFI Viperblock store backend")
-	err = efiVb.Backend.Init()
-	if err != nil {
+	if err := efiVb.Backend.Init(); err != nil {
 		slog.Error("Failed to initialize EFI Viperblock store backend", "err", err)
 		return errors.New(awserrors.ErrorServerInternal)
 	}
 
-	// Load the state from the remote backend
-	_, err = efiVb.LoadStateRequest("")
-	slog.Info("LoadStateRequest", "error", err)
+	// Distinguish "not found" (seed from template) from any other backend
+	// failure (transient S3 5xx, JSON unmarshal, etc.). Treating a transient
+	// failure as "not found" would silently re-seed a volume on every retry,
+	// clobbering guest-set BootOrder. Backends signal missing-object by
+	// wrapping os.ErrNotExist (see viperblock.classifyStateLoad).
+	_, loadErr := efiVb.LoadStateRequest("")
+	if loadErr != nil && !errors.Is(loadErr, os.ErrNotExist) {
+		slog.Error("Failed to load EFI volume state from backend", "name", efiVolumeName, "err", loadErr)
+		return errors.New(awserrors.ErrorServerInternal)
+	}
+	if loadErr != nil {
+		slog.Info("EFI volume does not yet exist, seeding from firmware VARS template", "name", efiVolumeName)
 
-	// Create EFI volume if it doesn't exist
-	if err != nil {
-		slog.Info("Volume does not yet exist, creating EFI disk ...")
-
-		// Open the chunk WAL (sharded or legacy)
+		var walErr error
 		if efiVb.UseShardedWAL {
-			err = efiVb.OpenShardedWAL()
+			walErr = efiVb.OpenShardedWAL()
 		} else {
-			err = efiVb.OpenWAL(&efiVb.WAL, fmt.Sprintf("%s/%s", efiVb.WAL.BaseDir, types.GetFilePath(types.FileTypeWALChunk, efiVb.WAL.WallNum.Load(), efiVb.GetVolume())))
+			walErr = efiVb.OpenWAL(&efiVb.WAL, fmt.Sprintf("%s/%s", efiVb.WAL.BaseDir, types.GetFilePath(types.FileTypeWALChunk, efiVb.WAL.WallNum.Load(), efiVb.GetVolume())))
 		}
-		if err != nil {
-			slog.Error("Failed to load WAL", "err", err)
+		if walErr != nil {
+			slog.Error("Failed to load WAL", "err", walErr)
 			return errors.New(awserrors.ErrorServerInternal)
 		}
 
-		// Open the block to object WAL
-		err = efiVb.OpenWAL(&efiVb.BlockToObjectWAL, fmt.Sprintf("%s/%s", efiVb.WAL.BaseDir, types.GetFilePath(types.FileTypeWALBlock, efiVb.BlockToObjectWAL.WallNum.Load(), efiVb.GetVolume())))
-		if err != nil {
+		if err := efiVb.OpenWAL(&efiVb.BlockToObjectWAL, fmt.Sprintf("%s/%s", efiVb.WAL.BaseDir, types.GetFilePath(types.FileTypeWALBlock, efiVb.BlockToObjectWAL.WallNum.Load(), efiVb.GetVolume()))); err != nil {
 			slog.Error("Failed to load block WAL", "err", err)
 			return errors.New(awserrors.ErrorServerInternal)
 		}
 
-		// Write an empty block to the EFI volume
-		if err := efiVb.WriteAt(0, make([]byte, efiVb.BlockSize)); err != nil {
-			slog.Error("Failed to write empty EFI block", "err", err)
+		if err := efiVb.WriteAt(0, template); err != nil {
+			slog.Error("Failed to seed EFI volume with VARS template", "err", err)
+			return errors.New(awserrors.ErrorServerInternal)
 		}
 		if err := efiVb.Flush(); err != nil {
 			slog.Error("Failed to flush EFI volume", "err", err)
+			return errors.New(awserrors.ErrorServerInternal)
 		}
 	}
 
-	slog.Info("Closing EFI")
-	err = efiVb.Close()
-	slog.Info("Close", "error", err)
-	if err != nil {
+	// Close is the durability boundary after WriteAt+Flush; if it fails the
+	// VARS volume may be partially written and QEMU pflash will refuse to
+	// start. Fail the launch loudly rather than enqueueing a corrupt volume.
+	if err := efiVb.Close(); err != nil {
 		slog.Error("Failed to close EFI Viperblock store", "err", err)
+		return errors.New(awserrors.ErrorServerInternal)
 	}
-
-	err = efiVb.RemoveLocalFiles()
-	if err != nil {
+	if err := efiVb.RemoveLocalFiles(); err != nil {
 		slog.Error("Failed to remove local files", "err", err)
 	}
 
@@ -1073,6 +1299,17 @@ func (s *InstanceServiceImpl) prepareEFIVolume(imageId string, volumeConfig vipe
 	return nil
 }
 
+// instanceArchitecture pulls the AWS architecture string off an
+// ec2.InstanceTypeInfo. Returns "" when the spec is nil or malformed; the
+// caller's firmware probe surfaces that as a clear error rather than a
+// silent BIOS fallback.
+func instanceArchitecture(it *ec2.InstanceTypeInfo) string {
+	if it == nil || it.ProcessorInfo == nil || len(it.ProcessorInfo.SupportedArchitectures) == 0 || it.ProcessorInfo.SupportedArchitectures[0] == nil {
+		return ""
+	}
+	return *it.ProcessorInfo.SupportedArchitectures[0]
+}
+
 // prepareCloudInitVolume creates cloud-init ISO with SSH keys and user data.
 // rootVolumeId is the per-instance root volume ID (not the AMI ID), ensuring
 // each instance gets its own cloud-init volume with fresh SSH keys and metadata.
@@ -1214,11 +1451,40 @@ func (s *InstanceServiceImpl) createCloudInitISO(input *ec2.RunInstancesInput, i
 		slog.Error("failed to read CA cert for guest cloud-init injection", "path", caCertPath, "error", err)
 	}
 
+	// Re-read AMI metadata for DistroFamily rather than persisting it on
+	// vm.VM — cloud-init renders once at first launch and the ISO is sealed,
+	// so no consumer needs DistroFamily after this point.
+	var distroFamily string
+	if s.amiLoader != nil && input.ImageId != nil && *input.ImageId != "" {
+		amiMeta, err := s.amiLoader.GetAMIConfig(*input.ImageId)
+		if err != nil {
+			slog.Warn("createCloudInitISO: AMI metadata fetch failed; cloud-init will render with debian-family defaults (RHEL guests will boot without networking and with wrong sudo group)", "imageId", *input.ImageId, "err", err)
+		} else {
+			distroFamily = amiMeta.DistroFamily
+		}
+	}
+
+	extraMACs := make([]string, 0, len(instance.ExtraENIs))
+	for _, extra := range instance.ExtraENIs {
+		extraMACs = append(extraMACs, extra.ENIMac)
+	}
+
+	sudoGroup := "sudo"
+	var rhelWriteFiles, rhelRunCmd string
+	if distroFamily == "rhel" {
+		sudoGroup = "wheel"
+		rhelWriteFiles, rhelRunCmd = buildRHELCloudInit(instance.ENIMac, instance.DevMAC, instance.MgmtMAC, instance.MgmtIP, extraMACs)
+	}
+
 	userData := CloudInitData{
-		Username:  "ec2-user",
-		SSHKey:    string(sshKey),
-		Hostname:  hostname,
-		CACertPEM: caCertPEM,
+		Username:       "ec2-user",
+		SSHKey:         string(sshKey),
+		Hostname:       hostname,
+		CACertPEM:      caCertPEM,
+		DistroFamily:   distroFamily,
+		SudoGroup:      sudoGroup,
+		RHELWriteFiles: rhelWriteFiles,
+		RHELRunCmd:     rhelRunCmd,
 	}
 
 	// Decode and classify user-data from RunInstances (base64-encoded).
@@ -1280,12 +1546,10 @@ func (s *InstanceServiceImpl) createCloudInitISO(input *ec2.RunInstancesInput, i
 
 	// Add network-config: per-interface when VPC+dev (suppresses dev default route),
 	// wildcard DHCP otherwise. Extra ENI MACs produce additional DHCP NICs for
-	// multi-subnet system VMs (multi-AZ ALBs).
-	extraMACs := make([]string, 0, len(instance.ExtraENIs))
-	for _, extra := range instance.ExtraENIs {
-		extraMACs = append(extraMACs, extra.ENIMac)
-	}
-	networkConfig := generateNetworkConfig(instance.ENIMac, instance.DevMAC, instance.MgmtMAC, instance.MgmtIP, extraMACs)
+	// multi-subnet system VMs (multi-AZ ALBs). RHEL-family guests get a
+	// "config: disabled" stub here so cloud-init doesn't drop a second NM
+	// keyfile alongside the ones we wrote via user-data write_files.
+	networkConfig := selectNetworkConfigForFamily(distroFamily, instance.ENIMac, instance.DevMAC, instance.MgmtMAC, instance.MgmtIP, extraMACs)
 	err = writer.AddFile(strings.NewReader(networkConfig), "network-config")
 	if err != nil {
 		slog.Error("failed to add network-config file", "err", err)
@@ -2020,6 +2284,43 @@ func (s *InstanceServiceImpl) deleteInstanceVolumes(instance *vm.VM, instanceID
 	_ = instanceID
 }
 
+// rollbackAutoAssignedPublicIP unwinds an auto-assign that proceeded past
+// ENI create + attach + IPAM allocation + ENI update but failed at the
+// vpc.add-nat barrier. Order matters: clear the ENI record first so any
+// concurrent DescribeNetworkInterfaces stops surfacing the unreachable IP,
+// then release the IPAM lease so the pool slot can be reused, then detach
+// and delete the ENI itself — the failing instance never enters vmMgr, so
+// terminateCleanup will not run and the ENI would otherwise leak. Detach
+// must precede DeleteNetworkInterface; the latter rejects in-use ENIs.
+func (s *InstanceServiceImpl) rollbackAutoAssignedPublicIP(accountID, instanceID, eniID, publicIP, poolName string) {
+	if s.eniCreator != nil {
+		if err := s.eniCreator.UpdateENIPublicIP(accountID, eniID, "", ""); err != nil {
+			slog.Warn("PrepareRunInstances: failed to clear ENI public IP during NAT-failure rollback",
+				"eniId", eniID, "publicIp", publicIP, "err", err)
+		}
+	}
+	if s.ipReleaser != nil {
+		if err := s.ipReleaser.ReleaseIP(poolName, publicIP); err != nil {
+			slog.Warn("PrepareRunInstances: failed to release public IP during NAT-failure rollback",
+				"publicIp", publicIP, "pool", poolName, "err", err)
+		}
+	}
+	if s.eniCreator != nil {
+		if err := s.eniCreator.DetachENI(accountID, eniID); err != nil {
+			slog.Warn("PrepareRunInstances: failed to detach ENI during NAT-failure rollback",
+				"eniId", eniID, "instanceId", instanceID, "err", err)
+		}
+	}
+	if s.eniDeleter != nil {
+		if _, err := s.eniDeleter.DeleteNetworkInterface(&ec2.DeleteNetworkInterfaceInput{
+			NetworkInterfaceId: &eniID,
+		}, accountID); err != nil {
+			slog.Warn("PrepareRunInstances: failed to delete ENI during NAT-failure rollback",
+				"eniId", eniID, "instanceId", instanceID, "err", err)
+		}
+	}
+}
+
 func (s *InstanceServiceImpl) releaseInstancePublicIP(instance *vm.VM, instanceID string) {
 	if instance.PublicIP == "" || instance.PublicIPPool == "" || s.ipReleaser == nil {
 		return

spinifex/handlers/ec2/instance/service_impl_test.go

@@ -272,6 +272,187 @@ func TestCloudInitTemplateRendering(t *testing.T) {
 	}
 }
 
+func TestFloorVolumeSizeToAMI(t *testing.T) {
+	loader := &fakeAMILoader{byID: map[string]viperblock.AMIMetadata{
+		"ami-rocky":   {VolumeSizeGiB: 10},
+		"ami-debian":  {VolumeSizeGiB: 3},
+		"ami-no-size": {VolumeSizeGiB: 0},
+	}}
+	const fourGiB = 4 * 1024 * 1024 * 1024
+	const tenGiB = 10 * 1024 * 1024 * 1024
+	const twentyGiB = 20 * 1024 * 1024 * 1024
+
+	t.Run("ami larger than requested rounds up", func(t *testing.T) {
+		assert.Equal(t, tenGiB, floorVolumeSizeToAMI(loader, "ami-rocky", fourGiB))
+	})
+	t.Run("ami smaller than requested keeps requested", func(t *testing.T) {
+		assert.Equal(t, fourGiB, floorVolumeSizeToAMI(loader, "ami-debian", fourGiB))
+	})
+	t.Run("requested larger than ami keeps requested", func(t *testing.T) {
+		assert.Equal(t, twentyGiB, floorVolumeSizeToAMI(loader, "ami-rocky", twentyGiB))
+	})
+	t.Run("missing VolumeSizeGiB keeps requested (legacy AMI)", func(t *testing.T) {
+		assert.Equal(t, fourGiB, floorVolumeSizeToAMI(loader, "ami-no-size", fourGiB))
+	})
+	t.Run("unknown AMI keeps requested", func(t *testing.T) {
+		assert.Equal(t, fourGiB, floorVolumeSizeToAMI(loader, "ami-unknown", fourGiB))
+	})
+	t.Run("non-ami image id keeps requested", func(t *testing.T) {
+		assert.Equal(t, fourGiB, floorVolumeSizeToAMI(loader, "vol-123", fourGiB))
+	})
+	t.Run("nil loader keeps requested", func(t *testing.T) {
+		assert.Equal(t, fourGiB, floorVolumeSizeToAMI(nil, "ami-rocky", fourGiB))
+	})
+}
+
+func TestBuildRHELCloudInit(t *testing.T) {
+	t.Run("empty eniMAC returns empty (wildcard path)", func(t *testing.T) {
+		wf, rc := buildRHELCloudInit("", "", "", "", nil)
+		assert.Empty(t, wf)
+		assert.Empty(t, rc)
+	})
+
+	t.Run("vpc0 only", func(t *testing.T) {
+		wf, rc := buildRHELCloudInit("02:00:00:00:00:01", "", "", "", nil)
+		// File mode + ownership are load-bearing — NM ignores anything else.
+		assert.Contains(t, wf, "/etc/NetworkManager/system-connections/vpc0.nmconnection")
+		assert.Contains(t, wf, "owner: root:root")
+		assert.Contains(t, wf, "permissions: '0600'")
+		assert.Contains(t, wf, "mac-address=02:00:00:00:00:01")
+		assert.Contains(t, wf, "dhcp-client-id=mac")
+		assert.Contains(t, wf, "method=auto")
+		assert.NotContains(t, wf, "never-default")
+		assert.Contains(t, rc, "  - [ restorecon, -R, /etc/NetworkManager/system-connections/ ]")
+		assert.Contains(t, rc, "  - [ nmcli, connection, reload ]")
+		assert.Contains(t, rc, "  - [ nmcli, connection, up, vpc0 ]")
+	})
+
+	t.Run("vpc0 + extra ENIs + dev + mgmt", func(t *testing.T) {
+		wf, rc := buildRHELCloudInit(
+			"02:00:00:00:00:01",
+			"02:00:00:00:00:99",
+			"02:00:00:00:00:aa", "10.250.0.5",
+			[]string{"02:00:00:00:00:02", "02:00:00:00:00:03"},
+		)
+		for _, name := range []string{"vpc0", "vpc1", "vpc2", "dev0", "mgmt0"} {
+			assert.Contains(t, wf, "/etc/NetworkManager/system-connections/"+name+".nmconnection", name)
+			assert.Contains(t, rc, "  - [ nmcli, connection, up, "+name+" ]", name)
+		}
+		// dev NIC must not install default route or DNS.
+		assert.Contains(t, wf, "never-default=true")
+		assert.Contains(t, wf, "ignore-auto-dns=true")
+		// mgmt NIC is static with the supplied IP.
+		assert.Contains(t, wf, "method=manual")
+		assert.Contains(t, wf, "addresses=10.250.0.5/24")
+	})
+
+	t.Run("empty extra MAC skipped", func(t *testing.T) {
+		wf, _ := buildRHELCloudInit(
+			"02:00:00:00:00:01", "", "", "",
+			[]string{"", "02:00:00:00:00:02"},
+		)
+		// Empty slot is skipped, but index advances — second valid MAC is vpc2.
+		assert.NotContains(t, wf, "vpc1.nmconnection")
+		assert.Contains(t, wf, "vpc2.nmconnection")
+	})
+}
+
+func TestCloudInitTemplate_FamilyBranching(t *testing.T) {
+	rhelWF, rhelRC := buildRHELCloudInit("02:00:00:11:22:33", "", "", "", nil)
+
+	t.Run("debian family: sudo group, no NM keyfiles", func(t *testing.T) {
+		var buf bytes.Buffer
+		tmpl := template.Must(template.New("c").Parse(cloudInitUserDataTemplate))
+		require.NoError(t, tmpl.Execute(&buf, CloudInitData{
+			Username: "ec2-user", SSHKey: "ssh-rsa AAA", Hostname: "spinifex-vm-deb",
+			DistroFamily: "debian", SudoGroup: "sudo",
+		}))
+		out := buf.String()
+		assert.Contains(t, out, "- sudo")
+		assert.NotContains(t, out, "- wheel")
+		assert.NotContains(t, out, "NetworkManager/system-connections")
+		assert.NotContains(t, out, "nmcli")
+		assert.NotContains(t, out, "restorecon")
+	})
+
+	t.Run("rhel family: wheel group, NM keyfile write_files + runcmd", func(t *testing.T) {
+		var buf bytes.Buffer
+		tmpl := template.Must(template.New("c").Parse(cloudInitUserDataTemplate))
+		require.NoError(t, tmpl.Execute(&buf, CloudInitData{
+			Username: "ec2-user", SSHKey: "ssh-rsa AAA", Hostname: "spinifex-vm-rhel",
+			DistroFamily: "rhel", SudoGroup: "wheel",
+			RHELWriteFiles: rhelWF, RHELRunCmd: rhelRC,
+		}))
+		out := buf.String()
+		assert.Contains(t, out, "- wheel")
+		assert.NotContains(t, out, "- sudo\n")
+		assert.Contains(t, out, "write_files:")
+		assert.Contains(t, out, "/etc/NetworkManager/system-connections/vpc0.nmconnection")
+		assert.Contains(t, out, "permissions: '0600'")
+		assert.Contains(t, out, "dhcp-client-id=mac")
+		assert.Contains(t, out, "runcmd:")
+		assert.Contains(t, out, "restorecon")
+		assert.Contains(t, out, "nmcli, connection, reload")
+		assert.Contains(t, out, "nmcli, connection, up, vpc0")
+	})
+
+	t.Run("rhel + user script: both blocks merged under single write_files/runcmd", func(t *testing.T) {
+		var buf bytes.Buffer
+		tmpl := template.Must(template.New("c").Parse(cloudInitUserDataTemplate))
+		require.NoError(t, tmpl.Execute(&buf, CloudInitData{
+			Username: "ec2-user", SSHKey: "ssh-rsa AAA", Hostname: "spinifex-vm-rhel",
+			DistroFamily: "rhel", SudoGroup: "wheel",
+			RHELWriteFiles: rhelWF, RHELRunCmd: rhelRC,
+			UserDataScript: "      #!/bin/bash\n      echo hi\n",
+		}))
+		out := buf.String()
+		// YAML disallows duplicate top-level keys; assert each appears exactly once.
+		assert.Equal(t, 1, strings.Count(out, "\nwrite_files:"))
+		assert.Equal(t, 1, strings.Count(out, "\nruncmd:"))
+		assert.Contains(t, out, "/etc/NetworkManager/system-connections/vpc0.nmconnection")
+		assert.Contains(t, out, "/tmp/cloud-init-startup.sh")
+		assert.Contains(t, out, "nmcli, connection, up, vpc0")
+		assert.Contains(t, out, `"/bin/bash", "/tmp/cloud-init-startup.sh"`)
+	})
+}
+
+func TestSelectNetworkConfigForFamily(t *testing.T) {
+	const eniMAC = "02:00:00:00:00:01"
+
+	t.Run("rhel returns disabled stub", func(t *testing.T) {
+		out := selectNetworkConfigForFamily("rhel", eniMAC, "", "", "", nil)
+		assert.Equal(t, cloudInitNetworkConfigDisabled, out)
+		assert.Contains(t, out, "config: disabled")
+		// Must not emit any netplan v2 keys — those would render alongside
+		// our NM keyfile and produce a competing cloud-init-enpXsY connection.
+		assert.NotContains(t, out, "ethernets:")
+		assert.NotContains(t, out, "version: 2")
+	})
+
+	t.Run("debian returns per-interface netplan", func(t *testing.T) {
+		out := selectNetworkConfigForFamily("debian", eniMAC, "", "", "", nil)
+		assert.Contains(t, out, "version: 2")
+		assert.Contains(t, out, "ethernets:")
+		assert.Contains(t, out, "vpc0:")
+		assert.Contains(t, out, "dhcp-identifier: mac")
+		assert.NotContains(t, out, "config: disabled")
+	})
+
+	t.Run("empty family falls through to netplan", func(t *testing.T) {
+		// Legacy AMIs without DistroFamily populated must keep today's behaviour.
+		out := selectNetworkConfigForFamily("", eniMAC, "", "", "", nil)
+		assert.Contains(t, out, "version: 2")
+		assert.NotContains(t, out, "config: disabled")
+	})
+
+	t.Run("alpine falls through to netplan", func(t *testing.T) {
+		// Alpine has no RHEL-style NM keyfile path; reuses netplan.
+		out := selectNetworkConfigForFamily("alpine", eniMAC, "", "", "", nil)
+		assert.Contains(t, out, "version: 2")
+		assert.NotContains(t, out, "config: disabled")
+	})
+}
+
 func TestCloudInitMetaTemplateRendering(t *testing.T) {
 	data := CloudInitMetaData{
 		InstanceID: "i-0123456789abcdef0",
@@ -2067,6 +2248,48 @@ func TestPrepareRunInstances_HappyPathNoENI(t *testing.T) {
 	}
 }
 
+// TestPrepareRunInstances_BootModePropagated pins that the AMI's BootMode
+// flows onto every prepared VM, so the launch path picks UEFI vs BIOS without
+// a second AMI lookup. Empty AMI BootMode (legacy) flows through as empty.
+func TestPrepareRunInstances_BootModePropagated(t *testing.T) {
+	tests := []struct {
+		name         string
+		amiBootMode  string
+		wantBootMode string
+	}{
+		{"legacy empty", "", ""},
+		{"bios", "bios", "bios"},
+		{"uefi", "uefi", "uefi"},
+		{"uefi-preferred", "uefi-preferred", "uefi-preferred"},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			types, _ := defaultPrepareInstanceTypes()
+			prov := &fakeResourceCapacityProvider{
+				instanceTypes: types,
+				canAllocFn:    func(_ *ec2.InstanceTypeInfo, count int) int { return count },
+			}
+			svc := &InstanceServiceImpl{
+				config:        &config.Config{},
+				instanceTypes: types,
+				amiLoader: &fakeAMILoader{byID: map[string]viperblock.AMIMetadata{
+					"ami-1": {ImageOwnerAlias: "acc", BootMode: tc.amiBootMode},
+				}},
+				resourceMgr: prov,
+			}
+			_, instances, _, err := svc.PrepareRunInstances(&ec2.RunInstancesInput{
+				InstanceType: aws.String("t3.micro"),
+				ImageId:      aws.String("ami-1"),
+				MinCount:     aws.Int64(1),
+				MaxCount:     aws.Int64(1),
+			}, "acc")
+			require.NoError(t, err)
+			require.Len(t, instances, 1)
+			assert.Equal(t, tc.wantBootMode, instances[0].BootMode)
+		})
+	}
+}
+
 func TestStartInstance_NotStopped(t *testing.T) {
 	id := "i-running"
 	mgr := mgrWith(map[string]*vm.VM{id: {ID: id, Status: vm.StateRunning}})
@@ -2141,8 +2364,11 @@ type fakeENICreator struct {
 	subnet        *SubnetInfo
 	createOut     *ec2.CreateNetworkInterfaceOutput
 	createErr     error
+	attachErr     error
 	attachCalls   int
 	updateCalls   int
+	clearCalls    int // updateCalls where publicIP is ""
+	detachCalls   int
 }
 
 func (f *fakeENICreator) GetDefaultSubnet(_ string) (*SubnetInfo, error) {
@@ -2168,11 +2394,22 @@ func (f *fakeENICreator) CreateNetworkInterface(_ *ec2.CreateNetworkInterfaceInp
 
 func (f *fakeENICreator) AttachENI(_, _, _ string, _ int64) (string, error) {
 	f.attachCalls++
+	if f.attachErr != nil {
+		return "", f.attachErr
+	}
 	return "attached", nil
 }
 
-func (f *fakeENICreator) UpdateENIPublicIP(_, _, _, _ string) error {
+func (f *fakeENICreator) DetachENI(_, _ string) error {
+	f.detachCalls++
+	return nil
+}
+
+func (f *fakeENICreator) UpdateENIPublicIP(_, _, publicIP, _ string) error {
 	f.updateCalls++
+	if publicIP == "" {
+		f.clearCalls++
+	}
 	return nil
 }
 
@@ -2253,6 +2490,14 @@ func TestPrepareRunInstances_PublicIPAutoAssigned(t *testing.T) {
 	ipam := &fakeIPAllocator{publicIP: "203.0.113.5", poolName: "pool-a"}
 	svc, _ := prepareSvcWithENI(t, eni, ipam)
 
+	// vpc.add-nat is now request-reply: the helper waits for vpcd to ack
+	// before the launch proceeds. The happy path needs a success responder.
+	sub, err := svc.natsConn.Subscribe("vpc.add-nat", func(msg *nats.Msg) {
+		_ = msg.Respond([]byte(`{"success":true}`))
+	})
+	require.NoError(t, err)
+	defer func() { _ = sub.Unsubscribe() }()
+
 	_, instances, _, err := svc.PrepareRunInstances(&ec2.RunInstancesInput{
 		InstanceType: aws.String("t3.micro"),
 		ImageId:      aws.String("ami-1"),
@@ -2267,6 +2512,112 @@ func TestPrepareRunInstances_PublicIPAutoAssigned(t *testing.T) {
 	assert.Equal(t, 1, eni.updateCalls)
 }
 
+// TestPrepareRunInstances_NATFailureRollsBackPublicIP regresses the silent
+// corruption where vpc.add-nat NACKs (or vpcd is offline) but the launch
+// returned an ec2.Instance carrying the unreachable IP, the ENI record was
+// updated with that IP, and the IPAM pool entry stayed allocated. With the
+// fix in place, NAT failure must drop the instance from the response, clear
+// the ENI public IP, release the IPAM lease, and deallocate capacity.
+func TestPrepareRunInstances_NATFailureRollsBackPublicIP(t *testing.T) {
+	eni := &fakeENICreator{
+		subnet: &SubnetInfo{SubnetID: "subnet-1", VpcID: "vpc-1", MapPublicIpOnLaunch: true},
+		createOut: &ec2.CreateNetworkInterfaceOutput{
+			NetworkInterface: &ec2.NetworkInterface{
+				NetworkInterfaceId: aws.String("eni-nat-fail"),
+				MacAddress:         aws.String("aa:bb:cc:dd:ee:ff"),
+				PrivateIpAddress:   aws.String("10.0.0.30"),
+				VpcId:              aws.String("vpc-1"),
+			},
+		},
+	}
+	ipam := &fakeIPAllocator{publicIP: "203.0.113.7", poolName: "pool-a"}
+	releaser := &fakePublicIPReleaser{}
+	deleter := &fakeENIDeleter{}
+	svc, prov := prepareSvcWithENI(t, eni, ipam)
+	svc.ipReleaser = releaser
+	svc.eniDeleter = deleter
+
+	// Stand up a vpcd-shaped responder that NACKs every add-nat — simulates
+	// vpcd online but unable to commit the OVN rule (northd lag, port
+	// missing, etc.).
+	sub, err := svc.natsConn.Subscribe("vpc.add-nat", func(msg *nats.Msg) {
+		_ = msg.Respond([]byte(`{"success":false,"error":"northd unavailable"}`))
+	})
+	require.NoError(t, err)
+	defer func() { _ = sub.Unsubscribe() }()
+
+	// Record vpc.delete-nat to assert the rollback neutralises any
+	// half-committed rule from a waitForFlowsHV timeout.
+	deleteNATCh := make(chan natWirePayload, 1)
+	delSub, err := svc.natsConn.Subscribe("vpc.delete-nat", func(msg *nats.Msg) {
+		var p natWirePayload
+		_ = json.Unmarshal(msg.Data, &p)
+		select {
+		case deleteNATCh <- p:
+		default:
+		}
+	})
+	require.NoError(t, err)
+	defer func() { _ = delSub.Unsubscribe() }()
+
+	_, instances, _, err := svc.PrepareRunInstances(&ec2.RunInstancesInput{
+		InstanceType: aws.String("t3.micro"),
+		ImageId:      aws.String("ami-1"),
+		SubnetId:     aws.String("subnet-1"),
+		MinCount:     aws.Int64(1),
+		MaxCount:     aws.Int64(1),
+	}, "acc")
+
+	// MinCount=1 with a single launch attempt that NAT-failed must drop
+	// below minimum and return ServerInternal — the wrapped natErr is not
+	// an AWS-known code, so the lookup misses and the wire fallback kicks
+	// in. The critical invariant is that no instance with the unreachable
+	// IP is surfaced.
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
+	assert.Empty(t, instances, "instance with failed NAT must not be returned")
+
+	// IPAM lease released back to the pool.
+	assert.Equal(t, "pool-a", releaser.pool, "IPAM ReleaseIP must be called with the originally allocated pool")
+	assert.Equal(t, "203.0.113.7", releaser.ip, "IPAM ReleaseIP must be called with the originally allocated IP")
+
+	// ENI record cleared: 2 update calls total (set + clear), 1 clear.
+	assert.Equal(t, 2, eni.updateCalls, "ENI public IP should be set once, then cleared on rollback")
+	assert.Equal(t, 1, eni.clearCalls, "ENI rollback should call UpdateENIPublicIP with empty publicIP")
+
+	// ENI itself must be detached + deleted; otherwise the dropped instance
+	// leaks an eniKV record (vmMgr.Insert never runs, so terminateCleanup
+	// never fires) and the subnet exhausts private IPs under vpcd brown-out.
+	assert.Equal(t, 1, eni.detachCalls, "rollback must detach the ENI before delete")
+	assert.Equal(t, []string{"eni-nat-fail"}, deleter.calls, "rollback must delete the auto-created ENI")
+
+	// Capacity deallocated so subsequent launches can use the slot.
+	require.Len(t, prov.deallocated, 1, "NAT failure must trigger Deallocate")
+
+	// vpc.delete-nat must be published so vpcd reaps any rule it committed
+	// after the AddNAT timeout — otherwise the orphan rule outlives the
+	// IPAM allocation and routes traffic for the next tenant that grabs
+	// the released IP.
+	select {
+	case got := <-deleteNATCh:
+		assert.Equal(t, "vpc-1", got.VpcId)
+		assert.Equal(t, "203.0.113.7", got.ExternalIP)
+		assert.Equal(t, "10.0.0.30", got.LogicalIP)
+		assert.Equal(t, "port-eni-nat-fail", got.PortName)
+	case <-time.After(time.Second):
+		t.Fatal("rollback must publish vpc.delete-nat to neutralise a half-committed rule")
+	}
+}
+
+// natWirePayload mirrors utils.natEvent for test-side decoding.
+type natWirePayload struct {
+	VpcId      string `json:"vpc_id"`
+	ExternalIP string `json:"external_ip"`
+	LogicalIP  string `json:"logical_ip"`
+	PortName   string `json:"port_name"`
+	MAC        string `json:"mac"`
+}
+
 func TestPrepareRunInstances_ENICreateFailureDeallocates(t *testing.T) {
 	eni := &fakeENICreator{createErr: errors.New("boom")}
 	svc, prov := prepareSvcWithENI(t, eni, nil)
@@ -2285,6 +2636,49 @@ func TestPrepareRunInstances_ENICreateFailureDeallocates(t *testing.T) {
 	require.Len(t, prov.deallocated, 1, "ENI failure must trigger deallocate")
 }
 
+// TestPrepareRunInstances_ENIAttachFailureRollsBack regresses the silent
+// corruption where AttachENI failure was logged-and-ignored, leaving the
+// auto-created ENI unattached in vpcService. Later RegisterTargets with
+// TargetType=instance enumerates ENIs by attachment.instance-id, finds
+// none, and drops the target — the customer's ALB has 0 healthy targets
+// with no log line connecting the two events. The dangling ENI also leaks
+// its auto-assigned EIP on terminate because DeleteNetworkInterface
+// cannot resolve the instance link. With the fix, AttachENI failure must
+// delete the auto-created ENI, deallocate capacity, and drop the instance
+// from the reservation.
+func TestPrepareRunInstances_ENIAttachFailureRollsBack(t *testing.T) {
+	eni := &fakeENICreator{
+		attachErr: errors.New("vpc attach refused"),
+		createOut: &ec2.CreateNetworkInterfaceOutput{
+			NetworkInterface: &ec2.NetworkInterface{
+				NetworkInterfaceId: aws.String("eni-attach-fail"),
+				MacAddress:         aws.String("aa:bb:cc:dd:ee:01"),
+				PrivateIpAddress:   aws.String("10.0.0.40"),
+				VpcId:              aws.String("vpc-1"),
+			},
+		},
+	}
+	deleter := &fakeENIDeleter{}
+	svc, prov := prepareSvcWithENI(t, eni, nil)
+	svc.eniDeleter = deleter
+
+	_, instances, _, err := svc.PrepareRunInstances(&ec2.RunInstancesInput{
+		InstanceType: aws.String("t3.micro"),
+		ImageId:      aws.String("ami-1"),
+		SubnetId:     aws.String("subnet-1"),
+		MinCount:     aws.Int64(1),
+		MaxCount:     aws.Int64(1),
+	}, "acc")
+
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
+	assert.Empty(t, instances, "instance with failed AttachENI must not be returned")
+
+	assert.Equal(t, 1, eni.attachCalls, "AttachENI must be attempted exactly once")
+	assert.Equal(t, []string{"eni-attach-fail"}, deleter.calls, "auto-created ENI must be deleted on attach failure")
+	require.Len(t, prov.deallocated, 1, "AttachENI failure must trigger Deallocate")
+}
+
 func TestPrepareRunInstances_NetworkInterfaceLifted(t *testing.T) {
 	// Terraform-style: subnet+SG come via NetworkInterfaces[0], not top-level.
 	eni := &fakeENICreator{
@@ -2393,6 +2787,25 @@ func TestRebootInstance_NotFound(t *testing.T) {
 	assert.Equal(t, awserrors.ErrorInvalidInstanceIDNotFound, err.Error())
 }
 
+// TestStartInstance_NotFound verifies that when vmMgr.Start cannot find the
+// instance (e.g. it was terminated between the daemon's pre-dispatch lookup
+// and the manager call), the service returns InvalidInstanceID.NotFound
+// rather than collapsing to a generic Server.InternalError. The AWS SDK
+// retries 500s, so the prior behaviour produced duplicate Start attempts
+// before the client surfaced the failure.
+func TestStartInstance_NotFound(t *testing.T) {
+	id := "i-missing"
+	mgr := mgrWith(nil)
+	svc := &InstanceServiceImpl{
+		vmMgr:       mgr,
+		resourceMgr: &fakeResourceCapacityProvider{},
+	}
+	instance := &vm.VM{ID: id, Status: vm.StateStopped, InstanceType: "unknown"}
+	err := svc.StartInstance(instance, spxtypes.EC2InstanceCommand{ID: id})
+	require.Error(t, err)
+	assert.Equal(t, awserrors.ErrorInvalidInstanceIDNotFound, err.Error())
+}
+
 // TestRunInstances_PrepareError covers the InstanceService-level RunInstances
 // (sync convenience method) error propagation when PrepareRunInstances rejects.
 func TestRunInstances_PrepareError(t *testing.T) {
@@ -2628,3 +3041,33 @@ func TestDescribeInstanceStatus_TagFilter(t *testing.T) {
 	require.Len(t, out.InstanceStatuses, 1)
 	assert.Equal(t, "i-tag", *out.InstanceStatuses[0].InstanceId)
 }
+
+// TestInstanceArchitecture pins the safe-extraction contract: malformed
+// InstanceTypeInfo returns "" rather than panicking, and the firmware probe
+// surfaces "" as a clear error on the launch path.
+func TestInstanceArchitecture(t *testing.T) {
+	tests := []struct {
+		name string
+		it   *ec2.InstanceTypeInfo
+		want string
+	}{
+		{"nil", nil, ""},
+		{"nil processor info", &ec2.InstanceTypeInfo{}, ""},
+		{"empty supported archs", &ec2.InstanceTypeInfo{ProcessorInfo: &ec2.ProcessorInfo{}}, ""},
+		{
+			"x86_64",
+			&ec2.InstanceTypeInfo{ProcessorInfo: &ec2.ProcessorInfo{SupportedArchitectures: []*string{aws.String("x86_64")}}},
+			"x86_64",
+		},
+		{
+			"arm64",
+			&ec2.InstanceTypeInfo{ProcessorInfo: &ec2.ProcessorInfo{SupportedArchitectures: []*string{aws.String("arm64")}}},
+			"arm64",
+		},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			assert.Equal(t, tc.want, instanceArchitecture(tc.it))
+		})
+	}
+}

spinifex/handlers/elbv2/launcher.go

@@ -82,6 +82,20 @@ type NICConfig struct {
 	RouteVia  string // next-hop for RouteDst
 }
 
+// RecoveryContext carries the per-VM fields the ELBv2 service needs to
+// reconstruct a SystemInstanceInput during host-reboot recovery, without
+// importing the vm package (which would cycle back through daemon).
+// MgmtMAC / MgmtIP are the values the daemon already allocated for this
+// instance on a prior launch; the service injects them back into NIC[1]
+// rather than re-allocating from the mgmt IPAM.
+type RecoveryContext struct {
+	InstanceID   string
+	InstanceType string
+	ENIMac       string
+	MgmtMAC      string
+	MgmtIP       string
+}
+
 // SystemInstanceOutput contains the result of a successful launch.
 type SystemInstanceOutput struct {
 	InstanceID string // e.g. "i-xxxxx"

spinifex/handlers/elbv2/service_impl.go

@@ -117,6 +117,9 @@ type ELBv2ServiceImpl struct {
 	ctx                        context.Context
 	cancel                     context.CancelFunc
 	hc                         *healthChecker
+
+	recoveryLBIndexMu sync.Mutex
+	recoveryLBIndex   map[string]*LoadBalancerRecord
 }
 
 // NewELBv2ServiceImplWithNATS creates an ELBv2 service backed by JetStream KV.
@@ -157,6 +160,77 @@ func (s *ELBv2ServiceImpl) Close() {
 	}
 }
 
+// ResetTargetHealthOnStartup invalidates persisted target HealthState across
+// all target groups by resetting every non-draining target to "initial". Run
+// once during daemon startup before the service begins serving requests.
+//
+// Rationale: target HealthState is persisted in JetStream KV, but the
+// healthChecker's per-target counters live only in process memory. After a
+// daemon restart (e.g. host reboot), the counters are gone but the stored
+// "healthy" claim remains — DescribeTargetHealth returns "healthy"
+// immediately even though the daemon has zero active health observations
+// and the LB system VM's lb-agent hasn't posted a fresh report yet. Tests
+// that gate on WaitForTargetsHealthy advance prematurely, then ALB traffic
+// lands on a backend that isn't actually up.
+//
+// Resetting to "initial" is the correct AWS-semantics representation of
+// "we don't know yet, give us a moment" — the next lb-agent heartbeat
+// drives the state machine forward through evaluateHealth.
+//
+// Multi-node caveat: in a multi-daemon cluster, each daemon that restarts
+// resets the SHARED KV state. A sibling daemon's in-memory counters would
+// then be out-of-sync with the persisted "initial" state until the next
+// heartbeat re-converges. Acceptable trade-off — heartbeats arrive within
+// one health-check interval (default 5-30s) and round-robin transiently
+// excluding a backend is safer than serving traffic to a dead one.
+//
+// Refs mulga-siv-119.
+func (s *ELBv2ServiceImpl) ResetTargetHealthOnStartup(ctx context.Context) error {
+	if s == nil || s.store == nil {
+		return nil
+	}
+	tgs, err := s.store.ListTargetGroups()
+	if err != nil {
+		return fmt.Errorf("list target groups: %w", err)
+	}
+	resetTGs := 0
+	resetTargets := 0
+	for _, tg := range tgs {
+		if ctx.Err() != nil {
+			return ctx.Err()
+		}
+		changed := false
+		for i := range tg.Targets {
+			t := &tg.Targets[i]
+			// Skip targets actively being drained — that state is driven by
+			// DeregisterTargets and must not be clobbered by a daemon restart.
+			if t.HealthState == TargetHealthDraining {
+				continue
+			}
+			if t.HealthState == TargetHealthInitial {
+				continue
+			}
+			t.HealthState = TargetHealthInitial
+			t.HealthDesc = "Target registration is in progress"
+			changed = true
+			resetTargets++
+		}
+		if changed {
+			if err := s.store.PutTargetGroup(tg); err != nil {
+				slog.Error("ResetTargetHealthOnStartup: persist failed",
+					"tgId", tg.TargetGroupID, "err", err)
+				continue
+			}
+			resetTGs++
+		}
+	}
+	if resetTargets > 0 {
+		slog.Info("ELBv2: reset target health on startup",
+			"targetsReset", resetTargets, "targetGroupsTouched", resetTGs)
+	}
+	return nil
+}
+
 // SetSystemInstanceTypeFunc sets a function that resolves the smallest available
 // instance type. Called at request time so it adapts to node capacity.
 func (s *ELBv2ServiceImpl) SetSystemInstanceTypeFunc(fn func() string) {
@@ -290,6 +364,146 @@ func (s *ELBv2ServiceImpl) buildMicrovmNICs(primaryIP, primaryMAC, primarySubnet
 	return nics
 }
 
+// describeENIs resolves the supplied ENI IDs in one VPC describe call and
+// returns them keyed by NetworkInterfaceId. Returns an empty map when the
+// VPC service is unwired or the describe errors — callers must tolerate
+// missing entries.
+func (s *ELBv2ServiceImpl) describeENIs(eniIDs []string, accountID string) map[string]*ec2.NetworkInterface {
+	eniDetails := make(map[string]*ec2.NetworkInterface, len(eniIDs))
+	if s.VPCService == nil || len(eniIDs) == 0 {
+		return eniDetails
+	}
+	eniPtrs := make([]*string, 0, len(eniIDs))
+	for _, id := range eniIDs {
+		eniPtrs = append(eniPtrs, aws.String(id))
+	}
+	result, err := s.VPCService.DescribeNetworkInterfaces(&ec2.DescribeNetworkInterfacesInput{
+		NetworkInterfaceIds: eniPtrs,
+	}, accountID)
+	if err != nil {
+		slog.Error("VPC describe ENIs failed", "count", len(eniIDs), "err", err)
+		return eniDetails
+	}
+	for _, eni := range result.NetworkInterfaces {
+		if eni.NetworkInterfaceId != nil {
+			eniDetails[*eni.NetworkInterfaceId] = eni
+		}
+	}
+	return eniDetails
+}
+
+// buildExtraENIInputs threads every non-primary ENI in eniIDs into an
+// ExtraENIInput, using eniDetails as the MAC/IP/SubnetID source. eniIDs[0]
+// is the primary ENI and is intentionally excluded. ENIs missing from
+// eniDetails are skipped (matching CreateLoadBalancer's original behaviour
+// when DescribeNetworkInterfaces drops an entry).
+func buildExtraENIInputs(eniIDs []string, eniDetails map[string]*ec2.NetworkInterface) []ExtraENIInput {
+	if len(eniIDs) <= 1 {
+		return nil
+	}
+	extras := make([]ExtraENIInput, 0, len(eniIDs)-1)
+	for i := 1; i < len(eniIDs); i++ {
+		eni := eniDetails[eniIDs[i]]
+		if eni == nil {
+			continue
+		}
+		extras = append(extras, ExtraENIInput{
+			ENIID:    eniIDs[i],
+			ENIMac:   aws.StringValue(eni.MacAddress),
+			ENIIP:    aws.StringValue(eni.PrivateIpAddress),
+			SubnetID: aws.StringValue(eni.SubnetId),
+		})
+	}
+	return extras
+}
+
+// loadRecoveryLBIndex returns the instanceID->LB map, populated lazily on
+// first successful call. Errors are not cached: a transient JetStream
+// failure at startup must not condemn every subsequent recovery candidate
+// in the same batch — the next caller retries.
+func (s *ELBv2ServiceImpl) loadRecoveryLBIndex() (map[string]*LoadBalancerRecord, error) {
+	s.recoveryLBIndexMu.Lock()
+	defer s.recoveryLBIndexMu.Unlock()
+	if s.recoveryLBIndex != nil {
+		return s.recoveryLBIndex, nil
+	}
+	lbs, err := s.store.ListLoadBalancers()
+	if err != nil {
+		return nil, fmt.Errorf("list lb records: %w", err)
+	}
+	index := make(map[string]*LoadBalancerRecord, len(lbs))
+	for _, r := range lbs {
+		if r.InstanceID != "" {
+			index[r.InstanceID] = r
+		}
+	}
+	s.recoveryLBIndex = index
+	return s.recoveryLBIndex, nil
+}
+
+// RebuildSystemInstanceInput reconstructs the launch input for an existing
+// system instance during host-reboot recovery. The instance->LB map is
+// memoised in s.recoveryLBIndex so N concurrent recovery candidates share a
+// single ListLoadBalancers call instead of issuing N×L JetStream round-trips.
+func (s *ELBv2ServiceImpl) RebuildSystemInstanceInput(ctx RecoveryContext) (*SystemInstanceInput, error) {
+	index, err := s.loadRecoveryLBIndex()
+	if err != nil {
+		return nil, err
+	}
+	lb := index[ctx.InstanceID]
+	if lb == nil {
+		return nil, fmt.Errorf("no LB record references instance %s", ctx.InstanceID)
+	}
+	if len(lb.Subnets) == 0 || len(lb.ENIs) == 0 {
+		return nil, fmt.Errorf("lb %s has no subnets/ENIs to rebuild from", lb.LoadBalancerID)
+	}
+
+	eniDetails := s.describeENIs(lb.ENIs, lb.AccountID)
+	// Multi-ENI rebuild needs an authoritative MAC/IP/subnet per extra, which
+	// only the VPC store supplies. Single-ENI ALBs can fall back to ctx.ENIMac
+	// + lb.VPCIP for the primary, so we only enforce the strict count match
+	// when extras are present.
+	if len(lb.ENIs) > 1 && len(eniDetails) < len(lb.ENIs) {
+		return nil, fmt.Errorf("describe lb %s ENIs: got %d/%d", lb.LoadBalancerID, len(eniDetails), len(lb.ENIs))
+	}
+	primaryMAC := ctx.ENIMac
+	if primary := eniDetails[lb.ENIs[0]]; primary != nil {
+		if mac := aws.StringValue(primary.MacAddress); mac != "" {
+			primaryMAC = mac
+		}
+	}
+	if primaryMAC == "" {
+		return nil, fmt.Errorf("no MAC for primary ENI %s of lb %s", lb.ENIs[0], lb.LoadBalancerID)
+	}
+	extraENIs := buildExtraENIInputs(lb.ENIs, eniDetails)
+
+	nics := s.buildMicrovmNICs(lb.VPCIP, primaryMAC, lb.Subnets[0], lb.ENIs[0], lb.Scheme, extraENIs, lb.AccountID)
+	// Re-inject the daemon-allocated mgmt NIC values so writeFwCfgBlobs
+	// emits the same netcfg the original launch produced — buildMicrovmNICs
+	// leaves NIC[1] MAC/CIDR blank because they are only known after the
+	// daemon's per-instance mgmt IP allocation.
+	if len(nics) > 1 && ctx.MgmtMAC != "" {
+		nics[1].MAC = ctx.MgmtMAC
+		if ctx.MgmtIP != "" {
+			nics[1].CIDR = ctx.MgmtIP + "/24"
+		}
+	}
+
+	return &SystemInstanceInput{
+		InstanceType: ctx.InstanceType,
+		SubnetID:     lb.Subnets[0],
+		ENIID:        lb.ENIs[0],
+		ENIMac:       primaryMAC,
+		ENIIP:        lb.VPCIP,
+		ExtraENIs:    extraENIs,
+		Scheme:       lb.Scheme,
+		AccountID:    lb.AccountID,
+		NICs:         nics,
+		LBAgentEnv:   s.buildLBAgentEnv(lb.LoadBalancerID),
+		CACert:       s.CACert,
+	}, nil
+}
+
 // resolveENIBindAddr looks up the private IP of the first ENI belonging to the ALB.
 // Returns empty string if VPC service is unavailable or no ENIs exist.
 func (s *ELBv2ServiceImpl) resolveENIBindAddr(lb *LoadBalancerRecord) string {
@@ -706,24 +920,7 @@ func (s *ELBv2ServiceImpl) CreateLoadBalancer(input *elbv2.CreateLoadBalancerInp
 	var hostPorts map[int]int
 	var launchFailed bool
 	if s.InstanceLauncher != nil && len(eniIDs) > 0 && len(subnets) > 0 {
-		// Resolve MAC/IP for every ENI in one describe call.
-		eniDetails := make(map[string]*ec2.NetworkInterface, len(eniIDs))
-		if s.VPCService != nil {
-			eniPtrs := make([]*string, 0, len(eniIDs))
-			for _, id := range eniIDs {
-				eniPtrs = append(eniPtrs, aws.String(id))
-			}
-			result, descErr := s.VPCService.DescribeNetworkInterfaces(&ec2.DescribeNetworkInterfacesInput{
-				NetworkInterfaceIds: eniPtrs,
-			}, accountID)
-			if descErr == nil {
-				for _, eni := range result.NetworkInterfaces {
-					if eni.NetworkInterfaceId != nil {
-						eniDetails[*eni.NetworkInterfaceId] = eni
-					}
-				}
-			}
-		}
+		eniDetails := s.describeENIs(eniIDs, accountID)
 
 		primary := eniDetails[eniIDs[0]]
 		primaryIP := ""
@@ -733,19 +930,7 @@ func (s *ELBv2ServiceImpl) CreateLoadBalancer(input *elbv2.CreateLoadBalancerInp
 			primaryMAC = aws.StringValue(primary.MacAddress)
 		}
 
-		extraENIInputs := make([]ExtraENIInput, 0, len(eniIDs)-1)
-		for i := 1; i < len(eniIDs); i++ {
-			eni := eniDetails[eniIDs[i]]
-			if eni == nil {
-				continue
-			}
-			extraENIInputs = append(extraENIInputs, ExtraENIInput{
-				ENIID:    eniIDs[i],
-				ENIMac:   aws.StringValue(eni.MacAddress),
-				ENIIP:    aws.StringValue(eni.PrivateIpAddress),
-				SubnetID: aws.StringValue(eni.SubnetId),
-			})
-		}
+		extraENIInputs := buildExtraENIInputs(eniIDs, eniDetails)
 
 		var launchInput *SystemInstanceInput
 		if s.GatewayURL == "" || s.SystemAccessKey == "" || s.SystemSecretKey == "" {

spinifex/handlers/elbv2/service_impl_health_test.go

@@ -160,3 +160,50 @@ func TestRegisterTargets_MultipleInstances(t *testing.T) {
 	assert.Equal(t, "10.0.1.50", tg.Targets[0].PrivateIP)
 	assert.Equal(t, "10.0.1.51", tg.Targets[1].PrivateIP)
 }
+
+// TestResetTargetHealthOnStartup_TransitionsAllNonDrainingToInitial verifies
+// that the startup reset rewrites every non-draining target's HealthState
+// back to "initial" so a daemon restart can't surface stale "healthy" claims
+// before the lb-agent has posted a fresh report. Refs mulga-siv-119.
+func TestResetTargetHealthOnStartup_TransitionsAllNonDrainingToInitial(t *testing.T) {
+	store := setupTestNATS(t)
+	svc := &ELBv2ServiceImpl{store: store}
+
+	tg := &TargetGroupRecord{
+		TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:000:targetgroup/test/tg-reset",
+		TargetGroupID:  "tg-reset",
+		Port:           80,
+		HealthCheck:    DefaultHealthCheck(),
+		Targets: []Target{
+			{Id: "i-h", HealthState: TargetHealthHealthy, HealthDesc: "Target is healthy", PrivateIP: "10.0.0.1"},
+			{Id: "i-u", HealthState: TargetHealthUnhealthy, HealthDesc: "Health check failed", PrivateIP: "10.0.0.2"},
+			{Id: "i-d", HealthState: TargetHealthDraining, HealthDesc: "Draining", PrivateIP: "10.0.0.3"},
+			{Id: "i-i", HealthState: TargetHealthInitial, PrivateIP: "10.0.0.4"},
+		},
+	}
+	require.NoError(t, store.PutTargetGroup(tg))
+
+	require.NoError(t, svc.ResetTargetHealthOnStartup(t.Context()))
+
+	got, err := store.GetTargetGroup("tg-reset")
+	require.NoError(t, err)
+	require.NotNil(t, got)
+	require.Len(t, got.Targets, 4)
+
+	byID := map[string]Target{}
+	for _, t := range got.Targets {
+		byID[t.Id] = t
+	}
+	assert.Equal(t, TargetHealthInitial, byID["i-h"].HealthState, "previously healthy must reset")
+	assert.Equal(t, TargetHealthInitial, byID["i-u"].HealthState, "previously unhealthy must reset")
+	assert.Equal(t, TargetHealthDraining, byID["i-d"].HealthState, "draining must be preserved")
+	assert.Equal(t, TargetHealthInitial, byID["i-i"].HealthState, "initial stays initial")
+}
+
+func TestResetTargetHealthOnStartup_NilSafe(t *testing.T) {
+	var svc *ELBv2ServiceImpl
+	assert.NoError(t, svc.ResetTargetHealthOnStartup(t.Context()))
+
+	svc2 := &ELBv2ServiceImpl{}
+	assert.NoError(t, svc2.ResetTargetHealthOnStartup(t.Context()))
+}

spinifex/handlers/elbv2/service_impl_vpc_test.go

@@ -712,3 +712,133 @@ func TestCreateLoadBalancer_NoSecurityGroupsFallsBackToDefault(t *testing.T) {
 	require.Len(t, albENI.Groups, 1)
 	assert.Equal(t, defaultSGID, *albENI.Groups[0].GroupId)
 }
+
+// TestRebuildSystemInstanceInput_HappyPath verifies that a recovering ALB
+// VM gets the same SystemInstanceInput shape the launch path produced —
+// otherwise writeFwCfgBlobs would emit a different netcfg/cacert blob than
+// the persisted QEMU command line references, and the relaunch would fail
+// at fw_cfg load time the same way the tmpfs wipe does today.
+func TestRebuildSystemInstanceInput_HappyPath(t *testing.T) {
+	svc, vpcSvc := setupTestServiceWithVPC(t)
+
+	subnets, err := vpcSvc.DescribeSubnets(&ec2.DescribeSubnetsInput{}, testAccountID)
+	require.NoError(t, err)
+	subnetID := *subnets.Subnets[0].SubnetId
+
+	mock := &mockSystemInstanceLauncher{
+		launchResult: &SystemInstanceOutput{
+			InstanceID: "i-recover-1",
+			PrivateIP:  "10.0.1.42",
+		},
+	}
+	svc.InstanceLauncher = mock
+	svc.GatewayURL = "https://10.0.0.1:9999"
+	svc.SystemAccessKey = "AKID"
+	svc.SystemSecretKey = "SECRET"
+	svc.CACert = "-----BEGIN CERTIFICATE-----\nfake\n-----END CERTIFICATE-----\n"
+
+	_, err = svc.CreateLoadBalancer(&elbv2.CreateLoadBalancerInput{
+		Name:    aws.String("recover-alb"),
+		Subnets: []*string{aws.String(subnetID)},
+		Scheme:  aws.String("internal"),
+	}, testAccountID)
+	require.NoError(t, err)
+	require.Len(t, mock.launchCalls, 1)
+	originalInput := mock.launchCalls[0]
+
+	ctx := RecoveryContext{
+		InstanceID:   "i-recover-1",
+		InstanceType: originalInput.InstanceType,
+		ENIMac:       originalInput.ENIMac,
+		MgmtMAC:      "02:a0:00:11:22:33",
+		MgmtIP:       "172.31.0.7",
+	}
+
+	rebuilt, err := svc.RebuildSystemInstanceInput(ctx)
+	require.NoError(t, err)
+	require.NotNil(t, rebuilt)
+
+	assert.Equal(t, originalInput.InstanceType, rebuilt.InstanceType)
+	assert.Equal(t, originalInput.SubnetID, rebuilt.SubnetID)
+	assert.Equal(t, originalInput.ENIID, rebuilt.ENIID)
+	assert.Equal(t, originalInput.ENIMac, rebuilt.ENIMac)
+	// ENIIP comes from the persisted lb.VPCIP (which the launcher's PrivateIP
+	// return value sets); in production the launcher echoes back the ENI's
+	// private IP, so this matches the original launch input. The test mock
+	// returns a synthetic value, so assert against the mock's PrivateIP.
+	assert.Equal(t, "10.0.1.42", rebuilt.ENIIP)
+	assert.Equal(t, originalInput.Scheme, rebuilt.Scheme)
+	assert.Equal(t, originalInput.AccountID, rebuilt.AccountID)
+	assert.Equal(t, originalInput.LBAgentEnv, rebuilt.LBAgentEnv)
+	assert.Equal(t, originalInput.CACert, rebuilt.CACert)
+
+	require.GreaterOrEqual(t, len(rebuilt.NICs), 2)
+	assert.True(t, rebuilt.NICs[0].IsDefault)
+	assert.Equal(t, "02:a0:00:11:22:33", rebuilt.NICs[1].MAC, "mgmt NIC MAC must come from RecoveryContext")
+	assert.Equal(t, "172.31.0.7/24", rebuilt.NICs[1].CIDR, "mgmt NIC CIDR must come from RecoveryContext")
+}
+
+// TestRebuildSystemInstanceInput_NoLBRecord verifies the error path used by
+// refreshSystemInstanceState to flag the instance recovery_failed when the
+// LB record was wiped from KV but the VM record survived — recovery must
+// not invent a fake input and silently boot the VM into an unowned state.
+func TestRebuildSystemInstanceInput_NoLBRecord(t *testing.T) {
+	svc := setupTestService(t)
+
+	_, err := svc.RebuildSystemInstanceInput(RecoveryContext{
+		InstanceID:   "i-ghost",
+		InstanceType: "sys.micro",
+	})
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "no LB record references instance i-ghost")
+}
+
+// TestRebuildSystemInstanceInput_MultiENI verifies that extras survive the
+// rebuild for multi-subnet ALBs — the create path passes one ExtraENIInput
+// per non-primary subnet, and recovery must produce the same shape so the
+// daemon wires the same set of taps and NICs the persisted QEMU args
+// reference.
+func TestRebuildSystemInstanceInput_MultiENI(t *testing.T) {
+	svc, vpcSvc := setupTestServiceWithVPC(t)
+
+	vpcs, _ := vpcSvc.DescribeVpcs(&ec2.DescribeVpcsInput{}, testAccountID)
+	vpcID := *vpcs.Vpcs[0].VpcId
+	sub1 := getTestSubnetID(t, vpcSvc, vpcID, "10.0.20.0/24", "us-east-1a")
+	sub2 := getTestSubnetID(t, vpcSvc, vpcID, "10.0.21.0/24", "us-east-1b")
+
+	mock := &mockSystemInstanceLauncher{
+		launchResult: &SystemInstanceOutput{InstanceID: "i-recover-multi", PrivateIP: "10.0.20.4"},
+	}
+	svc.InstanceLauncher = mock
+	svc.GatewayURL = "https://10.0.0.1:9999"
+	svc.SystemAccessKey = "AKID"
+	svc.SystemSecretKey = "SECRET"
+
+	_, err := svc.CreateLoadBalancer(&elbv2.CreateLoadBalancerInput{
+		Name:    aws.String("recover-multi-alb"),
+		Subnets: []*string{aws.String(sub1), aws.String(sub2)},
+	}, testAccountID)
+	require.NoError(t, err)
+	require.Len(t, mock.launchCalls, 1)
+	originalInput := mock.launchCalls[0]
+
+	rebuilt, err := svc.RebuildSystemInstanceInput(RecoveryContext{
+		InstanceID:   "i-recover-multi",
+		InstanceType: originalInput.InstanceType,
+		ENIMac:       originalInput.ENIMac,
+	})
+	require.NoError(t, err)
+	require.Len(t, rebuilt.ExtraENIs, len(originalInput.ExtraENIs))
+
+	originalByENI := map[string]ExtraENIInput{}
+	for _, e := range originalInput.ExtraENIs {
+		originalByENI[e.ENIID] = e
+	}
+	for _, e := range rebuilt.ExtraENIs {
+		orig, ok := originalByENI[e.ENIID]
+		require.True(t, ok, "extra ENI %s not in original launch", e.ENIID)
+		assert.Equal(t, orig.SubnetID, e.SubnetID)
+		assert.Equal(t, orig.ENIMac, e.ENIMac)
+		assert.Equal(t, orig.ENIIP, e.ENIIP)
+	}
+}

spinifex/lbagent/agent.go

@@ -22,6 +22,7 @@ import (
 
 	"github.com/aws/aws-sdk-go/aws/credentials"
 	v4 "github.com/aws/aws-sdk-go/aws/signer/v4"
+	"github.com/mulgadc/spinifex/internal/tlsconfig"
 )
 
 const (
@@ -77,7 +78,8 @@ func New(lbID, gatewayURL, accessKey, secretKey, region string) (*Agent, error)
 		Timeout: 10 * time.Second,
 		Transport: &http.Transport{
 			TLSClientConfig: &tls.Config{
-				MinVersion: tls.VersionTLS12,
+				MinVersion:       tls.VersionTLS13,
+				CurvePreferences: tlsconfig.Curves,
 			},
 			MaxIdleConns:    2,
 			IdleConnTimeout: 30 * time.Second,

spinifex/services/awsgw/awsgw.go

@@ -11,6 +11,7 @@ import (
 	"time"
 
 	"github.com/mulgadc/predastore/ratelimit"
+	"github.com/mulgadc/spinifex/internal/tlsconfig"
 	"github.com/mulgadc/spinifex/spinifex/admin"
 	"github.com/mulgadc/spinifex/spinifex/config"
 	"github.com/mulgadc/spinifex/spinifex/gateway"
@@ -190,8 +191,10 @@ func launchService(config *config.ClusterConfig) error {
 		Handler:           handler,
 		ReadHeaderTimeout: 10 * time.Second,
 		TLSConfig: &tls.Config{
-			Certificates: []tls.Certificate{cert},
-			NextProtos:   []string{"h2", "http/1.1"},
+			Certificates:     []tls.Certificate{cert},
+			NextProtos:       []string{"h2", "http/1.1"},
+			MinVersion:       tls.VersionTLS13,
+			CurvePreferences: tlsconfig.Curves,
 		},
 	}
 

spinifex/services/predastore/predastore_integration_test.go

@@ -32,6 +32,7 @@ import (
 	"github.com/aws/aws-sdk-go/aws/credentials"
 	"github.com/aws/aws-sdk-go/aws/session"
 	"github.com/aws/aws-sdk-go/service/s3"
+	"github.com/mulgadc/predastore/quic/quicclient"
 	"github.com/stretchr/testify/assert"
 )
 
@@ -57,10 +58,14 @@ var (
 )
 
 // generateTestCertificate generates a self-signed TLS certificate for testing
-func generateTestCertificate(certPath, keyPath string) error {
+// and returns a CertPool that trusts it. Predastore's QUIC client (commit
+// dce8618) verifies server certs against the OS trust store after
+// InsecureSkipVerify was dropped, so the test must inject the ephemeral CA
+// via quicclient.SetDefaultRootCAs before any dial.
+func generateTestCertificate(certPath, keyPath string) (*x509.CertPool, error) {
 	priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
 	if err != nil {
-		return err
+		return nil, err
 	}
 
 	notBefore := time.Now()
@@ -68,7 +73,7 @@ func generateTestCertificate(certPath, keyPath string) error {
 
 	serialNumber, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128))
 	if err != nil {
-		return err
+		return nil, err
 	}
 
 	template := x509.Certificate{
@@ -88,31 +93,40 @@ func generateTestCertificate(certPath, keyPath string) error {
 
 	derBytes, err := x509.CreateCertificate(rand.Reader, &template, &template, &priv.PublicKey, priv)
 	if err != nil {
-		return err
+		return nil, err
 	}
 
 	certOut, err := os.Create(certPath)
 	if err != nil {
-		return err
+		return nil, err
 	}
 	defer certOut.Close()
 
 	if err := pem.Encode(certOut, &pem.Block{Type: "CERTIFICATE", Bytes: derBytes}); err != nil {
-		return err
+		return nil, err
 	}
 
 	keyOut, err := os.Create(keyPath)
 	if err != nil {
-		return err
+		return nil, err
 	}
 	defer keyOut.Close()
 
 	privBytes, err := x509.MarshalECPrivateKey(priv)
 	if err != nil {
-		return err
+		return nil, err
+	}
+	if err := pem.Encode(keyOut, &pem.Block{Type: "EC PRIVATE KEY", Bytes: privBytes}); err != nil {
+		return nil, err
 	}
 
-	return pem.Encode(keyOut, &pem.Block{Type: "EC PRIVATE KEY", Bytes: privBytes})
+	parsed, err := x509.ParseCertificate(derBytes)
+	if err != nil {
+		return nil, err
+	}
+	pool := x509.NewCertPool()
+	pool.AddCert(parsed)
+	return pool, nil
 }
 
 // startPredastoreServer starts the predastore server once for all tests
@@ -142,11 +156,28 @@ func startPredastoreServer(t *testing.T) *Config {
 	certPath := filepath.Join(certDir, "test.crt")
 	keyPath := filepath.Join(certDir, "test.key")
 
-	err = generateTestCertificate(certPath, keyPath)
+	caPool, err := generateTestCertificate(certPath, keyPath)
 	if err != nil {
 		t.Fatalf("Failed to generate certificate: %v", err)
 	}
 
+	// Predastore's in-process QUIC client (s3d → shard nodes) verifies the
+	// server cert against the OS trust store. The ephemeral test cert is
+	// not in that store, so inject it as a package-level default. Persists
+	// for the lifetime of the test process — startPredastoreServer is a
+	// once-per-process singleton.
+	quicclient.SetDefaultRootCAs(caPool)
+
+	// Predastore's in-process s3db REST client (port 6660) has no public
+	// hook to override its RootCAs — it uses the OS trust store via the
+	// default tls.Config (RootCAs nil ⇒ x509.SystemCertPool, which Go reads
+	// from SSL_CERT_FILE on Unix). Point it at the ephemeral cert so the
+	// in-process bucket/object metadata calls trust their own server.
+	// x509.SystemCertPool is sync.Once-cached, so the pool is loaded on
+	// the first dial below while SSL_CERT_FILE is still set, and stays
+	// valid for the rest of the process even after t.Setenv reverts.
+	t.Setenv("SSL_CERT_FILE", certPath)
+
 	// Predastore mandates a 32-byte master key at mode 0600 (rejected otherwise
 	// by internal/keyfile.Load).
 	encryptionKeyPath := filepath.Join(testDir, "encryption.key")

spinifex/services/spinifexui/frontend/dist/assets/DescribeInstancesCommand.js

@@ -1 +0,0 @@
-import{Mt as e,in as t,rn as n,tn as r}from"./awsClient.js";var i=class extends n.classBuilder().ep(r).m(function(e,n,r,i){return[t(r,e.getEndpointParameterInstructions())]}).s(`AmazonEC2`,`DescribeInstances`,{}).n(`EC2Client`,`DescribeInstancesCommand`).sc(e).build(){};export{i as t};

spinifex/services/spinifexui/frontend/dist/assets/_.js

@@ -1 +1 @@
-import{n as e,t}from"./compiler-runtime.js";import{t as n}from"./error-banner.js";import{t as r}from"./useSuspenseQuery.js";import{t as i}from"./link.js";import{F as a,H as o,L as s,R as c}from"./SignatureV4.js";import{t as l}from"./back-link.js";import{n as u,r as d,t as f}from"./upload-button.js";import{r as p}from"./s3.js";import{t as m}from"./s32.js";import{h}from"./index.js";var g=t(),_=e();function v(){let e=(0,g.c)(24),{bucket:t,_splat:v}=h.useParams(),y=s(v??``),{data:b}=r(m(t,y)),x=p(),S;e[0]===b.Contents?S=e[1]:(S=b.Contents??[],e[0]=b.Contents,e[1]=S);let C=S,w;e[2]===b.CommonPrefixes?w=e[3]:(w=b.CommonPrefixes??[],e[2]=b.CommonPrefixes,e[3]=w);let T=w,E=y.split(`/`).filter(Boolean),D=[{name:t,path:`/s3/ls/${t}`},...E.map((e,n)=>({name:e,path:`/s3/ls/${t}/${`${E.slice(0,n+1).join(`/`)}/`}`}))],O=E.length>1?`/s3/ls/${t}/${E.slice(0,-1).join(`/`)}/`:`/s3/ls/${t}`,k;e[4]===O?k=e[5]:(k=(0,_.jsx)(l,{to:O,children:`Back`}),e[4]=O,e[5]=k);let A;e[6]!==t||e[7]!==y||e[8]!==x.isPending||e[9]!==x.mutateAsync?(A=(0,_.jsx)(f,{bucket:t,isPending:x.isPending,onUpload:x.mutateAsync,prefix:y}),e[6]=t,e[7]=y,e[8]=x.isPending,e[9]=x.mutateAsync,e[10]=A):A=e[10];let j;e[11]===x.error?j=e[12]:(j=x.error&&(0,_.jsx)(n,{error:x.error,msg:`Failed to upload object`}),e[11]=x.error,e[12]=j);let M;e[13]!==t||e[14]!==T||e[15]!==y?(M=T.length>0&&(0,_.jsxs)(`div`,{className:`mb-6`,children:[(0,_.jsx)(`h2`,{className:`mb-3 text-lg font-semibold`,children:`Folders`}),(0,_.jsx)(`div`,{className:`space-y-2`,children:T.map(e=>e.Prefix?(0,_.jsx)(d,{bucketName:t,folderName:o(c(e.Prefix,y)),fullPrefix:a(e.Prefix,y)},e.Prefix):null)})]}),e[13]=t,e[14]=T,e[15]=y,e[16]=M):M=e[16];let N;e[17]!==t||e[18]!==C||e[19]!==y?(N=C.length>0&&(0,_.jsxs)(`div`,{className:`space-y-2`,children:[(0,_.jsx)(`h2`,{className:`mb-3 text-lg font-semibold`,children:`Objects`}),C.map(e=>e.Key?(0,_.jsx)(u,{bucketName:t,displayName:c(e.Key,y),fullKey:a(e.Key,y),object:e},e.Key):null)]}),e[17]=t,e[18]=C,e[19]=y,e[20]=N):N=e[20];let P;return e[21]!==T.length||e[22]!==C.length?(P=C.length===0&&T.length===0&&(0,_.jsx)(`p`,{className:`text-muted-foreground`,children:`No objects found in this folder.`}),e[21]=T.length,e[22]=C.length,e[23]=P):P=e[23],(0,_.jsxs)(_.Fragment,{children:[k,(0,_.jsx)(`div`,{className:`mb-6`,children:(0,_.jsxs)(`div`,{className:`mb-2 flex items-center justify-between`,children:[(0,_.jsx)(`div`,{className:`flex items-center gap-2 text-sm text-muted-foreground`,children:D.map((e,t)=>(0,_.jsxs)(`div`,{className:`flex items-center gap-2`,children:[t>0&&(0,_.jsx)(`span`,{children:`/`}),t===D.length-1?(0,_.jsx)(`span`,{className:`text-foreground`,children:e.name}):(0,_.jsx)(i,{className:`hover:text-foreground`,to:e.path,children:e.name})]},e.path))}),A]})}),j,M,N,P]})}export{v as component};
+import{t as e}from"./error-banner.js";import{t}from"./useSuspenseQuery.js";import{t as n}from"./back-link.js";import{n as r,r as i,t as a}from"./upload-button.js";import{r as o}from"./s3.js";import{Ca as s,Oa as c,P as l,eo as u,h as d,no as f,so as p,wa as m,xa as h}from"./index.js";var g=u(),_=p();function v(){let u=(0,g.c)(24),{bucket:p,_splat:v}=d.useParams(),y=s(v??``),{data:b}=t(l(p,y)),x=o(),S;u[0]===b.Contents?S=u[1]:(S=b.Contents??[],u[0]=b.Contents,u[1]=S);let C=S,w;u[2]===b.CommonPrefixes?w=u[3]:(w=b.CommonPrefixes??[],u[2]=b.CommonPrefixes,u[3]=w);let T=w,E=y.split(`/`).filter(Boolean),D=[{name:p,path:`/s3/ls/${p}`},...E.map((e,t)=>({name:e,path:`/s3/ls/${p}/${`${E.slice(0,t+1).join(`/`)}/`}`}))],O=E.length>1?`/s3/ls/${p}/${E.slice(0,-1).join(`/`)}/`:`/s3/ls/${p}`,k;u[4]===O?k=u[5]:(k=(0,_.jsx)(n,{to:O,children:`Back`}),u[4]=O,u[5]=k);let A;u[6]!==p||u[7]!==y||u[8]!==x.isPending||u[9]!==x.mutateAsync?(A=(0,_.jsx)(a,{bucket:p,isPending:x.isPending,onUpload:x.mutateAsync,prefix:y}),u[6]=p,u[7]=y,u[8]=x.isPending,u[9]=x.mutateAsync,u[10]=A):A=u[10];let j;u[11]===x.error?j=u[12]:(j=x.error&&(0,_.jsx)(e,{error:x.error,msg:`Failed to upload object`}),u[11]=x.error,u[12]=j);let M;u[13]!==p||u[14]!==T||u[15]!==y?(M=T.length>0&&(0,_.jsxs)(`div`,{className:`mb-6`,children:[(0,_.jsx)(`h2`,{className:`mb-3 text-lg font-semibold`,children:`Folders`}),(0,_.jsx)(`div`,{className:`space-y-2`,children:T.map(e=>e.Prefix?(0,_.jsx)(i,{bucketName:p,folderName:c(m(e.Prefix,y)),fullPrefix:h(e.Prefix,y)},e.Prefix):null)})]}),u[13]=p,u[14]=T,u[15]=y,u[16]=M):M=u[16];let N;u[17]!==p||u[18]!==C||u[19]!==y?(N=C.length>0&&(0,_.jsxs)(`div`,{className:`space-y-2`,children:[(0,_.jsx)(`h2`,{className:`mb-3 text-lg font-semibold`,children:`Objects`}),C.map(e=>e.Key?(0,_.jsx)(r,{bucketName:p,displayName:m(e.Key,y),fullKey:h(e.Key,y),object:e},e.Key):null)]}),u[17]=p,u[18]=C,u[19]=y,u[20]=N):N=u[20];let P;return u[21]!==T.length||u[22]!==C.length?(P=C.length===0&&T.length===0&&(0,_.jsx)(`p`,{className:`text-muted-foreground`,children:`No objects found in this folder.`}),u[21]=T.length,u[22]=C.length,u[23]=P):P=u[23],(0,_.jsxs)(_.Fragment,{children:[k,(0,_.jsx)(`div`,{className:`mb-6`,children:(0,_.jsxs)(`div`,{className:`mb-2 flex items-center justify-between`,children:[(0,_.jsx)(`div`,{className:`flex items-center gap-2 text-sm text-muted-foreground`,children:D.map((e,t)=>(0,_.jsxs)(`div`,{className:`flex items-center gap-2`,children:[t>0&&(0,_.jsx)(`span`,{children:`/`}),t===D.length-1?(0,_.jsx)(`span`,{className:`text-foreground`,children:e.name}):(0,_.jsx)(f,{className:`hover:text-foreground`,to:e.path,children:e.name})]},e.path))}),A]})}),j,M,N,P]})}export{v as component};

spinifex/services/spinifexui/frontend/dist/assets/_bucket.js

@@ -1 +1 @@
-import{n as e,t}from"./compiler-runtime.js";import{t as n}from"./error-banner.js";import{t as r}from"./useSuspenseQuery.js";import{H as i}from"./SignatureV4.js";import{t as a}from"./back-link.js";import{n as o,r as s,t as c}from"./upload-button.js";import{r as l}from"./s3.js";import{t as u}from"./s32.js";import{g as d}from"./index.js";import{t as f}from"./page-heading.js";var p=t(),m=e();function h(){let e=(0,p.c)(31),{bucket:t}=d.useParams(),h;e[0]===t?h=e[1]:(h=u(t),e[0]=t,e[1]=h);let{data:g}=r(h),_=l(),v;e[2]===g.Contents?v=e[3]:(v=g.Contents??[],e[2]=g.Contents,e[3]=v);let y=v,b;e[4]===g.CommonPrefixes?b=e[5]:(b=g.CommonPrefixes??[],e[4]=g.CommonPrefixes,e[5]=b);let x=b,S;e[6]===Symbol.for(`react.memo_cache_sentinel`)?(S=(0,m.jsx)(a,{to:`/s3/ls`,children:`Back to buckets`}),e[6]=S):S=e[6];let C;e[7]!==t||e[8]!==_.isPending||e[9]!==_.mutateAsync?(C=(0,m.jsx)(c,{bucket:t,isPending:_.isPending,onUpload:_.mutateAsync}),e[7]=t,e[8]=_.isPending,e[9]=_.mutateAsync,e[10]=C):C=e[10];let w;e[11]!==t||e[12]!==C?(w=(0,m.jsx)(f,{actions:C,title:t}),e[11]=t,e[12]=C,e[13]=w):w=e[13];let T;e[14]===_.error?T=e[15]:(T=_.error&&(0,m.jsx)(n,{error:_.error,msg:`Failed to upload object`}),e[14]=_.error,e[15]=T);let E;e[16]!==t||e[17]!==x?(E=x.length>0&&(0,m.jsxs)(`div`,{className:`mb-6`,children:[(0,m.jsx)(`h2`,{className:`mb-3 text-lg font-semibold`,children:`Folders`}),(0,m.jsx)(`div`,{className:`space-y-2`,children:x.map(e=>e.Prefix?(0,m.jsx)(s,{bucketName:t,folderName:i(e.Prefix),fullPrefix:e.Prefix},e.Prefix):null)})]}),e[16]=t,e[17]=x,e[18]=E):E=e[18];let D;e[19]!==t||e[20]!==y?(D=y.length>0&&(0,m.jsxs)(`div`,{className:`space-y-2`,children:[(0,m.jsx)(`h2`,{className:`mb-3 text-lg font-semibold`,children:`Objects`}),y.map(e=>e.Key?(0,m.jsx)(o,{bucketName:t,displayName:e.Key,fullKey:e.Key,object:e},e.Key):null)]}),e[19]=t,e[20]=y,e[21]=D):D=e[21];let O;e[22]!==x.length||e[23]!==y.length?(O=y.length===0&&x.length===0&&(0,m.jsx)(`p`,{className:`text-muted-foreground`,children:`No objects found in this bucket.`}),e[22]=x.length,e[23]=y.length,e[24]=O):O=e[24];let k;return e[25]!==w||e[26]!==T||e[27]!==E||e[28]!==D||e[29]!==O?(k=(0,m.jsxs)(m.Fragment,{children:[S,w,T,E,D,O]}),e[25]=w,e[26]=T,e[27]=E,e[28]=D,e[29]=O,e[30]=k):k=e[30],k}export{h as component};
+import{t as e}from"./error-banner.js";import{t}from"./useSuspenseQuery.js";import{t as n}from"./back-link.js";import{n as r,r as i,t as a}from"./upload-button.js";import{r as o}from"./s3.js";import{C as s,Oa as c,P as l,eo as u,so as d}from"./index.js";import{t as f}from"./page-heading.js";var p=u(),m=d();function h(){let u=(0,p.c)(31),{bucket:d}=s.useParams(),h;u[0]===d?h=u[1]:(h=l(d),u[0]=d,u[1]=h);let{data:g}=t(h),_=o(),v;u[2]===g.Contents?v=u[3]:(v=g.Contents??[],u[2]=g.Contents,u[3]=v);let y=v,b;u[4]===g.CommonPrefixes?b=u[5]:(b=g.CommonPrefixes??[],u[4]=g.CommonPrefixes,u[5]=b);let x=b,S;u[6]===Symbol.for(`react.memo_cache_sentinel`)?(S=(0,m.jsx)(n,{to:`/s3/ls`,children:`Back to buckets`}),u[6]=S):S=u[6];let C;u[7]!==d||u[8]!==_.isPending||u[9]!==_.mutateAsync?(C=(0,m.jsx)(a,{bucket:d,isPending:_.isPending,onUpload:_.mutateAsync}),u[7]=d,u[8]=_.isPending,u[9]=_.mutateAsync,u[10]=C):C=u[10];let w;u[11]!==d||u[12]!==C?(w=(0,m.jsx)(f,{actions:C,title:d}),u[11]=d,u[12]=C,u[13]=w):w=u[13];let T;u[14]===_.error?T=u[15]:(T=_.error&&(0,m.jsx)(e,{error:_.error,msg:`Failed to upload object`}),u[14]=_.error,u[15]=T);let E;u[16]!==d||u[17]!==x?(E=x.length>0&&(0,m.jsxs)(`div`,{className:`mb-6`,children:[(0,m.jsx)(`h2`,{className:`mb-3 text-lg font-semibold`,children:`Folders`}),(0,m.jsx)(`div`,{className:`space-y-2`,children:x.map(e=>e.Prefix?(0,m.jsx)(i,{bucketName:d,folderName:c(e.Prefix),fullPrefix:e.Prefix},e.Prefix):null)})]}),u[16]=d,u[17]=x,u[18]=E):E=u[18];let D;u[19]!==d||u[20]!==y?(D=y.length>0&&(0,m.jsxs)(`div`,{className:`space-y-2`,children:[(0,m.jsx)(`h2`,{className:`mb-3 text-lg font-semibold`,children:`Objects`}),y.map(e=>e.Key?(0,m.jsx)(r,{bucketName:d,displayName:e.Key,fullKey:e.Key,object:e},e.Key):null)]}),u[19]=d,u[20]=y,u[21]=D):D=u[21];let O;u[22]!==x.length||u[23]!==y.length?(O=y.length===0&&x.length===0&&(0,m.jsx)(`p`,{className:`text-muted-foreground`,children:`No objects found in this bucket.`}),u[22]=x.length,u[23]=y.length,u[24]=O):O=u[24];let k;return u[25]!==w||u[26]!==T||u[27]!==E||u[28]!==D||u[29]!==O?(k=(0,m.jsxs)(m.Fragment,{children:[S,w,T,E,D,O]}),u[25]=w,u[26]=T,u[27]=E,u[28]=D,u[29]=O,u[30]=k):k=u[30],k}export{h as component};

spinifex/services/spinifexui/frontend/dist/assets/_id.js

@@ -1 +1 @@
-import{n as e,t}from"./compiler-runtime.js";import{t as n}from"./useSuspenseQuery.js";import{t as r}from"./back-link.js";import{n as i}from"./ec22.js";import{t as a}from"./index.js";import{t as o}from"./page-heading.js";import{t as s}from"./state-badge.js";import{t as c}from"./system-managed.js";import{t as l}from"./ami-details.js";var u=t(),d=e();function f(){let e=(0,u.c)(14),{id:t}=a.useParams(),f;e[0]===t?f=e[1]:(f=i(t),e[0]=t,e[1]=f);let{data:p}=n(f),m=p?.Images?.[0];if(!m?.ImageId||c(m)){let t;return e[2]===Symbol.for(`react.memo_cache_sentinel`)?(t=(0,d.jsxs)(d.Fragment,{children:[(0,d.jsx)(r,{to:`/ec2/describe-images`,children:`Back to images`}),(0,d.jsx)(`p`,{className:`text-muted-foreground`,children:`Image not found.`})]}),e[2]=t):t=e[2],t}let h;e[3]===Symbol.for(`react.memo_cache_sentinel`)?(h=(0,d.jsx)(r,{to:`/ec2/describe-images`,children:`Back to images`}),e[3]=h):h=e[3];let g;e[4]===m.State?g=e[5]:(g=(0,d.jsx)(s,{state:m.State}),e[4]=m.State,e[5]=g);let _;e[6]!==m.ImageId||e[7]!==g?(_=(0,d.jsx)(o,{actions:g,subtitle:`EC2 Image Details`,title:m.ImageId}),e[6]=m.ImageId,e[7]=g,e[8]=_):_=e[8];let v;e[9]===m?v=e[10]:(v=(0,d.jsx)(l,{image:m,showExtendedDetails:!0}),e[9]=m,e[10]=v);let y;return e[11]!==_||e[12]!==v?(y=(0,d.jsxs)(d.Fragment,{children:[h,(0,d.jsxs)(`div`,{className:`space-y-6`,children:[_,v]})]}),e[11]=_,e[12]=v,e[13]=y):y=e[13],y}export{f as component};
+import{t as e}from"./useSuspenseQuery.js";import{t}from"./back-link.js";import{L as n,eo as r,so as i,t as a}from"./index.js";import{t as o}from"./page-heading.js";import{t as s}from"./state-badge.js";import{t as c}from"./system-managed.js";import{t as l}from"./ami-details.js";var u=r(),d=i();function f(){let r=(0,u.c)(14),{id:i}=a.useParams(),f;r[0]===i?f=r[1]:(f=n(i),r[0]=i,r[1]=f);let{data:p}=e(f),m=p?.Images?.[0];if(!m?.ImageId||c(m)){let e;return r[2]===Symbol.for(`react.memo_cache_sentinel`)?(e=(0,d.jsxs)(d.Fragment,{children:[(0,d.jsx)(t,{to:`/ec2/describe-images`,children:`Back to images`}),(0,d.jsx)(`p`,{className:`text-muted-foreground`,children:`Image not found.`})]}),r[2]=e):e=r[2],e}let h;r[3]===Symbol.for(`react.memo_cache_sentinel`)?(h=(0,d.jsx)(t,{to:`/ec2/describe-images`,children:`Back to images`}),r[3]=h):h=r[3];let g;r[4]===m.State?g=r[5]:(g=(0,d.jsx)(s,{state:m.State}),r[4]=m.State,r[5]=g);let _;r[6]!==m.ImageId||r[7]!==g?(_=(0,d.jsx)(o,{actions:g,subtitle:`EC2 Image Details`,title:m.ImageId}),r[6]=m.ImageId,r[7]=g,r[8]=_):_=r[8];let v;r[9]===m?v=r[10]:(v=(0,d.jsx)(l,{image:m,showExtendedDetails:!0}),r[9]=m,r[10]=v);let y;return r[11]!==_||r[12]!==v?(y=(0,d.jsxs)(d.Fragment,{children:[h,(0,d.jsxs)(`div`,{className:`space-y-6`,children:[_,v]})]}),r[11]=_,r[12]=v,r[13]=y):y=r[13],y}export{f as component};

spinifex/services/spinifexui/frontend/dist/assets/_id11.js

@@ -1 +1 @@
-import{n as e,t}from"./compiler-runtime.js";import{t as n}from"./error-banner.js";import{t as r}from"./useSuspenseQuery.js";import{t as i}from"./useNavigate.js";import{t as a}from"./back-link.js";import{E as o}from"./ec2.js";import{v as s}from"./ec22.js";import{L as c,d as l}from"./index.js";import{t as u}from"./page-heading.js";import{a as d,r as f,t as p}from"./field.js";import{r as m,t as h}from"./zod.js";import{d as g}from"./ec23.js";import{t as _}from"./form-actions.js";var v=t(),y=e();function b(){let e=(0,v.c)(53),{id:t}=l.useParams(),b=i(),x;e[0]===t?x=e[1]:(x=s(t),e[0]=t,e[1]=x);let{data:S}=r(x),C=S.Volumes?.[0],w=C?.VolumeId,T=o(),E=C?.Size??1,D;e[2]===E?D=e[3]:(D=h(g.refine(e=>e.size>E,{message:`New size must be greater than current size (${E} GiB)`,path:[`size`]})),e[2]=E,e[3]=D);let O;e[4]===E?O=e[5]:(O={size:E},e[4]=E,e[5]=O);let k;e[6]!==D||e[7]!==O?(k={resolver:D,defaultValues:O},e[6]=D,e[7]=O,e[8]=k):k=e[8];let{handleSubmit:A,register:j,formState:M}=m(k),{errors:N,isSubmitting:P}=M;if(!w){let t;return e[9]===Symbol.for(`react.memo_cache_sentinel`)?(t=(0,y.jsxs)(y.Fragment,{children:[(0,y.jsx)(a,{to:`/ec2/describe-volumes`,children:`Back to volumes`}),(0,y.jsx)(`p`,{className:`text-muted-foreground`,children:`Volume not found.`})]}),e[9]=t):t=e[9],t}let F;e[10]!==T||e[11]!==b||e[12]!==w?(F=async e=>{await T.mutateAsync({...e,volumeId:w}),b({to:`/ec2/describe-volumes/$id`,params:{id:w}})},e[10]=T,e[11]=b,e[12]=w,e[13]=F):F=e[13];let I=F,L,R;e[14]===w?(L=e[15],R=e[16]):(L=(0,y.jsx)(a,{params:{id:w},to:`/ec2/describe-volumes/$id`,children:`Back to volume`}),R=(0,y.jsx)(u,{subtitle:w,title:`Resize Volume`}),e[14]=w,e[15]=L,e[16]=R);let z;e[17]===T.error?z=e[18]:(z=T.error&&(0,y.jsx)(n,{error:T.error,msg:`Failed to resize volume`}),e[17]=T.error,e[18]=z);let B;e[19]!==A||e[20]!==I?(B=A(I),e[19]=A,e[20]=I,e[21]=B):B=e[21];let V;e[22]===E?V=e[23]:(V=(0,y.jsx)(`div`,{className:`space-y-2`,children:(0,y.jsxs)(`p`,{className:`text-sm text-muted-foreground`,children:[`Current size: `,E,` GiB`]})}),e[22]=E,e[23]=V);let H;e[24]===Symbol.for(`react.memo_cache_sentinel`)?(H=(0,y.jsx)(d,{children:(0,y.jsx)(`label`,{htmlFor:`size`,children:`New size (GiB)`})}),e[24]=H):H=e[24];let U=!!N.size,W=E+1,G;e[25]===j?G=e[26]:(G=j(`size`,{valueAsNumber:!0}),e[25]=j,e[26]=G);let K;e[27]!==U||e[28]!==W||e[29]!==G?(K=(0,y.jsx)(c,{"aria-invalid":U,id:`size`,min:W,type:`number`,...G}),e[27]=U,e[28]=W,e[29]=G,e[30]=K):K=e[30];let q;e[31]===N.size?q=e[32]:(q=(0,y.jsx)(f,{errors:[N.size]}),e[31]=N.size,e[32]=q);let J;e[33]!==K||e[34]!==q?(J=(0,y.jsxs)(p,{children:[H,K,q]}),e[33]=K,e[34]=q,e[35]=J):J=e[35];let Y;e[36]!==b||e[37]!==w?(Y=async()=>await b({to:`/ec2/describe-volumes/$id`,params:{id:w}}),e[36]=b,e[37]=w,e[38]=Y):Y=e[38];let X;e[39]!==P||e[40]!==T.isPending||e[41]!==Y?(X=(0,y.jsx)(_,{isPending:T.isPending,isSubmitting:P,onCancel:Y,pendingLabel:`Resizing\\u2026`,submitLabel:`Resize Volume`}),e[39]=P,e[40]=T.isPending,e[41]=Y,e[42]=X):X=e[42];let Z;e[43]!==V||e[44]!==J||e[45]!==X||e[46]!==B?(Z=(0,y.jsxs)(`form`,{className:`max-w-4xl space-y-6`,onSubmit:B,children:[V,J,X]}),e[43]=V,e[44]=J,e[45]=X,e[46]=B,e[47]=Z):Z=e[47];let Q;return e[48]!==Z||e[49]!==L||e[50]!==R||e[51]!==z?(Q=(0,y.jsxs)(y.Fragment,{children:[L,R,z,Z]}),e[48]=Z,e[49]=L,e[50]=R,e[51]=z,e[52]=Q):Q=e[52],Q}export{b as component};
+import{t as e}from"./error-banner.js";import{t}from"./useSuspenseQuery.js";import{t as n}from"./back-link.js";import{E as r}from"./ec2.js";import{$ as i,d as a,eo as o,ro as s,so as c,yi as l}from"./index.js";import{t as u}from"./page-heading.js";import{a as d,r as f,t as p}from"./field.js";import{r as m,t as h}from"./zod.js";import{d as g}from"./ec22.js";import{t as _}from"./form-actions.js";var v=o(),y=c();function b(){let o=(0,v.c)(53),{id:c}=a.useParams(),b=s(),x;o[0]===c?x=o[1]:(x=i(c),o[0]=c,o[1]=x);let{data:S}=t(x),C=S.Volumes?.[0],w=C?.VolumeId,T=r(),E=C?.Size??1,D;o[2]===E?D=o[3]:(D=h(g.refine(e=>e.size>E,{message:`New size must be greater than current size (${E} GiB)`,path:[`size`]})),o[2]=E,o[3]=D);let O;o[4]===E?O=o[5]:(O={size:E},o[4]=E,o[5]=O);let k;o[6]!==D||o[7]!==O?(k={resolver:D,defaultValues:O},o[6]=D,o[7]=O,o[8]=k):k=o[8];let{handleSubmit:A,register:j,formState:M}=m(k),{errors:N,isSubmitting:P}=M;if(!w){let e;return o[9]===Symbol.for(`react.memo_cache_sentinel`)?(e=(0,y.jsxs)(y.Fragment,{children:[(0,y.jsx)(n,{to:`/ec2/describe-volumes`,children:`Back to volumes`}),(0,y.jsx)(`p`,{className:`text-muted-foreground`,children:`Volume not found.`})]}),o[9]=e):e=o[9],e}let F;o[10]!==T||o[11]!==b||o[12]!==w?(F=async e=>{await T.mutateAsync({...e,volumeId:w}),b({to:`/ec2/describe-volumes/$id`,params:{id:w}})},o[10]=T,o[11]=b,o[12]=w,o[13]=F):F=o[13];let I=F,L,R;o[14]===w?(L=o[15],R=o[16]):(L=(0,y.jsx)(n,{params:{id:w},to:`/ec2/describe-volumes/$id`,children:`Back to volume`}),R=(0,y.jsx)(u,{subtitle:w,title:`Resize Volume`}),o[14]=w,o[15]=L,o[16]=R);let z;o[17]===T.error?z=o[18]:(z=T.error&&(0,y.jsx)(e,{error:T.error,msg:`Failed to resize volume`}),o[17]=T.error,o[18]=z);let B;o[19]!==A||o[20]!==I?(B=A(I),o[19]=A,o[20]=I,o[21]=B):B=o[21];let V;o[22]===E?V=o[23]:(V=(0,y.jsx)(`div`,{className:`space-y-2`,children:(0,y.jsxs)(`p`,{className:`text-sm text-muted-foreground`,children:[`Current size: `,E,` GiB`]})}),o[22]=E,o[23]=V);let H;o[24]===Symbol.for(`react.memo_cache_sentinel`)?(H=(0,y.jsx)(d,{children:(0,y.jsx)(`label`,{htmlFor:`size`,children:`New size (GiB)`})}),o[24]=H):H=o[24];let U=!!N.size,W=E+1,G;o[25]===j?G=o[26]:(G=j(`size`,{valueAsNumber:!0}),o[25]=j,o[26]=G);let K;o[27]!==U||o[28]!==W||o[29]!==G?(K=(0,y.jsx)(l,{"aria-invalid":U,id:`size`,min:W,type:`number`,...G}),o[27]=U,o[28]=W,o[29]=G,o[30]=K):K=o[30];let q;o[31]===N.size?q=o[32]:(q=(0,y.jsx)(f,{errors:[N.size]}),o[31]=N.size,o[32]=q);let J;o[33]!==K||o[34]!==q?(J=(0,y.jsxs)(p,{children:[H,K,q]}),o[33]=K,o[34]=q,o[35]=J):J=o[35];let Y;o[36]!==b||o[37]!==w?(Y=async()=>await b({to:`/ec2/describe-volumes/$id`,params:{id:w}}),o[36]=b,o[37]=w,o[38]=Y):Y=o[38];let X;o[39]!==P||o[40]!==T.isPending||o[41]!==Y?(X=(0,y.jsx)(_,{isPending:T.isPending,isSubmitting:P,onCancel:Y,pendingLabel:`Resizing\\u2026`,submitLabel:`Resize Volume`}),o[39]=P,o[40]=T.isPending,o[41]=Y,o[42]=X):X=o[42];let Z;o[43]!==V||o[44]!==J||o[45]!==X||o[46]!==B?(Z=(0,y.jsxs)(`form`,{className:`max-w-4xl space-y-6`,onSubmit:B,children:[V,J,X]}),o[43]=V,o[44]=J,o[45]=X,o[46]=B,o[47]=Z):Z=o[47];let Q;return o[48]!==Z||o[49]!==L||o[50]!==R||o[51]!==z?(Q=(0,y.jsxs)(y.Fragment,{children:[L,R,z,Z]}),o[48]=Z,o[49]=L,o[50]=R,o[51]=z,o[52]=Q):Q=o[52],Q}export{b as component};

spinifex/services/spinifexui/frontend/dist/assets/_id12.js

@@ -1 +1 @@
-import{a as e,n as t,r as n}from"./compiler-runtime.js";import{t as r}from"./error-banner.js";import{t as i}from"./useSuspenseQuery.js";import{t as a}from"./link.js";import{t as o}from"./useNavigate.js";import{V as s}from"./SignatureV4.js";import{t as c}from"./back-link.js";import{t as l}from"./trash-2.js";import{t as u}from"./button.js";import{x as d}from"./ec2.js";import{_ as f,b as p}from"./ec22.js";import{f as m}from"./index.js";import{t as h}from"./page-heading.js";import{t as g}from"./state-badge.js";import{n as _,t as v}from"./detail-row.js";import{t as y}from"./delete-confirmation-dialog.js";var b=e(n()),x=t();function S(){let{id:e}=m.useParams(),t=o(),{data:n}=i(p(e)),{data:S}=i(f),C=n.Vpcs?.[0],w=d(),[T,E]=(0,b.useState)(!1),D=S.Subnets?.filter(t=>t.VpcId===e)??[],O=async()=>{try{await w.mutateAsync(e),t({to:`/ec2/describe-vpcs`})}finally{E(!1)}};if(!C?.VpcId)return(0,x.jsxs)(x.Fragment,{children:[(0,x.jsx)(c,{to:`/ec2/describe-vpcs`,children:`Back to VPCs`}),(0,x.jsx)(`p`,{className:`text-muted-foreground`,children:`VPC not found.`})]});let k=s(C.Tags);return(0,x.jsxs)(x.Fragment,{children:[(0,x.jsx)(c,{to:`/ec2/describe-vpcs`,children:`Back to VPCs`}),w.error&&(0,x.jsx)(r,{error:w.error,msg:`Failed to delete VPC`}),(0,x.jsxs)(`div`,{className:`space-y-6`,children:[(0,x.jsx)(h,{actions:(0,x.jsxs)(`div`,{className:`flex items-center gap-2`,children:[(0,x.jsxs)(u,{onClick:()=>E(!0),size:`sm`,variant:`destructive`,children:[(0,x.jsx)(l,{className:`size-4`}),`Delete`]}),(0,x.jsx)(g,{state:C.State})]}),subtitle:`VPC Details`,title:k?`${C.VpcId} (${k})`:C.VpcId}),(0,x.jsxs)(_,{children:[(0,x.jsx)(_.Header,{children:`VPC Information`}),(0,x.jsxs)(_.Content,{children:[(0,x.jsx)(v,{label:`VPC ID`,value:C.VpcId}),(0,x.jsx)(v,{label:`State`,value:C.State}),(0,x.jsx)(v,{label:`CIDR Block`,value:C.CidrBlock}),(0,x.jsx)(v,{label:`Is Default`,value:C.IsDefault?`Yes`:`No`}),(0,x.jsx)(v,{label:`DHCP Options ID`,value:C.DhcpOptionsId}),(0,x.jsx)(v,{label:`Instance Tenancy`,value:C.InstanceTenancy}),(0,x.jsx)(v,{label:`Owner ID`,value:C.OwnerId})]})]}),D.length>0&&(0,x.jsxs)(_,{children:[(0,x.jsxs)(_.Header,{children:[`Subnets (`,D.length,`)`]}),D.map(e=>e.SubnetId?(0,x.jsxs)(_.Content,{children:[(0,x.jsx)(v,{label:`Subnet ID`,value:(0,x.jsx)(a,{className:`text-primary hover:underline`,params:{id:e.SubnetId},to:`/ec2/describe-subnets/$id`,children:e.SubnetId})}),(0,x.jsx)(v,{label:`CIDR Block`,value:e.CidrBlock}),(0,x.jsx)(v,{label:`Availability Zone`,value:e.AvailabilityZone}),(0,x.jsx)(v,{label:`State`,value:e.State})]},e.SubnetId):null)]})]}),(0,x.jsx)(y,{description:`Are you sure you want to delete the VPC "${C.VpcId}"? This action cannot be undone. The VPC must have no associated subnets or internet gateways.`,isPending:w.isPending,onConfirm:O,onOpenChange:E,open:T,title:`Delete VPC`})]})}export{S as component};
+import{t as e}from"./error-banner.js";import{t}from"./useSuspenseQuery.js";import{t as n}from"./back-link.js";import{t as r}from"./trash-2.js";import{x as i}from"./ec2.js";import{Da as a,Do as o,Q as s,co as c,f as l,no as u,ro as d,so as f,tt as p,ya as m}from"./index.js";import{t as h}from"./page-heading.js";import{t as g}from"./state-badge.js";import{n as _,t as v}from"./detail-row.js";import{t as y}from"./delete-confirmation-dialog.js";var b=o(c()),x=f();function S(){let{id:o}=l.useParams(),c=d(),{data:f}=t(p(o)),{data:S}=t(s),C=f.Vpcs?.[0],w=i(),[T,E]=(0,b.useState)(!1),D=S.Subnets?.filter(e=>e.VpcId===o)??[],O=async()=>{try{await w.mutateAsync(o),c({to:`/ec2/describe-vpcs`})}finally{E(!1)}};if(!C?.VpcId)return(0,x.jsxs)(x.Fragment,{children:[(0,x.jsx)(n,{to:`/ec2/describe-vpcs`,children:`Back to VPCs`}),(0,x.jsx)(`p`,{className:`text-muted-foreground`,children:`VPC not found.`})]});let k=a(C.Tags);return(0,x.jsxs)(x.Fragment,{children:[(0,x.jsx)(n,{to:`/ec2/describe-vpcs`,children:`Back to VPCs`}),w.error&&(0,x.jsx)(e,{error:w.error,msg:`Failed to delete VPC`}),(0,x.jsxs)(`div`,{className:`space-y-6`,children:[(0,x.jsx)(h,{actions:(0,x.jsxs)(`div`,{className:`flex items-center gap-2`,children:[(0,x.jsxs)(m,{onClick:()=>E(!0),size:`sm`,variant:`destructive`,children:[(0,x.jsx)(r,{className:`size-4`}),`Delete`]}),(0,x.jsx)(g,{state:C.State})]}),subtitle:`VPC Details`,title:k?`${C.VpcId} (${k})`:C.VpcId}),(0,x.jsxs)(_,{children:[(0,x.jsx)(_.Header,{children:`VPC Information`}),(0,x.jsxs)(_.Content,{children:[(0,x.jsx)(v,{label:`VPC ID`,value:C.VpcId}),(0,x.jsx)(v,{label:`State`,value:C.State}),(0,x.jsx)(v,{label:`CIDR Block`,value:C.CidrBlock}),(0,x.jsx)(v,{label:`Is Default`,value:C.IsDefault?`Yes`:`No`}),(0,x.jsx)(v,{label:`DHCP Options ID`,value:C.DhcpOptionsId}),(0,x.jsx)(v,{label:`Instance Tenancy`,value:C.InstanceTenancy}),(0,x.jsx)(v,{label:`Owner ID`,value:C.OwnerId})]})]}),D.length>0&&(0,x.jsxs)(_,{children:[(0,x.jsxs)(_.Header,{children:[`Subnets (`,D.length,`)`]}),D.map(e=>e.SubnetId?(0,x.jsxs)(_.Content,{children:[(0,x.jsx)(v,{label:`Subnet ID`,value:(0,x.jsx)(u,{className:`text-primary hover:underline`,params:{id:e.SubnetId},to:`/ec2/describe-subnets/$id`,children:e.SubnetId})}),(0,x.jsx)(v,{label:`CIDR Block`,value:e.CidrBlock}),(0,x.jsx)(v,{label:`Availability Zone`,value:e.AvailabilityZone}),(0,x.jsx)(v,{label:`State`,value:e.State})]},e.SubnetId):null)]})]}),(0,x.jsx)(y,{description:`Are you sure you want to delete the VPC "${C.VpcId}"? This action cannot be undone. The VPC must have no associated subnets or internet gateways.`,isPending:w.isPending,onConfirm:O,onOpenChange:E,open:T,title:`Delete VPC`})]})}export{S as component};